2024-03-18T15:41:58.972 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-18T15:41:58.973 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_pdonnell@teuthology --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/ffsb}} --archive /home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608907 --name pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi --verbose -- /home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608907/orig.config.yaml 2024-03-18T15:41:59.032 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-18T15:41:59.119 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608907 branch: wip-batrick-testing-20240314.183243 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/ffsb}} email: pdonnell@redhat.com first_in_suite: false job_id: '7608907' kernel: client: branch: testing kdb: true sha1: distro ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-batrick-testing-20240314.183243 ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 5 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 filestore flush min: 0 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd heartbeat grace: 60 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - mons down, quorum - out of quorum - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN - slow metadata IO - SLOW_OPS - slow request - SLOW_OPS - slow request sha1: 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af subvols: create: 2 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - mds.dir_split install: ceph: flavor: default sha1: 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=secure - wsync syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-batrick-testing-20240314.183243 sha1: 4e19f704afd9957ea657ab858f9ecf76401fef6f owner: scheduled_pdonnell@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 4770 sha1: 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af sleep_before_teardown: 0 subset: 498/512 suite: fs suite_branch: wip-batrick-testing-20240314.183243 suite_path: /home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa suite_relpath: qa suite_repo: https://github.com/batrick/ceph suite_sha1: 4e19f704afd9957ea657ab858f9ecf76401fef6f syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi039.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHY1MDDrpKaPt02pg1LDMFSgGmc2dWtanXhMrgFGTcvqcQ0jx/3Sv8LUP0LN1UAcD79+qY1LEySKgfl2VsGvS4o= smithi194.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBfF0Vvutt5ltBy/buw4FS4VynvDztlWx3R/4TOA1t6sFlVqtej5vckm1rwGO6Aahuqy0ZWUiRVT+0nF1HTpNvM= smithi196.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKcVJ3bbZUAz5uSEqaEYjMFAq4WHt9Vv/N04A/u1+3Jbn4w7pTCkeOmOXuHBfzVxz6J1WmPK7gwbfF8ZPEPLLSA= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate true - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - workunit: clients: all: - suites/ffsb.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-18_14:19:46 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-18T15:41:59.120 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa; will attempt to use it 2024-03-18T15:41:59.120 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_batrick_ceph_4e19f704afd9957ea657ab858f9ecf76401fef6f/qa/tasks 2024-03-18T15:41:59.121 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-18T15:41:59.122 INFO:teuthology.task.internal:Checking packages... 2024-03-18T15:41:59.146 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af' 2024-03-18T15:41:59.147 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-18T15:41:59.147 INFO:teuthology.packaging:ref: None 2024-03-18T15:41:59.147 INFO:teuthology.packaging:tag: None 2024-03-18T15:41:59.147 INFO:teuthology.packaging:branch: wip-batrick-testing-20240314.183243 2024-03-18T15:41:59.147 INFO:teuthology.packaging:sha1: 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T15:41:59.147 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-batrick-testing-20240314.183243 2024-03-18T15:41:59.399 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2162.g3d78f5ba 2024-03-18T15:41:59.400 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-18T15:41:59.407 INFO:teuthology.task.internal:no buildpackages task found 2024-03-18T15:41:59.407 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-18T15:41:59.448 INFO:teuthology.task.internal:Saving configuration 2024-03-18T15:41:59.468 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-18T15:41:59.473 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-18T15:41:59.489 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi039.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608907', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-18 15:36:02.329550', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHY1MDDrpKaPt02pg1LDMFSgGmc2dWtanXhMrgFGTcvqcQ0jx/3Sv8LUP0LN1UAcD79+qY1LEySKgfl2VsGvS4o='} 2024-03-18T15:41:59.508 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi194.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608907', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-18 15:36:02.332011', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBfF0Vvutt5ltBy/buw4FS4VynvDztlWx3R/4TOA1t6sFlVqtej5vckm1rwGO6Aahuqy0ZWUiRVT+0nF1HTpNvM='} 2024-03-18T15:41:59.531 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi196.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608907', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-18 15:36:02.333123', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKcVJ3bbZUAz5uSEqaEYjMFAq4WHt9Vv/N04A/u1+3Jbn4w7pTCkeOmOXuHBfzVxz6J1WmPK7gwbfF8ZPEPLLSA='} 2024-03-18T15:41:59.531 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-18T15:41:59.537 INFO:teuthology.task.internal:roles: ubuntu@smithi039.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-03-18T15:41:59.538 INFO:teuthology.task.internal:roles: ubuntu@smithi194.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-03-18T15:41:59.538 INFO:teuthology.task.internal:roles: ubuntu@smithi196.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-03-18T15:41:59.538 INFO:teuthology.run_tasks:Running task console_log... 2024-03-18T15:41:59.656 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f4792351160>, signals=[15]) 2024-03-18T15:41:59.657 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-18T15:41:59.661 INFO:teuthology.task.internal:Opening connections... 2024-03-18T15:41:59.661 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi039.front.sepia.ceph.com 2024-03-18T15:41:59.664 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T15:41:59.778 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi194.front.sepia.ceph.com 2024-03-18T15:41:59.779 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi194.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T15:41:59.886 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi196.front.sepia.ceph.com 2024-03-18T15:41:59.887 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T15:42:00.001 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-18T15:42:00.009 DEBUG:teuthology.orchestra.run.smithi039:> uname -m 2024-03-18T15:42:00.032 INFO:teuthology.orchestra.run.smithi039.stdout:x86_64 2024-03-18T15:42:00.032 DEBUG:teuthology.orchestra.run.smithi039:> cat /etc/os-release 2024-03-18T15:42:00.090 INFO:teuthology.orchestra.run.smithi039.stdout:NAME="CentOS Stream" 2024-03-18T15:42:00.091 INFO:teuthology.orchestra.run.smithi039.stdout:VERSION="9" 2024-03-18T15:42:00.091 INFO:teuthology.orchestra.run.smithi039.stdout:ID="centos" 2024-03-18T15:42:00.091 INFO:teuthology.orchestra.run.smithi039.stdout:ID_LIKE="rhel fedora" 2024-03-18T15:42:00.091 INFO:teuthology.orchestra.run.smithi039.stdout:VERSION_ID="9" 2024-03-18T15:42:00.091 INFO:teuthology.orchestra.run.smithi039.stdout:PLATFORM_ID="platform:el9" 2024-03-18T15:42:00.091 INFO:teuthology.orchestra.run.smithi039.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-18T15:42:00.092 INFO:teuthology.orchestra.run.smithi039.stdout:ANSI_COLOR="0;31" 2024-03-18T15:42:00.092 INFO:teuthology.orchestra.run.smithi039.stdout:LOGO="fedora-logo-icon" 2024-03-18T15:42:00.092 INFO:teuthology.orchestra.run.smithi039.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-18T15:42:00.092 INFO:teuthology.orchestra.run.smithi039.stdout:HOME_URL="https://centos.org/" 2024-03-18T15:42:00.092 INFO:teuthology.orchestra.run.smithi039.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-18T15:42:00.092 INFO:teuthology.orchestra.run.smithi039.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-18T15:42:00.093 INFO:teuthology.orchestra.run.smithi039.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-18T15:42:00.094 INFO:teuthology.lock.ops:Updating smithi039.front.sepia.ceph.com on lock server 2024-03-18T15:42:00.115 DEBUG:teuthology.orchestra.run.smithi194:> uname -m 2024-03-18T15:42:00.132 INFO:teuthology.orchestra.run.smithi194.stdout:x86_64 2024-03-18T15:42:00.133 DEBUG:teuthology.orchestra.run.smithi194:> cat /etc/os-release 2024-03-18T15:42:00.189 INFO:teuthology.orchestra.run.smithi194.stdout:NAME="CentOS Stream" 2024-03-18T15:42:00.190 INFO:teuthology.orchestra.run.smithi194.stdout:VERSION="9" 2024-03-18T15:42:00.190 INFO:teuthology.orchestra.run.smithi194.stdout:ID="centos" 2024-03-18T15:42:00.190 INFO:teuthology.orchestra.run.smithi194.stdout:ID_LIKE="rhel fedora" 2024-03-18T15:42:00.190 INFO:teuthology.orchestra.run.smithi194.stdout:VERSION_ID="9" 2024-03-18T15:42:00.190 INFO:teuthology.orchestra.run.smithi194.stdout:PLATFORM_ID="platform:el9" 2024-03-18T15:42:00.190 INFO:teuthology.orchestra.run.smithi194.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-18T15:42:00.190 INFO:teuthology.orchestra.run.smithi194.stdout:ANSI_COLOR="0;31" 2024-03-18T15:42:00.190 INFO:teuthology.orchestra.run.smithi194.stdout:LOGO="fedora-logo-icon" 2024-03-18T15:42:00.190 INFO:teuthology.orchestra.run.smithi194.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-18T15:42:00.190 INFO:teuthology.orchestra.run.smithi194.stdout:HOME_URL="https://centos.org/" 2024-03-18T15:42:00.190 INFO:teuthology.orchestra.run.smithi194.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-18T15:42:00.190 INFO:teuthology.orchestra.run.smithi194.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-18T15:42:00.191 INFO:teuthology.orchestra.run.smithi194.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-18T15:42:00.191 INFO:teuthology.lock.ops:Updating smithi194.front.sepia.ceph.com on lock server 2024-03-18T15:42:00.206 DEBUG:teuthology.orchestra.run.smithi196:> uname -m 2024-03-18T15:42:00.223 INFO:teuthology.orchestra.run.smithi196.stdout:x86_64 2024-03-18T15:42:00.224 DEBUG:teuthology.orchestra.run.smithi196:> cat /etc/os-release 2024-03-18T15:42:00.281 INFO:teuthology.orchestra.run.smithi196.stdout:NAME="CentOS Stream" 2024-03-18T15:42:00.281 INFO:teuthology.orchestra.run.smithi196.stdout:VERSION="9" 2024-03-18T15:42:00.281 INFO:teuthology.orchestra.run.smithi196.stdout:ID="centos" 2024-03-18T15:42:00.281 INFO:teuthology.orchestra.run.smithi196.stdout:ID_LIKE="rhel fedora" 2024-03-18T15:42:00.281 INFO:teuthology.orchestra.run.smithi196.stdout:VERSION_ID="9" 2024-03-18T15:42:00.281 INFO:teuthology.orchestra.run.smithi196.stdout:PLATFORM_ID="platform:el9" 2024-03-18T15:42:00.281 INFO:teuthology.orchestra.run.smithi196.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-18T15:42:00.281 INFO:teuthology.orchestra.run.smithi196.stdout:ANSI_COLOR="0;31" 2024-03-18T15:42:00.282 INFO:teuthology.orchestra.run.smithi196.stdout:LOGO="fedora-logo-icon" 2024-03-18T15:42:00.282 INFO:teuthology.orchestra.run.smithi196.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-18T15:42:00.282 INFO:teuthology.orchestra.run.smithi196.stdout:HOME_URL="https://centos.org/" 2024-03-18T15:42:00.282 INFO:teuthology.orchestra.run.smithi196.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-18T15:42:00.282 INFO:teuthology.orchestra.run.smithi196.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-18T15:42:00.282 INFO:teuthology.orchestra.run.smithi196.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-18T15:42:00.282 INFO:teuthology.lock.ops:Updating smithi196.front.sepia.ceph.com on lock server 2024-03-18T15:42:00.306 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-18T15:42:00.316 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-18T15:42:00.349 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-18T15:42:00.349 DEBUG:teuthology.orchestra.run.smithi039:> test '!' -e /home/ubuntu/cephtest 2024-03-18T15:42:00.352 DEBUG:teuthology.orchestra.run.smithi194:> test '!' -e /home/ubuntu/cephtest 2024-03-18T15:42:00.354 DEBUG:teuthology.orchestra.run.smithi196:> test '!' -e /home/ubuntu/cephtest 2024-03-18T15:42:00.370 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-18T15:42:00.376 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-18T15:42:00.376 DEBUG:teuthology.orchestra.run.smithi039:> test -z $(ls -A /var/lib/ceph) 2024-03-18T15:42:00.409 DEBUG:teuthology.orchestra.run.smithi194:> test -z $(ls -A /var/lib/ceph) 2024-03-18T15:42:00.412 DEBUG:teuthology.orchestra.run.smithi196:> test -z $(ls -A /var/lib/ceph) 2024-03-18T15:42:00.706 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-18T15:42:00.785 INFO:teuthology.run_tasks:Running task kernel... 2024-03-18T15:42:00.800 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'} 2024-03-18T15:42:00.800 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2024-03-18T15:42:00.800 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2024-03-18T15:42:00.800 DEBUG:teuthology.orchestra.run.smithi039:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-18T15:42:00.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T15:42:00.818 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-03-18T15:42:01.051 DEBUG:teuthology.task.kernel:sha1 for client.0 is 3845b7ac715a3ff198582638d199a90e30e254da 2024-03-18T15:42:01.051 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "3845b7ac715a3ff198582638d199a90e30e254da"... 2024-03-18T15:42:01.052 DEBUG:teuthology.orchestra.run.smithi039:> uname -r 2024-03-18T15:42:01.069 INFO:teuthology.orchestra.run.smithi039.stdout:5.14.0-361.el9.x86_64 2024-03-18T15:42:01.069 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs 3845b7ac715a3ff198582638d199a90e30e254da 2024-03-18T15:42:01.070 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-03-18T15:42:01.070 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum upgrade -y linux-firmware 2024-03-18T15:42:03.136 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-18T15:42:04.669 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS-9 - BaseOS 6.5 MB/s | 2.0 MB 00:00 2024-03-18T15:42:14.151 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS Stream 9 - BaseOS 855 kB/s | 8.0 MB 00:09 2024-03-18T15:42:37.583 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS Stream 9 - AppStream 866 kB/s | 19 MB 00:22 2024-03-18T15:42:42.141 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS Stream 9 - CRB 7.1 MB/s | 6.1 MB 00:00 2024-03-18T15:42:45.825 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS Stream 9 - Extras packages 6.4 kB/s | 16 kB 00:02 2024-03-18T15:43:06.227 INFO:teuthology.orchestra.run.smithi039.stdout:Extra Packages for Enterprise Linux 1.0 MB/s | 21 MB 00:20 2024-03-18T15:43:11.349 INFO:teuthology.orchestra.run.smithi039.stdout:lab-extras 45 kB/s | 1.7 kB 00:00 2024-03-18T15:43:13.881 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-03-18T15:43:13.885 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-03-18T15:43:13.885 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repository Size 2024-03-18T15:43:13.885 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-03-18T15:43:13.885 INFO:teuthology.orchestra.run.smithi039.stdout:Upgrading: 2024-03-18T15:43:13.885 INFO:teuthology.orchestra.run.smithi039.stdout: linux-firmware noarch 20240219-143.el9 CentOS-BaseOS 375 M 2024-03-18T15:43:13.886 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T15:43:13.886 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2024-03-18T15:43:13.886 INFO:teuthology.orchestra.run.smithi039.stdout:================================================================================ 2024-03-18T15:43:13.886 INFO:teuthology.orchestra.run.smithi039.stdout:Upgrade 1 Package 2024-03-18T15:43:13.886 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T15:43:13.886 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 375 M 2024-03-18T15:43:13.886 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2024-03-18T15:43:23.174 INFO:teuthology.orchestra.run.smithi039.stdout:linux-firmware-20240219-143.el9.noarch.rpm 40 MB/s | 375 MB 00:09 2024-03-18T15:43:23.175 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2024-03-18T15:43:23.176 INFO:teuthology.orchestra.run.smithi039.stdout:Total 40 MB/s | 375 MB 00:09 2024-03-18T15:43:23.176 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2024-03-18T15:43:23.270 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2024-03-18T15:43:23.270 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2024-03-18T15:43:25.765 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2024-03-18T15:43:25.765 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2024-03-18T15:43:32.433 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2024-03-18T15:43:32.806 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-18T15:43:32.892 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-18T15:43:34.106 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2024-03-18T15:43:34.106 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-18T15:43:34.624 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-18T15:43:34.624 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T15:43:34.625 INFO:teuthology.orchestra.run.smithi039.stdout:Upgraded: 2024-03-18T15:43:34.625 INFO:teuthology.orchestra.run.smithi039.stdout: linux-firmware-20240219-143.el9.noarch 2024-03-18T15:43:34.625 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T15:43:34.626 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-03-18T15:43:34.908 INFO:teuthology.task.kernel:Downloading kernel 3845b7ac715a3ff198582638d199a90e30e254da on client.0... 2024-03-18T15:43:34.908 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3845b7ac715a3ff198582638d199a90e30e254da 2024-03-18T15:43:35.064 INFO:teuthology.task.kernel:fetching, builder baseurl is https://2.chacra.ceph.com/r/kernel/testing/3845b7ac715a3ff198582638d199a90e30e254da/centos/9/flavors/default/x86_64/ 2024-03-18T15:43:35.064 DEBUG:teuthology.orchestra.run.smithi039:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.8.0_rc7_g3845b7ac715a-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://2.chacra.ceph.com/r/kernel/testing/3845b7ac715a3ff198582638d199a90e30e254da/centos/9/flavors/default/x86_64/ --input-file=- 2024-03-18T15:43:35.067 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi039.front.sepia.ceph.com to complete... 2024-03-18T15:44:11.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18 15:44:11 URL:https://2.chacra.ceph.com/r/kernel/testing/3845b7ac715a3ff198582638d199a90e30e254da/centos/9/flavors/default/x86_64/kernel-6.8.0_rc7_g3845b7ac715a-1.x86_64.rpm [336075563/336075563] -> "/tmp/kernel.x86_64.rpm" [1] 2024-03-18T15:44:11.621 INFO:teuthology.orchestra.run.smithi039.stderr:FINISHED --2024-03-18 15:44:11-- 2024-03-18T15:44:11.622 INFO:teuthology.orchestra.run.smithi039.stderr:Total wall clock time: 37s 2024-03-18T15:44:11.622 INFO:teuthology.orchestra.run.smithi039.stderr:Downloaded: 1 files, 321M in 36s (8.89 MB/s) 2024-03-18T15:44:11.706 INFO:teuthology.task.kernel:Installing kernel 3845b7ac715a3ff198582638d199a90e30e254da on client.0... 2024-03-18T15:44:11.706 DEBUG:teuthology.orchestra.run.smithi039:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-03-18T15:44:13.740 INFO:teuthology.orchestra.run.smithi039.stdout:Verifying... ######################################## 2024-03-18T15:44:13.793 INFO:teuthology.orchestra.run.smithi039.stdout:Preparing... ######################################## 2024-03-18T15:44:13.877 INFO:teuthology.orchestra.run.smithi039.stdout:Updating / installing... 2024-03-18T15:44:21.726 INFO:teuthology.orchestra.run.smithi039.stdout:kernel-6.8.0_rc7_g3845b7ac715a-1 ######################################## 2024-03-18T15:44:36.028 INFO:teuthology.orchestra.run.smithi039.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-18T15:44:59.225 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi039.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-03-18T15:44:59.225 DEBUG:teuthology.orchestra.run.smithi039:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-03-18T15:44:59.299 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a 2024-03-18T15:44:59.300 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/System.map 2024-03-18T15:44:59.300 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/config 2024-03-18T15:44:59.300 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel 2024-03-18T15:44:59.300 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch 2024-03-18T15:44:59.300 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86 2024-03-18T15:44:59.300 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto 2024-03-18T15:44:59.300 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-03-18T15:44:59.300 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-03-18T15:44:59.300 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-03-18T15:44:59.300 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-03-18T15:44:59.300 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-03-18T15:44:59.300 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-03-18T15:44:59.301 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-03-18T15:44:59.301 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/crc32c-intel.ko 2024-03-18T15:44:59.301 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-03-18T15:44:59.301 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-03-18T15:44:59.301 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-03-18T15:44:59.301 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-03-18T15:44:59.301 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/serpent-avx2.ko 2024-03-18T15:44:59.301 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-03-18T15:44:59.301 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-03-18T15:44:59.301 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-03-18T15:44:59.301 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-03-18T15:44:59.301 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-03-18T15:44:59.301 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-03-18T15:44:59.301 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-03-18T15:44:59.301 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/kernel 2024-03-18T15:44:59.302 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/kernel/cpu 2024-03-18T15:44:59.302 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/kernel/cpu/mce 2024-03-18T15:44:59.302 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-03-18T15:44:59.302 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/kvm 2024-03-18T15:44:59.302 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/kvm/kvm-amd.ko 2024-03-18T15:44:59.302 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/kvm/kvm-intel.ko 2024-03-18T15:44:59.302 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/arch/x86/kvm/kvm.ko 2024-03-18T15:44:59.302 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto 2024-03-18T15:44:59.302 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/ansi_cprng.ko 2024-03-18T15:44:59.302 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/anubis.ko 2024-03-18T15:44:59.302 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/arc4.ko 2024-03-18T15:44:59.302 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/async_tx 2024-03-18T15:44:59.302 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/async_tx/async_memcpy.ko 2024-03-18T15:44:59.302 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/async_tx/async_pq.ko 2024-03-18T15:44:59.303 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/async_tx/async_raid6_recov.ko 2024-03-18T15:44:59.303 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/async_tx/async_tx.ko 2024-03-18T15:44:59.303 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/async_tx/async_xor.ko 2024-03-18T15:44:59.303 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/async_tx/raid6test.ko 2024-03-18T15:44:59.303 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/authenc.ko 2024-03-18T15:44:59.303 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/authencesn.ko 2024-03-18T15:44:59.303 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/blake2b_generic.ko 2024-03-18T15:44:59.303 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/blowfish_common.ko 2024-03-18T15:44:59.303 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/blowfish_generic.ko 2024-03-18T15:44:59.303 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/camellia_generic.ko 2024-03-18T15:44:59.303 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/cast5_generic.ko 2024-03-18T15:44:59.303 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/cast6_generic.ko 2024-03-18T15:44:59.303 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/cast_common.ko 2024-03-18T15:44:59.303 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/ccm.ko 2024-03-18T15:44:59.304 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/cmac.ko 2024-03-18T15:44:59.304 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/crc32_generic.ko 2024-03-18T15:44:59.304 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/crypto_engine.ko 2024-03-18T15:44:59.304 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/crypto_user.ko 2024-03-18T15:44:59.304 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/cts.ko 2024-03-18T15:44:59.304 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/des_generic.ko 2024-03-18T15:44:59.304 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/dh_generic.ko 2024-03-18T15:44:59.304 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/echainiv.ko 2024-03-18T15:44:59.304 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/essiv.ko 2024-03-18T15:44:59.304 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/fcrypt.ko 2024-03-18T15:44:59.304 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/khazad.ko 2024-03-18T15:44:59.304 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/lz4.ko 2024-03-18T15:44:59.304 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/lz4hc.ko 2024-03-18T15:44:59.304 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/md4.ko 2024-03-18T15:44:59.305 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/michael_mic.ko 2024-03-18T15:44:59.305 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/pcbc.ko 2024-03-18T15:44:59.305 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/pcrypt.ko 2024-03-18T15:44:59.305 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/rmd160.ko 2024-03-18T15:44:59.305 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/seed.ko 2024-03-18T15:44:59.305 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/serpent_generic.ko 2024-03-18T15:44:59.305 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/tcrypt.ko 2024-03-18T15:44:59.305 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/tea.ko 2024-03-18T15:44:59.305 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/twofish_common.ko 2024-03-18T15:44:59.305 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/twofish_generic.ko 2024-03-18T15:44:59.305 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/vmac.ko 2024-03-18T15:44:59.305 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/wp512.ko 2024-03-18T15:44:59.305 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/xcbc.ko 2024-03-18T15:44:59.305 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/xor.ko 2024-03-18T15:44:59.305 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/crypto/xxhash_generic.ko 2024-03-18T15:44:59.306 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers 2024-03-18T15:44:59.306 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/acpi 2024-03-18T15:44:59.306 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/acpi/acpi_ipmi.ko 2024-03-18T15:44:59.306 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/acpi/acpi_pad.ko 2024-03-18T15:44:59.306 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/acpi/custom_method.ko 2024-03-18T15:44:59.306 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/acpi/ec_sys.ko 2024-03-18T15:44:59.306 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/acpi/platform_profile.ko 2024-03-18T15:44:59.306 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/acpi/sbs.ko 2024-03-18T15:44:59.306 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/acpi/sbshc.ko 2024-03-18T15:44:59.306 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/acpi/video.ko 2024-03-18T15:44:59.306 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata 2024-03-18T15:44:59.306 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/acard-ahci.ko 2024-03-18T15:44:59.306 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/ahci_platform.ko 2024-03-18T15:44:59.306 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/ata_generic.ko 2024-03-18T15:44:59.306 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/libahci_platform.ko 2024-03-18T15:44:59.307 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_acpi.ko 2024-03-18T15:44:59.307 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_ali.ko 2024-03-18T15:44:59.307 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_amd.ko 2024-03-18T15:44:59.307 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_artop.ko 2024-03-18T15:44:59.307 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_atiixp.ko 2024-03-18T15:44:59.307 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_atp867x.ko 2024-03-18T15:44:59.307 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_cmd640.ko 2024-03-18T15:44:59.307 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_cmd64x.ko 2024-03-18T15:44:59.307 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_cypress.ko 2024-03-18T15:44:59.307 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_efar.ko 2024-03-18T15:44:59.307 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_hpt366.ko 2024-03-18T15:44:59.307 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_hpt37x.ko 2024-03-18T15:44:59.307 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_hpt3x2n.ko 2024-03-18T15:44:59.308 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_hpt3x3.ko 2024-03-18T15:44:59.308 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_it8213.ko 2024-03-18T15:44:59.308 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_it821x.ko 2024-03-18T15:44:59.308 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_jmicron.ko 2024-03-18T15:44:59.308 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_marvell.ko 2024-03-18T15:44:59.308 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_mpiix.ko 2024-03-18T15:44:59.308 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_netcell.ko 2024-03-18T15:44:59.308 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_ninja32.ko 2024-03-18T15:44:59.308 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_ns87410.ko 2024-03-18T15:44:59.308 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_ns87415.ko 2024-03-18T15:44:59.308 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_oldpiix.ko 2024-03-18T15:44:59.308 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_opti.ko 2024-03-18T15:44:59.308 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_optidma.ko 2024-03-18T15:44:59.309 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_pcmcia.ko 2024-03-18T15:44:59.309 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_pdc2027x.ko 2024-03-18T15:44:59.309 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-03-18T15:44:59.309 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_piccolo.ko 2024-03-18T15:44:59.309 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_rdc.ko 2024-03-18T15:44:59.309 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_sch.ko 2024-03-18T15:44:59.309 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_serverworks.ko 2024-03-18T15:44:59.309 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_sil680.ko 2024-03-18T15:44:59.309 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_sis.ko 2024-03-18T15:44:59.309 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_sl82c105.ko 2024-03-18T15:44:59.309 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_triflex.ko 2024-03-18T15:44:59.309 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pata_via.ko 2024-03-18T15:44:59.309 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/pdc_adma.ko 2024-03-18T15:44:59.309 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/sata_inic162x.ko 2024-03-18T15:44:59.310 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/sata_mv.ko 2024-03-18T15:44:59.310 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/sata_nv.ko 2024-03-18T15:44:59.310 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/sata_promise.ko 2024-03-18T15:44:59.310 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/sata_qstor.ko 2024-03-18T15:44:59.310 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/sata_sil.ko 2024-03-18T15:44:59.310 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/sata_sil24.ko 2024-03-18T15:44:59.310 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/sata_sis.ko 2024-03-18T15:44:59.310 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/sata_svw.ko 2024-03-18T15:44:59.310 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/sata_sx4.ko 2024-03-18T15:44:59.310 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/sata_uli.ko 2024-03-18T15:44:59.310 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/sata_via.ko 2024-03-18T15:44:59.310 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ata/sata_vsc.ko 2024-03-18T15:44:59.310 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/auxdisplay 2024-03-18T15:44:59.310 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/auxdisplay/ks0108.ko 2024-03-18T15:44:59.311 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/base 2024-03-18T15:44:59.311 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/base/regmap 2024-03-18T15:44:59.311 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/base/regmap/regmap-i2c.ko 2024-03-18T15:44:59.311 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/base/regmap/regmap-mmio.ko 2024-03-18T15:44:59.311 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/bcma 2024-03-18T15:44:59.311 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/bcma/bcma.ko 2024-03-18T15:44:59.311 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block 2024-03-18T15:44:59.311 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/aoe 2024-03-18T15:44:59.311 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/aoe/aoe.ko 2024-03-18T15:44:59.311 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/brd.ko 2024-03-18T15:44:59.311 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/drbd 2024-03-18T15:44:59.311 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/drbd/drbd.ko 2024-03-18T15:44:59.311 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/floppy.ko 2024-03-18T15:44:59.311 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/loop.ko 2024-03-18T15:44:59.312 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/mtip32xx 2024-03-18T15:44:59.312 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-03-18T15:44:59.312 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/nbd.ko 2024-03-18T15:44:59.312 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/null_blk 2024-03-18T15:44:59.312 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/null_blk/null_blk.ko 2024-03-18T15:44:59.312 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/pktcdvd.ko 2024-03-18T15:44:59.312 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/rbd.ko 2024-03-18T15:44:59.312 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/virtio_blk.ko 2024-03-18T15:44:59.312 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/xen-blkback 2024-03-18T15:44:59.312 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-03-18T15:44:59.312 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/xen-blkfront.ko 2024-03-18T15:44:59.312 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/zram 2024-03-18T15:44:59.312 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/block/zram/zram.ko 2024-03-18T15:44:59.312 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char 2024-03-18T15:44:59.312 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/hangcheck-timer.ko 2024-03-18T15:44:59.313 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/hw_random 2024-03-18T15:44:59.313 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/hw_random/amd-rng.ko 2024-03-18T15:44:59.313 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/hw_random/intel-rng.ko 2024-03-18T15:44:59.313 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-03-18T15:44:59.313 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/hw_random/via-rng.ko 2024-03-18T15:44:59.313 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/hw_random/virtio-rng.ko 2024-03-18T15:44:59.313 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/ipmi 2024-03-18T15:44:59.313 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-03-18T15:44:59.313 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-03-18T15:44:59.313 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-03-18T15:44:59.313 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/ipmi/ipmi_si.ko 2024-03-18T15:44:59.313 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-03-18T15:44:59.313 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-03-18T15:44:59.313 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/lp.ko 2024-03-18T15:44:59.314 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/mwave 2024-03-18T15:44:59.314 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/mwave/mwave.ko 2024-03-18T15:44:59.314 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/ppdev.ko 2024-03-18T15:44:59.314 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/tlclk.ko 2024-03-18T15:44:59.314 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/tpm 2024-03-18T15:44:59.314 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/tpm/tpm.ko 2024-03-18T15:44:59.314 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/tpm/tpm_atmel.ko 2024-03-18T15:44:59.314 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/tpm/tpm_infineon.ko 2024-03-18T15:44:59.314 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/tpm/tpm_nsc.ko 2024-03-18T15:44:59.314 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/tpm/tpm_tis.ko 2024-03-18T15:44:59.314 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-03-18T15:44:59.314 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/uv_mmtimer.ko 2024-03-18T15:44:59.314 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/char/virtio_console.ko 2024-03-18T15:44:59.314 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/cpufreq 2024-03-18T15:44:59.315 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-03-18T15:44:59.315 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-03-18T15:44:59.315 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/cpufreq/p4-clockmod.ko 2024-03-18T15:44:59.315 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-03-18T15:44:59.315 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/cpufreq/powernow-k8.ko 2024-03-18T15:44:59.315 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/cpufreq/speedstep-lib.ko 2024-03-18T15:44:59.315 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto 2024-03-18T15:44:59.315 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto/ccp 2024-03-18T15:44:59.315 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-03-18T15:44:59.315 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto/ccp/ccp.ko 2024-03-18T15:44:59.316 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto/intel 2024-03-18T15:44:59.316 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto/intel/qat 2024-03-18T15:44:59.316 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto/intel/qat/qat_common 2024-03-18T15:44:59.316 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-03-18T15:44:59.316 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-03-18T15:44:59.316 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-03-18T15:44:59.316 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto/padlock-aes.ko 2024-03-18T15:44:59.316 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto/padlock-sha.ko 2024-03-18T15:44:59.316 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto/virtio 2024-03-18T15:44:59.316 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-03-18T15:44:59.316 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/dca 2024-03-18T15:44:59.316 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/dca/dca.ko 2024-03-18T15:44:59.316 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/devfreq 2024-03-18T15:44:59.316 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-03-18T15:44:59.316 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/dma 2024-03-18T15:44:59.317 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/dma/dw 2024-03-18T15:44:59.317 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/dma/dw/dw_dmac.ko 2024-03-18T15:44:59.317 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/dma/ioat 2024-03-18T15:44:59.317 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/dma/ioat/ioatdma.ko 2024-03-18T15:44:59.317 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac 2024-03-18T15:44:59.317 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac/amd64_edac.ko 2024-03-18T15:44:59.317 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac/e752x_edac.ko 2024-03-18T15:44:59.317 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac/edac_mce_amd.ko 2024-03-18T15:44:59.317 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac/i3000_edac.ko 2024-03-18T15:44:59.317 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac/i3200_edac.ko 2024-03-18T15:44:59.317 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac/i5100_edac.ko 2024-03-18T15:44:59.317 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac/i5400_edac.ko 2024-03-18T15:44:59.317 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac/i7300_edac.ko 2024-03-18T15:44:59.317 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac/i7core_edac.ko 2024-03-18T15:44:59.318 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac/i82975x_edac.ko 2024-03-18T15:44:59.318 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac/ie31200_edac.ko 2024-03-18T15:44:59.318 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac/sb_edac.ko 2024-03-18T15:44:59.318 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/edac/x38_edac.ko 2024-03-18T15:44:59.318 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/firmware 2024-03-18T15:44:59.318 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/firmware/edd.ko 2024-03-18T15:44:59.318 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/firmware/iscsi_ibft.ko 2024-03-18T15:44:59.318 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/gpio 2024-03-18T15:44:59.318 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/gpio/gpio-ich.ko 2024-03-18T15:44:59.318 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/gpio/gpio-viperboard.ko 2024-03-18T15:44:59.318 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid 2024-03-18T15:44:59.318 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-appleir.ko 2024-03-18T15:44:59.318 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-aureal.ko 2024-03-18T15:44:59.319 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-axff.ko 2024-03-18T15:44:59.319 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-dr.ko 2024-03-18T15:44:59.319 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-elecom.ko 2024-03-18T15:44:59.319 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-elo.ko 2024-03-18T15:44:59.319 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-emsff.ko 2024-03-18T15:44:59.319 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-gaff.ko 2024-03-18T15:44:59.319 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-gt683r.ko 2024-03-18T15:44:59.319 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-gyration.ko 2024-03-18T15:44:59.319 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-holtek-kbd.ko 2024-03-18T15:44:59.319 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-holtek-mouse.ko 2024-03-18T15:44:59.319 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-holtekff.ko 2024-03-18T15:44:59.319 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-hyperv.ko 2024-03-18T15:44:59.319 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-icade.ko 2024-03-18T15:44:59.319 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-keytouch.ko 2024-03-18T15:44:59.320 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-kye.ko 2024-03-18T15:44:59.320 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-lcpower.ko 2024-03-18T15:44:59.320 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-led.ko 2024-03-18T15:44:59.320 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-lenovo.ko 2024-03-18T15:44:59.320 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-logitech-dj.ko 2024-03-18T15:44:59.320 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-03-18T15:44:59.320 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-multitouch.ko 2024-03-18T15:44:59.320 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-ortek.ko 2024-03-18T15:44:59.320 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-penmount.ko 2024-03-18T15:44:59.320 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-petalynx.ko 2024-03-18T15:44:59.320 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-picolcd.ko 2024-03-18T15:44:59.320 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-pl.ko 2024-03-18T15:44:59.320 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-plantronics.ko 2024-03-18T15:44:59.320 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-primax.ko 2024-03-18T15:44:59.321 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-rmi.ko 2024-03-18T15:44:59.321 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-roccat-arvo.ko 2024-03-18T15:44:59.321 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-roccat-common.ko 2024-03-18T15:44:59.321 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-roccat-isku.ko 2024-03-18T15:44:59.321 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-roccat-kone.ko 2024-03-18T15:44:59.321 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-03-18T15:44:59.321 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-roccat-konepure.ko 2024-03-18T15:44:59.321 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-03-18T15:44:59.321 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-roccat-lua.ko 2024-03-18T15:44:59.321 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-roccat-pyra.ko 2024-03-18T15:44:59.321 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-roccat-ryos.ko 2024-03-18T15:44:59.321 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-roccat-savu.ko 2024-03-18T15:44:59.321 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-roccat.ko 2024-03-18T15:44:59.321 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-saitek.ko 2024-03-18T15:44:59.322 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-samsung.ko 2024-03-18T15:44:59.322 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-sensor-hub.ko 2024-03-18T15:44:59.322 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-sjoy.ko 2024-03-18T15:44:59.322 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-sony.ko 2024-03-18T15:44:59.322 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-speedlink.ko 2024-03-18T15:44:59.322 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-steelseries.ko 2024-03-18T15:44:59.322 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-sunplus.ko 2024-03-18T15:44:59.322 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-thrustmaster.ko 2024-03-18T15:44:59.322 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-tivo.ko 2024-03-18T15:44:59.322 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-tmff.ko 2024-03-18T15:44:59.322 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-topseed.ko 2024-03-18T15:44:59.322 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-twinhan.ko 2024-03-18T15:44:59.322 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-uclogic.ko 2024-03-18T15:44:59.322 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-waltop.ko 2024-03-18T15:44:59.323 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-wiimote.ko 2024-03-18T15:44:59.323 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-xinmo.ko 2024-03-18T15:44:59.323 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-zpff.ko 2024-03-18T15:44:59.323 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/hid-zydacron.ko 2024-03-18T15:44:59.323 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/uhid.ko 2024-03-18T15:44:59.323 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hid/wacom.ko 2024-03-18T15:44:59.323 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hv 2024-03-18T15:44:59.323 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hv/hv_balloon.ko 2024-03-18T15:44:59.323 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hv/hv_utils.ko 2024-03-18T15:44:59.323 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hv/hv_vmbus.ko 2024-03-18T15:44:59.323 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon 2024-03-18T15:44:59.323 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/abituguru.ko 2024-03-18T15:44:59.323 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/abituguru3.ko 2024-03-18T15:44:59.323 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/acpi_power_meter.ko 2024-03-18T15:44:59.323 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ad7414.ko 2024-03-18T15:44:59.324 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ad7418.ko 2024-03-18T15:44:59.324 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/adc128d818.ko 2024-03-18T15:44:59.324 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/adm1025.ko 2024-03-18T15:44:59.324 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/adm1026.ko 2024-03-18T15:44:59.324 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/adm1029.ko 2024-03-18T15:44:59.324 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/adm1031.ko 2024-03-18T15:44:59.324 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/adm9240.ko 2024-03-18T15:44:59.324 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ads7828.ko 2024-03-18T15:44:59.324 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/adt7410.ko 2024-03-18T15:44:59.324 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/adt7411.ko 2024-03-18T15:44:59.324 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/adt7462.ko 2024-03-18T15:44:59.324 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/adt7470.ko 2024-03-18T15:44:59.324 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/adt7475.ko 2024-03-18T15:44:59.324 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/adt7x10.ko 2024-03-18T15:44:59.324 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/amc6821.ko 2024-03-18T15:44:59.325 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/applesmc.ko 2024-03-18T15:44:59.325 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/asb100.ko 2024-03-18T15:44:59.325 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/asc7621.ko 2024-03-18T15:44:59.325 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/asus_atk0110.ko 2024-03-18T15:44:59.325 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/atxp1.ko 2024-03-18T15:44:59.325 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/coretemp.ko 2024-03-18T15:44:59.325 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-03-18T15:44:59.325 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/dme1737.ko 2024-03-18T15:44:59.325 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ds1621.ko 2024-03-18T15:44:59.325 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ds620.ko 2024-03-18T15:44:59.325 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/emc1403.ko 2024-03-18T15:44:59.325 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/emc6w201.ko 2024-03-18T15:44:59.325 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/f71805f.ko 2024-03-18T15:44:59.325 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/f71882fg.ko 2024-03-18T15:44:59.326 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/f75375s.ko 2024-03-18T15:44:59.326 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/fam15h_power.ko 2024-03-18T15:44:59.326 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/fschmd.ko 2024-03-18T15:44:59.326 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/g760a.ko 2024-03-18T15:44:59.326 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/g762.ko 2024-03-18T15:44:59.326 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/gl518sm.ko 2024-03-18T15:44:59.326 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/gl520sm.ko 2024-03-18T15:44:59.326 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/hwmon-vid.ko 2024-03-18T15:44:59.326 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/i5500_temp.ko 2024-03-18T15:44:59.326 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/i5k_amb.ko 2024-03-18T15:44:59.326 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ibmaem.ko 2024-03-18T15:44:59.326 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ibmpex.ko 2024-03-18T15:44:59.326 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ina209.ko 2024-03-18T15:44:59.326 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ina2xx.ko 2024-03-18T15:44:59.327 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/it87.ko 2024-03-18T15:44:59.327 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/k10temp.ko 2024-03-18T15:44:59.327 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/k8temp.ko 2024-03-18T15:44:59.327 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lineage-pem.ko 2024-03-18T15:44:59.327 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm63.ko 2024-03-18T15:44:59.327 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm73.ko 2024-03-18T15:44:59.327 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm75.ko 2024-03-18T15:44:59.327 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm77.ko 2024-03-18T15:44:59.327 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm78.ko 2024-03-18T15:44:59.327 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm80.ko 2024-03-18T15:44:59.327 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm83.ko 2024-03-18T15:44:59.327 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm85.ko 2024-03-18T15:44:59.327 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm87.ko 2024-03-18T15:44:59.327 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm90.ko 2024-03-18T15:44:59.327 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm92.ko 2024-03-18T15:44:59.328 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm93.ko 2024-03-18T15:44:59.328 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm95234.ko 2024-03-18T15:44:59.328 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm95241.ko 2024-03-18T15:44:59.328 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/lm95245.ko 2024-03-18T15:44:59.328 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ltc2945.ko 2024-03-18T15:44:59.328 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ltc4151.ko 2024-03-18T15:44:59.328 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ltc4215.ko 2024-03-18T15:44:59.328 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ltc4222.ko 2024-03-18T15:44:59.328 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ltc4245.ko 2024-03-18T15:44:59.328 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ltc4260.ko 2024-03-18T15:44:59.328 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/ltc4261.ko 2024-03-18T15:44:59.328 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/max16065.ko 2024-03-18T15:44:59.328 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/max1619.ko 2024-03-18T15:44:59.328 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/max1668.ko 2024-03-18T15:44:59.329 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/max197.ko 2024-03-18T15:44:59.329 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/max6639.ko 2024-03-18T15:44:59.329 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/max6650.ko 2024-03-18T15:44:59.329 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/max6697.ko 2024-03-18T15:44:59.329 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/mcp3021.ko 2024-03-18T15:44:59.329 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/nct6683.ko 2024-03-18T15:44:59.329 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/nct6775-core.ko 2024-03-18T15:44:59.329 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/nct6775.ko 2024-03-18T15:44:59.329 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/nct7802.ko 2024-03-18T15:44:59.329 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pc87360.ko 2024-03-18T15:44:59.329 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pc87427.ko 2024-03-18T15:44:59.330 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pcf8591.ko 2024-03-18T15:44:59.330 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pmbus 2024-03-18T15:44:59.330 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-03-18T15:44:59.330 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-03-18T15:44:59.330 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-03-18T15:44:59.330 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pmbus/max16064.ko 2024-03-18T15:44:59.330 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pmbus/max34440.ko 2024-03-18T15:44:59.330 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pmbus/max8688.ko 2024-03-18T15:44:59.330 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-03-18T15:44:59.330 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-03-18T15:44:59.330 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-03-18T15:44:59.330 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-03-18T15:44:59.330 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-03-18T15:44:59.330 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-03-18T15:44:59.331 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/powr1220.ko 2024-03-18T15:44:59.331 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/sch5627.ko 2024-03-18T15:44:59.331 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/sch5636.ko 2024-03-18T15:44:59.331 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/sch56xx-common.ko 2024-03-18T15:44:59.331 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/sht15.ko 2024-03-18T15:44:59.331 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/sht21.ko 2024-03-18T15:44:59.331 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/shtc1.ko 2024-03-18T15:44:59.331 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/sis5595.ko 2024-03-18T15:44:59.331 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/smsc47b397.ko 2024-03-18T15:44:59.331 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/smsc47m1.ko 2024-03-18T15:44:59.331 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/smsc47m192.ko 2024-03-18T15:44:59.331 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/thmc50.ko 2024-03-18T15:44:59.331 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/tmp102.ko 2024-03-18T15:44:59.332 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/tmp103.ko 2024-03-18T15:44:59.332 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/tmp401.ko 2024-03-18T15:44:59.332 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/tmp421.ko 2024-03-18T15:44:59.332 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/via-cputemp.ko 2024-03-18T15:44:59.332 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/via686a.ko 2024-03-18T15:44:59.332 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/vt1211.ko 2024-03-18T15:44:59.332 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/vt8231.ko 2024-03-18T15:44:59.332 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/w83627ehf.ko 2024-03-18T15:44:59.332 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/w83627hf.ko 2024-03-18T15:44:59.332 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/w83781d.ko 2024-03-18T15:44:59.332 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/w83791d.ko 2024-03-18T15:44:59.332 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/w83792d.ko 2024-03-18T15:44:59.332 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/w83793.ko 2024-03-18T15:44:59.332 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/w83795.ko 2024-03-18T15:44:59.332 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/w83l785ts.ko 2024-03-18T15:44:59.333 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/hwmon/w83l786ng.ko 2024-03-18T15:44:59.333 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c 2024-03-18T15:44:59.333 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/algos 2024-03-18T15:44:59.333 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-03-18T15:44:59.333 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-03-18T15:44:59.333 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses 2024-03-18T15:44:59.333 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-03-18T15:44:59.333 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-03-18T15:44:59.333 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-03-18T15:44:59.333 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-03-18T15:44:59.333 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-03-18T15:44:59.333 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-03-18T15:44:59.333 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-03-18T15:44:59.333 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-03-18T15:44:59.334 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-i801.ko 2024-03-18T15:44:59.334 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-isch.ko 2024-03-18T15:44:59.334 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-03-18T15:44:59.334 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-03-18T15:44:59.334 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-03-18T15:44:59.334 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-parport.ko 2024-03-18T15:44:59.334 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-03-18T15:44:59.334 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-03-18T15:44:59.334 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-03-18T15:44:59.334 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-03-18T15:44:59.334 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-03-18T15:44:59.334 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-03-18T15:44:59.334 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-via.ko 2024-03-18T15:44:59.334 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-03-18T15:44:59.335 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-03-18T15:44:59.335 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/i2c-dev.ko 2024-03-18T15:44:59.335 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/i2c-mux.ko 2024-03-18T15:44:59.335 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-03-18T15:44:59.335 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/i2c-smbus.ko 2024-03-18T15:44:59.335 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/i2c/i2c-stub.ko 2024-03-18T15:44:59.335 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband 2024-03-18T15:44:59.335 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/core 2024-03-18T15:44:59.335 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/core/ib_cm.ko 2024-03-18T15:44:59.335 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/core/ib_core.ko 2024-03-18T15:44:59.335 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/core/ib_umad.ko 2024-03-18T15:44:59.335 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-03-18T15:44:59.335 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/core/iw_cm.ko 2024-03-18T15:44:59.335 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/core/rdma_cm.ko 2024-03-18T15:44:59.335 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-03-18T15:44:59.336 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/hw 2024-03-18T15:44:59.336 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/hw/cxgb4 2024-03-18T15:44:59.336 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-03-18T15:44:59.336 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/hw/mlx4 2024-03-18T15:44:59.336 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-03-18T15:44:59.336 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/hw/mlx5 2024-03-18T15:44:59.336 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-03-18T15:44:59.336 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/hw/mthca 2024-03-18T15:44:59.336 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-03-18T15:44:59.336 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/ulp 2024-03-18T15:44:59.336 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/ulp/ipoib 2024-03-18T15:44:59.336 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-03-18T15:44:59.336 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/ulp/iser 2024-03-18T15:44:59.336 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-03-18T15:44:59.337 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/ulp/isert 2024-03-18T15:44:59.337 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-03-18T15:44:59.337 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/ulp/srp 2024-03-18T15:44:59.337 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-03-18T15:44:59.337 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/ulp/srpt 2024-03-18T15:44:59.337 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-03-18T15:44:59.337 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input 2024-03-18T15:44:59.337 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/gameport 2024-03-18T15:44:59.337 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-03-18T15:44:59.337 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/gameport/fm801-gp.ko 2024-03-18T15:44:59.337 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/gameport/gameport.ko 2024-03-18T15:44:59.337 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/joydev.ko 2024-03-18T15:44:59.338 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/keyboard 2024-03-18T15:44:59.338 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/keyboard/gpio_keys.ko 2024-03-18T15:44:59.338 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/mouse 2024-03-18T15:44:59.338 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/mouse/appletouch.ko 2024-03-18T15:44:59.338 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/mouse/bcm5974.ko 2024-03-18T15:44:59.338 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/mouse/cyapatp.ko 2024-03-18T15:44:59.338 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/mouse/elan_i2c.ko 2024-03-18T15:44:59.338 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/mouse/sermouse.ko 2024-03-18T15:44:59.338 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-03-18T15:44:59.338 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/mouse/synaptics_usb.ko 2024-03-18T15:44:59.338 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/mouse/vsxxxaa.ko 2024-03-18T15:44:59.338 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/rmi4 2024-03-18T15:44:59.338 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/rmi4/rmi_core.ko 2024-03-18T15:44:59.339 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/serio 2024-03-18T15:44:59.339 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/serio/altera_ps2.ko 2024-03-18T15:44:59.339 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/serio/arc_ps2.ko 2024-03-18T15:44:59.339 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-03-18T15:44:59.339 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/serio/serio_raw.ko 2024-03-18T15:44:59.339 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/input/sparse-keymap.ko 2024-03-18T15:44:59.339 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds 2024-03-18T15:44:59.339 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/leds-blinkm.ko 2024-03-18T15:44:59.339 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/leds-lm3530.ko 2024-03-18T15:44:59.339 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/leds-lp3944.ko 2024-03-18T15:44:59.339 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/leds-ss4200.ko 2024-03-18T15:44:59.339 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/trigger 2024-03-18T15:44:59.339 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-03-18T15:44:59.339 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-03-18T15:44:59.339 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-03-18T15:44:59.340 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-03-18T15:44:59.340 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-03-18T15:44:59.340 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-03-18T15:44:59.340 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-03-18T15:44:59.340 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-03-18T15:44:59.340 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-03-18T15:44:59.340 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md 2024-03-18T15:44:59.340 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/bcache 2024-03-18T15:44:59.340 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/bcache/bcache.ko 2024-03-18T15:44:59.340 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-bio-prison.ko 2024-03-18T15:44:59.340 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-cache-smq.ko 2024-03-18T15:44:59.340 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-cache.ko 2024-03-18T15:44:59.340 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-crypt.ko 2024-03-18T15:44:59.340 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-delay.ko 2024-03-18T15:44:59.340 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-flakey.ko 2024-03-18T15:44:59.341 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-log-userspace.ko 2024-03-18T15:44:59.341 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-multipath.ko 2024-03-18T15:44:59.341 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-queue-length.ko 2024-03-18T15:44:59.341 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-raid.ko 2024-03-18T15:44:59.341 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-round-robin.ko 2024-03-18T15:44:59.341 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-service-time.ko 2024-03-18T15:44:59.341 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-switch.ko 2024-03-18T15:44:59.341 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-thin-pool.ko 2024-03-18T15:44:59.341 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/dm-verity.ko 2024-03-18T15:44:59.341 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/persistent-data 2024-03-18T15:44:59.341 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-03-18T15:44:59.341 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/raid0.ko 2024-03-18T15:44:59.341 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/raid1.ko 2024-03-18T15:44:59.341 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/raid10.ko 2024-03-18T15:44:59.342 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/md/raid456.ko 2024-03-18T15:44:59.342 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/message 2024-03-18T15:44:59.342 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/message/fusion 2024-03-18T15:44:59.342 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/message/fusion/mptbase.ko 2024-03-18T15:44:59.342 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/message/fusion/mptctl.ko 2024-03-18T15:44:59.342 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/message/fusion/mptfc.ko 2024-03-18T15:44:59.342 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/message/fusion/mptlan.ko 2024-03-18T15:44:59.342 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/message/fusion/mptsas.ko 2024-03-18T15:44:59.342 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/message/fusion/mptscsih.ko 2024-03-18T15:44:59.342 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/message/fusion/mptspi.ko 2024-03-18T15:44:59.342 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/mfd 2024-03-18T15:44:59.342 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/mfd/lpc_ich.ko 2024-03-18T15:44:59.342 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/mfd/lpc_sch.ko 2024-03-18T15:44:59.342 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/mfd/mfd-core.ko 2024-03-18T15:44:59.343 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/mfd/sm501.ko 2024-03-18T15:44:59.343 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/mfd/viperboard.ko 2024-03-18T15:44:59.343 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/mfd/vx855.ko 2024-03-18T15:44:59.343 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/mfd/wl1273-core.ko 2024-03-18T15:44:59.343 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc 2024-03-18T15:44:59.343 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/altera-stapl 2024-03-18T15:44:59.343 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-03-18T15:44:59.343 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/apds9802als.ko 2024-03-18T15:44:59.343 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/apds990x.ko 2024-03-18T15:44:59.343 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/bh1770glc.ko 2024-03-18T15:44:59.343 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/cb710 2024-03-18T15:44:59.343 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/cb710/cb710.ko 2024-03-18T15:44:59.344 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/eeprom 2024-03-18T15:44:59.344 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/eeprom/at24.ko 2024-03-18T15:44:59.344 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-03-18T15:44:59.344 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/eeprom/max6875.ko 2024-03-18T15:44:59.344 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/enclosure.ko 2024-03-18T15:44:59.344 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/hpilo.ko 2024-03-18T15:44:59.344 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/isl29003.ko 2024-03-18T15:44:59.344 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/isl29020.ko 2024-03-18T15:44:59.344 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/lis3lv02d 2024-03-18T15:44:59.344 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-03-18T15:44:59.344 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-03-18T15:44:59.344 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/mei 2024-03-18T15:44:59.344 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/mei/mei-me.ko 2024-03-18T15:44:59.344 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/mei/mei-txe.ko 2024-03-18T15:44:59.345 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/mei/mei.ko 2024-03-18T15:44:59.345 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/sgi-gru 2024-03-18T15:44:59.345 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/sgi-gru/gru.ko 2024-03-18T15:44:59.345 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/sgi-xp 2024-03-18T15:44:59.345 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/sgi-xp/xp.ko 2024-03-18T15:44:59.345 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/sgi-xp/xpc.ko 2024-03-18T15:44:59.345 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-03-18T15:44:59.345 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/tifm_7xx1.ko 2024-03-18T15:44:59.345 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/tifm_core.ko 2024-03-18T15:44:59.345 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/tsl2550.ko 2024-03-18T15:44:59.345 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/vmw_balloon.ko 2024-03-18T15:44:59.345 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/vmw_vmci 2024-03-18T15:44:59.345 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-03-18T15:44:59.345 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net 2024-03-18T15:44:59.345 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/bonding 2024-03-18T15:44:59.346 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/bonding/bonding.ko 2024-03-18T15:44:59.346 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/dsa 2024-03-18T15:44:59.346 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/dsa/b53 2024-03-18T15:44:59.346 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/dsa/b53/b53_common.ko 2024-03-18T15:44:59.346 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/dsa/bcm-sf2.ko 2024-03-18T15:44:59.346 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/dsa/mv88e6060.ko 2024-03-18T15:44:59.346 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/dsa/mv88e6xxx 2024-03-18T15:44:59.346 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-03-18T15:44:59.346 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/dummy.ko 2024-03-18T15:44:59.346 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/eql.ko 2024-03-18T15:44:59.346 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet 2024-03-18T15:44:59.346 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/3com 2024-03-18T15:44:59.346 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-03-18T15:44:59.347 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-03-18T15:44:59.347 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-03-18T15:44:59.347 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-03-18T15:44:59.347 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/8390 2024-03-18T15:44:59.347 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/8390/8390.ko 2024-03-18T15:44:59.347 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-03-18T15:44:59.347 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-03-18T15:44:59.347 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-03-18T15:44:59.347 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/adaptec 2024-03-18T15:44:59.347 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-03-18T15:44:59.347 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/agere 2024-03-18T15:44:59.347 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/agere/et131x.ko 2024-03-18T15:44:59.347 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/alteon 2024-03-18T15:44:59.347 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-03-18T15:44:59.348 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/altera 2024-03-18T15:44:59.348 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-03-18T15:44:59.348 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/amd 2024-03-18T15:44:59.348 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-03-18T15:44:59.348 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-03-18T15:44:59.348 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-03-18T15:44:59.348 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/atheros 2024-03-18T15:44:59.348 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/atheros/alx 2024-03-18T15:44:59.348 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-03-18T15:44:59.348 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/atheros/atl1c 2024-03-18T15:44:59.348 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-03-18T15:44:59.348 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/atheros/atl1e 2024-03-18T15:44:59.348 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-03-18T15:44:59.348 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/atheros/atlx 2024-03-18T15:44:59.349 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-03-18T15:44:59.349 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-03-18T15:44:59.349 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/broadcom 2024-03-18T15:44:59.349 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-03-18T15:44:59.349 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-03-18T15:44:59.349 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-03-18T15:44:59.349 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-03-18T15:44:59.349 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-03-18T15:44:59.349 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/broadcom/genet 2024-03-18T15:44:59.349 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-03-18T15:44:59.349 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-03-18T15:44:59.349 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/brocade 2024-03-18T15:44:59.349 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/brocade/bna 2024-03-18T15:44:59.349 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-03-18T15:44:59.350 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/chelsio 2024-03-18T15:44:59.350 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/chelsio/cxgb 2024-03-18T15:44:59.350 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-03-18T15:44:59.350 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-03-18T15:44:59.350 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-03-18T15:44:59.350 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-03-18T15:44:59.350 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-03-18T15:44:59.350 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-03-18T15:44:59.350 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-03-18T15:44:59.350 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-03-18T15:44:59.350 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-03-18T15:44:59.350 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/cisco 2024-03-18T15:44:59.350 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/cisco/enic 2024-03-18T15:44:59.350 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-03-18T15:44:59.351 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/dec 2024-03-18T15:44:59.351 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/dec/tulip 2024-03-18T15:44:59.351 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-03-18T15:44:59.351 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-03-18T15:44:59.351 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-03-18T15:44:59.351 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-03-18T15:44:59.351 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-03-18T15:44:59.351 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-03-18T15:44:59.351 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/dlink 2024-03-18T15:44:59.351 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-03-18T15:44:59.351 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-03-18T15:44:59.351 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/dnet.ko 2024-03-18T15:44:59.351 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/emulex 2024-03-18T15:44:59.351 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/emulex/benet 2024-03-18T15:44:59.352 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-03-18T15:44:59.352 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/ethoc.ko 2024-03-18T15:44:59.352 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/fealnx.ko 2024-03-18T15:44:59.352 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel 2024-03-18T15:44:59.352 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/e100.ko 2024-03-18T15:44:59.352 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/e1000 2024-03-18T15:44:59.352 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-03-18T15:44:59.352 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/e1000e 2024-03-18T15:44:59.352 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-03-18T15:44:59.352 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/fm10k 2024-03-18T15:44:59.352 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-03-18T15:44:59.352 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/i40e 2024-03-18T15:44:59.352 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-03-18T15:44:59.352 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/iavf 2024-03-18T15:44:59.353 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-03-18T15:44:59.353 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/igb 2024-03-18T15:44:59.353 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-03-18T15:44:59.353 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/igbvf 2024-03-18T15:44:59.353 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-03-18T15:44:59.353 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/ixgbe 2024-03-18T15:44:59.353 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-03-18T15:44:59.353 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/ixgbevf 2024-03-18T15:44:59.353 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-03-18T15:44:59.353 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/jme.ko 2024-03-18T15:44:59.353 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/marvell 2024-03-18T15:44:59.353 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-03-18T15:44:59.353 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/marvell/skge.ko 2024-03-18T15:44:59.353 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-03-18T15:44:59.354 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/mellanox 2024-03-18T15:44:59.354 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/mellanox/mlx4 2024-03-18T15:44:59.354 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-03-18T15:44:59.354 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-03-18T15:44:59.354 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/mellanox/mlx5 2024-03-18T15:44:59.354 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-03-18T15:44:59.354 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-03-18T15:44:59.354 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/micrel 2024-03-18T15:44:59.354 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-03-18T15:44:59.354 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/myricom 2024-03-18T15:44:59.354 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/myricom/myri10ge 2024-03-18T15:44:59.354 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-03-18T15:44:59.354 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/natsemi 2024-03-18T15:44:59.354 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-03-18T15:44:59.355 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-03-18T15:44:59.355 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/neterion 2024-03-18T15:44:59.355 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-03-18T15:44:59.355 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/nvidia 2024-03-18T15:44:59.355 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-03-18T15:44:59.355 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/packetengines 2024-03-18T15:44:59.355 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-03-18T15:44:59.355 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-03-18T15:44:59.355 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/qlogic 2024-03-18T15:44:59.355 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/qlogic/netxen 2024-03-18T15:44:59.355 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-03-18T15:44:59.355 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-03-18T15:44:59.355 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-03-18T15:44:59.355 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-03-18T15:44:59.355 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/rdc 2024-03-18T15:44:59.356 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-03-18T15:44:59.356 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/realtek 2024-03-18T15:44:59.356 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-03-18T15:44:59.356 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-03-18T15:44:59.356 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/realtek/atp.ko 2024-03-18T15:44:59.356 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-03-18T15:44:59.356 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/rocker 2024-03-18T15:44:59.356 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-03-18T15:44:59.356 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/sfc 2024-03-18T15:44:59.356 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-03-18T15:44:59.356 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/silan 2024-03-18T15:44:59.356 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-03-18T15:44:59.356 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/sis 2024-03-18T15:44:59.356 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/sis/sis190.ko 2024-03-18T15:44:59.357 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/sis/sis900.ko 2024-03-18T15:44:59.357 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/smsc 2024-03-18T15:44:59.357 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-03-18T15:44:59.357 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-03-18T15:44:59.357 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-03-18T15:44:59.357 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-03-18T15:44:59.357 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/stmicro 2024-03-18T15:44:59.357 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/stmicro/stmmac 2024-03-18T15:44:59.357 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-03-18T15:44:59.357 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-03-18T15:44:59.357 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/sun 2024-03-18T15:44:59.357 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/sun/cassini.ko 2024-03-18T15:44:59.357 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/sun/niu.ko 2024-03-18T15:44:59.357 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/sun/sungem.ko 2024-03-18T15:44:59.358 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-03-18T15:44:59.358 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/tehuti 2024-03-18T15:44:59.358 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-03-18T15:44:59.358 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/ti 2024-03-18T15:44:59.358 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/ti/tlan.ko 2024-03-18T15:44:59.358 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/via 2024-03-18T15:44:59.358 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-03-18T15:44:59.358 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-03-18T15:44:59.358 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/wiznet 2024-03-18T15:44:59.358 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-03-18T15:44:59.358 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-03-18T15:44:59.358 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/xircom 2024-03-18T15:44:59.358 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-03-18T15:44:59.358 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/geneve.ko 2024-03-18T15:44:59.359 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/hyperv 2024-03-18T15:44:59.359 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-03-18T15:44:59.359 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ieee802154 2024-03-18T15:44:59.359 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ieee802154/fakelb.ko 2024-03-18T15:44:59.359 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ifb.ko 2024-03-18T15:44:59.359 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ipvlan 2024-03-18T15:44:59.359 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ipvlan/ipvlan.ko 2024-03-18T15:44:59.359 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/macvlan.ko 2024-03-18T15:44:59.359 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/macvtap.ko 2024-03-18T15:44:59.359 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/mdio 2024-03-18T15:44:59.359 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/mdio.ko 2024-03-18T15:44:59.359 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-03-18T15:44:59.359 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-03-18T15:44:59.359 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/mdio/mdio-regmap.ko 2024-03-18T15:44:59.360 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/mii.ko 2024-03-18T15:44:59.360 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/net_failover.ko 2024-03-18T15:44:59.360 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/netconsole.ko 2024-03-18T15:44:59.360 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/nlmon.ko 2024-03-18T15:44:59.360 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/pcs 2024-03-18T15:44:59.360 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/pcs/pcs-lynx.ko 2024-03-18T15:44:59.360 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-03-18T15:44:59.360 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy 2024-03-18T15:44:59.360 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/amd.ko 2024-03-18T15:44:59.360 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-03-18T15:44:59.360 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-03-18T15:44:59.360 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/bcm7xxx.ko 2024-03-18T15:44:59.360 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/bcm87xx.ko 2024-03-18T15:44:59.360 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/broadcom.ko 2024-03-18T15:44:59.360 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/cicada.ko 2024-03-18T15:44:59.361 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/davicom.ko 2024-03-18T15:44:59.361 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/dp83640.ko 2024-03-18T15:44:59.361 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/et1011c.ko 2024-03-18T15:44:59.361 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/icplus.ko 2024-03-18T15:44:59.361 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/lxt.ko 2024-03-18T15:44:59.361 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/marvell.ko 2024-03-18T15:44:59.361 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/micrel.ko 2024-03-18T15:44:59.361 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/national.ko 2024-03-18T15:44:59.361 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/phylink.ko 2024-03-18T15:44:59.361 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/qsemi.ko 2024-03-18T15:44:59.361 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/realtek.ko 2024-03-18T15:44:59.361 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/smsc.ko 2024-03-18T15:44:59.361 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/ste10Xp.ko 2024-03-18T15:44:59.361 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/phy/vitesse.ko 2024-03-18T15:44:59.362 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ppp 2024-03-18T15:44:59.362 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ppp/bsd_comp.ko 2024-03-18T15:44:59.362 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ppp/ppp_async.ko 2024-03-18T15:44:59.362 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ppp/ppp_deflate.ko 2024-03-18T15:44:59.362 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ppp/ppp_generic.ko 2024-03-18T15:44:59.362 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ppp/ppp_mppe.ko 2024-03-18T15:44:59.362 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ppp/ppp_synctty.ko 2024-03-18T15:44:59.362 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ppp/pppoe.ko 2024-03-18T15:44:59.362 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ppp/pppox.ko 2024-03-18T15:44:59.362 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/ppp/pptp.ko 2024-03-18T15:44:59.362 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/slip 2024-03-18T15:44:59.362 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/slip/slhc.ko 2024-03-18T15:44:59.362 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/slip/slip.ko 2024-03-18T15:44:59.362 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/sungem_phy.ko 2024-03-18T15:44:59.362 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/tap.ko 2024-03-18T15:44:59.363 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/team 2024-03-18T15:44:59.363 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/team/team.ko 2024-03-18T15:44:59.363 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/team/team_mode_activebackup.ko 2024-03-18T15:44:59.363 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/team/team_mode_broadcast.ko 2024-03-18T15:44:59.363 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-03-18T15:44:59.363 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/team/team_mode_random.ko 2024-03-18T15:44:59.363 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-03-18T15:44:59.363 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/tun.ko 2024-03-18T15:44:59.363 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/veth.ko 2024-03-18T15:44:59.363 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/virtio_net.ko 2024-03-18T15:44:59.363 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/vmxnet3 2024-03-18T15:44:59.364 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-03-18T15:44:59.364 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/vxlan 2024-03-18T15:44:59.364 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/vxlan/vxlan.ko 2024-03-18T15:44:59.364 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/xen-netback 2024-03-18T15:44:59.364 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/xen-netback/xen-netback.ko 2024-03-18T15:44:59.364 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/net/xen-netfront.ko 2024-03-18T15:44:59.364 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ntb 2024-03-18T15:44:59.364 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ntb/ntb.ko 2024-03-18T15:44:59.364 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/nvme 2024-03-18T15:44:59.364 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/nvme/host 2024-03-18T15:44:59.364 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/nvme/host/nvme-core.ko 2024-03-18T15:44:59.364 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/nvme/host/nvme.ko 2024-03-18T15:44:59.364 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/parport 2024-03-18T15:44:59.364 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/parport/parport.ko 2024-03-18T15:44:59.365 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/parport/parport_cs.ko 2024-03-18T15:44:59.365 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/parport/parport_pc.ko 2024-03-18T15:44:59.365 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/parport/parport_serial.ko 2024-03-18T15:44:59.365 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pci 2024-03-18T15:44:59.365 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pci/hotplug 2024-03-18T15:44:59.365 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-03-18T15:44:59.365 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pci/pcie 2024-03-18T15:44:59.365 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pci/pcie/aer_inject.ko 2024-03-18T15:44:59.365 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pci/xen-pcifront.ko 2024-03-18T15:44:59.365 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pcmcia 2024-03-18T15:44:59.365 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pcmcia/i82092.ko 2024-03-18T15:44:59.365 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pcmcia/pd6729.ko 2024-03-18T15:44:59.365 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pcmcia/yenta_socket.ko 2024-03-18T15:44:59.365 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pinctrl 2024-03-18T15:44:59.366 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pinctrl/intel 2024-03-18T15:44:59.366 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-03-18T15:44:59.366 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform 2024-03-18T15:44:59.366 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86 2024-03-18T15:44:59.366 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/acer-wmi.ko 2024-03-18T15:44:59.366 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/acerhdf.ko 2024-03-18T15:44:59.366 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/apple-gmux.ko 2024-03-18T15:44:59.366 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/asus-laptop.ko 2024-03-18T15:44:59.366 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-03-18T15:44:59.366 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/asus-wmi.ko 2024-03-18T15:44:59.366 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/classmate-laptop.ko 2024-03-18T15:44:59.366 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-03-18T15:44:59.366 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-03-18T15:44:59.366 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-03-18T15:44:59.367 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-03-18T15:44:59.367 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/hdaps.ko 2024-03-18T15:44:59.367 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/intel 2024-03-18T15:44:59.367 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-03-18T15:44:59.367 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/intel_ips.ko 2024-03-18T15:44:59.367 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/msi-wmi.ko 2024-03-18T15:44:59.367 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/mxm-wmi.ko 2024-03-18T15:44:59.367 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-03-18T15:44:59.367 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/samsung-laptop.ko 2024-03-18T15:44:59.367 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/samsung-q10.ko 2024-03-18T15:44:59.367 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/topstar-laptop.ko 2024-03-18T15:44:59.367 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-03-18T15:44:59.367 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/toshiba_haps.ko 2024-03-18T15:44:59.367 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/wmi-bmof.ko 2024-03-18T15:44:59.368 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/platform/x86/wmi.ko 2024-03-18T15:44:59.368 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/powercap 2024-03-18T15:44:59.368 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/powercap/intel_rapl_common.ko 2024-03-18T15:44:59.368 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/powercap/intel_rapl_msr.ko 2024-03-18T15:44:59.368 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pps 2024-03-18T15:44:59.368 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pps/clients 2024-03-18T15:44:59.368 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pps/clients/pps-gpio.ko 2024-03-18T15:44:59.368 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pps/clients/pps-ldisc.ko 2024-03-18T15:44:59.368 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pps/clients/pps_parport.ko 2024-03-18T15:44:59.368 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/pps/pps_core.ko 2024-03-18T15:44:59.368 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ptp 2024-03-18T15:44:59.368 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ptp/ptp.ko 2024-03-18T15:44:59.368 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ptp/ptp_kvm.ko 2024-03-18T15:44:59.368 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc 2024-03-18T15:44:59.369 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-bq32k.ko 2024-03-18T15:44:59.369 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-ds1286.ko 2024-03-18T15:44:59.369 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-ds1307.ko 2024-03-18T15:44:59.369 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-ds1374.ko 2024-03-18T15:44:59.369 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-ds1511.ko 2024-03-18T15:44:59.369 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-ds1553.ko 2024-03-18T15:44:59.369 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-ds1672.ko 2024-03-18T15:44:59.369 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-ds1742.ko 2024-03-18T15:44:59.369 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-ds2404.ko 2024-03-18T15:44:59.369 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-ds3232.ko 2024-03-18T15:44:59.369 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-em3027.ko 2024-03-18T15:44:59.369 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-fm3130.ko 2024-03-18T15:44:59.369 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-isl12022.ko 2024-03-18T15:44:59.369 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-isl1208.ko 2024-03-18T15:44:59.369 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-m41t80.ko 2024-03-18T15:44:59.370 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-m48t35.ko 2024-03-18T15:44:59.370 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-m48t59.ko 2024-03-18T15:44:59.370 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-max6900.ko 2024-03-18T15:44:59.370 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-msm6242.ko 2024-03-18T15:44:59.370 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-pcf2127.ko 2024-03-18T15:44:59.370 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-pcf85063.ko 2024-03-18T15:44:59.370 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-pcf8523.ko 2024-03-18T15:44:59.370 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-pcf8563.ko 2024-03-18T15:44:59.370 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-pcf8583.ko 2024-03-18T15:44:59.370 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-rp5c01.ko 2024-03-18T15:44:59.370 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-rs5c372.ko 2024-03-18T15:44:59.370 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-03-18T15:44:59.370 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-rx8025.ko 2024-03-18T15:44:59.370 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-rx8581.ko 2024-03-18T15:44:59.371 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-03-18T15:44:59.371 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/rtc/rtc-x1205.ko 2024-03-18T15:44:59.371 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi 2024-03-18T15:44:59.371 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/3w-9xxx.ko 2024-03-18T15:44:59.371 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/3w-sas.ko 2024-03-18T15:44:59.371 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/3w-xxxx.ko 2024-03-18T15:44:59.371 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/BusLogic.ko 2024-03-18T15:44:59.371 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/a100u2w.ko 2024-03-18T15:44:59.371 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/aacraid 2024-03-18T15:44:59.371 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/aacraid/aacraid.ko 2024-03-18T15:44:59.371 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/advansys.ko 2024-03-18T15:44:59.371 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/aic7xxx 2024-03-18T15:44:59.371 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-03-18T15:44:59.371 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-03-18T15:44:59.372 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/am53c974.ko 2024-03-18T15:44:59.372 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/arcmsr 2024-03-18T15:44:59.372 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-03-18T15:44:59.372 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/atp870u.ko 2024-03-18T15:44:59.372 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/be2iscsi 2024-03-18T15:44:59.372 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-03-18T15:44:59.372 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/bfa 2024-03-18T15:44:59.372 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/bfa/bfa.ko 2024-03-18T15:44:59.372 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/bnx2fc 2024-03-18T15:44:59.372 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-03-18T15:44:59.372 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/bnx2i 2024-03-18T15:44:59.372 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-03-18T15:44:59.372 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/ch.ko 2024-03-18T15:44:59.372 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/csiostor 2024-03-18T15:44:59.372 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/csiostor/csiostor.ko 2024-03-18T15:44:59.373 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/cxgbi 2024-03-18T15:44:59.373 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/cxgbi/cxgb3i 2024-03-18T15:44:59.373 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-03-18T15:44:59.373 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/cxgbi/cxgb4i 2024-03-18T15:44:59.373 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-03-18T15:44:59.373 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-03-18T15:44:59.373 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/dc395x.ko 2024-03-18T15:44:59.373 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/device_handler 2024-03-18T15:44:59.373 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-03-18T15:44:59.373 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-03-18T15:44:59.373 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-03-18T15:44:59.373 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-03-18T15:44:59.373 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/dmx3191d.ko 2024-03-18T15:44:59.373 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/esas2r 2024-03-18T15:44:59.374 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/esas2r/esas2r.ko 2024-03-18T15:44:59.374 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/esp_scsi.ko 2024-03-18T15:44:59.374 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/fcoe 2024-03-18T15:44:59.374 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/fcoe/fcoe.ko 2024-03-18T15:44:59.374 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-03-18T15:44:59.374 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/fnic 2024-03-18T15:44:59.374 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/fnic/fnic.ko 2024-03-18T15:44:59.374 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/hpsa.ko 2024-03-18T15:44:59.374 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/hptiop.ko 2024-03-18T15:44:59.374 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/hv_storvsc.ko 2024-03-18T15:44:59.374 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/initio.ko 2024-03-18T15:44:59.374 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/ipr.ko 2024-03-18T15:44:59.374 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/ips.ko 2024-03-18T15:44:59.374 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/isci 2024-03-18T15:44:59.375 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/isci/isci.ko 2024-03-18T15:44:59.375 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-03-18T15:44:59.375 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/iscsi_tcp.ko 2024-03-18T15:44:59.375 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/libfc 2024-03-18T15:44:59.375 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/libfc/libfc.ko 2024-03-18T15:44:59.375 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/libiscsi.ko 2024-03-18T15:44:59.375 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/libiscsi_tcp.ko 2024-03-18T15:44:59.375 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/libsas 2024-03-18T15:44:59.375 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/libsas/libsas.ko 2024-03-18T15:44:59.375 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/lpfc 2024-03-18T15:44:59.375 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/lpfc/lpfc.ko 2024-03-18T15:44:59.375 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/megaraid 2024-03-18T15:44:59.375 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/megaraid.ko 2024-03-18T15:44:59.375 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-03-18T15:44:59.375 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-03-18T15:44:59.376 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-03-18T15:44:59.376 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/mpt3sas 2024-03-18T15:44:59.376 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-03-18T15:44:59.376 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/mvsas 2024-03-18T15:44:59.376 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/mvsas/mvsas.ko 2024-03-18T15:44:59.376 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/mvumi.ko 2024-03-18T15:44:59.376 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/pm8001 2024-03-18T15:44:59.376 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-03-18T15:44:59.376 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/pmcraid.ko 2024-03-18T15:44:59.376 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/qla1280.ko 2024-03-18T15:44:59.376 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/qla2xxx 2024-03-18T15:44:59.376 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-03-18T15:44:59.376 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-03-18T15:44:59.376 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/qla4xxx 2024-03-18T15:44:59.377 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-03-18T15:44:59.377 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/raid_class.ko 2024-03-18T15:44:59.377 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/scsi_debug.ko 2024-03-18T15:44:59.377 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/scsi_transport_fc.ko 2024-03-18T15:44:59.377 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-03-18T15:44:59.377 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/scsi_transport_sas.ko 2024-03-18T15:44:59.377 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/scsi_transport_spi.ko 2024-03-18T15:44:59.377 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/scsi_transport_srp.ko 2024-03-18T15:44:59.377 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/ses.ko 2024-03-18T15:44:59.377 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/st.ko 2024-03-18T15:44:59.377 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/stex.ko 2024-03-18T15:44:59.377 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/sym53c8xx_2 2024-03-18T15:44:59.377 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-03-18T15:44:59.377 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/virtio_scsi.ko 2024-03-18T15:44:59.377 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/vmw_pvscsi.ko 2024-03-18T15:44:59.378 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/scsi/wd719x.ko 2024-03-18T15:44:59.378 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ssb 2024-03-18T15:44:59.378 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ssb/ssb.ko 2024-03-18T15:44:59.378 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/target 2024-03-18T15:44:59.378 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/target/iscsi 2024-03-18T15:44:59.378 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-03-18T15:44:59.378 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/target/loopback 2024-03-18T15:44:59.378 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/target/loopback/tcm_loop.ko 2024-03-18T15:44:59.378 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/target/target_core_file.ko 2024-03-18T15:44:59.378 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/target/target_core_iblock.ko 2024-03-18T15:44:59.378 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/target/target_core_mod.ko 2024-03-18T15:44:59.378 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/target/target_core_pscsi.ko 2024-03-18T15:44:59.378 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/target/target_core_user.ko 2024-03-18T15:44:59.378 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/target/tcm_fc 2024-03-18T15:44:59.379 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-03-18T15:44:59.379 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal 2024-03-18T15:44:59.379 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel 2024-03-18T15:44:59.379 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal 2024-03-18T15:44:59.379 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-03-18T15:44:59.379 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-03-18T15:44:59.379 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-03-18T15:44:59.379 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-03-18T15:44:59.379 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-03-18T15:44:59.379 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-03-18T15:44:59.379 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-03-18T15:44:59.379 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-03-18T15:44:59.379 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-03-18T15:44:59.379 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-03-18T15:44:59.379 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-03-18T15:44:59.380 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-03-18T15:44:59.380 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-03-18T15:44:59.380 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-03-18T15:44:59.380 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-03-18T15:44:59.380 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-03-18T15:44:59.380 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-03-18T15:44:59.380 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-03-18T15:44:59.380 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/tty 2024-03-18T15:44:59.380 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/tty/ipwireless 2024-03-18T15:44:59.380 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-03-18T15:44:59.380 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/tty/n_gsm.ko 2024-03-18T15:44:59.380 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/tty/n_hdlc.ko 2024-03-18T15:44:59.380 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/tty/nozomi.ko 2024-03-18T15:44:59.380 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/tty/serial 2024-03-18T15:44:59.380 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/tty/serial/8250 2024-03-18T15:44:59.381 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-03-18T15:44:59.381 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/tty/serial/arc_uart.ko 2024-03-18T15:44:59.381 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/tty/serial/jsm 2024-03-18T15:44:59.381 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/tty/serial/jsm/jsm.ko 2024-03-18T15:44:59.381 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ufs 2024-03-18T15:44:59.381 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ufs/core 2024-03-18T15:44:59.381 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ufs/core/ufshcd-core.ko 2024-03-18T15:44:59.381 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ufs/host 2024-03-18T15:44:59.381 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-03-18T15:44:59.381 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/uio 2024-03-18T15:44:59.381 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/uio/uio.ko 2024-03-18T15:44:59.381 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/uio/uio_aec.ko 2024-03-18T15:44:59.381 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/uio/uio_cif.ko 2024-03-18T15:44:59.381 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/uio/uio_pci_generic.ko 2024-03-18T15:44:59.381 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/uio/uio_sercos3.ko 2024-03-18T15:44:59.382 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb 2024-03-18T15:44:59.382 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/class 2024-03-18T15:44:59.382 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/class/cdc-acm.ko 2024-03-18T15:44:59.382 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/class/cdc-wdm.ko 2024-03-18T15:44:59.382 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/class/usblp.ko 2024-03-18T15:44:59.382 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/class/usbtmc.ko 2024-03-18T15:44:59.382 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/host 2024-03-18T15:44:59.382 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/host/sl811-hcd.ko 2024-03-18T15:44:59.382 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/image 2024-03-18T15:44:59.382 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/image/mdc800.ko 2024-03-18T15:44:59.382 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/image/microtek.ko 2024-03-18T15:44:59.382 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc 2024-03-18T15:44:59.382 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/adutux.ko 2024-03-18T15:44:59.382 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/appledisplay.ko 2024-03-18T15:44:59.383 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/emi26.ko 2024-03-18T15:44:59.383 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/emi62.ko 2024-03-18T15:44:59.383 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/ezusb.ko 2024-03-18T15:44:59.383 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/idmouse.ko 2024-03-18T15:44:59.383 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/iowarrior.ko 2024-03-18T15:44:59.383 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/isight_firmware.ko 2024-03-18T15:44:59.383 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/ldusb.ko 2024-03-18T15:44:59.383 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/legousbtower.ko 2024-03-18T15:44:59.383 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/sisusbvga 2024-03-18T15:44:59.383 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-03-18T15:44:59.383 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/trancevibrator.ko 2024-03-18T15:44:59.383 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/usb3503.ko 2024-03-18T15:44:59.383 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/usblcd.ko 2024-03-18T15:44:59.383 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/usbsevseg.ko 2024-03-18T15:44:59.384 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/uss720.ko 2024-03-18T15:44:59.384 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/misc/yurex.ko 2024-03-18T15:44:59.384 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/phy 2024-03-18T15:44:59.384 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/phy/phy-generic.ko 2024-03-18T15:44:59.384 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial 2024-03-18T15:44:59.384 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/aircable.ko 2024-03-18T15:44:59.384 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/ark3116.ko 2024-03-18T15:44:59.384 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/belkin_sa.ko 2024-03-18T15:44:59.384 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/ch341.ko 2024-03-18T15:44:59.384 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/cp210x.ko 2024-03-18T15:44:59.384 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/cyberjack.ko 2024-03-18T15:44:59.384 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/cypress_m8.ko 2024-03-18T15:44:59.384 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/digi_acceleport.ko 2024-03-18T15:44:59.384 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/empeg.ko 2024-03-18T15:44:59.384 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/ftdi_sio.ko 2024-03-18T15:44:59.385 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/garmin_gps.ko 2024-03-18T15:44:59.385 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/io_edgeport.ko 2024-03-18T15:44:59.385 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/io_ti.ko 2024-03-18T15:44:59.385 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/ipaq.ko 2024-03-18T15:44:59.385 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/ipw.ko 2024-03-18T15:44:59.385 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/ir-usb.ko 2024-03-18T15:44:59.385 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-03-18T15:44:59.385 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/keyspan.ko 2024-03-18T15:44:59.385 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/keyspan_pda.ko 2024-03-18T15:44:59.385 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/kl5kusb105.ko 2024-03-18T15:44:59.385 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/kobil_sct.ko 2024-03-18T15:44:59.385 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/mct_u232.ko 2024-03-18T15:44:59.385 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/mos7720.ko 2024-03-18T15:44:59.385 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/mos7840.ko 2024-03-18T15:44:59.385 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/navman.ko 2024-03-18T15:44:59.386 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/omninet.ko 2024-03-18T15:44:59.386 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/opticon.ko 2024-03-18T15:44:59.386 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/option.ko 2024-03-18T15:44:59.386 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/oti6858.ko 2024-03-18T15:44:59.386 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/pl2303.ko 2024-03-18T15:44:59.386 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/qcaux.ko 2024-03-18T15:44:59.386 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/qcserial.ko 2024-03-18T15:44:59.386 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/quatech2.ko 2024-03-18T15:44:59.386 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/safe_serial.ko 2024-03-18T15:44:59.386 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/sierra.ko 2024-03-18T15:44:59.386 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/spcp8x5.ko 2024-03-18T15:44:59.386 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/ssu100.ko 2024-03-18T15:44:59.386 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/symbolserial.ko 2024-03-18T15:44:59.386 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-03-18T15:44:59.386 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-03-18T15:44:59.387 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/usb_debug.ko 2024-03-18T15:44:59.387 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/usb_wwan.ko 2024-03-18T15:44:59.387 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/visor.ko 2024-03-18T15:44:59.387 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/whiteheat.ko 2024-03-18T15:44:59.387 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/serial/xsens_mt.ko 2024-03-18T15:44:59.387 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage 2024-03-18T15:44:59.387 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/uas.ko 2024-03-18T15:44:59.387 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/ums-alauda.ko 2024-03-18T15:44:59.387 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/ums-cypress.ko 2024-03-18T15:44:59.387 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/ums-datafab.ko 2024-03-18T15:44:59.387 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-03-18T15:44:59.387 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/ums-freecom.ko 2024-03-18T15:44:59.387 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/ums-isd200.ko 2024-03-18T15:44:59.387 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-03-18T15:44:59.387 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/ums-karma.ko 2024-03-18T15:44:59.388 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/ums-onetouch.ko 2024-03-18T15:44:59.388 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/ums-realtek.ko 2024-03-18T15:44:59.388 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/ums-sddr09.ko 2024-03-18T15:44:59.388 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/ums-sddr55.ko 2024-03-18T15:44:59.388 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/ums-usbat.ko 2024-03-18T15:44:59.388 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/storage/usb-storage.ko 2024-03-18T15:44:59.388 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/usbip 2024-03-18T15:44:59.388 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/usbip/usbip-core.ko 2024-03-18T15:44:59.388 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/usbip/usbip-host.ko 2024-03-18T15:44:59.388 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-03-18T15:44:59.388 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/vfio 2024-03-18T15:44:59.388 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/vfio/pci 2024-03-18T15:44:59.388 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-03-18T15:44:59.388 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/vfio/pci/vfio-pci.ko 2024-03-18T15:44:59.388 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/vfio/vfio.ko 2024-03-18T15:44:59.389 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-03-18T15:44:59.389 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/vhost 2024-03-18T15:44:59.389 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/vhost/vhost.ko 2024-03-18T15:44:59.389 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/vhost/vhost_iotlb.ko 2024-03-18T15:44:59.389 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/vhost/vhost_net.ko 2024-03-18T15:44:59.389 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/vhost/vhost_scsi.ko 2024-03-18T15:44:59.389 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/video 2024-03-18T15:44:59.389 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/video/backlight 2024-03-18T15:44:59.389 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/video/backlight/apple_bl.ko 2024-03-18T15:44:59.389 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/video/backlight/lcd.ko 2024-03-18T15:44:59.389 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/video/backlight/platform_lcd.ko 2024-03-18T15:44:59.389 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/virtio 2024-03-18T15:44:59.389 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/virtio/virtio.ko 2024-03-18T15:44:59.389 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/virtio/virtio_balloon.ko 2024-03-18T15:44:59.389 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/virtio/virtio_mmio.ko 2024-03-18T15:44:59.390 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/virtio/virtio_pci.ko 2024-03-18T15:44:59.390 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-03-18T15:44:59.390 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-03-18T15:44:59.390 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/virtio/virtio_ring.ko 2024-03-18T15:44:59.390 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1 2024-03-18T15:44:59.390 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/masters 2024-03-18T15:44:59.390 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/masters/ds2482.ko 2024-03-18T15:44:59.390 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/masters/ds2490.ko 2024-03-18T15:44:59.390 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/slaves 2024-03-18T15:44:59.390 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-03-18T15:44:59.390 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-03-18T15:44:59.390 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-03-18T15:44:59.390 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-03-18T15:44:59.390 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-03-18T15:44:59.390 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-03-18T15:44:59.391 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-03-18T15:44:59.391 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-03-18T15:44:59.391 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-03-18T15:44:59.391 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/slaves/w1_smem.ko 2024-03-18T15:44:59.391 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/slaves/w1_therm.ko 2024-03-18T15:44:59.391 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/w1/wire.ko 2024-03-18T15:44:59.391 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog 2024-03-18T15:44:59.391 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/alim1535_wdt.ko 2024-03-18T15:44:59.391 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/alim7101_wdt.ko 2024-03-18T15:44:59.391 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/f71808e_wdt.ko 2024-03-18T15:44:59.391 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/hpwdt.ko 2024-03-18T15:44:59.391 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/i6300esb.ko 2024-03-18T15:44:59.391 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-03-18T15:44:59.391 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/iTCO_wdt.ko 2024-03-18T15:44:59.391 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/ib700wdt.ko 2024-03-18T15:44:59.392 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/ibmasr.ko 2024-03-18T15:44:59.392 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-03-18T15:44:59.392 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/it8712f_wdt.ko 2024-03-18T15:44:59.392 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/it87_wdt.ko 2024-03-18T15:44:59.392 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/machzwd.ko 2024-03-18T15:44:59.392 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/nv_tco.ko 2024-03-18T15:44:59.392 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/pcwd_pci.ko 2024-03-18T15:44:59.392 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/pcwd_usb.ko 2024-03-18T15:44:59.392 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-03-18T15:44:59.392 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/sch311x_wdt.ko 2024-03-18T15:44:59.392 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/softdog.ko 2024-03-18T15:44:59.392 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/sp5100_tco.ko 2024-03-18T15:44:59.392 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/via_wdt.ko 2024-03-18T15:44:59.392 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-03-18T15:44:59.392 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/w83877f_wdt.ko 2024-03-18T15:44:59.393 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/w83977f_wdt.ko 2024-03-18T15:44:59.393 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/wdt_pci.ko 2024-03-18T15:44:59.393 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/watchdog/xen_wdt.ko 2024-03-18T15:44:59.393 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/xen 2024-03-18T15:44:59.393 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/xen/xen-acpi-processor.ko 2024-03-18T15:44:59.393 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/xen/xen-evtchn.ko 2024-03-18T15:44:59.393 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/xen/xen-gntalloc.ko 2024-03-18T15:44:59.393 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/xen/xen-gntdev.ko 2024-03-18T15:44:59.393 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/xen/xen-pciback 2024-03-18T15:44:59.393 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-03-18T15:44:59.393 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/xen/xen-privcmd.ko 2024-03-18T15:44:59.393 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/xen/xenfs 2024-03-18T15:44:59.393 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/drivers/xen/xenfs/xenfs.ko 2024-03-18T15:44:59.393 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs 2024-03-18T15:44:59.393 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/9p 2024-03-18T15:44:59.394 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/9p/9p.ko 2024-03-18T15:44:59.394 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/affs 2024-03-18T15:44:59.394 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/affs/affs.ko 2024-03-18T15:44:59.394 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/befs 2024-03-18T15:44:59.394 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/befs/befs.ko 2024-03-18T15:44:59.394 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/binfmt_misc.ko 2024-03-18T15:44:59.394 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/btrfs 2024-03-18T15:44:59.394 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/btrfs/btrfs.ko 2024-03-18T15:44:59.394 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/cachefiles 2024-03-18T15:44:59.394 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/cachefiles/cachefiles.ko 2024-03-18T15:44:59.394 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ceph 2024-03-18T15:44:59.394 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ceph/ceph.ko 2024-03-18T15:44:59.394 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/coda 2024-03-18T15:44:59.394 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/coda/coda.ko 2024-03-18T15:44:59.394 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/cramfs 2024-03-18T15:44:59.395 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/cramfs/cramfs.ko 2024-03-18T15:44:59.395 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/dlm 2024-03-18T15:44:59.395 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/dlm/dlm.ko 2024-03-18T15:44:59.395 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ecryptfs 2024-03-18T15:44:59.395 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ecryptfs/ecryptfs.ko 2024-03-18T15:44:59.395 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/f2fs 2024-03-18T15:44:59.395 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/f2fs/f2fs.ko 2024-03-18T15:44:59.395 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/fat 2024-03-18T15:44:59.395 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/fat/fat.ko 2024-03-18T15:44:59.395 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/fat/msdos.ko 2024-03-18T15:44:59.395 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/fat/vfat.ko 2024-03-18T15:44:59.395 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/fuse 2024-03-18T15:44:59.395 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/fuse/cuse.ko 2024-03-18T15:44:59.395 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/fuse/fuse.ko 2024-03-18T15:44:59.395 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/gfs2 2024-03-18T15:44:59.396 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/gfs2/gfs2.ko 2024-03-18T15:44:59.396 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/hfs 2024-03-18T15:44:59.396 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/hfs/hfs.ko 2024-03-18T15:44:59.396 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/hfsplus 2024-03-18T15:44:59.396 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/hfsplus/hfsplus.ko 2024-03-18T15:44:59.396 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/isofs 2024-03-18T15:44:59.396 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/isofs/isofs.ko 2024-03-18T15:44:59.396 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/jfs 2024-03-18T15:44:59.396 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/jfs/jfs.ko 2024-03-18T15:44:59.396 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/lockd 2024-03-18T15:44:59.396 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/lockd/lockd.ko 2024-03-18T15:44:59.396 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/minix 2024-03-18T15:44:59.396 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/minix/minix.ko 2024-03-18T15:44:59.396 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/netfs 2024-03-18T15:44:59.397 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/netfs/netfs.ko 2024-03-18T15:44:59.397 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfs 2024-03-18T15:44:59.397 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfs/blocklayout 2024-03-18T15:44:59.397 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-03-18T15:44:59.397 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfs/filelayout 2024-03-18T15:44:59.397 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-03-18T15:44:59.397 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfs/flexfilelayout 2024-03-18T15:44:59.397 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-03-18T15:44:59.397 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfs/nfs.ko 2024-03-18T15:44:59.397 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfs/nfsv3.ko 2024-03-18T15:44:59.397 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfs/nfsv4.ko 2024-03-18T15:44:59.397 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfs_common 2024-03-18T15:44:59.397 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfs_common/grace.ko 2024-03-18T15:44:59.397 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfs_common/nfs_acl.ko 2024-03-18T15:44:59.397 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfsd 2024-03-18T15:44:59.398 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nfsd/nfsd.ko 2024-03-18T15:44:59.398 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nilfs2 2024-03-18T15:44:59.398 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nilfs2/nilfs2.ko 2024-03-18T15:44:59.398 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls 2024-03-18T15:44:59.398 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/mac-celtic.ko 2024-03-18T15:44:59.398 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/mac-centeuro.ko 2024-03-18T15:44:59.398 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/mac-croatian.ko 2024-03-18T15:44:59.398 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/mac-cyrillic.ko 2024-03-18T15:44:59.398 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/mac-gaelic.ko 2024-03-18T15:44:59.398 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/mac-greek.ko 2024-03-18T15:44:59.398 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/mac-iceland.ko 2024-03-18T15:44:59.398 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/mac-inuit.ko 2024-03-18T15:44:59.399 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/mac-roman.ko 2024-03-18T15:44:59.399 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/mac-romanian.ko 2024-03-18T15:44:59.399 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/mac-turkish.ko 2024-03-18T15:44:59.399 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp1250.ko 2024-03-18T15:44:59.399 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp1251.ko 2024-03-18T15:44:59.399 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp1255.ko 2024-03-18T15:44:59.399 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp737.ko 2024-03-18T15:44:59.399 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp775.ko 2024-03-18T15:44:59.399 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp850.ko 2024-03-18T15:44:59.399 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp852.ko 2024-03-18T15:44:59.399 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp855.ko 2024-03-18T15:44:59.399 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp857.ko 2024-03-18T15:44:59.399 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp860.ko 2024-03-18T15:44:59.400 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp861.ko 2024-03-18T15:44:59.400 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp862.ko 2024-03-18T15:44:59.400 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp863.ko 2024-03-18T15:44:59.400 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp864.ko 2024-03-18T15:44:59.400 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp865.ko 2024-03-18T15:44:59.400 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp866.ko 2024-03-18T15:44:59.400 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp869.ko 2024-03-18T15:44:59.400 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp874.ko 2024-03-18T15:44:59.400 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp932.ko 2024-03-18T15:44:59.400 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp936.ko 2024-03-18T15:44:59.400 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp949.ko 2024-03-18T15:44:59.400 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_cp950.ko 2024-03-18T15:44:59.400 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_euc-jp.ko 2024-03-18T15:44:59.400 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_iso8859-1.ko 2024-03-18T15:44:59.401 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_iso8859-13.ko 2024-03-18T15:44:59.401 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_iso8859-14.ko 2024-03-18T15:44:59.401 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_iso8859-15.ko 2024-03-18T15:44:59.401 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_iso8859-2.ko 2024-03-18T15:44:59.401 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_iso8859-3.ko 2024-03-18T15:44:59.401 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_iso8859-4.ko 2024-03-18T15:44:59.401 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_iso8859-5.ko 2024-03-18T15:44:59.401 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_iso8859-6.ko 2024-03-18T15:44:59.401 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_iso8859-7.ko 2024-03-18T15:44:59.401 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_iso8859-9.ko 2024-03-18T15:44:59.401 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_koi8-r.ko 2024-03-18T15:44:59.401 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_koi8-ru.ko 2024-03-18T15:44:59.401 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_koi8-u.ko 2024-03-18T15:44:59.401 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_ucs2_utils.ko 2024-03-18T15:44:59.401 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/nls/nls_utf8.ko 2024-03-18T15:44:59.402 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ocfs2 2024-03-18T15:44:59.402 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ocfs2/cluster 2024-03-18T15:44:59.402 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-03-18T15:44:59.402 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ocfs2/dlm 2024-03-18T15:44:59.402 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-03-18T15:44:59.402 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ocfs2/dlmfs 2024-03-18T15:44:59.402 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-03-18T15:44:59.402 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ocfs2/ocfs2.ko 2024-03-18T15:44:59.402 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-03-18T15:44:59.402 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-03-18T15:44:59.402 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-03-18T15:44:59.402 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/overlayfs 2024-03-18T15:44:59.402 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/overlayfs/overlay.ko 2024-03-18T15:44:59.403 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/pstore 2024-03-18T15:44:59.403 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/pstore/ramoops.ko 2024-03-18T15:44:59.403 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/reiserfs 2024-03-18T15:44:59.403 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/reiserfs/reiserfs.ko 2024-03-18T15:44:59.403 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/romfs 2024-03-18T15:44:59.403 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/romfs/romfs.ko 2024-03-18T15:44:59.403 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/smb 2024-03-18T15:44:59.403 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/smb/client 2024-03-18T15:44:59.403 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/smb/client/cifs.ko 2024-03-18T15:44:59.403 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/smb/common 2024-03-18T15:44:59.403 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/smb/common/cifs_arc4.ko 2024-03-18T15:44:59.403 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/smb/common/cifs_md4.ko 2024-03-18T15:44:59.403 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/squashfs 2024-03-18T15:44:59.404 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/squashfs/squashfs.ko 2024-03-18T15:44:59.404 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/sysv 2024-03-18T15:44:59.404 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/sysv/sysv.ko 2024-03-18T15:44:59.404 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/udf 2024-03-18T15:44:59.404 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/udf/udf.ko 2024-03-18T15:44:59.404 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ufs 2024-03-18T15:44:59.404 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/ufs/ufs.ko 2024-03-18T15:44:59.404 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/xfs 2024-03-18T15:44:59.404 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/fs/xfs/xfs.ko 2024-03-18T15:44:59.404 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/kernel 2024-03-18T15:44:59.404 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/kernel/rcu 2024-03-18T15:44:59.404 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/kernel/rcu/rcutorture.ko 2024-03-18T15:44:59.404 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/kernel/torture.ko 2024-03-18T15:44:59.404 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/kernel/trace 2024-03-18T15:44:59.404 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-03-18T15:44:59.405 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib 2024-03-18T15:44:59.405 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/asn1_encoder.ko 2024-03-18T15:44:59.405 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/crc-itu-t.ko 2024-03-18T15:44:59.405 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/crc8.ko 2024-03-18T15:44:59.405 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/crypto 2024-03-18T15:44:59.405 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/crypto/libarc4.ko 2024-03-18T15:44:59.405 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/crypto/libdes.ko 2024-03-18T15:44:59.405 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/libcrc32c.ko 2024-03-18T15:44:59.405 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/lru_cache.ko 2024-03-18T15:44:59.405 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/lz4 2024-03-18T15:44:59.405 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/lz4/lz4_compress.ko 2024-03-18T15:44:59.405 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/lz4/lz4hc_compress.ko 2024-03-18T15:44:59.405 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/math 2024-03-18T15:44:59.406 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/math/cordic.ko 2024-03-18T15:44:59.406 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/raid6 2024-03-18T15:44:59.406 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/raid6/raid6_pq.ko 2024-03-18T15:44:59.406 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/reed_solomon 2024-03-18T15:44:59.406 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/reed_solomon/reed_solomon.ko 2024-03-18T15:44:59.406 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/ts_bm.ko 2024-03-18T15:44:59.406 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/ts_fsm.ko 2024-03-18T15:44:59.406 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/ts_kmp.ko 2024-03-18T15:44:59.406 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/zstd 2024-03-18T15:44:59.406 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/lib/zstd/zstd_compress.ko 2024-03-18T15:44:59.406 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/mm 2024-03-18T15:44:59.406 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/mm/hwpoison-inject.ko 2024-03-18T15:44:59.406 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net 2024-03-18T15:44:59.406 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/6lowpan 2024-03-18T15:44:59.407 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/6lowpan/6lowpan.ko 2024-03-18T15:44:59.407 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/6lowpan/nhc_dest.ko 2024-03-18T15:44:59.407 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/6lowpan/nhc_fragment.ko 2024-03-18T15:44:59.407 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/6lowpan/nhc_hop.ko 2024-03-18T15:44:59.407 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/6lowpan/nhc_ipv6.ko 2024-03-18T15:44:59.407 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/6lowpan/nhc_mobility.ko 2024-03-18T15:44:59.407 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/6lowpan/nhc_routing.ko 2024-03-18T15:44:59.407 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/6lowpan/nhc_udp.ko 2024-03-18T15:44:59.407 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/802 2024-03-18T15:44:59.407 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/802/garp.ko 2024-03-18T15:44:59.407 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/802/mrp.ko 2024-03-18T15:44:59.407 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/802/p8022.ko 2024-03-18T15:44:59.407 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/802/psnap.ko 2024-03-18T15:44:59.407 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/802/stp.ko 2024-03-18T15:44:59.407 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/8021q 2024-03-18T15:44:59.407 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/8021q/8021q.ko 2024-03-18T15:44:59.408 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/9p 2024-03-18T15:44:59.408 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/9p/9pnet.ko 2024-03-18T15:44:59.408 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/9p/9pnet_fd.ko 2024-03-18T15:44:59.408 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/9p/9pnet_rdma.ko 2024-03-18T15:44:59.408 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/9p/9pnet_virtio.ko 2024-03-18T15:44:59.408 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/batman-adv 2024-03-18T15:44:59.408 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/batman-adv/batman-adv.ko 2024-03-18T15:44:59.408 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge 2024-03-18T15:44:59.408 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/br_netfilter.ko 2024-03-18T15:44:59.408 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/bridge.ko 2024-03-18T15:44:59.408 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter 2024-03-18T15:44:59.408 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-03-18T15:44:59.408 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_among.ko 2024-03-18T15:44:59.408 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_arp.ko 2024-03-18T15:44:59.409 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-03-18T15:44:59.409 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-03-18T15:44:59.409 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_ip.ko 2024-03-18T15:44:59.409 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-03-18T15:44:59.409 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_limit.ko 2024-03-18T15:44:59.409 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_log.ko 2024-03-18T15:44:59.409 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_mark.ko 2024-03-18T15:44:59.409 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-03-18T15:44:59.409 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-03-18T15:44:59.409 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-03-18T15:44:59.409 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-03-18T15:44:59.409 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_snat.ko 2024-03-18T15:44:59.409 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_stp.ko 2024-03-18T15:44:59.409 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-03-18T15:44:59.409 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-03-18T15:44:59.410 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-03-18T15:44:59.410 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-03-18T15:44:59.410 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/ebtables.ko 2024-03-18T15:44:59.410 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-03-18T15:44:59.410 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-03-18T15:44:59.410 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-03-18T15:44:59.410 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ceph 2024-03-18T15:44:59.410 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ceph/libceph.ko 2024-03-18T15:44:59.410 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/core 2024-03-18T15:44:59.410 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/core/failover.ko 2024-03-18T15:44:59.410 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/core/pktgen.ko 2024-03-18T15:44:59.410 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/dccp 2024-03-18T15:44:59.410 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/dccp/dccp.ko 2024-03-18T15:44:59.410 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/dccp/dccp_diag.ko 2024-03-18T15:44:59.410 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/dccp/dccp_ipv4.ko 2024-03-18T15:44:59.411 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/dccp/dccp_ipv6.ko 2024-03-18T15:44:59.411 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/dns_resolver 2024-03-18T15:44:59.411 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/dns_resolver/dns_resolver.ko 2024-03-18T15:44:59.411 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/dsa 2024-03-18T15:44:59.411 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/dsa/dsa_core.ko 2024-03-18T15:44:59.411 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/dsa/tag_brcm.ko 2024-03-18T15:44:59.411 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/dsa/tag_dsa.ko 2024-03-18T15:44:59.411 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/dsa/tag_none.ko 2024-03-18T15:44:59.411 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/dsa/tag_trailer.ko 2024-03-18T15:44:59.411 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ieee802154 2024-03-18T15:44:59.411 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ieee802154/6lowpan 2024-03-18T15:44:59.411 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-03-18T15:44:59.411 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ieee802154/ieee802154.ko 2024-03-18T15:44:59.411 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ieee802154/ieee802154_socket.ko 2024-03-18T15:44:59.411 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4 2024-03-18T15:44:59.412 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/ah4.ko 2024-03-18T15:44:59.412 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/esp4.ko 2024-03-18T15:44:59.412 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/fou.ko 2024-03-18T15:44:59.412 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/gre.ko 2024-03-18T15:44:59.412 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/inet_diag.ko 2024-03-18T15:44:59.412 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/ip_gre.ko 2024-03-18T15:44:59.412 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/ip_tunnel.ko 2024-03-18T15:44:59.412 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/ip_vti.ko 2024-03-18T15:44:59.412 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/ipcomp.ko 2024-03-18T15:44:59.412 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/ipip.ko 2024-03-18T15:44:59.412 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter 2024-03-18T15:44:59.412 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/arp_tables.ko 2024-03-18T15:44:59.412 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-03-18T15:44:59.412 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-03-18T15:44:59.412 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-03-18T15:44:59.413 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-03-18T15:44:59.413 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-03-18T15:44:59.413 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-03-18T15:44:59.413 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-03-18T15:44:59.413 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-03-18T15:44:59.413 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-03-18T15:44:59.413 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/iptable_security.ko 2024-03-18T15:44:59.413 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-03-18T15:44:59.413 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-03-18T15:44:59.413 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-03-18T15:44:59.413 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-03-18T15:44:59.413 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-03-18T15:44:59.413 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-03-18T15:44:59.413 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-03-18T15:44:59.413 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-03-18T15:44:59.414 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-03-18T15:44:59.414 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-03-18T15:44:59.414 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tcp_bic.ko 2024-03-18T15:44:59.414 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tcp_dctcp.ko 2024-03-18T15:44:59.414 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tcp_diag.ko 2024-03-18T15:44:59.414 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tcp_highspeed.ko 2024-03-18T15:44:59.414 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tcp_htcp.ko 2024-03-18T15:44:59.414 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tcp_hybla.ko 2024-03-18T15:44:59.414 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tcp_illinois.ko 2024-03-18T15:44:59.414 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tcp_lp.ko 2024-03-18T15:44:59.414 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tcp_scalable.ko 2024-03-18T15:44:59.414 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tcp_vegas.ko 2024-03-18T15:44:59.414 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tcp_veno.ko 2024-03-18T15:44:59.414 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tcp_westwood.ko 2024-03-18T15:44:59.414 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tcp_yeah.ko 2024-03-18T15:44:59.415 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/tunnel4.ko 2024-03-18T15:44:59.415 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/udp_diag.ko 2024-03-18T15:44:59.415 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/udp_tunnel.ko 2024-03-18T15:44:59.415 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv4/xfrm4_tunnel.ko 2024-03-18T15:44:59.415 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6 2024-03-18T15:44:59.415 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/ah6.ko 2024-03-18T15:44:59.415 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/esp6.ko 2024-03-18T15:44:59.415 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/fou6.ko 2024-03-18T15:44:59.415 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/ip6_tunnel.ko 2024-03-18T15:44:59.415 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-03-18T15:44:59.415 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/ip6_vti.ko 2024-03-18T15:44:59.415 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/ipcomp6.ko 2024-03-18T15:44:59.415 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter 2024-03-18T15:44:59.415 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-03-18T15:44:59.415 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-03-18T15:44:59.416 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-03-18T15:44:59.416 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-03-18T15:44:59.416 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-03-18T15:44:59.416 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-03-18T15:44:59.416 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-03-18T15:44:59.416 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-03-18T15:44:59.416 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-03-18T15:44:59.416 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-03-18T15:44:59.416 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-03-18T15:44:59.416 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-03-18T15:44:59.416 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-03-18T15:44:59.416 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-03-18T15:44:59.416 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-03-18T15:44:59.417 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-03-18T15:44:59.417 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-03-18T15:44:59.417 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-03-18T15:44:59.417 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-03-18T15:44:59.417 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-03-18T15:44:59.417 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-03-18T15:44:59.417 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-03-18T15:44:59.417 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-03-18T15:44:59.417 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-03-18T15:44:59.417 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/sit.ko 2024-03-18T15:44:59.417 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/tunnel6.ko 2024-03-18T15:44:59.417 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/ipv6/xfrm6_tunnel.ko 2024-03-18T15:44:59.417 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/key 2024-03-18T15:44:59.417 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/key/af_key.ko 2024-03-18T15:44:59.417 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/l2tp 2024-03-18T15:44:59.418 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/l2tp/l2tp_core.ko 2024-03-18T15:44:59.418 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/l2tp/l2tp_debugfs.ko 2024-03-18T15:44:59.418 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/l2tp/l2tp_eth.ko 2024-03-18T15:44:59.418 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/l2tp/l2tp_ip.ko 2024-03-18T15:44:59.418 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/l2tp/l2tp_ip6.ko 2024-03-18T15:44:59.418 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/l2tp/l2tp_netlink.ko 2024-03-18T15:44:59.418 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/l2tp/l2tp_ppp.ko 2024-03-18T15:44:59.418 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/llc 2024-03-18T15:44:59.418 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/llc/llc.ko 2024-03-18T15:44:59.418 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/mac802154 2024-03-18T15:44:59.418 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/mac802154/mac802154.ko 2024-03-18T15:44:59.418 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/mpls 2024-03-18T15:44:59.418 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/mpls/mpls_gso.ko 2024-03-18T15:44:59.418 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter 2024-03-18T15:44:59.419 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset 2024-03-18T15:44:59.419 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set.ko 2024-03-18T15:44:59.419 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-03-18T15:44:59.419 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-03-18T15:44:59.419 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-03-18T15:44:59.419 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-03-18T15:44:59.419 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-03-18T15:44:59.419 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-03-18T15:44:59.419 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-03-18T15:44:59.419 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-03-18T15:44:59.419 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-03-18T15:44:59.419 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-03-18T15:44:59.419 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-03-18T15:44:59.419 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-03-18T15:44:59.420 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-03-18T15:44:59.420 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-03-18T15:44:59.420 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-03-18T15:44:59.420 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs 2024-03-18T15:44:59.420 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs.ko 2024-03-18T15:44:59.420 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-03-18T15:44:59.420 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-03-18T15:44:59.420 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-03-18T15:44:59.420 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-03-18T15:44:59.420 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-03-18T15:44:59.420 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-03-18T15:44:59.420 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-03-18T15:44:59.420 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-03-18T15:44:59.421 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-03-18T15:44:59.421 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-03-18T15:44:59.421 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-03-18T15:44:59.421 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-03-18T15:44:59.421 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-03-18T15:44:59.421 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conncount.ko 2024-03-18T15:44:59.421 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conntrack.ko 2024-03-18T15:44:59.421 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-03-18T15:44:59.421 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-03-18T15:44:59.421 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-03-18T15:44:59.421 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conntrack_h323.ko 2024-03-18T15:44:59.421 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conntrack_irc.ko 2024-03-18T15:44:59.421 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-03-18T15:44:59.422 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-03-18T15:44:59.422 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-03-18T15:44:59.422 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conntrack_sane.ko 2024-03-18T15:44:59.422 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conntrack_sip.ko 2024-03-18T15:44:59.422 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-03-18T15:44:59.422 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-03-18T15:44:59.422 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_dup_netdev.ko 2024-03-18T15:44:59.422 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_flow_table.ko 2024-03-18T15:44:59.422 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_flow_table_inet.ko 2024-03-18T15:44:59.422 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_log_syslog.ko 2024-03-18T15:44:59.422 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_nat.ko 2024-03-18T15:44:59.422 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_nat_amanda.ko 2024-03-18T15:44:59.422 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_nat_ftp.ko 2024-03-18T15:44:59.422 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_nat_irc.ko 2024-03-18T15:44:59.422 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_nat_sip.ko 2024-03-18T15:44:59.423 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_nat_tftp.ko 2024-03-18T15:44:59.423 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_synproxy_core.ko 2024-03-18T15:44:59.423 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nf_tables.ko 2024-03-18T15:44:59.423 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nfnetlink.ko 2024-03-18T15:44:59.423 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nfnetlink_acct.ko 2024-03-18T15:44:59.423 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nfnetlink_log.ko 2024-03-18T15:44:59.423 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nfnetlink_osf.ko 2024-03-18T15:44:59.423 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nfnetlink_queue.ko 2024-03-18T15:44:59.423 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_chain_nat.ko 2024-03-18T15:44:59.423 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_compat.ko 2024-03-18T15:44:59.423 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_ct.ko 2024-03-18T15:44:59.423 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_dup_netdev.ko 2024-03-18T15:44:59.423 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_fib.ko 2024-03-18T15:44:59.423 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_fwd_netdev.ko 2024-03-18T15:44:59.423 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_hash.ko 2024-03-18T15:44:59.424 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_limit.ko 2024-03-18T15:44:59.424 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_log.ko 2024-03-18T15:44:59.424 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_masq.ko 2024-03-18T15:44:59.424 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_nat.ko 2024-03-18T15:44:59.424 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_numgen.ko 2024-03-18T15:44:59.424 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_queue.ko 2024-03-18T15:44:59.424 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_quota.ko 2024-03-18T15:44:59.424 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_redir.ko 2024-03-18T15:44:59.424 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_reject.ko 2024-03-18T15:44:59.424 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_reject_inet.ko 2024-03-18T15:44:59.424 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_socket.ko 2024-03-18T15:44:59.424 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/nft_tunnel.ko 2024-03-18T15:44:59.424 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_AUDIT.ko 2024-03-18T15:44:59.424 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_CHECKSUM.ko 2024-03-18T15:44:59.424 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_CLASSIFY.ko 2024-03-18T15:44:59.425 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-03-18T15:44:59.425 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_CT.ko 2024-03-18T15:44:59.425 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_DSCP.ko 2024-03-18T15:44:59.425 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_HL.ko 2024-03-18T15:44:59.425 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_HMARK.ko 2024-03-18T15:44:59.425 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_IDLETIMER.ko 2024-03-18T15:44:59.425 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_LED.ko 2024-03-18T15:44:59.425 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_LOG.ko 2024-03-18T15:44:59.425 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_MASQUERADE.ko 2024-03-18T15:44:59.425 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_NETMAP.ko 2024-03-18T15:44:59.425 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_NFLOG.ko 2024-03-18T15:44:59.425 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_NFQUEUE.ko 2024-03-18T15:44:59.425 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_RATEEST.ko 2024-03-18T15:44:59.425 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_REDIRECT.ko 2024-03-18T15:44:59.425 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_SECMARK.ko 2024-03-18T15:44:59.426 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_TCPMSS.ko 2024-03-18T15:44:59.426 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-03-18T15:44:59.426 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_TEE.ko 2024-03-18T15:44:59.426 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_TPROXY.ko 2024-03-18T15:44:59.426 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_TRACE.ko 2024-03-18T15:44:59.426 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_addrtype.ko 2024-03-18T15:44:59.426 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_bpf.ko 2024-03-18T15:44:59.426 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_cgroup.ko 2024-03-18T15:44:59.426 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_cluster.ko 2024-03-18T15:44:59.426 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_comment.ko 2024-03-18T15:44:59.426 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_connbytes.ko 2024-03-18T15:44:59.426 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_connlabel.ko 2024-03-18T15:44:59.426 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_connlimit.ko 2024-03-18T15:44:59.426 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_connmark.ko 2024-03-18T15:44:59.426 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_conntrack.ko 2024-03-18T15:44:59.427 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_cpu.ko 2024-03-18T15:44:59.427 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_dccp.ko 2024-03-18T15:44:59.427 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_devgroup.ko 2024-03-18T15:44:59.427 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_dscp.ko 2024-03-18T15:44:59.427 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_ecn.ko 2024-03-18T15:44:59.427 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_esp.ko 2024-03-18T15:44:59.427 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_hashlimit.ko 2024-03-18T15:44:59.427 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_helper.ko 2024-03-18T15:44:59.427 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_hl.ko 2024-03-18T15:44:59.427 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_ipcomp.ko 2024-03-18T15:44:59.427 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_iprange.ko 2024-03-18T15:44:59.427 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_ipvs.ko 2024-03-18T15:44:59.427 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_l2tp.ko 2024-03-18T15:44:59.427 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_length.ko 2024-03-18T15:44:59.428 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_limit.ko 2024-03-18T15:44:59.428 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_mac.ko 2024-03-18T15:44:59.428 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_mark.ko 2024-03-18T15:44:59.428 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_multiport.ko 2024-03-18T15:44:59.428 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_nat.ko 2024-03-18T15:44:59.428 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_nfacct.ko 2024-03-18T15:44:59.428 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_osf.ko 2024-03-18T15:44:59.428 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_owner.ko 2024-03-18T15:44:59.428 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_physdev.ko 2024-03-18T15:44:59.428 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_pkttype.ko 2024-03-18T15:44:59.428 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_policy.ko 2024-03-18T15:44:59.428 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_quota.ko 2024-03-18T15:44:59.428 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_rateest.ko 2024-03-18T15:44:59.428 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_realm.ko 2024-03-18T15:44:59.428 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_recent.ko 2024-03-18T15:44:59.429 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_sctp.ko 2024-03-18T15:44:59.429 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_set.ko 2024-03-18T15:44:59.429 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_socket.ko 2024-03-18T15:44:59.429 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_state.ko 2024-03-18T15:44:59.429 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_statistic.ko 2024-03-18T15:44:59.429 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_string.ko 2024-03-18T15:44:59.429 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_tcpmss.ko 2024-03-18T15:44:59.429 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_time.ko 2024-03-18T15:44:59.429 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netfilter/xt_u32.ko 2024-03-18T15:44:59.429 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netlink 2024-03-18T15:44:59.429 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/netlink/netlink_diag.ko 2024-03-18T15:44:59.429 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/nsh 2024-03-18T15:44:59.429 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/nsh/nsh.ko 2024-03-18T15:44:59.429 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/openvswitch 2024-03-18T15:44:59.430 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/openvswitch/openvswitch.ko 2024-03-18T15:44:59.430 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/openvswitch/vport-geneve.ko 2024-03-18T15:44:59.430 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/openvswitch/vport-gre.ko 2024-03-18T15:44:59.430 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/openvswitch/vport-vxlan.ko 2024-03-18T15:44:59.430 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/packet 2024-03-18T15:44:59.430 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/packet/af_packet_diag.ko 2024-03-18T15:44:59.430 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/rds 2024-03-18T15:44:59.430 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/rds/rds.ko 2024-03-18T15:44:59.430 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/rds/rds_rdma.ko 2024-03-18T15:44:59.430 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/rds/rds_tcp.ko 2024-03-18T15:44:59.430 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched 2024-03-18T15:44:59.430 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/act_csum.ko 2024-03-18T15:44:59.430 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/act_gact.ko 2024-03-18T15:44:59.430 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/act_mirred.ko 2024-03-18T15:44:59.431 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/act_nat.ko 2024-03-18T15:44:59.431 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/act_pedit.ko 2024-03-18T15:44:59.431 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/act_police.ko 2024-03-18T15:44:59.431 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/act_simple.ko 2024-03-18T15:44:59.431 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/act_skbedit.ko 2024-03-18T15:44:59.431 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/act_vlan.ko 2024-03-18T15:44:59.431 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/cls_basic.ko 2024-03-18T15:44:59.431 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/cls_bpf.ko 2024-03-18T15:44:59.431 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/cls_flow.ko 2024-03-18T15:44:59.431 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/cls_fw.ko 2024-03-18T15:44:59.431 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/cls_route.ko 2024-03-18T15:44:59.431 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/cls_u32.ko 2024-03-18T15:44:59.431 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/em_cmp.ko 2024-03-18T15:44:59.431 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/em_ipset.ko 2024-03-18T15:44:59.431 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/em_meta.ko 2024-03-18T15:44:59.432 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/em_nbyte.ko 2024-03-18T15:44:59.432 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/em_text.ko 2024-03-18T15:44:59.432 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/em_u32.ko 2024-03-18T15:44:59.432 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_choke.ko 2024-03-18T15:44:59.432 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_codel.ko 2024-03-18T15:44:59.432 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_drr.ko 2024-03-18T15:44:59.432 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_fq.ko 2024-03-18T15:44:59.432 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_gred.ko 2024-03-18T15:44:59.432 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_hfsc.ko 2024-03-18T15:44:59.432 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_hhf.ko 2024-03-18T15:44:59.432 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_htb.ko 2024-03-18T15:44:59.432 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_ingress.ko 2024-03-18T15:44:59.432 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_mqprio.ko 2024-03-18T15:44:59.432 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_mqprio_lib.ko 2024-03-18T15:44:59.432 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_multiq.ko 2024-03-18T15:44:59.433 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_netem.ko 2024-03-18T15:44:59.433 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_pie.ko 2024-03-18T15:44:59.433 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_plug.ko 2024-03-18T15:44:59.433 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_prio.ko 2024-03-18T15:44:59.433 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_qfq.ko 2024-03-18T15:44:59.433 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_red.ko 2024-03-18T15:44:59.433 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_sfb.ko 2024-03-18T15:44:59.433 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_sfq.ko 2024-03-18T15:44:59.433 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_tbf.ko 2024-03-18T15:44:59.433 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sched/sch_teql.ko 2024-03-18T15:44:59.433 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sctp 2024-03-18T15:44:59.433 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sctp/sctp.ko 2024-03-18T15:44:59.433 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sctp/sctp_diag.ko 2024-03-18T15:44:59.433 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sunrpc 2024-03-18T15:44:59.433 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sunrpc/auth_gss 2024-03-18T15:44:59.434 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-03-18T15:44:59.434 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-03-18T15:44:59.434 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sunrpc/sunrpc.ko 2024-03-18T15:44:59.434 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sunrpc/xprtrdma 2024-03-18T15:44:59.434 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-03-18T15:44:59.434 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/tipc 2024-03-18T15:44:59.434 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/tipc/diag.ko 2024-03-18T15:44:59.434 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/tipc/tipc.ko 2024-03-18T15:44:59.434 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/unix 2024-03-18T15:44:59.434 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/unix/unix_diag.ko 2024-03-18T15:44:59.434 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/vmw_vsock 2024-03-18T15:44:59.434 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-03-18T15:44:59.434 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-03-18T15:44:59.435 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/vmw_vsock/vsock.ko 2024-03-18T15:44:59.435 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/vmw_vsock/vsock_diag.ko 2024-03-18T15:44:59.435 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/vmw_vsock/vsock_loopback.ko 2024-03-18T15:44:59.435 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/xfrm 2024-03-18T15:44:59.435 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/net/xfrm/xfrm_ipcomp.ko 2024-03-18T15:44:59.435 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/security 2024-03-18T15:44:59.435 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/security/keys 2024-03-18T15:44:59.435 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/security/keys/encrypted-keys 2024-03-18T15:44:59.435 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-03-18T15:44:59.435 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/security/keys/trusted-keys 2024-03-18T15:44:59.435 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/security/keys/trusted-keys/trusted.ko 2024-03-18T15:44:59.435 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/virt 2024-03-18T15:44:59.435 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/virt/lib 2024-03-18T15:44:59.435 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/kernel/virt/lib/irqbypass.ko 2024-03-18T15:44:59.435 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/modules.alias 2024-03-18T15:44:59.435 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/modules.alias.bin 2024-03-18T15:44:59.436 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/modules.builtin 2024-03-18T15:44:59.436 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/modules.builtin.alias.bin 2024-03-18T15:44:59.436 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/modules.builtin.bin 2024-03-18T15:44:59.436 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/modules.builtin.modinfo 2024-03-18T15:44:59.436 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/modules.dep 2024-03-18T15:44:59.436 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/modules.dep.bin 2024-03-18T15:44:59.436 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/modules.devname 2024-03-18T15:44:59.436 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/modules.order 2024-03-18T15:44:59.436 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/modules.softdep 2024-03-18T15:44:59.436 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/modules.symbols 2024-03-18T15:44:59.436 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/modules.symbols.bin 2024-03-18T15:44:59.436 INFO:teuthology.orchestra.run.smithi039.stdout:/lib/modules/6.8.0-rc7-g3845b7ac715a/vmlinuz 2024-03-18T15:44:59.440 DEBUG:teuthology.task.kernel:get_image_version: 6.8.0-rc7-g3845b7ac715a 2024-03-18T15:44:59.441 DEBUG:teuthology.orchestra.run.smithi039:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-03-18T15:44:59.463 INFO:teuthology.orchestra.run.smithi039.stdout:postinstall scriptlet (using /bin/sh): 2024-03-18T15:44:59.464 INFO:teuthology.orchestra.run.smithi039.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-03-18T15:44:59.464 INFO:teuthology.orchestra.run.smithi039.stdout: /usr/bin/kernel-install add 6.8.0-rc7-g3845b7ac715a /lib/modules/6.8.0-rc7-g3845b7ac715a/vmlinuz 2024-03-18T15:44:59.464 INFO:teuthology.orchestra.run.smithi039.stdout:fi 2024-03-18T15:44:59.464 INFO:teuthology.orchestra.run.smithi039.stdout:for file in vmlinuz System.map config; do 2024-03-18T15:44:59.464 INFO:teuthology.orchestra.run.smithi039.stdout: if ! cmp --silent "/lib/modules/6.8.0-rc7-g3845b7ac715a/${file}" "/boot/${file}-6.8.0-rc7-g3845b7ac715a"; then 2024-03-18T15:44:59.464 INFO:teuthology.orchestra.run.smithi039.stdout: cp "/lib/modules/6.8.0-rc7-g3845b7ac715a/${file}" "/boot/${file}-6.8.0-rc7-g3845b7ac715a" 2024-03-18T15:44:59.464 INFO:teuthology.orchestra.run.smithi039.stdout: fi 2024-03-18T15:44:59.464 INFO:teuthology.orchestra.run.smithi039.stdout:done 2024-03-18T15:44:59.464 INFO:teuthology.orchestra.run.smithi039.stdout:preuninstall scriptlet (using /bin/sh): 2024-03-18T15:44:59.464 INFO:teuthology.orchestra.run.smithi039.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2024-03-18T15:44:59.464 INFO:teuthology.orchestra.run.smithi039.stdout:new-kernel-pkg --remove 6.8.0-rc7-g3845b7ac715a --rminitrd --initrdfile=/boot/initramfs-6.8.0-rc7-g3845b7ac715a.img 2024-03-18T15:44:59.464 INFO:teuthology.orchestra.run.smithi039.stdout:elif [ -x /usr/bin/kernel-install ]; then 2024-03-18T15:44:59.464 INFO:teuthology.orchestra.run.smithi039.stdout:kernel-install remove 6.8.0-rc7-g3845b7ac715a 2024-03-18T15:44:59.464 INFO:teuthology.orchestra.run.smithi039.stdout:fi 2024-03-18T15:44:59.465 INFO:teuthology.orchestra.run.smithi039.stdout:postuninstall scriptlet (using /bin/sh): 2024-03-18T15:44:59.465 INFO:teuthology.orchestra.run.smithi039.stdout:if [ -x /sbin/update-bootloader ]; then 2024-03-18T15:44:59.465 INFO:teuthology.orchestra.run.smithi039.stdout:/sbin/update-bootloader --remove 6.8.0-rc7-g3845b7ac715a 2024-03-18T15:44:59.465 INFO:teuthology.orchestra.run.smithi039.stdout:fi 2024-03-18T15:44:59.466 DEBUG:teuthology.orchestra.run.smithi039:> sudo rpm -qi grub2-tools 2024-03-18T15:44:59.552 INFO:teuthology.orchestra.run.smithi039.stdout:Name : grub2-tools 2024-03-18T15:44:59.553 INFO:teuthology.orchestra.run.smithi039.stdout:Epoch : 1 2024-03-18T15:44:59.553 INFO:teuthology.orchestra.run.smithi039.stdout:Version : 2.06 2024-03-18T15:44:59.553 INFO:teuthology.orchestra.run.smithi039.stdout:Release : 68.el9 2024-03-18T15:44:59.553 INFO:teuthology.orchestra.run.smithi039.stdout:Architecture: x86_64 2024-03-18T15:44:59.553 INFO:teuthology.orchestra.run.smithi039.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-18T15:44:59.553 INFO:teuthology.orchestra.run.smithi039.stdout:Group : Unspecified 2024-03-18T15:44:59.553 INFO:teuthology.orchestra.run.smithi039.stdout:Size : 8264254 2024-03-18T15:44:59.553 INFO:teuthology.orchestra.run.smithi039.stdout:License : GPLv3+ 2024-03-18T15:44:59.553 INFO:teuthology.orchestra.run.smithi039.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-18T15:44:59.553 INFO:teuthology.orchestra.run.smithi039.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-18T15:44:59.553 INFO:teuthology.orchestra.run.smithi039.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-18T15:44:59.553 INFO:teuthology.orchestra.run.smithi039.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-18T15:44:59.553 INFO:teuthology.orchestra.run.smithi039.stdout:Packager : builder@centos.org 2024-03-18T15:44:59.554 INFO:teuthology.orchestra.run.smithi039.stdout:Vendor : CentOS 2024-03-18T15:44:59.554 INFO:teuthology.orchestra.run.smithi039.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-18T15:44:59.554 INFO:teuthology.orchestra.run.smithi039.stdout:Summary : Support tools for GRUB. 2024-03-18T15:44:59.554 INFO:teuthology.orchestra.run.smithi039.stdout:Description : 2024-03-18T15:44:59.554 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T15:44:59.554 INFO:teuthology.orchestra.run.smithi039.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-18T15:44:59.554 INFO:teuthology.orchestra.run.smithi039.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-18T15:44:59.554 INFO:teuthology.orchestra.run.smithi039.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-18T15:44:59.554 INFO:teuthology.orchestra.run.smithi039.stdout:hardware devices. 2024-03-18T15:44:59.554 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T15:44:59.554 INFO:teuthology.orchestra.run.smithi039.stdout:This subpackage provides tools for support of all platforms. 2024-03-18T15:44:59.555 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-18T15:44:59.555 INFO:teuthology.task.kernel:Updating grub on smithi039 to boot 6.8.0-rc7-g3845b7ac715a 2024-03-18T15:44:59.555 DEBUG:teuthology.orchestra.run.smithi039:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-18T15:45:00.143 INFO:teuthology.orchestra.run.smithi039.stderr:Generating grub configuration file ... 2024-03-18T15:45:01.532 INFO:teuthology.orchestra.run.smithi039.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-18T15:45:01.560 INFO:teuthology.orchestra.run.smithi039.stderr:done 2024-03-18T15:45:01.562 DEBUG:teuthology.orchestra.run.smithi039:> mktemp 2024-03-18T15:45:01.577 INFO:teuthology.orchestra.run.smithi039.stdout:/tmp/tmp.gPOc5y0bCN 2024-03-18T15:45:01.577 DEBUG:teuthology.orchestra.run.smithi039:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.gPOc5y0bCN 2024-03-18T15:45:01.649 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 0666 /tmp/tmp.gPOc5y0bCN 2024-03-18T15:45:01.805 DEBUG:teuthology.orchestra.remote:smithi039:/tmp/tmp.gPOc5y0bCN is 6KB 2024-03-18T15:45:01.854 DEBUG:teuthology.orchestra.run.smithi039:> rm -fr /tmp/tmp.gPOc5y0bCN 2024-03-18T15:45:01.868 DEBUG:teuthology.orchestra.run.smithi039:> sudo /bin/ls /boot/loader/entries || true 2024-03-18T15:45:01.936 INFO:teuthology.orchestra.run.smithi039.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-18T15:45:01.936 INFO:teuthology.orchestra.run.smithi039.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-18T15:45:01.936 INFO:teuthology.orchestra.run.smithi039.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-18T15:45:01.936 INFO:teuthology.orchestra.run.smithi039.stdout:15949e3c88704300882482eddfd7baae-6.8.0-rc7-g3845b7ac715a.conf 2024-03-18T15:45:01.937 DEBUG:teuthology.orchestra.run.smithi039:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.8.0-rc7-g3845b7ac715a 2024-03-18T15:45:02.096 DEBUG:teuthology.orchestra.run.smithi039:> sudo shutdown -r now 2024-03-18T15:45:32.100 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-03-18T15:45:32.101 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-03-18T15:45:32.103 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T15:46:06.750 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.39 2024-03-18T15:46:15.752 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-03-18T15:46:15.753 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T15:46:18.814 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.39 2024-03-18T15:46:30.815 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-03-18T15:46:30.816 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T15:47:30.818 DEBUG:teuthology.orchestra.remote:timed out 2024-03-18T15:47:45.819 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-03-18T15:47:45.820 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T15:47:46.318 DEBUG:teuthology.orchestra.run.smithi039:> true 2024-03-18T15:47:46.994 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-03-18T15:47:46.994 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.8.0-rc7-g3845b7ac715a"... 2024-03-18T15:47:46.994 DEBUG:teuthology.orchestra.run.smithi039:> uname -r 2024-03-18T15:47:47.046 INFO:teuthology.orchestra.run.smithi039.stdout:6.8.0-rc7-g3845b7ac715a 2024-03-18T15:47:47.047 DEBUG:teuthology.task.kernel:current kernel version is 6.8.0-rc7-g3845b7ac715a vs 6.8.0-rc7-g3845b7ac715a 2024-03-18T15:47:47.047 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-18T15:47:47.047 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-18T15:47:48.048 DEBUG:teuthology.parallel:result is None 2024-03-18T15:47:48.048 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-18T15:47:48.055 INFO:teuthology.task.internal:Creating test directory... 2024-03-18T15:47:48.056 DEBUG:teuthology.orchestra.run.smithi039:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-18T15:47:48.063 DEBUG:teuthology.orchestra.run.smithi194:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-18T15:47:48.066 DEBUG:teuthology.orchestra.run.smithi196:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-18T15:47:48.114 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-18T15:47:48.122 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-18T15:47:48.151 INFO:teuthology.task.internal:Creating archive directory... 2024-03-18T15:47:48.152 DEBUG:teuthology.orchestra.run.smithi039:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-18T15:47:48.159 DEBUG:teuthology.orchestra.run.smithi194:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-18T15:47:48.161 DEBUG:teuthology.orchestra.run.smithi196:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-18T15:47:48.282 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-18T15:47:48.290 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-18T15:47:48.290 DEBUG:teuthology.orchestra.run.smithi194:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-18T15:47:48.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T15:47:48.307 DEBUG:teuthology.orchestra.run.smithi196:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-18T15:47:48.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T15:47:48.324 DEBUG:teuthology.orchestra.run.smithi039:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-18T15:47:48.331 DEBUG:teuthology.orchestra.run.smithi194:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-18T15:47:48.349 DEBUG:teuthology.orchestra.run.smithi196:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-18T15:47:48.386 INFO:teuthology.orchestra.run.smithi194.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-18T15:47:48.404 INFO:teuthology.orchestra.run.smithi194.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-18T15:47:48.413 INFO:teuthology.orchestra.run.smithi196.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-18T15:47:48.433 INFO:teuthology.orchestra.run.smithi196.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-18T15:47:48.487 INFO:teuthology.orchestra.run.smithi039.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-18T15:47:48.550 INFO:teuthology.orchestra.run.smithi039.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-18T15:47:48.555 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-18T15:47:48.563 INFO:teuthology.task.internal:Configuring sudo... 2024-03-18T15:47:48.563 DEBUG:teuthology.orchestra.run.smithi039:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-18T15:47:48.570 DEBUG:teuthology.orchestra.run.smithi194:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-18T15:47:48.572 DEBUG:teuthology.orchestra.run.smithi196:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-18T15:47:48.666 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-18T15:47:48.685 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-18T15:47:48.686 DEBUG:teuthology.orchestra.run.smithi039:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-18T15:47:48.693 DEBUG:teuthology.orchestra.run.smithi194:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-18T15:47:48.695 DEBUG:teuthology.orchestra.run.smithi196:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-18T15:47:48.744 DEBUG:teuthology.orchestra.run.smithi039:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-18T15:47:48.827 DEBUG:teuthology.orchestra.run.smithi039:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-18T15:47:48.951 DEBUG:teuthology.orchestra.run.smithi039:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-18T15:47:49.039 DEBUG:teuthology.orchestra.run.smithi039:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-18T15:47:49.142 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-03-18T15:47:49.142 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-18T15:47:49.235 DEBUG:teuthology.orchestra.run.smithi194:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-18T15:47:49.279 DEBUG:teuthology.orchestra.run.smithi194:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-18T15:47:49.339 DEBUG:teuthology.orchestra.run.smithi194:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-18T15:47:49.382 DEBUG:teuthology.orchestra.run.smithi194:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-18T15:47:49.426 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-03-18T15:47:49.426 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-18T15:47:49.494 DEBUG:teuthology.orchestra.run.smithi196:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-18T15:47:49.539 DEBUG:teuthology.orchestra.run.smithi196:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-18T15:47:49.597 DEBUG:teuthology.orchestra.run.smithi196:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-18T15:47:49.681 DEBUG:teuthology.orchestra.run.smithi196:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-18T15:47:49.727 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-18T15:47:49.727 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-18T15:47:49.796 DEBUG:teuthology.orchestra.run.smithi039:> sudo service rsyslog restart 2024-03-18T15:47:49.803 DEBUG:teuthology.orchestra.run.smithi194:> sudo service rsyslog restart 2024-03-18T15:47:49.805 DEBUG:teuthology.orchestra.run.smithi196:> sudo service rsyslog restart 2024-03-18T15:47:49.843 INFO:teuthology.orchestra.run.smithi194.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-18T15:47:49.874 INFO:teuthology.orchestra.run.smithi196.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-18T15:47:49.924 INFO:teuthology.orchestra.run.smithi039.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-18T15:47:50.420 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-18T15:47:50.428 INFO:teuthology.task.internal:Starting timer... 2024-03-18T15:47:50.428 INFO:teuthology.run_tasks:Running task pcp... 2024-03-18T15:47:50.470 INFO:teuthology.run_tasks:Running task selinux... 2024-03-18T15:47:50.502 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-03-18T15:47:50.503 DEBUG:teuthology.orchestra.run.smithi039:> sudo service auditd rotate 2024-03-18T15:47:50.634 INFO:teuthology.orchestra.run.smithi039.stdout:Rotating logs: 2024-03-18T15:47:50.638 DEBUG:teuthology.orchestra.run.smithi194:> sudo service auditd rotate 2024-03-18T15:47:50.697 INFO:teuthology.orchestra.run.smithi194.stdout:Rotating logs: 2024-03-18T15:47:50.699 DEBUG:teuthology.orchestra.run.smithi196:> sudo service auditd rotate 2024-03-18T15:47:50.756 INFO:teuthology.orchestra.run.smithi196.stdout:Rotating logs: 2024-03-18T15:47:50.758 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-18T15:47:50.758 DEBUG:teuthology.orchestra.run.smithi039:> /usr/sbin/getenforce 2024-03-18T15:47:50.815 INFO:teuthology.orchestra.run.smithi039.stdout:Permissive 2024-03-18T15:47:50.815 DEBUG:teuthology.orchestra.run.smithi194:> /usr/sbin/getenforce 2024-03-18T15:47:50.832 INFO:teuthology.orchestra.run.smithi194.stdout:Permissive 2024-03-18T15:47:50.833 DEBUG:teuthology.orchestra.run.smithi196:> /usr/sbin/getenforce 2024-03-18T15:47:50.857 INFO:teuthology.orchestra.run.smithi196.stdout:Permissive 2024-03-18T15:47:50.857 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi039.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi194.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi196.front.sepia.ceph.com': 'permissive'} 2024-03-18T15:47:50.857 DEBUG:teuthology.orchestra.run.smithi039:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-18T15:47:50.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T15:47:50.932 DEBUG:teuthology.orchestra.run.smithi194:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-18T15:47:50.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T15:47:50.959 DEBUG:teuthology.orchestra.run.smithi196:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-18T15:47:50.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T15:47:50.988 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-18T15:47:50.988 DEBUG:teuthology.orchestra.run.smithi039:> sudo /usr/sbin/setenforce permissive 2024-03-18T15:47:51.069 DEBUG:teuthology.orchestra.run.smithi194:> sudo /usr/sbin/setenforce permissive 2024-03-18T15:47:51.108 DEBUG:teuthology.orchestra.run.smithi196:> sudo /usr/sbin/setenforce permissive 2024-03-18T15:47:51.142 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-18T15:47:51.153 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-18T15:47:51.159 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-18T15:47:51.281 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-18T15:47:51.295 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-18T15:47:51.296 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi039.front.sepia.ceph.com,smithi194.front.sepia.ceph.com,smithi196.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-18T15:53:56.053 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi039.front.sepia.ceph.com'), Remote(name='ubuntu@smithi194.front.sepia.ceph.com'), Remote(name='ubuntu@smithi196.front.sepia.ceph.com')] 2024-03-18T15:53:56.054 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-03-18T15:53:56.055 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T15:53:56.206 DEBUG:teuthology.orchestra.run.smithi039:> true 2024-03-18T15:53:56.363 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi039.front.sepia.ceph.com' 2024-03-18T15:53:56.363 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-03-18T15:53:56.363 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi194.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T15:53:56.449 DEBUG:teuthology.orchestra.run.smithi194:> true 2024-03-18T15:53:56.516 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-03-18T15:53:56.517 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-03-18T15:53:56.517 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-18T15:53:56.604 DEBUG:teuthology.orchestra.run.smithi196:> true 2024-03-18T15:53:56.672 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-03-18T15:53:56.673 INFO:teuthology.run_tasks:Running task clock... 2024-03-18T15:53:56.687 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-18T15:53:56.687 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-18T15:53:56.687 DEBUG:teuthology.orchestra.run.smithi039:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-18T15:53:56.695 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-18T15:53:56.695 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-18T15:53:56.697 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-18T15:53:56.697 DEBUG:teuthology.orchestra.run.smithi196:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-18T15:53:56.725 INFO:teuthology.orchestra.run.smithi194.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-18T15:53:56.742 INFO:teuthology.orchestra.run.smithi194.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-18T15:53:56.745 INFO:teuthology.orchestra.run.smithi196.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-18T15:53:56.759 INFO:teuthology.orchestra.run.smithi196.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-18T15:53:56.771 INFO:teuthology.orchestra.run.smithi194.stderr:sudo: ntpd: command not found 2024-03-18T15:53:56.784 INFO:teuthology.orchestra.run.smithi194.stdout:506 Cannot talk to daemon 2024-03-18T15:53:56.788 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: ntpd: command not found 2024-03-18T15:53:56.794 INFO:teuthology.orchestra.run.smithi039.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-18T15:53:56.801 INFO:teuthology.orchestra.run.smithi194.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-18T15:53:56.801 INFO:teuthology.orchestra.run.smithi196.stdout:506 Cannot talk to daemon 2024-03-18T15:53:56.817 INFO:teuthology.orchestra.run.smithi194.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-18T15:53:56.818 INFO:teuthology.orchestra.run.smithi196.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-18T15:53:56.834 INFO:teuthology.orchestra.run.smithi196.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-18T15:53:56.849 INFO:teuthology.orchestra.run.smithi039.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-18T15:53:56.861 INFO:teuthology.orchestra.run.smithi194.stderr:bash: line 1: ntpq: command not found 2024-03-18T15:53:56.864 INFO:teuthology.orchestra.run.smithi194.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-18T15:53:56.864 INFO:teuthology.orchestra.run.smithi194.stdout:=============================================================================== 2024-03-18T15:53:56.864 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T15:53:56.864 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T15:53:56.865 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T15:53:56.865 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T15:53:56.877 INFO:teuthology.orchestra.run.smithi196.stderr:bash: line 1: ntpq: command not found 2024-03-18T15:53:56.881 INFO:teuthology.orchestra.run.smithi196.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-18T15:53:56.881 INFO:teuthology.orchestra.run.smithi196.stdout:=============================================================================== 2024-03-18T15:53:56.881 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T15:53:56.881 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T15:53:56.881 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T15:53:56.881 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T15:53:56.950 INFO:teuthology.orchestra.run.smithi039.stderr:sudo: ntpd: command not found 2024-03-18T15:53:56.994 INFO:teuthology.orchestra.run.smithi039.stdout:506 Cannot talk to daemon 2024-03-18T15:53:57.050 INFO:teuthology.orchestra.run.smithi039.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-18T15:53:57.103 INFO:teuthology.orchestra.run.smithi039.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-18T15:53:57.236 INFO:teuthology.orchestra.run.smithi039.stderr:bash: line 1: ntpq: command not found 2024-03-18T15:53:57.243 INFO:teuthology.orchestra.run.smithi039.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-18T15:53:57.243 INFO:teuthology.orchestra.run.smithi039.stdout:=============================================================================== 2024-03-18T15:53:57.243 INFO:teuthology.orchestra.run.smithi039.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T15:53:57.243 INFO:teuthology.orchestra.run.smithi039.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T15:53:57.243 INFO:teuthology.orchestra.run.smithi039.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T15:53:57.243 INFO:teuthology.orchestra.run.smithi039.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T15:53:57.245 INFO:teuthology.run_tasks:Running task install... 2024-03-18T15:53:57.253 DEBUG:teuthology.task.install:project ceph 2024-03-18T15:53:57.253 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af'}} 2024-03-18T15:53:57.254 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af'} 2024-03-18T15:53:57.254 INFO:teuthology.task.install:Using flavor: default 2024-03-18T15:53:57.259 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-18T15:53:57.260 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-18T15:53:57.260 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', 'tag': None, 'wait_for_package': False} 2024-03-18T15:53:57.260 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T15:53:57.263 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', 'tag': None, 'wait_for_package': False} 2024-03-18T15:53:57.263 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T15:53:57.264 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', 'tag': None, 'wait_for_package': False} 2024-03-18T15:53:57.264 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T15:53:57.412 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/ 2024-03-18T15:53:57.412 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2162.g3d78f5ba 2024-03-18T15:53:57.417 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/ 2024-03-18T15:53:57.417 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2162.g3d78f5ba 2024-03-18T15:53:57.525 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-18T15:53:57.525 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-03-18T15:53:57.525 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-18T15:53:57.531 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-18T15:53:57.531 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-18T15:53:57.531 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-18T15:53:57.559 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-18T15:53:57.560 DEBUG:teuthology.orchestra.run.smithi196:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-18T15:53:57.585 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/ 2024-03-18T15:53:57.585 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2162.g3d78f5ba 2024-03-18T15:53:57.626 DEBUG:teuthology.orchestra.run.smithi196:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-18T15:53:57.630 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-18T15:53:57.630 DEBUG:teuthology.orchestra.run.smithi039:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-18T15:53:57.706 DEBUG:teuthology.orchestra.run.smithi196:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-18T15:53:57.771 DEBUG:teuthology.orchestra.run.smithi039:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-18T15:53:57.772 INFO:teuthology.orchestra.run.smithi196.stdout:check_obsoletes = 1 2024-03-18T15:53:57.774 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum clean all 2024-03-18T15:53:57.821 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-18T15:53:57.821 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-03-18T15:53:57.821 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-18T15:53:57.852 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-18T15:53:57.853 DEBUG:teuthology.orchestra.run.smithi194:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-18T15:53:57.908 DEBUG:teuthology.orchestra.run.smithi039:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-18T15:53:57.924 DEBUG:teuthology.orchestra.run.smithi194:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-18T15:53:58.001 INFO:teuthology.orchestra.run.smithi039.stdout:check_obsoletes = 1 2024-03-18T15:53:58.006 DEBUG:teuthology.orchestra.run.smithi194:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-18T15:53:58.006 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum clean all 2024-03-18T15:53:58.009 INFO:teuthology.orchestra.run.smithi196.stdout:74 files removed 2024-03-18T15:53:58.035 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-18T15:53:58.074 INFO:teuthology.orchestra.run.smithi194.stdout:check_obsoletes = 1 2024-03-18T15:53:58.076 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum clean all 2024-03-18T15:53:58.309 INFO:teuthology.orchestra.run.smithi194.stdout:74 files removed 2024-03-18T15:53:58.336 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-18T15:53:58.369 INFO:teuthology.orchestra.run.smithi039.stdout:74 files removed 2024-03-18T15:53:58.412 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-18T15:53:58.754 INFO:teuthology.orchestra.run.smithi196.stdout:ceph packages for x86_64 194 kB/s | 83 kB 00:00 2024-03-18T15:53:58.948 INFO:teuthology.orchestra.run.smithi194.stdout:ceph packages for x86_64 258 kB/s | 83 kB 00:00 2024-03-18T15:53:59.197 INFO:teuthology.orchestra.run.smithi196.stdout:ceph noarch packages 25 kB/s | 10 kB 00:00 2024-03-18T15:53:59.390 INFO:teuthology.orchestra.run.smithi194.stdout:ceph noarch packages 25 kB/s | 10 kB 00:00 2024-03-18T15:53:59.454 INFO:teuthology.orchestra.run.smithi039.stdout:ceph packages for x86_64 157 kB/s | 83 kB 00:00 2024-03-18T15:53:59.631 INFO:teuthology.orchestra.run.smithi196.stdout:ceph source packages 4.7 kB/s | 1.9 kB 00:00 2024-03-18T15:53:59.832 INFO:teuthology.orchestra.run.smithi194.stdout:ceph source packages 4.6 kB/s | 1.9 kB 00:00 2024-03-18T15:54:00.079 INFO:teuthology.orchestra.run.smithi039.stdout:ceph noarch packages 19 kB/s | 10 kB 00:00 2024-03-18T15:54:00.450 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS-9 - AppStream 8.5 MB/s | 6.7 MB 00:00 2024-03-18T15:54:00.693 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS-9 - AppStream 8.0 MB/s | 6.7 MB 00:00 2024-03-18T15:54:00.755 INFO:teuthology.orchestra.run.smithi039.stdout:ceph source packages 3.2 kB/s | 1.9 kB 00:00 2024-03-18T15:54:01.859 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS-9 - AppStream 6.5 MB/s | 6.7 MB 00:01 2024-03-18T15:54:02.058 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS-9 - BaseOS 4.0 MB/s | 2.0 MB 00:00 2024-03-18T15:54:02.209 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS-9 - BaseOS 5.0 MB/s | 2.0 MB 00:00 2024-03-18T15:54:02.807 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - BaseOS 19 MB/s | 8.0 MB 00:00 2024-03-18T15:54:03.821 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS-9 - BaseOS 3.2 MB/s | 2.0 MB 00:00 2024-03-18T15:54:04.008 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 9 - BaseOS 5.4 MB/s | 8.0 MB 00:01 2024-03-18T15:54:07.072 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS Stream 9 - BaseOS 2.8 MB/s | 8.0 MB 00:02 2024-03-18T15:54:08.492 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 9 - AppStream 5.5 MB/s | 19 MB 00:03 2024-03-18T15:54:12.718 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 9 - CRB 8.3 MB/s | 6.1 MB 00:00 2024-03-18T15:54:14.435 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 9 - Extras packages 26 kB/s | 16 kB 00:00 2024-03-18T15:54:17.006 INFO:teuthology.orchestra.run.smithi194.stdout:Extra Packages for Enterprise Linux 8.4 MB/s | 21 MB 00:02 2024-03-18T15:54:22.020 INFO:teuthology.orchestra.run.smithi194.stdout:lab-extras 16 kB/s | 1.7 kB 00:00 2024-03-18T15:54:23.855 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS Stream 9 - AppStream 1.2 MB/s | 19 MB 00:15 2024-03-18T15:54:24.476 INFO:teuthology.orchestra.run.smithi194.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-18T15:54:24.477 INFO:teuthology.orchestra.run.smithi194.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-18T15:54:24.491 INFO:teuthology.orchestra.run.smithi194.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-18T15:54:24.492 INFO:teuthology.orchestra.run.smithi194.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-18T15:54:24.493 INFO:teuthology.orchestra.run.smithi194.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-18T15:54:24.494 INFO:teuthology.orchestra.run.smithi194.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-18T15:54:24.496 INFO:teuthology.orchestra.run.smithi194.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-18T15:54:24.499 INFO:teuthology.orchestra.run.smithi194.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-18T15:54:24.500 INFO:teuthology.orchestra.run.smithi194.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-18T15:54:24.579 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-03-18T15:54:24.594 INFO:teuthology.orchestra.run.smithi194.stdout:============================================================================================ 2024-03-18T15:54:24.594 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-03-18T15:54:24.594 INFO:teuthology.orchestra.run.smithi194.stdout:============================================================================================ 2024-03-18T15:54:24.595 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-03-18T15:54:24.595 INFO:teuthology.orchestra.run.smithi194.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-18T15:54:24.595 INFO:teuthology.orchestra.run.smithi194.stdout: ceph x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 6.6 k 2024-03-18T15:54:24.595 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-base x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.4 M 2024-03-18T15:54:24.595 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 890 k 2024-03-18T15:54:24.595 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 145 k 2024-03-18T15:54:24.595 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 1.5 M 2024-03-18T15:54:24.595 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 139 k 2024-03-18T15:54:24.595 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 1.8 M 2024-03-18T15:54:24.595 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 7.4 M 2024-03-18T15:54:24.595 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-rook noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 49 k 2024-03-18T15:54:24.595 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-radosgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 9.8 M 2024-03-18T15:54:24.595 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-test x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 48 M 2024-03-18T15:54:24.595 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-volume noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 264 k 2024-03-18T15:54:24.596 INFO:teuthology.orchestra.run.smithi194.stdout: cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 748 k 2024-03-18T15:54:24.596 INFO:teuthology.orchestra.run.smithi194.stdout: cephfs-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 221 k 2024-03-18T15:54:24.596 INFO:teuthology.orchestra.run.smithi194.stdout: cephfs-shell x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 27 k 2024-03-18T15:54:24.596 INFO:teuthology.orchestra.run.smithi194.stdout: cephfs-top noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 18 k 2024-03-18T15:54:24.596 INFO:teuthology.orchestra.run.smithi194.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-18T15:54:24.596 INFO:teuthology.orchestra.run.smithi194.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-18T15:54:24.596 INFO:teuthology.orchestra.run.smithi194.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-18T15:54:24.596 INFO:teuthology.orchestra.run.smithi194.stdout: libcephfs-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 32 k 2024-03-18T15:54:24.596 INFO:teuthology.orchestra.run.smithi194.stdout: libcephfs2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 747 k 2024-03-18T15:54:24.596 INFO:teuthology.orchestra.run.smithi194.stdout: librados-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 127 k 2024-03-18T15:54:24.596 INFO:teuthology.orchestra.run.smithi194.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-18T15:54:24.596 INFO:teuthology.orchestra.run.smithi194.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-18T15:54:24.596 INFO:teuthology.orchestra.run.smithi194.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-18T15:54:24.596 INFO:teuthology.orchestra.run.smithi194.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-18T15:54:24.596 INFO:teuthology.orchestra.run.smithi194.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-18T15:54:24.597 INFO:teuthology.orchestra.run.smithi194.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-18T15:54:24.597 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cephfs x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 162 k 2024-03-18T15:54:24.597 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rados x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 320 k 2024-03-18T15:54:24.597 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 298 k 2024-03-18T15:54:24.597 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 99 k 2024-03-18T15:54:24.597 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 85 k 2024-03-18T15:54:24.597 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-18T15:54:24.597 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-nbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 170 k 2024-03-18T15:54:24.597 INFO:teuthology.orchestra.run.smithi194.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-18T15:54:24.597 INFO:teuthology.orchestra.run.smithi194.stdout:Upgrading: 2024-03-18T15:54:24.597 INFO:teuthology.orchestra.run.smithi194.stdout: NetworkManager x86_64 1:1.46.0-1.el9 baseos 2.3 M 2024-03-18T15:54:24.597 INFO:teuthology.orchestra.run.smithi194.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 baseos 1.8 M 2024-03-18T15:54:24.597 INFO:teuthology.orchestra.run.smithi194.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 baseos 42 k 2024-03-18T15:54:24.597 INFO:teuthology.orchestra.run.smithi194.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 baseos 248 k 2024-03-18T15:54:24.598 INFO:teuthology.orchestra.run.smithi194.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-18T15:54:24.598 INFO:teuthology.orchestra.run.smithi194.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-18T15:54:24.598 INFO:teuthology.orchestra.run.smithi194.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-18T15:54:24.598 INFO:teuthology.orchestra.run.smithi194.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-18T15:54:24.598 INFO:teuthology.orchestra.run.smithi194.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-18T15:54:24.598 INFO:teuthology.orchestra.run.smithi194.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-18T15:54:24.598 INFO:teuthology.orchestra.run.smithi194.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-18T15:54:24.598 INFO:teuthology.orchestra.run.smithi194.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-18T15:54:24.598 INFO:teuthology.orchestra.run.smithi194.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-18T15:54:24.598 INFO:teuthology.orchestra.run.smithi194.stdout: librados2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.4 M 2024-03-18T15:54:24.598 INFO:teuthology.orchestra.run.smithi194.stdout: librbd1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-18T15:54:24.598 INFO:teuthology.orchestra.run.smithi194.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-18T15:54:24.598 INFO:teuthology.orchestra.run.smithi194.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-18T15:54:24.598 INFO:teuthology.orchestra.run.smithi194.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-18T15:54:24.598 INFO:teuthology.orchestra.run.smithi194.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-18T15:54:24.599 INFO:teuthology.orchestra.run.smithi194.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-18T15:54:24.599 INFO:teuthology.orchestra.run.smithi194.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-18T15:54:24.599 INFO:teuthology.orchestra.run.smithi194.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-18T15:54:24.599 INFO:teuthology.orchestra.run.smithi194.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-18T15:54:24.599 INFO:teuthology.orchestra.run.smithi194.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-18T15:54:24.599 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-03-18T15:54:24.599 INFO:teuthology.orchestra.run.smithi194.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-18T15:54:24.599 INFO:teuthology.orchestra.run.smithi194.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-18T15:54:24.599 INFO:teuthology.orchestra.run.smithi194.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-18T15:54:24.599 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 21 M 2024-03-18T15:54:24.599 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 34 k 2024-03-18T15:54:24.599 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mds x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 2.4 M 2024-03-18T15:54:24.599 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 248 k 2024-03-18T15:54:24.599 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mon x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 4.7 M 2024-03-18T15:54:24.600 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-osd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 17 M 2024-03-18T15:54:24.600 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 16 k 2024-03-18T15:54:24.600 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-selinux x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 25 k 2024-03-18T15:54:24.600 INFO:teuthology.orchestra.run.smithi194.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-18T15:54:24.600 INFO:teuthology.orchestra.run.smithi194.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-18T15:54:24.600 INFO:teuthology.orchestra.run.smithi194.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-18T15:54:24.600 INFO:teuthology.orchestra.run.smithi194.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-18T15:54:24.600 INFO:teuthology.orchestra.run.smithi194.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-18T15:54:24.600 INFO:teuthology.orchestra.run.smithi194.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-18T15:54:24.600 INFO:teuthology.orchestra.run.smithi194.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-18T15:54:24.600 INFO:teuthology.orchestra.run.smithi194.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-18T15:54:24.600 INFO:teuthology.orchestra.run.smithi194.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-18T15:54:24.600 INFO:teuthology.orchestra.run.smithi194.stdout: libcephsqlite x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 163 k 2024-03-18T15:54:24.600 INFO:teuthology.orchestra.run.smithi194.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-18T15:54:24.600 INFO:teuthology.orchestra.run.smithi194.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-18T15:54:24.601 INFO:teuthology.orchestra.run.smithi194.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-18T15:54:24.601 INFO:teuthology.orchestra.run.smithi194.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-18T15:54:24.601 INFO:teuthology.orchestra.run.smithi194.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-18T15:54:24.601 INFO:teuthology.orchestra.run.smithi194.stdout: libradosstriper1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 501 k 2024-03-18T15:54:24.601 INFO:teuthology.orchestra.run.smithi194.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-18T15:54:24.601 INFO:teuthology.orchestra.run.smithi194.stdout: librgw2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.1 M 2024-03-18T15:54:24.601 INFO:teuthology.orchestra.run.smithi194.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-18T15:54:24.601 INFO:teuthology.orchestra.run.smithi194.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-18T15:54:24.601 INFO:teuthology.orchestra.run.smithi194.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-18T15:54:24.601 INFO:teuthology.orchestra.run.smithi194.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-18T15:54:24.601 INFO:teuthology.orchestra.run.smithi194.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-18T15:54:24.601 INFO:teuthology.orchestra.run.smithi194.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-18T15:54:24.602 INFO:teuthology.orchestra.run.smithi194.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-18T15:54:24.602 INFO:teuthology.orchestra.run.smithi194.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-18T15:54:24.602 INFO:teuthology.orchestra.run.smithi194.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-18T15:54:24.602 INFO:teuthology.orchestra.run.smithi194.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-18T15:54:24.602 INFO:teuthology.orchestra.run.smithi194.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-18T15:54:24.602 INFO:teuthology.orchestra.run.smithi194.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-18T15:54:24.602 INFO:teuthology.orchestra.run.smithi194.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-18T15:54:24.602 INFO:teuthology.orchestra.run.smithi194.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-18T15:54:24.602 INFO:teuthology.orchestra.run.smithi194.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-18T15:54:24.602 INFO:teuthology.orchestra.run.smithi194.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-18T15:54:24.602 INFO:teuthology.orchestra.run.smithi194.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-18T15:54:24.602 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-18T15:54:24.602 INFO:teuthology.orchestra.run.smithi194.stdout: python3-ceph-argparse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 45 k 2024-03-18T15:54:24.602 INFO:teuthology.orchestra.run.smithi194.stdout: python3-ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 131 k 2024-03-18T15:54:24.602 INFO:teuthology.orchestra.run.smithi194.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-18T15:54:24.603 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-18T15:54:24.603 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-18T15:54:24.603 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-18T15:54:24.603 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-18T15:54:24.603 INFO:teuthology.orchestra.run.smithi194.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-18T15:54:24.603 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-18T15:54:24.603 INFO:teuthology.orchestra.run.smithi194.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-18T15:54:24.603 INFO:teuthology.orchestra.run.smithi194.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-18T15:54:24.603 INFO:teuthology.orchestra.run.smithi194.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-18T15:54:24.603 INFO:teuthology.orchestra.run.smithi194.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-18T15:54:24.603 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-18T15:54:24.603 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-18T15:54:24.603 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-18T15:54:24.603 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-18T15:54:24.604 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-18T15:54:24.604 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-18T15:54:24.604 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-18T15:54:24.604 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-18T15:54:24.604 INFO:teuthology.orchestra.run.smithi194.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-18T15:54:24.604 INFO:teuthology.orchestra.run.smithi194.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-18T15:54:24.604 INFO:teuthology.orchestra.run.smithi194.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-18T15:54:24.604 INFO:teuthology.orchestra.run.smithi194.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-18T15:54:24.604 INFO:teuthology.orchestra.run.smithi194.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-18T15:54:24.604 INFO:teuthology.orchestra.run.smithi194.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-18T15:54:24.604 INFO:teuthology.orchestra.run.smithi194.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-18T15:54:24.604 INFO:teuthology.orchestra.run.smithi194.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-18T15:54:24.604 INFO:teuthology.orchestra.run.smithi194.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-18T15:54:24.604 INFO:teuthology.orchestra.run.smithi194.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-18T15:54:24.604 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-18T15:54:24.604 INFO:teuthology.orchestra.run.smithi194.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-18T15:54:24.605 INFO:teuthology.orchestra.run.smithi194.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-18T15:54:24.605 INFO:teuthology.orchestra.run.smithi194.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-18T15:54:24.605 INFO:teuthology.orchestra.run.smithi194.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-18T15:54:24.605 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-18T15:54:24.605 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-18T15:54:24.605 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-18T15:54:24.605 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-18T15:54:24.605 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-18T15:54:24.605 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-18T15:54:24.605 INFO:teuthology.orchestra.run.smithi194.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-18T15:54:24.605 INFO:teuthology.orchestra.run.smithi194.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-18T15:54:24.605 INFO:teuthology.orchestra.run.smithi194.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-18T15:54:24.605 INFO:teuthology.orchestra.run.smithi194.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-18T15:54:24.605 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-18T15:54:24.605 INFO:teuthology.orchestra.run.smithi194.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-18T15:54:24.605 INFO:teuthology.orchestra.run.smithi194.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-18T15:54:24.606 INFO:teuthology.orchestra.run.smithi194.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-18T15:54:24.606 INFO:teuthology.orchestra.run.smithi194.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-18T15:54:24.606 INFO:teuthology.orchestra.run.smithi194.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-18T15:54:24.606 INFO:teuthology.orchestra.run.smithi194.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-18T15:54:24.606 INFO:teuthology.orchestra.run.smithi194.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-18T15:54:24.606 INFO:teuthology.orchestra.run.smithi194.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-18T15:54:24.606 INFO:teuthology.orchestra.run.smithi194.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-18T15:54:24.606 INFO:teuthology.orchestra.run.smithi194.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-18T15:54:24.606 INFO:teuthology.orchestra.run.smithi194.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-18T15:54:24.606 INFO:teuthology.orchestra.run.smithi194.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-18T15:54:24.606 INFO:teuthology.orchestra.run.smithi194.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-18T15:54:24.606 INFO:teuthology.orchestra.run.smithi194.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-18T15:54:24.606 INFO:teuthology.orchestra.run.smithi194.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-18T15:54:24.606 INFO:teuthology.orchestra.run.smithi194.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-18T15:54:24.607 INFO:teuthology.orchestra.run.smithi194.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-18T15:54:24.607 INFO:teuthology.orchestra.run.smithi194.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-18T15:54:24.607 INFO:teuthology.orchestra.run.smithi194.stdout:Installing weak dependencies: 2024-03-18T15:54:24.607 INFO:teuthology.orchestra.run.smithi194.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-18T15:54:24.607 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-18T15:54:24.607 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-03-18T15:54:24.607 INFO:teuthology.orchestra.run.smithi194.stdout:============================================================================================ 2024-03-18T15:54:24.607 INFO:teuthology.orchestra.run.smithi194.stdout:Install 150 Packages 2024-03-18T15:54:24.607 INFO:teuthology.orchestra.run.smithi194.stdout:Upgrade 24 Packages 2024-03-18T15:54:24.607 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-18T15:54:24.608 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 224 M 2024-03-18T15:54:24.608 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-03-18T15:54:25.971 INFO:teuthology.orchestra.run.smithi194.stdout:(1/174): ceph-19.0.0-2162.g3d78f5ba.el9.x86_64. 21 kB/s | 6.6 kB 00:00 2024-03-18T15:54:26.080 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - AppStream 868 kB/s | 19 MB 00:22 2024-03-18T15:54:26.638 INFO:teuthology.orchestra.run.smithi194.stdout:(2/174): ceph-fuse-19.0.0-2162.g3d78f5ba.el9.x8 1.3 MB/s | 890 kB 00:00 2024-03-18T15:54:26.771 INFO:teuthology.orchestra.run.smithi194.stdout:(3/174): ceph-immutable-object-cache-19.0.0-216 1.1 MB/s | 145 kB 00:00 2024-03-18T15:54:27.105 INFO:teuthology.orchestra.run.smithi194.stdout:(4/174): ceph-common-19.0.0-2162.g3d78f5ba.el9. 14 MB/s | 21 MB 00:01 2024-03-18T15:54:27.213 INFO:teuthology.orchestra.run.smithi194.stdout:(5/174): ceph-base-19.0.0-2162.g3d78f5ba.el9.x8 3.4 MB/s | 5.4 MB 00:01 2024-03-18T15:54:27.255 INFO:teuthology.orchestra.run.smithi194.stdout:(6/174): ceph-mgr-19.0.0-2162.g3d78f5ba.el9.x86 10 MB/s | 1.5 MB 00:00 2024-03-18T15:54:27.581 INFO:teuthology.orchestra.run.smithi194.stdout:(7/174): ceph-mds-19.0.0-2162.g3d78f5ba.el9.x86 2.9 MB/s | 2.4 MB 00:00 2024-03-18T15:54:28.247 INFO:teuthology.orchestra.run.smithi194.stdout:(8/174): ceph-osd-19.0.0-2162.g3d78f5ba.el9.x86 17 MB/s | 17 MB 00:00 2024-03-18T15:54:28.314 INFO:teuthology.orchestra.run.smithi194.stdout:(9/174): ceph-mon-19.0.0-2162.g3d78f5ba.el9.x86 4.3 MB/s | 4.7 MB 00:01 2024-03-18T15:54:28.340 INFO:teuthology.orchestra.run.smithi194.stdout:(10/174): ceph-selinux-19.0.0-2162.g3d78f5ba.el 274 kB/s | 25 kB 00:00 2024-03-18T15:54:28.390 INFO:teuthology.orchestra.run.smithi194.stdout:(11/174): cephfs-mirror-19.0.0-2162.g3d78f5ba.e 4.3 MB/s | 221 kB 00:00 2024-03-18T15:54:28.448 INFO:teuthology.orchestra.run.smithi194.stdout:(12/174): cephfs-shell-19.0.0-2162.g3d78f5ba.el 465 kB/s | 27 kB 00:00 2024-03-18T15:54:28.507 INFO:teuthology.orchestra.run.smithi194.stdout:(13/174): libcephfs-devel-19.0.0-2162.g3d78f5ba 544 kB/s | 32 kB 00:00 2024-03-18T15:54:28.607 INFO:teuthology.orchestra.run.smithi194.stdout:(14/174): libcephfs2-19.0.0-2162.g3d78f5ba.el9. 7.3 MB/s | 747 kB 00:00 2024-03-18T15:54:28.682 INFO:teuthology.orchestra.run.smithi194.stdout:(15/174): libcephsqlite-19.0.0-2162.g3d78f5ba.e 2.1 MB/s | 163 kB 00:00 2024-03-18T15:54:28.733 INFO:teuthology.orchestra.run.smithi194.stdout:(16/174): librados-devel-19.0.0-2162.g3d78f5ba. 2.5 MB/s | 127 kB 00:00 2024-03-18T15:54:28.808 INFO:teuthology.orchestra.run.smithi194.stdout:(17/174): libradosstriper1-19.0.0-2162.g3d78f5b 6.5 MB/s | 501 kB 00:00 2024-03-18T15:54:28.967 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS Stream 9 - CRB 5.9 MB/s | 6.1 MB 00:01 2024-03-18T15:54:29.041 INFO:teuthology.orchestra.run.smithi194.stdout:(18/174): ceph-radosgw-19.0.0-2162.g3d78f5ba.el 6.7 MB/s | 9.8 MB 00:01 2024-03-18T15:54:29.125 INFO:teuthology.orchestra.run.smithi194.stdout:(19/174): librgw2-19.0.0-2162.g3d78f5ba.el9.x86 16 MB/s | 5.1 MB 00:00 2024-03-18T15:54:29.142 INFO:teuthology.orchestra.run.smithi194.stdout:(20/174): python3-ceph-argparse-19.0.0-2162.g3d 451 kB/s | 45 kB 00:00 2024-03-18T15:54:29.167 INFO:teuthology.orchestra.run.smithi194.stdout:(21/174): python3-ceph-common-19.0.0-2162.g3d78 3.1 MB/s | 131 kB 00:00 2024-03-18T15:54:29.209 INFO:teuthology.orchestra.run.smithi194.stdout:(22/174): python3-cephfs-19.0.0-2162.g3d78f5ba. 2.4 MB/s | 162 kB 00:00 2024-03-18T15:54:29.234 INFO:teuthology.orchestra.run.smithi194.stdout:(23/174): python3-rados-19.0.0-2162.g3d78f5ba.e 4.7 MB/s | 320 kB 00:00 2024-03-18T15:54:29.276 INFO:teuthology.orchestra.run.smithi194.stdout:(24/174): python3-rbd-19.0.0-2162.g3d78f5ba.el9 4.3 MB/s | 298 kB 00:00 2024-03-18T15:54:29.302 INFO:teuthology.orchestra.run.smithi194.stdout:(25/174): python3-rgw-19.0.0-2162.g3d78f5ba.el9 1.5 MB/s | 99 kB 00:00 2024-03-18T15:54:29.335 INFO:teuthology.orchestra.run.smithi194.stdout:(26/174): rbd-fuse-19.0.0-2162.g3d78f5ba.el9.x8 1.4 MB/s | 85 kB 00:00 2024-03-18T15:54:29.427 INFO:teuthology.orchestra.run.smithi194.stdout:(27/174): rbd-nbd-19.0.0-2162.g3d78f5ba.el9.x86 1.8 MB/s | 170 kB 00:00 2024-03-18T15:54:29.502 INFO:teuthology.orchestra.run.smithi194.stdout:(28/174): rbd-mirror-19.0.0-2162.g3d78f5ba.el9. 15 MB/s | 3.1 MB 00:00 2024-03-18T15:54:29.519 INFO:teuthology.orchestra.run.smithi194.stdout:(29/174): ceph-grafana-dashboards-19.0.0-2162.g 370 kB/s | 34 kB 00:00 2024-03-18T15:54:29.544 INFO:teuthology.orchestra.run.smithi194.stdout:(30/174): ceph-mgr-cephadm-19.0.0-2162.g3d78f5b 3.3 MB/s | 139 kB 00:00 2024-03-18T15:54:29.820 INFO:teuthology.orchestra.run.smithi194.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2162.g3d78f 6.0 MB/s | 1.8 MB 00:00 2024-03-18T15:54:29.912 INFO:teuthology.orchestra.run.smithi194.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2162.g3d 2.6 MB/s | 248 kB 00:00 2024-03-18T15:54:30.029 INFO:teuthology.orchestra.run.smithi194.stdout:(33/174): ceph-mgr-diskprediction-local-19.0.0- 15 MB/s | 7.4 MB 00:00 2024-03-18T15:54:30.047 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.1 MB 00:00 2024-03-18T15:54:30.054 INFO:teuthology.orchestra.run.smithi194.stdout:(34/174): ceph-mgr-rook-19.0.0-2162.g3d78f5ba.e 347 kB/s | 49 kB 00:00 2024-03-18T15:54:30.080 INFO:teuthology.orchestra.run.smithi194.stdout:(35/174): ceph-prometheus-alerts-19.0.0-2162.g3 328 kB/s | 16 kB 00:00 2024-03-18T15:54:30.121 INFO:teuthology.orchestra.run.smithi194.stdout:(36/174): ceph-volume-19.0.0-2162.g3d78f5ba.el9 3.9 MB/s | 264 kB 00:00 2024-03-18T15:54:30.164 INFO:teuthology.orchestra.run.smithi194.stdout:(37/174): cephadm-19.0.0-2162.g3d78f5ba.el9.noa 8.7 MB/s | 748 kB 00:00 2024-03-18T15:54:30.189 INFO:teuthology.orchestra.run.smithi194.stdout:(38/174): cephfs-top-19.0.0-2162.g3d78f5ba.el9. 269 kB/s | 18 kB 00:00 2024-03-18T15:54:30.365 INFO:teuthology.orchestra.run.smithi194.stdout:(39/174): automake-1.16.2-8.el9.noarch.rpm 3.2 MB/s | 662 kB 00:00 2024-03-18T15:54:30.407 INFO:teuthology.orchestra.run.smithi194.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 4.1 MB/s | 920 kB 00:00 2024-03-18T15:54:30.432 INFO:teuthology.orchestra.run.smithi194.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 1.5 MB/s | 106 kB 00:00 2024-03-18T15:54:30.458 INFO:teuthology.orchestra.run.smithi194.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 6.0 MB/s | 307 kB 00:00 2024-03-18T15:54:30.475 INFO:teuthology.orchestra.run.smithi194.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 763 kB/s | 32 kB 00:00 2024-03-18T15:54:30.567 INFO:teuthology.orchestra.run.smithi194.stdout:(44/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 28 MB/s | 3.0 MB 00:00 2024-03-18T15:54:30.583 INFO:teuthology.orchestra.run.smithi194.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 159 kB/s | 17 kB 00:00 2024-03-18T15:54:30.609 INFO:teuthology.orchestra.run.smithi194.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 240 kB/s | 10 kB 00:00 2024-03-18T15:54:30.634 INFO:teuthology.orchestra.run.smithi194.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-18T15:54:30.651 INFO:teuthology.orchestra.run.smithi194.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-18T15:54:30.685 INFO:teuthology.orchestra.run.smithi194.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-03-18T15:54:30.710 INFO:teuthology.orchestra.run.smithi194.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 9.8 MB/s | 578 kB 00:00 2024-03-18T15:54:30.727 INFO:teuthology.orchestra.run.smithi194.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.1 MB/s | 46 kB 00:00 2024-03-18T15:54:30.752 INFO:teuthology.orchestra.run.smithi194.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-18T15:54:30.778 INFO:teuthology.orchestra.run.smithi194.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2024-03-18T15:54:30.803 INFO:teuthology.orchestra.run.smithi194.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.2 MB/s | 217 kB 00:00 2024-03-18T15:54:30.828 INFO:teuthology.orchestra.run.smithi194.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 687 kB/s | 34 kB 00:00 2024-03-18T15:54:30.928 INFO:teuthology.orchestra.run.smithi194.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 38 MB/s | 4.7 MB 00:00 2024-03-18T15:54:30.970 INFO:teuthology.orchestra.run.smithi194.stdout:(57/174): postgresql-13.11-1.el9.x86_64.rpm 11 MB/s | 1.5 MB 00:00 2024-03-18T15:54:30.995 INFO:teuthology.orchestra.run.smithi194.stdout:(58/174): postgresql-contrib-13.11-1.el9.x86_64 12 MB/s | 811 kB 00:00 2024-03-18T15:54:31.001 INFO:teuthology.orchestra.run.smithi039.stdout:CentOS Stream 9 - Extras packages 26 kB/s | 16 kB 00:00 2024-03-18T15:54:31.021 INFO:teuthology.orchestra.run.smithi194.stdout:(59/174): postgresql-private-libs-13.11-1.el9.x 2.7 MB/s | 134 kB 00:00 2024-03-18T15:54:31.163 INFO:teuthology.orchestra.run.smithi194.stdout:(60/174): postgresql-server-13.11-1.el9.x86_64. 34 MB/s | 5.7 MB 00:00 2024-03-18T15:54:31.196 INFO:teuthology.orchestra.run.smithi194.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 5.8 MB/s | 1.0 MB 00:00 2024-03-18T15:54:31.222 INFO:teuthology.orchestra.run.smithi194.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 78 kB 00:00 2024-03-18T15:54:31.355 INFO:teuthology.orchestra.run.smithi194.stdout:(63/174): python3-babel-2.9.1-2.el9.noarch.rpm 37 MB/s | 5.8 MB 00:00 2024-03-18T15:54:31.380 INFO:teuthology.orchestra.run.smithi194.stdout:(64/174): python3-jsonpatch-1.21-16.el9.noarch. 156 kB/s | 24 kB 00:00 2024-03-18T15:54:31.406 INFO:teuthology.orchestra.run.smithi194.stdout:(65/174): python3-jsonpointer-2.0-4.el9.noarch. 361 kB/s | 18 kB 00:00 2024-03-18T15:54:31.431 INFO:teuthology.orchestra.run.smithi194.stdout:(66/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-03-18T15:54:31.456 INFO:teuthology.orchestra.run.smithi194.stdout:(67/174): python3-markupsafe-1.1.1-12.el9.x86_6 630 kB/s | 32 kB 00:00 2024-03-18T15:54:31.582 INFO:teuthology.orchestra.run.smithi194.stdout:(68/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 32 MB/s | 4.8 MB 00:00 2024-03-18T15:54:31.607 INFO:teuthology.orchestra.run.smithi194.stdout:(69/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.4 MB/s | 217 kB 00:00 2024-03-18T15:54:31.624 INFO:teuthology.orchestra.run.smithi194.stdout:(70/174): python3-oauthlib-3.1.1-5.el9.noarch.r 4.1 MB/s | 174 kB 00:00 2024-03-18T15:54:31.648 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - Extras packages 34 kB/s | 16 kB 00:00 2024-03-18T15:54:31.657 INFO:teuthology.orchestra.run.smithi194.stdout:(71/174): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 69 kB 00:00 2024-03-18T15:54:31.683 INFO:teuthology.orchestra.run.smithi194.stdout:(72/174): python3-prettytable-0.7.2-27.el9.noar 702 kB/s | 41 kB 00:00 2024-03-18T15:54:31.708 INFO:teuthology.orchestra.run.smithi194.stdout:(73/174): python3-protobuf-3.14.0-13.el9.noarch 4.6 MB/s | 238 kB 00:00 2024-03-18T15:54:31.725 INFO:teuthology.orchestra.run.smithi194.stdout:(74/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.1 MB/s | 132 kB 00:00 2024-03-18T15:54:31.750 INFO:teuthology.orchestra.run.smithi194.stdout:(75/174): python3-pyasn1-modules-0.4.8-6.el9.no 4.9 MB/s | 211 kB 00:00 2024-03-18T15:54:31.776 INFO:teuthology.orchestra.run.smithi194.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 950 kB/s | 47 kB 00:00 2024-03-18T15:54:31.801 INFO:teuthology.orchestra.run.smithi194.stdout:(77/174): python3-requests-oauthlib-1.3.0-12.el 856 kB/s | 43 kB 00:00 2024-03-18T15:54:31.985 INFO:teuthology.orchestra.run.smithi194.stdout:(78/174): python3-toml-0.10.2-6.el9.noarch.rpm 212 kB/s | 39 kB 00:00 2024-03-18T15:54:32.218 INFO:teuthology.orchestra.run.smithi194.stdout:(79/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 35 MB/s | 16 MB 00:00 2024-03-18T15:54:32.243 INFO:teuthology.orchestra.run.smithi194.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 160 kB/s | 41 kB 00:00 2024-03-18T15:54:32.269 INFO:teuthology.orchestra.run.smithi194.stdout:(81/174): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-18T15:54:32.294 INFO:teuthology.orchestra.run.smithi194.stdout:(82/174): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 56 kB 00:00 2024-03-18T15:54:32.319 INFO:teuthology.orchestra.run.smithi194.stdout:(83/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-18T15:54:32.344 INFO:teuthology.orchestra.run.smithi194.stdout:(84/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 880 kB/s | 44 kB 00:00 2024-03-18T15:54:32.370 INFO:teuthology.orchestra.run.smithi194.stdout:(85/174): fuse-2.9.9-15.el9.x86_64.rpm 1.5 MB/s | 79 kB 00:00 2024-03-18T15:54:32.395 INFO:teuthology.orchestra.run.smithi194.stdout:(86/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-18T15:54:32.445 INFO:teuthology.orchestra.run.smithi194.stdout:(87/174): python3-cryptography-36.0.1-4.el9.x86 15 MB/s | 1.1 MB 00:00 2024-03-18T15:54:32.470 INFO:teuthology.orchestra.run.smithi194.stdout:(88/174): python3-ply-3.11-14.el9.noarch.rpm 1.4 MB/s | 103 kB 00:00 2024-03-18T15:54:32.496 INFO:teuthology.orchestra.run.smithi194.stdout:(89/174): python3-pycparser-2.20-6.el9.noarch.r 2.4 MB/s | 124 kB 00:00 2024-03-18T15:54:32.521 INFO:teuthology.orchestra.run.smithi194.stdout:(90/174): python3-urllib3-1.26.5-5.el9.noarch.r 3.7 MB/s | 187 kB 00:00 2024-03-18T15:54:32.696 INFO:teuthology.orchestra.run.smithi194.stdout:(91/174): jq-1.6-16.el9.x86_64.rpm 935 kB/s | 187 kB 00:00 2024-03-18T15:54:32.730 INFO:teuthology.orchestra.run.smithi194.stdout:(92/174): libgfortran-11.4.1-3.el9.x86_64.rpm 3.8 MB/s | 803 kB 00:00 2024-03-18T15:54:32.797 INFO:teuthology.orchestra.run.smithi194.stdout:(93/174): python3-requests-2.25.1-8.el9.noarch. 1.8 MB/s | 125 kB 00:00 2024-03-18T15:54:32.822 INFO:teuthology.orchestra.run.smithi194.stdout:(94/174): libquadmath-11.4.1-3.el9.x86_64.rpm 1.5 MB/s | 195 kB 00:00 2024-03-18T15:54:33.439 INFO:teuthology.orchestra.run.smithi194.stdout:(95/174): elfutils-libelf-devel-0.190-2.el9.x86 33 kB/s | 21 kB 00:00 2024-03-18T15:54:33.498 INFO:teuthology.orchestra.run.smithi194.stdout:(96/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 38 kB/s | 25 kB 00:00 2024-03-18T15:54:34.481 INFO:teuthology.orchestra.run.smithi194.stdout:(97/174): python3-devel-3.9.17-2.el9.x86_64.rpm 250 kB/s | 245 kB 00:00 2024-03-18T15:54:34.699 INFO:teuthology.orchestra.run.smithi196.stdout:Extra Packages for Enterprise Linux 7.0 MB/s | 21 MB 00:02 2024-03-18T15:54:35.515 INFO:teuthology.orchestra.run.smithi194.stdout:(98/174): ceph-test-19.0.0-2162.g3d78f5ba.el9.x 6.6 MB/s | 48 MB 00:07 2024-03-18T15:54:35.540 INFO:teuthology.orchestra.run.smithi194.stdout:(99/174): python3-jinja2-2.11.3-5.el9.noarch.rp 234 kB/s | 248 kB 00:01 2024-03-18T15:54:36.140 INFO:teuthology.orchestra.run.smithi194.stdout:(100/174): qatlib-service-23.11.0-1.el9.x86_64. 58 kB/s | 35 kB 00:00 2024-03-18T15:54:36.247 INFO:teuthology.orchestra.run.smithi039.stdout:Extra Packages for Enterprise Linux 4.1 MB/s | 21 MB 00:05 2024-03-18T15:54:36.432 INFO:teuthology.orchestra.run.smithi194.stdout:(101/174): qatlib-23.11.0-1.el9.x86_64.rpm 236 kB/s | 216 kB 00:00 2024-03-18T15:54:36.766 INFO:teuthology.orchestra.run.smithi194.stdout:(102/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 71 kB/s | 44 kB 00:00 2024-03-18T15:54:37.124 INFO:teuthology.orchestra.run.smithi194.stdout:(103/174): lua-devel-5.4.4-4.el9.x86_64.rpm 62 kB/s | 22 kB 00:00 2024-03-18T15:54:37.233 INFO:teuthology.orchestra.run.smithi194.stdout:(104/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 117 kB/s | 94 kB 00:00 2024-03-18T15:54:37.291 INFO:teuthology.orchestra.run.smithi194.stdout:(105/174): protobuf-compiler-3.14.0-13.el9.x86_ 5.0 MB/s | 863 kB 00:00 2024-03-18T15:54:37.567 INFO:teuthology.orchestra.run.smithi194.stdout:(106/174): gperftools-libs-2.9.1-2.el9.x86_64.r 1.1 MB/s | 309 kB 00:00 2024-03-18T15:54:37.609 INFO:teuthology.orchestra.run.smithi194.stdout:(107/174): abseil-cpp-20211102.0-3.el9.x86_64.r 1.4 MB/s | 548 kB 00:00 2024-03-18T15:54:37.634 INFO:teuthology.orchestra.run.smithi194.stdout:(108/174): grpc-data-1.46.7-10.el9.noarch.rpm 290 kB/s | 19 kB 00:00 2024-03-18T15:54:37.768 INFO:teuthology.orchestra.run.smithi194.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 195 kB/s | 26 kB 00:00 2024-03-18T15:54:37.876 INFO:teuthology.orchestra.run.smithi194.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 448 kB/s | 49 kB 00:00 2024-03-18T15:54:37.935 INFO:teuthology.orchestra.run.smithi194.stdout:(111/174): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-03-18T15:54:38.052 INFO:teuthology.orchestra.run.smithi194.stdout:(112/174): luarocks-3.9.2-1.el9.noarch.rpm 1.3 MB/s | 151 kB 00:00 2024-03-18T15:54:38.285 INFO:teuthology.orchestra.run.smithi194.stdout:(113/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.5 MB/s | 838 kB 00:00 2024-03-18T15:54:38.452 INFO:teuthology.orchestra.run.smithi194.stdout:(114/174): libarrow-9.0.0-10.el9.x86_64.rpm 5.2 MB/s | 4.4 MB 00:00 2024-03-18T15:54:38.503 INFO:teuthology.orchestra.run.smithi194.stdout:(115/174): python3-asyncssh-2.13.2-4.el9.noarch 2.5 MB/s | 548 kB 00:00 2024-03-18T15:54:38.536 INFO:teuthology.orchestra.run.smithi194.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 521 kB/s | 43 kB 00:00 2024-03-18T15:54:38.595 INFO:teuthology.orchestra.run.smithi194.stdout:(117/174): python3-certifi-2021.10.8-2.el9.noar 257 kB/s | 15 kB 00:00 2024-03-18T15:54:38.636 INFO:teuthology.orchestra.run.smithi194.stdout:(118/174): python3-cachetools-4.2.4-1.el9.noarc 240 kB/s | 32 kB 00:00 2024-03-18T15:54:38.679 INFO:teuthology.orchestra.run.smithi194.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 2.0 MB/s | 172 kB 00:00 2024-03-18T15:54:38.763 INFO:teuthology.orchestra.run.smithi194.stdout:(120/174): python3-cherrypy-18.6.1-2.el9.noarch 2.8 MB/s | 358 kB 00:00 2024-03-18T15:54:38.787 INFO:teuthology.orchestra.run.smithi194.stdout:(121/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.7 MB/s | 303 kB 00:00 2024-03-18T15:54:38.822 INFO:teuthology.orchestra.run.smithi194.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 609 kB/s | 35 kB 00:00 2024-03-18T15:54:38.880 INFO:teuthology.orchestra.run.smithi194.stdout:(123/174): python3-google-auth-2.28.0-1.el9.noa 2.2 MB/s | 203 kB 00:00 2024-03-18T15:54:38.988 INFO:teuthology.orchestra.run.smithi194.stdout:(124/174): python3-grpcio-tools-1.46.7-10.el9.x 1.3 MB/s | 144 kB 00:00 2024-03-18T15:54:39.072 INFO:teuthology.orchestra.run.smithi194.stdout:(125/174): python3-jaraco-8.2.1-3.el9.noarch.rp 128 kB/s | 11 kB 00:00 2024-03-18T15:54:39.147 INFO:teuthology.orchestra.run.smithi194.stdout:(126/174): python3-jaraco-classes-3.2.1-5.el9.n 236 kB/s | 18 kB 00:00 2024-03-18T15:54:39.197 INFO:teuthology.orchestra.run.smithi194.stdout:(127/174): python3-jaraco-collections-3.0.0-8.e 465 kB/s | 23 kB 00:00 2024-03-18T15:54:39.239 INFO:teuthology.orchestra.run.smithi194.stdout:(128/174): python3-grpcio-1.46.7-10.el9.x86_64. 4.9 MB/s | 2.0 MB 00:00 2024-03-18T15:54:39.273 INFO:teuthology.orchestra.run.smithi194.stdout:(129/174): python3-jaraco-functools-3.5.0-2.el9 258 kB/s | 19 kB 00:00 2024-03-18T15:54:39.306 INFO:teuthology.orchestra.run.smithi194.stdout:(130/174): python3-jaraco-text-3.2.0-6.el9.noar 295 kB/s | 20 kB 00:00 2024-03-18T15:54:39.390 INFO:teuthology.orchestra.run.smithi194.stdout:(131/174): python3-logutils-0.3.5-21.el9.noarch 556 kB/s | 46 kB 00:00 2024-03-18T15:54:39.423 INFO:teuthology.orchestra.run.smithi194.stdout:(132/174): python3-kubernetes-26.1.0-2.el9.noar 6.8 MB/s | 1.0 MB 00:00 2024-03-18T15:54:39.440 INFO:teuthology.orchestra.run.smithi194.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 1.5 MB/s | 79 kB 00:00 2024-03-18T15:54:39.518 INFO:teuthology.orchestra.run.smithi194.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 613 kB/s | 58 kB 00:00 2024-03-18T15:54:39.577 INFO:teuthology.orchestra.run.smithi194.stdout:(135/174): openssl-devel-3.0.7-27.el9.x86_64.rp 689 kB/s | 4.1 MB 00:06 2024-03-18T15:54:39.602 INFO:teuthology.orchestra.run.smithi194.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 197 kB/s | 16 kB 00:00 2024-03-18T15:54:39.627 INFO:teuthology.orchestra.run.smithi194.stdout:(137/174): python3-pecan-1.4.2-3.el9.noarch.rpm 1.4 MB/s | 272 kB 00:00 2024-03-18T15:54:39.677 INFO:teuthology.orchestra.run.smithi194.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 375 kB/s | 28 kB 00:00 2024-03-18T15:54:39.702 INFO:teuthology.orchestra.run.smithi194.stdout:(139/174): python3-repoze-lru-0.7-16.el9.noarch 408 kB/s | 31 kB 00:00 2024-03-18T15:54:39.736 INFO:teuthology.orchestra.run.smithi194.stdout:(140/174): python3-routes-2.5.1-5.el9.noarch.rp 3.2 MB/s | 188 kB 00:00 2024-03-18T15:54:39.761 INFO:teuthology.orchestra.run.smithi194.stdout:(141/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 489 kB/s | 90 kB 00:00 2024-03-18T15:54:39.803 INFO:teuthology.orchestra.run.smithi194.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 588 kB/s | 59 kB 00:00 2024-03-18T15:54:39.828 INFO:teuthology.orchestra.run.smithi194.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 388 kB/s | 36 kB 00:00 2024-03-18T15:54:39.854 INFO:teuthology.orchestra.run.smithi194.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 554 kB/s | 51 kB 00:00 2024-03-18T15:54:39.879 INFO:teuthology.orchestra.run.smithi194.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-03-18T15:54:39.937 INFO:teuthology.orchestra.run.smithi194.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 825 kB/s | 90 kB 00:00 2024-03-18T15:54:39.963 INFO:teuthology.orchestra.run.smithi194.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 239 kB/s | 20 kB 00:00 2024-03-18T15:54:39.982 INFO:teuthology.orchestra.run.smithi196.stdout:lab-extras 5.8 kB/s | 1.7 kB 00:00 2024-03-18T15:54:40.046 INFO:teuthology.orchestra.run.smithi194.stdout:(148/174): re2-20211101-3.el9.x86_64.rpm 1.7 MB/s | 192 kB 00:00 2024-03-18T15:54:40.163 INFO:teuthology.orchestra.run.smithi194.stdout:(149/174): python3-werkzeug-2.0.3-3.el9.1.noarc 1.3 MB/s | 427 kB 00:00 2024-03-18T15:54:40.372 INFO:teuthology.orchestra.run.smithi194.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 3.9 MB/s | 1.6 MB 00:00 2024-03-18T15:54:40.422 INFO:teuthology.orchestra.run.smithi194.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 6.3 MB/s | 321 kB 00:00 2024-03-18T15:54:40.789 INFO:teuthology.orchestra.run.smithi194.stdout:(152/174): librados2-19.0.0-2162.g3d78f5ba.el9. 4.6 MB/s | 3.4 MB 00:00 2024-03-18T15:54:41.031 INFO:teuthology.orchestra.run.smithi194.stdout:(153/174): NetworkManager-libnm-1.46.0-1.el9.x8 7.5 MB/s | 1.8 MB 00:00 2024-03-18T15:54:41.064 INFO:teuthology.orchestra.run.smithi194.stdout:(154/174): NetworkManager-team-1.46.0-1.el9.x86 1.2 MB/s | 42 kB 00:00 2024-03-18T15:54:41.115 INFO:teuthology.orchestra.run.smithi194.stdout:(155/174): librbd1-19.0.0-2162.g3d78f5ba.el9.x8 3.3 MB/s | 3.1 MB 00:00 2024-03-18T15:54:41.140 INFO:teuthology.orchestra.run.smithi194.stdout:(156/174): NetworkManager-tui-1.46.0-1.el9.x86_ 3.2 MB/s | 248 kB 00:00 2024-03-18T15:54:41.182 INFO:teuthology.orchestra.run.smithi194.stdout:(157/174): elfutils-debuginfod-client-0.190-2.e 877 kB/s | 36 kB 00:00 2024-03-18T15:54:41.207 INFO:teuthology.orchestra.run.smithi194.stdout:(158/174): acl-2.3.1-4.el9.x86_64.rpm 775 kB/s | 71 kB 00:00 2024-03-18T15:54:41.232 INFO:teuthology.orchestra.run.smithi194.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 3.7 MB/s | 192 kB 00:00 2024-03-18T15:54:41.274 INFO:teuthology.orchestra.run.smithi194.stdout:(160/174): libacl-2.3.1-4.el9.x86_64.rpm 555 kB/s | 23 kB 00:00 2024-03-18T15:54:41.299 INFO:teuthology.orchestra.run.smithi194.stdout:(161/174): elfutils-libs-0.190-2.el9.x86_64.rpm 2.7 MB/s | 256 kB 00:00 2024-03-18T15:54:41.325 INFO:teuthology.orchestra.run.smithi194.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 2.1 MB/s | 108 kB 00:00 2024-03-18T15:54:41.350 INFO:teuthology.orchestra.run.smithi194.stdout:(163/174): libfdisk-2.37.4-18.el9.x86_64.rpm 3.0 MB/s | 155 kB 00:00 2024-03-18T15:54:41.375 INFO:teuthology.orchestra.run.smithi194.stdout:(164/174): libmount-2.37.4-18.el9.x86_64.rpm 2.7 MB/s | 136 kB 00:00 2024-03-18T15:54:41.401 INFO:teuthology.orchestra.run.smithi194.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 1.2 MB/s | 64 kB 00:00 2024-03-18T15:54:41.426 INFO:teuthology.orchestra.run.smithi194.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 569 kB/s | 29 kB 00:00 2024-03-18T15:54:41.543 INFO:teuthology.orchestra.run.smithi194.stdout:(167/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 18 MB/s | 2.2 MB 00:00 2024-03-18T15:54:41.576 INFO:teuthology.orchestra.run.smithi194.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 6.9 MB/s | 1.2 MB 00:00 2024-03-18T15:54:41.627 INFO:teuthology.orchestra.run.smithi194.stdout:(169/174): util-linux-2.37.4-18.el9.x86_64.rpm 27 MB/s | 2.3 MB 00:00 2024-03-18T15:54:41.652 INFO:teuthology.orchestra.run.smithi194.stdout:(170/174): util-linux-core-2.37.4-18.el9.x86_64 6.0 MB/s | 465 kB 00:00 2024-03-18T15:54:41.677 INFO:teuthology.orchestra.run.smithi194.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 622 kB/s | 31 kB 00:00 2024-03-18T15:54:41.711 INFO:teuthology.orchestra.run.smithi194.stdout:(172/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 18 MB/s | 1.1 MB 00:00 2024-03-18T15:54:41.870 INFO:teuthology.orchestra.run.smithi194.stdout:(173/174): NetworkManager-1.46.0-1.el9.x86_64.r 1.6 MB/s | 2.3 MB 00:01 2024-03-18T15:54:42.308 INFO:teuthology.orchestra.run.smithi039.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-03-18T15:54:42.353 INFO:teuthology.orchestra.run.smithi194.stdout:(174/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 110 kB/s | 74 kB 00:00 2024-03-18T15:54:42.358 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-03-18T15:54:42.358 INFO:teuthology.orchestra.run.smithi194.stdout:Total 13 MB/s | 224 MB 00:17 2024-03-18T15:54:42.454 INFO:teuthology.orchestra.run.smithi196.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-18T15:54:42.455 INFO:teuthology.orchestra.run.smithi196.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-18T15:54:42.469 INFO:teuthology.orchestra.run.smithi196.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-18T15:54:42.471 INFO:teuthology.orchestra.run.smithi196.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-18T15:54:42.472 INFO:teuthology.orchestra.run.smithi196.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-18T15:54:42.473 INFO:teuthology.orchestra.run.smithi196.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-18T15:54:42.478 INFO:teuthology.orchestra.run.smithi196.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-18T15:54:42.478 INFO:teuthology.orchestra.run.smithi196.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-18T15:54:42.479 INFO:teuthology.orchestra.run.smithi196.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-18T15:54:42.557 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-18T15:54:42.580 INFO:teuthology.orchestra.run.smithi196.stdout:============================================================================================ 2024-03-18T15:54:42.580 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-03-18T15:54:42.580 INFO:teuthology.orchestra.run.smithi196.stdout:============================================================================================ 2024-03-18T15:54:42.580 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-18T15:54:42.580 INFO:teuthology.orchestra.run.smithi196.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-18T15:54:42.580 INFO:teuthology.orchestra.run.smithi196.stdout: ceph x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 6.6 k 2024-03-18T15:54:42.580 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-base x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.4 M 2024-03-18T15:54:42.581 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 890 k 2024-03-18T15:54:42.581 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 145 k 2024-03-18T15:54:42.581 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 1.5 M 2024-03-18T15:54:42.581 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 139 k 2024-03-18T15:54:42.581 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 1.8 M 2024-03-18T15:54:42.581 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 7.4 M 2024-03-18T15:54:42.581 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-rook noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 49 k 2024-03-18T15:54:42.581 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-radosgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 9.8 M 2024-03-18T15:54:42.581 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-test x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 48 M 2024-03-18T15:54:42.581 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-volume noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 264 k 2024-03-18T15:54:42.581 INFO:teuthology.orchestra.run.smithi196.stdout: cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 748 k 2024-03-18T15:54:42.581 INFO:teuthology.orchestra.run.smithi196.stdout: cephfs-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 221 k 2024-03-18T15:54:42.581 INFO:teuthology.orchestra.run.smithi196.stdout: cephfs-shell x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 27 k 2024-03-18T15:54:42.581 INFO:teuthology.orchestra.run.smithi196.stdout: cephfs-top noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 18 k 2024-03-18T15:54:42.581 INFO:teuthology.orchestra.run.smithi196.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-18T15:54:42.582 INFO:teuthology.orchestra.run.smithi196.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-18T15:54:42.582 INFO:teuthology.orchestra.run.smithi196.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-18T15:54:42.582 INFO:teuthology.orchestra.run.smithi196.stdout: libcephfs-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 32 k 2024-03-18T15:54:42.582 INFO:teuthology.orchestra.run.smithi196.stdout: libcephfs2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 747 k 2024-03-18T15:54:42.582 INFO:teuthology.orchestra.run.smithi196.stdout: librados-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 127 k 2024-03-18T15:54:42.582 INFO:teuthology.orchestra.run.smithi196.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-18T15:54:42.582 INFO:teuthology.orchestra.run.smithi196.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-18T15:54:42.582 INFO:teuthology.orchestra.run.smithi196.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-18T15:54:42.582 INFO:teuthology.orchestra.run.smithi196.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-18T15:54:42.582 INFO:teuthology.orchestra.run.smithi196.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-18T15:54:42.582 INFO:teuthology.orchestra.run.smithi196.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-18T15:54:42.582 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cephfs x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 162 k 2024-03-18T15:54:42.582 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rados x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 320 k 2024-03-18T15:54:42.583 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 298 k 2024-03-18T15:54:42.583 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 99 k 2024-03-18T15:54:42.583 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 85 k 2024-03-18T15:54:42.583 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-18T15:54:42.583 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-nbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 170 k 2024-03-18T15:54:42.583 INFO:teuthology.orchestra.run.smithi196.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-18T15:54:42.583 INFO:teuthology.orchestra.run.smithi196.stdout:Upgrading: 2024-03-18T15:54:42.583 INFO:teuthology.orchestra.run.smithi196.stdout: NetworkManager x86_64 1:1.46.0-1.el9 baseos 2.3 M 2024-03-18T15:54:42.583 INFO:teuthology.orchestra.run.smithi196.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 baseos 1.8 M 2024-03-18T15:54:42.583 INFO:teuthology.orchestra.run.smithi196.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 baseos 42 k 2024-03-18T15:54:42.583 INFO:teuthology.orchestra.run.smithi196.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 baseos 248 k 2024-03-18T15:54:42.583 INFO:teuthology.orchestra.run.smithi196.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-18T15:54:42.583 INFO:teuthology.orchestra.run.smithi196.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-18T15:54:42.583 INFO:teuthology.orchestra.run.smithi196.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-18T15:54:42.584 INFO:teuthology.orchestra.run.smithi196.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-18T15:54:42.584 INFO:teuthology.orchestra.run.smithi196.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-18T15:54:42.584 INFO:teuthology.orchestra.run.smithi196.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-18T15:54:42.584 INFO:teuthology.orchestra.run.smithi196.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-18T15:54:42.584 INFO:teuthology.orchestra.run.smithi196.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-18T15:54:42.584 INFO:teuthology.orchestra.run.smithi196.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-18T15:54:42.584 INFO:teuthology.orchestra.run.smithi196.stdout: librados2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.4 M 2024-03-18T15:54:42.584 INFO:teuthology.orchestra.run.smithi196.stdout: librbd1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-18T15:54:42.584 INFO:teuthology.orchestra.run.smithi196.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-18T15:54:42.584 INFO:teuthology.orchestra.run.smithi196.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-18T15:54:42.584 INFO:teuthology.orchestra.run.smithi196.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-18T15:54:42.584 INFO:teuthology.orchestra.run.smithi196.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-18T15:54:42.584 INFO:teuthology.orchestra.run.smithi196.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-18T15:54:42.585 INFO:teuthology.orchestra.run.smithi196.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-18T15:54:42.585 INFO:teuthology.orchestra.run.smithi196.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-18T15:54:42.585 INFO:teuthology.orchestra.run.smithi196.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-18T15:54:42.585 INFO:teuthology.orchestra.run.smithi196.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-18T15:54:42.585 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-03-18T15:54:42.585 INFO:teuthology.orchestra.run.smithi196.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-18T15:54:42.585 INFO:teuthology.orchestra.run.smithi196.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-18T15:54:42.585 INFO:teuthology.orchestra.run.smithi196.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-18T15:54:42.585 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 21 M 2024-03-18T15:54:42.585 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 34 k 2024-03-18T15:54:42.585 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mds x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 2.4 M 2024-03-18T15:54:42.585 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 248 k 2024-03-18T15:54:42.585 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mon x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 4.7 M 2024-03-18T15:54:42.585 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-osd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 17 M 2024-03-18T15:54:42.585 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 16 k 2024-03-18T15:54:42.586 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-selinux x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 25 k 2024-03-18T15:54:42.586 INFO:teuthology.orchestra.run.smithi196.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-18T15:54:42.586 INFO:teuthology.orchestra.run.smithi196.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-18T15:54:42.586 INFO:teuthology.orchestra.run.smithi196.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-18T15:54:42.586 INFO:teuthology.orchestra.run.smithi196.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-18T15:54:42.586 INFO:teuthology.orchestra.run.smithi196.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-18T15:54:42.586 INFO:teuthology.orchestra.run.smithi196.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-18T15:54:42.586 INFO:teuthology.orchestra.run.smithi196.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-18T15:54:42.586 INFO:teuthology.orchestra.run.smithi196.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-18T15:54:42.586 INFO:teuthology.orchestra.run.smithi196.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-18T15:54:42.586 INFO:teuthology.orchestra.run.smithi196.stdout: libcephsqlite x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 163 k 2024-03-18T15:54:42.586 INFO:teuthology.orchestra.run.smithi196.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-18T15:54:42.586 INFO:teuthology.orchestra.run.smithi196.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-18T15:54:42.587 INFO:teuthology.orchestra.run.smithi196.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-18T15:54:42.587 INFO:teuthology.orchestra.run.smithi196.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-18T15:54:42.587 INFO:teuthology.orchestra.run.smithi196.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-18T15:54:42.587 INFO:teuthology.orchestra.run.smithi196.stdout: libradosstriper1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 501 k 2024-03-18T15:54:42.587 INFO:teuthology.orchestra.run.smithi196.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-18T15:54:42.587 INFO:teuthology.orchestra.run.smithi196.stdout: librgw2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.1 M 2024-03-18T15:54:42.587 INFO:teuthology.orchestra.run.smithi196.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-18T15:54:42.587 INFO:teuthology.orchestra.run.smithi196.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-18T15:54:42.587 INFO:teuthology.orchestra.run.smithi196.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-18T15:54:42.587 INFO:teuthology.orchestra.run.smithi196.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-18T15:54:42.587 INFO:teuthology.orchestra.run.smithi196.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-18T15:54:42.587 INFO:teuthology.orchestra.run.smithi196.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-18T15:54:42.587 INFO:teuthology.orchestra.run.smithi196.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-18T15:54:42.588 INFO:teuthology.orchestra.run.smithi196.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-18T15:54:42.588 INFO:teuthology.orchestra.run.smithi196.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-18T15:54:42.588 INFO:teuthology.orchestra.run.smithi196.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-18T15:54:42.588 INFO:teuthology.orchestra.run.smithi196.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-18T15:54:42.588 INFO:teuthology.orchestra.run.smithi196.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-18T15:54:42.588 INFO:teuthology.orchestra.run.smithi196.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-18T15:54:42.588 INFO:teuthology.orchestra.run.smithi196.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-18T15:54:42.588 INFO:teuthology.orchestra.run.smithi196.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-18T15:54:42.588 INFO:teuthology.orchestra.run.smithi196.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-18T15:54:42.588 INFO:teuthology.orchestra.run.smithi196.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-18T15:54:42.588 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-18T15:54:42.588 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ceph-argparse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 45 k 2024-03-18T15:54:42.588 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 131 k 2024-03-18T15:54:42.588 INFO:teuthology.orchestra.run.smithi196.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-18T15:54:42.588 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-18T15:54:42.589 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-18T15:54:42.589 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-18T15:54:42.589 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-18T15:54:42.589 INFO:teuthology.orchestra.run.smithi196.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-18T15:54:42.589 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-18T15:54:42.589 INFO:teuthology.orchestra.run.smithi196.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-18T15:54:42.589 INFO:teuthology.orchestra.run.smithi196.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-18T15:54:42.589 INFO:teuthology.orchestra.run.smithi196.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-18T15:54:42.589 INFO:teuthology.orchestra.run.smithi196.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-18T15:54:42.589 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-18T15:54:42.589 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-18T15:54:42.589 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-18T15:54:42.589 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-18T15:54:42.589 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-18T15:54:42.589 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-18T15:54:42.590 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-18T15:54:42.590 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-18T15:54:42.590 INFO:teuthology.orchestra.run.smithi196.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-18T15:54:42.590 INFO:teuthology.orchestra.run.smithi196.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-18T15:54:42.590 INFO:teuthology.orchestra.run.smithi196.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-18T15:54:42.590 INFO:teuthology.orchestra.run.smithi196.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-18T15:54:42.590 INFO:teuthology.orchestra.run.smithi196.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-18T15:54:42.590 INFO:teuthology.orchestra.run.smithi196.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-18T15:54:42.590 INFO:teuthology.orchestra.run.smithi196.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-18T15:54:42.590 INFO:teuthology.orchestra.run.smithi196.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-18T15:54:42.590 INFO:teuthology.orchestra.run.smithi196.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-18T15:54:42.590 INFO:teuthology.orchestra.run.smithi196.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-18T15:54:42.590 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-18T15:54:42.590 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-18T15:54:42.591 INFO:teuthology.orchestra.run.smithi196.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-18T15:54:42.591 INFO:teuthology.orchestra.run.smithi196.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-18T15:54:42.591 INFO:teuthology.orchestra.run.smithi196.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-18T15:54:42.591 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-18T15:54:42.591 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-18T15:54:42.591 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-18T15:54:42.591 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-18T15:54:42.591 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-18T15:54:42.591 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-18T15:54:42.591 INFO:teuthology.orchestra.run.smithi196.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-18T15:54:42.591 INFO:teuthology.orchestra.run.smithi196.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-18T15:54:42.591 INFO:teuthology.orchestra.run.smithi196.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-18T15:54:42.591 INFO:teuthology.orchestra.run.smithi196.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-18T15:54:42.592 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-18T15:54:42.592 INFO:teuthology.orchestra.run.smithi196.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-18T15:54:42.592 INFO:teuthology.orchestra.run.smithi196.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-18T15:54:42.592 INFO:teuthology.orchestra.run.smithi196.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-18T15:54:42.592 INFO:teuthology.orchestra.run.smithi196.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-18T15:54:42.592 INFO:teuthology.orchestra.run.smithi196.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-18T15:54:42.592 INFO:teuthology.orchestra.run.smithi196.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-18T15:54:42.592 INFO:teuthology.orchestra.run.smithi196.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-18T15:54:42.592 INFO:teuthology.orchestra.run.smithi196.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-18T15:54:42.592 INFO:teuthology.orchestra.run.smithi196.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-18T15:54:42.592 INFO:teuthology.orchestra.run.smithi196.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-18T15:54:42.592 INFO:teuthology.orchestra.run.smithi196.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-18T15:54:42.592 INFO:teuthology.orchestra.run.smithi196.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-18T15:54:42.592 INFO:teuthology.orchestra.run.smithi196.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-18T15:54:42.592 INFO:teuthology.orchestra.run.smithi196.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-18T15:54:42.593 INFO:teuthology.orchestra.run.smithi196.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-18T15:54:42.593 INFO:teuthology.orchestra.run.smithi196.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-18T15:54:42.593 INFO:teuthology.orchestra.run.smithi196.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-18T15:54:42.593 INFO:teuthology.orchestra.run.smithi196.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-18T15:54:42.593 INFO:teuthology.orchestra.run.smithi196.stdout:Installing weak dependencies: 2024-03-18T15:54:42.593 INFO:teuthology.orchestra.run.smithi196.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-18T15:54:42.593 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-18T15:54:42.593 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-18T15:54:42.593 INFO:teuthology.orchestra.run.smithi196.stdout:============================================================================================ 2024-03-18T15:54:42.593 INFO:teuthology.orchestra.run.smithi196.stdout:Install 150 Packages 2024-03-18T15:54:42.593 INFO:teuthology.orchestra.run.smithi196.stdout:Upgrade 24 Packages 2024-03-18T15:54:42.593 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-18T15:54:42.594 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 224 M 2024-03-18T15:54:42.594 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-18T15:54:42.849 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-03-18T15:54:43.120 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-03-18T15:54:43.121 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-03-18T15:54:43.350 INFO:teuthology.orchestra.run.smithi196.stdout:(1/174): ceph-19.0.0-2162.g3d78f5ba.el9.x86_64. 34 kB/s | 6.6 kB 00:00 2024-03-18T15:54:43.758 INFO:teuthology.orchestra.run.smithi196.stdout:(2/174): ceph-base-19.0.0-2162.g3d78f5ba.el9.x8 8.9 MB/s | 5.4 MB 00:00 2024-03-18T15:54:43.783 INFO:teuthology.orchestra.run.smithi196.stdout:(3/174): ceph-fuse-19.0.0-2162.g3d78f5ba.el9.x8 2.0 MB/s | 890 kB 00:00 2024-03-18T15:54:43.808 INFO:teuthology.orchestra.run.smithi196.stdout:(4/174): ceph-immutable-object-cache-19.0.0-216 2.8 MB/s | 145 kB 00:00 2024-03-18T15:54:44.034 INFO:teuthology.orchestra.run.smithi196.stdout:(5/174): ceph-mgr-19.0.0-2162.g3d78f5ba.el9.x86 6.6 MB/s | 1.5 MB 00:00 2024-03-18T15:54:44.251 INFO:teuthology.orchestra.run.smithi196.stdout:(6/174): ceph-mds-19.0.0-2162.g3d78f5ba.el9.x86 5.1 MB/s | 2.4 MB 00:00 2024-03-18T15:54:44.827 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-03-18T15:54:44.828 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-03-18T15:54:44.834 INFO:teuthology.orchestra.run.smithi196.stdout:(7/174): ceph-common-19.0.0-2162.g3d78f5ba.el9. 12 MB/s | 21 MB 00:01 2024-03-18T15:54:45.101 INFO:teuthology.orchestra.run.smithi196.stdout:(8/174): ceph-mon-19.0.0-2162.g3d78f5ba.el9.x86 4.4 MB/s | 4.7 MB 00:01 2024-03-18T15:54:45.170 INFO:teuthology.orchestra.run.smithi039.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-18T15:54:45.172 INFO:teuthology.orchestra.run.smithi039.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-18T15:54:45.177 INFO:teuthology.orchestra.run.smithi196.stdout:(9/174): ceph-selinux-19.0.0-2162.g3d78f5ba.el9 335 kB/s | 25 kB 00:00 2024-03-18T15:54:45.196 INFO:teuthology.orchestra.run.smithi039.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-18T15:54:45.197 INFO:teuthology.orchestra.run.smithi039.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-18T15:54:45.199 INFO:teuthology.orchestra.run.smithi039.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-18T15:54:45.201 INFO:teuthology.orchestra.run.smithi039.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-18T15:54:45.203 INFO:teuthology.orchestra.run.smithi039.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-18T15:54:45.207 INFO:teuthology.orchestra.run.smithi039.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-18T15:54:45.209 INFO:teuthology.orchestra.run.smithi039.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-18T15:54:45.305 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2024-03-18T15:54:45.338 INFO:teuthology.orchestra.run.smithi039.stdout:============================================================================================ 2024-03-18T15:54:45.338 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repository Size 2024-03-18T15:54:45.338 INFO:teuthology.orchestra.run.smithi039.stdout:============================================================================================ 2024-03-18T15:54:45.338 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2024-03-18T15:54:45.338 INFO:teuthology.orchestra.run.smithi039.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-18T15:54:45.338 INFO:teuthology.orchestra.run.smithi039.stdout: ceph x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 6.6 k 2024-03-18T15:54:45.338 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-base x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.4 M 2024-03-18T15:54:45.338 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 890 k 2024-03-18T15:54:45.338 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 145 k 2024-03-18T15:54:45.339 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 1.5 M 2024-03-18T15:54:45.339 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 139 k 2024-03-18T15:54:45.339 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 1.8 M 2024-03-18T15:54:45.339 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 7.4 M 2024-03-18T15:54:45.339 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-rook noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 49 k 2024-03-18T15:54:45.339 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-radosgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 9.8 M 2024-03-18T15:54:45.339 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-test x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 48 M 2024-03-18T15:54:45.339 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-volume noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 264 k 2024-03-18T15:54:45.339 INFO:teuthology.orchestra.run.smithi039.stdout: cephadm noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 748 k 2024-03-18T15:54:45.339 INFO:teuthology.orchestra.run.smithi039.stdout: cephfs-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 221 k 2024-03-18T15:54:45.339 INFO:teuthology.orchestra.run.smithi039.stdout: cephfs-shell x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 27 k 2024-03-18T15:54:45.339 INFO:teuthology.orchestra.run.smithi039.stdout: cephfs-top noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 18 k 2024-03-18T15:54:45.339 INFO:teuthology.orchestra.run.smithi039.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-18T15:54:45.339 INFO:teuthology.orchestra.run.smithi039.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-18T15:54:45.340 INFO:teuthology.orchestra.run.smithi039.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-18T15:54:45.340 INFO:teuthology.orchestra.run.smithi039.stdout: libcephfs-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 32 k 2024-03-18T15:54:45.340 INFO:teuthology.orchestra.run.smithi039.stdout: libcephfs2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 747 k 2024-03-18T15:54:45.340 INFO:teuthology.orchestra.run.smithi039.stdout: librados-devel x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 127 k 2024-03-18T15:54:45.340 INFO:teuthology.orchestra.run.smithi039.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-18T15:54:45.340 INFO:teuthology.orchestra.run.smithi039.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-18T15:54:45.340 INFO:teuthology.orchestra.run.smithi039.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-18T15:54:45.340 INFO:teuthology.orchestra.run.smithi039.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-18T15:54:45.340 INFO:teuthology.orchestra.run.smithi039.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-18T15:54:45.340 INFO:teuthology.orchestra.run.smithi039.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-18T15:54:45.340 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cephfs x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 162 k 2024-03-18T15:54:45.340 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rados x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 320 k 2024-03-18T15:54:45.340 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 298 k 2024-03-18T15:54:45.340 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rgw x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 99 k 2024-03-18T15:54:45.341 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-fuse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 85 k 2024-03-18T15:54:45.341 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-mirror x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-18T15:54:45.341 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-nbd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 170 k 2024-03-18T15:54:45.341 INFO:teuthology.orchestra.run.smithi039.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-18T15:54:45.341 INFO:teuthology.orchestra.run.smithi039.stdout:Upgrading: 2024-03-18T15:54:45.341 INFO:teuthology.orchestra.run.smithi039.stdout: NetworkManager x86_64 1:1.46.0-1.el9 baseos 2.3 M 2024-03-18T15:54:45.341 INFO:teuthology.orchestra.run.smithi039.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 baseos 1.8 M 2024-03-18T15:54:45.341 INFO:teuthology.orchestra.run.smithi039.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 baseos 42 k 2024-03-18T15:54:45.341 INFO:teuthology.orchestra.run.smithi039.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 baseos 248 k 2024-03-18T15:54:45.341 INFO:teuthology.orchestra.run.smithi039.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-18T15:54:45.341 INFO:teuthology.orchestra.run.smithi039.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-18T15:54:45.341 INFO:teuthology.orchestra.run.smithi039.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-18T15:54:45.341 INFO:teuthology.orchestra.run.smithi039.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-18T15:54:45.341 INFO:teuthology.orchestra.run.smithi039.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-18T15:54:45.342 INFO:teuthology.orchestra.run.smithi039.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-18T15:54:45.342 INFO:teuthology.orchestra.run.smithi039.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-18T15:54:45.342 INFO:teuthology.orchestra.run.smithi039.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-18T15:54:45.342 INFO:teuthology.orchestra.run.smithi039.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-18T15:54:45.342 INFO:teuthology.orchestra.run.smithi039.stdout: librados2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.4 M 2024-03-18T15:54:45.342 INFO:teuthology.orchestra.run.smithi039.stdout: librbd1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 3.1 M 2024-03-18T15:54:45.342 INFO:teuthology.orchestra.run.smithi039.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-18T15:54:45.342 INFO:teuthology.orchestra.run.smithi039.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-18T15:54:45.342 INFO:teuthology.orchestra.run.smithi039.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-18T15:54:45.342 INFO:teuthology.orchestra.run.smithi039.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-18T15:54:45.342 INFO:teuthology.orchestra.run.smithi039.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-18T15:54:45.342 INFO:teuthology.orchestra.run.smithi039.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-18T15:54:45.342 INFO:teuthology.orchestra.run.smithi039.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-18T15:54:45.342 INFO:teuthology.orchestra.run.smithi039.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-18T15:54:45.343 INFO:teuthology.orchestra.run.smithi039.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-18T15:54:45.343 INFO:teuthology.orchestra.run.smithi039.stdout:Installing dependencies: 2024-03-18T15:54:45.343 INFO:teuthology.orchestra.run.smithi039.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-18T15:54:45.343 INFO:teuthology.orchestra.run.smithi039.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-18T15:54:45.343 INFO:teuthology.orchestra.run.smithi039.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-18T15:54:45.343 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 21 M 2024-03-18T15:54:45.343 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 34 k 2024-03-18T15:54:45.343 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mds x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 2.4 M 2024-03-18T15:54:45.343 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 248 k 2024-03-18T15:54:45.343 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mon x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 4.7 M 2024-03-18T15:54:45.343 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-osd x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 17 M 2024-03-18T15:54:45.343 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2162.g3d78f5ba.el9 ceph-noarch 16 k 2024-03-18T15:54:45.343 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-selinux x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 25 k 2024-03-18T15:54:45.343 INFO:teuthology.orchestra.run.smithi039.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-18T15:54:45.343 INFO:teuthology.orchestra.run.smithi039.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-18T15:54:45.344 INFO:teuthology.orchestra.run.smithi039.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-18T15:54:45.344 INFO:teuthology.orchestra.run.smithi039.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-18T15:54:45.344 INFO:teuthology.orchestra.run.smithi039.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-18T15:54:45.344 INFO:teuthology.orchestra.run.smithi039.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-18T15:54:45.344 INFO:teuthology.orchestra.run.smithi039.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-18T15:54:45.344 INFO:teuthology.orchestra.run.smithi039.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-18T15:54:45.344 INFO:teuthology.orchestra.run.smithi039.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-18T15:54:45.344 INFO:teuthology.orchestra.run.smithi039.stdout: libcephsqlite x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 163 k 2024-03-18T15:54:45.344 INFO:teuthology.orchestra.run.smithi039.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-18T15:54:45.344 INFO:teuthology.orchestra.run.smithi039.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-18T15:54:45.344 INFO:teuthology.orchestra.run.smithi039.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-18T15:54:45.344 INFO:teuthology.orchestra.run.smithi039.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-18T15:54:45.344 INFO:teuthology.orchestra.run.smithi039.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-18T15:54:45.344 INFO:teuthology.orchestra.run.smithi039.stdout: libradosstriper1 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 501 k 2024-03-18T15:54:45.345 INFO:teuthology.orchestra.run.smithi039.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-18T15:54:45.345 INFO:teuthology.orchestra.run.smithi039.stdout: librgw2 x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 5.1 M 2024-03-18T15:54:45.345 INFO:teuthology.orchestra.run.smithi039.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-18T15:54:45.345 INFO:teuthology.orchestra.run.smithi039.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-18T15:54:45.345 INFO:teuthology.orchestra.run.smithi039.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-18T15:54:45.345 INFO:teuthology.orchestra.run.smithi039.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-18T15:54:45.345 INFO:teuthology.orchestra.run.smithi039.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-18T15:54:45.345 INFO:teuthology.orchestra.run.smithi039.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-18T15:54:45.345 INFO:teuthology.orchestra.run.smithi039.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-18T15:54:45.345 INFO:teuthology.orchestra.run.smithi039.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-18T15:54:45.345 INFO:teuthology.orchestra.run.smithi039.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-18T15:54:45.345 INFO:teuthology.orchestra.run.smithi039.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-18T15:54:45.345 INFO:teuthology.orchestra.run.smithi039.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-18T15:54:45.345 INFO:teuthology.orchestra.run.smithi039.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-18T15:54:45.345 INFO:teuthology.orchestra.run.smithi039.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-18T15:54:45.346 INFO:teuthology.orchestra.run.smithi039.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-18T15:54:45.346 INFO:teuthology.orchestra.run.smithi039.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-18T15:54:45.346 INFO:teuthology.orchestra.run.smithi039.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-18T15:54:45.346 INFO:teuthology.orchestra.run.smithi039.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-18T15:54:45.346 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-18T15:54:45.346 INFO:teuthology.orchestra.run.smithi039.stdout: python3-ceph-argparse x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 45 k 2024-03-18T15:54:45.346 INFO:teuthology.orchestra.run.smithi039.stdout: python3-ceph-common x86_64 2:19.0.0-2162.g3d78f5ba.el9 ceph 131 k 2024-03-18T15:54:45.346 INFO:teuthology.orchestra.run.smithi039.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-18T15:54:45.346 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-18T15:54:45.346 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-18T15:54:45.346 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-18T15:54:45.346 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-18T15:54:45.346 INFO:teuthology.orchestra.run.smithi039.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-18T15:54:45.346 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-18T15:54:45.346 INFO:teuthology.orchestra.run.smithi039.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-18T15:54:45.347 INFO:teuthology.orchestra.run.smithi039.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-18T15:54:45.347 INFO:teuthology.orchestra.run.smithi039.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-18T15:54:45.347 INFO:teuthology.orchestra.run.smithi039.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-18T15:54:45.347 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-18T15:54:45.347 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-18T15:54:45.347 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-18T15:54:45.347 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-18T15:54:45.347 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-18T15:54:45.347 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-18T15:54:45.347 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-18T15:54:45.347 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-18T15:54:45.347 INFO:teuthology.orchestra.run.smithi039.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-18T15:54:45.347 INFO:teuthology.orchestra.run.smithi039.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-18T15:54:45.347 INFO:teuthology.orchestra.run.smithi039.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-18T15:54:45.348 INFO:teuthology.orchestra.run.smithi039.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-18T15:54:45.348 INFO:teuthology.orchestra.run.smithi039.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-18T15:54:45.348 INFO:teuthology.orchestra.run.smithi039.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-18T15:54:45.348 INFO:teuthology.orchestra.run.smithi039.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-18T15:54:45.348 INFO:teuthology.orchestra.run.smithi039.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-18T15:54:45.348 INFO:teuthology.orchestra.run.smithi039.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-18T15:54:45.348 INFO:teuthology.orchestra.run.smithi039.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-18T15:54:45.348 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-18T15:54:45.348 INFO:teuthology.orchestra.run.smithi039.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-18T15:54:45.348 INFO:teuthology.orchestra.run.smithi039.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-18T15:54:45.348 INFO:teuthology.orchestra.run.smithi039.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-18T15:54:45.348 INFO:teuthology.orchestra.run.smithi039.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-18T15:54:45.348 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-18T15:54:45.348 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-18T15:54:45.348 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-18T15:54:45.349 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-18T15:54:45.349 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-18T15:54:45.349 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-18T15:54:45.349 INFO:teuthology.orchestra.run.smithi039.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-18T15:54:45.349 INFO:teuthology.orchestra.run.smithi039.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-18T15:54:45.349 INFO:teuthology.orchestra.run.smithi039.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-18T15:54:45.349 INFO:teuthology.orchestra.run.smithi039.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-18T15:54:45.349 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-18T15:54:45.349 INFO:teuthology.orchestra.run.smithi039.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-18T15:54:45.349 INFO:teuthology.orchestra.run.smithi039.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-18T15:54:45.349 INFO:teuthology.orchestra.run.smithi039.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-18T15:54:45.349 INFO:teuthology.orchestra.run.smithi039.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-18T15:54:45.349 INFO:teuthology.orchestra.run.smithi039.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-18T15:54:45.349 INFO:teuthology.orchestra.run.smithi039.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-18T15:54:45.349 INFO:teuthology.orchestra.run.smithi039.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-18T15:54:45.350 INFO:teuthology.orchestra.run.smithi039.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-18T15:54:45.350 INFO:teuthology.orchestra.run.smithi039.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-18T15:54:45.350 INFO:teuthology.orchestra.run.smithi039.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-18T15:54:45.350 INFO:teuthology.orchestra.run.smithi039.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-18T15:54:45.350 INFO:teuthology.orchestra.run.smithi039.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-18T15:54:45.350 INFO:teuthology.orchestra.run.smithi039.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-18T15:54:45.350 INFO:teuthology.orchestra.run.smithi039.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-18T15:54:45.350 INFO:teuthology.orchestra.run.smithi039.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-18T15:54:45.350 INFO:teuthology.orchestra.run.smithi039.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-18T15:54:45.350 INFO:teuthology.orchestra.run.smithi039.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-18T15:54:45.350 INFO:teuthology.orchestra.run.smithi039.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-18T15:54:45.350 INFO:teuthology.orchestra.run.smithi039.stdout:Installing weak dependencies: 2024-03-18T15:54:45.350 INFO:teuthology.orchestra.run.smithi039.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-18T15:54:45.350 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T15:54:45.351 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2024-03-18T15:54:45.351 INFO:teuthology.orchestra.run.smithi039.stdout:============================================================================================ 2024-03-18T15:54:45.351 INFO:teuthology.orchestra.run.smithi039.stdout:Install 150 Packages 2024-03-18T15:54:45.351 INFO:teuthology.orchestra.run.smithi039.stdout:Upgrade 24 Packages 2024-03-18T15:54:45.351 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T15:54:45.351 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 224 M 2024-03-18T15:54:45.351 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2024-03-18T15:54:45.752 INFO:teuthology.orchestra.run.smithi196.stdout:(10/174): ceph-radosgw-19.0.0-2162.g3d78f5ba.el 11 MB/s | 9.8 MB 00:00 2024-03-18T15:54:45.944 INFO:teuthology.orchestra.run.smithi196.stdout:(11/174): ceph-osd-19.0.0-2162.g3d78f5ba.el9.x8 9.9 MB/s | 17 MB 00:01 2024-03-18T15:54:45.969 INFO:teuthology.orchestra.run.smithi196.stdout:(12/174): cephfs-mirror-19.0.0-2162.g3d78f5ba.e 1.0 MB/s | 221 kB 00:00 2024-03-18T15:54:45.986 INFO:teuthology.orchestra.run.smithi196.stdout:(13/174): cephfs-shell-19.0.0-2162.g3d78f5ba.el 651 kB/s | 27 kB 00:00 2024-03-18T15:54:46.011 INFO:teuthology.orchestra.run.smithi196.stdout:(14/174): libcephfs-devel-19.0.0-2162.g3d78f5ba 758 kB/s | 32 kB 00:00 2024-03-18T15:54:46.061 INFO:teuthology.orchestra.run.smithi196.stdout:(15/174): libcephfs2-19.0.0-2162.g3d78f5ba.el9. 9.7 MB/s | 747 kB 00:00 2024-03-18T15:54:46.087 INFO:teuthology.orchestra.run.smithi196.stdout:(16/174): libcephsqlite-19.0.0-2162.g3d78f5ba.e 2.1 MB/s | 163 kB 00:00 2024-03-18T15:54:46.120 INFO:teuthology.orchestra.run.smithi196.stdout:(17/174): librados-devel-19.0.0-2162.g3d78f5ba. 2.1 MB/s | 127 kB 00:00 2024-03-18T15:54:46.154 INFO:teuthology.orchestra.run.smithi196.stdout:(18/174): libradosstriper1-19.0.0-2162.g3d78f5b 7.3 MB/s | 501 kB 00:00 2024-03-18T15:54:46.254 INFO:teuthology.orchestra.run.smithi196.stdout:(19/174): python3-ceph-argparse-19.0.0-2162.g3d 452 kB/s | 45 kB 00:00 2024-03-18T15:54:46.293 INFO:teuthology.orchestra.run.smithi039.stdout:(1/174): ceph-19.0.0-2162.g3d78f5ba.el9.x86_64. 21 kB/s | 6.6 kB 00:00 2024-03-18T15:54:46.338 INFO:teuthology.orchestra.run.smithi196.stdout:(20/174): python3-ceph-common-19.0.0-2162.g3d78 1.5 MB/s | 131 kB 00:00 2024-03-18T15:54:46.471 INFO:teuthology.orchestra.run.smithi196.stdout:(21/174): librgw2-19.0.0-2162.g3d78f5ba.el9.x86 15 MB/s | 5.1 MB 00:00 2024-03-18T15:54:46.496 INFO:teuthology.orchestra.run.smithi196.stdout:(22/174): python3-cephfs-19.0.0-2162.g3d78f5ba. 1.0 MB/s | 162 kB 00:00 2024-03-18T15:54:46.555 INFO:teuthology.orchestra.run.smithi196.stdout:(23/174): python3-rados-19.0.0-2162.g3d78f5ba.e 3.8 MB/s | 320 kB 00:00 2024-03-18T15:54:46.597 INFO:teuthology.orchestra.run.smithi196.stdout:(24/174): python3-rbd-19.0.0-2162.g3d78f5ba.el9 2.9 MB/s | 298 kB 00:00 2024-03-18T15:54:46.618 INFO:teuthology.orchestra.run.smithi039.stdout:(2/174): ceph-fuse-19.0.0-2162.g3d78f5ba.el9.x8 2.7 MB/s | 890 kB 00:00 2024-03-18T15:54:46.647 INFO:teuthology.orchestra.run.smithi196.stdout:(25/174): python3-rgw-19.0.0-2162.g3d78f5ba.el9 1.1 MB/s | 99 kB 00:00 2024-03-18T15:54:46.672 INFO:teuthology.orchestra.run.smithi196.stdout:(26/174): rbd-fuse-19.0.0-2162.g3d78f5ba.el9.x8 1.1 MB/s | 85 kB 00:00 2024-03-18T15:54:46.702 INFO:teuthology.orchestra.run.smithi039.stdout:(3/174): ceph-immutable-object-cache-19.0.0-216 1.7 MB/s | 145 kB 00:00 2024-03-18T15:54:46.956 INFO:teuthology.orchestra.run.smithi196.stdout:(27/174): rbd-mirror-19.0.0-2162.g3d78f5ba.el9. 10 MB/s | 3.1 MB 00:00 2024-03-18T15:54:47.015 INFO:teuthology.orchestra.run.smithi196.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2162.g 584 kB/s | 34 kB 00:00 2024-03-18T15:54:47.069 INFO:teuthology.orchestra.run.smithi039.stdout:(4/174): ceph-base-19.0.0-2162.g3d78f5ba.el9.x8 4.9 MB/s | 5.4 MB 00:01 2024-03-18T15:54:47.073 INFO:teuthology.orchestra.run.smithi196.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2162.g3d78f5b 2.3 MB/s | 139 kB 00:00 2024-03-18T15:54:47.274 INFO:teuthology.orchestra.run.smithi196.stdout:(30/174): ceph-mgr-dashboard-19.0.0-2162.g3d78f 9.0 MB/s | 1.8 MB 00:00 2024-03-18T15:54:47.299 INFO:teuthology.orchestra.run.smithi196.stdout:(31/174): rbd-nbd-19.0.0-2162.g3d78f5ba.el9.x86 272 kB/s | 170 kB 00:00 2024-03-18T15:54:47.303 INFO:teuthology.orchestra.run.smithi039.stdout:(5/174): ceph-mds-19.0.0-2162.g3d78f5ba.el9.x86 3.9 MB/s | 2.4 MB 00:00 2024-03-18T15:54:47.353 INFO:teuthology.orchestra.run.smithi039.stdout:(6/174): ceph-mgr-19.0.0-2162.g3d78f5ba.el9.x86 5.3 MB/s | 1.5 MB 00:00 2024-03-18T15:54:47.383 INFO:teuthology.orchestra.run.smithi196.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2162.g3d 2.9 MB/s | 248 kB 00:00 2024-03-18T15:54:47.783 INFO:teuthology.orchestra.run.smithi196.stdout:(33/174): ceph-mgr-rook-19.0.0-2162.g3d78f5ba.e 123 kB/s | 49 kB 00:00 2024-03-18T15:54:47.866 INFO:teuthology.orchestra.run.smithi196.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2162.g3 198 kB/s | 16 kB 00:00 2024-03-18T15:54:47.975 INFO:teuthology.orchestra.run.smithi196.stdout:(35/174): ceph-mgr-diskprediction-local-19.0.0- 11 MB/s | 7.4 MB 00:00 2024-03-18T15:54:48.000 INFO:teuthology.orchestra.run.smithi196.stdout:(36/174): ceph-volume-19.0.0-2162.g3d78f5ba.el9 1.9 MB/s | 264 kB 00:00 2024-03-18T15:54:48.100 INFO:teuthology.orchestra.run.smithi196.stdout:(37/174): cephfs-top-19.0.0-2162.g3d78f5ba.el9. 182 kB/s | 18 kB 00:00 2024-03-18T15:54:48.134 INFO:teuthology.orchestra.run.smithi196.stdout:(38/174): cephadm-19.0.0-2162.g3d78f5ba.el9.noa 4.6 MB/s | 748 kB 00:00 2024-03-18T15:54:48.186 INFO:teuthology.orchestra.run.smithi039.stdout:(7/174): ceph-mon-19.0.0-2162.g3d78f5ba.el9.x86 5.3 MB/s | 4.7 MB 00:00 2024-03-18T15:54:48.360 INFO:teuthology.orchestra.run.smithi196.stdout:(39/174): automake-1.16.2-8.el9.noarch.rpm 2.5 MB/s | 662 kB 00:00 2024-03-18T15:54:48.402 INFO:teuthology.orchestra.run.smithi196.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 3.4 MB/s | 920 kB 00:00 2024-03-18T15:54:48.427 INFO:teuthology.orchestra.run.smithi196.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 1.6 MB/s | 106 kB 00:00 2024-03-18T15:54:48.469 INFO:teuthology.orchestra.run.smithi196.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 4.5 MB/s | 307 kB 00:00 2024-03-18T15:54:48.485 INFO:teuthology.orchestra.run.smithi196.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 542 kB/s | 32 kB 00:00 2024-03-18T15:54:48.561 INFO:teuthology.orchestra.run.smithi196.stdout:(44/174): flexiblas-openblas-openmp-3.0.4-8.el9 228 kB/s | 17 kB 00:00 2024-03-18T15:54:48.645 INFO:teuthology.orchestra.run.smithi196.stdout:(45/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 17 MB/s | 3.0 MB 00:00 2024-03-18T15:54:48.673 INFO:teuthology.orchestra.run.smithi196.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 91 kB/s | 10 kB 00:00 2024-03-18T15:54:48.698 INFO:teuthology.orchestra.run.smithi196.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.9 MB/s | 159 kB 00:00 2024-03-18T15:54:48.724 INFO:teuthology.orchestra.run.smithi196.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 887 kB/s | 44 kB 00:00 2024-03-18T15:54:48.758 INFO:teuthology.orchestra.run.smithi196.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-18T15:54:48.791 INFO:teuthology.orchestra.run.smithi196.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 8.6 MB/s | 578 kB 00:00 2024-03-18T15:54:48.817 INFO:teuthology.orchestra.run.smithi196.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 804 kB/s | 46 kB 00:00 2024-03-18T15:54:48.850 INFO:teuthology.orchestra.run.smithi196.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.7 MB/s | 282 kB 00:00 2024-03-18T15:54:48.875 INFO:teuthology.orchestra.run.smithi196.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 3.1 MB/s | 187 kB 00:00 2024-03-18T15:54:48.909 INFO:teuthology.orchestra.run.smithi196.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.7 MB/s | 217 kB 00:00 2024-03-18T15:54:48.926 INFO:teuthology.orchestra.run.smithi196.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 680 kB/s | 34 kB 00:00 2024-03-18T15:54:49.135 INFO:teuthology.orchestra.run.smithi196.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 21 MB/s | 4.7 MB 00:00 2024-03-18T15:54:49.168 INFO:teuthology.orchestra.run.smithi196.stdout:(57/174): postgresql-13.11-1.el9.x86_64.rpm 6.3 MB/s | 1.5 MB 00:00 2024-03-18T15:54:49.202 INFO:teuthology.orchestra.run.smithi196.stdout:(58/174): postgresql-contrib-13.11-1.el9.x86_64 12 MB/s | 811 kB 00:00 2024-03-18T15:54:49.227 INFO:teuthology.orchestra.run.smithi196.stdout:(59/174): postgresql-private-libs-13.11-1.el9.x 2.3 MB/s | 134 kB 00:00 2024-03-18T15:54:49.337 INFO:teuthology.orchestra.run.smithi039.stdout:(8/174): ceph-osd-19.0.0-2162.g3d78f5ba.el9.x86 8.5 MB/s | 17 MB 00:01 2024-03-18T15:54:49.386 INFO:teuthology.orchestra.run.smithi196.stdout:(60/174): postgresql-server-13.11-1.el9.x86_64. 31 MB/s | 5.7 MB 00:00 2024-03-18T15:54:49.419 INFO:teuthology.orchestra.run.smithi196.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 5.3 MB/s | 1.0 MB 00:00 2024-03-18T15:54:49.453 INFO:teuthology.orchestra.run.smithi196.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.2 MB/s | 78 kB 00:00 2024-03-18T15:54:49.495 INFO:teuthology.orchestra.run.smithi039.stdout:(9/174): ceph-radosgw-19.0.0-2162.g3d78f5ba.el9 7.5 MB/s | 9.8 MB 00:01 2024-03-18T15:54:49.521 INFO:teuthology.orchestra.run.smithi039.stdout:(10/174): ceph-selinux-19.0.0-2162.g3d78f5ba.el 138 kB/s | 25 kB 00:00 2024-03-18T15:54:49.579 INFO:teuthology.orchestra.run.smithi039.stdout:(11/174): cephfs-mirror-19.0.0-2162.g3d78f5ba.e 3.8 MB/s | 221 kB 00:00 2024-03-18T15:54:49.638 INFO:teuthology.orchestra.run.smithi039.stdout:(12/174): cephfs-shell-19.0.0-2162.g3d78f5ba.el 472 kB/s | 27 kB 00:00 2024-03-18T15:54:49.645 INFO:teuthology.orchestra.run.smithi196.stdout:(63/174): python3-babel-2.9.1-2.el9.noarch.rpm 26 MB/s | 5.8 MB 00:00 2024-03-18T15:54:49.662 INFO:teuthology.orchestra.run.smithi196.stdout:(64/174): python3-jsonpatch-1.21-16.el9.noarch. 117 kB/s | 24 kB 00:00 2024-03-18T15:54:49.687 INFO:teuthology.orchestra.run.smithi196.stdout:(65/174): python3-jsonpointer-2.0-4.el9.noarch. 435 kB/s | 18 kB 00:00 2024-03-18T15:54:49.696 INFO:teuthology.orchestra.run.smithi039.stdout:(13/174): libcephfs-devel-19.0.0-2162.g3d78f5ba 552 kB/s | 32 kB 00:00 2024-03-18T15:54:49.712 INFO:teuthology.orchestra.run.smithi196.stdout:(66/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-03-18T15:54:49.729 INFO:teuthology.orchestra.run.smithi196.stdout:(67/174): python3-markupsafe-1.1.1-12.el9.x86_6 756 kB/s | 32 kB 00:00 2024-03-18T15:54:49.822 INFO:teuthology.orchestra.run.smithi039.stdout:(14/174): libcephfs2-19.0.0-2162.g3d78f5ba.el9. 5.9 MB/s | 747 kB 00:00 2024-03-18T15:54:49.880 INFO:teuthology.orchestra.run.smithi039.stdout:(15/174): libcephsqlite-19.0.0-2162.g3d78f5ba.e 2.8 MB/s | 163 kB 00:00 2024-03-18T15:54:49.930 INFO:teuthology.orchestra.run.smithi196.stdout:(68/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 22 MB/s | 4.8 MB 00:00 2024-03-18T15:54:49.955 INFO:teuthology.orchestra.run.smithi196.stdout:(69/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 964 kB/s | 217 kB 00:00 2024-03-18T15:54:49.956 INFO:teuthology.orchestra.run.smithi039.stdout:(16/174): librados-devel-19.0.0-2162.g3d78f5ba. 1.7 MB/s | 127 kB 00:00 2024-03-18T15:54:49.980 INFO:teuthology.orchestra.run.smithi196.stdout:(70/174): python3-oauthlib-3.1.1-5.el9.noarch.r 3.4 MB/s | 174 kB 00:00 2024-03-18T15:54:50.005 INFO:teuthology.orchestra.run.smithi196.stdout:(71/174): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 69 kB 00:00 2024-03-18T15:54:50.031 INFO:teuthology.orchestra.run.smithi196.stdout:(72/174): python3-prettytable-0.7.2-27.el9.noar 822 kB/s | 41 kB 00:00 2024-03-18T15:54:50.056 INFO:teuthology.orchestra.run.smithi196.stdout:(73/174): python3-protobuf-3.14.0-13.el9.noarch 4.6 MB/s | 238 kB 00:00 2024-03-18T15:54:50.073 INFO:teuthology.orchestra.run.smithi039.stdout:(17/174): libradosstriper1-19.0.0-2162.g3d78f5b 4.2 MB/s | 501 kB 00:00 2024-03-18T15:54:50.082 INFO:teuthology.orchestra.run.smithi196.stdout:(74/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 132 kB 00:00 2024-03-18T15:54:50.107 INFO:teuthology.orchestra.run.smithi196.stdout:(75/174): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2024-03-18T15:54:50.132 INFO:teuthology.orchestra.run.smithi196.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 949 kB/s | 47 kB 00:00 2024-03-18T15:54:50.157 INFO:teuthology.orchestra.run.smithi196.stdout:(77/174): python3-requests-oauthlib-1.3.0-12.el 856 kB/s | 43 kB 00:00 2024-03-18T15:54:50.391 INFO:teuthology.orchestra.run.smithi196.stdout:(78/174): python3-toml-0.10.2-6.el9.noarch.rpm 166 kB/s | 39 kB 00:00 2024-03-18T15:54:50.666 INFO:teuthology.orchestra.run.smithi196.stdout:(79/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 29 MB/s | 16 MB 00:00 2024-03-18T15:54:50.681 INFO:teuthology.orchestra.run.smithi039.stdout:(18/174): librgw2-19.0.0-2162.g3d78f5ba.el9.x86 8.5 MB/s | 5.1 MB 00:00 2024-03-18T15:54:50.691 INFO:teuthology.orchestra.run.smithi196.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 137 kB/s | 41 kB 00:00 2024-03-18T15:54:50.716 INFO:teuthology.orchestra.run.smithi196.stdout:(81/174): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-18T15:54:50.740 INFO:teuthology.orchestra.run.smithi039.stdout:(19/174): python3-ceph-argparse-19.0.0-2162.g3d 791 kB/s | 45 kB 00:00 2024-03-18T15:54:50.742 INFO:teuthology.orchestra.run.smithi196.stdout:(82/174): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 56 kB 00:00 2024-03-18T15:54:50.767 INFO:teuthology.orchestra.run.smithi196.stdout:(83/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-18T15:54:50.790 INFO:teuthology.orchestra.run.smithi039.stdout:(20/174): python3-ceph-common-19.0.0-2162.g3d78 2.6 MB/s | 131 kB 00:00 2024-03-18T15:54:50.792 INFO:teuthology.orchestra.run.smithi196.stdout:(84/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 878 kB/s | 44 kB 00:00 2024-03-18T15:54:50.817 INFO:teuthology.orchestra.run.smithi196.stdout:(85/174): fuse-2.9.9-15.el9.x86_64.rpm 1.5 MB/s | 79 kB 00:00 2024-03-18T15:54:50.843 INFO:teuthology.orchestra.run.smithi196.stdout:(86/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-18T15:54:50.879 INFO:teuthology.orchestra.run.smithi039.stdout:(21/174): python3-cephfs-19.0.0-2162.g3d78f5ba. 1.8 MB/s | 162 kB 00:00 2024-03-18T15:54:50.893 INFO:teuthology.orchestra.run.smithi196.stdout:(87/174): python3-cryptography-36.0.1-4.el9.x86 15 MB/s | 1.1 MB 00:00 2024-03-18T15:54:50.918 INFO:teuthology.orchestra.run.smithi196.stdout:(88/174): python3-ply-3.11-14.el9.noarch.rpm 1.3 MB/s | 103 kB 00:00 2024-03-18T15:54:50.941 INFO:teuthology.orchestra.run.smithi039.stdout:(22/174): python3-rados-19.0.0-2162.g3d78f5ba.e 5.2 MB/s | 320 kB 00:00 2024-03-18T15:54:50.944 INFO:teuthology.orchestra.run.smithi196.stdout:(89/174): python3-pycparser-2.20-6.el9.noarch.r 2.4 MB/s | 124 kB 00:00 2024-03-18T15:54:51.008 INFO:teuthology.orchestra.run.smithi039.stdout:(23/174): python3-rbd-19.0.0-2162.g3d78f5ba.el9 4.4 MB/s | 298 kB 00:00 2024-03-18T15:54:51.034 INFO:teuthology.orchestra.run.smithi196.stdout:(90/174): python3-urllib3-1.26.5-5.el9.noarch.r 1.6 MB/s | 187 kB 00:00 2024-03-18T15:54:51.075 INFO:teuthology.orchestra.run.smithi039.stdout:(24/174): python3-rgw-19.0.0-2162.g3d78f5ba.el9 1.5 MB/s | 99 kB 00:00 2024-03-18T15:54:51.136 INFO:teuthology.orchestra.run.smithi196.stdout:(91/174): jq-1.6-16.el9.x86_64.rpm 977 kB/s | 187 kB 00:00 2024-03-18T15:54:51.142 INFO:teuthology.orchestra.run.smithi039.stdout:(25/174): rbd-fuse-19.0.0-2162.g3d78f5ba.el9.x8 1.3 MB/s | 85 kB 00:00 2024-03-18T15:54:51.186 INFO:teuthology.orchestra.run.smithi196.stdout:(92/174): libquadmath-11.4.1-3.el9.x86_64.rpm 3.8 MB/s | 195 kB 00:00 2024-03-18T15:54:51.236 INFO:teuthology.orchestra.run.smithi196.stdout:(93/174): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-03-18T15:54:51.270 INFO:teuthology.orchestra.run.smithi196.stdout:(94/174): libgfortran-11.4.1-3.el9.x86_64.rpm 3.3 MB/s | 803 kB 00:00 2024-03-18T15:54:51.543 INFO:teuthology.orchestra.run.smithi039.stdout:(26/174): rbd-mirror-19.0.0-2162.g3d78f5ba.el9. 7.7 MB/s | 3.1 MB 00:00 2024-03-18T15:54:51.628 INFO:teuthology.orchestra.run.smithi039.stdout:(27/174): rbd-nbd-19.0.0-2162.g3d78f5ba.el9.x86 2.0 MB/s | 170 kB 00:00 2024-03-18T15:54:51.664 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-03-18T15:54:51.685 INFO:teuthology.orchestra.run.smithi039.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2162.g 610 kB/s | 34 kB 00:00 2024-03-18T15:54:51.743 INFO:teuthology.orchestra.run.smithi039.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2162.g3d78f5b 2.4 MB/s | 139 kB 00:00 2024-03-18T15:54:51.781 INFO:teuthology.orchestra.run.smithi194.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-18T15:54:51.824 INFO:teuthology.orchestra.run.smithi194.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-18T15:54:51.855 INFO:teuthology.orchestra.run.smithi194.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-18T15:54:51.864 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-18T15:54:51.904 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-18T15:54:51.937 INFO:teuthology.orchestra.run.smithi196.stdout:(95/174): ceph-test-19.0.0-2162.g3d78f5ba.el9.x 7.0 MB/s | 48 MB 00:06 2024-03-18T15:54:51.944 INFO:teuthology.orchestra.run.smithi039.stdout:(30/174): ceph-mgr-dashboard-19.0.0-2162.g3d78f 9.0 MB/s | 1.8 MB 00:00 2024-03-18T15:54:51.957 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-18T15:54:51.962 INFO:teuthology.orchestra.run.smithi196.stdout:(96/174): elfutils-libelf-devel-0.190-2.el9.x86 30 kB/s | 21 kB 00:00 2024-03-18T15:54:51.991 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-18T15:54:51.996 INFO:teuthology.orchestra.run.smithi196.stdout:(97/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 35 kB/s | 25 kB 00:00 2024-03-18T15:54:52.016 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-18T15:54:52.058 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-18T15:54:52.421 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-18T15:54:52.451 INFO:teuthology.orchestra.run.smithi194.stdout: Upgrading : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-18T15:54:52.497 INFO:teuthology.orchestra.run.smithi196.stdout:(98/174): python3-devel-3.9.17-2.el9.x86_64.rpm 491 kB/s | 245 kB 00:00 2024-03-18T15:54:52.524 INFO:teuthology.orchestra.run.smithi196.stdout:(99/174): python3-jinja2-2.11.3-5.el9.noarch.rp 470 kB/s | 248 kB 00:00 2024-03-18T15:54:52.572 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-18T15:54:52.610 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-18T15:54:52.636 INFO:teuthology.orchestra.run.smithi039.stdout:(31/174): ceph-mgr-diskprediction-local-19.0.0- 11 MB/s | 7.4 MB 00:00 2024-03-18T15:54:52.694 INFO:teuthology.orchestra.run.smithi039.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2162.g3d 4.3 MB/s | 248 kB 00:00 2024-03-18T15:54:52.701 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-18T15:54:52.731 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/198 2024-03-18T15:54:52.753 INFO:teuthology.orchestra.run.smithi039.stdout:(33/174): ceph-mgr-rook-19.0.0-2162.g3d78f5ba.e 857 kB/s | 49 kB 00:00 2024-03-18T15:54:52.769 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-18T15:54:52.799 INFO:teuthology.orchestra.run.smithi196.stdout:(100/174): qatlib-service-23.11.0-1.el9.x86_64. 127 kB/s | 35 kB 00:00 2024-03-18T15:54:52.803 INFO:teuthology.orchestra.run.smithi039.stdout:(34/174): ceph-prometheus-alerts-19.0.0-2162.g3 335 kB/s | 16 kB 00:00 2024-03-18T15:54:52.870 INFO:teuthology.orchestra.run.smithi039.stdout:(35/174): ceph-volume-19.0.0-2162.g3d78f5ba.el9 3.9 MB/s | 264 kB 00:00 2024-03-18T15:54:52.945 INFO:teuthology.orchestra.run.smithi194.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-18T15:54:52.979 INFO:teuthology.orchestra.run.smithi039.stdout:(36/174): cephadm-19.0.0-2162.g3d78f5ba.el9.noa 6.8 MB/s | 748 kB 00:00 2024-03-18T15:54:52.991 INFO:teuthology.orchestra.run.smithi196.stdout:(101/174): qatlib-23.11.0-1.el9.x86_64.rpm 438 kB/s | 216 kB 00:00 2024-03-18T15:54:52.995 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-18T15:54:53.023 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-18T15:54:53.037 INFO:teuthology.orchestra.run.smithi039.stdout:(37/174): cephfs-top-19.0.0-2162.g3d78f5ba.el9. 318 kB/s | 18 kB 00:00 2024-03-18T15:54:53.110 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-18T15:54:53.132 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 17/198 2024-03-18T15:54:53.141 INFO:teuthology.orchestra.run.smithi196.stdout:(102/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 130 kB/s | 44 kB 00:00 2024-03-18T15:54:53.196 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 17/198 2024-03-18T15:54:53.233 INFO:teuthology.orchestra.run.smithi196.stdout:(103/174): lua-devel-5.4.4-4.el9.x86_64.rpm 243 kB/s | 22 kB 00:00 2024-03-18T15:54:53.236 INFO:teuthology.orchestra.run.smithi194.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-18T15:54:53.259 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-18T15:54:53.275 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-18T15:54:53.392 INFO:teuthology.orchestra.run.smithi196.stdout:(104/174): protobuf-compiler-3.14.0-13.el9.x86_ 5.3 MB/s | 863 kB 00:00 2024-03-18T15:54:53.417 INFO:teuthology.orchestra.run.smithi196.stdout:(105/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 220 kB/s | 94 kB 00:00 2024-03-18T15:54:53.467 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-18T15:54:53.498 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-18T15:54:53.517 INFO:teuthology.orchestra.run.smithi196.stdout:(106/174): gperftools-libs-2.9.1-2.el9.x86_64.r 3.0 MB/s | 309 kB 00:00 2024-03-18T15:54:53.530 INFO:teuthology.orchestra.run.smithi194.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-18T15:54:53.560 INFO:teuthology.orchestra.run.smithi196.stdout:(107/174): abseil-cpp-20211102.0-3.el9.x86_64.r 3.2 MB/s | 548 kB 00:00 2024-03-18T15:54:53.585 INFO:teuthology.orchestra.run.smithi196.stdout:(108/174): grpc-data-1.46.7-10.el9.noarch.rpm 290 kB/s | 19 kB 00:00 2024-03-18T15:54:53.618 INFO:teuthology.orchestra.run.smithi196.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 784 kB/s | 26 kB 00:00 2024-03-18T15:54:53.635 INFO:teuthology.orchestra.run.smithi194.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-18T15:54:53.652 INFO:teuthology.orchestra.run.smithi196.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 1.4 MB/s | 49 kB 00:00 2024-03-18T15:54:53.663 INFO:teuthology.orchestra.run.smithi039.stdout:(38/174): ceph-test-19.0.0-2162.g3d78f5ba.el9.x 11 MB/s | 48 MB 00:04 2024-03-18T15:54:53.686 INFO:teuthology.orchestra.run.smithi196.stdout:(111/174): libunwind-1.6.2-1.el9.x86_64.rpm 2.0 MB/s | 67 kB 00:00 2024-03-18T15:54:53.713 INFO:teuthology.orchestra.run.smithi039.stdout:(39/174): automake-1.16.2-8.el9.noarch.rpm 982 kB/s | 662 kB 00:00 2024-03-18T15:54:53.715 INFO:teuthology.orchestra.run.smithi194.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-18T15:54:53.744 INFO:teuthology.orchestra.run.smithi196.stdout:(112/174): libarrow-9.0.0-10.el9.x86_64.rpm 24 MB/s | 4.4 MB 00:00 2024-03-18T15:54:53.763 INFO:teuthology.orchestra.run.smithi039.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 9.5 MB/s | 920 kB 00:00 2024-03-18T15:54:53.770 INFO:teuthology.orchestra.run.smithi196.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 1.8 MB/s | 151 kB 00:00 2024-03-18T15:54:53.784 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-18T15:54:53.789 INFO:teuthology.orchestra.run.smithi039.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 1.5 MB/s | 106 kB 00:00 2024-03-18T15:54:53.812 INFO:teuthology.orchestra.run.smithi196.stdout:(114/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 12 MB/s | 838 kB 00:00 2024-03-18T15:54:53.831 INFO:teuthology.orchestra.run.smithi039.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 4.6 MB/s | 307 kB 00:00 2024-03-18T15:54:53.845 INFO:teuthology.orchestra.run.smithi196.stdout:(115/174): python3-asyncssh-2.13.2-4.el9.noarch 7.1 MB/s | 548 kB 00:00 2024-03-18T15:54:53.863 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-18T15:54:53.881 INFO:teuthology.orchestra.run.smithi039.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 358 kB/s | 32 kB 00:00 2024-03-18T15:54:53.895 INFO:teuthology.orchestra.run.smithi196.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 521 kB/s | 43 kB 00:00 2024-03-18T15:54:53.921 INFO:teuthology.orchestra.run.smithi196.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 427 kB/s | 32 kB 00:00 2024-03-18T15:54:53.927 INFO:teuthology.orchestra.run.smithi194.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-18T15:54:53.946 INFO:teuthology.orchestra.run.smithi196.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 298 kB/s | 15 kB 00:00 2024-03-18T15:54:53.971 INFO:teuthology.orchestra.run.smithi196.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 3.3 MB/s | 172 kB 00:00 2024-03-18T15:54:53.981 INFO:teuthology.orchestra.run.smithi039.stdout:(44/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 20 MB/s | 3.0 MB 00:00 2024-03-18T15:54:53.996 INFO:teuthology.orchestra.run.smithi196.stdout:(120/174): python3-cherrypy-18.6.1-2.el9.noarch 7.0 MB/s | 358 kB 00:00 2024-03-18T15:54:54.006 INFO:teuthology.orchestra.run.smithi039.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 156 kB/s | 17 kB 00:00 2024-03-18T15:54:54.022 INFO:teuthology.orchestra.run.smithi196.stdout:(121/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 5.9 MB/s | 303 kB 00:00 2024-03-18T15:54:54.047 INFO:teuthology.orchestra.run.smithi196.stdout:(122/174): python3-colorama-0.4.4-12.el9.noarch 704 kB/s | 35 kB 00:00 2024-03-18T15:54:54.048 INFO:teuthology.orchestra.run.smithi039.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 249 kB/s | 10 kB 00:00 2024-03-18T15:54:54.065 INFO:teuthology.orchestra.run.smithi039.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.7 MB/s | 159 kB 00:00 2024-03-18T15:54:54.072 INFO:teuthology.orchestra.run.smithi196.stdout:(123/174): python3-google-auth-2.28.0-1.el9.noa 4.0 MB/s | 203 kB 00:00 2024-03-18T15:54:54.090 INFO:teuthology.orchestra.run.smithi039.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.1 MB/s | 44 kB 00:00 2024-03-18T15:54:54.106 INFO:teuthology.orchestra.run.smithi196.stdout:(124/174): python3-grpcio-tools-1.46.7-10.el9.x 4.2 MB/s | 144 kB 00:00 2024-03-18T15:54:54.132 INFO:teuthology.orchestra.run.smithi039.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 9.8 MB/s | 661 kB 00:00 2024-03-18T15:54:54.148 INFO:teuthology.orchestra.run.smithi196.stdout:(125/174): python3-grpcio-1.46.7-10.el9.x86_64. 20 MB/s | 2.0 MB 00:00 2024-03-18T15:54:54.156 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-18T15:54:54.173 INFO:teuthology.orchestra.run.smithi196.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 160 kB/s | 11 kB 00:00 2024-03-18T15:54:54.174 INFO:teuthology.orchestra.run.smithi039.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 7.4 MB/s | 578 kB 00:00 2024-03-18T15:54:54.198 INFO:teuthology.orchestra.run.smithi196.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 352 kB/s | 18 kB 00:00 2024-03-18T15:54:54.200 INFO:teuthology.orchestra.run.smithi039.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 758 kB/s | 46 kB 00:00 2024-03-18T15:54:54.224 INFO:teuthology.orchestra.run.smithi196.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 462 kB/s | 23 kB 00:00 2024-03-18T15:54:54.249 INFO:teuthology.orchestra.run.smithi196.stdout:(129/174): python3-jaraco-functools-3.5.0-2.el9 385 kB/s | 19 kB 00:00 2024-03-18T15:54:54.268 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-18T15:54:54.268 INFO:teuthology.orchestra.run.smithi039.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.0 MB/s | 282 kB 00:00 2024-03-18T15:54:54.275 INFO:teuthology.orchestra.run.smithi196.stdout:(130/174): python3-jaraco-text-3.2.0-6.el9.noar 390 kB/s | 20 kB 00:00 2024-03-18T15:54:54.301 INFO:teuthology.orchestra.run.smithi039.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 1.9 MB/s | 187 kB 00:00 2024-03-18T15:54:54.308 INFO:teuthology.orchestra.run.smithi196.stdout:(131/174): python3-kubernetes-26.1.0-2.el9.noar 17 MB/s | 1.0 MB 00:00 2024-03-18T15:54:54.333 INFO:teuthology.orchestra.run.smithi196.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 792 kB/s | 46 kB 00:00 2024-03-18T15:54:54.334 INFO:teuthology.orchestra.run.smithi039.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.3 MB/s | 217 kB 00:00 2024-03-18T15:54:54.359 INFO:teuthology.orchestra.run.smithi196.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 1.5 MB/s | 79 kB 00:00 2024-03-18T15:54:54.360 INFO:teuthology.orchestra.run.smithi039.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 613 kB/s | 34 kB 00:00 2024-03-18T15:54:54.362 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-18T15:54:54.376 INFO:teuthology.orchestra.run.smithi196.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 1.3 MB/s | 58 kB 00:00 2024-03-18T15:54:54.390 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-18T15:54:54.416 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-18T15:54:54.434 INFO:teuthology.orchestra.run.smithi196.stdout:(135/174): openssl-devel-3.0.7-27.el9.x86_64.rp 1.7 MB/s | 4.1 MB 00:02 2024-03-18T15:54:54.451 INFO:teuthology.orchestra.run.smithi196.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 218 kB/s | 16 kB 00:00 2024-03-18T15:54:54.452 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-18T15:54:54.477 INFO:teuthology.orchestra.run.smithi196.stdout:(137/174): python3-pecan-1.4.2-3.el9.noarch.rpm 2.3 MB/s | 272 kB 00:00 2024-03-18T15:54:54.494 INFO:teuthology.orchestra.run.smithi196.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 674 kB/s | 28 kB 00:00 2024-03-18T15:54:54.501 INFO:teuthology.orchestra.run.smithi039.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 29 MB/s | 4.7 MB 00:00 2024-03-18T15:54:54.510 INFO:teuthology.orchestra.run.smithi196.stdout:(139/174): python3-repoze-lru-0.7-16.el9.noarch 922 kB/s | 31 kB 00:00 2024-03-18T15:54:54.527 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-18T15:54:54.536 INFO:teuthology.orchestra.run.smithi196.stdout:(140/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 891 kB/s | 90 kB 00:00 2024-03-18T15:54:54.552 INFO:teuthology.orchestra.run.smithi039.stdout:(57/174): postgresql-13.11-1.el9.x86_64.rpm 8.7 MB/s | 1.5 MB 00:00 2024-03-18T15:54:54.561 INFO:teuthology.orchestra.run.smithi196.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 1.2 MB/s | 59 kB 00:00 2024-03-18T15:54:54.580 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-18T15:54:54.585 INFO:teuthology.orchestra.run.smithi039.stdout:(58/174): postgresql-contrib-13.11-1.el9.x86_64 9.9 MB/s | 811 kB 00:00 2024-03-18T15:54:54.586 INFO:teuthology.orchestra.run.smithi196.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 714 kB/s | 36 kB 00:00 2024-03-18T15:54:54.612 INFO:teuthology.orchestra.run.smithi196.stdout:(143/174): python3-routes-2.5.1-5.el9.noarch.rp 1.6 MB/s | 188 kB 00:00 2024-03-18T15:54:54.635 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-18T15:54:54.636 INFO:teuthology.orchestra.run.smithi039.stdout:(59/174): postgresql-private-libs-13.11-1.el9.x 1.7 MB/s | 134 kB 00:00 2024-03-18T15:54:54.637 INFO:teuthology.orchestra.run.smithi196.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 676 kB/s | 51 kB 00:00 2024-03-18T15:54:54.662 INFO:teuthology.orchestra.run.smithi196.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 1.8 MB/s | 90 kB 00:00 2024-03-18T15:54:54.687 INFO:teuthology.orchestra.run.smithi196.stdout:(146/174): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-03-18T15:54:54.700 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-18T15:54:54.713 INFO:teuthology.orchestra.run.smithi196.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 399 kB/s | 20 kB 00:00 2024-03-18T15:54:54.734 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-18T15:54:54.738 INFO:teuthology.orchestra.run.smithi196.stdout:(148/174): re2-20211101-3.el9.x86_64.rpm 3.7 MB/s | 192 kB 00:00 2024-03-18T15:54:54.763 INFO:teuthology.orchestra.run.smithi196.stdout:(149/174): python3-werkzeug-2.0.3-3.el9.1.noarc 3.3 MB/s | 427 kB 00:00 2024-03-18T15:54:54.778 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-18T15:54:54.786 INFO:teuthology.orchestra.run.smithi039.stdout:(60/174): postgresql-server-13.11-1.el9.x86_64. 29 MB/s | 5.7 MB 00:00 2024-03-18T15:54:54.805 INFO:teuthology.orchestra.run.smithi196.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 17 MB/s | 1.6 MB 00:00 2024-03-18T15:54:54.810 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-18T15:54:54.828 INFO:teuthology.orchestra.run.smithi039.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 5.7 MB/s | 1.0 MB 00:00 2024-03-18T15:54:54.838 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-18T15:54:54.847 INFO:teuthology.orchestra.run.smithi196.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 7.6 MB/s | 321 kB 00:00 2024-03-18T15:54:54.887 INFO:teuthology.orchestra.run.smithi039.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 78 kB 00:00 2024-03-18T15:54:54.909 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-18T15:54:54.943 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 39/198 2024-03-18T15:54:54.986 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 40/198 2024-03-18T15:54:55.015 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 41/198 2024-03-18T15:54:55.045 INFO:teuthology.orchestra.run.smithi039.stdout:(63/174): python3-babel-2.9.1-2.el9.noarch.rpm 27 MB/s | 5.8 MB 00:00 2024-03-18T15:54:55.062 INFO:teuthology.orchestra.run.smithi039.stdout:(64/174): python3-jsonpatch-1.21-16.el9.noarch. 153 kB/s | 24 kB 00:00 2024-03-18T15:54:55.072 INFO:teuthology.orchestra.run.smithi196.stdout:(152/174): librados2-19.0.0-2162.g3d78f5ba.el9. 10 MB/s | 3.4 MB 00:00 2024-03-18T15:54:55.086 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-18T15:54:55.096 INFO:teuthology.orchestra.run.smithi039.stdout:(65/174): python3-jsonpointer-2.0-4.el9.noarch. 565 kB/s | 18 kB 00:00 2024-03-18T15:54:55.121 INFO:teuthology.orchestra.run.smithi039.stdout:(66/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.5 MB/s | 144 kB 00:00 2024-03-18T15:54:55.123 INFO:teuthology.orchestra.run.smithi196.stdout:(153/174): NetworkManager-1.46.0-1.el9.x86_64.r 8.3 MB/s | 2.3 MB 00:00 2024-03-18T15:54:55.129 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-18T15:54:55.155 INFO:teuthology.orchestra.run.smithi039.stdout:(67/174): python3-markupsafe-1.1.1-12.el9.x86_6 984 kB/s | 32 kB 00:00 2024-03-18T15:54:55.173 INFO:teuthology.orchestra.run.smithi196.stdout:(154/174): NetworkManager-team-1.46.0-1.el9.x86 850 kB/s | 42 kB 00:00 2024-03-18T15:54:55.204 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-18T15:54:55.249 INFO:teuthology.orchestra.run.smithi196.stdout:(155/174): librbd1-19.0.0-2162.g3d78f5ba.el9.x8 6.5 MB/s | 3.1 MB 00:00 2024-03-18T15:54:55.273 INFO:teuthology.orchestra.run.smithi196.stdout:(156/174): NetworkManager-tui-1.46.0-1.el9.x86_ 2.4 MB/s | 248 kB 00:00 2024-03-18T15:54:55.288 INFO:teuthology.orchestra.run.smithi039.stdout:(68/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 29 MB/s | 4.8 MB 00:00 2024-03-18T15:54:55.313 INFO:teuthology.orchestra.run.smithi039.stdout:(69/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.5 MB/s | 217 kB 00:00 2024-03-18T15:54:55.315 INFO:teuthology.orchestra.run.smithi196.stdout:(157/174): elfutils-debuginfod-client-0.190-2.e 880 kB/s | 36 kB 00:00 2024-03-18T15:54:55.336 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-18T15:54:55.355 INFO:teuthology.orchestra.run.smithi039.stdout:(70/174): python3-oauthlib-3.1.1-5.el9.noarch.r 4.2 MB/s | 174 kB 00:00 2024-03-18T15:54:55.358 INFO:teuthology.orchestra.run.smithi196.stdout:(158/174): acl-2.3.1-4.el9.x86_64.rpm 655 kB/s | 71 kB 00:00 2024-03-18T15:54:55.383 INFO:teuthology.orchestra.run.smithi196.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 2.8 MB/s | 192 kB 00:00 2024-03-18T15:54:55.390 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-18T15:54:55.412 INFO:teuthology.orchestra.run.smithi039.stdout:(71/174): python3-packaging-20.9-5.el9.noarch.r 708 kB/s | 69 kB 00:00 2024-03-18T15:54:55.425 INFO:teuthology.orchestra.run.smithi196.stdout:(160/174): elfutils-libs-0.190-2.el9.x86_64.rpm 3.7 MB/s | 256 kB 00:00 2024-03-18T15:54:55.439 INFO:teuthology.orchestra.run.smithi039.stdout:(72/174): python3-prettytable-0.7.2-27.el9.noar 502 kB/s | 41 kB 00:00 2024-03-18T15:54:55.450 INFO:teuthology.orchestra.run.smithi196.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 344 kB/s | 23 kB 00:00 2024-03-18T15:54:55.473 INFO:teuthology.orchestra.run.smithi039.stdout:(73/174): python3-protobuf-3.14.0-13.el9.noarch 3.9 MB/s | 238 kB 00:00 2024-03-18T15:54:55.475 INFO:teuthology.orchestra.run.smithi196.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 2.1 MB/s | 108 kB 00:00 2024-03-18T15:54:55.498 INFO:teuthology.orchestra.run.smithi039.stdout:(74/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.3 MB/s | 132 kB 00:00 2024-03-18T15:54:55.517 INFO:teuthology.orchestra.run.smithi196.stdout:(163/174): NetworkManager-libnm-1.46.0-1.el9.x8 4.1 MB/s | 1.8 MB 00:00 2024-03-18T15:54:55.532 INFO:teuthology.orchestra.run.smithi039.stdout:(75/174): python3-pyasn1-modules-0.4.8-6.el9.no 3.6 MB/s | 211 kB 00:00 2024-03-18T15:54:55.542 INFO:teuthology.orchestra.run.smithi196.stdout:(164/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.6 MB/s | 155 kB 00:00 2024-03-18T15:54:55.557 INFO:teuthology.orchestra.run.smithi039.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 857 kB/s | 47 kB 00:00 2024-03-18T15:54:55.568 INFO:teuthology.orchestra.run.smithi196.stdout:(165/174): libmount-2.37.4-18.el9.x86_64.rpm 1.4 MB/s | 136 kB 00:00 2024-03-18T15:54:55.582 INFO:teuthology.orchestra.run.smithi039.stdout:(77/174): python3-requests-oauthlib-1.3.0-12.el 877 kB/s | 43 kB 00:00 2024-03-18T15:54:55.593 INFO:teuthology.orchestra.run.smithi196.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 574 kB/s | 29 kB 00:00 2024-03-18T15:54:55.618 INFO:teuthology.orchestra.run.smithi196.stdout:(167/174): libsmartcols-2.37.4-18.el9.x86_64.rp 848 kB/s | 64 kB 00:00 2024-03-18T15:54:55.677 INFO:teuthology.orchestra.run.smithi196.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 11 MB/s | 1.2 MB 00:00 2024-03-18T15:54:55.727 INFO:teuthology.orchestra.run.smithi196.stdout:(169/174): util-linux-2.37.4-18.el9.x86_64.rpm 21 MB/s | 2.3 MB 00:00 2024-03-18T15:54:55.752 INFO:teuthology.orchestra.run.smithi196.stdout:(170/174): util-linux-core-2.37.4-18.el9.x86_64 6.1 MB/s | 465 kB 00:00 2024-03-18T15:54:55.786 INFO:teuthology.orchestra.run.smithi196.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 533 kB/s | 31 kB 00:00 2024-03-18T15:54:55.799 INFO:teuthology.orchestra.run.smithi039.stdout:(78/174): python3-toml-0.10.2-6.el9.noarch.rpm 192 kB/s | 39 kB 00:00 2024-03-18T15:54:55.845 INFO:teuthology.orchestra.run.smithi196.stdout:(172/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 12 MB/s | 1.1 MB 00:00 2024-03-18T15:54:55.912 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-18T15:54:55.978 INFO:teuthology.orchestra.run.smithi196.stdout:(173/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 5.6 MB/s | 2.2 MB 00:00 2024-03-18T15:54:55.995 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-18T15:54:56.000 INFO:teuthology.orchestra.run.smithi039.stdout:(79/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 36 MB/s | 16 MB 00:00 2024-03-18T15:54:56.025 INFO:teuthology.orchestra.run.smithi039.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 188 kB/s | 41 kB 00:00 2024-03-18T15:54:56.059 INFO:teuthology.orchestra.run.smithi039.stdout:(81/174): socat-1.7.4.1-5.el9.x86_64.rpm 5.1 MB/s | 300 kB 00:00 2024-03-18T15:54:56.073 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-18T15:54:56.093 INFO:teuthology.orchestra.run.smithi039.stdout:(82/174): uuid-1.6.2-55.el9.x86_64.rpm 891 kB/s | 56 kB 00:00 2024-03-18T15:54:56.114 INFO:teuthology.orchestra.run.smithi194.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-18T15:54:56.123 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-18T15:54:56.145 INFO:teuthology.orchestra.run.smithi039.stdout:(83/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 760 kB/s | 63 kB 00:00 2024-03-18T15:54:56.154 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-18T15:54:56.154 INFO:teuthology.orchestra.run.smithi196.stdout:(174/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 202 kB/s | 74 kB 00:00 2024-03-18T15:54:56.158 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-18T15:54:56.158 INFO:teuthology.orchestra.run.smithi196.stdout:Total 17 MB/s | 224 MB 00:13 2024-03-18T15:54:56.168 INFO:teuthology.orchestra.run.smithi039.stdout:(84/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 596 kB/s | 44 kB 00:00 2024-03-18T15:54:56.202 INFO:teuthology.orchestra.run.smithi039.stdout:(85/174): fuse-2.9.9-15.el9.x86_64.rpm 1.3 MB/s | 79 kB 00:00 2024-03-18T15:54:56.203 INFO:teuthology.orchestra.run.smithi194.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-18T15:54:56.235 INFO:teuthology.orchestra.run.smithi039.stdout:(86/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.6 MB/s | 241 kB 00:00 2024-03-18T15:54:56.240 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 53/198 2024-03-18T15:54:56.270 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-18T15:54:56.294 INFO:teuthology.orchestra.run.smithi039.stdout:(87/174): python3-cryptography-36.0.1-4.el9.x86 13 MB/s | 1.1 MB 00:00 2024-03-18T15:54:56.320 INFO:teuthology.orchestra.run.smithi039.stdout:(88/174): python3-ply-3.11-14.el9.noarch.rpm 1.4 MB/s | 103 kB 00:00 2024-03-18T15:54:56.353 INFO:teuthology.orchestra.run.smithi039.stdout:(89/174): python3-pycparser-2.20-6.el9.noarch.r 2.1 MB/s | 124 kB 00:00 2024-03-18T15:54:56.387 INFO:teuthology.orchestra.run.smithi039.stdout:(90/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.8 MB/s | 187 kB 00:00 2024-03-18T15:54:56.500 INFO:teuthology.orchestra.run.smithi194.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-18T15:54:56.628 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-18T15:54:56.652 INFO:teuthology.orchestra.run.smithi194.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-18T15:54:56.722 INFO:teuthology.orchestra.run.smithi194.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-18T15:54:56.764 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-18T15:54:56.901 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-18T15:54:56.901 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-18T15:54:56.905 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-18T15:54:56.966 INFO:teuthology.orchestra.run.smithi194.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-18T15:54:57.022 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-18T15:54:57.062 INFO:teuthology.orchestra.run.smithi039.stdout:(91/174): ceph-common-19.0.0-2162.g3d78f5ba.el9 1.9 MB/s | 21 MB 00:11 2024-03-18T15:54:57.087 INFO:teuthology.orchestra.run.smithi039.stdout:(92/174): jq-1.6-16.el9.x86_64.rpm 256 kB/s | 187 kB 00:00 2024-03-18T15:54:57.096 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-18T15:54:57.125 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-18T15:54:57.197 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-18T15:54:57.271 INFO:teuthology.orchestra.run.smithi039.stdout:(93/174): libgfortran-11.4.1-3.el9.x86_64.rpm 909 kB/s | 803 kB 00:00 2024-03-18T15:54:57.296 INFO:teuthology.orchestra.run.smithi039.stdout:(94/174): python3-requests-2.25.1-8.el9.noarch. 601 kB/s | 125 kB 00:00 2024-03-18T15:54:57.352 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-18T15:54:57.512 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-18T15:54:57.626 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-18T15:54:57.672 INFO:teuthology.orchestra.run.smithi039.stdout:(95/174): libquadmath-11.4.1-3.el9.x86_64.rpm 321 kB/s | 195 kB 00:00 2024-03-18T15:54:57.747 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-18T15:54:57.884 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-18T15:54:57.944 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-18T15:54:57.974 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-18T15:54:58.006 INFO:teuthology.orchestra.run.smithi039.stdout:(96/174): elfutils-libelf-devel-0.190-2.el9.x86 29 kB/s | 21 kB 00:00 2024-03-18T15:54:58.016 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-18T15:54:58.022 INFO:teuthology.orchestra.run.smithi039.stdout:(97/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 35 kB/s | 25 kB 00:00 2024-03-18T15:54:58.111 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-18T15:54:58.157 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-18T15:54:58.197 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-18T15:54:58.223 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-18T15:54:58.239 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-18T15:54:58.313 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-18T15:54:58.442 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-18T15:54:58.543 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-18T15:54:58.580 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-18T15:54:58.637 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-18T15:54:58.662 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-18T15:54:58.662 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-18T15:54:58.693 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-18T15:54:58.879 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-18T15:54:58.926 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-18T15:54:58.964 INFO:teuthology.orchestra.run.smithi039.stdout:(98/174): python3-devel-3.9.17-2.el9.x86_64.rpm 257 kB/s | 245 kB 00:00 2024-03-18T15:54:59.057 INFO:teuthology.orchestra.run.smithi039.stdout:(99/174): python3-jinja2-2.11.3-5.el9.noarch.rp 240 kB/s | 248 kB 00:01 2024-03-18T15:54:59.060 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-18T15:54:59.099 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-18T15:54:59.131 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-18T15:54:59.162 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-18T15:54:59.181 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-18T15:54:59.207 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-18T15:54:59.246 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-18T15:54:59.278 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-18T15:54:59.325 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-18T15:54:59.402 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-18T15:54:59.435 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-18T15:54:59.468 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-18T15:54:59.509 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-18T15:54:59.544 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-18T15:54:59.635 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-18T15:54:59.732 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-18T15:54:59.774 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-18T15:54:59.806 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-18T15:54:59.852 INFO:teuthology.orchestra.run.smithi039.stdout:(100/174): qatlib-service-23.11.0-1.el9.x86_64. 44 kB/s | 35 kB 00:00 2024-03-18T15:54:59.854 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-18T15:54:59.952 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-18T15:55:00.000 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-18T15:55:00.074 INFO:teuthology.orchestra.run.smithi039.stdout:(101/174): qatlib-23.11.0-1.el9.x86_64.rpm 195 kB/s | 216 kB 00:01 2024-03-18T15:55:00.615 INFO:teuthology.orchestra.run.smithi039.stdout:(102/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 58 kB/s | 44 kB 00:00 2024-03-18T15:55:00.707 INFO:teuthology.orchestra.run.smithi039.stdout:(103/174): lua-devel-5.4.4-4.el9.x86_64.rpm 247 kB/s | 22 kB 00:00 2024-03-18T15:55:00.743 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-18T15:55:00.788 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-18T15:55:00.821 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-18T15:55:00.850 INFO:teuthology.orchestra.run.smithi039.stdout:(104/174): protobuf-compiler-3.14.0-13.el9.x86_ 6.0 MB/s | 863 kB 00:00 2024-03-18T15:55:00.870 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-18T15:55:00.883 INFO:teuthology.orchestra.run.smithi039.stdout:(105/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 116 kB/s | 94 kB 00:00 2024-03-18T15:55:00.959 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-18T15:55:00.997 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-18T15:55:01.173 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-18T15:55:01.203 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-18T15:55:01.280 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-18T15:55:01.884 INFO:teuthology.orchestra.run.smithi039.stdout:(106/174): gperftools-libs-2.9.1-2.el9.x86_64.r 309 kB/s | 309 kB 00:00 2024-03-18T15:55:01.925 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-18T15:55:02.035 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-18T15:55:02.117 INFO:teuthology.orchestra.run.smithi039.stdout:(107/174): abseil-cpp-20211102.0-3.el9.x86_64.r 433 kB/s | 548 kB 00:01 2024-03-18T15:55:02.393 INFO:teuthology.orchestra.run.smithi039.stdout:(108/174): grpc-data-1.46.7-10.el9.noarch.rpm 38 kB/s | 19 kB 00:00 2024-03-18T15:55:02.901 INFO:teuthology.orchestra.run.smithi039.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 51 kB/s | 26 kB 00:00 2024-03-18T15:55:03.261 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-18T15:55:03.327 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-18T15:55:03.358 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-18T15:55:03.398 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-18T15:55:03.425 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-18T15:55:03.485 INFO:teuthology.orchestra.run.smithi039.stdout:(110/174): liboath-2.6.7-2.el9.x86_64.rpm 83 kB/s | 49 kB 00:00 2024-03-18T15:55:03.738 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-18T15:55:03.763 INFO:teuthology.orchestra.run.smithi194.stdout: Upgrading : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 124/198 2024-03-18T15:55:03.838 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 124/198 2024-03-18T15:55:03.868 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 125/198 2024-03-18T15:55:03.904 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-18T15:55:04.060 INFO:teuthology.orchestra.run.smithi039.stdout:(111/174): openssl-devel-3.0.7-27.el9.x86_64.rp 663 kB/s | 4.1 MB 00:06 2024-03-18T15:55:04.119 INFO:teuthology.orchestra.run.smithi039.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 107 kB/s | 67 kB 00:00 2024-03-18T15:55:04.367 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-18T15:55:04.400 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 128/198 2024-03-18T15:55:04.471 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 128/198 2024-03-18T15:55:04.501 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 129/198 2024-03-18T15:55:04.736 INFO:teuthology.orchestra.run.smithi039.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 224 kB/s | 151 kB 00:00 2024-03-18T15:55:05.303 INFO:teuthology.orchestra.run.smithi039.stdout:(114/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 709 kB/s | 838 kB 00:01 2024-03-18T15:55:05.665 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-18T15:55:05.797 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-18T15:55:05.833 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-18T15:55:05.836 INFO:teuthology.orchestra.run.smithi039.stdout:(115/174): python3-asyncssh-2.13.2-4.el9.noarch 499 kB/s | 548 kB 00:01 2024-03-18T15:55:05.861 INFO:teuthology.orchestra.run.smithi039.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 78 kB/s | 43 kB 00:00 2024-03-18T15:55:05.864 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-18T15:55:05.873 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-18T15:55:05.913 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-18T15:55:05.966 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-18T15:55:06.001 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-18T15:55:06.026 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-18T15:55:06.068 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-18T15:55:06.111 INFO:teuthology.orchestra.run.smithi039.stdout:(117/174): libarrow-9.0.0-10.el9.x86_64.rpm 1.1 MB/s | 4.4 MB 00:03 2024-03-18T15:55:06.404 INFO:teuthology.orchestra.run.smithi039.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 28 kB/s | 15 kB 00:00 2024-03-18T15:55:06.429 INFO:teuthology.orchestra.run.smithi039.stdout:(119/174): python3-cachetools-4.2.4-1.el9.noarc 54 kB/s | 32 kB 00:00 2024-03-18T15:55:06.435 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-18T15:55:06.560 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-18T15:55:06.665 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-18T15:55:06.695 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-18T15:55:06.723 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-18T15:55:06.756 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-18T15:55:06.786 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-18T15:55:06.815 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/198 2024-03-18T15:55:06.860 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-18T15:55:06.879 INFO:teuthology.orchestra.run.smithi039.stdout:(120/174): python3-cheroot-8.6.0-4.el9.noarch.r 225 kB/s | 172 kB 00:00 2024-03-18T15:55:07.025 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-18T15:55:07.080 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-18T15:55:07.108 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-18T15:55:07.195 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-18T15:55:07.218 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 17/198 2024-03-18T15:55:07.282 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 17/198 2024-03-18T15:55:07.293 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-18T15:55:07.313 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-18T15:55:07.334 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/198 2024-03-18T15:55:07.336 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-18T15:55:07.354 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-18T15:55:07.415 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/198 2024-03-18T15:55:07.415 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-18T15:55:07.415 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-18T15:55:07.415 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-18T15:55:07.429 INFO:teuthology.orchestra.run.smithi039.stdout:(121/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 303 kB/s | 303 kB 00:00 2024-03-18T15:55:07.443 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/198 2024-03-18T15:55:07.463 INFO:teuthology.orchestra.run.smithi039.stdout:(122/174): python3-cherrypy-18.6.1-2.el9.noarch 339 kB/s | 358 kB 00:01 2024-03-18T15:55:07.488 INFO:teuthology.orchestra.run.smithi039.stdout:(123/174): python3-colorama-0.4.4-12.el9.noarch 58 kB/s | 35 kB 00:00 2024-03-18T15:55:07.502 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-18T15:55:07.534 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-18T15:55:07.566 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-18T15:55:07.671 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-18T15:55:07.784 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-18T15:55:07.845 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-18T15:55:07.915 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-18T15:55:07.963 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-18T15:55:08.113 INFO:teuthology.orchestra.run.smithi039.stdout:(124/174): python3-google-auth-2.28.0-1.el9.noa 297 kB/s | 203 kB 00:00 2024-03-18T15:55:08.178 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-18T15:55:08.197 INFO:teuthology.orchestra.run.smithi039.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 204 kB/s | 144 kB 00:00 2024-03-18T15:55:08.283 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-18T15:55:08.379 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-18T15:55:08.409 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-18T15:55:08.436 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-18T15:55:08.471 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-18T15:55:08.547 INFO:teuthology.orchestra.run.smithi039.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 25 kB/s | 11 kB 00:00 2024-03-18T15:55:08.554 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-18T15:55:08.608 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-18T15:55:08.646 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-18T15:55:08.709 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-18T15:55:08.731 INFO:teuthology.orchestra.run.smithi039.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 38 kB/s | 18 kB 00:00 2024-03-18T15:55:08.745 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-18T15:55:08.789 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-18T15:55:08.821 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-18T15:55:08.849 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-18T15:55:08.918 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-18T15:55:08.946 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 39/198 2024-03-18T15:55:08.981 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 40/198 2024-03-18T15:55:09.006 INFO:teuthology.orchestra.run.smithi039.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 51 kB/s | 23 kB 00:00 2024-03-18T15:55:09.010 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 41/198 2024-03-18T15:55:09.078 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-18T15:55:09.125 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-18T15:55:09.190 INFO:teuthology.orchestra.run.smithi039.stdout:(129/174): python3-jaraco-functools-3.5.0-2.el9 42 kB/s | 19 kB 00:00 2024-03-18T15:55:09.198 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-18T15:55:09.332 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-18T15:55:09.393 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-18T15:55:09.518 INFO:teuthology.orchestra.run.smithi039.stdout:(130/174): python3-grpcio-1.46.7-10.el9.x86_64. 1.0 MB/s | 2.0 MB 00:02 2024-03-18T15:55:09.549 INFO:teuthology.orchestra.run.smithi039.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 36 kB/s | 20 kB 00:00 2024-03-18T15:55:09.911 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-18T15:55:09.989 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-18T15:55:10.024 INFO:teuthology.orchestra.run.smithi039.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 92 kB/s | 46 kB 00:00 2024-03-18T15:55:10.065 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-18T15:55:10.118 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-18T15:55:10.126 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-18T15:55:10.158 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-18T15:55:10.207 INFO:teuthology.orchestra.run.smithi039.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 120 kB/s | 79 kB 00:00 2024-03-18T15:55:10.215 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-18T15:55:10.253 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 53/198 2024-03-18T15:55:10.282 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-18T15:55:10.506 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-18T15:55:10.575 INFO:teuthology.orchestra.run.smithi039.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 105 kB/s | 58 kB 00:00 2024-03-18T15:55:10.634 INFO:teuthology.orchestra.run.smithi039.stdout:(135/174): python3-kubernetes-26.1.0-2.el9.noar 725 kB/s | 1.0 MB 00:01 2024-03-18T15:55:10.658 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-18T15:55:10.718 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-18T15:55:10.762 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-18T15:55:10.895 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-18T15:55:10.951 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-18T15:55:11.006 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-18T15:55:11.034 INFO:teuthology.orchestra.run.smithi039.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 36 kB/s | 16 kB 00:00 2024-03-18T15:55:11.078 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-18T15:55:11.110 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-18T15:55:11.117 INFO:teuthology.orchestra.run.smithi039.stdout:(137/174): python3-pecan-1.4.2-3.el9.noarch.rpm 300 kB/s | 272 kB 00:00 2024-03-18T15:55:11.147 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-18T15:55:11.193 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-18T15:55:11.209 INFO:teuthology.orchestra.run.smithi039.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 156 kB/s | 90 kB 00:00 2024-03-18T15:55:11.230 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-18T15:55:11.272 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-18T15:55:11.315 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-18T15:55:11.448 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-18T15:55:11.518 INFO:teuthology.orchestra.run.smithi039.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 58 kB/s | 28 kB 00:00 2024-03-18T15:55:11.701 INFO:teuthology.orchestra.run.smithi039.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 60 kB/s | 31 kB 00:00 2024-03-18T15:55:11.776 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-18T15:55:11.810 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-18T15:55:11.853 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-18T15:55:11.949 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-18T15:55:11.982 INFO:teuthology.orchestra.run.smithi039.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 244 kB/s | 188 kB 00:00 2024-03-18T15:55:12.021 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-18T15:55:12.049 INFO:teuthology.orchestra.run.smithi039.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 111 kB/s | 59 kB 00:00 2024-03-18T15:55:12.166 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-18T15:55:12.216 INFO:teuthology.orchestra.run.smithi039.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 70 kB/s | 36 kB 00:00 2024-03-18T15:55:12.291 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-18T15:55:12.425 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-18T15:55:12.503 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-18T15:55:12.626 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-18T15:55:12.732 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-18T15:55:12.774 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-18T15:55:12.776 INFO:teuthology.orchestra.run.smithi039.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 64 kB/s | 51 kB 00:00 2024-03-18T15:55:12.823 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-18T15:55:12.879 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-18T15:55:13.034 INFO:teuthology.orchestra.run.smithi039.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 234 kB/s | 230 kB 00:00 2024-03-18T15:55:13.061 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-18T15:55:13.068 INFO:teuthology.orchestra.run.smithi039.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 106 kB/s | 90 kB 00:00 2024-03-18T15:55:13.112 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-18T15:55:13.241 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-18T15:55:13.277 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-18T15:55:13.309 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-18T15:55:13.340 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-18T15:55:13.368 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-18T15:55:13.402 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-18T15:55:13.441 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-18T15:55:13.473 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-18T15:55:13.491 INFO:teuthology.orchestra.run.smithi039.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 44 kB/s | 20 kB 00:00 2024-03-18T15:55:13.503 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-18T15:55:13.579 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-18T15:55:13.613 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-18T15:55:13.655 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-18T15:55:13.676 INFO:teuthology.orchestra.run.smithi039.stdout:(148/174): python3-werkzeug-2.0.3-3.el9.1.noarc 475 kB/s | 427 kB 00:00 2024-03-18T15:55:13.696 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-18T15:55:13.739 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-18T15:55:13.760 INFO:teuthology.orchestra.run.smithi039.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 277 kB/s | 192 kB 00:00 2024-03-18T15:55:13.829 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-18T15:55:13.918 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-18T15:55:13.962 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-18T15:55:13.993 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-18T15:55:14.029 INFO:teuthology.orchestra.run.smithi039.stdout:(150/174): librados2-19.0.0-2162.g3d78f5ba.el9. 9.7 MB/s | 3.4 MB 00:00 2024-03-18T15:55:14.040 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-18T15:55:14.078 INFO:teuthology.orchestra.run.smithi039.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 6.6 MB/s | 321 kB 00:00 2024-03-18T15:55:14.138 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-18T15:55:14.195 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-18T15:55:14.203 INFO:teuthology.orchestra.run.smithi039.stdout:(152/174): librbd1-19.0.0-2162.g3d78f5ba.el9.x8 7.1 MB/s | 3.1 MB 00:00 2024-03-18T15:55:14.915 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-18T15:55:14.958 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-18T15:55:14.992 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-18T15:55:15.041 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-18T15:55:15.129 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-18T15:55:15.160 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-18T15:55:15.178 INFO:teuthology.orchestra.run.smithi039.stdout:(153/174): thrift-0.15.0-2.el9.x86_64.rpm 961 kB/s | 1.6 MB 00:01 2024-03-18T15:55:15.245 INFO:teuthology.orchestra.run.smithi039.stdout:(154/174): NetworkManager-libnm-1.46.0-1.el9.x8 1.7 MB/s | 1.8 MB 00:01 2024-03-18T15:55:15.336 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-18T15:55:15.370 INFO:teuthology.orchestra.run.smithi039.stdout:(155/174): NetworkManager-tui-1.46.0-1.el9.x86_ 2.0 MB/s | 248 kB 00:00 2024-03-18T15:55:15.374 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-18T15:55:15.451 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-18T15:55:15.479 INFO:teuthology.orchestra.run.smithi039.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 664 kB/s | 71 kB 00:00 2024-03-18T15:55:15.546 INFO:teuthology.orchestra.run.smithi039.stdout:(157/174): NetworkManager-team-1.46.0-1.el9.x86 116 kB/s | 42 kB 00:00 2024-03-18T15:55:15.588 INFO:teuthology.orchestra.run.smithi039.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 337 kB/s | 36 kB 00:00 2024-03-18T15:55:15.697 INFO:teuthology.orchestra.run.smithi039.stdout:(159/174): elfutils-libs-0.190-2.el9.x86_64.rpm 2.3 MB/s | 256 kB 00:00 2024-03-18T15:55:15.805 INFO:teuthology.orchestra.run.smithi039.stdout:(160/174): libacl-2.3.1-4.el9.x86_64.rpm 214 kB/s | 23 kB 00:00 2024-03-18T15:55:15.896 INFO:teuthology.orchestra.run.smithi039.stdout:(161/174): NetworkManager-1.46.0-1.el9.x86_64.r 1.3 MB/s | 2.3 MB 00:01 2024-03-18T15:55:15.930 INFO:teuthology.orchestra.run.smithi039.stdout:(162/174): elfutils-libelf-0.190-2.el9.x86_64.r 501 kB/s | 192 kB 00:00 2024-03-18T15:55:15.956 INFO:teuthology.orchestra.run.smithi039.stdout:(163/174): libblkid-2.37.4-18.el9.x86_64.rpm 724 kB/s | 108 kB 00:00 2024-03-18T15:55:16.006 INFO:teuthology.orchestra.run.smithi039.stdout:(164/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.4 MB/s | 155 kB 00:00 2024-03-18T15:55:16.040 INFO:teuthology.orchestra.run.smithi039.stdout:(165/174): libmount-2.37.4-18.el9.x86_64.rpm 1.2 MB/s | 136 kB 00:00 2024-03-18T15:55:16.065 INFO:teuthology.orchestra.run.smithi039.stdout:(166/174): libsmartcols-2.37.4-18.el9.x86_64.rp 590 kB/s | 64 kB 00:00 2024-03-18T15:55:16.092 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-18T15:55:16.115 INFO:teuthology.orchestra.run.smithi039.stdout:(167/174): libuuid-2.37.4-18.el9.x86_64.rpm 265 kB/s | 29 kB 00:00 2024-03-18T15:55:16.197 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-18T15:55:16.382 INFO:teuthology.orchestra.run.smithi039.stdout:(168/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 6.8 MB/s | 2.2 MB 00:00 2024-03-18T15:55:16.432 INFO:teuthology.orchestra.run.smithi039.stdout:(169/174): openssl-3.0.7-27.el9.x86_64.rpm 3.1 MB/s | 1.2 MB 00:00 2024-03-18T15:55:16.508 INFO:teuthology.orchestra.run.smithi039.stdout:(170/174): util-linux-core-2.37.4-18.el9.x86_64 3.7 MB/s | 465 kB 00:00 2024-03-18T15:55:16.542 INFO:teuthology.orchestra.run.smithi039.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 287 kB/s | 31 kB 00:00 2024-03-18T15:55:16.667 INFO:teuthology.orchestra.run.smithi039.stdout:(172/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 6.8 MB/s | 1.1 MB 00:00 2024-03-18T15:55:17.334 INFO:teuthology.orchestra.run.smithi039.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 94 kB/s | 74 kB 00:00 2024-03-18T15:55:17.415 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-18T15:55:17.492 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-18T15:55:17.492 INFO:teuthology.orchestra.run.smithi039.stdout:(174/174): util-linux-2.37.4-18.el9.x86_64.rpm 1.7 MB/s | 2.3 MB 00:01 2024-03-18T15:55:17.501 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2024-03-18T15:55:17.501 INFO:teuthology.orchestra.run.smithi039.stdout:Total 7.0 MB/s | 224 MB 00:32 2024-03-18T15:55:17.522 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-18T15:55:17.554 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-18T15:55:17.581 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-18T15:55:17.893 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-18T15:55:17.920 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 124/198 2024-03-18T15:55:18.010 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 124/198 2024-03-18T15:55:18.041 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 125/198 2024-03-18T15:55:18.077 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-18T15:55:18.536 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-18T15:55:18.556 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 128/198 2024-03-18T15:55:18.611 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 128/198 2024-03-18T15:55:18.641 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 129/198 2024-03-18T15:55:18.974 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2024-03-18T15:55:19.201 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/198 2024-03-18T15:55:19.202 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /sys 2024-03-18T15:55:19.202 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /proc 2024-03-18T15:55:19.202 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /mnt 2024-03-18T15:55:19.202 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /var/tmp 2024-03-18T15:55:19.202 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /home 2024-03-18T15:55:19.202 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /root 2024-03-18T15:55:19.202 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /tmp 2024-03-18T15:55:19.202 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-18T15:55:19.362 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 133/198 2024-03-18T15:55:19.371 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2024-03-18T15:55:19.371 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2024-03-18T15:55:19.378 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 133/198 2024-03-18T15:55:19.378 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T15:55:19.378 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-18T15:55:19.378 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-18T15:55:20.821 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-18T15:55:20.862 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-18T15:55:20.875 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 134/198 2024-03-18T15:55:20.900 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 134/198 2024-03-18T15:55:20.900 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T15:55:20.900 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-18T15:55:20.900 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-18T15:55:20.900 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-18T15:55:20.900 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-18T15:55:21.125 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/198 2024-03-18T15:55:21.148 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/198 2024-03-18T15:55:21.148 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T15:55:21.148 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-18T15:55:21.148 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-18T15:55:21.148 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-18T15:55:21.149 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-18T15:55:21.397 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-18T15:55:21.432 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/198 2024-03-18T15:55:21.496 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/198 2024-03-18T15:55:21.496 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-18T15:55:21.497 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-18T15:55:21.497 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-18T15:55:21.517 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/198 2024-03-18T15:55:21.584 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/198 2024-03-18T15:55:21.607 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/198 2024-03-18T15:55:21.607 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T15:55:21.607 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-18T15:55:21.607 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-18T15:55:21.607 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-18T15:55:21.607 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-18T15:55:21.695 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-18T15:55:21.898 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2024-03-18T15:55:21.900 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2024-03-18T15:55:22.079 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 138/198 2024-03-18T15:55:22.109 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 138/198 2024-03-18T15:55:22.141 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 139/198 2024-03-18T15:55:22.201 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 140/198 2024-03-18T15:55:22.224 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 141/198 2024-03-18T15:55:22.459 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 141/198 2024-03-18T15:55:22.491 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 142/198 2024-03-18T15:55:23.451 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 142/198 2024-03-18T15:55:23.475 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 143/198 2024-03-18T15:55:23.575 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 143/198 2024-03-18T15:55:23.740 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 144/198 2024-03-18T15:55:23.762 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 145/198 2024-03-18T15:55:23.785 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 145/198 2024-03-18T15:55:23.785 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T15:55:23.785 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-18T15:55:23.786 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-18T15:55:23.786 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-18T15:55:23.786 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-18T15:55:23.821 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 146/198 2024-03-18T15:55:23.832 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 146/198 2024-03-18T15:55:23.869 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 147/198 2024-03-18T15:55:24.738 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-18T15:55:24.765 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 149/198 2024-03-18T15:55:24.788 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 149/198 2024-03-18T15:55:24.788 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T15:55:24.788 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-18T15:55:24.788 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-18T15:55:24.788 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-18T15:55:24.788 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-18T15:55:24.832 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 150/198 2024-03-18T15:55:24.854 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 150/198 2024-03-18T15:55:24.854 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T15:55:24.854 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-18T15:55:24.854 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-18T15:55:24.899 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-18T15:55:24.922 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-18T15:55:24.922 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T15:55:24.922 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-18T15:55:24.922 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-18T15:55:24.923 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-18T15:55:24.923 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-18T15:55:25.234 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/198 2024-03-18T15:55:25.256 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/198 2024-03-18T15:55:25.256 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T15:55:25.256 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-18T15:55:25.256 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-18T15:55:25.256 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-18T15:55:25.256 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-18T15:55:29.515 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2024-03-18T15:55:29.611 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-18T15:55:29.687 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/198 2024-03-18T15:55:29.689 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-18T15:55:29.728 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-18T15:55:29.728 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 154/198 2024-03-18T15:55:29.759 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 155/198 2024-03-18T15:55:29.772 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-18T15:55:29.786 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-18T15:55:29.860 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-18T15:55:29.966 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-18T15:55:29.972 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 157/198 2024-03-18T15:55:30.054 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-18T15:55:30.099 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-18T15:55:30.135 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-18T15:55:30.175 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 159/198 2024-03-18T15:55:30.195 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-18T15:55:30.215 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-18T15:55:30.259 INFO:teuthology.orchestra.run.smithi194.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-18T15:55:30.302 INFO:teuthology.orchestra.run.smithi194.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-18T15:55:30.335 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 163/198 2024-03-18T15:55:30.358 INFO:teuthology.orchestra.run.smithi194.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-18T15:55:30.388 INFO:teuthology.orchestra.run.smithi194.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 165/198 2024-03-18T15:55:30.442 INFO:teuthology.orchestra.run.smithi194.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 166/198 2024-03-18T15:55:30.512 INFO:teuthology.orchestra.run.smithi194.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-18T15:55:30.544 INFO:teuthology.orchestra.run.smithi194.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-18T15:55:30.618 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-18T15:55:30.671 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-18T15:55:30.859 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-18T15:55:30.881 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-18T15:55:31.004 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-18T15:55:31.037 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/198 2024-03-18T15:55:31.101 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-18T15:55:31.167 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-18T15:55:31.209 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-18T15:55:31.229 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-18T15:55:31.426 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-18T15:55:31.513 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-18T15:55:31.546 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-18T15:55:31.684 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-18T15:55:31.714 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 17/198 2024-03-18T15:55:31.788 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 17/198 2024-03-18T15:55:31.847 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-18T15:55:31.899 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-18T15:55:31.961 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-18T15:55:32.049 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 170/198 2024-03-18T15:55:32.137 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-18T15:55:32.176 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-18T15:55:32.220 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-18T15:55:32.257 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-18T15:55:32.274 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-18T15:55:32.290 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-18T15:55:32.315 INFO:teuthology.orchestra.run.smithi194.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-18T15:55:32.317 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-18T15:55:32.340 INFO:teuthology.orchestra.run.smithi194.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-18T15:55:32.367 INFO:teuthology.orchestra.run.smithi194.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-18T15:55:32.367 INFO:teuthology.orchestra.run.smithi194.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-18T15:55:32.454 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-18T15:55:32.484 INFO:teuthology.orchestra.run.smithi194.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-18T15:55:32.484 INFO:teuthology.orchestra.run.smithi194.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-18T15:55:32.569 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-18T15:55:32.595 INFO:teuthology.orchestra.run.smithi194.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-18T15:55:32.636 INFO:teuthology.orchestra.run.smithi194.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-18T15:55:32.650 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-18T15:55:32.662 INFO:teuthology.orchestra.run.smithi194.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-18T15:55:32.687 INFO:teuthology.orchestra.run.smithi194.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-18T15:55:32.719 INFO:teuthology.orchestra.run.smithi194.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-18T15:55:32.759 INFO:teuthology.orchestra.run.smithi194.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-18T15:55:32.790 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-18T15:55:32.791 INFO:teuthology.orchestra.run.smithi194.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-18T15:55:32.816 INFO:teuthology.orchestra.run.smithi194.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-18T15:55:32.842 INFO:teuthology.orchestra.run.smithi194.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-18T15:55:32.867 INFO:teuthology.orchestra.run.smithi194.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-18T15:55:32.888 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-18T15:55:32.892 INFO:teuthology.orchestra.run.smithi194.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-18T15:55:32.899 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-18T15:55:32.899 INFO:teuthology.orchestra.run.smithi194.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-18T15:55:33.087 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-18T15:55:33.087 INFO:teuthology.orchestra.run.smithi194.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-18T15:55:33.092 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/198 2024-03-18T15:55:33.092 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /sys 2024-03-18T15:55:33.092 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /proc 2024-03-18T15:55:33.092 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /mnt 2024-03-18T15:55:33.092 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /var/tmp 2024-03-18T15:55:33.093 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /home 2024-03-18T15:55:33.093 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /root 2024-03-18T15:55:33.093 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /tmp 2024-03-18T15:55:33.093 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-18T15:55:33.146 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-18T15:55:33.178 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-18T15:55:33.193 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-18T15:55:33.202 INFO:teuthology.orchestra.run.smithi194.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-18T15:55:33.228 INFO:teuthology.orchestra.run.smithi194.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-18T15:55:33.258 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 133/198 2024-03-18T15:55:33.261 INFO:teuthology.orchestra.run.smithi194.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-18T15:55:33.273 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 133/198 2024-03-18T15:55:33.274 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T15:55:33.274 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-18T15:55:33.274 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-18T15:55:33.287 INFO:teuthology.orchestra.run.smithi194.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-18T15:55:33.352 INFO:teuthology.orchestra.run.smithi194.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-18T15:55:33.526 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-18T15:55:33.707 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-18T15:55:33.833 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-18T15:55:33.890 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-18T15:55:33.928 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-18T15:55:33.993 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-18T15:55:34.110 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-18T15:55:34.181 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-18T15:55:34.225 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-18T15:55:34.318 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-18T15:55:34.371 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-18T15:55:34.435 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-18T15:55:34.507 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-18T15:55:34.542 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-18T15:55:34.667 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-18T15:55:34.720 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 39/198 2024-03-18T15:55:34.744 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 134/198 2024-03-18T15:55:34.770 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 134/198 2024-03-18T15:55:34.770 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T15:55:34.770 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-18T15:55:34.770 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-18T15:55:34.773 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-18T15:55:34.774 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-18T15:55:34.783 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 40/198 2024-03-18T15:55:34.853 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 41/198 2024-03-18T15:55:34.988 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-18T15:55:34.995 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/198 2024-03-18T15:55:35.018 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/198 2024-03-18T15:55:35.018 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T15:55:35.018 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-18T15:55:35.018 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-18T15:55:35.018 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-18T15:55:35.018 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-18T15:55:35.056 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-18T15:55:35.161 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-18T15:55:35.357 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-18T15:55:35.425 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-18T15:55:35.462 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/198 2024-03-18T15:55:35.485 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/198 2024-03-18T15:55:35.485 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T15:55:35.485 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-18T15:55:35.485 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-18T15:55:35.485 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-18T15:55:35.485 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-18T15:55:35.572 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-18T15:55:35.949 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 138/198 2024-03-18T15:55:35.979 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 138/198 2024-03-18T15:55:36.011 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 139/198 2024-03-18T15:55:36.072 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 140/198 2024-03-18T15:55:36.103 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 141/198 2024-03-18T15:55:36.139 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-18T15:55:36.242 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-18T15:55:36.337 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 141/198 2024-03-18T15:55:36.357 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-18T15:55:36.370 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 142/198 2024-03-18T15:55:36.413 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-18T15:55:36.438 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-18T15:55:36.479 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-18T15:55:36.547 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-18T15:55:36.595 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 53/198 2024-03-18T15:55:36.636 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-18T15:55:37.062 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-18T15:55:37.291 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-18T15:55:37.332 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 142/198 2024-03-18T15:55:37.354 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 143/198 2024-03-18T15:55:37.376 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-18T15:55:37.452 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-18T15:55:37.454 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 143/198 2024-03-18T15:55:37.616 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 144/198 2024-03-18T15:55:37.647 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 145/198 2024-03-18T15:55:37.670 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 145/198 2024-03-18T15:55:37.670 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T15:55:37.670 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-18T15:55:37.670 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-18T15:55:37.670 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-18T15:55:37.671 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-18T15:55:37.671 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-18T15:55:37.715 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 146/198 2024-03-18T15:55:37.726 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 146/198 2024-03-18T15:55:37.771 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 147/198 2024-03-18T15:55:37.798 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-18T15:55:37.887 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-18T15:55:37.984 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-18T15:55:38.031 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-18T15:55:38.071 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-18T15:55:38.115 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-18T15:55:38.116 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 1/198 2024-03-18T15:55:38.116 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2/198 2024-03-18T15:55:38.116 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 3/198 2024-03-18T15:55:38.116 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 4/198 2024-03-18T15:55:38.116 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 5/198 2024-03-18T15:55:38.116 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 6/198 2024-03-18T15:55:38.116 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 7/198 2024-03-18T15:55:38.116 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 8/198 2024-03-18T15:55:38.116 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 9/198 2024-03-18T15:55:38.116 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-18T15:55:38.116 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-18T15:55:38.116 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/198 2024-03-18T15:55:38.116 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 13/198 2024-03-18T15:55:38.117 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 14/198 2024-03-18T15:55:38.117 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 15/198 2024-03-18T15:55:38.117 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-18T15:55:38.117 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 17/198 2024-03-18T15:55:38.117 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 18/198 2024-03-18T15:55:38.117 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 19/198 2024-03-18T15:55:38.117 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 20/198 2024-03-18T15:55:38.117 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 21/198 2024-03-18T15:55:38.117 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 22/198 2024-03-18T15:55:38.117 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 23/198 2024-03-18T15:55:38.117 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 24/198 2024-03-18T15:55:38.117 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 25/198 2024-03-18T15:55:38.117 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 26/198 2024-03-18T15:55:38.117 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 27/198 2024-03-18T15:55:38.118 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 28/198 2024-03-18T15:55:38.118 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 29/198 2024-03-18T15:55:38.118 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 30/198 2024-03-18T15:55:38.118 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 31/198 2024-03-18T15:55:38.118 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 32/198 2024-03-18T15:55:38.118 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 33/198 2024-03-18T15:55:38.118 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 34/198 2024-03-18T15:55:38.118 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 35/198 2024-03-18T15:55:38.118 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 36/198 2024-03-18T15:55:38.118 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 37/198 2024-03-18T15:55:38.118 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 38/198 2024-03-18T15:55:38.118 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 39/198 2024-03-18T15:55:38.118 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-18T15:55:38.118 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-18T15:55:38.118 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-18T15:55:38.119 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-18T15:55:38.119 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-18T15:55:38.119 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-18T15:55:38.119 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-18T15:55:38.119 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-18T15:55:38.119 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-18T15:55:38.119 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-18T15:55:38.119 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-18T15:55:38.119 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-18T15:55:38.119 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-18T15:55:38.119 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-18T15:55:38.119 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-18T15:55:38.120 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-18T15:55:38.120 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-18T15:55:38.120 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-18T15:55:38.120 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-18T15:55:38.120 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-18T15:55:38.120 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-18T15:55:38.120 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-18T15:55:38.120 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-18T15:55:38.120 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-18T15:55:38.121 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-18T15:55:38.121 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-18T15:55:38.121 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-18T15:55:38.121 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-18T15:55:38.121 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-18T15:55:38.121 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-18T15:55:38.121 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-18T15:55:38.121 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-18T15:55:38.121 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-18T15:55:38.121 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-18T15:55:38.121 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-18T15:55:38.121 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-18T15:55:38.121 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-18T15:55:38.121 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-18T15:55:38.121 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-18T15:55:38.122 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-18T15:55:38.122 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-18T15:55:38.122 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-18T15:55:38.122 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-18T15:55:38.122 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-18T15:55:38.122 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-18T15:55:38.122 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-18T15:55:38.122 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-18T15:55:38.122 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-18T15:55:38.122 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-18T15:55:38.122 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-18T15:55:38.122 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-18T15:55:38.122 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-18T15:55:38.123 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-18T15:55:38.123 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-18T15:55:38.123 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-18T15:55:38.123 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-18T15:55:38.123 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-18T15:55:38.123 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-18T15:55:38.123 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-18T15:55:38.123 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-18T15:55:38.123 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-18T15:55:38.123 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-18T15:55:38.123 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-18T15:55:38.123 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-18T15:55:38.123 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-18T15:55:38.123 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-18T15:55:38.123 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-18T15:55:38.124 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-18T15:55:38.124 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-18T15:55:38.124 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-18T15:55:38.124 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-18T15:55:38.124 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-18T15:55:38.124 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-18T15:55:38.124 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-18T15:55:38.124 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-18T15:55:38.124 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-18T15:55:38.124 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-18T15:55:38.124 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-18T15:55:38.124 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-18T15:55:38.124 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-18T15:55:38.125 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-18T15:55:38.125 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-18T15:55:38.125 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-18T15:55:38.125 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-18T15:55:38.125 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-18T15:55:38.125 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-18T15:55:38.125 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-18T15:55:38.125 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-18T15:55:38.125 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-18T15:55:38.125 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-18T15:55:38.125 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-18T15:55:38.125 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-18T15:55:38.125 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-18T15:55:38.125 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-18T15:55:38.126 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-18T15:55:38.126 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-18T15:55:38.126 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-18T15:55:38.126 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-18T15:55:38.126 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-18T15:55:38.126 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-18T15:55:38.126 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-18T15:55:38.126 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-18T15:55:38.126 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-18T15:55:38.126 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-18T15:55:38.126 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-18T15:55:38.126 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-18T15:55:38.126 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-18T15:55:38.126 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-18T15:55:38.126 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-18T15:55:38.127 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-18T15:55:38.127 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-18T15:55:38.127 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-18T15:55:38.127 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-18T15:55:38.127 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/198 2024-03-18T15:55:38.127 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-18T15:55:38.127 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-18T15:55:38.127 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-18T15:55:38.127 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/198 2024-03-18T15:55:38.127 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-18T15:55:38.127 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/198 2024-03-18T15:55:38.127 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-18T15:55:38.127 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/198 2024-03-18T15:55:38.127 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-18T15:55:38.127 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/198 2024-03-18T15:55:38.128 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-18T15:55:38.128 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-18T15:55:38.128 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-18T15:55:38.128 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-18T15:55:38.128 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-18T15:55:38.128 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-18T15:55:38.128 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-18T15:55:38.128 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-18T15:55:38.128 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-18T15:55:38.128 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-18T15:55:38.128 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-18T15:55:38.128 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-18T15:55:38.128 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-18T15:55:38.128 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-18T15:55:38.129 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-18T15:55:38.129 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-18T15:55:38.129 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-18T15:55:38.129 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-18T15:55:38.129 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-18T15:55:38.129 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-18T15:55:38.129 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-18T15:55:38.129 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-18T15:55:38.129 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-18T15:55:38.129 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-18T15:55:38.129 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-18T15:55:38.129 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-18T15:55:38.129 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-18T15:55:38.129 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-18T15:55:38.129 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-18T15:55:38.129 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-18T15:55:38.130 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-18T15:55:38.130 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-18T15:55:38.130 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-18T15:55:38.130 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-18T15:55:38.143 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-18T15:55:38.187 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-18T15:55:38.252 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-18T15:55:38.320 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-18T15:55:38.525 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-18T15:55:38.584 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-18T15:55:38.623 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-18T15:55:38.641 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-18T15:55:38.667 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 149/198 2024-03-18T15:55:38.689 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 149/198 2024-03-18T15:55:38.689 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T15:55:38.690 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-18T15:55:38.690 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-18T15:55:38.690 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-18T15:55:38.690 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-18T15:55:38.702 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-18T15:55:38.734 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 150/198 2024-03-18T15:55:38.756 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 150/198 2024-03-18T15:55:38.756 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T15:55:38.757 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-18T15:55:38.757 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-18T15:55:38.802 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-18T15:55:38.824 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-18T15:55:38.824 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T15:55:38.824 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-18T15:55:38.824 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-18T15:55:38.824 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-18T15:55:38.825 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-18T15:55:38.848 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-18T15:55:38.921 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-18T15:55:38.973 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-18T15:55:39.013 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-18T15:55:39.045 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-18T15:55:39.144 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/198 2024-03-18T15:55:39.167 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/198 2024-03-18T15:55:39.167 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T15:55:39.167 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-18T15:55:39.167 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-18T15:55:39.167 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-18T15:55:39.167 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-18T15:55:39.228 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-18T15:55:39.429 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-18T15:55:39.571 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-18T15:55:39.684 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-18T15:55:39.762 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-18T15:55:39.844 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-18T15:55:40.097 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-18T15:55:40.150 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-18T15:55:40.320 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-18T15:55:40.372 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-18T15:55:40.441 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-18T15:55:40.494 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-18T15:55:40.534 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-18T15:55:40.588 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-18T15:55:40.651 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-18T15:55:40.711 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-18T15:55:40.757 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-18T15:55:40.930 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-18T15:55:40.976 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-18T15:55:41.012 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-18T15:55:41.078 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-18T15:55:41.115 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-18T15:55:41.264 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-18T15:55:41.327 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-18T15:55:41.328 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-18T15:55:41.328 INFO:teuthology.orchestra.run.smithi194.stdout:Upgraded: 2024-03-18T15:55:41.328 INFO:teuthology.orchestra.run.smithi194.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-18T15:55:41.328 INFO:teuthology.orchestra.run.smithi194.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-18T15:55:41.328 INFO:teuthology.orchestra.run.smithi194.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-18T15:55:41.328 INFO:teuthology.orchestra.run.smithi194.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-18T15:55:41.328 INFO:teuthology.orchestra.run.smithi194.stdout: acl-2.3.1-4.el9.x86_64 2024-03-18T15:55:41.328 INFO:teuthology.orchestra.run.smithi194.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-18T15:55:41.328 INFO:teuthology.orchestra.run.smithi194.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-18T15:55:41.328 INFO:teuthology.orchestra.run.smithi194.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-18T15:55:41.328 INFO:teuthology.orchestra.run.smithi194.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-18T15:55:41.328 INFO:teuthology.orchestra.run.smithi194.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-18T15:55:41.329 INFO:teuthology.orchestra.run.smithi194.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-18T15:55:41.329 INFO:teuthology.orchestra.run.smithi194.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-18T15:55:41.329 INFO:teuthology.orchestra.run.smithi194.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-18T15:55:41.329 INFO:teuthology.orchestra.run.smithi194.stdout: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:55:41.329 INFO:teuthology.orchestra.run.smithi194.stdout: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:55:41.329 INFO:teuthology.orchestra.run.smithi194.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-18T15:55:41.329 INFO:teuthology.orchestra.run.smithi194.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-18T15:55:41.329 INFO:teuthology.orchestra.run.smithi194.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-18T15:55:41.329 INFO:teuthology.orchestra.run.smithi194.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-18T15:55:41.329 INFO:teuthology.orchestra.run.smithi194.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-18T15:55:41.329 INFO:teuthology.orchestra.run.smithi194.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-18T15:55:41.329 INFO:teuthology.orchestra.run.smithi194.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-18T15:55:41.329 INFO:teuthology.orchestra.run.smithi194.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-18T15:55:41.330 INFO:teuthology.orchestra.run.smithi194.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-18T15:55:41.330 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-03-18T15:55:41.330 INFO:teuthology.orchestra.run.smithi194.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-18T15:55:41.330 INFO:teuthology.orchestra.run.smithi194.stdout: automake-1.16.2-8.el9.noarch 2024-03-18T15:55:41.330 INFO:teuthology.orchestra.run.smithi194.stdout: bison-3.7.4-5.el9.x86_64 2024-03-18T15:55:41.330 INFO:teuthology.orchestra.run.smithi194.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-18T15:55:41.330 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:55:41.330 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:55:41.330 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:55:41.330 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:55:41.330 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:55:41.330 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:55:41.330 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:55:41.330 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:55:41.331 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:55:41.331 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:55:41.331 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:55:41.331 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:55:41.331 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:55:41.331 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:55:41.331 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:55:41.331 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:55:41.331 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:55:41.331 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:55:41.331 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:55:41.331 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:55:41.331 INFO:teuthology.orchestra.run.smithi194.stdout: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:55:41.331 INFO:teuthology.orchestra.run.smithi194.stdout: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:55:41.331 INFO:teuthology.orchestra.run.smithi194.stdout: cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:55:41.332 INFO:teuthology.orchestra.run.smithi194.stdout: cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:55:41.332 INFO:teuthology.orchestra.run.smithi194.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-18T15:55:41.332 INFO:teuthology.orchestra.run.smithi194.stdout: flex-2.6.4-9.el9.x86_64 2024-03-18T15:55:41.332 INFO:teuthology.orchestra.run.smithi194.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-18T15:55:41.332 INFO:teuthology.orchestra.run.smithi194.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-18T15:55:41.332 INFO:teuthology.orchestra.run.smithi194.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-18T15:55:41.332 INFO:teuthology.orchestra.run.smithi194.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-18T15:55:41.332 INFO:teuthology.orchestra.run.smithi194.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-18T15:55:41.332 INFO:teuthology.orchestra.run.smithi194.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-18T15:55:41.332 INFO:teuthology.orchestra.run.smithi194.stdout: jq-1.6-16.el9.x86_64 2024-03-18T15:55:41.332 INFO:teuthology.orchestra.run.smithi194.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-18T15:55:41.332 INFO:teuthology.orchestra.run.smithi194.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-18T15:55:41.332 INFO:teuthology.orchestra.run.smithi194.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-18T15:55:41.332 INFO:teuthology.orchestra.run.smithi194.stdout: libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:55:41.333 INFO:teuthology.orchestra.run.smithi194.stdout: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:55:41.333 INFO:teuthology.orchestra.run.smithi194.stdout: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:55:41.333 INFO:teuthology.orchestra.run.smithi194.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-18T15:55:41.333 INFO:teuthology.orchestra.run.smithi194.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-18T15:55:41.333 INFO:teuthology.orchestra.run.smithi194.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-18T15:55:41.333 INFO:teuthology.orchestra.run.smithi194.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-18T15:55:41.333 INFO:teuthology.orchestra.run.smithi194.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-18T15:55:41.333 INFO:teuthology.orchestra.run.smithi194.stdout: librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:55:41.333 INFO:teuthology.orchestra.run.smithi194.stdout: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:55:41.333 INFO:teuthology.orchestra.run.smithi194.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-18T15:55:41.333 INFO:teuthology.orchestra.run.smithi194.stdout: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:55:41.333 INFO:teuthology.orchestra.run.smithi194.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-18T15:55:41.333 INFO:teuthology.orchestra.run.smithi194.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-18T15:55:41.333 INFO:teuthology.orchestra.run.smithi194.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-18T15:55:41.334 INFO:teuthology.orchestra.run.smithi194.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-18T15:55:41.334 INFO:teuthology.orchestra.run.smithi194.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-18T15:55:41.334 INFO:teuthology.orchestra.run.smithi194.stdout: lua-5.4.4-4.el9.x86_64 2024-03-18T15:55:41.334 INFO:teuthology.orchestra.run.smithi194.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-18T15:55:41.334 INFO:teuthology.orchestra.run.smithi194.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-18T15:55:41.334 INFO:teuthology.orchestra.run.smithi194.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-18T15:55:41.334 INFO:teuthology.orchestra.run.smithi194.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-18T15:55:41.334 INFO:teuthology.orchestra.run.smithi194.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-18T15:55:41.334 INFO:teuthology.orchestra.run.smithi194.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-18T15:55:41.334 INFO:teuthology.orchestra.run.smithi194.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-18T15:55:41.334 INFO:teuthology.orchestra.run.smithi194.stdout: postgresql-13.11-1.el9.x86_64 2024-03-18T15:55:41.335 INFO:teuthology.orchestra.run.smithi194.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-18T15:55:41.335 INFO:teuthology.orchestra.run.smithi194.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-18T15:55:41.335 INFO:teuthology.orchestra.run.smithi194.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-18T15:55:41.335 INFO:teuthology.orchestra.run.smithi194.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-18T15:55:41.335 INFO:teuthology.orchestra.run.smithi194.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-18T15:55:41.335 INFO:teuthology.orchestra.run.smithi194.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-18T15:55:41.335 INFO:teuthology.orchestra.run.smithi194.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-18T15:55:41.335 INFO:teuthology.orchestra.run.smithi194.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-18T15:55:41.335 INFO:teuthology.orchestra.run.smithi194.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-18T15:55:41.335 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-18T15:55:41.335 INFO:teuthology.orchestra.run.smithi194.stdout: python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:55:41.335 INFO:teuthology.orchestra.run.smithi194.stdout: python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:55:41.335 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:55:41.335 INFO:teuthology.orchestra.run.smithi194.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-18T15:55:41.336 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-18T15:55:41.336 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-18T15:55:41.336 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-18T15:55:41.336 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-18T15:55:41.336 INFO:teuthology.orchestra.run.smithi194.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-18T15:55:41.336 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-18T15:55:41.336 INFO:teuthology.orchestra.run.smithi194.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-18T15:55:41.336 INFO:teuthology.orchestra.run.smithi194.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-18T15:55:41.336 INFO:teuthology.orchestra.run.smithi194.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-18T15:55:41.336 INFO:teuthology.orchestra.run.smithi194.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-18T15:55:41.336 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-18T15:55:41.336 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-18T15:55:41.336 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-18T15:55:41.336 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-18T15:55:41.337 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-18T15:55:41.337 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-18T15:55:41.337 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-18T15:55:41.337 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-18T15:55:41.337 INFO:teuthology.orchestra.run.smithi194.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-18T15:55:41.337 INFO:teuthology.orchestra.run.smithi194.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-18T15:55:41.337 INFO:teuthology.orchestra.run.smithi194.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-18T15:55:41.337 INFO:teuthology.orchestra.run.smithi194.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-18T15:55:41.337 INFO:teuthology.orchestra.run.smithi194.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-18T15:55:41.337 INFO:teuthology.orchestra.run.smithi194.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-18T15:55:41.337 INFO:teuthology.orchestra.run.smithi194.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-18T15:55:41.337 INFO:teuthology.orchestra.run.smithi194.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-18T15:55:41.337 INFO:teuthology.orchestra.run.smithi194.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-18T15:55:41.338 INFO:teuthology.orchestra.run.smithi194.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-18T15:55:41.338 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-18T15:55:41.338 INFO:teuthology.orchestra.run.smithi194.stdout: python3-ply-3.11-14.el9.noarch 2024-03-18T15:55:41.338 INFO:teuthology.orchestra.run.smithi194.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-18T15:55:41.338 INFO:teuthology.orchestra.run.smithi194.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-18T15:55:41.338 INFO:teuthology.orchestra.run.smithi194.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-18T15:55:41.338 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-18T15:55:41.338 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-18T15:55:41.338 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-18T15:55:41.338 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-18T15:55:41.338 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-18T15:55:41.338 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-18T15:55:41.338 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:55:41.338 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:55:41.339 INFO:teuthology.orchestra.run.smithi194.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-18T15:55:41.339 INFO:teuthology.orchestra.run.smithi194.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-18T15:55:41.339 INFO:teuthology.orchestra.run.smithi194.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-18T15:55:41.339 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:55:41.339 INFO:teuthology.orchestra.run.smithi194.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-18T15:55:41.339 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-18T15:55:41.339 INFO:teuthology.orchestra.run.smithi194.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-18T15:55:41.339 INFO:teuthology.orchestra.run.smithi194.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-18T15:55:41.340 INFO:teuthology.orchestra.run.smithi194.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-18T15:55:41.340 INFO:teuthology.orchestra.run.smithi194.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-18T15:55:41.340 INFO:teuthology.orchestra.run.smithi194.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-18T15:55:41.340 INFO:teuthology.orchestra.run.smithi194.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-18T15:55:41.340 INFO:teuthology.orchestra.run.smithi194.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-18T15:55:41.340 INFO:teuthology.orchestra.run.smithi194.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-18T15:55:41.340 INFO:teuthology.orchestra.run.smithi194.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-18T15:55:41.340 INFO:teuthology.orchestra.run.smithi194.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-18T15:55:41.341 INFO:teuthology.orchestra.run.smithi194.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-18T15:55:41.341 INFO:teuthology.orchestra.run.smithi194.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-18T15:55:41.341 INFO:teuthology.orchestra.run.smithi194.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-18T15:55:41.341 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:55:41.341 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:55:41.341 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:55:41.341 INFO:teuthology.orchestra.run.smithi194.stdout: re2-1:20211101-3.el9.x86_64 2024-03-18T15:55:41.341 INFO:teuthology.orchestra.run.smithi194.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-18T15:55:41.341 INFO:teuthology.orchestra.run.smithi194.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-18T15:55:41.341 INFO:teuthology.orchestra.run.smithi194.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-18T15:55:41.341 INFO:teuthology.orchestra.run.smithi194.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-18T15:55:41.341 INFO:teuthology.orchestra.run.smithi194.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-18T15:55:41.342 INFO:teuthology.orchestra.run.smithi194.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-18T15:55:41.342 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-03-18T15:55:41.342 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-03-18T15:55:41.387 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-18T15:55:41.444 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-18T15:55:41.479 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-18T15:55:41.538 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-18T15:55:41.693 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-18T15:55:41.780 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-18T15:55:41.807 DEBUG:teuthology.parallel:result is None 2024-03-18T15:55:42.787 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-18T15:55:42.904 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-18T15:55:42.942 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-18T15:55:43.020 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-18T15:55:43.166 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-18T15:55:43.221 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-18T15:55:43.482 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-18T15:55:43.525 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-18T15:55:43.590 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/198 2024-03-18T15:55:43.626 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-18T15:55:43.631 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 154/198 2024-03-18T15:55:43.661 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 155/198 2024-03-18T15:55:43.688 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-18T15:55:43.869 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 157/198 2024-03-18T15:55:43.996 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-18T15:55:44.037 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 159/198 2024-03-18T15:55:44.076 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-18T15:55:44.128 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-18T15:55:44.172 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-18T15:55:44.204 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 163/198 2024-03-18T15:55:44.256 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-18T15:55:44.291 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 165/198 2024-03-18T15:55:44.346 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 166/198 2024-03-18T15:55:44.407 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-18T15:55:44.440 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-18T15:55:44.493 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-18T15:55:44.634 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-18T15:55:45.016 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-18T15:55:45.054 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-18T15:55:45.075 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-18T15:55:45.896 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 170/198 2024-03-18T15:55:45.983 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-18T15:55:46.024 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-18T15:55:46.105 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-18T15:55:46.162 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-18T15:55:46.187 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-18T15:55:46.212 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-18T15:55:46.239 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-18T15:55:46.239 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-18T15:55:46.247 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-18T15:55:46.335 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-18T15:55:46.337 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-18T15:55:46.364 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-18T15:55:46.364 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-18T15:55:46.400 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-18T15:55:46.432 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-18T15:55:46.449 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-18T15:55:46.472 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-18T15:55:46.475 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-18T15:55:46.508 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-18T15:55:46.534 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-18T15:55:46.559 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-18T15:55:46.591 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-18T15:55:46.631 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-18T15:55:46.672 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-18T15:55:46.697 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-18T15:55:46.723 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-18T15:55:46.748 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-18T15:55:46.773 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-18T15:55:46.780 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-18T15:55:46.780 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-18T15:55:46.869 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-18T15:55:46.903 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 124/198 2024-03-18T15:55:46.959 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-18T15:55:46.959 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-18T15:55:47.024 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 124/198 2024-03-18T15:55:47.051 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-18T15:55:47.059 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 125/198 2024-03-18T15:55:47.075 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-18T15:55:47.101 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-18T15:55:47.134 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-18T15:55:47.148 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-18T15:55:47.160 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-18T15:55:47.224 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-18T15:55:47.703 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-18T15:55:47.730 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 128/198 2024-03-18T15:55:47.831 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 128/198 2024-03-18T15:55:47.883 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 129/198 2024-03-18T15:55:50.654 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-18T15:55:50.687 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-18T15:55:51.353 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 130/198 2024-03-18T15:55:51.399 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/198 2024-03-18T15:55:51.556 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 131/198 2024-03-18T15:55:51.556 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-18T15:55:51.556 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-18T15:55:51.556 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T15:55:51.642 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/198 2024-03-18T15:55:52.107 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-18T15:55:52.107 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 1/198 2024-03-18T15:55:52.107 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2/198 2024-03-18T15:55:52.107 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 3/198 2024-03-18T15:55:52.108 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 4/198 2024-03-18T15:55:52.108 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 5/198 2024-03-18T15:55:52.108 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 6/198 2024-03-18T15:55:52.108 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 7/198 2024-03-18T15:55:52.108 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 8/198 2024-03-18T15:55:52.108 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 9/198 2024-03-18T15:55:52.108 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-18T15:55:52.108 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-18T15:55:52.108 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/198 2024-03-18T15:55:52.108 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 13/198 2024-03-18T15:55:52.108 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 14/198 2024-03-18T15:55:52.108 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 15/198 2024-03-18T15:55:52.108 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-18T15:55:52.109 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 17/198 2024-03-18T15:55:52.109 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 18/198 2024-03-18T15:55:52.109 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 19/198 2024-03-18T15:55:52.109 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 20/198 2024-03-18T15:55:52.109 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 21/198 2024-03-18T15:55:52.109 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 22/198 2024-03-18T15:55:52.109 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 23/198 2024-03-18T15:55:52.109 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 24/198 2024-03-18T15:55:52.109 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 25/198 2024-03-18T15:55:52.109 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 26/198 2024-03-18T15:55:52.109 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 27/198 2024-03-18T15:55:52.109 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 28/198 2024-03-18T15:55:52.109 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 29/198 2024-03-18T15:55:52.109 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 30/198 2024-03-18T15:55:52.109 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 31/198 2024-03-18T15:55:52.110 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 32/198 2024-03-18T15:55:52.110 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 33/198 2024-03-18T15:55:52.110 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 34/198 2024-03-18T15:55:52.110 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 35/198 2024-03-18T15:55:52.110 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 36/198 2024-03-18T15:55:52.110 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 37/198 2024-03-18T15:55:52.110 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 38/198 2024-03-18T15:55:52.110 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 39/198 2024-03-18T15:55:52.110 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-18T15:55:52.110 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-18T15:55:52.110 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-18T15:55:52.110 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-18T15:55:52.110 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-18T15:55:52.110 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-18T15:55:52.111 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-18T15:55:52.111 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-18T15:55:52.111 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-18T15:55:52.111 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-18T15:55:52.111 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-18T15:55:52.111 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-18T15:55:52.111 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-18T15:55:52.111 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-18T15:55:52.111 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-18T15:55:52.111 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-18T15:55:52.111 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-18T15:55:52.111 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-18T15:55:52.111 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-18T15:55:52.111 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-18T15:55:52.111 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-18T15:55:52.112 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-18T15:55:52.112 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-18T15:55:52.112 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-18T15:55:52.112 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-18T15:55:52.112 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-18T15:55:52.112 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-18T15:55:52.112 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-18T15:55:52.112 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-18T15:55:52.112 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-18T15:55:52.112 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-18T15:55:52.112 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-18T15:55:52.112 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-18T15:55:52.112 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-18T15:55:52.113 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-18T15:55:52.113 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-18T15:55:52.113 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-18T15:55:52.113 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-18T15:55:52.113 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-18T15:55:52.113 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-18T15:55:52.113 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-18T15:55:52.113 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-18T15:55:52.113 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-18T15:55:52.113 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-18T15:55:52.113 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-18T15:55:52.113 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-18T15:55:52.113 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-18T15:55:52.113 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-18T15:55:52.113 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-18T15:55:52.114 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-18T15:55:52.114 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-18T15:55:52.114 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-18T15:55:52.114 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-18T15:55:52.114 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-18T15:55:52.114 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-18T15:55:52.114 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-18T15:55:52.114 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-18T15:55:52.114 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-18T15:55:52.114 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-18T15:55:52.114 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-18T15:55:52.114 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-18T15:55:52.114 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-18T15:55:52.114 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-18T15:55:52.115 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-18T15:55:52.115 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-18T15:55:52.115 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-18T15:55:52.115 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-18T15:55:52.115 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-18T15:55:52.115 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-18T15:55:52.115 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-18T15:55:52.115 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-18T15:55:52.115 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-18T15:55:52.115 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-18T15:55:52.115 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-18T15:55:52.115 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-18T15:55:52.115 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-18T15:55:52.115 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-18T15:55:52.115 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-18T15:55:52.116 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-18T15:55:52.116 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-18T15:55:52.116 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-18T15:55:52.116 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-18T15:55:52.116 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-18T15:55:52.116 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-18T15:55:52.116 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-18T15:55:52.116 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-18T15:55:52.116 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-18T15:55:52.116 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-18T15:55:52.116 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-18T15:55:52.116 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-18T15:55:52.116 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-18T15:55:52.116 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-18T15:55:52.117 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-18T15:55:52.117 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-18T15:55:52.117 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-18T15:55:52.117 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-18T15:55:52.117 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-18T15:55:52.117 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-18T15:55:52.117 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-18T15:55:52.117 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-18T15:55:52.117 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-18T15:55:52.117 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-18T15:55:52.117 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-18T15:55:52.117 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-18T15:55:52.117 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-18T15:55:52.117 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-18T15:55:52.118 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-18T15:55:52.118 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-18T15:55:52.118 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-18T15:55:52.118 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-18T15:55:52.118 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-18T15:55:52.118 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-18T15:55:52.118 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-18T15:55:52.118 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/198 2024-03-18T15:55:52.118 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-18T15:55:52.118 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-18T15:55:52.118 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-18T15:55:52.118 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/198 2024-03-18T15:55:52.118 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-18T15:55:52.118 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/198 2024-03-18T15:55:52.119 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-18T15:55:52.119 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/198 2024-03-18T15:55:52.119 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-18T15:55:52.119 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/198 2024-03-18T15:55:52.119 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-18T15:55:52.119 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-18T15:55:52.119 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-18T15:55:52.119 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-18T15:55:52.119 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-18T15:55:52.119 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-18T15:55:52.119 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-18T15:55:52.119 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-18T15:55:52.120 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-18T15:55:52.120 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-18T15:55:52.120 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-18T15:55:52.120 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-18T15:55:52.120 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-18T15:55:52.120 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-18T15:55:52.120 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-18T15:55:52.120 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-18T15:55:52.120 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-18T15:55:52.120 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-18T15:55:52.120 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-18T15:55:52.121 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-18T15:55:52.121 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-18T15:55:52.121 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-18T15:55:52.121 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-18T15:55:52.121 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-18T15:55:52.121 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-18T15:55:52.121 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-18T15:55:52.121 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-18T15:55:52.121 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-18T15:55:52.121 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-18T15:55:52.121 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-18T15:55:52.121 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-18T15:55:52.121 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-18T15:55:52.121 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-18T15:55:52.121 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-18T15:55:55.285 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-18T15:55:55.285 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-18T15:55:55.285 INFO:teuthology.orchestra.run.smithi196.stdout:Upgraded: 2024-03-18T15:55:55.285 INFO:teuthology.orchestra.run.smithi196.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-18T15:55:55.285 INFO:teuthology.orchestra.run.smithi196.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-18T15:55:55.285 INFO:teuthology.orchestra.run.smithi196.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-18T15:55:55.285 INFO:teuthology.orchestra.run.smithi196.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-18T15:55:55.285 INFO:teuthology.orchestra.run.smithi196.stdout: acl-2.3.1-4.el9.x86_64 2024-03-18T15:55:55.285 INFO:teuthology.orchestra.run.smithi196.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-18T15:55:55.285 INFO:teuthology.orchestra.run.smithi196.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-18T15:55:55.285 INFO:teuthology.orchestra.run.smithi196.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-18T15:55:55.286 INFO:teuthology.orchestra.run.smithi196.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-18T15:55:55.286 INFO:teuthology.orchestra.run.smithi196.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-18T15:55:55.286 INFO:teuthology.orchestra.run.smithi196.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-18T15:55:55.286 INFO:teuthology.orchestra.run.smithi196.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-18T15:55:55.286 INFO:teuthology.orchestra.run.smithi196.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-18T15:55:55.286 INFO:teuthology.orchestra.run.smithi196.stdout: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:55:55.286 INFO:teuthology.orchestra.run.smithi196.stdout: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:55:55.286 INFO:teuthology.orchestra.run.smithi196.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-18T15:55:55.286 INFO:teuthology.orchestra.run.smithi196.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-18T15:55:55.286 INFO:teuthology.orchestra.run.smithi196.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-18T15:55:55.286 INFO:teuthology.orchestra.run.smithi196.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-18T15:55:55.286 INFO:teuthology.orchestra.run.smithi196.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-18T15:55:55.286 INFO:teuthology.orchestra.run.smithi196.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-18T15:55:55.286 INFO:teuthology.orchestra.run.smithi196.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-18T15:55:55.287 INFO:teuthology.orchestra.run.smithi196.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-18T15:55:55.287 INFO:teuthology.orchestra.run.smithi196.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-18T15:55:55.287 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-18T15:55:55.287 INFO:teuthology.orchestra.run.smithi196.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-18T15:55:55.287 INFO:teuthology.orchestra.run.smithi196.stdout: automake-1.16.2-8.el9.noarch 2024-03-18T15:55:55.287 INFO:teuthology.orchestra.run.smithi196.stdout: bison-3.7.4-5.el9.x86_64 2024-03-18T15:55:55.287 INFO:teuthology.orchestra.run.smithi196.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-18T15:55:55.287 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:55:55.287 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:55:55.287 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:55:55.287 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:55:55.287 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:55:55.287 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:55:55.287 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:55:55.288 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:55:55.288 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:55:55.288 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:55:55.288 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:55:55.288 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:55:55.288 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:55:55.288 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:55:55.288 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:55:55.288 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:55:55.288 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:55:55.288 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:55:55.288 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:55:55.288 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:55:55.288 INFO:teuthology.orchestra.run.smithi196.stdout: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:55:55.288 INFO:teuthology.orchestra.run.smithi196.stdout: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:55:55.289 INFO:teuthology.orchestra.run.smithi196.stdout: cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:55:55.289 INFO:teuthology.orchestra.run.smithi196.stdout: cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:55:55.289 INFO:teuthology.orchestra.run.smithi196.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-18T15:55:55.289 INFO:teuthology.orchestra.run.smithi196.stdout: flex-2.6.4-9.el9.x86_64 2024-03-18T15:55:55.289 INFO:teuthology.orchestra.run.smithi196.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-18T15:55:55.289 INFO:teuthology.orchestra.run.smithi196.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-18T15:55:55.289 INFO:teuthology.orchestra.run.smithi196.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-18T15:55:55.289 INFO:teuthology.orchestra.run.smithi196.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-18T15:55:55.289 INFO:teuthology.orchestra.run.smithi196.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-18T15:55:55.289 INFO:teuthology.orchestra.run.smithi196.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-18T15:55:55.289 INFO:teuthology.orchestra.run.smithi196.stdout: jq-1.6-16.el9.x86_64 2024-03-18T15:55:55.289 INFO:teuthology.orchestra.run.smithi196.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-18T15:55:55.289 INFO:teuthology.orchestra.run.smithi196.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-18T15:55:55.289 INFO:teuthology.orchestra.run.smithi196.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-18T15:55:55.289 INFO:teuthology.orchestra.run.smithi196.stdout: libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:55:55.290 INFO:teuthology.orchestra.run.smithi196.stdout: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:55:55.290 INFO:teuthology.orchestra.run.smithi196.stdout: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:55:55.290 INFO:teuthology.orchestra.run.smithi196.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-18T15:55:55.290 INFO:teuthology.orchestra.run.smithi196.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-18T15:55:55.290 INFO:teuthology.orchestra.run.smithi196.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-18T15:55:55.290 INFO:teuthology.orchestra.run.smithi196.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-18T15:55:55.290 INFO:teuthology.orchestra.run.smithi196.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-18T15:55:55.290 INFO:teuthology.orchestra.run.smithi196.stdout: librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:55:55.290 INFO:teuthology.orchestra.run.smithi196.stdout: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:55:55.290 INFO:teuthology.orchestra.run.smithi196.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-18T15:55:55.290 INFO:teuthology.orchestra.run.smithi196.stdout: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:55:55.290 INFO:teuthology.orchestra.run.smithi196.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-18T15:55:55.290 INFO:teuthology.orchestra.run.smithi196.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-18T15:55:55.290 INFO:teuthology.orchestra.run.smithi196.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-18T15:55:55.291 INFO:teuthology.orchestra.run.smithi196.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-18T15:55:55.291 INFO:teuthology.orchestra.run.smithi196.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-18T15:55:55.291 INFO:teuthology.orchestra.run.smithi196.stdout: lua-5.4.4-4.el9.x86_64 2024-03-18T15:55:55.291 INFO:teuthology.orchestra.run.smithi196.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-18T15:55:55.291 INFO:teuthology.orchestra.run.smithi196.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-18T15:55:55.291 INFO:teuthology.orchestra.run.smithi196.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-18T15:55:55.291 INFO:teuthology.orchestra.run.smithi196.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-18T15:55:55.291 INFO:teuthology.orchestra.run.smithi196.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-18T15:55:55.291 INFO:teuthology.orchestra.run.smithi196.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-18T15:55:55.291 INFO:teuthology.orchestra.run.smithi196.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-18T15:55:55.291 INFO:teuthology.orchestra.run.smithi196.stdout: postgresql-13.11-1.el9.x86_64 2024-03-18T15:55:55.291 INFO:teuthology.orchestra.run.smithi196.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-18T15:55:55.291 INFO:teuthology.orchestra.run.smithi196.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-18T15:55:55.291 INFO:teuthology.orchestra.run.smithi196.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-18T15:55:55.291 INFO:teuthology.orchestra.run.smithi196.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-18T15:55:55.291 INFO:teuthology.orchestra.run.smithi196.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-18T15:55:55.292 INFO:teuthology.orchestra.run.smithi196.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-18T15:55:55.292 INFO:teuthology.orchestra.run.smithi196.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-18T15:55:55.292 INFO:teuthology.orchestra.run.smithi196.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-18T15:55:55.292 INFO:teuthology.orchestra.run.smithi196.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-18T15:55:55.292 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-18T15:55:55.292 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:55:55.292 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:55:55.292 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:55:55.292 INFO:teuthology.orchestra.run.smithi196.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-18T15:55:55.292 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-18T15:55:55.292 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-18T15:55:55.292 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-18T15:55:55.292 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-18T15:55:55.292 INFO:teuthology.orchestra.run.smithi196.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-18T15:55:55.292 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-18T15:55:55.293 INFO:teuthology.orchestra.run.smithi196.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-18T15:55:55.293 INFO:teuthology.orchestra.run.smithi196.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-18T15:55:55.293 INFO:teuthology.orchestra.run.smithi196.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-18T15:55:55.293 INFO:teuthology.orchestra.run.smithi196.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-18T15:55:55.293 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-18T15:55:55.293 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-18T15:55:55.293 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-18T15:55:55.293 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-18T15:55:55.293 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-18T15:55:55.293 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-18T15:55:55.293 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-18T15:55:55.293 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-18T15:55:55.293 INFO:teuthology.orchestra.run.smithi196.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-18T15:55:55.293 INFO:teuthology.orchestra.run.smithi196.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-18T15:55:55.293 INFO:teuthology.orchestra.run.smithi196.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-18T15:55:55.294 INFO:teuthology.orchestra.run.smithi196.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-18T15:55:55.294 INFO:teuthology.orchestra.run.smithi196.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-18T15:55:55.294 INFO:teuthology.orchestra.run.smithi196.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-18T15:55:55.294 INFO:teuthology.orchestra.run.smithi196.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-18T15:55:55.294 INFO:teuthology.orchestra.run.smithi196.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-18T15:55:55.294 INFO:teuthology.orchestra.run.smithi196.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-18T15:55:55.294 INFO:teuthology.orchestra.run.smithi196.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-18T15:55:55.294 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-18T15:55:55.294 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ply-3.11-14.el9.noarch 2024-03-18T15:55:55.294 INFO:teuthology.orchestra.run.smithi196.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-18T15:55:55.294 INFO:teuthology.orchestra.run.smithi196.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-18T15:55:55.294 INFO:teuthology.orchestra.run.smithi196.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-18T15:55:55.294 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-18T15:55:55.295 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-18T15:55:55.295 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-18T15:55:55.295 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-18T15:55:55.295 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-18T15:55:55.295 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-18T15:55:55.295 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:55:55.295 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:55:55.295 INFO:teuthology.orchestra.run.smithi196.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-18T15:55:55.295 INFO:teuthology.orchestra.run.smithi196.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-18T15:55:55.295 INFO:teuthology.orchestra.run.smithi196.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-18T15:55:55.295 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:55:55.295 INFO:teuthology.orchestra.run.smithi196.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-18T15:55:55.295 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-18T15:55:55.295 INFO:teuthology.orchestra.run.smithi196.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-18T15:55:55.295 INFO:teuthology.orchestra.run.smithi196.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-18T15:55:55.295 INFO:teuthology.orchestra.run.smithi196.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-18T15:55:55.296 INFO:teuthology.orchestra.run.smithi196.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-18T15:55:55.296 INFO:teuthology.orchestra.run.smithi196.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-18T15:55:55.296 INFO:teuthology.orchestra.run.smithi196.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-18T15:55:55.296 INFO:teuthology.orchestra.run.smithi196.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-18T15:55:55.296 INFO:teuthology.orchestra.run.smithi196.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-18T15:55:55.296 INFO:teuthology.orchestra.run.smithi196.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-18T15:55:55.296 INFO:teuthology.orchestra.run.smithi196.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-18T15:55:55.296 INFO:teuthology.orchestra.run.smithi196.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-18T15:55:55.296 INFO:teuthology.orchestra.run.smithi196.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-18T15:55:55.296 INFO:teuthology.orchestra.run.smithi196.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-18T15:55:55.296 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:55:55.296 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:55:55.296 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:55:55.296 INFO:teuthology.orchestra.run.smithi196.stdout: re2-1:20211101-3.el9.x86_64 2024-03-18T15:55:55.296 INFO:teuthology.orchestra.run.smithi196.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-18T15:55:55.297 INFO:teuthology.orchestra.run.smithi196.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-18T15:55:55.297 INFO:teuthology.orchestra.run.smithi196.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-18T15:55:55.297 INFO:teuthology.orchestra.run.smithi196.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-18T15:55:55.297 INFO:teuthology.orchestra.run.smithi196.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-18T15:55:55.297 INFO:teuthology.orchestra.run.smithi196.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-18T15:55:55.297 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-18T15:55:55.297 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-18T15:55:55.747 DEBUG:teuthology.parallel:result is None 2024-03-18T15:56:04.947 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 132/198 2024-03-18T15:56:04.947 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /sys 2024-03-18T15:56:04.947 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /proc 2024-03-18T15:56:04.947 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /mnt 2024-03-18T15:56:04.947 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /var/tmp 2024-03-18T15:56:04.947 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /home 2024-03-18T15:56:04.947 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /root 2024-03-18T15:56:04.947 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /tmp 2024-03-18T15:56:04.947 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T15:56:05.215 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 133/198 2024-03-18T15:56:05.262 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 133/198 2024-03-18T15:56:05.262 INFO:teuthology.orchestra.run.smithi039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T15:56:05.262 INFO:teuthology.orchestra.run.smithi039.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-18T15:56:05.262 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T15:56:06.933 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 134/198 2024-03-18T15:56:07.017 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 134/198 2024-03-18T15:56:07.017 INFO:teuthology.orchestra.run.smithi039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T15:56:07.018 INFO:teuthology.orchestra.run.smithi039.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-18T15:56:07.018 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-18T15:56:07.018 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-18T15:56:07.018 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T15:56:07.275 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/198 2024-03-18T15:56:07.349 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 135/198 2024-03-18T15:56:07.349 INFO:teuthology.orchestra.run.smithi039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T15:56:07.349 INFO:teuthology.orchestra.run.smithi039.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-18T15:56:07.349 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-18T15:56:07.349 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-18T15:56:07.349 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T15:56:07.859 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/198 2024-03-18T15:56:07.935 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 136/198 2024-03-18T15:56:07.935 INFO:teuthology.orchestra.run.smithi039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T15:56:07.935 INFO:teuthology.orchestra.run.smithi039.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-18T15:56:07.935 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-18T15:56:07.935 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-18T15:56:07.935 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T15:56:08.052 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-18T15:56:08.686 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 138/198 2024-03-18T15:56:08.720 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 138/198 2024-03-18T15:56:08.772 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 139/198 2024-03-18T15:56:08.876 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 140/198 2024-03-18T15:56:08.924 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 141/198 2024-03-18T15:56:09.274 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 141/198 2024-03-18T15:56:09.350 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 142/198 2024-03-18T15:56:10.538 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 142/198 2024-03-18T15:56:10.575 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 143/198 2024-03-18T15:56:10.726 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 143/198 2024-03-18T15:56:10.939 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 144/198 2024-03-18T15:56:10.969 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 145/198 2024-03-18T15:56:11.043 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 145/198 2024-03-18T15:56:11.043 INFO:teuthology.orchestra.run.smithi039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T15:56:11.043 INFO:teuthology.orchestra.run.smithi039.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-18T15:56:11.043 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-18T15:56:11.043 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-18T15:56:11.044 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T15:56:11.103 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 146/198 2024-03-18T15:56:11.142 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 146/198 2024-03-18T15:56:11.239 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 147/198 2024-03-18T15:56:12.254 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-18T15:56:12.297 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 149/198 2024-03-18T15:56:12.371 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 149/198 2024-03-18T15:56:12.371 INFO:teuthology.orchestra.run.smithi039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T15:56:12.371 INFO:teuthology.orchestra.run.smithi039.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-18T15:56:12.372 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-18T15:56:12.372 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-18T15:56:12.372 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T15:56:12.422 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 150/198 2024-03-18T15:56:12.502 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 150/198 2024-03-18T15:56:12.502 INFO:teuthology.orchestra.run.smithi039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T15:56:12.503 INFO:teuthology.orchestra.run.smithi039.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-18T15:56:12.503 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T15:56:12.573 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-18T15:56:12.652 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-18T15:56:12.652 INFO:teuthology.orchestra.run.smithi039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T15:56:12.652 INFO:teuthology.orchestra.run.smithi039.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-18T15:56:12.652 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-18T15:56:12.652 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-18T15:56:12.652 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T15:56:13.057 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/198 2024-03-18T15:56:13.149 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 152/198 2024-03-18T15:56:13.149 INFO:teuthology.orchestra.run.smithi039.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-18T15:56:13.149 INFO:teuthology.orchestra.run.smithi039.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-18T15:56:13.149 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-18T15:56:13.149 INFO:teuthology.orchestra.run.smithi039.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-18T15:56:13.149 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T15:56:18.264 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/198 2024-03-18T15:56:18.328 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 154/198 2024-03-18T15:56:18.371 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 155/198 2024-03-18T15:56:18.411 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-18T15:56:18.705 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 157/198 2024-03-18T15:56:18.839 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-18T15:56:18.922 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 159/198 2024-03-18T15:56:18.988 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-18T15:56:19.056 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-18T15:56:19.110 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-18T15:56:19.153 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 163/198 2024-03-18T15:56:19.196 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-18T15:56:19.249 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 165/198 2024-03-18T15:56:19.352 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 166/198 2024-03-18T15:56:19.473 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-18T15:56:19.512 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-18T15:56:20.465 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-18T15:56:20.511 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-18T15:56:20.570 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-18T15:56:22.022 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 170/198 2024-03-18T15:56:22.108 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-18T15:56:22.167 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-18T15:56:22.297 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-18T15:56:22.344 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-18T15:56:22.377 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-18T15:56:22.412 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-18T15:56:22.454 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-18T15:56:22.454 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-18T15:56:22.626 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-18T15:56:22.674 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-18T15:56:22.674 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-18T15:56:22.812 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-18T15:56:22.856 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-18T15:56:22.910 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-18T15:56:22.952 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-18T15:56:23.009 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-18T15:56:23.081 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-18T15:56:23.158 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-18T15:56:23.206 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-18T15:56:23.239 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-18T15:56:23.272 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-18T15:56:23.306 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-18T15:56:23.363 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-18T15:56:23.400 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-18T15:56:23.400 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-18T15:56:23.757 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-18T15:56:23.757 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-18T15:56:23.904 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-18T15:56:23.934 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-18T15:56:23.967 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-18T15:56:24.025 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-18T15:56:24.069 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-18T15:56:24.212 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-18T15:56:24.472 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-18T15:56:24.473 INFO:teuthology.orchestra.run.smithi039.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-03-18T15:56:24.473 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T15:56:29.353 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 1/198 2024-03-18T15:56:29.353 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2/198 2024-03-18T15:56:29.353 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 3/198 2024-03-18T15:56:29.353 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 4/198 2024-03-18T15:56:29.353 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2162.g3d78f 5/198 2024-03-18T15:56:29.353 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 6/198 2024-03-18T15:56:29.353 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 7/198 2024-03-18T15:56:29.354 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 8/198 2024-03-18T15:56:29.354 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 9/198 2024-03-18T15:56:29.354 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 10/198 2024-03-18T15:56:29.354 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 11/198 2024-03-18T15:56:29.354 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 12/198 2024-03-18T15:56:29.354 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 13/198 2024-03-18T15:56:29.354 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 14/198 2024-03-18T15:56:29.354 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_ 15/198 2024-03-18T15:56:29.354 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 16/198 2024-03-18T15:56:29.354 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 17/198 2024-03-18T15:56:29.354 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_6 18/198 2024-03-18T15:56:29.354 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86 19/198 2024-03-18T15:56:29.354 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 20/198 2024-03-18T15:56:29.354 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el 21/198 2024-03-18T15:56:29.355 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9. 22/198 2024-03-18T15:56:29.355 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_6 23/198 2024-03-18T15:56:29.355 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 24/198 2024-03-18T15:56:29.355 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 25/198 2024-03-18T15:56:29.355 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 26/198 2024-03-18T15:56:29.355 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 27/198 2024-03-18T15:56:29.370 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 28/198 2024-03-18T15:56:29.371 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 29/198 2024-03-18T15:56:29.371 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba. 30/198 2024-03-18T15:56:29.371 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noa 31/198 2024-03-18T15:56:29.371 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.n 32/198 2024-03-18T15:56:29.371 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d7 33/198 2024-03-18T15:56:29.371 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el 34/198 2024-03-18T15:56:29.371 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 35/198 2024-03-18T15:56:29.371 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.e 36/198 2024-03-18T15:56:29.371 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 37/198 2024-03-18T15:56:29.371 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 38/198 2024-03-18T15:56:29.371 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 39/198 2024-03-18T15:56:29.371 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-18T15:56:29.372 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-18T15:56:29.372 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-18T15:56:29.372 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-18T15:56:29.372 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-18T15:56:29.372 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-18T15:56:29.372 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-18T15:56:29.372 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-18T15:56:29.372 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-18T15:56:29.372 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-18T15:56:29.372 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-18T15:56:29.372 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-18T15:56:29.372 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-18T15:56:29.372 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-18T15:56:29.372 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-18T15:56:29.372 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-18T15:56:29.373 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-18T15:56:29.373 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-18T15:56:29.373 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-18T15:56:29.373 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-18T15:56:29.373 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-18T15:56:29.373 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-18T15:56:29.373 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-18T15:56:29.373 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-18T15:56:29.373 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-18T15:56:29.373 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-18T15:56:29.373 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-18T15:56:29.373 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-18T15:56:29.373 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-18T15:56:29.373 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-18T15:56:29.374 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-18T15:56:29.374 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-18T15:56:29.374 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-18T15:56:29.374 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-18T15:56:29.374 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-18T15:56:29.374 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-18T15:56:29.374 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-18T15:56:29.374 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-18T15:56:29.374 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-18T15:56:29.374 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-18T15:56:29.374 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-18T15:56:29.374 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-18T15:56:29.374 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-18T15:56:29.374 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-18T15:56:29.375 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-18T15:56:29.375 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-18T15:56:29.375 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-18T15:56:29.375 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-18T15:56:29.375 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-18T15:56:29.375 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-18T15:56:29.375 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-18T15:56:29.375 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-18T15:56:29.375 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-18T15:56:29.375 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-18T15:56:29.375 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-18T15:56:29.375 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-18T15:56:29.375 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-18T15:56:29.375 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-18T15:56:29.376 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-18T15:56:29.376 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-18T15:56:29.376 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-18T15:56:29.376 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-18T15:56:29.376 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-18T15:56:29.376 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-18T15:56:29.376 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-18T15:56:29.376 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-18T15:56:29.376 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-18T15:56:29.376 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-18T15:56:29.376 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-18T15:56:29.376 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-18T15:56:29.376 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-18T15:56:29.376 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-18T15:56:29.376 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-18T15:56:29.377 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-18T15:56:29.377 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-18T15:56:29.377 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-18T15:56:29.377 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-18T15:56:29.377 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-18T15:56:29.377 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-18T15:56:29.377 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-18T15:56:29.377 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-18T15:56:29.377 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-18T15:56:29.377 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-18T15:56:29.377 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-18T15:56:29.377 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-18T15:56:29.377 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-18T15:56:29.377 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-18T15:56:29.377 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-18T15:56:29.378 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-18T15:56:29.378 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-18T15:56:29.378 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-18T15:56:29.378 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-18T15:56:29.378 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-18T15:56:29.378 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-18T15:56:29.378 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-18T15:56:29.378 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-18T15:56:29.378 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-18T15:56:29.378 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-18T15:56:29.378 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-18T15:56:29.378 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-18T15:56:29.378 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-18T15:56:29.378 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-18T15:56:29.379 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-18T15:56:29.379 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-18T15:56:29.379 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-18T15:56:29.379 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-18T15:56:29.379 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-18T15:56:29.379 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-18T15:56:29.379 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-18T15:56:29.379 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-18T15:56:29.379 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-18T15:56:29.379 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 151/198 2024-03-18T15:56:29.379 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-18T15:56:29.379 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 153/198 2024-03-18T15:56:29.379 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-18T15:56:29.379 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-18T15:56:29.380 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-18T15:56:29.380 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/198 2024-03-18T15:56:29.380 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-18T15:56:29.380 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/198 2024-03-18T15:56:29.380 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-18T15:56:29.380 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/198 2024-03-18T15:56:29.380 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-18T15:56:29.380 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/198 2024-03-18T15:56:29.380 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-18T15:56:29.380 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-18T15:56:29.380 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-18T15:56:29.380 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-18T15:56:29.380 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-18T15:56:29.380 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-18T15:56:29.380 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-18T15:56:29.381 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-18T15:56:29.381 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-18T15:56:29.381 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-18T15:56:29.381 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-18T15:56:29.381 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-18T15:56:29.381 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-18T15:56:29.381 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-18T15:56:29.381 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-18T15:56:29.381 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-18T15:56:29.381 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-18T15:56:29.381 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-18T15:56:29.381 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-18T15:56:29.381 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-18T15:56:29.381 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-18T15:56:29.381 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-18T15:56:29.382 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-18T15:56:29.382 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-18T15:56:29.382 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-18T15:56:29.382 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-18T15:56:29.382 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-18T15:56:29.382 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-18T15:56:29.382 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-18T15:56:29.382 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-18T15:56:29.382 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-18T15:56:29.382 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-18T15:56:29.382 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-18T15:56:29.382 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-18T15:56:31.943 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-18T15:56:31.944 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T15:56:31.944 INFO:teuthology.orchestra.run.smithi039.stdout:Upgraded: 2024-03-18T15:56:31.944 INFO:teuthology.orchestra.run.smithi039.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-18T15:56:31.944 INFO:teuthology.orchestra.run.smithi039.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-18T15:56:31.944 INFO:teuthology.orchestra.run.smithi039.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-18T15:56:31.944 INFO:teuthology.orchestra.run.smithi039.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-18T15:56:31.944 INFO:teuthology.orchestra.run.smithi039.stdout: acl-2.3.1-4.el9.x86_64 2024-03-18T15:56:31.944 INFO:teuthology.orchestra.run.smithi039.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-18T15:56:31.944 INFO:teuthology.orchestra.run.smithi039.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-18T15:56:31.944 INFO:teuthology.orchestra.run.smithi039.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-18T15:56:31.944 INFO:teuthology.orchestra.run.smithi039.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-18T15:56:31.944 INFO:teuthology.orchestra.run.smithi039.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-18T15:56:31.944 INFO:teuthology.orchestra.run.smithi039.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-18T15:56:31.945 INFO:teuthology.orchestra.run.smithi039.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-18T15:56:31.945 INFO:teuthology.orchestra.run.smithi039.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-18T15:56:31.945 INFO:teuthology.orchestra.run.smithi039.stdout: librados2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:56:31.945 INFO:teuthology.orchestra.run.smithi039.stdout: librbd1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:56:31.945 INFO:teuthology.orchestra.run.smithi039.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-18T15:56:31.945 INFO:teuthology.orchestra.run.smithi039.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-18T15:56:31.945 INFO:teuthology.orchestra.run.smithi039.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-18T15:56:31.945 INFO:teuthology.orchestra.run.smithi039.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-18T15:56:31.945 INFO:teuthology.orchestra.run.smithi039.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-18T15:56:31.945 INFO:teuthology.orchestra.run.smithi039.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-18T15:56:31.945 INFO:teuthology.orchestra.run.smithi039.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-18T15:56:31.945 INFO:teuthology.orchestra.run.smithi039.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-18T15:56:31.945 INFO:teuthology.orchestra.run.smithi039.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-18T15:56:31.945 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2024-03-18T15:56:31.946 INFO:teuthology.orchestra.run.smithi039.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-18T15:56:31.946 INFO:teuthology.orchestra.run.smithi039.stdout: automake-1.16.2-8.el9.noarch 2024-03-18T15:56:31.946 INFO:teuthology.orchestra.run.smithi039.stdout: bison-3.7.4-5.el9.x86_64 2024-03-18T15:56:31.946 INFO:teuthology.orchestra.run.smithi039.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-18T15:56:31.946 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:56:31.946 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-base-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:56:31.946 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:56:31.946 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:56:31.946 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-grafana-dashboards-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:56:31.946 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-immutable-object-cache-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:56:31.946 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mds-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:56:31.946 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:56:31.946 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:56:31.946 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-dashboard-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:56:31.946 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:56:31.947 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-modules-core-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:56:31.947 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-rook-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:56:31.947 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mon-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:56:31.947 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-osd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:56:31.947 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-prometheus-alerts-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:56:31.947 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-radosgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:56:31.947 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-selinux-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:56:31.947 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-test-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:56:31.947 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-volume-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:56:31.947 INFO:teuthology.orchestra.run.smithi039.stdout: cephadm-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:56:31.947 INFO:teuthology.orchestra.run.smithi039.stdout: cephfs-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:56:31.947 INFO:teuthology.orchestra.run.smithi039.stdout: cephfs-shell-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:56:31.947 INFO:teuthology.orchestra.run.smithi039.stdout: cephfs-top-2:19.0.0-2162.g3d78f5ba.el9.noarch 2024-03-18T15:56:31.947 INFO:teuthology.orchestra.run.smithi039.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-18T15:56:31.947 INFO:teuthology.orchestra.run.smithi039.stdout: flex-2.6.4-9.el9.x86_64 2024-03-18T15:56:31.948 INFO:teuthology.orchestra.run.smithi039.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-18T15:56:31.948 INFO:teuthology.orchestra.run.smithi039.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-18T15:56:31.948 INFO:teuthology.orchestra.run.smithi039.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-18T15:56:31.948 INFO:teuthology.orchestra.run.smithi039.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-18T15:56:31.948 INFO:teuthology.orchestra.run.smithi039.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-18T15:56:31.948 INFO:teuthology.orchestra.run.smithi039.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-18T15:56:31.948 INFO:teuthology.orchestra.run.smithi039.stdout: jq-1.6-16.el9.x86_64 2024-03-18T15:56:31.948 INFO:teuthology.orchestra.run.smithi039.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-18T15:56:31.948 INFO:teuthology.orchestra.run.smithi039.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-18T15:56:31.948 INFO:teuthology.orchestra.run.smithi039.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-18T15:56:31.948 INFO:teuthology.orchestra.run.smithi039.stdout: libcephfs-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:56:31.948 INFO:teuthology.orchestra.run.smithi039.stdout: libcephfs2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:56:31.948 INFO:teuthology.orchestra.run.smithi039.stdout: libcephsqlite-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:56:31.948 INFO:teuthology.orchestra.run.smithi039.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-18T15:56:31.948 INFO:teuthology.orchestra.run.smithi039.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-18T15:56:31.949 INFO:teuthology.orchestra.run.smithi039.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-18T15:56:31.949 INFO:teuthology.orchestra.run.smithi039.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-18T15:56:31.949 INFO:teuthology.orchestra.run.smithi039.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-18T15:56:31.949 INFO:teuthology.orchestra.run.smithi039.stdout: librados-devel-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:56:31.949 INFO:teuthology.orchestra.run.smithi039.stdout: libradosstriper1-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:56:31.949 INFO:teuthology.orchestra.run.smithi039.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-18T15:56:31.949 INFO:teuthology.orchestra.run.smithi039.stdout: librgw2-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:56:31.949 INFO:teuthology.orchestra.run.smithi039.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-18T15:56:31.949 INFO:teuthology.orchestra.run.smithi039.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-18T15:56:31.949 INFO:teuthology.orchestra.run.smithi039.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-18T15:56:31.949 INFO:teuthology.orchestra.run.smithi039.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-18T15:56:31.949 INFO:teuthology.orchestra.run.smithi039.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-18T15:56:31.949 INFO:teuthology.orchestra.run.smithi039.stdout: lua-5.4.4-4.el9.x86_64 2024-03-18T15:56:31.949 INFO:teuthology.orchestra.run.smithi039.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-18T15:56:31.949 INFO:teuthology.orchestra.run.smithi039.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-18T15:56:31.950 INFO:teuthology.orchestra.run.smithi039.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-18T15:56:31.950 INFO:teuthology.orchestra.run.smithi039.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-18T15:56:31.950 INFO:teuthology.orchestra.run.smithi039.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-18T15:56:31.950 INFO:teuthology.orchestra.run.smithi039.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-18T15:56:31.950 INFO:teuthology.orchestra.run.smithi039.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-18T15:56:31.950 INFO:teuthology.orchestra.run.smithi039.stdout: postgresql-13.11-1.el9.x86_64 2024-03-18T15:56:31.950 INFO:teuthology.orchestra.run.smithi039.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-18T15:56:31.950 INFO:teuthology.orchestra.run.smithi039.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-18T15:56:31.950 INFO:teuthology.orchestra.run.smithi039.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-18T15:56:31.950 INFO:teuthology.orchestra.run.smithi039.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-18T15:56:31.950 INFO:teuthology.orchestra.run.smithi039.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-18T15:56:31.950 INFO:teuthology.orchestra.run.smithi039.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-18T15:56:31.950 INFO:teuthology.orchestra.run.smithi039.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-18T15:56:31.950 INFO:teuthology.orchestra.run.smithi039.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-18T15:56:31.950 INFO:teuthology.orchestra.run.smithi039.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-18T15:56:31.951 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-18T15:56:31.951 INFO:teuthology.orchestra.run.smithi039.stdout: python3-ceph-argparse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:56:31.951 INFO:teuthology.orchestra.run.smithi039.stdout: python3-ceph-common-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:56:31.951 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cephfs-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:56:31.951 INFO:teuthology.orchestra.run.smithi039.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-18T15:56:31.951 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-18T15:56:31.951 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-18T15:56:31.951 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-18T15:56:31.951 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-18T15:56:31.951 INFO:teuthology.orchestra.run.smithi039.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-18T15:56:31.951 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-18T15:56:31.951 INFO:teuthology.orchestra.run.smithi039.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-18T15:56:31.951 INFO:teuthology.orchestra.run.smithi039.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-18T15:56:31.952 INFO:teuthology.orchestra.run.smithi039.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-18T15:56:31.952 INFO:teuthology.orchestra.run.smithi039.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-18T15:56:31.952 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-18T15:56:31.952 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-18T15:56:31.952 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-18T15:56:31.952 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-18T15:56:31.952 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-18T15:56:31.952 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-18T15:56:31.952 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-18T15:56:31.952 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-18T15:56:31.952 INFO:teuthology.orchestra.run.smithi039.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-18T15:56:31.952 INFO:teuthology.orchestra.run.smithi039.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-18T15:56:31.952 INFO:teuthology.orchestra.run.smithi039.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-18T15:56:31.952 INFO:teuthology.orchestra.run.smithi039.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-18T15:56:31.952 INFO:teuthology.orchestra.run.smithi039.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-18T15:56:31.953 INFO:teuthology.orchestra.run.smithi039.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-18T15:56:31.953 INFO:teuthology.orchestra.run.smithi039.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-18T15:56:31.953 INFO:teuthology.orchestra.run.smithi039.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-18T15:56:31.953 INFO:teuthology.orchestra.run.smithi039.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-18T15:56:31.953 INFO:teuthology.orchestra.run.smithi039.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-18T15:56:31.953 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-18T15:56:31.953 INFO:teuthology.orchestra.run.smithi039.stdout: python3-ply-3.11-14.el9.noarch 2024-03-18T15:56:31.953 INFO:teuthology.orchestra.run.smithi039.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-18T15:56:31.953 INFO:teuthology.orchestra.run.smithi039.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-18T15:56:31.953 INFO:teuthology.orchestra.run.smithi039.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-18T15:56:31.953 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-18T15:56:31.953 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-18T15:56:31.953 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-18T15:56:31.953 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-18T15:56:31.953 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-18T15:56:31.954 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-18T15:56:31.954 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rados-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:56:31.954 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:56:31.954 INFO:teuthology.orchestra.run.smithi039.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-18T15:56:31.954 INFO:teuthology.orchestra.run.smithi039.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-18T15:56:31.954 INFO:teuthology.orchestra.run.smithi039.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-18T15:56:31.954 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rgw-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:56:31.954 INFO:teuthology.orchestra.run.smithi039.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-18T15:56:31.954 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-18T15:56:31.954 INFO:teuthology.orchestra.run.smithi039.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-18T15:56:31.954 INFO:teuthology.orchestra.run.smithi039.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-18T15:56:31.954 INFO:teuthology.orchestra.run.smithi039.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-18T15:56:31.954 INFO:teuthology.orchestra.run.smithi039.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-18T15:56:31.954 INFO:teuthology.orchestra.run.smithi039.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-18T15:56:31.955 INFO:teuthology.orchestra.run.smithi039.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-18T15:56:31.955 INFO:teuthology.orchestra.run.smithi039.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-18T15:56:31.955 INFO:teuthology.orchestra.run.smithi039.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-18T15:56:31.955 INFO:teuthology.orchestra.run.smithi039.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-18T15:56:31.955 INFO:teuthology.orchestra.run.smithi039.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-18T15:56:31.955 INFO:teuthology.orchestra.run.smithi039.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-18T15:56:31.955 INFO:teuthology.orchestra.run.smithi039.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-18T15:56:31.955 INFO:teuthology.orchestra.run.smithi039.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-18T15:56:31.955 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-fuse-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:56:31.955 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-mirror-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:56:31.955 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-nbd-2:19.0.0-2162.g3d78f5ba.el9.x86_64 2024-03-18T15:56:31.955 INFO:teuthology.orchestra.run.smithi039.stdout: re2-1:20211101-3.el9.x86_64 2024-03-18T15:56:31.955 INFO:teuthology.orchestra.run.smithi039.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-18T15:56:31.955 INFO:teuthology.orchestra.run.smithi039.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-18T15:56:31.955 INFO:teuthology.orchestra.run.smithi039.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-18T15:56:31.956 INFO:teuthology.orchestra.run.smithi039.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-18T15:56:31.956 INFO:teuthology.orchestra.run.smithi039.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-18T15:56:31.956 INFO:teuthology.orchestra.run.smithi039.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-18T15:56:31.956 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T15:56:31.956 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2024-03-18T15:56:32.718 DEBUG:teuthology.parallel:result is None 2024-03-18T15:56:32.719 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T15:56:32.866 DEBUG:teuthology.orchestra.run.smithi039:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-18T15:56:32.923 INFO:teuthology.orchestra.run.smithi039.stdout:19.0.0-2162.g3d78f5ba.el9 2024-03-18T15:56:32.924 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2162.g3d78f5ba.el9 2024-03-18T15:56:32.924 INFO:teuthology.task.install:The correct ceph version 19.0.0-2162.g3d78f5ba is installed. 2024-03-18T15:56:32.926 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T15:56:33.145 DEBUG:teuthology.orchestra.run.smithi194:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-18T15:56:33.171 INFO:teuthology.orchestra.run.smithi194.stdout:19.0.0-2162.g3d78f5ba.el9 2024-03-18T15:56:33.171 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2162.g3d78f5ba.el9 2024-03-18T15:56:33.171 INFO:teuthology.task.install:The correct ceph version 19.0.0-2162.g3d78f5ba is installed. 2024-03-18T15:56:33.173 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T15:56:33.407 DEBUG:teuthology.orchestra.run.smithi196:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-18T15:56:33.432 INFO:teuthology.orchestra.run.smithi196.stdout:19.0.0-2162.g3d78f5ba.el9 2024-03-18T15:56:33.432 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2162.g3d78f5ba.el9 2024-03-18T15:56:33.432 INFO:teuthology.task.install:The correct ceph version 19.0.0-2162.g3d78f5ba is installed. 2024-03-18T15:56:33.434 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-18T15:56:33.434 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-03-18T15:56:33.434 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-18T15:56:33.516 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-03-18T15:56:33.516 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-18T15:56:33.551 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-18T15:56:33.551 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-18T15:56:33.585 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-18T15:56:33.585 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-03-18T15:56:33.585 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/usr/bin/daemon-helper 2024-03-18T15:56:33.677 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-18T15:56:33.748 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-03-18T15:56:33.748 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/usr/bin/daemon-helper 2024-03-18T15:56:33.775 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-18T15:56:33.843 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-18T15:56:33.843 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/usr/bin/daemon-helper 2024-03-18T15:56:33.874 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-18T15:56:33.943 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-18T15:56:33.943 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-03-18T15:56:33.943 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-18T15:56:34.021 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-18T15:56:34.132 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-03-18T15:56:34.133 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-18T15:56:34.160 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-18T15:56:34.227 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-18T15:56:34.227 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-18T15:56:34.254 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-18T15:56:34.323 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-18T15:56:34.323 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-03-18T15:56:34.323 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/usr/bin/stdin-killer 2024-03-18T15:56:34.402 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-18T15:56:34.516 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-03-18T15:56:34.517 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/usr/bin/stdin-killer 2024-03-18T15:56:34.543 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-18T15:56:34.609 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-18T15:56:34.609 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/usr/bin/stdin-killer 2024-03-18T15:56:34.636 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-18T15:56:34.702 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-18T15:56:34.800 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'filestore flush min': 0, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd heartbeat grace': 60, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'mons down, quorum', 'out of quorum', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'slow metadata IO', 'SLOW_OPS', 'slow request', 'SLOW_OPS', 'slow request'], 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', 'subvols': {'create': 2}} 2024-03-18T15:56:34.800 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T15:56:34.800 INFO:tasks.cephadm:Cluster fsid is 1892d8fc-e540-11ee-95c9-87774f69a715 2024-03-18T15:56:34.801 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-18T15:56:34.801 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.39', 'mon.b': '172.21.15.194', 'mon.c': '172.21.15.196'} 2024-03-18T15:56:34.801 INFO:tasks.cephadm:First mon is mon.a on smithi039 2024-03-18T15:56:34.801 INFO:tasks.cephadm:First mgr is x 2024-03-18T15:56:34.801 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-18T15:56:34.801 DEBUG:teuthology.orchestra.run.smithi039:> sudo hostname $(hostname -s) 2024-03-18T15:56:34.876 DEBUG:teuthology.orchestra.run.smithi194:> sudo hostname $(hostname -s) 2024-03-18T15:56:34.903 DEBUG:teuthology.orchestra.run.smithi196:> sudo hostname $(hostname -s) 2024-03-18T15:56:34.930 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-18T15:56:34.930 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T15:56:35.081 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77658/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2162-g3d78f5ba', 'node_name': '172.21.5.34+adami04', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2162.g3d78f5ba'}, 'url': 'https://3.chacra.ceph.com/r/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-15 01:33:04.210510', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-batrick-testing-20240314.183243', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-03-18T15:56:35.230 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref wip-batrick-testing-20240314.183243, sha1 3d78f5ba07bf18880172c07d4d7b4ac00c9d30af from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T15:56:35.232 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/x86_64/flavors/default/cephadm 2024-03-18T15:56:35.232 DEBUG:teuthology.orchestra.run.smithi039:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-18T15:56:35.777 INFO:teuthology.orchestra.run.smithi039.stdout:-rw-r--r--. 1 ubuntu ubuntu 767536 Mar 18 15:56 /home/ubuntu/cephtest/cephadm 2024-03-18T15:56:35.777 DEBUG:teuthology.orchestra.run.smithi194:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-18T15:56:36.213 INFO:teuthology.orchestra.run.smithi194.stdout:-rw-r--r--. 1 ubuntu ubuntu 767536 Mar 18 15:56 /home/ubuntu/cephtest/cephadm 2024-03-18T15:56:36.213 DEBUG:teuthology.orchestra.run.smithi196:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/wip-batrick-testing-20240314.183243/3d78f5ba07bf18880172c07d4d7b4ac00c9d30af/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-18T15:56:37.043 INFO:teuthology.orchestra.run.smithi196.stdout:-rw-r--r--. 1 ubuntu ubuntu 767536 Mar 18 15:56 /home/ubuntu/cephtest/cephadm 2024-03-18T15:56:37.044 DEBUG:teuthology.orchestra.run.smithi039:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-18T15:56:37.098 DEBUG:teuthology.orchestra.run.smithi194:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-18T15:56:37.117 DEBUG:teuthology.orchestra.run.smithi196:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-18T15:56:37.151 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af on all hosts... 2024-03-18T15:56:37.151 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af pull 2024-03-18T15:56:37.155 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af pull 2024-03-18T15:56:37.159 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af pull 2024-03-18T15:56:38.011 INFO:teuthology.orchestra.run.smithi194.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af... 2024-03-18T15:56:38.280 INFO:teuthology.orchestra.run.smithi196.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af... 2024-03-18T15:56:38.809 INFO:teuthology.orchestra.run.smithi039.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af... 2024-03-18T15:57:05.113 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-03-18T15:57:05.113 INFO:teuthology.orchestra.run.smithi194.stdout: "ceph_version": "ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev)", 2024-03-18T15:57:05.113 INFO:teuthology.orchestra.run.smithi194.stdout: "image_id": "be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae", 2024-03-18T15:57:05.113 INFO:teuthology.orchestra.run.smithi194.stdout: "repo_digests": [ 2024-03-18T15:57:05.114 INFO:teuthology.orchestra.run.smithi194.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513" 2024-03-18T15:57:05.114 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-03-18T15:57:05.114 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-03-18T15:57:06.288 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-03-18T15:57:06.288 INFO:teuthology.orchestra.run.smithi196.stdout: "ceph_version": "ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev)", 2024-03-18T15:57:06.288 INFO:teuthology.orchestra.run.smithi196.stdout: "image_id": "be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae", 2024-03-18T15:57:06.288 INFO:teuthology.orchestra.run.smithi196.stdout: "repo_digests": [ 2024-03-18T15:57:06.288 INFO:teuthology.orchestra.run.smithi196.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513" 2024-03-18T15:57:06.289 INFO:teuthology.orchestra.run.smithi196.stdout: ] 2024-03-18T15:57:06.289 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-03-18T15:57:14.016 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-03-18T15:57:14.016 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph_version": "ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev)", 2024-03-18T15:57:14.016 INFO:teuthology.orchestra.run.smithi039.stdout: "image_id": "be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae", 2024-03-18T15:57:14.016 INFO:teuthology.orchestra.run.smithi039.stdout: "repo_digests": [ 2024-03-18T15:57:14.016 INFO:teuthology.orchestra.run.smithi039.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513" 2024-03-18T15:57:14.016 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-03-18T15:57:14.016 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-03-18T15:57:14.058 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /etc/ceph 2024-03-18T15:57:14.157 DEBUG:teuthology.orchestra.run.smithi194:> sudo mkdir -p /etc/ceph 2024-03-18T15:57:14.192 DEBUG:teuthology.orchestra.run.smithi196:> sudo mkdir -p /etc/ceph 2024-03-18T15:57:14.227 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 777 /etc/ceph 2024-03-18T15:57:14.333 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod 777 /etc/ceph 2024-03-18T15:57:14.365 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod 777 /etc/ceph 2024-03-18T15:57:14.396 INFO:tasks.cephadm:Writing seed config... 2024-03-18T15:57:14.397 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-03-18T15:57:14.397 INFO:tasks.cephadm: override: [client] debug client = 20 2024-03-18T15:57:14.397 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-03-18T15:57:14.397 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-03-18T15:57:14.397 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-03-18T15:57:14.397 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-03-18T15:57:14.398 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-03-18T15:57:14.398 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-03-18T15:57:14.398 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-03-18T15:57:14.398 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-03-18T15:57:14.398 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-03-18T15:57:14.398 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-03-18T15:57:14.398 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-03-18T15:57:14.398 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-03-18T15:57:14.398 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-03-18T15:57:14.398 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-03-18T15:57:14.398 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-03-18T15:57:14.398 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-03-18T15:57:14.398 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-03-18T15:57:14.398 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-03-18T15:57:14.398 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-03-18T15:57:14.399 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-18T15:57:14.399 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-18T15:57:14.399 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-18T15:57:14.399 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-18T15:57:14.399 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-18T15:57:14.399 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-03-18T15:57:14.399 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-03-18T15:57:14.399 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-03-18T15:57:14.399 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-03-18T15:57:14.399 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-03-18T15:57:14.399 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-18T15:57:14.399 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-18T15:57:14.399 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-03-18T15:57:14.399 INFO:tasks.cephadm: override: [osd] filestore flush min = 0 2024-03-18T15:57:14.399 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-03-18T15:57:14.399 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-03-18T15:57:14.399 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-03-18T15:57:14.400 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-03-18T15:57:14.400 INFO:tasks.cephadm: override: [osd] osd heartbeat grace = 60 2024-03-18T15:57:14.400 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-03-18T15:57:14.400 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-03-18T15:57:14.400 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-18T15:57:14.400 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-03-18T15:57:14.400 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-03-18T15:57:14.401 DEBUG:teuthology.orchestra.run.smithi039:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-18T15:57:14.460 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 1892d8fc-e540-11ee-95c9-87774f69a715 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 filestore flush min = 0 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd heartbeat grace = 60 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-03-18T15:57:14.460 DEBUG:teuthology.orchestra.run.smithi039:mon.a> sudo journalctl -f -n 0 -u ceph-1892d8fc-e540-11ee-95c9-87774f69a715@mon.a.service 2024-03-18T15:57:14.507 DEBUG:teuthology.orchestra.run.smithi039:mgr.x> sudo journalctl -f -n 0 -u ceph-1892d8fc-e540-11ee-95c9-87774f69a715@mgr.x.service 2024-03-18T15:57:14.554 INFO:tasks.cephadm:Bootstrapping... 2024-03-18T15:57:14.554 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af -v bootstrap --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.39 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-18T15:57:14.836 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2024-03-18T15:57:14.836 INFO:teuthology.orchestra.run.smithi039.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af', '-v', 'bootstrap', '--fsid', '1892d8fc-e540-11ee-95c9-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.39', '--skip-admin-label'] 2024-03-18T15:57:14.890 INFO:teuthology.orchestra.run.smithi039.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-18T15:57:14.891 INFO:teuthology.orchestra.run.smithi039.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-18T15:57:14.891 INFO:teuthology.orchestra.run.smithi039.stdout:Verifying podman|docker is present... 2024-03-18T15:57:14.941 INFO:teuthology.orchestra.run.smithi039.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-18T15:57:14.941 INFO:teuthology.orchestra.run.smithi039.stdout:Verifying lvm2 is present... 2024-03-18T15:57:14.941 INFO:teuthology.orchestra.run.smithi039.stdout:Verifying time synchronization is in place... 2024-03-18T15:57:14.958 INFO:teuthology.orchestra.run.smithi039.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-18T15:57:14.958 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-18T15:57:14.975 INFO:teuthology.orchestra.run.smithi039.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-18T15:57:14.976 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stdout inactive 2024-03-18T15:57:15.001 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stdout enabled 2024-03-18T15:57:15.024 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stdout active 2024-03-18T15:57:15.024 INFO:teuthology.orchestra.run.smithi039.stdout:Unit chronyd.service is enabled and running 2024-03-18T15:57:15.024 INFO:teuthology.orchestra.run.smithi039.stdout:Repeating the final host check... 2024-03-18T15:57:15.087 INFO:teuthology.orchestra.run.smithi039.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-18T15:57:15.088 INFO:teuthology.orchestra.run.smithi039.stdout:podman (/bin/podman) version 4.9.3 is present 2024-03-18T15:57:15.088 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl is present 2024-03-18T15:57:15.088 INFO:teuthology.orchestra.run.smithi039.stdout:lvcreate is present 2024-03-18T15:57:15.106 INFO:teuthology.orchestra.run.smithi039.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-18T15:57:15.106 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-18T15:57:15.127 INFO:teuthology.orchestra.run.smithi039.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-18T15:57:15.128 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stdout inactive 2024-03-18T15:57:15.150 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stdout enabled 2024-03-18T15:57:15.171 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stdout active 2024-03-18T15:57:15.172 INFO:teuthology.orchestra.run.smithi039.stdout:Unit chronyd.service is enabled and running 2024-03-18T15:57:15.172 INFO:teuthology.orchestra.run.smithi039.stdout:Host looks OK 2024-03-18T15:57:15.172 INFO:teuthology.orchestra.run.smithi039.stdout:Cluster fsid: 1892d8fc-e540-11ee-95c9-87774f69a715 2024-03-18T15:57:15.173 INFO:teuthology.orchestra.run.smithi039.stdout:Acquiring lock 139902683598272 on /run/cephadm/1892d8fc-e540-11ee-95c9-87774f69a715.lock 2024-03-18T15:57:15.173 INFO:teuthology.orchestra.run.smithi039.stdout:Lock 139902683598272 acquired on /run/cephadm/1892d8fc-e540-11ee-95c9-87774f69a715.lock 2024-03-18T15:57:15.174 INFO:teuthology.orchestra.run.smithi039.stdout:Verifying IP 172.21.15.39 port 3300 ... 2024-03-18T15:57:15.175 INFO:teuthology.orchestra.run.smithi039.stdout:Verifying IP 172.21.15.39 port 6789 ... 2024-03-18T15:57:15.176 INFO:teuthology.orchestra.run.smithi039.stdout:Base mon IP(s) is [172.21.15.39:3300, 172.21.15.39:6789], mon addrv is [v2:172.21.15.39:3300,v1:172.21.15.39:6789] 2024-03-18T15:57:15.191 INFO:teuthology.orchestra.run.smithi039.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.39 metric 100 2024-03-18T15:57:15.191 INFO:teuthology.orchestra.run.smithi039.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.39 metric 100 2024-03-18T15:57:15.201 INFO:teuthology.orchestra.run.smithi039.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-18T15:57:15.201 INFO:teuthology.orchestra.run.smithi039.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-03-18T15:57:15.201 INFO:teuthology.orchestra.run.smithi039.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-03-18T15:57:15.208 INFO:teuthology.orchestra.run.smithi039.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-18T15:57:15.209 INFO:teuthology.orchestra.run.smithi039.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-18T15:57:15.209 INFO:teuthology.orchestra.run.smithi039.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-18T15:57:15.209 INFO:teuthology.orchestra.run.smithi039.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-03-18T15:57:15.209 INFO:teuthology.orchestra.run.smithi039.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:1370/64 scope link noprefixroute 2024-03-18T15:57:15.209 INFO:teuthology.orchestra.run.smithi039.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-18T15:57:15.210 INFO:teuthology.orchestra.run.smithi039.stdout:Mon IP `172.21.15.39` is in CIDR network `172.21.0.0/20` 2024-03-18T15:57:15.210 INFO:teuthology.orchestra.run.smithi039.stdout:Mon IP `172.21.15.39` is in CIDR network `172.21.0.0/20` 2024-03-18T15:57:15.210 INFO:teuthology.orchestra.run.smithi039.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-18T15:57:15.211 INFO:teuthology.orchestra.run.smithi039.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-18T15:57:15.212 INFO:teuthology.orchestra.run.smithi039.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af... 2024-03-18T15:57:15.920 INFO:teuthology.orchestra.run.smithi039.stdout:/bin/podman: stdout be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae 2024-03-18T15:57:15.920 INFO:teuthology.orchestra.run.smithi039.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af... 2024-03-18T15:57:15.921 INFO:teuthology.orchestra.run.smithi039.stdout:/bin/podman: stderr Getting image source signatures 2024-03-18T15:57:15.921 INFO:teuthology.orchestra.run.smithi039.stdout:/bin/podman: stderr Copying blob sha256:66487374cb611c95435850c17b53ebb64c22ed9555f72cd43d678b60262a12b1 2024-03-18T15:57:15.921 INFO:teuthology.orchestra.run.smithi039.stdout:/bin/podman: stderr Copying blob sha256:f58b4670ef4403d11d1c912aa26a4dbcf73447df5302e9dc66f6a7fc0d9d1e47 2024-03-18T15:57:15.921 INFO:teuthology.orchestra.run.smithi039.stdout:/bin/podman: stderr Copying config sha256:be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae 2024-03-18T15:57:15.921 INFO:teuthology.orchestra.run.smithi039.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-18T15:57:25.231 INFO:teuthology.orchestra.run.smithi039.stdout:ceph: stdout ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev) 2024-03-18T15:57:25.231 INFO:teuthology.orchestra.run.smithi039.stdout:Ceph version: ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev) 2024-03-18T15:57:25.231 INFO:teuthology.orchestra.run.smithi039.stdout:Extracting ceph user uid/gid from container image... 2024-03-18T15:57:27.459 INFO:teuthology.orchestra.run.smithi039.stdout:stat: stdout 167 167 2024-03-18T15:57:27.459 INFO:teuthology.orchestra.run.smithi039.stdout:Creating initial keys... 2024-03-18T15:57:28.671 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph-authtool: stdout AQBnZPhl1Hm+ORAAM3c9AG+O7nMqpCn2ePgcdw== 2024-03-18T15:57:29.842 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph-authtool: stdout AQBpZPhl1ICKCRAAax9hfvJ6vVMLGTps6AUuSQ== 2024-03-18T15:57:31.789 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph-authtool: stdout AQBqZPhl+vxHExAAF1CoS8mB1VXIBGSZbaeJ0g== 2024-03-18T15:57:31.792 INFO:teuthology.orchestra.run.smithi039.stdout:Creating initial monmap... 2024-03-18T15:57:32.989 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-18T15:57:32.989 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-18T15:57:32.989 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 1892d8fc-e540-11ee-95c9-87774f69a715 2024-03-18T15:57:32.989 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-18T15:57:32.989 INFO:teuthology.orchestra.run.smithi039.stdout:monmaptool for a [v2:172.21.15.39:3300,v1:172.21.15.39:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-18T15:57:32.989 INFO:teuthology.orchestra.run.smithi039.stdout:setting min_mon_release = quincy 2024-03-18T15:57:32.989 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/monmaptool: set fsid to 1892d8fc-e540-11ee-95c9-87774f69a715 2024-03-18T15:57:32.989 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-18T15:57:32.989 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T15:57:32.989 INFO:teuthology.orchestra.run.smithi039.stdout:Creating mon... 2024-03-18T15:57:34.025 INFO:teuthology.orchestra.run.smithi039.stdout:create mon.a on 2024-03-18T15:57:34.437 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-18T15:57:34.678 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-18T15:57:35.004 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-1892d8fc-e540-11ee-95c9-87774f69a715.target → /etc/systemd/system/ceph-1892d8fc-e540-11ee-95c9-87774f69a715.target. 2024-03-18T15:57:35.005 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-1892d8fc-e540-11ee-95c9-87774f69a715.target → /etc/systemd/system/ceph-1892d8fc-e540-11ee-95c9-87774f69a715.target. 2024-03-18T15:57:35.441 INFO:teuthology.orchestra.run.smithi039.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-1892d8fc-e540-11ee-95c9-87774f69a715@mon.a 2024-03-18T15:57:35.441 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stderr Failed to reset failed state of unit ceph-1892d8fc-e540-11ee-95c9-87774f69a715@mon.a.service: Unit ceph-1892d8fc-e540-11ee-95c9-87774f69a715@mon.a.service not loaded. 2024-03-18T15:57:35.730 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-1892d8fc-e540-11ee-95c9-87774f69a715.target.wants/ceph-1892d8fc-e540-11ee-95c9-87774f69a715@mon.a.service → /etc/systemd/system/ceph-1892d8fc-e540-11ee-95c9-87774f69a715@.service. 2024-03-18T15:57:36.415 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:36 smithi039 podman[31338]: 2024-03-18 15:57:36.030306921 +0000 UTC m=+0.032948138 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T15:57:36.415 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:36 smithi039 podman[31338]: 2024-03-18 15:57:36.413264585 +0000 UTC m=+0.415905864 container init 3dbda31d5bf49a16ebab4a1fc91bb9d9b835801445c7041852ba58ac843d6368 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mon-a, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-18T15:57:36.527 INFO:teuthology.orchestra.run.smithi039.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-18T15:57:36.527 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stdout disabled 2024-03-18T15:57:36.542 INFO:teuthology.orchestra.run.smithi039.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-18T15:57:36.542 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stdout inactive 2024-03-18T15:57:36.542 INFO:teuthology.orchestra.run.smithi039.stdout:firewalld.service is not enabled 2024-03-18T15:57:36.542 INFO:teuthology.orchestra.run.smithi039.stdout:Not possible to enable service . firewalld.service is not available 2024-03-18T15:57:36.543 INFO:teuthology.orchestra.run.smithi039.stdout:Waiting for mon to start... 2024-03-18T15:57:36.543 INFO:teuthology.orchestra.run.smithi039.stdout:Waiting for mon... 2024-03-18T15:57:36.687 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:36 smithi039 podman[31338]: 2024-03-18 15:57:36.424069811 +0000 UTC m=+0.426711034 container start 3dbda31d5bf49a16ebab4a1fc91bb9d9b835801445c7041852ba58ac843d6368 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mon-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-18T15:57:36.711 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:36 smithi039 bash[31338]: 3dbda31d5bf49a16ebab4a1fc91bb9d9b835801445c7041852ba58ac843d6368 2024-03-18T15:57:36.711 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:36 smithi039 systemd[1]: Started Ceph mon.a for 1892d8fc-e540-11ee-95c9-87774f69a715. 2024-03-18T15:57:36.711 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:36 smithi039 ceph-mon[31357]: mkfs 1892d8fc-e540-11ee-95c9-87774f69a715 2024-03-18T15:57:36.711 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:36 smithi039 ceph-mon[31357]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-18T15:57:37.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:37 smithi039 ceph-mon[31357]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-18T15:57:37.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:37 smithi039 ceph-mon[31357]: monmap epoch 1 2024-03-18T15:57:37.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:37 smithi039 ceph-mon[31357]: fsid 1892d8fc-e540-11ee-95c9-87774f69a715 2024-03-18T15:57:37.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:37 smithi039 ceph-mon[31357]: last_changed 2024-03-18T15:57:32.305376+0000 2024-03-18T15:57:37.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:37 smithi039 ceph-mon[31357]: created 2024-03-18T15:57:32.305376+0000 2024-03-18T15:57:37.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:37 smithi039 ceph-mon[31357]: min_mon_release 19 (squid) 2024-03-18T15:57:37.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:37 smithi039 ceph-mon[31357]: election_strategy: 1 2024-03-18T15:57:37.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:37 smithi039 ceph-mon[31357]: 0: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.a 2024-03-18T15:57:37.838 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:37 smithi039 ceph-mon[31357]: fsmap 2024-03-18T15:57:37.838 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:37 smithi039 ceph-mon[31357]: osdmap e1: 0 total, 0 up, 0 in 2024-03-18T15:57:37.838 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:37 smithi039 ceph-mon[31357]: mgrmap e1: no daemons active 2024-03-18T15:57:37.838 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:37 smithi039 ceph-mon[31357]: from='client.? 172.21.15.39:0/3374176581' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-18T15:57:37.949 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout cluster: 2024-03-18T15:57:37.950 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout id: 1892d8fc-e540-11ee-95c9-87774f69a715 2024-03-18T15:57:37.950 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-18T15:57:37.951 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout 2024-03-18T15:57:37.951 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout services: 2024-03-18T15:57:37.957 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.720609s) 2024-03-18T15:57:37.957 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-18T15:57:37.957 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-18T15:57:37.957 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout 2024-03-18T15:57:37.957 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout data: 2024-03-18T15:57:37.957 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-18T15:57:37.957 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-18T15:57:37.957 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-18T15:57:37.957 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout pgs: 2024-03-18T15:57:37.958 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout 2024-03-18T15:57:37.958 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:37.130+0000 7f1a9c600700 1 Processor -- start 2024-03-18T15:57:37.958 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:37.130+0000 7f1a9c600700 1 -- start start 2024-03-18T15:57:37.958 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:37.130+0000 7f1a9c600700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1a94109e00 0x7f1a9410a220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:37.958 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:37.130+0000 7f1a9c600700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f1a9410a760 con 0x7f1a94109e00 2024-03-18T15:57:37.958 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:37.131+0000 7f1a99c00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1a94109e00 0x7f1a9410a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:37.958 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:37.132+0000 7f1a99c00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1a94109e00 0x7f1a9410a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:39014/0 (socket says 172.21.15.39:39014) 2024-03-18T15:57:37.958 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:37.132+0000 7f1a99c00700 1 -- 172.21.15.39:0/3531203413 learned_addr learned my addr 172.21.15.39:0/3531203413 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T15:57:37.958 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:37.132+0000 7f1a99c00700 1 -- 172.21.15.39:0/3531203413 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a9410af60 con 0x7f1a94109e00 2024-03-18T15:57:37.958 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:37.132+0000 7f1a99c00700 1 --2- 172.21.15.39:0/3531203413 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1a94109e00 0x7f1a9410a220 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f1a880098a0 tx=0x7f1a88009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3bb5afcd3ba067 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:37.958 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:37.133+0000 7f1a93e00700 1 -- 172.21.15.39:0/3531203413 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1a88004160 con 0x7f1a94109e00 2024-03-18T15:57:37.959 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:37.133+0000 7f1a93e00700 1 -- 172.21.15.39:0/3531203413 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a88004300 con 0x7f1a94109e00 2024-03-18T15:57:37.959 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:37.133+0000 7f1a93e00700 1 -- 172.21.15.39:0/3531203413 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1a880036a0 con 0x7f1a94109e00 2024-03-18T15:57:37.959 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:37.133+0000 7f1a9c600700 1 -- 172.21.15.39:0/3531203413 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1a94109e00 msgr2=0x7f1a9410a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:37.959 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:37.133+0000 7f1a9c600700 1 --2- 172.21.15.39:0/3531203413 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1a94109e00 0x7f1a9410a220 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f1a880098a0 tx=0x7f1a88009bb0 comp rx=0 tx=0).stop 2024-03-18T15:57:37.959 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:37.133+0000 7f1a9c600700 1 -- 172.21.15.39:0/3531203413 shutdown_connections 2024-03-18T15:57:37.959 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:37.133+0000 7f1a9c600700 1 --2- 172.21.15.39:0/3531203413 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1a94109e00 0x7f1a9410a220 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:37.959 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:37.134+0000 7f1a9c600700 1 -- 172.21.15.39:0/3531203413 >> 172.21.15.39:0/3531203413 conn(0x7f1a9407d210 msgr2=0x7f1a9407d620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:57:37.959 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:37.134+0000 7f1a9c600700 1 -- 172.21.15.39:0/3531203413 shutdown_connections 2024-03-18T15:57:37.959 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:37.134+0000 7f1a9c600700 1 -- 172.21.15.39:0/3531203413 wait complete. 2024-03-18T15:57:37.959 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:37.135+0000 7f1a9c600700 1 Processor -- start 2024-03-18T15:57:37.959 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:37.135+0000 7f1a9c600700 1 -- start start 2024-03-18T15:57:37.960 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:37.135+0000 7f1a9c600700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1a94109e00 0x7f1a9419d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:37.960 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:37.135+0000 7f1a9c600700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f1a9410b430 con 0x7f1a94109e00 2024-03-18T15:57:37.960 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:37.136+0000 7f1a99c00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1a94109e00 0x7f1a9419d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:37.960 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:37.136+0000 7f1a99c00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1a94109e00 0x7f1a9419d530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:39018/0 (socket says 172.21.15.39:39018) 2024-03-18T15:57:37.960 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:37.136+0000 7f1a99c00700 1 -- 172.21.15.39:0/3374176581 learned_addr learned my addr 172.21.15.39:0/3374176581 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T15:57:37.960 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:37.136+0000 7f1a99c00700 1 -- 172.21.15.39:0/3374176581 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a9419da70 con 0x7f1a94109e00 2024-03-18T15:57:37.960 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:37.136+0000 7f1a99c00700 1 --2- 172.21.15.39:0/3374176581 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1a94109e00 0x7f1a9419d530 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f1a88000c00 tx=0x7f1a88004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:37.960 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:37.137+0000 7f1a92000700 1 -- 172.21.15.39:0/3374176581 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1a88003db0 con 0x7f1a94109e00 2024-03-18T15:57:37.960 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:37.137+0000 7f1a92000700 1 -- 172.21.15.39:0/3374176581 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a88011630 con 0x7f1a94109e00 2024-03-18T15:57:37.960 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:37.137+0000 7f1a92000700 1 -- 172.21.15.39:0/3374176581 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1a88011950 con 0x7f1a94109e00 2024-03-18T15:57:37.960 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:37.137+0000 7f1a9c600700 1 -- 172.21.15.39:0/3374176581 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a9419dd00 con 0x7f1a94109e00 2024-03-18T15:57:37.960 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:37.137+0000 7f1a9c600700 1 -- 172.21.15.39:0/3374176581 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a941a09a0 con 0x7f1a94109e00 2024-03-18T15:57:37.962 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:37.137+0000 7f1a92000700 1 -- 172.21.15.39:0/3374176581 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f1a8801a460 con 0x7f1a94109e00 2024-03-18T15:57:37.962 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:37.137+0000 7f1a92000700 1 -- 172.21.15.39:0/3374176581 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f1a8801b780 con 0x7f1a94109e00 2024-03-18T15:57:37.962 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:37.138+0000 7f1a9c600700 1 -- 172.21.15.39:0/3374176581 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a94063900 con 0x7f1a94109e00 2024-03-18T15:57:37.962 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:37.141+0000 7f1a92000700 1 -- 172.21.15.39:0/3374176581 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f1a8801b980 con 0x7f1a94109e00 2024-03-18T15:57:37.963 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:37.223+0000 7f1a9c600700 1 -- 172.21.15.39:0/3374176581 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f1a9402ddc0 con 0x7f1a94109e00 2024-03-18T15:57:37.963 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:37.223+0000 7f1a92000700 1 -- 172.21.15.39:0/3374176581 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7f1a8801bb60 con 0x7f1a94109e00 2024-03-18T15:57:37.963 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:37.225+0000 7f1a9c600700 1 -- 172.21.15.39:0/3374176581 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1a94109e00 msgr2=0x7f1a9419d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:37.963 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:37.225+0000 7f1a9c600700 1 --2- 172.21.15.39:0/3374176581 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1a94109e00 0x7f1a9419d530 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f1a88000c00 tx=0x7f1a88004750 comp rx=0 tx=0).stop 2024-03-18T15:57:37.963 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:37.225+0000 7f1a9c600700 1 -- 172.21.15.39:0/3374176581 shutdown_connections 2024-03-18T15:57:37.963 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:37.225+0000 7f1a9c600700 1 --2- 172.21.15.39:0/3374176581 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1a94109e00 0x7f1a9419d530 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:37.963 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:37.225+0000 7f1a9c600700 1 -- 172.21.15.39:0/3374176581 >> 172.21.15.39:0/3374176581 conn(0x7f1a9407d210 msgr2=0x7f1a94193fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:57:37.963 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:37.225+0000 7f1a9c600700 1 -- 172.21.15.39:0/3374176581 shutdown_connections 2024-03-18T15:57:37.963 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:37.225+0000 7f1a9c600700 1 -- 172.21.15.39:0/3374176581 wait complete. 2024-03-18T15:57:37.963 INFO:teuthology.orchestra.run.smithi039.stdout:mon is available 2024-03-18T15:57:37.963 INFO:teuthology.orchestra.run.smithi039.stdout:Assimilating anything we can from ceph.conf... 2024-03-18T15:57:39.039 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:38 smithi039 ceph-mon[31357]: from='client.? 172.21.15.39:0/1425395519' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-18T15:57:39.039 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:38 smithi039 ceph-mon[31357]: from='client.? 172.21.15.39:0/1425395519' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-18T15:57:39.375 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout 2024-03-18T15:57:39.375 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout [global] 2024-03-18T15:57:39.375 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout fsid = 1892d8fc-e540-11ee-95c9-87774f69a715 2024-03-18T15:57:39.375 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.39:3300,v1:172.21.15.39:6789] 2024-03-18T15:57:39.375 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-18T15:57:39.376 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-18T15:57:39.376 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-18T15:57:39.376 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-18T15:57:39.376 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout 2024-03-18T15:57:39.376 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-18T15:57:39.377 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-18T15:57:39.377 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout 2024-03-18T15:57:39.377 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout [osd] 2024-03-18T15:57:39.377 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout filestore_flush_min = 0 2024-03-18T15:57:39.377 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-18T15:57:39.377 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-18T15:57:39.377 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-18T15:57:39.378 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-18T15:57:39.378 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-18T15:57:39.378 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:38.591+0000 7f24f7e00700 1 Processor -- start 2024-03-18T15:57:39.378 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:38.592+0000 7f24f7e00700 1 -- start start 2024-03-18T15:57:39.378 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:38.592+0000 7f24f7e00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f24f8082240 0x7f24f8110c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:39.379 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:38.592+0000 7f24f7e00700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f24f8111240 con 0x7f24f8082240 2024-03-18T15:57:39.379 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:38.593+0000 7f24f6a00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f24f8082240 0x7f24f8110c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:39.379 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:38.593+0000 7f24f6a00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f24f8082240 0x7f24f8110c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:36354/0 (socket says 172.21.15.39:36354) 2024-03-18T15:57:39.379 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:38.593+0000 7f24f6a00700 1 -- 172.21.15.39:0/147141354 learned_addr learned my addr 172.21.15.39:0/147141354 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T15:57:39.379 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:38.593+0000 7f24f6a00700 1 -- 172.21.15.39:0/147141354 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24f81113c0 con 0x7f24f8082240 2024-03-18T15:57:39.380 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:38.593+0000 7f24f6a00700 1 --2- 172.21.15.39:0/147141354 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f24f8082240 0x7f24f8110c70 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f24e80098a0 tx=0x7f24e8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=688242d9ddde52d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:39.380 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:38.594+0000 7f24f5600700 1 -- 172.21.15.39:0/147141354 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f24e8004160 con 0x7f24f8082240 2024-03-18T15:57:39.380 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:38.594+0000 7f24f5600700 1 -- 172.21.15.39:0/147141354 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24e8004300 con 0x7f24f8082240 2024-03-18T15:57:39.380 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:38.594+0000 7f24f5600700 1 -- 172.21.15.39:0/147141354 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f24e80036a0 con 0x7f24f8082240 2024-03-18T15:57:39.380 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:38.594+0000 7f24f7e00700 1 -- 172.21.15.39:0/147141354 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f24f8082240 msgr2=0x7f24f8110c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:39.381 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:38.594+0000 7f24f7e00700 1 --2- 172.21.15.39:0/147141354 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f24f8082240 0x7f24f8110c70 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f24e80098a0 tx=0x7f24e8009bb0 comp rx=0 tx=0).stop 2024-03-18T15:57:39.381 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:38.594+0000 7f24f7e00700 1 -- 172.21.15.39:0/147141354 shutdown_connections 2024-03-18T15:57:39.381 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:38.594+0000 7f24f7e00700 1 --2- 172.21.15.39:0/147141354 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f24f8082240 0x7f24f8110c70 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:39.381 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:38.594+0000 7f24f7e00700 1 -- 172.21.15.39:0/147141354 >> 172.21.15.39:0/147141354 conn(0x7f24f807d090 msgr2=0x7f24f807f4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:57:39.381 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:38.594+0000 7f24f7e00700 1 -- 172.21.15.39:0/147141354 shutdown_connections 2024-03-18T15:57:39.381 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:38.595+0000 7f24f7e00700 1 -- 172.21.15.39:0/147141354 wait complete. 2024-03-18T15:57:39.382 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:38.595+0000 7f24f7e00700 1 Processor -- start 2024-03-18T15:57:39.382 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:38.596+0000 7f24f7e00700 1 -- start start 2024-03-18T15:57:39.382 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:38.596+0000 7f24f7e00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f24f8082240 0x7f24f81a17b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:39.382 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:38.596+0000 7f24f7e00700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f24f81118f0 con 0x7f24f8082240 2024-03-18T15:57:39.382 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:38.596+0000 7f24f6a00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f24f8082240 0x7f24f81a17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:39.382 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:38.596+0000 7f24f6a00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f24f8082240 0x7f24f81a17b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:36360/0 (socket says 172.21.15.39:36360) 2024-03-18T15:57:39.383 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:38.596+0000 7f24f6a00700 1 -- 172.21.15.39:0/1425395519 learned_addr learned my addr 172.21.15.39:0/1425395519 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T15:57:39.383 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:38.597+0000 7f24f6a00700 1 -- 172.21.15.39:0/1425395519 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24f81a1cf0 con 0x7f24f8082240 2024-03-18T15:57:39.383 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:38.597+0000 7f24f6a00700 1 --2- 172.21.15.39:0/1425395519 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f24f8082240 0x7f24f81a17b0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f24e8000c00 tx=0x7f24e8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:39.383 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:38.597+0000 7f24ef400700 1 -- 172.21.15.39:0/1425395519 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f24e8003db0 con 0x7f24f8082240 2024-03-18T15:57:39.383 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:38.597+0000 7f24ef400700 1 -- 172.21.15.39:0/1425395519 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24e8011630 con 0x7f24f8082240 2024-03-18T15:57:39.383 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:38.597+0000 7f24ef400700 1 -- 172.21.15.39:0/1425395519 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f24e8011950 con 0x7f24f8082240 2024-03-18T15:57:39.384 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:38.597+0000 7f24f7e00700 1 -- 172.21.15.39:0/1425395519 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24f81a1f80 con 0x7f24f8082240 2024-03-18T15:57:39.384 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:38.597+0000 7f24f7e00700 1 -- 172.21.15.39:0/1425395519 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24f81a4c70 con 0x7f24f8082240 2024-03-18T15:57:39.384 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:38.598+0000 7f24ef400700 1 -- 172.21.15.39:0/1425395519 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f24e801a460 con 0x7f24f8082240 2024-03-18T15:57:39.384 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:38.598+0000 7f24ef400700 1 -- 172.21.15.39:0/1425395519 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f24e801b780 con 0x7f24f8082240 2024-03-18T15:57:39.384 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:38.599+0000 7f24f7e00700 1 -- 172.21.15.39:0/1425395519 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24f8063900 con 0x7f24f8082240 2024-03-18T15:57:39.385 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:38.602+0000 7f24ef400700 1 -- 172.21.15.39:0/1425395519 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f24e801b980 con 0x7f24f8082240 2024-03-18T15:57:39.385 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:38.679+0000 7f24f7e00700 1 -- 172.21.15.39:0/1425395519 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f24f81a2460 con 0x7f24f8082240 2024-03-18T15:57:39.385 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:38.689+0000 7f24ef400700 1 -- 172.21.15.39:0/1425395519 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+454 (secure 0 0 0) 0x7f24f81a2460 con 0x7f24f8082240 2024-03-18T15:57:39.385 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:38.691+0000 7f24f7e00700 1 -- 172.21.15.39:0/1425395519 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f24f8082240 msgr2=0x7f24f81a17b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:39.385 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:38.691+0000 7f24f7e00700 1 --2- 172.21.15.39:0/1425395519 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f24f8082240 0x7f24f81a17b0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f24e8000c00 tx=0x7f24e8004750 comp rx=0 tx=0).stop 2024-03-18T15:57:39.385 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:38.691+0000 7f24ef400700 1 -- 172.21.15.39:0/1425395519 <== mon.0 v2:172.21.15.39:3300/0 8 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f24e801ada0 con 0x7f24f8082240 2024-03-18T15:57:39.386 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:38.691+0000 7f24f7e00700 1 -- 172.21.15.39:0/1425395519 shutdown_connections 2024-03-18T15:57:39.386 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:38.691+0000 7f24f7e00700 1 --2- 172.21.15.39:0/1425395519 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f24f8082240 0x7f24f81a17b0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:39.386 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:38.691+0000 7f24f7e00700 1 -- 172.21.15.39:0/1425395519 >> 172.21.15.39:0/1425395519 conn(0x7f24f807d090 msgr2=0x7f24f807f3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:57:39.386 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:38.692+0000 7f24f7e00700 1 -- 172.21.15.39:0/1425395519 shutdown_connections 2024-03-18T15:57:39.386 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:38.692+0000 7f24f7e00700 1 -- 172.21.15.39:0/1425395519 wait complete. 2024-03-18T15:57:39.386 INFO:teuthology.orchestra.run.smithi039.stdout:Generating new minimal ceph.conf... 2024-03-18T15:57:40.331 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:40 smithi039 ceph-mon[31357]: from='client.? 172.21.15.39:0/4113064090' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:57:40.637 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:39.955+0000 7f1d7d400700 1 Processor -- start 2024-03-18T15:57:40.637 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:39.955+0000 7f1d7d400700 1 -- start start 2024-03-18T15:57:40.638 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:39.956+0000 7f1d7d400700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1d78107c20 0x7f1d78108040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:40.638 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:39.956+0000 7f1d7d400700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f1d78108610 con 0x7f1d78107c20 2024-03-18T15:57:40.638 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:39.956+0000 7f1d76800700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1d78107c20 0x7f1d78108040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:40.638 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:39.956+0000 7f1d76800700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1d78107c20 0x7f1d78108040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:36362/0 (socket says 172.21.15.39:36362) 2024-03-18T15:57:40.638 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:39.956+0000 7f1d76800700 1 -- 172.21.15.39:0/3046147773 learned_addr learned my addr 172.21.15.39:0/3046147773 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T15:57:40.638 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:39.957+0000 7f1d76800700 1 -- 172.21.15.39:0/3046147773 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d78108e20 con 0x7f1d78107c20 2024-03-18T15:57:40.638 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:39.957+0000 7f1d76800700 1 --2- 172.21.15.39:0/3046147773 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1d78107c20 0x7f1d78108040 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f1d680098a0 tx=0x7f1d68009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99f80e1770ec82c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:40.638 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:39.958+0000 7f1d75400700 1 -- 172.21.15.39:0/3046147773 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1d68004160 con 0x7f1d78107c20 2024-03-18T15:57:40.638 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:39.958+0000 7f1d75400700 1 -- 172.21.15.39:0/3046147773 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f1d68004300 con 0x7f1d78107c20 2024-03-18T15:57:40.638 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:39.958+0000 7f1d75400700 1 -- 172.21.15.39:0/3046147773 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1d68011b50 con 0x7f1d78107c20 2024-03-18T15:57:40.638 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:39.958+0000 7f1d7d400700 1 -- 172.21.15.39:0/3046147773 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1d78107c20 msgr2=0x7f1d78108040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:40.639 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:39.958+0000 7f1d7d400700 1 --2- 172.21.15.39:0/3046147773 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1d78107c20 0x7f1d78108040 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f1d680098a0 tx=0x7f1d68009bb0 comp rx=0 tx=0).stop 2024-03-18T15:57:40.639 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:39.959+0000 7f1d7d400700 1 -- 172.21.15.39:0/3046147773 shutdown_connections 2024-03-18T15:57:40.639 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:39.959+0000 7f1d7d400700 1 --2- 172.21.15.39:0/3046147773 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1d78107c20 0x7f1d78108040 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:40.639 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:39.959+0000 7f1d7d400700 1 -- 172.21.15.39:0/3046147773 >> 172.21.15.39:0/3046147773 conn(0x7f1d781031e0 msgr2=0x7f1d78105600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:57:40.639 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:39.959+0000 7f1d7d400700 1 -- 172.21.15.39:0/3046147773 shutdown_connections 2024-03-18T15:57:40.639 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:39.959+0000 7f1d7d400700 1 -- 172.21.15.39:0/3046147773 wait complete. 2024-03-18T15:57:40.639 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:39.960+0000 7f1d7d400700 1 Processor -- start 2024-03-18T15:57:40.639 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:39.960+0000 7f1d7d400700 1 -- start start 2024-03-18T15:57:40.639 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:39.960+0000 7f1d7d400700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1d78107c20 0x7f1d7819ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:40.639 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:39.960+0000 7f1d7d400700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f1d7810c1d0 con 0x7f1d78107c20 2024-03-18T15:57:40.639 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:39.961+0000 7f1d76800700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1d78107c20 0x7f1d7819ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:40.639 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:39.961+0000 7f1d76800700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1d78107c20 0x7f1d7819ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:36368/0 (socket says 172.21.15.39:36368) 2024-03-18T15:57:40.639 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:39.961+0000 7f1d76800700 1 -- 172.21.15.39:0/4113064090 learned_addr learned my addr 172.21.15.39:0/4113064090 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T15:57:40.640 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:39.961+0000 7f1d76800700 1 -- 172.21.15.39:0/4113064090 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d7819e330 con 0x7f1d78107c20 2024-03-18T15:57:40.640 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:39.961+0000 7f1d76800700 1 --2- 172.21.15.39:0/4113064090 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1d78107c20 0x7f1d7819ddf0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f1d68011850 tx=0x7f1d68011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:40.640 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:39.961+0000 7f1d6f400700 1 -- 172.21.15.39:0/4113064090 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1d68003af0 con 0x7f1d78107c20 2024-03-18T15:57:40.640 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:39.962+0000 7f1d6f400700 1 -- 172.21.15.39:0/4113064090 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f1d68020070 con 0x7f1d78107c20 2024-03-18T15:57:40.640 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:39.962+0000 7f1d7d400700 1 -- 172.21.15.39:0/4113064090 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d7819e5c0 con 0x7f1d78107c20 2024-03-18T15:57:40.640 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:39.962+0000 7f1d6f400700 1 -- 172.21.15.39:0/4113064090 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1d680195a0 con 0x7f1d78107c20 2024-03-18T15:57:40.640 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:39.962+0000 7f1d7d400700 1 -- 172.21.15.39:0/4113064090 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d781a12b0 con 0x7f1d78107c20 2024-03-18T15:57:40.640 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:39.962+0000 7f1d6f400700 1 -- 172.21.15.39:0/4113064090 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f1d680198a0 con 0x7f1d78107c20 2024-03-18T15:57:40.640 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:39.962+0000 7f1d6f400700 1 -- 172.21.15.39:0/4113064090 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f1d68011e00 con 0x7f1d78107c20 2024-03-18T15:57:40.640 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:39.963+0000 7f1d7d400700 1 -- 172.21.15.39:0/4113064090 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d78063900 con 0x7f1d78107c20 2024-03-18T15:57:40.640 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:39.966+0000 7f1d6f400700 1 -- 172.21.15.39:0/4113064090 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f1d68003c90 con 0x7f1d78107c20 2024-03-18T15:57:40.640 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:40.044+0000 7f1d7d400700 1 -- 172.21.15.39:0/4113064090 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f1d781a17f0 con 0x7f1d78107c20 2024-03-18T15:57:40.641 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:40.045+0000 7f1d6f400700 1 -- 172.21.15.39:0/4113064090 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7f1d68019d20 con 0x7f1d78107c20 2024-03-18T15:57:40.641 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:40.047+0000 7f1d7d400700 1 -- 172.21.15.39:0/4113064090 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1d78107c20 msgr2=0x7f1d7819ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:40.641 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:40.047+0000 7f1d7d400700 1 --2- 172.21.15.39:0/4113064090 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1d78107c20 0x7f1d7819ddf0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f1d68011850 tx=0x7f1d68011c10 comp rx=0 tx=0).stop 2024-03-18T15:57:40.641 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:40.047+0000 7f1d7d400700 1 -- 172.21.15.39:0/4113064090 shutdown_connections 2024-03-18T15:57:40.641 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:40.047+0000 7f1d7d400700 1 --2- 172.21.15.39:0/4113064090 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1d78107c20 0x7f1d7819ddf0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:40.641 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:40.047+0000 7f1d7d400700 1 -- 172.21.15.39:0/4113064090 >> 172.21.15.39:0/4113064090 conn(0x7f1d781031e0 msgr2=0x7f1d78103cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:57:40.641 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:40.047+0000 7f1d7d400700 1 -- 172.21.15.39:0/4113064090 shutdown_connections 2024-03-18T15:57:40.641 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:40.047+0000 7f1d7d400700 1 -- 172.21.15.39:0/4113064090 wait complete. 2024-03-18T15:57:40.641 INFO:teuthology.orchestra.run.smithi039.stdout:Restarting the monitor... 2024-03-18T15:57:41.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:40 smithi039 systemd[1]: Stopping Ceph mon.a for 1892d8fc-e540-11ee-95c9-87774f69a715... 2024-03-18T15:57:41.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:41 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mon-a[31353]: 2024-03-18T15:57:41.201+0000 7f957ee00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T15:57:41.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:41 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mon-a[31353]: 2024-03-18T15:57:41.201+0000 7f957ee00700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-18T15:57:41.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:41 smithi039 podman[31599]: 2024-03-18 15:57:41.294970435 +0000 UTC m=+0.260111274 container died 3dbda31d5bf49a16ebab4a1fc91bb9d9b835801445c7041852ba58ac843d6368 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mon-a, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-18T15:57:42.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:41 smithi039 podman[31599]: 2024-03-18 15:57:41.750395915 +0000 UTC m=+0.715536764 container cleanup 3dbda31d5bf49a16ebab4a1fc91bb9d9b835801445c7041852ba58ac843d6368 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mon-a, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-18T15:57:42.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:41 smithi039 bash[31599]: ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mon-a 2024-03-18T15:57:42.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:42 smithi039 podman[31613]: 2024-03-18 15:57:42.468270966 +0000 UTC m=+1.150719395 container remove 3dbda31d5bf49a16ebab4a1fc91bb9d9b835801445c7041852ba58ac843d6368 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mon-a, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307) 2024-03-18T15:57:43.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:42 smithi039 systemd[1]: ceph-1892d8fc-e540-11ee-95c9-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-18T15:57:43.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:42 smithi039 systemd[1]: Stopped Ceph mon.a for 1892d8fc-e540-11ee-95c9-87774f69a715. 2024-03-18T15:57:43.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:42 smithi039 systemd[1]: ceph-1892d8fc-e540-11ee-95c9-87774f69a715@mon.a.service: Consumed 2.018s CPU time. 2024-03-18T15:57:43.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 systemd[1]: Starting Ceph mon.a for 1892d8fc-e540-11ee-95c9-87774f69a715... 2024-03-18T15:57:43.677 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 podman[31702]: 2024-03-18 15:57:43.297155776 +0000 UTC m=+0.039183091 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T15:57:43.678 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 podman[31702]: 2024-03-18 15:57:43.394221488 +0000 UTC m=+0.136248826 container create c8c8b05ddbd74ac24f563dc2794fd9878b25451a36ad900909048bb8acfa9618 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mon-a, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-18T15:57:43.678 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 podman[31702]: 2024-03-18 15:57:43.622964939 +0000 UTC m=+0.364992281 container init c8c8b05ddbd74ac24f563dc2794fd9878b25451a36ad900909048bb8acfa9618 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mon-a, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-18T15:57:43.678 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 podman[31702]: 2024-03-18 15:57:43.631973416 +0000 UTC m=+0.374000732 container start c8c8b05ddbd74ac24f563dc2794fd9878b25451a36ad900909048bb8acfa9618 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mon-a, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-18T15:57:43.715 INFO:teuthology.orchestra.run.smithi039.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-18T15:57:44.031 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: set uid:gid to 167:167 (ceph:ceph) 2024-03-18T15:57:44.031 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev), process ceph-mon, pid 2 2024-03-18T15:57:44.031 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: pidfile_write: ignore empty --pid-file 2024-03-18T15:57:44.031 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: load: jerasure load: lrc 2024-03-18T15:57:44.031 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: RocksDB version: 7.9.2 2024-03-18T15:57:44.031 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Git sha 0 2024-03-18T15:57:44.032 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Compile date 2024-03-15 00:33:34 2024-03-18T15:57:44.032 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: DB SUMMARY 2024-03-18T15:57:44.032 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: DB Session ID: L20J71SJ2361NEVDCXTL 2024-03-18T15:57:44.032 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: CURRENT file: CURRENT 2024-03-18T15:57:44.032 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: IDENTITY file: IDENTITY 2024-03-18T15:57:44.032 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-18T15:57:44.032 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-18T15:57:44.032 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 103656 ; 2024-03-18T15:57:44.032 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.error_if_exists: 0 2024-03-18T15:57:44.032 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.create_if_missing: 0 2024-03-18T15:57:44.032 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.paranoid_checks: 1 2024-03-18T15:57:44.032 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-18T15:57:44.033 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-18T15:57:44.033 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-18T15:57:44.033 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.env: 0x5614422fdc60 2024-03-18T15:57:44.033 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.fs: PosixFileSystem 2024-03-18T15:57:44.033 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.info_log: 0x561443b03b00 2024-03-18T15:57:44.033 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.max_file_opening_threads: 16 2024-03-18T15:57:44.033 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.statistics: (nil) 2024-03-18T15:57:44.033 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.use_fsync: 0 2024-03-18T15:57:44.033 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.max_log_file_size: 0 2024-03-18T15:57:44.033 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-18T15:57:44.033 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-18T15:57:44.033 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.keep_log_file_num: 1000 2024-03-18T15:57:44.033 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.recycle_log_file_num: 0 2024-03-18T15:57:44.034 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.allow_fallocate: 1 2024-03-18T15:57:44.034 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.allow_mmap_reads: 0 2024-03-18T15:57:44.034 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.allow_mmap_writes: 0 2024-03-18T15:57:44.034 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.use_direct_reads: 0 2024-03-18T15:57:44.034 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-18T15:57:44.034 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.create_missing_column_families: 0 2024-03-18T15:57:44.034 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.db_log_dir: 2024-03-18T15:57:44.034 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.wal_dir: 2024-03-18T15:57:44.034 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-18T15:57:44.034 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-18T15:57:44.034 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-18T15:57:44.034 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-18T15:57:44.034 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-18T15:57:44.035 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-18T15:57:44.035 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.advise_random_on_open: 1 2024-03-18T15:57:44.035 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.db_write_buffer_size: 0 2024-03-18T15:57:44.035 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.write_buffer_manager: 0x561443ad61e0 2024-03-18T15:57:44.035 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-18T15:57:44.035 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-18T15:57:44.035 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-18T15:57:44.035 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.rate_limiter: (nil) 2024-03-18T15:57:44.035 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-18T15:57:44.035 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.wal_recovery_mode: 2 2024-03-18T15:57:44.035 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.enable_thread_tracking: 0 2024-03-18T15:57:44.035 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.enable_pipelined_write: 0 2024-03-18T15:57:44.035 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.unordered_write: 0 2024-03-18T15:57:44.035 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-18T15:57:44.035 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-18T15:57:44.036 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-18T15:57:44.036 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-18T15:57:44.036 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.row_cache: None 2024-03-18T15:57:44.036 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.wal_filter: None 2024-03-18T15:57:44.036 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-18T15:57:44.036 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.allow_ingest_behind: 0 2024-03-18T15:57:44.036 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.two_write_queues: 0 2024-03-18T15:57:44.036 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.manual_wal_flush: 0 2024-03-18T15:57:44.036 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.wal_compression: 0 2024-03-18T15:57:44.036 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.atomic_flush: 0 2024-03-18T15:57:44.036 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-18T15:57:44.036 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-18T15:57:44.036 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-18T15:57:44.036 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.log_readahead_size: 0 2024-03-18T15:57:44.037 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-18T15:57:44.037 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.best_efforts_recovery: 0 2024-03-18T15:57:44.037 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-18T15:57:44.037 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-18T15:57:44.037 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.allow_data_in_errors: 0 2024-03-18T15:57:44.037 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.db_host_id: __hostname__ 2024-03-18T15:57:44.037 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-18T15:57:44.037 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.max_background_jobs: 2 2024-03-18T15:57:44.037 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.max_background_compactions: -1 2024-03-18T15:57:44.037 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.max_subcompactions: 1 2024-03-18T15:57:44.037 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-18T15:57:44.037 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-18T15:57:44.037 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-18T15:57:44.037 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.max_total_wal_size: 0 2024-03-18T15:57:44.038 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-18T15:57:44.038 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-18T15:57:44.038 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-18T15:57:44.038 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-18T15:57:44.038 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.max_open_files: -1 2024-03-18T15:57:44.038 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.bytes_per_sync: 0 2024-03-18T15:57:44.038 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-18T15:57:44.038 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-18T15:57:44.038 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.compaction_readahead_size: 0 2024-03-18T15:57:44.038 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.max_background_flushes: -1 2024-03-18T15:57:44.038 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Compression algorithms supported: 2024-03-18T15:57:44.038 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-18T15:57:44.038 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: kZSTD supported: 0 2024-03-18T15:57:44.038 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: kXpressCompression supported: 0 2024-03-18T15:57:44.039 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: kLZ4HCCompression supported: 1 2024-03-18T15:57:44.039 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: kZlibCompression supported: 1 2024-03-18T15:57:44.039 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: kSnappyCompression supported: 1 2024-03-18T15:57:44.039 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: kLZ4Compression supported: 1 2024-03-18T15:57:44.039 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: kBZip2Compression supported: 0 2024-03-18T15:57:44.039 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-18T15:57:44.039 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-18T15:57:44.039 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-18T15:57:44.039 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-18T15:57:44.039 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-18T15:57:44.039 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.merge_operator: 2024-03-18T15:57:44.039 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.compaction_filter: None 2024-03-18T15:57:44.039 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.compaction_filter_factory: None 2024-03-18T15:57:44.039 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.sst_partitioner_factory: None 2024-03-18T15:57:44.040 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-18T15:57:44.040 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-18T15:57:44.040 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x561443b03c20) 2024-03-18T15:57:44.040 INFO:journalctl@ceph.mon.a.smithi039.stdout: cache_index_and_filter_blocks: 1 2024-03-18T15:57:44.040 INFO:journalctl@ceph.mon.a.smithi039.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-18T15:57:44.040 INFO:journalctl@ceph.mon.a.smithi039.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-18T15:57:44.040 INFO:journalctl@ceph.mon.a.smithi039.stdout: pin_top_level_index_and_filter: 1 2024-03-18T15:57:44.040 INFO:journalctl@ceph.mon.a.smithi039.stdout: index_type: 0 2024-03-18T15:57:44.040 INFO:journalctl@ceph.mon.a.smithi039.stdout: data_block_index_type: 0 2024-03-18T15:57:44.040 INFO:journalctl@ceph.mon.a.smithi039.stdout: index_shortening: 1 2024-03-18T15:57:44.040 INFO:journalctl@ceph.mon.a.smithi039.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-18T15:57:44.040 INFO:journalctl@ceph.mon.a.smithi039.stdout: checksum: 4 2024-03-18T15:57:44.041 INFO:journalctl@ceph.mon.a.smithi039.stdout: no_block_cache: 0 2024-03-18T15:57:44.041 INFO:journalctl@ceph.mon.a.smithi039.stdout: block_cache: 0x561443b09090 2024-03-18T15:57:44.041 INFO:journalctl@ceph.mon.a.smithi039.stdout: block_cache_name: BinnedLRUCache 2024-03-18T15:57:44.041 INFO:journalctl@ceph.mon.a.smithi039.stdout: block_cache_options: 2024-03-18T15:57:44.041 INFO:journalctl@ceph.mon.a.smithi039.stdout: capacity : 536870912 2024-03-18T15:57:44.041 INFO:journalctl@ceph.mon.a.smithi039.stdout: num_shard_bits : 4 2024-03-18T15:57:44.041 INFO:journalctl@ceph.mon.a.smithi039.stdout: strict_capacity_limit : 0 2024-03-18T15:57:44.041 INFO:journalctl@ceph.mon.a.smithi039.stdout: high_pri_pool_ratio: 0.000 2024-03-18T15:57:44.041 INFO:journalctl@ceph.mon.a.smithi039.stdout: block_cache_compressed: (nil) 2024-03-18T15:57:44.041 INFO:journalctl@ceph.mon.a.smithi039.stdout: persistent_cache: (nil) 2024-03-18T15:57:44.041 INFO:journalctl@ceph.mon.a.smithi039.stdout: block_size: 4096 2024-03-18T15:57:44.041 INFO:journalctl@ceph.mon.a.smithi039.stdout: block_size_deviation: 10 2024-03-18T15:57:44.042 INFO:journalctl@ceph.mon.a.smithi039.stdout: block_restart_interval: 16 2024-03-18T15:57:44.042 INFO:journalctl@ceph.mon.a.smithi039.stdout: index_block_restart_interval: 1 2024-03-18T15:57:44.042 INFO:journalctl@ceph.mon.a.smithi039.stdout: metadata_block_size: 4096 2024-03-18T15:57:44.042 INFO:journalctl@ceph.mon.a.smithi039.stdout: partition_filters: 0 2024-03-18T15:57:44.042 INFO:journalctl@ceph.mon.a.smithi039.stdout: use_delta_encoding: 1 2024-03-18T15:57:44.042 INFO:journalctl@ceph.mon.a.smithi039.stdout: filter_policy: bloomfilter 2024-03-18T15:57:44.042 INFO:journalctl@ceph.mon.a.smithi039.stdout: whole_key_filtering: 1 2024-03-18T15:57:44.042 INFO:journalctl@ceph.mon.a.smithi039.stdout: verify_compression: 0 2024-03-18T15:57:44.042 INFO:journalctl@ceph.mon.a.smithi039.stdout: read_amp_bytes_per_bit: 0 2024-03-18T15:57:44.042 INFO:journalctl@ceph.mon.a.smithi039.stdout: format_version: 5 2024-03-18T15:57:44.042 INFO:journalctl@ceph.mon.a.smithi039.stdout: enable_index_compression: 1 2024-03-18T15:57:44.042 INFO:journalctl@ceph.mon.a.smithi039.stdout: block_align: 0 2024-03-18T15:57:44.042 INFO:journalctl@ceph.mon.a.smithi039.stdout: max_auto_readahead_size: 262144 2024-03-18T15:57:44.043 INFO:journalctl@ceph.mon.a.smithi039.stdout: prepopulate_block_cache: 0 2024-03-18T15:57:44.043 INFO:journalctl@ceph.mon.a.smithi039.stdout: initial_auto_readahead_size: 8192 2024-03-18T15:57:44.043 INFO:journalctl@ceph.mon.a.smithi039.stdout: num_file_reads_for_auto_readahead: 2 2024-03-18T15:57:44.043 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.write_buffer_size: 33554432 2024-03-18T15:57:44.043 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.max_write_buffer_number: 2 2024-03-18T15:57:44.043 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.compression: NoCompression 2024-03-18T15:57:44.043 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.bottommost_compression: Disabled 2024-03-18T15:57:44.043 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.prefix_extractor: nullptr 2024-03-18T15:57:44.043 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-18T15:57:44.043 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.num_levels: 7 2024-03-18T15:57:44.043 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-18T15:57:44.043 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-18T15:57:44.043 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-18T15:57:44.043 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-18T15:57:44.043 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-18T15:57:44.044 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-18T15:57:44.044 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-18T15:57:44.044 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-18T15:57:44.044 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-18T15:57:44.044 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-18T15:57:44.044 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-18T15:57:44.044 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-18T15:57:44.044 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-18T15:57:44.044 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.compression_opts.level: 32767 2024-03-18T15:57:44.044 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.compression_opts.strategy: 0 2024-03-18T15:57:44.044 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-18T15:57:44.044 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-18T15:57:44.044 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-18T15:57:44.044 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-18T15:57:44.045 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.compression_opts.enabled: false 2024-03-18T15:57:44.045 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-18T15:57:44.045 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-18T15:57:44.045 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-18T15:57:44.045 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-18T15:57:44.045 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.target_file_size_base: 67108864 2024-03-18T15:57:44.045 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-18T15:57:44.045 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-18T15:57:44.045 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-18T15:57:44.045 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-18T15:57:44.045 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-18T15:57:44.045 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-18T15:57:44.045 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-18T15:57:44.045 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-18T15:57:44.045 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-18T15:57:44.046 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-18T15:57:44.046 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-18T15:57:44.046 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-18T15:57:44.046 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-18T15:57:44.046 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-18T15:57:44.046 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.arena_block_size: 1048576 2024-03-18T15:57:44.046 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-18T15:57:44.046 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-18T15:57:44.046 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.disable_auto_compactions: 0 2024-03-18T15:57:44.046 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-18T15:57:44.046 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-18T15:57:44.046 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-18T15:57:44.046 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-18T15:57:44.046 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-18T15:57:44.047 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-18T15:57:44.047 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-18T15:57:44.047 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-18T15:57:44.047 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-18T15:57:44.047 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-18T15:57:44.047 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.table_properties_collectors: 2024-03-18T15:57:44.047 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.inplace_update_support: 0 2024-03-18T15:57:44.047 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-18T15:57:44.047 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-18T15:57:44.047 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-18T15:57:44.047 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-18T15:57:44.047 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.bloom_locality: 0 2024-03-18T15:57:44.047 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.max_successive_merges: 0 2024-03-18T15:57:44.047 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-18T15:57:44.048 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.paranoid_file_checks: 0 2024-03-18T15:57:44.048 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.force_consistency_checks: 1 2024-03-18T15:57:44.048 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.report_bg_io_stats: 0 2024-03-18T15:57:44.048 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.ttl: 2592000 2024-03-18T15:57:44.048 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-18T15:57:44.048 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-18T15:57:44.048 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-18T15:57:44.048 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.enable_blob_files: false 2024-03-18T15:57:44.048 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.min_blob_size: 0 2024-03-18T15:57:44.048 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.blob_file_size: 268435456 2024-03-18T15:57:44.048 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-18T15:57:44.048 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-18T15:57:44.048 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-18T15:57:44.048 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-18T15:57:44.048 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-18T15:57:44.049 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.blob_file_starting_level: 0 2024-03-18T15:57:44.049 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-18T15:57:44.049 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-18T15:57:44.049 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-18T15:57:44.049 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 15670f8a-d8a7-4a03-a350-bf51e71254fe 2024-03-18T15:57:44.049 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710777463687942, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-18T15:57:44.049 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-18T15:57:44.049 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710777463689871, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 98892, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 320, "table_properties": {"data_size": 96861, "index_size": 313, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13820, "raw_average_key_size": 49, "raw_value_size": 89311, "raw_average_value_size": 320, "num_data_blocks": 12, "num_entries": 279, "num_filter_entries": 279, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710777463, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "15670f8a-d8a7-4a03-a350-bf51e71254fe", "db_session_id": "L20J71SJ2361NEVDCXTL", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-18T15:57:44.049 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710777463690017, "job": 1, "event": "recovery_finished"} 2024-03-18T15:57:44.049 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-18T15:57:44.049 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-18T15:57:44.049 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x561443bfc000 2024-03-18T15:57:44.049 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: DB pointer 0x561443be6000 2024-03-18T15:57:44.050 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-18T15:57:44.050 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-18T15:57:44.050 INFO:journalctl@ceph.mon.a.smithi039.stdout: ** DB Stats ** 2024-03-18T15:57:44.050 INFO:journalctl@ceph.mon.a.smithi039.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-18T15:57:44.050 INFO:journalctl@ceph.mon.a.smithi039.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-18T15:57:44.050 INFO:journalctl@ceph.mon.a.smithi039.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-18T15:57:44.050 INFO:journalctl@ceph.mon.a.smithi039.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-18T15:57:44.050 INFO:journalctl@ceph.mon.a.smithi039.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-18T15:57:44.050 INFO:journalctl@ceph.mon.a.smithi039.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-18T15:57:44.050 INFO:journalctl@ceph.mon.a.smithi039.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-18T15:57:44.050 INFO:journalctl@ceph.mon.a.smithi039.stdout: 2024-03-18T15:57:44.050 INFO:journalctl@ceph.mon.a.smithi039.stdout: ** Compaction Stats [default] ** 2024-03-18T15:57:44.050 INFO:journalctl@ceph.mon.a.smithi039.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-18T15:57:44.050 INFO:journalctl@ceph.mon.a.smithi039.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-18T15:57:44.051 INFO:journalctl@ceph.mon.a.smithi039.stdout: L0 2/0 98.41 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 75.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-18T15:57:44.051 INFO:journalctl@ceph.mon.a.smithi039.stdout: Sum 2/0 98.41 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 75.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-18T15:57:44.051 INFO:journalctl@ceph.mon.a.smithi039.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 75.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-18T15:57:44.051 INFO:journalctl@ceph.mon.a.smithi039.stdout: 2024-03-18T15:57:44.051 INFO:journalctl@ceph.mon.a.smithi039.stdout: ** Compaction Stats [default] ** 2024-03-18T15:57:44.051 INFO:journalctl@ceph.mon.a.smithi039.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-18T15:57:44.051 INFO:journalctl@ceph.mon.a.smithi039.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-18T15:57:44.051 INFO:journalctl@ceph.mon.a.smithi039.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 75.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-18T15:57:44.051 INFO:journalctl@ceph.mon.a.smithi039.stdout: 2024-03-18T15:57:44.051 INFO:journalctl@ceph.mon.a.smithi039.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-18T15:57:44.051 INFO:journalctl@ceph.mon.a.smithi039.stdout: 2024-03-18T15:57:44.051 INFO:journalctl@ceph.mon.a.smithi039.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-18T15:57:44.051 INFO:journalctl@ceph.mon.a.smithi039.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-18T15:57:44.051 INFO:journalctl@ceph.mon.a.smithi039.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-18T15:57:44.051 INFO:journalctl@ceph.mon.a.smithi039.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-18T15:57:44.052 INFO:journalctl@ceph.mon.a.smithi039.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-18T15:57:44.052 INFO:journalctl@ceph.mon.a.smithi039.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-18T15:57:44.052 INFO:journalctl@ceph.mon.a.smithi039.stdout: Cumulative compaction: 0.00 GB write, 13.49 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-18T15:57:44.052 INFO:journalctl@ceph.mon.a.smithi039.stdout: Interval compaction: 0.00 GB write, 13.49 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-18T15:57:44.052 INFO:journalctl@ceph.mon.a.smithi039.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-18T15:57:44.052 INFO:journalctl@ceph.mon.a.smithi039.stdout: Block cache BinnedLRUCache@0x561443b09090#2 capacity: 512.00 MB usage: 1.39 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-03-18T15:57:44.052 INFO:journalctl@ceph.mon.a.smithi039.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-03-18T15:57:44.052 INFO:journalctl@ceph.mon.a.smithi039.stdout: 2024-03-18T15:57:44.052 INFO:journalctl@ceph.mon.a.smithi039.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-18T15:57:44.052 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: starting mon.a rank 0 at public addrs [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] at bind addrs [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 1892d8fc-e540-11ee-95c9-87774f69a715 2024-03-18T15:57:44.052 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: mon.a@-1(???) e1 preinit fsid 1892d8fc-e540-11ee-95c9-87774f69a715 2024-03-18T15:57:44.052 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: mon.a@-1(???).mds e1 new map 2024-03-18T15:57:44.052 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: mon.a@-1(???).mds e1 print_map 2024-03-18T15:57:44.052 INFO:journalctl@ceph.mon.a.smithi039.stdout: e1 2024-03-18T15:57:44.053 INFO:journalctl@ceph.mon.a.smithi039.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-18T15:57:44.053 INFO:journalctl@ceph.mon.a.smithi039.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-18T15:57:44.053 INFO:journalctl@ceph.mon.a.smithi039.stdout: legacy client fscid: -1 2024-03-18T15:57:44.053 INFO:journalctl@ceph.mon.a.smithi039.stdout: 2024-03-18T15:57:44.053 INFO:journalctl@ceph.mon.a.smithi039.stdout: No filesystems configured 2024-03-18T15:57:44.053 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-18T15:57:44.053 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-18T15:57:44.053 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-18T15:57:44.053 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-18T15:57:44.053 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-18T15:57:44.053 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 bash[31702]: c8c8b05ddbd74ac24f563dc2794fd9878b25451a36ad900909048bb8acfa9618 2024-03-18T15:57:44.053 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 systemd[1]: Started Ceph mon.a for 1892d8fc-e540-11ee-95c9-87774f69a715. 2024-03-18T15:57:44.053 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-18T15:57:44.053 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: monmap epoch 1 2024-03-18T15:57:44.053 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: fsid 1892d8fc-e540-11ee-95c9-87774f69a715 2024-03-18T15:57:44.054 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: last_changed 2024-03-18T15:57:32.305376+0000 2024-03-18T15:57:44.054 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: created 2024-03-18T15:57:32.305376+0000 2024-03-18T15:57:44.054 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: min_mon_release 19 (squid) 2024-03-18T15:57:44.054 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: election_strategy: 1 2024-03-18T15:57:44.054 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: 0: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.a 2024-03-18T15:57:44.054 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: fsmap 2024-03-18T15:57:44.054 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: osdmap e1: 0 total, 0 up, 0 in 2024-03-18T15:57:44.054 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:43 smithi039 ceph-mon[31721]: mgrmap e1: no daemons active 2024-03-18T15:57:45.130 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:44.339+0000 7f00fe200700 1 Processor -- start 2024-03-18T15:57:45.130 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:44.339+0000 7f00fe200700 1 -- start start 2024-03-18T15:57:45.130 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:44.340+0000 7f00fe200700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00f807cd00 0x7f00f807b160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:45.130 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:44.340+0000 7f00fe200700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f00f807b730 con 0x7f00f807cd00 2024-03-18T15:57:45.130 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:44.340+0000 7f00f7400700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00f807cd00 0x7f00f807b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:45.130 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:44.341+0000 7f00f7400700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00f807cd00 0x7f00f807b160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:36370/0 (socket says 172.21.15.39:36370) 2024-03-18T15:57:45.130 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:44.341+0000 7f00f7400700 1 -- 172.21.15.39:0/1530219400 learned_addr learned my addr 172.21.15.39:0/1530219400 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T15:57:45.130 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:44.341+0000 7f00f7400700 1 -- 172.21.15.39:0/1530219400 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00f807b8b0 con 0x7f00f807cd00 2024-03-18T15:57:45.130 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:44.342+0000 7f00f7400700 1 --2- 172.21.15.39:0/1530219400 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00f807cd00 0x7f00f807b160 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f00e80098a0 tx=0x7f00e8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ec533940e67b0b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:45.130 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:44.342+0000 7f00f6000700 1 -- 172.21.15.39:0/1530219400 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f00e8004160 con 0x7f00f807cd00 2024-03-18T15:57:45.131 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:44.342+0000 7f00f6000700 1 -- 172.21.15.39:0/1530219400 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f00e8004300 con 0x7f00f807cd00 2024-03-18T15:57:45.131 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:44.342+0000 7f00f6000700 1 -- 172.21.15.39:0/1530219400 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f00e8011b50 con 0x7f00f807cd00 2024-03-18T15:57:45.131 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:44.343+0000 7f00fe200700 1 -- 172.21.15.39:0/1530219400 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00f807cd00 msgr2=0x7f00f807b160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:45.131 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:44.343+0000 7f00fe200700 1 --2- 172.21.15.39:0/1530219400 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00f807cd00 0x7f00f807b160 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f00e80098a0 tx=0x7f00e8009bb0 comp rx=0 tx=0).stop 2024-03-18T15:57:45.131 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:44.343+0000 7f00fe200700 1 -- 172.21.15.39:0/1530219400 shutdown_connections 2024-03-18T15:57:45.131 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:44.343+0000 7f00fe200700 1 --2- 172.21.15.39:0/1530219400 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00f807cd00 0x7f00f807b160 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:45.131 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:44.343+0000 7f00fe200700 1 -- 172.21.15.39:0/1530219400 >> 172.21.15.39:0/1530219400 conn(0x7f00f81031e0 msgr2=0x7f00f8105600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:57:45.131 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:44.343+0000 7f00fe200700 1 -- 172.21.15.39:0/1530219400 shutdown_connections 2024-03-18T15:57:45.131 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:44.343+0000 7f00fe200700 1 -- 172.21.15.39:0/1530219400 wait complete. 2024-03-18T15:57:45.132 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:44.344+0000 7f00fe200700 1 Processor -- start 2024-03-18T15:57:45.132 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:44.345+0000 7f00fe200700 1 -- start start 2024-03-18T15:57:45.132 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:44.345+0000 7f00fe200700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00f807cd00 0x7f00f819dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:45.132 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:44.345+0000 7f00fe200700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f00f810e1f0 con 0x7f00f807cd00 2024-03-18T15:57:45.132 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:44.345+0000 7f00f7400700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00f807cd00 0x7f00f819dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:45.132 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:44.345+0000 7f00f7400700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00f807cd00 0x7f00f819dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:36386/0 (socket says 172.21.15.39:36386) 2024-03-18T15:57:45.132 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:44.345+0000 7f00f7400700 1 -- 172.21.15.39:0/3950811353 learned_addr learned my addr 172.21.15.39:0/3950811353 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T15:57:45.132 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:44.346+0000 7f00f7400700 1 -- 172.21.15.39:0/3950811353 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00f819e2e0 con 0x7f00f807cd00 2024-03-18T15:57:45.132 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:44.346+0000 7f00f7400700 1 --2- 172.21.15.39:0/3950811353 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00f807cd00 0x7f00f819dda0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f00e8009fb0 tx=0x7f00e8011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:45.132 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:44.346+0000 7f00efe00700 1 -- 172.21.15.39:0/3950811353 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f00e80038b0 con 0x7f00f807cd00 2024-03-18T15:57:45.132 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:44.346+0000 7f00efe00700 1 -- 172.21.15.39:0/3950811353 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f00e8011e50 con 0x7f00f807cd00 2024-03-18T15:57:45.132 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:44.346+0000 7f00efe00700 1 -- 172.21.15.39:0/3950811353 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f00e8019420 con 0x7f00f807cd00 2024-03-18T15:57:45.132 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:44.346+0000 7f00fe200700 1 -- 172.21.15.39:0/3950811353 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00f819e570 con 0x7f00f807cd00 2024-03-18T15:57:45.133 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:44.347+0000 7f00fe200700 1 -- 172.21.15.39:0/3950811353 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00f81a1260 con 0x7f00f807cd00 2024-03-18T15:57:45.133 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:44.347+0000 7f00efe00700 1 -- 172.21.15.39:0/3950811353 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f00e8019ab0 con 0x7f00f807cd00 2024-03-18T15:57:45.133 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:44.347+0000 7f00efe00700 1 -- 172.21.15.39:0/3950811353 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f00e801a430 con 0x7f00f807cd00 2024-03-18T15:57:45.133 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:44.348+0000 7f00fe200700 1 -- 172.21.15.39:0/3950811353 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00f8063900 con 0x7f00f807cd00 2024-03-18T15:57:45.133 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:44.351+0000 7f00efe00700 1 -- 172.21.15.39:0/3950811353 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f00e8003a50 con 0x7f00f807cd00 2024-03-18T15:57:45.133 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:44.428+0000 7f00fe200700 1 -- 172.21.15.39:0/3950811353 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7f00f81a1430 con 0x7f00f807cd00 2024-03-18T15:57:45.133 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:44.437+0000 7f00efe00700 1 -- 172.21.15.39:0/3950811353 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7f00e8032030 con 0x7f00f807cd00 2024-03-18T15:57:45.133 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:44.438+0000 7f00fe200700 1 -- 172.21.15.39:0/3950811353 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00f807cd00 msgr2=0x7f00f819dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:45.133 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:44.438+0000 7f00fe200700 1 --2- 172.21.15.39:0/3950811353 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00f807cd00 0x7f00f819dda0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f00e8009fb0 tx=0x7f00e8011bf0 comp rx=0 tx=0).stop 2024-03-18T15:57:45.134 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:44.439+0000 7f00fe200700 1 -- 172.21.15.39:0/3950811353 shutdown_connections 2024-03-18T15:57:45.134 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:44.439+0000 7f00fe200700 1 --2- 172.21.15.39:0/3950811353 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00f807cd00 0x7f00f819dda0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:45.134 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:44.439+0000 7f00fe200700 1 -- 172.21.15.39:0/3950811353 >> 172.21.15.39:0/3950811353 conn(0x7f00f81031e0 msgr2=0x7f00f8103d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:57:45.134 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:44.439+0000 7f00fe200700 1 -- 172.21.15.39:0/3950811353 shutdown_connections 2024-03-18T15:57:45.134 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:44.439+0000 7f00fe200700 1 -- 172.21.15.39:0/3950811353 wait complete. 2024-03-18T15:57:45.135 INFO:teuthology.orchestra.run.smithi039.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-18T15:57:45.153 INFO:teuthology.orchestra.run.smithi039.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-18T15:57:45.153 INFO:teuthology.orchestra.run.smithi039.stdout:Creating mgr... 2024-03-18T15:57:45.154 INFO:teuthology.orchestra.run.smithi039.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-18T15:57:45.155 INFO:teuthology.orchestra.run.smithi039.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-18T15:57:45.588 INFO:teuthology.orchestra.run.smithi039.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-1892d8fc-e540-11ee-95c9-87774f69a715@mgr.x 2024-03-18T15:57:45.588 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stderr Failed to reset failed state of unit ceph-1892d8fc-e540-11ee-95c9-87774f69a715@mgr.x.service: Unit ceph-1892d8fc-e540-11ee-95c9-87774f69a715@mgr.x.service not loaded. 2024-03-18T15:57:45.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:45 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/3950811353' entity='client.admin' 2024-03-18T15:57:45.849 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-1892d8fc-e540-11ee-95c9-87774f69a715.target.wants/ceph-1892d8fc-e540-11ee-95c9-87774f69a715@mgr.x.service → /etc/systemd/system/ceph-1892d8fc-e540-11ee-95c9-87774f69a715@.service. 2024-03-18T15:57:46.238 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:57:45 smithi039 systemd[1]: Starting Ceph mgr.x for 1892d8fc-e540-11ee-95c9-87774f69a715... 2024-03-18T15:57:46.498 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:57:46 smithi039 podman[31944]: 2024-03-18 15:57:46.143683389 +0000 UTC m=+0.034595890 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af 2024-03-18T15:57:46.498 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:57:46 smithi039 podman[31944]: 2024-03-18 15:57:46.255890302 +0000 UTC m=+0.146802851 container create 0ad604841d44848bd8834bb6b35f810ea8be6ba0a381cb69c436dacfc47017b9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T15:57:46.498 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:57:46 smithi039 podman[31944]: 2024-03-18 15:57:46.496484933 +0000 UTC m=+0.387397505 container init 0ad604841d44848bd8834bb6b35f810ea8be6ba0a381cb69c436dacfc47017b9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T15:57:46.615 INFO:teuthology.orchestra.run.smithi039.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-18T15:57:46.615 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stdout disabled 2024-03-18T15:57:46.634 INFO:teuthology.orchestra.run.smithi039.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-18T15:57:46.634 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stdout inactive 2024-03-18T15:57:46.634 INFO:teuthology.orchestra.run.smithi039.stdout:firewalld.service is not enabled 2024-03-18T15:57:46.634 INFO:teuthology.orchestra.run.smithi039.stdout:Not possible to enable service . firewalld.service is not available 2024-03-18T15:57:46.658 INFO:teuthology.orchestra.run.smithi039.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-18T15:57:46.659 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stdout disabled 2024-03-18T15:57:46.680 INFO:teuthology.orchestra.run.smithi039.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-18T15:57:46.680 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stdout inactive 2024-03-18T15:57:46.681 INFO:teuthology.orchestra.run.smithi039.stdout:firewalld.service is not enabled 2024-03-18T15:57:46.681 INFO:teuthology.orchestra.run.smithi039.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-18T15:57:46.681 INFO:teuthology.orchestra.run.smithi039.stdout:Waiting for mgr to start... 2024-03-18T15:57:46.681 INFO:teuthology.orchestra.run.smithi039.stdout:Waiting for mgr... 2024-03-18T15:57:46.831 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:57:46 smithi039 podman[31944]: 2024-03-18 15:57:46.506574105 +0000 UTC m=+0.397486641 container start 0ad604841d44848bd8834bb6b35f810ea8be6ba0a381cb69c436dacfc47017b9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-18T15:57:46.831 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:57:46 smithi039 bash[31944]: 0ad604841d44848bd8834bb6b35f810ea8be6ba0a381cb69c436dacfc47017b9 2024-03-18T15:57:46.831 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:57:46 smithi039 systemd[1]: Started Ceph mgr.x for 1892d8fc-e540-11ee-95c9-87774f69a715. 2024-03-18T15:57:47.366 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:57:47 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T15:57:47.319+0000 7ff476a9b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T15:57:47.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:47 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/1039665941' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-18T15:57:47.836 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:57:47 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T15:57:47.462+0000 7ff476a9b200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T15:57:48.256 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout 2024-03-18T15:57:48.256 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout { 2024-03-18T15:57:48.257 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "fsid": "1892d8fc-e540-11ee-95c9-87774f69a715", 2024-03-18T15:57:48.257 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "health": { 2024-03-18T15:57:48.257 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-18T15:57:48.257 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-18T15:57:48.258 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-18T15:57:48.258 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2024-03-18T15:57:48.258 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-18T15:57:48.258 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-18T15:57:48.260 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout 0 2024-03-18T15:57:48.260 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout ], 2024-03-18T15:57:48.260 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-18T15:57:48.260 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "a" 2024-03-18T15:57:48.260 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout ], 2024-03-18T15:57:48.260 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "quorum_age": 3, 2024-03-18T15:57:48.260 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-18T15:57:48.260 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T15:57:48.261 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-18T15:57:48.261 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-18T15:57:48.261 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2024-03-18T15:57:48.261 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-18T15:57:48.261 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T15:57:48.261 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-18T15:57:48.261 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-18T15:57:48.261 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-18T15:57:48.261 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-18T15:57:48.261 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-18T15:57:48.264 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-18T15:57:48.264 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2024-03-18T15:57:48.265 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-18T15:57:48.265 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-18T15:57:48.265 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-18T15:57:48.265 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-18T15:57:48.265 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-18T15:57:48.265 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-18T15:57:48.265 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-18T15:57:48.265 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-18T15:57:48.265 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-18T15:57:48.265 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2024-03-18T15:57:48.265 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-18T15:57:48.265 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T15:57:48.265 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-18T15:57:48.265 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-18T15:57:48.266 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2024-03-18T15:57:48.266 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-18T15:57:48.266 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-18T15:57:48.266 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-18T15:57:48.266 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-18T15:57:48.266 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-18T15:57:48.266 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-18T15:57:48.266 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "restful" 2024-03-18T15:57:48.266 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout ], 2024-03-18T15:57:48.266 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-18T15:57:48.266 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2024-03-18T15:57:48.266 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-18T15:57:48.266 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T15:57:48.266 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "modified": "2024-03-18T15:57:36.508025+0000", 2024-03-18T15:57:48.271 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-18T15:57:48.271 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2024-03-18T15:57:48.271 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-18T15:57:48.271 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout } 2024-03-18T15:57:48.272 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:47.271+0000 7fa7bac00700 1 Processor -- start 2024-03-18T15:57:48.272 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:47.271+0000 7fa7bac00700 1 -- start start 2024-03-18T15:57:48.272 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:47.272+0000 7fa7bac00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7b4106510 0x7fa7b4106930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:48.272 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:47.272+0000 7fa7bac00700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fa7b4075950 con 0x7fa7b4106510 2024-03-18T15:57:48.272 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:47.272+0000 7fa7b3e00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7b4106510 0x7fa7b4106930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:48.272 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:47.272+0000 7fa7b3e00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7b4106510 0x7fa7b4106930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:36404/0 (socket says 172.21.15.39:36404) 2024-03-18T15:57:48.272 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:47.272+0000 7fa7b3e00700 1 -- 172.21.15.39:0/3762858390 learned_addr learned my addr 172.21.15.39:0/3762858390 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T15:57:48.272 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:47.273+0000 7fa7b3e00700 1 -- 172.21.15.39:0/3762858390 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7b4106e70 con 0x7fa7b4106510 2024-03-18T15:57:48.272 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:47.273+0000 7fa7b3e00700 1 --2- 172.21.15.39:0/3762858390 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7b4106510 0x7fa7b4106930 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fa7a40098a0 tx=0x7fa7a4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b43a6317f99bc6d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:48.272 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:47.273+0000 7fa7b2a00700 1 -- 172.21.15.39:0/3762858390 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa7a4004160 con 0x7fa7b4106510 2024-03-18T15:57:48.272 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:47.273+0000 7fa7b2a00700 1 -- 172.21.15.39:0/3762858390 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fa7a4004300 con 0x7fa7b4106510 2024-03-18T15:57:48.272 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:47.273+0000 7fa7b2a00700 1 -- 172.21.15.39:0/3762858390 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa7a4011b50 con 0x7fa7b4106510 2024-03-18T15:57:48.272 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:47.274+0000 7fa7bac00700 1 -- 172.21.15.39:0/3762858390 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7b4106510 msgr2=0x7fa7b4106930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:48.273 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:47.274+0000 7fa7bac00700 1 --2- 172.21.15.39:0/3762858390 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7b4106510 0x7fa7b4106930 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fa7a40098a0 tx=0x7fa7a4009bb0 comp rx=0 tx=0).stop 2024-03-18T15:57:48.273 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:47.274+0000 7fa7bac00700 1 -- 172.21.15.39:0/3762858390 shutdown_connections 2024-03-18T15:57:48.273 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:47.274+0000 7fa7bac00700 1 --2- 172.21.15.39:0/3762858390 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7b4106510 0x7fa7b4106930 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:48.273 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:47.274+0000 7fa7bac00700 1 -- 172.21.15.39:0/3762858390 >> 172.21.15.39:0/3762858390 conn(0x7fa7b41020d0 msgr2=0x7fa7b41044f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:57:48.273 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:47.275+0000 7fa7bac00700 1 -- 172.21.15.39:0/3762858390 shutdown_connections 2024-03-18T15:57:48.273 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:47.275+0000 7fa7bac00700 1 -- 172.21.15.39:0/3762858390 wait complete. 2024-03-18T15:57:48.273 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:47.276+0000 7fa7bac00700 1 Processor -- start 2024-03-18T15:57:48.273 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:47.276+0000 7fa7bac00700 1 -- start start 2024-03-18T15:57:48.273 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:47.276+0000 7fa7bac00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7b4106510 0x7fa7b419dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:48.273 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:47.276+0000 7fa7bac00700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fa7b4107e90 con 0x7fa7b4106510 2024-03-18T15:57:48.273 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:47.277+0000 7fa7b3e00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7b4106510 0x7fa7b419dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:48.273 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:47.277+0000 7fa7b3e00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7b4106510 0x7fa7b419dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:36420/0 (socket says 172.21.15.39:36420) 2024-03-18T15:57:48.273 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:47.277+0000 7fa7b3e00700 1 -- 172.21.15.39:0/1039665941 learned_addr learned my addr 172.21.15.39:0/1039665941 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T15:57:48.273 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:47.277+0000 7fa7b3e00700 1 -- 172.21.15.39:0/1039665941 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7b419e250 con 0x7fa7b4106510 2024-03-18T15:57:48.274 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:47.277+0000 7fa7b3e00700 1 --2- 172.21.15.39:0/1039665941 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7b4106510 0x7fa7b419dd10 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fa7a4011670 tx=0x7fa7a4011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:48.274 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:47.278+0000 7fa7b0c00700 1 -- 172.21.15.39:0/1039665941 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa7a40039c0 con 0x7fa7b4106510 2024-03-18T15:57:48.274 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:47.278+0000 7fa7b0c00700 1 -- 172.21.15.39:0/1039665941 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fa7a4011e50 con 0x7fa7b4106510 2024-03-18T15:57:48.274 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:47.278+0000 7fa7bac00700 1 -- 172.21.15.39:0/1039665941 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7b419e4e0 con 0x7fa7b4106510 2024-03-18T15:57:48.274 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:47.278+0000 7fa7b0c00700 1 -- 172.21.15.39:0/1039665941 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa7a401a5c0 con 0x7fa7b4106510 2024-03-18T15:57:48.274 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:47.278+0000 7fa7bac00700 1 -- 172.21.15.39:0/1039665941 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7b41a11c0 con 0x7fa7b4106510 2024-03-18T15:57:48.274 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:47.278+0000 7fa7b0c00700 1 -- 172.21.15.39:0/1039665941 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fa7a401ac00 con 0x7fa7b4106510 2024-03-18T15:57:48.274 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:47.278+0000 7fa7b0c00700 1 -- 172.21.15.39:0/1039665941 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fa7a4019700 con 0x7fa7b4106510 2024-03-18T15:57:48.274 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:47.279+0000 7fa7bac00700 1 -- 172.21.15.39:0/1039665941 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7b4063900 con 0x7fa7b4106510 2024-03-18T15:57:48.274 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:47.282+0000 7fa7b0c00700 1 -- 172.21.15.39:0/1039665941 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fa7a4003b60 con 0x7fa7b4106510 2024-03-18T15:57:48.274 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:47.359+0000 7fa7bac00700 1 -- 172.21.15.39:0/1039665941 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fa7b402ddc0 con 0x7fa7b4106510 2024-03-18T15:57:48.274 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:47.359+0000 7fa7b0c00700 1 -- 172.21.15.39:0/1039665941 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fa7a40199a0 con 0x7fa7b4106510 2024-03-18T15:57:48.274 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:47.361+0000 7fa7bac00700 1 -- 172.21.15.39:0/1039665941 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7b4106510 msgr2=0x7fa7b419dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:48.275 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:47.361+0000 7fa7bac00700 1 --2- 172.21.15.39:0/1039665941 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7b4106510 0x7fa7b419dd10 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fa7a4011670 tx=0x7fa7a4011b60 comp rx=0 tx=0).stop 2024-03-18T15:57:48.275 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:47.361+0000 7fa7bac00700 1 -- 172.21.15.39:0/1039665941 shutdown_connections 2024-03-18T15:57:48.275 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:47.361+0000 7fa7bac00700 1 --2- 172.21.15.39:0/1039665941 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7b4106510 0x7fa7b419dd10 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:48.276 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:47.361+0000 7fa7bac00700 1 -- 172.21.15.39:0/1039665941 >> 172.21.15.39:0/1039665941 conn(0x7fa7b41020d0 msgr2=0x7fa7b4194970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:57:48.276 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:47.361+0000 7fa7bac00700 1 -- 172.21.15.39:0/1039665941 shutdown_connections 2024-03-18T15:57:48.276 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:47.361+0000 7fa7bac00700 1 -- 172.21.15.39:0/1039665941 wait complete. 2024-03-18T15:57:48.276 INFO:teuthology.orchestra.run.smithi039.stdout:mgr not available, waiting (1/15)... 2024-03-18T15:57:48.587 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:57:48 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T15:57:48.314+0000 7ff476a9b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T15:57:48.587 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:57:48 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T15:57:48.440+0000 7ff476a9b200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T15:57:49.086 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:57:48 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T15:57:48.625+0000 7ff476a9b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T15:57:49.086 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:57:48 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T15:57:48.753+0000 7ff476a9b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T15:57:49.086 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:57:48 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T15:57:48.867+0000 7ff476a9b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T15:57:49.473 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:57:49 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T15:57:49.211+0000 7ff476a9b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T15:57:49.473 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:57:49 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T15:57:49.355+0000 7ff476a9b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T15:57:49.821 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:57:49 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T15:57:49.470+0000 7ff476a9b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T15:57:49.821 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:57:49 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T15:57:49.597+0000 7ff476a9b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T15:57:49.821 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:57:49 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T15:57:49.818+0000 7ff476a9b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T15:57:51.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:51 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/1434152056' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-18T15:57:51.336 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:57:51 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T15:57:51.209+0000 7ff476a9b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T15:57:51.640 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:57:51 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T15:57:51.396+0000 7ff476a9b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T15:57:51.649 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout 2024-03-18T15:57:51.660 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout { 2024-03-18T15:57:51.660 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "fsid": "1892d8fc-e540-11ee-95c9-87774f69a715", 2024-03-18T15:57:51.660 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "health": { 2024-03-18T15:57:51.660 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-18T15:57:51.660 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-18T15:57:51.660 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-18T15:57:51.660 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2024-03-18T15:57:51.660 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-18T15:57:51.660 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-18T15:57:51.660 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout 0 2024-03-18T15:57:51.660 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout ], 2024-03-18T15:57:51.660 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-18T15:57:51.661 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "a" 2024-03-18T15:57:51.661 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout ], 2024-03-18T15:57:51.661 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "quorum_age": 7, 2024-03-18T15:57:51.661 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-18T15:57:51.661 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T15:57:51.661 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-18T15:57:51.661 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-18T15:57:51.661 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2024-03-18T15:57:51.661 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-18T15:57:51.661 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T15:57:51.661 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-18T15:57:51.661 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-18T15:57:51.661 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-18T15:57:51.661 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-18T15:57:51.661 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-18T15:57:51.662 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-18T15:57:51.662 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2024-03-18T15:57:51.662 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-18T15:57:51.662 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-18T15:57:51.662 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-18T15:57:51.662 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-18T15:57:51.662 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-18T15:57:51.662 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-18T15:57:51.662 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-18T15:57:51.662 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-18T15:57:51.662 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-18T15:57:51.662 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2024-03-18T15:57:51.662 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-18T15:57:51.662 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T15:57:51.663 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-18T15:57:51.663 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-18T15:57:51.663 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2024-03-18T15:57:51.663 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-18T15:57:51.663 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-18T15:57:51.663 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-18T15:57:51.663 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-18T15:57:51.663 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-18T15:57:51.663 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-18T15:57:51.663 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "restful" 2024-03-18T15:57:51.663 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout ], 2024-03-18T15:57:51.663 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-18T15:57:51.663 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2024-03-18T15:57:51.663 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-18T15:57:51.663 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T15:57:51.664 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "modified": "2024-03-18T15:57:36.508025+0000", 2024-03-18T15:57:51.664 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-18T15:57:51.664 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2024-03-18T15:57:51.664 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-18T15:57:51.664 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout } 2024-03-18T15:57:51.664 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:50.885+0000 7fd439e00700 1 Processor -- start 2024-03-18T15:57:51.664 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:50.885+0000 7fd439e00700 1 -- start start 2024-03-18T15:57:51.664 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:50.885+0000 7fd439e00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd4341064f0 0x7fd434106910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:51.664 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:50.885+0000 7fd439e00700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fd434075970 con 0x7fd4341064f0 2024-03-18T15:57:51.664 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:50.886+0000 7fd432a00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd4341064f0 0x7fd434106910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:51.664 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:50.886+0000 7fd432a00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd4341064f0 0x7fd434106910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58920/0 (socket says 172.21.15.39:58920) 2024-03-18T15:57:51.664 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:50.886+0000 7fd432a00700 1 -- 172.21.15.39:0/1456223193 learned_addr learned my addr 172.21.15.39:0/1456223193 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T15:57:51.664 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:50.886+0000 7fd432a00700 1 -- 172.21.15.39:0/1456223193 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd434106e50 con 0x7fd4341064f0 2024-03-18T15:57:51.664 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:50.887+0000 7fd432a00700 1 --2- 172.21.15.39:0/1456223193 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd4341064f0 0x7fd434106910 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd4240098a0 tx=0x7fd424009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19896f4e35afb40a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:51.664 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:50.887+0000 7fd431600700 1 -- 172.21.15.39:0/1456223193 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd424004160 con 0x7fd4341064f0 2024-03-18T15:57:51.665 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:50.887+0000 7fd431600700 1 -- 172.21.15.39:0/1456223193 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fd424004300 con 0x7fd4341064f0 2024-03-18T15:57:51.665 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:50.887+0000 7fd431600700 1 -- 172.21.15.39:0/1456223193 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd424011b50 con 0x7fd4341064f0 2024-03-18T15:57:51.665 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:50.888+0000 7fd439e00700 1 -- 172.21.15.39:0/1456223193 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd4341064f0 msgr2=0x7fd434106910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:51.665 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:50.888+0000 7fd439e00700 1 --2- 172.21.15.39:0/1456223193 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd4341064f0 0x7fd434106910 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd4240098a0 tx=0x7fd424009bb0 comp rx=0 tx=0).stop 2024-03-18T15:57:51.665 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:50.888+0000 7fd439e00700 1 -- 172.21.15.39:0/1456223193 shutdown_connections 2024-03-18T15:57:51.665 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:50.888+0000 7fd439e00700 1 --2- 172.21.15.39:0/1456223193 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd4341064f0 0x7fd434106910 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:51.665 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:50.888+0000 7fd439e00700 1 -- 172.21.15.39:0/1456223193 >> 172.21.15.39:0/1456223193 conn(0x7fd4341020d0 msgr2=0x7fd4341044f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:57:51.665 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:50.888+0000 7fd439e00700 1 -- 172.21.15.39:0/1456223193 shutdown_connections 2024-03-18T15:57:51.665 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:50.888+0000 7fd439e00700 1 -- 172.21.15.39:0/1456223193 wait complete. 2024-03-18T15:57:51.665 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:50.889+0000 7fd439e00700 1 Processor -- start 2024-03-18T15:57:51.665 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:50.890+0000 7fd439e00700 1 -- start start 2024-03-18T15:57:51.665 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:50.890+0000 7fd439e00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd4341064f0 0x7fd43419dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:51.665 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:50.890+0000 7fd439e00700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fd434107e70 con 0x7fd4341064f0 2024-03-18T15:57:51.665 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:50.890+0000 7fd432a00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd4341064f0 0x7fd43419dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:51.666 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:50.890+0000 7fd432a00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd4341064f0 0x7fd43419dd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58936/0 (socket says 172.21.15.39:58936) 2024-03-18T15:57:51.666 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:50.890+0000 7fd432a00700 1 -- 172.21.15.39:0/1434152056 learned_addr learned my addr 172.21.15.39:0/1434152056 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T15:57:51.666 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:50.891+0000 7fd432a00700 1 -- 172.21.15.39:0/1434152056 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd43419e290 con 0x7fd4341064f0 2024-03-18T15:57:51.666 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:50.891+0000 7fd432a00700 1 --2- 172.21.15.39:0/1434152056 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd4341064f0 0x7fd43419dd50 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd424011b60 tx=0x7fd424011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:51.666 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:50.891+0000 7fd438a00700 1 -- 172.21.15.39:0/1434152056 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd424003b20 con 0x7fd4341064f0 2024-03-18T15:57:51.666 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:50.891+0000 7fd438a00700 1 -- 172.21.15.39:0/1434152056 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fd424003cc0 con 0x7fd4341064f0 2024-03-18T15:57:51.666 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:50.891+0000 7fd439e00700 1 -- 172.21.15.39:0/1434152056 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd43419e520 con 0x7fd4341064f0 2024-03-18T15:57:51.666 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:50.891+0000 7fd438a00700 1 -- 172.21.15.39:0/1434152056 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd424024840 con 0x7fd4341064f0 2024-03-18T15:57:51.666 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:50.892+0000 7fd438a00700 1 -- 172.21.15.39:0/1434152056 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fd424022020 con 0x7fd4341064f0 2024-03-18T15:57:51.666 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:50.892+0000 7fd439e00700 1 -- 172.21.15.39:0/1434152056 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4341a1210 con 0x7fd4341064f0 2024-03-18T15:57:51.666 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:50.892+0000 7fd438a00700 1 -- 172.21.15.39:0/1434152056 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fd424011e00 con 0x7fd4341064f0 2024-03-18T15:57:51.666 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:50.893+0000 7fd439e00700 1 -- 172.21.15.39:0/1434152056 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd434063900 con 0x7fd4341064f0 2024-03-18T15:57:51.666 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:50.896+0000 7fd438a00700 1 -- 172.21.15.39:0/1434152056 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fd424032080 con 0x7fd4341064f0 2024-03-18T15:57:51.667 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:50.974+0000 7fd439e00700 1 -- 172.21.15.39:0/1434152056 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fd43402de60 con 0x7fd4341064f0 2024-03-18T15:57:51.667 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:50.975+0000 7fd438a00700 1 -- 172.21.15.39:0/1434152056 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fd42401a430 con 0x7fd4341064f0 2024-03-18T15:57:51.667 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:50.976+0000 7fd439e00700 1 -- 172.21.15.39:0/1434152056 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd4341064f0 msgr2=0x7fd43419dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:51.667 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:50.976+0000 7fd439e00700 1 --2- 172.21.15.39:0/1434152056 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd4341064f0 0x7fd43419dd50 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd424011b60 tx=0x7fd424011c40 comp rx=0 tx=0).stop 2024-03-18T15:57:51.667 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:50.976+0000 7fd439e00700 1 -- 172.21.15.39:0/1434152056 shutdown_connections 2024-03-18T15:57:51.667 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:50.976+0000 7fd439e00700 1 --2- 172.21.15.39:0/1434152056 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd4341064f0 0x7fd43419dd50 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:51.667 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:50.976+0000 7fd439e00700 1 -- 172.21.15.39:0/1434152056 >> 172.21.15.39:0/1434152056 conn(0x7fd4341020d0 msgr2=0x7fd4341949b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:57:51.667 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:50.976+0000 7fd439e00700 1 -- 172.21.15.39:0/1434152056 shutdown_connections 2024-03-18T15:57:51.667 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:50.976+0000 7fd439e00700 1 -- 172.21.15.39:0/1434152056 wait complete. 2024-03-18T15:57:51.667 INFO:teuthology.orchestra.run.smithi039.stdout:mgr not available, waiting (2/15)... 2024-03-18T15:57:51.961 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:57:51 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T15:57:51.838+0000 7ff476a9b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T15:57:51.961 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:57:51 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T15:57:51.958+0000 7ff476a9b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T15:57:52.477 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:57:52 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T15:57:52.223+0000 7ff476a9b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T15:57:52.477 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:57:52 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T15:57:52.474+0000 7ff476a9b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T15:57:53.205 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:57:52 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T15:57:52.857+0000 7ff476a9b200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T15:57:53.205 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:57:53 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T15:57:53.203+0000 7ff476a9b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T15:57:53.799 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:57:53 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T15:57:53.516+0000 7ff476a9b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T15:57:54.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:54 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/915093327' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-18T15:57:55.079 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:57:54 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T15:57:54.619+0000 7ff476a9b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T15:57:55.088 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout 2024-03-18T15:57:55.088 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout { 2024-03-18T15:57:55.088 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "fsid": "1892d8fc-e540-11ee-95c9-87774f69a715", 2024-03-18T15:57:55.089 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "health": { 2024-03-18T15:57:55.089 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-18T15:57:55.089 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-18T15:57:55.089 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-18T15:57:55.089 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2024-03-18T15:57:55.089 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-18T15:57:55.089 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-18T15:57:55.089 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout 0 2024-03-18T15:57:55.089 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout ], 2024-03-18T15:57:55.089 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-18T15:57:55.089 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "a" 2024-03-18T15:57:55.089 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout ], 2024-03-18T15:57:55.089 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "quorum_age": 10, 2024-03-18T15:57:55.089 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-18T15:57:55.089 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T15:57:55.090 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-18T15:57:55.090 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-18T15:57:55.090 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2024-03-18T15:57:55.090 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-18T15:57:55.090 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T15:57:55.090 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-18T15:57:55.090 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-18T15:57:55.098 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-18T15:57:55.098 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-18T15:57:55.098 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-18T15:57:55.098 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-18T15:57:55.098 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2024-03-18T15:57:55.098 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-18T15:57:55.099 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-18T15:57:55.099 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-18T15:57:55.099 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-18T15:57:55.099 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-18T15:57:55.099 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-18T15:57:55.099 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-18T15:57:55.099 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-18T15:57:55.099 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-18T15:57:55.099 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2024-03-18T15:57:55.099 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-18T15:57:55.099 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T15:57:55.099 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-18T15:57:55.099 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-18T15:57:55.099 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2024-03-18T15:57:55.099 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-18T15:57:55.100 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-18T15:57:55.100 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-18T15:57:55.100 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-18T15:57:55.100 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-18T15:57:55.100 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-18T15:57:55.100 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "restful" 2024-03-18T15:57:55.100 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout ], 2024-03-18T15:57:55.100 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-18T15:57:55.100 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2024-03-18T15:57:55.100 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-18T15:57:55.100 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T15:57:55.100 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "modified": "2024-03-18T15:57:36.508025+0000", 2024-03-18T15:57:55.100 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-18T15:57:55.100 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2024-03-18T15:57:55.101 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-18T15:57:55.101 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout } 2024-03-18T15:57:55.101 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:54.200+0000 7fcf0da00700 1 Processor -- start 2024-03-18T15:57:55.101 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:54.200+0000 7fcf0da00700 1 -- start start 2024-03-18T15:57:55.101 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:54.200+0000 7fcf0da00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcf08107c30 0x7fcf08108050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:55.101 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:54.201+0000 7fcf0da00700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fcf08108620 con 0x7fcf08107c30 2024-03-18T15:57:55.101 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:54.201+0000 7fcf06a00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcf08107c30 0x7fcf08108050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:55.101 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:54.201+0000 7fcf06a00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcf08107c30 0x7fcf08108050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58950/0 (socket says 172.21.15.39:58950) 2024-03-18T15:57:55.101 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:54.201+0000 7fcf06a00700 1 -- 172.21.15.39:0/1835263201 learned_addr learned my addr 172.21.15.39:0/1835263201 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T15:57:55.101 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:54.202+0000 7fcf06a00700 1 -- 172.21.15.39:0/1835263201 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf08108e30 con 0x7fcf08107c30 2024-03-18T15:57:55.101 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:54.202+0000 7fcf06a00700 1 --2- 172.21.15.39:0/1835263201 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcf08107c30 0x7fcf08108050 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fcef80098a0 tx=0x7fcef8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77a9e5670781c7c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:55.101 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:54.202+0000 7fcf05600700 1 -- 172.21.15.39:0/1835263201 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcef8004160 con 0x7fcf08107c30 2024-03-18T15:57:55.101 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:54.202+0000 7fcf05600700 1 -- 172.21.15.39:0/1835263201 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fcef8004300 con 0x7fcf08107c30 2024-03-18T15:57:55.101 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:54.202+0000 7fcf05600700 1 -- 172.21.15.39:0/1835263201 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcef8011b50 con 0x7fcf08107c30 2024-03-18T15:57:55.102 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:54.203+0000 7fcf0da00700 1 -- 172.21.15.39:0/1835263201 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcf08107c30 msgr2=0x7fcf08108050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:55.102 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:54.203+0000 7fcf0da00700 1 --2- 172.21.15.39:0/1835263201 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcf08107c30 0x7fcf08108050 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fcef80098a0 tx=0x7fcef8009bb0 comp rx=0 tx=0).stop 2024-03-18T15:57:55.102 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:54.203+0000 7fcf0da00700 1 -- 172.21.15.39:0/1835263201 shutdown_connections 2024-03-18T15:57:55.102 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:54.203+0000 7fcf0da00700 1 --2- 172.21.15.39:0/1835263201 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcf08107c30 0x7fcf08108050 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:55.102 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:54.203+0000 7fcf0da00700 1 -- 172.21.15.39:0/1835263201 >> 172.21.15.39:0/1835263201 conn(0x7fcf08103210 msgr2=0x7fcf08105630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:57:55.102 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:54.204+0000 7fcf0da00700 1 -- 172.21.15.39:0/1835263201 shutdown_connections 2024-03-18T15:57:55.102 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:54.204+0000 7fcf0da00700 1 -- 172.21.15.39:0/1835263201 wait complete. 2024-03-18T15:57:55.103 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:54.205+0000 7fcf0da00700 1 Processor -- start 2024-03-18T15:57:55.103 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:54.205+0000 7fcf0da00700 1 -- start start 2024-03-18T15:57:55.103 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:54.205+0000 7fcf0da00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcf08107c30 0x7fcf0819de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:55.103 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:54.205+0000 7fcf0da00700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fcf0810c1e0 con 0x7fcf08107c30 2024-03-18T15:57:55.103 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:54.205+0000 7fcf06a00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcf08107c30 0x7fcf0819de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:55.103 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:54.205+0000 7fcf06a00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcf08107c30 0x7fcf0819de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58952/0 (socket says 172.21.15.39:58952) 2024-03-18T15:57:55.103 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:54.205+0000 7fcf06a00700 1 -- 172.21.15.39:0/915093327 learned_addr learned my addr 172.21.15.39:0/915093327 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T15:57:55.103 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:54.206+0000 7fcf06a00700 1 -- 172.21.15.39:0/915093327 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf0819e340 con 0x7fcf08107c30 2024-03-18T15:57:55.103 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:54.206+0000 7fcf06a00700 1 --2- 172.21.15.39:0/915093327 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcf08107c30 0x7fcf0819de00 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fcef8011850 tx=0x7fcef8011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:55.103 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:54.206+0000 7fceff400700 1 -- 172.21.15.39:0/915093327 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcef8003af0 con 0x7fcf08107c30 2024-03-18T15:57:55.103 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:54.206+0000 7fceff400700 1 -- 172.21.15.39:0/915093327 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fcef8020070 con 0x7fcf08107c30 2024-03-18T15:57:55.103 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:54.206+0000 7fcf0da00700 1 -- 172.21.15.39:0/915093327 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf0819e5d0 con 0x7fcf08107c30 2024-03-18T15:57:55.104 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:54.206+0000 7fceff400700 1 -- 172.21.15.39:0/915093327 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcef80195a0 con 0x7fcf08107c30 2024-03-18T15:57:55.104 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:54.207+0000 7fcf0da00700 1 -- 172.21.15.39:0/915093327 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf081a12c0 con 0x7fcf08107c30 2024-03-18T15:57:55.104 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:54.207+0000 7fceff400700 1 -- 172.21.15.39:0/915093327 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fcef8019be0 con 0x7fcf08107c30 2024-03-18T15:57:55.104 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:54.207+0000 7fceff400700 1 -- 172.21.15.39:0/915093327 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fcef8011e00 con 0x7fcf08107c30 2024-03-18T15:57:55.104 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:54.208+0000 7fcf0da00700 1 -- 172.21.15.39:0/915093327 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcf08063900 con 0x7fcf08107c30 2024-03-18T15:57:55.104 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:54.210+0000 7fceff400700 1 -- 172.21.15.39:0/915093327 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fcef8003c90 con 0x7fcf08107c30 2024-03-18T15:57:55.104 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:54.290+0000 7fcf0da00700 1 -- 172.21.15.39:0/915093327 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fcf081a1490 con 0x7fcf08107c30 2024-03-18T15:57:55.104 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:54.291+0000 7fceff400700 1 -- 172.21.15.39:0/915093327 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1240 (secure 0 0 0) 0x7fcef8025460 con 0x7fcf08107c30 2024-03-18T15:57:55.104 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:54.292+0000 7fcf0da00700 1 -- 172.21.15.39:0/915093327 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcf08107c30 msgr2=0x7fcf0819de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:55.104 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:54.292+0000 7fcf0da00700 1 --2- 172.21.15.39:0/915093327 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcf08107c30 0x7fcf0819de00 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fcef8011850 tx=0x7fcef8011c10 comp rx=0 tx=0).stop 2024-03-18T15:57:55.104 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:54.292+0000 7fcf0da00700 1 -- 172.21.15.39:0/915093327 shutdown_connections 2024-03-18T15:57:55.104 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:54.292+0000 7fcf0da00700 1 --2- 172.21.15.39:0/915093327 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcf08107c30 0x7fcf0819de00 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:55.104 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:54.292+0000 7fcf0da00700 1 -- 172.21.15.39:0/915093327 >> 172.21.15.39:0/915093327 conn(0x7fcf08103210 msgr2=0x7fcf08103df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:57:55.104 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:54.293+0000 7fcf0da00700 1 -- 172.21.15.39:0/915093327 shutdown_connections 2024-03-18T15:57:55.105 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:54.293+0000 7fcf0da00700 1 -- 172.21.15.39:0/915093327 wait complete. 2024-03-18T15:57:55.105 INFO:teuthology.orchestra.run.smithi039.stdout:mgr not available, waiting (3/15)... 2024-03-18T15:57:55.778 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:57:55 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T15:57:55.355+0000 7ff476a9b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T15:57:55.779 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:57:55 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T15:57:55.488+0000 7ff476a9b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T15:57:56.086 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:57:55 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T15:57:55.776+0000 7ff476a9b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T15:57:56.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:55 smithi039 ceph-mon[31721]: Activating manager daemon x 2024-03-18T15:57:56.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:55 smithi039 ceph-mon[31721]: mgrmap e2: x(active, starting, since 0.012395s) 2024-03-18T15:57:56.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:55 smithi039 ceph-mon[31721]: from='mgr.14100 172.21.15.39:0/3098390752' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-18T15:57:56.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:55 smithi039 ceph-mon[31721]: from='mgr.14100 172.21.15.39:0/3098390752' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-18T15:57:56.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:55 smithi039 ceph-mon[31721]: from='mgr.14100 172.21.15.39:0/3098390752' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-18T15:57:56.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:55 smithi039 ceph-mon[31721]: from='mgr.14100 172.21.15.39:0/3098390752' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T15:57:56.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:55 smithi039 ceph-mon[31721]: from='mgr.14100 172.21.15.39:0/3098390752' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-18T15:57:56.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:55 smithi039 ceph-mon[31721]: Manager daemon x is now available 2024-03-18T15:57:56.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:55 smithi039 ceph-mon[31721]: from='mgr.14100 172.21.15.39:0/3098390752' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-18T15:57:56.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:55 smithi039 ceph-mon[31721]: from='mgr.14100 172.21.15.39:0/3098390752' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-18T15:57:56.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:55 smithi039 ceph-mon[31721]: from='mgr.14100 172.21.15.39:0/3098390752' entity='mgr.x' 2024-03-18T15:57:56.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:55 smithi039 ceph-mon[31721]: from='mgr.14100 172.21.15.39:0/3098390752' entity='mgr.x' 2024-03-18T15:57:57.250 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:56 smithi039 ceph-mon[31721]: from='mgr.14100 172.21.15.39:0/3098390752' entity='mgr.x' 2024-03-18T15:57:57.250 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:56 smithi039 ceph-mon[31721]: mgrmap e3: x(active, since 1.02022s) 2024-03-18T15:57:58.649 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout 2024-03-18T15:57:58.649 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout { 2024-03-18T15:57:58.649 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "fsid": "1892d8fc-e540-11ee-95c9-87774f69a715", 2024-03-18T15:57:58.649 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "health": { 2024-03-18T15:57:58.649 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-18T15:57:58.649 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-18T15:57:58.649 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-18T15:57:58.649 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2024-03-18T15:57:58.649 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-18T15:57:58.650 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-18T15:57:58.650 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout 0 2024-03-18T15:57:58.650 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout ], 2024-03-18T15:57:58.650 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-18T15:57:58.650 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "a" 2024-03-18T15:57:58.650 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout ], 2024-03-18T15:57:58.650 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "quorum_age": 14, 2024-03-18T15:57:58.650 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-18T15:57:58.650 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T15:57:58.650 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-18T15:57:58.650 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-18T15:57:58.650 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2024-03-18T15:57:58.650 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-18T15:57:58.650 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T15:57:58.651 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-18T15:57:58.651 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-18T15:57:58.651 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-18T15:57:58.651 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-18T15:57:58.651 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-18T15:57:58.651 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-18T15:57:58.651 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2024-03-18T15:57:58.651 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-18T15:57:58.651 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-18T15:57:58.651 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-18T15:57:58.651 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-18T15:57:58.651 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-18T15:57:58.651 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-18T15:57:58.651 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-18T15:57:58.651 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-18T15:57:58.652 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-18T15:57:58.652 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2024-03-18T15:57:58.652 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-18T15:57:58.652 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T15:57:58.652 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-18T15:57:58.652 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-18T15:57:58.652 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2024-03-18T15:57:58.652 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-18T15:57:58.652 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-18T15:57:58.652 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-18T15:57:58.652 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-18T15:57:58.652 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-18T15:57:58.652 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-18T15:57:58.652 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "restful" 2024-03-18T15:57:58.652 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout ], 2024-03-18T15:57:58.653 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-18T15:57:58.653 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2024-03-18T15:57:58.653 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-18T15:57:58.653 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-18T15:57:58.653 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "modified": "2024-03-18T15:57:36.508025+0000", 2024-03-18T15:57:58.653 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-18T15:57:58.653 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout }, 2024-03-18T15:57:58.653 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-18T15:57:58.653 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout } 2024-03-18T15:57:58.653 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:57.724+0000 7f3102400700 1 Processor -- start 2024-03-18T15:57:58.653 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:57.724+0000 7f3102400700 1 -- start start 2024-03-18T15:57:58.653 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:57.725+0000 7f3102400700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f30fc10ec60 0x7f30fc111050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:58.653 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:57.725+0000 7f3102400700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f30fc082470 con 0x7f30fc10ec60 2024-03-18T15:57:58.653 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:57.725+0000 7f30fb400700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f30fc10ec60 0x7f30fc111050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:58.654 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:57.725+0000 7f30fb400700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f30fc10ec60 0x7f30fc111050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:59020/0 (socket says 172.21.15.39:59020) 2024-03-18T15:57:58.654 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:57.725+0000 7f30fb400700 1 -- 172.21.15.39:0/1840811821 learned_addr learned my addr 172.21.15.39:0/1840811821 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T15:57:58.654 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:57.726+0000 7f30fb400700 1 -- 172.21.15.39:0/1840811821 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30fc111590 con 0x7f30fc10ec60 2024-03-18T15:57:58.654 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:57.726+0000 7f30fb400700 1 --2- 172.21.15.39:0/1840811821 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f30fc10ec60 0x7f30fc111050 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f30ec0098a0 tx=0x7f30ec009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9648fae0c9b643f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:58.654 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:57.727+0000 7f30fa000700 1 -- 172.21.15.39:0/1840811821 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f30ec004160 con 0x7f30fc10ec60 2024-03-18T15:57:58.654 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:57.727+0000 7f30fa000700 1 -- 172.21.15.39:0/1840811821 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f30ec004300 con 0x7f30fc10ec60 2024-03-18T15:57:58.654 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:57.727+0000 7f30fa000700 1 -- 172.21.15.39:0/1840811821 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f30ec011b50 con 0x7f30fc10ec60 2024-03-18T15:57:58.654 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:57.728+0000 7f3102400700 1 -- 172.21.15.39:0/1840811821 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f30fc10ec60 msgr2=0x7f30fc111050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:58.654 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:57.728+0000 7f3102400700 1 --2- 172.21.15.39:0/1840811821 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f30fc10ec60 0x7f30fc111050 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f30ec0098a0 tx=0x7f30ec009bb0 comp rx=0 tx=0).stop 2024-03-18T15:57:58.654 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:57.728+0000 7f3102400700 1 -- 172.21.15.39:0/1840811821 shutdown_connections 2024-03-18T15:57:58.654 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:57.728+0000 7f3102400700 1 --2- 172.21.15.39:0/1840811821 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f30fc10ec60 0x7f30fc111050 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:58.654 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:57.728+0000 7f3102400700 1 -- 172.21.15.39:0/1840811821 >> 172.21.15.39:0/1840811821 conn(0x7f30fc07d210 msgr2=0x7f30fc07f630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:57:58.654 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:57.728+0000 7f3102400700 1 -- 172.21.15.39:0/1840811821 shutdown_connections 2024-03-18T15:57:58.654 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:57.728+0000 7f3102400700 1 -- 172.21.15.39:0/1840811821 wait complete. 2024-03-18T15:57:58.655 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:57.729+0000 7f3102400700 1 Processor -- start 2024-03-18T15:57:58.655 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:57.729+0000 7f3102400700 1 -- start start 2024-03-18T15:57:58.655 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:57.729+0000 7f3102400700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f30fc10ec60 0x7f30fc1a2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:58.655 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:57.729+0000 7f3102400700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f30fc1125a0 con 0x7f30fc10ec60 2024-03-18T15:57:58.655 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:57.730+0000 7f30fb400700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f30fc10ec60 0x7f30fc1a2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:58.655 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:57.730+0000 7f30fb400700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f30fc10ec60 0x7f30fc1a2160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:59032/0 (socket says 172.21.15.39:59032) 2024-03-18T15:57:58.655 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:57.730+0000 7f30fb400700 1 -- 172.21.15.39:0/4259339919 learned_addr learned my addr 172.21.15.39:0/4259339919 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T15:57:58.655 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:57.730+0000 7f30fb400700 1 -- 172.21.15.39:0/4259339919 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30fc1a26a0 con 0x7f30fc10ec60 2024-03-18T15:57:58.655 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:57.730+0000 7f30fb400700 1 --2- 172.21.15.39:0/4259339919 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f30fc10ec60 0x7f30fc1a2160 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f30ec011b60 tx=0x7f30ec011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:58.655 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:57.731+0000 7f30f3e00700 1 -- 172.21.15.39:0/4259339919 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f30ec003970 con 0x7f30fc10ec60 2024-03-18T15:57:58.655 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:57.731+0000 7f30f3e00700 1 -- 172.21.15.39:0/4259339919 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f30ec003b10 con 0x7f30fc10ec60 2024-03-18T15:57:58.655 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:57.731+0000 7f30f3e00700 1 -- 172.21.15.39:0/4259339919 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f30ec024610 con 0x7f30fc10ec60 2024-03-18T15:57:58.655 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:57.731+0000 7f3102400700 1 -- 172.21.15.39:0/4259339919 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30fc1a2930 con 0x7f30fc10ec60 2024-03-18T15:57:58.655 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:57.731+0000 7f3102400700 1 -- 172.21.15.39:0/4259339919 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30fc1a5620 con 0x7f30fc10ec60 2024-03-18T15:57:58.656 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:57.732+0000 7f3102400700 1 -- 172.21.15.39:0/4259339919 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30fc063900 con 0x7f30fc10ec60 2024-03-18T15:57:58.656 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:57.732+0000 7f30f3e00700 1 -- 172.21.15.39:0/4259339919 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 3) ==== 45989+0+0 (secure 0 0 0) 0x7f30ec01a9f0 con 0x7f30fc10ec60 2024-03-18T15:57:58.656 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:57.733+0000 7f30f3e00700 1 --2- 172.21.15.39:0/4259339919 >> [v2:172.21.15.39:6800/1217503189,v1:172.21.15.39:6801/1217503189] conn(0x7f30e0038e00 0x7f30e003b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:57:58.656 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:57.733+0000 7f30f3e00700 1 -- 172.21.15.39:0/4259339919 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f30ec011e00 con 0x7f30fc10ec60 2024-03-18T15:57:58.656 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:57.733+0000 7f30faa00700 1 --2- 172.21.15.39:0/4259339919 >> [v2:172.21.15.39:6800/1217503189,v1:172.21.15.39:6801/1217503189] conn(0x7f30e0038e00 0x7f30e003b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:57:58.656 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:57.734+0000 7f30faa00700 1 --2- 172.21.15.39:0/4259339919 >> [v2:172.21.15.39:6800/1217503189,v1:172.21.15.39:6801/1217503189] conn(0x7f30e0038e00 0x7f30e003b2c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f30e8006fd0 tx=0x7f30e8006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:57:58.656 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:57.739+0000 7f30f3e00700 1 -- 172.21.15.39:0/4259339919 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f30ec00b3e0 con 0x7f30fc10ec60 2024-03-18T15:57:58.656 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:57.844+0000 7f30f3e00700 1 -- 172.21.15.39:0/4259339919 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mgrmap(e 4) ==== 46095+0+0 (secure 0 0 0) 0x7f30ec02c350 con 0x7f30fc10ec60 2024-03-18T15:57:58.656 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:58.051+0000 7f3102400700 1 -- 172.21.15.39:0/4259339919 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f30fc1a5890 con 0x7f30fc10ec60 2024-03-18T15:57:58.656 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:58.052+0000 7f30f3e00700 1 -- 172.21.15.39:0/4259339919 <== mon.0 v2:172.21.15.39:3300/0 8 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f30ec02b2f0 con 0x7f30fc10ec60 2024-03-18T15:57:58.656 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:58.055+0000 7f3102400700 1 -- 172.21.15.39:0/4259339919 >> [v2:172.21.15.39:6800/1217503189,v1:172.21.15.39:6801/1217503189] conn(0x7f30e0038e00 msgr2=0x7f30e003b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:58.656 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:58.055+0000 7f3102400700 1 --2- 172.21.15.39:0/4259339919 >> [v2:172.21.15.39:6800/1217503189,v1:172.21.15.39:6801/1217503189] conn(0x7f30e0038e00 0x7f30e003b2c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f30e8006fd0 tx=0x7f30e8006e40 comp rx=0 tx=0).stop 2024-03-18T15:57:58.656 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:58.055+0000 7f3102400700 1 -- 172.21.15.39:0/4259339919 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f30fc10ec60 msgr2=0x7f30fc1a2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:57:58.656 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:58.055+0000 7f3102400700 1 --2- 172.21.15.39:0/4259339919 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f30fc10ec60 0x7f30fc1a2160 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f30ec011b60 tx=0x7f30ec011c40 comp rx=0 tx=0).stop 2024-03-18T15:57:58.657 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:58.055+0000 7f3102400700 1 -- 172.21.15.39:0/4259339919 shutdown_connections 2024-03-18T15:57:58.657 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:58.055+0000 7f3102400700 1 --2- 172.21.15.39:0/4259339919 >> [v2:172.21.15.39:6800/1217503189,v1:172.21.15.39:6801/1217503189] conn(0x7f30e0038e00 0x7f30e003b2c0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:58.657 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:58.055+0000 7f3102400700 1 --2- 172.21.15.39:0/4259339919 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f30fc10ec60 0x7f30fc1a2160 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:57:58.657 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:58.055+0000 7f3102400700 1 -- 172.21.15.39:0/4259339919 >> 172.21.15.39:0/4259339919 conn(0x7f30fc07d210 msgr2=0x7f30fc07dd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:57:58.657 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:58.055+0000 7f3102400700 1 -- 172.21.15.39:0/4259339919 shutdown_connections 2024-03-18T15:57:58.657 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:58.055+0000 7f3102400700 1 -- 172.21.15.39:0/4259339919 wait complete. 2024-03-18T15:57:58.657 INFO:teuthology.orchestra.run.smithi039.stdout:mgr is available 2024-03-18T15:57:58.973 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:58 smithi039 ceph-mon[31721]: mgrmap e4: x(active, since 2s) 2024-03-18T15:57:58.973 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:58 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/4259339919' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-18T15:58:00.142 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:57:59 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/470651851' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-18T15:58:00.160 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout 2024-03-18T15:58:00.160 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout [global] 2024-03-18T15:58:00.160 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout fsid = 1892d8fc-e540-11ee-95c9-87774f69a715 2024-03-18T15:58:00.160 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-18T15:58:00.160 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-18T15:58:00.160 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-18T15:58:00.160 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-18T15:58:00.160 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout 2024-03-18T15:58:00.161 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-18T15:58:00.161 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-18T15:58:00.161 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout 2024-03-18T15:58:00.161 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout [osd] 2024-03-18T15:58:00.161 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout filestore_flush_min = 0 2024-03-18T15:58:00.161 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-18T15:58:00.161 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-18T15:58:00.161 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-18T15:58:00.161 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-18T15:58:00.161 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-18T15:58:00.161 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:59.264+0000 7f2a61800700 1 Processor -- start 2024-03-18T15:58:00.161 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:59.264+0000 7f2a61800700 1 -- start start 2024-03-18T15:58:00.161 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:59.265+0000 7f2a61800700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2a5c109e00 0x7f2a5c10a220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:00.161 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:59.265+0000 7f2a61800700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f2a5c10a760 con 0x7f2a5c109e00 2024-03-18T15:58:00.162 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:59.265+0000 7f2a5aa00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2a5c109e00 0x7f2a5c10a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:00.162 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:59.265+0000 7f2a5aa00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2a5c109e00 0x7f2a5c10a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:46712/0 (socket says 172.21.15.39:46712) 2024-03-18T15:58:00.162 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:59.265+0000 7f2a5aa00700 1 -- 172.21.15.39:0/838950611 learned_addr learned my addr 172.21.15.39:0/838950611 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T15:58:00.162 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:59.266+0000 7f2a5aa00700 1 -- 172.21.15.39:0/838950611 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a5c10af60 con 0x7f2a5c109e00 2024-03-18T15:58:00.162 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:59.266+0000 7f2a5aa00700 1 --2- 172.21.15.39:0/838950611 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2a5c109e00 0x7f2a5c10a220 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2a4c0098a0 tx=0x7f2a4c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56ca9652c17d4c9e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:00.162 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:59.266+0000 7f2a5a000700 1 -- 172.21.15.39:0/838950611 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2a4c004160 con 0x7f2a5c109e00 2024-03-18T15:58:00.162 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:59.267+0000 7f2a5a000700 1 -- 172.21.15.39:0/838950611 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f2a4c004300 con 0x7f2a5c109e00 2024-03-18T15:58:00.162 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:59.267+0000 7f2a5a000700 1 -- 172.21.15.39:0/838950611 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2a4c011b50 con 0x7f2a5c109e00 2024-03-18T15:58:00.162 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:59.267+0000 7f2a61800700 1 -- 172.21.15.39:0/838950611 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2a5c109e00 msgr2=0x7f2a5c10a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:00.162 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:59.267+0000 7f2a61800700 1 --2- 172.21.15.39:0/838950611 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2a5c109e00 0x7f2a5c10a220 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2a4c0098a0 tx=0x7f2a4c009bb0 comp rx=0 tx=0).stop 2024-03-18T15:58:00.162 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:59.268+0000 7f2a61800700 1 -- 172.21.15.39:0/838950611 shutdown_connections 2024-03-18T15:58:00.162 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:59.268+0000 7f2a61800700 1 --2- 172.21.15.39:0/838950611 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2a5c109e00 0x7f2a5c10a220 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:00.162 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:59.268+0000 7f2a61800700 1 -- 172.21.15.39:0/838950611 >> 172.21.15.39:0/838950611 conn(0x7f2a5c07d210 msgr2=0x7f2a5c07d620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:00.163 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:59.268+0000 7f2a61800700 1 -- 172.21.15.39:0/838950611 shutdown_connections 2024-03-18T15:58:00.163 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:59.268+0000 7f2a61800700 1 -- 172.21.15.39:0/838950611 wait complete. 2024-03-18T15:58:00.163 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:59.269+0000 7f2a61800700 1 Processor -- start 2024-03-18T15:58:00.163 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:59.269+0000 7f2a61800700 1 -- start start 2024-03-18T15:58:00.163 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:59.269+0000 7f2a61800700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2a5c109e00 0x7f2a5c19dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:00.163 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:59.269+0000 7f2a61800700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f2a5c10e220 con 0x7f2a5c109e00 2024-03-18T15:58:00.163 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:59.270+0000 7f2a5aa00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2a5c109e00 0x7f2a5c19dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:00.163 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:59.270+0000 7f2a5aa00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2a5c109e00 0x7f2a5c19dda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:46724/0 (socket says 172.21.15.39:46724) 2024-03-18T15:58:00.163 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:59.270+0000 7f2a5aa00700 1 -- 172.21.15.39:0/470651851 learned_addr learned my addr 172.21.15.39:0/470651851 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T15:58:00.163 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:59.270+0000 7f2a5aa00700 1 -- 172.21.15.39:0/470651851 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a5c19e2e0 con 0x7f2a5c109e00 2024-03-18T15:58:00.163 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:59.270+0000 7f2a5aa00700 1 --2- 172.21.15.39:0/470651851 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2a5c109e00 0x7f2a5c19dda0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f2a4c011b60 tx=0x7f2a4c011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:00.163 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:59.271+0000 7f2a53400700 1 -- 172.21.15.39:0/470651851 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2a4c003970 con 0x7f2a5c109e00 2024-03-18T15:58:00.163 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:59.271+0000 7f2a53400700 1 -- 172.21.15.39:0/470651851 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f2a4c003b10 con 0x7f2a5c109e00 2024-03-18T15:58:00.163 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:59.271+0000 7f2a53400700 1 -- 172.21.15.39:0/470651851 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2a4c024610 con 0x7f2a5c109e00 2024-03-18T15:58:00.163 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:59.271+0000 7f2a61800700 1 -- 172.21.15.39:0/470651851 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a5c19e570 con 0x7f2a5c109e00 2024-03-18T15:58:00.164 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:59.271+0000 7f2a61800700 1 -- 172.21.15.39:0/470651851 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a5c1a1260 con 0x7f2a5c109e00 2024-03-18T15:58:00.164 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:59.272+0000 7f2a53400700 1 -- 172.21.15.39:0/470651851 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 4) ==== 46095+0+0 (secure 0 0 0) 0x7f2a4c01a9f0 con 0x7f2a5c109e00 2024-03-18T15:58:00.164 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:59.272+0000 7f2a61800700 1 -- 172.21.15.39:0/470651851 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a5c10e3a0 con 0x7f2a5c109e00 2024-03-18T15:58:00.164 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:59.272+0000 7f2a53400700 1 --2- 172.21.15.39:0/470651851 >> [v2:172.21.15.39:6800/1217503189,v1:172.21.15.39:6801/1217503189] conn(0x7f2a40038f20 0x7f2a4003b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:00.164 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:59.272+0000 7f2a53400700 1 -- 172.21.15.39:0/470651851 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f2a4c011e00 con 0x7f2a5c109e00 2024-03-18T15:58:00.164 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:59.273+0000 7f2a53e00700 1 --2- 172.21.15.39:0/470651851 >> [v2:172.21.15.39:6800/1217503189,v1:172.21.15.39:6801/1217503189] conn(0x7f2a40038f20 0x7f2a4003b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:00.164 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:59.273+0000 7f2a53e00700 1 --2- 172.21.15.39:0/470651851 >> [v2:172.21.15.39:6800/1217503189,v1:172.21.15.39:6801/1217503189] conn(0x7f2a40038f20 0x7f2a4003b3e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2a44006fd0 tx=0x7f2a44006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:00.164 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:59.279+0000 7f2a53400700 1 -- 172.21.15.39:0/470651851 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2a4c019e10 con 0x7f2a5c109e00 2024-03-18T15:58:00.164 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:59.505+0000 7f2a61800700 1 -- 172.21.15.39:0/470651851 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f2a5c063900 con 0x7f2a5c109e00 2024-03-18T15:58:00.164 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:59.506+0000 7f2a53400700 1 -- 172.21.15.39:0/470651851 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+398 (secure 0 0 0) 0x7f2a4c04c9f0 con 0x7f2a5c109e00 2024-03-18T15:58:00.164 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:59.510+0000 7f2a61800700 1 -- 172.21.15.39:0/470651851 >> [v2:172.21.15.39:6800/1217503189,v1:172.21.15.39:6801/1217503189] conn(0x7f2a40038f20 msgr2=0x7f2a4003b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:00.164 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:59.510+0000 7f2a61800700 1 --2- 172.21.15.39:0/470651851 >> [v2:172.21.15.39:6800/1217503189,v1:172.21.15.39:6801/1217503189] conn(0x7f2a40038f20 0x7f2a4003b3e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2a44006fd0 tx=0x7f2a44006e40 comp rx=0 tx=0).stop 2024-03-18T15:58:00.164 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:59.510+0000 7f2a61800700 1 -- 172.21.15.39:0/470651851 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2a5c109e00 msgr2=0x7f2a5c19dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:00.164 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:59.510+0000 7f2a61800700 1 --2- 172.21.15.39:0/470651851 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2a5c109e00 0x7f2a5c19dda0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f2a4c011b60 tx=0x7f2a4c011c40 comp rx=0 tx=0).stop 2024-03-18T15:58:00.165 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:59.510+0000 7f2a61800700 1 -- 172.21.15.39:0/470651851 shutdown_connections 2024-03-18T15:58:00.165 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:59.510+0000 7f2a61800700 1 --2- 172.21.15.39:0/470651851 >> [v2:172.21.15.39:6800/1217503189,v1:172.21.15.39:6801/1217503189] conn(0x7f2a40038f20 0x7f2a4003b3e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:00.165 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:59.510+0000 7f2a61800700 1 --2- 172.21.15.39:0/470651851 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2a5c109e00 0x7f2a5c19dda0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:00.165 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:59.510+0000 7f2a61800700 1 -- 172.21.15.39:0/470651851 >> 172.21.15.39:0/470651851 conn(0x7f2a5c07d210 msgr2=0x7f2a5c1074d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:00.165 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:59.510+0000 7f2a61800700 1 -- 172.21.15.39:0/470651851 shutdown_connections 2024-03-18T15:58:00.165 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:57:59.510+0000 7f2a61800700 1 -- 172.21.15.39:0/470651851 wait complete. 2024-03-18T15:58:00.165 INFO:teuthology.orchestra.run.smithi039.stdout:Enabling cephadm module... 2024-03-18T15:58:01.461 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:01 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/1074828101' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-18T15:58:01.462 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:58:01 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: ignoring --setuser ceph since I am not root 2024-03-18T15:58:01.462 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:58:01 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: ignoring --setgroup ceph since I am not root 2024-03-18T15:58:01.801 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:00.789+0000 7f135d000700 1 Processor -- start 2024-03-18T15:58:01.801 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:00.789+0000 7f135d000700 1 -- start start 2024-03-18T15:58:01.801 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:00.790+0000 7f135d000700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f135810ec60 0x7f1358111050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:01.801 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:00.790+0000 7f135d000700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f1358082470 con 0x7f135810ec60 2024-03-18T15:58:01.802 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:00.790+0000 7f1355e00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f135810ec60 0x7f1358111050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:01.802 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:00.791+0000 7f1355e00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f135810ec60 0x7f1358111050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:46740/0 (socket says 172.21.15.39:46740) 2024-03-18T15:58:01.803 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:00.791+0000 7f1355e00700 1 -- 172.21.15.39:0/987785995 learned_addr learned my addr 172.21.15.39:0/987785995 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T15:58:01.803 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:00.791+0000 7f1355e00700 1 -- 172.21.15.39:0/987785995 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1358111590 con 0x7f135810ec60 2024-03-18T15:58:01.803 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:00.792+0000 7f1355e00700 1 --2- 172.21.15.39:0/987785995 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f135810ec60 0x7f1358111050 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f13480098a0 tx=0x7f1348009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8b75d1d19aa823a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:01.803 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:00.792+0000 7f1354a00700 1 -- 172.21.15.39:0/987785995 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1348004160 con 0x7f135810ec60 2024-03-18T15:58:01.805 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:00.792+0000 7f1354a00700 1 -- 172.21.15.39:0/987785995 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f1348004300 con 0x7f135810ec60 2024-03-18T15:58:01.805 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:00.792+0000 7f1354a00700 1 -- 172.21.15.39:0/987785995 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1348011b50 con 0x7f135810ec60 2024-03-18T15:58:01.805 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:00.793+0000 7f135d000700 1 -- 172.21.15.39:0/987785995 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f135810ec60 msgr2=0x7f1358111050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:01.805 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:00.793+0000 7f135d000700 1 --2- 172.21.15.39:0/987785995 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f135810ec60 0x7f1358111050 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f13480098a0 tx=0x7f1348009bb0 comp rx=0 tx=0).stop 2024-03-18T15:58:01.805 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:00.793+0000 7f135d000700 1 -- 172.21.15.39:0/987785995 shutdown_connections 2024-03-18T15:58:01.805 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:00.793+0000 7f135d000700 1 --2- 172.21.15.39:0/987785995 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f135810ec60 0x7f1358111050 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:01.805 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:00.793+0000 7f135d000700 1 -- 172.21.15.39:0/987785995 >> 172.21.15.39:0/987785995 conn(0x7f135807d210 msgr2=0x7f135807f630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:01.805 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:00.793+0000 7f135d000700 1 -- 172.21.15.39:0/987785995 shutdown_connections 2024-03-18T15:58:01.805 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:00.793+0000 7f135d000700 1 -- 172.21.15.39:0/987785995 wait complete. 2024-03-18T15:58:01.805 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:00.795+0000 7f135d000700 1 Processor -- start 2024-03-18T15:58:01.805 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:00.795+0000 7f135d000700 1 -- start start 2024-03-18T15:58:01.806 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:00.795+0000 7f135d000700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f135810ec60 0x7f13581a2190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:01.806 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:00.795+0000 7f135d000700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f13581125a0 con 0x7f135810ec60 2024-03-18T15:58:01.806 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:00.796+0000 7f1355e00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f135810ec60 0x7f13581a2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:01.806 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:00.796+0000 7f1355e00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f135810ec60 0x7f13581a2190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:46756/0 (socket says 172.21.15.39:46756) 2024-03-18T15:58:01.810 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:00.796+0000 7f1355e00700 1 -- 172.21.15.39:0/1074828101 learned_addr learned my addr 172.21.15.39:0/1074828101 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T15:58:01.810 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:00.796+0000 7f1355e00700 1 -- 172.21.15.39:0/1074828101 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13581a26d0 con 0x7f135810ec60 2024-03-18T15:58:01.810 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:00.796+0000 7f1355e00700 1 --2- 172.21.15.39:0/1074828101 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f135810ec60 0x7f13581a2190 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1348011b60 tx=0x7f1348011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:01.810 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:00.797+0000 7f134ea00700 1 -- 172.21.15.39:0/1074828101 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1348003970 con 0x7f135810ec60 2024-03-18T15:58:01.810 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:00.797+0000 7f134ea00700 1 -- 172.21.15.39:0/1074828101 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f1348003b10 con 0x7f135810ec60 2024-03-18T15:58:01.811 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:00.797+0000 7f134ea00700 1 -- 172.21.15.39:0/1074828101 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1348024610 con 0x7f135810ec60 2024-03-18T15:58:01.811 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:00.797+0000 7f135d000700 1 -- 172.21.15.39:0/1074828101 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13581a2960 con 0x7f135810ec60 2024-03-18T15:58:01.811 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:00.797+0000 7f135d000700 1 -- 172.21.15.39:0/1074828101 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13581a5650 con 0x7f135810ec60 2024-03-18T15:58:01.811 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:00.798+0000 7f135d000700 1 -- 172.21.15.39:0/1074828101 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1358063900 con 0x7f135810ec60 2024-03-18T15:58:01.811 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:00.798+0000 7f134ea00700 1 -- 172.21.15.39:0/1074828101 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 4) ==== 46095+0+0 (secure 0 0 0) 0x7f134801a9f0 con 0x7f135810ec60 2024-03-18T15:58:01.811 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:00.799+0000 7f134ea00700 1 --2- 172.21.15.39:0/1074828101 >> [v2:172.21.15.39:6800/1217503189,v1:172.21.15.39:6801/1217503189] conn(0x7f133c038f20 0x7f133c03b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:01.811 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:00.799+0000 7f134ea00700 1 -- 172.21.15.39:0/1074828101 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f1348011e00 con 0x7f135810ec60 2024-03-18T15:58:01.811 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:00.799+0000 7f1355400700 1 --2- 172.21.15.39:0/1074828101 >> [v2:172.21.15.39:6800/1217503189,v1:172.21.15.39:6801/1217503189] conn(0x7f133c038f20 0x7f133c03b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:01.811 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:00.800+0000 7f1355400700 1 --2- 172.21.15.39:0/1074828101 >> [v2:172.21.15.39:6800/1217503189,v1:172.21.15.39:6801/1217503189] conn(0x7f133c038f20 0x7f133c03b3e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1344006fd0 tx=0x7f1344006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:01.811 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:00.804+0000 7f134ea00700 1 -- 172.21.15.39:0/1074828101 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f134800b3e0 con 0x7f135810ec60 2024-03-18T15:58:01.811 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:01.090+0000 7f135d000700 1 -- 172.21.15.39:0/1074828101 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7f13581a5820 con 0x7f135810ec60 2024-03-18T15:58:01.811 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:01.144+0000 7f134ea00700 1 -- 172.21.15.39:0/1074828101 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7f13581a5820 con 0x7f135810ec60 2024-03-18T15:58:01.811 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:01.147+0000 7f134ea00700 1 -- 172.21.15.39:0/1074828101 <== mon.0 v2:172.21.15.39:3300/0 8 ==== mgrmap(e 5) ==== 46106+0+0 (secure 0 0 0) 0x7f13480172b0 con 0x7f135810ec60 2024-03-18T15:58:01.812 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:01.148+0000 7f135d000700 1 -- 172.21.15.39:0/1074828101 >> [v2:172.21.15.39:6800/1217503189,v1:172.21.15.39:6801/1217503189] conn(0x7f133c038f20 msgr2=0x7f133c03b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:01.812 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:01.148+0000 7f135d000700 1 --2- 172.21.15.39:0/1074828101 >> [v2:172.21.15.39:6800/1217503189,v1:172.21.15.39:6801/1217503189] conn(0x7f133c038f20 0x7f133c03b3e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1344006fd0 tx=0x7f1344006e40 comp rx=0 tx=0).stop 2024-03-18T15:58:01.812 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:01.148+0000 7f135d000700 1 -- 172.21.15.39:0/1074828101 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f135810ec60 msgr2=0x7f13581a2190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:01.812 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:01.148+0000 7f135d000700 1 --2- 172.21.15.39:0/1074828101 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f135810ec60 0x7f13581a2190 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1348011b60 tx=0x7f1348011c40 comp rx=0 tx=0).stop 2024-03-18T15:58:01.812 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:01.148+0000 7f135d000700 1 -- 172.21.15.39:0/1074828101 shutdown_connections 2024-03-18T15:58:01.812 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:01.148+0000 7f135d000700 1 --2- 172.21.15.39:0/1074828101 >> [v2:172.21.15.39:6800/1217503189,v1:172.21.15.39:6801/1217503189] conn(0x7f133c038f20 0x7f133c03b3e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:01.812 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:01.148+0000 7f135d000700 1 --2- 172.21.15.39:0/1074828101 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f135810ec60 0x7f13581a2190 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:01.812 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:01.148+0000 7f135d000700 1 -- 172.21.15.39:0/1074828101 >> 172.21.15.39:0/1074828101 conn(0x7f135807d210 msgr2=0x7f135807dd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:01.812 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:01.149+0000 7f135d000700 1 -- 172.21.15.39:0/1074828101 shutdown_connections 2024-03-18T15:58:01.812 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:01.149+0000 7f135d000700 1 -- 172.21.15.39:0/1074828101 wait complete. 2024-03-18T15:58:02.086 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:58:01 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T15:58:01.963+0000 7f57bd8c7200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T15:58:02.406 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:58:02 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T15:58:02.090+0000 7f57bd8c7200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T15:58:02.407 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:02 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/1074828101' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-18T15:58:02.407 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:02 smithi039 ceph-mon[31721]: mgrmap e5: x(active, since 5s) 2024-03-18T15:58:03.072 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:58:02 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T15:58:02.943+0000 7f57bd8c7200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T15:58:03.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:03 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/3911689061' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-18T15:58:03.336 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:58:03 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T15:58:03.069+0000 7f57bd8c7200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T15:58:03.336 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:58:03 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T15:58:03.254+0000 7f57bd8c7200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T15:58:03.455 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout { 2024-03-18T15:58:03.455 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-18T15:58:03.455 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-18T15:58:03.455 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-18T15:58:03.455 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-18T15:58:03.455 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout } 2024-03-18T15:58:03.455 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:02.404+0000 7efd60800700 1 Processor -- start 2024-03-18T15:58:03.456 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:02.404+0000 7efd60800700 1 -- start start 2024-03-18T15:58:03.456 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:02.405+0000 7efd60800700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd58108680 0x7efd5810aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:03.456 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:02.405+0000 7efd60800700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7efd580759f0 con 0x7efd58108680 2024-03-18T15:58:03.456 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:02.405+0000 7efd5de00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd58108680 0x7efd5810aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:03.456 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:02.405+0000 7efd5de00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd58108680 0x7efd5810aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:46784/0 (socket says 172.21.15.39:46784) 2024-03-18T15:58:03.456 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:02.405+0000 7efd5de00700 1 -- 172.21.15.39:0/1762274227 learned_addr learned my addr 172.21.15.39:0/1762274227 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T15:58:03.456 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:02.406+0000 7efd5de00700 1 -- 172.21.15.39:0/1762274227 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd5810afb0 con 0x7efd58108680 2024-03-18T15:58:03.456 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:02.406+0000 7efd5de00700 1 --2- 172.21.15.39:0/1762274227 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd58108680 0x7efd5810aa70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7efd4c0098a0 tx=0x7efd4c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=737a906319522450 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:03.456 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:02.406+0000 7efd5ca00700 1 -- 172.21.15.39:0/1762274227 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efd4c004160 con 0x7efd58108680 2024-03-18T15:58:03.456 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:02.406+0000 7efd5ca00700 1 -- 172.21.15.39:0/1762274227 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7efd4c004300 con 0x7efd58108680 2024-03-18T15:58:03.456 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:02.406+0000 7efd5ca00700 1 -- 172.21.15.39:0/1762274227 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efd4c011b50 con 0x7efd58108680 2024-03-18T15:58:03.456 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:02.407+0000 7efd60800700 1 -- 172.21.15.39:0/1762274227 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd58108680 msgr2=0x7efd5810aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:03.456 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:02.407+0000 7efd60800700 1 --2- 172.21.15.39:0/1762274227 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd58108680 0x7efd5810aa70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7efd4c0098a0 tx=0x7efd4c009bb0 comp rx=0 tx=0).stop 2024-03-18T15:58:03.457 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:02.407+0000 7efd60800700 1 -- 172.21.15.39:0/1762274227 shutdown_connections 2024-03-18T15:58:03.457 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:02.407+0000 7efd60800700 1 --2- 172.21.15.39:0/1762274227 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd58108680 0x7efd5810aa70 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:03.457 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:02.407+0000 7efd60800700 1 -- 172.21.15.39:0/1762274227 >> 172.21.15.39:0/1762274227 conn(0x7efd581020d0 msgr2=0x7efd581044f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:03.457 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:02.407+0000 7efd60800700 1 -- 172.21.15.39:0/1762274227 shutdown_connections 2024-03-18T15:58:03.457 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:02.408+0000 7efd60800700 1 -- 172.21.15.39:0/1762274227 wait complete. 2024-03-18T15:58:03.457 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:02.409+0000 7efd60800700 1 Processor -- start 2024-03-18T15:58:03.457 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:02.409+0000 7efd60800700 1 -- start start 2024-03-18T15:58:03.457 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:02.409+0000 7efd60800700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd58108680 0x7efd5819dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:03.457 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:02.409+0000 7efd60800700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7efd5810bf70 con 0x7efd58108680 2024-03-18T15:58:03.457 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:02.409+0000 7efd5de00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd58108680 0x7efd5819dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:03.457 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:02.409+0000 7efd5de00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd58108680 0x7efd5819dd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:46792/0 (socket says 172.21.15.39:46792) 2024-03-18T15:58:03.457 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:02.409+0000 7efd5de00700 1 -- 172.21.15.39:0/3911689061 learned_addr learned my addr 172.21.15.39:0/3911689061 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T15:58:03.457 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:02.410+0000 7efd5de00700 1 -- 172.21.15.39:0/3911689061 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd5819e280 con 0x7efd58108680 2024-03-18T15:58:03.457 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:02.410+0000 7efd5de00700 1 --2- 172.21.15.39:0/3911689061 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd58108680 0x7efd5819dd40 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7efd4c011b60 tx=0x7efd4c011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:03.458 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:02.410+0000 7efd52a00700 1 -- 172.21.15.39:0/3911689061 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efd4c003b20 con 0x7efd58108680 2024-03-18T15:58:03.458 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:02.410+0000 7efd52a00700 1 -- 172.21.15.39:0/3911689061 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7efd4c003cc0 con 0x7efd58108680 2024-03-18T15:58:03.458 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:02.410+0000 7efd52a00700 1 -- 172.21.15.39:0/3911689061 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efd4c024840 con 0x7efd58108680 2024-03-18T15:58:03.458 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:02.410+0000 7efd60800700 1 -- 172.21.15.39:0/3911689061 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd5819e510 con 0x7efd58108680 2024-03-18T15:58:03.458 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:02.411+0000 7efd60800700 1 -- 172.21.15.39:0/3911689061 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd581a1200 con 0x7efd58108680 2024-03-18T15:58:03.458 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:02.412+0000 7efd60800700 1 -- 172.21.15.39:0/3911689061 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd58063900 con 0x7efd58108680 2024-03-18T15:58:03.458 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:02.412+0000 7efd52a00700 1 -- 172.21.15.39:0/3911689061 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) ==== 46106+0+0 (secure 0 0 0) 0x7efd4c022020 con 0x7efd58108680 2024-03-18T15:58:03.458 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:02.412+0000 7efd52a00700 1 --2- 172.21.15.39:0/3911689061 >> [v2:172.21.15.39:6800/1217503189,v1:172.21.15.39:6801/1217503189] conn(0x7efd40038f20 0x7efd4003b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:03.458 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:02.412+0000 7efd52a00700 1 -- 172.21.15.39:0/3911689061 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7efd4c011e00 con 0x7efd58108680 2024-03-18T15:58:03.458 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:02.413+0000 7efd5d400700 1 -- 172.21.15.39:0/3911689061 >> [v2:172.21.15.39:6800/1217503189,v1:172.21.15.39:6801/1217503189] conn(0x7efd40038f20 msgr2=0x7efd4003b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.39:6800/1217503189 2024-03-18T15:58:03.458 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:02.413+0000 7efd5d400700 1 --2- 172.21.15.39:0/3911689061 >> [v2:172.21.15.39:6800/1217503189,v1:172.21.15.39:6801/1217503189] conn(0x7efd40038f20 0x7efd4003b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T15:58:03.458 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:02.418+0000 7efd52a00700 1 -- 172.21.15.39:0/3911689061 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7efd4c04c9a0 con 0x7efd58108680 2024-03-18T15:58:03.458 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:02.614+0000 7efd5d400700 1 -- 172.21.15.39:0/3911689061 >> [v2:172.21.15.39:6800/1217503189,v1:172.21.15.39:6801/1217503189] conn(0x7efd40038f20 msgr2=0x7efd4003b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.39:6800/1217503189 2024-03-18T15:58:03.458 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:02.614+0000 7efd5d400700 1 --2- 172.21.15.39:0/3911689061 >> [v2:172.21.15.39:6800/1217503189,v1:172.21.15.39:6801/1217503189] conn(0x7efd40038f20 0x7efd4003b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T15:58:03.459 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:02.704+0000 7efd60800700 1 -- 172.21.15.39:0/3911689061 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7efd581a13d0 con 0x7efd58108680 2024-03-18T15:58:03.459 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:02.705+0000 7efd52a00700 1 -- 172.21.15.39:0/3911689061 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7efd581a13d0 con 0x7efd58108680 2024-03-18T15:58:03.459 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:02.708+0000 7efd60800700 1 -- 172.21.15.39:0/3911689061 >> [v2:172.21.15.39:6800/1217503189,v1:172.21.15.39:6801/1217503189] conn(0x7efd40038f20 msgr2=0x7efd4003b3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T15:58:03.459 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:02.708+0000 7efd60800700 1 --2- 172.21.15.39:0/3911689061 >> [v2:172.21.15.39:6800/1217503189,v1:172.21.15.39:6801/1217503189] conn(0x7efd40038f20 0x7efd4003b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:03.459 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:02.708+0000 7efd60800700 1 -- 172.21.15.39:0/3911689061 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd58108680 msgr2=0x7efd5819dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:03.459 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:02.708+0000 7efd60800700 1 --2- 172.21.15.39:0/3911689061 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd58108680 0x7efd5819dd40 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7efd4c011b60 tx=0x7efd4c011c40 comp rx=0 tx=0).stop 2024-03-18T15:58:03.459 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:02.708+0000 7efd60800700 1 -- 172.21.15.39:0/3911689061 shutdown_connections 2024-03-18T15:58:03.459 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:02.708+0000 7efd60800700 1 --2- 172.21.15.39:0/3911689061 >> [v2:172.21.15.39:6800/1217503189,v1:172.21.15.39:6801/1217503189] conn(0x7efd40038f20 0x7efd4003b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:03.459 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:02.708+0000 7efd60800700 1 --2- 172.21.15.39:0/3911689061 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efd58108680 0x7efd5819dd40 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:03.459 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:02.708+0000 7efd60800700 1 -- 172.21.15.39:0/3911689061 >> 172.21.15.39:0/3911689061 conn(0x7efd581020d0 msgr2=0x7efd58102bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:03.459 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:02.709+0000 7efd60800700 1 -- 172.21.15.39:0/3911689061 shutdown_connections 2024-03-18T15:58:03.459 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:02.709+0000 7efd60800700 1 -- 172.21.15.39:0/3911689061 wait complete. 2024-03-18T15:58:03.459 INFO:teuthology.orchestra.run.smithi039.stdout:Waiting for the mgr to restart... 2024-03-18T15:58:03.459 INFO:teuthology.orchestra.run.smithi039.stdout:Waiting for mgr epoch 5... 2024-03-18T15:58:03.650 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:58:03 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T15:58:03.384+0000 7f57bd8c7200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T15:58:03.650 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:58:03 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T15:58:03.502+0000 7f57bd8c7200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T15:58:03.903 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:58:03 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T15:58:03.850+0000 7f57bd8c7200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T15:58:04.248 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:58:04 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T15:58:03.998+0000 7f57bd8c7200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T15:58:04.248 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:58:04 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T15:58:04.118+0000 7f57bd8c7200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T15:58:04.248 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:58:04 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T15:58:04.245+0000 7f57bd8c7200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T15:58:04.586 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:58:04 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T15:58:04.469+0000 7f57bd8c7200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T15:58:06.086 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:58:05 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T15:58:05.786+0000 7f57bd8c7200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T15:58:06.086 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:58:05 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T15:58:05.965+0000 7f57bd8c7200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T15:58:06.783 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:58:06 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T15:58:06.392+0000 7f57bd8c7200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T15:58:06.783 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:58:06 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T15:58:06.515+0000 7f57bd8c7200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T15:58:07.035 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:58:06 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T15:58:06.781+0000 7f57bd8c7200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T15:58:07.336 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:58:07 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T15:58:07.033+0000 7f57bd8c7200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T15:58:07.746 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:58:07 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T15:58:07.419+0000 7f57bd8c7200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T15:58:07.746 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:58:07 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T15:58:07.743+0000 7f57bd8c7200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T15:58:08.336 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:58:08 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T15:58:08.056+0000 7f57bd8c7200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T15:58:09.586 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:58:09 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T15:58:09.119+0000 7f57bd8c7200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T15:58:10.262 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:58:09 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T15:58:09.838+0000 7f57bd8c7200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T15:58:10.262 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:58:09 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T15:58:09.971+0000 7f57bd8c7200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T15:58:10.586 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:58:10 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T15:58:10.260+0000 7f57bd8c7200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T15:58:10.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:10 smithi039 ceph-mon[31721]: Active manager daemon x restarted 2024-03-18T15:58:10.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:10 smithi039 ceph-mon[31721]: Activating manager daemon x 2024-03-18T15:58:10.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:10 smithi039 ceph-mon[31721]: osdmap e2: 0 total, 0 up, 0 in 2024-03-18T15:58:10.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:10 smithi039 ceph-mon[31721]: mgrmap e6: x(active, starting, since 0.0091342s) 2024-03-18T15:58:10.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:10 smithi039 ceph-mon[31721]: from='mgr.14120 172.21.15.39:0/2752076370' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T15:58:10.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:10 smithi039 ceph-mon[31721]: from='mgr.14120 172.21.15.39:0/2752076370' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-18T15:58:10.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:10 smithi039 ceph-mon[31721]: from='mgr.14120 172.21.15.39:0/2752076370' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-18T15:58:10.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:10 smithi039 ceph-mon[31721]: from='mgr.14120 172.21.15.39:0/2752076370' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-18T15:58:10.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:10 smithi039 ceph-mon[31721]: from='mgr.14120 172.21.15.39:0/2752076370' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-18T15:58:10.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:10 smithi039 ceph-mon[31721]: Manager daemon x is now available 2024-03-18T15:58:11.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:11 smithi039 ceph-mon[31721]: Found migration_current of "None". Setting to last migration. 2024-03-18T15:58:11.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:11 smithi039 ceph-mon[31721]: from='mgr.14120 172.21.15.39:0/2752076370' entity='mgr.x' 2024-03-18T15:58:11.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:11 smithi039 ceph-mon[31721]: from='mgr.14120 172.21.15.39:0/2752076370' entity='mgr.x' 2024-03-18T15:58:11.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:11 smithi039 ceph-mon[31721]: from='mgr.14120 172.21.15.39:0/2752076370' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:58:11.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:11 smithi039 ceph-mon[31721]: from='mgr.14120 172.21.15.39:0/2752076370' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:58:11.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:11 smithi039 ceph-mon[31721]: from='mgr.14120 172.21.15.39:0/2752076370' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:58:11.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:11 smithi039 ceph-mon[31721]: from='mgr.14120 172.21.15.39:0/2752076370' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-18T15:58:11.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:11 smithi039 ceph-mon[31721]: from='mgr.14120 172.21.15.39:0/2752076370' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-18T15:58:11.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:11 smithi039 ceph-mon[31721]: from='mgr.14120 172.21.15.39:0/2752076370' entity='mgr.x' 2024-03-18T15:58:11.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:11 smithi039 ceph-mon[31721]: from='mgr.14120 172.21.15.39:0/2752076370' entity='mgr.x' 2024-03-18T15:58:11.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:11 smithi039 ceph-mon[31721]: mgrmap e7: x(active, since 1.01616s) 2024-03-18T15:58:12.004 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout { 2024-03-18T15:58:12.004 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-18T15:58:12.004 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-18T15:58:12.004 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout } 2024-03-18T15:58:12.004 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:04.050+0000 7ff19a200700 1 Processor -- start 2024-03-18T15:58:12.004 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:04.050+0000 7ff19a200700 1 -- start start 2024-03-18T15:58:12.005 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:04.051+0000 7ff19a200700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff194108740 0x7ff19410ab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:12.005 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:04.051+0000 7ff19a200700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7ff194075930 con 0x7ff194108740 2024-03-18T15:58:12.005 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:04.051+0000 7ff193400700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff194108740 0x7ff19410ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:12.005 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:04.051+0000 7ff193400700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff194108740 0x7ff19410ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:46804/0 (socket says 172.21.15.39:46804) 2024-03-18T15:58:12.005 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:04.051+0000 7ff193400700 1 -- 172.21.15.39:0/2841341039 learned_addr learned my addr 172.21.15.39:0/2841341039 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T15:58:12.005 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:04.052+0000 7ff193400700 1 -- 172.21.15.39:0/2841341039 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff19410b070 con 0x7ff194108740 2024-03-18T15:58:12.005 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:04.052+0000 7ff193400700 1 --2- 172.21.15.39:0/2841341039 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff194108740 0x7ff19410ab30 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff1840098a0 tx=0x7ff184009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76d5deb6cacbc80a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:12.005 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:04.052+0000 7ff192000700 1 -- 172.21.15.39:0/2841341039 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff184004160 con 0x7ff194108740 2024-03-18T15:58:12.005 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:04.052+0000 7ff192000700 1 -- 172.21.15.39:0/2841341039 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7ff184004300 con 0x7ff194108740 2024-03-18T15:58:12.005 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:04.052+0000 7ff192000700 1 -- 172.21.15.39:0/2841341039 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff184011b50 con 0x7ff194108740 2024-03-18T15:58:12.005 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:04.053+0000 7ff19a200700 1 -- 172.21.15.39:0/2841341039 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff194108740 msgr2=0x7ff19410ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:12.005 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:04.053+0000 7ff19a200700 1 --2- 172.21.15.39:0/2841341039 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff194108740 0x7ff19410ab30 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff1840098a0 tx=0x7ff184009bb0 comp rx=0 tx=0).stop 2024-03-18T15:58:12.005 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:04.053+0000 7ff19a200700 1 -- 172.21.15.39:0/2841341039 shutdown_connections 2024-03-18T15:58:12.006 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:04.053+0000 7ff19a200700 1 --2- 172.21.15.39:0/2841341039 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff194108740 0x7ff19410ab30 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:12.006 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:04.053+0000 7ff19a200700 1 -- 172.21.15.39:0/2841341039 >> 172.21.15.39:0/2841341039 conn(0x7ff1941020d0 msgr2=0x7ff1941044f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:12.006 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:04.053+0000 7ff19a200700 1 -- 172.21.15.39:0/2841341039 shutdown_connections 2024-03-18T15:58:12.006 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:04.053+0000 7ff19a200700 1 -- 172.21.15.39:0/2841341039 wait complete. 2024-03-18T15:58:12.006 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:04.055+0000 7ff19a200700 1 Processor -- start 2024-03-18T15:58:12.006 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:04.055+0000 7ff19a200700 1 -- start start 2024-03-18T15:58:12.006 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:04.055+0000 7ff19a200700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff194108740 0x7ff19419bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:12.006 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:04.055+0000 7ff19a200700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7ff19410bf70 con 0x7ff194108740 2024-03-18T15:58:12.006 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:04.056+0000 7ff193400700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff194108740 0x7ff19419bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:12.006 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:04.056+0000 7ff193400700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff194108740 0x7ff19419bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:46818/0 (socket says 172.21.15.39:46818) 2024-03-18T15:58:12.006 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:04.056+0000 7ff193400700 1 -- 172.21.15.39:0/3785548990 learned_addr learned my addr 172.21.15.39:0/3785548990 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T15:58:12.006 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:04.056+0000 7ff193400700 1 -- 172.21.15.39:0/3785548990 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff19419bff0 con 0x7ff194108740 2024-03-18T15:58:12.006 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:04.056+0000 7ff193400700 1 --2- 172.21.15.39:0/3785548990 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff194108740 0x7ff19419bab0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff184011b60 tx=0x7ff184011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:12.006 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:04.057+0000 7ff18be00700 1 -- 172.21.15.39:0/3785548990 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff184003b20 con 0x7ff194108740 2024-03-18T15:58:12.007 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:04.057+0000 7ff18be00700 1 -- 172.21.15.39:0/3785548990 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7ff184003cc0 con 0x7ff194108740 2024-03-18T15:58:12.007 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:04.057+0000 7ff19a200700 1 -- 172.21.15.39:0/3785548990 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff19419c280 con 0x7ff194108740 2024-03-18T15:58:12.007 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:04.057+0000 7ff18be00700 1 -- 172.21.15.39:0/3785548990 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff184024840 con 0x7ff194108740 2024-03-18T15:58:12.007 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:04.057+0000 7ff19a200700 1 -- 172.21.15.39:0/3785548990 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff19419ef70 con 0x7ff194108740 2024-03-18T15:58:12.007 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:04.058+0000 7ff18be00700 1 -- 172.21.15.39:0/3785548990 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) ==== 46106+0+0 (secure 0 0 0) 0x7ff184022020 con 0x7ff194108740 2024-03-18T15:58:12.007 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:04.059+0000 7ff18be00700 1 --2- 172.21.15.39:0/3785548990 >> [v2:172.21.15.39:6800/1217503189,v1:172.21.15.39:6801/1217503189] conn(0x7ff178038f20 0x7ff17803b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:12.007 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:04.059+0000 7ff18be00700 1 -- 172.21.15.39:0/3785548990 --> [v2:172.21.15.39:6800/1217503189,v1:172.21.15.39:6801/1217503189] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7ff17803baf0 con 0x7ff178038f20 2024-03-18T15:58:12.007 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:04.059+0000 7ff18be00700 1 -- 172.21.15.39:0/3785548990 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7ff184011e00 con 0x7ff194108740 2024-03-18T15:58:12.007 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:04.059+0000 7ff192a00700 1 -- 172.21.15.39:0/3785548990 >> [v2:172.21.15.39:6800/1217503189,v1:172.21.15.39:6801/1217503189] conn(0x7ff178038f20 msgr2=0x7ff17803b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.39:6800/1217503189 2024-03-18T15:58:12.007 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:04.059+0000 7ff192a00700 1 --2- 172.21.15.39:0/3785548990 >> [v2:172.21.15.39:6800/1217503189,v1:172.21.15.39:6801/1217503189] conn(0x7ff178038f20 0x7ff17803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T15:58:12.007 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:04.260+0000 7ff192a00700 1 -- 172.21.15.39:0/3785548990 >> [v2:172.21.15.39:6800/1217503189,v1:172.21.15.39:6801/1217503189] conn(0x7ff178038f20 msgr2=0x7ff17803b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.39:6800/1217503189 2024-03-18T15:58:12.008 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:04.260+0000 7ff192a00700 1 --2- 172.21.15.39:0/3785548990 >> [v2:172.21.15.39:6800/1217503189,v1:172.21.15.39:6801/1217503189] conn(0x7ff178038f20 0x7ff17803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T15:58:12.008 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:04.661+0000 7ff192a00700 1 -- 172.21.15.39:0/3785548990 >> [v2:172.21.15.39:6800/1217503189,v1:172.21.15.39:6801/1217503189] conn(0x7ff178038f20 msgr2=0x7ff17803b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.39:6800/1217503189 2024-03-18T15:58:12.008 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:04.661+0000 7ff192a00700 1 --2- 172.21.15.39:0/3785548990 >> [v2:172.21.15.39:6800/1217503189,v1:172.21.15.39:6801/1217503189] conn(0x7ff178038f20 0x7ff17803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-18T15:58:12.008 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:05.462+0000 7ff192a00700 1 -- 172.21.15.39:0/3785548990 >> [v2:172.21.15.39:6800/1217503189,v1:172.21.15.39:6801/1217503189] conn(0x7ff178038f20 msgr2=0x7ff17803b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.39:6800/1217503189 2024-03-18T15:58:12.008 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:05.462+0000 7ff192a00700 1 --2- 172.21.15.39:0/3785548990 >> [v2:172.21.15.39:6800/1217503189,v1:172.21.15.39:6801/1217503189] conn(0x7ff178038f20 0x7ff17803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-18T15:58:12.008 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:07.064+0000 7ff192a00700 1 -- 172.21.15.39:0/3785548990 >> [v2:172.21.15.39:6800/1217503189,v1:172.21.15.39:6801/1217503189] conn(0x7ff178038f20 msgr2=0x7ff17803b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.39:6800/1217503189 2024-03-18T15:58:12.008 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:07.064+0000 7ff192a00700 1 --2- 172.21.15.39:0/3785548990 >> [v2:172.21.15.39:6800/1217503189,v1:172.21.15.39:6801/1217503189] conn(0x7ff178038f20 0x7ff17803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-18T15:58:12.008 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:10.268+0000 7ff192a00700 1 -- 172.21.15.39:0/3785548990 >> [v2:172.21.15.39:6800/1217503189,v1:172.21.15.39:6801/1217503189] conn(0x7ff178038f20 msgr2=0x7ff17803b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.39:6800/1217503189 2024-03-18T15:58:12.008 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:10.268+0000 7ff192a00700 1 --2- 172.21.15.39:0/3785548990 >> [v2:172.21.15.39:6800/1217503189,v1:172.21.15.39:6801/1217503189] conn(0x7ff178038f20 0x7ff17803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-03-18T15:58:12.008 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:10.273+0000 7ff18be00700 1 -- 172.21.15.39:0/3785548990 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mgrmap(e 6) ==== 45873+0+0 (secure 0 0 0) 0x7ff184024af0 con 0x7ff194108740 2024-03-18T15:58:12.009 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:10.273+0000 7ff18be00700 1 -- 172.21.15.39:0/3785548990 >> [v2:172.21.15.39:6800/1217503189,v1:172.21.15.39:6801/1217503189] conn(0x7ff178038f20 msgr2=0x7ff17803b3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T15:58:12.009 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:10.273+0000 7ff18be00700 1 --2- 172.21.15.39:0/3785548990 >> [v2:172.21.15.39:6800/1217503189,v1:172.21.15.39:6801/1217503189] conn(0x7ff178038f20 0x7ff17803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:12.009 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:11.276+0000 7ff18be00700 1 -- 172.21.15.39:0/3785548990 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mgrmap(e 7) ==== 46000+0+0 (secure 0 0 0) 0x7ff18404e530 con 0x7ff194108740 2024-03-18T15:58:12.009 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:11.276+0000 7ff18be00700 1 --2- 172.21.15.39:0/3785548990 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7ff17803c810 0x7ff17803ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:12.009 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:11.276+0000 7ff18be00700 1 -- 172.21.15.39:0/3785548990 --> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7ff184016e70 con 0x7ff17803c810 2024-03-18T15:58:12.009 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:11.279+0000 7ff192a00700 1 --2- 172.21.15.39:0/3785548990 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7ff17803c810 0x7ff17803ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:12.009 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:11.280+0000 7ff192a00700 1 --2- 172.21.15.39:0/3785548990 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7ff17803c810 0x7ff17803ec00 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7ff180003a10 tx=0x7ff180009290 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:12.009 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:11.282+0000 7ff18be00700 1 -- 172.21.15.39:0/3785548990 <== mgr.14120 v2:172.21.15.39:6800/1810238546 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7ff184016e70 con 0x7ff17803c810 2024-03-18T15:58:12.009 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:11.289+0000 7ff19a200700 1 -- 172.21.15.39:0/3785548990 --> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7ff194063900 con 0x7ff17803c810 2024-03-18T15:58:12.009 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:11.290+0000 7ff18be00700 1 -- 172.21.15.39:0/3785548990 <== mgr.14120 v2:172.21.15.39:6800/1810238546 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7ff194063900 con 0x7ff17803c810 2024-03-18T15:58:12.009 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:11.290+0000 7ff19a200700 1 -- 172.21.15.39:0/3785548990 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7ff17803c810 msgr2=0x7ff17803ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:12.009 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:11.290+0000 7ff19a200700 1 --2- 172.21.15.39:0/3785548990 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7ff17803c810 0x7ff17803ec00 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7ff180003a10 tx=0x7ff180009290 comp rx=0 tx=0).stop 2024-03-18T15:58:12.010 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:11.290+0000 7ff19a200700 1 -- 172.21.15.39:0/3785548990 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff194108740 msgr2=0x7ff19419bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:12.010 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:11.290+0000 7ff19a200700 1 --2- 172.21.15.39:0/3785548990 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff194108740 0x7ff19419bab0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff184011b60 tx=0x7ff184011c40 comp rx=0 tx=0).stop 2024-03-18T15:58:12.010 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:11.290+0000 7ff19a200700 1 -- 172.21.15.39:0/3785548990 shutdown_connections 2024-03-18T15:58:12.010 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:11.290+0000 7ff19a200700 1 --2- 172.21.15.39:0/3785548990 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7ff17803c810 0x7ff17803ec00 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:12.010 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:11.290+0000 7ff19a200700 1 --2- 172.21.15.39:0/3785548990 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff194108740 0x7ff19419bab0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:12.010 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:11.290+0000 7ff19a200700 1 --2- 172.21.15.39:0/3785548990 >> [v2:172.21.15.39:6800/1217503189,v1:172.21.15.39:6801/1217503189] conn(0x7ff178038f20 0x7ff17803b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:12.010 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:11.290+0000 7ff19a200700 1 -- 172.21.15.39:0/3785548990 >> 172.21.15.39:0/3785548990 conn(0x7ff1941020d0 msgr2=0x7ff194102bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:12.010 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:11.291+0000 7ff19a200700 1 -- 172.21.15.39:0/3785548990 shutdown_connections 2024-03-18T15:58:12.010 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:11.291+0000 7ff19a200700 1 -- 172.21.15.39:0/3785548990 wait complete. 2024-03-18T15:58:12.010 INFO:teuthology.orchestra.run.smithi039.stdout:mgr epoch 5 is available 2024-03-18T15:58:12.011 INFO:teuthology.orchestra.run.smithi039.stdout:Setting orchestrator backend to cephadm... 2024-03-18T15:58:12.551 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:12 smithi039 ceph-mon[31721]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-18T15:58:12.552 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:12 smithi039 ceph-mon[31721]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-18T15:58:12.552 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:12 smithi039 ceph-mon[31721]: [18/Mar/2024:15:58:11] ENGINE Bus STARTING 2024-03-18T15:58:12.552 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:12 smithi039 ceph-mon[31721]: [18/Mar/2024:15:58:11] ENGINE Serving on http://172.21.15.39:8765 2024-03-18T15:58:12.552 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:12 smithi039 ceph-mon[31721]: [18/Mar/2024:15:58:12] ENGINE Serving on https://172.21.15.39:7150 2024-03-18T15:58:12.552 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:12 smithi039 ceph-mon[31721]: [18/Mar/2024:15:58:12] ENGINE Bus STARTED 2024-03-18T15:58:12.552 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:12 smithi039 ceph-mon[31721]: from='mgr.14120 172.21.15.39:0/2752076370' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:58:13.698 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:13 smithi039 ceph-mon[31721]: mgrmap e8: x(active, since 2s) 2024-03-18T15:58:13.698 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:13 smithi039 ceph-mon[31721]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:58:13.698 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:13 smithi039 ceph-mon[31721]: from='mgr.14120 172.21.15.39:0/2752076370' entity='mgr.x' 2024-03-18T15:58:13.698 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:13 smithi039 ceph-mon[31721]: from='mgr.14120 172.21.15.39:0/2752076370' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:58:13.720 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:12.721+0000 7fc983600700 1 Processor -- start 2024-03-18T15:58:13.720 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:12.721+0000 7fc983600700 1 -- start start 2024-03-18T15:58:13.720 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:12.721+0000 7fc983600700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc97c108700 0x7fc97c10aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:13.721 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:12.721+0000 7fc983600700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fc97c075970 con 0x7fc97c108700 2024-03-18T15:58:13.721 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:12.722+0000 7fc980c00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc97c108700 0x7fc97c10aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:13.721 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:12.722+0000 7fc980c00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc97c108700 0x7fc97c10aaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:33312/0 (socket says 172.21.15.39:33312) 2024-03-18T15:58:13.721 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:12.722+0000 7fc980c00700 1 -- 172.21.15.39:0/1270813666 learned_addr learned my addr 172.21.15.39:0/1270813666 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T15:58:13.721 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:12.722+0000 7fc980c00700 1 -- 172.21.15.39:0/1270813666 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc97c10b030 con 0x7fc97c108700 2024-03-18T15:58:13.721 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:12.723+0000 7fc980c00700 1 --2- 172.21.15.39:0/1270813666 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc97c108700 0x7fc97c10aaf0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc96c0098a0 tx=0x7fc96c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51b811b2d045112a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:13.721 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:12.723+0000 7fc97be00700 1 -- 172.21.15.39:0/1270813666 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc96c004160 con 0x7fc97c108700 2024-03-18T15:58:13.721 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:12.723+0000 7fc97be00700 1 -- 172.21.15.39:0/1270813666 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fc96c004300 con 0x7fc97c108700 2024-03-18T15:58:13.721 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:12.723+0000 7fc97be00700 1 -- 172.21.15.39:0/1270813666 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc96c011b50 con 0x7fc97c108700 2024-03-18T15:58:13.721 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:12.724+0000 7fc983600700 1 -- 172.21.15.39:0/1270813666 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc97c108700 msgr2=0x7fc97c10aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:13.721 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:12.724+0000 7fc983600700 1 --2- 172.21.15.39:0/1270813666 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc97c108700 0x7fc97c10aaf0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc96c0098a0 tx=0x7fc96c009bb0 comp rx=0 tx=0).stop 2024-03-18T15:58:13.721 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:12.724+0000 7fc983600700 1 -- 172.21.15.39:0/1270813666 shutdown_connections 2024-03-18T15:58:13.722 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:12.724+0000 7fc983600700 1 --2- 172.21.15.39:0/1270813666 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc97c108700 0x7fc97c10aaf0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:13.722 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:12.724+0000 7fc983600700 1 -- 172.21.15.39:0/1270813666 >> 172.21.15.39:0/1270813666 conn(0x7fc97c1020d0 msgr2=0x7fc97c1044f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:13.722 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:12.724+0000 7fc983600700 1 -- 172.21.15.39:0/1270813666 shutdown_connections 2024-03-18T15:58:13.722 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:12.724+0000 7fc983600700 1 -- 172.21.15.39:0/1270813666 wait complete. 2024-03-18T15:58:13.722 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:12.725+0000 7fc983600700 1 Processor -- start 2024-03-18T15:58:13.722 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:12.725+0000 7fc983600700 1 -- start start 2024-03-18T15:58:13.722 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:12.725+0000 7fc983600700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc97c108700 0x7fc97c19bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:13.722 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:12.725+0000 7fc983600700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fc97c10bf80 con 0x7fc97c108700 2024-03-18T15:58:13.722 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:12.726+0000 7fc980c00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc97c108700 0x7fc97c19bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:13.722 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:12.726+0000 7fc980c00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc97c108700 0x7fc97c19bb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:33322/0 (socket says 172.21.15.39:33322) 2024-03-18T15:58:13.722 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:12.726+0000 7fc980c00700 1 -- 172.21.15.39:0/1940998157 learned_addr learned my addr 172.21.15.39:0/1940998157 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T15:58:13.722 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:12.726+0000 7fc980c00700 1 -- 172.21.15.39:0/1940998157 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc97c19c0a0 con 0x7fc97c108700 2024-03-18T15:58:13.722 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:12.726+0000 7fc980c00700 1 --2- 172.21.15.39:0/1940998157 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc97c108700 0x7fc97c19bb60 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fc96c011b60 tx=0x7fc96c011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:13.722 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:12.727+0000 7fc97a000700 1 -- 172.21.15.39:0/1940998157 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc96c003b20 con 0x7fc97c108700 2024-03-18T15:58:13.723 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:12.727+0000 7fc97a000700 1 -- 172.21.15.39:0/1940998157 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fc96c003cc0 con 0x7fc97c108700 2024-03-18T15:58:13.723 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:12.727+0000 7fc983600700 1 -- 172.21.15.39:0/1940998157 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc97c19c330 con 0x7fc97c108700 2024-03-18T15:58:13.723 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:12.727+0000 7fc97a000700 1 -- 172.21.15.39:0/1940998157 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc96c024840 con 0x7fc97c108700 2024-03-18T15:58:13.723 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:12.727+0000 7fc983600700 1 -- 172.21.15.39:0/1940998157 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc97c19efd0 con 0x7fc97c108700 2024-03-18T15:58:13.723 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:12.728+0000 7fc97a000700 1 -- 172.21.15.39:0/1940998157 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7fc96c022020 con 0x7fc97c108700 2024-03-18T15:58:13.723 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:12.728+0000 7fc983600700 1 -- 172.21.15.39:0/1940998157 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc97c063900 con 0x7fc97c108700 2024-03-18T15:58:13.723 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:12.728+0000 7fc97a000700 1 --2- 172.21.15.39:0/1940998157 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7fc960038f70 0x7fc96003b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:13.723 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:12.729+0000 7fc97a000700 1 -- 172.21.15.39:0/1940998157 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fc96c011e00 con 0x7fc97c108700 2024-03-18T15:58:13.723 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:12.729+0000 7fc973e00700 1 --2- 172.21.15.39:0/1940998157 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7fc960038f70 0x7fc96003b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:13.723 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:12.729+0000 7fc973e00700 1 --2- 172.21.15.39:0/1940998157 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7fc960038f70 0x7fc96003b430 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fc964006fd0 tx=0x7fc964006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:13.723 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:12.735+0000 7fc97a000700 1 -- 172.21.15.39:0/1940998157 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc96c04c960 con 0x7fc97c108700 2024-03-18T15:58:13.723 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:12.977+0000 7fc983600700 1 -- 172.21.15.39:0/1940998157 --> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7fc97c1032d0 con 0x7fc960038f70 2024-03-18T15:58:13.723 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:12.991+0000 7fc97a000700 1 -- 172.21.15.39:0/1940998157 <== mgr.14120 v2:172.21.15.39:6800/1810238546 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fc97c1032d0 con 0x7fc960038f70 2024-03-18T15:58:13.724 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:12.994+0000 7fc983600700 1 -- 172.21.15.39:0/1940998157 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7fc960038f70 msgr2=0x7fc96003b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:13.724 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:12.994+0000 7fc983600700 1 --2- 172.21.15.39:0/1940998157 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7fc960038f70 0x7fc96003b430 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fc964006fd0 tx=0x7fc964006e40 comp rx=0 tx=0).stop 2024-03-18T15:58:13.724 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:12.994+0000 7fc983600700 1 -- 172.21.15.39:0/1940998157 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc97c108700 msgr2=0x7fc97c19bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:13.724 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:12.994+0000 7fc983600700 1 --2- 172.21.15.39:0/1940998157 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc97c108700 0x7fc97c19bb60 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fc96c011b60 tx=0x7fc96c011c40 comp rx=0 tx=0).stop 2024-03-18T15:58:13.724 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:12.994+0000 7fc983600700 1 -- 172.21.15.39:0/1940998157 shutdown_connections 2024-03-18T15:58:13.724 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:12.994+0000 7fc983600700 1 --2- 172.21.15.39:0/1940998157 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7fc960038f70 0x7fc96003b430 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:13.724 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:12.994+0000 7fc983600700 1 --2- 172.21.15.39:0/1940998157 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc97c108700 0x7fc97c19bb60 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:13.724 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:12.994+0000 7fc983600700 1 -- 172.21.15.39:0/1940998157 >> 172.21.15.39:0/1940998157 conn(0x7fc97c1020d0 msgr2=0x7fc97c102bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:13.724 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:12.994+0000 7fc983600700 1 -- 172.21.15.39:0/1940998157 shutdown_connections 2024-03-18T15:58:13.724 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:12.995+0000 7fc983600700 1 -- 172.21.15.39:0/1940998157 wait complete. 2024-03-18T15:58:15.263 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-18T15:58:15.265 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:14.312+0000 7f0991e00700 1 Processor -- start 2024-03-18T15:58:15.265 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:14.312+0000 7f0991e00700 1 -- start start 2024-03-18T15:58:15.265 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:14.313+0000 7f0991e00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f098c1086c0 0x7f098c10aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:15.265 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:14.313+0000 7f0991e00700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f098c0759b0 con 0x7f098c1086c0 2024-03-18T15:58:15.265 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:14.313+0000 7f098aa00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f098c1086c0 0x7f098c10aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:15.265 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:14.313+0000 7f098aa00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f098c1086c0 0x7f098c10aab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:33324/0 (socket says 172.21.15.39:33324) 2024-03-18T15:58:15.265 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:14.313+0000 7f098aa00700 1 -- 172.21.15.39:0/1287411245 learned_addr learned my addr 172.21.15.39:0/1287411245 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T15:58:15.265 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:14.314+0000 7f098aa00700 1 -- 172.21.15.39:0/1287411245 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f098c10aff0 con 0x7f098c1086c0 2024-03-18T15:58:15.269 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:14.314+0000 7f098aa00700 1 --2- 172.21.15.39:0/1287411245 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f098c1086c0 0x7f098c10aab0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f097c0098a0 tx=0x7f097c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=635a2f21de317256 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:15.270 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:14.315+0000 7f0989600700 1 -- 172.21.15.39:0/1287411245 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f097c004160 con 0x7f098c1086c0 2024-03-18T15:58:15.270 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:14.315+0000 7f0989600700 1 -- 172.21.15.39:0/1287411245 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f097c004300 con 0x7f098c1086c0 2024-03-18T15:58:15.270 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:14.315+0000 7f0989600700 1 -- 172.21.15.39:0/1287411245 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f097c011b50 con 0x7f098c1086c0 2024-03-18T15:58:15.270 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:14.315+0000 7f0991e00700 1 -- 172.21.15.39:0/1287411245 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f098c1086c0 msgr2=0x7f098c10aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:15.270 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:14.315+0000 7f0991e00700 1 --2- 172.21.15.39:0/1287411245 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f098c1086c0 0x7f098c10aab0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f097c0098a0 tx=0x7f097c009bb0 comp rx=0 tx=0).stop 2024-03-18T15:58:15.270 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:14.316+0000 7f0991e00700 1 -- 172.21.15.39:0/1287411245 shutdown_connections 2024-03-18T15:58:15.270 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:14.316+0000 7f0991e00700 1 --2- 172.21.15.39:0/1287411245 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f098c1086c0 0x7f098c10aab0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:15.270 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:14.316+0000 7f0991e00700 1 -- 172.21.15.39:0/1287411245 >> 172.21.15.39:0/1287411245 conn(0x7f098c1020d0 msgr2=0x7f098c1044f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:15.270 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:14.316+0000 7f0991e00700 1 -- 172.21.15.39:0/1287411245 shutdown_connections 2024-03-18T15:58:15.270 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:14.316+0000 7f0991e00700 1 -- 172.21.15.39:0/1287411245 wait complete. 2024-03-18T15:58:15.270 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:14.317+0000 7f0991e00700 1 Processor -- start 2024-03-18T15:58:15.271 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:14.317+0000 7f0991e00700 1 -- start start 2024-03-18T15:58:15.271 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:14.317+0000 7f0991e00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f098c1086c0 0x7f098c19dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:15.271 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:14.317+0000 7f0991e00700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f098c10bfa0 con 0x7f098c1086c0 2024-03-18T15:58:15.271 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:14.318+0000 7f098aa00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f098c1086c0 0x7f098c19dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:15.271 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:14.318+0000 7f098aa00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f098c1086c0 0x7f098c19dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:33334/0 (socket says 172.21.15.39:33334) 2024-03-18T15:58:15.271 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:14.318+0000 7f098aa00700 1 -- 172.21.15.39:0/2714581084 learned_addr learned my addr 172.21.15.39:0/2714581084 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T15:58:15.271 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:14.318+0000 7f098aa00700 1 -- 172.21.15.39:0/2714581084 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f098c19e250 con 0x7f098c1086c0 2024-03-18T15:58:15.271 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:14.319+0000 7f098aa00700 1 --2- 172.21.15.39:0/2714581084 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f098c1086c0 0x7f098c19dd10 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f097c011b60 tx=0x7f097c011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:15.271 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:14.319+0000 7f0990a00700 1 -- 172.21.15.39:0/2714581084 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f097c003b20 con 0x7f098c1086c0 2024-03-18T15:58:15.271 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:14.319+0000 7f0990a00700 1 -- 172.21.15.39:0/2714581084 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f097c003cc0 con 0x7f098c1086c0 2024-03-18T15:58:15.271 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:14.319+0000 7f0990a00700 1 -- 172.21.15.39:0/2714581084 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f097c024840 con 0x7f098c1086c0 2024-03-18T15:58:15.271 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:14.319+0000 7f0991e00700 1 -- 172.21.15.39:0/2714581084 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f098c19e4e0 con 0x7f098c1086c0 2024-03-18T15:58:15.271 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:14.319+0000 7f0991e00700 1 -- 172.21.15.39:0/2714581084 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f098c1a11c0 con 0x7f098c1086c0 2024-03-18T15:58:15.272 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:14.320+0000 7f0990a00700 1 -- 172.21.15.39:0/2714581084 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f097c022020 con 0x7f098c1086c0 2024-03-18T15:58:15.272 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:14.321+0000 7f0990a00700 1 --2- 172.21.15.39:0/2714581084 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7f0970038f70 0x7f097003b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:15.272 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:14.321+0000 7f0991e00700 1 -- 172.21.15.39:0/2714581084 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f098c063900 con 0x7f098c1086c0 2024-03-18T15:58:15.272 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:14.321+0000 7f0990a00700 1 -- 172.21.15.39:0/2714581084 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f097c011e00 con 0x7f098c1086c0 2024-03-18T15:58:15.272 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:14.321+0000 7f098a000700 1 --2- 172.21.15.39:0/2714581084 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7f0970038f70 0x7f097003b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:15.272 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:14.322+0000 7f098a000700 1 --2- 172.21.15.39:0/2714581084 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7f0970038f70 0x7f097003b430 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0978006fd0 tx=0x7f0978006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:15.272 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:14.327+0000 7f0990a00700 1 -- 172.21.15.39:0/2714581084 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f097c00b3e0 con 0x7f098c1086c0 2024-03-18T15:58:15.274 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:14.555+0000 7f0991e00700 1 -- 172.21.15.39:0/2714581084 --> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7f098c103260 con 0x7f0970038f70 2024-03-18T15:58:15.274 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:14.556+0000 7f0990a00700 1 -- 172.21.15.39:0/2714581084 <== mgr.14120 v2:172.21.15.39:6800/1810238546 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7f098c103260 con 0x7f0970038f70 2024-03-18T15:58:15.274 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:14.560+0000 7f0991e00700 1 -- 172.21.15.39:0/2714581084 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7f0970038f70 msgr2=0x7f097003b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:15.274 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:14.560+0000 7f0991e00700 1 --2- 172.21.15.39:0/2714581084 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7f0970038f70 0x7f097003b430 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0978006fd0 tx=0x7f0978006e40 comp rx=0 tx=0).stop 2024-03-18T15:58:15.274 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:14.560+0000 7f0991e00700 1 -- 172.21.15.39:0/2714581084 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f098c1086c0 msgr2=0x7f098c19dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:15.274 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:14.560+0000 7f0991e00700 1 --2- 172.21.15.39:0/2714581084 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f098c1086c0 0x7f098c19dd10 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f097c011b60 tx=0x7f097c011c40 comp rx=0 tx=0).stop 2024-03-18T15:58:15.274 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:14.560+0000 7f0991e00700 1 -- 172.21.15.39:0/2714581084 shutdown_connections 2024-03-18T15:58:15.274 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:14.560+0000 7f0991e00700 1 --2- 172.21.15.39:0/2714581084 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7f0970038f70 0x7f097003b430 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:15.274 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:14.560+0000 7f0991e00700 1 --2- 172.21.15.39:0/2714581084 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f098c1086c0 0x7f098c19dd10 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:15.275 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:14.560+0000 7f0991e00700 1 -- 172.21.15.39:0/2714581084 >> 172.21.15.39:0/2714581084 conn(0x7f098c1020d0 msgr2=0x7f098c102b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:15.275 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:14.560+0000 7f0991e00700 1 -- 172.21.15.39:0/2714581084 shutdown_connections 2024-03-18T15:58:15.275 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:14.561+0000 7f0991e00700 1 -- 172.21.15.39:0/2714581084 wait complete. 2024-03-18T15:58:15.275 INFO:teuthology.orchestra.run.smithi039.stdout:Generating ssh key... 2024-03-18T15:58:15.554 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:15 smithi039 ceph-mon[31721]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:58:16.589 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:16 smithi039 ceph-mon[31721]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:58:16.589 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:16 smithi039 ceph-mon[31721]: Generating ssh key... 2024-03-18T15:58:16.590 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:58:16 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: Generating public/private rsa key pair. 2024-03-18T15:58:16.590 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:58:16 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: Your identification has been saved in /tmp/tmp3vt9qrpo/key. 2024-03-18T15:58:16.590 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:58:16 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: Your public key has been saved in /tmp/tmp3vt9qrpo/key.pub. 2024-03-18T15:58:16.590 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:58:16 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: The key fingerprint is: 2024-03-18T15:58:16.590 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:58:16 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: SHA256:QZYYk3CGCsVgD3CQp+tYCdS5wAstECENauq1s0BGak0 ceph-1892d8fc-e540-11ee-95c9-87774f69a715 2024-03-18T15:58:16.590 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:58:16 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: The key's randomart image is: 2024-03-18T15:58:16.590 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:58:16 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: +---[RSA 3072]----+ 2024-03-18T15:58:16.590 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:58:16 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: |&#+ oo=+o. | 2024-03-18T15:58:16.590 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:58:16 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: |O=Bo.oo+. | 2024-03-18T15:58:16.591 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:58:16 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: |+BoE. . | 2024-03-18T15:58:16.591 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:58:16 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: |B.+. . | 2024-03-18T15:58:16.591 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:58:16 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: |o* + S | 2024-03-18T15:58:16.591 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:58:16 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: |* + . | 2024-03-18T15:58:16.591 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:58:16 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: |o+ o | 2024-03-18T15:58:16.591 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:58:16 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: |... o | 2024-03-18T15:58:16.591 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:58:16 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: | . | 2024-03-18T15:58:16.591 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:58:16 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: +----[SHA256]-----+ 2024-03-18T15:58:17.034 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:15.868+0000 7f8a6dc00700 1 Processor -- start 2024-03-18T15:58:17.035 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:15.868+0000 7f8a6dc00700 1 -- start start 2024-03-18T15:58:17.035 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:15.869+0000 7f8a6dc00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a68109db0 0x7f8a6810a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:17.035 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:15.869+0000 7f8a6dc00700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f8a6810a7a0 con 0x7f8a68109db0 2024-03-18T15:58:17.036 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:15.869+0000 7f8a66a00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a68109db0 0x7f8a6810a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:17.037 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:15.870+0000 7f8a66a00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a68109db0 0x7f8a6810a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:33346/0 (socket says 172.21.15.39:33346) 2024-03-18T15:58:17.037 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:15.870+0000 7f8a66a00700 1 -- 172.21.15.39:0/1337597456 learned_addr learned my addr 172.21.15.39:0/1337597456 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T15:58:17.037 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:15.870+0000 7f8a66a00700 1 -- 172.21.15.39:0/1337597456 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a6810afb0 con 0x7f8a68109db0 2024-03-18T15:58:17.037 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:15.870+0000 7f8a66a00700 1 --2- 172.21.15.39:0/1337597456 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a68109db0 0x7f8a6810a1d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f8a580098a0 tx=0x7f8a58009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f900e0bb6d44ba40 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:17.037 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:15.871+0000 7f8a65600700 1 -- 172.21.15.39:0/1337597456 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8a58004160 con 0x7f8a68109db0 2024-03-18T15:58:17.037 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:15.871+0000 7f8a65600700 1 -- 172.21.15.39:0/1337597456 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f8a58004300 con 0x7f8a68109db0 2024-03-18T15:58:17.037 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:15.871+0000 7f8a65600700 1 -- 172.21.15.39:0/1337597456 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8a58011b50 con 0x7f8a68109db0 2024-03-18T15:58:17.037 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:15.871+0000 7f8a6dc00700 1 -- 172.21.15.39:0/1337597456 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a68109db0 msgr2=0x7f8a6810a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:17.037 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:15.872+0000 7f8a6dc00700 1 --2- 172.21.15.39:0/1337597456 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a68109db0 0x7f8a6810a1d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f8a580098a0 tx=0x7f8a58009bb0 comp rx=0 tx=0).stop 2024-03-18T15:58:17.037 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:15.872+0000 7f8a6dc00700 1 -- 172.21.15.39:0/1337597456 shutdown_connections 2024-03-18T15:58:17.042 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:15.872+0000 7f8a6dc00700 1 --2- 172.21.15.39:0/1337597456 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a68109db0 0x7f8a6810a1d0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:17.042 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:15.872+0000 7f8a6dc00700 1 -- 172.21.15.39:0/1337597456 >> 172.21.15.39:0/1337597456 conn(0x7f8a68105510 msgr2=0x7f8a68107900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:17.042 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:15.872+0000 7f8a6dc00700 1 -- 172.21.15.39:0/1337597456 shutdown_connections 2024-03-18T15:58:17.042 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:15.872+0000 7f8a6dc00700 1 -- 172.21.15.39:0/1337597456 wait complete. 2024-03-18T15:58:17.042 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:15.873+0000 7f8a6dc00700 1 Processor -- start 2024-03-18T15:58:17.042 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:15.873+0000 7f8a6dc00700 1 -- start start 2024-03-18T15:58:17.042 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:15.874+0000 7f8a6dc00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a68109db0 0x7f8a6819dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:17.042 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:15.874+0000 7f8a6dc00700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f8a6810e360 con 0x7f8a68109db0 2024-03-18T15:58:17.042 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:15.874+0000 7f8a66a00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a68109db0 0x7f8a6819dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:17.042 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:15.874+0000 7f8a66a00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a68109db0 0x7f8a6819dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:33352/0 (socket says 172.21.15.39:33352) 2024-03-18T15:58:17.043 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:15.874+0000 7f8a66a00700 1 -- 172.21.15.39:0/2045545736 learned_addr learned my addr 172.21.15.39:0/2045545736 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T15:58:17.043 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:15.875+0000 7f8a66a00700 1 -- 172.21.15.39:0/2045545736 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a6819e250 con 0x7f8a68109db0 2024-03-18T15:58:17.043 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:15.875+0000 7f8a66a00700 1 --2- 172.21.15.39:0/2045545736 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a68109db0 0x7f8a6819dd10 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f8a58000c00 tx=0x7f8a58011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:17.043 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:15.875+0000 7f8a5f400700 1 -- 172.21.15.39:0/2045545736 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8a58003b20 con 0x7f8a68109db0 2024-03-18T15:58:17.043 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:15.875+0000 7f8a5f400700 1 -- 172.21.15.39:0/2045545736 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f8a58003cc0 con 0x7f8a68109db0 2024-03-18T15:58:17.043 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:15.875+0000 7f8a6dc00700 1 -- 172.21.15.39:0/2045545736 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a6819e4e0 con 0x7f8a68109db0 2024-03-18T15:58:17.043 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:15.875+0000 7f8a5f400700 1 -- 172.21.15.39:0/2045545736 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8a58024910 con 0x7f8a68109db0 2024-03-18T15:58:17.043 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:15.876+0000 7f8a6dc00700 1 -- 172.21.15.39:0/2045545736 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a681a11c0 con 0x7f8a68109db0 2024-03-18T15:58:17.043 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:15.877+0000 7f8a5f400700 1 -- 172.21.15.39:0/2045545736 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f8a58022020 con 0x7f8a68109db0 2024-03-18T15:58:17.043 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:15.877+0000 7f8a6dc00700 1 -- 172.21.15.39:0/2045545736 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a68063900 con 0x7f8a68109db0 2024-03-18T15:58:17.043 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:15.877+0000 7f8a5f400700 1 --2- 172.21.15.39:0/2045545736 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7f8a4c038f70 0x7f8a4c03b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:17.043 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:15.877+0000 7f8a5f400700 1 -- 172.21.15.39:0/2045545736 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f8a58011e00 con 0x7f8a68109db0 2024-03-18T15:58:17.043 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:15.878+0000 7f8a66000700 1 --2- 172.21.15.39:0/2045545736 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7f8a4c038f70 0x7f8a4c03b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:17.044 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:15.878+0000 7f8a66000700 1 --2- 172.21.15.39:0/2045545736 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7f8a4c038f70 0x7f8a4c03b430 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f8a54006fd0 tx=0x7f8a54006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:17.044 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:15.883+0000 7f8a5f400700 1 -- 172.21.15.39:0/2045545736 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8a58016260 con 0x7f8a68109db0 2024-03-18T15:58:17.044 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:16.109+0000 7f8a6dc00700 1 -- 172.21.15.39:0/2045545736 --> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7f8a68107750 con 0x7f8a4c038f70 2024-03-18T15:58:17.044 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:16.440+0000 7f8a5f400700 1 -- 172.21.15.39:0/2045545736 <== mgr.14120 v2:172.21.15.39:6800/1810238546 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f8a68107750 con 0x7f8a4c038f70 2024-03-18T15:58:17.044 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:16.443+0000 7f8a6dc00700 1 -- 172.21.15.39:0/2045545736 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7f8a4c038f70 msgr2=0x7f8a4c03b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:17.044 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:16.443+0000 7f8a6dc00700 1 --2- 172.21.15.39:0/2045545736 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7f8a4c038f70 0x7f8a4c03b430 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f8a54006fd0 tx=0x7f8a54006e40 comp rx=0 tx=0).stop 2024-03-18T15:58:17.044 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:16.443+0000 7f8a6dc00700 1 -- 172.21.15.39:0/2045545736 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a68109db0 msgr2=0x7f8a6819dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:17.044 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:16.443+0000 7f8a6dc00700 1 --2- 172.21.15.39:0/2045545736 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a68109db0 0x7f8a6819dd10 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f8a58000c00 tx=0x7f8a58011c10 comp rx=0 tx=0).stop 2024-03-18T15:58:17.044 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:16.443+0000 7f8a6dc00700 1 -- 172.21.15.39:0/2045545736 shutdown_connections 2024-03-18T15:58:17.044 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:16.443+0000 7f8a6dc00700 1 --2- 172.21.15.39:0/2045545736 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7f8a4c038f70 0x7f8a4c03b430 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:17.044 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:16.443+0000 7f8a6dc00700 1 --2- 172.21.15.39:0/2045545736 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a68109db0 0x7f8a6819dd10 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:17.044 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:16.443+0000 7f8a6dc00700 1 -- 172.21.15.39:0/2045545736 >> 172.21.15.39:0/2045545736 conn(0x7f8a68105510 msgr2=0x7f8a68107040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:17.044 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:16.443+0000 7f8a6dc00700 1 -- 172.21.15.39:0/2045545736 shutdown_connections 2024-03-18T15:58:17.045 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:16.443+0000 7f8a6dc00700 1 -- 172.21.15.39:0/2045545736 wait complete. 2024-03-18T15:58:17.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:17 smithi039 ceph-mon[31721]: from='mgr.14120 172.21.15.39:0/2752076370' entity='mgr.x' 2024-03-18T15:58:17.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:17 smithi039 ceph-mon[31721]: from='mgr.14120 172.21.15.39:0/2752076370' entity='mgr.x' 2024-03-18T15:58:18.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:18 smithi039 ceph-mon[31721]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:58:18.638 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJK+CTQQ3Wg71D32RPxs4NoPmY8FpotvbTS2bAm99J85Gi9xX+K5lXncSe03PD7evIDOV7mIhbq6TzF2WFEbCoxGehisf1+fprn5BZrquL3L17pdY59T6aVxSnQjYOUNmq3kPNHBg6OjyT0ePvWVtyjyLdbG4ImaUivgMwv2w1L2wUSyGNB2/lBhocZjLxbzQy3atQHfT/llEt2G9TffPCk+7wAM3K+IkHwhwwaoh9qj9nLHTLroOmIe9uKZxwa4ZTshKJBLj89l2+Cm2mhmNsBB7i0AAYSP5QVBIdzjN9Sw/66Kxrg3aZobJt9v6lD0MndegLgPcuCJT5cwbWtqc/bWksPGtUdfqtIzcbIHTUYDqJcw74ZYAeOV2vZeGdMDi2dE++qE2d8QekhdFsUSX8ABzsaLljIZhy1H2eAU9dMTAvyvdAKCDYfajBXXJXXCdUy99c3rUmaf5lx51mwz4sccliZLFkqvsRkBF17SrewDxLSLWJ0k7Hq8eydXPMvKs= ceph-1892d8fc-e540-11ee-95c9-87774f69a715 2024-03-18T15:58:18.638 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:17.675+0000 7fe121800700 1 Processor -- start 2024-03-18T15:58:18.638 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:17.675+0000 7fe121800700 1 -- start start 2024-03-18T15:58:18.638 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:17.675+0000 7fe121800700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe11c10ec60 0x7fe11c111050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:18.638 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:17.675+0000 7fe121800700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fe11c082470 con 0x7fe11c10ec60 2024-03-18T15:58:18.638 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:17.676+0000 7fe11aa00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe11c10ec60 0x7fe11c111050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:18.638 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:17.676+0000 7fe11aa00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe11c10ec60 0x7fe11c111050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:33360/0 (socket says 172.21.15.39:33360) 2024-03-18T15:58:18.638 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:17.676+0000 7fe11aa00700 1 -- 172.21.15.39:0/292637104 learned_addr learned my addr 172.21.15.39:0/292637104 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T15:58:18.638 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:17.676+0000 7fe11aa00700 1 -- 172.21.15.39:0/292637104 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe11c111590 con 0x7fe11c10ec60 2024-03-18T15:58:18.639 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:17.677+0000 7fe11aa00700 1 --2- 172.21.15.39:0/292637104 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe11c10ec60 0x7fe11c111050 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe10c0098a0 tx=0x7fe10c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3f61434117bcc87 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:18.639 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:17.677+0000 7fe11a000700 1 -- 172.21.15.39:0/292637104 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe10c004160 con 0x7fe11c10ec60 2024-03-18T15:58:18.639 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:17.677+0000 7fe11a000700 1 -- 172.21.15.39:0/292637104 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe10c004300 con 0x7fe11c10ec60 2024-03-18T15:58:18.639 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:17.677+0000 7fe11a000700 1 -- 172.21.15.39:0/292637104 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe10c011b50 con 0x7fe11c10ec60 2024-03-18T15:58:18.639 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:17.677+0000 7fe121800700 1 -- 172.21.15.39:0/292637104 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe11c10ec60 msgr2=0x7fe11c111050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:18.639 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:17.677+0000 7fe121800700 1 --2- 172.21.15.39:0/292637104 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe11c10ec60 0x7fe11c111050 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe10c0098a0 tx=0x7fe10c009bb0 comp rx=0 tx=0).stop 2024-03-18T15:58:18.639 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:17.678+0000 7fe121800700 1 -- 172.21.15.39:0/292637104 shutdown_connections 2024-03-18T15:58:18.639 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:17.678+0000 7fe121800700 1 --2- 172.21.15.39:0/292637104 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe11c10ec60 0x7fe11c111050 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:18.639 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:17.678+0000 7fe121800700 1 -- 172.21.15.39:0/292637104 >> 172.21.15.39:0/292637104 conn(0x7fe11c07d210 msgr2=0x7fe11c07f630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:18.639 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:17.678+0000 7fe121800700 1 -- 172.21.15.39:0/292637104 shutdown_connections 2024-03-18T15:58:18.639 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:17.678+0000 7fe121800700 1 -- 172.21.15.39:0/292637104 wait complete. 2024-03-18T15:58:18.639 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:17.679+0000 7fe121800700 1 Processor -- start 2024-03-18T15:58:18.640 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:17.679+0000 7fe121800700 1 -- start start 2024-03-18T15:58:18.640 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:17.679+0000 7fe121800700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe11c10ec60 0x7fe11c1a2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:18.640 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:17.679+0000 7fe121800700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fe11c1125a0 con 0x7fe11c10ec60 2024-03-18T15:58:18.640 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:17.679+0000 7fe11aa00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe11c10ec60 0x7fe11c1a2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:18.640 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:17.679+0000 7fe11aa00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe11c10ec60 0x7fe11c1a2160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:33374/0 (socket says 172.21.15.39:33374) 2024-03-18T15:58:18.640 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:17.679+0000 7fe11aa00700 1 -- 172.21.15.39:0/3169932483 learned_addr learned my addr 172.21.15.39:0/3169932483 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T15:58:18.640 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:17.680+0000 7fe11aa00700 1 -- 172.21.15.39:0/3169932483 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe11c1a26a0 con 0x7fe11c10ec60 2024-03-18T15:58:18.640 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:17.680+0000 7fe11aa00700 1 --2- 172.21.15.39:0/3169932483 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe11c10ec60 0x7fe11c1a2160 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe10c011b60 tx=0x7fe10c011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:18.640 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:17.680+0000 7fe113e00700 1 -- 172.21.15.39:0/3169932483 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe10c003970 con 0x7fe11c10ec60 2024-03-18T15:58:18.640 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:17.680+0000 7fe113e00700 1 -- 172.21.15.39:0/3169932483 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe10c003b10 con 0x7fe11c10ec60 2024-03-18T15:58:18.640 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:17.680+0000 7fe113e00700 1 -- 172.21.15.39:0/3169932483 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe10c024610 con 0x7fe11c10ec60 2024-03-18T15:58:18.640 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:17.680+0000 7fe121800700 1 -- 172.21.15.39:0/3169932483 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe11c1a2930 con 0x7fe11c10ec60 2024-03-18T15:58:18.640 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:17.681+0000 7fe121800700 1 -- 172.21.15.39:0/3169932483 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe11c1a5620 con 0x7fe11c10ec60 2024-03-18T15:58:18.640 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:17.682+0000 7fe121800700 1 -- 172.21.15.39:0/3169932483 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe11c063900 con 0x7fe11c10ec60 2024-03-18T15:58:18.641 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:17.682+0000 7fe113e00700 1 -- 172.21.15.39:0/3169932483 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7fe10c01a9f0 con 0x7fe11c10ec60 2024-03-18T15:58:18.641 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:17.682+0000 7fe113e00700 1 --2- 172.21.15.39:0/3169932483 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7fe100038f70 0x7fe10003b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:18.641 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:17.683+0000 7fe113e00700 1 -- 172.21.15.39:0/3169932483 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fe10c011e00 con 0x7fe11c10ec60 2024-03-18T15:58:18.641 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:17.683+0000 7fe112000700 1 --2- 172.21.15.39:0/3169932483 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7fe100038f70 0x7fe10003b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:18.641 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:17.684+0000 7fe112000700 1 --2- 172.21.15.39:0/3169932483 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7fe100038f70 0x7fe10003b430 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe10400ad30 tx=0x7fe1040093f0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:18.641 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:17.688+0000 7fe113e00700 1 -- 172.21.15.39:0/3169932483 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe10c00b3e0 con 0x7fe11c10ec60 2024-03-18T15:58:18.641 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:17.912+0000 7fe121800700 1 -- 172.21.15.39:0/3169932483 --> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7fe11c07e450 con 0x7fe100038f70 2024-03-18T15:58:18.641 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:17.914+0000 7fe113e00700 1 -- 172.21.15.39:0/3169932483 <== mgr.14120 v2:172.21.15.39:6800/1810238546 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7fe11c07e450 con 0x7fe100038f70 2024-03-18T15:58:18.641 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:17.917+0000 7fe121800700 1 -- 172.21.15.39:0/3169932483 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7fe100038f70 msgr2=0x7fe10003b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:18.641 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:17.917+0000 7fe121800700 1 --2- 172.21.15.39:0/3169932483 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7fe100038f70 0x7fe10003b430 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe10400ad30 tx=0x7fe1040093f0 comp rx=0 tx=0).stop 2024-03-18T15:58:18.641 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:17.917+0000 7fe121800700 1 -- 172.21.15.39:0/3169932483 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe11c10ec60 msgr2=0x7fe11c1a2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:18.641 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:17.917+0000 7fe121800700 1 --2- 172.21.15.39:0/3169932483 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe11c10ec60 0x7fe11c1a2160 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe10c011b60 tx=0x7fe10c011c40 comp rx=0 tx=0).stop 2024-03-18T15:58:18.641 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:17.918+0000 7fe121800700 1 -- 172.21.15.39:0/3169932483 shutdown_connections 2024-03-18T15:58:18.641 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:17.918+0000 7fe121800700 1 --2- 172.21.15.39:0/3169932483 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7fe100038f70 0x7fe10003b430 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:18.642 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:17.918+0000 7fe121800700 1 --2- 172.21.15.39:0/3169932483 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe11c10ec60 0x7fe11c1a2160 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:18.642 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:17.918+0000 7fe121800700 1 -- 172.21.15.39:0/3169932483 >> 172.21.15.39:0/3169932483 conn(0x7fe11c07d210 msgr2=0x7fe11c07dd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:18.642 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:17.918+0000 7fe121800700 1 -- 172.21.15.39:0/3169932483 shutdown_connections 2024-03-18T15:58:18.642 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:17.918+0000 7fe121800700 1 -- 172.21.15.39:0/3169932483 wait complete. 2024-03-18T15:58:18.642 INFO:teuthology.orchestra.run.smithi039.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-18T15:58:18.642 INFO:teuthology.orchestra.run.smithi039.stdout:Adding key to root@localhost authorized_keys... 2024-03-18T15:58:18.642 INFO:teuthology.orchestra.run.smithi039.stdout:Adding host smithi039... 2024-03-18T15:58:20.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:20 smithi039 ceph-mon[31721]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi039", "addr": "172.21.15.39", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:58:21.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:21 smithi039 ceph-mon[31721]: Deploying cephadm binary to smithi039 2024-03-18T15:58:26.851 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout Added host 'smithi039' with addr '172.21.15.39' 2024-03-18T15:58:26.851 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:19.207+0000 7fbdf4e00700 1 Processor -- start 2024-03-18T15:58:26.852 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:19.207+0000 7fbdf4e00700 1 -- start start 2024-03-18T15:58:26.852 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:19.208+0000 7fbdf4e00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbdf0109c30 0x7fbdf010a050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:26.852 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:19.208+0000 7fbdf4e00700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fbdf010a620 con 0x7fbdf0109c30 2024-03-18T15:58:26.852 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:19.208+0000 7fbdef400700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbdf0109c30 0x7fbdf010a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:26.852 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:19.208+0000 7fbdef400700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbdf0109c30 0x7fbdf010a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:52248/0 (socket says 172.21.15.39:52248) 2024-03-18T15:58:26.852 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:19.208+0000 7fbdef400700 1 -- 172.21.15.39:0/1450865060 learned_addr learned my addr 172.21.15.39:0/1450865060 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T15:58:26.852 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:19.208+0000 7fbdef400700 1 -- 172.21.15.39:0/1450865060 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdf010ae30 con 0x7fbdf0109c30 2024-03-18T15:58:26.852 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:19.209+0000 7fbdef400700 1 --2- 172.21.15.39:0/1450865060 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbdf0109c30 0x7fbdf010a050 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fbde00098a0 tx=0x7fbde0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43652e34c1e58467 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:26.852 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:19.209+0000 7fbdee000700 1 -- 172.21.15.39:0/1450865060 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbde0004160 con 0x7fbdf0109c30 2024-03-18T15:58:26.852 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:19.209+0000 7fbdee000700 1 -- 172.21.15.39:0/1450865060 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fbde0004300 con 0x7fbdf0109c30 2024-03-18T15:58:26.852 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:19.209+0000 7fbdee000700 1 -- 172.21.15.39:0/1450865060 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbde0011b50 con 0x7fbdf0109c30 2024-03-18T15:58:26.852 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:19.210+0000 7fbdf4e00700 1 -- 172.21.15.39:0/1450865060 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbdf0109c30 msgr2=0x7fbdf010a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:26.852 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:19.210+0000 7fbdf4e00700 1 --2- 172.21.15.39:0/1450865060 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbdf0109c30 0x7fbdf010a050 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fbde00098a0 tx=0x7fbde0009bb0 comp rx=0 tx=0).stop 2024-03-18T15:58:26.852 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:19.210+0000 7fbdf4e00700 1 -- 172.21.15.39:0/1450865060 shutdown_connections 2024-03-18T15:58:26.853 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:19.210+0000 7fbdf4e00700 1 --2- 172.21.15.39:0/1450865060 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbdf0109c30 0x7fbdf010a050 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:26.853 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:19.210+0000 7fbdf4e00700 1 -- 172.21.15.39:0/1450865060 >> 172.21.15.39:0/1450865060 conn(0x7fbdf007d000 msgr2=0x7fbdf007d430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:26.853 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:19.210+0000 7fbdf4e00700 1 -- 172.21.15.39:0/1450865060 shutdown_connections 2024-03-18T15:58:26.853 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:19.210+0000 7fbdf4e00700 1 -- 172.21.15.39:0/1450865060 wait complete. 2024-03-18T15:58:26.853 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:19.211+0000 7fbdf4e00700 1 Processor -- start 2024-03-18T15:58:26.853 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:19.211+0000 7fbdf4e00700 1 -- start start 2024-03-18T15:58:26.853 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:19.212+0000 7fbdf4e00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbdf0109c30 0x7fbdf019dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:26.853 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:19.212+0000 7fbdf4e00700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fbdf010e1e0 con 0x7fbdf0109c30 2024-03-18T15:58:26.853 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:19.212+0000 7fbdef400700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbdf0109c30 0x7fbdf019dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:26.853 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:19.212+0000 7fbdef400700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbdf0109c30 0x7fbdf019dbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:52260/0 (socket says 172.21.15.39:52260) 2024-03-18T15:58:26.853 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:19.212+0000 7fbdef400700 1 -- 172.21.15.39:0/1265263612 learned_addr learned my addr 172.21.15.39:0/1265263612 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T15:58:26.853 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:19.212+0000 7fbdef400700 1 -- 172.21.15.39:0/1265263612 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdf019e120 con 0x7fbdf0109c30 2024-03-18T15:58:26.853 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:19.213+0000 7fbdef400700 1 --2- 172.21.15.39:0/1265263612 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbdf0109c30 0x7fbdf019dbe0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fbde0000c00 tx=0x7fbde0011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:26.853 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:19.213+0000 7fbde7e00700 1 -- 172.21.15.39:0/1265263612 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbde0003b20 con 0x7fbdf0109c30 2024-03-18T15:58:26.854 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:19.213+0000 7fbde7e00700 1 -- 172.21.15.39:0/1265263612 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fbde0003cc0 con 0x7fbdf0109c30 2024-03-18T15:58:26.854 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:19.213+0000 7fbde7e00700 1 -- 172.21.15.39:0/1265263612 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbde0024910 con 0x7fbdf0109c30 2024-03-18T15:58:26.854 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:19.213+0000 7fbdf4e00700 1 -- 172.21.15.39:0/1265263612 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbdf019e3b0 con 0x7fbdf0109c30 2024-03-18T15:58:26.854 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:19.213+0000 7fbdf4e00700 1 -- 172.21.15.39:0/1265263612 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbdf01a10a0 con 0x7fbdf0109c30 2024-03-18T15:58:26.854 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:19.214+0000 7fbdf4e00700 1 -- 172.21.15.39:0/1265263612 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbdf0063900 con 0x7fbdf0109c30 2024-03-18T15:58:26.854 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:19.215+0000 7fbde7e00700 1 -- 172.21.15.39:0/1265263612 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7fbde0022020 con 0x7fbdf0109c30 2024-03-18T15:58:26.854 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:19.215+0000 7fbde7e00700 1 --2- 172.21.15.39:0/1265263612 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7fbdd4038f70 0x7fbdd403b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:26.854 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:19.215+0000 7fbde7e00700 1 -- 172.21.15.39:0/1265263612 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fbde0011e00 con 0x7fbdf0109c30 2024-03-18T15:58:26.854 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:19.215+0000 7fbdeea00700 1 --2- 172.21.15.39:0/1265263612 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7fbdd4038f70 0x7fbdd403b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:26.854 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:19.216+0000 7fbdeea00700 1 --2- 172.21.15.39:0/1265263612 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7fbdd4038f70 0x7fbdd403b430 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fbddc006fd0 tx=0x7fbddc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:26.856 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:19.221+0000 7fbde7e00700 1 -- 172.21.15.39:0/1265263612 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fbde004c910 con 0x7fbdf0109c30 2024-03-18T15:58:26.856 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:19.451+0000 7fbdf4e00700 1 -- 172.21.15.39:0/1265263612 --> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi039", "addr": "172.21.15.39", "target": ["mon-mgr", ""]}) -- 0x7fbdf0107bc0 con 0x7fbdd4038f70 2024-03-18T15:58:26.856 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:26.224+0000 7fbde7e00700 1 -- 172.21.15.39:0/1265263612 <== mgr.14120 v2:172.21.15.39:6800/1810238546 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7fbdf0107bc0 con 0x7fbdd4038f70 2024-03-18T15:58:26.856 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:26.228+0000 7fbdf4e00700 1 -- 172.21.15.39:0/1265263612 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7fbdd4038f70 msgr2=0x7fbdd403b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:26.856 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:26.228+0000 7fbdf4e00700 1 --2- 172.21.15.39:0/1265263612 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7fbdd4038f70 0x7fbdd403b430 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fbddc006fd0 tx=0x7fbddc006e40 comp rx=0 tx=0).stop 2024-03-18T15:58:26.856 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:26.228+0000 7fbdf4e00700 1 -- 172.21.15.39:0/1265263612 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbdf0109c30 msgr2=0x7fbdf019dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:26.856 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:26.228+0000 7fbdf4e00700 1 --2- 172.21.15.39:0/1265263612 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbdf0109c30 0x7fbdf019dbe0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fbde0000c00 tx=0x7fbde0011c10 comp rx=0 tx=0).stop 2024-03-18T15:58:26.856 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:26.228+0000 7fbdf4e00700 1 -- 172.21.15.39:0/1265263612 shutdown_connections 2024-03-18T15:58:26.857 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:26.228+0000 7fbdf4e00700 1 --2- 172.21.15.39:0/1265263612 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7fbdd4038f70 0x7fbdd403b430 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:26.857 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:26.228+0000 7fbdf4e00700 1 --2- 172.21.15.39:0/1265263612 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbdf0109c30 0x7fbdf019dbe0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:26.857 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:26.228+0000 7fbdf4e00700 1 -- 172.21.15.39:0/1265263612 >> 172.21.15.39:0/1265263612 conn(0x7fbdf007d000 msgr2=0x7fbdf01074b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:26.857 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:26.229+0000 7fbdf4e00700 1 -- 172.21.15.39:0/1265263612 shutdown_connections 2024-03-18T15:58:26.857 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:26.229+0000 7fbdf4e00700 1 -- 172.21.15.39:0/1265263612 wait complete. 2024-03-18T15:58:26.857 INFO:teuthology.orchestra.run.smithi039.stdout:Deploying unmanaged mon service... 2024-03-18T15:58:27.407 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:27 smithi039 ceph-mon[31721]: from='mgr.14120 172.21.15.39:0/2752076370' entity='mgr.x' 2024-03-18T15:58:27.407 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:27 smithi039 ceph-mon[31721]: Added host smithi039 2024-03-18T15:58:27.407 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:27 smithi039 ceph-mon[31721]: from='mgr.14120 172.21.15.39:0/2752076370' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:58:28.467 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-18T15:58:28.468 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:27.470+0000 7fd98a400700 1 Processor -- start 2024-03-18T15:58:28.468 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:27.470+0000 7fd98a400700 1 -- start start 2024-03-18T15:58:28.468 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:27.471+0000 7fd98a400700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd98407cd40 0x7fd98407b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:28.468 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:27.471+0000 7fd98a400700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fd98407b770 con 0x7fd98407cd40 2024-03-18T15:58:28.468 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:27.471+0000 7fd989000700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd98407cd40 0x7fd98407b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:28.468 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:27.471+0000 7fd989000700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd98407cd40 0x7fd98407b1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:52268/0 (socket says 172.21.15.39:52268) 2024-03-18T15:58:28.468 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:27.471+0000 7fd989000700 1 -- 172.21.15.39:0/2285443930 learned_addr learned my addr 172.21.15.39:0/2285443930 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T15:58:28.468 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:27.472+0000 7fd989000700 1 -- 172.21.15.39:0/2285443930 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd98407b8f0 con 0x7fd98407cd40 2024-03-18T15:58:28.468 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:27.472+0000 7fd989000700 1 --2- 172.21.15.39:0/2285443930 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd98407cd40 0x7fd98407b1a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd9740098a0 tx=0x7fd974009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c9fb636c06a98fa server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:28.469 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:27.473+0000 7fd983e00700 1 -- 172.21.15.39:0/2285443930 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd974004160 con 0x7fd98407cd40 2024-03-18T15:58:28.469 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:27.473+0000 7fd983e00700 1 -- 172.21.15.39:0/2285443930 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fd974004300 con 0x7fd98407cd40 2024-03-18T15:58:28.469 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:27.473+0000 7fd983e00700 1 -- 172.21.15.39:0/2285443930 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd974011b50 con 0x7fd98407cd40 2024-03-18T15:58:28.469 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:27.473+0000 7fd98a400700 1 -- 172.21.15.39:0/2285443930 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd98407cd40 msgr2=0x7fd98407b1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:28.469 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:27.473+0000 7fd98a400700 1 --2- 172.21.15.39:0/2285443930 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd98407cd40 0x7fd98407b1a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd9740098a0 tx=0x7fd974009bb0 comp rx=0 tx=0).stop 2024-03-18T15:58:28.469 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:27.473+0000 7fd98a400700 1 -- 172.21.15.39:0/2285443930 shutdown_connections 2024-03-18T15:58:28.469 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:27.473+0000 7fd98a400700 1 --2- 172.21.15.39:0/2285443930 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd98407cd40 0x7fd98407b1a0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:28.469 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:27.473+0000 7fd98a400700 1 -- 172.21.15.39:0/2285443930 >> 172.21.15.39:0/2285443930 conn(0x7fd984103050 msgr2=0x7fd984105470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:28.469 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:27.474+0000 7fd98a400700 1 -- 172.21.15.39:0/2285443930 shutdown_connections 2024-03-18T15:58:28.469 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:27.474+0000 7fd98a400700 1 -- 172.21.15.39:0/2285443930 wait complete. 2024-03-18T15:58:28.469 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:27.475+0000 7fd98a400700 1 Processor -- start 2024-03-18T15:58:28.469 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:27.475+0000 7fd98a400700 1 -- start start 2024-03-18T15:58:28.470 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:27.475+0000 7fd98a400700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd98407cd40 0x7fd98419dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:28.470 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:27.475+0000 7fd98a400700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fd98410e000 con 0x7fd98407cd40 2024-03-18T15:58:28.470 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:27.475+0000 7fd989000700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd98407cd40 0x7fd98419dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:28.470 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:27.475+0000 7fd989000700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd98407cd40 0x7fd98419dc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:52284/0 (socket says 172.21.15.39:52284) 2024-03-18T15:58:28.470 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:27.475+0000 7fd989000700 1 -- 172.21.15.39:0/3762934954 learned_addr learned my addr 172.21.15.39:0/3762934954 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T15:58:28.470 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:27.476+0000 7fd989000700 1 -- 172.21.15.39:0/3762934954 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd98419e140 con 0x7fd98407cd40 2024-03-18T15:58:28.470 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:27.476+0000 7fd989000700 1 --2- 172.21.15.39:0/3762934954 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd98407cd40 0x7fd98419dc00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd974011850 tx=0x7fd974011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:28.470 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:27.477+0000 7fd982000700 1 -- 172.21.15.39:0/3762934954 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd974003af0 con 0x7fd98407cd40 2024-03-18T15:58:28.470 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:27.477+0000 7fd982000700 1 -- 172.21.15.39:0/3762934954 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fd974020070 con 0x7fd98407cd40 2024-03-18T15:58:28.470 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:27.477+0000 7fd982000700 1 -- 172.21.15.39:0/3762934954 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd9740195a0 con 0x7fd98407cd40 2024-03-18T15:58:28.470 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:27.477+0000 7fd98a400700 1 -- 172.21.15.39:0/3762934954 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd98419e3d0 con 0x7fd98407cd40 2024-03-18T15:58:28.470 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:27.477+0000 7fd98a400700 1 -- 172.21.15.39:0/3762934954 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9841a10c0 con 0x7fd98407cd40 2024-03-18T15:58:28.470 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:27.478+0000 7fd98a400700 1 -- 172.21.15.39:0/3762934954 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd984063900 con 0x7fd98407cd40 2024-03-18T15:58:28.470 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:27.479+0000 7fd982000700 1 -- 172.21.15.39:0/3762934954 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7fd97401a430 con 0x7fd98407cd40 2024-03-18T15:58:28.471 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:27.479+0000 7fd982000700 1 --2- 172.21.15.39:0/3762934954 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7fd968038f70 0x7fd96803b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:28.471 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:27.479+0000 7fd982000700 1 -- 172.21.15.39:0/3762934954 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fd974011e00 con 0x7fd98407cd40 2024-03-18T15:58:28.471 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:27.479+0000 7fd97be00700 1 --2- 172.21.15.39:0/3762934954 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7fd968038f70 0x7fd96803b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:28.471 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:27.480+0000 7fd97be00700 1 --2- 172.21.15.39:0/3762934954 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7fd968038f70 0x7fd96803b430 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fd970006fd0 tx=0x7fd970006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:28.471 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:27.485+0000 7fd982000700 1 -- 172.21.15.39:0/3762934954 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd97401a720 con 0x7fd98407cd40 2024-03-18T15:58:28.471 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:27.728+0000 7fd98a400700 1 -- 172.21.15.39:0/3762934954 --> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7fd9841042c0 con 0x7fd968038f70 2024-03-18T15:58:28.471 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:27.739+0000 7fd982000700 1 -- 172.21.15.39:0/3762934954 <== mgr.14120 v2:172.21.15.39:6800/1810238546 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fd9841042c0 con 0x7fd968038f70 2024-03-18T15:58:28.471 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:27.742+0000 7fd98a400700 1 -- 172.21.15.39:0/3762934954 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7fd968038f70 msgr2=0x7fd96803b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:28.471 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:27.742+0000 7fd98a400700 1 --2- 172.21.15.39:0/3762934954 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7fd968038f70 0x7fd96803b430 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fd970006fd0 tx=0x7fd970006e40 comp rx=0 tx=0).stop 2024-03-18T15:58:28.471 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:27.742+0000 7fd98a400700 1 -- 172.21.15.39:0/3762934954 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd98407cd40 msgr2=0x7fd98419dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:28.471 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:27.742+0000 7fd98a400700 1 --2- 172.21.15.39:0/3762934954 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd98407cd40 0x7fd98419dc00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd974011850 tx=0x7fd974011c10 comp rx=0 tx=0).stop 2024-03-18T15:58:28.471 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:27.742+0000 7fd98a400700 1 -- 172.21.15.39:0/3762934954 shutdown_connections 2024-03-18T15:58:28.471 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:27.742+0000 7fd98a400700 1 --2- 172.21.15.39:0/3762934954 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7fd968038f70 0x7fd96803b430 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:28.472 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:27.742+0000 7fd98a400700 1 --2- 172.21.15.39:0/3762934954 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd98407cd40 0x7fd98419dc00 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:28.472 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:27.742+0000 7fd98a400700 1 -- 172.21.15.39:0/3762934954 >> 172.21.15.39:0/3762934954 conn(0x7fd984103050 msgr2=0x7fd984103bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:28.472 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:27.742+0000 7fd98a400700 1 -- 172.21.15.39:0/3762934954 shutdown_connections 2024-03-18T15:58:28.472 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:27.742+0000 7fd98a400700 1 -- 172.21.15.39:0/3762934954 wait complete. 2024-03-18T15:58:28.472 INFO:teuthology.orchestra.run.smithi039.stdout:Deploying unmanaged mgr service... 2024-03-18T15:58:29.055 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:28 smithi039 ceph-mon[31721]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:58:29.055 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:28 smithi039 ceph-mon[31721]: Saving service mon spec with placement count:5 2024-03-18T15:58:29.055 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:28 smithi039 ceph-mon[31721]: from='mgr.14120 172.21.15.39:0/2752076370' entity='mgr.x' 2024-03-18T15:58:30.490 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-18T15:58:30.490 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:29.108+0000 7fd080200700 1 Processor -- start 2024-03-18T15:58:30.490 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:29.109+0000 7fd080200700 1 -- start start 2024-03-18T15:58:30.491 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:29.109+0000 7fd080200700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd07810ec60 0x7fd078111050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:30.491 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:29.109+0000 7fd080200700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fd078082470 con 0x7fd07810ec60 2024-03-18T15:58:30.491 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:29.109+0000 7fd07d800700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd07810ec60 0x7fd078111050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:30.491 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:29.110+0000 7fd07d800700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd07810ec60 0x7fd078111050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:40938/0 (socket says 172.21.15.39:40938) 2024-03-18T15:58:30.491 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:29.110+0000 7fd07d800700 1 -- 172.21.15.39:0/1090809046 learned_addr learned my addr 172.21.15.39:0/1090809046 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T15:58:30.492 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:29.110+0000 7fd07d800700 1 -- 172.21.15.39:0/1090809046 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd078111590 con 0x7fd07810ec60 2024-03-18T15:58:30.494 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:29.110+0000 7fd07d800700 1 --2- 172.21.15.39:0/1090809046 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd07810ec60 0x7fd078111050 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd06c0098a0 tx=0x7fd06c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2da431a4b6de392a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:30.494 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:29.111+0000 7fd077e00700 1 -- 172.21.15.39:0/1090809046 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd06c004160 con 0x7fd07810ec60 2024-03-18T15:58:30.494 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:29.111+0000 7fd077e00700 1 -- 172.21.15.39:0/1090809046 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fd06c004300 con 0x7fd07810ec60 2024-03-18T15:58:30.494 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:29.111+0000 7fd077e00700 1 -- 172.21.15.39:0/1090809046 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd06c011b50 con 0x7fd07810ec60 2024-03-18T15:58:30.494 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:29.111+0000 7fd080200700 1 -- 172.21.15.39:0/1090809046 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd07810ec60 msgr2=0x7fd078111050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:30.494 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:29.111+0000 7fd080200700 1 --2- 172.21.15.39:0/1090809046 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd07810ec60 0x7fd078111050 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd06c0098a0 tx=0x7fd06c009bb0 comp rx=0 tx=0).stop 2024-03-18T15:58:30.494 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:29.111+0000 7fd080200700 1 -- 172.21.15.39:0/1090809046 shutdown_connections 2024-03-18T15:58:30.494 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:29.111+0000 7fd080200700 1 --2- 172.21.15.39:0/1090809046 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd07810ec60 0x7fd078111050 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:30.494 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:29.111+0000 7fd080200700 1 -- 172.21.15.39:0/1090809046 >> 172.21.15.39:0/1090809046 conn(0x7fd07807d210 msgr2=0x7fd07807f630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:30.494 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:29.112+0000 7fd080200700 1 -- 172.21.15.39:0/1090809046 shutdown_connections 2024-03-18T15:58:30.494 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:29.112+0000 7fd080200700 1 -- 172.21.15.39:0/1090809046 wait complete. 2024-03-18T15:58:30.495 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:29.113+0000 7fd080200700 1 Processor -- start 2024-03-18T15:58:30.495 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:29.113+0000 7fd080200700 1 -- start start 2024-03-18T15:58:30.495 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:29.113+0000 7fd080200700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd07810ec60 0x7fd0781a2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:30.495 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:29.113+0000 7fd080200700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fd0781125a0 con 0x7fd07810ec60 2024-03-18T15:58:30.495 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:29.114+0000 7fd07d800700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd07810ec60 0x7fd0781a2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:30.495 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:29.114+0000 7fd07d800700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd07810ec60 0x7fd0781a2140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:40946/0 (socket says 172.21.15.39:40946) 2024-03-18T15:58:30.495 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:29.114+0000 7fd07d800700 1 -- 172.21.15.39:0/3936804897 learned_addr learned my addr 172.21.15.39:0/3936804897 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T15:58:30.498 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:29.114+0000 7fd07d800700 1 -- 172.21.15.39:0/3936804897 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0781a2680 con 0x7fd07810ec60 2024-03-18T15:58:30.498 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:29.114+0000 7fd07d800700 1 --2- 172.21.15.39:0/3936804897 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd07810ec60 0x7fd0781a2140 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd06c011b60 tx=0x7fd06c011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:30.498 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:29.115+0000 7fd076000700 1 -- 172.21.15.39:0/3936804897 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd06c003970 con 0x7fd07810ec60 2024-03-18T15:58:30.498 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:29.115+0000 7fd076000700 1 -- 172.21.15.39:0/3936804897 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fd06c003b10 con 0x7fd07810ec60 2024-03-18T15:58:30.498 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:29.115+0000 7fd076000700 1 -- 172.21.15.39:0/3936804897 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd06c024610 con 0x7fd07810ec60 2024-03-18T15:58:30.498 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:29.115+0000 7fd080200700 1 -- 172.21.15.39:0/3936804897 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0781a2910 con 0x7fd07810ec60 2024-03-18T15:58:30.498 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:29.115+0000 7fd080200700 1 -- 172.21.15.39:0/3936804897 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0781a5600 con 0x7fd07810ec60 2024-03-18T15:58:30.498 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:29.116+0000 7fd076000700 1 -- 172.21.15.39:0/3936804897 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7fd06c01a9f0 con 0x7fd07810ec60 2024-03-18T15:58:30.498 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:29.116+0000 7fd080200700 1 -- 172.21.15.39:0/3936804897 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd078063900 con 0x7fd07810ec60 2024-03-18T15:58:30.499 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:29.117+0000 7fd076000700 1 --2- 172.21.15.39:0/3936804897 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7fd060038f70 0x7fd06003b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:30.499 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:29.117+0000 7fd076000700 1 -- 172.21.15.39:0/3936804897 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fd06c011e00 con 0x7fd07810ec60 2024-03-18T15:58:30.499 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:29.117+0000 7fd07ce00700 1 --2- 172.21.15.39:0/3936804897 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7fd060038f70 0x7fd06003b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:30.499 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:29.118+0000 7fd07ce00700 1 --2- 172.21.15.39:0/3936804897 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7fd060038f70 0x7fd06003b430 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fd064006fd0 tx=0x7fd064006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:30.499 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:29.123+0000 7fd076000700 1 -- 172.21.15.39:0/3936804897 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd06c00b3e0 con 0x7fd07810ec60 2024-03-18T15:58:30.499 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:29.374+0000 7fd080200700 1 -- 172.21.15.39:0/3936804897 --> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7fd07807e410 con 0x7fd060038f70 2024-03-18T15:58:30.499 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:29.383+0000 7fd076000700 1 -- 172.21.15.39:0/3936804897 <== mgr.14120 v2:172.21.15.39:6800/1810238546 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fd07807e410 con 0x7fd060038f70 2024-03-18T15:58:30.499 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:29.386+0000 7fd080200700 1 -- 172.21.15.39:0/3936804897 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7fd060038f70 msgr2=0x7fd06003b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:30.499 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:29.386+0000 7fd080200700 1 --2- 172.21.15.39:0/3936804897 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7fd060038f70 0x7fd06003b430 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fd064006fd0 tx=0x7fd064006e40 comp rx=0 tx=0).stop 2024-03-18T15:58:30.499 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:29.386+0000 7fd080200700 1 -- 172.21.15.39:0/3936804897 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd07810ec60 msgr2=0x7fd0781a2140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:30.499 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:29.386+0000 7fd080200700 1 --2- 172.21.15.39:0/3936804897 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd07810ec60 0x7fd0781a2140 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd06c011b60 tx=0x7fd06c011c40 comp rx=0 tx=0).stop 2024-03-18T15:58:30.499 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:29.386+0000 7fd080200700 1 -- 172.21.15.39:0/3936804897 shutdown_connections 2024-03-18T15:58:30.499 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:29.386+0000 7fd080200700 1 --2- 172.21.15.39:0/3936804897 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7fd060038f70 0x7fd06003b430 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:30.500 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:29.386+0000 7fd080200700 1 --2- 172.21.15.39:0/3936804897 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd07810ec60 0x7fd0781a2140 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:30.500 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:29.386+0000 7fd080200700 1 -- 172.21.15.39:0/3936804897 >> 172.21.15.39:0/3936804897 conn(0x7fd07807d210 msgr2=0x7fd07807dd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:30.500 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:29.386+0000 7fd080200700 1 -- 172.21.15.39:0/3936804897 shutdown_connections 2024-03-18T15:58:30.500 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:29.387+0000 7fd080200700 1 -- 172.21.15.39:0/3936804897 wait complete. 2024-03-18T15:58:30.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:30 smithi039 ceph-mon[31721]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:58:30.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:30 smithi039 ceph-mon[31721]: Saving service mgr spec with placement count:2 2024-03-18T15:58:30.654 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:30 smithi039 ceph-mon[31721]: from='mgr.14120 172.21.15.39:0/2752076370' entity='mgr.x' 2024-03-18T15:58:32.305 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:31 smithi039 ceph-mon[31721]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:58:32.305 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:31 smithi039 ceph-mon[31721]: from='mgr.14120 172.21.15.39:0/2752076370' entity='mgr.x' 2024-03-18T15:58:32.305 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:31 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/1343674591' entity='client.admin' 2024-03-18T15:58:32.305 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:31 smithi039 ceph-mon[31721]: from='mgr.14120 172.21.15.39:0/2752076370' entity='mgr.x' 2024-03-18T15:58:32.324 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:31.389+0000 7f862ce00700 1 Processor -- start 2024-03-18T15:58:32.324 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:31.390+0000 7f862ce00700 1 -- start start 2024-03-18T15:58:32.324 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:31.390+0000 7f862ce00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8628107be0 0x7f8628108000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:32.324 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:31.390+0000 7f862ce00700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f86281085d0 con 0x7f8628107be0 2024-03-18T15:58:32.324 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:31.391+0000 7f8625e00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8628107be0 0x7f8628108000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:32.325 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:31.391+0000 7f8625e00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8628107be0 0x7f8628108000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:40956/0 (socket says 172.21.15.39:40956) 2024-03-18T15:58:32.325 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:31.391+0000 7f8625e00700 1 -- 172.21.15.39:0/1197023730 learned_addr learned my addr 172.21.15.39:0/1197023730 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T15:58:32.325 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:31.391+0000 7f8625e00700 1 -- 172.21.15.39:0/1197023730 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8628108de0 con 0x7f8628107be0 2024-03-18T15:58:32.325 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:31.392+0000 7f8625e00700 1 --2- 172.21.15.39:0/1197023730 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8628107be0 0x7f8628108000 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f86180098a0 tx=0x7f8618009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a6a129c6a30da14 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:32.325 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:31.392+0000 7f8625400700 1 -- 172.21.15.39:0/1197023730 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8618004160 con 0x7f8628107be0 2024-03-18T15:58:32.325 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:31.392+0000 7f8625400700 1 -- 172.21.15.39:0/1197023730 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8618004300 con 0x7f8628107be0 2024-03-18T15:58:32.325 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:31.392+0000 7f8625400700 1 -- 172.21.15.39:0/1197023730 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8618011b20 con 0x7f8628107be0 2024-03-18T15:58:32.325 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:31.393+0000 7f862ce00700 1 -- 172.21.15.39:0/1197023730 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8628107be0 msgr2=0x7f8628108000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:32.325 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:31.393+0000 7f862ce00700 1 --2- 172.21.15.39:0/1197023730 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8628107be0 0x7f8628108000 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f86180098a0 tx=0x7f8618009bb0 comp rx=0 tx=0).stop 2024-03-18T15:58:32.325 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:31.393+0000 7f862ce00700 1 -- 172.21.15.39:0/1197023730 shutdown_connections 2024-03-18T15:58:32.326 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:31.393+0000 7f862ce00700 1 --2- 172.21.15.39:0/1197023730 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8628107be0 0x7f8628108000 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:32.326 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:31.393+0000 7f862ce00700 1 -- 172.21.15.39:0/1197023730 >> 172.21.15.39:0/1197023730 conn(0x7f8628103180 msgr2=0x7f86281055c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:32.326 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:31.393+0000 7f862ce00700 1 -- 172.21.15.39:0/1197023730 shutdown_connections 2024-03-18T15:58:32.326 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:31.393+0000 7f862ce00700 1 -- 172.21.15.39:0/1197023730 wait complete. 2024-03-18T15:58:32.326 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:31.394+0000 7f862ce00700 1 Processor -- start 2024-03-18T15:58:32.326 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:31.394+0000 7f862ce00700 1 -- start start 2024-03-18T15:58:32.326 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:31.394+0000 7f862ce00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8628107be0 0x7f862819bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:32.326 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:31.395+0000 7f862ce00700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f862810c190 con 0x7f8628107be0 2024-03-18T15:58:32.326 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:31.395+0000 7f8625e00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8628107be0 0x7f862819bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:32.326 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:31.395+0000 7f8625e00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8628107be0 0x7f862819bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:40960/0 (socket says 172.21.15.39:40960) 2024-03-18T15:58:32.326 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:31.395+0000 7f8625e00700 1 -- 172.21.15.39:0/1343674591 learned_addr learned my addr 172.21.15.39:0/1343674591 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T15:58:32.327 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:31.395+0000 7f8625e00700 1 -- 172.21.15.39:0/1343674591 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f862819bff0 con 0x7f8628107be0 2024-03-18T15:58:32.327 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:31.395+0000 7f8625e00700 1 --2- 172.21.15.39:0/1343674591 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8628107be0 0x7f862819bab0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f86180117e0 tx=0x7f8618011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:32.327 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:31.396+0000 7f861ea00700 1 -- 172.21.15.39:0/1343674591 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8618003bf0 con 0x7f8628107be0 2024-03-18T15:58:32.327 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:31.396+0000 7f861ea00700 1 -- 172.21.15.39:0/1343674591 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f861801a910 con 0x7f8628107be0 2024-03-18T15:58:32.327 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:31.396+0000 7f861ea00700 1 -- 172.21.15.39:0/1343674591 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8618024970 con 0x7f8628107be0 2024-03-18T15:58:32.327 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:31.396+0000 7f862ce00700 1 -- 172.21.15.39:0/1343674591 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f862819c280 con 0x7f8628107be0 2024-03-18T15:58:32.327 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:31.396+0000 7f862ce00700 1 -- 172.21.15.39:0/1343674591 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f862819c6e0 con 0x7f8628107be0 2024-03-18T15:58:32.327 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:31.397+0000 7f862ce00700 1 -- 172.21.15.39:0/1343674591 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8628063900 con 0x7f8628107be0 2024-03-18T15:58:32.327 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:31.397+0000 7f861ea00700 1 -- 172.21.15.39:0/1343674591 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7f8618022020 con 0x7f8628107be0 2024-03-18T15:58:32.327 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:31.398+0000 7f861ea00700 1 --2- 172.21.15.39:0/1343674591 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7f860c038f70 0x7f860c03b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:32.327 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:31.398+0000 7f861ea00700 1 -- 172.21.15.39:0/1343674591 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f861804d1b0 con 0x7f8628107be0 2024-03-18T15:58:32.327 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:31.398+0000 7f861fe00700 1 --2- 172.21.15.39:0/1343674591 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7f860c038f70 0x7f860c03b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:32.328 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:31.399+0000 7f861fe00700 1 --2- 172.21.15.39:0/1343674591 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7f860c038f70 0x7f860c03b430 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f8610006fd0 tx=0x7f8610006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:32.328 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:31.403+0000 7f861ea00700 1 -- 172.21.15.39:0/1343674591 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f861801ab70 con 0x7f8628107be0 2024-03-18T15:58:32.328 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:31.631+0000 7f862ce00700 1 -- 172.21.15.39:0/1343674591 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7f862819ef70 con 0x7f8628107be0 2024-03-18T15:58:32.328 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:31.640+0000 7f861ea00700 1 -- 172.21.15.39:0/1343674591 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7f861802d3f0 con 0x7f8628107be0 2024-03-18T15:58:32.328 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:31.643+0000 7f862ce00700 1 -- 172.21.15.39:0/1343674591 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7f860c038f70 msgr2=0x7f860c03b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:32.328 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:31.644+0000 7f862ce00700 1 --2- 172.21.15.39:0/1343674591 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7f860c038f70 0x7f860c03b430 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f8610006fd0 tx=0x7f8610006e40 comp rx=0 tx=0).stop 2024-03-18T15:58:32.328 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:31.644+0000 7f862ce00700 1 -- 172.21.15.39:0/1343674591 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8628107be0 msgr2=0x7f862819bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:32.328 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:31.644+0000 7f862ce00700 1 --2- 172.21.15.39:0/1343674591 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8628107be0 0x7f862819bab0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f86180117e0 tx=0x7f8618011ba0 comp rx=0 tx=0).stop 2024-03-18T15:58:32.328 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:31.644+0000 7f862ce00700 1 -- 172.21.15.39:0/1343674591 shutdown_connections 2024-03-18T15:58:32.328 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:31.644+0000 7f862ce00700 1 --2- 172.21.15.39:0/1343674591 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7f860c038f70 0x7f860c03b430 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:32.328 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:31.644+0000 7f862ce00700 1 --2- 172.21.15.39:0/1343674591 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8628107be0 0x7f862819bab0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:32.328 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:31.644+0000 7f862ce00700 1 -- 172.21.15.39:0/1343674591 >> 172.21.15.39:0/1343674591 conn(0x7f8628103180 msgr2=0x7f8628103cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:32.328 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:31.644+0000 7f862ce00700 1 -- 172.21.15.39:0/1343674591 shutdown_connections 2024-03-18T15:58:32.328 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:31.644+0000 7f862ce00700 1 -- 172.21.15.39:0/1343674591 wait complete. 2024-03-18T15:58:33.931 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:32.864+0000 7faabca00700 1 Processor -- start 2024-03-18T15:58:33.932 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:32.864+0000 7faabca00700 1 -- start start 2024-03-18T15:58:33.932 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:32.865+0000 7faabca00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faab8109e10 0x7faab810a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:33.932 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:32.865+0000 7faabca00700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7faab810a770 con 0x7faab8109e10 2024-03-18T15:58:33.932 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:32.865+0000 7faab5e00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faab8109e10 0x7faab810a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:33.932 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:32.866+0000 7faab5e00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faab8109e10 0x7faab810a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:40972/0 (socket says 172.21.15.39:40972) 2024-03-18T15:58:33.932 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:32.866+0000 7faab5e00700 1 -- 172.21.15.39:0/2189422563 learned_addr learned my addr 172.21.15.39:0/2189422563 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T15:58:33.932 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:32.866+0000 7faab5e00700 1 -- 172.21.15.39:0/2189422563 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faab810af70 con 0x7faab8109e10 2024-03-18T15:58:33.937 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:32.866+0000 7faab5e00700 1 --2- 172.21.15.39:0/2189422563 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faab8109e10 0x7faab810a230 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7faaa80098a0 tx=0x7faaa8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a41c503ded5b465 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:33.937 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:32.867+0000 7faab4a00700 1 -- 172.21.15.39:0/2189422563 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faaa8004160 con 0x7faab8109e10 2024-03-18T15:58:33.937 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:32.867+0000 7faab4a00700 1 -- 172.21.15.39:0/2189422563 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faaa8004300 con 0x7faab8109e10 2024-03-18T15:58:33.937 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:32.867+0000 7faab4a00700 1 -- 172.21.15.39:0/2189422563 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faaa8011b20 con 0x7faab8109e10 2024-03-18T15:58:33.937 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:32.867+0000 7faabca00700 1 -- 172.21.15.39:0/2189422563 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faab8109e10 msgr2=0x7faab810a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:33.937 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:32.868+0000 7faabca00700 1 --2- 172.21.15.39:0/2189422563 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faab8109e10 0x7faab810a230 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7faaa80098a0 tx=0x7faaa8009bb0 comp rx=0 tx=0).stop 2024-03-18T15:58:33.937 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:32.868+0000 7faabca00700 1 -- 172.21.15.39:0/2189422563 shutdown_connections 2024-03-18T15:58:33.938 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:32.868+0000 7faabca00700 1 --2- 172.21.15.39:0/2189422563 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faab8109e10 0x7faab810a230 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:33.938 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:32.868+0000 7faabca00700 1 -- 172.21.15.39:0/2189422563 >> 172.21.15.39:0/2189422563 conn(0x7faab807d220 msgr2=0x7faab807d630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:33.938 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:32.868+0000 7faabca00700 1 -- 172.21.15.39:0/2189422563 shutdown_connections 2024-03-18T15:58:33.938 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:32.868+0000 7faabca00700 1 -- 172.21.15.39:0/2189422563 wait complete. 2024-03-18T15:58:33.938 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:32.869+0000 7faabca00700 1 Processor -- start 2024-03-18T15:58:33.938 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:32.869+0000 7faabca00700 1 -- start start 2024-03-18T15:58:33.938 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:32.869+0000 7faabca00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faab8109e10 0x7faab81a64c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:33.938 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:32.869+0000 7faabca00700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7faab810e3b0 con 0x7faab8109e10 2024-03-18T15:58:33.938 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:32.870+0000 7faab5e00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faab8109e10 0x7faab81a64c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:33.938 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:32.870+0000 7faab5e00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faab8109e10 0x7faab81a64c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:40982/0 (socket says 172.21.15.39:40982) 2024-03-18T15:58:33.938 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:32.870+0000 7faab5e00700 1 -- 172.21.15.39:0/3919226154 learned_addr learned my addr 172.21.15.39:0/3919226154 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T15:58:33.938 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:32.870+0000 7faab5e00700 1 -- 172.21.15.39:0/3919226154 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faab81a6a00 con 0x7faab8109e10 2024-03-18T15:58:33.938 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:32.870+0000 7faab5e00700 1 --2- 172.21.15.39:0/3919226154 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faab8109e10 0x7faab81a64c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7faaa80117e0 tx=0x7faaa8011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:33.939 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:32.871+0000 7faaaea00700 1 -- 172.21.15.39:0/3919226154 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faaa8003ba0 con 0x7faab8109e10 2024-03-18T15:58:33.939 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:32.871+0000 7faaaea00700 1 -- 172.21.15.39:0/3919226154 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faaa801a910 con 0x7faab8109e10 2024-03-18T15:58:33.939 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:32.871+0000 7faabca00700 1 -- 172.21.15.39:0/3919226154 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faab81a6c90 con 0x7faab8109e10 2024-03-18T15:58:33.939 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:32.871+0000 7faaaea00700 1 -- 172.21.15.39:0/3919226154 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faaa80248f0 con 0x7faab8109e10 2024-03-18T15:58:33.939 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:32.871+0000 7faabca00700 1 -- 172.21.15.39:0/3919226154 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faab81a70f0 con 0x7faab8109e10 2024-03-18T15:58:33.939 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:32.872+0000 7faabca00700 1 -- 172.21.15.39:0/3919226154 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faab8063900 con 0x7faab8109e10 2024-03-18T15:58:33.939 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:32.873+0000 7faaaea00700 1 -- 172.21.15.39:0/3919226154 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7faaa8022020 con 0x7faab8109e10 2024-03-18T15:58:33.939 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:32.873+0000 7faaaea00700 1 --2- 172.21.15.39:0/3919226154 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7faa9c038f20 0x7faa9c03b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:33.939 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:32.873+0000 7faaaea00700 1 -- 172.21.15.39:0/3919226154 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7faaa804d7d0 con 0x7faab8109e10 2024-03-18T15:58:33.939 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:32.874+0000 7faab5400700 1 --2- 172.21.15.39:0/3919226154 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7faa9c038f20 0x7faa9c03b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:33.939 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:32.874+0000 7faab5400700 1 --2- 172.21.15.39:0/3919226154 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7faa9c038f20 0x7faa9c03b3e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7faaa4006fd0 tx=0x7faaa4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:33.940 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:32.878+0000 7faaaea00700 1 -- 172.21.15.39:0/3919226154 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7faaa801ab70 con 0x7faab8109e10 2024-03-18T15:58:33.940 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:33.107+0000 7faabca00700 1 -- 172.21.15.39:0/3919226154 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7faab81a9980 con 0x7faab8109e10 2024-03-18T15:58:33.940 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:33.109+0000 7faaaea00700 1 -- 172.21.15.39:0/3919226154 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7faaa804d120 con 0x7faab8109e10 2024-03-18T15:58:33.940 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:33.113+0000 7faabca00700 1 -- 172.21.15.39:0/3919226154 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7faa9c038f20 msgr2=0x7faa9c03b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:33.940 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:33.113+0000 7faabca00700 1 --2- 172.21.15.39:0/3919226154 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7faa9c038f20 0x7faa9c03b3e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7faaa4006fd0 tx=0x7faaa4006e40 comp rx=0 tx=0).stop 2024-03-18T15:58:33.940 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:33.113+0000 7faabca00700 1 -- 172.21.15.39:0/3919226154 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faab8109e10 msgr2=0x7faab81a64c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:33.940 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:33.113+0000 7faabca00700 1 --2- 172.21.15.39:0/3919226154 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faab8109e10 0x7faab81a64c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7faaa80117e0 tx=0x7faaa8011ba0 comp rx=0 tx=0).stop 2024-03-18T15:58:33.940 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:33.113+0000 7faabca00700 1 -- 172.21.15.39:0/3919226154 shutdown_connections 2024-03-18T15:58:33.940 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:33.113+0000 7faabca00700 1 --2- 172.21.15.39:0/3919226154 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7faa9c038f20 0x7faa9c03b3e0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:33.940 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:33.113+0000 7faabca00700 1 --2- 172.21.15.39:0/3919226154 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faab8109e10 0x7faab81a64c0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:33.940 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:33.113+0000 7faabca00700 1 -- 172.21.15.39:0/3919226154 >> 172.21.15.39:0/3919226154 conn(0x7faab807d220 msgr2=0x7faab8107370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:33.940 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:33.113+0000 7faabca00700 1 -- 172.21.15.39:0/3919226154 shutdown_connections 2024-03-18T15:58:33.940 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:33.113+0000 7faabca00700 1 -- 172.21.15.39:0/3919226154 wait complete. 2024-03-18T15:58:33.940 INFO:teuthology.orchestra.run.smithi039.stdout:Enabling the dashboard module... 2024-03-18T15:58:34.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:34 smithi039 ceph-mon[31721]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:58:34.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:34 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/3919226154' entity='client.admin' 2024-03-18T15:58:36.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:35 smithi039 ceph-mon[31721]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:58:36.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:35 smithi039 ceph-mon[31721]: from='mgr.14120 172.21.15.39:0/2752076370' entity='mgr.x' 2024-03-18T15:58:36.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:35 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/308307086' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-18T15:58:36.511 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:58:36 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: ignoring --setuser ceph since I am not root 2024-03-18T15:58:36.511 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:58:36 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: ignoring --setgroup ceph since I am not root 2024-03-18T15:58:36.804 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:34.925+0000 7fdfb8800700 1 Processor -- start 2024-03-18T15:58:36.804 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:34.925+0000 7fdfb8800700 1 -- start start 2024-03-18T15:58:36.804 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:34.925+0000 7fdfb8800700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdfb0106b20 0x7fdfb0106f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:36.804 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:34.925+0000 7fdfb8800700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fdfb0107510 con 0x7fdfb0106b20 2024-03-18T15:58:36.804 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:34.926+0000 7fdfb5e00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdfb0106b20 0x7fdfb0106f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:36.804 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:34.926+0000 7fdfb5e00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdfb0106b20 0x7fdfb0106f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:40994/0 (socket says 172.21.15.39:40994) 2024-03-18T15:58:36.804 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:34.926+0000 7fdfb5e00700 1 -- 172.21.15.39:0/3570127199 learned_addr learned my addr 172.21.15.39:0/3570127199 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T15:58:36.804 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:34.926+0000 7fdfb5e00700 1 -- 172.21.15.39:0/3570127199 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfb0107d20 con 0x7fdfb0106b20 2024-03-18T15:58:36.805 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:34.927+0000 7fdfb5e00700 1 --2- 172.21.15.39:0/3570127199 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdfb0106b20 0x7fdfb0106f40 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fdfa40098a0 tx=0x7fdfa4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74bd754a31cac707 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:36.805 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:34.927+0000 7fdfb4a00700 1 -- 172.21.15.39:0/3570127199 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdfa4004160 con 0x7fdfb0106b20 2024-03-18T15:58:36.805 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:34.927+0000 7fdfb4a00700 1 -- 172.21.15.39:0/3570127199 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdfa4004300 con 0x7fdfb0106b20 2024-03-18T15:58:36.805 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:34.927+0000 7fdfb4a00700 1 -- 172.21.15.39:0/3570127199 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdfa4011b20 con 0x7fdfb0106b20 2024-03-18T15:58:36.805 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:34.928+0000 7fdfb8800700 1 -- 172.21.15.39:0/3570127199 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdfb0106b20 msgr2=0x7fdfb0106f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:36.805 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:34.928+0000 7fdfb8800700 1 --2- 172.21.15.39:0/3570127199 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdfb0106b20 0x7fdfb0106f40 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fdfa40098a0 tx=0x7fdfa4009bb0 comp rx=0 tx=0).stop 2024-03-18T15:58:36.805 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:34.928+0000 7fdfb8800700 1 -- 172.21.15.39:0/3570127199 shutdown_connections 2024-03-18T15:58:36.805 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:34.928+0000 7fdfb8800700 1 --2- 172.21.15.39:0/3570127199 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdfb0106b20 0x7fdfb0106f40 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:36.805 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:34.928+0000 7fdfb8800700 1 -- 172.21.15.39:0/3570127199 >> 172.21.15.39:0/3570127199 conn(0x7fdfb01020c0 msgr2=0x7fdfb0104500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:36.805 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:34.928+0000 7fdfb8800700 1 -- 172.21.15.39:0/3570127199 shutdown_connections 2024-03-18T15:58:36.805 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:34.929+0000 7fdfb8800700 1 -- 172.21.15.39:0/3570127199 wait complete. 2024-03-18T15:58:36.805 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:34.931+0000 7fdfb8800700 1 Processor -- start 2024-03-18T15:58:36.805 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:34.931+0000 7fdfb8800700 1 -- start start 2024-03-18T15:58:36.806 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:34.931+0000 7fdfb8800700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdfb0106b20 0x7fdfb019bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:36.806 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:34.931+0000 7fdfb8800700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fdfb010b0d0 con 0x7fdfb0106b20 2024-03-18T15:58:36.806 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:34.932+0000 7fdfb5e00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdfb0106b20 0x7fdfb019bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:36.806 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:34.932+0000 7fdfb5e00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdfb0106b20 0x7fdfb019bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:41000/0 (socket says 172.21.15.39:41000) 2024-03-18T15:58:36.806 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:34.932+0000 7fdfb5e00700 1 -- 172.21.15.39:0/308307086 learned_addr learned my addr 172.21.15.39:0/308307086 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T15:58:36.806 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:34.932+0000 7fdfb5e00700 1 -- 172.21.15.39:0/308307086 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfb019c070 con 0x7fdfb0106b20 2024-03-18T15:58:36.806 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:34.932+0000 7fdfb5e00700 1 --2- 172.21.15.39:0/308307086 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdfb0106b20 0x7fdfb019bb30 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fdfa40117e0 tx=0x7fdfa4011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:36.806 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:34.933+0000 7fdfaaa00700 1 -- 172.21.15.39:0/308307086 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdfa4003bf0 con 0x7fdfb0106b20 2024-03-18T15:58:36.806 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:34.933+0000 7fdfaaa00700 1 -- 172.21.15.39:0/308307086 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdfa401a910 con 0x7fdfb0106b20 2024-03-18T15:58:36.806 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:34.933+0000 7fdfb8800700 1 -- 172.21.15.39:0/308307086 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfb019c300 con 0x7fdfb0106b20 2024-03-18T15:58:36.806 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:34.933+0000 7fdfaaa00700 1 -- 172.21.15.39:0/308307086 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fdfa4024970 con 0x7fdfb0106b20 2024-03-18T15:58:36.807 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:34.933+0000 7fdfb8800700 1 -- 172.21.15.39:0/308307086 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfb019c760 con 0x7fdfb0106b20 2024-03-18T15:58:36.807 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:34.935+0000 7fdfaaa00700 1 -- 172.21.15.39:0/308307086 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 8) ==== 46106+0+0 (secure 0 0 0) 0x7fdfa4022020 con 0x7fdfb0106b20 2024-03-18T15:58:36.807 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:34.935+0000 7fdfb8800700 1 -- 172.21.15.39:0/308307086 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdfb0063900 con 0x7fdfb0106b20 2024-03-18T15:58:36.807 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:34.935+0000 7fdfaaa00700 1 --2- 172.21.15.39:0/308307086 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7fdf98038f20 0x7fdf9803b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:36.807 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:34.935+0000 7fdfaaa00700 1 -- 172.21.15.39:0/308307086 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fdfa404d690 con 0x7fdfb0106b20 2024-03-18T15:58:36.807 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:34.935+0000 7fdfb5400700 1 --2- 172.21.15.39:0/308307086 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7fdf98038f20 0x7fdf9803b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:36.807 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:34.936+0000 7fdfb5400700 1 --2- 172.21.15.39:0/308307086 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7fdf98038f20 0x7fdf9803b3e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fdfa0006fd0 tx=0x7fdfa0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:36.807 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:34.941+0000 7fdfaaa00700 1 -- 172.21.15.39:0/308307086 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fdfa4017ca0 con 0x7fdfb0106b20 2024-03-18T15:58:36.808 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:35.233+0000 7fdfb8800700 1 -- 172.21.15.39:0/308307086 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7fdfb0062ee0 con 0x7fdfb0106b20 2024-03-18T15:58:36.808 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:36.117+0000 7fdfaaa00700 1 -- 172.21.15.39:0/308307086 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7fdfa404cfe0 con 0x7fdfb0106b20 2024-03-18T15:58:36.808 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:36.121+0000 7fdfaaa00700 1 -- 172.21.15.39:0/308307086 <== mon.0 v2:172.21.15.39:3300/0 8 ==== mgrmap(e 9) ==== 46119+0+0 (secure 0 0 0) 0x7fdfa402d3f0 con 0x7fdfb0106b20 2024-03-18T15:58:36.808 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:36.123+0000 7fdfb8800700 1 -- 172.21.15.39:0/308307086 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7fdf98038f20 msgr2=0x7fdf9803b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:36.808 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:36.123+0000 7fdfb8800700 1 --2- 172.21.15.39:0/308307086 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7fdf98038f20 0x7fdf9803b3e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fdfa0006fd0 tx=0x7fdfa0006e40 comp rx=0 tx=0).stop 2024-03-18T15:58:36.808 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:36.123+0000 7fdfb8800700 1 -- 172.21.15.39:0/308307086 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdfb0106b20 msgr2=0x7fdfb019bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:36.808 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:36.123+0000 7fdfb8800700 1 --2- 172.21.15.39:0/308307086 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdfb0106b20 0x7fdfb019bb30 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fdfa40117e0 tx=0x7fdfa4011ba0 comp rx=0 tx=0).stop 2024-03-18T15:58:36.809 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:36.123+0000 7fdfb8800700 1 -- 172.21.15.39:0/308307086 shutdown_connections 2024-03-18T15:58:36.809 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:36.123+0000 7fdfb8800700 1 --2- 172.21.15.39:0/308307086 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7fdf98038f20 0x7fdf9803b3e0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:36.809 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:36.123+0000 7fdfb8800700 1 --2- 172.21.15.39:0/308307086 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdfb0106b20 0x7fdfb019bb30 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:36.809 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:36.123+0000 7fdfb8800700 1 -- 172.21.15.39:0/308307086 >> 172.21.15.39:0/308307086 conn(0x7fdfb01020c0 msgr2=0x7fdfb0102ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:36.809 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:36.124+0000 7fdfb8800700 1 -- 172.21.15.39:0/308307086 shutdown_connections 2024-03-18T15:58:36.809 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:36.124+0000 7fdfb8800700 1 -- 172.21.15.39:0/308307086 wait complete. 2024-03-18T15:58:37.062 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:58:36 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T15:58:36.963+0000 7f6f269de200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T15:58:37.336 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:58:37 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T15:58:37.091+0000 7f6f269de200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T15:58:37.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:37 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/308307086' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-18T15:58:37.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:37 smithi039 ceph-mon[31721]: mgrmap e9: x(active, since 25s) 2024-03-18T15:58:38.250 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:38 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/178830935' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-18T15:58:38.251 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:58:37 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T15:58:37.934+0000 7f6f269de200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T15:58:38.251 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:58:38 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T15:58:38.064+0000 7f6f269de200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T15:58:38.251 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:58:38 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T15:58:38.248+0000 7f6f269de200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T15:58:38.388 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout { 2024-03-18T15:58:38.388 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-18T15:58:38.388 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-18T15:58:38.388 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-18T15:58:38.389 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-18T15:58:38.389 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout } 2024-03-18T15:58:38.389 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:37.407+0000 7f546da00700 1 Processor -- start 2024-03-18T15:58:38.389 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:37.407+0000 7f546da00700 1 -- start start 2024-03-18T15:58:38.390 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:37.408+0000 7f546da00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5468107bf0 0x7f5468108010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:38.390 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:37.408+0000 7f546da00700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f54681085e0 con 0x7f5468107bf0 2024-03-18T15:58:38.390 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:37.408+0000 7f5466a00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5468107bf0 0x7f5468108010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:38.390 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:37.408+0000 7f5466a00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5468107bf0 0x7f5468108010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:41032/0 (socket says 172.21.15.39:41032) 2024-03-18T15:58:38.391 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:37.408+0000 7f5466a00700 1 -- 172.21.15.39:0/284394337 learned_addr learned my addr 172.21.15.39:0/284394337 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T15:58:38.391 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:37.409+0000 7f5466a00700 1 -- 172.21.15.39:0/284394337 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5468108df0 con 0x7f5468107bf0 2024-03-18T15:58:38.391 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:37.409+0000 7f5466a00700 1 --2- 172.21.15.39:0/284394337 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5468107bf0 0x7f5468108010 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f54580098a0 tx=0x7f5458009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3483036438d9ef0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:38.392 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:37.409+0000 7f5465600700 1 -- 172.21.15.39:0/284394337 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5458004160 con 0x7f5468107bf0 2024-03-18T15:58:38.392 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:37.409+0000 7f5465600700 1 -- 172.21.15.39:0/284394337 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5458004300 con 0x7f5468107bf0 2024-03-18T15:58:38.392 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:37.409+0000 7f5465600700 1 -- 172.21.15.39:0/284394337 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5458011b20 con 0x7f5468107bf0 2024-03-18T15:58:38.392 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:37.410+0000 7f546da00700 1 -- 172.21.15.39:0/284394337 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5468107bf0 msgr2=0x7f5468108010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:38.396 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:37.410+0000 7f546da00700 1 --2- 172.21.15.39:0/284394337 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5468107bf0 0x7f5468108010 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f54580098a0 tx=0x7f5458009bb0 comp rx=0 tx=0).stop 2024-03-18T15:58:38.396 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:37.410+0000 7f546da00700 1 -- 172.21.15.39:0/284394337 shutdown_connections 2024-03-18T15:58:38.396 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:37.410+0000 7f546da00700 1 --2- 172.21.15.39:0/284394337 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5468107bf0 0x7f5468108010 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:38.396 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:37.410+0000 7f546da00700 1 -- 172.21.15.39:0/284394337 >> 172.21.15.39:0/284394337 conn(0x7f54681031b0 msgr2=0x7f54681055d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:38.396 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:37.410+0000 7f546da00700 1 -- 172.21.15.39:0/284394337 shutdown_connections 2024-03-18T15:58:38.396 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:37.410+0000 7f546da00700 1 -- 172.21.15.39:0/284394337 wait complete. 2024-03-18T15:58:38.396 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:37.412+0000 7f546da00700 1 Processor -- start 2024-03-18T15:58:38.396 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:37.412+0000 7f546da00700 1 -- start start 2024-03-18T15:58:38.397 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:37.412+0000 7f546da00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5468107bf0 0x7f546807db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:38.397 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:37.412+0000 7f546da00700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f546810c1a0 con 0x7f5468107bf0 2024-03-18T15:58:38.397 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:37.413+0000 7f5466a00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5468107bf0 0x7f546807db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:38.397 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:37.413+0000 7f5466a00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5468107bf0 0x7f546807db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:41040/0 (socket says 172.21.15.39:41040) 2024-03-18T15:58:38.397 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:37.413+0000 7f5466a00700 1 -- 172.21.15.39:0/178830935 learned_addr learned my addr 172.21.15.39:0/178830935 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T15:58:38.397 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:37.413+0000 7f5466a00700 1 -- 172.21.15.39:0/178830935 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f546807e040 con 0x7f5468107bf0 2024-03-18T15:58:38.397 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:37.413+0000 7f5466a00700 1 --2- 172.21.15.39:0/178830935 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5468107bf0 0x7f546807db00 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5458000c00 tx=0x7f5458011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:38.397 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:37.414+0000 7f545f400700 1 -- 172.21.15.39:0/178830935 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5458003b60 con 0x7f5468107bf0 2024-03-18T15:58:38.397 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:37.414+0000 7f545f400700 1 -- 172.21.15.39:0/178830935 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f545801a910 con 0x7f5468107bf0 2024-03-18T15:58:38.398 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:37.414+0000 7f546da00700 1 -- 172.21.15.39:0/178830935 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f546807c230 con 0x7f5468107bf0 2024-03-18T15:58:38.398 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:37.414+0000 7f545f400700 1 -- 172.21.15.39:0/178830935 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5458024930 con 0x7f5468107bf0 2024-03-18T15:58:38.398 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:37.414+0000 7f546da00700 1 -- 172.21.15.39:0/178830935 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f546807c6f0 con 0x7f5468107bf0 2024-03-18T15:58:38.398 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:37.415+0000 7f546da00700 1 -- 172.21.15.39:0/178830935 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5468063900 con 0x7f5468107bf0 2024-03-18T15:58:38.398 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:37.415+0000 7f545f400700 1 -- 172.21.15.39:0/178830935 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 9) ==== 46119+0+0 (secure 0 0 0) 0x7f5458022020 con 0x7f5468107bf0 2024-03-18T15:58:38.398 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:37.416+0000 7f545f400700 1 --2- 172.21.15.39:0/178830935 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7f544c038fd0 0x7f544c03b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:38.398 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:37.416+0000 7f545f400700 1 -- 172.21.15.39:0/178830935 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f545804d040 con 0x7f5468107bf0 2024-03-18T15:58:38.398 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:37.416+0000 7f5466000700 1 -- 172.21.15.39:0/178830935 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7f544c038fd0 msgr2=0x7f544c03b490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.39:6800/1810238546 2024-03-18T15:58:38.398 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:37.416+0000 7f5466000700 1 --2- 172.21.15.39:0/178830935 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7f544c038fd0 0x7f544c03b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T15:58:38.399 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:37.421+0000 7f545f400700 1 -- 172.21.15.39:0/178830935 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f545802ddb0 con 0x7f5468107bf0 2024-03-18T15:58:38.399 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:37.617+0000 7f5466000700 1 -- 172.21.15.39:0/178830935 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7f544c038fd0 msgr2=0x7f544c03b490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.39:6800/1810238546 2024-03-18T15:58:38.399 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:37.617+0000 7f5466000700 1 --2- 172.21.15.39:0/178830935 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7f544c038fd0 0x7f544c03b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T15:58:38.399 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:37.703+0000 7f546da00700 1 -- 172.21.15.39:0/178830935 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f546807ca30 con 0x7f5468107bf0 2024-03-18T15:58:38.399 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:37.704+0000 7f545f400700 1 -- 172.21.15.39:0/178830935 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7f546807ca30 con 0x7f5468107bf0 2024-03-18T15:58:38.399 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:37.707+0000 7f546da00700 1 -- 172.21.15.39:0/178830935 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7f544c038fd0 msgr2=0x7f544c03b490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T15:58:38.399 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:37.707+0000 7f546da00700 1 --2- 172.21.15.39:0/178830935 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7f544c038fd0 0x7f544c03b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:38.401 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:37.707+0000 7f546da00700 1 -- 172.21.15.39:0/178830935 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5468107bf0 msgr2=0x7f546807db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:38.401 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:37.707+0000 7f546da00700 1 --2- 172.21.15.39:0/178830935 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5468107bf0 0x7f546807db00 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5458000c00 tx=0x7f5458011c10 comp rx=0 tx=0).stop 2024-03-18T15:58:38.401 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:37.707+0000 7f546da00700 1 -- 172.21.15.39:0/178830935 shutdown_connections 2024-03-18T15:58:38.402 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:37.707+0000 7f546da00700 1 --2- 172.21.15.39:0/178830935 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7f544c038fd0 0x7f544c03b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:38.402 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:37.707+0000 7f546da00700 1 --2- 172.21.15.39:0/178830935 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5468107bf0 0x7f546807db00 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:38.402 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:37.707+0000 7f546da00700 1 -- 172.21.15.39:0/178830935 >> 172.21.15.39:0/178830935 conn(0x7f54681031b0 msgr2=0x7f5468103c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:38.402 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:37.708+0000 7f546da00700 1 -- 172.21.15.39:0/178830935 shutdown_connections 2024-03-18T15:58:38.402 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:37.708+0000 7f546da00700 1 -- 172.21.15.39:0/178830935 wait complete. 2024-03-18T15:58:38.402 INFO:teuthology.orchestra.run.smithi039.stdout:Waiting for the mgr to restart... 2024-03-18T15:58:38.402 INFO:teuthology.orchestra.run.smithi039.stdout:Waiting for mgr epoch 9... 2024-03-18T15:58:38.501 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:58:38 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T15:58:38.381+0000 7f6f269de200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T15:58:38.502 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:58:38 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T15:58:38.499+0000 7f6f269de200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T15:58:39.086 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:58:38 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T15:58:38.857+0000 7f6f269de200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T15:58:39.086 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:58:39 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T15:58:39.008+0000 7f6f269de200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T15:58:39.482 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:58:39 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T15:58:39.127+0000 7f6f269de200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T15:58:39.482 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:58:39 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T15:58:39.254+0000 7f6f269de200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T15:58:39.482 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:58:39 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T15:58:39.479+0000 7f6f269de200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T15:58:41.086 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:58:40 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T15:58:40.806+0000 7f6f269de200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T15:58:41.087 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:58:40 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T15:58:40.984+0000 7f6f269de200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T15:58:41.807 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:58:41 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T15:58:41.414+0000 7f6f269de200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T15:58:41.807 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:58:41 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T15:58:41.537+0000 7f6f269de200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T15:58:41.807 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:58:41 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T15:58:41.804+0000 7f6f269de200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T15:58:42.336 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:58:42 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T15:58:42.058+0000 7f6f269de200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T15:58:42.778 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:58:42 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T15:58:42.448+0000 7f6f269de200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T15:58:43.086 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:58:42 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T15:58:42.775+0000 7f6f269de200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T15:58:43.430 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:58:43 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T15:58:43.091+0000 7f6f269de200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T15:58:44.586 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:58:44 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T15:58:44.182+0000 7f6f269de200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T15:58:45.047 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:58:44 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T15:58:44.910+0000 7f6f269de200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T15:58:45.047 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:58:45 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T15:58:45.044+0000 7f6f269de200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T15:58:45.836 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:58:45 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T15:58:45.335+0000 7f6f269de200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T15:58:45.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:45 smithi039 ceph-mon[31721]: Active manager daemon x restarted 2024-03-18T15:58:45.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:45 smithi039 ceph-mon[31721]: Activating manager daemon x 2024-03-18T15:58:45.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:45 smithi039 ceph-mon[31721]: osdmap e3: 0 total, 0 up, 0 in 2024-03-18T15:58:45.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:45 smithi039 ceph-mon[31721]: mgrmap e10: x(active, starting, since 0.01097s) 2024-03-18T15:58:45.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:45 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T15:58:45.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:45 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-18T15:58:45.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:45 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-18T15:58:45.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:45 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-18T15:58:45.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:45 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-18T15:58:45.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:45 smithi039 ceph-mon[31721]: Manager daemon x is now available 2024-03-18T15:58:46.700 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:46 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:58:46.700 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:46 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:58:46.700 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:46 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-18T15:58:46.700 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:46 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-18T15:58:46.700 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:46 smithi039 ceph-mon[31721]: [18/Mar/2024:15:58:45] ENGINE Bus STARTING 2024-03-18T15:58:46.700 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:46 smithi039 ceph-mon[31721]: [18/Mar/2024:15:58:46] ENGINE Serving on http://172.21.15.39:8765 2024-03-18T15:58:46.700 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:46 smithi039 ceph-mon[31721]: [18/Mar/2024:15:58:46] ENGINE Serving on https://172.21.15.39:7150 2024-03-18T15:58:46.700 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:46 smithi039 ceph-mon[31721]: [18/Mar/2024:15:58:46] ENGINE Bus STARTED 2024-03-18T15:58:46.700 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:46 smithi039 ceph-mon[31721]: mgrmap e11: x(active, since 1.00909s) 2024-03-18T15:58:47.004 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout { 2024-03-18T15:58:47.004 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-18T15:58:47.004 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-18T15:58:47.004 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout } 2024-03-18T15:58:47.004 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:38.984+0000 7f122a200700 1 Processor -- start 2024-03-18T15:58:47.012 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:38.984+0000 7f122a200700 1 -- start start 2024-03-18T15:58:47.012 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:38.985+0000 7f122a200700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f12241086e0 0x7f122410aad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:47.012 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:38.985+0000 7f122a200700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f1224075990 con 0x7f12241086e0 2024-03-18T15:58:47.012 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:38.985+0000 7f1223400700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f12241086e0 0x7f122410aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:47.012 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:38.985+0000 7f1223400700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f12241086e0 0x7f122410aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:56406/0 (socket says 172.21.15.39:56406) 2024-03-18T15:58:47.012 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:38.985+0000 7f1223400700 1 -- 172.21.15.39:0/1941604987 learned_addr learned my addr 172.21.15.39:0/1941604987 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T15:58:47.012 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:38.986+0000 7f1223400700 1 -- 172.21.15.39:0/1941604987 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f122410b010 con 0x7f12241086e0 2024-03-18T15:58:47.013 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:38.986+0000 7f1223400700 1 --2- 172.21.15.39:0/1941604987 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f12241086e0 0x7f122410aad0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f12140098a0 tx=0x7f1214009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4983dc47ba124eb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:47.013 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:38.986+0000 7f1222000700 1 -- 172.21.15.39:0/1941604987 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1214004160 con 0x7f12241086e0 2024-03-18T15:58:47.013 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:38.986+0000 7f1222000700 1 -- 172.21.15.39:0/1941604987 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1214004300 con 0x7f12241086e0 2024-03-18T15:58:47.013 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:38.986+0000 7f1222000700 1 -- 172.21.15.39:0/1941604987 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1214011b20 con 0x7f12241086e0 2024-03-18T15:58:47.013 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:38.987+0000 7f122a200700 1 -- 172.21.15.39:0/1941604987 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f12241086e0 msgr2=0x7f122410aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:47.013 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:38.987+0000 7f122a200700 1 --2- 172.21.15.39:0/1941604987 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f12241086e0 0x7f122410aad0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f12140098a0 tx=0x7f1214009bb0 comp rx=0 tx=0).stop 2024-03-18T15:58:47.013 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:38.987+0000 7f122a200700 1 -- 172.21.15.39:0/1941604987 shutdown_connections 2024-03-18T15:58:47.013 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:38.987+0000 7f122a200700 1 --2- 172.21.15.39:0/1941604987 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f12241086e0 0x7f122410aad0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:47.013 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:38.987+0000 7f122a200700 1 -- 172.21.15.39:0/1941604987 >> 172.21.15.39:0/1941604987 conn(0x7f12241020d0 msgr2=0x7f12241044f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:47.013 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:38.987+0000 7f122a200700 1 -- 172.21.15.39:0/1941604987 shutdown_connections 2024-03-18T15:58:47.013 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:38.988+0000 7f122a200700 1 -- 172.21.15.39:0/1941604987 wait complete. 2024-03-18T15:58:47.013 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:38.989+0000 7f122a200700 1 Processor -- start 2024-03-18T15:58:47.013 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:38.989+0000 7f122a200700 1 -- start start 2024-03-18T15:58:47.013 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:38.990+0000 7f122a200700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f12241086e0 0x7f1224107f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:47.014 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:38.990+0000 7f122a200700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f122410c020 con 0x7f12241086e0 2024-03-18T15:58:47.014 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:38.990+0000 7f1223400700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f12241086e0 0x7f1224107f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:47.014 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:38.990+0000 7f1223400700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f12241086e0 0x7f1224107f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:56420/0 (socket says 172.21.15.39:56420) 2024-03-18T15:58:47.014 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:38.990+0000 7f1223400700 1 -- 172.21.15.39:0/572106094 learned_addr learned my addr 172.21.15.39:0/572106094 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T15:58:47.014 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:38.990+0000 7f1223400700 1 -- 172.21.15.39:0/572106094 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12241084a0 con 0x7f12241086e0 2024-03-18T15:58:47.014 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:38.991+0000 7f1223400700 1 --2- 172.21.15.39:0/572106094 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f12241086e0 0x7f1224107f60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f12140117e0 tx=0x7f1214011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:47.014 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:38.991+0000 7f121be00700 1 -- 172.21.15.39:0/572106094 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1214003bf0 con 0x7f12241086e0 2024-03-18T15:58:47.014 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:38.991+0000 7f121be00700 1 -- 172.21.15.39:0/572106094 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f121401a910 con 0x7f12241086e0 2024-03-18T15:58:47.014 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:38.991+0000 7f122a200700 1 -- 172.21.15.39:0/572106094 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1224106690 con 0x7f12241086e0 2024-03-18T15:58:47.014 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:38.991+0000 7f121be00700 1 -- 172.21.15.39:0/572106094 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1214024970 con 0x7f12241086e0 2024-03-18T15:58:47.014 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:38.991+0000 7f122a200700 1 -- 172.21.15.39:0/572106094 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1224106b50 con 0x7f12241086e0 2024-03-18T15:58:47.014 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:38.992+0000 7f121be00700 1 -- 172.21.15.39:0/572106094 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 9) ==== 46119+0+0 (secure 0 0 0) 0x7f1214022020 con 0x7f12241086e0 2024-03-18T15:58:47.014 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:38.993+0000 7f121be00700 1 --2- 172.21.15.39:0/572106094 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7f1208038f20 0x7f120803b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:47.015 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:38.993+0000 7f122a200700 1 -- 172.21.15.39:0/572106094 --> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f1224063900 con 0x7f1208038f20 2024-03-18T15:58:47.015 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:38.993+0000 7f121be00700 1 -- 172.21.15.39:0/572106094 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f121404d2e0 con 0x7f12241086e0 2024-03-18T15:58:47.015 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:38.993+0000 7f1222a00700 1 -- 172.21.15.39:0/572106094 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7f1208038f20 msgr2=0x7f120803b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.39:6800/1810238546 2024-03-18T15:58:47.015 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:38.993+0000 7f1222a00700 1 --2- 172.21.15.39:0/572106094 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7f1208038f20 0x7f120803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T15:58:47.015 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:39.194+0000 7f1222a00700 1 -- 172.21.15.39:0/572106094 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7f1208038f20 msgr2=0x7f120803b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.39:6800/1810238546 2024-03-18T15:58:47.015 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:39.194+0000 7f1222a00700 1 --2- 172.21.15.39:0/572106094 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7f1208038f20 0x7f120803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-18T15:58:47.015 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:39.595+0000 7f1222a00700 1 -- 172.21.15.39:0/572106094 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7f1208038f20 msgr2=0x7f120803b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.39:6800/1810238546 2024-03-18T15:58:47.015 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:39.595+0000 7f1222a00700 1 --2- 172.21.15.39:0/572106094 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7f1208038f20 0x7f120803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-18T15:58:47.015 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:40.396+0000 7f1222a00700 1 -- 172.21.15.39:0/572106094 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7f1208038f20 msgr2=0x7f120803b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.39:6800/1810238546 2024-03-18T15:58:47.015 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:40.396+0000 7f1222a00700 1 --2- 172.21.15.39:0/572106094 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7f1208038f20 0x7f120803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-18T15:58:47.016 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:41.998+0000 7f1222a00700 1 -- 172.21.15.39:0/572106094 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7f1208038f20 msgr2=0x7f120803b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.39:6800/1810238546 2024-03-18T15:58:47.016 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:41.998+0000 7f1222a00700 1 --2- 172.21.15.39:0/572106094 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7f1208038f20 0x7f120803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-18T15:58:47.016 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:45.201+0000 7f1222a00700 1 -- 172.21.15.39:0/572106094 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7f1208038f20 msgr2=0x7f120803b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.39:6800/1810238546 2024-03-18T15:58:47.016 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:45.201+0000 7f1222a00700 1 --2- 172.21.15.39:0/572106094 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7f1208038f20 0x7f120803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-03-18T15:58:47.016 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:45.347+0000 7f121be00700 1 -- 172.21.15.39:0/572106094 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mgrmap(e 10) ==== 45886+0+0 (secure 0 0 0) 0x7f121402d910 con 0x7f12241086e0 2024-03-18T15:58:47.016 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:45.347+0000 7f121be00700 1 -- 172.21.15.39:0/572106094 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7f1208038f20 msgr2=0x7f120803b3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T15:58:47.016 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:45.347+0000 7f121be00700 1 --2- 172.21.15.39:0/572106094 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7f1208038f20 0x7f120803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:47.016 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:46.346+0000 7f121be00700 1 -- 172.21.15.39:0/572106094 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mgrmap(e 11) ==== 46013+0+0 (secure 0 0 0) 0x7f121404ef60 con 0x7f12241086e0 2024-03-18T15:58:47.016 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:46.346+0000 7f121be00700 1 --2- 172.21.15.39:0/572106094 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f120803c7b0 0x7f120803eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:47.016 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:46.346+0000 7f121be00700 1 -- 172.21.15.39:0/572106094 --> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f12140068a0 con 0x7f120803c7b0 2024-03-18T15:58:47.016 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:46.348+0000 7f1222a00700 1 --2- 172.21.15.39:0/572106094 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f120803c7b0 0x7f120803eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:47.016 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:46.349+0000 7f1222a00700 1 --2- 172.21.15.39:0/572106094 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f120803c7b0 0x7f120803eba0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f1210003a10 tx=0x7f1210009270 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:47.016 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:46.350+0000 7f121be00700 1 -- 172.21.15.39:0/572106094 <== mgr.14152 v2:172.21.15.39:6800/2749214286 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f12140068a0 con 0x7f120803c7b0 2024-03-18T15:58:47.016 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:46.357+0000 7f122a200700 1 -- 172.21.15.39:0/572106094 --> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f1224103280 con 0x7f120803c7b0 2024-03-18T15:58:47.017 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:46.358+0000 7f121be00700 1 -- 172.21.15.39:0/572106094 <== mgr.14152 v2:172.21.15.39:6800/2749214286 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7f1224103280 con 0x7f120803c7b0 2024-03-18T15:58:47.017 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:46.358+0000 7f122a200700 1 -- 172.21.15.39:0/572106094 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f120803c7b0 msgr2=0x7f120803eba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:47.017 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:46.358+0000 7f122a200700 1 --2- 172.21.15.39:0/572106094 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f120803c7b0 0x7f120803eba0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f1210003a10 tx=0x7f1210009270 comp rx=0 tx=0).stop 2024-03-18T15:58:47.017 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:46.358+0000 7f122a200700 1 -- 172.21.15.39:0/572106094 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f12241086e0 msgr2=0x7f1224107f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:47.017 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:46.358+0000 7f122a200700 1 --2- 172.21.15.39:0/572106094 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f12241086e0 0x7f1224107f60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f12140117e0 tx=0x7f1214011c10 comp rx=0 tx=0).stop 2024-03-18T15:58:47.017 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:46.359+0000 7f122a200700 1 -- 172.21.15.39:0/572106094 shutdown_connections 2024-03-18T15:58:47.017 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:46.359+0000 7f122a200700 1 --2- 172.21.15.39:0/572106094 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f120803c7b0 0x7f120803eba0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:47.017 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:46.359+0000 7f122a200700 1 --2- 172.21.15.39:0/572106094 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f12241086e0 0x7f1224107f60 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:47.017 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:46.359+0000 7f122a200700 1 --2- 172.21.15.39:0/572106094 >> [v2:172.21.15.39:6800/1810238546,v1:172.21.15.39:6801/1810238546] conn(0x7f1208038f20 0x7f120803b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:47.017 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:46.359+0000 7f122a200700 1 -- 172.21.15.39:0/572106094 >> 172.21.15.39:0/572106094 conn(0x7f12241020d0 msgr2=0x7f1224102b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:47.017 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:46.359+0000 7f122a200700 1 -- 172.21.15.39:0/572106094 shutdown_connections 2024-03-18T15:58:47.017 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:46.359+0000 7f122a200700 1 -- 172.21.15.39:0/572106094 wait complete. 2024-03-18T15:58:47.018 INFO:teuthology.orchestra.run.smithi039.stdout:mgr epoch 9 is available 2024-03-18T15:58:47.018 INFO:teuthology.orchestra.run.smithi039.stdout:Generating a dashboard self-signed certificate... 2024-03-18T15:58:47.634 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:47 smithi039 ceph-mon[31721]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-18T15:58:47.634 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:47 smithi039 ceph-mon[31721]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-18T15:58:48.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:48 smithi039 ceph-mon[31721]: mgrmap e12: x(active, since 2s) 2024-03-18T15:58:48.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:48 smithi039 ceph-mon[31721]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:58:48.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:48 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:58:48.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:48 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:58:48.949 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-18T15:58:48.949 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:47.886+0000 7fdea9600700 1 Processor -- start 2024-03-18T15:58:48.949 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:47.886+0000 7fdea9600700 1 -- start start 2024-03-18T15:58:48.949 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:47.886+0000 7fdea9600700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdea407de70 0x7fdea407c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:48.949 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:47.886+0000 7fdea9600700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fdea407c8a0 con 0x7fdea407de70 2024-03-18T15:58:48.949 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:47.887+0000 7fdea2a00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdea407de70 0x7fdea407c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:48.949 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:47.887+0000 7fdea2a00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdea407de70 0x7fdea407c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:56502/0 (socket says 172.21.15.39:56502) 2024-03-18T15:58:48.950 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:47.887+0000 7fdea2a00700 1 -- 172.21.15.39:0/3195753994 learned_addr learned my addr 172.21.15.39:0/3195753994 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T15:58:48.950 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:47.888+0000 7fdea2a00700 1 -- 172.21.15.39:0/3195753994 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdea407ca20 con 0x7fdea407de70 2024-03-18T15:58:48.950 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:47.888+0000 7fdea2a00700 1 --2- 172.21.15.39:0/3195753994 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdea407de70 0x7fdea407c2d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fde940098a0 tx=0x7fde94009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ff8a5e37ac4c74d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:48.950 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:47.888+0000 7fdea1600700 1 -- 172.21.15.39:0/3195753994 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fde94004160 con 0x7fdea407de70 2024-03-18T15:58:48.950 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:47.888+0000 7fdea1600700 1 -- 172.21.15.39:0/3195753994 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde94004300 con 0x7fdea407de70 2024-03-18T15:58:48.950 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:47.888+0000 7fdea1600700 1 -- 172.21.15.39:0/3195753994 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fde94011b20 con 0x7fdea407de70 2024-03-18T15:58:48.950 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:47.889+0000 7fdea9600700 1 -- 172.21.15.39:0/3195753994 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdea407de70 msgr2=0x7fdea407c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:48.950 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:47.889+0000 7fdea9600700 1 --2- 172.21.15.39:0/3195753994 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdea407de70 0x7fdea407c2d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fde940098a0 tx=0x7fde94009bb0 comp rx=0 tx=0).stop 2024-03-18T15:58:48.950 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:47.889+0000 7fdea9600700 1 -- 172.21.15.39:0/3195753994 shutdown_connections 2024-03-18T15:58:48.950 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:47.889+0000 7fdea9600700 1 --2- 172.21.15.39:0/3195753994 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdea407de70 0x7fdea407c2d0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:48.950 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:47.889+0000 7fdea9600700 1 -- 172.21.15.39:0/3195753994 >> 172.21.15.39:0/3195753994 conn(0x7fdea4103180 msgr2=0x7fdea4105600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:48.950 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:47.889+0000 7fdea9600700 1 -- 172.21.15.39:0/3195753994 shutdown_connections 2024-03-18T15:58:48.951 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:47.889+0000 7fdea9600700 1 -- 172.21.15.39:0/3195753994 wait complete. 2024-03-18T15:58:48.951 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:47.890+0000 7fdea9600700 1 Processor -- start 2024-03-18T15:58:48.951 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:47.891+0000 7fdea9600700 1 -- start start 2024-03-18T15:58:48.951 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:47.891+0000 7fdea9600700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdea407de70 0x7fdea419ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:48.951 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:47.891+0000 7fdea9600700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fdea410e290 con 0x7fdea407de70 2024-03-18T15:58:48.951 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:47.891+0000 7fdea2a00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdea407de70 0x7fdea419ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:48.951 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:47.891+0000 7fdea2a00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdea407de70 0x7fdea419ddd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:56510/0 (socket says 172.21.15.39:56510) 2024-03-18T15:58:48.951 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:47.891+0000 7fdea2a00700 1 -- 172.21.15.39:0/4205534163 learned_addr learned my addr 172.21.15.39:0/4205534163 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T15:58:48.951 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:47.892+0000 7fdea2a00700 1 -- 172.21.15.39:0/4205534163 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdea419e310 con 0x7fdea407de70 2024-03-18T15:58:48.951 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:47.892+0000 7fdea2a00700 1 --2- 172.21.15.39:0/4205534163 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdea407de70 0x7fdea419ddd0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fde94000c00 tx=0x7fde94011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:48.951 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:47.892+0000 7fde9b400700 1 -- 172.21.15.39:0/4205534163 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fde94003b60 con 0x7fdea407de70 2024-03-18T15:58:48.951 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:47.892+0000 7fde9b400700 1 -- 172.21.15.39:0/4205534163 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde94003d00 con 0x7fdea407de70 2024-03-18T15:58:48.951 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:47.892+0000 7fde9b400700 1 -- 172.21.15.39:0/4205534163 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fde940248a0 con 0x7fdea407de70 2024-03-18T15:58:48.951 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:47.892+0000 7fdea9600700 1 -- 172.21.15.39:0/4205534163 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdea419e5a0 con 0x7fdea407de70 2024-03-18T15:58:48.952 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:47.892+0000 7fdea9600700 1 -- 172.21.15.39:0/4205534163 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdea419ea00 con 0x7fdea407de70 2024-03-18T15:58:48.952 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:47.893+0000 7fdea9600700 1 -- 172.21.15.39:0/4205534163 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdea4063900 con 0x7fdea407de70 2024-03-18T15:58:48.952 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:47.894+0000 7fde9b400700 1 -- 172.21.15.39:0/4205534163 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 12) ==== 46119+0+0 (secure 0 0 0) 0x7fde94022020 con 0x7fdea407de70 2024-03-18T15:58:48.952 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:47.894+0000 7fde9b400700 1 --2- 172.21.15.39:0/4205534163 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fde88038fd0 0x7fde8803b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:48.952 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:47.894+0000 7fde9b400700 1 -- 172.21.15.39:0/4205534163 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fde9404d0a0 con 0x7fdea407de70 2024-03-18T15:58:48.952 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:47.895+0000 7fdea2000700 1 --2- 172.21.15.39:0/4205534163 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fde88038fd0 0x7fde8803b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:48.952 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:47.895+0000 7fdea2000700 1 --2- 172.21.15.39:0/4205534163 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fde88038fd0 0x7fde8803b490 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fde90006fd0 tx=0x7fde90006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:48.952 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:47.900+0000 7fde9b400700 1 -- 172.21.15.39:0/4205534163 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fde94016260 con 0x7fdea407de70 2024-03-18T15:58:48.952 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:48.135+0000 7fdea9600700 1 -- 172.21.15.39:0/4205534163 --> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7fdea4104480 con 0x7fde88038fd0 2024-03-18T15:58:48.952 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:48.230+0000 7fde9b400700 1 -- 172.21.15.39:0/4205534163 <== mgr.14152 v2:172.21.15.39:6800/2749214286 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7fdea4104480 con 0x7fde88038fd0 2024-03-18T15:58:48.952 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:48.234+0000 7fdea9600700 1 -- 172.21.15.39:0/4205534163 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fde88038fd0 msgr2=0x7fde8803b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:48.952 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:48.234+0000 7fdea9600700 1 --2- 172.21.15.39:0/4205534163 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fde88038fd0 0x7fde8803b490 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fde90006fd0 tx=0x7fde90006e40 comp rx=0 tx=0).stop 2024-03-18T15:58:48.952 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:48.234+0000 7fdea9600700 1 -- 172.21.15.39:0/4205534163 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdea407de70 msgr2=0x7fdea419ddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:48.952 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:48.234+0000 7fdea9600700 1 --2- 172.21.15.39:0/4205534163 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdea407de70 0x7fdea419ddd0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fde94000c00 tx=0x7fde94011c10 comp rx=0 tx=0).stop 2024-03-18T15:58:48.953 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:48.234+0000 7fdea9600700 1 -- 172.21.15.39:0/4205534163 shutdown_connections 2024-03-18T15:58:48.953 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:48.234+0000 7fdea9600700 1 --2- 172.21.15.39:0/4205534163 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fde88038fd0 0x7fde8803b490 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:48.953 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:48.234+0000 7fdea9600700 1 --2- 172.21.15.39:0/4205534163 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdea407de70 0x7fdea419ddd0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:48.953 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:48.234+0000 7fdea9600700 1 -- 172.21.15.39:0/4205534163 >> 172.21.15.39:0/4205534163 conn(0x7fdea4103180 msgr2=0x7fdea4103d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:48.953 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:48.234+0000 7fdea9600700 1 -- 172.21.15.39:0/4205534163 shutdown_connections 2024-03-18T15:58:48.953 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:48.234+0000 7fdea9600700 1 -- 172.21.15.39:0/4205534163 wait complete. 2024-03-18T15:58:48.953 INFO:teuthology.orchestra.run.smithi039.stdout:Creating initial admin user... 2024-03-18T15:58:49.806 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:49 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:58:50.823 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$17JwZum33wMAo0zbxO/xvOU5KgeueKl9ZYVhMzhnKv6fWlP7jX08O", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1710777530, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-18T15:58:50.823 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:49.524+0000 7f3f96e00700 1 Processor -- start 2024-03-18T15:58:50.823 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:49.525+0000 7f3f96e00700 1 -- start start 2024-03-18T15:58:50.823 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:49.525+0000 7f3f96e00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3f90107a70 0x7f3f90107e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:50.824 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:49.525+0000 7f3f96e00700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f3f90108460 con 0x7f3f90107a70 2024-03-18T15:58:50.824 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:49.526+0000 7f3f95a00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3f90107a70 0x7f3f90107e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:50.824 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:49.526+0000 7f3f95a00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3f90107a70 0x7f3f90107e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:46812/0 (socket says 172.21.15.39:46812) 2024-03-18T15:58:50.824 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:49.526+0000 7f3f95a00700 1 -- 172.21.15.39:0/2544574594 learned_addr learned my addr 172.21.15.39:0/2544574594 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T15:58:50.824 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:49.526+0000 7f3f95a00700 1 -- 172.21.15.39:0/2544574594 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f90108c70 con 0x7f3f90107a70 2024-03-18T15:58:50.824 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:49.526+0000 7f3f95a00700 1 --2- 172.21.15.39:0/2544574594 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3f90107a70 0x7f3f90107e90 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3f840098a0 tx=0x7f3f84009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cbdbd09c695d68f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:50.824 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:49.527+0000 7f3f8fe00700 1 -- 172.21.15.39:0/2544574594 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3f84004160 con 0x7f3f90107a70 2024-03-18T15:58:50.824 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:49.527+0000 7f3f8fe00700 1 -- 172.21.15.39:0/2544574594 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f84004300 con 0x7f3f90107a70 2024-03-18T15:58:50.824 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:49.527+0000 7f3f8fe00700 1 -- 172.21.15.39:0/2544574594 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3f84011b20 con 0x7f3f90107a70 2024-03-18T15:58:50.824 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:49.527+0000 7f3f96e00700 1 -- 172.21.15.39:0/2544574594 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3f90107a70 msgr2=0x7f3f90107e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:50.824 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:49.527+0000 7f3f96e00700 1 --2- 172.21.15.39:0/2544574594 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3f90107a70 0x7f3f90107e90 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3f840098a0 tx=0x7f3f84009bb0 comp rx=0 tx=0).stop 2024-03-18T15:58:50.824 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:49.528+0000 7f3f96e00700 1 -- 172.21.15.39:0/2544574594 shutdown_connections 2024-03-18T15:58:50.824 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:49.528+0000 7f3f96e00700 1 --2- 172.21.15.39:0/2544574594 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3f90107a70 0x7f3f90107e90 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:50.825 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:49.528+0000 7f3f96e00700 1 -- 172.21.15.39:0/2544574594 >> 172.21.15.39:0/2544574594 conn(0x7f3f90103050 msgr2=0x7f3f90105470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:50.825 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:49.528+0000 7f3f96e00700 1 -- 172.21.15.39:0/2544574594 shutdown_connections 2024-03-18T15:58:50.825 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:49.528+0000 7f3f96e00700 1 -- 172.21.15.39:0/2544574594 wait complete. 2024-03-18T15:58:50.825 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:49.529+0000 7f3f96e00700 1 Processor -- start 2024-03-18T15:58:50.825 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:49.529+0000 7f3f96e00700 1 -- start start 2024-03-18T15:58:50.825 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:49.529+0000 7f3f96e00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3f90107a70 0x7f3f9007c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:50.825 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:49.529+0000 7f3f96e00700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f3f9010c020 con 0x7f3f90107a70 2024-03-18T15:58:50.825 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:49.529+0000 7f3f95a00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3f90107a70 0x7f3f9007c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:50.825 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:49.530+0000 7f3f95a00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3f90107a70 0x7f3f9007c9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:46826/0 (socket says 172.21.15.39:46826) 2024-03-18T15:58:50.825 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:49.530+0000 7f3f95a00700 1 -- 172.21.15.39:0/891278591 learned_addr learned my addr 172.21.15.39:0/891278591 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T15:58:50.825 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:49.530+0000 7f3f95a00700 1 -- 172.21.15.39:0/891278591 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f9007cf30 con 0x7f3f90107a70 2024-03-18T15:58:50.825 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:49.530+0000 7f3f95a00700 1 --2- 172.21.15.39:0/891278591 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3f90107a70 0x7f3f9007c9f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3f84000c00 tx=0x7f3f84011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:50.825 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:49.530+0000 7f3f8e000700 1 -- 172.21.15.39:0/891278591 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3f84003b60 con 0x7f3f90107a70 2024-03-18T15:58:50.825 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:49.530+0000 7f3f8e000700 1 -- 172.21.15.39:0/891278591 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f8401a910 con 0x7f3f90107a70 2024-03-18T15:58:50.825 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:49.530+0000 7f3f8e000700 1 -- 172.21.15.39:0/891278591 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3f84024930 con 0x7f3f90107a70 2024-03-18T15:58:50.826 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:49.531+0000 7f3f96e00700 1 -- 172.21.15.39:0/891278591 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f9007b120 con 0x7f3f90107a70 2024-03-18T15:58:50.826 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:49.531+0000 7f3f96e00700 1 -- 172.21.15.39:0/891278591 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f9007b5e0 con 0x7f3f90107a70 2024-03-18T15:58:50.826 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:49.532+0000 7f3f8e000700 1 -- 172.21.15.39:0/891278591 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 12) ==== 46119+0+0 (secure 0 0 0) 0x7f3f84022020 con 0x7f3f90107a70 2024-03-18T15:58:50.826 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:49.532+0000 7f3f96e00700 1 -- 172.21.15.39:0/891278591 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f90063900 con 0x7f3f90107a70 2024-03-18T15:58:50.826 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:49.532+0000 7f3f8e000700 1 --2- 172.21.15.39:0/891278591 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3f78038fd0 0x7f3f7803b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:50.826 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:49.532+0000 7f3f8e000700 1 -- 172.21.15.39:0/891278591 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f3f8404d1e0 con 0x7f3f90107a70 2024-03-18T15:58:50.827 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:49.533+0000 7f3f95000700 1 --2- 172.21.15.39:0/891278591 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3f78038fd0 0x7f3f7803b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:50.827 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:49.533+0000 7f3f95000700 1 --2- 172.21.15.39:0/891278591 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3f78038fd0 0x7f3f7803b490 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3f7c006fd0 tx=0x7f3f7c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:50.827 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:49.539+0000 7f3f8e000700 1 -- 172.21.15.39:0/891278591 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3f8400b3e0 con 0x7f3f90107a70 2024-03-18T15:58:50.827 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:49.804+0000 7f3f96e00700 1 -- 172.21.15.39:0/891278591 --> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7f3f9007bd50 con 0x7f3f78038fd0 2024-03-18T15:58:50.827 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:50.096+0000 7f3f8e000700 1 -- 172.21.15.39:0/891278591 <== mgr.14152 v2:172.21.15.39:6800/2749214286 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7f3f9007bd50 con 0x7f3f78038fd0 2024-03-18T15:58:50.827 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:50.098+0000 7f3f96e00700 1 -- 172.21.15.39:0/891278591 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3f78038fd0 msgr2=0x7f3f7803b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:50.827 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:50.098+0000 7f3f96e00700 1 --2- 172.21.15.39:0/891278591 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3f78038fd0 0x7f3f7803b490 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3f7c006fd0 tx=0x7f3f7c006e40 comp rx=0 tx=0).stop 2024-03-18T15:58:50.827 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:50.098+0000 7f3f96e00700 1 -- 172.21.15.39:0/891278591 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3f90107a70 msgr2=0x7f3f9007c9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:50.827 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:50.098+0000 7f3f96e00700 1 --2- 172.21.15.39:0/891278591 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3f90107a70 0x7f3f9007c9f0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3f84000c00 tx=0x7f3f84011c10 comp rx=0 tx=0).stop 2024-03-18T15:58:50.827 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:50.098+0000 7f3f96e00700 1 -- 172.21.15.39:0/891278591 shutdown_connections 2024-03-18T15:58:50.827 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:50.098+0000 7f3f96e00700 1 --2- 172.21.15.39:0/891278591 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3f78038fd0 0x7f3f7803b490 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:50.827 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:50.098+0000 7f3f96e00700 1 --2- 172.21.15.39:0/891278591 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3f90107a70 0x7f3f9007c9f0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:50.828 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:50.098+0000 7f3f96e00700 1 -- 172.21.15.39:0/891278591 >> 172.21.15.39:0/891278591 conn(0x7f3f90103050 msgr2=0x7f3f90103af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:50.828 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:50.099+0000 7f3f96e00700 1 -- 172.21.15.39:0/891278591 shutdown_connections 2024-03-18T15:58:50.828 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:50.099+0000 7f3f96e00700 1 -- 172.21.15.39:0/891278591 wait complete. 2024-03-18T15:58:50.828 INFO:teuthology.orchestra.run.smithi039.stdout:Fetching dashboard port number... 2024-03-18T15:58:51.418 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:51 smithi039 ceph-mon[31721]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:58:51.418 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:51 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:58:51.418 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:51 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:58:52.220 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:52 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/2509286430' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-18T15:58:52.483 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stdout 8443 2024-03-18T15:58:52.484 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:51.415+0000 7f7fd7200700 1 Processor -- start 2024-03-18T15:58:52.484 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:51.415+0000 7f7fd7200700 1 -- start start 2024-03-18T15:58:52.484 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:51.416+0000 7f7fd7200700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7fd807cce0 0x7f7fd807b140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:52.484 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:51.416+0000 7f7fd7200700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f7fd807b710 con 0x7f7fd807cce0 2024-03-18T15:58:52.484 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:51.416+0000 7f7fd5e00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7fd807cce0 0x7f7fd807b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:52.484 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:51.416+0000 7f7fd5e00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7fd807cce0 0x7f7fd807b140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:46842/0 (socket says 172.21.15.39:46842) 2024-03-18T15:58:52.487 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:51.416+0000 7f7fd5e00700 1 -- 172.21.15.39:0/2426478938 learned_addr learned my addr 172.21.15.39:0/2426478938 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T15:58:52.487 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:51.417+0000 7f7fd5e00700 1 -- 172.21.15.39:0/2426478938 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fd807b890 con 0x7f7fd807cce0 2024-03-18T15:58:52.487 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:51.417+0000 7f7fd5e00700 1 --2- 172.21.15.39:0/2426478938 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7fd807cce0 0x7f7fd807b140 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7fc80098a0 tx=0x7f7fc8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55c735823bf89dcf server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:52.487 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:51.418+0000 7f7fd4a00700 1 -- 172.21.15.39:0/2426478938 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7fc8004160 con 0x7f7fd807cce0 2024-03-18T15:58:52.487 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:51.418+0000 7f7fd4a00700 1 -- 172.21.15.39:0/2426478938 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7fc8004300 con 0x7f7fd807cce0 2024-03-18T15:58:52.488 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:51.418+0000 7f7fd4a00700 1 -- 172.21.15.39:0/2426478938 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7fc8011b20 con 0x7f7fd807cce0 2024-03-18T15:58:52.488 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:51.418+0000 7f7fd7200700 1 -- 172.21.15.39:0/2426478938 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7fd807cce0 msgr2=0x7f7fd807b140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:52.488 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:51.418+0000 7f7fd7200700 1 --2- 172.21.15.39:0/2426478938 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7fd807cce0 0x7f7fd807b140 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7fc80098a0 tx=0x7f7fc8009bb0 comp rx=0 tx=0).stop 2024-03-18T15:58:52.488 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:51.419+0000 7f7fd7200700 1 -- 172.21.15.39:0/2426478938 shutdown_connections 2024-03-18T15:58:52.488 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:51.419+0000 7f7fd7200700 1 --2- 172.21.15.39:0/2426478938 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7fd807cce0 0x7f7fd807b140 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:52.488 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:51.419+0000 7f7fd7200700 1 -- 172.21.15.39:0/2426478938 >> 172.21.15.39:0/2426478938 conn(0x7f7fd8103050 msgr2=0x7f7fd8105470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:52.488 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:51.419+0000 7f7fd7200700 1 -- 172.21.15.39:0/2426478938 shutdown_connections 2024-03-18T15:58:52.488 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:51.419+0000 7f7fd7200700 1 -- 172.21.15.39:0/2426478938 wait complete. 2024-03-18T15:58:52.488 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:51.420+0000 7f7fd7200700 1 Processor -- start 2024-03-18T15:58:52.488 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:51.420+0000 7f7fd7200700 1 -- start start 2024-03-18T15:58:52.488 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:51.420+0000 7f7fd7200700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7fd807cce0 0x7f7fd819db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:52.488 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:51.420+0000 7f7fd7200700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f7fd810e0b0 con 0x7f7fd807cce0 2024-03-18T15:58:52.488 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:51.420+0000 7f7fd5e00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7fd807cce0 0x7f7fd819db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:52.488 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:51.420+0000 7f7fd5e00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7fd807cce0 0x7f7fd819db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:46854/0 (socket says 172.21.15.39:46854) 2024-03-18T15:58:52.489 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:51.421+0000 7f7fd5e00700 1 -- 172.21.15.39:0/2509286430 learned_addr learned my addr 172.21.15.39:0/2509286430 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T15:58:52.489 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:51.421+0000 7f7fd5e00700 1 -- 172.21.15.39:0/2509286430 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fd819e0d0 con 0x7f7fd807cce0 2024-03-18T15:58:52.489 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:51.421+0000 7f7fd5e00700 1 --2- 172.21.15.39:0/2509286430 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7fd807cce0 0x7f7fd819db90 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7fc8011b60 tx=0x7f7fc8011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:52.489 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:51.421+0000 7f7fcea00700 1 -- 172.21.15.39:0/2509286430 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7fc8003b60 con 0x7f7fd807cce0 2024-03-18T15:58:52.492 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:51.421+0000 7f7fcea00700 1 -- 172.21.15.39:0/2509286430 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7fc801a910 con 0x7f7fd807cce0 2024-03-18T15:58:52.492 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:51.421+0000 7f7fd7200700 1 -- 172.21.15.39:0/2509286430 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fd819e360 con 0x7f7fd807cce0 2024-03-18T15:58:52.492 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:51.421+0000 7f7fcea00700 1 -- 172.21.15.39:0/2509286430 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7fc8024930 con 0x7f7fd807cce0 2024-03-18T15:58:52.492 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:51.422+0000 7f7fd7200700 1 -- 172.21.15.39:0/2509286430 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fd819e7c0 con 0x7f7fd807cce0 2024-03-18T15:58:52.492 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:51.423+0000 7f7fd7200700 1 -- 172.21.15.39:0/2509286430 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7fd8063900 con 0x7f7fd807cce0 2024-03-18T15:58:52.492 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:51.423+0000 7f7fcea00700 1 -- 172.21.15.39:0/2509286430 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 12) ==== 46119+0+0 (secure 0 0 0) 0x7f7fc8022020 con 0x7f7fd807cce0 2024-03-18T15:58:52.492 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:51.423+0000 7f7fcea00700 1 --2- 172.21.15.39:0/2509286430 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f7fbc038fd0 0x7f7fbc03b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:52.492 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:51.424+0000 7f7fcea00700 1 -- 172.21.15.39:0/2509286430 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f7fc804d230 con 0x7f7fd807cce0 2024-03-18T15:58:52.492 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:51.424+0000 7f7fd5400700 1 --2- 172.21.15.39:0/2509286430 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f7fbc038fd0 0x7f7fbc03b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:52.492 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:51.424+0000 7f7fd5400700 1 --2- 172.21.15.39:0/2509286430 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f7fbc038fd0 0x7f7fbc03b490 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f7fc4006fd0 tx=0x7f7fc4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:52.492 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:51.429+0000 7f7fcea00700 1 -- 172.21.15.39:0/2509286430 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7fc80212c0 con 0x7f7fd807cce0 2024-03-18T15:58:52.492 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:51.683+0000 7f7fd7200700 1 -- 172.21.15.39:0/2509286430 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7f7fd81a1250 con 0x7f7fd807cce0 2024-03-18T15:58:52.493 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:51.684+0000 7f7fcea00700 1 -- 172.21.15.39:0/2509286430 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7f7fc802a2a0 con 0x7f7fd807cce0 2024-03-18T15:58:52.493 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:51.687+0000 7f7fd7200700 1 -- 172.21.15.39:0/2509286430 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f7fbc038fd0 msgr2=0x7f7fbc03b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:52.493 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:51.687+0000 7f7fd7200700 1 --2- 172.21.15.39:0/2509286430 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f7fbc038fd0 0x7f7fbc03b490 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f7fc4006fd0 tx=0x7f7fc4006e40 comp rx=0 tx=0).stop 2024-03-18T15:58:52.493 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:51.687+0000 7f7fd7200700 1 -- 172.21.15.39:0/2509286430 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7fd807cce0 msgr2=0x7f7fd819db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:52.493 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:51.687+0000 7f7fd7200700 1 --2- 172.21.15.39:0/2509286430 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7fd807cce0 0x7f7fd819db90 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7fc8011b60 tx=0x7f7fc8011c40 comp rx=0 tx=0).stop 2024-03-18T15:58:52.493 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:51.687+0000 7f7fd7200700 1 -- 172.21.15.39:0/2509286430 shutdown_connections 2024-03-18T15:58:52.493 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:51.687+0000 7f7fd7200700 1 --2- 172.21.15.39:0/2509286430 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f7fbc038fd0 0x7f7fbc03b490 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:52.493 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:51.687+0000 7f7fd7200700 1 --2- 172.21.15.39:0/2509286430 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7fd807cce0 0x7f7fd819db90 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:52.493 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:51.687+0000 7f7fd7200700 1 -- 172.21.15.39:0/2509286430 >> 172.21.15.39:0/2509286430 conn(0x7f7fd8103050 msgr2=0x7f7fd8103af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:52.493 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:51.688+0000 7f7fd7200700 1 -- 172.21.15.39:0/2509286430 shutdown_connections 2024-03-18T15:58:52.493 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:51.688+0000 7f7fd7200700 1 -- 172.21.15.39:0/2509286430 wait complete. 2024-03-18T15:58:52.518 INFO:teuthology.orchestra.run.smithi039.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-18T15:58:52.518 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stdout disabled 2024-03-18T15:58:52.544 INFO:teuthology.orchestra.run.smithi039.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-18T15:58:52.544 INFO:teuthology.orchestra.run.smithi039.stdout:systemctl: stdout inactive 2024-03-18T15:58:52.545 INFO:teuthology.orchestra.run.smithi039.stdout:firewalld.service is not enabled 2024-03-18T15:58:52.545 INFO:teuthology.orchestra.run.smithi039.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-18T15:58:52.552 INFO:teuthology.orchestra.run.smithi039.stdout:Ceph Dashboard is now available at: 2024-03-18T15:58:52.552 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T15:58:52.552 INFO:teuthology.orchestra.run.smithi039.stdout: URL: https://smithi039.front.sepia.ceph.com:8443/ 2024-03-18T15:58:52.552 INFO:teuthology.orchestra.run.smithi039.stdout: User: admin 2024-03-18T15:58:52.552 INFO:teuthology.orchestra.run.smithi039.stdout: Password: m96ghb50r7 2024-03-18T15:58:52.552 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T15:58:52.553 INFO:teuthology.orchestra.run.smithi039.stdout:Saving cluster configuration to /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/config directory 2024-03-18T15:58:52.555 INFO:teuthology.orchestra.run.smithi039.stdout:Enabling autotune for osd_memory_target 2024-03-18T15:58:53.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:53 smithi039 ceph-mon[31721]: mgrmap e13: x(active, since 6s) 2024-03-18T15:58:55.057 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:54.092+0000 7fb87ee00700 1 Processor -- start 2024-03-18T15:58:55.058 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:54.092+0000 7fb87ee00700 1 -- start start 2024-03-18T15:58:55.058 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:54.092+0000 7fb87ee00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb87807cd00 0x7fb87807b160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:55.058 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:54.092+0000 7fb87ee00700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fb87807b730 con 0x7fb87807cd00 2024-03-18T15:58:55.058 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:54.093+0000 7fb877e00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb87807cd00 0x7fb87807b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:55.058 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:54.093+0000 7fb877e00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb87807cd00 0x7fb87807b160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:46856/0 (socket says 172.21.15.39:46856) 2024-03-18T15:58:55.058 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:54.093+0000 7fb877e00700 1 -- 172.21.15.39:0/2060556144 learned_addr learned my addr 172.21.15.39:0/2060556144 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T15:58:55.058 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:54.093+0000 7fb877e00700 1 -- 172.21.15.39:0/2060556144 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb87807b8b0 con 0x7fb87807cd00 2024-03-18T15:58:55.058 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:54.094+0000 7fb877e00700 1 --2- 172.21.15.39:0/2060556144 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb87807cd00 0x7fb87807b160 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb8680098a0 tx=0x7fb868009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eaffbe184c36fe1f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:55.058 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:54.094+0000 7fb876a00700 1 -- 172.21.15.39:0/2060556144 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb868004160 con 0x7fb87807cd00 2024-03-18T15:58:55.058 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:54.094+0000 7fb876a00700 1 -- 172.21.15.39:0/2060556144 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb868004300 con 0x7fb87807cd00 2024-03-18T15:58:55.058 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:54.094+0000 7fb876a00700 1 -- 172.21.15.39:0/2060556144 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb868011b20 con 0x7fb87807cd00 2024-03-18T15:58:55.058 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:54.095+0000 7fb87ee00700 1 -- 172.21.15.39:0/2060556144 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb87807cd00 msgr2=0x7fb87807b160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:55.059 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:54.095+0000 7fb87ee00700 1 --2- 172.21.15.39:0/2060556144 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb87807cd00 0x7fb87807b160 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb8680098a0 tx=0x7fb868009bb0 comp rx=0 tx=0).stop 2024-03-18T15:58:55.059 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:54.095+0000 7fb87ee00700 1 -- 172.21.15.39:0/2060556144 shutdown_connections 2024-03-18T15:58:55.059 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:54.095+0000 7fb87ee00700 1 --2- 172.21.15.39:0/2060556144 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb87807cd00 0x7fb87807b160 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:55.059 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:54.095+0000 7fb87ee00700 1 -- 172.21.15.39:0/2060556144 >> 172.21.15.39:0/2060556144 conn(0x7fb8781031e0 msgr2=0x7fb878105600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:55.059 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:54.095+0000 7fb87ee00700 1 -- 172.21.15.39:0/2060556144 shutdown_connections 2024-03-18T15:58:55.059 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:54.095+0000 7fb87ee00700 1 -- 172.21.15.39:0/2060556144 wait complete. 2024-03-18T15:58:55.059 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:54.096+0000 7fb87ee00700 1 Processor -- start 2024-03-18T15:58:55.059 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:54.096+0000 7fb87ee00700 1 -- start start 2024-03-18T15:58:55.059 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:54.097+0000 7fb87ee00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb87807cd00 0x7fb87819dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:55.059 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:54.097+0000 7fb87ee00700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fb87810e2a0 con 0x7fb87807cd00 2024-03-18T15:58:55.059 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:54.097+0000 7fb877e00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb87807cd00 0x7fb87819dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:55.059 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:54.097+0000 7fb877e00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb87807cd00 0x7fb87819dd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:46870/0 (socket says 172.21.15.39:46870) 2024-03-18T15:58:55.059 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:54.097+0000 7fb877e00700 1 -- 172.21.15.39:0/3766676549 learned_addr learned my addr 172.21.15.39:0/3766676549 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T15:58:55.060 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:54.097+0000 7fb877e00700 1 -- 172.21.15.39:0/3766676549 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb87819e270 con 0x7fb87807cd00 2024-03-18T15:58:55.060 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:54.098+0000 7fb877e00700 1 --2- 172.21.15.39:0/3766676549 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb87807cd00 0x7fb87819dd30 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb868011b60 tx=0x7fb868011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:55.060 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:54.098+0000 7fb874c00700 1 -- 172.21.15.39:0/3766676549 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb868003b60 con 0x7fb87807cd00 2024-03-18T15:58:55.060 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:54.098+0000 7fb874c00700 1 -- 172.21.15.39:0/3766676549 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb86801a910 con 0x7fb87807cd00 2024-03-18T15:58:55.060 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:54.098+0000 7fb87ee00700 1 -- 172.21.15.39:0/3766676549 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb87819e500 con 0x7fb87807cd00 2024-03-18T15:58:55.060 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:54.098+0000 7fb874c00700 1 -- 172.21.15.39:0/3766676549 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb868024930 con 0x7fb87807cd00 2024-03-18T15:58:55.060 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:54.098+0000 7fb87ee00700 1 -- 172.21.15.39:0/3766676549 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb87819e960 con 0x7fb87807cd00 2024-03-18T15:58:55.060 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:54.099+0000 7fb87ee00700 1 -- 172.21.15.39:0/3766676549 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb878063900 con 0x7fb87807cd00 2024-03-18T15:58:55.060 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:54.100+0000 7fb874c00700 1 -- 172.21.15.39:0/3766676549 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7fb868022020 con 0x7fb87807cd00 2024-03-18T15:58:55.060 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:54.100+0000 7fb874c00700 1 --2- 172.21.15.39:0/3766676549 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb85c039070 0x7fb85c03b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:55.060 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:54.100+0000 7fb874c00700 1 -- 172.21.15.39:0/3766676549 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fb86804d2c0 con 0x7fb87807cd00 2024-03-18T15:58:55.060 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:54.101+0000 7fb877400700 1 --2- 172.21.15.39:0/3766676549 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb85c039070 0x7fb85c03b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:55.060 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:54.101+0000 7fb877400700 1 --2- 172.21.15.39:0/3766676549 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb85c039070 0x7fb85c03b530 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb864006fd0 tx=0x7fb864006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:55.060 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:54.106+0000 7fb874c00700 1 -- 172.21.15.39:0/3766676549 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb86800b3e0 con 0x7fb87807cd00 2024-03-18T15:58:55.061 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:54.330+0000 7fb87ee00700 1 -- 172.21.15.39:0/3766676549 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7fb8781a13d0 con 0x7fb87807cd00 2024-03-18T15:58:55.061 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:54.331+0000 7fb874c00700 1 -- 172.21.15.39:0/3766676549 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7fb86802dde0 con 0x7fb87807cd00 2024-03-18T15:58:55.061 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:54.334+0000 7fb87ee00700 1 -- 172.21.15.39:0/3766676549 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb85c039070 msgr2=0x7fb85c03b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:55.061 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:54.334+0000 7fb87ee00700 1 --2- 172.21.15.39:0/3766676549 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb85c039070 0x7fb85c03b530 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb864006fd0 tx=0x7fb864006e40 comp rx=0 tx=0).stop 2024-03-18T15:58:55.061 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:54.334+0000 7fb87ee00700 1 -- 172.21.15.39:0/3766676549 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb87807cd00 msgr2=0x7fb87819dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:55.061 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:54.334+0000 7fb87ee00700 1 --2- 172.21.15.39:0/3766676549 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb87807cd00 0x7fb87819dd30 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb868011b60 tx=0x7fb868011c40 comp rx=0 tx=0).stop 2024-03-18T15:58:55.061 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:54.334+0000 7fb87ee00700 1 -- 172.21.15.39:0/3766676549 shutdown_connections 2024-03-18T15:58:55.061 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:54.334+0000 7fb87ee00700 1 --2- 172.21.15.39:0/3766676549 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb85c039070 0x7fb85c03b530 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:55.061 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:54.334+0000 7fb87ee00700 1 --2- 172.21.15.39:0/3766676549 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb87807cd00 0x7fb87819dd30 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:55.061 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:54.334+0000 7fb87ee00700 1 -- 172.21.15.39:0/3766676549 >> 172.21.15.39:0/3766676549 conn(0x7fb8781031e0 msgr2=0x7fb878103cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:55.061 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:54.334+0000 7fb87ee00700 1 -- 172.21.15.39:0/3766676549 shutdown_connections 2024-03-18T15:58:55.061 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:54.334+0000 7fb87ee00700 1 -- 172.21.15.39:0/3766676549 wait complete. 2024-03-18T15:58:56.700 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:55.660+0000 7f422d600700 1 Processor -- start 2024-03-18T15:58:56.700 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:55.660+0000 7f422d600700 1 -- start start 2024-03-18T15:58:56.700 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:55.661+0000 7f422d600700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4228109dd0 0x7f422810a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:56.700 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:55.661+0000 7f422d600700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f422810a7c0 con 0x7f4228109dd0 2024-03-18T15:58:56.702 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:55.661+0000 7f4226a00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4228109dd0 0x7f422810a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:56.702 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:55.661+0000 7f4226a00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4228109dd0 0x7f422810a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:46874/0 (socket says 172.21.15.39:46874) 2024-03-18T15:58:56.702 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:55.661+0000 7f4226a00700 1 -- 172.21.15.39:0/2501960702 learned_addr learned my addr 172.21.15.39:0/2501960702 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T15:58:56.702 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:55.662+0000 7f4226a00700 1 -- 172.21.15.39:0/2501960702 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f422810afd0 con 0x7f4228109dd0 2024-03-18T15:58:56.702 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:55.662+0000 7f4226a00700 1 --2- 172.21.15.39:0/2501960702 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4228109dd0 0x7f422810a1f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f42180098a0 tx=0x7f4218009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76a0200ebab2b343 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:56.702 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:55.662+0000 7f4225600700 1 -- 172.21.15.39:0/2501960702 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4218004160 con 0x7f4228109dd0 2024-03-18T15:58:56.702 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:55.662+0000 7f4225600700 1 -- 172.21.15.39:0/2501960702 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4218004300 con 0x7f4228109dd0 2024-03-18T15:58:56.702 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:55.662+0000 7f4225600700 1 -- 172.21.15.39:0/2501960702 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4218011b20 con 0x7f4228109dd0 2024-03-18T15:58:56.709 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:55.663+0000 7f422d600700 1 -- 172.21.15.39:0/2501960702 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4228109dd0 msgr2=0x7f422810a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:56.709 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:55.663+0000 7f422d600700 1 --2- 172.21.15.39:0/2501960702 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4228109dd0 0x7f422810a1f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f42180098a0 tx=0x7f4218009bb0 comp rx=0 tx=0).stop 2024-03-18T15:58:56.709 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:55.663+0000 7f422d600700 1 -- 172.21.15.39:0/2501960702 shutdown_connections 2024-03-18T15:58:56.709 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:55.663+0000 7f422d600700 1 --2- 172.21.15.39:0/2501960702 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4228109dd0 0x7f422810a1f0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:56.710 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:55.663+0000 7f422d600700 1 -- 172.21.15.39:0/2501960702 >> 172.21.15.39:0/2501960702 conn(0x7f422807d1e0 msgr2=0x7f422807d5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:56.710 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:55.663+0000 7f422d600700 1 -- 172.21.15.39:0/2501960702 shutdown_connections 2024-03-18T15:58:56.710 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:55.663+0000 7f422d600700 1 -- 172.21.15.39:0/2501960702 wait complete. 2024-03-18T15:58:56.710 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:55.664+0000 7f422d600700 1 Processor -- start 2024-03-18T15:58:56.710 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:55.664+0000 7f422d600700 1 -- start start 2024-03-18T15:58:56.710 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:55.665+0000 7f422d600700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4228109dd0 0x7f422819dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:56.710 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:55.665+0000 7f422d600700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f422810e380 con 0x7f4228109dd0 2024-03-18T15:58:56.710 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:55.665+0000 7f4226a00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4228109dd0 0x7f422819dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:56.710 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:55.665+0000 7f4226a00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4228109dd0 0x7f422819dd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:46884/0 (socket says 172.21.15.39:46884) 2024-03-18T15:58:56.710 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:55.665+0000 7f4226a00700 1 -- 172.21.15.39:0/2849373666 learned_addr learned my addr 172.21.15.39:0/2849373666 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T15:58:56.710 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:55.665+0000 7f4226a00700 1 -- 172.21.15.39:0/2849373666 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f422819e260 con 0x7f4228109dd0 2024-03-18T15:58:56.711 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:55.666+0000 7f4226a00700 1 --2- 172.21.15.39:0/2849373666 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4228109dd0 0x7f422819dd20 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4218000c00 tx=0x7f4218011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:56.711 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:55.666+0000 7f421f400700 1 -- 172.21.15.39:0/2849373666 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4218003b60 con 0x7f4228109dd0 2024-03-18T15:58:56.711 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:55.666+0000 7f421f400700 1 -- 172.21.15.39:0/2849373666 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f421801a910 con 0x7f4228109dd0 2024-03-18T15:58:56.711 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:55.666+0000 7f421f400700 1 -- 172.21.15.39:0/2849373666 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4218024930 con 0x7f4228109dd0 2024-03-18T15:58:56.711 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:55.666+0000 7f422d600700 1 -- 172.21.15.39:0/2849373666 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f422819e4f0 con 0x7f4228109dd0 2024-03-18T15:58:56.711 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:55.666+0000 7f422d600700 1 -- 172.21.15.39:0/2849373666 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f422819e950 con 0x7f4228109dd0 2024-03-18T15:58:56.711 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:55.667+0000 7f421f400700 1 -- 172.21.15.39:0/2849373666 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f4218022020 con 0x7f4228109dd0 2024-03-18T15:58:56.711 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:55.667+0000 7f422d600700 1 -- 172.21.15.39:0/2849373666 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4228063900 con 0x7f4228109dd0 2024-03-18T15:58:56.711 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:55.668+0000 7f421f400700 1 --2- 172.21.15.39:0/2849373666 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f420c039070 0x7f420c03b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:58:56.711 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:55.668+0000 7f421f400700 1 -- 172.21.15.39:0/2849373666 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f421804d1f0 con 0x7f4228109dd0 2024-03-18T15:58:56.711 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:55.668+0000 7f4226000700 1 --2- 172.21.15.39:0/2849373666 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f420c039070 0x7f420c03b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:58:56.711 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:55.669+0000 7f4226000700 1 --2- 172.21.15.39:0/2849373666 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f420c039070 0x7f420c03b530 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4214006fd0 tx=0x7f4214006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:58:56.712 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:55.673+0000 7f421f400700 1 -- 172.21.15.39:0/2849373666 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f421801ab70 con 0x7f4228109dd0 2024-03-18T15:58:56.712 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:56.022+0000 7f422d600700 1 -- 172.21.15.39:0/2849373666 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7f42281a13b0 con 0x7f4228109dd0 2024-03-18T15:58:56.712 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:56.028+0000 7f421f400700 1 -- 172.21.15.39:0/2849373666 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v27)=0 set mgr/dashboard/cluster/status v27) ==== 153+0+0 (secure 0 0 0) 0x7f421802d3f0 con 0x7f4228109dd0 2024-03-18T15:58:56.712 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:56.031+0000 7f422d600700 1 -- 172.21.15.39:0/2849373666 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f420c039070 msgr2=0x7f420c03b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:56.712 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:56.031+0000 7f422d600700 1 --2- 172.21.15.39:0/2849373666 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f420c039070 0x7f420c03b530 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f4214006fd0 tx=0x7f4214006e40 comp rx=0 tx=0).stop 2024-03-18T15:58:56.712 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:56.031+0000 7f422d600700 1 -- 172.21.15.39:0/2849373666 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4228109dd0 msgr2=0x7f422819dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:58:56.712 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:56.031+0000 7f422d600700 1 --2- 172.21.15.39:0/2849373666 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4228109dd0 0x7f422819dd20 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4218000c00 tx=0x7f4218011c10 comp rx=0 tx=0).stop 2024-03-18T15:58:56.712 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:56.031+0000 7f422d600700 1 -- 172.21.15.39:0/2849373666 shutdown_connections 2024-03-18T15:58:56.712 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:56.031+0000 7f422d600700 1 --2- 172.21.15.39:0/2849373666 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f420c039070 0x7f420c03b530 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:56.712 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:56.031+0000 7f422d600700 1 --2- 172.21.15.39:0/2849373666 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4228109dd0 0x7f422819dd20 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:58:56.712 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:56.031+0000 7f422d600700 1 -- 172.21.15.39:0/2849373666 >> 172.21.15.39:0/2849373666 conn(0x7f422807d1e0 msgr2=0x7f4228107460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:58:56.712 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:56.031+0000 7f422d600700 1 -- 172.21.15.39:0/2849373666 shutdown_connections 2024-03-18T15:58:56.712 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr 2024-03-18T15:58:56.032+0000 7f422d600700 1 -- 172.21.15.39:0/2849373666 wait complete. 2024-03-18T15:58:56.713 INFO:teuthology.orchestra.run.smithi039.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-18T15:58:56.713 INFO:teuthology.orchestra.run.smithi039.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-18T15:58:56.713 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T15:58:56.713 INFO:teuthology.orchestra.run.smithi039.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-18T15:58:56.713 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T15:58:56.713 INFO:teuthology.orchestra.run.smithi039.stdout:Or, if you are only running a single cluster on this host: 2024-03-18T15:58:56.713 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T15:58:56.713 INFO:teuthology.orchestra.run.smithi039.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-18T15:58:56.713 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T15:58:56.713 INFO:teuthology.orchestra.run.smithi039.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-18T15:58:56.713 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T15:58:56.713 INFO:teuthology.orchestra.run.smithi039.stdout: ceph telemetry on 2024-03-18T15:58:56.713 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T15:58:56.713 INFO:teuthology.orchestra.run.smithi039.stdout:For more information see: 2024-03-18T15:58:56.713 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T15:58:56.714 INFO:teuthology.orchestra.run.smithi039.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-18T15:58:56.714 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T15:58:56.714 INFO:teuthology.orchestra.run.smithi039.stdout:Bootstrap complete. 2024-03-18T15:58:56.820 INFO:tasks.cephadm:Fetching config... 2024-03-18T15:58:56.820 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-03-18T15:58:56.820 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-18T15:58:56.907 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-18T15:58:56.907 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-03-18T15:58:56.907 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-18T15:58:56.995 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-18T15:58:56.996 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-03-18T15:58:56.996 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-18T15:58:57.130 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-18T15:58:57.130 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-03-18T15:58:57.130 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-18T15:58:57.222 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-18T15:58:57.223 DEBUG:teuthology.orchestra.run.smithi039:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJK+CTQQ3Wg71D32RPxs4NoPmY8FpotvbTS2bAm99J85Gi9xX+K5lXncSe03PD7evIDOV7mIhbq6TzF2WFEbCoxGehisf1+fprn5BZrquL3L17pdY59T6aVxSnQjYOUNmq3kPNHBg6OjyT0ePvWVtyjyLdbG4ImaUivgMwv2w1L2wUSyGNB2/lBhocZjLxbzQy3atQHfT/llEt2G9TffPCk+7wAM3K+IkHwhwwaoh9qj9nLHTLroOmIe9uKZxwa4ZTshKJBLj89l2+Cm2mhmNsBB7i0AAYSP5QVBIdzjN9Sw/66Kxrg3aZobJt9v6lD0MndegLgPcuCJT5cwbWtqc/bWksPGtUdfqtIzcbIHTUYDqJcw74ZYAeOV2vZeGdMDi2dE++qE2d8QekhdFsUSX8ABzsaLljIZhy1H2eAU9dMTAvyvdAKCDYfajBXXJXXCdUy99c3rUmaf5lx51mwz4sccliZLFkqvsRkBF17SrewDxLSLWJ0k7Hq8eydXPMvKs= ceph-1892d8fc-e540-11ee-95c9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-18T15:58:57.300 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:58:57 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/2849373666' entity='client.admin' 2024-03-18T15:58:57.355 INFO:teuthology.orchestra.run.smithi039.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJK+CTQQ3Wg71D32RPxs4NoPmY8FpotvbTS2bAm99J85Gi9xX+K5lXncSe03PD7evIDOV7mIhbq6TzF2WFEbCoxGehisf1+fprn5BZrquL3L17pdY59T6aVxSnQjYOUNmq3kPNHBg6OjyT0ePvWVtyjyLdbG4ImaUivgMwv2w1L2wUSyGNB2/lBhocZjLxbzQy3atQHfT/llEt2G9TffPCk+7wAM3K+IkHwhwwaoh9qj9nLHTLroOmIe9uKZxwa4ZTshKJBLj89l2+Cm2mhmNsBB7i0AAYSP5QVBIdzjN9Sw/66Kxrg3aZobJt9v6lD0MndegLgPcuCJT5cwbWtqc/bWksPGtUdfqtIzcbIHTUYDqJcw74ZYAeOV2vZeGdMDi2dE++qE2d8QekhdFsUSX8ABzsaLljIZhy1H2eAU9dMTAvyvdAKCDYfajBXXJXXCdUy99c3rUmaf5lx51mwz4sccliZLFkqvsRkBF17SrewDxLSLWJ0k7Hq8eydXPMvKs= ceph-1892d8fc-e540-11ee-95c9-87774f69a715 2024-03-18T15:58:57.409 DEBUG:teuthology.orchestra.run.smithi194:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJK+CTQQ3Wg71D32RPxs4NoPmY8FpotvbTS2bAm99J85Gi9xX+K5lXncSe03PD7evIDOV7mIhbq6TzF2WFEbCoxGehisf1+fprn5BZrquL3L17pdY59T6aVxSnQjYOUNmq3kPNHBg6OjyT0ePvWVtyjyLdbG4ImaUivgMwv2w1L2wUSyGNB2/lBhocZjLxbzQy3atQHfT/llEt2G9TffPCk+7wAM3K+IkHwhwwaoh9qj9nLHTLroOmIe9uKZxwa4ZTshKJBLj89l2+Cm2mhmNsBB7i0AAYSP5QVBIdzjN9Sw/66Kxrg3aZobJt9v6lD0MndegLgPcuCJT5cwbWtqc/bWksPGtUdfqtIzcbIHTUYDqJcw74ZYAeOV2vZeGdMDi2dE++qE2d8QekhdFsUSX8ABzsaLljIZhy1H2eAU9dMTAvyvdAKCDYfajBXXJXXCdUy99c3rUmaf5lx51mwz4sccliZLFkqvsRkBF17SrewDxLSLWJ0k7Hq8eydXPMvKs= ceph-1892d8fc-e540-11ee-95c9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-18T15:58:57.454 INFO:teuthology.orchestra.run.smithi194.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJK+CTQQ3Wg71D32RPxs4NoPmY8FpotvbTS2bAm99J85Gi9xX+K5lXncSe03PD7evIDOV7mIhbq6TzF2WFEbCoxGehisf1+fprn5BZrquL3L17pdY59T6aVxSnQjYOUNmq3kPNHBg6OjyT0ePvWVtyjyLdbG4ImaUivgMwv2w1L2wUSyGNB2/lBhocZjLxbzQy3atQHfT/llEt2G9TffPCk+7wAM3K+IkHwhwwaoh9qj9nLHTLroOmIe9uKZxwa4ZTshKJBLj89l2+Cm2mhmNsBB7i0AAYSP5QVBIdzjN9Sw/66Kxrg3aZobJt9v6lD0MndegLgPcuCJT5cwbWtqc/bWksPGtUdfqtIzcbIHTUYDqJcw74ZYAeOV2vZeGdMDi2dE++qE2d8QekhdFsUSX8ABzsaLljIZhy1H2eAU9dMTAvyvdAKCDYfajBXXJXXCdUy99c3rUmaf5lx51mwz4sccliZLFkqvsRkBF17SrewDxLSLWJ0k7Hq8eydXPMvKs= ceph-1892d8fc-e540-11ee-95c9-87774f69a715 2024-03-18T15:58:57.469 DEBUG:teuthology.orchestra.run.smithi196:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJK+CTQQ3Wg71D32RPxs4NoPmY8FpotvbTS2bAm99J85Gi9xX+K5lXncSe03PD7evIDOV7mIhbq6TzF2WFEbCoxGehisf1+fprn5BZrquL3L17pdY59T6aVxSnQjYOUNmq3kPNHBg6OjyT0ePvWVtyjyLdbG4ImaUivgMwv2w1L2wUSyGNB2/lBhocZjLxbzQy3atQHfT/llEt2G9TffPCk+7wAM3K+IkHwhwwaoh9qj9nLHTLroOmIe9uKZxwa4ZTshKJBLj89l2+Cm2mhmNsBB7i0AAYSP5QVBIdzjN9Sw/66Kxrg3aZobJt9v6lD0MndegLgPcuCJT5cwbWtqc/bWksPGtUdfqtIzcbIHTUYDqJcw74ZYAeOV2vZeGdMDi2dE++qE2d8QekhdFsUSX8ABzsaLljIZhy1H2eAU9dMTAvyvdAKCDYfajBXXJXXCdUy99c3rUmaf5lx51mwz4sccliZLFkqvsRkBF17SrewDxLSLWJ0k7Hq8eydXPMvKs= ceph-1892d8fc-e540-11ee-95c9-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-18T15:58:57.515 INFO:teuthology.orchestra.run.smithi196.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJK+CTQQ3Wg71D32RPxs4NoPmY8FpotvbTS2bAm99J85Gi9xX+K5lXncSe03PD7evIDOV7mIhbq6TzF2WFEbCoxGehisf1+fprn5BZrquL3L17pdY59T6aVxSnQjYOUNmq3kPNHBg6OjyT0ePvWVtyjyLdbG4ImaUivgMwv2w1L2wUSyGNB2/lBhocZjLxbzQy3atQHfT/llEt2G9TffPCk+7wAM3K+IkHwhwwaoh9qj9nLHTLroOmIe9uKZxwa4ZTshKJBLj89l2+Cm2mhmNsBB7i0AAYSP5QVBIdzjN9Sw/66Kxrg3aZobJt9v6lD0MndegLgPcuCJT5cwbWtqc/bWksPGtUdfqtIzcbIHTUYDqJcw74ZYAeOV2vZeGdMDi2dE++qE2d8QekhdFsUSX8ABzsaLljIZhy1H2eAU9dMTAvyvdAKCDYfajBXXJXXCdUy99c3rUmaf5lx51mwz4sccliZLFkqvsRkBF17SrewDxLSLWJ0k7Hq8eydXPMvKs= ceph-1892d8fc-e540-11ee-95c9-87774f69a715 2024-03-18T15:58:57.530 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-18T15:58:57.971 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:59:00.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:00.441+0000 7ff6f1200700 1 -- 172.21.15.39:0/1503429566 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff6ec102790 msgr2=0x7ff6ec104b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:00.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:00.441+0000 7ff6f1200700 1 --2- 172.21.15.39:0/1503429566 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff6ec102790 0x7ff6ec104b80 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc009910 tx=0x7ff6dc009c20 comp rx=0 tx=0).stop 2024-03-18T15:59:00.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:00.442+0000 7ff6f1200700 1 -- 172.21.15.39:0/1503429566 shutdown_connections 2024-03-18T15:59:00.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:00.442+0000 7ff6f1200700 1 --2- 172.21.15.39:0/1503429566 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff6ec102790 0x7ff6ec104b80 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:00.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:00.442+0000 7ff6f1200700 1 -- 172.21.15.39:0/1503429566 >> 172.21.15.39:0/1503429566 conn(0x7ff6ec0fc1a0 msgr2=0x7ff6ec0fe5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:00.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:00.442+0000 7ff6f1200700 1 -- 172.21.15.39:0/1503429566 shutdown_connections 2024-03-18T15:59:00.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:00.442+0000 7ff6f1200700 1 -- 172.21.15.39:0/1503429566 wait complete. 2024-03-18T15:59:00.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:00.443+0000 7ff6f1200700 1 Processor -- start 2024-03-18T15:59:00.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:00.443+0000 7ff6f1200700 1 -- start start 2024-03-18T15:59:00.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:00.443+0000 7ff6f1200700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff6ec102790 0x7ff6ec197370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:00.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:00.443+0000 7ff6f1200700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7ff6ec107d30 con 0x7ff6ec102790 2024-03-18T15:59:00.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:00.444+0000 7ff6e9e00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff6ec102790 0x7ff6ec197370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:00.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:00.444+0000 7ff6e9e00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff6ec102790 0x7ff6ec197370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:54946/0 (socket says 172.21.15.39:54946) 2024-03-18T15:59:00.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:00.444+0000 7ff6e9e00700 1 -- 172.21.15.39:0/2370302965 learned_addr learned my addr 172.21.15.39:0/2370302965 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T15:59:00.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:00.444+0000 7ff6e9e00700 1 -- 172.21.15.39:0/2370302965 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6ec1978b0 con 0x7ff6ec102790 2024-03-18T15:59:00.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:00.444+0000 7ff6e9e00700 1 --2- 172.21.15.39:0/2370302965 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff6ec102790 0x7ff6ec197370 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc004fa0 tx=0x7ff6dc005080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:00.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:00.445+0000 7ff6e3400700 1 -- 172.21.15.39:0/2370302965 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff6dc027070 con 0x7ff6ec102790 2024-03-18T15:59:00.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:00.445+0000 7ff6f1200700 1 -- 172.21.15.39:0/2370302965 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6ec197ba0 con 0x7ff6ec102790 2024-03-18T15:59:00.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:00.445+0000 7ff6e3400700 1 -- 172.21.15.39:0/2370302965 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6dc00be40 con 0x7ff6ec102790 2024-03-18T15:59:00.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:00.445+0000 7ff6e3400700 1 -- 172.21.15.39:0/2370302965 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff6dc00f460 con 0x7ff6ec102790 2024-03-18T15:59:00.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:00.445+0000 7ff6f1200700 1 -- 172.21.15.39:0/2370302965 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6ec19a830 con 0x7ff6ec102790 2024-03-18T15:59:00.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:00.446+0000 7ff6e3400700 1 -- 172.21.15.39:0/2370302965 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7ff6dc02a070 con 0x7ff6ec102790 2024-03-18T15:59:00.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:00.446+0000 7ff6f1200700 1 -- 172.21.15.39:0/2370302965 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6ec1084b0 con 0x7ff6ec102790 2024-03-18T15:59:00.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:00.447+0000 7ff6e3400700 1 --2- 172.21.15.39:0/2370302965 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff6d0039020 0x7ff6d003b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:00.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:00.447+0000 7ff6e3400700 1 -- 172.21.15.39:0/2370302965 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7ff6dc01e030 con 0x7ff6ec102790 2024-03-18T15:59:00.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:00.447+0000 7ff6e1400700 1 --2- 172.21.15.39:0/2370302965 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff6d0039020 0x7ff6d003b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:00.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:00.448+0000 7ff6e1400700 1 --2- 172.21.15.39:0/2370302965 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff6d0039020 0x7ff6d003b4e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ff6d4006fd0 tx=0x7ff6d4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:00.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:00.453+0000 7ff6e3400700 1 -- 172.21.15.39:0/2370302965 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff6dc013070 con 0x7ff6ec102790 2024-03-18T15:59:00.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:00.678+0000 7ff6f1200700 1 -- 172.21.15.39:0/2370302965 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7ff6ec02dd60 con 0x7ff6ec102790 2024-03-18T15:59:00.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:00.688+0000 7ff6e3400700 1 -- 172.21.15.39:0/2370302965 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7ff6dc00e3e0 con 0x7ff6ec102790 2024-03-18T15:59:00.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:00.691+0000 7ff6f1200700 1 -- 172.21.15.39:0/2370302965 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff6d0039020 msgr2=0x7ff6d003b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:00.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:00.691+0000 7ff6f1200700 1 --2- 172.21.15.39:0/2370302965 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff6d0039020 0x7ff6d003b4e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ff6d4006fd0 tx=0x7ff6d4006e40 comp rx=0 tx=0).stop 2024-03-18T15:59:00.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:00.691+0000 7ff6f1200700 1 -- 172.21.15.39:0/2370302965 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff6ec102790 msgr2=0x7ff6ec197370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:00.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:00.691+0000 7ff6f1200700 1 --2- 172.21.15.39:0/2370302965 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff6ec102790 0x7ff6ec197370 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc004fa0 tx=0x7ff6dc005080 comp rx=0 tx=0).stop 2024-03-18T15:59:00.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:00.691+0000 7ff6f1200700 1 -- 172.21.15.39:0/2370302965 shutdown_connections 2024-03-18T15:59:00.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:00.691+0000 7ff6f1200700 1 --2- 172.21.15.39:0/2370302965 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff6d0039020 0x7ff6d003b4e0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:00.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:00.691+0000 7ff6f1200700 1 --2- 172.21.15.39:0/2370302965 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff6ec102790 0x7ff6ec197370 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:00.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:00.691+0000 7ff6f1200700 1 -- 172.21.15.39:0/2370302965 >> 172.21.15.39:0/2370302965 conn(0x7ff6ec0fc1a0 msgr2=0x7ff6ec0fcbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:00.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:00.692+0000 7ff6f1200700 1 -- 172.21.15.39:0/2370302965 shutdown_connections 2024-03-18T15:59:00.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:00.692+0000 7ff6f1200700 1 -- 172.21.15.39:0/2370302965 wait complete. 2024-03-18T15:59:01.002 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:00 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:01.002 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:00 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:01.002 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:00 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi039", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:59:01.002 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:00 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:01.003 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:00 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:59:01.003 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:00 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:01.003 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:00 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/2370302965' entity='client.admin' 2024-03-18T15:59:01.003 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:00 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:59:01.003 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:00 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:01.332 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-18T15:59:01.332 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-18T15:59:01.794 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:59:03.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:03.791+0000 7faf0d800700 1 -- 172.21.15.39:0/3758226157 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faf08076c80 msgr2=0x7faf080770a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:03.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:03.791+0000 7faf0d800700 1 --2- 172.21.15.39:0/3758226157 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faf08076c80 0x7faf080770a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7faef8009910 tx=0x7faef8009c20 comp rx=0 tx=0).stop 2024-03-18T15:59:03.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:03.791+0000 7faf0d800700 1 -- 172.21.15.39:0/3758226157 shutdown_connections 2024-03-18T15:59:03.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:03.791+0000 7faf0d800700 1 --2- 172.21.15.39:0/3758226157 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faf08076c80 0x7faf080770a0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:03.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:03.792+0000 7faf0d800700 1 -- 172.21.15.39:0/3758226157 >> 172.21.15.39:0/3758226157 conn(0x7faf080ff4d0 msgr2=0x7faf08101910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:03.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:03.792+0000 7faf0d800700 1 -- 172.21.15.39:0/3758226157 shutdown_connections 2024-03-18T15:59:03.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:03.792+0000 7faf0d800700 1 -- 172.21.15.39:0/3758226157 wait complete. 2024-03-18T15:59:03.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:03.793+0000 7faf0d800700 1 Processor -- start 2024-03-18T15:59:03.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:03.793+0000 7faf0d800700 1 -- start start 2024-03-18T15:59:03.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:03.794+0000 7faf0d800700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faf08076c80 0x7faf0819d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:03.795 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:03.794+0000 7faf0d800700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7faf0810e420 con 0x7faf08076c80 2024-03-18T15:59:03.795 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:03.794+0000 7faf06a00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faf08076c80 0x7faf0819d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:03.795 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:03.794+0000 7faf06a00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faf08076c80 0x7faf0819d9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:54968/0 (socket says 172.21.15.39:54968) 2024-03-18T15:59:03.796 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:03.794+0000 7faf06a00700 1 -- 172.21.15.39:0/1520750418 learned_addr learned my addr 172.21.15.39:0/1520750418 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T15:59:03.796 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:03.794+0000 7faf06a00700 1 -- 172.21.15.39:0/1520750418 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf0819df30 con 0x7faf08076c80 2024-03-18T15:59:03.796 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:03.795+0000 7faf06a00700 1 --2- 172.21.15.39:0/1520750418 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faf08076c80 0x7faf0819d9f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7faef8004fa0 tx=0x7faef8005080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:03.796 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:03.795+0000 7faeff400700 1 -- 172.21.15.39:0/1520750418 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faef8027070 con 0x7faf08076c80 2024-03-18T15:59:03.796 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:03.795+0000 7faeff400700 1 -- 172.21.15.39:0/1520750418 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faef800be40 con 0x7faf08076c80 2024-03-18T15:59:03.796 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:03.795+0000 7faf0d800700 1 -- 172.21.15.39:0/1520750418 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf0819e1c0 con 0x7faf08076c80 2024-03-18T15:59:03.797 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:03.795+0000 7faeff400700 1 -- 172.21.15.39:0/1520750418 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7faef800f460 con 0x7faf08076c80 2024-03-18T15:59:03.797 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:03.795+0000 7faf0d800700 1 -- 172.21.15.39:0/1520750418 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf081a0eb0 con 0x7faf08076c80 2024-03-18T15:59:03.797 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:03.796+0000 7faeff400700 1 -- 172.21.15.39:0/1520750418 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7faef802a070 con 0x7faf08076c80 2024-03-18T15:59:03.797 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:03.796+0000 7faf0d800700 1 -- 172.21.15.39:0/1520750418 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf0810eb30 con 0x7faf08076c80 2024-03-18T15:59:03.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:03.797+0000 7faeff400700 1 --2- 172.21.15.39:0/1520750418 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7faeec038fc0 0x7faeec03b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:03.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:03.797+0000 7faeff400700 1 -- 172.21.15.39:0/1520750418 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7faef801e030 con 0x7faf08076c80 2024-03-18T15:59:03.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:03.797+0000 7faf06000700 1 --2- 172.21.15.39:0/1520750418 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7faeec038fc0 0x7faeec03b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:03.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:03.798+0000 7faf06000700 1 --2- 172.21.15.39:0/1520750418 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7faeec038fc0 0x7faeec03b480 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7faef4006fd0 tx=0x7faef4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:03.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:03.803+0000 7faeff400700 1 -- 172.21.15.39:0/1520750418 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7faef8013070 con 0x7faf08076c80 2024-03-18T15:59:04.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:04.040+0000 7faf0d800700 1 -- 172.21.15.39:0/1520750418 --> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7faf0802dc80 con 0x7faeec038fc0 2024-03-18T15:59:04.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:04.047+0000 7faeff400700 1 -- 172.21.15.39:0/1520750418 <== mgr.14152 v2:172.21.15.39:6800/2749214286 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7faf0802dc80 con 0x7faeec038fc0 2024-03-18T15:59:04.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:04.050+0000 7faf0d800700 1 -- 172.21.15.39:0/1520750418 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7faeec038fc0 msgr2=0x7faeec03b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:04.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:04.050+0000 7faf0d800700 1 --2- 172.21.15.39:0/1520750418 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7faeec038fc0 0x7faeec03b480 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7faef4006fd0 tx=0x7faef4006e40 comp rx=0 tx=0).stop 2024-03-18T15:59:04.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:04.050+0000 7faf0d800700 1 -- 172.21.15.39:0/1520750418 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faf08076c80 msgr2=0x7faf0819d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:04.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:04.050+0000 7faf0d800700 1 --2- 172.21.15.39:0/1520750418 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faf08076c80 0x7faf0819d9f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7faef8004fa0 tx=0x7faef8005080 comp rx=0 tx=0).stop 2024-03-18T15:59:04.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:04.050+0000 7faf0d800700 1 -- 172.21.15.39:0/1520750418 shutdown_connections 2024-03-18T15:59:04.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:04.050+0000 7faf0d800700 1 --2- 172.21.15.39:0/1520750418 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7faeec038fc0 0x7faeec03b480 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:04.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:04.051+0000 7faf0d800700 1 --2- 172.21.15.39:0/1520750418 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faf08076c80 0x7faf0819d9f0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:04.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:04.051+0000 7faf0d800700 1 -- 172.21.15.39:0/1520750418 >> 172.21.15.39:0/1520750418 conn(0x7faf080ff4d0 msgr2=0x7faf080fff60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:04.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:04.051+0000 7faf0d800700 1 -- 172.21.15.39:0/1520750418 shutdown_connections 2024-03-18T15:59:04.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:04.051+0000 7faf0d800700 1 -- 172.21.15.39:0/1520750418 wait complete. 2024-03-18T15:59:04.798 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi194 2024-03-18T15:59:04.798 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-03-18T15:59:04.798 DEBUG:teuthology.orchestra.run.smithi194:> dd of=/etc/ceph/ceph.conf 2024-03-18T15:59:04.818 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-03-18T15:59:04.818 DEBUG:teuthology.orchestra.run.smithi194:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-18T15:59:04.875 INFO:tasks.cephadm:Adding host smithi194 to orchestrator... 2024-03-18T15:59:04.875 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph orch host add smithi194 2024-03-18T15:59:05.303 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:59:05.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:05 smithi039 ceph-mon[31721]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:59:05.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:05 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:05.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:05 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:59:05.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:05 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:59:05.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:05 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:59:05.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:05 smithi039 ceph-mon[31721]: Updating smithi039:/etc/ceph/ceph.conf 2024-03-18T15:59:06.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:06 smithi039 ceph-mon[31721]: Updating smithi039:/var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T15:59:07.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:07.328+0000 7fb870800700 1 -- 172.21.15.39:0/3087768718 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb8681009c0 msgr2=0x7fb868100da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:07.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:07.328+0000 7fb870800700 1 --2- 172.21.15.39:0/3087768718 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb8681009c0 0x7fb868100da0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb85c009910 tx=0x7fb85c009c20 comp rx=0 tx=0).stop 2024-03-18T15:59:07.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:07.328+0000 7fb870800700 1 -- 172.21.15.39:0/3087768718 shutdown_connections 2024-03-18T15:59:07.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:07.328+0000 7fb870800700 1 --2- 172.21.15.39:0/3087768718 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb8681009c0 0x7fb868100da0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:07.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:07.328+0000 7fb870800700 1 -- 172.21.15.39:0/3087768718 >> 172.21.15.39:0/3087768718 conn(0x7fb8680777d0 msgr2=0x7fb8680feae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:07.330 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:07 smithi039 ceph-mon[31721]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:59:07.331 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:07 smithi039 ceph-mon[31721]: Updating smithi039:/etc/ceph/ceph.client.admin.keyring 2024-03-18T15:59:07.331 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:07 smithi039 ceph-mon[31721]: Updating smithi039:/var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-18T15:59:07.331 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:07 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:07.331 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:07 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:07.331 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:07 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:07.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:07.329+0000 7fb870800700 1 -- 172.21.15.39:0/3087768718 shutdown_connections 2024-03-18T15:59:07.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:07.329+0000 7fb870800700 1 -- 172.21.15.39:0/3087768718 wait complete. 2024-03-18T15:59:07.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:07.330+0000 7fb870800700 1 Processor -- start 2024-03-18T15:59:07.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:07.330+0000 7fb870800700 1 -- start start 2024-03-18T15:59:07.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:07.330+0000 7fb870800700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb8681009c0 0x7fb86819dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:07.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:07.330+0000 7fb870800700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fb86810e500 con 0x7fb8681009c0 2024-03-18T15:59:07.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:07.331+0000 7fb86de00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb8681009c0 0x7fb86819dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:07.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:07.331+0000 7fb86de00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb8681009c0 0x7fb86819dd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:54984/0 (socket says 172.21.15.39:54984) 2024-03-18T15:59:07.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:07.331+0000 7fb86de00700 1 -- 172.21.15.39:0/2111603755 learned_addr learned my addr 172.21.15.39:0/2111603755 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T15:59:07.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:07.331+0000 7fb86de00700 1 -- 172.21.15.39:0/2111603755 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb86819e2b0 con 0x7fb8681009c0 2024-03-18T15:59:07.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:07.331+0000 7fb86de00700 1 --2- 172.21.15.39:0/2111603755 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb8681009c0 0x7fb86819dd70 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb85c005850 tx=0x7fb85c005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:07.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:07.332+0000 7fb862a00700 1 -- 172.21.15.39:0/2111603755 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb85c026070 con 0x7fb8681009c0 2024-03-18T15:59:07.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:07.332+0000 7fb862a00700 1 -- 172.21.15.39:0/2111603755 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb85c00bc30 con 0x7fb8681009c0 2024-03-18T15:59:07.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:07.332+0000 7fb870800700 1 -- 172.21.15.39:0/2111603755 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb86819e540 con 0x7fb8681009c0 2024-03-18T15:59:07.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:07.332+0000 7fb862a00700 1 -- 172.21.15.39:0/2111603755 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb85c003a20 con 0x7fb8681009c0 2024-03-18T15:59:07.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:07.332+0000 7fb870800700 1 -- 172.21.15.39:0/2111603755 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb86819e9a0 con 0x7fb8681009c0 2024-03-18T15:59:07.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:07.333+0000 7fb870800700 1 -- 172.21.15.39:0/2111603755 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb868069fa0 con 0x7fb8681009c0 2024-03-18T15:59:07.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:07.333+0000 7fb862a00700 1 -- 172.21.15.39:0/2111603755 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7fb85c029070 con 0x7fb8681009c0 2024-03-18T15:59:07.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:07.334+0000 7fb862a00700 1 --2- 172.21.15.39:0/2111603755 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb850039020 0x7fb85003b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:07.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:07.334+0000 7fb862a00700 1 -- 172.21.15.39:0/2111603755 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fb85c01e070 con 0x7fb8681009c0 2024-03-18T15:59:07.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:07.334+0000 7fb86d400700 1 --2- 172.21.15.39:0/2111603755 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb850039020 0x7fb85003b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:07.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:07.335+0000 7fb86d400700 1 --2- 172.21.15.39:0/2111603755 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb850039020 0x7fb85003b4e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fb858006fd0 tx=0x7fb858006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:07.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:07.339+0000 7fb862a00700 1 -- 172.21.15.39:0/2111603755 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb85c013070 con 0x7fb8681009c0 2024-03-18T15:59:07.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:07.571+0000 7fb870800700 1 -- 172.21.15.39:0/2111603755 --> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi194", "target": ["mon-mgr", ""]}) -- 0x7fb86819f0d0 con 0x7fb850039020 2024-03-18T15:59:09.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:09 smithi039 ceph-mon[31721]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:59:09.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:09 smithi039 ceph-mon[31721]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi194", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:59:10.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:10 smithi039 ceph-mon[31721]: Deploying cephadm binary to smithi194 2024-03-18T15:59:11.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:11 smithi039 ceph-mon[31721]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:59:12.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:12.547+0000 7fb862a00700 1 -- 172.21.15.39:0/2111603755 <== mgr.14152 v2:172.21.15.39:6800/2749214286 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7fb86819f0d0 con 0x7fb850039020 2024-03-18T15:59:12.548 INFO:teuthology.orchestra.run.smithi039.stdout:Added host 'smithi194' with addr '172.21.15.194' 2024-03-18T15:59:12.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:12.550+0000 7fb870800700 1 -- 172.21.15.39:0/2111603755 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb850039020 msgr2=0x7fb85003b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:12.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:12.550+0000 7fb870800700 1 --2- 172.21.15.39:0/2111603755 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb850039020 0x7fb85003b4e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fb858006fd0 tx=0x7fb858006e40 comp rx=0 tx=0).stop 2024-03-18T15:59:12.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:12.551+0000 7fb870800700 1 -- 172.21.15.39:0/2111603755 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb8681009c0 msgr2=0x7fb86819dd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:12.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:12.551+0000 7fb870800700 1 --2- 172.21.15.39:0/2111603755 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb8681009c0 0x7fb86819dd70 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb85c005850 tx=0x7fb85c005030 comp rx=0 tx=0).stop 2024-03-18T15:59:12.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:12.551+0000 7fb870800700 1 -- 172.21.15.39:0/2111603755 shutdown_connections 2024-03-18T15:59:12.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:12.551+0000 7fb870800700 1 --2- 172.21.15.39:0/2111603755 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb850039020 0x7fb85003b4e0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:12.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:12.551+0000 7fb870800700 1 --2- 172.21.15.39:0/2111603755 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb8681009c0 0x7fb86819dd70 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:12.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:12.551+0000 7fb870800700 1 -- 172.21.15.39:0/2111603755 >> 172.21.15.39:0/2111603755 conn(0x7fb8680777d0 msgr2=0x7fb8680feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:12.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:12.551+0000 7fb870800700 1 -- 172.21.15.39:0/2111603755 shutdown_connections 2024-03-18T15:59:12.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:12.551+0000 7fb870800700 1 -- 172.21.15.39:0/2111603755 wait complete. 2024-03-18T15:59:13.271 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph orch host ls --format=json 2024-03-18T15:59:13.358 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:13 smithi039 ceph-mon[31721]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:59:13.358 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:13 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:13.358 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:13 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:59:13.736 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:59:14.144 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:14 smithi039 ceph-mon[31721]: Added host smithi194 2024-03-18T15:59:14.144 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:14 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:14.144 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:14 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:15.316 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:15 smithi039 ceph-mon[31721]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:59:15.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:15.717+0000 7f9910200700 1 -- 172.21.15.39:0/2660649997 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f99080746a0 msgr2=0x7f9908074a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:15.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:15.717+0000 7f9910200700 1 --2- 172.21.15.39:0/2660649997 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f99080746a0 0x7f9908074a80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f98fc009910 tx=0x7f98fc009c20 comp rx=0 tx=0).stop 2024-03-18T15:59:15.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:15.717+0000 7f9910200700 1 -- 172.21.15.39:0/2660649997 shutdown_connections 2024-03-18T15:59:15.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:15.717+0000 7f9910200700 1 --2- 172.21.15.39:0/2660649997 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f99080746a0 0x7f9908074a80 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:15.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:15.717+0000 7f9910200700 1 -- 172.21.15.39:0/2660649997 >> 172.21.15.39:0/2660649997 conn(0x7f99080fd7b0 msgr2=0x7f99080ffbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:15.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:15.717+0000 7f9910200700 1 -- 172.21.15.39:0/2660649997 shutdown_connections 2024-03-18T15:59:15.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:15.717+0000 7f9910200700 1 -- 172.21.15.39:0/2660649997 wait complete. 2024-03-18T15:59:15.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:15.718+0000 7f9910200700 1 Processor -- start 2024-03-18T15:59:15.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:15.718+0000 7f9910200700 1 -- start start 2024-03-18T15:59:15.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:15.719+0000 7f9910200700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f99080746a0 0x7f9908101c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:15.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:15.719+0000 7f9910200700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f990810e540 con 0x7f99080746a0 2024-03-18T15:59:15.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:15.719+0000 7f990d800700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f99080746a0 0x7f9908101c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:15.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:15.719+0000 7f990d800700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f99080746a0 0x7f9908101c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:38072/0 (socket says 172.21.15.39:38072) 2024-03-18T15:59:15.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:15.719+0000 7f990d800700 1 -- 172.21.15.39:0/1450294428 learned_addr learned my addr 172.21.15.39:0/1450294428 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T15:59:15.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:15.720+0000 7f990d800700 1 -- 172.21.15.39:0/1450294428 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9908105bc0 con 0x7f99080746a0 2024-03-18T15:59:15.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:15.720+0000 7f990d800700 1 --2- 172.21.15.39:0/1450294428 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f99080746a0 0x7f9908101c60 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f98fc006010 tx=0x7f98fc005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:15.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:15.721+0000 7f9906000700 1 -- 172.21.15.39:0/1450294428 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f98fc026070 con 0x7f99080746a0 2024-03-18T15:59:15.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:15.721+0000 7f9910200700 1 -- 172.21.15.39:0/1450294428 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99081021a0 con 0x7f99080746a0 2024-03-18T15:59:15.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:15.721+0000 7f9910200700 1 -- 172.21.15.39:0/1450294428 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9908102600 con 0x7f99080746a0 2024-03-18T15:59:15.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:15.721+0000 7f9906000700 1 -- 172.21.15.39:0/1450294428 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f98fc00b750 con 0x7f99080746a0 2024-03-18T15:59:15.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:15.721+0000 7f9906000700 1 -- 172.21.15.39:0/1450294428 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f98fc030060 con 0x7f99080746a0 2024-03-18T15:59:15.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:15.722+0000 7f9910200700 1 -- 172.21.15.39:0/1450294428 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98e8005330 con 0x7f99080746a0 2024-03-18T15:59:15.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:15.722+0000 7f9906000700 1 -- 172.21.15.39:0/1450294428 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f98fc029070 con 0x7f99080746a0 2024-03-18T15:59:15.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:15.723+0000 7f9906000700 1 --2- 172.21.15.39:0/1450294428 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f98f0039020 0x7f98f003b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:15.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:15.723+0000 7f9906000700 1 -- 172.21.15.39:0/1450294428 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f98fc01e070 con 0x7f99080746a0 2024-03-18T15:59:15.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:15.723+0000 7f990ce00700 1 --2- 172.21.15.39:0/1450294428 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f98f0039020 0x7f98f003b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:15.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:15.724+0000 7f990ce00700 1 --2- 172.21.15.39:0/1450294428 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f98f0039020 0x7f98f003b4e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f98f4006fd0 tx=0x7f98f4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:15.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:15.729+0000 7f9906000700 1 -- 172.21.15.39:0/1450294428 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f98fc013070 con 0x7f99080746a0 2024-03-18T15:59:15.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:15.962+0000 7f9910200700 1 -- 172.21.15.39:0/1450294428 --> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f98e8000bc0 con 0x7f98f0039020 2024-03-18T15:59:15.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:15.964+0000 7f9906000700 1 -- 172.21.15.39:0/1450294428 <== mgr.14152 v2:172.21.15.39:6800/2749214286 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+160 (secure 0 0 0) 0x7f98e8000bc0 con 0x7f98f0039020 2024-03-18T15:59:15.965 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T15:59:15.965 INFO:teuthology.orchestra.run.smithi039.stdout:[{"addr": "172.21.15.39", "hostname": "smithi039", "labels": [], "status": ""}, {"addr": "172.21.15.194", "hostname": "smithi194", "labels": [], "status": ""}] 2024-03-18T15:59:15.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:15.967+0000 7f9910200700 1 -- 172.21.15.39:0/1450294428 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f98f0039020 msgr2=0x7f98f003b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:15.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:15.967+0000 7f9910200700 1 --2- 172.21.15.39:0/1450294428 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f98f0039020 0x7f98f003b4e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f98f4006fd0 tx=0x7f98f4006e40 comp rx=0 tx=0).stop 2024-03-18T15:59:15.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:15.967+0000 7f9910200700 1 -- 172.21.15.39:0/1450294428 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f99080746a0 msgr2=0x7f9908101c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:15.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:15.967+0000 7f9910200700 1 --2- 172.21.15.39:0/1450294428 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f99080746a0 0x7f9908101c60 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f98fc006010 tx=0x7f98fc005030 comp rx=0 tx=0).stop 2024-03-18T15:59:15.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:15.967+0000 7f9910200700 1 -- 172.21.15.39:0/1450294428 shutdown_connections 2024-03-18T15:59:15.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:15.967+0000 7f9910200700 1 --2- 172.21.15.39:0/1450294428 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f98f0039020 0x7f98f003b4e0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:15.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:15.967+0000 7f9910200700 1 --2- 172.21.15.39:0/1450294428 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f99080746a0 0x7f9908101c60 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:15.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:15.967+0000 7f9910200700 1 -- 172.21.15.39:0/1450294428 >> 172.21.15.39:0/1450294428 conn(0x7f99080fd7b0 msgr2=0x7f9908105fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:15.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:15.967+0000 7f9910200700 1 -- 172.21.15.39:0/1450294428 shutdown_connections 2024-03-18T15:59:15.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:15.967+0000 7f9910200700 1 -- 172.21.15.39:0/1450294428 wait complete. 2024-03-18T15:59:16.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:16 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:16.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:16 smithi039 ceph-mon[31721]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:59:16.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:16 smithi039 ceph-mon[31721]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T15:59:16.751 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi196 2024-03-18T15:59:16.752 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-18T15:59:16.752 DEBUG:teuthology.orchestra.run.smithi196:> dd of=/etc/ceph/ceph.conf 2024-03-18T15:59:16.771 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-18T15:59:16.771 DEBUG:teuthology.orchestra.run.smithi196:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-18T15:59:16.830 INFO:tasks.cephadm:Adding host smithi196 to orchestrator... 2024-03-18T15:59:16.830 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph orch host add smithi196 2024-03-18T15:59:17.277 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:59:18.787 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:18 smithi039 ceph-mon[31721]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:59:18.787 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:18 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:19.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:19.110+0000 7f0c58000700 1 -- 172.21.15.39:0/1716873649 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0c50101f00 msgr2=0x7f0c501022e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:19.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:19.110+0000 7f0c58000700 1 --2- 172.21.15.39:0/1716873649 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0c50101f00 0x7f0c501022e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f0c44009910 tx=0x7f0c44009c20 comp rx=0 tx=0).stop 2024-03-18T15:59:19.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:19.110+0000 7f0c58000700 1 -- 172.21.15.39:0/1716873649 shutdown_connections 2024-03-18T15:59:19.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:19.110+0000 7f0c58000700 1 --2- 172.21.15.39:0/1716873649 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0c50101f00 0x7f0c501022e0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:19.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:19.110+0000 7f0c58000700 1 -- 172.21.15.39:0/1716873649 >> 172.21.15.39:0/1716873649 conn(0x7f0c500fd7b0 msgr2=0x7f0c500ffbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:19.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:19.110+0000 7f0c58000700 1 -- 172.21.15.39:0/1716873649 shutdown_connections 2024-03-18T15:59:19.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:19.111+0000 7f0c58000700 1 -- 172.21.15.39:0/1716873649 wait complete. 2024-03-18T15:59:19.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:19.112+0000 7f0c58000700 1 Processor -- start 2024-03-18T15:59:19.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:19.112+0000 7f0c58000700 1 -- start start 2024-03-18T15:59:19.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:19.112+0000 7f0c58000700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0c50101f00 0x7f0c5019d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:19.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:19.112+0000 7f0c58000700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f0c5010c2d0 con 0x7f0c50101f00 2024-03-18T15:59:19.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:19.112+0000 7f0c55600700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0c50101f00 0x7f0c5019d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:19.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:19.112+0000 7f0c55600700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0c50101f00 0x7f0c5019d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:54932/0 (socket says 172.21.15.39:54932) 2024-03-18T15:59:19.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:19.112+0000 7f0c55600700 1 -- 172.21.15.39:0/3074273449 learned_addr learned my addr 172.21.15.39:0/3074273449 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T15:59:19.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:19.113+0000 7f0c55600700 1 -- 172.21.15.39:0/3074273449 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c5019d990 con 0x7f0c50101f00 2024-03-18T15:59:19.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:19.113+0000 7f0c55600700 1 --2- 172.21.15.39:0/3074273449 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0c50101f00 0x7f0c5019d450 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0c44006010 tx=0x7f0c44005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:19.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:19.113+0000 7f0c4e000700 1 -- 172.21.15.39:0/3074273449 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0c44026070 con 0x7f0c50101f00 2024-03-18T15:59:19.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:19.114+0000 7f0c4e000700 1 -- 172.21.15.39:0/3074273449 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0c4400bbd0 con 0x7f0c50101f00 2024-03-18T15:59:19.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:19.114+0000 7f0c4e000700 1 -- 172.21.15.39:0/3074273449 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0c44003910 con 0x7f0c50101f00 2024-03-18T15:59:19.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:19.114+0000 7f0c58000700 1 -- 172.21.15.39:0/3074273449 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c5019dc80 con 0x7f0c50101f00 2024-03-18T15:59:19.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:19.114+0000 7f0c58000700 1 -- 172.21.15.39:0/3074273449 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c50197610 con 0x7f0c50101f00 2024-03-18T15:59:19.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:19.115+0000 7f0c4e000700 1 -- 172.21.15.39:0/3074273449 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f0c44029070 con 0x7f0c50101f00 2024-03-18T15:59:19.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:19.115+0000 7f0c58000700 1 -- 172.21.15.39:0/3074273449 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c501033e0 con 0x7f0c50101f00 2024-03-18T15:59:19.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:19.115+0000 7f0c4e000700 1 --2- 172.21.15.39:0/3074273449 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f0c38039020 0x7f0c3803b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:19.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:19.115+0000 7f0c4e000700 1 -- 172.21.15.39:0/3074273449 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f0c4401e070 con 0x7f0c50101f00 2024-03-18T15:59:19.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:19.116+0000 7f0c54c00700 1 --2- 172.21.15.39:0/3074273449 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f0c38039020 0x7f0c3803b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:19.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:19.116+0000 7f0c54c00700 1 --2- 172.21.15.39:0/3074273449 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f0c38039020 0x7f0c3803b4e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f0c3c006fd0 tx=0x7f0c3c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:19.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:19.121+0000 7f0c4e000700 1 -- 172.21.15.39:0/3074273449 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0c44013070 con 0x7f0c50101f00 2024-03-18T15:59:19.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:19.354+0000 7f0c58000700 1 -- 172.21.15.39:0/3074273449 --> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi196", "target": ["mon-mgr", ""]}) -- 0x7f0c50198b40 con 0x7f0c38039020 2024-03-18T15:59:19.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:19 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:19.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:19 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:19.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:19 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:19.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:19 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi194", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:59:19.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:19 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:59:19.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:19 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:59:19.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:19 smithi039 ceph-mon[31721]: Updating smithi194:/etc/ceph/ceph.conf 2024-03-18T15:59:19.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:19 smithi039 ceph-mon[31721]: Updating smithi194:/var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T15:59:19.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:19 smithi039 ceph-mon[31721]: Updating smithi194:/etc/ceph/ceph.client.admin.keyring 2024-03-18T15:59:21.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:20 smithi039 ceph-mon[31721]: Updating smithi194:/var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-18T15:59:21.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:20 smithi039 ceph-mon[31721]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi196", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:59:21.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:20 smithi039 ceph-mon[31721]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:59:21.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:20 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:21.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:20 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:21.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:20 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:21.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:20 smithi039 ceph-mon[31721]: Deploying cephadm binary to smithi196 2024-03-18T15:59:23.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:22 smithi039 ceph-mon[31721]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:59:24.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:24.259+0000 7f0c4e000700 1 -- 172.21.15.39:0/3074273449 <== mgr.14152 v2:172.21.15.39:6800/2749214286 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f0c50198b40 con 0x7f0c38039020 2024-03-18T15:59:24.262 INFO:teuthology.orchestra.run.smithi039.stdout:Added host 'smithi196' with addr '172.21.15.196' 2024-03-18T15:59:24.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:24.263+0000 7f0c58000700 1 -- 172.21.15.39:0/3074273449 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f0c38039020 msgr2=0x7f0c3803b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:24.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:24.263+0000 7f0c58000700 1 --2- 172.21.15.39:0/3074273449 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f0c38039020 0x7f0c3803b4e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f0c3c006fd0 tx=0x7f0c3c006e40 comp rx=0 tx=0).stop 2024-03-18T15:59:24.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:24.263+0000 7f0c58000700 1 -- 172.21.15.39:0/3074273449 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0c50101f00 msgr2=0x7f0c5019d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:24.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:24.263+0000 7f0c58000700 1 --2- 172.21.15.39:0/3074273449 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0c50101f00 0x7f0c5019d450 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0c44006010 tx=0x7f0c44005030 comp rx=0 tx=0).stop 2024-03-18T15:59:24.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:24.263+0000 7f0c58000700 1 -- 172.21.15.39:0/3074273449 shutdown_connections 2024-03-18T15:59:24.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:24.263+0000 7f0c58000700 1 --2- 172.21.15.39:0/3074273449 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f0c38039020 0x7f0c3803b4e0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:24.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:24.263+0000 7f0c58000700 1 --2- 172.21.15.39:0/3074273449 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0c50101f00 0x7f0c5019d450 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:24.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:24.264+0000 7f0c58000700 1 -- 172.21.15.39:0/3074273449 >> 172.21.15.39:0/3074273449 conn(0x7f0c500fd7b0 msgr2=0x7f0c501038d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:24.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:24.264+0000 7f0c58000700 1 -- 172.21.15.39:0/3074273449 shutdown_connections 2024-03-18T15:59:24.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:24.264+0000 7f0c58000700 1 -- 172.21.15.39:0/3074273449 wait complete. 2024-03-18T15:59:25.176 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph orch host ls --format=json 2024-03-18T15:59:25.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:25 smithi039 ceph-mon[31721]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:59:25.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:25 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:25.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:25 smithi039 ceph-mon[31721]: Added host smithi196 2024-03-18T15:59:25.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:25 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:59:25.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:25 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:25.651 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:59:26.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:26 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:26.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:26 smithi039 ceph-mon[31721]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:59:27.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:27.723+0000 7fe562200700 1 -- 172.21.15.39:0/3707973285 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe55c10e1f0 msgr2=0x7fe55c10e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:27.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:27.723+0000 7fe562200700 1 --2- 172.21.15.39:0/3707973285 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe55c10e1f0 0x7fe55c10e5d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fe550009910 tx=0x7fe550009c20 comp rx=0 tx=0).stop 2024-03-18T15:59:27.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:27.723+0000 7fe562200700 1 -- 172.21.15.39:0/3707973285 shutdown_connections 2024-03-18T15:59:27.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:27.723+0000 7fe562200700 1 --2- 172.21.15.39:0/3707973285 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe55c10e1f0 0x7fe55c10e5d0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:27.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:27.723+0000 7fe562200700 1 -- 172.21.15.39:0/3707973285 >> 172.21.15.39:0/3707973285 conn(0x7fe55c0778b0 msgr2=0x7fe55c079cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:27.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:27.723+0000 7fe562200700 1 -- 172.21.15.39:0/3707973285 shutdown_connections 2024-03-18T15:59:27.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:27.723+0000 7fe562200700 1 -- 172.21.15.39:0/3707973285 wait complete. 2024-03-18T15:59:27.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:27.724+0000 7fe562200700 1 Processor -- start 2024-03-18T15:59:27.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:27.725+0000 7fe562200700 1 -- start start 2024-03-18T15:59:27.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:27.725+0000 7fe562200700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe55c10e1f0 0x7fe55c1a3780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:27.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:27.725+0000 7fe562200700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fe55c112740 con 0x7fe55c10e1f0 2024-03-18T15:59:27.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:27.725+0000 7fe560e00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe55c10e1f0 0x7fe55c1a3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:27.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:27.725+0000 7fe560e00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe55c10e1f0 0x7fe55c1a3780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:54938/0 (socket says 172.21.15.39:54938) 2024-03-18T15:59:27.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:27.725+0000 7fe560e00700 1 -- 172.21.15.39:0/3560867602 learned_addr learned my addr 172.21.15.39:0/3560867602 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T15:59:27.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:27.726+0000 7fe560e00700 1 -- 172.21.15.39:0/3560867602 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe55c1a3cc0 con 0x7fe55c10e1f0 2024-03-18T15:59:27.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:27.726+0000 7fe560e00700 1 --2- 172.21.15.39:0/3560867602 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe55c10e1f0 0x7fe55c1a3780 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe550006010 tx=0x7fe550005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:27.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:27.726+0000 7fe559600700 1 -- 172.21.15.39:0/3560867602 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe550026070 con 0x7fe55c10e1f0 2024-03-18T15:59:27.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:27.726+0000 7fe559600700 1 -- 172.21.15.39:0/3560867602 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe55000bbd0 con 0x7fe55c10e1f0 2024-03-18T15:59:27.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:27.726+0000 7fe559600700 1 -- 172.21.15.39:0/3560867602 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe550003a30 con 0x7fe55c10e1f0 2024-03-18T15:59:27.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:27.726+0000 7fe562200700 1 -- 172.21.15.39:0/3560867602 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe55c1a3fb0 con 0x7fe55c10e1f0 2024-03-18T15:59:27.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:27.727+0000 7fe562200700 1 -- 172.21.15.39:0/3560867602 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe55c19d940 con 0x7fe55c10e1f0 2024-03-18T15:59:27.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:27.728+0000 7fe559600700 1 -- 172.21.15.39:0/3560867602 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7fe550029070 con 0x7fe55c10e1f0 2024-03-18T15:59:27.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:27.728+0000 7fe562200700 1 -- 172.21.15.39:0/3560867602 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe55c109830 con 0x7fe55c10e1f0 2024-03-18T15:59:27.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:27.728+0000 7fe559600700 1 --2- 172.21.15.39:0/3560867602 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fe544039020 0x7fe54403b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:27.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:27.728+0000 7fe559600700 1 -- 172.21.15.39:0/3560867602 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fe55001e070 con 0x7fe55c10e1f0 2024-03-18T15:59:27.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:27.729+0000 7fe55be00700 1 --2- 172.21.15.39:0/3560867602 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fe544039020 0x7fe54403b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:27.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:27.729+0000 7fe55be00700 1 --2- 172.21.15.39:0/3560867602 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fe544039020 0x7fe54403b4e0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fe548006fd0 tx=0x7fe548006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:27.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:27.734+0000 7fe559600700 1 -- 172.21.15.39:0/3560867602 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe550013070 con 0x7fe55c10e1f0 2024-03-18T15:59:27.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:27.966+0000 7fe562200700 1 -- 172.21.15.39:0/3560867602 --> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe55c19ef80 con 0x7fe544039020 2024-03-18T15:59:27.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:27.968+0000 7fe559600700 1 -- 172.21.15.39:0/3560867602 <== mgr.14152 v2:172.21.15.39:6800/2749214286 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+240 (secure 0 0 0) 0x7fe55c19ef80 con 0x7fe544039020 2024-03-18T15:59:27.970 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T15:59:27.970 INFO:teuthology.orchestra.run.smithi039.stdout:[{"addr": "172.21.15.39", "hostname": "smithi039", "labels": [], "status": ""}, {"addr": "172.21.15.194", "hostname": "smithi194", "labels": [], "status": ""}, {"addr": "172.21.15.196", "hostname": "smithi196", "labels": [], "status": ""}] 2024-03-18T15:59:27.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:27.971+0000 7fe562200700 1 -- 172.21.15.39:0/3560867602 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fe544039020 msgr2=0x7fe54403b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:27.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:27.971+0000 7fe562200700 1 --2- 172.21.15.39:0/3560867602 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fe544039020 0x7fe54403b4e0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fe548006fd0 tx=0x7fe548006e40 comp rx=0 tx=0).stop 2024-03-18T15:59:27.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:27.971+0000 7fe562200700 1 -- 172.21.15.39:0/3560867602 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe55c10e1f0 msgr2=0x7fe55c1a3780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:27.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:27.971+0000 7fe562200700 1 --2- 172.21.15.39:0/3560867602 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe55c10e1f0 0x7fe55c1a3780 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe550006010 tx=0x7fe550005050 comp rx=0 tx=0).stop 2024-03-18T15:59:27.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:27.971+0000 7fe562200700 1 -- 172.21.15.39:0/3560867602 shutdown_connections 2024-03-18T15:59:27.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:27.971+0000 7fe562200700 1 --2- 172.21.15.39:0/3560867602 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fe544039020 0x7fe54403b4e0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:27.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:27.971+0000 7fe562200700 1 --2- 172.21.15.39:0/3560867602 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe55c10e1f0 0x7fe55c1a3780 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:27.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:27.971+0000 7fe562200700 1 -- 172.21.15.39:0/3560867602 >> 172.21.15.39:0/3560867602 conn(0x7fe55c0778b0 msgr2=0x7fe55c077c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:27.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:27.971+0000 7fe562200700 1 -- 172.21.15.39:0/3560867602 shutdown_connections 2024-03-18T15:59:27.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:27.971+0000 7fe562200700 1 -- 172.21.15.39:0/3560867602 wait complete. 2024-03-18T15:59:28.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:28 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:28.739 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-18T15:59:28.739 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph osd crush tunables default 2024-03-18T15:59:29.208 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:59:29.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:29 smithi039 ceph-mon[31721]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:59:29.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:29 smithi039 ceph-mon[31721]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T15:59:31.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:31.288+0000 7f45afe00700 1 -- 172.21.15.39:0/3838719990 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f45b0074700 msgr2=0x7f45b0074ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:31.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:31.288+0000 7f45afe00700 1 --2- 172.21.15.39:0/3838719990 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f45b0074700 0x7f45b0074ae0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f45a0009910 tx=0x7f45a0009c20 comp rx=0 tx=0).stop 2024-03-18T15:59:31.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:31.289+0000 7f45afe00700 1 -- 172.21.15.39:0/3838719990 shutdown_connections 2024-03-18T15:59:31.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:31.289+0000 7f45afe00700 1 --2- 172.21.15.39:0/3838719990 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f45b0074700 0x7f45b0074ae0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:31.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:31.289+0000 7f45afe00700 1 -- 172.21.15.39:0/3838719990 >> 172.21.15.39:0/3838719990 conn(0x7f45b00fd600 msgr2=0x7f45b00ffa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:31.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:31.289+0000 7f45afe00700 1 -- 172.21.15.39:0/3838719990 shutdown_connections 2024-03-18T15:59:31.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:31.289+0000 7f45afe00700 1 -- 172.21.15.39:0/3838719990 wait complete. 2024-03-18T15:59:31.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:31.290+0000 7f45afe00700 1 Processor -- start 2024-03-18T15:59:31.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:31.290+0000 7f45afe00700 1 -- start start 2024-03-18T15:59:31.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:31.290+0000 7f45afe00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f45b0074700 0x7f45b019f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:31.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:31.290+0000 7f45afe00700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f45b010e3a0 con 0x7f45b0074700 2024-03-18T15:59:31.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:31.291+0000 7f45aea00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f45b0074700 0x7f45b019f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:31.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:31.291+0000 7f45aea00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f45b0074700 0x7f45b019f600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:48898/0 (socket says 172.21.15.39:48898) 2024-03-18T15:59:31.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:31.291+0000 7f45aea00700 1 -- 172.21.15.39:0/3277886616 learned_addr learned my addr 172.21.15.39:0/3277886616 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T15:59:31.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:31.292+0000 7f45aea00700 1 -- 172.21.15.39:0/3277886616 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45b019fb40 con 0x7f45b0074700 2024-03-18T15:59:31.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:31.292+0000 7f45aea00700 1 --2- 172.21.15.39:0/3277886616 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f45b0074700 0x7f45b019f600 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f45a0006010 tx=0x7f45a0005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:31.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:31.292+0000 7f45a7400700 1 -- 172.21.15.39:0/3277886616 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f45a0026070 con 0x7f45b0074700 2024-03-18T15:59:31.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:31.292+0000 7f45a7400700 1 -- 172.21.15.39:0/3277886616 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f45a000bc30 con 0x7f45b0074700 2024-03-18T15:59:31.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:31.292+0000 7f45afe00700 1 -- 172.21.15.39:0/3277886616 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45b019fdd0 con 0x7f45b0074700 2024-03-18T15:59:31.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:31.293+0000 7f45a7400700 1 -- 172.21.15.39:0/3277886616 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f45a0003ab0 con 0x7f45b0074700 2024-03-18T15:59:31.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:31.293+0000 7f45afe00700 1 -- 172.21.15.39:0/3277886616 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45b01996d0 con 0x7f45b0074700 2024-03-18T15:59:31.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:31.294+0000 7f45afe00700 1 -- 172.21.15.39:0/3277886616 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45b0075db0 con 0x7f45b0074700 2024-03-18T15:59:31.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:31.294+0000 7f45a7400700 1 -- 172.21.15.39:0/3277886616 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f45a0029070 con 0x7f45b0074700 2024-03-18T15:59:31.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:31.295+0000 7f45a7400700 1 --2- 172.21.15.39:0/3277886616 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f4594039020 0x7f459403b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:31.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:31.295+0000 7f45a7400700 1 -- 172.21.15.39:0/3277886616 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f45a001e070 con 0x7f45b0074700 2024-03-18T15:59:31.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:31.295+0000 7f45ae000700 1 --2- 172.21.15.39:0/3277886616 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f4594039020 0x7f459403b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:31.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:31.296+0000 7f45ae000700 1 --2- 172.21.15.39:0/3277886616 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f4594039020 0x7f459403b4e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f459c006fd0 tx=0x7f459c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:31.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:31.301+0000 7f45a7400700 1 -- 172.21.15.39:0/3277886616 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f45a0013070 con 0x7f45b0074700 2024-03-18T15:59:31.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:31.539+0000 7f45afe00700 1 -- 172.21.15.39:0/3277886616 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f45b0063930 con 0x7f45b0074700 2024-03-18T15:59:31.546 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:31 smithi039 ceph-mon[31721]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:59:31.546 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:31 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:31.546 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:31 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:31.546 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:31 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:31.546 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:31 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:31.546 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:31 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi196", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:59:31.546 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:31 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:59:31.546 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:31 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:59:31.546 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:31 smithi039 ceph-mon[31721]: Updating smithi196:/etc/ceph/ceph.conf 2024-03-18T15:59:32.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:32.417+0000 7f45a7400700 1 -- 172.21.15.39:0/3277886616 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7f45b0063930 con 0x7f45b0074700 2024-03-18T15:59:32.420 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:32 smithi039 ceph-mon[31721]: Updating smithi196:/var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T15:59:32.420 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:32 smithi039 ceph-mon[31721]: Updating smithi196:/etc/ceph/ceph.client.admin.keyring 2024-03-18T15:59:32.420 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:32 smithi039 ceph-mon[31721]: Updating smithi196:/var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-18T15:59:32.420 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:32 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:32.420 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:32 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:32.420 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:32 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:32.421 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:32 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/3277886616' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-18T15:59:32.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:32.421+0000 7f45afe00700 1 -- 172.21.15.39:0/3277886616 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f4594039020 msgr2=0x7f459403b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:32.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:32.421+0000 7f45afe00700 1 --2- 172.21.15.39:0/3277886616 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f4594039020 0x7f459403b4e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f459c006fd0 tx=0x7f459c006e40 comp rx=0 tx=0).stop 2024-03-18T15:59:32.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:32.421+0000 7f45afe00700 1 -- 172.21.15.39:0/3277886616 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f45b0074700 msgr2=0x7f45b019f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:32.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:32.421+0000 7f45afe00700 1 --2- 172.21.15.39:0/3277886616 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f45b0074700 0x7f45b019f600 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f45a0006010 tx=0x7f45a0005050 comp rx=0 tx=0).stop 2024-03-18T15:59:32.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:32.421+0000 7f45afe00700 1 -- 172.21.15.39:0/3277886616 shutdown_connections 2024-03-18T15:59:32.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:32.421+0000 7f45afe00700 1 --2- 172.21.15.39:0/3277886616 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f4594039020 0x7f459403b4e0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:32.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:32.421+0000 7f45afe00700 1 --2- 172.21.15.39:0/3277886616 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f45b0074700 0x7f45b019f600 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:32.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:32.421+0000 7f45afe00700 1 -- 172.21.15.39:0/3277886616 >> 172.21.15.39:0/3277886616 conn(0x7f45b00fd600 msgr2=0x7f45b0105d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:32.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:32.421+0000 7f45afe00700 1 -- 172.21.15.39:0/3277886616 shutdown_connections 2024-03-18T15:59:32.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:32.422+0000 7f45afe00700 1 -- 172.21.15.39:0/3277886616 wait complete. 2024-03-18T15:59:32.425 INFO:teuthology.orchestra.run.smithi039.stderr:adjusted tunables profile to default 2024-03-18T15:59:33.282 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:33 smithi039 ceph-mon[31721]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:59:33.282 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:33 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/3277886616' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-18T15:59:33.283 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:33 smithi039 ceph-mon[31721]: osdmap e4: 0 total, 0 up, 0 in 2024-03-18T15:59:33.351 INFO:tasks.cephadm:Adding mon.a on smithi039 2024-03-18T15:59:33.352 INFO:tasks.cephadm:Adding mon.b on smithi194 2024-03-18T15:59:33.352 INFO:tasks.cephadm:Adding mon.c on smithi196 2024-03-18T15:59:33.352 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph orch apply mon '3;smithi039:172.21.15.39=a;smithi194:172.21.15.194=b;smithi196:172.21.15.196=c' 2024-03-18T15:59:33.589 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T15:59:34.848 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:34.845+0000 7f6f40929700 1 -- 172.21.15.196:0/2504160483 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f3c100e00 msgr2=0x7f6f3c1011e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:34.848 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:34.845+0000 7f6f40929700 1 --2- 172.21.15.196:0/2504160483 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f3c100e00 0x7f6f3c1011e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f6f2c00b0a0 tx=0x7f6f2c00b3b0 comp rx=0 tx=0).stop 2024-03-18T15:59:34.848 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:34.845+0000 7f6f40929700 1 -- 172.21.15.196:0/2504160483 shutdown_connections 2024-03-18T15:59:34.848 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:34.845+0000 7f6f40929700 1 --2- 172.21.15.196:0/2504160483 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f3c100e00 0x7f6f3c1011e0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:34.848 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:34.846+0000 7f6f40929700 1 -- 172.21.15.196:0/2504160483 >> 172.21.15.196:0/2504160483 conn(0x7f6f3c077810 msgr2=0x7f6f3c0feae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:34.848 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:34.846+0000 7f6f40929700 1 -- 172.21.15.196:0/2504160483 shutdown_connections 2024-03-18T15:59:34.848 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:34.846+0000 7f6f40929700 1 -- 172.21.15.196:0/2504160483 wait complete. 2024-03-18T15:59:34.849 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:34.846+0000 7f6f40929700 1 Processor -- start 2024-03-18T15:59:34.849 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:34.846+0000 7f6f40929700 1 -- start start 2024-03-18T15:59:34.849 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:34.846+0000 7f6f40929700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f3c100e00 0x7f6f3c198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:34.849 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:34.846+0000 7f6f40929700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f6f3c10b1d0 con 0x7f6f3c100e00 2024-03-18T15:59:34.850 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:34.847+0000 7f6f3a59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f3c100e00 0x7f6f3c198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:34.850 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:34.847+0000 7f6f3a59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f3c100e00 0x7f6f3c198600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.196:59466/0 (socket says 172.21.15.196:59466) 2024-03-18T15:59:34.850 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:34.847+0000 7f6f3a59c700 1 -- 172.21.15.196:0/1230776113 learned_addr learned my addr 172.21.15.196:0/1230776113 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-18T15:59:34.850 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:34.848+0000 7f6f3a59c700 1 -- 172.21.15.196:0/1230776113 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f3c198b40 con 0x7f6f3c100e00 2024-03-18T15:59:34.850 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:34.848+0000 7f6f3a59c700 1 --2- 172.21.15.196:0/1230776113 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f3c100e00 0x7f6f3c198600 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6f2c015040 tx=0x7f6f2c003d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:34.850 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:34.848+0000 7f6f337fe700 1 -- 172.21.15.196:0/1230776113 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6f2c017040 con 0x7f6f3c100e00 2024-03-18T15:59:34.850 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:34.848+0000 7f6f40929700 1 -- 172.21.15.196:0/1230776113 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f3c198dd0 con 0x7f6f3c100e00 2024-03-18T15:59:34.851 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:34.848+0000 7f6f40929700 1 -- 172.21.15.196:0/1230776113 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f3c199290 con 0x7f6f3c100e00 2024-03-18T15:59:34.851 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:34.848+0000 7f6f337fe700 1 -- 172.21.15.196:0/1230776113 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f2c022070 con 0x7f6f3c100e00 2024-03-18T15:59:34.851 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:34.848+0000 7f6f337fe700 1 -- 172.21.15.196:0/1230776113 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6f2c009410 con 0x7f6f3c100e00 2024-03-18T15:59:34.851 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:34.849+0000 7f6f40929700 1 -- 172.21.15.196:0/1230776113 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f3c1022e0 con 0x7f6f3c100e00 2024-03-18T15:59:34.852 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:34.850+0000 7f6f337fe700 1 -- 172.21.15.196:0/1230776113 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f6f2c00bc10 con 0x7f6f3c100e00 2024-03-18T15:59:34.852 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:34.850+0000 7f6f337fe700 1 --2- 172.21.15.196:0/1230776113 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f6f28039160 0x7f6f2803b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:34.852 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:34.850+0000 7f6f337fe700 1 -- 172.21.15.196:0/1230776113 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f6f2c00f050 con 0x7f6f3c100e00 2024-03-18T15:59:34.852 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:34.851+0000 7f6f39d9b700 1 --2- 172.21.15.196:0/1230776113 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f6f28039160 0x7f6f2803b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:34.853 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:34.852+0000 7f6f39d9b700 1 --2- 172.21.15.196:0/1230776113 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f6f28039160 0x7f6f2803b620 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6f24006fd0 tx=0x7f6f24006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:34.867 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:34.866+0000 7f6f337fe700 1 -- 172.21.15.196:0/1230776113 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6f2c007e20 con 0x7f6f3c100e00 2024-03-18T15:59:35.074 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:35.073+0000 7f6f40929700 1 -- 172.21.15.196:0/1230776113 --> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi039:172.21.15.39=a;smithi194:172.21.15.194=b;smithi196:172.21.15.196=c", "target": ["mon-mgr", ""]}) -- 0x7f6f3c063900 con 0x7f6f28039160 2024-03-18T15:59:35.089 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:35.088+0000 7f6f337fe700 1 -- 172.21.15.196:0/1230776113 <== mgr.14152 v2:172.21.15.39:6800/2749214286 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f6f3c063900 con 0x7f6f28039160 2024-03-18T15:59:35.090 INFO:teuthology.orchestra.run.smithi196.stdout:Scheduled mon update... 2024-03-18T15:59:35.092 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:35.090+0000 7f6f40929700 1 -- 172.21.15.196:0/1230776113 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f6f28039160 msgr2=0x7f6f2803b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:35.092 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:35.090+0000 7f6f40929700 1 --2- 172.21.15.196:0/1230776113 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f6f28039160 0x7f6f2803b620 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6f24006fd0 tx=0x7f6f24006e40 comp rx=0 tx=0).stop 2024-03-18T15:59:35.092 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:35.090+0000 7f6f40929700 1 -- 172.21.15.196:0/1230776113 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f3c100e00 msgr2=0x7f6f3c198600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:35.092 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:35.090+0000 7f6f40929700 1 --2- 172.21.15.196:0/1230776113 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f3c100e00 0x7f6f3c198600 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6f2c015040 tx=0x7f6f2c003d50 comp rx=0 tx=0).stop 2024-03-18T15:59:35.092 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:35.090+0000 7f6f40929700 1 -- 172.21.15.196:0/1230776113 shutdown_connections 2024-03-18T15:59:35.092 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:35.090+0000 7f6f40929700 1 --2- 172.21.15.196:0/1230776113 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f6f28039160 0x7f6f2803b620 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:35.092 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:35.090+0000 7f6f40929700 1 --2- 172.21.15.196:0/1230776113 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f3c100e00 0x7f6f3c198600 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:35.092 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:35.090+0000 7f6f40929700 1 -- 172.21.15.196:0/1230776113 >> 172.21.15.196:0/1230776113 conn(0x7f6f3c077810 msgr2=0x7f6f3c1027d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:35.092 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:35.090+0000 7f6f40929700 1 -- 172.21.15.196:0/1230776113 shutdown_connections 2024-03-18T15:59:35.092 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:35.090+0000 7f6f40929700 1 -- 172.21.15.196:0/1230776113 wait complete. 2024-03-18T15:59:35.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:35 smithi039 ceph-mon[31721]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:59:35.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:35 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:35.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:35 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:59:35.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:35 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:59:35.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:35 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:59:35.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:35 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:35.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:35 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T15:59:35.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:35 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:59:35.708 DEBUG:teuthology.orchestra.run.smithi194:mon.b> sudo journalctl -f -n 0 -u ceph-1892d8fc-e540-11ee-95c9-87774f69a715@mon.b.service 2024-03-18T15:59:35.711 DEBUG:teuthology.orchestra.run.smithi196:mon.c> sudo journalctl -f -n 0 -u ceph-1892d8fc-e540-11ee-95c9-87774f69a715@mon.c.service 2024-03-18T15:59:35.713 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-18T15:59:35.713 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-18T15:59:36.378 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T15:59:36.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:36 smithi039 ceph-mon[31721]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi039:172.21.15.39=a;smithi194:172.21.15.194=b;smithi196:172.21.15.196=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:59:36.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:36 smithi039 ceph-mon[31721]: Saving service mon spec with placement smithi039:172.21.15.39=a;smithi194:172.21.15.194=b;smithi196:172.21.15.196=c;count:3 2024-03-18T15:59:36.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:36 smithi039 ceph-mon[31721]: Deploying daemon mon.c on smithi196 2024-03-18T15:59:37.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:37 smithi039 ceph-mon[31721]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:59:39.147 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:39.145+0000 7ff78d78f700 1 -- 172.21.15.196:0/107911885 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff788074700 msgr2=0x7ff788074ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:39.148 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:39.145+0000 7ff78d78f700 1 --2- 172.21.15.196:0/107911885 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff788074700 0x7ff788074ae0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7ff778009910 tx=0x7ff778009c20 comp rx=0 tx=0).stop 2024-03-18T15:59:39.148 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:39.145+0000 7ff78d78f700 1 -- 172.21.15.196:0/107911885 shutdown_connections 2024-03-18T15:59:39.148 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:39.145+0000 7ff78d78f700 1 --2- 172.21.15.196:0/107911885 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff788074700 0x7ff788074ae0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:39.148 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:39.145+0000 7ff78d78f700 1 -- 172.21.15.196:0/107911885 >> 172.21.15.196:0/107911885 conn(0x7ff7880fd800 msgr2=0x7ff7880ffc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:39.148 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:39.145+0000 7ff78d78f700 1 -- 172.21.15.196:0/107911885 shutdown_connections 2024-03-18T15:59:39.148 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:39.145+0000 7ff78d78f700 1 -- 172.21.15.196:0/107911885 wait complete. 2024-03-18T15:59:39.148 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:39.145+0000 7ff78d78f700 1 Processor -- start 2024-03-18T15:59:39.148 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:39.146+0000 7ff78d78f700 1 -- start start 2024-03-18T15:59:39.148 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:39.146+0000 7ff78d78f700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff788074700 0x7ff78819f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:39.148 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:39.146+0000 7ff78d78f700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7ff78810e580 con 0x7ff788074700 2024-03-18T15:59:39.149 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:39.146+0000 7ff786ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff788074700 0x7ff78819f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:39.150 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:39.146+0000 7ff786ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff788074700 0x7ff78819f6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.196:59488/0 (socket says 172.21.15.196:59488) 2024-03-18T15:59:39.150 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:39.146+0000 7ff786ffd700 1 -- 172.21.15.196:0/119451341 learned_addr learned my addr 172.21.15.196:0/119451341 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-18T15:59:39.150 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:39.147+0000 7ff786ffd700 1 -- 172.21.15.196:0/119451341 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff78819fc00 con 0x7ff788074700 2024-03-18T15:59:39.150 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:39.147+0000 7ff786ffd700 1 --2- 172.21.15.196:0/119451341 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff788074700 0x7ff78819f6c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff778006010 tx=0x7ff778005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:39.150 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:39.148+0000 7ff76ffff700 1 -- 172.21.15.196:0/119451341 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff778026070 con 0x7ff788074700 2024-03-18T15:59:39.150 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:39.148+0000 7ff78d78f700 1 -- 172.21.15.196:0/119451341 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff78819fe90 con 0x7ff788074700 2024-03-18T15:59:39.150 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:39.148+0000 7ff78d78f700 1 -- 172.21.15.196:0/119451341 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff788199890 con 0x7ff788074700 2024-03-18T15:59:39.150 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:39.148+0000 7ff76ffff700 1 -- 172.21.15.196:0/119451341 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff778004830 con 0x7ff788074700 2024-03-18T15:59:39.150 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:39.148+0000 7ff76ffff700 1 -- 172.21.15.196:0/119451341 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff778030060 con 0x7ff788074700 2024-03-18T15:59:39.151 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:39.149+0000 7ff78d78f700 1 -- 172.21.15.196:0/119451341 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff774005330 con 0x7ff788074700 2024-03-18T15:59:39.151 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:39.149+0000 7ff76ffff700 1 -- 172.21.15.196:0/119451341 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7ff778029070 con 0x7ff788074700 2024-03-18T15:59:39.151 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:39.150+0000 7ff76ffff700 1 --2- 172.21.15.196:0/119451341 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff770038fc0 0x7ff77003b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:39.151 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:39.150+0000 7ff76ffff700 1 -- 172.21.15.196:0/119451341 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7ff77801e070 con 0x7ff788074700 2024-03-18T15:59:39.152 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:39.150+0000 7ff7867fc700 1 --2- 172.21.15.196:0/119451341 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff770038fc0 0x7ff77003b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:39.152 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:39.151+0000 7ff7867fc700 1 --2- 172.21.15.196:0/119451341 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff770038fc0 0x7ff77003b480 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff77c006fd0 tx=0x7ff77c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:39.166 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:39.165+0000 7ff76ffff700 1 -- 172.21.15.196:0/119451341 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff778013070 con 0x7ff788074700 2024-03-18T15:59:39.461 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:39.460+0000 7ff78d78f700 1 -- 172.21.15.196:0/119451341 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7ff7740058f0 con 0x7ff788074700 2024-03-18T15:59:39.462 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:39.461+0000 7ff76ffff700 1 -- 172.21.15.196:0/119451341 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7ff778031250 con 0x7ff788074700 2024-03-18T15:59:39.462 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-18T15:59:39.463 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":1,"fsid":"1892d8fc-e540-11ee-95c9-87774f69a715","modified":"2024-03-18T15:57:32.305376Z","created":"2024-03-18T15:57:32.305376Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-18T15:59:39.465 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:39.463+0000 7ff78d78f700 1 -- 172.21.15.196:0/119451341 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff770038fc0 msgr2=0x7ff77003b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:39.465 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:39.463+0000 7ff78d78f700 1 --2- 172.21.15.196:0/119451341 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff770038fc0 0x7ff77003b480 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff77c006fd0 tx=0x7ff77c006e40 comp rx=0 tx=0).stop 2024-03-18T15:59:39.465 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:39.463+0000 7ff78d78f700 1 -- 172.21.15.196:0/119451341 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff788074700 msgr2=0x7ff78819f6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:39.465 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:39.463+0000 7ff78d78f700 1 --2- 172.21.15.196:0/119451341 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff788074700 0x7ff78819f6c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff778006010 tx=0x7ff778005050 comp rx=0 tx=0).stop 2024-03-18T15:59:39.465 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:39.463+0000 7ff78d78f700 1 -- 172.21.15.196:0/119451341 shutdown_connections 2024-03-18T15:59:39.465 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:39.463+0000 7ff78d78f700 1 --2- 172.21.15.196:0/119451341 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff770038fc0 0x7ff77003b480 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:39.465 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:39.463+0000 7ff78d78f700 1 --2- 172.21.15.196:0/119451341 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff788074700 0x7ff78819f6c0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:39.465 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:39.463+0000 7ff78d78f700 1 -- 172.21.15.196:0/119451341 >> 172.21.15.196:0/119451341 conn(0x7ff7880fd800 msgr2=0x7ff788105f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:39.465 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:39.463+0000 7ff78d78f700 1 -- 172.21.15.196:0/119451341 shutdown_connections 2024-03-18T15:59:39.465 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:39.463+0000 7ff78d78f700 1 -- 172.21.15.196:0/119451341 wait complete. 2024-03-18T15:59:39.466 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 1 2024-03-18T15:59:39.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:39 smithi039 ceph-mon[31721]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:59:40.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:40 smithi039 ceph-mon[31721]: from='client.? 172.21.15.196:0/119451341' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-18T15:59:40.617 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:40 smithi196 ceph-mon[36307]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-18T15:59:40.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:40 smithi196 bash[36289]: 83e5d1a96c4638f038cca2c77e66c118994e205a95cd7d1088ea996a47cb513b 2024-03-18T15:59:40.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:40 smithi196 systemd[1]: Started Ceph mon.c for 1892d8fc-e540-11ee-95c9-87774f69a715. 2024-03-18T15:59:41.180 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-18T15:59:41.180 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-18T15:59:41.414 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T15:59:45.836 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:45.833+0000 7ff015194700 1 -- 172.21.15.196:0/1866355617 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff010100f00 msgr2=0x7feff8005610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:45.836 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:45.833+0000 7ff015194700 1 --2- 172.21.15.196:0/1866355617 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff010100f00 0x7feff8005610 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ff004005fa0 tx=0x7ff00400ffc0 comp rx=0 tx=0).stop 2024-03-18T15:59:45.836 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:45.833+0000 7ff015194700 1 -- 172.21.15.196:0/1866355617 shutdown_connections 2024-03-18T15:59:45.837 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:45.833+0000 7ff015194700 1 --2- 172.21.15.196:0/1866355617 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff010100f00 0x7feff8005610 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:45.837 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:45.833+0000 7ff015194700 1 -- 172.21.15.196:0/1866355617 >> 172.21.15.196:0/1866355617 conn(0x7ff0100fc770 msgr2=0x7ff0100feb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:45.837 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:45.833+0000 7ff015194700 1 -- 172.21.15.196:0/1866355617 shutdown_connections 2024-03-18T15:59:45.837 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:45.833+0000 7ff015194700 1 -- 172.21.15.196:0/1866355617 wait complete. 2024-03-18T15:59:45.837 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:45.834+0000 7ff015194700 1 Processor -- start 2024-03-18T15:59:45.837 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:45.834+0000 7ff015194700 1 -- start start 2024-03-18T15:59:45.837 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:45.834+0000 7ff015194700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff010100f00 0x7ff01019d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:45.837 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:45.834+0000 7ff015194700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff01019db40 0x7ff0101976d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:45.837 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:45.834+0000 7ff015194700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7ff010108d30 con 0x7ff010100f00 2024-03-18T15:59:45.837 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:45.834+0000 7ff015194700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7ff010108bb0 con 0x7ff01019db40 2024-03-18T15:59:45.837 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:45.834+0000 7ff00e59c700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff01019db40 0x7ff0101976d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:45.837 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:45.834+0000 7ff00e59c700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff01019db40 0x7ff0101976d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.196:52622/0 (socket says 172.21.15.196:52622) 2024-03-18T15:59:45.837 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:45.834+0000 7ff00e59c700 1 -- 172.21.15.196:0/3249888235 learned_addr learned my addr 172.21.15.196:0/3249888235 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-18T15:59:45.838 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:45.834+0000 7ff00e59c700 1 -- 172.21.15.196:0/3249888235 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff01019db40 msgr2=0x7ff0101976d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-18T15:59:45.838 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:45.834+0000 7ff00e59c700 1 -- 172.21.15.196:0/3249888235 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff01019db40 msgr2=0x7ff0101976d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-18T15:59:45.838 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:45.834+0000 7ff00e59c700 1 --2- 172.21.15.196:0/3249888235 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff01019db40 0x7ff0101976d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-18T15:59:45.838 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:45.834+0000 7ff00e59c700 1 --2- 172.21.15.196:0/3249888235 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff01019db40 0x7ff0101976d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T15:59:45.838 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:45.835+0000 7ff00ed9d700 1 --2- 172.21.15.196:0/3249888235 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff010100f00 0x7ff01019d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:45.839 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:45.835+0000 7ff00ed9d700 1 -- 172.21.15.196:0/3249888235 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff01019db40 msgr2=0x7ff0101976d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T15:59:45.839 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:45.835+0000 7ff00ed9d700 1 --2- 172.21.15.196:0/3249888235 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff01019db40 0x7ff0101976d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:45.839 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:45.835+0000 7ff00ed9d700 1 -- 172.21.15.196:0/3249888235 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff01019e0c0 con 0x7ff010100f00 2024-03-18T15:59:45.839 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:45.836+0000 7ff00ed9d700 1 --2- 172.21.15.196:0/3249888235 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff010100f00 0x7ff01019d600 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff004000c00 tx=0x7ff004004d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:45.839 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:45.836+0000 7feff7fff700 1 -- 172.21.15.196:0/3249888235 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7ff0040053b0 con 0x7ff010100f00 2024-03-18T15:59:45.840 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:45.836+0000 7ff015194700 1 -- 172.21.15.196:0/3249888235 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff010197c70 con 0x7ff010100f00 2024-03-18T15:59:45.840 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:45.836+0000 7feff7fff700 1 -- 172.21.15.196:0/3249888235 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff004005550 con 0x7ff010100f00 2024-03-18T15:59:45.840 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:45.836+0000 7ff015194700 1 -- 172.21.15.196:0/3249888235 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0101981d0 con 0x7ff010100f00 2024-03-18T15:59:45.840 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:45.836+0000 7feff7fff700 1 -- 172.21.15.196:0/3249888235 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7ff004011670 con 0x7ff010100f00 2024-03-18T15:59:45.840 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:45.837+0000 7ff015194700 1 -- 172.21.15.196:0/3249888235 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0101023e0 con 0x7ff010100f00 2024-03-18T15:59:45.840 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:45.837+0000 7feff7fff700 1 -- 172.21.15.196:0/3249888235 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7ff004004030 con 0x7ff010100f00 2024-03-18T15:59:45.840 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:45.838+0000 7feff7fff700 1 --2- 172.21.15.196:0/3249888235 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7feffc038d80 0x7feffc03b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:45.840 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:45.838+0000 7feff7fff700 1 -- 172.21.15.196:0/3249888235 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7ff00404e5c0 con 0x7ff010100f00 2024-03-18T15:59:45.840 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:45.838+0000 7ff00e59c700 1 --2- 172.21.15.196:0/3249888235 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7feffc038d80 0x7feffc03b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:45.841 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:45.839+0000 7ff00e59c700 1 --2- 172.21.15.196:0/3249888235 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7feffc038d80 0x7feffc03b240 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff000009180 tx=0x7ff000008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:45.854 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:45.853+0000 7feff7fff700 1 -- 172.21.15.196:0/3249888235 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff004022070 con 0x7ff010100f00 2024-03-18T15:59:45.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:45 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T15:59:45.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:45 smithi196 ceph-mon[36307]: mon.a calling monitor election 2024-03-18T15:59:45.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:45 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:59:45.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:45 smithi196 ceph-mon[36307]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:59:45.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:45 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:59:45.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:45 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:59:45.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:45 smithi196 ceph-mon[36307]: mon.c calling monitor election 2024-03-18T15:59:45.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:45 smithi196 ceph-mon[36307]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:59:45.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:45 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:59:45.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:45 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:59:45.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:45 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:59:45.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:45 smithi196 ceph-mon[36307]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-18T15:59:45.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:45 smithi196 ceph-mon[36307]: monmap epoch 2 2024-03-18T15:59:45.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:45 smithi196 ceph-mon[36307]: fsid 1892d8fc-e540-11ee-95c9-87774f69a715 2024-03-18T15:59:45.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:45 smithi196 ceph-mon[36307]: last_changed 2024-03-18T15:59:40.606884+0000 2024-03-18T15:59:45.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:45 smithi196 ceph-mon[36307]: created 2024-03-18T15:57:32.305376+0000 2024-03-18T15:59:45.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:45 smithi196 ceph-mon[36307]: min_mon_release 19 (squid) 2024-03-18T15:59:45.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:45 smithi196 ceph-mon[36307]: election_strategy: 1 2024-03-18T15:59:45.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:45 smithi196 ceph-mon[36307]: 0: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.a 2024-03-18T15:59:45.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:45 smithi196 ceph-mon[36307]: 1: [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] mon.c 2024-03-18T15:59:45.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:45 smithi196 ceph-mon[36307]: fsmap 2024-03-18T15:59:45.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:45 smithi196 ceph-mon[36307]: osdmap e4: 0 total, 0 up, 0 in 2024-03-18T15:59:45.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:45 smithi196 ceph-mon[36307]: mgrmap e13: x(active, since 60s) 2024-03-18T15:59:45.980 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:45 smithi196 ceph-mon[36307]: overall HEALTH_OK 2024-03-18T15:59:45.980 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:45 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:45.980 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:45 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:45.995 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:45 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T15:59:45.995 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:45 smithi039 ceph-mon[31721]: mon.a calling monitor election 2024-03-18T15:59:45.995 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:45 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:59:45.995 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:45 smithi039 ceph-mon[31721]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:59:45.995 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:45 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:59:45.995 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:45 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:59:45.995 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:45 smithi039 ceph-mon[31721]: mon.c calling monitor election 2024-03-18T15:59:45.995 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:45 smithi039 ceph-mon[31721]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:59:45.995 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:45 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:59:45.995 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:45 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:59:45.995 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:45 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:59:45.995 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:45 smithi039 ceph-mon[31721]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-18T15:59:45.995 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:45 smithi039 ceph-mon[31721]: monmap epoch 2 2024-03-18T15:59:45.996 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:45 smithi039 ceph-mon[31721]: fsid 1892d8fc-e540-11ee-95c9-87774f69a715 2024-03-18T15:59:45.996 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:45 smithi039 ceph-mon[31721]: last_changed 2024-03-18T15:59:40.606884+0000 2024-03-18T15:59:45.996 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:45 smithi039 ceph-mon[31721]: created 2024-03-18T15:57:32.305376+0000 2024-03-18T15:59:45.996 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:45 smithi039 ceph-mon[31721]: min_mon_release 19 (squid) 2024-03-18T15:59:45.996 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:45 smithi039 ceph-mon[31721]: election_strategy: 1 2024-03-18T15:59:45.996 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:45 smithi039 ceph-mon[31721]: 0: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.a 2024-03-18T15:59:45.996 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:45 smithi039 ceph-mon[31721]: 1: [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] mon.c 2024-03-18T15:59:45.996 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:45 smithi039 ceph-mon[31721]: fsmap 2024-03-18T15:59:45.996 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:45 smithi039 ceph-mon[31721]: osdmap e4: 0 total, 0 up, 0 in 2024-03-18T15:59:45.996 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:45 smithi039 ceph-mon[31721]: mgrmap e13: x(active, since 60s) 2024-03-18T15:59:45.996 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:45 smithi039 ceph-mon[31721]: overall HEALTH_OK 2024-03-18T15:59:45.996 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:45 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:45.996 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:45 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:46.131 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:46.129+0000 7ff015194700 1 -- 172.21.15.196:0/3249888235 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7ff010198540 con 0x7ff010100f00 2024-03-18T15:59:46.131 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:46.130+0000 7feff7fff700 1 -- 172.21.15.196:0/3249888235 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7ff0040103d0 con 0x7ff010100f00 2024-03-18T15:59:46.132 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-18T15:59:46.132 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":2,"fsid":"1892d8fc-e540-11ee-95c9-87774f69a715","modified":"2024-03-18T15:59:40.606884Z","created":"2024-03-18T15:57:32.305376Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-18T15:59:46.134 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:46.132+0000 7ff015194700 1 -- 172.21.15.196:0/3249888235 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7feffc038d80 msgr2=0x7feffc03b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:46.134 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:46.132+0000 7ff015194700 1 --2- 172.21.15.196:0/3249888235 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7feffc038d80 0x7feffc03b240 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff000009180 tx=0x7ff000008040 comp rx=0 tx=0).stop 2024-03-18T15:59:46.134 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:46.132+0000 7ff015194700 1 -- 172.21.15.196:0/3249888235 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff010100f00 msgr2=0x7ff01019d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:46.134 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:46.132+0000 7ff015194700 1 --2- 172.21.15.196:0/3249888235 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff010100f00 0x7ff01019d600 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff004000c00 tx=0x7ff004004d60 comp rx=0 tx=0).stop 2024-03-18T15:59:46.134 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:46.132+0000 7ff015194700 1 -- 172.21.15.196:0/3249888235 shutdown_connections 2024-03-18T15:59:46.134 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:46.132+0000 7ff015194700 1 --2- 172.21.15.196:0/3249888235 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7feffc038d80 0x7feffc03b240 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:46.134 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:46.132+0000 7ff015194700 1 --2- 172.21.15.196:0/3249888235 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff010100f00 0x7ff01019d600 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:46.134 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:46.132+0000 7ff015194700 1 --2- 172.21.15.196:0/3249888235 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff01019db40 0x7ff0101976d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:46.134 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:46.132+0000 7ff015194700 1 -- 172.21.15.196:0/3249888235 >> 172.21.15.196:0/3249888235 conn(0x7ff0100fc770 msgr2=0x7ff0100fdb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:46.135 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:46.132+0000 7ff015194700 1 -- 172.21.15.196:0/3249888235 shutdown_connections 2024-03-18T15:59:46.135 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:46.132+0000 7ff015194700 1 -- 172.21.15.196:0/3249888235 wait complete. 2024-03-18T15:59:46.135 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 2 2024-03-18T15:59:46.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:46 smithi196 ceph-mon[36307]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:59:46.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:46 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:46.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:46 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T15:59:46.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:46 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:59:46.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:46 smithi196 ceph-mon[36307]: Deploying daemon mon.b on smithi194 2024-03-18T15:59:46.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:46 smithi196 ceph-mon[36307]: from='client.? 172.21.15.196:0/3249888235' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-18T15:59:46.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:46 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:59:47.016 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:46 smithi039 ceph-mon[31721]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:59:47.016 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:46 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:47.016 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:46 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T15:59:47.016 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:46 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:59:47.016 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:46 smithi039 ceph-mon[31721]: Deploying daemon mon.b on smithi194 2024-03-18T15:59:47.016 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:46 smithi039 ceph-mon[31721]: from='client.? 172.21.15.196:0/3249888235' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-18T15:59:47.016 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:46 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:59:47.732 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-18T15:59:47.732 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph mon dump -f json 2024-03-18T15:59:47.977 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T15:59:48.977 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:48 smithi196 ceph-mon[36307]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:59:49.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:48 smithi039 ceph-mon[31721]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:59:49.462 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:49.460+0000 7f691cf3e700 1 -- 172.21.15.196:0/3014925356 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f6910005670 con 0x7f6918102010 2024-03-18T15:59:49.462 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:49.461+0000 7f69201a4700 1 -- 172.21.15.196:0/3014925356 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6918102010 msgr2=0x7f69181023f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:49.463 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:49.461+0000 7f69201a4700 1 --2- 172.21.15.196:0/3014925356 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6918102010 0x7f69181023f0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6910009910 tx=0x7f6910009c20 comp rx=0 tx=0).stop 2024-03-18T15:59:49.463 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:49.461+0000 7f69201a4700 1 -- 172.21.15.196:0/3014925356 shutdown_connections 2024-03-18T15:59:49.463 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:49.461+0000 7f69201a4700 1 --2- 172.21.15.196:0/3014925356 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6918102010 0x7f69181023f0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:49.463 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:49.461+0000 7f69201a4700 1 -- 172.21.15.196:0/3014925356 >> 172.21.15.196:0/3014925356 conn(0x7f69180fd8a0 msgr2=0x7f69180ffcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:49.463 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:49.461+0000 7f69201a4700 1 -- 172.21.15.196:0/3014925356 shutdown_connections 2024-03-18T15:59:49.463 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:49.461+0000 7f69201a4700 1 -- 172.21.15.196:0/3014925356 wait complete. 2024-03-18T15:59:49.465 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:49.461+0000 7f69201a4700 1 Processor -- start 2024-03-18T15:59:49.465 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:49.461+0000 7f69201a4700 1 -- start start 2024-03-18T15:59:49.465 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:49.462+0000 7f69201a4700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6918102010 0x7f691819f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:49.465 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:49.462+0000 7f69201a4700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f691819fe10 0x7f6918199a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:49.465 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:49.462+0000 7f69201a4700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f6918075150 con 0x7f691819fe10 2024-03-18T15:59:49.465 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:49.462+0000 7f69201a4700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f6918074fd0 con 0x7f6918102010 2024-03-18T15:59:49.465 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:49.462+0000 7f691df40700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6918102010 0x7f691819f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:49.465 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:49.462+0000 7f691df40700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6918102010 0x7f691819f8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.196:52638/0 (socket says 172.21.15.196:52638) 2024-03-18T15:59:49.465 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:49.462+0000 7f691df40700 1 -- 172.21.15.196:0/3656513348 learned_addr learned my addr 172.21.15.196:0/3656513348 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-18T15:59:49.466 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:49.462+0000 7f691df40700 1 -- 172.21.15.196:0/3656513348 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6918102010 msgr2=0x7f691819f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-18T15:59:49.466 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:49.462+0000 7f691df40700 1 -- 172.21.15.196:0/3656513348 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6918102010 msgr2=0x7f691819f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-18T15:59:49.466 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:49.462+0000 7f691df40700 1 --2- 172.21.15.196:0/3656513348 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6918102010 0x7f691819f8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-18T15:59:49.466 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:49.462+0000 7f691df40700 1 --2- 172.21.15.196:0/3656513348 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6918102010 0x7f691819f8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-18T15:59:49.466 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:49.462+0000 7f691d73f700 1 --2- 172.21.15.196:0/3656513348 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f691819fe10 0x7f6918199a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:49.466 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:49.463+0000 7f691d73f700 1 -- 172.21.15.196:0/3656513348 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6918102010 msgr2=0x7f691819f8d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T15:59:49.466 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:49.463+0000 7f691d73f700 1 --2- 172.21.15.196:0/3656513348 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6918102010 0x7f691819f8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:49.466 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:49.463+0000 7f691d73f700 1 -- 172.21.15.196:0/3656513348 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69181a0450 con 0x7f691819fe10 2024-03-18T15:59:49.466 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:49.463+0000 7f691d73f700 1 --2- 172.21.15.196:0/3656513348 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f691819fe10 0x7f6918199a90 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f690800b4d0 tx=0x7f690800b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:49.466 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:49.463+0000 7f690effd700 1 -- 172.21.15.196:0/3656513348 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f6908011630 con 0x7f691819fe10 2024-03-18T15:59:49.466 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:49.463+0000 7f69201a4700 1 -- 172.21.15.196:0/3656513348 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6918199fd0 con 0x7f691819fe10 2024-03-18T15:59:49.466 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:49.464+0000 7f69201a4700 1 -- 172.21.15.196:0/3656513348 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f691819a560 con 0x7f691819fe10 2024-03-18T15:59:49.466 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:49.464+0000 7f690effd700 1 -- 172.21.15.196:0/3656513348 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6908011cb0 con 0x7f691819fe10 2024-03-18T15:59:49.466 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:49.464+0000 7f690effd700 1 -- 172.21.15.196:0/3656513348 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f690801ae10 con 0x7f691819fe10 2024-03-18T15:59:49.468 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:49.464+0000 7f69201a4700 1 -- 172.21.15.196:0/3656513348 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68fc005330 con 0x7f691819fe10 2024-03-18T15:59:49.468 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:49.465+0000 7f690effd700 1 -- 172.21.15.196:0/3656513348 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f6908018020 con 0x7f691819fe10 2024-03-18T15:59:49.468 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:49.465+0000 7f690effd700 1 --2- 172.21.15.196:0/3656513348 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f6904038d60 0x7f690403b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:49.468 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:49.465+0000 7f690effd700 1 -- 172.21.15.196:0/3656513348 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f690804bce0 con 0x7f691819fe10 2024-03-18T15:59:49.468 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:49.466+0000 7f691df40700 1 --2- 172.21.15.196:0/3656513348 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f6904038d60 0x7f690403b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:49.468 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:49.467+0000 7f691df40700 1 --2- 172.21.15.196:0/3656513348 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f6904038d60 0x7f690403b220 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f691000b5c0 tx=0x7f6910009650 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:49.482 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:49.481+0000 7f690effd700 1 -- 172.21.15.196:0/3656513348 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f690801f020 con 0x7f691819fe10 2024-03-18T15:59:49.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 podman[36181]: 2024-03-18 15:59:49.386173334 +0000 UTC m=+0.120513556 container create 171673f257b832d7b4f3919b261fcc05d3f32115e8c9b4c0eb2bffa122103658 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mon-b, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD) 2024-03-18T15:59:49.665 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:49.663+0000 7f690effd700 1 -- 172.21.15.196:0/3656513348 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6908010970 con 0x7f691819fe10 2024-03-18T15:59:49.760 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:49.758+0000 7f69201a4700 1 -- 172.21.15.196:0/3656513348 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f68fc0058f0 con 0x7f691819fe10 2024-03-18T15:59:49.781 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 podman[36181]: 2024-03-18 15:59:49.605225205 +0000 UTC m=+0.339565437 container init 171673f257b832d7b4f3919b261fcc05d3f32115e8c9b4c0eb2bffa122103658 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mon-b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, ceph=True, GIT_CLEAN=True) 2024-03-18T15:59:49.781 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 podman[36181]: 2024-03-18 15:59:49.608298665 +0000 UTC m=+0.342638894 container start 171673f257b832d7b4f3919b261fcc05d3f32115e8c9b4c0eb2bffa122103658 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mon-b, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2) 2024-03-18T15:59:49.782 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: set uid:gid to 167:167 (ceph:ceph) 2024-03-18T15:59:49.782 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev), process ceph-mon, pid 2 2024-03-18T15:59:49.782 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: pidfile_write: ignore empty --pid-file 2024-03-18T15:59:49.782 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: load: jerasure load: lrc 2024-03-18T15:59:49.782 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: RocksDB version: 7.9.2 2024-03-18T15:59:49.782 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Git sha 0 2024-03-18T15:59:49.782 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Compile date 2024-03-15 00:33:34 2024-03-18T15:59:49.782 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: DB SUMMARY 2024-03-18T15:59:49.782 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: DB Session ID: P1L3QRB5SNQ3J2TTLAQ0 2024-03-18T15:59:49.782 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: CURRENT file: CURRENT 2024-03-18T15:59:49.782 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: IDENTITY file: IDENTITY 2024-03-18T15:59:49.782 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-18T15:59:49.782 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-03-18T15:59:49.783 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-03-18T15:59:49.783 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.error_if_exists: 0 2024-03-18T15:59:49.783 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.create_if_missing: 0 2024-03-18T15:59:49.783 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.paranoid_checks: 1 2024-03-18T15:59:49.783 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-18T15:59:49.783 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-18T15:59:49.783 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-18T15:59:49.783 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.env: 0x561217919c60 2024-03-18T15:59:49.783 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.fs: PosixFileSystem 2024-03-18T15:59:49.783 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.info_log: 0x56121900bac0 2024-03-18T15:59:49.783 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.max_file_opening_threads: 16 2024-03-18T15:59:49.785 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.statistics: (nil) 2024-03-18T15:59:49.785 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.use_fsync: 0 2024-03-18T15:59:49.785 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.max_log_file_size: 0 2024-03-18T15:59:49.785 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-18T15:59:49.786 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-18T15:59:49.786 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.keep_log_file_num: 1000 2024-03-18T15:59:49.786 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.recycle_log_file_num: 0 2024-03-18T15:59:49.786 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.allow_fallocate: 1 2024-03-18T15:59:49.786 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.allow_mmap_reads: 0 2024-03-18T15:59:49.786 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.allow_mmap_writes: 0 2024-03-18T15:59:49.786 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.use_direct_reads: 0 2024-03-18T15:59:49.786 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-18T15:59:49.786 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.create_missing_column_families: 0 2024-03-18T15:59:49.786 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.db_log_dir: 2024-03-18T15:59:49.786 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.wal_dir: 2024-03-18T15:59:49.786 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-18T15:59:49.786 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-18T15:59:49.786 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-18T15:59:49.787 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-18T15:59:49.787 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-18T15:59:49.787 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-18T15:59:49.787 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.advise_random_on_open: 1 2024-03-18T15:59:49.787 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.db_write_buffer_size: 0 2024-03-18T15:59:49.787 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.write_buffer_manager: 0x561218fde1e0 2024-03-18T15:59:49.787 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-18T15:59:49.787 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-18T15:59:49.787 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-18T15:59:49.787 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.rate_limiter: (nil) 2024-03-18T15:59:49.787 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-18T15:59:49.787 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.wal_recovery_mode: 2 2024-03-18T15:59:49.787 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.enable_thread_tracking: 0 2024-03-18T15:59:49.787 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.enable_pipelined_write: 0 2024-03-18T15:59:49.788 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.unordered_write: 0 2024-03-18T15:59:49.788 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-18T15:59:49.788 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-18T15:59:49.788 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-18T15:59:49.788 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-18T15:59:49.788 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.row_cache: None 2024-03-18T15:59:49.788 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.wal_filter: None 2024-03-18T15:59:49.788 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-18T15:59:49.788 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.allow_ingest_behind: 0 2024-03-18T15:59:49.788 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.two_write_queues: 0 2024-03-18T15:59:49.788 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.manual_wal_flush: 0 2024-03-18T15:59:49.788 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.wal_compression: 0 2024-03-18T15:59:49.788 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.atomic_flush: 0 2024-03-18T15:59:49.788 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-18T15:59:49.789 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-18T15:59:49.789 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-18T15:59:49.789 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.log_readahead_size: 0 2024-03-18T15:59:49.789 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-18T15:59:49.789 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.best_efforts_recovery: 0 2024-03-18T15:59:49.789 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-18T15:59:49.789 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-18T15:59:49.789 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.allow_data_in_errors: 0 2024-03-18T15:59:49.789 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.db_host_id: __hostname__ 2024-03-18T15:59:49.789 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-18T15:59:49.789 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.max_background_jobs: 2 2024-03-18T15:59:49.789 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.max_background_compactions: -1 2024-03-18T15:59:49.789 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.max_subcompactions: 1 2024-03-18T15:59:49.789 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-18T15:59:49.789 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-18T15:59:49.790 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-18T15:59:49.790 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.max_total_wal_size: 0 2024-03-18T15:59:49.790 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-18T15:59:49.790 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-18T15:59:49.790 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-18T15:59:49.790 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-18T15:59:49.790 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.max_open_files: -1 2024-03-18T15:59:49.790 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.bytes_per_sync: 0 2024-03-18T15:59:49.790 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-18T15:59:49.790 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-18T15:59:49.790 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.compaction_readahead_size: 0 2024-03-18T15:59:49.810 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.max_background_flushes: -1 2024-03-18T15:59:49.810 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Compression algorithms supported: 2024-03-18T15:59:49.810 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-18T15:59:49.810 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: kZSTD supported: 0 2024-03-18T15:59:49.810 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: kXpressCompression supported: 0 2024-03-18T15:59:49.811 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: kLZ4HCCompression supported: 1 2024-03-18T15:59:49.811 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: kZlibCompression supported: 1 2024-03-18T15:59:49.811 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: kSnappyCompression supported: 1 2024-03-18T15:59:49.811 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: kLZ4Compression supported: 1 2024-03-18T15:59:49.811 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: kBZip2Compression supported: 0 2024-03-18T15:59:49.811 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-18T15:59:49.811 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-18T15:59:49.811 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-03-18T15:59:49.811 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-18T15:59:49.811 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-18T15:59:49.811 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.merge_operator: 2024-03-18T15:59:49.811 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.compaction_filter: None 2024-03-18T15:59:49.811 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.compaction_filter_factory: None 2024-03-18T15:59:49.811 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.sst_partitioner_factory: None 2024-03-18T15:59:49.812 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-18T15:59:49.812 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-18T15:59:49.812 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56121900bbc0) 2024-03-18T15:59:49.812 INFO:journalctl@ceph.mon.b.smithi194.stdout: cache_index_and_filter_blocks: 1 2024-03-18T15:59:49.812 INFO:journalctl@ceph.mon.b.smithi194.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-18T15:59:49.812 INFO:journalctl@ceph.mon.b.smithi194.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-18T15:59:49.812 INFO:journalctl@ceph.mon.b.smithi194.stdout: pin_top_level_index_and_filter: 1 2024-03-18T15:59:49.812 INFO:journalctl@ceph.mon.b.smithi194.stdout: index_type: 0 2024-03-18T15:59:49.813 INFO:journalctl@ceph.mon.b.smithi194.stdout: data_block_index_type: 0 2024-03-18T15:59:49.813 INFO:journalctl@ceph.mon.b.smithi194.stdout: index_shortening: 1 2024-03-18T15:59:49.813 INFO:journalctl@ceph.mon.b.smithi194.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-18T15:59:49.813 INFO:journalctl@ceph.mon.b.smithi194.stdout: checksum: 4 2024-03-18T15:59:49.813 INFO:journalctl@ceph.mon.b.smithi194.stdout: no_block_cache: 0 2024-03-18T15:59:49.813 INFO:journalctl@ceph.mon.b.smithi194.stdout: block_cache: 0x561219011090 2024-03-18T15:59:49.813 INFO:journalctl@ceph.mon.b.smithi194.stdout: block_cache_name: BinnedLRUCache 2024-03-18T15:59:49.813 INFO:journalctl@ceph.mon.b.smithi194.stdout: block_cache_options: 2024-03-18T15:59:49.813 INFO:journalctl@ceph.mon.b.smithi194.stdout: capacity : 536870912 2024-03-18T15:59:49.813 INFO:journalctl@ceph.mon.b.smithi194.stdout: num_shard_bits : 4 2024-03-18T15:59:49.813 INFO:journalctl@ceph.mon.b.smithi194.stdout: strict_capacity_limit : 0 2024-03-18T15:59:49.813 INFO:journalctl@ceph.mon.b.smithi194.stdout: high_pri_pool_ratio: 0.000 2024-03-18T15:59:49.813 INFO:journalctl@ceph.mon.b.smithi194.stdout: block_cache_compressed: (nil) 2024-03-18T15:59:49.813 INFO:journalctl@ceph.mon.b.smithi194.stdout: persistent_cache: (nil) 2024-03-18T15:59:49.813 INFO:journalctl@ceph.mon.b.smithi194.stdout: block_size: 4096 2024-03-18T15:59:49.814 INFO:journalctl@ceph.mon.b.smithi194.stdout: block_size_deviation: 10 2024-03-18T15:59:49.814 INFO:journalctl@ceph.mon.b.smithi194.stdout: block_restart_interval: 16 2024-03-18T15:59:49.814 INFO:journalctl@ceph.mon.b.smithi194.stdout: index_block_restart_interval: 1 2024-03-18T15:59:49.814 INFO:journalctl@ceph.mon.b.smithi194.stdout: metadata_block_size: 4096 2024-03-18T15:59:49.814 INFO:journalctl@ceph.mon.b.smithi194.stdout: partition_filters: 0 2024-03-18T15:59:49.814 INFO:journalctl@ceph.mon.b.smithi194.stdout: use_delta_encoding: 1 2024-03-18T15:59:49.814 INFO:journalctl@ceph.mon.b.smithi194.stdout: filter_policy: bloomfilter 2024-03-18T15:59:49.814 INFO:journalctl@ceph.mon.b.smithi194.stdout: whole_key_filtering: 1 2024-03-18T15:59:49.814 INFO:journalctl@ceph.mon.b.smithi194.stdout: verify_compression: 0 2024-03-18T15:59:49.814 INFO:journalctl@ceph.mon.b.smithi194.stdout: read_amp_bytes_per_bit: 0 2024-03-18T15:59:49.814 INFO:journalctl@ceph.mon.b.smithi194.stdout: format_version: 5 2024-03-18T15:59:49.814 INFO:journalctl@ceph.mon.b.smithi194.stdout: enable_index_compression: 1 2024-03-18T15:59:49.814 INFO:journalctl@ceph.mon.b.smithi194.stdout: block_align: 0 2024-03-18T15:59:49.814 INFO:journalctl@ceph.mon.b.smithi194.stdout: max_auto_readahead_size: 262144 2024-03-18T15:59:49.815 INFO:journalctl@ceph.mon.b.smithi194.stdout: prepopulate_block_cache: 0 2024-03-18T15:59:49.815 INFO:journalctl@ceph.mon.b.smithi194.stdout: initial_auto_readahead_size: 8192 2024-03-18T15:59:49.815 INFO:journalctl@ceph.mon.b.smithi194.stdout: num_file_reads_for_auto_readahead: 2 2024-03-18T15:59:49.815 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.write_buffer_size: 33554432 2024-03-18T15:59:49.815 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.max_write_buffer_number: 2 2024-03-18T15:59:49.815 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.compression: NoCompression 2024-03-18T15:59:49.815 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.bottommost_compression: Disabled 2024-03-18T15:59:49.815 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.prefix_extractor: nullptr 2024-03-18T15:59:49.815 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-18T15:59:49.815 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.num_levels: 7 2024-03-18T15:59:49.815 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-18T15:59:49.815 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-18T15:59:49.815 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-18T15:59:49.815 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-18T15:59:49.816 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-18T15:59:49.816 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-18T15:59:49.816 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-18T15:59:49.816 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-18T15:59:49.816 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-18T15:59:49.816 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-18T15:59:49.816 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-18T15:59:49.816 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-18T15:59:49.816 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-18T15:59:49.816 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.compression_opts.level: 32767 2024-03-18T15:59:49.816 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.compression_opts.strategy: 0 2024-03-18T15:59:49.816 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-18T15:59:49.816 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-18T15:59:49.816 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-18T15:59:49.816 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-18T15:59:49.817 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.compression_opts.enabled: false 2024-03-18T15:59:49.817 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-18T15:59:49.817 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-18T15:59:49.817 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-18T15:59:49.817 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-18T15:59:49.817 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.target_file_size_base: 67108864 2024-03-18T15:59:49.817 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-18T15:59:49.817 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-18T15:59:49.817 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-18T15:59:49.817 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-18T15:59:49.817 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-18T15:59:49.817 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-18T15:59:49.817 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-18T15:59:49.817 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-18T15:59:49.817 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-18T15:59:49.818 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-18T15:59:49.818 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-18T15:59:49.818 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-18T15:59:49.818 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-18T15:59:49.818 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-18T15:59:49.818 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.arena_block_size: 1048576 2024-03-18T15:59:49.818 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-18T15:59:49.818 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-18T15:59:49.818 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.disable_auto_compactions: 0 2024-03-18T15:59:49.818 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-18T15:59:49.818 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-18T15:59:49.818 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-18T15:59:49.818 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-18T15:59:49.818 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-18T15:59:49.819 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-18T15:59:49.819 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-18T15:59:49.819 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-18T15:59:49.819 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-18T15:59:49.819 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-18T15:59:49.819 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.table_properties_collectors: 2024-03-18T15:59:49.819 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.inplace_update_support: 0 2024-03-18T15:59:49.819 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-18T15:59:49.819 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-18T15:59:49.819 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-18T15:59:49.819 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-18T15:59:49.819 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.bloom_locality: 0 2024-03-18T15:59:49.819 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.max_successive_merges: 0 2024-03-18T15:59:49.819 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-18T15:59:49.819 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.paranoid_file_checks: 0 2024-03-18T15:59:49.820 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.force_consistency_checks: 1 2024-03-18T15:59:49.820 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.report_bg_io_stats: 0 2024-03-18T15:59:49.820 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.ttl: 2592000 2024-03-18T15:59:49.820 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-18T15:59:49.820 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-18T15:59:49.820 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-18T15:59:49.820 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.enable_blob_files: false 2024-03-18T15:59:49.820 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.min_blob_size: 0 2024-03-18T15:59:49.820 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.blob_file_size: 268435456 2024-03-18T15:59:49.820 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-18T15:59:49.820 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-18T15:59:49.821 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-18T15:59:49.821 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-18T15:59:49.821 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-18T15:59:49.821 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.blob_file_starting_level: 0 2024-03-18T15:59:49.821 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-18T15:59:49.821 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-18T15:59:49.821 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-18T15:59:49.821 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 59a94952-fb20-48a9-877e-5b316c5f2473 2024-03-18T15:59:49.822 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710777589638278, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-18T15:59:49.822 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-18T15:59:49.822 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710777589638707, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1710777589, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "59a94952-fb20-48a9-877e-5b316c5f2473", "db_session_id": "P1L3QRB5SNQ3J2TTLAQ0", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-18T15:59:49.822 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: EVENT_LOG_v1 {"time_micros": 1710777589638786, "job": 1, "event": "recovery_finished"} 2024-03-18T15:59:49.822 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-18T15:59:49.822 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-18T15:59:49.822 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x561219104000 2024-03-18T15:59:49.822 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: DB pointer 0x5612190ee000 2024-03-18T15:59:49.822 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-18T15:59:49.823 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-18T15:59:49.823 INFO:journalctl@ceph.mon.b.smithi194.stdout: ** DB Stats ** 2024-03-18T15:59:49.823 INFO:journalctl@ceph.mon.b.smithi194.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-18T15:59:49.823 INFO:journalctl@ceph.mon.b.smithi194.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-18T15:59:49.823 INFO:journalctl@ceph.mon.b.smithi194.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-18T15:59:49.823 INFO:journalctl@ceph.mon.b.smithi194.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-18T15:59:49.823 INFO:journalctl@ceph.mon.b.smithi194.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-18T15:59:49.823 INFO:journalctl@ceph.mon.b.smithi194.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-18T15:59:49.823 INFO:journalctl@ceph.mon.b.smithi194.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-18T15:59:49.823 INFO:journalctl@ceph.mon.b.smithi194.stdout: 2024-03-18T15:59:49.823 INFO:journalctl@ceph.mon.b.smithi194.stdout: ** Compaction Stats [default] ** 2024-03-18T15:59:49.823 INFO:journalctl@ceph.mon.b.smithi194.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-18T15:59:49.823 INFO:journalctl@ceph.mon.b.smithi194.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-18T15:59:49.824 INFO:journalctl@ceph.mon.b.smithi194.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-18T15:59:49.824 INFO:journalctl@ceph.mon.b.smithi194.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-18T15:59:49.824 INFO:journalctl@ceph.mon.b.smithi194.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-18T15:59:49.824 INFO:journalctl@ceph.mon.b.smithi194.stdout: 2024-03-18T15:59:49.824 INFO:journalctl@ceph.mon.b.smithi194.stdout: ** Compaction Stats [default] ** 2024-03-18T15:59:49.824 INFO:journalctl@ceph.mon.b.smithi194.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-18T15:59:49.824 INFO:journalctl@ceph.mon.b.smithi194.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-18T15:59:49.824 INFO:journalctl@ceph.mon.b.smithi194.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-18T15:59:49.824 INFO:journalctl@ceph.mon.b.smithi194.stdout: 2024-03-18T15:59:49.824 INFO:journalctl@ceph.mon.b.smithi194.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-18T15:59:49.824 INFO:journalctl@ceph.mon.b.smithi194.stdout: 2024-03-18T15:59:49.824 INFO:journalctl@ceph.mon.b.smithi194.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-18T15:59:49.824 INFO:journalctl@ceph.mon.b.smithi194.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-18T15:59:49.824 INFO:journalctl@ceph.mon.b.smithi194.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-18T15:59:49.824 INFO:journalctl@ceph.mon.b.smithi194.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-18T15:59:49.825 INFO:journalctl@ceph.mon.b.smithi194.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-18T15:59:49.825 INFO:journalctl@ceph.mon.b.smithi194.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-18T15:59:49.825 INFO:journalctl@ceph.mon.b.smithi194.stdout: Cumulative compaction: 0.00 GB write, 0.66 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-18T15:59:49.825 INFO:journalctl@ceph.mon.b.smithi194.stdout: Interval compaction: 0.00 GB write, 0.66 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-18T15:59:49.825 INFO:journalctl@ceph.mon.b.smithi194.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-18T15:59:49.825 INFO:journalctl@ceph.mon.b.smithi194.stdout: Block cache BinnedLRUCache@0x561219011090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.3e-05 secs_since: 0 2024-03-18T15:59:49.825 INFO:journalctl@ceph.mon.b.smithi194.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-18T15:59:49.825 INFO:journalctl@ceph.mon.b.smithi194.stdout: 2024-03-18T15:59:49.825 INFO:journalctl@ceph.mon.b.smithi194.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-18T15:59:49.825 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-03-18T15:59:49.826 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: using public_addr v2:172.21.15.194:0/0 -> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] 2024-03-18T15:59:49.826 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: starting mon.b rank -1 at public addrs [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] at bind addrs [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 1892d8fc-e540-11ee-95c9-87774f69a715 2024-03-18T15:59:49.826 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: mon.b@-1(???) e0 preinit fsid 1892d8fc-e540-11ee-95c9-87774f69a715 2024-03-18T15:59:49.826 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: mon.b@-1(synchronizing).mds e1 new map 2024-03-18T15:59:49.826 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: mon.b@-1(synchronizing).mds e1 print_map 2024-03-18T15:59:49.826 INFO:journalctl@ceph.mon.b.smithi194.stdout: e1 2024-03-18T15:59:49.826 INFO:journalctl@ceph.mon.b.smithi194.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-18T15:59:49.826 INFO:journalctl@ceph.mon.b.smithi194.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-18T15:59:49.826 INFO:journalctl@ceph.mon.b.smithi194.stdout: legacy client fscid: -1 2024-03-18T15:59:49.826 INFO:journalctl@ceph.mon.b.smithi194.stdout: 2024-03-18T15:59:49.826 INFO:journalctl@ceph.mon.b.smithi194.stdout: No filesystems configured 2024-03-18T15:59:49.826 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-18T15:59:49.826 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-18T15:59:49.827 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-18T15:59:49.827 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-18T15:59:49.827 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-18T15:59:49.827 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-18T15:59:49.827 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-18T15:59:49.827 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-18T15:59:49.827 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-18T15:59:49.827 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-18T15:59:49.827 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:59:49.827 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:49.827 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:59:49.827 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: Added host smithi194 2024-03-18T15:59:49.827 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:49.827 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:49.828 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:59:49.828 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:49.828 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:59:49.828 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T15:59:49.828 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:59:49.828 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:49.828 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:49.828 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:49.828 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:49.828 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi194", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:59:49.828 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:59:49.828 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:59:49.828 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: Updating smithi194:/etc/ceph/ceph.conf 2024-03-18T15:59:49.828 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: Updating smithi194:/var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T15:59:49.828 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: Updating smithi194:/etc/ceph/ceph.client.admin.keyring 2024-03-18T15:59:49.829 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: Updating smithi194:/var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-18T15:59:49.829 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi196", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:59:49.829 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:59:49.829 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:49.829 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:49.829 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:49.829 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: Deploying cephadm binary to smithi196 2024-03-18T15:59:49.829 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:59:49.829 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:59:49.829 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:49.829 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: Added host smithi196 2024-03-18T15:59:49.829 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:59:49.829 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:49.829 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:49.830 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:59:49.830 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:49.830 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:59:49.830 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T15:59:49.830 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:59:49.830 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:49.830 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:49.830 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:49.830 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:49.831 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi196", "name": "osd_memory_target"}]: dispatch 2024-03-18T15:59:49.831 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:59:49.831 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:59:49.831 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: Updating smithi196:/etc/ceph/ceph.conf 2024-03-18T15:59:49.831 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: Updating smithi196:/var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T15:59:49.831 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: Updating smithi196:/etc/ceph/ceph.client.admin.keyring 2024-03-18T15:59:49.831 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: Updating smithi196:/var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/config/ceph.client.admin.keyring 2024-03-18T15:59:49.831 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:49.831 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:49.831 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:49.831 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/3277886616' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-18T15:59:49.831 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:59:49.831 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/3277886616' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-18T15:59:49.832 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: osdmap e4: 0 total, 0 up, 0 in 2024-03-18T15:59:49.832 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:59:49.832 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:49.832 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:59:49.832 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:59:49.832 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T15:59:49.832 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:49.832 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T15:59:49.832 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:59:49.832 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi039:172.21.15.39=a;smithi194:172.21.15.194=b;smithi196:172.21.15.196=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T15:59:49.832 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: Saving service mon spec with placement smithi039:172.21.15.39=a;smithi194:172.21.15.194=b;smithi196:172.21.15.196=c;count:3 2024-03-18T15:59:49.832 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: Deploying daemon mon.c on smithi196 2024-03-18T15:59:49.832 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:59:49.832 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:59:49.833 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: from='client.? 172.21.15.196:0/119451341' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-18T15:59:49.833 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T15:59:49.833 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: mon.a calling monitor election 2024-03-18T15:59:49.834 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:59:49.834 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:59:49.834 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:59:49.834 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:59:49.834 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: mon.c calling monitor election 2024-03-18T15:59:49.834 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:59:49.834 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:59:49.834 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:59:49.834 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:59:49.834 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-18T15:59:49.834 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: monmap epoch 2 2024-03-18T15:59:49.834 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: fsid 1892d8fc-e540-11ee-95c9-87774f69a715 2024-03-18T15:59:49.835 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: last_changed 2024-03-18T15:59:40.606884+0000 2024-03-18T15:59:49.835 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: created 2024-03-18T15:57:32.305376+0000 2024-03-18T15:59:49.835 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: min_mon_release 19 (squid) 2024-03-18T15:59:49.835 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: election_strategy: 1 2024-03-18T15:59:49.835 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: 0: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.a 2024-03-18T15:59:49.835 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: 1: [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] mon.c 2024-03-18T15:59:49.835 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: fsmap 2024-03-18T15:59:49.835 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: osdmap e4: 0 total, 0 up, 0 in 2024-03-18T15:59:49.835 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: mgrmap e13: x(active, since 60s) 2024-03-18T15:59:49.835 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: overall HEALTH_OK 2024-03-18T15:59:49.836 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:49.836 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:49.836 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:59:49.836 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:49.836 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T15:59:49.836 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:59:49.836 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: Deploying daemon mon.b on smithi194 2024-03-18T15:59:49.836 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: from='client.? 172.21.15.196:0/3249888235' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-18T15:59:49.836 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:59:49.836 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:59:49.836 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 ceph-mon[36200]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-18T15:59:49.836 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 bash[36181]: 171673f257b832d7b4f3919b261fcc05d3f32115e8c9b4c0eb2bffa122103658 2024-03-18T15:59:49.836 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:49 smithi194 systemd[1]: Started Ceph mon.b for 1892d8fc-e540-11ee-95c9-87774f69a715. 2024-03-18T15:59:54.687 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:54.685+0000 7f690effd700 1 -- 172.21.15.196:0/3656513348 <== mon.0 v2:172.21.15.39:3300/0 8 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1283 (secure 0 0 0) 0x7f6908010ba0 con 0x7f691819fe10 2024-03-18T15:59:54.687 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-18T15:59:54.688 INFO:teuthology.orchestra.run.smithi196.stdout:{"epoch":3,"fsid":"1892d8fc-e540-11ee-95c9-87774f69a715","modified":"2024-03-18T15:59:49.662052Z","created":"2024-03-18T15:57:32.305376Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:3300","nonce":0},{"type":"v1","addr":"172.21.15.39:6789","nonce":0}]},"addr":"172.21.15.39:6789/0","public_addr":"172.21.15.39:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-18T15:59:54.690 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:54.689+0000 7f69201a4700 1 -- 172.21.15.196:0/3656513348 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f6904038d60 msgr2=0x7f690403b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:54.690 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:54.689+0000 7f69201a4700 1 --2- 172.21.15.196:0/3656513348 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f6904038d60 0x7f690403b220 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f691000b5c0 tx=0x7f6910009650 comp rx=0 tx=0).stop 2024-03-18T15:59:54.690 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:54.689+0000 7f69201a4700 1 -- 172.21.15.196:0/3656513348 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f691819fe10 msgr2=0x7f6918199a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:54.690 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:54.689+0000 7f69201a4700 1 --2- 172.21.15.196:0/3656513348 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f691819fe10 0x7f6918199a90 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f690800b4d0 tx=0x7f690800b890 comp rx=0 tx=0).stop 2024-03-18T15:59:54.690 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:54.689+0000 7f69201a4700 1 -- 172.21.15.196:0/3656513348 shutdown_connections 2024-03-18T15:59:54.691 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:54.689+0000 7f69201a4700 1 --2- 172.21.15.196:0/3656513348 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f6904038d60 0x7f690403b220 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:54.691 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:54.689+0000 7f69201a4700 1 --2- 172.21.15.196:0/3656513348 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6918102010 0x7f691819f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:54.691 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:54.689+0000 7f69201a4700 1 --2- 172.21.15.196:0/3656513348 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f691819fe10 0x7f6918199a90 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:54.691 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:54.689+0000 7f69201a4700 1 -- 172.21.15.196:0/3656513348 >> 172.21.15.196:0/3656513348 conn(0x7f69180fd8a0 msgr2=0x7f69180fec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:54.691 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:54.689+0000 7f69201a4700 1 -- 172.21.15.196:0/3656513348 shutdown_connections 2024-03-18T15:59:54.691 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T15:59:54.689+0000 7f69201a4700 1 -- 172.21.15.196:0/3656513348 wait complete. 2024-03-18T15:59:54.691 INFO:teuthology.orchestra.run.smithi196.stderr:dumped monmap epoch 3 2024-03-18T15:59:54.965 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:54 smithi196 ceph-mon[36307]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:59:54.965 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:54 smithi196 ceph-mon[36307]: mon.c calling monitor election 2024-03-18T15:59:54.965 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:54 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T15:59:54.965 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:54 smithi196 ceph-mon[36307]: mon.a calling monitor election 2024-03-18T15:59:54.965 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:54 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:59:54.965 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:54 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:59:54.965 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:54 smithi196 ceph-mon[36307]: from='client.? 172.21.15.196:0/3656513348' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-18T15:59:54.966 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:54 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:59:54.966 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:54 smithi196 ceph-mon[36307]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:59:54.966 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:54 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:59:54.966 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:54 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:59:54.966 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:54 smithi196 ceph-mon[36307]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:59:54.966 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:54 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:59:54.966 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:54 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:59:54.966 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:54 smithi196 ceph-mon[36307]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-18T15:59:54.966 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:54 smithi196 ceph-mon[36307]: monmap epoch 3 2024-03-18T15:59:54.966 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:54 smithi196 ceph-mon[36307]: fsid 1892d8fc-e540-11ee-95c9-87774f69a715 2024-03-18T15:59:54.966 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:54 smithi196 ceph-mon[36307]: last_changed 2024-03-18T15:59:49.662052+0000 2024-03-18T15:59:54.966 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:54 smithi196 ceph-mon[36307]: created 2024-03-18T15:57:32.305376+0000 2024-03-18T15:59:54.966 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:54 smithi196 ceph-mon[36307]: min_mon_release 19 (squid) 2024-03-18T15:59:54.967 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:54 smithi196 ceph-mon[36307]: election_strategy: 1 2024-03-18T15:59:54.967 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:54 smithi196 ceph-mon[36307]: 0: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.a 2024-03-18T15:59:54.967 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:54 smithi196 ceph-mon[36307]: 1: [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] mon.c 2024-03-18T15:59:54.967 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:54 smithi196 ceph-mon[36307]: 2: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-03-18T15:59:54.967 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:54 smithi196 ceph-mon[36307]: fsmap 2024-03-18T15:59:54.967 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:54 smithi196 ceph-mon[36307]: osdmap e4: 0 total, 0 up, 0 in 2024-03-18T15:59:54.967 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:54 smithi196 ceph-mon[36307]: mgrmap e13: x(active, since 69s) 2024-03-18T15:59:54.967 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:54 smithi196 ceph-mon[36307]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-18T15:59:54.967 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:54 smithi196 ceph-mon[36307]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-18T15:59:54.967 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:54 smithi196 ceph-mon[36307]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-18T15:59:54.967 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:54 smithi196 ceph-mon[36307]: mon.b (rank 2) addr [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] is down (out of quorum) 2024-03-18T15:59:54.967 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:54 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:54.967 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:54 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:54.967 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:54 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:54.968 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:54 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:55.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:54 smithi039 ceph-mon[31721]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:59:55.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:54 smithi039 ceph-mon[31721]: mon.c calling monitor election 2024-03-18T15:59:55.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:54 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T15:59:55.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:54 smithi039 ceph-mon[31721]: mon.a calling monitor election 2024-03-18T15:59:55.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:54 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:59:55.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:54 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:59:55.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:54 smithi039 ceph-mon[31721]: from='client.? 172.21.15.196:0/3656513348' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-18T15:59:55.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:54 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:59:55.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:54 smithi039 ceph-mon[31721]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:59:55.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:54 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:59:55.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:54 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:59:55.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:54 smithi039 ceph-mon[31721]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:59:55.088 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:54 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:59:55.088 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:54 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:59:55.088 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:54 smithi039 ceph-mon[31721]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-18T15:59:55.088 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:54 smithi039 ceph-mon[31721]: monmap epoch 3 2024-03-18T15:59:55.088 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:54 smithi039 ceph-mon[31721]: fsid 1892d8fc-e540-11ee-95c9-87774f69a715 2024-03-18T15:59:55.088 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:54 smithi039 ceph-mon[31721]: last_changed 2024-03-18T15:59:49.662052+0000 2024-03-18T15:59:55.088 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:54 smithi039 ceph-mon[31721]: created 2024-03-18T15:57:32.305376+0000 2024-03-18T15:59:55.088 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:54 smithi039 ceph-mon[31721]: min_mon_release 19 (squid) 2024-03-18T15:59:55.088 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:54 smithi039 ceph-mon[31721]: election_strategy: 1 2024-03-18T15:59:55.088 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:54 smithi039 ceph-mon[31721]: 0: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.a 2024-03-18T15:59:55.088 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:54 smithi039 ceph-mon[31721]: 1: [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] mon.c 2024-03-18T15:59:55.088 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:54 smithi039 ceph-mon[31721]: 2: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-03-18T15:59:55.088 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:54 smithi039 ceph-mon[31721]: fsmap 2024-03-18T15:59:55.088 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:54 smithi039 ceph-mon[31721]: osdmap e4: 0 total, 0 up, 0 in 2024-03-18T15:59:55.088 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:54 smithi039 ceph-mon[31721]: mgrmap e13: x(active, since 69s) 2024-03-18T15:59:55.088 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:54 smithi039 ceph-mon[31721]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-18T15:59:55.089 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:54 smithi039 ceph-mon[31721]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-18T15:59:55.089 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:54 smithi039 ceph-mon[31721]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-18T15:59:55.089 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:54 smithi039 ceph-mon[31721]: mon.b (rank 2) addr [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] is down (out of quorum) 2024-03-18T15:59:55.089 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:54 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:55.089 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:54 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:55.089 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:54 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:55.089 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:54 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:55.258 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-18T15:59:55.259 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph config generate-minimal-conf 2024-03-18T15:59:55.698 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T15:59:56.030 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:55 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:59:56.030 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:55 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:56.030 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:55 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:59:56.227 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:55 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:59:56.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:55 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:56.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:55 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:59:56.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:56 smithi194 ceph-mon[36200]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:59:56.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:56 smithi194 ceph-mon[36200]: mon.c calling monitor election 2024-03-18T15:59:56.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:56 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T15:59:56.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:56 smithi194 ceph-mon[36200]: mon.a calling monitor election 2024-03-18T15:59:56.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:56 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:59:56.991 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:56 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T15:59:56.991 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:56 smithi194 ceph-mon[36200]: from='client.? 172.21.15.196:0/3656513348' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-18T15:59:56.991 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:56 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:59:56.991 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:56 smithi194 ceph-mon[36200]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:59:56.991 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:56 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:59:56.991 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:56 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:59:56.991 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:56 smithi194 ceph-mon[36200]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:59:56.991 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:56 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:59:56.991 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:56 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:59:56.991 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:56 smithi194 ceph-mon[36200]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-18T15:59:56.991 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:56 smithi194 ceph-mon[36200]: monmap epoch 3 2024-03-18T15:59:56.991 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:56 smithi194 ceph-mon[36200]: fsid 1892d8fc-e540-11ee-95c9-87774f69a715 2024-03-18T15:59:56.992 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:56 smithi194 ceph-mon[36200]: last_changed 2024-03-18T15:59:49.662052+0000 2024-03-18T15:59:56.992 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:56 smithi194 ceph-mon[36200]: created 2024-03-18T15:57:32.305376+0000 2024-03-18T15:59:56.992 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:56 smithi194 ceph-mon[36200]: min_mon_release 19 (squid) 2024-03-18T15:59:56.992 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:56 smithi194 ceph-mon[36200]: election_strategy: 1 2024-03-18T15:59:56.992 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:56 smithi194 ceph-mon[36200]: 0: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.a 2024-03-18T15:59:56.992 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:56 smithi194 ceph-mon[36200]: 1: [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] mon.c 2024-03-18T15:59:56.992 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:56 smithi194 ceph-mon[36200]: 2: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-03-18T15:59:56.992 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:56 smithi194 ceph-mon[36200]: fsmap 2024-03-18T15:59:56.992 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:56 smithi194 ceph-mon[36200]: osdmap e4: 0 total, 0 up, 0 in 2024-03-18T15:59:56.992 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:56 smithi194 ceph-mon[36200]: mgrmap e13: x(active, since 69s) 2024-03-18T15:59:56.992 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:56 smithi194 ceph-mon[36200]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-18T15:59:56.992 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:56 smithi194 ceph-mon[36200]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-18T15:59:56.992 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:56 smithi194 ceph-mon[36200]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-18T15:59:56.993 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:56 smithi194 ceph-mon[36200]: mon.b (rank 2) addr [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] is down (out of quorum) 2024-03-18T15:59:56.993 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:56 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:56.993 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:56 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:56.993 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:56 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:56.993 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:56 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:56.993 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:56 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T15:59:56.993 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:56 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:56.993 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:56 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:59:57.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:57 smithi039 ceph-mon[31721]: mon.b calling monitor election 2024-03-18T15:59:57.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:57 smithi039 ceph-mon[31721]: Updating smithi039:/etc/ceph/ceph.conf 2024-03-18T15:59:57.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:57 smithi039 ceph-mon[31721]: Updating smithi194:/etc/ceph/ceph.conf 2024-03-18T15:59:57.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:57 smithi039 ceph-mon[31721]: Updating smithi196:/etc/ceph/ceph.conf 2024-03-18T15:59:57.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:57 smithi039 ceph-mon[31721]: mon.b calling monitor election 2024-03-18T15:59:57.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:57 smithi039 ceph-mon[31721]: mon.c calling monitor election 2024-03-18T15:59:57.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:57 smithi039 ceph-mon[31721]: mon.a calling monitor election 2024-03-18T15:59:57.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:57 smithi039 ceph-mon[31721]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-18T15:59:57.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:57 smithi039 ceph-mon[31721]: monmap epoch 3 2024-03-18T15:59:57.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:57 smithi039 ceph-mon[31721]: fsid 1892d8fc-e540-11ee-95c9-87774f69a715 2024-03-18T15:59:57.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:57 smithi039 ceph-mon[31721]: last_changed 2024-03-18T15:59:49.662052+0000 2024-03-18T15:59:57.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:57 smithi039 ceph-mon[31721]: created 2024-03-18T15:57:32.305376+0000 2024-03-18T15:59:57.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:57 smithi039 ceph-mon[31721]: min_mon_release 19 (squid) 2024-03-18T15:59:57.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:57 smithi039 ceph-mon[31721]: election_strategy: 1 2024-03-18T15:59:57.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:57 smithi039 ceph-mon[31721]: 0: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.a 2024-03-18T15:59:57.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:57 smithi039 ceph-mon[31721]: 1: [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] mon.c 2024-03-18T15:59:57.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:57 smithi039 ceph-mon[31721]: 2: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-03-18T15:59:57.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:57 smithi039 ceph-mon[31721]: fsmap 2024-03-18T15:59:57.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:57 smithi039 ceph-mon[31721]: osdmap e4: 0 total, 0 up, 0 in 2024-03-18T15:59:57.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:57 smithi039 ceph-mon[31721]: mgrmap e13: x(active, since 71s) 2024-03-18T15:59:57.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:57 smithi039 ceph-mon[31721]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-18T15:59:57.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:57 smithi039 ceph-mon[31721]: Cluster is now healthy 2024-03-18T15:59:57.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:57 smithi039 ceph-mon[31721]: overall HEALTH_OK 2024-03-18T15:59:57.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:57 smithi039 ceph-mon[31721]: Updating smithi196:/var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T15:59:57.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:57 smithi039 ceph-mon[31721]: Updating smithi194:/var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T15:59:57.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:57 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:57.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:57 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:57.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:57 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:57.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:57 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:57.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:57 smithi039 ceph-mon[31721]: Updating smithi039:/var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T15:59:57.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:57 smithi196 ceph-mon[36307]: mon.b calling monitor election 2024-03-18T15:59:57.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:57 smithi196 ceph-mon[36307]: Updating smithi039:/etc/ceph/ceph.conf 2024-03-18T15:59:57.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:57 smithi196 ceph-mon[36307]: Updating smithi194:/etc/ceph/ceph.conf 2024-03-18T15:59:57.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:57 smithi196 ceph-mon[36307]: Updating smithi196:/etc/ceph/ceph.conf 2024-03-18T15:59:57.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:57 smithi196 ceph-mon[36307]: mon.b calling monitor election 2024-03-18T15:59:57.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:57 smithi196 ceph-mon[36307]: mon.c calling monitor election 2024-03-18T15:59:57.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:57 smithi196 ceph-mon[36307]: mon.a calling monitor election 2024-03-18T15:59:57.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:57 smithi196 ceph-mon[36307]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-18T15:59:57.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:57 smithi196 ceph-mon[36307]: monmap epoch 3 2024-03-18T15:59:57.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:57 smithi196 ceph-mon[36307]: fsid 1892d8fc-e540-11ee-95c9-87774f69a715 2024-03-18T15:59:57.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:57 smithi196 ceph-mon[36307]: last_changed 2024-03-18T15:59:49.662052+0000 2024-03-18T15:59:57.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:57 smithi196 ceph-mon[36307]: created 2024-03-18T15:57:32.305376+0000 2024-03-18T15:59:57.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:57 smithi196 ceph-mon[36307]: min_mon_release 19 (squid) 2024-03-18T15:59:57.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:57 smithi196 ceph-mon[36307]: election_strategy: 1 2024-03-18T15:59:57.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:57 smithi196 ceph-mon[36307]: 0: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.a 2024-03-18T15:59:57.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:57 smithi196 ceph-mon[36307]: 1: [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] mon.c 2024-03-18T15:59:57.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:57 smithi196 ceph-mon[36307]: 2: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-03-18T15:59:57.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:57 smithi196 ceph-mon[36307]: fsmap 2024-03-18T15:59:57.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:57 smithi196 ceph-mon[36307]: osdmap e4: 0 total, 0 up, 0 in 2024-03-18T15:59:57.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:57 smithi196 ceph-mon[36307]: mgrmap e13: x(active, since 71s) 2024-03-18T15:59:57.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:57 smithi196 ceph-mon[36307]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-18T15:59:57.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:57 smithi196 ceph-mon[36307]: Cluster is now healthy 2024-03-18T15:59:57.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:57 smithi196 ceph-mon[36307]: overall HEALTH_OK 2024-03-18T15:59:57.730 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:57 smithi196 ceph-mon[36307]: Updating smithi196:/var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T15:59:57.730 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:57 smithi196 ceph-mon[36307]: Updating smithi194:/var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T15:59:57.730 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:57 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:57.730 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:57 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:57.730 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:57 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:57.730 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:57 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:57.730 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:57 smithi196 ceph-mon[36307]: Updating smithi039:/var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T15:59:57.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:57 smithi194 ceph-mon[36200]: mon.b calling monitor election 2024-03-18T15:59:57.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:57 smithi194 ceph-mon[36200]: Updating smithi039:/etc/ceph/ceph.conf 2024-03-18T15:59:57.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:57 smithi194 ceph-mon[36200]: Updating smithi194:/etc/ceph/ceph.conf 2024-03-18T15:59:57.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:57 smithi194 ceph-mon[36200]: Updating smithi196:/etc/ceph/ceph.conf 2024-03-18T15:59:57.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:57 smithi194 ceph-mon[36200]: mon.b calling monitor election 2024-03-18T15:59:57.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:57 smithi194 ceph-mon[36200]: mon.c calling monitor election 2024-03-18T15:59:57.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:57 smithi194 ceph-mon[36200]: mon.a calling monitor election 2024-03-18T15:59:57.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:57 smithi194 ceph-mon[36200]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-18T15:59:57.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:57 smithi194 ceph-mon[36200]: monmap epoch 3 2024-03-18T15:59:57.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:57 smithi194 ceph-mon[36200]: fsid 1892d8fc-e540-11ee-95c9-87774f69a715 2024-03-18T15:59:57.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:57 smithi194 ceph-mon[36200]: last_changed 2024-03-18T15:59:49.662052+0000 2024-03-18T15:59:57.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:57 smithi194 ceph-mon[36200]: created 2024-03-18T15:57:32.305376+0000 2024-03-18T15:59:57.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:57 smithi194 ceph-mon[36200]: min_mon_release 19 (squid) 2024-03-18T15:59:57.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:57 smithi194 ceph-mon[36200]: election_strategy: 1 2024-03-18T15:59:57.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:57 smithi194 ceph-mon[36200]: 0: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.a 2024-03-18T15:59:57.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:57 smithi194 ceph-mon[36200]: 1: [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] mon.c 2024-03-18T15:59:57.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:57 smithi194 ceph-mon[36200]: 2: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.b 2024-03-18T15:59:57.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:57 smithi194 ceph-mon[36200]: fsmap 2024-03-18T15:59:57.742 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:57 smithi194 ceph-mon[36200]: osdmap e4: 0 total, 0 up, 0 in 2024-03-18T15:59:57.742 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:57 smithi194 ceph-mon[36200]: mgrmap e13: x(active, since 71s) 2024-03-18T15:59:57.742 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:57 smithi194 ceph-mon[36200]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-18T15:59:57.742 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:57 smithi194 ceph-mon[36200]: Cluster is now healthy 2024-03-18T15:59:57.742 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:57 smithi194 ceph-mon[36200]: overall HEALTH_OK 2024-03-18T15:59:57.742 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:57 smithi194 ceph-mon[36200]: Updating smithi196:/var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T15:59:57.742 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:57 smithi194 ceph-mon[36200]: Updating smithi194:/var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T15:59:57.742 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:57 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:57.742 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:57 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:57.742 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:57 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:57.742 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:57 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:57.742 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:57 smithi194 ceph-mon[36200]: Updating smithi039:/var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/config/ceph.conf 2024-03-18T15:59:57.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:57.752+0000 7f5f07e00700 1 -- 172.21.15.39:0/807093031 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5f00102e60 msgr2=0x7f5f00103240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:57.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:57.752+0000 7f5f07e00700 1 --2- 172.21.15.39:0/807093031 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5f00102e60 0x7f5f00103240 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f5ef4009910 tx=0x7f5ef4009c20 comp rx=0 tx=0).stop 2024-03-18T15:59:57.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:57.752+0000 7f5f07e00700 1 -- 172.21.15.39:0/807093031 shutdown_connections 2024-03-18T15:59:57.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:57.752+0000 7f5f07e00700 1 --2- 172.21.15.39:0/807093031 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5f00102e60 0x7f5f00103240 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:57.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:57.752+0000 7f5f07e00700 1 -- 172.21.15.39:0/807093031 >> 172.21.15.39:0/807093031 conn(0x7f5f00077810 msgr2=0x7f5f000feae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:57.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:57.752+0000 7f5f07e00700 1 -- 172.21.15.39:0/807093031 shutdown_connections 2024-03-18T15:59:57.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:57.752+0000 7f5f07e00700 1 -- 172.21.15.39:0/807093031 wait complete. 2024-03-18T15:59:57.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:57.754+0000 7f5f07e00700 1 Processor -- start 2024-03-18T15:59:57.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:57.754+0000 7f5f07e00700 1 -- start start 2024-03-18T15:59:57.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:57.754+0000 7f5f07e00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5f00102e60 0x7f5f0019f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:57.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:57.754+0000 7f5f07e00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5f0019fe70 0x7f5f00199b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:57.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:57.754+0000 7f5f07e00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5f0019a080 0x7f5f0019a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:57.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:57.754+0000 7f5f07e00700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f5f0010c300 con 0x7f5f00102e60 2024-03-18T15:59:57.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:57.754+0000 7f5f07e00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f5f0010c180 con 0x7f5f0019fe70 2024-03-18T15:59:57.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:57.754+0000 7f5f07e00700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f5f0010c480 con 0x7f5f0019a080 2024-03-18T15:59:57.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:57.755+0000 7f5f05400700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5f00102e60 0x7f5f0019f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:57.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:57.755+0000 7f5f05400700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5f00102e60 0x7f5f0019f930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51384/0 (socket says 172.21.15.39:51384) 2024-03-18T15:59:57.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:57.755+0000 7f5f05400700 1 -- 172.21.15.39:0/2044607013 learned_addr learned my addr 172.21.15.39:0/2044607013 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T15:59:57.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:57.755+0000 7f5f05e00700 1 --2- 172.21.15.39:0/2044607013 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5f0019a080 0x7f5f0019a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:57.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:57.755+0000 7f5f04a00700 1 --2- 172.21.15.39:0/2044607013 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5f0019fe70 0x7f5f00199b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:57.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:57.755+0000 7f5f05400700 1 -- 172.21.15.39:0/2044607013 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5f0019a080 msgr2=0x7f5f0019a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:57.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:57.755+0000 7f5f05400700 1 --2- 172.21.15.39:0/2044607013 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5f0019a080 0x7f5f0019a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:57.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:57.755+0000 7f5f05400700 1 -- 172.21.15.39:0/2044607013 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5f0019fe70 msgr2=0x7f5f00199b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:57.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:57.755+0000 7f5f05400700 1 --2- 172.21.15.39:0/2044607013 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5f0019fe70 0x7f5f00199b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:57.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:57.755+0000 7f5f05400700 1 -- 172.21.15.39:0/2044607013 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f001a6800 con 0x7f5f00102e60 2024-03-18T15:59:57.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:57.756+0000 7f5f05400700 1 --2- 172.21.15.39:0/2044607013 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5f00102e60 0x7f5f0019f930 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f5ef4009fd0 tx=0x7f5ef400bde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:57.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:57.756+0000 7f5efe000700 1 -- 172.21.15.39:0/2044607013 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ef4026070 con 0x7f5f00102e60 2024-03-18T15:59:57.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:57.756+0000 7f5efe000700 1 -- 172.21.15.39:0/2044607013 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5ef4005da0 con 0x7f5f00102e60 2024-03-18T15:59:57.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:57.756+0000 7f5efe000700 1 -- 172.21.15.39:0/2044607013 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ef40038e0 con 0x7f5f00102e60 2024-03-18T15:59:57.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:57.756+0000 7f5f07e00700 1 -- 172.21.15.39:0/2044607013 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f001a69d0 con 0x7f5f00102e60 2024-03-18T15:59:57.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:57.756+0000 7f5f07e00700 1 -- 172.21.15.39:0/2044607013 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f001a6e60 con 0x7f5f00102e60 2024-03-18T15:59:57.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:57.757+0000 7f5efe000700 1 -- 172.21.15.39:0/2044607013 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f5ef4029070 con 0x7f5f00102e60 2024-03-18T15:59:57.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:57.758+0000 7f5f07e00700 1 -- 172.21.15.39:0/2044607013 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f00063900 con 0x7f5f00102e60 2024-03-18T15:59:57.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:57.758+0000 7f5efe000700 1 --2- 172.21.15.39:0/2044607013 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f5ee8039450 0x7f5ee803b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T15:59:57.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:57.758+0000 7f5efe000700 1 -- 172.21.15.39:0/2044607013 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f5ef404cbd0 con 0x7f5f00102e60 2024-03-18T15:59:57.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:57.758+0000 7f5f04a00700 1 --2- 172.21.15.39:0/2044607013 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f5ee8039450 0x7f5ee803b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T15:59:57.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:57.759+0000 7f5f04a00700 1 --2- 172.21.15.39:0/2044607013 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f5ee8039450 0x7f5ee803b910 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5eec009ad0 tx=0x7f5eec009210 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T15:59:57.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:57.764+0000 7f5efe000700 1 -- 172.21.15.39:0/2044607013 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f5ef4013070 con 0x7f5f00102e60 2024-03-18T15:59:57.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:57.993+0000 7f5f07e00700 1 -- 172.21.15.39:0/2044607013 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f5f0002db70 con 0x7f5f00102e60 2024-03-18T15:59:57.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:57.994+0000 7f5efe000700 1 -- 172.21.15.39:0/2044607013 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+275 (secure 0 0 0) 0x7f5ef4030090 con 0x7f5f00102e60 2024-03-18T15:59:57.995 INFO:teuthology.orchestra.run.smithi039.stdout:# minimal ceph.conf for 1892d8fc-e540-11ee-95c9-87774f69a715 2024-03-18T15:59:57.996 INFO:teuthology.orchestra.run.smithi039.stdout:[global] 2024-03-18T15:59:57.996 INFO:teuthology.orchestra.run.smithi039.stdout: fsid = 1892d8fc-e540-11ee-95c9-87774f69a715 2024-03-18T15:59:57.996 INFO:teuthology.orchestra.run.smithi039.stdout: mon_host = [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] 2024-03-18T15:59:57.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:57.998+0000 7f5f07e00700 1 -- 172.21.15.39:0/2044607013 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f5ee8039450 msgr2=0x7f5ee803b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:57.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:57.998+0000 7f5f07e00700 1 --2- 172.21.15.39:0/2044607013 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f5ee8039450 0x7f5ee803b910 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5eec009ad0 tx=0x7f5eec009210 comp rx=0 tx=0).stop 2024-03-18T15:59:58.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:57.998+0000 7f5f07e00700 1 -- 172.21.15.39:0/2044607013 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5f00102e60 msgr2=0x7f5f0019f930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T15:59:58.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:57.998+0000 7f5f07e00700 1 --2- 172.21.15.39:0/2044607013 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5f00102e60 0x7f5f0019f930 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f5ef4009fd0 tx=0x7f5ef400bde0 comp rx=0 tx=0).stop 2024-03-18T15:59:58.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:57.998+0000 7f5f07e00700 1 -- 172.21.15.39:0/2044607013 shutdown_connections 2024-03-18T15:59:58.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:57.998+0000 7f5f07e00700 1 --2- 172.21.15.39:0/2044607013 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f5ee8039450 0x7f5ee803b910 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:58.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:57.998+0000 7f5f07e00700 1 --2- 172.21.15.39:0/2044607013 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5f0019a080 0x7f5f0019a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:58.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:57.998+0000 7f5f07e00700 1 --2- 172.21.15.39:0/2044607013 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5f00102e60 0x7f5f0019f930 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:58.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:57.998+0000 7f5f07e00700 1 --2- 172.21.15.39:0/2044607013 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5f0019fe70 0x7f5f00199b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T15:59:58.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:57.998+0000 7f5f07e00700 1 -- 172.21.15.39:0/2044607013 >> 172.21.15.39:0/2044607013 conn(0x7f5f00077810 msgr2=0x7f5f00108160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T15:59:58.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:57.998+0000 7f5f07e00700 1 -- 172.21.15.39:0/2044607013 shutdown_connections 2024-03-18T15:59:58.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T15:59:57.998+0000 7f5f07e00700 1 -- 172.21.15.39:0/2044607013 wait complete. 2024-03-18T15:59:58.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:58 smithi039 ceph-mon[31721]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:59:58.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:58 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:59:58.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:58 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:58.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:58 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:58.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:58 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:58.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:58 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/2044607013' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:59:58.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:58 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:58.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:58 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:58.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:58 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:58.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:58 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:58.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:58 smithi039 ceph-mon[31721]: Reconfiguring mon.a (unknown last config time)... 2024-03-18T15:59:58.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:58 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T15:59:58.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:58 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-18T15:59:58.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:58 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:59:58.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 15:59:58 smithi039 ceph-mon[31721]: Reconfiguring daemon mon.a on smithi039 2024-03-18T15:59:58.673 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-18T15:59:58.673 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-03-18T15:59:58.674 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/etc/ceph/ceph.conf 2024-03-18T15:59:58.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:58 smithi196 ceph-mon[36307]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:59:58.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:58 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:59:58.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:58 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:58.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:58 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:58.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:58 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:58.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:58 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/2044607013' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:59:58.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:58 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:58.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:58 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:58.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:58 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:58.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:58 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:58.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:58 smithi196 ceph-mon[36307]: Reconfiguring mon.a (unknown last config time)... 2024-03-18T15:59:58.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:58 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T15:59:58.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:58 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-18T15:59:58.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:58 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:59:58.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 15:59:58 smithi196 ceph-mon[36307]: Reconfiguring daemon mon.a on smithi039 2024-03-18T15:59:58.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:58 smithi194 ceph-mon[36200]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T15:59:58.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:58 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T15:59:58.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:58 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:58.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:58 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:58.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:58 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:58.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:58 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/2044607013' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:59:58.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:58 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:58.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:58 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:58.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:58 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:58.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:58 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T15:59:58.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:58 smithi194 ceph-mon[36200]: Reconfiguring mon.a (unknown last config time)... 2024-03-18T15:59:58.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:58 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T15:59:58.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:58 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-18T15:59:58.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:58 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T15:59:58.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 15:59:58 smithi194 ceph-mon[36200]: Reconfiguring daemon mon.a on smithi039 2024-03-18T15:59:58.752 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-03-18T15:59:58.752 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-18T15:59:58.863 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-03-18T15:59:58.863 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/etc/ceph/ceph.conf 2024-03-18T15:59:58.886 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 15:59:58 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T15:59:58.662+0000 7f6edbc00700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-18T15:59:58.898 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-03-18T15:59:58.898 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-18T15:59:58.968 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-18T15:59:58.968 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/etc/ceph/ceph.conf 2024-03-18T15:59:58.998 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-18T15:59:58.998 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-18T15:59:59.065 INFO:tasks.cephadm:Adding mgr.x on smithi039 2024-03-18T15:59:59.065 INFO:tasks.cephadm:Adding mgr.y on smithi194 2024-03-18T15:59:59.065 INFO:tasks.cephadm:Adding mgr.z on smithi196 2024-03-18T15:59:59.065 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph orch apply mgr '3;smithi039=x;smithi194=y;smithi196=z' 2024-03-18T15:59:59.333 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T16:00:00.517 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:00 smithi196 ceph-mon[36307]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T16:00:00.517 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:00 smithi196 ceph-mon[36307]: overall HEALTH_OK 2024-03-18T16:00:00.517 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:00 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:00.517 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:00 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:00.517 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:00 smithi196 ceph-mon[36307]: Reconfiguring mon.b (monmap changed)... 2024-03-18T16:00:00.517 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:00 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T16:00:00.517 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:00 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-18T16:00:00.518 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:00 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:00:00.518 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:00 smithi196 ceph-mon[36307]: Reconfiguring daemon mon.b on smithi194 2024-03-18T16:00:00.709 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:00:00.705+0000 7f99a7841700 1 -- 172.21.15.196:0/1299597057 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f99a0103470 msgr2=0x7f99a0103890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:00:00.709 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:00:00.705+0000 7f99a7841700 1 --2- 172.21.15.196:0/1299597057 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f99a0103470 0x7f99a0103890 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f999c009910 tx=0x7f999c009c20 comp rx=0 tx=0).stop 2024-03-18T16:00:00.709 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:00:00.705+0000 7f99a7841700 1 -- 172.21.15.196:0/1299597057 shutdown_connections 2024-03-18T16:00:00.710 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:00:00.705+0000 7f99a7841700 1 --2- 172.21.15.196:0/1299597057 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f99a0103470 0x7f99a0103890 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:00.710 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:00:00.705+0000 7f99a7841700 1 -- 172.21.15.196:0/1299597057 >> 172.21.15.196:0/1299597057 conn(0x7f99a00fd3d0 msgr2=0x7f99a00ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:00:00.710 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:00:00.705+0000 7f99a7841700 1 -- 172.21.15.196:0/1299597057 shutdown_connections 2024-03-18T16:00:00.710 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:00:00.705+0000 7f99a7841700 1 -- 172.21.15.196:0/1299597057 wait complete. 2024-03-18T16:00:00.710 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:00:00.706+0000 7f99a7841700 1 Processor -- start 2024-03-18T16:00:00.710 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:00:00.706+0000 7f99a7841700 1 -- start start 2024-03-18T16:00:00.711 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:00:00.706+0000 7f99a7841700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f99a0103470 0x7f99a0197720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:00.711 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:00:00.706+0000 7f99a7841700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f99a0197c60 0x7f99a019ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:00.711 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:00:00.706+0000 7f99a7841700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f99a01980e0 0x7f99a019f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:00.711 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:00:00.706+0000 7f99a7841700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f99a0107ee0 con 0x7f99a0197c60 2024-03-18T16:00:00.711 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:00:00.706+0000 7f99a7841700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f99a0107d60 con 0x7f99a0103470 2024-03-18T16:00:00.711 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:00:00.706+0000 7f99a7841700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f99a0108060 con 0x7f99a01980e0 2024-03-18T16:00:00.712 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:00:00.706+0000 7f99a5dde700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f99a01980e0 0x7f99a019f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:00.712 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:00:00.706+0000 7f99a5dde700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f99a01980e0 0x7f99a019f260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.196:56056/0 (socket says 172.21.15.196:56056) 2024-03-18T16:00:00.712 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:00:00.706+0000 7f99a5dde700 1 -- 172.21.15.196:0/1431391097 learned_addr learned my addr 172.21.15.196:0/1431391097 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-18T16:00:00.712 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:00:00.706+0000 7f99a55dd700 1 --2- 172.21.15.196:0/1431391097 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f99a0103470 0x7f99a0197720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:00.712 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:00:00.706+0000 7f99a4ddc700 1 --2- 172.21.15.196:0/1431391097 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f99a0197c60 0x7f99a019ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:00.712 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:00:00.706+0000 7f99a5dde700 1 -- 172.21.15.196:0/1431391097 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f99a0103470 msgr2=0x7f99a0197720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:00:00.713 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:00:00.706+0000 7f99a5dde700 1 --2- 172.21.15.196:0/1431391097 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f99a0103470 0x7f99a0197720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:00.713 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:00:00.706+0000 7f99a5dde700 1 -- 172.21.15.196:0/1431391097 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f99a0197c60 msgr2=0x7f99a019ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:00:00.713 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:00:00.706+0000 7f99a5dde700 1 --2- 172.21.15.196:0/1431391097 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f99a0197c60 0x7f99a019ccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:00.713 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:00:00.706+0000 7f99a5dde700 1 -- 172.21.15.196:0/1431391097 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99a019fb30 con 0x7f99a01980e0 2024-03-18T16:00:00.713 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:00:00.707+0000 7f99a5dde700 1 --2- 172.21.15.196:0/1431391097 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f99a01980e0 0x7f99a019f260 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f999800ba50 tx=0x7f999800be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:00:00.713 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:00:00.707+0000 7f99967fc700 1 -- 172.21.15.196:0/1431391097 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9998019070 con 0x7f99a01980e0 2024-03-18T16:00:00.714 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:00:00.707+0000 7f99967fc700 1 -- 172.21.15.196:0/1431391097 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9998014440 con 0x7f99a01980e0 2024-03-18T16:00:00.714 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:00:00.707+0000 7f99a7841700 1 -- 172.21.15.196:0/1431391097 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99a019fe20 con 0x7f99a01980e0 2024-03-18T16:00:00.714 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:00:00.707+0000 7f99967fc700 1 -- 172.21.15.196:0/1431391097 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9998007590 con 0x7f99a01980e0 2024-03-18T16:00:00.714 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:00:00.707+0000 7f99a7841700 1 -- 172.21.15.196:0/1431391097 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99a01a0300 con 0x7f99a01980e0 2024-03-18T16:00:00.716 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:00:00.708+0000 7f99967fc700 1 -- 172.21.15.196:0/1431391097 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f999801c070 con 0x7f99a01980e0 2024-03-18T16:00:00.716 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:00:00.708+0000 7f99a7841700 1 -- 172.21.15.196:0/1431391097 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99a0063900 con 0x7f99a01980e0 2024-03-18T16:00:00.717 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:00:00.708+0000 7f99967fc700 1 --2- 172.21.15.196:0/1431391097 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f998c039450 0x7f998c03b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:00.717 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:00:00.708+0000 7f99967fc700 1 -- 172.21.15.196:0/1431391097 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f999804b6a0 con 0x7f99a01980e0 2024-03-18T16:00:00.717 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:00:00.709+0000 7f99a55dd700 1 --2- 172.21.15.196:0/1431391097 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f998c039450 0x7f998c03b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:00.717 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:00:00.710+0000 7f99a55dd700 1 --2- 172.21.15.196:0/1431391097 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f998c039450 0x7f998c03b910 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f999c005330 tx=0x7f999c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:00:00.717 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:00:00.713+0000 7f99967fc700 1 -- 172.21.15.196:0/1431391097 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9998014910 con 0x7f99a01980e0 2024-03-18T16:00:00.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:00 smithi194 ceph-mon[36200]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T16:00:00.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:00 smithi194 ceph-mon[36200]: overall HEALTH_OK 2024-03-18T16:00:00.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:00 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:00.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:00 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:00.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:00 smithi194 ceph-mon[36200]: Reconfiguring mon.b (monmap changed)... 2024-03-18T16:00:00.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:00 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T16:00:00.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:00 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-18T16:00:00.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:00 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:00:00.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:00 smithi194 ceph-mon[36200]: Reconfiguring daemon mon.b on smithi194 2024-03-18T16:00:00.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:00 smithi039 ceph-mon[31721]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T16:00:00.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:00 smithi039 ceph-mon[31721]: overall HEALTH_OK 2024-03-18T16:00:00.838 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:00 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:00.838 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:00 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:00.838 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:00 smithi039 ceph-mon[31721]: Reconfiguring mon.b (monmap changed)... 2024-03-18T16:00:00.838 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:00 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T16:00:00.838 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:00 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-18T16:00:00.838 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:00 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:00:00.839 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:00 smithi039 ceph-mon[31721]: Reconfiguring daemon mon.b on smithi194 2024-03-18T16:00:00.919 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:00:00.917+0000 7f99a7841700 1 -- 172.21.15.196:0/1431391097 --> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi039=x;smithi194=y;smithi196=z", "target": ["mon-mgr", ""]}) -- 0x7f99a00627a0 con 0x7f998c039450 2024-03-18T16:00:00.933 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:00:00.931+0000 7f99967fc700 1 -- 172.21.15.196:0/1431391097 <== mgr.14152 v2:172.21.15.39:6800/2749214286 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f99a00627a0 con 0x7f998c039450 2024-03-18T16:00:00.934 INFO:teuthology.orchestra.run.smithi196.stdout:Scheduled mgr update... 2024-03-18T16:00:00.935 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:00:00.934+0000 7f99a7841700 1 -- 172.21.15.196:0/1431391097 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f998c039450 msgr2=0x7f998c03b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:00:00.936 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:00:00.934+0000 7f99a7841700 1 --2- 172.21.15.196:0/1431391097 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f998c039450 0x7f998c03b910 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f999c005330 tx=0x7f999c005fb0 comp rx=0 tx=0).stop 2024-03-18T16:00:00.936 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:00:00.934+0000 7f99a7841700 1 -- 172.21.15.196:0/1431391097 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f99a01980e0 msgr2=0x7f99a019f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:00:00.936 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:00:00.934+0000 7f99a7841700 1 --2- 172.21.15.196:0/1431391097 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f99a01980e0 0x7f99a019f260 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f999800ba50 tx=0x7f999800be10 comp rx=0 tx=0).stop 2024-03-18T16:00:00.936 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:00:00.934+0000 7f99a7841700 1 -- 172.21.15.196:0/1431391097 shutdown_connections 2024-03-18T16:00:00.936 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:00:00.934+0000 7f99a7841700 1 --2- 172.21.15.196:0/1431391097 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f998c039450 0x7f998c03b910 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:00.936 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:00:00.934+0000 7f99a7841700 1 --2- 172.21.15.196:0/1431391097 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f99a01980e0 0x7f99a019f260 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:00.936 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:00:00.934+0000 7f99a7841700 1 --2- 172.21.15.196:0/1431391097 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f99a0103470 0x7f99a0197720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:00.936 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:00:00.934+0000 7f99a7841700 1 --2- 172.21.15.196:0/1431391097 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f99a0197c60 0x7f99a019ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:00.936 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:00:00.934+0000 7f99a7841700 1 -- 172.21.15.196:0/1431391097 >> 172.21.15.196:0/1431391097 conn(0x7f99a00fd3d0 msgr2=0x7f99a00fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:00:00.936 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:00:00.934+0000 7f99a7841700 1 -- 172.21.15.196:0/1431391097 shutdown_connections 2024-03-18T16:00:00.936 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:00:00.934+0000 7f99a7841700 1 -- 172.21.15.196:0/1431391097 wait complete. 2024-03-18T16:00:01.633 DEBUG:teuthology.orchestra.run.smithi194:mgr.y> sudo journalctl -f -n 0 -u ceph-1892d8fc-e540-11ee-95c9-87774f69a715@mgr.y.service 2024-03-18T16:00:01.636 DEBUG:teuthology.orchestra.run.smithi196:mgr.z> sudo journalctl -f -n 0 -u ceph-1892d8fc-e540-11ee-95c9-87774f69a715@mgr.z.service 2024-03-18T16:00:01.638 INFO:tasks.cephadm:Deploying OSDs... 2024-03-18T16:00:01.638 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-03-18T16:00:01.638 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/scratch_devs of=/dev/stdout 2024-03-18T16:00:01.697 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-18T16:00:01.697 DEBUG:teuthology.orchestra.run.smithi039:> stat /dev/vg_nvme/lv_1 2024-03-18T16:00:01.750 INFO:teuthology.orchestra.run.smithi039.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-18T16:00:01.750 INFO:teuthology.orchestra.run.smithi039.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T16:00:01.750 INFO:teuthology.orchestra.run.smithi039.stdout:Device: 5h/5d Inode: 633 Links: 1 2024-03-18T16:00:01.750 INFO:teuthology.orchestra.run.smithi039.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T16:00:01.750 INFO:teuthology.orchestra.run.smithi039.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T16:00:01.750 INFO:teuthology.orchestra.run.smithi039.stdout:Access: 2024-03-18 15:58:58.351506231 +0000 2024-03-18T16:00:01.751 INFO:teuthology.orchestra.run.smithi039.stdout:Modify: 2024-03-18 15:53:24.261182256 +0000 2024-03-18T16:00:01.751 INFO:teuthology.orchestra.run.smithi039.stdout:Change: 2024-03-18 15:53:24.261182256 +0000 2024-03-18T16:00:01.751 INFO:teuthology.orchestra.run.smithi039.stdout: Birth: 2024-03-18 15:53:24.261182256 +0000 2024-03-18T16:00:01.751 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-18T16:00:01.844 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records in 2024-03-18T16:00:01.844 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records out 2024-03-18T16:00:01.844 INFO:teuthology.orchestra.run.smithi039.stderr:512 bytes copied, 0.000552972 s, 926 kB/s 2024-03-18T16:00:01.849 DEBUG:teuthology.orchestra.run.smithi039:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-18T16:00:01.908 DEBUG:teuthology.orchestra.run.smithi039:> stat /dev/vg_nvme/lv_2 2024-03-18T16:00:01.962 INFO:teuthology.orchestra.run.smithi039.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-18T16:00:01.962 INFO:teuthology.orchestra.run.smithi039.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T16:00:01.962 INFO:teuthology.orchestra.run.smithi039.stdout:Device: 5h/5d Inode: 652 Links: 1 2024-03-18T16:00:01.962 INFO:teuthology.orchestra.run.smithi039.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T16:00:01.962 INFO:teuthology.orchestra.run.smithi039.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T16:00:01.962 INFO:teuthology.orchestra.run.smithi039.stdout:Access: 2024-03-18 15:58:58.405505146 +0000 2024-03-18T16:00:01.962 INFO:teuthology.orchestra.run.smithi039.stdout:Modify: 2024-03-18 15:53:24.853170315 +0000 2024-03-18T16:00:01.962 INFO:teuthology.orchestra.run.smithi039.stdout:Change: 2024-03-18 15:53:24.853170315 +0000 2024-03-18T16:00:01.963 INFO:teuthology.orchestra.run.smithi039.stdout: Birth: 2024-03-18 15:53:24.853170315 +0000 2024-03-18T16:00:01.963 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-18T16:00:02.046 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records in 2024-03-18T16:00:02.046 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records out 2024-03-18T16:00:02.046 INFO:teuthology.orchestra.run.smithi039.stderr:512 bytes copied, 0.000399135 s, 1.3 MB/s 2024-03-18T16:00:02.049 DEBUG:teuthology.orchestra.run.smithi039:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-18T16:00:02.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:01 smithi039 ceph-mon[31721]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi039=x;smithi194=y;smithi196=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:00:02.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:01 smithi039 ceph-mon[31721]: Saving service mgr spec with placement smithi039=x;smithi194=y;smithi196=z;count:3 2024-03-18T16:00:02.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:01 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:02.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:01 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:02.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:01 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:02.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:01 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T16:00:02.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:01 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-18T16:00:02.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:01 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:00:02.110 DEBUG:teuthology.orchestra.run.smithi039:> stat /dev/vg_nvme/lv_3 2024-03-18T16:00:02.210 INFO:teuthology.orchestra.run.smithi039.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-18T16:00:02.210 INFO:teuthology.orchestra.run.smithi039.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T16:00:02.210 INFO:teuthology.orchestra.run.smithi039.stdout:Device: 5h/5d Inode: 669 Links: 1 2024-03-18T16:00:02.211 INFO:teuthology.orchestra.run.smithi039.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T16:00:02.211 INFO:teuthology.orchestra.run.smithi039.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T16:00:02.211 INFO:teuthology.orchestra.run.smithi039.stdout:Access: 2024-03-18 15:58:58.459504059 +0000 2024-03-18T16:00:02.211 INFO:teuthology.orchestra.run.smithi039.stdout:Modify: 2024-03-18 15:53:25.445158375 +0000 2024-03-18T16:00:02.211 INFO:teuthology.orchestra.run.smithi039.stdout:Change: 2024-03-18 15:53:25.445158375 +0000 2024-03-18T16:00:02.211 INFO:teuthology.orchestra.run.smithi039.stdout: Birth: 2024-03-18 15:53:25.445158375 +0000 2024-03-18T16:00:02.211 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-18T16:00:02.227 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:01 smithi196 ceph-mon[36307]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi039=x;smithi194=y;smithi196=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:00:02.227 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:01 smithi196 ceph-mon[36307]: Saving service mgr spec with placement smithi039=x;smithi194=y;smithi196=z;count:3 2024-03-18T16:00:02.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:01 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:02.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:01 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:02.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:01 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:02.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:01 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T16:00:02.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:01 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-18T16:00:02.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:01 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:00:02.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:01 smithi194 ceph-mon[36200]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi039=x;smithi194=y;smithi196=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:00:02.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:01 smithi194 ceph-mon[36200]: Saving service mgr spec with placement smithi039=x;smithi194=y;smithi196=z;count:3 2024-03-18T16:00:02.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:01 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:02.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:01 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:02.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:01 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:02.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:01 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-18T16:00:02.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:01 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-18T16:00:02.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:01 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:00:02.305 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records in 2024-03-18T16:00:02.306 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records out 2024-03-18T16:00:02.306 INFO:teuthology.orchestra.run.smithi039.stderr:512 bytes copied, 0.000411329 s, 1.2 MB/s 2024-03-18T16:00:02.309 DEBUG:teuthology.orchestra.run.smithi039:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-18T16:00:02.373 DEBUG:teuthology.orchestra.run.smithi039:> stat /dev/vg_nvme/lv_4 2024-03-18T16:00:02.426 INFO:teuthology.orchestra.run.smithi039.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-18T16:00:02.426 INFO:teuthology.orchestra.run.smithi039.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T16:00:02.426 INFO:teuthology.orchestra.run.smithi039.stdout:Device: 5h/5d Inode: 690 Links: 1 2024-03-18T16:00:02.426 INFO:teuthology.orchestra.run.smithi039.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T16:00:02.426 INFO:teuthology.orchestra.run.smithi039.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T16:00:02.426 INFO:teuthology.orchestra.run.smithi039.stdout:Access: 2024-03-18 15:58:58.515502934 +0000 2024-03-18T16:00:02.426 INFO:teuthology.orchestra.run.smithi039.stdout:Modify: 2024-03-18 15:53:26.056146051 +0000 2024-03-18T16:00:02.426 INFO:teuthology.orchestra.run.smithi039.stdout:Change: 2024-03-18 15:53:26.056146051 +0000 2024-03-18T16:00:02.426 INFO:teuthology.orchestra.run.smithi039.stdout: Birth: 2024-03-18 15:53:26.056146051 +0000 2024-03-18T16:00:02.427 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-18T16:00:02.523 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records in 2024-03-18T16:00:02.524 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records out 2024-03-18T16:00:02.524 INFO:teuthology.orchestra.run.smithi039.stderr:512 bytes copied, 0.000466085 s, 1.1 MB/s 2024-03-18T16:00:02.528 DEBUG:teuthology.orchestra.run.smithi039:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-18T16:00:02.589 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-03-18T16:00:02.590 DEBUG:teuthology.orchestra.run.smithi194:> dd if=/scratch_devs of=/dev/stdout 2024-03-18T16:00:02.609 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-18T16:00:02.609 DEBUG:teuthology.orchestra.run.smithi194:> stat /dev/vg_nvme/lv_1 2024-03-18T16:00:02.667 INFO:teuthology.orchestra.run.smithi194.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-18T16:00:02.667 INFO:teuthology.orchestra.run.smithi194.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T16:00:02.667 INFO:teuthology.orchestra.run.smithi194.stdout:Device: 5h/5d Inode: 660 Links: 1 2024-03-18T16:00:02.667 INFO:teuthology.orchestra.run.smithi194.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T16:00:02.667 INFO:teuthology.orchestra.run.smithi194.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T16:00:02.667 INFO:teuthology.orchestra.run.smithi194.stdout:Access: 2024-03-18 15:59:49.089181840 +0000 2024-03-18T16:00:02.667 INFO:teuthology.orchestra.run.smithi194.stdout:Modify: 2024-03-18 15:52:03.229801096 +0000 2024-03-18T16:00:02.668 INFO:teuthology.orchestra.run.smithi194.stdout:Change: 2024-03-18 15:52:03.229801096 +0000 2024-03-18T16:00:02.668 INFO:teuthology.orchestra.run.smithi194.stdout: Birth: 2024-03-18 15:52:03.229801096 +0000 2024-03-18T16:00:02.668 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-18T16:00:02.736 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records in 2024-03-18T16:00:02.736 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records out 2024-03-18T16:00:02.736 INFO:teuthology.orchestra.run.smithi194.stderr:512 bytes copied, 0.000281057 s, 1.8 MB/s 2024-03-18T16:00:02.737 DEBUG:teuthology.orchestra.run.smithi194:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-18T16:00:02.794 DEBUG:teuthology.orchestra.run.smithi194:> stat /dev/vg_nvme/lv_2 2024-03-18T16:00:02.850 INFO:teuthology.orchestra.run.smithi194.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-18T16:00:02.850 INFO:teuthology.orchestra.run.smithi194.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T16:00:02.850 INFO:teuthology.orchestra.run.smithi194.stdout:Device: 5h/5d Inode: 679 Links: 1 2024-03-18T16:00:02.850 INFO:teuthology.orchestra.run.smithi194.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T16:00:02.850 INFO:teuthology.orchestra.run.smithi194.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T16:00:02.850 INFO:teuthology.orchestra.run.smithi194.stdout:Access: 2024-03-18 15:59:49.090181820 +0000 2024-03-18T16:00:02.850 INFO:teuthology.orchestra.run.smithi194.stdout:Modify: 2024-03-18 15:52:03.584793549 +0000 2024-03-18T16:00:02.850 INFO:teuthology.orchestra.run.smithi194.stdout:Change: 2024-03-18 15:52:03.584793549 +0000 2024-03-18T16:00:02.850 INFO:teuthology.orchestra.run.smithi194.stdout: Birth: 2024-03-18 15:52:03.584793549 +0000 2024-03-18T16:00:02.851 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-18T16:00:02.917 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records in 2024-03-18T16:00:02.917 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records out 2024-03-18T16:00:02.917 INFO:teuthology.orchestra.run.smithi194.stderr:512 bytes copied, 0.000202544 s, 2.5 MB/s 2024-03-18T16:00:02.918 DEBUG:teuthology.orchestra.run.smithi194:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-18T16:00:02.974 DEBUG:teuthology.orchestra.run.smithi194:> stat /dev/vg_nvme/lv_3 2024-03-18T16:00:02.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:02 smithi194 ceph-mon[36200]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T16:00:02.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:02 smithi194 ceph-mon[36200]: Reconfiguring mon.c (monmap changed)... 2024-03-18T16:00:02.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:02 smithi194 ceph-mon[36200]: Reconfiguring daemon mon.c on smithi196 2024-03-18T16:00:03.004 INFO:teuthology.orchestra.run.smithi194.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-18T16:00:03.004 INFO:teuthology.orchestra.run.smithi194.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T16:00:03.005 INFO:teuthology.orchestra.run.smithi194.stdout:Device: 5h/5d Inode: 700 Links: 1 2024-03-18T16:00:03.005 INFO:teuthology.orchestra.run.smithi194.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T16:00:03.005 INFO:teuthology.orchestra.run.smithi194.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T16:00:03.005 INFO:teuthology.orchestra.run.smithi194.stdout:Access: 2024-03-18 15:59:49.090181820 +0000 2024-03-18T16:00:03.005 INFO:teuthology.orchestra.run.smithi194.stdout:Modify: 2024-03-18 15:52:03.947785832 +0000 2024-03-18T16:00:03.005 INFO:teuthology.orchestra.run.smithi194.stdout:Change: 2024-03-18 15:52:03.947785832 +0000 2024-03-18T16:00:03.005 INFO:teuthology.orchestra.run.smithi194.stdout: Birth: 2024-03-18 15:52:03.947785832 +0000 2024-03-18T16:00:03.005 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-18T16:00:03.070 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records in 2024-03-18T16:00:03.070 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records out 2024-03-18T16:00:03.070 INFO:teuthology.orchestra.run.smithi194.stderr:512 bytes copied, 0.000169346 s, 3.0 MB/s 2024-03-18T16:00:03.071 DEBUG:teuthology.orchestra.run.smithi194:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-18T16:00:03.126 DEBUG:teuthology.orchestra.run.smithi194:> stat /dev/vg_nvme/lv_4 2024-03-18T16:00:03.181 INFO:teuthology.orchestra.run.smithi194.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-18T16:00:03.181 INFO:teuthology.orchestra.run.smithi194.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T16:00:03.181 INFO:teuthology.orchestra.run.smithi194.stdout:Device: 5h/5d Inode: 719 Links: 1 2024-03-18T16:00:03.181 INFO:teuthology.orchestra.run.smithi194.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T16:00:03.181 INFO:teuthology.orchestra.run.smithi194.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T16:00:03.181 INFO:teuthology.orchestra.run.smithi194.stdout:Access: 2024-03-18 15:59:49.090181820 +0000 2024-03-18T16:00:03.181 INFO:teuthology.orchestra.run.smithi194.stdout:Modify: 2024-03-18 15:52:04.296778412 +0000 2024-03-18T16:00:03.181 INFO:teuthology.orchestra.run.smithi194.stdout:Change: 2024-03-18 15:52:04.296778412 +0000 2024-03-18T16:00:03.181 INFO:teuthology.orchestra.run.smithi194.stdout: Birth: 2024-03-18 15:52:04.296778412 +0000 2024-03-18T16:00:03.182 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-18T16:00:03.227 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:02 smithi196 ceph-mon[36307]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T16:00:03.227 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:02 smithi196 ceph-mon[36307]: Reconfiguring mon.c (monmap changed)... 2024-03-18T16:00:03.227 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:02 smithi196 ceph-mon[36307]: Reconfiguring daemon mon.c on smithi196 2024-03-18T16:00:03.246 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records in 2024-03-18T16:00:03.247 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records out 2024-03-18T16:00:03.247 INFO:teuthology.orchestra.run.smithi194.stderr:512 bytes copied, 0.000218056 s, 2.3 MB/s 2024-03-18T16:00:03.248 DEBUG:teuthology.orchestra.run.smithi194:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-18T16:00:03.305 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-18T16:00:03.305 DEBUG:teuthology.orchestra.run.smithi196:> dd if=/scratch_devs of=/dev/stdout 2024-03-18T16:00:03.320 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-18T16:00:03.320 DEBUG:teuthology.orchestra.run.smithi196:> stat /dev/vg_nvme/lv_1 2024-03-18T16:00:03.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:02 smithi039 ceph-mon[31721]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T16:00:03.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:02 smithi039 ceph-mon[31721]: Reconfiguring mon.c (monmap changed)... 2024-03-18T16:00:03.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:02 smithi039 ceph-mon[31721]: Reconfiguring daemon mon.c on smithi196 2024-03-18T16:00:03.375 INFO:teuthology.orchestra.run.smithi196.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-18T16:00:03.375 INFO:teuthology.orchestra.run.smithi196.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T16:00:03.375 INFO:teuthology.orchestra.run.smithi196.stdout:Device: 5h/5d Inode: 663 Links: 1 2024-03-18T16:00:03.375 INFO:teuthology.orchestra.run.smithi196.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T16:00:03.375 INFO:teuthology.orchestra.run.smithi196.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T16:00:03.375 INFO:teuthology.orchestra.run.smithi196.stdout:Access: 2024-03-18 15:59:39.905285797 +0000 2024-03-18T16:00:03.376 INFO:teuthology.orchestra.run.smithi196.stdout:Modify: 2024-03-18 15:51:58.590197312 +0000 2024-03-18T16:00:03.376 INFO:teuthology.orchestra.run.smithi196.stdout:Change: 2024-03-18 15:51:58.590197312 +0000 2024-03-18T16:00:03.376 INFO:teuthology.orchestra.run.smithi196.stdout: Birth: 2024-03-18 15:51:58.590197312 +0000 2024-03-18T16:00:03.376 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-18T16:00:03.442 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records in 2024-03-18T16:00:03.442 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records out 2024-03-18T16:00:03.442 INFO:teuthology.orchestra.run.smithi196.stderr:512 bytes copied, 0.000197874 s, 2.6 MB/s 2024-03-18T16:00:03.444 DEBUG:teuthology.orchestra.run.smithi196:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-18T16:00:03.501 DEBUG:teuthology.orchestra.run.smithi196:> stat /dev/vg_nvme/lv_2 2024-03-18T16:00:03.555 INFO:teuthology.orchestra.run.smithi196.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-18T16:00:03.555 INFO:teuthology.orchestra.run.smithi196.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T16:00:03.555 INFO:teuthology.orchestra.run.smithi196.stdout:Device: 5h/5d Inode: 681 Links: 1 2024-03-18T16:00:03.555 INFO:teuthology.orchestra.run.smithi196.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T16:00:03.556 INFO:teuthology.orchestra.run.smithi196.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T16:00:03.556 INFO:teuthology.orchestra.run.smithi196.stdout:Access: 2024-03-18 15:59:39.905285797 +0000 2024-03-18T16:00:03.556 INFO:teuthology.orchestra.run.smithi196.stdout:Modify: 2024-03-18 15:51:58.944189681 +0000 2024-03-18T16:00:03.556 INFO:teuthology.orchestra.run.smithi196.stdout:Change: 2024-03-18 15:51:58.944189681 +0000 2024-03-18T16:00:03.556 INFO:teuthology.orchestra.run.smithi196.stdout: Birth: 2024-03-18 15:51:58.944189681 +0000 2024-03-18T16:00:03.556 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-18T16:00:03.622 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records in 2024-03-18T16:00:03.623 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records out 2024-03-18T16:00:03.623 INFO:teuthology.orchestra.run.smithi196.stderr:512 bytes copied, 0.000180951 s, 2.8 MB/s 2024-03-18T16:00:03.624 DEBUG:teuthology.orchestra.run.smithi196:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-18T16:00:03.682 DEBUG:teuthology.orchestra.run.smithi196:> stat /dev/vg_nvme/lv_3 2024-03-18T16:00:03.737 INFO:teuthology.orchestra.run.smithi196.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-18T16:00:03.737 INFO:teuthology.orchestra.run.smithi196.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T16:00:03.737 INFO:teuthology.orchestra.run.smithi196.stdout:Device: 5h/5d Inode: 701 Links: 1 2024-03-18T16:00:03.737 INFO:teuthology.orchestra.run.smithi196.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T16:00:03.737 INFO:teuthology.orchestra.run.smithi196.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T16:00:03.738 INFO:teuthology.orchestra.run.smithi196.stdout:Access: 2024-03-18 15:59:39.905285797 +0000 2024-03-18T16:00:03.738 INFO:teuthology.orchestra.run.smithi196.stdout:Modify: 2024-03-18 15:51:59.277182503 +0000 2024-03-18T16:00:03.738 INFO:teuthology.orchestra.run.smithi196.stdout:Change: 2024-03-18 15:51:59.277182503 +0000 2024-03-18T16:00:03.738 INFO:teuthology.orchestra.run.smithi196.stdout: Birth: 2024-03-18 15:51:59.277182503 +0000 2024-03-18T16:00:03.738 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-18T16:00:03.802 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records in 2024-03-18T16:00:03.802 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records out 2024-03-18T16:00:03.803 INFO:teuthology.orchestra.run.smithi196.stderr:512 bytes copied, 0.000169667 s, 3.0 MB/s 2024-03-18T16:00:03.803 DEBUG:teuthology.orchestra.run.smithi196:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-18T16:00:03.859 DEBUG:teuthology.orchestra.run.smithi196:> stat /dev/vg_nvme/lv_4 2024-03-18T16:00:03.913 INFO:teuthology.orchestra.run.smithi196.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-18T16:00:03.913 INFO:teuthology.orchestra.run.smithi196.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-18T16:00:03.913 INFO:teuthology.orchestra.run.smithi196.stdout:Device: 5h/5d Inode: 718 Links: 1 2024-03-18T16:00:03.914 INFO:teuthology.orchestra.run.smithi196.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T16:00:03.914 INFO:teuthology.orchestra.run.smithi196.stdout:Context: system_u:object_r:device_t:s0 2024-03-18T16:00:03.914 INFO:teuthology.orchestra.run.smithi196.stdout:Access: 2024-03-18 15:59:39.906285776 +0000 2024-03-18T16:00:03.914 INFO:teuthology.orchestra.run.smithi196.stdout:Modify: 2024-03-18 15:51:59.608175368 +0000 2024-03-18T16:00:03.914 INFO:teuthology.orchestra.run.smithi196.stdout:Change: 2024-03-18 15:51:59.608175368 +0000 2024-03-18T16:00:03.914 INFO:teuthology.orchestra.run.smithi196.stdout: Birth: 2024-03-18 15:51:59.608175368 +0000 2024-03-18T16:00:03.914 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-18T16:00:03.978 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records in 2024-03-18T16:00:03.978 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records out 2024-03-18T16:00:03.979 INFO:teuthology.orchestra.run.smithi196.stderr:512 bytes copied, 0.000192321 s, 2.7 MB/s 2024-03-18T16:00:03.980 DEBUG:teuthology.orchestra.run.smithi196:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-18T16:00:04.037 INFO:tasks.cephadm:Deploying osd.0 on smithi039 with /dev/vg_nvme/lv_4... 2024-03-18T16:00:04.037 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-18T16:00:04.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:04 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:04.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:04 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:04.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:04 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:00:04.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:04 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:00:04.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:04 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:00:04.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:04 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:04.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:04 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-18T16:00:04.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:04 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-18T16:00:04.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:04 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-18T16:00:04.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:04 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:00:04.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:04 smithi039 ceph-mon[31721]: Deploying daemon mgr.z on smithi196 2024-03-18T16:00:04.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:04 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:04.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:04 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:04.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:04 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:00:04.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:04 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:00:04.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:04 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:00:04.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:04 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:04.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:04 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-18T16:00:04.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:04 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-18T16:00:04.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:04 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-18T16:00:04.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:04 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:00:04.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:04 smithi196 ceph-mon[36307]: Deploying daemon mgr.z on smithi196 2024-03-18T16:00:04.489 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T16:00:04.489 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:04 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:04.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:04 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:04.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:04 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:00:04.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:04 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:00:04.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:04 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:00:04.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:04 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:04.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:04 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-18T16:00:04.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:04 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-18T16:00:04.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:04 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-18T16:00:04.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:04 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:00:04.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:04 smithi194 ceph-mon[36200]: Deploying daemon mgr.z on smithi196 2024-03-18T16:00:05.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:05 smithi196 ceph-mon[36307]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T16:00:05.228 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 18 16:00:05 smithi196 systemd[1]: Starting Ceph mgr.z for 1892d8fc-e540-11ee-95c9-87774f69a715... 2024-03-18T16:00:05.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:05 smithi039 ceph-mon[31721]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T16:00:05.391 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:05 smithi194 ceph-mon[36200]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T16:00:05.606 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 18 16:00:05 smithi196 podman[38155]: 2024-03-18 16:00:05.247249907 +0000 UTC m=+0.017507657 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T16:00:05.607 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 18 16:00:05 smithi196 podman[38155]: 2024-03-18 16:00:05.3609205 +0000 UTC m=+0.131178237 container create fce4512a2469ffb976389bcb604d420920bf42ecada7daf775dd69e34accc9bc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-z, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True) 2024-03-18T16:00:05.607 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 18 16:00:05 smithi196 podman[38155]: 2024-03-18 16:00:05.53052133 +0000 UTC m=+0.300779073 container init fce4512a2469ffb976389bcb604d420920bf42ecada7daf775dd69e34accc9bc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-z, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T16:00:05.607 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 18 16:00:05 smithi196 podman[38155]: 2024-03-18 16:00:05.533209193 +0000 UTC m=+0.303466929 container start fce4512a2469ffb976389bcb604d420920bf42ecada7daf775dd69e34accc9bc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-z, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-18T16:00:05.607 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 18 16:00:05 smithi196 ceph-mgr[38174]: -- 172.21.15.196:0/837511212 <== mon.1 v2:172.21.15.196:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5582287805a0 con 0x558228754800 2024-03-18T16:00:05.607 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 18 16:00:05 smithi196 bash[38155]: fce4512a2469ffb976389bcb604d420920bf42ecada7daf775dd69e34accc9bc 2024-03-18T16:00:05.977 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 18 16:00:05 smithi196 systemd[1]: Started Ceph mgr.z for 1892d8fc-e540-11ee-95c9-87774f69a715. 2024-03-18T16:00:06.478 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 18 16:00:06 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-z[38170]: 2024-03-18T16:00:06.077+0000 7fb3cc536200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T16:00:06.478 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 18 16:00:06 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-z[38170]: 2024-03-18T16:00:06.171+0000 7fb3cc536200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T16:00:06.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:06 smithi039 ceph-mon[31721]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T16:00:06.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:06 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:06.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:06 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:06.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:06 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:06.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:06 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-18T16:00:06.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:06 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-18T16:00:06.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:06 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-18T16:00:06.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:06 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:00:06.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:06 smithi039 ceph-mon[31721]: Deploying daemon mgr.y on smithi194 2024-03-18T16:00:06.978 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 18 16:00:06 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-z[38170]: 2024-03-18T16:00:06.810+0000 7fb3cc536200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T16:00:06.978 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 18 16:00:06 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-z[38170]: 2024-03-18T16:00:06.899+0000 7fb3cc536200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T16:00:06.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:06 smithi196 ceph-mon[36307]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T16:00:06.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:06 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:06.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:06 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:06.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:06 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:06.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:06 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-18T16:00:06.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:06 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-18T16:00:06.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:06 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-18T16:00:06.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:06 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:00:06.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:06 smithi196 ceph-mon[36307]: Deploying daemon mgr.y on smithi194 2024-03-18T16:00:06.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:06 smithi194 ceph-mon[36200]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T16:00:06.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:06 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:06.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:06 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:06.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:06 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:06.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:06 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-18T16:00:06.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:06 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-18T16:00:06.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:06 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-18T16:00:06.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:06 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:00:06.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:06 smithi194 ceph-mon[36200]: Deploying daemon mgr.y on smithi194 2024-03-18T16:00:07.391 INFO:teuthology.orchestra.run.smithi039.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-18T16:00:07.391 INFO:teuthology.orchestra.run.smithi039.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-18T16:00:07.391 INFO:teuthology.orchestra.run.smithi039.stderr: stderr: 10+0 records in 2024-03-18T16:00:07.391 INFO:teuthology.orchestra.run.smithi039.stderr:10+0 records out 2024-03-18T16:00:07.391 INFO:teuthology.orchestra.run.smithi039.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0398444 s, 263 MB/s 2024-03-18T16:00:07.391 INFO:teuthology.orchestra.run.smithi039.stderr:--> Zapping successful for: 2024-03-18T16:00:07.477 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 18 16:00:07 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-z[38170]: 2024-03-18T16:00:07.040+0000 7fb3cc536200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T16:00:07.478 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 18 16:00:07 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-z[38170]: 2024-03-18T16:00:07.133+0000 7fb3cc536200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T16:00:07.478 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 18 16:00:07 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-z[38170]: 2024-03-18T16:00:07.215+0000 7fb3cc536200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T16:00:07.740 INFO:journalctl@ceph.mgr.y.smithi194.stdout:Mar 18 16:00:07 smithi194 systemd[1]: Starting Ceph mgr.y for 1892d8fc-e540-11ee-95c9-87774f69a715... 2024-03-18T16:00:07.778 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 18 16:00:07 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-z[38170]: 2024-03-18T16:00:07.486+0000 7fb3cc536200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T16:00:07.778 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 18 16:00:07 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-z[38170]: 2024-03-18T16:00:07.594+0000 7fb3cc536200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T16:00:07.778 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 18 16:00:07 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-z[38170]: 2024-03-18T16:00:07.682+0000 7fb3cc536200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T16:00:07.778 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 18 16:00:07 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-z[38170]: 2024-03-18T16:00:07.776+0000 7fb3cc536200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T16:00:08.082 INFO:journalctl@ceph.mgr.y.smithi194.stdout:Mar 18 16:00:07 smithi194 podman[37581]: 2024-03-18 16:00:07.820848674 +0000 UTC m=+0.099857105 container create d24c93dbcadc45d14bcf4862cfcae33e74e10adeeae40f5f599c2e4c3724ddfa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-y, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T16:00:08.082 INFO:journalctl@ceph.mgr.y.smithi194.stdout:Mar 18 16:00:07 smithi194 podman[37581]: 2024-03-18 16:00:07.738940081 +0000 UTC m=+0.017948530 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T16:00:08.082 INFO:journalctl@ceph.mgr.y.smithi194.stdout:Mar 18 16:00:08 smithi194 podman[37581]: 2024-03-18 16:00:08.014742122 +0000 UTC m=+0.293750554 container init d24c93dbcadc45d14bcf4862cfcae33e74e10adeeae40f5f599c2e4c3724ddfa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-y, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-18T16:00:08.082 INFO:journalctl@ceph.mgr.y.smithi194.stdout:Mar 18 16:00:08 smithi194 podman[37581]: 2024-03-18 16:00:08.017328676 +0000 UTC m=+0.296337102 container start d24c93dbcadc45d14bcf4862cfcae33e74e10adeeae40f5f599c2e4c3724ddfa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-y, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T16:00:08.082 INFO:journalctl@ceph.mgr.y.smithi194.stdout:Mar 18 16:00:08 smithi194 ceph-mgr[37599]: -- 172.21.15.194:0/1352820690 <== mon.2 v2:172.21.15.194:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a7987185a0 con 0x55a7986ec800 2024-03-18T16:00:08.082 INFO:journalctl@ceph.mgr.y.smithi194.stdout:Mar 18 16:00:08 smithi194 bash[37581]: d24c93dbcadc45d14bcf4862cfcae33e74e10adeeae40f5f599c2e4c3724ddfa 2024-03-18T16:00:08.227 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 18 16:00:07 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-z[38170]: 2024-03-18T16:00:07.952+0000 7fb3cc536200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T16:00:08.245 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi039:vg_nvme/lv_4 2024-03-18T16:00:08.490 INFO:journalctl@ceph.mgr.y.smithi194.stdout:Mar 18 16:00:08 smithi194 systemd[1]: Started Ceph mgr.y for 1892d8fc-e540-11ee-95c9-87774f69a715. 2024-03-18T16:00:08.667 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T16:00:08.990 INFO:journalctl@ceph.mgr.y.smithi194.stdout:Mar 18 16:00:08 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-y[37595]: 2024-03-18T16:00:08.567+0000 7fd58a450200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-18T16:00:08.990 INFO:journalctl@ceph.mgr.y.smithi194.stdout:Mar 18 16:00:08 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-y[37595]: 2024-03-18T16:00:08.664+0000 7fd58a450200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-18T16:00:09.228 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 18 16:00:08 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-z[38170]: 2024-03-18T16:00:08.946+0000 7fb3cc536200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T16:00:09.228 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 18 16:00:09 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-z[38170]: 2024-03-18T16:00:09.100+0000 7fb3cc536200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T16:00:09.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:09 smithi196 ceph-mon[36307]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T16:00:09.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:09 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:09.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:09 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:09.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:09 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:09.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:09 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:09.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:09 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:00:09.450 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:09 smithi194 ceph-mon[36200]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T16:00:09.450 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:09 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:09.450 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:09 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:09.450 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:09 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:09.451 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:09 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:09.451 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:09 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:00:09.451 INFO:journalctl@ceph.mgr.y.smithi194.stdout:Mar 18 16:00:09 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-y[37595]: 2024-03-18T16:00:09.353+0000 7fd58a450200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-18T16:00:09.451 INFO:journalctl@ceph.mgr.y.smithi194.stdout:Mar 18 16:00:09 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-y[37595]: 2024-03-18T16:00:09.448+0000 7fd58a450200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-18T16:00:09.517 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:09 smithi039 ceph-mon[31721]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T16:00:09.517 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:09 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:09.517 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:09 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:09.517 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:09 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:09.517 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:09 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:09.517 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:09 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:00:09.727 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 18 16:00:09 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-z[38170]: 2024-03-18T16:00:09.443+0000 7fb3cc536200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T16:00:09.728 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 18 16:00:09 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-z[38170]: 2024-03-18T16:00:09.531+0000 7fb3cc536200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T16:00:09.740 INFO:journalctl@ceph.mgr.y.smithi194.stdout:Mar 18 16:00:09 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-y[37595]: 2024-03-18T16:00:09.592+0000 7fd58a450200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-18T16:00:09.740 INFO:journalctl@ceph.mgr.y.smithi194.stdout:Mar 18 16:00:09 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-y[37595]: 2024-03-18T16:00:09.686+0000 7fd58a450200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-18T16:00:10.056 INFO:journalctl@ceph.mgr.y.smithi194.stdout:Mar 18 16:00:09 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-y[37595]: 2024-03-18T16:00:09.786+0000 7fd58a450200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-18T16:00:10.056 INFO:journalctl@ceph.mgr.y.smithi194.stdout:Mar 18 16:00:10 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-y[37595]: 2024-03-18T16:00:10.054+0000 7fd58a450200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-18T16:00:10.197 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 18 16:00:09 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-z[38170]: 2024-03-18T16:00:09.728+0000 7fb3cc536200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T16:00:10.197 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 18 16:00:09 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-z[38170]: 2024-03-18T16:00:09.913+0000 7fb3cc536200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T16:00:10.343 INFO:journalctl@ceph.mgr.y.smithi194.stdout:Mar 18 16:00:10 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-y[37595]: 2024-03-18T16:00:10.162+0000 7fd58a450200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-18T16:00:10.343 INFO:journalctl@ceph.mgr.y.smithi194.stdout:Mar 18 16:00:10 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-y[37595]: 2024-03-18T16:00:10.248+0000 7fd58a450200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-18T16:00:10.343 INFO:journalctl@ceph.mgr.y.smithi194.stdout:Mar 18 16:00:10 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-y[37595]: 2024-03-18T16:00:10.341+0000 7fd58a450200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-18T16:00:10.458 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 18 16:00:10 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-z[38170]: 2024-03-18T16:00:10.195+0000 7fb3cc536200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T16:00:10.458 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 18 16:00:10 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-z[38170]: 2024-03-18T16:00:10.456+0000 7fb3cc536200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T16:00:10.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:00:10.541+0000 7f376dc00700 1 -- 172.21.15.39:0/2988721538 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3768107fb0 msgr2=0x7f376810a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:00:10.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:00:10.541+0000 7f376dc00700 1 --2- 172.21.15.39:0/2988721538 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3768107fb0 0x7f376810a420 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f376000b0a0 tx=0x7f376000b3b0 comp rx=0 tx=0).stop 2024-03-18T16:00:10.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:00:10.541+0000 7f376dc00700 1 -- 172.21.15.39:0/2988721538 shutdown_connections 2024-03-18T16:00:10.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:00:10.541+0000 7f376dc00700 1 --2- 172.21.15.39:0/2988721538 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3768107fb0 0x7f376810a420 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:10.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:00:10.541+0000 7f376dc00700 1 --2- 172.21.15.39:0/2988721538 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3768102d50 0x7f3768105140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:10.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:00:10.541+0000 7f376dc00700 1 --2- 172.21.15.39:0/2988721538 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3768105680 0x7f3768107a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:10.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:00:10.541+0000 7f376dc00700 1 -- 172.21.15.39:0/2988721538 >> 172.21.15.39:0/2988721538 conn(0x7f37680fc760 msgr2=0x7f37680feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:00:10.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:00:10.541+0000 7f376dc00700 1 -- 172.21.15.39:0/2988721538 shutdown_connections 2024-03-18T16:00:10.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:00:10.541+0000 7f376dc00700 1 -- 172.21.15.39:0/2988721538 wait complete. 2024-03-18T16:00:10.545 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:00:10.542+0000 7f376dc00700 1 Processor -- start 2024-03-18T16:00:10.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:00:10.542+0000 7f376dc00700 1 -- start start 2024-03-18T16:00:10.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:00:10.543+0000 7f376dc00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3768102d50 0x7f37681026b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:10.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:00:10.543+0000 7f376dc00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3768105680 0x7f3768100d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:10.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:00:10.543+0000 7f376dc00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3768107fb0 0x7f3768101240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:10.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:00:10.543+0000 7f376dc00700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f376810dec0 con 0x7f3768107fb0 2024-03-18T16:00:10.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:00:10.543+0000 7f376dc00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f376810dd40 con 0x7f3768105680 2024-03-18T16:00:10.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:00:10.543+0000 7f376dc00700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f376810e040 con 0x7f3768102d50 2024-03-18T16:00:10.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:00:10.543+0000 7f3766a00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3768102d50 0x7f37681026b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:10.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:00:10.543+0000 7f3766a00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3768102d50 0x7f37681026b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:42250/0 (socket says 172.21.15.39:42250) 2024-03-18T16:00:10.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:00:10.543+0000 7f3766a00700 1 -- 172.21.15.39:0/344702481 learned_addr learned my addr 172.21.15.39:0/344702481 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:00:10.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:00:10.543+0000 7f3766000700 1 --2- 172.21.15.39:0/344702481 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3768105680 0x7f3768100d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:10.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:00:10.543+0000 7f3767400700 1 --2- 172.21.15.39:0/344702481 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3768107fb0 0x7f3768101240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:10.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:00:10.543+0000 7f3766a00700 1 -- 172.21.15.39:0/344702481 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3768105680 msgr2=0x7f3768100d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:00:10.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:00:10.543+0000 7f3766a00700 1 --2- 172.21.15.39:0/344702481 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3768105680 0x7f3768100d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:10.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:00:10.543+0000 7f3766a00700 1 -- 172.21.15.39:0/344702481 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3768107fb0 msgr2=0x7f3768101240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:00:10.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:00:10.544+0000 7f3766a00700 1 --2- 172.21.15.39:0/344702481 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3768107fb0 0x7f3768101240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:10.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:00:10.544+0000 7f3766a00700 1 -- 172.21.15.39:0/344702481 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3768101af0 con 0x7f3768102d50 2024-03-18T16:00:10.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:00:10.544+0000 7f3767400700 1 --2- 172.21.15.39:0/344702481 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3768107fb0 0x7f3768101240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:00:10.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:00:10.544+0000 7f3766a00700 1 --2- 172.21.15.39:0/344702481 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3768102d50 0x7f37681026b0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f375800e9a0 tx=0x7f375800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:00:10.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:00:10.544+0000 7f375f400700 1 -- 172.21.15.39:0/344702481 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f375800cbd0 con 0x7f3768102d50 2024-03-18T16:00:10.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:00:10.544+0000 7f375f400700 1 -- 172.21.15.39:0/344702481 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f375800cd70 con 0x7f3768102d50 2024-03-18T16:00:10.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:00:10.544+0000 7f376dc00700 1 -- 172.21.15.39:0/344702481 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37681a2630 con 0x7f3768102d50 2024-03-18T16:00:10.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:00:10.544+0000 7f375f400700 1 -- 172.21.15.39:0/344702481 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3758018750 con 0x7f3768102d50 2024-03-18T16:00:10.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:00:10.544+0000 7f376dc00700 1 -- 172.21.15.39:0/344702481 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37681a2b70 con 0x7f3768102d50 2024-03-18T16:00:10.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:00:10.545+0000 7f376dc00700 1 -- 172.21.15.39:0/344702481 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3768068f90 con 0x7f3768102d50 2024-03-18T16:00:10.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:00:10.546+0000 7f375f400700 1 -- 172.21.15.39:0/344702481 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 13) ==== 46162+0+0 (secure 0 0 0) 0x7f3758010430 con 0x7f3768102d50 2024-03-18T16:00:10.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:00:10.546+0000 7f375f400700 1 --2- 172.21.15.39:0/344702481 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f374c039200 0x7f374c03b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:10.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:00:10.546+0000 7f375f400700 1 -- 172.21.15.39:0/344702481 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f375804c540 con 0x7f3768102d50 2024-03-18T16:00:10.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:00:10.546+0000 7f3766000700 1 --2- 172.21.15.39:0/344702481 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f374c039200 0x7f374c03b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:10.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:00:10.547+0000 7f3766000700 1 --2- 172.21.15.39:0/344702481 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f374c039200 0x7f374c03b6c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3754009c60 tx=0x7f3754009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:00:10.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:00:10.551+0000 7f375f400700 1 -- 172.21.15.39:0/344702481 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3758020360 con 0x7f3768102d50 2024-03-18T16:00:10.727 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 18 16:00:10 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-z[38170]: 2024-03-18T16:00:10.688+0000 7fb3cc536200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T16:00:10.739 INFO:journalctl@ceph.mgr.y.smithi194.stdout:Mar 18 16:00:10 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-y[37595]: 2024-03-18T16:00:10.518+0000 7fd58a450200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-18T16:00:10.796 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:00:10.794+0000 7f376dc00700 1 -- 172.21.15.39:0/344702481 --> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi039:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f3768062710 con 0x7f374c039200 2024-03-18T16:00:10.796 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:10 smithi039 ceph-mon[31721]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T16:00:10.796 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:10 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:10.796 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:10 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:10.796 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:10 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:10.796 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:10 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:10.796 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:10 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:00:10.797 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:10 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:00:10.797 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:10 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:10.797 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:10 smithi039 ceph-mon[31721]: Reconfiguring mgr.x (unknown last config time)... 2024-03-18T16:00:10.797 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:10 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-18T16:00:10.797 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:10 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-18T16:00:10.797 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:10 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:00:10.797 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:10 smithi039 ceph-mon[31721]: Reconfiguring daemon mgr.x on smithi039 2024-03-18T16:00:10.797 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:10 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:11.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:10 smithi196 ceph-mon[36307]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T16:00:11.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:10 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:11.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:10 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:11.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:10 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:11.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:10 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:11.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:10 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:00:11.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:10 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:00:11.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:10 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:11.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:10 smithi196 ceph-mon[36307]: Reconfiguring mgr.x (unknown last config time)... 2024-03-18T16:00:11.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:10 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-18T16:00:11.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:10 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-18T16:00:11.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:10 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:00:11.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:10 smithi196 ceph-mon[36307]: Reconfiguring daemon mgr.x on smithi039 2024-03-18T16:00:11.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:10 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:11.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:10 smithi194 ceph-mon[36200]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T16:00:11.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:10 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:11.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:10 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:11.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:10 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:11.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:10 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:11.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:10 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:00:11.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:10 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:00:11.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:10 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:11.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:10 smithi194 ceph-mon[36200]: Reconfiguring mgr.x (unknown last config time)... 2024-03-18T16:00:11.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:10 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-18T16:00:11.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:10 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-18T16:00:11.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:10 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:00:11.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:10 smithi194 ceph-mon[36200]: Reconfiguring daemon mgr.x on smithi039 2024-03-18T16:00:11.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:10 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:11.740 INFO:journalctl@ceph.mgr.y.smithi194.stdout:Mar 18 16:00:11 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-y[37595]: 2024-03-18T16:00:11.480+0000 7fd58a450200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-18T16:00:11.740 INFO:journalctl@ceph.mgr.y.smithi194.stdout:Mar 18 16:00:11 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-y[37595]: 2024-03-18T16:00:11.609+0000 7fd58a450200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-18T16:00:11.787 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 18 16:00:11 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-z[38170]: 2024-03-18T16:00:11.476+0000 7fb3cc536200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T16:00:11.983 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:11 smithi039 ceph-mon[31721]: from='client.24118 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi039:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:00:11.983 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:11 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T16:00:11.983 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:11 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T16:00:11.983 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:11 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:00:12.044 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:11 smithi196 ceph-mon[36307]: from='client.24118 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi039:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:00:12.044 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:11 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T16:00:12.044 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:11 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T16:00:12.044 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:11 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:00:12.226 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:11 smithi194 ceph-mon[36200]: from='client.24118 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi039:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:00:12.227 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:11 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T16:00:12.227 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:11 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T16:00:12.227 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:11 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:00:12.227 INFO:journalctl@ceph.mgr.y.smithi194.stdout:Mar 18 16:00:11 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-y[37595]: 2024-03-18T16:00:11.944+0000 7fd58a450200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-18T16:00:12.227 INFO:journalctl@ceph.mgr.y.smithi194.stdout:Mar 18 16:00:12 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-y[37595]: 2024-03-18T16:00:12.031+0000 7fd58a450200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-18T16:00:12.227 INFO:journalctl@ceph.mgr.y.smithi194.stdout:Mar 18 16:00:12 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-y[37595]: 2024-03-18T16:00:12.225+0000 7fd58a450200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-18T16:00:12.351 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 18 16:00:12 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-z[38170]: 2024-03-18T16:00:12.042+0000 7fb3cc536200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T16:00:12.351 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 18 16:00:12 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-z[38170]: 2024-03-18T16:00:12.139+0000 7fb3cc536200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T16:00:12.352 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 18 16:00:12 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-z[38170]: 2024-03-18T16:00:12.350+0000 7fb3cc536200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T16:00:12.490 INFO:journalctl@ceph.mgr.y.smithi194.stdout:Mar 18 16:00:12 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-y[37595]: 2024-03-18T16:00:12.409+0000 7fd58a450200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-18T16:00:12.950 INFO:journalctl@ceph.mgr.y.smithi194.stdout:Mar 18 16:00:12 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-y[37595]: 2024-03-18T16:00:12.689+0000 7fd58a450200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-18T16:00:12.950 INFO:journalctl@ceph.mgr.y.smithi194.stdout:Mar 18 16:00:12 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-y[37595]: 2024-03-18T16:00:12.948+0000 7fd58a450200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-18T16:00:13.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:00:13.203+0000 7f375f400700 1 -- 172.21.15.39:0/344702481 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mgrmap(e 14) ==== 91698+0+0 (secure 0 0 0) 0x7f3758018c70 con 0x7f3768102d50 2024-03-18T16:00:13.207 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:13 smithi039 ceph-mon[31721]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T16:00:13.207 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:13 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:13.207 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:13 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:13.207 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:13 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:00:13.207 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:13 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:00:13.207 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:13 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:00:13.207 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:13 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:13.207 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:13 smithi039 ceph-mon[31721]: Standby manager daemon z started 2024-03-18T16:00:13.207 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:13 smithi039 ceph-mon[31721]: from='mgr.? 172.21.15.196:0/778126944' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-18T16:00:13.207 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:13 smithi039 ceph-mon[31721]: from='mgr.? 172.21.15.196:0/778126944' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-18T16:00:13.207 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:13 smithi039 ceph-mon[31721]: from='mgr.? 172.21.15.196:0/778126944' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-18T16:00:13.207 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:13 smithi039 ceph-mon[31721]: from='mgr.? 172.21.15.196:0/778126944' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-18T16:00:13.240 INFO:journalctl@ceph.mgr.y.smithi194.stdout:Mar 18 16:00:13 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-y[37595]: 2024-03-18T16:00:13.182+0000 7fd58a450200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-18T16:00:13.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:13 smithi194 ceph-mon[36200]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T16:00:13.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:13 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:13.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:13 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:13.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:13 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:00:13.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:13 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:00:13.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:13 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:00:13.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:13 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:13.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:13 smithi194 ceph-mon[36200]: Standby manager daemon z started 2024-03-18T16:00:13.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:13 smithi194 ceph-mon[36200]: from='mgr.? 172.21.15.196:0/778126944' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-18T16:00:13.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:13 smithi194 ceph-mon[36200]: from='mgr.? 172.21.15.196:0/778126944' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-18T16:00:13.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:13 smithi194 ceph-mon[36200]: from='mgr.? 172.21.15.196:0/778126944' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-18T16:00:13.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:13 smithi194 ceph-mon[36200]: from='mgr.? 172.21.15.196:0/778126944' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-18T16:00:13.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:13 smithi196 ceph-mon[36307]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T16:00:13.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:13 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:13.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:13 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:13.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:13 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:00:13.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:13 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:00:13.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:13 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:00:13.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:13 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:13.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:13 smithi196 ceph-mon[36307]: Standby manager daemon z started 2024-03-18T16:00:13.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:13 smithi196 ceph-mon[36307]: from='mgr.? 172.21.15.196:0/778126944' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-18T16:00:13.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:13 smithi196 ceph-mon[36307]: from='mgr.? 172.21.15.196:0/778126944' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-18T16:00:13.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:13 smithi196 ceph-mon[36307]: from='mgr.? 172.21.15.196:0/778126944' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-18T16:00:13.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:13 smithi196 ceph-mon[36307]: from='mgr.? 172.21.15.196:0/778126944' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-18T16:00:14.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:14 smithi194 ceph-mon[36200]: mgrmap e14: x(active, since 87s), standbys: z 2024-03-18T16:00:14.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:14 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-18T16:00:14.240 INFO:journalctl@ceph.mgr.y.smithi194.stdout:Mar 18 16:00:13 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-y[37595]: 2024-03-18T16:00:13.981+0000 7fd58a450200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-18T16:00:14.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:14 smithi196 ceph-mon[36307]: mgrmap e14: x(active, since 87s), standbys: z 2024-03-18T16:00:14.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:14 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-18T16:00:14.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:14 smithi039 ceph-mon[31721]: mgrmap e14: x(active, since 87s), standbys: z 2024-03-18T16:00:14.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:14 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-18T16:00:14.857 INFO:journalctl@ceph.mgr.y.smithi194.stdout:Mar 18 16:00:14 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-y[37595]: 2024-03-18T16:00:14.549+0000 7fd58a450200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-18T16:00:14.857 INFO:journalctl@ceph.mgr.y.smithi194.stdout:Mar 18 16:00:14 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-y[37595]: 2024-03-18T16:00:14.645+0000 7fd58a450200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-18T16:00:14.857 INFO:journalctl@ceph.mgr.y.smithi194.stdout:Mar 18 16:00:14 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-y[37595]: 2024-03-18T16:00:14.855+0000 7fd58a450200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-18T16:00:15.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:00:15.231+0000 7f375f400700 1 -- 172.21.15.39:0/344702481 <== mon.1 v2:172.21.15.196:3300/0 8 ==== mgrmap(e 15) ==== 137234+0+0 (secure 0 0 0) 0x7f375805f2c0 con 0x7f3768102d50 2024-03-18T16:00:15.490 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:15 smithi039 ceph-mon[31721]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T16:00:15.490 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:15 smithi039 ceph-mon[31721]: Standby manager daemon y started 2024-03-18T16:00:15.490 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:15 smithi039 ceph-mon[31721]: from='mgr.? 172.21.15.194:0/3878194911' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-18T16:00:15.490 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:15 smithi039 ceph-mon[31721]: from='mgr.? 172.21.15.194:0/3878194911' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-18T16:00:15.490 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:15 smithi039 ceph-mon[31721]: from='mgr.? 172.21.15.194:0/3878194911' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-18T16:00:15.490 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:15 smithi039 ceph-mon[31721]: from='mgr.? 172.21.15.194:0/3878194911' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-18T16:00:15.490 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:15 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/3261085776' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "771e0558-be62-4436-94b0-6364751911c4"}]: dispatch 2024-03-18T16:00:15.490 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:15 smithi039 ceph-mon[31721]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "771e0558-be62-4436-94b0-6364751911c4"}]: dispatch 2024-03-18T16:00:15.490 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:15 smithi039 ceph-mon[31721]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "771e0558-be62-4436-94b0-6364751911c4"}]': finished 2024-03-18T16:00:15.491 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:15 smithi039 ceph-mon[31721]: osdmap e5: 1 total, 0 up, 1 in 2024-03-18T16:00:15.491 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:15 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T16:00:15.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:15 smithi196 ceph-mon[36307]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T16:00:15.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:15 smithi196 ceph-mon[36307]: Standby manager daemon y started 2024-03-18T16:00:15.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:15 smithi196 ceph-mon[36307]: from='mgr.? 172.21.15.194:0/3878194911' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-18T16:00:15.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:15 smithi196 ceph-mon[36307]: from='mgr.? 172.21.15.194:0/3878194911' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-18T16:00:15.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:15 smithi196 ceph-mon[36307]: from='mgr.? 172.21.15.194:0/3878194911' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-18T16:00:15.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:15 smithi196 ceph-mon[36307]: from='mgr.? 172.21.15.194:0/3878194911' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-18T16:00:15.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:15 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/3261085776' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "771e0558-be62-4436-94b0-6364751911c4"}]: dispatch 2024-03-18T16:00:15.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:15 smithi196 ceph-mon[36307]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "771e0558-be62-4436-94b0-6364751911c4"}]: dispatch 2024-03-18T16:00:15.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:15 smithi196 ceph-mon[36307]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "771e0558-be62-4436-94b0-6364751911c4"}]': finished 2024-03-18T16:00:15.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:15 smithi196 ceph-mon[36307]: osdmap e5: 1 total, 0 up, 1 in 2024-03-18T16:00:15.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:15 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T16:00:15.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:15 smithi194 ceph-mon[36200]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T16:00:15.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:15 smithi194 ceph-mon[36200]: Standby manager daemon y started 2024-03-18T16:00:15.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:15 smithi194 ceph-mon[36200]: from='mgr.? 172.21.15.194:0/3878194911' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-18T16:00:15.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:15 smithi194 ceph-mon[36200]: from='mgr.? 172.21.15.194:0/3878194911' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-18T16:00:15.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:15 smithi194 ceph-mon[36200]: from='mgr.? 172.21.15.194:0/3878194911' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-18T16:00:15.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:15 smithi194 ceph-mon[36200]: from='mgr.? 172.21.15.194:0/3878194911' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-18T16:00:15.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:15 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/3261085776' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "771e0558-be62-4436-94b0-6364751911c4"}]: dispatch 2024-03-18T16:00:15.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:15 smithi194 ceph-mon[36200]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "771e0558-be62-4436-94b0-6364751911c4"}]: dispatch 2024-03-18T16:00:15.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:15 smithi194 ceph-mon[36200]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "771e0558-be62-4436-94b0-6364751911c4"}]': finished 2024-03-18T16:00:15.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:15 smithi194 ceph-mon[36200]: osdmap e5: 1 total, 0 up, 1 in 2024-03-18T16:00:15.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:15 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T16:00:16.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:16 smithi194 ceph-mon[36200]: mgrmap e15: x(active, since 89s), standbys: z, y 2024-03-18T16:00:16.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:16 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-18T16:00:16.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:16 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/3481484234' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T16:00:16.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:16 smithi039 ceph-mon[31721]: mgrmap e15: x(active, since 89s), standbys: z, y 2024-03-18T16:00:16.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:16 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-18T16:00:16.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:16 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/3481484234' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T16:00:16.727 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:16 smithi196 ceph-mon[36307]: mgrmap e15: x(active, since 89s), standbys: z, y 2024-03-18T16:00:16.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:16 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-18T16:00:16.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:16 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/3481484234' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T16:00:17.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:17 smithi039 ceph-mon[31721]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T16:00:17.727 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:17 smithi196 ceph-mon[36307]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T16:00:17.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:17 smithi194 ceph-mon[36200]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T16:00:19.542 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:19 smithi039 ceph-mon[31721]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T16:00:19.727 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:19 smithi196 ceph-mon[36307]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T16:00:19.739 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:19 smithi194 ceph-mon[36200]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T16:00:21.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:21 smithi039 ceph-mon[31721]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T16:00:21.727 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:21 smithi196 ceph-mon[36307]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T16:00:21.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:21 smithi194 ceph-mon[36200]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T16:00:23.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:23 smithi039 ceph-mon[31721]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T16:00:23.727 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:23 smithi196 ceph-mon[36307]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T16:00:23.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:23 smithi194 ceph-mon[36200]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T16:00:24.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:24 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-18T16:00:24.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:24 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:00:24.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:24 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-18T16:00:24.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:24 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:00:24.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:24 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-18T16:00:24.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:24 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:00:25.576 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:25 smithi039 ceph-mon[31721]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T16:00:25.576 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:25 smithi039 ceph-mon[31721]: Deploying daemon osd.0 on smithi039 2024-03-18T16:00:25.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:25 smithi196 ceph-mon[36307]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T16:00:25.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:25 smithi196 ceph-mon[36307]: Deploying daemon osd.0 on smithi039 2024-03-18T16:00:25.739 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:25 smithi194 ceph-mon[36200]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T16:00:25.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:25 smithi194 ceph-mon[36200]: Deploying daemon osd.0 on smithi039 2024-03-18T16:00:26.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:26 smithi039 ceph-mon[31721]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T16:00:26.727 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:26 smithi196 ceph-mon[36307]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T16:00:26.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:26 smithi194 ceph-mon[36200]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T16:00:28.555 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:28 smithi039 ceph-mon[31721]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T16:00:28.727 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:28 smithi196 ceph-mon[36307]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T16:00:28.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:28 smithi194 ceph-mon[36200]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T16:00:30.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:30 smithi039 ceph-mon[31721]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T16:00:30.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:30 smithi196 ceph-mon[36307]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T16:00:30.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:30 smithi194 ceph-mon[36200]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T16:00:33.258 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:33 smithi039 ceph-mon[31721]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T16:00:33.258 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:33 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:33.258 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:33 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:33.258 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:33 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:00:33.477 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:33 smithi196 ceph-mon[36307]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T16:00:33.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:33 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:33.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:33 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:33.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:33 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:00:33.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:33 smithi194 ceph-mon[36200]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T16:00:33.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:33 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:33.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:33 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:33.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:33 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:00:35.291 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:35 smithi194 ceph-mon[36200]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T16:00:35.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:35 smithi039 ceph-mon[31721]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T16:00:35.477 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:35 smithi196 ceph-mon[36307]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T16:00:36.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:36 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:36.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:36 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:36.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:36 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:00:36.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:36 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:00:36.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:36 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:36.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:36 smithi039 ceph-mon[31721]: from='osd.0 [v2:172.21.15.39:6802/4090743635,v1:172.21.15.39:6803/4090743635]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-18T16:00:36.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:36 smithi039 ceph-mon[31721]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-18T16:00:36.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:36 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:36.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:36 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:36.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:36 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:00:36.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:36 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:00:36.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:36 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:36.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:36 smithi196 ceph-mon[36307]: from='osd.0 [v2:172.21.15.39:6802/4090743635,v1:172.21.15.39:6803/4090743635]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-18T16:00:36.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:36 smithi196 ceph-mon[36307]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-18T16:00:36.485 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:36 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:36.486 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:36 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:36.486 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:36 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:00:36.486 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:36 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:00:36.486 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:36 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:36.486 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:36 smithi194 ceph-mon[36200]: from='osd.0 [v2:172.21.15.39:6802/4090743635,v1:172.21.15.39:6803/4090743635]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-18T16:00:36.486 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:36 smithi194 ceph-mon[36200]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-18T16:00:36.997 INFO:teuthology.orchestra.run.smithi039.stdout:Created osd(s) 0 on host 'smithi039' 2024-03-18T16:00:36.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:00:36.994+0000 7f375f400700 1 -- 172.21.15.39:0/344702481 <== mgr.14152 v2:172.21.15.39:6800/2749214286 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f3768062710 con 0x7f374c039200 2024-03-18T16:00:37.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:00:36.998+0000 7f376dc00700 1 -- 172.21.15.39:0/344702481 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f374c039200 msgr2=0x7f374c03b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:00:37.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:00:36.998+0000 7f376dc00700 1 --2- 172.21.15.39:0/344702481 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f374c039200 0x7f374c03b6c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f3754009c60 tx=0x7f3754009260 comp rx=0 tx=0).stop 2024-03-18T16:00:37.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:00:36.998+0000 7f376dc00700 1 -- 172.21.15.39:0/344702481 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3768102d50 msgr2=0x7f37681026b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:00:37.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:00:36.998+0000 7f376dc00700 1 --2- 172.21.15.39:0/344702481 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3768102d50 0x7f37681026b0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f375800e9a0 tx=0x7f375800ecb0 comp rx=0 tx=0).stop 2024-03-18T16:00:37.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:00:36.999+0000 7f376dc00700 1 -- 172.21.15.39:0/344702481 shutdown_connections 2024-03-18T16:00:37.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:00:36.999+0000 7f376dc00700 1 --2- 172.21.15.39:0/344702481 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f374c039200 0x7f374c03b6c0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:37.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:00:36.999+0000 7f376dc00700 1 --2- 172.21.15.39:0/344702481 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3768107fb0 0x7f3768101240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:37.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:00:36.999+0000 7f376dc00700 1 --2- 172.21.15.39:0/344702481 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3768102d50 0x7f37681026b0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:37.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:00:36.999+0000 7f376dc00700 1 --2- 172.21.15.39:0/344702481 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3768105680 0x7f3768100d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:37.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:00:36.999+0000 7f376dc00700 1 -- 172.21.15.39:0/344702481 >> 172.21.15.39:0/344702481 conn(0x7f37680fc760 msgr2=0x7f3768103690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:00:37.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:00:36.999+0000 7f376dc00700 1 -- 172.21.15.39:0/344702481 shutdown_connections 2024-03-18T16:00:37.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:00:36.999+0000 7f376dc00700 1 -- 172.21.15.39:0/344702481 wait complete. 2024-03-18T16:00:37.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:37 smithi196 ceph-mon[36307]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T16:00:37.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:37 smithi196 ceph-mon[36307]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-18T16:00:37.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:37 smithi196 ceph-mon[36307]: from='osd.0 [v2:172.21.15.39:6802/4090743635,v1:172.21.15.39:6803/4090743635]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]: dispatch 2024-03-18T16:00:37.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:37 smithi196 ceph-mon[36307]: osdmap e6: 1 total, 0 up, 1 in 2024-03-18T16:00:37.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:37 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T16:00:37.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:37 smithi196 ceph-mon[36307]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]: dispatch 2024-03-18T16:00:37.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:37 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:00:37.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:37 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:37.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:37 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:37.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:37 smithi194 ceph-mon[36200]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T16:00:37.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:37 smithi194 ceph-mon[36200]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-18T16:00:37.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:37 smithi194 ceph-mon[36200]: from='osd.0 [v2:172.21.15.39:6802/4090743635,v1:172.21.15.39:6803/4090743635]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]: dispatch 2024-03-18T16:00:37.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:37 smithi194 ceph-mon[36200]: osdmap e6: 1 total, 0 up, 1 in 2024-03-18T16:00:37.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:37 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T16:00:37.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:37 smithi194 ceph-mon[36200]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]: dispatch 2024-03-18T16:00:37.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:37 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:00:37.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:37 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:37.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:37 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:37.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:37 smithi039 ceph-mon[31721]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T16:00:37.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:37 smithi039 ceph-mon[31721]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-18T16:00:37.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:37 smithi039 ceph-mon[31721]: from='osd.0 [v2:172.21.15.39:6802/4090743635,v1:172.21.15.39:6803/4090743635]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]: dispatch 2024-03-18T16:00:37.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:37 smithi039 ceph-mon[31721]: osdmap e6: 1 total, 0 up, 1 in 2024-03-18T16:00:37.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:37 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T16:00:37.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:37 smithi039 ceph-mon[31721]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]: dispatch 2024-03-18T16:00:37.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:37 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:00:37.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:37 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:37.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:37 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:37.804 DEBUG:teuthology.orchestra.run.smithi039:osd.0> sudo journalctl -f -n 0 -u ceph-1892d8fc-e540-11ee-95c9-87774f69a715@osd.0.service 2024-03-18T16:00:37.810 INFO:tasks.cephadm:Deploying osd.1 on smithi194 with /dev/vg_nvme/lv_4... 2024-03-18T16:00:37.811 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-18T16:00:38.056 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T16:00:38.303 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:38 smithi039 ceph-mon[31721]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]': finished 2024-03-18T16:00:38.303 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:38 smithi039 ceph-mon[31721]: osdmap e7: 1 total, 0 up, 1 in 2024-03-18T16:00:38.303 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:38 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T16:00:38.303 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:38 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T16:00:38.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:38 smithi196 ceph-mon[36307]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]': finished 2024-03-18T16:00:38.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:38 smithi196 ceph-mon[36307]: osdmap e7: 1 total, 0 up, 1 in 2024-03-18T16:00:38.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:38 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T16:00:38.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:38 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T16:00:38.485 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:38 smithi194 ceph-mon[36200]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]': finished 2024-03-18T16:00:38.485 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:38 smithi194 ceph-mon[36200]: osdmap e7: 1 total, 0 up, 1 in 2024-03-18T16:00:38.485 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:38 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T16:00:38.485 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:38 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T16:00:39.397 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:39 smithi194 ceph-mon[36200]: purged_snaps scrub starts 2024-03-18T16:00:39.397 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:39 smithi194 ceph-mon[36200]: purged_snaps scrub ok 2024-03-18T16:00:39.397 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:39 smithi194 ceph-mon[36200]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T16:00:39.397 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:39 smithi194 ceph-mon[36200]: osd.0 [v2:172.21.15.39:6802/4090743635,v1:172.21.15.39:6803/4090743635] boot 2024-03-18T16:00:39.397 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:39 smithi194 ceph-mon[36200]: osdmap e8: 1 total, 1 up, 1 in 2024-03-18T16:00:39.397 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:39 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T16:00:39.456 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:39 smithi039 ceph-mon[31721]: purged_snaps scrub starts 2024-03-18T16:00:39.456 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:39 smithi039 ceph-mon[31721]: purged_snaps scrub ok 2024-03-18T16:00:39.456 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:39 smithi039 ceph-mon[31721]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T16:00:39.456 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:39 smithi039 ceph-mon[31721]: osd.0 [v2:172.21.15.39:6802/4090743635,v1:172.21.15.39:6803/4090743635] boot 2024-03-18T16:00:39.456 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:39 smithi039 ceph-mon[31721]: osdmap e8: 1 total, 1 up, 1 in 2024-03-18T16:00:39.456 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:39 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T16:00:39.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:39 smithi196 ceph-mon[36307]: purged_snaps scrub starts 2024-03-18T16:00:39.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:39 smithi196 ceph-mon[36307]: purged_snaps scrub ok 2024-03-18T16:00:39.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:39 smithi196 ceph-mon[36307]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-18T16:00:39.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:39 smithi196 ceph-mon[36307]: osd.0 [v2:172.21.15.39:6802/4090743635,v1:172.21.15.39:6803/4090743635] boot 2024-03-18T16:00:39.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:39 smithi196 ceph-mon[36307]: osdmap e8: 1 total, 1 up, 1 in 2024-03-18T16:00:39.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:39 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-18T16:00:39.926 INFO:teuthology.orchestra.run.smithi194.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-18T16:00:39.926 INFO:teuthology.orchestra.run.smithi194.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-18T16:00:39.926 INFO:teuthology.orchestra.run.smithi194.stderr: stderr: 10+0 records in 2024-03-18T16:00:39.926 INFO:teuthology.orchestra.run.smithi194.stderr:10+0 records out 2024-03-18T16:00:39.926 INFO:teuthology.orchestra.run.smithi194.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112129 s, 935 MB/s 2024-03-18T16:00:39.926 INFO:teuthology.orchestra.run.smithi194.stderr:--> Zapping successful for: 2024-03-18T16:00:40.469 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:40 smithi194 ceph-mon[36200]: osdmap e9: 1 total, 1 up, 1 in 2024-03-18T16:00:40.477 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:40 smithi196 ceph-mon[36307]: osdmap e9: 1 total, 1 up, 1 in 2024-03-18T16:00:40.495 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi194:vg_nvme/lv_4 2024-03-18T16:00:40.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:40 smithi039 ceph-mon[31721]: osdmap e9: 1 total, 1 up, 1 in 2024-03-18T16:00:40.728 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T16:00:41.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:41 smithi196 ceph-mon[36307]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T16:00:41.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:41 smithi194 ceph-mon[36200]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T16:00:41.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:41 smithi039 ceph-mon[31721]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T16:00:42.261 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:00:42.257+0000 7f2e99c96700 1 -- 172.21.15.194:0/993540949 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2e94103500 msgr2=0x7f2e94103960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:00:42.261 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:00:42.257+0000 7f2e99c96700 1 --2- 172.21.15.194:0/993540949 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2e94103500 0x7f2e94103960 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2e880099c0 tx=0x7f2e88009cd0 comp rx=0 tx=0).stop 2024-03-18T16:00:42.261 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:00:42.257+0000 7f2e99c96700 1 -- 172.21.15.194:0/993540949 shutdown_connections 2024-03-18T16:00:42.261 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:00:42.257+0000 7f2e99c96700 1 --2- 172.21.15.194:0/993540949 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2e94103ea0 0x7f2e9410a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:42.261 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:00:42.257+0000 7f2e99c96700 1 --2- 172.21.15.194:0/993540949 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2e94102300 0x7f2e94102720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:42.261 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:00:42.257+0000 7f2e99c96700 1 --2- 172.21.15.194:0/993540949 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2e94103500 0x7f2e94103960 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:42.261 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:00:42.257+0000 7f2e99c96700 1 -- 172.21.15.194:0/993540949 >> 172.21.15.194:0/993540949 conn(0x7f2e940fd8e0 msgr2=0x7f2e940ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:00:42.261 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:00:42.257+0000 7f2e99c96700 1 -- 172.21.15.194:0/993540949 shutdown_connections 2024-03-18T16:00:42.261 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:00:42.257+0000 7f2e99c96700 1 -- 172.21.15.194:0/993540949 wait complete. 2024-03-18T16:00:42.262 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:00:42.257+0000 7f2e99c96700 1 Processor -- start 2024-03-18T16:00:42.262 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:00:42.258+0000 7f2e99c96700 1 -- start start 2024-03-18T16:00:42.262 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:00:42.258+0000 7f2e99c96700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2e94102300 0x7f2e94198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:42.262 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:00:42.258+0000 7f2e99c96700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2e94103500 0x7f2e94198ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:42.262 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:00:42.258+0000 7f2e99c96700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2e94103ea0 0x7f2e9419fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:42.262 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:00:42.258+0000 7f2e99c96700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f2e9410df60 con 0x7f2e94103500 2024-03-18T16:00:42.262 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:00:42.258+0000 7f2e99c96700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f2e9410dde0 con 0x7f2e94102300 2024-03-18T16:00:42.262 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:00:42.258+0000 7f2e99c96700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f2e9410e0e0 con 0x7f2e94103ea0 2024-03-18T16:00:42.262 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:00:42.258+0000 7f2e937fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2e94102300 0x7f2e94198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:42.262 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:00:42.258+0000 7f2e937fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2e94102300 0x7f2e94198570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:58328/0 (socket says 172.21.15.194:58328) 2024-03-18T16:00:42.262 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:00:42.258+0000 7f2e937fe700 1 -- 172.21.15.194:0/798635065 learned_addr learned my addr 172.21.15.194:0/798635065 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-03-18T16:00:42.262 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:00:42.258+0000 7f2e93fff700 1 --2- 172.21.15.194:0/798635065 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2e94103ea0 0x7f2e9419fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:42.262 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:00:42.258+0000 7f2e937fe700 1 -- 172.21.15.194:0/798635065 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2e94103ea0 msgr2=0x7f2e9419fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:00:42.263 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:00:42.258+0000 7f2e937fe700 1 --2- 172.21.15.194:0/798635065 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2e94103ea0 0x7f2e9419fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:42.263 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:00:42.258+0000 7f2e937fe700 1 -- 172.21.15.194:0/798635065 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2e94103500 msgr2=0x7f2e94198ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:00:42.263 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:00:42.258+0000 7f2e937fe700 1 --2- 172.21.15.194:0/798635065 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2e94103500 0x7f2e94198ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:00:42.263 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:00:42.258+0000 7f2e937fe700 1 -- 172.21.15.194:0/798635065 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e941a01d0 con 0x7f2e94102300 2024-03-18T16:00:42.263 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:00:42.258+0000 7f2e937fe700 1 --2- 172.21.15.194:0/798635065 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2e94102300 0x7f2e94198570 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2e8400d980 tx=0x7f2e8400dd40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:00:42.263 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:00:42.259+0000 7f2e90ff9700 1 -- 172.21.15.194:0/798635065 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e84017070 con 0x7f2e94102300 2024-03-18T16:00:42.263 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:00:42.259+0000 7f2e90ff9700 1 -- 172.21.15.194:0/798635065 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e84009b40 con 0x7f2e94102300 2024-03-18T16:00:42.263 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:00:42.259+0000 7f2e90ff9700 1 -- 172.21.15.194:0/798635065 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e8400f620 con 0x7f2e94102300 2024-03-18T16:00:42.263 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:00:42.259+0000 7f2e99c96700 1 -- 172.21.15.194:0/798635065 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e94074560 con 0x7f2e94102300 2024-03-18T16:00:42.263 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:00:42.259+0000 7f2e99c96700 1 -- 172.21.15.194:0/798635065 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e94074b20 con 0x7f2e94102300 2024-03-18T16:00:42.264 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:00:42.260+0000 7f2e99c96700 1 -- 172.21.15.194:0/798635065 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e94068f90 con 0x7f2e94102300 2024-03-18T16:00:42.264 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:00:42.261+0000 7f2e90ff9700 1 -- 172.21.15.194:0/798635065 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 15) ==== 137234+0+0 (secure 0 0 0) 0x7f2e84010460 con 0x7f2e94102300 2024-03-18T16:00:42.264 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:00:42.261+0000 7f2e90ff9700 1 --2- 172.21.15.194:0/798635065 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f2e7c0a2880 0x7f2e7c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:00:42.265 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:00:42.261+0000 7f2e90ff9700 1 -- 172.21.15.194:0/798635065 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7f2e840ccbd0 con 0x7f2e94102300 2024-03-18T16:00:42.265 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:00:42.262+0000 7f2e92ffd700 1 --2- 172.21.15.194:0/798635065 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f2e7c0a2880 0x7f2e7c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:00:42.265 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:00:42.263+0000 7f2e92ffd700 1 --2- 172.21.15.194:0/798635065 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f2e7c0a2880 0x7f2e7c0a4d40 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2e88003890 tx=0x7f2e88000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:00:42.267 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:00:42.266+0000 7f2e90ff9700 1 -- 172.21.15.194:0/798635065 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2e8409b6e0 con 0x7f2e94102300 2024-03-18T16:00:42.471 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:00:42.469+0000 7f2e99c96700 1 -- 172.21.15.194:0/798635065 --> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f2e94075f70 con 0x7f2e7c0a2880 2024-03-18T16:00:43.456 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:43 smithi194 ceph-mon[36200]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T16:00:43.456 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:43 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:43.456 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:43 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:43.457 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:43 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T16:00:43.457 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:43 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:00:43.457 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:43 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T16:00:43.457 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:43 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:00:43.457 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:43 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:43.457 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:43 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:00:43.457 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:43 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:00:43.457 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:43 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:43.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:43 smithi196 ceph-mon[36307]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T16:00:43.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:43 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:43.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:43 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:43.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:43 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T16:00:43.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:43 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:00:43.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:43 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T16:00:43.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:43 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:00:43.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:43 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:43.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:43 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:00:43.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:43 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:00:43.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:43 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:43.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:43 smithi039 ceph-mon[31721]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T16:00:43.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:43 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:43.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:43 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:43.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:43 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T16:00:43.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:43 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:00:43.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:43 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T16:00:43.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:43 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:00:43.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:43 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:43.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:43 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:00:43.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:43 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:00:43.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:43 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:44.353 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:44 smithi194 ceph-mon[36200]: Detected new or changed devices on smithi039 2024-03-18T16:00:44.353 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:44 smithi194 ceph-mon[36200]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:00:44.353 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:44 smithi194 ceph-mon[36200]: Adjusting osd_memory_target on smithi039 to 17091M 2024-03-18T16:00:44.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:44 smithi039 ceph-mon[31721]: Detected new or changed devices on smithi039 2024-03-18T16:00:44.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:44 smithi039 ceph-mon[31721]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:00:44.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:44 smithi039 ceph-mon[31721]: Adjusting osd_memory_target on smithi039 to 17091M 2024-03-18T16:00:44.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:44 smithi196 ceph-mon[36307]: Detected new or changed devices on smithi039 2024-03-18T16:00:44.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:44 smithi196 ceph-mon[36307]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:00:44.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:44 smithi196 ceph-mon[36307]: Adjusting osd_memory_target on smithi039 to 17091M 2024-03-18T16:00:45.489 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:45 smithi039 ceph-mon[31721]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T16:00:45.490 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:45 smithi039 ceph-mon[31721]: from='client.? 172.21.15.194:0/3465116621' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0430eee1-cf34-4277-b255-c7d9f07ad19a"}]: dispatch 2024-03-18T16:00:45.490 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:45 smithi039 ceph-mon[31721]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0430eee1-cf34-4277-b255-c7d9f07ad19a"}]: dispatch 2024-03-18T16:00:45.490 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:45 smithi039 ceph-mon[31721]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0430eee1-cf34-4277-b255-c7d9f07ad19a"}]': finished 2024-03-18T16:00:45.490 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:45 smithi039 ceph-mon[31721]: osdmap e10: 2 total, 1 up, 2 in 2024-03-18T16:00:45.490 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:45 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T16:00:45.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:45 smithi194 ceph-mon[36200]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T16:00:45.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:45 smithi194 ceph-mon[36200]: from='client.? 172.21.15.194:0/3465116621' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0430eee1-cf34-4277-b255-c7d9f07ad19a"}]: dispatch 2024-03-18T16:00:45.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:45 smithi194 ceph-mon[36200]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0430eee1-cf34-4277-b255-c7d9f07ad19a"}]: dispatch 2024-03-18T16:00:45.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:45 smithi194 ceph-mon[36200]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0430eee1-cf34-4277-b255-c7d9f07ad19a"}]': finished 2024-03-18T16:00:45.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:45 smithi194 ceph-mon[36200]: osdmap e10: 2 total, 1 up, 2 in 2024-03-18T16:00:45.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:45 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T16:00:45.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:45 smithi196 ceph-mon[36307]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T16:00:45.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:45 smithi196 ceph-mon[36307]: from='client.? 172.21.15.194:0/3465116621' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0430eee1-cf34-4277-b255-c7d9f07ad19a"}]: dispatch 2024-03-18T16:00:45.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:45 smithi196 ceph-mon[36307]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0430eee1-cf34-4277-b255-c7d9f07ad19a"}]: dispatch 2024-03-18T16:00:45.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:45 smithi196 ceph-mon[36307]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0430eee1-cf34-4277-b255-c7d9f07ad19a"}]': finished 2024-03-18T16:00:45.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:45 smithi196 ceph-mon[36307]: osdmap e10: 2 total, 1 up, 2 in 2024-03-18T16:00:45.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:45 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T16:00:46.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:46 smithi039 ceph-mon[31721]: from='client.? 172.21.15.194:0/4148819650' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T16:00:46.727 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:46 smithi196 ceph-mon[36307]: from='client.? 172.21.15.194:0/4148819650' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T16:00:46.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:46 smithi194 ceph-mon[36200]: from='client.? 172.21.15.194:0/4148819650' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T16:00:47.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:47 smithi039 ceph-mon[31721]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T16:00:47.727 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:47 smithi196 ceph-mon[36307]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T16:00:47.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:47 smithi194 ceph-mon[36200]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T16:00:49.467 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:49 smithi194 ceph-mon[36200]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T16:00:49.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:49 smithi039 ceph-mon[31721]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T16:00:49.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:49 smithi196 ceph-mon[36307]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T16:00:51.388 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:51 smithi194 ceph-mon[36200]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T16:00:51.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:51 smithi039 ceph-mon[31721]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T16:00:51.727 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:51 smithi196 ceph-mon[36307]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T16:00:53.324 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:53 smithi194 ceph-mon[36200]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T16:00:53.324 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:53 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-18T16:00:53.324 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:53 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:00:53.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:53 smithi039 ceph-mon[31721]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T16:00:53.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:53 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-18T16:00:53.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:53 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:00:53.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:53 smithi196 ceph-mon[36307]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T16:00:53.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:53 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-18T16:00:53.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:53 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:00:54.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:54 smithi039 ceph-mon[31721]: Deploying daemon osd.1 on smithi194 2024-03-18T16:00:54.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:54 smithi194 ceph-mon[36200]: Deploying daemon osd.1 on smithi194 2024-03-18T16:00:54.727 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:54 smithi196 ceph-mon[36307]: Deploying daemon osd.1 on smithi194 2024-03-18T16:00:55.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:55 smithi194 ceph-mon[36200]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T16:00:55.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:55 smithi039 ceph-mon[31721]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T16:00:55.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:55 smithi196 ceph-mon[36307]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T16:00:56.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:56 smithi194 ceph-mon[36200]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T16:00:56.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:56 smithi196 ceph-mon[36307]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T16:00:56.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:56 smithi039 ceph-mon[31721]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T16:00:59.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:58 smithi039 ceph-mon[31721]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T16:00:59.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:58 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:59.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:58 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:59.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:00:58 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:00:59.127 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:58 smithi194 ceph-mon[36200]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T16:00:59.127 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:58 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:59.127 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:58 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:59.127 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:00:58 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:00:59.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:58 smithi196 ceph-mon[36307]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T16:00:59.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:58 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:59.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:58 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:00:59.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:00:58 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:01:01.410 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:01 smithi194 ceph-mon[36200]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T16:01:01.410 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:01 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:01.410 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:01 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:01.410 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:01 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:01:01.410 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:01 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:01:01.410 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:01 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:01.463 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:01:01.461+0000 7f2e90ff9700 1 -- 172.21.15.194:0/798635065 <== mgr.14152 v2:172.21.15.39:6800/2749214286 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f2e94075f70 con 0x7f2e7c0a2880 2024-03-18T16:01:01.464 INFO:teuthology.orchestra.run.smithi194.stdout:Created osd(s) 1 on host 'smithi194' 2024-03-18T16:01:01.466 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:01:01.464+0000 7f2e99c96700 1 -- 172.21.15.194:0/798635065 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f2e7c0a2880 msgr2=0x7f2e7c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:01:01.466 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:01:01.464+0000 7f2e99c96700 1 --2- 172.21.15.194:0/798635065 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f2e7c0a2880 0x7f2e7c0a4d40 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2e88003890 tx=0x7f2e88000bc0 comp rx=0 tx=0).stop 2024-03-18T16:01:01.466 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:01:01.464+0000 7f2e99c96700 1 -- 172.21.15.194:0/798635065 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2e94102300 msgr2=0x7f2e94198570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:01:01.466 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:01:01.464+0000 7f2e99c96700 1 --2- 172.21.15.194:0/798635065 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2e94102300 0x7f2e94198570 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2e8400d980 tx=0x7f2e8400dd40 comp rx=0 tx=0).stop 2024-03-18T16:01:01.466 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:01:01.464+0000 7f2e99c96700 1 -- 172.21.15.194:0/798635065 shutdown_connections 2024-03-18T16:01:01.466 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:01:01.464+0000 7f2e99c96700 1 --2- 172.21.15.194:0/798635065 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f2e7c0a2880 0x7f2e7c0a4d40 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:01.467 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:01:01.464+0000 7f2e99c96700 1 --2- 172.21.15.194:0/798635065 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2e94103ea0 0x7f2e9419fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:01.467 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:01:01.464+0000 7f2e99c96700 1 --2- 172.21.15.194:0/798635065 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2e94102300 0x7f2e94198570 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:01.467 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:01:01.464+0000 7f2e99c96700 1 --2- 172.21.15.194:0/798635065 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2e94103500 0x7f2e94198ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:01.467 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:01:01.464+0000 7f2e99c96700 1 -- 172.21.15.194:0/798635065 >> 172.21.15.194:0/798635065 conn(0x7f2e940fd8e0 msgr2=0x7f2e94106880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:01:01.467 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:01:01.464+0000 7f2e99c96700 1 -- 172.21.15.194:0/798635065 shutdown_connections 2024-03-18T16:01:01.467 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:01:01.464+0000 7f2e99c96700 1 -- 172.21.15.194:0/798635065 wait complete. 2024-03-18T16:01:01.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:01 smithi039 ceph-mon[31721]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T16:01:01.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:01 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:01.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:01 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:01.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:01 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:01:01.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:01 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:01:01.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:01 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:01.689 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:01 smithi196 ceph-mon[36307]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T16:01:01.689 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:01 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:01.690 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:01 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:01.690 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:01 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:01:01.690 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:01 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:01:01.690 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:01 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:02.104 DEBUG:teuthology.orchestra.run.smithi194:osd.1> sudo journalctl -f -n 0 -u ceph-1892d8fc-e540-11ee-95c9-87774f69a715@osd.1.service 2024-03-18T16:01:02.107 INFO:tasks.cephadm:Deploying osd.2 on smithi196 with /dev/vg_nvme/lv_4... 2024-03-18T16:01:02.107 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-18T16:01:02.350 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T16:01:02.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:02 smithi196 ceph-mon[36307]: pgmap v68: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T16:01:02.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:02 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:02.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:02 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:02.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:02 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:01:02.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:02 smithi196 ceph-mon[36307]: from='osd.1 [v2:172.21.15.194:6800/2800329805,v1:172.21.15.194:6801/2800329805]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-18T16:01:02.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:02 smithi196 ceph-mon[36307]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-18T16:01:02.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:02 smithi194 ceph-mon[36200]: pgmap v68: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T16:01:02.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:02 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:02.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:02 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:02.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:02 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:01:02.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:02 smithi194 ceph-mon[36200]: from='osd.1 [v2:172.21.15.194:6800/2800329805,v1:172.21.15.194:6801/2800329805]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-18T16:01:02.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:02 smithi194 ceph-mon[36200]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-18T16:01:02.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:02 smithi039 ceph-mon[31721]: pgmap v68: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T16:01:02.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:02 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:02.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:02 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:02.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:02 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:01:02.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:02 smithi039 ceph-mon[31721]: from='osd.1 [v2:172.21.15.194:6800/2800329805,v1:172.21.15.194:6801/2800329805]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-18T16:01:02.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:02 smithi039 ceph-mon[31721]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-18T16:01:03.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:03 smithi196 ceph-mon[36307]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-18T16:01:03.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:03 smithi196 ceph-mon[36307]: from='osd.1 [v2:172.21.15.194:6800/2800329805,v1:172.21.15.194:6801/2800329805]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-03-18T16:01:03.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:03 smithi196 ceph-mon[36307]: osdmap e11: 2 total, 1 up, 2 in 2024-03-18T16:01:03.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:03 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T16:01:03.685 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:03 smithi196 ceph-mon[36307]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-03-18T16:01:03.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:03 smithi194 ceph-mon[36200]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-18T16:01:03.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:03 smithi194 ceph-mon[36200]: from='osd.1 [v2:172.21.15.194:6800/2800329805,v1:172.21.15.194:6801/2800329805]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-03-18T16:01:03.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:03 smithi194 ceph-mon[36200]: osdmap e11: 2 total, 1 up, 2 in 2024-03-18T16:01:03.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:03 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T16:01:03.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:03 smithi194 ceph-mon[36200]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-03-18T16:01:03.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:03 smithi039 ceph-mon[31721]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-18T16:01:03.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:03 smithi039 ceph-mon[31721]: from='osd.1 [v2:172.21.15.194:6800/2800329805,v1:172.21.15.194:6801/2800329805]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-03-18T16:01:03.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:03 smithi039 ceph-mon[31721]: osdmap e11: 2 total, 1 up, 2 in 2024-03-18T16:01:03.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:03 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T16:01:03.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:03 smithi039 ceph-mon[31721]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-03-18T16:01:04.239 INFO:teuthology.orchestra.run.smithi196.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-18T16:01:04.239 INFO:teuthology.orchestra.run.smithi196.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-18T16:01:04.240 INFO:teuthology.orchestra.run.smithi196.stderr: stderr: 10+0 records in 2024-03-18T16:01:04.240 INFO:teuthology.orchestra.run.smithi196.stderr:10+0 records out 2024-03-18T16:01:04.240 INFO:teuthology.orchestra.run.smithi196.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111165 s, 943 MB/s 2024-03-18T16:01:04.240 INFO:teuthology.orchestra.run.smithi196.stderr:--> Zapping successful for: 2024-03-18T16:01:04.480 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:04 smithi194 ceph-mon[36200]: pgmap v70: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T16:01:04.481 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:04 smithi194 ceph-mon[36200]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-03-18T16:01:04.481 INFO:journalctl@ceph.osd.1.smithi194.stdout:Mar 18 16:01:04 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-1[40784]: 2024-03-18T16:01:04.200+0000 7f3cd44e8700 -1 osd.1 0 waiting for initial osdmap 2024-03-18T16:01:04.481 INFO:journalctl@ceph.osd.1.smithi194.stdout:Mar 18 16:01:04 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-1[40784]: 2024-03-18T16:01:04.204+0000 7f3ccc2d3700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T16:01:04.539 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:04 smithi196 ceph-mon[36307]: pgmap v70: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T16:01:04.540 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:04 smithi196 ceph-mon[36307]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-03-18T16:01:04.540 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:04 smithi196 ceph-mon[36307]: osdmap e12: 2 total, 1 up, 2 in 2024-03-18T16:01:04.540 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:04 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T16:01:04.540 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:04 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T16:01:04.540 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:04 smithi196 ceph-mon[36307]: from='osd.1 ' entity='osd.1' 2024-03-18T16:01:04.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:04 smithi194 ceph-mon[36200]: osdmap e12: 2 total, 1 up, 2 in 2024-03-18T16:01:04.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:04 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T16:01:04.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:04 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T16:01:04.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:04 smithi194 ceph-mon[36200]: from='osd.1 ' entity='osd.1' 2024-03-18T16:01:04.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:04 smithi039 ceph-mon[31721]: pgmap v70: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T16:01:04.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:04 smithi039 ceph-mon[31721]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-03-18T16:01:04.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:04 smithi039 ceph-mon[31721]: osdmap e12: 2 total, 1 up, 2 in 2024-03-18T16:01:04.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:04 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T16:01:04.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:04 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T16:01:04.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:04 smithi039 ceph-mon[31721]: from='osd.1 ' entity='osd.1' 2024-03-18T16:01:04.902 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi196:vg_nvme/lv_4 2024-03-18T16:01:05.151 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T16:01:05.518 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:05 smithi196 ceph-mon[36307]: purged_snaps scrub starts 2024-03-18T16:01:05.518 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:05 smithi196 ceph-mon[36307]: purged_snaps scrub ok 2024-03-18T16:01:05.518 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:05 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T16:01:05.518 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:05 smithi196 ceph-mon[36307]: Detected new or changed devices on smithi194 2024-03-18T16:01:05.518 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:05 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:05.518 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:05 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:05.518 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:05 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:01:05.519 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:05 smithi196 ceph-mon[36307]: Adjusting osd_memory_target on smithi194 to 17080M 2024-03-18T16:01:05.519 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:05 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:05.519 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:05 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:01:05.519 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:05 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:01:05.519 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:05 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:05.519 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:05 smithi196 ceph-mon[36307]: osd.1 [v2:172.21.15.194:6800/2800329805,v1:172.21.15.194:6801/2800329805] boot 2024-03-18T16:01:05.519 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:05 smithi196 ceph-mon[36307]: osdmap e13: 2 total, 2 up, 2 in 2024-03-18T16:01:05.519 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:05 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T16:01:05.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:05 smithi194 ceph-mon[36200]: purged_snaps scrub starts 2024-03-18T16:01:05.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:05 smithi194 ceph-mon[36200]: purged_snaps scrub ok 2024-03-18T16:01:05.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:05 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T16:01:05.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:05 smithi194 ceph-mon[36200]: Detected new or changed devices on smithi194 2024-03-18T16:01:05.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:05 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:05.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:05 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:05.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:05 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:01:05.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:05 smithi194 ceph-mon[36200]: Adjusting osd_memory_target on smithi194 to 17080M 2024-03-18T16:01:05.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:05 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:05.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:05 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:01:05.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:05 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:01:05.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:05 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:05.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:05 smithi194 ceph-mon[36200]: osd.1 [v2:172.21.15.194:6800/2800329805,v1:172.21.15.194:6801/2800329805] boot 2024-03-18T16:01:05.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:05 smithi194 ceph-mon[36200]: osdmap e13: 2 total, 2 up, 2 in 2024-03-18T16:01:05.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:05 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T16:01:05.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:05 smithi039 ceph-mon[31721]: purged_snaps scrub starts 2024-03-18T16:01:05.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:05 smithi039 ceph-mon[31721]: purged_snaps scrub ok 2024-03-18T16:01:05.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:05 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T16:01:05.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:05 smithi039 ceph-mon[31721]: Detected new or changed devices on smithi194 2024-03-18T16:01:05.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:05 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:05.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:05 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:05.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:05 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:01:05.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:05 smithi039 ceph-mon[31721]: Adjusting osd_memory_target on smithi194 to 17080M 2024-03-18T16:01:05.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:05 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:05.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:05 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:01:05.838 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:05 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:01:05.838 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:05 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:05.838 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:05 smithi039 ceph-mon[31721]: osd.1 [v2:172.21.15.194:6800/2800329805,v1:172.21.15.194:6801/2800329805] boot 2024-03-18T16:01:05.838 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:05 smithi039 ceph-mon[31721]: osdmap e13: 2 total, 2 up, 2 in 2024-03-18T16:01:05.838 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:05 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-18T16:01:06.687 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:01:06.684+0000 7f8faa4e9700 1 -- 172.21.15.196:0/1972470044 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8fa4101240 msgr2=0x7f8fa4101660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:01:06.687 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:01:06.684+0000 7f8faa4e9700 1 --2- 172.21.15.196:0/1972470044 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8fa4101240 0x7f8fa4101660 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f8f94009990 tx=0x7f8f94009ca0 comp rx=0 tx=0).stop 2024-03-18T16:01:06.687 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:01:06.684+0000 7f8faa4e9700 1 -- 172.21.15.196:0/1972470044 shutdown_connections 2024-03-18T16:01:06.688 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:01:06.684+0000 7f8faa4e9700 1 --2- 172.21.15.196:0/1972470044 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8fa4102de0 0x7f8fa4109690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:06.688 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:01:06.684+0000 7f8faa4e9700 1 --2- 172.21.15.196:0/1972470044 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8fa4101240 0x7f8fa4101660 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:06.688 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:01:06.684+0000 7f8faa4e9700 1 --2- 172.21.15.196:0/1972470044 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8fa4102440 0x7f8fa41028a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:06.688 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:01:06.684+0000 7f8faa4e9700 1 -- 172.21.15.196:0/1972470044 >> 172.21.15.196:0/1972470044 conn(0x7f8fa40fc7c0 msgr2=0x7f8fa40fec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:01:06.688 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:01:06.684+0000 7f8faa4e9700 1 -- 172.21.15.196:0/1972470044 shutdown_connections 2024-03-18T16:01:06.688 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:01:06.684+0000 7f8faa4e9700 1 -- 172.21.15.196:0/1972470044 wait complete. 2024-03-18T16:01:06.688 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:01:06.684+0000 7f8faa4e9700 1 Processor -- start 2024-03-18T16:01:06.688 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:01:06.684+0000 7f8faa4e9700 1 -- start start 2024-03-18T16:01:06.688 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:01:06.685+0000 7f8faa4e9700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8fa4101240 0x7f8fa406b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:01:06.688 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:01:06.685+0000 7f8faa4e9700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8fa4102440 0x7f8fa419f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:01:06.688 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:01:06.685+0000 7f8faa4e9700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8fa4102de0 0x7f8fa41a1c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:01:06.688 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:01:06.685+0000 7f8faa4e9700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f8fa410cea0 con 0x7f8fa4102de0 2024-03-18T16:01:06.688 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:01:06.685+0000 7f8faa4e9700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f8fa410cd20 con 0x7f8fa4101240 2024-03-18T16:01:06.688 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:01:06.685+0000 7f8faa4e9700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f8fa410d020 con 0x7f8fa4102440 2024-03-18T16:01:06.688 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:01:06.685+0000 7f8fa37fe700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8fa4102440 0x7f8fa419f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:01:06.689 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:01:06.685+0000 7f8fa3fff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8fa4101240 0x7f8fa406b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:01:06.689 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:01:06.685+0000 7f8fa37fe700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8fa4102440 0x7f8fa419f6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.196:49464/0 (socket says 172.21.15.196:49464) 2024-03-18T16:01:06.689 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:01:06.685+0000 7f8fa37fe700 1 -- 172.21.15.196:0/1252067654 learned_addr learned my addr 172.21.15.196:0/1252067654 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-18T16:01:06.689 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:01:06.685+0000 7f8fa37fe700 1 -- 172.21.15.196:0/1252067654 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8fa4101240 msgr2=0x7f8fa406b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:01:06.689 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:01:06.685+0000 7f8fa37fe700 1 --2- 172.21.15.196:0/1252067654 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8fa4101240 0x7f8fa406b550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:06.689 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:01:06.685+0000 7f8fa37fe700 1 -- 172.21.15.196:0/1252067654 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8fa4102de0 msgr2=0x7f8fa41a1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:01:06.689 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:01:06.685+0000 7f8fa37fe700 1 --2- 172.21.15.196:0/1252067654 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8fa4102de0 0x7f8fa41a1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:06.689 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:01:06.685+0000 7f8fa37fe700 1 -- 172.21.15.196:0/1252067654 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fa41a2410 con 0x7f8fa4102440 2024-03-18T16:01:06.689 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:01:06.685+0000 7f8fa37fe700 1 --2- 172.21.15.196:0/1252067654 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8fa4102440 0x7f8fa419f6e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f8f9800b530 tx=0x7f8f9800b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:01:06.689 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:01:06.685+0000 7f8fa17fa700 1 -- 172.21.15.196:0/1252067654 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f98015020 con 0x7f8fa4102440 2024-03-18T16:01:06.689 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:01:06.685+0000 7f8fa17fa700 1 -- 172.21.15.196:0/1252067654 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f98011d10 con 0x7f8fa4102440 2024-03-18T16:01:06.689 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:01:06.685+0000 7f8faa4e9700 1 -- 172.21.15.196:0/1252067654 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fa41a2700 con 0x7f8fa4102440 2024-03-18T16:01:06.689 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:01:06.685+0000 7f8fa17fa700 1 -- 172.21.15.196:0/1252067654 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f9800f2e0 con 0x7f8fa4102440 2024-03-18T16:01:06.689 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:01:06.686+0000 7f8faa4e9700 1 -- 172.21.15.196:0/1252067654 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fa41a2cc0 con 0x7f8fa4102440 2024-03-18T16:01:06.690 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:01:06.687+0000 7f8faa4e9700 1 -- 172.21.15.196:0/1252067654 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8fa4068f90 con 0x7f8fa4102440 2024-03-18T16:01:06.690 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:01:06.688+0000 7f8fa17fa700 1 -- 172.21.15.196:0/1252067654 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 15) ==== 137234+0+0 (secure 0 0 0) 0x7f8f98018020 con 0x7f8fa4102440 2024-03-18T16:01:06.691 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:01:06.688+0000 7f8fa17fa700 1 --2- 172.21.15.196:0/1252067654 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f8f8c0a2800 0x7f8f8c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:01:06.691 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:01:06.688+0000 7f8fa17fa700 1 -- 172.21.15.196:0/1252067654 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2309+0+0 (secure 0 0 0) 0x7f8f980cca40 con 0x7f8fa4102440 2024-03-18T16:01:06.691 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:01:06.689+0000 7f8fa3fff700 1 --2- 172.21.15.196:0/1252067654 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f8f8c0a2800 0x7f8f8c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:01:06.691 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:01:06.690+0000 7f8fa3fff700 1 --2- 172.21.15.196:0/1252067654 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f8f8c0a2800 0x7f8f8c0a4cc0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8f94003890 tx=0x7f8f94005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:01:06.694 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:01:06.693+0000 7f8fa17fa700 1 -- 172.21.15.196:0/1252067654 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8f9801f020 con 0x7f8fa4102440 2024-03-18T16:01:06.900 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:01:06.899+0000 7f8faa4e9700 1 -- 172.21.15.196:0/1252067654 --> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f8fa4062710 con 0x7f8f8c0a2800 2024-03-18T16:01:07.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:07 smithi196 ceph-mon[36307]: pgmap v73: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T16:01:07.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:07 smithi196 ceph-mon[36307]: osdmap e14: 2 total, 2 up, 2 in 2024-03-18T16:01:07.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:07 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T16:01:07.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:07 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T16:01:07.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:07 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:01:07.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:07 smithi194 ceph-mon[36200]: pgmap v73: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T16:01:07.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:07 smithi194 ceph-mon[36200]: osdmap e14: 2 total, 2 up, 2 in 2024-03-18T16:01:07.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:07 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T16:01:07.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:07 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T16:01:07.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:07 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:01:07.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:07 smithi039 ceph-mon[31721]: pgmap v73: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-18T16:01:07.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:07 smithi039 ceph-mon[31721]: osdmap e14: 2 total, 2 up, 2 in 2024-03-18T16:01:07.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:07 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T16:01:07.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:07 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T16:01:07.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:07 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:01:08.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:08 smithi196 ceph-mon[36307]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:01:08.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:08 smithi194 ceph-mon[36200]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:01:08.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:08 smithi039 ceph-mon[31721]: from='client.24166 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:01:09.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:09 smithi194 ceph-mon[36200]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T16:01:09.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:09 smithi039 ceph-mon[31721]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T16:01:09.709 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:09 smithi196 ceph-mon[36307]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T16:01:10.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:10 smithi194 ceph-mon[36200]: from='client.? 172.21.15.196:0/2186123335' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3a718da4-2511-4ec1-b1a8-eafd6cce0df0"}]: dispatch 2024-03-18T16:01:10.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:10 smithi194 ceph-mon[36200]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3a718da4-2511-4ec1-b1a8-eafd6cce0df0"}]: dispatch 2024-03-18T16:01:10.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:10 smithi194 ceph-mon[36200]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3a718da4-2511-4ec1-b1a8-eafd6cce0df0"}]': finished 2024-03-18T16:01:10.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:10 smithi194 ceph-mon[36200]: osdmap e15: 3 total, 2 up, 3 in 2024-03-18T16:01:10.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:10 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T16:01:10.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:10 smithi194 ceph-mon[36200]: from='client.? 172.21.15.196:0/1879420833' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T16:01:10.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:10 smithi039 ceph-mon[31721]: from='client.? 172.21.15.196:0/2186123335' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3a718da4-2511-4ec1-b1a8-eafd6cce0df0"}]: dispatch 2024-03-18T16:01:10.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:10 smithi039 ceph-mon[31721]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3a718da4-2511-4ec1-b1a8-eafd6cce0df0"}]: dispatch 2024-03-18T16:01:10.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:10 smithi039 ceph-mon[31721]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3a718da4-2511-4ec1-b1a8-eafd6cce0df0"}]': finished 2024-03-18T16:01:10.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:10 smithi039 ceph-mon[31721]: osdmap e15: 3 total, 2 up, 3 in 2024-03-18T16:01:10.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:10 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T16:01:10.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:10 smithi039 ceph-mon[31721]: from='client.? 172.21.15.196:0/1879420833' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T16:01:10.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:10 smithi196 ceph-mon[36307]: from='client.? 172.21.15.196:0/2186123335' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3a718da4-2511-4ec1-b1a8-eafd6cce0df0"}]: dispatch 2024-03-18T16:01:10.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:10 smithi196 ceph-mon[36307]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3a718da4-2511-4ec1-b1a8-eafd6cce0df0"}]: dispatch 2024-03-18T16:01:10.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:10 smithi196 ceph-mon[36307]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3a718da4-2511-4ec1-b1a8-eafd6cce0df0"}]': finished 2024-03-18T16:01:10.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:10 smithi196 ceph-mon[36307]: osdmap e15: 3 total, 2 up, 3 in 2024-03-18T16:01:10.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:10 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T16:01:10.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:10 smithi196 ceph-mon[36307]: from='client.? 172.21.15.196:0/1879420833' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T16:01:11.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:11 smithi039 ceph-mon[31721]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T16:01:11.727 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:11 smithi196 ceph-mon[36307]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T16:01:11.739 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:11 smithi194 ceph-mon[36200]: pgmap v77: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T16:01:13.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:13 smithi039 ceph-mon[31721]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T16:01:13.596 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:13 smithi196 ceph-mon[36307]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T16:01:13.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:13 smithi194 ceph-mon[36200]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T16:01:15.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:15 smithi196 ceph-mon[36307]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T16:01:15.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:15 smithi039 ceph-mon[31721]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T16:01:15.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:15 smithi194 ceph-mon[36200]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T16:01:16.477 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:16 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-18T16:01:16.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:16 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:01:16.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:16 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-18T16:01:16.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:16 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:01:16.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:16 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-18T16:01:16.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:16 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:01:17.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:17 smithi039 ceph-mon[31721]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T16:01:17.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:17 smithi039 ceph-mon[31721]: Deploying daemon osd.2 on smithi196 2024-03-18T16:01:17.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:17 smithi196 ceph-mon[36307]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T16:01:17.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:17 smithi196 ceph-mon[36307]: Deploying daemon osd.2 on smithi196 2024-03-18T16:01:17.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:17 smithi194 ceph-mon[36200]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T16:01:17.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:17 smithi194 ceph-mon[36200]: Deploying daemon osd.2 on smithi196 2024-03-18T16:01:19.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:19 smithi039 ceph-mon[31721]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T16:01:19.590 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:19 smithi196 ceph-mon[36307]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T16:01:19.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:19 smithi194 ceph-mon[36200]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T16:01:21.477 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:21 smithi196 ceph-mon[36307]: pgmap v82: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T16:01:21.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:21 smithi039 ceph-mon[31721]: pgmap v82: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T16:01:21.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:21 smithi194 ceph-mon[36200]: pgmap v82: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T16:01:22.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:22 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:22.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:22 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:01:22.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:22 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:22.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:22 smithi196 ceph-mon[36307]: pgmap v83: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T16:01:22.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:22 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:22.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:22 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:01:22.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:22 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:22.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:22 smithi194 ceph-mon[36200]: pgmap v83: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T16:01:22.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:22 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:22.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:22 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:01:22.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:22 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:22.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:22 smithi039 ceph-mon[31721]: pgmap v83: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T16:01:24.832 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:24 smithi196 ceph-mon[36307]: pgmap v84: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T16:01:24.833 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:24 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:24.833 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:24 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:24.833 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:24 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:01:24.833 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:24 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:01:24.833 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:24 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:24.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:24 smithi039 ceph-mon[31721]: pgmap v84: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T16:01:24.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:24 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:24.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:24 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:24.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:24 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:01:24.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:24 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:01:24.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:24 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:24.885 INFO:teuthology.orchestra.run.smithi196.stdout:Created osd(s) 2 on host 'smithi196' 2024-03-18T16:01:24.885 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:01:24.883+0000 7f8fa17fa700 1 -- 172.21.15.196:0/1252067654 <== mgr.14152 v2:172.21.15.39:6800/2749214286 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f8fa4062710 con 0x7f8f8c0a2800 2024-03-18T16:01:24.886 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:01:24.885+0000 7f8faa4e9700 1 -- 172.21.15.196:0/1252067654 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f8f8c0a2800 msgr2=0x7f8f8c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:01:24.886 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:01:24.885+0000 7f8faa4e9700 1 --2- 172.21.15.196:0/1252067654 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f8f8c0a2800 0x7f8f8c0a4cc0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8f94003890 tx=0x7f8f94005fd0 comp rx=0 tx=0).stop 2024-03-18T16:01:24.886 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:01:24.885+0000 7f8faa4e9700 1 -- 172.21.15.196:0/1252067654 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8fa4102440 msgr2=0x7f8fa419f6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:01:24.887 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:01:24.885+0000 7f8faa4e9700 1 --2- 172.21.15.196:0/1252067654 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8fa4102440 0x7f8fa419f6e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f8f9800b530 tx=0x7f8f9800b840 comp rx=0 tx=0).stop 2024-03-18T16:01:24.887 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:01:24.885+0000 7f8faa4e9700 1 -- 172.21.15.196:0/1252067654 shutdown_connections 2024-03-18T16:01:24.887 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:01:24.885+0000 7f8faa4e9700 1 --2- 172.21.15.196:0/1252067654 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f8f8c0a2800 0x7f8f8c0a4cc0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:24.887 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:01:24.885+0000 7f8faa4e9700 1 --2- 172.21.15.196:0/1252067654 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8fa4102de0 0x7f8fa41a1c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:24.887 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:01:24.885+0000 7f8faa4e9700 1 --2- 172.21.15.196:0/1252067654 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8fa4101240 0x7f8fa406b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:24.887 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:01:24.885+0000 7f8faa4e9700 1 --2- 172.21.15.196:0/1252067654 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8fa4102440 0x7f8fa419f6e0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:24.887 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:01:24.885+0000 7f8faa4e9700 1 -- 172.21.15.196:0/1252067654 >> 172.21.15.196:0/1252067654 conn(0x7f8fa40fc7c0 msgr2=0x7f8fa4105740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:01:24.887 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:01:24.885+0000 7f8faa4e9700 1 -- 172.21.15.196:0/1252067654 shutdown_connections 2024-03-18T16:01:24.887 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:01:24.885+0000 7f8faa4e9700 1 -- 172.21.15.196:0/1252067654 wait complete. 2024-03-18T16:01:24.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:24 smithi194 ceph-mon[36200]: pgmap v84: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T16:01:24.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:24 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:24.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:24 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:24.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:24 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:01:24.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:24 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:01:24.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:24 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:25.503 DEBUG:teuthology.orchestra.run.smithi196:osd.2> sudo journalctl -f -n 0 -u ceph-1892d8fc-e540-11ee-95c9-87774f69a715@osd.2.service 2024-03-18T16:01:25.506 INFO:tasks.cephadm:Deploying osd.3 on smithi039 with /dev/vg_nvme/lv_3... 2024-03-18T16:01:25.506 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-18T16:01:25.974 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T16:01:26.129 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:25 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:26.129 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:25 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:26.129 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:25 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:01:26.130 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:25 smithi039 ceph-mon[31721]: from='osd.2 [v2:172.21.15.196:6800/3935874719,v1:172.21.15.196:6801/3935874719]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-18T16:01:26.130 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:25 smithi039 ceph-mon[31721]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-18T16:01:26.200 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:25 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:26.200 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:25 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:26.200 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:25 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:01:26.200 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:25 smithi196 ceph-mon[36307]: from='osd.2 [v2:172.21.15.196:6800/3935874719,v1:172.21.15.196:6801/3935874719]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-18T16:01:26.200 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:25 smithi196 ceph-mon[36307]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-18T16:01:26.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:25 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:26.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:25 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:26.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:25 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:01:26.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:25 smithi194 ceph-mon[36200]: from='osd.2 [v2:172.21.15.196:6800/3935874719,v1:172.21.15.196:6801/3935874719]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-18T16:01:26.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:25 smithi194 ceph-mon[36200]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-18T16:01:27.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:26 smithi196 ceph-mon[36307]: pgmap v85: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T16:01:27.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:26 smithi196 ceph-mon[36307]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-18T16:01:27.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:26 smithi196 ceph-mon[36307]: from='osd.2 [v2:172.21.15.196:6800/3935874719,v1:172.21.15.196:6801/3935874719]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-03-18T16:01:27.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:26 smithi196 ceph-mon[36307]: osdmap e16: 3 total, 2 up, 3 in 2024-03-18T16:01:27.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:26 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T16:01:27.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:26 smithi196 ceph-mon[36307]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-03-18T16:01:27.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:26 smithi194 ceph-mon[36200]: pgmap v85: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T16:01:27.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:26 smithi194 ceph-mon[36200]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-18T16:01:27.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:26 smithi194 ceph-mon[36200]: from='osd.2 [v2:172.21.15.196:6800/3935874719,v1:172.21.15.196:6801/3935874719]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-03-18T16:01:27.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:26 smithi194 ceph-mon[36200]: osdmap e16: 3 total, 2 up, 3 in 2024-03-18T16:01:27.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:26 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T16:01:27.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:26 smithi194 ceph-mon[36200]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-03-18T16:01:27.270 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:26 smithi039 ceph-mon[31721]: pgmap v85: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T16:01:27.270 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:26 smithi039 ceph-mon[31721]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-18T16:01:27.270 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:26 smithi039 ceph-mon[31721]: from='osd.2 [v2:172.21.15.196:6800/3935874719,v1:172.21.15.196:6801/3935874719]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-03-18T16:01:27.270 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:26 smithi039 ceph-mon[31721]: osdmap e16: 3 total, 2 up, 3 in 2024-03-18T16:01:27.270 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:26 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T16:01:27.270 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:26 smithi039 ceph-mon[31721]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-03-18T16:01:27.961 INFO:journalctl@ceph.osd.2.smithi196.stdout:Mar 18 16:01:27 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-2[41457]: 2024-03-18T16:01:27.672+0000 7fc0eee95700 -1 osd.2 0 waiting for initial osdmap 2024-03-18T16:01:27.961 INFO:journalctl@ceph.osd.2.smithi196.stdout:Mar 18 16:01:27 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-2[41457]: 2024-03-18T16:01:27.676+0000 7fc0e6c80700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T16:01:28.230 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:27 smithi196 ceph-mon[36307]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-03-18T16:01:28.230 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:27 smithi196 ceph-mon[36307]: osdmap e17: 3 total, 2 up, 3 in 2024-03-18T16:01:28.230 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:27 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T16:01:28.231 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:27 smithi196 ceph-mon[36307]: from='osd.2 ' entity='osd.2' 2024-03-18T16:01:28.231 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:27 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T16:01:28.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:27 smithi194 ceph-mon[36200]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-03-18T16:01:28.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:27 smithi194 ceph-mon[36200]: osdmap e17: 3 total, 2 up, 3 in 2024-03-18T16:01:28.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:27 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T16:01:28.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:27 smithi194 ceph-mon[36200]: from='osd.2 ' entity='osd.2' 2024-03-18T16:01:28.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:27 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T16:01:28.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:27 smithi039 ceph-mon[31721]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-03-18T16:01:28.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:27 smithi039 ceph-mon[31721]: osdmap e17: 3 total, 2 up, 3 in 2024-03-18T16:01:28.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:27 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T16:01:28.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:27 smithi039 ceph-mon[31721]: from='osd.2 ' entity='osd.2' 2024-03-18T16:01:28.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:27 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T16:01:28.805 INFO:teuthology.orchestra.run.smithi039.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-18T16:01:28.805 INFO:teuthology.orchestra.run.smithi039.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-18T16:01:28.805 INFO:teuthology.orchestra.run.smithi039.stderr: stderr: 10+0 records in 2024-03-18T16:01:28.806 INFO:teuthology.orchestra.run.smithi039.stderr:10+0 records out 2024-03-18T16:01:28.806 INFO:teuthology.orchestra.run.smithi039.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0339037 s, 309 MB/s 2024-03-18T16:01:28.806 INFO:teuthology.orchestra.run.smithi039.stderr: stderr: 2024-03-18T16:01:28.806 INFO:teuthology.orchestra.run.smithi039.stderr:--> Zapping successful for: 2024-03-18T16:01:29.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:28 smithi039 ceph-mon[31721]: purged_snaps scrub starts 2024-03-18T16:01:29.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:28 smithi039 ceph-mon[31721]: purged_snaps scrub ok 2024-03-18T16:01:29.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:28 smithi039 ceph-mon[31721]: pgmap v88: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T16:01:29.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:28 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:29.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:28 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:29.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:28 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:01:29.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:28 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:29.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:28 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:01:29.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:28 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:01:29.088 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:28 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:29.088 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:28 smithi039 ceph-mon[31721]: osd.2 [v2:172.21.15.196:6800/3935874719,v1:172.21.15.196:6801/3935874719] boot 2024-03-18T16:01:29.088 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:28 smithi039 ceph-mon[31721]: osdmap e18: 3 total, 3 up, 3 in 2024-03-18T16:01:29.088 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:28 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T16:01:29.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:28 smithi196 ceph-mon[36307]: purged_snaps scrub starts 2024-03-18T16:01:29.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:28 smithi196 ceph-mon[36307]: purged_snaps scrub ok 2024-03-18T16:01:29.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:28 smithi196 ceph-mon[36307]: pgmap v88: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T16:01:29.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:28 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:29.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:28 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:29.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:28 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:01:29.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:28 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:29.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:28 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:01:29.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:28 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:01:29.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:28 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:29.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:28 smithi196 ceph-mon[36307]: osd.2 [v2:172.21.15.196:6800/3935874719,v1:172.21.15.196:6801/3935874719] boot 2024-03-18T16:01:29.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:28 smithi196 ceph-mon[36307]: osdmap e18: 3 total, 3 up, 3 in 2024-03-18T16:01:29.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:28 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T16:01:29.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:28 smithi194 ceph-mon[36200]: purged_snaps scrub starts 2024-03-18T16:01:29.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:28 smithi194 ceph-mon[36200]: purged_snaps scrub ok 2024-03-18T16:01:29.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:28 smithi194 ceph-mon[36200]: pgmap v88: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-18T16:01:29.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:28 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:29.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:28 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:29.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:28 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:01:29.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:28 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:29.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:28 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:01:29.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:28 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:01:29.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:28 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:29.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:28 smithi194 ceph-mon[36200]: osd.2 [v2:172.21.15.196:6800/3935874719,v1:172.21.15.196:6801/3935874719] boot 2024-03-18T16:01:29.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:28 smithi194 ceph-mon[36200]: osdmap e18: 3 total, 3 up, 3 in 2024-03-18T16:01:29.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:28 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-18T16:01:29.597 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi039:vg_nvme/lv_3 2024-03-18T16:01:30.041 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T16:01:30.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:30 smithi039 ceph-mon[31721]: Detected new or changed devices on smithi196 2024-03-18T16:01:30.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:30 smithi039 ceph-mon[31721]: Adjusting osd_memory_target on smithi196 to 17080M 2024-03-18T16:01:30.477 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:30 smithi196 ceph-mon[36307]: Detected new or changed devices on smithi196 2024-03-18T16:01:30.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:30 smithi196 ceph-mon[36307]: Adjusting osd_memory_target on smithi196 to 17080M 2024-03-18T16:01:30.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:30 smithi194 ceph-mon[36200]: Detected new or changed devices on smithi196 2024-03-18T16:01:30.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:30 smithi194 ceph-mon[36200]: Adjusting osd_memory_target on smithi196 to 17080M 2024-03-18T16:01:31.216 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:31 smithi039 ceph-mon[31721]: pgmap v90: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-18T16:01:31.216 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:31 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-18T16:01:31.216 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:31 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-18T16:01:31.216 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:31 smithi039 ceph-mon[31721]: osdmap e19: 3 total, 3 up, 3 in 2024-03-18T16:01:31.216 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:31 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-18T16:01:31.436 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:31 smithi196 ceph-mon[36307]: pgmap v90: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-18T16:01:31.436 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:31 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-18T16:01:31.436 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:31 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-18T16:01:31.436 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:31 smithi196 ceph-mon[36307]: osdmap e19: 3 total, 3 up, 3 in 2024-03-18T16:01:31.436 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:31 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-18T16:01:31.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:31 smithi194 ceph-mon[36200]: pgmap v90: 0 pgs: ; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-18T16:01:31.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:31 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-18T16:01:31.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:31 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-18T16:01:31.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:31 smithi194 ceph-mon[36200]: osdmap e19: 3 total, 3 up, 3 in 2024-03-18T16:01:31.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:31 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-18T16:01:32.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:01:32.093+0000 7f1e87800700 1 -- 172.21.15.39:0/1968443861 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1e80102e30 msgr2=0x7f1e80105220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:01:32.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:01:32.093+0000 7f1e87800700 1 --2- 172.21.15.39:0/1968443861 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1e80102e30 0x7f1e80105220 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f1e74009990 tx=0x7f1e74009ca0 comp rx=0 tx=0).stop 2024-03-18T16:01:32.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:01:32.094+0000 7f1e87800700 1 -- 172.21.15.39:0/1968443861 shutdown_connections 2024-03-18T16:01:32.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:01:32.094+0000 7f1e87800700 1 --2- 172.21.15.39:0/1968443861 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e80108090 0x7f1e8010a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:32.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:01:32.094+0000 7f1e87800700 1 --2- 172.21.15.39:0/1968443861 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1e80102e30 0x7f1e80105220 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:32.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:01:32.094+0000 7f1e87800700 1 --2- 172.21.15.39:0/1968443861 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1e80105760 0x7f1e80107b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:32.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:01:32.094+0000 7f1e87800700 1 -- 172.21.15.39:0/1968443861 >> 172.21.15.39:0/1968443861 conn(0x7f1e800fc7a0 msgr2=0x7f1e800febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:01:32.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:01:32.094+0000 7f1e87800700 1 -- 172.21.15.39:0/1968443861 shutdown_connections 2024-03-18T16:01:32.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:01:32.094+0000 7f1e87800700 1 -- 172.21.15.39:0/1968443861 wait complete. 2024-03-18T16:01:32.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:01:32.095+0000 7f1e87800700 1 Processor -- start 2024-03-18T16:01:32.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:01:32.095+0000 7f1e87800700 1 -- start start 2024-03-18T16:01:32.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:01:32.096+0000 7f1e87800700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1e80102e30 0x7f1e8019a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:01:32.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:01:32.096+0000 7f1e87800700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e80105760 0x7f1e8019ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:01:32.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:01:32.096+0000 7f1e87800700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1e80108090 0x7f1e801a1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:01:32.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:01:32.096+0000 7f1e87800700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f1e8010df20 con 0x7f1e80105760 2024-03-18T16:01:32.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:01:32.096+0000 7f1e87800700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f1e8010dda0 con 0x7f1e80108090 2024-03-18T16:01:32.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:01:32.096+0000 7f1e87800700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f1e8010e0a0 con 0x7f1e80102e30 2024-03-18T16:01:32.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:01:32.096+0000 7f1e84e00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1e80102e30 0x7f1e8019a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:01:32.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:01:32.096+0000 7f1e84e00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1e80102e30 0x7f1e8019a700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:47956/0 (socket says 172.21.15.39:47956) 2024-03-18T16:01:32.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:01:32.096+0000 7f1e84e00700 1 -- 172.21.15.39:0/3498917891 learned_addr learned my addr 172.21.15.39:0/3498917891 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:01:32.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:01:32.097+0000 7f1e85800700 1 --2- 172.21.15.39:0/3498917891 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1e80108090 0x7f1e801a1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:01:32.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:01:32.097+0000 7f1e84e00700 1 -- 172.21.15.39:0/3498917891 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1e80108090 msgr2=0x7f1e801a1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:01:32.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:01:32.097+0000 7f1e84e00700 1 --2- 172.21.15.39:0/3498917891 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1e80108090 0x7f1e801a1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:32.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:01:32.097+0000 7f1e84e00700 1 -- 172.21.15.39:0/3498917891 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e80105760 msgr2=0x7f1e8019ac40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T16:01:32.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:01:32.098+0000 7f1e7fe00700 1 --2- 172.21.15.39:0/3498917891 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e80105760 0x7f1e8019ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:01:32.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:01:32.099+0000 7f1e84e00700 1 --2- 172.21.15.39:0/3498917891 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e80105760 0x7f1e8019ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:32.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:01:32.099+0000 7f1e84e00700 1 -- 172.21.15.39:0/3498917891 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e801a2360 con 0x7f1e80102e30 2024-03-18T16:01:32.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:01:32.099+0000 7f1e85800700 1 --2- 172.21.15.39:0/3498917891 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1e80108090 0x7f1e801a1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:01:32.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:01:32.100+0000 7f1e84e00700 1 --2- 172.21.15.39:0/3498917891 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1e80102e30 0x7f1e8019a700 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1e740098e0 tx=0x7f1e7400bc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:01:32.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:01:32.100+0000 7f1e7fe00700 1 --2- 172.21.15.39:0/3498917891 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e80105760 0x7f1e8019ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:01:32.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:01:32.100+0000 7f1e7d600700 1 -- 172.21.15.39:0/3498917891 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e74026070 con 0x7f1e80102e30 2024-03-18T16:01:32.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:01:32.100+0000 7f1e7d600700 1 -- 172.21.15.39:0/3498917891 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1e74003ae0 con 0x7f1e80102e30 2024-03-18T16:01:32.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:01:32.100+0000 7f1e7d600700 1 -- 172.21.15.39:0/3498917891 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e74021670 con 0x7f1e80102e30 2024-03-18T16:01:32.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:01:32.100+0000 7f1e87800700 1 -- 172.21.15.39:0/3498917891 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e801a25f0 con 0x7f1e80102e30 2024-03-18T16:01:32.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:01:32.101+0000 7f1e87800700 1 -- 172.21.15.39:0/3498917891 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e801937b0 con 0x7f1e80102e30 2024-03-18T16:01:32.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:01:32.103+0000 7f1e87800700 1 -- 172.21.15.39:0/3498917891 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e80068f90 con 0x7f1e80102e30 2024-03-18T16:01:32.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:01:32.105+0000 7f1e7d600700 1 -- 172.21.15.39:0/3498917891 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f1e74029070 con 0x7f1e80102e30 2024-03-18T16:01:32.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:01:32.107+0000 7f1e7d600700 1 --2- 172.21.15.39:0/3498917891 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1e680a2910 0x7f1e680a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:01:32.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:01:32.107+0000 7f1e7d600700 1 -- 172.21.15.39:0/3498917891 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 3237+0+0 (secure 0 0 0) 0x7f1e740ceb60 con 0x7f1e80102e30 2024-03-18T16:01:32.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:01:32.107+0000 7f1e7fe00700 1 --2- 172.21.15.39:0/3498917891 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1e680a2910 0x7f1e680a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:01:32.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:01:32.108+0000 7f1e7fe00700 1 --2- 172.21.15.39:0/3498917891 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1e680a2910 0x7f1e680a4dd0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f1e6c005fd0 tx=0x7f1e6c005e20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:01:32.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:01:32.108+0000 7f1e7d600700 1 -- 172.21.15.39:0/3498917891 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1e74013070 con 0x7f1e80102e30 2024-03-18T16:01:32.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:32 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-18T16:01:32.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:32 smithi039 ceph-mon[31721]: osdmap e20: 3 total, 3 up, 3 in 2024-03-18T16:01:32.337 INFO:journalctl@ceph.osd.0.smithi039.stdout:Mar 18 16:01:31 smithi039 sudo[48415]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-18T16:01:32.337 INFO:journalctl@ceph.osd.0.smithi039.stdout:Mar 18 16:01:31 smithi039 sudo[48415]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T16:01:32.338 INFO:journalctl@ceph.osd.0.smithi039.stdout:Mar 18 16:01:31 smithi039 sudo[48415]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T16:01:32.338 INFO:journalctl@ceph.osd.0.smithi039.stdout:Mar 18 16:01:32 smithi039 sudo[48415]: pam_unix(sudo:session): session closed for user root 2024-03-18T16:01:32.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:01:32.348+0000 7f1e87800700 1 -- 172.21.15.39:0/3498917891 --> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi039:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f1e80062710 con 0x7f1e680a2910 2024-03-18T16:01:32.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:32 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-18T16:01:32.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:32 smithi196 ceph-mon[36307]: osdmap e20: 3 total, 3 up, 3 in 2024-03-18T16:01:32.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:32 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-18T16:01:32.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:32 smithi194 ceph-mon[36200]: osdmap e20: 3 total, 3 up, 3 in 2024-03-18T16:01:32.836 INFO:journalctl@ceph.osd.0.smithi039.stdout:Mar 18 16:01:32 smithi039 sudo[48505]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-18T16:01:32.836 INFO:journalctl@ceph.osd.0.smithi039.stdout:Mar 18 16:01:32 smithi039 sudo[48505]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T16:01:32.837 INFO:journalctl@ceph.osd.0.smithi039.stdout:Mar 18 16:01:32 smithi039 sudo[48505]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T16:01:32.837 INFO:journalctl@ceph.osd.0.smithi039.stdout:Mar 18 16:01:32 smithi039 sudo[48505]: pam_unix(sudo:session): session closed for user root 2024-03-18T16:01:33.138 INFO:journalctl@ceph.osd.1.smithi194.stdout:Mar 18 16:01:32 smithi194 sudo[44965]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-18T16:01:33.139 INFO:journalctl@ceph.osd.1.smithi194.stdout:Mar 18 16:01:32 smithi194 sudo[44965]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T16:01:33.139 INFO:journalctl@ceph.osd.1.smithi194.stdout:Mar 18 16:01:32 smithi194 sudo[44965]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T16:01:33.139 INFO:journalctl@ceph.osd.1.smithi194.stdout:Mar 18 16:01:32 smithi194 sudo[44965]: pam_unix(sudo:session): session closed for user root 2024-03-18T16:01:33.139 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:33 smithi194 ceph-mon[36200]: pgmap v93: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-18T16:01:33.139 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:33 smithi194 ceph-mon[36200]: osdmap e21: 3 total, 3 up, 3 in 2024-03-18T16:01:33.139 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:33 smithi194 ceph-mon[36200]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-18T16:01:33.139 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:33 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T16:01:33.139 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:33 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T16:01:33.139 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:33 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:01:33.139 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:33 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:33.375 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:33 smithi039 ceph-mon[31721]: pgmap v93: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-18T16:01:33.375 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:33 smithi039 ceph-mon[31721]: osdmap e21: 3 total, 3 up, 3 in 2024-03-18T16:01:33.375 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:33 smithi039 ceph-mon[31721]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-18T16:01:33.375 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:33 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T16:01:33.375 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:33 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T16:01:33.375 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:33 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:01:33.375 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:33 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:33.413 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:33 smithi196 ceph-mon[36307]: pgmap v93: 1 pgs: 1 creating+peering; 0 B data, 88 MiB used, 268 GiB / 268 GiB avail 2024-03-18T16:01:33.413 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:33 smithi196 ceph-mon[36307]: osdmap e21: 3 total, 3 up, 3 in 2024-03-18T16:01:33.413 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:33 smithi196 ceph-mon[36307]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-18T16:01:33.413 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:33 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T16:01:33.413 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:33 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T16:01:33.413 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:33 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:01:33.413 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:33 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:33.490 INFO:journalctl@ceph.osd.1.smithi194.stdout:Mar 18 16:01:33 smithi194 sudo[44968]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-18T16:01:33.490 INFO:journalctl@ceph.osd.1.smithi194.stdout:Mar 18 16:01:33 smithi194 sudo[44968]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T16:01:33.490 INFO:journalctl@ceph.osd.1.smithi194.stdout:Mar 18 16:01:33 smithi194 sudo[44968]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T16:01:33.490 INFO:journalctl@ceph.osd.1.smithi194.stdout:Mar 18 16:01:33 smithi194 sudo[44968]: pam_unix(sudo:session): session closed for user root 2024-03-18T16:01:33.666 INFO:journalctl@ceph.osd.2.smithi196.stdout:Mar 18 16:01:33 smithi196 sudo[45113]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-18T16:01:33.666 INFO:journalctl@ceph.osd.2.smithi196.stdout:Mar 18 16:01:33 smithi196 sudo[45113]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T16:01:33.667 INFO:journalctl@ceph.osd.2.smithi196.stdout:Mar 18 16:01:33 smithi196 sudo[45113]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T16:01:33.667 INFO:journalctl@ceph.osd.2.smithi196.stdout:Mar 18 16:01:33 smithi196 sudo[45113]: pam_unix(sudo:session): session closed for user root 2024-03-18T16:01:33.978 INFO:journalctl@ceph.osd.2.smithi196.stdout:Mar 18 16:01:33 smithi196 sudo[45116]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-18T16:01:33.978 INFO:journalctl@ceph.osd.2.smithi196.stdout:Mar 18 16:01:33 smithi196 sudo[45116]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T16:01:33.978 INFO:journalctl@ceph.osd.2.smithi196.stdout:Mar 18 16:01:33 smithi196 sudo[45116]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T16:01:33.978 INFO:journalctl@ceph.osd.2.smithi196.stdout:Mar 18 16:01:33 smithi196 sudo[45116]: pam_unix(sudo:session): session closed for user root 2024-03-18T16:01:34.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:34 smithi039 sudo[48583]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-18T16:01:34.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:34 smithi039 sudo[48583]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T16:01:34.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:34 smithi039 sudo[48583]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T16:01:34.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:34 smithi039 sudo[48583]: pam_unix(sudo:session): session closed for user root 2024-03-18T16:01:34.848 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:34 smithi039 sudo[48612]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-18T16:01:34.849 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:34 smithi039 sudo[48612]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T16:01:34.849 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:34 smithi039 sudo[48612]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T16:01:34.849 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:34 smithi039 sudo[48612]: pam_unix(sudo:session): session closed for user root 2024-03-18T16:01:35.340 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:35 smithi196 sudo[45119]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-18T16:01:35.340 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:35 smithi196 sudo[45119]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T16:01:35.340 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:35 smithi196 sudo[45119]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T16:01:35.340 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:35 smithi196 sudo[45119]: pam_unix(sudo:session): session closed for user root 2024-03-18T16:01:35.600 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:35 smithi194 ceph-mon[36200]: from='client.24199 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi039:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:01:35.600 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:35 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:35.600 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:35 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:35.600 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:35 smithi194 ceph-mon[36200]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-18T16:01:35.600 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:35 smithi194 sudo[44971]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-18T16:01:35.601 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:35 smithi194 sudo[44971]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T16:01:35.601 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:35 smithi194 sudo[44971]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T16:01:35.601 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:35 smithi194 sudo[44971]: pam_unix(sudo:session): session closed for user root 2024-03-18T16:01:35.727 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:35 smithi196 sudo[45122]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-18T16:01:35.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:35 smithi196 sudo[45122]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T16:01:35.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:35 smithi196 sudo[45122]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T16:01:35.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:35 smithi196 sudo[45122]: pam_unix(sudo:session): session closed for user root 2024-03-18T16:01:35.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:35 smithi196 ceph-mon[36307]: from='client.24199 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi039:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:01:35.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:35 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:35.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:35 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:35.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:35 smithi196 ceph-mon[36307]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-18T16:01:35.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:35 smithi039 ceph-mon[31721]: from='client.24199 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi039:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:01:35.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:35 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:35.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:35 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:35.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:35 smithi039 ceph-mon[31721]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-18T16:01:35.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:35 smithi194 sudo[44974]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-18T16:01:35.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:35 smithi194 sudo[44974]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-18T16:01:35.991 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:35 smithi194 sudo[44974]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-18T16:01:35.991 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:35 smithi194 sudo[44974]: pam_unix(sudo:session): session closed for user root 2024-03-18T16:01:36.633 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:36 smithi039 ceph-mon[31721]: pgmap v95: 1 pgs: 1 creating+peering; 0 B data, 89 MiB used, 268 GiB / 268 GiB avail 2024-03-18T16:01:36.633 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:36 smithi039 ceph-mon[31721]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-18T16:01:36.633 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:36 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T16:01:36.633 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:36 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T16:01:36.633 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:36 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T16:01:36.634 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:36 smithi039 ceph-mon[31721]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-18T16:01:36.634 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:36 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T16:01:36.634 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:36 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T16:01:36.634 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:36 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T16:01:36.634 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:36 smithi039 ceph-mon[31721]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-18T16:01:36.634 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:36 smithi039 ceph-mon[31721]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-18T16:01:36.634 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:36 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T16:01:36.634 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:36 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T16:01:36.634 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:36 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T16:01:36.634 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:36 smithi039 ceph-mon[31721]: pgmap v96: 1 pgs: 1 creating+peering; 0 B data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-18T16:01:36.634 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:36 smithi039 ceph-mon[31721]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-18T16:01:36.634 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:36 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/4214032070' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6f9c37f9-0ab9-4038-8919-afd4b39c4165"}]: dispatch 2024-03-18T16:01:36.634 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:36 smithi039 ceph-mon[31721]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6f9c37f9-0ab9-4038-8919-afd4b39c4165"}]: dispatch 2024-03-18T16:01:36.635 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:36 smithi039 ceph-mon[31721]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6f9c37f9-0ab9-4038-8919-afd4b39c4165"}]': finished 2024-03-18T16:01:36.635 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:36 smithi039 ceph-mon[31721]: osdmap e22: 4 total, 3 up, 4 in 2024-03-18T16:01:36.635 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:36 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T16:01:36.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:36 smithi196 ceph-mon[36307]: pgmap v95: 1 pgs: 1 creating+peering; 0 B data, 89 MiB used, 268 GiB / 268 GiB avail 2024-03-18T16:01:36.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:36 smithi196 ceph-mon[36307]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-18T16:01:36.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:36 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T16:01:36.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:36 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T16:01:36.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:36 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T16:01:36.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:36 smithi196 ceph-mon[36307]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-18T16:01:36.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:36 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T16:01:36.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:36 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T16:01:36.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:36 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T16:01:36.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:36 smithi196 ceph-mon[36307]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-18T16:01:36.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:36 smithi196 ceph-mon[36307]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-18T16:01:36.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:36 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T16:01:36.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:36 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T16:01:36.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:36 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T16:01:36.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:36 smithi196 ceph-mon[36307]: pgmap v96: 1 pgs: 1 creating+peering; 0 B data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-18T16:01:36.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:36 smithi196 ceph-mon[36307]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-18T16:01:36.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:36 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/4214032070' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6f9c37f9-0ab9-4038-8919-afd4b39c4165"}]: dispatch 2024-03-18T16:01:36.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:36 smithi196 ceph-mon[36307]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6f9c37f9-0ab9-4038-8919-afd4b39c4165"}]: dispatch 2024-03-18T16:01:36.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:36 smithi196 ceph-mon[36307]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6f9c37f9-0ab9-4038-8919-afd4b39c4165"}]': finished 2024-03-18T16:01:36.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:36 smithi196 ceph-mon[36307]: osdmap e22: 4 total, 3 up, 4 in 2024-03-18T16:01:36.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:36 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T16:01:36.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:36 smithi194 ceph-mon[36200]: pgmap v95: 1 pgs: 1 creating+peering; 0 B data, 89 MiB used, 268 GiB / 268 GiB avail 2024-03-18T16:01:36.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:36 smithi194 ceph-mon[36200]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-18T16:01:36.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:36 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T16:01:36.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:36 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T16:01:36.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:36 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T16:01:36.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:36 smithi194 ceph-mon[36200]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-18T16:01:36.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:36 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T16:01:36.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:36 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T16:01:36.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:36 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T16:01:36.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:36 smithi194 ceph-mon[36200]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-18T16:01:36.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:36 smithi194 ceph-mon[36200]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-18T16:01:36.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:36 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T16:01:36.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:36 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T16:01:36.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:36 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T16:01:36.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:36 smithi194 ceph-mon[36200]: pgmap v96: 1 pgs: 1 creating+peering; 0 B data, 90 MiB used, 268 GiB / 268 GiB avail 2024-03-18T16:01:36.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:36 smithi194 ceph-mon[36200]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-18T16:01:36.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:36 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/4214032070' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6f9c37f9-0ab9-4038-8919-afd4b39c4165"}]: dispatch 2024-03-18T16:01:36.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:36 smithi194 ceph-mon[36200]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6f9c37f9-0ab9-4038-8919-afd4b39c4165"}]: dispatch 2024-03-18T16:01:36.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:36 smithi194 ceph-mon[36200]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6f9c37f9-0ab9-4038-8919-afd4b39c4165"}]': finished 2024-03-18T16:01:36.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:36 smithi194 ceph-mon[36200]: osdmap e22: 4 total, 3 up, 4 in 2024-03-18T16:01:36.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:36 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T16:01:37.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:37 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/2218526056' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T16:01:37.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:37 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/2218526056' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T16:01:37.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:37 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/2218526056' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T16:01:38.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:38 smithi196 ceph-mon[36307]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-03-18T16:01:38.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:38 smithi194 ceph-mon[36200]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-03-18T16:01:38.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:38 smithi039 ceph-mon[31721]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 95 MiB used, 268 GiB / 268 GiB avail 2024-03-18T16:01:40.571 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:40 smithi039 ceph-mon[31721]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail 2024-03-18T16:01:40.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:40 smithi196 ceph-mon[36307]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail 2024-03-18T16:01:40.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:40 smithi194 ceph-mon[36200]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail 2024-03-18T16:01:42.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:42 smithi196 ceph-mon[36307]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 49 KiB/s wr, 4 op/s 2024-03-18T16:01:42.739 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:42 smithi194 ceph-mon[36200]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 49 KiB/s wr, 4 op/s 2024-03-18T16:01:42.788 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:42 smithi039 ceph-mon[31721]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 49 KiB/s wr, 4 op/s 2024-03-18T16:01:44.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:44 smithi196 ceph-mon[36307]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail; 1.2 KiB/s rd, 36 KiB/s wr, 3 op/s 2024-03-18T16:01:44.739 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:44 smithi194 ceph-mon[36200]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail; 1.2 KiB/s rd, 36 KiB/s wr, 3 op/s 2024-03-18T16:01:44.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:44 smithi039 ceph-mon[31721]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail; 1.2 KiB/s rd, 36 KiB/s wr, 3 op/s 2024-03-18T16:01:45.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:45 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-18T16:01:45.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:45 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:01:45.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:45 smithi196 ceph-mon[36307]: Deploying daemon osd.3 on smithi039 2024-03-18T16:01:45.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:45 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-18T16:01:45.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:45 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:01:45.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:45 smithi194 ceph-mon[36200]: Deploying daemon osd.3 on smithi039 2024-03-18T16:01:45.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:45 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-18T16:01:45.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:45 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:01:45.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:45 smithi039 ceph-mon[31721]: Deploying daemon osd.3 on smithi039 2024-03-18T16:01:46.627 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:46 smithi039 ceph-mon[31721]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail; 983 B/s rd, 28 KiB/s wr, 2 op/s 2024-03-18T16:01:46.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:46 smithi194 ceph-mon[36200]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail; 983 B/s rd, 28 KiB/s wr, 2 op/s 2024-03-18T16:01:46.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:46 smithi196 ceph-mon[36307]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail; 983 B/s rd, 28 KiB/s wr, 2 op/s 2024-03-18T16:01:48.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:48 smithi194 ceph-mon[36200]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail; 810 B/s rd, 23 KiB/s wr, 2 op/s 2024-03-18T16:01:48.800 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:48 smithi039 ceph-mon[31721]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail; 810 B/s rd, 23 KiB/s wr, 2 op/s 2024-03-18T16:01:48.977 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:48 smithi196 ceph-mon[36307]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail; 810 B/s rd, 23 KiB/s wr, 2 op/s 2024-03-18T16:01:50.817 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:50 smithi039 ceph-mon[31721]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-03-18T16:01:50.977 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:50 smithi196 ceph-mon[36307]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-03-18T16:01:50.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:50 smithi194 ceph-mon[36200]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-03-18T16:01:52.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:52 smithi039 ceph-mon[31721]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-03-18T16:01:52.977 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:52 smithi196 ceph-mon[36307]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-03-18T16:01:52.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:52 smithi194 ceph-mon[36200]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-03-18T16:01:54.642 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:54 smithi039 ceph-mon[31721]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail 2024-03-18T16:01:54.642 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:54 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:54.642 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:54 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:54.642 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:54 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:01:54.727 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:54 smithi196 ceph-mon[36307]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail 2024-03-18T16:01:54.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:54 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:54.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:54 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:54.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:54 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:01:54.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:54 smithi194 ceph-mon[36200]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail 2024-03-18T16:01:54.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:54 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:54.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:54 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:54.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:54 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:01:56.723 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:56 smithi039 ceph-mon[31721]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail 2024-03-18T16:01:56.727 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:56 smithi196 ceph-mon[36307]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail 2024-03-18T16:01:56.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:56 smithi194 ceph-mon[36200]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail 2024-03-18T16:01:58.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:58 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:58.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:58 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:58.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:58 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:01:58.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:58 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:01:58.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:58 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:58.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:58 smithi039 ceph-mon[31721]: from='osd.3 [v2:172.21.15.39:6810/893345939,v1:172.21.15.39:6811/893345939]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-18T16:01:58.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:58 smithi039 ceph-mon[31721]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-18T16:01:58.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:58 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:58.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:58 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:58.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:58 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:01:58.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:58 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:01:58.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:58 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:58.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:58 smithi196 ceph-mon[36307]: from='osd.3 [v2:172.21.15.39:6810/893345939,v1:172.21.15.39:6811/893345939]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-18T16:01:58.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:58 smithi196 ceph-mon[36307]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-18T16:01:58.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:58 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:58.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:58 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:58.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:58 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:01:58.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:58 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:01:58.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:58 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:58.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:58 smithi194 ceph-mon[36200]: from='osd.3 [v2:172.21.15.39:6810/893345939,v1:172.21.15.39:6811/893345939]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-18T16:01:58.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:58 smithi194 ceph-mon[36200]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-18T16:01:58.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:01:58.552+0000 7f1e7d600700 1 -- 172.21.15.39:0/3498917891 <== mgr.14152 v2:172.21.15.39:6800/2749214286 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f1e80062710 con 0x7f1e680a2910 2024-03-18T16:01:58.555 INFO:teuthology.orchestra.run.smithi039.stdout:Created osd(s) 3 on host 'smithi039' 2024-03-18T16:01:58.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:01:58.556+0000 7f1e87800700 1 -- 172.21.15.39:0/3498917891 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1e680a2910 msgr2=0x7f1e680a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:01:58.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:01:58.556+0000 7f1e87800700 1 --2- 172.21.15.39:0/3498917891 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1e680a2910 0x7f1e680a4dd0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f1e6c005fd0 tx=0x7f1e6c005e20 comp rx=0 tx=0).stop 2024-03-18T16:01:58.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:01:58.556+0000 7f1e87800700 1 -- 172.21.15.39:0/3498917891 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1e80102e30 msgr2=0x7f1e8019a700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:01:58.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:01:58.556+0000 7f1e87800700 1 --2- 172.21.15.39:0/3498917891 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1e80102e30 0x7f1e8019a700 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1e740098e0 tx=0x7f1e7400bc60 comp rx=0 tx=0).stop 2024-03-18T16:01:58.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:01:58.556+0000 7f1e87800700 1 -- 172.21.15.39:0/3498917891 shutdown_connections 2024-03-18T16:01:58.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:01:58.556+0000 7f1e87800700 1 --2- 172.21.15.39:0/3498917891 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1e680a2910 0x7f1e680a4dd0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:58.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:01:58.556+0000 7f1e87800700 1 --2- 172.21.15.39:0/3498917891 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1e80108090 0x7f1e801a1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:58.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:01:58.556+0000 7f1e87800700 1 --2- 172.21.15.39:0/3498917891 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1e80102e30 0x7f1e8019a700 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:58.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:01:58.556+0000 7f1e87800700 1 --2- 172.21.15.39:0/3498917891 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1e80105760 0x7f1e8019ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:01:58.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:01:58.556+0000 7f1e87800700 1 -- 172.21.15.39:0/3498917891 >> 172.21.15.39:0/3498917891 conn(0x7f1e800fc7a0 msgr2=0x7f1e80102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:01:58.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:01:58.557+0000 7f1e87800700 1 -- 172.21.15.39:0/3498917891 shutdown_connections 2024-03-18T16:01:58.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:01:58.557+0000 7f1e87800700 1 -- 172.21.15.39:0/3498917891 wait complete. 2024-03-18T16:01:59.385 DEBUG:teuthology.orchestra.run.smithi039:osd.3> sudo journalctl -f -n 0 -u ceph-1892d8fc-e540-11ee-95c9-87774f69a715@osd.3.service 2024-03-18T16:01:59.390 INFO:tasks.cephadm:Deploying osd.4 on smithi194 with /dev/vg_nvme/lv_3... 2024-03-18T16:01:59.390 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-18T16:01:59.579 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:59 smithi039 ceph-mon[31721]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail 2024-03-18T16:01:59.580 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:59 smithi039 ceph-mon[31721]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-18T16:01:59.580 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:59 smithi039 ceph-mon[31721]: from='osd.3 [v2:172.21.15.39:6810/893345939,v1:172.21.15.39:6811/893345939]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]: dispatch 2024-03-18T16:01:59.580 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:59 smithi039 ceph-mon[31721]: osdmap e23: 4 total, 3 up, 4 in 2024-03-18T16:01:59.580 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:59 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T16:01:59.580 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:59 smithi039 ceph-mon[31721]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]: dispatch 2024-03-18T16:01:59.580 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:59 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:59.580 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:59 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:59.580 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:01:59 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:01:59.636 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T16:01:59.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:59 smithi196 ceph-mon[36307]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail 2024-03-18T16:01:59.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:59 smithi196 ceph-mon[36307]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-18T16:01:59.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:59 smithi196 ceph-mon[36307]: from='osd.3 [v2:172.21.15.39:6810/893345939,v1:172.21.15.39:6811/893345939]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]: dispatch 2024-03-18T16:01:59.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:59 smithi196 ceph-mon[36307]: osdmap e23: 4 total, 3 up, 4 in 2024-03-18T16:01:59.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:59 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T16:01:59.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:59 smithi196 ceph-mon[36307]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]: dispatch 2024-03-18T16:01:59.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:59 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:59.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:59 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:59.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:01:59 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:01:59.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:59 smithi194 ceph-mon[36200]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail 2024-03-18T16:01:59.742 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:59 smithi194 ceph-mon[36200]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-18T16:01:59.742 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:59 smithi194 ceph-mon[36200]: from='osd.3 [v2:172.21.15.39:6810/893345939,v1:172.21.15.39:6811/893345939]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]: dispatch 2024-03-18T16:01:59.742 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:59 smithi194 ceph-mon[36200]: osdmap e23: 4 total, 3 up, 4 in 2024-03-18T16:01:59.742 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:59 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T16:01:59.742 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:59 smithi194 ceph-mon[36200]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]: dispatch 2024-03-18T16:01:59.742 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:59 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:59.742 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:59 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:01:59.742 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:01:59 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:02:00.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:00 smithi194 ceph-mon[36200]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]': finished 2024-03-18T16:02:00.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:00 smithi194 ceph-mon[36200]: osdmap e24: 4 total, 3 up, 4 in 2024-03-18T16:02:00.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:00 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T16:02:00.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:00 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T16:02:00.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:00 smithi194 ceph-mon[36200]: osd.3 [v2:172.21.15.39:6810/893345939,v1:172.21.15.39:6811/893345939] boot 2024-03-18T16:02:00.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:00 smithi194 ceph-mon[36200]: osdmap e25: 4 total, 4 up, 4 in 2024-03-18T16:02:00.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:00 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T16:02:00.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:00 smithi039 ceph-mon[31721]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]': finished 2024-03-18T16:02:00.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:00 smithi039 ceph-mon[31721]: osdmap e24: 4 total, 3 up, 4 in 2024-03-18T16:02:00.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:00 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T16:02:00.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:00 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T16:02:00.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:00 smithi039 ceph-mon[31721]: osd.3 [v2:172.21.15.39:6810/893345939,v1:172.21.15.39:6811/893345939] boot 2024-03-18T16:02:00.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:00 smithi039 ceph-mon[31721]: osdmap e25: 4 total, 4 up, 4 in 2024-03-18T16:02:00.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:00 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T16:02:00.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:00 smithi196 ceph-mon[36307]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]': finished 2024-03-18T16:02:00.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:00 smithi196 ceph-mon[36307]: osdmap e24: 4 total, 3 up, 4 in 2024-03-18T16:02:00.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:00 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T16:02:00.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:00 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T16:02:00.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:00 smithi196 ceph-mon[36307]: osd.3 [v2:172.21.15.39:6810/893345939,v1:172.21.15.39:6811/893345939] boot 2024-03-18T16:02:00.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:00 smithi196 ceph-mon[36307]: osdmap e25: 4 total, 4 up, 4 in 2024-03-18T16:02:00.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:00 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-18T16:02:01.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:01 smithi194 ceph-mon[36200]: purged_snaps scrub starts 2024-03-18T16:02:01.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:01 smithi194 ceph-mon[36200]: purged_snaps scrub ok 2024-03-18T16:02:01.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:01 smithi194 ceph-mon[36200]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail 2024-03-18T16:02:01.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:01 smithi194 ceph-mon[36200]: osdmap e26: 4 total, 4 up, 4 in 2024-03-18T16:02:01.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:01 smithi039 ceph-mon[31721]: purged_snaps scrub starts 2024-03-18T16:02:01.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:01 smithi039 ceph-mon[31721]: purged_snaps scrub ok 2024-03-18T16:02:01.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:01 smithi039 ceph-mon[31721]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail 2024-03-18T16:02:01.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:01 smithi039 ceph-mon[31721]: osdmap e26: 4 total, 4 up, 4 in 2024-03-18T16:02:01.611 INFO:teuthology.orchestra.run.smithi194.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-18T16:02:01.611 INFO:teuthology.orchestra.run.smithi194.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-18T16:02:01.611 INFO:teuthology.orchestra.run.smithi194.stderr: stderr: 10+0 records in 2024-03-18T16:02:01.611 INFO:teuthology.orchestra.run.smithi194.stderr:10+0 records out 2024-03-18T16:02:01.611 INFO:teuthology.orchestra.run.smithi194.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0117956 s, 889 MB/s 2024-03-18T16:02:01.611 INFO:teuthology.orchestra.run.smithi194.stderr:--> Zapping successful for: 2024-03-18T16:02:01.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:01 smithi196 ceph-mon[36307]: purged_snaps scrub starts 2024-03-18T16:02:01.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:01 smithi196 ceph-mon[36307]: purged_snaps scrub ok 2024-03-18T16:02:01.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:01 smithi196 ceph-mon[36307]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 99 MiB used, 268 GiB / 268 GiB avail 2024-03-18T16:02:01.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:01 smithi196 ceph-mon[36307]: osdmap e26: 4 total, 4 up, 4 in 2024-03-18T16:02:02.303 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi194:vg_nvme/lv_3 2024-03-18T16:02:02.544 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T16:02:03.175 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:02 smithi039 ceph-mon[31721]: pgmap v114: 1 pgs: 1 remapped+peering; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-18T16:02:03.175 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:02 smithi039 ceph-mon[31721]: osdmap e27: 4 total, 4 up, 4 in 2024-03-18T16:02:03.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:02 smithi196 ceph-mon[36307]: pgmap v114: 1 pgs: 1 remapped+peering; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-18T16:02:03.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:02 smithi196 ceph-mon[36307]: osdmap e27: 4 total, 4 up, 4 in 2024-03-18T16:02:03.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:02 smithi194 ceph-mon[36200]: pgmap v114: 1 pgs: 1 remapped+peering; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-18T16:02:03.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:02 smithi194 ceph-mon[36200]: osdmap e27: 4 total, 4 up, 4 in 2024-03-18T16:02:03.966 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:03 smithi194 ceph-mon[36200]: osdmap e28: 4 total, 4 up, 4 in 2024-03-18T16:02:04.167 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:02:04.163+0000 7ff48f951700 1 -- 172.21.15.194:0/2350385217 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff488102300 msgr2=0x7ff488102720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:02:04.167 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:02:04.163+0000 7ff48f951700 1 --2- 172.21.15.194:0/2350385217 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff488102300 0x7ff488102720 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff4840099c0 tx=0x7ff484009cd0 comp rx=0 tx=0).stop 2024-03-18T16:02:04.167 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:02:04.163+0000 7ff48f951700 1 -- 172.21.15.194:0/2350385217 shutdown_connections 2024-03-18T16:02:04.167 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:02:04.163+0000 7ff48f951700 1 --2- 172.21.15.194:0/2350385217 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff488103ea0 0x7ff48810a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:04.167 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:02:04.163+0000 7ff48f951700 1 --2- 172.21.15.194:0/2350385217 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff488102300 0x7ff488102720 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:04.167 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:02:04.163+0000 7ff48f951700 1 --2- 172.21.15.194:0/2350385217 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff488103500 0x7ff488103960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:04.167 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:02:04.163+0000 7ff48f951700 1 -- 172.21.15.194:0/2350385217 >> 172.21.15.194:0/2350385217 conn(0x7ff4880fd8e0 msgr2=0x7ff4880ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:02:04.167 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:02:04.163+0000 7ff48f951700 1 -- 172.21.15.194:0/2350385217 shutdown_connections 2024-03-18T16:02:04.167 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:02:04.163+0000 7ff48f951700 1 -- 172.21.15.194:0/2350385217 wait complete. 2024-03-18T16:02:04.168 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:02:04.164+0000 7ff48f951700 1 Processor -- start 2024-03-18T16:02:04.168 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:02:04.164+0000 7ff48f951700 1 -- start start 2024-03-18T16:02:04.168 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:02:04.164+0000 7ff48f951700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff488102300 0x7ff488076070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:04.168 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:02:04.164+0000 7ff48f951700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff488103500 0x7ff4880746c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:04.168 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:02:04.164+0000 7ff48f951700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff488103ea0 0x7ff488074c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:04.168 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:02:04.164+0000 7ff48f951700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7ff48810dfc0 con 0x7ff488103500 2024-03-18T16:02:04.168 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:02:04.164+0000 7ff48f951700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ff48810de40 con 0x7ff488103ea0 2024-03-18T16:02:04.168 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:02:04.164+0000 7ff48f951700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7ff48810e140 con 0x7ff488102300 2024-03-18T16:02:04.168 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:02:04.164+0000 7ff48deee700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff488103ea0 0x7ff488074c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:04.168 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:02:04.164+0000 7ff48deee700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff488103ea0 0x7ff488074c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:57960/0 (socket says 172.21.15.194:57960) 2024-03-18T16:02:04.168 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:02:04.164+0000 7ff48deee700 1 -- 172.21.15.194:0/1984239060 learned_addr learned my addr 172.21.15.194:0/1984239060 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-03-18T16:02:04.168 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:02:04.164+0000 7ff48d6ed700 1 --2- 172.21.15.194:0/1984239060 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff488102300 0x7ff488076070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:04.169 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:02:04.164+0000 7ff48deee700 1 -- 172.21.15.194:0/1984239060 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff488102300 msgr2=0x7ff488076070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:02:04.169 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:02:04.164+0000 7ff48deee700 1 --2- 172.21.15.194:0/1984239060 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff488102300 0x7ff488076070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:04.169 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:02:04.164+0000 7ff48deee700 1 -- 172.21.15.194:0/1984239060 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff488103500 msgr2=0x7ff4880746c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:02:04.169 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:02:04.164+0000 7ff48deee700 1 --2- 172.21.15.194:0/1984239060 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff488103500 0x7ff4880746c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:04.169 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:02:04.164+0000 7ff48deee700 1 -- 172.21.15.194:0/1984239060 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4880754b0 con 0x7ff488103ea0 2024-03-18T16:02:04.169 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:02:04.164+0000 7ff48deee700 1 --2- 172.21.15.194:0/1984239060 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff488103ea0 0x7ff488074c00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff48000b970 tx=0x7ff48000bc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:02:04.169 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:02:04.164+0000 7ff47e7fc700 1 -- 172.21.15.194:0/1984239060 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff480019070 con 0x7ff488103ea0 2024-03-18T16:02:04.169 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:02:04.165+0000 7ff47e7fc700 1 -- 172.21.15.194:0/1984239060 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff48000ce10 con 0x7ff488103ea0 2024-03-18T16:02:04.169 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:02:04.165+0000 7ff48f951700 1 -- 172.21.15.194:0/1984239060 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4881a2820 con 0x7ff488103ea0 2024-03-18T16:02:04.169 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:02:04.165+0000 7ff47e7fc700 1 -- 172.21.15.194:0/1984239060 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff480007370 con 0x7ff488103ea0 2024-03-18T16:02:04.169 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:02:04.165+0000 7ff48f951700 1 -- 172.21.15.194:0/1984239060 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4881a2db0 con 0x7ff488103ea0 2024-03-18T16:02:04.170 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:02:04.166+0000 7ff48f951700 1 -- 172.21.15.194:0/1984239060 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff488068f90 con 0x7ff488103ea0 2024-03-18T16:02:04.170 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:02:04.167+0000 7ff47e7fc700 1 -- 172.21.15.194:0/1984239060 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7ff48001c070 con 0x7ff488103ea0 2024-03-18T16:02:04.170 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:02:04.167+0000 7ff47e7fc700 1 --2- 172.21.15.194:0/1984239060 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff4740a2910 0x7ff4740a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:04.170 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:02:04.167+0000 7ff47e7fc700 1 -- 172.21.15.194:0/1984239060 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 3669+0+0 (secure 0 0 0) 0x7ff4800ccba0 con 0x7ff488103ea0 2024-03-18T16:02:04.171 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:02:04.168+0000 7ff48d6ed700 1 --2- 172.21.15.194:0/1984239060 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff4740a2910 0x7ff4740a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:04.171 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:02:04.168+0000 7ff48d6ed700 1 --2- 172.21.15.194:0/1984239060 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff4740a2910 0x7ff4740a4dd0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff48400b580 tx=0x7ff484005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:02:04.173 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:02:04.172+0000 7ff47e7fc700 1 -- 172.21.15.194:0/1984239060 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff48009ae60 con 0x7ff488103ea0 2024-03-18T16:02:04.227 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:03 smithi196 ceph-mon[36307]: osdmap e28: 4 total, 4 up, 4 in 2024-03-18T16:02:04.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:03 smithi039 ceph-mon[31721]: osdmap e28: 4 total, 4 up, 4 in 2024-03-18T16:02:04.381 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:02:04.379+0000 7ff48f951700 1 -- 172.21.15.194:0/1984239060 --> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7ff488062710 con 0x7ff4740a2910 2024-03-18T16:02:05.223 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:04 smithi194 ceph-mon[36200]: pgmap v117: 1 pgs: 1 remapped+peering; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-18T16:02:05.223 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:04 smithi194 ceph-mon[36200]: osdmap e29: 4 total, 4 up, 4 in 2024-03-18T16:02:05.224 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:04 smithi194 ceph-mon[36200]: Detected new or changed devices on smithi039 2024-03-18T16:02:05.224 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:04 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:02:05.224 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:04 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:02:05.224 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:04 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:02:05.224 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:04 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:02:05.224 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:04 smithi194 ceph-mon[36200]: Adjusting osd_memory_target on smithi039 to 8545M 2024-03-18T16:02:05.224 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:04 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:02:05.224 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:04 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:02:05.224 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:04 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:02:05.224 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:04 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:02:05.224 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:04 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T16:02:05.224 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:04 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T16:02:05.224 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:04 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:02:05.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:04 smithi196 ceph-mon[36307]: pgmap v117: 1 pgs: 1 remapped+peering; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-18T16:02:05.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:04 smithi196 ceph-mon[36307]: osdmap e29: 4 total, 4 up, 4 in 2024-03-18T16:02:05.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:04 smithi196 ceph-mon[36307]: Detected new or changed devices on smithi039 2024-03-18T16:02:05.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:04 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:02:05.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:04 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:02:05.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:04 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:02:05.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:04 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:02:05.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:04 smithi196 ceph-mon[36307]: Adjusting osd_memory_target on smithi039 to 8545M 2024-03-18T16:02:05.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:04 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:02:05.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:04 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:02:05.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:04 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:02:05.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:04 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:02:05.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:04 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T16:02:05.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:04 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T16:02:05.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:04 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:02:05.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:04 smithi039 ceph-mon[31721]: pgmap v117: 1 pgs: 1 remapped+peering; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-18T16:02:05.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:04 smithi039 ceph-mon[31721]: osdmap e29: 4 total, 4 up, 4 in 2024-03-18T16:02:05.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:04 smithi039 ceph-mon[31721]: Detected new or changed devices on smithi039 2024-03-18T16:02:05.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:04 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:02:05.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:04 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:02:05.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:04 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:02:05.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:04 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:02:05.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:04 smithi039 ceph-mon[31721]: Adjusting osd_memory_target on smithi039 to 8545M 2024-03-18T16:02:05.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:04 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:02:05.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:04 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:02:05.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:04 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:02:05.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:04 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:02:05.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:04 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T16:02:05.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:04 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T16:02:05.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:04 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:02:06.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:05 smithi196 ceph-mon[36307]: from='client.24218 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:02:06.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:05 smithi194 ceph-mon[36200]: from='client.24218 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:02:06.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:05 smithi039 ceph-mon[31721]: from='client.24218 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:02:07.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:06 smithi196 ceph-mon[36307]: pgmap v119: 1 pgs: 1 remapped+peering; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-18T16:02:07.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:06 smithi194 ceph-mon[36200]: pgmap v119: 1 pgs: 1 remapped+peering; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-18T16:02:07.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:06 smithi039 ceph-mon[31721]: pgmap v119: 1 pgs: 1 remapped+peering; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail 2024-03-18T16:02:08.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:07 smithi196 ceph-mon[36307]: from='client.? 172.21.15.194:0/262399952' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e795bb0e-4791-4537-acda-1a321e19bc9c"}]: dispatch 2024-03-18T16:02:08.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:07 smithi196 ceph-mon[36307]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e795bb0e-4791-4537-acda-1a321e19bc9c"}]: dispatch 2024-03-18T16:02:08.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:07 smithi196 ceph-mon[36307]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e795bb0e-4791-4537-acda-1a321e19bc9c"}]': finished 2024-03-18T16:02:08.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:07 smithi196 ceph-mon[36307]: osdmap e30: 5 total, 4 up, 5 in 2024-03-18T16:02:08.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:07 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T16:02:08.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:07 smithi196 ceph-mon[36307]: from='client.? 172.21.15.194:0/2224413971' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T16:02:08.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:07 smithi194 ceph-mon[36200]: from='client.? 172.21.15.194:0/262399952' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e795bb0e-4791-4537-acda-1a321e19bc9c"}]: dispatch 2024-03-18T16:02:08.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:07 smithi194 ceph-mon[36200]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e795bb0e-4791-4537-acda-1a321e19bc9c"}]: dispatch 2024-03-18T16:02:08.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:07 smithi194 ceph-mon[36200]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e795bb0e-4791-4537-acda-1a321e19bc9c"}]': finished 2024-03-18T16:02:08.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:07 smithi194 ceph-mon[36200]: osdmap e30: 5 total, 4 up, 5 in 2024-03-18T16:02:08.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:07 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T16:02:08.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:07 smithi194 ceph-mon[36200]: from='client.? 172.21.15.194:0/2224413971' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T16:02:08.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:07 smithi039 ceph-mon[31721]: from='client.? 172.21.15.194:0/262399952' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e795bb0e-4791-4537-acda-1a321e19bc9c"}]: dispatch 2024-03-18T16:02:08.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:07 smithi039 ceph-mon[31721]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e795bb0e-4791-4537-acda-1a321e19bc9c"}]: dispatch 2024-03-18T16:02:08.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:07 smithi039 ceph-mon[31721]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e795bb0e-4791-4537-acda-1a321e19bc9c"}]': finished 2024-03-18T16:02:08.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:07 smithi039 ceph-mon[31721]: osdmap e30: 5 total, 4 up, 5 in 2024-03-18T16:02:08.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:07 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T16:02:08.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:07 smithi039 ceph-mon[31721]: from='client.? 172.21.15.194:0/2224413971' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T16:02:09.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:09 smithi039 ceph-mon[31721]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail; 104 KiB/s, 0 objects/s recovering 2024-03-18T16:02:09.727 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:09 smithi196 ceph-mon[36307]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail; 104 KiB/s, 0 objects/s recovering 2024-03-18T16:02:09.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:09 smithi194 ceph-mon[36200]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 125 MiB used, 357 GiB / 358 GiB avail; 104 KiB/s, 0 objects/s recovering 2024-03-18T16:02:11.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:11 smithi194 ceph-mon[36200]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-03-18T16:02:11.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:11 smithi039 ceph-mon[31721]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-03-18T16:02:11.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:11 smithi196 ceph-mon[36307]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-03-18T16:02:13.527 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:13 smithi194 ceph-mon[36200]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-18T16:02:13.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:13 smithi039 ceph-mon[31721]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-18T16:02:13.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:13 smithi196 ceph-mon[36307]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-18T16:02:15.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:15 smithi194 ceph-mon[36200]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-03-18T16:02:15.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:15 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-18T16:02:15.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:15 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:02:15.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:15 smithi039 ceph-mon[31721]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-03-18T16:02:15.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:15 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-18T16:02:15.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:15 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:02:15.727 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:15 smithi196 ceph-mon[36307]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-03-18T16:02:15.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:15 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-18T16:02:15.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:15 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:02:16.535 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:16 smithi194 ceph-mon[36200]: Deploying daemon osd.4 on smithi194 2024-03-18T16:02:16.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:16 smithi039 ceph-mon[31721]: Deploying daemon osd.4 on smithi194 2024-03-18T16:02:16.727 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:16 smithi196 ceph-mon[36307]: Deploying daemon osd.4 on smithi194 2024-03-18T16:02:17.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:17 smithi039 ceph-mon[31721]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-18T16:02:17.600 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:17 smithi194 ceph-mon[36200]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-18T16:02:17.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:17 smithi196 ceph-mon[36307]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-18T16:02:19.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:19 smithi194 ceph-mon[36200]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-03-18T16:02:19.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:19 smithi039 ceph-mon[31721]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-03-18T16:02:19.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:19 smithi196 ceph-mon[36307]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-03-18T16:02:21.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:21 smithi196 ceph-mon[36307]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-03-18T16:02:21.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:21 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:02:21.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:21 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:02:21.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:21 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:02:21.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:21 smithi194 ceph-mon[36200]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-03-18T16:02:21.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:21 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:02:21.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:21 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:02:21.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:21 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:02:21.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:21 smithi039 ceph-mon[31721]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-03-18T16:02:21.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:21 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:02:21.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:21 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:02:21.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:21 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:02:23.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:23 smithi196 ceph-mon[36307]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-03-18T16:02:23.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:23 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:02:23.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:23 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:02:23.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:23 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:02:23.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:23 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:02:23.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:23 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:02:23.489 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:23 smithi194 ceph-mon[36200]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-03-18T16:02:23.489 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:23 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:02:23.489 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:23 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:02:23.489 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:23 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:02:23.489 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:23 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:02:23.489 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:23 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:02:23.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:23 smithi039 ceph-mon[31721]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-03-18T16:02:23.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:23 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:02:23.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:23 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:02:23.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:23 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:02:23.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:23 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:02:23.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:23 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:02:23.877 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:02:23.875+0000 7ff47e7fc700 1 -- 172.21.15.194:0/1984239060 <== mgr.14152 v2:172.21.15.39:6800/2749214286 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7ff488062710 con 0x7ff4740a2910 2024-03-18T16:02:23.878 INFO:teuthology.orchestra.run.smithi194.stdout:Created osd(s) 4 on host 'smithi194' 2024-03-18T16:02:23.880 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:02:23.877+0000 7ff48f951700 1 -- 172.21.15.194:0/1984239060 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff4740a2910 msgr2=0x7ff4740a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:02:23.880 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:02:23.878+0000 7ff48f951700 1 --2- 172.21.15.194:0/1984239060 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff4740a2910 0x7ff4740a4dd0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff48400b580 tx=0x7ff484005fb0 comp rx=0 tx=0).stop 2024-03-18T16:02:23.880 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:02:23.878+0000 7ff48f951700 1 -- 172.21.15.194:0/1984239060 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff488103ea0 msgr2=0x7ff488074c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:02:23.880 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:02:23.878+0000 7ff48f951700 1 --2- 172.21.15.194:0/1984239060 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff488103ea0 0x7ff488074c00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff48000b970 tx=0x7ff48000bc80 comp rx=0 tx=0).stop 2024-03-18T16:02:23.880 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:02:23.878+0000 7ff48f951700 1 -- 172.21.15.194:0/1984239060 shutdown_connections 2024-03-18T16:02:23.880 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:02:23.878+0000 7ff48f951700 1 --2- 172.21.15.194:0/1984239060 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff4740a2910 0x7ff4740a4dd0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:23.881 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:02:23.878+0000 7ff48f951700 1 --2- 172.21.15.194:0/1984239060 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff488103ea0 0x7ff488074c00 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:23.881 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:02:23.878+0000 7ff48f951700 1 --2- 172.21.15.194:0/1984239060 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff488102300 0x7ff488076070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:23.881 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:02:23.878+0000 7ff48f951700 1 --2- 172.21.15.194:0/1984239060 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff488103500 0x7ff4880746c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:23.881 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:02:23.878+0000 7ff48f951700 1 -- 172.21.15.194:0/1984239060 >> 172.21.15.194:0/1984239060 conn(0x7ff4880fd8e0 msgr2=0x7ff488106940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:02:23.881 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:02:23.878+0000 7ff48f951700 1 -- 172.21.15.194:0/1984239060 shutdown_connections 2024-03-18T16:02:23.881 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:02:23.878+0000 7ff48f951700 1 -- 172.21.15.194:0/1984239060 wait complete. 2024-03-18T16:02:24.704 DEBUG:teuthology.orchestra.run.smithi194:osd.4> sudo journalctl -f -n 0 -u ceph-1892d8fc-e540-11ee-95c9-87774f69a715@osd.4.service 2024-03-18T16:02:24.706 INFO:tasks.cephadm:Deploying osd.5 on smithi196 with /dev/vg_nvme/lv_3... 2024-03-18T16:02:24.707 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-18T16:02:24.954 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T16:02:24.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:24 smithi194 ceph-mon[36200]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-03-18T16:02:24.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:24 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:02:24.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:24 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:02:24.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:24 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:02:24.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:24 smithi194 ceph-mon[36200]: from='osd.4 [v2:172.21.15.194:6808/800423088,v1:172.21.15.194:6809/800423088]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-18T16:02:24.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:24 smithi194 ceph-mon[36200]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-18T16:02:25.075 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:24 smithi196 ceph-mon[36307]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-03-18T16:02:25.075 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:24 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:02:25.076 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:24 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:02:25.076 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:24 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:02:25.076 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:24 smithi196 ceph-mon[36307]: from='osd.4 [v2:172.21.15.194:6808/800423088,v1:172.21.15.194:6809/800423088]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-18T16:02:25.076 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:24 smithi196 ceph-mon[36307]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-18T16:02:25.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:24 smithi039 ceph-mon[31721]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-03-18T16:02:25.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:24 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:02:25.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:24 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:02:25.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:24 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:02:25.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:24 smithi039 ceph-mon[31721]: from='osd.4 [v2:172.21.15.194:6808/800423088,v1:172.21.15.194:6809/800423088]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-18T16:02:25.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:24 smithi039 ceph-mon[31721]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-18T16:02:26.101 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:25 smithi194 ceph-mon[36200]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-18T16:02:26.101 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:25 smithi194 ceph-mon[36200]: from='osd.4 [v2:172.21.15.194:6808/800423088,v1:172.21.15.194:6809/800423088]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-03-18T16:02:26.101 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:25 smithi194 ceph-mon[36200]: osdmap e31: 5 total, 4 up, 5 in 2024-03-18T16:02:26.102 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:25 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T16:02:26.102 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:25 smithi194 ceph-mon[36200]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-03-18T16:02:26.115 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:25 smithi196 ceph-mon[36307]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-18T16:02:26.115 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:25 smithi196 ceph-mon[36307]: from='osd.4 [v2:172.21.15.194:6808/800423088,v1:172.21.15.194:6809/800423088]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-03-18T16:02:26.115 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:25 smithi196 ceph-mon[36307]: osdmap e31: 5 total, 4 up, 5 in 2024-03-18T16:02:26.115 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:25 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T16:02:26.115 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:25 smithi196 ceph-mon[36307]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-03-18T16:02:26.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:25 smithi039 ceph-mon[31721]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-18T16:02:26.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:25 smithi039 ceph-mon[31721]: from='osd.4 [v2:172.21.15.194:6808/800423088,v1:172.21.15.194:6809/800423088]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-03-18T16:02:26.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:25 smithi039 ceph-mon[31721]: osdmap e31: 5 total, 4 up, 5 in 2024-03-18T16:02:26.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:25 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T16:02:26.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:25 smithi039 ceph-mon[31721]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-03-18T16:02:26.740 INFO:journalctl@ceph.osd.4.smithi194.stdout:Mar 18 16:02:26 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-4[47784]: 2024-03-18T16:02:26.548+0000 7f5889f56700 -1 osd.4 0 waiting for initial osdmap 2024-03-18T16:02:26.741 INFO:journalctl@ceph.osd.4.smithi194.stdout:Mar 18 16:02:26 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-4[47784]: 2024-03-18T16:02:26.555+0000 7f5881d41700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T16:02:26.823 INFO:teuthology.orchestra.run.smithi196.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-18T16:02:26.823 INFO:teuthology.orchestra.run.smithi196.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-18T16:02:26.823 INFO:teuthology.orchestra.run.smithi196.stderr: stderr: 10+0 records in 2024-03-18T16:02:26.823 INFO:teuthology.orchestra.run.smithi196.stderr:10+0 records out 2024-03-18T16:02:26.823 INFO:teuthology.orchestra.run.smithi196.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113058 s, 927 MB/s 2024-03-18T16:02:26.823 INFO:teuthology.orchestra.run.smithi196.stderr:--> Zapping successful for: 2024-03-18T16:02:27.152 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:26 smithi196 ceph-mon[36307]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-03-18T16:02:27.152 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:26 smithi196 ceph-mon[36307]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-03-18T16:02:27.152 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:26 smithi196 ceph-mon[36307]: osdmap e32: 5 total, 4 up, 5 in 2024-03-18T16:02:27.152 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:26 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T16:02:27.152 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:26 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T16:02:27.152 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:26 smithi196 ceph-mon[36307]: from='osd.4 ' entity='osd.4' 2024-03-18T16:02:27.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:26 smithi194 ceph-mon[36200]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-03-18T16:02:27.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:26 smithi194 ceph-mon[36200]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-03-18T16:02:27.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:26 smithi194 ceph-mon[36200]: osdmap e32: 5 total, 4 up, 5 in 2024-03-18T16:02:27.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:26 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T16:02:27.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:26 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T16:02:27.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:26 smithi194 ceph-mon[36200]: from='osd.4 ' entity='osd.4' 2024-03-18T16:02:27.298 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi196:vg_nvme/lv_3 2024-03-18T16:02:27.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:26 smithi039 ceph-mon[31721]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-03-18T16:02:27.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:26 smithi039 ceph-mon[31721]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-03-18T16:02:27.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:26 smithi039 ceph-mon[31721]: osdmap e32: 5 total, 4 up, 5 in 2024-03-18T16:02:27.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:26 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T16:02:27.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:26 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T16:02:27.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:26 smithi039 ceph-mon[31721]: from='osd.4 ' entity='osd.4' 2024-03-18T16:02:27.535 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T16:02:27.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:27 smithi194 ceph-mon[36200]: purged_snaps scrub starts 2024-03-18T16:02:27.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:27 smithi194 ceph-mon[36200]: purged_snaps scrub ok 2024-03-18T16:02:27.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:27 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T16:02:27.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:27 smithi194 ceph-mon[36200]: osd.4 [v2:172.21.15.194:6808/800423088,v1:172.21.15.194:6809/800423088] boot 2024-03-18T16:02:27.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:27 smithi194 ceph-mon[36200]: osdmap e33: 5 total, 5 up, 5 in 2024-03-18T16:02:27.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:27 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T16:02:27.991 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:27 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:02:27.991 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:27 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:02:27.991 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:27 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:02:27.991 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:27 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:02:27.991 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:27 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:02:27.991 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:27 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:02:27.991 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:27 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:02:27.991 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:27 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:02:28.169 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:27 smithi039 ceph-mon[31721]: purged_snaps scrub starts 2024-03-18T16:02:28.170 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:27 smithi039 ceph-mon[31721]: purged_snaps scrub ok 2024-03-18T16:02:28.170 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:27 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T16:02:28.170 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:27 smithi039 ceph-mon[31721]: osd.4 [v2:172.21.15.194:6808/800423088,v1:172.21.15.194:6809/800423088] boot 2024-03-18T16:02:28.170 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:27 smithi039 ceph-mon[31721]: osdmap e33: 5 total, 5 up, 5 in 2024-03-18T16:02:28.170 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:27 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T16:02:28.170 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:27 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:02:28.170 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:27 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:02:28.170 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:27 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:02:28.170 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:27 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:02:28.170 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:27 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:02:28.170 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:27 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:02:28.171 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:27 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:02:28.171 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:27 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:02:28.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:27 smithi196 ceph-mon[36307]: purged_snaps scrub starts 2024-03-18T16:02:28.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:27 smithi196 ceph-mon[36307]: purged_snaps scrub ok 2024-03-18T16:02:28.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:27 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T16:02:28.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:27 smithi196 ceph-mon[36307]: osd.4 [v2:172.21.15.194:6808/800423088,v1:172.21.15.194:6809/800423088] boot 2024-03-18T16:02:28.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:27 smithi196 ceph-mon[36307]: osdmap e33: 5 total, 5 up, 5 in 2024-03-18T16:02:28.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:27 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-18T16:02:28.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:27 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:02:28.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:27 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:02:28.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:27 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:02:28.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:27 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:02:28.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:27 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:02:28.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:27 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:02:28.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:27 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:02:28.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:27 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:02:29.098 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:28 smithi196 ceph-mon[36307]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-03-18T16:02:29.098 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:28 smithi196 ceph-mon[36307]: Detected new or changed devices on smithi194 2024-03-18T16:02:29.099 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:28 smithi196 ceph-mon[36307]: Adjusting osd_memory_target on smithi194 to 8540M 2024-03-18T16:02:29.103 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:02:29.100+0000 7fa2345cb700 1 -- 172.21.15.196:0/2469898526 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa22c103fe0 msgr2=0x7fa22c10a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:02:29.103 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:02:29.100+0000 7fa2345cb700 1 --2- 172.21.15.196:0/2469898526 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa22c103fe0 0x7fa22c10a890 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa22400b0a0 tx=0x7fa22400b3b0 comp rx=0 tx=0).stop 2024-03-18T16:02:29.104 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:02:29.100+0000 7fa2345cb700 1 -- 172.21.15.196:0/2469898526 shutdown_connections 2024-03-18T16:02:29.104 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:02:29.100+0000 7fa2345cb700 1 --2- 172.21.15.196:0/2469898526 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa22c103fe0 0x7fa22c10a890 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:29.104 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:02:29.100+0000 7fa2345cb700 1 --2- 172.21.15.196:0/2469898526 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa22c1005c0 0x7fa22c1009e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:29.104 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:02:29.100+0000 7fa2345cb700 1 --2- 172.21.15.196:0/2469898526 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa22c103640 0x7fa22c103aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:29.104 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:02:29.100+0000 7fa2345cb700 1 -- 172.21.15.196:0/2469898526 >> 172.21.15.196:0/2469898526 conn(0x7fa22c0fbf40 msgr2=0x7fa22c0fe360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:02:29.104 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:02:29.100+0000 7fa2345cb700 1 -- 172.21.15.196:0/2469898526 shutdown_connections 2024-03-18T16:02:29.104 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:02:29.100+0000 7fa2345cb700 1 -- 172.21.15.196:0/2469898526 wait complete. 2024-03-18T16:02:29.104 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:02:29.101+0000 7fa2345cb700 1 Processor -- start 2024-03-18T16:02:29.104 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:02:29.101+0000 7fa2345cb700 1 -- start start 2024-03-18T16:02:29.104 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:02:29.101+0000 7fa2345cb700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa22c1005c0 0x7fa22c102fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:29.104 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:02:29.101+0000 7fa2345cb700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa22c103640 0x7fa22c1015f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:29.104 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:02:29.101+0000 7fa2345cb700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa22c103fe0 0x7fa22c101b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:29.104 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:02:29.101+0000 7fa2345cb700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fa22c10df50 con 0x7fa22c103fe0 2024-03-18T16:02:29.105 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:02:29.101+0000 7fa2345cb700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fa22c10ddd0 con 0x7fa22c1005c0 2024-03-18T16:02:29.105 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:02:29.101+0000 7fa2345cb700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fa22c10e0d0 con 0x7fa22c103640 2024-03-18T16:02:29.105 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:02:29.101+0000 7fa231b66700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa22c103640 0x7fa22c1015f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:29.105 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:02:29.101+0000 7fa231b66700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa22c103640 0x7fa22c1015f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.196:60118/0 (socket says 172.21.15.196:60118) 2024-03-18T16:02:29.105 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:02:29.101+0000 7fa231b66700 1 -- 172.21.15.196:0/2658650035 learned_addr learned my addr 172.21.15.196:0/2658650035 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-18T16:02:29.105 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:02:29.101+0000 7fa232367700 1 --2- 172.21.15.196:0/2658650035 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa22c1005c0 0x7fa22c102fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:29.105 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:02:29.101+0000 7fa231b66700 1 -- 172.21.15.196:0/2658650035 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa22c1005c0 msgr2=0x7fa22c102fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:02:29.105 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:02:29.101+0000 7fa231b66700 1 --2- 172.21.15.196:0/2658650035 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa22c1005c0 0x7fa22c102fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:29.105 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:02:29.101+0000 7fa231b66700 1 -- 172.21.15.196:0/2658650035 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa22c103fe0 msgr2=0x7fa22c101b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:02:29.105 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:02:29.101+0000 7fa231b66700 1 --2- 172.21.15.196:0/2658650035 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa22c103fe0 0x7fa22c101b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:29.105 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:02:29.101+0000 7fa231b66700 1 -- 172.21.15.196:0/2658650035 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa22c1023e0 con 0x7fa22c103640 2024-03-18T16:02:29.105 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:02:29.101+0000 7fa232367700 1 --2- 172.21.15.196:0/2658650035 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa22c1005c0 0x7fa22c102fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:02:29.105 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:02:29.101+0000 7fa231b66700 1 --2- 172.21.15.196:0/2658650035 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa22c103640 0x7fa22c1015f0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa21c00b530 tx=0x7fa21c00b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:02:29.105 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:02:29.102+0000 7fa2237fe700 1 -- 172.21.15.196:0/2658650035 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa21c015020 con 0x7fa22c103640 2024-03-18T16:02:29.106 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:02:29.102+0000 7fa2345cb700 1 -- 172.21.15.196:0/2658650035 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa22c1a2660 con 0x7fa22c103640 2024-03-18T16:02:29.106 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:02:29.102+0000 7fa2345cb700 1 -- 172.21.15.196:0/2658650035 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa22c1a2ba0 con 0x7fa22c103640 2024-03-18T16:02:29.106 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:02:29.102+0000 7fa2237fe700 1 -- 172.21.15.196:0/2658650035 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa21c011d10 con 0x7fa22c103640 2024-03-18T16:02:29.106 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:02:29.102+0000 7fa2237fe700 1 -- 172.21.15.196:0/2658650035 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa21c010e50 con 0x7fa22c103640 2024-03-18T16:02:29.107 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:02:29.103+0000 7fa2345cb700 1 -- 172.21.15.196:0/2658650035 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa22c068f90 con 0x7fa22c103640 2024-03-18T16:02:29.107 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:02:29.104+0000 7fa2237fe700 1 -- 172.21.15.196:0/2658650035 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fa21c018020 con 0x7fa22c103640 2024-03-18T16:02:29.107 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:02:29.104+0000 7fa2237fe700 1 --2- 172.21.15.196:0/2658650035 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa2180a2ab0 0x7fa2180a4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:29.107 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:02:29.105+0000 7fa2237fe700 1 -- 172.21.15.196:0/2658650035 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 4101+0+0 (secure 0 0 0) 0x7fa21c0cd7c0 con 0x7fa22c103640 2024-03-18T16:02:29.107 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:02:29.105+0000 7fa232367700 1 --2- 172.21.15.196:0/2658650035 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa2180a2ab0 0x7fa2180a4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:29.107 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:02:29.106+0000 7fa232367700 1 --2- 172.21.15.196:0/2658650035 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa2180a2ab0 0x7fa2180a4f70 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa228005950 tx=0x7fa228005f50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:02:29.110 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:02:29.109+0000 7fa2237fe700 1 -- 172.21.15.196:0/2658650035 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa21c01f020 con 0x7fa22c103640 2024-03-18T16:02:29.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:28 smithi194 ceph-mon[36200]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-03-18T16:02:29.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:28 smithi194 ceph-mon[36200]: Detected new or changed devices on smithi194 2024-03-18T16:02:29.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:28 smithi194 ceph-mon[36200]: Adjusting osd_memory_target on smithi194 to 8540M 2024-03-18T16:02:29.313 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:02:29.311+0000 7fa2345cb700 1 -- 172.21.15.196:0/2658650035 --> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fa22c062710 con 0x7fa2180a2ab0 2024-03-18T16:02:29.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:28 smithi039 ceph-mon[31721]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 124 MiB used, 357 GiB / 358 GiB avail 2024-03-18T16:02:29.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:28 smithi039 ceph-mon[31721]: Detected new or changed devices on smithi194 2024-03-18T16:02:29.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:28 smithi039 ceph-mon[31721]: Adjusting osd_memory_target on smithi194 to 8540M 2024-03-18T16:02:30.168 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:29 smithi196 ceph-mon[36307]: osdmap e34: 5 total, 5 up, 5 in 2024-03-18T16:02:30.168 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:29 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T16:02:30.168 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:29 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T16:02:30.168 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:29 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:02:30.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:29 smithi194 ceph-mon[36200]: osdmap e34: 5 total, 5 up, 5 in 2024-03-18T16:02:30.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:29 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T16:02:30.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:29 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T16:02:30.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:29 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:02:30.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:29 smithi039 ceph-mon[31721]: osdmap e34: 5 total, 5 up, 5 in 2024-03-18T16:02:30.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:29 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T16:02:30.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:29 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T16:02:30.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:29 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:02:31.138 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:30 smithi196 ceph-mon[36307]: from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:02:31.138 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:30 smithi196 ceph-mon[36307]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-18T16:02:31.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:30 smithi194 ceph-mon[36200]: from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:02:31.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:30 smithi194 ceph-mon[36200]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-18T16:02:31.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:30 smithi039 ceph-mon[31721]: from='client.24247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:02:31.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:30 smithi039 ceph-mon[31721]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-18T16:02:33.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:32 smithi196 ceph-mon[36307]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-18T16:02:33.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:32 smithi196 ceph-mon[36307]: from='client.? 172.21.15.196:0/1360428953' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "026e175e-6acb-4b4c-b235-8ccee8ae1149"}]: dispatch 2024-03-18T16:02:33.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:32 smithi196 ceph-mon[36307]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "026e175e-6acb-4b4c-b235-8ccee8ae1149"}]: dispatch 2024-03-18T16:02:33.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:32 smithi196 ceph-mon[36307]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "026e175e-6acb-4b4c-b235-8ccee8ae1149"}]': finished 2024-03-18T16:02:33.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:32 smithi196 ceph-mon[36307]: osdmap e35: 6 total, 5 up, 6 in 2024-03-18T16:02:33.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:32 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T16:02:33.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:32 smithi196 ceph-mon[36307]: from='client.? 172.21.15.196:0/3103312845' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T16:02:33.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:32 smithi194 ceph-mon[36200]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-18T16:02:33.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:32 smithi194 ceph-mon[36200]: from='client.? 172.21.15.196:0/1360428953' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "026e175e-6acb-4b4c-b235-8ccee8ae1149"}]: dispatch 2024-03-18T16:02:33.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:32 smithi194 ceph-mon[36200]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "026e175e-6acb-4b4c-b235-8ccee8ae1149"}]: dispatch 2024-03-18T16:02:33.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:32 smithi194 ceph-mon[36200]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "026e175e-6acb-4b4c-b235-8ccee8ae1149"}]': finished 2024-03-18T16:02:33.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:32 smithi194 ceph-mon[36200]: osdmap e35: 6 total, 5 up, 6 in 2024-03-18T16:02:33.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:32 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T16:02:33.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:32 smithi194 ceph-mon[36200]: from='client.? 172.21.15.196:0/3103312845' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T16:02:33.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:32 smithi039 ceph-mon[31721]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-18T16:02:33.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:32 smithi039 ceph-mon[31721]: from='client.? 172.21.15.196:0/1360428953' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "026e175e-6acb-4b4c-b235-8ccee8ae1149"}]: dispatch 2024-03-18T16:02:33.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:32 smithi039 ceph-mon[31721]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "026e175e-6acb-4b4c-b235-8ccee8ae1149"}]: dispatch 2024-03-18T16:02:33.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:32 smithi039 ceph-mon[31721]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "026e175e-6acb-4b4c-b235-8ccee8ae1149"}]': finished 2024-03-18T16:02:33.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:32 smithi039 ceph-mon[31721]: osdmap e35: 6 total, 5 up, 6 in 2024-03-18T16:02:33.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:32 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T16:02:33.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:32 smithi039 ceph-mon[31721]: from='client.? 172.21.15.196:0/3103312845' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T16:02:35.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:34 smithi196 ceph-mon[36307]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-18T16:02:35.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:34 smithi194 ceph-mon[36200]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-18T16:02:35.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:34 smithi039 ceph-mon[31721]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-18T16:02:37.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:36 smithi196 ceph-mon[36307]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-18T16:02:37.235 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:36 smithi194 ceph-mon[36200]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-18T16:02:37.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:36 smithi039 ceph-mon[31721]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-18T16:02:39.175 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:38 smithi196 ceph-mon[36307]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-18T16:02:39.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:38 smithi194 ceph-mon[36200]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-18T16:02:39.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:38 smithi039 ceph-mon[31721]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-18T16:02:40.189 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:39 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-18T16:02:40.189 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:39 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:02:40.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:39 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-18T16:02:40.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:39 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:02:40.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:39 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-18T16:02:40.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:39 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:02:41.227 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:40 smithi196 ceph-mon[36307]: Deploying daemon osd.5 on smithi196 2024-03-18T16:02:41.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:40 smithi196 ceph-mon[36307]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-18T16:02:41.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:40 smithi194 ceph-mon[36200]: Deploying daemon osd.5 on smithi196 2024-03-18T16:02:41.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:40 smithi194 ceph-mon[36200]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-18T16:02:41.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:40 smithi039 ceph-mon[31721]: Deploying daemon osd.5 on smithi196 2024-03-18T16:02:41.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:40 smithi039 ceph-mon[31721]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-18T16:02:43.184 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:42 smithi196 ceph-mon[36307]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-18T16:02:43.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:42 smithi194 ceph-mon[36200]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-18T16:02:43.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:42 smithi039 ceph-mon[31721]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-18T16:02:45.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:44 smithi196 ceph-mon[36307]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-18T16:02:45.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:44 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:02:45.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:44 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:02:45.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:44 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:02:45.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:44 smithi194 ceph-mon[36200]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-18T16:02:45.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:44 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:02:45.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:44 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:02:45.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:44 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:02:45.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:44 smithi039 ceph-mon[31721]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-18T16:02:45.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:44 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:02:45.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:44 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:02:45.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:44 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:02:47.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:46 smithi196 ceph-mon[36307]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-18T16:02:47.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:46 smithi194 ceph-mon[36200]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-18T16:02:47.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:46 smithi039 ceph-mon[31721]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-18T16:02:48.802 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:02:48.800+0000 7fa2237fe700 1 -- 172.21.15.196:0/2658650035 <== mgr.14152 v2:172.21.15.39:6800/2749214286 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fa22c062710 con 0x7fa2180a2ab0 2024-03-18T16:02:48.803 INFO:teuthology.orchestra.run.smithi196.stdout:Created osd(s) 5 on host 'smithi196' 2024-03-18T16:02:48.804 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:02:48.803+0000 7fa2345cb700 1 -- 172.21.15.196:0/2658650035 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa2180a2ab0 msgr2=0x7fa2180a4f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:02:48.805 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:02:48.803+0000 7fa2345cb700 1 --2- 172.21.15.196:0/2658650035 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa2180a2ab0 0x7fa2180a4f70 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa228005950 tx=0x7fa228005f50 comp rx=0 tx=0).stop 2024-03-18T16:02:48.805 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:02:48.803+0000 7fa2345cb700 1 -- 172.21.15.196:0/2658650035 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa22c103640 msgr2=0x7fa22c1015f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:02:48.805 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:02:48.803+0000 7fa2345cb700 1 --2- 172.21.15.196:0/2658650035 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa22c103640 0x7fa22c1015f0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa21c00b530 tx=0x7fa21c00b8f0 comp rx=0 tx=0).stop 2024-03-18T16:02:48.805 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:02:48.803+0000 7fa2345cb700 1 -- 172.21.15.196:0/2658650035 shutdown_connections 2024-03-18T16:02:48.805 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:02:48.803+0000 7fa2345cb700 1 --2- 172.21.15.196:0/2658650035 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa2180a2ab0 0x7fa2180a4f70 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:48.805 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:02:48.803+0000 7fa2345cb700 1 --2- 172.21.15.196:0/2658650035 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa22c103fe0 0x7fa22c101b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:48.805 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:02:48.803+0000 7fa2345cb700 1 --2- 172.21.15.196:0/2658650035 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa22c1005c0 0x7fa22c102fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:48.805 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:02:48.803+0000 7fa2345cb700 1 --2- 172.21.15.196:0/2658650035 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa22c103640 0x7fa22c1015f0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:48.805 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:02:48.803+0000 7fa2345cb700 1 -- 172.21.15.196:0/2658650035 >> 172.21.15.196:0/2658650035 conn(0x7fa22c0fbf40 msgr2=0x7fa22c1068e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:02:48.805 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:02:48.803+0000 7fa2345cb700 1 -- 172.21.15.196:0/2658650035 shutdown_connections 2024-03-18T16:02:48.805 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:02:48.803+0000 7fa2345cb700 1 -- 172.21.15.196:0/2658650035 wait complete. 2024-03-18T16:02:48.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:48 smithi196 ceph-mon[36307]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-18T16:02:48.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:48 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:02:48.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:48 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:02:48.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:48 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:02:48.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:48 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:02:48.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:48 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:02:48.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:48 smithi196 ceph-mon[36307]: from='osd.5 [v2:172.21.15.196:6808/2746224525,v1:172.21.15.196:6809/2746224525]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-18T16:02:48.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:48 smithi196 ceph-mon[36307]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-18T16:02:48.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:48 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:02:48.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:48 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:02:48.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:48 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:02:49.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:48 smithi194 ceph-mon[36200]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-18T16:02:49.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:48 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:02:49.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:48 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:02:49.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:48 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:02:49.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:48 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:02:49.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:48 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:02:49.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:48 smithi194 ceph-mon[36200]: from='osd.5 [v2:172.21.15.196:6808/2746224525,v1:172.21.15.196:6809/2746224525]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-18T16:02:49.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:48 smithi194 ceph-mon[36200]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-18T16:02:49.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:48 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:02:49.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:48 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:02:49.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:48 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:02:49.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:48 smithi039 ceph-mon[31721]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-18T16:02:49.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:48 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:02:49.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:48 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:02:49.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:48 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:02:49.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:48 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:02:49.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:48 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:02:49.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:48 smithi039 ceph-mon[31721]: from='osd.5 [v2:172.21.15.196:6808/2746224525,v1:172.21.15.196:6809/2746224525]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-18T16:02:49.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:48 smithi039 ceph-mon[31721]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-18T16:02:49.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:48 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:02:49.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:48 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:02:49.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:48 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:02:49.342 DEBUG:teuthology.orchestra.run.smithi196:osd.5> sudo journalctl -f -n 0 -u ceph-1892d8fc-e540-11ee-95c9-87774f69a715@osd.5.service 2024-03-18T16:02:49.344 INFO:tasks.cephadm:Deploying osd.6 on smithi039 with /dev/vg_nvme/lv_2... 2024-03-18T16:02:49.344 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-18T16:02:49.811 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T16:02:50.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:49 smithi194 ceph-mon[36200]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-18T16:02:50.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:49 smithi194 ceph-mon[36200]: from='osd.5 [v2:172.21.15.196:6808/2746224525,v1:172.21.15.196:6809/2746224525]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-03-18T16:02:50.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:49 smithi194 ceph-mon[36200]: osdmap e36: 6 total, 5 up, 6 in 2024-03-18T16:02:50.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:49 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T16:02:50.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:49 smithi194 ceph-mon[36200]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-03-18T16:02:50.295 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:49 smithi039 ceph-mon[31721]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-18T16:02:50.295 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:49 smithi039 ceph-mon[31721]: from='osd.5 [v2:172.21.15.196:6808/2746224525,v1:172.21.15.196:6809/2746224525]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-03-18T16:02:50.295 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:49 smithi039 ceph-mon[31721]: osdmap e36: 6 total, 5 up, 6 in 2024-03-18T16:02:50.295 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:49 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T16:02:50.295 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:49 smithi039 ceph-mon[31721]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-03-18T16:02:50.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:50 smithi196 ceph-mon[36307]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-18T16:02:50.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:50 smithi196 ceph-mon[36307]: from='osd.5 [v2:172.21.15.196:6808/2746224525,v1:172.21.15.196:6809/2746224525]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-03-18T16:02:50.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:50 smithi196 ceph-mon[36307]: osdmap e36: 6 total, 5 up, 6 in 2024-03-18T16:02:50.434 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:50 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T16:02:50.435 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:50 smithi196 ceph-mon[36307]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-03-18T16:02:50.728 INFO:journalctl@ceph.osd.5.smithi196.stdout:Mar 18 16:02:50 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-5[48008]: 2024-03-18T16:02:50.652+0000 7f95cf47d700 -1 osd.5 0 waiting for initial osdmap 2024-03-18T16:02:50.728 INFO:journalctl@ceph.osd.5.smithi196.stdout:Mar 18 16:02:50 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-5[48008]: 2024-03-18T16:02:50.660+0000 7f95c7268700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T16:02:51.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:50 smithi196 ceph-mon[36307]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-18T16:02:51.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:50 smithi196 ceph-mon[36307]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-03-18T16:02:51.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:50 smithi196 ceph-mon[36307]: osdmap e37: 6 total, 5 up, 6 in 2024-03-18T16:02:51.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:50 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T16:02:51.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:50 smithi196 ceph-mon[36307]: from='osd.5 ' entity='osd.5' 2024-03-18T16:02:51.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:50 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T16:02:51.290 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:50 smithi039 ceph-mon[31721]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-18T16:02:51.291 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:50 smithi039 ceph-mon[31721]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-03-18T16:02:51.291 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:50 smithi039 ceph-mon[31721]: osdmap e37: 6 total, 5 up, 6 in 2024-03-18T16:02:51.291 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:50 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T16:02:51.291 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:50 smithi039 ceph-mon[31721]: from='osd.5 ' entity='osd.5' 2024-03-18T16:02:51.291 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:50 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T16:02:51.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:50 smithi194 ceph-mon[36200]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-18T16:02:51.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:50 smithi194 ceph-mon[36200]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-03-18T16:02:51.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:50 smithi194 ceph-mon[36200]: osdmap e37: 6 total, 5 up, 6 in 2024-03-18T16:02:51.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:50 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T16:02:51.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:50 smithi194 ceph-mon[36200]: from='osd.5 ' entity='osd.5' 2024-03-18T16:02:51.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:50 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T16:02:52.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:52 smithi039 ceph-mon[31721]: purged_snaps scrub starts 2024-03-18T16:02:52.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:52 smithi039 ceph-mon[31721]: purged_snaps scrub ok 2024-03-18T16:02:52.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:52 smithi039 ceph-mon[31721]: osd.5 [v2:172.21.15.196:6808/2746224525,v1:172.21.15.196:6809/2746224525] boot 2024-03-18T16:02:52.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:52 smithi039 ceph-mon[31721]: osdmap e38: 6 total, 6 up, 6 in 2024-03-18T16:02:52.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:52 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T16:02:52.450 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:52 smithi196 ceph-mon[36307]: purged_snaps scrub starts 2024-03-18T16:02:52.451 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:52 smithi196 ceph-mon[36307]: purged_snaps scrub ok 2024-03-18T16:02:52.451 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:52 smithi196 ceph-mon[36307]: osd.5 [v2:172.21.15.196:6808/2746224525,v1:172.21.15.196:6809/2746224525] boot 2024-03-18T16:02:52.451 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:52 smithi196 ceph-mon[36307]: osdmap e38: 6 total, 6 up, 6 in 2024-03-18T16:02:52.451 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:52 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T16:02:52.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:52 smithi194 ceph-mon[36200]: purged_snaps scrub starts 2024-03-18T16:02:52.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:52 smithi194 ceph-mon[36200]: purged_snaps scrub ok 2024-03-18T16:02:52.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:52 smithi194 ceph-mon[36200]: osd.5 [v2:172.21.15.196:6808/2746224525,v1:172.21.15.196:6809/2746224525] boot 2024-03-18T16:02:52.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:52 smithi194 ceph-mon[36200]: osdmap e38: 6 total, 6 up, 6 in 2024-03-18T16:02:52.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:52 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-18T16:02:52.825 INFO:teuthology.orchestra.run.smithi039.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-18T16:02:52.825 INFO:teuthology.orchestra.run.smithi039.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-18T16:02:52.825 INFO:teuthology.orchestra.run.smithi039.stderr: stderr: 10+0 records in 2024-03-18T16:02:52.825 INFO:teuthology.orchestra.run.smithi039.stderr:10+0 records out 2024-03-18T16:02:52.825 INFO:teuthology.orchestra.run.smithi039.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0338338 s, 310 MB/s 2024-03-18T16:02:52.825 INFO:teuthology.orchestra.run.smithi039.stderr:--> Zapping successful for: 2024-03-18T16:02:53.203 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:53 smithi039 ceph-mon[31721]: pgmap v151: 1 pgs: 1 peering; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-18T16:02:53.204 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:53 smithi039 ceph-mon[31721]: osdmap e39: 6 total, 6 up, 6 in 2024-03-18T16:02:53.204 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:53 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:02:53.204 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:53 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:02:53.204 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:53 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:02:53.204 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:53 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:02:53.204 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:53 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:02:53.204 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:53 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:02:53.204 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:53 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:02:53.204 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:53 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:02:53.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:53 smithi196 ceph-mon[36307]: pgmap v151: 1 pgs: 1 peering; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-18T16:02:53.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:53 smithi196 ceph-mon[36307]: osdmap e39: 6 total, 6 up, 6 in 2024-03-18T16:02:53.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:53 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:02:53.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:53 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:02:53.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:53 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:02:53.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:53 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:02:53.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:53 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:02:53.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:53 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:02:53.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:53 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:02:53.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:53 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:02:53.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:53 smithi194 ceph-mon[36200]: pgmap v151: 1 pgs: 1 peering; 577 KiB data, 155 MiB used, 447 GiB / 447 GiB avail 2024-03-18T16:02:53.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:53 smithi194 ceph-mon[36200]: osdmap e39: 6 total, 6 up, 6 in 2024-03-18T16:02:53.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:53 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:02:53.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:53 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:02:53.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:53 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:02:53.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:53 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:02:53.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:53 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:02:53.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:53 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:02:53.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:53 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:02:53.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:53 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:02:53.620 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi039:vg_nvme/lv_2 2024-03-18T16:02:54.063 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T16:02:54.302 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:54 smithi039 ceph-mon[31721]: Detected new or changed devices on smithi196 2024-03-18T16:02:54.302 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:54 smithi039 ceph-mon[31721]: Adjusting osd_memory_target on smithi196 to 8540M 2024-03-18T16:02:54.302 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:54 smithi039 ceph-mon[31721]: osdmap e40: 6 total, 6 up, 6 in 2024-03-18T16:02:54.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:54 smithi196 ceph-mon[36307]: Detected new or changed devices on smithi196 2024-03-18T16:02:54.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:54 smithi196 ceph-mon[36307]: Adjusting osd_memory_target on smithi196 to 8540M 2024-03-18T16:02:54.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:54 smithi196 ceph-mon[36307]: osdmap e40: 6 total, 6 up, 6 in 2024-03-18T16:02:54.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:54 smithi194 ceph-mon[36200]: Detected new or changed devices on smithi196 2024-03-18T16:02:54.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:54 smithi194 ceph-mon[36200]: Adjusting osd_memory_target on smithi196 to 8540M 2024-03-18T16:02:54.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:54 smithi194 ceph-mon[36200]: osdmap e40: 6 total, 6 up, 6 in 2024-03-18T16:02:55.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:55 smithi039 ceph-mon[31721]: pgmap v154: 1 pgs: 1 peering; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-03-18T16:02:55.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:55 smithi039 ceph-mon[31721]: osdmap e41: 6 total, 6 up, 6 in 2024-03-18T16:02:55.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:55 smithi196 ceph-mon[36307]: pgmap v154: 1 pgs: 1 peering; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-03-18T16:02:55.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:55 smithi196 ceph-mon[36307]: osdmap e41: 6 total, 6 up, 6 in 2024-03-18T16:02:55.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:55 smithi194 ceph-mon[36200]: pgmap v154: 1 pgs: 1 peering; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-03-18T16:02:55.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:55 smithi194 ceph-mon[36200]: osdmap e41: 6 total, 6 up, 6 in 2024-03-18T16:02:56.325 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:56 smithi039 ceph-mon[31721]: osdmap e42: 6 total, 6 up, 6 in 2024-03-18T16:02:56.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:02:56.359+0000 7f7cfde00700 1 -- 172.21.15.39:0/3098686386 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7cf8105650 msgr2=0x7f7cf8105ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:02:56.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:02:56.359+0000 7f7cfde00700 1 --2- 172.21.15.39:0/3098686386 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7cf8105650 0x7f7cf8105ab0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f7ce4009920 tx=0x7f7ce4009c30 comp rx=0 tx=0).stop 2024-03-18T16:02:56.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:02:56.359+0000 7f7cfde00700 1 -- 172.21.15.39:0/3098686386 shutdown_connections 2024-03-18T16:02:56.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:02:56.359+0000 7f7cfde00700 1 --2- 172.21.15.39:0/3098686386 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7cf8105ff0 0x7f7cf810c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:56.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:02:56.359+0000 7f7cfde00700 1 --2- 172.21.15.39:0/3098686386 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7cf8104450 0x7f7cf8104870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:56.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:02:56.359+0000 7f7cfde00700 1 --2- 172.21.15.39:0/3098686386 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7cf8105650 0x7f7cf8105ab0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:56.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:02:56.359+0000 7f7cfde00700 1 -- 172.21.15.39:0/3098686386 >> 172.21.15.39:0/3098686386 conn(0x7f7cf80ff9f0 msgr2=0x7f7cf8101e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:02:56.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:02:56.360+0000 7f7cfde00700 1 -- 172.21.15.39:0/3098686386 shutdown_connections 2024-03-18T16:02:56.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:02:56.360+0000 7f7cfde00700 1 -- 172.21.15.39:0/3098686386 wait complete. 2024-03-18T16:02:56.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:02:56.361+0000 7f7cfde00700 1 Processor -- start 2024-03-18T16:02:56.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:02:56.361+0000 7f7cfde00700 1 -- start start 2024-03-18T16:02:56.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:02:56.362+0000 7f7cfde00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7cf8104450 0x7f7cf819a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:56.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:02:56.362+0000 7f7cfde00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7cf8105650 0x7f7cf819ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:56.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:02:56.362+0000 7f7cfde00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7cf8105ff0 0x7f7cf81a1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:56.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:02:56.362+0000 7f7cfde00700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f7cf8110090 con 0x7f7cf8104450 2024-03-18T16:02:56.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:02:56.362+0000 7f7cfde00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f7cf810ff10 con 0x7f7cf8105650 2024-03-18T16:02:56.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:02:56.362+0000 7f7cfde00700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f7cf8110210 con 0x7f7cf8105ff0 2024-03-18T16:02:56.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:02:56.362+0000 7f7cfca00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7cf8104450 0x7f7cf819a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:56.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:02:56.362+0000 7f7cfca00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7cf8104450 0x7f7cf819a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:53940/0 (socket says 172.21.15.39:53940) 2024-03-18T16:02:56.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:02:56.362+0000 7f7cfca00700 1 -- 172.21.15.39:0/3738368027 learned_addr learned my addr 172.21.15.39:0/3738368027 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:02:56.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:02:56.362+0000 7f7cfd400700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7cf8105ff0 0x7f7cf81a1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:56.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:02:56.362+0000 7f7cf7e00700 1 --2- 172.21.15.39:0/3738368027 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7cf8105650 0x7f7cf819ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:56.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:02:56.363+0000 7f7cfd400700 1 -- 172.21.15.39:0/3738368027 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7cf8105650 msgr2=0x7f7cf819ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:02:56.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:02:56.363+0000 7f7cfd400700 1 --2- 172.21.15.39:0/3738368027 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7cf8105650 0x7f7cf819ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:56.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:02:56.363+0000 7f7cfd400700 1 -- 172.21.15.39:0/3738368027 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7cf8104450 msgr2=0x7f7cf819a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:02:56.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:02:56.363+0000 7f7cfd400700 1 --2- 172.21.15.39:0/3738368027 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7cf8104450 0x7f7cf819a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:02:56.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:02:56.363+0000 7f7cfd400700 1 -- 172.21.15.39:0/3738368027 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cf81a2340 con 0x7f7cf8105ff0 2024-03-18T16:02:56.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:02:56.363+0000 7f7cfca00700 1 --2- 172.21.15.39:0/3738368027 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7cf8104450 0x7f7cf819a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:02:56.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:02:56.363+0000 7f7cfd400700 1 --2- 172.21.15.39:0/3738368027 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7cf8105ff0 0x7f7cf81a1ca0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7cf0009f60 tx=0x7f7cf0009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:02:56.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:02:56.364+0000 7f7cf5600700 1 -- 172.21.15.39:0/3738368027 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cf00092e0 con 0x7f7cf8105ff0 2024-03-18T16:02:56.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:02:56.364+0000 7f7cf5600700 1 -- 172.21.15.39:0/3738368027 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7cf0004bb0 con 0x7f7cf8105ff0 2024-03-18T16:02:56.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:02:56.364+0000 7f7cfde00700 1 -- 172.21.15.39:0/3738368027 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cf81a25d0 con 0x7f7cf8105ff0 2024-03-18T16:02:56.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:02:56.364+0000 7f7cf5600700 1 -- 172.21.15.39:0/3738368027 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cf0004280 con 0x7f7cf8105ff0 2024-03-18T16:02:56.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:02:56.364+0000 7f7cfde00700 1 -- 172.21.15.39:0/3738368027 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cf81a2b10 con 0x7f7cf8105ff0 2024-03-18T16:02:56.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:02:56.365+0000 7f7cfde00700 1 -- 172.21.15.39:0/3738368027 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7cf8068f90 con 0x7f7cf8105ff0 2024-03-18T16:02:56.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:02:56.367+0000 7f7cf5600700 1 -- 172.21.15.39:0/3738368027 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f7cf001b030 con 0x7f7cf8105ff0 2024-03-18T16:02:56.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:02:56.368+0000 7f7cf5600700 1 --2- 172.21.15.39:0/3738368027 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f7ce00a2910 0x7f7ce00a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:02:56.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:02:56.368+0000 7f7cf5600700 1 -- 172.21.15.39:0/3738368027 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 4533+0+0 (secure 0 0 0) 0x7f7cf00cd7d0 con 0x7f7cf8105ff0 2024-03-18T16:02:56.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:02:56.368+0000 7f7cfca00700 1 --2- 172.21.15.39:0/3738368027 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f7ce00a2910 0x7f7ce00a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:02:56.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:02:56.369+0000 7f7cfca00700 1 --2- 172.21.15.39:0/3738368027 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f7ce00a2910 0x7f7ce00a4dd0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7cec009fd0 tx=0x7f7cec009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:02:56.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:02:56.370+0000 7f7cf5600700 1 -- 172.21.15.39:0/3738368027 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7cf0022020 con 0x7f7cf8105ff0 2024-03-18T16:02:56.477 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:56 smithi196 ceph-mon[36307]: osdmap e42: 6 total, 6 up, 6 in 2024-03-18T16:02:56.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:56 smithi194 ceph-mon[36200]: osdmap e42: 6 total, 6 up, 6 in 2024-03-18T16:02:56.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:02:56.611+0000 7f7cfde00700 1 -- 172.21.15.39:0/3738368027 --> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi039:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f7cf8062710 con 0x7f7ce00a2910 2024-03-18T16:02:57.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:57 smithi039 ceph-mon[31721]: pgmap v157: 1 pgs: 1 peering; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-03-18T16:02:57.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:57 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T16:02:57.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:57 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T16:02:57.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:57 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:02:57.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:57 smithi196 ceph-mon[36307]: pgmap v157: 1 pgs: 1 peering; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-03-18T16:02:57.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:57 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T16:02:57.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:57 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T16:02:57.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:57 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:02:57.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:57 smithi194 ceph-mon[36200]: pgmap v157: 1 pgs: 1 peering; 577 KiB data, 186 MiB used, 536 GiB / 536 GiB avail 2024-03-18T16:02:57.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:57 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T16:02:57.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:57 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T16:02:57.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:57 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:02:58.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:58 smithi039 ceph-mon[31721]: from='client.24274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi039:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:02:58.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:58 smithi196 ceph-mon[36307]: from='client.24274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi039:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:02:58.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:58 smithi194 ceph-mon[36200]: from='client.24274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi039:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:02:59.173 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:02:59 smithi039 ceph-mon[31721]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-03-18T16:02:59.379 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:02:59 smithi196 ceph-mon[36307]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-03-18T16:02:59.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:02:59 smithi194 ceph-mon[36200]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 107 KiB/s, 0 objects/s recovering 2024-03-18T16:03:01.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:01 smithi196 ceph-mon[36307]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-03-18T16:03:01.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:01 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/2132936787' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c1b183ff-383f-4c5c-9075-f76bab1c0bb3"}]: dispatch 2024-03-18T16:03:01.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:01 smithi196 ceph-mon[36307]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c1b183ff-383f-4c5c-9075-f76bab1c0bb3"}]: dispatch 2024-03-18T16:03:01.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:01 smithi196 ceph-mon[36307]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c1b183ff-383f-4c5c-9075-f76bab1c0bb3"}]': finished 2024-03-18T16:03:01.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:01 smithi196 ceph-mon[36307]: osdmap e43: 7 total, 6 up, 7 in 2024-03-18T16:03:01.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:01 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T16:03:01.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:01 smithi194 ceph-mon[36200]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-03-18T16:03:01.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:01 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/2132936787' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c1b183ff-383f-4c5c-9075-f76bab1c0bb3"}]: dispatch 2024-03-18T16:03:01.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:01 smithi194 ceph-mon[36200]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c1b183ff-383f-4c5c-9075-f76bab1c0bb3"}]: dispatch 2024-03-18T16:03:01.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:01 smithi194 ceph-mon[36200]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c1b183ff-383f-4c5c-9075-f76bab1c0bb3"}]': finished 2024-03-18T16:03:01.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:01 smithi194 ceph-mon[36200]: osdmap e43: 7 total, 6 up, 7 in 2024-03-18T16:03:01.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:01 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T16:03:01.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:01 smithi039 ceph-mon[31721]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-03-18T16:03:01.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:01 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/2132936787' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c1b183ff-383f-4c5c-9075-f76bab1c0bb3"}]: dispatch 2024-03-18T16:03:01.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:01 smithi039 ceph-mon[31721]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c1b183ff-383f-4c5c-9075-f76bab1c0bb3"}]: dispatch 2024-03-18T16:03:01.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:01 smithi039 ceph-mon[31721]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c1b183ff-383f-4c5c-9075-f76bab1c0bb3"}]': finished 2024-03-18T16:03:01.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:01 smithi039 ceph-mon[31721]: osdmap e43: 7 total, 6 up, 7 in 2024-03-18T16:03:01.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:01 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T16:03:02.477 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:02 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/2948029294' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T16:03:02.492 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:02 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/2948029294' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T16:03:02.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:02 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/2948029294' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T16:03:03.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:03 smithi196 ceph-mon[36307]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-03-18T16:03:03.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:03 smithi194 ceph-mon[36200]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-03-18T16:03:03.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:03 smithi039 ceph-mon[31721]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-03-18T16:03:05.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:05 smithi039 ceph-mon[31721]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-03-18T16:03:05.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:05 smithi196 ceph-mon[36307]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-03-18T16:03:05.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:05 smithi194 ceph-mon[36200]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-03-18T16:03:07.256 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:07 smithi039 ceph-mon[31721]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-18T16:03:07.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:07 smithi196 ceph-mon[36307]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-18T16:03:07.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:07 smithi194 ceph-mon[36200]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-18T16:03:09.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:09 smithi196 ceph-mon[36307]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-18T16:03:09.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:09 smithi194 ceph-mon[36200]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-18T16:03:09.512 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:09 smithi039 ceph-mon[31721]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-18T16:03:10.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:10 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-18T16:03:10.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:10 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:03:10.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:10 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-18T16:03:10.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:10 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:03:10.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:10 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-18T16:03:10.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:10 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:03:11.219 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:11 smithi039 ceph-mon[31721]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-18T16:03:11.219 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:11 smithi039 ceph-mon[31721]: Deploying daemon osd.6 on smithi039 2024-03-18T16:03:11.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:11 smithi196 ceph-mon[36307]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-18T16:03:11.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:11 smithi196 ceph-mon[36307]: Deploying daemon osd.6 on smithi039 2024-03-18T16:03:11.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:11 smithi194 ceph-mon[36200]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-18T16:03:11.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:11 smithi194 ceph-mon[36200]: Deploying daemon osd.6 on smithi039 2024-03-18T16:03:13.447 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:13 smithi039 ceph-mon[31721]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-18T16:03:13.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:13 smithi196 ceph-mon[36307]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-18T16:03:13.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:13 smithi194 ceph-mon[36200]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-18T16:03:15.332 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:15 smithi039 ceph-mon[31721]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-18T16:03:15.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:15 smithi196 ceph-mon[36307]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-18T16:03:15.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:15 smithi194 ceph-mon[36200]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-18T16:03:17.297 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:17 smithi039 ceph-mon[31721]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-18T16:03:17.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:17 smithi196 ceph-mon[36307]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-18T16:03:17.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:17 smithi194 ceph-mon[36200]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-18T16:03:19.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:19 smithi039 ceph-mon[31721]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-18T16:03:19.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:19 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:03:19.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:19 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:03:19.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:19 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:03:19.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:19 smithi196 ceph-mon[36307]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-18T16:03:19.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:19 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:03:19.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:19 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:03:19.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:19 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:03:19.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:19 smithi194 ceph-mon[36200]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-18T16:03:19.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:19 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:03:19.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:19 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:03:19.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:19 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:03:21.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:21 smithi039 ceph-mon[31721]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-18T16:03:21.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:21 smithi196 ceph-mon[36307]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-18T16:03:21.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:21 smithi194 ceph-mon[36200]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-18T16:03:23.367 INFO:teuthology.orchestra.run.smithi039.stdout:Created osd(s) 6 on host 'smithi039' 2024-03-18T16:03:23.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:03:23.365+0000 7f7cf5600700 1 -- 172.21.15.39:0/3738368027 <== mgr.14152 v2:172.21.15.39:6800/2749214286 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f7cf8062710 con 0x7f7ce00a2910 2024-03-18T16:03:23.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:03:23.369+0000 7f7cfde00700 1 -- 172.21.15.39:0/3738368027 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f7ce00a2910 msgr2=0x7f7ce00a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:03:23.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:03:23.369+0000 7f7cfde00700 1 --2- 172.21.15.39:0/3738368027 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f7ce00a2910 0x7f7ce00a4dd0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7cec009fd0 tx=0x7f7cec009310 comp rx=0 tx=0).stop 2024-03-18T16:03:23.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:03:23.369+0000 7f7cfde00700 1 -- 172.21.15.39:0/3738368027 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7cf8105ff0 msgr2=0x7f7cf81a1ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:03:23.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:03:23.369+0000 7f7cfde00700 1 --2- 172.21.15.39:0/3738368027 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7cf8105ff0 0x7f7cf81a1ca0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7cf0009f60 tx=0x7f7cf0009f90 comp rx=0 tx=0).stop 2024-03-18T16:03:23.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:03:23.369+0000 7f7cfde00700 1 -- 172.21.15.39:0/3738368027 shutdown_connections 2024-03-18T16:03:23.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:03:23.369+0000 7f7cfde00700 1 --2- 172.21.15.39:0/3738368027 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f7ce00a2910 0x7f7ce00a4dd0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:23.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:03:23.369+0000 7f7cfde00700 1 --2- 172.21.15.39:0/3738368027 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7cf8105ff0 0x7f7cf81a1ca0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:23.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:03:23.369+0000 7f7cfde00700 1 --2- 172.21.15.39:0/3738368027 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7cf8104450 0x7f7cf819a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:23.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:03:23.369+0000 7f7cfde00700 1 --2- 172.21.15.39:0/3738368027 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7cf8105650 0x7f7cf819ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:23.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:03:23.369+0000 7f7cfde00700 1 -- 172.21.15.39:0/3738368027 >> 172.21.15.39:0/3738368027 conn(0x7f7cf80ff9f0 msgr2=0x7f7cf81089d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:03:23.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:03:23.370+0000 7f7cfde00700 1 -- 172.21.15.39:0/3738368027 shutdown_connections 2024-03-18T16:03:23.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:03:23.370+0000 7f7cfde00700 1 -- 172.21.15.39:0/3738368027 wait complete. 2024-03-18T16:03:23.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:23 smithi196 ceph-mon[36307]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-18T16:03:23.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:23 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:03:23.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:23 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:03:23.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:23 smithi196 ceph-mon[36307]: from='osd.6 [v2:172.21.15.39:6818/1213371331,v1:172.21.15.39:6819/1213371331]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-18T16:03:23.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:23 smithi196 ceph-mon[36307]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-18T16:03:23.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:23 smithi194 ceph-mon[36200]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-18T16:03:23.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:23 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:03:23.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:23 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:03:23.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:23 smithi194 ceph-mon[36200]: from='osd.6 [v2:172.21.15.39:6818/1213371331,v1:172.21.15.39:6819/1213371331]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-18T16:03:23.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:23 smithi194 ceph-mon[36200]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-18T16:03:23.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:23 smithi039 ceph-mon[31721]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-18T16:03:23.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:23 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:03:23.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:23 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:03:23.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:23 smithi039 ceph-mon[31721]: from='osd.6 [v2:172.21.15.39:6818/1213371331,v1:172.21.15.39:6819/1213371331]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-18T16:03:23.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:23 smithi039 ceph-mon[31721]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-18T16:03:24.224 DEBUG:teuthology.orchestra.run.smithi039:osd.6> sudo journalctl -f -n 0 -u ceph-1892d8fc-e540-11ee-95c9-87774f69a715@osd.6.service 2024-03-18T16:03:24.230 INFO:tasks.cephadm:Deploying osd.7 on smithi194 with /dev/vg_nvme/lv_2... 2024-03-18T16:03:24.230 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-18T16:03:24.473 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T16:03:24.477 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:24 smithi196 ceph-mon[36307]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-18T16:03:24.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:24 smithi196 ceph-mon[36307]: osdmap e44: 7 total, 6 up, 7 in 2024-03-18T16:03:24.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:24 smithi196 ceph-mon[36307]: from='osd.6 [v2:172.21.15.39:6818/1213371331,v1:172.21.15.39:6819/1213371331]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]: dispatch 2024-03-18T16:03:24.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:24 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T16:03:24.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:24 smithi196 ceph-mon[36307]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]: dispatch 2024-03-18T16:03:24.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:24 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:03:24.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:24 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:03:24.478 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:24 smithi039 ceph-mon[31721]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-18T16:03:24.478 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:24 smithi039 ceph-mon[31721]: osdmap e44: 7 total, 6 up, 7 in 2024-03-18T16:03:24.479 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:24 smithi039 ceph-mon[31721]: from='osd.6 [v2:172.21.15.39:6818/1213371331,v1:172.21.15.39:6819/1213371331]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]: dispatch 2024-03-18T16:03:24.479 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:24 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T16:03:24.479 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:24 smithi039 ceph-mon[31721]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]: dispatch 2024-03-18T16:03:24.479 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:24 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:03:24.479 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:24 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:03:24.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:24 smithi194 ceph-mon[36200]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-18T16:03:24.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:24 smithi194 ceph-mon[36200]: osdmap e44: 7 total, 6 up, 7 in 2024-03-18T16:03:24.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:24 smithi194 ceph-mon[36200]: from='osd.6 [v2:172.21.15.39:6818/1213371331,v1:172.21.15.39:6819/1213371331]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]: dispatch 2024-03-18T16:03:24.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:24 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T16:03:24.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:24 smithi194 ceph-mon[36200]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]: dispatch 2024-03-18T16:03:24.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:24 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:03:24.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:24 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:03:25.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:25 smithi196 ceph-mon[36307]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-18T16:03:25.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:25 smithi196 ceph-mon[36307]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]': finished 2024-03-18T16:03:25.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:25 smithi196 ceph-mon[36307]: osdmap e45: 7 total, 6 up, 7 in 2024-03-18T16:03:25.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:25 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T16:03:25.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:25 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T16:03:25.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:25 smithi196 ceph-mon[36307]: from='osd.6 ' entity='osd.6' 2024-03-18T16:03:25.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:25 smithi194 ceph-mon[36200]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-18T16:03:25.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:25 smithi194 ceph-mon[36200]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]': finished 2024-03-18T16:03:25.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:25 smithi194 ceph-mon[36200]: osdmap e45: 7 total, 6 up, 7 in 2024-03-18T16:03:25.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:25 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T16:03:25.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:25 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T16:03:25.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:25 smithi194 ceph-mon[36200]: from='osd.6 ' entity='osd.6' 2024-03-18T16:03:25.532 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:25 smithi039 ceph-mon[31721]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-18T16:03:25.532 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:25 smithi039 ceph-mon[31721]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]': finished 2024-03-18T16:03:25.532 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:25 smithi039 ceph-mon[31721]: osdmap e45: 7 total, 6 up, 7 in 2024-03-18T16:03:25.532 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:25 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T16:03:25.532 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:25 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T16:03:25.532 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:25 smithi039 ceph-mon[31721]: from='osd.6 ' entity='osd.6' 2024-03-18T16:03:25.532 INFO:journalctl@ceph.osd.6.smithi039.stdout:Mar 18 16:03:25 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-6[57699]: 2024-03-18T16:03:25.122+0000 7f68d9400700 -1 osd.6 0 waiting for initial osdmap 2024-03-18T16:03:25.532 INFO:journalctl@ceph.osd.6.smithi039.stdout:Mar 18 16:03:25 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-6[57699]: 2024-03-18T16:03:25.156+0000 7f68ce800700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T16:03:26.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:26 smithi196 ceph-mon[36307]: purged_snaps scrub starts 2024-03-18T16:03:26.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:26 smithi196 ceph-mon[36307]: purged_snaps scrub ok 2024-03-18T16:03:26.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:26 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T16:03:26.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:26 smithi196 ceph-mon[36307]: osd.6 [v2:172.21.15.39:6818/1213371331,v1:172.21.15.39:6819/1213371331] boot 2024-03-18T16:03:26.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:26 smithi196 ceph-mon[36307]: osdmap e46: 7 total, 7 up, 7 in 2024-03-18T16:03:26.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:26 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T16:03:26.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:26 smithi194 ceph-mon[36200]: purged_snaps scrub starts 2024-03-18T16:03:26.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:26 smithi194 ceph-mon[36200]: purged_snaps scrub ok 2024-03-18T16:03:26.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:26 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T16:03:26.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:26 smithi194 ceph-mon[36200]: osd.6 [v2:172.21.15.39:6818/1213371331,v1:172.21.15.39:6819/1213371331] boot 2024-03-18T16:03:26.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:26 smithi194 ceph-mon[36200]: osdmap e46: 7 total, 7 up, 7 in 2024-03-18T16:03:26.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:26 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T16:03:26.507 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:26 smithi039 ceph-mon[31721]: purged_snaps scrub starts 2024-03-18T16:03:26.508 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:26 smithi039 ceph-mon[31721]: purged_snaps scrub ok 2024-03-18T16:03:26.508 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:26 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T16:03:26.508 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:26 smithi039 ceph-mon[31721]: osd.6 [v2:172.21.15.39:6818/1213371331,v1:172.21.15.39:6819/1213371331] boot 2024-03-18T16:03:26.508 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:26 smithi039 ceph-mon[31721]: osdmap e46: 7 total, 7 up, 7 in 2024-03-18T16:03:26.508 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:26 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-18T16:03:26.676 INFO:teuthology.orchestra.run.smithi194.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-18T16:03:26.676 INFO:teuthology.orchestra.run.smithi194.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-18T16:03:26.677 INFO:teuthology.orchestra.run.smithi194.stderr: stderr: 10+0 records in 2024-03-18T16:03:26.677 INFO:teuthology.orchestra.run.smithi194.stderr:10+0 records out 2024-03-18T16:03:26.677 INFO:teuthology.orchestra.run.smithi194.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.013141 s, 798 MB/s 2024-03-18T16:03:26.677 INFO:teuthology.orchestra.run.smithi194.stderr:--> Zapping successful for: 2024-03-18T16:03:27.368 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:27 smithi194 ceph-mon[36200]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-18T16:03:27.398 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi194:vg_nvme/lv_2 2024-03-18T16:03:27.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:27 smithi039 ceph-mon[31721]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-18T16:03:27.644 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T16:03:27.727 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:27 smithi196 ceph-mon[36307]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 187 MiB used, 536 GiB / 536 GiB avail 2024-03-18T16:03:28.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:28 smithi194 ceph-mon[36200]: osdmap e47: 7 total, 7 up, 7 in 2024-03-18T16:03:28.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:28 smithi039 ceph-mon[31721]: osdmap e47: 7 total, 7 up, 7 in 2024-03-18T16:03:28.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:28 smithi196 ceph-mon[36307]: osdmap e47: 7 total, 7 up, 7 in 2024-03-18T16:03:29.316 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:03:29.313+0000 7f17b5e5d700 1 -- 172.21.15.194:0/3762940010 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f17b0103ea0 msgr2=0x7f17b010a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:03:29.316 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:03:29.313+0000 7f17b5e5d700 1 --2- 172.21.15.194:0/3762940010 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f17b0103ea0 0x7f17b010a750 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f17a800b0d0 tx=0x7f17a800b3e0 comp rx=0 tx=0).stop 2024-03-18T16:03:29.316 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:03:29.313+0000 7f17b5e5d700 1 -- 172.21.15.194:0/3762940010 shutdown_connections 2024-03-18T16:03:29.317 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:03:29.313+0000 7f17b5e5d700 1 --2- 172.21.15.194:0/3762940010 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f17b0103ea0 0x7f17b010a750 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:29.317 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:03:29.313+0000 7f17b5e5d700 1 --2- 172.21.15.194:0/3762940010 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f17b0102300 0x7f17b0102720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:29.317 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:03:29.313+0000 7f17b5e5d700 1 --2- 172.21.15.194:0/3762940010 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f17b0103500 0x7f17b0103960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:29.317 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:03:29.313+0000 7f17b5e5d700 1 -- 172.21.15.194:0/3762940010 >> 172.21.15.194:0/3762940010 conn(0x7f17b00fd8e0 msgr2=0x7f17b00ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:03:29.317 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:03:29.313+0000 7f17b5e5d700 1 -- 172.21.15.194:0/3762940010 shutdown_connections 2024-03-18T16:03:29.317 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:03:29.313+0000 7f17b5e5d700 1 -- 172.21.15.194:0/3762940010 wait complete. 2024-03-18T16:03:29.317 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:03:29.314+0000 7f17b5e5d700 1 Processor -- start 2024-03-18T16:03:29.317 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:03:29.314+0000 7f17b5e5d700 1 -- start start 2024-03-18T16:03:29.317 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:03:29.314+0000 7f17b5e5d700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f17b0102300 0x7f17b0076070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:03:29.317 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:03:29.314+0000 7f17b5e5d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f17b0103500 0x7f17b00746c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:03:29.317 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:03:29.314+0000 7f17b5e5d700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f17b0103ea0 0x7f17b0074c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:03:29.317 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:03:29.314+0000 7f17b5e5d700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f17b010dfe0 con 0x7f17b0103500 2024-03-18T16:03:29.317 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:03:29.314+0000 7f17b5e5d700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f17b010de60 con 0x7f17b0103ea0 2024-03-18T16:03:29.318 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:03:29.314+0000 7f17b5e5d700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f17b010e160 con 0x7f17b0102300 2024-03-18T16:03:29.318 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:03:29.314+0000 7f17af7fe700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f17b0102300 0x7f17b0076070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:03:29.318 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:03:29.314+0000 7f17af7fe700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f17b0102300 0x7f17b0076070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.194:55452/0 (socket says 172.21.15.194:55452) 2024-03-18T16:03:29.318 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:03:29.314+0000 7f17af7fe700 1 -- 172.21.15.194:0/837122601 learned_addr learned my addr 172.21.15.194:0/837122601 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-03-18T16:03:29.318 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:03:29.314+0000 7f17affff700 1 --2- 172.21.15.194:0/837122601 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f17b0103ea0 0x7f17b0074c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:03:29.318 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:03:29.314+0000 7f17affff700 1 -- 172.21.15.194:0/837122601 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f17b0102300 msgr2=0x7f17b0076070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:03:29.318 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:03:29.314+0000 7f17affff700 1 --2- 172.21.15.194:0/837122601 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f17b0102300 0x7f17b0076070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:29.318 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:03:29.314+0000 7f17affff700 1 -- 172.21.15.194:0/837122601 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f17b0103500 msgr2=0x7f17b00746c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:03:29.318 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:03:29.314+0000 7f17affff700 1 --2- 172.21.15.194:0/837122601 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f17b0103500 0x7f17b00746c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:29.318 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:03:29.314+0000 7f17affff700 1 -- 172.21.15.194:0/837122601 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17b00754b0 con 0x7f17b0103ea0 2024-03-18T16:03:29.318 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:03:29.314+0000 7f17affff700 1 --2- 172.21.15.194:0/837122601 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f17b0103ea0 0x7f17b0074c00 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f17a8007fa0 tx=0x7f17a8007d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:03:29.319 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:03:29.316+0000 7f17ad7fa700 1 -- 172.21.15.194:0/837122601 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17a8021070 con 0x7f17b0103ea0 2024-03-18T16:03:29.319 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:03:29.316+0000 7f17ad7fa700 1 -- 172.21.15.194:0/837122601 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f17a801dd40 con 0x7f17b0103ea0 2024-03-18T16:03:29.319 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:03:29.316+0000 7f17ad7fa700 1 -- 172.21.15.194:0/837122601 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17a801bc80 con 0x7f17b0103ea0 2024-03-18T16:03:29.319 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:03:29.316+0000 7f17b5e5d700 1 -- 172.21.15.194:0/837122601 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17b01a26d0 con 0x7f17b0103ea0 2024-03-18T16:03:29.320 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:03:29.316+0000 7f17b5e5d700 1 -- 172.21.15.194:0/837122601 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17b01a2c60 con 0x7f17b0103ea0 2024-03-18T16:03:29.320 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:03:29.317+0000 7f17b5e5d700 1 -- 172.21.15.194:0/837122601 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17b0068f90 con 0x7f17b0103ea0 2024-03-18T16:03:29.320 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:03:29.318+0000 7f17ad7fa700 1 -- 172.21.15.194:0/837122601 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f17a8024070 con 0x7f17b0103ea0 2024-03-18T16:03:29.320 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:03:29.318+0000 7f17ad7fa700 1 --2- 172.21.15.194:0/837122601 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f179c0a2990 0x7f179c0a4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:03:29.320 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:03:29.319+0000 7f17ad7fa700 1 -- 172.21.15.194:0/837122601 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 4965+0+0 (secure 0 0 0) 0x7f17a80ceef0 con 0x7f17b0103ea0 2024-03-18T16:03:29.320 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:03:29.319+0000 7f17af7fe700 1 --2- 172.21.15.194:0/837122601 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f179c0a2990 0x7f179c0a4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:03:29.321 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:03:29.320+0000 7f17af7fe700 1 --2- 172.21.15.194:0/837122601 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f179c0a2990 0x7f179c0a4e50 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f17a0009aa0 tx=0x7f17a0009240 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:03:29.324 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:03:29.323+0000 7f17ad7fa700 1 -- 172.21.15.194:0/837122601 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f17a809cc90 con 0x7f17b0103ea0 2024-03-18T16:03:29.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:29 smithi194 ceph-mon[36200]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-03-18T16:03:29.531 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:03:29.530+0000 7f17b5e5d700 1 -- 172.21.15.194:0/837122601 --> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f17b0062710 con 0x7f179c0a2990 2024-03-18T16:03:29.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:29 smithi039 ceph-mon[31721]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-03-18T16:03:29.628 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:29 smithi196 ceph-mon[36307]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-03-18T16:03:30.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:30 smithi194 ceph-mon[36200]: Detected new or changed devices on smithi039 2024-03-18T16:03:30.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:30 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:03:30.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:30 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:03:30.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:30 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:03:30.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:30 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:03:30.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:30 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:03:30.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:30 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:03:30.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:30 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:03:30.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:30 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:03:30.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:30 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:03:30.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:30 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T16:03:30.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:30 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T16:03:30.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:30 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:03:30.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:30 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:03:30.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:30 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:03:30.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:30 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:03:30.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:30 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:03:30.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:30 smithi039 ceph-mon[31721]: Detected new or changed devices on smithi039 2024-03-18T16:03:30.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:30 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:03:30.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:30 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:03:30.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:30 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:03:30.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:30 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:03:30.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:30 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:03:30.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:30 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:03:30.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:30 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:03:30.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:30 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:03:30.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:30 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:03:30.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:30 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T16:03:30.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:30 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T16:03:30.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:30 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:03:30.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:30 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:03:30.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:30 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:03:30.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:30 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:03:30.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:30 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:03:30.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:30 smithi196 ceph-mon[36307]: Detected new or changed devices on smithi039 2024-03-18T16:03:30.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:30 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:03:30.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:30 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:03:30.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:30 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:03:30.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:30 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:03:30.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:30 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:03:30.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:30 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:03:30.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:30 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:03:30.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:30 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:03:30.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:30 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:03:30.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:30 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T16:03:30.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:30 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T16:03:30.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:30 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:03:30.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:30 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:03:30.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:30 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:03:30.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:30 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:03:30.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:30 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:03:31.468 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:31 smithi194 ceph-mon[36200]: Adjusting osd_memory_target on smithi039 to 5697M 2024-03-18T16:03:31.468 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:31 smithi194 ceph-mon[36200]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-03-18T16:03:31.469 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:31 smithi194 ceph-mon[36200]: from='client.24281 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:03:31.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:31 smithi039 ceph-mon[31721]: Adjusting osd_memory_target on smithi039 to 5697M 2024-03-18T16:03:31.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:31 smithi039 ceph-mon[31721]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-03-18T16:03:31.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:31 smithi039 ceph-mon[31721]: from='client.24281 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:03:31.727 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:31 smithi196 ceph-mon[36307]: Adjusting osd_memory_target on smithi039 to 5697M 2024-03-18T16:03:31.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:31 smithi196 ceph-mon[36307]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-03-18T16:03:31.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:31 smithi196 ceph-mon[36307]: from='client.24281 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:03:32.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:32 smithi039 ceph-mon[31721]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-03-18T16:03:32.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:32 smithi039 ceph-mon[31721]: from='client.? 172.21.15.194:0/386770000' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "612f8697-957d-4ef4-a87f-241b36f9d13d"}]: dispatch 2024-03-18T16:03:32.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:32 smithi039 ceph-mon[31721]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "612f8697-957d-4ef4-a87f-241b36f9d13d"}]: dispatch 2024-03-18T16:03:32.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:32 smithi039 ceph-mon[31721]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "612f8697-957d-4ef4-a87f-241b36f9d13d"}]': finished 2024-03-18T16:03:32.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:32 smithi039 ceph-mon[31721]: osdmap e48: 8 total, 7 up, 8 in 2024-03-18T16:03:32.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:32 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T16:03:32.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:32 smithi194 ceph-mon[36200]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-03-18T16:03:32.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:32 smithi194 ceph-mon[36200]: from='client.? 172.21.15.194:0/386770000' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "612f8697-957d-4ef4-a87f-241b36f9d13d"}]: dispatch 2024-03-18T16:03:32.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:32 smithi194 ceph-mon[36200]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "612f8697-957d-4ef4-a87f-241b36f9d13d"}]: dispatch 2024-03-18T16:03:32.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:32 smithi194 ceph-mon[36200]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "612f8697-957d-4ef4-a87f-241b36f9d13d"}]': finished 2024-03-18T16:03:32.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:32 smithi194 ceph-mon[36200]: osdmap e48: 8 total, 7 up, 8 in 2024-03-18T16:03:32.639 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:32 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T16:03:32.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:32 smithi196 ceph-mon[36307]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-03-18T16:03:32.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:32 smithi196 ceph-mon[36307]: from='client.? 172.21.15.194:0/386770000' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "612f8697-957d-4ef4-a87f-241b36f9d13d"}]: dispatch 2024-03-18T16:03:32.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:32 smithi196 ceph-mon[36307]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "612f8697-957d-4ef4-a87f-241b36f9d13d"}]: dispatch 2024-03-18T16:03:32.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:32 smithi196 ceph-mon[36307]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "612f8697-957d-4ef4-a87f-241b36f9d13d"}]': finished 2024-03-18T16:03:32.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:32 smithi196 ceph-mon[36307]: osdmap e48: 8 total, 7 up, 8 in 2024-03-18T16:03:32.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:32 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T16:03:33.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:33 smithi039 ceph-mon[31721]: from='client.? 172.21.15.194:0/3164013258' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T16:03:33.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:33 smithi196 ceph-mon[36307]: from='client.? 172.21.15.194:0/3164013258' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T16:03:33.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:33 smithi194 ceph-mon[36200]: from='client.? 172.21.15.194:0/3164013258' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T16:03:34.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:34 smithi039 ceph-mon[31721]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-03-18T16:03:34.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:34 smithi196 ceph-mon[36307]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-03-18T16:03:34.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:34 smithi194 ceph-mon[36200]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-03-18T16:03:36.635 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:36 smithi194 ceph-mon[36200]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-03-18T16:03:36.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:36 smithi039 ceph-mon[31721]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-03-18T16:03:36.977 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:36 smithi196 ceph-mon[36307]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-03-18T16:03:38.629 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:38 smithi194 ceph-mon[36200]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-03-18T16:03:38.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:38 smithi039 ceph-mon[31721]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-03-18T16:03:38.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:38 smithi196 ceph-mon[36307]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-03-18T16:03:40.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:40 smithi194 ceph-mon[36200]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-03-18T16:03:40.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:40 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-18T16:03:40.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:40 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:03:40.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:40 smithi194 ceph-mon[36200]: Deploying daemon osd.7 on smithi194 2024-03-18T16:03:40.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:40 smithi039 ceph-mon[31721]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-03-18T16:03:40.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:40 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-18T16:03:40.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:40 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:03:40.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:40 smithi039 ceph-mon[31721]: Deploying daemon osd.7 on smithi194 2024-03-18T16:03:40.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:40 smithi196 ceph-mon[36307]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-03-18T16:03:40.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:40 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-18T16:03:40.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:40 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:03:40.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:40 smithi196 ceph-mon[36307]: Deploying daemon osd.7 on smithi194 2024-03-18T16:03:42.806 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:42 smithi194 ceph-mon[36200]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-03-18T16:03:42.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:42 smithi039 ceph-mon[31721]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-03-18T16:03:42.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:42 smithi196 ceph-mon[36307]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-03-18T16:03:44.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:44 smithi039 ceph-mon[31721]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-03-18T16:03:44.889 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:44 smithi194 ceph-mon[36200]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-03-18T16:03:44.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:44 smithi196 ceph-mon[36307]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-03-18T16:03:46.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:46 smithi194 ceph-mon[36200]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-03-18T16:03:46.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:46 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:03:46.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:46 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:03:46.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:46 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:03:46.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:46 smithi039 ceph-mon[31721]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-03-18T16:03:46.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:46 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:03:46.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:46 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:03:46.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:46 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:03:46.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:46 smithi196 ceph-mon[36307]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-03-18T16:03:46.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:46 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:03:46.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:46 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:03:46.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:46 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:03:48.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:48 smithi039 ceph-mon[31721]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-03-18T16:03:48.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:48 smithi196 ceph-mon[36307]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-03-18T16:03:48.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:48 smithi194 ceph-mon[36200]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-03-18T16:03:49.421 INFO:teuthology.orchestra.run.smithi194.stdout:Created osd(s) 7 on host 'smithi194' 2024-03-18T16:03:49.422 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:03:49.419+0000 7f17ad7fa700 1 -- 172.21.15.194:0/837122601 <== mgr.14152 v2:172.21.15.39:6800/2749214286 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f17b0062710 con 0x7f179c0a2990 2024-03-18T16:03:49.424 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:03:49.422+0000 7f17b5e5d700 1 -- 172.21.15.194:0/837122601 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f179c0a2990 msgr2=0x7f179c0a4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:03:49.424 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:03:49.422+0000 7f17b5e5d700 1 --2- 172.21.15.194:0/837122601 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f179c0a2990 0x7f179c0a4e50 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f17a0009aa0 tx=0x7f17a0009240 comp rx=0 tx=0).stop 2024-03-18T16:03:49.424 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:03:49.422+0000 7f17b5e5d700 1 -- 172.21.15.194:0/837122601 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f17b0103ea0 msgr2=0x7f17b0074c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:03:49.424 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:03:49.422+0000 7f17b5e5d700 1 --2- 172.21.15.194:0/837122601 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f17b0103ea0 0x7f17b0074c00 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f17a8007fa0 tx=0x7f17a8007d10 comp rx=0 tx=0).stop 2024-03-18T16:03:49.424 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:03:49.422+0000 7f17b5e5d700 1 -- 172.21.15.194:0/837122601 shutdown_connections 2024-03-18T16:03:49.424 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:03:49.422+0000 7f17b5e5d700 1 --2- 172.21.15.194:0/837122601 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f179c0a2990 0x7f179c0a4e50 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:49.424 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:03:49.422+0000 7f17b5e5d700 1 --2- 172.21.15.194:0/837122601 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f17b0103ea0 0x7f17b0074c00 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:49.424 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:03:49.422+0000 7f17b5e5d700 1 --2- 172.21.15.194:0/837122601 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f17b0102300 0x7f17b0076070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:49.424 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:03:49.422+0000 7f17b5e5d700 1 --2- 172.21.15.194:0/837122601 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f17b0103500 0x7f17b00746c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:49.424 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:03:49.422+0000 7f17b5e5d700 1 -- 172.21.15.194:0/837122601 >> 172.21.15.194:0/837122601 conn(0x7f17b00fd8e0 msgr2=0x7f17b0106800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:03:49.424 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:03:49.422+0000 7f17b5e5d700 1 -- 172.21.15.194:0/837122601 shutdown_connections 2024-03-18T16:03:49.425 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:03:49.422+0000 7f17b5e5d700 1 -- 172.21.15.194:0/837122601 wait complete. 2024-03-18T16:03:49.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:49 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:03:49.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:49 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:03:49.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:49 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:03:49.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:49 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:03:49.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:49 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:03:49.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:49 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:03:49.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:49 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:03:49.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:49 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:03:49.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:49 smithi194 ceph-mon[36200]: from='osd.7 [v2:172.21.15.194:6816/3299162444,v1:172.21.15.194:6817/3299162444]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-18T16:03:49.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:49 smithi194 ceph-mon[36200]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-18T16:03:49.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:49 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:03:49.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:49 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:03:49.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:49 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:03:49.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:49 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:03:49.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:49 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:03:49.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:49 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:03:49.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:49 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:03:49.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:49 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:03:49.838 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:49 smithi039 ceph-mon[31721]: from='osd.7 [v2:172.21.15.194:6816/3299162444,v1:172.21.15.194:6817/3299162444]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-18T16:03:49.838 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:49 smithi039 ceph-mon[31721]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-18T16:03:49.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:49 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:03:49.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:49 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:03:49.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:49 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:03:49.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:49 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:03:49.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:49 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:03:49.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:49 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:03:49.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:49 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:03:49.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:49 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:03:49.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:49 smithi196 ceph-mon[36307]: from='osd.7 [v2:172.21.15.194:6816/3299162444,v1:172.21.15.194:6817/3299162444]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-18T16:03:49.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:49 smithi196 ceph-mon[36307]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-18T16:03:50.090 DEBUG:teuthology.orchestra.run.smithi194:osd.7> sudo journalctl -f -n 0 -u ceph-1892d8fc-e540-11ee-95c9-87774f69a715@osd.7.service 2024-03-18T16:03:50.092 INFO:tasks.cephadm:Deploying osd.8 on smithi196 with /dev/vg_nvme/lv_2... 2024-03-18T16:03:50.093 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-18T16:03:50.337 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T16:03:50.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:50 smithi196 ceph-mon[36307]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-03-18T16:03:50.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:50 smithi196 ceph-mon[36307]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-18T16:03:50.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:50 smithi196 ceph-mon[36307]: osdmap e49: 8 total, 7 up, 8 in 2024-03-18T16:03:50.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:50 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T16:03:50.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:50 smithi196 ceph-mon[36307]: from='osd.7 [v2:172.21.15.194:6816/3299162444,v1:172.21.15.194:6817/3299162444]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-03-18T16:03:50.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:50 smithi196 ceph-mon[36307]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-03-18T16:03:50.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:50 smithi194 ceph-mon[36200]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-03-18T16:03:50.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:50 smithi194 ceph-mon[36200]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-18T16:03:50.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:50 smithi194 ceph-mon[36200]: osdmap e49: 8 total, 7 up, 8 in 2024-03-18T16:03:50.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:50 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T16:03:50.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:50 smithi194 ceph-mon[36200]: from='osd.7 [v2:172.21.15.194:6816/3299162444,v1:172.21.15.194:6817/3299162444]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-03-18T16:03:50.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:50 smithi194 ceph-mon[36200]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-03-18T16:03:50.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:50 smithi039 ceph-mon[31721]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-03-18T16:03:50.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:50 smithi039 ceph-mon[31721]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-18T16:03:50.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:50 smithi039 ceph-mon[31721]: osdmap e49: 8 total, 7 up, 8 in 2024-03-18T16:03:50.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:50 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T16:03:50.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:50 smithi039 ceph-mon[31721]: from='osd.7 [v2:172.21.15.194:6816/3299162444,v1:172.21.15.194:6817/3299162444]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-03-18T16:03:50.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:50 smithi039 ceph-mon[31721]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-03-18T16:03:51.575 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 18 16:03:51 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-7[54818]: 2024-03-18T16:03:51.427+0000 7f28ccf51700 -1 osd.7 0 waiting for initial osdmap 2024-03-18T16:03:51.575 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 18 16:03:51 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-7[54818]: 2024-03-18T16:03:51.436+0000 7f28c8d44700 -1 osd.7 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T16:03:52.117 INFO:teuthology.orchestra.run.smithi196.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-18T16:03:52.118 INFO:teuthology.orchestra.run.smithi196.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-18T16:03:52.118 INFO:teuthology.orchestra.run.smithi196.stderr: stderr: 10+0 records in 2024-03-18T16:03:52.118 INFO:teuthology.orchestra.run.smithi196.stderr:10+0 records out 2024-03-18T16:03:52.118 INFO:teuthology.orchestra.run.smithi196.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0134236 s, 781 MB/s 2024-03-18T16:03:52.118 INFO:teuthology.orchestra.run.smithi196.stderr:--> Zapping successful for: 2024-03-18T16:03:52.727 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:52 smithi196 ceph-mon[36307]: purged_snaps scrub starts 2024-03-18T16:03:52.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:52 smithi196 ceph-mon[36307]: purged_snaps scrub ok 2024-03-18T16:03:52.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:52 smithi196 ceph-mon[36307]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-03-18T16:03:52.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:52 smithi196 ceph-mon[36307]: osdmap e50: 8 total, 7 up, 8 in 2024-03-18T16:03:52.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:52 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T16:03:52.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:52 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T16:03:52.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:52 smithi196 ceph-mon[36307]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-03-18T16:03:52.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:52 smithi194 ceph-mon[36200]: purged_snaps scrub starts 2024-03-18T16:03:52.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:52 smithi194 ceph-mon[36200]: purged_snaps scrub ok 2024-03-18T16:03:52.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:52 smithi194 ceph-mon[36200]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-03-18T16:03:52.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:52 smithi194 ceph-mon[36200]: osdmap e50: 8 total, 7 up, 8 in 2024-03-18T16:03:52.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:52 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T16:03:52.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:52 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T16:03:52.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:52 smithi194 ceph-mon[36200]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-03-18T16:03:52.804 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi196:vg_nvme/lv_2 2024-03-18T16:03:52.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:52 smithi039 ceph-mon[31721]: purged_snaps scrub starts 2024-03-18T16:03:52.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:52 smithi039 ceph-mon[31721]: purged_snaps scrub ok 2024-03-18T16:03:52.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:52 smithi039 ceph-mon[31721]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-03-18T16:03:52.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:52 smithi039 ceph-mon[31721]: osdmap e50: 8 total, 7 up, 8 in 2024-03-18T16:03:52.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:52 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T16:03:52.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:52 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T16:03:52.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:52 smithi039 ceph-mon[31721]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 218 MiB used, 626 GiB / 626 GiB avail 2024-03-18T16:03:53.046 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T16:03:53.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:53 smithi196 ceph-mon[36307]: osd.7 [v2:172.21.15.194:6816/3299162444,v1:172.21.15.194:6817/3299162444] boot 2024-03-18T16:03:53.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:53 smithi196 ceph-mon[36307]: osdmap e51: 8 total, 8 up, 8 in 2024-03-18T16:03:53.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:53 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T16:03:53.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:53 smithi196 ceph-mon[36307]: Detected new or changed devices on smithi194 2024-03-18T16:03:53.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:53 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:03:53.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:53 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:03:53.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:53 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:03:53.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:53 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:03:53.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:53 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:03:53.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:53 smithi196 ceph-mon[36307]: Adjusting osd_memory_target on smithi194 to 5693M 2024-03-18T16:03:53.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:53 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:03:53.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:53 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:03:53.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:53 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:03:53.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:53 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:03:53.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:53 smithi196 ceph-mon[36307]: osdmap e52: 8 total, 8 up, 8 in 2024-03-18T16:03:53.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:53 smithi194 ceph-mon[36200]: osd.7 [v2:172.21.15.194:6816/3299162444,v1:172.21.15.194:6817/3299162444] boot 2024-03-18T16:03:53.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:53 smithi194 ceph-mon[36200]: osdmap e51: 8 total, 8 up, 8 in 2024-03-18T16:03:53.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:53 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T16:03:53.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:53 smithi194 ceph-mon[36200]: Detected new or changed devices on smithi194 2024-03-18T16:03:53.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:53 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:03:53.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:53 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:03:53.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:53 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:03:53.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:53 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:03:53.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:53 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:03:53.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:53 smithi194 ceph-mon[36200]: Adjusting osd_memory_target on smithi194 to 5693M 2024-03-18T16:03:53.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:53 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:03:53.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:53 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:03:53.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:53 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:03:53.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:53 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:03:53.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:53 smithi194 ceph-mon[36200]: osdmap e52: 8 total, 8 up, 8 in 2024-03-18T16:03:53.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:53 smithi039 ceph-mon[31721]: osd.7 [v2:172.21.15.194:6816/3299162444,v1:172.21.15.194:6817/3299162444] boot 2024-03-18T16:03:53.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:53 smithi039 ceph-mon[31721]: osdmap e51: 8 total, 8 up, 8 in 2024-03-18T16:03:53.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:53 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-18T16:03:53.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:53 smithi039 ceph-mon[31721]: Detected new or changed devices on smithi194 2024-03-18T16:03:53.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:53 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:03:53.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:53 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:03:53.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:53 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:03:53.838 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:53 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:03:53.838 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:53 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:03:53.838 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:53 smithi039 ceph-mon[31721]: Adjusting osd_memory_target on smithi194 to 5693M 2024-03-18T16:03:53.838 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:53 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:03:53.838 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:53 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:03:53.838 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:53 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:03:53.838 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:53 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:03:53.838 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:53 smithi039 ceph-mon[31721]: osdmap e52: 8 total, 8 up, 8 in 2024-03-18T16:03:54.705 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:03:54.702+0000 7fadb1891700 1 -- 172.21.15.196:0/3497570945 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fadac103500 msgr2=0x7fadac103960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:03:54.705 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:03:54.702+0000 7fadb1891700 1 --2- 172.21.15.196:0/3497570945 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fadac103500 0x7fadac103960 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fada0009990 tx=0x7fada0009ca0 comp rx=0 tx=0).stop 2024-03-18T16:03:54.705 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:03:54.702+0000 7fadb1891700 1 -- 172.21.15.196:0/3497570945 shutdown_connections 2024-03-18T16:03:54.705 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:03:54.702+0000 7fadb1891700 1 --2- 172.21.15.196:0/3497570945 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fadac103ea0 0x7fadac10a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:54.705 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:03:54.702+0000 7fadb1891700 1 --2- 172.21.15.196:0/3497570945 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fadac102300 0x7fadac102720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:54.705 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:03:54.702+0000 7fadb1891700 1 --2- 172.21.15.196:0/3497570945 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fadac103500 0x7fadac103960 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:54.705 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:03:54.702+0000 7fadb1891700 1 -- 172.21.15.196:0/3497570945 >> 172.21.15.196:0/3497570945 conn(0x7fadac0fd8e0 msgr2=0x7fadac0ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:03:54.705 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:03:54.702+0000 7fadb1891700 1 -- 172.21.15.196:0/3497570945 shutdown_connections 2024-03-18T16:03:54.705 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:03:54.702+0000 7fadb1891700 1 -- 172.21.15.196:0/3497570945 wait complete. 2024-03-18T16:03:54.706 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:54 smithi196 ceph-mon[36307]: pgmap v196: 1 pgs: 1 unknown; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-03-18T16:03:54.706 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:54 smithi196 ceph-mon[36307]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-03-18T16:03:54.707 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:03:54.703+0000 7fadb1891700 1 Processor -- start 2024-03-18T16:03:54.707 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:03:54.703+0000 7fadb1891700 1 -- start start 2024-03-18T16:03:54.708 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:03:54.703+0000 7fadb1891700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fadac102300 0x7fadac19a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:03:54.708 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:03:54.703+0000 7fadb1891700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fadac103500 0x7fadac19ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:03:54.708 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:03:54.703+0000 7fadb1891700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fadac103ea0 0x7fadac1a1da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:03:54.708 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:03:54.704+0000 7fadb1891700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fadac10df60 con 0x7fadac102300 2024-03-18T16:03:54.708 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:03:54.704+0000 7fadb1891700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fadac10dde0 con 0x7fadac103500 2024-03-18T16:03:54.708 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:03:54.704+0000 7fadb1891700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fadac10e0e0 con 0x7fadac103ea0 2024-03-18T16:03:54.708 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:03:54.704+0000 7fadab7fe700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fadac103ea0 0x7fadac1a1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:03:54.708 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:03:54.704+0000 7fadab7fe700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fadac103ea0 0x7fadac1a1da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.196:54948/0 (socket says 172.21.15.196:54948) 2024-03-18T16:03:54.708 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:03:54.704+0000 7fadab7fe700 1 -- 172.21.15.196:0/3189127676 learned_addr learned my addr 172.21.15.196:0/3189127676 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-18T16:03:54.708 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:03:54.704+0000 7fadaa7fc700 1 --2- 172.21.15.196:0/3189127676 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fadac103500 0x7fadac19ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:03:54.708 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:03:54.704+0000 7fadaaffd700 1 --2- 172.21.15.196:0/3189127676 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fadac102300 0x7fadac19a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:03:54.708 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:03:54.704+0000 7fadab7fe700 1 -- 172.21.15.196:0/3189127676 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fadac103500 msgr2=0x7fadac19ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:03:54.708 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:03:54.704+0000 7fadab7fe700 1 --2- 172.21.15.196:0/3189127676 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fadac103500 0x7fadac19ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:54.709 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:03:54.704+0000 7fadab7fe700 1 -- 172.21.15.196:0/3189127676 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fadac102300 msgr2=0x7fadac19a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:03:54.709 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:03:54.704+0000 7fadab7fe700 1 --2- 172.21.15.196:0/3189127676 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fadac102300 0x7fadac19a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:03:54.709 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:03:54.704+0000 7fadab7fe700 1 -- 172.21.15.196:0/3189127676 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadac1a2440 con 0x7fadac103ea0 2024-03-18T16:03:54.709 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:03:54.704+0000 7fadab7fe700 1 --2- 172.21.15.196:0/3189127676 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fadac103ea0 0x7fadac1a1da0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fada400add0 tx=0x7fada400c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:03:54.709 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:03:54.704+0000 7fadb088f700 1 -- 172.21.15.196:0/3189127676 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fada4017020 con 0x7fadac103ea0 2024-03-18T16:03:54.709 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:03:54.704+0000 7fadb1891700 1 -- 172.21.15.196:0/3189127676 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fadac1a2730 con 0x7fadac103ea0 2024-03-18T16:03:54.709 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:03:54.704+0000 7fadb088f700 1 -- 172.21.15.196:0/3189127676 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fada400ce20 con 0x7fadac103ea0 2024-03-18T16:03:54.709 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:03:54.705+0000 7fadb088f700 1 -- 172.21.15.196:0/3189127676 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fada40053e0 con 0x7fadac103ea0 2024-03-18T16:03:54.709 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:03:54.705+0000 7fadb1891700 1 -- 172.21.15.196:0/3189127676 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fadac1a2cf0 con 0x7fadac103ea0 2024-03-18T16:03:54.710 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:03:54.706+0000 7fadb1891700 1 -- 172.21.15.196:0/3189127676 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fadac068f90 con 0x7fadac103ea0 2024-03-18T16:03:54.710 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:03:54.707+0000 7fadb088f700 1 -- 172.21.15.196:0/3189127676 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fada400c940 con 0x7fadac103ea0 2024-03-18T16:03:54.710 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:03:54.707+0000 7fadb088f700 1 --2- 172.21.15.196:0/3189127676 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fad940a2890 0x7fad940a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:03:54.710 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:03:54.707+0000 7fadb088f700 1 -- 172.21.15.196:0/3189127676 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 5426+0+0 (secure 0 0 0) 0x7fada40cd790 con 0x7fadac103ea0 2024-03-18T16:03:54.710 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:03:54.708+0000 7fadaaffd700 1 --2- 172.21.15.196:0/3189127676 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fad940a2890 0x7fad940a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:03:54.710 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:03:54.709+0000 7fadaaffd700 1 --2- 172.21.15.196:0/3189127676 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fad940a2890 0x7fad940a4d50 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fad9c006fd0 tx=0x7fad9c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:03:54.714 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:03:54.712+0000 7fadb088f700 1 -- 172.21.15.196:0/3189127676 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fada4021020 con 0x7fadac103ea0 2024-03-18T16:03:54.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:54 smithi194 ceph-mon[36200]: pgmap v196: 1 pgs: 1 unknown; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-03-18T16:03:54.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:54 smithi194 ceph-mon[36200]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-03-18T16:03:54.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:54 smithi039 ceph-mon[31721]: pgmap v196: 1 pgs: 1 unknown; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-03-18T16:03:54.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:54 smithi039 ceph-mon[31721]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-03-18T16:03:54.918 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:03:54.916+0000 7fadb1891700 1 -- 172.21.15.196:0/3189127676 --> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fadac10a3f0 con 0x7fad940a2890 2024-03-18T16:03:55.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:55 smithi196 ceph-mon[36307]: osdmap e53: 8 total, 8 up, 8 in 2024-03-18T16:03:55.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:55 smithi196 ceph-mon[36307]: from='client.24316 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:03:55.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:55 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T16:03:55.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:55 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T16:03:55.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:55 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:03:55.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:55 smithi194 ceph-mon[36200]: osdmap e53: 8 total, 8 up, 8 in 2024-03-18T16:03:55.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:55 smithi194 ceph-mon[36200]: from='client.24316 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:03:55.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:55 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T16:03:55.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:55 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T16:03:55.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:55 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:03:55.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:55 smithi039 ceph-mon[31721]: osdmap e53: 8 total, 8 up, 8 in 2024-03-18T16:03:55.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:55 smithi039 ceph-mon[31721]: from='client.24316 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:03:55.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:55 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T16:03:55.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:55 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T16:03:55.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:55 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:03:56.534 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:56 smithi196 ceph-mon[36307]: pgmap v198: 1 pgs: 1 unknown; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-03-18T16:03:56.534 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:56 smithi196 ceph-mon[36307]: osdmap e54: 8 total, 8 up, 8 in 2024-03-18T16:03:56.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:56 smithi194 ceph-mon[36200]: pgmap v198: 1 pgs: 1 unknown; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-03-18T16:03:56.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:56 smithi194 ceph-mon[36200]: osdmap e54: 8 total, 8 up, 8 in 2024-03-18T16:03:56.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:56 smithi039 ceph-mon[31721]: pgmap v198: 1 pgs: 1 unknown; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-03-18T16:03:56.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:56 smithi039 ceph-mon[31721]: osdmap e54: 8 total, 8 up, 8 in 2024-03-18T16:03:57.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:57 smithi196 ceph-mon[36307]: osdmap e55: 8 total, 8 up, 8 in 2024-03-18T16:03:57.736 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:57 smithi039 ceph-mon[31721]: osdmap e55: 8 total, 8 up, 8 in 2024-03-18T16:03:57.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:57 smithi194 ceph-mon[36200]: osdmap e55: 8 total, 8 up, 8 in 2024-03-18T16:03:58.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:58 smithi194 ceph-mon[36200]: pgmap v201: 1 pgs: 1 unknown; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-03-18T16:03:58.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:58 smithi194 ceph-mon[36200]: from='client.? 172.21.15.196:0/442091658' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "836a9107-4d4f-4594-b8dc-0301e64f59d9"}]: dispatch 2024-03-18T16:03:58.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:58 smithi194 ceph-mon[36200]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "836a9107-4d4f-4594-b8dc-0301e64f59d9"}]: dispatch 2024-03-18T16:03:58.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:58 smithi194 ceph-mon[36200]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "836a9107-4d4f-4594-b8dc-0301e64f59d9"}]': finished 2024-03-18T16:03:58.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:58 smithi194 ceph-mon[36200]: osdmap e56: 9 total, 8 up, 9 in 2024-03-18T16:03:58.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:58 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T16:03:58.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:58 smithi194 ceph-mon[36200]: from='client.? 172.21.15.196:0/3911594110' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T16:03:58.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:58 smithi039 ceph-mon[31721]: pgmap v201: 1 pgs: 1 unknown; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-03-18T16:03:58.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:58 smithi039 ceph-mon[31721]: from='client.? 172.21.15.196:0/442091658' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "836a9107-4d4f-4594-b8dc-0301e64f59d9"}]: dispatch 2024-03-18T16:03:58.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:58 smithi039 ceph-mon[31721]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "836a9107-4d4f-4594-b8dc-0301e64f59d9"}]: dispatch 2024-03-18T16:03:58.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:58 smithi039 ceph-mon[31721]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "836a9107-4d4f-4594-b8dc-0301e64f59d9"}]': finished 2024-03-18T16:03:58.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:58 smithi039 ceph-mon[31721]: osdmap e56: 9 total, 8 up, 9 in 2024-03-18T16:03:58.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:58 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T16:03:58.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:58 smithi039 ceph-mon[31721]: from='client.? 172.21.15.196:0/3911594110' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T16:03:58.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:58 smithi196 ceph-mon[36307]: pgmap v201: 1 pgs: 1 unknown; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-03-18T16:03:58.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:58 smithi196 ceph-mon[36307]: from='client.? 172.21.15.196:0/442091658' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "836a9107-4d4f-4594-b8dc-0301e64f59d9"}]: dispatch 2024-03-18T16:03:58.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:58 smithi196 ceph-mon[36307]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "836a9107-4d4f-4594-b8dc-0301e64f59d9"}]: dispatch 2024-03-18T16:03:58.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:58 smithi196 ceph-mon[36307]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "836a9107-4d4f-4594-b8dc-0301e64f59d9"}]': finished 2024-03-18T16:03:58.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:58 smithi196 ceph-mon[36307]: osdmap e56: 9 total, 8 up, 9 in 2024-03-18T16:03:58.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:58 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T16:03:58.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:58 smithi196 ceph-mon[36307]: from='client.? 172.21.15.196:0/3911594110' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T16:03:59.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:59 smithi039 ceph-mon[31721]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-03-18T16:03:59.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:03:59 smithi039 ceph-mon[31721]: Cluster is now healthy 2024-03-18T16:03:59.878 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:59 smithi196 ceph-mon[36307]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-03-18T16:03:59.879 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:03:59 smithi196 ceph-mon[36307]: Cluster is now healthy 2024-03-18T16:03:59.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:59 smithi194 ceph-mon[36200]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-03-18T16:03:59.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:03:59 smithi194 ceph-mon[36200]: Cluster is now healthy 2024-03-18T16:04:00.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:00 smithi039 ceph-mon[31721]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 246 MiB used, 715 GiB / 715 GiB avail 2024-03-18T16:04:00.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:00 smithi196 ceph-mon[36307]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 246 MiB used, 715 GiB / 715 GiB avail 2024-03-18T16:04:00.992 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:00 smithi194 ceph-mon[36200]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 246 MiB used, 715 GiB / 715 GiB avail 2024-03-18T16:04:02.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:02 smithi039 ceph-mon[31721]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-03-18T16:04:02.882 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:02 smithi196 ceph-mon[36307]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-03-18T16:04:02.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:02 smithi194 ceph-mon[36200]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 715 GiB / 715 GiB avail 2024-03-18T16:04:04.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:04 smithi196 ceph-mon[36307]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 246 MiB used, 715 GiB / 715 GiB avail 2024-03-18T16:04:04.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:04 smithi039 ceph-mon[31721]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 246 MiB used, 715 GiB / 715 GiB avail 2024-03-18T16:04:04.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:04 smithi194 ceph-mon[36200]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 246 MiB used, 715 GiB / 715 GiB avail 2024-03-18T16:04:05.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:05 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-18T16:04:05.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:05 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:04:05.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:05 smithi039 ceph-mon[31721]: Deploying daemon osd.8 on smithi196 2024-03-18T16:04:05.934 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:05 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-18T16:04:05.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:05 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:04:05.935 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:05 smithi196 ceph-mon[36307]: Deploying daemon osd.8 on smithi196 2024-03-18T16:04:05.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:05 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-18T16:04:05.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:05 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:04:05.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:05 smithi194 ceph-mon[36200]: Deploying daemon osd.8 on smithi196 2024-03-18T16:04:06.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:06 smithi039 ceph-mon[31721]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 246 MiB used, 715 GiB / 715 GiB avail 2024-03-18T16:04:06.891 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:06 smithi194 ceph-mon[36200]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 246 MiB used, 715 GiB / 715 GiB avail 2024-03-18T16:04:06.971 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:06 smithi196 ceph-mon[36307]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 246 MiB used, 715 GiB / 715 GiB avail 2024-03-18T16:04:08.835 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:08 smithi196 ceph-mon[36307]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 246 MiB used, 715 GiB / 715 GiB avail 2024-03-18T16:04:08.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:08 smithi039 ceph-mon[31721]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 246 MiB used, 715 GiB / 715 GiB avail 2024-03-18T16:04:08.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:08 smithi194 ceph-mon[36200]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 246 MiB used, 715 GiB / 715 GiB avail 2024-03-18T16:04:10.693 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:10 smithi196 ceph-mon[36307]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 246 MiB used, 715 GiB / 715 GiB avail 2024-03-18T16:04:10.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:10 smithi039 ceph-mon[31721]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 246 MiB used, 715 GiB / 715 GiB avail 2024-03-18T16:04:10.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:10 smithi194 ceph-mon[36200]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 246 MiB used, 715 GiB / 715 GiB avail 2024-03-18T16:04:12.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:12 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:04:12.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:12 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:04:12.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:12 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:04:12.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:12 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:04:12.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:12 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:04:12.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:12 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:04:12.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:12 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:04:12.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:12 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:04:12.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:12 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:04:13.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:13 smithi196 ceph-mon[36307]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 246 MiB used, 715 GiB / 715 GiB avail 2024-03-18T16:04:13.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:13 smithi039 ceph-mon[31721]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 246 MiB used, 715 GiB / 715 GiB avail 2024-03-18T16:04:13.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:13 smithi194 ceph-mon[36200]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 246 MiB used, 715 GiB / 715 GiB avail 2024-03-18T16:04:14.556 INFO:teuthology.orchestra.run.smithi196.stdout:Created osd(s) 8 on host 'smithi196' 2024-03-18T16:04:14.556 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:04:14.553+0000 7fadb088f700 1 -- 172.21.15.196:0/3189127676 <== mgr.14152 v2:172.21.15.39:6800/2749214286 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fadac10a3f0 con 0x7fad940a2890 2024-03-18T16:04:14.558 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:04:14.556+0000 7fadb1891700 1 -- 172.21.15.196:0/3189127676 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fad940a2890 msgr2=0x7fad940a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:04:14.558 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:04:14.556+0000 7fadb1891700 1 --2- 172.21.15.196:0/3189127676 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fad940a2890 0x7fad940a4d50 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fad9c006fd0 tx=0x7fad9c008040 comp rx=0 tx=0).stop 2024-03-18T16:04:14.558 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:04:14.556+0000 7fadb1891700 1 -- 172.21.15.196:0/3189127676 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fadac103ea0 msgr2=0x7fadac1a1da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:04:14.558 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:04:14.556+0000 7fadb1891700 1 --2- 172.21.15.196:0/3189127676 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fadac103ea0 0x7fadac1a1da0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fada400add0 tx=0x7fada400c330 comp rx=0 tx=0).stop 2024-03-18T16:04:14.558 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:04:14.556+0000 7fadb1891700 1 -- 172.21.15.196:0/3189127676 shutdown_connections 2024-03-18T16:04:14.558 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:04:14.556+0000 7fadb1891700 1 --2- 172.21.15.196:0/3189127676 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fad940a2890 0x7fad940a4d50 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:14.558 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:04:14.556+0000 7fadb1891700 1 --2- 172.21.15.196:0/3189127676 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fadac103ea0 0x7fadac1a1da0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:14.558 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:04:14.556+0000 7fadb1891700 1 --2- 172.21.15.196:0/3189127676 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fadac102300 0x7fadac19a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:14.558 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:04:14.556+0000 7fadb1891700 1 --2- 172.21.15.196:0/3189127676 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fadac103500 0x7fadac19ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:14.558 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:04:14.556+0000 7fadb1891700 1 -- 172.21.15.196:0/3189127676 >> 172.21.15.196:0/3189127676 conn(0x7fadac0fd8e0 msgr2=0x7fadac108cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:04:14.558 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:04:14.556+0000 7fadb1891700 1 -- 172.21.15.196:0/3189127676 shutdown_connections 2024-03-18T16:04:14.559 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:04:14.556+0000 7fadb1891700 1 -- 172.21.15.196:0/3189127676 wait complete. 2024-03-18T16:04:15.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:15 smithi196 ceph-mon[36307]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 246 MiB used, 715 GiB / 715 GiB avail 2024-03-18T16:04:15.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:15 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:04:15.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:15 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:04:15.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:15 smithi196 ceph-mon[36307]: from='osd.8 [v2:172.21.15.196:6816/2456143145,v1:172.21.15.196:6817/2456143145]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-18T16:04:15.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:15 smithi196 ceph-mon[36307]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-18T16:04:15.274 DEBUG:teuthology.orchestra.run.smithi196:osd.8> sudo journalctl -f -n 0 -u ceph-1892d8fc-e540-11ee-95c9-87774f69a715@osd.8.service 2024-03-18T16:04:15.276 INFO:tasks.cephadm:Deploying osd.9 on smithi039 with /dev/vg_nvme/lv_1... 2024-03-18T16:04:15.277 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-18T16:04:15.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:15 smithi039 ceph-mon[31721]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 246 MiB used, 715 GiB / 715 GiB avail 2024-03-18T16:04:15.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:15 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:04:15.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:15 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:04:15.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:15 smithi039 ceph-mon[31721]: from='osd.8 [v2:172.21.15.196:6816/2456143145,v1:172.21.15.196:6817/2456143145]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-18T16:04:15.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:15 smithi039 ceph-mon[31721]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-18T16:04:15.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:15 smithi194 ceph-mon[36200]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 246 MiB used, 715 GiB / 715 GiB avail 2024-03-18T16:04:15.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:15 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:04:15.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:15 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:04:15.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:15 smithi194 ceph-mon[36200]: from='osd.8 [v2:172.21.15.196:6816/2456143145,v1:172.21.15.196:6817/2456143145]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-18T16:04:15.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:15 smithi194 ceph-mon[36200]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-18T16:04:15.743 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T16:04:16.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:16 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:04:16.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:16 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:04:16.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:16 smithi039 ceph-mon[31721]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 246 MiB used, 715 GiB / 715 GiB avail 2024-03-18T16:04:16.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:16 smithi039 ceph-mon[31721]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-18T16:04:16.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:16 smithi039 ceph-mon[31721]: from='osd.8 [v2:172.21.15.196:6816/2456143145,v1:172.21.15.196:6817/2456143145]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-03-18T16:04:16.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:16 smithi039 ceph-mon[31721]: osdmap e57: 9 total, 8 up, 9 in 2024-03-18T16:04:16.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:16 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T16:04:16.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:16 smithi039 ceph-mon[31721]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-03-18T16:04:16.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:16 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:04:16.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:16 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:04:16.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:16 smithi196 ceph-mon[36307]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 246 MiB used, 715 GiB / 715 GiB avail 2024-03-18T16:04:16.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:16 smithi196 ceph-mon[36307]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-18T16:04:16.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:16 smithi196 ceph-mon[36307]: from='osd.8 [v2:172.21.15.196:6816/2456143145,v1:172.21.15.196:6817/2456143145]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-03-18T16:04:16.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:16 smithi196 ceph-mon[36307]: osdmap e57: 9 total, 8 up, 9 in 2024-03-18T16:04:16.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:16 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T16:04:16.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:16 smithi196 ceph-mon[36307]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-03-18T16:04:16.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:16 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:04:16.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:16 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:04:16.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:16 smithi194 ceph-mon[36200]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 246 MiB used, 715 GiB / 715 GiB avail 2024-03-18T16:04:16.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:16 smithi194 ceph-mon[36200]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-18T16:04:16.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:16 smithi194 ceph-mon[36200]: from='osd.8 [v2:172.21.15.196:6816/2456143145,v1:172.21.15.196:6817/2456143145]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-03-18T16:04:16.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:16 smithi194 ceph-mon[36200]: osdmap e57: 9 total, 8 up, 9 in 2024-03-18T16:04:16.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:16 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T16:04:16.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:16 smithi194 ceph-mon[36200]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-03-18T16:04:17.342 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 18 16:04:17 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-8[54576]: 2024-03-18T16:04:17.219+0000 7fb116fe0700 -1 osd.8 0 waiting for initial osdmap 2024-03-18T16:04:17.342 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 18 16:04:17 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-8[54576]: 2024-03-18T16:04:17.234+0000 7fb10e5ca700 -1 osd.8 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T16:04:17.727 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:17 smithi196 ceph-mon[36307]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-03-18T16:04:17.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:17 smithi196 ceph-mon[36307]: osdmap e58: 9 total, 8 up, 9 in 2024-03-18T16:04:17.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:17 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T16:04:17.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:17 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T16:04:17.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:17 smithi196 ceph-mon[36307]: from='osd.8 ' entity='osd.8' 2024-03-18T16:04:17.793 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:17 smithi039 ceph-mon[31721]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-03-18T16:04:17.793 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:17 smithi039 ceph-mon[31721]: osdmap e58: 9 total, 8 up, 9 in 2024-03-18T16:04:17.793 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:17 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T16:04:17.793 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:17 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T16:04:17.793 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:17 smithi039 ceph-mon[31721]: from='osd.8 ' entity='osd.8' 2024-03-18T16:04:17.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:17 smithi194 ceph-mon[36200]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-03-18T16:04:17.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:17 smithi194 ceph-mon[36200]: osdmap e58: 9 total, 8 up, 9 in 2024-03-18T16:04:17.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:17 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T16:04:17.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:17 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T16:04:17.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:17 smithi194 ceph-mon[36200]: from='osd.8 ' entity='osd.8' 2024-03-18T16:04:18.652 INFO:teuthology.orchestra.run.smithi039.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-18T16:04:18.652 INFO:teuthology.orchestra.run.smithi039.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-18T16:04:18.652 INFO:teuthology.orchestra.run.smithi039.stderr: stderr: 10+0 records in 2024-03-18T16:04:18.652 INFO:teuthology.orchestra.run.smithi039.stderr:10+0 records out 2024-03-18T16:04:18.653 INFO:teuthology.orchestra.run.smithi039.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.033845 s, 310 MB/s 2024-03-18T16:04:18.653 INFO:teuthology.orchestra.run.smithi039.stderr:--> Zapping successful for: 2024-03-18T16:04:18.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:18 smithi039 ceph-mon[31721]: purged_snaps scrub starts 2024-03-18T16:04:18.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:18 smithi039 ceph-mon[31721]: purged_snaps scrub ok 2024-03-18T16:04:18.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:18 smithi039 ceph-mon[31721]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 246 MiB used, 715 GiB / 715 GiB avail 2024-03-18T16:04:18.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:18 smithi039 ceph-mon[31721]: osd.8 [v2:172.21.15.196:6816/2456143145,v1:172.21.15.196:6817/2456143145] boot 2024-03-18T16:04:18.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:18 smithi039 ceph-mon[31721]: osdmap e59: 9 total, 9 up, 9 in 2024-03-18T16:04:18.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:18 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T16:04:18.838 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:18 smithi039 ceph-mon[31721]: osdmap e60: 9 total, 9 up, 9 in 2024-03-18T16:04:18.847 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:18 smithi196 ceph-mon[36307]: purged_snaps scrub starts 2024-03-18T16:04:18.847 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:18 smithi196 ceph-mon[36307]: purged_snaps scrub ok 2024-03-18T16:04:18.847 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:18 smithi196 ceph-mon[36307]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 246 MiB used, 715 GiB / 715 GiB avail 2024-03-18T16:04:18.847 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:18 smithi196 ceph-mon[36307]: osd.8 [v2:172.21.15.196:6816/2456143145,v1:172.21.15.196:6817/2456143145] boot 2024-03-18T16:04:18.847 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:18 smithi196 ceph-mon[36307]: osdmap e59: 9 total, 9 up, 9 in 2024-03-18T16:04:18.847 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:18 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T16:04:18.847 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:18 smithi196 ceph-mon[36307]: osdmap e60: 9 total, 9 up, 9 in 2024-03-18T16:04:18.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:18 smithi194 ceph-mon[36200]: purged_snaps scrub starts 2024-03-18T16:04:18.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:18 smithi194 ceph-mon[36200]: purged_snaps scrub ok 2024-03-18T16:04:18.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:18 smithi194 ceph-mon[36200]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 246 MiB used, 715 GiB / 715 GiB avail 2024-03-18T16:04:18.991 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:18 smithi194 ceph-mon[36200]: osd.8 [v2:172.21.15.196:6816/2456143145,v1:172.21.15.196:6817/2456143145] boot 2024-03-18T16:04:18.991 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:18 smithi194 ceph-mon[36200]: osdmap e59: 9 total, 9 up, 9 in 2024-03-18T16:04:18.991 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:18 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-18T16:04:18.991 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:18 smithi194 ceph-mon[36200]: osdmap e60: 9 total, 9 up, 9 in 2024-03-18T16:04:19.604 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi039:vg_nvme/lv_1 2024-03-18T16:04:20.068 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T16:04:20.219 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:19 smithi039 ceph-mon[31721]: Detected new or changed devices on smithi196 2024-03-18T16:04:20.219 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:19 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:04:20.219 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:19 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:04:20.219 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:19 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:04:20.219 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:19 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:04:20.219 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:19 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:04:20.219 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:19 smithi039 ceph-mon[31721]: Adjusting osd_memory_target on smithi196 to 5693M 2024-03-18T16:04:20.220 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:19 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:04:20.220 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:19 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:04:20.220 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:19 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:04:20.220 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:19 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:04:20.220 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:19 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:04:20.220 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:19 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:04:20.220 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:19 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:04:20.220 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:19 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:04:20.220 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:19 smithi039 ceph-mon[31721]: osdmap e61: 9 total, 9 up, 9 in 2024-03-18T16:04:20.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:19 smithi196 ceph-mon[36307]: Detected new or changed devices on smithi196 2024-03-18T16:04:20.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:19 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:04:20.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:19 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:04:20.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:19 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:04:20.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:19 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:04:20.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:19 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:04:20.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:19 smithi196 ceph-mon[36307]: Adjusting osd_memory_target on smithi196 to 5693M 2024-03-18T16:04:20.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:19 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:04:20.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:19 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:04:20.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:19 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:04:20.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:19 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:04:20.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:19 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:04:20.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:19 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:04:20.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:19 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:04:20.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:19 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:04:20.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:19 smithi196 ceph-mon[36307]: osdmap e61: 9 total, 9 up, 9 in 2024-03-18T16:04:20.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:19 smithi194 ceph-mon[36200]: Detected new or changed devices on smithi196 2024-03-18T16:04:20.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:19 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:04:20.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:19 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:04:20.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:19 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:04:20.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:19 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:04:20.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:19 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:04:20.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:19 smithi194 ceph-mon[36200]: Adjusting osd_memory_target on smithi196 to 5693M 2024-03-18T16:04:20.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:19 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:04:20.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:19 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:04:20.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:19 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:04:20.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:19 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:04:20.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:19 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:04:20.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:19 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:04:20.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:19 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:04:20.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:19 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:04:20.242 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:19 smithi194 ceph-mon[36200]: osdmap e61: 9 total, 9 up, 9 in 2024-03-18T16:04:21.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:20 smithi196 ceph-mon[36307]: pgmap v217: 1 pgs: 1 peering; 577 KiB data, 616 MiB used, 804 GiB / 805 GiB avail 2024-03-18T16:04:21.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:20 smithi196 ceph-mon[36307]: osdmap e62: 9 total, 9 up, 9 in 2024-03-18T16:04:21.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:20 smithi194 ceph-mon[36200]: pgmap v217: 1 pgs: 1 peering; 577 KiB data, 616 MiB used, 804 GiB / 805 GiB avail 2024-03-18T16:04:21.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:20 smithi194 ceph-mon[36200]: osdmap e62: 9 total, 9 up, 9 in 2024-03-18T16:04:21.262 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:20 smithi039 ceph-mon[31721]: pgmap v217: 1 pgs: 1 peering; 577 KiB data, 616 MiB used, 804 GiB / 805 GiB avail 2024-03-18T16:04:21.262 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:20 smithi039 ceph-mon[31721]: osdmap e62: 9 total, 9 up, 9 in 2024-03-18T16:04:22.123 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:21 smithi039 ceph-mon[31721]: osdmap e63: 9 total, 9 up, 9 in 2024-03-18T16:04:22.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:21 smithi196 ceph-mon[36307]: osdmap e63: 9 total, 9 up, 9 in 2024-03-18T16:04:22.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:21 smithi194 ceph-mon[36200]: osdmap e63: 9 total, 9 up, 9 in 2024-03-18T16:04:22.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:04:22.402+0000 7f8a29a00700 1 -- 172.21.15.39:0/220588713 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8a24102da0 msgr2=0x7f8a24105190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:04:22.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:04:22.402+0000 7f8a29a00700 1 --2- 172.21.15.39:0/220588713 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8a24102da0 0x7f8a24105190 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f8a14009990 tx=0x7f8a14009ca0 comp rx=0 tx=0).stop 2024-03-18T16:04:22.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:04:22.402+0000 7f8a29a00700 1 -- 172.21.15.39:0/220588713 shutdown_connections 2024-03-18T16:04:22.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:04:22.402+0000 7f8a29a00700 1 --2- 172.21.15.39:0/220588713 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8a24108000 0x7f8a2410a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:22.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:04:22.402+0000 7f8a29a00700 1 --2- 172.21.15.39:0/220588713 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8a24102da0 0x7f8a24105190 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:22.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:04:22.403+0000 7f8a29a00700 1 --2- 172.21.15.39:0/220588713 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a241056d0 0x7f8a24107ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:22.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:04:22.403+0000 7f8a29a00700 1 -- 172.21.15.39:0/220588713 >> 172.21.15.39:0/220588713 conn(0x7f8a240fc7d0 msgr2=0x7f8a240febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:04:22.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:04:22.403+0000 7f8a29a00700 1 -- 172.21.15.39:0/220588713 shutdown_connections 2024-03-18T16:04:22.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:04:22.403+0000 7f8a29a00700 1 -- 172.21.15.39:0/220588713 wait complete. 2024-03-18T16:04:22.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:04:22.404+0000 7f8a29a00700 1 Processor -- start 2024-03-18T16:04:22.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:04:22.404+0000 7f8a29a00700 1 -- start start 2024-03-18T16:04:22.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:04:22.405+0000 7f8a29a00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8a24102da0 0x7f8a24102700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:22.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:04:22.405+0000 7f8a29a00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8a241056d0 0x7f8a24100d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:22.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:04:22.405+0000 7f8a29a00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a24108000 0x7f8a24101290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:22.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:04:22.405+0000 7f8a29a00700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f8a2410df70 con 0x7f8a24108000 2024-03-18T16:04:22.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:04:22.405+0000 7f8a29a00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f8a2410ddf0 con 0x7f8a241056d0 2024-03-18T16:04:22.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:04:22.405+0000 7f8a29a00700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f8a2410e0f0 con 0x7f8a24102da0 2024-03-18T16:04:22.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:04:22.405+0000 7f8a22a00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8a24102da0 0x7f8a24102700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:22.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:04:22.405+0000 7f8a23400700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a24108000 0x7f8a24101290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:22.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:04:22.405+0000 7f8a22a00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8a24102da0 0x7f8a24102700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:46926/0 (socket says 172.21.15.39:46926) 2024-03-18T16:04:22.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:04:22.405+0000 7f8a22000700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8a241056d0 0x7f8a24100d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:22.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:04:22.405+0000 7f8a22a00700 1 -- 172.21.15.39:0/2153710267 learned_addr learned my addr 172.21.15.39:0/2153710267 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:04:22.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:04:22.406+0000 7f8a22a00700 1 -- 172.21.15.39:0/2153710267 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8a241056d0 msgr2=0x7f8a24100d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:04:22.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:04:22.406+0000 7f8a22a00700 1 --2- 172.21.15.39:0/2153710267 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8a241056d0 0x7f8a24100d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:22.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:04:22.406+0000 7f8a22a00700 1 -- 172.21.15.39:0/2153710267 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a24108000 msgr2=0x7f8a24101290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:04:22.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:04:22.406+0000 7f8a22a00700 1 --2- 172.21.15.39:0/2153710267 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a24108000 0x7f8a24101290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:22.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:04:22.406+0000 7f8a22a00700 1 -- 172.21.15.39:0/2153710267 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a24101b40 con 0x7f8a24102da0 2024-03-18T16:04:22.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:04:22.406+0000 7f8a22000700 1 --2- 172.21.15.39:0/2153710267 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8a241056d0 0x7f8a24100d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:04:22.409 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:04:22.406+0000 7f8a22a00700 1 --2- 172.21.15.39:0/2153710267 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8a24102da0 0x7f8a24102700 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f8a14005910 tx=0x7f8a1400bbb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:04:22.409 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:04:22.406+0000 7f8a1b400700 1 -- 172.21.15.39:0/2153710267 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a14026070 con 0x7f8a24102da0 2024-03-18T16:04:22.409 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:04:22.406+0000 7f8a1b400700 1 -- 172.21.15.39:0/2153710267 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a14003d70 con 0x7f8a24102da0 2024-03-18T16:04:22.409 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:04:22.406+0000 7f8a29a00700 1 -- 172.21.15.39:0/2153710267 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a241a26f0 con 0x7f8a24102da0 2024-03-18T16:04:22.409 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:04:22.406+0000 7f8a1b400700 1 -- 172.21.15.39:0/2153710267 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a14021410 con 0x7f8a24102da0 2024-03-18T16:04:22.409 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:04:22.407+0000 7f8a29a00700 1 -- 172.21.15.39:0/2153710267 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a241a2b40 con 0x7f8a24102da0 2024-03-18T16:04:22.409 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:04:22.408+0000 7f8a29a00700 1 -- 172.21.15.39:0/2153710267 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a24068f90 con 0x7f8a24102da0 2024-03-18T16:04:22.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:04:22.410+0000 7f8a1b400700 1 -- 172.21.15.39:0/2153710267 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f8a14029070 con 0x7f8a24102da0 2024-03-18T16:04:22.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:04:22.411+0000 7f8a1b400700 1 --2- 172.21.15.39:0/2153710267 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f8a080a2910 0x7f8a080a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:22.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:04:22.411+0000 7f8a1b400700 1 -- 172.21.15.39:0/2153710267 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(63..63 src has 1..63) ==== 5829+0+0 (secure 0 0 0) 0x7f8a140cedb0 con 0x7f8a24102da0 2024-03-18T16:04:22.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:04:22.411+0000 7f8a22000700 1 --2- 172.21.15.39:0/2153710267 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f8a080a2910 0x7f8a080a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:22.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:04:22.412+0000 7f8a22000700 1 --2- 172.21.15.39:0/2153710267 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f8a080a2910 0x7f8a080a4dd0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f8a10011490 tx=0x7f8a10009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:04:22.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:04:22.415+0000 7f8a1b400700 1 -- 172.21.15.39:0/2153710267 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8a140d3070 con 0x7f8a24102da0 2024-03-18T16:04:22.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:04:22.653+0000 7f8a29a00700 1 -- 172.21.15.39:0/2153710267 --> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi039:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f8a24062710 con 0x7f8a080a2910 2024-03-18T16:04:23.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:22 smithi196 ceph-mon[36307]: pgmap v221: 1 pgs: 1 peering; 577 KiB data, 616 MiB used, 804 GiB / 805 GiB avail 2024-03-18T16:04:23.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:22 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T16:04:23.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:22 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T16:04:23.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:22 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:04:23.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:22 smithi194 ceph-mon[36200]: pgmap v221: 1 pgs: 1 peering; 577 KiB data, 616 MiB used, 804 GiB / 805 GiB avail 2024-03-18T16:04:23.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:22 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T16:04:23.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:22 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T16:04:23.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:22 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:04:23.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:22 smithi039 ceph-mon[31721]: pgmap v221: 1 pgs: 1 peering; 577 KiB data, 616 MiB used, 804 GiB / 805 GiB avail 2024-03-18T16:04:23.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:22 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T16:04:23.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:22 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T16:04:23.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:22 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:04:24.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:23 smithi196 ceph-mon[36307]: from='client.24343 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi039:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:04:24.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:23 smithi194 ceph-mon[36200]: from='client.24343 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi039:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:04:24.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:23 smithi039 ceph-mon[31721]: from='client.24343 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi039:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:04:25.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:24 smithi039 ceph-mon[31721]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 804 GiB / 805 GiB avail; 118 KiB/s, 0 objects/s recovering 2024-03-18T16:04:25.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:24 smithi196 ceph-mon[36307]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 804 GiB / 805 GiB avail; 118 KiB/s, 0 objects/s recovering 2024-03-18T16:04:25.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:24 smithi194 ceph-mon[36200]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 278 MiB used, 804 GiB / 805 GiB avail; 118 KiB/s, 0 objects/s recovering 2024-03-18T16:04:27.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:26 smithi196 ceph-mon[36307]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-18T16:04:27.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:26 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/1295402239' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4f30b960-12e3-4b16-bd28-fcc988139b58"}]: dispatch 2024-03-18T16:04:27.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:26 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/1295402239' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4f30b960-12e3-4b16-bd28-fcc988139b58"}]': finished 2024-03-18T16:04:27.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:26 smithi196 ceph-mon[36307]: osdmap e64: 10 total, 9 up, 10 in 2024-03-18T16:04:27.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:26 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T16:04:27.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:26 smithi194 ceph-mon[36200]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-18T16:04:27.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:26 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/1295402239' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4f30b960-12e3-4b16-bd28-fcc988139b58"}]: dispatch 2024-03-18T16:04:27.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:26 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/1295402239' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4f30b960-12e3-4b16-bd28-fcc988139b58"}]': finished 2024-03-18T16:04:27.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:26 smithi194 ceph-mon[36200]: osdmap e64: 10 total, 9 up, 10 in 2024-03-18T16:04:27.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:26 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T16:04:27.290 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:26 smithi039 ceph-mon[31721]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-18T16:04:27.290 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:26 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/1295402239' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4f30b960-12e3-4b16-bd28-fcc988139b58"}]: dispatch 2024-03-18T16:04:27.290 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:26 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/1295402239' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4f30b960-12e3-4b16-bd28-fcc988139b58"}]': finished 2024-03-18T16:04:27.291 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:26 smithi039 ceph-mon[31721]: osdmap e64: 10 total, 9 up, 10 in 2024-03-18T16:04:27.291 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:26 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T16:04:28.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:27 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/2485213683' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T16:04:28.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:27 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/2485213683' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T16:04:28.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:27 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/2485213683' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T16:04:29.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:28 smithi196 ceph-mon[36307]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-03-18T16:04:29.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:28 smithi194 ceph-mon[36200]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-03-18T16:04:29.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:28 smithi039 ceph-mon[31721]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-03-18T16:04:31.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:30 smithi196 ceph-mon[36307]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-03-18T16:04:31.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:30 smithi194 ceph-mon[36200]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-03-18T16:04:31.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:30 smithi039 ceph-mon[31721]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-03-18T16:04:33.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:32 smithi194 ceph-mon[36200]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-18T16:04:33.260 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:32 smithi039 ceph-mon[31721]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-18T16:04:33.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:32 smithi196 ceph-mon[36307]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-18T16:04:35.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:34 smithi039 ceph-mon[31721]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-03-18T16:04:35.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:34 smithi196 ceph-mon[36307]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-03-18T16:04:35.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:34 smithi194 ceph-mon[36200]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-03-18T16:04:37.292 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:36 smithi194 ceph-mon[36200]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-03-18T16:04:37.292 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:36 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-18T16:04:37.293 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:36 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:04:37.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:36 smithi039 ceph-mon[31721]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-03-18T16:04:37.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:36 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-18T16:04:37.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:36 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:04:37.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:37 smithi196 ceph-mon[36307]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-03-18T16:04:37.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:37 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-18T16:04:37.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:37 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:04:38.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:38 smithi039 ceph-mon[31721]: Deploying daemon osd.9 on smithi039 2024-03-18T16:04:38.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:38 smithi196 ceph-mon[36307]: Deploying daemon osd.9 on smithi039 2024-03-18T16:04:38.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:38 smithi194 ceph-mon[36200]: Deploying daemon osd.9 on smithi039 2024-03-18T16:04:39.272 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:39 smithi039 ceph-mon[31721]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-03-18T16:04:39.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:39 smithi196 ceph-mon[36307]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-03-18T16:04:39.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:39 smithi194 ceph-mon[36200]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-03-18T16:04:41.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:41 smithi039 ceph-mon[31721]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-03-18T16:04:41.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:41 smithi196 ceph-mon[36307]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-03-18T16:04:41.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:41 smithi194 ceph-mon[36200]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-03-18T16:04:43.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:43 smithi039 ceph-mon[31721]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-03-18T16:04:43.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:43 smithi196 ceph-mon[36307]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-03-18T16:04:43.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:43 smithi194 ceph-mon[36200]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-03-18T16:04:45.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:45 smithi039 ceph-mon[31721]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-03-18T16:04:45.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:45 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:04:45.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:45 smithi196 ceph-mon[36307]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-03-18T16:04:45.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:45 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:04:45.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:45 smithi194 ceph-mon[36200]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-03-18T16:04:45.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:45 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:04:46.287 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:46 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:04:46.288 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:46 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:04:46.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:46 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:04:46.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:46 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:04:46.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:46 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:04:46.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:46 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:04:47.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:47 smithi039 ceph-mon[31721]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-03-18T16:04:47.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:47 smithi196 ceph-mon[36307]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-03-18T16:04:47.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:47 smithi194 ceph-mon[36200]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-03-18T16:04:49.256 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:49 smithi039 ceph-mon[31721]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-03-18T16:04:49.256 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:49 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:04:49.256 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:49 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:04:49.256 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:49 smithi039 ceph-mon[31721]: from='osd.9 [v2:172.21.15.39:6826/1848634433,v1:172.21.15.39:6827/1848634433]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-18T16:04:49.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:49 smithi196 ceph-mon[36307]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-03-18T16:04:49.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:49 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:04:49.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:49 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:04:49.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:49 smithi196 ceph-mon[36307]: from='osd.9 [v2:172.21.15.39:6826/1848634433,v1:172.21.15.39:6827/1848634433]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-18T16:04:49.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:49 smithi194 ceph-mon[36200]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-03-18T16:04:49.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:49 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:04:49.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:49 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:04:49.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:49 smithi194 ceph-mon[36200]: from='osd.9 [v2:172.21.15.39:6826/1848634433,v1:172.21.15.39:6827/1848634433]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-18T16:04:50.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:04:50.205+0000 7f8a1b400700 1 -- 172.21.15.39:0/2153710267 <== mgr.14152 v2:172.21.15.39:6800/2749214286 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f8a24062710 con 0x7f8a080a2910 2024-03-18T16:04:50.207 INFO:teuthology.orchestra.run.smithi039.stdout:Created osd(s) 9 on host 'smithi039' 2024-03-18T16:04:50.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:04:50.209+0000 7f8a29a00700 1 -- 172.21.15.39:0/2153710267 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f8a080a2910 msgr2=0x7f8a080a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:04:50.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:04:50.209+0000 7f8a29a00700 1 --2- 172.21.15.39:0/2153710267 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f8a080a2910 0x7f8a080a4dd0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f8a10011490 tx=0x7f8a10009260 comp rx=0 tx=0).stop 2024-03-18T16:04:50.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:04:50.209+0000 7f8a29a00700 1 -- 172.21.15.39:0/2153710267 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8a24102da0 msgr2=0x7f8a24102700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:04:50.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:04:50.209+0000 7f8a29a00700 1 --2- 172.21.15.39:0/2153710267 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8a24102da0 0x7f8a24102700 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f8a14005910 tx=0x7f8a1400bbb0 comp rx=0 tx=0).stop 2024-03-18T16:04:50.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:04:50.209+0000 7f8a29a00700 1 -- 172.21.15.39:0/2153710267 shutdown_connections 2024-03-18T16:04:50.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:04:50.209+0000 7f8a29a00700 1 --2- 172.21.15.39:0/2153710267 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f8a080a2910 0x7f8a080a4dd0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:50.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:04:50.209+0000 7f8a29a00700 1 --2- 172.21.15.39:0/2153710267 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a24108000 0x7f8a24101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:50.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:04:50.209+0000 7f8a29a00700 1 --2- 172.21.15.39:0/2153710267 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8a24102da0 0x7f8a24102700 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:50.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:04:50.209+0000 7f8a29a00700 1 --2- 172.21.15.39:0/2153710267 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8a241056d0 0x7f8a24100d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:50.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:04:50.209+0000 7f8a29a00700 1 -- 172.21.15.39:0/2153710267 >> 172.21.15.39:0/2153710267 conn(0x7f8a240fc7d0 msgr2=0x7f8a24103840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:04:50.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:04:50.209+0000 7f8a29a00700 1 -- 172.21.15.39:0/2153710267 shutdown_connections 2024-03-18T16:04:50.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:04:50.209+0000 7f8a29a00700 1 -- 172.21.15.39:0/2153710267 wait complete. 2024-03-18T16:04:50.851 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:50 smithi039 ceph-mon[31721]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-03-18T16:04:50.852 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:50 smithi039 ceph-mon[31721]: from='osd.9 [v2:172.21.15.39:6826/1848634433,v1:172.21.15.39:6827/1848634433]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-18T16:04:50.852 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:50 smithi039 ceph-mon[31721]: osdmap e65: 10 total, 9 up, 10 in 2024-03-18T16:04:50.852 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:50 smithi039 ceph-mon[31721]: from='osd.9 [v2:172.21.15.39:6826/1848634433,v1:172.21.15.39:6827/1848634433]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]: dispatch 2024-03-18T16:04:50.852 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:50 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T16:04:50.852 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:50 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:04:50.852 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:50 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:04:50.852 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:50 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:04:50.852 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:50 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:04:50.852 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:50 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:04:50.852 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:50 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:04:51.007 DEBUG:teuthology.orchestra.run.smithi039:osd.9> sudo journalctl -f -n 0 -u ceph-1892d8fc-e540-11ee-95c9-87774f69a715@osd.9.service 2024-03-18T16:04:51.012 INFO:tasks.cephadm:Deploying osd.10 on smithi194 with /dev/vg_nvme/lv_1... 2024-03-18T16:04:51.013 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-18T16:04:51.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:50 smithi196 ceph-mon[36307]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-03-18T16:04:51.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:50 smithi196 ceph-mon[36307]: from='osd.9 [v2:172.21.15.39:6826/1848634433,v1:172.21.15.39:6827/1848634433]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-18T16:04:51.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:50 smithi196 ceph-mon[36307]: osdmap e65: 10 total, 9 up, 10 in 2024-03-18T16:04:51.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:50 smithi196 ceph-mon[36307]: from='osd.9 [v2:172.21.15.39:6826/1848634433,v1:172.21.15.39:6827/1848634433]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]: dispatch 2024-03-18T16:04:51.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:50 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T16:04:51.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:50 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:04:51.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:50 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:04:51.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:50 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:04:51.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:50 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:04:51.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:50 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:04:51.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:50 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:04:51.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:50 smithi194 ceph-mon[36200]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-03-18T16:04:51.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:50 smithi194 ceph-mon[36200]: from='osd.9 [v2:172.21.15.39:6826/1848634433,v1:172.21.15.39:6827/1848634433]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-18T16:04:51.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:50 smithi194 ceph-mon[36200]: osdmap e65: 10 total, 9 up, 10 in 2024-03-18T16:04:51.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:50 smithi194 ceph-mon[36200]: from='osd.9 [v2:172.21.15.39:6826/1848634433,v1:172.21.15.39:6827/1848634433]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]: dispatch 2024-03-18T16:04:51.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:50 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T16:04:51.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:50 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:04:51.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:50 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:04:51.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:50 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:04:51.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:50 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:04:51.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:50 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:04:51.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:50 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:04:51.261 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T16:04:51.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:51 smithi194 ceph-mon[36200]: from='osd.9 [v2:172.21.15.39:6826/1848634433,v1:172.21.15.39:6827/1848634433]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]': finished 2024-03-18T16:04:51.991 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:51 smithi194 ceph-mon[36200]: osdmap e66: 10 total, 9 up, 10 in 2024-03-18T16:04:51.991 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:51 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T16:04:51.991 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:51 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T16:04:52.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:51 smithi039 ceph-mon[31721]: from='osd.9 [v2:172.21.15.39:6826/1848634433,v1:172.21.15.39:6827/1848634433]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]': finished 2024-03-18T16:04:52.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:51 smithi039 ceph-mon[31721]: osdmap e66: 10 total, 9 up, 10 in 2024-03-18T16:04:52.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:51 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T16:04:52.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:51 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T16:04:52.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:51 smithi196 ceph-mon[36307]: from='osd.9 [v2:172.21.15.39:6826/1848634433,v1:172.21.15.39:6827/1848634433]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi039", "root=default"]}]': finished 2024-03-18T16:04:52.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:51 smithi196 ceph-mon[36307]: osdmap e66: 10 total, 9 up, 10 in 2024-03-18T16:04:52.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:51 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T16:04:52.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:51 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T16:04:52.986 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:52 smithi194 ceph-mon[36200]: purged_snaps scrub starts 2024-03-18T16:04:52.986 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:52 smithi194 ceph-mon[36200]: purged_snaps scrub ok 2024-03-18T16:04:52.986 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:52 smithi194 ceph-mon[36200]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-03-18T16:04:52.986 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:52 smithi194 ceph-mon[36200]: osd.9 [v2:172.21.15.39:6826/1848634433,v1:172.21.15.39:6827/1848634433] boot 2024-03-18T16:04:52.986 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:52 smithi194 ceph-mon[36200]: osdmap e67: 10 total, 10 up, 10 in 2024-03-18T16:04:52.986 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:52 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T16:04:53.008 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:52 smithi039 ceph-mon[31721]: purged_snaps scrub starts 2024-03-18T16:04:53.008 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:52 smithi039 ceph-mon[31721]: purged_snaps scrub ok 2024-03-18T16:04:53.008 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:52 smithi039 ceph-mon[31721]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-03-18T16:04:53.008 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:52 smithi039 ceph-mon[31721]: osd.9 [v2:172.21.15.39:6826/1848634433,v1:172.21.15.39:6827/1848634433] boot 2024-03-18T16:04:53.008 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:52 smithi039 ceph-mon[31721]: osdmap e67: 10 total, 10 up, 10 in 2024-03-18T16:04:53.008 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:52 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T16:04:53.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:52 smithi196 ceph-mon[36307]: purged_snaps scrub starts 2024-03-18T16:04:53.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:52 smithi196 ceph-mon[36307]: purged_snaps scrub ok 2024-03-18T16:04:53.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:52 smithi196 ceph-mon[36307]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 277 MiB used, 804 GiB / 805 GiB avail 2024-03-18T16:04:53.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:52 smithi196 ceph-mon[36307]: osd.9 [v2:172.21.15.39:6826/1848634433,v1:172.21.15.39:6827/1848634433] boot 2024-03-18T16:04:53.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:52 smithi196 ceph-mon[36307]: osdmap e67: 10 total, 10 up, 10 in 2024-03-18T16:04:53.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:52 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-18T16:04:53.333 INFO:teuthology.orchestra.run.smithi194.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-18T16:04:53.333 INFO:teuthology.orchestra.run.smithi194.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-18T16:04:53.333 INFO:teuthology.orchestra.run.smithi194.stderr: stderr: 10+0 records in 2024-03-18T16:04:53.333 INFO:teuthology.orchestra.run.smithi194.stderr:10+0 records out 2024-03-18T16:04:53.334 INFO:teuthology.orchestra.run.smithi194.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113962 s, 920 MB/s 2024-03-18T16:04:53.334 INFO:teuthology.orchestra.run.smithi194.stderr:--> Zapping successful for: 2024-03-18T16:04:53.973 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:53 smithi194 ceph-mon[36200]: osdmap e68: 10 total, 10 up, 10 in 2024-03-18T16:04:54.001 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi194:vg_nvme/lv_1 2024-03-18T16:04:54.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:53 smithi196 ceph-mon[36307]: osdmap e68: 10 total, 10 up, 10 in 2024-03-18T16:04:54.244 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T16:04:54.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:53 smithi039 ceph-mon[31721]: osdmap e68: 10 total, 10 up, 10 in 2024-03-18T16:04:55.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:54 smithi039 ceph-mon[31721]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-03-18T16:04:55.227 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:54 smithi196 ceph-mon[36307]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-03-18T16:04:55.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:54 smithi194 ceph-mon[36200]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-03-18T16:04:55.878 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:04:55.876+0000 7f1f7d6b3700 1 -- 172.21.15.194:0/354009111 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1f78107fe0 msgr2=0x7f1f7810a450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:04:55.879 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:04:55.876+0000 7f1f7d6b3700 1 --2- 172.21.15.194:0/354009111 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1f78107fe0 0x7f1f7810a450 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f1f7000b0a0 tx=0x7f1f7000b3b0 comp rx=0 tx=0).stop 2024-03-18T16:04:55.879 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:04:55.876+0000 7f1f7d6b3700 1 -- 172.21.15.194:0/354009111 shutdown_connections 2024-03-18T16:04:55.879 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:04:55.876+0000 7f1f7d6b3700 1 --2- 172.21.15.194:0/354009111 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1f78107fe0 0x7f1f7810a450 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:55.879 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:04:55.876+0000 7f1f7d6b3700 1 --2- 172.21.15.194:0/354009111 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f78102d80 0x7f1f78105170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:55.879 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:04:55.876+0000 7f1f7d6b3700 1 --2- 172.21.15.194:0/354009111 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1f781056b0 0x7f1f78107aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:55.879 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:04:55.876+0000 7f1f7d6b3700 1 -- 172.21.15.194:0/354009111 >> 172.21.15.194:0/354009111 conn(0x7f1f780fc790 msgr2=0x7f1f780febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:04:55.879 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:04:55.876+0000 7f1f7d6b3700 1 -- 172.21.15.194:0/354009111 shutdown_connections 2024-03-18T16:04:55.879 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:04:55.876+0000 7f1f7d6b3700 1 -- 172.21.15.194:0/354009111 wait complete. 2024-03-18T16:04:55.881 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:04:55.877+0000 7f1f7d6b3700 1 Processor -- start 2024-03-18T16:04:55.881 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:04:55.877+0000 7f1f7d6b3700 1 -- start start 2024-03-18T16:04:55.881 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:04:55.878+0000 7f1f7d6b3700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1f78102d80 0x7f1f781026e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:55.881 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:04:55.878+0000 7f1f7d6b3700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f781056b0 0x7f1f78100d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:55.881 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:04:55.878+0000 7f1f7d6b3700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1f78107fe0 0x7f1f78101270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:55.881 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:04:55.878+0000 7f1f7d6b3700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f1f7810df40 con 0x7f1f781056b0 2024-03-18T16:04:55.881 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:04:55.878+0000 7f1f7d6b3700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f1f7810ddc0 con 0x7f1f78107fe0 2024-03-18T16:04:55.881 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:04:55.878+0000 7f1f7d6b3700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f1f7810e0c0 con 0x7f1f78102d80 2024-03-18T16:04:55.881 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:04:55.878+0000 7f1f777fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1f78107fe0 0x7f1f78101270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:55.881 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:04:55.878+0000 7f1f777fe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1f78107fe0 0x7f1f78101270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:56774/0 (socket says 172.21.15.194:56774) 2024-03-18T16:04:55.882 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:04:55.878+0000 7f1f76ffd700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1f78102d80 0x7f1f781026e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:55.882 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:04:55.878+0000 7f1f777fe700 1 -- 172.21.15.194:0/1715104517 learned_addr learned my addr 172.21.15.194:0/1715104517 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-03-18T16:04:55.882 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:04:55.878+0000 7f1f777fe700 1 -- 172.21.15.194:0/1715104517 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1f78102d80 msgr2=0x7f1f781026e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:04:55.882 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:04:55.878+0000 7f1f777fe700 1 --2- 172.21.15.194:0/1715104517 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1f78102d80 0x7f1f781026e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:55.882 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:04:55.878+0000 7f1f777fe700 1 -- 172.21.15.194:0/1715104517 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f781056b0 msgr2=0x7f1f78100d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:04:55.882 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:04:55.878+0000 7f1f777fe700 1 --2- 172.21.15.194:0/1715104517 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f781056b0 0x7f1f78100d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:04:55.882 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:04:55.878+0000 7f1f777fe700 1 -- 172.21.15.194:0/1715104517 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f78101b20 con 0x7f1f78107fe0 2024-03-18T16:04:55.882 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:04:55.878+0000 7f1f777fe700 1 --2- 172.21.15.194:0/1715104517 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1f78107fe0 0x7f1f78101270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1f70007f00 tx=0x7f1f70004e40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:04:55.882 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:04:55.879+0000 7f1f5ffff700 1 -- 172.21.15.194:0/1715104517 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f70021070 con 0x7f1f78107fe0 2024-03-18T16:04:55.882 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:04:55.879+0000 7f1f5ffff700 1 -- 172.21.15.194:0/1715104517 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f7001d930 con 0x7f1f78107fe0 2024-03-18T16:04:55.882 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:04:55.879+0000 7f1f7d6b3700 1 -- 172.21.15.194:0/1715104517 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f781a2680 con 0x7f1f78107fe0 2024-03-18T16:04:55.882 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:04:55.879+0000 7f1f5ffff700 1 -- 172.21.15.194:0/1715104517 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f700055a0 con 0x7f1f78107fe0 2024-03-18T16:04:55.882 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:04:55.879+0000 7f1f7d6b3700 1 -- 172.21.15.194:0/1715104517 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f781a2b30 con 0x7f1f78107fe0 2024-03-18T16:04:55.883 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:04:55.880+0000 7f1f7d6b3700 1 -- 172.21.15.194:0/1715104517 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f78068f90 con 0x7f1f78107fe0 2024-03-18T16:04:55.883 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:04:55.882+0000 7f1f5ffff700 1 -- 172.21.15.194:0/1715104517 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f1f70024070 con 0x7f1f78107fe0 2024-03-18T16:04:55.884 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:04:55.882+0000 7f1f5ffff700 1 --2- 172.21.15.194:0/1715104517 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1f600a2910 0x7f1f600a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:04:55.884 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:04:55.882+0000 7f1f5ffff700 1 -- 172.21.15.194:0/1715104517 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 6261+0+0 (secure 0 0 0) 0x7f1f7000f020 con 0x7f1f78107fe0 2024-03-18T16:04:55.884 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:04:55.883+0000 7f1f76ffd700 1 --2- 172.21.15.194:0/1715104517 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1f600a2910 0x7f1f600a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:04:55.885 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:04:55.884+0000 7f1f76ffd700 1 --2- 172.21.15.194:0/1715104517 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1f600a2910 0x7f1f600a4dd0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f1f68009a10 tx=0x7f1f68009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:04:55.887 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:04:55.885+0000 7f1f5ffff700 1 -- 172.21.15.194:0/1715104517 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1f7002b020 con 0x7f1f78107fe0 2024-03-18T16:04:56.092 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:04:56.090+0000 7f1f7d6b3700 1 -- 172.21.15.194:0/1715104517 --> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f1f78062710 con 0x7f1f600a2910 2024-03-18T16:04:57.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:57 smithi194 ceph-mon[36200]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-18T16:04:57.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:57 smithi194 ceph-mon[36200]: Detected new or changed devices on smithi039 2024-03-18T16:04:57.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:57 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:04:57.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:57 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:04:57.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:57 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:04:57.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:57 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:04:57.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:57 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:04:57.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:57 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:04:57.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:57 smithi194 ceph-mon[36200]: Adjusting osd_memory_target on smithi039 to 4272M 2024-03-18T16:04:57.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:57 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:04:57.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:57 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:04:57.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:57 smithi194 ceph-mon[36200]: from='client.24350 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:04:57.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:57 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:04:57.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:57 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T16:04:57.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:57 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T16:04:57.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:57 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:04:57.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:57 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:04:57.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:57 smithi039 ceph-mon[31721]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-18T16:04:57.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:57 smithi039 ceph-mon[31721]: Detected new or changed devices on smithi039 2024-03-18T16:04:57.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:57 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:04:57.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:57 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:04:57.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:57 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:04:57.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:57 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:04:57.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:57 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:04:57.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:57 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:04:57.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:57 smithi039 ceph-mon[31721]: Adjusting osd_memory_target on smithi039 to 4272M 2024-03-18T16:04:57.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:57 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:04:57.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:57 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:04:57.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:57 smithi039 ceph-mon[31721]: from='client.24350 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:04:57.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:57 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:04:57.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:57 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T16:04:57.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:57 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T16:04:57.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:57 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:04:57.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:57 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:04:57.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:57 smithi196 ceph-mon[36307]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-18T16:04:57.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:57 smithi196 ceph-mon[36307]: Detected new or changed devices on smithi039 2024-03-18T16:04:57.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:57 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:04:57.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:57 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:04:57.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:57 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:04:57.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:57 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:04:57.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:57 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:04:57.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:57 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:04:57.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:57 smithi196 ceph-mon[36307]: Adjusting osd_memory_target on smithi039 to 4272M 2024-03-18T16:04:57.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:57 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:04:57.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:57 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:04:57.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:57 smithi196 ceph-mon[36307]: from='client.24350 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:04:57.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:57 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:04:57.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:57 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T16:04:57.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:57 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T16:04:57.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:57 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:04:57.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:57 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:04:59.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:59 smithi039 ceph-mon[31721]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-18T16:04:59.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:59 smithi039 ceph-mon[31721]: from='client.? 172.21.15.194:0/133898622' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "620f90e8-23e6-4bc0-8269-7219b9e1aebd"}]: dispatch 2024-03-18T16:04:59.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:59 smithi039 ceph-mon[31721]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "620f90e8-23e6-4bc0-8269-7219b9e1aebd"}]: dispatch 2024-03-18T16:04:59.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:59 smithi039 ceph-mon[31721]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "620f90e8-23e6-4bc0-8269-7219b9e1aebd"}]': finished 2024-03-18T16:04:59.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:59 smithi039 ceph-mon[31721]: osdmap e69: 11 total, 10 up, 11 in 2024-03-18T16:04:59.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:04:59 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T16:04:59.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:59 smithi196 ceph-mon[36307]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-18T16:04:59.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:59 smithi196 ceph-mon[36307]: from='client.? 172.21.15.194:0/133898622' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "620f90e8-23e6-4bc0-8269-7219b9e1aebd"}]: dispatch 2024-03-18T16:04:59.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:59 smithi196 ceph-mon[36307]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "620f90e8-23e6-4bc0-8269-7219b9e1aebd"}]: dispatch 2024-03-18T16:04:59.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:59 smithi196 ceph-mon[36307]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "620f90e8-23e6-4bc0-8269-7219b9e1aebd"}]': finished 2024-03-18T16:04:59.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:59 smithi196 ceph-mon[36307]: osdmap e69: 11 total, 10 up, 11 in 2024-03-18T16:04:59.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:04:59 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T16:04:59.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:59 smithi194 ceph-mon[36200]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-18T16:04:59.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:59 smithi194 ceph-mon[36200]: from='client.? 172.21.15.194:0/133898622' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "620f90e8-23e6-4bc0-8269-7219b9e1aebd"}]: dispatch 2024-03-18T16:04:59.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:59 smithi194 ceph-mon[36200]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "620f90e8-23e6-4bc0-8269-7219b9e1aebd"}]: dispatch 2024-03-18T16:04:59.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:59 smithi194 ceph-mon[36200]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "620f90e8-23e6-4bc0-8269-7219b9e1aebd"}]': finished 2024-03-18T16:04:59.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:59 smithi194 ceph-mon[36200]: osdmap e69: 11 total, 10 up, 11 in 2024-03-18T16:04:59.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:04:59 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T16:05:00.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:00 smithi039 ceph-mon[31721]: from='client.? 172.21.15.194:0/2584022559' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T16:05:00.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:00 smithi196 ceph-mon[36307]: from='client.? 172.21.15.194:0/2584022559' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T16:05:00.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:00 smithi194 ceph-mon[36200]: from='client.? 172.21.15.194:0/2584022559' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T16:05:01.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:01 smithi039 ceph-mon[31721]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-18T16:05:01.477 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:01 smithi196 ceph-mon[36307]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-18T16:05:01.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:01 smithi194 ceph-mon[36200]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-18T16:05:03.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:03 smithi194 ceph-mon[36200]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-18T16:05:03.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:03 smithi039 ceph-mon[31721]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-18T16:05:03.477 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:03 smithi196 ceph-mon[36307]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-18T16:05:05.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:05 smithi194 ceph-mon[36200]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-18T16:05:05.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:05 smithi196 ceph-mon[36307]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-18T16:05:05.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:05 smithi039 ceph-mon[31721]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-18T16:05:07.411 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:07 smithi194 ceph-mon[36200]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-18T16:05:07.411 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:07 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-18T16:05:07.411 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:07 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:05:07.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:07 smithi196 ceph-mon[36307]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-18T16:05:07.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:07 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-18T16:05:07.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:07 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:05:07.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:07 smithi039 ceph-mon[31721]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-18T16:05:07.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:07 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-18T16:05:07.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:07 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:05:08.458 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:08 smithi194 ceph-mon[36200]: Deploying daemon osd.10 on smithi194 2024-03-18T16:05:08.477 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:08 smithi196 ceph-mon[36307]: Deploying daemon osd.10 on smithi194 2024-03-18T16:05:08.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:08 smithi039 ceph-mon[31721]: Deploying daemon osd.10 on smithi194 2024-03-18T16:05:09.458 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:09 smithi194 ceph-mon[36200]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-18T16:05:09.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:09 smithi196 ceph-mon[36307]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-18T16:05:09.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:09 smithi039 ceph-mon[31721]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-18T16:05:11.232 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:11 smithi194 ceph-mon[36200]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-18T16:05:11.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:11 smithi196 ceph-mon[36307]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-18T16:05:11.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:11 smithi039 ceph-mon[31721]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-18T16:05:13.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:13 smithi196 ceph-mon[36307]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-18T16:05:13.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:13 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:05:13.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:13 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:05:13.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:13 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:05:13.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:13 smithi194 ceph-mon[36200]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-18T16:05:13.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:13 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:05:13.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:13 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:05:13.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:13 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:05:13.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:13 smithi039 ceph-mon[31721]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-18T16:05:13.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:13 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:05:13.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:13 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:05:13.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:13 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:05:15.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:15 smithi194 ceph-mon[36200]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-18T16:05:15.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:15 smithi196 ceph-mon[36307]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-18T16:05:15.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:15 smithi039 ceph-mon[31721]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-18T16:05:16.135 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:05:16.133+0000 7f1f5ffff700 1 -- 172.21.15.194:0/1715104517 <== mgr.14152 v2:172.21.15.39:6800/2749214286 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f1f78062710 con 0x7f1f600a2910 2024-03-18T16:05:16.136 INFO:teuthology.orchestra.run.smithi194.stdout:Created osd(s) 10 on host 'smithi194' 2024-03-18T16:05:16.138 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:05:16.136+0000 7f1f7d6b3700 1 -- 172.21.15.194:0/1715104517 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1f600a2910 msgr2=0x7f1f600a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:05:16.138 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:05:16.136+0000 7f1f7d6b3700 1 --2- 172.21.15.194:0/1715104517 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1f600a2910 0x7f1f600a4dd0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f1f68009a10 tx=0x7f1f68009190 comp rx=0 tx=0).stop 2024-03-18T16:05:16.138 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:05:16.136+0000 7f1f7d6b3700 1 -- 172.21.15.194:0/1715104517 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1f78107fe0 msgr2=0x7f1f78101270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:05:16.138 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:05:16.136+0000 7f1f7d6b3700 1 --2- 172.21.15.194:0/1715104517 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1f78107fe0 0x7f1f78101270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1f70007f00 tx=0x7f1f70004e40 comp rx=0 tx=0).stop 2024-03-18T16:05:16.138 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:05:16.136+0000 7f1f7d6b3700 1 -- 172.21.15.194:0/1715104517 shutdown_connections 2024-03-18T16:05:16.138 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:05:16.136+0000 7f1f7d6b3700 1 --2- 172.21.15.194:0/1715104517 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1f600a2910 0x7f1f600a4dd0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:16.138 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:05:16.136+0000 7f1f7d6b3700 1 --2- 172.21.15.194:0/1715104517 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1f78107fe0 0x7f1f78101270 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:16.138 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:05:16.136+0000 7f1f7d6b3700 1 --2- 172.21.15.194:0/1715104517 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1f78102d80 0x7f1f781026e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:16.139 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:05:16.136+0000 7f1f7d6b3700 1 --2- 172.21.15.194:0/1715104517 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f781056b0 0x7f1f78100d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:16.139 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:05:16.136+0000 7f1f7d6b3700 1 -- 172.21.15.194:0/1715104517 >> 172.21.15.194:0/1715104517 conn(0x7f1f780fc790 msgr2=0x7f1f781036c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:05:16.139 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:05:16.136+0000 7f1f7d6b3700 1 -- 172.21.15.194:0/1715104517 shutdown_connections 2024-03-18T16:05:16.139 INFO:teuthology.orchestra.run.smithi194.stderr:2024-03-18T16:05:16.136+0000 7f1f7d6b3700 1 -- 172.21.15.194:0/1715104517 wait complete. 2024-03-18T16:05:16.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:16 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:05:16.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:16 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:05:16.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:16 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:05:16.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:16 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:05:16.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:16 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:05:16.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:16 smithi194 ceph-mon[36200]: from='osd.10 [v2:172.21.15.194:6824/3725064809,v1:172.21.15.194:6825/3725064809]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-18T16:05:16.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:16 smithi194 ceph-mon[36200]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-18T16:05:16.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:16 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:05:16.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:16 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:05:16.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:16 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:05:16.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:16 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:05:16.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:16 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:05:16.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:16 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:05:16.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:16 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:05:16.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:16 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:05:16.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:16 smithi039 ceph-mon[31721]: from='osd.10 [v2:172.21.15.194:6824/3725064809,v1:172.21.15.194:6825/3725064809]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-18T16:05:16.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:16 smithi039 ceph-mon[31721]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-18T16:05:16.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:16 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:05:16.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:16 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:05:16.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:16 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:05:16.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:16 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:05:16.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:16 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:05:16.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:16 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:05:16.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:16 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:05:16.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:16 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:05:16.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:16 smithi196 ceph-mon[36307]: from='osd.10 [v2:172.21.15.194:6824/3725064809,v1:172.21.15.194:6825/3725064809]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-18T16:05:16.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:16 smithi196 ceph-mon[36307]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-18T16:05:16.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:16 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:05:16.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:16 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:05:16.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:16 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:05:16.841 DEBUG:teuthology.orchestra.run.smithi194:osd.10> sudo journalctl -f -n 0 -u ceph-1892d8fc-e540-11ee-95c9-87774f69a715@osd.10.service 2024-03-18T16:05:16.843 INFO:tasks.cephadm:Deploying osd.11 on smithi196 with /dev/vg_nvme/lv_1... 2024-03-18T16:05:16.843 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-18T16:05:17.090 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T16:05:17.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:17 smithi196 ceph-mon[36307]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-18T16:05:17.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:17 smithi196 ceph-mon[36307]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-18T16:05:17.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:17 smithi196 ceph-mon[36307]: from='osd.10 [v2:172.21.15.194:6824/3725064809,v1:172.21.15.194:6825/3725064809]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-03-18T16:05:17.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:17 smithi196 ceph-mon[36307]: osdmap e70: 11 total, 10 up, 11 in 2024-03-18T16:05:17.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:17 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T16:05:17.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:17 smithi196 ceph-mon[36307]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-03-18T16:05:17.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:17 smithi194 ceph-mon[36200]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-18T16:05:17.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:17 smithi194 ceph-mon[36200]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-18T16:05:17.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:17 smithi194 ceph-mon[36200]: from='osd.10 [v2:172.21.15.194:6824/3725064809,v1:172.21.15.194:6825/3725064809]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-03-18T16:05:17.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:17 smithi194 ceph-mon[36200]: osdmap e70: 11 total, 10 up, 11 in 2024-03-18T16:05:17.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:17 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T16:05:17.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:17 smithi194 ceph-mon[36200]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-03-18T16:05:17.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:17 smithi039 ceph-mon[31721]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-18T16:05:17.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:17 smithi039 ceph-mon[31721]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-18T16:05:17.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:17 smithi039 ceph-mon[31721]: from='osd.10 [v2:172.21.15.194:6824/3725064809,v1:172.21.15.194:6825/3725064809]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-03-18T16:05:17.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:17 smithi039 ceph-mon[31721]: osdmap e70: 11 total, 10 up, 11 in 2024-03-18T16:05:17.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:17 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T16:05:17.838 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:17 smithi039 ceph-mon[31721]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-03-18T16:05:18.348 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:18 smithi194 ceph-mon[36200]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-03-18T16:05:18.348 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:18 smithi194 ceph-mon[36200]: osdmap e71: 11 total, 10 up, 11 in 2024-03-18T16:05:18.348 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:18 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T16:05:18.348 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:18 smithi194 ceph-mon[36200]: from='osd.10 ' entity='osd.10' 2024-03-18T16:05:18.348 INFO:journalctl@ceph.osd.10.smithi194.stdout:Mar 18 16:05:18 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-10[61903]: 2024-03-18T16:05:18.030+0000 7f86ccde7700 -1 osd.10 0 waiting for initial osdmap 2024-03-18T16:05:18.349 INFO:journalctl@ceph.osd.10.smithi194.stdout:Mar 18 16:05:18 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-10[61903]: 2024-03-18T16:05:18.041+0000 7f86c4bd2700 -1 osd.10 71 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T16:05:18.531 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:18 smithi196 ceph-mon[36307]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-03-18T16:05:18.531 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:18 smithi196 ceph-mon[36307]: osdmap e71: 11 total, 10 up, 11 in 2024-03-18T16:05:18.532 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:18 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T16:05:18.532 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:18 smithi196 ceph-mon[36307]: from='osd.10 ' entity='osd.10' 2024-03-18T16:05:18.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:18 smithi039 ceph-mon[31721]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-03-18T16:05:18.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:18 smithi039 ceph-mon[31721]: osdmap e71: 11 total, 10 up, 11 in 2024-03-18T16:05:18.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:18 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T16:05:18.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:18 smithi039 ceph-mon[31721]: from='osd.10 ' entity='osd.10' 2024-03-18T16:05:19.304 INFO:teuthology.orchestra.run.smithi196.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-18T16:05:19.305 INFO:teuthology.orchestra.run.smithi196.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-18T16:05:19.305 INFO:teuthology.orchestra.run.smithi196.stderr: stderr: 10+0 records in 2024-03-18T16:05:19.305 INFO:teuthology.orchestra.run.smithi196.stderr:10+0 records out 2024-03-18T16:05:19.305 INFO:teuthology.orchestra.run.smithi196.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0112718 s, 930 MB/s 2024-03-18T16:05:19.305 INFO:teuthology.orchestra.run.smithi196.stderr:--> Zapping successful for: 2024-03-18T16:05:19.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:19 smithi194 ceph-mon[36200]: purged_snaps scrub starts 2024-03-18T16:05:19.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:19 smithi194 ceph-mon[36200]: purged_snaps scrub ok 2024-03-18T16:05:19.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:19 smithi194 ceph-mon[36200]: pgmap v257: 1 pgs: 1 peering; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-18T16:05:19.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:19 smithi194 ceph-mon[36200]: osd.10 [v2:172.21.15.194:6824/3725064809,v1:172.21.15.194:6825/3725064809] boot 2024-03-18T16:05:19.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:19 smithi194 ceph-mon[36200]: osdmap e72: 11 total, 11 up, 11 in 2024-03-18T16:05:19.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:19 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T16:05:19.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:19 smithi194 ceph-mon[36200]: osdmap e73: 11 total, 11 up, 11 in 2024-03-18T16:05:19.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:19 smithi039 ceph-mon[31721]: purged_snaps scrub starts 2024-03-18T16:05:19.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:19 smithi039 ceph-mon[31721]: purged_snaps scrub ok 2024-03-18T16:05:19.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:19 smithi039 ceph-mon[31721]: pgmap v257: 1 pgs: 1 peering; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-18T16:05:19.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:19 smithi039 ceph-mon[31721]: osd.10 [v2:172.21.15.194:6824/3725064809,v1:172.21.15.194:6825/3725064809] boot 2024-03-18T16:05:19.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:19 smithi039 ceph-mon[31721]: osdmap e72: 11 total, 11 up, 11 in 2024-03-18T16:05:19.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:19 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T16:05:19.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:19 smithi039 ceph-mon[31721]: osdmap e73: 11 total, 11 up, 11 in 2024-03-18T16:05:19.597 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:19 smithi196 ceph-mon[36307]: purged_snaps scrub starts 2024-03-18T16:05:19.597 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:19 smithi196 ceph-mon[36307]: purged_snaps scrub ok 2024-03-18T16:05:19.597 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:19 smithi196 ceph-mon[36307]: pgmap v257: 1 pgs: 1 peering; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-18T16:05:19.597 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:19 smithi196 ceph-mon[36307]: osd.10 [v2:172.21.15.194:6824/3725064809,v1:172.21.15.194:6825/3725064809] boot 2024-03-18T16:05:19.598 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:19 smithi196 ceph-mon[36307]: osdmap e72: 11 total, 11 up, 11 in 2024-03-18T16:05:19.598 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:19 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-18T16:05:19.598 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:19 smithi196 ceph-mon[36307]: osdmap e73: 11 total, 11 up, 11 in 2024-03-18T16:05:19.944 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph orch daemon add osd smithi196:vg_nvme/lv_1 2024-03-18T16:05:20.198 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T16:05:21.298 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:21 smithi196 ceph-mon[36307]: pgmap v260: 1 pgs: 1 peering; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-03-18T16:05:21.298 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:21 smithi196 ceph-mon[36307]: Detected new or changed devices on smithi194 2024-03-18T16:05:21.298 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:21 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:05:21.298 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:21 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:05:21.298 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:21 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:05:21.298 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:21 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:05:21.298 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:21 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:05:21.298 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:21 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:05:21.298 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:21 smithi196 ceph-mon[36307]: Adjusting osd_memory_target on smithi194 to 4270M 2024-03-18T16:05:21.298 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:21 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:05:21.299 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:21 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:05:21.299 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:21 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:05:21.299 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:21 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:05:21.299 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:21 smithi196 ceph-mon[36307]: osdmap e74: 11 total, 11 up, 11 in 2024-03-18T16:05:21.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:21 smithi039 ceph-mon[31721]: pgmap v260: 1 pgs: 1 peering; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-03-18T16:05:21.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:21 smithi039 ceph-mon[31721]: Detected new or changed devices on smithi194 2024-03-18T16:05:21.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:21 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:05:21.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:21 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:05:21.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:21 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:05:21.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:21 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:05:21.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:21 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:05:21.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:21 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:05:21.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:21 smithi039 ceph-mon[31721]: Adjusting osd_memory_target on smithi194 to 4270M 2024-03-18T16:05:21.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:21 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:05:21.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:21 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:05:21.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:21 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:05:21.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:21 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:05:21.589 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:21 smithi039 ceph-mon[31721]: osdmap e74: 11 total, 11 up, 11 in 2024-03-18T16:05:21.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:21 smithi194 ceph-mon[36200]: pgmap v260: 1 pgs: 1 peering; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-03-18T16:05:21.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:21 smithi194 ceph-mon[36200]: Detected new or changed devices on smithi194 2024-03-18T16:05:21.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:21 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:05:21.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:21 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:05:21.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:21 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:05:21.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:21 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:05:21.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:21 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:05:21.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:21 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:05:21.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:21 smithi194 ceph-mon[36200]: Adjusting osd_memory_target on smithi194 to 4270M 2024-03-18T16:05:21.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:21 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:05:21.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:21 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:05:21.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:21 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:05:21.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:21 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:05:21.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:21 smithi194 ceph-mon[36200]: osdmap e74: 11 total, 11 up, 11 in 2024-03-18T16:05:21.847 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:21.844+0000 7ff8d2840700 1 -- 172.21.15.196:0/2185568694 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff8cc107ff0 msgr2=0x7ff8cc10a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:05:21.848 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:21.844+0000 7ff8d2840700 1 --2- 172.21.15.196:0/2185568694 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff8cc107ff0 0x7ff8cc10a460 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff8c400b0a0 tx=0x7ff8c400b3b0 comp rx=0 tx=0).stop 2024-03-18T16:05:21.848 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:21.844+0000 7ff8d2840700 1 -- 172.21.15.196:0/2185568694 shutdown_connections 2024-03-18T16:05:21.848 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:21.844+0000 7ff8d2840700 1 --2- 172.21.15.196:0/2185568694 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff8cc107ff0 0x7ff8cc10a460 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:21.848 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:21.844+0000 7ff8d2840700 1 --2- 172.21.15.196:0/2185568694 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff8cc102d90 0x7ff8cc105180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:21.848 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:21.844+0000 7ff8d2840700 1 --2- 172.21.15.196:0/2185568694 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff8cc1056c0 0x7ff8cc107ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:21.848 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:21.844+0000 7ff8d2840700 1 -- 172.21.15.196:0/2185568694 >> 172.21.15.196:0/2185568694 conn(0x7ff8cc0fc7c0 msgr2=0x7ff8cc0febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:05:21.848 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:21.844+0000 7ff8d2840700 1 -- 172.21.15.196:0/2185568694 shutdown_connections 2024-03-18T16:05:21.848 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:21.844+0000 7ff8d2840700 1 -- 172.21.15.196:0/2185568694 wait complete. 2024-03-18T16:05:21.848 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:21.844+0000 7ff8d2840700 1 Processor -- start 2024-03-18T16:05:21.848 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:21.844+0000 7ff8d2840700 1 -- start start 2024-03-18T16:05:21.848 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:21.845+0000 7ff8d2840700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff8cc102d90 0x7ff8cc1026f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:21.849 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:21.845+0000 7ff8d2840700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff8cc1056c0 0x7ff8cc100d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:21.849 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:21.845+0000 7ff8d2840700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff8cc107ff0 0x7ff8cc101280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:21.849 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:21.845+0000 7ff8d2840700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7ff8cc10df30 con 0x7ff8cc102d90 2024-03-18T16:05:21.849 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:21.845+0000 7ff8d2840700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ff8cc10ddb0 con 0x7ff8cc107ff0 2024-03-18T16:05:21.849 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:21.845+0000 7ff8d2840700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7ff8cc10e0b0 con 0x7ff8cc1056c0 2024-03-18T16:05:21.849 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:21.845+0000 7ff8cb7fe700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff8cc1056c0 0x7ff8cc100d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:21.849 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:21.845+0000 7ff8cb7fe700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff8cc1056c0 0x7ff8cc100d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.196:49882/0 (socket says 172.21.15.196:49882) 2024-03-18T16:05:21.849 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:21.845+0000 7ff8cb7fe700 1 -- 172.21.15.196:0/3502059380 learned_addr learned my addr 172.21.15.196:0/3502059380 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-18T16:05:21.849 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:21.845+0000 7ff8d0ddd700 1 --2- 172.21.15.196:0/3502059380 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff8cc107ff0 0x7ff8cc101280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:21.849 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:21.845+0000 7ff8cb7fe700 1 -- 172.21.15.196:0/3502059380 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff8cc107ff0 msgr2=0x7ff8cc101280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:05:21.849 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:21.845+0000 7ff8cb7fe700 1 --2- 172.21.15.196:0/3502059380 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff8cc107ff0 0x7ff8cc101280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:21.850 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:21.845+0000 7ff8cb7fe700 1 -- 172.21.15.196:0/3502059380 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff8cc102d90 msgr2=0x7ff8cc1026f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:05:21.850 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:21.845+0000 7ff8cb7fe700 1 --2- 172.21.15.196:0/3502059380 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff8cc102d90 0x7ff8cc1026f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:21.850 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:21.845+0000 7ff8cb7fe700 1 -- 172.21.15.196:0/3502059380 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8cc1017c0 con 0x7ff8cc1056c0 2024-03-18T16:05:21.850 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:21.845+0000 7ff8cb7fe700 1 --2- 172.21.15.196:0/3502059380 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff8cc1056c0 0x7ff8cc100d40 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff8c000d950 tx=0x7ff8c000dd10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:05:21.850 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:21.845+0000 7ff8c97fa700 1 -- 172.21.15.196:0/3502059380 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8c0016070 con 0x7ff8cc1056c0 2024-03-18T16:05:21.850 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:21.845+0000 7ff8d2840700 1 -- 172.21.15.196:0/3502059380 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8cc101ab0 con 0x7ff8cc1056c0 2024-03-18T16:05:21.850 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:21.845+0000 7ff8c97fa700 1 -- 172.21.15.196:0/3502059380 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff8c0009b40 con 0x7ff8cc1056c0 2024-03-18T16:05:21.850 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:21.845+0000 7ff8d2840700 1 -- 172.21.15.196:0/3502059380 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8cc1a6ff0 con 0x7ff8cc1056c0 2024-03-18T16:05:21.850 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:21.845+0000 7ff8c97fa700 1 -- 172.21.15.196:0/3502059380 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8c000f620 con 0x7ff8cc1056c0 2024-03-18T16:05:21.851 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:21.846+0000 7ff8d2840700 1 -- 172.21.15.196:0/3502059380 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8cc068f90 con 0x7ff8cc1056c0 2024-03-18T16:05:21.851 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:21.847+0000 7ff8c97fa700 1 -- 172.21.15.196:0/3502059380 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7ff8c0019070 con 0x7ff8cc1056c0 2024-03-18T16:05:21.851 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:21.848+0000 7ff8c97fa700 1 --2- 172.21.15.196:0/3502059380 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff8b40a2890 0x7ff8b40a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:21.851 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:21.848+0000 7ff8c97fa700 1 -- 172.21.15.196:0/3502059380 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 6694+0+0 (secure 0 0 0) 0x7ff8c00ce460 con 0x7ff8cc1056c0 2024-03-18T16:05:21.852 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:21.848+0000 7ff8cbfff700 1 --2- 172.21.15.196:0/3502059380 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff8b40a2890 0x7ff8b40a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:21.852 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:21.849+0000 7ff8cbfff700 1 --2- 172.21.15.196:0/3502059380 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff8b40a2890 0x7ff8b40a4d50 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff8bc0097b0 tx=0x7ff8bc006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:05:21.854 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:21.852+0000 7ff8c97fa700 1 -- 172.21.15.196:0/3502059380 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff8c009bb50 con 0x7ff8cc1056c0 2024-03-18T16:05:22.059 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:22.057+0000 7ff8d2840700 1 -- 172.21.15.196:0/3502059380 --> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7ff8cc062710 con 0x7ff8b40a2890 2024-03-18T16:05:22.694 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:22 smithi196 ceph-mon[36307]: osdmap e75: 11 total, 11 up, 11 in 2024-03-18T16:05:22.694 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:22 smithi196 ceph-mon[36307]: pgmap v263: 1 pgs: 1 peering; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-03-18T16:05:22.694 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:22 smithi196 ceph-mon[36307]: from='client.24394 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:05:22.694 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:22 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T16:05:22.694 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:22 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T16:05:22.694 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:22 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:05:22.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:22 smithi194 ceph-mon[36200]: osdmap e75: 11 total, 11 up, 11 in 2024-03-18T16:05:22.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:22 smithi194 ceph-mon[36200]: pgmap v263: 1 pgs: 1 peering; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-03-18T16:05:22.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:22 smithi194 ceph-mon[36200]: from='client.24394 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:05:22.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:22 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T16:05:22.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:22 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T16:05:22.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:22 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:05:22.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:22 smithi039 ceph-mon[31721]: osdmap e75: 11 total, 11 up, 11 in 2024-03-18T16:05:22.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:22 smithi039 ceph-mon[31721]: pgmap v263: 1 pgs: 1 peering; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-03-18T16:05:22.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:22 smithi039 ceph-mon[31721]: from='client.24394 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi196:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:05:22.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:22 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-18T16:05:22.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:22 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-18T16:05:22.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:22 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:05:23.691 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:23 smithi196 ceph-mon[36307]: osdmap e76: 11 total, 11 up, 11 in 2024-03-18T16:05:23.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:23 smithi194 ceph-mon[36200]: osdmap e76: 11 total, 11 up, 11 in 2024-03-18T16:05:23.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:23 smithi039 ceph-mon[31721]: osdmap e76: 11 total, 11 up, 11 in 2024-03-18T16:05:24.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:24 smithi196 ceph-mon[36307]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 140 KiB/s, 0 objects/s recovering 2024-03-18T16:05:24.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:24 smithi194 ceph-mon[36200]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 140 KiB/s, 0 objects/s recovering 2024-03-18T16:05:24.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:24 smithi039 ceph-mon[31721]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail; 140 KiB/s, 0 objects/s recovering 2024-03-18T16:05:25.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:25 smithi196 ceph-mon[36307]: from='client.? 172.21.15.196:0/1061835788' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "867a2535-8235-4ceb-a698-eb86869d6d20"}]: dispatch 2024-03-18T16:05:25.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:25 smithi196 ceph-mon[36307]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "867a2535-8235-4ceb-a698-eb86869d6d20"}]: dispatch 2024-03-18T16:05:25.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:25 smithi196 ceph-mon[36307]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "867a2535-8235-4ceb-a698-eb86869d6d20"}]': finished 2024-03-18T16:05:25.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:25 smithi196 ceph-mon[36307]: osdmap e77: 12 total, 11 up, 12 in 2024-03-18T16:05:25.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:25 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T16:05:25.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:25 smithi194 ceph-mon[36200]: from='client.? 172.21.15.196:0/1061835788' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "867a2535-8235-4ceb-a698-eb86869d6d20"}]: dispatch 2024-03-18T16:05:25.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:25 smithi194 ceph-mon[36200]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "867a2535-8235-4ceb-a698-eb86869d6d20"}]: dispatch 2024-03-18T16:05:25.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:25 smithi194 ceph-mon[36200]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "867a2535-8235-4ceb-a698-eb86869d6d20"}]': finished 2024-03-18T16:05:25.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:25 smithi194 ceph-mon[36200]: osdmap e77: 12 total, 11 up, 12 in 2024-03-18T16:05:25.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:25 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T16:05:25.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:25 smithi039 ceph-mon[31721]: from='client.? 172.21.15.196:0/1061835788' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "867a2535-8235-4ceb-a698-eb86869d6d20"}]: dispatch 2024-03-18T16:05:25.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:25 smithi039 ceph-mon[31721]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "867a2535-8235-4ceb-a698-eb86869d6d20"}]: dispatch 2024-03-18T16:05:25.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:25 smithi039 ceph-mon[31721]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "867a2535-8235-4ceb-a698-eb86869d6d20"}]': finished 2024-03-18T16:05:25.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:25 smithi039 ceph-mon[31721]: osdmap e77: 12 total, 11 up, 12 in 2024-03-18T16:05:25.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:25 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T16:05:26.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:26 smithi196 ceph-mon[36307]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail; 113 KiB/s, 0 objects/s recovering 2024-03-18T16:05:26.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:26 smithi196 ceph-mon[36307]: from='client.? 172.21.15.196:0/1079867647' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T16:05:26.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:26 smithi194 ceph-mon[36200]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail; 113 KiB/s, 0 objects/s recovering 2024-03-18T16:05:26.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:26 smithi194 ceph-mon[36200]: from='client.? 172.21.15.196:0/1079867647' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T16:05:26.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:26 smithi039 ceph-mon[31721]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail; 113 KiB/s, 0 objects/s recovering 2024-03-18T16:05:26.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:26 smithi039 ceph-mon[31721]: from='client.? 172.21.15.196:0/1079867647' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-18T16:05:28.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:28 smithi039 ceph-mon[31721]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-03-18T16:05:28.884 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:28 smithi196 ceph-mon[36307]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-03-18T16:05:28.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:28 smithi194 ceph-mon[36200]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-03-18T16:05:30.814 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:30 smithi196 ceph-mon[36307]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-18T16:05:30.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:30 smithi039 ceph-mon[31721]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-18T16:05:30.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:30 smithi194 ceph-mon[36200]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-18T16:05:32.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:32 smithi039 ceph-mon[31721]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-03-18T16:05:32.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:32 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-18T16:05:32.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:32 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:05:32.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:32 smithi039 ceph-mon[31721]: Deploying daemon osd.11 on smithi196 2024-03-18T16:05:32.945 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:32 smithi196 ceph-mon[36307]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-03-18T16:05:32.945 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:32 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-18T16:05:32.945 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:32 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:05:32.945 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:32 smithi196 ceph-mon[36307]: Deploying daemon osd.11 on smithi196 2024-03-18T16:05:32.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:32 smithi194 ceph-mon[36200]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-03-18T16:05:32.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:32 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-18T16:05:32.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:32 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:05:32.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:32 smithi194 ceph-mon[36200]: Deploying daemon osd.11 on smithi196 2024-03-18T16:05:34.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:34 smithi039 ceph-mon[31721]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-03-18T16:05:34.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:34 smithi196 ceph-mon[36307]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-03-18T16:05:34.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:34 smithi194 ceph-mon[36200]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-03-18T16:05:36.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:36 smithi039 ceph-mon[31721]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-03-18T16:05:36.959 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:36 smithi196 ceph-mon[36307]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-03-18T16:05:36.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:36 smithi194 ceph-mon[36200]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-03-18T16:05:39.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:39 smithi039 ceph-mon[31721]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-03-18T16:05:39.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:39 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:05:39.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:39 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:05:39.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:39 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:05:39.693 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:39 smithi194 ceph-mon[36200]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-03-18T16:05:39.693 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:39 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:05:39.693 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:39 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:05:39.694 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:39 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:05:39.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:39 smithi196 ceph-mon[36307]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-03-18T16:05:39.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:39 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:05:39.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:39 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:05:39.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:39 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:05:41.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:41 smithi196 ceph-mon[36307]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-03-18T16:05:41.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:41 smithi039 ceph-mon[31721]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-03-18T16:05:41.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:41 smithi194 ceph-mon[36200]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-03-18T16:05:42.246 INFO:teuthology.orchestra.run.smithi196.stdout:Created osd(s) 11 on host 'smithi196' 2024-03-18T16:05:42.247 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:42.244+0000 7ff8c97fa700 1 -- 172.21.15.196:0/3502059380 <== mgr.14152 v2:172.21.15.39:6800/2749214286 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7ff8cc062710 con 0x7ff8b40a2890 2024-03-18T16:05:42.248 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:42.246+0000 7ff8d2840700 1 -- 172.21.15.196:0/3502059380 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff8b40a2890 msgr2=0x7ff8b40a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:05:42.248 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:42.246+0000 7ff8d2840700 1 --2- 172.21.15.196:0/3502059380 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff8b40a2890 0x7ff8b40a4d50 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff8bc0097b0 tx=0x7ff8bc006d20 comp rx=0 tx=0).stop 2024-03-18T16:05:42.248 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:42.246+0000 7ff8d2840700 1 -- 172.21.15.196:0/3502059380 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff8cc1056c0 msgr2=0x7ff8cc100d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:05:42.248 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:42.246+0000 7ff8d2840700 1 --2- 172.21.15.196:0/3502059380 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff8cc1056c0 0x7ff8cc100d40 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff8c000d950 tx=0x7ff8c000dd10 comp rx=0 tx=0).stop 2024-03-18T16:05:42.249 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:42.246+0000 7ff8d2840700 1 -- 172.21.15.196:0/3502059380 shutdown_connections 2024-03-18T16:05:42.249 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:42.246+0000 7ff8d2840700 1 --2- 172.21.15.196:0/3502059380 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff8b40a2890 0x7ff8b40a4d50 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:42.249 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:42.246+0000 7ff8d2840700 1 --2- 172.21.15.196:0/3502059380 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff8cc107ff0 0x7ff8cc101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:42.249 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:42.246+0000 7ff8d2840700 1 --2- 172.21.15.196:0/3502059380 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff8cc102d90 0x7ff8cc1026f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:42.249 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:42.246+0000 7ff8d2840700 1 --2- 172.21.15.196:0/3502059380 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff8cc1056c0 0x7ff8cc100d40 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:42.249 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:42.246+0000 7ff8d2840700 1 -- 172.21.15.196:0/3502059380 >> 172.21.15.196:0/3502059380 conn(0x7ff8cc0fc7c0 msgr2=0x7ff8cc1036d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:05:42.249 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:42.246+0000 7ff8d2840700 1 -- 172.21.15.196:0/3502059380 shutdown_connections 2024-03-18T16:05:42.249 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:42.246+0000 7ff8d2840700 1 -- 172.21.15.196:0/3502059380 wait complete. 2024-03-18T16:05:42.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:42 smithi039 ceph-mon[31721]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-03-18T16:05:42.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:42 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:05:42.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:42 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:05:42.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:42 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:05:42.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:42 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:05:42.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:42 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:05:42.838 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:42 smithi039 ceph-mon[31721]: from='osd.11 [v2:172.21.15.196:6824/2501212422,v1:172.21.15.196:6825/2501212422]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-18T16:05:42.838 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:42 smithi039 ceph-mon[31721]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-18T16:05:42.838 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:42 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:05:42.838 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:42 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:05:42.838 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:42 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:05:42.894 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:42 smithi196 ceph-mon[36307]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-03-18T16:05:42.894 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:42 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:05:42.894 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:42 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:05:42.894 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:42 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:05:42.894 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:42 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:05:42.894 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:42 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:05:42.895 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:42 smithi196 ceph-mon[36307]: from='osd.11 [v2:172.21.15.196:6824/2501212422,v1:172.21.15.196:6825/2501212422]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-18T16:05:42.895 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:42 smithi196 ceph-mon[36307]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-18T16:05:42.895 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:42 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:05:42.895 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:42 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:05:42.895 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:42 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:05:42.921 DEBUG:teuthology.orchestra.run.smithi196:osd.11> sudo journalctl -f -n 0 -u ceph-1892d8fc-e540-11ee-95c9-87774f69a715@osd.11.service 2024-03-18T16:05:42.923 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-03-18T16:05:42.924 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph osd stat -f json 2024-03-18T16:05:42.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:42 smithi194 ceph-mon[36200]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-03-18T16:05:42.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:42 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:05:42.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:42 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:05:42.991 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:42 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:05:42.991 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:42 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:05:42.991 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:42 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:05:42.991 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:42 smithi194 ceph-mon[36200]: from='osd.11 [v2:172.21.15.196:6824/2501212422,v1:172.21.15.196:6825/2501212422]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-18T16:05:42.991 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:42 smithi194 ceph-mon[36200]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-18T16:05:42.991 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:42 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:05:42.991 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:42 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:05:42.991 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:42 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:05:43.405 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T16:05:43.797 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:43 smithi039 ceph-mon[31721]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-18T16:05:43.797 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:43 smithi039 ceph-mon[31721]: from='osd.11 [v2:172.21.15.196:6824/2501212422,v1:172.21.15.196:6825/2501212422]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-03-18T16:05:43.797 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:43 smithi039 ceph-mon[31721]: osdmap e78: 12 total, 11 up, 12 in 2024-03-18T16:05:43.797 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:43 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T16:05:43.797 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:43 smithi039 ceph-mon[31721]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-03-18T16:05:43.826 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:43 smithi196 ceph-mon[36307]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-18T16:05:43.826 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:43 smithi196 ceph-mon[36307]: from='osd.11 [v2:172.21.15.196:6824/2501212422,v1:172.21.15.196:6825/2501212422]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-03-18T16:05:43.826 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:43 smithi196 ceph-mon[36307]: osdmap e78: 12 total, 11 up, 12 in 2024-03-18T16:05:43.826 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:43 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T16:05:43.827 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:43 smithi196 ceph-mon[36307]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-03-18T16:05:43.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:43 smithi194 ceph-mon[36200]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-18T16:05:43.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:43 smithi194 ceph-mon[36200]: from='osd.11 [v2:172.21.15.196:6824/2501212422,v1:172.21.15.196:6825/2501212422]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-03-18T16:05:43.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:43 smithi194 ceph-mon[36200]: osdmap e78: 12 total, 11 up, 12 in 2024-03-18T16:05:43.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:43 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T16:05:43.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:43 smithi194 ceph-mon[36200]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]: dispatch 2024-03-18T16:05:44.437 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 18 16:05:44 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-11[61670]: 2024-03-18T16:05:44.245+0000 7fb12ee9e700 -1 osd.11 0 waiting for initial osdmap 2024-03-18T16:05:44.437 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 18 16:05:44 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-11[61670]: 2024-03-18T16:05:44.260+0000 7fb126488700 -1 osd.11 79 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-18T16:05:44.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:44 smithi196 ceph-mon[36307]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-03-18T16:05:44.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:44 smithi196 ceph-mon[36307]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-03-18T16:05:44.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:44 smithi196 ceph-mon[36307]: osdmap e79: 12 total, 11 up, 12 in 2024-03-18T16:05:44.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:44 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T16:05:44.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:44 smithi196 ceph-mon[36307]: from='osd.11 ' entity='osd.11' 2024-03-18T16:05:44.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:44 smithi039 ceph-mon[31721]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-03-18T16:05:44.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:44 smithi039 ceph-mon[31721]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-03-18T16:05:44.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:44 smithi039 ceph-mon[31721]: osdmap e79: 12 total, 11 up, 12 in 2024-03-18T16:05:44.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:44 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T16:05:44.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:44 smithi039 ceph-mon[31721]: from='osd.11 ' entity='osd.11' 2024-03-18T16:05:44.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:44 smithi194 ceph-mon[36200]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-03-18T16:05:44.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:44 smithi194 ceph-mon[36200]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi196", "root=default"]}]': finished 2024-03-18T16:05:44.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:44 smithi194 ceph-mon[36200]: osdmap e79: 12 total, 11 up, 12 in 2024-03-18T16:05:44.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:44 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T16:05:44.991 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:44 smithi194 ceph-mon[36200]: from='osd.11 ' entity='osd.11' 2024-03-18T16:05:45.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:45.767+0000 7f82c9e00700 1 -- 172.21.15.39:0/492227727 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f82c4102240 msgr2=0x7f82c4102660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:05:45.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:45.767+0000 7f82c9e00700 1 --2- 172.21.15.39:0/492227727 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f82c4102240 0x7f82c4102660 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f82b80099c0 tx=0x7f82b8009cd0 comp rx=0 tx=0).stop 2024-03-18T16:05:45.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:45.767+0000 7f82c9e00700 1 -- 172.21.15.39:0/492227727 shutdown_connections 2024-03-18T16:05:45.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:45.767+0000 7f82c9e00700 1 --2- 172.21.15.39:0/492227727 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f82c4103de0 0x7f82c410a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:45.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:45.767+0000 7f82c9e00700 1 --2- 172.21.15.39:0/492227727 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f82c4102240 0x7f82c4102660 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:45.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:45.767+0000 7f82c9e00700 1 --2- 172.21.15.39:0/492227727 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f82c4103440 0x7f82c41038a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:45.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:45.767+0000 7f82c9e00700 1 -- 172.21.15.39:0/492227727 >> 172.21.15.39:0/492227727 conn(0x7f82c40fd7e0 msgr2=0x7f82c40ffc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:05:45.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:45.767+0000 7f82c9e00700 1 -- 172.21.15.39:0/492227727 shutdown_connections 2024-03-18T16:05:45.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:45.767+0000 7f82c9e00700 1 -- 172.21.15.39:0/492227727 wait complete. 2024-03-18T16:05:45.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:45.768+0000 7f82c9e00700 1 Processor -- start 2024-03-18T16:05:45.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:45.769+0000 7f82c9e00700 1 -- start start 2024-03-18T16:05:45.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:45.769+0000 7f82c9e00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f82c4102240 0x7f82c4198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:45.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:45.769+0000 7f82c9e00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f82c4103440 0x7f82c4198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:45.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:45.769+0000 7f82c9e00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f82c4103de0 0x7f82c419f9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:45.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:45.769+0000 7f82c9e00700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f82c410de50 con 0x7f82c4103440 2024-03-18T16:05:45.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:45.769+0000 7f82c9e00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f82c410dcd0 con 0x7f82c4103de0 2024-03-18T16:05:45.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:45.769+0000 7f82c9e00700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f82c410dfd0 con 0x7f82c4102240 2024-03-18T16:05:45.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:45.770+0000 7f82c3e00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f82c4103440 0x7f82c4198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:45.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:45.770+0000 7f82c8a00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f82c4102240 0x7f82c4198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:45.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:45.770+0000 7f82c3e00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f82c4103440 0x7f82c4198970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58896/0 (socket says 172.21.15.39:58896) 2024-03-18T16:05:45.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:45.770+0000 7f82c3e00700 1 -- 172.21.15.39:0/3213830152 learned_addr learned my addr 172.21.15.39:0/3213830152 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:05:45.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:45.770+0000 7f82c9400700 1 --2- 172.21.15.39:0/3213830152 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f82c4103de0 0x7f82c419f9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:45.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:45.770+0000 7f82c9400700 1 -- 172.21.15.39:0/3213830152 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f82c4102240 msgr2=0x7f82c4198430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:05:45.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:45.770+0000 7f82c9400700 1 --2- 172.21.15.39:0/3213830152 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f82c4102240 0x7f82c4198430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:45.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:45.770+0000 7f82c9400700 1 -- 172.21.15.39:0/3213830152 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f82c4103440 msgr2=0x7f82c4198970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:05:45.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:45.770+0000 7f82c9400700 1 --2- 172.21.15.39:0/3213830152 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f82c4103440 0x7f82c4198970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:45.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:45.770+0000 7f82c9400700 1 -- 172.21.15.39:0/3213830152 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82c41a0090 con 0x7f82c4103de0 2024-03-18T16:05:45.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:45.770+0000 7f82c3e00700 1 --2- 172.21.15.39:0/3213830152 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f82c4103440 0x7f82c4198970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:05:45.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:45.770+0000 7f82c9400700 1 --2- 172.21.15.39:0/3213830152 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f82c4103de0 0x7f82c419f9f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f82bc00efc0 tx=0x7f82bc00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:05:45.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:45.771+0000 7f82c1600700 1 -- 172.21.15.39:0/3213830152 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82bc0092e0 con 0x7f82c4103de0 2024-03-18T16:05:45.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:45.771+0000 7f82c1600700 1 -- 172.21.15.39:0/3213830152 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f82bc004bb0 con 0x7f82c4103de0 2024-03-18T16:05:45.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:45.771+0000 7f82c9e00700 1 -- 172.21.15.39:0/3213830152 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82c41a0320 con 0x7f82c4103de0 2024-03-18T16:05:45.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:45.771+0000 7f82c9e00700 1 -- 172.21.15.39:0/3213830152 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82c41a0860 con 0x7f82c4103de0 2024-03-18T16:05:45.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:45.771+0000 7f82c1600700 1 -- 172.21.15.39:0/3213830152 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82bc004280 con 0x7f82c4103de0 2024-03-18T16:05:45.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:45.772+0000 7f82c9e00700 1 -- 172.21.15.39:0/3213830152 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82c4068f90 con 0x7f82c4103de0 2024-03-18T16:05:45.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:45.774+0000 7f82c1600700 1 -- 172.21.15.39:0/3213830152 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f82bc01b030 con 0x7f82c4103de0 2024-03-18T16:05:45.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:45.777+0000 7f82c1600700 1 --2- 172.21.15.39:0/3213830152 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f82ac0a2910 0x7f82ac0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:45.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:45.777+0000 7f82c1600700 1 -- 172.21.15.39:0/3213830152 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 7127+0+0 (secure 0 0 0) 0x7f82bc0ce2b0 con 0x7f82c4103de0 2024-03-18T16:05:45.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:45.777+0000 7f82c8a00700 1 --2- 172.21.15.39:0/3213830152 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f82ac0a2910 0x7f82ac0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:45.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:45.778+0000 7f82c8a00700 1 --2- 172.21.15.39:0/3213830152 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f82ac0a2910 0x7f82ac0a4dd0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f82b800b580 tx=0x7f82b8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:05:45.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:45.778+0000 7f82c1600700 1 -- 172.21.15.39:0/3213830152 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f82bc022020 con 0x7f82c4103de0 2024-03-18T16:05:45.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:45 smithi039 ceph-mon[31721]: purged_snaps scrub starts 2024-03-18T16:05:45.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:45 smithi039 ceph-mon[31721]: purged_snaps scrub ok 2024-03-18T16:05:45.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:45 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T16:05:45.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:45 smithi039 ceph-mon[31721]: osd.11 [v2:172.21.15.196:6824/2501212422,v1:172.21.15.196:6825/2501212422] boot 2024-03-18T16:05:45.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:45 smithi039 ceph-mon[31721]: osdmap e80: 12 total, 12 up, 12 in 2024-03-18T16:05:45.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:45 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T16:05:45.927 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:45 smithi196 ceph-mon[36307]: purged_snaps scrub starts 2024-03-18T16:05:45.927 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:45 smithi196 ceph-mon[36307]: purged_snaps scrub ok 2024-03-18T16:05:45.927 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:45 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T16:05:45.928 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:45 smithi196 ceph-mon[36307]: osd.11 [v2:172.21.15.196:6824/2501212422,v1:172.21.15.196:6825/2501212422] boot 2024-03-18T16:05:45.928 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:45 smithi196 ceph-mon[36307]: osdmap e80: 12 total, 12 up, 12 in 2024-03-18T16:05:45.928 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:45 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T16:05:45.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:45 smithi194 ceph-mon[36200]: purged_snaps scrub starts 2024-03-18T16:05:45.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:45 smithi194 ceph-mon[36200]: purged_snaps scrub ok 2024-03-18T16:05:45.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:45 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T16:05:45.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:45 smithi194 ceph-mon[36200]: osd.11 [v2:172.21.15.196:6824/2501212422,v1:172.21.15.196:6825/2501212422] boot 2024-03-18T16:05:45.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:45 smithi194 ceph-mon[36200]: osdmap e80: 12 total, 12 up, 12 in 2024-03-18T16:05:45.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:45 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-18T16:05:46.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:46.005+0000 7f82c9e00700 1 -- 172.21.15.39:0/3213830152 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f82c402dc10 con 0x7f82c4103de0 2024-03-18T16:05:46.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:46.006+0000 7f82c1600700 1 -- 172.21.15.39:0/3213830152 <== mon.2 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v80) ==== 74+0+133 (secure 0 0 0) 0x7f82c402dc10 con 0x7f82c4103de0 2024-03-18T16:05:46.007 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:05:46.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:46.008+0000 7f82c9e00700 1 -- 172.21.15.39:0/3213830152 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f82ac0a2910 msgr2=0x7f82ac0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:05:46.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:46.008+0000 7f82c9e00700 1 --2- 172.21.15.39:0/3213830152 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f82ac0a2910 0x7f82ac0a4dd0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f82b800b580 tx=0x7f82b8005fb0 comp rx=0 tx=0).stop 2024-03-18T16:05:46.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:46.008+0000 7f82c9e00700 1 -- 172.21.15.39:0/3213830152 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f82c4103de0 msgr2=0x7f82c419f9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:05:46.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:46.008+0000 7f82c9e00700 1 --2- 172.21.15.39:0/3213830152 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f82c4103de0 0x7f82c419f9f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f82bc00efc0 tx=0x7f82bc00c3a0 comp rx=0 tx=0).stop 2024-03-18T16:05:46.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:46.009+0000 7f82c9e00700 1 -- 172.21.15.39:0/3213830152 shutdown_connections 2024-03-18T16:05:46.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:46.009+0000 7f82c9e00700 1 --2- 172.21.15.39:0/3213830152 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f82ac0a2910 0x7f82ac0a4dd0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:46.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:46.009+0000 7f82c9e00700 1 --2- 172.21.15.39:0/3213830152 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f82c4103de0 0x7f82c419f9f0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:46.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:46.009+0000 7f82c9e00700 1 --2- 172.21.15.39:0/3213830152 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f82c4102240 0x7f82c4198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:46.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:46.009+0000 7f82c9e00700 1 --2- 172.21.15.39:0/3213830152 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f82c4103440 0x7f82c4198970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:46.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:46.009+0000 7f82c9e00700 1 -- 172.21.15.39:0/3213830152 >> 172.21.15.39:0/3213830152 conn(0x7f82c40fd7e0 msgr2=0x7f82c4106740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:05:46.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:46.009+0000 7f82c9e00700 1 -- 172.21.15.39:0/3213830152 shutdown_connections 2024-03-18T16:05:46.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:46.009+0000 7f82c9e00700 1 -- 172.21.15.39:0/3213830152 wait complete. 2024-03-18T16:05:46.811 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":80,"num_osds":12,"num_up_osds":12,"osd_up_since":1710777945,"num_in_osds":12,"osd_in_since":1710777924,"num_remapped_pgs":0} 2024-03-18T16:05:46.812 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph osd dump --format=json 2024-03-18T16:05:46.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:46 smithi039 ceph-mon[31721]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-03-18T16:05:46.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:46 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/3213830152' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-18T16:05:46.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:46 smithi039 ceph-mon[31721]: Detected new or changed devices on smithi196 2024-03-18T16:05:46.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:46 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:05:46.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:46 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:05:46.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:46 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:05:46.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:46 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:05:46.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:46 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:05:46.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:46 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:05:46.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:46 smithi039 ceph-mon[31721]: Adjusting osd_memory_target on smithi196 to 4270M 2024-03-18T16:05:46.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:46 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:05:46.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:46 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:05:46.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:46 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:05:46.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:46 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:05:46.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:46 smithi196 ceph-mon[36307]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-03-18T16:05:46.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:46 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/3213830152' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-18T16:05:46.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:46 smithi196 ceph-mon[36307]: Detected new or changed devices on smithi196 2024-03-18T16:05:46.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:46 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:05:46.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:46 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:05:46.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:46 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:05:46.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:46 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:05:46.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:46 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:05:46.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:46 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:05:46.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:46 smithi196 ceph-mon[36307]: Adjusting osd_memory_target on smithi196 to 4270M 2024-03-18T16:05:46.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:46 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:05:46.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:46 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:05:46.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:46 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:05:46.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:46 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:05:46.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:46 smithi194 ceph-mon[36200]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-03-18T16:05:46.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:46 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/3213830152' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-18T16:05:46.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:46 smithi194 ceph-mon[36200]: Detected new or changed devices on smithi196 2024-03-18T16:05:46.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:46 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:05:46.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:46 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:05:46.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:46 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:05:46.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:46 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:05:46.991 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:46 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:05:46.991 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:46 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:05:46.991 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:46 smithi194 ceph-mon[36200]: Adjusting osd_memory_target on smithi196 to 4270M 2024-03-18T16:05:46.991 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:46 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:05:46.991 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:46 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:05:46.991 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:46 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:05:46.991 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:46 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:05:47.240 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T16:05:47.688 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:47 smithi039 ceph-mon[31721]: osdmap e81: 12 total, 12 up, 12 in 2024-03-18T16:05:47.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:47 smithi196 ceph-mon[36307]: osdmap e81: 12 total, 12 up, 12 in 2024-03-18T16:05:47.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:47 smithi194 ceph-mon[36200]: osdmap e81: 12 total, 12 up, 12 in 2024-03-18T16:05:48.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:48 smithi039 ceph-mon[31721]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:05:48.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:48 smithi196 ceph-mon[36307]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:05:48.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:48 smithi194 ceph-mon[36200]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:05:49.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:49.445+0000 7fb9a6400700 1 -- 172.21.15.39:0/3876314095 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb9a006a920 msgr2=0x7fb9a010f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:05:49.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:49.445+0000 7fb9a6400700 1 --2- 172.21.15.39:0/3876314095 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb9a006a920 0x7fb9a010f160 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fb99800b0d0 tx=0x7fb99800b3e0 comp rx=0 tx=0).stop 2024-03-18T16:05:49.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:49.445+0000 7fb9a6400700 1 -- 172.21.15.39:0/3876314095 shutdown_connections 2024-03-18T16:05:49.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:49.445+0000 7fb9a6400700 1 --2- 172.21.15.39:0/3876314095 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb9a006a920 0x7fb9a010f160 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:49.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:49.445+0000 7fb9a6400700 1 --2- 172.21.15.39:0/3876314095 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb9a0108320 0x7fb9a0108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:49.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:49.445+0000 7fb9a6400700 1 --2- 172.21.15.39:0/3876314095 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb9a0069f80 0x7fb9a006a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:49.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:49.445+0000 7fb9a6400700 1 -- 172.21.15.39:0/3876314095 >> 172.21.15.39:0/3876314095 conn(0x7fb9a0078100 msgr2=0x7fb9a0078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:05:49.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:49.445+0000 7fb9a6400700 1 -- 172.21.15.39:0/3876314095 shutdown_connections 2024-03-18T16:05:49.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:49.445+0000 7fb9a6400700 1 -- 172.21.15.39:0/3876314095 wait complete. 2024-03-18T16:05:49.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:49.446+0000 7fb9a6400700 1 Processor -- start 2024-03-18T16:05:49.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:49.446+0000 7fb9a6400700 1 -- start start 2024-03-18T16:05:49.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:49.447+0000 7fb9a6400700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb9a0069f80 0x7fb9a01a08d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:49.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:49.447+0000 7fb9a6400700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb9a006a920 0x7fb9a01a0e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:49.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:49.447+0000 7fb9a6400700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb9a0108320 0x7fb9a019aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:49.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:49.447+0000 7fb9a6400700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fb9a01125f0 con 0x7fb9a0108320 2024-03-18T16:05:49.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:49.447+0000 7fb9a6400700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fb9a0112470 con 0x7fb9a0069f80 2024-03-18T16:05:49.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:49.447+0000 7fb9a6400700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fb9a0112770 con 0x7fb9a006a920 2024-03-18T16:05:49.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:49.447+0000 7fb99fe00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb9a0108320 0x7fb9a019aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:49.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:49.447+0000 7fb996a00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb9a006a920 0x7fb9a01a0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:49.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:49.447+0000 7fb99f400700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb9a0069f80 0x7fb9a01a08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:49.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:49.447+0000 7fb996a00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb9a006a920 0x7fb9a01a0e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:36746/0 (socket says 172.21.15.39:36746) 2024-03-18T16:05:49.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:49.447+0000 7fb99fe00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb9a0108320 0x7fb9a019aae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:42104/0 (socket says 172.21.15.39:42104) 2024-03-18T16:05:49.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:49.447+0000 7fb996a00700 1 -- 172.21.15.39:0/4025593964 learned_addr learned my addr 172.21.15.39:0/4025593964 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:05:49.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:49.448+0000 7fb99fe00700 1 -- 172.21.15.39:0/4025593964 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb9a006a920 msgr2=0x7fb9a01a0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:05:49.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:49.448+0000 7fb99fe00700 1 --2- 172.21.15.39:0/4025593964 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb9a006a920 0x7fb9a01a0e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:49.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:49.448+0000 7fb99fe00700 1 -- 172.21.15.39:0/4025593964 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb9a0069f80 msgr2=0x7fb9a01a08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:05:49.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:49.448+0000 7fb99fe00700 1 --2- 172.21.15.39:0/4025593964 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb9a0069f80 0x7fb9a01a08d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:49.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:49.448+0000 7fb99fe00700 1 -- 172.21.15.39:0/4025593964 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9a019b240 con 0x7fb9a0108320 2024-03-18T16:05:49.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:49.448+0000 7fb99fe00700 1 --2- 172.21.15.39:0/4025593964 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb9a0108320 0x7fb9a019aae0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb99800d010 tx=0x7fb998009810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:05:49.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:49.448+0000 7fb99cc00700 1 -- 172.21.15.39:0/4025593964 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb998003a00 con 0x7fb9a0108320 2024-03-18T16:05:49.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:49.448+0000 7fb99cc00700 1 -- 172.21.15.39:0/4025593964 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb998022070 con 0x7fb9a0108320 2024-03-18T16:05:49.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:49.448+0000 7fb9a6400700 1 -- 172.21.15.39:0/4025593964 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9a019b4d0 con 0x7fb9a0108320 2024-03-18T16:05:49.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:49.448+0000 7fb99cc00700 1 -- 172.21.15.39:0/4025593964 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb99801b6c0 con 0x7fb9a0108320 2024-03-18T16:05:49.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:49.449+0000 7fb9a6400700 1 -- 172.21.15.39:0/4025593964 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9a01a77a0 con 0x7fb9a0108320 2024-03-18T16:05:49.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:49.450+0000 7fb9a6400700 1 -- 172.21.15.39:0/4025593964 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9a0068f90 con 0x7fb9a0108320 2024-03-18T16:05:49.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:49.452+0000 7fb99cc00700 1 -- 172.21.15.39:0/4025593964 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fb998003ba0 con 0x7fb9a0108320 2024-03-18T16:05:49.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:49.453+0000 7fb99cc00700 1 --2- 172.21.15.39:0/4025593964 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb9840a2910 0x7fb9840a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:49.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:49.454+0000 7fb99cc00700 1 -- 172.21.15.39:0/4025593964 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fb99800f020 con 0x7fb9a0108320 2024-03-18T16:05:49.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:49.454+0000 7fb99f400700 1 --2- 172.21.15.39:0/4025593964 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb9840a2910 0x7fb9840a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:49.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:49.454+0000 7fb99f400700 1 --2- 172.21.15.39:0/4025593964 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb9840a2910 0x7fb9840a4dd0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb9a0107b30 tx=0x7fb990009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:05:49.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:49.456+0000 7fb99cc00700 1 -- 172.21.15.39:0/4025593964 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb99809caf0 con 0x7fb9a0108320 2024-03-18T16:05:49.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:49.685+0000 7fb9a6400700 1 -- 172.21.15.39:0/4025593964 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb9a01a7970 con 0x7fb9a0108320 2024-03-18T16:05:49.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:49.687+0000 7fb99cc00700 1 -- 172.21.15.39:0/4025593964 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18470 (secure 0 0 0) 0x7fb99809c3f0 con 0x7fb9a0108320 2024-03-18T16:05:49.689 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:05:49.690 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":81,"fsid":"1892d8fc-e540-11ee-95c9-87774f69a715","created":"2024-03-18T15:57:36.507008+0000","modified":"2024-03-18T16:05:46.544944+0000","last_up_change":"2024-03-18T16:05:45.250629+0000","last_in_change":"2024-03-18T16:05:24.841337+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T16:01:30.048247+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"771e0558-be62-4436-94b0-6364751911c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6803","nonce":4090743635}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6805","nonce":4090743635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6809","nonce":4090743635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6807","nonce":4090743635}]},"public_addr":"172.21.15.39:6803/4090743635","cluster_addr":"172.21.15.39:6805/4090743635","heartbeat_back_addr":"172.21.15.39:6809/4090743635","heartbeat_front_addr":"172.21.15.39:6807/4090743635","state":["exists","up"]},{"osd":1,"uuid":"0430eee1-cf34-4277-b255-c7d9f07ad19a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6801","nonce":2800329805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6803","nonce":2800329805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6807","nonce":2800329805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6805","nonce":2800329805}]},"public_addr":"172.21.15.194:6801/2800329805","cluster_addr":"172.21.15.194:6803/2800329805","heartbeat_back_addr":"172.21.15.194:6807/2800329805","heartbeat_front_addr":"172.21.15.194:6805/2800329805","state":["exists","up"]},{"osd":2,"uuid":"3a718da4-2511-4ec1-b1a8-eafd6cce0df0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6801","nonce":3935874719}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6803","nonce":3935874719}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6807","nonce":3935874719}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6805","nonce":3935874719}]},"public_addr":"172.21.15.196:6801/3935874719","cluster_addr":"172.21.15.196:6803/3935874719","heartbeat_back_addr":"172.21.15.196:6807/3935874719","heartbeat_front_addr":"172.21.15.196:6805/3935874719","state":["exists","up"]},{"osd":3,"uuid":"6f9c37f9-0ab9-4038-8919-afd4b39c4165","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6811","nonce":893345939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6813","nonce":893345939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6817","nonce":893345939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6815","nonce":893345939}]},"public_addr":"172.21.15.39:6811/893345939","cluster_addr":"172.21.15.39:6813/893345939","heartbeat_back_addr":"172.21.15.39:6817/893345939","heartbeat_front_addr":"172.21.15.39:6815/893345939","state":["exists","up"]},{"osd":4,"uuid":"e795bb0e-4791-4537-acda-1a321e19bc9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6809","nonce":800423088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6811","nonce":800423088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6815","nonce":800423088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6813","nonce":800423088}]},"public_addr":"172.21.15.194:6809/800423088","cluster_addr":"172.21.15.194:6811/800423088","heartbeat_back_addr":"172.21.15.194:6815/800423088","heartbeat_front_addr":"172.21.15.194:6813/800423088","state":["exists","up"]},{"osd":5,"uuid":"026e175e-6acb-4b4c-b235-8ccee8ae1149","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6809","nonce":2746224525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6811","nonce":2746224525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6815","nonce":2746224525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6813","nonce":2746224525}]},"public_addr":"172.21.15.196:6809/2746224525","cluster_addr":"172.21.15.196:6811/2746224525","heartbeat_back_addr":"172.21.15.196:6815/2746224525","heartbeat_front_addr":"172.21.15.196:6813/2746224525","state":["exists","up"]},{"osd":6,"uuid":"c1b183ff-383f-4c5c-9075-f76bab1c0bb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6819","nonce":1213371331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6821","nonce":1213371331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6825","nonce":1213371331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6823","nonce":1213371331}]},"public_addr":"172.21.15.39:6819/1213371331","cluster_addr":"172.21.15.39:6821/1213371331","heartbeat_back_addr":"172.21.15.39:6825/1213371331","heartbeat_front_addr":"172.21.15.39:6823/1213371331","state":["exists","up"]},{"osd":7,"uuid":"612f8697-957d-4ef4-a87f-241b36f9d13d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6817","nonce":3299162444}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6819","nonce":3299162444}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6823","nonce":3299162444}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6821","nonce":3299162444}]},"public_addr":"172.21.15.194:6817/3299162444","cluster_addr":"172.21.15.194:6819/3299162444","heartbeat_back_addr":"172.21.15.194:6823/3299162444","heartbeat_front_addr":"172.21.15.194:6821/3299162444","state":["exists","up"]},{"osd":8,"uuid":"836a9107-4d4f-4594-b8dc-0301e64f59d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6817","nonce":2456143145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6819","nonce":2456143145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6823","nonce":2456143145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6821","nonce":2456143145}]},"public_addr":"172.21.15.196:6817/2456143145","cluster_addr":"172.21.15.196:6819/2456143145","heartbeat_back_addr":"172.21.15.196:6823/2456143145","heartbeat_front_addr":"172.21.15.196:6821/2456143145","state":["exists","up"]},{"osd":9,"uuid":"4f30b960-12e3-4b16-bd28-fcc988139b58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6827","nonce":1848634433}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6829","nonce":1848634433}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6833","nonce":1848634433}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6831","nonce":1848634433}]},"public_addr":"172.21.15.39:6827/1848634433","cluster_addr":"172.21.15.39:6829/1848634433","heartbeat_back_addr":"172.21.15.39:6833/1848634433","heartbeat_front_addr":"172.21.15.39:6831/1848634433","state":["exists","up"]},{"osd":10,"uuid":"620f90e8-23e6-4bc0-8269-7219b9e1aebd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6825","nonce":3725064809}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6827","nonce":3725064809}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6831","nonce":3725064809}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6829","nonce":3725064809}]},"public_addr":"172.21.15.194:6825/3725064809","cluster_addr":"172.21.15.194:6827/3725064809","heartbeat_back_addr":"172.21.15.194:6831/3725064809","heartbeat_front_addr":"172.21.15.194:6829/3725064809","state":["exists","up"]},{"osd":11,"uuid":"867a2535-8235-4ceb-a698-eb86869d6d20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6825","nonce":2501212422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6827","nonce":2501212422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6831","nonce":2501212422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6829","nonce":2501212422}]},"public_addr":"172.21.15.196:6825/2501212422","cluster_addr":"172.21.15.196:6827/2501212422","heartbeat_back_addr":"172.21.15.196:6831/2501212422","heartbeat_front_addr":"172.21.15.196:6829/2501212422","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:00:36.931566+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:01:02.614024+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:01:26.193265+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:01:58.308833+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:02:25.049546+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:02:49.748750+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:03:23.582153+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:03:50.447321+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:04:15.812129+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:04:49.998047+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:05:17.087924+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:05:43.167637+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.39:0/3482086140":"2024-03-19T15:58:45.340294+0000","172.21.15.39:0/3844306284":"2024-03-19T15:58:45.340294+0000","172.21.15.39:6801/1810238546":"2024-03-19T15:58:45.340294+0000","172.21.15.39:6800/1810238546":"2024-03-19T15:58:45.340294+0000","172.21.15.39:0/2528397488":"2024-03-19T15:58:10.264798+0000","172.21.15.39:0/2240981254":"2024-03-19T15:58:10.264798+0000","172.21.15.39:6800/1217503189":"2024-03-19T15:58:10.264798+0000","172.21.15.39:0/3754876370":"2024-03-19T15:58:10.264798+0000","172.21.15.39:6801/1217503189":"2024-03-19T15:58:10.264798+0000","172.21.15.39:0/1874637628":"2024-03-19T15:58:45.340294+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T16:05:49.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:49.691+0000 7fb9a6400700 1 -- 172.21.15.39:0/4025593964 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb9840a2910 msgr2=0x7fb9840a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:05:49.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:49.691+0000 7fb9a6400700 1 --2- 172.21.15.39:0/4025593964 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb9840a2910 0x7fb9840a4dd0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb9a0107b30 tx=0x7fb990009260 comp rx=0 tx=0).stop 2024-03-18T16:05:49.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:49.691+0000 7fb9a6400700 1 -- 172.21.15.39:0/4025593964 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb9a0108320 msgr2=0x7fb9a019aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:05:49.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:49.691+0000 7fb9a6400700 1 --2- 172.21.15.39:0/4025593964 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb9a0108320 0x7fb9a019aae0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb99800d010 tx=0x7fb998009810 comp rx=0 tx=0).stop 2024-03-18T16:05:49.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:49.691+0000 7fb9a6400700 1 -- 172.21.15.39:0/4025593964 shutdown_connections 2024-03-18T16:05:49.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:49.691+0000 7fb9a6400700 1 --2- 172.21.15.39:0/4025593964 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb9840a2910 0x7fb9840a4dd0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:49.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:49.691+0000 7fb9a6400700 1 --2- 172.21.15.39:0/4025593964 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb9a0108320 0x7fb9a019aae0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:49.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:49.691+0000 7fb9a6400700 1 --2- 172.21.15.39:0/4025593964 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb9a0069f80 0x7fb9a01a08d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:49.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:49.691+0000 7fb9a6400700 1 --2- 172.21.15.39:0/4025593964 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb9a006a920 0x7fb9a01a0e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:49.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:49.691+0000 7fb9a6400700 1 -- 172.21.15.39:0/4025593964 >> 172.21.15.39:0/4025593964 conn(0x7fb9a0078100 msgr2=0x7fb9a00ffc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:05:49.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:49.691+0000 7fb9a6400700 1 -- 172.21.15.39:0/4025593964 shutdown_connections 2024-03-18T16:05:49.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:49.691+0000 7fb9a6400700 1 -- 172.21.15.39:0/4025593964 wait complete. 2024-03-18T16:05:50.697 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T16:01:30.048247+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T16:05:50.698 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-18T16:05:50.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:50 smithi194 ceph-mon[36200]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:05:50.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:50 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/4025593964' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:05:50.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:50 smithi039 ceph-mon[31721]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:05:50.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:50 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/4025593964' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:05:50.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:50 smithi196 ceph-mon[36307]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:05:50.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:50 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/4025593964' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:05:51.216 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T16:05:52.754 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:52 smithi039 ceph-mon[31721]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:05:52.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:52 smithi196 ceph-mon[36307]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:05:52.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:52 smithi194 ceph-mon[36200]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:05:53.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:53.551+0000 7ff1e5e00700 1 -- 172.21.15.39:0/1847475942 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff1e01088e0 msgr2=0x7ff1e0108d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:05:53.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:53.551+0000 7ff1e5e00700 1 --2- 172.21.15.39:0/1847475942 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff1e01088e0 0x7ff1e0108d60 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff1cc0098f0 tx=0x7ff1cc009c00 comp rx=0 tx=0).stop 2024-03-18T16:05:53.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:53.551+0000 7ff1e5e00700 1 -- 172.21.15.39:0/1847475942 shutdown_connections 2024-03-18T16:05:53.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:53.551+0000 7ff1e5e00700 1 --2- 172.21.15.39:0/1847475942 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff1e01092a0 0x7ff1e01157b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:53.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:53.551+0000 7ff1e5e00700 1 --2- 172.21.15.39:0/1847475942 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1e010e8e0 0x7ff1e010ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:53.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:53.551+0000 7ff1e5e00700 1 --2- 172.21.15.39:0/1847475942 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff1e01088e0 0x7ff1e0108d60 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:53.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:53.551+0000 7ff1e5e00700 1 -- 172.21.15.39:0/1847475942 >> 172.21.15.39:0/1847475942 conn(0x7ff1e0075aa0 msgr2=0x7ff1e0077ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:05:53.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:53.552+0000 7ff1e5e00700 1 -- 172.21.15.39:0/1847475942 shutdown_connections 2024-03-18T16:05:53.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:53.552+0000 7ff1e5e00700 1 -- 172.21.15.39:0/1847475942 wait complete. 2024-03-18T16:05:53.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:53.553+0000 7ff1e5e00700 1 Processor -- start 2024-03-18T16:05:53.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:53.553+0000 7ff1e5e00700 1 -- start start 2024-03-18T16:05:53.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:53.553+0000 7ff1e5e00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff1e01088e0 0x7ff1e019f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:53.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:53.554+0000 7ff1e5e00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff1e01092a0 0x7ff1e019f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:53.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:53.554+0000 7ff1e5e00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1e010e8e0 0x7ff1e01a3a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:53.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:53.554+0000 7ff1e5e00700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7ff1e0118cc0 con 0x7ff1e010e8e0 2024-03-18T16:05:53.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:53.554+0000 7ff1e5e00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ff1e0118b40 con 0x7ff1e01088e0 2024-03-18T16:05:53.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:53.554+0000 7ff1e5e00700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7ff1e0118e40 con 0x7ff1e01092a0 2024-03-18T16:05:53.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:53.554+0000 7ff1df400700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1e010e8e0 0x7ff1e01a3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:53.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:53.554+0000 7ff1de000700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff1e01092a0 0x7ff1e019f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:53.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:53.554+0000 7ff1de000700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff1e01092a0 0x7ff1e019f6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:36758/0 (socket says 172.21.15.39:36758) 2024-03-18T16:05:53.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:53.554+0000 7ff1dea00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff1e01088e0 0x7ff1e019f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:53.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:53.554+0000 7ff1de000700 1 -- 172.21.15.39:0/1024591517 learned_addr learned my addr 172.21.15.39:0/1024591517 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:05:53.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:53.554+0000 7ff1de000700 1 -- 172.21.15.39:0/1024591517 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff1e01088e0 msgr2=0x7ff1e019f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:05:53.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:53.554+0000 7ff1de000700 1 --2- 172.21.15.39:0/1024591517 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff1e01088e0 0x7ff1e019f170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:53.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:53.554+0000 7ff1de000700 1 -- 172.21.15.39:0/1024591517 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1e010e8e0 msgr2=0x7ff1e01a3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:05:53.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:53.554+0000 7ff1de000700 1 --2- 172.21.15.39:0/1024591517 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1e010e8e0 0x7ff1e01a3a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:53.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:53.554+0000 7ff1de000700 1 -- 172.21.15.39:0/1024591517 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1e01a4180 con 0x7ff1e01092a0 2024-03-18T16:05:53.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:53.554+0000 7ff1dea00700 1 --2- 172.21.15.39:0/1024591517 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff1e01088e0 0x7ff1e019f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:05:53.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:53.555+0000 7ff1de000700 1 --2- 172.21.15.39:0/1024591517 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff1e01092a0 0x7ff1e019f6b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff1cc005950 tx=0x7ff1cc00fb20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:05:53.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:53.555+0000 7ff1e4a00700 1 -- 172.21.15.39:0/1024591517 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1cc0051f0 con 0x7ff1e01092a0 2024-03-18T16:05:53.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:53.555+0000 7ff1e5e00700 1 -- 172.21.15.39:0/1024591517 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1e01a4410 con 0x7ff1e01092a0 2024-03-18T16:05:53.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:53.555+0000 7ff1e4a00700 1 -- 172.21.15.39:0/1024591517 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff1cc00b560 con 0x7ff1e01092a0 2024-03-18T16:05:53.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:53.555+0000 7ff1e4a00700 1 -- 172.21.15.39:0/1024591517 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1cc003a40 con 0x7ff1e01092a0 2024-03-18T16:05:53.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:53.555+0000 7ff1e5e00700 1 -- 172.21.15.39:0/1024591517 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1e01abcf0 con 0x7ff1e01092a0 2024-03-18T16:05:53.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:53.556+0000 7ff1e5e00700 1 -- 172.21.15.39:0/1024591517 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1e0068f90 con 0x7ff1e01092a0 2024-03-18T16:05:53.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:53.559+0000 7ff1e4a00700 1 -- 172.21.15.39:0/1024591517 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7ff1cc00ba80 con 0x7ff1e01092a0 2024-03-18T16:05:53.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:53.560+0000 7ff1e4a00700 1 --2- 172.21.15.39:0/1024591517 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff1c40a2890 0x7ff1c40a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:53.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:53.560+0000 7ff1e4a00700 1 -- 172.21.15.39:0/1024591517 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7ff1cc0cf790 con 0x7ff1e01092a0 2024-03-18T16:05:53.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:53.560+0000 7ff1dea00700 1 --2- 172.21.15.39:0/1024591517 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff1c40a2890 0x7ff1c40a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:53.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:53.561+0000 7ff1dea00700 1 --2- 172.21.15.39:0/1024591517 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff1c40a2890 0x7ff1c40a4d50 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ff1d0005fd0 tx=0x7ff1d0009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:05:53.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:53.562+0000 7ff1e4a00700 1 -- 172.21.15.39:0/1024591517 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff1cc05e740 con 0x7ff1e01092a0 2024-03-18T16:05:53.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:53.788+0000 7ff1e5e00700 1 -- 172.21.15.39:0/1024591517 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff1e01a04d0 con 0x7ff1e01092a0 2024-03-18T16:05:53.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:53.789+0000 7ff1e4a00700 1 -- 172.21.15.39:0/1024591517 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7ff1cc013030 con 0x7ff1e01092a0 2024-03-18T16:05:53.791 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2024-03-18T16:05:53.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:53.792+0000 7ff1e5e00700 1 -- 172.21.15.39:0/1024591517 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff1c40a2890 msgr2=0x7ff1c40a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:05:53.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:53.792+0000 7ff1e5e00700 1 --2- 172.21.15.39:0/1024591517 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff1c40a2890 0x7ff1c40a4d50 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ff1d0005fd0 tx=0x7ff1d0009310 comp rx=0 tx=0).stop 2024-03-18T16:05:53.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:53.792+0000 7ff1e5e00700 1 -- 172.21.15.39:0/1024591517 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff1e01092a0 msgr2=0x7ff1e019f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:05:53.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:53.792+0000 7ff1e5e00700 1 --2- 172.21.15.39:0/1024591517 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff1e01092a0 0x7ff1e019f6b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff1cc005950 tx=0x7ff1cc00fb20 comp rx=0 tx=0).stop 2024-03-18T16:05:53.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:53.793+0000 7ff1e5e00700 1 -- 172.21.15.39:0/1024591517 shutdown_connections 2024-03-18T16:05:53.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:53.793+0000 7ff1e5e00700 1 --2- 172.21.15.39:0/1024591517 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff1c40a2890 0x7ff1c40a4d50 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:53.795 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:53.793+0000 7ff1e5e00700 1 --2- 172.21.15.39:0/1024591517 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1e010e8e0 0x7ff1e01a3a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:53.795 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:53.793+0000 7ff1e5e00700 1 --2- 172.21.15.39:0/1024591517 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff1e01088e0 0x7ff1e019f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:53.795 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:53.793+0000 7ff1e5e00700 1 --2- 172.21.15.39:0/1024591517 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff1e01092a0 0x7ff1e019f6b0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:53.795 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:53.793+0000 7ff1e5e00700 1 -- 172.21.15.39:0/1024591517 >> 172.21.15.39:0/1024591517 conn(0x7ff1e0075aa0 msgr2=0x7ff1e0076040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:05:53.795 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:53.793+0000 7ff1e5e00700 1 -- 172.21.15.39:0/1024591517 shutdown_connections 2024-03-18T16:05:53.795 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:53.793+0000 7ff1e5e00700 1 -- 172.21.15.39:0/1024591517 wait complete. 2024-03-18T16:05:54.641 INFO:tasks.cephadm:Adding mds.a on smithi039 2024-03-18T16:05:54.641 INFO:tasks.cephadm:Adding mds.d on smithi039 2024-03-18T16:05:54.641 INFO:tasks.cephadm:Adding mds.g on smithi039 2024-03-18T16:05:54.641 INFO:tasks.cephadm:Adding mds.j on smithi039 2024-03-18T16:05:54.641 INFO:tasks.cephadm:Adding mds.b on smithi194 2024-03-18T16:05:54.641 INFO:tasks.cephadm:Adding mds.e on smithi194 2024-03-18T16:05:54.641 INFO:tasks.cephadm:Adding mds.h on smithi194 2024-03-18T16:05:54.641 INFO:tasks.cephadm:Adding mds.k on smithi194 2024-03-18T16:05:54.641 INFO:tasks.cephadm:Adding mds.c on smithi196 2024-03-18T16:05:54.641 INFO:tasks.cephadm:Adding mds.f on smithi196 2024-03-18T16:05:54.642 INFO:tasks.cephadm:Adding mds.i on smithi196 2024-03-18T16:05:54.642 INFO:tasks.cephadm:Adding mds.l on smithi196 2024-03-18T16:05:54.642 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph orch apply mds all '12;smithi039=a;smithi039=d;smithi039=g;smithi039=j;smithi194=b;smithi194=e;smithi194=h;smithi194=k;smithi196=c;smithi196=f;smithi196=i;smithi196=l' 2024-03-18T16:05:54.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:54 smithi039 ceph-mon[31721]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:05:54.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:54 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/1024591517' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T16:05:54.889 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T16:05:54.977 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:54 smithi196 ceph-mon[36307]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:05:54.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:54 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/1024591517' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T16:05:54.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:54 smithi194 ceph-mon[36200]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:05:54.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:54 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/1024591517' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T16:05:56.560 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:56.556+0000 7f41b0efe700 1 -- 172.21.15.196:0/2802822657 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f41ac1056c0 msgr2=0x7f41ac107ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:05:56.560 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:56.556+0000 7f41b0efe700 1 --2- 172.21.15.196:0/2802822657 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f41ac1056c0 0x7f41ac107ab0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f41a0009990 tx=0x7f41a0009ca0 comp rx=0 tx=0).stop 2024-03-18T16:05:56.560 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:56.556+0000 7f41b0efe700 1 -- 172.21.15.196:0/2802822657 shutdown_connections 2024-03-18T16:05:56.560 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:56.556+0000 7f41b0efe700 1 --2- 172.21.15.196:0/2802822657 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f41ac107ff0 0x7f41ac10a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:56.560 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:56.556+0000 7f41b0efe700 1 --2- 172.21.15.196:0/2802822657 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f41ac102d90 0x7f41ac105180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:56.561 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:56.556+0000 7f41b0efe700 1 --2- 172.21.15.196:0/2802822657 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f41ac1056c0 0x7f41ac107ab0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:56.561 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:56.556+0000 7f41b0efe700 1 -- 172.21.15.196:0/2802822657 >> 172.21.15.196:0/2802822657 conn(0x7f41ac0fc7c0 msgr2=0x7f41ac0febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:05:56.561 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:56.556+0000 7f41b0efe700 1 -- 172.21.15.196:0/2802822657 shutdown_connections 2024-03-18T16:05:56.561 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:56.556+0000 7f41b0efe700 1 -- 172.21.15.196:0/2802822657 wait complete. 2024-03-18T16:05:56.561 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:56.557+0000 7f41b0efe700 1 Processor -- start 2024-03-18T16:05:56.561 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:56.557+0000 7f41b0efe700 1 -- start start 2024-03-18T16:05:56.561 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:56.557+0000 7f41b0efe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f41ac102d90 0x7f41ac198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:56.561 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:56.557+0000 7f41b0efe700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f41ac1056c0 0x7f41ac198a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:56.561 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:56.557+0000 7f41b0efe700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f41ac107ff0 0x7f41ac19faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:56.561 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:56.557+0000 7f41b0efe700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f41ac10df50 con 0x7f41ac102d90 2024-03-18T16:05:56.561 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:56.557+0000 7f41b0efe700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f41ac10ddd0 con 0x7f41ac107ff0 2024-03-18T16:05:56.561 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:56.557+0000 7f41b0efe700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f41ac10e0d0 con 0x7f41ac1056c0 2024-03-18T16:05:56.561 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:56.558+0000 7f41a9d9b700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f41ac1056c0 0x7f41ac198a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:56.562 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:56.558+0000 7f41a9d9b700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f41ac1056c0 0x7f41ac198a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.196:49648/0 (socket says 172.21.15.196:49648) 2024-03-18T16:05:56.562 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:56.558+0000 7f41a9d9b700 1 -- 172.21.15.196:0/2715261516 learned_addr learned my addr 172.21.15.196:0/2715261516 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-03-18T16:05:56.562 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:56.558+0000 7f41aad9d700 1 --2- 172.21.15.196:0/2715261516 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f41ac107ff0 0x7f41ac19faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:56.562 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:56.558+0000 7f41a9d9b700 1 -- 172.21.15.196:0/2715261516 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f41ac107ff0 msgr2=0x7f41ac19faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:05:56.562 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:56.558+0000 7f41a9d9b700 1 --2- 172.21.15.196:0/2715261516 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f41ac107ff0 0x7f41ac19faa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:56.562 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:56.558+0000 7f41a9d9b700 1 -- 172.21.15.196:0/2715261516 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f41ac102d90 msgr2=0x7f41ac198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:05:56.562 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:56.558+0000 7f41a9d9b700 1 --2- 172.21.15.196:0/2715261516 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f41ac102d90 0x7f41ac198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:56.562 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:56.558+0000 7f41a9d9b700 1 -- 172.21.15.196:0/2715261516 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41ac1a0140 con 0x7f41ac1056c0 2024-03-18T16:05:56.562 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:56.558+0000 7f41a9d9b700 1 --2- 172.21.15.196:0/2715261516 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f41ac1056c0 0x7f41ac198a70 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f41a00047b0 tx=0x7f41a0009f50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:05:56.562 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:56.558+0000 7f419b7fe700 1 -- 172.21.15.196:0/2715261516 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41a0027070 con 0x7f41ac1056c0 2024-03-18T16:05:56.562 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:56.558+0000 7f419b7fe700 1 -- 172.21.15.196:0/2715261516 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41a0022470 con 0x7f41ac1056c0 2024-03-18T16:05:56.562 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:56.558+0000 7f419b7fe700 1 -- 172.21.15.196:0/2715261516 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41a000f6c0 con 0x7f41ac1056c0 2024-03-18T16:05:56.562 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:56.558+0000 7f41b0efe700 1 -- 172.21.15.196:0/2715261516 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41ac1a03d0 con 0x7f41ac1056c0 2024-03-18T16:05:56.563 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:56.558+0000 7f41b0efe700 1 -- 172.21.15.196:0/2715261516 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41ac1a0900 con 0x7f41ac1056c0 2024-03-18T16:05:56.563 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:56.559+0000 7f41b0efe700 1 -- 172.21.15.196:0/2715261516 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41ac068f90 con 0x7f41ac1056c0 2024-03-18T16:05:56.563 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:56.560+0000 7f419b7fe700 1 -- 172.21.15.196:0/2715261516 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f41a002a070 con 0x7f41ac1056c0 2024-03-18T16:05:56.564 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:56.561+0000 7f419b7fe700 1 --2- 172.21.15.196:0/2715261516 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f41940a2910 0x7f41940a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:56.564 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:56.561+0000 7f419b7fe700 1 -- 172.21.15.196:0/2715261516 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f41a00cfa40 con 0x7f41ac1056c0 2024-03-18T16:05:56.564 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:56.561+0000 7f41aa59c700 1 --2- 172.21.15.196:0/2715261516 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f41940a2910 0x7f41940a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:56.564 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:56.562+0000 7f41aa59c700 1 --2- 172.21.15.196:0/2715261516 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f41940a2910 0x7f41940a4dd0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f419c005fd0 tx=0x7f419c005d00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:05:56.566 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:56.565+0000 7f419b7fe700 1 -- 172.21.15.196:0/2715261516 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f41a0031070 con 0x7f41ac1056c0 2024-03-18T16:05:56.772 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:56.770+0000 7f41b0efe700 1 -- 172.21.15.196:0/2715261516 --> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi039=a;smithi039=d;smithi039=g;smithi039=j;smithi194=b;smithi194=e;smithi194=h;smithi194=k;smithi196=c;smithi196=f;smithi196=i;smithi196=l", "target": ["mon-mgr", ""]}) -- 0x7f41ac0628d0 con 0x7f41940a2910 2024-03-18T16:05:56.789 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:56.787+0000 7f419b7fe700 1 -- 172.21.15.196:0/2715261516 <== mgr.14152 v2:172.21.15.39:6800/2749214286 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7f41ac0628d0 con 0x7f41940a2910 2024-03-18T16:05:56.789 INFO:teuthology.orchestra.run.smithi196.stdout:Scheduled mds.all update... 2024-03-18T16:05:56.791 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:56.790+0000 7f41b0efe700 1 -- 172.21.15.196:0/2715261516 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f41940a2910 msgr2=0x7f41940a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:05:56.792 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:56.790+0000 7f41b0efe700 1 --2- 172.21.15.196:0/2715261516 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f41940a2910 0x7f41940a4dd0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f419c005fd0 tx=0x7f419c005d00 comp rx=0 tx=0).stop 2024-03-18T16:05:56.792 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:56.790+0000 7f41b0efe700 1 -- 172.21.15.196:0/2715261516 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f41ac1056c0 msgr2=0x7f41ac198a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:05:56.792 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:56.790+0000 7f41b0efe700 1 --2- 172.21.15.196:0/2715261516 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f41ac1056c0 0x7f41ac198a70 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f41a00047b0 tx=0x7f41a0009f50 comp rx=0 tx=0).stop 2024-03-18T16:05:56.792 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:56.790+0000 7f41b0efe700 1 -- 172.21.15.196:0/2715261516 shutdown_connections 2024-03-18T16:05:56.792 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:56.790+0000 7f41b0efe700 1 --2- 172.21.15.196:0/2715261516 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f41940a2910 0x7f41940a4dd0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:56.792 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:56.790+0000 7f41b0efe700 1 --2- 172.21.15.196:0/2715261516 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f41ac107ff0 0x7f41ac19faa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:56.792 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:56.790+0000 7f41b0efe700 1 --2- 172.21.15.196:0/2715261516 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f41ac102d90 0x7f41ac198530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:56.792 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:56.790+0000 7f41b0efe700 1 --2- 172.21.15.196:0/2715261516 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f41ac1056c0 0x7f41ac198a70 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:56.792 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:56.790+0000 7f41b0efe700 1 -- 172.21.15.196:0/2715261516 >> 172.21.15.196:0/2715261516 conn(0x7f41ac0fc7c0 msgr2=0x7f41ac1036d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:05:56.792 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:56.790+0000 7f41b0efe700 1 -- 172.21.15.196:0/2715261516 shutdown_connections 2024-03-18T16:05:56.792 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-18T16:05:56.790+0000 7f41b0efe700 1 -- 172.21.15.196:0/2715261516 wait complete. 2024-03-18T16:05:56.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:56 smithi039 ceph-mon[31721]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:05:56.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:56 smithi196 ceph-mon[36307]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:05:56.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:56 smithi194 ceph-mon[36200]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:05:57.427 DEBUG:teuthology.orchestra.run.smithi039:mds.a> sudo journalctl -f -n 0 -u ceph-1892d8fc-e540-11ee-95c9-87774f69a715@mds.a.service 2024-03-18T16:05:57.438 DEBUG:teuthology.orchestra.run.smithi039:mds.d> sudo journalctl -f -n 0 -u ceph-1892d8fc-e540-11ee-95c9-87774f69a715@mds.d.service 2024-03-18T16:05:57.487 DEBUG:teuthology.orchestra.run.smithi039:mds.g> sudo journalctl -f -n 0 -u ceph-1892d8fc-e540-11ee-95c9-87774f69a715@mds.g.service 2024-03-18T16:05:57.533 DEBUG:teuthology.orchestra.run.smithi039:mds.j> sudo journalctl -f -n 0 -u ceph-1892d8fc-e540-11ee-95c9-87774f69a715@mds.j.service 2024-03-18T16:05:57.577 DEBUG:teuthology.orchestra.run.smithi194:mds.b> sudo journalctl -f -n 0 -u ceph-1892d8fc-e540-11ee-95c9-87774f69a715@mds.b.service 2024-03-18T16:05:57.580 DEBUG:teuthology.orchestra.run.smithi194:mds.e> sudo journalctl -f -n 0 -u ceph-1892d8fc-e540-11ee-95c9-87774f69a715@mds.e.service 2024-03-18T16:05:57.623 DEBUG:teuthology.orchestra.run.smithi194:mds.h> sudo journalctl -f -n 0 -u ceph-1892d8fc-e540-11ee-95c9-87774f69a715@mds.h.service 2024-03-18T16:05:57.665 DEBUG:teuthology.orchestra.run.smithi194:mds.k> sudo journalctl -f -n 0 -u ceph-1892d8fc-e540-11ee-95c9-87774f69a715@mds.k.service 2024-03-18T16:05:57.708 DEBUG:teuthology.orchestra.run.smithi196:mds.c> sudo journalctl -f -n 0 -u ceph-1892d8fc-e540-11ee-95c9-87774f69a715@mds.c.service 2024-03-18T16:05:57.711 DEBUG:teuthology.orchestra.run.smithi196:mds.f> sudo journalctl -f -n 0 -u ceph-1892d8fc-e540-11ee-95c9-87774f69a715@mds.f.service 2024-03-18T16:05:57.753 DEBUG:teuthology.orchestra.run.smithi196:mds.i> sudo journalctl -f -n 0 -u ceph-1892d8fc-e540-11ee-95c9-87774f69a715@mds.i.service 2024-03-18T16:05:57.796 DEBUG:teuthology.orchestra.run.smithi196:mds.l> sudo journalctl -f -n 0 -u ceph-1892d8fc-e540-11ee-95c9-87774f69a715@mds.l.service 2024-03-18T16:05:57.839 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-03-18T16:05:57.840 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T16:05:57.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:57 smithi196 ceph-mon[36307]: from='client.24439 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi039=a;smithi039=d;smithi039=g;smithi039=j;smithi194=b;smithi194=e;smithi194=h;smithi194=k;smithi196=c;smithi196=f;smithi196=i;smithi196=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:05:57.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:57 smithi196 ceph-mon[36307]: Saving service mds.all spec with placement smithi039=a;smithi039=d;smithi039=g;smithi039=j;smithi194=b;smithi194=e;smithi194=h;smithi194=k;smithi196=c;smithi196=f;smithi196=i;smithi196=l;count:12 2024-03-18T16:05:57.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:57 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:05:57.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:57 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:05:57.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:57 smithi194 ceph-mon[36200]: from='client.24439 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi039=a;smithi039=d;smithi039=g;smithi039=j;smithi194=b;smithi194=e;smithi194=h;smithi194=k;smithi196=c;smithi196=f;smithi196=i;smithi196=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:05:57.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:57 smithi194 ceph-mon[36200]: Saving service mds.all spec with placement smithi039=a;smithi039=d;smithi039=g;smithi039=j;smithi194=b;smithi194=e;smithi194=h;smithi194=k;smithi196=c;smithi196=f;smithi196=i;smithi196=l;count:12 2024-03-18T16:05:57.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:57 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:05:57.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:57 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:05:58.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.073+0000 7fe084a00640 1 -- 172.21.15.39:0/208308976 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe080104970 msgr2=0x7fe080106d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:05:58.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.073+0000 7fe084a00640 1 --2- 172.21.15.39:0/208308976 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe080104970 0x7fe080106d60 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe06c009900 tx=0x7fe06c02bc30 comp rx=0 tx=0).stop 2024-03-18T16:05:58.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.074+0000 7fe084a00640 1 -- 172.21.15.39:0/208308976 shutdown_connections 2024-03-18T16:05:58.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.074+0000 7fe084a00640 1 --2- 172.21.15.39:0/208308976 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe0801072a0 0x7fe0801096f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:58.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.074+0000 7fe084a00640 1 --2- 172.21.15.39:0/208308976 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe080104970 0x7fe080106d60 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:58.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.074+0000 7fe084a00640 1 --2- 172.21.15.39:0/208308976 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe080102040 0x7fe080104430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:58.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.074+0000 7fe084a00640 1 -- 172.21.15.39:0/208308976 >> 172.21.15.39:0/208308976 conn(0x7fe0800fba30 msgr2=0x7fe0800fde70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:05:58.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.074+0000 7fe084a00640 1 -- 172.21.15.39:0/208308976 shutdown_connections 2024-03-18T16:05:58.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.074+0000 7fe084a00640 1 -- 172.21.15.39:0/208308976 wait complete. 2024-03-18T16:05:58.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.075+0000 7fe084a00640 1 Processor -- start 2024-03-18T16:05:58.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.075+0000 7fe084a00640 1 -- start start 2024-03-18T16:05:58.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.075+0000 7fe084a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe080102040 0x7fe080199b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:58.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.075+0000 7fe084a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe080104970 0x7fe08019a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:58.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.075+0000 7fe084a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe0801072a0 0x7fe0801a10b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:58.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.075+0000 7fe084a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fe08010d290 con 0x7fe0801072a0 2024-03-18T16:05:58.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.075+0000 7fe084a00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fe08010d110 con 0x7fe080102040 2024-03-18T16:05:58.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.075+0000 7fe084a00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fe08010d410 con 0x7fe080104970 2024-03-18T16:05:58.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.075+0000 7fe07ea00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe080104970 0x7fe08019a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:58.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.075+0000 7fe07f400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe080102040 0x7fe080199b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:58.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.076+0000 7fe07ea00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe080104970 0x7fe08019a040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:53914/0 (socket says 172.21.15.39:53914) 2024-03-18T16:05:58.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.076+0000 7fe07f400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe080102040 0x7fe080199b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:50030/0 (socket says 172.21.15.39:50030) 2024-03-18T16:05:58.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.076+0000 7fe07ea00640 1 -- 172.21.15.39:0/1320571241 learned_addr learned my addr 172.21.15.39:0/1320571241 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:05:58.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.076+0000 7fe07fe00640 1 --2- 172.21.15.39:0/1320571241 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe0801072a0 0x7fe0801a10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:58.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.076+0000 7fe07ea00640 1 -- 172.21.15.39:0/1320571241 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe080102040 msgr2=0x7fe080199b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:05:58.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.076+0000 7fe07ea00640 1 --2- 172.21.15.39:0/1320571241 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe080102040 0x7fe080199b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:58.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.076+0000 7fe07ea00640 1 -- 172.21.15.39:0/1320571241 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe0801072a0 msgr2=0x7fe0801a10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:05:58.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.076+0000 7fe07ea00640 1 --2- 172.21.15.39:0/1320571241 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe0801072a0 0x7fe0801a10b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:58.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.076+0000 7fe07ea00640 1 -- 172.21.15.39:0/1320571241 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0801a1720 con 0x7fe080104970 2024-03-18T16:05:58.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.076+0000 7fe07ea00640 1 --2- 172.21.15.39:0/1320571241 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe080104970 0x7fe08019a040 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe06c000c00 tx=0x7fe06c02c6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:05:58.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.077+0000 7fe077e00640 1 -- 172.21.15.39:0/1320571241 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe06c043020 con 0x7fe080104970 2024-03-18T16:05:58.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.077+0000 7fe077e00640 1 -- 172.21.15.39:0/1320571241 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe06c0054e0 con 0x7fe080104970 2024-03-18T16:05:58.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.077+0000 7fe084a00640 1 -- 172.21.15.39:0/1320571241 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0801a19b0 con 0x7fe080104970 2024-03-18T16:05:58.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.077+0000 7fe077e00640 1 -- 172.21.15.39:0/1320571241 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe06c003ce0 con 0x7fe080104970 2024-03-18T16:05:58.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.077+0000 7fe084a00640 1 -- 172.21.15.39:0/1320571241 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0801a1f20 con 0x7fe080104970 2024-03-18T16:05:58.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.079+0000 7fe084a00640 1 -- 172.21.15.39:0/1320571241 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe080068940 con 0x7fe080104970 2024-03-18T16:05:58.081 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.080+0000 7fe077e00640 1 -- 172.21.15.39:0/1320571241 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fe06c046020 con 0x7fe080104970 2024-03-18T16:05:58.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.082+0000 7fe077e00640 1 --2- 172.21.15.39:0/1320571241 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fe0640a2880 0x7fe0640a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:58.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.082+0000 7fe077e00640 1 -- 172.21.15.39:0/1320571241 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fe06c0f0680 con 0x7fe080104970 2024-03-18T16:05:58.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.082+0000 7fe07f400640 1 --2- 172.21.15.39:0/1320571241 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fe0640a2880 0x7fe0640a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:58.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.082+0000 7fe07f400640 1 --2- 172.21.15.39:0/1320571241 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fe0640a2880 0x7fe0640a4d40 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fe070009bd0 tx=0x7fe070005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:05:58.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:57 smithi039 ceph-mon[31721]: from='client.24439 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi039=a;smithi039=d;smithi039=g;smithi039=j;smithi194=b;smithi194=e;smithi194=h;smithi194=k;smithi196=c;smithi196=f;smithi196=i;smithi196=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:05:58.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:57 smithi039 ceph-mon[31721]: Saving service mds.all spec with placement smithi039=a;smithi039=d;smithi039=g;smithi039=j;smithi194=b;smithi194=e;smithi194=h;smithi194=k;smithi196=c;smithi196=f;smithi196=i;smithi196=l;count:12 2024-03-18T16:05:58.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:57 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:05:58.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:57 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:05:58.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.090+0000 7fe077e00640 1 -- 172.21.15.39:0/1320571241 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe06c04d020 con 0x7fe080104970 2024-03-18T16:05:58.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.297+0000 7fe084a00640 1 -- 172.21.15.39:0/1320571241 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe0800623e0 con 0x7fe080104970 2024-03-18T16:05:58.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.299+0000 7fe077e00640 1 -- 172.21.15.39:0/1320571241 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18470 (secure 0 0 0) 0x7fe06c0bdb20 con 0x7fe080104970 2024-03-18T16:05:58.301 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:05:58.301 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":81,"fsid":"1892d8fc-e540-11ee-95c9-87774f69a715","created":"2024-03-18T15:57:36.507008+0000","modified":"2024-03-18T16:05:46.544944+0000","last_up_change":"2024-03-18T16:05:45.250629+0000","last_in_change":"2024-03-18T16:05:24.841337+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T16:01:30.048247+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"771e0558-be62-4436-94b0-6364751911c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6803","nonce":4090743635}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6805","nonce":4090743635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6809","nonce":4090743635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6807","nonce":4090743635}]},"public_addr":"172.21.15.39:6803/4090743635","cluster_addr":"172.21.15.39:6805/4090743635","heartbeat_back_addr":"172.21.15.39:6809/4090743635","heartbeat_front_addr":"172.21.15.39:6807/4090743635","state":["exists","up"]},{"osd":1,"uuid":"0430eee1-cf34-4277-b255-c7d9f07ad19a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6801","nonce":2800329805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6803","nonce":2800329805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6807","nonce":2800329805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6805","nonce":2800329805}]},"public_addr":"172.21.15.194:6801/2800329805","cluster_addr":"172.21.15.194:6803/2800329805","heartbeat_back_addr":"172.21.15.194:6807/2800329805","heartbeat_front_addr":"172.21.15.194:6805/2800329805","state":["exists","up"]},{"osd":2,"uuid":"3a718da4-2511-4ec1-b1a8-eafd6cce0df0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6801","nonce":3935874719}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6803","nonce":3935874719}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6807","nonce":3935874719}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6805","nonce":3935874719}]},"public_addr":"172.21.15.196:6801/3935874719","cluster_addr":"172.21.15.196:6803/3935874719","heartbeat_back_addr":"172.21.15.196:6807/3935874719","heartbeat_front_addr":"172.21.15.196:6805/3935874719","state":["exists","up"]},{"osd":3,"uuid":"6f9c37f9-0ab9-4038-8919-afd4b39c4165","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6811","nonce":893345939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6813","nonce":893345939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6817","nonce":893345939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6815","nonce":893345939}]},"public_addr":"172.21.15.39:6811/893345939","cluster_addr":"172.21.15.39:6813/893345939","heartbeat_back_addr":"172.21.15.39:6817/893345939","heartbeat_front_addr":"172.21.15.39:6815/893345939","state":["exists","up"]},{"osd":4,"uuid":"e795bb0e-4791-4537-acda-1a321e19bc9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6809","nonce":800423088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6811","nonce":800423088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6815","nonce":800423088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6813","nonce":800423088}]},"public_addr":"172.21.15.194:6809/800423088","cluster_addr":"172.21.15.194:6811/800423088","heartbeat_back_addr":"172.21.15.194:6815/800423088","heartbeat_front_addr":"172.21.15.194:6813/800423088","state":["exists","up"]},{"osd":5,"uuid":"026e175e-6acb-4b4c-b235-8ccee8ae1149","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6809","nonce":2746224525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6811","nonce":2746224525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6815","nonce":2746224525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6813","nonce":2746224525}]},"public_addr":"172.21.15.196:6809/2746224525","cluster_addr":"172.21.15.196:6811/2746224525","heartbeat_back_addr":"172.21.15.196:6815/2746224525","heartbeat_front_addr":"172.21.15.196:6813/2746224525","state":["exists","up"]},{"osd":6,"uuid":"c1b183ff-383f-4c5c-9075-f76bab1c0bb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6819","nonce":1213371331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6821","nonce":1213371331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6825","nonce":1213371331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6823","nonce":1213371331}]},"public_addr":"172.21.15.39:6819/1213371331","cluster_addr":"172.21.15.39:6821/1213371331","heartbeat_back_addr":"172.21.15.39:6825/1213371331","heartbeat_front_addr":"172.21.15.39:6823/1213371331","state":["exists","up"]},{"osd":7,"uuid":"612f8697-957d-4ef4-a87f-241b36f9d13d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6817","nonce":3299162444}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6819","nonce":3299162444}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6823","nonce":3299162444}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6821","nonce":3299162444}]},"public_addr":"172.21.15.194:6817/3299162444","cluster_addr":"172.21.15.194:6819/3299162444","heartbeat_back_addr":"172.21.15.194:6823/3299162444","heartbeat_front_addr":"172.21.15.194:6821/3299162444","state":["exists","up"]},{"osd":8,"uuid":"836a9107-4d4f-4594-b8dc-0301e64f59d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6817","nonce":2456143145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6819","nonce":2456143145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6823","nonce":2456143145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6821","nonce":2456143145}]},"public_addr":"172.21.15.196:6817/2456143145","cluster_addr":"172.21.15.196:6819/2456143145","heartbeat_back_addr":"172.21.15.196:6823/2456143145","heartbeat_front_addr":"172.21.15.196:6821/2456143145","state":["exists","up"]},{"osd":9,"uuid":"4f30b960-12e3-4b16-bd28-fcc988139b58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6827","nonce":1848634433}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6829","nonce":1848634433}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6833","nonce":1848634433}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6831","nonce":1848634433}]},"public_addr":"172.21.15.39:6827/1848634433","cluster_addr":"172.21.15.39:6829/1848634433","heartbeat_back_addr":"172.21.15.39:6833/1848634433","heartbeat_front_addr":"172.21.15.39:6831/1848634433","state":["exists","up"]},{"osd":10,"uuid":"620f90e8-23e6-4bc0-8269-7219b9e1aebd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6825","nonce":3725064809}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6827","nonce":3725064809}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6831","nonce":3725064809}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6829","nonce":3725064809}]},"public_addr":"172.21.15.194:6825/3725064809","cluster_addr":"172.21.15.194:6827/3725064809","heartbeat_back_addr":"172.21.15.194:6831/3725064809","heartbeat_front_addr":"172.21.15.194:6829/3725064809","state":["exists","up"]},{"osd":11,"uuid":"867a2535-8235-4ceb-a698-eb86869d6d20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6825","nonce":2501212422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6827","nonce":2501212422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6831","nonce":2501212422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6829","nonce":2501212422}]},"public_addr":"172.21.15.196:6825/2501212422","cluster_addr":"172.21.15.196:6827/2501212422","heartbeat_back_addr":"172.21.15.196:6831/2501212422","heartbeat_front_addr":"172.21.15.196:6829/2501212422","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:00:36.931566+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:01:02.614024+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:01:26.193265+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:01:58.308833+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:02:25.049546+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:02:49.748750+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:03:23.582153+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:03:50.447321+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:04:15.812129+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:04:49.998047+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:05:17.087924+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:05:43.167637+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.39:0/3482086140":"2024-03-19T15:58:45.340294+0000","172.21.15.39:0/3844306284":"2024-03-19T15:58:45.340294+0000","172.21.15.39:6801/1810238546":"2024-03-19T15:58:45.340294+0000","172.21.15.39:6800/1810238546":"2024-03-19T15:58:45.340294+0000","172.21.15.39:0/2528397488":"2024-03-19T15:58:10.264798+0000","172.21.15.39:0/2240981254":"2024-03-19T15:58:10.264798+0000","172.21.15.39:6800/1217503189":"2024-03-19T15:58:10.264798+0000","172.21.15.39:0/3754876370":"2024-03-19T15:58:10.264798+0000","172.21.15.39:6801/1217503189":"2024-03-19T15:58:10.264798+0000","172.21.15.39:0/1874637628":"2024-03-19T15:58:45.340294+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T16:05:58.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.302+0000 7fe084a00640 1 -- 172.21.15.39:0/1320571241 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fe0640a2880 msgr2=0x7fe0640a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:05:58.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.302+0000 7fe084a00640 1 --2- 172.21.15.39:0/1320571241 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fe0640a2880 0x7fe0640a4d40 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fe070009bd0 tx=0x7fe070005f90 comp rx=0 tx=0).stop 2024-03-18T16:05:58.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.302+0000 7fe084a00640 1 -- 172.21.15.39:0/1320571241 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe080104970 msgr2=0x7fe08019a040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:05:58.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.302+0000 7fe084a00640 1 --2- 172.21.15.39:0/1320571241 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe080104970 0x7fe08019a040 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe06c000c00 tx=0x7fe06c02c6c0 comp rx=0 tx=0).stop 2024-03-18T16:05:58.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.302+0000 7fe084a00640 1 -- 172.21.15.39:0/1320571241 shutdown_connections 2024-03-18T16:05:58.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.302+0000 7fe084a00640 1 --2- 172.21.15.39:0/1320571241 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fe0640a2880 0x7fe0640a4d40 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:58.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.302+0000 7fe084a00640 1 --2- 172.21.15.39:0/1320571241 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe0801072a0 0x7fe0801a10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:58.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.302+0000 7fe084a00640 1 --2- 172.21.15.39:0/1320571241 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe080104970 0x7fe08019a040 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:58.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.302+0000 7fe084a00640 1 --2- 172.21.15.39:0/1320571241 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe080102040 0x7fe080199b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:58.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.302+0000 7fe084a00640 1 -- 172.21.15.39:0/1320571241 >> 172.21.15.39:0/1320571241 conn(0x7fe0800fba30 msgr2=0x7fe080101670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:05:58.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.302+0000 7fe084a00640 1 -- 172.21.15.39:0/1320571241 shutdown_connections 2024-03-18T16:05:58.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.302+0000 7fe084a00640 1 -- 172.21.15.39:0/1320571241 wait complete. 2024-03-18T16:05:58.324 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T16:01:30.048247+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T16:05:58.324 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T16:05:58.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.551+0000 7f004a800640 1 -- 172.21.15.39:0/1905869485 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0044108ff0 msgr2=0x7f0044109470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:05:58.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.551+0000 7f004a800640 1 --2- 172.21.15.39:0/1905869485 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0044108ff0 0x7f0044109470 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f0030009900 tx=0x7f003002bc30 comp rx=0 tx=0).stop 2024-03-18T16:05:58.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.551+0000 7f004a800640 1 -- 172.21.15.39:0/1905869485 shutdown_connections 2024-03-18T16:05:58.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.551+0000 7f004a800640 1 --2- 172.21.15.39:0/1905869485 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f00441099b0 0x7f0044110240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:58.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.551+0000 7f004a800640 1 --2- 172.21.15.39:0/1905869485 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0044108ff0 0x7f0044109470 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:58.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.551+0000 7f004a800640 1 --2- 172.21.15.39:0/1905869485 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0044107e00 0x7f0044108220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:58.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.551+0000 7f004a800640 1 -- 172.21.15.39:0/1905869485 >> 172.21.15.39:0/1905869485 conn(0x7f0044076da0 msgr2=0x7f00440791e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:05:58.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.551+0000 7f004a800640 1 -- 172.21.15.39:0/1905869485 shutdown_connections 2024-03-18T16:05:58.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.551+0000 7f004a800640 1 -- 172.21.15.39:0/1905869485 wait complete. 2024-03-18T16:05:58.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.552+0000 7f004a800640 1 Processor -- start 2024-03-18T16:05:58.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.552+0000 7f004a800640 1 -- start start 2024-03-18T16:05:58.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.553+0000 7f004a800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0044107e00 0x7f004419e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:58.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.553+0000 7f004a800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0044108ff0 0x7f004419e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:58.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.553+0000 7f004a800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00441099b0 0x7f00441a56b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:58.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.553+0000 7f004a800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f0044113a80 con 0x7f00441099b0 2024-03-18T16:05:58.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.553+0000 7f004a800640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f0044113900 con 0x7f0044108ff0 2024-03-18T16:05:58.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.553+0000 7f004a800640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f0044113c00 con 0x7f0044107e00 2024-03-18T16:05:58.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.553+0000 7f0043400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0044107e00 0x7f004419e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:58.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.553+0000 7f0043e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00441099b0 0x7f00441a56b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:58.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.553+0000 7f0043400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0044107e00 0x7f004419e100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:53932/0 (socket says 172.21.15.39:53932) 2024-03-18T16:05:58.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.553+0000 7f0043400640 1 -- 172.21.15.39:0/3739789338 learned_addr learned my addr 172.21.15.39:0/3739789338 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:05:58.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.553+0000 7f0042a00640 1 --2- 172.21.15.39:0/3739789338 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0044108ff0 0x7f004419e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:58.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.553+0000 7f0043400640 1 -- 172.21.15.39:0/3739789338 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0044108ff0 msgr2=0x7f004419e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:05:58.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.553+0000 7f0043400640 1 --2- 172.21.15.39:0/3739789338 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0044108ff0 0x7f004419e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:58.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.553+0000 7f0043400640 1 -- 172.21.15.39:0/3739789338 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00441099b0 msgr2=0x7f00441a56b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:05:58.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.553+0000 7f0043400640 1 --2- 172.21.15.39:0/3739789338 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00441099b0 0x7f00441a56b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:58.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.553+0000 7f0043400640 1 -- 172.21.15.39:0/3739789338 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00441a5d20 con 0x7f0044107e00 2024-03-18T16:05:58.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.553+0000 7f0043e00640 1 --2- 172.21.15.39:0/3739789338 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00441099b0 0x7f00441a56b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:05:58.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.554+0000 7f0043400640 1 --2- 172.21.15.39:0/3739789338 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0044107e00 0x7f004419e100 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f003400e9c0 tx=0x7f003400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:05:58.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.554+0000 7f003be00640 1 -- 172.21.15.39:0/3739789338 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f003400cde0 con 0x7f0044107e00 2024-03-18T16:05:58.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.554+0000 7f003be00640 1 -- 172.21.15.39:0/3739789338 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0034010910 con 0x7f0044107e00 2024-03-18T16:05:58.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.554+0000 7f004a800640 1 -- 172.21.15.39:0/3739789338 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00441a5fb0 con 0x7f0044107e00 2024-03-18T16:05:58.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.554+0000 7f003be00640 1 -- 172.21.15.39:0/3739789338 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0034018a10 con 0x7f0044107e00 2024-03-18T16:05:58.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.554+0000 7f004a800640 1 -- 172.21.15.39:0/3739789338 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00441a6470 con 0x7f0044107e00 2024-03-18T16:05:58.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.555+0000 7f004a800640 1 -- 172.21.15.39:0/3739789338 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f004410f120 con 0x7f0044107e00 2024-03-18T16:05:58.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.557+0000 7f003be00640 1 -- 172.21.15.39:0/3739789338 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f003401f020 con 0x7f0044107e00 2024-03-18T16:05:58.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.558+0000 7f003be00640 1 --2- 172.21.15.39:0/3739789338 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f00280a2800 0x7f00280a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:58.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.559+0000 7f003be00640 1 -- 172.21.15.39:0/3739789338 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f00340ce7e0 con 0x7f0044107e00 2024-03-18T16:05:58.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.559+0000 7f0042a00640 1 --2- 172.21.15.39:0/3739789338 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f00280a2800 0x7f00280a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:58.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.559+0000 7f0042a00640 1 --2- 172.21.15.39:0/3739789338 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f00280a2800 0x7f00280a4cc0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f003002c7b0 tx=0x7f0030005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:05:58.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.567+0000 7f003be00640 1 -- 172.21.15.39:0/3739789338 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f003409bc80 con 0x7f0044107e00 2024-03-18T16:05:58.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.774+0000 7f004a800640 1 -- 172.21.15.39:0/3739789338 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f0044062480 con 0x7f0044107e00 2024-03-18T16:05:58.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.775+0000 7f003be00640 1 -- 172.21.15.39:0/3739789338 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f003409b580 con 0x7f0044107e00 2024-03-18T16:05:58.777 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2024-03-18T16:05:58.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.778+0000 7f004a800640 1 -- 172.21.15.39:0/3739789338 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f00280a2800 msgr2=0x7f00280a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:05:58.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.778+0000 7f004a800640 1 --2- 172.21.15.39:0/3739789338 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f00280a2800 0x7f00280a4cc0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f003002c7b0 tx=0x7f0030005fd0 comp rx=0 tx=0).stop 2024-03-18T16:05:58.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.778+0000 7f004a800640 1 -- 172.21.15.39:0/3739789338 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0044107e00 msgr2=0x7f004419e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:05:58.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.778+0000 7f004a800640 1 --2- 172.21.15.39:0/3739789338 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0044107e00 0x7f004419e100 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f003400e9c0 tx=0x7f003400ee80 comp rx=0 tx=0).stop 2024-03-18T16:05:58.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.778+0000 7f004a800640 1 -- 172.21.15.39:0/3739789338 shutdown_connections 2024-03-18T16:05:58.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.778+0000 7f004a800640 1 --2- 172.21.15.39:0/3739789338 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0044108ff0 0x7f004419e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:58.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.778+0000 7f004a800640 1 --2- 172.21.15.39:0/3739789338 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f00280a2800 0x7f00280a4cc0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:58.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.778+0000 7f004a800640 1 --2- 172.21.15.39:0/3739789338 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f00441099b0 0x7f00441a56b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:58.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.778+0000 7f004a800640 1 --2- 172.21.15.39:0/3739789338 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0044107e00 0x7f004419e100 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:58.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.778+0000 7f004a800640 1 -- 172.21.15.39:0/3739789338 >> 172.21.15.39:0/3739789338 conn(0x7f0044076da0 msgr2=0x7f004410c2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:05:58.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.779+0000 7f004a800640 1 -- 172.21.15.39:0/3739789338 shutdown_connections 2024-03-18T16:05:58.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:58.779+0000 7f004a800640 1 -- 172.21.15.39:0/3739789338 wait complete. 2024-03-18T16:05:58.803 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:05:58.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:58 smithi196 ceph-mon[36307]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:05:58.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:58 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:05:58.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:58 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:05:58.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:58 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:05:58.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:58 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T16:05:58.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:58 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T16:05:58.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:58 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:05:58.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:58 smithi196 ceph-mon[36307]: Deploying daemon mds.l on smithi196 2024-03-18T16:05:58.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:58 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/1320571241' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:05:58.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:58 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/3739789338' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T16:05:59.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:59.031+0000 7f11f7200640 1 -- 172.21.15.39:0/761373634 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f11f8103810 msgr2=0x7f11f8103c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:05:59.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:59.031+0000 7f11f7200640 1 --2- 172.21.15.39:0/761373634 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f11f8103810 0x7f11f8103c30 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f11e8009900 tx=0x7f11e802bc30 comp rx=0 tx=0).stop 2024-03-18T16:05:59.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:59.031+0000 7f11f7200640 1 -- 172.21.15.39:0/761373634 shutdown_connections 2024-03-18T16:05:59.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:59.031+0000 7f11f7200640 1 --2- 172.21.15.39:0/761373634 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f11f8104a00 0x7f11f8104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:59.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:59.031+0000 7f11f7200640 1 --2- 172.21.15.39:0/761373634 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f11f81053c0 0x7f11f810bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:59.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:59.031+0000 7f11f7200640 1 --2- 172.21.15.39:0/761373634 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f11f8103810 0x7f11f8103c30 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:59.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:59.031+0000 7f11f7200640 1 -- 172.21.15.39:0/761373634 >> 172.21.15.39:0/761373634 conn(0x7f11f80fed80 msgr2=0x7f11f81011c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:05:59.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:59.031+0000 7f11f7200640 1 -- 172.21.15.39:0/761373634 shutdown_connections 2024-03-18T16:05:59.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:59.031+0000 7f11f7200640 1 -- 172.21.15.39:0/761373634 wait complete. 2024-03-18T16:05:59.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:59.032+0000 7f11f7200640 1 Processor -- start 2024-03-18T16:05:59.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:59.032+0000 7f11f7200640 1 -- start start 2024-03-18T16:05:59.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:59.032+0000 7f11f7200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f11f8103810 0x7f11f8199af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:59.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:59.032+0000 7f11f7200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f11f8104a00 0x7f11f819a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:59.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:59.032+0000 7f11f7200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f11f81053c0 0x7f11f81a10a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:59.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:59.032+0000 7f11f7200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f11f810f4b0 con 0x7f11f8104a00 2024-03-18T16:05:59.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:59.032+0000 7f11f7200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f11f810f330 con 0x7f11f81053c0 2024-03-18T16:05:59.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:59.032+0000 7f11f7200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f11f810f630 con 0x7f11f8103810 2024-03-18T16:05:59.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:59.032+0000 7f11ed400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f11f8104a00 0x7f11f819a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:59.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:59.032+0000 7f11ed400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f11f8104a00 0x7f11f819a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:33784/0 (socket says 172.21.15.39:33784) 2024-03-18T16:05:59.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:59.033+0000 7f11ed400640 1 -- 172.21.15.39:0/2590947870 learned_addr learned my addr 172.21.15.39:0/2590947870 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:05:59.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:59.033+0000 7f11f5e00640 1 --2- 172.21.15.39:0/2590947870 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f11f8103810 0x7f11f8199af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:59.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:59.033+0000 7f11f6800640 1 --2- 172.21.15.39:0/2590947870 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f11f81053c0 0x7f11f81a10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:59.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:59.033+0000 7f11ed400640 1 -- 172.21.15.39:0/2590947870 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f11f8103810 msgr2=0x7f11f8199af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:05:59.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:59.033+0000 7f11ed400640 1 --2- 172.21.15.39:0/2590947870 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f11f8103810 0x7f11f8199af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:59.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:59.033+0000 7f11ed400640 1 -- 172.21.15.39:0/2590947870 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f11f81053c0 msgr2=0x7f11f81a10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:05:59.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:59.033+0000 7f11ed400640 1 --2- 172.21.15.39:0/2590947870 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f11f81053c0 0x7f11f81a10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:59.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:59.033+0000 7f11ed400640 1 -- 172.21.15.39:0/2590947870 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11f81a1710 con 0x7f11f8104a00 2024-03-18T16:05:59.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:59.033+0000 7f11f6800640 1 --2- 172.21.15.39:0/2590947870 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f11f81053c0 0x7f11f81a10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:05:59.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:59.033+0000 7f11ed400640 1 --2- 172.21.15.39:0/2590947870 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f11f8104a00 0x7f11f819a030 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f11e000e9f0 tx=0x7f11e000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:05:59.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:59.034+0000 7f11ef400640 1 -- 172.21.15.39:0/2590947870 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11e000cd80 con 0x7f11f8104a00 2024-03-18T16:05:59.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:59.034+0000 7f11f7200640 1 -- 172.21.15.39:0/2590947870 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11f81a1a00 con 0x7f11f8104a00 2024-03-18T16:05:59.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:59.034+0000 7f11ef400640 1 -- 172.21.15.39:0/2590947870 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11e0010910 con 0x7f11f8104a00 2024-03-18T16:05:59.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:59.034+0000 7f11ef400640 1 -- 172.21.15.39:0/2590947870 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11e00188f0 con 0x7f11f8104a00 2024-03-18T16:05:59.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:59.034+0000 7f11f7200640 1 -- 172.21.15.39:0/2590947870 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11f81a1fa0 con 0x7f11f8104a00 2024-03-18T16:05:59.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:59.035+0000 7f11f7200640 1 -- 172.21.15.39:0/2590947870 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11f810ab30 con 0x7f11f8104a00 2024-03-18T16:05:59.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:59.038+0000 7f11ef400640 1 -- 172.21.15.39:0/2590947870 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f11e0010ac0 con 0x7f11f8104a00 2024-03-18T16:05:59.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:59.040+0000 7f11ef400640 1 --2- 172.21.15.39:0/2590947870 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f11d80a2880 0x7f11d80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:05:59.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:59.040+0000 7f11ef400640 1 -- 172.21.15.39:0/2590947870 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f11e00ced70 con 0x7f11f8104a00 2024-03-18T16:05:59.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:59.040+0000 7f11f5e00640 1 --2- 172.21.15.39:0/2590947870 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f11d80a2880 0x7f11d80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:05:59.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:59.041+0000 7f11f5e00640 1 --2- 172.21.15.39:0/2590947870 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f11d80a2880 0x7f11d80a4d40 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f11e802c170 tx=0x7f11e802df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:05:59.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:59.042+0000 7f11ef400640 1 -- 172.21.15.39:0/2590947870 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f11e009bbc0 con 0x7f11f8104a00 2024-03-18T16:05:59.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:58 smithi039 ceph-mon[31721]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:05:59.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:58 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:05:59.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:58 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:05:59.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:58 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:05:59.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:58 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T16:05:59.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:58 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T16:05:59.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:58 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:05:59.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:58 smithi039 ceph-mon[31721]: Deploying daemon mds.l on smithi196 2024-03-18T16:05:59.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:58 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/1320571241' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:05:59.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:58 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/3739789338' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T16:05:59.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:58 smithi194 ceph-mon[36200]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:05:59.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:58 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:05:59.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:58 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:05:59.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:58 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:05:59.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:58 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T16:05:59.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:58 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T16:05:59.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:58 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:05:59.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:58 smithi194 ceph-mon[36200]: Deploying daemon mds.l on smithi196 2024-03-18T16:05:59.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:58 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/1320571241' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:05:59.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:58 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/3739789338' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T16:05:59.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:59.302+0000 7f11f7200640 1 -- 172.21.15.39:0/2590947870 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f11f802d1b0 con 0x7f11f8104a00 2024-03-18T16:05:59.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:59.303+0000 7f11ef400640 1 -- 172.21.15.39:0/2590947870 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7f11e009fc60 con 0x7f11f8104a00 2024-03-18T16:05:59.305 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:05:59.305 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-03-18T16:05:59.305 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 1 2024-03-18T16:05:59.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:59.306+0000 7f11f7200640 1 -- 172.21.15.39:0/2590947870 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f11d80a2880 msgr2=0x7f11d80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:05:59.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:59.306+0000 7f11f7200640 1 --2- 172.21.15.39:0/2590947870 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f11d80a2880 0x7f11d80a4d40 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f11e802c170 tx=0x7f11e802df90 comp rx=0 tx=0).stop 2024-03-18T16:05:59.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:59.306+0000 7f11f7200640 1 -- 172.21.15.39:0/2590947870 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f11f8104a00 msgr2=0x7f11f819a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:05:59.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:59.306+0000 7f11f7200640 1 --2- 172.21.15.39:0/2590947870 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f11f8104a00 0x7f11f819a030 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f11e000e9f0 tx=0x7f11e000eeb0 comp rx=0 tx=0).stop 2024-03-18T16:05:59.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:59.306+0000 7f11f7200640 1 -- 172.21.15.39:0/2590947870 shutdown_connections 2024-03-18T16:05:59.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:59.306+0000 7f11f7200640 1 --2- 172.21.15.39:0/2590947870 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f11f81053c0 0x7f11f81a10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:59.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:59.306+0000 7f11f7200640 1 --2- 172.21.15.39:0/2590947870 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f11d80a2880 0x7f11d80a4d40 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:59.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:59.306+0000 7f11f7200640 1 --2- 172.21.15.39:0/2590947870 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f11f8104a00 0x7f11f819a030 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:59.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:59.306+0000 7f11f7200640 1 --2- 172.21.15.39:0/2590947870 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f11f8103810 0x7f11f8199af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:05:59.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:59.306+0000 7f11f7200640 1 -- 172.21.15.39:0/2590947870 >> 172.21.15.39:0/2590947870 conn(0x7f11f80fed80 msgr2=0x7f11f8107d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:05:59.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:59.307+0000 7f11f7200640 1 -- 172.21.15.39:0/2590947870 shutdown_connections 2024-03-18T16:05:59.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:05:59.307+0000 7f11f7200640 1 -- 172.21.15.39:0/2590947870 wait complete. 2024-03-18T16:06:00.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:05:59 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/2590947870' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:06:00.111 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:05:59 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/2590947870' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:06:00.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:05:59 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/2590947870' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:06:00.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:00 smithi196 ceph-mon[36307]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:06:00.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:00 smithi196 ceph-mon[36307]: mds.? [v2:172.21.15.196:6832/3874209621,v1:172.21.15.196:6833/3874209621] up:boot 2024-03-18T16:06:00.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:00 smithi196 ceph-mon[36307]: fsmap 1 up:standby 2024-03-18T16:06:00.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:00 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-18T16:06:00.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:00 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:00.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:00 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:00.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:00 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:00.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:00 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T16:06:00.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:00 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T16:06:00.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:00 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:06:01.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:00 smithi039 ceph-mon[31721]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:06:01.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:00 smithi039 ceph-mon[31721]: mds.? [v2:172.21.15.196:6832/3874209621,v1:172.21.15.196:6833/3874209621] up:boot 2024-03-18T16:06:01.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:00 smithi039 ceph-mon[31721]: fsmap 1 up:standby 2024-03-18T16:06:01.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:00 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-18T16:06:01.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:00 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:01.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:00 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:01.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:00 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:01.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:00 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T16:06:01.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:00 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T16:06:01.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:00 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:06:01.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:00 smithi194 ceph-mon[36200]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:06:01.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:00 smithi194 ceph-mon[36200]: mds.? [v2:172.21.15.196:6832/3874209621,v1:172.21.15.196:6833/3874209621] up:boot 2024-03-18T16:06:01.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:00 smithi194 ceph-mon[36200]: fsmap 1 up:standby 2024-03-18T16:06:01.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:00 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-18T16:06:01.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:00 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:01.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:00 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:01.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:00 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:01.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:00 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T16:06:01.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:00 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T16:06:01.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:00 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:06:01.328 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:06:01.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:01.556+0000 7fa646c00640 1 -- 172.21.15.39:0/3254840453 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa640103190 msgr2=0x7fa640109a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:01.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:01.556+0000 7fa646c00640 1 --2- 172.21.15.39:0/3254840453 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa640103190 0x7fa640109a20 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa638009be0 tx=0x7fa63802bef0 comp rx=0 tx=0).stop 2024-03-18T16:06:01.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:01.556+0000 7fa646c00640 1 -- 172.21.15.39:0/3254840453 shutdown_connections 2024-03-18T16:06:01.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:01.556+0000 7fa646c00640 1 --2- 172.21.15.39:0/3254840453 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa6401027d0 0x7fa640102c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:01.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:01.556+0000 7fa646c00640 1 --2- 172.21.15.39:0/3254840453 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa640103190 0x7fa640109a20 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:01.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:01.556+0000 7fa646c00640 1 --2- 172.21.15.39:0/3254840453 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa6401015e0 0x7fa640101a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:01.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:01.556+0000 7fa646c00640 1 -- 172.21.15.39:0/3254840453 >> 172.21.15.39:0/3254840453 conn(0x7fa6400fcb70 msgr2=0x7fa6400fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:01.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:01.556+0000 7fa646c00640 1 -- 172.21.15.39:0/3254840453 shutdown_connections 2024-03-18T16:06:01.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:01.556+0000 7fa646c00640 1 -- 172.21.15.39:0/3254840453 wait complete. 2024-03-18T16:06:01.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:01.557+0000 7fa646c00640 1 Processor -- start 2024-03-18T16:06:01.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:01.557+0000 7fa646c00640 1 -- start start 2024-03-18T16:06:01.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:01.558+0000 7fa646c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa6401015e0 0x7fa640075590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:01.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:01.558+0000 7fa646c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa6401027d0 0x7fa640073bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:01.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:01.558+0000 7fa646c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa640103190 0x7fa640074110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:01.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:01.558+0000 7fa646c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fa64010d240 con 0x7fa640103190 2024-03-18T16:06:01.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:01.558+0000 7fa646c00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fa64010d0c0 con 0x7fa6401027d0 2024-03-18T16:06:01.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:01.558+0000 7fa646c00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fa64010d3c0 con 0x7fa6401015e0 2024-03-18T16:06:01.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:01.558+0000 7fa646200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa640103190 0x7fa640074110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:01.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:01.558+0000 7fa646200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa640103190 0x7fa640074110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:33804/0 (socket says 172.21.15.39:33804) 2024-03-18T16:06:01.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:01.558+0000 7fa646200640 1 -- 172.21.15.39:0/2943501846 learned_addr learned my addr 172.21.15.39:0/2943501846 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:06:01.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:01.558+0000 7fa645800640 1 --2- 172.21.15.39:0/2943501846 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa6401015e0 0x7fa640075590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:01.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:01.558+0000 7fa644e00640 1 --2- 172.21.15.39:0/2943501846 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa6401027d0 0x7fa640073bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:01.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:01.559+0000 7fa646200640 1 -- 172.21.15.39:0/2943501846 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa6401015e0 msgr2=0x7fa640075590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:01.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:01.559+0000 7fa646200640 1 --2- 172.21.15.39:0/2943501846 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa6401015e0 0x7fa640075590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:01.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:01.559+0000 7fa646200640 1 -- 172.21.15.39:0/2943501846 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa6401027d0 msgr2=0x7fa640073bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:01.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:01.559+0000 7fa646200640 1 --2- 172.21.15.39:0/2943501846 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa6401027d0 0x7fa640073bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:01.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:01.559+0000 7fa646200640 1 -- 172.21.15.39:0/2943501846 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6400749a0 con 0x7fa640103190 2024-03-18T16:06:01.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:01.559+0000 7fa646200640 1 --2- 172.21.15.39:0/2943501846 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa640103190 0x7fa640074110 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fa638036040 tx=0x7fa6380337b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:01.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:01.559+0000 7fa63e000640 1 -- 172.21.15.39:0/2943501846 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa638043020 con 0x7fa640103190 2024-03-18T16:06:01.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:01.559+0000 7fa646c00640 1 -- 172.21.15.39:0/2943501846 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6401a1980 con 0x7fa640103190 2024-03-18T16:06:01.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:01.559+0000 7fa63e000640 1 -- 172.21.15.39:0/2943501846 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa63802ca60 con 0x7fa640103190 2024-03-18T16:06:01.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:01.559+0000 7fa63e000640 1 -- 172.21.15.39:0/2943501846 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa638034690 con 0x7fa640103190 2024-03-18T16:06:01.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:01.559+0000 7fa646c00640 1 -- 172.21.15.39:0/2943501846 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6401a1f20 con 0x7fa640103190 2024-03-18T16:06:01.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:01.561+0000 7fa646c00640 1 -- 172.21.15.39:0/2943501846 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa640108900 con 0x7fa640103190 2024-03-18T16:06:01.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:01.564+0000 7fa63e000640 1 -- 172.21.15.39:0/2943501846 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fa638046020 con 0x7fa640103190 2024-03-18T16:06:01.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:01.565+0000 7fa63e000640 1 --2- 172.21.15.39:0/2943501846 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa6280a2800 0x7fa6280a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:01.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:01.566+0000 7fa63e000640 1 -- 172.21.15.39:0/2943501846 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fa6380f1420 con 0x7fa640103190 2024-03-18T16:06:01.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:01.566+0000 7fa645800640 1 --2- 172.21.15.39:0/2943501846 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa6280a2800 0x7fa6280a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:01.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:01.566+0000 7fa645800640 1 --2- 172.21.15.39:0/2943501846 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa6280a2800 0x7fa6280a4cc0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fa640102630 tx=0x7fa634009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:01.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:01.568+0000 7fa63e000640 1 -- 172.21.15.39:0/2943501846 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa63804d020 con 0x7fa640103190 2024-03-18T16:06:01.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:01.813+0000 7fa646c00640 1 -- 172.21.15.39:0/2943501846 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa64002d1b0 con 0x7fa640103190 2024-03-18T16:06:01.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:01.817+0000 7fa63e000640 1 -- 172.21.15.39:0/2943501846 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7fa6380be8c0 con 0x7fa640103190 2024-03-18T16:06:01.819 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:06:01.819 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24463,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6833/3874209621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":3874209621},{"type":"v1","addr":"172.21.15.196:6833","nonce":3874209621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-18T16:06:01.819 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 2 2024-03-18T16:06:01.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:01.820+0000 7fa646c00640 1 -- 172.21.15.39:0/2943501846 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa6280a2800 msgr2=0x7fa6280a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:01.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:01.820+0000 7fa646c00640 1 --2- 172.21.15.39:0/2943501846 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa6280a2800 0x7fa6280a4cc0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fa640102630 tx=0x7fa634009260 comp rx=0 tx=0).stop 2024-03-18T16:06:01.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:01.820+0000 7fa646c00640 1 -- 172.21.15.39:0/2943501846 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa640103190 msgr2=0x7fa640074110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:01.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:01.820+0000 7fa646c00640 1 --2- 172.21.15.39:0/2943501846 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa640103190 0x7fa640074110 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fa638036040 tx=0x7fa6380337b0 comp rx=0 tx=0).stop 2024-03-18T16:06:01.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:01.821+0000 7fa646c00640 1 -- 172.21.15.39:0/2943501846 shutdown_connections 2024-03-18T16:06:01.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:01.821+0000 7fa646c00640 1 --2- 172.21.15.39:0/2943501846 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa6401027d0 0x7fa640073bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:01.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:01.821+0000 7fa646c00640 1 --2- 172.21.15.39:0/2943501846 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa6280a2800 0x7fa6280a4cc0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:01.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:01.821+0000 7fa646c00640 1 --2- 172.21.15.39:0/2943501846 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa640103190 0x7fa640074110 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:01.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:01.821+0000 7fa646c00640 1 --2- 172.21.15.39:0/2943501846 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa6401015e0 0x7fa640075590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:01.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:01.821+0000 7fa646c00640 1 -- 172.21.15.39:0/2943501846 >> 172.21.15.39:0/2943501846 conn(0x7fa6400fcb70 msgr2=0x7fa640105ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:01.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:01.821+0000 7fa646c00640 1 -- 172.21.15.39:0/2943501846 shutdown_connections 2024-03-18T16:06:01.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:01.821+0000 7fa646c00640 1 -- 172.21.15.39:0/2943501846 wait complete. 2024-03-18T16:06:01.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:01 smithi194 ceph-mon[36200]: Deploying daemon mds.k on smithi194 2024-03-18T16:06:02.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:01 smithi039 ceph-mon[31721]: Deploying daemon mds.k on smithi194 2024-03-18T16:06:02.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:01 smithi196 ceph-mon[36307]: Deploying daemon mds.k on smithi194 2024-03-18T16:06:03.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:02 smithi039 ceph-mon[31721]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:06:03.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:02 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/2943501846' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:06:03.207 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:02 smithi194 ceph-mon[36200]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:06:03.207 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:02 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/2943501846' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:06:03.207 INFO:journalctl@ceph.mds.k.smithi194.stdout:Mar 18 16:06:02 smithi194 systemd[1]: Starting Ceph mds.k for 1892d8fc-e540-11ee-95c9-87774f69a715... 2024-03-18T16:06:03.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:02 smithi196 ceph-mon[36307]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:06:03.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:02 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/2943501846' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:06:03.469 INFO:journalctl@ceph.mds.k.smithi194.stdout:Mar 18 16:06:03 smithi194 podman[66340]: 2024-03-18 16:06:03.108253888 +0000 UTC m=+0.018018562 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T16:06:03.469 INFO:journalctl@ceph.mds.k.smithi194.stdout:Mar 18 16:06:03 smithi194 podman[66340]: 2024-03-18 16:06:03.224170768 +0000 UTC m=+0.133935427 container create e2b93397987b5b558f9b45764540063e06cd63ba9e482acef598b9dbb7819433 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-k, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-18T16:06:03.740 INFO:journalctl@ceph.mds.k.smithi194.stdout:Mar 18 16:06:03 smithi194 podman[66340]: 2024-03-18 16:06:03.468542809 +0000 UTC m=+0.378307472 container init e2b93397987b5b558f9b45764540063e06cd63ba9e482acef598b9dbb7819433 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-k, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-18T16:06:03.740 INFO:journalctl@ceph.mds.k.smithi194.stdout:Mar 18 16:06:03 smithi194 podman[66340]: 2024-03-18 16:06:03.471427832 +0000 UTC m=+0.381192484 container start e2b93397987b5b558f9b45764540063e06cd63ba9e482acef598b9dbb7819433 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-k, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-18T16:06:03.740 INFO:journalctl@ceph.mds.k.smithi194.stdout:Mar 18 16:06:03 smithi194 ceph-mds[66360]: -- 172.21.15.194:0/169246996 <== mon.2 v2:172.21.15.194:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56167d81fa40 con 0x56167d7f2c00 2024-03-18T16:06:03.740 INFO:journalctl@ceph.mds.k.smithi194.stdout:Mar 18 16:06:03 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-k[66356]: starting mds.k at 2024-03-18T16:06:03.740 INFO:journalctl@ceph.mds.k.smithi194.stdout:Mar 18 16:06:03 smithi194 bash[66340]: e2b93397987b5b558f9b45764540063e06cd63ba9e482acef598b9dbb7819433 2024-03-18T16:06:03.740 INFO:journalctl@ceph.mds.k.smithi194.stdout:Mar 18 16:06:03 smithi194 systemd[1]: Started Ceph mds.k for 1892d8fc-e540-11ee-95c9-87774f69a715. 2024-03-18T16:06:03.840 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:06:04.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:04.039+0000 7f75ba200640 1 -- 172.21.15.39:0/751080153 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f75b4104bf0 msgr2=0x7f75b4105070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:04.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:04.039+0000 7f75ba200640 1 --2- 172.21.15.39:0/751080153 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f75b4104bf0 0x7f75b4105070 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f759c009900 tx=0x7f759c02bc30 comp rx=0 tx=0).stop 2024-03-18T16:06:04.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:04.039+0000 7f75ba200640 1 -- 172.21.15.39:0/751080153 shutdown_connections 2024-03-18T16:06:04.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:04.039+0000 7f75ba200640 1 --2- 172.21.15.39:0/751080153 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f75b41055b0 0x7f75b410be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:04.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:04.039+0000 7f75ba200640 1 --2- 172.21.15.39:0/751080153 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f75b4104bf0 0x7f75b4105070 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:04.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:04.039+0000 7f75ba200640 1 --2- 172.21.15.39:0/751080153 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f75b4103a00 0x7f75b4103e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:04.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:04.039+0000 7f75ba200640 1 -- 172.21.15.39:0/751080153 >> 172.21.15.39:0/751080153 conn(0x7f75b40fef90 msgr2=0x7f75b41013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:04.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:04.039+0000 7f75ba200640 1 -- 172.21.15.39:0/751080153 shutdown_connections 2024-03-18T16:06:04.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:04.039+0000 7f75ba200640 1 -- 172.21.15.39:0/751080153 wait complete. 2024-03-18T16:06:04.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:04.040+0000 7f75ba200640 1 Processor -- start 2024-03-18T16:06:04.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:04.040+0000 7f75ba200640 1 -- start start 2024-03-18T16:06:04.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:04.040+0000 7f75ba200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f75b4103a00 0x7f75b4199cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:04.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:04.040+0000 7f75ba200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f75b4104bf0 0x7f75b419a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:04.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:04.040+0000 7f75ba200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f75b41055b0 0x7f75b41a1260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:04.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:04.041+0000 7f75ba200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f75b410f660 con 0x7f75b4103a00 2024-03-18T16:06:04.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:04.041+0000 7f75ba200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f75b410f4e0 con 0x7f75b41055b0 2024-03-18T16:06:04.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:04.041+0000 7f75ba200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f75b410f7e0 con 0x7f75b4104bf0 2024-03-18T16:06:04.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:04.041+0000 7f75b3400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f75b4103a00 0x7f75b4199cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:04.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:04.041+0000 7f75b3400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f75b4103a00 0x7f75b4199cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:33816/0 (socket says 172.21.15.39:33816) 2024-03-18T16:06:04.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:04.041+0000 7f75b3400640 1 -- 172.21.15.39:0/3152995461 learned_addr learned my addr 172.21.15.39:0/3152995461 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:06:04.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:04.041+0000 7f75b3e00640 1 --2- 172.21.15.39:0/3152995461 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f75b41055b0 0x7f75b41a1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:04.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:04.041+0000 7f75aaa00640 1 --2- 172.21.15.39:0/3152995461 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f75b4104bf0 0x7f75b419a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:04.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:04.041+0000 7f75b3400640 1 -- 172.21.15.39:0/3152995461 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f75b4104bf0 msgr2=0x7f75b419a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:04.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:04.041+0000 7f75b3400640 1 --2- 172.21.15.39:0/3152995461 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f75b4104bf0 0x7f75b419a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:04.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:04.041+0000 7f75b3400640 1 -- 172.21.15.39:0/3152995461 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f75b41055b0 msgr2=0x7f75b41a1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:04.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:04.041+0000 7f75b3400640 1 --2- 172.21.15.39:0/3152995461 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f75b41055b0 0x7f75b41a1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:04.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:04.041+0000 7f75b3400640 1 -- 172.21.15.39:0/3152995461 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75b41a1900 con 0x7f75b4103a00 2024-03-18T16:06:04.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:04.041+0000 7f75b3e00640 1 --2- 172.21.15.39:0/3152995461 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f75b41055b0 0x7f75b41a1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:06:04.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:04.042+0000 7f75b3400640 1 --2- 172.21.15.39:0/3152995461 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f75b4103a00 0x7f75b4199cf0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f75a400e9c0 tx=0x7f75a400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:04.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:04.042+0000 7f75b0c00640 1 -- 172.21.15.39:0/3152995461 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a400cd80 con 0x7f75b4103a00 2024-03-18T16:06:04.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:04.042+0000 7f75b0c00640 1 -- 172.21.15.39:0/3152995461 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f75a40035f0 con 0x7f75b4103a00 2024-03-18T16:06:04.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:04.042+0000 7f75b0c00640 1 -- 172.21.15.39:0/3152995461 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a4010690 con 0x7f75b4103a00 2024-03-18T16:06:04.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:04.042+0000 7f75ba200640 1 -- 172.21.15.39:0/3152995461 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75b41a1bf0 con 0x7f75b4103a00 2024-03-18T16:06:04.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:04.042+0000 7f75ba200640 1 -- 172.21.15.39:0/3152995461 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75b41a2140 con 0x7f75b4103a00 2024-03-18T16:06:04.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:04.044+0000 7f75ba200640 1 -- 172.21.15.39:0/3152995461 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75b410ad20 con 0x7f75b4103a00 2024-03-18T16:06:04.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:04.045+0000 7f75b0c00640 1 -- 172.21.15.39:0/3152995461 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f75a40037a0 con 0x7f75b4103a00 2024-03-18T16:06:04.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:04.047+0000 7f75b0c00640 1 --2- 172.21.15.39:0/3152995461 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f75980a2800 0x7f75980a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:04.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:04.048+0000 7f75b0c00640 1 -- 172.21.15.39:0/3152995461 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f75a40cea80 con 0x7f75b4103a00 2024-03-18T16:06:04.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:04.048+0000 7f75aaa00640 1 --2- 172.21.15.39:0/3152995461 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f75980a2800 0x7f75980a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:04.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:04.049+0000 7f75aaa00640 1 --2- 172.21.15.39:0/3152995461 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f75980a2800 0x7f75980a4cc0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f759c02c7b0 tx=0x7f759c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:04.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:04.052+0000 7f75b0c00640 1 -- 172.21.15.39:0/3152995461 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f75a40d0e20 con 0x7f75b4103a00 2024-03-18T16:06:04.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:04.315+0000 7f75ba200640 1 -- 172.21.15.39:0/3152995461 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f75b40623e0 con 0x7f75b4103a00 2024-03-18T16:06:04.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:04.316+0000 7f75b0c00640 1 -- 172.21.15.39:0/3152995461 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2221 (secure 0 0 0) 0x7f75a409bf20 con 0x7f75b4103a00 2024-03-18T16:06:04.319 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:06:04.319 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/931350738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":931350738},{"type":"v1","addr":"172.21.15.194:6833","nonce":931350738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24463,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6833/3874209621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":3874209621},{"type":"v1","addr":"172.21.15.196:6833","nonce":3874209621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-18T16:06:04.319 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 3 2024-03-18T16:06:04.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:04.319+0000 7f75ba200640 1 -- 172.21.15.39:0/3152995461 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f75980a2800 msgr2=0x7f75980a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:04.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:04.319+0000 7f75ba200640 1 --2- 172.21.15.39:0/3152995461 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f75980a2800 0x7f75980a4cc0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f759c02c7b0 tx=0x7f759c005fd0 comp rx=0 tx=0).stop 2024-03-18T16:06:04.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:04.319+0000 7f75ba200640 1 -- 172.21.15.39:0/3152995461 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f75b4103a00 msgr2=0x7f75b4199cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:04.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:04.319+0000 7f75ba200640 1 --2- 172.21.15.39:0/3152995461 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f75b4103a00 0x7f75b4199cf0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f75a400e9c0 tx=0x7f75a400ee80 comp rx=0 tx=0).stop 2024-03-18T16:06:04.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:04.319+0000 7f75ba200640 1 -- 172.21.15.39:0/3152995461 shutdown_connections 2024-03-18T16:06:04.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:04.319+0000 7f75ba200640 1 --2- 172.21.15.39:0/3152995461 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f75b41055b0 0x7f75b41a1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:04.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:04.319+0000 7f75ba200640 1 --2- 172.21.15.39:0/3152995461 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f75980a2800 0x7f75980a4cc0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:04.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:04.319+0000 7f75ba200640 1 --2- 172.21.15.39:0/3152995461 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f75b4104bf0 0x7f75b419a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:04.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:04.319+0000 7f75ba200640 1 --2- 172.21.15.39:0/3152995461 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f75b4103a00 0x7f75b4199cf0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:04.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:04.319+0000 7f75ba200640 1 -- 172.21.15.39:0/3152995461 >> 172.21.15.39:0/3152995461 conn(0x7f75b40fef90 msgr2=0x7f75b4107ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:04.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:04.319+0000 7f75ba200640 1 -- 172.21.15.39:0/3152995461 shutdown_connections 2024-03-18T16:06:04.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:04.319+0000 7f75ba200640 1 -- 172.21.15.39:0/3152995461 wait complete. 2024-03-18T16:06:04.880 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:04 smithi039 ceph-mon[31721]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:06:04.880 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:04 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:04.880 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:04 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:04.880 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:04 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:04.880 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:04 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T16:06:04.880 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:04 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T16:06:04.880 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:04 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:06:04.880 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:04 smithi039 ceph-mon[31721]: Deploying daemon mds.j on smithi039 2024-03-18T16:06:04.880 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:04 smithi039 ceph-mon[31721]: mds.? [v2:172.21.15.194:6832/931350738,v1:172.21.15.194:6833/931350738] up:boot 2024-03-18T16:06:04.880 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:04 smithi039 ceph-mon[31721]: fsmap 2 up:standby 2024-03-18T16:06:04.880 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:04 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-18T16:06:04.880 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:04 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/3152995461' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:06:04.886 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:04 smithi194 ceph-mon[36200]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:06:04.886 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:04 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:04.886 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:04 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:04.886 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:04 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:04.886 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:04 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T16:06:04.887 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:04 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T16:06:04.887 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:04 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:06:04.887 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:04 smithi194 ceph-mon[36200]: Deploying daemon mds.j on smithi039 2024-03-18T16:06:04.887 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:04 smithi194 ceph-mon[36200]: mds.? [v2:172.21.15.194:6832/931350738,v1:172.21.15.194:6833/931350738] up:boot 2024-03-18T16:06:04.887 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:04 smithi194 ceph-mon[36200]: fsmap 2 up:standby 2024-03-18T16:06:04.887 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:04 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-18T16:06:04.887 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:04 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/3152995461' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:06:04.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:04 smithi196 ceph-mon[36307]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:06:04.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:04 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:04.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:04 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:04.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:04 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:04.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:04 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T16:06:04.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:04 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T16:06:04.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:04 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:06:04.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:04 smithi196 ceph-mon[36307]: Deploying daemon mds.j on smithi039 2024-03-18T16:06:04.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:04 smithi196 ceph-mon[36307]: mds.? [v2:172.21.15.194:6832/931350738,v1:172.21.15.194:6833/931350738] up:boot 2024-03-18T16:06:04.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:04 smithi196 ceph-mon[36307]: fsmap 2 up:standby 2024-03-18T16:06:04.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:04 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-18T16:06:04.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:04 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/3152995461' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:06:06.343 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:06:06.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:06.584+0000 7f484a800640 1 -- 172.21.15.39:0/550196949 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4844102380 msgr2=0x7f4844104770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:06.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:06.584+0000 7f484a800640 1 --2- 172.21.15.39:0/550196949 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4844102380 0x7f4844104770 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4834009900 tx=0x7f483402bc50 comp rx=0 tx=0).stop 2024-03-18T16:06:06.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:06.585+0000 7f484a800640 1 -- 172.21.15.39:0/550196949 shutdown_connections 2024-03-18T16:06:06.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:06.585+0000 7f484a800640 1 --2- 172.21.15.39:0/550196949 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f48441075e0 0x7f48441099d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:06.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:06.585+0000 7f484a800640 1 --2- 172.21.15.39:0/550196949 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4844104cb0 0x7f48441070a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:06.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:06.585+0000 7f484a800640 1 --2- 172.21.15.39:0/550196949 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4844102380 0x7f4844104770 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:06.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:06.585+0000 7f484a800640 1 -- 172.21.15.39:0/550196949 >> 172.21.15.39:0/550196949 conn(0x7f48440fbd10 msgr2=0x7f48440fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:06.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:06.585+0000 7f484a800640 1 -- 172.21.15.39:0/550196949 shutdown_connections 2024-03-18T16:06:06.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:06.585+0000 7f484a800640 1 -- 172.21.15.39:0/550196949 wait complete. 2024-03-18T16:06:06.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:06.586+0000 7f484a800640 1 Processor -- start 2024-03-18T16:06:06.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:06.586+0000 7f484a800640 1 -- start start 2024-03-18T16:06:06.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:06.586+0000 7f484a800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4844102380 0x7f4844197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:06.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:06.586+0000 7f484a800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4844104cb0 0x7f4844198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:06.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:06.586+0000 7f484a800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f48441075e0 0x7f484419f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:06.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:06.586+0000 7f484a800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f484410d4d0 con 0x7f4844104cb0 2024-03-18T16:06:06.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:06.586+0000 7f484a800640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f484410d350 con 0x7f48441075e0 2024-03-18T16:06:06.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:06.586+0000 7f484a800640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f484410d650 con 0x7f4844102380 2024-03-18T16:06:06.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:06.586+0000 7f4842a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4844104cb0 0x7f4844198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:06.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:06.586+0000 7f4843400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4844102380 0x7f4844197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:06.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:06.586+0000 7f4843400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4844102380 0x7f4844197b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:54006/0 (socket says 172.21.15.39:54006) 2024-03-18T16:06:06.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:06.586+0000 7f4843400640 1 -- 172.21.15.39:0/3643219293 learned_addr learned my addr 172.21.15.39:0/3643219293 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:06:06.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:06.586+0000 7f4843e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f48441075e0 0x7f484419f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:06.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:06.587+0000 7f4843400640 1 -- 172.21.15.39:0/3643219293 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f48441075e0 msgr2=0x7f484419f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:06.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:06.587+0000 7f4843400640 1 --2- 172.21.15.39:0/3643219293 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f48441075e0 0x7f484419f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:06.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:06.587+0000 7f4843400640 1 -- 172.21.15.39:0/3643219293 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4844104cb0 msgr2=0x7f4844198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:06.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:06.587+0000 7f4843400640 1 --2- 172.21.15.39:0/3643219293 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4844104cb0 0x7f4844198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:06.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:06.587+0000 7f4843400640 1 -- 172.21.15.39:0/3643219293 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f484419f740 con 0x7f4844102380 2024-03-18T16:06:06.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:06.587+0000 7f4843400640 1 --2- 172.21.15.39:0/3643219293 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4844102380 0x7f4844197b20 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f483402c260 tx=0x7f483402c670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:06.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:06.588+0000 7f483be00640 1 -- 172.21.15.39:0/3643219293 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4834041070 con 0x7f4844102380 2024-03-18T16:06:06.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:06.588+0000 7f484a800640 1 -- 172.21.15.39:0/3643219293 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f484419f9d0 con 0x7f4844102380 2024-03-18T16:06:06.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:06.588+0000 7f484a800640 1 -- 172.21.15.39:0/3643219293 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f484419ff70 con 0x7f4844102380 2024-03-18T16:06:06.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:06.588+0000 7f483be00640 1 -- 172.21.15.39:0/3643219293 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4834004450 con 0x7f4844102380 2024-03-18T16:06:06.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:06.588+0000 7f483be00640 1 -- 172.21.15.39:0/3643219293 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f483404b060 con 0x7f4844102380 2024-03-18T16:06:06.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:06.589+0000 7f484a800640 1 -- 172.21.15.39:0/3643219293 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48440fd800 con 0x7f4844102380 2024-03-18T16:06:06.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:06.591+0000 7f483be00640 1 -- 172.21.15.39:0/3643219293 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f4834044070 con 0x7f4844102380 2024-03-18T16:06:06.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:06.592+0000 7f483be00640 1 --2- 172.21.15.39:0/3643219293 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f48280a2800 0x7f48280a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:06.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:06.592+0000 7f483be00640 1 -- 172.21.15.39:0/3643219293 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f48340efe40 con 0x7f4844102380 2024-03-18T16:06:06.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:06.592+0000 7f4842a00640 1 --2- 172.21.15.39:0/3643219293 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f48280a2800 0x7f48280a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:06.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:06.593+0000 7f4842a00640 1 --2- 172.21.15.39:0/3643219293 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f48280a2800 0x7f48280a4cc0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f4844199030 tx=0x7f4830005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:06.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:06.596+0000 7f483be00640 1 -- 172.21.15.39:0/3643219293 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f48340bcc90 con 0x7f4844102380 2024-03-18T16:06:06.729 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:06 smithi039 ceph-mon[31721]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:06:06.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:06.859+0000 7f484a800640 1 -- 172.21.15.39:0/3643219293 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f48440623e0 con 0x7f4844102380 2024-03-18T16:06:06.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:06.860+0000 7f483be00640 1 -- 172.21.15.39:0/3643219293 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2221 (secure 0 0 0) 0x7f48340c0d30 con 0x7f4844102380 2024-03-18T16:06:06.863 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:06:06.863 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/931350738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":931350738},{"type":"v1","addr":"172.21.15.194:6833","nonce":931350738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24463,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6833/3874209621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":3874209621},{"type":"v1","addr":"172.21.15.196:6833","nonce":3874209621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-18T16:06:06.863 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 3 2024-03-18T16:06:06.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:06.863+0000 7f484a800640 1 -- 172.21.15.39:0/3643219293 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f48280a2800 msgr2=0x7f48280a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:06.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:06.863+0000 7f484a800640 1 --2- 172.21.15.39:0/3643219293 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f48280a2800 0x7f48280a4cc0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f4844199030 tx=0x7f4830005f90 comp rx=0 tx=0).stop 2024-03-18T16:06:06.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:06.863+0000 7f484a800640 1 -- 172.21.15.39:0/3643219293 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4844102380 msgr2=0x7f4844197b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:06.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:06.863+0000 7f484a800640 1 --2- 172.21.15.39:0/3643219293 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4844102380 0x7f4844197b20 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f483402c260 tx=0x7f483402c670 comp rx=0 tx=0).stop 2024-03-18T16:06:06.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:06.863+0000 7f484a800640 1 -- 172.21.15.39:0/3643219293 shutdown_connections 2024-03-18T16:06:06.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:06.863+0000 7f484a800640 1 --2- 172.21.15.39:0/3643219293 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f48441075e0 0x7f484419f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:06.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:06.863+0000 7f484a800640 1 --2- 172.21.15.39:0/3643219293 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f48280a2800 0x7f48280a4cc0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:06.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:06.863+0000 7f484a800640 1 --2- 172.21.15.39:0/3643219293 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4844104cb0 0x7f4844198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:06.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:06.863+0000 7f484a800640 1 --2- 172.21.15.39:0/3643219293 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4844102380 0x7f4844197b20 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:06.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:06.863+0000 7f484a800640 1 -- 172.21.15.39:0/3643219293 >> 172.21.15.39:0/3643219293 conn(0x7f48440fbd10 msgr2=0x7f4844102e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:06.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:06.863+0000 7f484a800640 1 -- 172.21.15.39:0/3643219293 shutdown_connections 2024-03-18T16:06:06.867 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:06.863+0000 7f484a800640 1 -- 172.21.15.39:0/3643219293 wait complete. 2024-03-18T16:06:06.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:06 smithi196 ceph-mon[36307]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:06:06.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:06 smithi194 ceph-mon[36200]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:06:07.407 INFO:journalctl@ceph.mds.j.smithi039.stdout:Mar 18 16:06:07 smithi039 podman[69903]: 2024-03-18 16:06:07.108280855 +0000 UTC m=+0.151335739 container create 035dc69f8af4d5f70d5c9118e036e407dff1166cc04312a2ed1143a20af63cc2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-j, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T16:06:07.407 INFO:journalctl@ceph.mds.j.smithi039.stdout:Mar 18 16:06:07 smithi039 podman[69903]: 2024-03-18 16:06:07.40482659 +0000 UTC m=+0.447881480 container init 035dc69f8af4d5f70d5c9118e036e407dff1166cc04312a2ed1143a20af63cc2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-j, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-03-18T16:06:07.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:07 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/3643219293' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:06:07.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:07 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:07.837 INFO:journalctl@ceph.mds.j.smithi039.stdout:Mar 18 16:06:07 smithi039 podman[69903]: 2024-03-18 16:06:07.416257749 +0000 UTC m=+0.459312617 container start 035dc69f8af4d5f70d5c9118e036e407dff1166cc04312a2ed1143a20af63cc2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-j, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-18T16:06:07.837 INFO:journalctl@ceph.mds.j.smithi039.stdout:Mar 18 16:06:07 smithi039 ceph-mds[69922]: -- 172.21.15.39:0/4143995765 <== mon.1 v2:172.21.15.196:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ba92111a40 con 0x55ba920e4c00 2024-03-18T16:06:07.837 INFO:journalctl@ceph.mds.j.smithi039.stdout:Mar 18 16:06:07 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-j[69918]: starting mds.j at 2024-03-18T16:06:07.837 INFO:journalctl@ceph.mds.j.smithi039.stdout:Mar 18 16:06:07 smithi039 bash[69903]: 035dc69f8af4d5f70d5c9118e036e407dff1166cc04312a2ed1143a20af63cc2 2024-03-18T16:06:07.837 INFO:journalctl@ceph.mds.j.smithi039.stdout:Mar 18 16:06:07 smithi039 systemd[1]: Started Ceph mds.j for 1892d8fc-e540-11ee-95c9-87774f69a715. 2024-03-18T16:06:07.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:07 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/3643219293' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:06:07.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:07 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:07.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:07 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/3643219293' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:06:07.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:07 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:08.887 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:06:08.906 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:08 smithi039 ceph-mon[31721]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:06:08.906 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:08 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:08.907 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:08 smithi039 ceph-mon[31721]: mds.? [v2:172.21.15.39:6834/2520226608,v1:172.21.15.39:6835/2520226608] up:boot 2024-03-18T16:06:08.907 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:08 smithi039 ceph-mon[31721]: fsmap 3 up:standby 2024-03-18T16:06:08.907 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:08 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-18T16:06:08.907 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:08 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:08.907 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:08 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T16:06:08.907 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:08 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T16:06:08.907 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:08 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:06:08.907 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:08 smithi039 ceph-mon[31721]: Deploying daemon mds.d on smithi039 2024-03-18T16:06:08.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:08 smithi196 ceph-mon[36307]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:06:08.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:08 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:08.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:08 smithi196 ceph-mon[36307]: mds.? [v2:172.21.15.39:6834/2520226608,v1:172.21.15.39:6835/2520226608] up:boot 2024-03-18T16:06:08.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:08 smithi196 ceph-mon[36307]: fsmap 3 up:standby 2024-03-18T16:06:08.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:08 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-18T16:06:08.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:08 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:08.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:08 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T16:06:08.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:08 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T16:06:08.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:08 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:06:08.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:08 smithi196 ceph-mon[36307]: Deploying daemon mds.d on smithi039 2024-03-18T16:06:08.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:08 smithi194 ceph-mon[36200]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:06:08.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:08 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:08.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:08 smithi194 ceph-mon[36200]: mds.? [v2:172.21.15.39:6834/2520226608,v1:172.21.15.39:6835/2520226608] up:boot 2024-03-18T16:06:08.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:08 smithi194 ceph-mon[36200]: fsmap 3 up:standby 2024-03-18T16:06:08.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:08 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-18T16:06:08.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:08 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:08.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:08 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T16:06:08.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:08 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T16:06:08.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:08 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:06:08.991 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:08 smithi194 ceph-mon[36200]: Deploying daemon mds.d on smithi039 2024-03-18T16:06:09.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:09.108+0000 7fce4aa00640 1 -- 172.21.15.39:0/1087841821 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fce441018c0 msgr2=0x7fce44101d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:09.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:09.108+0000 7fce4aa00640 1 --2- 172.21.15.39:0/1087841821 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fce441018c0 0x7fce44101d40 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fce30009900 tx=0x7fce3002bc30 comp rx=0 tx=0).stop 2024-03-18T16:06:09.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:09.108+0000 7fce4aa00640 1 -- 172.21.15.39:0/1087841821 shutdown_connections 2024-03-18T16:06:09.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:09.108+0000 7fce4aa00640 1 --2- 172.21.15.39:0/1087841821 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fce441018c0 0x7fce44101d40 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:09.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:09.108+0000 7fce4aa00640 1 --2- 172.21.15.39:0/1087841821 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fce44102280 0x7fce44108b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:09.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:09.108+0000 7fce4aa00640 1 --2- 172.21.15.39:0/1087841821 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fce441006d0 0x7fce44100af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:09.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:09.108+0000 7fce4aa00640 1 -- 172.21.15.39:0/1087841821 >> 172.21.15.39:0/1087841821 conn(0x7fce440fbc40 msgr2=0x7fce440fe080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:09.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:09.108+0000 7fce4aa00640 1 -- 172.21.15.39:0/1087841821 shutdown_connections 2024-03-18T16:06:09.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:09.108+0000 7fce4aa00640 1 -- 172.21.15.39:0/1087841821 wait complete. 2024-03-18T16:06:09.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:09.109+0000 7fce4aa00640 1 Processor -- start 2024-03-18T16:06:09.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:09.109+0000 7fce4aa00640 1 -- start start 2024-03-18T16:06:09.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:09.110+0000 7fce4aa00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fce441006d0 0x7fce4410b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:09.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:09.110+0000 7fce4aa00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fce441018c0 0x7fce44109690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:09.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:09.110+0000 7fce4aa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fce44102280 0x7fce44109bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:09.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:09.110+0000 7fce4aa00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fce4410d470 con 0x7fce44102280 2024-03-18T16:06:09.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:09.110+0000 7fce4aa00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fce4410d2f0 con 0x7fce441018c0 2024-03-18T16:06:09.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:09.110+0000 7fce4aa00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fce4410d5f0 con 0x7fce441006d0 2024-03-18T16:06:09.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:09.110+0000 7fce48a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fce44102280 0x7fce44109bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:09.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:09.110+0000 7fce43e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fce441006d0 0x7fce4410b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:09.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:09.110+0000 7fce48a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fce44102280 0x7fce44109bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:35346/0 (socket says 172.21.15.39:35346) 2024-03-18T16:06:09.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:09.110+0000 7fce48a00640 1 -- 172.21.15.39:0/2405836094 learned_addr learned my addr 172.21.15.39:0/2405836094 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:06:09.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:09.110+0000 7fce43400640 1 --2- 172.21.15.39:0/2405836094 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fce441018c0 0x7fce44109690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:09.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:09.111+0000 7fce43e00640 1 -- 172.21.15.39:0/2405836094 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fce441018c0 msgr2=0x7fce44109690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:09.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:09.111+0000 7fce43e00640 1 --2- 172.21.15.39:0/2405836094 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fce441018c0 0x7fce44109690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:09.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:09.111+0000 7fce43e00640 1 -- 172.21.15.39:0/2405836094 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fce44102280 msgr2=0x7fce44109bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:09.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:09.111+0000 7fce43e00640 1 --2- 172.21.15.39:0/2405836094 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fce44102280 0x7fce44109bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:09.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:09.111+0000 7fce43e00640 1 -- 172.21.15.39:0/2405836094 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce4410a460 con 0x7fce441006d0 2024-03-18T16:06:09.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:09.111+0000 7fce43e00640 1 --2- 172.21.15.39:0/2405836094 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fce441006d0 0x7fce4410b050 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fce3400e9c0 tx=0x7fce3400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:09.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:09.111+0000 7fce40c00640 1 -- 172.21.15.39:0/2405836094 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce3400ccc0 con 0x7fce441006d0 2024-03-18T16:06:09.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:09.111+0000 7fce40c00640 1 -- 172.21.15.39:0/2405836094 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fce340035f0 con 0x7fce441006d0 2024-03-18T16:06:09.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:09.111+0000 7fce4aa00640 1 -- 172.21.15.39:0/2405836094 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce441a1c10 con 0x7fce441006d0 2024-03-18T16:06:09.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:09.111+0000 7fce40c00640 1 -- 172.21.15.39:0/2405836094 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce3400ce60 con 0x7fce441006d0 2024-03-18T16:06:09.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:09.111+0000 7fce4aa00640 1 -- 172.21.15.39:0/2405836094 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce441a20e0 con 0x7fce441006d0 2024-03-18T16:06:09.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:09.112+0000 7fce4aa00640 1 -- 172.21.15.39:0/2405836094 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce44068640 con 0x7fce441006d0 2024-03-18T16:06:09.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:09.114+0000 7fce40c00640 1 -- 172.21.15.39:0/2405836094 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fce340037a0 con 0x7fce441006d0 2024-03-18T16:06:09.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:09.116+0000 7fce40c00640 1 --2- 172.21.15.39:0/2405836094 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fce280a2800 0x7fce280a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:09.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:09.116+0000 7fce40c00640 1 -- 172.21.15.39:0/2405836094 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fce340cebb0 con 0x7fce441006d0 2024-03-18T16:06:09.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:09.116+0000 7fce43400640 1 --2- 172.21.15.39:0/2405836094 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fce280a2800 0x7fce280a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:09.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:09.117+0000 7fce43400640 1 --2- 172.21.15.39:0/2405836094 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fce280a2800 0x7fce280a4cc0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fce3002c7b0 tx=0x7fce30005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:09.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:09.118+0000 7fce40c00640 1 -- 172.21.15.39:0/2405836094 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fce3409c050 con 0x7fce441006d0 2024-03-18T16:06:09.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:09.367+0000 7fce4aa00640 1 -- 172.21.15.39:0/2405836094 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fce441079f0 con 0x7fce441006d0 2024-03-18T16:06:09.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:09.368+0000 7fce40c00640 1 -- 172.21.15.39:0/2405836094 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3046 (secure 0 0 0) 0x7fce3409b950 con 0x7fce441006d0 2024-03-18T16:06:09.370 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:06:09.370 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2520226608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2520226608},{"type":"v1","addr":"172.21.15.39:6835","nonce":2520226608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/931350738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":931350738},{"type":"v1","addr":"172.21.15.194:6833","nonce":931350738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24463,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6833/3874209621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":3874209621},{"type":"v1","addr":"172.21.15.196:6833","nonce":3874209621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-18T16:06:09.371 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 4 2024-03-18T16:06:09.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:09.370+0000 7fce4aa00640 1 -- 172.21.15.39:0/2405836094 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fce280a2800 msgr2=0x7fce280a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:09.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:09.370+0000 7fce4aa00640 1 --2- 172.21.15.39:0/2405836094 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fce280a2800 0x7fce280a4cc0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fce3002c7b0 tx=0x7fce30005fd0 comp rx=0 tx=0).stop 2024-03-18T16:06:09.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:09.370+0000 7fce4aa00640 1 -- 172.21.15.39:0/2405836094 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fce441006d0 msgr2=0x7fce4410b050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:09.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:09.370+0000 7fce4aa00640 1 --2- 172.21.15.39:0/2405836094 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fce441006d0 0x7fce4410b050 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fce3400e9c0 tx=0x7fce3400ee80 comp rx=0 tx=0).stop 2024-03-18T16:06:09.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:09.371+0000 7fce4aa00640 1 -- 172.21.15.39:0/2405836094 shutdown_connections 2024-03-18T16:06:09.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:09.371+0000 7fce4aa00640 1 --2- 172.21.15.39:0/2405836094 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fce441018c0 0x7fce44109690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:09.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:09.371+0000 7fce4aa00640 1 --2- 172.21.15.39:0/2405836094 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fce280a2800 0x7fce280a4cc0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:09.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:09.371+0000 7fce4aa00640 1 --2- 172.21.15.39:0/2405836094 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fce44102280 0x7fce44109bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:09.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:09.371+0000 7fce4aa00640 1 --2- 172.21.15.39:0/2405836094 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fce441006d0 0x7fce4410b050 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:09.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:09.371+0000 7fce4aa00640 1 -- 172.21.15.39:0/2405836094 >> 172.21.15.39:0/2405836094 conn(0x7fce440fbc40 msgr2=0x7fce44104bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:09.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:09.371+0000 7fce4aa00640 1 -- 172.21.15.39:0/2405836094 shutdown_connections 2024-03-18T16:06:09.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:09.371+0000 7fce4aa00640 1 -- 172.21.15.39:0/2405836094 wait complete. 2024-03-18T16:06:09.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:09 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/2405836094' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:06:09.983 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:09 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/2405836094' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:06:09.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:09 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/2405836094' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:06:10.767 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:10 smithi039 ceph-mon[31721]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:06:10.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:10 smithi196 ceph-mon[36307]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:06:10.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:10 smithi194 ceph-mon[36200]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:06:11.087 INFO:journalctl@ceph.mds.d.smithi039.stdout:Mar 18 16:06:10 smithi039 systemd[1]: Starting Ceph mds.d for 1892d8fc-e540-11ee-95c9-87774f69a715... 2024-03-18T16:06:11.392 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:06:11.415 INFO:journalctl@ceph.mds.d.smithi039.stdout:Mar 18 16:06:11 smithi039 podman[70251]: 2024-03-18 16:06:11.041090884 +0000 UTC m=+0.036439406 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T16:06:11.415 INFO:journalctl@ceph.mds.d.smithi039.stdout:Mar 18 16:06:11 smithi039 podman[70251]: 2024-03-18 16:06:11.1675512 +0000 UTC m=+0.162899742 container create 312ac9d705b0775801f310404d320dc8dd3396a702b0398dc7d4ac34500f8df6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-d, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-18T16:06:11.415 INFO:journalctl@ceph.mds.d.smithi039.stdout:Mar 18 16:06:11 smithi039 podman[70251]: 2024-03-18 16:06:11.413003848 +0000 UTC m=+0.408352433 container init 312ac9d705b0775801f310404d320dc8dd3396a702b0398dc7d4ac34500f8df6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-d, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-18T16:06:11.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:11.610+0000 7f968b200640 1 -- 172.21.15.39:0/1540549992 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9684102820 msgr2=0x7f9684102ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:11.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:11.610+0000 7f968b200640 1 --2- 172.21.15.39:0/1540549992 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9684102820 0x7f9684102ca0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f9674009900 tx=0x7f967402bc30 comp rx=0 tx=0).stop 2024-03-18T16:06:11.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:11.610+0000 7f968b200640 1 -- 172.21.15.39:0/1540549992 shutdown_connections 2024-03-18T16:06:11.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:11.610+0000 7f968b200640 1 --2- 172.21.15.39:0/1540549992 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f96841031e0 0x7f9684109a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:11.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:11.610+0000 7f968b200640 1 --2- 172.21.15.39:0/1540549992 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9684102820 0x7f9684102ca0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:11.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:11.610+0000 7f968b200640 1 --2- 172.21.15.39:0/1540549992 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9684101630 0x7f9684101a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:11.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:11.610+0000 7f968b200640 1 -- 172.21.15.39:0/1540549992 >> 172.21.15.39:0/1540549992 conn(0x7f96840fcbc0 msgr2=0x7f96840fefe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:11.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:11.610+0000 7f968b200640 1 -- 172.21.15.39:0/1540549992 shutdown_connections 2024-03-18T16:06:11.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:11.610+0000 7f968b200640 1 -- 172.21.15.39:0/1540549992 wait complete. 2024-03-18T16:06:11.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:11.612+0000 7f968b200640 1 Processor -- start 2024-03-18T16:06:11.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:11.612+0000 7f968b200640 1 -- start start 2024-03-18T16:06:11.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:11.612+0000 7f968b200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9684101630 0x7f9684075590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:11.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:11.612+0000 7f968b200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9684102820 0x7f9684073bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:11.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:11.612+0000 7f968b200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f96841031e0 0x7f9684074110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:11.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:11.612+0000 7f968b200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f968410d2e0 con 0x7f9684102820 2024-03-18T16:06:11.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:11.612+0000 7f968b200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f968410d160 con 0x7f96841031e0 2024-03-18T16:06:11.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:11.612+0000 7f968b200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f968410d460 con 0x7f9684101630 2024-03-18T16:06:11.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:11.612+0000 7f9689e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9684101630 0x7f9684075590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:11.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:11.612+0000 7f9689e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9684101630 0x7f9684075590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:36086/0 (socket says 172.21.15.39:36086) 2024-03-18T16:06:11.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:11.612+0000 7f9689e00640 1 -- 172.21.15.39:0/3235533388 learned_addr learned my addr 172.21.15.39:0/3235533388 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:06:11.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:11.612+0000 7f9689400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9684102820 0x7f9684073bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:11.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:11.612+0000 7f968a800640 1 --2- 172.21.15.39:0/3235533388 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f96841031e0 0x7f9684074110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:11.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:11.613+0000 7f968a800640 1 -- 172.21.15.39:0/3235533388 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9684101630 msgr2=0x7f9684075590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:11.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:11.613+0000 7f968a800640 1 --2- 172.21.15.39:0/3235533388 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9684101630 0x7f9684075590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:11.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:11.613+0000 7f968a800640 1 -- 172.21.15.39:0/3235533388 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9684102820 msgr2=0x7f9684073bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:11.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:11.613+0000 7f968a800640 1 --2- 172.21.15.39:0/3235533388 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9684102820 0x7f9684073bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:11.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:11.613+0000 7f968a800640 1 -- 172.21.15.39:0/3235533388 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9684074650 con 0x7f96841031e0 2024-03-18T16:06:11.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:11.613+0000 7f9689e00640 1 --2- 172.21.15.39:0/3235533388 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9684101630 0x7f9684075590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:06:11.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:11.613+0000 7f9689400640 1 --2- 172.21.15.39:0/3235533388 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9684102820 0x7f9684073bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:06:11.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:11.613+0000 7f968a800640 1 --2- 172.21.15.39:0/3235533388 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f96841031e0 0x7f9684074110 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f968000ec50 tx=0x7f968000c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:11.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:11.613+0000 7f967ea00640 1 -- 172.21.15.39:0/3235533388 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9680003b70 con 0x7f96841031e0 2024-03-18T16:06:11.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:11.614+0000 7f967ea00640 1 -- 172.21.15.39:0/3235533388 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9680003d10 con 0x7f96841031e0 2024-03-18T16:06:11.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:11.614+0000 7f967ea00640 1 -- 172.21.15.39:0/3235533388 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9680010890 con 0x7f96841031e0 2024-03-18T16:06:11.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:11.614+0000 7f968b200640 1 -- 172.21.15.39:0/3235533388 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9684074940 con 0x7f96841031e0 2024-03-18T16:06:11.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:11.614+0000 7f968b200640 1 -- 172.21.15.39:0/3235533388 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96841a62f0 con 0x7f96841031e0 2024-03-18T16:06:11.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:11.615+0000 7f968b200640 1 -- 172.21.15.39:0/3235533388 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96841088f0 con 0x7f96841031e0 2024-03-18T16:06:11.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:11.617+0000 7f967ea00640 1 -- 172.21.15.39:0/3235533388 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f9680004b40 con 0x7f96841031e0 2024-03-18T16:06:11.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:11.618+0000 7f967ea00640 1 --2- 172.21.15.39:0/3235533388 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f966c0a2800 0x7f966c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:11.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:11.619+0000 7f967ea00640 1 -- 172.21.15.39:0/3235533388 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f96800cebc0 con 0x7f96841031e0 2024-03-18T16:06:11.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:11.619+0000 7f9689e00640 1 --2- 172.21.15.39:0/3235533388 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f966c0a2800 0x7f966c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:11.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:11.619+0000 7f9689e00640 1 --2- 172.21.15.39:0/3235533388 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f966c0a2800 0x7f966c0a4cc0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f96780095f0 tx=0x7f9678005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:11.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:11.621+0000 7f967ea00640 1 -- 172.21.15.39:0/3235533388 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f968009c060 con 0x7f96841031e0 2024-03-18T16:06:11.837 INFO:journalctl@ceph.mds.d.smithi039.stdout:Mar 18 16:06:11 smithi039 podman[70251]: 2024-03-18 16:06:11.422583236 +0000 UTC m=+0.417931762 container start 312ac9d705b0775801f310404d320dc8dd3396a702b0398dc7d4ac34500f8df6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-d, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T16:06:11.837 INFO:journalctl@ceph.mds.d.smithi039.stdout:Mar 18 16:06:11 smithi039 ceph-mds[70277]: -- 172.21.15.39:0/2887363979 <== mon.0 v2:172.21.15.39:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5581ed3b5a40 con 0x5581ed388800 2024-03-18T16:06:11.837 INFO:journalctl@ceph.mds.d.smithi039.stdout:Mar 18 16:06:11 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-d[70266]: starting mds.d at 2024-03-18T16:06:11.837 INFO:journalctl@ceph.mds.d.smithi039.stdout:Mar 18 16:06:11 smithi039 bash[70251]: 312ac9d705b0775801f310404d320dc8dd3396a702b0398dc7d4ac34500f8df6 2024-03-18T16:06:11.837 INFO:journalctl@ceph.mds.d.smithi039.stdout:Mar 18 16:06:11 smithi039 systemd[1]: Started Ceph mds.d for 1892d8fc-e540-11ee-95c9-87774f69a715. 2024-03-18T16:06:11.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:11.872+0000 7f968b200640 1 -- 172.21.15.39:0/3235533388 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f968402d1b0 con 0x7f96841031e0 2024-03-18T16:06:11.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:11.873+0000 7f967ea00640 1 -- 172.21.15.39:0/3235533388 <== mon.2 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3868 (secure 0 0 0) 0x7f968009b960 con 0x7f96841031e0 2024-03-18T16:06:11.876 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:06:11.876 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2520226608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2520226608},{"type":"v1","addr":"172.21.15.39:6835","nonce":2520226608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/719341763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":719341763},{"type":"v1","addr":"172.21.15.39:6837","nonce":719341763}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/931350738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":931350738},{"type":"v1","addr":"172.21.15.194:6833","nonce":931350738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24463,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6833/3874209621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":3874209621},{"type":"v1","addr":"172.21.15.196:6833","nonce":3874209621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-18T16:06:11.876 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 5 2024-03-18T16:06:11.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:11.876+0000 7f968b200640 1 -- 172.21.15.39:0/3235533388 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f966c0a2800 msgr2=0x7f966c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:11.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:11.876+0000 7f968b200640 1 --2- 172.21.15.39:0/3235533388 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f966c0a2800 0x7f966c0a4cc0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f96780095f0 tx=0x7f9678005f90 comp rx=0 tx=0).stop 2024-03-18T16:06:11.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:11.876+0000 7f968b200640 1 -- 172.21.15.39:0/3235533388 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f96841031e0 msgr2=0x7f9684074110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:11.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:11.876+0000 7f968b200640 1 --2- 172.21.15.39:0/3235533388 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f96841031e0 0x7f9684074110 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f968000ec50 tx=0x7f968000c6a0 comp rx=0 tx=0).stop 2024-03-18T16:06:11.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:11.876+0000 7f968b200640 1 -- 172.21.15.39:0/3235533388 shutdown_connections 2024-03-18T16:06:11.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:11.876+0000 7f968b200640 1 --2- 172.21.15.39:0/3235533388 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f96841031e0 0x7f9684074110 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:11.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:11.876+0000 7f968b200640 1 --2- 172.21.15.39:0/3235533388 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f966c0a2800 0x7f966c0a4cc0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:11.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:11.876+0000 7f968b200640 1 --2- 172.21.15.39:0/3235533388 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9684102820 0x7f9684073bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:11.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:11.876+0000 7f968b200640 1 --2- 172.21.15.39:0/3235533388 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9684101630 0x7f9684075590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:11.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:11.876+0000 7f968b200640 1 -- 172.21.15.39:0/3235533388 >> 172.21.15.39:0/3235533388 conn(0x7f96840fcbc0 msgr2=0x7f9684105ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:11.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:11.876+0000 7f968b200640 1 -- 172.21.15.39:0/3235533388 shutdown_connections 2024-03-18T16:06:11.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:11.876+0000 7f968b200640 1 -- 172.21.15.39:0/3235533388 wait complete. 2024-03-18T16:06:12.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:12 smithi196 ceph-mon[36307]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:06:12.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:12 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:12.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:12 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:12.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:12 smithi196 ceph-mon[36307]: mds.? [v2:172.21.15.39:6836/719341763,v1:172.21.15.39:6837/719341763] up:boot 2024-03-18T16:06:12.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:12 smithi196 ceph-mon[36307]: fsmap 4 up:standby 2024-03-18T16:06:12.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:12 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-18T16:06:12.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:12 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:12.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:12 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T16:06:12.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:12 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T16:06:12.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:12 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:06:12.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:12 smithi196 ceph-mon[36307]: Deploying daemon mds.e on smithi194 2024-03-18T16:06:12.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:12 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/3235533388' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:06:12.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:12 smithi194 ceph-mon[36200]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:06:12.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:12 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:12.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:12 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:12.991 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:12 smithi194 ceph-mon[36200]: mds.? [v2:172.21.15.39:6836/719341763,v1:172.21.15.39:6837/719341763] up:boot 2024-03-18T16:06:12.991 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:12 smithi194 ceph-mon[36200]: fsmap 4 up:standby 2024-03-18T16:06:12.991 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:12 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-18T16:06:12.991 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:12 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:12.991 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:12 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T16:06:12.991 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:12 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T16:06:12.991 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:12 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:06:12.991 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:12 smithi194 ceph-mon[36200]: Deploying daemon mds.e on smithi194 2024-03-18T16:06:12.991 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:12 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/3235533388' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:06:13.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:12 smithi039 ceph-mon[31721]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:06:13.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:12 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:13.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:12 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:13.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:12 smithi039 ceph-mon[31721]: mds.? [v2:172.21.15.39:6836/719341763,v1:172.21.15.39:6837/719341763] up:boot 2024-03-18T16:06:13.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:12 smithi039 ceph-mon[31721]: fsmap 4 up:standby 2024-03-18T16:06:13.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:12 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-18T16:06:13.088 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:12 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:13.088 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:12 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T16:06:13.088 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:12 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T16:06:13.088 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:12 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:06:13.088 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:12 smithi039 ceph-mon[31721]: Deploying daemon mds.e on smithi194 2024-03-18T16:06:13.088 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:12 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/3235533388' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:06:13.899 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:06:14.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:14.130+0000 7f02b2800640 1 -- 172.21.15.39:0/3911123987 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f02ac104bf0 msgr2=0x7f02ac105070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:14.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:14.130+0000 7f02b2800640 1 --2- 172.21.15.39:0/3911123987 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f02ac104bf0 0x7f02ac105070 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0294009900 tx=0x7f029402bc30 comp rx=0 tx=0).stop 2024-03-18T16:06:14.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:14.131+0000 7f02b2800640 1 -- 172.21.15.39:0/3911123987 shutdown_connections 2024-03-18T16:06:14.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:14.131+0000 7f02b2800640 1 --2- 172.21.15.39:0/3911123987 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f02ac104bf0 0x7f02ac105070 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:14.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:14.131+0000 7f02b2800640 1 --2- 172.21.15.39:0/3911123987 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f02ac1055b0 0x7f02ac10be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:14.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:14.131+0000 7f02b2800640 1 --2- 172.21.15.39:0/3911123987 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f02ac103a00 0x7f02ac103e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:14.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:14.131+0000 7f02b2800640 1 -- 172.21.15.39:0/3911123987 >> 172.21.15.39:0/3911123987 conn(0x7f02ac0fef90 msgr2=0x7f02ac1013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:14.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:14.131+0000 7f02b2800640 1 -- 172.21.15.39:0/3911123987 shutdown_connections 2024-03-18T16:06:14.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:14.131+0000 7f02b2800640 1 -- 172.21.15.39:0/3911123987 wait complete. 2024-03-18T16:06:14.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:14.132+0000 7f02b2800640 1 Processor -- start 2024-03-18T16:06:14.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:14.132+0000 7f02b2800640 1 -- start start 2024-03-18T16:06:14.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:14.132+0000 7f02b2800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f02ac103a00 0x7f02ac199d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:14.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:14.132+0000 7f02b2800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f02ac104bf0 0x7f02ac19a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:14.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:14.132+0000 7f02b2800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f02ac1055b0 0x7f02ac1a1280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:14.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:14.132+0000 7f02b2800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f02ac10f6d0 con 0x7f02ac103a00 2024-03-18T16:06:14.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:14.132+0000 7f02b2800640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f02ac10f550 con 0x7f02ac1055b0 2024-03-18T16:06:14.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:14.132+0000 7f02b2800640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f02ac10f850 con 0x7f02ac104bf0 2024-03-18T16:06:14.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:14.132+0000 7f02ab400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f02ac103a00 0x7f02ac199d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:14.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:14.132+0000 7f02abe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f02ac1055b0 0x7f02ac1a1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:14.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:14.132+0000 7f02ab400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f02ac103a00 0x7f02ac199d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:35392/0 (socket says 172.21.15.39:35392) 2024-03-18T16:06:14.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:14.132+0000 7f02ab400640 1 -- 172.21.15.39:0/16703379 learned_addr learned my addr 172.21.15.39:0/16703379 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:06:14.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:14.132+0000 7f02a2a00640 1 --2- 172.21.15.39:0/16703379 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f02ac104bf0 0x7f02ac19a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:14.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:14.133+0000 7f02abe00640 1 -- 172.21.15.39:0/16703379 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f02ac104bf0 msgr2=0x7f02ac19a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:14.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:14.133+0000 7f02abe00640 1 --2- 172.21.15.39:0/16703379 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f02ac104bf0 0x7f02ac19a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:14.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:14.133+0000 7f02abe00640 1 -- 172.21.15.39:0/16703379 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f02ac103a00 msgr2=0x7f02ac199d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:14.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:14.133+0000 7f02abe00640 1 --2- 172.21.15.39:0/16703379 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f02ac103a00 0x7f02ac199d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:14.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:14.133+0000 7f02abe00640 1 -- 172.21.15.39:0/16703379 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02ac1a1920 con 0x7f02ac1055b0 2024-03-18T16:06:14.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:14.133+0000 7f02ab400640 1 --2- 172.21.15.39:0/16703379 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f02ac103a00 0x7f02ac199d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:06:14.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:14.133+0000 7f02abe00640 1 --2- 172.21.15.39:0/16703379 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f02ac1055b0 0x7f02ac1a1280 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f02a400ec50 tx=0x7f02a400c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:14.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:14.133+0000 7f02a8c00640 1 -- 172.21.15.39:0/16703379 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02a4003b70 con 0x7f02ac1055b0 2024-03-18T16:06:14.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:14.133+0000 7f02a8c00640 1 -- 172.21.15.39:0/16703379 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02a4003d10 con 0x7f02ac1055b0 2024-03-18T16:06:14.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:14.133+0000 7f02a8c00640 1 -- 172.21.15.39:0/16703379 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02a4010890 con 0x7f02ac1055b0 2024-03-18T16:06:14.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:14.133+0000 7f02b2800640 1 -- 172.21.15.39:0/16703379 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02ac1a1c10 con 0x7f02ac1055b0 2024-03-18T16:06:14.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:14.133+0000 7f02b2800640 1 -- 172.21.15.39:0/16703379 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02ac1a2110 con 0x7f02ac1055b0 2024-03-18T16:06:14.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:14.135+0000 7f02b2800640 1 -- 172.21.15.39:0/16703379 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02ac10ad20 con 0x7f02ac1055b0 2024-03-18T16:06:14.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:14.136+0000 7f02a8c00640 1 -- 172.21.15.39:0/16703379 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f02a4003fc0 con 0x7f02ac1055b0 2024-03-18T16:06:14.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:14.138+0000 7f02a8c00640 1 --2- 172.21.15.39:0/16703379 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f02900a2800 0x7f02900a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:14.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:14.138+0000 7f02a8c00640 1 -- 172.21.15.39:0/16703379 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f02a40cec70 con 0x7f02ac1055b0 2024-03-18T16:06:14.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:14.138+0000 7f02ab400640 1 --2- 172.21.15.39:0/16703379 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f02900a2800 0x7f02900a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:14.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:14.138+0000 7f02ab400640 1 --2- 172.21.15.39:0/16703379 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f02900a2800 0x7f02900a4cc0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f02ac19b1e0 tx=0x7f029c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:14.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:14.142+0000 7f02a8c00640 1 -- 172.21.15.39:0/16703379 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f02a409c110 con 0x7f02ac1055b0 2024-03-18T16:06:14.230 INFO:journalctl@ceph.mds.e.smithi194.stdout:Mar 18 16:06:14 smithi194 systemd[1]: Starting Ceph mds.e for 1892d8fc-e540-11ee-95c9-87774f69a715... 2024-03-18T16:06:14.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:14.388+0000 7f02b2800640 1 -- 172.21.15.39:0/16703379 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f02ac02d1b0 con 0x7f02ac1055b0 2024-03-18T16:06:14.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:14.389+0000 7f02a8c00640 1 -- 172.21.15.39:0/16703379 <== mon.2 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3868 (secure 0 0 0) 0x7f02a409ba10 con 0x7f02ac1055b0 2024-03-18T16:06:14.392 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:06:14.392 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2520226608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2520226608},{"type":"v1","addr":"172.21.15.39:6835","nonce":2520226608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/719341763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":719341763},{"type":"v1","addr":"172.21.15.39:6837","nonce":719341763}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/931350738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":931350738},{"type":"v1","addr":"172.21.15.194:6833","nonce":931350738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24463,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6833/3874209621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":3874209621},{"type":"v1","addr":"172.21.15.196:6833","nonce":3874209621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-18T16:06:14.392 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 5 2024-03-18T16:06:14.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:14.392+0000 7f02b2800640 1 -- 172.21.15.39:0/16703379 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f02900a2800 msgr2=0x7f02900a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:14.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:14.392+0000 7f02b2800640 1 --2- 172.21.15.39:0/16703379 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f02900a2800 0x7f02900a4cc0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f02ac19b1e0 tx=0x7f029c005f90 comp rx=0 tx=0).stop 2024-03-18T16:06:14.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:14.392+0000 7f02b2800640 1 -- 172.21.15.39:0/16703379 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f02ac1055b0 msgr2=0x7f02ac1a1280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:14.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:14.392+0000 7f02b2800640 1 --2- 172.21.15.39:0/16703379 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f02ac1055b0 0x7f02ac1a1280 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f02a400ec50 tx=0x7f02a400c6a0 comp rx=0 tx=0).stop 2024-03-18T16:06:14.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:14.392+0000 7f02b2800640 1 -- 172.21.15.39:0/16703379 shutdown_connections 2024-03-18T16:06:14.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:14.392+0000 7f02b2800640 1 --2- 172.21.15.39:0/16703379 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f02ac1055b0 0x7f02ac1a1280 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:14.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:14.392+0000 7f02b2800640 1 --2- 172.21.15.39:0/16703379 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f02900a2800 0x7f02900a4cc0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:14.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:14.392+0000 7f02b2800640 1 --2- 172.21.15.39:0/16703379 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f02ac104bf0 0x7f02ac19a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:14.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:14.392+0000 7f02b2800640 1 --2- 172.21.15.39:0/16703379 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f02ac103a00 0x7f02ac199d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:14.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:14.392+0000 7f02b2800640 1 -- 172.21.15.39:0/16703379 >> 172.21.15.39:0/16703379 conn(0x7f02ac0fef90 msgr2=0x7f02ac107ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:14.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:14.392+0000 7f02b2800640 1 -- 172.21.15.39:0/16703379 shutdown_connections 2024-03-18T16:06:14.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:14.393+0000 7f02b2800640 1 -- 172.21.15.39:0/16703379 wait complete. 2024-03-18T16:06:14.490 INFO:journalctl@ceph.mds.e.smithi194.stdout:Mar 18 16:06:14 smithi194 podman[66660]: 2024-03-18 16:06:14.131757938 +0000 UTC m=+0.017860797 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T16:06:14.490 INFO:journalctl@ceph.mds.e.smithi194.stdout:Mar 18 16:06:14 smithi194 podman[66660]: 2024-03-18 16:06:14.262038046 +0000 UTC m=+0.148140895 container create 315ebcc53fe14edef9440845eb72064d8ea0151204ec81fc0ba8aaf4e00f67ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-e, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307) 2024-03-18T16:06:14.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:14 smithi196 ceph-mon[36307]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:06:14.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:14 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/16703379' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:06:14.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:14 smithi194 ceph-mon[36200]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:06:14.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:14 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/16703379' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:06:14.990 INFO:journalctl@ceph.mds.e.smithi194.stdout:Mar 18 16:06:14 smithi194 podman[66660]: 2024-03-18 16:06:14.506228709 +0000 UTC m=+0.392331581 container init 315ebcc53fe14edef9440845eb72064d8ea0151204ec81fc0ba8aaf4e00f67ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-e, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T16:06:14.991 INFO:journalctl@ceph.mds.e.smithi194.stdout:Mar 18 16:06:14 smithi194 podman[66660]: 2024-03-18 16:06:14.508913205 +0000 UTC m=+0.395016066 container start 315ebcc53fe14edef9440845eb72064d8ea0151204ec81fc0ba8aaf4e00f67ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-e, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2) 2024-03-18T16:06:14.991 INFO:journalctl@ceph.mds.e.smithi194.stdout:Mar 18 16:06:14 smithi194 ceph-mds[66679]: -- 172.21.15.194:0/2592100524 <== mon.2 v2:172.21.15.194:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55aaefe53a40 con 0x55aaefe26c00 2024-03-18T16:06:14.991 INFO:journalctl@ceph.mds.e.smithi194.stdout:Mar 18 16:06:14 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-e[66675]: starting mds.e at 2024-03-18T16:06:14.991 INFO:journalctl@ceph.mds.e.smithi194.stdout:Mar 18 16:06:14 smithi194 bash[66660]: 315ebcc53fe14edef9440845eb72064d8ea0151204ec81fc0ba8aaf4e00f67ec 2024-03-18T16:06:14.991 INFO:journalctl@ceph.mds.e.smithi194.stdout:Mar 18 16:06:14 smithi194 systemd[1]: Started Ceph mds.e for 1892d8fc-e540-11ee-95c9-87774f69a715. 2024-03-18T16:06:15.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:14 smithi039 ceph-mon[31721]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:06:15.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:14 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/16703379' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:06:15.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:15 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:15.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:15 smithi196 ceph-mon[36307]: mds.? [v2:172.21.15.194:6834/188385470,v1:172.21.15.194:6835/188385470] up:boot 2024-03-18T16:06:15.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:15 smithi196 ceph-mon[36307]: fsmap 5 up:standby 2024-03-18T16:06:15.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:15 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-18T16:06:15.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:15 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:15.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:15 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:15.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:15 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T16:06:15.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:15 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T16:06:15.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:15 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:06:15.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:15 smithi196 ceph-mon[36307]: Deploying daemon mds.f on smithi196 2024-03-18T16:06:15.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:15 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:15.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:15 smithi194 ceph-mon[36200]: mds.? [v2:172.21.15.194:6834/188385470,v1:172.21.15.194:6835/188385470] up:boot 2024-03-18T16:06:15.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:15 smithi194 ceph-mon[36200]: fsmap 5 up:standby 2024-03-18T16:06:15.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:15 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-18T16:06:15.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:15 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:15.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:15 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:15.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:15 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T16:06:15.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:15 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T16:06:15.991 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:15 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:06:15.991 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:15 smithi194 ceph-mon[36200]: Deploying daemon mds.f on smithi196 2024-03-18T16:06:16.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:15 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:16.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:15 smithi039 ceph-mon[31721]: mds.? [v2:172.21.15.194:6834/188385470,v1:172.21.15.194:6835/188385470] up:boot 2024-03-18T16:06:16.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:15 smithi039 ceph-mon[31721]: fsmap 5 up:standby 2024-03-18T16:06:16.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:15 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-18T16:06:16.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:15 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:16.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:15 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:16.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:15 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T16:06:16.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:15 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T16:06:16.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:15 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:06:16.088 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:15 smithi039 ceph-mon[31721]: Deploying daemon mds.f on smithi196 2024-03-18T16:06:16.413 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:06:16.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:16.645+0000 7f3b0aa00640 1 -- 172.21.15.39:0/2309910348 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3b041055b0 msgr2=0x7f3b0410be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:16.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:16.645+0000 7f3b0aa00640 1 --2- 172.21.15.39:0/2309910348 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3b041055b0 0x7f3b0410be40 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3afc009ca0 tx=0x7f3afc02c100 comp rx=0 tx=0).stop 2024-03-18T16:06:16.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:16.645+0000 7f3b0aa00640 1 -- 172.21.15.39:0/2309910348 shutdown_connections 2024-03-18T16:06:16.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:16.645+0000 7f3b0aa00640 1 --2- 172.21.15.39:0/2309910348 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3b041055b0 0x7f3b0410be40 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:16.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:16.645+0000 7f3b0aa00640 1 --2- 172.21.15.39:0/2309910348 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3b04104bf0 0x7f3b04105070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:16.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:16.645+0000 7f3b0aa00640 1 --2- 172.21.15.39:0/2309910348 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3b04103a00 0x7f3b04103e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:16.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:16.645+0000 7f3b0aa00640 1 -- 172.21.15.39:0/2309910348 >> 172.21.15.39:0/2309910348 conn(0x7f3b040fef90 msgr2=0x7f3b041013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:16.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:16.645+0000 7f3b0aa00640 1 -- 172.21.15.39:0/2309910348 shutdown_connections 2024-03-18T16:06:16.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:16.645+0000 7f3b0aa00640 1 -- 172.21.15.39:0/2309910348 wait complete. 2024-03-18T16:06:16.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:16.646+0000 7f3b0aa00640 1 Processor -- start 2024-03-18T16:06:16.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:16.647+0000 7f3b0aa00640 1 -- start start 2024-03-18T16:06:16.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:16.647+0000 7f3b0aa00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3b04103a00 0x7f3b04199c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:16.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:16.647+0000 7f3b0aa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3b04104bf0 0x7f3b0419a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:16.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:16.647+0000 7f3b0aa00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3b041055b0 0x7f3b041a11d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:16.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:16.647+0000 7f3b0aa00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f3b0410f660 con 0x7f3b04104bf0 2024-03-18T16:06:16.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:16.647+0000 7f3b0aa00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f3b0410f4e0 con 0x7f3b041055b0 2024-03-18T16:06:16.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:16.647+0000 7f3b0aa00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f3b0410f7e0 con 0x7f3b04103a00 2024-03-18T16:06:16.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:16.647+0000 7f3b03e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3b04103a00 0x7f3b04199c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:16.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:16.647+0000 7f3b03e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3b04103a00 0x7f3b04199c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:36120/0 (socket says 172.21.15.39:36120) 2024-03-18T16:06:16.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:16.647+0000 7f3b03e00640 1 -- 172.21.15.39:0/2547190720 learned_addr learned my addr 172.21.15.39:0/2547190720 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:06:16.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:16.647+0000 7f3b03400640 1 --2- 172.21.15.39:0/2547190720 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3b04104bf0 0x7f3b0419a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:16.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:16.647+0000 7f3b08a00640 1 --2- 172.21.15.39:0/2547190720 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3b041055b0 0x7f3b041a11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:16.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:16.647+0000 7f3b03e00640 1 -- 172.21.15.39:0/2547190720 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3b041055b0 msgr2=0x7f3b041a11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:16.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:16.648+0000 7f3b03e00640 1 --2- 172.21.15.39:0/2547190720 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3b041055b0 0x7f3b041a11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:16.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:16.648+0000 7f3b03e00640 1 -- 172.21.15.39:0/2547190720 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3b04104bf0 msgr2=0x7f3b0419a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:16.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:16.648+0000 7f3b03e00640 1 --2- 172.21.15.39:0/2547190720 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3b04104bf0 0x7f3b0419a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:16.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:16.648+0000 7f3b03e00640 1 -- 172.21.15.39:0/2547190720 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b041a1870 con 0x7f3b04103a00 2024-03-18T16:06:16.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:16.648+0000 7f3b03400640 1 --2- 172.21.15.39:0/2547190720 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3b04104bf0 0x7f3b0419a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:06:16.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:16.648+0000 7f3b03e00640 1 --2- 172.21.15.39:0/2547190720 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3b04103a00 0x7f3b04199c60 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f3af400e9c0 tx=0x7f3af400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:16.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:16.648+0000 7f3b00c00640 1 -- 172.21.15.39:0/2547190720 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3af400cd80 con 0x7f3b04103a00 2024-03-18T16:06:16.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:16.648+0000 7f3b00c00640 1 -- 172.21.15.39:0/2547190720 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3af40035f0 con 0x7f3b04103a00 2024-03-18T16:06:16.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:16.648+0000 7f3b00c00640 1 -- 172.21.15.39:0/2547190720 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3af4010690 con 0x7f3b04103a00 2024-03-18T16:06:16.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:16.648+0000 7f3b0aa00640 1 -- 172.21.15.39:0/2547190720 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b041a1b60 con 0x7f3b04103a00 2024-03-18T16:06:16.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:16.648+0000 7f3b0aa00640 1 -- 172.21.15.39:0/2547190720 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b041a2060 con 0x7f3b04103a00 2024-03-18T16:06:16.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:16.649+0000 7f3b0aa00640 1 -- 172.21.15.39:0/2547190720 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b0410acc0 con 0x7f3b04103a00 2024-03-18T16:06:16.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:16.652+0000 7f3b00c00640 1 -- 172.21.15.39:0/2547190720 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f3af40037a0 con 0x7f3b04103a00 2024-03-18T16:06:16.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:16.653+0000 7f3b00c00640 1 --2- 172.21.15.39:0/2547190720 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3ae80a2880 0x7f3ae80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:16.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:16.653+0000 7f3b00c00640 1 -- 172.21.15.39:0/2547190720 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f3af40ceaf0 con 0x7f3b04103a00 2024-03-18T16:06:16.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:16.653+0000 7f3b03400640 1 --2- 172.21.15.39:0/2547190720 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3ae80a2880 0x7f3ae80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:16.656 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:16.654+0000 7f3b03400640 1 --2- 172.21.15.39:0/2547190720 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3ae80a2880 0x7f3ae80a4d40 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f3af0009fc0 tx=0x7f3af0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:16.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:16.656+0000 7f3b00c00640 1 -- 172.21.15.39:0/2547190720 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3af409bf90 con 0x7f3b04103a00 2024-03-18T16:06:16.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:16 smithi039 ceph-mon[31721]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:06:16.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:16.905+0000 7f3b0aa00640 1 -- 172.21.15.39:0/2547190720 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3b0402d1b0 con 0x7f3b04103a00 2024-03-18T16:06:16.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:16.906+0000 7f3b00c00640 1 -- 172.21.15.39:0/2547190720 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4693 (secure 0 0 0) 0x7f3af409b890 con 0x7f3b04103a00 2024-03-18T16:06:16.909 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:06:16.909 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2520226608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2520226608},{"type":"v1","addr":"172.21.15.39:6835","nonce":2520226608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/719341763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":719341763},{"type":"v1","addr":"172.21.15.39:6837","nonce":719341763}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/931350738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":931350738},{"type":"v1","addr":"172.21.15.194:6833","nonce":931350738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24463,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6833/3874209621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":3874209621},{"type":"v1","addr":"172.21.15.196:6833","nonce":3874209621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24494,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6835/188385470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":188385470},{"type":"v1","addr":"172.21.15.194:6835","nonce":188385470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-03-18T16:06:16.909 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 6 2024-03-18T16:06:16.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:16.909+0000 7f3b0aa00640 1 -- 172.21.15.39:0/2547190720 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3ae80a2880 msgr2=0x7f3ae80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:16.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:16.909+0000 7f3b0aa00640 1 --2- 172.21.15.39:0/2547190720 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3ae80a2880 0x7f3ae80a4d40 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f3af0009fc0 tx=0x7f3af0005f90 comp rx=0 tx=0).stop 2024-03-18T16:06:16.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:16.909+0000 7f3b0aa00640 1 -- 172.21.15.39:0/2547190720 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3b04103a00 msgr2=0x7f3b04199c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:16.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:16.909+0000 7f3b0aa00640 1 --2- 172.21.15.39:0/2547190720 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3b04103a00 0x7f3b04199c60 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f3af400e9c0 tx=0x7f3af400ee80 comp rx=0 tx=0).stop 2024-03-18T16:06:16.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:16.909+0000 7f3b0aa00640 1 -- 172.21.15.39:0/2547190720 shutdown_connections 2024-03-18T16:06:16.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:16.909+0000 7f3b0aa00640 1 --2- 172.21.15.39:0/2547190720 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3b041055b0 0x7f3b041a11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:16.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:16.909+0000 7f3b0aa00640 1 --2- 172.21.15.39:0/2547190720 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3ae80a2880 0x7f3ae80a4d40 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:16.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:16.909+0000 7f3b0aa00640 1 --2- 172.21.15.39:0/2547190720 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3b04104bf0 0x7f3b0419a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:16.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:16.909+0000 7f3b0aa00640 1 --2- 172.21.15.39:0/2547190720 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3b04103a00 0x7f3b04199c60 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:16.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:16.909+0000 7f3b0aa00640 1 -- 172.21.15.39:0/2547190720 >> 172.21.15.39:0/2547190720 conn(0x7f3b040fef90 msgr2=0x7f3b04107e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:16.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:16.909+0000 7f3b0aa00640 1 -- 172.21.15.39:0/2547190720 shutdown_connections 2024-03-18T16:06:16.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:16.909+0000 7f3b0aa00640 1 -- 172.21.15.39:0/2547190720 wait complete. 2024-03-18T16:06:16.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:16 smithi196 ceph-mon[36307]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:06:16.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:16 smithi194 ceph-mon[36200]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:06:17.728 INFO:journalctl@ceph.mds.f.smithi196.stdout:Mar 18 16:06:17 smithi196 systemd[1]: Started Ceph mds.f for 1892d8fc-e540-11ee-95c9-87774f69a715. 2024-03-18T16:06:17.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:17 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/2547190720' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:06:17.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:17 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:17.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:17 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:17.991 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:17 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:17.991 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:17 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T16:06:17.991 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:17 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T16:06:17.991 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:17 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:06:18.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:17 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/2547190720' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:06:18.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:17 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:18.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:17 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:18.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:17 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:18.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:17 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T16:06:18.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:17 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T16:06:18.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:17 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:06:18.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:17 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/2547190720' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:06:18.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:17 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:18.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:17 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:18.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:17 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:18.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:17 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T16:06:18.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:17 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T16:06:18.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:17 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:06:18.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:18 smithi039 ceph-mon[31721]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:06:18.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:18 smithi039 ceph-mon[31721]: Deploying daemon mds.c on smithi196 2024-03-18T16:06:18.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:18 smithi039 ceph-mon[31721]: mds.? [v2:172.21.15.196:6834/3949984309,v1:172.21.15.196:6835/3949984309] up:boot 2024-03-18T16:06:18.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:18 smithi039 ceph-mon[31721]: fsmap 6 up:standby 2024-03-18T16:06:18.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:18 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-18T16:06:18.933 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:06:18.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:18 smithi196 ceph-mon[36307]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:06:18.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:18 smithi196 ceph-mon[36307]: Deploying daemon mds.c on smithi196 2024-03-18T16:06:18.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:18 smithi196 ceph-mon[36307]: mds.? [v2:172.21.15.196:6834/3949984309,v1:172.21.15.196:6835/3949984309] up:boot 2024-03-18T16:06:18.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:18 smithi196 ceph-mon[36307]: fsmap 6 up:standby 2024-03-18T16:06:18.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:18 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-18T16:06:18.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:18 smithi194 ceph-mon[36200]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:06:18.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:18 smithi194 ceph-mon[36200]: Deploying daemon mds.c on smithi196 2024-03-18T16:06:18.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:18 smithi194 ceph-mon[36200]: mds.? [v2:172.21.15.196:6834/3949984309,v1:172.21.15.196:6835/3949984309] up:boot 2024-03-18T16:06:18.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:18 smithi194 ceph-mon[36200]: fsmap 6 up:standby 2024-03-18T16:06:18.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:18 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-18T16:06:19.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:19.166+0000 7f10dbe00640 1 -- 172.21.15.39:0/3158864992 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f10dc108e10 msgr2=0x7f10dc109290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:19.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:19.166+0000 7f10dbe00640 1 --2- 172.21.15.39:0/3158864992 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f10dc108e10 0x7f10dc109290 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f10c4009900 tx=0x7f10c402bc30 comp rx=0 tx=0).stop 2024-03-18T16:06:19.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:19.166+0000 7f10dbe00640 1 -- 172.21.15.39:0/3158864992 shutdown_connections 2024-03-18T16:06:19.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:19.166+0000 7f10dbe00640 1 --2- 172.21.15.39:0/3158864992 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f10dc108e10 0x7f10dc109290 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:19.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:19.166+0000 7f10dbe00640 1 --2- 172.21.15.39:0/3158864992 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f10dc1097d0 0x7f10dc110060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:19.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:19.166+0000 7f10dbe00640 1 --2- 172.21.15.39:0/3158864992 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f10dc107c20 0x7f10dc108040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:19.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:19.166+0000 7f10dbe00640 1 -- 172.21.15.39:0/3158864992 >> 172.21.15.39:0/3158864992 conn(0x7f10dc076bb0 msgr2=0x7f10dc078ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:19.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:19.166+0000 7f10dbe00640 1 -- 172.21.15.39:0/3158864992 shutdown_connections 2024-03-18T16:06:19.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:19.166+0000 7f10dbe00640 1 -- 172.21.15.39:0/3158864992 wait complete. 2024-03-18T16:06:19.172 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:19.167+0000 7f10dbe00640 1 Processor -- start 2024-03-18T16:06:19.172 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:19.167+0000 7f10dbe00640 1 -- start start 2024-03-18T16:06:19.172 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:19.168+0000 7f10dbe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f10dc107c20 0x7f10dc19dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:19.172 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:19.168+0000 7f10dbe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f10dc108e10 0x7f10dc19e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:19.172 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:19.168+0000 7f10dbe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f10dc1097d0 0x7f10dc1a5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:19.172 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:19.168+0000 7f10dbe00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f10dc1138f0 con 0x7f10dc108e10 2024-03-18T16:06:19.172 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:19.168+0000 7f10dbe00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f10dc113770 con 0x7f10dc107c20 2024-03-18T16:06:19.172 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:19.168+0000 7f10dbe00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f10dc113a70 con 0x7f10dc1097d0 2024-03-18T16:06:19.172 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:19.168+0000 7f10d2000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f10dc108e10 0x7f10dc19e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:19.172 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:19.168+0000 7f10d2000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f10dc108e10 0x7f10dc19e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:34378/0 (socket says 172.21.15.39:34378) 2024-03-18T16:06:19.172 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:19.168+0000 7f10d2000640 1 -- 172.21.15.39:0/1616297517 learned_addr learned my addr 172.21.15.39:0/1616297517 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:06:19.172 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:19.168+0000 7f10db400640 1 --2- 172.21.15.39:0/1616297517 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f10dc1097d0 0x7f10dc1a5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:19.172 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:19.168+0000 7f10daa00640 1 --2- 172.21.15.39:0/1616297517 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f10dc107c20 0x7f10dc19dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:19.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:19.169+0000 7f10d2000640 1 -- 172.21.15.39:0/1616297517 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f10dc1097d0 msgr2=0x7f10dc1a5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:19.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:19.169+0000 7f10d2000640 1 --2- 172.21.15.39:0/1616297517 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f10dc1097d0 0x7f10dc1a5490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:19.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:19.169+0000 7f10d2000640 1 -- 172.21.15.39:0/1616297517 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f10dc107c20 msgr2=0x7f10dc19dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:19.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:19.169+0000 7f10d2000640 1 --2- 172.21.15.39:0/1616297517 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f10dc107c20 0x7f10dc19dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:19.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:19.169+0000 7f10d2000640 1 -- 172.21.15.39:0/1616297517 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10dc1a5b00 con 0x7f10dc108e10 2024-03-18T16:06:19.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:19.169+0000 7f10db400640 1 --2- 172.21.15.39:0/1616297517 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f10dc1097d0 0x7f10dc1a5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:06:19.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:19.169+0000 7f10daa00640 1 --2- 172.21.15.39:0/1616297517 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f10dc107c20 0x7f10dc19dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:06:19.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:19.169+0000 7f10d2000640 1 --2- 172.21.15.39:0/1616297517 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f10dc108e10 0x7f10dc19e420 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f10c4000c00 tx=0x7f10c402c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:19.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:19.169+0000 7f10d3e00640 1 -- 172.21.15.39:0/1616297517 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10c4043020 con 0x7f10dc108e10 2024-03-18T16:06:19.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:19.169+0000 7f10d3e00640 1 -- 172.21.15.39:0/1616297517 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f10c40054e0 con 0x7f10dc108e10 2024-03-18T16:06:19.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:19.169+0000 7f10d3e00640 1 -- 172.21.15.39:0/1616297517 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10c4003c40 con 0x7f10dc108e10 2024-03-18T16:06:19.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:19.169+0000 7f10dbe00640 1 -- 172.21.15.39:0/1616297517 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10dc1a5d90 con 0x7f10dc108e10 2024-03-18T16:06:19.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:19.170+0000 7f10dbe00640 1 -- 172.21.15.39:0/1616297517 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10dc1a6230 con 0x7f10dc108e10 2024-03-18T16:06:19.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:19.171+0000 7f10dbe00640 1 -- 172.21.15.39:0/1616297517 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10dc10ef40 con 0x7f10dc108e10 2024-03-18T16:06:19.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:19.174+0000 7f10d3e00640 1 -- 172.21.15.39:0/1616297517 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f10c4046020 con 0x7f10dc108e10 2024-03-18T16:06:19.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:19.175+0000 7f10d3e00640 1 --2- 172.21.15.39:0/1616297517 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f10c00a2880 0x7f10c00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:19.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:19.175+0000 7f10d3e00640 1 -- 172.21.15.39:0/1616297517 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f10c40f0620 con 0x7f10dc108e10 2024-03-18T16:06:19.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:19.175+0000 7f10daa00640 1 --2- 172.21.15.39:0/1616297517 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f10c00a2880 0x7f10c00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:19.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:19.176+0000 7f10daa00640 1 --2- 172.21.15.39:0/1616297517 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f10c00a2880 0x7f10c00a4d40 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f10dc108c70 tx=0x7f10cc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:19.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:19.180+0000 7f10d3e00640 1 -- 172.21.15.39:0/1616297517 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f10c404d020 con 0x7f10dc108e10 2024-03-18T16:06:19.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:19.413+0000 7f10dbe00640 1 -- 172.21.15.39:0/1616297517 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f10dc02d1b0 con 0x7f10dc108e10 2024-03-18T16:06:19.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:19.414+0000 7f10d3e00640 1 -- 172.21.15.39:0/1616297517 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5521 (secure 0 0 0) 0x7f10c40bdac0 con 0x7f10dc108e10 2024-03-18T16:06:19.417 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:06:19.417 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2520226608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2520226608},{"type":"v1","addr":"172.21.15.39:6835","nonce":2520226608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/719341763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":719341763},{"type":"v1","addr":"172.21.15.39:6837","nonce":719341763}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/931350738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":931350738},{"type":"v1","addr":"172.21.15.194:6833","nonce":931350738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24463,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6833/3874209621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":3874209621},{"type":"v1","addr":"172.21.15.196:6833","nonce":3874209621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24494,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6835/188385470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":188385470},{"type":"v1","addr":"172.21.15.194:6835","nonce":188385470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24523,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6835/3949984309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":3949984309},{"type":"v1","addr":"172.21.15.196:6835","nonce":3949984309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-03-18T16:06:19.417 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 7 2024-03-18T16:06:19.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:19.417+0000 7f10dbe00640 1 -- 172.21.15.39:0/1616297517 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f10c00a2880 msgr2=0x7f10c00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:19.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:19.417+0000 7f10dbe00640 1 --2- 172.21.15.39:0/1616297517 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f10c00a2880 0x7f10c00a4d40 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f10dc108c70 tx=0x7f10cc005f90 comp rx=0 tx=0).stop 2024-03-18T16:06:19.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:19.417+0000 7f10dbe00640 1 -- 172.21.15.39:0/1616297517 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f10dc108e10 msgr2=0x7f10dc19e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:19.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:19.417+0000 7f10dbe00640 1 --2- 172.21.15.39:0/1616297517 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f10dc108e10 0x7f10dc19e420 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f10c4000c00 tx=0x7f10c402c650 comp rx=0 tx=0).stop 2024-03-18T16:06:19.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:19.417+0000 7f10dbe00640 1 -- 172.21.15.39:0/1616297517 shutdown_connections 2024-03-18T16:06:19.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:19.417+0000 7f10dbe00640 1 --2- 172.21.15.39:0/1616297517 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f10c00a2880 0x7f10c00a4d40 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:19.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:19.417+0000 7f10dbe00640 1 --2- 172.21.15.39:0/1616297517 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f10dc1097d0 0x7f10dc1a5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:19.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:19.417+0000 7f10dbe00640 1 --2- 172.21.15.39:0/1616297517 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f10dc108e10 0x7f10dc19e420 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:19.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:19.417+0000 7f10dbe00640 1 --2- 172.21.15.39:0/1616297517 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f10dc107c20 0x7f10dc19dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:19.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:19.417+0000 7f10dbe00640 1 -- 172.21.15.39:0/1616297517 >> 172.21.15.39:0/1616297517 conn(0x7f10dc076bb0 msgr2=0x7f10dc10c110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:19.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:19.417+0000 7f10dbe00640 1 -- 172.21.15.39:0/1616297517 shutdown_connections 2024-03-18T16:06:19.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:19.417+0000 7f10dbe00640 1 -- 172.21.15.39:0/1616297517 wait complete. 2024-03-18T16:06:19.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:19 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/1616297517' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:06:19.979 INFO:journalctl@ceph.mds.c.smithi196.stdout:Mar 18 16:06:19 smithi196 systemd[1]: Starting Ceph mds.c for 1892d8fc-e540-11ee-95c9-87774f69a715... 2024-03-18T16:06:20.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:19 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/1616297517' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:06:20.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:19 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/1616297517' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:06:20.400 INFO:journalctl@ceph.mds.c.smithi196.stdout:Mar 18 16:06:20 smithi196 podman[66670]: 2024-03-18 16:06:20.020748819 +0000 UTC m=+0.017646507 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T16:06:20.400 INFO:journalctl@ceph.mds.c.smithi196.stdout:Mar 18 16:06:20 smithi196 podman[66670]: 2024-03-18 16:06:20.146071558 +0000 UTC m=+0.142969232 container create 4a10438865ac82f5d005ddf2d90f1604f3e5471805c97ae5dac394aef7a40de5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-c, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T16:06:20.400 INFO:journalctl@ceph.mds.c.smithi196.stdout:Mar 18 16:06:20 smithi196 podman[66670]: 2024-03-18 16:06:20.398644694 +0000 UTC m=+0.395542369 container init 4a10438865ac82f5d005ddf2d90f1604f3e5471805c97ae5dac394aef7a40de5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-c, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T16:06:20.728 INFO:journalctl@ceph.mds.c.smithi196.stdout:Mar 18 16:06:20 smithi196 podman[66670]: 2024-03-18 16:06:20.401823576 +0000 UTC m=+0.398721252 container start 4a10438865ac82f5d005ddf2d90f1604f3e5471805c97ae5dac394aef7a40de5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-c, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-18T16:06:20.728 INFO:journalctl@ceph.mds.c.smithi196.stdout:Mar 18 16:06:20 smithi196 ceph-mds[66689]: -- 172.21.15.196:0/2392533914 <== mon.1 v2:172.21.15.196:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5639dbe4da40 con 0x5639dbe20800 2024-03-18T16:06:20.728 INFO:journalctl@ceph.mds.c.smithi196.stdout:Mar 18 16:06:20 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-c[66685]: starting mds.c at 2024-03-18T16:06:20.728 INFO:journalctl@ceph.mds.c.smithi196.stdout:Mar 18 16:06:20 smithi196 bash[66670]: 4a10438865ac82f5d005ddf2d90f1604f3e5471805c97ae5dac394aef7a40de5 2024-03-18T16:06:20.728 INFO:journalctl@ceph.mds.c.smithi196.stdout:Mar 18 16:06:20 smithi196 systemd[1]: Started Ceph mds.c for 1892d8fc-e540-11ee-95c9-87774f69a715. 2024-03-18T16:06:21.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:20 smithi039 ceph-mon[31721]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:06:21.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:20 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:21.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:20 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:21.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:20 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:21.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:20 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T16:06:21.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:20 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T16:06:21.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:20 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:06:21.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:20 smithi196 ceph-mon[36307]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:06:21.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:20 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:21.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:20 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:21.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:20 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:21.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:20 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T16:06:21.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:20 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T16:06:21.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:20 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:06:21.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:20 smithi194 ceph-mon[36200]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:06:21.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:20 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:21.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:20 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:21.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:20 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:21.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:20 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T16:06:21.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:20 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T16:06:21.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:20 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:06:21.436 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:06:21.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:21.666+0000 7f9107000640 1 -- 172.21.15.39:0/164158685 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f91001097d0 msgr2=0x7f9100110060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:21.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:21.666+0000 7f9107000640 1 --2- 172.21.15.39:0/164158685 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f91001097d0 0x7f9100110060 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f90f8009900 tx=0x7f90f802bc50 comp rx=0 tx=0).stop 2024-03-18T16:06:21.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:21.666+0000 7f9107000640 1 -- 172.21.15.39:0/164158685 shutdown_connections 2024-03-18T16:06:21.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:21.666+0000 7f9107000640 1 --2- 172.21.15.39:0/164158685 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f91001097d0 0x7f9100110060 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:21.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:21.666+0000 7f9107000640 1 --2- 172.21.15.39:0/164158685 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9100108e10 0x7f9100109290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:21.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:21.666+0000 7f9107000640 1 --2- 172.21.15.39:0/164158685 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9100107c20 0x7f9100108040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:21.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:21.666+0000 7f9107000640 1 -- 172.21.15.39:0/164158685 >> 172.21.15.39:0/164158685 conn(0x7f9100076bb0 msgr2=0x7f9100078ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:21.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:21.666+0000 7f9107000640 1 -- 172.21.15.39:0/164158685 shutdown_connections 2024-03-18T16:06:21.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:21.666+0000 7f9107000640 1 -- 172.21.15.39:0/164158685 wait complete. 2024-03-18T16:06:21.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:21.667+0000 7f9107000640 1 Processor -- start 2024-03-18T16:06:21.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:21.667+0000 7f9107000640 1 -- start start 2024-03-18T16:06:21.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:21.667+0000 7f9107000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9100107c20 0x7f910019deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:21.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:21.667+0000 7f9107000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9100108e10 0x7f910019e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:21.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:21.667+0000 7f9107000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f91001097d0 0x7f91001a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:21.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:21.667+0000 7f9107000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f91001138d0 con 0x7f9100108e10 2024-03-18T16:06:21.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:21.667+0000 7f9107000640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f9100113750 con 0x7f9100107c20 2024-03-18T16:06:21.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:21.667+0000 7f9107000640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f9100113a50 con 0x7f91001097d0 2024-03-18T16:06:21.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:21.668+0000 7f9105c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9100107c20 0x7f910019deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:21.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:21.668+0000 7f9105c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9100107c20 0x7f910019deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:50572/0 (socket says 172.21.15.39:50572) 2024-03-18T16:06:21.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:21.668+0000 7f9105c00640 1 -- 172.21.15.39:0/3816061548 learned_addr learned my addr 172.21.15.39:0/3816061548 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:06:21.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:21.668+0000 7f9106600640 1 --2- 172.21.15.39:0/3816061548 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f91001097d0 0x7f91001a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:21.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:21.668+0000 7f9105200640 1 --2- 172.21.15.39:0/3816061548 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9100108e10 0x7f910019e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:21.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:21.668+0000 7f9105c00640 1 -- 172.21.15.39:0/3816061548 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f91001097d0 msgr2=0x7f91001a5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:21.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:21.668+0000 7f9105c00640 1 --2- 172.21.15.39:0/3816061548 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f91001097d0 0x7f91001a5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:21.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:21.668+0000 7f9105c00640 1 -- 172.21.15.39:0/3816061548 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9100108e10 msgr2=0x7f910019e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:21.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:21.668+0000 7f9105c00640 1 --2- 172.21.15.39:0/3816061548 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9100108e10 0x7f910019e3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:21.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:21.668+0000 7f9105c00640 1 -- 172.21.15.39:0/3816061548 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91001a5ad0 con 0x7f9100107c20 2024-03-18T16:06:21.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:21.668+0000 7f9105c00640 1 --2- 172.21.15.39:0/3816061548 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9100107c20 0x7f910019deb0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f90f400ebc0 tx=0x7f90f400c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:21.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:21.669+0000 7f90fe000640 1 -- 172.21.15.39:0/3816061548 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90f4003b70 con 0x7f9100107c20 2024-03-18T16:06:21.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:21.669+0000 7f9107000640 1 -- 172.21.15.39:0/3816061548 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91001a5dc0 con 0x7f9100107c20 2024-03-18T16:06:21.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:21.669+0000 7f90fe000640 1 -- 172.21.15.39:0/3816061548 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f90f4003d10 con 0x7f9100107c20 2024-03-18T16:06:21.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:21.669+0000 7f90fe000640 1 -- 172.21.15.39:0/3816061548 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90f4019980 con 0x7f9100107c20 2024-03-18T16:06:21.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:21.669+0000 7f9107000640 1 -- 172.21.15.39:0/3816061548 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91001a6230 con 0x7f9100107c20 2024-03-18T16:06:21.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:21.670+0000 7f9107000640 1 -- 172.21.15.39:0/3816061548 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f910010eee0 con 0x7f9100107c20 2024-03-18T16:06:21.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:21.672+0000 7f90fe000640 1 -- 172.21.15.39:0/3816061548 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f90f4019b20 con 0x7f9100107c20 2024-03-18T16:06:21.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:21.673+0000 7f90fe000640 1 --2- 172.21.15.39:0/3816061548 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f90e80a2800 0x7f90e80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:21.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:21.673+0000 7f90fe000640 1 -- 172.21.15.39:0/3816061548 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f90f40cee10 con 0x7f9100107c20 2024-03-18T16:06:21.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:21.674+0000 7f9105200640 1 --2- 172.21.15.39:0/3816061548 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f90e80a2800 0x7f90e80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:21.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:21.674+0000 7f9105200640 1 --2- 172.21.15.39:0/3816061548 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f90e80a2800 0x7f90e80a4cc0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f910019f3c0 tx=0x7f90ec005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:21.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:21.677+0000 7f90fe000640 1 -- 172.21.15.39:0/3816061548 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f90f40d2030 con 0x7f9100107c20 2024-03-18T16:06:21.776 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:21 smithi039 ceph-mon[31721]: Deploying daemon mds.h on smithi194 2024-03-18T16:06:21.776 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:21 smithi039 ceph-mon[31721]: mds.? [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] up:boot 2024-03-18T16:06:21.776 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:21 smithi039 ceph-mon[31721]: fsmap 7 up:standby 2024-03-18T16:06:21.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:21.918+0000 7f9107000640 1 -- 172.21.15.39:0/3816061548 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f910002d1b0 con 0x7f9100107c20 2024-03-18T16:06:21.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:21.920+0000 7f90fe000640 1 -- 172.21.15.39:0/3816061548 <== mon.2 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6346 (secure 0 0 0) 0x7f910002d1b0 con 0x7f9100107c20 2024-03-18T16:06:21.922 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:06:21.922 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2520226608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2520226608},{"type":"v1","addr":"172.21.15.39:6835","nonce":2520226608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/719341763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":719341763},{"type":"v1","addr":"172.21.15.39:6837","nonce":719341763}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/931350738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":931350738},{"type":"v1","addr":"172.21.15.194:6833","nonce":931350738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24463,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6833/3874209621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":3874209621},{"type":"v1","addr":"172.21.15.196:6833","nonce":3874209621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24494,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6835/188385470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":188385470},{"type":"v1","addr":"172.21.15.194:6835","nonce":188385470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24523,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6835/3949984309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":3949984309},{"type":"v1","addr":"172.21.15.196:6835","nonce":3949984309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24535,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6837/895247251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":895247251},{"type":"v1","addr":"172.21.15.196:6837","nonce":895247251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-03-18T16:06:21.923 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 8 2024-03-18T16:06:21.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:21.923+0000 7f9107000640 1 -- 172.21.15.39:0/3816061548 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f90e80a2800 msgr2=0x7f90e80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:21.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:21.923+0000 7f9107000640 1 --2- 172.21.15.39:0/3816061548 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f90e80a2800 0x7f90e80a4cc0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f910019f3c0 tx=0x7f90ec005f90 comp rx=0 tx=0).stop 2024-03-18T16:06:21.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:21.923+0000 7f9107000640 1 -- 172.21.15.39:0/3816061548 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9100107c20 msgr2=0x7f910019deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:21.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:21.923+0000 7f9107000640 1 --2- 172.21.15.39:0/3816061548 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9100107c20 0x7f910019deb0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f90f400ebc0 tx=0x7f90f400c6a0 comp rx=0 tx=0).stop 2024-03-18T16:06:21.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:21.923+0000 7f9107000640 1 -- 172.21.15.39:0/3816061548 shutdown_connections 2024-03-18T16:06:21.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:21.923+0000 7f9107000640 1 --2- 172.21.15.39:0/3816061548 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f90e80a2800 0x7f90e80a4cc0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:21.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:21.923+0000 7f9107000640 1 --2- 172.21.15.39:0/3816061548 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f91001097d0 0x7f91001a5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:21.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:21.923+0000 7f9107000640 1 --2- 172.21.15.39:0/3816061548 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9100108e10 0x7f910019e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:21.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:21.923+0000 7f9107000640 1 --2- 172.21.15.39:0/3816061548 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9100107c20 0x7f910019deb0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:21.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:21.923+0000 7f9107000640 1 -- 172.21.15.39:0/3816061548 >> 172.21.15.39:0/3816061548 conn(0x7f9100076bb0 msgr2=0x7f910010c0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:21.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:21.923+0000 7f9107000640 1 -- 172.21.15.39:0/3816061548 shutdown_connections 2024-03-18T16:06:21.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:21.923+0000 7f9107000640 1 -- 172.21.15.39:0/3816061548 wait complete. 2024-03-18T16:06:21.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:21 smithi194 ceph-mon[36200]: Deploying daemon mds.h on smithi194 2024-03-18T16:06:21.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:21 smithi194 ceph-mon[36200]: mds.? [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] up:boot 2024-03-18T16:06:21.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:21 smithi194 ceph-mon[36200]: fsmap 7 up:standby 2024-03-18T16:06:21.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:21 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-18T16:06:21.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:21 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-18T16:06:22.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:21 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-18T16:06:22.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:21 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-18T16:06:22.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:21 smithi196 ceph-mon[36307]: Deploying daemon mds.h on smithi194 2024-03-18T16:06:22.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:21 smithi196 ceph-mon[36307]: mds.? [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] up:boot 2024-03-18T16:06:22.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:21 smithi196 ceph-mon[36307]: fsmap 7 up:standby 2024-03-18T16:06:22.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:21 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-18T16:06:22.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:21 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-18T16:06:22.972 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:22 smithi194 ceph-mon[36200]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:06:22.972 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:22 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/3816061548' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:06:23.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:22 smithi039 ceph-mon[31721]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:06:23.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:22 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/3816061548' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:06:23.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:22 smithi196 ceph-mon[36307]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:06:23.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:22 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/3816061548' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:06:23.490 INFO:journalctl@ceph.mds.h.smithi194.stdout:Mar 18 16:06:23 smithi194 systemd[1]: Started Ceph mds.h for 1892d8fc-e540-11ee-95c9-87774f69a715. 2024-03-18T16:06:23.945 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:06:24.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:24.174+0000 7f4e05c00640 1 -- 172.21.15.39:0/152898846 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4e001015f0 msgr2=0x7f4e00101a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:24.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:24.174+0000 7f4e05c00640 1 --2- 172.21.15.39:0/152898846 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4e001015f0 0x7f4e00101a10 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f4df0009900 tx=0x7f4df002bc30 comp rx=0 tx=0).stop 2024-03-18T16:06:24.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:24.174+0000 7f4e05c00640 1 -- 172.21.15.39:0/152898846 shutdown_connections 2024-03-18T16:06:24.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:24.174+0000 7f4e05c00640 1 --2- 172.21.15.39:0/152898846 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4e001027e0 0x7f4e00102c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:24.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:24.174+0000 7f4e05c00640 1 --2- 172.21.15.39:0/152898846 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4e001031a0 0x7f4e00109a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:24.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:24.174+0000 7f4e05c00640 1 --2- 172.21.15.39:0/152898846 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4e001015f0 0x7f4e00101a10 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:24.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:24.174+0000 7f4e05c00640 1 -- 172.21.15.39:0/152898846 >> 172.21.15.39:0/152898846 conn(0x7f4e000fcb40 msgr2=0x7f4e000fefa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:24.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:24.174+0000 7f4e05c00640 1 -- 172.21.15.39:0/152898846 shutdown_connections 2024-03-18T16:06:24.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:24.174+0000 7f4e05c00640 1 -- 172.21.15.39:0/152898846 wait complete. 2024-03-18T16:06:24.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:24.175+0000 7f4e05c00640 1 Processor -- start 2024-03-18T16:06:24.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:24.175+0000 7f4e05c00640 1 -- start start 2024-03-18T16:06:24.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:24.175+0000 7f4e05c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4e001015f0 0x7f4e00199b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:24.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:24.175+0000 7f4e05c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4e001027e0 0x7f4e0019a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:24.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:24.175+0000 7f4e05c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4e001031a0 0x7f4e001a10f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:24.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:24.175+0000 7f4e05c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f4e0010d260 con 0x7f4e001027e0 2024-03-18T16:06:24.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:24.175+0000 7f4e05c00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f4e0010d0e0 con 0x7f4e001015f0 2024-03-18T16:06:24.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:24.175+0000 7f4e05c00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f4e0010d3e0 con 0x7f4e001031a0 2024-03-18T16:06:24.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:24.176+0000 7f4dff400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4e001027e0 0x7f4e0019a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:24.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:24.176+0000 7f4dff400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4e001027e0 0x7f4e0019a080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:34428/0 (socket says 172.21.15.39:34428) 2024-03-18T16:06:24.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:24.176+0000 7f4dff400640 1 -- 172.21.15.39:0/2757085811 learned_addr learned my addr 172.21.15.39:0/2757085811 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:06:24.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:24.176+0000 7f4dffe00640 1 --2- 172.21.15.39:0/2757085811 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4e001015f0 0x7f4e00199b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:24.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:24.176+0000 7f4e05200640 1 --2- 172.21.15.39:0/2757085811 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4e001031a0 0x7f4e001a10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:24.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:24.176+0000 7f4dff400640 1 -- 172.21.15.39:0/2757085811 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4e001031a0 msgr2=0x7f4e001a10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:24.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:24.176+0000 7f4dff400640 1 --2- 172.21.15.39:0/2757085811 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4e001031a0 0x7f4e001a10f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:24.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:24.176+0000 7f4dff400640 1 -- 172.21.15.39:0/2757085811 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4e001015f0 msgr2=0x7f4e00199b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:24.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:24.176+0000 7f4dff400640 1 --2- 172.21.15.39:0/2757085811 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4e001015f0 0x7f4e00199b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:24.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:24.176+0000 7f4dff400640 1 -- 172.21.15.39:0/2757085811 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e001a1760 con 0x7f4e001027e0 2024-03-18T16:06:24.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:24.176+0000 7f4dff400640 1 --2- 172.21.15.39:0/2757085811 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4e001027e0 0x7f4e0019a080 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f4dec00ea60 tx=0x7f4dec00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:24.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:24.177+0000 7f4dfcc00640 1 -- 172.21.15.39:0/2757085811 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dec00cde0 con 0x7f4e001027e0 2024-03-18T16:06:24.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:24.177+0000 7f4dfcc00640 1 -- 172.21.15.39:0/2757085811 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4dec0035f0 con 0x7f4e001027e0 2024-03-18T16:06:24.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:24.177+0000 7f4e05c00640 1 -- 172.21.15.39:0/2757085811 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e001a1a50 con 0x7f4e001027e0 2024-03-18T16:06:24.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:24.177+0000 7f4dfcc00640 1 -- 172.21.15.39:0/2757085811 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dec010670 con 0x7f4e001027e0 2024-03-18T16:06:24.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:24.177+0000 7f4e05c00640 1 -- 172.21.15.39:0/2757085811 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e001a1fa0 con 0x7f4e001027e0 2024-03-18T16:06:24.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:24.178+0000 7f4e05c00640 1 -- 172.21.15.39:0/2757085811 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e001a2230 con 0x7f4e001027e0 2024-03-18T16:06:24.183 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:24.181+0000 7f4dfcc00640 1 -- 172.21.15.39:0/2757085811 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f4dec0037a0 con 0x7f4e001027e0 2024-03-18T16:06:24.185 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:24.182+0000 7f4dfcc00640 1 --2- 172.21.15.39:0/2757085811 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f4de40a2800 0x7f4de40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:24.185 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:24.183+0000 7f4dfcc00640 1 -- 172.21.15.39:0/2757085811 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f4dec0cea10 con 0x7f4e001027e0 2024-03-18T16:06:24.185 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:24.183+0000 7f4dffe00640 1 --2- 172.21.15.39:0/2757085811 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f4de40a2800 0x7f4de40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:24.185 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:24.183+0000 7f4dffe00640 1 --2- 172.21.15.39:0/2757085811 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f4de40a2800 0x7f4de40a4cc0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f4df0005ab0 tx=0x7f4df0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:24.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:24.185+0000 7f4dfcc00640 1 -- 172.21.15.39:0/2757085811 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4dec09bf40 con 0x7f4e001027e0 2024-03-18T16:06:24.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:24.428+0000 7f4e05c00640 1 -- 172.21.15.39:0/2757085811 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4e001a2400 con 0x7f4e001027e0 2024-03-18T16:06:24.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:24.429+0000 7f4dfcc00640 1 -- 172.21.15.39:0/2757085811 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7174 (secure 0 0 0) 0x7f4dec09b840 con 0x7f4e001027e0 2024-03-18T16:06:24.432 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:06:24.432 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2520226608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2520226608},{"type":"v1","addr":"172.21.15.39:6835","nonce":2520226608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/719341763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":719341763},{"type":"v1","addr":"172.21.15.39:6837","nonce":719341763}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/931350738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":931350738},{"type":"v1","addr":"172.21.15.194:6833","nonce":931350738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24463,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6833/3874209621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":3874209621},{"type":"v1","addr":"172.21.15.196:6833","nonce":3874209621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24494,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6835/188385470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":188385470},{"type":"v1","addr":"172.21.15.194:6835","nonce":188385470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24523,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6835/3949984309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":3949984309},{"type":"v1","addr":"172.21.15.196:6835","nonce":3949984309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6837/1301464246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":1301464246},{"type":"v1","addr":"172.21.15.194:6837","nonce":1301464246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24535,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6837/895247251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":895247251},{"type":"v1","addr":"172.21.15.196:6837","nonce":895247251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-03-18T16:06:24.433 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 9 2024-03-18T16:06:24.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:24.432+0000 7f4e05c00640 1 -- 172.21.15.39:0/2757085811 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f4de40a2800 msgr2=0x7f4de40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:24.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:24.432+0000 7f4e05c00640 1 --2- 172.21.15.39:0/2757085811 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f4de40a2800 0x7f4de40a4cc0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f4df0005ab0 tx=0x7f4df0005fd0 comp rx=0 tx=0).stop 2024-03-18T16:06:24.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:24.432+0000 7f4e05c00640 1 -- 172.21.15.39:0/2757085811 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4e001027e0 msgr2=0x7f4e0019a080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:24.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:24.432+0000 7f4e05c00640 1 --2- 172.21.15.39:0/2757085811 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4e001027e0 0x7f4e0019a080 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f4dec00ea60 tx=0x7f4dec00ef20 comp rx=0 tx=0).stop 2024-03-18T16:06:24.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:24.432+0000 7f4e05c00640 1 -- 172.21.15.39:0/2757085811 shutdown_connections 2024-03-18T16:06:24.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:24.432+0000 7f4e05c00640 1 --2- 172.21.15.39:0/2757085811 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f4de40a2800 0x7f4de40a4cc0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:24.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:24.432+0000 7f4e05c00640 1 --2- 172.21.15.39:0/2757085811 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4e001031a0 0x7f4e001a10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:24.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:24.432+0000 7f4e05c00640 1 --2- 172.21.15.39:0/2757085811 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4e001027e0 0x7f4e0019a080 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:24.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:24.432+0000 7f4e05c00640 1 --2- 172.21.15.39:0/2757085811 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4e001015f0 0x7f4e00199b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:24.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:24.432+0000 7f4e05c00640 1 -- 172.21.15.39:0/2757085811 >> 172.21.15.39:0/2757085811 conn(0x7f4e000fcb40 msgr2=0x7f4e00073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:24.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:24.433+0000 7f4e05c00640 1 -- 172.21.15.39:0/2757085811 shutdown_connections 2024-03-18T16:06:24.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:24.433+0000 7f4e05c00640 1 -- 172.21.15.39:0/2757085811 wait complete. 2024-03-18T16:06:24.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:24 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:24.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:24 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:24.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:24 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:24.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:24 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T16:06:24.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:24 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T16:06:24.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:24 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:06:24.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:24 smithi196 ceph-mon[36307]: Deploying daemon mds.i on smithi196 2024-03-18T16:06:24.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:24 smithi196 ceph-mon[36307]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:06:24.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:24 smithi196 ceph-mon[36307]: mds.? [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] up:boot 2024-03-18T16:06:24.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:24 smithi196 ceph-mon[36307]: fsmap 8 up:standby 2024-03-18T16:06:24.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:24 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-18T16:06:24.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:24 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:24.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:24 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:24.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:24 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:24.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:24 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T16:06:24.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:24 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T16:06:24.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:24 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:06:24.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:24 smithi194 ceph-mon[36200]: Deploying daemon mds.i on smithi196 2024-03-18T16:06:24.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:24 smithi194 ceph-mon[36200]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:06:24.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:24 smithi194 ceph-mon[36200]: mds.? [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] up:boot 2024-03-18T16:06:24.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:24 smithi194 ceph-mon[36200]: fsmap 8 up:standby 2024-03-18T16:06:24.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:24 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-18T16:06:24.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:24 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:24.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:24 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:24.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:24 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:24.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:24 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T16:06:24.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:24 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T16:06:24.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:24 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:06:24.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:24 smithi039 ceph-mon[31721]: Deploying daemon mds.i on smithi196 2024-03-18T16:06:24.838 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:24 smithi039 ceph-mon[31721]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:06:24.838 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:24 smithi039 ceph-mon[31721]: mds.? [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] up:boot 2024-03-18T16:06:24.838 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:24 smithi039 ceph-mon[31721]: fsmap 8 up:standby 2024-03-18T16:06:24.838 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:24 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-18T16:06:25.701 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:25 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/2757085811' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:06:25.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:25 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/2757085811' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:06:25.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:25 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/2757085811' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:06:25.978 INFO:journalctl@ceph.mds.i.smithi196.stdout:Mar 18 16:06:25 smithi196 podman[66972]: 2024-03-18 16:06:25.926833149 +0000 UTC m=+0.126390737 container create d4b43a4f0cf797248641000842847c04b69a1313be7ff0d780c8e3358420527d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-i, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T16:06:26.456 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:06:26.478 INFO:journalctl@ceph.mds.i.smithi196.stdout:Mar 18 16:06:26 smithi196 podman[66972]: 2024-03-18 16:06:26.171274896 +0000 UTC m=+0.370832499 container init d4b43a4f0cf797248641000842847c04b69a1313be7ff0d780c8e3358420527d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-i, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T16:06:26.478 INFO:journalctl@ceph.mds.i.smithi196.stdout:Mar 18 16:06:26 smithi196 podman[66972]: 2024-03-18 16:06:26.174153161 +0000 UTC m=+0.373710742 container start d4b43a4f0cf797248641000842847c04b69a1313be7ff0d780c8e3358420527d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-i, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307) 2024-03-18T16:06:26.478 INFO:journalctl@ceph.mds.i.smithi196.stdout:Mar 18 16:06:26 smithi196 ceph-mds[66991]: -- 172.21.15.196:0/1281940398 <== mon.1 v2:172.21.15.196:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555aebd95a40 con 0x555aebd69000 2024-03-18T16:06:26.478 INFO:journalctl@ceph.mds.i.smithi196.stdout:Mar 18 16:06:26 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-i[66987]: starting mds.i at 2024-03-18T16:06:26.478 INFO:journalctl@ceph.mds.i.smithi196.stdout:Mar 18 16:06:26 smithi196 bash[66972]: d4b43a4f0cf797248641000842847c04b69a1313be7ff0d780c8e3358420527d 2024-03-18T16:06:26.478 INFO:journalctl@ceph.mds.i.smithi196.stdout:Mar 18 16:06:26 smithi196 systemd[1]: Started Ceph mds.i for 1892d8fc-e540-11ee-95c9-87774f69a715. 2024-03-18T16:06:26.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:26.653+0000 7f0aa3c00640 1 -- 172.21.15.39:0/500706343 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0a9c107e00 msgr2=0x7f0a9c108220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:26.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:26.653+0000 7f0aa3c00640 1 --2- 172.21.15.39:0/500706343 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0a9c107e00 0x7f0a9c108220 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0a94009ca0 tx=0x7f0a9402c100 comp rx=0 tx=0).stop 2024-03-18T16:06:26.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:26.653+0000 7f0aa3c00640 1 -- 172.21.15.39:0/500706343 shutdown_connections 2024-03-18T16:06:26.656 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:26.653+0000 7f0aa3c00640 1 --2- 172.21.15.39:0/500706343 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0a9c1099b0 0x7f0a9c110240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:26.656 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:26.653+0000 7f0aa3c00640 1 --2- 172.21.15.39:0/500706343 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0a9c108ff0 0x7f0a9c109470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:26.656 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:26.653+0000 7f0aa3c00640 1 --2- 172.21.15.39:0/500706343 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0a9c107e00 0x7f0a9c108220 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:26.656 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:26.653+0000 7f0aa3c00640 1 -- 172.21.15.39:0/500706343 >> 172.21.15.39:0/500706343 conn(0x7f0a9c076dc0 msgr2=0x7f0a9c0791e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:26.656 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:26.653+0000 7f0aa3c00640 1 -- 172.21.15.39:0/500706343 shutdown_connections 2024-03-18T16:06:26.656 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:26.653+0000 7f0aa3c00640 1 -- 172.21.15.39:0/500706343 wait complete. 2024-03-18T16:06:26.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:26.654+0000 7f0aa3c00640 1 Processor -- start 2024-03-18T16:06:26.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:26.654+0000 7f0aa3c00640 1 -- start start 2024-03-18T16:06:26.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:26.654+0000 7f0aa3c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0a9c107e00 0x7f0a9c19e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:26.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:26.654+0000 7f0aa3c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0a9c108ff0 0x7f0a9c19e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:26.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:26.654+0000 7f0aa3c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0a9c1099b0 0x7f0a9c1a56f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:26.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:26.654+0000 7f0aa3c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f0a9c113a70 con 0x7f0a9c107e00 2024-03-18T16:06:26.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:26.654+0000 7f0aa3c00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f0a9c1138f0 con 0x7f0a9c108ff0 2024-03-18T16:06:26.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:26.654+0000 7f0aa3c00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f0a9c113bf0 con 0x7f0a9c1099b0 2024-03-18T16:06:26.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:26.655+0000 7f0aa1200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0a9c107e00 0x7f0a9c19e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:26.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:26.655+0000 7f0aa1200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0a9c107e00 0x7f0a9c19e140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:34446/0 (socket says 172.21.15.39:34446) 2024-03-18T16:06:26.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:26.655+0000 7f0aa1200640 1 -- 172.21.15.39:0/3923804651 learned_addr learned my addr 172.21.15.39:0/3923804651 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:06:26.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:26.655+0000 7f0aa1c00640 1 --2- 172.21.15.39:0/3923804651 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0a9c1099b0 0x7f0a9c1a56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:26.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:26.655+0000 7f0a9be00640 1 --2- 172.21.15.39:0/3923804651 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0a9c108ff0 0x7f0a9c19e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:26.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:26.655+0000 7f0aa1200640 1 -- 172.21.15.39:0/3923804651 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0a9c1099b0 msgr2=0x7f0a9c1a56f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:26.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:26.655+0000 7f0aa1200640 1 --2- 172.21.15.39:0/3923804651 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0a9c1099b0 0x7f0a9c1a56f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:26.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:26.655+0000 7f0aa1200640 1 -- 172.21.15.39:0/3923804651 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0a9c108ff0 msgr2=0x7f0a9c19e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:26.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:26.655+0000 7f0aa1200640 1 --2- 172.21.15.39:0/3923804651 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0a9c108ff0 0x7f0a9c19e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:26.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:26.655+0000 7f0aa1200640 1 -- 172.21.15.39:0/3923804651 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a9c1a5d60 con 0x7f0a9c107e00 2024-03-18T16:06:26.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:26.655+0000 7f0a9be00640 1 --2- 172.21.15.39:0/3923804651 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0a9c108ff0 0x7f0a9c19e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:06:26.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:26.655+0000 7f0aa1200640 1 --2- 172.21.15.39:0/3923804651 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0a9c107e00 0x7f0a9c19e140 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f0a940324c0 tx=0x7f0a94035d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:26.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:26.656+0000 7f0a99600640 1 -- 172.21.15.39:0/3923804651 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a94039040 con 0x7f0a9c107e00 2024-03-18T16:06:26.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:26.656+0000 7f0a99600640 1 -- 172.21.15.39:0/3923804651 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0a94044070 con 0x7f0a9c107e00 2024-03-18T16:06:26.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:26.656+0000 7f0aa3c00640 1 -- 172.21.15.39:0/3923804651 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a9c1a5ff0 con 0x7f0a9c107e00 2024-03-18T16:06:26.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:26.656+0000 7f0a99600640 1 -- 172.21.15.39:0/3923804651 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a94004370 con 0x7f0a9c107e00 2024-03-18T16:06:26.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:26.656+0000 7f0aa3c00640 1 -- 172.21.15.39:0/3923804651 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a9c1a6400 con 0x7f0a9c107e00 2024-03-18T16:06:26.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:26.657+0000 7f0aa3c00640 1 -- 172.21.15.39:0/3923804651 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a9c10f1a0 con 0x7f0a9c107e00 2024-03-18T16:06:26.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:26.659+0000 7f0a99600640 1 -- 172.21.15.39:0/3923804651 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f0a94004510 con 0x7f0a9c107e00 2024-03-18T16:06:26.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:26.660+0000 7f0a99600640 1 --2- 172.21.15.39:0/3923804651 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f0a800a2800 0x7f0a800a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:26.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:26.661+0000 7f0a99600640 1 -- 172.21.15.39:0/3923804651 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f0a940f1780 con 0x7f0a9c107e00 2024-03-18T16:06:26.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:26.661+0000 7f0a9be00640 1 --2- 172.21.15.39:0/3923804651 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f0a800a2800 0x7f0a800a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:26.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:26.661+0000 7f0a9be00640 1 --2- 172.21.15.39:0/3923804651 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f0a800a2800 0x7f0a800a4cc0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f0a9c19f650 tx=0x7f0a88005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:26.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:26.668+0000 7f0a99600640 1 -- 172.21.15.39:0/3923804651 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0a940bec20 con 0x7f0a9c107e00 2024-03-18T16:06:26.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:26.915+0000 7f0aa3c00640 1 -- 172.21.15.39:0/3923804651 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0a9c02d1b0 con 0x7f0a9c107e00 2024-03-18T16:06:26.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:26.917+0000 7f0a99600640 1 -- 172.21.15.39:0/3923804651 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8004 (secure 0 0 0) 0x7f0a940be520 con 0x7f0a9c107e00 2024-03-18T16:06:26.919 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:06:26.919 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2520226608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2520226608},{"type":"v1","addr":"172.21.15.39:6835","nonce":2520226608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/719341763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":719341763},{"type":"v1","addr":"172.21.15.39:6837","nonce":719341763}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/931350738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":931350738},{"type":"v1","addr":"172.21.15.194:6833","nonce":931350738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24463,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6833/3874209621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":3874209621},{"type":"v1","addr":"172.21.15.196:6833","nonce":3874209621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24494,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6835/188385470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":188385470},{"type":"v1","addr":"172.21.15.194:6835","nonce":188385470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24523,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6835/3949984309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":3949984309},{"type":"v1","addr":"172.21.15.196:6835","nonce":3949984309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6837/1301464246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":1301464246},{"type":"v1","addr":"172.21.15.194:6837","nonce":1301464246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24535,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6837/895247251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":895247251},{"type":"v1","addr":"172.21.15.196:6837","nonce":895247251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24556,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6839/2001427268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6838","nonce":2001427268},{"type":"v1","addr":"172.21.15.196:6839","nonce":2001427268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-03-18T16:06:26.920 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 10 2024-03-18T16:06:26.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:26.919+0000 7f0aa3c00640 1 -- 172.21.15.39:0/3923804651 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f0a800a2800 msgr2=0x7f0a800a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:26.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:26.919+0000 7f0aa3c00640 1 --2- 172.21.15.39:0/3923804651 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f0a800a2800 0x7f0a800a4cc0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f0a9c19f650 tx=0x7f0a88005f90 comp rx=0 tx=0).stop 2024-03-18T16:06:26.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:26.919+0000 7f0aa3c00640 1 -- 172.21.15.39:0/3923804651 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0a9c107e00 msgr2=0x7f0a9c19e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:26.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:26.919+0000 7f0aa3c00640 1 --2- 172.21.15.39:0/3923804651 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0a9c107e00 0x7f0a9c19e140 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f0a940324c0 tx=0x7f0a94035d90 comp rx=0 tx=0).stop 2024-03-18T16:06:26.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:26.920+0000 7f0aa3c00640 1 -- 172.21.15.39:0/3923804651 shutdown_connections 2024-03-18T16:06:26.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:26.920+0000 7f0aa3c00640 1 --2- 172.21.15.39:0/3923804651 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0a9c108ff0 0x7f0a9c19e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:26.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:26.920+0000 7f0aa3c00640 1 --2- 172.21.15.39:0/3923804651 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f0a800a2800 0x7f0a800a4cc0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:26.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:26.920+0000 7f0aa3c00640 1 --2- 172.21.15.39:0/3923804651 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0a9c1099b0 0x7f0a9c1a56f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:26.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:26.920+0000 7f0aa3c00640 1 --2- 172.21.15.39:0/3923804651 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0a9c107e00 0x7f0a9c19e140 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:26.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:26.920+0000 7f0aa3c00640 1 -- 172.21.15.39:0/3923804651 >> 172.21.15.39:0/3923804651 conn(0x7f0a9c076dc0 msgr2=0x7f0a9c10c370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:26.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:26.920+0000 7f0aa3c00640 1 -- 172.21.15.39:0/3923804651 shutdown_connections 2024-03-18T16:06:26.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:26.920+0000 7f0aa3c00640 1 -- 172.21.15.39:0/3923804651 wait complete. 2024-03-18T16:06:27.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:27 smithi039 ceph-mon[31721]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:06:27.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:27 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:27.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:27 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:27.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:27 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:27.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:27 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T16:06:27.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:27 smithi039 ceph-mon[31721]: mds.? [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] up:boot 2024-03-18T16:06:27.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:27 smithi039 ceph-mon[31721]: fsmap 9 up:standby 2024-03-18T16:06:27.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:27 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-18T16:06:27.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:27 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T16:06:27.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:27 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:06:27.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:27 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/3923804651' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:06:27.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:27 smithi196 ceph-mon[36307]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:06:27.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:27 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:27.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:27 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:27.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:27 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:27.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:27 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T16:06:27.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:27 smithi196 ceph-mon[36307]: mds.? [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] up:boot 2024-03-18T16:06:27.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:27 smithi196 ceph-mon[36307]: fsmap 9 up:standby 2024-03-18T16:06:27.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:27 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-18T16:06:27.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:27 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T16:06:27.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:27 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:06:27.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:27 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/3923804651' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:06:27.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:27 smithi194 ceph-mon[36200]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:06:27.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:27 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:27.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:27 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:27.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:27 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:27.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:27 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T16:06:27.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:27 smithi194 ceph-mon[36200]: mds.? [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] up:boot 2024-03-18T16:06:27.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:27 smithi194 ceph-mon[36200]: fsmap 9 up:standby 2024-03-18T16:06:27.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:27 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-18T16:06:27.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:27 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T16:06:27.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:27 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:06:27.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:27 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/3923804651' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:06:28.665 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:28 smithi196 ceph-mon[36307]: Deploying daemon mds.g on smithi039 2024-03-18T16:06:28.695 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:28 smithi039 ceph-mon[31721]: Deploying daemon mds.g on smithi039 2024-03-18T16:06:28.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:28 smithi194 ceph-mon[36200]: Deploying daemon mds.g on smithi039 2024-03-18T16:06:28.942 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:06:29.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:29.422+0000 7f851b800640 1 -- 172.21.15.39:0/4191768627 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8514103a00 msgr2=0x7f8514103e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:29.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:29.422+0000 7f851b800640 1 --2- 172.21.15.39:0/4191768627 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8514103a00 0x7f8514103e20 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f8500009900 tx=0x7f850002bc80 comp rx=0 tx=0).stop 2024-03-18T16:06:29.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:29.422+0000 7f851b800640 1 -- 172.21.15.39:0/4191768627 shutdown_connections 2024-03-18T16:06:29.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:29.422+0000 7f851b800640 1 --2- 172.21.15.39:0/4191768627 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8514104bf0 0x7f8514105070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:29.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:29.422+0000 7f851b800640 1 --2- 172.21.15.39:0/4191768627 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f85141055b0 0x7f851410be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:29.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:29.422+0000 7f851b800640 1 --2- 172.21.15.39:0/4191768627 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8514103a00 0x7f8514103e20 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:29.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:29.422+0000 7f851b800640 1 -- 172.21.15.39:0/4191768627 >> 172.21.15.39:0/4191768627 conn(0x7f85140fef90 msgr2=0x7f85141013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:29.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:29.423+0000 7f851b800640 1 -- 172.21.15.39:0/4191768627 shutdown_connections 2024-03-18T16:06:29.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:29.423+0000 7f851b800640 1 -- 172.21.15.39:0/4191768627 wait complete. 2024-03-18T16:06:29.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:29.425+0000 7f851b800640 1 Processor -- start 2024-03-18T16:06:29.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:29.425+0000 7f851b800640 1 -- start start 2024-03-18T16:06:29.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:29.425+0000 7f851b800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8514104bf0 0x7f8514195be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:29.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:29.425+0000 7f851b800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f85141055b0 0x7f8514196120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:29.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:29.425+0000 7f851b800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f851419b1c0 0x7f851419d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:29.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:29.426+0000 7f851b800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f851410f7e0 con 0x7f851419b1c0 2024-03-18T16:06:29.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:29.426+0000 7f851b800640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f851410f660 con 0x7f85141055b0 2024-03-18T16:06:29.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:29.426+0000 7f851b800640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f851410f960 con 0x7f8514104bf0 2024-03-18T16:06:29.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:29.426+0000 7f8518e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8514104bf0 0x7f8514195be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:29.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:29.426+0000 7f8518e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8514104bf0 0x7f8514195be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:49790/0 (socket says 172.21.15.39:49790) 2024-03-18T16:06:29.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:29.426+0000 7f8518e00640 1 -- 172.21.15.39:0/4137179831 learned_addr learned my addr 172.21.15.39:0/4137179831 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:06:29.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:29.426+0000 7f8513e00640 1 --2- 172.21.15.39:0/4137179831 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f85141055b0 0x7f8514196120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:29.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:29.426+0000 7f8519800640 1 --2- 172.21.15.39:0/4137179831 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f851419b1c0 0x7f851419d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:29.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:29.426+0000 7f8518e00640 1 -- 172.21.15.39:0/4137179831 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f85141055b0 msgr2=0x7f8514196120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:29.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:29.426+0000 7f8518e00640 1 --2- 172.21.15.39:0/4137179831 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f85141055b0 0x7f8514196120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:29.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:29.426+0000 7f8518e00640 1 -- 172.21.15.39:0/4137179831 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f851419b1c0 msgr2=0x7f851419d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:29.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:29.426+0000 7f8518e00640 1 --2- 172.21.15.39:0/4137179831 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f851419b1c0 0x7f851419d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:29.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:29.426+0000 7f8518e00640 1 -- 172.21.15.39:0/4137179831 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f851419dc40 con 0x7f8514104bf0 2024-03-18T16:06:29.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:29.428+0000 7f8518e00640 1 --2- 172.21.15.39:0/4137179831 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8514104bf0 0x7f8514195be0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8500036040 tx=0x7f8500004040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:29.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:29.428+0000 7f8511600640 1 -- 172.21.15.39:0/4137179831 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8500041070 con 0x7f8514104bf0 2024-03-18T16:06:29.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:29.428+0000 7f8511600640 1 -- 172.21.15.39:0/4137179831 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f85000054e0 con 0x7f8514104bf0 2024-03-18T16:06:29.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:29.428+0000 7f8511600640 1 -- 172.21.15.39:0/4137179831 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85000346a0 con 0x7f8514104bf0 2024-03-18T16:06:29.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:29.428+0000 7f851b800640 1 -- 172.21.15.39:0/4137179831 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f851419ded0 con 0x7f8514104bf0 2024-03-18T16:06:29.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:29.429+0000 7f851b800640 1 -- 172.21.15.39:0/4137179831 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f851419e370 con 0x7f8514104bf0 2024-03-18T16:06:29.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:29.430+0000 7f851b800640 1 -- 172.21.15.39:0/4137179831 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84f40052c0 con 0x7f8514104bf0 2024-03-18T16:06:29.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:29.435+0000 7f8511600640 1 -- 172.21.15.39:0/4137179831 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f8500044070 con 0x7f8514104bf0 2024-03-18T16:06:29.440 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:29.438+0000 7f8511600640 1 --2- 172.21.15.39:0/4137179831 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f84f80a29d0 0x7f84f80a4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:29.440 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:29.438+0000 7f8511600640 1 -- 172.21.15.39:0/4137179831 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f85000f0470 con 0x7f8514104bf0 2024-03-18T16:06:29.440 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:29.439+0000 7f8513e00640 1 --2- 172.21.15.39:0/4137179831 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f84f80a29d0 0x7f84f80a4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:29.440 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:29.439+0000 7f8511600640 1 -- 172.21.15.39:0/4137179831 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f85000bd990 con 0x7f8514104bf0 2024-03-18T16:06:29.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:29.444+0000 7f8513e00640 1 --2- 172.21.15.39:0/4137179831 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f84f80a29d0 0x7f84f80a4e90 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f8514196d00 tx=0x7f850c0094f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:29.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:29 smithi039 ceph-mon[31721]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:06:29.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:29.709+0000 7f851b800640 1 -- 172.21.15.39:0/4137179831 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f84f40055b0 con 0x7f8514104bf0 2024-03-18T16:06:29.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:29.709+0000 7f8511600640 1 -- 172.21.15.39:0/4137179831 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8004 (secure 0 0 0) 0x7f85000bd990 con 0x7f8514104bf0 2024-03-18T16:06:29.711 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:06:29.711 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2520226608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2520226608},{"type":"v1","addr":"172.21.15.39:6835","nonce":2520226608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/719341763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":719341763},{"type":"v1","addr":"172.21.15.39:6837","nonce":719341763}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/931350738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":931350738},{"type":"v1","addr":"172.21.15.194:6833","nonce":931350738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24463,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6833/3874209621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":3874209621},{"type":"v1","addr":"172.21.15.196:6833","nonce":3874209621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24494,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6835/188385470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":188385470},{"type":"v1","addr":"172.21.15.194:6835","nonce":188385470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24523,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6835/3949984309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":3949984309},{"type":"v1","addr":"172.21.15.196:6835","nonce":3949984309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6837/1301464246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":1301464246},{"type":"v1","addr":"172.21.15.194:6837","nonce":1301464246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24535,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6837/895247251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":895247251},{"type":"v1","addr":"172.21.15.196:6837","nonce":895247251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24556,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6839/2001427268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6838","nonce":2001427268},{"type":"v1","addr":"172.21.15.196:6839","nonce":2001427268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-03-18T16:06:29.711 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 10 2024-03-18T16:06:29.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:29.713+0000 7f851b800640 1 -- 172.21.15.39:0/4137179831 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f84f80a29d0 msgr2=0x7f84f80a4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:29.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:29.713+0000 7f851b800640 1 --2- 172.21.15.39:0/4137179831 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f84f80a29d0 0x7f84f80a4e90 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f8514196d00 tx=0x7f850c0094f0 comp rx=0 tx=0).stop 2024-03-18T16:06:29.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:29.713+0000 7f851b800640 1 -- 172.21.15.39:0/4137179831 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8514104bf0 msgr2=0x7f8514195be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:29.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:29.713+0000 7f851b800640 1 --2- 172.21.15.39:0/4137179831 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8514104bf0 0x7f8514195be0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f8500036040 tx=0x7f8500004040 comp rx=0 tx=0).stop 2024-03-18T16:06:29.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:29.713+0000 7f851b800640 1 -- 172.21.15.39:0/4137179831 shutdown_connections 2024-03-18T16:06:29.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:29.713+0000 7f851b800640 1 --2- 172.21.15.39:0/4137179831 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f85141055b0 0x7f8514196120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:29.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:29.713+0000 7f851b800640 1 --2- 172.21.15.39:0/4137179831 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f84f80a29d0 0x7f84f80a4e90 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:29.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:29.713+0000 7f851b800640 1 --2- 172.21.15.39:0/4137179831 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f851419b1c0 0x7f851419d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:29.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:29.713+0000 7f851b800640 1 --2- 172.21.15.39:0/4137179831 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8514104bf0 0x7f8514195be0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:29.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:29.713+0000 7f851b800640 1 -- 172.21.15.39:0/4137179831 >> 172.21.15.39:0/4137179831 conn(0x7f85140fef90 msgr2=0x7f8514077d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:29.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:29.713+0000 7f851b800640 1 -- 172.21.15.39:0/4137179831 shutdown_connections 2024-03-18T16:06:29.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:29.713+0000 7f851b800640 1 -- 172.21.15.39:0/4137179831 wait complete. 2024-03-18T16:06:29.727 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:29 smithi196 ceph-mon[36307]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:06:29.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:29 smithi194 ceph-mon[36200]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:06:30.233 INFO:journalctl@ceph.mds.g.smithi039.stdout:Mar 18 16:06:30 smithi039 podman[70901]: 2024-03-18 16:06:30.06653602 +0000 UTC m=+0.158108572 container create d8fdf2cc8f12929b85f928ed3fca3609551801f4285ccd1e7b3eaab8cab9fcdf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-g, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T16:06:30.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:30 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/4137179831' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:06:30.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:30 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/4137179831' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:06:30.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:30 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/4137179831' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:06:30.837 INFO:journalctl@ceph.mds.g.smithi039.stdout:Mar 18 16:06:30 smithi039 podman[70901]: 2024-03-18 16:06:30.345270814 +0000 UTC m=+0.436843375 container init d8fdf2cc8f12929b85f928ed3fca3609551801f4285ccd1e7b3eaab8cab9fcdf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-g, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-18T16:06:30.837 INFO:journalctl@ceph.mds.g.smithi039.stdout:Mar 18 16:06:30 smithi039 podman[70901]: 2024-03-18 16:06:30.35380142 +0000 UTC m=+0.445373946 container start d8fdf2cc8f12929b85f928ed3fca3609551801f4285ccd1e7b3eaab8cab9fcdf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-g, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-03-18T16:06:30.837 INFO:journalctl@ceph.mds.g.smithi039.stdout:Mar 18 16:06:30 smithi039 ceph-mds[70926]: -- 172.21.15.39:0/1700795656 <== mon.0 v2:172.21.15.39:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563cd27dda40 con 0x563cd27b1000 2024-03-18T16:06:30.837 INFO:journalctl@ceph.mds.g.smithi039.stdout:Mar 18 16:06:30 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-g[70920]: starting mds.g at 2024-03-18T16:06:30.837 INFO:journalctl@ceph.mds.g.smithi039.stdout:Mar 18 16:06:30 smithi039 bash[70901]: d8fdf2cc8f12929b85f928ed3fca3609551801f4285ccd1e7b3eaab8cab9fcdf 2024-03-18T16:06:30.837 INFO:journalctl@ceph.mds.g.smithi039.stdout:Mar 18 16:06:30 smithi039 systemd[1]: Started Ceph mds.g for 1892d8fc-e540-11ee-95c9-87774f69a715. 2024-03-18T16:06:31.706 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:31 smithi039 ceph-mon[31721]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:06:31.706 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:31 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:31.706 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:31 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:31.706 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:31 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:31.706 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:31 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T16:06:31.706 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:31 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T16:06:31.706 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:31 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:06:31.706 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:31 smithi039 ceph-mon[31721]: Deploying daemon mds.a on smithi039 2024-03-18T16:06:31.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:31 smithi196 ceph-mon[36307]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:06:31.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:31 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:31.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:31 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:31.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:31 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:31.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:31 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T16:06:31.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:31 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T16:06:31.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:31 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:06:31.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:31 smithi196 ceph-mon[36307]: Deploying daemon mds.a on smithi039 2024-03-18T16:06:31.736 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:06:31.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:31 smithi194 ceph-mon[36200]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:06:31.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:31 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:31.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:31 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:31.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:31 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:31.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:31 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T16:06:31.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:31 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T16:06:31.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:31 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:06:31.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:31 smithi194 ceph-mon[36200]: Deploying daemon mds.a on smithi039 2024-03-18T16:06:31.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:31.957+0000 7fd1ff200640 1 -- 172.21.15.39:0/4059058770 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd20010dbc0 msgr2=0x7fd20010ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:31.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:31.957+0000 7fd1ff200640 1 --2- 172.21.15.39:0/4059058770 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd20010dbc0 0x7fd20010ffb0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd1f8009cd0 tx=0x7fd1f802c0f0 comp rx=0 tx=0).stop 2024-03-18T16:06:31.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:31.957+0000 7fd1ff200640 1 -- 172.21.15.39:0/4059058770 shutdown_connections 2024-03-18T16:06:31.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:31.957+0000 7fd1ff200640 1 --2- 172.21.15.39:0/4059058770 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd20010dbc0 0x7fd20010ffb0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:31.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:31.957+0000 7fd1ff200640 1 --2- 172.21.15.39:0/4059058770 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd200074240 0x7fd2000746c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:31.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:31.957+0000 7fd1ff200640 1 --2- 172.21.15.39:0/4059058770 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd200075820 0x7fd200073c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:31.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:31.957+0000 7fd1ff200640 1 -- 172.21.15.39:0/4059058770 >> 172.21.15.39:0/4059058770 conn(0x7fd2000fcb40 msgr2=0x7fd2000fefa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:31.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:31.957+0000 7fd1ff200640 1 -- 172.21.15.39:0/4059058770 shutdown_connections 2024-03-18T16:06:31.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:31.957+0000 7fd1ff200640 1 -- 172.21.15.39:0/4059058770 wait complete. 2024-03-18T16:06:31.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:31.958+0000 7fd1ff200640 1 Processor -- start 2024-03-18T16:06:31.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:31.958+0000 7fd1ff200640 1 -- start start 2024-03-18T16:06:31.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:31.958+0000 7fd1ff200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd200074240 0x7fd20019deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:31.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:31.958+0000 7fd1ff200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd200075820 0x7fd20019e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:31.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:31.958+0000 7fd1ff200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd20010dbc0 0x7fd2001a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:31.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:31.958+0000 7fd1ff200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fd200113860 con 0x7fd200074240 2024-03-18T16:06:31.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:31.958+0000 7fd1ff200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fd2001136e0 con 0x7fd200075820 2024-03-18T16:06:31.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:31.958+0000 7fd1ff200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fd2001139e0 con 0x7fd20010dbc0 2024-03-18T16:06:31.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:31.958+0000 7fd1fde00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd200074240 0x7fd20019deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:31.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:31.959+0000 7fd1fde00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd200074240 0x7fd20019deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43230/0 (socket says 172.21.15.39:43230) 2024-03-18T16:06:31.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:31.959+0000 7fd1fde00640 1 -- 172.21.15.39:0/3633550894 learned_addr learned my addr 172.21.15.39:0/3633550894 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:06:31.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:31.959+0000 7fd1fd400640 1 --2- 172.21.15.39:0/3633550894 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd200075820 0x7fd20019e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:31.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:31.959+0000 7fd1fe800640 1 --2- 172.21.15.39:0/3633550894 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd20010dbc0 0x7fd2001a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:31.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:31.959+0000 7fd1fde00640 1 -- 172.21.15.39:0/3633550894 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd20010dbc0 msgr2=0x7fd2001a5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:31.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:31.959+0000 7fd1fde00640 1 --2- 172.21.15.39:0/3633550894 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd20010dbc0 0x7fd2001a5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:31.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:31.959+0000 7fd1fde00640 1 -- 172.21.15.39:0/3633550894 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd200075820 msgr2=0x7fd20019e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:31.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:31.959+0000 7fd1fde00640 1 --2- 172.21.15.39:0/3633550894 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd200075820 0x7fd20019e3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:31.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:31.959+0000 7fd1fde00640 1 -- 172.21.15.39:0/3633550894 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2001a5ad0 con 0x7fd200074240 2024-03-18T16:06:31.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:31.959+0000 7fd1fde00640 1 --2- 172.21.15.39:0/3633550894 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd200074240 0x7fd20019deb0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd1f000e9f0 tx=0x7fd1f000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:31.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:31.959+0000 7fd1f6a00640 1 -- 172.21.15.39:0/3633550894 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1f000cd80 con 0x7fd200074240 2024-03-18T16:06:31.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:31.959+0000 7fd1f6a00640 1 -- 172.21.15.39:0/3633550894 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1f0010910 con 0x7fd200074240 2024-03-18T16:06:31.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:31.960+0000 7fd1f6a00640 1 -- 172.21.15.39:0/3633550894 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1f0018990 con 0x7fd200074240 2024-03-18T16:06:31.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:31.960+0000 7fd1ff200640 1 -- 172.21.15.39:0/3633550894 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2001a5dc0 con 0x7fd200074240 2024-03-18T16:06:31.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:31.960+0000 7fd1ff200640 1 -- 172.21.15.39:0/3633550894 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2001a6230 con 0x7fd200074240 2024-03-18T16:06:31.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:31.961+0000 7fd1ff200640 1 -- 172.21.15.39:0/3633550894 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd200103d90 con 0x7fd200074240 2024-03-18T16:06:31.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:31.963+0000 7fd1f6a00640 1 -- 172.21.15.39:0/3633550894 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fd1f0010ac0 con 0x7fd200074240 2024-03-18T16:06:31.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:31.964+0000 7fd1f6a00640 1 --2- 172.21.15.39:0/3633550894 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd1e40a2880 0x7fd1e40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:31.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:31.964+0000 7fd1f6a00640 1 -- 172.21.15.39:0/3633550894 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fd1f00ce8f0 con 0x7fd200074240 2024-03-18T16:06:31.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:31.965+0000 7fd1fd400640 1 --2- 172.21.15.39:0/3633550894 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd1e40a2880 0x7fd1e40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:31.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:31.965+0000 7fd1fd400640 1 --2- 172.21.15.39:0/3633550894 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd1e40a2880 0x7fd1e40a4d40 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd20019f3c0 tx=0x7fd1ec005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:31.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:31.969+0000 7fd1f6a00640 1 -- 172.21.15.39:0/3633550894 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd1f009b740 con 0x7fd200074240 2024-03-18T16:06:32.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:32.215+0000 7fd1ff200640 1 -- 172.21.15.39:0/3633550894 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd20002d3f0 con 0x7fd200074240 2024-03-18T16:06:32.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:32.217+0000 7fd1f6a00640 1 -- 172.21.15.39:0/3633550894 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8827 (secure 0 0 0) 0x7fd1f009f7e0 con 0x7fd200074240 2024-03-18T16:06:32.219 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:06:32.219 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2520226608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2520226608},{"type":"v1","addr":"172.21.15.39:6835","nonce":2520226608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/719341763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":719341763},{"type":"v1","addr":"172.21.15.39:6837","nonce":719341763}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14445,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6839/656513813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":656513813},{"type":"v1","addr":"172.21.15.39:6839","nonce":656513813}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/931350738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":931350738},{"type":"v1","addr":"172.21.15.194:6833","nonce":931350738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24463,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6833/3874209621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":3874209621},{"type":"v1","addr":"172.21.15.196:6833","nonce":3874209621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24494,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6835/188385470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":188385470},{"type":"v1","addr":"172.21.15.194:6835","nonce":188385470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24523,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6835/3949984309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":3949984309},{"type":"v1","addr":"172.21.15.196:6835","nonce":3949984309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6837/1301464246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":1301464246},{"type":"v1","addr":"172.21.15.194:6837","nonce":1301464246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24535,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6837/895247251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":895247251},{"type":"v1","addr":"172.21.15.196:6837","nonce":895247251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24556,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6839/2001427268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6838","nonce":2001427268},{"type":"v1","addr":"172.21.15.196:6839","nonce":2001427268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-03-18T16:06:32.219 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 11 2024-03-18T16:06:32.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:32.220+0000 7fd1ff200640 1 -- 172.21.15.39:0/3633550894 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd1e40a2880 msgr2=0x7fd1e40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:32.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:32.220+0000 7fd1ff200640 1 --2- 172.21.15.39:0/3633550894 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd1e40a2880 0x7fd1e40a4d40 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd20019f3c0 tx=0x7fd1ec005f90 comp rx=0 tx=0).stop 2024-03-18T16:06:32.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:32.220+0000 7fd1ff200640 1 -- 172.21.15.39:0/3633550894 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd200074240 msgr2=0x7fd20019deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:32.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:32.220+0000 7fd1ff200640 1 --2- 172.21.15.39:0/3633550894 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd200074240 0x7fd20019deb0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd1f000e9f0 tx=0x7fd1f000eeb0 comp rx=0 tx=0).stop 2024-03-18T16:06:32.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:32.220+0000 7fd1ff200640 1 -- 172.21.15.39:0/3633550894 shutdown_connections 2024-03-18T16:06:32.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:32.220+0000 7fd1ff200640 1 --2- 172.21.15.39:0/3633550894 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd200075820 0x7fd20019e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:32.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:32.220+0000 7fd1ff200640 1 --2- 172.21.15.39:0/3633550894 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd1e40a2880 0x7fd1e40a4d40 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:32.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:32.220+0000 7fd1ff200640 1 --2- 172.21.15.39:0/3633550894 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd20010dbc0 0x7fd2001a5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:32.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:32.220+0000 7fd1ff200640 1 --2- 172.21.15.39:0/3633550894 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd200074240 0x7fd20019deb0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:32.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:32.220+0000 7fd1ff200640 1 -- 172.21.15.39:0/3633550894 >> 172.21.15.39:0/3633550894 conn(0x7fd2000fcb40 msgr2=0x7fd200100f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:32.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:32.220+0000 7fd1ff200640 1 -- 172.21.15.39:0/3633550894 shutdown_connections 2024-03-18T16:06:32.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:32.220+0000 7fd1ff200640 1 -- 172.21.15.39:0/3633550894 wait complete. 2024-03-18T16:06:32.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:32 smithi196 ceph-mon[36307]: mds.? [v2:172.21.15.39:6838/656513813,v1:172.21.15.39:6839/656513813] up:boot 2024-03-18T16:06:32.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:32 smithi196 ceph-mon[36307]: fsmap 10 up:standby 2024-03-18T16:06:32.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:32 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-18T16:06:32.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:32 smithi196 ceph-mon[36307]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:06:32.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:32 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/3633550894' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:06:32.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:32 smithi194 ceph-mon[36200]: mds.? [v2:172.21.15.39:6838/656513813,v1:172.21.15.39:6839/656513813] up:boot 2024-03-18T16:06:32.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:32 smithi194 ceph-mon[36200]: fsmap 10 up:standby 2024-03-18T16:06:32.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:32 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-18T16:06:32.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:32 smithi194 ceph-mon[36200]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:06:32.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:32 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/3633550894' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:06:32.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:32 smithi039 ceph-mon[31721]: mds.? [v2:172.21.15.39:6838/656513813,v1:172.21.15.39:6839/656513813] up:boot 2024-03-18T16:06:32.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:32 smithi039 ceph-mon[31721]: fsmap 10 up:standby 2024-03-18T16:06:32.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:32 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-18T16:06:32.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:32 smithi039 ceph-mon[31721]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:06:32.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:32 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/3633550894' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:06:34.222 INFO:journalctl@ceph.mds.a.smithi039.stdout:Mar 18 16:06:33 smithi039 systemd[1]: Starting Ceph mds.a for 1892d8fc-e540-11ee-95c9-87774f69a715... 2024-03-18T16:06:34.243 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:06:34.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:34.473+0000 7f0e26800640 1 -- 172.21.15.39:0/3137166070 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0e20108e10 msgr2=0x7f0e20109290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:34.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:34.473+0000 7f0e26800640 1 --2- 172.21.15.39:0/3137166070 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0e20108e10 0x7f0e20109290 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f0e0c009860 tx=0x7f0e0c02bb70 comp rx=0 tx=0).stop 2024-03-18T16:06:34.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:34.473+0000 7f0e26800640 1 -- 172.21.15.39:0/3137166070 shutdown_connections 2024-03-18T16:06:34.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:34.473+0000 7f0e26800640 1 --2- 172.21.15.39:0/3137166070 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0e20108e10 0x7f0e20109290 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:34.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:34.473+0000 7f0e26800640 1 --2- 172.21.15.39:0/3137166070 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0e201097d0 0x7f0e20110060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:34.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:34.473+0000 7f0e26800640 1 --2- 172.21.15.39:0/3137166070 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0e20107c20 0x7f0e20108040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:34.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:34.473+0000 7f0e26800640 1 -- 172.21.15.39:0/3137166070 >> 172.21.15.39:0/3137166070 conn(0x7f0e20076bb0 msgr2=0x7f0e20078ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:34.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:34.473+0000 7f0e26800640 1 -- 172.21.15.39:0/3137166070 shutdown_connections 2024-03-18T16:06:34.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:34.473+0000 7f0e26800640 1 -- 172.21.15.39:0/3137166070 wait complete. 2024-03-18T16:06:34.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:34.474+0000 7f0e26800640 1 Processor -- start 2024-03-18T16:06:34.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:34.474+0000 7f0e26800640 1 -- start start 2024-03-18T16:06:34.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:34.474+0000 7f0e26800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0e20107c20 0x7f0e2019def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:34.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:34.475+0000 7f0e26800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0e20108e10 0x7f0e2019e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:34.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:34.475+0000 7f0e26800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0e201097d0 0x7f0e201a54a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:34.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:34.475+0000 7f0e26800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f0e20113880 con 0x7f0e20107c20 2024-03-18T16:06:34.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:34.475+0000 7f0e26800640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f0e20113700 con 0x7f0e20108e10 2024-03-18T16:06:34.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:34.475+0000 7f0e26800640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f0e20113a00 con 0x7f0e201097d0 2024-03-18T16:06:34.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:34.475+0000 7f0e25400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0e20107c20 0x7f0e2019def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:34.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:34.475+0000 7f0e25400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0e20107c20 0x7f0e2019def0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43250/0 (socket says 172.21.15.39:43250) 2024-03-18T16:06:34.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:34.475+0000 7f0e25400640 1 -- 172.21.15.39:0/1744699270 learned_addr learned my addr 172.21.15.39:0/1744699270 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:06:34.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:34.475+0000 7f0e24a00640 1 --2- 172.21.15.39:0/1744699270 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0e20108e10 0x7f0e2019e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:34.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:34.475+0000 7f0e25e00640 1 --2- 172.21.15.39:0/1744699270 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0e201097d0 0x7f0e201a54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:34.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:34.475+0000 7f0e25400640 1 -- 172.21.15.39:0/1744699270 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0e201097d0 msgr2=0x7f0e201a54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:34.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:34.475+0000 7f0e25400640 1 --2- 172.21.15.39:0/1744699270 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0e201097d0 0x7f0e201a54a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:34.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:34.475+0000 7f0e25400640 1 -- 172.21.15.39:0/1744699270 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0e20108e10 msgr2=0x7f0e2019e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:34.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:34.475+0000 7f0e25400640 1 --2- 172.21.15.39:0/1744699270 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0e20108e10 0x7f0e2019e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:34.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:34.475+0000 7f0e25400640 1 -- 172.21.15.39:0/1744699270 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e201a5b10 con 0x7f0e20107c20 2024-03-18T16:06:34.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:34.475+0000 7f0e24a00640 1 --2- 172.21.15.39:0/1744699270 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0e20108e10 0x7f0e2019e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:06:34.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:34.476+0000 7f0e25400640 1 --2- 172.21.15.39:0/1744699270 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0e20107c20 0x7f0e2019def0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f0e1400e9a0 tx=0x7f0e1400ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:34.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:34.476+0000 7f0e1e000640 1 -- 172.21.15.39:0/1744699270 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e1400cd80 con 0x7f0e20107c20 2024-03-18T16:06:34.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:34.476+0000 7f0e1e000640 1 -- 172.21.15.39:0/1744699270 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0e14010910 con 0x7f0e20107c20 2024-03-18T16:06:34.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:34.476+0000 7f0e26800640 1 -- 172.21.15.39:0/1744699270 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e201a5e00 con 0x7f0e20107c20 2024-03-18T16:06:34.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:34.476+0000 7f0e1e000640 1 -- 172.21.15.39:0/1744699270 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e14018a30 con 0x7f0e20107c20 2024-03-18T16:06:34.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:34.476+0000 7f0e26800640 1 -- 172.21.15.39:0/1744699270 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e201a6270 con 0x7f0e20107c20 2024-03-18T16:06:34.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:34.478+0000 7f0e26800640 1 -- 172.21.15.39:0/1744699270 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e2010ef90 con 0x7f0e20107c20 2024-03-18T16:06:34.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:34.480+0000 7f0e1e000640 1 -- 172.21.15.39:0/1744699270 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f0e14010ac0 con 0x7f0e20107c20 2024-03-18T16:06:34.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:34.482+0000 7f0e1e000640 1 --2- 172.21.15.39:0/1744699270 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f0e080a2880 0x7f0e080a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:34.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:34.483+0000 7f0e1e000640 1 -- 172.21.15.39:0/1744699270 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f0e140ceff0 con 0x7f0e20107c20 2024-03-18T16:06:34.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:34.483+0000 7f0e24a00640 1 --2- 172.21.15.39:0/1744699270 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f0e080a2880 0x7f0e080a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:34.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:34.484+0000 7f0e24a00640 1 --2- 172.21.15.39:0/1744699270 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f0e080a2880 0x7f0e080a4d40 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f0e0c02c180 tx=0x7f0e0c02e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:34.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:34.485+0000 7f0e1e000640 1 -- 172.21.15.39:0/1744699270 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0e14026070 con 0x7f0e20107c20 2024-03-18T16:06:34.538 INFO:journalctl@ceph.mds.a.smithi039.stdout:Mar 18 16:06:34 smithi039 podman[71252]: 2024-03-18 16:06:34.125076853 +0000 UTC m=+0.036292196 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T16:06:34.538 INFO:journalctl@ceph.mds.a.smithi039.stdout:Mar 18 16:06:34 smithi039 podman[71252]: 2024-03-18 16:06:34.256979805 +0000 UTC m=+0.168195188 container create 26e6fb6d37c490c0f3548d10178c69baef4828e6f3c7fde4e02aa5e1f466ba39 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-a, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0) 2024-03-18T16:06:34.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:34.734+0000 7f0e26800640 1 -- 172.21.15.39:0/1744699270 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0e2002d380 con 0x7f0e20107c20 2024-03-18T16:06:34.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:34.734+0000 7f0e1e000640 1 -- 172.21.15.39:0/1744699270 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8827 (secure 0 0 0) 0x7f0e1409c4b0 con 0x7f0e20107c20 2024-03-18T16:06:34.736 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:06:34.736 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2520226608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2520226608},{"type":"v1","addr":"172.21.15.39:6835","nonce":2520226608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/719341763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":719341763},{"type":"v1","addr":"172.21.15.39:6837","nonce":719341763}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14445,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6839/656513813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":656513813},{"type":"v1","addr":"172.21.15.39:6839","nonce":656513813}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/931350738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":931350738},{"type":"v1","addr":"172.21.15.194:6833","nonce":931350738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24463,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6833/3874209621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":3874209621},{"type":"v1","addr":"172.21.15.196:6833","nonce":3874209621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24494,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6835/188385470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":188385470},{"type":"v1","addr":"172.21.15.194:6835","nonce":188385470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24523,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6835/3949984309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":3949984309},{"type":"v1","addr":"172.21.15.196:6835","nonce":3949984309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6837/1301464246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":1301464246},{"type":"v1","addr":"172.21.15.194:6837","nonce":1301464246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24535,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6837/895247251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":895247251},{"type":"v1","addr":"172.21.15.196:6837","nonce":895247251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24556,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6839/2001427268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6838","nonce":2001427268},{"type":"v1","addr":"172.21.15.196:6839","nonce":2001427268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-03-18T16:06:34.736 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 11 2024-03-18T16:06:34.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:34.737+0000 7f0e26800640 1 -- 172.21.15.39:0/1744699270 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f0e080a2880 msgr2=0x7f0e080a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:34.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:34.737+0000 7f0e26800640 1 --2- 172.21.15.39:0/1744699270 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f0e080a2880 0x7f0e080a4d40 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f0e0c02c180 tx=0x7f0e0c02e040 comp rx=0 tx=0).stop 2024-03-18T16:06:34.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:34.737+0000 7f0e26800640 1 -- 172.21.15.39:0/1744699270 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0e20107c20 msgr2=0x7f0e2019def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:34.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:34.737+0000 7f0e26800640 1 --2- 172.21.15.39:0/1744699270 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0e20107c20 0x7f0e2019def0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f0e1400e9a0 tx=0x7f0e1400ee60 comp rx=0 tx=0).stop 2024-03-18T16:06:34.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:34.737+0000 7f0e26800640 1 -- 172.21.15.39:0/1744699270 shutdown_connections 2024-03-18T16:06:34.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:34.737+0000 7f0e26800640 1 --2- 172.21.15.39:0/1744699270 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0e20108e10 0x7f0e2019e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:34.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:34.737+0000 7f0e26800640 1 --2- 172.21.15.39:0/1744699270 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f0e080a2880 0x7f0e080a4d40 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:34.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:34.737+0000 7f0e26800640 1 --2- 172.21.15.39:0/1744699270 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0e201097d0 0x7f0e201a54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:34.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:34.737+0000 7f0e26800640 1 --2- 172.21.15.39:0/1744699270 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0e20107c20 0x7f0e2019def0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:34.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:34.737+0000 7f0e26800640 1 -- 172.21.15.39:0/1744699270 >> 172.21.15.39:0/1744699270 conn(0x7f0e20076bb0 msgr2=0x7f0e2010c160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:34.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:34.738+0000 7f0e26800640 1 -- 172.21.15.39:0/1744699270 shutdown_connections 2024-03-18T16:06:34.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:34.738+0000 7f0e26800640 1 -- 172.21.15.39:0/1744699270 wait complete. 2024-03-18T16:06:34.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:34 smithi039 ceph-mon[31721]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:06:34.837 INFO:journalctl@ceph.mds.a.smithi039.stdout:Mar 18 16:06:34 smithi039 podman[71252]: 2024-03-18 16:06:34.578161181 +0000 UTC m=+0.489376567 container init 26e6fb6d37c490c0f3548d10178c69baef4828e6f3c7fde4e02aa5e1f466ba39 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-18T16:06:34.837 INFO:journalctl@ceph.mds.a.smithi039.stdout:Mar 18 16:06:34 smithi039 podman[71252]: 2024-03-18 16:06:34.587635448 +0000 UTC m=+0.498850813 container start 26e6fb6d37c490c0f3548d10178c69baef4828e6f3c7fde4e02aa5e1f466ba39 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-a, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T16:06:34.837 INFO:journalctl@ceph.mds.a.smithi039.stdout:Mar 18 16:06:34 smithi039 ceph-mds[71310]: -- 172.21.15.39:0/1479684222 <== mon.2 v2:172.21.15.194:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5635dfca5a40 con 0x5635dfc78800 2024-03-18T16:06:34.837 INFO:journalctl@ceph.mds.a.smithi039.stdout:Mar 18 16:06:34 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-a[71287]: starting mds.a at 2024-03-18T16:06:34.837 INFO:journalctl@ceph.mds.a.smithi039.stdout:Mar 18 16:06:34 smithi039 bash[71252]: 26e6fb6d37c490c0f3548d10178c69baef4828e6f3c7fde4e02aa5e1f466ba39 2024-03-18T16:06:34.837 INFO:journalctl@ceph.mds.a.smithi039.stdout:Mar 18 16:06:34 smithi039 systemd[1]: Started Ceph mds.a for 1892d8fc-e540-11ee-95c9-87774f69a715. 2024-03-18T16:06:34.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:34 smithi196 ceph-mon[36307]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:06:34.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:34 smithi194 ceph-mon[36200]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:06:35.764 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:35 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/1744699270' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:06:35.765 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:35 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:35.765 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:35 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:35.765 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:35 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:35.765 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:35 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T16:06:35.765 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:35 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T16:06:35.765 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:35 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:06:35.765 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:35 smithi194 ceph-mon[36200]: Deploying daemon mds.b on smithi194 2024-03-18T16:06:35.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:35 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/1744699270' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:06:35.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:35 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:35.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:35 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:35.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:35 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:35.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:35 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T16:06:35.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:35 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T16:06:35.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:35 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:06:35.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:35 smithi039 ceph-mon[31721]: Deploying daemon mds.b on smithi194 2024-03-18T16:06:35.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:35 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/1744699270' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:06:35.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:35 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:35.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:35 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:35.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:35 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:35.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:35 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-18T16:06:35.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:35 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-18T16:06:35.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:35 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:06:35.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:35 smithi196 ceph-mon[36307]: Deploying daemon mds.b on smithi194 2024-03-18T16:06:36.758 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:06:36.792 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:36 smithi194 ceph-mon[36200]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:06:36.793 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:36 smithi194 ceph-mon[36200]: mds.? [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] up:boot 2024-03-18T16:06:36.793 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:36 smithi194 ceph-mon[36200]: fsmap 11 up:standby 2024-03-18T16:06:36.793 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:36 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-18T16:06:36.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:36 smithi039 ceph-mon[31721]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:06:36.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:36 smithi039 ceph-mon[31721]: mds.? [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] up:boot 2024-03-18T16:06:36.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:36 smithi039 ceph-mon[31721]: fsmap 11 up:standby 2024-03-18T16:06:36.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:36 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-18T16:06:36.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:36 smithi196 ceph-mon[36307]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:06:36.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:36 smithi196 ceph-mon[36307]: mds.? [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] up:boot 2024-03-18T16:06:36.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:36 smithi196 ceph-mon[36307]: fsmap 11 up:standby 2024-03-18T16:06:36.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:36 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-18T16:06:36.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:36.992+0000 7f5987e00640 1 -- 172.21.15.39:0/1221696207 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f59801099b0 msgr2=0x7f5980110240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:36.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:36.992+0000 7f5987e00640 1 --2- 172.21.15.39:0/1221696207 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f59801099b0 0x7f5980110240 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5978009ca0 tx=0x7f597802c0f0 comp rx=0 tx=0).stop 2024-03-18T16:06:36.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:36.992+0000 7f5987e00640 1 -- 172.21.15.39:0/1221696207 shutdown_connections 2024-03-18T16:06:36.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:36.992+0000 7f5987e00640 1 --2- 172.21.15.39:0/1221696207 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f59801099b0 0x7f5980110240 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:36.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:36.992+0000 7f5987e00640 1 --2- 172.21.15.39:0/1221696207 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5980108ff0 0x7f5980109470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:36.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:36.992+0000 7f5987e00640 1 --2- 172.21.15.39:0/1221696207 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5980107e00 0x7f5980108220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:36.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:36.992+0000 7f5987e00640 1 -- 172.21.15.39:0/1221696207 >> 172.21.15.39:0/1221696207 conn(0x7f5980076dc0 msgr2=0x7f59800791e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:36.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:36.992+0000 7f5987e00640 1 -- 172.21.15.39:0/1221696207 shutdown_connections 2024-03-18T16:06:36.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:36.992+0000 7f5987e00640 1 -- 172.21.15.39:0/1221696207 wait complete. 2024-03-18T16:06:36.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:36.993+0000 7f5987e00640 1 Processor -- start 2024-03-18T16:06:36.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:36.993+0000 7f5987e00640 1 -- start start 2024-03-18T16:06:36.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:36.994+0000 7f5987e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5980107e00 0x7f598019e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:36.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:36.994+0000 7f5987e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5980108ff0 0x7f598019e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:36.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:36.994+0000 7f5987e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f59801099b0 0x7f59801a55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:36.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:36.994+0000 7f5987e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f5980113a60 con 0x7f5980107e00 2024-03-18T16:06:36.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:36.994+0000 7f5987e00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f59801138e0 con 0x7f59801099b0 2024-03-18T16:06:36.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:36.994+0000 7f5987e00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f5980113be0 con 0x7f5980108ff0 2024-03-18T16:06:36.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:36.994+0000 7f5985400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5980107e00 0x7f598019e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:36.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:36.994+0000 7f5984a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5980108ff0 0x7f598019e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:36.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:36.994+0000 7f5985400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5980107e00 0x7f598019e040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43282/0 (socket says 172.21.15.39:43282) 2024-03-18T16:06:36.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:36.994+0000 7f5985400640 1 -- 172.21.15.39:0/2918601414 learned_addr learned my addr 172.21.15.39:0/2918601414 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:06:36.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:36.994+0000 7f5985e00640 1 --2- 172.21.15.39:0/2918601414 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f59801099b0 0x7f59801a55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:36.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:36.994+0000 7f5985400640 1 -- 172.21.15.39:0/2918601414 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5980108ff0 msgr2=0x7f598019e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:36.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:36.994+0000 7f5985400640 1 --2- 172.21.15.39:0/2918601414 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5980108ff0 0x7f598019e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:36.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:36.994+0000 7f5985400640 1 -- 172.21.15.39:0/2918601414 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f59801099b0 msgr2=0x7f59801a55f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:36.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:36.994+0000 7f5985400640 1 --2- 172.21.15.39:0/2918601414 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f59801099b0 0x7f59801a55f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:36.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:36.994+0000 7f5985400640 1 -- 172.21.15.39:0/2918601414 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59801a5c60 con 0x7f5980107e00 2024-03-18T16:06:36.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:36.995+0000 7f5985400640 1 --2- 172.21.15.39:0/2918601414 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5980107e00 0x7f598019e040 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f597400e9c0 tx=0x7f597400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:36.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:36.995+0000 7f597e000640 1 -- 172.21.15.39:0/2918601414 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f597400cd80 con 0x7f5980107e00 2024-03-18T16:06:36.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:36.995+0000 7f597e000640 1 -- 172.21.15.39:0/2918601414 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f59740035f0 con 0x7f5980107e00 2024-03-18T16:06:36.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:36.995+0000 7f5987e00640 1 -- 172.21.15.39:0/2918601414 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59801a5f50 con 0x7f5980107e00 2024-03-18T16:06:36.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:36.995+0000 7f597e000640 1 -- 172.21.15.39:0/2918601414 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5974010690 con 0x7f5980107e00 2024-03-18T16:06:36.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:36.995+0000 7f5987e00640 1 -- 172.21.15.39:0/2918601414 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59801a63b0 con 0x7f5980107e00 2024-03-18T16:06:36.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:36.996+0000 7f5987e00640 1 -- 172.21.15.39:0/2918601414 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f598010f0c0 con 0x7f5980107e00 2024-03-18T16:06:37.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:37.000+0000 7f597e000640 1 -- 172.21.15.39:0/2918601414 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f59740037a0 con 0x7f5980107e00 2024-03-18T16:06:37.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:37.001+0000 7f597e000640 1 --2- 172.21.15.39:0/2918601414 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f59680a2800 0x7f59680a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:37.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:37.001+0000 7f597e000640 1 -- 172.21.15.39:0/2918601414 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f59740ceaf0 con 0x7f5980107e00 2024-03-18T16:06:37.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:37.002+0000 7f5984a00640 1 --2- 172.21.15.39:0/2918601414 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f59680a2800 0x7f59680a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:37.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:37.002+0000 7f5984a00640 1 --2- 172.21.15.39:0/2918601414 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f59680a2800 0x7f59680a4cc0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f598019f550 tx=0x7f596c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:37.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:37.004+0000 7f597e000640 1 -- 172.21.15.39:0/2918601414 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f597409c020 con 0x7f5980107e00 2024-03-18T16:06:37.187 INFO:journalctl@ceph.mds.b.smithi194.stdout:Mar 18 16:06:36 smithi194 systemd[1]: Starting Ceph mds.b for 1892d8fc-e540-11ee-95c9-87774f69a715... 2024-03-18T16:06:37.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:37.248+0000 7f5987e00640 1 -- 172.21.15.39:0/2918601414 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f598002d1b0 con 0x7f5980107e00 2024-03-18T16:06:37.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:37.250+0000 7f597e000640 1 -- 172.21.15.39:0/2918601414 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9653 (secure 0 0 0) 0x7f597409b920 con 0x7f5980107e00 2024-03-18T16:06:37.251 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:06:37.252 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2520226608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2520226608},{"type":"v1","addr":"172.21.15.39:6835","nonce":2520226608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/719341763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":719341763},{"type":"v1","addr":"172.21.15.39:6837","nonce":719341763}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14445,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6839/656513813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":656513813},{"type":"v1","addr":"172.21.15.39:6839","nonce":656513813}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14463,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6841/3515326042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3515326042},{"type":"v1","addr":"172.21.15.39:6841","nonce":3515326042}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/931350738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":931350738},{"type":"v1","addr":"172.21.15.194:6833","nonce":931350738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24463,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6833/3874209621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":3874209621},{"type":"v1","addr":"172.21.15.196:6833","nonce":3874209621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24494,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6835/188385470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":188385470},{"type":"v1","addr":"172.21.15.194:6835","nonce":188385470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24523,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6835/3949984309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":3949984309},{"type":"v1","addr":"172.21.15.196:6835","nonce":3949984309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6837/1301464246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":1301464246},{"type":"v1","addr":"172.21.15.194:6837","nonce":1301464246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24535,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6837/895247251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":895247251},{"type":"v1","addr":"172.21.15.196:6837","nonce":895247251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24556,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6839/2001427268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6838","nonce":2001427268},{"type":"v1","addr":"172.21.15.196:6839","nonce":2001427268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-03-18T16:06:37.252 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 12 2024-03-18T16:06:37.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:37.253+0000 7f5987e00640 1 -- 172.21.15.39:0/2918601414 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f59680a2800 msgr2=0x7f59680a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:37.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:37.253+0000 7f5987e00640 1 --2- 172.21.15.39:0/2918601414 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f59680a2800 0x7f59680a4cc0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f598019f550 tx=0x7f596c00b410 comp rx=0 tx=0).stop 2024-03-18T16:06:37.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:37.253+0000 7f5987e00640 1 -- 172.21.15.39:0/2918601414 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5980107e00 msgr2=0x7f598019e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:37.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:37.253+0000 7f5987e00640 1 --2- 172.21.15.39:0/2918601414 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5980107e00 0x7f598019e040 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f597400e9c0 tx=0x7f597400ee80 comp rx=0 tx=0).stop 2024-03-18T16:06:37.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:37.253+0000 7f5987e00640 1 -- 172.21.15.39:0/2918601414 shutdown_connections 2024-03-18T16:06:37.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:37.253+0000 7f5987e00640 1 --2- 172.21.15.39:0/2918601414 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f59801099b0 0x7f59801a55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:37.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:37.253+0000 7f5987e00640 1 --2- 172.21.15.39:0/2918601414 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f59680a2800 0x7f59680a4cc0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:37.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:37.253+0000 7f5987e00640 1 --2- 172.21.15.39:0/2918601414 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5980108ff0 0x7f598019e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:37.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:37.253+0000 7f5987e00640 1 --2- 172.21.15.39:0/2918601414 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5980107e00 0x7f598019e040 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:37.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:37.253+0000 7f5987e00640 1 -- 172.21.15.39:0/2918601414 >> 172.21.15.39:0/2918601414 conn(0x7f5980076dc0 msgr2=0x7f598010c290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:37.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:37.253+0000 7f5987e00640 1 -- 172.21.15.39:0/2918601414 shutdown_connections 2024-03-18T16:06:37.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:37.253+0000 7f5987e00640 1 -- 172.21.15.39:0/2918601414 wait complete. 2024-03-18T16:06:37.490 INFO:journalctl@ceph.mds.b.smithi194.stdout:Mar 18 16:06:37 smithi194 podman[67272]: 2024-03-18 16:06:37.186615832 +0000 UTC m=+0.110525696 container create 13068c0a3b14b351dd6554019e42de5779fda2bec182626b7bb1ce9e2c71edf4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-b, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-18T16:06:37.491 INFO:journalctl@ceph.mds.b.smithi194.stdout:Mar 18 16:06:37 smithi194 podman[67272]: 2024-03-18 16:06:37.093858922 +0000 UTC m=+0.017768799 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T16:06:37.491 INFO:journalctl@ceph.mds.b.smithi194.stdout:Mar 18 16:06:37 smithi194 podman[67272]: 2024-03-18 16:06:37.397379054 +0000 UTC m=+0.321288942 container init 13068c0a3b14b351dd6554019e42de5779fda2bec182626b7bb1ce9e2c71edf4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240307) 2024-03-18T16:06:37.491 INFO:journalctl@ceph.mds.b.smithi194.stdout:Mar 18 16:06:37 smithi194 podman[67272]: 2024-03-18 16:06:37.400030952 +0000 UTC m=+0.323940836 container start 13068c0a3b14b351dd6554019e42de5779fda2bec182626b7bb1ce9e2c71edf4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-b, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-18T16:06:37.491 INFO:journalctl@ceph.mds.b.smithi194.stdout:Mar 18 16:06:37 smithi194 ceph-mds[67291]: -- 172.21.15.194:0/2036001462 <== mon.2 v2:172.21.15.194:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e748585a40 con 0x55e748558c00 2024-03-18T16:06:37.491 INFO:journalctl@ceph.mds.b.smithi194.stdout:Mar 18 16:06:37 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-b[67287]: starting mds.b at 2024-03-18T16:06:37.491 INFO:journalctl@ceph.mds.b.smithi194.stdout:Mar 18 16:06:37 smithi194 bash[67272]: 13068c0a3b14b351dd6554019e42de5779fda2bec182626b7bb1ce9e2c71edf4 2024-03-18T16:06:37.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:37 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/2918601414' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:06:37.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:37 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:37.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:37 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/2918601414' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:06:37.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:37 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:37.990 INFO:journalctl@ceph.mds.b.smithi194.stdout:Mar 18 16:06:37 smithi194 systemd[1]: Started Ceph mds.b for 1892d8fc-e540-11ee-95c9-87774f69a715. 2024-03-18T16:06:37.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:37 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/2918601414' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:06:37.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:37 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:38.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:38 smithi039 ceph-mon[31721]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:06:38.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:38 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:38.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:38 smithi039 ceph-mon[31721]: mds.? [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] up:boot 2024-03-18T16:06:38.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:38 smithi039 ceph-mon[31721]: fsmap 12 up:standby 2024-03-18T16:06:38.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:38 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-18T16:06:38.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:38 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:38.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:38 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:38.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:38 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:38.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:38 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:06:38.890 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:38 smithi194 ceph-mon[36200]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:06:38.890 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:38 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:38.890 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:38 smithi194 ceph-mon[36200]: mds.? [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] up:boot 2024-03-18T16:06:38.890 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:38 smithi194 ceph-mon[36200]: fsmap 12 up:standby 2024-03-18T16:06:38.890 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:38 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-18T16:06:38.890 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:38 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:38.890 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:38 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:38.890 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:38 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:38.890 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:38 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:06:38.900 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:38 smithi196 ceph-mon[36307]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:06:38.900 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:38 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:38.900 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:38 smithi196 ceph-mon[36307]: mds.? [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] up:boot 2024-03-18T16:06:38.900 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:38 smithi196 ceph-mon[36307]: fsmap 12 up:standby 2024-03-18T16:06:38.900 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:38 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-18T16:06:38.900 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:38 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:38.900 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:38 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:38.900 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:38 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:38.900 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:38 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:06:39.276 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:06:39.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.480+0000 7f74a0200640 1 -- 172.21.15.39:0/1264422729 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7498100a20 msgr2=0x7f7498100e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:39.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.480+0000 7f74a0200640 1 --2- 172.21.15.39:0/1264422729 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7498100a20 0x7f7498100e80 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f7484009860 tx=0x7f748402bc00 comp rx=0 tx=0).stop 2024-03-18T16:06:39.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.480+0000 7f74a0200640 1 -- 172.21.15.39:0/1264422729 shutdown_connections 2024-03-18T16:06:39.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.480+0000 7f74a0200640 1 --2- 172.21.15.39:0/1264422729 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f74981013c0 0x7f7498101840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:39.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.480+0000 7f74a0200640 1 --2- 172.21.15.39:0/1264422729 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7498100a20 0x7f7498100e80 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:39.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.480+0000 7f74a0200640 1 --2- 172.21.15.39:0/1264422729 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f74981000c0 0x7f74981004e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:39.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.480+0000 7f74a0200640 1 -- 172.21.15.39:0/1264422729 >> 172.21.15.39:0/1264422729 conn(0x7f74980fbc40 msgr2=0x7f74980fe080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:39.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.480+0000 7f74a0200640 1 -- 172.21.15.39:0/1264422729 shutdown_connections 2024-03-18T16:06:39.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.480+0000 7f74a0200640 1 -- 172.21.15.39:0/1264422729 wait complete. 2024-03-18T16:06:39.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.481+0000 7f74a0200640 1 Processor -- start 2024-03-18T16:06:39.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.481+0000 7f74a0200640 1 -- start start 2024-03-18T16:06:39.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.482+0000 7f74a0200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f74981000c0 0x7f7498199d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:39.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.482+0000 7f74a0200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7498100a20 0x7f749819a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:39.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.482+0000 7f74a0200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f74981013c0 0x7f74981a1270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:39.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.482+0000 7f74a0200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f749810f630 con 0x7f74981000c0 2024-03-18T16:06:39.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.482+0000 7f74a0200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f749810f4b0 con 0x7f7498100a20 2024-03-18T16:06:39.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.482+0000 7f74a0200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f749810f7b0 con 0x7f74981013c0 2024-03-18T16:06:39.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.482+0000 7f749d800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f74981000c0 0x7f7498199d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:39.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.482+0000 7f749d800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f74981000c0 0x7f7498199d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:60306/0 (socket says 172.21.15.39:60306) 2024-03-18T16:06:39.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.482+0000 7f749d800640 1 -- 172.21.15.39:0/2106069353 learned_addr learned my addr 172.21.15.39:0/2106069353 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:06:39.487 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.482+0000 7f749ce00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7498100a20 0x7f749819a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:39.487 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.482+0000 7f749e200640 1 --2- 172.21.15.39:0/2106069353 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f74981013c0 0x7f74981a1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:39.487 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.482+0000 7f749d800640 1 -- 172.21.15.39:0/2106069353 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f74981013c0 msgr2=0x7f74981a1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:39.487 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.482+0000 7f749d800640 1 --2- 172.21.15.39:0/2106069353 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f74981013c0 0x7f74981a1270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:39.487 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.482+0000 7f749d800640 1 -- 172.21.15.39:0/2106069353 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7498100a20 msgr2=0x7f749819a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:39.487 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.482+0000 7f749d800640 1 --2- 172.21.15.39:0/2106069353 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7498100a20 0x7f749819a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:39.487 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.482+0000 7f749d800640 1 -- 172.21.15.39:0/2106069353 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74981a1910 con 0x7f74981000c0 2024-03-18T16:06:39.487 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.482+0000 7f749ce00640 1 --2- 172.21.15.39:0/2106069353 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7498100a20 0x7f749819a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:06:39.487 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.483+0000 7f749d800640 1 --2- 172.21.15.39:0/2106069353 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f74981000c0 0x7f7498199d00 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f748c00e9c0 tx=0x7f748c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:39.487 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.483+0000 7f7496000640 1 -- 172.21.15.39:0/2106069353 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f748c00ccc0 con 0x7f74981000c0 2024-03-18T16:06:39.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.483+0000 7f7496000640 1 -- 172.21.15.39:0/2106069353 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f748c005500 con 0x7f74981000c0 2024-03-18T16:06:39.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.483+0000 7f74a0200640 1 -- 172.21.15.39:0/2106069353 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74981a1c00 con 0x7f74981000c0 2024-03-18T16:06:39.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.483+0000 7f7496000640 1 -- 172.21.15.39:0/2106069353 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f748c010430 con 0x7f74981000c0 2024-03-18T16:06:39.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.483+0000 7f74a0200640 1 -- 172.21.15.39:0/2106069353 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74981a2150 con 0x7f74981000c0 2024-03-18T16:06:39.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.484+0000 7f74a0200640 1 -- 172.21.15.39:0/2106069353 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f749810af90 con 0x7f74981000c0 2024-03-18T16:06:39.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.486+0000 7f7496000640 1 -- 172.21.15.39:0/2106069353 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f748c0046c0 con 0x7f74981000c0 2024-03-18T16:06:39.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.487+0000 7f7496000640 1 --2- 172.21.15.39:0/2106069353 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f74800a2800 0x7f74800a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:39.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.488+0000 7f7496000640 1 -- 172.21.15.39:0/2106069353 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f748c0cea00 con 0x7f74981000c0 2024-03-18T16:06:39.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.488+0000 7f749ce00640 1 --2- 172.21.15.39:0/2106069353 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f74800a2800 0x7f74800a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:39.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.489+0000 7f749ce00640 1 --2- 172.21.15.39:0/2106069353 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f74800a2800 0x7f74800a4cc0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f7484009830 tx=0x7f7484034b50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:39.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.492+0000 7f7496000640 1 -- 172.21.15.39:0/2106069353 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f748c09bea0 con 0x7f74981000c0 2024-03-18T16:06:39.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.760+0000 7f74a0200640 1 -- 172.21.15.39:0/2106069353 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f749802d1b0 con 0x7f74981000c0 2024-03-18T16:06:39.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.761+0000 7f7496000640 1 -- 172.21.15.39:0/2106069353 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10479 (secure 0 0 0) 0x7f748c09b7a0 con 0x7f74981000c0 2024-03-18T16:06:39.763 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:06:39.763 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2520226608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2520226608},{"type":"v1","addr":"172.21.15.39:6835","nonce":2520226608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/719341763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":719341763},{"type":"v1","addr":"172.21.15.39:6837","nonce":719341763}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14445,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6839/656513813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":656513813},{"type":"v1","addr":"172.21.15.39:6839","nonce":656513813}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14463,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6841/3515326042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3515326042},{"type":"v1","addr":"172.21.15.39:6841","nonce":3515326042}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/931350738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":931350738},{"type":"v1","addr":"172.21.15.194:6833","nonce":931350738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24463,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6833/3874209621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":3874209621},{"type":"v1","addr":"172.21.15.196:6833","nonce":3874209621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24494,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6835/188385470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":188385470},{"type":"v1","addr":"172.21.15.194:6835","nonce":188385470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24523,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6835/3949984309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":3949984309},{"type":"v1","addr":"172.21.15.196:6835","nonce":3949984309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6837/1301464246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":1301464246},{"type":"v1","addr":"172.21.15.194:6837","nonce":1301464246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24535,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6837/895247251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":895247251},{"type":"v1","addr":"172.21.15.196:6837","nonce":895247251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24556,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6839/2001427268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6838","nonce":2001427268},{"type":"v1","addr":"172.21.15.196:6839","nonce":2001427268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24575,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6839/274911158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":274911158},{"type":"v1","addr":"172.21.15.194:6839","nonce":274911158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-03-18T16:06:39.763 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 13 2024-03-18T16:06:39.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.764+0000 7f74a0200640 1 -- 172.21.15.39:0/2106069353 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f74800a2800 msgr2=0x7f74800a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:39.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.764+0000 7f74a0200640 1 --2- 172.21.15.39:0/2106069353 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f74800a2800 0x7f74800a4cc0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f7484009830 tx=0x7f7484034b50 comp rx=0 tx=0).stop 2024-03-18T16:06:39.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.764+0000 7f74a0200640 1 -- 172.21.15.39:0/2106069353 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f74981000c0 msgr2=0x7f7498199d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:39.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.764+0000 7f74a0200640 1 --2- 172.21.15.39:0/2106069353 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f74981000c0 0x7f7498199d00 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f748c00e9c0 tx=0x7f748c00ee80 comp rx=0 tx=0).stop 2024-03-18T16:06:39.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.764+0000 7f74a0200640 1 -- 172.21.15.39:0/2106069353 shutdown_connections 2024-03-18T16:06:39.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.764+0000 7f74a0200640 1 --2- 172.21.15.39:0/2106069353 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7498100a20 0x7f749819a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:39.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.764+0000 7f74a0200640 1 --2- 172.21.15.39:0/2106069353 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f74800a2800 0x7f74800a4cc0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:39.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.764+0000 7f74a0200640 1 --2- 172.21.15.39:0/2106069353 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f74981013c0 0x7f74981a1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:39.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.764+0000 7f74a0200640 1 --2- 172.21.15.39:0/2106069353 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f74981000c0 0x7f7498199d00 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:39.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.764+0000 7f74a0200640 1 -- 172.21.15.39:0/2106069353 >> 172.21.15.39:0/2106069353 conn(0x7f74980fbc40 msgr2=0x7f74980fd360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:39.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.764+0000 7f74a0200640 1 -- 172.21.15.39:0/2106069353 shutdown_connections 2024-03-18T16:06:39.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.764+0000 7f74a0200640 1 -- 172.21.15.39:0/2106069353 wait complete. 2024-03-18T16:06:39.788 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T16:06:39.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.981+0000 7fced0200640 1 -- 172.21.15.39:0/3780193820 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcec8103a00 msgr2=0x7fcec8103e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:39.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.981+0000 7fced0200640 1 --2- 172.21.15.39:0/3780193820 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcec8103a00 0x7fcec8103e20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fcebc009900 tx=0x7fcebc02bc30 comp rx=0 tx=0).stop 2024-03-18T16:06:39.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.981+0000 7fced0200640 1 -- 172.21.15.39:0/3780193820 shutdown_connections 2024-03-18T16:06:39.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.981+0000 7fced0200640 1 --2- 172.21.15.39:0/3780193820 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcec81055b0 0x7fcec810be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:39.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.981+0000 7fced0200640 1 --2- 172.21.15.39:0/3780193820 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcec8104bf0 0x7fcec8105070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:39.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.981+0000 7fced0200640 1 --2- 172.21.15.39:0/3780193820 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcec8103a00 0x7fcec8103e20 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:39.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.981+0000 7fced0200640 1 -- 172.21.15.39:0/3780193820 >> 172.21.15.39:0/3780193820 conn(0x7fcec80fef90 msgr2=0x7fcec81013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:39.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.981+0000 7fced0200640 1 -- 172.21.15.39:0/3780193820 shutdown_connections 2024-03-18T16:06:39.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.981+0000 7fced0200640 1 -- 172.21.15.39:0/3780193820 wait complete. 2024-03-18T16:06:39.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.982+0000 7fced0200640 1 Processor -- start 2024-03-18T16:06:39.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.982+0000 7fced0200640 1 -- start start 2024-03-18T16:06:39.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.983+0000 7fced0200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcec8103a00 0x7fcec8199d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:39.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.983+0000 7fced0200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcec8104bf0 0x7fcec819a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:39.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.983+0000 7fced0200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcec81055b0 0x7fcec81a1260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:39.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.983+0000 7fced0200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fcec810f6a0 con 0x7fcec8103a00 2024-03-18T16:06:39.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.983+0000 7fced0200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fcec810f520 con 0x7fcec8104bf0 2024-03-18T16:06:39.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.983+0000 7fced0200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fcec810f820 con 0x7fcec81055b0 2024-03-18T16:06:39.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.983+0000 7fcecd800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcec8103a00 0x7fcec8199d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:39.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.983+0000 7fcecd800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcec8103a00 0x7fcec8199d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:60324/0 (socket says 172.21.15.39:60324) 2024-03-18T16:06:39.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.983+0000 7fcecd800640 1 -- 172.21.15.39:0/2084665538 learned_addr learned my addr 172.21.15.39:0/2084665538 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:06:39.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.983+0000 7fcecce00640 1 --2- 172.21.15.39:0/2084665538 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcec8104bf0 0x7fcec819a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:39.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.983+0000 7fcece200640 1 --2- 172.21.15.39:0/2084665538 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcec81055b0 0x7fcec81a1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:39.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.984+0000 7fcecd800640 1 -- 172.21.15.39:0/2084665538 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcec81055b0 msgr2=0x7fcec81a1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:39.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.984+0000 7fcecd800640 1 --2- 172.21.15.39:0/2084665538 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcec81055b0 0x7fcec81a1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:39.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.984+0000 7fcecd800640 1 -- 172.21.15.39:0/2084665538 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcec8104bf0 msgr2=0x7fcec819a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:39.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.984+0000 7fcecd800640 1 --2- 172.21.15.39:0/2084665538 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcec8104bf0 0x7fcec819a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:39.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.984+0000 7fcecd800640 1 -- 172.21.15.39:0/2084665538 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcec81a1900 con 0x7fcec8103a00 2024-03-18T16:06:39.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.984+0000 7fcecce00640 1 --2- 172.21.15.39:0/2084665538 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcec8104bf0 0x7fcec819a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:06:39.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.984+0000 7fcecd800640 1 --2- 172.21.15.39:0/2084665538 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcec8103a00 0x7fcec8199d90 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fcebc000c00 tx=0x7fcebc02c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:39.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.984+0000 7fcec6000640 1 -- 172.21.15.39:0/2084665538 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcebc043020 con 0x7fcec8103a00 2024-03-18T16:06:39.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.984+0000 7fcec6000640 1 -- 172.21.15.39:0/2084665538 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcebc0054e0 con 0x7fcec8103a00 2024-03-18T16:06:39.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.984+0000 7fced0200640 1 -- 172.21.15.39:0/2084665538 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcec81a1b90 con 0x7fcec8103a00 2024-03-18T16:06:39.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.984+0000 7fcec6000640 1 -- 172.21.15.39:0/2084665538 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcebc003c40 con 0x7fcec8103a00 2024-03-18T16:06:39.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.985+0000 7fced0200640 1 -- 172.21.15.39:0/2084665538 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcec81a20e0 con 0x7fcec8103a00 2024-03-18T16:06:39.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.986+0000 7fced0200640 1 -- 172.21.15.39:0/2084665538 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcec8068a90 con 0x7fcec8103a00 2024-03-18T16:06:39.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.989+0000 7fcec6000640 1 -- 172.21.15.39:0/2084665538 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fcebc046020 con 0x7fcec8103a00 2024-03-18T16:06:39.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.991+0000 7fcec6000640 1 --2- 172.21.15.39:0/2084665538 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fceb00a2800 0x7fceb00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:39.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.991+0000 7fcec6000640 1 -- 172.21.15.39:0/2084665538 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fcebc0f0a00 con 0x7fcec8103a00 2024-03-18T16:06:39.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.991+0000 7fcecce00640 1 --2- 172.21.15.39:0/2084665538 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fceb00a2800 0x7fceb00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:39.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.992+0000 7fcecce00640 1 --2- 172.21.15.39:0/2084665538 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fceb00a2800 0x7fceb00a4cc0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fceb4009fc0 tx=0x7fceb4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:39.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:39.994+0000 7fcec6000640 1 -- 172.21.15.39:0/2084665538 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fcebc04d020 con 0x7fcec8103a00 2024-03-18T16:06:40.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.210+0000 7fced0200640 1 -- 172.21.15.39:0/2084665538 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fcec810ad20 con 0x7fcec8103a00 2024-03-18T16:06:40.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.214+0000 7fcec6000640 1 -- 172.21.15.39:0/2084665538 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18470 (secure 0 0 0) 0x7fcebc0bdf40 con 0x7fcec8103a00 2024-03-18T16:06:40.216 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:06:40.216 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":81,"fsid":"1892d8fc-e540-11ee-95c9-87774f69a715","created":"2024-03-18T15:57:36.507008+0000","modified":"2024-03-18T16:05:46.544944+0000","last_up_change":"2024-03-18T16:05:45.250629+0000","last_in_change":"2024-03-18T16:05:24.841337+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T16:01:30.048247+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"771e0558-be62-4436-94b0-6364751911c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6803","nonce":4090743635}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6805","nonce":4090743635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6809","nonce":4090743635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6807","nonce":4090743635}]},"public_addr":"172.21.15.39:6803/4090743635","cluster_addr":"172.21.15.39:6805/4090743635","heartbeat_back_addr":"172.21.15.39:6809/4090743635","heartbeat_front_addr":"172.21.15.39:6807/4090743635","state":["exists","up"]},{"osd":1,"uuid":"0430eee1-cf34-4277-b255-c7d9f07ad19a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6801","nonce":2800329805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6803","nonce":2800329805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6807","nonce":2800329805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6805","nonce":2800329805}]},"public_addr":"172.21.15.194:6801/2800329805","cluster_addr":"172.21.15.194:6803/2800329805","heartbeat_back_addr":"172.21.15.194:6807/2800329805","heartbeat_front_addr":"172.21.15.194:6805/2800329805","state":["exists","up"]},{"osd":2,"uuid":"3a718da4-2511-4ec1-b1a8-eafd6cce0df0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6801","nonce":3935874719}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6803","nonce":3935874719}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6807","nonce":3935874719}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6805","nonce":3935874719}]},"public_addr":"172.21.15.196:6801/3935874719","cluster_addr":"172.21.15.196:6803/3935874719","heartbeat_back_addr":"172.21.15.196:6807/3935874719","heartbeat_front_addr":"172.21.15.196:6805/3935874719","state":["exists","up"]},{"osd":3,"uuid":"6f9c37f9-0ab9-4038-8919-afd4b39c4165","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6811","nonce":893345939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6813","nonce":893345939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6817","nonce":893345939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6815","nonce":893345939}]},"public_addr":"172.21.15.39:6811/893345939","cluster_addr":"172.21.15.39:6813/893345939","heartbeat_back_addr":"172.21.15.39:6817/893345939","heartbeat_front_addr":"172.21.15.39:6815/893345939","state":["exists","up"]},{"osd":4,"uuid":"e795bb0e-4791-4537-acda-1a321e19bc9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6809","nonce":800423088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6811","nonce":800423088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6815","nonce":800423088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6813","nonce":800423088}]},"public_addr":"172.21.15.194:6809/800423088","cluster_addr":"172.21.15.194:6811/800423088","heartbeat_back_addr":"172.21.15.194:6815/800423088","heartbeat_front_addr":"172.21.15.194:6813/800423088","state":["exists","up"]},{"osd":5,"uuid":"026e175e-6acb-4b4c-b235-8ccee8ae1149","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6809","nonce":2746224525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6811","nonce":2746224525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6815","nonce":2746224525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6813","nonce":2746224525}]},"public_addr":"172.21.15.196:6809/2746224525","cluster_addr":"172.21.15.196:6811/2746224525","heartbeat_back_addr":"172.21.15.196:6815/2746224525","heartbeat_front_addr":"172.21.15.196:6813/2746224525","state":["exists","up"]},{"osd":6,"uuid":"c1b183ff-383f-4c5c-9075-f76bab1c0bb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6819","nonce":1213371331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6821","nonce":1213371331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6825","nonce":1213371331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6823","nonce":1213371331}]},"public_addr":"172.21.15.39:6819/1213371331","cluster_addr":"172.21.15.39:6821/1213371331","heartbeat_back_addr":"172.21.15.39:6825/1213371331","heartbeat_front_addr":"172.21.15.39:6823/1213371331","state":["exists","up"]},{"osd":7,"uuid":"612f8697-957d-4ef4-a87f-241b36f9d13d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6817","nonce":3299162444}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6819","nonce":3299162444}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6823","nonce":3299162444}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6821","nonce":3299162444}]},"public_addr":"172.21.15.194:6817/3299162444","cluster_addr":"172.21.15.194:6819/3299162444","heartbeat_back_addr":"172.21.15.194:6823/3299162444","heartbeat_front_addr":"172.21.15.194:6821/3299162444","state":["exists","up"]},{"osd":8,"uuid":"836a9107-4d4f-4594-b8dc-0301e64f59d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6817","nonce":2456143145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6819","nonce":2456143145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6823","nonce":2456143145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6821","nonce":2456143145}]},"public_addr":"172.21.15.196:6817/2456143145","cluster_addr":"172.21.15.196:6819/2456143145","heartbeat_back_addr":"172.21.15.196:6823/2456143145","heartbeat_front_addr":"172.21.15.196:6821/2456143145","state":["exists","up"]},{"osd":9,"uuid":"4f30b960-12e3-4b16-bd28-fcc988139b58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6827","nonce":1848634433}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6829","nonce":1848634433}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6833","nonce":1848634433}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6831","nonce":1848634433}]},"public_addr":"172.21.15.39:6827/1848634433","cluster_addr":"172.21.15.39:6829/1848634433","heartbeat_back_addr":"172.21.15.39:6833/1848634433","heartbeat_front_addr":"172.21.15.39:6831/1848634433","state":["exists","up"]},{"osd":10,"uuid":"620f90e8-23e6-4bc0-8269-7219b9e1aebd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6825","nonce":3725064809}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6827","nonce":3725064809}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6831","nonce":3725064809}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6829","nonce":3725064809}]},"public_addr":"172.21.15.194:6825/3725064809","cluster_addr":"172.21.15.194:6827/3725064809","heartbeat_back_addr":"172.21.15.194:6831/3725064809","heartbeat_front_addr":"172.21.15.194:6829/3725064809","state":["exists","up"]},{"osd":11,"uuid":"867a2535-8235-4ceb-a698-eb86869d6d20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6825","nonce":2501212422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6827","nonce":2501212422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6831","nonce":2501212422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6829","nonce":2501212422}]},"public_addr":"172.21.15.196:6825/2501212422","cluster_addr":"172.21.15.196:6827/2501212422","heartbeat_back_addr":"172.21.15.196:6831/2501212422","heartbeat_front_addr":"172.21.15.196:6829/2501212422","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:00:36.931566+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:01:02.614024+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:01:26.193265+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:01:58.308833+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:02:25.049546+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:02:49.748750+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:03:23.582153+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:03:50.447321+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:04:15.812129+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:04:49.998047+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:05:17.087924+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:05:43.167637+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.39:0/3482086140":"2024-03-19T15:58:45.340294+0000","172.21.15.39:0/3844306284":"2024-03-19T15:58:45.340294+0000","172.21.15.39:6801/1810238546":"2024-03-19T15:58:45.340294+0000","172.21.15.39:6800/1810238546":"2024-03-19T15:58:45.340294+0000","172.21.15.39:0/2528397488":"2024-03-19T15:58:10.264798+0000","172.21.15.39:0/2240981254":"2024-03-19T15:58:10.264798+0000","172.21.15.39:6800/1217503189":"2024-03-19T15:58:10.264798+0000","172.21.15.39:0/3754876370":"2024-03-19T15:58:10.264798+0000","172.21.15.39:6801/1217503189":"2024-03-19T15:58:10.264798+0000","172.21.15.39:0/1874637628":"2024-03-19T15:58:45.340294+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T16:06:40.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.217+0000 7fced0200640 1 -- 172.21.15.39:0/2084665538 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fceb00a2800 msgr2=0x7fceb00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:40.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.217+0000 7fced0200640 1 --2- 172.21.15.39:0/2084665538 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fceb00a2800 0x7fceb00a4cc0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fceb4009fc0 tx=0x7fceb4005f90 comp rx=0 tx=0).stop 2024-03-18T16:06:40.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.217+0000 7fced0200640 1 -- 172.21.15.39:0/2084665538 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcec8103a00 msgr2=0x7fcec8199d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:40.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.217+0000 7fced0200640 1 --2- 172.21.15.39:0/2084665538 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcec8103a00 0x7fcec8199d90 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fcebc000c00 tx=0x7fcebc02c650 comp rx=0 tx=0).stop 2024-03-18T16:06:40.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.217+0000 7fced0200640 1 -- 172.21.15.39:0/2084665538 shutdown_connections 2024-03-18T16:06:40.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.217+0000 7fced0200640 1 --2- 172.21.15.39:0/2084665538 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcec8104bf0 0x7fcec819a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:40.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.217+0000 7fced0200640 1 --2- 172.21.15.39:0/2084665538 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fceb00a2800 0x7fceb00a4cc0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:40.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.217+0000 7fced0200640 1 --2- 172.21.15.39:0/2084665538 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcec81055b0 0x7fcec81a1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:40.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.217+0000 7fced0200640 1 --2- 172.21.15.39:0/2084665538 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcec8103a00 0x7fcec8199d90 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:40.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.217+0000 7fced0200640 1 -- 172.21.15.39:0/2084665538 >> 172.21.15.39:0/2084665538 conn(0x7fcec80fef90 msgr2=0x7fcec8107ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:40.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.217+0000 7fced0200640 1 -- 172.21.15.39:0/2084665538 shutdown_connections 2024-03-18T16:06:40.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.217+0000 7fced0200640 1 -- 172.21.15.39:0/2084665538 wait complete. 2024-03-18T16:06:40.238 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T16:01:30.048247+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-18T16:06:40.238 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T16:06:40.440 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.437+0000 7fb921a00640 1 -- 172.21.15.39:0/2960374047 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb91c103a20 msgr2=0x7fb91c103e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:40.440 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.437+0000 7fb921a00640 1 --2- 172.21.15.39:0/2960374047 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb91c103a20 0x7fb91c103e40 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fb90c009900 tx=0x7fb90c02bc30 comp rx=0 tx=0).stop 2024-03-18T16:06:40.440 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.437+0000 7fb921a00640 1 -- 172.21.15.39:0/2960374047 shutdown_connections 2024-03-18T16:06:40.440 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.437+0000 7fb921a00640 1 --2- 172.21.15.39:0/2960374047 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb91c1055d0 0x7fb91c10be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:40.440 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.437+0000 7fb921a00640 1 --2- 172.21.15.39:0/2960374047 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb91c104c10 0x7fb91c105090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:40.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.437+0000 7fb921a00640 1 --2- 172.21.15.39:0/2960374047 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb91c103a20 0x7fb91c103e40 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:40.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.437+0000 7fb921a00640 1 -- 172.21.15.39:0/2960374047 >> 172.21.15.39:0/2960374047 conn(0x7fb91c0fef90 msgr2=0x7fb91c1013d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:40.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.437+0000 7fb921a00640 1 -- 172.21.15.39:0/2960374047 shutdown_connections 2024-03-18T16:06:40.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.437+0000 7fb921a00640 1 -- 172.21.15.39:0/2960374047 wait complete. 2024-03-18T16:06:40.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.438+0000 7fb921a00640 1 Processor -- start 2024-03-18T16:06:40.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.439+0000 7fb921a00640 1 -- start start 2024-03-18T16:06:40.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.439+0000 7fb921a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb91c103a20 0x7fb91c199da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:40.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.439+0000 7fb921a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb91c104c10 0x7fb91c19a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:40.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.439+0000 7fb921a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb91c1055d0 0x7fb91c1a1310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:40.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.439+0000 7fb921a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fb91c10f6e0 con 0x7fb91c1055d0 2024-03-18T16:06:40.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.439+0000 7fb921a00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fb91c10f560 con 0x7fb91c103a20 2024-03-18T16:06:40.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.439+0000 7fb921a00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fb91c10f860 con 0x7fb91c104c10 2024-03-18T16:06:40.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.439+0000 7fb91aa00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb91c103a20 0x7fb91c199da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:40.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.439+0000 7fb91b400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb91c1055d0 0x7fb91c1a1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:40.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.439+0000 7fb91aa00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb91c103a20 0x7fb91c199da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:56500/0 (socket says 172.21.15.39:56500) 2024-03-18T16:06:40.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.439+0000 7fb91aa00640 1 -- 172.21.15.39:0/193356851 learned_addr learned my addr 172.21.15.39:0/193356851 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:06:40.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.439+0000 7fb91a000640 1 --2- 172.21.15.39:0/193356851 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb91c104c10 0x7fb91c19a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:40.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.440+0000 7fb91aa00640 1 -- 172.21.15.39:0/193356851 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb91c104c10 msgr2=0x7fb91c19a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:40.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.440+0000 7fb91aa00640 1 --2- 172.21.15.39:0/193356851 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb91c104c10 0x7fb91c19a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:40.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.440+0000 7fb91aa00640 1 -- 172.21.15.39:0/193356851 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb91c1055d0 msgr2=0x7fb91c1a1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:40.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.440+0000 7fb91aa00640 1 --2- 172.21.15.39:0/193356851 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb91c1055d0 0x7fb91c1a1310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:40.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.440+0000 7fb91aa00640 1 -- 172.21.15.39:0/193356851 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb91c1a19b0 con 0x7fb91c103a20 2024-03-18T16:06:40.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.440+0000 7fb91aa00640 1 --2- 172.21.15.39:0/193356851 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb91c103a20 0x7fb91c199da0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb90c000c00 tx=0x7fb90c02c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:40.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.440+0000 7fb913400640 1 -- 172.21.15.39:0/193356851 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb90c043020 con 0x7fb91c103a20 2024-03-18T16:06:40.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.440+0000 7fb913400640 1 -- 172.21.15.39:0/193356851 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb90c0054e0 con 0x7fb91c103a20 2024-03-18T16:06:40.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.440+0000 7fb921a00640 1 -- 172.21.15.39:0/193356851 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb91c1a1c40 con 0x7fb91c103a20 2024-03-18T16:06:40.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.440+0000 7fb913400640 1 -- 172.21.15.39:0/193356851 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb90c003cd0 con 0x7fb91c103a20 2024-03-18T16:06:40.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.441+0000 7fb921a00640 1 -- 172.21.15.39:0/193356851 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb91c1a2190 con 0x7fb91c103a20 2024-03-18T16:06:40.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.442+0000 7fb921a00640 1 -- 172.21.15.39:0/193356851 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb91c068a20 con 0x7fb91c103a20 2024-03-18T16:06:40.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.444+0000 7fb913400640 1 -- 172.21.15.39:0/193356851 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fb90c046020 con 0x7fb91c103a20 2024-03-18T16:06:40.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.445+0000 7fb913400640 1 --2- 172.21.15.39:0/193356851 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb9000a2800 0x7fb9000a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:40.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.446+0000 7fb913400640 1 -- 172.21.15.39:0/193356851 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fb90c0f0700 con 0x7fb91c103a20 2024-03-18T16:06:40.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.446+0000 7fb91a000640 1 --2- 172.21.15.39:0/193356851 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb9000a2800 0x7fb9000a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:40.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.447+0000 7fb91a000640 1 --2- 172.21.15.39:0/193356851 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb9000a2800 0x7fb9000a4cc0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fb908009bd0 tx=0x7fb908005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:40.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.449+0000 7fb913400640 1 -- 172.21.15.39:0/193356851 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb90c04d020 con 0x7fb91c103a20 2024-03-18T16:06:40.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.678+0000 7fb921a00640 1 -- 172.21.15.39:0/193356851 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb91c0623e0 con 0x7fb91c103a20 2024-03-18T16:06:40.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.679+0000 7fb913400640 1 -- 172.21.15.39:0/193356851 <== mon.2 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7fb90c0bdba0 con 0x7fb91c103a20 2024-03-18T16:06:40.680 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2024-03-18T16:06:40.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.681+0000 7fb921a00640 1 -- 172.21.15.39:0/193356851 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb9000a2800 msgr2=0x7fb9000a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:40.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.681+0000 7fb921a00640 1 --2- 172.21.15.39:0/193356851 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb9000a2800 0x7fb9000a4cc0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fb908009bd0 tx=0x7fb908005f90 comp rx=0 tx=0).stop 2024-03-18T16:06:40.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.681+0000 7fb921a00640 1 -- 172.21.15.39:0/193356851 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb91c103a20 msgr2=0x7fb91c199da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:40.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.681+0000 7fb921a00640 1 --2- 172.21.15.39:0/193356851 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb91c103a20 0x7fb91c199da0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb90c000c00 tx=0x7fb90c02c650 comp rx=0 tx=0).stop 2024-03-18T16:06:40.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.681+0000 7fb921a00640 1 -- 172.21.15.39:0/193356851 shutdown_connections 2024-03-18T16:06:40.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.681+0000 7fb921a00640 1 --2- 172.21.15.39:0/193356851 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb9000a2800 0x7fb9000a4cc0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:40.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.681+0000 7fb921a00640 1 --2- 172.21.15.39:0/193356851 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb91c1055d0 0x7fb91c1a1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:40.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.681+0000 7fb921a00640 1 --2- 172.21.15.39:0/193356851 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb91c104c10 0x7fb91c19a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:40.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.681+0000 7fb921a00640 1 --2- 172.21.15.39:0/193356851 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb91c103a20 0x7fb91c199da0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:40.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.681+0000 7fb921a00640 1 -- 172.21.15.39:0/193356851 >> 172.21.15.39:0/193356851 conn(0x7fb91c0fef90 msgr2=0x7fb91c107f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:40.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.681+0000 7fb921a00640 1 -- 172.21.15.39:0/193356851 shutdown_connections 2024-03-18T16:06:40.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.681+0000 7fb921a00640 1 -- 172.21.15.39:0/193356851 wait complete. 2024-03-18T16:06:40.710 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-18T16:06:40.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.934+0000 7f092e800640 1 -- 172.21.15.39:0/123515670 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0928102a10 msgr2=0x7f0928102e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:40.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.934+0000 7f092e800640 1 --2- 172.21.15.39:0/123515670 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0928102a10 0x7f0928102e90 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0914009900 tx=0x7f091402bc30 comp rx=0 tx=0).stop 2024-03-18T16:06:40.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.934+0000 7f092e800640 1 -- 172.21.15.39:0/123515670 shutdown_connections 2024-03-18T16:06:40.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.934+0000 7f092e800640 1 --2- 172.21.15.39:0/123515670 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0928102a10 0x7f0928102e90 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:40.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.934+0000 7f092e800640 1 --2- 172.21.15.39:0/123515670 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f09281033d0 0x7f0928109c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:40.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.934+0000 7f092e800640 1 --2- 172.21.15.39:0/123515670 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0928101820 0x7f0928101c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:40.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.934+0000 7f092e800640 1 -- 172.21.15.39:0/123515670 >> 172.21.15.39:0/123515670 conn(0x7f09280fcd90 msgr2=0x7f09280ff1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:40.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.934+0000 7f092e800640 1 -- 172.21.15.39:0/123515670 shutdown_connections 2024-03-18T16:06:40.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.934+0000 7f092e800640 1 -- 172.21.15.39:0/123515670 wait complete. 2024-03-18T16:06:40.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.935+0000 7f092e800640 1 Processor -- start 2024-03-18T16:06:40.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.935+0000 7f092e800640 1 -- start start 2024-03-18T16:06:40.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.935+0000 7f092e800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0928101820 0x7f0928199d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:40.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.935+0000 7f092e800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0928102a10 0x7f092819a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:40.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.935+0000 7f092e800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f09281033d0 0x7f09281a1300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:40.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.935+0000 7f092e800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f092810d4d0 con 0x7f0928102a10 2024-03-18T16:06:40.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.935+0000 7f092e800640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f092810d350 con 0x7f0928101820 2024-03-18T16:06:40.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.935+0000 7f092e800640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f092810d650 con 0x7f09281033d0 2024-03-18T16:06:40.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.936+0000 7f0926a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0928102a10 0x7f092819a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:40.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.936+0000 7f0927400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0928101820 0x7f0928199d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:40.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.936+0000 7f0927400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0928101820 0x7f0928199d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:56528/0 (socket says 172.21.15.39:56528) 2024-03-18T16:06:40.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.936+0000 7f0927400640 1 -- 172.21.15.39:0/3969535029 learned_addr learned my addr 172.21.15.39:0/3969535029 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:06:40.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.936+0000 7f0927e00640 1 --2- 172.21.15.39:0/3969535029 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f09281033d0 0x7f09281a1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:40.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.936+0000 7f0927400640 1 -- 172.21.15.39:0/3969535029 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f09281033d0 msgr2=0x7f09281a1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:40.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.936+0000 7f0927400640 1 --2- 172.21.15.39:0/3969535029 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f09281033d0 0x7f09281a1300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:40.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.936+0000 7f0927400640 1 -- 172.21.15.39:0/3969535029 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0928102a10 msgr2=0x7f092819a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:40.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.936+0000 7f0927400640 1 --2- 172.21.15.39:0/3969535029 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0928102a10 0x7f092819a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:40.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.936+0000 7f0927400640 1 -- 172.21.15.39:0/3969535029 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09281a1970 con 0x7f0928101820 2024-03-18T16:06:40.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.936+0000 7f0927400640 1 --2- 172.21.15.39:0/3969535029 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0928101820 0x7f0928199d50 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f091800e9c0 tx=0x7f091800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:40.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.937+0000 7f091fe00640 1 -- 172.21.15.39:0/3969535029 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f091800cd80 con 0x7f0928101820 2024-03-18T16:06:40.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.937+0000 7f091fe00640 1 -- 172.21.15.39:0/3969535029 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09180035f0 con 0x7f0928101820 2024-03-18T16:06:40.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.937+0000 7f092e800640 1 -- 172.21.15.39:0/3969535029 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09281a1c60 con 0x7f0928101820 2024-03-18T16:06:40.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.937+0000 7f091fe00640 1 -- 172.21.15.39:0/3969535029 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0918010690 con 0x7f0928101820 2024-03-18T16:06:40.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.937+0000 7f092e800640 1 -- 172.21.15.39:0/3969535029 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09281a21d0 con 0x7f0928101820 2024-03-18T16:06:40.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.938+0000 7f092e800640 1 -- 172.21.15.39:0/3969535029 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09281a23a0 con 0x7f0928101820 2024-03-18T16:06:40.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.940+0000 7f091fe00640 1 -- 172.21.15.39:0/3969535029 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f09180037a0 con 0x7f0928101820 2024-03-18T16:06:40.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.941+0000 7f091fe00640 1 --2- 172.21.15.39:0/3969535029 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f090c0a2880 0x7f090c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:40.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.941+0000 7f091fe00640 1 -- 172.21.15.39:0/3969535029 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f09180ce970 con 0x7f0928101820 2024-03-18T16:06:40.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.941+0000 7f0926a00640 1 --2- 172.21.15.39:0/3969535029 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f090c0a2880 0x7f090c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:40.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.942+0000 7f0926a00640 1 --2- 172.21.15.39:0/3969535029 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f090c0a2880 0x7f090c0a4d40 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f091402c240 tx=0x7f091402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:40.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:40.945+0000 7f091fe00640 1 -- 172.21.15.39:0/3969535029 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f091809be10 con 0x7f0928101820 2024-03-18T16:06:40.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:40 smithi196 ceph-mon[36307]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:06:40.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:40 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:40.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:40 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:40.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:40 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:40.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:40 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:40.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:40 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/2106069353' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:06:40.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:40 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/2084665538' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:06:40.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:40 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:40.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:40 smithi194 ceph-mon[36200]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:06:40.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:40 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:40.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:40 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:40.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:40 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:40.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:40 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:40.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:40 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/2106069353' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:06:40.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:40 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/2084665538' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:06:40.991 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:40 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:41.014 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:40 smithi039 ceph-mon[31721]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:06:41.015 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:40 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:41.015 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:40 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:41.015 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:40 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:41.015 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:40 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:41.015 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:40 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/2106069353' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:06:41.015 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:40 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/2084665538' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:06:41.016 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:40 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:41.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:41.169+0000 7f092e800640 1 -- 172.21.15.39:0/3969535029 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f09281a2570 con 0x7f0928101820 2024-03-18T16:06:41.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:41.170+0000 7f091fe00640 1 -- 172.21.15.39:0/3969535029 <== mon.2 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v81) ==== 84+0+65 (secure 0 0 0) 0x7f091809b710 con 0x7f0928101820 2024-03-18T16:06:41.172 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:06:41.172 INFO:teuthology.orchestra.run.smithi039.stdout:[ 2024-03-18T16:06:41.172 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:06:41.172 INFO:teuthology.orchestra.run.smithi039.stdout: "poolnum": 1, 2024-03-18T16:06:41.172 INFO:teuthology.orchestra.run.smithi039.stdout: "poolname": ".mgr" 2024-03-18T16:06:41.172 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:06:41.172 INFO:teuthology.orchestra.run.smithi039.stdout:] 2024-03-18T16:06:41.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:41.172+0000 7f092e800640 1 -- 172.21.15.39:0/3969535029 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f090c0a2880 msgr2=0x7f090c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:41.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:41.172+0000 7f092e800640 1 --2- 172.21.15.39:0/3969535029 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f090c0a2880 0x7f090c0a4d40 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f091402c240 tx=0x7f091402df90 comp rx=0 tx=0).stop 2024-03-18T16:06:41.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:41.173+0000 7f092e800640 1 -- 172.21.15.39:0/3969535029 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0928101820 msgr2=0x7f0928199d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:41.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:41.173+0000 7f092e800640 1 --2- 172.21.15.39:0/3969535029 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0928101820 0x7f0928199d50 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f091800e9c0 tx=0x7f091800ee80 comp rx=0 tx=0).stop 2024-03-18T16:06:41.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:41.173+0000 7f092e800640 1 -- 172.21.15.39:0/3969535029 shutdown_connections 2024-03-18T16:06:41.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:41.173+0000 7f092e800640 1 --2- 172.21.15.39:0/3969535029 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f090c0a2880 0x7f090c0a4d40 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:41.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:41.173+0000 7f092e800640 1 --2- 172.21.15.39:0/3969535029 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f09281033d0 0x7f09281a1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:41.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:41.173+0000 7f092e800640 1 --2- 172.21.15.39:0/3969535029 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0928102a10 0x7f092819a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:41.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:41.173+0000 7f092e800640 1 --2- 172.21.15.39:0/3969535029 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0928101820 0x7f0928199d50 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:41.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:41.173+0000 7f092e800640 1 -- 172.21.15.39:0/3969535029 >> 172.21.15.39:0/3969535029 conn(0x7f09280fcd90 msgr2=0x7f0928073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:41.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:41.173+0000 7f092e800640 1 -- 172.21.15.39:0/3969535029 shutdown_connections 2024-03-18T16:06:41.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:41.173+0000 7f092e800640 1 -- 172.21.15.39:0/3969535029 wait complete. 2024-03-18T16:06:41.192 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-18T16:06:41.192 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-18T16:06:41.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:41.390+0000 7f7816c00640 1 -- 172.21.15.39:0/4052184294 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f78101017f0 msgr2=0x7f7810101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:41.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:41.390+0000 7f7816c00640 1 --2- 172.21.15.39:0/4052184294 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f78101017f0 0x7f7810101c10 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f7800009900 tx=0x7f780002bc30 comp rx=0 tx=0).stop 2024-03-18T16:06:41.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:41.390+0000 7f7816c00640 1 -- 172.21.15.39:0/4052184294 shutdown_connections 2024-03-18T16:06:41.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:41.390+0000 7f7816c00640 1 --2- 172.21.15.39:0/4052184294 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f78101033a0 0x7f7810109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:41.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:41.390+0000 7f7816c00640 1 --2- 172.21.15.39:0/4052184294 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f78101029e0 0x7f7810102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:41.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:41.390+0000 7f7816c00640 1 --2- 172.21.15.39:0/4052184294 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f78101017f0 0x7f7810101c10 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:41.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:41.390+0000 7f7816c00640 1 -- 172.21.15.39:0/4052184294 >> 172.21.15.39:0/4052184294 conn(0x7f78100fcd80 msgr2=0x7f78100ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:41.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:41.390+0000 7f7816c00640 1 -- 172.21.15.39:0/4052184294 shutdown_connections 2024-03-18T16:06:41.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:41.390+0000 7f7816c00640 1 -- 172.21.15.39:0/4052184294 wait complete. 2024-03-18T16:06:41.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:41.391+0000 7f7816c00640 1 Processor -- start 2024-03-18T16:06:41.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:41.391+0000 7f7816c00640 1 -- start start 2024-03-18T16:06:41.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:41.392+0000 7f7816c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f78101017f0 0x7f7810197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:41.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:41.392+0000 7f7816c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f78101029e0 0x7f7810198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:41.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:41.392+0000 7f7816c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f78101033a0 0x7f781019f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:41.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:41.392+0000 7f7816c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f781010d4c0 con 0x7f78101033a0 2024-03-18T16:06:41.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:41.392+0000 7f7816c00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f781010d340 con 0x7f78101029e0 2024-03-18T16:06:41.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:41.392+0000 7f7816c00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f781010d640 con 0x7f78101017f0 2024-03-18T16:06:41.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:41.392+0000 7f780fe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f78101017f0 0x7f7810197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:41.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:41.392+0000 7f780fe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f78101017f0 0x7f7810197b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:54828/0 (socket says 172.21.15.39:54828) 2024-03-18T16:06:41.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:41.392+0000 7f780fe00640 1 -- 172.21.15.39:0/506654043 learned_addr learned my addr 172.21.15.39:0/506654043 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:06:41.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:41.392+0000 7f780f400640 1 --2- 172.21.15.39:0/506654043 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f78101029e0 0x7f7810198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:41.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:41.392+0000 7f7814c00640 1 --2- 172.21.15.39:0/506654043 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f78101033a0 0x7f781019f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:41.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:41.393+0000 7f780fe00640 1 -- 172.21.15.39:0/506654043 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f78101029e0 msgr2=0x7f7810198080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:41.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:41.393+0000 7f780fe00640 1 --2- 172.21.15.39:0/506654043 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f78101029e0 0x7f7810198080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:41.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:41.393+0000 7f780fe00640 1 -- 172.21.15.39:0/506654043 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f78101033a0 msgr2=0x7f781019f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:41.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:41.393+0000 7f780fe00640 1 --2- 172.21.15.39:0/506654043 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f78101033a0 0x7f781019f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:41.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:41.393+0000 7f780fe00640 1 -- 172.21.15.39:0/506654043 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f781019f760 con 0x7f78101017f0 2024-03-18T16:06:41.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:41.393+0000 7f780f400640 1 --2- 172.21.15.39:0/506654043 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f78101029e0 0x7f7810198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:06:41.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:41.393+0000 7f780fe00640 1 --2- 172.21.15.39:0/506654043 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f78101017f0 0x7f7810197b40 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7800000c00 tx=0x7f780002c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:41.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:41.393+0000 7f780cc00640 1 -- 172.21.15.39:0/506654043 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7800043020 con 0x7f78101017f0 2024-03-18T16:06:41.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:41.393+0000 7f780cc00640 1 -- 172.21.15.39:0/506654043 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78000054e0 con 0x7f78101017f0 2024-03-18T16:06:41.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:41.393+0000 7f780cc00640 1 -- 172.21.15.39:0/506654043 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7800003c40 con 0x7f78101017f0 2024-03-18T16:06:41.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:41.394+0000 7f7816c00640 1 -- 172.21.15.39:0/506654043 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7810073a10 con 0x7f78101017f0 2024-03-18T16:06:41.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:41.394+0000 7f7816c00640 1 -- 172.21.15.39:0/506654043 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7810073fb0 con 0x7f78101017f0 2024-03-18T16:06:41.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:41.395+0000 7f7816c00640 1 -- 172.21.15.39:0/506654043 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7810108b90 con 0x7f78101017f0 2024-03-18T16:06:41.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:41.398+0000 7f780cc00640 1 -- 172.21.15.39:0/506654043 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f7800046020 con 0x7f78101017f0 2024-03-18T16:06:41.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:41.399+0000 7f780cc00640 1 --2- 172.21.15.39:0/506654043 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f77f40a2880 0x7f77f40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:41.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:41.400+0000 7f780cc00640 1 -- 172.21.15.39:0/506654043 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f78000f0550 con 0x7f78101017f0 2024-03-18T16:06:41.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:41.400+0000 7f780f400640 1 --2- 172.21.15.39:0/506654043 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f77f40a2880 0x7f77f40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:41.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:41.400+0000 7f780f400640 1 --2- 172.21.15.39:0/506654043 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f77f40a2880 0x7f77f40a4d40 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f7810199050 tx=0x7f77fc012040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:41.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:41.404+0000 7f780cc00640 1 -- 172.21.15.39:0/506654043 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f780004d020 con 0x7f78101017f0 2024-03-18T16:06:41.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:41.608+0000 7f7816c00640 1 -- 172.21.15.39:0/506654043 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f781002d3d0 con 0x7f78101017f0 2024-03-18T16:06:41.896 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:41 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/193356851' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T16:06:41.896 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:41 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:41.896 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:41 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:41.896 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:41 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:06:41.896 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:41 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:06:41.896 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:41 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:41.896 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:41 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:06:41.897 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:41 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/3969535029' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-18T16:06:41.933 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:41 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/193356851' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T16:06:41.933 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:41 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:41.933 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:41 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:41.933 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:41 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:06:41.933 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:41 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:06:41.933 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:41 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:41.933 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:41 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:06:41.934 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:41 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/3969535029' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-18T16:06:42.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:41 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/193356851' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T16:06:42.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:41 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:42.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:41 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:42.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:41 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:06:42.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:41 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:06:42.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:41 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:42.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:41 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:06:42.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:41 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/3969535029' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-18T16:06:42.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:42.618+0000 7f780cc00640 1 -- 172.21.15.39:0/506654043 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v82) ==== 138+0+0 (secure 0 0 0) 0x7f78000bda90 con 0x7f78101017f0 2024-03-18T16:06:42.621 INFO:teuthology.orchestra.run.smithi039.stderr:pool 'cephfs_metadata' created 2024-03-18T16:06:42.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:42.624+0000 7f7816c00640 1 -- 172.21.15.39:0/506654043 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f77f40a2880 msgr2=0x7f77f40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:42.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:42.624+0000 7f7816c00640 1 --2- 172.21.15.39:0/506654043 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f77f40a2880 0x7f77f40a4d40 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f7810199050 tx=0x7f77fc012040 comp rx=0 tx=0).stop 2024-03-18T16:06:42.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:42.625+0000 7f7816c00640 1 -- 172.21.15.39:0/506654043 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f78101017f0 msgr2=0x7f7810197b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:42.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:42.625+0000 7f7816c00640 1 --2- 172.21.15.39:0/506654043 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f78101017f0 0x7f7810197b40 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7800000c00 tx=0x7f780002c650 comp rx=0 tx=0).stop 2024-03-18T16:06:42.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:42.626+0000 7f7816c00640 1 -- 172.21.15.39:0/506654043 shutdown_connections 2024-03-18T16:06:42.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:42.626+0000 7f7816c00640 1 --2- 172.21.15.39:0/506654043 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f78101029e0 0x7f7810198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:42.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:42.626+0000 7f7816c00640 1 --2- 172.21.15.39:0/506654043 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f77f40a2880 0x7f77f40a4d40 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:42.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:42.626+0000 7f7816c00640 1 --2- 172.21.15.39:0/506654043 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f78101033a0 0x7f781019f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:42.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:42.626+0000 7f7816c00640 1 --2- 172.21.15.39:0/506654043 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f78101017f0 0x7f7810197b40 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:42.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:42.626+0000 7f7816c00640 1 -- 172.21.15.39:0/506654043 >> 172.21.15.39:0/506654043 conn(0x7f78100fcd80 msgr2=0x7f7810105d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:42.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:42.627+0000 7f7816c00640 1 -- 172.21.15.39:0/506654043 shutdown_connections 2024-03-18T16:06:42.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:42.627+0000 7f7816c00640 1 -- 172.21.15.39:0/506654043 wait complete. 2024-03-18T16:06:42.654 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-03-18T16:06:42.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:42.860+0000 7fcbd5200640 1 -- 172.21.15.39:0/3870316356 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcbd0103a20 msgr2=0x7fcbd0103e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:42.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:42.860+0000 7fcbd5200640 1 --2- 172.21.15.39:0/3870316356 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcbd0103a20 0x7fcbd0103e40 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fcbc0009860 tx=0x7fcbc002bb70 comp rx=0 tx=0).stop 2024-03-18T16:06:42.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:42.860+0000 7fcbd5200640 1 -- 172.21.15.39:0/3870316356 shutdown_connections 2024-03-18T16:06:42.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:42.860+0000 7fcbd5200640 1 --2- 172.21.15.39:0/3870316356 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcbd01055d0 0x7fcbd010be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:42.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:42.860+0000 7fcbd5200640 1 --2- 172.21.15.39:0/3870316356 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcbd0104c10 0x7fcbd0105090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:42.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:42.860+0000 7fcbd5200640 1 --2- 172.21.15.39:0/3870316356 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcbd0103a20 0x7fcbd0103e40 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:42.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:42.860+0000 7fcbd5200640 1 -- 172.21.15.39:0/3870316356 >> 172.21.15.39:0/3870316356 conn(0x7fcbd00fef90 msgr2=0x7fcbd01013d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:42.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:42.860+0000 7fcbd5200640 1 -- 172.21.15.39:0/3870316356 shutdown_connections 2024-03-18T16:06:42.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:42.860+0000 7fcbd5200640 1 -- 172.21.15.39:0/3870316356 wait complete. 2024-03-18T16:06:42.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:42.861+0000 7fcbd5200640 1 Processor -- start 2024-03-18T16:06:42.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:42.861+0000 7fcbd5200640 1 -- start start 2024-03-18T16:06:42.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:42.861+0000 7fcbd5200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcbd0103a20 0x7fcbd0199de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:42.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:42.861+0000 7fcbd5200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcbd0104c10 0x7fcbd019a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:42.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:42.861+0000 7fcbd5200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcbd01055d0 0x7fcbd01a1390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:42.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:42.861+0000 7fcbd5200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fcbd010f690 con 0x7fcbd0104c10 2024-03-18T16:06:42.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:42.861+0000 7fcbd5200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fcbd010f510 con 0x7fcbd0103a20 2024-03-18T16:06:42.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:42.861+0000 7fcbd5200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fcbd010f810 con 0x7fcbd01055d0 2024-03-18T16:06:42.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:42.861+0000 7fcbcde00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcbd0103a20 0x7fcbd0199de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:42.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:42.861+0000 7fcbcd400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcbd0104c10 0x7fcbd019a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:42.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:42.861+0000 7fcbcde00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcbd0103a20 0x7fcbd0199de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:56562/0 (socket says 172.21.15.39:56562) 2024-03-18T16:06:42.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:42.861+0000 7fcbcde00640 1 -- 172.21.15.39:0/3947090173 learned_addr learned my addr 172.21.15.39:0/3947090173 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:06:42.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:42.862+0000 7fcbce800640 1 --2- 172.21.15.39:0/3947090173 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcbd01055d0 0x7fcbd01a1390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:42.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:42.862+0000 7fcbcde00640 1 -- 172.21.15.39:0/3947090173 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcbd01055d0 msgr2=0x7fcbd01a1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:42.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:42.862+0000 7fcbcde00640 1 --2- 172.21.15.39:0/3947090173 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcbd01055d0 0x7fcbd01a1390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:42.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:42.862+0000 7fcbcde00640 1 -- 172.21.15.39:0/3947090173 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcbd0104c10 msgr2=0x7fcbd019a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:42.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:42.862+0000 7fcbcde00640 1 --2- 172.21.15.39:0/3947090173 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcbd0104c10 0x7fcbd019a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:42.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:42.862+0000 7fcbcde00640 1 -- 172.21.15.39:0/3947090173 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbd01a1a00 con 0x7fcbd0103a20 2024-03-18T16:06:42.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:42.862+0000 7fcbcde00640 1 --2- 172.21.15.39:0/3947090173 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcbd0103a20 0x7fcbd0199de0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fcbc002c0b0 tx=0x7fcbc0032430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:42.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:42.862+0000 7fcbc6a00640 1 -- 172.21.15.39:0/3947090173 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbc0041020 con 0x7fcbd0103a20 2024-03-18T16:06:42.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:42.863+0000 7fcbd5200640 1 -- 172.21.15.39:0/3947090173 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbd01a1c90 con 0x7fcbd0103a20 2024-03-18T16:06:42.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:42.863+0000 7fcbc6a00640 1 -- 172.21.15.39:0/3947090173 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcbc002c9f0 con 0x7fcbd0103a20 2024-03-18T16:06:42.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:42.863+0000 7fcbc6a00640 1 -- 172.21.15.39:0/3947090173 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbc0039040 con 0x7fcbd0103a20 2024-03-18T16:06:42.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:42.863+0000 7fcbd5200640 1 -- 172.21.15.39:0/3947090173 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbd01a2180 con 0x7fcbd0103a20 2024-03-18T16:06:42.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:42.864+0000 7fcbd5200640 1 -- 172.21.15.39:0/3947090173 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbd010ae00 con 0x7fcbd0103a20 2024-03-18T16:06:42.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:42.865+0000 7fcbc6a00640 1 -- 172.21.15.39:0/3947090173 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fcbc0044020 con 0x7fcbd0103a20 2024-03-18T16:06:42.868 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:42.867+0000 7fcbc6a00640 1 --2- 172.21.15.39:0/3947090173 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fcbb40a2800 0x7fcbb40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:42.868 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:42.867+0000 7fcbc6a00640 1 -- 172.21.15.39:0/3947090173 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7497+0+0 (secure 0 0 0) 0x7fcbc00f00a0 con 0x7fcbd0103a20 2024-03-18T16:06:42.868 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:42.867+0000 7fcbcd400640 1 --2- 172.21.15.39:0/3947090173 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fcbb40a2800 0x7fcbb40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:42.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:42.868+0000 7fcbcd400640 1 --2- 172.21.15.39:0/3947090173 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fcbb40a2800 0x7fcbb40a4cc0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fcbd019b2f0 tx=0x7fcbbc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:42.872 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:42.871+0000 7fcbc6a00640 1 -- 172.21.15.39:0/3947090173 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fcbc004b020 con 0x7fcbd0103a20 2024-03-18T16:06:42.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:42 smithi196 ceph-mon[36307]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:06:42.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:42 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/506654043' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-18T16:06:42.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:42 smithi196 ceph-mon[36307]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-18T16:06:42.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:42 smithi194 ceph-mon[36200]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:06:42.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:42 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/506654043' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-18T16:06:42.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:42 smithi194 ceph-mon[36200]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-18T16:06:43.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:42 smithi039 ceph-mon[31721]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:06:43.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:42 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/506654043' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-18T16:06:43.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:42 smithi039 ceph-mon[31721]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-18T16:06:43.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:43.114+0000 7fcbd5200640 1 -- 172.21.15.39:0/3947090173 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7fcbd002d590 con 0x7fcbd0103a20 2024-03-18T16:06:43.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:43.691+0000 7fcbc6a00640 1 -- 172.21.15.39:0/3947090173 <== mon.2 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v83) ==== 171+0+0 (secure 0 0 0) 0x7fcbd002d590 con 0x7fcbd0103a20 2024-03-18T16:06:43.693 INFO:teuthology.orchestra.run.smithi039.stderr:pool 'cephfs_data' created 2024-03-18T16:06:43.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:43.696+0000 7fcbd5200640 1 -- 172.21.15.39:0/3947090173 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fcbb40a2800 msgr2=0x7fcbb40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:43.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:43.696+0000 7fcbd5200640 1 --2- 172.21.15.39:0/3947090173 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fcbb40a2800 0x7fcbb40a4cc0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fcbd019b2f0 tx=0x7fcbbc005f90 comp rx=0 tx=0).stop 2024-03-18T16:06:43.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:43.696+0000 7fcbd5200640 1 -- 172.21.15.39:0/3947090173 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcbd0103a20 msgr2=0x7fcbd0199de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:43.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:43.696+0000 7fcbd5200640 1 --2- 172.21.15.39:0/3947090173 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcbd0103a20 0x7fcbd0199de0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fcbc002c0b0 tx=0x7fcbc0032430 comp rx=0 tx=0).stop 2024-03-18T16:06:43.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:43.698+0000 7fcbd5200640 1 -- 172.21.15.39:0/3947090173 shutdown_connections 2024-03-18T16:06:43.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:43.698+0000 7fcbd5200640 1 --2- 172.21.15.39:0/3947090173 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fcbb40a2800 0x7fcbb40a4cc0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:43.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:43.698+0000 7fcbd5200640 1 --2- 172.21.15.39:0/3947090173 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcbd01055d0 0x7fcbd01a1390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:43.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:43.698+0000 7fcbd5200640 1 --2- 172.21.15.39:0/3947090173 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcbd0104c10 0x7fcbd019a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:43.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:43.698+0000 7fcbd5200640 1 --2- 172.21.15.39:0/3947090173 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcbd0103a20 0x7fcbd0199de0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:43.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:43.698+0000 7fcbd5200640 1 -- 172.21.15.39:0/3947090173 >> 172.21.15.39:0/3947090173 conn(0x7fcbd00fef90 msgr2=0x7fcbd0107fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:43.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:43.699+0000 7fcbd5200640 1 -- 172.21.15.39:0/3947090173 shutdown_connections 2024-03-18T16:06:43.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:43.699+0000 7fcbd5200640 1 -- 172.21.15.39:0/3947090173 wait complete. 2024-03-18T16:06:43.732 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-18T16:06:43.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:43.916+0000 7ff586400640 1 -- 172.21.15.39:0/1921140847 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff5800757c0 msgr2=0x7ff580073c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:43.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:43.916+0000 7ff586400640 1 --2- 172.21.15.39:0/1921140847 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff5800757c0 0x7ff580073c10 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff570009ca0 tx=0x7ff57002c190 comp rx=0 tx=0).stop 2024-03-18T16:06:43.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:43.917+0000 7ff586400640 1 -- 172.21.15.39:0/1921140847 shutdown_connections 2024-03-18T16:06:43.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:43.917+0000 7ff586400640 1 --2- 172.21.15.39:0/1921140847 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff580109680 0x7ff58010bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:43.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:43.917+0000 7ff586400640 1 --2- 172.21.15.39:0/1921140847 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff5800741e0 0x7ff580074640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:43.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:43.917+0000 7ff586400640 1 --2- 172.21.15.39:0/1921140847 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff5800757c0 0x7ff580073c10 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:43.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:43.917+0000 7ff586400640 1 -- 172.21.15.39:0/1921140847 >> 172.21.15.39:0/1921140847 conn(0x7ff5800fcd80 msgr2=0x7ff5800ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:43.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:43.917+0000 7ff586400640 1 -- 172.21.15.39:0/1921140847 shutdown_connections 2024-03-18T16:06:43.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:43.917+0000 7ff586400640 1 -- 172.21.15.39:0/1921140847 wait complete. 2024-03-18T16:06:43.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:43.918+0000 7ff586400640 1 Processor -- start 2024-03-18T16:06:43.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:43.918+0000 7ff586400640 1 -- start start 2024-03-18T16:06:43.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:43.918+0000 7ff586400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff5800741e0 0x7ff580199d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:43.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:43.918+0000 7ff586400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff5800757c0 0x7ff58019a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:43.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:43.919+0000 7ff586400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff580109680 0x7ff5801a12a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:43.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:43.919+0000 7ff586400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7ff58010f640 con 0x7ff580109680 2024-03-18T16:06:43.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:43.919+0000 7ff586400640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ff58010f4c0 con 0x7ff5800757c0 2024-03-18T16:06:43.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:43.919+0000 7ff586400640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7ff58010f7c0 con 0x7ff5800741e0 2024-03-18T16:06:43.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:43.919+0000 7ff57f400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff5800741e0 0x7ff580199d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:43.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:43.919+0000 7ff57f400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff5800741e0 0x7ff580199d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:54872/0 (socket says 172.21.15.39:54872) 2024-03-18T16:06:43.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:43.919+0000 7ff57f400640 1 -- 172.21.15.39:0/1935843605 learned_addr learned my addr 172.21.15.39:0/1935843605 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:06:43.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:43.919+0000 7ff57fe00640 1 --2- 172.21.15.39:0/1935843605 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff580109680 0x7ff5801a12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:43.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:43.919+0000 7ff576a00640 1 --2- 172.21.15.39:0/1935843605 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff5800757c0 0x7ff58019a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:43.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:43.919+0000 7ff57f400640 1 -- 172.21.15.39:0/1935843605 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff5800757c0 msgr2=0x7ff58019a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:43.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:43.919+0000 7ff57f400640 1 --2- 172.21.15.39:0/1935843605 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff5800757c0 0x7ff58019a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:43.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:43.919+0000 7ff57f400640 1 -- 172.21.15.39:0/1935843605 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff580109680 msgr2=0x7ff5801a12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:43.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:43.919+0000 7ff57f400640 1 --2- 172.21.15.39:0/1935843605 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff580109680 0x7ff5801a12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:43.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:43.919+0000 7ff57f400640 1 -- 172.21.15.39:0/1935843605 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5801a1940 con 0x7ff5800741e0 2024-03-18T16:06:43.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:43.919+0000 7ff576a00640 1 --2- 172.21.15.39:0/1935843605 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff5800757c0 0x7ff58019a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:06:43.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:43.920+0000 7ff57f400640 1 --2- 172.21.15.39:0/1935843605 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff5800741e0 0x7ff580199d30 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff5700326d0 tx=0x7ff570035f60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:43.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:43.920+0000 7ff57cc00640 1 -- 172.21.15.39:0/1935843605 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff570039040 con 0x7ff5800741e0 2024-03-18T16:06:43.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:43.920+0000 7ff57cc00640 1 -- 172.21.15.39:0/1935843605 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff570044070 con 0x7ff5800741e0 2024-03-18T16:06:43.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:43.920+0000 7ff586400640 1 -- 172.21.15.39:0/1935843605 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5801a1bd0 con 0x7ff5800741e0 2024-03-18T16:06:43.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:43.920+0000 7ff57cc00640 1 -- 172.21.15.39:0/1935843605 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5700043a0 con 0x7ff5800741e0 2024-03-18T16:06:43.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:43.920+0000 7ff586400640 1 -- 172.21.15.39:0/1935843605 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5801a20e0 con 0x7ff5800741e0 2024-03-18T16:06:43.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:43.921+0000 7ff586400640 1 -- 172.21.15.39:0/1935843605 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff580108440 con 0x7ff5800741e0 2024-03-18T16:06:43.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:43.923+0000 7ff57cc00640 1 -- 172.21.15.39:0/1935843605 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7ff570004540 con 0x7ff5800741e0 2024-03-18T16:06:43.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:43.925+0000 7ff57cc00640 1 --2- 172.21.15.39:0/1935843605 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff5640a2880 0x7ff5640a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:43.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:43.925+0000 7ff57cc00640 1 -- 172.21.15.39:0/1935843605 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7879+0+0 (secure 0 0 0) 0x7ff5700f0ec0 con 0x7ff5800741e0 2024-03-18T16:06:43.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:43.925+0000 7ff576a00640 1 --2- 172.21.15.39:0/1935843605 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff5640a2880 0x7ff5640a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:43.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:43.926+0000 7ff576a00640 1 --2- 172.21.15.39:0/1935843605 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff5640a2880 0x7ff5640a4d40 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ff568009c30 tx=0x7ff568005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:43.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:43.929+0000 7ff57cc00640 1 -- 172.21.15.39:0/1935843605 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff5700be070 con 0x7ff5800741e0 2024-03-18T16:06:43.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:43 smithi196 ceph-mon[36307]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-18T16:06:43.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:43 smithi196 ceph-mon[36307]: osdmap e82: 12 total, 12 up, 12 in 2024-03-18T16:06:43.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:43 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:43.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:43 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:43.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:43 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:43.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:43 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:43.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:43 smithi194 ceph-mon[36200]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-18T16:06:43.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:43 smithi194 ceph-mon[36200]: osdmap e82: 12 total, 12 up, 12 in 2024-03-18T16:06:43.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:43 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:43.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:43 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:43.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:43 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:43.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:43 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:44.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:43 smithi039 ceph-mon[31721]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-18T16:06:44.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:43 smithi039 ceph-mon[31721]: osdmap e82: 12 total, 12 up, 12 in 2024-03-18T16:06:44.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:43 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:44.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:43 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:44.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:43 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:44.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:43 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:44.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:44.177+0000 7ff586400640 1 -- 172.21.15.39:0/1935843605 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7ff580062480 con 0x7ff5800741e0 2024-03-18T16:06:44.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:44.826+0000 7ff57cc00640 1 -- 172.21.15.39:0/1935843605 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7ff5700bd970 con 0x7ff5800741e0 2024-03-18T16:06:44.827 INFO:teuthology.orchestra.run.smithi039.stderr:new fs with metadata pool 2 and data pool 3 2024-03-18T16:06:44.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:44.832+0000 7ff586400640 1 -- 172.21.15.39:0/1935843605 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff5640a2880 msgr2=0x7ff5640a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:44.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:44.832+0000 7ff586400640 1 --2- 172.21.15.39:0/1935843605 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff5640a2880 0x7ff5640a4d40 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ff568009c30 tx=0x7ff568005f90 comp rx=0 tx=0).stop 2024-03-18T16:06:44.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:44.832+0000 7ff586400640 1 -- 172.21.15.39:0/1935843605 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff5800741e0 msgr2=0x7ff580199d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:44.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:44.832+0000 7ff586400640 1 --2- 172.21.15.39:0/1935843605 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff5800741e0 0x7ff580199d30 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff5700326d0 tx=0x7ff570035f60 comp rx=0 tx=0).stop 2024-03-18T16:06:44.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:44.832+0000 7ff586400640 1 -- 172.21.15.39:0/1935843605 shutdown_connections 2024-03-18T16:06:44.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:44.832+0000 7ff586400640 1 --2- 172.21.15.39:0/1935843605 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff5800757c0 0x7ff58019a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:44.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:44.832+0000 7ff586400640 1 --2- 172.21.15.39:0/1935843605 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff5640a2880 0x7ff5640a4d40 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:44.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:44.832+0000 7ff586400640 1 --2- 172.21.15.39:0/1935843605 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff580109680 0x7ff5801a12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:44.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:44.832+0000 7ff586400640 1 --2- 172.21.15.39:0/1935843605 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff5800741e0 0x7ff580199d30 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:44.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:44.832+0000 7ff586400640 1 -- 172.21.15.39:0/1935843605 >> 172.21.15.39:0/1935843605 conn(0x7ff5800fcd80 msgr2=0x7ff580105610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:44.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:44.833+0000 7ff586400640 1 -- 172.21.15.39:0/1935843605 shutdown_connections 2024-03-18T16:06:44.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:44.834+0000 7ff586400640 1 -- 172.21.15.39:0/1935843605 wait complete. 2024-03-18T16:06:44.855 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-03-18T16:06:44.855 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-03-18T16:06:44.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:44 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/3947090173' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-18T16:06:44.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:44 smithi196 ceph-mon[36307]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-18T16:06:44.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:44 smithi196 ceph-mon[36307]: pgmap v311: 33 pgs: 3 creating+peering, 1 active+clean, 29 unknown; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:06:44.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:44 smithi196 ceph-mon[36307]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-18T16:06:44.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:44 smithi196 ceph-mon[36307]: osdmap e83: 12 total, 12 up, 12 in 2024-03-18T16:06:44.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:44 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:44.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:44 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:44.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:44 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:06:44.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:44 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:06:44.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:44 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:44.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:44 smithi196 ceph-mon[36307]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-18T16:06:44.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:44 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/1935843605' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-18T16:06:44.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:44 smithi196 ceph-mon[36307]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-18T16:06:44.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:44 smithi196 ceph-mon[36307]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-18T16:06:44.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:44 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/3947090173' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-18T16:06:44.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:44 smithi194 ceph-mon[36200]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-18T16:06:44.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:44 smithi194 ceph-mon[36200]: pgmap v311: 33 pgs: 3 creating+peering, 1 active+clean, 29 unknown; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:06:44.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:44 smithi194 ceph-mon[36200]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-18T16:06:44.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:44 smithi194 ceph-mon[36200]: osdmap e83: 12 total, 12 up, 12 in 2024-03-18T16:06:44.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:44 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:44.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:44 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:44.991 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:44 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:06:44.991 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:44 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:06:44.991 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:44 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:44.991 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:44 smithi194 ceph-mon[36200]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-18T16:06:44.991 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:44 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/1935843605' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-18T16:06:44.991 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:44 smithi194 ceph-mon[36200]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-18T16:06:44.991 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:44 smithi194 ceph-mon[36200]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-18T16:06:45.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.063+0000 7fa639000640 1 -- 172.21.15.39:0/1377279533 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa634108ef0 msgr2=0x7fa634109370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:45.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.063+0000 7fa639000640 1 --2- 172.21.15.39:0/1377279533 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa634108ef0 0x7fa634109370 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa61c009900 tx=0x7fa61c02bc30 comp rx=0 tx=0).stop 2024-03-18T16:06:45.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.064+0000 7fa639000640 1 -- 172.21.15.39:0/1377279533 shutdown_connections 2024-03-18T16:06:45.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.064+0000 7fa639000640 1 --2- 172.21.15.39:0/1377279533 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa634108ef0 0x7fa634109370 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:45.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.064+0000 7fa639000640 1 --2- 172.21.15.39:0/1377279533 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa6341098b0 0x7fa634110140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:45.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.064+0000 7fa639000640 1 --2- 172.21.15.39:0/1377279533 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa634107d00 0x7fa634108120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:45.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.064+0000 7fa639000640 1 -- 172.21.15.39:0/1377279533 >> 172.21.15.39:0/1377279533 conn(0x7fa634076cb0 msgr2=0x7fa6340790d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:45.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.064+0000 7fa639000640 1 -- 172.21.15.39:0/1377279533 shutdown_connections 2024-03-18T16:06:45.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.064+0000 7fa639000640 1 -- 172.21.15.39:0/1377279533 wait complete. 2024-03-18T16:06:45.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.065+0000 7fa639000640 1 Processor -- start 2024-03-18T16:06:45.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.065+0000 7fa639000640 1 -- start start 2024-03-18T16:06:45.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.065+0000 7fa639000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa634107d00 0x7fa63419dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:45.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.065+0000 7fa639000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa634108ef0 0x7fa63419e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:45.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.065+0000 7fa639000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa6341098b0 0x7fa6341a5590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:45.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.065+0000 7fa639000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fa6341139d0 con 0x7fa634107d00 2024-03-18T16:06:45.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.065+0000 7fa639000640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fa634113850 con 0x7fa634108ef0 2024-03-18T16:06:45.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.065+0000 7fa639000640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fa634113b50 con 0x7fa6341098b0 2024-03-18T16:06:45.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.066+0000 7fa62be00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa634108ef0 0x7fa63419e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:45.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.066+0000 7fa633400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa634107d00 0x7fa63419dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:45.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.066+0000 7fa62be00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa634108ef0 0x7fa63419e520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:56610/0 (socket says 172.21.15.39:56610) 2024-03-18T16:06:45.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.066+0000 7fa62be00640 1 -- 172.21.15.39:0/1517444994 learned_addr learned my addr 172.21.15.39:0/1517444994 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:06:45.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.066+0000 7fa633e00640 1 --2- 172.21.15.39:0/1517444994 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa6341098b0 0x7fa6341a5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:45.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.066+0000 7fa62be00640 1 -- 172.21.15.39:0/1517444994 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa6341098b0 msgr2=0x7fa6341a5590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:45.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.066+0000 7fa62be00640 1 --2- 172.21.15.39:0/1517444994 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa6341098b0 0x7fa6341a5590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:45.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.066+0000 7fa62be00640 1 -- 172.21.15.39:0/1517444994 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa634107d00 msgr2=0x7fa63419dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:45.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.066+0000 7fa62be00640 1 --2- 172.21.15.39:0/1517444994 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa634107d00 0x7fa63419dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:45.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.066+0000 7fa62be00640 1 -- 172.21.15.39:0/1517444994 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6341a5c00 con 0x7fa634108ef0 2024-03-18T16:06:45.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.067+0000 7fa62be00640 1 --2- 172.21.15.39:0/1517444994 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa634108ef0 0x7fa63419e520 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa61c005ab0 tx=0x7fa61c02c6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:45.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.067+0000 7fa630c00640 1 -- 172.21.15.39:0/1517444994 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa61c043020 con 0x7fa634108ef0 2024-03-18T16:06:45.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.067+0000 7fa630c00640 1 -- 172.21.15.39:0/1517444994 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa61c0054e0 con 0x7fa634108ef0 2024-03-18T16:06:45.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.067+0000 7fa639000640 1 -- 172.21.15.39:0/1517444994 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6341a5e90 con 0x7fa634108ef0 2024-03-18T16:06:45.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.067+0000 7fa630c00640 1 -- 172.21.15.39:0/1517444994 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa61c003ce0 con 0x7fa634108ef0 2024-03-18T16:06:45.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.067+0000 7fa639000640 1 -- 172.21.15.39:0/1517444994 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6341a6350 con 0x7fa634108ef0 2024-03-18T16:06:45.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.068+0000 7fa639000640 1 -- 172.21.15.39:0/1517444994 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa63410f020 con 0x7fa634108ef0 2024-03-18T16:06:45.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.070+0000 7fa630c00640 1 -- 172.21.15.39:0/1517444994 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fa61c046020 con 0x7fa634108ef0 2024-03-18T16:06:45.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.071+0000 7fa630c00640 1 --2- 172.21.15.39:0/1517444994 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa6180a2800 0x7fa6180a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:45.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.072+0000 7fa630c00640 1 -- 172.21.15.39:0/1517444994 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7979+0+0 (secure 0 0 0) 0x7fa61c0f0a30 con 0x7fa634108ef0 2024-03-18T16:06:45.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.072+0000 7fa633400640 1 --2- 172.21.15.39:0/1517444994 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa6180a2800 0x7fa6180a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:45.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.072+0000 7fa633400640 1 --2- 172.21.15.39:0/1517444994 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa6180a2800 0x7fa6180a4cc0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fa624005fd0 tx=0x7fa62400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:45.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.075+0000 7fa630c00640 1 -- 172.21.15.39:0/1517444994 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa61c04d020 con 0x7fa634108ef0 2024-03-18T16:06:45.104 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:44 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/3947090173' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-18T16:06:45.104 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:44 smithi039 ceph-mon[31721]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-18T16:06:45.104 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:44 smithi039 ceph-mon[31721]: pgmap v311: 33 pgs: 3 creating+peering, 1 active+clean, 29 unknown; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:06:45.105 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:44 smithi039 ceph-mon[31721]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-18T16:06:45.105 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:44 smithi039 ceph-mon[31721]: osdmap e83: 12 total, 12 up, 12 in 2024-03-18T16:06:45.105 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:44 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:45.105 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:44 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:45.105 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:44 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:06:45.105 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:44 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:06:45.105 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:44 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:06:45.105 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:44 smithi039 ceph-mon[31721]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-18T16:06:45.105 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:44 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/1935843605' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-18T16:06:45.105 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:44 smithi039 ceph-mon[31721]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-18T16:06:45.105 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:44 smithi039 ceph-mon[31721]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-18T16:06:45.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.285+0000 7fa639000640 1 -- 172.21.15.39:0/1517444994 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7fa6340623e0 con 0x7fa634108ef0 2024-03-18T16:06:45.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.687+0000 7fa630c00640 1 -- 172.21.15.39:0/1517444994 <== mon.2 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v85) ==== 157+0+0 (secure 0 0 0) 0x7fa61c0bdb80 con 0x7fa634108ef0 2024-03-18T16:06:45.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.692+0000 7fa639000640 1 -- 172.21.15.39:0/1517444994 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa6180a2800 msgr2=0x7fa6180a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:45.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.692+0000 7fa639000640 1 --2- 172.21.15.39:0/1517444994 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa6180a2800 0x7fa6180a4cc0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fa624005fd0 tx=0x7fa62400b410 comp rx=0 tx=0).stop 2024-03-18T16:06:45.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.692+0000 7fa639000640 1 -- 172.21.15.39:0/1517444994 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa634108ef0 msgr2=0x7fa63419e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:45.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.692+0000 7fa639000640 1 --2- 172.21.15.39:0/1517444994 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa634108ef0 0x7fa63419e520 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa61c005ab0 tx=0x7fa61c02c6c0 comp rx=0 tx=0).stop 2024-03-18T16:06:45.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.692+0000 7fa639000640 1 -- 172.21.15.39:0/1517444994 shutdown_connections 2024-03-18T16:06:45.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.692+0000 7fa639000640 1 --2- 172.21.15.39:0/1517444994 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa634108ef0 0x7fa63419e520 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:45.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.692+0000 7fa639000640 1 --2- 172.21.15.39:0/1517444994 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa6180a2800 0x7fa6180a4cc0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:45.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.692+0000 7fa639000640 1 --2- 172.21.15.39:0/1517444994 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa6341098b0 0x7fa6341a5590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:45.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.692+0000 7fa639000640 1 --2- 172.21.15.39:0/1517444994 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa634107d00 0x7fa63419dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:45.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.692+0000 7fa639000640 1 -- 172.21.15.39:0/1517444994 >> 172.21.15.39:0/1517444994 conn(0x7fa634076cb0 msgr2=0x7fa63410c1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:45.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.692+0000 7fa639000640 1 -- 172.21.15.39:0/1517444994 shutdown_connections 2024-03-18T16:06:45.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.692+0000 7fa639000640 1 -- 172.21.15.39:0/1517444994 wait complete. 2024-03-18T16:06:45.719 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-03-18T16:06:45.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.951+0000 7f6408a00640 1 -- 172.21.15.39:0/3514889682 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f64041053e0 msgr2=0x7f640410bc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:45.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.951+0000 7f6408a00640 1 --2- 172.21.15.39:0/3514889682 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f64041053e0 0x7f640410bc70 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f63fc009900 tx=0x7f63fc02bc30 comp rx=0 tx=0).stop 2024-03-18T16:06:45.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.952+0000 7f6408a00640 1 -- 172.21.15.39:0/3514889682 shutdown_connections 2024-03-18T16:06:45.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.952+0000 7f6408a00640 1 --2- 172.21.15.39:0/3514889682 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f64041053e0 0x7f640410bc70 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:45.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.952+0000 7f6408a00640 1 --2- 172.21.15.39:0/3514889682 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6404104a20 0x7f6404104ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:45.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.952+0000 7f6408a00640 1 --2- 172.21.15.39:0/3514889682 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6404103830 0x7f6404103c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:45.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.952+0000 7f6408a00640 1 -- 172.21.15.39:0/3514889682 >> 172.21.15.39:0/3514889682 conn(0x7f64040fedc0 msgr2=0x7f64041011e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:45.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.952+0000 7f6408a00640 1 -- 172.21.15.39:0/3514889682 shutdown_connections 2024-03-18T16:06:45.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.952+0000 7f6408a00640 1 -- 172.21.15.39:0/3514889682 wait complete. 2024-03-18T16:06:45.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.953+0000 7f6408a00640 1 Processor -- start 2024-03-18T16:06:45.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.953+0000 7f6408a00640 1 -- start start 2024-03-18T16:06:45.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.953+0000 7f6408a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6404103830 0x7f6404199a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:45.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.953+0000 7f6408a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6404104a20 0x7f6404199fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:45.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.953+0000 7f6408a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f64041053e0 0x7f64041a1020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:45.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.953+0000 7f6408a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f640410f4b0 con 0x7f6404103830 2024-03-18T16:06:45.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.953+0000 7f6408a00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f640410f330 con 0x7f6404104a20 2024-03-18T16:06:45.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.953+0000 7f6408a00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f640410f630 con 0x7f64041053e0 2024-03-18T16:06:45.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.953+0000 7f6402800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6404103830 0x7f6404199a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:45.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.953+0000 7f6401e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6404104a20 0x7f6404199fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:45.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.953+0000 7f6401e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6404104a20 0x7f6404199fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:56634/0 (socket says 172.21.15.39:56634) 2024-03-18T16:06:45.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.953+0000 7f6401e00640 1 -- 172.21.15.39:0/4205432933 learned_addr learned my addr 172.21.15.39:0/4205432933 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:06:45.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.954+0000 7f6403200640 1 --2- 172.21.15.39:0/4205432933 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f64041053e0 0x7f64041a1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:45.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.954+0000 7f6401e00640 1 -- 172.21.15.39:0/4205432933 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f64041053e0 msgr2=0x7f64041a1020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:45.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.954+0000 7f6401e00640 1 --2- 172.21.15.39:0/4205432933 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f64041053e0 0x7f64041a1020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:45.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.954+0000 7f6401e00640 1 -- 172.21.15.39:0/4205432933 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6404103830 msgr2=0x7f6404199a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:45.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.954+0000 7f6401e00640 1 --2- 172.21.15.39:0/4205432933 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6404103830 0x7f6404199a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:45.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.954+0000 7f6401e00640 1 -- 172.21.15.39:0/4205432933 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64041a1690 con 0x7f6404104a20 2024-03-18T16:06:45.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.954+0000 7f6403200640 1 --2- 172.21.15.39:0/4205432933 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f64041053e0 0x7f64041a1020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:06:45.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.954+0000 7f6402800640 1 --2- 172.21.15.39:0/4205432933 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6404103830 0x7f6404199a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:06:45.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.954+0000 7f6401e00640 1 --2- 172.21.15.39:0/4205432933 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6404104a20 0x7f6404199fb0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f63ec00e9f0 tx=0x7f63ec00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:45.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.954+0000 7f63fb400640 1 -- 172.21.15.39:0/4205432933 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63ec00cd80 con 0x7f6404104a20 2024-03-18T16:06:45.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.954+0000 7f63fb400640 1 -- 172.21.15.39:0/4205432933 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f63ec010910 con 0x7f6404104a20 2024-03-18T16:06:45.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.954+0000 7f63fb400640 1 -- 172.21.15.39:0/4205432933 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63ec018990 con 0x7f6404104a20 2024-03-18T16:06:45.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.954+0000 7f6408a00640 1 -- 172.21.15.39:0/4205432933 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64041a1980 con 0x7f6404104a20 2024-03-18T16:06:45.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.955+0000 7f6408a00640 1 -- 172.21.15.39:0/4205432933 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64041a1e80 con 0x7f6404104a20 2024-03-18T16:06:45.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.956+0000 7f6408a00640 1 -- 172.21.15.39:0/4205432933 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f640410aaf0 con 0x7f6404104a20 2024-03-18T16:06:45.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.958+0000 7f63fb400640 1 -- 172.21.15.39:0/4205432933 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f63ec010ac0 con 0x7f6404104a20 2024-03-18T16:06:45.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.959+0000 7f63fb400640 1 --2- 172.21.15.39:0/4205432933 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f63e80a2880 0x7f63e80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:45.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.959+0000 7f63fb400640 1 -- 172.21.15.39:0/4205432933 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 8277+0+0 (secure 0 0 0) 0x7f63ec0cee50 con 0x7f6404104a20 2024-03-18T16:06:45.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.959+0000 7f6402800640 1 --2- 172.21.15.39:0/4205432933 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f63e80a2880 0x7f63e80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:45.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.960+0000 7f6402800640 1 --2- 172.21.15.39:0/4205432933 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f63e80a2880 0x7f63e80a4d40 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f6404104880 tx=0x7f63f4009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:45.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:45.963+0000 7f63fb400640 1 -- 172.21.15.39:0/4205432933 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f63ec09be60 con 0x7f6404104a20 2024-03-18T16:06:45.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:45 smithi196 ceph-mon[36307]: osdmap e84: 12 total, 12 up, 12 in 2024-03-18T16:06:45.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:45 smithi196 ceph-mon[36307]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-18T16:06:45.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:45 smithi196 ceph-mon[36307]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-18T16:06:45.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:45 smithi196 ceph-mon[36307]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-18T16:06:45.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:45 smithi196 ceph-mon[36307]: mds.? [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] up:active 2024-03-18T16:06:45.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:45 smithi196 ceph-mon[36307]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-18T16:06:45.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:45 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/1517444994' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-18T16:06:45.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:45 smithi196 ceph-mon[36307]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-18T16:06:45.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:45 smithi194 ceph-mon[36200]: osdmap e84: 12 total, 12 up, 12 in 2024-03-18T16:06:45.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:45 smithi194 ceph-mon[36200]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-18T16:06:45.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:45 smithi194 ceph-mon[36200]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-18T16:06:45.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:45 smithi194 ceph-mon[36200]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-18T16:06:45.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:45 smithi194 ceph-mon[36200]: mds.? [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] up:active 2024-03-18T16:06:45.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:45 smithi194 ceph-mon[36200]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-18T16:06:45.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:45 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/1517444994' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-18T16:06:45.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:45 smithi194 ceph-mon[36200]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-18T16:06:46.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:45 smithi039 ceph-mon[31721]: osdmap e84: 12 total, 12 up, 12 in 2024-03-18T16:06:46.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:45 smithi039 ceph-mon[31721]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-18T16:06:46.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:45 smithi039 ceph-mon[31721]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-18T16:06:46.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:45 smithi039 ceph-mon[31721]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-18T16:06:46.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:45 smithi039 ceph-mon[31721]: mds.? [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] up:active 2024-03-18T16:06:46.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:45 smithi039 ceph-mon[31721]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-18T16:06:46.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:45 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/1517444994' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-18T16:06:46.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:45 smithi039 ceph-mon[31721]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-18T16:06:46.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:46.165+0000 7f6408a00640 1 -- 172.21.15.39:0/4205432933 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f640402d1b0 con 0x7f6404104a20 2024-03-18T16:06:46.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:46 smithi196 ceph-mon[36307]: pgmap v314: 97 pgs: 11 creating+peering, 17 active+clean, 69 unknown; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:06:46.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:46 smithi196 ceph-mon[36307]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-18T16:06:46.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:46 smithi196 ceph-mon[36307]: osdmap e85: 12 total, 12 up, 12 in 2024-03-18T16:06:46.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:46 smithi196 ceph-mon[36307]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-18T16:06:46.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:46 smithi196 ceph-mon[36307]: Cluster is now healthy 2024-03-18T16:06:46.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:46 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/4205432933' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-18T16:06:46.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:46 smithi196 ceph-mon[36307]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-18T16:06:46.989 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:46 smithi039 ceph-mon[31721]: pgmap v314: 97 pgs: 11 creating+peering, 17 active+clean, 69 unknown; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:06:46.989 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:46 smithi039 ceph-mon[31721]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-18T16:06:46.989 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:46 smithi039 ceph-mon[31721]: osdmap e85: 12 total, 12 up, 12 in 2024-03-18T16:06:46.989 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:46 smithi039 ceph-mon[31721]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-18T16:06:46.989 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:46 smithi039 ceph-mon[31721]: Cluster is now healthy 2024-03-18T16:06:46.989 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:46 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/4205432933' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-18T16:06:46.989 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:46 smithi039 ceph-mon[31721]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-18T16:06:46.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:46 smithi194 ceph-mon[36200]: pgmap v314: 97 pgs: 11 creating+peering, 17 active+clean, 69 unknown; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:06:46.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:46 smithi194 ceph-mon[36200]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-18T16:06:46.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:46 smithi194 ceph-mon[36200]: osdmap e85: 12 total, 12 up, 12 in 2024-03-18T16:06:46.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:46 smithi194 ceph-mon[36200]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-18T16:06:46.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:46 smithi194 ceph-mon[36200]: Cluster is now healthy 2024-03-18T16:06:46.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:46 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/4205432933' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-18T16:06:46.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:46 smithi194 ceph-mon[36200]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-18T16:06:47.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:47.996+0000 7f63fb400640 1 -- 172.21.15.39:0/4205432933 <== mon.2 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v87) ==== 228+0+0 (secure 0 0 0) 0x7f63ec09b760 con 0x7f6404104a20 2024-03-18T16:06:48.000 INFO:teuthology.orchestra.run.smithi039.stderr:pool 'cephfs_data_ec' created 2024-03-18T16:06:48.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:48.005+0000 7f6408a00640 1 -- 172.21.15.39:0/4205432933 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f63e80a2880 msgr2=0x7f63e80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:48.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:48.005+0000 7f6408a00640 1 --2- 172.21.15.39:0/4205432933 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f63e80a2880 0x7f63e80a4d40 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f6404104880 tx=0x7f63f4009500 comp rx=0 tx=0).stop 2024-03-18T16:06:48.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:48.005+0000 7f6408a00640 1 -- 172.21.15.39:0/4205432933 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6404104a20 msgr2=0x7f6404199fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:48.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:48.005+0000 7f6408a00640 1 --2- 172.21.15.39:0/4205432933 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6404104a20 0x7f6404199fb0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f63ec00e9f0 tx=0x7f63ec00eeb0 comp rx=0 tx=0).stop 2024-03-18T16:06:48.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:48.006+0000 7f6408a00640 1 -- 172.21.15.39:0/4205432933 shutdown_connections 2024-03-18T16:06:48.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:48.006+0000 7f6408a00640 1 --2- 172.21.15.39:0/4205432933 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6404104a20 0x7f6404199fb0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:48.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:48.006+0000 7f6408a00640 1 --2- 172.21.15.39:0/4205432933 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f63e80a2880 0x7f63e80a4d40 secure :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f6404104880 tx=0x7f63f4009500 comp rx=0 tx=0).stop 2024-03-18T16:06:48.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:48.006+0000 7f6408a00640 1 --2- 172.21.15.39:0/4205432933 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f64041053e0 0x7f64041a1020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:48.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:48.006+0000 7f6408a00640 1 --2- 172.21.15.39:0/4205432933 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6404103830 0x7f6404199a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:48.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:48.006+0000 7f6408a00640 1 -- 172.21.15.39:0/4205432933 >> 172.21.15.39:0/4205432933 conn(0x7f64040fedc0 msgr2=0x7f6404107cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:48.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:48.008+0000 7f6408a00640 1 -- 172.21.15.39:0/4205432933 shutdown_connections 2024-03-18T16:06:48.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:48.009+0000 7f6408a00640 1 -- 172.21.15.39:0/4205432933 wait complete. 2024-03-18T16:06:48.037 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-03-18T16:06:48.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:48.232+0000 7f3452200640 1 -- 172.21.15.39:0/1719889790 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f344c104a00 msgr2=0x7f344c104e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:48.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:48.232+0000 7f3452200640 1 --2- 172.21.15.39:0/1719889790 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f344c104a00 0x7f344c104e80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3434009900 tx=0x7f343402bc30 comp rx=0 tx=0).stop 2024-03-18T16:06:48.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:48.232+0000 7f3452200640 1 -- 172.21.15.39:0/1719889790 shutdown_connections 2024-03-18T16:06:48.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:48.232+0000 7f3452200640 1 --2- 172.21.15.39:0/1719889790 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f344c104a00 0x7f344c104e80 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:48.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:48.232+0000 7f3452200640 1 --2- 172.21.15.39:0/1719889790 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f344c1053c0 0x7f344c10bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:48.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:48.232+0000 7f3452200640 1 --2- 172.21.15.39:0/1719889790 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f344c103810 0x7f344c103c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:48.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:48.232+0000 7f3452200640 1 -- 172.21.15.39:0/1719889790 >> 172.21.15.39:0/1719889790 conn(0x7f344c0fed80 msgr2=0x7f344c1011c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:48.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:48.232+0000 7f3452200640 1 -- 172.21.15.39:0/1719889790 shutdown_connections 2024-03-18T16:06:48.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:48.232+0000 7f3452200640 1 -- 172.21.15.39:0/1719889790 wait complete. 2024-03-18T16:06:48.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:48.233+0000 7f3452200640 1 Processor -- start 2024-03-18T16:06:48.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:48.234+0000 7f3452200640 1 -- start start 2024-03-18T16:06:48.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:48.234+0000 7f3452200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f344c103810 0x7f344c199b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:48.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:48.234+0000 7f3452200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f344c104a00 0x7f344c19a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:48.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:48.234+0000 7f3452200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f344c1053c0 0x7f344c1a10e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:48.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:48.234+0000 7f3452200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f344c10f490 con 0x7f344c103810 2024-03-18T16:06:48.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:48.234+0000 7f3452200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f344c10f310 con 0x7f344c1053c0 2024-03-18T16:06:48.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:48.234+0000 7f3452200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f344c10f610 con 0x7f344c104a00 2024-03-18T16:06:48.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:48.234+0000 7f3450e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f344c103810 0x7f344c199b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:48.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:48.234+0000 7f3451800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f344c1053c0 0x7f344c1a10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:48.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:48.234+0000 7f3450e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f344c103810 0x7f344c199b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43400/0 (socket says 172.21.15.39:43400) 2024-03-18T16:06:48.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:48.234+0000 7f3450e00640 1 -- 172.21.15.39:0/2913802053 learned_addr learned my addr 172.21.15.39:0/2913802053 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:06:48.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:48.234+0000 7f3443e00640 1 --2- 172.21.15.39:0/2913802053 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f344c104a00 0x7f344c19a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:48.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:48.235+0000 7f3451800640 1 -- 172.21.15.39:0/2913802053 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f344c104a00 msgr2=0x7f344c19a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:48.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:48.235+0000 7f3451800640 1 --2- 172.21.15.39:0/2913802053 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f344c104a00 0x7f344c19a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:48.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:48.235+0000 7f3451800640 1 -- 172.21.15.39:0/2913802053 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f344c103810 msgr2=0x7f344c199b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:48.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:48.235+0000 7f3451800640 1 --2- 172.21.15.39:0/2913802053 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f344c103810 0x7f344c199b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:48.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:48.235+0000 7f3451800640 1 -- 172.21.15.39:0/2913802053 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f344c1a1750 con 0x7f344c1053c0 2024-03-18T16:06:48.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:48.235+0000 7f3443e00640 1 --2- 172.21.15.39:0/2913802053 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f344c104a00 0x7f344c19a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:06:48.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:48.235+0000 7f3451800640 1 --2- 172.21.15.39:0/2913802053 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f344c1053c0 0x7f344c1a10e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f344400da10 tx=0x7f344400ded0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:48.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:48.235+0000 7f344a000640 1 -- 172.21.15.39:0/2913802053 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f344400bd70 con 0x7f344c1053c0 2024-03-18T16:06:48.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:48.235+0000 7f344a000640 1 -- 172.21.15.39:0/2913802053 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3444003e50 con 0x7f344c1053c0 2024-03-18T16:06:48.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:48.235+0000 7f3452200640 1 -- 172.21.15.39:0/2913802053 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f344c1a1a40 con 0x7f344c1053c0 2024-03-18T16:06:48.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:48.235+0000 7f344a000640 1 -- 172.21.15.39:0/2913802053 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34440106a0 con 0x7f344c1053c0 2024-03-18T16:06:48.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:48.235+0000 7f3452200640 1 -- 172.21.15.39:0/2913802053 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f344c1a1f60 con 0x7f344c1053c0 2024-03-18T16:06:48.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:48.236+0000 7f3452200640 1 -- 172.21.15.39:0/2913802053 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f344c10abb0 con 0x7f344c1053c0 2024-03-18T16:06:48.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:48.238+0000 7f344a000640 1 -- 172.21.15.39:0/2913802053 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f3444003970 con 0x7f344c1053c0 2024-03-18T16:06:48.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:48.240+0000 7f344a000640 1 --2- 172.21.15.39:0/2913802053 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f34300a2880 0x7f34300a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:48.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:48.240+0000 7f344a000640 1 -- 172.21.15.39:0/2913802053 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8770+0+0 (secure 0 0 0) 0x7f34440cf100 con 0x7f344c1053c0 2024-03-18T16:06:48.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:48.240+0000 7f3450e00640 1 --2- 172.21.15.39:0/2913802053 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f34300a2880 0x7f34300a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:48.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:48.241+0000 7f3450e00640 1 --2- 172.21.15.39:0/2913802053 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f34300a2880 0x7f34300a4d40 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f343c005fd0 tx=0x7f343c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:48.245 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:48.244+0000 7f344a000640 1 -- 172.21.15.39:0/2913802053 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f344409bf20 con 0x7f344c1053c0 2024-03-18T16:06:48.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:47 smithi039 ceph-mon[31721]: osdmap e86: 12 total, 12 up, 12 in 2024-03-18T16:06:48.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:48.444+0000 7f3452200640 1 -- 172.21.15.39:0/2913802053 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7f344c062450 con 0x7f344c1053c0 2024-03-18T16:06:48.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:47 smithi196 ceph-mon[36307]: osdmap e86: 12 total, 12 up, 12 in 2024-03-18T16:06:48.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:47 smithi194 ceph-mon[36200]: osdmap e86: 12 total, 12 up, 12 in 2024-03-18T16:06:48.990 INFO:journalctl@ceph.mds.b.smithi194.stdout:Mar 18 16:06:48 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-b[67287]: 2024-03-18T16:06:48.581+0000 7fa042c70700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-18T16:06:48.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:48.997+0000 7f344a000640 1 -- 172.21.15.39:0/2913802053 <== mon.2 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v88) ==== 169+0+0 (secure 0 0 0) 0x7f344409b820 con 0x7f344c1053c0 2024-03-18T16:06:48.999 INFO:teuthology.orchestra.run.smithi039.stderr:set pool 4 allow_ec_overwrites to true 2024-03-18T16:06:49.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:49.001+0000 7f3452200640 1 -- 172.21.15.39:0/2913802053 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f34300a2880 msgr2=0x7f34300a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:49.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:49.001+0000 7f3452200640 1 --2- 172.21.15.39:0/2913802053 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f34300a2880 0x7f34300a4d40 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f343c005fd0 tx=0x7f343c00b410 comp rx=0 tx=0).stop 2024-03-18T16:06:49.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:49.001+0000 7f3452200640 1 -- 172.21.15.39:0/2913802053 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f344c1053c0 msgr2=0x7f344c1a10e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:49.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:49.001+0000 7f3452200640 1 --2- 172.21.15.39:0/2913802053 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f344c1053c0 0x7f344c1a10e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f344400da10 tx=0x7f344400ded0 comp rx=0 tx=0).stop 2024-03-18T16:06:49.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:49.001+0000 7f3452200640 1 -- 172.21.15.39:0/2913802053 shutdown_connections 2024-03-18T16:06:49.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:49.001+0000 7f3452200640 1 --2- 172.21.15.39:0/2913802053 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f344c1053c0 0x7f344c1a10e0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:49.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:49.001+0000 7f3452200640 1 --2- 172.21.15.39:0/2913802053 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f34300a2880 0x7f34300a4d40 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:49.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:49.001+0000 7f3452200640 1 --2- 172.21.15.39:0/2913802053 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f344c104a00 0x7f344c19a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:49.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:49.001+0000 7f3452200640 1 --2- 172.21.15.39:0/2913802053 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f344c103810 0x7f344c199b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:49.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:49.001+0000 7f3452200640 1 -- 172.21.15.39:0/2913802053 >> 172.21.15.39:0/2913802053 conn(0x7f344c0fed80 msgr2=0x7f344c107d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:49.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:49.002+0000 7f3452200640 1 -- 172.21.15.39:0/2913802053 shutdown_connections 2024-03-18T16:06:49.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:49.002+0000 7f3452200640 1 -- 172.21.15.39:0/2913802053 wait complete. 2024-03-18T16:06:49.035 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-03-18T16:06:49.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:49.215+0000 7fc22b200640 1 -- 172.21.15.39:0/1170457112 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc22c1015e0 msgr2=0x7fc22c101a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:49.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:49.215+0000 7fc22b200640 1 --2- 172.21.15.39:0/1170457112 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc22c1015e0 0x7fc22c101a00 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc21c009860 tx=0x7fc21c02bb70 comp rx=0 tx=0).stop 2024-03-18T16:06:49.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:49.216+0000 7fc22b200640 1 -- 172.21.15.39:0/1170457112 shutdown_connections 2024-03-18T16:06:49.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:49.216+0000 7fc22b200640 1 --2- 172.21.15.39:0/1170457112 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc22c103190 0x7fc22c109a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:49.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:49.216+0000 7fc22b200640 1 --2- 172.21.15.39:0/1170457112 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc22c1027d0 0x7fc22c102c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:49.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:49.216+0000 7fc22b200640 1 --2- 172.21.15.39:0/1170457112 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc22c1015e0 0x7fc22c101a00 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:49.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:49.216+0000 7fc22b200640 1 -- 172.21.15.39:0/1170457112 >> 172.21.15.39:0/1170457112 conn(0x7fc22c0fcb70 msgr2=0x7fc22c0fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:49.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:49.216+0000 7fc22b200640 1 -- 172.21.15.39:0/1170457112 shutdown_connections 2024-03-18T16:06:49.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:49.216+0000 7fc22b200640 1 -- 172.21.15.39:0/1170457112 wait complete. 2024-03-18T16:06:49.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:49.217+0000 7fc22b200640 1 Processor -- start 2024-03-18T16:06:49.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:49.217+0000 7fc22b200640 1 -- start start 2024-03-18T16:06:49.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:49.217+0000 7fc22b200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc22c1015e0 0x7fc22c199ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:49.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:49.217+0000 7fc22b200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc22c1027d0 0x7fc22c19a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:49.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:49.217+0000 7fc22b200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc22c103190 0x7fc22c1a1070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:49.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:49.217+0000 7fc22b200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fc22c10d230 con 0x7fc22c1015e0 2024-03-18T16:06:49.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:49.217+0000 7fc22b200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fc22c10d0b0 con 0x7fc22c1027d0 2024-03-18T16:06:49.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:49.217+0000 7fc22b200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fc22c10d3b0 con 0x7fc22c103190 2024-03-18T16:06:49.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:49.217+0000 7fc229e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc22c1015e0 0x7fc22c199ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:49.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:49.217+0000 7fc229400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc22c1027d0 0x7fc22c19a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:49.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:49.217+0000 7fc229400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc22c1027d0 0x7fc22c19a000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:36774/0 (socket says 172.21.15.39:36774) 2024-03-18T16:06:49.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:49.217+0000 7fc229e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc22c1015e0 0x7fc22c199ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43410/0 (socket says 172.21.15.39:43410) 2024-03-18T16:06:49.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:49.217+0000 7fc229400640 1 -- 172.21.15.39:0/966800324 learned_addr learned my addr 172.21.15.39:0/966800324 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:06:49.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:49.217+0000 7fc22a800640 1 --2- 172.21.15.39:0/966800324 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc22c103190 0x7fc22c1a1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:49.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:49.218+0000 7fc229e00640 1 -- 172.21.15.39:0/966800324 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc22c103190 msgr2=0x7fc22c1a1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:49.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:49.218+0000 7fc229e00640 1 --2- 172.21.15.39:0/966800324 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc22c103190 0x7fc22c1a1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:49.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:49.218+0000 7fc229e00640 1 -- 172.21.15.39:0/966800324 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc22c1027d0 msgr2=0x7fc22c19a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:49.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:49.218+0000 7fc229e00640 1 --2- 172.21.15.39:0/966800324 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc22c1027d0 0x7fc22c19a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:49.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:49.218+0000 7fc229e00640 1 -- 172.21.15.39:0/966800324 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc22c1a16e0 con 0x7fc22c1015e0 2024-03-18T16:06:49.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:49.218+0000 7fc229400640 1 --2- 172.21.15.39:0/966800324 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc22c1027d0 0x7fc22c19a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:06:49.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:49.218+0000 7fc229e00640 1 --2- 172.21.15.39:0/966800324 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc22c1015e0 0x7fc22c199ac0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fc21c009780 tx=0x7fc21c032430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:49.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:49.218+0000 7fc222a00640 1 -- 172.21.15.39:0/966800324 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc21c041020 con 0x7fc22c1015e0 2024-03-18T16:06:49.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:49.218+0000 7fc222a00640 1 -- 172.21.15.39:0/966800324 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc21c02c9f0 con 0x7fc22c1015e0 2024-03-18T16:06:49.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:49.218+0000 7fc22b200640 1 -- 172.21.15.39:0/966800324 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc22c1a1970 con 0x7fc22c1015e0 2024-03-18T16:06:49.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:49.218+0000 7fc222a00640 1 -- 172.21.15.39:0/966800324 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc21c039040 con 0x7fc22c1015e0 2024-03-18T16:06:49.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:49.219+0000 7fc22b200640 1 -- 172.21.15.39:0/966800324 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc22c1a1e80 con 0x7fc22c1015e0 2024-03-18T16:06:49.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:49.220+0000 7fc22b200640 1 -- 172.21.15.39:0/966800324 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc22c1a2050 con 0x7fc22c1015e0 2024-03-18T16:06:49.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:49.222+0000 7fc222a00640 1 -- 172.21.15.39:0/966800324 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fc21c044020 con 0x7fc22c1015e0 2024-03-18T16:06:49.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:49.223+0000 7fc222a00640 1 --2- 172.21.15.39:0/966800324 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fc2100a2880 0x7fc2100a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:49.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:49.223+0000 7fc222a00640 1 -- 172.21.15.39:0/966800324 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8770+0+0 (secure 0 0 0) 0x7fc21c0f0400 con 0x7fc22c1015e0 2024-03-18T16:06:49.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:49.223+0000 7fc229400640 1 --2- 172.21.15.39:0/966800324 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fc2100a2880 0x7fc2100a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:49.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:49.224+0000 7fc229400640 1 --2- 172.21.15.39:0/966800324 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fc2100a2880 0x7fc2100a4d40 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fc2180095f0 tx=0x7fc218005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:49.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:49.228+0000 7fc222a00640 1 -- 172.21.15.39:0/966800324 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc21c04b020 con 0x7fc22c1015e0 2024-03-18T16:06:49.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:49 smithi039 ceph-mon[31721]: pgmap v317: 97 pgs: 11 creating+peering, 66 active+clean, 20 unknown; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s wr, 3 op/s 2024-03-18T16:06:49.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:49 smithi039 ceph-mon[31721]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-18T16:06:49.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:49 smithi039 ceph-mon[31721]: osdmap e87: 12 total, 12 up, 12 in 2024-03-18T16:06:49.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:49 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/2913802053' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-18T16:06:49.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:49 smithi039 ceph-mon[31721]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-18T16:06:49.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:49 smithi196 ceph-mon[36307]: pgmap v317: 97 pgs: 11 creating+peering, 66 active+clean, 20 unknown; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s wr, 3 op/s 2024-03-18T16:06:49.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:49 smithi196 ceph-mon[36307]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-18T16:06:49.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:49 smithi196 ceph-mon[36307]: osdmap e87: 12 total, 12 up, 12 in 2024-03-18T16:06:49.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:49 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/2913802053' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-18T16:06:49.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:49 smithi196 ceph-mon[36307]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-18T16:06:49.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:49 smithi194 ceph-mon[36200]: pgmap v317: 97 pgs: 11 creating+peering, 66 active+clean, 20 unknown; 577 KiB data, 373 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s wr, 3 op/s 2024-03-18T16:06:49.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:49 smithi194 ceph-mon[36200]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-18T16:06:49.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:49 smithi194 ceph-mon[36200]: osdmap e87: 12 total, 12 up, 12 in 2024-03-18T16:06:49.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:49 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/2913802053' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-18T16:06:49.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:49 smithi194 ceph-mon[36200]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-18T16:06:49.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:49.509+0000 7fc22b200640 1 -- 172.21.15.39:0/966800324 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7fc22c1a2440 con 0x7fc22c1015e0 2024-03-18T16:06:50.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.026+0000 7fc222a00640 1 -- 172.21.15.39:0/966800324 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v16) ==== 137+0+0 (secure 0 0 0) 0x7fc21c0bd220 con 0x7fc22c1015e0 2024-03-18T16:06:50.028 INFO:teuthology.orchestra.run.smithi039.stderr:added data pool 4 to fsmap 2024-03-18T16:06:50.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.031+0000 7fc22b200640 1 -- 172.21.15.39:0/966800324 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fc2100a2880 msgr2=0x7fc2100a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:50.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.031+0000 7fc22b200640 1 --2- 172.21.15.39:0/966800324 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fc2100a2880 0x7fc2100a4d40 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fc2180095f0 tx=0x7fc218005f90 comp rx=0 tx=0).stop 2024-03-18T16:06:50.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.031+0000 7fc22b200640 1 -- 172.21.15.39:0/966800324 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc22c1015e0 msgr2=0x7fc22c199ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:50.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.031+0000 7fc22b200640 1 --2- 172.21.15.39:0/966800324 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc22c1015e0 0x7fc22c199ac0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fc21c009780 tx=0x7fc21c032430 comp rx=0 tx=0).stop 2024-03-18T16:06:50.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.031+0000 7fc22b200640 1 -- 172.21.15.39:0/966800324 shutdown_connections 2024-03-18T16:06:50.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.031+0000 7fc22b200640 1 --2- 172.21.15.39:0/966800324 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc22c1027d0 0x7fc22c19a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:50.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.031+0000 7fc22b200640 1 --2- 172.21.15.39:0/966800324 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fc2100a2880 0x7fc2100a4d40 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:50.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.031+0000 7fc22b200640 1 --2- 172.21.15.39:0/966800324 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc22c103190 0x7fc22c1a1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:50.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.031+0000 7fc22b200640 1 --2- 172.21.15.39:0/966800324 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc22c1015e0 0x7fc22c199ac0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:50.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.031+0000 7fc22b200640 1 -- 172.21.15.39:0/966800324 >> 172.21.15.39:0/966800324 conn(0x7fc22c0fcb70 msgr2=0x7fc22c073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:50.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.032+0000 7fc22b200640 1 -- 172.21.15.39:0/966800324 shutdown_connections 2024-03-18T16:06:50.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.032+0000 7fc22b200640 1 -- 172.21.15.39:0/966800324 wait complete. 2024-03-18T16:06:50.056 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:06:50.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.278+0000 7fede5a00640 1 -- 172.21.15.39:0/867510991 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fede0103a00 msgr2=0x7fede0103e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:50.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.278+0000 7fede5a00640 1 --2- 172.21.15.39:0/867510991 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fede0103a00 0x7fede0103e20 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fedd0009900 tx=0x7fedd002bc40 comp rx=0 tx=0).stop 2024-03-18T16:06:50.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.278+0000 7fede5a00640 1 -- 172.21.15.39:0/867510991 shutdown_connections 2024-03-18T16:06:50.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.278+0000 7fede5a00640 1 --2- 172.21.15.39:0/867510991 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fede01055b0 0x7fede010be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:50.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.278+0000 7fede5a00640 1 --2- 172.21.15.39:0/867510991 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fede0104bf0 0x7fede0105070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:50.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.278+0000 7fede5a00640 1 --2- 172.21.15.39:0/867510991 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fede0103a00 0x7fede0103e20 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:50.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.278+0000 7fede5a00640 1 -- 172.21.15.39:0/867510991 >> 172.21.15.39:0/867510991 conn(0x7fede00fef90 msgr2=0x7fede01013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:50.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.278+0000 7fede5a00640 1 -- 172.21.15.39:0/867510991 shutdown_connections 2024-03-18T16:06:50.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.278+0000 7fede5a00640 1 -- 172.21.15.39:0/867510991 wait complete. 2024-03-18T16:06:50.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.279+0000 7fede5a00640 1 Processor -- start 2024-03-18T16:06:50.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.280+0000 7fede5a00640 1 -- start start 2024-03-18T16:06:50.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.280+0000 7fede5a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fede0103a00 0x7fede0199d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:50.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.280+0000 7fede5a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fede0104bf0 0x7fede019a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:50.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.280+0000 7fede5a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fede01055b0 0x7fede01a1280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:50.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.280+0000 7fede5a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fede010f6a0 con 0x7fede01055b0 2024-03-18T16:06:50.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.280+0000 7fede5a00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fede010f520 con 0x7fede0103a00 2024-03-18T16:06:50.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.280+0000 7fede5a00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fede010f820 con 0x7fede0104bf0 2024-03-18T16:06:50.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.281+0000 7feddea00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fede0103a00 0x7fede0199d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:50.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.281+0000 7feddea00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fede0103a00 0x7fede0199d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:36798/0 (socket says 172.21.15.39:36798) 2024-03-18T16:06:50.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.281+0000 7feddea00640 1 -- 172.21.15.39:0/2232133980 learned_addr learned my addr 172.21.15.39:0/2232133980 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:06:50.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.281+0000 7fedd7e00640 1 --2- 172.21.15.39:0/2232133980 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fede0104bf0 0x7fede019a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:50.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.281+0000 7feddf400640 1 --2- 172.21.15.39:0/2232133980 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fede01055b0 0x7fede01a1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:50.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.281+0000 7feddea00640 1 -- 172.21.15.39:0/2232133980 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fede0104bf0 msgr2=0x7fede019a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:50.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.281+0000 7feddea00640 1 --2- 172.21.15.39:0/2232133980 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fede0104bf0 0x7fede019a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:50.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.281+0000 7feddea00640 1 -- 172.21.15.39:0/2232133980 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fede01055b0 msgr2=0x7fede01a1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:50.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.281+0000 7feddea00640 1 --2- 172.21.15.39:0/2232133980 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fede01055b0 0x7fede01a1280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:50.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.281+0000 7feddea00640 1 -- 172.21.15.39:0/2232133980 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fede01a1920 con 0x7fede0103a00 2024-03-18T16:06:50.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.281+0000 7feddea00640 1 --2- 172.21.15.39:0/2232133980 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fede0103a00 0x7fede0199d10 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fedd002efd0 tx=0x7fedd002c660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:50.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.281+0000 7fedd7400640 1 -- 172.21.15.39:0/2232133980 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedd0041070 con 0x7fede0103a00 2024-03-18T16:06:50.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.282+0000 7fedd7400640 1 -- 172.21.15.39:0/2232133980 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedd0004450 con 0x7fede0103a00 2024-03-18T16:06:50.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.282+0000 7fede5a00640 1 -- 172.21.15.39:0/2232133980 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fede01a1bb0 con 0x7fede0103a00 2024-03-18T16:06:50.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.282+0000 7fedd7400640 1 -- 172.21.15.39:0/2232133980 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedd0003e50 con 0x7fede0103a00 2024-03-18T16:06:50.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.282+0000 7fede5a00640 1 -- 172.21.15.39:0/2232133980 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fede01a2070 con 0x7fede0103a00 2024-03-18T16:06:50.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.283+0000 7fede5a00640 1 -- 172.21.15.39:0/2232133980 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fede010ad20 con 0x7fede0103a00 2024-03-18T16:06:50.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.286+0000 7fedd7400640 1 -- 172.21.15.39:0/2232133980 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fedd0044070 con 0x7fede0103a00 2024-03-18T16:06:50.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.288+0000 7fedd7400640 1 --2- 172.21.15.39:0/2232133980 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fedc40a29a0 0x7fedc40a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:50.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.288+0000 7fedd7400640 1 -- 172.21.15.39:0/2232133980 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fedd00f0900 con 0x7fede0103a00 2024-03-18T16:06:50.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.288+0000 7fedd7e00640 1 --2- 172.21.15.39:0/2232133980 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fedc40a29a0 0x7fedc40a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:50.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.289+0000 7fedd7e00640 1 --2- 172.21.15.39:0/2232133980 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fedc40a29a0 0x7fedc40a4e60 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fedc8009bd0 tx=0x7fedc8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:50.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.291+0000 7fedd7400640 1 -- 172.21.15.39:0/2232133980 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fedd00bd0b0 con 0x7fede0103a00 2024-03-18T16:06:50.305 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:50 smithi194 ceph-mon[36200]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-18T16:06:50.305 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:50 smithi194 ceph-mon[36200]: osdmap e88: 12 total, 12 up, 12 in 2024-03-18T16:06:50.305 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:50 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/966800324' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-18T16:06:50.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:50 smithi039 ceph-mon[31721]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-18T16:06:50.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:50 smithi039 ceph-mon[31721]: osdmap e88: 12 total, 12 up, 12 in 2024-03-18T16:06:50.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:50 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/966800324' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-18T16:06:50.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:50 smithi196 ceph-mon[36307]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-18T16:06:50.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:50 smithi196 ceph-mon[36307]: osdmap e88: 12 total, 12 up, 12 in 2024-03-18T16:06:50.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:50 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/966800324' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-18T16:06:50.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.534+0000 7fede5a00640 1 -- 172.21.15.39:0/2232133980 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fede0062410 con 0x7fede0103a00 2024-03-18T16:06:50.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.535+0000 7fedd7400640 1 -- 172.21.15.39:0/2232133980 <== mon.2 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+11797 (secure 0 0 0) 0x7fedd00c1150 con 0x7fede0103a00 2024-03-18T16:06:50.536 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:06:50.537 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2520226608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2520226608},{"type":"v1","addr":"172.21.15.39:6835","nonce":2520226608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/719341763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":719341763},{"type":"v1","addr":"172.21.15.39:6837","nonce":719341763}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14445,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6839/656513813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":656513813},{"type":"v1","addr":"172.21.15.39:6839","nonce":656513813}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14463,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6841/3515326042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3515326042},{"type":"v1","addr":"172.21.15.39:6841","nonce":3515326042}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/931350738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":931350738},{"type":"v1","addr":"172.21.15.194:6833","nonce":931350738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24463,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6833/3874209621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":3874209621},{"type":"v1","addr":"172.21.15.196:6833","nonce":3874209621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24494,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6835/188385470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":188385470},{"type":"v1","addr":"172.21.15.194:6835","nonce":188385470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24523,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6835/3949984309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":3949984309},{"type":"v1","addr":"172.21.15.196:6835","nonce":3949984309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6837/1301464246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":1301464246},{"type":"v1","addr":"172.21.15.194:6837","nonce":1301464246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24535,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6837/895247251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":895247251},{"type":"v1","addr":"172.21.15.196:6837","nonce":895247251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24556,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6839/2001427268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6838","nonce":2001427268},{"type":"v1","addr":"172.21.15.196:6839","nonce":2001427268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:06:44.180425+0000","modified":"2024-03-18T16:06:49.511040+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24575},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24575":{"gid":24575,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.194:6839/274911158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":274911158},{"type":"v1","addr":"172.21.15.194:6839","nonce":274911158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24575,"qdb_cluster":[24575]},"id":1}]} 2024-03-18T16:06:50.537 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 16 2024-03-18T16:06:50.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.537+0000 7fede5a00640 1 -- 172.21.15.39:0/2232133980 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fedc40a29a0 msgr2=0x7fedc40a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:50.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.537+0000 7fede5a00640 1 --2- 172.21.15.39:0/2232133980 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fedc40a29a0 0x7fedc40a4e60 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fedc8009bd0 tx=0x7fedc8005f90 comp rx=0 tx=0).stop 2024-03-18T16:06:50.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.537+0000 7fede5a00640 1 -- 172.21.15.39:0/2232133980 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fede0103a00 msgr2=0x7fede0199d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:50.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.537+0000 7fede5a00640 1 --2- 172.21.15.39:0/2232133980 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fede0103a00 0x7fede0199d10 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fedd002efd0 tx=0x7fedd002c660 comp rx=0 tx=0).stop 2024-03-18T16:06:50.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.537+0000 7fede5a00640 1 -- 172.21.15.39:0/2232133980 shutdown_connections 2024-03-18T16:06:50.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.537+0000 7fede5a00640 1 --2- 172.21.15.39:0/2232133980 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fedc40a29a0 0x7fedc40a4e60 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:50.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.537+0000 7fede5a00640 1 --2- 172.21.15.39:0/2232133980 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fede01055b0 0x7fede01a1280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:50.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.537+0000 7fede5a00640 1 --2- 172.21.15.39:0/2232133980 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fede0104bf0 0x7fede019a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:50.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.537+0000 7fede5a00640 1 --2- 172.21.15.39:0/2232133980 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fede0103a00 0x7fede0199d10 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:50.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.537+0000 7fede5a00640 1 -- 172.21.15.39:0/2232133980 >> 172.21.15.39:0/2232133980 conn(0x7fede00fef90 msgr2=0x7fede0107ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:50.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.538+0000 7fede5a00640 1 -- 172.21.15.39:0/2232133980 shutdown_connections 2024-03-18T16:06:50.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.538+0000 7fede5a00640 1 -- 172.21.15.39:0/2232133980 wait complete. 2024-03-18T16:06:50.557 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T16:06:50.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.777+0000 7f89e6400640 1 -- 172.21.15.39:0/3321470525 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f89e01072d0 msgr2=0x7f89e0109720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:50.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.777+0000 7f89e6400640 1 --2- 172.21.15.39:0/3321470525 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f89e01072d0 0x7f89e0109720 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f89d8009be0 tx=0x7f89d802bef0 comp rx=0 tx=0).stop 2024-03-18T16:06:50.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.777+0000 7f89e6400640 1 -- 172.21.15.39:0/3321470525 shutdown_connections 2024-03-18T16:06:50.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.777+0000 7f89e6400640 1 --2- 172.21.15.39:0/3321470525 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f89e01072d0 0x7f89e0109720 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:50.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.777+0000 7f89e6400640 1 --2- 172.21.15.39:0/3321470525 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f89e01049a0 0x7f89e0106d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:50.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.777+0000 7f89e6400640 1 --2- 172.21.15.39:0/3321470525 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f89e0102070 0x7f89e0104460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:50.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.777+0000 7f89e6400640 1 -- 172.21.15.39:0/3321470525 >> 172.21.15.39:0/3321470525 conn(0x7f89e00fba60 msgr2=0x7f89e00fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:50.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.777+0000 7f89e6400640 1 -- 172.21.15.39:0/3321470525 shutdown_connections 2024-03-18T16:06:50.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.777+0000 7f89e6400640 1 -- 172.21.15.39:0/3321470525 wait complete. 2024-03-18T16:06:50.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.778+0000 7f89e6400640 1 Processor -- start 2024-03-18T16:06:50.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.778+0000 7f89e6400640 1 -- start start 2024-03-18T16:06:50.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.778+0000 7f89e6400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f89e0102070 0x7f89e01019e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:50.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.778+0000 7f89e6400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f89e01049a0 0x7f89e0100020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:50.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.778+0000 7f89e6400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f89e01072d0 0x7f89e0100560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:50.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.778+0000 7f89e6400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f89e010d280 con 0x7f89e0102070 2024-03-18T16:06:50.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.778+0000 7f89e6400640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f89e010d100 con 0x7f89e01049a0 2024-03-18T16:06:50.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.778+0000 7f89e6400640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f89e010d400 con 0x7f89e01072d0 2024-03-18T16:06:50.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.779+0000 7f89e5000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f89e0102070 0x7f89e01019e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:50.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.779+0000 7f89d7e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f89e01049a0 0x7f89e0100020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:50.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.779+0000 7f89e5000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f89e0102070 0x7f89e01019e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43444/0 (socket says 172.21.15.39:43444) 2024-03-18T16:06:50.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.779+0000 7f89e5000640 1 -- 172.21.15.39:0/1910740187 learned_addr learned my addr 172.21.15.39:0/1910740187 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:06:50.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.779+0000 7f89e5a00640 1 --2- 172.21.15.39:0/1910740187 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f89e01072d0 0x7f89e0100560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:50.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.779+0000 7f89e5000640 1 -- 172.21.15.39:0/1910740187 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f89e01072d0 msgr2=0x7f89e0100560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:50.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.779+0000 7f89e5000640 1 --2- 172.21.15.39:0/1910740187 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f89e01072d0 0x7f89e0100560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:50.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.779+0000 7f89e5000640 1 -- 172.21.15.39:0/1910740187 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f89e01049a0 msgr2=0x7f89e0100020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:50.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.779+0000 7f89e5000640 1 --2- 172.21.15.39:0/1910740187 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f89e01049a0 0x7f89e0100020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:50.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.779+0000 7f89e5000640 1 -- 172.21.15.39:0/1910740187 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89e0100df0 con 0x7f89e0102070 2024-03-18T16:06:50.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.779+0000 7f89d7e00640 1 --2- 172.21.15.39:0/1910740187 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f89e01049a0 0x7f89e0100020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:06:50.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.779+0000 7f89e5000640 1 --2- 172.21.15.39:0/1910740187 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f89e0102070 0x7f89e01019e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f89d000e9f0 tx=0x7f89d000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:50.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.780+0000 7f89de000640 1 -- 172.21.15.39:0/1910740187 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d000cd80 con 0x7f89e0102070 2024-03-18T16:06:50.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.780+0000 7f89de000640 1 -- 172.21.15.39:0/1910740187 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89d0005500 con 0x7f89e0102070 2024-03-18T16:06:50.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.780+0000 7f89e6400640 1 -- 172.21.15.39:0/1910740187 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89e01a1980 con 0x7f89e0102070 2024-03-18T16:06:50.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.780+0000 7f89de000640 1 -- 172.21.15.39:0/1910740187 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89d0010690 con 0x7f89e0102070 2024-03-18T16:06:50.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.780+0000 7f89e6400640 1 -- 172.21.15.39:0/1910740187 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89e01a1e80 con 0x7f89e0102070 2024-03-18T16:06:50.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.781+0000 7f89e6400640 1 -- 172.21.15.39:0/1910740187 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89e00fd550 con 0x7f89e0102070 2024-03-18T16:06:50.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.783+0000 7f89de000640 1 -- 172.21.15.39:0/1910740187 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f89d00046c0 con 0x7f89e0102070 2024-03-18T16:06:50.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.784+0000 7f89de000640 1 --2- 172.21.15.39:0/1910740187 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f89c40a2800 0x7f89c40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:50.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.785+0000 7f89de000640 1 -- 172.21.15.39:0/1910740187 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f89d00cf300 con 0x7f89e0102070 2024-03-18T16:06:50.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.785+0000 7f89d7e00640 1 --2- 172.21.15.39:0/1910740187 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f89c40a2800 0x7f89c40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:50.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.785+0000 7f89d7e00640 1 --2- 172.21.15.39:0/1910740187 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f89c40a2800 0x7f89c40a4cc0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f89c80096c0 tx=0x7f89c8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:50.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.787+0000 7f89de000640 1 -- 172.21.15.39:0/1910740187 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f89d0053740 con 0x7f89e0102070 2024-03-18T16:06:50.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.990+0000 7f89e6400640 1 -- 172.21.15.39:0/1910740187 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f89e002d380 con 0x7f89e0102070 2024-03-18T16:06:50.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.993+0000 7f89de000640 1 -- 172.21.15.39:0/1910740187 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24142 (secure 0 0 0) 0x7f89d009c160 con 0x7f89e0102070 2024-03-18T16:06:50.995 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:06:50.996 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":89,"fsid":"1892d8fc-e540-11ee-95c9-87774f69a715","created":"2024-03-18T15:57:36.507008+0000","modified":"2024-03-18T16:06:49.510653+0000","last_up_change":"2024-03-18T16:05:45.250629+0000","last_in_change":"2024-03-18T16:05:24.841337+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T16:01:30.048247+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T16:06:42.191791+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T16:06:43.670325+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T16:06:47.523388+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"771e0558-be62-4436-94b0-6364751911c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6803","nonce":4090743635}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6805","nonce":4090743635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6809","nonce":4090743635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6807","nonce":4090743635}]},"public_addr":"172.21.15.39:6803/4090743635","cluster_addr":"172.21.15.39:6805/4090743635","heartbeat_back_addr":"172.21.15.39:6809/4090743635","heartbeat_front_addr":"172.21.15.39:6807/4090743635","state":["exists","up"]},{"osd":1,"uuid":"0430eee1-cf34-4277-b255-c7d9f07ad19a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6801","nonce":2800329805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6803","nonce":2800329805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6807","nonce":2800329805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6805","nonce":2800329805}]},"public_addr":"172.21.15.194:6801/2800329805","cluster_addr":"172.21.15.194:6803/2800329805","heartbeat_back_addr":"172.21.15.194:6807/2800329805","heartbeat_front_addr":"172.21.15.194:6805/2800329805","state":["exists","up"]},{"osd":2,"uuid":"3a718da4-2511-4ec1-b1a8-eafd6cce0df0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6801","nonce":3935874719}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6803","nonce":3935874719}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6807","nonce":3935874719}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6805","nonce":3935874719}]},"public_addr":"172.21.15.196:6801/3935874719","cluster_addr":"172.21.15.196:6803/3935874719","heartbeat_back_addr":"172.21.15.196:6807/3935874719","heartbeat_front_addr":"172.21.15.196:6805/3935874719","state":["exists","up"]},{"osd":3,"uuid":"6f9c37f9-0ab9-4038-8919-afd4b39c4165","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6811","nonce":893345939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6813","nonce":893345939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6817","nonce":893345939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6815","nonce":893345939}]},"public_addr":"172.21.15.39:6811/893345939","cluster_addr":"172.21.15.39:6813/893345939","heartbeat_back_addr":"172.21.15.39:6817/893345939","heartbeat_front_addr":"172.21.15.39:6815/893345939","state":["exists","up"]},{"osd":4,"uuid":"e795bb0e-4791-4537-acda-1a321e19bc9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6809","nonce":800423088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6811","nonce":800423088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6815","nonce":800423088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6813","nonce":800423088}]},"public_addr":"172.21.15.194:6809/800423088","cluster_addr":"172.21.15.194:6811/800423088","heartbeat_back_addr":"172.21.15.194:6815/800423088","heartbeat_front_addr":"172.21.15.194:6813/800423088","state":["exists","up"]},{"osd":5,"uuid":"026e175e-6acb-4b4c-b235-8ccee8ae1149","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6809","nonce":2746224525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6811","nonce":2746224525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6815","nonce":2746224525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6813","nonce":2746224525}]},"public_addr":"172.21.15.196:6809/2746224525","cluster_addr":"172.21.15.196:6811/2746224525","heartbeat_back_addr":"172.21.15.196:6815/2746224525","heartbeat_front_addr":"172.21.15.196:6813/2746224525","state":["exists","up"]},{"osd":6,"uuid":"c1b183ff-383f-4c5c-9075-f76bab1c0bb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6819","nonce":1213371331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6821","nonce":1213371331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6825","nonce":1213371331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6823","nonce":1213371331}]},"public_addr":"172.21.15.39:6819/1213371331","cluster_addr":"172.21.15.39:6821/1213371331","heartbeat_back_addr":"172.21.15.39:6825/1213371331","heartbeat_front_addr":"172.21.15.39:6823/1213371331","state":["exists","up"]},{"osd":7,"uuid":"612f8697-957d-4ef4-a87f-241b36f9d13d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6817","nonce":3299162444}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6819","nonce":3299162444}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6823","nonce":3299162444}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6821","nonce":3299162444}]},"public_addr":"172.21.15.194:6817/3299162444","cluster_addr":"172.21.15.194:6819/3299162444","heartbeat_back_addr":"172.21.15.194:6823/3299162444","heartbeat_front_addr":"172.21.15.194:6821/3299162444","state":["exists","up"]},{"osd":8,"uuid":"836a9107-4d4f-4594-b8dc-0301e64f59d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6817","nonce":2456143145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6819","nonce":2456143145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6823","nonce":2456143145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6821","nonce":2456143145}]},"public_addr":"172.21.15.196:6817/2456143145","cluster_addr":"172.21.15.196:6819/2456143145","heartbeat_back_addr":"172.21.15.196:6823/2456143145","heartbeat_front_addr":"172.21.15.196:6821/2456143145","state":["exists","up"]},{"osd":9,"uuid":"4f30b960-12e3-4b16-bd28-fcc988139b58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6827","nonce":1848634433}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6829","nonce":1848634433}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6833","nonce":1848634433}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6831","nonce":1848634433}]},"public_addr":"172.21.15.39:6827/1848634433","cluster_addr":"172.21.15.39:6829/1848634433","heartbeat_back_addr":"172.21.15.39:6833/1848634433","heartbeat_front_addr":"172.21.15.39:6831/1848634433","state":["exists","up"]},{"osd":10,"uuid":"620f90e8-23e6-4bc0-8269-7219b9e1aebd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6825","nonce":3725064809}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6827","nonce":3725064809}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6831","nonce":3725064809}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6829","nonce":3725064809}]},"public_addr":"172.21.15.194:6825/3725064809","cluster_addr":"172.21.15.194:6827/3725064809","heartbeat_back_addr":"172.21.15.194:6831/3725064809","heartbeat_front_addr":"172.21.15.194:6829/3725064809","state":["exists","up"]},{"osd":11,"uuid":"867a2535-8235-4ceb-a698-eb86869d6d20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6825","nonce":2501212422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6827","nonce":2501212422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6831","nonce":2501212422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6829","nonce":2501212422}]},"public_addr":"172.21.15.196:6825/2501212422","cluster_addr":"172.21.15.196:6827/2501212422","heartbeat_back_addr":"172.21.15.196:6831/2501212422","heartbeat_front_addr":"172.21.15.196:6829/2501212422","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:00:36.931566+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:01:02.614024+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:01:26.193265+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:01:58.308833+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:02:25.049546+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:02:49.748750+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:03:23.582153+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:03:50.447321+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:04:15.812129+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:04:49.998047+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:05:17.087924+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:05:43.167637+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.39:0/3482086140":"2024-03-19T15:58:45.340294+0000","172.21.15.39:0/3844306284":"2024-03-19T15:58:45.340294+0000","172.21.15.39:6801/1810238546":"2024-03-19T15:58:45.340294+0000","172.21.15.39:6800/1810238546":"2024-03-19T15:58:45.340294+0000","172.21.15.39:0/2528397488":"2024-03-19T15:58:10.264798+0000","172.21.15.39:0/2240981254":"2024-03-19T15:58:10.264798+0000","172.21.15.39:6800/1217503189":"2024-03-19T15:58:10.264798+0000","172.21.15.39:0/3754876370":"2024-03-19T15:58:10.264798+0000","172.21.15.39:6801/1217503189":"2024-03-19T15:58:10.264798+0000","172.21.15.39:0/1874637628":"2024-03-19T15:58:45.340294+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T16:06:50.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.996+0000 7f89e6400640 1 -- 172.21.15.39:0/1910740187 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f89c40a2800 msgr2=0x7f89c40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:50.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.996+0000 7f89e6400640 1 --2- 172.21.15.39:0/1910740187 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f89c40a2800 0x7f89c40a4cc0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f89c80096c0 tx=0x7f89c8009260 comp rx=0 tx=0).stop 2024-03-18T16:06:50.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.996+0000 7f89e6400640 1 -- 172.21.15.39:0/1910740187 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f89e0102070 msgr2=0x7f89e01019e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:50.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.996+0000 7f89e6400640 1 --2- 172.21.15.39:0/1910740187 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f89e0102070 0x7f89e01019e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f89d000e9f0 tx=0x7f89d000edf0 comp rx=0 tx=0).stop 2024-03-18T16:06:50.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.996+0000 7f89e6400640 1 -- 172.21.15.39:0/1910740187 shutdown_connections 2024-03-18T16:06:50.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.996+0000 7f89e6400640 1 --2- 172.21.15.39:0/1910740187 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f89e01049a0 0x7f89e0100020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:50.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.996+0000 7f89e6400640 1 --2- 172.21.15.39:0/1910740187 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f89c40a2800 0x7f89c40a4cc0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:50.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.996+0000 7f89e6400640 1 --2- 172.21.15.39:0/1910740187 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f89e01072d0 0x7f89e0100560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:50.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.996+0000 7f89e6400640 1 --2- 172.21.15.39:0/1910740187 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f89e0102070 0x7f89e01019e0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:50.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.996+0000 7f89e6400640 1 -- 172.21.15.39:0/1910740187 >> 172.21.15.39:0/1910740187 conn(0x7f89e00fba60 msgr2=0x7f89e01029b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:50.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.997+0000 7f89e6400640 1 -- 172.21.15.39:0/1910740187 shutdown_connections 2024-03-18T16:06:50.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:50.997+0000 7f89e6400640 1 -- 172.21.15.39:0/1910740187 wait complete. 2024-03-18T16:06:51.018 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T16:06:51.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.236+0000 7f1cfba00640 1 -- 172.21.15.39:0/1628962654 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1cf4103a00 msgr2=0x7f1cf4103e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:51.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.236+0000 7f1cfba00640 1 --2- 172.21.15.39:0/1628962654 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1cf4103a00 0x7f1cf4103e20 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f1ce8009900 tx=0x7f1ce802bc30 comp rx=0 tx=0).stop 2024-03-18T16:06:51.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.236+0000 7f1cfba00640 1 -- 172.21.15.39:0/1628962654 shutdown_connections 2024-03-18T16:06:51.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.236+0000 7f1cfba00640 1 --2- 172.21.15.39:0/1628962654 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1cf41055b0 0x7f1cf410be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:51.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.236+0000 7f1cfba00640 1 --2- 172.21.15.39:0/1628962654 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1cf4104bf0 0x7f1cf4105070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:51.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.236+0000 7f1cfba00640 1 --2- 172.21.15.39:0/1628962654 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1cf4103a00 0x7f1cf4103e20 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:51.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.236+0000 7f1cfba00640 1 -- 172.21.15.39:0/1628962654 >> 172.21.15.39:0/1628962654 conn(0x7f1cf40fef90 msgr2=0x7f1cf41013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:51.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.236+0000 7f1cfba00640 1 -- 172.21.15.39:0/1628962654 shutdown_connections 2024-03-18T16:06:51.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.236+0000 7f1cfba00640 1 -- 172.21.15.39:0/1628962654 wait complete. 2024-03-18T16:06:51.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.237+0000 7f1cfba00640 1 Processor -- start 2024-03-18T16:06:51.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.237+0000 7f1cfba00640 1 -- start start 2024-03-18T16:06:51.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.237+0000 7f1cfba00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1cf4103a00 0x7f1cf4199d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:51.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.238+0000 7f1cfba00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1cf4104bf0 0x7f1cf419a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:51.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.238+0000 7f1cfba00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1cf41055b0 0x7f1cf41a1290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:51.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.238+0000 7f1cfba00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f1cf410f6c0 con 0x7f1cf4104bf0 2024-03-18T16:06:51.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.238+0000 7f1cfba00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f1cf410f540 con 0x7f1cf41055b0 2024-03-18T16:06:51.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.238+0000 7f1cfba00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f1cf410f840 con 0x7f1cf4103a00 2024-03-18T16:06:51.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.238+0000 7f1cf3e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1cf4104bf0 0x7f1cf419a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:51.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.238+0000 7f1cf3e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1cf4104bf0 0x7f1cf419a260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43462/0 (socket says 172.21.15.39:43462) 2024-03-18T16:06:51.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.238+0000 7f1cf3e00640 1 -- 172.21.15.39:0/3094248891 learned_addr learned my addr 172.21.15.39:0/3094248891 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:06:51.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.238+0000 7f1cf9a00640 1 --2- 172.21.15.39:0/3094248891 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1cf41055b0 0x7f1cf41a1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:51.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.238+0000 7f1cf9000640 1 --2- 172.21.15.39:0/3094248891 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1cf4103a00 0x7f1cf4199d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:51.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.238+0000 7f1cf3e00640 1 -- 172.21.15.39:0/3094248891 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1cf4103a00 msgr2=0x7f1cf4199d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:51.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.238+0000 7f1cf3e00640 1 --2- 172.21.15.39:0/3094248891 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1cf4103a00 0x7f1cf4199d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:51.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.238+0000 7f1cf3e00640 1 -- 172.21.15.39:0/3094248891 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1cf41055b0 msgr2=0x7f1cf41a1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:51.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.238+0000 7f1cf3e00640 1 --2- 172.21.15.39:0/3094248891 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1cf41055b0 0x7f1cf41a1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:51.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.238+0000 7f1cf3e00640 1 -- 172.21.15.39:0/3094248891 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cf41a1930 con 0x7f1cf4104bf0 2024-03-18T16:06:51.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.238+0000 7f1cf9a00640 1 --2- 172.21.15.39:0/3094248891 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1cf41055b0 0x7f1cf41a1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:06:51.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.239+0000 7f1cf3e00640 1 --2- 172.21.15.39:0/3094248891 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1cf4104bf0 0x7f1cf419a260 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f1ce000e9f0 tx=0x7f1ce000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:51.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.239+0000 7f1cf1600640 1 -- 172.21.15.39:0/3094248891 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ce000cd80 con 0x7f1cf4104bf0 2024-03-18T16:06:51.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.239+0000 7f1cfba00640 1 -- 172.21.15.39:0/3094248891 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1cf41a1c20 con 0x7f1cf4104bf0 2024-03-18T16:06:51.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.239+0000 7f1cf1600640 1 -- 172.21.15.39:0/3094248891 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ce0005500 con 0x7f1cf4104bf0 2024-03-18T16:06:51.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.239+0000 7f1cf1600640 1 -- 172.21.15.39:0/3094248891 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ce0010690 con 0x7f1cf4104bf0 2024-03-18T16:06:51.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.239+0000 7f1cfba00640 1 -- 172.21.15.39:0/3094248891 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1cf41a20f0 con 0x7f1cf4104bf0 2024-03-18T16:06:51.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.240+0000 7f1cfba00640 1 -- 172.21.15.39:0/3094248891 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1cf410ad20 con 0x7f1cf4104bf0 2024-03-18T16:06:51.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.243+0000 7f1cf1600640 1 -- 172.21.15.39:0/3094248891 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f1ce00046c0 con 0x7f1cf4104bf0 2024-03-18T16:06:51.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.244+0000 7f1cf1600640 1 --2- 172.21.15.39:0/3094248891 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1cdc0a2800 0x7f1cdc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:51.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.244+0000 7f1cf1600640 1 -- 172.21.15.39:0/3094248891 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f1ce00cf180 con 0x7f1cf4104bf0 2024-03-18T16:06:51.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.245+0000 7f1cf9000640 1 --2- 172.21.15.39:0/3094248891 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1cdc0a2800 0x7f1cdc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:51.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.245+0000 7f1cf9000640 1 --2- 172.21.15.39:0/3094248891 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1cdc0a2800 0x7f1cdc0a4cc0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f1ce802c170 tx=0x7f1ce802df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:51.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.249+0000 7f1cf1600640 1 -- 172.21.15.39:0/3094248891 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1ce009bf80 con 0x7f1cf4104bf0 2024-03-18T16:06:51.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:51 smithi039 ceph-mon[31721]: pgmap v320: 129 pgs: 6 creating+peering, 97 active+clean, 26 unknown; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s wr, 11 op/s 2024-03-18T16:06:51.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:51 smithi039 ceph-mon[31721]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-18T16:06:51.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:51 smithi039 ceph-mon[31721]: osdmap e89: 12 total, 12 up, 12 in 2024-03-18T16:06:51.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:51 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/966800324' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-18T16:06:51.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:51 smithi039 ceph-mon[31721]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-18T16:06:51.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:51 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/2232133980' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:06:51.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:51 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/1910740187' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:06:51.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.457+0000 7f1cfba00640 1 -- 172.21.15.39:0/3094248891 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1cf4062480 con 0x7f1cf4104bf0 2024-03-18T16:06:51.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.461+0000 7f1cf1600640 1 -- 172.21.15.39:0/3094248891 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24142 (secure 0 0 0) 0x7f1ce009b880 con 0x7f1cf4104bf0 2024-03-18T16:06:51.463 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:06:51.463 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":89,"fsid":"1892d8fc-e540-11ee-95c9-87774f69a715","created":"2024-03-18T15:57:36.507008+0000","modified":"2024-03-18T16:06:49.510653+0000","last_up_change":"2024-03-18T16:05:45.250629+0000","last_in_change":"2024-03-18T16:05:24.841337+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T16:01:30.048247+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T16:06:42.191791+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T16:06:43.670325+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T16:06:47.523388+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"771e0558-be62-4436-94b0-6364751911c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6803","nonce":4090743635}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6805","nonce":4090743635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6809","nonce":4090743635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6807","nonce":4090743635}]},"public_addr":"172.21.15.39:6803/4090743635","cluster_addr":"172.21.15.39:6805/4090743635","heartbeat_back_addr":"172.21.15.39:6809/4090743635","heartbeat_front_addr":"172.21.15.39:6807/4090743635","state":["exists","up"]},{"osd":1,"uuid":"0430eee1-cf34-4277-b255-c7d9f07ad19a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6801","nonce":2800329805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6803","nonce":2800329805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6807","nonce":2800329805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6805","nonce":2800329805}]},"public_addr":"172.21.15.194:6801/2800329805","cluster_addr":"172.21.15.194:6803/2800329805","heartbeat_back_addr":"172.21.15.194:6807/2800329805","heartbeat_front_addr":"172.21.15.194:6805/2800329805","state":["exists","up"]},{"osd":2,"uuid":"3a718da4-2511-4ec1-b1a8-eafd6cce0df0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6801","nonce":3935874719}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6803","nonce":3935874719}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6807","nonce":3935874719}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6805","nonce":3935874719}]},"public_addr":"172.21.15.196:6801/3935874719","cluster_addr":"172.21.15.196:6803/3935874719","heartbeat_back_addr":"172.21.15.196:6807/3935874719","heartbeat_front_addr":"172.21.15.196:6805/3935874719","state":["exists","up"]},{"osd":3,"uuid":"6f9c37f9-0ab9-4038-8919-afd4b39c4165","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6811","nonce":893345939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6813","nonce":893345939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6817","nonce":893345939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6815","nonce":893345939}]},"public_addr":"172.21.15.39:6811/893345939","cluster_addr":"172.21.15.39:6813/893345939","heartbeat_back_addr":"172.21.15.39:6817/893345939","heartbeat_front_addr":"172.21.15.39:6815/893345939","state":["exists","up"]},{"osd":4,"uuid":"e795bb0e-4791-4537-acda-1a321e19bc9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6809","nonce":800423088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6811","nonce":800423088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6815","nonce":800423088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6813","nonce":800423088}]},"public_addr":"172.21.15.194:6809/800423088","cluster_addr":"172.21.15.194:6811/800423088","heartbeat_back_addr":"172.21.15.194:6815/800423088","heartbeat_front_addr":"172.21.15.194:6813/800423088","state":["exists","up"]},{"osd":5,"uuid":"026e175e-6acb-4b4c-b235-8ccee8ae1149","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6809","nonce":2746224525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6811","nonce":2746224525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6815","nonce":2746224525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6813","nonce":2746224525}]},"public_addr":"172.21.15.196:6809/2746224525","cluster_addr":"172.21.15.196:6811/2746224525","heartbeat_back_addr":"172.21.15.196:6815/2746224525","heartbeat_front_addr":"172.21.15.196:6813/2746224525","state":["exists","up"]},{"osd":6,"uuid":"c1b183ff-383f-4c5c-9075-f76bab1c0bb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6819","nonce":1213371331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6821","nonce":1213371331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6825","nonce":1213371331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6823","nonce":1213371331}]},"public_addr":"172.21.15.39:6819/1213371331","cluster_addr":"172.21.15.39:6821/1213371331","heartbeat_back_addr":"172.21.15.39:6825/1213371331","heartbeat_front_addr":"172.21.15.39:6823/1213371331","state":["exists","up"]},{"osd":7,"uuid":"612f8697-957d-4ef4-a87f-241b36f9d13d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6817","nonce":3299162444}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6819","nonce":3299162444}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6823","nonce":3299162444}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6821","nonce":3299162444}]},"public_addr":"172.21.15.194:6817/3299162444","cluster_addr":"172.21.15.194:6819/3299162444","heartbeat_back_addr":"172.21.15.194:6823/3299162444","heartbeat_front_addr":"172.21.15.194:6821/3299162444","state":["exists","up"]},{"osd":8,"uuid":"836a9107-4d4f-4594-b8dc-0301e64f59d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6817","nonce":2456143145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6819","nonce":2456143145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6823","nonce":2456143145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6821","nonce":2456143145}]},"public_addr":"172.21.15.196:6817/2456143145","cluster_addr":"172.21.15.196:6819/2456143145","heartbeat_back_addr":"172.21.15.196:6823/2456143145","heartbeat_front_addr":"172.21.15.196:6821/2456143145","state":["exists","up"]},{"osd":9,"uuid":"4f30b960-12e3-4b16-bd28-fcc988139b58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6827","nonce":1848634433}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6829","nonce":1848634433}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6833","nonce":1848634433}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6831","nonce":1848634433}]},"public_addr":"172.21.15.39:6827/1848634433","cluster_addr":"172.21.15.39:6829/1848634433","heartbeat_back_addr":"172.21.15.39:6833/1848634433","heartbeat_front_addr":"172.21.15.39:6831/1848634433","state":["exists","up"]},{"osd":10,"uuid":"620f90e8-23e6-4bc0-8269-7219b9e1aebd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6825","nonce":3725064809}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6827","nonce":3725064809}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6831","nonce":3725064809}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6829","nonce":3725064809}]},"public_addr":"172.21.15.194:6825/3725064809","cluster_addr":"172.21.15.194:6827/3725064809","heartbeat_back_addr":"172.21.15.194:6831/3725064809","heartbeat_front_addr":"172.21.15.194:6829/3725064809","state":["exists","up"]},{"osd":11,"uuid":"867a2535-8235-4ceb-a698-eb86869d6d20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6825","nonce":2501212422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6827","nonce":2501212422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6831","nonce":2501212422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6829","nonce":2501212422}]},"public_addr":"172.21.15.196:6825/2501212422","cluster_addr":"172.21.15.196:6827/2501212422","heartbeat_back_addr":"172.21.15.196:6831/2501212422","heartbeat_front_addr":"172.21.15.196:6829/2501212422","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:00:36.931566+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:01:02.614024+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:01:26.193265+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:01:58.308833+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:02:25.049546+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:02:49.748750+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:03:23.582153+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:03:50.447321+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:04:15.812129+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:04:49.998047+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:05:17.087924+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:05:43.167637+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.39:0/3482086140":"2024-03-19T15:58:45.340294+0000","172.21.15.39:0/3844306284":"2024-03-19T15:58:45.340294+0000","172.21.15.39:6801/1810238546":"2024-03-19T15:58:45.340294+0000","172.21.15.39:6800/1810238546":"2024-03-19T15:58:45.340294+0000","172.21.15.39:0/2528397488":"2024-03-19T15:58:10.264798+0000","172.21.15.39:0/2240981254":"2024-03-19T15:58:10.264798+0000","172.21.15.39:6800/1217503189":"2024-03-19T15:58:10.264798+0000","172.21.15.39:0/3754876370":"2024-03-19T15:58:10.264798+0000","172.21.15.39:6801/1217503189":"2024-03-19T15:58:10.264798+0000","172.21.15.39:0/1874637628":"2024-03-19T15:58:45.340294+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T16:06:51.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.464+0000 7f1cfba00640 1 -- 172.21.15.39:0/3094248891 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1cdc0a2800 msgr2=0x7f1cdc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:51.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.464+0000 7f1cfba00640 1 --2- 172.21.15.39:0/3094248891 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1cdc0a2800 0x7f1cdc0a4cc0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f1ce802c170 tx=0x7f1ce802df90 comp rx=0 tx=0).stop 2024-03-18T16:06:51.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.464+0000 7f1cfba00640 1 -- 172.21.15.39:0/3094248891 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1cf4104bf0 msgr2=0x7f1cf419a260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:51.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.464+0000 7f1cfba00640 1 --2- 172.21.15.39:0/3094248891 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1cf4104bf0 0x7f1cf419a260 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f1ce000e9f0 tx=0x7f1ce000edf0 comp rx=0 tx=0).stop 2024-03-18T16:06:51.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.464+0000 7f1cfba00640 1 -- 172.21.15.39:0/3094248891 shutdown_connections 2024-03-18T16:06:51.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.464+0000 7f1cfba00640 1 --2- 172.21.15.39:0/3094248891 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1cf41055b0 0x7f1cf41a1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:51.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.464+0000 7f1cfba00640 1 --2- 172.21.15.39:0/3094248891 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1cdc0a2800 0x7f1cdc0a4cc0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:51.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.464+0000 7f1cfba00640 1 --2- 172.21.15.39:0/3094248891 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1cf4104bf0 0x7f1cf419a260 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:51.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.464+0000 7f1cfba00640 1 --2- 172.21.15.39:0/3094248891 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1cf4103a00 0x7f1cf4199d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:51.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.464+0000 7f1cfba00640 1 -- 172.21.15.39:0/3094248891 >> 172.21.15.39:0/3094248891 conn(0x7f1cf40fef90 msgr2=0x7f1cf4107ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:51.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.464+0000 7f1cfba00640 1 -- 172.21.15.39:0/3094248891 shutdown_connections 2024-03-18T16:06:51.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.464+0000 7f1cfba00640 1 -- 172.21.15.39:0/3094248891 wait complete. 2024-03-18T16:06:51.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:51 smithi196 ceph-mon[36307]: pgmap v320: 129 pgs: 6 creating+peering, 97 active+clean, 26 unknown; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s wr, 11 op/s 2024-03-18T16:06:51.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:51 smithi196 ceph-mon[36307]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-18T16:06:51.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:51 smithi196 ceph-mon[36307]: osdmap e89: 12 total, 12 up, 12 in 2024-03-18T16:06:51.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:51 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/966800324' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-18T16:06:51.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:51 smithi196 ceph-mon[36307]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-18T16:06:51.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:51 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/2232133980' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:06:51.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:51 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/1910740187' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:06:51.503 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:06:51.504 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:51 smithi194 ceph-mon[36200]: pgmap v320: 129 pgs: 6 creating+peering, 97 active+clean, 26 unknown; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s wr, 11 op/s 2024-03-18T16:06:51.504 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:51 smithi194 ceph-mon[36200]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-18T16:06:51.504 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:51 smithi194 ceph-mon[36200]: osdmap e89: 12 total, 12 up, 12 in 2024-03-18T16:06:51.504 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:51 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/966800324' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-18T16:06:51.505 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:51 smithi194 ceph-mon[36200]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-18T16:06:51.505 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:51 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/2232133980' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:06:51.505 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:51 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/1910740187' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:06:51.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.728+0000 7fddcca00640 1 -- 172.21.15.39:0/63128298 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fddc81053c0 msgr2=0x7fddc810bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:51.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.728+0000 7fddcca00640 1 --2- 172.21.15.39:0/63128298 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fddc81053c0 0x7fddc810bc50 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fddc0009900 tx=0x7fddc002bc50 comp rx=0 tx=0).stop 2024-03-18T16:06:51.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.729+0000 7fddcca00640 1 -- 172.21.15.39:0/63128298 shutdown_connections 2024-03-18T16:06:51.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.729+0000 7fddcca00640 1 --2- 172.21.15.39:0/63128298 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fddc81053c0 0x7fddc810bc50 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:51.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.729+0000 7fddcca00640 1 --2- 172.21.15.39:0/63128298 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fddc8104a00 0x7fddc8104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:51.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.729+0000 7fddcca00640 1 --2- 172.21.15.39:0/63128298 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fddc8103810 0x7fddc8103c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:51.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.729+0000 7fddcca00640 1 -- 172.21.15.39:0/63128298 >> 172.21.15.39:0/63128298 conn(0x7fddc80fed80 msgr2=0x7fddc81011c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:51.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.729+0000 7fddcca00640 1 -- 172.21.15.39:0/63128298 shutdown_connections 2024-03-18T16:06:51.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.729+0000 7fddcca00640 1 -- 172.21.15.39:0/63128298 wait complete. 2024-03-18T16:06:51.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.730+0000 7fddcca00640 1 Processor -- start 2024-03-18T16:06:51.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.730+0000 7fddcca00640 1 -- start start 2024-03-18T16:06:51.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.730+0000 7fddcca00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fddc8103810 0x7fddc8199a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:51.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.730+0000 7fddcca00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fddc8104a00 0x7fddc8199fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:51.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.730+0000 7fddcca00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fddc81053c0 0x7fddc81a1030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:51.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.730+0000 7fddcca00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fddc810f4c0 con 0x7fddc8104a00 2024-03-18T16:06:51.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.730+0000 7fddcca00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fddc810f340 con 0x7fddc8103810 2024-03-18T16:06:51.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.730+0000 7fddcca00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fddc810f640 con 0x7fddc81053c0 2024-03-18T16:06:51.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.730+0000 7fddbea00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fddc8104a00 0x7fddc8199fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:51.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.730+0000 7fddc7400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fddc8103810 0x7fddc8199a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:51.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.730+0000 7fddbea00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fddc8104a00 0x7fddc8199fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43484/0 (socket says 172.21.15.39:43484) 2024-03-18T16:06:51.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.730+0000 7fddbea00640 1 -- 172.21.15.39:0/285802235 learned_addr learned my addr 172.21.15.39:0/285802235 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:06:51.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.730+0000 7fddc7e00640 1 --2- 172.21.15.39:0/285802235 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fddc81053c0 0x7fddc81a1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:51.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.731+0000 7fddc7400640 1 -- 172.21.15.39:0/285802235 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fddc81053c0 msgr2=0x7fddc81a1030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:51.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.731+0000 7fddc7400640 1 --2- 172.21.15.39:0/285802235 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fddc81053c0 0x7fddc81a1030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:51.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.731+0000 7fddc7400640 1 -- 172.21.15.39:0/285802235 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fddc8104a00 msgr2=0x7fddc8199fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:51.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.731+0000 7fddc7400640 1 --2- 172.21.15.39:0/285802235 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fddc8104a00 0x7fddc8199fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:51.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.731+0000 7fddc7400640 1 -- 172.21.15.39:0/285802235 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddc81a16a0 con 0x7fddc8103810 2024-03-18T16:06:51.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.731+0000 7fddbea00640 1 --2- 172.21.15.39:0/285802235 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fddc8104a00 0x7fddc8199fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:06:51.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.731+0000 7fddc7400640 1 --2- 172.21.15.39:0/285802235 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fddc8103810 0x7fddc8199a80 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fddb8005fd0 tx=0x7fddb800ef90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:51.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.731+0000 7fddc4c00640 1 -- 172.21.15.39:0/285802235 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddb8003970 con 0x7fddc8103810 2024-03-18T16:06:51.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.731+0000 7fddc4c00640 1 -- 172.21.15.39:0/285802235 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fddb8004030 con 0x7fddc8103810 2024-03-18T16:06:51.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.731+0000 7fddc4c00640 1 -- 172.21.15.39:0/285802235 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fddb80107f0 con 0x7fddc8103810 2024-03-18T16:06:51.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.731+0000 7fddcca00640 1 -- 172.21.15.39:0/285802235 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fddc81a1990 con 0x7fddc8103810 2024-03-18T16:06:51.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.731+0000 7fddcca00640 1 -- 172.21.15.39:0/285802235 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fddc81a1e90 con 0x7fddc8103810 2024-03-18T16:06:51.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.732+0000 7fddcca00640 1 -- 172.21.15.39:0/285802235 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fddc8068940 con 0x7fddc8103810 2024-03-18T16:06:51.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.734+0000 7fddc4c00640 1 -- 172.21.15.39:0/285802235 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fddb80041e0 con 0x7fddc8103810 2024-03-18T16:06:51.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.736+0000 7fddc4c00640 1 --2- 172.21.15.39:0/285802235 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fddac0a2880 0x7fddac0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:51.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.736+0000 7fddc4c00640 1 -- 172.21.15.39:0/285802235 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fddb80cf440 con 0x7fddc8103810 2024-03-18T16:06:51.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.736+0000 7fddbea00640 1 --2- 172.21.15.39:0/285802235 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fddac0a2880 0x7fddac0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:51.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.736+0000 7fddbea00640 1 --2- 172.21.15.39:0/285802235 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fddac0a2880 0x7fddac0a4d40 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fddb00095f0 tx=0x7fddb0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:51.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.740+0000 7fddc4c00640 1 -- 172.21.15.39:0/285802235 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fddb809c2a0 con 0x7fddc8103810 2024-03-18T16:06:51.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.982+0000 7fddcca00640 1 -- 172.21.15.39:0/285802235 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fddc810ab20 con 0x7fddc8103810 2024-03-18T16:06:51.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.984+0000 7fddc4c00640 1 -- 172.21.15.39:0/285802235 <== mon.2 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+11797 (secure 0 0 0) 0x7fddc810ab20 con 0x7fddc8103810 2024-03-18T16:06:51.985 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:06:51.985 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2520226608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2520226608},{"type":"v1","addr":"172.21.15.39:6835","nonce":2520226608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/719341763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":719341763},{"type":"v1","addr":"172.21.15.39:6837","nonce":719341763}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14445,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6839/656513813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":656513813},{"type":"v1","addr":"172.21.15.39:6839","nonce":656513813}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14463,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6841/3515326042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3515326042},{"type":"v1","addr":"172.21.15.39:6841","nonce":3515326042}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24446,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/931350738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":931350738},{"type":"v1","addr":"172.21.15.194:6833","nonce":931350738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24463,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6833/3874209621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":3874209621},{"type":"v1","addr":"172.21.15.196:6833","nonce":3874209621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24494,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6835/188385470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":188385470},{"type":"v1","addr":"172.21.15.194:6835","nonce":188385470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24523,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6835/3949984309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":3949984309},{"type":"v1","addr":"172.21.15.196:6835","nonce":3949984309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24524,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6837/1301464246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":1301464246},{"type":"v1","addr":"172.21.15.194:6837","nonce":1301464246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24535,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6837/895247251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":895247251},{"type":"v1","addr":"172.21.15.196:6837","nonce":895247251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24556,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6839/2001427268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6838","nonce":2001427268},{"type":"v1","addr":"172.21.15.196:6839","nonce":2001427268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-18T16:06:44.180425+0000","modified":"2024-03-18T16:06:49.511040+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24575},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24575":{"gid":24575,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.194:6839/274911158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":274911158},{"type":"v1","addr":"172.21.15.194:6839","nonce":274911158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24575,"qdb_cluster":[24575]},"id":1}]} 2024-03-18T16:06:51.986 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 16 2024-03-18T16:06:51.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.986+0000 7fddcca00640 1 -- 172.21.15.39:0/285802235 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fddac0a2880 msgr2=0x7fddac0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:51.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.986+0000 7fddcca00640 1 --2- 172.21.15.39:0/285802235 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fddac0a2880 0x7fddac0a4d40 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fddb00095f0 tx=0x7fddb0005f90 comp rx=0 tx=0).stop 2024-03-18T16:06:51.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.986+0000 7fddcca00640 1 -- 172.21.15.39:0/285802235 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fddc8103810 msgr2=0x7fddc8199a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:51.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.986+0000 7fddcca00640 1 --2- 172.21.15.39:0/285802235 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fddc8103810 0x7fddc8199a80 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fddb8005fd0 tx=0x7fddb800ef90 comp rx=0 tx=0).stop 2024-03-18T16:06:51.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.986+0000 7fddcca00640 1 -- 172.21.15.39:0/285802235 shutdown_connections 2024-03-18T16:06:51.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.986+0000 7fddcca00640 1 --2- 172.21.15.39:0/285802235 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fddac0a2880 0x7fddac0a4d40 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:51.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.986+0000 7fddcca00640 1 --2- 172.21.15.39:0/285802235 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fddc81053c0 0x7fddc81a1030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:51.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.987+0000 7fddcca00640 1 --2- 172.21.15.39:0/285802235 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fddc8104a00 0x7fddc8199fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:51.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.987+0000 7fddcca00640 1 --2- 172.21.15.39:0/285802235 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fddc8103810 0x7fddc8199a80 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:51.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.987+0000 7fddcca00640 1 -- 172.21.15.39:0/285802235 >> 172.21.15.39:0/285802235 conn(0x7fddc80fed80 msgr2=0x7fddc8107ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:51.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.987+0000 7fddcca00640 1 -- 172.21.15.39:0/285802235 shutdown_connections 2024-03-18T16:06:51.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:51.987+0000 7fddcca00640 1 -- 172.21.15.39:0/285802235 wait complete. 2024-03-18T16:06:52.006 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T16:06:52.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.220+0000 7fb800a00640 1 -- 172.21.15.39:0/3416480690 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb7fc107e00 msgr2=0x7fb7fc108220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:52.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.220+0000 7fb800a00640 1 --2- 172.21.15.39:0/3416480690 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb7fc107e00 0x7fb7fc108220 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec009900 tx=0x7fb7ec02bc30 comp rx=0 tx=0).stop 2024-03-18T16:06:52.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.220+0000 7fb800a00640 1 -- 172.21.15.39:0/3416480690 shutdown_connections 2024-03-18T16:06:52.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.220+0000 7fb800a00640 1 --2- 172.21.15.39:0/3416480690 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb7fc1099b0 0x7fb7fc110240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:52.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.220+0000 7fb800a00640 1 --2- 172.21.15.39:0/3416480690 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb7fc108ff0 0x7fb7fc109470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:52.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.220+0000 7fb800a00640 1 --2- 172.21.15.39:0/3416480690 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb7fc107e00 0x7fb7fc108220 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:52.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.220+0000 7fb800a00640 1 -- 172.21.15.39:0/3416480690 >> 172.21.15.39:0/3416480690 conn(0x7fb7fc076dc0 msgr2=0x7fb7fc0791e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:52.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.220+0000 7fb800a00640 1 -- 172.21.15.39:0/3416480690 shutdown_connections 2024-03-18T16:06:52.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.220+0000 7fb800a00640 1 -- 172.21.15.39:0/3416480690 wait complete. 2024-03-18T16:06:52.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.221+0000 7fb800a00640 1 Processor -- start 2024-03-18T16:06:52.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.222+0000 7fb800a00640 1 -- start start 2024-03-18T16:06:52.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.222+0000 7fb800a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb7fc107e00 0x7fb7fc19e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:52.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.222+0000 7fb800a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb7fc108ff0 0x7fb7fc19e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:52.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.222+0000 7fb800a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb7fc1099b0 0x7fb7fc1a56c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:52.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.222+0000 7fb800a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fb7fc113aa0 con 0x7fb7fc107e00 2024-03-18T16:06:52.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.222+0000 7fb800a00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fb7fc113920 con 0x7fb7fc108ff0 2024-03-18T16:06:52.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.222+0000 7fb800a00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fb7fc113c20 con 0x7fb7fc1099b0 2024-03-18T16:06:52.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.222+0000 7fb7f3e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb7fc108ff0 0x7fb7fc19e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:52.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.222+0000 7fb7f9e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb7fc107e00 0x7fb7fc19e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:52.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.222+0000 7fb7f3e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb7fc108ff0 0x7fb7fc19e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:36874/0 (socket says 172.21.15.39:36874) 2024-03-18T16:06:52.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.222+0000 7fb7f9e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb7fc107e00 0x7fb7fc19e110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43504/0 (socket says 172.21.15.39:43504) 2024-03-18T16:06:52.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.222+0000 7fb7f3e00640 1 -- 172.21.15.39:0/968245156 learned_addr learned my addr 172.21.15.39:0/968245156 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:06:52.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.222+0000 7fb7fa800640 1 --2- 172.21.15.39:0/968245156 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb7fc1099b0 0x7fb7fc1a56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:52.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.223+0000 7fb7f3e00640 1 -- 172.21.15.39:0/968245156 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb7fc1099b0 msgr2=0x7fb7fc1a56c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:52.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.223+0000 7fb7f3e00640 1 --2- 172.21.15.39:0/968245156 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb7fc1099b0 0x7fb7fc1a56c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:52.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.223+0000 7fb7f3e00640 1 -- 172.21.15.39:0/968245156 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb7fc107e00 msgr2=0x7fb7fc19e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:52.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.223+0000 7fb7f3e00640 1 --2- 172.21.15.39:0/968245156 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb7fc107e00 0x7fb7fc19e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:52.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.223+0000 7fb7f3e00640 1 -- 172.21.15.39:0/968245156 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7fc1a5d30 con 0x7fb7fc108ff0 2024-03-18T16:06:52.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.223+0000 7fb7fa800640 1 --2- 172.21.15.39:0/968245156 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb7fc1099b0 0x7fb7fc1a56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:06:52.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.223+0000 7fb7f3e00640 1 --2- 172.21.15.39:0/968245156 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb7fc108ff0 0x7fb7fc19e650 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb7e400d950 tx=0x7fb7e400de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:52.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.223+0000 7fb7f2a00640 1 -- 172.21.15.39:0/968245156 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7e4017070 con 0x7fb7fc108ff0 2024-03-18T16:06:52.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.223+0000 7fb7f2a00640 1 -- 172.21.15.39:0/968245156 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb7e4009c30 con 0x7fb7fc108ff0 2024-03-18T16:06:52.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.223+0000 7fb7f2a00640 1 -- 172.21.15.39:0/968245156 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7e400f6b0 con 0x7fb7fc108ff0 2024-03-18T16:06:52.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.223+0000 7fb800a00640 1 -- 172.21.15.39:0/968245156 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7fc1a5fc0 con 0x7fb7fc108ff0 2024-03-18T16:06:52.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.223+0000 7fb800a00640 1 -- 172.21.15.39:0/968245156 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7fc1a6530 con 0x7fb7fc108ff0 2024-03-18T16:06:52.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.224+0000 7fb800a00640 1 -- 172.21.15.39:0/968245156 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7fc10f120 con 0x7fb7fc108ff0 2024-03-18T16:06:52.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.226+0000 7fb7f2a00640 1 -- 172.21.15.39:0/968245156 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fb7e40109d0 con 0x7fb7fc108ff0 2024-03-18T16:06:52.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.227+0000 7fb7f2a00640 1 --2- 172.21.15.39:0/968245156 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb7e00a2880 0x7fb7e00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:52.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.227+0000 7fb7f2a00640 1 -- 172.21.15.39:0/968245156 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fb7e40ce8a0 con 0x7fb7fc108ff0 2024-03-18T16:06:52.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.227+0000 7fb7f9e00640 1 --2- 172.21.15.39:0/968245156 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb7e00a2880 0x7fb7e00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:52.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.228+0000 7fb7f9e00640 1 --2- 172.21.15.39:0/968245156 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb7e00a2880 0x7fb7e00a4d40 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec02c170 tx=0x7fb7ec02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:52.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.231+0000 7fb7f2a00640 1 -- 172.21.15.39:0/968245156 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb7e4009de0 con 0x7fb7fc108ff0 2024-03-18T16:06:52.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:52 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/3094248891' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:06:52.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:52 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/285802235' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:06:52.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.428+0000 7fb800a00640 1 -- 172.21.15.39:0/968245156 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb7fc0623e0 con 0x7fb7fc108ff0 2024-03-18T16:06:52.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.430+0000 7fb7f2a00640 1 -- 172.21.15.39:0/968245156 <== mon.2 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24142 (secure 0 0 0) 0x7fb7e409b6a0 con 0x7fb7fc108ff0 2024-03-18T16:06:52.431 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:06:52.432 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":89,"fsid":"1892d8fc-e540-11ee-95c9-87774f69a715","created":"2024-03-18T15:57:36.507008+0000","modified":"2024-03-18T16:06:49.510653+0000","last_up_change":"2024-03-18T16:05:45.250629+0000","last_in_change":"2024-03-18T16:05:24.841337+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T16:01:30.048247+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T16:06:42.191791+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T16:06:43.670325+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T16:06:47.523388+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"771e0558-be62-4436-94b0-6364751911c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6803","nonce":4090743635}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6805","nonce":4090743635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6809","nonce":4090743635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6807","nonce":4090743635}]},"public_addr":"172.21.15.39:6803/4090743635","cluster_addr":"172.21.15.39:6805/4090743635","heartbeat_back_addr":"172.21.15.39:6809/4090743635","heartbeat_front_addr":"172.21.15.39:6807/4090743635","state":["exists","up"]},{"osd":1,"uuid":"0430eee1-cf34-4277-b255-c7d9f07ad19a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6801","nonce":2800329805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6803","nonce":2800329805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6807","nonce":2800329805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6805","nonce":2800329805}]},"public_addr":"172.21.15.194:6801/2800329805","cluster_addr":"172.21.15.194:6803/2800329805","heartbeat_back_addr":"172.21.15.194:6807/2800329805","heartbeat_front_addr":"172.21.15.194:6805/2800329805","state":["exists","up"]},{"osd":2,"uuid":"3a718da4-2511-4ec1-b1a8-eafd6cce0df0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6801","nonce":3935874719}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6803","nonce":3935874719}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6807","nonce":3935874719}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6805","nonce":3935874719}]},"public_addr":"172.21.15.196:6801/3935874719","cluster_addr":"172.21.15.196:6803/3935874719","heartbeat_back_addr":"172.21.15.196:6807/3935874719","heartbeat_front_addr":"172.21.15.196:6805/3935874719","state":["exists","up"]},{"osd":3,"uuid":"6f9c37f9-0ab9-4038-8919-afd4b39c4165","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6811","nonce":893345939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6813","nonce":893345939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6817","nonce":893345939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6815","nonce":893345939}]},"public_addr":"172.21.15.39:6811/893345939","cluster_addr":"172.21.15.39:6813/893345939","heartbeat_back_addr":"172.21.15.39:6817/893345939","heartbeat_front_addr":"172.21.15.39:6815/893345939","state":["exists","up"]},{"osd":4,"uuid":"e795bb0e-4791-4537-acda-1a321e19bc9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6809","nonce":800423088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6811","nonce":800423088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6815","nonce":800423088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6813","nonce":800423088}]},"public_addr":"172.21.15.194:6809/800423088","cluster_addr":"172.21.15.194:6811/800423088","heartbeat_back_addr":"172.21.15.194:6815/800423088","heartbeat_front_addr":"172.21.15.194:6813/800423088","state":["exists","up"]},{"osd":5,"uuid":"026e175e-6acb-4b4c-b235-8ccee8ae1149","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6809","nonce":2746224525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6811","nonce":2746224525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6815","nonce":2746224525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6813","nonce":2746224525}]},"public_addr":"172.21.15.196:6809/2746224525","cluster_addr":"172.21.15.196:6811/2746224525","heartbeat_back_addr":"172.21.15.196:6815/2746224525","heartbeat_front_addr":"172.21.15.196:6813/2746224525","state":["exists","up"]},{"osd":6,"uuid":"c1b183ff-383f-4c5c-9075-f76bab1c0bb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6819","nonce":1213371331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6821","nonce":1213371331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6825","nonce":1213371331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6823","nonce":1213371331}]},"public_addr":"172.21.15.39:6819/1213371331","cluster_addr":"172.21.15.39:6821/1213371331","heartbeat_back_addr":"172.21.15.39:6825/1213371331","heartbeat_front_addr":"172.21.15.39:6823/1213371331","state":["exists","up"]},{"osd":7,"uuid":"612f8697-957d-4ef4-a87f-241b36f9d13d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6817","nonce":3299162444}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6819","nonce":3299162444}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6823","nonce":3299162444}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6821","nonce":3299162444}]},"public_addr":"172.21.15.194:6817/3299162444","cluster_addr":"172.21.15.194:6819/3299162444","heartbeat_back_addr":"172.21.15.194:6823/3299162444","heartbeat_front_addr":"172.21.15.194:6821/3299162444","state":["exists","up"]},{"osd":8,"uuid":"836a9107-4d4f-4594-b8dc-0301e64f59d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6817","nonce":2456143145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6819","nonce":2456143145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6823","nonce":2456143145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6821","nonce":2456143145}]},"public_addr":"172.21.15.196:6817/2456143145","cluster_addr":"172.21.15.196:6819/2456143145","heartbeat_back_addr":"172.21.15.196:6823/2456143145","heartbeat_front_addr":"172.21.15.196:6821/2456143145","state":["exists","up"]},{"osd":9,"uuid":"4f30b960-12e3-4b16-bd28-fcc988139b58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6827","nonce":1848634433}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6829","nonce":1848634433}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6833","nonce":1848634433}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6831","nonce":1848634433}]},"public_addr":"172.21.15.39:6827/1848634433","cluster_addr":"172.21.15.39:6829/1848634433","heartbeat_back_addr":"172.21.15.39:6833/1848634433","heartbeat_front_addr":"172.21.15.39:6831/1848634433","state":["exists","up"]},{"osd":10,"uuid":"620f90e8-23e6-4bc0-8269-7219b9e1aebd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6825","nonce":3725064809}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6827","nonce":3725064809}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6831","nonce":3725064809}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6829","nonce":3725064809}]},"public_addr":"172.21.15.194:6825/3725064809","cluster_addr":"172.21.15.194:6827/3725064809","heartbeat_back_addr":"172.21.15.194:6831/3725064809","heartbeat_front_addr":"172.21.15.194:6829/3725064809","state":["exists","up"]},{"osd":11,"uuid":"867a2535-8235-4ceb-a698-eb86869d6d20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6825","nonce":2501212422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6827","nonce":2501212422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6831","nonce":2501212422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6829","nonce":2501212422}]},"public_addr":"172.21.15.196:6825/2501212422","cluster_addr":"172.21.15.196:6827/2501212422","heartbeat_back_addr":"172.21.15.196:6831/2501212422","heartbeat_front_addr":"172.21.15.196:6829/2501212422","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:00:36.931566+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:01:02.614024+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:01:26.193265+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:01:58.308833+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:02:25.049546+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:02:49.748750+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:03:23.582153+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:03:50.447321+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:04:15.812129+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:04:49.998047+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:05:17.087924+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:05:43.167637+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.39:0/3482086140":"2024-03-19T15:58:45.340294+0000","172.21.15.39:0/3844306284":"2024-03-19T15:58:45.340294+0000","172.21.15.39:6801/1810238546":"2024-03-19T15:58:45.340294+0000","172.21.15.39:6800/1810238546":"2024-03-19T15:58:45.340294+0000","172.21.15.39:0/2528397488":"2024-03-19T15:58:10.264798+0000","172.21.15.39:0/2240981254":"2024-03-19T15:58:10.264798+0000","172.21.15.39:6800/1217503189":"2024-03-19T15:58:10.264798+0000","172.21.15.39:0/3754876370":"2024-03-19T15:58:10.264798+0000","172.21.15.39:6801/1217503189":"2024-03-19T15:58:10.264798+0000","172.21.15.39:0/1874637628":"2024-03-19T15:58:45.340294+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T16:06:52.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.432+0000 7fb800a00640 1 -- 172.21.15.39:0/968245156 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb7e00a2880 msgr2=0x7fb7e00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:52.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.432+0000 7fb800a00640 1 --2- 172.21.15.39:0/968245156 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb7e00a2880 0x7fb7e00a4d40 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec02c170 tx=0x7fb7ec02df90 comp rx=0 tx=0).stop 2024-03-18T16:06:52.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.432+0000 7fb800a00640 1 -- 172.21.15.39:0/968245156 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb7fc108ff0 msgr2=0x7fb7fc19e650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:52.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.432+0000 7fb800a00640 1 --2- 172.21.15.39:0/968245156 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb7fc108ff0 0x7fb7fc19e650 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb7e400d950 tx=0x7fb7e400de10 comp rx=0 tx=0).stop 2024-03-18T16:06:52.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.432+0000 7fb800a00640 1 -- 172.21.15.39:0/968245156 shutdown_connections 2024-03-18T16:06:52.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.432+0000 7fb800a00640 1 --2- 172.21.15.39:0/968245156 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb7fc108ff0 0x7fb7fc19e650 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:52.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.432+0000 7fb800a00640 1 --2- 172.21.15.39:0/968245156 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb7e00a2880 0x7fb7e00a4d40 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:52.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.432+0000 7fb800a00640 1 --2- 172.21.15.39:0/968245156 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb7fc1099b0 0x7fb7fc1a56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:52.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.432+0000 7fb800a00640 1 --2- 172.21.15.39:0/968245156 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb7fc107e00 0x7fb7fc19e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:52.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.432+0000 7fb800a00640 1 -- 172.21.15.39:0/968245156 >> 172.21.15.39:0/968245156 conn(0x7fb7fc076dc0 msgr2=0x7fb7fc10c2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:52.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.433+0000 7fb800a00640 1 -- 172.21.15.39:0/968245156 shutdown_connections 2024-03-18T16:06:52.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.433+0000 7fb800a00640 1 -- 172.21.15.39:0/968245156 wait complete. 2024-03-18T16:06:52.451 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 16, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-18T16:06:44.180425+0000', 'modified': '2024-03-18T16:06:49.511040+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24575}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24575': {'gid': 24575, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.194:6839/274911158', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.194:6838', 'nonce': 274911158}, {'type': 'v1', 'addr': '172.21.15.194:6839', 'nonce': 274911158}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 24575, 'qdb_cluster': [24575]} 2024-03-18T16:06:52.451 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-18T16:06:52.451 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-18T16:06:52.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:52 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/3094248891' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:06:52.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:52 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/285802235' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:06:52.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:52 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/3094248891' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:06:52.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:52 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/285802235' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:06:52.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.666+0000 7ffaa2800640 1 -- 172.21.15.39:0/2400581625 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa9c100700 msgr2=0x7ffa9c100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:52.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.666+0000 7ffaa2800640 1 --2- 172.21.15.39:0/2400581625 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa9c100700 0x7ffa9c100b20 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ffa8c009900 tx=0x7ffa8c02bc30 comp rx=0 tx=0).stop 2024-03-18T16:06:52.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.667+0000 7ffaa2800640 1 -- 172.21.15.39:0/2400581625 shutdown_connections 2024-03-18T16:06:52.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.667+0000 7ffaa2800640 1 --2- 172.21.15.39:0/2400581625 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffa9c1022b0 0x7ffa9c108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:52.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.667+0000 7ffaa2800640 1 --2- 172.21.15.39:0/2400581625 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffa9c1018f0 0x7ffa9c101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:52.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.667+0000 7ffaa2800640 1 --2- 172.21.15.39:0/2400581625 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa9c100700 0x7ffa9c100b20 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:52.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.667+0000 7ffaa2800640 1 -- 172.21.15.39:0/2400581625 >> 172.21.15.39:0/2400581625 conn(0x7ffa9c0fbc70 msgr2=0x7ffa9c0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:52.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.667+0000 7ffaa2800640 1 -- 172.21.15.39:0/2400581625 shutdown_connections 2024-03-18T16:06:52.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.667+0000 7ffaa2800640 1 -- 172.21.15.39:0/2400581625 wait complete. 2024-03-18T16:06:52.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.668+0000 7ffaa2800640 1 Processor -- start 2024-03-18T16:06:52.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.668+0000 7ffaa2800640 1 -- start start 2024-03-18T16:06:52.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.669+0000 7ffaa2800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffa9c100700 0x7ffa9c199d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:52.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.669+0000 7ffaa2800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffa9c1018f0 0x7ffa9c19a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:52.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.669+0000 7ffaa2800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa9c1022b0 0x7ffa9c1a1320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:52.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.669+0000 7ffaa2800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7ffa9c10c380 con 0x7ffa9c1018f0 2024-03-18T16:06:52.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.669+0000 7ffaa2800640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ffa9c10c200 con 0x7ffa9c1022b0 2024-03-18T16:06:52.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.669+0000 7ffaa2800640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7ffa9c10c500 con 0x7ffa9c100700 2024-03-18T16:06:52.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.669+0000 7ffa92a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffa9c1018f0 0x7ffa9c19a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:52.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.669+0000 7ffa92a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffa9c1018f0 0x7ffa9c19a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43530/0 (socket says 172.21.15.39:43530) 2024-03-18T16:06:52.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.669+0000 7ffa92a00640 1 -- 172.21.15.39:0/4084391263 learned_addr learned my addr 172.21.15.39:0/4084391263 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:06:52.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.669+0000 7ffa9be00640 1 --2- 172.21.15.39:0/4084391263 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa9c1022b0 0x7ffa9c1a1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:52.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.669+0000 7ffa9b400640 1 --2- 172.21.15.39:0/4084391263 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffa9c100700 0x7ffa9c199d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:52.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.669+0000 7ffa92a00640 1 -- 172.21.15.39:0/4084391263 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffa9c100700 msgr2=0x7ffa9c199d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:52.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.669+0000 7ffa92a00640 1 --2- 172.21.15.39:0/4084391263 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffa9c100700 0x7ffa9c199d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:52.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.669+0000 7ffa92a00640 1 -- 172.21.15.39:0/4084391263 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa9c1022b0 msgr2=0x7ffa9c1a1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:52.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.669+0000 7ffa92a00640 1 --2- 172.21.15.39:0/4084391263 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa9c1022b0 0x7ffa9c1a1320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:52.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.669+0000 7ffa92a00640 1 -- 172.21.15.39:0/4084391263 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa9c1a1990 con 0x7ffa9c1018f0 2024-03-18T16:06:52.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.670+0000 7ffa92a00640 1 --2- 172.21.15.39:0/4084391263 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffa9c1018f0 0x7ffa9c19a2b0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7ffa8400e9f0 tx=0x7ffa8400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:52.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.670+0000 7ffa98c00640 1 -- 172.21.15.39:0/4084391263 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa8400cd80 con 0x7ffa9c1018f0 2024-03-18T16:06:52.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.670+0000 7ffa98c00640 1 -- 172.21.15.39:0/4084391263 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa84010910 con 0x7ffa9c1018f0 2024-03-18T16:06:52.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.670+0000 7ffaa2800640 1 -- 172.21.15.39:0/4084391263 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa9c1a1c80 con 0x7ffa9c1018f0 2024-03-18T16:06:52.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.670+0000 7ffa98c00640 1 -- 172.21.15.39:0/4084391263 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa84018990 con 0x7ffa9c1018f0 2024-03-18T16:06:52.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.670+0000 7ffaa2800640 1 -- 172.21.15.39:0/4084391263 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa9c1a21d0 con 0x7ffa9c1018f0 2024-03-18T16:06:52.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.674+0000 7ffaa562c740 1 Processor -- start 2024-03-18T16:06:52.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.675+0000 7ffaa562c740 1 -- start start 2024-03-18T16:06:52.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.675+0000 7ffa98c00640 1 -- 172.21.15.39:0/4084391263 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7ffa84010ac0 con 0x7ffa9c1018f0 2024-03-18T16:06:52.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.675+0000 7ffaa562c740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5602ca2ab160 0x5602ca2ab580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:52.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.675+0000 7ffaa562c740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5602ca2abb50 0x5602ca2b0650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:52.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.675+0000 7ffaa562c740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5602ca2b0cc0 0x5602ca2b3150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:52.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.675+0000 7ffaa562c740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5602ca20fc50 con 0x5602ca2abb50 2024-03-18T16:06:52.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.675+0000 7ffaa562c740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5602ca1b89d0 con 0x5602ca2b0cc0 2024-03-18T16:06:52.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.675+0000 7ffaa562c740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x5602ca20bcf0 con 0x5602ca2ab160 2024-03-18T16:06:52.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.675+0000 7ffa92a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5602ca2b0cc0 0x5602ca2b3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:52.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.675+0000 7ffa9b400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5602ca2abb50 0x5602ca2b0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:52.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.675+0000 7ffa92a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5602ca2b0cc0 0x5602ca2b3150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:36918/0 (socket says 172.21.15.39:36918) 2024-03-18T16:06:52.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.675+0000 7ffa92a00640 1 -- 172.21.15.39:0/1084222993 learned_addr learned my addr 172.21.15.39:0/1084222993 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:06:52.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.675+0000 7ffa9be00640 1 --2- 172.21.15.39:0/1084222993 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5602ca2ab160 0x5602ca2ab580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:52.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.676+0000 7ffa92a00640 1 -- 172.21.15.39:0/1084222993 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5602ca2ab160 msgr2=0x5602ca2ab580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:06:52.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.676+0000 7ffa92a00640 1 --2- 172.21.15.39:0/1084222993 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5602ca2ab160 0x5602ca2ab580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:52.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.676+0000 7ffa92a00640 1 -- 172.21.15.39:0/1084222993 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5602ca2abb50 msgr2=0x5602ca2b0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:06:52.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.676+0000 7ffa92a00640 1 --2- 172.21.15.39:0/1084222993 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5602ca2abb50 0x5602ca2b0650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:52.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.676+0000 7ffa92a00640 1 -- 172.21.15.39:0/1084222993 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5602ca1d2f10 con 0x5602ca2b0cc0 2024-03-18T16:06:52.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.676+0000 7ffa92a00640 1 --2- 172.21.15.39:0/1084222993 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5602ca2b0cc0 0x5602ca2b3150 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ffa840d71b0 tx=0x7ffa840d7670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea39fcefe135afc5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:52.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.676+0000 7ffa90c00640 1 -- 172.21.15.39:0/1084222993 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa840e2070 con 0x5602ca2b0cc0 2024-03-18T16:06:52.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.676+0000 7ffa90c00640 1 -- 172.21.15.39:0/1084222993 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa840cfb00 con 0x5602ca2b0cc0 2024-03-18T16:06:52.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.676+0000 7ffa90c00640 1 -- 172.21.15.39:0/1084222993 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa840dd9c0 con 0x5602ca2b0cc0 2024-03-18T16:06:52.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.677+0000 7ffa98c00640 1 --2- 172.21.15.39:0/4084391263 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ffa800a2800 0x7ffa800a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:52.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.677+0000 7ffa98c00640 1 -- 172.21.15.39:0/4084391263 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7ffa840ceff0 con 0x7ffa9c1018f0 2024-03-18T16:06:52.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.677+0000 7ffaa562c740 1 -- 172.21.15.39:0/1084222993 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5602ca2b0cc0 msgr2=0x5602ca2b3150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:52.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.677+0000 7ffaa562c740 1 --2- 172.21.15.39:0/1084222993 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5602ca2b0cc0 0x5602ca2b3150 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ffa840d71b0 tx=0x7ffa840d7670 comp rx=0 tx=0).stop 2024-03-18T16:06:52.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.677+0000 7ffa9be00640 1 --2- 172.21.15.39:0/4084391263 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ffa800a2800 0x7ffa800a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:52.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.677+0000 7ffaa562c740 1 -- 172.21.15.39:0/1084222993 shutdown_connections 2024-03-18T16:06:52.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.677+0000 7ffaa562c740 1 --2- 172.21.15.39:0/1084222993 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5602ca2b0cc0 0x5602ca2b3150 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:52.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.677+0000 7ffaa562c740 1 --2- 172.21.15.39:0/1084222993 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5602ca2abb50 0x5602ca2b0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:52.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.677+0000 7ffaa562c740 1 --2- 172.21.15.39:0/1084222993 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5602ca2ab160 0x5602ca2ab580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:52.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.677+0000 7ffaa562c740 1 -- 172.21.15.39:0/1084222993 >> 172.21.15.39:0/1084222993 conn(0x5602ca2a66d0 msgr2=0x5602ca2a6ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:52.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.677+0000 7ffaa562c740 1 -- 172.21.15.39:0/1084222993 shutdown_connections 2024-03-18T16:06:52.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.677+0000 7ffaa562c740 1 -- 172.21.15.39:0/1084222993 wait complete. 2024-03-18T16:06:52.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.678+0000 7ffa9be00640 1 --2- 172.21.15.39:0/4084391263 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ffa800a2800 0x7ffa800a4cc0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x5602ca2ac3b0 tx=0x7ffa94009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:52.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.678+0000 7ffaa562c740 1 -- start start 2024-03-18T16:06:52.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.678+0000 7ffaa562c740 1 Processor -- start 2024-03-18T16:06:52.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.678+0000 7ffaa562c740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5602ca2ab160 0x5602ca2b52b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:52.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.678+0000 7ffaa562c740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5602ca2abb50 0x5602ca2b57f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:52.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.678+0000 7ffaa562c740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5602ca2b0cc0 0x5602ca2bc7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:52.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.678+0000 7ffaa562c740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5602ca2b3c70 con 0x5602ca2abb50 2024-03-18T16:06:52.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.679+0000 7ffaa562c740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5602ca1cbe20 con 0x5602ca2ab160 2024-03-18T16:06:52.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.679+0000 7ffaa562c740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x5602ca2b3df0 con 0x5602ca2b0cc0 2024-03-18T16:06:52.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.679+0000 7ffaa562c740 10 client.0 _ll_register_callbacks cb 0x5602ca1cb7f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:06:52.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.679+0000 7ffa9b400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5602ca2ab160 0x5602ca2b52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:52.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.679+0000 7ffa9b400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5602ca2ab160 0x5602ca2b52b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:36920/0 (socket says 172.21.15.39:36920) 2024-03-18T16:06:52.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.679+0000 7ffa92a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5602ca2abb50 0x5602ca2b57f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:52.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.679+0000 7ffa9b400640 1 -- 172.21.15.39:0/1106468368 learned_addr learned my addr 172.21.15.39:0/1106468368 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:06:52.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.679+0000 7ffa9be00640 1 --2- 172.21.15.39:0/1106468368 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5602ca2b0cc0 0x5602ca2bc7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:52.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.679+0000 7ffa9b400640 1 -- 172.21.15.39:0/1106468368 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5602ca2b0cc0 msgr2=0x5602ca2bc7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:06:52.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.679+0000 7ffa9b400640 1 --2- 172.21.15.39:0/1106468368 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5602ca2b0cc0 0x5602ca2bc7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:52.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.679+0000 7ffa9b400640 1 -- 172.21.15.39:0/1106468368 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5602ca2abb50 msgr2=0x5602ca2b57f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:06:52.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.679+0000 7ffa9b400640 1 --2- 172.21.15.39:0/1106468368 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5602ca2abb50 0x5602ca2b57f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:52.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.679+0000 7ffa9b400640 1 -- 172.21.15.39:0/1106468368 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5602c9f242d0 con 0x5602ca2ab160 2024-03-18T16:06:52.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.679+0000 7ffa9be00640 1 --2- 172.21.15.39:0/1106468368 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5602ca2b0cc0 0x5602ca2bc7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:06:52.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.679+0000 7ffa92a00640 1 --2- 172.21.15.39:0/1106468368 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5602ca2abb50 0x5602ca2b57f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:06:52.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.679+0000 7ffa9b400640 1 --2- 172.21.15.39:0/1106468368 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5602ca2ab160 0x5602ca2b52b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7ffa8c02cfd0 tx=0x7ffa8c005bb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9869a7d51f6f2c19 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:52.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.680+0000 7ffa89600640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:06:52.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.680+0000 7ffa89600640 1 -- 172.21.15.39:0/1106468368 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa8c004380 con 0x5602ca2ab160 2024-03-18T16:06:52.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.680+0000 7ffa89600640 1 -- 172.21.15.39:0/1106468368 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7ffa8c0046c0 con 0x5602ca2ab160 2024-03-18T16:06:52.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.680+0000 7ffaa562c740 1 -- 172.21.15.39:0/1106468368 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffa840cfb00 con 0x5602ca2ab160 2024-03-18T16:06:52.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.680+0000 7ffa89600640 1 -- 172.21.15.39:0/1106468368 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7ffa8c0048c0 con 0x5602ca2ab160 2024-03-18T16:06:52.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.680+0000 7ffaa562c740 10 client.24683 fetch_fsmap learned FSMap version 16 2024-03-18T16:06:52.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.680+0000 7ffaa562c740 1 -- 172.21.15.39:0/1106468368 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=4+}) -- 0x5602c9fee6b0 con 0x5602ca2ab160 2024-03-18T16:06:52.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.680+0000 7ffa89600640 1 -- 172.21.15.39:0/1106468368 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa840cfb00 con 0x5602ca2ab160 2024-03-18T16:06:52.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.680+0000 7ffa89600640 1 -- 172.21.15.39:0/1106468368 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 16) ==== 8342+0+0 (secure 0 0 0) 0x7ffa8c0035f0 con 0x5602ca2ab160 2024-03-18T16:06:52.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.681+0000 7ffaa562c740 10 client.24683 fetch_fsmap finished waiting for FSMap version 16 2024-03-18T16:06:52.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.681+0000 7ffaa562c740 10 client.24683 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-18T16:06:52.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.681+0000 7ffaa562c740 20 client.24683 populate_metadata read hostname 'smithi039' 2024-03-18T16:06:52.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.681+0000 7ffaa562c740 1 --2- 172.21.15.39:0/1106468368 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5602ca2c2050 0x5602ca2c34c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:52.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.681+0000 7ffaa562c740 4 client.24683 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:06:52.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.681+0000 7ffaa562c740 1 -- 172.21.15.39:0/1106468368 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5602ca26d2b0 con 0x5602ca2c2050 2024-03-18T16:06:52.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.681+0000 7ffa92a00640 1 --2- 172.21.15.39:0/1106468368 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5602ca2c2050 0x5602ca2c34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:52.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.681+0000 7ffa92a00640 1 --2- 172.21.15.39:0/1106468368 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5602ca2c2050 0x5602ca2c34c0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=19c0e931a9185a86 server_cookie=38c8e677b802c43c in_seq=0 out_seq=0 2024-03-18T16:06:52.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.681+0000 7ffa89600640 10 client.24683 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:06:52.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.682+0000 7ffa89600640 1 -- 172.21.15.39:0/1106468368 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5602ca26d2b0 con 0x5602ca2c2050 2024-03-18T16:06:52.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.682+0000 7ffa89600640 10 client.24683 handle_command_reply: tid=0 2024-03-18T16:06:52.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.682+0000 7ffaa562c740 1 client.24683 shutdown 2024-03-18T16:06:52.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.683+0000 7ffaa562c740 1 -- 172.21.15.39:0/1106468368 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5602ca2ab160 msgr2=0x5602ca2b52b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:52.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.683+0000 7ffaa562c740 1 --2- 172.21.15.39:0/1106468368 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5602ca2ab160 0x5602ca2b52b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7ffa8c02cfd0 tx=0x7ffa8c005bb0 comp rx=0 tx=0).stop 2024-03-18T16:06:52.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.683+0000 7ffaa562c740 1 -- 172.21.15.39:0/1106468368 shutdown_connections 2024-03-18T16:06:52.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.683+0000 7ffaa562c740 1 --2- 172.21.15.39:0/1106468368 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5602ca2c2050 0x5602ca2c34c0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:52.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.683+0000 7ffaa562c740 1 --2- 172.21.15.39:0/1106468368 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5602ca2b0cc0 0x5602ca2bc7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:52.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.683+0000 7ffaa562c740 1 --2- 172.21.15.39:0/1106468368 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5602ca2abb50 0x5602ca2b57f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:52.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.683+0000 7ffa89600640 0 client.24683 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:06:52.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.683+0000 7ffaa562c740 1 --2- 172.21.15.39:0/1106468368 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5602ca2ab160 0x5602ca2b52b0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:52.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.683+0000 7ffaa562c740 1 -- 172.21.15.39:0/1106468368 >> 172.21.15.39:0/1106468368 conn(0x5602ca2a66d0 msgr2=0x5602ca2a5a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:52.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.683+0000 7ffaa562c740 1 -- 172.21.15.39:0/1106468368 shutdown_connections 2024-03-18T16:06:52.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.683+0000 7ffaa562c740 1 -- 172.21.15.39:0/1106468368 wait complete. 2024-03-18T16:06:52.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.683+0000 7ffaa562c740 20 client.24683 trim_cache size 0 max 16384 2024-03-18T16:06:52.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.703+0000 7ffaa562c740 1 Processor -- start 2024-03-18T16:06:52.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.703+0000 7ffaa562c740 1 -- start start 2024-03-18T16:06:52.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.703+0000 7ffaa562c740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5602ca2ab160 0x5602ca2aa040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:52.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.703+0000 7ffaa562c740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5602ca2abb50 0x5602ca2bc780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:52.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.703+0000 7ffaa562c740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5602ca2b0cc0 0x5602ca2ac3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:52.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.703+0000 7ffaa562c740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5602ca2c1800 con 0x5602ca2abb50 2024-03-18T16:06:52.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.703+0000 7ffaa562c740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5602ca2c1ed0 con 0x5602ca2b0cc0 2024-03-18T16:06:52.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.703+0000 7ffaa562c740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x5602ca2c25a0 con 0x5602ca2ab160 2024-03-18T16:06:52.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.703+0000 7ffa9be00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5602ca2b0cc0 0x5602ca2ac3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:52.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.703+0000 7ffa92a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5602ca2abb50 0x5602ca2bc780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:52.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.703+0000 7ffa9be00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5602ca2b0cc0 0x5602ca2ac3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:36930/0 (socket says 172.21.15.39:36930) 2024-03-18T16:06:52.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.703+0000 7ffa9be00640 1 -- 172.21.15.39:0/4081947006 learned_addr learned my addr 172.21.15.39:0/4081947006 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:06:52.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.703+0000 7ffa9b400640 1 --2- 172.21.15.39:0/4081947006 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5602ca2ab160 0x5602ca2aa040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:52.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.704+0000 7ffa9be00640 1 -- 172.21.15.39:0/4081947006 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5602ca2ab160 msgr2=0x5602ca2aa040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:06:52.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.704+0000 7ffa9be00640 1 --2- 172.21.15.39:0/4081947006 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5602ca2ab160 0x5602ca2aa040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:52.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.704+0000 7ffa9be00640 1 -- 172.21.15.39:0/4081947006 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5602ca2abb50 msgr2=0x5602ca2bc780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:06:52.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.704+0000 7ffa9be00640 1 --2- 172.21.15.39:0/4081947006 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5602ca2abb50 0x5602ca2bc780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:52.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.704+0000 7ffa9be00640 1 -- 172.21.15.39:0/4081947006 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5602ca27c750 con 0x5602ca2b0cc0 2024-03-18T16:06:52.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.704+0000 7ffa9be00640 1 --2- 172.21.15.39:0/4081947006 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5602ca2b0cc0 0x5602ca2ac3e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ffa94003f30 tx=0x7ffa94016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2adac8805803bc8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:52.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.704+0000 7ffa8be00640 1 -- 172.21.15.39:0/4081947006 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa94004800 con 0x5602ca2b0cc0 2024-03-18T16:06:52.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.704+0000 7ffa8be00640 1 -- 172.21.15.39:0/4081947006 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa9401e910 con 0x5602ca2b0cc0 2024-03-18T16:06:52.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.705+0000 7ffa8be00640 1 -- 172.21.15.39:0/4081947006 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa9401c850 con 0x5602ca2b0cc0 2024-03-18T16:06:52.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.705+0000 7ffaa562c740 1 -- 172.21.15.39:0/4081947006 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5602ca2b0cc0 msgr2=0x5602ca2ac3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:52.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.705+0000 7ffaa562c740 1 --2- 172.21.15.39:0/4081947006 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5602ca2b0cc0 0x5602ca2ac3e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ffa94003f30 tx=0x7ffa94016000 comp rx=0 tx=0).stop 2024-03-18T16:06:52.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.705+0000 7ffaa562c740 1 -- 172.21.15.39:0/4081947006 shutdown_connections 2024-03-18T16:06:52.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.705+0000 7ffaa562c740 1 --2- 172.21.15.39:0/4081947006 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5602ca2b0cc0 0x5602ca2ac3e0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:52.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.705+0000 7ffaa562c740 1 --2- 172.21.15.39:0/4081947006 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5602ca2abb50 0x5602ca2bc780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:52.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.705+0000 7ffaa562c740 1 --2- 172.21.15.39:0/4081947006 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5602ca2ab160 0x5602ca2aa040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:52.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.705+0000 7ffaa562c740 1 -- 172.21.15.39:0/4081947006 >> 172.21.15.39:0/4081947006 conn(0x5602ca2a66d0 msgr2=0x5602ca2a9930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:52.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.705+0000 7ffaa562c740 1 -- 172.21.15.39:0/4081947006 shutdown_connections 2024-03-18T16:06:52.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.705+0000 7ffaa562c740 1 -- 172.21.15.39:0/4081947006 wait complete. 2024-03-18T16:06:52.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.706+0000 7ffaa562c740 1 -- start start 2024-03-18T16:06:52.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.706+0000 7ffaa562c740 1 Processor -- start 2024-03-18T16:06:52.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.706+0000 7ffaa562c740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5602ca2ab160 0x5602ca2b8610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:52.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.706+0000 7ffaa562c740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5602ca2abb50 0x5602ca2b8b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:52.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.706+0000 7ffaa562c740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5602ca2b0cc0 0x5602ca2bbb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:52.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.706+0000 7ffaa562c740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5602ca2c3340 con 0x5602ca2b0cc0 2024-03-18T16:06:52.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.706+0000 7ffaa562c740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5602ca2c2c70 con 0x5602ca2abb50 2024-03-18T16:06:52.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.706+0000 7ffaa562c740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x5602ca2c3a10 con 0x5602ca2ab160 2024-03-18T16:06:52.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.706+0000 7ffaa562c740 10 client.0 _ll_register_callbacks cb 0x7ffa8c005d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:06:52.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.707+0000 7ffa92a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5602ca2abb50 0x5602ca2b8b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:52.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.707+0000 7ffa9be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5602ca2b0cc0 0x5602ca2bbb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:52.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.707+0000 7ffa9b400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5602ca2ab160 0x5602ca2b8610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:52.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.707+0000 7ffa92a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5602ca2abb50 0x5602ca2b8b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:36940/0 (socket says 172.21.15.39:36940) 2024-03-18T16:06:52.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.707+0000 7ffa9be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5602ca2b0cc0 0x5602ca2bbb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43564/0 (socket says 172.21.15.39:43564) 2024-03-18T16:06:52.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.707+0000 7ffa92a00640 1 -- 172.21.15.39:0/3402510385 learned_addr learned my addr 172.21.15.39:0/3402510385 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:06:52.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.707+0000 7ffa9b400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5602ca2ab160 0x5602ca2b8610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:53938/0 (socket says 172.21.15.39:53938) 2024-03-18T16:06:52.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.707+0000 7ffa92a00640 1 -- 172.21.15.39:0/3402510385 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5602ca2ab160 msgr2=0x5602ca2b8610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:06:52.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.707+0000 7ffa92a00640 1 --2- 172.21.15.39:0/3402510385 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5602ca2ab160 0x5602ca2b8610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:52.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.707+0000 7ffa92a00640 1 -- 172.21.15.39:0/3402510385 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5602ca2b0cc0 msgr2=0x5602ca2bbb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:06:52.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.707+0000 7ffa92a00640 1 --2- 172.21.15.39:0/3402510385 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5602ca2b0cc0 0x5602ca2bbb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:52.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.707+0000 7ffa92a00640 1 -- 172.21.15.39:0/3402510385 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5602ca2bc160 con 0x5602ca2abb50 2024-03-18T16:06:52.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.707+0000 7ffa92a00640 1 --2- 172.21.15.39:0/3402510385 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5602ca2abb50 0x5602ca2b8b50 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ffa84097c40 tx=0x7ffa840d12e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ba40f9ff16bc241 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:52.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.707+0000 7ffa89600640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:06:52.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.707+0000 7ffa89600640 1 -- 172.21.15.39:0/3402510385 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa84097a80 con 0x5602ca2abb50 2024-03-18T16:06:52.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.707+0000 7ffa89600640 1 -- 172.21.15.39:0/3402510385 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7ffa840d2bf0 con 0x5602ca2abb50 2024-03-18T16:06:52.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.708+0000 7ffaa562c740 1 -- 172.21.15.39:0/3402510385 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffa9401e910 con 0x5602ca2abb50 2024-03-18T16:06:52.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.708+0000 7ffa89600640 1 -- 172.21.15.39:0/3402510385 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7ffa8409b140 con 0x5602ca2abb50 2024-03-18T16:06:52.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.708+0000 7ffaa562c740 10 client.24689 fetch_fsmap learned FSMap version 16 2024-03-18T16:06:52.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.708+0000 7ffaa562c740 1 -- 172.21.15.39:0/3402510385 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=4+}) -- 0x5602ca328560 con 0x5602ca2abb50 2024-03-18T16:06:52.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.708+0000 7ffa89600640 1 -- 172.21.15.39:0/3402510385 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa9401e910 con 0x5602ca2abb50 2024-03-18T16:06:52.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.708+0000 7ffa89600640 1 -- 172.21.15.39:0/3402510385 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 16) ==== 8342+0+0 (secure 0 0 0) 0x7ffa840860b0 con 0x5602ca2abb50 2024-03-18T16:06:52.716 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.708+0000 7ffaa562c740 10 client.24689 fetch_fsmap finished waiting for FSMap version 16 2024-03-18T16:06:52.716 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.708+0000 7ffaa562c740 10 client.24689 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-18T16:06:52.716 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.708+0000 7ffaa562c740 20 client.24689 populate_metadata read hostname 'smithi039' 2024-03-18T16:06:52.716 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.708+0000 7ffaa562c740 1 --2- 172.21.15.39:0/3402510385 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5602ca32cc20 0x5602ca32b930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:52.716 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.708+0000 7ffaa562c740 4 client.24689 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T16:06:52.716 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.708+0000 7ffaa562c740 1 -- 172.21.15.39:0/3402510385 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "status"}) -- 0x5602ca21d190 con 0x5602ca32cc20 2024-03-18T16:06:52.716 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.708+0000 7ffa9b400640 1 --2- 172.21.15.39:0/3402510385 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5602ca32cc20 0x5602ca32b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:52.716 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.709+0000 7ffa9b400640 1 --2- 172.21.15.39:0/3402510385 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5602ca32cc20 0x5602ca32b930 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a9fb5267bdd383f7 server_cookie=b2627617f2e72f1b in_seq=0 out_seq=0 2024-03-18T16:06:52.716 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.709+0000 7ffa89600640 10 client.24689 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:06:52.716 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.709+0000 7ffa89600640 1 -- 172.21.15.39:0/3402510385 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+325 (crc 0 0 0) 0x5602ca21d190 con 0x5602ca32cc20 2024-03-18T16:06:52.716 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.709+0000 7ffa89600640 10 client.24689 handle_command_reply: tid=0 2024-03-18T16:06:52.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.709+0000 7ffaa562c740 1 client.24689 shutdown 2024-03-18T16:06:52.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.709+0000 7ffaa562c740 1 -- 172.21.15.39:0/3402510385 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5602ca2abb50 msgr2=0x5602ca2b8b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:52.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.709+0000 7ffaa562c740 1 --2- 172.21.15.39:0/3402510385 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5602ca2abb50 0x5602ca2b8b50 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ffa84097c40 tx=0x7ffa840d12e0 comp rx=0 tx=0).stop 2024-03-18T16:06:52.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.709+0000 7ffaa562c740 1 -- 172.21.15.39:0/3402510385 shutdown_connections 2024-03-18T16:06:52.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.709+0000 7ffaa562c740 1 --2- 172.21.15.39:0/3402510385 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5602ca32cc20 0x5602ca32b930 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:52.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.709+0000 7ffaa562c740 1 --2- 172.21.15.39:0/3402510385 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5602ca2abb50 0x5602ca2b8b50 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:52.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.709+0000 7ffa89600640 0 client.24689 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:06:52.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.709+0000 7ffaa562c740 1 --2- 172.21.15.39:0/3402510385 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5602ca2b0cc0 0x5602ca2bbb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:52.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.709+0000 7ffaa562c740 1 --2- 172.21.15.39:0/3402510385 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5602ca2ab160 0x5602ca2b8610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:52.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.709+0000 7ffaa562c740 1 -- 172.21.15.39:0/3402510385 >> 172.21.15.39:0/3402510385 conn(0x5602ca2a66d0 msgr2=0x5602ca2aa1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:52.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.709+0000 7ffaa562c740 1 -- 172.21.15.39:0/3402510385 shutdown_connections 2024-03-18T16:06:52.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.710+0000 7ffaa562c740 1 -- 172.21.15.39:0/3402510385 wait complete. 2024-03-18T16:06:52.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.710+0000 7ffaa562c740 20 client.24689 trim_cache size 0 max 16384 2024-03-18T16:06:52.718 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-03-18T16:06:52.718 INFO:teuthology.orchestra.run.smithi039.stdout: "cluster_fsid": "1892d8fc-e540-11ee-95c9-87774f69a715", 2024-03-18T16:06:52.718 INFO:teuthology.orchestra.run.smithi039.stdout: "whoami": 0, 2024-03-18T16:06:52.718 INFO:teuthology.orchestra.run.smithi039.stdout: "id": 24575, 2024-03-18T16:06:52.718 INFO:teuthology.orchestra.run.smithi039.stdout: "want_state": "up:active", 2024-03-18T16:06:52.718 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "up:active", 2024-03-18T16:06:52.718 INFO:teuthology.orchestra.run.smithi039.stdout: "fs_name": "cephfs", 2024-03-18T16:06:52.718 INFO:teuthology.orchestra.run.smithi039.stdout: "rank_uptime": 7.885367864, 2024-03-18T16:06:52.718 INFO:teuthology.orchestra.run.smithi039.stdout: "mdsmap_epoch": 16, 2024-03-18T16:06:52.718 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_epoch": 89, 2024-03-18T16:06:52.718 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_epoch_barrier": 89, 2024-03-18T16:06:52.719 INFO:teuthology.orchestra.run.smithi039.stdout: "uptime": 15.284474622999999 2024-03-18T16:06:52.719 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-03-18T16:06:52.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.710+0000 7ffaa2800640 1 -- 172.21.15.39:0/4084391263 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ffa800a2800 msgr2=0x7ffa800a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:52.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.710+0000 7ffaa2800640 1 --2- 172.21.15.39:0/4084391263 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ffa800a2800 0x7ffa800a4cc0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x5602ca2ac3b0 tx=0x7ffa94009500 comp rx=0 tx=0).stop 2024-03-18T16:06:52.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.710+0000 7ffaa2800640 1 -- 172.21.15.39:0/4084391263 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffa9c1018f0 msgr2=0x7ffa9c19a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:52.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.710+0000 7ffaa2800640 1 --2- 172.21.15.39:0/4084391263 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffa9c1018f0 0x7ffa9c19a2b0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7ffa8400e9f0 tx=0x7ffa8400eeb0 comp rx=0 tx=0).stop 2024-03-18T16:06:52.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.710+0000 7ffaa2800640 1 -- 172.21.15.39:0/4084391263 shutdown_connections 2024-03-18T16:06:52.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.710+0000 7ffaa2800640 1 --2- 172.21.15.39:0/4084391263 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa9c1022b0 0x7ffa9c1a1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:52.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.710+0000 7ffaa2800640 1 --2- 172.21.15.39:0/4084391263 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ffa800a2800 0x7ffa800a4cc0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:52.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.710+0000 7ffaa2800640 1 --2- 172.21.15.39:0/4084391263 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffa9c1018f0 0x7ffa9c19a2b0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:52.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.710+0000 7ffaa2800640 1 --2- 172.21.15.39:0/4084391263 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffa9c100700 0x7ffa9c199d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:52.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.710+0000 7ffaa2800640 1 -- 172.21.15.39:0/4084391263 >> 172.21.15.39:0/4084391263 conn(0x7ffa9c0fbc70 msgr2=0x7ffa9c0fd7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:52.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.710+0000 7ffaa2800640 1 -- 172.21.15.39:0/4084391263 shutdown_connections 2024-03-18T16:06:52.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:52.711+0000 7ffaa2800640 1 -- 172.21.15.39:0/4084391263 wait complete. 2024-03-18T16:06:52.730 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-18T16:06:52.730 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-18T16:06:52.730 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph fs ls 2024-03-18T16:06:53.188 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T16:06:53.335 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:53 smithi039 ceph-mon[31721]: pgmap v322: 129 pgs: 6 creating+peering, 106 active+clean, 17 unknown; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s wr, 8 op/s 2024-03-18T16:06:53.335 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:53 smithi039 ceph-mon[31721]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-18T16:06:53.335 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:53 smithi039 ceph-mon[31721]: Cluster is now healthy 2024-03-18T16:06:53.335 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:53 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/968245156' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:06:53.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:53 smithi196 ceph-mon[36307]: pgmap v322: 129 pgs: 6 creating+peering, 106 active+clean, 17 unknown; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s wr, 8 op/s 2024-03-18T16:06:53.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:53 smithi196 ceph-mon[36307]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-18T16:06:53.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:53 smithi196 ceph-mon[36307]: Cluster is now healthy 2024-03-18T16:06:53.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:53 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/968245156' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:06:53.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:53 smithi194 ceph-mon[36200]: pgmap v322: 129 pgs: 6 creating+peering, 106 active+clean, 17 unknown; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s wr, 8 op/s 2024-03-18T16:06:53.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:53 smithi194 ceph-mon[36200]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-18T16:06:53.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:53 smithi194 ceph-mon[36200]: Cluster is now healthy 2024-03-18T16:06:53.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:53 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/968245156' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:06:55.288 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:55 smithi039 ceph-mon[31721]: pgmap v323: 129 pgs: 4 creating+peering, 125 active+clean; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s wr, 5 op/s 2024-03-18T16:06:55.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:55 smithi196 ceph-mon[36307]: pgmap v323: 129 pgs: 4 creating+peering, 125 active+clean; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s wr, 5 op/s 2024-03-18T16:06:55.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:55 smithi194 ceph-mon[36200]: pgmap v323: 129 pgs: 4 creating+peering, 125 active+clean; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s wr, 5 op/s 2024-03-18T16:06:55.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:55.582+0000 7f4b50a00700 1 -- 172.21.15.39:0/396684669 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4b4c06a920 msgr2=0x7f4b4c10f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:55.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:55.582+0000 7f4b50a00700 1 --2- 172.21.15.39:0/396684669 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4b4c06a920 0x7f4b4c10f160 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4b4400b0a0 tx=0x7f4b4400b3b0 comp rx=0 tx=0).stop 2024-03-18T16:06:55.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:55.582+0000 7f4b50a00700 1 -- 172.21.15.39:0/396684669 shutdown_connections 2024-03-18T16:06:55.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:55.582+0000 7f4b50a00700 1 --2- 172.21.15.39:0/396684669 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4b4c06a920 0x7f4b4c10f160 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:55.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:55.582+0000 7f4b50a00700 1 --2- 172.21.15.39:0/396684669 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4b4c108320 0x7f4b4c108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:55.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:55.582+0000 7f4b50a00700 1 --2- 172.21.15.39:0/396684669 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4b4c069f80 0x7f4b4c06a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:55.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:55.582+0000 7f4b50a00700 1 -- 172.21.15.39:0/396684669 >> 172.21.15.39:0/396684669 conn(0x7f4b4c078100 msgr2=0x7f4b4c078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:55.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:55.582+0000 7f4b50a00700 1 -- 172.21.15.39:0/396684669 shutdown_connections 2024-03-18T16:06:55.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:55.582+0000 7f4b50a00700 1 -- 172.21.15.39:0/396684669 wait complete. 2024-03-18T16:06:55.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:55.583+0000 7f4b50a00700 1 Processor -- start 2024-03-18T16:06:55.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:55.584+0000 7f4b50a00700 1 -- start start 2024-03-18T16:06:55.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:55.584+0000 7f4b50a00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4b4c069f80 0x7f4b4c1a08a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:55.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:55.584+0000 7f4b50a00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4b4c06a920 0x7f4b4c1a0de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:55.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:55.584+0000 7f4b50a00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4b4c108320 0x7f4b4c19aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:55.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:55.584+0000 7f4b50a00700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f4b4c1125f0 con 0x7f4b4c108320 2024-03-18T16:06:55.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:55.584+0000 7f4b50a00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f4b4c112470 con 0x7f4b4c069f80 2024-03-18T16:06:55.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:55.584+0000 7f4b50a00700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f4b4c112770 con 0x7f4b4c06a920 2024-03-18T16:06:55.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:55.584+0000 7f4b49e00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4b4c069f80 0x7f4b4c1a08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:55.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:55.585+0000 7f4b49e00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4b4c069f80 0x7f4b4c1a08a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:36966/0 (socket says 172.21.15.39:36966) 2024-03-18T16:06:55.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:55.585+0000 7f4b49e00700 1 -- 172.21.15.39:0/1279778231 learned_addr learned my addr 172.21.15.39:0/1279778231 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:06:55.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:55.585+0000 7f4b49400700 1 --2- 172.21.15.39:0/1279778231 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4b4c06a920 0x7f4b4c1a0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:55.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:55.585+0000 7f4b4a800700 1 --2- 172.21.15.39:0/1279778231 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4b4c108320 0x7f4b4c19aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:55.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:55.585+0000 7f4b49e00700 1 -- 172.21.15.39:0/1279778231 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4b4c06a920 msgr2=0x7f4b4c1a0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:55.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:55.585+0000 7f4b49e00700 1 --2- 172.21.15.39:0/1279778231 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4b4c06a920 0x7f4b4c1a0de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:55.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:55.585+0000 7f4b49e00700 1 -- 172.21.15.39:0/1279778231 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4b4c108320 msgr2=0x7f4b4c19aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:55.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:55.585+0000 7f4b49e00700 1 --2- 172.21.15.39:0/1279778231 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4b4c108320 0x7f4b4c19aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:55.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:55.585+0000 7f4b49e00700 1 -- 172.21.15.39:0/1279778231 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b4c19b1c0 con 0x7f4b4c069f80 2024-03-18T16:06:55.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:55.585+0000 7f4b4a800700 1 --2- 172.21.15.39:0/1279778231 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4b4c108320 0x7f4b4c19aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:06:55.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:55.585+0000 7f4b49e00700 1 --2- 172.21.15.39:0/1279778231 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4b4c069f80 0x7f4b4c1a08a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4b3c00d980 tx=0x7f4b3c00dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:55.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:55.586+0000 7f4b42a00700 1 -- 172.21.15.39:0/1279778231 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b3c01c070 con 0x7f4b4c069f80 2024-03-18T16:06:55.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:55.586+0000 7f4b42a00700 1 -- 172.21.15.39:0/1279778231 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b3c016d60 con 0x7f4b4c069f80 2024-03-18T16:06:55.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:55.586+0000 7f4b42a00700 1 -- 172.21.15.39:0/1279778231 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b3c009990 con 0x7f4b4c069f80 2024-03-18T16:06:55.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:55.586+0000 7f4b50a00700 1 -- 172.21.15.39:0/1279778231 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b4c19b4b0 con 0x7f4b4c069f80 2024-03-18T16:06:55.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:55.586+0000 7f4b50a00700 1 -- 172.21.15.39:0/1279778231 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b4c1090a0 con 0x7f4b4c069f80 2024-03-18T16:06:55.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:55.587+0000 7f4b50a00700 1 -- 172.21.15.39:0/1279778231 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b4c068f90 con 0x7f4b4c069f80 2024-03-18T16:06:55.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:55.589+0000 7f4b42a00700 1 -- 172.21.15.39:0/1279778231 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f4b3c004f30 con 0x7f4b4c069f80 2024-03-18T16:06:55.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:55.590+0000 7f4b42a00700 1 --2- 172.21.15.39:0/1279778231 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f4b300a2910 0x7f4b300a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:06:55.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:55.590+0000 7f4b42a00700 1 -- 172.21.15.39:0/1279778231 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f4b3c0d69e0 con 0x7f4b4c069f80 2024-03-18T16:06:55.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:55.590+0000 7f4b49400700 1 --2- 172.21.15.39:0/1279778231 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f4b300a2910 0x7f4b300a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:06:55.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:55.591+0000 7f4b49400700 1 --2- 172.21.15.39:0/1279778231 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f4b300a2910 0x7f4b300a4dd0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f4b4c103220 tx=0x7f4b38009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:06:55.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:55.593+0000 7f4b42a00700 1 -- 172.21.15.39:0/1279778231 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4b3c0a37e0 con 0x7f4b4c069f80 2024-03-18T16:06:55.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:55.867+0000 7f4b50a00700 1 -- 172.21.15.39:0/1279778231 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f4b4c109360 con 0x7f4b4c069f80 2024-03-18T16:06:55.870 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:55.867+0000 7f4b42a00700 1 -- 172.21.15.39:0/1279778231 <== mon.2 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) ==== 53+0+88 (secure 0 0 0) 0x7f4b3c0a30e0 con 0x7f4b4c069f80 2024-03-18T16:06:55.870 INFO:teuthology.orchestra.run.smithi039.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-18T16:06:55.873 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:55.870+0000 7f4b50a00700 1 -- 172.21.15.39:0/1279778231 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f4b300a2910 msgr2=0x7f4b300a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:55.873 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:55.870+0000 7f4b50a00700 1 --2- 172.21.15.39:0/1279778231 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f4b300a2910 0x7f4b300a4dd0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f4b4c103220 tx=0x7f4b38009260 comp rx=0 tx=0).stop 2024-03-18T16:06:55.873 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:55.870+0000 7f4b50a00700 1 -- 172.21.15.39:0/1279778231 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4b4c069f80 msgr2=0x7f4b4c1a08a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:06:55.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:55.870+0000 7f4b50a00700 1 --2- 172.21.15.39:0/1279778231 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4b4c069f80 0x7f4b4c1a08a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4b3c00d980 tx=0x7f4b3c00dc90 comp rx=0 tx=0).stop 2024-03-18T16:06:55.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:55.871+0000 7f4b50a00700 1 -- 172.21.15.39:0/1279778231 shutdown_connections 2024-03-18T16:06:55.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:55.871+0000 7f4b50a00700 1 --2- 172.21.15.39:0/1279778231 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f4b300a2910 0x7f4b300a4dd0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:55.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:55.871+0000 7f4b50a00700 1 --2- 172.21.15.39:0/1279778231 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4b4c108320 0x7f4b4c19aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:55.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:55.871+0000 7f4b50a00700 1 --2- 172.21.15.39:0/1279778231 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4b4c069f80 0x7f4b4c1a08a0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:55.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:55.871+0000 7f4b50a00700 1 --2- 172.21.15.39:0/1279778231 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4b4c06a920 0x7f4b4c1a0de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:06:55.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:55.871+0000 7f4b50a00700 1 -- 172.21.15.39:0/1279778231 >> 172.21.15.39:0/1279778231 conn(0x7f4b4c078100 msgr2=0x7f4b4c10d4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:06:55.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:55.871+0000 7f4b50a00700 1 -- 172.21.15.39:0/1279778231 shutdown_connections 2024-03-18T16:06:55.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:55.871+0000 7f4b50a00700 1 -- 172.21.15.39:0/1279778231 wait complete. 2024-03-18T16:06:56.311 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:56 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/1279778231' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-18T16:06:56.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:56 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/1279778231' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-18T16:06:56.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:56 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/1279778231' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-18T16:06:56.862 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-18T16:06:56.862 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-18T16:06:56.862 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi039.front.sepia.ceph.com 2024-03-18T16:06:56.862 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-03-18T16:06:56.862 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-18T16:06:56.862 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-18T16:06:56.863 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-18T16:06:56.863 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-18T16:06:56.863 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-03-18T16:06:56.863 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T16:06:56.863 DEBUG:teuthology.orchestra.run.smithi039:> ip addr 2024-03-18T16:06:56.921 INFO:teuthology.orchestra.run.smithi039.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-18T16:06:56.921 INFO:teuthology.orchestra.run.smithi039.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-18T16:06:56.921 INFO:teuthology.orchestra.run.smithi039.stdout: inet 127.0.0.1/8 scope host lo 2024-03-18T16:06:56.921 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-03-18T16:06:56.921 INFO:teuthology.orchestra.run.smithi039.stdout: inet6 ::1/128 scope host 2024-03-18T16:06:56.921 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-03-18T16:06:56.921 INFO:teuthology.orchestra.run.smithi039.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T16:06:56.921 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 0c:c4:7a:6b:fe:c6 brd ff:ff:ff:ff:ff:ff 2024-03-18T16:06:56.922 INFO:teuthology.orchestra.run.smithi039.stdout: altname enp7s0f0 2024-03-18T16:06:56.922 INFO:teuthology.orchestra.run.smithi039.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T16:06:56.922 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 0c:c4:7a:6b:fe:c7 brd ff:ff:ff:ff:ff:ff 2024-03-18T16:06:56.922 INFO:teuthology.orchestra.run.smithi039.stdout: altname enp7s0f1 2024-03-18T16:06:56.922 INFO:teuthology.orchestra.run.smithi039.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-18T16:06:56.922 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 0c:c4:7a:bd:13:70 brd ff:ff:ff:ff:ff:ff 2024-03-18T16:06:56.922 INFO:teuthology.orchestra.run.smithi039.stdout: altname enp1s0f0 2024-03-18T16:06:56.922 INFO:teuthology.orchestra.run.smithi039.stdout: inet 172.21.15.39/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-18T16:06:56.922 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft 42020sec preferred_lft 42020sec 2024-03-18T16:06:56.922 INFO:teuthology.orchestra.run.smithi039.stdout: inet6 fe80::ec4:7aff:febd:1370/64 scope link noprefixroute 2024-03-18T16:06:56.922 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-03-18T16:06:56.922 INFO:teuthology.orchestra.run.smithi039.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T16:06:56.922 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 0c:c4:7a:bd:13:71 brd ff:ff:ff:ff:ff:ff 2024-03-18T16:06:56.923 INFO:teuthology.orchestra.run.smithi039.stdout: altname enp1s0f1 2024-03-18T16:06:56.924 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-18T16:06:56.924 DEBUG:teuthology.orchestra.run.smithi039:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T16:06:56.924 DEBUG:teuthology.orchestra.run.smithi039:> set -e 2024-03-18T16:06:56.924 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip link add name ceph-brx type bridge 2024-03-18T16:06:56.924 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip addr flush dev ceph-brx 2024-03-18T16:06:56.924 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip link set ceph-brx up 2024-03-18T16:06:56.924 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-18T16:06:56.924 DEBUG:teuthology.orchestra.run.smithi039:> ') 2024-03-18T16:06:57.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:57 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T16:06:57.335 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:57 smithi039 ceph-mon[31721]: pgmap v324: 129 pgs: 129 active+clean; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 1 op/s 2024-03-18T16:06:57.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:57 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T16:06:57.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:57 smithi196 ceph-mon[36307]: pgmap v324: 129 pgs: 129 active+clean; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 1 op/s 2024-03-18T16:06:57.483 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T16:06:57.483 DEBUG:teuthology.orchestra.run.smithi039:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-18T16:06:57.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:57 smithi194 ceph-mon[36200]: pgmap v324: 129 pgs: 129 active+clean; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s wr, 1 op/s 2024-03-18T16:06:57.559 INFO:teuthology.orchestra.run.smithi039.stdout:1 2024-03-18T16:06:57.563 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T16:06:57.563 DEBUG:teuthology.orchestra.run.smithi039:> route 2024-03-18T16:06:57.619 INFO:teuthology.orchestra.run.smithi039.stdout:Kernel IP routing table 2024-03-18T16:06:57.619 INFO:teuthology.orchestra.run.smithi039.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-18T16:06:57.619 INFO:teuthology.orchestra.run.smithi039.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-18T16:06:57.619 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-18T16:06:57.619 INFO:teuthology.orchestra.run.smithi039.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-18T16:06:57.620 DEBUG:teuthology.orchestra.run.smithi039:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T16:06:57.620 DEBUG:teuthology.orchestra.run.smithi039:> set -e 2024-03-18T16:06:57.620 DEBUG:teuthology.orchestra.run.smithi039:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-18T16:06:57.620 DEBUG:teuthology.orchestra.run.smithi039:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-18T16:06:57.620 DEBUG:teuthology.orchestra.run.smithi039:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-18T16:06:57.620 DEBUG:teuthology.orchestra.run.smithi039:> ') 2024-03-18T16:06:57.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:57 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T16:06:58.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:58 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T16:06:58.480 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T16:06:58.481 DEBUG:teuthology.orchestra.run.smithi039:> ip netns list 2024-03-18T16:06:58.538 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T16:06:58.538 DEBUG:teuthology.orchestra.run.smithi039:> ip netns list-id 2024-03-18T16:06:58.593 DEBUG:teuthology.orchestra.run.smithi039:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T16:06:58.593 DEBUG:teuthology.orchestra.run.smithi039:> set -e 2024-03-18T16:06:58.594 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-18T16:06:58.594 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-03-18T16:06:58.594 DEBUG:teuthology.orchestra.run.smithi039:> ') 2024-03-18T16:06:58.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:58 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T16:06:58.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:58 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T16:06:58.818 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-03-18T16:06:58.818 DEBUG:teuthology.orchestra.run.smithi039:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T16:06:58.819 DEBUG:teuthology.orchestra.run.smithi039:> set -e 2024-03-18T16:06:58.819 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-03-18T16:06:58.819 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-18T16:06:58.819 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-03-18T16:06:58.819 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-03-18T16:06:58.819 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-03-18T16:06:58.819 DEBUG:teuthology.orchestra.run.smithi039:> ') 2024-03-18T16:06:58.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:58 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T16:06:59.186 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:59 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T16:06:59.196 DEBUG:teuthology.orchestra.run.smithi039:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T16:06:59.196 DEBUG:teuthology.orchestra.run.smithi039:> set -e 2024-03-18T16:06:59.197 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip link set brx.0 up 2024-03-18T16:06:59.197 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip link set dev brx.0 master ceph-brx 2024-03-18T16:06:59.197 DEBUG:teuthology.orchestra.run.smithi039:> ') 2024-03-18T16:06:59.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:59 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T16:06:59.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:06:59 smithi039 ceph-mon[31721]: pgmap v325: 129 pgs: 129 active+clean; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 361 B/s wr, 0 op/s 2024-03-18T16:06:59.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:06:59 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T16:06:59.438 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-03-18T16:06:59.439 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T16:06:59.439 DEBUG:teuthology.orchestra.run.smithi039:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-03-18T16:06:59.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:06:59 smithi196 ceph-mon[36307]: pgmap v325: 129 pgs: 129 active+clean; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 361 B/s wr, 0 op/s 2024-03-18T16:06:59.484 INFO:teuthology.orchestra.run.smithi039.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-03-18T16:06:59.484 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T16:06:59.484 DEBUG:teuthology.orchestra.run.smithi039:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-03-18T16:06:59.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:06:59 smithi194 ceph-mon[36200]: pgmap v325: 129 pgs: 129 active+clean; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 361 B/s wr, 0 op/s 2024-03-18T16:06:59.578 DEBUG:teuthology.orchestra.run.smithi039:> sudo modprobe fuse 2024-03-18T16:06:59.680 DEBUG:teuthology.orchestra.run.smithi039:> cat /proc/self/mounts | awk '{print $2}' 2024-03-18T16:06:59.731 INFO:teuthology.orchestra.run.smithi039.stdout:/proc 2024-03-18T16:06:59.731 INFO:teuthology.orchestra.run.smithi039.stdout:/sys 2024-03-18T16:06:59.731 INFO:teuthology.orchestra.run.smithi039.stdout:/dev 2024-03-18T16:06:59.731 INFO:teuthology.orchestra.run.smithi039.stdout:/sys/kernel/security 2024-03-18T16:06:59.732 INFO:teuthology.orchestra.run.smithi039.stdout:/dev/shm 2024-03-18T16:06:59.732 INFO:teuthology.orchestra.run.smithi039.stdout:/dev/pts 2024-03-18T16:06:59.732 INFO:teuthology.orchestra.run.smithi039.stdout:/run 2024-03-18T16:06:59.732 INFO:teuthology.orchestra.run.smithi039.stdout:/sys/fs/cgroup 2024-03-18T16:06:59.732 INFO:teuthology.orchestra.run.smithi039.stdout:/sys/fs/pstore 2024-03-18T16:06:59.732 INFO:teuthology.orchestra.run.smithi039.stdout:/sys/fs/bpf 2024-03-18T16:06:59.732 INFO:teuthology.orchestra.run.smithi039.stdout:/ 2024-03-18T16:06:59.732 INFO:teuthology.orchestra.run.smithi039.stdout:/sys/fs/selinux 2024-03-18T16:06:59.732 INFO:teuthology.orchestra.run.smithi039.stdout:/proc/sys/fs/binfmt_misc 2024-03-18T16:06:59.732 INFO:teuthology.orchestra.run.smithi039.stdout:/dev/mqueue 2024-03-18T16:06:59.732 INFO:teuthology.orchestra.run.smithi039.stdout:/dev/hugepages 2024-03-18T16:06:59.732 INFO:teuthology.orchestra.run.smithi039.stdout:/sys/kernel/debug 2024-03-18T16:06:59.732 INFO:teuthology.orchestra.run.smithi039.stdout:/sys/kernel/tracing 2024-03-18T16:06:59.733 INFO:teuthology.orchestra.run.smithi039.stdout:/sys/fs/fuse/connections 2024-03-18T16:06:59.733 INFO:teuthology.orchestra.run.smithi039.stdout:/sys/kernel/config 2024-03-18T16:06:59.733 INFO:teuthology.orchestra.run.smithi039.stdout:/run/credentials/systemd-sysctl.service 2024-03-18T16:06:59.733 INFO:teuthology.orchestra.run.smithi039.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-18T16:06:59.733 INFO:teuthology.orchestra.run.smithi039.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-18T16:06:59.733 INFO:teuthology.orchestra.run.smithi039.stdout:/var/lib/nfs/rpc_pipefs 2024-03-18T16:06:59.733 INFO:teuthology.orchestra.run.smithi039.stdout:/run/user/1000 2024-03-18T16:06:59.733 INFO:teuthology.orchestra.run.smithi039.stdout:/var/lib/ceph 2024-03-18T16:06:59.733 INFO:teuthology.orchestra.run.smithi039.stdout:/var/lib/containers/storage/overlay 2024-03-18T16:06:59.733 INFO:teuthology.orchestra.run.smithi039.stdout:/var/lib/containers/storage/overlay/2409865ff32c9c04587444b8e37d7769407dc287c8a03c1342b9a86a150759fe/merged 2024-03-18T16:06:59.733 INFO:teuthology.orchestra.run.smithi039.stdout:/var/lib/containers/storage/overlay/b7654a442e809015b7eb859164a05d70d130cef137883e73bc780926e96a7643/merged 2024-03-18T16:06:59.733 INFO:teuthology.orchestra.run.smithi039.stdout:/run/user/0 2024-03-18T16:06:59.733 INFO:teuthology.orchestra.run.smithi039.stdout:/proc/sys/fs/binfmt_misc 2024-03-18T16:06:59.733 INFO:teuthology.orchestra.run.smithi039.stdout:/var/lib/containers/storage/overlay/90a75a6e76063e82797b9956dba964f848eb7a73e7d9ab973c1b7a66f477400a/merged 2024-03-18T16:06:59.733 INFO:teuthology.orchestra.run.smithi039.stdout:/var/lib/containers/storage/overlay/2b8cf1bae4484043534c37887a6b7c605a6069d23ede3575cc2a3150261fce84/merged 2024-03-18T16:06:59.734 INFO:teuthology.orchestra.run.smithi039.stdout:/var/lib/containers/storage/overlay/bfd06074c846d81c6a55892d973e556317f738cc99c70b42a301b0490c8723f9/merged 2024-03-18T16:06:59.734 INFO:teuthology.orchestra.run.smithi039.stdout:/var/lib/containers/storage/overlay/741b8fc3f666ba6902debb4c8fa112e0433ee386bbe9a4ba902c1eb92c4f29d3/merged 2024-03-18T16:06:59.734 INFO:teuthology.orchestra.run.smithi039.stdout:/var/lib/containers/storage/overlay/11187decc0d15d6eedc5740c05776f48a05f50c783e2bd23d09866c7af609970/merged 2024-03-18T16:06:59.734 INFO:teuthology.orchestra.run.smithi039.stdout:/var/lib/containers/storage/overlay/b95dd69952c1e2dfe2418a47cb70cc4c3e3325baf4dc7797ff6687961e7d4721/merged 2024-03-18T16:06:59.734 INFO:teuthology.orchestra.run.smithi039.stdout:/var/lib/containers/storage/overlay/43780c00f7e5b493b95793fb504686b5951cd76a9bebd45c531e6517ef630e5b/merged 2024-03-18T16:06:59.734 INFO:teuthology.orchestra.run.smithi039.stdout:/var/lib/containers/storage/overlay/013887cbd11a2f3962fdee72b04569a2833163420edbd32b8b3ae20055104d9b/merged 2024-03-18T16:06:59.734 INFO:teuthology.orchestra.run.smithi039.stdout:/run/netns 2024-03-18T16:06:59.734 INFO:teuthology.orchestra.run.smithi039.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-18T16:06:59.734 INFO:teuthology.orchestra.run.smithi039.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-18T16:06:59.735 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T16:06:59.735 DEBUG:teuthology.orchestra.run.smithi039:> ls /sys/fs/fuse/connections 2024-03-18T16:06:59.782 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-18T16:06:59.782 DEBUG:teuthology.orchestra.run.smithi039:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-03-18T16:06:59.830 DEBUG:teuthology.orchestra.run.smithi039:> sudo modprobe fuse 2024-03-18T16:06:59.911 DEBUG:teuthology.orchestra.run.smithi039:> cat /proc/self/mounts | awk '{print $2}' 2024-03-18T16:06:59.956 INFO:teuthology.orchestra.run.smithi039.stdout:/proc 2024-03-18T16:06:59.956 INFO:teuthology.orchestra.run.smithi039.stdout:/sys 2024-03-18T16:06:59.956 INFO:teuthology.orchestra.run.smithi039.stdout:/dev 2024-03-18T16:06:59.956 INFO:teuthology.orchestra.run.smithi039.stdout:/sys/kernel/security 2024-03-18T16:06:59.957 INFO:teuthology.orchestra.run.smithi039.stdout:/dev/shm 2024-03-18T16:06:59.957 INFO:teuthology.orchestra.run.smithi039.stdout:/dev/pts 2024-03-18T16:06:59.957 INFO:teuthology.orchestra.run.smithi039.stdout:/run 2024-03-18T16:06:59.957 INFO:teuthology.orchestra.run.smithi039.stdout:/sys/fs/cgroup 2024-03-18T16:06:59.957 INFO:teuthology.orchestra.run.smithi039.stdout:/sys/fs/pstore 2024-03-18T16:06:59.957 INFO:teuthology.orchestra.run.smithi039.stdout:/sys/fs/bpf 2024-03-18T16:06:59.957 INFO:teuthology.orchestra.run.smithi039.stdout:/ 2024-03-18T16:06:59.957 INFO:teuthology.orchestra.run.smithi039.stdout:/sys/fs/selinux 2024-03-18T16:06:59.957 INFO:teuthology.orchestra.run.smithi039.stdout:/proc/sys/fs/binfmt_misc 2024-03-18T16:06:59.957 INFO:teuthology.orchestra.run.smithi039.stdout:/dev/mqueue 2024-03-18T16:06:59.957 INFO:teuthology.orchestra.run.smithi039.stdout:/dev/hugepages 2024-03-18T16:06:59.957 INFO:teuthology.orchestra.run.smithi039.stdout:/sys/kernel/debug 2024-03-18T16:06:59.957 INFO:teuthology.orchestra.run.smithi039.stdout:/sys/kernel/tracing 2024-03-18T16:06:59.957 INFO:teuthology.orchestra.run.smithi039.stdout:/sys/fs/fuse/connections 2024-03-18T16:06:59.957 INFO:teuthology.orchestra.run.smithi039.stdout:/sys/kernel/config 2024-03-18T16:06:59.958 INFO:teuthology.orchestra.run.smithi039.stdout:/run/credentials/systemd-sysctl.service 2024-03-18T16:06:59.958 INFO:teuthology.orchestra.run.smithi039.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-18T16:06:59.958 INFO:teuthology.orchestra.run.smithi039.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-18T16:06:59.958 INFO:teuthology.orchestra.run.smithi039.stdout:/var/lib/nfs/rpc_pipefs 2024-03-18T16:06:59.958 INFO:teuthology.orchestra.run.smithi039.stdout:/run/user/1000 2024-03-18T16:06:59.958 INFO:teuthology.orchestra.run.smithi039.stdout:/var/lib/ceph 2024-03-18T16:06:59.958 INFO:teuthology.orchestra.run.smithi039.stdout:/var/lib/containers/storage/overlay 2024-03-18T16:06:59.958 INFO:teuthology.orchestra.run.smithi039.stdout:/var/lib/containers/storage/overlay/2409865ff32c9c04587444b8e37d7769407dc287c8a03c1342b9a86a150759fe/merged 2024-03-18T16:06:59.958 INFO:teuthology.orchestra.run.smithi039.stdout:/var/lib/containers/storage/overlay/b7654a442e809015b7eb859164a05d70d130cef137883e73bc780926e96a7643/merged 2024-03-18T16:06:59.958 INFO:teuthology.orchestra.run.smithi039.stdout:/run/user/0 2024-03-18T16:06:59.958 INFO:teuthology.orchestra.run.smithi039.stdout:/proc/sys/fs/binfmt_misc 2024-03-18T16:06:59.958 INFO:teuthology.orchestra.run.smithi039.stdout:/var/lib/containers/storage/overlay/90a75a6e76063e82797b9956dba964f848eb7a73e7d9ab973c1b7a66f477400a/merged 2024-03-18T16:06:59.958 INFO:teuthology.orchestra.run.smithi039.stdout:/var/lib/containers/storage/overlay/2b8cf1bae4484043534c37887a6b7c605a6069d23ede3575cc2a3150261fce84/merged 2024-03-18T16:06:59.958 INFO:teuthology.orchestra.run.smithi039.stdout:/var/lib/containers/storage/overlay/bfd06074c846d81c6a55892d973e556317f738cc99c70b42a301b0490c8723f9/merged 2024-03-18T16:06:59.958 INFO:teuthology.orchestra.run.smithi039.stdout:/var/lib/containers/storage/overlay/741b8fc3f666ba6902debb4c8fa112e0433ee386bbe9a4ba902c1eb92c4f29d3/merged 2024-03-18T16:06:59.959 INFO:teuthology.orchestra.run.smithi039.stdout:/var/lib/containers/storage/overlay/11187decc0d15d6eedc5740c05776f48a05f50c783e2bd23d09866c7af609970/merged 2024-03-18T16:06:59.959 INFO:teuthology.orchestra.run.smithi039.stdout:/var/lib/containers/storage/overlay/b95dd69952c1e2dfe2418a47cb70cc4c3e3325baf4dc7797ff6687961e7d4721/merged 2024-03-18T16:06:59.959 INFO:teuthology.orchestra.run.smithi039.stdout:/var/lib/containers/storage/overlay/43780c00f7e5b493b95793fb504686b5951cd76a9bebd45c531e6517ef630e5b/merged 2024-03-18T16:06:59.959 INFO:teuthology.orchestra.run.smithi039.stdout:/var/lib/containers/storage/overlay/013887cbd11a2f3962fdee72b04569a2833163420edbd32b8b3ae20055104d9b/merged 2024-03-18T16:06:59.959 INFO:teuthology.orchestra.run.smithi039.stdout:/run/netns 2024-03-18T16:06:59.959 INFO:teuthology.orchestra.run.smithi039.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-18T16:06:59.959 INFO:teuthology.orchestra.run.smithi039.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-18T16:06:59.960 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T16:06:59.960 DEBUG:teuthology.orchestra.run.smithi039:> ls /sys/fs/fuse/connections 2024-03-18T16:07:00.052 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi039.stderr:2024-03-18T16:07:00.050+0000 7efe46785480 -1 init, newargv = 0x557b1f74a5e0 newargc=15 2024-03-18T16:07:00.053 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi039.stderr:ceph-fuse[73442]: starting ceph client 2024-03-18T16:07:00.063 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi039.stderr:ceph-fuse[73442]: starting fuse 2024-03-18T16:07:01.150 DEBUG:teuthology.orchestra.run.smithi039:> sudo modprobe fuse 2024-03-18T16:07:01.246 DEBUG:teuthology.orchestra.run.smithi039:> cat /proc/self/mounts | awk '{print $2}' 2024-03-18T16:07:01.300 INFO:teuthology.orchestra.run.smithi039.stdout:/proc 2024-03-18T16:07:01.300 INFO:teuthology.orchestra.run.smithi039.stdout:/sys 2024-03-18T16:07:01.300 INFO:teuthology.orchestra.run.smithi039.stdout:/dev 2024-03-18T16:07:01.300 INFO:teuthology.orchestra.run.smithi039.stdout:/sys/kernel/security 2024-03-18T16:07:01.300 INFO:teuthology.orchestra.run.smithi039.stdout:/dev/shm 2024-03-18T16:07:01.300 INFO:teuthology.orchestra.run.smithi039.stdout:/dev/pts 2024-03-18T16:07:01.300 INFO:teuthology.orchestra.run.smithi039.stdout:/run 2024-03-18T16:07:01.300 INFO:teuthology.orchestra.run.smithi039.stdout:/sys/fs/cgroup 2024-03-18T16:07:01.300 INFO:teuthology.orchestra.run.smithi039.stdout:/sys/fs/pstore 2024-03-18T16:07:01.300 INFO:teuthology.orchestra.run.smithi039.stdout:/sys/fs/bpf 2024-03-18T16:07:01.301 INFO:teuthology.orchestra.run.smithi039.stdout:/ 2024-03-18T16:07:01.301 INFO:teuthology.orchestra.run.smithi039.stdout:/sys/fs/selinux 2024-03-18T16:07:01.301 INFO:teuthology.orchestra.run.smithi039.stdout:/proc/sys/fs/binfmt_misc 2024-03-18T16:07:01.301 INFO:teuthology.orchestra.run.smithi039.stdout:/dev/mqueue 2024-03-18T16:07:01.301 INFO:teuthology.orchestra.run.smithi039.stdout:/dev/hugepages 2024-03-18T16:07:01.301 INFO:teuthology.orchestra.run.smithi039.stdout:/sys/kernel/debug 2024-03-18T16:07:01.301 INFO:teuthology.orchestra.run.smithi039.stdout:/sys/kernel/tracing 2024-03-18T16:07:01.301 INFO:teuthology.orchestra.run.smithi039.stdout:/sys/fs/fuse/connections 2024-03-18T16:07:01.301 INFO:teuthology.orchestra.run.smithi039.stdout:/sys/kernel/config 2024-03-18T16:07:01.301 INFO:teuthology.orchestra.run.smithi039.stdout:/run/credentials/systemd-sysctl.service 2024-03-18T16:07:01.301 INFO:teuthology.orchestra.run.smithi039.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-18T16:07:01.301 INFO:teuthology.orchestra.run.smithi039.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-18T16:07:01.301 INFO:teuthology.orchestra.run.smithi039.stdout:/var/lib/nfs/rpc_pipefs 2024-03-18T16:07:01.301 INFO:teuthology.orchestra.run.smithi039.stdout:/run/user/1000 2024-03-18T16:07:01.302 INFO:teuthology.orchestra.run.smithi039.stdout:/var/lib/ceph 2024-03-18T16:07:01.302 INFO:teuthology.orchestra.run.smithi039.stdout:/var/lib/containers/storage/overlay 2024-03-18T16:07:01.302 INFO:teuthology.orchestra.run.smithi039.stdout:/var/lib/containers/storage/overlay/2409865ff32c9c04587444b8e37d7769407dc287c8a03c1342b9a86a150759fe/merged 2024-03-18T16:07:01.302 INFO:teuthology.orchestra.run.smithi039.stdout:/var/lib/containers/storage/overlay/b7654a442e809015b7eb859164a05d70d130cef137883e73bc780926e96a7643/merged 2024-03-18T16:07:01.302 INFO:teuthology.orchestra.run.smithi039.stdout:/run/user/0 2024-03-18T16:07:01.302 INFO:teuthology.orchestra.run.smithi039.stdout:/proc/sys/fs/binfmt_misc 2024-03-18T16:07:01.302 INFO:teuthology.orchestra.run.smithi039.stdout:/var/lib/containers/storage/overlay/90a75a6e76063e82797b9956dba964f848eb7a73e7d9ab973c1b7a66f477400a/merged 2024-03-18T16:07:01.302 INFO:teuthology.orchestra.run.smithi039.stdout:/var/lib/containers/storage/overlay/2b8cf1bae4484043534c37887a6b7c605a6069d23ede3575cc2a3150261fce84/merged 2024-03-18T16:07:01.302 INFO:teuthology.orchestra.run.smithi039.stdout:/var/lib/containers/storage/overlay/bfd06074c846d81c6a55892d973e556317f738cc99c70b42a301b0490c8723f9/merged 2024-03-18T16:07:01.302 INFO:teuthology.orchestra.run.smithi039.stdout:/var/lib/containers/storage/overlay/741b8fc3f666ba6902debb4c8fa112e0433ee386bbe9a4ba902c1eb92c4f29d3/merged 2024-03-18T16:07:01.302 INFO:teuthology.orchestra.run.smithi039.stdout:/var/lib/containers/storage/overlay/11187decc0d15d6eedc5740c05776f48a05f50c783e2bd23d09866c7af609970/merged 2024-03-18T16:07:01.302 INFO:teuthology.orchestra.run.smithi039.stdout:/var/lib/containers/storage/overlay/b95dd69952c1e2dfe2418a47cb70cc4c3e3325baf4dc7797ff6687961e7d4721/merged 2024-03-18T16:07:01.302 INFO:teuthology.orchestra.run.smithi039.stdout:/var/lib/containers/storage/overlay/43780c00f7e5b493b95793fb504686b5951cd76a9bebd45c531e6517ef630e5b/merged 2024-03-18T16:07:01.302 INFO:teuthology.orchestra.run.smithi039.stdout:/var/lib/containers/storage/overlay/013887cbd11a2f3962fdee72b04569a2833163420edbd32b8b3ae20055104d9b/merged 2024-03-18T16:07:01.303 INFO:teuthology.orchestra.run.smithi039.stdout:/run/netns 2024-03-18T16:07:01.303 INFO:teuthology.orchestra.run.smithi039.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-18T16:07:01.303 INFO:teuthology.orchestra.run.smithi039.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-18T16:07:01.303 INFO:teuthology.orchestra.run.smithi039.stdout:/home/ubuntu/cephtest/mnt.admin 2024-03-18T16:07:01.304 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T16:07:01.304 DEBUG:teuthology.orchestra.run.smithi039:> ls /sys/fs/fuse/connections 2024-03-18T16:07:01.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:01 smithi039 ceph-mon[31721]: pgmap v326: 129 pgs: 129 active+clean; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:07:01.359 INFO:teuthology.orchestra.run.smithi039.stdout:118 2024-03-18T16:07:01.359 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [118] 2024-03-18T16:07:01.359 DEBUG:teuthology.orchestra.run.smithi039:> sudo stdin-killer -- python3 -c ' 2024-03-18T16:07:01.359 DEBUG:teuthology.orchestra.run.smithi039:> import glob 2024-03-18T16:07:01.360 DEBUG:teuthology.orchestra.run.smithi039:> import re 2024-03-18T16:07:01.360 DEBUG:teuthology.orchestra.run.smithi039:> import os 2024-03-18T16:07:01.360 DEBUG:teuthology.orchestra.run.smithi039:> import subprocess 2024-03-18T16:07:01.360 DEBUG:teuthology.orchestra.run.smithi039:> 2024-03-18T16:07:01.360 DEBUG:teuthology.orchestra.run.smithi039:> def _find_admin_socket(client_name): 2024-03-18T16:07:01.360 DEBUG:teuthology.orchestra.run.smithi039:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-03-18T16:07:01.360 DEBUG:teuthology.orchestra.run.smithi039:> files = glob.glob(asok_path) 2024-03-18T16:07:01.360 DEBUG:teuthology.orchestra.run.smithi039:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-03-18T16:07:01.360 DEBUG:teuthology.orchestra.run.smithi039:> 2024-03-18T16:07:01.360 DEBUG:teuthology.orchestra.run.smithi039:> # Given a non-glob path, it better be there 2024-03-18T16:07:01.360 DEBUG:teuthology.orchestra.run.smithi039:> if "*" not in asok_path: 2024-03-18T16:07:01.360 DEBUG:teuthology.orchestra.run.smithi039:> assert(len(files) == 1) 2024-03-18T16:07:01.360 DEBUG:teuthology.orchestra.run.smithi039:> return files[0] 2024-03-18T16:07:01.360 DEBUG:teuthology.orchestra.run.smithi039:> 2024-03-18T16:07:01.360 DEBUG:teuthology.orchestra.run.smithi039:> for f in files: 2024-03-18T16:07:01.361 DEBUG:teuthology.orchestra.run.smithi039:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-18T16:07:01.361 DEBUG:teuthology.orchestra.run.smithi039:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-18T16:07:01.361 DEBUG:teuthology.orchestra.run.smithi039:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-18T16:07:01.361 DEBUG:teuthology.orchestra.run.smithi039:> contents = proc_f.read() 2024-03-18T16:07:01.361 DEBUG:teuthology.orchestra.run.smithi039:> if mountpoint in contents: 2024-03-18T16:07:01.361 DEBUG:teuthology.orchestra.run.smithi039:> return f 2024-03-18T16:07:01.361 DEBUG:teuthology.orchestra.run.smithi039:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-18T16:07:01.361 DEBUG:teuthology.orchestra.run.smithi039:> 2024-03-18T16:07:01.361 DEBUG:teuthology.orchestra.run.smithi039:> print(_find_admin_socket("client.admin")) 2024-03-18T16:07:01.361 DEBUG:teuthology.orchestra.run.smithi039:> ' 2024-03-18T16:07:01.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:01 smithi196 ceph-mon[36307]: pgmap v326: 129 pgs: 129 active+clean; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:07:01.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:01 smithi194 ceph-mon[36200]: pgmap v326: 129 pgs: 129 active+clean; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:07:01.592 INFO:teuthology.orchestra.run.smithi039.stdout:/var/run/ceph/ceph-client.admin.73442.asok 2024-03-18T16:07:01.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T16:07:01.610 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.73442.asok 2024-03-18T16:07:01.610 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T16:07:01.610 DEBUG:teuthology.orchestra.run.smithi039:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.73442.asok status 2024-03-18T16:07:01.853 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-03-18T16:07:01.853 INFO:teuthology.orchestra.run.smithi039.stdout: "metadata": { 2024-03-18T16:07:01.853 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph_sha1": "3d78f5ba07bf18880172c07d4d7b4ac00c9d30af", 2024-03-18T16:07:01.853 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph_version": "ceph version 19.0.0-2162-g3d78f5ba (3d78f5ba07bf18880172c07d4d7b4ac00c9d30af) squid (dev)", 2024-03-18T16:07:01.854 INFO:teuthology.orchestra.run.smithi039.stdout: "entity_id": "admin", 2024-03-18T16:07:01.854 INFO:teuthology.orchestra.run.smithi039.stdout: "hostname": "smithi039", 2024-03-18T16:07:01.854 INFO:teuthology.orchestra.run.smithi039.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-03-18T16:07:01.854 INFO:teuthology.orchestra.run.smithi039.stdout: "pid": "73442", 2024-03-18T16:07:01.854 INFO:teuthology.orchestra.run.smithi039.stdout: "root": "/" 2024-03-18T16:07:01.854 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:07:01.854 INFO:teuthology.orchestra.run.smithi039.stdout: "dentry_count": 0, 2024-03-18T16:07:01.854 INFO:teuthology.orchestra.run.smithi039.stdout: "dentry_pinned_count": 0, 2024-03-18T16:07:01.854 INFO:teuthology.orchestra.run.smithi039.stdout: "id": 14595, 2024-03-18T16:07:01.854 INFO:teuthology.orchestra.run.smithi039.stdout: "inst": { 2024-03-18T16:07:01.854 INFO:teuthology.orchestra.run.smithi039.stdout: "name": { 2024-03-18T16:07:01.854 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "client", 2024-03-18T16:07:01.854 INFO:teuthology.orchestra.run.smithi039.stdout: "num": 14595 2024-03-18T16:07:01.854 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:07:01.855 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": { 2024-03-18T16:07:01.855 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-03-18T16:07:01.855 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "192.168.0.1:0", 2024-03-18T16:07:01.855 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 554339595 2024-03-18T16:07:01.855 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:07:01.855 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:07:01.855 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": { 2024-03-18T16:07:01.855 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-03-18T16:07:01.855 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "192.168.0.1:0", 2024-03-18T16:07:01.855 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 554339595 2024-03-18T16:07:01.855 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:07:01.855 INFO:teuthology.orchestra.run.smithi039.stdout: "inst_str": "client.14595 192.168.0.1:0/554339595", 2024-03-18T16:07:01.855 INFO:teuthology.orchestra.run.smithi039.stdout: "addr_str": "192.168.0.1:0/554339595", 2024-03-18T16:07:01.855 INFO:teuthology.orchestra.run.smithi039.stdout: "inode_count": 1, 2024-03-18T16:07:01.855 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_epoch": 16, 2024-03-18T16:07:01.856 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_epoch": 89, 2024-03-18T16:07:01.856 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_epoch_barrier": 0, 2024-03-18T16:07:01.856 INFO:teuthology.orchestra.run.smithi039.stdout: "blocklisted": false, 2024-03-18T16:07:01.856 INFO:teuthology.orchestra.run.smithi039.stdout: "fs_name": "cephfs" 2024-03-18T16:07:01.856 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-03-18T16:07:01.870 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T16:07:01.871 DEBUG:teuthology.orchestra.run.smithi039:> stat --file-system '--printf=%T 2024-03-18T16:07:01.871 DEBUG:teuthology.orchestra.run.smithi039:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-03-18T16:07:01.925 INFO:teuthology.orchestra.run.smithi039.stdout:fuseblk 2024-03-18T16:07:01.925 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-03-18T16:07:01.925 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T16:07:01.925 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-03-18T16:07:02.014 DEBUG:teuthology.orchestra.run.smithi039:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-03-18T16:07:02.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-18T16:07:02.146 INFO:teuthology.orchestra.run.smithi039.stdout:# file: . 2024-03-18T16:07:02.147 INFO:teuthology.orchestra.run.smithi039.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-03-18T16:07:02.147 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:07:02.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T16:07:02.157 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-03-18T16:07:02.157 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T16:07:02.207 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T16:07:02.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.479+0000 7f5350e00640 1 -- 172.21.15.39:0/2575649198 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f534c075820 msgr2=0x7f534c073c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:02.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.479+0000 7f5350e00640 1 --2- 172.21.15.39:0/2575649198 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f534c075820 0x7f534c073c70 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f533c009930 tx=0x7f533c02bc20 comp rx=0 tx=0).stop 2024-03-18T16:07:02.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.479+0000 7f5350e00640 1 -- 172.21.15.39:0/2575649198 shutdown_connections 2024-03-18T16:07:02.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.479+0000 7f5350e00640 1 --2- 172.21.15.39:0/2575649198 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f534c10dbb0 0x7f534c10ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:02.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.479+0000 7f5350e00640 1 --2- 172.21.15.39:0/2575649198 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f534c074240 0x7f534c0746c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:02.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.479+0000 7f5350e00640 1 --2- 172.21.15.39:0/2575649198 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f534c075820 0x7f534c073c70 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:02.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.479+0000 7f5350e00640 1 -- 172.21.15.39:0/2575649198 >> 172.21.15.39:0/2575649198 conn(0x7f534c0fcb70 msgr2=0x7f534c0fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:02.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.480+0000 7f5350e00640 1 -- 172.21.15.39:0/2575649198 shutdown_connections 2024-03-18T16:07:02.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.480+0000 7f5350e00640 1 -- 172.21.15.39:0/2575649198 wait complete. 2024-03-18T16:07:02.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.481+0000 7f5350e00640 1 Processor -- start 2024-03-18T16:07:02.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.481+0000 7f5350e00640 1 -- start start 2024-03-18T16:07:02.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.481+0000 7f5350e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f534c074240 0x7f534c19df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:02.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.481+0000 7f5350e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f534c075820 0x7f534c19e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:02.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.481+0000 7f5350e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f534c10dbb0 0x7f534c1a54b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:02.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.481+0000 7f5350e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f534c113830 con 0x7f534c074240 2024-03-18T16:07:02.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.481+0000 7f5350e00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f534c1136b0 con 0x7f534c075820 2024-03-18T16:07:02.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.481+0000 7f5350e00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f534c1139b0 con 0x7f534c10dbb0 2024-03-18T16:07:02.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.482+0000 7f534aa00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f534c075820 0x7f534c19e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:02.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.482+0000 7f534aa00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f534c075820 0x7f534c19e440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:49672/0 (socket says 172.21.15.39:49672) 2024-03-18T16:07:02.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.482+0000 7f534aa00640 1 -- 172.21.15.39:0/3875156555 learned_addr learned my addr 172.21.15.39:0/3875156555 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:07:02.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.482+0000 7f534b400640 1 --2- 172.21.15.39:0/3875156555 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f534c074240 0x7f534c19df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:02.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.482+0000 7f534be00640 1 --2- 172.21.15.39:0/3875156555 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f534c10dbb0 0x7f534c1a54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:02.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.482+0000 7f534aa00640 1 -- 172.21.15.39:0/3875156555 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f534c10dbb0 msgr2=0x7f534c1a54b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:02.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.482+0000 7f534aa00640 1 --2- 172.21.15.39:0/3875156555 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f534c10dbb0 0x7f534c1a54b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:02.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.482+0000 7f534aa00640 1 -- 172.21.15.39:0/3875156555 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f534c074240 msgr2=0x7f534c19df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:02.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.482+0000 7f534aa00640 1 --2- 172.21.15.39:0/3875156555 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f534c074240 0x7f534c19df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:02.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.482+0000 7f534aa00640 1 -- 172.21.15.39:0/3875156555 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f534c1a5b20 con 0x7f534c075820 2024-03-18T16:07:02.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.482+0000 7f534aa00640 1 --2- 172.21.15.39:0/3875156555 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f534c075820 0x7f534c19e440 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f533800e9f0 tx=0x7f533800edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:02.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.483+0000 7f5343e00640 1 -- 172.21.15.39:0/3875156555 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f533800ccc0 con 0x7f534c075820 2024-03-18T16:07:02.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.483+0000 7f5343e00640 1 -- 172.21.15.39:0/3875156555 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5338010910 con 0x7f534c075820 2024-03-18T16:07:02.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.483+0000 7f5350e00640 1 -- 172.21.15.39:0/3875156555 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f534c1a5e10 con 0x7f534c075820 2024-03-18T16:07:02.487 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.483+0000 7f5343e00640 1 -- 172.21.15.39:0/3875156555 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53380188a0 con 0x7f534c075820 2024-03-18T16:07:02.487 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.483+0000 7f5350e00640 1 -- 172.21.15.39:0/3875156555 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f534c1a62d0 con 0x7f534c075820 2024-03-18T16:07:02.487 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.485+0000 7f5350e00640 1 -- 172.21.15.39:0/3875156555 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f534c103d80 con 0x7f534c075820 2024-03-18T16:07:02.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.487+0000 7f5343e00640 1 -- 172.21.15.39:0/3875156555 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f5338010ac0 con 0x7f534c075820 2024-03-18T16:07:02.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.488+0000 7f5343e00640 1 --2- 172.21.15.39:0/3875156555 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f53300a2800 0x7f53300a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:02.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.488+0000 7f5343e00640 1 -- 172.21.15.39:0/3875156555 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f53380cf120 con 0x7f534c075820 2024-03-18T16:07:02.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.488+0000 7f534b400640 1 --2- 172.21.15.39:0/3875156555 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f53300a2800 0x7f53300a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:02.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.489+0000 7f534b400640 1 --2- 172.21.15.39:0/3875156555 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f53300a2800 0x7f53300a4cc0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f533c009900 tx=0x7f533c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:02.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.492+0000 7f5343e00640 1 -- 172.21.15.39:0/3875156555 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f533809bf20 con 0x7f534c075820 2024-03-18T16:07:02.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.699+0000 7f5350e00640 1 -- 172.21.15.39:0/3875156555 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f534c062410 con 0x7f534c075820 2024-03-18T16:07:02.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.701+0000 7f5343e00640 1 -- 172.21.15.39:0/3875156555 <== mon.2 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24142 (secure 0 0 0) 0x7f533809b820 con 0x7f534c075820 2024-03-18T16:07:02.704 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:07:02.704 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":89,"fsid":"1892d8fc-e540-11ee-95c9-87774f69a715","created":"2024-03-18T15:57:36.507008+0000","modified":"2024-03-18T16:06:49.510653+0000","last_up_change":"2024-03-18T16:05:45.250629+0000","last_in_change":"2024-03-18T16:05:24.841337+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T16:01:30.048247+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T16:06:42.191791+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T16:06:43.670325+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T16:06:47.523388+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"771e0558-be62-4436-94b0-6364751911c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6803","nonce":4090743635}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6805","nonce":4090743635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6809","nonce":4090743635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6807","nonce":4090743635}]},"public_addr":"172.21.15.39:6803/4090743635","cluster_addr":"172.21.15.39:6805/4090743635","heartbeat_back_addr":"172.21.15.39:6809/4090743635","heartbeat_front_addr":"172.21.15.39:6807/4090743635","state":["exists","up"]},{"osd":1,"uuid":"0430eee1-cf34-4277-b255-c7d9f07ad19a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6801","nonce":2800329805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6803","nonce":2800329805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6807","nonce":2800329805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6805","nonce":2800329805}]},"public_addr":"172.21.15.194:6801/2800329805","cluster_addr":"172.21.15.194:6803/2800329805","heartbeat_back_addr":"172.21.15.194:6807/2800329805","heartbeat_front_addr":"172.21.15.194:6805/2800329805","state":["exists","up"]},{"osd":2,"uuid":"3a718da4-2511-4ec1-b1a8-eafd6cce0df0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6801","nonce":3935874719}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6803","nonce":3935874719}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6807","nonce":3935874719}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6805","nonce":3935874719}]},"public_addr":"172.21.15.196:6801/3935874719","cluster_addr":"172.21.15.196:6803/3935874719","heartbeat_back_addr":"172.21.15.196:6807/3935874719","heartbeat_front_addr":"172.21.15.196:6805/3935874719","state":["exists","up"]},{"osd":3,"uuid":"6f9c37f9-0ab9-4038-8919-afd4b39c4165","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6811","nonce":893345939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6813","nonce":893345939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6817","nonce":893345939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6815","nonce":893345939}]},"public_addr":"172.21.15.39:6811/893345939","cluster_addr":"172.21.15.39:6813/893345939","heartbeat_back_addr":"172.21.15.39:6817/893345939","heartbeat_front_addr":"172.21.15.39:6815/893345939","state":["exists","up"]},{"osd":4,"uuid":"e795bb0e-4791-4537-acda-1a321e19bc9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6809","nonce":800423088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6811","nonce":800423088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6815","nonce":800423088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6813","nonce":800423088}]},"public_addr":"172.21.15.194:6809/800423088","cluster_addr":"172.21.15.194:6811/800423088","heartbeat_back_addr":"172.21.15.194:6815/800423088","heartbeat_front_addr":"172.21.15.194:6813/800423088","state":["exists","up"]},{"osd":5,"uuid":"026e175e-6acb-4b4c-b235-8ccee8ae1149","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6809","nonce":2746224525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6811","nonce":2746224525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6815","nonce":2746224525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6813","nonce":2746224525}]},"public_addr":"172.21.15.196:6809/2746224525","cluster_addr":"172.21.15.196:6811/2746224525","heartbeat_back_addr":"172.21.15.196:6815/2746224525","heartbeat_front_addr":"172.21.15.196:6813/2746224525","state":["exists","up"]},{"osd":6,"uuid":"c1b183ff-383f-4c5c-9075-f76bab1c0bb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6819","nonce":1213371331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6821","nonce":1213371331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6825","nonce":1213371331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6823","nonce":1213371331}]},"public_addr":"172.21.15.39:6819/1213371331","cluster_addr":"172.21.15.39:6821/1213371331","heartbeat_back_addr":"172.21.15.39:6825/1213371331","heartbeat_front_addr":"172.21.15.39:6823/1213371331","state":["exists","up"]},{"osd":7,"uuid":"612f8697-957d-4ef4-a87f-241b36f9d13d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6817","nonce":3299162444}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6819","nonce":3299162444}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6823","nonce":3299162444}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6821","nonce":3299162444}]},"public_addr":"172.21.15.194:6817/3299162444","cluster_addr":"172.21.15.194:6819/3299162444","heartbeat_back_addr":"172.21.15.194:6823/3299162444","heartbeat_front_addr":"172.21.15.194:6821/3299162444","state":["exists","up"]},{"osd":8,"uuid":"836a9107-4d4f-4594-b8dc-0301e64f59d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6817","nonce":2456143145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6819","nonce":2456143145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6823","nonce":2456143145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6821","nonce":2456143145}]},"public_addr":"172.21.15.196:6817/2456143145","cluster_addr":"172.21.15.196:6819/2456143145","heartbeat_back_addr":"172.21.15.196:6823/2456143145","heartbeat_front_addr":"172.21.15.196:6821/2456143145","state":["exists","up"]},{"osd":9,"uuid":"4f30b960-12e3-4b16-bd28-fcc988139b58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6827","nonce":1848634433}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6829","nonce":1848634433}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6833","nonce":1848634433}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6831","nonce":1848634433}]},"public_addr":"172.21.15.39:6827/1848634433","cluster_addr":"172.21.15.39:6829/1848634433","heartbeat_back_addr":"172.21.15.39:6833/1848634433","heartbeat_front_addr":"172.21.15.39:6831/1848634433","state":["exists","up"]},{"osd":10,"uuid":"620f90e8-23e6-4bc0-8269-7219b9e1aebd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6825","nonce":3725064809}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6827","nonce":3725064809}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6831","nonce":3725064809}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6829","nonce":3725064809}]},"public_addr":"172.21.15.194:6825/3725064809","cluster_addr":"172.21.15.194:6827/3725064809","heartbeat_back_addr":"172.21.15.194:6831/3725064809","heartbeat_front_addr":"172.21.15.194:6829/3725064809","state":["exists","up"]},{"osd":11,"uuid":"867a2535-8235-4ceb-a698-eb86869d6d20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6825","nonce":2501212422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6827","nonce":2501212422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6831","nonce":2501212422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6829","nonce":2501212422}]},"public_addr":"172.21.15.196:6825/2501212422","cluster_addr":"172.21.15.196:6827/2501212422","heartbeat_back_addr":"172.21.15.196:6831/2501212422","heartbeat_front_addr":"172.21.15.196:6829/2501212422","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:00:36.931566+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:01:02.614024+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:01:26.193265+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:01:58.308833+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:02:25.049546+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:02:49.748750+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:03:23.582153+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:03:50.447321+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:04:15.812129+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:04:49.998047+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:05:17.087924+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:05:43.167637+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.39:0/3482086140":"2024-03-19T15:58:45.340294+0000","172.21.15.39:0/3844306284":"2024-03-19T15:58:45.340294+0000","172.21.15.39:6801/1810238546":"2024-03-19T15:58:45.340294+0000","172.21.15.39:6800/1810238546":"2024-03-19T15:58:45.340294+0000","172.21.15.39:0/2528397488":"2024-03-19T15:58:10.264798+0000","172.21.15.39:0/2240981254":"2024-03-19T15:58:10.264798+0000","172.21.15.39:6800/1217503189":"2024-03-19T15:58:10.264798+0000","172.21.15.39:0/3754876370":"2024-03-19T15:58:10.264798+0000","172.21.15.39:6801/1217503189":"2024-03-19T15:58:10.264798+0000","172.21.15.39:0/1874637628":"2024-03-19T15:58:45.340294+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T16:07:02.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.704+0000 7f5350e00640 1 -- 172.21.15.39:0/3875156555 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f53300a2800 msgr2=0x7f53300a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:02.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.704+0000 7f5350e00640 1 --2- 172.21.15.39:0/3875156555 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f53300a2800 0x7f53300a4cc0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f533c009900 tx=0x7f533c000bc0 comp rx=0 tx=0).stop 2024-03-18T16:07:02.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.704+0000 7f5350e00640 1 -- 172.21.15.39:0/3875156555 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f534c075820 msgr2=0x7f534c19e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:02.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.704+0000 7f5350e00640 1 --2- 172.21.15.39:0/3875156555 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f534c075820 0x7f534c19e440 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f533800e9f0 tx=0x7f533800edf0 comp rx=0 tx=0).stop 2024-03-18T16:07:02.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.704+0000 7f5350e00640 1 -- 172.21.15.39:0/3875156555 shutdown_connections 2024-03-18T16:07:02.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.704+0000 7f5350e00640 1 --2- 172.21.15.39:0/3875156555 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f534c075820 0x7f534c19e440 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:02.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.704+0000 7f5350e00640 1 --2- 172.21.15.39:0/3875156555 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f53300a2800 0x7f53300a4cc0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:02.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.704+0000 7f5350e00640 1 --2- 172.21.15.39:0/3875156555 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f534c10dbb0 0x7f534c1a54b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:02.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.704+0000 7f5350e00640 1 --2- 172.21.15.39:0/3875156555 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f534c074240 0x7f534c19df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:02.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.704+0000 7f5350e00640 1 -- 172.21.15.39:0/3875156555 >> 172.21.15.39:0/3875156555 conn(0x7f534c0fcb70 msgr2=0x7f534c100f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:02.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.704+0000 7f5350e00640 1 -- 172.21.15.39:0/3875156555 shutdown_connections 2024-03-18T16:07:02.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.704+0000 7f5350e00640 1 -- 172.21.15.39:0/3875156555 wait complete. 2024-03-18T16:07:02.729 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T16:01:30.048247+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T16:06:42.191791+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T16:06:43.670325+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T16:06:47.523388+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T16:07:02.730 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T16:07:02.956 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.953+0000 7f947b200640 1 -- 172.21.15.39:0/4222208938 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f947c102140 msgr2=0x7f947c104530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:02.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.953+0000 7f947b200640 1 --2- 172.21.15.39:0/4222208938 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f947c102140 0x7f947c104530 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f946c009ca0 tx=0x7f946c02c100 comp rx=0 tx=0).stop 2024-03-18T16:07:02.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.954+0000 7f947b200640 1 -- 172.21.15.39:0/4222208938 shutdown_connections 2024-03-18T16:07:02.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.954+0000 7f947b200640 1 --2- 172.21.15.39:0/4222208938 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f947c1073a0 0x7f947c1097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:02.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.954+0000 7f947b200640 1 --2- 172.21.15.39:0/4222208938 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f947c104a70 0x7f947c106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:02.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.954+0000 7f947b200640 1 --2- 172.21.15.39:0/4222208938 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f947c102140 0x7f947c104530 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:02.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.954+0000 7f947b200640 1 -- 172.21.15.39:0/4222208938 >> 172.21.15.39:0/4222208938 conn(0x7f947c0fbb50 msgr2=0x7f947c0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:02.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.954+0000 7f947b200640 1 -- 172.21.15.39:0/4222208938 shutdown_connections 2024-03-18T16:07:02.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.954+0000 7f947b200640 1 -- 172.21.15.39:0/4222208938 wait complete. 2024-03-18T16:07:02.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.955+0000 7f947b200640 1 Processor -- start 2024-03-18T16:07:02.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.955+0000 7f947b200640 1 -- start start 2024-03-18T16:07:02.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.955+0000 7f947b200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f947c102140 0x7f947c1979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:02.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.955+0000 7f947b200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f947c104a70 0x7f947c197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:02.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.955+0000 7f947b200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f947c1073a0 0x7f947c19efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:02.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.955+0000 7f947b200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f947c10d350 con 0x7f947c102140 2024-03-18T16:07:02.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.955+0000 7f947b200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f947c10d1d0 con 0x7f947c1073a0 2024-03-18T16:07:02.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.955+0000 7f947b200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f947c10d4d0 con 0x7f947c104a70 2024-03-18T16:07:02.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.955+0000 7f9479e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f947c102140 0x7f947c1979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:02.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.955+0000 7f947a800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f947c1073a0 0x7f947c19efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:02.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.955+0000 7f947a800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f947c1073a0 0x7f947c19efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:49682/0 (socket says 172.21.15.39:49682) 2024-03-18T16:07:02.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.956+0000 7f947a800640 1 -- 172.21.15.39:0/1943419460 learned_addr learned my addr 172.21.15.39:0/1943419460 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:07:02.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.955+0000 7f9479e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f947c102140 0x7f947c1979f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58606/0 (socket says 172.21.15.39:58606) 2024-03-18T16:07:02.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.956+0000 7f9479400640 1 --2- 172.21.15.39:0/1943419460 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f947c104a70 0x7f947c197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:02.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.956+0000 7f947a800640 1 -- 172.21.15.39:0/1943419460 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f947c104a70 msgr2=0x7f947c197f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:02.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.956+0000 7f947a800640 1 --2- 172.21.15.39:0/1943419460 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f947c104a70 0x7f947c197f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:02.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.956+0000 7f947a800640 1 -- 172.21.15.39:0/1943419460 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f947c102140 msgr2=0x7f947c1979f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:02.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.956+0000 7f947a800640 1 --2- 172.21.15.39:0/1943419460 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f947c102140 0x7f947c1979f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:02.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.956+0000 7f947a800640 1 -- 172.21.15.39:0/1943419460 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f947c19f610 con 0x7f947c1073a0 2024-03-18T16:07:02.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.956+0000 7f9479e00640 1 --2- 172.21.15.39:0/1943419460 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f947c102140 0x7f947c1979f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:07:02.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.956+0000 7f947a800640 1 --2- 172.21.15.39:0/1943419460 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f947c1073a0 0x7f947c19efa0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f947400e9b0 tx=0x7f947400ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:02.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.956+0000 7f9472a00640 1 -- 172.21.15.39:0/1943419460 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f947400cd80 con 0x7f947c1073a0 2024-03-18T16:07:02.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.956+0000 7f9472a00640 1 -- 172.21.15.39:0/1943419460 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94740035f0 con 0x7f947c1073a0 2024-03-18T16:07:02.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.956+0000 7f9472a00640 1 -- 172.21.15.39:0/1943419460 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9474010670 con 0x7f947c1073a0 2024-03-18T16:07:02.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.956+0000 7f947b200640 1 -- 172.21.15.39:0/1943419460 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f947c19f900 con 0x7f947c1073a0 2024-03-18T16:07:02.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.957+0000 7f947b200640 1 -- 172.21.15.39:0/1943419460 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f947c19fe50 con 0x7f947c1073a0 2024-03-18T16:07:02.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.958+0000 7f947b200640 1 -- 172.21.15.39:0/1943419460 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f947c0fd640 con 0x7f947c1073a0 2024-03-18T16:07:02.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.959+0000 7f9472a00640 1 -- 172.21.15.39:0/1943419460 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f94740037a0 con 0x7f947c1073a0 2024-03-18T16:07:02.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.961+0000 7f9472a00640 1 --2- 172.21.15.39:0/1943419460 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f94600a2880 0x7f94600a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:02.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.961+0000 7f9472a00640 1 -- 172.21.15.39:0/1943419460 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f94740cef80 con 0x7f947c1073a0 2024-03-18T16:07:02.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.961+0000 7f9479e00640 1 --2- 172.21.15.39:0/1943419460 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f94600a2880 0x7f94600a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:02.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.962+0000 7f9479e00640 1 --2- 172.21.15.39:0/1943419460 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f94600a2880 0x7f94600a4d40 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f946c006010 tx=0x7f946c032590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:02.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:02.965+0000 7f9472a00640 1 -- 172.21.15.39:0/1943419460 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f947409bd80 con 0x7f947c1073a0 2024-03-18T16:07:03.172 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.170+0000 7f947b200640 1 -- 172.21.15.39:0/1943419460 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f947c0623e0 con 0x7f947c1073a0 2024-03-18T16:07:03.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.170+0000 7f9472a00640 1 -- 172.21.15.39:0/1943419460 <== mon.2 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) ==== 93+0+10 (secure 0 0 0) 0x7f947409b680 con 0x7f947c1073a0 2024-03-18T16:07:03.173 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2024-03-18T16:07:03.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.173+0000 7f947b200640 1 -- 172.21.15.39:0/1943419460 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f94600a2880 msgr2=0x7f94600a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:03.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.173+0000 7f947b200640 1 --2- 172.21.15.39:0/1943419460 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f94600a2880 0x7f94600a4d40 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f946c006010 tx=0x7f946c032590 comp rx=0 tx=0).stop 2024-03-18T16:07:03.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.174+0000 7f947b200640 1 -- 172.21.15.39:0/1943419460 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f947c1073a0 msgr2=0x7f947c19efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:03.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.174+0000 7f947b200640 1 --2- 172.21.15.39:0/1943419460 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f947c1073a0 0x7f947c19efa0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f947400e9b0 tx=0x7f947400ee70 comp rx=0 tx=0).stop 2024-03-18T16:07:03.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.174+0000 7f947b200640 1 -- 172.21.15.39:0/1943419460 shutdown_connections 2024-03-18T16:07:03.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.174+0000 7f947b200640 1 --2- 172.21.15.39:0/1943419460 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f947c1073a0 0x7f947c19efa0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:03.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.174+0000 7f947b200640 1 --2- 172.21.15.39:0/1943419460 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f94600a2880 0x7f94600a4d40 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:03.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.174+0000 7f947b200640 1 --2- 172.21.15.39:0/1943419460 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f947c104a70 0x7f947c197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:03.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.174+0000 7f947b200640 1 --2- 172.21.15.39:0/1943419460 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f947c102140 0x7f947c1979f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:03.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.174+0000 7f947b200640 1 -- 172.21.15.39:0/1943419460 >> 172.21.15.39:0/1943419460 conn(0x7f947c0fbb50 msgr2=0x7f947c102a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:03.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.174+0000 7f947b200640 1 -- 172.21.15.39:0/1943419460 shutdown_connections 2024-03-18T16:07:03.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.174+0000 7f947b200640 1 -- 172.21.15.39:0/1943419460 wait complete. 2024-03-18T16:07:03.196 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T16:07:03.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:03 smithi039 ceph-mon[31721]: pgmap v327: 129 pgs: 129 active+clean; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:07:03.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:03 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/3875156555' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:07:03.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.422+0000 7fb489a00640 1 -- 172.21.15.39:0/995473170 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb4841016c0 msgr2=0x7fb484101ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:03.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.422+0000 7fb489a00640 1 --2- 172.21.15.39:0/995473170 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb4841016c0 0x7fb484101ae0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb474009900 tx=0x7fb47402bc30 comp rx=0 tx=0).stop 2024-03-18T16:07:03.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.422+0000 7fb489a00640 1 -- 172.21.15.39:0/995473170 shutdown_connections 2024-03-18T16:07:03.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.422+0000 7fb489a00640 1 --2- 172.21.15.39:0/995473170 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb484103270 0x7fb484109b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:03.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.422+0000 7fb489a00640 1 --2- 172.21.15.39:0/995473170 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb4841028b0 0x7fb484102d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:03.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.422+0000 7fb489a00640 1 --2- 172.21.15.39:0/995473170 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb4841016c0 0x7fb484101ae0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:03.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.422+0000 7fb489a00640 1 -- 172.21.15.39:0/995473170 >> 172.21.15.39:0/995473170 conn(0x7fb4840fcc50 msgr2=0x7fb4840ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:03.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.423+0000 7fb489a00640 1 -- 172.21.15.39:0/995473170 shutdown_connections 2024-03-18T16:07:03.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.423+0000 7fb489a00640 1 -- 172.21.15.39:0/995473170 wait complete. 2024-03-18T16:07:03.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.424+0000 7fb489a00640 1 Processor -- start 2024-03-18T16:07:03.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.424+0000 7fb489a00640 1 -- start start 2024-03-18T16:07:03.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.424+0000 7fb489a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb4841016c0 0x7fb484199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:03.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.424+0000 7fb489a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb4841028b0 0x7fb48419a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:03.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.424+0000 7fb489a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb484103270 0x7fb4841a1210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:03.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.425+0000 7fb489a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fb48410d360 con 0x7fb4841028b0 2024-03-18T16:07:03.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.425+0000 7fb489a00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fb48410d1e0 con 0x7fb484103270 2024-03-18T16:07:03.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.425+0000 7fb489a00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fb48410d4e0 con 0x7fb4841016c0 2024-03-18T16:07:03.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.425+0000 7fb489000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb484103270 0x7fb4841a1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:03.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.425+0000 7fb483400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb4841028b0 0x7fb48419a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:03.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.425+0000 7fb489000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb484103270 0x7fb4841a1210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:49696/0 (socket says 172.21.15.39:49696) 2024-03-18T16:07:03.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.425+0000 7fb489000640 1 -- 172.21.15.39:0/2239186669 learned_addr learned my addr 172.21.15.39:0/2239186669 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:07:03.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.425+0000 7fb483e00640 1 --2- 172.21.15.39:0/2239186669 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb4841016c0 0x7fb484199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:03.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.425+0000 7fb489000640 1 -- 172.21.15.39:0/2239186669 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb4841016c0 msgr2=0x7fb484199c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:03.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.425+0000 7fb489000640 1 --2- 172.21.15.39:0/2239186669 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb4841016c0 0x7fb484199c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:03.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.425+0000 7fb489000640 1 -- 172.21.15.39:0/2239186669 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb4841028b0 msgr2=0x7fb48419a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:03.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.425+0000 7fb489000640 1 --2- 172.21.15.39:0/2239186669 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb4841028b0 0x7fb48419a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:03.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.425+0000 7fb489000640 1 -- 172.21.15.39:0/2239186669 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4841a18b0 con 0x7fb484103270 2024-03-18T16:07:03.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.425+0000 7fb483e00640 1 --2- 172.21.15.39:0/2239186669 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb4841016c0 0x7fb484199c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:07:03.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.426+0000 7fb489000640 1 --2- 172.21.15.39:0/2239186669 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb484103270 0x7fb4841a1210 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fb47c00eb10 tx=0x7fb47c00c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:03.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.426+0000 7fb480c00640 1 -- 172.21.15.39:0/2239186669 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb47c004b20 con 0x7fb484103270 2024-03-18T16:07:03.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.426+0000 7fb480c00640 1 -- 172.21.15.39:0/2239186669 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb47c004cc0 con 0x7fb484103270 2024-03-18T16:07:03.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.426+0000 7fb480c00640 1 -- 172.21.15.39:0/2239186669 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb47c010760 con 0x7fb484103270 2024-03-18T16:07:03.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.426+0000 7fb489a00640 1 -- 172.21.15.39:0/2239186669 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4841a1ba0 con 0x7fb484103270 2024-03-18T16:07:03.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.426+0000 7fb489a00640 1 -- 172.21.15.39:0/2239186669 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4841a20c0 con 0x7fb484103270 2024-03-18T16:07:03.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.427+0000 7fb489a00640 1 -- 172.21.15.39:0/2239186669 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4841a22f0 con 0x7fb484103270 2024-03-18T16:07:03.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.429+0000 7fb480c00640 1 -- 172.21.15.39:0/2239186669 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fb47c005020 con 0x7fb484103270 2024-03-18T16:07:03.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.431+0000 7fb480c00640 1 --2- 172.21.15.39:0/2239186669 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb4680a2880 0x7fb4680a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:03.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.431+0000 7fb480c00640 1 -- 172.21.15.39:0/2239186669 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fb47c0cf230 con 0x7fb484103270 2024-03-18T16:07:03.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.431+0000 7fb483e00640 1 --2- 172.21.15.39:0/2239186669 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb4680a2880 0x7fb4680a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:03.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.432+0000 7fb483e00640 1 --2- 172.21.15.39:0/2239186669 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb4680a2880 0x7fb4680a4d40 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb47402c240 tx=0x7fb47402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:03.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.435+0000 7fb480c00640 1 -- 172.21.15.39:0/2239186669 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb47c09c090 con 0x7fb484103270 2024-03-18T16:07:03.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:03 smithi196 ceph-mon[36307]: pgmap v327: 129 pgs: 129 active+clean; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:07:03.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:03 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/3875156555' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:07:03.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:03 smithi194 ceph-mon[36200]: pgmap v327: 129 pgs: 129 active+clean; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:07:03.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:03 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/3875156555' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:07:03.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.647+0000 7fb489a00640 1 -- 172.21.15.39:0/2239186669 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fb4841a2560 con 0x7fb484103270 2024-03-18T16:07:03.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.647+0000 7fb480c00640 1 -- 172.21.15.39:0/2239186669 <== mon.2 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) ==== 104+0+11 (secure 0 0 0) 0x7fb47c09b990 con 0x7fb484103270 2024-03-18T16:07:03.650 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 32 2024-03-18T16:07:03.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.650+0000 7fb489a00640 1 -- 172.21.15.39:0/2239186669 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb4680a2880 msgr2=0x7fb4680a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:03.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.650+0000 7fb489a00640 1 --2- 172.21.15.39:0/2239186669 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb4680a2880 0x7fb4680a4d40 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb47402c240 tx=0x7fb47402df90 comp rx=0 tx=0).stop 2024-03-18T16:07:03.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.650+0000 7fb489a00640 1 -- 172.21.15.39:0/2239186669 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb484103270 msgr2=0x7fb4841a1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:03.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.650+0000 7fb489a00640 1 --2- 172.21.15.39:0/2239186669 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb484103270 0x7fb4841a1210 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fb47c00eb10 tx=0x7fb47c00c2d0 comp rx=0 tx=0).stop 2024-03-18T16:07:03.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.650+0000 7fb489a00640 1 -- 172.21.15.39:0/2239186669 shutdown_connections 2024-03-18T16:07:03.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.650+0000 7fb489a00640 1 --2- 172.21.15.39:0/2239186669 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb484103270 0x7fb4841a1210 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:03.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.650+0000 7fb489a00640 1 --2- 172.21.15.39:0/2239186669 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb4680a2880 0x7fb4680a4d40 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:03.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.650+0000 7fb489a00640 1 --2- 172.21.15.39:0/2239186669 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb4841028b0 0x7fb48419a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:03.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.650+0000 7fb489a00640 1 --2- 172.21.15.39:0/2239186669 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb4841016c0 0x7fb484199c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:03.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.650+0000 7fb489a00640 1 -- 172.21.15.39:0/2239186669 >> 172.21.15.39:0/2239186669 conn(0x7fb4840fcc50 msgr2=0x7fb484073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:03.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.651+0000 7fb489a00640 1 -- 172.21.15.39:0/2239186669 shutdown_connections 2024-03-18T16:07:03.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.651+0000 7fb489a00640 1 -- 172.21.15.39:0/2239186669 wait complete. 2024-03-18T16:07:03.672 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T16:07:03.894 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.891+0000 7fdd45000640 1 -- 172.21.15.39:0/907868090 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdd401028b0 msgr2=0x7fdd40102d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:03.894 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.891+0000 7fdd45000640 1 --2- 172.21.15.39:0/907868090 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdd401028b0 0x7fdd40102d30 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fdd28009900 tx=0x7fdd2802bc40 comp rx=0 tx=0).stop 2024-03-18T16:07:03.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.892+0000 7fdd45000640 1 -- 172.21.15.39:0/907868090 shutdown_connections 2024-03-18T16:07:03.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.892+0000 7fdd45000640 1 --2- 172.21.15.39:0/907868090 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdd401028b0 0x7fdd40102d30 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:03.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.892+0000 7fdd45000640 1 --2- 172.21.15.39:0/907868090 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdd40103270 0x7fdd40109b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:03.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.892+0000 7fdd45000640 1 --2- 172.21.15.39:0/907868090 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdd401016c0 0x7fdd40101ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:03.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.892+0000 7fdd45000640 1 -- 172.21.15.39:0/907868090 >> 172.21.15.39:0/907868090 conn(0x7fdd400fcc50 msgr2=0x7fdd400ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:03.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.892+0000 7fdd45000640 1 -- 172.21.15.39:0/907868090 shutdown_connections 2024-03-18T16:07:03.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.892+0000 7fdd45000640 1 -- 172.21.15.39:0/907868090 wait complete. 2024-03-18T16:07:03.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.893+0000 7fdd45000640 1 Processor -- start 2024-03-18T16:07:03.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.893+0000 7fdd45000640 1 -- start start 2024-03-18T16:07:03.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.893+0000 7fdd45000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdd401016c0 0x7fdd401979b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:03.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.893+0000 7fdd45000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdd401028b0 0x7fdd40197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:03.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.893+0000 7fdd45000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdd40103270 0x7fdd4019ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:03.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.893+0000 7fdd45000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fdd4010d370 con 0x7fdd40103270 2024-03-18T16:07:03.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.893+0000 7fdd45000640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fdd4010d1f0 con 0x7fdd401016c0 2024-03-18T16:07:03.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.893+0000 7fdd45000640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fdd4010d4f0 con 0x7fdd401028b0 2024-03-18T16:07:03.898 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.893+0000 7fdd3f400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdd401016c0 0x7fdd401979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:03.898 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.893+0000 7fdd3f400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdd401016c0 0x7fdd401979b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:49714/0 (socket says 172.21.15.39:49714) 2024-03-18T16:07:03.898 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.893+0000 7fdd3f400640 1 -- 172.21.15.39:0/2884888472 learned_addr learned my addr 172.21.15.39:0/2884888472 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:07:03.898 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.893+0000 7fdd37e00640 1 --2- 172.21.15.39:0/2884888472 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdd401028b0 0x7fdd40197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:03.898 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.893+0000 7fdd3fe00640 1 --2- 172.21.15.39:0/2884888472 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdd40103270 0x7fdd4019ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:03.898 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.894+0000 7fdd3f400640 1 -- 172.21.15.39:0/2884888472 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdd401028b0 msgr2=0x7fdd40197ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:03.898 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.894+0000 7fdd3f400640 1 --2- 172.21.15.39:0/2884888472 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdd401028b0 0x7fdd40197ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:03.898 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.894+0000 7fdd3f400640 1 -- 172.21.15.39:0/2884888472 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdd40103270 msgr2=0x7fdd4019ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:03.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.894+0000 7fdd3f400640 1 --2- 172.21.15.39:0/2884888472 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdd40103270 0x7fdd4019ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:03.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.894+0000 7fdd3f400640 1 -- 172.21.15.39:0/2884888472 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd4019f5d0 con 0x7fdd401016c0 2024-03-18T16:07:03.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.894+0000 7fdd3f400640 1 --2- 172.21.15.39:0/2884888472 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdd401016c0 0x7fdd401979b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fdd3000e9c0 tx=0x7fdd3000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:03.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.894+0000 7fdd3cc00640 1 -- 172.21.15.39:0/2884888472 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd3000cd80 con 0x7fdd401016c0 2024-03-18T16:07:03.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.894+0000 7fdd3cc00640 1 -- 172.21.15.39:0/2884888472 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd300035f0 con 0x7fdd401016c0 2024-03-18T16:07:03.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.894+0000 7fdd3cc00640 1 -- 172.21.15.39:0/2884888472 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd30010690 con 0x7fdd401016c0 2024-03-18T16:07:03.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.894+0000 7fdd45000640 1 -- 172.21.15.39:0/2884888472 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd4019f8c0 con 0x7fdd401016c0 2024-03-18T16:07:03.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.895+0000 7fdd45000640 1 -- 172.21.15.39:0/2884888472 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd4019fe30 con 0x7fdd401016c0 2024-03-18T16:07:03.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.896+0000 7fdd45000640 1 -- 172.21.15.39:0/2884888472 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd40108980 con 0x7fdd401016c0 2024-03-18T16:07:03.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.898+0000 7fdd3cc00640 1 -- 172.21.15.39:0/2884888472 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fdd300037a0 con 0x7fdd401016c0 2024-03-18T16:07:03.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.899+0000 7fdd3cc00640 1 --2- 172.21.15.39:0/2884888472 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fdd240a2880 0x7fdd240a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:03.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.899+0000 7fdd3cc00640 1 -- 172.21.15.39:0/2884888472 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fdd300ceff0 con 0x7fdd401016c0 2024-03-18T16:07:03.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.899+0000 7fdd37e00640 1 --2- 172.21.15.39:0/2884888472 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fdd240a2880 0x7fdd240a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:03.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.900+0000 7fdd37e00640 1 --2- 172.21.15.39:0/2884888472 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fdd240a2880 0x7fdd240a4d40 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fdd2802c250 tx=0x7fdd2802cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:03.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:03.901+0000 7fdd3cc00640 1 -- 172.21.15.39:0/2884888472 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fdd3009be10 con 0x7fdd401016c0 2024-03-18T16:07:04.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.104+0000 7fdd45000640 1 -- 172.21.15.39:0/2884888472 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fdd4002d420 con 0x7fdd401016c0 2024-03-18T16:07:04.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.105+0000 7fdd3cc00640 1 -- 172.21.15.39:0/2884888472 <== mon.2 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) ==== 100+0+11 (secure 0 0 0) 0x7fdd3009b710 con 0x7fdd401016c0 2024-03-18T16:07:04.108 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 64 2024-03-18T16:07:04.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.108+0000 7fdd45000640 1 -- 172.21.15.39:0/2884888472 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fdd240a2880 msgr2=0x7fdd240a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:04.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.108+0000 7fdd45000640 1 --2- 172.21.15.39:0/2884888472 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fdd240a2880 0x7fdd240a4d40 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fdd2802c250 tx=0x7fdd2802cf90 comp rx=0 tx=0).stop 2024-03-18T16:07:04.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.108+0000 7fdd45000640 1 -- 172.21.15.39:0/2884888472 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdd401016c0 msgr2=0x7fdd401979b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:04.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.108+0000 7fdd45000640 1 --2- 172.21.15.39:0/2884888472 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdd401016c0 0x7fdd401979b0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fdd3000e9c0 tx=0x7fdd3000ee80 comp rx=0 tx=0).stop 2024-03-18T16:07:04.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.108+0000 7fdd45000640 1 -- 172.21.15.39:0/2884888472 shutdown_connections 2024-03-18T16:07:04.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.108+0000 7fdd45000640 1 --2- 172.21.15.39:0/2884888472 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fdd240a2880 0x7fdd240a4d40 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:04.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.108+0000 7fdd45000640 1 --2- 172.21.15.39:0/2884888472 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdd40103270 0x7fdd4019ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:04.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.108+0000 7fdd45000640 1 --2- 172.21.15.39:0/2884888472 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdd401028b0 0x7fdd40197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:04.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.108+0000 7fdd45000640 1 --2- 172.21.15.39:0/2884888472 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdd401016c0 0x7fdd401979b0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:04.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.108+0000 7fdd45000640 1 -- 172.21.15.39:0/2884888472 >> 172.21.15.39:0/2884888472 conn(0x7fdd400fcc50 msgr2=0x7fdd40105b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:04.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.108+0000 7fdd45000640 1 -- 172.21.15.39:0/2884888472 shutdown_connections 2024-03-18T16:07:04.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.108+0000 7fdd45000640 1 -- 172.21.15.39:0/2884888472 wait complete. 2024-03-18T16:07:04.130 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T16:07:04.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:04 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/1943419460' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T16:07:04.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:04 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/2239186669' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T16:07:04.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.352+0000 7ff5f1a00640 1 -- 172.21.15.39:0/244317686 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff5ec103900 msgr2=0x7ff5ec103d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:04.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.352+0000 7ff5f1a00640 1 --2- 172.21.15.39:0/244317686 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff5ec103900 0x7ff5ec103d20 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7ff5dc009900 tx=0x7ff5dc02bc30 comp rx=0 tx=0).stop 2024-03-18T16:07:04.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.352+0000 7ff5f1a00640 1 -- 172.21.15.39:0/244317686 shutdown_connections 2024-03-18T16:07:04.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.352+0000 7ff5f1a00640 1 --2- 172.21.15.39:0/244317686 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff5ec1054b0 0x7ff5ec10bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:04.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.352+0000 7ff5f1a00640 1 --2- 172.21.15.39:0/244317686 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff5ec104af0 0x7ff5ec104f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:04.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.352+0000 7ff5f1a00640 1 --2- 172.21.15.39:0/244317686 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff5ec103900 0x7ff5ec103d20 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:04.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.352+0000 7ff5f1a00640 1 -- 172.21.15.39:0/244317686 >> 172.21.15.39:0/244317686 conn(0x7ff5ec0fee90 msgr2=0x7ff5ec1012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:04.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.352+0000 7ff5f1a00640 1 -- 172.21.15.39:0/244317686 shutdown_connections 2024-03-18T16:07:04.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.352+0000 7ff5f1a00640 1 -- 172.21.15.39:0/244317686 wait complete. 2024-03-18T16:07:04.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.354+0000 7ff5f1a00640 1 Processor -- start 2024-03-18T16:07:04.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.354+0000 7ff5f1a00640 1 -- start start 2024-03-18T16:07:04.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.354+0000 7ff5f1a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff5ec103900 0x7ff5ec199c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:04.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.354+0000 7ff5f1a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff5ec104af0 0x7ff5ec19a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:04.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.354+0000 7ff5f1a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff5ec1054b0 0x7ff5ec1a11d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:04.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.354+0000 7ff5f1a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7ff5ec10f570 con 0x7ff5ec104af0 2024-03-18T16:07:04.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.355+0000 7ff5f1a00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ff5ec10f3f0 con 0x7ff5ec103900 2024-03-18T16:07:04.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.355+0000 7ff5f1a00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7ff5ec10f6f0 con 0x7ff5ec1054b0 2024-03-18T16:07:04.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.355+0000 7ff5ebe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff5ec103900 0x7ff5ec199c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:04.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.355+0000 7ff5ebe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff5ec103900 0x7ff5ec199c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:49726/0 (socket says 172.21.15.39:49726) 2024-03-18T16:07:04.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.355+0000 7ff5ebe00640 1 -- 172.21.15.39:0/1151730733 learned_addr learned my addr 172.21.15.39:0/1151730733 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:07:04.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.355+0000 7ff5e3e00640 1 --2- 172.21.15.39:0/1151730733 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff5ec104af0 0x7ff5ec19a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:04.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.355+0000 7ff5f1000640 1 --2- 172.21.15.39:0/1151730733 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff5ec1054b0 0x7ff5ec1a11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:04.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.355+0000 7ff5ebe00640 1 -- 172.21.15.39:0/1151730733 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff5ec1054b0 msgr2=0x7ff5ec1a11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:04.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.355+0000 7ff5ebe00640 1 --2- 172.21.15.39:0/1151730733 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff5ec1054b0 0x7ff5ec1a11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:04.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.355+0000 7ff5ebe00640 1 -- 172.21.15.39:0/1151730733 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff5ec104af0 msgr2=0x7ff5ec19a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:04.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.355+0000 7ff5ebe00640 1 --2- 172.21.15.39:0/1151730733 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff5ec104af0 0x7ff5ec19a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:04.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.355+0000 7ff5ebe00640 1 -- 172.21.15.39:0/1151730733 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5ec1a1870 con 0x7ff5ec103900 2024-03-18T16:07:04.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.356+0000 7ff5ebe00640 1 --2- 172.21.15.39:0/1151730733 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff5ec103900 0x7ff5ec199c60 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff5dc000c00 tx=0x7ff5dc02c640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:04.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.356+0000 7ff5e9600640 1 -- 172.21.15.39:0/1151730733 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5dc041070 con 0x7ff5ec103900 2024-03-18T16:07:04.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.356+0000 7ff5e9600640 1 -- 172.21.15.39:0/1151730733 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff5dc004450 con 0x7ff5ec103900 2024-03-18T16:07:04.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.356+0000 7ff5e9600640 1 -- 172.21.15.39:0/1151730733 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5dc003df0 con 0x7ff5ec103900 2024-03-18T16:07:04.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.356+0000 7ff5f1a00640 1 -- 172.21.15.39:0/1151730733 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5ec1a1b00 con 0x7ff5ec103900 2024-03-18T16:07:04.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.356+0000 7ff5f1a00640 1 -- 172.21.15.39:0/1151730733 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5ec1a2040 con 0x7ff5ec103900 2024-03-18T16:07:04.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.357+0000 7ff5f1a00640 1 -- 172.21.15.39:0/1151730733 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5ec10aca0 con 0x7ff5ec103900 2024-03-18T16:07:04.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.359+0000 7ff5e9600640 1 -- 172.21.15.39:0/1151730733 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7ff5dc044070 con 0x7ff5ec103900 2024-03-18T16:07:04.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.360+0000 7ff5e9600640 1 --2- 172.21.15.39:0/1151730733 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff5d00a2880 0x7ff5d00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:04.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.360+0000 7ff5e9600640 1 -- 172.21.15.39:0/1151730733 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7ff5dc0f07e0 con 0x7ff5ec103900 2024-03-18T16:07:04.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.360+0000 7ff5e3e00640 1 --2- 172.21.15.39:0/1151730733 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff5d00a2880 0x7ff5d00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:04.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.361+0000 7ff5e3e00640 1 --2- 172.21.15.39:0/1151730733 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff5d00a2880 0x7ff5d00a4d40 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff5d4009bd0 tx=0x7ff5d4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:04.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.364+0000 7ff5e9600640 1 -- 172.21.15.39:0/1151730733 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff5dc0bd5e0 con 0x7ff5ec103900 2024-03-18T16:07:04.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:04 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/1943419460' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T16:07:04.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:04 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/2239186669' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T16:07:04.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:04 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/1943419460' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T16:07:04.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:04 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/2239186669' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T16:07:04.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.567+0000 7ff5f1a00640 1 -- 172.21.15.39:0/1151730733 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7ff5ec02d1b0 con 0x7ff5ec103900 2024-03-18T16:07:04.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.568+0000 7ff5e9600640 1 -- 172.21.15.39:0/1151730733 <== mon.2 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) ==== 103+0+11 (secure 0 0 0) 0x7ff5dc0bcee0 con 0x7ff5ec103900 2024-03-18T16:07:04.570 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 32 2024-03-18T16:07:04.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.571+0000 7ff5f1a00640 1 -- 172.21.15.39:0/1151730733 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff5d00a2880 msgr2=0x7ff5d00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:04.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.571+0000 7ff5f1a00640 1 --2- 172.21.15.39:0/1151730733 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff5d00a2880 0x7ff5d00a4d40 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff5d4009bd0 tx=0x7ff5d4005f90 comp rx=0 tx=0).stop 2024-03-18T16:07:04.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.571+0000 7ff5f1a00640 1 -- 172.21.15.39:0/1151730733 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff5ec103900 msgr2=0x7ff5ec199c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:04.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.571+0000 7ff5f1a00640 1 --2- 172.21.15.39:0/1151730733 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff5ec103900 0x7ff5ec199c60 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff5dc000c00 tx=0x7ff5dc02c640 comp rx=0 tx=0).stop 2024-03-18T16:07:04.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.571+0000 7ff5f1a00640 1 -- 172.21.15.39:0/1151730733 shutdown_connections 2024-03-18T16:07:04.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.571+0000 7ff5f1a00640 1 --2- 172.21.15.39:0/1151730733 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff5d00a2880 0x7ff5d00a4d40 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:04.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.571+0000 7ff5f1a00640 1 --2- 172.21.15.39:0/1151730733 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff5ec1054b0 0x7ff5ec1a11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:04.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.571+0000 7ff5f1a00640 1 --2- 172.21.15.39:0/1151730733 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff5ec104af0 0x7ff5ec19a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:04.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.571+0000 7ff5f1a00640 1 --2- 172.21.15.39:0/1151730733 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff5ec103900 0x7ff5ec199c60 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:04.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.571+0000 7ff5f1a00640 1 -- 172.21.15.39:0/1151730733 >> 172.21.15.39:0/1151730733 conn(0x7ff5ec0fee90 msgr2=0x7ff5ec107e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:04.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.571+0000 7ff5f1a00640 1 -- 172.21.15.39:0/1151730733 shutdown_connections 2024-03-18T16:07:04.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.571+0000 7ff5f1a00640 1 -- 172.21.15.39:0/1151730733 wait complete. 2024-03-18T16:07:04.594 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-18T16:07:04.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.816+0000 7fdd87e00640 1 -- 172.21.15.39:0/4207032501 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdd88104a70 msgr2=0x7fdd88106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:04.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.816+0000 7fdd87e00640 1 --2- 172.21.15.39:0/4207032501 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdd88104a70 0x7fdd88106e60 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fdd70009900 tx=0x7fdd7002bc40 comp rx=0 tx=0).stop 2024-03-18T16:07:04.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.816+0000 7fdd87e00640 1 -- 172.21.15.39:0/4207032501 shutdown_connections 2024-03-18T16:07:04.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.816+0000 7fdd87e00640 1 --2- 172.21.15.39:0/4207032501 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdd88104a70 0x7fdd88106e60 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:04.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.816+0000 7fdd87e00640 1 --2- 172.21.15.39:0/4207032501 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdd881073a0 0x7fdd881097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:04.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.816+0000 7fdd87e00640 1 --2- 172.21.15.39:0/4207032501 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdd88102140 0x7fdd88104530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:04.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.816+0000 7fdd87e00640 1 -- 172.21.15.39:0/4207032501 >> 172.21.15.39:0/4207032501 conn(0x7fdd880fbb50 msgr2=0x7fdd880fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:04.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.816+0000 7fdd87e00640 1 -- 172.21.15.39:0/4207032501 shutdown_connections 2024-03-18T16:07:04.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.816+0000 7fdd87e00640 1 -- 172.21.15.39:0/4207032501 wait complete. 2024-03-18T16:07:04.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.818+0000 7fdd87e00640 1 Processor -- start 2024-03-18T16:07:04.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.818+0000 7fdd87e00640 1 -- start start 2024-03-18T16:07:04.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.818+0000 7fdd87e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdd88102140 0x7fdd88101ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:04.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.818+0000 7fdd87e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdd88104a70 0x7fdd881000f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:04.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.818+0000 7fdd87e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdd881073a0 0x7fdd88100630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:04.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.818+0000 7fdd87e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fdd8810d380 con 0x7fdd881073a0 2024-03-18T16:07:04.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.818+0000 7fdd87e00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fdd8810d200 con 0x7fdd88102140 2024-03-18T16:07:04.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.818+0000 7fdd87e00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fdd8810d500 con 0x7fdd88104a70 2024-03-18T16:07:04.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.819+0000 7fdd86a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdd88102140 0x7fdd88101ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:04.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.819+0000 7fdd86a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdd88102140 0x7fdd88101ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:49744/0 (socket says 172.21.15.39:49744) 2024-03-18T16:07:04.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.819+0000 7fdd86a00640 1 -- 172.21.15.39:0/292627902 learned_addr learned my addr 172.21.15.39:0/292627902 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:07:04.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.819+0000 7fdd87400640 1 --2- 172.21.15.39:0/292627902 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdd881073a0 0x7fdd88100630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:04.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.819+0000 7fdd7e000640 1 --2- 172.21.15.39:0/292627902 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdd88104a70 0x7fdd881000f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:04.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.819+0000 7fdd86a00640 1 -- 172.21.15.39:0/292627902 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdd88104a70 msgr2=0x7fdd881000f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:04.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.819+0000 7fdd86a00640 1 --2- 172.21.15.39:0/292627902 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdd88104a70 0x7fdd881000f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:04.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.819+0000 7fdd86a00640 1 -- 172.21.15.39:0/292627902 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdd881073a0 msgr2=0x7fdd88100630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:04.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.819+0000 7fdd86a00640 1 --2- 172.21.15.39:0/292627902 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdd881073a0 0x7fdd88100630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:04.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.819+0000 7fdd86a00640 1 -- 172.21.15.39:0/292627902 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd88100ec0 con 0x7fdd88102140 2024-03-18T16:07:04.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.820+0000 7fdd86a00640 1 --2- 172.21.15.39:0/292627902 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdd88102140 0x7fdd88101ab0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fdd7800ebc0 tx=0x7fdd7800c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:04.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.820+0000 7fdd8ca00640 1 -- 172.21.15.39:0/292627902 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd78010630 con 0x7fdd88102140 2024-03-18T16:07:04.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.820+0000 7fdd8ca00640 1 -- 172.21.15.39:0/292627902 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd78010cb0 con 0x7fdd88102140 2024-03-18T16:07:04.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.820+0000 7fdd87e00640 1 -- 172.21.15.39:0/292627902 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd881a1a80 con 0x7fdd88102140 2024-03-18T16:07:04.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.820+0000 7fdd8ca00640 1 -- 172.21.15.39:0/292627902 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd78018bd0 con 0x7fdd88102140 2024-03-18T16:07:04.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.820+0000 7fdd87e00640 1 -- 172.21.15.39:0/292627902 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd881a1ff0 con 0x7fdd88102140 2024-03-18T16:07:04.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.821+0000 7fdd87e00640 1 -- 172.21.15.39:0/292627902 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd880fd640 con 0x7fdd88102140 2024-03-18T16:07:04.825 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.823+0000 7fdd8ca00640 1 -- 172.21.15.39:0/292627902 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fdd780107d0 con 0x7fdd88102140 2024-03-18T16:07:04.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.824+0000 7fdd8ca00640 1 --2- 172.21.15.39:0/292627902 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fdd6c0a2880 0x7fdd6c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:04.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.824+0000 7fdd8ca00640 1 -- 172.21.15.39:0/292627902 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fdd780cf260 con 0x7fdd88102140 2024-03-18T16:07:04.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.825+0000 7fdd7e000640 1 --2- 172.21.15.39:0/292627902 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fdd6c0a2880 0x7fdd6c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:04.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.825+0000 7fdd7e000640 1 --2- 172.21.15.39:0/292627902 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fdd6c0a2880 0x7fdd6c0a4d40 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fdd7002c180 tx=0x7fdd7002cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:04.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:04.829+0000 7fdd8ca00640 1 -- 172.21.15.39:0/292627902 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fdd7809ba10 con 0x7fdd88102140 2024-03-18T16:07:05.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.044+0000 7fdd87e00640 1 -- 172.21.15.39:0/292627902 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fdd880623e0 con 0x7fdd88102140 2024-03-18T16:07:05.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.044+0000 7fdd8ca00640 1 -- 172.21.15.39:0/292627902 <== mon.2 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v89) ==== 81+0+592 (secure 0 0 0) 0x7fdd7809fab0 con 0x7fdd88102140 2024-03-18T16:07:05.047 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/3482086140 2024-03-19T15:58:45.340294+0000 2024-03-18T16:07:05.047 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/3844306284 2024-03-19T15:58:45.340294+0000 2024-03-18T16:07:05.047 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:6801/1810238546 2024-03-19T15:58:45.340294+0000 2024-03-18T16:07:05.047 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:6800/1810238546 2024-03-19T15:58:45.340294+0000 2024-03-18T16:07:05.047 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/2528397488 2024-03-19T15:58:10.264798+0000 2024-03-18T16:07:05.048 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/2240981254 2024-03-19T15:58:10.264798+0000 2024-03-18T16:07:05.048 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:6800/1217503189 2024-03-19T15:58:10.264798+0000 2024-03-18T16:07:05.048 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/3754876370 2024-03-19T15:58:10.264798+0000 2024-03-18T16:07:05.048 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:6801/1217503189 2024-03-19T15:58:10.264798+0000 2024-03-18T16:07:05.048 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/1874637628 2024-03-19T15:58:45.340294+0000 2024-03-18T16:07:05.048 INFO:teuthology.orchestra.run.smithi039.stderr:listed 10 entries 2024-03-18T16:07:05.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.047+0000 7fdd87e00640 1 -- 172.21.15.39:0/292627902 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fdd6c0a2880 msgr2=0x7fdd6c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:05.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.047+0000 7fdd87e00640 1 --2- 172.21.15.39:0/292627902 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fdd6c0a2880 0x7fdd6c0a4d40 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fdd7002c180 tx=0x7fdd7002cf90 comp rx=0 tx=0).stop 2024-03-18T16:07:05.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.048+0000 7fdd87e00640 1 -- 172.21.15.39:0/292627902 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdd88102140 msgr2=0x7fdd88101ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:05.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.048+0000 7fdd87e00640 1 --2- 172.21.15.39:0/292627902 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdd88102140 0x7fdd88101ab0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fdd7800ebc0 tx=0x7fdd7800c6a0 comp rx=0 tx=0).stop 2024-03-18T16:07:05.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.048+0000 7fdd87e00640 1 -- 172.21.15.39:0/292627902 shutdown_connections 2024-03-18T16:07:05.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.048+0000 7fdd87e00640 1 --2- 172.21.15.39:0/292627902 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fdd6c0a2880 0x7fdd6c0a4d40 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:05.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.048+0000 7fdd87e00640 1 --2- 172.21.15.39:0/292627902 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdd881073a0 0x7fdd88100630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:05.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.048+0000 7fdd87e00640 1 --2- 172.21.15.39:0/292627902 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdd88104a70 0x7fdd881000f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:05.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.048+0000 7fdd87e00640 1 --2- 172.21.15.39:0/292627902 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdd88102140 0x7fdd88101ab0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:05.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.048+0000 7fdd87e00640 1 -- 172.21.15.39:0/292627902 >> 172.21.15.39:0/292627902 conn(0x7fdd880fbb50 msgr2=0x7fdd88102a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:05.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.048+0000 7fdd87e00640 1 -- 172.21.15.39:0/292627902 shutdown_connections 2024-03-18T16:07:05.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.048+0000 7fdd87e00640 1 -- 172.21.15.39:0/292627902 wait complete. 2024-03-18T16:07:05.071 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-03-18T16:07:05.071 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T16:07:05.126 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T16:07:05.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:05 smithi039 ceph-mon[31721]: pgmap v328: 129 pgs: 129 active+clean; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:07:05.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:05 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/2884888472' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T16:07:05.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:05 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/1151730733' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T16:07:05.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:05 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/292627902' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-18T16:07:05.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.349+0000 7f1599600640 1 -- 172.21.15.39:0/467672626 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f15941029c0 msgr2=0x7f1594104e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:05.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.349+0000 7f1599600640 1 --2- 172.21.15.39:0/467672626 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f15941029c0 0x7f1594104e30 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f157c009900 tx=0x7f157c02bc30 comp rx=0 tx=0).stop 2024-03-18T16:07:05.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.349+0000 7f1599600640 1 -- 172.21.15.39:0/467672626 shutdown_connections 2024-03-18T16:07:05.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.349+0000 7f1599600640 1 --2- 172.21.15.39:0/467672626 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f15941029c0 0x7f1594104e30 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:05.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.349+0000 7f1599600640 1 --2- 172.21.15.39:0/467672626 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1594105370 0x7f15941057f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:05.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.349+0000 7f1599600640 1 --2- 172.21.15.39:0/467672626 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1594100050 0x7f1594102480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:05.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.349+0000 7f1599600640 1 -- 172.21.15.39:0/467672626 >> 172.21.15.39:0/467672626 conn(0x7f15940fbc30 msgr2=0x7f15940fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:05.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.349+0000 7f1599600640 1 -- 172.21.15.39:0/467672626 shutdown_connections 2024-03-18T16:07:05.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.349+0000 7f1599600640 1 -- 172.21.15.39:0/467672626 wait complete. 2024-03-18T16:07:05.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.350+0000 7f1599600640 1 Processor -- start 2024-03-18T16:07:05.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.350+0000 7f1599600640 1 -- start start 2024-03-18T16:07:05.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.350+0000 7f1599600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1594100050 0x7f159419e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:05.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.350+0000 7f1599600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f15941029c0 0x7f159419e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:05.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.350+0000 7f1599600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1594105370 0x7f15941a5680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:05.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.350+0000 7f1599600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f1594113a80 con 0x7f15941029c0 2024-03-18T16:07:05.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.350+0000 7f1599600640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f1594113900 con 0x7f1594105370 2024-03-18T16:07:05.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.350+0000 7f1599600640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f1594113c00 con 0x7f1594100050 2024-03-18T16:07:05.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.351+0000 7f158a000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f15941029c0 0x7f159419e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:05.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.351+0000 7f1593400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1594105370 0x7f15941a5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:05.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.351+0000 7f1592a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1594100050 0x7f159419e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:05.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.351+0000 7f1593400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1594105370 0x7f15941a5680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:49756/0 (socket says 172.21.15.39:49756) 2024-03-18T16:07:05.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.351+0000 7f1593400640 1 -- 172.21.15.39:0/1092152846 learned_addr learned my addr 172.21.15.39:0/1092152846 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:07:05.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.351+0000 7f1593400640 1 -- 172.21.15.39:0/1092152846 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1594100050 msgr2=0x7f159419e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:05.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.351+0000 7f1593400640 1 --2- 172.21.15.39:0/1092152846 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1594100050 0x7f159419e0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:05.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.351+0000 7f1593400640 1 -- 172.21.15.39:0/1092152846 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f15941029c0 msgr2=0x7f159419e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:05.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.351+0000 7f1593400640 1 --2- 172.21.15.39:0/1092152846 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f15941029c0 0x7f159419e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:05.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.351+0000 7f1593400640 1 -- 172.21.15.39:0/1092152846 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15941a5cf0 con 0x7f1594105370 2024-03-18T16:07:05.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.351+0000 7f1593400640 1 --2- 172.21.15.39:0/1092152846 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1594105370 0x7f15941a5680 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f158c00d750 tx=0x7f158c00dc10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:05.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.352+0000 7f158be00640 1 -- 172.21.15.39:0/1092152846 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f158c017070 con 0x7f1594105370 2024-03-18T16:07:05.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.352+0000 7f158be00640 1 -- 172.21.15.39:0/1092152846 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f158c005500 con 0x7f1594105370 2024-03-18T16:07:05.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.352+0000 7f1599600640 1 -- 172.21.15.39:0/1092152846 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15941a5fe0 con 0x7f1594105370 2024-03-18T16:07:05.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.352+0000 7f158be00640 1 -- 172.21.15.39:0/1092152846 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f158c004d90 con 0x7f1594105370 2024-03-18T16:07:05.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.352+0000 7f1599600640 1 -- 172.21.15.39:0/1092152846 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15941a64a0 con 0x7f1594105370 2024-03-18T16:07:05.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.353+0000 7f1599600640 1 -- 172.21.15.39:0/1092152846 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15940fd720 con 0x7f1594105370 2024-03-18T16:07:05.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.355+0000 7f158be00640 1 -- 172.21.15.39:0/1092152846 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f158c004f30 con 0x7f1594105370 2024-03-18T16:07:05.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.356+0000 7f158be00640 1 --2- 172.21.15.39:0/1092152846 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f15780a2880 0x7f15780a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:05.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.356+0000 7f158be00640 1 -- 172.21.15.39:0/1092152846 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f158c0ce540 con 0x7f1594105370 2024-03-18T16:07:05.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.356+0000 7f1592a00640 1 --2- 172.21.15.39:0/1092152846 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f15780a2880 0x7f15780a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:05.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.357+0000 7f1592a00640 1 --2- 172.21.15.39:0/1092152846 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f15780a2880 0x7f15780a4d40 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f1584005fd0 tx=0x7f1584009680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:05.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.360+0000 7f158be00640 1 -- 172.21.15.39:0/1092152846 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f158c09ad50 con 0x7f1594105370 2024-03-18T16:07:05.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:05 smithi196 ceph-mon[36307]: pgmap v328: 129 pgs: 129 active+clean; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:07:05.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:05 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/2884888472' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T16:07:05.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:05 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/1151730733' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T16:07:05.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:05 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/292627902' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-18T16:07:05.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:05 smithi194 ceph-mon[36200]: pgmap v328: 129 pgs: 129 active+clean; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:07:05.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:05 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/2884888472' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T16:07:05.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:05 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/1151730733' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T16:07:05.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:05 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/292627902' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-18T16:07:05.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.563+0000 7f1599600640 1 -- 172.21.15.39:0/1092152846 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f15940623e0 con 0x7f1594105370 2024-03-18T16:07:05.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.565+0000 7f158be00640 1 -- 172.21.15.39:0/1092152846 <== mon.2 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24142 (secure 0 0 0) 0x7f158c09edf0 con 0x7f1594105370 2024-03-18T16:07:05.567 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:07:05.567 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":89,"fsid":"1892d8fc-e540-11ee-95c9-87774f69a715","created":"2024-03-18T15:57:36.507008+0000","modified":"2024-03-18T16:06:49.510653+0000","last_up_change":"2024-03-18T16:05:45.250629+0000","last_in_change":"2024-03-18T16:05:24.841337+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T16:01:30.048247+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T16:06:42.191791+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T16:06:43.670325+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T16:06:47.523388+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"771e0558-be62-4436-94b0-6364751911c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6803","nonce":4090743635}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6805","nonce":4090743635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6809","nonce":4090743635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6807","nonce":4090743635}]},"public_addr":"172.21.15.39:6803/4090743635","cluster_addr":"172.21.15.39:6805/4090743635","heartbeat_back_addr":"172.21.15.39:6809/4090743635","heartbeat_front_addr":"172.21.15.39:6807/4090743635","state":["exists","up"]},{"osd":1,"uuid":"0430eee1-cf34-4277-b255-c7d9f07ad19a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6801","nonce":2800329805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6803","nonce":2800329805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6807","nonce":2800329805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6805","nonce":2800329805}]},"public_addr":"172.21.15.194:6801/2800329805","cluster_addr":"172.21.15.194:6803/2800329805","heartbeat_back_addr":"172.21.15.194:6807/2800329805","heartbeat_front_addr":"172.21.15.194:6805/2800329805","state":["exists","up"]},{"osd":2,"uuid":"3a718da4-2511-4ec1-b1a8-eafd6cce0df0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6801","nonce":3935874719}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6803","nonce":3935874719}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6807","nonce":3935874719}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6805","nonce":3935874719}]},"public_addr":"172.21.15.196:6801/3935874719","cluster_addr":"172.21.15.196:6803/3935874719","heartbeat_back_addr":"172.21.15.196:6807/3935874719","heartbeat_front_addr":"172.21.15.196:6805/3935874719","state":["exists","up"]},{"osd":3,"uuid":"6f9c37f9-0ab9-4038-8919-afd4b39c4165","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6811","nonce":893345939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6813","nonce":893345939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6817","nonce":893345939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6815","nonce":893345939}]},"public_addr":"172.21.15.39:6811/893345939","cluster_addr":"172.21.15.39:6813/893345939","heartbeat_back_addr":"172.21.15.39:6817/893345939","heartbeat_front_addr":"172.21.15.39:6815/893345939","state":["exists","up"]},{"osd":4,"uuid":"e795bb0e-4791-4537-acda-1a321e19bc9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6809","nonce":800423088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6811","nonce":800423088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6815","nonce":800423088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6813","nonce":800423088}]},"public_addr":"172.21.15.194:6809/800423088","cluster_addr":"172.21.15.194:6811/800423088","heartbeat_back_addr":"172.21.15.194:6815/800423088","heartbeat_front_addr":"172.21.15.194:6813/800423088","state":["exists","up"]},{"osd":5,"uuid":"026e175e-6acb-4b4c-b235-8ccee8ae1149","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6809","nonce":2746224525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6811","nonce":2746224525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6815","nonce":2746224525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6813","nonce":2746224525}]},"public_addr":"172.21.15.196:6809/2746224525","cluster_addr":"172.21.15.196:6811/2746224525","heartbeat_back_addr":"172.21.15.196:6815/2746224525","heartbeat_front_addr":"172.21.15.196:6813/2746224525","state":["exists","up"]},{"osd":6,"uuid":"c1b183ff-383f-4c5c-9075-f76bab1c0bb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6819","nonce":1213371331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6821","nonce":1213371331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6825","nonce":1213371331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6823","nonce":1213371331}]},"public_addr":"172.21.15.39:6819/1213371331","cluster_addr":"172.21.15.39:6821/1213371331","heartbeat_back_addr":"172.21.15.39:6825/1213371331","heartbeat_front_addr":"172.21.15.39:6823/1213371331","state":["exists","up"]},{"osd":7,"uuid":"612f8697-957d-4ef4-a87f-241b36f9d13d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6817","nonce":3299162444}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6819","nonce":3299162444}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6823","nonce":3299162444}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6821","nonce":3299162444}]},"public_addr":"172.21.15.194:6817/3299162444","cluster_addr":"172.21.15.194:6819/3299162444","heartbeat_back_addr":"172.21.15.194:6823/3299162444","heartbeat_front_addr":"172.21.15.194:6821/3299162444","state":["exists","up"]},{"osd":8,"uuid":"836a9107-4d4f-4594-b8dc-0301e64f59d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6817","nonce":2456143145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6819","nonce":2456143145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6823","nonce":2456143145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6821","nonce":2456143145}]},"public_addr":"172.21.15.196:6817/2456143145","cluster_addr":"172.21.15.196:6819/2456143145","heartbeat_back_addr":"172.21.15.196:6823/2456143145","heartbeat_front_addr":"172.21.15.196:6821/2456143145","state":["exists","up"]},{"osd":9,"uuid":"4f30b960-12e3-4b16-bd28-fcc988139b58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6827","nonce":1848634433}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6829","nonce":1848634433}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6833","nonce":1848634433}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6831","nonce":1848634433}]},"public_addr":"172.21.15.39:6827/1848634433","cluster_addr":"172.21.15.39:6829/1848634433","heartbeat_back_addr":"172.21.15.39:6833/1848634433","heartbeat_front_addr":"172.21.15.39:6831/1848634433","state":["exists","up"]},{"osd":10,"uuid":"620f90e8-23e6-4bc0-8269-7219b9e1aebd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6825","nonce":3725064809}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6827","nonce":3725064809}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6831","nonce":3725064809}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6829","nonce":3725064809}]},"public_addr":"172.21.15.194:6825/3725064809","cluster_addr":"172.21.15.194:6827/3725064809","heartbeat_back_addr":"172.21.15.194:6831/3725064809","heartbeat_front_addr":"172.21.15.194:6829/3725064809","state":["exists","up"]},{"osd":11,"uuid":"867a2535-8235-4ceb-a698-eb86869d6d20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6825","nonce":2501212422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6827","nonce":2501212422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6831","nonce":2501212422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6829","nonce":2501212422}]},"public_addr":"172.21.15.196:6825/2501212422","cluster_addr":"172.21.15.196:6827/2501212422","heartbeat_back_addr":"172.21.15.196:6831/2501212422","heartbeat_front_addr":"172.21.15.196:6829/2501212422","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:00:36.931566+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:01:02.614024+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:01:26.193265+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:01:58.308833+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:02:25.049546+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:02:49.748750+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:03:23.582153+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:03:50.447321+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:04:15.812129+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:04:49.998047+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:05:17.087924+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:05:43.167637+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.39:0/3482086140":"2024-03-19T15:58:45.340294+0000","172.21.15.39:0/3844306284":"2024-03-19T15:58:45.340294+0000","172.21.15.39:6801/1810238546":"2024-03-19T15:58:45.340294+0000","172.21.15.39:6800/1810238546":"2024-03-19T15:58:45.340294+0000","172.21.15.39:0/2528397488":"2024-03-19T15:58:10.264798+0000","172.21.15.39:0/2240981254":"2024-03-19T15:58:10.264798+0000","172.21.15.39:6800/1217503189":"2024-03-19T15:58:10.264798+0000","172.21.15.39:0/3754876370":"2024-03-19T15:58:10.264798+0000","172.21.15.39:6801/1217503189":"2024-03-19T15:58:10.264798+0000","172.21.15.39:0/1874637628":"2024-03-19T15:58:45.340294+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T16:07:05.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.567+0000 7f1599600640 1 -- 172.21.15.39:0/1092152846 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f15780a2880 msgr2=0x7f15780a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:05.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.567+0000 7f1599600640 1 --2- 172.21.15.39:0/1092152846 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f15780a2880 0x7f15780a4d40 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f1584005fd0 tx=0x7f1584009680 comp rx=0 tx=0).stop 2024-03-18T16:07:05.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.567+0000 7f1599600640 1 -- 172.21.15.39:0/1092152846 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1594105370 msgr2=0x7f15941a5680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:05.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.567+0000 7f1599600640 1 --2- 172.21.15.39:0/1092152846 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1594105370 0x7f15941a5680 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f158c00d750 tx=0x7f158c00dc10 comp rx=0 tx=0).stop 2024-03-18T16:07:05.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.567+0000 7f1599600640 1 -- 172.21.15.39:0/1092152846 shutdown_connections 2024-03-18T16:07:05.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.567+0000 7f1599600640 1 --2- 172.21.15.39:0/1092152846 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1594105370 0x7f15941a5680 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:05.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.567+0000 7f1599600640 1 --2- 172.21.15.39:0/1092152846 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f15780a2880 0x7f15780a4d40 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:05.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.567+0000 7f1599600640 1 --2- 172.21.15.39:0/1092152846 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f15941029c0 0x7f159419e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:05.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.567+0000 7f1599600640 1 --2- 172.21.15.39:0/1092152846 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1594100050 0x7f159419e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:05.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.567+0000 7f1599600640 1 -- 172.21.15.39:0/1092152846 >> 172.21.15.39:0/1092152846 conn(0x7f15940fbc30 msgr2=0x7f15941009d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:05.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.567+0000 7f1599600640 1 -- 172.21.15.39:0/1092152846 shutdown_connections 2024-03-18T16:07:05.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.567+0000 7f1599600640 1 -- 172.21.15.39:0/1092152846 wait complete. 2024-03-18T16:07:05.589 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T16:01:30.048247+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T16:06:42.191791+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T16:06:43.670325+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T16:06:47.523388+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T16:07:05.589 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T16:07:05.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.807+0000 7f1fb5600640 1 -- 172.21.15.39:0/2441357418 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1fb01098b0 msgr2=0x7f1fb0110140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:05.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.807+0000 7f1fb5600640 1 --2- 172.21.15.39:0/2441357418 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1fb01098b0 0x7f1fb0110140 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f1fa8009ca0 tx=0x7f1fa802c0f0 comp rx=0 tx=0).stop 2024-03-18T16:07:05.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.807+0000 7f1fb5600640 1 -- 172.21.15.39:0/2441357418 shutdown_connections 2024-03-18T16:07:05.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.807+0000 7f1fb5600640 1 --2- 172.21.15.39:0/2441357418 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1fb0108ef0 0x7f1fb0109370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:05.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.807+0000 7f1fb5600640 1 --2- 172.21.15.39:0/2441357418 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1fb01098b0 0x7f1fb0110140 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:05.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.807+0000 7f1fb5600640 1 --2- 172.21.15.39:0/2441357418 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1fb0107d00 0x7f1fb0108120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:05.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.807+0000 7f1fb5600640 1 -- 172.21.15.39:0/2441357418 >> 172.21.15.39:0/2441357418 conn(0x7f1fb0076cb0 msgr2=0x7f1fb00790d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:05.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.807+0000 7f1fb5600640 1 -- 172.21.15.39:0/2441357418 shutdown_connections 2024-03-18T16:07:05.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.807+0000 7f1fb5600640 1 -- 172.21.15.39:0/2441357418 wait complete. 2024-03-18T16:07:05.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.808+0000 7f1fb5600640 1 Processor -- start 2024-03-18T16:07:05.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.808+0000 7f1fb5600640 1 -- start start 2024-03-18T16:07:05.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.809+0000 7f1fb5600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1fb0107d00 0x7f1fb019dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:05.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.809+0000 7f1fb5600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1fb0108ef0 0x7f1fb019e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:05.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.809+0000 7f1fb5600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1fb01098b0 0x7f1fb01a5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:05.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.809+0000 7f1fb5600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f1fb0113980 con 0x7f1fb01098b0 2024-03-18T16:07:05.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.809+0000 7f1fb5600640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f1fb0113800 con 0x7f1fb0107d00 2024-03-18T16:07:05.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.809+0000 7f1fb5600640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f1fb0113b00 con 0x7f1fb0108ef0 2024-03-18T16:07:05.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.809+0000 7f1fafe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1fb0107d00 0x7f1fb019dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:05.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.809+0000 7f1fb4c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1fb01098b0 0x7f1fb01a5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:05.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.809+0000 7f1fafe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1fb0107d00 0x7f1fb019dfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:49776/0 (socket says 172.21.15.39:49776) 2024-03-18T16:07:05.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.809+0000 7f1fafe00640 1 -- 172.21.15.39:0/2907393389 learned_addr learned my addr 172.21.15.39:0/2907393389 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:07:05.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.809+0000 7f1faf400640 1 --2- 172.21.15.39:0/2907393389 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1fb0108ef0 0x7f1fb019e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:05.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.809+0000 7f1fafe00640 1 -- 172.21.15.39:0/2907393389 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1fb0108ef0 msgr2=0x7f1fb019e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:05.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.809+0000 7f1fafe00640 1 --2- 172.21.15.39:0/2907393389 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1fb0108ef0 0x7f1fb019e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:05.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.809+0000 7f1fafe00640 1 -- 172.21.15.39:0/2907393389 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1fb01098b0 msgr2=0x7f1fb01a5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:05.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.809+0000 7f1fafe00640 1 --2- 172.21.15.39:0/2907393389 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1fb01098b0 0x7f1fb01a5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:05.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.809+0000 7f1fafe00640 1 -- 172.21.15.39:0/2907393389 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fb01a5bd0 con 0x7f1fb0107d00 2024-03-18T16:07:05.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.810+0000 7f1fafe00640 1 --2- 172.21.15.39:0/2907393389 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1fb0107d00 0x7f1fb019dfb0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f1fa000e9c0 tx=0x7f1fa000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:05.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.810+0000 7f1facc00640 1 -- 172.21.15.39:0/2907393389 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fa000cd80 con 0x7f1fb0107d00 2024-03-18T16:07:05.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.810+0000 7f1facc00640 1 -- 172.21.15.39:0/2907393389 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1fa00035f0 con 0x7f1fb0107d00 2024-03-18T16:07:05.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.810+0000 7f1facc00640 1 -- 172.21.15.39:0/2907393389 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fa0010690 con 0x7f1fb0107d00 2024-03-18T16:07:05.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.810+0000 7f1fb5600640 1 -- 172.21.15.39:0/2907393389 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fb01a5ec0 con 0x7f1fb0107d00 2024-03-18T16:07:05.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.810+0000 7f1fb5600640 1 -- 172.21.15.39:0/2907393389 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fb01a6320 con 0x7f1fb0107d00 2024-03-18T16:07:05.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.811+0000 7f1fb5600640 1 -- 172.21.15.39:0/2907393389 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1fb0068940 con 0x7f1fb0107d00 2024-03-18T16:07:05.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.813+0000 7f1facc00640 1 -- 172.21.15.39:0/2907393389 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f1fa00037a0 con 0x7f1fb0107d00 2024-03-18T16:07:05.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.815+0000 7f1facc00640 1 --2- 172.21.15.39:0/2907393389 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1f940a2880 0x7f1f940a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:05.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.815+0000 7f1facc00640 1 -- 172.21.15.39:0/2907393389 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f1fa00cefd0 con 0x7f1fb0107d00 2024-03-18T16:07:05.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.815+0000 7f1faf400640 1 --2- 172.21.15.39:0/2907393389 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1f940a2880 0x7f1f940a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:05.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.816+0000 7f1faf400640 1 --2- 172.21.15.39:0/2907393389 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1f940a2880 0x7f1f940a4d40 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f1f980095f0 tx=0x7f1f98005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:05.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:05.818+0000 7f1facc00640 1 -- 172.21.15.39:0/2907393389 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1fa009bdd0 con 0x7f1fb0107d00 2024-03-18T16:07:06.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.020+0000 7f1fb5600640 1 -- 172.21.15.39:0/2907393389 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1fb00623e0 con 0x7f1fb0107d00 2024-03-18T16:07:06.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.021+0000 7f1facc00640 1 -- 172.21.15.39:0/2907393389 <== mon.2 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) ==== 93+0+10 (secure 0 0 0) 0x7f1fa009b6d0 con 0x7f1fb0107d00 2024-03-18T16:07:06.023 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2024-03-18T16:07:06.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.023+0000 7f1fb5600640 1 -- 172.21.15.39:0/2907393389 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1f940a2880 msgr2=0x7f1f940a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:06.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.023+0000 7f1fb5600640 1 --2- 172.21.15.39:0/2907393389 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1f940a2880 0x7f1f940a4d40 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f1f980095f0 tx=0x7f1f98005f90 comp rx=0 tx=0).stop 2024-03-18T16:07:06.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.023+0000 7f1fb5600640 1 -- 172.21.15.39:0/2907393389 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1fb0107d00 msgr2=0x7f1fb019dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:06.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.023+0000 7f1fb5600640 1 --2- 172.21.15.39:0/2907393389 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1fb0107d00 0x7f1fb019dfb0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f1fa000e9c0 tx=0x7f1fa000ee80 comp rx=0 tx=0).stop 2024-03-18T16:07:06.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.023+0000 7f1fb5600640 1 -- 172.21.15.39:0/2907393389 shutdown_connections 2024-03-18T16:07:06.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.023+0000 7f1fb5600640 1 --2- 172.21.15.39:0/2907393389 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1f940a2880 0x7f1f940a4d40 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:06.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.023+0000 7f1fb5600640 1 --2- 172.21.15.39:0/2907393389 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1fb01098b0 0x7f1fb01a5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:06.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.023+0000 7f1fb5600640 1 --2- 172.21.15.39:0/2907393389 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1fb0108ef0 0x7f1fb019e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:06.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.023+0000 7f1fb5600640 1 --2- 172.21.15.39:0/2907393389 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1fb0107d00 0x7f1fb019dfb0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:06.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.023+0000 7f1fb5600640 1 -- 172.21.15.39:0/2907393389 >> 172.21.15.39:0/2907393389 conn(0x7f1fb0076cb0 msgr2=0x7f1fb010c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:06.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.024+0000 7f1fb5600640 1 -- 172.21.15.39:0/2907393389 shutdown_connections 2024-03-18T16:07:06.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.024+0000 7f1fb5600640 1 -- 172.21.15.39:0/2907393389 wait complete. 2024-03-18T16:07:06.046 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T16:07:06.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.274+0000 7fdfd3e00640 1 -- 172.21.15.39:0/2240244081 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdfd4109570 msgr2=0x7fdfd410b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:06.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.274+0000 7fdfd3e00640 1 --2- 172.21.15.39:0/2240244081 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdfd4109570 0x7fdfd410b9e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fdfcc009ca0 tx=0x7fdfcc02c0f0 comp rx=0 tx=0).stop 2024-03-18T16:07:06.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.274+0000 7fdfd3e00640 1 -- 172.21.15.39:0/2240244081 shutdown_connections 2024-03-18T16:07:06.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.274+0000 7fdfd3e00640 1 --2- 172.21.15.39:0/2240244081 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdfd4109570 0x7fdfd410b9e0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:06.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.274+0000 7fdfd3e00640 1 --2- 172.21.15.39:0/2240244081 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdfd4074200 0x7fdfd4074680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:06.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.274+0000 7fdfd3e00640 1 --2- 172.21.15.39:0/2240244081 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdfd40757e0 0x7fdfd4073c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:06.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.274+0000 7fdfd3e00640 1 -- 172.21.15.39:0/2240244081 >> 172.21.15.39:0/2240244081 conn(0x7fdfd40fcc50 msgr2=0x7fdfd40ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:06.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.274+0000 7fdfd3e00640 1 -- 172.21.15.39:0/2240244081 shutdown_connections 2024-03-18T16:07:06.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.274+0000 7fdfd3e00640 1 -- 172.21.15.39:0/2240244081 wait complete. 2024-03-18T16:07:06.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.275+0000 7fdfd3e00640 1 Processor -- start 2024-03-18T16:07:06.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.275+0000 7fdfd3e00640 1 -- start start 2024-03-18T16:07:06.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.276+0000 7fdfd3e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdfd4074200 0x7fdfd4199b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:06.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.276+0000 7fdfd3e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdfd40757e0 0x7fdfd419a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:06.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.276+0000 7fdfd3e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdfd4109570 0x7fdfd41a1100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:06.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.276+0000 7fdfd3e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fdfd410f520 con 0x7fdfd40757e0 2024-03-18T16:07:06.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.276+0000 7fdfd3e00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fdfd410f3a0 con 0x7fdfd4074200 2024-03-18T16:07:06.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.276+0000 7fdfd3e00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fdfd410f6a0 con 0x7fdfd4109570 2024-03-18T16:07:06.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.276+0000 7fdfd2a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdfd4074200 0x7fdfd4199b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:06.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.276+0000 7fdfd2a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdfd4074200 0x7fdfd4199b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:49786/0 (socket says 172.21.15.39:49786) 2024-03-18T16:07:06.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.276+0000 7fdfd2a00640 1 -- 172.21.15.39:0/2235630573 learned_addr learned my addr 172.21.15.39:0/2235630573 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:07:06.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.276+0000 7fdfd2000640 1 --2- 172.21.15.39:0/2235630573 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdfd40757e0 0x7fdfd419a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:06.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.276+0000 7fdfd3400640 1 --2- 172.21.15.39:0/2235630573 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdfd4109570 0x7fdfd41a1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:06.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.276+0000 7fdfd2a00640 1 -- 172.21.15.39:0/2235630573 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdfd4109570 msgr2=0x7fdfd41a1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:06.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.276+0000 7fdfd2a00640 1 --2- 172.21.15.39:0/2235630573 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdfd4109570 0x7fdfd41a1100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:06.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.276+0000 7fdfd2a00640 1 -- 172.21.15.39:0/2235630573 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdfd40757e0 msgr2=0x7fdfd419a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:06.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.276+0000 7fdfd2a00640 1 --2- 172.21.15.39:0/2235630573 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdfd40757e0 0x7fdfd419a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:06.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.277+0000 7fdfd2a00640 1 -- 172.21.15.39:0/2235630573 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfd41a1770 con 0x7fdfd4074200 2024-03-18T16:07:06.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.277+0000 7fdfd2a00640 1 --2- 172.21.15.39:0/2235630573 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdfd4074200 0x7fdfd4199b50 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fdfc400d950 tx=0x7fdfc400de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:06.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.277+0000 7fdfcb400640 1 -- 172.21.15.39:0/2235630573 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc4017030 con 0x7fdfd4074200 2024-03-18T16:07:06.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.277+0000 7fdfcb400640 1 -- 172.21.15.39:0/2235630573 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdfc4004020 con 0x7fdfd4074200 2024-03-18T16:07:06.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.277+0000 7fdfcb400640 1 -- 172.21.15.39:0/2235630573 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc40047a0 con 0x7fdfd4074200 2024-03-18T16:07:06.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.277+0000 7fdfd3e00640 1 -- 172.21.15.39:0/2235630573 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfd41a1a60 con 0x7fdfd4074200 2024-03-18T16:07:06.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.277+0000 7fdfd3e00640 1 -- 172.21.15.39:0/2235630573 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfd41a1f60 con 0x7fdfd4074200 2024-03-18T16:07:06.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.278+0000 7fdfd3e00640 1 -- 172.21.15.39:0/2235630573 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdfd41082f0 con 0x7fdfd4074200 2024-03-18T16:07:06.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.280+0000 7fdfcb400640 1 -- 172.21.15.39:0/2235630573 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fdfc40041d0 con 0x7fdfd4074200 2024-03-18T16:07:06.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.281+0000 7fdfcb400640 1 --2- 172.21.15.39:0/2235630573 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fdfb80a2880 0x7fdfb80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:06.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.282+0000 7fdfcb400640 1 -- 172.21.15.39:0/2235630573 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fdfc40ce5a0 con 0x7fdfd4074200 2024-03-18T16:07:06.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.282+0000 7fdfd2000640 1 --2- 172.21.15.39:0/2235630573 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fdfb80a2880 0x7fdfb80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:06.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.282+0000 7fdfd2000640 1 --2- 172.21.15.39:0/2235630573 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fdfb80a2880 0x7fdfb80a4d40 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fdfd419b060 tx=0x7fdfc0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:06.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.286+0000 7fdfcb400640 1 -- 172.21.15.39:0/2235630573 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fdfc409b3a0 con 0x7fdfd4074200 2024-03-18T16:07:06.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:06 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/1092152846' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:07:06.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:06 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/2907393389' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T16:07:06.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:06 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/1092152846' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:07:06.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:06 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/2907393389' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T16:07:06.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:06 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/1092152846' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:07:06.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:06 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/2907393389' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T16:07:06.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.490+0000 7fdfd3e00640 1 -- 172.21.15.39:0/2235630573 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fdfd40623e0 con 0x7fdfd4074200 2024-03-18T16:07:06.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.491+0000 7fdfcb400640 1 -- 172.21.15.39:0/2235630573 <== mon.2 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) ==== 104+0+11 (secure 0 0 0) 0x7fdfc409aca0 con 0x7fdfd4074200 2024-03-18T16:07:06.493 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 32 2024-03-18T16:07:06.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.494+0000 7fdfd3e00640 1 -- 172.21.15.39:0/2235630573 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fdfb80a2880 msgr2=0x7fdfb80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:06.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.494+0000 7fdfd3e00640 1 --2- 172.21.15.39:0/2235630573 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fdfb80a2880 0x7fdfb80a4d40 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fdfd419b060 tx=0x7fdfc0005f90 comp rx=0 tx=0).stop 2024-03-18T16:07:06.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.494+0000 7fdfd3e00640 1 -- 172.21.15.39:0/2235630573 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdfd4074200 msgr2=0x7fdfd4199b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:06.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.494+0000 7fdfd3e00640 1 --2- 172.21.15.39:0/2235630573 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdfd4074200 0x7fdfd4199b50 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fdfc400d950 tx=0x7fdfc400de10 comp rx=0 tx=0).stop 2024-03-18T16:07:06.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.494+0000 7fdfd3e00640 1 -- 172.21.15.39:0/2235630573 shutdown_connections 2024-03-18T16:07:06.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.494+0000 7fdfd3e00640 1 --2- 172.21.15.39:0/2235630573 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fdfb80a2880 0x7fdfb80a4d40 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:06.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.494+0000 7fdfd3e00640 1 --2- 172.21.15.39:0/2235630573 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdfd4109570 0x7fdfd41a1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:06.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.494+0000 7fdfd3e00640 1 --2- 172.21.15.39:0/2235630573 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdfd40757e0 0x7fdfd419a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:06.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.494+0000 7fdfd3e00640 1 --2- 172.21.15.39:0/2235630573 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdfd4074200 0x7fdfd4199b50 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:06.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.494+0000 7fdfd3e00640 1 -- 172.21.15.39:0/2235630573 >> 172.21.15.39:0/2235630573 conn(0x7fdfd40fcc50 msgr2=0x7fdfd41054c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:06.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.494+0000 7fdfd3e00640 1 -- 172.21.15.39:0/2235630573 shutdown_connections 2024-03-18T16:07:06.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.494+0000 7fdfd3e00640 1 -- 172.21.15.39:0/2235630573 wait complete. 2024-03-18T16:07:06.516 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T16:07:06.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.731+0000 7f2dbd800640 1 -- 172.21.15.39:0/2203117246 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2db8100a40 msgr2=0x7f2db8100ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:06.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.731+0000 7f2dbd800640 1 --2- 172.21.15.39:0/2203117246 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2db8100a40 0x7f2db8100ea0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f2da4009900 tx=0x7f2da402bcc0 comp rx=0 tx=0).stop 2024-03-18T16:07:06.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.731+0000 7f2dbd800640 1 -- 172.21.15.39:0/2203117246 shutdown_connections 2024-03-18T16:07:06.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.731+0000 7f2dbd800640 1 --2- 172.21.15.39:0/2203117246 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2db8100a40 0x7f2db8100ea0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:06.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.731+0000 7f2dbd800640 1 --2- 172.21.15.39:0/2203117246 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2db81013e0 0x7f2db8101860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:06.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.731+0000 7f2dbd800640 1 --2- 172.21.15.39:0/2203117246 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2db81000e0 0x7f2db8100500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:06.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.731+0000 7f2dbd800640 1 -- 172.21.15.39:0/2203117246 >> 172.21.15.39:0/2203117246 conn(0x7f2db80fbc80 msgr2=0x7f2db80fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:06.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.731+0000 7f2dbd800640 1 -- 172.21.15.39:0/2203117246 shutdown_connections 2024-03-18T16:07:06.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.731+0000 7f2dbd800640 1 -- 172.21.15.39:0/2203117246 wait complete. 2024-03-18T16:07:06.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.732+0000 7f2dbd800640 1 Processor -- start 2024-03-18T16:07:06.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.732+0000 7f2dbd800640 1 -- start start 2024-03-18T16:07:06.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.732+0000 7f2dbd800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2db81000e0 0x7f2db8199d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:06.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.732+0000 7f2dbd800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2db8100a40 0x7f2db819a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:06.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.732+0000 7f2dbd800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2db81013e0 0x7f2db81a1310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:06.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.732+0000 7f2dbd800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f2db810f6c0 con 0x7f2db8100a40 2024-03-18T16:07:06.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.732+0000 7f2dbd800640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f2db810f540 con 0x7f2db81013e0 2024-03-18T16:07:06.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.732+0000 7f2dbd800640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f2db810f840 con 0x7f2db81000e0 2024-03-18T16:07:06.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.733+0000 7f2db6000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2db8100a40 0x7f2db819a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:06.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.733+0000 7f2db6000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2db8100a40 0x7f2db819a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58758/0 (socket says 172.21.15.39:58758) 2024-03-18T16:07:06.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.733+0000 7f2db6000640 1 -- 172.21.15.39:0/4120569663 learned_addr learned my addr 172.21.15.39:0/4120569663 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:07:06.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.733+0000 7f2db7400640 1 --2- 172.21.15.39:0/4120569663 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2db81013e0 0x7f2db81a1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:06.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.733+0000 7f2db6a00640 1 --2- 172.21.15.39:0/4120569663 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2db81000e0 0x7f2db8199d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:06.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.733+0000 7f2db6000640 1 -- 172.21.15.39:0/4120569663 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2db81000e0 msgr2=0x7f2db8199d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:06.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.733+0000 7f2db6000640 1 --2- 172.21.15.39:0/4120569663 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2db81000e0 0x7f2db8199d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:06.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.733+0000 7f2db6000640 1 -- 172.21.15.39:0/4120569663 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2db81013e0 msgr2=0x7f2db81a1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:06.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.733+0000 7f2db6000640 1 --2- 172.21.15.39:0/4120569663 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2db81013e0 0x7f2db81a1310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:06.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.733+0000 7f2db6000640 1 -- 172.21.15.39:0/4120569663 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2db81a1980 con 0x7f2db8100a40 2024-03-18T16:07:06.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.733+0000 7f2db7400640 1 --2- 172.21.15.39:0/4120569663 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2db81013e0 0x7f2db81a1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:07:06.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.733+0000 7f2db6000640 1 --2- 172.21.15.39:0/4120569663 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2db8100a40 0x7f2db819a2a0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f2da4000c00 tx=0x7f2da402c6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:06.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.734+0000 7f2daf400640 1 -- 172.21.15.39:0/4120569663 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2da4043020 con 0x7f2db8100a40 2024-03-18T16:07:06.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.734+0000 7f2dbd800640 1 -- 172.21.15.39:0/4120569663 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2db81a1c10 con 0x7f2db8100a40 2024-03-18T16:07:06.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.734+0000 7f2daf400640 1 -- 172.21.15.39:0/4120569663 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2da40054e0 con 0x7f2db8100a40 2024-03-18T16:07:06.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.734+0000 7f2daf400640 1 -- 172.21.15.39:0/4120569663 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2da4003b20 con 0x7f2db8100a40 2024-03-18T16:07:06.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.734+0000 7f2dbd800640 1 -- 172.21.15.39:0/4120569663 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2db81a2160 con 0x7f2db8100a40 2024-03-18T16:07:06.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.735+0000 7f2dbd800640 1 -- 172.21.15.39:0/4120569663 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2db810b020 con 0x7f2db8100a40 2024-03-18T16:07:06.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.739+0000 7f2daf400640 1 -- 172.21.15.39:0/4120569663 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f2da4046020 con 0x7f2db8100a40 2024-03-18T16:07:06.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.740+0000 7f2daf400640 1 --2- 172.21.15.39:0/4120569663 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f2d9c0a2800 0x7f2d9c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:06.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.740+0000 7f2daf400640 1 -- 172.21.15.39:0/4120569663 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f2da40f1070 con 0x7f2db8100a40 2024-03-18T16:07:06.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.740+0000 7f2db6a00640 1 --2- 172.21.15.39:0/4120569663 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f2d9c0a2800 0x7f2d9c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:06.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.741+0000 7f2db6a00640 1 --2- 172.21.15.39:0/4120569663 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f2d9c0a2800 0x7f2d9c0a4cc0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f2da8009bd0 tx=0x7f2da8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:06.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.743+0000 7f2daf400640 1 -- 172.21.15.39:0/4120569663 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2da404d020 con 0x7f2db8100a40 2024-03-18T16:07:06.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.946+0000 7f2dbd800640 1 -- 172.21.15.39:0/4120569663 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f2db802d340 con 0x7f2db8100a40 2024-03-18T16:07:06.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.947+0000 7f2daf400640 1 -- 172.21.15.39:0/4120569663 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) ==== 100+0+11 (secure 0 0 0) 0x7f2da40bde70 con 0x7f2db8100a40 2024-03-18T16:07:06.950 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 64 2024-03-18T16:07:06.952 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.950+0000 7f2dbd800640 1 -- 172.21.15.39:0/4120569663 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f2d9c0a2800 msgr2=0x7f2d9c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:06.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.950+0000 7f2dbd800640 1 --2- 172.21.15.39:0/4120569663 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f2d9c0a2800 0x7f2d9c0a4cc0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f2da8009bd0 tx=0x7f2da8005f90 comp rx=0 tx=0).stop 2024-03-18T16:07:06.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.950+0000 7f2dbd800640 1 -- 172.21.15.39:0/4120569663 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2db8100a40 msgr2=0x7f2db819a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:06.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.950+0000 7f2dbd800640 1 --2- 172.21.15.39:0/4120569663 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2db8100a40 0x7f2db819a2a0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f2da4000c00 tx=0x7f2da402c6e0 comp rx=0 tx=0).stop 2024-03-18T16:07:06.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.950+0000 7f2dbd800640 1 -- 172.21.15.39:0/4120569663 shutdown_connections 2024-03-18T16:07:06.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.950+0000 7f2dbd800640 1 --2- 172.21.15.39:0/4120569663 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2db81013e0 0x7f2db81a1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:06.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.950+0000 7f2dbd800640 1 --2- 172.21.15.39:0/4120569663 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f2d9c0a2800 0x7f2d9c0a4cc0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:06.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.950+0000 7f2dbd800640 1 --2- 172.21.15.39:0/4120569663 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2db8100a40 0x7f2db819a2a0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:06.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.950+0000 7f2dbd800640 1 --2- 172.21.15.39:0/4120569663 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2db81000e0 0x7f2db8199d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:06.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.950+0000 7f2dbd800640 1 -- 172.21.15.39:0/4120569663 >> 172.21.15.39:0/4120569663 conn(0x7f2db80fbc80 msgr2=0x7f2db80fd3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:06.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.950+0000 7f2dbd800640 1 -- 172.21.15.39:0/4120569663 shutdown_connections 2024-03-18T16:07:06.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:06.950+0000 7f2dbd800640 1 -- 172.21.15.39:0/4120569663 wait complete. 2024-03-18T16:07:06.972 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T16:07:07.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.198+0000 7fc83b200640 1 -- 172.21.15.39:0/2903679400 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc83c108ef0 msgr2=0x7fc83c109370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:07.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.198+0000 7fc83b200640 1 --2- 172.21.15.39:0/2903679400 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc83c108ef0 0x7fc83c109370 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc824009900 tx=0x7fc82402bc50 comp rx=0 tx=0).stop 2024-03-18T16:07:07.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.198+0000 7fc83b200640 1 -- 172.21.15.39:0/2903679400 shutdown_connections 2024-03-18T16:07:07.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.198+0000 7fc83b200640 1 --2- 172.21.15.39:0/2903679400 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc83c108ef0 0x7fc83c109370 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:07.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.198+0000 7fc83b200640 1 --2- 172.21.15.39:0/2903679400 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc83c1098b0 0x7fc83c110140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:07.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.198+0000 7fc83b200640 1 --2- 172.21.15.39:0/2903679400 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc83c107d00 0x7fc83c108120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:07.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.198+0000 7fc83b200640 1 -- 172.21.15.39:0/2903679400 >> 172.21.15.39:0/2903679400 conn(0x7fc83c076cb0 msgr2=0x7fc83c0790d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:07.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.198+0000 7fc83b200640 1 -- 172.21.15.39:0/2903679400 shutdown_connections 2024-03-18T16:07:07.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.198+0000 7fc83b200640 1 -- 172.21.15.39:0/2903679400 wait complete. 2024-03-18T16:07:07.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.199+0000 7fc83b200640 1 Processor -- start 2024-03-18T16:07:07.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.200+0000 7fc83b200640 1 -- start start 2024-03-18T16:07:07.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.200+0000 7fc83b200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc83c107d00 0x7fc83c19dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:07.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.200+0000 7fc83b200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc83c108ef0 0x7fc83c19e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:07.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.200+0000 7fc83b200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc83c1098b0 0x7fc83c1a5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:07.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.200+0000 7fc83b200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fc83c1139b0 con 0x7fc83c1098b0 2024-03-18T16:07:07.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.200+0000 7fc83b200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fc83c113830 con 0x7fc83c107d00 2024-03-18T16:07:07.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.200+0000 7fc83b200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fc83c113b30 con 0x7fc83c108ef0 2024-03-18T16:07:07.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.200+0000 7fc839e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc83c107d00 0x7fc83c19dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:07.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.200+0000 7fc839e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc83c107d00 0x7fc83c19dfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:49820/0 (socket says 172.21.15.39:49820) 2024-03-18T16:07:07.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.200+0000 7fc839e00640 1 -- 172.21.15.39:0/2264411632 learned_addr learned my addr 172.21.15.39:0/2264411632 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:07:07.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.200+0000 7fc839400640 1 --2- 172.21.15.39:0/2264411632 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc83c108ef0 0x7fc83c19e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:07.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.200+0000 7fc83a800640 1 --2- 172.21.15.39:0/2264411632 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc83c1098b0 0x7fc83c1a5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:07.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.201+0000 7fc839e00640 1 -- 172.21.15.39:0/2264411632 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc83c108ef0 msgr2=0x7fc83c19e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:07.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.201+0000 7fc839e00640 1 --2- 172.21.15.39:0/2264411632 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc83c108ef0 0x7fc83c19e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:07.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.201+0000 7fc839e00640 1 -- 172.21.15.39:0/2264411632 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc83c1098b0 msgr2=0x7fc83c1a5580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:07.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.201+0000 7fc839e00640 1 --2- 172.21.15.39:0/2264411632 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc83c1098b0 0x7fc83c1a5580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:07.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.201+0000 7fc839e00640 1 -- 172.21.15.39:0/2264411632 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc83c1a5bf0 con 0x7fc83c107d00 2024-03-18T16:07:07.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.201+0000 7fc839e00640 1 --2- 172.21.15.39:0/2264411632 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc83c107d00 0x7fc83c19dfd0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc82c00ebc0 tx=0x7fc82c00c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:07.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.201+0000 7fc832a00640 1 -- 172.21.15.39:0/2264411632 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc82c003970 con 0x7fc83c107d00 2024-03-18T16:07:07.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.201+0000 7fc832a00640 1 -- 172.21.15.39:0/2264411632 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc82c004030 con 0x7fc83c107d00 2024-03-18T16:07:07.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.201+0000 7fc832a00640 1 -- 172.21.15.39:0/2264411632 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc82c0197b0 con 0x7fc83c107d00 2024-03-18T16:07:07.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.201+0000 7fc83b200640 1 -- 172.21.15.39:0/2264411632 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc83c1a5ee0 con 0x7fc83c107d00 2024-03-18T16:07:07.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.202+0000 7fc83b200640 1 -- 172.21.15.39:0/2264411632 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc83c1a6350 con 0x7fc83c107d00 2024-03-18T16:07:07.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.203+0000 7fc83b200640 1 -- 172.21.15.39:0/2264411632 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc83c10f020 con 0x7fc83c107d00 2024-03-18T16:07:07.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.204+0000 7fc832a00640 1 -- 172.21.15.39:0/2264411632 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7fc82c0041e0 con 0x7fc83c107d00 2024-03-18T16:07:07.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.206+0000 7fc832a00640 1 --2- 172.21.15.39:0/2264411632 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fc8200a2880 0x7fc8200a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:07.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.206+0000 7fc832a00640 1 -- 172.21.15.39:0/2264411632 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fc82c0cf520 con 0x7fc83c107d00 2024-03-18T16:07:07.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.206+0000 7fc839400640 1 --2- 172.21.15.39:0/2264411632 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fc8200a2880 0x7fc8200a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:07.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.207+0000 7fc839400640 1 --2- 172.21.15.39:0/2264411632 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fc8200a2880 0x7fc8200a4d40 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fc82402c260 tx=0x7fc824000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:07.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.210+0000 7fc832a00640 1 -- 172.21.15.39:0/2264411632 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc82c09c320 con 0x7fc83c107d00 2024-03-18T16:07:07.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:07 smithi039 ceph-mon[31721]: pgmap v329: 129 pgs: 129 active+clean; 581 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-18T16:07:07.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:07 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/2235630573' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T16:07:07.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:07 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/4120569663' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T16:07:07.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.415+0000 7fc83b200640 1 -- 172.21.15.39:0/2264411632 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fc83c02d1b0 con 0x7fc83c107d00 2024-03-18T16:07:07.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.416+0000 7fc832a00640 1 -- 172.21.15.39:0/2264411632 <== mon.2 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) ==== 103+0+11 (secure 0 0 0) 0x7fc82c086020 con 0x7fc83c107d00 2024-03-18T16:07:07.419 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 32 2024-03-18T16:07:07.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.419+0000 7fc83b200640 1 -- 172.21.15.39:0/2264411632 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fc8200a2880 msgr2=0x7fc8200a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:07.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.419+0000 7fc83b200640 1 --2- 172.21.15.39:0/2264411632 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fc8200a2880 0x7fc8200a4d40 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fc82402c260 tx=0x7fc824000bc0 comp rx=0 tx=0).stop 2024-03-18T16:07:07.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.419+0000 7fc83b200640 1 -- 172.21.15.39:0/2264411632 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc83c107d00 msgr2=0x7fc83c19dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:07.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.419+0000 7fc83b200640 1 --2- 172.21.15.39:0/2264411632 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc83c107d00 0x7fc83c19dfd0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc82c00ebc0 tx=0x7fc82c00c2d0 comp rx=0 tx=0).stop 2024-03-18T16:07:07.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.419+0000 7fc83b200640 1 -- 172.21.15.39:0/2264411632 shutdown_connections 2024-03-18T16:07:07.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.419+0000 7fc83b200640 1 --2- 172.21.15.39:0/2264411632 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fc8200a2880 0x7fc8200a4d40 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:07.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.419+0000 7fc83b200640 1 --2- 172.21.15.39:0/2264411632 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc83c1098b0 0x7fc83c1a5580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:07.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.419+0000 7fc83b200640 1 --2- 172.21.15.39:0/2264411632 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc83c108ef0 0x7fc83c19e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:07.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.419+0000 7fc83b200640 1 --2- 172.21.15.39:0/2264411632 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc83c107d00 0x7fc83c19dfd0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:07.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.419+0000 7fc83b200640 1 -- 172.21.15.39:0/2264411632 >> 172.21.15.39:0/2264411632 conn(0x7fc83c076cb0 msgr2=0x7fc83c10c1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:07.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.419+0000 7fc83b200640 1 -- 172.21.15.39:0/2264411632 shutdown_connections 2024-03-18T16:07:07.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.419+0000 7fc83b200640 1 -- 172.21.15.39:0/2264411632 wait complete. 2024-03-18T16:07:07.443 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-18T16:07:07.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:07 smithi196 ceph-mon[36307]: pgmap v329: 129 pgs: 129 active+clean; 581 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-18T16:07:07.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:07 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/2235630573' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T16:07:07.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:07 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/4120569663' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T16:07:07.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:07 smithi194 ceph-mon[36200]: pgmap v329: 129 pgs: 129 active+clean; 581 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-18T16:07:07.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:07 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/2235630573' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T16:07:07.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:07 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/4120569663' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T16:07:07.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.667+0000 7f1a79c00640 1 -- 172.21.15.39:0/2039358680 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1a74107d00 msgr2=0x7f1a74108120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:07.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.667+0000 7f1a79c00640 1 --2- 172.21.15.39:0/2039358680 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1a74107d00 0x7f1a74108120 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1a64009900 tx=0x7f1a6402bc30 comp rx=0 tx=0).stop 2024-03-18T16:07:07.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.667+0000 7f1a79c00640 1 -- 172.21.15.39:0/2039358680 shutdown_connections 2024-03-18T16:07:07.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.667+0000 7f1a79c00640 1 --2- 172.21.15.39:0/2039358680 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1a741098b0 0x7f1a74110140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:07.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.667+0000 7f1a79c00640 1 --2- 172.21.15.39:0/2039358680 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1a74108ef0 0x7f1a74109370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:07.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.667+0000 7f1a79c00640 1 --2- 172.21.15.39:0/2039358680 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1a74107d00 0x7f1a74108120 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:07.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.667+0000 7f1a79c00640 1 -- 172.21.15.39:0/2039358680 >> 172.21.15.39:0/2039358680 conn(0x7f1a74076cb0 msgr2=0x7f1a740790d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:07.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.667+0000 7f1a79c00640 1 -- 172.21.15.39:0/2039358680 shutdown_connections 2024-03-18T16:07:07.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.667+0000 7f1a79c00640 1 -- 172.21.15.39:0/2039358680 wait complete. 2024-03-18T16:07:07.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.668+0000 7f1a79c00640 1 Processor -- start 2024-03-18T16:07:07.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.668+0000 7f1a79c00640 1 -- start start 2024-03-18T16:07:07.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.669+0000 7f1a79c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1a74107d00 0x7f1a7419e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:07.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.669+0000 7f1a79c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1a74108ef0 0x7f1a7419e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:07.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.669+0000 7f1a79c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1a741098b0 0x7f1a741a5580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:07.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.669+0000 7f1a79c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f1a741139a0 con 0x7f1a741098b0 2024-03-18T16:07:07.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.669+0000 7f1a79c00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f1a74113820 con 0x7f1a74107d00 2024-03-18T16:07:07.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.669+0000 7f1a79c00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f1a74113b20 con 0x7f1a74108ef0 2024-03-18T16:07:07.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.669+0000 7f1a73e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1a74107d00 0x7f1a7419e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:07.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.669+0000 7f1a73e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1a74107d00 0x7f1a7419e070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:49844/0 (socket says 172.21.15.39:49844) 2024-03-18T16:07:07.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.669+0000 7f1a73e00640 1 -- 172.21.15.39:0/345700039 learned_addr learned my addr 172.21.15.39:0/345700039 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:07:07.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.669+0000 7f1a79200640 1 --2- 172.21.15.39:0/345700039 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1a741098b0 0x7f1a741a5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:07.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.669+0000 7f1a6b400640 1 --2- 172.21.15.39:0/345700039 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1a74108ef0 0x7f1a7419e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:07.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.669+0000 7f1a73e00640 1 -- 172.21.15.39:0/345700039 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1a74108ef0 msgr2=0x7f1a7419e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:07.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.669+0000 7f1a73e00640 1 --2- 172.21.15.39:0/345700039 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1a74108ef0 0x7f1a7419e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:07.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.669+0000 7f1a73e00640 1 -- 172.21.15.39:0/345700039 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1a741098b0 msgr2=0x7f1a741a5580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:07.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.669+0000 7f1a73e00640 1 --2- 172.21.15.39:0/345700039 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1a741098b0 0x7f1a741a5580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:07.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.669+0000 7f1a73e00640 1 -- 172.21.15.39:0/345700039 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a741a5bf0 con 0x7f1a74107d00 2024-03-18T16:07:07.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.670+0000 7f1a73e00640 1 --2- 172.21.15.39:0/345700039 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1a74107d00 0x7f1a7419e070 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f1a64000c00 tx=0x7f1a6402c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:07.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.670+0000 7f1a71600640 1 -- 172.21.15.39:0/345700039 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a64043020 con 0x7f1a74107d00 2024-03-18T16:07:07.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.670+0000 7f1a71600640 1 -- 172.21.15.39:0/345700039 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1a640054e0 con 0x7f1a74107d00 2024-03-18T16:07:07.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.670+0000 7f1a79c00640 1 -- 172.21.15.39:0/345700039 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a741a5e80 con 0x7f1a74107d00 2024-03-18T16:07:07.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.670+0000 7f1a71600640 1 -- 172.21.15.39:0/345700039 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a64003c40 con 0x7f1a74107d00 2024-03-18T16:07:07.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.670+0000 7f1a79c00640 1 -- 172.21.15.39:0/345700039 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a741a62e0 con 0x7f1a74107d00 2024-03-18T16:07:07.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.672+0000 7f1a79c00640 1 -- 172.21.15.39:0/345700039 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a74068970 con 0x7f1a74107d00 2024-03-18T16:07:07.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.673+0000 7f1a71600640 1 -- 172.21.15.39:0/345700039 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f1a64046020 con 0x7f1a74107d00 2024-03-18T16:07:07.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.674+0000 7f1a71600640 1 --2- 172.21.15.39:0/345700039 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1a580a2800 0x7f1a580a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:07.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.675+0000 7f1a71600640 1 -- 172.21.15.39:0/345700039 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f1a640f0cc0 con 0x7f1a74107d00 2024-03-18T16:07:07.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.675+0000 7f1a6b400640 1 --2- 172.21.15.39:0/345700039 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1a580a2800 0x7f1a580a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:07.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.675+0000 7f1a6b400640 1 --2- 172.21.15.39:0/345700039 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1a580a2800 0x7f1a580a4cc0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f1a7419f4e0 tx=0x7f1a5c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:07.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.678+0000 7f1a71600640 1 -- 172.21.15.39:0/345700039 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1a74068970 con 0x7f1a74107d00 2024-03-18T16:07:07.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.898+0000 7f1a79c00640 1 -- 172.21.15.39:0/345700039 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f1a740623e0 con 0x7f1a74107d00 2024-03-18T16:07:07.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.899+0000 7f1a71600640 1 -- 172.21.15.39:0/345700039 <== mon.2 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v89) ==== 81+0+592 (secure 0 0 0) 0x7f1a640f7080 con 0x7f1a74107d00 2024-03-18T16:07:07.902 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/3482086140 2024-03-19T15:58:45.340294+0000 2024-03-18T16:07:07.902 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/3844306284 2024-03-19T15:58:45.340294+0000 2024-03-18T16:07:07.902 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:6801/1810238546 2024-03-19T15:58:45.340294+0000 2024-03-18T16:07:07.902 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:6800/1810238546 2024-03-19T15:58:45.340294+0000 2024-03-18T16:07:07.902 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/2528397488 2024-03-19T15:58:10.264798+0000 2024-03-18T16:07:07.902 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/2240981254 2024-03-19T15:58:10.264798+0000 2024-03-18T16:07:07.902 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:6800/1217503189 2024-03-19T15:58:10.264798+0000 2024-03-18T16:07:07.902 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/3754876370 2024-03-19T15:58:10.264798+0000 2024-03-18T16:07:07.902 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:6801/1217503189 2024-03-19T15:58:10.264798+0000 2024-03-18T16:07:07.902 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/1874637628 2024-03-19T15:58:45.340294+0000 2024-03-18T16:07:07.903 INFO:teuthology.orchestra.run.smithi039.stderr:listed 10 entries 2024-03-18T16:07:07.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.902+0000 7f1a79c00640 1 -- 172.21.15.39:0/345700039 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1a580a2800 msgr2=0x7f1a580a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:07.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.902+0000 7f1a79c00640 1 --2- 172.21.15.39:0/345700039 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1a580a2800 0x7f1a580a4cc0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f1a7419f4e0 tx=0x7f1a5c005f90 comp rx=0 tx=0).stop 2024-03-18T16:07:07.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.902+0000 7f1a79c00640 1 -- 172.21.15.39:0/345700039 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1a74107d00 msgr2=0x7f1a7419e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:07.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.902+0000 7f1a79c00640 1 --2- 172.21.15.39:0/345700039 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1a74107d00 0x7f1a7419e070 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f1a64000c00 tx=0x7f1a6402c650 comp rx=0 tx=0).stop 2024-03-18T16:07:07.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.902+0000 7f1a79c00640 1 -- 172.21.15.39:0/345700039 shutdown_connections 2024-03-18T16:07:07.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.902+0000 7f1a79c00640 1 --2- 172.21.15.39:0/345700039 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1a580a2800 0x7f1a580a4cc0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:07.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.902+0000 7f1a79c00640 1 --2- 172.21.15.39:0/345700039 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1a741098b0 0x7f1a741a5580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:07.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.902+0000 7f1a79c00640 1 --2- 172.21.15.39:0/345700039 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1a74108ef0 0x7f1a7419e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:07.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.902+0000 7f1a79c00640 1 --2- 172.21.15.39:0/345700039 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1a74107d00 0x7f1a7419e070 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:07.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.902+0000 7f1a79c00640 1 -- 172.21.15.39:0/345700039 >> 172.21.15.39:0/345700039 conn(0x7f1a74076cb0 msgr2=0x7f1a7410c1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:07.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.902+0000 7f1a79c00640 1 -- 172.21.15.39:0/345700039 shutdown_connections 2024-03-18T16:07:07.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:07.902+0000 7f1a79c00640 1 -- 172.21.15.39:0/345700039 wait complete. 2024-03-18T16:07:07.922 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi039.front.sepia.ceph.com... 2024-03-18T16:07:07.922 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T16:07:07.922 DEBUG:teuthology.orchestra.run.smithi039:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-03-18T16:07:08.007 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi039.stderr:ceph-fuse[73442]: fuse finished with error 0 and tester_r 0 2024-03-18T16:07:08.015 INFO:teuthology.orchestra.run:waiting for 300 2024-03-18T16:07:08.230 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:08 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/2264411632' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T16:07:08.230 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:08 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/345700039' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-18T16:07:08.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:08 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/2264411632' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T16:07:08.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:08 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/345700039' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-18T16:07:08.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:08 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/2264411632' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T16:07:08.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:08 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/345700039' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-18T16:07:09.391 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:09 smithi194 ceph-mon[36200]: pgmap v330: 129 pgs: 129 active+clean; 581 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-18T16:07:09.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:09 smithi196 ceph-mon[36307]: pgmap v330: 129 pgs: 129 active+clean; 581 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-18T16:07:09.589 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:09 smithi039 ceph-mon[31721]: pgmap v330: 129 pgs: 129 active+clean; 581 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-03-18T16:07:11.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:11 smithi196 ceph-mon[36307]: pgmap v331: 129 pgs: 129 active+clean; 581 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-18T16:07:11.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:11 smithi194 ceph-mon[36200]: pgmap v331: 129 pgs: 129 active+clean; 581 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-18T16:07:11.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:11 smithi039 ceph-mon[31721]: pgmap v331: 129 pgs: 129 active+clean; 581 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-18T16:07:13.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:13 smithi196 ceph-mon[36307]: pgmap v332: 129 pgs: 129 active+clean; 581 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-18T16:07:13.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:13 smithi194 ceph-mon[36200]: pgmap v332: 129 pgs: 129 active+clean; 581 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-18T16:07:13.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:13 smithi039 ceph-mon[31721]: pgmap v332: 129 pgs: 129 active+clean; 581 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-18T16:07:14.116 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi039.front.sepia.ceph.com 2024-03-18T16:07:14.117 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T16:07:14.117 DEBUG:teuthology.orchestra.run.smithi039:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-03-18T16:07:14.209 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T16:07:14.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.441+0000 7f30b3200640 1 -- 172.21.15.39:0/3018513380 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f30b4103810 msgr2=0x7f30b4103c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:14.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.441+0000 7f30b3200640 1 --2- 172.21.15.39:0/3018513380 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f30b4103810 0x7f30b4103c30 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f30a4009900 tx=0x7f30a402bc50 comp rx=0 tx=0).stop 2024-03-18T16:07:14.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.441+0000 7f30b3200640 1 -- 172.21.15.39:0/3018513380 shutdown_connections 2024-03-18T16:07:14.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.441+0000 7f30b3200640 1 --2- 172.21.15.39:0/3018513380 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f30b41053c0 0x7f30b410bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:14.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.441+0000 7f30b3200640 1 --2- 172.21.15.39:0/3018513380 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f30b4104a00 0x7f30b4104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:14.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.441+0000 7f30b3200640 1 --2- 172.21.15.39:0/3018513380 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f30b4103810 0x7f30b4103c30 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:14.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.441+0000 7f30b3200640 1 -- 172.21.15.39:0/3018513380 >> 172.21.15.39:0/3018513380 conn(0x7f30b40fed80 msgr2=0x7f30b41011c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:14.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.441+0000 7f30b3200640 1 -- 172.21.15.39:0/3018513380 shutdown_connections 2024-03-18T16:07:14.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.441+0000 7f30b3200640 1 -- 172.21.15.39:0/3018513380 wait complete. 2024-03-18T16:07:14.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.442+0000 7f30b3200640 1 Processor -- start 2024-03-18T16:07:14.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.443+0000 7f30b3200640 1 -- start start 2024-03-18T16:07:14.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.443+0000 7f30b3200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f30b4103810 0x7f30b4199b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:14.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.443+0000 7f30b3200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f30b4104a00 0x7f30b419a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:14.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.443+0000 7f30b3200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f30b41053c0 0x7f30b41a10c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:14.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.443+0000 7f30b3200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f30b410f4d0 con 0x7f30b41053c0 2024-03-18T16:07:14.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.443+0000 7f30b3200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f30b410f350 con 0x7f30b4104a00 2024-03-18T16:07:14.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.443+0000 7f30b3200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f30b410f650 con 0x7f30b4103810 2024-03-18T16:07:14.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.443+0000 7f30b1e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f30b4103810 0x7f30b4199b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:14.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.443+0000 7f30b2800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f30b41053c0 0x7f30b41a10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:14.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.443+0000 7f30a9400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f30b4104a00 0x7f30b419a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:14.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.443+0000 7f30b1e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f30b4103810 0x7f30b4199b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:38512/0 (socket says 172.21.15.39:38512) 2024-03-18T16:07:14.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.443+0000 7f30b2800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f30b41053c0 0x7f30b41a10c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:46966/0 (socket says 172.21.15.39:46966) 2024-03-18T16:07:14.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.443+0000 7f30b1e00640 1 -- 172.21.15.39:0/2352628561 learned_addr learned my addr 172.21.15.39:0/2352628561 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:07:14.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.444+0000 7f30a9400640 1 -- 172.21.15.39:0/2352628561 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f30b4103810 msgr2=0x7f30b4199b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:14.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.444+0000 7f30a9400640 1 --2- 172.21.15.39:0/2352628561 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f30b4103810 0x7f30b4199b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:14.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.444+0000 7f30a9400640 1 -- 172.21.15.39:0/2352628561 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f30b41053c0 msgr2=0x7f30b41a10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:14.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.444+0000 7f30a9400640 1 --2- 172.21.15.39:0/2352628561 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f30b41053c0 0x7f30b41a10c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:14.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.444+0000 7f30a9400640 1 -- 172.21.15.39:0/2352628561 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30b41a1730 con 0x7f30b4104a00 2024-03-18T16:07:14.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.444+0000 7f30b1e00640 1 --2- 172.21.15.39:0/2352628561 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f30b4103810 0x7f30b4199b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:07:14.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.444+0000 7f30b2800640 1 --2- 172.21.15.39:0/2352628561 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f30b41053c0 0x7f30b41a10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:07:14.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.444+0000 7f30a9400640 1 --2- 172.21.15.39:0/2352628561 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f30b4104a00 0x7f30b419a050 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f309c00e9f0 tx=0x7f309c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:14.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.445+0000 7f30ab400640 1 -- 172.21.15.39:0/2352628561 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f309c00ccc0 con 0x7f30b4104a00 2024-03-18T16:07:14.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.445+0000 7f30ab400640 1 -- 172.21.15.39:0/2352628561 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f309c005020 con 0x7f30b4104a00 2024-03-18T16:07:14.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.445+0000 7f30b3200640 1 -- 172.21.15.39:0/2352628561 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30b41a1a20 con 0x7f30b4104a00 2024-03-18T16:07:14.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.445+0000 7f30ab400640 1 -- 172.21.15.39:0/2352628561 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f309c0049e0 con 0x7f30b4104a00 2024-03-18T16:07:14.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.445+0000 7f30b3200640 1 -- 172.21.15.39:0/2352628561 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30b41a1f70 con 0x7f30b4104a00 2024-03-18T16:07:14.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.446+0000 7f30b3200640 1 -- 172.21.15.39:0/2352628561 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30b410ab30 con 0x7f30b4104a00 2024-03-18T16:07:14.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.448+0000 7f30ab400640 1 -- 172.21.15.39:0/2352628561 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f309c005560 con 0x7f30b4104a00 2024-03-18T16:07:14.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.449+0000 7f30ab400640 1 --2- 172.21.15.39:0/2352628561 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f30980a2880 0x7f30980a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:14.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.449+0000 7f30ab400640 1 -- 172.21.15.39:0/2352628561 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f309c0cec40 con 0x7f30b4104a00 2024-03-18T16:07:14.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.449+0000 7f30b1e00640 1 --2- 172.21.15.39:0/2352628561 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f30980a2880 0x7f30980a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:14.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.450+0000 7f30b1e00640 1 --2- 172.21.15.39:0/2352628561 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f30980a2880 0x7f30980a4d40 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f30a402c900 tx=0x7f30a4000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:14.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.453+0000 7f30ab400640 1 -- 172.21.15.39:0/2352628561 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f309c0051d0 con 0x7f30b4104a00 2024-03-18T16:07:14.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.659+0000 7f30b3200640 1 -- 172.21.15.39:0/2352628561 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f30b40623e0 con 0x7f30b4104a00 2024-03-18T16:07:14.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.661+0000 7f30ab400640 1 -- 172.21.15.39:0/2352628561 <== mon.2 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24142 (secure 0 0 0) 0x7f309c080e60 con 0x7f30b4104a00 2024-03-18T16:07:14.664 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:07:14.664 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":89,"fsid":"1892d8fc-e540-11ee-95c9-87774f69a715","created":"2024-03-18T15:57:36.507008+0000","modified":"2024-03-18T16:06:49.510653+0000","last_up_change":"2024-03-18T16:05:45.250629+0000","last_in_change":"2024-03-18T16:05:24.841337+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T16:01:30.048247+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T16:06:42.191791+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T16:06:43.670325+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T16:06:47.523388+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"771e0558-be62-4436-94b0-6364751911c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6803","nonce":4090743635}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6805","nonce":4090743635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6809","nonce":4090743635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6807","nonce":4090743635}]},"public_addr":"172.21.15.39:6803/4090743635","cluster_addr":"172.21.15.39:6805/4090743635","heartbeat_back_addr":"172.21.15.39:6809/4090743635","heartbeat_front_addr":"172.21.15.39:6807/4090743635","state":["exists","up"]},{"osd":1,"uuid":"0430eee1-cf34-4277-b255-c7d9f07ad19a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6801","nonce":2800329805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6803","nonce":2800329805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6807","nonce":2800329805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6805","nonce":2800329805}]},"public_addr":"172.21.15.194:6801/2800329805","cluster_addr":"172.21.15.194:6803/2800329805","heartbeat_back_addr":"172.21.15.194:6807/2800329805","heartbeat_front_addr":"172.21.15.194:6805/2800329805","state":["exists","up"]},{"osd":2,"uuid":"3a718da4-2511-4ec1-b1a8-eafd6cce0df0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6801","nonce":3935874719}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6803","nonce":3935874719}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6807","nonce":3935874719}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6805","nonce":3935874719}]},"public_addr":"172.21.15.196:6801/3935874719","cluster_addr":"172.21.15.196:6803/3935874719","heartbeat_back_addr":"172.21.15.196:6807/3935874719","heartbeat_front_addr":"172.21.15.196:6805/3935874719","state":["exists","up"]},{"osd":3,"uuid":"6f9c37f9-0ab9-4038-8919-afd4b39c4165","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6811","nonce":893345939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6813","nonce":893345939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6817","nonce":893345939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6815","nonce":893345939}]},"public_addr":"172.21.15.39:6811/893345939","cluster_addr":"172.21.15.39:6813/893345939","heartbeat_back_addr":"172.21.15.39:6817/893345939","heartbeat_front_addr":"172.21.15.39:6815/893345939","state":["exists","up"]},{"osd":4,"uuid":"e795bb0e-4791-4537-acda-1a321e19bc9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6809","nonce":800423088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6811","nonce":800423088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6815","nonce":800423088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6813","nonce":800423088}]},"public_addr":"172.21.15.194:6809/800423088","cluster_addr":"172.21.15.194:6811/800423088","heartbeat_back_addr":"172.21.15.194:6815/800423088","heartbeat_front_addr":"172.21.15.194:6813/800423088","state":["exists","up"]},{"osd":5,"uuid":"026e175e-6acb-4b4c-b235-8ccee8ae1149","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6809","nonce":2746224525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6811","nonce":2746224525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6815","nonce":2746224525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6813","nonce":2746224525}]},"public_addr":"172.21.15.196:6809/2746224525","cluster_addr":"172.21.15.196:6811/2746224525","heartbeat_back_addr":"172.21.15.196:6815/2746224525","heartbeat_front_addr":"172.21.15.196:6813/2746224525","state":["exists","up"]},{"osd":6,"uuid":"c1b183ff-383f-4c5c-9075-f76bab1c0bb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6819","nonce":1213371331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6821","nonce":1213371331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6825","nonce":1213371331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6823","nonce":1213371331}]},"public_addr":"172.21.15.39:6819/1213371331","cluster_addr":"172.21.15.39:6821/1213371331","heartbeat_back_addr":"172.21.15.39:6825/1213371331","heartbeat_front_addr":"172.21.15.39:6823/1213371331","state":["exists","up"]},{"osd":7,"uuid":"612f8697-957d-4ef4-a87f-241b36f9d13d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6817","nonce":3299162444}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6819","nonce":3299162444}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6823","nonce":3299162444}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6821","nonce":3299162444}]},"public_addr":"172.21.15.194:6817/3299162444","cluster_addr":"172.21.15.194:6819/3299162444","heartbeat_back_addr":"172.21.15.194:6823/3299162444","heartbeat_front_addr":"172.21.15.194:6821/3299162444","state":["exists","up"]},{"osd":8,"uuid":"836a9107-4d4f-4594-b8dc-0301e64f59d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6817","nonce":2456143145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6819","nonce":2456143145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6823","nonce":2456143145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6821","nonce":2456143145}]},"public_addr":"172.21.15.196:6817/2456143145","cluster_addr":"172.21.15.196:6819/2456143145","heartbeat_back_addr":"172.21.15.196:6823/2456143145","heartbeat_front_addr":"172.21.15.196:6821/2456143145","state":["exists","up"]},{"osd":9,"uuid":"4f30b960-12e3-4b16-bd28-fcc988139b58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6827","nonce":1848634433}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6829","nonce":1848634433}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6833","nonce":1848634433}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6831","nonce":1848634433}]},"public_addr":"172.21.15.39:6827/1848634433","cluster_addr":"172.21.15.39:6829/1848634433","heartbeat_back_addr":"172.21.15.39:6833/1848634433","heartbeat_front_addr":"172.21.15.39:6831/1848634433","state":["exists","up"]},{"osd":10,"uuid":"620f90e8-23e6-4bc0-8269-7219b9e1aebd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6825","nonce":3725064809}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6827","nonce":3725064809}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6831","nonce":3725064809}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6829","nonce":3725064809}]},"public_addr":"172.21.15.194:6825/3725064809","cluster_addr":"172.21.15.194:6827/3725064809","heartbeat_back_addr":"172.21.15.194:6831/3725064809","heartbeat_front_addr":"172.21.15.194:6829/3725064809","state":["exists","up"]},{"osd":11,"uuid":"867a2535-8235-4ceb-a698-eb86869d6d20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6825","nonce":2501212422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6827","nonce":2501212422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6831","nonce":2501212422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6829","nonce":2501212422}]},"public_addr":"172.21.15.196:6825/2501212422","cluster_addr":"172.21.15.196:6827/2501212422","heartbeat_back_addr":"172.21.15.196:6831/2501212422","heartbeat_front_addr":"172.21.15.196:6829/2501212422","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:00:36.931566+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:01:02.614024+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:01:26.193265+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:01:58.308833+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:02:25.049546+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:02:49.748750+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:03:23.582153+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:03:50.447321+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:04:15.812129+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:04:49.998047+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:05:17.087924+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:05:43.167637+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.39:0/3482086140":"2024-03-19T15:58:45.340294+0000","172.21.15.39:0/3844306284":"2024-03-19T15:58:45.340294+0000","172.21.15.39:6801/1810238546":"2024-03-19T15:58:45.340294+0000","172.21.15.39:6800/1810238546":"2024-03-19T15:58:45.340294+0000","172.21.15.39:0/2528397488":"2024-03-19T15:58:10.264798+0000","172.21.15.39:0/2240981254":"2024-03-19T15:58:10.264798+0000","172.21.15.39:6800/1217503189":"2024-03-19T15:58:10.264798+0000","172.21.15.39:0/3754876370":"2024-03-19T15:58:10.264798+0000","172.21.15.39:6801/1217503189":"2024-03-19T15:58:10.264798+0000","172.21.15.39:0/1874637628":"2024-03-19T15:58:45.340294+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T16:07:14.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.664+0000 7f30b3200640 1 -- 172.21.15.39:0/2352628561 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f30980a2880 msgr2=0x7f30980a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:14.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.664+0000 7f30b3200640 1 --2- 172.21.15.39:0/2352628561 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f30980a2880 0x7f30980a4d40 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f30a402c900 tx=0x7f30a4000bc0 comp rx=0 tx=0).stop 2024-03-18T16:07:14.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.664+0000 7f30b3200640 1 -- 172.21.15.39:0/2352628561 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f30b4104a00 msgr2=0x7f30b419a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:14.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.664+0000 7f30b3200640 1 --2- 172.21.15.39:0/2352628561 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f30b4104a00 0x7f30b419a050 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f309c00e9f0 tx=0x7f309c00eeb0 comp rx=0 tx=0).stop 2024-03-18T16:07:14.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.664+0000 7f30b3200640 1 -- 172.21.15.39:0/2352628561 shutdown_connections 2024-03-18T16:07:14.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.664+0000 7f30b3200640 1 --2- 172.21.15.39:0/2352628561 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f30b4104a00 0x7f30b419a050 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:14.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.664+0000 7f30b3200640 1 --2- 172.21.15.39:0/2352628561 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f30980a2880 0x7f30980a4d40 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:14.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.664+0000 7f30b3200640 1 --2- 172.21.15.39:0/2352628561 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f30b41053c0 0x7f30b41a10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:14.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.664+0000 7f30b3200640 1 --2- 172.21.15.39:0/2352628561 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f30b4103810 0x7f30b4199b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:14.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.664+0000 7f30b3200640 1 -- 172.21.15.39:0/2352628561 >> 172.21.15.39:0/2352628561 conn(0x7f30b40fed80 msgr2=0x7f30b4107d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:14.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.664+0000 7f30b3200640 1 -- 172.21.15.39:0/2352628561 shutdown_connections 2024-03-18T16:07:14.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.664+0000 7f30b3200640 1 -- 172.21.15.39:0/2352628561 wait complete. 2024-03-18T16:07:14.687 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T16:07:14.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.906+0000 7f944ba00640 1 -- 172.21.15.39:0/2573151711 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f94440757c0 msgr2=0x7f9444073c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:14.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.906+0000 7f944ba00640 1 --2- 172.21.15.39:0/2573151711 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f94440757c0 0x7f9444073c10 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f9434009ca0 tx=0x7f943402c190 comp rx=0 tx=0).stop 2024-03-18T16:07:14.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.906+0000 7f944ba00640 1 -- 172.21.15.39:0/2573151711 shutdown_connections 2024-03-18T16:07:14.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.906+0000 7f944ba00640 1 --2- 172.21.15.39:0/2573151711 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9444109680 0x7f944410bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:14.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.906+0000 7f944ba00640 1 --2- 172.21.15.39:0/2573151711 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f94440741e0 0x7f9444074640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:14.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.906+0000 7f944ba00640 1 --2- 172.21.15.39:0/2573151711 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f94440757c0 0x7f9444073c10 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:14.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.906+0000 7f944ba00640 1 -- 172.21.15.39:0/2573151711 >> 172.21.15.39:0/2573151711 conn(0x7f94440fcd80 msgr2=0x7f94440ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:14.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.906+0000 7f944ba00640 1 -- 172.21.15.39:0/2573151711 shutdown_connections 2024-03-18T16:07:14.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.906+0000 7f944ba00640 1 -- 172.21.15.39:0/2573151711 wait complete. 2024-03-18T16:07:14.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.907+0000 7f944ba00640 1 Processor -- start 2024-03-18T16:07:14.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.907+0000 7f944ba00640 1 -- start start 2024-03-18T16:07:14.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.907+0000 7f944ba00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f94440741e0 0x7f9444199d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:14.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.907+0000 7f944ba00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f94440757c0 0x7f944419a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:14.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.907+0000 7f944ba00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9444109680 0x7f94441a1270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:14.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.907+0000 7f944ba00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f944410f640 con 0x7f94440757c0 2024-03-18T16:07:14.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.907+0000 7f944ba00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f944410f4c0 con 0x7f9444109680 2024-03-18T16:07:14.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.907+0000 7f944ba00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f944410f7c0 con 0x7f94440741e0 2024-03-18T16:07:14.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.908+0000 7f9449a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9444109680 0x7f94441a1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:14.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.908+0000 7f9443e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f94440757c0 0x7f944419a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:14.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.908+0000 7f9449a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9444109680 0x7f94441a1270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:54944/0 (socket says 172.21.15.39:54944) 2024-03-18T16:07:14.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.908+0000 7f9449a00640 1 -- 172.21.15.39:0/291296249 learned_addr learned my addr 172.21.15.39:0/291296249 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:07:14.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.908+0000 7f9449000640 1 --2- 172.21.15.39:0/291296249 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f94440741e0 0x7f9444199d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:14.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.908+0000 7f9449a00640 1 -- 172.21.15.39:0/291296249 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f94440741e0 msgr2=0x7f9444199d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:14.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.908+0000 7f9449a00640 1 --2- 172.21.15.39:0/291296249 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f94440741e0 0x7f9444199d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:14.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.908+0000 7f9449a00640 1 -- 172.21.15.39:0/291296249 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f94440757c0 msgr2=0x7f944419a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:14.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.908+0000 7f9449a00640 1 --2- 172.21.15.39:0/291296249 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f94440757c0 0x7f944419a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:14.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.908+0000 7f9449a00640 1 -- 172.21.15.39:0/291296249 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94441a1910 con 0x7f9444109680 2024-03-18T16:07:14.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.908+0000 7f9449a00640 1 --2- 172.21.15.39:0/291296249 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9444109680 0x7f94441a1270 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f943c00d950 tx=0x7f943c00de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:14.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.909+0000 7f9441600640 1 -- 172.21.15.39:0/291296249 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f943c017030 con 0x7f9444109680 2024-03-18T16:07:14.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.909+0000 7f9441600640 1 -- 172.21.15.39:0/291296249 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f943c004020 con 0x7f9444109680 2024-03-18T16:07:14.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.909+0000 7f9441600640 1 -- 172.21.15.39:0/291296249 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f943c003c00 con 0x7f9444109680 2024-03-18T16:07:14.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.909+0000 7f944ba00640 1 -- 172.21.15.39:0/291296249 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94441a1c00 con 0x7f9444109680 2024-03-18T16:07:14.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.909+0000 7f944ba00640 1 -- 172.21.15.39:0/291296249 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94441a2100 con 0x7f9444109680 2024-03-18T16:07:14.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.910+0000 7f944ba00640 1 -- 172.21.15.39:0/291296249 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9444108440 con 0x7f9444109680 2024-03-18T16:07:14.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.912+0000 7f9441600640 1 -- 172.21.15.39:0/291296249 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f943c01a070 con 0x7f9444109680 2024-03-18T16:07:14.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.913+0000 7f9441600640 1 --2- 172.21.15.39:0/291296249 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f94280a2880 0x7f94280a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:14.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.913+0000 7f9441600640 1 -- 172.21.15.39:0/291296249 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f943c0ce550 con 0x7f9444109680 2024-03-18T16:07:14.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.913+0000 7f9449000640 1 --2- 172.21.15.39:0/291296249 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f94280a2880 0x7f94280a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:14.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.914+0000 7f9449000640 1 --2- 172.21.15.39:0/291296249 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f94280a2880 0x7f94280a4d40 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f943402c7a0 tx=0x7f9434032670 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:14.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:14.917+0000 7f9441600640 1 -- 172.21.15.39:0/291296249 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f943c021070 con 0x7f9444109680 2024-03-18T16:07:15.124 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:15.122+0000 7f944ba00640 1 -- 172.21.15.39:0/291296249 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9444062410 con 0x7f9444109680 2024-03-18T16:07:15.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:15.124+0000 7f9441600640 1 -- 172.21.15.39:0/291296249 <== mon.2 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24142 (secure 0 0 0) 0x7f943c09b350 con 0x7f9444109680 2024-03-18T16:07:15.127 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:07:15.127 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":89,"fsid":"1892d8fc-e540-11ee-95c9-87774f69a715","created":"2024-03-18T15:57:36.507008+0000","modified":"2024-03-18T16:06:49.510653+0000","last_up_change":"2024-03-18T16:05:45.250629+0000","last_in_change":"2024-03-18T16:05:24.841337+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T16:01:30.048247+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T16:06:42.191791+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T16:06:43.670325+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T16:06:47.523388+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"771e0558-be62-4436-94b0-6364751911c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6803","nonce":4090743635}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6805","nonce":4090743635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6809","nonce":4090743635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6807","nonce":4090743635}]},"public_addr":"172.21.15.39:6803/4090743635","cluster_addr":"172.21.15.39:6805/4090743635","heartbeat_back_addr":"172.21.15.39:6809/4090743635","heartbeat_front_addr":"172.21.15.39:6807/4090743635","state":["exists","up"]},{"osd":1,"uuid":"0430eee1-cf34-4277-b255-c7d9f07ad19a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6801","nonce":2800329805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6803","nonce":2800329805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6807","nonce":2800329805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6805","nonce":2800329805}]},"public_addr":"172.21.15.194:6801/2800329805","cluster_addr":"172.21.15.194:6803/2800329805","heartbeat_back_addr":"172.21.15.194:6807/2800329805","heartbeat_front_addr":"172.21.15.194:6805/2800329805","state":["exists","up"]},{"osd":2,"uuid":"3a718da4-2511-4ec1-b1a8-eafd6cce0df0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6801","nonce":3935874719}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6803","nonce":3935874719}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6807","nonce":3935874719}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6805","nonce":3935874719}]},"public_addr":"172.21.15.196:6801/3935874719","cluster_addr":"172.21.15.196:6803/3935874719","heartbeat_back_addr":"172.21.15.196:6807/3935874719","heartbeat_front_addr":"172.21.15.196:6805/3935874719","state":["exists","up"]},{"osd":3,"uuid":"6f9c37f9-0ab9-4038-8919-afd4b39c4165","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6811","nonce":893345939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6813","nonce":893345939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6817","nonce":893345939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6815","nonce":893345939}]},"public_addr":"172.21.15.39:6811/893345939","cluster_addr":"172.21.15.39:6813/893345939","heartbeat_back_addr":"172.21.15.39:6817/893345939","heartbeat_front_addr":"172.21.15.39:6815/893345939","state":["exists","up"]},{"osd":4,"uuid":"e795bb0e-4791-4537-acda-1a321e19bc9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6809","nonce":800423088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6811","nonce":800423088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6815","nonce":800423088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6813","nonce":800423088}]},"public_addr":"172.21.15.194:6809/800423088","cluster_addr":"172.21.15.194:6811/800423088","heartbeat_back_addr":"172.21.15.194:6815/800423088","heartbeat_front_addr":"172.21.15.194:6813/800423088","state":["exists","up"]},{"osd":5,"uuid":"026e175e-6acb-4b4c-b235-8ccee8ae1149","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6809","nonce":2746224525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6811","nonce":2746224525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6815","nonce":2746224525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6813","nonce":2746224525}]},"public_addr":"172.21.15.196:6809/2746224525","cluster_addr":"172.21.15.196:6811/2746224525","heartbeat_back_addr":"172.21.15.196:6815/2746224525","heartbeat_front_addr":"172.21.15.196:6813/2746224525","state":["exists","up"]},{"osd":6,"uuid":"c1b183ff-383f-4c5c-9075-f76bab1c0bb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6819","nonce":1213371331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6821","nonce":1213371331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6825","nonce":1213371331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6823","nonce":1213371331}]},"public_addr":"172.21.15.39:6819/1213371331","cluster_addr":"172.21.15.39:6821/1213371331","heartbeat_back_addr":"172.21.15.39:6825/1213371331","heartbeat_front_addr":"172.21.15.39:6823/1213371331","state":["exists","up"]},{"osd":7,"uuid":"612f8697-957d-4ef4-a87f-241b36f9d13d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6817","nonce":3299162444}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6819","nonce":3299162444}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6823","nonce":3299162444}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6821","nonce":3299162444}]},"public_addr":"172.21.15.194:6817/3299162444","cluster_addr":"172.21.15.194:6819/3299162444","heartbeat_back_addr":"172.21.15.194:6823/3299162444","heartbeat_front_addr":"172.21.15.194:6821/3299162444","state":["exists","up"]},{"osd":8,"uuid":"836a9107-4d4f-4594-b8dc-0301e64f59d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6817","nonce":2456143145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6819","nonce":2456143145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6823","nonce":2456143145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6821","nonce":2456143145}]},"public_addr":"172.21.15.196:6817/2456143145","cluster_addr":"172.21.15.196:6819/2456143145","heartbeat_back_addr":"172.21.15.196:6823/2456143145","heartbeat_front_addr":"172.21.15.196:6821/2456143145","state":["exists","up"]},{"osd":9,"uuid":"4f30b960-12e3-4b16-bd28-fcc988139b58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6827","nonce":1848634433}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6829","nonce":1848634433}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6833","nonce":1848634433}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6831","nonce":1848634433}]},"public_addr":"172.21.15.39:6827/1848634433","cluster_addr":"172.21.15.39:6829/1848634433","heartbeat_back_addr":"172.21.15.39:6833/1848634433","heartbeat_front_addr":"172.21.15.39:6831/1848634433","state":["exists","up"]},{"osd":10,"uuid":"620f90e8-23e6-4bc0-8269-7219b9e1aebd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6825","nonce":3725064809}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6827","nonce":3725064809}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6831","nonce":3725064809}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6829","nonce":3725064809}]},"public_addr":"172.21.15.194:6825/3725064809","cluster_addr":"172.21.15.194:6827/3725064809","heartbeat_back_addr":"172.21.15.194:6831/3725064809","heartbeat_front_addr":"172.21.15.194:6829/3725064809","state":["exists","up"]},{"osd":11,"uuid":"867a2535-8235-4ceb-a698-eb86869d6d20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6825","nonce":2501212422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6827","nonce":2501212422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6831","nonce":2501212422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6829","nonce":2501212422}]},"public_addr":"172.21.15.196:6825/2501212422","cluster_addr":"172.21.15.196:6827/2501212422","heartbeat_back_addr":"172.21.15.196:6831/2501212422","heartbeat_front_addr":"172.21.15.196:6829/2501212422","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:00:36.931566+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:01:02.614024+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:01:26.193265+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:01:58.308833+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:02:25.049546+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:02:49.748750+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:03:23.582153+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:03:50.447321+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:04:15.812129+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:04:49.998047+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:05:17.087924+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:05:43.167637+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.39:0/3482086140":"2024-03-19T15:58:45.340294+0000","172.21.15.39:0/3844306284":"2024-03-19T15:58:45.340294+0000","172.21.15.39:6801/1810238546":"2024-03-19T15:58:45.340294+0000","172.21.15.39:6800/1810238546":"2024-03-19T15:58:45.340294+0000","172.21.15.39:0/2528397488":"2024-03-19T15:58:10.264798+0000","172.21.15.39:0/2240981254":"2024-03-19T15:58:10.264798+0000","172.21.15.39:6800/1217503189":"2024-03-19T15:58:10.264798+0000","172.21.15.39:0/3754876370":"2024-03-19T15:58:10.264798+0000","172.21.15.39:6801/1217503189":"2024-03-19T15:58:10.264798+0000","172.21.15.39:0/1874637628":"2024-03-19T15:58:45.340294+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T16:07:15.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:15.126+0000 7f944ba00640 1 -- 172.21.15.39:0/291296249 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f94280a2880 msgr2=0x7f94280a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:15.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:15.126+0000 7f944ba00640 1 --2- 172.21.15.39:0/291296249 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f94280a2880 0x7f94280a4d40 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f943402c7a0 tx=0x7f9434032670 comp rx=0 tx=0).stop 2024-03-18T16:07:15.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:15.126+0000 7f944ba00640 1 -- 172.21.15.39:0/291296249 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9444109680 msgr2=0x7f94441a1270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:15.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:15.126+0000 7f944ba00640 1 --2- 172.21.15.39:0/291296249 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9444109680 0x7f94441a1270 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f943c00d950 tx=0x7f943c00de10 comp rx=0 tx=0).stop 2024-03-18T16:07:15.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:15.126+0000 7f944ba00640 1 -- 172.21.15.39:0/291296249 shutdown_connections 2024-03-18T16:07:15.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:15.126+0000 7f944ba00640 1 --2- 172.21.15.39:0/291296249 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9444109680 0x7f94441a1270 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:15.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:15.126+0000 7f944ba00640 1 --2- 172.21.15.39:0/291296249 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f94280a2880 0x7f94280a4d40 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:15.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:15.126+0000 7f944ba00640 1 --2- 172.21.15.39:0/291296249 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f94440757c0 0x7f944419a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:15.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:15.126+0000 7f944ba00640 1 --2- 172.21.15.39:0/291296249 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f94440741e0 0x7f9444199d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:15.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:15.126+0000 7f944ba00640 1 -- 172.21.15.39:0/291296249 >> 172.21.15.39:0/291296249 conn(0x7f94440fcd80 msgr2=0x7f9444105610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:15.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:15.127+0000 7f944ba00640 1 -- 172.21.15.39:0/291296249 shutdown_connections 2024-03-18T16:07:15.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:15.127+0000 7f944ba00640 1 -- 172.21.15.39:0/291296249 wait complete. 2024-03-18T16:07:15.149 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-18T16:07:15.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:15.366+0000 7f5942200640 1 -- 172.21.15.39:0/1967165226 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f593c0757e0 msgr2=0x7f593c073c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:15.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:15.366+0000 7f5942200640 1 --2- 172.21.15.39:0/1967165226 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f593c0757e0 0x7f593c073c30 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f5930009900 tx=0x7f593002bc30 comp rx=0 tx=0).stop 2024-03-18T16:07:15.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:15.366+0000 7f5942200640 1 -- 172.21.15.39:0/1967165226 shutdown_connections 2024-03-18T16:07:15.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:15.366+0000 7f5942200640 1 --2- 172.21.15.39:0/1967165226 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f593c109570 0x7f593c10b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:15.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:15.366+0000 7f5942200640 1 --2- 172.21.15.39:0/1967165226 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f593c074200 0x7f593c074680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:15.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:15.366+0000 7f5942200640 1 --2- 172.21.15.39:0/1967165226 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f593c0757e0 0x7f593c073c30 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:15.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:15.366+0000 7f5942200640 1 -- 172.21.15.39:0/1967165226 >> 172.21.15.39:0/1967165226 conn(0x7f593c0fcc50 msgr2=0x7f593c0ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:15.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:15.366+0000 7f5942200640 1 -- 172.21.15.39:0/1967165226 shutdown_connections 2024-03-18T16:07:15.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:15.367+0000 7f5942200640 1 -- 172.21.15.39:0/1967165226 wait complete. 2024-03-18T16:07:15.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:15.368+0000 7f5942200640 1 Processor -- start 2024-03-18T16:07:15.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:15.368+0000 7f5942200640 1 -- start start 2024-03-18T16:07:15.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:15.368+0000 7f5942200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f593c074200 0x7f593c199ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:15.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:15.368+0000 7f5942200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f593c0757e0 0x7f593c19a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:15.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:15.368+0000 7f5942200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f593c109570 0x7f593c1a1150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:15.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:15.368+0000 7f5942200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f593c10f570 con 0x7f593c074200 2024-03-18T16:07:15.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:15.368+0000 7f5942200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f593c10f3f0 con 0x7f593c109570 2024-03-18T16:07:15.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:15.368+0000 7f5942200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f593c10f6f0 con 0x7f593c0757e0 2024-03-18T16:07:15.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:15.369+0000 7f5940e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f593c074200 0x7f593c199ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:15.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:15.369+0000 7f5941800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f593c109570 0x7f593c1a1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:15.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:15.369+0000 7f5941800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f593c109570 0x7f593c1a1150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:54966/0 (socket says 172.21.15.39:54966) 2024-03-18T16:07:15.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:15.369+0000 7f5941800640 1 -- 172.21.15.39:0/2987157937 learned_addr learned my addr 172.21.15.39:0/2987157937 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:07:15.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:15.369+0000 7f593be00640 1 --2- 172.21.15.39:0/2987157937 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f593c0757e0 0x7f593c19a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:15.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:15.369+0000 7f5941800640 1 -- 172.21.15.39:0/2987157937 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f593c0757e0 msgr2=0x7f593c19a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:15.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:15.369+0000 7f5941800640 1 --2- 172.21.15.39:0/2987157937 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f593c0757e0 0x7f593c19a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:15.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:15.369+0000 7f5941800640 1 -- 172.21.15.39:0/2987157937 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f593c074200 msgr2=0x7f593c199ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:15.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:15.369+0000 7f5941800640 1 --2- 172.21.15.39:0/2987157937 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f593c074200 0x7f593c199ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:15.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:15.369+0000 7f5941800640 1 -- 172.21.15.39:0/2987157937 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f593c1a17c0 con 0x7f593c109570 2024-03-18T16:07:15.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:15.369+0000 7f5941800640 1 --2- 172.21.15.39:0/2987157937 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f593c109570 0x7f593c1a1150 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f593400eb10 tx=0x7f593400c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:15.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:15.370+0000 7f5939600640 1 -- 172.21.15.39:0/2987157937 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5934004b20 con 0x7f593c109570 2024-03-18T16:07:15.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:15.370+0000 7f5939600640 1 -- 172.21.15.39:0/2987157937 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5934004cc0 con 0x7f593c109570 2024-03-18T16:07:15.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:15.370+0000 7f5939600640 1 -- 172.21.15.39:0/2987157937 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5934010760 con 0x7f593c109570 2024-03-18T16:07:15.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:15.370+0000 7f5942200640 1 -- 172.21.15.39:0/2987157937 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f593c1a1ab0 con 0x7f593c109570 2024-03-18T16:07:15.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:15.370+0000 7f5942200640 1 -- 172.21.15.39:0/2987157937 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f593c1a2050 con 0x7f593c109570 2024-03-18T16:07:15.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:15.371+0000 7f5942200640 1 -- 172.21.15.39:0/2987157937 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f593c1082f0 con 0x7f593c109570 2024-03-18T16:07:15.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:15.373+0000 7f5939600640 1 -- 172.21.15.39:0/2987157937 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f5934005020 con 0x7f593c109570 2024-03-18T16:07:15.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:15.374+0000 7f5939600640 1 --2- 172.21.15.39:0/2987157937 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f59240a2880 0x7f59240a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:15.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:15.374+0000 7f5939600640 1 -- 172.21.15.39:0/2987157937 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f59340cf230 con 0x7f593c109570 2024-03-18T16:07:15.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:15.374+0000 7f5940e00640 1 --2- 172.21.15.39:0/2987157937 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f59240a2880 0x7f59240a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:15.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:15.375+0000 7f5940e00640 1 --2- 172.21.15.39:0/2987157937 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f59240a2880 0x7f59240a4d40 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f593002c240 tx=0x7f593002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:15.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:15.378+0000 7f5939600640 1 -- 172.21.15.39:0/2987157937 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f593409c130 con 0x7f593c109570 2024-03-18T16:07:15.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:15 smithi196 ceph-mon[36307]: pgmap v333: 129 pgs: 129 active+clean; 581 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-18T16:07:15.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:15 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/2352628561' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:07:15.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:15 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/291296249' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:07:15.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:15 smithi194 ceph-mon[36200]: pgmap v333: 129 pgs: 129 active+clean; 581 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-18T16:07:15.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:15 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/2352628561' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:07:15.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:15 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/291296249' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:07:15.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:15 smithi039 ceph-mon[31721]: pgmap v333: 129 pgs: 129 active+clean; 581 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-18T16:07:15.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:15 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/2352628561' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:07:15.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:15 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/291296249' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:07:15.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:15.624+0000 7f5942200640 1 -- 172.21.15.39:0/2987157937 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f593c02d1b0 con 0x7f593c109570 2024-03-18T16:07:16.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:16.166+0000 7f5939600640 1 -- 172.21.15.39:0/2987157937 <== mon.2 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v17) ==== 118+0+0 (secure 0 0 0) 0x7f593409ba30 con 0x7f593c109570 2024-03-18T16:07:16.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:16.171+0000 7f5942200640 1 -- 172.21.15.39:0/2987157937 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f59240a2880 msgr2=0x7f59240a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:16.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:16.171+0000 7f5942200640 1 --2- 172.21.15.39:0/2987157937 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f59240a2880 0x7f59240a4d40 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f593002c240 tx=0x7f593002df90 comp rx=0 tx=0).stop 2024-03-18T16:07:16.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:16.171+0000 7f5942200640 1 -- 172.21.15.39:0/2987157937 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f593c109570 msgr2=0x7f593c1a1150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:16.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:16.171+0000 7f5942200640 1 --2- 172.21.15.39:0/2987157937 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f593c109570 0x7f593c1a1150 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f593400eb10 tx=0x7f593400c2d0 comp rx=0 tx=0).stop 2024-03-18T16:07:16.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:16.171+0000 7f5942200640 1 -- 172.21.15.39:0/2987157937 shutdown_connections 2024-03-18T16:07:16.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:16.171+0000 7f5942200640 1 --2- 172.21.15.39:0/2987157937 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f593c109570 0x7f593c1a1150 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:16.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:16.171+0000 7f5942200640 1 --2- 172.21.15.39:0/2987157937 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f59240a2880 0x7f59240a4d40 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:16.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:16.171+0000 7f5942200640 1 --2- 172.21.15.39:0/2987157937 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f593c0757e0 0x7f593c19a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:16.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:16.171+0000 7f5942200640 1 --2- 172.21.15.39:0/2987157937 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f593c074200 0x7f593c199ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:16.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:16.171+0000 7f5942200640 1 -- 172.21.15.39:0/2987157937 >> 172.21.15.39:0/2987157937 conn(0x7f593c0fcc50 msgr2=0x7f593c1054c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:16.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:16.171+0000 7f5942200640 1 -- 172.21.15.39:0/2987157937 shutdown_connections 2024-03-18T16:07:16.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:16.171+0000 7f5942200640 1 -- 172.21.15.39:0/2987157937 wait complete. 2024-03-18T16:07:16.200 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2}} 2024-03-18T16:07:16.200 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5 2024-03-18T16:07:16.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:16.428+0000 7f83b2600640 1 -- 172.21.15.39:0/2433148328 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f83ac105d20 msgr2=0x7f83ac108110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:16.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:16.428+0000 7f83b2600640 1 --2- 172.21.15.39:0/2433148328 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f83ac105d20 0x7f83ac108110 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f83a4009be0 tx=0x7f83a402bef0 comp rx=0 tx=0).stop 2024-03-18T16:07:16.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:16.428+0000 7f83b2600640 1 -- 172.21.15.39:0/2433148328 shutdown_connections 2024-03-18T16:07:16.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:16.428+0000 7f83b2600640 1 --2- 172.21.15.39:0/2433148328 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f83ac105d20 0x7f83ac108110 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:16.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:16.428+0000 7f83b2600640 1 --2- 172.21.15.39:0/2433148328 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83ac101020 0x7f83ac1014a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:16.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:16.428+0000 7f83b2600640 1 --2- 172.21.15.39:0/2433148328 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f83ac067c60 0x7f83ac100ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:16.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:16.428+0000 7f83b2600640 1 -- 172.21.15.39:0/2433148328 >> 172.21.15.39:0/2433148328 conn(0x7f83ac0fae60 msgr2=0x7f83ac0fd2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:16.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:16.428+0000 7f83b2600640 1 -- 172.21.15.39:0/2433148328 shutdown_connections 2024-03-18T16:07:16.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:16.429+0000 7f83b2600640 1 -- 172.21.15.39:0/2433148328 wait complete. 2024-03-18T16:07:16.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:16.430+0000 7f83b2600640 1 Processor -- start 2024-03-18T16:07:16.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:16.430+0000 7f83b2600640 1 -- start start 2024-03-18T16:07:16.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:16.430+0000 7f83b2600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83ac067c60 0x7f83ac199ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:16.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:16.430+0000 7f83b2600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f83ac101020 0x7f83ac19a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:16.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:16.430+0000 7f83b2600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f83ac105d20 0x7f83ac1a1150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:16.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:16.430+0000 7f83b2600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f83ac10d3d0 con 0x7f83ac067c60 2024-03-18T16:07:16.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:16.430+0000 7f83b2600640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f83ac10d250 con 0x7f83ac101020 2024-03-18T16:07:16.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:16.430+0000 7f83b2600640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f83ac10d550 con 0x7f83ac105d20 2024-03-18T16:07:16.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:16.431+0000 7f83abe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f83ac101020 0x7f83ac19a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:16.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:16.431+0000 7f83b1200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83ac067c60 0x7f83ac199ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:16.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:16.431+0000 7f83abe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f83ac101020 0x7f83ac19a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:54990/0 (socket says 172.21.15.39:54990) 2024-03-18T16:07:16.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:16.431+0000 7f83abe00640 1 -- 172.21.15.39:0/3802534630 learned_addr learned my addr 172.21.15.39:0/3802534630 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:07:16.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:16.431+0000 7f83b1c00640 1 --2- 172.21.15.39:0/3802534630 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f83ac105d20 0x7f83ac1a1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:16.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:16.431+0000 7f83abe00640 1 -- 172.21.15.39:0/3802534630 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f83ac105d20 msgr2=0x7f83ac1a1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:16.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:16.431+0000 7f83abe00640 1 --2- 172.21.15.39:0/3802534630 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f83ac105d20 0x7f83ac1a1150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:16.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:16.431+0000 7f83abe00640 1 -- 172.21.15.39:0/3802534630 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83ac067c60 msgr2=0x7f83ac199ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:16.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:16.431+0000 7f83abe00640 1 --2- 172.21.15.39:0/3802534630 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83ac067c60 0x7f83ac199ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:16.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:16.431+0000 7f83abe00640 1 -- 172.21.15.39:0/3802534630 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83ac1a17c0 con 0x7f83ac101020 2024-03-18T16:07:16.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:16.431+0000 7f83abe00640 1 --2- 172.21.15.39:0/3802534630 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f83ac101020 0x7f83ac19a0e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f839c00e9a0 tx=0x7f839c00ee60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:16.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:16.432+0000 7f83a9600640 1 -- 172.21.15.39:0/3802534630 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f839c00cd80 con 0x7f83ac101020 2024-03-18T16:07:16.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:16.432+0000 7f83a9600640 1 -- 172.21.15.39:0/3802534630 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f839c010910 con 0x7f83ac101020 2024-03-18T16:07:16.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:16.432+0000 7f83b2600640 1 -- 172.21.15.39:0/3802534630 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83ac1a1ab0 con 0x7f83ac101020 2024-03-18T16:07:16.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:16.432+0000 7f83a9600640 1 -- 172.21.15.39:0/3802534630 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f839c018a30 con 0x7f83ac101020 2024-03-18T16:07:16.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:16.432+0000 7f83b2600640 1 -- 172.21.15.39:0/3802534630 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83ac1a2050 con 0x7f83ac101020 2024-03-18T16:07:16.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:16.433+0000 7f83b2600640 1 -- 172.21.15.39:0/3802534630 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83ac1a2280 con 0x7f83ac101020 2024-03-18T16:07:16.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:16.436+0000 7f83a9600640 1 -- 172.21.15.39:0/3802534630 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f839c010ac0 con 0x7f83ac101020 2024-03-18T16:07:16.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:16.437+0000 7f83a9600640 1 --2- 172.21.15.39:0/3802534630 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f83940a2880 0x7f83940a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:16.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:16.437+0000 7f83a9600640 1 -- 172.21.15.39:0/3802534630 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f839c0cf030 con 0x7f83ac101020 2024-03-18T16:07:16.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:16.437+0000 7f83b1200640 1 --2- 172.21.15.39:0/3802534630 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f83940a2880 0x7f83940a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:16.440 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:16.438+0000 7f83b1200640 1 --2- 172.21.15.39:0/3802534630 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f83940a2880 0x7f83940a4d40 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f83ac100940 tx=0x7f83a0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:16.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:16.441+0000 7f83a9600640 1 -- 172.21.15.39:0/3802534630 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f839c053740 con 0x7f83ac101020 2024-03-18T16:07:16.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:16 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/2987157937' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-18T16:07:16.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:16 smithi196 ceph-mon[36307]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-18T16:07:16.489 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:16 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/2987157937' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-18T16:07:16.489 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:16 smithi039 ceph-mon[31721]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-18T16:07:16.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:16 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/2987157937' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-18T16:07:16.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:16 smithi194 ceph-mon[36200]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-18T16:07:16.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:16.689+0000 7f83b2600640 1 -- 172.21.15.39:0/3802534630 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) -- 0x7f83ac1a2450 con 0x7f83ac101020 2024-03-18T16:07:17.189 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:17.187+0000 7f83a9600640 1 -- 172.21.15.39:0/3802534630 <== mon.2 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v18) ==== 105+0+0 (secure 0 0 0) 0x7f839c09be90 con 0x7f83ac101020 2024-03-18T16:07:17.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:17.190+0000 7f83b2600640 1 -- 172.21.15.39:0/3802534630 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f83940a2880 msgr2=0x7f83940a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:17.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:17.190+0000 7f83b2600640 1 --2- 172.21.15.39:0/3802534630 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f83940a2880 0x7f83940a4d40 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f83ac100940 tx=0x7f83a0005f90 comp rx=0 tx=0).stop 2024-03-18T16:07:17.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:17.190+0000 7f83b2600640 1 -- 172.21.15.39:0/3802534630 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f83ac101020 msgr2=0x7f83ac19a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:17.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:17.190+0000 7f83b2600640 1 --2- 172.21.15.39:0/3802534630 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f83ac101020 0x7f83ac19a0e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f839c00e9a0 tx=0x7f839c00ee60 comp rx=0 tx=0).stop 2024-03-18T16:07:17.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:17.190+0000 7f83b2600640 1 -- 172.21.15.39:0/3802534630 shutdown_connections 2024-03-18T16:07:17.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:17.190+0000 7f83b2600640 1 --2- 172.21.15.39:0/3802534630 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f83ac101020 0x7f83ac19a0e0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:17.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:17.190+0000 7f83b2600640 1 --2- 172.21.15.39:0/3802534630 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f83940a2880 0x7f83940a4d40 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:17.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:17.190+0000 7f83b2600640 1 --2- 172.21.15.39:0/3802534630 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f83ac105d20 0x7f83ac1a1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:17.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:17.190+0000 7f83b2600640 1 --2- 172.21.15.39:0/3802534630 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83ac067c60 0x7f83ac199ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:17.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:17.190+0000 7f83b2600640 1 -- 172.21.15.39:0/3802534630 >> 172.21.15.39:0/3802534630 conn(0x7f83ac0fae60 msgr2=0x7f83ac0fb240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:17.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:17.191+0000 7f83b2600640 1 -- 172.21.15.39:0/3802534630 shutdown_connections 2024-03-18T16:07:17.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:17.191+0000 7f83b2600640 1 -- 172.21.15.39:0/3802534630 wait complete. 2024-03-18T16:07:17.221 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-03-18T16:07:17.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:17.431+0000 7f09c4a00640 1 -- 172.21.15.39:0/3866051350 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f09c01016c0 msgr2=0x7f09c0101ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:17.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:17.431+0000 7f09c4a00640 1 --2- 172.21.15.39:0/3866051350 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f09c01016c0 0x7f09c0101ae0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f09b0009900 tx=0x7f09b002bc50 comp rx=0 tx=0).stop 2024-03-18T16:07:17.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:17.432+0000 7f09c4a00640 1 -- 172.21.15.39:0/3866051350 shutdown_connections 2024-03-18T16:07:17.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:17.432+0000 7f09c4a00640 1 --2- 172.21.15.39:0/3866051350 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f09c0103270 0x7f09c0109b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:17.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:17.432+0000 7f09c4a00640 1 --2- 172.21.15.39:0/3866051350 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f09c01028b0 0x7f09c0102d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:17.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:17.432+0000 7f09c4a00640 1 --2- 172.21.15.39:0/3866051350 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f09c01016c0 0x7f09c0101ae0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:17.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:17.432+0000 7f09c4a00640 1 -- 172.21.15.39:0/3866051350 >> 172.21.15.39:0/3866051350 conn(0x7f09c00fcc50 msgr2=0x7f09c00ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:17.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:17.432+0000 7f09c4a00640 1 -- 172.21.15.39:0/3866051350 shutdown_connections 2024-03-18T16:07:17.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:17.432+0000 7f09c4a00640 1 -- 172.21.15.39:0/3866051350 wait complete. 2024-03-18T16:07:17.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:17.433+0000 7f09c4a00640 1 Processor -- start 2024-03-18T16:07:17.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:17.433+0000 7f09c4a00640 1 -- start start 2024-03-18T16:07:17.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:17.433+0000 7f09c4a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f09c01016c0 0x7f09c0197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:17.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:17.433+0000 7f09c4a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f09c01028b0 0x7f09c0197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:17.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:17.433+0000 7f09c4a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f09c0103270 0x7f09c019efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:17.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:17.433+0000 7f09c4a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f09c010d310 con 0x7f09c01028b0 2024-03-18T16:07:17.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:17.433+0000 7f09c4a00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f09c010d190 con 0x7f09c0103270 2024-03-18T16:07:17.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:17.433+0000 7f09c4a00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f09c010d490 con 0x7f09c01016c0 2024-03-18T16:07:17.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:17.433+0000 7f09be800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f09c01016c0 0x7f09c0197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:17.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:17.433+0000 7f09bf200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f09c0103270 0x7f09c019efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:17.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:17.433+0000 7f09be800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f09c01016c0 0x7f09c0197a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:38606/0 (socket says 172.21.15.39:38606) 2024-03-18T16:07:17.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:17.433+0000 7f09be800640 1 -- 172.21.15.39:0/1968967381 learned_addr learned my addr 172.21.15.39:0/1968967381 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:07:17.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:17.433+0000 7f09bde00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f09c01028b0 0x7f09c0197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:17.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:17.434+0000 7f09bf200640 1 -- 172.21.15.39:0/1968967381 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f09c01016c0 msgr2=0x7f09c0197a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:17.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:17.434+0000 7f09bf200640 1 --2- 172.21.15.39:0/1968967381 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f09c01016c0 0x7f09c0197a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:17.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:17.434+0000 7f09bf200640 1 -- 172.21.15.39:0/1968967381 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f09c01028b0 msgr2=0x7f09c0197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:17.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:17.434+0000 7f09bf200640 1 --2- 172.21.15.39:0/1968967381 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f09c01028b0 0x7f09c0197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:17.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:17.434+0000 7f09bf200640 1 -- 172.21.15.39:0/1968967381 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09c019f650 con 0x7f09c0103270 2024-03-18T16:07:17.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:17.434+0000 7f09bf200640 1 --2- 172.21.15.39:0/1968967381 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f09c0103270 0x7f09c019efe0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f09b8009fd0 tx=0x7f09b800ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:17.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:17.434+0000 7f09b7400640 1 -- 172.21.15.39:0/1968967381 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09b800ce20 con 0x7f09c0103270 2024-03-18T16:07:17.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:17.434+0000 7f09b7400640 1 -- 172.21.15.39:0/1968967381 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09b8005020 con 0x7f09c0103270 2024-03-18T16:07:17.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:17.434+0000 7f09c4a00640 1 -- 172.21.15.39:0/1968967381 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09c019f940 con 0x7f09c0103270 2024-03-18T16:07:17.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:17.434+0000 7f09b7400640 1 -- 172.21.15.39:0/1968967381 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09b8010720 con 0x7f09c0103270 2024-03-18T16:07:17.440 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:17.435+0000 7f09c4a00640 1 -- 172.21.15.39:0/1968967381 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09c019fe90 con 0x7f09c0103270 2024-03-18T16:07:17.440 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:17.436+0000 7f09c4a00640 1 -- 172.21.15.39:0/1968967381 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09c01089e0 con 0x7f09c0103270 2024-03-18T16:07:17.440 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:17.438+0000 7f09b7400640 1 -- 172.21.15.39:0/1968967381 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f09b8005540 con 0x7f09c0103270 2024-03-18T16:07:17.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:17.439+0000 7f09b7400640 1 --2- 172.21.15.39:0/1968967381 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f09a40a2880 0x7f09a40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:17.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:17.439+0000 7f09b7400640 1 -- 172.21.15.39:0/1968967381 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f09b80cf2b0 con 0x7f09c0103270 2024-03-18T16:07:17.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:17.439+0000 7f09be800640 1 --2- 172.21.15.39:0/1968967381 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f09a40a2880 0x7f09a40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:17.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:17.440+0000 7f09be800640 1 --2- 172.21.15.39:0/1968967381 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f09a40a2880 0x7f09a40a4d40 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f09b002c900 tx=0x7f09b0005b50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:17.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:17.443+0000 7f09b7400640 1 -- 172.21.15.39:0/1968967381 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f09b809c0b0 con 0x7f09c0103270 2024-03-18T16:07:17.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:17 smithi196 ceph-mon[36307]: pgmap v334: 129 pgs: 129 active+clean; 581 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-18T16:07:17.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:17 smithi196 ceph-mon[36307]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-18T16:07:17.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:17 smithi196 ceph-mon[36307]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-18T16:07:17.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:17 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/3802534630' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-18T16:07:17.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:17 smithi196 ceph-mon[36307]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-18T16:07:17.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:17 smithi194 ceph-mon[36200]: pgmap v334: 129 pgs: 129 active+clean; 581 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-18T16:07:17.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:17 smithi194 ceph-mon[36200]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-18T16:07:17.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:17 smithi194 ceph-mon[36200]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-18T16:07:17.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:17 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/3802534630' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-18T16:07:17.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:17 smithi194 ceph-mon[36200]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-18T16:07:17.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:17 smithi039 ceph-mon[31721]: pgmap v334: 129 pgs: 129 active+clean; 581 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-18T16:07:17.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:17 smithi039 ceph-mon[31721]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-18T16:07:17.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:17 smithi039 ceph-mon[31721]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-18T16:07:17.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:17 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/3802534630' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-18T16:07:17.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:17 smithi039 ceph-mon[31721]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-18T16:07:17.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:17.688+0000 7f09c4a00640 1 -- 172.21.15.39:0/1968967381 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7f09c002d1b0 con 0x7f09c0103270 2024-03-18T16:07:18.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:18.208+0000 7f09b7400640 1 -- 172.21.15.39:0/1968967381 <== mon.2 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v20) ==== 121+0+0 (secure 0 0 0) 0x7f09b809b9b0 con 0x7f09c0103270 2024-03-18T16:07:18.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:18.211+0000 7f09c4a00640 1 -- 172.21.15.39:0/1968967381 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f09a40a2880 msgr2=0x7f09a40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:18.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:18.211+0000 7f09c4a00640 1 --2- 172.21.15.39:0/1968967381 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f09a40a2880 0x7f09a40a4d40 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f09b002c900 tx=0x7f09b0005b50 comp rx=0 tx=0).stop 2024-03-18T16:07:18.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:18.211+0000 7f09c4a00640 1 -- 172.21.15.39:0/1968967381 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f09c0103270 msgr2=0x7f09c019efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:18.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:18.211+0000 7f09c4a00640 1 --2- 172.21.15.39:0/1968967381 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f09c0103270 0x7f09c019efe0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f09b8009fd0 tx=0x7f09b800ef10 comp rx=0 tx=0).stop 2024-03-18T16:07:18.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:18.211+0000 7f09c4a00640 1 -- 172.21.15.39:0/1968967381 shutdown_connections 2024-03-18T16:07:18.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:18.211+0000 7f09c4a00640 1 --2- 172.21.15.39:0/1968967381 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f09c0103270 0x7f09c019efe0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:18.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:18.211+0000 7f09c4a00640 1 --2- 172.21.15.39:0/1968967381 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f09a40a2880 0x7f09a40a4d40 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:18.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:18.211+0000 7f09c4a00640 1 --2- 172.21.15.39:0/1968967381 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f09c01028b0 0x7f09c0197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:18.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:18.211+0000 7f09c4a00640 1 --2- 172.21.15.39:0/1968967381 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f09c01016c0 0x7f09c0197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:18.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:18.211+0000 7f09c4a00640 1 -- 172.21.15.39:0/1968967381 >> 172.21.15.39:0/1968967381 conn(0x7f09c00fcc50 msgr2=0x7f09c0105bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:18.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:18.211+0000 7f09c4a00640 1 -- 172.21.15.39:0/1968967381 shutdown_connections 2024-03-18T16:07:18.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:18.211+0000 7f09c4a00640 1 -- 172.21.15.39:0/1968967381 wait complete. 2024-03-18T16:07:18.240 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-03-18T16:07:18.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:18.439+0000 7f3f3ba00640 1 -- 172.21.15.39:0/1827286956 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3f34104c70 msgr2=0x7f3f34107060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:18.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:18.439+0000 7f3f3ba00640 1 --2- 172.21.15.39:0/1827286956 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3f34104c70 0x7f3f34107060 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f3f1c009be0 tx=0x7f3f1c02bef0 comp rx=0 tx=0).stop 2024-03-18T16:07:18.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:18.439+0000 7f3f3ba00640 1 -- 172.21.15.39:0/1827286956 shutdown_connections 2024-03-18T16:07:18.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:18.439+0000 7f3f3ba00640 1 --2- 172.21.15.39:0/1827286956 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3f34104c70 0x7f3f34107060 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:18.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:18.439+0000 7f3f3ba00640 1 --2- 172.21.15.39:0/1827286956 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3f341075a0 0x7f3f34109990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:18.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:18.439+0000 7f3f3ba00640 1 --2- 172.21.15.39:0/1827286956 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3f34102340 0x7f3f34104730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:18.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:18.439+0000 7f3f3ba00640 1 -- 172.21.15.39:0/1827286956 >> 172.21.15.39:0/1827286956 conn(0x7f3f340fbc70 msgr2=0x7f3f340fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:18.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:18.439+0000 7f3f3ba00640 1 -- 172.21.15.39:0/1827286956 shutdown_connections 2024-03-18T16:07:18.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:18.439+0000 7f3f3ba00640 1 -- 172.21.15.39:0/1827286956 wait complete. 2024-03-18T16:07:18.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:18.440+0000 7f3f3ba00640 1 Processor -- start 2024-03-18T16:07:18.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:18.441+0000 7f3f3ba00640 1 -- start start 2024-03-18T16:07:18.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:18.441+0000 7f3f3ba00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3f34102340 0x7f3f34101cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:18.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:18.441+0000 7f3f3ba00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3f34104c70 0x7f3f341002f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:18.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:18.441+0000 7f3f3ba00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3f341075a0 0x7f3f34100830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:18.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:18.441+0000 7f3f3ba00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f3f3410d4b0 con 0x7f3f34102340 2024-03-18T16:07:18.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:18.441+0000 7f3f3ba00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f3f3410d330 con 0x7f3f341075a0 2024-03-18T16:07:18.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:18.441+0000 7f3f3ba00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f3f3410d630 con 0x7f3f34104c70 2024-03-18T16:07:18.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:18.441+0000 7f3f33e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3f34104c70 0x7f3f341002f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:18.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:18.441+0000 7f3f33e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3f34104c70 0x7f3f341002f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:60818/0 (socket says 172.21.15.39:60818) 2024-03-18T16:07:18.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:18.441+0000 7f3f33e00640 1 -- 172.21.15.39:0/3955502210 learned_addr learned my addr 172.21.15.39:0/3955502210 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:07:18.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:18.441+0000 7f3f39a00640 1 --2- 172.21.15.39:0/3955502210 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3f341075a0 0x7f3f34100830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:18.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:18.441+0000 7f3f39000640 1 --2- 172.21.15.39:0/3955502210 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3f34102340 0x7f3f34101cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:18.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:18.442+0000 7f3f33e00640 1 -- 172.21.15.39:0/3955502210 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3f341075a0 msgr2=0x7f3f34100830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:18.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:18.442+0000 7f3f33e00640 1 --2- 172.21.15.39:0/3955502210 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3f341075a0 0x7f3f34100830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:18.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:18.442+0000 7f3f33e00640 1 -- 172.21.15.39:0/3955502210 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3f34102340 msgr2=0x7f3f34101cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:18.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:18.442+0000 7f3f33e00640 1 --2- 172.21.15.39:0/3955502210 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3f34102340 0x7f3f34101cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:18.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:18.442+0000 7f3f33e00640 1 -- 172.21.15.39:0/3955502210 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f341010c0 con 0x7f3f34104c70 2024-03-18T16:07:18.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:18.442+0000 7f3f39000640 1 --2- 172.21.15.39:0/3955502210 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3f34102340 0x7f3f34101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:07:18.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:18.442+0000 7f3f39a00640 1 --2- 172.21.15.39:0/3955502210 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3f341075a0 0x7f3f34100830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:07:18.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:18.442+0000 7f3f33e00640 1 --2- 172.21.15.39:0/3955502210 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3f34104c70 0x7f3f341002f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3f1c02c500 tx=0x7f3f1c0337b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:18.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:18.442+0000 7f3f31600640 1 -- 172.21.15.39:0/3955502210 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f1c043020 con 0x7f3f34104c70 2024-03-18T16:07:18.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:18.442+0000 7f3f3ba00640 1 -- 172.21.15.39:0/3955502210 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f341a1cb0 con 0x7f3f34104c70 2024-03-18T16:07:18.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:18.442+0000 7f3f31600640 1 -- 172.21.15.39:0/3955502210 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f1c003b80 con 0x7f3f34104c70 2024-03-18T16:07:18.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:18.442+0000 7f3f31600640 1 -- 172.21.15.39:0/3955502210 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f1c03e7b0 con 0x7f3f34104c70 2024-03-18T16:07:18.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:18.442+0000 7f3f3ba00640 1 -- 172.21.15.39:0/3955502210 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f341a21a0 con 0x7f3f34104c70 2024-03-18T16:07:18.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:18.444+0000 7f3f3ba00640 1 -- 172.21.15.39:0/3955502210 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f340fd760 con 0x7f3f34104c70 2024-03-18T16:07:18.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:18.445+0000 7f3f31600640 1 -- 172.21.15.39:0/3955502210 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f3f1c046020 con 0x7f3f34104c70 2024-03-18T16:07:18.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:18.447+0000 7f3f31600640 1 --2- 172.21.15.39:0/3955502210 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3f140a2800 0x7f3f140a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:18.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:18.447+0000 7f3f31600640 1 -- 172.21.15.39:0/3955502210 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f3f1c0f1440 con 0x7f3f34104c70 2024-03-18T16:07:18.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:18.447+0000 7f3f39000640 1 --2- 172.21.15.39:0/3955502210 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3f140a2800 0x7f3f140a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:18.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:18.448+0000 7f3f39000640 1 --2- 172.21.15.39:0/3955502210 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3f140a2800 0x7f3f140a4cc0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f3f24005fd0 tx=0x7f3f24009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:18.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:18.456+0000 7f3f31600640 1 -- 172.21.15.39:0/3955502210 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3f1c04d020 con 0x7f3f34104c70 2024-03-18T16:07:18.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:18 smithi196 ceph-mon[36307]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-18T16:07:18.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:18 smithi196 ceph-mon[36307]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-03-18T16:07:18.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:18 smithi196 ceph-mon[36307]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-18T16:07:18.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:18 smithi196 ceph-mon[36307]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-18T16:07:18.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:18 smithi196 ceph-mon[36307]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-03-18T16:07:18.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:18 smithi196 ceph-mon[36307]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-18T16:07:18.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:18 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/1968967381' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-18T16:07:18.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:18 smithi196 ceph-mon[36307]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-18T16:07:18.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:18 smithi194 ceph-mon[36200]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-18T16:07:18.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:18 smithi194 ceph-mon[36200]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-03-18T16:07:18.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:18 smithi194 ceph-mon[36200]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-18T16:07:18.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:18 smithi194 ceph-mon[36200]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-18T16:07:18.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:18 smithi194 ceph-mon[36200]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-03-18T16:07:18.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:18 smithi194 ceph-mon[36200]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-18T16:07:18.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:18 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/1968967381' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-18T16:07:18.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:18 smithi194 ceph-mon[36200]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-18T16:07:18.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:18 smithi039 ceph-mon[31721]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-18T16:07:18.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:18 smithi039 ceph-mon[31721]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-03-18T16:07:18.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:18 smithi039 ceph-mon[31721]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-18T16:07:18.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:18 smithi039 ceph-mon[31721]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-18T16:07:18.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:18 smithi039 ceph-mon[31721]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-03-18T16:07:18.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:18 smithi039 ceph-mon[31721]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-03-18T16:07:18.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:18 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/1968967381' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-18T16:07:18.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:18 smithi039 ceph-mon[31721]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-18T16:07:18.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:18.691+0000 7f3f3ba00640 1 -- 172.21.15.39:0/3955502210 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7f3f340623e0 con 0x7f3f34104c70 2024-03-18T16:07:18.990 INFO:journalctl@ceph.mds.b.smithi194.stdout:Mar 18 16:07:18 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-b[67287]: 2024-03-18T16:07:18.582+0000 7fa042c70700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-18T16:07:19.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:19.224+0000 7f3f31600640 1 -- 172.21.15.39:0/3955502210 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v22) ==== 115+0+0 (secure 0 0 0) 0x7f3f1c0be240 con 0x7f3f34104c70 2024-03-18T16:07:19.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:19.227+0000 7f3f3ba00640 1 -- 172.21.15.39:0/3955502210 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3f140a2800 msgr2=0x7f3f140a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:19.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:19.227+0000 7f3f3ba00640 1 --2- 172.21.15.39:0/3955502210 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3f140a2800 0x7f3f140a4cc0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f3f24005fd0 tx=0x7f3f24009260 comp rx=0 tx=0).stop 2024-03-18T16:07:19.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:19.227+0000 7f3f3ba00640 1 -- 172.21.15.39:0/3955502210 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3f34104c70 msgr2=0x7f3f341002f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:19.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:19.227+0000 7f3f3ba00640 1 --2- 172.21.15.39:0/3955502210 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3f34104c70 0x7f3f341002f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3f1c02c500 tx=0x7f3f1c0337b0 comp rx=0 tx=0).stop 2024-03-18T16:07:19.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:19.227+0000 7f3f3ba00640 1 -- 172.21.15.39:0/3955502210 shutdown_connections 2024-03-18T16:07:19.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:19.227+0000 7f3f3ba00640 1 --2- 172.21.15.39:0/3955502210 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3f341075a0 0x7f3f34100830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:19.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:19.228+0000 7f3f3ba00640 1 --2- 172.21.15.39:0/3955502210 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3f140a2800 0x7f3f140a4cc0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:19.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:19.228+0000 7f3f3ba00640 1 --2- 172.21.15.39:0/3955502210 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3f34104c70 0x7f3f341002f0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:19.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:19.228+0000 7f3f3ba00640 1 --2- 172.21.15.39:0/3955502210 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3f34102340 0x7f3f34101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:19.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:19.228+0000 7f3f3ba00640 1 -- 172.21.15.39:0/3955502210 >> 172.21.15.39:0/3955502210 conn(0x7f3f340fbc70 msgr2=0x7f3f34102dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:19.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:19.228+0000 7f3f3ba00640 1 -- 172.21.15.39:0/3955502210 shutdown_connections 2024-03-18T16:07:19.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:19.228+0000 7f3f3ba00640 1 -- 172.21.15.39:0/3955502210 wait complete. 2024-03-18T16:07:19.255 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2} subvols for filesystem 'cephfs' 2024-03-18T16:07:19.256 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-03-18T16:07:19.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:19.450+0000 7f76ed600640 1 -- 172.21.15.39:0/1279254558 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f76e81055d0 msgr2=0x7f76e810be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:19.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:19.450+0000 7f76ed600640 1 --2- 172.21.15.39:0/1279254558 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f76e81055d0 0x7f76e810be60 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f76e0009900 tx=0x7f76e002bc30 comp rx=0 tx=0).stop 2024-03-18T16:07:19.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:19.450+0000 7f76ed600640 1 -- 172.21.15.39:0/1279254558 shutdown_connections 2024-03-18T16:07:19.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:19.450+0000 7f76ed600640 1 --2- 172.21.15.39:0/1279254558 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f76e8104c10 0x7f76e8105090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:19.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:19.450+0000 7f76ed600640 1 --2- 172.21.15.39:0/1279254558 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f76e81055d0 0x7f76e810be60 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:19.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:19.450+0000 7f76ed600640 1 --2- 172.21.15.39:0/1279254558 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f76e8103a20 0x7f76e8103e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:19.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:19.450+0000 7f76ed600640 1 -- 172.21.15.39:0/1279254558 >> 172.21.15.39:0/1279254558 conn(0x7f76e80fef90 msgr2=0x7f76e81013d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:19.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:19.450+0000 7f76ed600640 1 -- 172.21.15.39:0/1279254558 shutdown_connections 2024-03-18T16:07:19.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:19.450+0000 7f76ed600640 1 -- 172.21.15.39:0/1279254558 wait complete. 2024-03-18T16:07:19.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:19.452+0000 7f76ed600640 1 Processor -- start 2024-03-18T16:07:19.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:19.452+0000 7f76ed600640 1 -- start start 2024-03-18T16:07:19.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:19.452+0000 7f76ed600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f76e8103a20 0x7f76e8199cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:19.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:19.452+0000 7f76ed600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f76e8104c10 0x7f76e819a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:19.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:19.452+0000 7f76ed600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f76e81055d0 0x7f76e81a1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:19.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:19.452+0000 7f76ed600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f76e810f6d0 con 0x7f76e8103a20 2024-03-18T16:07:19.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:19.452+0000 7f76ed600640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f76e810f550 con 0x7f76e8104c10 2024-03-18T16:07:19.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:19.452+0000 7f76ed600640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f76e810f850 con 0x7f76e81055d0 2024-03-18T16:07:19.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:19.452+0000 7f76e6a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f76e8103a20 0x7f76e8199cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:19.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:19.453+0000 7f76e6000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f76e8104c10 0x7f76e819a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:19.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:19.453+0000 7f76e6000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f76e8104c10 0x7f76e819a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:56368/0 (socket says 172.21.15.39:56368) 2024-03-18T16:07:19.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:19.453+0000 7f76e6000640 1 -- 172.21.15.39:0/4055695335 learned_addr learned my addr 172.21.15.39:0/4055695335 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:07:19.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:19.453+0000 7f76e7400640 1 --2- 172.21.15.39:0/4055695335 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f76e81055d0 0x7f76e81a1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:19.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:19.453+0000 7f76e6000640 1 -- 172.21.15.39:0/4055695335 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f76e81055d0 msgr2=0x7f76e81a1250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:19.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:19.453+0000 7f76e6000640 1 --2- 172.21.15.39:0/4055695335 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f76e81055d0 0x7f76e81a1250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:19.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:19.453+0000 7f76e6000640 1 -- 172.21.15.39:0/4055695335 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f76e8103a20 msgr2=0x7f76e8199cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:19.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:19.453+0000 7f76e6000640 1 --2- 172.21.15.39:0/4055695335 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f76e8103a20 0x7f76e8199cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:19.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:19.453+0000 7f76e6000640 1 -- 172.21.15.39:0/4055695335 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76e81a18f0 con 0x7f76e8104c10 2024-03-18T16:07:19.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:19.453+0000 7f76e6000640 1 --2- 172.21.15.39:0/4055695335 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f76e8104c10 0x7f76e819a1f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f76d000ea60 tx=0x7f76d000ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:19.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:19.453+0000 7f76df400640 1 -- 172.21.15.39:0/4055695335 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76d000cde0 con 0x7f76e8104c10 2024-03-18T16:07:19.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:19.454+0000 7f76ed600640 1 -- 172.21.15.39:0/4055695335 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76e81a1be0 con 0x7f76e8104c10 2024-03-18T16:07:19.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:19.454+0000 7f76df400640 1 -- 172.21.15.39:0/4055695335 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76d00035f0 con 0x7f76e8104c10 2024-03-18T16:07:19.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:19.454+0000 7f76df400640 1 -- 172.21.15.39:0/4055695335 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76d0010670 con 0x7f76e8104c10 2024-03-18T16:07:19.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:19.454+0000 7f76ed600640 1 -- 172.21.15.39:0/4055695335 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76e81a2180 con 0x7f76e8104c10 2024-03-18T16:07:19.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:19.455+0000 7f76ed600640 1 -- 172.21.15.39:0/4055695335 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76e810ace0 con 0x7f76e8104c10 2024-03-18T16:07:19.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:19.456+0000 7f76df400640 1 -- 172.21.15.39:0/4055695335 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f76d00037a0 con 0x7f76e8104c10 2024-03-18T16:07:19.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:19.458+0000 7f76df400640 1 --2- 172.21.15.39:0/4055695335 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f76cc0a2800 0x7f76cc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:19.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:19.458+0000 7f76df400640 1 -- 172.21.15.39:0/4055695335 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f76d00cefe0 con 0x7f76e8104c10 2024-03-18T16:07:19.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:19.458+0000 7f76e6a00640 1 --2- 172.21.15.39:0/4055695335 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f76cc0a2800 0x7f76cc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:19.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:19.459+0000 7f76e6a00640 1 --2- 172.21.15.39:0/4055695335 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f76cc0a2800 0x7f76cc0a4cc0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f76e819ab80 tx=0x7f76d8009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:19.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:19.462+0000 7f76df400640 1 -- 172.21.15.39:0/4055695335 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f76d009bde0 con 0x7f76e8104c10 2024-03-18T16:07:19.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:19 smithi196 ceph-mon[36307]: pgmap v335: 129 pgs: 129 active+clean; 581 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-03-18T16:07:19.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:19 smithi196 ceph-mon[36307]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-18T16:07:19.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:19 smithi196 ceph-mon[36307]: mds.? [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] up:active 2024-03-18T16:07:19.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:19 smithi196 ceph-mon[36307]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-18T16:07:19.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:19 smithi196 ceph-mon[36307]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-03-18T16:07:19.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:19 smithi196 ceph-mon[36307]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2024-03-18T16:07:19.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:19 smithi196 ceph-mon[36307]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-03-18T16:07:19.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:19 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/3955502210' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-18T16:07:19.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:19 smithi196 ceph-mon[36307]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-18T16:07:19.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:19 smithi194 ceph-mon[36200]: pgmap v335: 129 pgs: 129 active+clean; 581 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-03-18T16:07:19.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:19 smithi194 ceph-mon[36200]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-18T16:07:19.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:19 smithi194 ceph-mon[36200]: mds.? [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] up:active 2024-03-18T16:07:19.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:19 smithi194 ceph-mon[36200]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-18T16:07:19.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:19 smithi194 ceph-mon[36200]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-03-18T16:07:19.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:19 smithi194 ceph-mon[36200]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2024-03-18T16:07:19.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:19 smithi194 ceph-mon[36200]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-03-18T16:07:19.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:19 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/3955502210' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-18T16:07:19.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:19 smithi194 ceph-mon[36200]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-18T16:07:19.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:19 smithi039 ceph-mon[31721]: pgmap v335: 129 pgs: 129 active+clean; 581 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-03-18T16:07:19.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:19 smithi039 ceph-mon[31721]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-18T16:07:19.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:19 smithi039 ceph-mon[31721]: mds.? [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] up:active 2024-03-18T16:07:19.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:19 smithi039 ceph-mon[31721]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-18T16:07:19.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:19 smithi039 ceph-mon[31721]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-03-18T16:07:19.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:19 smithi039 ceph-mon[31721]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:creating} 9 up:standby 2024-03-18T16:07:19.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:19 smithi039 ceph-mon[31721]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-03-18T16:07:19.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:19 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/3955502210' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-18T16:07:19.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:19 smithi039 ceph-mon[31721]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-18T16:07:19.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:19.694+0000 7f76ed600640 1 -- 172.21.15.39:0/4055695335 --> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f76e8061c20 con 0x7f76cc0a2800 2024-03-18T16:07:19.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:19.996+0000 7f76df400640 1 -- 172.21.15.39:0/4055695335 <== mgr.14152 v2:172.21.15.39:6800/2749214286 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f76e8061c20 con 0x7f76cc0a2800 2024-03-18T16:07:20.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.000+0000 7f76ed600640 1 -- 172.21.15.39:0/4055695335 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f76cc0a2800 msgr2=0x7f76cc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:20.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.000+0000 7f76ed600640 1 --2- 172.21.15.39:0/4055695335 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f76cc0a2800 0x7f76cc0a4cc0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f76e819ab80 tx=0x7f76d8009500 comp rx=0 tx=0).stop 2024-03-18T16:07:20.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.000+0000 7f76ed600640 1 -- 172.21.15.39:0/4055695335 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f76e8104c10 msgr2=0x7f76e819a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:20.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.000+0000 7f76ed600640 1 --2- 172.21.15.39:0/4055695335 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f76e8104c10 0x7f76e819a1f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f76d000ea60 tx=0x7f76d000ef20 comp rx=0 tx=0).stop 2024-03-18T16:07:20.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.000+0000 7f76ed600640 1 -- 172.21.15.39:0/4055695335 shutdown_connections 2024-03-18T16:07:20.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.000+0000 7f76ed600640 1 --2- 172.21.15.39:0/4055695335 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f76e8104c10 0x7f76e819a1f0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:20.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.000+0000 7f76ed600640 1 --2- 172.21.15.39:0/4055695335 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f76cc0a2800 0x7f76cc0a4cc0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:20.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.000+0000 7f76ed600640 1 --2- 172.21.15.39:0/4055695335 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f76e81055d0 0x7f76e81a1250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:20.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.000+0000 7f76ed600640 1 --2- 172.21.15.39:0/4055695335 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f76e8103a20 0x7f76e8199cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:20.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.000+0000 7f76ed600640 1 -- 172.21.15.39:0/4055695335 >> 172.21.15.39:0/4055695335 conn(0x7f76e80fef90 msgr2=0x7f76e8107eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:20.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.001+0000 7f76ed600640 1 -- 172.21.15.39:0/4055695335 shutdown_connections 2024-03-18T16:07:20.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.001+0000 7f76ed600640 1 -- 172.21.15.39:0/4055695335 wait complete. 2024-03-18T16:07:20.025 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 16:07:19 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T16:07:19.704+0000 7f6e7c400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T16:07:20.025 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 16:07:19 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T16:07:19.704+0000 7f6e7c400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T16:07:20.025 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 16:07:19 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T16:07:19.704+0000 7f6e7c400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T16:07:20.025 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 16:07:19 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T16:07:19.704+0000 7f6e7c400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T16:07:20.025 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 16:07:19 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T16:07:19.704+0000 7f6e7c400700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-18T16:07:20.027 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-03-18T16:07:20.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.230+0000 7f3f05200640 1 -- 172.21.15.39:0/1006327264 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3f00077d80 msgr2=0x7f3f00078620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:20.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.230+0000 7f3f05200640 1 --2- 172.21.15.39:0/1006327264 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3f00077d80 0x7f3f00078620 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3ef8009ca0 tx=0x7f3ef802c100 comp rx=0 tx=0).stop 2024-03-18T16:07:20.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.230+0000 7f3f05200640 1 -- 172.21.15.39:0/1006327264 shutdown_connections 2024-03-18T16:07:20.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.230+0000 7f3f05200640 1 --2- 172.21.15.39:0/1006327264 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3f000773e0 0x7f3f00077840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:20.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.230+0000 7f3f05200640 1 --2- 172.21.15.39:0/1006327264 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3f00077d80 0x7f3f00078620 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:20.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.230+0000 7f3f05200640 1 --2- 172.21.15.39:0/1006327264 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3f000761a0 0x7f3f000765c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:20.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.230+0000 7f3f05200640 1 -- 172.21.15.39:0/1006327264 >> 172.21.15.39:0/1006327264 conn(0x7f3f000fef90 msgr2=0x7f3f001013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:20.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.230+0000 7f3f05200640 1 -- 172.21.15.39:0/1006327264 shutdown_connections 2024-03-18T16:07:20.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.230+0000 7f3f05200640 1 -- 172.21.15.39:0/1006327264 wait complete. 2024-03-18T16:07:20.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.231+0000 7f3f05200640 1 Processor -- start 2024-03-18T16:07:20.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.231+0000 7f3f05200640 1 -- start start 2024-03-18T16:07:20.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.231+0000 7f3f05200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3f000761a0 0x7f3f0019e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:20.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.231+0000 7f3f05200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3f000773e0 0x7f3f0019e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:20.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.231+0000 7f3f05200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3f00077d80 0x7f3f001a5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:20.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.231+0000 7f3f05200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f3f00113a90 con 0x7f3f000773e0 2024-03-18T16:07:20.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.231+0000 7f3f05200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f3f00113910 con 0x7f3f000761a0 2024-03-18T16:07:20.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.231+0000 7f3f05200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f3f00113c10 con 0x7f3f00077d80 2024-03-18T16:07:20.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.232+0000 7f3efde00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3f000761a0 0x7f3f0019e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:20.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.232+0000 7f3efd400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3f000773e0 0x7f3f0019e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:20.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.232+0000 7f3efe800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3f00077d80 0x7f3f001a5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:20.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.232+0000 7f3efde00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3f000761a0 0x7f3f0019e0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:56410/0 (socket says 172.21.15.39:56410) 2024-03-18T16:07:20.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.232+0000 7f3efde00640 1 -- 172.21.15.39:0/380584082 learned_addr learned my addr 172.21.15.39:0/380584082 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:07:20.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.232+0000 7f3efde00640 1 -- 172.21.15.39:0/380584082 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3f00077d80 msgr2=0x7f3f001a5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:20.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.232+0000 7f3efde00640 1 --2- 172.21.15.39:0/380584082 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3f00077d80 0x7f3f001a5670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:20.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.232+0000 7f3efde00640 1 -- 172.21.15.39:0/380584082 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3f000773e0 msgr2=0x7f3f0019e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:20.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.232+0000 7f3efde00640 1 --2- 172.21.15.39:0/380584082 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3f000773e0 0x7f3f0019e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:20.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.232+0000 7f3efde00640 1 -- 172.21.15.39:0/380584082 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f001a5ce0 con 0x7f3f000761a0 2024-03-18T16:07:20.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.232+0000 7f3efe800640 1 --2- 172.21.15.39:0/380584082 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3f00077d80 0x7f3f001a5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:07:20.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.233+0000 7f3efde00640 1 --2- 172.21.15.39:0/380584082 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3f000761a0 0x7f3f0019e0c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f3ef000e9f0 tx=0x7f3ef000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:20.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.234+0000 7f3ef6a00640 1 -- 172.21.15.39:0/380584082 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ef000cd80 con 0x7f3f000761a0 2024-03-18T16:07:20.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.234+0000 7f3ef6a00640 1 -- 172.21.15.39:0/380584082 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ef0010910 con 0x7f3f000761a0 2024-03-18T16:07:20.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.234+0000 7f3ef6a00640 1 -- 172.21.15.39:0/380584082 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ef0018990 con 0x7f3f000761a0 2024-03-18T16:07:20.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.234+0000 7f3f05200640 1 -- 172.21.15.39:0/380584082 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f001a5fd0 con 0x7f3f000761a0 2024-03-18T16:07:20.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.235+0000 7f3f05200640 1 -- 172.21.15.39:0/380584082 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f001a63e0 con 0x7f3f000761a0 2024-03-18T16:07:20.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.236+0000 7f3f05200640 1 -- 172.21.15.39:0/380584082 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f00110370 con 0x7f3f000761a0 2024-03-18T16:07:20.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.238+0000 7f3ef6a00640 1 -- 172.21.15.39:0/380584082 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f3ef0018b30 con 0x7f3f000761a0 2024-03-18T16:07:20.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.239+0000 7f3ef6a00640 1 --2- 172.21.15.39:0/380584082 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3ee40a2880 0x7f3ee40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:20.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.239+0000 7f3ef6a00640 1 -- 172.21.15.39:0/380584082 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f3ef00cef70 con 0x7f3f000761a0 2024-03-18T16:07:20.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.239+0000 7f3efd400640 1 --2- 172.21.15.39:0/380584082 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3ee40a2880 0x7f3ee40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:20.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.240+0000 7f3efd400640 1 --2- 172.21.15.39:0/380584082 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3ee40a2880 0x7f3ee40a4d40 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f3f0019f5d0 tx=0x7f3eec005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:20.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.243+0000 7f3ef6a00640 1 -- 172.21.15.39:0/380584082 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3ef009b720 con 0x7f3f000761a0 2024-03-18T16:07:20.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:20 smithi039 ceph-mon[31721]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-18T16:07:20.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:20 smithi039 ceph-mon[31721]: mds.? [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] up:active 2024-03-18T16:07:20.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:20 smithi039 ceph-mon[31721]: daemon mds.h assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-03-18T16:07:20.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:20 smithi039 ceph-mon[31721]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 9 up:standby 2024-03-18T16:07:20.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:20 smithi039 ceph-mon[31721]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=h=up:creating} 8 up:standby 2024-03-18T16:07:20.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:20 smithi039 ceph-mon[31721]: daemon mds.h is now active in filesystem cephfs as rank 3 2024-03-18T16:07:20.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:20 smithi039 ceph-mon[31721]: mds.? [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] up:active 2024-03-18T16:07:20.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:20 smithi039 ceph-mon[31721]: daemon mds.a assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-03-18T16:07:20.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:20 smithi039 ceph-mon[31721]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-18T16:07:20.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:20 smithi039 ceph-mon[31721]: Cluster is now healthy 2024-03-18T16:07:20.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:20 smithi039 ceph-mon[31721]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=h=up:active} 4 up:standby-replay 4 up:standby 2024-03-18T16:07:20.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:20 smithi039 ceph-mon[31721]: fsmap cephfs:5 4 up:standby-replay 1 up:creating 3 up:standby 4 up:active 2024-03-18T16:07:20.339 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:20 smithi039 ceph-mon[31721]: daemon mds.a is now active in filesystem cephfs as rank 4 2024-03-18T16:07:20.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:20 smithi196 ceph-mon[36307]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-18T16:07:20.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:20 smithi196 ceph-mon[36307]: mds.? [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] up:active 2024-03-18T16:07:20.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:20 smithi196 ceph-mon[36307]: daemon mds.h assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-03-18T16:07:20.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:20 smithi196 ceph-mon[36307]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 9 up:standby 2024-03-18T16:07:20.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:20 smithi196 ceph-mon[36307]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=h=up:creating} 8 up:standby 2024-03-18T16:07:20.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:20 smithi196 ceph-mon[36307]: daemon mds.h is now active in filesystem cephfs as rank 3 2024-03-18T16:07:20.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:20 smithi196 ceph-mon[36307]: mds.? [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] up:active 2024-03-18T16:07:20.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:20 smithi196 ceph-mon[36307]: daemon mds.a assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-03-18T16:07:20.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:20 smithi196 ceph-mon[36307]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-18T16:07:20.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:20 smithi196 ceph-mon[36307]: Cluster is now healthy 2024-03-18T16:07:20.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:20 smithi196 ceph-mon[36307]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=h=up:active} 4 up:standby-replay 4 up:standby 2024-03-18T16:07:20.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:20 smithi196 ceph-mon[36307]: fsmap cephfs:5 4 up:standby-replay 1 up:creating 3 up:standby 4 up:active 2024-03-18T16:07:20.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:20 smithi196 ceph-mon[36307]: daemon mds.a is now active in filesystem cephfs as rank 4 2024-03-18T16:07:20.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.477+0000 7f3f05200640 1 -- 172.21.15.39:0/380584082 --> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f3f00061cb0 con 0x7f3ee40a2880 2024-03-18T16:07:20.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:20 smithi194 ceph-mon[36200]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-18T16:07:20.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:20 smithi194 ceph-mon[36200]: mds.? [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] up:active 2024-03-18T16:07:20.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:20 smithi194 ceph-mon[36200]: daemon mds.h assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-03-18T16:07:20.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:20 smithi194 ceph-mon[36200]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=c=up:active} 9 up:standby 2024-03-18T16:07:20.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:20 smithi194 ceph-mon[36200]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=h=up:creating} 8 up:standby 2024-03-18T16:07:20.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:20 smithi194 ceph-mon[36200]: daemon mds.h is now active in filesystem cephfs as rank 3 2024-03-18T16:07:20.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:20 smithi194 ceph-mon[36200]: mds.? [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] up:active 2024-03-18T16:07:20.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:20 smithi194 ceph-mon[36200]: daemon mds.a assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-03-18T16:07:20.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:20 smithi194 ceph-mon[36200]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-18T16:07:20.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:20 smithi194 ceph-mon[36200]: Cluster is now healthy 2024-03-18T16:07:20.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:20 smithi194 ceph-mon[36200]: fsmap cephfs:4 {0=b=up:active,1=i=up:active,2=c=up:active,3=h=up:active} 4 up:standby-replay 4 up:standby 2024-03-18T16:07:20.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:20 smithi194 ceph-mon[36200]: fsmap cephfs:5 4 up:standby-replay 1 up:creating 3 up:standby 4 up:active 2024-03-18T16:07:20.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:20 smithi194 ceph-mon[36200]: daemon mds.a is now active in filesystem cephfs as rank 4 2024-03-18T16:07:20.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.506+0000 7f3ef6a00640 1 -- 172.21.15.39:0/380584082 <== mgr.14152 v2:172.21.15.39:6800/2749214286 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f3f00061cb0 con 0x7f3ee40a2880 2024-03-18T16:07:20.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.509+0000 7f3f05200640 1 -- 172.21.15.39:0/380584082 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3ee40a2880 msgr2=0x7f3ee40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:20.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.509+0000 7f3f05200640 1 --2- 172.21.15.39:0/380584082 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3ee40a2880 0x7f3ee40a4d40 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f3f0019f5d0 tx=0x7f3eec005f90 comp rx=0 tx=0).stop 2024-03-18T16:07:20.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.509+0000 7f3f05200640 1 -- 172.21.15.39:0/380584082 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3f000761a0 msgr2=0x7f3f0019e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:20.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.509+0000 7f3f05200640 1 --2- 172.21.15.39:0/380584082 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3f000761a0 0x7f3f0019e0c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f3ef000e9f0 tx=0x7f3ef000eeb0 comp rx=0 tx=0).stop 2024-03-18T16:07:20.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.509+0000 7f3f05200640 1 -- 172.21.15.39:0/380584082 shutdown_connections 2024-03-18T16:07:20.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.509+0000 7f3f05200640 1 --2- 172.21.15.39:0/380584082 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3ee40a2880 0x7f3ee40a4d40 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:20.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.509+0000 7f3f05200640 1 --2- 172.21.15.39:0/380584082 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3f00077d80 0x7f3f001a5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:20.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.509+0000 7f3f05200640 1 --2- 172.21.15.39:0/380584082 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3f000773e0 0x7f3f0019e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:20.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.509+0000 7f3f05200640 1 --2- 172.21.15.39:0/380584082 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3f000761a0 0x7f3f0019e0c0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:20.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.509+0000 7f3f05200640 1 -- 172.21.15.39:0/380584082 >> 172.21.15.39:0/380584082 conn(0x7f3f000fef90 msgr2=0x7f3f00100a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:20.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.509+0000 7f3f05200640 1 -- 172.21.15.39:0/380584082 shutdown_connections 2024-03-18T16:07:20.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.509+0000 7f3f05200640 1 -- 172.21.15.39:0/380584082 wait complete. 2024-03-18T16:07:20.531 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-03-18T16:07:20.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.769+0000 7f16b9400640 1 -- 172.21.15.39:0/4017456297 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f16b4108ff0 msgr2=0x7f16b4109470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:20.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.769+0000 7f16b9400640 1 --2- 172.21.15.39:0/4017456297 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f16b4108ff0 0x7f16b4109470 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f16a0009900 tx=0x7f16a002bc50 comp rx=0 tx=0).stop 2024-03-18T16:07:20.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.769+0000 7f16b9400640 1 -- 172.21.15.39:0/4017456297 shutdown_connections 2024-03-18T16:07:20.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.769+0000 7f16b9400640 1 --2- 172.21.15.39:0/4017456297 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f16b41099b0 0x7f16b4110240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:20.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.769+0000 7f16b9400640 1 --2- 172.21.15.39:0/4017456297 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f16b4108ff0 0x7f16b4109470 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:20.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.769+0000 7f16b9400640 1 --2- 172.21.15.39:0/4017456297 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f16b4107e00 0x7f16b4108220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:20.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.769+0000 7f16b9400640 1 -- 172.21.15.39:0/4017456297 >> 172.21.15.39:0/4017456297 conn(0x7f16b4076da0 msgr2=0x7f16b40791e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:20.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.769+0000 7f16b9400640 1 -- 172.21.15.39:0/4017456297 shutdown_connections 2024-03-18T16:07:20.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.769+0000 7f16b9400640 1 -- 172.21.15.39:0/4017456297 wait complete. 2024-03-18T16:07:20.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.770+0000 7f16b9400640 1 Processor -- start 2024-03-18T16:07:20.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.771+0000 7f16b9400640 1 -- start start 2024-03-18T16:07:20.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.771+0000 7f16b9400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f16b4107e00 0x7f16b419e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:20.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.771+0000 7f16b9400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f16b4108ff0 0x7f16b419e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:20.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.771+0000 7f16b9400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f16b41099b0 0x7f16b41a5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:20.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.771+0000 7f16b9400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f16b4113a60 con 0x7f16b4107e00 2024-03-18T16:07:20.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.771+0000 7f16b9400640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f16b41138e0 con 0x7f16b41099b0 2024-03-18T16:07:20.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.771+0000 7f16b9400640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f16b4113be0 con 0x7f16b4108ff0 2024-03-18T16:07:20.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.771+0000 7f16b1e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f16b4108ff0 0x7f16b419e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:20.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.771+0000 7f16b2800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f16b4107e00 0x7f16b419e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:20.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.771+0000 7f16b1e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f16b4108ff0 0x7f16b419e6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:60878/0 (socket says 172.21.15.39:60878) 2024-03-18T16:07:20.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.771+0000 7f16b2800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f16b4107e00 0x7f16b419e180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:47622/0 (socket says 172.21.15.39:47622) 2024-03-18T16:07:20.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.771+0000 7f16b1e00640 1 -- 172.21.15.39:0/2979615443 learned_addr learned my addr 172.21.15.39:0/2979615443 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:07:20.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.771+0000 7f16b3200640 1 --2- 172.21.15.39:0/2979615443 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f16b41099b0 0x7f16b41a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:20.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.772+0000 7f16b1e00640 1 -- 172.21.15.39:0/2979615443 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f16b41099b0 msgr2=0x7f16b41a5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:20.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.772+0000 7f16b1e00640 1 --2- 172.21.15.39:0/2979615443 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f16b41099b0 0x7f16b41a5730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:20.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.772+0000 7f16b1e00640 1 -- 172.21.15.39:0/2979615443 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f16b4107e00 msgr2=0x7f16b419e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:20.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.772+0000 7f16b1e00640 1 --2- 172.21.15.39:0/2979615443 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f16b4107e00 0x7f16b419e180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:20.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.772+0000 7f16b1e00640 1 -- 172.21.15.39:0/2979615443 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16b41a5da0 con 0x7f16b4108ff0 2024-03-18T16:07:20.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.772+0000 7f16b1e00640 1 --2- 172.21.15.39:0/2979615443 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f16b4108ff0 0x7f16b419e6c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f16a0009230 tx=0x7f16a002c670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:20.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.772+0000 7f16ab400640 1 -- 172.21.15.39:0/2979615443 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16a0042020 con 0x7f16b4108ff0 2024-03-18T16:07:20.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.772+0000 7f16ab400640 1 -- 172.21.15.39:0/2979615443 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16a0004450 con 0x7f16b4108ff0 2024-03-18T16:07:20.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.772+0000 7f16ab400640 1 -- 172.21.15.39:0/2979615443 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16a0003e20 con 0x7f16b4108ff0 2024-03-18T16:07:20.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.772+0000 7f16b9400640 1 -- 172.21.15.39:0/2979615443 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16b41a6030 con 0x7f16b4108ff0 2024-03-18T16:07:20.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.773+0000 7f16b9400640 1 -- 172.21.15.39:0/2979615443 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16b41a6580 con 0x7f16b4108ff0 2024-03-18T16:07:20.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.774+0000 7f16b9400640 1 -- 172.21.15.39:0/2979615443 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16b410f120 con 0x7f16b4108ff0 2024-03-18T16:07:20.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.776+0000 7f16ab400640 1 -- 172.21.15.39:0/2979615443 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f16a0045020 con 0x7f16b4108ff0 2024-03-18T16:07:20.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.777+0000 7f16ab400640 1 --2- 172.21.15.39:0/2979615443 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f16980a2800 0x7f16980a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:20.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.777+0000 7f16ab400640 1 -- 172.21.15.39:0/2979615443 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f16a00f10e0 con 0x7f16b4108ff0 2024-03-18T16:07:20.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.777+0000 7f16b2800640 1 --2- 172.21.15.39:0/2979615443 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f16980a2800 0x7f16980a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:20.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.778+0000 7f16b2800640 1 --2- 172.21.15.39:0/2979615443 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f16980a2800 0x7f16980a4cc0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f16b4108e50 tx=0x7f16a4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:20.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:20.784+0000 7f16ab400640 1 -- 172.21.15.39:0/2979615443 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f16a004c020 con 0x7f16b4108ff0 2024-03-18T16:07:21.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.038+0000 7f16b9400640 1 -- 172.21.15.39:0/2979615443 --> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7f16b4061c20 con 0x7f16980a2800 2024-03-18T16:07:21.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.045+0000 7f16ab400640 1 -- 172.21.15.39:0/2979615443 <== mgr.14152 v2:172.21.15.39:6800/2749214286 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f16b4061c20 con 0x7f16980a2800 2024-03-18T16:07:21.047 INFO:teuthology.orchestra.run.smithi039.stdout:/volumes/qa/sv_0/2ad3ded7-edfe-4ee8-9f64-ad7c68bdb53c 2024-03-18T16:07:21.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.047+0000 7f16b9400640 1 -- 172.21.15.39:0/2979615443 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f16980a2800 msgr2=0x7f16980a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:21.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.047+0000 7f16b9400640 1 --2- 172.21.15.39:0/2979615443 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f16980a2800 0x7f16980a4cc0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f16b4108e50 tx=0x7f16a4005f90 comp rx=0 tx=0).stop 2024-03-18T16:07:21.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.048+0000 7f16b9400640 1 -- 172.21.15.39:0/2979615443 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f16b4108ff0 msgr2=0x7f16b419e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:21.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.048+0000 7f16b9400640 1 --2- 172.21.15.39:0/2979615443 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f16b4108ff0 0x7f16b419e6c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f16a0009230 tx=0x7f16a002c670 comp rx=0 tx=0).stop 2024-03-18T16:07:21.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.048+0000 7f16b9400640 1 -- 172.21.15.39:0/2979615443 shutdown_connections 2024-03-18T16:07:21.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.048+0000 7f16b9400640 1 --2- 172.21.15.39:0/2979615443 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f16b41099b0 0x7f16b41a5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:21.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.048+0000 7f16b9400640 1 --2- 172.21.15.39:0/2979615443 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f16980a2800 0x7f16980a4cc0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:21.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.048+0000 7f16b9400640 1 --2- 172.21.15.39:0/2979615443 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f16b4108ff0 0x7f16b419e6c0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:21.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.048+0000 7f16b9400640 1 --2- 172.21.15.39:0/2979615443 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f16b4107e00 0x7f16b419e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:21.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.048+0000 7f16b9400640 1 -- 172.21.15.39:0/2979615443 >> 172.21.15.39:0/2979615443 conn(0x7f16b4076da0 msgr2=0x7f16b410c2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:21.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.048+0000 7f16b9400640 1 -- 172.21.15.39:0/2979615443 shutdown_connections 2024-03-18T16:07:21.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.048+0000 7f16b9400640 1 -- 172.21.15.39:0/2979615443 wait complete. 2024-03-18T16:07:21.069 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-03-18T16:07:21.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.286+0000 7f832ee00640 1 -- 172.21.15.39:0/3451643756 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f83281028d0 msgr2=0x7f8328102d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:21.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.286+0000 7f832ee00640 1 --2- 172.21.15.39:0/3451643756 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f83281028d0 0x7f8328102d50 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f8314009900 tx=0x7f831402bc40 comp rx=0 tx=0).stop 2024-03-18T16:07:21.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.286+0000 7f832ee00640 1 -- 172.21.15.39:0/3451643756 shutdown_connections 2024-03-18T16:07:21.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.286+0000 7f832ee00640 1 --2- 172.21.15.39:0/3451643756 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f83281028d0 0x7f8328102d50 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:21.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.286+0000 7f832ee00640 1 --2- 172.21.15.39:0/3451643756 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8328103290 0x7f8328109b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:21.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.286+0000 7f832ee00640 1 --2- 172.21.15.39:0/3451643756 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f83281016e0 0x7f8328101b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:21.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.286+0000 7f832ee00640 1 -- 172.21.15.39:0/3451643756 >> 172.21.15.39:0/3451643756 conn(0x7f83280fcc50 msgr2=0x7f83280ff090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:21.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.286+0000 7f832ee00640 1 -- 172.21.15.39:0/3451643756 shutdown_connections 2024-03-18T16:07:21.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.287+0000 7f832ee00640 1 -- 172.21.15.39:0/3451643756 wait complete. 2024-03-18T16:07:21.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.287+0000 7f832ee00640 1 Processor -- start 2024-03-18T16:07:21.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.288+0000 7f832ee00640 1 -- start start 2024-03-18T16:07:21.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.288+0000 7f832ee00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f83281016e0 0x7f8328075590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:21.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.288+0000 7f832ee00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f83281028d0 0x7f8328073bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:21.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.288+0000 7f832ee00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8328103290 0x7f8328074110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:21.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.288+0000 7f832ee00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f832810d3b0 con 0x7f8328103290 2024-03-18T16:07:21.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.288+0000 7f832ee00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f832810d230 con 0x7f83281028d0 2024-03-18T16:07:21.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.288+0000 7f832ee00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f832810d530 con 0x7f83281016e0 2024-03-18T16:07:21.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.288+0000 7f832da00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f83281016e0 0x7f8328075590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:21.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.288+0000 7f832da00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f83281016e0 0x7f8328075590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:60890/0 (socket says 172.21.15.39:60890) 2024-03-18T16:07:21.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.288+0000 7f832da00640 1 -- 172.21.15.39:0/2192097248 learned_addr learned my addr 172.21.15.39:0/2192097248 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:07:21.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.288+0000 7f832e400640 1 --2- 172.21.15.39:0/2192097248 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8328103290 0x7f8328074110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:21.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.288+0000 7f832d000640 1 --2- 172.21.15.39:0/2192097248 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f83281028d0 0x7f8328073bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:21.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.289+0000 7f832da00640 1 -- 172.21.15.39:0/2192097248 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f83281028d0 msgr2=0x7f8328073bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:21.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.289+0000 7f832da00640 1 --2- 172.21.15.39:0/2192097248 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f83281028d0 0x7f8328073bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:21.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.289+0000 7f832da00640 1 -- 172.21.15.39:0/2192097248 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8328103290 msgr2=0x7f8328074110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:21.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.289+0000 7f832da00640 1 --2- 172.21.15.39:0/2192097248 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8328103290 0x7f8328074110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:21.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.289+0000 7f832da00640 1 -- 172.21.15.39:0/2192097248 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83280749a0 con 0x7f83281016e0 2024-03-18T16:07:21.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.289+0000 7f832da00640 1 --2- 172.21.15.39:0/2192097248 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f83281016e0 0x7f8328075590 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f831c00e9c0 tx=0x7f831c00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:21.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.289+0000 7f8326000640 1 -- 172.21.15.39:0/2192097248 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f831c00cd80 con 0x7f83281016e0 2024-03-18T16:07:21.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.289+0000 7f8326000640 1 -- 172.21.15.39:0/2192097248 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f831c0035f0 con 0x7f83281016e0 2024-03-18T16:07:21.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.289+0000 7f832ee00640 1 -- 172.21.15.39:0/2192097248 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83281a1b40 con 0x7f83281016e0 2024-03-18T16:07:21.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.289+0000 7f8326000640 1 -- 172.21.15.39:0/2192097248 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f831c010690 con 0x7f83281016e0 2024-03-18T16:07:21.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.289+0000 7f832ee00640 1 -- 172.21.15.39:0/2192097248 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83281a20b0 con 0x7f83281016e0 2024-03-18T16:07:21.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.291+0000 7f832ee00640 1 -- 172.21.15.39:0/2192097248 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83280684f0 con 0x7f83281016e0 2024-03-18T16:07:21.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.292+0000 7f8326000640 1 -- 172.21.15.39:0/2192097248 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f831c004020 con 0x7f83281016e0 2024-03-18T16:07:21.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.293+0000 7f8326000640 1 --2- 172.21.15.39:0/2192097248 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f83100a2880 0x7f83100a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:21.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.294+0000 7f8326000640 1 -- 172.21.15.39:0/2192097248 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f831c0cf0b0 con 0x7f83281016e0 2024-03-18T16:07:21.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.294+0000 7f832d000640 1 --2- 172.21.15.39:0/2192097248 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f83100a2880 0x7f83100a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:21.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.295+0000 7f832d000640 1 --2- 172.21.15.39:0/2192097248 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f83100a2880 0x7f83100a4d40 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f831402c250 tx=0x7f831402cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:21.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.302+0000 7f8326000640 1 -- 172.21.15.39:0/2192097248 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f831c09beb0 con 0x7f83281016e0 2024-03-18T16:07:21.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:21 smithi039 ceph-mon[31721]: pgmap v336: 129 pgs: 129 active+clean; 582 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s wr, 3 op/s 2024-03-18T16:07:21.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:21 smithi039 ceph-mon[31721]: from='client.24815 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:07:21.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:21 smithi039 ceph-mon[31721]: mds.? [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] up:active 2024-03-18T16:07:21.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:21 smithi039 ceph-mon[31721]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-03-18T16:07:21.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:21 smithi039 ceph-mon[31721]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T16:07:21.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:21 smithi194 ceph-mon[36200]: pgmap v336: 129 pgs: 129 active+clean; 582 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s wr, 3 op/s 2024-03-18T16:07:21.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:21 smithi194 ceph-mon[36200]: from='client.24815 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:07:21.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:21 smithi194 ceph-mon[36200]: mds.? [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] up:active 2024-03-18T16:07:21.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:21 smithi194 ceph-mon[36200]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-03-18T16:07:21.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:21 smithi194 ceph-mon[36200]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T16:07:21.544 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.543+0000 7f832ee00640 1 -- 172.21.15.39:0/2192097248 --> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f8328061d70 con 0x7f83100a2880 2024-03-18T16:07:21.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.572+0000 7f8326000640 1 -- 172.21.15.39:0/2192097248 <== mgr.14152 v2:172.21.15.39:6800/2749214286 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f8328061d70 con 0x7f83100a2880 2024-03-18T16:07:21.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.574+0000 7f832ee00640 1 -- 172.21.15.39:0/2192097248 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f83100a2880 msgr2=0x7f83100a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:21.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.574+0000 7f832ee00640 1 --2- 172.21.15.39:0/2192097248 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f83100a2880 0x7f83100a4d40 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f831402c250 tx=0x7f831402cf90 comp rx=0 tx=0).stop 2024-03-18T16:07:21.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.574+0000 7f832ee00640 1 -- 172.21.15.39:0/2192097248 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f83281016e0 msgr2=0x7f8328075590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:21.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.574+0000 7f832ee00640 1 --2- 172.21.15.39:0/2192097248 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f83281016e0 0x7f8328075590 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f831c00e9c0 tx=0x7f831c00ee80 comp rx=0 tx=0).stop 2024-03-18T16:07:21.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.575+0000 7f832ee00640 1 -- 172.21.15.39:0/2192097248 shutdown_connections 2024-03-18T16:07:21.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.575+0000 7f832ee00640 1 --2- 172.21.15.39:0/2192097248 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f83281028d0 0x7f8328073bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:21.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.575+0000 7f832ee00640 1 --2- 172.21.15.39:0/2192097248 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f83100a2880 0x7f83100a4d40 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:21.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.575+0000 7f832ee00640 1 --2- 172.21.15.39:0/2192097248 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8328103290 0x7f8328074110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:21.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.575+0000 7f832ee00640 1 --2- 172.21.15.39:0/2192097248 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f83281016e0 0x7f8328075590 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:21.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.575+0000 7f832ee00640 1 -- 172.21.15.39:0/2192097248 >> 172.21.15.39:0/2192097248 conn(0x7f83280fcc50 msgr2=0x7f8328105bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:21.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.575+0000 7f832ee00640 1 -- 172.21.15.39:0/2192097248 shutdown_connections 2024-03-18T16:07:21.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.575+0000 7f832ee00640 1 -- 172.21.15.39:0/2192097248 wait complete. 2024-03-18T16:07:21.595 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-03-18T16:07:21.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:21 smithi196 ceph-mon[36307]: pgmap v336: 129 pgs: 129 active+clean; 582 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s wr, 3 op/s 2024-03-18T16:07:21.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:21 smithi196 ceph-mon[36307]: from='client.24815 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:07:21.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:21 smithi196 ceph-mon[36307]: mds.? [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] up:active 2024-03-18T16:07:21.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:21 smithi196 ceph-mon[36307]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-03-18T16:07:21.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:21 smithi196 ceph-mon[36307]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T16:07:21.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.826+0000 7f9477e00640 1 -- 172.21.15.39:0/4099069991 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9478104ad0 msgr2=0x7f9478104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:21.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.826+0000 7f9477e00640 1 --2- 172.21.15.39:0/4099069991 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9478104ad0 0x7f9478104f50 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f9464009900 tx=0x7f946402bc30 comp rx=0 tx=0).stop 2024-03-18T16:07:21.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.826+0000 7f9477e00640 1 -- 172.21.15.39:0/4099069991 shutdown_connections 2024-03-18T16:07:21.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.826+0000 7f9477e00640 1 --2- 172.21.15.39:0/4099069991 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9478105490 0x7f947810bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:21.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.826+0000 7f9477e00640 1 --2- 172.21.15.39:0/4099069991 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9478104ad0 0x7f9478104f50 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:21.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.826+0000 7f9477e00640 1 --2- 172.21.15.39:0/4099069991 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f94781038e0 0x7f9478103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:21.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.826+0000 7f9477e00640 1 -- 172.21.15.39:0/4099069991 >> 172.21.15.39:0/4099069991 conn(0x7f94780fee50 msgr2=0x7f9478101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:21.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.826+0000 7f9477e00640 1 -- 172.21.15.39:0/4099069991 shutdown_connections 2024-03-18T16:07:21.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.826+0000 7f9477e00640 1 -- 172.21.15.39:0/4099069991 wait complete. 2024-03-18T16:07:21.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.827+0000 7f9477e00640 1 Processor -- start 2024-03-18T16:07:21.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.827+0000 7f9477e00640 1 -- start start 2024-03-18T16:07:21.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.828+0000 7f9477e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f94781038e0 0x7f9478199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:21.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.828+0000 7f9477e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9478104ad0 0x7f947819a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:21.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.828+0000 7f9477e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9478105490 0x7f94781a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:21.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.828+0000 7f9477e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f947810f560 con 0x7f9478105490 2024-03-18T16:07:21.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.828+0000 7f9477e00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f947810f3e0 con 0x7f9478104ad0 2024-03-18T16:07:21.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.828+0000 7f9477e00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f947810f6e0 con 0x7f94781038e0 2024-03-18T16:07:21.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.828+0000 7f9476a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f94781038e0 0x7f9478199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:21.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.828+0000 7f9476a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f94781038e0 0x7f9478199c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:60898/0 (socket says 172.21.15.39:60898) 2024-03-18T16:07:21.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.828+0000 7f9476a00640 1 -- 172.21.15.39:0/2143997655 learned_addr learned my addr 172.21.15.39:0/2143997655 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:07:21.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.828+0000 7f9476000640 1 --2- 172.21.15.39:0/2143997655 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9478104ad0 0x7f947819a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:21.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.828+0000 7f9477400640 1 --2- 172.21.15.39:0/2143997655 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9478105490 0x7f94781a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:21.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.828+0000 7f9476a00640 1 -- 172.21.15.39:0/2143997655 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9478104ad0 msgr2=0x7f947819a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:21.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.828+0000 7f9476a00640 1 --2- 172.21.15.39:0/2143997655 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9478104ad0 0x7f947819a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:21.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.828+0000 7f9476a00640 1 -- 172.21.15.39:0/2143997655 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9478105490 msgr2=0x7f94781a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:21.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.828+0000 7f9476a00640 1 --2- 172.21.15.39:0/2143997655 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9478105490 0x7f94781a1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:21.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.828+0000 7f9476a00640 1 -- 172.21.15.39:0/2143997655 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94781a1830 con 0x7f94781038e0 2024-03-18T16:07:21.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.829+0000 7f9476a00640 1 --2- 172.21.15.39:0/2143997655 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f94781038e0 0x7f9478199c20 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f946800e9c0 tx=0x7f946800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:21.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.829+0000 7f946f400640 1 -- 172.21.15.39:0/2143997655 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f946800cd80 con 0x7f94781038e0 2024-03-18T16:07:21.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.829+0000 7f946f400640 1 -- 172.21.15.39:0/2143997655 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94680035f0 con 0x7f94781038e0 2024-03-18T16:07:21.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.829+0000 7f946f400640 1 -- 172.21.15.39:0/2143997655 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9468010690 con 0x7f94781038e0 2024-03-18T16:07:21.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.829+0000 7f9477e00640 1 -- 172.21.15.39:0/2143997655 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94781a1b20 con 0x7f94781038e0 2024-03-18T16:07:21.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.829+0000 7f9477e00640 1 -- 172.21.15.39:0/2143997655 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94781a2020 con 0x7f94781038e0 2024-03-18T16:07:21.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.830+0000 7f9477e00640 1 -- 172.21.15.39:0/2143997655 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f947810ac00 con 0x7f94781038e0 2024-03-18T16:07:21.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.833+0000 7f946f400640 1 -- 172.21.15.39:0/2143997655 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 16) ==== 137290+0+0 (secure 0 0 0) 0x7f94680037a0 con 0x7f94781038e0 2024-03-18T16:07:21.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.834+0000 7f946f400640 1 --2- 172.21.15.39:0/2143997655 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f945c0a2800 0x7f945c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:21.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.834+0000 7f946f400640 1 -- 172.21.15.39:0/2143997655 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f94680cf0f0 con 0x7f94781038e0 2024-03-18T16:07:21.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.834+0000 7f9476000640 1 --2- 172.21.15.39:0/2143997655 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f945c0a2800 0x7f945c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:21.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.835+0000 7f9476000640 1 --2- 172.21.15.39:0/2143997655 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f945c0a2800 0x7f945c0a4cc0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f946402c7b0 tx=0x7f9464005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:21.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:21.838+0000 7f946f400640 1 -- 172.21.15.39:0/2143997655 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f946809bef0 con 0x7f94781038e0 2024-03-18T16:07:22.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.074+0000 7f9477e00640 1 -- 172.21.15.39:0/2143997655 --> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f9478061c20 con 0x7f945c0a2800 2024-03-18T16:07:22.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.081+0000 7f946f400640 1 -- 172.21.15.39:0/2143997655 <== mgr.14152 v2:172.21.15.39:6800/2749214286 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f9478061c20 con 0x7f945c0a2800 2024-03-18T16:07:22.082 INFO:teuthology.orchestra.run.smithi039.stdout:/volumes/qa/sv_1/ba28ee63-cdd7-4699-94b4-1139b8b6c674 2024-03-18T16:07:22.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.083+0000 7f9477e00640 1 -- 172.21.15.39:0/2143997655 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f945c0a2800 msgr2=0x7f945c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:22.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.083+0000 7f9477e00640 1 --2- 172.21.15.39:0/2143997655 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f945c0a2800 0x7f945c0a4cc0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f946402c7b0 tx=0x7f9464005fd0 comp rx=0 tx=0).stop 2024-03-18T16:07:22.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.083+0000 7f9477e00640 1 -- 172.21.15.39:0/2143997655 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f94781038e0 msgr2=0x7f9478199c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:22.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.083+0000 7f9477e00640 1 --2- 172.21.15.39:0/2143997655 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f94781038e0 0x7f9478199c20 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f946800e9c0 tx=0x7f946800ee80 comp rx=0 tx=0).stop 2024-03-18T16:07:22.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.084+0000 7f9477e00640 1 -- 172.21.15.39:0/2143997655 shutdown_connections 2024-03-18T16:07:22.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.084+0000 7f9477e00640 1 --2- 172.21.15.39:0/2143997655 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9478104ad0 0x7f947819a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:22.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.084+0000 7f9477e00640 1 --2- 172.21.15.39:0/2143997655 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f945c0a2800 0x7f945c0a4cc0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:22.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.084+0000 7f9477e00640 1 --2- 172.21.15.39:0/2143997655 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9478105490 0x7f94781a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:22.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.084+0000 7f9477e00640 1 --2- 172.21.15.39:0/2143997655 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f94781038e0 0x7f9478199c20 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:22.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.084+0000 7f9477e00640 1 -- 172.21.15.39:0/2143997655 >> 172.21.15.39:0/2143997655 conn(0x7f94780fee50 msgr2=0x7f9478107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:22.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.084+0000 7f9477e00640 1 -- 172.21.15.39:0/2143997655 shutdown_connections 2024-03-18T16:07:22.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.084+0000 7f9477e00640 1 -- 172.21.15.39:0/2143997655 wait complete. 2024-03-18T16:07:22.107 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:07:22.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.359+0000 7f7742000640 1 -- 172.21.15.39:0/2746422182 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f773c1072e0 msgr2=0x7f773c1096d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:22.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.359+0000 7f7742000640 1 --2- 172.21.15.39:0/2746422182 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f773c1072e0 0x7f773c1096d0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f7734009ca0 tx=0x7f773402c100 comp rx=0 tx=0).stop 2024-03-18T16:07:22.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.359+0000 7f7742000640 1 -- 172.21.15.39:0/2746422182 shutdown_connections 2024-03-18T16:07:22.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.359+0000 7f7742000640 1 --2- 172.21.15.39:0/2746422182 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f773c1072e0 0x7f773c1096d0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:22.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.359+0000 7f7742000640 1 --2- 172.21.15.39:0/2746422182 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f773c0ffb30 0x7f773c106da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:22.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.359+0000 7f7742000640 1 --2- 172.21.15.39:0/2746422182 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f773c0ff1d0 0x7f773c0ff5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:22.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.359+0000 7f7742000640 1 -- 172.21.15.39:0/2746422182 >> 172.21.15.39:0/2746422182 conn(0x7f773c0fad90 msgr2=0x7f773c0fd1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:22.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.359+0000 7f7742000640 1 -- 172.21.15.39:0/2746422182 shutdown_connections 2024-03-18T16:07:22.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.359+0000 7f7742000640 1 -- 172.21.15.39:0/2746422182 wait complete. 2024-03-18T16:07:22.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.361+0000 7f7742000640 1 Processor -- start 2024-03-18T16:07:22.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.361+0000 7f7742000640 1 -- start start 2024-03-18T16:07:22.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.361+0000 7f7742000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f773c0ff1d0 0x7f773c199b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:22.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.361+0000 7f7742000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f773c0ffb30 0x7f773c19a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:22.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.361+0000 7f7742000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f773c1072e0 0x7f773c1a10e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:22.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.361+0000 7f7742000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f773c10d260 con 0x7f773c1072e0 2024-03-18T16:07:22.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.361+0000 7f7742000640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f773c10d0e0 con 0x7f773c0ff1d0 2024-03-18T16:07:22.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.361+0000 7f7742000640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f773c10d3e0 con 0x7f773c0ffb30 2024-03-18T16:07:22.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.362+0000 7f7741600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f773c1072e0 0x7f773c1a10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:22.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.362+0000 7f7740c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f773c0ff1d0 0x7f773c199b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:22.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.362+0000 7f773be00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f773c0ffb30 0x7f773c19a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:22.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.362+0000 7f7740c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f773c0ff1d0 0x7f773c199b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:56472/0 (socket says 172.21.15.39:56472) 2024-03-18T16:07:22.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.362+0000 7f7741600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f773c1072e0 0x7f773c1a10e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:47682/0 (socket says 172.21.15.39:47682) 2024-03-18T16:07:22.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.362+0000 7f773be00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f773c0ffb30 0x7f773c19a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:60918/0 (socket says 172.21.15.39:60918) 2024-03-18T16:07:22.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.362+0000 7f7740c00640 1 -- 172.21.15.39:0/3889527143 learned_addr learned my addr 172.21.15.39:0/3889527143 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:07:22.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.362+0000 7f773be00640 1 -- 172.21.15.39:0/3889527143 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f773c0ff1d0 msgr2=0x7f773c199b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:22.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.362+0000 7f773be00640 1 --2- 172.21.15.39:0/3889527143 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f773c0ff1d0 0x7f773c199b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:22.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.362+0000 7f773be00640 1 -- 172.21.15.39:0/3889527143 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f773c1072e0 msgr2=0x7f773c1a10e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:22.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.362+0000 7f773be00640 1 --2- 172.21.15.39:0/3889527143 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f773c1072e0 0x7f773c1a10e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:22.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.362+0000 7f773be00640 1 -- 172.21.15.39:0/3889527143 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f773c1a1750 con 0x7f773c0ffb30 2024-03-18T16:07:22.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.362+0000 7f7741600640 1 --2- 172.21.15.39:0/3889527143 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f773c1072e0 0x7f773c1a10e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:07:22.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.362+0000 7f773be00640 1 --2- 172.21.15.39:0/3889527143 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f773c0ffb30 0x7f773c19a0c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f772800e9c0 tx=0x7f772800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:22.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.363+0000 7f7739600640 1 -- 172.21.15.39:0/3889527143 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f772800ccc0 con 0x7f773c0ffb30 2024-03-18T16:07:22.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.363+0000 7f7739600640 1 -- 172.21.15.39:0/3889527143 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f77280035f0 con 0x7f773c0ffb30 2024-03-18T16:07:22.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.363+0000 7f7739600640 1 -- 172.21.15.39:0/3889527143 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f772800ce60 con 0x7f773c0ffb30 2024-03-18T16:07:22.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.363+0000 7f7742000640 1 -- 172.21.15.39:0/3889527143 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f773c1a1a40 con 0x7f773c0ffb30 2024-03-18T16:07:22.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.363+0000 7f7742000640 1 -- 172.21.15.39:0/3889527143 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f773c1a1f40 con 0x7f773c0ffb30 2024-03-18T16:07:22.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.364+0000 7f7742000640 1 -- 172.21.15.39:0/3889527143 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f773c1a21d0 con 0x7f773c0ffb30 2024-03-18T16:07:22.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.367+0000 7f7739600640 1 -- 172.21.15.39:0/3889527143 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f77280037a0 con 0x7f773c0ffb30 2024-03-18T16:07:22.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.368+0000 7f7739600640 1 --2- 172.21.15.39:0/3889527143 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f77240a2910 0x7f77240a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:22.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.368+0000 7f7739600640 1 -- 172.21.15.39:0/3889527143 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f77280cf2a0 con 0x7f773c0ffb30 2024-03-18T16:07:22.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.368+0000 7f7740c00640 1 --2- 172.21.15.39:0/3889527143 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f77240a2910 0x7f77240a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:22.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.369+0000 7f7740c00640 1 --2- 172.21.15.39:0/3889527143 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f77240a2910 0x7f77240a4dd0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f7730005fd0 tx=0x7f773000b4a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:22.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.372+0000 7f7739600640 1 -- 172.21.15.39:0/3889527143 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f772809bf40 con 0x7f773c0ffb30 2024-03-18T16:07:22.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:22 smithi039 ceph-mon[31721]: from='client.24827 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:07:22.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:22 smithi039 ceph-mon[31721]: from='client.24853 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:07:22.666 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.664+0000 7f7742000640 1 -- 172.21.15.39:0/3889527143 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f773c1a23a0 con 0x7f773c0ffb30 2024-03-18T16:07:22.666 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.665+0000 7f7739600640 1 -- 172.21.15.39:0/3889527143 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11932 (secure 0 0 0) 0x7f772809b840 con 0x7f773c0ffb30 2024-03-18T16:07:22.667 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:07:22.667 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2520226608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2520226608},{"type":"v1","addr":"172.21.15.39:6835","nonce":2520226608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/719341763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":719341763},{"type":"v1","addr":"172.21.15.39:6837","nonce":719341763}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T16:06:44.180425+0000","modified":"2024-03-18T16:07:20.994405+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24575,"mds_1":24556,"mds_2":24535,"mds_3":24524,"mds_4":14463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14445":{"gid":14445,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.39:6839/656513813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":656513813},{"type":"v1","addr":"172.21.15.39:6839","nonce":656513813}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14463":{"gid":14463,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":13,"addr":"172.21.15.39:6841/3515326042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3515326042},{"type":"v1","addr":"172.21.15.39:6841","nonce":3515326042}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6833/931350738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":931350738},{"type":"v1","addr":"172.21.15.194:6833","nonce":931350738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6833/3874209621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":3874209621},{"type":"v1","addr":"172.21.15.196:6833","nonce":3874209621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6835/188385470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":188385470},{"type":"v1","addr":"172.21.15.194:6835","nonce":188385470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6835/3949984309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":3949984309},{"type":"v1","addr":"172.21.15.196:6835","nonce":3949984309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":15,"addr":"172.21.15.194:6837/1301464246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":1301464246},{"type":"v1","addr":"172.21.15.194:6837","nonce":1301464246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":16,"addr":"172.21.15.196:6837/895247251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":895247251},{"type":"v1","addr":"172.21.15.196:6837","nonce":895247251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.196:6839/2001427268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6838","nonce":2001427268},{"type":"v1","addr":"172.21.15.196:6839","nonce":2001427268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24575":{"gid":24575,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.194:6839/274911158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":274911158},{"type":"v1","addr":"172.21.15.194:6839","nonce":274911158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24575,"qdb_cluster":[24556,24535,24575,24524,14463]},"id":1}]} 2024-03-18T16:07:22.670 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 27 2024-03-18T16:07:22.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.668+0000 7f7742000640 1 -- 172.21.15.39:0/3889527143 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f77240a2910 msgr2=0x7f77240a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:22.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.668+0000 7f7742000640 1 --2- 172.21.15.39:0/3889527143 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f77240a2910 0x7f77240a4dd0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f7730005fd0 tx=0x7f773000b4a0 comp rx=0 tx=0).stop 2024-03-18T16:07:22.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.668+0000 7f7742000640 1 -- 172.21.15.39:0/3889527143 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f773c0ffb30 msgr2=0x7f773c19a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:22.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.668+0000 7f7742000640 1 --2- 172.21.15.39:0/3889527143 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f773c0ffb30 0x7f773c19a0c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f772800e9c0 tx=0x7f772800ee80 comp rx=0 tx=0).stop 2024-03-18T16:07:22.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.668+0000 7f7742000640 1 -- 172.21.15.39:0/3889527143 shutdown_connections 2024-03-18T16:07:22.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.668+0000 7f7742000640 1 --2- 172.21.15.39:0/3889527143 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f77240a2910 0x7f77240a4dd0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:22.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.668+0000 7f7742000640 1 --2- 172.21.15.39:0/3889527143 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f773c1072e0 0x7f773c1a10e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:22.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.668+0000 7f7742000640 1 --2- 172.21.15.39:0/3889527143 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f773c0ffb30 0x7f773c19a0c0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:22.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.668+0000 7f7742000640 1 --2- 172.21.15.39:0/3889527143 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f773c0ff1d0 0x7f773c199b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:22.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.668+0000 7f7742000640 1 -- 172.21.15.39:0/3889527143 >> 172.21.15.39:0/3889527143 conn(0x7f773c0fad90 msgr2=0x7f773c0fd180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:22.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.668+0000 7f7742000640 1 -- 172.21.15.39:0/3889527143 shutdown_connections 2024-03-18T16:07:22.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.668+0000 7f7742000640 1 -- 172.21.15.39:0/3889527143 wait complete. 2024-03-18T16:07:22.693 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T16:07:22.727 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:22 smithi196 ceph-mon[36307]: from='client.24827 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:07:22.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:22 smithi196 ceph-mon[36307]: from='client.24853 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:07:22.745 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:22 smithi194 ceph-mon[36200]: from='client.24827 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:07:22.745 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:22 smithi194 ceph-mon[36200]: from='client.24853 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:07:22.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.943+0000 7f1ef1a00640 1 -- 172.21.15.39:0/3321316298 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1eec0ffe60 msgr2=0x7f1eec102290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:22.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.943+0000 7f1ef1a00640 1 --2- 172.21.15.39:0/3321316298 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1eec0ffe60 0x7f1eec102290 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f1edc009ca0 tx=0x7f1edc02c100 comp rx=0 tx=0).stop 2024-03-18T16:07:22.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.944+0000 7f1ef1a00640 1 -- 172.21.15.39:0/3321316298 shutdown_connections 2024-03-18T16:07:22.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.944+0000 7f1ef1a00640 1 --2- 172.21.15.39:0/3321316298 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1eec105180 0x7f1eec105600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:22.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.944+0000 7f1ef1a00640 1 --2- 172.21.15.39:0/3321316298 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1eec1027d0 0x7f1eec104c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:22.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.944+0000 7f1ef1a00640 1 --2- 172.21.15.39:0/3321316298 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1eec0ffe60 0x7f1eec102290 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:22.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.944+0000 7f1ef1a00640 1 -- 172.21.15.39:0/3321316298 >> 172.21.15.39:0/3321316298 conn(0x7f1eec0fba60 msgr2=0x7f1eec0fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:22.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.944+0000 7f1ef1a00640 1 -- 172.21.15.39:0/3321316298 shutdown_connections 2024-03-18T16:07:22.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.944+0000 7f1ef1a00640 1 -- 172.21.15.39:0/3321316298 wait complete. 2024-03-18T16:07:22.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.945+0000 7f1ef1a00640 1 Processor -- start 2024-03-18T16:07:22.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.945+0000 7f1ef1a00640 1 -- start start 2024-03-18T16:07:22.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.946+0000 7f1ef1a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1eec0ffe60 0x7f1eec19dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:22.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.946+0000 7f1ef1a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1eec1027d0 0x7f1eec19e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:22.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.946+0000 7f1ef1a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1eec105180 0x7f1eec1a5450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:22.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.946+0000 7f1ef1a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f1eec113840 con 0x7f1eec1027d0 2024-03-18T16:07:22.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.946+0000 7f1ef1a00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f1eec1136c0 con 0x7f1eec105180 2024-03-18T16:07:22.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.946+0000 7f1ef1a00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f1eec1139c0 con 0x7f1eec0ffe60 2024-03-18T16:07:22.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.946+0000 7f1eebe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1eec0ffe60 0x7f1eec19dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:22.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.946+0000 7f1eebe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1eec0ffe60 0x7f1eec19dea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:60936/0 (socket says 172.21.15.39:60936) 2024-03-18T16:07:22.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.946+0000 7f1eebe00640 1 -- 172.21.15.39:0/2946819666 learned_addr learned my addr 172.21.15.39:0/2946819666 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:07:22.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.946+0000 7f1eeb400640 1 --2- 172.21.15.39:0/2946819666 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1eec1027d0 0x7f1eec19e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:22.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.946+0000 7f1ef1000640 1 --2- 172.21.15.39:0/2946819666 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1eec105180 0x7f1eec1a5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:22.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.946+0000 7f1eebe00640 1 -- 172.21.15.39:0/2946819666 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1eec105180 msgr2=0x7f1eec1a5450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:22.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.946+0000 7f1eebe00640 1 --2- 172.21.15.39:0/2946819666 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1eec105180 0x7f1eec1a5450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:22.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.946+0000 7f1eebe00640 1 -- 172.21.15.39:0/2946819666 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1eec1027d0 msgr2=0x7f1eec19e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:22.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.946+0000 7f1eebe00640 1 --2- 172.21.15.39:0/2946819666 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1eec1027d0 0x7f1eec19e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:22.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.946+0000 7f1eebe00640 1 -- 172.21.15.39:0/2946819666 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1eec1a5ac0 con 0x7f1eec0ffe60 2024-03-18T16:07:22.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.947+0000 7f1eebe00640 1 --2- 172.21.15.39:0/2946819666 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1eec0ffe60 0x7f1eec19dea0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f1edc0325f0 tx=0x7f1edc035d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:22.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.947+0000 7f1ee8c00640 1 -- 172.21.15.39:0/2946819666 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1edc039040 con 0x7f1eec0ffe60 2024-03-18T16:07:22.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.947+0000 7f1ee8c00640 1 -- 172.21.15.39:0/2946819666 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1edc044070 con 0x7f1eec0ffe60 2024-03-18T16:07:22.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.947+0000 7f1ef1a00640 1 -- 172.21.15.39:0/2946819666 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1eec1a5d50 con 0x7f1eec0ffe60 2024-03-18T16:07:22.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.947+0000 7f1ee8c00640 1 -- 172.21.15.39:0/2946819666 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1edc0044d0 con 0x7f1eec0ffe60 2024-03-18T16:07:22.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.947+0000 7f1ef1a00640 1 -- 172.21.15.39:0/2946819666 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1eec1a61b0 con 0x7f1eec0ffe60 2024-03-18T16:07:22.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.948+0000 7f1ef1a00640 1 -- 172.21.15.39:0/2946819666 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1eec0fd550 con 0x7f1eec0ffe60 2024-03-18T16:07:22.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.950+0000 7f1ee8c00640 1 -- 172.21.15.39:0/2946819666 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f1edc004670 con 0x7f1eec0ffe60 2024-03-18T16:07:22.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.951+0000 7f1ee8c00640 1 --2- 172.21.15.39:0/2946819666 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1ed00a2910 0x7f1ed00a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:22.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.952+0000 7f1ee8c00640 1 -- 172.21.15.39:0/2946819666 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f1edc0f1580 con 0x7f1eec0ffe60 2024-03-18T16:07:22.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.952+0000 7f1eeb400640 1 --2- 172.21.15.39:0/2946819666 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1ed00a2910 0x7f1ed00a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:22.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.952+0000 7f1eeb400640 1 --2- 172.21.15.39:0/2946819666 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1ed00a2910 0x7f1ed00a4dd0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f1eec19f3b0 tx=0x7f1ed4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:22.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:22.954+0000 7f1ee8c00640 1 -- 172.21.15.39:0/2946819666 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1edc0be450 con 0x7f1eec0ffe60 2024-03-18T16:07:23.185 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.184+0000 7f1ef1a00640 1 -- 172.21.15.39:0/2946819666 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1eec062480 con 0x7f1eec0ffe60 2024-03-18T16:07:23.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.186+0000 7f1ee8c00640 1 -- 172.21.15.39:0/2946819666 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24142 (secure 0 0 0) 0x7f1edc0bdd50 con 0x7f1eec0ffe60 2024-03-18T16:07:23.188 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:07:23.188 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":89,"fsid":"1892d8fc-e540-11ee-95c9-87774f69a715","created":"2024-03-18T15:57:36.507008+0000","modified":"2024-03-18T16:06:49.510653+0000","last_up_change":"2024-03-18T16:05:45.250629+0000","last_in_change":"2024-03-18T16:05:24.841337+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T16:01:30.048247+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T16:06:42.191791+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T16:06:43.670325+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T16:06:47.523388+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"771e0558-be62-4436-94b0-6364751911c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6803","nonce":4090743635}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6805","nonce":4090743635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6809","nonce":4090743635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6807","nonce":4090743635}]},"public_addr":"172.21.15.39:6803/4090743635","cluster_addr":"172.21.15.39:6805/4090743635","heartbeat_back_addr":"172.21.15.39:6809/4090743635","heartbeat_front_addr":"172.21.15.39:6807/4090743635","state":["exists","up"]},{"osd":1,"uuid":"0430eee1-cf34-4277-b255-c7d9f07ad19a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6801","nonce":2800329805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6803","nonce":2800329805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6807","nonce":2800329805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6805","nonce":2800329805}]},"public_addr":"172.21.15.194:6801/2800329805","cluster_addr":"172.21.15.194:6803/2800329805","heartbeat_back_addr":"172.21.15.194:6807/2800329805","heartbeat_front_addr":"172.21.15.194:6805/2800329805","state":["exists","up"]},{"osd":2,"uuid":"3a718da4-2511-4ec1-b1a8-eafd6cce0df0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6801","nonce":3935874719}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6803","nonce":3935874719}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6807","nonce":3935874719}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6805","nonce":3935874719}]},"public_addr":"172.21.15.196:6801/3935874719","cluster_addr":"172.21.15.196:6803/3935874719","heartbeat_back_addr":"172.21.15.196:6807/3935874719","heartbeat_front_addr":"172.21.15.196:6805/3935874719","state":["exists","up"]},{"osd":3,"uuid":"6f9c37f9-0ab9-4038-8919-afd4b39c4165","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6811","nonce":893345939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6813","nonce":893345939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6817","nonce":893345939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6815","nonce":893345939}]},"public_addr":"172.21.15.39:6811/893345939","cluster_addr":"172.21.15.39:6813/893345939","heartbeat_back_addr":"172.21.15.39:6817/893345939","heartbeat_front_addr":"172.21.15.39:6815/893345939","state":["exists","up"]},{"osd":4,"uuid":"e795bb0e-4791-4537-acda-1a321e19bc9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6809","nonce":800423088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6811","nonce":800423088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6815","nonce":800423088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6813","nonce":800423088}]},"public_addr":"172.21.15.194:6809/800423088","cluster_addr":"172.21.15.194:6811/800423088","heartbeat_back_addr":"172.21.15.194:6815/800423088","heartbeat_front_addr":"172.21.15.194:6813/800423088","state":["exists","up"]},{"osd":5,"uuid":"026e175e-6acb-4b4c-b235-8ccee8ae1149","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6809","nonce":2746224525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6811","nonce":2746224525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6815","nonce":2746224525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6813","nonce":2746224525}]},"public_addr":"172.21.15.196:6809/2746224525","cluster_addr":"172.21.15.196:6811/2746224525","heartbeat_back_addr":"172.21.15.196:6815/2746224525","heartbeat_front_addr":"172.21.15.196:6813/2746224525","state":["exists","up"]},{"osd":6,"uuid":"c1b183ff-383f-4c5c-9075-f76bab1c0bb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6819","nonce":1213371331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6821","nonce":1213371331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6825","nonce":1213371331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6823","nonce":1213371331}]},"public_addr":"172.21.15.39:6819/1213371331","cluster_addr":"172.21.15.39:6821/1213371331","heartbeat_back_addr":"172.21.15.39:6825/1213371331","heartbeat_front_addr":"172.21.15.39:6823/1213371331","state":["exists","up"]},{"osd":7,"uuid":"612f8697-957d-4ef4-a87f-241b36f9d13d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6817","nonce":3299162444}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6819","nonce":3299162444}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6823","nonce":3299162444}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6821","nonce":3299162444}]},"public_addr":"172.21.15.194:6817/3299162444","cluster_addr":"172.21.15.194:6819/3299162444","heartbeat_back_addr":"172.21.15.194:6823/3299162444","heartbeat_front_addr":"172.21.15.194:6821/3299162444","state":["exists","up"]},{"osd":8,"uuid":"836a9107-4d4f-4594-b8dc-0301e64f59d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6817","nonce":2456143145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6819","nonce":2456143145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6823","nonce":2456143145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6821","nonce":2456143145}]},"public_addr":"172.21.15.196:6817/2456143145","cluster_addr":"172.21.15.196:6819/2456143145","heartbeat_back_addr":"172.21.15.196:6823/2456143145","heartbeat_front_addr":"172.21.15.196:6821/2456143145","state":["exists","up"]},{"osd":9,"uuid":"4f30b960-12e3-4b16-bd28-fcc988139b58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6827","nonce":1848634433}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6829","nonce":1848634433}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6833","nonce":1848634433}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6831","nonce":1848634433}]},"public_addr":"172.21.15.39:6827/1848634433","cluster_addr":"172.21.15.39:6829/1848634433","heartbeat_back_addr":"172.21.15.39:6833/1848634433","heartbeat_front_addr":"172.21.15.39:6831/1848634433","state":["exists","up"]},{"osd":10,"uuid":"620f90e8-23e6-4bc0-8269-7219b9e1aebd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6825","nonce":3725064809}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6827","nonce":3725064809}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6831","nonce":3725064809}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6829","nonce":3725064809}]},"public_addr":"172.21.15.194:6825/3725064809","cluster_addr":"172.21.15.194:6827/3725064809","heartbeat_back_addr":"172.21.15.194:6831/3725064809","heartbeat_front_addr":"172.21.15.194:6829/3725064809","state":["exists","up"]},{"osd":11,"uuid":"867a2535-8235-4ceb-a698-eb86869d6d20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6825","nonce":2501212422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6827","nonce":2501212422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6831","nonce":2501212422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6829","nonce":2501212422}]},"public_addr":"172.21.15.196:6825/2501212422","cluster_addr":"172.21.15.196:6827/2501212422","heartbeat_back_addr":"172.21.15.196:6831/2501212422","heartbeat_front_addr":"172.21.15.196:6829/2501212422","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:00:36.931566+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:01:02.614024+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:01:26.193265+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:01:58.308833+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:02:25.049546+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:02:49.748750+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:03:23.582153+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:03:50.447321+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:04:15.812129+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:04:49.998047+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:05:17.087924+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:05:43.167637+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.39:0/3482086140":"2024-03-19T15:58:45.340294+0000","172.21.15.39:0/3844306284":"2024-03-19T15:58:45.340294+0000","172.21.15.39:6801/1810238546":"2024-03-19T15:58:45.340294+0000","172.21.15.39:6800/1810238546":"2024-03-19T15:58:45.340294+0000","172.21.15.39:0/2528397488":"2024-03-19T15:58:10.264798+0000","172.21.15.39:0/2240981254":"2024-03-19T15:58:10.264798+0000","172.21.15.39:6800/1217503189":"2024-03-19T15:58:10.264798+0000","172.21.15.39:0/3754876370":"2024-03-19T15:58:10.264798+0000","172.21.15.39:6801/1217503189":"2024-03-19T15:58:10.264798+0000","172.21.15.39:0/1874637628":"2024-03-19T15:58:45.340294+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T16:07:23.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.188+0000 7f1ef1a00640 1 -- 172.21.15.39:0/2946819666 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1ed00a2910 msgr2=0x7f1ed00a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:23.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.188+0000 7f1ef1a00640 1 --2- 172.21.15.39:0/2946819666 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1ed00a2910 0x7f1ed00a4dd0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f1eec19f3b0 tx=0x7f1ed4005f90 comp rx=0 tx=0).stop 2024-03-18T16:07:23.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.188+0000 7f1ef1a00640 1 -- 172.21.15.39:0/2946819666 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1eec0ffe60 msgr2=0x7f1eec19dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:23.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.188+0000 7f1ef1a00640 1 --2- 172.21.15.39:0/2946819666 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1eec0ffe60 0x7f1eec19dea0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f1edc0325f0 tx=0x7f1edc035d10 comp rx=0 tx=0).stop 2024-03-18T16:07:23.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.189+0000 7f1ef1a00640 1 -- 172.21.15.39:0/2946819666 shutdown_connections 2024-03-18T16:07:23.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.189+0000 7f1ef1a00640 1 --2- 172.21.15.39:0/2946819666 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1eec105180 0x7f1eec1a5450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:23.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.189+0000 7f1ef1a00640 1 --2- 172.21.15.39:0/2946819666 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1ed00a2910 0x7f1ed00a4dd0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:23.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.189+0000 7f1ef1a00640 1 --2- 172.21.15.39:0/2946819666 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1eec1027d0 0x7f1eec19e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:23.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.189+0000 7f1ef1a00640 1 --2- 172.21.15.39:0/2946819666 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1eec0ffe60 0x7f1eec19dea0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:23.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.189+0000 7f1ef1a00640 1 -- 172.21.15.39:0/2946819666 >> 172.21.15.39:0/2946819666 conn(0x7f1eec0fba60 msgr2=0x7f1eec103160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:23.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.189+0000 7f1ef1a00640 1 -- 172.21.15.39:0/2946819666 shutdown_connections 2024-03-18T16:07:23.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.189+0000 7f1ef1a00640 1 -- 172.21.15.39:0/2946819666 wait complete. 2024-03-18T16:07:23.210 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-18T16:07:23.211 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T16:07:23.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.421+0000 7f5093e00640 1 -- 172.21.15.39:0/919309277 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f50941007c0 msgr2=0x7f5094100c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:23.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.421+0000 7f5093e00640 1 --2- 172.21.15.39:0/919309277 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f50941007c0 0x7f5094100c20 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5080009900 tx=0x7f508002bc40 comp rx=0 tx=0).stop 2024-03-18T16:07:23.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.421+0000 7f5093e00640 1 -- 172.21.15.39:0/919309277 shutdown_connections 2024-03-18T16:07:23.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.421+0000 7f5093e00640 1 --2- 172.21.15.39:0/919309277 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f50941007c0 0x7f5094100c20 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:23.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.421+0000 7f5093e00640 1 --2- 172.21.15.39:0/919309277 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5094101160 0x7f50941015e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:23.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.421+0000 7f5093e00640 1 --2- 172.21.15.39:0/919309277 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f50940ffe60 0x7f5094100280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:23.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.421+0000 7f5093e00640 1 -- 172.21.15.39:0/919309277 >> 172.21.15.39:0/919309277 conn(0x7f50940fba60 msgr2=0x7f50940fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:23.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.421+0000 7f5093e00640 1 -- 172.21.15.39:0/919309277 shutdown_connections 2024-03-18T16:07:23.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.421+0000 7f5093e00640 1 -- 172.21.15.39:0/919309277 wait complete. 2024-03-18T16:07:23.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.422+0000 7f5093e00640 1 Processor -- start 2024-03-18T16:07:23.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.422+0000 7f5093e00640 1 -- start start 2024-03-18T16:07:23.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.423+0000 7f5093e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f50940ffe60 0x7f5094199a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:23.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.423+0000 7f5093e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f50941007c0 0x7f5094199f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:23.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.423+0000 7f5093e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5094101160 0x7f50941a1000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:23.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.423+0000 7f5093e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f509410f430 con 0x7f5094101160 2024-03-18T16:07:23.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.423+0000 7f5093e00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f509410f2b0 con 0x7f50940ffe60 2024-03-18T16:07:23.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.423+0000 7f5093e00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f509410f5b0 con 0x7f50941007c0 2024-03-18T16:07:23.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.423+0000 7f5092000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f50941007c0 0x7f5094199f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:23.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.423+0000 7f5092000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f50941007c0 0x7f5094199f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:60952/0 (socket says 172.21.15.39:60952) 2024-03-18T16:07:23.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.423+0000 7f5092000640 1 -- 172.21.15.39:0/3539988401 learned_addr learned my addr 172.21.15.39:0/3539988401 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:07:23.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.423+0000 7f5092a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f50940ffe60 0x7f5094199a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:23.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.423+0000 7f5093400640 1 --2- 172.21.15.39:0/3539988401 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5094101160 0x7f50941a1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:23.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.424+0000 7f5092000640 1 -- 172.21.15.39:0/3539988401 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f50940ffe60 msgr2=0x7f5094199a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:23.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.424+0000 7f5092000640 1 --2- 172.21.15.39:0/3539988401 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f50940ffe60 0x7f5094199a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:23.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.424+0000 7f5092000640 1 -- 172.21.15.39:0/3539988401 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5094101160 msgr2=0x7f50941a1000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:23.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.424+0000 7f5092000640 1 --2- 172.21.15.39:0/3539988401 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5094101160 0x7f50941a1000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:23.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.424+0000 7f5092000640 1 -- 172.21.15.39:0/3539988401 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50941a1670 con 0x7f50941007c0 2024-03-18T16:07:23.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.424+0000 7f5092a00640 1 --2- 172.21.15.39:0/3539988401 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f50940ffe60 0x7f5094199a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:07:23.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.424+0000 7f5093400640 1 --2- 172.21.15.39:0/3539988401 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5094101160 0x7f50941a1000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:07:23.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.424+0000 7f5092000640 1 --2- 172.21.15.39:0/3539988401 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f50941007c0 0x7f5094199f90 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f5080009230 tx=0x7f508002c660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:23.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.424+0000 7f508b400640 1 -- 172.21.15.39:0/3539988401 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5080041070 con 0x7f50941007c0 2024-03-18T16:07:23.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.424+0000 7f508b400640 1 -- 172.21.15.39:0/3539988401 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5080004450 con 0x7f50941007c0 2024-03-18T16:07:23.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.424+0000 7f5093e00640 1 -- 172.21.15.39:0/3539988401 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50941a1900 con 0x7f50941007c0 2024-03-18T16:07:23.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.424+0000 7f508b400640 1 -- 172.21.15.39:0/3539988401 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5080003e00 con 0x7f50941007c0 2024-03-18T16:07:23.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.425+0000 7f5093e00640 1 -- 172.21.15.39:0/3539988401 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50941a1df0 con 0x7f50941007c0 2024-03-18T16:07:23.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.426+0000 7f5093e00640 1 -- 172.21.15.39:0/3539988401 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f509410ad00 con 0x7f50941007c0 2024-03-18T16:07:23.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.429+0000 7f508b400640 1 -- 172.21.15.39:0/3539988401 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f5080044070 con 0x7f50941007c0 2024-03-18T16:07:23.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.430+0000 7f508b400640 1 --2- 172.21.15.39:0/3539988401 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f50780a2890 0x7f50780a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:23.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.430+0000 7f508b400640 1 -- 172.21.15.39:0/3539988401 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f50800f0ca0 con 0x7f50941007c0 2024-03-18T16:07:23.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.431+0000 7f5092a00640 1 --2- 172.21.15.39:0/3539988401 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f50780a2890 0x7f50780a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:23.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.431+0000 7f5092a00640 1 --2- 172.21.15.39:0/3539988401 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f50780a2890 0x7f50780a4d50 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5084009fc0 tx=0x7f5084005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:23.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.431+0000 7f508b400640 1 -- 172.21.15.39:0/3539988401 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f50800bdab0 con 0x7f50941007c0 2024-03-18T16:07:23.585 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:23 smithi194 ceph-mon[36200]: pgmap v337: 129 pgs: 129 active+clean; 582 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s rd, 1.6 KiB/s wr, 5 op/s 2024-03-18T16:07:23.585 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:23 smithi194 ceph-mon[36200]: from='client.24859 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:07:23.585 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:23 smithi194 ceph-mon[36200]: from='client.24865 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:07:23.585 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:23 smithi194 ceph-mon[36200]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-03-18T16:07:23.585 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:23 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/3889527143' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:07:23.585 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:23 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/2946819666' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:07:23.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:23 smithi039 ceph-mon[31721]: pgmap v337: 129 pgs: 129 active+clean; 582 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s rd, 1.6 KiB/s wr, 5 op/s 2024-03-18T16:07:23.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:23 smithi039 ceph-mon[31721]: from='client.24859 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:07:23.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:23 smithi039 ceph-mon[31721]: from='client.24865 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:07:23.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:23 smithi039 ceph-mon[31721]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-03-18T16:07:23.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:23 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/3889527143' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:07:23.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:23 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/2946819666' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:07:23.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.631+0000 7f5093e00640 1 -- 172.21.15.39:0/3539988401 --> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5094061c20 con 0x7f50780a2890 2024-03-18T16:07:23.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.654+0000 7f508b400640 1 -- 172.21.15.39:0/3539988401 <== mgr.14152 v2:172.21.15.39:6800/2749214286 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340439 (secure 0 0 0) 0x7f5094061c20 con 0x7f50780a2890 2024-03-18T16:07:23.660 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:07:23.662 INFO:teuthology.orchestra.run.smithi039.stderr:dumped all 2024-03-18T16:07:23.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.661+0000 7f5093e00640 1 -- 172.21.15.39:0/3539988401 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f50780a2890 msgr2=0x7f50780a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:23.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.661+0000 7f5093e00640 1 --2- 172.21.15.39:0/3539988401 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f50780a2890 0x7f50780a4d50 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5084009fc0 tx=0x7f5084005f90 comp rx=0 tx=0).stop 2024-03-18T16:07:23.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.661+0000 7f5093e00640 1 -- 172.21.15.39:0/3539988401 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f50941007c0 msgr2=0x7f5094199f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:23.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.661+0000 7f5093e00640 1 --2- 172.21.15.39:0/3539988401 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f50941007c0 0x7f5094199f90 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f5080009230 tx=0x7f508002c660 comp rx=0 tx=0).stop 2024-03-18T16:07:23.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.661+0000 7f5093e00640 1 -- 172.21.15.39:0/3539988401 shutdown_connections 2024-03-18T16:07:23.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.661+0000 7f5093e00640 1 --2- 172.21.15.39:0/3539988401 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f50780a2890 0x7f50780a4d50 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:23.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.661+0000 7f5093e00640 1 --2- 172.21.15.39:0/3539988401 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5094101160 0x7f50941a1000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:23.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.661+0000 7f5093e00640 1 --2- 172.21.15.39:0/3539988401 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f50941007c0 0x7f5094199f90 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:23.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.661+0000 7f5093e00640 1 --2- 172.21.15.39:0/3539988401 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f50940ffe60 0x7f5094199a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:23.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.661+0000 7f5093e00640 1 -- 172.21.15.39:0/3539988401 >> 172.21.15.39:0/3539988401 conn(0x7f50940fba60 msgr2=0x7f50940fd100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:23.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.661+0000 7f5093e00640 1 -- 172.21.15.39:0/3539988401 shutdown_connections 2024-03-18T16:07:23.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.661+0000 7f5093e00640 1 -- 172.21.15.39:0/3539988401 wait complete. 2024-03-18T16:07:23.683 INFO:teuthology.orchestra.run.smithi039.stdout:{"pg_ready":true,"pg_map":{"version":338,"stamp":"2024-03-18T16:07:23.502398+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":600635,"num_objects":96,"num_object_clones":0,"num_object_copies":290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96,"num_whiteouts":0,"num_read":164,"num_read_kb":139,"num_write":387,"num_write_kb":3008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":62,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":315,"ondisk_log_size":315,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":387740,"kb_used_data":13628,"kb_used_omap":18,"kb_used_meta":373997,"kb_avail":1124554084,"statfs":{"total":1151940427776,"available":1151543382016,"internally_reserved":0,"allocated":13955072,"data_stored":7555264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":382973243},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5627,"num_objects":72,"num_object_clones":0,"num_object_copies":218,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":0,"num_read":38,"num_read_kb":30,"num_write":157,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":48,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006268"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.151383+0000","last_change":"2024-03-18T16:06:44.736511+0000","last_active":"2024-03-18T16:06:50.151383+0000","last_peered":"2024-03-18T16:06:50.151383+0000","last_clean":"2024-03-18T16:06:50.151383+0000","last_became_active":"2024-03-18T16:06:44.736182+0000","last_became_peered":"2024-03-18T16:06:44.736182+0000","last_unstale":"2024-03-18T16:06:50.151383+0000","last_undegraded":"2024-03-18T16:06:50.151383+0000","last_fullsized":"2024-03-18T16:06:50.151383+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:08:57.060268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.288006+0000","last_change":"2024-03-18T16:06:44.733658+0000","last_active":"2024-03-18T16:06:50.288006+0000","last_peered":"2024-03-18T16:06:50.288006+0000","last_clean":"2024-03-18T16:06:50.288006+0000","last_became_active":"2024-03-18T16:06:44.733279+0000","last_became_peered":"2024-03-18T16:06:44.733279+0000","last_unstale":"2024-03-18T16:06:50.288006+0000","last_undegraded":"2024-03-18T16:06:50.288006+0000","last_fullsized":"2024-03-18T16:06:50.288006+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:59:46.783025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.286651+0000","last_change":"2024-03-18T16:06:44.723949+0000","last_active":"2024-03-18T16:06:50.286651+0000","last_peered":"2024-03-18T16:06:50.286651+0000","last_clean":"2024-03-18T16:06:50.286651+0000","last_became_active":"2024-03-18T16:06:44.723614+0000","last_became_peered":"2024-03-18T16:06:44.723614+0000","last_unstale":"2024-03-18T16:06:50.286651+0000","last_undegraded":"2024-03-18T16:06:50.286651+0000","last_fullsized":"2024-03-18T16:06:50.286651+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:03:17.876097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.265883+0000","last_change":"2024-03-18T16:06:44.711058+0000","last_active":"2024-03-18T16:06:50.265883+0000","last_peered":"2024-03-18T16:06:50.265883+0000","last_clean":"2024-03-18T16:06:50.265883+0000","last_became_active":"2024-03-18T16:06:44.708873+0000","last_became_peered":"2024-03-18T16:06:44.708873+0000","last_unstale":"2024-03-18T16:06:50.265883+0000","last_undegraded":"2024-03-18T16:06:50.265883+0000","last_fullsized":"2024-03-18T16:06:50.265883+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:58:08.182651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.285682+0000","last_change":"2024-03-18T16:06:44.721480+0000","last_active":"2024-03-18T16:06:50.285682+0000","last_peered":"2024-03-18T16:06:50.285682+0000","last_clean":"2024-03-18T16:06:50.285682+0000","last_became_active":"2024-03-18T16:06:44.721101+0000","last_became_peered":"2024-03-18T16:06:44.721101+0000","last_unstale":"2024-03-18T16:06:50.285682+0000","last_undegraded":"2024-03-18T16:06:50.285682+0000","last_fullsized":"2024-03-18T16:06:50.285682+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:19:43.341221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.014165+0000","last_change":"2024-03-18T16:06:44.716443+0000","last_active":"2024-03-18T16:06:50.014165+0000","last_peered":"2024-03-18T16:06:50.014165+0000","last_clean":"2024-03-18T16:06:50.014165+0000","last_became_active":"2024-03-18T16:06:44.715969+0000","last_became_peered":"2024-03-18T16:06:44.715969+0000","last_unstale":"2024-03-18T16:06:50.014165+0000","last_undegraded":"2024-03-18T16:06:50.014165+0000","last_fullsized":"2024-03-18T16:06:50.014165+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:57:14.714415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.043165+0000","last_change":"2024-03-18T16:06:44.711439+0000","last_active":"2024-03-18T16:06:50.043165+0000","last_peered":"2024-03-18T16:06:50.043165+0000","last_clean":"2024-03-18T16:06:50.043165+0000","last_became_active":"2024-03-18T16:06:44.711032+0000","last_became_peered":"2024-03-18T16:06:44.711032+0000","last_unstale":"2024-03-18T16:06:50.043165+0000","last_undegraded":"2024-03-18T16:06:50.043165+0000","last_fullsized":"2024-03-18T16:06:50.043165+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:19:56.516867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.286621+0000","last_change":"2024-03-18T16:06:44.723629+0000","last_active":"2024-03-18T16:06:50.286621+0000","last_peered":"2024-03-18T16:06:50.286621+0000","last_clean":"2024-03-18T16:06:50.286621+0000","last_became_active":"2024-03-18T16:06:44.721789+0000","last_became_peered":"2024-03-18T16:06:44.721789+0000","last_unstale":"2024-03-18T16:06:50.286621+0000","last_undegraded":"2024-03-18T16:06:50.286621+0000","last_fullsized":"2024-03-18T16:06:50.286621+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:07:59.653357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.285652+0000","last_change":"2024-03-18T16:06:44.710978+0000","last_active":"2024-03-18T16:06:50.285652+0000","last_peered":"2024-03-18T16:06:50.285652+0000","last_clean":"2024-03-18T16:06:50.285652+0000","last_became_active":"2024-03-18T16:06:44.710520+0000","last_became_peered":"2024-03-18T16:06:44.710520+0000","last_unstale":"2024-03-18T16:06:50.285652+0000","last_undegraded":"2024-03-18T16:06:50.285652+0000","last_fullsized":"2024-03-18T16:06:50.285652+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:54:53.234025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.014645+0000","last_change":"2024-03-18T16:06:44.716231+0000","last_active":"2024-03-18T16:06:50.014645+0000","last_peered":"2024-03-18T16:06:50.014645+0000","last_clean":"2024-03-18T16:06:50.014645+0000","last_became_active":"2024-03-18T16:06:44.714900+0000","last_became_peered":"2024-03-18T16:06:44.714900+0000","last_unstale":"2024-03-18T16:06:50.014645+0000","last_undegraded":"2024-03-18T16:06:50.014645+0000","last_fullsized":"2024-03-18T16:06:50.014645+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:38:54.964729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.043595+0000","last_change":"2024-03-18T16:06:44.710942+0000","last_active":"2024-03-18T16:06:50.043595+0000","last_peered":"2024-03-18T16:06:50.043595+0000","last_clean":"2024-03-18T16:06:50.043595+0000","last_became_active":"2024-03-18T16:06:44.710588+0000","last_became_peered":"2024-03-18T16:06:44.710588+0000","last_unstale":"2024-03-18T16:06:50.043595+0000","last_undegraded":"2024-03-18T16:06:50.043595+0000","last_fullsized":"2024-03-18T16:06:50.043595+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:43:19.320034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.265773+0000","last_change":"2024-03-18T16:06:44.714763+0000","last_active":"2024-03-18T16:06:50.265773+0000","last_peered":"2024-03-18T16:06:50.265773+0000","last_clean":"2024-03-18T16:06:50.265773+0000","last_became_active":"2024-03-18T16:06:44.714031+0000","last_became_peered":"2024-03-18T16:06:44.714031+0000","last_unstale":"2024-03-18T16:06:50.265773+0000","last_undegraded":"2024-03-18T16:06:50.265773+0000","last_fullsized":"2024-03-18T16:06:50.265773+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:19:25.917093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.286907+0000","last_change":"2024-03-18T16:06:44.710062+0000","last_active":"2024-03-18T16:06:50.286907+0000","last_peered":"2024-03-18T16:06:50.286907+0000","last_clean":"2024-03-18T16:06:50.286907+0000","last_became_active":"2024-03-18T16:06:44.709281+0000","last_became_peered":"2024-03-18T16:06:44.709281+0000","last_unstale":"2024-03-18T16:06:50.286907+0000","last_undegraded":"2024-03-18T16:06:50.286907+0000","last_fullsized":"2024-03-18T16:06:50.286907+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:51:30.100947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.285918+0000","last_change":"2024-03-18T16:06:44.734178+0000","last_active":"2024-03-18T16:06:50.285918+0000","last_peered":"2024-03-18T16:06:50.285918+0000","last_clean":"2024-03-18T16:06:50.285918+0000","last_became_active":"2024-03-18T16:06:44.732538+0000","last_became_peered":"2024-03-18T16:06:44.732538+0000","last_unstale":"2024-03-18T16:06:50.285918+0000","last_undegraded":"2024-03-18T16:06:50.285918+0000","last_fullsized":"2024-03-18T16:06:50.285918+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:14:26.904402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.285701+0000","last_change":"2024-03-18T16:06:44.734383+0000","last_active":"2024-03-18T16:06:50.285701+0000","last_peered":"2024-03-18T16:06:50.285701+0000","last_clean":"2024-03-18T16:06:50.285701+0000","last_became_active":"2024-03-18T16:06:44.731404+0000","last_became_peered":"2024-03-18T16:06:44.731404+0000","last_unstale":"2024-03-18T16:06:50.285701+0000","last_undegraded":"2024-03-18T16:06:50.285701+0000","last_fullsized":"2024-03-18T16:06:50.285701+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:57:03.510629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.285098+0000","last_change":"2024-03-18T16:06:44.728165+0000","last_active":"2024-03-18T16:06:50.285098+0000","last_peered":"2024-03-18T16:06:50.285098+0000","last_clean":"2024-03-18T16:06:50.285098+0000","last_became_active":"2024-03-18T16:06:44.727842+0000","last_became_peered":"2024-03-18T16:06:44.727842+0000","last_unstale":"2024-03-18T16:06:50.285098+0000","last_undegraded":"2024-03-18T16:06:50.285098+0000","last_fullsized":"2024-03-18T16:06:50.285098+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:55:04.464916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.151190+0000","last_change":"2024-03-18T16:06:44.730893+0000","last_active":"2024-03-18T16:06:50.151190+0000","last_peered":"2024-03-18T16:06:50.151190+0000","last_clean":"2024-03-18T16:06:50.151190+0000","last_became_active":"2024-03-18T16:06:44.730544+0000","last_became_peered":"2024-03-18T16:06:44.730544+0000","last_unstale":"2024-03-18T16:06:50.151190+0000","last_undegraded":"2024-03-18T16:06:50.151190+0000","last_fullsized":"2024-03-18T16:06:50.151190+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:16:37.918096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.019939+0000","last_change":"2024-03-18T16:06:44.706036+0000","last_active":"2024-03-18T16:06:50.019939+0000","last_peered":"2024-03-18T16:06:50.019939+0000","last_clean":"2024-03-18T16:06:50.019939+0000","last_became_active":"2024-03-18T16:06:44.705281+0000","last_became_peered":"2024-03-18T16:06:44.705281+0000","last_unstale":"2024-03-18T16:06:50.019939+0000","last_undegraded":"2024-03-18T16:06:50.019939+0000","last_fullsized":"2024-03-18T16:06:50.019939+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:52:20.202110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.015179+0000","last_change":"2024-03-18T16:06:44.727543+0000","last_active":"2024-03-18T16:06:50.015179+0000","last_peered":"2024-03-18T16:06:50.015179+0000","last_clean":"2024-03-18T16:06:50.015179+0000","last_became_active":"2024-03-18T16:06:44.727135+0000","last_became_peered":"2024-03-18T16:06:44.727135+0000","last_unstale":"2024-03-18T16:06:50.015179+0000","last_undegraded":"2024-03-18T16:06:50.015179+0000","last_fullsized":"2024-03-18T16:06:50.015179+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:52:42.742279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.014611+0000","last_change":"2024-03-18T16:06:49.028004+0000","last_active":"2024-03-18T16:06:50.014611+0000","last_peered":"2024-03-18T16:06:50.014611+0000","last_clean":"2024-03-18T16:06:50.014611+0000","last_became_active":"2024-03-18T16:06:49.027640+0000","last_became_peered":"2024-03-18T16:06:49.027640+0000","last_unstale":"2024-03-18T16:06:50.014611+0000","last_undegraded":"2024-03-18T16:06:50.014611+0000","last_fullsized":"2024-03-18T16:06:50.014611+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:23:07.926349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'13","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:07:20.006936+0000","last_change":"2024-03-18T16:06:43.709540+0000","last_active":"2024-03-18T16:07:20.006936+0000","last_peered":"2024-03-18T16:07:20.006936+0000","last_clean":"2024-03-18T16:07:20.006936+0000","last_became_active":"2024-03-18T16:06:43.709170+0000","last_became_peered":"2024-03-18T16:06:43.709170+0000","last_unstale":"2024-03-18T16:07:20.006936+0000","last_undegraded":"2024-03-18T16:07:20.006936+0000","last_fullsized":"2024-03-18T16:07:20.006936+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:12:53.934079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.288197+0000","last_change":"2024-03-18T16:06:44.730192+0000","last_active":"2024-03-18T16:06:50.288197+0000","last_peered":"2024-03-18T16:06:50.288197+0000","last_clean":"2024-03-18T16:06:50.288197+0000","last_became_active":"2024-03-18T16:06:44.728071+0000","last_became_peered":"2024-03-18T16:06:44.728071+0000","last_unstale":"2024-03-18T16:06:50.288197+0000","last_undegraded":"2024-03-18T16:06:50.288197+0000","last_fullsized":"2024-03-18T16:06:50.288197+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:14:59.992958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.265629+0000","last_change":"2024-03-18T16:06:49.023963+0000","last_active":"2024-03-18T16:06:50.265629+0000","last_peered":"2024-03-18T16:06:50.265629+0000","last_clean":"2024-03-18T16:06:50.265629+0000","last_became_active":"2024-03-18T16:06:49.023568+0000","last_became_peered":"2024-03-18T16:06:49.023568+0000","last_unstale":"2024-03-18T16:06:50.265629+0000","last_undegraded":"2024-03-18T16:06:50.265629+0000","last_fullsized":"2024-03-18T16:06:50.265629+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:15:19.730842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.288348+0000","last_change":"2024-03-18T16:06:43.722167+0000","last_active":"2024-03-18T16:06:50.288348+0000","last_peered":"2024-03-18T16:06:50.288348+0000","last_clean":"2024-03-18T16:06:50.288348+0000","last_became_active":"2024-03-18T16:06:43.721782+0000","last_became_peered":"2024-03-18T16:06:43.721782+0000","last_unstale":"2024-03-18T16:06:50.288348+0000","last_undegraded":"2024-03-18T16:06:50.288348+0000","last_fullsized":"2024-03-18T16:06:50.288348+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:08:54.400637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.287490+0000","last_change":"2024-03-18T16:06:44.729666+0000","last_active":"2024-03-18T16:06:50.287490+0000","last_peered":"2024-03-18T16:06:50.287490+0000","last_clean":"2024-03-18T16:06:50.287490+0000","last_became_active":"2024-03-18T16:06:44.728842+0000","last_became_peered":"2024-03-18T16:06:44.728842+0000","last_unstale":"2024-03-18T16:06:50.287490+0000","last_undegraded":"2024-03-18T16:06:50.287490+0000","last_fullsized":"2024-03-18T16:06:50.287490+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:01:42.438000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.265815+0000","last_change":"2024-03-18T16:06:49.031169+0000","last_active":"2024-03-18T16:06:50.265815+0000","last_peered":"2024-03-18T16:06:50.265815+0000","last_clean":"2024-03-18T16:06:50.265815+0000","last_became_active":"2024-03-18T16:06:49.030801+0000","last_became_peered":"2024-03-18T16:06:49.030801+0000","last_unstale":"2024-03-18T16:06:50.265815+0000","last_undegraded":"2024-03-18T16:06:50.265815+0000","last_fullsized":"2024-03-18T16:06:50.265815+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:10:54.880882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'12","reported_seq":35,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:07:20.004185+0000","last_change":"2024-03-18T16:06:43.707111+0000","last_active":"2024-03-18T16:07:20.004185+0000","last_peered":"2024-03-18T16:07:20.004185+0000","last_clean":"2024-03-18T16:07:20.004185+0000","last_became_active":"2024-03-18T16:06:43.706671+0000","last_became_peered":"2024-03-18T16:06:43.706671+0000","last_unstale":"2024-03-18T16:07:20.004185+0000","last_undegraded":"2024-03-18T16:07:20.004185+0000","last_fullsized":"2024-03-18T16:07:20.004185+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:18:34.295641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.285292+0000","last_change":"2024-03-18T16:06:44.722100+0000","last_active":"2024-03-18T16:06:50.285292+0000","last_peered":"2024-03-18T16:06:50.285292+0000","last_clean":"2024-03-18T16:06:50.285292+0000","last_became_active":"2024-03-18T16:06:44.721673+0000","last_became_peered":"2024-03-18T16:06:44.721673+0000","last_unstale":"2024-03-18T16:06:50.285292+0000","last_undegraded":"2024-03-18T16:06:50.285292+0000","last_fullsized":"2024-03-18T16:06:50.285292+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:41:22.624778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.265583+0000","last_change":"2024-03-18T16:06:49.027966+0000","last_active":"2024-03-18T16:06:50.265583+0000","last_peered":"2024-03-18T16:06:50.265583+0000","last_clean":"2024-03-18T16:06:50.265583+0000","last_became_active":"2024-03-18T16:06:49.027567+0000","last_became_peered":"2024-03-18T16:06:49.027567+0000","last_unstale":"2024-03-18T16:06:50.265583+0000","last_undegraded":"2024-03-18T16:06:50.265583+0000","last_fullsized":"2024-03-18T16:06:50.265583+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:54:28.543019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:07:20.003614+0000","last_change":"2024-03-18T16:06:43.704023+0000","last_active":"2024-03-18T16:07:20.003614+0000","last_peered":"2024-03-18T16:07:20.003614+0000","last_clean":"2024-03-18T16:07:20.003614+0000","last_became_active":"2024-03-18T16:06:43.703690+0000","last_became_peered":"2024-03-18T16:06:43.703690+0000","last_unstale":"2024-03-18T16:07:20.003614+0000","last_undegraded":"2024-03-18T16:07:20.003614+0000","last_fullsized":"2024-03-18T16:07:20.003614+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:20:02.344346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.285990+0000","last_change":"2024-03-18T16:06:44.708803+0000","last_active":"2024-03-18T16:06:50.285990+0000","last_peered":"2024-03-18T16:06:50.285990+0000","last_clean":"2024-03-18T16:06:50.285990+0000","last_became_active":"2024-03-18T16:06:44.707863+0000","last_became_peered":"2024-03-18T16:06:44.707863+0000","last_unstale":"2024-03-18T16:06:50.285990+0000","last_undegraded":"2024-03-18T16:06:50.285990+0000","last_fullsized":"2024-03-18T16:06:50.285990+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:40:58.800490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.015091+0000","last_change":"2024-03-18T16:06:49.012310+0000","last_active":"2024-03-18T16:06:50.015091+0000","last_peered":"2024-03-18T16:06:50.015091+0000","last_clean":"2024-03-18T16:06:50.015091+0000","last_became_active":"2024-03-18T16:06:49.011844+0000","last_became_peered":"2024-03-18T16:06:49.011844+0000","last_unstale":"2024-03-18T16:06:50.015091+0000","last_undegraded":"2024-03-18T16:06:50.015091+0000","last_fullsized":"2024-03-18T16:06:50.015091+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:16:42.652205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'6","reported_seq":34,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:07:20.983908+0000","last_change":"2024-03-18T16:06:43.705653+0000","last_active":"2024-03-18T16:07:20.983908+0000","last_peered":"2024-03-18T16:07:20.983908+0000","last_clean":"2024-03-18T16:07:20.983908+0000","last_became_active":"2024-03-18T16:06:43.705320+0000","last_became_peered":"2024-03-18T16:06:43.705320+0000","last_unstale":"2024-03-18T16:07:20.983908+0000","last_undegraded":"2024-03-18T16:07:20.983908+0000","last_fullsized":"2024-03-18T16:07:20.983908+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:10:10.827338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.286101+0000","last_change":"2024-03-18T16:06:44.720045+0000","last_active":"2024-03-18T16:06:50.286101+0000","last_peered":"2024-03-18T16:06:50.286101+0000","last_clean":"2024-03-18T16:06:50.286101+0000","last_became_active":"2024-03-18T16:06:44.719669+0000","last_became_peered":"2024-03-18T16:06:44.719669+0000","last_unstale":"2024-03-18T16:06:50.286101+0000","last_undegraded":"2024-03-18T16:06:50.286101+0000","last_fullsized":"2024-03-18T16:06:50.286101+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:23:47.464057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.014824+0000","last_change":"2024-03-18T16:06:49.032576+0000","last_active":"2024-03-18T16:06:50.014824+0000","last_peered":"2024-03-18T16:06:50.014824+0000","last_clean":"2024-03-18T16:06:50.014824+0000","last_became_active":"2024-03-18T16:06:49.032126+0000","last_became_peered":"2024-03-18T16:06:49.032126+0000","last_unstale":"2024-03-18T16:06:50.014824+0000","last_undegraded":"2024-03-18T16:06:50.014824+0000","last_fullsized":"2024-03-18T16:06:50.014824+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:41:54.822070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:07:19.242451+0000","last_change":"2024-03-18T16:06:43.706364+0000","last_active":"2024-03-18T16:07:19.242451+0000","last_peered":"2024-03-18T16:07:19.242451+0000","last_clean":"2024-03-18T16:07:19.242451+0000","last_became_active":"2024-03-18T16:06:43.705984+0000","last_became_peered":"2024-03-18T16:06:43.705984+0000","last_unstale":"2024-03-18T16:07:19.242451+0000","last_undegraded":"2024-03-18T16:07:19.242451+0000","last_fullsized":"2024-03-18T16:07:19.242451+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:46:06.503000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.286117+0000","last_change":"2024-03-18T16:06:44.721019+0000","last_active":"2024-03-18T16:06:50.286117+0000","last_peered":"2024-03-18T16:06:50.286117+0000","last_clean":"2024-03-18T16:06:50.286117+0000","last_became_active":"2024-03-18T16:06:44.720665+0000","last_became_peered":"2024-03-18T16:06:44.720665+0000","last_unstale":"2024-03-18T16:06:50.286117+0000","last_undegraded":"2024-03-18T16:06:50.286117+0000","last_fullsized":"2024-03-18T16:06:50.286117+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:58:34.180653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.151293+0000","last_change":"2024-03-18T16:06:49.022919+0000","last_active":"2024-03-18T16:06:50.151293+0000","last_peered":"2024-03-18T16:06:50.151293+0000","last_clean":"2024-03-18T16:06:50.151293+0000","last_became_active":"2024-03-18T16:06:49.022514+0000","last_became_peered":"2024-03-18T16:06:49.022514+0000","last_unstale":"2024-03-18T16:06:50.151293+0000","last_undegraded":"2024-03-18T16:06:50.151293+0000","last_fullsized":"2024-03-18T16:06:50.151293+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:46:39.510181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:07:18.229817+0000","last_change":"2024-03-18T16:06:43.713025+0000","last_active":"2024-03-18T16:07:18.229817+0000","last_peered":"2024-03-18T16:07:18.229817+0000","last_clean":"2024-03-18T16:07:18.229817+0000","last_became_active":"2024-03-18T16:06:43.711469+0000","last_became_peered":"2024-03-18T16:06:43.711469+0000","last_unstale":"2024-03-18T16:07:18.229817+0000","last_undegraded":"2024-03-18T16:07:18.229817+0000","last_fullsized":"2024-03-18T16:07:18.229817+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:30:41.193229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.286584+0000","last_change":"2024-03-18T16:06:44.723953+0000","last_active":"2024-03-18T16:06:50.286584+0000","last_peered":"2024-03-18T16:06:50.286584+0000","last_clean":"2024-03-18T16:06:50.286584+0000","last_became_active":"2024-03-18T16:06:44.723266+0000","last_became_peered":"2024-03-18T16:06:44.723266+0000","last_unstale":"2024-03-18T16:06:50.286584+0000","last_undegraded":"2024-03-18T16:06:50.286584+0000","last_fullsized":"2024-03-18T16:06:50.286584+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:41:10.067297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.287077+0000","last_change":"2024-03-18T16:06:49.039472+0000","last_active":"2024-03-18T16:06:50.287077+0000","last_peered":"2024-03-18T16:06:50.287077+0000","last_clean":"2024-03-18T16:06:50.287077+0000","last_became_active":"2024-03-18T16:06:49.038889+0000","last_became_peered":"2024-03-18T16:06:49.038889+0000","last_unstale":"2024-03-18T16:06:50.287077+0000","last_undegraded":"2024-03-18T16:06:50.287077+0000","last_fullsized":"2024-03-18T16:06:50.287077+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:03:05.345315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:07:19.980692+0000","last_change":"2024-03-18T16:06:43.705873+0000","last_active":"2024-03-18T16:07:19.980692+0000","last_peered":"2024-03-18T16:07:19.980692+0000","last_clean":"2024-03-18T16:07:19.980692+0000","last_became_active":"2024-03-18T16:06:43.705548+0000","last_became_peered":"2024-03-18T16:06:43.705548+0000","last_unstale":"2024-03-18T16:07:19.980692+0000","last_undegraded":"2024-03-18T16:07:19.980692+0000","last_fullsized":"2024-03-18T16:07:19.980692+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:52:03.790888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.018009+0000","last_change":"2024-03-18T16:06:44.718526+0000","last_active":"2024-03-18T16:06:50.018009+0000","last_peered":"2024-03-18T16:06:50.018009+0000","last_clean":"2024-03-18T16:06:50.018009+0000","last_became_active":"2024-03-18T16:06:44.717266+0000","last_became_peered":"2024-03-18T16:06:44.717266+0000","last_unstale":"2024-03-18T16:06:50.018009+0000","last_undegraded":"2024-03-18T16:06:50.018009+0000","last_fullsized":"2024-03-18T16:06:50.018009+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:55:09.340354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.287346+0000","last_change":"2024-03-18T16:06:49.021971+0000","last_active":"2024-03-18T16:06:50.287346+0000","last_peered":"2024-03-18T16:06:50.287346+0000","last_clean":"2024-03-18T16:06:50.287346+0000","last_became_active":"2024-03-18T16:06:49.021577+0000","last_became_peered":"2024-03-18T16:06:49.021577+0000","last_unstale":"2024-03-18T16:06:50.287346+0000","last_undegraded":"2024-03-18T16:06:50.287346+0000","last_fullsized":"2024-03-18T16:06:50.287346+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:47:44.768559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'4","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:07:17.207443+0000","last_change":"2024-03-18T16:06:43.705636+0000","last_active":"2024-03-18T16:07:17.207443+0000","last_peered":"2024-03-18T16:07:17.207443+0000","last_clean":"2024-03-18T16:07:17.207443+0000","last_became_active":"2024-03-18T16:06:43.705268+0000","last_became_peered":"2024-03-18T16:06:43.705268+0000","last_unstale":"2024-03-18T16:07:17.207443+0000","last_undegraded":"2024-03-18T16:07:17.207443+0000","last_fullsized":"2024-03-18T16:07:17.207443+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:37:31.367130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.285718+0000","last_change":"2024-03-18T16:06:44.712237+0000","last_active":"2024-03-18T16:06:50.285718+0000","last_peered":"2024-03-18T16:06:50.285718+0000","last_clean":"2024-03-18T16:06:50.285718+0000","last_became_active":"2024-03-18T16:06:44.711677+0000","last_became_peered":"2024-03-18T16:06:44.711677+0000","last_unstale":"2024-03-18T16:06:50.285718+0000","last_undegraded":"2024-03-18T16:06:50.285718+0000","last_fullsized":"2024-03-18T16:06:50.285718+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:16:56.326962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.018043+0000","last_change":"2024-03-18T16:06:49.034578+0000","last_active":"2024-03-18T16:06:50.018043+0000","last_peered":"2024-03-18T16:06:50.018043+0000","last_clean":"2024-03-18T16:06:50.018043+0000","last_became_active":"2024-03-18T16:06:49.031099+0000","last_became_peered":"2024-03-18T16:06:49.031099+0000","last_unstale":"2024-03-18T16:06:50.018043+0000","last_undegraded":"2024-03-18T16:06:50.018043+0000","last_fullsized":"2024-03-18T16:06:50.018043+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:11:08.042550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:07:20.004223+0000","last_change":"2024-03-18T16:06:43.708151+0000","last_active":"2024-03-18T16:07:20.004223+0000","last_peered":"2024-03-18T16:07:20.004223+0000","last_clean":"2024-03-18T16:07:20.004223+0000","last_became_active":"2024-03-18T16:06:43.707810+0000","last_became_peered":"2024-03-18T16:06:43.707810+0000","last_unstale":"2024-03-18T16:07:20.004223+0000","last_undegraded":"2024-03-18T16:07:20.004223+0000","last_fullsized":"2024-03-18T16:07:20.004223+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:30:35.745772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.287628+0000","last_change":"2024-03-18T16:06:44.729777+0000","last_active":"2024-03-18T16:06:50.287628+0000","last_peered":"2024-03-18T16:06:50.287628+0000","last_clean":"2024-03-18T16:06:50.287628+0000","last_became_active":"2024-03-18T16:06:44.729244+0000","last_became_peered":"2024-03-18T16:06:44.729244+0000","last_unstale":"2024-03-18T16:06:50.287628+0000","last_undegraded":"2024-03-18T16:06:50.287628+0000","last_fullsized":"2024-03-18T16:06:50.287628+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:52:05.420443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.265917+0000","last_change":"2024-03-18T16:06:49.014087+0000","last_active":"2024-03-18T16:06:50.265917+0000","last_peered":"2024-03-18T16:06:50.265917+0000","last_clean":"2024-03-18T16:06:50.265917+0000","last_became_active":"2024-03-18T16:06:49.013556+0000","last_became_peered":"2024-03-18T16:06:49.013556+0000","last_unstale":"2024-03-18T16:06:50.265917+0000","last_undegraded":"2024-03-18T16:06:50.265917+0000","last_fullsized":"2024-03-18T16:06:50.265917+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:13:59.638977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:07:19.247654+0000","last_change":"2024-03-18T16:06:43.715280+0000","last_active":"2024-03-18T16:07:19.247654+0000","last_peered":"2024-03-18T16:07:19.247654+0000","last_clean":"2024-03-18T16:07:19.247654+0000","last_became_active":"2024-03-18T16:06:43.714936+0000","last_became_peered":"2024-03-18T16:06:43.714936+0000","last_unstale":"2024-03-18T16:07:19.247654+0000","last_undegraded":"2024-03-18T16:07:19.247654+0000","last_fullsized":"2024-03-18T16:07:19.247654+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:07:38.746932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.285428+0000","last_change":"2024-03-18T16:06:44.709004+0000","last_active":"2024-03-18T16:06:50.285428+0000","last_peered":"2024-03-18T16:06:50.285428+0000","last_clean":"2024-03-18T16:06:50.285428+0000","last_became_active":"2024-03-18T16:06:44.708574+0000","last_became_peered":"2024-03-18T16:06:44.708574+0000","last_unstale":"2024-03-18T16:06:50.285428+0000","last_undegraded":"2024-03-18T16:06:50.285428+0000","last_fullsized":"2024-03-18T16:06:50.285428+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:02:50.982110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.018789+0000","last_change":"2024-03-18T16:06:49.029806+0000","last_active":"2024-03-18T16:06:50.018789+0000","last_peered":"2024-03-18T16:06:50.018789+0000","last_clean":"2024-03-18T16:06:50.018789+0000","last_became_active":"2024-03-18T16:06:49.029404+0000","last_became_peered":"2024-03-18T16:06:49.029404+0000","last_unstale":"2024-03-18T16:06:50.018789+0000","last_undegraded":"2024-03-18T16:06:50.018789+0000","last_fullsized":"2024-03-18T16:06:50.018789+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:38:37.706620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'12","reported_seq":34,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:07:17.201229+0000","last_change":"2024-03-18T16:06:43.702910+0000","last_active":"2024-03-18T16:07:17.201229+0000","last_peered":"2024-03-18T16:07:17.201229+0000","last_clean":"2024-03-18T16:07:17.201229+0000","last_became_active":"2024-03-18T16:06:43.702557+0000","last_became_peered":"2024-03-18T16:06:43.702557+0000","last_unstale":"2024-03-18T16:07:17.201229+0000","last_undegraded":"2024-03-18T16:07:17.201229+0000","last_fullsized":"2024-03-18T16:07:17.201229+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:55:15.362987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.285820+0000","last_change":"2024-03-18T16:06:44.709075+0000","last_active":"2024-03-18T16:06:50.285820+0000","last_peered":"2024-03-18T16:06:50.285820+0000","last_clean":"2024-03-18T16:06:50.285820+0000","last_became_active":"2024-03-18T16:06:44.708366+0000","last_became_peered":"2024-03-18T16:06:44.708366+0000","last_unstale":"2024-03-18T16:06:50.285820+0000","last_undegraded":"2024-03-18T16:06:50.285820+0000","last_fullsized":"2024-03-18T16:06:50.285820+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:44:18.486385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.286659+0000","last_change":"2024-03-18T16:06:49.037380+0000","last_active":"2024-03-18T16:06:50.286659+0000","last_peered":"2024-03-18T16:06:50.286659+0000","last_clean":"2024-03-18T16:06:50.286659+0000","last_became_active":"2024-03-18T16:06:49.037021+0000","last_became_peered":"2024-03-18T16:06:49.037021+0000","last_unstale":"2024-03-18T16:06:50.286659+0000","last_undegraded":"2024-03-18T16:06:50.286659+0000","last_fullsized":"2024-03-18T16:06:50.286659+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:43:03.482631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:07:18.225050+0000","last_change":"2024-03-18T16:06:43.722287+0000","last_active":"2024-03-18T16:07:18.225050+0000","last_peered":"2024-03-18T16:07:18.225050+0000","last_clean":"2024-03-18T16:07:18.225050+0000","last_became_active":"2024-03-18T16:06:43.721919+0000","last_became_peered":"2024-03-18T16:06:43.721919+0000","last_unstale":"2024-03-18T16:07:18.225050+0000","last_undegraded":"2024-03-18T16:07:18.225050+0000","last_fullsized":"2024-03-18T16:07:18.225050+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:10:07.606112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.288783+0000","last_change":"2024-03-18T16:06:44.727204+0000","last_active":"2024-03-18T16:06:50.288783+0000","last_peered":"2024-03-18T16:06:50.288783+0000","last_clean":"2024-03-18T16:06:50.288783+0000","last_became_active":"2024-03-18T16:06:44.726775+0000","last_became_peered":"2024-03-18T16:06:44.726775+0000","last_unstale":"2024-03-18T16:06:50.288783+0000","last_undegraded":"2024-03-18T16:06:50.288783+0000","last_fullsized":"2024-03-18T16:06:50.288783+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:29:44.842972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.151291+0000","last_change":"2024-03-18T16:06:49.022508+0000","last_active":"2024-03-18T16:06:50.151291+0000","last_peered":"2024-03-18T16:06:50.151291+0000","last_clean":"2024-03-18T16:06:50.151291+0000","last_became_active":"2024-03-18T16:06:49.022017+0000","last_became_peered":"2024-03-18T16:06:49.022017+0000","last_unstale":"2024-03-18T16:06:50.151291+0000","last_undegraded":"2024-03-18T16:06:50.151291+0000","last_fullsized":"2024-03-18T16:06:50.151291+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:52:45.892880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:07:20.006950+0000","last_change":"2024-03-18T16:06:43.719618+0000","last_active":"2024-03-18T16:07:20.006950+0000","last_peered":"2024-03-18T16:07:20.006950+0000","last_clean":"2024-03-18T16:07:20.006950+0000","last_became_active":"2024-03-18T16:06:43.719161+0000","last_became_peered":"2024-03-18T16:06:43.719161+0000","last_unstale":"2024-03-18T16:07:20.006950+0000","last_undegraded":"2024-03-18T16:07:20.006950+0000","last_fullsized":"2024-03-18T16:07:20.006950+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:39:53.077878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.043367+0000","last_change":"2024-03-18T16:06:44.717829+0000","last_active":"2024-03-18T16:06:50.043367+0000","last_peered":"2024-03-18T16:06:50.043367+0000","last_clean":"2024-03-18T16:06:50.043367+0000","last_became_active":"2024-03-18T16:06:44.717473+0000","last_became_peered":"2024-03-18T16:06:44.717473+0000","last_unstale":"2024-03-18T16:06:50.043367+0000","last_undegraded":"2024-03-18T16:06:50.043367+0000","last_fullsized":"2024-03-18T16:06:50.043367+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:10:00.183613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.019958+0000","last_change":"2024-03-18T16:06:49.036766+0000","last_active":"2024-03-18T16:06:50.019958+0000","last_peered":"2024-03-18T16:06:50.019958+0000","last_clean":"2024-03-18T16:06:50.019958+0000","last_became_active":"2024-03-18T16:06:49.034617+0000","last_became_peered":"2024-03-18T16:06:49.034617+0000","last_unstale":"2024-03-18T16:06:50.019958+0000","last_undegraded":"2024-03-18T16:06:50.019958+0000","last_fullsized":"2024-03-18T16:06:50.019958+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:18:59.033781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'5","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:07:20.985685+0000","last_change":"2024-03-18T16:06:43.702452+0000","last_active":"2024-03-18T16:07:20.985685+0000","last_peered":"2024-03-18T16:07:20.985685+0000","last_clean":"2024-03-18T16:07:20.985685+0000","last_became_active":"2024-03-18T16:06:43.702141+0000","last_became_peered":"2024-03-18T16:06:43.702141+0000","last_unstale":"2024-03-18T16:07:20.985685+0000","last_undegraded":"2024-03-18T16:07:20.985685+0000","last_fullsized":"2024-03-18T16:07:20.985685+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:44:05.366101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.043170+0000","last_change":"2024-03-18T16:06:44.711403+0000","last_active":"2024-03-18T16:06:50.043170+0000","last_peered":"2024-03-18T16:06:50.043170+0000","last_clean":"2024-03-18T16:06:50.043170+0000","last_became_active":"2024-03-18T16:06:44.710654+0000","last_became_peered":"2024-03-18T16:06:44.710654+0000","last_unstale":"2024-03-18T16:06:50.043170+0000","last_undegraded":"2024-03-18T16:06:50.043170+0000","last_fullsized":"2024-03-18T16:06:50.043170+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:21:43.633274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.287436+0000","last_change":"2024-03-18T16:06:49.039192+0000","last_active":"2024-03-18T16:06:50.287436+0000","last_peered":"2024-03-18T16:06:50.287436+0000","last_clean":"2024-03-18T16:06:50.287436+0000","last_became_active":"2024-03-18T16:06:49.038421+0000","last_became_peered":"2024-03-18T16:06:49.038421+0000","last_unstale":"2024-03-18T16:06:50.287436+0000","last_undegraded":"2024-03-18T16:06:50.287436+0000","last_fullsized":"2024-03-18T16:06:50.287436+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:54:56.944209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":40,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:07:19.978151+0000","last_change":"2024-03-18T16:06:43.712523+0000","last_active":"2024-03-18T16:07:19.978151+0000","last_peered":"2024-03-18T16:07:19.978151+0000","last_clean":"2024-03-18T16:07:19.978151+0000","last_became_active":"2024-03-18T16:06:43.712151+0000","last_became_peered":"2024-03-18T16:06:43.712151+0000","last_unstale":"2024-03-18T16:07:19.978151+0000","last_undegraded":"2024-03-18T16:07:19.978151+0000","last_fullsized":"2024-03-18T16:07:19.978151+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:17:19.651293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.285772+0000","last_change":"2024-03-18T16:06:44.713522+0000","last_active":"2024-03-18T16:06:50.285772+0000","last_peered":"2024-03-18T16:06:50.285772+0000","last_clean":"2024-03-18T16:06:50.285772+0000","last_became_active":"2024-03-18T16:06:44.713152+0000","last_became_peered":"2024-03-18T16:06:44.713152+0000","last_unstale":"2024-03-18T16:06:50.285772+0000","last_undegraded":"2024-03-18T16:06:50.285772+0000","last_fullsized":"2024-03-18T16:06:50.285772+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:56:01.213384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.285697+0000","last_change":"2024-03-18T16:06:49.028943+0000","last_active":"2024-03-18T16:06:50.285697+0000","last_peered":"2024-03-18T16:06:50.285697+0000","last_clean":"2024-03-18T16:06:50.285697+0000","last_became_active":"2024-03-18T16:06:49.028453+0000","last_became_peered":"2024-03-18T16:06:49.028453+0000","last_unstale":"2024-03-18T16:06:50.285697+0000","last_undegraded":"2024-03-18T16:06:50.285697+0000","last_fullsized":"2024-03-18T16:06:50.285697+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:16:30.318649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:07:19.977899+0000","last_change":"2024-03-18T16:06:43.706840+0000","last_active":"2024-03-18T16:07:19.977899+0000","last_peered":"2024-03-18T16:07:19.977899+0000","last_clean":"2024-03-18T16:07:19.977899+0000","last_became_active":"2024-03-18T16:06:43.706486+0000","last_became_peered":"2024-03-18T16:06:43.706486+0000","last_unstale":"2024-03-18T16:07:19.977899+0000","last_undegraded":"2024-03-18T16:07:19.977899+0000","last_fullsized":"2024-03-18T16:07:19.977899+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:16:44.542471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.151070+0000","last_change":"2024-03-18T16:06:44.716284+0000","last_active":"2024-03-18T16:06:50.151070+0000","last_peered":"2024-03-18T16:06:50.151070+0000","last_clean":"2024-03-18T16:06:50.151070+0000","last_became_active":"2024-03-18T16:06:44.715911+0000","last_became_peered":"2024-03-18T16:06:44.715911+0000","last_unstale":"2024-03-18T16:06:50.151070+0000","last_undegraded":"2024-03-18T16:06:50.151070+0000","last_fullsized":"2024-03-18T16:06:50.151070+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:48:34.839514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.286032+0000","last_change":"2024-03-18T16:06:49.059061+0000","last_active":"2024-03-18T16:06:50.286032+0000","last_peered":"2024-03-18T16:06:50.286032+0000","last_clean":"2024-03-18T16:06:50.286032+0000","last_became_active":"2024-03-18T16:06:49.058619+0000","last_became_peered":"2024-03-18T16:06:49.058619+0000","last_unstale":"2024-03-18T16:06:50.286032+0000","last_undegraded":"2024-03-18T16:06:50.286032+0000","last_fullsized":"2024-03-18T16:06:50.286032+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:44:52.970785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.019296+0000","last_change":"2024-03-18T16:06:43.715766+0000","last_active":"2024-03-18T16:06:50.019296+0000","last_peered":"2024-03-18T16:06:50.019296+0000","last_clean":"2024-03-18T16:06:50.019296+0000","last_became_active":"2024-03-18T16:06:43.715380+0000","last_became_peered":"2024-03-18T16:06:43.715380+0000","last_unstale":"2024-03-18T16:06:50.019296+0000","last_undegraded":"2024-03-18T16:06:50.019296+0000","last_fullsized":"2024-03-18T16:06:50.019296+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:13:44.195934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.014473+0000","last_change":"2024-03-18T16:06:44.728150+0000","last_active":"2024-03-18T16:06:50.014473+0000","last_peered":"2024-03-18T16:06:50.014473+0000","last_clean":"2024-03-18T16:06:50.014473+0000","last_became_active":"2024-03-18T16:06:44.727769+0000","last_became_peered":"2024-03-18T16:06:44.727769+0000","last_unstale":"2024-03-18T16:06:50.014473+0000","last_undegraded":"2024-03-18T16:06:50.014473+0000","last_fullsized":"2024-03-18T16:06:50.014473+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:28:16.023225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.288303+0000","last_change":"2024-03-18T16:06:49.033671+0000","last_active":"2024-03-18T16:06:50.288303+0000","last_peered":"2024-03-18T16:06:50.288303+0000","last_clean":"2024-03-18T16:06:50.288303+0000","last_became_active":"2024-03-18T16:06:49.033046+0000","last_became_peered":"2024-03-18T16:06:49.033046+0000","last_unstale":"2024-03-18T16:06:50.288303+0000","last_undegraded":"2024-03-18T16:06:50.288303+0000","last_fullsized":"2024-03-18T16:06:50.288303+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:26:37.020521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'5","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:07:21.010151+0000","last_change":"2024-03-18T16:06:43.712636+0000","last_active":"2024-03-18T16:07:21.010151+0000","last_peered":"2024-03-18T16:07:21.010151+0000","last_clean":"2024-03-18T16:07:21.010151+0000","last_became_active":"2024-03-18T16:06:43.712300+0000","last_became_peered":"2024-03-18T16:06:43.712300+0000","last_unstale":"2024-03-18T16:07:21.010151+0000","last_undegraded":"2024-03-18T16:07:21.010151+0000","last_fullsized":"2024-03-18T16:07:21.010151+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:48:20.386640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.014268+0000","last_change":"2024-03-18T16:06:44.731583+0000","last_active":"2024-03-18T16:06:50.014268+0000","last_peered":"2024-03-18T16:06:50.014268+0000","last_clean":"2024-03-18T16:06:50.014268+0000","last_became_active":"2024-03-18T16:06:44.731183+0000","last_became_peered":"2024-03-18T16:06:44.731183+0000","last_unstale":"2024-03-18T16:06:50.014268+0000","last_undegraded":"2024-03-18T16:06:50.014268+0000","last_fullsized":"2024-03-18T16:06:50.014268+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:23:05.098101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.288606+0000","last_change":"2024-03-18T16:06:49.060130+0000","last_active":"2024-03-18T16:06:50.288606+0000","last_peered":"2024-03-18T16:06:50.288606+0000","last_clean":"2024-03-18T16:06:50.288606+0000","last_became_active":"2024-03-18T16:06:49.059109+0000","last_became_peered":"2024-03-18T16:06:49.059109+0000","last_unstale":"2024-03-18T16:06:50.288606+0000","last_undegraded":"2024-03-18T16:06:50.288606+0000","last_fullsized":"2024-03-18T16:06:50.288606+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:41:40.008047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.285940+0000","last_change":"2024-03-18T16:06:44.705793+0000","last_active":"2024-03-18T16:06:50.285940+0000","last_peered":"2024-03-18T16:06:50.285940+0000","last_clean":"2024-03-18T16:06:50.285940+0000","last_became_active":"2024-03-18T16:06:44.705426+0000","last_became_peered":"2024-03-18T16:06:44.705426+0000","last_unstale":"2024-03-18T16:06:50.285940+0000","last_undegraded":"2024-03-18T16:06:50.285940+0000","last_fullsized":"2024-03-18T16:06:50.285940+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:14:31.198829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'5","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:07:20.985121+0000","last_change":"2024-03-18T16:06:43.721057+0000","last_active":"2024-03-18T16:07:20.985121+0000","last_peered":"2024-03-18T16:07:20.985121+0000","last_clean":"2024-03-18T16:07:20.985121+0000","last_became_active":"2024-03-18T16:06:43.720661+0000","last_became_peered":"2024-03-18T16:06:43.720661+0000","last_unstale":"2024-03-18T16:07:20.985121+0000","last_undegraded":"2024-03-18T16:07:20.985121+0000","last_fullsized":"2024-03-18T16:07:20.985121+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:08:18.050511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:52.454967+0000","last_change":"2024-03-18T16:06:49.021836+0000","last_active":"2024-03-18T16:06:52.454967+0000","last_peered":"2024-03-18T16:06:52.454967+0000","last_clean":"2024-03-18T16:06:52.454967+0000","last_became_active":"2024-03-18T16:06:49.021474+0000","last_became_peered":"2024-03-18T16:06:49.021474+0000","last_unstale":"2024-03-18T16:06:52.454967+0000","last_undegraded":"2024-03-18T16:06:52.454967+0000","last_fullsized":"2024-03-18T16:06:52.454967+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:46:59.100154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.287039+0000","last_change":"2024-03-18T16:06:44.725555+0000","last_active":"2024-03-18T16:06:50.287039+0000","last_peered":"2024-03-18T16:06:50.287039+0000","last_clean":"2024-03-18T16:06:50.287039+0000","last_became_active":"2024-03-18T16:06:44.719880+0000","last_became_peered":"2024-03-18T16:06:44.719880+0000","last_unstale":"2024-03-18T16:06:50.287039+0000","last_undegraded":"2024-03-18T16:06:50.287039+0000","last_fullsized":"2024-03-18T16:06:50.287039+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:45:17.830876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:07:20.009759+0000","last_change":"2024-03-18T16:06:43.700814+0000","last_active":"2024-03-18T16:07:20.009759+0000","last_peered":"2024-03-18T16:07:20.009759+0000","last_clean":"2024-03-18T16:07:20.009759+0000","last_became_active":"2024-03-18T16:06:43.698902+0000","last_became_peered":"2024-03-18T16:06:43.698902+0000","last_unstale":"2024-03-18T16:07:20.009759+0000","last_undegraded":"2024-03-18T16:07:20.009759+0000","last_fullsized":"2024-03-18T16:07:20.009759+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:20:20.100033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":344,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.043041+0000","last_change":"2024-03-18T16:05:22.379397+0000","last_active":"2024-03-18T16:06:50.043041+0000","last_peered":"2024-03-18T16:06:50.043041+0000","last_clean":"2024-03-18T16:06:50.043041+0000","last_became_active":"2024-03-18T16:05:22.378830+0000","last_became_peered":"2024-03-18T16:05:22.378830+0000","last_unstale":"2024-03-18T16:06:50.043041+0000","last_undegraded":"2024-03-18T16:06:50.043041+0000","last_fullsized":"2024-03-18T16:06:50.043041+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:01:30.048984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:01:30.048984+0000","last_clean_scrub_stamp":"2024-03-18T16:01:30.048984+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:35:45.796495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.286309+0000","last_change":"2024-03-18T16:06:49.058931+0000","last_active":"2024-03-18T16:06:50.286309+0000","last_peered":"2024-03-18T16:06:50.286309+0000","last_clean":"2024-03-18T16:06:50.286309+0000","last_became_active":"2024-03-18T16:06:49.058497+0000","last_became_peered":"2024-03-18T16:06:49.058497+0000","last_unstale":"2024-03-18T16:06:50.286309+0000","last_undegraded":"2024-03-18T16:06:50.286309+0000","last_fullsized":"2024-03-18T16:06:50.286309+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:41:51.874582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.285700+0000","last_change":"2024-03-18T16:06:44.710332+0000","last_active":"2024-03-18T16:06:50.285700+0000","last_peered":"2024-03-18T16:06:50.285700+0000","last_clean":"2024-03-18T16:06:50.285700+0000","last_became_active":"2024-03-18T16:06:44.709952+0000","last_became_peered":"2024-03-18T16:06:44.709952+0000","last_unstale":"2024-03-18T16:06:50.285700+0000","last_undegraded":"2024-03-18T16:06:50.285700+0000","last_fullsized":"2024-03-18T16:06:50.285700+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:24:26.653511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:07:18.224446+0000","last_change":"2024-03-18T16:06:43.708265+0000","last_active":"2024-03-18T16:07:18.224446+0000","last_peered":"2024-03-18T16:07:18.224446+0000","last_clean":"2024-03-18T16:07:18.224446+0000","last_became_active":"2024-03-18T16:06:43.707466+0000","last_became_peered":"2024-03-18T16:06:43.707466+0000","last_unstale":"2024-03-18T16:07:18.224446+0000","last_undegraded":"2024-03-18T16:07:18.224446+0000","last_fullsized":"2024-03-18T16:07:18.224446+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:45:40.065396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.151226+0000","last_change":"2024-03-18T16:06:49.022401+0000","last_active":"2024-03-18T16:06:50.151226+0000","last_peered":"2024-03-18T16:06:50.151226+0000","last_clean":"2024-03-18T16:06:50.151226+0000","last_became_active":"2024-03-18T16:06:49.021966+0000","last_became_peered":"2024-03-18T16:06:49.021966+0000","last_unstale":"2024-03-18T16:06:50.151226+0000","last_undegraded":"2024-03-18T16:06:50.151226+0000","last_fullsized":"2024-03-18T16:06:50.151226+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:04:12.434969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:07:19.978716+0000","last_change":"2024-03-18T16:06:43.713249+0000","last_active":"2024-03-18T16:07:19.978716+0000","last_peered":"2024-03-18T16:07:19.978716+0000","last_clean":"2024-03-18T16:07:19.978716+0000","last_became_active":"2024-03-18T16:06:43.712897+0000","last_became_peered":"2024-03-18T16:06:43.712897+0000","last_unstale":"2024-03-18T16:07:19.978716+0000","last_undegraded":"2024-03-18T16:07:19.978716+0000","last_fullsized":"2024-03-18T16:07:19.978716+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:20:04.642568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.288425+0000","last_change":"2024-03-18T16:06:44.710034+0000","last_active":"2024-03-18T16:06:50.288425+0000","last_peered":"2024-03-18T16:06:50.288425+0000","last_clean":"2024-03-18T16:06:50.288425+0000","last_became_active":"2024-03-18T16:06:44.708457+0000","last_became_peered":"2024-03-18T16:06:44.708457+0000","last_unstale":"2024-03-18T16:06:50.288425+0000","last_undegraded":"2024-03-18T16:06:50.288425+0000","last_fullsized":"2024-03-18T16:06:50.288425+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:40:00.504382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.286557+0000","last_change":"2024-03-18T16:06:49.059345+0000","last_active":"2024-03-18T16:06:50.286557+0000","last_peered":"2024-03-18T16:06:50.286557+0000","last_clean":"2024-03-18T16:06:50.286557+0000","last_became_active":"2024-03-18T16:06:49.058675+0000","last_became_peered":"2024-03-18T16:06:49.058675+0000","last_unstale":"2024-03-18T16:06:50.286557+0000","last_undegraded":"2024-03-18T16:06:50.286557+0000","last_fullsized":"2024-03-18T16:06:50.286557+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:03:33.730074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'9","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:07:20.833925+0000","last_change":"2024-03-18T16:06:43.724589+0000","last_active":"2024-03-18T16:07:20.833925+0000","last_peered":"2024-03-18T16:07:20.833925+0000","last_clean":"2024-03-18T16:07:20.833925+0000","last_became_active":"2024-03-18T16:06:43.724125+0000","last_became_peered":"2024-03-18T16:06:43.724125+0000","last_unstale":"2024-03-18T16:07:20.833925+0000","last_undegraded":"2024-03-18T16:07:20.833925+0000","last_fullsized":"2024-03-18T16:07:20.833925+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:08:34.921118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.287632+0000","last_change":"2024-03-18T16:06:44.727162+0000","last_active":"2024-03-18T16:06:50.287632+0000","last_peered":"2024-03-18T16:06:50.287632+0000","last_clean":"2024-03-18T16:06:50.287632+0000","last_became_active":"2024-03-18T16:06:44.726552+0000","last_became_peered":"2024-03-18T16:06:44.726552+0000","last_unstale":"2024-03-18T16:06:50.287632+0000","last_undegraded":"2024-03-18T16:06:50.287632+0000","last_fullsized":"2024-03-18T16:06:50.287632+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:36:57.974999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.014394+0000","last_change":"2024-03-18T16:06:49.008434+0000","last_active":"2024-03-18T16:06:50.014394+0000","last_peered":"2024-03-18T16:06:50.014394+0000","last_clean":"2024-03-18T16:06:50.014394+0000","last_became_active":"2024-03-18T16:06:49.007944+0000","last_became_peered":"2024-03-18T16:06:49.007944+0000","last_unstale":"2024-03-18T16:06:50.014394+0000","last_undegraded":"2024-03-18T16:06:50.014394+0000","last_fullsized":"2024-03-18T16:06:50.014394+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:01:24.450371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:07:20.010508+0000","last_change":"2024-03-18T16:06:43.711797+0000","last_active":"2024-03-18T16:07:20.010508+0000","last_peered":"2024-03-18T16:07:20.010508+0000","last_clean":"2024-03-18T16:07:20.010508+0000","last_became_active":"2024-03-18T16:06:43.711493+0000","last_became_peered":"2024-03-18T16:06:43.711493+0000","last_unstale":"2024-03-18T16:07:20.010508+0000","last_undegraded":"2024-03-18T16:07:20.010508+0000","last_fullsized":"2024-03-18T16:07:20.010508+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:50:29.258166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.288039+0000","last_change":"2024-03-18T16:06:44.735910+0000","last_active":"2024-03-18T16:06:50.288039+0000","last_peered":"2024-03-18T16:06:50.288039+0000","last_clean":"2024-03-18T16:06:50.288039+0000","last_became_active":"2024-03-18T16:06:44.735504+0000","last_became_peered":"2024-03-18T16:06:44.735504+0000","last_unstale":"2024-03-18T16:06:50.288039+0000","last_undegraded":"2024-03-18T16:06:50.288039+0000","last_fullsized":"2024-03-18T16:06:50.288039+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:02:18.538217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.265920+0000","last_change":"2024-03-18T16:06:49.059130+0000","last_active":"2024-03-18T16:06:50.265920+0000","last_peered":"2024-03-18T16:06:50.265920+0000","last_clean":"2024-03-18T16:06:50.265920+0000","last_became_active":"2024-03-18T16:06:49.058701+0000","last_became_peered":"2024-03-18T16:06:49.058701+0000","last_unstale":"2024-03-18T16:06:50.265920+0000","last_undegraded":"2024-03-18T16:06:50.265920+0000","last_fullsized":"2024-03-18T16:06:50.265920+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:54:52.842786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:07:17.209659+0000","last_change":"2024-03-18T16:06:43.714414+0000","last_active":"2024-03-18T16:07:17.209659+0000","last_peered":"2024-03-18T16:07:17.209659+0000","last_clean":"2024-03-18T16:07:17.209659+0000","last_became_active":"2024-03-18T16:06:43.714029+0000","last_became_peered":"2024-03-18T16:06:43.714029+0000","last_unstale":"2024-03-18T16:07:17.209659+0000","last_undegraded":"2024-03-18T16:07:17.209659+0000","last_fullsized":"2024-03-18T16:07:17.209659+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:16:39.060378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.285335+0000","last_change":"2024-03-18T16:06:44.707491+0000","last_active":"2024-03-18T16:06:50.285335+0000","last_peered":"2024-03-18T16:06:50.285335+0000","last_clean":"2024-03-18T16:06:50.285335+0000","last_became_active":"2024-03-18T16:06:44.707157+0000","last_became_peered":"2024-03-18T16:06:44.707157+0000","last_unstale":"2024-03-18T16:06:50.285335+0000","last_undegraded":"2024-03-18T16:06:50.285335+0000","last_fullsized":"2024-03-18T16:06:50.285335+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:19:08.132911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.020397+0000","last_change":"2024-03-18T16:06:49.035041+0000","last_active":"2024-03-18T16:06:50.020397+0000","last_peered":"2024-03-18T16:06:50.020397+0000","last_clean":"2024-03-18T16:06:50.020397+0000","last_became_active":"2024-03-18T16:06:49.033887+0000","last_became_peered":"2024-03-18T16:06:49.033887+0000","last_unstale":"2024-03-18T16:06:50.020397+0000","last_undegraded":"2024-03-18T16:06:50.020397+0000","last_fullsized":"2024-03-18T16:06:50.020397+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:10:36.269741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:07:20.009415+0000","last_change":"2024-03-18T16:06:43.717755+0000","last_active":"2024-03-18T16:07:20.009415+0000","last_peered":"2024-03-18T16:07:20.009415+0000","last_clean":"2024-03-18T16:07:20.009415+0000","last_became_active":"2024-03-18T16:06:43.717278+0000","last_became_peered":"2024-03-18T16:06:43.717278+0000","last_unstale":"2024-03-18T16:07:20.009415+0000","last_undegraded":"2024-03-18T16:07:20.009415+0000","last_fullsized":"2024-03-18T16:07:20.009415+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:48:09.390659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.150854+0000","last_change":"2024-03-18T16:06:44.730858+0000","last_active":"2024-03-18T16:06:50.150854+0000","last_peered":"2024-03-18T16:06:50.150854+0000","last_clean":"2024-03-18T16:06:50.150854+0000","last_became_active":"2024-03-18T16:06:44.730500+0000","last_became_peered":"2024-03-18T16:06:44.730500+0000","last_unstale":"2024-03-18T16:06:50.150854+0000","last_undegraded":"2024-03-18T16:06:50.150854+0000","last_fullsized":"2024-03-18T16:06:50.150854+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:22:03.590834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:52.881119+0000","last_change":"2024-03-18T16:06:49.014717+0000","last_active":"2024-03-18T16:06:52.881119+0000","last_peered":"2024-03-18T16:06:52.881119+0000","last_clean":"2024-03-18T16:06:52.881119+0000","last_became_active":"2024-03-18T16:06:49.014313+0000","last_became_peered":"2024-03-18T16:06:49.014313+0000","last_unstale":"2024-03-18T16:06:52.881119+0000","last_undegraded":"2024-03-18T16:06:52.881119+0000","last_fullsized":"2024-03-18T16:06:52.881119+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:06:12.168888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.014435+0000","last_change":"2024-03-18T16:06:43.719496+0000","last_active":"2024-03-18T16:06:50.014435+0000","last_peered":"2024-03-18T16:06:50.014435+0000","last_clean":"2024-03-18T16:06:50.014435+0000","last_became_active":"2024-03-18T16:06:43.719139+0000","last_became_peered":"2024-03-18T16:06:43.719139+0000","last_unstale":"2024-03-18T16:06:50.014435+0000","last_undegraded":"2024-03-18T16:06:50.014435+0000","last_fullsized":"2024-03-18T16:06:50.014435+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:10:01.250923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.287622+0000","last_change":"2024-03-18T16:06:44.737792+0000","last_active":"2024-03-18T16:06:50.287622+0000","last_peered":"2024-03-18T16:06:50.287622+0000","last_clean":"2024-03-18T16:06:50.287622+0000","last_became_active":"2024-03-18T16:06:44.736311+0000","last_became_peered":"2024-03-18T16:06:44.736311+0000","last_unstale":"2024-03-18T16:06:50.287622+0000","last_undegraded":"2024-03-18T16:06:50.287622+0000","last_fullsized":"2024-03-18T16:06:50.287622+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:43:56.110205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.265955+0000","last_change":"2024-03-18T16:06:49.058893+0000","last_active":"2024-03-18T16:06:50.265955+0000","last_peered":"2024-03-18T16:06:50.265955+0000","last_clean":"2024-03-18T16:06:50.265955+0000","last_became_active":"2024-03-18T16:06:49.058477+0000","last_became_peered":"2024-03-18T16:06:49.058477+0000","last_unstale":"2024-03-18T16:06:50.265955+0000","last_undegraded":"2024-03-18T16:06:50.265955+0000","last_fullsized":"2024-03-18T16:06:50.265955+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:50:51.890348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:07:17.205076+0000","last_change":"2024-03-18T16:06:43.703073+0000","last_active":"2024-03-18T16:07:17.205076+0000","last_peered":"2024-03-18T16:07:17.205076+0000","last_clean":"2024-03-18T16:07:17.205076+0000","last_became_active":"2024-03-18T16:06:43.702708+0000","last_became_peered":"2024-03-18T16:06:43.702708+0000","last_unstale":"2024-03-18T16:07:17.205076+0000","last_undegraded":"2024-03-18T16:07:17.205076+0000","last_fullsized":"2024-03-18T16:07:17.205076+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:38:41.454458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.043293+0000","last_change":"2024-03-18T16:06:44.713622+0000","last_active":"2024-03-18T16:06:50.043293+0000","last_peered":"2024-03-18T16:06:50.043293+0000","last_clean":"2024-03-18T16:06:50.043293+0000","last_became_active":"2024-03-18T16:06:44.713160+0000","last_became_peered":"2024-03-18T16:06:44.713160+0000","last_unstale":"2024-03-18T16:06:50.043293+0000","last_undegraded":"2024-03-18T16:06:50.043293+0000","last_fullsized":"2024-03-18T16:06:50.043293+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:49:21.886543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.285320+0000","last_change":"2024-03-18T16:06:49.032597+0000","last_active":"2024-03-18T16:06:50.285320+0000","last_peered":"2024-03-18T16:06:50.285320+0000","last_clean":"2024-03-18T16:06:50.285320+0000","last_became_active":"2024-03-18T16:06:49.032250+0000","last_became_peered":"2024-03-18T16:06:49.032250+0000","last_unstale":"2024-03-18T16:06:50.285320+0000","last_undegraded":"2024-03-18T16:06:50.285320+0000","last_fullsized":"2024-03-18T16:06:50.285320+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:16:45.761978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:07:18.230261+0000","last_change":"2024-03-18T16:06:43.717161+0000","last_active":"2024-03-18T16:07:18.230261+0000","last_peered":"2024-03-18T16:07:18.230261+0000","last_clean":"2024-03-18T16:07:18.230261+0000","last_became_active":"2024-03-18T16:06:43.716657+0000","last_became_peered":"2024-03-18T16:06:43.716657+0000","last_unstale":"2024-03-18T16:07:18.230261+0000","last_undegraded":"2024-03-18T16:07:18.230261+0000","last_fullsized":"2024-03-18T16:07:18.230261+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:23:48.194779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.043700+0000","last_change":"2024-03-18T16:06:44.709777+0000","last_active":"2024-03-18T16:06:50.043700+0000","last_peered":"2024-03-18T16:06:50.043700+0000","last_clean":"2024-03-18T16:06:50.043700+0000","last_became_active":"2024-03-18T16:06:44.709409+0000","last_became_peered":"2024-03-18T16:06:44.709409+0000","last_unstale":"2024-03-18T16:06:50.043700+0000","last_undegraded":"2024-03-18T16:06:50.043700+0000","last_fullsized":"2024-03-18T16:06:50.043700+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:37:12.288262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.286126+0000","last_change":"2024-03-18T16:06:49.040872+0000","last_active":"2024-03-18T16:06:50.286126+0000","last_peered":"2024-03-18T16:06:50.286126+0000","last_clean":"2024-03-18T16:06:50.286126+0000","last_became_active":"2024-03-18T16:06:49.039807+0000","last_became_peered":"2024-03-18T16:06:49.039807+0000","last_unstale":"2024-03-18T16:06:50.286126+0000","last_undegraded":"2024-03-18T16:06:50.286126+0000","last_fullsized":"2024-03-18T16:06:50.286126+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:09:27.045450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:07:19.977630+0000","last_change":"2024-03-18T16:06:43.723675+0000","last_active":"2024-03-18T16:07:19.977630+0000","last_peered":"2024-03-18T16:07:19.977630+0000","last_clean":"2024-03-18T16:07:19.977630+0000","last_became_active":"2024-03-18T16:06:43.723334+0000","last_became_peered":"2024-03-18T16:06:43.723334+0000","last_unstale":"2024-03-18T16:07:19.977630+0000","last_undegraded":"2024-03-18T16:07:19.977630+0000","last_fullsized":"2024-03-18T16:07:19.977630+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:01:36.553362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.043434+0000","last_change":"2024-03-18T16:06:44.723045+0000","last_active":"2024-03-18T16:06:50.043434+0000","last_peered":"2024-03-18T16:06:50.043434+0000","last_clean":"2024-03-18T16:06:50.043434+0000","last_became_active":"2024-03-18T16:06:44.722688+0000","last_became_peered":"2024-03-18T16:06:44.722688+0000","last_unstale":"2024-03-18T16:06:50.043434+0000","last_undegraded":"2024-03-18T16:06:50.043434+0000","last_fullsized":"2024-03-18T16:06:50.043434+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:46:21.720007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.285387+0000","last_change":"2024-03-18T16:06:49.011981+0000","last_active":"2024-03-18T16:06:50.285387+0000","last_peered":"2024-03-18T16:06:50.285387+0000","last_clean":"2024-03-18T16:06:50.285387+0000","last_became_active":"2024-03-18T16:06:49.011621+0000","last_became_peered":"2024-03-18T16:06:49.011621+0000","last_unstale":"2024-03-18T16:06:50.285387+0000","last_undegraded":"2024-03-18T16:06:50.285387+0000","last_fullsized":"2024-03-18T16:06:50.285387+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:11:42.382721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:07:20.010635+0000","last_change":"2024-03-18T16:06:43.725404+0000","last_active":"2024-03-18T16:07:20.010635+0000","last_peered":"2024-03-18T16:07:20.010635+0000","last_clean":"2024-03-18T16:07:20.010635+0000","last_became_active":"2024-03-18T16:06:43.724117+0000","last_became_peered":"2024-03-18T16:06:43.724117+0000","last_unstale":"2024-03-18T16:07:20.010635+0000","last_undegraded":"2024-03-18T16:07:20.010635+0000","last_fullsized":"2024-03-18T16:07:20.010635+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:58:43.243070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.043631+0000","last_change":"2024-03-18T16:06:44.709979+0000","last_active":"2024-03-18T16:06:50.043631+0000","last_peered":"2024-03-18T16:06:50.043631+0000","last_clean":"2024-03-18T16:06:50.043631+0000","last_became_active":"2024-03-18T16:06:44.709591+0000","last_became_peered":"2024-03-18T16:06:44.709591+0000","last_unstale":"2024-03-18T16:06:50.043631+0000","last_undegraded":"2024-03-18T16:06:50.043631+0000","last_fullsized":"2024-03-18T16:06:50.043631+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:56:37.537248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.150979+0000","last_change":"2024-03-18T16:06:44.706641+0000","last_active":"2024-03-18T16:06:50.150979+0000","last_peered":"2024-03-18T16:06:50.150979+0000","last_clean":"2024-03-18T16:06:50.150979+0000","last_became_active":"2024-03-18T16:06:44.706267+0000","last_became_peered":"2024-03-18T16:06:44.706267+0000","last_unstale":"2024-03-18T16:06:50.150979+0000","last_undegraded":"2024-03-18T16:06:50.150979+0000","last_fullsized":"2024-03-18T16:06:50.150979+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:10:30.072914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.014840+0000","last_change":"2024-03-18T16:06:44.714836+0000","last_active":"2024-03-18T16:06:50.014840+0000","last_peered":"2024-03-18T16:06:50.014840+0000","last_clean":"2024-03-18T16:06:50.014840+0000","last_became_active":"2024-03-18T16:06:44.714454+0000","last_became_peered":"2024-03-18T16:06:44.714454+0000","last_unstale":"2024-03-18T16:06:50.014840+0000","last_undegraded":"2024-03-18T16:06:50.014840+0000","last_fullsized":"2024-03-18T16:06:50.014840+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:05:24.532238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.014510+0000","last_change":"2024-03-18T16:06:44.733183+0000","last_active":"2024-03-18T16:06:50.014510+0000","last_peered":"2024-03-18T16:06:50.014510+0000","last_clean":"2024-03-18T16:06:50.014510+0000","last_became_active":"2024-03-18T16:06:44.732909+0000","last_became_peered":"2024-03-18T16:06:44.732909+0000","last_unstale":"2024-03-18T16:06:50.014510+0000","last_undegraded":"2024-03-18T16:06:50.014510+0000","last_fullsized":"2024-03-18T16:06:50.014510+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:28:07.636692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.286380+0000","last_change":"2024-03-18T16:06:44.731050+0000","last_active":"2024-03-18T16:06:50.286380+0000","last_peered":"2024-03-18T16:06:50.286380+0000","last_clean":"2024-03-18T16:06:50.286380+0000","last_became_active":"2024-03-18T16:06:44.730634+0000","last_became_peered":"2024-03-18T16:06:44.730634+0000","last_unstale":"2024-03-18T16:06:50.286380+0000","last_undegraded":"2024-03-18T16:06:50.286380+0000","last_fullsized":"2024-03-18T16:06:50.286380+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:07:18.619130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.288532+0000","last_change":"2024-03-18T16:06:44.735190+0000","last_active":"2024-03-18T16:06:50.288532+0000","last_peered":"2024-03-18T16:06:50.288532+0000","last_clean":"2024-03-18T16:06:50.288532+0000","last_became_active":"2024-03-18T16:06:44.734312+0000","last_became_peered":"2024-03-18T16:06:44.734312+0000","last_unstale":"2024-03-18T16:06:50.288532+0000","last_undegraded":"2024-03-18T16:06:50.288532+0000","last_fullsized":"2024-03-18T16:06:50.288532+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:59:50.209515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.151329+0000","last_change":"2024-03-18T16:06:44.705385+0000","last_active":"2024-03-18T16:06:50.151329+0000","last_peered":"2024-03-18T16:06:50.151329+0000","last_clean":"2024-03-18T16:06:50.151329+0000","last_became_active":"2024-03-18T16:06:44.705011+0000","last_became_peered":"2024-03-18T16:06:44.705011+0000","last_unstale":"2024-03-18T16:06:50.151329+0000","last_undegraded":"2024-03-18T16:06:50.151329+0000","last_fullsized":"2024-03-18T16:06:50.151329+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:50:02.152851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.015132+0000","last_change":"2024-03-18T16:06:44.709621+0000","last_active":"2024-03-18T16:06:50.015132+0000","last_peered":"2024-03-18T16:06:50.015132+0000","last_clean":"2024-03-18T16:06:50.015132+0000","last_became_active":"2024-03-18T16:06:44.709275+0000","last_became_peered":"2024-03-18T16:06:44.709275+0000","last_unstale":"2024-03-18T16:06:50.015132+0000","last_undegraded":"2024-03-18T16:06:50.015132+0000","last_fullsized":"2024-03-18T16:06:50.015132+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:41:17.346661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.287832+0000","last_change":"2024-03-18T16:06:44.719827+0000","last_active":"2024-03-18T16:06:50.287832+0000","last_peered":"2024-03-18T16:06:50.287832+0000","last_clean":"2024-03-18T16:06:50.287832+0000","last_became_active":"2024-03-18T16:06:44.719432+0000","last_became_peered":"2024-03-18T16:06:44.719432+0000","last_unstale":"2024-03-18T16:06:50.287832+0000","last_undegraded":"2024-03-18T16:06:50.287832+0000","last_fullsized":"2024-03-18T16:06:50.287832+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:13:42.344211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.285992+0000","last_change":"2024-03-18T16:06:44.714526+0000","last_active":"2024-03-18T16:06:50.285992+0000","last_peered":"2024-03-18T16:06:50.285992+0000","last_clean":"2024-03-18T16:06:50.285992+0000","last_became_active":"2024-03-18T16:06:44.714184+0000","last_became_peered":"2024-03-18T16:06:44.714184+0000","last_unstale":"2024-03-18T16:06:50.285992+0000","last_undegraded":"2024-03-18T16:06:50.285992+0000","last_fullsized":"2024-03-18T16:06:50.285992+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:31:07.917143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.287038+0000","last_change":"2024-03-18T16:06:44.730905+0000","last_active":"2024-03-18T16:06:50.287038+0000","last_peered":"2024-03-18T16:06:50.287038+0000","last_clean":"2024-03-18T16:06:50.287038+0000","last_became_active":"2024-03-18T16:06:44.729002+0000","last_became_peered":"2024-03-18T16:06:44.729002+0000","last_unstale":"2024-03-18T16:06:50.287038+0000","last_undegraded":"2024-03-18T16:06:50.287038+0000","last_fullsized":"2024-03-18T16:06:50.287038+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:49:18.867571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.285382+0000","last_change":"2024-03-18T16:06:44.727608+0000","last_active":"2024-03-18T16:06:50.285382+0000","last_peered":"2024-03-18T16:06:50.285382+0000","last_clean":"2024-03-18T16:06:50.285382+0000","last_became_active":"2024-03-18T16:06:44.727096+0000","last_became_peered":"2024-03-18T16:06:44.727096+0000","last_unstale":"2024-03-18T16:06:50.285382+0000","last_undegraded":"2024-03-18T16:06:50.285382+0000","last_fullsized":"2024-03-18T16:06:50.285382+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:08:20.767443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.286521+0000","last_change":"2024-03-18T16:06:44.731823+0000","last_active":"2024-03-18T16:06:50.286521+0000","last_peered":"2024-03-18T16:06:50.286521+0000","last_clean":"2024-03-18T16:06:50.286521+0000","last_became_active":"2024-03-18T16:06:44.731429+0000","last_became_peered":"2024-03-18T16:06:44.731429+0000","last_unstale":"2024-03-18T16:06:50.286521+0000","last_undegraded":"2024-03-18T16:06:50.286521+0000","last_fullsized":"2024-03-18T16:06:50.286521+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:36:28.752466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.151152+0000","last_change":"2024-03-18T16:06:44.714354+0000","last_active":"2024-03-18T16:06:50.151152+0000","last_peered":"2024-03-18T16:06:50.151152+0000","last_clean":"2024-03-18T16:06:50.151152+0000","last_became_active":"2024-03-18T16:06:44.713969+0000","last_became_peered":"2024-03-18T16:06:44.713969+0000","last_unstale":"2024-03-18T16:06:50.151152+0000","last_undegraded":"2024-03-18T16:06:50.151152+0000","last_fullsized":"2024-03-18T16:06:50.151152+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:18:49.502178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":10033,"num_objects":92,"num_object_clones":0,"num_object_copies":276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92,"num_whiteouts":0,"num_read":38,"num_read_kb":30,"num_write":204,"num_write_kb":68,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":62,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":50217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":176,"ondisk_log_size":176,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383702,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":995328,"data_stored":478407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645337,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36920,"kb_used_data":1584,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708232,"statfs":{"total":95995035648,"available":95957229568,"internally_reserved":0,"allocated":1622016,"data_stored":1073003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808863,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1028096,"data_stored":486529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070503,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1019904,"data_stored":482811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332139,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28596,"kb_used_data":1580,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716556,"statfs":{"total":95995035648,"available":95965753344,"internally_reserved":0,"allocated":1617920,"data_stored":1079308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495664,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32152,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713000,"statfs":{"total":95995035648,"available":95962112000,"internally_reserved":0,"allocated":999424,"data_stored":477937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574924,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36312,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708840,"statfs":{"total":95995035648,"available":95957852160,"internally_reserved":0,"allocated":999424,"data_stored":481639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738451,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1024000,"data_stored":479393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411400,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36320,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708832,"statfs":{"total":95995035648,"available":95957843968,"internally_reserved":0,"allocated":1007616,"data_stored":481713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182466,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1007616,"data_stored":477623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920829,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":1007616,"data_stored":482011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757304,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36924,"kb_used_data":1588,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708228,"statfs":{"total":95995035648,"available":95957225472,"internally_reserved":0,"allocated":1626112,"data_stored":1074890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":24893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":5192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":5544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T16:07:23.688 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T16:07:23.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:23 smithi196 ceph-mon[36307]: pgmap v337: 129 pgs: 129 active+clean; 582 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s rd, 1.6 KiB/s wr, 5 op/s 2024-03-18T16:07:23.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:23 smithi196 ceph-mon[36307]: from='client.24859 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:07:23.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:23 smithi196 ceph-mon[36307]: from='client.24865 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:07:23.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:23 smithi196 ceph-mon[36307]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-03-18T16:07:23.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:23 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/3889527143' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:07:23.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:23 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/2946819666' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:07:23.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.941+0000 7f5943200640 1 -- 172.21.15.39:0/282518704 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f593c104bf0 msgr2=0x7f593c105070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:23.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.941+0000 7f5943200640 1 --2- 172.21.15.39:0/282518704 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f593c104bf0 0x7f593c105070 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f5924009900 tx=0x7f592402bc30 comp rx=0 tx=0).stop 2024-03-18T16:07:23.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.941+0000 7f5943200640 1 -- 172.21.15.39:0/282518704 shutdown_connections 2024-03-18T16:07:23.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.941+0000 7f5943200640 1 --2- 172.21.15.39:0/282518704 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f593c104bf0 0x7f593c105070 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:23.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.941+0000 7f5943200640 1 --2- 172.21.15.39:0/282518704 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f593c1055b0 0x7f593c10be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:23.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.941+0000 7f5943200640 1 --2- 172.21.15.39:0/282518704 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f593c103a00 0x7f593c103e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:23.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.941+0000 7f5943200640 1 -- 172.21.15.39:0/282518704 >> 172.21.15.39:0/282518704 conn(0x7f593c0fef90 msgr2=0x7f593c1013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:23.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.941+0000 7f5943200640 1 -- 172.21.15.39:0/282518704 shutdown_connections 2024-03-18T16:07:23.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.942+0000 7f5943200640 1 -- 172.21.15.39:0/282518704 wait complete. 2024-03-18T16:07:23.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.943+0000 7f5943200640 1 Processor -- start 2024-03-18T16:07:23.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.943+0000 7f5943200640 1 -- start start 2024-03-18T16:07:23.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.943+0000 7f5943200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f593c103a00 0x7f593c199d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:23.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.943+0000 7f5943200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f593c104bf0 0x7f593c19a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:23.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.943+0000 7f5943200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f593c1055b0 0x7f593c1a1280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:23.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.943+0000 7f5943200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f593c10f6d0 con 0x7f593c103a00 2024-03-18T16:07:23.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.943+0000 7f5943200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f593c10f550 con 0x7f593c104bf0 2024-03-18T16:07:23.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.943+0000 7f5943200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f593c10f850 con 0x7f593c1055b0 2024-03-18T16:07:23.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.943+0000 7f593be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f593c103a00 0x7f593c199d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:23.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.943+0000 7f5941200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f593c1055b0 0x7f593c1a1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:23.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.943+0000 7f5933e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f593c104bf0 0x7f593c19a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:23.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.943+0000 7f5941200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f593c1055b0 0x7f593c1a1280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:60984/0 (socket says 172.21.15.39:60984) 2024-03-18T16:07:23.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.943+0000 7f5933e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f593c104bf0 0x7f593c19a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:56530/0 (socket says 172.21.15.39:56530) 2024-03-18T16:07:23.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.943+0000 7f5941200640 1 -- 172.21.15.39:0/1964043729 learned_addr learned my addr 172.21.15.39:0/1964043729 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:07:23.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.944+0000 7f5933e00640 1 -- 172.21.15.39:0/1964043729 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f593c1055b0 msgr2=0x7f593c1a1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:23.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.944+0000 7f5933e00640 1 --2- 172.21.15.39:0/1964043729 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f593c1055b0 0x7f593c1a1280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:23.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.944+0000 7f5933e00640 1 -- 172.21.15.39:0/1964043729 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f593c103a00 msgr2=0x7f593c199d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:23.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.944+0000 7f5933e00640 1 --2- 172.21.15.39:0/1964043729 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f593c103a00 0x7f593c199d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:23.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.944+0000 7f5933e00640 1 -- 172.21.15.39:0/1964043729 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f593c1a1920 con 0x7f593c104bf0 2024-03-18T16:07:23.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.944+0000 7f5933e00640 1 --2- 172.21.15.39:0/1964043729 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f593c104bf0 0x7f593c19a250 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f5924036040 tx=0x7f592402c6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:23.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.944+0000 7f5939600640 1 -- 172.21.15.39:0/1964043729 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5924043020 con 0x7f593c104bf0 2024-03-18T16:07:23.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.944+0000 7f5939600640 1 -- 172.21.15.39:0/1964043729 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f59240054e0 con 0x7f593c104bf0 2024-03-18T16:07:23.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.945+0000 7f5943200640 1 -- 172.21.15.39:0/1964043729 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f593c1a1bb0 con 0x7f593c104bf0 2024-03-18T16:07:23.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.945+0000 7f5939600640 1 -- 172.21.15.39:0/1964043729 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5924003ca0 con 0x7f593c104bf0 2024-03-18T16:07:23.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.945+0000 7f5943200640 1 -- 172.21.15.39:0/1964043729 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f593c1a2120 con 0x7f593c104bf0 2024-03-18T16:07:23.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.947+0000 7f5943200640 1 -- 172.21.15.39:0/1964043729 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f593c068a90 con 0x7f593c104bf0 2024-03-18T16:07:23.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.948+0000 7f5939600640 1 -- 172.21.15.39:0/1964043729 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f5924046020 con 0x7f593c104bf0 2024-03-18T16:07:23.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.949+0000 7f5939600640 1 --2- 172.21.15.39:0/1964043729 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f59200a2910 0x7f59200a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:23.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.949+0000 7f5939600640 1 -- 172.21.15.39:0/1964043729 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f59240f0de0 con 0x7f593c104bf0 2024-03-18T16:07:23.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.949+0000 7f593be00640 1 --2- 172.21.15.39:0/1964043729 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f59200a2910 0x7f59200a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:23.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.950+0000 7f593be00640 1 --2- 172.21.15.39:0/1964043729 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f59200a2910 0x7f59200a4dd0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f593c19b1e0 tx=0x7f592c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:23.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:23.953+0000 7f5939600640 1 -- 172.21.15.39:0/1964043729 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f592404d020 con 0x7f593c104bf0 2024-03-18T16:07:23.990 INFO:journalctl@ceph.mds.b.smithi194.stdout:Mar 18 16:07:23 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-b[67287]: 2024-03-18T16:07:23.583+0000 7fa042c70700 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2024-03-18T16:07:23.990 INFO:journalctl@ceph.mds.b.smithi194.stdout:Mar 18 16:07:23 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-b[67287]: 2024-03-18T16:07:23.583+0000 7fa042c70700 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2024-03-18T16:07:24.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:24.187+0000 7f5943200640 1 -- 172.21.15.39:0/1964043729 --> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f593c061ce0 con 0x7f59200a2910 2024-03-18T16:07:24.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:24.205+0000 7f5939600640 1 -- 172.21.15.39:0/1964043729 <== mgr.14152 v2:172.21.15.39:6800/2749214286 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340439 (secure 0 0 0) 0x7f593c061ce0 con 0x7f59200a2910 2024-03-18T16:07:24.208 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:07:24.213 INFO:teuthology.orchestra.run.smithi039.stderr:dumped all 2024-03-18T16:07:24.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:24.214+0000 7f5943200640 1 -- 172.21.15.39:0/1964043729 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f59200a2910 msgr2=0x7f59200a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:24.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:24.214+0000 7f5943200640 1 --2- 172.21.15.39:0/1964043729 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f59200a2910 0x7f59200a4dd0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f593c19b1e0 tx=0x7f592c005f90 comp rx=0 tx=0).stop 2024-03-18T16:07:24.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:24.214+0000 7f5943200640 1 -- 172.21.15.39:0/1964043729 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f593c104bf0 msgr2=0x7f593c19a250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:24.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:24.214+0000 7f5943200640 1 --2- 172.21.15.39:0/1964043729 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f593c104bf0 0x7f593c19a250 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f5924036040 tx=0x7f592402c6c0 comp rx=0 tx=0).stop 2024-03-18T16:07:24.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:24.214+0000 7f5943200640 1 -- 172.21.15.39:0/1964043729 shutdown_connections 2024-03-18T16:07:24.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:24.214+0000 7f5943200640 1 --2- 172.21.15.39:0/1964043729 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f593c104bf0 0x7f593c19a250 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:24.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:24.214+0000 7f5943200640 1 --2- 172.21.15.39:0/1964043729 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f59200a2910 0x7f59200a4dd0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:24.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:24.214+0000 7f5943200640 1 --2- 172.21.15.39:0/1964043729 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f593c1055b0 0x7f593c1a1280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:24.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:24.214+0000 7f5943200640 1 --2- 172.21.15.39:0/1964043729 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f593c103a00 0x7f593c199d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:24.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:24.214+0000 7f5943200640 1 -- 172.21.15.39:0/1964043729 >> 172.21.15.39:0/1964043729 conn(0x7f593c0fef90 msgr2=0x7f593c107ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:24.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:24.214+0000 7f5943200640 1 -- 172.21.15.39:0/1964043729 shutdown_connections 2024-03-18T16:07:24.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:24.214+0000 7f5943200640 1 -- 172.21.15.39:0/1964043729 wait complete. 2024-03-18T16:07:24.239 INFO:teuthology.orchestra.run.smithi039.stdout:{"pg_ready":true,"pg_map":{"version":338,"stamp":"2024-03-18T16:07:23.502398+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":600635,"num_objects":96,"num_object_clones":0,"num_object_copies":290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":96,"num_whiteouts":0,"num_read":164,"num_read_kb":139,"num_write":387,"num_write_kb":3008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":62,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":315,"ondisk_log_size":315,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":387740,"kb_used_data":13628,"kb_used_omap":18,"kb_used_meta":373997,"kb_avail":1124554084,"statfs":{"total":1151940427776,"available":1151543382016,"internally_reserved":0,"allocated":13955072,"data_stored":7555264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":382973243},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5627,"num_objects":72,"num_object_clones":0,"num_object_copies":218,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":0,"num_read":38,"num_read_kb":30,"num_write":157,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":48,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.006268"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.151383+0000","last_change":"2024-03-18T16:06:44.736511+0000","last_active":"2024-03-18T16:06:50.151383+0000","last_peered":"2024-03-18T16:06:50.151383+0000","last_clean":"2024-03-18T16:06:50.151383+0000","last_became_active":"2024-03-18T16:06:44.736182+0000","last_became_peered":"2024-03-18T16:06:44.736182+0000","last_unstale":"2024-03-18T16:06:50.151383+0000","last_undegraded":"2024-03-18T16:06:50.151383+0000","last_fullsized":"2024-03-18T16:06:50.151383+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:08:57.060268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.288006+0000","last_change":"2024-03-18T16:06:44.733658+0000","last_active":"2024-03-18T16:06:50.288006+0000","last_peered":"2024-03-18T16:06:50.288006+0000","last_clean":"2024-03-18T16:06:50.288006+0000","last_became_active":"2024-03-18T16:06:44.733279+0000","last_became_peered":"2024-03-18T16:06:44.733279+0000","last_unstale":"2024-03-18T16:06:50.288006+0000","last_undegraded":"2024-03-18T16:06:50.288006+0000","last_fullsized":"2024-03-18T16:06:50.288006+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:59:46.783025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.286651+0000","last_change":"2024-03-18T16:06:44.723949+0000","last_active":"2024-03-18T16:06:50.286651+0000","last_peered":"2024-03-18T16:06:50.286651+0000","last_clean":"2024-03-18T16:06:50.286651+0000","last_became_active":"2024-03-18T16:06:44.723614+0000","last_became_peered":"2024-03-18T16:06:44.723614+0000","last_unstale":"2024-03-18T16:06:50.286651+0000","last_undegraded":"2024-03-18T16:06:50.286651+0000","last_fullsized":"2024-03-18T16:06:50.286651+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:03:17.876097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.265883+0000","last_change":"2024-03-18T16:06:44.711058+0000","last_active":"2024-03-18T16:06:50.265883+0000","last_peered":"2024-03-18T16:06:50.265883+0000","last_clean":"2024-03-18T16:06:50.265883+0000","last_became_active":"2024-03-18T16:06:44.708873+0000","last_became_peered":"2024-03-18T16:06:44.708873+0000","last_unstale":"2024-03-18T16:06:50.265883+0000","last_undegraded":"2024-03-18T16:06:50.265883+0000","last_fullsized":"2024-03-18T16:06:50.265883+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:58:08.182651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.285682+0000","last_change":"2024-03-18T16:06:44.721480+0000","last_active":"2024-03-18T16:06:50.285682+0000","last_peered":"2024-03-18T16:06:50.285682+0000","last_clean":"2024-03-18T16:06:50.285682+0000","last_became_active":"2024-03-18T16:06:44.721101+0000","last_became_peered":"2024-03-18T16:06:44.721101+0000","last_unstale":"2024-03-18T16:06:50.285682+0000","last_undegraded":"2024-03-18T16:06:50.285682+0000","last_fullsized":"2024-03-18T16:06:50.285682+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:19:43.341221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.014165+0000","last_change":"2024-03-18T16:06:44.716443+0000","last_active":"2024-03-18T16:06:50.014165+0000","last_peered":"2024-03-18T16:06:50.014165+0000","last_clean":"2024-03-18T16:06:50.014165+0000","last_became_active":"2024-03-18T16:06:44.715969+0000","last_became_peered":"2024-03-18T16:06:44.715969+0000","last_unstale":"2024-03-18T16:06:50.014165+0000","last_undegraded":"2024-03-18T16:06:50.014165+0000","last_fullsized":"2024-03-18T16:06:50.014165+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:57:14.714415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.043165+0000","last_change":"2024-03-18T16:06:44.711439+0000","last_active":"2024-03-18T16:06:50.043165+0000","last_peered":"2024-03-18T16:06:50.043165+0000","last_clean":"2024-03-18T16:06:50.043165+0000","last_became_active":"2024-03-18T16:06:44.711032+0000","last_became_peered":"2024-03-18T16:06:44.711032+0000","last_unstale":"2024-03-18T16:06:50.043165+0000","last_undegraded":"2024-03-18T16:06:50.043165+0000","last_fullsized":"2024-03-18T16:06:50.043165+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:19:56.516867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.286621+0000","last_change":"2024-03-18T16:06:44.723629+0000","last_active":"2024-03-18T16:06:50.286621+0000","last_peered":"2024-03-18T16:06:50.286621+0000","last_clean":"2024-03-18T16:06:50.286621+0000","last_became_active":"2024-03-18T16:06:44.721789+0000","last_became_peered":"2024-03-18T16:06:44.721789+0000","last_unstale":"2024-03-18T16:06:50.286621+0000","last_undegraded":"2024-03-18T16:06:50.286621+0000","last_fullsized":"2024-03-18T16:06:50.286621+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:07:59.653357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.285652+0000","last_change":"2024-03-18T16:06:44.710978+0000","last_active":"2024-03-18T16:06:50.285652+0000","last_peered":"2024-03-18T16:06:50.285652+0000","last_clean":"2024-03-18T16:06:50.285652+0000","last_became_active":"2024-03-18T16:06:44.710520+0000","last_became_peered":"2024-03-18T16:06:44.710520+0000","last_unstale":"2024-03-18T16:06:50.285652+0000","last_undegraded":"2024-03-18T16:06:50.285652+0000","last_fullsized":"2024-03-18T16:06:50.285652+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:54:53.234025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.014645+0000","last_change":"2024-03-18T16:06:44.716231+0000","last_active":"2024-03-18T16:06:50.014645+0000","last_peered":"2024-03-18T16:06:50.014645+0000","last_clean":"2024-03-18T16:06:50.014645+0000","last_became_active":"2024-03-18T16:06:44.714900+0000","last_became_peered":"2024-03-18T16:06:44.714900+0000","last_unstale":"2024-03-18T16:06:50.014645+0000","last_undegraded":"2024-03-18T16:06:50.014645+0000","last_fullsized":"2024-03-18T16:06:50.014645+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:38:54.964729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.043595+0000","last_change":"2024-03-18T16:06:44.710942+0000","last_active":"2024-03-18T16:06:50.043595+0000","last_peered":"2024-03-18T16:06:50.043595+0000","last_clean":"2024-03-18T16:06:50.043595+0000","last_became_active":"2024-03-18T16:06:44.710588+0000","last_became_peered":"2024-03-18T16:06:44.710588+0000","last_unstale":"2024-03-18T16:06:50.043595+0000","last_undegraded":"2024-03-18T16:06:50.043595+0000","last_fullsized":"2024-03-18T16:06:50.043595+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:43:19.320034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.265773+0000","last_change":"2024-03-18T16:06:44.714763+0000","last_active":"2024-03-18T16:06:50.265773+0000","last_peered":"2024-03-18T16:06:50.265773+0000","last_clean":"2024-03-18T16:06:50.265773+0000","last_became_active":"2024-03-18T16:06:44.714031+0000","last_became_peered":"2024-03-18T16:06:44.714031+0000","last_unstale":"2024-03-18T16:06:50.265773+0000","last_undegraded":"2024-03-18T16:06:50.265773+0000","last_fullsized":"2024-03-18T16:06:50.265773+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:19:25.917093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.286907+0000","last_change":"2024-03-18T16:06:44.710062+0000","last_active":"2024-03-18T16:06:50.286907+0000","last_peered":"2024-03-18T16:06:50.286907+0000","last_clean":"2024-03-18T16:06:50.286907+0000","last_became_active":"2024-03-18T16:06:44.709281+0000","last_became_peered":"2024-03-18T16:06:44.709281+0000","last_unstale":"2024-03-18T16:06:50.286907+0000","last_undegraded":"2024-03-18T16:06:50.286907+0000","last_fullsized":"2024-03-18T16:06:50.286907+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:51:30.100947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.285918+0000","last_change":"2024-03-18T16:06:44.734178+0000","last_active":"2024-03-18T16:06:50.285918+0000","last_peered":"2024-03-18T16:06:50.285918+0000","last_clean":"2024-03-18T16:06:50.285918+0000","last_became_active":"2024-03-18T16:06:44.732538+0000","last_became_peered":"2024-03-18T16:06:44.732538+0000","last_unstale":"2024-03-18T16:06:50.285918+0000","last_undegraded":"2024-03-18T16:06:50.285918+0000","last_fullsized":"2024-03-18T16:06:50.285918+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:14:26.904402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.285701+0000","last_change":"2024-03-18T16:06:44.734383+0000","last_active":"2024-03-18T16:06:50.285701+0000","last_peered":"2024-03-18T16:06:50.285701+0000","last_clean":"2024-03-18T16:06:50.285701+0000","last_became_active":"2024-03-18T16:06:44.731404+0000","last_became_peered":"2024-03-18T16:06:44.731404+0000","last_unstale":"2024-03-18T16:06:50.285701+0000","last_undegraded":"2024-03-18T16:06:50.285701+0000","last_fullsized":"2024-03-18T16:06:50.285701+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:57:03.510629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.285098+0000","last_change":"2024-03-18T16:06:44.728165+0000","last_active":"2024-03-18T16:06:50.285098+0000","last_peered":"2024-03-18T16:06:50.285098+0000","last_clean":"2024-03-18T16:06:50.285098+0000","last_became_active":"2024-03-18T16:06:44.727842+0000","last_became_peered":"2024-03-18T16:06:44.727842+0000","last_unstale":"2024-03-18T16:06:50.285098+0000","last_undegraded":"2024-03-18T16:06:50.285098+0000","last_fullsized":"2024-03-18T16:06:50.285098+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:55:04.464916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.151190+0000","last_change":"2024-03-18T16:06:44.730893+0000","last_active":"2024-03-18T16:06:50.151190+0000","last_peered":"2024-03-18T16:06:50.151190+0000","last_clean":"2024-03-18T16:06:50.151190+0000","last_became_active":"2024-03-18T16:06:44.730544+0000","last_became_peered":"2024-03-18T16:06:44.730544+0000","last_unstale":"2024-03-18T16:06:50.151190+0000","last_undegraded":"2024-03-18T16:06:50.151190+0000","last_fullsized":"2024-03-18T16:06:50.151190+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:16:37.918096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.019939+0000","last_change":"2024-03-18T16:06:44.706036+0000","last_active":"2024-03-18T16:06:50.019939+0000","last_peered":"2024-03-18T16:06:50.019939+0000","last_clean":"2024-03-18T16:06:50.019939+0000","last_became_active":"2024-03-18T16:06:44.705281+0000","last_became_peered":"2024-03-18T16:06:44.705281+0000","last_unstale":"2024-03-18T16:06:50.019939+0000","last_undegraded":"2024-03-18T16:06:50.019939+0000","last_fullsized":"2024-03-18T16:06:50.019939+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:52:20.202110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.015179+0000","last_change":"2024-03-18T16:06:44.727543+0000","last_active":"2024-03-18T16:06:50.015179+0000","last_peered":"2024-03-18T16:06:50.015179+0000","last_clean":"2024-03-18T16:06:50.015179+0000","last_became_active":"2024-03-18T16:06:44.727135+0000","last_became_peered":"2024-03-18T16:06:44.727135+0000","last_unstale":"2024-03-18T16:06:50.015179+0000","last_undegraded":"2024-03-18T16:06:50.015179+0000","last_fullsized":"2024-03-18T16:06:50.015179+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:52:42.742279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.014611+0000","last_change":"2024-03-18T16:06:49.028004+0000","last_active":"2024-03-18T16:06:50.014611+0000","last_peered":"2024-03-18T16:06:50.014611+0000","last_clean":"2024-03-18T16:06:50.014611+0000","last_became_active":"2024-03-18T16:06:49.027640+0000","last_became_peered":"2024-03-18T16:06:49.027640+0000","last_unstale":"2024-03-18T16:06:50.014611+0000","last_undegraded":"2024-03-18T16:06:50.014611+0000","last_fullsized":"2024-03-18T16:06:50.014611+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:23:07.926349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'13","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:07:20.006936+0000","last_change":"2024-03-18T16:06:43.709540+0000","last_active":"2024-03-18T16:07:20.006936+0000","last_peered":"2024-03-18T16:07:20.006936+0000","last_clean":"2024-03-18T16:07:20.006936+0000","last_became_active":"2024-03-18T16:06:43.709170+0000","last_became_peered":"2024-03-18T16:06:43.709170+0000","last_unstale":"2024-03-18T16:07:20.006936+0000","last_undegraded":"2024-03-18T16:07:20.006936+0000","last_fullsized":"2024-03-18T16:07:20.006936+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:12:53.934079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.288197+0000","last_change":"2024-03-18T16:06:44.730192+0000","last_active":"2024-03-18T16:06:50.288197+0000","last_peered":"2024-03-18T16:06:50.288197+0000","last_clean":"2024-03-18T16:06:50.288197+0000","last_became_active":"2024-03-18T16:06:44.728071+0000","last_became_peered":"2024-03-18T16:06:44.728071+0000","last_unstale":"2024-03-18T16:06:50.288197+0000","last_undegraded":"2024-03-18T16:06:50.288197+0000","last_fullsized":"2024-03-18T16:06:50.288197+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:14:59.992958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.265629+0000","last_change":"2024-03-18T16:06:49.023963+0000","last_active":"2024-03-18T16:06:50.265629+0000","last_peered":"2024-03-18T16:06:50.265629+0000","last_clean":"2024-03-18T16:06:50.265629+0000","last_became_active":"2024-03-18T16:06:49.023568+0000","last_became_peered":"2024-03-18T16:06:49.023568+0000","last_unstale":"2024-03-18T16:06:50.265629+0000","last_undegraded":"2024-03-18T16:06:50.265629+0000","last_fullsized":"2024-03-18T16:06:50.265629+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:15:19.730842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.288348+0000","last_change":"2024-03-18T16:06:43.722167+0000","last_active":"2024-03-18T16:06:50.288348+0000","last_peered":"2024-03-18T16:06:50.288348+0000","last_clean":"2024-03-18T16:06:50.288348+0000","last_became_active":"2024-03-18T16:06:43.721782+0000","last_became_peered":"2024-03-18T16:06:43.721782+0000","last_unstale":"2024-03-18T16:06:50.288348+0000","last_undegraded":"2024-03-18T16:06:50.288348+0000","last_fullsized":"2024-03-18T16:06:50.288348+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:08:54.400637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.287490+0000","last_change":"2024-03-18T16:06:44.729666+0000","last_active":"2024-03-18T16:06:50.287490+0000","last_peered":"2024-03-18T16:06:50.287490+0000","last_clean":"2024-03-18T16:06:50.287490+0000","last_became_active":"2024-03-18T16:06:44.728842+0000","last_became_peered":"2024-03-18T16:06:44.728842+0000","last_unstale":"2024-03-18T16:06:50.287490+0000","last_undegraded":"2024-03-18T16:06:50.287490+0000","last_fullsized":"2024-03-18T16:06:50.287490+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:01:42.438000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.265815+0000","last_change":"2024-03-18T16:06:49.031169+0000","last_active":"2024-03-18T16:06:50.265815+0000","last_peered":"2024-03-18T16:06:50.265815+0000","last_clean":"2024-03-18T16:06:50.265815+0000","last_became_active":"2024-03-18T16:06:49.030801+0000","last_became_peered":"2024-03-18T16:06:49.030801+0000","last_unstale":"2024-03-18T16:06:50.265815+0000","last_undegraded":"2024-03-18T16:06:50.265815+0000","last_fullsized":"2024-03-18T16:06:50.265815+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:10:54.880882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'12","reported_seq":35,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:07:20.004185+0000","last_change":"2024-03-18T16:06:43.707111+0000","last_active":"2024-03-18T16:07:20.004185+0000","last_peered":"2024-03-18T16:07:20.004185+0000","last_clean":"2024-03-18T16:07:20.004185+0000","last_became_active":"2024-03-18T16:06:43.706671+0000","last_became_peered":"2024-03-18T16:06:43.706671+0000","last_unstale":"2024-03-18T16:07:20.004185+0000","last_undegraded":"2024-03-18T16:07:20.004185+0000","last_fullsized":"2024-03-18T16:07:20.004185+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:18:34.295641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.285292+0000","last_change":"2024-03-18T16:06:44.722100+0000","last_active":"2024-03-18T16:06:50.285292+0000","last_peered":"2024-03-18T16:06:50.285292+0000","last_clean":"2024-03-18T16:06:50.285292+0000","last_became_active":"2024-03-18T16:06:44.721673+0000","last_became_peered":"2024-03-18T16:06:44.721673+0000","last_unstale":"2024-03-18T16:06:50.285292+0000","last_undegraded":"2024-03-18T16:06:50.285292+0000","last_fullsized":"2024-03-18T16:06:50.285292+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:41:22.624778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.265583+0000","last_change":"2024-03-18T16:06:49.027966+0000","last_active":"2024-03-18T16:06:50.265583+0000","last_peered":"2024-03-18T16:06:50.265583+0000","last_clean":"2024-03-18T16:06:50.265583+0000","last_became_active":"2024-03-18T16:06:49.027567+0000","last_became_peered":"2024-03-18T16:06:49.027567+0000","last_unstale":"2024-03-18T16:06:50.265583+0000","last_undegraded":"2024-03-18T16:06:50.265583+0000","last_fullsized":"2024-03-18T16:06:50.265583+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:54:28.543019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:07:20.003614+0000","last_change":"2024-03-18T16:06:43.704023+0000","last_active":"2024-03-18T16:07:20.003614+0000","last_peered":"2024-03-18T16:07:20.003614+0000","last_clean":"2024-03-18T16:07:20.003614+0000","last_became_active":"2024-03-18T16:06:43.703690+0000","last_became_peered":"2024-03-18T16:06:43.703690+0000","last_unstale":"2024-03-18T16:07:20.003614+0000","last_undegraded":"2024-03-18T16:07:20.003614+0000","last_fullsized":"2024-03-18T16:07:20.003614+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:20:02.344346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.285990+0000","last_change":"2024-03-18T16:06:44.708803+0000","last_active":"2024-03-18T16:06:50.285990+0000","last_peered":"2024-03-18T16:06:50.285990+0000","last_clean":"2024-03-18T16:06:50.285990+0000","last_became_active":"2024-03-18T16:06:44.707863+0000","last_became_peered":"2024-03-18T16:06:44.707863+0000","last_unstale":"2024-03-18T16:06:50.285990+0000","last_undegraded":"2024-03-18T16:06:50.285990+0000","last_fullsized":"2024-03-18T16:06:50.285990+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:40:58.800490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.015091+0000","last_change":"2024-03-18T16:06:49.012310+0000","last_active":"2024-03-18T16:06:50.015091+0000","last_peered":"2024-03-18T16:06:50.015091+0000","last_clean":"2024-03-18T16:06:50.015091+0000","last_became_active":"2024-03-18T16:06:49.011844+0000","last_became_peered":"2024-03-18T16:06:49.011844+0000","last_unstale":"2024-03-18T16:06:50.015091+0000","last_undegraded":"2024-03-18T16:06:50.015091+0000","last_fullsized":"2024-03-18T16:06:50.015091+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:16:42.652205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'6","reported_seq":34,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:07:20.983908+0000","last_change":"2024-03-18T16:06:43.705653+0000","last_active":"2024-03-18T16:07:20.983908+0000","last_peered":"2024-03-18T16:07:20.983908+0000","last_clean":"2024-03-18T16:07:20.983908+0000","last_became_active":"2024-03-18T16:06:43.705320+0000","last_became_peered":"2024-03-18T16:06:43.705320+0000","last_unstale":"2024-03-18T16:07:20.983908+0000","last_undegraded":"2024-03-18T16:07:20.983908+0000","last_fullsized":"2024-03-18T16:07:20.983908+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:10:10.827338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.286101+0000","last_change":"2024-03-18T16:06:44.720045+0000","last_active":"2024-03-18T16:06:50.286101+0000","last_peered":"2024-03-18T16:06:50.286101+0000","last_clean":"2024-03-18T16:06:50.286101+0000","last_became_active":"2024-03-18T16:06:44.719669+0000","last_became_peered":"2024-03-18T16:06:44.719669+0000","last_unstale":"2024-03-18T16:06:50.286101+0000","last_undegraded":"2024-03-18T16:06:50.286101+0000","last_fullsized":"2024-03-18T16:06:50.286101+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:23:47.464057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.014824+0000","last_change":"2024-03-18T16:06:49.032576+0000","last_active":"2024-03-18T16:06:50.014824+0000","last_peered":"2024-03-18T16:06:50.014824+0000","last_clean":"2024-03-18T16:06:50.014824+0000","last_became_active":"2024-03-18T16:06:49.032126+0000","last_became_peered":"2024-03-18T16:06:49.032126+0000","last_unstale":"2024-03-18T16:06:50.014824+0000","last_undegraded":"2024-03-18T16:06:50.014824+0000","last_fullsized":"2024-03-18T16:06:50.014824+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:41:54.822070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":29,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:07:19.242451+0000","last_change":"2024-03-18T16:06:43.706364+0000","last_active":"2024-03-18T16:07:19.242451+0000","last_peered":"2024-03-18T16:07:19.242451+0000","last_clean":"2024-03-18T16:07:19.242451+0000","last_became_active":"2024-03-18T16:06:43.705984+0000","last_became_peered":"2024-03-18T16:06:43.705984+0000","last_unstale":"2024-03-18T16:07:19.242451+0000","last_undegraded":"2024-03-18T16:07:19.242451+0000","last_fullsized":"2024-03-18T16:07:19.242451+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:46:06.503000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.286117+0000","last_change":"2024-03-18T16:06:44.721019+0000","last_active":"2024-03-18T16:06:50.286117+0000","last_peered":"2024-03-18T16:06:50.286117+0000","last_clean":"2024-03-18T16:06:50.286117+0000","last_became_active":"2024-03-18T16:06:44.720665+0000","last_became_peered":"2024-03-18T16:06:44.720665+0000","last_unstale":"2024-03-18T16:06:50.286117+0000","last_undegraded":"2024-03-18T16:06:50.286117+0000","last_fullsized":"2024-03-18T16:06:50.286117+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:58:34.180653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.151293+0000","last_change":"2024-03-18T16:06:49.022919+0000","last_active":"2024-03-18T16:06:50.151293+0000","last_peered":"2024-03-18T16:06:50.151293+0000","last_clean":"2024-03-18T16:06:50.151293+0000","last_became_active":"2024-03-18T16:06:49.022514+0000","last_became_peered":"2024-03-18T16:06:49.022514+0000","last_unstale":"2024-03-18T16:06:50.151293+0000","last_undegraded":"2024-03-18T16:06:50.151293+0000","last_fullsized":"2024-03-18T16:06:50.151293+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:46:39.510181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:07:18.229817+0000","last_change":"2024-03-18T16:06:43.713025+0000","last_active":"2024-03-18T16:07:18.229817+0000","last_peered":"2024-03-18T16:07:18.229817+0000","last_clean":"2024-03-18T16:07:18.229817+0000","last_became_active":"2024-03-18T16:06:43.711469+0000","last_became_peered":"2024-03-18T16:06:43.711469+0000","last_unstale":"2024-03-18T16:07:18.229817+0000","last_undegraded":"2024-03-18T16:07:18.229817+0000","last_fullsized":"2024-03-18T16:07:18.229817+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:30:41.193229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.286584+0000","last_change":"2024-03-18T16:06:44.723953+0000","last_active":"2024-03-18T16:06:50.286584+0000","last_peered":"2024-03-18T16:06:50.286584+0000","last_clean":"2024-03-18T16:06:50.286584+0000","last_became_active":"2024-03-18T16:06:44.723266+0000","last_became_peered":"2024-03-18T16:06:44.723266+0000","last_unstale":"2024-03-18T16:06:50.286584+0000","last_undegraded":"2024-03-18T16:06:50.286584+0000","last_fullsized":"2024-03-18T16:06:50.286584+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:41:10.067297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.287077+0000","last_change":"2024-03-18T16:06:49.039472+0000","last_active":"2024-03-18T16:06:50.287077+0000","last_peered":"2024-03-18T16:06:50.287077+0000","last_clean":"2024-03-18T16:06:50.287077+0000","last_became_active":"2024-03-18T16:06:49.038889+0000","last_became_peered":"2024-03-18T16:06:49.038889+0000","last_unstale":"2024-03-18T16:06:50.287077+0000","last_undegraded":"2024-03-18T16:06:50.287077+0000","last_fullsized":"2024-03-18T16:06:50.287077+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:03:05.345315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:07:19.980692+0000","last_change":"2024-03-18T16:06:43.705873+0000","last_active":"2024-03-18T16:07:19.980692+0000","last_peered":"2024-03-18T16:07:19.980692+0000","last_clean":"2024-03-18T16:07:19.980692+0000","last_became_active":"2024-03-18T16:06:43.705548+0000","last_became_peered":"2024-03-18T16:06:43.705548+0000","last_unstale":"2024-03-18T16:07:19.980692+0000","last_undegraded":"2024-03-18T16:07:19.980692+0000","last_fullsized":"2024-03-18T16:07:19.980692+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:52:03.790888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.018009+0000","last_change":"2024-03-18T16:06:44.718526+0000","last_active":"2024-03-18T16:06:50.018009+0000","last_peered":"2024-03-18T16:06:50.018009+0000","last_clean":"2024-03-18T16:06:50.018009+0000","last_became_active":"2024-03-18T16:06:44.717266+0000","last_became_peered":"2024-03-18T16:06:44.717266+0000","last_unstale":"2024-03-18T16:06:50.018009+0000","last_undegraded":"2024-03-18T16:06:50.018009+0000","last_fullsized":"2024-03-18T16:06:50.018009+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:55:09.340354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.287346+0000","last_change":"2024-03-18T16:06:49.021971+0000","last_active":"2024-03-18T16:06:50.287346+0000","last_peered":"2024-03-18T16:06:50.287346+0000","last_clean":"2024-03-18T16:06:50.287346+0000","last_became_active":"2024-03-18T16:06:49.021577+0000","last_became_peered":"2024-03-18T16:06:49.021577+0000","last_unstale":"2024-03-18T16:06:50.287346+0000","last_undegraded":"2024-03-18T16:06:50.287346+0000","last_fullsized":"2024-03-18T16:06:50.287346+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:47:44.768559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'4","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:07:17.207443+0000","last_change":"2024-03-18T16:06:43.705636+0000","last_active":"2024-03-18T16:07:17.207443+0000","last_peered":"2024-03-18T16:07:17.207443+0000","last_clean":"2024-03-18T16:07:17.207443+0000","last_became_active":"2024-03-18T16:06:43.705268+0000","last_became_peered":"2024-03-18T16:06:43.705268+0000","last_unstale":"2024-03-18T16:07:17.207443+0000","last_undegraded":"2024-03-18T16:07:17.207443+0000","last_fullsized":"2024-03-18T16:07:17.207443+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:37:31.367130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.285718+0000","last_change":"2024-03-18T16:06:44.712237+0000","last_active":"2024-03-18T16:06:50.285718+0000","last_peered":"2024-03-18T16:06:50.285718+0000","last_clean":"2024-03-18T16:06:50.285718+0000","last_became_active":"2024-03-18T16:06:44.711677+0000","last_became_peered":"2024-03-18T16:06:44.711677+0000","last_unstale":"2024-03-18T16:06:50.285718+0000","last_undegraded":"2024-03-18T16:06:50.285718+0000","last_fullsized":"2024-03-18T16:06:50.285718+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:16:56.326962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.018043+0000","last_change":"2024-03-18T16:06:49.034578+0000","last_active":"2024-03-18T16:06:50.018043+0000","last_peered":"2024-03-18T16:06:50.018043+0000","last_clean":"2024-03-18T16:06:50.018043+0000","last_became_active":"2024-03-18T16:06:49.031099+0000","last_became_peered":"2024-03-18T16:06:49.031099+0000","last_unstale":"2024-03-18T16:06:50.018043+0000","last_undegraded":"2024-03-18T16:06:50.018043+0000","last_fullsized":"2024-03-18T16:06:50.018043+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:11:08.042550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:07:20.004223+0000","last_change":"2024-03-18T16:06:43.708151+0000","last_active":"2024-03-18T16:07:20.004223+0000","last_peered":"2024-03-18T16:07:20.004223+0000","last_clean":"2024-03-18T16:07:20.004223+0000","last_became_active":"2024-03-18T16:06:43.707810+0000","last_became_peered":"2024-03-18T16:06:43.707810+0000","last_unstale":"2024-03-18T16:07:20.004223+0000","last_undegraded":"2024-03-18T16:07:20.004223+0000","last_fullsized":"2024-03-18T16:07:20.004223+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:30:35.745772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.287628+0000","last_change":"2024-03-18T16:06:44.729777+0000","last_active":"2024-03-18T16:06:50.287628+0000","last_peered":"2024-03-18T16:06:50.287628+0000","last_clean":"2024-03-18T16:06:50.287628+0000","last_became_active":"2024-03-18T16:06:44.729244+0000","last_became_peered":"2024-03-18T16:06:44.729244+0000","last_unstale":"2024-03-18T16:06:50.287628+0000","last_undegraded":"2024-03-18T16:06:50.287628+0000","last_fullsized":"2024-03-18T16:06:50.287628+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:52:05.420443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.265917+0000","last_change":"2024-03-18T16:06:49.014087+0000","last_active":"2024-03-18T16:06:50.265917+0000","last_peered":"2024-03-18T16:06:50.265917+0000","last_clean":"2024-03-18T16:06:50.265917+0000","last_became_active":"2024-03-18T16:06:49.013556+0000","last_became_peered":"2024-03-18T16:06:49.013556+0000","last_unstale":"2024-03-18T16:06:50.265917+0000","last_undegraded":"2024-03-18T16:06:50.265917+0000","last_fullsized":"2024-03-18T16:06:50.265917+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:13:59.638977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:07:19.247654+0000","last_change":"2024-03-18T16:06:43.715280+0000","last_active":"2024-03-18T16:07:19.247654+0000","last_peered":"2024-03-18T16:07:19.247654+0000","last_clean":"2024-03-18T16:07:19.247654+0000","last_became_active":"2024-03-18T16:06:43.714936+0000","last_became_peered":"2024-03-18T16:06:43.714936+0000","last_unstale":"2024-03-18T16:07:19.247654+0000","last_undegraded":"2024-03-18T16:07:19.247654+0000","last_fullsized":"2024-03-18T16:07:19.247654+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:07:38.746932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.285428+0000","last_change":"2024-03-18T16:06:44.709004+0000","last_active":"2024-03-18T16:06:50.285428+0000","last_peered":"2024-03-18T16:06:50.285428+0000","last_clean":"2024-03-18T16:06:50.285428+0000","last_became_active":"2024-03-18T16:06:44.708574+0000","last_became_peered":"2024-03-18T16:06:44.708574+0000","last_unstale":"2024-03-18T16:06:50.285428+0000","last_undegraded":"2024-03-18T16:06:50.285428+0000","last_fullsized":"2024-03-18T16:06:50.285428+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:02:50.982110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.018789+0000","last_change":"2024-03-18T16:06:49.029806+0000","last_active":"2024-03-18T16:06:50.018789+0000","last_peered":"2024-03-18T16:06:50.018789+0000","last_clean":"2024-03-18T16:06:50.018789+0000","last_became_active":"2024-03-18T16:06:49.029404+0000","last_became_peered":"2024-03-18T16:06:49.029404+0000","last_unstale":"2024-03-18T16:06:50.018789+0000","last_undegraded":"2024-03-18T16:06:50.018789+0000","last_fullsized":"2024-03-18T16:06:50.018789+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:38:37.706620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'12","reported_seq":34,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:07:17.201229+0000","last_change":"2024-03-18T16:06:43.702910+0000","last_active":"2024-03-18T16:07:17.201229+0000","last_peered":"2024-03-18T16:07:17.201229+0000","last_clean":"2024-03-18T16:07:17.201229+0000","last_became_active":"2024-03-18T16:06:43.702557+0000","last_became_peered":"2024-03-18T16:06:43.702557+0000","last_unstale":"2024-03-18T16:07:17.201229+0000","last_undegraded":"2024-03-18T16:07:17.201229+0000","last_fullsized":"2024-03-18T16:07:17.201229+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:55:15.362987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.285820+0000","last_change":"2024-03-18T16:06:44.709075+0000","last_active":"2024-03-18T16:06:50.285820+0000","last_peered":"2024-03-18T16:06:50.285820+0000","last_clean":"2024-03-18T16:06:50.285820+0000","last_became_active":"2024-03-18T16:06:44.708366+0000","last_became_peered":"2024-03-18T16:06:44.708366+0000","last_unstale":"2024-03-18T16:06:50.285820+0000","last_undegraded":"2024-03-18T16:06:50.285820+0000","last_fullsized":"2024-03-18T16:06:50.285820+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:44:18.486385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.286659+0000","last_change":"2024-03-18T16:06:49.037380+0000","last_active":"2024-03-18T16:06:50.286659+0000","last_peered":"2024-03-18T16:06:50.286659+0000","last_clean":"2024-03-18T16:06:50.286659+0000","last_became_active":"2024-03-18T16:06:49.037021+0000","last_became_peered":"2024-03-18T16:06:49.037021+0000","last_unstale":"2024-03-18T16:06:50.286659+0000","last_undegraded":"2024-03-18T16:06:50.286659+0000","last_fullsized":"2024-03-18T16:06:50.286659+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:43:03.482631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:07:18.225050+0000","last_change":"2024-03-18T16:06:43.722287+0000","last_active":"2024-03-18T16:07:18.225050+0000","last_peered":"2024-03-18T16:07:18.225050+0000","last_clean":"2024-03-18T16:07:18.225050+0000","last_became_active":"2024-03-18T16:06:43.721919+0000","last_became_peered":"2024-03-18T16:06:43.721919+0000","last_unstale":"2024-03-18T16:07:18.225050+0000","last_undegraded":"2024-03-18T16:07:18.225050+0000","last_fullsized":"2024-03-18T16:07:18.225050+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:10:07.606112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.288783+0000","last_change":"2024-03-18T16:06:44.727204+0000","last_active":"2024-03-18T16:06:50.288783+0000","last_peered":"2024-03-18T16:06:50.288783+0000","last_clean":"2024-03-18T16:06:50.288783+0000","last_became_active":"2024-03-18T16:06:44.726775+0000","last_became_peered":"2024-03-18T16:06:44.726775+0000","last_unstale":"2024-03-18T16:06:50.288783+0000","last_undegraded":"2024-03-18T16:06:50.288783+0000","last_fullsized":"2024-03-18T16:06:50.288783+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:29:44.842972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.151291+0000","last_change":"2024-03-18T16:06:49.022508+0000","last_active":"2024-03-18T16:06:50.151291+0000","last_peered":"2024-03-18T16:06:50.151291+0000","last_clean":"2024-03-18T16:06:50.151291+0000","last_became_active":"2024-03-18T16:06:49.022017+0000","last_became_peered":"2024-03-18T16:06:49.022017+0000","last_unstale":"2024-03-18T16:06:50.151291+0000","last_undegraded":"2024-03-18T16:06:50.151291+0000","last_fullsized":"2024-03-18T16:06:50.151291+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:52:45.892880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:07:20.006950+0000","last_change":"2024-03-18T16:06:43.719618+0000","last_active":"2024-03-18T16:07:20.006950+0000","last_peered":"2024-03-18T16:07:20.006950+0000","last_clean":"2024-03-18T16:07:20.006950+0000","last_became_active":"2024-03-18T16:06:43.719161+0000","last_became_peered":"2024-03-18T16:06:43.719161+0000","last_unstale":"2024-03-18T16:07:20.006950+0000","last_undegraded":"2024-03-18T16:07:20.006950+0000","last_fullsized":"2024-03-18T16:07:20.006950+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:39:53.077878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.043367+0000","last_change":"2024-03-18T16:06:44.717829+0000","last_active":"2024-03-18T16:06:50.043367+0000","last_peered":"2024-03-18T16:06:50.043367+0000","last_clean":"2024-03-18T16:06:50.043367+0000","last_became_active":"2024-03-18T16:06:44.717473+0000","last_became_peered":"2024-03-18T16:06:44.717473+0000","last_unstale":"2024-03-18T16:06:50.043367+0000","last_undegraded":"2024-03-18T16:06:50.043367+0000","last_fullsized":"2024-03-18T16:06:50.043367+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:10:00.183613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.019958+0000","last_change":"2024-03-18T16:06:49.036766+0000","last_active":"2024-03-18T16:06:50.019958+0000","last_peered":"2024-03-18T16:06:50.019958+0000","last_clean":"2024-03-18T16:06:50.019958+0000","last_became_active":"2024-03-18T16:06:49.034617+0000","last_became_peered":"2024-03-18T16:06:49.034617+0000","last_unstale":"2024-03-18T16:06:50.019958+0000","last_undegraded":"2024-03-18T16:06:50.019958+0000","last_fullsized":"2024-03-18T16:06:50.019958+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:18:59.033781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'5","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:07:20.985685+0000","last_change":"2024-03-18T16:06:43.702452+0000","last_active":"2024-03-18T16:07:20.985685+0000","last_peered":"2024-03-18T16:07:20.985685+0000","last_clean":"2024-03-18T16:07:20.985685+0000","last_became_active":"2024-03-18T16:06:43.702141+0000","last_became_peered":"2024-03-18T16:06:43.702141+0000","last_unstale":"2024-03-18T16:07:20.985685+0000","last_undegraded":"2024-03-18T16:07:20.985685+0000","last_fullsized":"2024-03-18T16:07:20.985685+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:44:05.366101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.043170+0000","last_change":"2024-03-18T16:06:44.711403+0000","last_active":"2024-03-18T16:06:50.043170+0000","last_peered":"2024-03-18T16:06:50.043170+0000","last_clean":"2024-03-18T16:06:50.043170+0000","last_became_active":"2024-03-18T16:06:44.710654+0000","last_became_peered":"2024-03-18T16:06:44.710654+0000","last_unstale":"2024-03-18T16:06:50.043170+0000","last_undegraded":"2024-03-18T16:06:50.043170+0000","last_fullsized":"2024-03-18T16:06:50.043170+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:21:43.633274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.287436+0000","last_change":"2024-03-18T16:06:49.039192+0000","last_active":"2024-03-18T16:06:50.287436+0000","last_peered":"2024-03-18T16:06:50.287436+0000","last_clean":"2024-03-18T16:06:50.287436+0000","last_became_active":"2024-03-18T16:06:49.038421+0000","last_became_peered":"2024-03-18T16:06:49.038421+0000","last_unstale":"2024-03-18T16:06:50.287436+0000","last_undegraded":"2024-03-18T16:06:50.287436+0000","last_fullsized":"2024-03-18T16:06:50.287436+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:54:56.944209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":40,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:07:19.978151+0000","last_change":"2024-03-18T16:06:43.712523+0000","last_active":"2024-03-18T16:07:19.978151+0000","last_peered":"2024-03-18T16:07:19.978151+0000","last_clean":"2024-03-18T16:07:19.978151+0000","last_became_active":"2024-03-18T16:06:43.712151+0000","last_became_peered":"2024-03-18T16:06:43.712151+0000","last_unstale":"2024-03-18T16:07:19.978151+0000","last_undegraded":"2024-03-18T16:07:19.978151+0000","last_fullsized":"2024-03-18T16:07:19.978151+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:17:19.651293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.285772+0000","last_change":"2024-03-18T16:06:44.713522+0000","last_active":"2024-03-18T16:06:50.285772+0000","last_peered":"2024-03-18T16:06:50.285772+0000","last_clean":"2024-03-18T16:06:50.285772+0000","last_became_active":"2024-03-18T16:06:44.713152+0000","last_became_peered":"2024-03-18T16:06:44.713152+0000","last_unstale":"2024-03-18T16:06:50.285772+0000","last_undegraded":"2024-03-18T16:06:50.285772+0000","last_fullsized":"2024-03-18T16:06:50.285772+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:56:01.213384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.285697+0000","last_change":"2024-03-18T16:06:49.028943+0000","last_active":"2024-03-18T16:06:50.285697+0000","last_peered":"2024-03-18T16:06:50.285697+0000","last_clean":"2024-03-18T16:06:50.285697+0000","last_became_active":"2024-03-18T16:06:49.028453+0000","last_became_peered":"2024-03-18T16:06:49.028453+0000","last_unstale":"2024-03-18T16:06:50.285697+0000","last_undegraded":"2024-03-18T16:06:50.285697+0000","last_fullsized":"2024-03-18T16:06:50.285697+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:16:30.318649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:07:19.977899+0000","last_change":"2024-03-18T16:06:43.706840+0000","last_active":"2024-03-18T16:07:19.977899+0000","last_peered":"2024-03-18T16:07:19.977899+0000","last_clean":"2024-03-18T16:07:19.977899+0000","last_became_active":"2024-03-18T16:06:43.706486+0000","last_became_peered":"2024-03-18T16:06:43.706486+0000","last_unstale":"2024-03-18T16:07:19.977899+0000","last_undegraded":"2024-03-18T16:07:19.977899+0000","last_fullsized":"2024-03-18T16:07:19.977899+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:16:44.542471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.151070+0000","last_change":"2024-03-18T16:06:44.716284+0000","last_active":"2024-03-18T16:06:50.151070+0000","last_peered":"2024-03-18T16:06:50.151070+0000","last_clean":"2024-03-18T16:06:50.151070+0000","last_became_active":"2024-03-18T16:06:44.715911+0000","last_became_peered":"2024-03-18T16:06:44.715911+0000","last_unstale":"2024-03-18T16:06:50.151070+0000","last_undegraded":"2024-03-18T16:06:50.151070+0000","last_fullsized":"2024-03-18T16:06:50.151070+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:48:34.839514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.286032+0000","last_change":"2024-03-18T16:06:49.059061+0000","last_active":"2024-03-18T16:06:50.286032+0000","last_peered":"2024-03-18T16:06:50.286032+0000","last_clean":"2024-03-18T16:06:50.286032+0000","last_became_active":"2024-03-18T16:06:49.058619+0000","last_became_peered":"2024-03-18T16:06:49.058619+0000","last_unstale":"2024-03-18T16:06:50.286032+0000","last_undegraded":"2024-03-18T16:06:50.286032+0000","last_fullsized":"2024-03-18T16:06:50.286032+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:44:52.970785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.019296+0000","last_change":"2024-03-18T16:06:43.715766+0000","last_active":"2024-03-18T16:06:50.019296+0000","last_peered":"2024-03-18T16:06:50.019296+0000","last_clean":"2024-03-18T16:06:50.019296+0000","last_became_active":"2024-03-18T16:06:43.715380+0000","last_became_peered":"2024-03-18T16:06:43.715380+0000","last_unstale":"2024-03-18T16:06:50.019296+0000","last_undegraded":"2024-03-18T16:06:50.019296+0000","last_fullsized":"2024-03-18T16:06:50.019296+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:13:44.195934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.014473+0000","last_change":"2024-03-18T16:06:44.728150+0000","last_active":"2024-03-18T16:06:50.014473+0000","last_peered":"2024-03-18T16:06:50.014473+0000","last_clean":"2024-03-18T16:06:50.014473+0000","last_became_active":"2024-03-18T16:06:44.727769+0000","last_became_peered":"2024-03-18T16:06:44.727769+0000","last_unstale":"2024-03-18T16:06:50.014473+0000","last_undegraded":"2024-03-18T16:06:50.014473+0000","last_fullsized":"2024-03-18T16:06:50.014473+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:28:16.023225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.288303+0000","last_change":"2024-03-18T16:06:49.033671+0000","last_active":"2024-03-18T16:06:50.288303+0000","last_peered":"2024-03-18T16:06:50.288303+0000","last_clean":"2024-03-18T16:06:50.288303+0000","last_became_active":"2024-03-18T16:06:49.033046+0000","last_became_peered":"2024-03-18T16:06:49.033046+0000","last_unstale":"2024-03-18T16:06:50.288303+0000","last_undegraded":"2024-03-18T16:06:50.288303+0000","last_fullsized":"2024-03-18T16:06:50.288303+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:26:37.020521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'5","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:07:21.010151+0000","last_change":"2024-03-18T16:06:43.712636+0000","last_active":"2024-03-18T16:07:21.010151+0000","last_peered":"2024-03-18T16:07:21.010151+0000","last_clean":"2024-03-18T16:07:21.010151+0000","last_became_active":"2024-03-18T16:06:43.712300+0000","last_became_peered":"2024-03-18T16:06:43.712300+0000","last_unstale":"2024-03-18T16:07:21.010151+0000","last_undegraded":"2024-03-18T16:07:21.010151+0000","last_fullsized":"2024-03-18T16:07:21.010151+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:48:20.386640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.014268+0000","last_change":"2024-03-18T16:06:44.731583+0000","last_active":"2024-03-18T16:06:50.014268+0000","last_peered":"2024-03-18T16:06:50.014268+0000","last_clean":"2024-03-18T16:06:50.014268+0000","last_became_active":"2024-03-18T16:06:44.731183+0000","last_became_peered":"2024-03-18T16:06:44.731183+0000","last_unstale":"2024-03-18T16:06:50.014268+0000","last_undegraded":"2024-03-18T16:06:50.014268+0000","last_fullsized":"2024-03-18T16:06:50.014268+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:23:05.098101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.288606+0000","last_change":"2024-03-18T16:06:49.060130+0000","last_active":"2024-03-18T16:06:50.288606+0000","last_peered":"2024-03-18T16:06:50.288606+0000","last_clean":"2024-03-18T16:06:50.288606+0000","last_became_active":"2024-03-18T16:06:49.059109+0000","last_became_peered":"2024-03-18T16:06:49.059109+0000","last_unstale":"2024-03-18T16:06:50.288606+0000","last_undegraded":"2024-03-18T16:06:50.288606+0000","last_fullsized":"2024-03-18T16:06:50.288606+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:41:40.008047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.285940+0000","last_change":"2024-03-18T16:06:44.705793+0000","last_active":"2024-03-18T16:06:50.285940+0000","last_peered":"2024-03-18T16:06:50.285940+0000","last_clean":"2024-03-18T16:06:50.285940+0000","last_became_active":"2024-03-18T16:06:44.705426+0000","last_became_peered":"2024-03-18T16:06:44.705426+0000","last_unstale":"2024-03-18T16:06:50.285940+0000","last_undegraded":"2024-03-18T16:06:50.285940+0000","last_fullsized":"2024-03-18T16:06:50.285940+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:14:31.198829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'5","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:07:20.985121+0000","last_change":"2024-03-18T16:06:43.721057+0000","last_active":"2024-03-18T16:07:20.985121+0000","last_peered":"2024-03-18T16:07:20.985121+0000","last_clean":"2024-03-18T16:07:20.985121+0000","last_became_active":"2024-03-18T16:06:43.720661+0000","last_became_peered":"2024-03-18T16:06:43.720661+0000","last_unstale":"2024-03-18T16:07:20.985121+0000","last_undegraded":"2024-03-18T16:07:20.985121+0000","last_fullsized":"2024-03-18T16:07:20.985121+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:08:18.050511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:52.454967+0000","last_change":"2024-03-18T16:06:49.021836+0000","last_active":"2024-03-18T16:06:52.454967+0000","last_peered":"2024-03-18T16:06:52.454967+0000","last_clean":"2024-03-18T16:06:52.454967+0000","last_became_active":"2024-03-18T16:06:49.021474+0000","last_became_peered":"2024-03-18T16:06:49.021474+0000","last_unstale":"2024-03-18T16:06:52.454967+0000","last_undegraded":"2024-03-18T16:06:52.454967+0000","last_fullsized":"2024-03-18T16:06:52.454967+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:46:59.100154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.287039+0000","last_change":"2024-03-18T16:06:44.725555+0000","last_active":"2024-03-18T16:06:50.287039+0000","last_peered":"2024-03-18T16:06:50.287039+0000","last_clean":"2024-03-18T16:06:50.287039+0000","last_became_active":"2024-03-18T16:06:44.719880+0000","last_became_peered":"2024-03-18T16:06:44.719880+0000","last_unstale":"2024-03-18T16:06:50.287039+0000","last_undegraded":"2024-03-18T16:06:50.287039+0000","last_fullsized":"2024-03-18T16:06:50.287039+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:45:17.830876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:07:20.009759+0000","last_change":"2024-03-18T16:06:43.700814+0000","last_active":"2024-03-18T16:07:20.009759+0000","last_peered":"2024-03-18T16:07:20.009759+0000","last_clean":"2024-03-18T16:07:20.009759+0000","last_became_active":"2024-03-18T16:06:43.698902+0000","last_became_peered":"2024-03-18T16:06:43.698902+0000","last_unstale":"2024-03-18T16:07:20.009759+0000","last_undegraded":"2024-03-18T16:07:20.009759+0000","last_fullsized":"2024-03-18T16:07:20.009759+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:20:20.100033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":344,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.043041+0000","last_change":"2024-03-18T16:05:22.379397+0000","last_active":"2024-03-18T16:06:50.043041+0000","last_peered":"2024-03-18T16:06:50.043041+0000","last_clean":"2024-03-18T16:06:50.043041+0000","last_became_active":"2024-03-18T16:05:22.378830+0000","last_became_peered":"2024-03-18T16:05:22.378830+0000","last_unstale":"2024-03-18T16:06:50.043041+0000","last_undegraded":"2024-03-18T16:06:50.043041+0000","last_fullsized":"2024-03-18T16:06:50.043041+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:01:30.048984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:01:30.048984+0000","last_clean_scrub_stamp":"2024-03-18T16:01:30.048984+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:35:45.796495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.286309+0000","last_change":"2024-03-18T16:06:49.058931+0000","last_active":"2024-03-18T16:06:50.286309+0000","last_peered":"2024-03-18T16:06:50.286309+0000","last_clean":"2024-03-18T16:06:50.286309+0000","last_became_active":"2024-03-18T16:06:49.058497+0000","last_became_peered":"2024-03-18T16:06:49.058497+0000","last_unstale":"2024-03-18T16:06:50.286309+0000","last_undegraded":"2024-03-18T16:06:50.286309+0000","last_fullsized":"2024-03-18T16:06:50.286309+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:41:51.874582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.285700+0000","last_change":"2024-03-18T16:06:44.710332+0000","last_active":"2024-03-18T16:06:50.285700+0000","last_peered":"2024-03-18T16:06:50.285700+0000","last_clean":"2024-03-18T16:06:50.285700+0000","last_became_active":"2024-03-18T16:06:44.709952+0000","last_became_peered":"2024-03-18T16:06:44.709952+0000","last_unstale":"2024-03-18T16:06:50.285700+0000","last_undegraded":"2024-03-18T16:06:50.285700+0000","last_fullsized":"2024-03-18T16:06:50.285700+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:24:26.653511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:07:18.224446+0000","last_change":"2024-03-18T16:06:43.708265+0000","last_active":"2024-03-18T16:07:18.224446+0000","last_peered":"2024-03-18T16:07:18.224446+0000","last_clean":"2024-03-18T16:07:18.224446+0000","last_became_active":"2024-03-18T16:06:43.707466+0000","last_became_peered":"2024-03-18T16:06:43.707466+0000","last_unstale":"2024-03-18T16:07:18.224446+0000","last_undegraded":"2024-03-18T16:07:18.224446+0000","last_fullsized":"2024-03-18T16:07:18.224446+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:45:40.065396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.151226+0000","last_change":"2024-03-18T16:06:49.022401+0000","last_active":"2024-03-18T16:06:50.151226+0000","last_peered":"2024-03-18T16:06:50.151226+0000","last_clean":"2024-03-18T16:06:50.151226+0000","last_became_active":"2024-03-18T16:06:49.021966+0000","last_became_peered":"2024-03-18T16:06:49.021966+0000","last_unstale":"2024-03-18T16:06:50.151226+0000","last_undegraded":"2024-03-18T16:06:50.151226+0000","last_fullsized":"2024-03-18T16:06:50.151226+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:04:12.434969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:07:19.978716+0000","last_change":"2024-03-18T16:06:43.713249+0000","last_active":"2024-03-18T16:07:19.978716+0000","last_peered":"2024-03-18T16:07:19.978716+0000","last_clean":"2024-03-18T16:07:19.978716+0000","last_became_active":"2024-03-18T16:06:43.712897+0000","last_became_peered":"2024-03-18T16:06:43.712897+0000","last_unstale":"2024-03-18T16:07:19.978716+0000","last_undegraded":"2024-03-18T16:07:19.978716+0000","last_fullsized":"2024-03-18T16:07:19.978716+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:20:04.642568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.288425+0000","last_change":"2024-03-18T16:06:44.710034+0000","last_active":"2024-03-18T16:06:50.288425+0000","last_peered":"2024-03-18T16:06:50.288425+0000","last_clean":"2024-03-18T16:06:50.288425+0000","last_became_active":"2024-03-18T16:06:44.708457+0000","last_became_peered":"2024-03-18T16:06:44.708457+0000","last_unstale":"2024-03-18T16:06:50.288425+0000","last_undegraded":"2024-03-18T16:06:50.288425+0000","last_fullsized":"2024-03-18T16:06:50.288425+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:40:00.504382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.286557+0000","last_change":"2024-03-18T16:06:49.059345+0000","last_active":"2024-03-18T16:06:50.286557+0000","last_peered":"2024-03-18T16:06:50.286557+0000","last_clean":"2024-03-18T16:06:50.286557+0000","last_became_active":"2024-03-18T16:06:49.058675+0000","last_became_peered":"2024-03-18T16:06:49.058675+0000","last_unstale":"2024-03-18T16:06:50.286557+0000","last_undegraded":"2024-03-18T16:06:50.286557+0000","last_fullsized":"2024-03-18T16:06:50.286557+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:03:33.730074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'9","reported_seq":32,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:07:20.833925+0000","last_change":"2024-03-18T16:06:43.724589+0000","last_active":"2024-03-18T16:07:20.833925+0000","last_peered":"2024-03-18T16:07:20.833925+0000","last_clean":"2024-03-18T16:07:20.833925+0000","last_became_active":"2024-03-18T16:06:43.724125+0000","last_became_peered":"2024-03-18T16:06:43.724125+0000","last_unstale":"2024-03-18T16:07:20.833925+0000","last_undegraded":"2024-03-18T16:07:20.833925+0000","last_fullsized":"2024-03-18T16:07:20.833925+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:08:34.921118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.287632+0000","last_change":"2024-03-18T16:06:44.727162+0000","last_active":"2024-03-18T16:06:50.287632+0000","last_peered":"2024-03-18T16:06:50.287632+0000","last_clean":"2024-03-18T16:06:50.287632+0000","last_became_active":"2024-03-18T16:06:44.726552+0000","last_became_peered":"2024-03-18T16:06:44.726552+0000","last_unstale":"2024-03-18T16:06:50.287632+0000","last_undegraded":"2024-03-18T16:06:50.287632+0000","last_fullsized":"2024-03-18T16:06:50.287632+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:36:57.974999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.014394+0000","last_change":"2024-03-18T16:06:49.008434+0000","last_active":"2024-03-18T16:06:50.014394+0000","last_peered":"2024-03-18T16:06:50.014394+0000","last_clean":"2024-03-18T16:06:50.014394+0000","last_became_active":"2024-03-18T16:06:49.007944+0000","last_became_peered":"2024-03-18T16:06:49.007944+0000","last_unstale":"2024-03-18T16:06:50.014394+0000","last_undegraded":"2024-03-18T16:06:50.014394+0000","last_fullsized":"2024-03-18T16:06:50.014394+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:01:24.450371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:07:20.010508+0000","last_change":"2024-03-18T16:06:43.711797+0000","last_active":"2024-03-18T16:07:20.010508+0000","last_peered":"2024-03-18T16:07:20.010508+0000","last_clean":"2024-03-18T16:07:20.010508+0000","last_became_active":"2024-03-18T16:06:43.711493+0000","last_became_peered":"2024-03-18T16:06:43.711493+0000","last_unstale":"2024-03-18T16:07:20.010508+0000","last_undegraded":"2024-03-18T16:07:20.010508+0000","last_fullsized":"2024-03-18T16:07:20.010508+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:50:29.258166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.288039+0000","last_change":"2024-03-18T16:06:44.735910+0000","last_active":"2024-03-18T16:06:50.288039+0000","last_peered":"2024-03-18T16:06:50.288039+0000","last_clean":"2024-03-18T16:06:50.288039+0000","last_became_active":"2024-03-18T16:06:44.735504+0000","last_became_peered":"2024-03-18T16:06:44.735504+0000","last_unstale":"2024-03-18T16:06:50.288039+0000","last_undegraded":"2024-03-18T16:06:50.288039+0000","last_fullsized":"2024-03-18T16:06:50.288039+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:02:18.538217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.265920+0000","last_change":"2024-03-18T16:06:49.059130+0000","last_active":"2024-03-18T16:06:50.265920+0000","last_peered":"2024-03-18T16:06:50.265920+0000","last_clean":"2024-03-18T16:06:50.265920+0000","last_became_active":"2024-03-18T16:06:49.058701+0000","last_became_peered":"2024-03-18T16:06:49.058701+0000","last_unstale":"2024-03-18T16:06:50.265920+0000","last_undegraded":"2024-03-18T16:06:50.265920+0000","last_fullsized":"2024-03-18T16:06:50.265920+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:54:52.842786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:07:17.209659+0000","last_change":"2024-03-18T16:06:43.714414+0000","last_active":"2024-03-18T16:07:17.209659+0000","last_peered":"2024-03-18T16:07:17.209659+0000","last_clean":"2024-03-18T16:07:17.209659+0000","last_became_active":"2024-03-18T16:06:43.714029+0000","last_became_peered":"2024-03-18T16:06:43.714029+0000","last_unstale":"2024-03-18T16:07:17.209659+0000","last_undegraded":"2024-03-18T16:07:17.209659+0000","last_fullsized":"2024-03-18T16:07:17.209659+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:16:39.060378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.285335+0000","last_change":"2024-03-18T16:06:44.707491+0000","last_active":"2024-03-18T16:06:50.285335+0000","last_peered":"2024-03-18T16:06:50.285335+0000","last_clean":"2024-03-18T16:06:50.285335+0000","last_became_active":"2024-03-18T16:06:44.707157+0000","last_became_peered":"2024-03-18T16:06:44.707157+0000","last_unstale":"2024-03-18T16:06:50.285335+0000","last_undegraded":"2024-03-18T16:06:50.285335+0000","last_fullsized":"2024-03-18T16:06:50.285335+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:19:08.132911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.020397+0000","last_change":"2024-03-18T16:06:49.035041+0000","last_active":"2024-03-18T16:06:50.020397+0000","last_peered":"2024-03-18T16:06:50.020397+0000","last_clean":"2024-03-18T16:06:50.020397+0000","last_became_active":"2024-03-18T16:06:49.033887+0000","last_became_peered":"2024-03-18T16:06:49.033887+0000","last_unstale":"2024-03-18T16:06:50.020397+0000","last_undegraded":"2024-03-18T16:06:50.020397+0000","last_fullsized":"2024-03-18T16:06:50.020397+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:10:36.269741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":30,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:07:20.009415+0000","last_change":"2024-03-18T16:06:43.717755+0000","last_active":"2024-03-18T16:07:20.009415+0000","last_peered":"2024-03-18T16:07:20.009415+0000","last_clean":"2024-03-18T16:07:20.009415+0000","last_became_active":"2024-03-18T16:06:43.717278+0000","last_became_peered":"2024-03-18T16:06:43.717278+0000","last_unstale":"2024-03-18T16:07:20.009415+0000","last_undegraded":"2024-03-18T16:07:20.009415+0000","last_fullsized":"2024-03-18T16:07:20.009415+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:48:09.390659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.150854+0000","last_change":"2024-03-18T16:06:44.730858+0000","last_active":"2024-03-18T16:06:50.150854+0000","last_peered":"2024-03-18T16:06:50.150854+0000","last_clean":"2024-03-18T16:06:50.150854+0000","last_became_active":"2024-03-18T16:06:44.730500+0000","last_became_peered":"2024-03-18T16:06:44.730500+0000","last_unstale":"2024-03-18T16:06:50.150854+0000","last_undegraded":"2024-03-18T16:06:50.150854+0000","last_fullsized":"2024-03-18T16:06:50.150854+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:22:03.590834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:52.881119+0000","last_change":"2024-03-18T16:06:49.014717+0000","last_active":"2024-03-18T16:06:52.881119+0000","last_peered":"2024-03-18T16:06:52.881119+0000","last_clean":"2024-03-18T16:06:52.881119+0000","last_became_active":"2024-03-18T16:06:49.014313+0000","last_became_peered":"2024-03-18T16:06:49.014313+0000","last_unstale":"2024-03-18T16:06:52.881119+0000","last_undegraded":"2024-03-18T16:06:52.881119+0000","last_fullsized":"2024-03-18T16:06:52.881119+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:06:12.168888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.014435+0000","last_change":"2024-03-18T16:06:43.719496+0000","last_active":"2024-03-18T16:06:50.014435+0000","last_peered":"2024-03-18T16:06:50.014435+0000","last_clean":"2024-03-18T16:06:50.014435+0000","last_became_active":"2024-03-18T16:06:43.719139+0000","last_became_peered":"2024-03-18T16:06:43.719139+0000","last_unstale":"2024-03-18T16:06:50.014435+0000","last_undegraded":"2024-03-18T16:06:50.014435+0000","last_fullsized":"2024-03-18T16:06:50.014435+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:10:01.250923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.287622+0000","last_change":"2024-03-18T16:06:44.737792+0000","last_active":"2024-03-18T16:06:50.287622+0000","last_peered":"2024-03-18T16:06:50.287622+0000","last_clean":"2024-03-18T16:06:50.287622+0000","last_became_active":"2024-03-18T16:06:44.736311+0000","last_became_peered":"2024-03-18T16:06:44.736311+0000","last_unstale":"2024-03-18T16:06:50.287622+0000","last_undegraded":"2024-03-18T16:06:50.287622+0000","last_fullsized":"2024-03-18T16:06:50.287622+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:43:56.110205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.265955+0000","last_change":"2024-03-18T16:06:49.058893+0000","last_active":"2024-03-18T16:06:50.265955+0000","last_peered":"2024-03-18T16:06:50.265955+0000","last_clean":"2024-03-18T16:06:50.265955+0000","last_became_active":"2024-03-18T16:06:49.058477+0000","last_became_peered":"2024-03-18T16:06:49.058477+0000","last_unstale":"2024-03-18T16:06:50.265955+0000","last_undegraded":"2024-03-18T16:06:50.265955+0000","last_fullsized":"2024-03-18T16:06:50.265955+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:50:51.890348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:07:17.205076+0000","last_change":"2024-03-18T16:06:43.703073+0000","last_active":"2024-03-18T16:07:17.205076+0000","last_peered":"2024-03-18T16:07:17.205076+0000","last_clean":"2024-03-18T16:07:17.205076+0000","last_became_active":"2024-03-18T16:06:43.702708+0000","last_became_peered":"2024-03-18T16:06:43.702708+0000","last_unstale":"2024-03-18T16:07:17.205076+0000","last_undegraded":"2024-03-18T16:07:17.205076+0000","last_fullsized":"2024-03-18T16:07:17.205076+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:38:41.454458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.043293+0000","last_change":"2024-03-18T16:06:44.713622+0000","last_active":"2024-03-18T16:06:50.043293+0000","last_peered":"2024-03-18T16:06:50.043293+0000","last_clean":"2024-03-18T16:06:50.043293+0000","last_became_active":"2024-03-18T16:06:44.713160+0000","last_became_peered":"2024-03-18T16:06:44.713160+0000","last_unstale":"2024-03-18T16:06:50.043293+0000","last_undegraded":"2024-03-18T16:06:50.043293+0000","last_fullsized":"2024-03-18T16:06:50.043293+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:49:21.886543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.285320+0000","last_change":"2024-03-18T16:06:49.032597+0000","last_active":"2024-03-18T16:06:50.285320+0000","last_peered":"2024-03-18T16:06:50.285320+0000","last_clean":"2024-03-18T16:06:50.285320+0000","last_became_active":"2024-03-18T16:06:49.032250+0000","last_became_peered":"2024-03-18T16:06:49.032250+0000","last_unstale":"2024-03-18T16:06:50.285320+0000","last_undegraded":"2024-03-18T16:06:50.285320+0000","last_fullsized":"2024-03-18T16:06:50.285320+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:16:45.761978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:07:18.230261+0000","last_change":"2024-03-18T16:06:43.717161+0000","last_active":"2024-03-18T16:07:18.230261+0000","last_peered":"2024-03-18T16:07:18.230261+0000","last_clean":"2024-03-18T16:07:18.230261+0000","last_became_active":"2024-03-18T16:06:43.716657+0000","last_became_peered":"2024-03-18T16:06:43.716657+0000","last_unstale":"2024-03-18T16:07:18.230261+0000","last_undegraded":"2024-03-18T16:07:18.230261+0000","last_fullsized":"2024-03-18T16:07:18.230261+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:23:48.194779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.043700+0000","last_change":"2024-03-18T16:06:44.709777+0000","last_active":"2024-03-18T16:06:50.043700+0000","last_peered":"2024-03-18T16:06:50.043700+0000","last_clean":"2024-03-18T16:06:50.043700+0000","last_became_active":"2024-03-18T16:06:44.709409+0000","last_became_peered":"2024-03-18T16:06:44.709409+0000","last_unstale":"2024-03-18T16:06:50.043700+0000","last_undegraded":"2024-03-18T16:06:50.043700+0000","last_fullsized":"2024-03-18T16:06:50.043700+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:37:12.288262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.286126+0000","last_change":"2024-03-18T16:06:49.040872+0000","last_active":"2024-03-18T16:06:50.286126+0000","last_peered":"2024-03-18T16:06:50.286126+0000","last_clean":"2024-03-18T16:06:50.286126+0000","last_became_active":"2024-03-18T16:06:49.039807+0000","last_became_peered":"2024-03-18T16:06:49.039807+0000","last_unstale":"2024-03-18T16:06:50.286126+0000","last_undegraded":"2024-03-18T16:06:50.286126+0000","last_fullsized":"2024-03-18T16:06:50.286126+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:09:27.045450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:07:19.977630+0000","last_change":"2024-03-18T16:06:43.723675+0000","last_active":"2024-03-18T16:07:19.977630+0000","last_peered":"2024-03-18T16:07:19.977630+0000","last_clean":"2024-03-18T16:07:19.977630+0000","last_became_active":"2024-03-18T16:06:43.723334+0000","last_became_peered":"2024-03-18T16:06:43.723334+0000","last_unstale":"2024-03-18T16:07:19.977630+0000","last_undegraded":"2024-03-18T16:07:19.977630+0000","last_fullsized":"2024-03-18T16:07:19.977630+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:01:36.553362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.043434+0000","last_change":"2024-03-18T16:06:44.723045+0000","last_active":"2024-03-18T16:06:50.043434+0000","last_peered":"2024-03-18T16:06:50.043434+0000","last_clean":"2024-03-18T16:06:50.043434+0000","last_became_active":"2024-03-18T16:06:44.722688+0000","last_became_peered":"2024-03-18T16:06:44.722688+0000","last_unstale":"2024-03-18T16:06:50.043434+0000","last_undegraded":"2024-03-18T16:06:50.043434+0000","last_fullsized":"2024-03-18T16:06:50.043434+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:46:21.720007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.285387+0000","last_change":"2024-03-18T16:06:49.011981+0000","last_active":"2024-03-18T16:06:50.285387+0000","last_peered":"2024-03-18T16:06:50.285387+0000","last_clean":"2024-03-18T16:06:50.285387+0000","last_became_active":"2024-03-18T16:06:49.011621+0000","last_became_peered":"2024-03-18T16:06:49.011621+0000","last_unstale":"2024-03-18T16:06:50.285387+0000","last_undegraded":"2024-03-18T16:06:50.285387+0000","last_fullsized":"2024-03-18T16:06:50.285387+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:11:42.382721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:07:20.010635+0000","last_change":"2024-03-18T16:06:43.725404+0000","last_active":"2024-03-18T16:07:20.010635+0000","last_peered":"2024-03-18T16:07:20.010635+0000","last_clean":"2024-03-18T16:07:20.010635+0000","last_became_active":"2024-03-18T16:06:43.724117+0000","last_became_peered":"2024-03-18T16:06:43.724117+0000","last_unstale":"2024-03-18T16:07:20.010635+0000","last_undegraded":"2024-03-18T16:07:20.010635+0000","last_fullsized":"2024-03-18T16:07:20.010635+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:58:43.243070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.043631+0000","last_change":"2024-03-18T16:06:44.709979+0000","last_active":"2024-03-18T16:06:50.043631+0000","last_peered":"2024-03-18T16:06:50.043631+0000","last_clean":"2024-03-18T16:06:50.043631+0000","last_became_active":"2024-03-18T16:06:44.709591+0000","last_became_peered":"2024-03-18T16:06:44.709591+0000","last_unstale":"2024-03-18T16:06:50.043631+0000","last_undegraded":"2024-03-18T16:06:50.043631+0000","last_fullsized":"2024-03-18T16:06:50.043631+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:56:37.537248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.150979+0000","last_change":"2024-03-18T16:06:44.706641+0000","last_active":"2024-03-18T16:06:50.150979+0000","last_peered":"2024-03-18T16:06:50.150979+0000","last_clean":"2024-03-18T16:06:50.150979+0000","last_became_active":"2024-03-18T16:06:44.706267+0000","last_became_peered":"2024-03-18T16:06:44.706267+0000","last_unstale":"2024-03-18T16:06:50.150979+0000","last_undegraded":"2024-03-18T16:06:50.150979+0000","last_fullsized":"2024-03-18T16:06:50.150979+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:10:30.072914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.014840+0000","last_change":"2024-03-18T16:06:44.714836+0000","last_active":"2024-03-18T16:06:50.014840+0000","last_peered":"2024-03-18T16:06:50.014840+0000","last_clean":"2024-03-18T16:06:50.014840+0000","last_became_active":"2024-03-18T16:06:44.714454+0000","last_became_peered":"2024-03-18T16:06:44.714454+0000","last_unstale":"2024-03-18T16:06:50.014840+0000","last_undegraded":"2024-03-18T16:06:50.014840+0000","last_fullsized":"2024-03-18T16:06:50.014840+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:05:24.532238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.014510+0000","last_change":"2024-03-18T16:06:44.733183+0000","last_active":"2024-03-18T16:06:50.014510+0000","last_peered":"2024-03-18T16:06:50.014510+0000","last_clean":"2024-03-18T16:06:50.014510+0000","last_became_active":"2024-03-18T16:06:44.732909+0000","last_became_peered":"2024-03-18T16:06:44.732909+0000","last_unstale":"2024-03-18T16:06:50.014510+0000","last_undegraded":"2024-03-18T16:06:50.014510+0000","last_fullsized":"2024-03-18T16:06:50.014510+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:28:07.636692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.286380+0000","last_change":"2024-03-18T16:06:44.731050+0000","last_active":"2024-03-18T16:06:50.286380+0000","last_peered":"2024-03-18T16:06:50.286380+0000","last_clean":"2024-03-18T16:06:50.286380+0000","last_became_active":"2024-03-18T16:06:44.730634+0000","last_became_peered":"2024-03-18T16:06:44.730634+0000","last_unstale":"2024-03-18T16:06:50.286380+0000","last_undegraded":"2024-03-18T16:06:50.286380+0000","last_fullsized":"2024-03-18T16:06:50.286380+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:07:18.619130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.288532+0000","last_change":"2024-03-18T16:06:44.735190+0000","last_active":"2024-03-18T16:06:50.288532+0000","last_peered":"2024-03-18T16:06:50.288532+0000","last_clean":"2024-03-18T16:06:50.288532+0000","last_became_active":"2024-03-18T16:06:44.734312+0000","last_became_peered":"2024-03-18T16:06:44.734312+0000","last_unstale":"2024-03-18T16:06:50.288532+0000","last_undegraded":"2024-03-18T16:06:50.288532+0000","last_fullsized":"2024-03-18T16:06:50.288532+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:59:50.209515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.151329+0000","last_change":"2024-03-18T16:06:44.705385+0000","last_active":"2024-03-18T16:06:50.151329+0000","last_peered":"2024-03-18T16:06:50.151329+0000","last_clean":"2024-03-18T16:06:50.151329+0000","last_became_active":"2024-03-18T16:06:44.705011+0000","last_became_peered":"2024-03-18T16:06:44.705011+0000","last_unstale":"2024-03-18T16:06:50.151329+0000","last_undegraded":"2024-03-18T16:06:50.151329+0000","last_fullsized":"2024-03-18T16:06:50.151329+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:50:02.152851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.015132+0000","last_change":"2024-03-18T16:06:44.709621+0000","last_active":"2024-03-18T16:06:50.015132+0000","last_peered":"2024-03-18T16:06:50.015132+0000","last_clean":"2024-03-18T16:06:50.015132+0000","last_became_active":"2024-03-18T16:06:44.709275+0000","last_became_peered":"2024-03-18T16:06:44.709275+0000","last_unstale":"2024-03-18T16:06:50.015132+0000","last_undegraded":"2024-03-18T16:06:50.015132+0000","last_fullsized":"2024-03-18T16:06:50.015132+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:41:17.346661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.287832+0000","last_change":"2024-03-18T16:06:44.719827+0000","last_active":"2024-03-18T16:06:50.287832+0000","last_peered":"2024-03-18T16:06:50.287832+0000","last_clean":"2024-03-18T16:06:50.287832+0000","last_became_active":"2024-03-18T16:06:44.719432+0000","last_became_peered":"2024-03-18T16:06:44.719432+0000","last_unstale":"2024-03-18T16:06:50.287832+0000","last_undegraded":"2024-03-18T16:06:50.287832+0000","last_fullsized":"2024-03-18T16:06:50.287832+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:13:42.344211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.285992+0000","last_change":"2024-03-18T16:06:44.714526+0000","last_active":"2024-03-18T16:06:50.285992+0000","last_peered":"2024-03-18T16:06:50.285992+0000","last_clean":"2024-03-18T16:06:50.285992+0000","last_became_active":"2024-03-18T16:06:44.714184+0000","last_became_peered":"2024-03-18T16:06:44.714184+0000","last_unstale":"2024-03-18T16:06:50.285992+0000","last_undegraded":"2024-03-18T16:06:50.285992+0000","last_fullsized":"2024-03-18T16:06:50.285992+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:31:07.917143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.287038+0000","last_change":"2024-03-18T16:06:44.730905+0000","last_active":"2024-03-18T16:06:50.287038+0000","last_peered":"2024-03-18T16:06:50.287038+0000","last_clean":"2024-03-18T16:06:50.287038+0000","last_became_active":"2024-03-18T16:06:44.729002+0000","last_became_peered":"2024-03-18T16:06:44.729002+0000","last_unstale":"2024-03-18T16:06:50.287038+0000","last_undegraded":"2024-03-18T16:06:50.287038+0000","last_fullsized":"2024-03-18T16:06:50.287038+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:49:18.867571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.285382+0000","last_change":"2024-03-18T16:06:44.727608+0000","last_active":"2024-03-18T16:06:50.285382+0000","last_peered":"2024-03-18T16:06:50.285382+0000","last_clean":"2024-03-18T16:06:50.285382+0000","last_became_active":"2024-03-18T16:06:44.727096+0000","last_became_peered":"2024-03-18T16:06:44.727096+0000","last_unstale":"2024-03-18T16:06:50.285382+0000","last_undegraded":"2024-03-18T16:06:50.285382+0000","last_fullsized":"2024-03-18T16:06:50.285382+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:08:20.767443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.286521+0000","last_change":"2024-03-18T16:06:44.731823+0000","last_active":"2024-03-18T16:06:50.286521+0000","last_peered":"2024-03-18T16:06:50.286521+0000","last_clean":"2024-03-18T16:06:50.286521+0000","last_became_active":"2024-03-18T16:06:44.731429+0000","last_became_peered":"2024-03-18T16:06:44.731429+0000","last_unstale":"2024-03-18T16:06:50.286521+0000","last_undegraded":"2024-03-18T16:06:50.286521+0000","last_fullsized":"2024-03-18T16:06:50.286521+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:36:28.752466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-18T16:06:50.151152+0000","last_change":"2024-03-18T16:06:44.714354+0000","last_active":"2024-03-18T16:06:50.151152+0000","last_peered":"2024-03-18T16:06:50.151152+0000","last_clean":"2024-03-18T16:06:50.151152+0000","last_became_active":"2024-03-18T16:06:44.713969+0000","last_became_peered":"2024-03-18T16:06:44.713969+0000","last_unstale":"2024-03-18T16:06:50.151152+0000","last_undegraded":"2024-03-18T16:06:50.151152+0000","last_fullsized":"2024-03-18T16:06:50.151152+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:18:49.502178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":10033,"num_objects":92,"num_object_clones":0,"num_object_copies":276,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":92,"num_whiteouts":0,"num_read":38,"num_read_kb":30,"num_write":204,"num_write_kb":68,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":62,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":50217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":176,"ondisk_log_size":176,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383702,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":995328,"data_stored":478407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645337,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36920,"kb_used_data":1584,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708232,"statfs":{"total":95995035648,"available":95957229568,"internally_reserved":0,"allocated":1622016,"data_stored":1073003,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808863,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1028096,"data_stored":486529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070503,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1019904,"data_stored":482811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332139,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28596,"kb_used_data":1580,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716556,"statfs":{"total":95995035648,"available":95965753344,"internally_reserved":0,"allocated":1617920,"data_stored":1079308,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495664,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32152,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713000,"statfs":{"total":95995035648,"available":95962112000,"internally_reserved":0,"allocated":999424,"data_stored":477937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574924,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36312,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708840,"statfs":{"total":95995035648,"available":95957852160,"internally_reserved":0,"allocated":999424,"data_stored":481639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738451,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1024000,"data_stored":479393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411400,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36320,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708832,"statfs":{"total":95995035648,"available":95957843968,"internally_reserved":0,"allocated":1007616,"data_stored":481713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182466,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1007616,"data_stored":477623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920829,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":1007616,"data_stored":482011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757304,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36924,"kb_used_data":1588,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708228,"statfs":{"total":95995035648,"available":95957225472,"internally_reserved":0,"allocated":1626112,"data_stored":1074890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":24893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":5192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":5544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T16:07:24.244 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-18T16:07:24.244 INFO:tasks.cephadm:Setting up client nodes... 2024-03-18T16:07:24.244 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-18T16:07:24.742 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T16:07:25.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:25 smithi039 ceph-mon[31721]: pgmap v338: 129 pgs: 129 active+clean; 587 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4.2 KiB/s wr, 16 op/s 2024-03-18T16:07:25.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:25 smithi039 ceph-mon[31721]: from='client.24883 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T16:07:25.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:25 smithi039 ceph-mon[31721]: from='client.24869 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T16:07:25.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:25 smithi196 ceph-mon[36307]: pgmap v338: 129 pgs: 129 active+clean; 587 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4.2 KiB/s wr, 16 op/s 2024-03-18T16:07:25.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:25 smithi196 ceph-mon[36307]: from='client.24883 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T16:07:25.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:25 smithi196 ceph-mon[36307]: from='client.24869 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T16:07:25.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:25 smithi194 ceph-mon[36200]: pgmap v338: 129 pgs: 129 active+clean; 587 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4.2 KiB/s wr, 16 op/s 2024-03-18T16:07:25.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:25 smithi194 ceph-mon[36200]: from='client.24883 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T16:07:25.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:25 smithi194 ceph-mon[36200]: from='client.24869 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T16:07:27.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:27.111+0000 7fe257200700 1 -- 172.21.15.39:0/2769357960 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe258101060 msgr2=0x7fe258101480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:27.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:27.111+0000 7fe257200700 1 --2- 172.21.15.39:0/2769357960 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe258101060 0x7fe258101480 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fe248009920 tx=0x7fe248009c30 comp rx=0 tx=0).stop 2024-03-18T16:07:27.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:27.111+0000 7fe257200700 1 -- 172.21.15.39:0/2769357960 shutdown_connections 2024-03-18T16:07:27.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:27.111+0000 7fe257200700 1 --2- 172.21.15.39:0/2769357960 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe258102c00 0x7fe2581094b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:27.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:27.111+0000 7fe257200700 1 --2- 172.21.15.39:0/2769357960 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe258101060 0x7fe258101480 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:27.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:27.111+0000 7fe257200700 1 --2- 172.21.15.39:0/2769357960 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe258102260 0x7fe2581026c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:27.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:27.111+0000 7fe257200700 1 -- 172.21.15.39:0/2769357960 >> 172.21.15.39:0/2769357960 conn(0x7fe2580fc600 msgr2=0x7fe2580fea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:27.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:27.111+0000 7fe257200700 1 -- 172.21.15.39:0/2769357960 shutdown_connections 2024-03-18T16:07:27.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:27.111+0000 7fe257200700 1 -- 172.21.15.39:0/2769357960 wait complete. 2024-03-18T16:07:27.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:27.112+0000 7fe257200700 1 Processor -- start 2024-03-18T16:07:27.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:27.113+0000 7fe257200700 1 -- start start 2024-03-18T16:07:27.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:27.113+0000 7fe257200700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe258101060 0x7fe25819a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:27.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:27.113+0000 7fe257200700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe258102260 0x7fe25819aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:27.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:27.113+0000 7fe257200700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe258102c00 0x7fe2581a1b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:27.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:27.113+0000 7fe257200700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fe25810cc70 con 0x7fe258101060 2024-03-18T16:07:27.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:27.113+0000 7fe257200700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fe25810caf0 con 0x7fe258102c00 2024-03-18T16:07:27.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:27.113+0000 7fe257200700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fe25810cdf0 con 0x7fe258102260 2024-03-18T16:07:27.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:27.113+0000 7fe255400700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe258102260 0x7fe25819aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:27.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:27.113+0000 7fe255400700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe258102260 0x7fe25819aab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:32770/0 (socket says 172.21.15.39:32770) 2024-03-18T16:07:27.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:27.114+0000 7fe255400700 1 -- 172.21.15.39:0/1276329770 learned_addr learned my addr 172.21.15.39:0/1276329770 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:07:27.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:27.114+0000 7fe255e00700 1 --2- 172.21.15.39:0/1276329770 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe258101060 0x7fe25819a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:27.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:27.114+0000 7fe256800700 1 --2- 172.21.15.39:0/1276329770 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe258102c00 0x7fe2581a1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:27.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:27.114+0000 7fe255400700 1 -- 172.21.15.39:0/1276329770 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe258102c00 msgr2=0x7fe2581a1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:27.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:27.114+0000 7fe255400700 1 --2- 172.21.15.39:0/1276329770 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe258102c00 0x7fe2581a1b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:27.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:27.114+0000 7fe255400700 1 -- 172.21.15.39:0/1276329770 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe258101060 msgr2=0x7fe25819a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:27.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:27.114+0000 7fe255400700 1 --2- 172.21.15.39:0/1276329770 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe258101060 0x7fe25819a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:27.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:27.114+0000 7fe255400700 1 -- 172.21.15.39:0/1276329770 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2581a21d0 con 0x7fe258102260 2024-03-18T16:07:27.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:27.114+0000 7fe255400700 1 --2- 172.21.15.39:0/1276329770 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe258102260 0x7fe25819aab0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe24400ec40 tx=0x7fe24400ef50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:27.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:27.115+0000 7fe24ea00700 1 -- 172.21.15.39:0/1276329770 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe24400ccf0 con 0x7fe258102260 2024-03-18T16:07:27.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:27.115+0000 7fe24ea00700 1 -- 172.21.15.39:0/1276329770 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe24400ce90 con 0x7fe258102260 2024-03-18T16:07:27.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:27.115+0000 7fe257200700 1 -- 172.21.15.39:0/1276329770 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2581a24c0 con 0x7fe258102260 2024-03-18T16:07:27.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:27.115+0000 7fe24ea00700 1 -- 172.21.15.39:0/1276329770 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe244010690 con 0x7fe258102260 2024-03-18T16:07:27.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:27.115+0000 7fe257200700 1 -- 172.21.15.39:0/1276329770 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2581a2a00 con 0x7fe258102260 2024-03-18T16:07:27.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:27.116+0000 7fe257200700 1 -- 172.21.15.39:0/1276329770 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe258063900 con 0x7fe258102260 2024-03-18T16:07:27.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:27.118+0000 7fe24ea00700 1 -- 172.21.15.39:0/1276329770 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fe244003e10 con 0x7fe258102260 2024-03-18T16:07:27.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:27.119+0000 7fe24ea00700 1 --2- 172.21.15.39:0/1276329770 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fe23c0a29d0 0x7fe23c0a4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:27.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:27.119+0000 7fe24ea00700 1 -- 172.21.15.39:0/1276329770 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fe2440cf200 con 0x7fe258102260 2024-03-18T16:07:27.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:27.119+0000 7fe255e00700 1 --2- 172.21.15.39:0/1276329770 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fe23c0a29d0 0x7fe23c0a4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:27.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:27.120+0000 7fe255e00700 1 --2- 172.21.15.39:0/1276329770 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fe23c0a29d0 0x7fe23c0a4e90 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fe24800fb70 tx=0x7fe248019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:27.124 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:27.123+0000 7fe24ea00700 1 -- 172.21.15.39:0/1276329770 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe24409bea0 con 0x7fe258102260 2024-03-18T16:07:27.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:27.476+0000 7fe257200700 1 -- 172.21.15.39:0/1276329770 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7fe2581a2f20 con 0x7fe258102260 2024-03-18T16:07:27.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:27.483+0000 7fe24ea00700 1 -- 172.21.15.39:0/1276329770 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7fe24409b7a0 con 0x7fe258102260 2024-03-18T16:07:27.484 INFO:teuthology.orchestra.run.smithi039.stdout:[client.0] 2024-03-18T16:07:27.484 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQC/ZvhlPh15HBAArv5dMWbjUFDLkVLy0nHNbg== 2024-03-18T16:07:27.487 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:27.486+0000 7fe257200700 1 -- 172.21.15.39:0/1276329770 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fe23c0a29d0 msgr2=0x7fe23c0a4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:27.487 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:27.486+0000 7fe257200700 1 --2- 172.21.15.39:0/1276329770 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fe23c0a29d0 0x7fe23c0a4e90 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fe24800fb70 tx=0x7fe248019040 comp rx=0 tx=0).stop 2024-03-18T16:07:27.487 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:27.486+0000 7fe257200700 1 -- 172.21.15.39:0/1276329770 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe258102260 msgr2=0x7fe25819aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:27.487 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:27.486+0000 7fe257200700 1 --2- 172.21.15.39:0/1276329770 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe258102260 0x7fe25819aab0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe24400ec40 tx=0x7fe24400ef50 comp rx=0 tx=0).stop 2024-03-18T16:07:27.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:27.486+0000 7fe257200700 1 -- 172.21.15.39:0/1276329770 shutdown_connections 2024-03-18T16:07:27.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:27.486+0000 7fe257200700 1 --2- 172.21.15.39:0/1276329770 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fe23c0a29d0 0x7fe23c0a4e90 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:27.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:27.486+0000 7fe257200700 1 --2- 172.21.15.39:0/1276329770 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe258102c00 0x7fe2581a1b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:27.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:27.486+0000 7fe257200700 1 --2- 172.21.15.39:0/1276329770 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe258101060 0x7fe25819a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:27.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:27.486+0000 7fe257200700 1 --2- 172.21.15.39:0/1276329770 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe258102260 0x7fe25819aab0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:27.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:27.486+0000 7fe257200700 1 -- 172.21.15.39:0/1276329770 >> 172.21.15.39:0/1276329770 conn(0x7fe2580fc600 msgr2=0x7fe2580fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:27.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:27.486+0000 7fe257200700 1 -- 172.21.15.39:0/1276329770 shutdown_connections 2024-03-18T16:07:27.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:27.486+0000 7fe257200700 1 -- 172.21.15.39:0/1276329770 wait complete. 2024-03-18T16:07:27.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:27 smithi039 ceph-mon[31721]: pgmap v339: 129 pgs: 129 active+clean; 629 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 8.4 KiB/s wr, 20 op/s 2024-03-18T16:07:27.727 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:27 smithi196 ceph-mon[36307]: pgmap v339: 129 pgs: 129 active+clean; 629 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 8.4 KiB/s wr, 20 op/s 2024-03-18T16:07:27.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:27 smithi194 ceph-mon[36200]: pgmap v339: 129 pgs: 129 active+clean; 629 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 8.4 KiB/s wr, 20 op/s 2024-03-18T16:07:28.304 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-03-18T16:07:28.304 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-18T16:07:28.304 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-18T16:07:28.458 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-18T16:07:28.458 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-18T16:07:28.459 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph mgr dump --format=json 2024-03-18T16:07:28.582 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:28 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/1276329770' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-18T16:07:28.583 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:28 smithi039 ceph-mon[31721]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-18T16:07:28.583 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:28 smithi039 ceph-mon[31721]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-18T16:07:28.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:28 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/1276329770' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-18T16:07:28.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:28 smithi196 ceph-mon[36307]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-18T16:07:28.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:28 smithi196 ceph-mon[36307]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-18T16:07:28.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:28 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/1276329770' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-18T16:07:28.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:28 smithi194 ceph-mon[36200]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-18T16:07:28.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:28 smithi194 ceph-mon[36200]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-18T16:07:28.968 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T16:07:29.469 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:29 smithi039 ceph-mon[31721]: pgmap v340: 129 pgs: 129 active+clean; 629 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 8.5 KiB/s wr, 23 op/s 2024-03-18T16:07:29.629 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:29 smithi196 ceph-mon[36307]: pgmap v340: 129 pgs: 129 active+clean; 629 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 8.5 KiB/s wr, 23 op/s 2024-03-18T16:07:29.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:29 smithi194 ceph-mon[36200]: pgmap v340: 129 pgs: 129 active+clean; 629 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 8.5 KiB/s wr, 23 op/s 2024-03-18T16:07:31.450 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:31 smithi039 ceph-mon[31721]: pgmap v341: 129 pgs: 129 active+clean; 637 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.4 KiB/s wr, 25 op/s 2024-03-18T16:07:31.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:31.511+0000 7f0f36a00700 1 -- 172.21.15.39:0/1700656113 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f30104500 msgr2=0x7f0f30104980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:31.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:31.511+0000 7f0f36a00700 1 --2- 172.21.15.39:0/1700656113 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f30104500 0x7f0f30104980 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f0f1c0099c0 tx=0x7f0f1c009cd0 comp rx=0 tx=0).stop 2024-03-18T16:07:31.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:31.511+0000 7f0f36a00700 1 -- 172.21.15.39:0/1700656113 shutdown_connections 2024-03-18T16:07:31.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:31.511+0000 7f0f36a00700 1 --2- 172.21.15.39:0/1700656113 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0f30104ec0 0x7f0f30111380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:31.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:31.511+0000 7f0f36a00700 1 --2- 172.21.15.39:0/1700656113 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0f3010a5b0 0x7f0f3010a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:31.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:31.511+0000 7f0f36a00700 1 --2- 172.21.15.39:0/1700656113 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f30104500 0x7f0f30104980 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:31.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:31.511+0000 7f0f36a00700 1 -- 172.21.15.39:0/1700656113 >> 172.21.15.39:0/1700656113 conn(0x7f0f300fffd0 msgr2=0x7f0f301023f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:31.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:31.511+0000 7f0f36a00700 1 -- 172.21.15.39:0/1700656113 shutdown_connections 2024-03-18T16:07:31.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:31.511+0000 7f0f36a00700 1 -- 172.21.15.39:0/1700656113 wait complete. 2024-03-18T16:07:31.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:31.512+0000 7f0f36a00700 1 Processor -- start 2024-03-18T16:07:31.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:31.512+0000 7f0f36a00700 1 -- start start 2024-03-18T16:07:31.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:31.513+0000 7f0f36a00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f30104500 0x7f0f301a0860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:31.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:31.513+0000 7f0f36a00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0f30104ec0 0x7f0f301a0da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:31.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:31.513+0000 7f0f36a00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0f3010a5b0 0x7f0f3019aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:31.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:31.513+0000 7f0f36a00700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f0f30114850 con 0x7f0f30104500 2024-03-18T16:07:31.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:31.513+0000 7f0f36a00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f0f301146d0 con 0x7f0f30104ec0 2024-03-18T16:07:31.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:31.513+0000 7f0f36a00700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f0f301149d0 con 0x7f0f3010a5b0 2024-03-18T16:07:31.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:31.513+0000 7f0f2fe00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f30104500 0x7f0f301a0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:31.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:31.513+0000 7f0f2fe00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f30104500 0x7f0f301a0860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:59808/0 (socket says 172.21.15.39:59808) 2024-03-18T16:07:31.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:31.513+0000 7f0f34a00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0f3010a5b0 0x7f0f3019aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:31.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:31.513+0000 7f0f2fe00700 1 -- 172.21.15.39:0/1444581979 learned_addr learned my addr 172.21.15.39:0/1444581979 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:07:31.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:31.513+0000 7f0f2f400700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0f30104ec0 0x7f0f301a0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:31.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:31.514+0000 7f0f34a00700 1 -- 172.21.15.39:0/1444581979 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0f30104ec0 msgr2=0x7f0f301a0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:31.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:31.514+0000 7f0f34a00700 1 --2- 172.21.15.39:0/1444581979 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0f30104ec0 0x7f0f301a0da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:31.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:31.514+0000 7f0f34a00700 1 -- 172.21.15.39:0/1444581979 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f30104500 msgr2=0x7f0f301a0860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:31.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:31.514+0000 7f0f34a00700 1 --2- 172.21.15.39:0/1444581979 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f30104500 0x7f0f301a0860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:31.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:31.514+0000 7f0f34a00700 1 -- 172.21.15.39:0/1444581979 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f3019b1d0 con 0x7f0f3010a5b0 2024-03-18T16:07:31.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:31.514+0000 7f0f2f400700 1 --2- 172.21.15.39:0/1444581979 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0f30104ec0 0x7f0f301a0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:07:31.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:31.514+0000 7f0f2fe00700 1 --2- 172.21.15.39:0/1444581979 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f30104500 0x7f0f301a0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:07:31.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:31.514+0000 7f0f34a00700 1 --2- 172.21.15.39:0/1444581979 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0f3010a5b0 0x7f0f3019aa70 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f0f2800efc0 tx=0x7f0f2800c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:31.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:31.514+0000 7f0f2cc00700 1 -- 172.21.15.39:0/1444581979 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f280092e0 con 0x7f0f3010a5b0 2024-03-18T16:07:31.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:31.514+0000 7f0f2cc00700 1 -- 172.21.15.39:0/1444581979 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f28004bb0 con 0x7f0f3010a5b0 2024-03-18T16:07:31.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:31.514+0000 7f0f36a00700 1 -- 172.21.15.39:0/1444581979 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f3019b460 con 0x7f0f3010a5b0 2024-03-18T16:07:31.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:31.515+0000 7f0f2cc00700 1 -- 172.21.15.39:0/1444581979 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f28004280 con 0x7f0f3010a5b0 2024-03-18T16:07:31.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:31.515+0000 7f0f36a00700 1 -- 172.21.15.39:0/1444581979 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f301a7730 con 0x7f0f3010a5b0 2024-03-18T16:07:31.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:31.516+0000 7f0f36a00700 1 -- 172.21.15.39:0/1444581979 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f30068f90 con 0x7f0f3010a5b0 2024-03-18T16:07:31.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:31.518+0000 7f0f2cc00700 1 -- 172.21.15.39:0/1444581979 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f0f2801b030 con 0x7f0f3010a5b0 2024-03-18T16:07:31.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:31.519+0000 7f0f2cc00700 1 --2- 172.21.15.39:0/1444581979 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f0f140a29a0 0x7f0f140a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:31.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:31.519+0000 7f0f2cc00700 1 -- 172.21.15.39:0/1444581979 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f0f280cea40 con 0x7f0f3010a5b0 2024-03-18T16:07:31.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:31.519+0000 7f0f2fe00700 1 --2- 172.21.15.39:0/1444581979 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f0f140a29a0 0x7f0f140a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:31.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:31.520+0000 7f0f2fe00700 1 --2- 172.21.15.39:0/1444581979 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f0f140a29a0 0x7f0f140a4e60 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f0f301097e0 tx=0x7f0f20009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:31.522 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:31.521+0000 7f0f2cc00700 1 -- 172.21.15.39:0/1444581979 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0f28022020 con 0x7f0f3010a5b0 2024-03-18T16:07:31.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:31 smithi196 ceph-mon[36307]: pgmap v341: 129 pgs: 129 active+clean; 637 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.4 KiB/s wr, 25 op/s 2024-03-18T16:07:31.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:31 smithi194 ceph-mon[36200]: pgmap v341: 129 pgs: 129 active+clean; 637 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.4 KiB/s wr, 25 op/s 2024-03-18T16:07:31.825 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:31.823+0000 7f0f36a00700 1 -- 172.21.15.39:0/1444581979 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f0f3002db70 con 0x7f0f3010a5b0 2024-03-18T16:07:31.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:31.832+0000 7f0f2cc00700 1 -- 172.21.15.39:0/1444581979 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+262660 (secure 0 0 0) 0x7f0f28010040 con 0x7f0f3010a5b0 2024-03-18T16:07:31.835 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:07:31.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:31.838+0000 7f0f36a00700 1 -- 172.21.15.39:0/1444581979 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f0f140a29a0 msgr2=0x7f0f140a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:31.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:31.838+0000 7f0f36a00700 1 --2- 172.21.15.39:0/1444581979 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f0f140a29a0 0x7f0f140a4e60 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f0f301097e0 tx=0x7f0f20009260 comp rx=0 tx=0).stop 2024-03-18T16:07:31.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:31.838+0000 7f0f36a00700 1 -- 172.21.15.39:0/1444581979 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0f3010a5b0 msgr2=0x7f0f3019aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:31.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:31.838+0000 7f0f36a00700 1 --2- 172.21.15.39:0/1444581979 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0f3010a5b0 0x7f0f3019aa70 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f0f2800efc0 tx=0x7f0f2800c3a0 comp rx=0 tx=0).stop 2024-03-18T16:07:31.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:31.838+0000 7f0f36a00700 1 -- 172.21.15.39:0/1444581979 shutdown_connections 2024-03-18T16:07:31.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:31.839+0000 7f0f36a00700 1 --2- 172.21.15.39:0/1444581979 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f0f140a29a0 0x7f0f140a4e60 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:31.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:31.839+0000 7f0f36a00700 1 --2- 172.21.15.39:0/1444581979 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0f3010a5b0 0x7f0f3019aa70 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:31.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:31.839+0000 7f0f36a00700 1 --2- 172.21.15.39:0/1444581979 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0f30104500 0x7f0f301a0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:31.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:31.839+0000 7f0f36a00700 1 --2- 172.21.15.39:0/1444581979 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0f30104ec0 0x7f0f301a0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:31.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:31.839+0000 7f0f36a00700 1 -- 172.21.15.39:0/1444581979 >> 172.21.15.39:0/1444581979 conn(0x7f0f300fffd0 msgr2=0x7f0f301015c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:31.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:31.839+0000 7f0f36a00700 1 -- 172.21.15.39:0/1444581979 shutdown_connections 2024-03-18T16:07:31.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:31.839+0000 7f0f36a00700 1 -- 172.21.15.39:0/1444581979 wait complete. 2024-03-18T16:07:32.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:32 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/1444581979' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-18T16:07:32.705 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":2749214286},{"type":"v1","addr":"172.21.15.39:6801","nonce":2749214286}]},"active_addr":"172.21.15.39:6801/2749214286","active_change":"2024-03-18T15:58:45.340556+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24113,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.39:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.39:0","nonce":3746088120}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.39:0","nonce":265589236}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.39:0","nonce":1969807796}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.39:0","nonce":2682283296}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.39:0","nonce":1154490444}]}]} 2024-03-18T16:07:32.709 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-18T16:07:32.709 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-18T16:07:32.710 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph osd dump --format=json 2024-03-18T16:07:32.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:32 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/1444581979' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-18T16:07:32.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:32 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/1444581979' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-18T16:07:33.206 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T16:07:33.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:33 smithi039 ceph-mon[31721]: pgmap v342: 129 pgs: 129 active+clean; 637 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.6 KiB/s wr, 22 op/s 2024-03-18T16:07:33.727 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:33 smithi196 ceph-mon[36307]: pgmap v342: 129 pgs: 129 active+clean; 637 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.6 KiB/s wr, 22 op/s 2024-03-18T16:07:33.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:33 smithi194 ceph-mon[36200]: pgmap v342: 129 pgs: 129 active+clean; 637 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.6 KiB/s wr, 22 op/s 2024-03-18T16:07:35.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:35.494+0000 7f5033200700 1 -- 172.21.15.39:0/2676877901 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f502c1134a0 msgr2=0x7f502c115890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:35.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:35.494+0000 7f5033200700 1 --2- 172.21.15.39:0/2676877901 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f502c1134a0 0x7f502c115890 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f502400b0d0 tx=0x7f502400b3e0 comp rx=0 tx=0).stop 2024-03-18T16:07:35.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:35.495+0000 7f5033200700 1 -- 172.21.15.39:0/2676877901 shutdown_connections 2024-03-18T16:07:35.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:35.495+0000 7f5033200700 1 --2- 172.21.15.39:0/2676877901 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f502c1134a0 0x7f502c115890 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:35.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:35.495+0000 7f5033200700 1 --2- 172.21.15.39:0/2676877901 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f502c074500 0x7f502c0748e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:35.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:35.495+0000 7f5033200700 1 --2- 172.21.15.39:0/2676877901 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f502c074e20 0x7f502c112f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:35.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:35.495+0000 7f5033200700 1 -- 172.21.15.39:0/2676877901 >> 172.21.15.39:0/2676877901 conn(0x7f502c079430 msgr2=0x7f502c079840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:35.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:35.495+0000 7f5033200700 1 -- 172.21.15.39:0/2676877901 shutdown_connections 2024-03-18T16:07:35.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:35.495+0000 7f5033200700 1 -- 172.21.15.39:0/2676877901 wait complete. 2024-03-18T16:07:35.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:35.496+0000 7f5033200700 1 Processor -- start 2024-03-18T16:07:35.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:35.496+0000 7f5033200700 1 -- start start 2024-03-18T16:07:35.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:35.496+0000 7f5033200700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f502c074500 0x7f502c19f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:35.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:35.496+0000 7f5033200700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f502c074e20 0x7f502c19f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:35.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:35.496+0000 7f5033200700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f502c1134a0 0x7f502c1a3920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:35.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:35.496+0000 7f5033200700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f502c118ce0 con 0x7f502c074e20 2024-03-18T16:07:35.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:35.497+0000 7f5033200700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f502c118b60 con 0x7f502c1134a0 2024-03-18T16:07:35.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:35.497+0000 7f5033200700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f502c118e60 con 0x7f502c074500 2024-03-18T16:07:35.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:35.497+0000 7f502be00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f502c074500 0x7f502c19f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:35.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:35.497+0000 7f502be00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f502c074500 0x7f502c19f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:53684/0 (socket says 172.21.15.39:53684) 2024-03-18T16:07:35.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:35.497+0000 7f502be00700 1 -- 172.21.15.39:0/1885026447 learned_addr learned my addr 172.21.15.39:0/1885026447 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:07:35.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:35.497+0000 7f5031200700 1 --2- 172.21.15.39:0/1885026447 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f502c1134a0 0x7f502c1a3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:35.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:35.497+0000 7f502b400700 1 --2- 172.21.15.39:0/1885026447 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f502c074e20 0x7f502c19f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:35.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:35.497+0000 7f502be00700 1 -- 172.21.15.39:0/1885026447 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f502c1134a0 msgr2=0x7f502c1a3920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:35.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:35.497+0000 7f502be00700 1 --2- 172.21.15.39:0/1885026447 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f502c1134a0 0x7f502c1a3920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:35.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:35.497+0000 7f502be00700 1 -- 172.21.15.39:0/1885026447 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f502c074e20 msgr2=0x7f502c19f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:35.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:35.497+0000 7f502be00700 1 --2- 172.21.15.39:0/1885026447 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f502c074e20 0x7f502c19f590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:35.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:35.497+0000 7f502be00700 1 -- 172.21.15.39:0/1885026447 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f502c1a4060 con 0x7f502c074500 2024-03-18T16:07:35.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:35.497+0000 7f502be00700 1 --2- 172.21.15.39:0/1885026447 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f502c074500 0x7f502c19f050 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f501c00eb30 tx=0x7f501c00ee40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:35.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:35.498+0000 7f5028c00700 1 -- 172.21.15.39:0/1885026447 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f501c00cc90 con 0x7f502c074500 2024-03-18T16:07:35.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:35.498+0000 7f5028c00700 1 -- 172.21.15.39:0/1885026447 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f501c004500 con 0x7f502c074500 2024-03-18T16:07:35.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:35.498+0000 7f5033200700 1 -- 172.21.15.39:0/1885026447 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f502c1a4350 con 0x7f502c074500 2024-03-18T16:07:35.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:35.498+0000 7f5028c00700 1 -- 172.21.15.39:0/1885026447 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f501c010430 con 0x7f502c074500 2024-03-18T16:07:35.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:35.498+0000 7f5033200700 1 -- 172.21.15.39:0/1885026447 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f502c1abbd0 con 0x7f502c074500 2024-03-18T16:07:35.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:35.499+0000 7f5033200700 1 -- 172.21.15.39:0/1885026447 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f502c068f90 con 0x7f502c074500 2024-03-18T16:07:35.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:35.501+0000 7f5028c00700 1 -- 172.21.15.39:0/1885026447 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f501c003f70 con 0x7f502c074500 2024-03-18T16:07:35.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:35.503+0000 7f5028c00700 1 --2- 172.21.15.39:0/1885026447 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f50100a29a0 0x7f50100a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:35.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:35.504+0000 7f5028c00700 1 -- 172.21.15.39:0/1885026447 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f501c0cf150 con 0x7f502c074500 2024-03-18T16:07:35.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:35.504+0000 7f502b400700 1 --2- 172.21.15.39:0/1885026447 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f50100a29a0 0x7f50100a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:35.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:35.505+0000 7f502b400700 1 --2- 172.21.15.39:0/1885026447 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f50100a29a0 0x7f50100a4e60 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f5018009680 tx=0x7f5018009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:35.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:35.506+0000 7f5028c00700 1 -- 172.21.15.39:0/1885026447 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f501c09bdf0 con 0x7f502c074500 2024-03-18T16:07:35.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:35 smithi039 ceph-mon[31721]: pgmap v343: 129 pgs: 129 active+clean; 637 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.8 KiB/s wr, 22 op/s 2024-03-18T16:07:35.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:35 smithi196 ceph-mon[36307]: pgmap v343: 129 pgs: 129 active+clean; 637 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.8 KiB/s wr, 22 op/s 2024-03-18T16:07:35.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:35 smithi194 ceph-mon[36200]: pgmap v343: 129 pgs: 129 active+clean; 637 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 7.8 KiB/s wr, 22 op/s 2024-03-18T16:07:35.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:35.742+0000 7f5033200700 1 -- 172.21.15.39:0/1885026447 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f502c1abe60 con 0x7f502c074500 2024-03-18T16:07:35.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:35.744+0000 7f5028c00700 1 -- 172.21.15.39:0/1885026447 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24142 (secure 0 0 0) 0x7f501c09b6f0 con 0x7f502c074500 2024-03-18T16:07:35.746 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:07:35.746 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":89,"fsid":"1892d8fc-e540-11ee-95c9-87774f69a715","created":"2024-03-18T15:57:36.507008+0000","modified":"2024-03-18T16:06:49.510653+0000","last_up_change":"2024-03-18T16:05:45.250629+0000","last_in_change":"2024-03-18T16:05:24.841337+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T16:01:30.048247+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T16:06:42.191791+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T16:06:43.670325+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T16:06:47.523388+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"771e0558-be62-4436-94b0-6364751911c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6803","nonce":4090743635}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6805","nonce":4090743635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6809","nonce":4090743635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6807","nonce":4090743635}]},"public_addr":"172.21.15.39:6803/4090743635","cluster_addr":"172.21.15.39:6805/4090743635","heartbeat_back_addr":"172.21.15.39:6809/4090743635","heartbeat_front_addr":"172.21.15.39:6807/4090743635","state":["exists","up"]},{"osd":1,"uuid":"0430eee1-cf34-4277-b255-c7d9f07ad19a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6801","nonce":2800329805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6803","nonce":2800329805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6807","nonce":2800329805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6805","nonce":2800329805}]},"public_addr":"172.21.15.194:6801/2800329805","cluster_addr":"172.21.15.194:6803/2800329805","heartbeat_back_addr":"172.21.15.194:6807/2800329805","heartbeat_front_addr":"172.21.15.194:6805/2800329805","state":["exists","up"]},{"osd":2,"uuid":"3a718da4-2511-4ec1-b1a8-eafd6cce0df0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6801","nonce":3935874719}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6803","nonce":3935874719}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6807","nonce":3935874719}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6805","nonce":3935874719}]},"public_addr":"172.21.15.196:6801/3935874719","cluster_addr":"172.21.15.196:6803/3935874719","heartbeat_back_addr":"172.21.15.196:6807/3935874719","heartbeat_front_addr":"172.21.15.196:6805/3935874719","state":["exists","up"]},{"osd":3,"uuid":"6f9c37f9-0ab9-4038-8919-afd4b39c4165","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6811","nonce":893345939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6813","nonce":893345939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6817","nonce":893345939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6815","nonce":893345939}]},"public_addr":"172.21.15.39:6811/893345939","cluster_addr":"172.21.15.39:6813/893345939","heartbeat_back_addr":"172.21.15.39:6817/893345939","heartbeat_front_addr":"172.21.15.39:6815/893345939","state":["exists","up"]},{"osd":4,"uuid":"e795bb0e-4791-4537-acda-1a321e19bc9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6809","nonce":800423088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6811","nonce":800423088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6815","nonce":800423088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6813","nonce":800423088}]},"public_addr":"172.21.15.194:6809/800423088","cluster_addr":"172.21.15.194:6811/800423088","heartbeat_back_addr":"172.21.15.194:6815/800423088","heartbeat_front_addr":"172.21.15.194:6813/800423088","state":["exists","up"]},{"osd":5,"uuid":"026e175e-6acb-4b4c-b235-8ccee8ae1149","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6809","nonce":2746224525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6811","nonce":2746224525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6815","nonce":2746224525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6813","nonce":2746224525}]},"public_addr":"172.21.15.196:6809/2746224525","cluster_addr":"172.21.15.196:6811/2746224525","heartbeat_back_addr":"172.21.15.196:6815/2746224525","heartbeat_front_addr":"172.21.15.196:6813/2746224525","state":["exists","up"]},{"osd":6,"uuid":"c1b183ff-383f-4c5c-9075-f76bab1c0bb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6819","nonce":1213371331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6821","nonce":1213371331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6825","nonce":1213371331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6823","nonce":1213371331}]},"public_addr":"172.21.15.39:6819/1213371331","cluster_addr":"172.21.15.39:6821/1213371331","heartbeat_back_addr":"172.21.15.39:6825/1213371331","heartbeat_front_addr":"172.21.15.39:6823/1213371331","state":["exists","up"]},{"osd":7,"uuid":"612f8697-957d-4ef4-a87f-241b36f9d13d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6817","nonce":3299162444}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6819","nonce":3299162444}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6823","nonce":3299162444}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6821","nonce":3299162444}]},"public_addr":"172.21.15.194:6817/3299162444","cluster_addr":"172.21.15.194:6819/3299162444","heartbeat_back_addr":"172.21.15.194:6823/3299162444","heartbeat_front_addr":"172.21.15.194:6821/3299162444","state":["exists","up"]},{"osd":8,"uuid":"836a9107-4d4f-4594-b8dc-0301e64f59d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6817","nonce":2456143145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6819","nonce":2456143145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6823","nonce":2456143145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6821","nonce":2456143145}]},"public_addr":"172.21.15.196:6817/2456143145","cluster_addr":"172.21.15.196:6819/2456143145","heartbeat_back_addr":"172.21.15.196:6823/2456143145","heartbeat_front_addr":"172.21.15.196:6821/2456143145","state":["exists","up"]},{"osd":9,"uuid":"4f30b960-12e3-4b16-bd28-fcc988139b58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6827","nonce":1848634433}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6829","nonce":1848634433}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6833","nonce":1848634433}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6831","nonce":1848634433}]},"public_addr":"172.21.15.39:6827/1848634433","cluster_addr":"172.21.15.39:6829/1848634433","heartbeat_back_addr":"172.21.15.39:6833/1848634433","heartbeat_front_addr":"172.21.15.39:6831/1848634433","state":["exists","up"]},{"osd":10,"uuid":"620f90e8-23e6-4bc0-8269-7219b9e1aebd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6825","nonce":3725064809}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6827","nonce":3725064809}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6831","nonce":3725064809}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6829","nonce":3725064809}]},"public_addr":"172.21.15.194:6825/3725064809","cluster_addr":"172.21.15.194:6827/3725064809","heartbeat_back_addr":"172.21.15.194:6831/3725064809","heartbeat_front_addr":"172.21.15.194:6829/3725064809","state":["exists","up"]},{"osd":11,"uuid":"867a2535-8235-4ceb-a698-eb86869d6d20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6825","nonce":2501212422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6827","nonce":2501212422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6831","nonce":2501212422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6829","nonce":2501212422}]},"public_addr":"172.21.15.196:6825/2501212422","cluster_addr":"172.21.15.196:6827/2501212422","heartbeat_back_addr":"172.21.15.196:6831/2501212422","heartbeat_front_addr":"172.21.15.196:6829/2501212422","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:00:36.931566+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:01:02.614024+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:01:26.193265+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:01:58.308833+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:02:25.049546+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:02:49.748750+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:03:23.582153+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:03:50.447321+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:04:15.812129+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:04:49.998047+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:05:17.087924+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:05:43.167637+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.39:0/3482086140":"2024-03-19T15:58:45.340294+0000","172.21.15.39:0/3844306284":"2024-03-19T15:58:45.340294+0000","172.21.15.39:6801/1810238546":"2024-03-19T15:58:45.340294+0000","172.21.15.39:6800/1810238546":"2024-03-19T15:58:45.340294+0000","172.21.15.39:0/2528397488":"2024-03-19T15:58:10.264798+0000","172.21.15.39:0/2240981254":"2024-03-19T15:58:10.264798+0000","172.21.15.39:6800/1217503189":"2024-03-19T15:58:10.264798+0000","172.21.15.39:0/3754876370":"2024-03-19T15:58:10.264798+0000","172.21.15.39:6801/1217503189":"2024-03-19T15:58:10.264798+0000","172.21.15.39:0/1874637628":"2024-03-19T15:58:45.340294+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T16:07:35.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:35.747+0000 7f5033200700 1 -- 172.21.15.39:0/1885026447 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f50100a29a0 msgr2=0x7f50100a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:35.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:35.747+0000 7f5033200700 1 --2- 172.21.15.39:0/1885026447 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f50100a29a0 0x7f50100a4e60 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f5018009680 tx=0x7f5018009260 comp rx=0 tx=0).stop 2024-03-18T16:07:35.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:35.747+0000 7f5033200700 1 -- 172.21.15.39:0/1885026447 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f502c074500 msgr2=0x7f502c19f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:35.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:35.747+0000 7f5033200700 1 --2- 172.21.15.39:0/1885026447 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f502c074500 0x7f502c19f050 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f501c00eb30 tx=0x7f501c00ee40 comp rx=0 tx=0).stop 2024-03-18T16:07:35.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:35.747+0000 7f5033200700 1 -- 172.21.15.39:0/1885026447 shutdown_connections 2024-03-18T16:07:35.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:35.747+0000 7f5033200700 1 --2- 172.21.15.39:0/1885026447 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f50100a29a0 0x7f50100a4e60 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:35.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:35.747+0000 7f5033200700 1 --2- 172.21.15.39:0/1885026447 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f502c1134a0 0x7f502c1a3920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:35.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:35.747+0000 7f5033200700 1 --2- 172.21.15.39:0/1885026447 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f502c074500 0x7f502c19f050 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:35.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:35.747+0000 7f5033200700 1 --2- 172.21.15.39:0/1885026447 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f502c074e20 0x7f502c19f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:35.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:35.747+0000 7f5033200700 1 -- 172.21.15.39:0/1885026447 >> 172.21.15.39:0/1885026447 conn(0x7f502c079430 msgr2=0x7f502c112200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:35.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:35.747+0000 7f5033200700 1 -- 172.21.15.39:0/1885026447 shutdown_connections 2024-03-18T16:07:35.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:35.748+0000 7f5033200700 1 -- 172.21.15.39:0/1885026447 wait complete. 2024-03-18T16:07:36.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:36 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/1885026447' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:07:36.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:36 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/1885026447' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:07:36.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:36 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/1885026447' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:07:36.781 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-18T16:07:36.781 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph osd dump --format=json 2024-03-18T16:07:37.265 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T16:07:37.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:37 smithi039 ceph-mon[31721]: pgmap v344: 129 pgs: 129 active+clean; 642 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.6 KiB/s wr, 14 op/s 2024-03-18T16:07:37.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:37 smithi196 ceph-mon[36307]: pgmap v344: 129 pgs: 129 active+clean; 642 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.6 KiB/s wr, 14 op/s 2024-03-18T16:07:37.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:37 smithi194 ceph-mon[36200]: pgmap v344: 129 pgs: 129 active+clean; 642 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.6 KiB/s wr, 14 op/s 2024-03-18T16:07:39.557 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:39 smithi039 ceph-mon[31721]: pgmap v345: 129 pgs: 129 active+clean; 642 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.4 KiB/s wr, 11 op/s 2024-03-18T16:07:39.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:39.582+0000 7f265a200700 1 -- 172.21.15.39:0/2594886219 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2654103d50 msgr2=0x7f265410c240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:39.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:39.582+0000 7f265a200700 1 --2- 172.21.15.39:0/2594886219 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2654103d50 0x7f265410c240 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f2640009990 tx=0x7f2640009ca0 comp rx=0 tx=0).stop 2024-03-18T16:07:39.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:39.582+0000 7f265a200700 1 -- 172.21.15.39:0/2594886219 shutdown_connections 2024-03-18T16:07:39.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:39.582+0000 7f265a200700 1 --2- 172.21.15.39:0/2594886219 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f265410c810 0x7f265410ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:39.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:39.582+0000 7f265a200700 1 --2- 172.21.15.39:0/2594886219 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2654103430 0x7f2654103810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:39.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:39.582+0000 7f265a200700 1 --2- 172.21.15.39:0/2594886219 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2654103d50 0x7f265410c240 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:39.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:39.582+0000 7f265a200700 1 -- 172.21.15.39:0/2594886219 >> 172.21.15.39:0/2594886219 conn(0x7f26540fcba0 msgr2=0x7f26540fefc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:39.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:39.582+0000 7f265a200700 1 -- 172.21.15.39:0/2594886219 shutdown_connections 2024-03-18T16:07:39.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:39.582+0000 7f265a200700 1 -- 172.21.15.39:0/2594886219 wait complete. 2024-03-18T16:07:39.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:39.583+0000 7f265a200700 1 Processor -- start 2024-03-18T16:07:39.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:39.584+0000 7f265a200700 1 -- start start 2024-03-18T16:07:39.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:39.584+0000 7f265a200700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2654103430 0x7f26541a08a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:39.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:39.584+0000 7f265a200700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2654103d50 0x7f26541a0de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:39.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:39.584+0000 7f265a200700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f265410c810 0x7f265419aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:39.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:39.584+0000 7f265a200700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f2654112520 con 0x7f2654103430 2024-03-18T16:07:39.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:39.584+0000 7f265a200700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f26541123a0 con 0x7f265410c810 2024-03-18T16:07:39.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:39.584+0000 7f265a200700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f26541126a0 con 0x7f2654103d50 2024-03-18T16:07:39.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:39.585+0000 7f2658e00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2654103430 0x7f26541a08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:39.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:39.585+0000 7f2653e00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2654103d50 0x7f26541a0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:39.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:39.585+0000 7f2659800700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f265410c810 0x7f265419aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:39.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:39.585+0000 7f2658e00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2654103430 0x7f26541a08a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:46878/0 (socket says 172.21.15.39:46878) 2024-03-18T16:07:39.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:39.585+0000 7f2658e00700 1 -- 172.21.15.39:0/2695710483 learned_addr learned my addr 172.21.15.39:0/2695710483 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:07:39.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:39.585+0000 7f2659800700 1 -- 172.21.15.39:0/2695710483 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2654103d50 msgr2=0x7f26541a0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:39.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:39.585+0000 7f2659800700 1 --2- 172.21.15.39:0/2695710483 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2654103d50 0x7f26541a0de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:39.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:39.585+0000 7f2659800700 1 -- 172.21.15.39:0/2695710483 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2654103430 msgr2=0x7f26541a08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:39.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:39.585+0000 7f2659800700 1 --2- 172.21.15.39:0/2695710483 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2654103430 0x7f26541a08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:39.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:39.585+0000 7f2659800700 1 -- 172.21.15.39:0/2695710483 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f265419b1c0 con 0x7f265410c810 2024-03-18T16:07:39.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:39.585+0000 7f2658e00700 1 --2- 172.21.15.39:0/2695710483 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2654103430 0x7f26541a08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:07:39.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:39.585+0000 7f2659800700 1 --2- 172.21.15.39:0/2695710483 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f265410c810 0x7f265419aa60 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f264c00e970 tx=0x7f264c00ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:39.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:39.586+0000 7f2651600700 1 -- 172.21.15.39:0/2695710483 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f264c004220 con 0x7f265410c810 2024-03-18T16:07:39.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:39.586+0000 7f2651600700 1 -- 172.21.15.39:0/2695710483 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f264c004d10 con 0x7f265410c810 2024-03-18T16:07:39.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:39.586+0000 7f2651600700 1 -- 172.21.15.39:0/2695710483 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f264c010700 con 0x7f265410c810 2024-03-18T16:07:39.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:39.586+0000 7f265a200700 1 -- 172.21.15.39:0/2695710483 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f265419b4b0 con 0x7f265410c810 2024-03-18T16:07:39.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:39.586+0000 7f265a200700 1 -- 172.21.15.39:0/2695710483 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26541a7770 con 0x7f265410c810 2024-03-18T16:07:39.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:39.587+0000 7f265a200700 1 -- 172.21.15.39:0/2695710483 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2654068f90 con 0x7f265410c810 2024-03-18T16:07:39.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:39.589+0000 7f2651600700 1 -- 172.21.15.39:0/2695710483 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f264c004750 con 0x7f265410c810 2024-03-18T16:07:39.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:39.591+0000 7f2651600700 1 --2- 172.21.15.39:0/2695710483 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f263c0a29a0 0x7f263c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:39.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:39.591+0000 7f2651600700 1 -- 172.21.15.39:0/2695710483 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f264c0ceb20 con 0x7f265410c810 2024-03-18T16:07:39.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:39.591+0000 7f2658e00700 1 --2- 172.21.15.39:0/2695710483 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f263c0a29a0 0x7f263c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:39.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:39.591+0000 7f2658e00700 1 --2- 172.21.15.39:0/2695710483 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f263c0a29a0 0x7f263c0a4e60 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f2648009c10 tx=0x7f264800b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:39.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:39.594+0000 7f2651600700 1 -- 172.21.15.39:0/2695710483 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f264c09b7c0 con 0x7f265410c810 2024-03-18T16:07:39.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:39 smithi196 ceph-mon[36307]: pgmap v345: 129 pgs: 129 active+clean; 642 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.4 KiB/s wr, 11 op/s 2024-03-18T16:07:39.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:39 smithi194 ceph-mon[36200]: pgmap v345: 129 pgs: 129 active+clean; 642 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.4 KiB/s wr, 11 op/s 2024-03-18T16:07:39.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:39.836+0000 7f265a200700 1 -- 172.21.15.39:0/2695710483 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f265419bf90 con 0x7f265410c810 2024-03-18T16:07:39.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:39.839+0000 7f2651600700 1 -- 172.21.15.39:0/2695710483 <== mon.2 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24142 (secure 0 0 0) 0x7f264c09b0c0 con 0x7f265410c810 2024-03-18T16:07:39.841 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:07:39.842 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":89,"fsid":"1892d8fc-e540-11ee-95c9-87774f69a715","created":"2024-03-18T15:57:36.507008+0000","modified":"2024-03-18T16:06:49.510653+0000","last_up_change":"2024-03-18T16:05:45.250629+0000","last_in_change":"2024-03-18T16:05:24.841337+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T16:01:30.048247+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T16:06:42.191791+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T16:06:43.670325+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T16:06:47.523388+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"771e0558-be62-4436-94b0-6364751911c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6803","nonce":4090743635}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6805","nonce":4090743635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6809","nonce":4090743635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6807","nonce":4090743635}]},"public_addr":"172.21.15.39:6803/4090743635","cluster_addr":"172.21.15.39:6805/4090743635","heartbeat_back_addr":"172.21.15.39:6809/4090743635","heartbeat_front_addr":"172.21.15.39:6807/4090743635","state":["exists","up"]},{"osd":1,"uuid":"0430eee1-cf34-4277-b255-c7d9f07ad19a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6801","nonce":2800329805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6803","nonce":2800329805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6807","nonce":2800329805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6805","nonce":2800329805}]},"public_addr":"172.21.15.194:6801/2800329805","cluster_addr":"172.21.15.194:6803/2800329805","heartbeat_back_addr":"172.21.15.194:6807/2800329805","heartbeat_front_addr":"172.21.15.194:6805/2800329805","state":["exists","up"]},{"osd":2,"uuid":"3a718da4-2511-4ec1-b1a8-eafd6cce0df0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6801","nonce":3935874719}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6803","nonce":3935874719}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6807","nonce":3935874719}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6805","nonce":3935874719}]},"public_addr":"172.21.15.196:6801/3935874719","cluster_addr":"172.21.15.196:6803/3935874719","heartbeat_back_addr":"172.21.15.196:6807/3935874719","heartbeat_front_addr":"172.21.15.196:6805/3935874719","state":["exists","up"]},{"osd":3,"uuid":"6f9c37f9-0ab9-4038-8919-afd4b39c4165","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6811","nonce":893345939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6813","nonce":893345939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6817","nonce":893345939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6815","nonce":893345939}]},"public_addr":"172.21.15.39:6811/893345939","cluster_addr":"172.21.15.39:6813/893345939","heartbeat_back_addr":"172.21.15.39:6817/893345939","heartbeat_front_addr":"172.21.15.39:6815/893345939","state":["exists","up"]},{"osd":4,"uuid":"e795bb0e-4791-4537-acda-1a321e19bc9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6809","nonce":800423088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6811","nonce":800423088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6815","nonce":800423088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6813","nonce":800423088}]},"public_addr":"172.21.15.194:6809/800423088","cluster_addr":"172.21.15.194:6811/800423088","heartbeat_back_addr":"172.21.15.194:6815/800423088","heartbeat_front_addr":"172.21.15.194:6813/800423088","state":["exists","up"]},{"osd":5,"uuid":"026e175e-6acb-4b4c-b235-8ccee8ae1149","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6809","nonce":2746224525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6811","nonce":2746224525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6815","nonce":2746224525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6813","nonce":2746224525}]},"public_addr":"172.21.15.196:6809/2746224525","cluster_addr":"172.21.15.196:6811/2746224525","heartbeat_back_addr":"172.21.15.196:6815/2746224525","heartbeat_front_addr":"172.21.15.196:6813/2746224525","state":["exists","up"]},{"osd":6,"uuid":"c1b183ff-383f-4c5c-9075-f76bab1c0bb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6819","nonce":1213371331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6821","nonce":1213371331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6825","nonce":1213371331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6823","nonce":1213371331}]},"public_addr":"172.21.15.39:6819/1213371331","cluster_addr":"172.21.15.39:6821/1213371331","heartbeat_back_addr":"172.21.15.39:6825/1213371331","heartbeat_front_addr":"172.21.15.39:6823/1213371331","state":["exists","up"]},{"osd":7,"uuid":"612f8697-957d-4ef4-a87f-241b36f9d13d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6817","nonce":3299162444}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6819","nonce":3299162444}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6823","nonce":3299162444}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6821","nonce":3299162444}]},"public_addr":"172.21.15.194:6817/3299162444","cluster_addr":"172.21.15.194:6819/3299162444","heartbeat_back_addr":"172.21.15.194:6823/3299162444","heartbeat_front_addr":"172.21.15.194:6821/3299162444","state":["exists","up"]},{"osd":8,"uuid":"836a9107-4d4f-4594-b8dc-0301e64f59d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6817","nonce":2456143145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6819","nonce":2456143145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6823","nonce":2456143145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6821","nonce":2456143145}]},"public_addr":"172.21.15.196:6817/2456143145","cluster_addr":"172.21.15.196:6819/2456143145","heartbeat_back_addr":"172.21.15.196:6823/2456143145","heartbeat_front_addr":"172.21.15.196:6821/2456143145","state":["exists","up"]},{"osd":9,"uuid":"4f30b960-12e3-4b16-bd28-fcc988139b58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6827","nonce":1848634433}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6829","nonce":1848634433}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6833","nonce":1848634433}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6831","nonce":1848634433}]},"public_addr":"172.21.15.39:6827/1848634433","cluster_addr":"172.21.15.39:6829/1848634433","heartbeat_back_addr":"172.21.15.39:6833/1848634433","heartbeat_front_addr":"172.21.15.39:6831/1848634433","state":["exists","up"]},{"osd":10,"uuid":"620f90e8-23e6-4bc0-8269-7219b9e1aebd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6825","nonce":3725064809}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6827","nonce":3725064809}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6831","nonce":3725064809}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6829","nonce":3725064809}]},"public_addr":"172.21.15.194:6825/3725064809","cluster_addr":"172.21.15.194:6827/3725064809","heartbeat_back_addr":"172.21.15.194:6831/3725064809","heartbeat_front_addr":"172.21.15.194:6829/3725064809","state":["exists","up"]},{"osd":11,"uuid":"867a2535-8235-4ceb-a698-eb86869d6d20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6825","nonce":2501212422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6827","nonce":2501212422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6831","nonce":2501212422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6829","nonce":2501212422}]},"public_addr":"172.21.15.196:6825/2501212422","cluster_addr":"172.21.15.196:6827/2501212422","heartbeat_back_addr":"172.21.15.196:6831/2501212422","heartbeat_front_addr":"172.21.15.196:6829/2501212422","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:00:36.931566+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:01:02.614024+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:01:26.193265+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:01:58.308833+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:02:25.049546+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:02:49.748750+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:03:23.582153+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:03:50.447321+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:04:15.812129+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:04:49.998047+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:05:17.087924+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:05:43.167637+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.39:0/3482086140":"2024-03-19T15:58:45.340294+0000","172.21.15.39:0/3844306284":"2024-03-19T15:58:45.340294+0000","172.21.15.39:6801/1810238546":"2024-03-19T15:58:45.340294+0000","172.21.15.39:6800/1810238546":"2024-03-19T15:58:45.340294+0000","172.21.15.39:0/2528397488":"2024-03-19T15:58:10.264798+0000","172.21.15.39:0/2240981254":"2024-03-19T15:58:10.264798+0000","172.21.15.39:6800/1217503189":"2024-03-19T15:58:10.264798+0000","172.21.15.39:0/3754876370":"2024-03-19T15:58:10.264798+0000","172.21.15.39:6801/1217503189":"2024-03-19T15:58:10.264798+0000","172.21.15.39:0/1874637628":"2024-03-19T15:58:45.340294+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T16:07:39.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:39.843+0000 7f265a200700 1 -- 172.21.15.39:0/2695710483 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f263c0a29a0 msgr2=0x7f263c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:39.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:39.843+0000 7f265a200700 1 --2- 172.21.15.39:0/2695710483 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f263c0a29a0 0x7f263c0a4e60 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f2648009c10 tx=0x7f264800b040 comp rx=0 tx=0).stop 2024-03-18T16:07:39.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:39.843+0000 7f265a200700 1 -- 172.21.15.39:0/2695710483 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f265410c810 msgr2=0x7f265419aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:39.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:39.843+0000 7f265a200700 1 --2- 172.21.15.39:0/2695710483 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f265410c810 0x7f265419aa60 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f264c00e970 tx=0x7f264c00ec80 comp rx=0 tx=0).stop 2024-03-18T16:07:39.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:39.843+0000 7f265a200700 1 -- 172.21.15.39:0/2695710483 shutdown_connections 2024-03-18T16:07:39.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:39.843+0000 7f265a200700 1 --2- 172.21.15.39:0/2695710483 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f263c0a29a0 0x7f263c0a4e60 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:39.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:39.843+0000 7f265a200700 1 --2- 172.21.15.39:0/2695710483 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f265410c810 0x7f265419aa60 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:39.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:39.843+0000 7f265a200700 1 --2- 172.21.15.39:0/2695710483 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2654103430 0x7f26541a08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:39.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:39.843+0000 7f265a200700 1 --2- 172.21.15.39:0/2695710483 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2654103d50 0x7f26541a0de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:39.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:39.843+0000 7f265a200700 1 -- 172.21.15.39:0/2695710483 >> 172.21.15.39:0/2695710483 conn(0x7f26540fcba0 msgr2=0x7f26540fd810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:39.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:39.844+0000 7f265a200700 1 -- 172.21.15.39:0/2695710483 shutdown_connections 2024-03-18T16:07:39.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:39.844+0000 7f265a200700 1 -- 172.21.15.39:0/2695710483 wait complete. 2024-03-18T16:07:40.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:40 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/2695710483' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:07:40.724 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-18T16:07:40.724 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-18T16:07:40.725 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-18T16:07:40.725 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-18T16:07:40.725 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-18T16:07:40.726 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-18T16:07:40.726 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-18T16:07:40.727 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-18T16:07:40.727 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph tell osd.8 flush_pg_stats 2024-03-18T16:07:40.728 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph tell osd.9 flush_pg_stats 2024-03-18T16:07:40.728 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph tell osd.10 flush_pg_stats 2024-03-18T16:07:40.728 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph tell osd.11 flush_pg_stats 2024-03-18T16:07:40.731 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:40 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/2695710483' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:07:40.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:40 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/2695710483' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:07:41.318 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:41 smithi039 ceph-mon[31721]: pgmap v346: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.6 KiB/s wr, 11 op/s 2024-03-18T16:07:41.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:41 smithi196 ceph-mon[36307]: pgmap v346: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.6 KiB/s wr, 11 op/s 2024-03-18T16:07:41.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:41 smithi194 ceph-mon[36200]: pgmap v346: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.6 KiB/s wr, 11 op/s 2024-03-18T16:07:41.926 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T16:07:41.948 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T16:07:41.963 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T16:07:41.975 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T16:07:41.983 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T16:07:42.734 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T16:07:42.734 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T16:07:42.736 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T16:07:42.738 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T16:07:42.740 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T16:07:42.740 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T16:07:43.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:43 smithi039 ceph-mon[31721]: pgmap v347: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 682 B/s wr, 9 op/s 2024-03-18T16:07:43.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:43 smithi196 ceph-mon[36307]: pgmap v347: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 682 B/s wr, 9 op/s 2024-03-18T16:07:43.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:43 smithi194 ceph-mon[36200]: pgmap v347: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 682 B/s wr, 9 op/s 2024-03-18T16:07:44.474 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:44 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:07:44.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:44 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:07:44.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:44 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:07:45.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:45 smithi194 ceph-mon[36200]: pgmap v348: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 682 B/s wr, 10 op/s 2024-03-18T16:07:45.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:45 smithi039 ceph-mon[31721]: pgmap v348: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 682 B/s wr, 10 op/s 2024-03-18T16:07:45.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:45 smithi196 ceph-mon[36307]: pgmap v348: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 682 B/s wr, 10 op/s 2024-03-18T16:07:46.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:46 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]: dispatch 2024-03-18T16:07:46.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:46 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8]}]: dispatch 2024-03-18T16:07:46.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:46 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 7]}]: dispatch 2024-03-18T16:07:46.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:46 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]: dispatch 2024-03-18T16:07:46.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:46 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 6]}]: dispatch 2024-03-18T16:07:46.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:46 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]: dispatch 2024-03-18T16:07:46.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:46 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [9, 10]}]: dispatch 2024-03-18T16:07:46.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:46 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [5, 8]}]: dispatch 2024-03-18T16:07:46.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:46 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]: dispatch 2024-03-18T16:07:46.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:46 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]: dispatch 2024-03-18T16:07:46.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:46 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:07:46.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:46 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:07:46.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:46 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:07:46.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:46 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:07:46.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:46 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]: dispatch 2024-03-18T16:07:46.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:46 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8]}]: dispatch 2024-03-18T16:07:46.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:46 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 7]}]: dispatch 2024-03-18T16:07:46.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:46 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]: dispatch 2024-03-18T16:07:46.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:46 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 6]}]: dispatch 2024-03-18T16:07:46.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:46 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]: dispatch 2024-03-18T16:07:46.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:46 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [9, 10]}]: dispatch 2024-03-18T16:07:46.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:46 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [5, 8]}]: dispatch 2024-03-18T16:07:46.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:46 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]: dispatch 2024-03-18T16:07:46.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:46 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]: dispatch 2024-03-18T16:07:46.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:46 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:07:46.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:46 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:07:46.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:46 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:07:46.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:46 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:07:46.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:46 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]: dispatch 2024-03-18T16:07:46.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:46 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8]}]: dispatch 2024-03-18T16:07:46.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:46 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 7]}]: dispatch 2024-03-18T16:07:46.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:46 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]: dispatch 2024-03-18T16:07:46.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:46 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 6]}]: dispatch 2024-03-18T16:07:46.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:46 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]: dispatch 2024-03-18T16:07:46.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:46 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [9, 10]}]: dispatch 2024-03-18T16:07:46.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:46 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [5, 8]}]: dispatch 2024-03-18T16:07:46.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:46 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]: dispatch 2024-03-18T16:07:46.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:46 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]: dispatch 2024-03-18T16:07:46.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:46 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:07:46.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:46 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:07:46.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:46 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:07:46.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:46 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:07:47.699 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:47 smithi039 ceph-mon[31721]: pgmap v349: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 682 B/s wr, 10 op/s 2024-03-18T16:07:47.699 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:47 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2024-03-18T16:07:47.699 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:47 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8]}]': finished 2024-03-18T16:07:47.700 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:47 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 7]}]': finished 2024-03-18T16:07:47.700 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:47 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]': finished 2024-03-18T16:07:47.700 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:47 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 6]}]': finished 2024-03-18T16:07:47.700 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:47 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]': finished 2024-03-18T16:07:47.700 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:47 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [9, 10]}]': finished 2024-03-18T16:07:47.700 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:47 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [5, 8]}]': finished 2024-03-18T16:07:47.700 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:47 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]': finished 2024-03-18T16:07:47.700 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:47 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]': finished 2024-03-18T16:07:47.700 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:47 smithi039 ceph-mon[31721]: osdmap e90: 12 total, 12 up, 12 in 2024-03-18T16:07:47.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:47 smithi196 ceph-mon[36307]: pgmap v349: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 682 B/s wr, 10 op/s 2024-03-18T16:07:47.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:47 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2024-03-18T16:07:47.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:47 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8]}]': finished 2024-03-18T16:07:47.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:47 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 7]}]': finished 2024-03-18T16:07:47.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:47 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]': finished 2024-03-18T16:07:47.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:47 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 6]}]': finished 2024-03-18T16:07:47.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:47 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]': finished 2024-03-18T16:07:47.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:47 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [9, 10]}]': finished 2024-03-18T16:07:47.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:47 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [5, 8]}]': finished 2024-03-18T16:07:47.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:47 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]': finished 2024-03-18T16:07:47.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:47 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]': finished 2024-03-18T16:07:47.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:47 smithi196 ceph-mon[36307]: osdmap e90: 12 total, 12 up, 12 in 2024-03-18T16:07:47.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:47 smithi194 ceph-mon[36200]: pgmap v349: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 682 B/s wr, 10 op/s 2024-03-18T16:07:47.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:47 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.c", "id": [3, 9]}]': finished 2024-03-18T16:07:47.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:47 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8]}]': finished 2024-03-18T16:07:47.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:47 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1d", "id": [4, 7]}]': finished 2024-03-18T16:07:47.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:47 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.20", "id": [5, 9]}]': finished 2024-03-18T16:07:47.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:47 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 6]}]': finished 2024-03-18T16:07:47.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:47 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]': finished 2024-03-18T16:07:47.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:47 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.5", "id": [9, 10]}]': finished 2024-03-18T16:07:47.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:47 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [5, 8]}]': finished 2024-03-18T16:07:47.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:47 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 8]}]': finished 2024-03-18T16:07:47.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:47 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]': finished 2024-03-18T16:07:47.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:47 smithi194 ceph-mon[36200]: osdmap e90: 12 total, 12 up, 12 in 2024-03-18T16:07:48.619 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:48 smithi039 ceph-mon[31721]: osdmap e91: 12 total, 12 up, 12 in 2024-03-18T16:07:48.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:48 smithi196 ceph-mon[36307]: osdmap e91: 12 total, 12 up, 12 in 2024-03-18T16:07:48.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:48 smithi194 ceph-mon[36200]: osdmap e91: 12 total, 12 up, 12 in 2024-03-18T16:07:49.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:49 smithi039 ceph-mon[31721]: pgmap v352: 129 pgs: 3 peering, 126 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s; 14 B/s, 0 objects/s recovering 2024-03-18T16:07:49.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:49 smithi196 ceph-mon[36307]: pgmap v352: 129 pgs: 3 peering, 126 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s; 14 B/s, 0 objects/s recovering 2024-03-18T16:07:49.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:49 smithi194 ceph-mon[36200]: pgmap v352: 129 pgs: 3 peering, 126 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s; 14 B/s, 0 objects/s recovering 2024-03-18T16:07:50.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:50 smithi196 ceph-mon[36307]: pgmap v353: 129 pgs: 3 peering, 126 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s; 14 B/s, 0 objects/s recovering 2024-03-18T16:07:50.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:50 smithi194 ceph-mon[36200]: pgmap v353: 129 pgs: 3 peering, 126 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s; 14 B/s, 0 objects/s recovering 2024-03-18T16:07:50.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:50 smithi039 ceph-mon[31721]: pgmap v353: 129 pgs: 3 peering, 126 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s; 14 B/s, 0 objects/s recovering 2024-03-18T16:07:51.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:51 smithi196 ceph-mon[36307]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-03-18T16:07:51.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:51 smithi194 ceph-mon[36200]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-03-18T16:07:52.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:51 smithi039 ceph-mon[31721]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-03-18T16:07:52.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:52 smithi039 ceph-mon[31721]: pgmap v354: 129 pgs: 3 peering, 126 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 9 op/s; 14 B/s, 0 objects/s recovering 2024-03-18T16:07:52.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:52 smithi196 ceph-mon[36307]: pgmap v354: 129 pgs: 3 peering, 126 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 9 op/s; 14 B/s, 0 objects/s recovering 2024-03-18T16:07:52.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:52 smithi194 ceph-mon[36200]: pgmap v354: 129 pgs: 3 peering, 126 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 9 op/s; 14 B/s, 0 objects/s recovering 2024-03-18T16:07:53.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:53.421+0000 7fc8ba600700 1 -- 172.21.15.39:0/4061457056 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc8b4104ec0 msgr2=0x7fc8b4111380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:53.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:53.421+0000 7fc8ba600700 1 --2- 172.21.15.39:0/4061457056 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc8b4104ec0 0x7fc8b4111380 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc8ac00b0a0 tx=0x7fc8ac00b3b0 comp rx=0 tx=0).stop 2024-03-18T16:07:53.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:53.421+0000 7fc8ba600700 1 -- 172.21.15.39:0/4061457056 shutdown_connections 2024-03-18T16:07:53.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:53.421+0000 7fc8ba600700 1 --2- 172.21.15.39:0/4061457056 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc8b4104ec0 0x7fc8b4111380 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:53.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:53.421+0000 7fc8ba600700 1 --2- 172.21.15.39:0/4061457056 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc8b410a5b0 0x7fc8b410a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:53.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:53.421+0000 7fc8ba600700 1 --2- 172.21.15.39:0/4061457056 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc8b4104500 0x7fc8b4104980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:53.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:53.421+0000 7fc8ba600700 1 -- 172.21.15.39:0/4061457056 >> 172.21.15.39:0/4061457056 conn(0x7fc8b40fffd0 msgr2=0x7fc8b41023f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:53.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:53.421+0000 7fc8ba600700 1 -- 172.21.15.39:0/4061457056 shutdown_connections 2024-03-18T16:07:53.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:53.421+0000 7fc8ba600700 1 -- 172.21.15.39:0/4061457056 wait complete. 2024-03-18T16:07:53.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:53.422+0000 7fc8ba600700 1 Processor -- start 2024-03-18T16:07:53.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:53.423+0000 7fc8ba600700 1 -- start start 2024-03-18T16:07:53.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:53.423+0000 7fc8ba600700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc8b4104500 0x7fc8b41a0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:53.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:53.423+0000 7fc8ba600700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc8b4104ec0 0x7fc8b41a0d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:53.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:53.423+0000 7fc8ba600700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc8b410a5b0 0x7fc8b419aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:53.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:53.423+0000 7fc8ba600700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fc8b4114840 con 0x7fc8b410a5b0 2024-03-18T16:07:53.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:53.423+0000 7fc8ba600700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fc8b41146c0 con 0x7fc8b4104ec0 2024-03-18T16:07:53.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:53.423+0000 7fc8ba600700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fc8b41149c0 con 0x7fc8b4104500 2024-03-18T16:07:53.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:53.423+0000 7fc8b3400700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc8b4104500 0x7fc8b41a0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:53.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:53.423+0000 7fc8b2a00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc8b4104ec0 0x7fc8b41a0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:53.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:53.423+0000 7fc8b3400700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc8b4104500 0x7fc8b41a0810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:49722/0 (socket says 172.21.15.39:49722) 2024-03-18T16:07:53.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:53.423+0000 7fc8b2a00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc8b4104ec0 0x7fc8b41a0d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:49396/0 (socket says 172.21.15.39:49396) 2024-03-18T16:07:53.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:53.423+0000 7fc8b3400700 1 -- 172.21.15.39:0/195278966 learned_addr learned my addr 172.21.15.39:0/195278966 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:07:53.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:53.424+0000 7fc8b2a00700 1 -- 172.21.15.39:0/195278966 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc8b4104500 msgr2=0x7fc8b41a0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:53.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:53.424+0000 7fc8b2a00700 1 --2- 172.21.15.39:0/195278966 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc8b4104500 0x7fc8b41a0810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:53.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:53.424+0000 7fc8b2a00700 1 -- 172.21.15.39:0/195278966 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc8b410a5b0 msgr2=0x7fc8b419aa20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T16:07:53.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:53.424+0000 7fc8b2a00700 1 --2- 172.21.15.39:0/195278966 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc8b410a5b0 0x7fc8b419aa20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:53.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:53.424+0000 7fc8b2a00700 1 -- 172.21.15.39:0/195278966 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8b419b180 con 0x7fc8b4104ec0 2024-03-18T16:07:53.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:53.424+0000 7fc8b2a00700 1 --2- 172.21.15.39:0/195278966 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc8b4104ec0 0x7fc8b41a0d50 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc8a000e9f0 tx=0x7fc8a000ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:53.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:53.424+0000 7fc8abe00700 1 -- 172.21.15.39:0/195278966 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8a000cb70 con 0x7fc8b4104ec0 2024-03-18T16:07:53.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:53.424+0000 7fc8ba600700 1 -- 172.21.15.39:0/195278966 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8b419b470 con 0x7fc8b4104ec0 2024-03-18T16:07:53.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:53.424+0000 7fc8abe00700 1 -- 172.21.15.39:0/195278966 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc8a0004500 con 0x7fc8b4104ec0 2024-03-18T16:07:53.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:53.424+0000 7fc8abe00700 1 -- 172.21.15.39:0/195278966 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8a0010430 con 0x7fc8b4104ec0 2024-03-18T16:07:53.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:53.425+0000 7fc8ba600700 1 -- 172.21.15.39:0/195278966 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8b41a76e0 con 0x7fc8b4104ec0 2024-03-18T16:07:53.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:53.426+0000 7fc8ba600700 1 -- 172.21.15.39:0/195278966 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc8b4062a60 con 0x7fc8b4104ec0 2024-03-18T16:07:53.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:53.428+0000 7fc8abe00700 1 -- 172.21.15.39:0/195278966 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fc8a0003ed0 con 0x7fc8b4104ec0 2024-03-18T16:07:53.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:53.430+0000 7fc8abe00700 1 --2- 172.21.15.39:0/195278966 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fc8980a29a0 0x7fc8980a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:53.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:53.430+0000 7fc8abe00700 1 -- 172.21.15.39:0/195278966 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fc8a00cf2e0 con 0x7fc8b4104ec0 2024-03-18T16:07:53.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:53.430+0000 7fc8b3400700 1 --2- 172.21.15.39:0/195278966 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fc8980a29a0 0x7fc8980a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:53.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:53.430+0000 7fc8abe00700 1 --2- 172.21.15.39:0/195278966 >> [v2:172.21.15.194:6816/3299162444,v1:172.21.15.194:6817/3299162444] conn(0x7fc8980adb00 0x7fc8980affa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:53.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:53.430+0000 7fc8abe00700 1 -- 172.21.15.39:0/195278966 --> [v2:172.21.15.194:6816/3299162444,v1:172.21.15.194:6817/3299162444] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc8a001aec0 con 0x7fc8980adb00 2024-03-18T16:07:53.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:53.430+0000 7fc8abe00700 1 -- 172.21.15.39:0/195278966 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7fc8a00cf690 con 0x7fc8b4104ec0 2024-03-18T16:07:53.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:53.431+0000 7fc8b3e00700 1 --2- 172.21.15.39:0/195278966 >> [v2:172.21.15.194:6816/3299162444,v1:172.21.15.194:6817/3299162444] conn(0x7fc8980adb00 0x7fc8980affa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:53.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:53.431+0000 7fc8b3400700 1 --2- 172.21.15.39:0/195278966 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fc8980a29a0 0x7fc8980a4e60 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fc8b41095f0 tx=0x7fc8a4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:53.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:53.431+0000 7fc8b3e00700 1 --2- 172.21.15.39:0/195278966 >> [v2:172.21.15.194:6816/3299162444,v1:172.21.15.194:6817/3299162444] conn(0x7fc8980adb00 0x7fc8980affa0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:53.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:53.432+0000 7fc8abe00700 1 -- 172.21.15.39:0/195278966 <== osd.7 v2:172.21.15.194:6816/3299162444 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc8a001aec0 con 0x7fc8980adb00 2024-03-18T16:07:53.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:53.451+0000 7fc8ba600700 1 -- 172.21.15.39:0/195278966 --> [v2:172.21.15.194:6816/3299162444,v1:172.21.15.194:6817/3299162444] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc8b4062e60 con 0x7fc8980adb00 2024-03-18T16:07:53.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:53.451+0000 7fc8abe00700 1 -- 172.21.15.39:0/195278966 <== osd.7 v2:172.21.15.194:6816/3299162444 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fc8b4062e60 con 0x7fc8980adb00 2024-03-18T16:07:53.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:53.452+0000 7fc8ba600700 1 -- 172.21.15.39:0/195278966 >> [v2:172.21.15.194:6816/3299162444,v1:172.21.15.194:6817/3299162444] conn(0x7fc8980adb00 msgr2=0x7fc8980affa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:53.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:53.452+0000 7fc8ba600700 1 --2- 172.21.15.39:0/195278966 >> [v2:172.21.15.194:6816/3299162444,v1:172.21.15.194:6817/3299162444] conn(0x7fc8980adb00 0x7fc8980affa0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:53.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:53.452+0000 7fc8ba600700 1 -- 172.21.15.39:0/195278966 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fc8980a29a0 msgr2=0x7fc8980a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:53.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:53.452+0000 7fc8ba600700 1 --2- 172.21.15.39:0/195278966 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fc8980a29a0 0x7fc8980a4e60 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fc8b41095f0 tx=0x7fc8a4009260 comp rx=0 tx=0).stop 2024-03-18T16:07:53.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:53.452+0000 7fc8ba600700 1 -- 172.21.15.39:0/195278966 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc8b4104ec0 msgr2=0x7fc8b41a0d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:53.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:53.452+0000 7fc8ba600700 1 --2- 172.21.15.39:0/195278966 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc8b4104ec0 0x7fc8b41a0d50 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc8a000e9f0 tx=0x7fc8a000ed00 comp rx=0 tx=0).stop 2024-03-18T16:07:53.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:53.452+0000 7fc8b3e00700 1 -- 172.21.15.39:0/195278966 reap_dead start 2024-03-18T16:07:53.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:53.452+0000 7fc8ba600700 1 -- 172.21.15.39:0/195278966 shutdown_connections 2024-03-18T16:07:53.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:53.452+0000 7fc8ba600700 1 -- 172.21.15.39:0/195278966 >> 172.21.15.39:0/195278966 conn(0x7fc8b40fffd0 msgr2=0x7fc8b4101590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:53.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:53.452+0000 7fc8ba600700 1 -- 172.21.15.39:0/195278966 shutdown_connections 2024-03-18T16:07:53.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:53.453+0000 7fc8ba600700 1 -- 172.21.15.39:0/195278966 wait complete. 2024-03-18T16:07:53.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:53 smithi196 ceph-mon[36307]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-03-18T16:07:53.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:53 smithi196 ceph-mon[36307]: Cluster is now healthy 2024-03-18T16:07:53.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:53 smithi194 ceph-mon[36200]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-03-18T16:07:53.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:53 smithi194 ceph-mon[36200]: Cluster is now healthy 2024-03-18T16:07:54.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:53 smithi039 ceph-mon[31721]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-03-18T16:07:54.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:53 smithi039 ceph-mon[31721]: Cluster is now healthy 2024-03-18T16:07:54.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:54 smithi196 ceph-mon[36307]: pgmap v355: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 9 op/s; 14 B/s, 0 objects/s recovering 2024-03-18T16:07:54.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:54 smithi194 ceph-mon[36200]: pgmap v355: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 9 op/s; 14 B/s, 0 objects/s recovering 2024-03-18T16:07:55.076 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:54 smithi039 ceph-mon[31721]: pgmap v355: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 9 op/s; 14 B/s, 0 objects/s recovering 2024-03-18T16:07:55.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:55.774+0000 7f7941000700 1 -- 172.21.15.39:0/636745804 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f793c102f30 msgr2=0x7f793c106f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:55.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:55.774+0000 7f7941000700 1 --2- 172.21.15.39:0/636745804 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f793c102f30 0x7f793c106f70 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f79280099c0 tx=0x7f7928009cd0 comp rx=0 tx=0).stop 2024-03-18T16:07:55.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:55.774+0000 7f7941000700 1 -- 172.21.15.39:0/636745804 shutdown_connections 2024-03-18T16:07:55.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:55.774+0000 7f7941000700 1 --2- 172.21.15.39:0/636745804 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f793c107650 0x7f793c10f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:55.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:55.774+0000 7f7941000700 1 --2- 172.21.15.39:0/636745804 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f793c102580 0x7f793c102960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:55.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:55.774+0000 7f7941000700 1 --2- 172.21.15.39:0/636745804 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f793c102f30 0x7f793c106f70 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:55.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:55.774+0000 7f7941000700 1 -- 172.21.15.39:0/636745804 >> 172.21.15.39:0/636745804 conn(0x7f793c079430 msgr2=0x7f793c079840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:55.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:55.775+0000 7f7941000700 1 -- 172.21.15.39:0/636745804 shutdown_connections 2024-03-18T16:07:55.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:55.775+0000 7f7941000700 1 -- 172.21.15.39:0/636745804 wait complete. 2024-03-18T16:07:55.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:55.776+0000 7f7941000700 1 Processor -- start 2024-03-18T16:07:55.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:55.776+0000 7f7941000700 1 -- start start 2024-03-18T16:07:55.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:55.777+0000 7f7941000700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f793c102580 0x7f793c0745d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:55.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:55.777+0000 7f7941000700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f793c102f30 0x7f793c074b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:55.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:55.777+0000 7f7941000700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f793c107650 0x7f793c075050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:55.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:55.777+0000 7f7941000700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f793c112660 con 0x7f793c107650 2024-03-18T16:07:55.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:55.777+0000 7f7941000700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f793c1124e0 con 0x7f793c102f30 2024-03-18T16:07:55.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:55.777+0000 7f7941000700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f793c1127e0 con 0x7f793c102580 2024-03-18T16:07:55.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:55.777+0000 7f7939e00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f793c102580 0x7f793c0745d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:55.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:55.777+0000 7f7939e00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f793c102580 0x7f793c0745d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:49744/0 (socket says 172.21.15.39:49744) 2024-03-18T16:07:55.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:55.777+0000 7f7939e00700 1 -- 172.21.15.39:0/1786066999 learned_addr learned my addr 172.21.15.39:0/1786066999 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:07:55.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:55.777+0000 7f7939400700 1 --2- 172.21.15.39:0/1786066999 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f793c102f30 0x7f793c074b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:55.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:55.777+0000 7f793a800700 1 --2- 172.21.15.39:0/1786066999 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f793c107650 0x7f793c075050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:55.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:55.778+0000 7f7939e00700 1 -- 172.21.15.39:0/1786066999 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f793c102f30 msgr2=0x7f793c074b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:55.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:55.778+0000 7f7939e00700 1 --2- 172.21.15.39:0/1786066999 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f793c102f30 0x7f793c074b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:55.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:55.778+0000 7f7939e00700 1 -- 172.21.15.39:0/1786066999 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f793c107650 msgr2=0x7f793c075050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:55.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:55.778+0000 7f7939e00700 1 --2- 172.21.15.39:0/1786066999 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f793c107650 0x7f793c075050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:55.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:55.778+0000 7f7939e00700 1 -- 172.21.15.39:0/1786066999 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f793c1a7080 con 0x7f793c102580 2024-03-18T16:07:55.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:55.778+0000 7f7939400700 1 --2- 172.21.15.39:0/1786066999 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f793c102f30 0x7f793c074b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:07:55.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:55.778+0000 7f7939e00700 1 --2- 172.21.15.39:0/1786066999 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f793c102580 0x7f793c0745d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f792c00eb30 tx=0x7f792c00ee40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:55.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:55.778+0000 7f7932a00700 1 -- 172.21.15.39:0/1786066999 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f792c00cbd0 con 0x7f793c102580 2024-03-18T16:07:55.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:55.778+0000 7f7932a00700 1 -- 172.21.15.39:0/1786066999 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f792c004500 con 0x7f793c102580 2024-03-18T16:07:55.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:55.778+0000 7f7932a00700 1 -- 172.21.15.39:0/1786066999 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f792c010430 con 0x7f793c102580 2024-03-18T16:07:55.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:55.778+0000 7f7941000700 1 -- 172.21.15.39:0/1786066999 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f793c1a7370 con 0x7f793c102580 2024-03-18T16:07:55.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:55.779+0000 7f7941000700 1 -- 172.21.15.39:0/1786066999 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f793c1a7830 con 0x7f793c102580 2024-03-18T16:07:55.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:55.779+0000 7f7941000700 1 -- 172.21.15.39:0/1786066999 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f793c0629e0 con 0x7f793c102580 2024-03-18T16:07:55.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:55.781+0000 7f7932a00700 1 -- 172.21.15.39:0/1786066999 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f792c003f00 con 0x7f793c102580 2024-03-18T16:07:55.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:55.783+0000 7f7932a00700 1 --2- 172.21.15.39:0/1786066999 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f79200a29a0 0x7f79200a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:55.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:55.783+0000 7f7932a00700 1 -- 172.21.15.39:0/1786066999 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f792c0cf260 con 0x7f793c102580 2024-03-18T16:07:55.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:55.783+0000 7f7939400700 1 --2- 172.21.15.39:0/1786066999 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f79200a29a0 0x7f79200a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:55.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:55.783+0000 7f7932a00700 1 --2- 172.21.15.39:0/1786066999 >> [v2:172.21.15.196:6800/3935874719,v1:172.21.15.196:6801/3935874719] conn(0x7f79200adb00 0x7f79200affa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:55.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:55.783+0000 7f7932a00700 1 -- 172.21.15.39:0/1786066999 --> [v2:172.21.15.196:6800/3935874719,v1:172.21.15.196:6801/3935874719] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f792c01ee50 con 0x7f79200adb00 2024-03-18T16:07:55.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:55.783+0000 7f7932a00700 1 -- 172.21.15.39:0/1786066999 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f792c0d6050 con 0x7f793c102580 2024-03-18T16:07:55.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:55.784+0000 7f793a800700 1 --2- 172.21.15.39:0/1786066999 >> [v2:172.21.15.196:6800/3935874719,v1:172.21.15.196:6801/3935874719] conn(0x7f79200adb00 0x7f79200affa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:55.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:55.784+0000 7f7939400700 1 --2- 172.21.15.39:0/1786066999 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f79200a29a0 0x7f79200a4e60 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f7928000c00 tx=0x7f7928005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:55.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:55.784+0000 7f793a800700 1 --2- 172.21.15.39:0/1786066999 >> [v2:172.21.15.196:6800/3935874719,v1:172.21.15.196:6801/3935874719] conn(0x7f79200adb00 0x7f79200affa0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:55.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:55.785+0000 7f7932a00700 1 -- 172.21.15.39:0/1786066999 <== osd.2 v2:172.21.15.196:6800/3935874719 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f792c01ee50 con 0x7f79200adb00 2024-03-18T16:07:55.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:55.803+0000 7f7941000700 1 -- 172.21.15.39:0/1786066999 --> [v2:172.21.15.196:6800/3935874719,v1:172.21.15.196:6801/3935874719] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f793c062e60 con 0x7f79200adb00 2024-03-18T16:07:55.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:55.804+0000 7f7932a00700 1 -- 172.21.15.39:0/1786066999 <== osd.2 v2:172.21.15.196:6800/3935874719 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f793c062e60 con 0x7f79200adb00 2024-03-18T16:07:55.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:55.804+0000 7f7941000700 1 -- 172.21.15.39:0/1786066999 >> [v2:172.21.15.196:6800/3935874719,v1:172.21.15.196:6801/3935874719] conn(0x7f79200adb00 msgr2=0x7f79200affa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:55.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:55.804+0000 7f7941000700 1 --2- 172.21.15.39:0/1786066999 >> [v2:172.21.15.196:6800/3935874719,v1:172.21.15.196:6801/3935874719] conn(0x7f79200adb00 0x7f79200affa0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:55.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:55.804+0000 7f7941000700 1 -- 172.21.15.39:0/1786066999 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f79200a29a0 msgr2=0x7f79200a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:55.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:55.804+0000 7f7941000700 1 --2- 172.21.15.39:0/1786066999 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f79200a29a0 0x7f79200a4e60 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f7928000c00 tx=0x7f7928005fb0 comp rx=0 tx=0).stop 2024-03-18T16:07:55.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:55.804+0000 7f7941000700 1 -- 172.21.15.39:0/1786066999 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f793c102580 msgr2=0x7f793c0745d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:55.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:55.804+0000 7f7941000700 1 --2- 172.21.15.39:0/1786066999 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f793c102580 0x7f793c0745d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f792c00eb30 tx=0x7f792c00ee40 comp rx=0 tx=0).stop 2024-03-18T16:07:55.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:55.804+0000 7f793a800700 1 -- 172.21.15.39:0/1786066999 reap_dead start 2024-03-18T16:07:55.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:55.804+0000 7f7941000700 1 -- 172.21.15.39:0/1786066999 shutdown_connections 2024-03-18T16:07:55.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:55.804+0000 7f7941000700 1 -- 172.21.15.39:0/1786066999 >> 172.21.15.39:0/1786066999 conn(0x7f793c079430 msgr2=0x7f793c100370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:55.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:55.805+0000 7f7941000700 1 -- 172.21.15.39:0/1786066999 shutdown_connections 2024-03-18T16:07:55.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:55.805+0000 7f7941000700 1 -- 172.21.15.39:0/1786066999 wait complete. 2024-03-18T16:07:56.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:56 smithi196 ceph-mon[36307]: pgmap v356: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 10 op/s; 12 B/s, 0 objects/s recovering 2024-03-18T16:07:56.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:56 smithi194 ceph-mon[36200]: pgmap v356: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 10 op/s; 12 B/s, 0 objects/s recovering 2024-03-18T16:07:57.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:56 smithi039 ceph-mon[31721]: pgmap v356: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 10 op/s; 12 B/s, 0 objects/s recovering 2024-03-18T16:07:58.875 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:07:58 smithi039 ceph-mon[31721]: pgmap v357: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 10 op/s; 11 B/s, 0 objects/s recovering 2024-03-18T16:07:58.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:07:58 smithi196 ceph-mon[36307]: pgmap v357: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 10 op/s; 11 B/s, 0 objects/s recovering 2024-03-18T16:07:58.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:07:58 smithi194 ceph-mon[36200]: pgmap v357: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 10 op/s; 11 B/s, 0 objects/s recovering 2024-03-18T16:07:59.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.942+0000 7fb618c00700 1 -- 172.21.15.39:0/3051383478 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb614074500 msgr2=0x7fb6140748e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:59.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.942+0000 7fb618c00700 1 --2- 172.21.15.39:0/3051383478 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb614074500 0x7fb6140748e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fb604009990 tx=0x7fb604009ca0 comp rx=0 tx=0).stop 2024-03-18T16:07:59.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.943+0000 7fb618c00700 1 -- 172.21.15.39:0/3051383478 shutdown_connections 2024-03-18T16:07:59.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.943+0000 7fb618c00700 1 --2- 172.21.15.39:0/3051383478 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb6141134a0 0x7fb614115890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:59.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.943+0000 7fb618c00700 1 --2- 172.21.15.39:0/3051383478 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb614074500 0x7fb6140748e0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:59.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.943+0000 7fb618c00700 1 --2- 172.21.15.39:0/3051383478 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb614074e20 0x7fb614112f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:59.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.943+0000 7fb618c00700 1 -- 172.21.15.39:0/3051383478 >> 172.21.15.39:0/3051383478 conn(0x7fb614079430 msgr2=0x7fb614079840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:59.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.943+0000 7fb618c00700 1 -- 172.21.15.39:0/3051383478 shutdown_connections 2024-03-18T16:07:59.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.943+0000 7fb618c00700 1 -- 172.21.15.39:0/3051383478 wait complete. 2024-03-18T16:07:59.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.944+0000 7fb618c00700 1 Processor -- start 2024-03-18T16:07:59.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.944+0000 7fb618c00700 1 -- start start 2024-03-18T16:07:59.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.944+0000 7fb618c00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb614074500 0x7fb61419f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:59.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.944+0000 7fb618c00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb614074e20 0x7fb61419f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:59.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.944+0000 7fb618c00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb6141134a0 0x7fb6141a39d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:59.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.944+0000 7fb618c00700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fb614118cf0 con 0x7fb614074500 2024-03-18T16:07:59.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.944+0000 7fb618c00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fb614118b70 con 0x7fb614074e20 2024-03-18T16:07:59.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.944+0000 7fb618c00700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fb614118e70 con 0x7fb6141134a0 2024-03-18T16:07:59.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.945+0000 7fb611400700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb614074e20 0x7fb61419f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:59.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.945+0000 7fb611400700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb614074e20 0x7fb61419f640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:44716/0 (socket says 172.21.15.39:44716) 2024-03-18T16:07:59.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.945+0000 7fb611e00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb614074500 0x7fb61419f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:59.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.945+0000 7fb611400700 1 -- 172.21.15.39:0/3700398956 learned_addr learned my addr 172.21.15.39:0/3700398956 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:07:59.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.945+0000 7fb612800700 1 --2- 172.21.15.39:0/3700398956 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb6141134a0 0x7fb6141a39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:59.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.945+0000 7fb611400700 1 -- 172.21.15.39:0/3700398956 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb6141134a0 msgr2=0x7fb6141a39d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:59.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.945+0000 7fb611400700 1 --2- 172.21.15.39:0/3700398956 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb6141134a0 0x7fb6141a39d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:59.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.945+0000 7fb611400700 1 -- 172.21.15.39:0/3700398956 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb614074500 msgr2=0x7fb61419f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:59.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.945+0000 7fb611400700 1 --2- 172.21.15.39:0/3700398956 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb614074500 0x7fb61419f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:59.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.945+0000 7fb611400700 1 -- 172.21.15.39:0/3700398956 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6141a4110 con 0x7fb614074e20 2024-03-18T16:07:59.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.945+0000 7fb612800700 1 --2- 172.21.15.39:0/3700398956 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb6141134a0 0x7fb6141a39d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:07:59.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.945+0000 7fb611400700 1 --2- 172.21.15.39:0/3700398956 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb614074e20 0x7fb61419f640 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fb60000e950 tx=0x7fb60000ec60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:59.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.946+0000 7fb60aa00700 1 -- 172.21.15.39:0/3700398956 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb60000eea0 con 0x7fb614074e20 2024-03-18T16:07:59.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.946+0000 7fb60aa00700 1 -- 172.21.15.39:0/3700398956 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb600004510 con 0x7fb614074e20 2024-03-18T16:07:59.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.946+0000 7fb618c00700 1 -- 172.21.15.39:0/3700398956 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6141a4400 con 0x7fb614074e20 2024-03-18T16:07:59.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.946+0000 7fb60aa00700 1 -- 172.21.15.39:0/3700398956 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb600010430 con 0x7fb614074e20 2024-03-18T16:07:59.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.946+0000 7fb618c00700 1 -- 172.21.15.39:0/3700398956 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6141abc80 con 0x7fb614074e20 2024-03-18T16:07:59.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.947+0000 7fb618c00700 1 -- 172.21.15.39:0/3700398956 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb614112c40 con 0x7fb614074e20 2024-03-18T16:07:59.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.949+0000 7fb60aa00700 1 -- 172.21.15.39:0/3700398956 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fb600003eb0 con 0x7fb614074e20 2024-03-18T16:07:59.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.950+0000 7fb60aa00700 1 --2- 172.21.15.39:0/3700398956 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb5f80a29a0 0x7fb5f80a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:59.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.950+0000 7fb60aa00700 1 -- 172.21.15.39:0/3700398956 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fb6000cf260 con 0x7fb614074e20 2024-03-18T16:07:59.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.950+0000 7fb611e00700 1 --2- 172.21.15.39:0/3700398956 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb5f80a29a0 0x7fb5f80a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:59.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.951+0000 7fb60aa00700 1 --2- 172.21.15.39:0/3700398956 >> [v2:172.21.15.196:6816/2456143145,v1:172.21.15.196:6817/2456143145] conn(0x7fb5f80adb00 0x7fb5f80affa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:59.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.951+0000 7fb60aa00700 1 -- 172.21.15.39:0/3700398956 --> [v2:172.21.15.196:6816/2456143145,v1:172.21.15.196:6817/2456143145] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb5f80b0670 con 0x7fb5f80adb00 2024-03-18T16:07:59.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.951+0000 7fb60aa00700 1 -- 172.21.15.39:0/3700398956 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7fb6000d6050 con 0x7fb614074e20 2024-03-18T16:07:59.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.951+0000 7fb612800700 1 --2- 172.21.15.39:0/3700398956 >> [v2:172.21.15.196:6816/2456143145,v1:172.21.15.196:6817/2456143145] conn(0x7fb5f80adb00 0x7fb5f80affa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:59.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.951+0000 7fb611e00700 1 --2- 172.21.15.39:0/3700398956 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb5f80a29a0 0x7fb5f80a4e60 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb60400b2e0 tx=0x7fb604005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:59.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.951+0000 7fb612800700 1 --2- 172.21.15.39:0/3700398956 >> [v2:172.21.15.196:6816/2456143145,v1:172.21.15.196:6817/2456143145] conn(0x7fb5f80adb00 0x7fb5f80affa0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:59.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.952+0000 7fb60aa00700 1 -- 172.21.15.39:0/3700398956 <== osd.8 v2:172.21.15.196:6816/2456143145 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb5f80b0670 con 0x7fb5f80adb00 2024-03-18T16:07:59.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.971+0000 7fb618c00700 1 -- 172.21.15.39:0/3700398956 --> [v2:172.21.15.196:6816/2456143145,v1:172.21.15.196:6817/2456143145] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb614075c30 con 0x7fb5f80adb00 2024-03-18T16:07:59.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.971+0000 7fb60aa00700 1 -- 172.21.15.39:0/3700398956 <== osd.8 v2:172.21.15.196:6816/2456143145 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fb614075c30 con 0x7fb5f80adb00 2024-03-18T16:07:59.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.972+0000 7fb618c00700 1 -- 172.21.15.39:0/3700398956 >> [v2:172.21.15.196:6816/2456143145,v1:172.21.15.196:6817/2456143145] conn(0x7fb5f80adb00 msgr2=0x7fb5f80affa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:59.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.972+0000 7fb618c00700 1 --2- 172.21.15.39:0/3700398956 >> [v2:172.21.15.196:6816/2456143145,v1:172.21.15.196:6817/2456143145] conn(0x7fb5f80adb00 0x7fb5f80affa0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:59.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.972+0000 7fb618c00700 1 -- 172.21.15.39:0/3700398956 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb5f80a29a0 msgr2=0x7fb5f80a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:59.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.972+0000 7fb618c00700 1 --2- 172.21.15.39:0/3700398956 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb5f80a29a0 0x7fb5f80a4e60 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb60400b2e0 tx=0x7fb604005fd0 comp rx=0 tx=0).stop 2024-03-18T16:07:59.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.972+0000 7fb618c00700 1 -- 172.21.15.39:0/3700398956 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb614074e20 msgr2=0x7fb61419f640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:59.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.972+0000 7fb618c00700 1 --2- 172.21.15.39:0/3700398956 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb614074e20 0x7fb61419f640 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fb60000e950 tx=0x7fb60000ec60 comp rx=0 tx=0).stop 2024-03-18T16:07:59.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.972+0000 7fb612800700 1 -- 172.21.15.39:0/3700398956 reap_dead start 2024-03-18T16:07:59.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.972+0000 7fb618c00700 1 -- 172.21.15.39:0/3700398956 shutdown_connections 2024-03-18T16:07:59.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.972+0000 7fb618c00700 1 -- 172.21.15.39:0/3700398956 >> 172.21.15.39:0/3700398956 conn(0x7fb614079430 msgr2=0x7fb614112260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:59.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.973+0000 7fb618c00700 1 -- 172.21.15.39:0/3700398956 shutdown_connections 2024-03-18T16:07:59.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.973+0000 7fb618c00700 1 -- 172.21.15.39:0/3700398956 wait complete. 2024-03-18T16:07:59.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.324+0000 7f144cc00700 1 -- 172.21.15.39:0/2234036099 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1448102320 msgr2=0x7f14481027a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:59.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.324+0000 7f144cc00700 1 --2- 172.21.15.39:0/2234036099 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1448102320 0x7f14481027a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f14340098f0 tx=0x7f1434009c00 comp rx=0 tx=0).stop 2024-03-18T16:07:59.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.325+0000 7f144cc00700 1 -- 172.21.15.39:0/2234036099 shutdown_connections 2024-03-18T16:07:59.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.325+0000 7f144cc00700 1 --2- 172.21.15.39:0/2234036099 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1448102ce0 0x7f144810f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:59.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.325+0000 7f144cc00700 1 --2- 172.21.15.39:0/2234036099 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1448108300 0x7f14481086e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:59.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.325+0000 7f144cc00700 1 --2- 172.21.15.39:0/2234036099 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1448102320 0x7f14481027a0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:59.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.325+0000 7f144cc00700 1 -- 172.21.15.39:0/2234036099 >> 172.21.15.39:0/2234036099 conn(0x7f1448079400 msgr2=0x7f1448079810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:59.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.325+0000 7f144cc00700 1 -- 172.21.15.39:0/2234036099 shutdown_connections 2024-03-18T16:07:59.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.325+0000 7f144cc00700 1 -- 172.21.15.39:0/2234036099 wait complete. 2024-03-18T16:07:59.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.326+0000 7f144cc00700 1 Processor -- start 2024-03-18T16:07:59.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.326+0000 7f144cc00700 1 -- start start 2024-03-18T16:07:59.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.327+0000 7f144cc00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1448102320 0x7f144819e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:59.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.327+0000 7f144cc00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1448102ce0 0x7f144819ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:59.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.327+0000 7f144cc00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1448108300 0x7f14481988b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:59.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.327+0000 7f144cc00700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f14481125d0 con 0x7f1448108300 2024-03-18T16:07:59.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.327+0000 7f144cc00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f1448112450 con 0x7f1448102320 2024-03-18T16:07:59.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.327+0000 7f144cc00700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f1448112750 con 0x7f1448102ce0 2024-03-18T16:07:59.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.327+0000 7f1445e00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1448102320 0x7f144819e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:59.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.327+0000 7f1445e00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1448102320 0x7f144819e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:44700/0 (socket says 172.21.15.39:44700) 2024-03-18T16:07:59.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.327+0000 7f1445e00700 1 -- 172.21.15.39:0/243555102 learned_addr learned my addr 172.21.15.39:0/243555102 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:07:59.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.327+0000 7f1446800700 1 --2- 172.21.15.39:0/243555102 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1448108300 0x7f14481988b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:59.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.327+0000 7f1445400700 1 --2- 172.21.15.39:0/243555102 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1448102ce0 0x7f144819ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:59.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.327+0000 7f1445e00700 1 -- 172.21.15.39:0/243555102 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1448102ce0 msgr2=0x7f144819ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:59.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.327+0000 7f1445e00700 1 --2- 172.21.15.39:0/243555102 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1448102ce0 0x7f144819ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:59.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.327+0000 7f1445e00700 1 -- 172.21.15.39:0/243555102 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1448108300 msgr2=0x7f14481988b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:59.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.327+0000 7f1445e00700 1 --2- 172.21.15.39:0/243555102 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1448108300 0x7f14481988b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:59.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.327+0000 7f1445e00700 1 -- 172.21.15.39:0/243555102 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1448199010 con 0x7f1448102320 2024-03-18T16:07:59.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.327+0000 7f1446800700 1 --2- 172.21.15.39:0/243555102 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1448108300 0x7f14481988b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:07:59.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.328+0000 7f1445e00700 1 --2- 172.21.15.39:0/243555102 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1448102320 0x7f144819e6a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f143800e9c0 tx=0x7f143800ed80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:59.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.328+0000 7f143ea00700 1 -- 172.21.15.39:0/243555102 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f143800cbd0 con 0x7f1448102320 2024-03-18T16:07:59.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.328+0000 7f143ea00700 1 -- 172.21.15.39:0/243555102 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f143800cd70 con 0x7f1448102320 2024-03-18T16:07:59.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.328+0000 7f144cc00700 1 -- 172.21.15.39:0/243555102 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1448199300 con 0x7f1448102320 2024-03-18T16:07:59.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.328+0000 7f143ea00700 1 -- 172.21.15.39:0/243555102 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1438010430 con 0x7f1448102320 2024-03-18T16:07:59.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.328+0000 7f144cc00700 1 -- 172.21.15.39:0/243555102 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14481a5570 con 0x7f1448102320 2024-03-18T16:07:59.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.329+0000 7f144cc00700 1 -- 172.21.15.39:0/243555102 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1448062a60 con 0x7f1448102320 2024-03-18T16:07:59.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.331+0000 7f143ea00700 1 -- 172.21.15.39:0/243555102 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f1438004da0 con 0x7f1448102320 2024-03-18T16:07:59.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.332+0000 7f143ea00700 1 --2- 172.21.15.39:0/243555102 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f142c0a29a0 0x7f142c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:59.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.332+0000 7f143ea00700 1 -- 172.21.15.39:0/243555102 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f14380cf2d0 con 0x7f1448102320 2024-03-18T16:07:59.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.332+0000 7f1445400700 1 --2- 172.21.15.39:0/243555102 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f142c0a29a0 0x7f142c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:59.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.333+0000 7f143ea00700 1 --2- 172.21.15.39:0/243555102 >> [v2:172.21.15.194:6808/800423088,v1:172.21.15.194:6809/800423088] conn(0x7f142c0adb00 0x7f142c0affa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:07:59.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.333+0000 7f143ea00700 1 -- 172.21.15.39:0/243555102 --> [v2:172.21.15.194:6808/800423088,v1:172.21.15.194:6809/800423088] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f142c0b0670 con 0x7f142c0adb00 2024-03-18T16:07:59.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.333+0000 7f143ea00700 1 -- 172.21.15.39:0/243555102 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f14380cf680 con 0x7f1448102320 2024-03-18T16:07:59.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.333+0000 7f1446800700 1 --2- 172.21.15.39:0/243555102 >> [v2:172.21.15.194:6808/800423088,v1:172.21.15.194:6809/800423088] conn(0x7f142c0adb00 0x7f142c0affa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:07:59.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.333+0000 7f1446800700 1 --2- 172.21.15.39:0/243555102 >> [v2:172.21.15.194:6808/800423088,v1:172.21.15.194:6809/800423088] conn(0x7f142c0adb00 0x7f142c0affa0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:59.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.333+0000 7f1445400700 1 --2- 172.21.15.39:0/243555102 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f142c0a29a0 0x7f142c0a4e60 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f1434009810 tx=0x7f1434019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:07:59.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.334+0000 7f143ea00700 1 -- 172.21.15.39:0/243555102 <== osd.4 v2:172.21.15.194:6808/800423088 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f142c0b0670 con 0x7f142c0adb00 2024-03-18T16:07:59.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.355+0000 7f144cc00700 1 -- 172.21.15.39:0/243555102 --> [v2:172.21.15.194:6808/800423088,v1:172.21.15.194:6809/800423088] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1448062ee0 con 0x7f142c0adb00 2024-03-18T16:07:59.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.356+0000 7f143ea00700 1 -- 172.21.15.39:0/243555102 <== osd.4 v2:172.21.15.194:6808/800423088 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f1448062ee0 con 0x7f142c0adb00 2024-03-18T16:07:59.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.356+0000 7f144cc00700 1 -- 172.21.15.39:0/243555102 >> [v2:172.21.15.194:6808/800423088,v1:172.21.15.194:6809/800423088] conn(0x7f142c0adb00 msgr2=0x7f142c0affa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:59.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.356+0000 7f144cc00700 1 --2- 172.21.15.39:0/243555102 >> [v2:172.21.15.194:6808/800423088,v1:172.21.15.194:6809/800423088] conn(0x7f142c0adb00 0x7f142c0affa0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:07:59.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.357+0000 7f144cc00700 1 -- 172.21.15.39:0/243555102 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f142c0a29a0 msgr2=0x7f142c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:59.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.357+0000 7f144cc00700 1 --2- 172.21.15.39:0/243555102 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f142c0a29a0 0x7f142c0a4e60 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f1434009810 tx=0x7f1434019040 comp rx=0 tx=0).stop 2024-03-18T16:07:59.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.357+0000 7f144cc00700 1 -- 172.21.15.39:0/243555102 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1448102320 msgr2=0x7f144819e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:07:59.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.357+0000 7f144cc00700 1 --2- 172.21.15.39:0/243555102 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1448102320 0x7f144819e6a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f143800e9c0 tx=0x7f143800ed80 comp rx=0 tx=0).stop 2024-03-18T16:07:59.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.357+0000 7f1446800700 1 -- 172.21.15.39:0/243555102 reap_dead start 2024-03-18T16:07:59.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.357+0000 7f144cc00700 1 -- 172.21.15.39:0/243555102 shutdown_connections 2024-03-18T16:07:59.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.357+0000 7f144cc00700 1 -- 172.21.15.39:0/243555102 >> 172.21.15.39:0/243555102 conn(0x7f1448079400 msgr2=0x7f1448101010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:07:59.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.357+0000 7f144cc00700 1 -- 172.21.15.39:0/243555102 shutdown_connections 2024-03-18T16:07:59.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:58.357+0000 7f144cc00700 1 -- 172.21.15.39:0/243555102 wait complete. 2024-03-18T16:08:00.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:59.047+0000 7f76fb200700 1 -- 172.21.15.39:0/3595894193 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f76f4102f30 msgr2=0x7f76f4106f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:00.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:59.047+0000 7f76fb200700 1 --2- 172.21.15.39:0/3595894193 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f76f4102f30 0x7f76f4106f70 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f76e0009990 tx=0x7f76e0009ca0 comp rx=0 tx=0).stop 2024-03-18T16:08:00.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:59.047+0000 7f76fb200700 1 -- 172.21.15.39:0/3595894193 shutdown_connections 2024-03-18T16:08:00.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:59.047+0000 7f76fb200700 1 --2- 172.21.15.39:0/3595894193 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f76f4107650 0x7f76f410f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:00.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:59.047+0000 7f76fb200700 1 --2- 172.21.15.39:0/3595894193 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f76f4102580 0x7f76f4102960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:00.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:59.047+0000 7f76fb200700 1 --2- 172.21.15.39:0/3595894193 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f76f4102f30 0x7f76f4106f70 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:00.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:59.047+0000 7f76fb200700 1 -- 172.21.15.39:0/3595894193 >> 172.21.15.39:0/3595894193 conn(0x7f76f4079430 msgr2=0x7f76f4079840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:08:00.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:59.047+0000 7f76fb200700 1 -- 172.21.15.39:0/3595894193 shutdown_connections 2024-03-18T16:08:00.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:59.047+0000 7f76fb200700 1 -- 172.21.15.39:0/3595894193 wait complete. 2024-03-18T16:08:00.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:59.048+0000 7f76fb200700 1 Processor -- start 2024-03-18T16:08:00.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:59.049+0000 7f76fb200700 1 -- start start 2024-03-18T16:08:00.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:59.049+0000 7f76fb200700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f76f4102580 0x7f76f419ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:00.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:59.049+0000 7f76fb200700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f76f4102f30 0x7f76f419b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:00.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:59.049+0000 7f76fb200700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f76f4107650 0x7f76f419f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:00.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:59.049+0000 7f76fb200700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f76f4112660 con 0x7f76f4107650 2024-03-18T16:08:00.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:59.049+0000 7f76fb200700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f76f41124e0 con 0x7f76f4102580 2024-03-18T16:08:00.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:59.049+0000 7f76fb200700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f76f41127e0 con 0x7f76f4102f30 2024-03-18T16:08:00.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:59.049+0000 7f76f3e00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f76f4102580 0x7f76f419ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:00.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:59.049+0000 7f76f3e00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f76f4102580 0x7f76f419ad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:44736/0 (socket says 172.21.15.39:44736) 2024-03-18T16:08:00.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:59.049+0000 7f76f3e00700 1 -- 172.21.15.39:0/2558359212 learned_addr learned my addr 172.21.15.39:0/2558359212 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:08:00.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:59.049+0000 7f76f3400700 1 --2- 172.21.15.39:0/2558359212 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f76f4102f30 0x7f76f419b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:00.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:59.049+0000 7f76f9200700 1 --2- 172.21.15.39:0/2558359212 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f76f4107650 0x7f76f419f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:00.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:59.050+0000 7f76f3e00700 1 -- 172.21.15.39:0/2558359212 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f76f4102f30 msgr2=0x7f76f419b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:00.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:59.050+0000 7f76f3e00700 1 --2- 172.21.15.39:0/2558359212 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f76f4102f30 0x7f76f419b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:00.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:59.050+0000 7f76f3e00700 1 -- 172.21.15.39:0/2558359212 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f76f4107650 msgr2=0x7f76f419f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:00.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:59.050+0000 7f76f3e00700 1 --2- 172.21.15.39:0/2558359212 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f76f4107650 0x7f76f419f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:00.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:59.050+0000 7f76f3e00700 1 -- 172.21.15.39:0/2558359212 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76f419fd30 con 0x7f76f4102580 2024-03-18T16:08:00.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:59.050+0000 7f76f3e00700 1 --2- 172.21.15.39:0/2558359212 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f76f4102580 0x7f76f419ad70 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f76e400e9a0 tx=0x7f76e400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:00.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:59.050+0000 7f76f0c00700 1 -- 172.21.15.39:0/2558359212 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76e400ca10 con 0x7f76f4102580 2024-03-18T16:08:00.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:59.050+0000 7f76f0c00700 1 -- 172.21.15.39:0/2558359212 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76e400cbb0 con 0x7f76f4102580 2024-03-18T16:08:00.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:59.050+0000 7f76f0c00700 1 -- 172.21.15.39:0/2558359212 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76e4018680 con 0x7f76f4102580 2024-03-18T16:08:00.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:59.050+0000 7f76fb200700 1 -- 172.21.15.39:0/2558359212 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76f41a0020 con 0x7f76f4102580 2024-03-18T16:08:00.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:59.050+0000 7f76fb200700 1 -- 172.21.15.39:0/2558359212 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76f41a78a0 con 0x7f76f4102580 2024-03-18T16:08:00.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:59.052+0000 7f76fb200700 1 -- 172.21.15.39:0/2558359212 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f76f4101470 con 0x7f76f4102580 2024-03-18T16:08:00.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:59.053+0000 7f76f0c00700 1 -- 172.21.15.39:0/2558359212 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f76e4010430 con 0x7f76f4102580 2024-03-18T16:08:00.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:59.055+0000 7f76f0c00700 1 --2- 172.21.15.39:0/2558359212 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f76d80a29a0 0x7f76d80a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:00.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:59.055+0000 7f76f0c00700 1 -- 172.21.15.39:0/2558359212 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f76e40cf160 con 0x7f76f4102580 2024-03-18T16:08:00.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:59.055+0000 7f76f3400700 1 --2- 172.21.15.39:0/2558359212 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f76d80a29a0 0x7f76d80a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:00.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:59.056+0000 7f76f0c00700 1 --2- 172.21.15.39:0/2558359212 >> [v2:172.21.15.39:6826/1848634433,v1:172.21.15.39:6827/1848634433] conn(0x7f76d80adb00 0x7f76d80affa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:00.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:59.056+0000 7f76f0c00700 1 -- 172.21.15.39:0/2558359212 --> [v2:172.21.15.39:6826/1848634433,v1:172.21.15.39:6827/1848634433] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f76e4015e80 con 0x7f76d80adb00 2024-03-18T16:08:00.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:59.056+0000 7f76f0c00700 1 -- 172.21.15.39:0/2558359212 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f76e40d6050 con 0x7f76f4102580 2024-03-18T16:08:00.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:59.056+0000 7f76f9200700 1 --2- 172.21.15.39:0/2558359212 >> [v2:172.21.15.39:6826/1848634433,v1:172.21.15.39:6827/1848634433] conn(0x7f76d80adb00 0x7f76d80affa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:00.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:59.056+0000 7f76f3400700 1 --2- 172.21.15.39:0/2558359212 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f76d80a29a0 0x7f76d80a4e60 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f76f4108fc0 tx=0x7f76e000b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:00.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:59.057+0000 7f76f9200700 1 --2- 172.21.15.39:0/2558359212 >> [v2:172.21.15.39:6826/1848634433,v1:172.21.15.39:6827/1848634433] conn(0x7f76d80adb00 0x7f76d80affa0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:00.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:59.058+0000 7f76f0c00700 1 -- 172.21.15.39:0/2558359212 <== osd.9 v2:172.21.15.39:6826/1848634433 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f76e4015e80 con 0x7f76d80adb00 2024-03-18T16:08:00.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:59.076+0000 7f76fb200700 1 -- 172.21.15.39:0/2558359212 --> [v2:172.21.15.39:6826/1848634433,v1:172.21.15.39:6827/1848634433] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f76f4103a50 con 0x7f76d80adb00 2024-03-18T16:08:00.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:59.076+0000 7f76f0c00700 1 -- 172.21.15.39:0/2558359212 <== osd.9 v2:172.21.15.39:6826/1848634433 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f76f4103a50 con 0x7f76d80adb00 2024-03-18T16:08:00.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:59.077+0000 7f76fb200700 1 -- 172.21.15.39:0/2558359212 >> [v2:172.21.15.39:6826/1848634433,v1:172.21.15.39:6827/1848634433] conn(0x7f76d80adb00 msgr2=0x7f76d80affa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:00.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:59.077+0000 7f76fb200700 1 --2- 172.21.15.39:0/2558359212 >> [v2:172.21.15.39:6826/1848634433,v1:172.21.15.39:6827/1848634433] conn(0x7f76d80adb00 0x7f76d80affa0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:00.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:59.077+0000 7f76fb200700 1 -- 172.21.15.39:0/2558359212 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f76d80a29a0 msgr2=0x7f76d80a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:00.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:59.077+0000 7f76fb200700 1 --2- 172.21.15.39:0/2558359212 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f76d80a29a0 0x7f76d80a4e60 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f76f4108fc0 tx=0x7f76e000b260 comp rx=0 tx=0).stop 2024-03-18T16:08:00.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:59.077+0000 7f76fb200700 1 -- 172.21.15.39:0/2558359212 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f76f4102580 msgr2=0x7f76f419ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:00.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:59.077+0000 7f76fb200700 1 --2- 172.21.15.39:0/2558359212 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f76f4102580 0x7f76f419ad70 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f76e400e9a0 tx=0x7f76e400ecb0 comp rx=0 tx=0).stop 2024-03-18T16:08:00.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:59.077+0000 7f76f9200700 1 -- 172.21.15.39:0/2558359212 reap_dead start 2024-03-18T16:08:00.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:59.077+0000 7f76fb200700 1 -- 172.21.15.39:0/2558359212 shutdown_connections 2024-03-18T16:08:00.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:59.077+0000 7f76fb200700 1 -- 172.21.15.39:0/2558359212 >> 172.21.15.39:0/2558359212 conn(0x7f76f4079430 msgr2=0x7f76f410d9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:08:00.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:59.077+0000 7f76fb200700 1 -- 172.21.15.39:0/2558359212 shutdown_connections 2024-03-18T16:08:00.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:07:59.077+0000 7f76fb200700 1 -- 172.21.15.39:0/2558359212 wait complete. 2024-03-18T16:08:00.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:00.776+0000 7f9566600700 1 -- 172.21.15.39:0/3217086750 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9560102320 msgr2=0x7f95601027a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:00.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:00.776+0000 7f9566600700 1 --2- 172.21.15.39:0/3217086750 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9560102320 0x7f95601027a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f9548009990 tx=0x7f9548009ca0 comp rx=0 tx=0).stop 2024-03-18T16:08:00.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:00.776+0000 7f9566600700 1 -- 172.21.15.39:0/3217086750 shutdown_connections 2024-03-18T16:08:00.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:00.776+0000 7f9566600700 1 --2- 172.21.15.39:0/3217086750 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9560102ce0 0x7f956010f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:00.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:00.776+0000 7f9566600700 1 --2- 172.21.15.39:0/3217086750 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9560108300 0x7f95601086e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:00.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:00.776+0000 7f9566600700 1 --2- 172.21.15.39:0/3217086750 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9560102320 0x7f95601027a0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:00.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:00.776+0000 7f9566600700 1 -- 172.21.15.39:0/3217086750 >> 172.21.15.39:0/3217086750 conn(0x7f9560079400 msgr2=0x7f9560079810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:08:00.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:00.776+0000 7f9566600700 1 -- 172.21.15.39:0/3217086750 shutdown_connections 2024-03-18T16:08:00.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:00.776+0000 7f9566600700 1 -- 172.21.15.39:0/3217086750 wait complete. 2024-03-18T16:08:00.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:00.777+0000 7f9566600700 1 Processor -- start 2024-03-18T16:08:00.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:00.778+0000 7f9566600700 1 -- start start 2024-03-18T16:08:00.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:00.778+0000 7f9566600700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9560102320 0x7f95601a08e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:00.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:00.778+0000 7f9566600700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9560102ce0 0x7f95601a0e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:00.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:00.778+0000 7f9566600700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9560108300 0x7f956019aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:00.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:00.778+0000 7f9566600700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f95601125d0 con 0x7f9560108300 2024-03-18T16:08:00.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:00.778+0000 7f9566600700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f9560112450 con 0x7f9560102ce0 2024-03-18T16:08:00.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:00.778+0000 7f9566600700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f9560112750 con 0x7f9560102320 2024-03-18T16:08:00.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:00.779+0000 7f955f400700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9560102320 0x7f95601a08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:00.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:00.779+0000 7f955f400700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9560102320 0x7f95601a08e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:44774/0 (socket says 172.21.15.39:44774) 2024-03-18T16:08:00.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:00.779+0000 7f9556a00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9560102ce0 0x7f95601a0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:00.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:00.779+0000 7f955f400700 1 -- 172.21.15.39:0/2043252894 learned_addr learned my addr 172.21.15.39:0/2043252894 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:08:00.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:00.779+0000 7f955fe00700 1 --2- 172.21.15.39:0/2043252894 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9560108300 0x7f956019aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:00.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:00.779+0000 7f955f400700 1 -- 172.21.15.39:0/2043252894 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9560102ce0 msgr2=0x7f95601a0e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:00.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:00.779+0000 7f955f400700 1 --2- 172.21.15.39:0/2043252894 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9560102ce0 0x7f95601a0e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:00.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:00.779+0000 7f955f400700 1 -- 172.21.15.39:0/2043252894 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9560108300 msgr2=0x7f956019aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:00.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:00.779+0000 7f955f400700 1 --2- 172.21.15.39:0/2043252894 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9560108300 0x7f956019aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:00.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:00.779+0000 7f955f400700 1 -- 172.21.15.39:0/2043252894 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f956019b200 con 0x7f9560102320 2024-03-18T16:08:00.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:00.779+0000 7f9556a00700 1 --2- 172.21.15.39:0/2043252894 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9560102ce0 0x7f95601a0e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:08:00.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:00.780+0000 7f955f400700 1 --2- 172.21.15.39:0/2043252894 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9560102320 0x7f95601a08e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f955000ef10 tx=0x7f955000c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:00.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:00.780+0000 7f955cc00700 1 -- 172.21.15.39:0/2043252894 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f955000c9b0 con 0x7f9560102320 2024-03-18T16:08:00.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:00.780+0000 7f955cc00700 1 -- 172.21.15.39:0/2043252894 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9550019070 con 0x7f9560102320 2024-03-18T16:08:00.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:00.780+0000 7f955cc00700 1 -- 172.21.15.39:0/2043252894 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9550007e60 con 0x7f9560102320 2024-03-18T16:08:00.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:00.780+0000 7f9566600700 1 -- 172.21.15.39:0/2043252894 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f956019b490 con 0x7f9560102320 2024-03-18T16:08:00.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:00.780+0000 7f9566600700 1 -- 172.21.15.39:0/2043252894 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95601a77b0 con 0x7f9560102320 2024-03-18T16:08:00.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:00.781+0000 7f9566600700 1 -- 172.21.15.39:0/2043252894 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f9560076200 con 0x7f9560102320 2024-03-18T16:08:00.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:00.783+0000 7f955cc00700 1 -- 172.21.15.39:0/2043252894 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f9550012050 con 0x7f9560102320 2024-03-18T16:08:00.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:00.784+0000 7f955cc00700 1 --2- 172.21.15.39:0/2043252894 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f95440a29a0 0x7f95440a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:00.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:00.785+0000 7f955cc00700 1 -- 172.21.15.39:0/2043252894 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f95500cee30 con 0x7f9560102320 2024-03-18T16:08:00.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:00.785+0000 7f9556a00700 1 --2- 172.21.15.39:0/2043252894 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f95440a29a0 0x7f95440a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:00.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:00.785+0000 7f955cc00700 1 --2- 172.21.15.39:0/2043252894 >> [v2:172.21.15.194:6800/2800329805,v1:172.21.15.194:6801/2800329805] conn(0x7f95440adb00 0x7f95440affa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:00.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:00.785+0000 7f955cc00700 1 -- 172.21.15.39:0/2043252894 --> [v2:172.21.15.194:6800/2800329805,v1:172.21.15.194:6801/2800329805] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9550008ec0 con 0x7f95440adb00 2024-03-18T16:08:00.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:00.785+0000 7f955cc00700 1 -- 172.21.15.39:0/2043252894 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f95500d1450 con 0x7f9560102320 2024-03-18T16:08:00.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:00.785+0000 7f955fe00700 1 --2- 172.21.15.39:0/2043252894 >> [v2:172.21.15.194:6800/2800329805,v1:172.21.15.194:6801/2800329805] conn(0x7f95440adb00 0x7f95440affa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:00.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:00.786+0000 7f9556a00700 1 --2- 172.21.15.39:0/2043252894 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f95440a29a0 0x7f95440a4e60 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f9560079d70 tx=0x7f954800b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:00.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:00.786+0000 7f955fe00700 1 --2- 172.21.15.39:0/2043252894 >> [v2:172.21.15.194:6800/2800329805,v1:172.21.15.194:6801/2800329805] conn(0x7f95440adb00 0x7f95440affa0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:00.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:00.787+0000 7f955cc00700 1 -- 172.21.15.39:0/2043252894 <== osd.1 v2:172.21.15.194:6800/2800329805 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9550008ec0 con 0x7f95440adb00 2024-03-18T16:08:00.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:00.806+0000 7f9566600700 1 -- 172.21.15.39:0/2043252894 --> [v2:172.21.15.194:6800/2800329805,v1:172.21.15.194:6801/2800329805] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f95601039b0 con 0x7f95440adb00 2024-03-18T16:08:00.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:00.807+0000 7f955cc00700 1 -- 172.21.15.39:0/2043252894 <== osd.1 v2:172.21.15.194:6800/2800329805 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f95601039b0 con 0x7f95440adb00 2024-03-18T16:08:00.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:00.807+0000 7f9566600700 1 -- 172.21.15.39:0/2043252894 >> [v2:172.21.15.194:6800/2800329805,v1:172.21.15.194:6801/2800329805] conn(0x7f95440adb00 msgr2=0x7f95440affa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:00.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:00.807+0000 7f9566600700 1 --2- 172.21.15.39:0/2043252894 >> [v2:172.21.15.194:6800/2800329805,v1:172.21.15.194:6801/2800329805] conn(0x7f95440adb00 0x7f95440affa0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:00.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:00.807+0000 7f9566600700 1 -- 172.21.15.39:0/2043252894 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f95440a29a0 msgr2=0x7f95440a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:00.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:00.807+0000 7f9566600700 1 --2- 172.21.15.39:0/2043252894 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f95440a29a0 0x7f95440a4e60 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f9560079d70 tx=0x7f954800b260 comp rx=0 tx=0).stop 2024-03-18T16:08:00.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:00.808+0000 7f9566600700 1 -- 172.21.15.39:0/2043252894 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9560102320 msgr2=0x7f95601a08e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:00.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:00.808+0000 7f9566600700 1 --2- 172.21.15.39:0/2043252894 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9560102320 0x7f95601a08e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f955000ef10 tx=0x7f955000c3a0 comp rx=0 tx=0).stop 2024-03-18T16:08:00.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:00.808+0000 7f955fe00700 1 -- 172.21.15.39:0/2043252894 reap_dead start 2024-03-18T16:08:00.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:00.808+0000 7f9566600700 1 -- 172.21.15.39:0/2043252894 shutdown_connections 2024-03-18T16:08:00.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:00.808+0000 7f9566600700 1 -- 172.21.15.39:0/2043252894 >> 172.21.15.39:0/2043252894 conn(0x7f9560079400 msgr2=0x7f9560074790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:08:00.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:00.808+0000 7f9566600700 1 -- 172.21.15.39:0/2043252894 shutdown_connections 2024-03-18T16:08:00.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:00.808+0000 7f9566600700 1 -- 172.21.15.39:0/2043252894 wait complete. 2024-03-18T16:08:00.948 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:08:00 smithi039 ceph-mon[31721]: pgmap v358: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s 2024-03-18T16:08:00.977 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:08:00 smithi196 ceph-mon[36307]: pgmap v358: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s 2024-03-18T16:08:00.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:08:00 smithi194 ceph-mon[36200]: pgmap v358: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 11 op/s 2024-03-18T16:08:02.196 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T16:08:02.729 INFO:teuthology.orchestra.run.smithi039.stdout:55834574933 2024-03-18T16:08:02.730 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-18T16:08:02.791 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:08:02 smithi039 ceph-mon[31721]: pgmap v359: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-03-18T16:08:02.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:08:02 smithi196 ceph-mon[36307]: pgmap v359: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-03-18T16:08:02.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:08:02 smithi194 ceph-mon[36200]: pgmap v359: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-03-18T16:08:03.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:02.503+0000 7f4be7e00700 1 -- 172.21.15.39:0/2326047409 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4be006b420 msgr2=0x7f4be0106ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:03.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:02.503+0000 7f4be7e00700 1 --2- 172.21.15.39:0/2326047409 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4be006b420 0x7f4be0106ed0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f4bcc009990 tx=0x7f4bcc009ca0 comp rx=0 tx=0).stop 2024-03-18T16:08:03.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:02.503+0000 7f4be7e00700 1 -- 172.21.15.39:0/2326047409 shutdown_connections 2024-03-18T16:08:03.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:02.503+0000 7f4be7e00700 1 --2- 172.21.15.39:0/2326047409 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4be01075b0 0x7f4be010f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:03.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:02.503+0000 7f4be7e00700 1 --2- 172.21.15.39:0/2326047409 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4be006aa70 0x7f4be006ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:03.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:02.503+0000 7f4be7e00700 1 --2- 172.21.15.39:0/2326047409 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4be006b420 0x7f4be0106ed0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:03.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:02.503+0000 7f4be7e00700 1 -- 172.21.15.39:0/2326047409 >> 172.21.15.39:0/2326047409 conn(0x7f4be0077220 msgr2=0x7f4be0077630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:08:03.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:02.503+0000 7f4be7e00700 1 -- 172.21.15.39:0/2326047409 shutdown_connections 2024-03-18T16:08:03.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:02.503+0000 7f4be7e00700 1 -- 172.21.15.39:0/2326047409 wait complete. 2024-03-18T16:08:03.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:02.505+0000 7f4be7e00700 1 Processor -- start 2024-03-18T16:08:03.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:02.505+0000 7f4be7e00700 1 -- start start 2024-03-18T16:08:03.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:02.505+0000 7f4be7e00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4be006aa70 0x7f4be019e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:03.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:02.505+0000 7f4be7e00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4be006b420 0x7f4be019ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:03.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:02.506+0000 7f4be7e00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4be01075b0 0x7f4be0198900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:03.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:02.506+0000 7f4be7e00700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f4be01125e0 con 0x7f4be006b420 2024-03-18T16:08:03.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:02.506+0000 7f4be7e00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f4be0112460 con 0x7f4be01075b0 2024-03-18T16:08:03.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:02.506+0000 7f4be7e00700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f4be0112760 con 0x7f4be006aa70 2024-03-18T16:08:03.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:02.506+0000 7f4be5400700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4be006aa70 0x7f4be019e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:03.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:02.506+0000 7f4be5400700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4be006aa70 0x7f4be019e740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:44792/0 (socket says 172.21.15.39:44792) 2024-03-18T16:08:03.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:02.506+0000 7f4be5400700 1 -- 172.21.15.39:0/291228997 learned_addr learned my addr 172.21.15.39:0/291228997 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:08:03.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:02.506+0000 7f4be5e00700 1 --2- 172.21.15.39:0/291228997 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4be01075b0 0x7f4be0198900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:03.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:02.506+0000 7f4be4a00700 1 --2- 172.21.15.39:0/291228997 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4be006b420 0x7f4be019ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:03.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:02.506+0000 7f4be5400700 1 -- 172.21.15.39:0/291228997 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4be01075b0 msgr2=0x7f4be0198900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:03.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:02.506+0000 7f4be5400700 1 --2- 172.21.15.39:0/291228997 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4be01075b0 0x7f4be0198900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:03.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:02.506+0000 7f4be5400700 1 -- 172.21.15.39:0/291228997 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4be006b420 msgr2=0x7f4be019ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:03.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:02.506+0000 7f4be5400700 1 --2- 172.21.15.39:0/291228997 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4be006b420 0x7f4be019ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:03.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:02.506+0000 7f4be5400700 1 -- 172.21.15.39:0/291228997 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4be0199060 con 0x7f4be006aa70 2024-03-18T16:08:03.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:02.506+0000 7f4be4a00700 1 --2- 172.21.15.39:0/291228997 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4be006b420 0x7f4be019ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:08:03.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:02.506+0000 7f4be5e00700 1 --2- 172.21.15.39:0/291228997 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4be01075b0 0x7f4be0198900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:08:03.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:02.507+0000 7f4be5400700 1 --2- 172.21.15.39:0/291228997 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4be006aa70 0x7f4be019e740 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f4bd400e9c0 tx=0x7f4bd400ed80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:03.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:02.507+0000 7f4bde000700 1 -- 172.21.15.39:0/291228997 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bd400cbd0 con 0x7f4be006aa70 2024-03-18T16:08:03.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:02.507+0000 7f4bde000700 1 -- 172.21.15.39:0/291228997 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4bd400cd70 con 0x7f4be006aa70 2024-03-18T16:08:03.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:02.507+0000 7f4be7e00700 1 -- 172.21.15.39:0/291228997 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4be0199350 con 0x7f4be006aa70 2024-03-18T16:08:03.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:02.507+0000 7f4bde000700 1 -- 172.21.15.39:0/291228997 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bd4010430 con 0x7f4be006aa70 2024-03-18T16:08:03.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:02.507+0000 7f4be7e00700 1 -- 172.21.15.39:0/291228997 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4be01a5610 con 0x7f4be006aa70 2024-03-18T16:08:03.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:02.508+0000 7f4be7e00700 1 -- 172.21.15.39:0/291228997 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4be0062a60 con 0x7f4be006aa70 2024-03-18T16:08:03.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:02.510+0000 7f4bde000700 1 -- 172.21.15.39:0/291228997 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f4bd4004da0 con 0x7f4be006aa70 2024-03-18T16:08:03.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:02.512+0000 7f4bde000700 1 --2- 172.21.15.39:0/291228997 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f4bc80a2a20 0x7f4bc80a4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:03.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:02.512+0000 7f4bde000700 1 -- 172.21.15.39:0/291228997 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f4bd40cf650 con 0x7f4be006aa70 2024-03-18T16:08:03.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:02.512+0000 7f4be4a00700 1 --2- 172.21.15.39:0/291228997 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f4bc80a2a20 0x7f4bc80a4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:03.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:02.512+0000 7f4bde000700 1 --2- 172.21.15.39:0/291228997 >> [v2:172.21.15.196:6808/2746224525,v1:172.21.15.196:6809/2746224525] conn(0x7f4bc80adb80 0x7f4bc80b0020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:03.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:02.512+0000 7f4bde000700 1 -- 172.21.15.39:0/291228997 --> [v2:172.21.15.196:6808/2746224525,v1:172.21.15.196:6809/2746224525] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4bd4004500 con 0x7f4bc80adb80 2024-03-18T16:08:03.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:02.512+0000 7f4bde000700 1 -- 172.21.15.39:0/291228997 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f4bd40cfa40 con 0x7f4be006aa70 2024-03-18T16:08:03.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:02.513+0000 7f4be5e00700 1 --2- 172.21.15.39:0/291228997 >> [v2:172.21.15.196:6808/2746224525,v1:172.21.15.196:6809/2746224525] conn(0x7f4bc80adb80 0x7f4bc80b0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:03.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:02.513+0000 7f4be4a00700 1 --2- 172.21.15.39:0/291228997 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f4bc80a2a20 0x7f4bc80a4ee0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f4bcc006010 tx=0x7f4bcc01a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:03.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:02.513+0000 7f4be5e00700 1 --2- 172.21.15.39:0/291228997 >> [v2:172.21.15.196:6808/2746224525,v1:172.21.15.196:6809/2746224525] conn(0x7f4bc80adb80 0x7f4bc80b0020 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:03.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:02.514+0000 7f4bde000700 1 -- 172.21.15.39:0/291228997 <== osd.5 v2:172.21.15.196:6808/2746224525 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4bd4004500 con 0x7f4bc80adb80 2024-03-18T16:08:03.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:02.532+0000 7f4be7e00700 1 -- 172.21.15.39:0/291228997 --> [v2:172.21.15.196:6808/2746224525,v1:172.21.15.196:6809/2746224525] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f4be0062e60 con 0x7f4bc80adb80 2024-03-18T16:08:03.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:02.532+0000 7f4bde000700 1 -- 172.21.15.39:0/291228997 <== osd.5 v2:172.21.15.196:6808/2746224525 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f4be0062e60 con 0x7f4bc80adb80 2024-03-18T16:08:03.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:02.532+0000 7f4be7e00700 1 -- 172.21.15.39:0/291228997 >> [v2:172.21.15.196:6808/2746224525,v1:172.21.15.196:6809/2746224525] conn(0x7f4bc80adb80 msgr2=0x7f4bc80b0020 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:03.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:02.532+0000 7f4be7e00700 1 --2- 172.21.15.39:0/291228997 >> [v2:172.21.15.196:6808/2746224525,v1:172.21.15.196:6809/2746224525] conn(0x7f4bc80adb80 0x7f4bc80b0020 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:03.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:02.533+0000 7f4be7e00700 1 -- 172.21.15.39:0/291228997 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f4bc80a2a20 msgr2=0x7f4bc80a4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:03.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:02.533+0000 7f4be7e00700 1 --2- 172.21.15.39:0/291228997 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f4bc80a2a20 0x7f4bc80a4ee0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f4bcc006010 tx=0x7f4bcc01a040 comp rx=0 tx=0).stop 2024-03-18T16:08:03.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:02.533+0000 7f4be7e00700 1 -- 172.21.15.39:0/291228997 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4be006aa70 msgr2=0x7f4be019e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:03.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:02.533+0000 7f4be7e00700 1 --2- 172.21.15.39:0/291228997 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4be006aa70 0x7f4be019e740 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f4bd400e9c0 tx=0x7f4bd400ed80 comp rx=0 tx=0).stop 2024-03-18T16:08:03.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:02.533+0000 7f4be5e00700 1 -- 172.21.15.39:0/291228997 reap_dead start 2024-03-18T16:08:03.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:02.533+0000 7f4be7e00700 1 -- 172.21.15.39:0/291228997 shutdown_connections 2024-03-18T16:08:03.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:02.533+0000 7f4be7e00700 1 -- 172.21.15.39:0/291228997 >> 172.21.15.39:0/291228997 conn(0x7f4be0077220 msgr2=0x7f4be0100980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:08:03.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:02.533+0000 7f4be7e00700 1 -- 172.21.15.39:0/291228997 shutdown_connections 2024-03-18T16:08:03.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:02.533+0000 7f4be7e00700 1 -- 172.21.15.39:0/291228997 wait complete. 2024-03-18T16:08:03.759 INFO:teuthology.orchestra.run.smithi039.stdout:77309411408 2024-03-18T16:08:03.760 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-18T16:08:04.172 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:04.029+0000 7f35c4400700 1 -- 172.21.15.39:0/2423961588 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f35bc108300 msgr2=0x7f35bc1086e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:04.172 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:04.029+0000 7f35c4400700 1 --2- 172.21.15.39:0/2423961588 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f35bc108300 0x7f35bc1086e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f35b00098f0 tx=0x7f35b0009c00 comp rx=0 tx=0).stop 2024-03-18T16:08:04.172 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:04.031+0000 7f35c4400700 1 -- 172.21.15.39:0/2423961588 shutdown_connections 2024-03-18T16:08:04.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:04.031+0000 7f35c4400700 1 --2- 172.21.15.39:0/2423961588 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f35bc102ce0 0x7f35bc10f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:04.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:04.031+0000 7f35c4400700 1 --2- 172.21.15.39:0/2423961588 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f35bc108300 0x7f35bc1086e0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:04.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:04.031+0000 7f35c4400700 1 --2- 172.21.15.39:0/2423961588 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f35bc102320 0x7f35bc1027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:04.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:04.031+0000 7f35c4400700 1 -- 172.21.15.39:0/2423961588 >> 172.21.15.39:0/2423961588 conn(0x7f35bc079400 msgr2=0x7f35bc079810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:08:04.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:04.031+0000 7f35c4400700 1 -- 172.21.15.39:0/2423961588 shutdown_connections 2024-03-18T16:08:04.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:04.031+0000 7f35c4400700 1 -- 172.21.15.39:0/2423961588 wait complete. 2024-03-18T16:08:04.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:04.033+0000 7f35c4400700 1 Processor -- start 2024-03-18T16:08:04.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:04.033+0000 7f35c4400700 1 -- start start 2024-03-18T16:08:04.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:04.033+0000 7f35c4400700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f35bc102320 0x7f35bc19e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:04.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:04.033+0000 7f35c4400700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f35bc102ce0 0x7f35bc19ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:04.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:04.033+0000 7f35c4400700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f35bc108300 0x7f35bc1988b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:04.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:04.033+0000 7f35c4400700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f35bc1125d0 con 0x7f35bc102ce0 2024-03-18T16:08:04.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:04.034+0000 7f35c4400700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f35bc112450 con 0x7f35bc102320 2024-03-18T16:08:04.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:04.034+0000 7f35c4400700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f35bc112750 con 0x7f35bc108300 2024-03-18T16:08:04.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:04.034+0000 7f35c1a00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f35bc102320 0x7f35bc19e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:04.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:04.034+0000 7f35c1a00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f35bc102320 0x7f35bc19e6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:44812/0 (socket says 172.21.15.39:44812) 2024-03-18T16:08:04.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:04.034+0000 7f35c1000700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f35bc102ce0 0x7f35bc19ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:04.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:04.034+0000 7f35c1a00700 1 -- 172.21.15.39:0/333917503 learned_addr learned my addr 172.21.15.39:0/333917503 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:08:04.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:04.034+0000 7f35c2400700 1 --2- 172.21.15.39:0/333917503 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f35bc108300 0x7f35bc1988b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:04.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:04.034+0000 7f35c1a00700 1 -- 172.21.15.39:0/333917503 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f35bc108300 msgr2=0x7f35bc1988b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:04.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:04.034+0000 7f35c1a00700 1 --2- 172.21.15.39:0/333917503 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f35bc108300 0x7f35bc1988b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:04.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:04.034+0000 7f35c1a00700 1 -- 172.21.15.39:0/333917503 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f35bc102ce0 msgr2=0x7f35bc19ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:04.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:04.034+0000 7f35c1a00700 1 --2- 172.21.15.39:0/333917503 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f35bc102ce0 0x7f35bc19ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:04.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:04.034+0000 7f35c1a00700 1 -- 172.21.15.39:0/333917503 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35bc199010 con 0x7f35bc102320 2024-03-18T16:08:04.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:04.035+0000 7f35c1a00700 1 --2- 172.21.15.39:0/333917503 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f35bc102320 0x7f35bc19e6f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f35b0009fd0 tx=0x7f35b00094b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:04.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:04.035+0000 7f35ba000700 1 -- 172.21.15.39:0/333917503 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35b0025070 con 0x7f35bc102320 2024-03-18T16:08:04.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:04.035+0000 7f35ba000700 1 -- 172.21.15.39:0/333917503 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f35b000b560 con 0x7f35bc102320 2024-03-18T16:08:04.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:04.035+0000 7f35ba000700 1 -- 172.21.15.39:0/333917503 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35b0004c30 con 0x7f35bc102320 2024-03-18T16:08:04.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:04.036+0000 7f35c4400700 1 -- 172.21.15.39:0/333917503 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35bc1992a0 con 0x7f35bc102320 2024-03-18T16:08:04.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:04.036+0000 7f35c4400700 1 -- 172.21.15.39:0/333917503 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35bc074500 con 0x7f35bc102320 2024-03-18T16:08:04.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:04.037+0000 7f35c4400700 1 -- 172.21.15.39:0/333917503 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f35bc062a60 con 0x7f35bc102320 2024-03-18T16:08:04.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:04.039+0000 7f35ba000700 1 -- 172.21.15.39:0/333917503 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f35b0028070 con 0x7f35bc102320 2024-03-18T16:08:04.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:04.041+0000 7f35ba000700 1 --2- 172.21.15.39:0/333917503 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f35a40a2a20 0x7f35a40a4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:04.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:04.041+0000 7f35ba000700 1 -- 172.21.15.39:0/333917503 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f35b00cfb50 con 0x7f35bc102320 2024-03-18T16:08:04.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:04.041+0000 7f35ba000700 1 --2- 172.21.15.39:0/333917503 >> [v2:172.21.15.39:6802/4090743635,v1:172.21.15.39:6803/4090743635] conn(0x7f35a40ada50 0x7f35a40afef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:04.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:04.041+0000 7f35ba000700 1 -- 172.21.15.39:0/333917503 --> [v2:172.21.15.39:6802/4090743635,v1:172.21.15.39:6803/4090743635] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f35a40b05c0 con 0x7f35a40ada50 2024-03-18T16:08:04.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:04.041+0000 7f35ba000700 1 -- 172.21.15.39:0/333917503 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f35b00cff40 con 0x7f35bc102320 2024-03-18T16:08:04.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:04.043+0000 7f35c1000700 1 --2- 172.21.15.39:0/333917503 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f35a40a2a20 0x7f35a40a4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:04.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:04.043+0000 7f35c2400700 1 --2- 172.21.15.39:0/333917503 >> [v2:172.21.15.39:6802/4090743635,v1:172.21.15.39:6803/4090743635] conn(0x7f35a40ada50 0x7f35a40afef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:04.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:04.044+0000 7f35c1000700 1 --2- 172.21.15.39:0/333917503 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f35a40a2a20 0x7f35a40a4ee0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f35bc079a70 tx=0x7f35a8012040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:04.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:04.044+0000 7f35c2400700 1 --2- 172.21.15.39:0/333917503 >> [v2:172.21.15.39:6802/4090743635,v1:172.21.15.39:6803/4090743635] conn(0x7f35a40ada50 0x7f35a40afef0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:04.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:04.045+0000 7f35ba000700 1 -- 172.21.15.39:0/333917503 <== osd.0 v2:172.21.15.39:6802/4090743635 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f35a40b05c0 con 0x7f35a40ada50 2024-03-18T16:08:04.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:04.067+0000 7f35c4400700 1 -- 172.21.15.39:0/333917503 --> [v2:172.21.15.39:6802/4090743635,v1:172.21.15.39:6803/4090743635] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f35bc062f30 con 0x7f35a40ada50 2024-03-18T16:08:04.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:04.068+0000 7f35ba000700 1 -- 172.21.15.39:0/333917503 <== osd.0 v2:172.21.15.39:6802/4090743635 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f35bc062f30 con 0x7f35a40ada50 2024-03-18T16:08:04.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:04.069+0000 7f35c4400700 1 -- 172.21.15.39:0/333917503 >> [v2:172.21.15.39:6802/4090743635,v1:172.21.15.39:6803/4090743635] conn(0x7f35a40ada50 msgr2=0x7f35a40afef0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:04.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:04.069+0000 7f35c4400700 1 --2- 172.21.15.39:0/333917503 >> [v2:172.21.15.39:6802/4090743635,v1:172.21.15.39:6803/4090743635] conn(0x7f35a40ada50 0x7f35a40afef0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:04.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:04.069+0000 7f35c4400700 1 -- 172.21.15.39:0/333917503 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f35a40a2a20 msgr2=0x7f35a40a4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:04.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:04.069+0000 7f35c4400700 1 --2- 172.21.15.39:0/333917503 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f35a40a2a20 0x7f35a40a4ee0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f35bc079a70 tx=0x7f35a8012040 comp rx=0 tx=0).stop 2024-03-18T16:08:04.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:04.069+0000 7f35c4400700 1 -- 172.21.15.39:0/333917503 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f35bc102320 msgr2=0x7f35bc19e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:04.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:04.069+0000 7f35c4400700 1 --2- 172.21.15.39:0/333917503 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f35bc102320 0x7f35bc19e6f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f35b0009fd0 tx=0x7f35b00094b0 comp rx=0 tx=0).stop 2024-03-18T16:08:04.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:04.069+0000 7f35c2400700 1 -- 172.21.15.39:0/333917503 reap_dead start 2024-03-18T16:08:04.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:04.069+0000 7f35c4400700 1 -- 172.21.15.39:0/333917503 shutdown_connections 2024-03-18T16:08:04.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:04.069+0000 7f35c4400700 1 -- 172.21.15.39:0/333917503 >> 172.21.15.39:0/333917503 conn(0x7f35bc079400 msgr2=0x7f35bc100ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:08:04.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:04.069+0000 7f35c4400700 1 -- 172.21.15.39:0/333917503 shutdown_connections 2024-03-18T16:08:04.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:04.069+0000 7f35c4400700 1 -- 172.21.15.39:0/333917503 wait complete. 2024-03-18T16:08:04.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:08:04 smithi196 ceph-mon[36307]: pgmap v360: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-18T16:08:04.979 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:08:04 smithi039 ceph-mon[31721]: pgmap v360: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-18T16:08:04.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:08:04 smithi194 ceph-mon[36200]: pgmap v360: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 10 op/s 2024-03-18T16:08:05.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.148+0000 7f043e800700 1 -- 172.21.15.39:0/1627180422 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f04381071c0 msgr2=0x7f0438069f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:05.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.148+0000 7f043e800700 1 --2- 172.21.15.39:0/1627180422 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f04381071c0 0x7f0438069f80 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f0428009990 tx=0x7f0428009ca0 comp rx=0 tx=0).stop 2024-03-18T16:08:05.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.148+0000 7f043e800700 1 -- 172.21.15.39:0/1627180422 shutdown_connections 2024-03-18T16:08:05.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.148+0000 7f043e800700 1 --2- 172.21.15.39:0/1627180422 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0438104cd0 0x7f0438105160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:05.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.148+0000 7f043e800700 1 --2- 172.21.15.39:0/1627180422 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f04381071c0 0x7f0438069f80 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:05.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.148+0000 7f043e800700 1 --2- 172.21.15.39:0/1627180422 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f043806a4c0 0x7f043806a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:05.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.148+0000 7f043e800700 1 -- 172.21.15.39:0/1627180422 >> 172.21.15.39:0/1627180422 conn(0x7f0438078100 msgr2=0x7f0438078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:08:05.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.148+0000 7f043e800700 1 -- 172.21.15.39:0/1627180422 shutdown_connections 2024-03-18T16:08:05.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.148+0000 7f043e800700 1 -- 172.21.15.39:0/1627180422 wait complete. 2024-03-18T16:08:05.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.150+0000 7f043e800700 1 Processor -- start 2024-03-18T16:08:05.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.150+0000 7f043e800700 1 -- start start 2024-03-18T16:08:05.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.150+0000 7f043e800700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f043806a4c0 0x7f043819f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:05.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.150+0000 7f043e800700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0438104cd0 0x7f043819f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:05.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.151+0000 7f043e800700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04381071c0 0x7f04381a3960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:05.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.151+0000 7f043e800700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f0438118bc0 con 0x7f04381071c0 2024-03-18T16:08:05.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.151+0000 7f043e800700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f0438118a40 con 0x7f043806a4c0 2024-03-18T16:08:05.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.151+0000 7f043e800700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f0438118d40 con 0x7f0438104cd0 2024-03-18T16:08:05.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.151+0000 7f0437400700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f043806a4c0 0x7f043819f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:05.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.151+0000 7f0436a00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0438104cd0 0x7f043819f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:05.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.151+0000 7f0437400700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f043806a4c0 0x7f043819f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:44830/0 (socket says 172.21.15.39:44830) 2024-03-18T16:08:05.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.151+0000 7f0437400700 1 -- 172.21.15.39:0/455032349 learned_addr learned my addr 172.21.15.39:0/455032349 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:08:05.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.151+0000 7f0436a00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0438104cd0 0x7f043819f5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:44830/0 (socket says 172.21.15.39:44830) 2024-03-18T16:08:05.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.151+0000 7f0437e00700 1 --2- 172.21.15.39:0/455032349 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04381071c0 0x7f04381a3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:05.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.151+0000 7f0437400700 1 -- 172.21.15.39:0/455032349 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0438104cd0 msgr2=0x7f043819f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:05.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.151+0000 7f0437400700 1 --2- 172.21.15.39:0/455032349 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0438104cd0 0x7f043819f5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:05.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.151+0000 7f0437400700 1 -- 172.21.15.39:0/455032349 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04381071c0 msgr2=0x7f04381a3960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:05.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.151+0000 7f0437400700 1 --2- 172.21.15.39:0/455032349 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04381071c0 0x7f04381a3960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:05.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.151+0000 7f0437400700 1 -- 172.21.15.39:0/455032349 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04381a40a0 con 0x7f043806a4c0 2024-03-18T16:08:05.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.151+0000 7f0436a00700 1 --2- 172.21.15.39:0/455032349 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0438104cd0 0x7f043819f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:08:05.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.152+0000 7f0437400700 1 --2- 172.21.15.39:0/455032349 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f043806a4c0 0x7f043819f090 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f0428000c00 tx=0x7f042800bab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:05.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.152+0000 7f042fe00700 1 -- 172.21.15.39:0/455032349 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0428027070 con 0x7f043806a4c0 2024-03-18T16:08:05.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.152+0000 7f042fe00700 1 -- 172.21.15.39:0/455032349 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0428003d70 con 0x7f043806a4c0 2024-03-18T16:08:05.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.152+0000 7f043e800700 1 -- 172.21.15.39:0/455032349 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04381a4330 con 0x7f043806a4c0 2024-03-18T16:08:05.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.152+0000 7f042fe00700 1 -- 172.21.15.39:0/455032349 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f042800f460 con 0x7f043806a4c0 2024-03-18T16:08:05.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.152+0000 7f043e800700 1 -- 172.21.15.39:0/455032349 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04381abc10 con 0x7f043806a4c0 2024-03-18T16:08:05.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.153+0000 7f043e800700 1 -- 172.21.15.39:0/455032349 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0438103e20 con 0x7f043806a4c0 2024-03-18T16:08:05.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.155+0000 7f042fe00700 1 -- 172.21.15.39:0/455032349 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f042802a070 con 0x7f043806a4c0 2024-03-18T16:08:05.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.157+0000 7f042fe00700 1 --2- 172.21.15.39:0/455032349 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f041c0a29a0 0x7f041c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:05.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.158+0000 7f042fe00700 1 -- 172.21.15.39:0/455032349 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f04280d0ad0 con 0x7f043806a4c0 2024-03-18T16:08:05.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.158+0000 7f0436a00700 1 --2- 172.21.15.39:0/455032349 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f041c0a29a0 0x7f041c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:05.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.158+0000 7f042fe00700 1 --2- 172.21.15.39:0/455032349 >> [v2:172.21.15.39:6810/893345939,v1:172.21.15.39:6811/893345939] conn(0x7f041c0adb00 0x7f041c0affa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:05.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.158+0000 7f042fe00700 1 -- 172.21.15.39:0/455032349 --> [v2:172.21.15.39:6810/893345939,v1:172.21.15.39:6811/893345939] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0428009030 con 0x7f041c0adb00 2024-03-18T16:08:05.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.158+0000 7f042fe00700 1 -- 172.21.15.39:0/455032349 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f04280d0e80 con 0x7f043806a4c0 2024-03-18T16:08:05.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.158+0000 7f0437e00700 1 --2- 172.21.15.39:0/455032349 >> [v2:172.21.15.39:6810/893345939,v1:172.21.15.39:6811/893345939] conn(0x7f041c0adb00 0x7f041c0affa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:05.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.159+0000 7f0436a00700 1 --2- 172.21.15.39:0/455032349 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f041c0a29a0 0x7f041c0a4e60 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f0438078990 tx=0x7f0424009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:05.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.159+0000 7f0437e00700 1 --2- 172.21.15.39:0/455032349 >> [v2:172.21.15.39:6810/893345939,v1:172.21.15.39:6811/893345939] conn(0x7f041c0adb00 0x7f041c0affa0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:05.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.160+0000 7f042fe00700 1 -- 172.21.15.39:0/455032349 <== osd.3 v2:172.21.15.39:6810/893345939 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0428009030 con 0x7f041c0adb00 2024-03-18T16:08:05.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.198+0000 7f043e800700 1 -- 172.21.15.39:0/455032349 --> [v2:172.21.15.39:6810/893345939,v1:172.21.15.39:6811/893345939] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f04381017b0 con 0x7f041c0adb00 2024-03-18T16:08:05.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.199+0000 7f042fe00700 1 -- 172.21.15.39:0/455032349 <== osd.3 v2:172.21.15.39:6810/893345939 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f04381017b0 con 0x7f041c0adb00 2024-03-18T16:08:05.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.200+0000 7f043e800700 1 -- 172.21.15.39:0/455032349 >> [v2:172.21.15.39:6810/893345939,v1:172.21.15.39:6811/893345939] conn(0x7f041c0adb00 msgr2=0x7f041c0affa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:05.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.200+0000 7f043e800700 1 --2- 172.21.15.39:0/455032349 >> [v2:172.21.15.39:6810/893345939,v1:172.21.15.39:6811/893345939] conn(0x7f041c0adb00 0x7f041c0affa0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:05.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.200+0000 7f043e800700 1 -- 172.21.15.39:0/455032349 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f041c0a29a0 msgr2=0x7f041c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:05.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.200+0000 7f043e800700 1 --2- 172.21.15.39:0/455032349 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f041c0a29a0 0x7f041c0a4e60 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f0438078990 tx=0x7f0424009260 comp rx=0 tx=0).stop 2024-03-18T16:08:05.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.200+0000 7f043e800700 1 -- 172.21.15.39:0/455032349 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f043806a4c0 msgr2=0x7f043819f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:05.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.200+0000 7f043e800700 1 --2- 172.21.15.39:0/455032349 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f043806a4c0 0x7f043819f090 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f0428000c00 tx=0x7f042800bab0 comp rx=0 tx=0).stop 2024-03-18T16:08:05.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.200+0000 7f0437e00700 1 -- 172.21.15.39:0/455032349 reap_dead start 2024-03-18T16:08:05.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.200+0000 7f043e800700 1 -- 172.21.15.39:0/455032349 shutdown_connections 2024-03-18T16:08:05.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.200+0000 7f043e800700 1 -- 172.21.15.39:0/455032349 >> 172.21.15.39:0/455032349 conn(0x7f0438078100 msgr2=0x7f04381033c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:08:05.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.200+0000 7f043e800700 1 -- 172.21.15.39:0/455032349 shutdown_connections 2024-03-18T16:08:05.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.200+0000 7f043e800700 1 -- 172.21.15.39:0/455032349 wait complete. 2024-03-18T16:08:06.471 INFO:teuthology.orchestra.run.smithi039.stdout:287762808871 2024-03-18T16:08:06.471 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.9 2024-03-18T16:08:06.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:08:06 smithi039 ceph-mon[31721]: pgmap v361: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-03-18T16:08:06.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:08:06 smithi196 ceph-mon[36307]: pgmap v361: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-03-18T16:08:06.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:08:06 smithi194 ceph-mon[36200]: pgmap v361: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s 2024-03-18T16:08:07.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:07.331+0000 7f7b4ec00700 1 -- 172.21.15.39:0/2029874384 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7b48108320 msgr2=0x7f7b48108700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:07.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:07.331+0000 7f7b4ec00700 1 --2- 172.21.15.39:0/2029874384 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7b48108320 0x7f7b48108700 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f7b38009990 tx=0x7f7b38009ca0 comp rx=0 tx=0).stop 2024-03-18T16:08:07.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:07.331+0000 7f7b4ec00700 1 -- 172.21.15.39:0/2029874384 shutdown_connections 2024-03-18T16:08:07.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:07.331+0000 7f7b4ec00700 1 --2- 172.21.15.39:0/2029874384 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7b4806a920 0x7f7b4810f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:07.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:07.331+0000 7f7b4ec00700 1 --2- 172.21.15.39:0/2029874384 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7b48108320 0x7f7b48108700 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:07.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:07.331+0000 7f7b4ec00700 1 --2- 172.21.15.39:0/2029874384 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7b48069f80 0x7f7b4806a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:07.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:07.331+0000 7f7b4ec00700 1 -- 172.21.15.39:0/2029874384 >> 172.21.15.39:0/2029874384 conn(0x7f7b48078100 msgr2=0x7f7b48078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:08:07.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:07.331+0000 7f7b4ec00700 1 -- 172.21.15.39:0/2029874384 shutdown_connections 2024-03-18T16:08:07.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:07.331+0000 7f7b4ec00700 1 -- 172.21.15.39:0/2029874384 wait complete. 2024-03-18T16:08:07.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:07.332+0000 7f7b4ec00700 1 Processor -- start 2024-03-18T16:08:07.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:07.332+0000 7f7b4ec00700 1 -- start start 2024-03-18T16:08:07.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:07.333+0000 7f7b4ec00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7b48069f80 0x7f7b4819ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:07.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:07.333+0000 7f7b4ec00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7b4806a920 0x7f7b4819b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:07.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:07.333+0000 7f7b4ec00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7b48108320 0x7f7b4819f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:07.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:07.333+0000 7f7b4ec00700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f7b48112560 con 0x7f7b4806a920 2024-03-18T16:08:07.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:07.333+0000 7f7b4ec00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f7b481123e0 con 0x7f7b48069f80 2024-03-18T16:08:07.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:07.333+0000 7f7b4ec00700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f7b481126e0 con 0x7f7b48108320 2024-03-18T16:08:07.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:07.333+0000 7f7b47e00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7b48069f80 0x7f7b4819ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:07.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:07.333+0000 7f7b47e00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7b48069f80 0x7f7b4819ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:44852/0 (socket says 172.21.15.39:44852) 2024-03-18T16:08:07.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:07.333+0000 7f7b47e00700 1 -- 172.21.15.39:0/3015351942 learned_addr learned my addr 172.21.15.39:0/3015351942 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:08:07.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:07.333+0000 7f7b3fe00700 1 --2- 172.21.15.39:0/3015351942 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7b4806a920 0x7f7b4819b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:07.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:07.333+0000 7f7b4cc00700 1 --2- 172.21.15.39:0/3015351942 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7b48108320 0x7f7b4819f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:07.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:07.334+0000 7f7b47e00700 1 -- 172.21.15.39:0/3015351942 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7b48108320 msgr2=0x7f7b4819f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:07.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:07.334+0000 7f7b47e00700 1 --2- 172.21.15.39:0/3015351942 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7b48108320 0x7f7b4819f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:07.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:07.334+0000 7f7b47e00700 1 -- 172.21.15.39:0/3015351942 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7b4806a920 msgr2=0x7f7b4819b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:07.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:07.334+0000 7f7b47e00700 1 --2- 172.21.15.39:0/3015351942 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7b4806a920 0x7f7b4819b290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:07.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:07.334+0000 7f7b47e00700 1 -- 172.21.15.39:0/3015351942 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b4819fd60 con 0x7f7b48069f80 2024-03-18T16:08:07.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:07.334+0000 7f7b47e00700 1 --2- 172.21.15.39:0/3015351942 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7b48069f80 0x7f7b4819ad50 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f7b38000c00 tx=0x7f7b3800b610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:07.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:07.334+0000 7f7b45600700 1 -- 172.21.15.39:0/3015351942 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b38027070 con 0x7f7b48069f80 2024-03-18T16:08:07.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:07.334+0000 7f7b45600700 1 -- 172.21.15.39:0/3015351942 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b38022470 con 0x7f7b48069f80 2024-03-18T16:08:07.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:07.334+0000 7f7b4ec00700 1 -- 172.21.15.39:0/3015351942 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b4819fff0 con 0x7f7b48069f80 2024-03-18T16:08:07.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:07.334+0000 7f7b4ec00700 1 -- 172.21.15.39:0/3015351942 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b481a78d0 con 0x7f7b48069f80 2024-03-18T16:08:07.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:07.335+0000 7f7b45600700 1 -- 172.21.15.39:0/3015351942 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b38017650 con 0x7f7b48069f80 2024-03-18T16:08:07.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:07.335+0000 7f7b4ec00700 1 -- 172.21.15.39:0/3015351942 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7b4810de90 con 0x7f7b48069f80 2024-03-18T16:08:07.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:07.339+0000 7f7b45600700 1 -- 172.21.15.39:0/3015351942 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f7b3802a070 con 0x7f7b48069f80 2024-03-18T16:08:07.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:07.340+0000 7f7b45600700 1 --2- 172.21.15.39:0/3015351942 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f7b2c0a29a0 0x7f7b2c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:07.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:07.341+0000 7f7b45600700 1 -- 172.21.15.39:0/3015351942 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f7b380d07a0 con 0x7f7b48069f80 2024-03-18T16:08:07.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:07.341+0000 7f7b3fe00700 1 --2- 172.21.15.39:0/3015351942 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f7b2c0a29a0 0x7f7b2c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:07.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:07.341+0000 7f7b45600700 1 --2- 172.21.15.39:0/3015351942 >> [v2:172.21.15.196:6824/2501212422,v1:172.21.15.196:6825/2501212422] conn(0x7f7b2c0adb00 0x7f7b2c0affa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:07.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:07.341+0000 7f7b45600700 1 -- 172.21.15.39:0/3015351942 --> [v2:172.21.15.196:6824/2501212422,v1:172.21.15.196:6825/2501212422] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7b38010ec0 con 0x7f7b2c0adb00 2024-03-18T16:08:07.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:07.341+0000 7f7b45600700 1 -- 172.21.15.39:0/3015351942 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f7b380d0b50 con 0x7f7b48069f80 2024-03-18T16:08:07.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:07.341+0000 7f7b4cc00700 1 --2- 172.21.15.39:0/3015351942 >> [v2:172.21.15.196:6824/2501212422,v1:172.21.15.196:6825/2501212422] conn(0x7f7b2c0adb00 0x7f7b2c0affa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:07.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:07.342+0000 7f7b3fe00700 1 --2- 172.21.15.39:0/3015351942 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f7b2c0a29a0 0x7f7b2c0a4e60 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f7b481032d0 tx=0x7f7b30009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:07.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:07.342+0000 7f7b4cc00700 1 --2- 172.21.15.39:0/3015351942 >> [v2:172.21.15.196:6824/2501212422,v1:172.21.15.196:6825/2501212422] conn(0x7f7b2c0adb00 0x7f7b2c0affa0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:07.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:07.343+0000 7f7b45600700 1 -- 172.21.15.39:0/3015351942 <== osd.11 v2:172.21.15.196:6824/2501212422 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7b38010ec0 con 0x7f7b2c0adb00 2024-03-18T16:08:07.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:07.361+0000 7f7b4ec00700 1 -- 172.21.15.39:0/3015351942 --> [v2:172.21.15.196:6824/2501212422,v1:172.21.15.196:6825/2501212422] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7b481039d0 con 0x7f7b2c0adb00 2024-03-18T16:08:07.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:07.361+0000 7f7b45600700 1 -- 172.21.15.39:0/3015351942 <== osd.11 v2:172.21.15.196:6824/2501212422 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f7b481039d0 con 0x7f7b2c0adb00 2024-03-18T16:08:07.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:07.362+0000 7f7b4ec00700 1 -- 172.21.15.39:0/3015351942 >> [v2:172.21.15.196:6824/2501212422,v1:172.21.15.196:6825/2501212422] conn(0x7f7b2c0adb00 msgr2=0x7f7b2c0affa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:07.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:07.362+0000 7f7b4ec00700 1 --2- 172.21.15.39:0/3015351942 >> [v2:172.21.15.196:6824/2501212422,v1:172.21.15.196:6825/2501212422] conn(0x7f7b2c0adb00 0x7f7b2c0affa0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:07.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:07.362+0000 7f7b4ec00700 1 -- 172.21.15.39:0/3015351942 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f7b2c0a29a0 msgr2=0x7f7b2c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:07.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:07.362+0000 7f7b4ec00700 1 --2- 172.21.15.39:0/3015351942 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f7b2c0a29a0 0x7f7b2c0a4e60 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f7b481032d0 tx=0x7f7b30009260 comp rx=0 tx=0).stop 2024-03-18T16:08:07.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:07.362+0000 7f7b4ec00700 1 -- 172.21.15.39:0/3015351942 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7b48069f80 msgr2=0x7f7b4819ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:07.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:07.362+0000 7f7b4ec00700 1 --2- 172.21.15.39:0/3015351942 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7b48069f80 0x7f7b4819ad50 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f7b38000c00 tx=0x7f7b3800b610 comp rx=0 tx=0).stop 2024-03-18T16:08:07.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:07.362+0000 7f7b4cc00700 1 -- 172.21.15.39:0/3015351942 reap_dead start 2024-03-18T16:08:07.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:07.362+0000 7f7b4ec00700 1 -- 172.21.15.39:0/3015351942 shutdown_connections 2024-03-18T16:08:07.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:07.362+0000 7f7b4ec00700 1 -- 172.21.15.39:0/3015351942 >> 172.21.15.39:0/3015351942 conn(0x7f7b48078100 msgr2=0x7f7b4810d4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:08:07.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:07.363+0000 7f7b4ec00700 1 -- 172.21.15.39:0/3015351942 shutdown_connections 2024-03-18T16:08:07.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:07.363+0000 7f7b4ec00700 1 -- 172.21.15.39:0/3015351942 wait complete. 2024-03-18T16:08:08.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.326+0000 7f0ef2800700 1 -- 172.21.15.39:0/329533383 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0eec10a590 msgr2=0x7f0eec10a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:08.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.326+0000 7f0ef2800700 1 --2- 172.21.15.39:0/329533383 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0eec10a590 0x7f0eec10a970 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f0edc009990 tx=0x7f0edc009ca0 comp rx=0 tx=0).stop 2024-03-18T16:08:08.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.326+0000 7f0ef2800700 1 -- 172.21.15.39:0/329533383 shutdown_connections 2024-03-18T16:08:08.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.326+0000 7f0ef2800700 1 --2- 172.21.15.39:0/329533383 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0eec104ea0 0x7f0eec111360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:08.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.326+0000 7f0ef2800700 1 --2- 172.21.15.39:0/329533383 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0eec10a590 0x7f0eec10a970 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:08.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.326+0000 7f0ef2800700 1 --2- 172.21.15.39:0/329533383 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0eec1044e0 0x7f0eec104960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:08.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.326+0000 7f0ef2800700 1 -- 172.21.15.39:0/329533383 >> 172.21.15.39:0/329533383 conn(0x7f0eec0fffd0 msgr2=0x7f0eec1023f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:08:08.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.327+0000 7f0ef2800700 1 -- 172.21.15.39:0/329533383 shutdown_connections 2024-03-18T16:08:08.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.327+0000 7f0ef2800700 1 -- 172.21.15.39:0/329533383 wait complete. 2024-03-18T16:08:08.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.328+0000 7f0ef2800700 1 Processor -- start 2024-03-18T16:08:08.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.328+0000 7f0ef2800700 1 -- start start 2024-03-18T16:08:08.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.328+0000 7f0ef2800700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0eec1044e0 0x7f0eec1a0860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:08.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.328+0000 7f0ef2800700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0eec104ea0 0x7f0eec1a0da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:08.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.329+0000 7f0ef2800700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0eec10a590 0x7f0eec19aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:08.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.329+0000 7f0ef2800700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f0eec114820 con 0x7f0eec10a590 2024-03-18T16:08:08.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.329+0000 7f0ef2800700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f0eec1146a0 con 0x7f0eec104ea0 2024-03-18T16:08:08.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.329+0000 7f0ef2800700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f0eec1149a0 con 0x7f0eec1044e0 2024-03-18T16:08:08.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.329+0000 7f0eeb400700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0eec1044e0 0x7f0eec1a0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:08.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.329+0000 7f0eeb400700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0eec1044e0 0x7f0eec1a0860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:44840/0 (socket says 172.21.15.39:44840) 2024-03-18T16:08:08.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.329+0000 7f0eeb400700 1 -- 172.21.15.39:0/1649963053 learned_addr learned my addr 172.21.15.39:0/1649963053 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:08:08.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.329+0000 7f0eeaa00700 1 --2- 172.21.15.39:0/1649963053 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0eec104ea0 0x7f0eec1a0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:08.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.329+0000 7f0eebe00700 1 --2- 172.21.15.39:0/1649963053 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0eec10a590 0x7f0eec19aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:08.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.329+0000 7f0eeb400700 1 -- 172.21.15.39:0/1649963053 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0eec104ea0 msgr2=0x7f0eec1a0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:08.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.329+0000 7f0eeb400700 1 --2- 172.21.15.39:0/1649963053 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0eec104ea0 0x7f0eec1a0da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:08.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.329+0000 7f0eeb400700 1 -- 172.21.15.39:0/1649963053 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0eec10a590 msgr2=0x7f0eec19aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:08.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.329+0000 7f0eeb400700 1 --2- 172.21.15.39:0/1649963053 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0eec10a590 0x7f0eec19aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:08.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.329+0000 7f0eeb400700 1 -- 172.21.15.39:0/1649963053 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0eec19b180 con 0x7f0eec1044e0 2024-03-18T16:08:08.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.329+0000 7f0eeb400700 1 --2- 172.21.15.39:0/1649963053 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0eec1044e0 0x7f0eec1a0860 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f0edc005910 tx=0x7f0edc0047b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:08.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.330+0000 7f0ee3e00700 1 -- 172.21.15.39:0/1649963053 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0edc027070 con 0x7f0eec1044e0 2024-03-18T16:08:08.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.330+0000 7f0ee3e00700 1 -- 172.21.15.39:0/1649963053 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0edc00b6c0 con 0x7f0eec1044e0 2024-03-18T16:08:08.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.330+0000 7f0ee3e00700 1 -- 172.21.15.39:0/1649963053 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0edc017650 con 0x7f0eec1044e0 2024-03-18T16:08:08.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.330+0000 7f0ef2800700 1 -- 172.21.15.39:0/1649963053 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0eec19b410 con 0x7f0eec1044e0 2024-03-18T16:08:08.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.330+0000 7f0ef2800700 1 -- 172.21.15.39:0/1649963053 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0eec1a7730 con 0x7f0eec1044e0 2024-03-18T16:08:08.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.331+0000 7f0ef2800700 1 -- 172.21.15.39:0/1649963053 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0eec062a60 con 0x7f0eec1044e0 2024-03-18T16:08:08.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.333+0000 7f0ee3e00700 1 -- 172.21.15.39:0/1649963053 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f0edc02a070 con 0x7f0eec1044e0 2024-03-18T16:08:08.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.334+0000 7f0ee3e00700 1 --2- 172.21.15.39:0/1649963053 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f0ed00a29a0 0x7f0ed00a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:08.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.334+0000 7f0ee3e00700 1 -- 172.21.15.39:0/1649963053 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f0edc0d0690 con 0x7f0eec1044e0 2024-03-18T16:08:08.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.334+0000 7f0eeaa00700 1 --2- 172.21.15.39:0/1649963053 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f0ed00a29a0 0x7f0ed00a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:08.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.334+0000 7f0ee3e00700 1 --2- 172.21.15.39:0/1649963053 >> [v2:172.21.15.39:6818/1213371331,v1:172.21.15.39:6819/1213371331] conn(0x7f0ed00adb00 0x7f0ed00affa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:08.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.335+0000 7f0ee3e00700 1 -- 172.21.15.39:0/1649963053 --> [v2:172.21.15.39:6818/1213371331,v1:172.21.15.39:6819/1213371331] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0edc00fe50 con 0x7f0ed00adb00 2024-03-18T16:08:08.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.335+0000 7f0ee3e00700 1 -- 172.21.15.39:0/1649963053 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f0edc0d0a40 con 0x7f0eec1044e0 2024-03-18T16:08:08.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.335+0000 7f0eebe00700 1 --2- 172.21.15.39:0/1649963053 >> [v2:172.21.15.39:6818/1213371331,v1:172.21.15.39:6819/1213371331] conn(0x7f0ed00adb00 0x7f0ed00affa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:08.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.335+0000 7f0eeaa00700 1 --2- 172.21.15.39:0/1649963053 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f0ed00a29a0 0x7f0ed00a4e60 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f0eec109b90 tx=0x7f0ed8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:08.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.337+0000 7f0eebe00700 1 --2- 172.21.15.39:0/1649963053 >> [v2:172.21.15.39:6818/1213371331,v1:172.21.15.39:6819/1213371331] conn(0x7f0ed00adb00 0x7f0ed00affa0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:08.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.338+0000 7f0ee3e00700 1 -- 172.21.15.39:0/1649963053 <== osd.6 v2:172.21.15.39:6818/1213371331 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0edc00fe50 con 0x7f0ed00adb00 2024-03-18T16:08:08.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.356+0000 7f0ef2800700 1 -- 172.21.15.39:0/1649963053 --> [v2:172.21.15.39:6818/1213371331,v1:172.21.15.39:6819/1213371331] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0eec062f30 con 0x7f0ed00adb00 2024-03-18T16:08:08.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.357+0000 7f0ee3e00700 1 -- 172.21.15.39:0/1649963053 <== osd.6 v2:172.21.15.39:6818/1213371331 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f0eec062f30 con 0x7f0ed00adb00 2024-03-18T16:08:08.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.357+0000 7f0ef2800700 1 -- 172.21.15.39:0/1649963053 >> [v2:172.21.15.39:6818/1213371331,v1:172.21.15.39:6819/1213371331] conn(0x7f0ed00adb00 msgr2=0x7f0ed00affa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:08.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.357+0000 7f0ef2800700 1 --2- 172.21.15.39:0/1649963053 >> [v2:172.21.15.39:6818/1213371331,v1:172.21.15.39:6819/1213371331] conn(0x7f0ed00adb00 0x7f0ed00affa0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:08.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.357+0000 7f0ef2800700 1 -- 172.21.15.39:0/1649963053 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f0ed00a29a0 msgr2=0x7f0ed00a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:08.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.357+0000 7f0ef2800700 1 --2- 172.21.15.39:0/1649963053 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f0ed00a29a0 0x7f0ed00a4e60 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f0eec109b90 tx=0x7f0ed8009260 comp rx=0 tx=0).stop 2024-03-18T16:08:08.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.357+0000 7f0ef2800700 1 -- 172.21.15.39:0/1649963053 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0eec1044e0 msgr2=0x7f0eec1a0860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:08.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.357+0000 7f0ef2800700 1 --2- 172.21.15.39:0/1649963053 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0eec1044e0 0x7f0eec1a0860 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f0edc005910 tx=0x7f0edc0047b0 comp rx=0 tx=0).stop 2024-03-18T16:08:08.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.357+0000 7f0eebe00700 1 -- 172.21.15.39:0/1649963053 reap_dead start 2024-03-18T16:08:08.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.358+0000 7f0ef2800700 1 -- 172.21.15.39:0/1649963053 shutdown_connections 2024-03-18T16:08:08.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.358+0000 7f0ef2800700 1 -- 172.21.15.39:0/1649963053 >> 172.21.15.39:0/1649963053 conn(0x7f0eec0fffd0 msgr2=0x7f0eec101640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:08:08.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.358+0000 7f0ef2800700 1 -- 172.21.15.39:0/1649963053 shutdown_connections 2024-03-18T16:08:08.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:05.358+0000 7f0ef2800700 1 -- 172.21.15.39:0/1649963053 wait complete. 2024-03-18T16:08:08.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:08:08 smithi039 ceph-mon[31721]: pgmap v362: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T16:08:08.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:08:08 smithi196 ceph-mon[36307]: pgmap v362: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T16:08:08.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:08:08 smithi194 ceph-mon[36200]: pgmap v362: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T16:08:09.152 INFO:teuthology.orchestra.run.smithi039.stdout:34359738460 2024-03-18T16:08:09.152 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-18T16:08:10.076 INFO:teuthology.orchestra.run.smithi039.stdout:219043332146 2024-03-18T16:08:10.077 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-18T16:08:10.610 INFO:teuthology.orchestra.run.smithi039.stdout:253403070511 2024-03-18T16:08:10.611 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.8 2024-03-18T16:08:10.826 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:08:10 smithi039 ceph-mon[31721]: pgmap v363: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T16:08:10.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:10.976+0000 7f45e9400700 1 -- 172.21.15.39:0/236256282 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f45e4102f30 msgr2=0x7f45e4106f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:10.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:10.976+0000 7f45e9400700 1 --2- 172.21.15.39:0/236256282 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f45e4102f30 0x7f45e4106f70 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f45d00098f0 tx=0x7f45d0009c00 comp rx=0 tx=0).stop 2024-03-18T16:08:10.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:10.976+0000 7f45e9400700 1 -- 172.21.15.39:0/236256282 shutdown_connections 2024-03-18T16:08:10.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:10.976+0000 7f45e9400700 1 --2- 172.21.15.39:0/236256282 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f45e4107650 0x7f45e410f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:10.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:10.976+0000 7f45e9400700 1 --2- 172.21.15.39:0/236256282 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f45e4102580 0x7f45e4102960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:10.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:10.976+0000 7f45e9400700 1 --2- 172.21.15.39:0/236256282 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f45e4102f30 0x7f45e4106f70 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:10.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:10.976+0000 7f45e9400700 1 -- 172.21.15.39:0/236256282 >> 172.21.15.39:0/236256282 conn(0x7f45e4079430 msgr2=0x7f45e4079840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:08:10.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:10.976+0000 7f45e9400700 1 -- 172.21.15.39:0/236256282 shutdown_connections 2024-03-18T16:08:10.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:10.976+0000 7f45e9400700 1 -- 172.21.15.39:0/236256282 wait complete. 2024-03-18T16:08:10.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:10.977+0000 7f45e9400700 1 Processor -- start 2024-03-18T16:08:10.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:10.977+0000 7f45e9400700 1 -- start start 2024-03-18T16:08:10.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:10.978+0000 7f45e9400700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f45e4102580 0x7f45e419adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:10.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:10.978+0000 7f45e9400700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f45e4102f30 0x7f45e419b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:10.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:10.978+0000 7f45e9400700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f45e4107650 0x7f45e419f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:10.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:10.978+0000 7f45e9400700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f45e4112660 con 0x7f45e4102f30 2024-03-18T16:08:10.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:10.978+0000 7f45e9400700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f45e41124e0 con 0x7f45e4107650 2024-03-18T16:08:10.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:10.978+0000 7f45e9400700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f45e41127e0 con 0x7f45e4102580 2024-03-18T16:08:10.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:10.979+0000 7f45e1e00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f45e4102f30 0x7f45e419b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:10.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:10.979+0000 7f45e2800700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f45e4102580 0x7f45e419adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:10.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:10.979+0000 7f45e2800700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f45e4102580 0x7f45e419adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:43022/0 (socket says 172.21.15.39:43022) 2024-03-18T16:08:10.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:10.979+0000 7f45e2800700 1 -- 172.21.15.39:0/4255245467 learned_addr learned my addr 172.21.15.39:0/4255245467 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:08:10.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:10.979+0000 7f45e3200700 1 --2- 172.21.15.39:0/4255245467 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f45e4107650 0x7f45e419f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:10.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:10.979+0000 7f45e2800700 1 -- 172.21.15.39:0/4255245467 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f45e4107650 msgr2=0x7f45e419f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:10.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:10.979+0000 7f45e2800700 1 --2- 172.21.15.39:0/4255245467 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f45e4107650 0x7f45e419f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:10.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:10.979+0000 7f45e2800700 1 -- 172.21.15.39:0/4255245467 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f45e4102f30 msgr2=0x7f45e419b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:10.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:10.979+0000 7f45e2800700 1 --2- 172.21.15.39:0/4255245467 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f45e4102f30 0x7f45e419b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:10.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:10.980+0000 7f45e2800700 1 -- 172.21.15.39:0/4255245467 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45e419fdc0 con 0x7f45e4102580 2024-03-18T16:08:10.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:10.980+0000 7f45e3200700 1 --2- 172.21.15.39:0/4255245467 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f45e4107650 0x7f45e419f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:08:10.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:10.980+0000 7f45e2800700 1 --2- 172.21.15.39:0/4255245467 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f45e4102580 0x7f45e419adb0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f45d400e9a0 tx=0x7f45d400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:10.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:10.980+0000 7f45db400700 1 -- 172.21.15.39:0/4255245467 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45d400ca10 con 0x7f45e4102580 2024-03-18T16:08:10.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:10.980+0000 7f45db400700 1 -- 172.21.15.39:0/4255245467 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f45d400cbb0 con 0x7f45e4102580 2024-03-18T16:08:10.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:10.980+0000 7f45e9400700 1 -- 172.21.15.39:0/4255245467 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45e41a00b0 con 0x7f45e4102580 2024-03-18T16:08:10.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:10.980+0000 7f45db400700 1 -- 172.21.15.39:0/4255245467 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45d4018680 con 0x7f45e4102580 2024-03-18T16:08:10.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:10.980+0000 7f45e9400700 1 -- 172.21.15.39:0/4255245467 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45e41a7930 con 0x7f45e4102580 2024-03-18T16:08:10.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:10.982+0000 7f45e9400700 1 -- 172.21.15.39:0/4255245467 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f45e410e8c0 con 0x7f45e4102580 2024-03-18T16:08:10.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:10.984+0000 7f45db400700 1 -- 172.21.15.39:0/4255245467 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f45d4010430 con 0x7f45e4102580 2024-03-18T16:08:10.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:10.985+0000 7f45db400700 1 --2- 172.21.15.39:0/4255245467 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f45c80a29a0 0x7f45c80a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:10.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:10.985+0000 7f45db400700 1 -- 172.21.15.39:0/4255245467 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f45d40cf160 con 0x7f45e4102580 2024-03-18T16:08:10.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:10.986+0000 7f45db400700 1 --2- 172.21.15.39:0/4255245467 >> [v2:172.21.15.194:6824/3725064809,v1:172.21.15.194:6825/3725064809] conn(0x7f45c80adb00 0x7f45c80affa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:10.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:10.986+0000 7f45db400700 1 -- 172.21.15.39:0/4255245467 --> [v2:172.21.15.194:6824/3725064809,v1:172.21.15.194:6825/3725064809] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f45d4015e80 con 0x7f45c80adb00 2024-03-18T16:08:10.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:10.986+0000 7f45db400700 1 -- 172.21.15.39:0/4255245467 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f45d40d6050 con 0x7f45e4102580 2024-03-18T16:08:10.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:10.986+0000 7f45e1e00700 1 --2- 172.21.15.39:0/4255245467 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f45c80a29a0 0x7f45c80a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:10.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:10.986+0000 7f45e3200700 1 --2- 172.21.15.39:0/4255245467 >> [v2:172.21.15.194:6824/3725064809,v1:172.21.15.194:6825/3725064809] conn(0x7f45c80adb00 0x7f45c80affa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:10.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:10.986+0000 7f45e1e00700 1 --2- 172.21.15.39:0/4255245467 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f45c80a29a0 0x7f45c80a4e60 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f45d0009fd0 tx=0x7f45d001a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:10.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:10.987+0000 7f45e3200700 1 --2- 172.21.15.39:0/4255245467 >> [v2:172.21.15.194:6824/3725064809,v1:172.21.15.194:6825/3725064809] conn(0x7f45c80adb00 0x7f45c80affa0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:10.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:10.987+0000 7f45db400700 1 -- 172.21.15.39:0/4255245467 <== osd.10 v2:172.21.15.194:6824/3725064809 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f45d4015e80 con 0x7f45c80adb00 2024-03-18T16:08:10.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:08:10 smithi194 ceph-mon[36200]: pgmap v363: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T16:08:11.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:11.006+0000 7f45e9400700 1 -- 172.21.15.39:0/4255245467 --> [v2:172.21.15.194:6824/3725064809,v1:172.21.15.194:6825/3725064809] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f45e4103a50 con 0x7f45c80adb00 2024-03-18T16:08:11.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:11.006+0000 7f45db400700 1 -- 172.21.15.39:0/4255245467 <== osd.10 v2:172.21.15.194:6824/3725064809 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f45e4103a50 con 0x7f45c80adb00 2024-03-18T16:08:11.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:11.006+0000 7f45e9400700 1 -- 172.21.15.39:0/4255245467 >> [v2:172.21.15.194:6824/3725064809,v1:172.21.15.194:6825/3725064809] conn(0x7f45c80adb00 msgr2=0x7f45c80affa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:11.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:11.006+0000 7f45e9400700 1 --2- 172.21.15.39:0/4255245467 >> [v2:172.21.15.194:6824/3725064809,v1:172.21.15.194:6825/3725064809] conn(0x7f45c80adb00 0x7f45c80affa0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:11.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:11.007+0000 7f45e9400700 1 -- 172.21.15.39:0/4255245467 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f45c80a29a0 msgr2=0x7f45c80a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:11.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:11.007+0000 7f45e9400700 1 --2- 172.21.15.39:0/4255245467 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f45c80a29a0 0x7f45c80a4e60 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f45d0009fd0 tx=0x7f45d001a040 comp rx=0 tx=0).stop 2024-03-18T16:08:11.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:11.007+0000 7f45e9400700 1 -- 172.21.15.39:0/4255245467 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f45e4102580 msgr2=0x7f45e419adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:11.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:11.007+0000 7f45e9400700 1 --2- 172.21.15.39:0/4255245467 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f45e4102580 0x7f45e419adb0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f45d400e9a0 tx=0x7f45d400ecb0 comp rx=0 tx=0).stop 2024-03-18T16:08:11.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:11.007+0000 7f45e3200700 1 -- 172.21.15.39:0/4255245467 reap_dead start 2024-03-18T16:08:11.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:11.007+0000 7f45e9400700 1 -- 172.21.15.39:0/4255245467 shutdown_connections 2024-03-18T16:08:11.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:11.007+0000 7f45e9400700 1 -- 172.21.15.39:0/4255245467 >> 172.21.15.39:0/4255245467 conn(0x7f45e4079430 msgr2=0x7f45e4100660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:08:11.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:11.007+0000 7f45e9400700 1 -- 172.21.15.39:0/4255245467 shutdown_connections 2024-03-18T16:08:11.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:11.007+0000 7f45e9400700 1 -- 172.21.15.39:0/4255245467 wait complete. 2024-03-18T16:08:11.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:08:10 smithi196 ceph-mon[36307]: pgmap v363: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T16:08:11.716 INFO:teuthology.orchestra.run.smithi039.stdout:107374182475 2024-03-18T16:08:11.717 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-18T16:08:12.568 INFO:teuthology.orchestra.run.smithi039.stdout:141733920837 2024-03-18T16:08:12.568 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-18T16:08:12.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:08:12 smithi039 ceph-mon[31721]: pgmap v364: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-03-18T16:08:12.897 INFO:teuthology.orchestra.run.smithi039.stdout:163208757313 2024-03-18T16:08:12.897 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-18T16:08:12.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:08:12 smithi196 ceph-mon[36307]: pgmap v364: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-03-18T16:08:13.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:08:12 smithi194 ceph-mon[36200]: pgmap v364: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 8 op/s 2024-03-18T16:08:13.593 INFO:teuthology.orchestra.run.smithi039.stdout:309237645348 2024-03-18T16:08:13.593 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.10 2024-03-18T16:08:14.164 INFO:teuthology.orchestra.run.smithi039.stdout:197568495673 2024-03-18T16:08:14.164 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-18T16:08:14.349 INFO:teuthology.orchestra.run.smithi039.stdout:343597383712 2024-03-18T16:08:14.350 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph osd last-stat-seq osd.11 2024-03-18T16:08:14.711 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T16:08:14.717 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T16:08:14.719 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T16:08:14.731 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T16:08:14.734 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T16:08:14.738 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T16:08:14.754 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T16:08:14.773 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T16:08:14.774 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T16:08:14.774 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T16:08:15.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:08:14 smithi039 ceph-mon[31721]: pgmap v365: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 10 op/s 2024-03-18T16:08:15.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:08:14 smithi196 ceph-mon[36307]: pgmap v365: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 10 op/s 2024-03-18T16:08:15.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:08:14 smithi194 ceph-mon[36200]: pgmap v365: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 10 op/s 2024-03-18T16:08:17.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:08:16 smithi039 ceph-mon[31721]: pgmap v366: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T16:08:17.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:08:16 smithi196 ceph-mon[36307]: pgmap v366: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T16:08:17.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:08:16 smithi194 ceph-mon[36200]: pgmap v366: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T16:08:19.030 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:08:18 smithi039 ceph-mon[31721]: pgmap v367: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-03-18T16:08:19.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:08:18 smithi196 ceph-mon[36307]: pgmap v367: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-03-18T16:08:19.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:08:18 smithi194 ceph-mon[36200]: pgmap v367: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-03-18T16:08:21.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:08:20 smithi039 ceph-mon[31721]: pgmap v368: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T16:08:21.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:08:20 smithi196 ceph-mon[36307]: pgmap v368: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T16:08:21.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:08:20 smithi194 ceph-mon[36200]: pgmap v368: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T16:08:22.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:22.212+0000 7febdfe00700 1 -- 172.21.15.39:0/2245884178 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7febd8102d10 msgr2=0x7febd810f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:22.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:22.212+0000 7febdfe00700 1 --2- 172.21.15.39:0/2245884178 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7febd8102d10 0x7febd810f170 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7febd000b0a0 tx=0x7febd000b3b0 comp rx=0 tx=0).stop 2024-03-18T16:08:22.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:22.213+0000 7febdfe00700 1 -- 172.21.15.39:0/2245884178 shutdown_connections 2024-03-18T16:08:22.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:22.213+0000 7febdfe00700 1 --2- 172.21.15.39:0/2245884178 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7febd8102d10 0x7febd810f170 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:22.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:22.213+0000 7febdfe00700 1 --2- 172.21.15.39:0/2245884178 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7febd81083a0 0x7febd8108780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:22.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:22.213+0000 7febdfe00700 1 --2- 172.21.15.39:0/2245884178 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7febd8102350 0x7febd81027d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:22.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:22.213+0000 7febdfe00700 1 -- 172.21.15.39:0/2245884178 >> 172.21.15.39:0/2245884178 conn(0x7febd8079390 msgr2=0x7febd80797a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:08:22.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:22.213+0000 7febdfe00700 1 -- 172.21.15.39:0/2245884178 shutdown_connections 2024-03-18T16:08:22.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:22.213+0000 7febdfe00700 1 -- 172.21.15.39:0/2245884178 wait complete. 2024-03-18T16:08:22.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:22.214+0000 7febdfe00700 1 Processor -- start 2024-03-18T16:08:22.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:22.214+0000 7febdfe00700 1 -- start start 2024-03-18T16:08:22.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:22.215+0000 7febdfe00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7febd8102350 0x7febd819e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:22.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:22.215+0000 7febdfe00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7febd8102d10 0x7febd819ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:22.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:22.215+0000 7febdfe00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7febd81083a0 0x7febd81988d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:22.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:22.215+0000 7febdfe00700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7febd81125d0 con 0x7febd8102350 2024-03-18T16:08:22.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:22.215+0000 7febdfe00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7febd8112450 con 0x7febd81083a0 2024-03-18T16:08:22.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:22.215+0000 7febdfe00700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7febd8112750 con 0x7febd8102d10 2024-03-18T16:08:22.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:22.215+0000 7febdca00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7febd8102d10 0x7febd819ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:22.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:22.215+0000 7febdca00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7febd8102d10 0x7febd819ec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:56248/0 (socket says 172.21.15.39:56248) 2024-03-18T16:08:22.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:22.215+0000 7febdd400700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7febd8102350 0x7febd819e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:22.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:22.215+0000 7febdca00700 1 -- 172.21.15.39:0/161196134 learned_addr learned my addr 172.21.15.39:0/161196134 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:08:22.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:22.215+0000 7febdde00700 1 --2- 172.21.15.39:0/161196134 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7febd81083a0 0x7febd81988d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:22.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:22.216+0000 7febdca00700 1 -- 172.21.15.39:0/161196134 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7febd81083a0 msgr2=0x7febd81988d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:22.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:22.216+0000 7febdca00700 1 --2- 172.21.15.39:0/161196134 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7febd81083a0 0x7febd81988d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:22.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:22.216+0000 7febdca00700 1 -- 172.21.15.39:0/161196134 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7febd8102350 msgr2=0x7febd819e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:22.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:22.216+0000 7febdca00700 1 --2- 172.21.15.39:0/161196134 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7febd8102350 0x7febd819e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:22.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:22.216+0000 7febdca00700 1 -- 172.21.15.39:0/161196134 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febd8199030 con 0x7febd8102d10 2024-03-18T16:08:22.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:22.216+0000 7febdde00700 1 --2- 172.21.15.39:0/161196134 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7febd81083a0 0x7febd81988d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:08:22.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:22.216+0000 7febdca00700 1 --2- 172.21.15.39:0/161196134 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7febd8102d10 0x7febd819ec00 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7febc400e9a0 tx=0x7febc400ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:22.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:22.216+0000 7febd6000700 1 -- 172.21.15.39:0/161196134 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febc400ca10 con 0x7febd8102d10 2024-03-18T16:08:22.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:22.217+0000 7febd6000700 1 -- 172.21.15.39:0/161196134 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7febc400cbb0 con 0x7febd8102d10 2024-03-18T16:08:22.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:22.217+0000 7febdfe00700 1 -- 172.21.15.39:0/161196134 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febd8199320 con 0x7febd8102d10 2024-03-18T16:08:22.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:22.217+0000 7febd6000700 1 -- 172.21.15.39:0/161196134 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febc4018680 con 0x7febd8102d10 2024-03-18T16:08:22.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:22.217+0000 7febdfe00700 1 -- 172.21.15.39:0/161196134 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febd8074500 con 0x7febd8102d10 2024-03-18T16:08:22.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:22.218+0000 7febdfe00700 1 -- 172.21.15.39:0/161196134 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7febd8068f90 con 0x7febd8102d10 2024-03-18T16:08:22.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:22.220+0000 7febd6000700 1 -- 172.21.15.39:0/161196134 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7febc4010430 con 0x7febd8102d10 2024-03-18T16:08:22.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:22.222+0000 7febd6000700 1 --2- 172.21.15.39:0/161196134 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7febc00a29a0 0x7febc00a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:22.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:22.222+0000 7febd6000700 1 -- 172.21.15.39:0/161196134 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7febc40cf160 con 0x7febd8102d10 2024-03-18T16:08:22.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:22.223+0000 7febdd400700 1 --2- 172.21.15.39:0/161196134 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7febc00a29a0 0x7febc00a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:22.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:22.223+0000 7febdd400700 1 --2- 172.21.15.39:0/161196134 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7febc00a29a0 0x7febc00a4e60 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7febcc009680 tx=0x7febcc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:22.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:22.230+0000 7febd6000700 1 -- 172.21.15.39:0/161196134 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7febc409bce0 con 0x7febd8102d10 2024-03-18T16:08:22.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:22.471+0000 7febdfe00700 1 -- 172.21.15.39:0/161196134 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7febd802db70 con 0x7febd8102d10 2024-03-18T16:08:22.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:22.472+0000 7febd6000700 1 -- 172.21.15.39:0/161196134 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7febc409b5e0 con 0x7febd8102d10 2024-03-18T16:08:22.474 INFO:teuthology.orchestra.run.smithi039.stdout:107374182479 2024-03-18T16:08:22.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:22.475+0000 7febdfe00700 1 -- 172.21.15.39:0/161196134 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7febc00a29a0 msgr2=0x7febc00a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:22.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:22.475+0000 7febdfe00700 1 --2- 172.21.15.39:0/161196134 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7febc00a29a0 0x7febc00a4e60 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7febcc009680 tx=0x7febcc009260 comp rx=0 tx=0).stop 2024-03-18T16:08:22.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:22.475+0000 7febdfe00700 1 -- 172.21.15.39:0/161196134 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7febd8102d10 msgr2=0x7febd819ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:22.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:22.475+0000 7febdfe00700 1 --2- 172.21.15.39:0/161196134 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7febd8102d10 0x7febd819ec00 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7febc400e9a0 tx=0x7febc400ecb0 comp rx=0 tx=0).stop 2024-03-18T16:08:22.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:22.475+0000 7febdfe00700 1 -- 172.21.15.39:0/161196134 shutdown_connections 2024-03-18T16:08:22.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:22.475+0000 7febdfe00700 1 --2- 172.21.15.39:0/161196134 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7febc00a29a0 0x7febc00a4e60 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:22.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:22.475+0000 7febdfe00700 1 --2- 172.21.15.39:0/161196134 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7febd81083a0 0x7febd81988d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:22.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:22.475+0000 7febdfe00700 1 --2- 172.21.15.39:0/161196134 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7febd8102350 0x7febd819e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:22.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:22.475+0000 7febdfe00700 1 --2- 172.21.15.39:0/161196134 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7febd8102d10 0x7febd819ec00 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:22.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:22.475+0000 7febdfe00700 1 -- 172.21.15.39:0/161196134 >> 172.21.15.39:0/161196134 conn(0x7febd8079390 msgr2=0x7febd8100fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:08:22.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:22.475+0000 7febdfe00700 1 -- 172.21.15.39:0/161196134 shutdown_connections 2024-03-18T16:08:22.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:22.475+0000 7febdfe00700 1 -- 172.21.15.39:0/161196134 wait complete. 2024-03-18T16:08:23.002 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:08:22 smithi039 ceph-mon[31721]: pgmap v369: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-18T16:08:23.002 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:08:22 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/161196134' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-18T16:08:23.166 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T16:08:23.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:08:22 smithi196 ceph-mon[36307]: pgmap v369: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-18T16:08:23.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:08:22 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/161196134' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-18T16:08:23.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:08:22 smithi194 ceph-mon[36200]: pgmap v369: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-18T16:08:23.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:08:22 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/161196134' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-18T16:08:25.074 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:08:24 smithi039 ceph-mon[31721]: pgmap v370: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 9 op/s 2024-03-18T16:08:25.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:08:24 smithi196 ceph-mon[36307]: pgmap v370: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 9 op/s 2024-03-18T16:08:25.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:08:24 smithi194 ceph-mon[36200]: pgmap v370: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 9 op/s 2024-03-18T16:08:27.052 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:08:26 smithi039 ceph-mon[31721]: pgmap v371: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-03-18T16:08:27.227 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:08:26 smithi196 ceph-mon[36307]: pgmap v371: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-03-18T16:08:27.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:08:26 smithi194 ceph-mon[36200]: pgmap v371: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-03-18T16:08:29.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:08:28 smithi039 ceph-mon[31721]: pgmap v372: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T16:08:29.227 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:08:28 smithi196 ceph-mon[36307]: pgmap v372: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T16:08:29.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:08:28 smithi194 ceph-mon[36200]: pgmap v372: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T16:08:30.969 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:08:30 smithi039 ceph-mon[31721]: pgmap v373: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T16:08:30.969 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:08:30 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/3769853816' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-18T16:08:31.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:30.125+0000 7fcaf0800700 1 -- 172.21.15.39:0/1864690550 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcae81075b0 msgr2=0x7fcae810f180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:31.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:30.125+0000 7fcaf0800700 1 --2- 172.21.15.39:0/1864690550 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcae81075b0 0x7fcae810f180 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fcae4009f90 tx=0x7fcae400b330 comp rx=0 tx=0).stop 2024-03-18T16:08:31.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:30.125+0000 7fcaf0800700 1 -- 172.21.15.39:0/1864690550 shutdown_connections 2024-03-18T16:08:31.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:30.126+0000 7fcaf0800700 1 --2- 172.21.15.39:0/1864690550 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcae81075b0 0x7fcae810f180 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:31.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:30.126+0000 7fcaf0800700 1 --2- 172.21.15.39:0/1864690550 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcae806aa70 0x7fcae806ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:31.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:30.126+0000 7fcaf0800700 1 --2- 172.21.15.39:0/1864690550 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcae806b420 0x7fcae8106ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:31.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:30.126+0000 7fcaf0800700 1 -- 172.21.15.39:0/1864690550 >> 172.21.15.39:0/1864690550 conn(0x7fcae80772b0 msgr2=0x7fcae80776c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:08:31.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:30.126+0000 7fcaf0800700 1 -- 172.21.15.39:0/1864690550 shutdown_connections 2024-03-18T16:08:31.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:30.126+0000 7fcaf0800700 1 -- 172.21.15.39:0/1864690550 wait complete. 2024-03-18T16:08:31.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:30.128+0000 7fcaf0800700 1 Processor -- start 2024-03-18T16:08:31.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:30.128+0000 7fcaf0800700 1 -- start start 2024-03-18T16:08:31.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:30.129+0000 7fcaf0800700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcae806aa70 0x7fcae81a0840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:31.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:30.129+0000 7fcaf0800700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcae806b420 0x7fcae81a0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:31.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:30.129+0000 7fcaf0800700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcae81075b0 0x7fcae819aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:31.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:30.129+0000 7fcaf0800700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fcae81125e0 con 0x7fcae81075b0 2024-03-18T16:08:31.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:30.129+0000 7fcaf0800700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fcae8112460 con 0x7fcae806aa70 2024-03-18T16:08:31.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:30.129+0000 7fcaf0800700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fcae8112760 con 0x7fcae806b420 2024-03-18T16:08:31.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:30.129+0000 7fcaede00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcae806aa70 0x7fcae81a0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:31.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:30.129+0000 7fcaede00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcae806aa70 0x7fcae81a0840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:57030/0 (socket says 172.21.15.39:57030) 2024-03-18T16:08:31.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:30.129+0000 7fcaede00700 1 -- 172.21.15.39:0/3769853816 learned_addr learned my addr 172.21.15.39:0/3769853816 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:08:31.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:30.129+0000 7fcaed400700 1 --2- 172.21.15.39:0/3769853816 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcae806b420 0x7fcae81a0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:31.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:30.129+0000 7fcaee800700 1 --2- 172.21.15.39:0/3769853816 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcae81075b0 0x7fcae819aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:31.049 INFO:teuthology.orchestra.run.smithi039.stdout:309237645352 2024-03-18T16:08:31.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:30.129+0000 7fcaede00700 1 -- 172.21.15.39:0/3769853816 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcae806b420 msgr2=0x7fcae81a0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:31.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:30.129+0000 7fcaede00700 1 --2- 172.21.15.39:0/3769853816 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcae806b420 0x7fcae81a0d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:31.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:30.129+0000 7fcaede00700 1 -- 172.21.15.39:0/3769853816 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcae81075b0 msgr2=0x7fcae819aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:31.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:30.129+0000 7fcaede00700 1 --2- 172.21.15.39:0/3769853816 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcae81075b0 0x7fcae819aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:31.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:30.129+0000 7fcaede00700 1 -- 172.21.15.39:0/3769853816 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcae819b1b0 con 0x7fcae806aa70 2024-03-18T16:08:31.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:30.130+0000 7fcaede00700 1 --2- 172.21.15.39:0/3769853816 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcae806aa70 0x7fcae81a0840 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fcadc00ec10 tx=0x7fcadc00ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:31.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:30.130+0000 7fcae2a00700 1 -- 172.21.15.39:0/3769853816 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcadc00cd40 con 0x7fcae806aa70 2024-03-18T16:08:31.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:30.130+0000 7fcae2a00700 1 -- 172.21.15.39:0/3769853816 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcadc004500 con 0x7fcae806aa70 2024-03-18T16:08:31.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:30.130+0000 7fcaf0800700 1 -- 172.21.15.39:0/3769853816 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcae819b4a0 con 0x7fcae806aa70 2024-03-18T16:08:31.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:30.130+0000 7fcae2a00700 1 -- 172.21.15.39:0/3769853816 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcadc010690 con 0x7fcae806aa70 2024-03-18T16:08:31.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:30.130+0000 7fcaf0800700 1 -- 172.21.15.39:0/3769853816 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcae81a7710 con 0x7fcae806aa70 2024-03-18T16:08:31.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:30.131+0000 7fcaf0800700 1 -- 172.21.15.39:0/3769853816 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcae8068f90 con 0x7fcae806aa70 2024-03-18T16:08:31.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:30.133+0000 7fcae2a00700 1 -- 172.21.15.39:0/3769853816 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fcadc003f10 con 0x7fcae806aa70 2024-03-18T16:08:31.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:30.135+0000 7fcae2a00700 1 --2- 172.21.15.39:0/3769853816 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fcad00a2920 0x7fcad00a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:31.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:30.135+0000 7fcae2a00700 1 -- 172.21.15.39:0/3769853816 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fcadc0cf370 con 0x7fcae806aa70 2024-03-18T16:08:31.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:30.136+0000 7fcaed400700 1 --2- 172.21.15.39:0/3769853816 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fcad00a2920 0x7fcad00a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:31.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:30.136+0000 7fcaed400700 1 --2- 172.21.15.39:0/3769853816 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fcad00a2920 0x7fcad00a4de0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fcae81088c0 tx=0x7fcad8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:31.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:30.139+0000 7fcae2a00700 1 -- 172.21.15.39:0/3769853816 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fcadc09bef0 con 0x7fcae806aa70 2024-03-18T16:08:31.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:30.433+0000 7fcaf0800700 1 -- 172.21.15.39:0/3769853816 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7fcae819bf30 con 0x7fcae806aa70 2024-03-18T16:08:31.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:30.434+0000 7fcae2a00700 1 -- 172.21.15.39:0/3769853816 <== mon.2 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7fcadc09b7f0 con 0x7fcae806aa70 2024-03-18T16:08:31.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:30.437+0000 7fcaf0800700 1 -- 172.21.15.39:0/3769853816 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fcad00a2920 msgr2=0x7fcad00a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:31.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:30.437+0000 7fcaf0800700 1 --2- 172.21.15.39:0/3769853816 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fcad00a2920 0x7fcad00a4de0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fcae81088c0 tx=0x7fcad8009260 comp rx=0 tx=0).stop 2024-03-18T16:08:31.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:30.437+0000 7fcaf0800700 1 -- 172.21.15.39:0/3769853816 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcae806aa70 msgr2=0x7fcae81a0840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:31.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:30.437+0000 7fcaf0800700 1 --2- 172.21.15.39:0/3769853816 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcae806aa70 0x7fcae81a0840 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fcadc00ec10 tx=0x7fcadc00ef20 comp rx=0 tx=0).stop 2024-03-18T16:08:31.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:30.437+0000 7fcaf0800700 1 -- 172.21.15.39:0/3769853816 shutdown_connections 2024-03-18T16:08:31.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:30.437+0000 7fcaf0800700 1 --2- 172.21.15.39:0/3769853816 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fcad00a2920 0x7fcad00a4de0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:31.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:30.437+0000 7fcaf0800700 1 --2- 172.21.15.39:0/3769853816 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcae81075b0 0x7fcae819aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:31.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:30.437+0000 7fcaf0800700 1 --2- 172.21.15.39:0/3769853816 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcae806aa70 0x7fcae81a0840 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:31.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:30.437+0000 7fcaf0800700 1 --2- 172.21.15.39:0/3769853816 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcae806b420 0x7fcae81a0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:31.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:30.437+0000 7fcaf0800700 1 -- 172.21.15.39:0/3769853816 >> 172.21.15.39:0/3769853816 conn(0x7fcae80772b0 msgr2=0x7fcae810d610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:08:31.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:30.437+0000 7fcaf0800700 1 -- 172.21.15.39:0/3769853816 shutdown_connections 2024-03-18T16:08:31.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:30.437+0000 7fcaf0800700 1 -- 172.21.15.39:0/3769853816 wait complete. 2024-03-18T16:08:31.227 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:08:30 smithi196 ceph-mon[36307]: pgmap v373: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T16:08:31.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:08:30 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/3769853816' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-18T16:08:31.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:08:30 smithi194 ceph-mon[36200]: pgmap v373: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T16:08:31.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:08:30 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/3769853816' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-18T16:08:32.348 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182475 got 107374182479 for osd.3 2024-03-18T16:08:32.348 DEBUG:teuthology.parallel:result is None 2024-03-18T16:08:33.002 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:08:32 smithi039 ceph-mon[31721]: pgmap v374: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-18T16:08:33.003 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:08:32 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/605615859' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-18T16:08:33.003 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:08:32 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/3107225586' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-18T16:08:33.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:08:32 smithi196 ceph-mon[36307]: pgmap v374: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-18T16:08:33.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:08:32 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/605615859' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-18T16:08:33.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:08:32 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/3107225586' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-18T16:08:33.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:08:32 smithi194 ceph-mon[36200]: pgmap v374: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-18T16:08:33.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:08:32 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/605615859' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-18T16:08:33.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:08:32 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/3107225586' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-18T16:08:33.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:32.102+0000 7ff4a8e00700 1 -- 172.21.15.39:0/3780740699 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff4a4069f80 msgr2=0x7ff4a406a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:33.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:32.102+0000 7ff4a8e00700 1 --2- 172.21.15.39:0/3780740699 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff4a4069f80 0x7ff4a406a3e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff490009920 tx=0x7ff490009c30 comp rx=0 tx=0).stop 2024-03-18T16:08:33.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:32.102+0000 7ff4a8e00700 1 -- 172.21.15.39:0/3780740699 shutdown_connections 2024-03-18T16:08:33.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:32.102+0000 7ff4a8e00700 1 --2- 172.21.15.39:0/3780740699 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4a406a920 0x7ff4a410f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:33.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:32.102+0000 7ff4a8e00700 1 --2- 172.21.15.39:0/3780740699 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff4a4108320 0x7ff4a4108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:33.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:32.102+0000 7ff4a8e00700 1 --2- 172.21.15.39:0/3780740699 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff4a4069f80 0x7ff4a406a3e0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:33.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:32.102+0000 7ff4a8e00700 1 -- 172.21.15.39:0/3780740699 >> 172.21.15.39:0/3780740699 conn(0x7ff4a4078100 msgr2=0x7ff4a4078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:08:33.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:32.102+0000 7ff4a8e00700 1 -- 172.21.15.39:0/3780740699 shutdown_connections 2024-03-18T16:08:33.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:32.103+0000 7ff4a8e00700 1 -- 172.21.15.39:0/3780740699 wait complete. 2024-03-18T16:08:33.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:32.103+0000 7ff4a8e00700 1 Processor -- start 2024-03-18T16:08:33.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:32.104+0000 7ff4a8e00700 1 -- start start 2024-03-18T16:08:33.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:32.104+0000 7ff4a8e00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4a4069f80 0x7ff4a4109170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:33.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:32.104+0000 7ff4a8e00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff4a406a920 0x7ff4a41096b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:33.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:32.104+0000 7ff4a8e00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff4a4108320 0x7ff4a4109bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:33.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:32.104+0000 7ff4a8e00700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7ff4a41125d0 con 0x7ff4a4069f80 2024-03-18T16:08:33.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:32.104+0000 7ff4a8e00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ff4a4112450 con 0x7ff4a406a920 2024-03-18T16:08:33.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:32.104+0000 7ff4a8e00700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7ff4a4112750 con 0x7ff4a4108320 2024-03-18T16:08:33.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:32.104+0000 7ff4a1400700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff4a406a920 0x7ff4a41096b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:33.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:32.104+0000 7ff4a1400700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff4a406a920 0x7ff4a41096b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:57052/0 (socket says 172.21.15.39:57052) 2024-03-18T16:08:33.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:32.104+0000 7ff4a1400700 1 -- 172.21.15.39:0/3107225586 learned_addr learned my addr 172.21.15.39:0/3107225586 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:08:33.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:32.104+0000 7ff4a1e00700 1 --2- 172.21.15.39:0/3107225586 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4a4069f80 0x7ff4a4109170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:33.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:32.104+0000 7ff4a2800700 1 --2- 172.21.15.39:0/3107225586 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff4a4108320 0x7ff4a4109bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:33.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:32.105+0000 7ff4a1400700 1 -- 172.21.15.39:0/3107225586 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff4a4108320 msgr2=0x7ff4a4109bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:33.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:32.105+0000 7ff4a1400700 1 --2- 172.21.15.39:0/3107225586 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff4a4108320 0x7ff4a4109bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:33.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:32.105+0000 7ff4a1400700 1 -- 172.21.15.39:0/3107225586 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4a4069f80 msgr2=0x7ff4a4109170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:33.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:32.105+0000 7ff4a1400700 1 --2- 172.21.15.39:0/3107225586 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4a4069f80 0x7ff4a4109170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:33.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:32.105+0000 7ff4a1400700 1 -- 172.21.15.39:0/3107225586 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4a41ab3f0 con 0x7ff4a406a920 2024-03-18T16:08:33.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:32.105+0000 7ff4a1400700 1 --2- 172.21.15.39:0/3107225586 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff4a406a920 0x7ff4a41096b0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff490009230 tx=0x7ff49000fa70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:33.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:32.105+0000 7ff49aa00700 1 -- 172.21.15.39:0/3107225586 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff490027070 con 0x7ff4a406a920 2024-03-18T16:08:33.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:32.105+0000 7ff49aa00700 1 -- 172.21.15.39:0/3107225586 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff490005660 con 0x7ff4a406a920 2024-03-18T16:08:33.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:32.105+0000 7ff4a8e00700 1 -- 172.21.15.39:0/3107225586 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4a41ab680 con 0x7ff4a406a920 2024-03-18T16:08:33.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:32.106+0000 7ff49aa00700 1 -- 172.21.15.39:0/3107225586 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff490004930 con 0x7ff4a406a920 2024-03-18T16:08:33.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:32.106+0000 7ff4a8e00700 1 -- 172.21.15.39:0/3107225586 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4a41abad0 con 0x7ff4a406a920 2024-03-18T16:08:33.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:32.107+0000 7ff4a8e00700 1 -- 172.21.15.39:0/3107225586 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4a41039d0 con 0x7ff4a406a920 2024-03-18T16:08:33.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:32.109+0000 7ff49aa00700 1 -- 172.21.15.39:0/3107225586 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7ff49002a070 con 0x7ff4a406a920 2024-03-18T16:08:33.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:32.111+0000 7ff49aa00700 1 --2- 172.21.15.39:0/3107225586 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff4880a29a0 0x7ff4880a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:33.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:32.111+0000 7ff49aa00700 1 -- 172.21.15.39:0/3107225586 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7ff4900cf8f0 con 0x7ff4a406a920 2024-03-18T16:08:33.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:32.111+0000 7ff4a1e00700 1 --2- 172.21.15.39:0/3107225586 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff4880a29a0 0x7ff4880a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:33.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:32.112+0000 7ff4a1e00700 1 --2- 172.21.15.39:0/3107225586 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff4880a29a0 0x7ff4880a4e60 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ff494009750 tx=0x7ff494009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:33.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:32.118+0000 7ff49aa00700 1 -- 172.21.15.39:0/3107225586 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff49005e740 con 0x7ff4a406a920 2024-03-18T16:08:33.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:32.359+0000 7ff4a8e00700 1 -- 172.21.15.39:0/3107225586 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7ff4a41abdd0 con 0x7ff4a406a920 2024-03-18T16:08:33.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:32.359+0000 7ff49aa00700 1 -- 172.21.15.39:0/3107225586 <== mon.2 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7ff490013030 con 0x7ff4a406a920 2024-03-18T16:08:33.993 INFO:teuthology.orchestra.run.smithi039.stdout:219043332153 2024-03-18T16:08:33.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:32.362+0000 7ff4a8e00700 1 -- 172.21.15.39:0/3107225586 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff4880a29a0 msgr2=0x7ff4880a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:33.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:32.362+0000 7ff4a8e00700 1 --2- 172.21.15.39:0/3107225586 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff4880a29a0 0x7ff4880a4e60 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ff494009750 tx=0x7ff494009310 comp rx=0 tx=0).stop 2024-03-18T16:08:33.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:32.362+0000 7ff4a8e00700 1 -- 172.21.15.39:0/3107225586 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff4a406a920 msgr2=0x7ff4a41096b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:33.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:32.362+0000 7ff4a8e00700 1 --2- 172.21.15.39:0/3107225586 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff4a406a920 0x7ff4a41096b0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff490009230 tx=0x7ff49000fa70 comp rx=0 tx=0).stop 2024-03-18T16:08:33.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:32.362+0000 7ff4a8e00700 1 -- 172.21.15.39:0/3107225586 shutdown_connections 2024-03-18T16:08:33.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:32.362+0000 7ff4a8e00700 1 --2- 172.21.15.39:0/3107225586 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff4880a29a0 0x7ff4880a4e60 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:33.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:32.362+0000 7ff4a8e00700 1 --2- 172.21.15.39:0/3107225586 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff4a4108320 0x7ff4a4109bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:33.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:32.362+0000 7ff4a8e00700 1 --2- 172.21.15.39:0/3107225586 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff4a4069f80 0x7ff4a4109170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:33.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:32.362+0000 7ff4a8e00700 1 --2- 172.21.15.39:0/3107225586 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff4a406a920 0x7ff4a41096b0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:33.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:32.362+0000 7ff4a8e00700 1 -- 172.21.15.39:0/3107225586 >> 172.21.15.39:0/3107225586 conn(0x7ff4a4078100 msgr2=0x7ff4a410d4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:08:33.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:32.362+0000 7ff4a8e00700 1 -- 172.21.15.39:0/3107225586 shutdown_connections 2024-03-18T16:08:33.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:32.362+0000 7ff4a8e00700 1 -- 172.21.15.39:0/3107225586 wait complete. 2024-03-18T16:08:34.386 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.382+0000 7efce1600700 1 -- 172.21.15.39:0/2989922007 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efcdc06a920 msgr2=0x7efcdc10f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:34.386 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.382+0000 7efce1600700 1 --2- 172.21.15.39:0/2989922007 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efcdc06a920 0x7efcdc10f160 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7efcd400b0d0 tx=0x7efcd400b3e0 comp rx=0 tx=0).stop 2024-03-18T16:08:34.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.382+0000 7efce1600700 1 -- 172.21.15.39:0/2989922007 shutdown_connections 2024-03-18T16:08:34.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.382+0000 7efce1600700 1 --2- 172.21.15.39:0/2989922007 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efcdc06a920 0x7efcdc10f160 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:34.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.382+0000 7efce1600700 1 --2- 172.21.15.39:0/2989922007 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efcdc108320 0x7efcdc108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:34.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.382+0000 7efce1600700 1 --2- 172.21.15.39:0/2989922007 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efcdc069f80 0x7efcdc06a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:34.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.382+0000 7efce1600700 1 -- 172.21.15.39:0/2989922007 >> 172.21.15.39:0/2989922007 conn(0x7efcdc078100 msgr2=0x7efcdc078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:08:34.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.382+0000 7efce1600700 1 -- 172.21.15.39:0/2989922007 shutdown_connections 2024-03-18T16:08:34.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.382+0000 7efce1600700 1 -- 172.21.15.39:0/2989922007 wait complete. 2024-03-18T16:08:34.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.384+0000 7efce1600700 1 Processor -- start 2024-03-18T16:08:34.388 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.384+0000 7efce1600700 1 -- start start 2024-03-18T16:08:34.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.384+0000 7efce1600700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efcdc069f80 0x7efcdc1a08a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:34.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.384+0000 7efce1600700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efcdc06a920 0x7efcdc1a0de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:34.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.384+0000 7efce1600700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efcdc108320 0x7efcdc19aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:34.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.384+0000 7efce1600700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7efcdc112570 con 0x7efcdc069f80 2024-03-18T16:08:34.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.384+0000 7efce1600700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7efcdc1123f0 con 0x7efcdc06a920 2024-03-18T16:08:34.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.384+0000 7efce1600700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7efcdc1126f0 con 0x7efcdc108320 2024-03-18T16:08:34.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.385+0000 7efcdaa00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efcdc069f80 0x7efcdc1a08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:34.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.385+0000 7efcda000700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efcdc06a920 0x7efcdc1a0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:34.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.385+0000 7efcdaa00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efcdc069f80 0x7efcdc1a08a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:32844/0 (socket says 172.21.15.39:32844) 2024-03-18T16:08:34.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.385+0000 7efcdaa00700 1 -- 172.21.15.39:0/2467322931 learned_addr learned my addr 172.21.15.39:0/2467322931 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:08:34.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.385+0000 7efcda000700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efcdc06a920 0x7efcdc1a0de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:57092/0 (socket says 172.21.15.39:57092) 2024-03-18T16:08:34.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.385+0000 7efcdb400700 1 --2- 172.21.15.39:0/2467322931 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efcdc108320 0x7efcdc19aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:34.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.385+0000 7efcda000700 1 -- 172.21.15.39:0/2467322931 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efcdc108320 msgr2=0x7efcdc19aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:34.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.385+0000 7efcda000700 1 --2- 172.21.15.39:0/2467322931 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efcdc108320 0x7efcdc19aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:34.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.385+0000 7efcda000700 1 -- 172.21.15.39:0/2467322931 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efcdc069f80 msgr2=0x7efcdc1a08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:34.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.385+0000 7efcda000700 1 --2- 172.21.15.39:0/2467322931 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efcdc069f80 0x7efcdc1a08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:34.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.385+0000 7efcda000700 1 -- 172.21.15.39:0/2467322931 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcdc19b1c0 con 0x7efcdc06a920 2024-03-18T16:08:34.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.385+0000 7efcdaa00700 1 --2- 172.21.15.39:0/2467322931 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efcdc069f80 0x7efcdc1a08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:08:34.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.385+0000 7efcda000700 1 --2- 172.21.15.39:0/2467322931 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efcdc06a920 0x7efcdc1a0de0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7efcc800e9f0 tx=0x7efcc800ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:34.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.386+0000 7efcd3400700 1 -- 172.21.15.39:0/2467322931 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcc800cbd0 con 0x7efcdc06a920 2024-03-18T16:08:34.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.386+0000 7efcd3400700 1 -- 172.21.15.39:0/2467322931 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efcc800cd70 con 0x7efcdc06a920 2024-03-18T16:08:34.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.386+0000 7efce1600700 1 -- 172.21.15.39:0/2467322931 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efcdc19b4b0 con 0x7efcdc06a920 2024-03-18T16:08:34.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.386+0000 7efce1600700 1 -- 172.21.15.39:0/2467322931 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efcdc1090a0 con 0x7efcdc06a920 2024-03-18T16:08:34.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.386+0000 7efcd3400700 1 -- 172.21.15.39:0/2467322931 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcc8018710 con 0x7efcdc06a920 2024-03-18T16:08:34.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.388+0000 7efce1600700 1 -- 172.21.15.39:0/2467322931 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efcbc005330 con 0x7efcdc06a920 2024-03-18T16:08:34.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.389+0000 7efcd3400700 1 -- 172.21.15.39:0/2467322931 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7efcc80188b0 con 0x7efcdc06a920 2024-03-18T16:08:34.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.391+0000 7efcd3400700 1 --2- 172.21.15.39:0/2467322931 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7efcc00a29a0 0x7efcc00a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:34.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.391+0000 7efcd3400700 1 -- 172.21.15.39:0/2467322931 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7efcc80cf0c0 con 0x7efcdc06a920 2024-03-18T16:08:34.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.391+0000 7efcdaa00700 1 --2- 172.21.15.39:0/2467322931 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7efcc00a29a0 0x7efcc00a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:34.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.392+0000 7efcdaa00700 1 --2- 172.21.15.39:0/2467322931 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7efcc00a29a0 0x7efcc00a4e60 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7efcdc19bba0 tx=0x7efccc018040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:34.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.399+0000 7efcd3400700 1 -- 172.21.15.39:0/2467322931 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7efcc809bc40 con 0x7efcdc06a920 2024-03-18T16:08:34.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.678+0000 7efce1600700 1 -- 172.21.15.39:0/2467322931 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7efcbc0058f0 con 0x7efcdc06a920 2024-03-18T16:08:34.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.678+0000 7efcd3400700 1 -- 172.21.15.39:0/2467322931 <== mon.2 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7efcc809b540 con 0x7efcdc06a920 2024-03-18T16:08:34.681 INFO:teuthology.orchestra.run.smithi039.stdout:77309411415 2024-03-18T16:08:34.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.681+0000 7efce1600700 1 -- 172.21.15.39:0/2467322931 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7efcc00a29a0 msgr2=0x7efcc00a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:34.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.681+0000 7efce1600700 1 --2- 172.21.15.39:0/2467322931 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7efcc00a29a0 0x7efcc00a4e60 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7efcdc19bba0 tx=0x7efccc018040 comp rx=0 tx=0).stop 2024-03-18T16:08:34.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.681+0000 7efce1600700 1 -- 172.21.15.39:0/2467322931 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efcdc06a920 msgr2=0x7efcdc1a0de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:34.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.681+0000 7efce1600700 1 --2- 172.21.15.39:0/2467322931 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efcdc06a920 0x7efcdc1a0de0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7efcc800e9f0 tx=0x7efcc800ed00 comp rx=0 tx=0).stop 2024-03-18T16:08:34.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.681+0000 7efce1600700 1 -- 172.21.15.39:0/2467322931 shutdown_connections 2024-03-18T16:08:34.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.681+0000 7efce1600700 1 --2- 172.21.15.39:0/2467322931 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7efcc00a29a0 0x7efcc00a4e60 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:34.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.681+0000 7efce1600700 1 --2- 172.21.15.39:0/2467322931 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efcdc108320 0x7efcdc19aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:34.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.681+0000 7efce1600700 1 --2- 172.21.15.39:0/2467322931 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efcdc069f80 0x7efcdc1a08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:34.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.681+0000 7efce1600700 1 --2- 172.21.15.39:0/2467322931 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efcdc06a920 0x7efcdc1a0de0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:34.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.681+0000 7efce1600700 1 -- 172.21.15.39:0/2467322931 >> 172.21.15.39:0/2467322931 conn(0x7efcdc078100 msgr2=0x7efcdc10d4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:08:34.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.682+0000 7efce1600700 1 -- 172.21.15.39:0/2467322931 shutdown_connections 2024-03-18T16:08:34.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.682+0000 7efce1600700 1 -- 172.21.15.39:0/2467322931 wait complete. 2024-03-18T16:08:35.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:08:34 smithi039 ceph-mon[31721]: pgmap v375: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T16:08:35.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:08:34 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/3667602652' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-18T16:08:35.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:08:34 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/2467322931' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-18T16:08:35.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.981+0000 7f073fc00700 1 -- 172.21.15.39:0/702508233 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0738074500 msgr2=0x7f07380748e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:35.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.981+0000 7f073fc00700 1 --2- 172.21.15.39:0/702508233 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0738074500 0x7f07380748e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f072c009fc0 tx=0x7f072c00b330 comp rx=0 tx=0).stop 2024-03-18T16:08:35.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.981+0000 7f073fc00700 1 -- 172.21.15.39:0/702508233 shutdown_connections 2024-03-18T16:08:35.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.981+0000 7f073fc00700 1 --2- 172.21.15.39:0/702508233 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f07381134a0 0x7f0738115890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:35.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.981+0000 7f073fc00700 1 --2- 172.21.15.39:0/702508233 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0738074500 0x7f07380748e0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:35.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.981+0000 7f073fc00700 1 --2- 172.21.15.39:0/702508233 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0738074e20 0x7f0738112f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:35.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.981+0000 7f073fc00700 1 -- 172.21.15.39:0/702508233 >> 172.21.15.39:0/702508233 conn(0x7f0738079430 msgr2=0x7f0738079840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:08:35.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.982+0000 7f073fc00700 1 -- 172.21.15.39:0/702508233 shutdown_connections 2024-03-18T16:08:35.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.982+0000 7f073fc00700 1 -- 172.21.15.39:0/702508233 wait complete. 2024-03-18T16:08:35.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.983+0000 7f073fc00700 1 Processor -- start 2024-03-18T16:08:35.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.983+0000 7f073fc00700 1 -- start start 2024-03-18T16:08:35.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.984+0000 7f073fc00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0738074500 0x7f073819f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:35.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.984+0000 7f073fc00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0738074e20 0x7f073819f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:35.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.984+0000 7f073fc00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f07381134a0 0x7f07381a39e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:35.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.984+0000 7f073fc00700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f0738118ce0 con 0x7f07381134a0 2024-03-18T16:08:35.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.984+0000 7f073fc00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f0738118b60 con 0x7f0738074500 2024-03-18T16:08:35.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.984+0000 7f073fc00700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f0738118e60 con 0x7f0738074e20 2024-03-18T16:08:35.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.984+0000 7f073d200700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0738074500 0x7f073819f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:35.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.984+0000 7f073d200700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0738074500 0x7f073819f110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:57106/0 (socket says 172.21.15.39:57106) 2024-03-18T16:08:35.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.984+0000 7f073d200700 1 -- 172.21.15.39:0/2194615024 learned_addr learned my addr 172.21.15.39:0/2194615024 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:08:35.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.984+0000 7f073dc00700 1 --2- 172.21.15.39:0/2194615024 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f07381134a0 0x7f07381a39e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:35.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.984+0000 7f0737e00700 1 --2- 172.21.15.39:0/2194615024 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0738074e20 0x7f073819f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:35.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.985+0000 7f073d200700 1 -- 172.21.15.39:0/2194615024 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0738074e20 msgr2=0x7f073819f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:35.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.985+0000 7f073d200700 1 --2- 172.21.15.39:0/2194615024 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0738074e20 0x7f073819f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:35.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.985+0000 7f073d200700 1 -- 172.21.15.39:0/2194615024 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f07381134a0 msgr2=0x7f07381a39e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:35.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.985+0000 7f073d200700 1 --2- 172.21.15.39:0/2194615024 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f07381134a0 0x7f07381a39e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:35.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.985+0000 7f073d200700 1 -- 172.21.15.39:0/2194615024 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07381a4120 con 0x7f0738074500 2024-03-18T16:08:35.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.985+0000 7f073d200700 1 --2- 172.21.15.39:0/2194615024 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0738074500 0x7f073819f110 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f072c004f00 tx=0x7f072c004f30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:35.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.986+0000 7f0735600700 1 -- 172.21.15.39:0/2194615024 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f072c021070 con 0x7f0738074500 2024-03-18T16:08:35.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.986+0000 7f0735600700 1 -- 172.21.15.39:0/2194615024 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f072c007ac0 con 0x7f0738074500 2024-03-18T16:08:35.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.986+0000 7f0735600700 1 -- 172.21.15.39:0/2194615024 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f072c0052a0 con 0x7f0738074500 2024-03-18T16:08:35.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.986+0000 7f073fc00700 1 -- 172.21.15.39:0/2194615024 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07381a43b0 con 0x7f0738074500 2024-03-18T16:08:35.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.986+0000 7f073fc00700 1 -- 172.21.15.39:0/2194615024 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07381abc90 con 0x7f0738074500 2024-03-18T16:08:35.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.987+0000 7f073fc00700 1 -- 172.21.15.39:0/2194615024 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0738075c30 con 0x7f0738074500 2024-03-18T16:08:35.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.989+0000 7f0735600700 1 -- 172.21.15.39:0/2194615024 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f072c024070 con 0x7f0738074500 2024-03-18T16:08:35.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.990+0000 7f0735600700 1 --2- 172.21.15.39:0/2194615024 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f07200a29a0 0x7f07200a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:35.183 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.990+0000 7f0735600700 1 -- 172.21.15.39:0/2194615024 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f072c00f020 con 0x7f0738074500 2024-03-18T16:08:35.183 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.990+0000 7f0737e00700 1 --2- 172.21.15.39:0/2194615024 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f07200a29a0 0x7f07200a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:35.183 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.991+0000 7f0737e00700 1 --2- 172.21.15.39:0/2194615024 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f07200a29a0 0x7f07200a4e60 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f073810fdc0 tx=0x7f0724009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:35.183 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.994+0000 7f0735600700 1 -- 172.21.15.39:0/2194615024 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f072c02b020 con 0x7f0738074500 2024-03-18T16:08:35.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:08:34 smithi196 ceph-mon[36307]: pgmap v375: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T16:08:35.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:08:34 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/3667602652' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-18T16:08:35.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:08:34 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/2467322931' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-18T16:08:35.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:08:34 smithi194 ceph-mon[36200]: pgmap v375: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T16:08:35.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:08:34 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/3667602652' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-18T16:08:35.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:08:34 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/2467322931' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-18T16:08:35.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:35.241+0000 7f073fc00700 1 -- 172.21.15.39:0/2194615024 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f07381a0470 con 0x7f0738074500 2024-03-18T16:08:35.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:35.241+0000 7f0735600700 1 -- 172.21.15.39:0/2194615024 <== mon.2 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f072c02e350 con 0x7f0738074500 2024-03-18T16:08:35.244 INFO:teuthology.orchestra.run.smithi039.stdout:55834574939 2024-03-18T16:08:35.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:35.244+0000 7f073fc00700 1 -- 172.21.15.39:0/2194615024 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f07200a29a0 msgr2=0x7f07200a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:35.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:35.244+0000 7f073fc00700 1 --2- 172.21.15.39:0/2194615024 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f07200a29a0 0x7f07200a4e60 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f073810fdc0 tx=0x7f0724009260 comp rx=0 tx=0).stop 2024-03-18T16:08:35.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:35.244+0000 7f073fc00700 1 -- 172.21.15.39:0/2194615024 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0738074500 msgr2=0x7f073819f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:35.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:35.244+0000 7f073fc00700 1 --2- 172.21.15.39:0/2194615024 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0738074500 0x7f073819f110 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f072c004f00 tx=0x7f072c004f30 comp rx=0 tx=0).stop 2024-03-18T16:08:35.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:35.244+0000 7f073fc00700 1 -- 172.21.15.39:0/2194615024 shutdown_connections 2024-03-18T16:08:35.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:35.244+0000 7f073fc00700 1 --2- 172.21.15.39:0/2194615024 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f07200a29a0 0x7f07200a4e60 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:35.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:35.244+0000 7f073fc00700 1 --2- 172.21.15.39:0/2194615024 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f07381134a0 0x7f07381a39e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:35.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:35.244+0000 7f073fc00700 1 --2- 172.21.15.39:0/2194615024 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0738074500 0x7f073819f110 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:35.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:35.244+0000 7f073fc00700 1 --2- 172.21.15.39:0/2194615024 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0738074e20 0x7f073819f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:35.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:35.244+0000 7f073fc00700 1 -- 172.21.15.39:0/2194615024 >> 172.21.15.39:0/2194615024 conn(0x7f0738079430 msgr2=0x7f0738112260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:08:35.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:35.245+0000 7f073fc00700 1 -- 172.21.15.39:0/2194615024 shutdown_connections 2024-03-18T16:08:35.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:35.245+0000 7f073fc00700 1 -- 172.21.15.39:0/2194615024 wait complete. 2024-03-18T16:08:35.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:31.694+0000 7f8722200700 1 -- 172.21.15.39:0/2959437966 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f871c108320 msgr2=0x7f871c108700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:35.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:31.694+0000 7f8722200700 1 --2- 172.21.15.39:0/2959437966 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f871c108320 0x7f871c108700 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f870c009990 tx=0x7f870c009ca0 comp rx=0 tx=0).stop 2024-03-18T16:08:35.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:31.694+0000 7f8722200700 1 -- 172.21.15.39:0/2959437966 shutdown_connections 2024-03-18T16:08:35.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:31.694+0000 7f8722200700 1 --2- 172.21.15.39:0/2959437966 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f871c06a920 0x7f871c10f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:35.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:31.694+0000 7f8722200700 1 --2- 172.21.15.39:0/2959437966 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f871c108320 0x7f871c108700 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:35.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:31.694+0000 7f8722200700 1 --2- 172.21.15.39:0/2959437966 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f871c069f80 0x7f871c06a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:35.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:31.694+0000 7f8722200700 1 -- 172.21.15.39:0/2959437966 >> 172.21.15.39:0/2959437966 conn(0x7f871c078100 msgr2=0x7f871c078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:08:35.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:31.694+0000 7f8722200700 1 -- 172.21.15.39:0/2959437966 shutdown_connections 2024-03-18T16:08:35.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:31.694+0000 7f8722200700 1 -- 172.21.15.39:0/2959437966 wait complete. 2024-03-18T16:08:35.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:31.695+0000 7f8722200700 1 Processor -- start 2024-03-18T16:08:35.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:31.695+0000 7f8722200700 1 -- start start 2024-03-18T16:08:35.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:31.696+0000 7f8722200700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f871c069f80 0x7f871c1a0840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:35.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:31.696+0000 7f8722200700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f871c06a920 0x7f871c1a0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:35.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:31.696+0000 7f8722200700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f871c108320 0x7f871c19aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:35.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:31.696+0000 7f8722200700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f871c112580 con 0x7f871c069f80 2024-03-18T16:08:35.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:31.696+0000 7f8722200700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f871c112400 con 0x7f871c108320 2024-03-18T16:08:35.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:31.696+0000 7f8722200700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f871c112700 con 0x7f871c06a920 2024-03-18T16:08:35.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:31.696+0000 7f871b400700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f871c069f80 0x7f871c1a0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:35.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:31.696+0000 7f871aa00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f871c06a920 0x7f871c1a0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:35.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:31.696+0000 7f871b400700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f871c069f80 0x7f871c1a0840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:32794/0 (socket says 172.21.15.39:32794) 2024-03-18T16:08:35.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:31.696+0000 7f871b400700 1 -- 172.21.15.39:0/605615859 learned_addr learned my addr 172.21.15.39:0/605615859 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:08:35.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:31.696+0000 7f871be00700 1 --2- 172.21.15.39:0/605615859 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f871c108320 0x7f871c19aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:35.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:31.696+0000 7f871aa00700 1 -- 172.21.15.39:0/605615859 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f871c108320 msgr2=0x7f871c19aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:35.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:31.696+0000 7f871aa00700 1 --2- 172.21.15.39:0/605615859 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f871c108320 0x7f871c19aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:35.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:31.696+0000 7f871aa00700 1 -- 172.21.15.39:0/605615859 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f871c069f80 msgr2=0x7f871c1a0840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:35.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:31.696+0000 7f871aa00700 1 --2- 172.21.15.39:0/605615859 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f871c069f80 0x7f871c1a0840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:35.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:31.696+0000 7f871aa00700 1 -- 172.21.15.39:0/605615859 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f871c19b160 con 0x7f871c06a920 2024-03-18T16:08:35.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:31.697+0000 7f871aa00700 1 --2- 172.21.15.39:0/605615859 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f871c06a920 0x7f871c1a0d80 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f870800e9f0 tx=0x7f870800ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:35.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:31.697+0000 7f8713e00700 1 -- 172.21.15.39:0/605615859 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8708004220 con 0x7f871c06a920 2024-03-18T16:08:35.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:31.697+0000 7f8713e00700 1 -- 172.21.15.39:0/605615859 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8708004d10 con 0x7f871c06a920 2024-03-18T16:08:35.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:31.697+0000 7f8722200700 1 -- 172.21.15.39:0/605615859 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f871c19b450 con 0x7f871c06a920 2024-03-18T16:08:35.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:31.697+0000 7f8713e00700 1 -- 172.21.15.39:0/605615859 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8708005020 con 0x7f871c06a920 2024-03-18T16:08:35.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:31.697+0000 7f8722200700 1 -- 172.21.15.39:0/605615859 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f871c1a7710 con 0x7f871c06a920 2024-03-18T16:08:35.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:31.699+0000 7f8722200700 1 -- 172.21.15.39:0/605615859 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f871c068f90 con 0x7f871c06a920 2024-03-18T16:08:35.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:31.701+0000 7f8713e00700 1 -- 172.21.15.39:0/605615859 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f8708004750 con 0x7f871c06a920 2024-03-18T16:08:35.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:31.702+0000 7f8713e00700 1 --2- 172.21.15.39:0/605615859 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f87000a29a0 0x7f87000a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:35.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:31.702+0000 7f8713e00700 1 -- 172.21.15.39:0/605615859 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f87080cede0 con 0x7f871c06a920 2024-03-18T16:08:35.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:31.702+0000 7f871b400700 1 --2- 172.21.15.39:0/605615859 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f87000a29a0 0x7f87000a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:35.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:31.703+0000 7f871b400700 1 --2- 172.21.15.39:0/605615859 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f87000a29a0 0x7f87000a4e60 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f870c006010 tx=0x7f870c009590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:35.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:31.706+0000 7f8713e00700 1 -- 172.21.15.39:0/605615859 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f870809b960 con 0x7f871c06a920 2024-03-18T16:08:35.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:31.961+0000 7f8722200700 1 -- 172.21.15.39:0/605615859 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f871c1a79a0 con 0x7f871c06a920 2024-03-18T16:08:35.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:31.961+0000 7f8713e00700 1 -- 172.21.15.39:0/605615859 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f870809b260 con 0x7f871c06a920 2024-03-18T16:08:35.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:31.964+0000 7f8722200700 1 -- 172.21.15.39:0/605615859 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f87000a29a0 msgr2=0x7f87000a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:35.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:31.964+0000 7f8722200700 1 --2- 172.21.15.39:0/605615859 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f87000a29a0 0x7f87000a4e60 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f870c006010 tx=0x7f870c009590 comp rx=0 tx=0).stop 2024-03-18T16:08:35.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:31.964+0000 7f8722200700 1 -- 172.21.15.39:0/605615859 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f871c06a920 msgr2=0x7f871c1a0d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:35.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:31.964+0000 7f8722200700 1 --2- 172.21.15.39:0/605615859 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f871c06a920 0x7f871c1a0d80 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f870800e9f0 tx=0x7f870800ed00 comp rx=0 tx=0).stop 2024-03-18T16:08:35.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:31.964+0000 7f8722200700 1 -- 172.21.15.39:0/605615859 shutdown_connections 2024-03-18T16:08:35.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:31.964+0000 7f8722200700 1 --2- 172.21.15.39:0/605615859 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f87000a29a0 0x7f87000a4e60 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:35.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:31.964+0000 7f8722200700 1 --2- 172.21.15.39:0/605615859 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f871c108320 0x7f871c19aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:35.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:31.964+0000 7f8722200700 1 --2- 172.21.15.39:0/605615859 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f871c069f80 0x7f871c1a0840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:35.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:31.964+0000 7f8722200700 1 --2- 172.21.15.39:0/605615859 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f871c06a920 0x7f871c1a0d80 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:35.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:31.964+0000 7f8722200700 1 -- 172.21.15.39:0/605615859 >> 172.21.15.39:0/605615859 conn(0x7f871c078100 msgr2=0x7f871c10d4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:08:35.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:31.965+0000 7f8722200700 1 -- 172.21.15.39:0/605615859 shutdown_connections 2024-03-18T16:08:35.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:31.965+0000 7f8722200700 1 -- 172.21.15.39:0/605615859 wait complete. 2024-03-18T16:08:35.764 INFO:teuthology.orchestra.run.smithi039.stdout:34359738465 2024-03-18T16:08:36.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:08:35 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/2194615024' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-18T16:08:36.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:08:35 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/2194615024' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-18T16:08:36.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:08:35 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/2194615024' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-18T16:08:36.904 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411408 got 77309411415 for osd.2 2024-03-18T16:08:36.904 DEBUG:teuthology.parallel:result is None 2024-03-18T16:08:36.930 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:08:36 smithi039 ceph-mon[31721]: pgmap v376: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T16:08:37.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:08:36 smithi196 ceph-mon[36307]: pgmap v376: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T16:08:37.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:08:36 smithi194 ceph-mon[36200]: pgmap v376: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T16:08:37.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.273+0000 7f3e07200700 1 -- 172.21.15.39:0/2511663287 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3e08101310 msgr2=0x7f3e081016f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:37.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.273+0000 7f3e07200700 1 --2- 172.21.15.39:0/2511663287 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3e08101310 0x7f3e081016f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f3df80099c0 tx=0x7f3df8009cd0 comp rx=0 tx=0).stop 2024-03-18T16:08:37.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.273+0000 7f3e07200700 1 -- 172.21.15.39:0/2511663287 shutdown_connections 2024-03-18T16:08:37.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.273+0000 7f3e07200700 1 --2- 172.21.15.39:0/2511663287 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3e081063f0 0x7f3e0810dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:37.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.273+0000 7f3e07200700 1 --2- 172.21.15.39:0/2511663287 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3e08101310 0x7f3e081016f0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:37.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.273+0000 7f3e07200700 1 --2- 172.21.15.39:0/2511663287 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3e08101cc0 0x7f3e08105d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:37.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.273+0000 7f3e07200700 1 -- 172.21.15.39:0/2511663287 >> 172.21.15.39:0/2511663287 conn(0x7f3e080fcba0 msgr2=0x7f3e080fefc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:08:37.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.273+0000 7f3e07200700 1 -- 172.21.15.39:0/2511663287 shutdown_connections 2024-03-18T16:08:37.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.274+0000 7f3e07200700 1 -- 172.21.15.39:0/2511663287 wait complete. 2024-03-18T16:08:37.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.275+0000 7f3e07200700 1 Processor -- start 2024-03-18T16:08:37.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.275+0000 7f3e07200700 1 -- start start 2024-03-18T16:08:37.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.276+0000 7f3e07200700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3e08101310 0x7f3e0810e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:37.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.276+0000 7f3e07200700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3e08101cc0 0x7f3e0810eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:37.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.276+0000 7f3e07200700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3e081063f0 0x7f3e0810f420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:37.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.276+0000 7f3e07200700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f3e08112530 con 0x7f3e08101310 2024-03-18T16:08:37.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.276+0000 7f3e07200700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f3e081123b0 con 0x7f3e08101cc0 2024-03-18T16:08:37.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.276+0000 7f3e07200700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f3e081126b0 con 0x7f3e081063f0 2024-03-18T16:08:37.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.276+0000 7f3e05400700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3e08101cc0 0x7f3e0810eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:37.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.276+0000 7f3e05e00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3e08101310 0x7f3e0810e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:37.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.276+0000 7f3e05400700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3e08101cc0 0x7f3e0810eee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:57076/0 (socket says 172.21.15.39:57076) 2024-03-18T16:08:37.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.276+0000 7f3e05400700 1 -- 172.21.15.39:0/3667602652 learned_addr learned my addr 172.21.15.39:0/3667602652 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:08:37.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.276+0000 7f3e05e00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3e08101310 0x7f3e0810e9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:32840/0 (socket says 172.21.15.39:32840) 2024-03-18T16:08:37.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.277+0000 7f3e05400700 1 -- 172.21.15.39:0/3667602652 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3e081063f0 msgr2=0x7f3e0810f420 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T16:08:37.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.277+0000 7f3e05400700 1 --2- 172.21.15.39:0/3667602652 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3e081063f0 0x7f3e0810f420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:37.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.277+0000 7f3e05400700 1 -- 172.21.15.39:0/3667602652 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3e08101310 msgr2=0x7f3e0810e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:37.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.277+0000 7f3e05400700 1 --2- 172.21.15.39:0/3667602652 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3e08101310 0x7f3e0810e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:37.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.277+0000 7f3e05400700 1 -- 172.21.15.39:0/3667602652 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e081a6e90 con 0x7f3e08101cc0 2024-03-18T16:08:37.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.277+0000 7f3e05400700 1 --2- 172.21.15.39:0/3667602652 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3e08101cc0 0x7f3e0810eee0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3df400ea30 tx=0x7f3df400ed40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:37.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.277+0000 7f3dfea00700 1 -- 172.21.15.39:0/3667602652 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3df400cbd0 con 0x7f3e08101cc0 2024-03-18T16:08:37.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.277+0000 7f3dfea00700 1 -- 172.21.15.39:0/3667602652 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3df4004500 con 0x7f3e08101cc0 2024-03-18T16:08:37.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.277+0000 7f3dfea00700 1 -- 172.21.15.39:0/3667602652 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3df4010430 con 0x7f3e08101cc0 2024-03-18T16:08:37.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.277+0000 7f3e07200700 1 -- 172.21.15.39:0/3667602652 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e081a7180 con 0x7f3e08101cc0 2024-03-18T16:08:37.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.278+0000 7f3e07200700 1 -- 172.21.15.39:0/3667602652 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e081a7670 con 0x7f3e08101cc0 2024-03-18T16:08:37.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.279+0000 7f3e07200700 1 -- 172.21.15.39:0/3667602652 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e08068f90 con 0x7f3e08101cc0 2024-03-18T16:08:37.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.281+0000 7f3dfea00700 1 -- 172.21.15.39:0/3667602652 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f3df4004020 con 0x7f3e08101cc0 2024-03-18T16:08:37.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.282+0000 7f3dfea00700 1 --2- 172.21.15.39:0/3667602652 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3dec0a29a0 0x7f3dec0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:37.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.282+0000 7f3dfea00700 1 -- 172.21.15.39:0/3667602652 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f3df40cf0d0 con 0x7f3e08101cc0 2024-03-18T16:08:37.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.282+0000 7f3e05e00700 1 --2- 172.21.15.39:0/3667602652 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3dec0a29a0 0x7f3dec0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:37.256 INFO:teuthology.orchestra.run.smithi039.stdout:253403070518 2024-03-18T16:08:37.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.283+0000 7f3e05e00700 1 --2- 172.21.15.39:0/3667602652 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3dec0a29a0 0x7f3dec0a4e60 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f3df8000c00 tx=0x7f3df8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:37.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.286+0000 7f3dfea00700 1 -- 172.21.15.39:0/3667602652 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3df409bc50 con 0x7f3e08101cc0 2024-03-18T16:08:37.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.541+0000 7f3e07200700 1 -- 172.21.15.39:0/3667602652 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7f3e0810fed0 con 0x7f3e08101cc0 2024-03-18T16:08:37.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.541+0000 7f3dfea00700 1 -- 172.21.15.39:0/3667602652 <== mon.2 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f3df409b550 con 0x7f3e08101cc0 2024-03-18T16:08:37.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.545+0000 7f3e07200700 1 -- 172.21.15.39:0/3667602652 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3dec0a29a0 msgr2=0x7f3dec0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:37.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.545+0000 7f3e07200700 1 --2- 172.21.15.39:0/3667602652 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3dec0a29a0 0x7f3dec0a4e60 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f3df8000c00 tx=0x7f3df8005fb0 comp rx=0 tx=0).stop 2024-03-18T16:08:37.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.545+0000 7f3e07200700 1 -- 172.21.15.39:0/3667602652 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3e08101cc0 msgr2=0x7f3e0810eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:37.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.545+0000 7f3e07200700 1 --2- 172.21.15.39:0/3667602652 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3e08101cc0 0x7f3e0810eee0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3df400ea30 tx=0x7f3df400ed40 comp rx=0 tx=0).stop 2024-03-18T16:08:37.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.545+0000 7f3e07200700 1 -- 172.21.15.39:0/3667602652 shutdown_connections 2024-03-18T16:08:37.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.545+0000 7f3e07200700 1 --2- 172.21.15.39:0/3667602652 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3dec0a29a0 0x7f3dec0a4e60 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:37.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.545+0000 7f3e07200700 1 --2- 172.21.15.39:0/3667602652 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3e081063f0 0x7f3e0810f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:37.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.545+0000 7f3e07200700 1 --2- 172.21.15.39:0/3667602652 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3e08101310 0x7f3e0810e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:37.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.545+0000 7f3e07200700 1 --2- 172.21.15.39:0/3667602652 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3e08101cc0 0x7f3e0810eee0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:37.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.545+0000 7f3e07200700 1 -- 172.21.15.39:0/3667602652 >> 172.21.15.39:0/3667602652 conn(0x7f3e080fcba0 msgr2=0x7f3e080fe910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:08:37.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.545+0000 7f3e07200700 1 -- 172.21.15.39:0/3667602652 shutdown_connections 2024-03-18T16:08:37.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:34.545+0000 7f3e07200700 1 -- 172.21.15.39:0/3667602652 wait complete. 2024-03-18T16:08:38.033 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:08:37 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/3081443511' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-18T16:08:38.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:08:37 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/3081443511' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-18T16:08:38.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:08:37 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/3081443511' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-18T16:08:38.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:36.664+0000 7f6952000700 1 -- 172.21.15.39:0/3126648547 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f694c108320 msgr2=0x7f694c108700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:38.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:36.664+0000 7f6952000700 1 --2- 172.21.15.39:0/3126648547 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f694c108320 0x7f694c108700 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f693c009920 tx=0x7f693c009c30 comp rx=0 tx=0).stop 2024-03-18T16:08:38.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:36.664+0000 7f6952000700 1 -- 172.21.15.39:0/3126648547 shutdown_connections 2024-03-18T16:08:38.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:36.664+0000 7f6952000700 1 --2- 172.21.15.39:0/3126648547 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f694c06a920 0x7f694c10f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:38.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:36.664+0000 7f6952000700 1 --2- 172.21.15.39:0/3126648547 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f694c108320 0x7f694c108700 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:38.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:36.664+0000 7f6952000700 1 --2- 172.21.15.39:0/3126648547 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f694c069f80 0x7f694c06a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:38.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:36.664+0000 7f6952000700 1 -- 172.21.15.39:0/3126648547 >> 172.21.15.39:0/3126648547 conn(0x7f694c078100 msgr2=0x7f694c078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:08:38.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:36.664+0000 7f6952000700 1 -- 172.21.15.39:0/3126648547 shutdown_connections 2024-03-18T16:08:38.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:36.664+0000 7f6952000700 1 -- 172.21.15.39:0/3126648547 wait complete. 2024-03-18T16:08:38.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:36.666+0000 7f6952000700 1 Processor -- start 2024-03-18T16:08:38.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:36.666+0000 7f6952000700 1 -- start start 2024-03-18T16:08:38.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:36.666+0000 7f6952000700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f694c069f80 0x7f694c1012a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:38.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:36.666+0000 7f6952000700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f694c06a920 0x7f694c1017e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:38.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:36.666+0000 7f6952000700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f694c108320 0x7f694c101d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:38.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:36.666+0000 7f6952000700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f694c1125b0 con 0x7f694c069f80 2024-03-18T16:08:38.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:36.666+0000 7f6952000700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f694c112430 con 0x7f694c06a920 2024-03-18T16:08:38.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:36.666+0000 7f6952000700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f694c112730 con 0x7f694c108320 2024-03-18T16:08:38.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:36.667+0000 7f694b400700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f694c069f80 0x7f694c1012a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:38.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:36.667+0000 7f694b400700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f694c069f80 0x7f694c1012a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:32892/0 (socket says 172.21.15.39:32892) 2024-03-18T16:08:38.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:36.667+0000 7f694b400700 1 -- 172.21.15.39:0/3081443511 learned_addr learned my addr 172.21.15.39:0/3081443511 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:08:38.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:36.667+0000 7f694aa00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f694c06a920 0x7f694c1017e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:38.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:36.667+0000 7f694be00700 1 --2- 172.21.15.39:0/3081443511 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f694c108320 0x7f694c101d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:38.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:36.667+0000 7f694aa00700 1 -- 172.21.15.39:0/3081443511 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f694c108320 msgr2=0x7f694c101d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:38.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:36.667+0000 7f694aa00700 1 --2- 172.21.15.39:0/3081443511 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f694c108320 0x7f694c101d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:38.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:36.667+0000 7f694aa00700 1 -- 172.21.15.39:0/3081443511 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f694c069f80 msgr2=0x7f694c1012a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:38.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:36.667+0000 7f694aa00700 1 --2- 172.21.15.39:0/3081443511 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f694c069f80 0x7f694c1012a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:38.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:36.667+0000 7f694aa00700 1 -- 172.21.15.39:0/3081443511 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f694c1a7030 con 0x7f694c06a920 2024-03-18T16:08:38.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:36.667+0000 7f694b400700 1 --2- 172.21.15.39:0/3081443511 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f694c069f80 0x7f694c1012a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:08:38.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:36.667+0000 7f694aa00700 1 --2- 172.21.15.39:0/3081443511 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f694c06a920 0x7f694c1017e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f693800e9c0 tx=0x7f693800ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:38.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:36.668+0000 7f6943e00700 1 -- 172.21.15.39:0/3081443511 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f693800c9c0 con 0x7f694c06a920 2024-03-18T16:08:38.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:36.668+0000 7f6943e00700 1 -- 172.21.15.39:0/3081443511 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f693800cb60 con 0x7f694c06a920 2024-03-18T16:08:38.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:36.668+0000 7f6952000700 1 -- 172.21.15.39:0/3081443511 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f694c1a7320 con 0x7f694c06a920 2024-03-18T16:08:38.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:36.668+0000 7f6943e00700 1 -- 172.21.15.39:0/3081443511 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6938018680 con 0x7f694c06a920 2024-03-18T16:08:38.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:36.668+0000 7f6952000700 1 -- 172.21.15.39:0/3081443511 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f694c1a78b0 con 0x7f694c06a920 2024-03-18T16:08:38.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:36.669+0000 7f6952000700 1 -- 172.21.15.39:0/3081443511 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f694c068f90 con 0x7f694c06a920 2024-03-18T16:08:38.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:36.672+0000 7f6943e00700 1 -- 172.21.15.39:0/3081443511 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f693800cd10 con 0x7f694c06a920 2024-03-18T16:08:38.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:36.674+0000 7f6943e00700 1 --2- 172.21.15.39:0/3081443511 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f69300a29a0 0x7f69300a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:38.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:36.674+0000 7f6943e00700 1 -- 172.21.15.39:0/3081443511 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f69380cefe0 con 0x7f694c06a920 2024-03-18T16:08:38.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:36.674+0000 7f694b400700 1 --2- 172.21.15.39:0/3081443511 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f69300a29a0 0x7f69300a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:38.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:36.675+0000 7f694b400700 1 --2- 172.21.15.39:0/3081443511 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f69300a29a0 0x7f69300a4e60 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f693c009680 tx=0x7f693c019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:38.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:36.678+0000 7f6943e00700 1 -- 172.21.15.39:0/3081443511 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f693809bba0 con 0x7f694c06a920 2024-03-18T16:08:38.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:36.928+0000 7f6952000700 1 -- 172.21.15.39:0/3081443511 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f694c02db70 con 0x7f694c06a920 2024-03-18T16:08:38.457 INFO:teuthology.orchestra.run.smithi039.stdout:141733920844 2024-03-18T16:08:38.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:36.928+0000 7f6943e00700 1 -- 172.21.15.39:0/3081443511 <== mon.2 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f693809b4a0 con 0x7f694c06a920 2024-03-18T16:08:38.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:36.932+0000 7f6952000700 1 -- 172.21.15.39:0/3081443511 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f69300a29a0 msgr2=0x7f69300a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:38.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:36.932+0000 7f6952000700 1 --2- 172.21.15.39:0/3081443511 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f69300a29a0 0x7f69300a4e60 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f693c009680 tx=0x7f693c019040 comp rx=0 tx=0).stop 2024-03-18T16:08:38.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:36.932+0000 7f6952000700 1 -- 172.21.15.39:0/3081443511 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f694c06a920 msgr2=0x7f694c1017e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:38.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:36.932+0000 7f6952000700 1 --2- 172.21.15.39:0/3081443511 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f694c06a920 0x7f694c1017e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f693800e9c0 tx=0x7f693800ecd0 comp rx=0 tx=0).stop 2024-03-18T16:08:38.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:36.932+0000 7f6952000700 1 -- 172.21.15.39:0/3081443511 shutdown_connections 2024-03-18T16:08:38.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:36.932+0000 7f6952000700 1 --2- 172.21.15.39:0/3081443511 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f69300a29a0 0x7f69300a4e60 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:38.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:36.932+0000 7f6952000700 1 --2- 172.21.15.39:0/3081443511 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f694c108320 0x7f694c101d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:38.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:36.932+0000 7f6952000700 1 --2- 172.21.15.39:0/3081443511 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f694c069f80 0x7f694c1012a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:38.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:36.932+0000 7f6952000700 1 --2- 172.21.15.39:0/3081443511 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f694c06a920 0x7f694c1017e0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:38.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:36.932+0000 7f6952000700 1 -- 172.21.15.39:0/3081443511 >> 172.21.15.39:0/3081443511 conn(0x7f694c078100 msgr2=0x7f694c0fff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:08:38.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:36.932+0000 7f6952000700 1 -- 172.21.15.39:0/3081443511 shutdown_connections 2024-03-18T16:08:38.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:36.932+0000 7f6952000700 1 -- 172.21.15.39:0/3081443511 wait complete. 2024-03-18T16:08:38.995 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574933 got 55834574939 for osd.1 2024-03-18T16:08:38.995 DEBUG:teuthology.parallel:result is None 2024-03-18T16:08:39.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:08:38 smithi039 ceph-mon[31721]: pgmap v377: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T16:08:39.227 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:08:38 smithi196 ceph-mon[36307]: pgmap v377: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T16:08:39.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:08:38 smithi194 ceph-mon[36200]: pgmap v377: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T16:08:39.620 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070511 got 253403070518 for osd.8 2024-03-18T16:08:39.620 DEBUG:teuthology.parallel:result is None 2024-03-18T16:08:40.078 INFO:tasks.cephadm.ceph_manager.ceph:need seq 309237645348 got 309237645352 for osd.10 2024-03-18T16:08:40.078 DEBUG:teuthology.parallel:result is None 2024-03-18T16:08:40.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:40.058+0000 7f3e16200700 1 -- 172.21.15.39:0/4190337394 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3e10108320 msgr2=0x7f3e10108700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:40.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:40.058+0000 7f3e16200700 1 --2- 172.21.15.39:0/4190337394 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3e10108320 0x7f3e10108700 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f3e0000b0a0 tx=0x7f3e0000b3b0 comp rx=0 tx=0).stop 2024-03-18T16:08:40.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:40.058+0000 7f3e16200700 1 -- 172.21.15.39:0/4190337394 shutdown_connections 2024-03-18T16:08:40.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:40.058+0000 7f3e16200700 1 --2- 172.21.15.39:0/4190337394 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3e1006a920 0x7f3e1010f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:40.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:40.058+0000 7f3e16200700 1 --2- 172.21.15.39:0/4190337394 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3e10108320 0x7f3e10108700 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:40.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:40.058+0000 7f3e16200700 1 --2- 172.21.15.39:0/4190337394 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3e10069f80 0x7f3e1006a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:40.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:40.058+0000 7f3e16200700 1 -- 172.21.15.39:0/4190337394 >> 172.21.15.39:0/4190337394 conn(0x7f3e10078100 msgr2=0x7f3e10078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:08:40.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:40.058+0000 7f3e16200700 1 -- 172.21.15.39:0/4190337394 shutdown_connections 2024-03-18T16:08:40.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:40.058+0000 7f3e16200700 1 -- 172.21.15.39:0/4190337394 wait complete. 2024-03-18T16:08:40.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:40.059+0000 7f3e16200700 1 Processor -- start 2024-03-18T16:08:40.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:40.060+0000 7f3e16200700 1 -- start start 2024-03-18T16:08:40.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:40.060+0000 7f3e16200700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3e10069f80 0x7f3e101a0880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:40.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:40.060+0000 7f3e16200700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3e1006a920 0x7f3e101a0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:40.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:40.060+0000 7f3e16200700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3e10108320 0x7f3e1019aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:40.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:40.060+0000 7f3e16200700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f3e101125b0 con 0x7f3e10108320 2024-03-18T16:08:40.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:40.060+0000 7f3e16200700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f3e10112430 con 0x7f3e10069f80 2024-03-18T16:08:40.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:40.060+0000 7f3e16200700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f3e10112730 con 0x7f3e1006a920 2024-03-18T16:08:40.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:40.060+0000 7f3e0f400700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3e10069f80 0x7f3e101a0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:40.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:40.060+0000 7f3e0ea00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3e1006a920 0x7f3e101a0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:40.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:40.060+0000 7f3e0f400700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3e10069f80 0x7f3e101a0880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:38302/0 (socket says 172.21.15.39:38302) 2024-03-18T16:08:40.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:40.060+0000 7f3e0f400700 1 -- 172.21.15.39:0/3600996381 learned_addr learned my addr 172.21.15.39:0/3600996381 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:08:40.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:40.061+0000 7f3e0fe00700 1 --2- 172.21.15.39:0/3600996381 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3e10108320 0x7f3e1019aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:40.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:40.061+0000 7f3e0f400700 1 -- 172.21.15.39:0/3600996381 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3e1006a920 msgr2=0x7f3e101a0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:40.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:40.061+0000 7f3e0f400700 1 --2- 172.21.15.39:0/3600996381 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3e1006a920 0x7f3e101a0dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:40.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:40.061+0000 7f3e0f400700 1 -- 172.21.15.39:0/3600996381 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3e10108320 msgr2=0x7f3e1019aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:40.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:40.061+0000 7f3e0f400700 1 --2- 172.21.15.39:0/3600996381 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3e10108320 0x7f3e1019aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:40.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:40.061+0000 7f3e0f400700 1 -- 172.21.15.39:0/3600996381 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e1019b1a0 con 0x7f3e10069f80 2024-03-18T16:08:40.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:40.061+0000 7f3e0ea00700 1 --2- 172.21.15.39:0/3600996381 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3e1006a920 0x7f3e101a0dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:08:40.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:40.061+0000 7f3e0f400700 1 --2- 172.21.15.39:0/3600996381 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3e10069f80 0x7f3e101a0880 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f3e000062a0 tx=0x7f3e00004ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:40.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:40.061+0000 7f3e07e00700 1 -- 172.21.15.39:0/3600996381 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e00021070 con 0x7f3e10069f80 2024-03-18T16:08:40.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:40.062+0000 7f3e07e00700 1 -- 172.21.15.39:0/3600996381 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e00007b80 con 0x7f3e10069f80 2024-03-18T16:08:40.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:40.062+0000 7f3e16200700 1 -- 172.21.15.39:0/3600996381 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e1019b430 con 0x7f3e10069f80 2024-03-18T16:08:40.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:40.062+0000 7f3e07e00700 1 -- 172.21.15.39:0/3600996381 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e000052a0 con 0x7f3e10069f80 2024-03-18T16:08:40.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:40.062+0000 7f3e16200700 1 -- 172.21.15.39:0/3600996381 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e101a7750 con 0x7f3e10069f80 2024-03-18T16:08:40.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:40.063+0000 7f3e05600700 1 -- 172.21.15.39:0/3600996381 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3de8005330 con 0x7f3e10069f80 2024-03-18T16:08:40.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:40.065+0000 7f3e07e00700 1 -- 172.21.15.39:0/3600996381 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7f3e00024070 con 0x7f3e10069f80 2024-03-18T16:08:40.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:40.066+0000 7f3e07e00700 1 --2- 172.21.15.39:0/3600996381 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3df40a29a0 0x7f3df40a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:40.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:40.066+0000 7f3e07e00700 1 -- 172.21.15.39:0/3600996381 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f3e0000f020 con 0x7f3e10069f80 2024-03-18T16:08:40.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:40.066+0000 7f3e0ea00700 1 --2- 172.21.15.39:0/3600996381 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3df40a29a0 0x7f3df40a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:40.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:40.067+0000 7f3e0ea00700 1 --2- 172.21.15.39:0/3600996381 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3df40a29a0 0x7f3df40a4e60 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f3e1010f5a0 tx=0x7f3df8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:40.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:40.070+0000 7f3e07e00700 1 -- 172.21.15.39:0/3600996381 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3e0002b020 con 0x7f3e10069f80 2024-03-18T16:08:40.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:40.299+0000 7f3e05600700 1 -- 172.21.15.39:0/3600996381 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f3de80058f0 con 0x7f3e10069f80 2024-03-18T16:08:40.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:40.300+0000 7f3e07e00700 1 -- 172.21.15.39:0/3600996381 <== mon.2 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f3e0009d0a0 con 0x7f3e10069f80 2024-03-18T16:08:40.302 INFO:teuthology.orchestra.run.smithi039.stdout:197568495680 2024-03-18T16:08:40.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:40.303+0000 7f3e05600700 1 -- 172.21.15.39:0/3600996381 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3df40a29a0 msgr2=0x7f3df40a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:40.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:40.303+0000 7f3e05600700 1 --2- 172.21.15.39:0/3600996381 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3df40a29a0 0x7f3df40a4e60 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f3e1010f5a0 tx=0x7f3df8009260 comp rx=0 tx=0).stop 2024-03-18T16:08:40.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:40.303+0000 7f3e05600700 1 -- 172.21.15.39:0/3600996381 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3e10069f80 msgr2=0x7f3e101a0880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:40.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:40.303+0000 7f3e05600700 1 --2- 172.21.15.39:0/3600996381 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3e10069f80 0x7f3e101a0880 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f3e000062a0 tx=0x7f3e00004ce0 comp rx=0 tx=0).stop 2024-03-18T16:08:40.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:40.303+0000 7f3e05600700 1 -- 172.21.15.39:0/3600996381 shutdown_connections 2024-03-18T16:08:40.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:40.303+0000 7f3e05600700 1 --2- 172.21.15.39:0/3600996381 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3df40a29a0 0x7f3df40a4e60 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:40.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:40.303+0000 7f3e05600700 1 --2- 172.21.15.39:0/3600996381 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3e10108320 0x7f3e1019aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:40.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:40.303+0000 7f3e05600700 1 --2- 172.21.15.39:0/3600996381 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3e10069f80 0x7f3e101a0880 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:40.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:40.303+0000 7f3e05600700 1 --2- 172.21.15.39:0/3600996381 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3e1006a920 0x7f3e101a0dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:40.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:40.303+0000 7f3e05600700 1 -- 172.21.15.39:0/3600996381 >> 172.21.15.39:0/3600996381 conn(0x7f3e10078100 msgr2=0x7f3e100ffc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:08:40.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:40.303+0000 7f3e05600700 1 -- 172.21.15.39:0/3600996381 shutdown_connections 2024-03-18T16:08:40.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:40.303+0000 7f3e05600700 1 -- 172.21.15.39:0/3600996381 wait complete. 2024-03-18T16:08:40.569 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738460 got 34359738465 for osd.0 2024-03-18T16:08:40.569 DEBUG:teuthology.parallel:result is None 2024-03-18T16:08:41.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:08:40 smithi039 ceph-mon[31721]: pgmap v378: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T16:08:41.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:08:40 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/3600996381' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-18T16:08:41.226 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332146 got 219043332153 for osd.7 2024-03-18T16:08:41.227 DEBUG:teuthology.parallel:result is None 2024-03-18T16:08:41.227 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:08:40 smithi196 ceph-mon[36307]: pgmap v378: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T16:08:41.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:08:40 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/3600996381' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-18T16:08:41.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:08:40 smithi194 ceph-mon[36200]: pgmap v378: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T16:08:41.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:08:40 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/3600996381' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-18T16:08:41.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:41.435+0000 7fe0a3600700 1 -- 172.21.15.39:0/2672996108 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe09c074500 msgr2=0x7fe09c0748e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:41.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:41.435+0000 7fe0a3600700 1 --2- 172.21.15.39:0/2672996108 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe09c074500 0x7fe09c0748e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fe090009920 tx=0x7fe090009c30 comp rx=0 tx=0).stop 2024-03-18T16:08:41.438 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T16:08:41.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:41.436+0000 7fe0a3600700 1 -- 172.21.15.39:0/2672996108 shutdown_connections 2024-03-18T16:08:41.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:41.436+0000 7fe0a3600700 1 --2- 172.21.15.39:0/2672996108 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe09c1134a0 0x7fe09c115890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:41.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:41.436+0000 7fe0a3600700 1 --2- 172.21.15.39:0/2672996108 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe09c074500 0x7fe09c0748e0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:41.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:41.436+0000 7fe0a3600700 1 --2- 172.21.15.39:0/2672996108 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe09c074e20 0x7fe09c112f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:41.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:41.436+0000 7fe0a3600700 1 -- 172.21.15.39:0/2672996108 >> 172.21.15.39:0/2672996108 conn(0x7fe09c079430 msgr2=0x7fe09c079840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:08:41.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:41.436+0000 7fe0a3600700 1 -- 172.21.15.39:0/2672996108 shutdown_connections 2024-03-18T16:08:41.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:41.436+0000 7fe0a3600700 1 -- 172.21.15.39:0/2672996108 wait complete. 2024-03-18T16:08:41.440 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:41.438+0000 7fe0a3600700 1 Processor -- start 2024-03-18T16:08:41.440 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:41.438+0000 7fe0a3600700 1 -- start start 2024-03-18T16:08:41.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:41.438+0000 7fe0a3600700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe09c074500 0x7fe09c19f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:41.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:41.438+0000 7fe0a3600700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe09c074e20 0x7fe09c19f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:41.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:41.438+0000 7fe0a3600700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe09c1134a0 0x7fe09c1a3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:41.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:41.438+0000 7fe0a3600700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fe09c118c90 con 0x7fe09c1134a0 2024-03-18T16:08:41.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:41.438+0000 7fe0a3600700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fe09c118b10 con 0x7fe09c074500 2024-03-18T16:08:41.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:41.439+0000 7fe0a3600700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fe09c118e10 con 0x7fe09c074e20 2024-03-18T16:08:41.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:41.439+0000 7fe0a0c00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe09c074500 0x7fe09c19f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:41.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:41.439+0000 7fe0a0c00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe09c074500 0x7fe09c19f1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:38314/0 (socket says 172.21.15.39:38314) 2024-03-18T16:08:41.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:41.439+0000 7fe0a0c00700 1 -- 172.21.15.39:0/1018634402 learned_addr learned my addr 172.21.15.39:0/1018634402 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:08:41.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:41.439+0000 7fe0a1600700 1 --2- 172.21.15.39:0/1018634402 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe09c1134a0 0x7fe09c1a3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:41.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:41.439+0000 7fe09be00700 1 --2- 172.21.15.39:0/1018634402 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe09c074e20 0x7fe09c19f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:41.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:41.439+0000 7fe09be00700 1 -- 172.21.15.39:0/1018634402 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe09c074500 msgr2=0x7fe09c19f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:41.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:41.440+0000 7fe09be00700 1 --2- 172.21.15.39:0/1018634402 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe09c074500 0x7fe09c19f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:41.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:41.440+0000 7fe09be00700 1 -- 172.21.15.39:0/1018634402 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe09c1134a0 msgr2=0x7fe09c1a3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:41.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:41.440+0000 7fe09be00700 1 --2- 172.21.15.39:0/1018634402 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe09c1134a0 0x7fe09c1a3ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:41.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:41.440+0000 7fe09be00700 1 -- 172.21.15.39:0/1018634402 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe09c1a41f0 con 0x7fe09c074e20 2024-03-18T16:08:41.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:41.440+0000 7fe0a1600700 1 --2- 172.21.15.39:0/1018634402 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe09c1134a0 0x7fe09c1a3ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:08:41.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:41.440+0000 7fe0a0c00700 1 --2- 172.21.15.39:0/1018634402 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe09c074500 0x7fe09c19f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:08:41.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:41.440+0000 7fe09be00700 1 --2- 172.21.15.39:0/1018634402 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe09c074e20 0x7fe09c19f720 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fe088005fd0 tx=0x7fe08800ee70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:41.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:41.440+0000 7fe099600700 1 -- 172.21.15.39:0/1018634402 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe08800cbd0 con 0x7fe09c074e20 2024-03-18T16:08:41.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:41.440+0000 7fe099600700 1 -- 172.21.15.39:0/1018634402 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe08800cd70 con 0x7fe09c074e20 2024-03-18T16:08:41.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:41.440+0000 7fe0a3600700 1 -- 172.21.15.39:0/1018634402 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe09c1a44e0 con 0x7fe09c074e20 2024-03-18T16:08:41.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:41.440+0000 7fe099600700 1 -- 172.21.15.39:0/1018634402 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0880188b0 con 0x7fe09c074e20 2024-03-18T16:08:41.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:41.440+0000 7fe0a3600700 1 -- 172.21.15.39:0/1018634402 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe09c1abd60 con 0x7fe09c074e20 2024-03-18T16:08:41.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:41.442+0000 7fe0a3600700 1 -- 172.21.15.39:0/1018634402 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe09c068f90 con 0x7fe09c074e20 2024-03-18T16:08:41.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:41.444+0000 7fe099600700 1 -- 172.21.15.39:0/1018634402 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7fe08801f020 con 0x7fe09c074e20 2024-03-18T16:08:41.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:41.446+0000 7fe099600700 1 --2- 172.21.15.39:0/1018634402 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fe0840a29a0 0x7fe0840a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:41.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:41.446+0000 7fe099600700 1 -- 172.21.15.39:0/1018634402 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7fe0880ceec0 con 0x7fe09c074e20 2024-03-18T16:08:41.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:41.446+0000 7fe0a0c00700 1 --2- 172.21.15.39:0/1018634402 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fe0840a29a0 0x7fe0840a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:41.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:41.447+0000 7fe0a0c00700 1 --2- 172.21.15.39:0/1018634402 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fe0840a29a0 0x7fe0840a4e60 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fe090009680 tx=0x7fe090019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:41.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:41.449+0000 7fe099600700 1 -- 172.21.15.39:0/1018634402 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe08809ba40 con 0x7fe09c074e20 2024-03-18T16:08:41.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:41.700+0000 7fe0a3600700 1 -- 172.21.15.39:0/1018634402 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7fe09c1abff0 con 0x7fe09c074e20 2024-03-18T16:08:41.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:41.700+0000 7fe099600700 1 -- 172.21.15.39:0/1018634402 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fe08809b340 con 0x7fe09c074e20 2024-03-18T16:08:41.703 INFO:teuthology.orchestra.run.smithi039.stdout:287762808879 2024-03-18T16:08:41.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:41.703+0000 7fe0a3600700 1 -- 172.21.15.39:0/1018634402 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fe0840a29a0 msgr2=0x7fe0840a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:41.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:41.703+0000 7fe0a3600700 1 --2- 172.21.15.39:0/1018634402 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fe0840a29a0 0x7fe0840a4e60 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fe090009680 tx=0x7fe090019040 comp rx=0 tx=0).stop 2024-03-18T16:08:41.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:41.703+0000 7fe0a3600700 1 -- 172.21.15.39:0/1018634402 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe09c074e20 msgr2=0x7fe09c19f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:41.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:41.703+0000 7fe0a3600700 1 --2- 172.21.15.39:0/1018634402 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe09c074e20 0x7fe09c19f720 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fe088005fd0 tx=0x7fe08800ee70 comp rx=0 tx=0).stop 2024-03-18T16:08:41.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:41.704+0000 7fe0a3600700 1 -- 172.21.15.39:0/1018634402 shutdown_connections 2024-03-18T16:08:41.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:41.704+0000 7fe0a3600700 1 --2- 172.21.15.39:0/1018634402 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fe0840a29a0 0x7fe0840a4e60 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:41.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:41.704+0000 7fe0a3600700 1 --2- 172.21.15.39:0/1018634402 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe09c1134a0 0x7fe09c1a3ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:41.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:41.704+0000 7fe0a3600700 1 --2- 172.21.15.39:0/1018634402 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe09c074500 0x7fe09c19f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:41.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:41.704+0000 7fe0a3600700 1 --2- 172.21.15.39:0/1018634402 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe09c074e20 0x7fe09c19f720 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:41.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:41.704+0000 7fe0a3600700 1 -- 172.21.15.39:0/1018634402 >> 172.21.15.39:0/1018634402 conn(0x7fe09c079430 msgr2=0x7fe09c1122d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:08:41.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:41.704+0000 7fe0a3600700 1 -- 172.21.15.39:0/1018634402 shutdown_connections 2024-03-18T16:08:41.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:41.704+0000 7fe0a3600700 1 -- 172.21.15.39:0/1018634402 wait complete. 2024-03-18T16:08:41.989 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:08:41 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/1018634402' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-18T16:08:42.040 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920837 got 141733920844 for osd.4 2024-03-18T16:08:42.040 DEBUG:teuthology.parallel:result is None 2024-03-18T16:08:42.227 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:08:41 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/1018634402' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-18T16:08:42.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:08:41 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/1018634402' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-18T16:08:43.181 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:08:42 smithi039 ceph-mon[31721]: pgmap v379: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-18T16:08:43.185 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495673 got 197568495680 for osd.6 2024-03-18T16:08:43.185 DEBUG:teuthology.parallel:result is None 2024-03-18T16:08:43.227 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:08:42 smithi196 ceph-mon[36307]: pgmap v379: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-18T16:08:43.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:08:42 smithi194 ceph-mon[36200]: pgmap v379: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-18T16:08:44.151 INFO:tasks.cephadm.ceph_manager.ceph:need seq 287762808871 got 287762808879 for osd.9 2024-03-18T16:08:44.151 DEBUG:teuthology.parallel:result is None 2024-03-18T16:08:45.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:08:44 smithi039 ceph-mon[31721]: pgmap v380: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T16:08:45.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:08:44 smithi196 ceph-mon[36307]: pgmap v380: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T16:08:45.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:08:44 smithi194 ceph-mon[36200]: pgmap v380: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T16:08:45.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:45.495+0000 7feb76200700 1 -- 172.21.15.39:0/3316070268 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feb68004d10 msgr2=0x7feb68005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:45.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:45.495+0000 7feb76200700 1 --2- 172.21.15.39:0/3316070268 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feb68004d10 0x7feb68005190 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7feb640098f0 tx=0x7feb64009c00 comp rx=0 tx=0).stop 2024-03-18T16:08:45.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:45.498+0000 7feb76200700 1 -- 172.21.15.39:0/3316070268 shutdown_connections 2024-03-18T16:08:45.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:45.498+0000 7feb76200700 1 --2- 172.21.15.39:0/3316070268 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7feb68098720 0x7feb68098ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:45.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:45.498+0000 7feb76200700 1 --2- 172.21.15.39:0/3316070268 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7feb6809ad20 0x7feb680047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:45.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:45.498+0000 7feb76200700 1 --2- 172.21.15.39:0/3316070268 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feb68004d10 0x7feb68005190 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:45.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:45.499+0000 7feb76200700 1 -- 172.21.15.39:0/3316070268 >> 172.21.15.39:0/3316070268 conn(0x7feb6800a7a0 msgr2=0x7feb6800abb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:08:45.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:45.502+0000 7feb76200700 1 -- 172.21.15.39:0/3316070268 shutdown_connections 2024-03-18T16:08:45.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:45.502+0000 7feb76200700 1 -- 172.21.15.39:0/3316070268 wait complete. 2024-03-18T16:08:45.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:45.504+0000 7feb76200700 1 Processor -- start 2024-03-18T16:08:45.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:45.504+0000 7feb76200700 1 -- start start 2024-03-18T16:08:45.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:45.504+0000 7feb76200700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7feb68004d10 0x7feb680a9290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:45.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:45.504+0000 7feb76200700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7feb68098720 0x7feb680a97d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:45.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:45.504+0000 7feb76200700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feb6809ad20 0x7feb680a2570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:45.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:45.505+0000 7feb76200700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7feb680ac660 con 0x7feb6809ad20 2024-03-18T16:08:45.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:45.505+0000 7feb76200700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7feb680ac4e0 con 0x7feb68098720 2024-03-18T16:08:45.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:45.505+0000 7feb76200700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7feb680ac7e0 con 0x7feb68004d10 2024-03-18T16:08:45.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:45.505+0000 7feb6f400700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7feb68004d10 0x7feb680a9290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:45.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:45.505+0000 7feb6ea00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7feb68098720 0x7feb680a97d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:45.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:45.505+0000 7feb6f400700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7feb68004d10 0x7feb680a9290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:49174/0 (socket says 172.21.15.39:49174) 2024-03-18T16:08:45.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:45.505+0000 7feb6ea00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7feb68098720 0x7feb680a97d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:38330/0 (socket says 172.21.15.39:38330) 2024-03-18T16:08:45.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:45.505+0000 7feb6f400700 1 -- 172.21.15.39:0/619012794 learned_addr learned my addr 172.21.15.39:0/619012794 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:08:45.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:45.506+0000 7feb6ea00700 1 -- 172.21.15.39:0/619012794 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7feb68004d10 msgr2=0x7feb680a9290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:45.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:45.506+0000 7feb6ea00700 1 --2- 172.21.15.39:0/619012794 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7feb68004d10 0x7feb680a9290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:45.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:45.506+0000 7feb6ea00700 1 -- 172.21.15.39:0/619012794 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feb6809ad20 msgr2=0x7feb680a2570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:45.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:45.506+0000 7feb6ea00700 1 --2- 172.21.15.39:0/619012794 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feb6809ad20 0x7feb680a2570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:45.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:45.506+0000 7feb6ea00700 1 -- 172.21.15.39:0/619012794 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb680a2e40 con 0x7feb68098720 2024-03-18T16:08:45.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:45.506+0000 7feb6ea00700 1 --2- 172.21.15.39:0/619012794 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7feb68098720 0x7feb680a97d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7feb64000c00 tx=0x7feb640049d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:45.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:45.506+0000 7feb63e00700 1 -- 172.21.15.39:0/619012794 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb64025070 con 0x7feb68098720 2024-03-18T16:08:45.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:45.507+0000 7feb76200700 1 -- 172.21.15.39:0/619012794 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb680a30d0 con 0x7feb68098720 2024-03-18T16:08:45.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:45.507+0000 7feb63e00700 1 -- 172.21.15.39:0/619012794 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb6400faf0 con 0x7feb68098720 2024-03-18T16:08:45.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:45.507+0000 7feb63e00700 1 -- 172.21.15.39:0/619012794 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb6400be70 con 0x7feb68098720 2024-03-18T16:08:45.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:45.507+0000 7feb76200700 1 -- 172.21.15.39:0/619012794 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb680a7040 con 0x7feb68098720 2024-03-18T16:08:45.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:45.509+0000 7feb76200700 1 -- 172.21.15.39:0/619012794 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb68095260 con 0x7feb68098720 2024-03-18T16:08:45.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:45.512+0000 7feb63e00700 1 -- 172.21.15.39:0/619012794 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 17) ==== 137341+0+0 (secure 0 0 0) 0x7feb64028070 con 0x7feb68098720 2024-03-18T16:08:45.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:45.515+0000 7feb63e00700 1 --2- 172.21.15.39:0/619012794 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7feb580a29e0 0x7feb580a4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:45.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:45.515+0000 7feb63e00700 1 -- 172.21.15.39:0/619012794 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7feb640cf6e0 con 0x7feb68098720 2024-03-18T16:08:45.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:45.515+0000 7feb6f400700 1 --2- 172.21.15.39:0/619012794 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7feb580a29e0 0x7feb580a4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:45.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:45.518+0000 7feb6f400700 1 --2- 172.21.15.39:0/619012794 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7feb580a29e0 0x7feb580a4ea0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7feb70051930 tx=0x7feb70051050 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:45.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:45.521+0000 7feb63e00700 1 -- 172.21.15.39:0/619012794 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7feb6405c740 con 0x7feb68098720 2024-03-18T16:08:45.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:45.826+0000 7feb76200700 1 -- 172.21.15.39:0/619012794 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7feb680a72f0 con 0x7feb68098720 2024-03-18T16:08:45.830 INFO:teuthology.orchestra.run.smithi039.stdout:163208757321 2024-03-18T16:08:45.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:45.826+0000 7feb63e00700 1 -- 172.21.15.39:0/619012794 <== mon.2 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7feb64013030 con 0x7feb68098720 2024-03-18T16:08:45.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:45.830+0000 7feb76200700 1 -- 172.21.15.39:0/619012794 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7feb580a29e0 msgr2=0x7feb580a4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:45.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:45.830+0000 7feb76200700 1 --2- 172.21.15.39:0/619012794 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7feb580a29e0 0x7feb580a4ea0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7feb70051930 tx=0x7feb70051050 comp rx=0 tx=0).stop 2024-03-18T16:08:45.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:45.830+0000 7feb76200700 1 -- 172.21.15.39:0/619012794 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7feb68098720 msgr2=0x7feb680a97d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:45.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:45.830+0000 7feb76200700 1 --2- 172.21.15.39:0/619012794 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7feb68098720 0x7feb680a97d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7feb64000c00 tx=0x7feb640049d0 comp rx=0 tx=0).stop 2024-03-18T16:08:45.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:45.831+0000 7feb76200700 1 -- 172.21.15.39:0/619012794 shutdown_connections 2024-03-18T16:08:45.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:45.831+0000 7feb76200700 1 --2- 172.21.15.39:0/619012794 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7feb580a29e0 0x7feb580a4ea0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:45.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:45.831+0000 7feb76200700 1 --2- 172.21.15.39:0/619012794 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7feb6809ad20 0x7feb680a2570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:45.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:45.831+0000 7feb76200700 1 --2- 172.21.15.39:0/619012794 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7feb68004d10 0x7feb680a9290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:45.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:45.831+0000 7feb76200700 1 --2- 172.21.15.39:0/619012794 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7feb68098720 0x7feb680a97d0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:45.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:45.831+0000 7feb76200700 1 -- 172.21.15.39:0/619012794 >> 172.21.15.39:0/619012794 conn(0x7feb6800a7a0 msgr2=0x7feb68095b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:08:45.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:45.831+0000 7feb76200700 1 -- 172.21.15.39:0/619012794 shutdown_connections 2024-03-18T16:08:45.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:45.831+0000 7feb76200700 1 -- 172.21.15.39:0/619012794 wait complete. 2024-03-18T16:08:46.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:08:45 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/619012794' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-18T16:08:46.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:08:45 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/619012794' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-18T16:08:46.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:08:45 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/619012794' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-18T16:08:46.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:46.842+0000 7f9eca200700 1 -- 172.21.15.39:0/891584000 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9ec4102f30 msgr2=0x7f9ec4106f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:46.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:46.842+0000 7f9eca200700 1 --2- 172.21.15.39:0/891584000 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9ec4102f30 0x7f9ec4106f70 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f9eb0009990 tx=0x7f9eb0009ca0 comp rx=0 tx=0).stop 2024-03-18T16:08:46.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:46.842+0000 7f9eca200700 1 -- 172.21.15.39:0/891584000 shutdown_connections 2024-03-18T16:08:46.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:46.842+0000 7f9eca200700 1 --2- 172.21.15.39:0/891584000 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9ec4107650 0x7f9ec410f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:46.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:46.842+0000 7f9eca200700 1 --2- 172.21.15.39:0/891584000 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9ec4102580 0x7f9ec4102960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:46.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:46.842+0000 7f9eca200700 1 --2- 172.21.15.39:0/891584000 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9ec4102f30 0x7f9ec4106f70 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:46.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:46.842+0000 7f9eca200700 1 -- 172.21.15.39:0/891584000 >> 172.21.15.39:0/891584000 conn(0x7f9ec4079430 msgr2=0x7f9ec4079840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:08:46.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:46.842+0000 7f9eca200700 1 -- 172.21.15.39:0/891584000 shutdown_connections 2024-03-18T16:08:46.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:46.842+0000 7f9eca200700 1 -- 172.21.15.39:0/891584000 wait complete. 2024-03-18T16:08:46.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:46.843+0000 7f9eca200700 1 Processor -- start 2024-03-18T16:08:46.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:46.843+0000 7f9eca200700 1 -- start start 2024-03-18T16:08:46.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:46.844+0000 7f9eca200700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9ec4102580 0x7f9ec419e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:46.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:46.844+0000 7f9eca200700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9ec4102f30 0x7f9ec419ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:46.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:46.844+0000 7f9eca200700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9ec4107650 0x7f9ec4198980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:46.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:46.844+0000 7f9eca200700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f9ec4112660 con 0x7f9ec4102580 2024-03-18T16:08:46.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:46.844+0000 7f9eca200700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f9ec41124e0 con 0x7f9ec4107650 2024-03-18T16:08:46.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:46.844+0000 7f9eca200700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f9ec41127e0 con 0x7f9ec4102f30 2024-03-18T16:08:46.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:46.844+0000 7f9ec2a00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9ec4102f30 0x7f9ec419ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:46.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:46.844+0000 7f9ec3400700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9ec4102580 0x7f9ec419e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:46.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:46.844+0000 7f9ec2a00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9ec4102f30 0x7f9ec419ecb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:49188/0 (socket says 172.21.15.39:49188) 2024-03-18T16:08:46.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:46.844+0000 7f9ec2a00700 1 -- 172.21.15.39:0/632861833 learned_addr learned my addr 172.21.15.39:0/632861833 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:08:46.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:46.844+0000 7f9ec3e00700 1 --2- 172.21.15.39:0/632861833 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9ec4107650 0x7f9ec4198980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:46.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:46.845+0000 7f9ec2a00700 1 -- 172.21.15.39:0/632861833 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9ec4107650 msgr2=0x7f9ec4198980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:46.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:46.845+0000 7f9ec2a00700 1 --2- 172.21.15.39:0/632861833 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9ec4107650 0x7f9ec4198980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:46.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:46.845+0000 7f9ec2a00700 1 -- 172.21.15.39:0/632861833 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9ec4102580 msgr2=0x7f9ec419e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:46.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:46.845+0000 7f9ec2a00700 1 --2- 172.21.15.39:0/632861833 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9ec4102580 0x7f9ec419e770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:46.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:46.845+0000 7f9ec2a00700 1 -- 172.21.15.39:0/632861833 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ec41990e0 con 0x7f9ec4102f30 2024-03-18T16:08:46.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:46.845+0000 7f9ec3e00700 1 --2- 172.21.15.39:0/632861833 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9ec4107650 0x7f9ec4198980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:08:46.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:46.845+0000 7f9ec2a00700 1 --2- 172.21.15.39:0/632861833 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9ec4102f30 0x7f9ec419ecb0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9eb0000c00 tx=0x7f9eb0005590 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:46.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:46.845+0000 7f9ebbe00700 1 -- 172.21.15.39:0/632861833 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9eb0027070 con 0x7f9ec4102f30 2024-03-18T16:08:46.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:46.845+0000 7f9ebbe00700 1 -- 172.21.15.39:0/632861833 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9eb00056b0 con 0x7f9ec4102f30 2024-03-18T16:08:46.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:46.845+0000 7f9eca200700 1 -- 172.21.15.39:0/632861833 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ec4199370 con 0x7f9ec4102f30 2024-03-18T16:08:46.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:46.845+0000 7f9ebbe00700 1 -- 172.21.15.39:0/632861833 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9eb00173f0 con 0x7f9ec4102f30 2024-03-18T16:08:46.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:46.846+0000 7f9eca200700 1 -- 172.21.15.39:0/632861833 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ec4074500 con 0x7f9ec4102f30 2024-03-18T16:08:46.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:46.847+0000 7f9eca200700 1 -- 172.21.15.39:0/632861833 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ec4103a50 con 0x7f9ec4102f30 2024-03-18T16:08:46.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:46.849+0000 7f9ebbe00700 1 -- 172.21.15.39:0/632861833 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f9eb002a070 con 0x7f9ec4102f30 2024-03-18T16:08:46.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:46.850+0000 7f9ebbe00700 1 --2- 172.21.15.39:0/632861833 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f9ea80a2890 0x7f9ea80a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:46.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:46.850+0000 7f9ebbe00700 1 -- 172.21.15.39:0/632861833 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9092+0+0 (secure 0 0 0) 0x7f9eb00cfb70 con 0x7f9ec4102f30 2024-03-18T16:08:46.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:46.850+0000 7f9ec3400700 1 --2- 172.21.15.39:0/632861833 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f9ea80a2890 0x7f9ea80a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:46.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:46.851+0000 7f9ec3400700 1 --2- 172.21.15.39:0/632861833 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f9ea80a2890 0x7f9ea80a4d50 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f9ec4107390 tx=0x7f9eb4012040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:46.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:46.859+0000 7f9ebbe00700 1 -- 172.21.15.39:0/632861833 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f9eb0013070 con 0x7f9ec4102f30 2024-03-18T16:08:47.016 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:08:46 smithi039 ceph-mon[31721]: pgmap v381: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T16:08:47.016 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:08:46 smithi039 ceph-mon[31721]: mgrmap e18: x(active, since 10m), standbys: z, y 2024-03-18T16:08:47.016 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:08:46 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]: dispatch 2024-03-18T16:08:47.016 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:08:46 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]: dispatch 2024-03-18T16:08:47.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:47.132+0000 7f9eca200700 1 -- 172.21.15.39:0/632861833 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7f9ec4068f90 con 0x7f9ec4102f30 2024-03-18T16:08:47.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:47.133+0000 7f9ebbe00700 1 -- 172.21.15.39:0/632861833 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f9eb009c8e0 con 0x7f9ec4102f30 2024-03-18T16:08:47.136 INFO:teuthology.orchestra.run.smithi039.stdout:343597383719 2024-03-18T16:08:47.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:47.136+0000 7f9eca200700 1 -- 172.21.15.39:0/632861833 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f9ea80a2890 msgr2=0x7f9ea80a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:47.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:47.136+0000 7f9eca200700 1 --2- 172.21.15.39:0/632861833 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f9ea80a2890 0x7f9ea80a4d50 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f9ec4107390 tx=0x7f9eb4012040 comp rx=0 tx=0).stop 2024-03-18T16:08:47.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:47.136+0000 7f9eca200700 1 -- 172.21.15.39:0/632861833 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9ec4102f30 msgr2=0x7f9ec419ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:47.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:47.136+0000 7f9eca200700 1 --2- 172.21.15.39:0/632861833 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9ec4102f30 0x7f9ec419ecb0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9eb0000c00 tx=0x7f9eb0005590 comp rx=0 tx=0).stop 2024-03-18T16:08:47.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:47.136+0000 7f9eca200700 1 -- 172.21.15.39:0/632861833 shutdown_connections 2024-03-18T16:08:47.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:47.136+0000 7f9eca200700 1 --2- 172.21.15.39:0/632861833 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f9ea80a2890 0x7f9ea80a4d50 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:47.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:47.136+0000 7f9eca200700 1 --2- 172.21.15.39:0/632861833 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9ec4107650 0x7f9ec4198980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:47.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:47.136+0000 7f9eca200700 1 --2- 172.21.15.39:0/632861833 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9ec4102580 0x7f9ec419e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:47.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:47.136+0000 7f9eca200700 1 --2- 172.21.15.39:0/632861833 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9ec4102f30 0x7f9ec419ecb0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:47.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:47.136+0000 7f9eca200700 1 -- 172.21.15.39:0/632861833 >> 172.21.15.39:0/632861833 conn(0x7f9ec4079430 msgr2=0x7f9ec4100370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:08:47.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:47.136+0000 7f9eca200700 1 -- 172.21.15.39:0/632861833 shutdown_connections 2024-03-18T16:08:47.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:47.136+0000 7f9eca200700 1 -- 172.21.15.39:0/632861833 wait complete. 2024-03-18T16:08:47.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:08:46 smithi196 ceph-mon[36307]: pgmap v381: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T16:08:47.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:08:46 smithi196 ceph-mon[36307]: mgrmap e18: x(active, since 10m), standbys: z, y 2024-03-18T16:08:47.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:08:46 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]: dispatch 2024-03-18T16:08:47.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:08:46 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]: dispatch 2024-03-18T16:08:47.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:08:46 smithi194 ceph-mon[36200]: pgmap v381: 129 pgs: 129 active+clean; 643 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T16:08:47.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:08:46 smithi194 ceph-mon[36200]: mgrmap e18: x(active, since 10m), standbys: z, y 2024-03-18T16:08:47.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:08:46 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]: dispatch 2024-03-18T16:08:47.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:08:46 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]: dispatch 2024-03-18T16:08:47.459 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757313 got 163208757321 for osd.5 2024-03-18T16:08:47.459 DEBUG:teuthology.parallel:result is None 2024-03-18T16:08:48.209 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:08:47 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]': finished 2024-03-18T16:08:48.209 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:08:47 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]': finished 2024-03-18T16:08:48.209 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:08:47 smithi039 ceph-mon[31721]: osdmap e92: 12 total, 12 up, 12 in 2024-03-18T16:08:48.210 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:08:47 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:08:48.210 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:08:47 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:08:48.210 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:08:47 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/632861833' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-18T16:08:48.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:08:47 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]': finished 2024-03-18T16:08:48.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:08:47 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]': finished 2024-03-18T16:08:48.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:08:47 smithi196 ceph-mon[36307]: osdmap e92: 12 total, 12 up, 12 in 2024-03-18T16:08:48.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:08:47 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:08:48.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:08:47 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:08:48.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:08:47 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/632861833' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-18T16:08:48.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:08:47 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.f", "id": [8, 11]}]': finished 2024-03-18T16:08:48.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:08:47 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]': finished 2024-03-18T16:08:48.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:08:47 smithi194 ceph-mon[36200]: osdmap e92: 12 total, 12 up, 12 in 2024-03-18T16:08:48.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:08:47 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:08:48.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:08:47 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:08:48.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:08:47 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/632861833' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-18T16:08:48.251 INFO:tasks.cephadm.ceph_manager.ceph:need seq 343597383712 got 343597383719 for osd.11 2024-03-18T16:08:48.251 DEBUG:teuthology.parallel:result is None 2024-03-18T16:08:48.251 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-18T16:08:48.251 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-18T16:08:48.688 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T16:08:49.096 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:08:48 smithi039 ceph-mon[31721]: pgmap v383: 129 pgs: 129 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-18T16:08:49.097 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:08:48 smithi039 ceph-mon[31721]: osdmap e93: 12 total, 12 up, 12 in 2024-03-18T16:08:49.097 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:08:48 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:08:49.097 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:08:48 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:08:49.097 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:08:48 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:08:49.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:08:48 smithi196 ceph-mon[36307]: pgmap v383: 129 pgs: 129 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-18T16:08:49.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:08:48 smithi196 ceph-mon[36307]: osdmap e93: 12 total, 12 up, 12 in 2024-03-18T16:08:49.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:08:48 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:08:49.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:08:48 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:08:49.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:08:48 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:08:49.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:08:48 smithi194 ceph-mon[36200]: pgmap v383: 129 pgs: 129 active+clean; 644 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 102 B/s wr, 10 op/s 2024-03-18T16:08:49.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:08:48 smithi194 ceph-mon[36200]: osdmap e93: 12 total, 12 up, 12 in 2024-03-18T16:08:49.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:08:48 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:08:49.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:08:48 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:08:49.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:08:48 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:08:51.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:51.073+0000 7f6576c00700 1 -- 172.21.15.39:0/2274435148 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6570103d00 msgr2=0x7f657010c1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:51.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:51.073+0000 7f6576c00700 1 --2- 172.21.15.39:0/2274435148 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6570103d00 0x7f657010c1f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f655c009990 tx=0x7f655c009ca0 comp rx=0 tx=0).stop 2024-03-18T16:08:51.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:51.073+0000 7f6576c00700 1 -- 172.21.15.39:0/2274435148 shutdown_connections 2024-03-18T16:08:51.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:51.073+0000 7f6576c00700 1 --2- 172.21.15.39:0/2274435148 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f657010c7c0 0x7f657010ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:51.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:51.073+0000 7f6576c00700 1 --2- 172.21.15.39:0/2274435148 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f65701033e0 0x7f65701037c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:51.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:51.073+0000 7f6576c00700 1 --2- 172.21.15.39:0/2274435148 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6570103d00 0x7f657010c1f0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:51.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:51.073+0000 7f6576c00700 1 -- 172.21.15.39:0/2274435148 >> 172.21.15.39:0/2274435148 conn(0x7f65700fcbb0 msgr2=0x7f65700fefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:08:51.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:51.073+0000 7f6576c00700 1 -- 172.21.15.39:0/2274435148 shutdown_connections 2024-03-18T16:08:51.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:51.073+0000 7f6576c00700 1 -- 172.21.15.39:0/2274435148 wait complete. 2024-03-18T16:08:51.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:51.074+0000 7f6576c00700 1 Processor -- start 2024-03-18T16:08:51.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:51.074+0000 7f6576c00700 1 -- start start 2024-03-18T16:08:51.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:51.075+0000 7f6576c00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f65701033e0 0x7f657019e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:51.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:51.075+0000 7f6576c00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6570103d00 0x7f657019ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:51.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:51.075+0000 7f6576c00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f657010c7c0 0x7f65701988c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:51.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:51.075+0000 7f6576c00700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f6570112520 con 0x7f657010c7c0 2024-03-18T16:08:51.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:51.075+0000 7f6576c00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f65701123a0 con 0x7f6570103d00 2024-03-18T16:08:51.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:51.075+0000 7f6576c00700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f65701126a0 con 0x7f65701033e0 2024-03-18T16:08:51.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:51.075+0000 7f6575800700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f65701033e0 0x7f657019e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:51.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:51.075+0000 7f6575800700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f65701033e0 0x7f657019e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:54176/0 (socket says 172.21.15.39:54176) 2024-03-18T16:08:51.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:51.075+0000 7f6574e00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6570103d00 0x7f657019ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:51.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:51.075+0000 7f6575800700 1 -- 172.21.15.39:0/1131747149 learned_addr learned my addr 172.21.15.39:0/1131747149 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:08:51.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:51.075+0000 7f6576200700 1 --2- 172.21.15.39:0/1131747149 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f657010c7c0 0x7f65701988c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:51.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:51.076+0000 7f6574e00700 1 -- 172.21.15.39:0/1131747149 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f65701033e0 msgr2=0x7f657019e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:51.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:51.076+0000 7f6574e00700 1 --2- 172.21.15.39:0/1131747149 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f65701033e0 0x7f657019e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:51.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:51.076+0000 7f6574e00700 1 -- 172.21.15.39:0/1131747149 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f657010c7c0 msgr2=0x7f65701988c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:51.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:51.076+0000 7f6574e00700 1 --2- 172.21.15.39:0/1131747149 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f657010c7c0 0x7f65701988c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:51.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:51.076+0000 7f6574e00700 1 -- 172.21.15.39:0/1131747149 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6570199020 con 0x7f6570103d00 2024-03-18T16:08:51.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:51.076+0000 7f6575800700 1 --2- 172.21.15.39:0/1131747149 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f65701033e0 0x7f657019e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:08:51.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:51.076+0000 7f6574e00700 1 --2- 172.21.15.39:0/1131747149 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6570103d00 0x7f657019ebf0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f655c000c00 tx=0x7f655c00bab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:51.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:51.076+0000 7f656e000700 1 -- 172.21.15.39:0/1131747149 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f655c026070 con 0x7f6570103d00 2024-03-18T16:08:51.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:51.076+0000 7f656e000700 1 -- 172.21.15.39:0/1131747149 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f655c003d70 con 0x7f6570103d00 2024-03-18T16:08:51.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:51.076+0000 7f6576c00700 1 -- 172.21.15.39:0/1131747149 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65701992b0 con 0x7f6570103d00 2024-03-18T16:08:51.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:51.076+0000 7f656e000700 1 -- 172.21.15.39:0/1131747149 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f655c021410 con 0x7f6570103d00 2024-03-18T16:08:51.081 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:51.076+0000 7f6576c00700 1 -- 172.21.15.39:0/1131747149 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65701a5580 con 0x7f6570103d00 2024-03-18T16:08:51.081 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:51.078+0000 7f6576c00700 1 -- 172.21.15.39:0/1131747149 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6570068f90 con 0x7f6570103d00 2024-03-18T16:08:51.081 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:51.079+0000 7f656e000700 1 -- 172.21.15.39:0/1131747149 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f655c029070 con 0x7f6570103d00 2024-03-18T16:08:51.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:51.081+0000 7f656e000700 1 --2- 172.21.15.39:0/1131747149 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f65580a2910 0x7f65580a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:51.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:51.081+0000 7f656e000700 1 -- 172.21.15.39:0/1131747149 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f655c0cfb00 con 0x7f6570103d00 2024-03-18T16:08:51.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:51.081+0000 7f6575800700 1 --2- 172.21.15.39:0/1131747149 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f65580a2910 0x7f65580a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:51.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:51.082+0000 7f6575800700 1 --2- 172.21.15.39:0/1131747149 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f65580a2910 0x7f65580a4dd0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f657010f0a0 tx=0x7f6564009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:51.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:51.085+0000 7f656e000700 1 -- 172.21.15.39:0/1131747149 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f655c05d740 con 0x7f6570103d00 2024-03-18T16:08:51.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:08:50 smithi196 ceph-mon[36307]: pgmap v385: 129 pgs: 129 active+clean; 644 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 639 B/s wr, 12 op/s 2024-03-18T16:08:51.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:08:50 smithi194 ceph-mon[36200]: pgmap v385: 129 pgs: 129 active+clean; 644 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 639 B/s wr, 12 op/s 2024-03-18T16:08:51.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:51.305+0000 7f6576c00700 1 -- 172.21.15.39:0/1131747149 --> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f6570062710 con 0x7f65580a2910 2024-03-18T16:08:51.308 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:08:50 smithi039 ceph-mon[31721]: pgmap v385: 129 pgs: 129 active+clean; 644 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 639 B/s wr, 12 op/s 2024-03-18T16:08:51.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:51.317+0000 7f656e000700 1 -- 172.21.15.39:0/1131747149 <== mgr.14152 v2:172.21.15.39:6800/2749214286 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340659 (secure 0 0 0) 0x7f6570062710 con 0x7f65580a2910 2024-03-18T16:08:51.327 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:08:51.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:51.324+0000 7f6576c00700 1 -- 172.21.15.39:0/1131747149 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f65580a2910 msgr2=0x7f65580a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:51.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:51.324+0000 7f6576c00700 1 --2- 172.21.15.39:0/1131747149 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f65580a2910 0x7f65580a4dd0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f657010f0a0 tx=0x7f6564009260 comp rx=0 tx=0).stop 2024-03-18T16:08:51.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:51.324+0000 7f6576c00700 1 -- 172.21.15.39:0/1131747149 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6570103d00 msgr2=0x7f657019ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:51.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:51.324+0000 7f6576c00700 1 --2- 172.21.15.39:0/1131747149 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6570103d00 0x7f657019ebf0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f655c000c00 tx=0x7f655c00bab0 comp rx=0 tx=0).stop 2024-03-18T16:08:51.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:51.324+0000 7f6576c00700 1 -- 172.21.15.39:0/1131747149 shutdown_connections 2024-03-18T16:08:51.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:51.324+0000 7f6576c00700 1 --2- 172.21.15.39:0/1131747149 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f65580a2910 0x7f65580a4dd0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:51.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:51.324+0000 7f6576c00700 1 --2- 172.21.15.39:0/1131747149 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f657010c7c0 0x7f65701988c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:51.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:51.324+0000 7f6576c00700 1 --2- 172.21.15.39:0/1131747149 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f65701033e0 0x7f657019e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:51.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:51.324+0000 7f6576c00700 1 --2- 172.21.15.39:0/1131747149 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6570103d00 0x7f657019ebf0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:51.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:51.324+0000 7f6576c00700 1 -- 172.21.15.39:0/1131747149 >> 172.21.15.39:0/1131747149 conn(0x7f65700fcbb0 msgr2=0x7f65700fd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:08:51.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:51.325+0000 7f6576c00700 1 -- 172.21.15.39:0/1131747149 shutdown_connections 2024-03-18T16:08:51.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:51.325+0000 7f6576c00700 1 -- 172.21.15.39:0/1131747149 wait complete. 2024-03-18T16:08:51.331 INFO:teuthology.orchestra.run.smithi039.stderr:dumped all 2024-03-18T16:08:52.165 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:08:51 smithi039 ceph-mon[31721]: from='client.25040 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T16:08:52.225 INFO:teuthology.orchestra.run.smithi039.stdout:{"pg_ready":true,"pg_map":{"version":385,"stamp":"2024-03-18T16:08:49.540609+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":659297,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1051,"num_read_kb":649,"num_write":432,"num_write_kb":3083,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1180853,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":793,"ondisk_log_size":793,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":418,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":396788,"kb_used_data":14316,"kb_used_omap":18,"kb_used_meta":382317,"kb_avail":1124545036,"statfs":{"total":1151940427776,"available":1151534116864,"internally_reserved":0,"allocated":14659584,"data_stored":8019824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":391492923},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":390,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":97,"num_read_kb":47,"num_write":7,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003274"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T16:08:45.519289+0000","last_change":"2024-03-18T16:06:44.736511+0000","last_active":"2024-03-18T16:08:45.519289+0000","last_peered":"2024-03-18T16:08:45.519289+0000","last_clean":"2024-03-18T16:08:45.519289+0000","last_became_active":"2024-03-18T16:06:44.736182+0000","last_became_peered":"2024-03-18T16:06:44.736182+0000","last_unstale":"2024-03-18T16:08:45.519289+0000","last_undegraded":"2024-03-18T16:08:45.519289+0000","last_fullsized":"2024-03-18T16:08:45.519289+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:08:57.060268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.961742+0000","last_change":"2024-03-18T16:06:44.733658+0000","last_active":"2024-03-18T16:08:47.961742+0000","last_peered":"2024-03-18T16:08:47.961742+0000","last_clean":"2024-03-18T16:08:47.961742+0000","last_became_active":"2024-03-18T16:06:44.733279+0000","last_became_peered":"2024-03-18T16:06:44.733279+0000","last_unstale":"2024-03-18T16:08:47.961742+0000","last_undegraded":"2024-03-18T16:08:47.961742+0000","last_fullsized":"2024-03-18T16:08:47.961742+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:59:46.783025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.961398+0000","last_change":"2024-03-18T16:06:44.723949+0000","last_active":"2024-03-18T16:08:47.961398+0000","last_peered":"2024-03-18T16:08:47.961398+0000","last_clean":"2024-03-18T16:08:47.961398+0000","last_became_active":"2024-03-18T16:06:44.723614+0000","last_became_peered":"2024-03-18T16:06:44.723614+0000","last_unstale":"2024-03-18T16:08:47.961398+0000","last_undegraded":"2024-03-18T16:08:47.961398+0000","last_fullsized":"2024-03-18T16:08:47.961398+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:03:17.876097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883864+0000","last_change":"2024-03-18T16:06:44.711058+0000","last_active":"2024-03-18T16:08:47.883864+0000","last_peered":"2024-03-18T16:08:47.883864+0000","last_clean":"2024-03-18T16:08:47.883864+0000","last_became_active":"2024-03-18T16:06:44.708873+0000","last_became_peered":"2024-03-18T16:06:44.708873+0000","last_unstale":"2024-03-18T16:08:47.883864+0000","last_undegraded":"2024-03-18T16:08:47.883864+0000","last_fullsized":"2024-03-18T16:08:47.883864+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:58:08.182651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":30,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-18T16:08:46.877060+0000","last_change":"2024-03-18T16:06:44.721480+0000","last_active":"2024-03-18T16:08:46.877060+0000","last_peered":"2024-03-18T16:08:46.877060+0000","last_clean":"2024-03-18T16:08:46.877060+0000","last_became_active":"2024-03-18T16:06:44.721101+0000","last_became_peered":"2024-03-18T16:06:44.721101+0000","last_unstale":"2024-03-18T16:08:46.877060+0000","last_undegraded":"2024-03-18T16:08:46.877060+0000","last_fullsized":"2024-03-18T16:08:46.877060+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:19:43.341221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.879808+0000","last_change":"2024-03-18T16:06:44.716443+0000","last_active":"2024-03-18T16:08:47.879808+0000","last_peered":"2024-03-18T16:08:47.879808+0000","last_clean":"2024-03-18T16:08:47.879808+0000","last_became_active":"2024-03-18T16:06:44.715969+0000","last_became_peered":"2024-03-18T16:06:44.715969+0000","last_unstale":"2024-03-18T16:08:47.879808+0000","last_undegraded":"2024-03-18T16:08:47.879808+0000","last_fullsized":"2024-03-18T16:08:47.879808+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:57:14.714415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.177820+0000","last_change":"2024-03-18T16:06:44.711439+0000","last_active":"2024-03-18T16:08:47.177820+0000","last_peered":"2024-03-18T16:08:47.177820+0000","last_clean":"2024-03-18T16:08:47.177820+0000","last_became_active":"2024-03-18T16:06:44.711032+0000","last_became_peered":"2024-03-18T16:06:44.711032+0000","last_unstale":"2024-03-18T16:08:47.177820+0000","last_undegraded":"2024-03-18T16:08:47.177820+0000","last_fullsized":"2024-03-18T16:08:47.177820+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:19:56.516867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":20,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T16:08:45.506919+0000","last_change":"2024-03-18T16:07:47.358866+0000","last_active":"2024-03-18T16:08:45.506919+0000","last_peered":"2024-03-18T16:08:45.506919+0000","last_clean":"2024-03-18T16:08:45.506919+0000","last_became_active":"2024-03-18T16:07:47.357743+0000","last_became_peered":"2024-03-18T16:07:47.357743+0000","last_unstale":"2024-03-18T16:08:45.506919+0000","last_undegraded":"2024-03-18T16:08:45.506919+0000","last_fullsized":"2024-03-18T16:08:45.506919+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:13:12.361025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,10],"acting":[9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T16:08:45.500567+0000","last_change":"2024-03-18T16:06:44.710978+0000","last_active":"2024-03-18T16:08:45.500567+0000","last_peered":"2024-03-18T16:08:45.500567+0000","last_clean":"2024-03-18T16:08:45.500567+0000","last_became_active":"2024-03-18T16:06:44.710520+0000","last_became_peered":"2024-03-18T16:06:44.710520+0000","last_unstale":"2024-03-18T16:08:45.500567+0000","last_undegraded":"2024-03-18T16:08:45.500567+0000","last_fullsized":"2024-03-18T16:08:45.500567+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:54:53.234025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.880362+0000","last_change":"2024-03-18T16:06:44.716231+0000","last_active":"2024-03-18T16:08:47.880362+0000","last_peered":"2024-03-18T16:08:47.880362+0000","last_clean":"2024-03-18T16:08:47.880362+0000","last_became_active":"2024-03-18T16:06:44.714900+0000","last_became_peered":"2024-03-18T16:06:44.714900+0000","last_unstale":"2024-03-18T16:08:47.880362+0000","last_undegraded":"2024-03-18T16:08:47.880362+0000","last_fullsized":"2024-03-18T16:08:47.880362+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:38:54.964729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.178264+0000","last_change":"2024-03-18T16:06:44.710942+0000","last_active":"2024-03-18T16:08:47.178264+0000","last_peered":"2024-03-18T16:08:47.178264+0000","last_clean":"2024-03-18T16:08:47.178264+0000","last_became_active":"2024-03-18T16:06:44.710588+0000","last_became_peered":"2024-03-18T16:06:44.710588+0000","last_unstale":"2024-03-18T16:08:47.178264+0000","last_undegraded":"2024-03-18T16:08:47.178264+0000","last_fullsized":"2024-03-18T16:08:47.178264+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:43:19.320034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883785+0000","last_change":"2024-03-18T16:06:44.714763+0000","last_active":"2024-03-18T16:08:47.883785+0000","last_peered":"2024-03-18T16:08:47.883785+0000","last_clean":"2024-03-18T16:08:47.883785+0000","last_became_active":"2024-03-18T16:06:44.714031+0000","last_became_peered":"2024-03-18T16:06:44.714031+0000","last_unstale":"2024-03-18T16:08:47.883785+0000","last_undegraded":"2024-03-18T16:08:47.883785+0000","last_fullsized":"2024-03-18T16:08:47.883785+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:19:25.917093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T16:08:45.537253+0000","last_change":"2024-03-18T16:06:44.710062+0000","last_active":"2024-03-18T16:08:45.537253+0000","last_peered":"2024-03-18T16:08:45.537253+0000","last_clean":"2024-03-18T16:08:45.537253+0000","last_became_active":"2024-03-18T16:06:44.709281+0000","last_became_peered":"2024-03-18T16:06:44.709281+0000","last_unstale":"2024-03-18T16:08:45.537253+0000","last_undegraded":"2024-03-18T16:08:45.537253+0000","last_fullsized":"2024-03-18T16:08:45.537253+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:51:30.100947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.960744+0000","last_change":"2024-03-18T16:06:44.734178+0000","last_active":"2024-03-18T16:08:47.960744+0000","last_peered":"2024-03-18T16:08:47.960744+0000","last_clean":"2024-03-18T16:08:47.960744+0000","last_became_active":"2024-03-18T16:06:44.732538+0000","last_became_peered":"2024-03-18T16:06:44.732538+0000","last_unstale":"2024-03-18T16:08:47.960744+0000","last_undegraded":"2024-03-18T16:08:47.960744+0000","last_fullsized":"2024-03-18T16:08:47.960744+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:14:26.904402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.960737+0000","last_change":"2024-03-18T16:07:47.412629+0000","last_active":"2024-03-18T16:08:47.960737+0000","last_peered":"2024-03-18T16:08:47.960737+0000","last_clean":"2024-03-18T16:08:47.960737+0000","last_became_active":"2024-03-18T16:07:47.411714+0000","last_became_peered":"2024-03-18T16:07:47.411714+0000","last_unstale":"2024-03-18T16:08:47.960737+0000","last_undegraded":"2024-03-18T16:08:47.960737+0000","last_fullsized":"2024-03-18T16:08:47.960737+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:25:50.843323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T16:07:47.353621+0000","last_change":"2024-03-18T16:06:44.728165+0000","last_active":"2024-03-18T16:07:47.353621+0000","last_peered":"2024-03-18T16:07:47.353621+0000","last_clean":"2024-03-18T16:07:47.353621+0000","last_became_active":"2024-03-18T16:06:44.727842+0000","last_became_peered":"2024-03-18T16:06:44.727842+0000","last_unstale":"2024-03-18T16:07:47.353621+0000","last_undegraded":"2024-03-18T16:07:47.353621+0000","last_fullsized":"2024-03-18T16:07:47.353621+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:55:04.464916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T16:08:45.519528+0000","last_change":"2024-03-18T16:06:44.730893+0000","last_active":"2024-03-18T16:08:45.519528+0000","last_peered":"2024-03-18T16:08:45.519528+0000","last_clean":"2024-03-18T16:08:45.519528+0000","last_became_active":"2024-03-18T16:06:44.730544+0000","last_became_peered":"2024-03-18T16:06:44.730544+0000","last_unstale":"2024-03-18T16:08:45.519528+0000","last_undegraded":"2024-03-18T16:08:45.519528+0000","last_fullsized":"2024-03-18T16:08:45.519528+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:16:37.918096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T16:08:45.506472+0000","last_change":"2024-03-18T16:06:44.706036+0000","last_active":"2024-03-18T16:08:45.506472+0000","last_peered":"2024-03-18T16:08:45.506472+0000","last_clean":"2024-03-18T16:08:45.506472+0000","last_became_active":"2024-03-18T16:06:44.705281+0000","last_became_peered":"2024-03-18T16:06:44.705281+0000","last_unstale":"2024-03-18T16:08:45.506472+0000","last_undegraded":"2024-03-18T16:08:45.506472+0000","last_fullsized":"2024-03-18T16:08:45.506472+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:52:20.202110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.884459+0000","last_change":"2024-03-18T16:06:44.727543+0000","last_active":"2024-03-18T16:08:47.884459+0000","last_peered":"2024-03-18T16:08:47.884459+0000","last_clean":"2024-03-18T16:08:47.884459+0000","last_became_active":"2024-03-18T16:06:44.727135+0000","last_became_peered":"2024-03-18T16:06:44.727135+0000","last_unstale":"2024-03-18T16:08:47.884459+0000","last_undegraded":"2024-03-18T16:08:47.884459+0000","last_fullsized":"2024-03-18T16:08:47.884459+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:52:42.742279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883506+0000","last_change":"2024-03-18T16:06:49.028004+0000","last_active":"2024-03-18T16:08:47.883506+0000","last_peered":"2024-03-18T16:08:47.883506+0000","last_clean":"2024-03-18T16:08:47.883506+0000","last_became_active":"2024-03-18T16:06:49.027640+0000","last_became_peered":"2024-03-18T16:06:49.027640+0000","last_unstale":"2024-03-18T16:08:47.883506+0000","last_undegraded":"2024-03-18T16:08:47.883506+0000","last_fullsized":"2024-03-18T16:08:47.883506+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:23:07.926349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'13","reported_seq":127,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T16:08:45.171451+0000","last_change":"2024-03-18T16:06:43.709540+0000","last_active":"2024-03-18T16:08:45.171451+0000","last_peered":"2024-03-18T16:08:45.171451+0000","last_clean":"2024-03-18T16:08:45.171451+0000","last_became_active":"2024-03-18T16:06:43.709170+0000","last_became_peered":"2024-03-18T16:06:43.709170+0000","last_unstale":"2024-03-18T16:08:45.171451+0000","last_undegraded":"2024-03-18T16:08:45.171451+0000","last_fullsized":"2024-03-18T16:08:45.171451+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:12:53.934079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":87,"num_read_kb":87,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T16:07:47.410879+0000","last_change":"2024-03-18T16:06:44.730192+0000","last_active":"2024-03-18T16:07:47.410879+0000","last_peered":"2024-03-18T16:07:47.410879+0000","last_clean":"2024-03-18T16:07:47.410879+0000","last_became_active":"2024-03-18T16:06:44.728071+0000","last_became_peered":"2024-03-18T16:06:44.728071+0000","last_unstale":"2024-03-18T16:07:47.410879+0000","last_undegraded":"2024-03-18T16:07:47.410879+0000","last_fullsized":"2024-03-18T16:07:47.410879+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:14:59.992958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883279+0000","last_change":"2024-03-18T16:06:49.023963+0000","last_active":"2024-03-18T16:08:47.883279+0000","last_peered":"2024-03-18T16:08:47.883279+0000","last_clean":"2024-03-18T16:08:47.883279+0000","last_became_active":"2024-03-18T16:06:49.023568+0000","last_became_peered":"2024-03-18T16:06:49.023568+0000","last_unstale":"2024-03-18T16:08:47.883279+0000","last_undegraded":"2024-03-18T16:08:47.883279+0000","last_fullsized":"2024-03-18T16:08:47.883279+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:15:19.730842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T16:07:47.410923+0000","last_change":"2024-03-18T16:06:43.722167+0000","last_active":"2024-03-18T16:07:47.410923+0000","last_peered":"2024-03-18T16:07:47.410923+0000","last_clean":"2024-03-18T16:07:47.410923+0000","last_became_active":"2024-03-18T16:06:43.721782+0000","last_became_peered":"2024-03-18T16:06:43.721782+0000","last_unstale":"2024-03-18T16:07:47.410923+0000","last_undegraded":"2024-03-18T16:07:47.410923+0000","last_fullsized":"2024-03-18T16:07:47.410923+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:08:54.400637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.961989+0000","last_change":"2024-03-18T16:06:44.729666+0000","last_active":"2024-03-18T16:08:47.961989+0000","last_peered":"2024-03-18T16:08:47.961989+0000","last_clean":"2024-03-18T16:08:47.961989+0000","last_became_active":"2024-03-18T16:06:44.728842+0000","last_became_peered":"2024-03-18T16:06:44.728842+0000","last_unstale":"2024-03-18T16:08:47.961989+0000","last_undegraded":"2024-03-18T16:08:47.961989+0000","last_fullsized":"2024-03-18T16:08:47.961989+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:01:42.438000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883419+0000","last_change":"2024-03-18T16:06:49.031169+0000","last_active":"2024-03-18T16:08:47.883419+0000","last_peered":"2024-03-18T16:08:47.883419+0000","last_clean":"2024-03-18T16:08:47.883419+0000","last_became_active":"2024-03-18T16:06:49.030801+0000","last_became_peered":"2024-03-18T16:06:49.030801+0000","last_unstale":"2024-03-18T16:08:47.883419+0000","last_undegraded":"2024-03-18T16:08:47.883419+0000","last_fullsized":"2024-03-18T16:08:47.883419+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:10:54.880882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'12","reported_seq":41,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-18T16:08:46.877350+0000","last_change":"2024-03-18T16:06:43.707111+0000","last_active":"2024-03-18T16:08:46.877350+0000","last_peered":"2024-03-18T16:08:46.877350+0000","last_clean":"2024-03-18T16:08:46.877350+0000","last_became_active":"2024-03-18T16:06:43.706671+0000","last_became_peered":"2024-03-18T16:06:43.706671+0000","last_unstale":"2024-03-18T16:08:46.877350+0000","last_undegraded":"2024-03-18T16:08:46.877350+0000","last_fullsized":"2024-03-18T16:08:46.877350+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:18:34.295641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T16:07:47.353511+0000","last_change":"2024-03-18T16:06:44.722100+0000","last_active":"2024-03-18T16:07:47.353511+0000","last_peered":"2024-03-18T16:07:47.353511+0000","last_clean":"2024-03-18T16:07:47.353511+0000","last_became_active":"2024-03-18T16:06:44.721673+0000","last_became_peered":"2024-03-18T16:06:44.721673+0000","last_unstale":"2024-03-18T16:07:47.353511+0000","last_undegraded":"2024-03-18T16:07:47.353511+0000","last_fullsized":"2024-03-18T16:07:47.353511+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:41:22.624778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883330+0000","last_change":"2024-03-18T16:06:49.027966+0000","last_active":"2024-03-18T16:08:47.883330+0000","last_peered":"2024-03-18T16:08:47.883330+0000","last_clean":"2024-03-18T16:08:47.883330+0000","last_became_active":"2024-03-18T16:06:49.027567+0000","last_became_peered":"2024-03-18T16:06:49.027567+0000","last_unstale":"2024-03-18T16:08:47.883330+0000","last_undegraded":"2024-03-18T16:08:47.883330+0000","last_fullsized":"2024-03-18T16:08:47.883330+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:54:28.543019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-18T16:08:46.877290+0000","last_change":"2024-03-18T16:06:43.704023+0000","last_active":"2024-03-18T16:08:46.877290+0000","last_peered":"2024-03-18T16:08:46.877290+0000","last_clean":"2024-03-18T16:08:46.877290+0000","last_became_active":"2024-03-18T16:06:43.703690+0000","last_became_peered":"2024-03-18T16:06:43.703690+0000","last_unstale":"2024-03-18T16:08:46.877290+0000","last_undegraded":"2024-03-18T16:08:46.877290+0000","last_fullsized":"2024-03-18T16:08:46.877290+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:20:02.344346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":47,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-18T16:08:46.881170+0000","last_change":"2024-03-18T16:07:47.354361+0000","last_active":"2024-03-18T16:08:46.881170+0000","last_peered":"2024-03-18T16:08:46.881170+0000","last_clean":"2024-03-18T16:08:46.881170+0000","last_became_active":"2024-03-18T16:07:47.353964+0000","last_became_peered":"2024-03-18T16:07:47.353964+0000","last_unstale":"2024-03-18T16:08:46.881170+0000","last_undegraded":"2024-03-18T16:08:46.881170+0000","last_fullsized":"2024-03-18T16:08:46.881170+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:56:11.405218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.884296+0000","last_change":"2024-03-18T16:06:49.012310+0000","last_active":"2024-03-18T16:08:47.884296+0000","last_peered":"2024-03-18T16:08:47.884296+0000","last_clean":"2024-03-18T16:08:47.884296+0000","last_became_active":"2024-03-18T16:06:49.011844+0000","last_became_peered":"2024-03-18T16:06:49.011844+0000","last_unstale":"2024-03-18T16:08:47.884296+0000","last_undegraded":"2024-03-18T16:08:47.884296+0000","last_fullsized":"2024-03-18T16:08:47.884296+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:16:42.652205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"91'7","reported_seq":126,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-18T16:08:46.881242+0000","last_change":"2024-03-18T16:06:43.705653+0000","last_active":"2024-03-18T16:08:46.881242+0000","last_peered":"2024-03-18T16:08:46.881242+0000","last_clean":"2024-03-18T16:08:46.881242+0000","last_became_active":"2024-03-18T16:06:43.705320+0000","last_became_peered":"2024-03-18T16:06:43.705320+0000","last_unstale":"2024-03-18T16:08:46.881242+0000","last_undegraded":"2024-03-18T16:08:46.881242+0000","last_fullsized":"2024-03-18T16:08:46.881242+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:10:10.827338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":91,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-18T16:08:46.876953+0000","last_change":"2024-03-18T16:06:44.720045+0000","last_active":"2024-03-18T16:08:46.876953+0000","last_peered":"2024-03-18T16:08:46.876953+0000","last_clean":"2024-03-18T16:08:46.876953+0000","last_became_active":"2024-03-18T16:06:44.719669+0000","last_became_peered":"2024-03-18T16:06:44.719669+0000","last_unstale":"2024-03-18T16:08:46.876953+0000","last_undegraded":"2024-03-18T16:08:46.876953+0000","last_fullsized":"2024-03-18T16:08:46.876953+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:23:47.464057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.884057+0000","last_change":"2024-03-18T16:06:49.032576+0000","last_active":"2024-03-18T16:08:47.884057+0000","last_peered":"2024-03-18T16:08:47.884057+0000","last_clean":"2024-03-18T16:08:47.884057+0000","last_became_active":"2024-03-18T16:06:49.032126+0000","last_became_peered":"2024-03-18T16:06:49.032126+0000","last_unstale":"2024-03-18T16:08:47.884057+0000","last_undegraded":"2024-03-18T16:08:47.884057+0000","last_fullsized":"2024-03-18T16:08:47.884057+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:41:54.822070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.879762+0000","last_change":"2024-03-18T16:06:43.706364+0000","last_active":"2024-03-18T16:08:47.879762+0000","last_peered":"2024-03-18T16:08:47.879762+0000","last_clean":"2024-03-18T16:08:47.879762+0000","last_became_active":"2024-03-18T16:06:43.705984+0000","last_became_peered":"2024-03-18T16:06:43.705984+0000","last_unstale":"2024-03-18T16:08:47.879762+0000","last_undegraded":"2024-03-18T16:08:47.879762+0000","last_fullsized":"2024-03-18T16:08:47.879762+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:46:06.503000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-18T16:08:46.881426+0000","last_change":"2024-03-18T16:06:44.721019+0000","last_active":"2024-03-18T16:08:46.881426+0000","last_peered":"2024-03-18T16:08:46.881426+0000","last_clean":"2024-03-18T16:08:46.881426+0000","last_became_active":"2024-03-18T16:06:44.720665+0000","last_became_peered":"2024-03-18T16:06:44.720665+0000","last_unstale":"2024-03-18T16:08:46.881426+0000","last_undegraded":"2024-03-18T16:08:46.881426+0000","last_fullsized":"2024-03-18T16:08:46.881426+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:58:34.180653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T16:07:47.348790+0000","last_change":"2024-03-18T16:06:49.022919+0000","last_active":"2024-03-18T16:07:47.348790+0000","last_peered":"2024-03-18T16:07:47.348790+0000","last_clean":"2024-03-18T16:07:47.348790+0000","last_became_active":"2024-03-18T16:06:49.022514+0000","last_became_peered":"2024-03-18T16:06:49.022514+0000","last_unstale":"2024-03-18T16:07:47.348790+0000","last_undegraded":"2024-03-18T16:07:47.348790+0000","last_fullsized":"2024-03-18T16:07:47.348790+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:46:39.510181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T16:07:47.408970+0000","last_change":"2024-03-18T16:06:43.713025+0000","last_active":"2024-03-18T16:07:47.408970+0000","last_peered":"2024-03-18T16:07:47.408970+0000","last_clean":"2024-03-18T16:07:47.408970+0000","last_became_active":"2024-03-18T16:06:43.711469+0000","last_became_peered":"2024-03-18T16:06:43.711469+0000","last_unstale":"2024-03-18T16:07:47.408970+0000","last_undegraded":"2024-03-18T16:07:47.408970+0000","last_fullsized":"2024-03-18T16:07:47.408970+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:30:41.193229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T16:07:47.407580+0000","last_change":"2024-03-18T16:06:44.723953+0000","last_active":"2024-03-18T16:07:47.407580+0000","last_peered":"2024-03-18T16:07:47.407580+0000","last_clean":"2024-03-18T16:07:47.407580+0000","last_became_active":"2024-03-18T16:06:44.723266+0000","last_became_peered":"2024-03-18T16:06:44.723266+0000","last_unstale":"2024-03-18T16:07:47.407580+0000","last_undegraded":"2024-03-18T16:07:47.407580+0000","last_fullsized":"2024-03-18T16:07:47.407580+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:41:10.067297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.961683+0000","last_change":"2024-03-18T16:06:49.039472+0000","last_active":"2024-03-18T16:08:47.961683+0000","last_peered":"2024-03-18T16:08:47.961683+0000","last_clean":"2024-03-18T16:08:47.961683+0000","last_became_active":"2024-03-18T16:06:49.038889+0000","last_became_peered":"2024-03-18T16:06:49.038889+0000","last_unstale":"2024-03-18T16:08:47.961683+0000","last_undegraded":"2024-03-18T16:08:47.961683+0000","last_fullsized":"2024-03-18T16:08:47.961683+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:03:05.345315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-18T16:08:46.881499+0000","last_change":"2024-03-18T16:06:43.705873+0000","last_active":"2024-03-18T16:08:46.881499+0000","last_peered":"2024-03-18T16:08:46.881499+0000","last_clean":"2024-03-18T16:08:46.881499+0000","last_became_active":"2024-03-18T16:06:43.705548+0000","last_became_peered":"2024-03-18T16:06:43.705548+0000","last_unstale":"2024-03-18T16:08:46.881499+0000","last_undegraded":"2024-03-18T16:08:46.881499+0000","last_fullsized":"2024-03-18T16:08:46.881499+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:52:03.790888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T16:08:45.519041+0000","last_change":"2024-03-18T16:06:44.718526+0000","last_active":"2024-03-18T16:08:45.519041+0000","last_peered":"2024-03-18T16:08:45.519041+0000","last_clean":"2024-03-18T16:08:45.519041+0000","last_became_active":"2024-03-18T16:06:44.717266+0000","last_became_peered":"2024-03-18T16:06:44.717266+0000","last_unstale":"2024-03-18T16:08:45.519041+0000","last_undegraded":"2024-03-18T16:08:45.519041+0000","last_fullsized":"2024-03-18T16:08:45.519041+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:55:09.340354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T16:07:47.408065+0000","last_change":"2024-03-18T16:06:49.021971+0000","last_active":"2024-03-18T16:07:47.408065+0000","last_peered":"2024-03-18T16:07:47.408065+0000","last_clean":"2024-03-18T16:07:47.408065+0000","last_became_active":"2024-03-18T16:06:49.021577+0000","last_became_peered":"2024-03-18T16:06:49.021577+0000","last_unstale":"2024-03-18T16:07:47.408065+0000","last_undegraded":"2024-03-18T16:07:47.408065+0000","last_fullsized":"2024-03-18T16:07:47.408065+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:47:44.768559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"91'5","reported_seq":215,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:49.074685+0000","last_change":"2024-03-18T16:06:43.705636+0000","last_active":"2024-03-18T16:08:49.074685+0000","last_peered":"2024-03-18T16:08:49.074685+0000","last_clean":"2024-03-18T16:08:49.074685+0000","last_became_active":"2024-03-18T16:06:43.705268+0000","last_became_peered":"2024-03-18T16:06:43.705268+0000","last_unstale":"2024-03-18T16:08:49.074685+0000","last_undegraded":"2024-03-18T16:08:49.074685+0000","last_fullsized":"2024-03-18T16:08:49.074685+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:37:31.367130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":180,"num_read_kb":180,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T16:08:45.503571+0000","last_change":"2024-03-18T16:06:44.712237+0000","last_active":"2024-03-18T16:08:45.503571+0000","last_peered":"2024-03-18T16:08:45.503571+0000","last_clean":"2024-03-18T16:08:45.503571+0000","last_became_active":"2024-03-18T16:06:44.711677+0000","last_became_peered":"2024-03-18T16:06:44.711677+0000","last_unstale":"2024-03-18T16:08:45.503571+0000","last_undegraded":"2024-03-18T16:08:45.503571+0000","last_fullsized":"2024-03-18T16:08:45.503571+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:16:56.326962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T16:07:47.349360+0000","last_change":"2024-03-18T16:06:49.034578+0000","last_active":"2024-03-18T16:07:47.349360+0000","last_peered":"2024-03-18T16:07:47.349360+0000","last_clean":"2024-03-18T16:07:47.349360+0000","last_became_active":"2024-03-18T16:06:49.031099+0000","last_became_peered":"2024-03-18T16:06:49.031099+0000","last_unstale":"2024-03-18T16:07:47.349360+0000","last_undegraded":"2024-03-18T16:07:47.349360+0000","last_fullsized":"2024-03-18T16:07:47.349360+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:11:08.042550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.178256+0000","last_change":"2024-03-18T16:06:43.708151+0000","last_active":"2024-03-18T16:08:47.178256+0000","last_peered":"2024-03-18T16:08:47.178256+0000","last_clean":"2024-03-18T16:08:47.178256+0000","last_became_active":"2024-03-18T16:06:43.707810+0000","last_became_peered":"2024-03-18T16:06:43.707810+0000","last_unstale":"2024-03-18T16:08:47.178256+0000","last_undegraded":"2024-03-18T16:08:47.178256+0000","last_fullsized":"2024-03-18T16:08:47.178256+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:30:35.745772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.962153+0000","last_change":"2024-03-18T16:06:44.729777+0000","last_active":"2024-03-18T16:08:47.962153+0000","last_peered":"2024-03-18T16:08:47.962153+0000","last_clean":"2024-03-18T16:08:47.962153+0000","last_became_active":"2024-03-18T16:06:44.729244+0000","last_became_peered":"2024-03-18T16:06:44.729244+0000","last_unstale":"2024-03-18T16:08:47.962153+0000","last_undegraded":"2024-03-18T16:08:47.962153+0000","last_fullsized":"2024-03-18T16:08:47.962153+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:52:05.420443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883601+0000","last_change":"2024-03-18T16:07:47.355131+0000","last_active":"2024-03-18T16:08:47.883601+0000","last_peered":"2024-03-18T16:08:47.883601+0000","last_clean":"2024-03-18T16:08:47.883601+0000","last_became_active":"2024-03-18T16:07:47.354608+0000","last_became_peered":"2024-03-18T16:07:47.354608+0000","last_unstale":"2024-03-18T16:08:47.883601+0000","last_undegraded":"2024-03-18T16:08:47.883601+0000","last_fullsized":"2024-03-18T16:08:47.883601+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:12:57.979168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-18T16:08:46.881098+0000","last_change":"2024-03-18T16:06:43.715280+0000","last_active":"2024-03-18T16:08:46.881098+0000","last_peered":"2024-03-18T16:08:46.881098+0000","last_clean":"2024-03-18T16:08:46.881098+0000","last_became_active":"2024-03-18T16:06:43.714936+0000","last_became_peered":"2024-03-18T16:06:43.714936+0000","last_unstale":"2024-03-18T16:08:46.881098+0000","last_undegraded":"2024-03-18T16:08:46.881098+0000","last_fullsized":"2024-03-18T16:08:46.881098+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:07:38.746932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T16:08:45.506747+0000","last_change":"2024-03-18T16:06:44.709004+0000","last_active":"2024-03-18T16:08:45.506747+0000","last_peered":"2024-03-18T16:08:45.506747+0000","last_clean":"2024-03-18T16:08:45.506747+0000","last_became_active":"2024-03-18T16:06:44.708574+0000","last_became_peered":"2024-03-18T16:06:44.708574+0000","last_unstale":"2024-03-18T16:08:45.506747+0000","last_undegraded":"2024-03-18T16:08:45.506747+0000","last_fullsized":"2024-03-18T16:08:45.506747+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:02:50.982110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T16:07:47.349263+0000","last_change":"2024-03-18T16:06:49.029806+0000","last_active":"2024-03-18T16:07:47.349263+0000","last_peered":"2024-03-18T16:07:47.349263+0000","last_clean":"2024-03-18T16:07:47.349263+0000","last_became_active":"2024-03-18T16:06:49.029404+0000","last_became_peered":"2024-03-18T16:06:49.029404+0000","last_unstale":"2024-03-18T16:07:47.349263+0000","last_undegraded":"2024-03-18T16:07:47.349263+0000","last_fullsized":"2024-03-18T16:07:47.349263+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:38:37.706620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"91'37","reported_seq":256,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:49.075039+0000","last_change":"2024-03-18T16:06:43.702910+0000","last_active":"2024-03-18T16:08:49.075039+0000","last_peered":"2024-03-18T16:08:49.075039+0000","last_clean":"2024-03-18T16:08:49.075039+0000","last_became_active":"2024-03-18T16:06:43.702557+0000","last_became_peered":"2024-03-18T16:06:43.702557+0000","last_unstale":"2024-03-18T16:08:49.075039+0000","last_undegraded":"2024-03-18T16:08:49.075039+0000","last_fullsized":"2024-03-18T16:08:49.075039+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:55:15.362987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61741,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":189,"num_read_kb":155,"num_write":39,"num_write_kb":78,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-18T16:08:46.877413+0000","last_change":"2024-03-18T16:06:44.709075+0000","last_active":"2024-03-18T16:08:46.877413+0000","last_peered":"2024-03-18T16:08:46.877413+0000","last_clean":"2024-03-18T16:08:46.877413+0000","last_became_active":"2024-03-18T16:06:44.708366+0000","last_became_peered":"2024-03-18T16:06:44.708366+0000","last_unstale":"2024-03-18T16:08:46.877413+0000","last_undegraded":"2024-03-18T16:08:46.877413+0000","last_fullsized":"2024-03-18T16:08:46.877413+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:44:18.486385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-18T16:08:46.881369+0000","last_change":"2024-03-18T16:06:49.037380+0000","last_active":"2024-03-18T16:08:46.881369+0000","last_peered":"2024-03-18T16:08:46.881369+0000","last_clean":"2024-03-18T16:08:46.881369+0000","last_became_active":"2024-03-18T16:06:49.037021+0000","last_became_peered":"2024-03-18T16:06:49.037021+0000","last_unstale":"2024-03-18T16:08:46.881369+0000","last_undegraded":"2024-03-18T16:08:46.881369+0000","last_fullsized":"2024-03-18T16:08:46.881369+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:43:03.482631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T16:07:47.346986+0000","last_change":"2024-03-18T16:06:43.722287+0000","last_active":"2024-03-18T16:07:47.346986+0000","last_peered":"2024-03-18T16:07:47.346986+0000","last_clean":"2024-03-18T16:07:47.346986+0000","last_became_active":"2024-03-18T16:06:43.721919+0000","last_became_peered":"2024-03-18T16:06:43.721919+0000","last_unstale":"2024-03-18T16:07:47.346986+0000","last_undegraded":"2024-03-18T16:07:47.346986+0000","last_fullsized":"2024-03-18T16:07:47.346986+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:10:07.606112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T16:08:45.524649+0000","last_change":"2024-03-18T16:06:44.727204+0000","last_active":"2024-03-18T16:08:45.524649+0000","last_peered":"2024-03-18T16:08:45.524649+0000","last_clean":"2024-03-18T16:08:45.524649+0000","last_became_active":"2024-03-18T16:06:44.726775+0000","last_became_peered":"2024-03-18T16:06:44.726775+0000","last_unstale":"2024-03-18T16:08:45.524649+0000","last_undegraded":"2024-03-18T16:08:45.524649+0000","last_fullsized":"2024-03-18T16:08:45.524649+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:29:44.842972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T16:07:47.348313+0000","last_change":"2024-03-18T16:06:49.022508+0000","last_active":"2024-03-18T16:07:47.348313+0000","last_peered":"2024-03-18T16:07:47.348313+0000","last_clean":"2024-03-18T16:07:47.348313+0000","last_became_active":"2024-03-18T16:06:49.022017+0000","last_became_peered":"2024-03-18T16:06:49.022017+0000","last_unstale":"2024-03-18T16:07:47.348313+0000","last_undegraded":"2024-03-18T16:07:47.348313+0000","last_fullsized":"2024-03-18T16:07:47.348313+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:52:45.892880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-18T16:08:46.880999+0000","last_change":"2024-03-18T16:06:43.719618+0000","last_active":"2024-03-18T16:08:46.880999+0000","last_peered":"2024-03-18T16:08:46.880999+0000","last_clean":"2024-03-18T16:08:46.880999+0000","last_became_active":"2024-03-18T16:06:43.719161+0000","last_became_peered":"2024-03-18T16:06:43.719161+0000","last_unstale":"2024-03-18T16:08:46.880999+0000","last_undegraded":"2024-03-18T16:08:46.880999+0000","last_fullsized":"2024-03-18T16:08:46.880999+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:39:53.077878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":42,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.177980+0000","last_change":"2024-03-18T16:06:44.717829+0000","last_active":"2024-03-18T16:08:47.177980+0000","last_peered":"2024-03-18T16:08:47.177980+0000","last_clean":"2024-03-18T16:08:47.177980+0000","last_became_active":"2024-03-18T16:06:44.717473+0000","last_became_peered":"2024-03-18T16:06:44.717473+0000","last_unstale":"2024-03-18T16:08:47.177980+0000","last_undegraded":"2024-03-18T16:08:47.177980+0000","last_fullsized":"2024-03-18T16:08:47.177980+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:10:00.183613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T16:07:47.350853+0000","last_change":"2024-03-18T16:06:49.036766+0000","last_active":"2024-03-18T16:07:47.350853+0000","last_peered":"2024-03-18T16:07:47.350853+0000","last_clean":"2024-03-18T16:07:47.350853+0000","last_became_active":"2024-03-18T16:06:49.034617+0000","last_became_peered":"2024-03-18T16:06:49.034617+0000","last_unstale":"2024-03-18T16:07:47.350853+0000","last_undegraded":"2024-03-18T16:07:47.350853+0000","last_fullsized":"2024-03-18T16:07:47.350853+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:18:59.033781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'5","reported_seq":125,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883370+0000","last_change":"2024-03-18T16:06:43.702452+0000","last_active":"2024-03-18T16:08:47.883370+0000","last_peered":"2024-03-18T16:08:47.883370+0000","last_clean":"2024-03-18T16:08:47.883370+0000","last_became_active":"2024-03-18T16:06:43.702141+0000","last_became_peered":"2024-03-18T16:06:43.702141+0000","last_unstale":"2024-03-18T16:08:47.883370+0000","last_undegraded":"2024-03-18T16:08:47.883370+0000","last_fullsized":"2024-03-18T16:08:47.883370+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:44:05.366101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":91,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.177865+0000","last_change":"2024-03-18T16:06:44.711403+0000","last_active":"2024-03-18T16:08:47.177865+0000","last_peered":"2024-03-18T16:08:47.177865+0000","last_clean":"2024-03-18T16:08:47.177865+0000","last_became_active":"2024-03-18T16:06:44.710654+0000","last_became_peered":"2024-03-18T16:06:44.710654+0000","last_unstale":"2024-03-18T16:08:47.177865+0000","last_undegraded":"2024-03-18T16:08:47.177865+0000","last_fullsized":"2024-03-18T16:08:47.177865+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:21:43.633274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T16:07:47.409226+0000","last_change":"2024-03-18T16:06:49.039192+0000","last_active":"2024-03-18T16:07:47.409226+0000","last_peered":"2024-03-18T16:07:47.409226+0000","last_clean":"2024-03-18T16:07:47.409226+0000","last_became_active":"2024-03-18T16:06:49.038421+0000","last_became_peered":"2024-03-18T16:06:49.038421+0000","last_unstale":"2024-03-18T16:07:47.409226+0000","last_undegraded":"2024-03-18T16:07:47.409226+0000","last_fullsized":"2024-03-18T16:07:47.409226+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:54:56.944209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":46,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-18T16:08:46.876824+0000","last_change":"2024-03-18T16:06:43.712523+0000","last_active":"2024-03-18T16:08:46.876824+0000","last_peered":"2024-03-18T16:08:46.876824+0000","last_clean":"2024-03-18T16:08:46.876824+0000","last_became_active":"2024-03-18T16:06:43.712151+0000","last_became_peered":"2024-03-18T16:06:43.712151+0000","last_unstale":"2024-03-18T16:08:46.876824+0000","last_undegraded":"2024-03-18T16:08:46.876824+0000","last_fullsized":"2024-03-18T16:08:46.876824+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:17:19.651293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":38,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-18T16:08:46.881304+0000","last_change":"2024-03-18T16:07:47.355115+0000","last_active":"2024-03-18T16:08:46.881304+0000","last_peered":"2024-03-18T16:08:46.881304+0000","last_clean":"2024-03-18T16:08:46.881304+0000","last_became_active":"2024-03-18T16:07:47.354744+0000","last_became_peered":"2024-03-18T16:07:47.354744+0000","last_unstale":"2024-03-18T16:08:46.881304+0000","last_undegraded":"2024-03-18T16:08:46.881304+0000","last_fullsized":"2024-03-18T16:08:46.881304+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:14:27.644891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-18T16:08:46.881196+0000","last_change":"2024-03-18T16:06:49.028943+0000","last_active":"2024-03-18T16:08:46.881196+0000","last_peered":"2024-03-18T16:08:46.881196+0000","last_clean":"2024-03-18T16:08:46.881196+0000","last_became_active":"2024-03-18T16:06:49.028453+0000","last_became_peered":"2024-03-18T16:06:49.028453+0000","last_unstale":"2024-03-18T16:08:46.881196+0000","last_undegraded":"2024-03-18T16:08:46.881196+0000","last_fullsized":"2024-03-18T16:08:46.881196+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:16:30.318649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T16:07:47.350674+0000","last_change":"2024-03-18T16:06:43.706840+0000","last_active":"2024-03-18T16:07:47.350674+0000","last_peered":"2024-03-18T16:07:47.350674+0000","last_clean":"2024-03-18T16:07:47.350674+0000","last_became_active":"2024-03-18T16:06:43.706486+0000","last_became_peered":"2024-03-18T16:06:43.706486+0000","last_unstale":"2024-03-18T16:07:47.350674+0000","last_undegraded":"2024-03-18T16:07:47.350674+0000","last_fullsized":"2024-03-18T16:07:47.350674+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:16:44.542471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T16:08:45.492422+0000","last_change":"2024-03-18T16:06:44.716284+0000","last_active":"2024-03-18T16:08:45.492422+0000","last_peered":"2024-03-18T16:08:45.492422+0000","last_clean":"2024-03-18T16:08:45.492422+0000","last_became_active":"2024-03-18T16:06:44.715911+0000","last_became_peered":"2024-03-18T16:06:44.715911+0000","last_unstale":"2024-03-18T16:08:45.492422+0000","last_undegraded":"2024-03-18T16:08:45.492422+0000","last_fullsized":"2024-03-18T16:08:45.492422+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:48:34.839514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-18T16:08:46.877074+0000","last_change":"2024-03-18T16:06:49.059061+0000","last_active":"2024-03-18T16:08:46.877074+0000","last_peered":"2024-03-18T16:08:46.877074+0000","last_clean":"2024-03-18T16:08:46.877074+0000","last_became_active":"2024-03-18T16:06:49.058619+0000","last_became_peered":"2024-03-18T16:06:49.058619+0000","last_unstale":"2024-03-18T16:08:46.877074+0000","last_undegraded":"2024-03-18T16:08:46.877074+0000","last_fullsized":"2024-03-18T16:08:46.877074+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:44:52.970785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T16:07:47.349883+0000","last_change":"2024-03-18T16:06:43.715766+0000","last_active":"2024-03-18T16:07:47.349883+0000","last_peered":"2024-03-18T16:07:47.349883+0000","last_clean":"2024-03-18T16:07:47.349883+0000","last_became_active":"2024-03-18T16:06:43.715380+0000","last_became_peered":"2024-03-18T16:06:43.715380+0000","last_unstale":"2024-03-18T16:07:47.349883+0000","last_undegraded":"2024-03-18T16:07:47.349883+0000","last_fullsized":"2024-03-18T16:07:47.349883+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:13:44.195934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.880175+0000","last_change":"2024-03-18T16:06:44.728150+0000","last_active":"2024-03-18T16:08:47.880175+0000","last_peered":"2024-03-18T16:08:47.880175+0000","last_clean":"2024-03-18T16:08:47.880175+0000","last_became_active":"2024-03-18T16:06:44.727769+0000","last_became_peered":"2024-03-18T16:06:44.727769+0000","last_unstale":"2024-03-18T16:08:47.880175+0000","last_undegraded":"2024-03-18T16:08:47.880175+0000","last_fullsized":"2024-03-18T16:08:47.880175+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:28:16.023225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T16:07:47.408441+0000","last_change":"2024-03-18T16:06:49.033671+0000","last_active":"2024-03-18T16:07:47.408441+0000","last_peered":"2024-03-18T16:07:47.408441+0000","last_clean":"2024-03-18T16:07:47.408441+0000","last_became_active":"2024-03-18T16:06:49.033046+0000","last_became_peered":"2024-03-18T16:06:49.033046+0000","last_unstale":"2024-03-18T16:07:47.408441+0000","last_undegraded":"2024-03-18T16:07:47.408441+0000","last_fullsized":"2024-03-18T16:07:47.408441+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:26:37.020521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'5","reported_seq":122,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.178006+0000","last_change":"2024-03-18T16:06:43.712636+0000","last_active":"2024-03-18T16:08:47.178006+0000","last_peered":"2024-03-18T16:08:47.178006+0000","last_clean":"2024-03-18T16:08:47.178006+0000","last_became_active":"2024-03-18T16:06:43.712300+0000","last_became_peered":"2024-03-18T16:06:43.712300+0000","last_unstale":"2024-03-18T16:08:47.178006+0000","last_undegraded":"2024-03-18T16:08:47.178006+0000","last_fullsized":"2024-03-18T16:08:47.178006+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:48:20.386640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":88,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.880029+0000","last_change":"2024-03-18T16:06:44.731583+0000","last_active":"2024-03-18T16:08:47.880029+0000","last_peered":"2024-03-18T16:08:47.880029+0000","last_clean":"2024-03-18T16:08:47.880029+0000","last_became_active":"2024-03-18T16:06:44.731183+0000","last_became_peered":"2024-03-18T16:06:44.731183+0000","last_unstale":"2024-03-18T16:08:47.880029+0000","last_undegraded":"2024-03-18T16:08:47.880029+0000","last_fullsized":"2024-03-18T16:08:47.880029+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:23:05.098101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T16:07:47.409465+0000","last_change":"2024-03-18T16:06:49.060130+0000","last_active":"2024-03-18T16:07:47.409465+0000","last_peered":"2024-03-18T16:07:47.409465+0000","last_clean":"2024-03-18T16:07:47.409465+0000","last_became_active":"2024-03-18T16:06:49.059109+0000","last_became_peered":"2024-03-18T16:06:49.059109+0000","last_unstale":"2024-03-18T16:07:47.409465+0000","last_undegraded":"2024-03-18T16:07:47.409465+0000","last_fullsized":"2024-03-18T16:07:47.409465+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:41:40.008047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":35,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-18T16:08:46.880920+0000","last_change":"2024-03-18T16:06:44.705793+0000","last_active":"2024-03-18T16:08:46.880920+0000","last_peered":"2024-03-18T16:08:46.880920+0000","last_clean":"2024-03-18T16:08:46.880920+0000","last_became_active":"2024-03-18T16:06:44.705426+0000","last_became_peered":"2024-03-18T16:06:44.705426+0000","last_unstale":"2024-03-18T16:08:46.880920+0000","last_undegraded":"2024-03-18T16:08:46.880920+0000","last_fullsized":"2024-03-18T16:08:46.880920+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:14:31.198829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'5","reported_seq":119,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T16:08:45.112558+0000","last_change":"2024-03-18T16:06:43.721057+0000","last_active":"2024-03-18T16:08:45.112558+0000","last_peered":"2024-03-18T16:08:45.112558+0000","last_clean":"2024-03-18T16:08:45.112558+0000","last_became_active":"2024-03-18T16:06:43.720661+0000","last_became_peered":"2024-03-18T16:06:43.720661+0000","last_unstale":"2024-03-18T16:08:45.112558+0000","last_undegraded":"2024-03-18T16:08:45.112558+0000","last_fullsized":"2024-03-18T16:08:45.112558+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:08:18.050511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":88,"num_read_kb":88,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.178376+0000","last_change":"2024-03-18T16:07:47.361447+0000","last_active":"2024-03-18T16:08:47.178376+0000","last_peered":"2024-03-18T16:08:47.178376+0000","last_clean":"2024-03-18T16:08:47.178376+0000","last_became_active":"2024-03-18T16:07:47.353684+0000","last_became_peered":"2024-03-18T16:07:47.353684+0000","last_unstale":"2024-03-18T16:08:47.178376+0000","last_undegraded":"2024-03-18T16:08:47.178376+0000","last_fullsized":"2024-03-18T16:08:47.178376+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:59:53.025620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,10],"acting":[7,5,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T16:07:47.408020+0000","last_change":"2024-03-18T16:06:44.725555+0000","last_active":"2024-03-18T16:07:47.408020+0000","last_peered":"2024-03-18T16:07:47.408020+0000","last_clean":"2024-03-18T16:07:47.408020+0000","last_became_active":"2024-03-18T16:06:44.719880+0000","last_became_peered":"2024-03-18T16:06:44.719880+0000","last_unstale":"2024-03-18T16:07:47.408020+0000","last_undegraded":"2024-03-18T16:07:47.408020+0000","last_fullsized":"2024-03-18T16:07:47.408020+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:45:17.830876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883578+0000","last_change":"2024-03-18T16:06:43.700814+0000","last_active":"2024-03-18T16:08:47.883578+0000","last_peered":"2024-03-18T16:08:47.883578+0000","last_clean":"2024-03-18T16:08:47.883578+0000","last_became_active":"2024-03-18T16:06:43.698902+0000","last_became_peered":"2024-03-18T16:06:43.698902+0000","last_unstale":"2024-03-18T16:08:47.883578+0000","last_undegraded":"2024-03-18T16:08:47.883578+0000","last_fullsized":"2024-03-18T16:08:47.883578+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:20:20.100033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":351,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.177745+0000","last_change":"2024-03-18T16:05:22.379397+0000","last_active":"2024-03-18T16:08:47.177745+0000","last_peered":"2024-03-18T16:08:47.177745+0000","last_clean":"2024-03-18T16:08:47.177745+0000","last_became_active":"2024-03-18T16:05:22.378830+0000","last_became_peered":"2024-03-18T16:05:22.378830+0000","last_unstale":"2024-03-18T16:08:47.177745+0000","last_undegraded":"2024-03-18T16:08:47.177745+0000","last_fullsized":"2024-03-18T16:08:47.177745+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:01:30.048984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:01:30.048984+0000","last_clean_scrub_stamp":"2024-03-18T16:01:30.048984+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:35:45.796495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-18T16:08:46.881389+0000","last_change":"2024-03-18T16:06:49.058931+0000","last_active":"2024-03-18T16:08:46.881389+0000","last_peered":"2024-03-18T16:08:46.881389+0000","last_clean":"2024-03-18T16:08:46.881389+0000","last_became_active":"2024-03-18T16:06:49.058497+0000","last_became_peered":"2024-03-18T16:06:49.058497+0000","last_unstale":"2024-03-18T16:08:46.881389+0000","last_undegraded":"2024-03-18T16:08:46.881389+0000","last_fullsized":"2024-03-18T16:08:46.881389+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:41:51.874582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T16:08:45.531104+0000","last_change":"2024-03-18T16:06:44.710332+0000","last_active":"2024-03-18T16:08:45.531104+0000","last_peered":"2024-03-18T16:08:45.531104+0000","last_clean":"2024-03-18T16:08:45.531104+0000","last_became_active":"2024-03-18T16:06:44.709952+0000","last_became_peered":"2024-03-18T16:06:44.709952+0000","last_unstale":"2024-03-18T16:08:45.531104+0000","last_undegraded":"2024-03-18T16:08:45.531104+0000","last_fullsized":"2024-03-18T16:08:45.531104+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:24:26.653511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.177836+0000","last_change":"2024-03-18T16:06:43.708265+0000","last_active":"2024-03-18T16:08:47.177836+0000","last_peered":"2024-03-18T16:08:47.177836+0000","last_clean":"2024-03-18T16:08:47.177836+0000","last_became_active":"2024-03-18T16:06:43.707466+0000","last_became_peered":"2024-03-18T16:06:43.707466+0000","last_unstale":"2024-03-18T16:08:47.177836+0000","last_undegraded":"2024-03-18T16:08:47.177836+0000","last_fullsized":"2024-03-18T16:08:47.177836+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:45:40.065396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T16:07:47.348160+0000","last_change":"2024-03-18T16:06:49.022401+0000","last_active":"2024-03-18T16:07:47.348160+0000","last_peered":"2024-03-18T16:07:47.348160+0000","last_clean":"2024-03-18T16:07:47.348160+0000","last_became_active":"2024-03-18T16:06:49.021966+0000","last_became_peered":"2024-03-18T16:06:49.021966+0000","last_unstale":"2024-03-18T16:07:47.348160+0000","last_undegraded":"2024-03-18T16:07:47.348160+0000","last_fullsized":"2024-03-18T16:07:47.348160+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:04:12.434969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-18T16:08:46.877182+0000","last_change":"2024-03-18T16:06:43.713249+0000","last_active":"2024-03-18T16:08:46.877182+0000","last_peered":"2024-03-18T16:08:46.877182+0000","last_clean":"2024-03-18T16:08:46.877182+0000","last_became_active":"2024-03-18T16:06:43.712897+0000","last_became_peered":"2024-03-18T16:06:43.712897+0000","last_unstale":"2024-03-18T16:08:46.877182+0000","last_undegraded":"2024-03-18T16:08:46.877182+0000","last_fullsized":"2024-03-18T16:08:46.877182+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:20:04.642568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T16:08:45.535352+0000","last_change":"2024-03-18T16:06:44.710034+0000","last_active":"2024-03-18T16:08:45.535352+0000","last_peered":"2024-03-18T16:08:45.535352+0000","last_clean":"2024-03-18T16:08:45.535352+0000","last_became_active":"2024-03-18T16:06:44.708457+0000","last_became_peered":"2024-03-18T16:06:44.708457+0000","last_unstale":"2024-03-18T16:08:45.535352+0000","last_undegraded":"2024-03-18T16:08:45.535352+0000","last_fullsized":"2024-03-18T16:08:45.535352+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:40:00.504382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T16:07:47.407177+0000","last_change":"2024-03-18T16:06:49.059345+0000","last_active":"2024-03-18T16:07:47.407177+0000","last_peered":"2024-03-18T16:07:47.407177+0000","last_clean":"2024-03-18T16:07:47.407177+0000","last_became_active":"2024-03-18T16:06:49.058675+0000","last_became_peered":"2024-03-18T16:06:49.058675+0000","last_unstale":"2024-03-18T16:07:47.407177+0000","last_undegraded":"2024-03-18T16:07:47.407177+0000","last_fullsized":"2024-03-18T16:07:47.407177+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:03:33.730074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'14","reported_seq":41,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T16:07:47.351825+0000","last_change":"2024-03-18T16:06:43.724589+0000","last_active":"2024-03-18T16:07:47.351825+0000","last_peered":"2024-03-18T16:07:47.351825+0000","last_clean":"2024-03-18T16:07:47.351825+0000","last_became_active":"2024-03-18T16:06:43.724125+0000","last_became_peered":"2024-03-18T16:06:43.724125+0000","last_unstale":"2024-03-18T16:07:47.351825+0000","last_undegraded":"2024-03-18T16:07:47.351825+0000","last_fullsized":"2024-03-18T16:07:47.351825+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:08:34.921118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T16:07:47.410559+0000","last_change":"2024-03-18T16:06:44.727162+0000","last_active":"2024-03-18T16:07:47.410559+0000","last_peered":"2024-03-18T16:07:47.410559+0000","last_clean":"2024-03-18T16:07:47.410559+0000","last_became_active":"2024-03-18T16:06:44.726552+0000","last_became_peered":"2024-03-18T16:06:44.726552+0000","last_unstale":"2024-03-18T16:07:47.410559+0000","last_undegraded":"2024-03-18T16:07:47.410559+0000","last_fullsized":"2024-03-18T16:07:47.410559+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:36:57.974999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883760+0000","last_change":"2024-03-18T16:06:49.008434+0000","last_active":"2024-03-18T16:08:47.883760+0000","last_peered":"2024-03-18T16:08:47.883760+0000","last_clean":"2024-03-18T16:08:47.883760+0000","last_became_active":"2024-03-18T16:06:49.007944+0000","last_became_peered":"2024-03-18T16:06:49.007944+0000","last_unstale":"2024-03-18T16:08:47.883760+0000","last_undegraded":"2024-03-18T16:08:47.883760+0000","last_fullsized":"2024-03-18T16:08:47.883760+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:01:24.450371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883875+0000","last_change":"2024-03-18T16:06:43.711797+0000","last_active":"2024-03-18T16:08:47.883875+0000","last_peered":"2024-03-18T16:08:47.883875+0000","last_clean":"2024-03-18T16:08:47.883875+0000","last_became_active":"2024-03-18T16:06:43.711493+0000","last_became_peered":"2024-03-18T16:06:43.711493+0000","last_unstale":"2024-03-18T16:08:47.883875+0000","last_undegraded":"2024-03-18T16:08:47.883875+0000","last_fullsized":"2024-03-18T16:08:47.883875+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:50:29.258166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.961855+0000","last_change":"2024-03-18T16:06:44.735910+0000","last_active":"2024-03-18T16:08:47.961855+0000","last_peered":"2024-03-18T16:08:47.961855+0000","last_clean":"2024-03-18T16:08:47.961855+0000","last_became_active":"2024-03-18T16:06:44.735504+0000","last_became_peered":"2024-03-18T16:06:44.735504+0000","last_unstale":"2024-03-18T16:08:47.961855+0000","last_undegraded":"2024-03-18T16:08:47.961855+0000","last_fullsized":"2024-03-18T16:08:47.961855+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:02:18.538217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883643+0000","last_change":"2024-03-18T16:06:49.059130+0000","last_active":"2024-03-18T16:08:47.883643+0000","last_peered":"2024-03-18T16:08:47.883643+0000","last_clean":"2024-03-18T16:08:47.883643+0000","last_became_active":"2024-03-18T16:06:49.058701+0000","last_became_peered":"2024-03-18T16:06:49.058701+0000","last_unstale":"2024-03-18T16:08:47.883643+0000","last_undegraded":"2024-03-18T16:08:47.883643+0000","last_fullsized":"2024-03-18T16:08:47.883643+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:54:52.842786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T16:07:47.354154+0000","last_change":"2024-03-18T16:06:43.714414+0000","last_active":"2024-03-18T16:07:47.354154+0000","last_peered":"2024-03-18T16:07:47.354154+0000","last_clean":"2024-03-18T16:07:47.354154+0000","last_became_active":"2024-03-18T16:06:43.714029+0000","last_became_peered":"2024-03-18T16:06:43.714029+0000","last_unstale":"2024-03-18T16:07:47.354154+0000","last_undegraded":"2024-03-18T16:07:47.354154+0000","last_fullsized":"2024-03-18T16:07:47.354154+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:16:39.060378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":28,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-18T16:08:46.876845+0000","last_change":"2024-03-18T16:06:44.707491+0000","last_active":"2024-03-18T16:08:46.876845+0000","last_peered":"2024-03-18T16:08:46.876845+0000","last_clean":"2024-03-18T16:08:46.876845+0000","last_became_active":"2024-03-18T16:06:44.707157+0000","last_became_peered":"2024-03-18T16:06:44.707157+0000","last_unstale":"2024-03-18T16:08:46.876845+0000","last_undegraded":"2024-03-18T16:08:46.876845+0000","last_fullsized":"2024-03-18T16:08:46.876845+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:19:08.132911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-18T16:08:46.877329+0000","last_change":"2024-03-18T16:07:47.411002+0000","last_active":"2024-03-18T16:08:46.877329+0000","last_peered":"2024-03-18T16:08:46.877329+0000","last_clean":"2024-03-18T16:08:46.877329+0000","last_became_active":"2024-03-18T16:07:47.410575+0000","last_became_peered":"2024-03-18T16:07:47.410575+0000","last_unstale":"2024-03-18T16:08:46.877329+0000","last_undegraded":"2024-03-18T16:08:46.877329+0000","last_fullsized":"2024-03-18T16:08:46.877329+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:37:25.311205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,2,5],"acting":[8,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T16:07:47.407682+0000","last_change":"2024-03-18T16:06:43.717755+0000","last_active":"2024-03-18T16:07:47.407682+0000","last_peered":"2024-03-18T16:07:47.407682+0000","last_clean":"2024-03-18T16:07:47.407682+0000","last_became_active":"2024-03-18T16:06:43.717278+0000","last_became_peered":"2024-03-18T16:06:43.717278+0000","last_unstale":"2024-03-18T16:07:47.407682+0000","last_undegraded":"2024-03-18T16:07:47.407682+0000","last_fullsized":"2024-03-18T16:07:47.407682+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:48:09.390659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T16:08:45.519173+0000","last_change":"2024-03-18T16:06:44.730858+0000","last_active":"2024-03-18T16:08:45.519173+0000","last_peered":"2024-03-18T16:08:45.519173+0000","last_clean":"2024-03-18T16:08:45.519173+0000","last_became_active":"2024-03-18T16:06:44.730500+0000","last_became_peered":"2024-03-18T16:06:44.730500+0000","last_unstale":"2024-03-18T16:08:45.519173+0000","last_undegraded":"2024-03-18T16:08:45.519173+0000","last_fullsized":"2024-03-18T16:08:45.519173+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:22:03.590834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883553+0000","last_change":"2024-03-18T16:06:49.014717+0000","last_active":"2024-03-18T16:08:47.883553+0000","last_peered":"2024-03-18T16:08:47.883553+0000","last_clean":"2024-03-18T16:08:47.883553+0000","last_became_active":"2024-03-18T16:06:49.014313+0000","last_became_peered":"2024-03-18T16:06:49.014313+0000","last_unstale":"2024-03-18T16:08:47.883553+0000","last_undegraded":"2024-03-18T16:08:47.883553+0000","last_fullsized":"2024-03-18T16:08:47.883553+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:06:12.168888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'3","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883632+0000","last_change":"2024-03-18T16:06:43.719496+0000","last_active":"2024-03-18T16:08:47.883632+0000","last_peered":"2024-03-18T16:08:47.883632+0000","last_clean":"2024-03-18T16:08:47.883632+0000","last_became_active":"2024-03-18T16:06:43.719139+0000","last_became_peered":"2024-03-18T16:06:43.719139+0000","last_unstale":"2024-03-18T16:08:47.883632+0000","last_undegraded":"2024-03-18T16:08:47.883632+0000","last_fullsized":"2024-03-18T16:08:47.883632+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:10:01.250923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T16:07:47.408988+0000","last_change":"2024-03-18T16:06:44.737792+0000","last_active":"2024-03-18T16:07:47.408988+0000","last_peered":"2024-03-18T16:07:47.408988+0000","last_clean":"2024-03-18T16:07:47.408988+0000","last_became_active":"2024-03-18T16:06:44.736311+0000","last_became_peered":"2024-03-18T16:06:44.736311+0000","last_unstale":"2024-03-18T16:07:47.408988+0000","last_undegraded":"2024-03-18T16:07:47.408988+0000","last_fullsized":"2024-03-18T16:07:47.408988+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:43:56.110205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883609+0000","last_change":"2024-03-18T16:06:49.058893+0000","last_active":"2024-03-18T16:08:47.883609+0000","last_peered":"2024-03-18T16:08:47.883609+0000","last_clean":"2024-03-18T16:08:47.883609+0000","last_became_active":"2024-03-18T16:06:49.058477+0000","last_became_peered":"2024-03-18T16:06:49.058477+0000","last_unstale":"2024-03-18T16:08:47.883609+0000","last_undegraded":"2024-03-18T16:08:47.883609+0000","last_fullsized":"2024-03-18T16:08:47.883609+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:50:51.890348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":33,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-18T16:08:46.877015+0000","last_change":"2024-03-18T16:06:43.703073+0000","last_active":"2024-03-18T16:08:46.877015+0000","last_peered":"2024-03-18T16:08:46.877015+0000","last_clean":"2024-03-18T16:08:46.877015+0000","last_became_active":"2024-03-18T16:06:43.702708+0000","last_became_peered":"2024-03-18T16:06:43.702708+0000","last_unstale":"2024-03-18T16:08:46.877015+0000","last_undegraded":"2024-03-18T16:08:46.877015+0000","last_fullsized":"2024-03-18T16:08:46.877015+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:38:41.454458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.178041+0000","last_change":"2024-03-18T16:06:44.713622+0000","last_active":"2024-03-18T16:08:47.178041+0000","last_peered":"2024-03-18T16:08:47.178041+0000","last_clean":"2024-03-18T16:08:47.178041+0000","last_became_active":"2024-03-18T16:06:44.713160+0000","last_became_peered":"2024-03-18T16:06:44.713160+0000","last_unstale":"2024-03-18T16:08:47.178041+0000","last_undegraded":"2024-03-18T16:08:47.178041+0000","last_fullsized":"2024-03-18T16:08:47.178041+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:49:21.886543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T16:07:47.411958+0000","last_change":"2024-03-18T16:07:47.411958+0000","last_active":"2024-03-18T16:07:47.411958+0000","last_peered":"2024-03-18T16:07:47.411958+0000","last_clean":"2024-03-18T16:07:47.411958+0000","last_became_active":"2024-03-18T16:07:47.411433+0000","last_became_peered":"2024-03-18T16:07:47.411433+0000","last_unstale":"2024-03-18T16:07:47.411958+0000","last_undegraded":"2024-03-18T16:07:47.411958+0000","last_fullsized":"2024-03-18T16:07:47.411958+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:58:47.853897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6,7],"acting":[1,8,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'5","reported_seq":126,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.884800+0000","last_change":"2024-03-18T16:06:43.717161+0000","last_active":"2024-03-18T16:08:47.884800+0000","last_peered":"2024-03-18T16:08:47.884800+0000","last_clean":"2024-03-18T16:08:47.884800+0000","last_became_active":"2024-03-18T16:06:43.716657+0000","last_became_peered":"2024-03-18T16:06:43.716657+0000","last_unstale":"2024-03-18T16:08:47.884800+0000","last_undegraded":"2024-03-18T16:08:47.884800+0000","last_fullsized":"2024-03-18T16:08:47.884800+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:23:48.194779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":91,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":34,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.178162+0000","last_change":"2024-03-18T16:06:44.709777+0000","last_active":"2024-03-18T16:08:47.178162+0000","last_peered":"2024-03-18T16:08:47.178162+0000","last_clean":"2024-03-18T16:08:47.178162+0000","last_became_active":"2024-03-18T16:06:44.709409+0000","last_became_peered":"2024-03-18T16:06:44.709409+0000","last_unstale":"2024-03-18T16:08:47.178162+0000","last_undegraded":"2024-03-18T16:08:47.178162+0000","last_fullsized":"2024-03-18T16:08:47.178162+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:37:12.288262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.961469+0000","last_change":"2024-03-18T16:06:49.040872+0000","last_active":"2024-03-18T16:08:47.961469+0000","last_peered":"2024-03-18T16:08:47.961469+0000","last_clean":"2024-03-18T16:08:47.961469+0000","last_became_active":"2024-03-18T16:06:49.039807+0000","last_became_peered":"2024-03-18T16:06:49.039807+0000","last_unstale":"2024-03-18T16:08:47.961469+0000","last_undegraded":"2024-03-18T16:08:47.961469+0000","last_fullsized":"2024-03-18T16:08:47.961469+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:09:27.045450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"89'3","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-18T16:08:46.881217+0000","last_change":"2024-03-18T16:06:43.723675+0000","last_active":"2024-03-18T16:08:46.881217+0000","last_peered":"2024-03-18T16:08:46.881217+0000","last_clean":"2024-03-18T16:08:46.881217+0000","last_became_active":"2024-03-18T16:06:43.723334+0000","last_became_peered":"2024-03-18T16:06:43.723334+0000","last_unstale":"2024-03-18T16:08:46.881217+0000","last_undegraded":"2024-03-18T16:08:46.881217+0000","last_fullsized":"2024-03-18T16:08:46.881217+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:01:36.553362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":39,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.178014+0000","last_change":"2024-03-18T16:07:47.359134+0000","last_active":"2024-03-18T16:08:47.178014+0000","last_peered":"2024-03-18T16:08:47.178014+0000","last_clean":"2024-03-18T16:08:47.178014+0000","last_became_active":"2024-03-18T16:07:47.358692+0000","last_became_peered":"2024-03-18T16:07:47.358692+0000","last_unstale":"2024-03-18T16:08:47.178014+0000","last_undegraded":"2024-03-18T16:08:47.178014+0000","last_fullsized":"2024-03-18T16:08:47.178014+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:48:22.929034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":20,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-18T16:08:46.880618+0000","last_change":"2024-03-18T16:06:49.011981+0000","last_active":"2024-03-18T16:08:46.880618+0000","last_peered":"2024-03-18T16:08:46.880618+0000","last_clean":"2024-03-18T16:08:46.880618+0000","last_became_active":"2024-03-18T16:06:49.011621+0000","last_became_peered":"2024-03-18T16:06:49.011621+0000","last_unstale":"2024-03-18T16:08:46.880618+0000","last_undegraded":"2024-03-18T16:08:46.880618+0000","last_fullsized":"2024-03-18T16:08:46.880618+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:11:42.382721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T16:07:47.409073+0000","last_change":"2024-03-18T16:06:43.725404+0000","last_active":"2024-03-18T16:07:47.409073+0000","last_peered":"2024-03-18T16:07:47.409073+0000","last_clean":"2024-03-18T16:07:47.409073+0000","last_became_active":"2024-03-18T16:06:43.724117+0000","last_became_peered":"2024-03-18T16:06:43.724117+0000","last_unstale":"2024-03-18T16:07:47.409073+0000","last_undegraded":"2024-03-18T16:07:47.409073+0000","last_fullsized":"2024-03-18T16:07:47.409073+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:58:43.243070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":36,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.178343+0000","last_change":"2024-03-18T16:06:44.709979+0000","last_active":"2024-03-18T16:08:47.178343+0000","last_peered":"2024-03-18T16:08:47.178343+0000","last_clean":"2024-03-18T16:08:47.178343+0000","last_became_active":"2024-03-18T16:06:44.709591+0000","last_became_peered":"2024-03-18T16:06:44.709591+0000","last_unstale":"2024-03-18T16:08:47.178343+0000","last_undegraded":"2024-03-18T16:08:47.178343+0000","last_fullsized":"2024-03-18T16:08:47.178343+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:56:37.537248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T16:08:45.499645+0000","last_change":"2024-03-18T16:07:47.354636+0000","last_active":"2024-03-18T16:08:45.499645+0000","last_peered":"2024-03-18T16:08:45.499645+0000","last_clean":"2024-03-18T16:08:45.499645+0000","last_became_active":"2024-03-18T16:07:47.354219+0000","last_became_peered":"2024-03-18T16:07:47.354219+0000","last_unstale":"2024-03-18T16:08:45.499645+0000","last_undegraded":"2024-03-18T16:08:45.499645+0000","last_fullsized":"2024-03-18T16:08:45.499645+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:35:48.094404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,9],"acting":[11,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883718+0000","last_change":"2024-03-18T16:06:44.714836+0000","last_active":"2024-03-18T16:08:47.883718+0000","last_peered":"2024-03-18T16:08:47.883718+0000","last_clean":"2024-03-18T16:08:47.883718+0000","last_became_active":"2024-03-18T16:06:44.714454+0000","last_became_peered":"2024-03-18T16:06:44.714454+0000","last_unstale":"2024-03-18T16:08:47.883718+0000","last_undegraded":"2024-03-18T16:08:47.883718+0000","last_fullsized":"2024-03-18T16:08:47.883718+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:05:24.532238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883502+0000","last_change":"2024-03-18T16:06:44.733183+0000","last_active":"2024-03-18T16:08:47.883502+0000","last_peered":"2024-03-18T16:08:47.883502+0000","last_clean":"2024-03-18T16:08:47.883502+0000","last_became_active":"2024-03-18T16:06:44.732909+0000","last_became_peered":"2024-03-18T16:06:44.732909+0000","last_unstale":"2024-03-18T16:08:47.883502+0000","last_undegraded":"2024-03-18T16:08:47.883502+0000","last_fullsized":"2024-03-18T16:08:47.883502+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:28:07.636692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.961752+0000","last_change":"2024-03-18T16:06:44.731050+0000","last_active":"2024-03-18T16:08:47.961752+0000","last_peered":"2024-03-18T16:08:47.961752+0000","last_clean":"2024-03-18T16:08:47.961752+0000","last_became_active":"2024-03-18T16:06:44.730634+0000","last_became_peered":"2024-03-18T16:06:44.730634+0000","last_unstale":"2024-03-18T16:08:47.961752+0000","last_undegraded":"2024-03-18T16:08:47.961752+0000","last_fullsized":"2024-03-18T16:08:47.961752+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:07:18.619130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T16:08:45.535599+0000","last_change":"2024-03-18T16:06:44.735190+0000","last_active":"2024-03-18T16:08:45.535599+0000","last_peered":"2024-03-18T16:08:45.535599+0000","last_clean":"2024-03-18T16:08:45.535599+0000","last_became_active":"2024-03-18T16:06:44.734312+0000","last_became_peered":"2024-03-18T16:06:44.734312+0000","last_unstale":"2024-03-18T16:08:45.535599+0000","last_undegraded":"2024-03-18T16:08:45.535599+0000","last_fullsized":"2024-03-18T16:08:45.535599+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:59:50.209515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T16:08:45.492622+0000","last_change":"2024-03-18T16:06:44.705385+0000","last_active":"2024-03-18T16:08:45.492622+0000","last_peered":"2024-03-18T16:08:45.492622+0000","last_clean":"2024-03-18T16:08:45.492622+0000","last_became_active":"2024-03-18T16:06:44.705011+0000","last_became_peered":"2024-03-18T16:06:44.705011+0000","last_unstale":"2024-03-18T16:08:45.492622+0000","last_undegraded":"2024-03-18T16:08:45.492622+0000","last_fullsized":"2024-03-18T16:08:45.492622+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:50:02.152851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.885230+0000","last_change":"2024-03-18T16:06:44.709621+0000","last_active":"2024-03-18T16:08:47.885230+0000","last_peered":"2024-03-18T16:08:47.885230+0000","last_clean":"2024-03-18T16:08:47.885230+0000","last_became_active":"2024-03-18T16:06:44.709275+0000","last_became_peered":"2024-03-18T16:06:44.709275+0000","last_unstale":"2024-03-18T16:08:47.885230+0000","last_undegraded":"2024-03-18T16:08:47.885230+0000","last_fullsized":"2024-03-18T16:08:47.885230+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:41:17.346661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T16:07:47.410785+0000","last_change":"2024-03-18T16:06:44.719827+0000","last_active":"2024-03-18T16:07:47.410785+0000","last_peered":"2024-03-18T16:07:47.410785+0000","last_clean":"2024-03-18T16:07:47.410785+0000","last_became_active":"2024-03-18T16:06:44.719432+0000","last_became_peered":"2024-03-18T16:06:44.719432+0000","last_unstale":"2024-03-18T16:07:47.410785+0000","last_undegraded":"2024-03-18T16:07:47.410785+0000","last_fullsized":"2024-03-18T16:07:47.410785+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:13:42.344211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":32,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-03-18T16:08:46.880762+0000","last_change":"2024-03-18T16:06:44.714526+0000","last_active":"2024-03-18T16:08:46.880762+0000","last_peered":"2024-03-18T16:08:46.880762+0000","last_clean":"2024-03-18T16:08:46.880762+0000","last_became_active":"2024-03-18T16:06:44.714184+0000","last_became_peered":"2024-03-18T16:06:44.714184+0000","last_unstale":"2024-03-18T16:08:46.880762+0000","last_undegraded":"2024-03-18T16:08:46.880762+0000","last_fullsized":"2024-03-18T16:08:46.880762+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:31:07.917143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T16:07:47.407837+0000","last_change":"2024-03-18T16:06:44.730905+0000","last_active":"2024-03-18T16:07:47.407837+0000","last_peered":"2024-03-18T16:07:47.407837+0000","last_clean":"2024-03-18T16:07:47.407837+0000","last_became_active":"2024-03-18T16:06:44.729002+0000","last_became_peered":"2024-03-18T16:06:44.729002+0000","last_unstale":"2024-03-18T16:07:47.407837+0000","last_undegraded":"2024-03-18T16:07:47.407837+0000","last_fullsized":"2024-03-18T16:07:47.407837+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:49:18.867571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T16:08:45.513862+0000","last_change":"2024-03-18T16:06:44.727608+0000","last_active":"2024-03-18T16:08:45.513862+0000","last_peered":"2024-03-18T16:08:45.513862+0000","last_clean":"2024-03-18T16:08:45.513862+0000","last_became_active":"2024-03-18T16:06:44.727096+0000","last_became_peered":"2024-03-18T16:06:44.727096+0000","last_unstale":"2024-03-18T16:08:45.513862+0000","last_undegraded":"2024-03-18T16:08:45.513862+0000","last_fullsized":"2024-03-18T16:08:45.513862+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:08:20.767443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T16:08:45.538302+0000","last_change":"2024-03-18T16:06:44.731823+0000","last_active":"2024-03-18T16:08:45.538302+0000","last_peered":"2024-03-18T16:08:45.538302+0000","last_clean":"2024-03-18T16:08:45.538302+0000","last_became_active":"2024-03-18T16:06:44.731429+0000","last_became_peered":"2024-03-18T16:06:44.731429+0000","last_unstale":"2024-03-18T16:08:45.538302+0000","last_undegraded":"2024-03-18T16:08:45.538302+0000","last_fullsized":"2024-03-18T16:08:45.538302+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:36:28.752466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-18T16:08:45.495190+0000","last_change":"2024-03-18T16:06:44.714354+0000","last_active":"2024-03-18T16:08:45.495190+0000","last_peered":"2024-03-18T16:08:45.495190+0000","last_clean":"2024-03-18T16:08:45.495190+0000","last_became_active":"2024-03-18T16:06:44.713969+0000","last_became_peered":"2024-03-18T16:06:44.713969+0000","last_unstale":"2024-03-18T16:08:45.495190+0000","last_undegraded":"2024-03-18T16:08:45.495190+0000","last_fullsized":"2024-03-18T16:08:45.495190+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:18:49.502178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":438,"ondisk_log_size":438,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":68695,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":925,"num_read_kb":540,"num_write":249,"num_write_kb":143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":557056,"data_stored":205941,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":216,"ondisk_log_size":216,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383720,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1032192,"data_stored":501441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645356,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37064,"kb_used_data":1720,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708088,"statfs":{"total":95995035648,"available":95957082112,"internally_reserved":0,"allocated":1761280,"data_stored":1172860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808881,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1056768,"data_stored":501371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070521,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36364,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708788,"statfs":{"total":95995035648,"available":95957798912,"internally_reserved":0,"allocated":1052672,"data_stored":501749,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332157,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28628,"kb_used_data":1612,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716524,"statfs":{"total":95995035648,"available":95965720576,"internally_reserved":0,"allocated":1650688,"data_stored":1098387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495682,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32184,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712968,"statfs":{"total":95995035648,"available":95962079232,"internally_reserved":0,"allocated":1032192,"data_stored":496875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574942,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36412,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708740,"statfs":{"total":95995035648,"available":95957749760,"internally_reserved":0,"allocated":1097728,"data_stored":559397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738469,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1073152,"data_stored":507739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411419,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36460,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708692,"statfs":{"total":95995035648,"available":95957700608,"internally_reserved":0,"allocated":1142784,"data_stored":579096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182484,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28040,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717112,"statfs":{"total":95995035648,"available":95966322688,"internally_reserved":0,"allocated":1044480,"data_stored":496583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920847,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36372,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708780,"statfs":{"total":95995035648,"available":95957790720,"internally_reserved":0,"allocated":1056768,"data_stored":510357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757322,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36956,"kb_used_data":1620,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708196,"statfs":{"total":95995035648,"available":95957192704,"internally_reserved":0,"allocated":1658880,"data_stored":1093969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":63417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":64177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":63477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T16:08:52.231 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph pg dump --format=json 2024-03-18T16:08:52.231 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:08:51 smithi196 ceph-mon[36307]: from='client.25040 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T16:08:52.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:08:51 smithi194 ceph-mon[36200]: from='client.25040 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T16:08:52.727 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T16:08:53.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:08:52 smithi196 ceph-mon[36307]: pgmap v386: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 639 B/s wr, 10 op/s; 11 B/s, 1 keys/s, 0 objects/s recovering 2024-03-18T16:08:53.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:08:52 smithi194 ceph-mon[36200]: pgmap v386: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 639 B/s wr, 10 op/s; 11 B/s, 1 keys/s, 0 objects/s recovering 2024-03-18T16:08:53.269 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:08:52 smithi039 ceph-mon[31721]: pgmap v386: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 639 B/s wr, 10 op/s; 11 B/s, 1 keys/s, 0 objects/s recovering 2024-03-18T16:08:55.085 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:08:54 smithi039 ceph-mon[31721]: pgmap v387: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 639 B/s wr, 10 op/s; 11 B/s, 1 keys/s, 0 objects/s recovering 2024-03-18T16:08:55.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:55.110+0000 7f1521c00700 1 -- 172.21.15.39:0/1208397030 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f151c108320 msgr2=0x7f151c108700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:55.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:55.110+0000 7f1521c00700 1 --2- 172.21.15.39:0/1208397030 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f151c108320 0x7f151c108700 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f150c009990 tx=0x7f150c009ca0 comp rx=0 tx=0).stop 2024-03-18T16:08:55.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:55.110+0000 7f1521c00700 1 -- 172.21.15.39:0/1208397030 shutdown_connections 2024-03-18T16:08:55.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:55.110+0000 7f1521c00700 1 --2- 172.21.15.39:0/1208397030 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f151c06a920 0x7f151c10f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:55.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:55.110+0000 7f1521c00700 1 --2- 172.21.15.39:0/1208397030 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f151c108320 0x7f151c108700 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:55.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:55.110+0000 7f1521c00700 1 --2- 172.21.15.39:0/1208397030 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f151c069f80 0x7f151c06a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:55.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:55.110+0000 7f1521c00700 1 -- 172.21.15.39:0/1208397030 >> 172.21.15.39:0/1208397030 conn(0x7f151c078100 msgr2=0x7f151c078510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:08:55.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:55.110+0000 7f1521c00700 1 -- 172.21.15.39:0/1208397030 shutdown_connections 2024-03-18T16:08:55.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:55.110+0000 7f1521c00700 1 -- 172.21.15.39:0/1208397030 wait complete. 2024-03-18T16:08:55.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:55.112+0000 7f1521c00700 1 Processor -- start 2024-03-18T16:08:55.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:55.112+0000 7f1521c00700 1 -- start start 2024-03-18T16:08:55.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:55.112+0000 7f1521c00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f151c069f80 0x7f151c1a08f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:55.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:55.112+0000 7f1521c00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f151c06a920 0x7f151c1a0e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:55.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:55.112+0000 7f1521c00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f151c108320 0x7f151c19aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:55.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:55.112+0000 7f1521c00700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f151c1125b0 con 0x7f151c069f80 2024-03-18T16:08:55.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:55.112+0000 7f1521c00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f151c112430 con 0x7f151c06a920 2024-03-18T16:08:55.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:55.112+0000 7f1521c00700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f151c112730 con 0x7f151c108320 2024-03-18T16:08:55.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:55.113+0000 7f151a000700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f151c06a920 0x7f151c1a0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:55.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:55.113+0000 7f151aa00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f151c069f80 0x7f151c1a08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:55.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:55.113+0000 7f151a000700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f151c06a920 0x7f151c1a0e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:44802/0 (socket says 172.21.15.39:44802) 2024-03-18T16:08:55.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:55.113+0000 7f151a000700 1 -- 172.21.15.39:0/3514610424 learned_addr learned my addr 172.21.15.39:0/3514610424 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:08:55.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:55.113+0000 7f151b400700 1 --2- 172.21.15.39:0/3514610424 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f151c108320 0x7f151c19aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:55.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:55.113+0000 7f151a000700 1 -- 172.21.15.39:0/3514610424 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f151c108320 msgr2=0x7f151c19aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:55.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:55.113+0000 7f151a000700 1 --2- 172.21.15.39:0/3514610424 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f151c108320 0x7f151c19aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:55.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:55.113+0000 7f151a000700 1 -- 172.21.15.39:0/3514610424 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f151c069f80 msgr2=0x7f151c1a08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:55.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:55.113+0000 7f151a000700 1 --2- 172.21.15.39:0/3514610424 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f151c069f80 0x7f151c1a08f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:55.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:55.113+0000 7f151a000700 1 -- 172.21.15.39:0/3514610424 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f151c19b210 con 0x7f151c06a920 2024-03-18T16:08:55.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:55.113+0000 7f151b400700 1 --2- 172.21.15.39:0/3514610424 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f151c108320 0x7f151c19aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:08:55.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:55.114+0000 7f151a000700 1 --2- 172.21.15.39:0/3514610424 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f151c06a920 0x7f151c1a0e30 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f150800e9a0 tx=0x7f150800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:55.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:55.114+0000 7f1513400700 1 -- 172.21.15.39:0/3514610424 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f150800ca10 con 0x7f151c06a920 2024-03-18T16:08:55.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:55.114+0000 7f1513400700 1 -- 172.21.15.39:0/3514610424 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f150800cbb0 con 0x7f151c06a920 2024-03-18T16:08:55.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:55.114+0000 7f1513400700 1 -- 172.21.15.39:0/3514610424 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1508018680 con 0x7f151c06a920 2024-03-18T16:08:55.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:55.114+0000 7f1521c00700 1 -- 172.21.15.39:0/3514610424 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f151c19b500 con 0x7f151c06a920 2024-03-18T16:08:55.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:55.114+0000 7f1521c00700 1 -- 172.21.15.39:0/3514610424 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f151c1a77c0 con 0x7f151c06a920 2024-03-18T16:08:55.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:55.115+0000 7f1521c00700 1 -- 172.21.15.39:0/3514610424 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f151c068f90 con 0x7f151c06a920 2024-03-18T16:08:55.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:55.119+0000 7f1513400700 1 -- 172.21.15.39:0/3514610424 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f1508010430 con 0x7f151c06a920 2024-03-18T16:08:55.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:55.120+0000 7f1513400700 1 --2- 172.21.15.39:0/3514610424 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f15000a2910 0x7f15000a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:55.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:55.120+0000 7f1513400700 1 -- 172.21.15.39:0/3514610424 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f15080cefe0 con 0x7f151c06a920 2024-03-18T16:08:55.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:55.121+0000 7f151aa00700 1 --2- 172.21.15.39:0/3514610424 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f15000a2910 0x7f15000a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:55.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:55.121+0000 7f151aa00700 1 --2- 172.21.15.39:0/3514610424 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f15000a2910 0x7f15000a4dd0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f151c19bbf0 tx=0x7f150c00b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:55.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:55.124+0000 7f1513400700 1 -- 172.21.15.39:0/3514610424 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f150809bcb0 con 0x7f151c06a920 2024-03-18T16:08:55.227 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:08:54 smithi196 ceph-mon[36307]: pgmap v387: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 639 B/s wr, 10 op/s; 11 B/s, 1 keys/s, 0 objects/s recovering 2024-03-18T16:08:55.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:08:54 smithi194 ceph-mon[36200]: pgmap v387: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 639 B/s wr, 10 op/s; 11 B/s, 1 keys/s, 0 objects/s recovering 2024-03-18T16:08:55.383 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:55.380+0000 7f1521c00700 1 -- 172.21.15.39:0/3514610424 --> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f151c100380 con 0x7f15000a2910 2024-03-18T16:08:55.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:55.396+0000 7f1513400700 1 -- 172.21.15.39:0/3514610424 <== mgr.14152 v2:172.21.15.39:6800/2749214286 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340611 (secure 0 0 0) 0x7f151c100380 con 0x7f15000a2910 2024-03-18T16:08:55.407 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:08:55.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:55.405+0000 7f1521c00700 1 -- 172.21.15.39:0/3514610424 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f15000a2910 msgr2=0x7f15000a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:55.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:55.405+0000 7f1521c00700 1 --2- 172.21.15.39:0/3514610424 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f15000a2910 0x7f15000a4dd0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f151c19bbf0 tx=0x7f150c00b260 comp rx=0 tx=0).stop 2024-03-18T16:08:55.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:55.405+0000 7f1521c00700 1 -- 172.21.15.39:0/3514610424 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f151c06a920 msgr2=0x7f151c1a0e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:55.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:55.405+0000 7f1521c00700 1 --2- 172.21.15.39:0/3514610424 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f151c06a920 0x7f151c1a0e30 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f150800e9a0 tx=0x7f150800ecb0 comp rx=0 tx=0).stop 2024-03-18T16:08:55.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:55.406+0000 7f1521c00700 1 -- 172.21.15.39:0/3514610424 shutdown_connections 2024-03-18T16:08:55.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:55.406+0000 7f1521c00700 1 --2- 172.21.15.39:0/3514610424 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f15000a2910 0x7f15000a4dd0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:55.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:55.406+0000 7f1521c00700 1 --2- 172.21.15.39:0/3514610424 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f151c108320 0x7f151c19aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:55.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:55.406+0000 7f1521c00700 1 --2- 172.21.15.39:0/3514610424 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f151c069f80 0x7f151c1a08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:55.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:55.406+0000 7f1521c00700 1 --2- 172.21.15.39:0/3514610424 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f151c06a920 0x7f151c1a0e30 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:55.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:55.406+0000 7f1521c00700 1 -- 172.21.15.39:0/3514610424 >> 172.21.15.39:0/3514610424 conn(0x7f151c078100 msgr2=0x7f151c0ffc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:08:55.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:55.406+0000 7f1521c00700 1 -- 172.21.15.39:0/3514610424 shutdown_connections 2024-03-18T16:08:55.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:55.406+0000 7f1521c00700 1 -- 172.21.15.39:0/3514610424 wait complete. 2024-03-18T16:08:55.411 INFO:teuthology.orchestra.run.smithi039.stderr:dumped all 2024-03-18T16:08:56.325 INFO:teuthology.orchestra.run.smithi039.stdout:{"pg_ready":true,"pg_map":{"version":387,"stamp":"2024-03-18T16:08:53.542625+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":659297,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1075,"num_read_kb":658,"num_write":432,"num_write_kb":3083,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":1180943,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":855,"ondisk_log_size":855,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":401292,"kb_used_data":14612,"kb_used_omap":18,"kb_used_meta":386477,"kb_avail":1124540532,"statfs":{"total":1151940427776,"available":1151529504768,"internally_reserved":0,"allocated":14962688,"data_stored":8188985,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":395752763},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":390,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":76,"num_read_kb":36,"num_write":7,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":90,"num_keys_recovered":10,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003271"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.882222+0000","last_change":"2024-03-18T16:06:44.736511+0000","last_active":"2024-03-18T16:08:47.882222+0000","last_peered":"2024-03-18T16:08:47.882222+0000","last_clean":"2024-03-18T16:08:47.882222+0000","last_became_active":"2024-03-18T16:06:44.736182+0000","last_became_peered":"2024-03-18T16:06:44.736182+0000","last_unstale":"2024-03-18T16:08:47.882222+0000","last_undegraded":"2024-03-18T16:08:47.882222+0000","last_fullsized":"2024-03-18T16:08:47.882222+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:08:57.060268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.961793+0000","last_change":"2024-03-18T16:06:44.733658+0000","last_active":"2024-03-18T16:08:47.961793+0000","last_peered":"2024-03-18T16:08:47.961793+0000","last_clean":"2024-03-18T16:08:47.961793+0000","last_became_active":"2024-03-18T16:06:44.733279+0000","last_became_peered":"2024-03-18T16:06:44.733279+0000","last_unstale":"2024-03-18T16:08:47.961793+0000","last_undegraded":"2024-03-18T16:08:47.961793+0000","last_fullsized":"2024-03-18T16:08:47.961793+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:59:46.783025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.961451+0000","last_change":"2024-03-18T16:06:44.723949+0000","last_active":"2024-03-18T16:08:47.961451+0000","last_peered":"2024-03-18T16:08:47.961451+0000","last_clean":"2024-03-18T16:08:47.961451+0000","last_became_active":"2024-03-18T16:06:44.723614+0000","last_became_peered":"2024-03-18T16:06:44.723614+0000","last_unstale":"2024-03-18T16:08:47.961451+0000","last_undegraded":"2024-03-18T16:08:47.961451+0000","last_fullsized":"2024-03-18T16:08:47.961451+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:03:17.876097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883864+0000","last_change":"2024-03-18T16:06:44.711058+0000","last_active":"2024-03-18T16:08:47.883864+0000","last_peered":"2024-03-18T16:08:47.883864+0000","last_clean":"2024-03-18T16:08:47.883864+0000","last_became_active":"2024-03-18T16:06:44.708873+0000","last_became_peered":"2024-03-18T16:06:44.708873+0000","last_unstale":"2024-03-18T16:08:47.883864+0000","last_undegraded":"2024-03-18T16:08:47.883864+0000","last_fullsized":"2024-03-18T16:08:47.883864+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:58:08.182651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.884673+0000","last_change":"2024-03-18T16:06:44.721480+0000","last_active":"2024-03-18T16:08:47.884673+0000","last_peered":"2024-03-18T16:08:47.884673+0000","last_clean":"2024-03-18T16:08:47.884673+0000","last_became_active":"2024-03-18T16:06:44.721101+0000","last_became_peered":"2024-03-18T16:06:44.721101+0000","last_unstale":"2024-03-18T16:08:47.884673+0000","last_undegraded":"2024-03-18T16:08:47.884673+0000","last_fullsized":"2024-03-18T16:08:47.884673+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:19:43.341221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.879808+0000","last_change":"2024-03-18T16:06:44.716443+0000","last_active":"2024-03-18T16:08:47.879808+0000","last_peered":"2024-03-18T16:08:47.879808+0000","last_clean":"2024-03-18T16:08:47.879808+0000","last_became_active":"2024-03-18T16:06:44.715969+0000","last_became_peered":"2024-03-18T16:06:44.715969+0000","last_unstale":"2024-03-18T16:08:47.879808+0000","last_undegraded":"2024-03-18T16:08:47.879808+0000","last_fullsized":"2024-03-18T16:08:47.879808+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:57:14.714415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.960327+0000","last_change":"2024-03-18T16:06:44.711439+0000","last_active":"2024-03-18T16:08:47.960327+0000","last_peered":"2024-03-18T16:08:47.960327+0000","last_clean":"2024-03-18T16:08:47.960327+0000","last_became_active":"2024-03-18T16:06:44.711032+0000","last_became_peered":"2024-03-18T16:06:44.711032+0000","last_unstale":"2024-03-18T16:08:47.960327+0000","last_undegraded":"2024-03-18T16:08:47.960327+0000","last_fullsized":"2024-03-18T16:08:47.960327+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:19:56.516867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.881902+0000","last_change":"2024-03-18T16:07:47.358866+0000","last_active":"2024-03-18T16:08:47.881902+0000","last_peered":"2024-03-18T16:08:47.881902+0000","last_clean":"2024-03-18T16:08:47.881902+0000","last_became_active":"2024-03-18T16:07:47.357743+0000","last_became_peered":"2024-03-18T16:07:47.357743+0000","last_unstale":"2024-03-18T16:08:47.881902+0000","last_undegraded":"2024-03-18T16:08:47.881902+0000","last_fullsized":"2024-03-18T16:08:47.881902+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:13:12.361025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,10],"acting":[9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.960719+0000","last_change":"2024-03-18T16:06:44.710978+0000","last_active":"2024-03-18T16:08:47.960719+0000","last_peered":"2024-03-18T16:08:47.960719+0000","last_clean":"2024-03-18T16:08:47.960719+0000","last_became_active":"2024-03-18T16:06:44.710520+0000","last_became_peered":"2024-03-18T16:06:44.710520+0000","last_unstale":"2024-03-18T16:08:47.960719+0000","last_undegraded":"2024-03-18T16:08:47.960719+0000","last_fullsized":"2024-03-18T16:08:47.960719+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:54:53.234025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.880362+0000","last_change":"2024-03-18T16:06:44.716231+0000","last_active":"2024-03-18T16:08:47.880362+0000","last_peered":"2024-03-18T16:08:47.880362+0000","last_clean":"2024-03-18T16:08:47.880362+0000","last_became_active":"2024-03-18T16:06:44.714900+0000","last_became_peered":"2024-03-18T16:06:44.714900+0000","last_unstale":"2024-03-18T16:08:47.880362+0000","last_undegraded":"2024-03-18T16:08:47.880362+0000","last_fullsized":"2024-03-18T16:08:47.880362+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:38:54.964729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.961174+0000","last_change":"2024-03-18T16:06:44.710942+0000","last_active":"2024-03-18T16:08:47.961174+0000","last_peered":"2024-03-18T16:08:47.961174+0000","last_clean":"2024-03-18T16:08:47.961174+0000","last_became_active":"2024-03-18T16:06:44.710588+0000","last_became_peered":"2024-03-18T16:06:44.710588+0000","last_unstale":"2024-03-18T16:08:47.961174+0000","last_undegraded":"2024-03-18T16:08:47.961174+0000","last_fullsized":"2024-03-18T16:08:47.961174+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:43:19.320034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883785+0000","last_change":"2024-03-18T16:06:44.714763+0000","last_active":"2024-03-18T16:08:47.883785+0000","last_peered":"2024-03-18T16:08:47.883785+0000","last_clean":"2024-03-18T16:08:47.883785+0000","last_became_active":"2024-03-18T16:06:44.714031+0000","last_became_peered":"2024-03-18T16:06:44.714031+0000","last_unstale":"2024-03-18T16:08:47.883785+0000","last_undegraded":"2024-03-18T16:08:47.883785+0000","last_fullsized":"2024-03-18T16:08:47.883785+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:19:25.917093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.962580+0000","last_change":"2024-03-18T16:06:44.710062+0000","last_active":"2024-03-18T16:08:47.962580+0000","last_peered":"2024-03-18T16:08:47.962580+0000","last_clean":"2024-03-18T16:08:47.962580+0000","last_became_active":"2024-03-18T16:06:44.709281+0000","last_became_peered":"2024-03-18T16:06:44.709281+0000","last_unstale":"2024-03-18T16:08:47.962580+0000","last_undegraded":"2024-03-18T16:08:47.962580+0000","last_fullsized":"2024-03-18T16:08:47.962580+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:51:30.100947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.960744+0000","last_change":"2024-03-18T16:06:44.734178+0000","last_active":"2024-03-18T16:08:47.960744+0000","last_peered":"2024-03-18T16:08:47.960744+0000","last_clean":"2024-03-18T16:08:47.960744+0000","last_became_active":"2024-03-18T16:06:44.732538+0000","last_became_peered":"2024-03-18T16:06:44.732538+0000","last_unstale":"2024-03-18T16:08:47.960744+0000","last_undegraded":"2024-03-18T16:08:47.960744+0000","last_fullsized":"2024-03-18T16:08:47.960744+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:14:26.904402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.960737+0000","last_change":"2024-03-18T16:07:47.412629+0000","last_active":"2024-03-18T16:08:47.960737+0000","last_peered":"2024-03-18T16:08:47.960737+0000","last_clean":"2024-03-18T16:08:47.960737+0000","last_became_active":"2024-03-18T16:07:47.411714+0000","last_became_peered":"2024-03-18T16:07:47.411714+0000","last_unstale":"2024-03-18T16:08:47.960737+0000","last_undegraded":"2024-03-18T16:08:47.960737+0000","last_fullsized":"2024-03-18T16:08:47.960737+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:25:50.843323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.959845+0000","last_change":"2024-03-18T16:06:44.728165+0000","last_active":"2024-03-18T16:08:47.959845+0000","last_peered":"2024-03-18T16:08:47.959845+0000","last_clean":"2024-03-18T16:08:47.959845+0000","last_became_active":"2024-03-18T16:06:44.727842+0000","last_became_peered":"2024-03-18T16:06:44.727842+0000","last_unstale":"2024-03-18T16:08:47.959845+0000","last_undegraded":"2024-03-18T16:08:47.959845+0000","last_fullsized":"2024-03-18T16:08:47.959845+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:55:04.464916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.881915+0000","last_change":"2024-03-18T16:06:44.730893+0000","last_active":"2024-03-18T16:08:47.881915+0000","last_peered":"2024-03-18T16:08:47.881915+0000","last_clean":"2024-03-18T16:08:47.881915+0000","last_became_active":"2024-03-18T16:06:44.730544+0000","last_became_peered":"2024-03-18T16:06:44.730544+0000","last_unstale":"2024-03-18T16:08:47.881915+0000","last_undegraded":"2024-03-18T16:08:47.881915+0000","last_fullsized":"2024-03-18T16:08:47.881915+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:16:37.918096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.884269+0000","last_change":"2024-03-18T16:06:44.706036+0000","last_active":"2024-03-18T16:08:47.884269+0000","last_peered":"2024-03-18T16:08:47.884269+0000","last_clean":"2024-03-18T16:08:47.884269+0000","last_became_active":"2024-03-18T16:06:44.705281+0000","last_became_peered":"2024-03-18T16:06:44.705281+0000","last_unstale":"2024-03-18T16:08:47.884269+0000","last_undegraded":"2024-03-18T16:08:47.884269+0000","last_fullsized":"2024-03-18T16:08:47.884269+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:52:20.202110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.884459+0000","last_change":"2024-03-18T16:06:44.727543+0000","last_active":"2024-03-18T16:08:47.884459+0000","last_peered":"2024-03-18T16:08:47.884459+0000","last_clean":"2024-03-18T16:08:47.884459+0000","last_became_active":"2024-03-18T16:06:44.727135+0000","last_became_peered":"2024-03-18T16:06:44.727135+0000","last_unstale":"2024-03-18T16:08:47.884459+0000","last_undegraded":"2024-03-18T16:08:47.884459+0000","last_fullsized":"2024-03-18T16:08:47.884459+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:52:42.742279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883506+0000","last_change":"2024-03-18T16:06:49.028004+0000","last_active":"2024-03-18T16:08:47.883506+0000","last_peered":"2024-03-18T16:08:47.883506+0000","last_clean":"2024-03-18T16:08:47.883506+0000","last_became_active":"2024-03-18T16:06:49.027640+0000","last_became_peered":"2024-03-18T16:06:49.027640+0000","last_unstale":"2024-03-18T16:08:47.883506+0000","last_undegraded":"2024-03-18T16:08:47.883506+0000","last_fullsized":"2024-03-18T16:08:47.883506+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:23:07.926349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'13","reported_seq":137,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:51.185265+0000","last_change":"2024-03-18T16:06:43.709540+0000","last_active":"2024-03-18T16:08:51.185265+0000","last_peered":"2024-03-18T16:08:51.185265+0000","last_clean":"2024-03-18T16:08:51.185265+0000","last_became_active":"2024-03-18T16:06:43.709170+0000","last_became_peered":"2024-03-18T16:06:43.709170+0000","last_unstale":"2024-03-18T16:08:51.185265+0000","last_undegraded":"2024-03-18T16:08:51.185265+0000","last_fullsized":"2024-03-18T16:08:51.185265+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:12:53.934079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":92,"num_read_kb":92,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.886428+0000","last_change":"2024-03-18T16:06:44.730192+0000","last_active":"2024-03-18T16:08:47.886428+0000","last_peered":"2024-03-18T16:08:47.886428+0000","last_clean":"2024-03-18T16:08:47.886428+0000","last_became_active":"2024-03-18T16:06:44.728071+0000","last_became_peered":"2024-03-18T16:06:44.728071+0000","last_unstale":"2024-03-18T16:08:47.886428+0000","last_undegraded":"2024-03-18T16:08:47.886428+0000","last_fullsized":"2024-03-18T16:08:47.886428+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:14:59.992958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883279+0000","last_change":"2024-03-18T16:06:49.023963+0000","last_active":"2024-03-18T16:08:47.883279+0000","last_peered":"2024-03-18T16:08:47.883279+0000","last_clean":"2024-03-18T16:08:47.883279+0000","last_became_active":"2024-03-18T16:06:49.023568+0000","last_became_peered":"2024-03-18T16:06:49.023568+0000","last_unstale":"2024-03-18T16:08:47.883279+0000","last_undegraded":"2024-03-18T16:08:47.883279+0000","last_fullsized":"2024-03-18T16:08:47.883279+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:15:19.730842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.886440+0000","last_change":"2024-03-18T16:06:43.722167+0000","last_active":"2024-03-18T16:08:47.886440+0000","last_peered":"2024-03-18T16:08:47.886440+0000","last_clean":"2024-03-18T16:08:47.886440+0000","last_became_active":"2024-03-18T16:06:43.721782+0000","last_became_peered":"2024-03-18T16:06:43.721782+0000","last_unstale":"2024-03-18T16:08:47.886440+0000","last_undegraded":"2024-03-18T16:08:47.886440+0000","last_fullsized":"2024-03-18T16:08:47.886440+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:08:54.400637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.962047+0000","last_change":"2024-03-18T16:06:44.729666+0000","last_active":"2024-03-18T16:08:47.962047+0000","last_peered":"2024-03-18T16:08:47.962047+0000","last_clean":"2024-03-18T16:08:47.962047+0000","last_became_active":"2024-03-18T16:06:44.728842+0000","last_became_peered":"2024-03-18T16:06:44.728842+0000","last_unstale":"2024-03-18T16:08:47.962047+0000","last_undegraded":"2024-03-18T16:08:47.962047+0000","last_fullsized":"2024-03-18T16:08:47.962047+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:01:42.438000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883419+0000","last_change":"2024-03-18T16:06:49.031169+0000","last_active":"2024-03-18T16:08:47.883419+0000","last_peered":"2024-03-18T16:08:47.883419+0000","last_clean":"2024-03-18T16:08:47.883419+0000","last_became_active":"2024-03-18T16:06:49.030801+0000","last_became_peered":"2024-03-18T16:06:49.030801+0000","last_unstale":"2024-03-18T16:08:47.883419+0000","last_undegraded":"2024-03-18T16:08:47.883419+0000","last_fullsized":"2024-03-18T16:08:47.883419+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:10:54.880882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'12","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.885369+0000","last_change":"2024-03-18T16:06:43.707111+0000","last_active":"2024-03-18T16:08:47.885369+0000","last_peered":"2024-03-18T16:08:47.885369+0000","last_clean":"2024-03-18T16:08:47.885369+0000","last_became_active":"2024-03-18T16:06:43.706671+0000","last_became_peered":"2024-03-18T16:06:43.706671+0000","last_unstale":"2024-03-18T16:08:47.885369+0000","last_undegraded":"2024-03-18T16:08:47.885369+0000","last_fullsized":"2024-03-18T16:08:47.885369+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:18:34.295641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.960332+0000","last_change":"2024-03-18T16:06:44.722100+0000","last_active":"2024-03-18T16:08:47.960332+0000","last_peered":"2024-03-18T16:08:47.960332+0000","last_clean":"2024-03-18T16:08:47.960332+0000","last_became_active":"2024-03-18T16:06:44.721673+0000","last_became_peered":"2024-03-18T16:06:44.721673+0000","last_unstale":"2024-03-18T16:08:47.960332+0000","last_undegraded":"2024-03-18T16:08:47.960332+0000","last_fullsized":"2024-03-18T16:08:47.960332+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:41:22.624778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883330+0000","last_change":"2024-03-18T16:06:49.027966+0000","last_active":"2024-03-18T16:08:47.883330+0000","last_peered":"2024-03-18T16:08:47.883330+0000","last_clean":"2024-03-18T16:08:47.883330+0000","last_became_active":"2024-03-18T16:06:49.027567+0000","last_became_peered":"2024-03-18T16:06:49.027567+0000","last_unstale":"2024-03-18T16:08:47.883330+0000","last_undegraded":"2024-03-18T16:08:47.883330+0000","last_fullsized":"2024-03-18T16:08:47.883330+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:54:28.543019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.885045+0000","last_change":"2024-03-18T16:06:43.704023+0000","last_active":"2024-03-18T16:08:47.885045+0000","last_peered":"2024-03-18T16:08:47.885045+0000","last_clean":"2024-03-18T16:08:47.885045+0000","last_became_active":"2024-03-18T16:06:43.703690+0000","last_became_peered":"2024-03-18T16:06:43.703690+0000","last_unstale":"2024-03-18T16:08:47.885045+0000","last_undegraded":"2024-03-18T16:08:47.885045+0000","last_fullsized":"2024-03-18T16:08:47.885045+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:20:02.344346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":49,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.886088+0000","last_change":"2024-03-18T16:07:47.354361+0000","last_active":"2024-03-18T16:08:47.886088+0000","last_peered":"2024-03-18T16:08:47.886088+0000","last_clean":"2024-03-18T16:08:47.886088+0000","last_became_active":"2024-03-18T16:07:47.353964+0000","last_became_peered":"2024-03-18T16:07:47.353964+0000","last_unstale":"2024-03-18T16:08:47.886088+0000","last_undegraded":"2024-03-18T16:08:47.886088+0000","last_fullsized":"2024-03-18T16:08:47.886088+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:56:11.405218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.884296+0000","last_change":"2024-03-18T16:06:49.012310+0000","last_active":"2024-03-18T16:08:47.884296+0000","last_peered":"2024-03-18T16:08:47.884296+0000","last_clean":"2024-03-18T16:08:47.884296+0000","last_became_active":"2024-03-18T16:06:49.011844+0000","last_became_peered":"2024-03-18T16:06:49.011844+0000","last_unstale":"2024-03-18T16:08:47.884296+0000","last_undegraded":"2024-03-18T16:08:47.884296+0000","last_fullsized":"2024-03-18T16:08:47.884296+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:16:42.652205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"91'7","reported_seq":133,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:50.076706+0000","last_change":"2024-03-18T16:06:43.705653+0000","last_active":"2024-03-18T16:08:50.076706+0000","last_peered":"2024-03-18T16:08:50.076706+0000","last_clean":"2024-03-18T16:08:50.076706+0000","last_became_active":"2024-03-18T16:06:43.705320+0000","last_became_peered":"2024-03-18T16:06:43.705320+0000","last_unstale":"2024-03-18T16:08:50.076706+0000","last_undegraded":"2024-03-18T16:08:50.076706+0000","last_fullsized":"2024-03-18T16:08:50.076706+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:10:10.827338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.884799+0000","last_change":"2024-03-18T16:06:44.720045+0000","last_active":"2024-03-18T16:08:47.884799+0000","last_peered":"2024-03-18T16:08:47.884799+0000","last_clean":"2024-03-18T16:08:47.884799+0000","last_became_active":"2024-03-18T16:06:44.719669+0000","last_became_peered":"2024-03-18T16:06:44.719669+0000","last_unstale":"2024-03-18T16:08:47.884799+0000","last_undegraded":"2024-03-18T16:08:47.884799+0000","last_fullsized":"2024-03-18T16:08:47.884799+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:23:47.464057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.884057+0000","last_change":"2024-03-18T16:06:49.032576+0000","last_active":"2024-03-18T16:08:47.884057+0000","last_peered":"2024-03-18T16:08:47.884057+0000","last_clean":"2024-03-18T16:08:47.884057+0000","last_became_active":"2024-03-18T16:06:49.032126+0000","last_became_peered":"2024-03-18T16:06:49.032126+0000","last_unstale":"2024-03-18T16:08:47.884057+0000","last_undegraded":"2024-03-18T16:08:47.884057+0000","last_fullsized":"2024-03-18T16:08:47.884057+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:41:54.822070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.879762+0000","last_change":"2024-03-18T16:06:43.706364+0000","last_active":"2024-03-18T16:08:47.879762+0000","last_peered":"2024-03-18T16:08:47.879762+0000","last_clean":"2024-03-18T16:08:47.879762+0000","last_became_active":"2024-03-18T16:06:43.705984+0000","last_became_peered":"2024-03-18T16:06:43.705984+0000","last_unstale":"2024-03-18T16:08:47.879762+0000","last_undegraded":"2024-03-18T16:08:47.879762+0000","last_fullsized":"2024-03-18T16:08:47.879762+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:46:06.503000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.885963+0000","last_change":"2024-03-18T16:06:44.721019+0000","last_active":"2024-03-18T16:08:47.885963+0000","last_peered":"2024-03-18T16:08:47.885963+0000","last_clean":"2024-03-18T16:08:47.885963+0000","last_became_active":"2024-03-18T16:06:44.720665+0000","last_became_peered":"2024-03-18T16:06:44.720665+0000","last_unstale":"2024-03-18T16:08:47.885963+0000","last_undegraded":"2024-03-18T16:08:47.885963+0000","last_fullsized":"2024-03-18T16:08:47.885963+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:58:34.180653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.881812+0000","last_change":"2024-03-18T16:06:49.022919+0000","last_active":"2024-03-18T16:08:47.881812+0000","last_peered":"2024-03-18T16:08:47.881812+0000","last_clean":"2024-03-18T16:08:47.881812+0000","last_became_active":"2024-03-18T16:06:49.022514+0000","last_became_peered":"2024-03-18T16:06:49.022514+0000","last_unstale":"2024-03-18T16:08:47.881812+0000","last_undegraded":"2024-03-18T16:08:47.881812+0000","last_fullsized":"2024-03-18T16:08:47.881812+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:46:39.510181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.961358+0000","last_change":"2024-03-18T16:06:43.713025+0000","last_active":"2024-03-18T16:08:47.961358+0000","last_peered":"2024-03-18T16:08:47.961358+0000","last_clean":"2024-03-18T16:08:47.961358+0000","last_became_active":"2024-03-18T16:06:43.711469+0000","last_became_peered":"2024-03-18T16:06:43.711469+0000","last_unstale":"2024-03-18T16:08:47.961358+0000","last_undegraded":"2024-03-18T16:08:47.961358+0000","last_fullsized":"2024-03-18T16:08:47.961358+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:30:41.193229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883113+0000","last_change":"2024-03-18T16:06:44.723953+0000","last_active":"2024-03-18T16:08:47.883113+0000","last_peered":"2024-03-18T16:08:47.883113+0000","last_clean":"2024-03-18T16:08:47.883113+0000","last_became_active":"2024-03-18T16:06:44.723266+0000","last_became_peered":"2024-03-18T16:06:44.723266+0000","last_unstale":"2024-03-18T16:08:47.883113+0000","last_undegraded":"2024-03-18T16:08:47.883113+0000","last_fullsized":"2024-03-18T16:08:47.883113+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:41:10.067297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.961683+0000","last_change":"2024-03-18T16:06:49.039472+0000","last_active":"2024-03-18T16:08:47.961683+0000","last_peered":"2024-03-18T16:08:47.961683+0000","last_clean":"2024-03-18T16:08:47.961683+0000","last_became_active":"2024-03-18T16:06:49.038889+0000","last_became_peered":"2024-03-18T16:06:49.038889+0000","last_unstale":"2024-03-18T16:08:47.961683+0000","last_undegraded":"2024-03-18T16:08:47.961683+0000","last_fullsized":"2024-03-18T16:08:47.961683+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:03:05.345315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.885712+0000","last_change":"2024-03-18T16:06:43.705873+0000","last_active":"2024-03-18T16:08:47.885712+0000","last_peered":"2024-03-18T16:08:47.885712+0000","last_clean":"2024-03-18T16:08:47.885712+0000","last_became_active":"2024-03-18T16:06:43.705548+0000","last_became_peered":"2024-03-18T16:06:43.705548+0000","last_unstale":"2024-03-18T16:08:47.885712+0000","last_undegraded":"2024-03-18T16:08:47.885712+0000","last_fullsized":"2024-03-18T16:08:47.885712+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:52:03.790888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.882633+0000","last_change":"2024-03-18T16:06:44.718526+0000","last_active":"2024-03-18T16:08:47.882633+0000","last_peered":"2024-03-18T16:08:47.882633+0000","last_clean":"2024-03-18T16:08:47.882633+0000","last_became_active":"2024-03-18T16:06:44.717266+0000","last_became_peered":"2024-03-18T16:06:44.717266+0000","last_unstale":"2024-03-18T16:08:47.882633+0000","last_undegraded":"2024-03-18T16:08:47.882633+0000","last_fullsized":"2024-03-18T16:08:47.882633+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:55:09.340354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.885032+0000","last_change":"2024-03-18T16:06:49.021971+0000","last_active":"2024-03-18T16:08:47.885032+0000","last_peered":"2024-03-18T16:08:47.885032+0000","last_clean":"2024-03-18T16:08:47.885032+0000","last_became_active":"2024-03-18T16:06:49.021577+0000","last_became_peered":"2024-03-18T16:06:49.021577+0000","last_unstale":"2024-03-18T16:08:47.885032+0000","last_undegraded":"2024-03-18T16:08:47.885032+0000","last_fullsized":"2024-03-18T16:08:47.885032+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:47:44.768559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"91'5","reported_seq":215,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:49.074685+0000","last_change":"2024-03-18T16:06:43.705636+0000","last_active":"2024-03-18T16:08:49.074685+0000","last_peered":"2024-03-18T16:08:49.074685+0000","last_clean":"2024-03-18T16:08:49.074685+0000","last_became_active":"2024-03-18T16:06:43.705268+0000","last_became_peered":"2024-03-18T16:06:43.705268+0000","last_unstale":"2024-03-18T16:08:49.074685+0000","last_undegraded":"2024-03-18T16:08:49.074685+0000","last_fullsized":"2024-03-18T16:08:49.074685+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:37:31.367130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":180,"num_read_kb":180,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.960527+0000","last_change":"2024-03-18T16:06:44.712237+0000","last_active":"2024-03-18T16:08:47.960527+0000","last_peered":"2024-03-18T16:08:47.960527+0000","last_clean":"2024-03-18T16:08:47.960527+0000","last_became_active":"2024-03-18T16:06:44.711677+0000","last_became_peered":"2024-03-18T16:06:44.711677+0000","last_unstale":"2024-03-18T16:08:47.960527+0000","last_undegraded":"2024-03-18T16:08:47.960527+0000","last_fullsized":"2024-03-18T16:08:47.960527+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:16:56.326962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.882148+0000","last_change":"2024-03-18T16:06:49.034578+0000","last_active":"2024-03-18T16:08:47.882148+0000","last_peered":"2024-03-18T16:08:47.882148+0000","last_clean":"2024-03-18T16:08:47.882148+0000","last_became_active":"2024-03-18T16:06:49.031099+0000","last_became_peered":"2024-03-18T16:06:49.031099+0000","last_unstale":"2024-03-18T16:08:47.882148+0000","last_undegraded":"2024-03-18T16:08:47.882148+0000","last_fullsized":"2024-03-18T16:08:47.882148+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:11:08.042550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.961164+0000","last_change":"2024-03-18T16:06:43.708151+0000","last_active":"2024-03-18T16:08:47.961164+0000","last_peered":"2024-03-18T16:08:47.961164+0000","last_clean":"2024-03-18T16:08:47.961164+0000","last_became_active":"2024-03-18T16:06:43.707810+0000","last_became_peered":"2024-03-18T16:06:43.707810+0000","last_unstale":"2024-03-18T16:08:47.961164+0000","last_undegraded":"2024-03-18T16:08:47.961164+0000","last_fullsized":"2024-03-18T16:08:47.961164+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:30:35.745772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.962153+0000","last_change":"2024-03-18T16:06:44.729777+0000","last_active":"2024-03-18T16:08:47.962153+0000","last_peered":"2024-03-18T16:08:47.962153+0000","last_clean":"2024-03-18T16:08:47.962153+0000","last_became_active":"2024-03-18T16:06:44.729244+0000","last_became_peered":"2024-03-18T16:06:44.729244+0000","last_unstale":"2024-03-18T16:08:47.962153+0000","last_undegraded":"2024-03-18T16:08:47.962153+0000","last_fullsized":"2024-03-18T16:08:47.962153+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:52:05.420443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883601+0000","last_change":"2024-03-18T16:07:47.355131+0000","last_active":"2024-03-18T16:08:47.883601+0000","last_peered":"2024-03-18T16:08:47.883601+0000","last_clean":"2024-03-18T16:08:47.883601+0000","last_became_active":"2024-03-18T16:07:47.354608+0000","last_became_peered":"2024-03-18T16:07:47.354608+0000","last_unstale":"2024-03-18T16:08:47.883601+0000","last_undegraded":"2024-03-18T16:08:47.883601+0000","last_fullsized":"2024-03-18T16:08:47.883601+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:12:57.979168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.886026+0000","last_change":"2024-03-18T16:06:43.715280+0000","last_active":"2024-03-18T16:08:47.886026+0000","last_peered":"2024-03-18T16:08:47.886026+0000","last_clean":"2024-03-18T16:08:47.886026+0000","last_became_active":"2024-03-18T16:06:43.714936+0000","last_became_peered":"2024-03-18T16:06:43.714936+0000","last_unstale":"2024-03-18T16:08:47.886026+0000","last_undegraded":"2024-03-18T16:08:47.886026+0000","last_fullsized":"2024-03-18T16:08:47.886026+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:07:38.746932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.960557+0000","last_change":"2024-03-18T16:06:44.709004+0000","last_active":"2024-03-18T16:08:47.960557+0000","last_peered":"2024-03-18T16:08:47.960557+0000","last_clean":"2024-03-18T16:08:47.960557+0000","last_became_active":"2024-03-18T16:06:44.708574+0000","last_became_peered":"2024-03-18T16:06:44.708574+0000","last_unstale":"2024-03-18T16:08:47.960557+0000","last_undegraded":"2024-03-18T16:08:47.960557+0000","last_fullsized":"2024-03-18T16:08:47.960557+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:02:50.982110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883004+0000","last_change":"2024-03-18T16:06:49.029806+0000","last_active":"2024-03-18T16:08:47.883004+0000","last_peered":"2024-03-18T16:08:47.883004+0000","last_clean":"2024-03-18T16:08:47.883004+0000","last_became_active":"2024-03-18T16:06:49.029404+0000","last_became_peered":"2024-03-18T16:06:49.029404+0000","last_unstale":"2024-03-18T16:08:47.883004+0000","last_undegraded":"2024-03-18T16:08:47.883004+0000","last_fullsized":"2024-03-18T16:08:47.883004+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:38:37.706620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"91'37","reported_seq":256,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:49.075039+0000","last_change":"2024-03-18T16:06:43.702910+0000","last_active":"2024-03-18T16:08:49.075039+0000","last_peered":"2024-03-18T16:08:49.075039+0000","last_clean":"2024-03-18T16:08:49.075039+0000","last_became_active":"2024-03-18T16:06:43.702557+0000","last_became_peered":"2024-03-18T16:06:43.702557+0000","last_unstale":"2024-03-18T16:08:49.075039+0000","last_undegraded":"2024-03-18T16:08:49.075039+0000","last_fullsized":"2024-03-18T16:08:49.075039+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:55:15.362987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61741,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":189,"num_read_kb":155,"num_write":39,"num_write_kb":78,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.884914+0000","last_change":"2024-03-18T16:06:44.709075+0000","last_active":"2024-03-18T16:08:47.884914+0000","last_peered":"2024-03-18T16:08:47.884914+0000","last_clean":"2024-03-18T16:08:47.884914+0000","last_became_active":"2024-03-18T16:06:44.708366+0000","last_became_peered":"2024-03-18T16:06:44.708366+0000","last_unstale":"2024-03-18T16:08:47.884914+0000","last_undegraded":"2024-03-18T16:08:47.884914+0000","last_fullsized":"2024-03-18T16:08:47.884914+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:44:18.486385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.886392+0000","last_change":"2024-03-18T16:06:49.037380+0000","last_active":"2024-03-18T16:08:47.886392+0000","last_peered":"2024-03-18T16:08:47.886392+0000","last_clean":"2024-03-18T16:08:47.886392+0000","last_became_active":"2024-03-18T16:06:49.037021+0000","last_became_peered":"2024-03-18T16:06:49.037021+0000","last_unstale":"2024-03-18T16:08:47.886392+0000","last_undegraded":"2024-03-18T16:08:47.886392+0000","last_fullsized":"2024-03-18T16:08:47.886392+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:43:03.482631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.896395+0000","last_change":"2024-03-18T16:08:47.895964+0000","last_active":"2024-03-18T16:08:47.896395+0000","last_peered":"2024-03-18T16:08:47.896395+0000","last_clean":"2024-03-18T16:08:47.896395+0000","last_became_active":"2024-03-18T16:08:47.886729+0000","last_became_peered":"2024-03-18T16:08:47.886729+0000","last_unstale":"2024-03-18T16:08:47.896395+0000","last_undegraded":"2024-03-18T16:08:47.896395+0000","last_fullsized":"2024-03-18T16:08:47.896395+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:58:02.557326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":68,"num_keys_recovered":10,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.961928+0000","last_change":"2024-03-18T16:06:44.727204+0000","last_active":"2024-03-18T16:08:47.961928+0000","last_peered":"2024-03-18T16:08:47.961928+0000","last_clean":"2024-03-18T16:08:47.961928+0000","last_became_active":"2024-03-18T16:06:44.726775+0000","last_became_peered":"2024-03-18T16:06:44.726775+0000","last_unstale":"2024-03-18T16:08:47.961928+0000","last_undegraded":"2024-03-18T16:08:47.961928+0000","last_fullsized":"2024-03-18T16:08:47.961928+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:29:44.842972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.881431+0000","last_change":"2024-03-18T16:06:49.022508+0000","last_active":"2024-03-18T16:08:47.881431+0000","last_peered":"2024-03-18T16:08:47.881431+0000","last_clean":"2024-03-18T16:08:47.881431+0000","last_became_active":"2024-03-18T16:06:49.022017+0000","last_became_peered":"2024-03-18T16:06:49.022017+0000","last_unstale":"2024-03-18T16:08:47.881431+0000","last_undegraded":"2024-03-18T16:08:47.881431+0000","last_fullsized":"2024-03-18T16:08:47.881431+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:52:45.892880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.885830+0000","last_change":"2024-03-18T16:06:43.719618+0000","last_active":"2024-03-18T16:08:47.885830+0000","last_peered":"2024-03-18T16:08:47.885830+0000","last_clean":"2024-03-18T16:08:47.885830+0000","last_became_active":"2024-03-18T16:06:43.719161+0000","last_became_peered":"2024-03-18T16:06:43.719161+0000","last_unstale":"2024-03-18T16:08:47.885830+0000","last_undegraded":"2024-03-18T16:08:47.885830+0000","last_fullsized":"2024-03-18T16:08:47.885830+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:39:53.077878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.960867+0000","last_change":"2024-03-18T16:06:44.717829+0000","last_active":"2024-03-18T16:08:47.960867+0000","last_peered":"2024-03-18T16:08:47.960867+0000","last_clean":"2024-03-18T16:08:47.960867+0000","last_became_active":"2024-03-18T16:06:44.717473+0000","last_became_peered":"2024-03-18T16:06:44.717473+0000","last_unstale":"2024-03-18T16:08:47.960867+0000","last_undegraded":"2024-03-18T16:08:47.960867+0000","last_fullsized":"2024-03-18T16:08:47.960867+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:10:00.183613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.884313+0000","last_change":"2024-03-18T16:06:49.036766+0000","last_active":"2024-03-18T16:08:47.884313+0000","last_peered":"2024-03-18T16:08:47.884313+0000","last_clean":"2024-03-18T16:08:47.884313+0000","last_became_active":"2024-03-18T16:06:49.034617+0000","last_became_peered":"2024-03-18T16:06:49.034617+0000","last_unstale":"2024-03-18T16:08:47.884313+0000","last_undegraded":"2024-03-18T16:08:47.884313+0000","last_fullsized":"2024-03-18T16:08:47.884313+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:18:59.033781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'5","reported_seq":130,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:51.124500+0000","last_change":"2024-03-18T16:06:43.702452+0000","last_active":"2024-03-18T16:08:51.124500+0000","last_peered":"2024-03-18T16:08:51.124500+0000","last_clean":"2024-03-18T16:08:51.124500+0000","last_became_active":"2024-03-18T16:06:43.702141+0000","last_became_peered":"2024-03-18T16:06:43.702141+0000","last_unstale":"2024-03-18T16:08:51.124500+0000","last_undegraded":"2024-03-18T16:08:51.124500+0000","last_fullsized":"2024-03-18T16:08:51.124500+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:44:05.366101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":96,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.960118+0000","last_change":"2024-03-18T16:06:44.711403+0000","last_active":"2024-03-18T16:08:47.960118+0000","last_peered":"2024-03-18T16:08:47.960118+0000","last_clean":"2024-03-18T16:08:47.960118+0000","last_became_active":"2024-03-18T16:06:44.710654+0000","last_became_peered":"2024-03-18T16:06:44.710654+0000","last_unstale":"2024-03-18T16:08:47.960118+0000","last_undegraded":"2024-03-18T16:08:47.960118+0000","last_fullsized":"2024-03-18T16:08:47.960118+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:21:43.633274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.962752+0000","last_change":"2024-03-18T16:06:49.039192+0000","last_active":"2024-03-18T16:08:47.962752+0000","last_peered":"2024-03-18T16:08:47.962752+0000","last_clean":"2024-03-18T16:08:47.962752+0000","last_became_active":"2024-03-18T16:06:49.038421+0000","last_became_peered":"2024-03-18T16:06:49.038421+0000","last_unstale":"2024-03-18T16:08:47.962752+0000","last_undegraded":"2024-03-18T16:08:47.962752+0000","last_fullsized":"2024-03-18T16:08:47.962752+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:54:56.944209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":48,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.884532+0000","last_change":"2024-03-18T16:06:43.712523+0000","last_active":"2024-03-18T16:08:47.884532+0000","last_peered":"2024-03-18T16:08:47.884532+0000","last_clean":"2024-03-18T16:08:47.884532+0000","last_became_active":"2024-03-18T16:06:43.712151+0000","last_became_peered":"2024-03-18T16:06:43.712151+0000","last_unstale":"2024-03-18T16:08:47.884532+0000","last_undegraded":"2024-03-18T16:08:47.884532+0000","last_fullsized":"2024-03-18T16:08:47.884532+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:17:19.651293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.884846+0000","last_change":"2024-03-18T16:07:47.355115+0000","last_active":"2024-03-18T16:08:47.884846+0000","last_peered":"2024-03-18T16:08:47.884846+0000","last_clean":"2024-03-18T16:08:47.884846+0000","last_became_active":"2024-03-18T16:07:47.354744+0000","last_became_peered":"2024-03-18T16:07:47.354744+0000","last_unstale":"2024-03-18T16:08:47.884846+0000","last_undegraded":"2024-03-18T16:08:47.884846+0000","last_fullsized":"2024-03-18T16:08:47.884846+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:14:27.644891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.884730+0000","last_change":"2024-03-18T16:06:49.028943+0000","last_active":"2024-03-18T16:08:47.884730+0000","last_peered":"2024-03-18T16:08:47.884730+0000","last_clean":"2024-03-18T16:08:47.884730+0000","last_became_active":"2024-03-18T16:06:49.028453+0000","last_became_peered":"2024-03-18T16:06:49.028453+0000","last_unstale":"2024-03-18T16:08:47.884730+0000","last_undegraded":"2024-03-18T16:08:47.884730+0000","last_fullsized":"2024-03-18T16:08:47.884730+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:16:30.318649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":17,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.899433+0000","last_change":"2024-03-18T16:08:47.899048+0000","last_active":"2024-03-18T16:08:47.899433+0000","last_peered":"2024-03-18T16:08:47.899433+0000","last_clean":"2024-03-18T16:08:47.899433+0000","last_became_active":"2024-03-18T16:08:47.887283+0000","last_became_peered":"2024-03-18T16:08:47.887283+0000","last_unstale":"2024-03-18T16:08:47.899433+0000","last_undegraded":"2024-03-18T16:08:47.899433+0000","last_fullsized":"2024-03-18T16:08:47.899433+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:17:17.924210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.881447+0000","last_change":"2024-03-18T16:06:44.716284+0000","last_active":"2024-03-18T16:08:47.881447+0000","last_peered":"2024-03-18T16:08:47.881447+0000","last_clean":"2024-03-18T16:08:47.881447+0000","last_became_active":"2024-03-18T16:06:44.715911+0000","last_became_peered":"2024-03-18T16:06:44.715911+0000","last_unstale":"2024-03-18T16:08:47.881447+0000","last_undegraded":"2024-03-18T16:08:47.881447+0000","last_fullsized":"2024-03-18T16:08:47.881447+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:48:34.839514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.884968+0000","last_change":"2024-03-18T16:06:49.059061+0000","last_active":"2024-03-18T16:08:47.884968+0000","last_peered":"2024-03-18T16:08:47.884968+0000","last_clean":"2024-03-18T16:08:47.884968+0000","last_became_active":"2024-03-18T16:06:49.058619+0000","last_became_peered":"2024-03-18T16:06:49.058619+0000","last_unstale":"2024-03-18T16:08:47.884968+0000","last_undegraded":"2024-03-18T16:08:47.884968+0000","last_fullsized":"2024-03-18T16:08:47.884968+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:44:52.970785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883689+0000","last_change":"2024-03-18T16:06:43.715766+0000","last_active":"2024-03-18T16:08:47.883689+0000","last_peered":"2024-03-18T16:08:47.883689+0000","last_clean":"2024-03-18T16:08:47.883689+0000","last_became_active":"2024-03-18T16:06:43.715380+0000","last_became_peered":"2024-03-18T16:06:43.715380+0000","last_unstale":"2024-03-18T16:08:47.883689+0000","last_undegraded":"2024-03-18T16:08:47.883689+0000","last_fullsized":"2024-03-18T16:08:47.883689+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:13:44.195934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.880175+0000","last_change":"2024-03-18T16:06:44.728150+0000","last_active":"2024-03-18T16:08:47.880175+0000","last_peered":"2024-03-18T16:08:47.880175+0000","last_clean":"2024-03-18T16:08:47.880175+0000","last_became_active":"2024-03-18T16:06:44.727769+0000","last_became_peered":"2024-03-18T16:06:44.727769+0000","last_unstale":"2024-03-18T16:08:47.880175+0000","last_undegraded":"2024-03-18T16:08:47.880175+0000","last_fullsized":"2024-03-18T16:08:47.880175+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:28:16.023225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.961702+0000","last_change":"2024-03-18T16:06:49.033671+0000","last_active":"2024-03-18T16:08:47.961702+0000","last_peered":"2024-03-18T16:08:47.961702+0000","last_clean":"2024-03-18T16:08:47.961702+0000","last_became_active":"2024-03-18T16:06:49.033046+0000","last_became_peered":"2024-03-18T16:06:49.033046+0000","last_unstale":"2024-03-18T16:08:47.961702+0000","last_undegraded":"2024-03-18T16:08:47.961702+0000","last_fullsized":"2024-03-18T16:08:47.961702+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:26:37.020521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'5","reported_seq":130,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:52.188194+0000","last_change":"2024-03-18T16:06:43.712636+0000","last_active":"2024-03-18T16:08:52.188194+0000","last_peered":"2024-03-18T16:08:52.188194+0000","last_clean":"2024-03-18T16:08:52.188194+0000","last_became_active":"2024-03-18T16:06:43.712300+0000","last_became_peered":"2024-03-18T16:06:43.712300+0000","last_unstale":"2024-03-18T16:08:52.188194+0000","last_undegraded":"2024-03-18T16:08:52.188194+0000","last_fullsized":"2024-03-18T16:08:52.188194+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:48:20.386640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":93,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.880029+0000","last_change":"2024-03-18T16:06:44.731583+0000","last_active":"2024-03-18T16:08:47.880029+0000","last_peered":"2024-03-18T16:08:47.880029+0000","last_clean":"2024-03-18T16:08:47.880029+0000","last_became_active":"2024-03-18T16:06:44.731183+0000","last_became_peered":"2024-03-18T16:06:44.731183+0000","last_unstale":"2024-03-18T16:08:47.880029+0000","last_undegraded":"2024-03-18T16:08:47.880029+0000","last_fullsized":"2024-03-18T16:08:47.880029+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:23:05.098101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.962301+0000","last_change":"2024-03-18T16:06:49.060130+0000","last_active":"2024-03-18T16:08:47.962301+0000","last_peered":"2024-03-18T16:08:47.962301+0000","last_clean":"2024-03-18T16:08:47.962301+0000","last_became_active":"2024-03-18T16:06:49.059109+0000","last_became_peered":"2024-03-18T16:06:49.059109+0000","last_unstale":"2024-03-18T16:08:47.962301+0000","last_undegraded":"2024-03-18T16:08:47.962301+0000","last_fullsized":"2024-03-18T16:08:47.962301+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:41:40.008047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.885977+0000","last_change":"2024-03-18T16:06:44.705793+0000","last_active":"2024-03-18T16:08:47.885977+0000","last_peered":"2024-03-18T16:08:47.885977+0000","last_clean":"2024-03-18T16:08:47.885977+0000","last_became_active":"2024-03-18T16:06:44.705426+0000","last_became_peered":"2024-03-18T16:06:44.705426+0000","last_unstale":"2024-03-18T16:08:47.885977+0000","last_undegraded":"2024-03-18T16:08:47.885977+0000","last_fullsized":"2024-03-18T16:08:47.885977+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:14:31.198829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'5","reported_seq":129,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:51.123970+0000","last_change":"2024-03-18T16:06:43.721057+0000","last_active":"2024-03-18T16:08:51.123970+0000","last_peered":"2024-03-18T16:08:51.123970+0000","last_clean":"2024-03-18T16:08:51.123970+0000","last_became_active":"2024-03-18T16:06:43.720661+0000","last_became_peered":"2024-03-18T16:06:43.720661+0000","last_unstale":"2024-03-18T16:08:51.123970+0000","last_undegraded":"2024-03-18T16:08:51.123970+0000","last_fullsized":"2024-03-18T16:08:51.123970+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:08:18.050511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":93,"num_read_kb":93,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.961269+0000","last_change":"2024-03-18T16:07:47.361447+0000","last_active":"2024-03-18T16:08:47.961269+0000","last_peered":"2024-03-18T16:08:47.961269+0000","last_clean":"2024-03-18T16:08:47.961269+0000","last_became_active":"2024-03-18T16:07:47.353684+0000","last_became_peered":"2024-03-18T16:07:47.353684+0000","last_unstale":"2024-03-18T16:08:47.961269+0000","last_undegraded":"2024-03-18T16:08:47.961269+0000","last_fullsized":"2024-03-18T16:08:47.961269+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:59:53.025620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,10],"acting":[7,5,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.885471+0000","last_change":"2024-03-18T16:06:44.725555+0000","last_active":"2024-03-18T16:08:47.885471+0000","last_peered":"2024-03-18T16:08:47.885471+0000","last_clean":"2024-03-18T16:08:47.885471+0000","last_became_active":"2024-03-18T16:06:44.719880+0000","last_became_peered":"2024-03-18T16:06:44.719880+0000","last_unstale":"2024-03-18T16:08:47.885471+0000","last_undegraded":"2024-03-18T16:08:47.885471+0000","last_fullsized":"2024-03-18T16:08:47.885471+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:45:17.830876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883578+0000","last_change":"2024-03-18T16:06:43.700814+0000","last_active":"2024-03-18T16:08:47.883578+0000","last_peered":"2024-03-18T16:08:47.883578+0000","last_clean":"2024-03-18T16:08:47.883578+0000","last_became_active":"2024-03-18T16:06:43.698902+0000","last_became_peered":"2024-03-18T16:06:43.698902+0000","last_unstale":"2024-03-18T16:08:47.883578+0000","last_undegraded":"2024-03-18T16:08:47.883578+0000","last_fullsized":"2024-03-18T16:08:47.883578+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:20:20.100033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":353,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.959985+0000","last_change":"2024-03-18T16:05:22.379397+0000","last_active":"2024-03-18T16:08:47.959985+0000","last_peered":"2024-03-18T16:08:47.959985+0000","last_clean":"2024-03-18T16:08:47.959985+0000","last_became_active":"2024-03-18T16:05:22.378830+0000","last_became_peered":"2024-03-18T16:05:22.378830+0000","last_unstale":"2024-03-18T16:08:47.959985+0000","last_undegraded":"2024-03-18T16:08:47.959985+0000","last_fullsized":"2024-03-18T16:08:47.959985+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:01:30.048984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:01:30.048984+0000","last_clean_scrub_stamp":"2024-03-18T16:01:30.048984+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:35:45.796495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.885183+0000","last_change":"2024-03-18T16:06:49.058931+0000","last_active":"2024-03-18T16:08:47.885183+0000","last_peered":"2024-03-18T16:08:47.885183+0000","last_clean":"2024-03-18T16:08:47.885183+0000","last_became_active":"2024-03-18T16:06:49.058497+0000","last_became_peered":"2024-03-18T16:06:49.058497+0000","last_unstale":"2024-03-18T16:08:47.885183+0000","last_undegraded":"2024-03-18T16:08:47.885183+0000","last_fullsized":"2024-03-18T16:08:47.885183+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:41:51.874582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.961259+0000","last_change":"2024-03-18T16:06:44.710332+0000","last_active":"2024-03-18T16:08:47.961259+0000","last_peered":"2024-03-18T16:08:47.961259+0000","last_clean":"2024-03-18T16:08:47.961259+0000","last_became_active":"2024-03-18T16:06:44.709952+0000","last_became_peered":"2024-03-18T16:06:44.709952+0000","last_unstale":"2024-03-18T16:08:47.961259+0000","last_undegraded":"2024-03-18T16:08:47.961259+0000","last_fullsized":"2024-03-18T16:08:47.961259+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:24:26.653511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.960126+0000","last_change":"2024-03-18T16:06:43.708265+0000","last_active":"2024-03-18T16:08:47.960126+0000","last_peered":"2024-03-18T16:08:47.960126+0000","last_clean":"2024-03-18T16:08:47.960126+0000","last_became_active":"2024-03-18T16:06:43.707466+0000","last_became_peered":"2024-03-18T16:06:43.707466+0000","last_unstale":"2024-03-18T16:08:47.960126+0000","last_undegraded":"2024-03-18T16:08:47.960126+0000","last_fullsized":"2024-03-18T16:08:47.960126+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:45:40.065396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.881967+0000","last_change":"2024-03-18T16:06:49.022401+0000","last_active":"2024-03-18T16:08:47.881967+0000","last_peered":"2024-03-18T16:08:47.881967+0000","last_clean":"2024-03-18T16:08:47.881967+0000","last_became_active":"2024-03-18T16:06:49.021966+0000","last_became_peered":"2024-03-18T16:06:49.021966+0000","last_unstale":"2024-03-18T16:08:47.881967+0000","last_undegraded":"2024-03-18T16:08:47.881967+0000","last_fullsized":"2024-03-18T16:08:47.881967+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:04:12.434969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.885248+0000","last_change":"2024-03-18T16:06:43.713249+0000","last_active":"2024-03-18T16:08:47.885248+0000","last_peered":"2024-03-18T16:08:47.885248+0000","last_clean":"2024-03-18T16:08:47.885248+0000","last_became_active":"2024-03-18T16:06:43.712897+0000","last_became_peered":"2024-03-18T16:06:43.712897+0000","last_unstale":"2024-03-18T16:08:47.885248+0000","last_undegraded":"2024-03-18T16:08:47.885248+0000","last_fullsized":"2024-03-18T16:08:47.885248+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:20:04.642568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.961929+0000","last_change":"2024-03-18T16:06:44.710034+0000","last_active":"2024-03-18T16:08:47.961929+0000","last_peered":"2024-03-18T16:08:47.961929+0000","last_clean":"2024-03-18T16:08:47.961929+0000","last_became_active":"2024-03-18T16:06:44.708457+0000","last_became_peered":"2024-03-18T16:06:44.708457+0000","last_unstale":"2024-03-18T16:08:47.961929+0000","last_undegraded":"2024-03-18T16:08:47.961929+0000","last_fullsized":"2024-03-18T16:08:47.961929+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:40:00.504382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883057+0000","last_change":"2024-03-18T16:06:49.059345+0000","last_active":"2024-03-18T16:08:47.883057+0000","last_peered":"2024-03-18T16:08:47.883057+0000","last_clean":"2024-03-18T16:08:47.883057+0000","last_became_active":"2024-03-18T16:06:49.058675+0000","last_became_peered":"2024-03-18T16:06:49.058675+0000","last_unstale":"2024-03-18T16:08:47.883057+0000","last_undegraded":"2024-03-18T16:08:47.883057+0000","last_fullsized":"2024-03-18T16:08:47.883057+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:03:33.730074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'14","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.884382+0000","last_change":"2024-03-18T16:06:43.724589+0000","last_active":"2024-03-18T16:08:47.884382+0000","last_peered":"2024-03-18T16:08:47.884382+0000","last_clean":"2024-03-18T16:08:47.884382+0000","last_became_active":"2024-03-18T16:06:43.724125+0000","last_became_peered":"2024-03-18T16:06:43.724125+0000","last_unstale":"2024-03-18T16:08:47.884382+0000","last_undegraded":"2024-03-18T16:08:47.884382+0000","last_fullsized":"2024-03-18T16:08:47.884382+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:08:34.921118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.886169+0000","last_change":"2024-03-18T16:06:44.727162+0000","last_active":"2024-03-18T16:08:47.886169+0000","last_peered":"2024-03-18T16:08:47.886169+0000","last_clean":"2024-03-18T16:08:47.886169+0000","last_became_active":"2024-03-18T16:06:44.726552+0000","last_became_peered":"2024-03-18T16:06:44.726552+0000","last_unstale":"2024-03-18T16:08:47.886169+0000","last_undegraded":"2024-03-18T16:08:47.886169+0000","last_fullsized":"2024-03-18T16:08:47.886169+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:36:57.974999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883760+0000","last_change":"2024-03-18T16:06:49.008434+0000","last_active":"2024-03-18T16:08:47.883760+0000","last_peered":"2024-03-18T16:08:47.883760+0000","last_clean":"2024-03-18T16:08:47.883760+0000","last_became_active":"2024-03-18T16:06:49.007944+0000","last_became_peered":"2024-03-18T16:06:49.007944+0000","last_unstale":"2024-03-18T16:08:47.883760+0000","last_undegraded":"2024-03-18T16:08:47.883760+0000","last_fullsized":"2024-03-18T16:08:47.883760+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:01:24.450371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883928+0000","last_change":"2024-03-18T16:06:43.711797+0000","last_active":"2024-03-18T16:08:47.883928+0000","last_peered":"2024-03-18T16:08:47.883928+0000","last_clean":"2024-03-18T16:08:47.883928+0000","last_became_active":"2024-03-18T16:06:43.711493+0000","last_became_peered":"2024-03-18T16:06:43.711493+0000","last_unstale":"2024-03-18T16:08:47.883928+0000","last_undegraded":"2024-03-18T16:08:47.883928+0000","last_fullsized":"2024-03-18T16:08:47.883928+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:50:29.258166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.961855+0000","last_change":"2024-03-18T16:06:44.735910+0000","last_active":"2024-03-18T16:08:47.961855+0000","last_peered":"2024-03-18T16:08:47.961855+0000","last_clean":"2024-03-18T16:08:47.961855+0000","last_became_active":"2024-03-18T16:06:44.735504+0000","last_became_peered":"2024-03-18T16:06:44.735504+0000","last_unstale":"2024-03-18T16:08:47.961855+0000","last_undegraded":"2024-03-18T16:08:47.961855+0000","last_fullsized":"2024-03-18T16:08:47.961855+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:02:18.538217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883643+0000","last_change":"2024-03-18T16:06:49.059130+0000","last_active":"2024-03-18T16:08:47.883643+0000","last_peered":"2024-03-18T16:08:47.883643+0000","last_clean":"2024-03-18T16:08:47.883643+0000","last_became_active":"2024-03-18T16:06:49.058701+0000","last_became_peered":"2024-03-18T16:06:49.058701+0000","last_unstale":"2024-03-18T16:08:47.883643+0000","last_undegraded":"2024-03-18T16:08:47.883643+0000","last_fullsized":"2024-03-18T16:08:47.883643+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:54:52.842786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.959869+0000","last_change":"2024-03-18T16:06:43.714414+0000","last_active":"2024-03-18T16:08:47.959869+0000","last_peered":"2024-03-18T16:08:47.959869+0000","last_clean":"2024-03-18T16:08:47.959869+0000","last_became_active":"2024-03-18T16:06:43.714029+0000","last_became_peered":"2024-03-18T16:06:43.714029+0000","last_unstale":"2024-03-18T16:08:47.959869+0000","last_undegraded":"2024-03-18T16:08:47.959869+0000","last_fullsized":"2024-03-18T16:08:47.959869+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:16:39.060378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.884372+0000","last_change":"2024-03-18T16:06:44.707491+0000","last_active":"2024-03-18T16:08:47.884372+0000","last_peered":"2024-03-18T16:08:47.884372+0000","last_clean":"2024-03-18T16:08:47.884372+0000","last_became_active":"2024-03-18T16:06:44.707157+0000","last_became_peered":"2024-03-18T16:06:44.707157+0000","last_unstale":"2024-03-18T16:08:47.884372+0000","last_undegraded":"2024-03-18T16:08:47.884372+0000","last_fullsized":"2024-03-18T16:08:47.884372+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:19:08.132911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.885895+0000","last_change":"2024-03-18T16:07:47.411002+0000","last_active":"2024-03-18T16:08:47.885895+0000","last_peered":"2024-03-18T16:08:47.885895+0000","last_clean":"2024-03-18T16:08:47.885895+0000","last_became_active":"2024-03-18T16:07:47.410575+0000","last_became_peered":"2024-03-18T16:07:47.410575+0000","last_unstale":"2024-03-18T16:08:47.885895+0000","last_undegraded":"2024-03-18T16:08:47.885895+0000","last_fullsized":"2024-03-18T16:08:47.885895+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:37:25.311205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,2,5],"acting":[8,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.884761+0000","last_change":"2024-03-18T16:06:43.717755+0000","last_active":"2024-03-18T16:08:47.884761+0000","last_peered":"2024-03-18T16:08:47.884761+0000","last_clean":"2024-03-18T16:08:47.884761+0000","last_became_active":"2024-03-18T16:06:43.717278+0000","last_became_peered":"2024-03-18T16:06:43.717278+0000","last_unstale":"2024-03-18T16:08:47.884761+0000","last_undegraded":"2024-03-18T16:08:47.884761+0000","last_fullsized":"2024-03-18T16:08:47.884761+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:48:09.390659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.881255+0000","last_change":"2024-03-18T16:06:44.730858+0000","last_active":"2024-03-18T16:08:47.881255+0000","last_peered":"2024-03-18T16:08:47.881255+0000","last_clean":"2024-03-18T16:08:47.881255+0000","last_became_active":"2024-03-18T16:06:44.730500+0000","last_became_peered":"2024-03-18T16:06:44.730500+0000","last_unstale":"2024-03-18T16:08:47.881255+0000","last_undegraded":"2024-03-18T16:08:47.881255+0000","last_fullsized":"2024-03-18T16:08:47.881255+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:22:03.590834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883553+0000","last_change":"2024-03-18T16:06:49.014717+0000","last_active":"2024-03-18T16:08:47.883553+0000","last_peered":"2024-03-18T16:08:47.883553+0000","last_clean":"2024-03-18T16:08:47.883553+0000","last_became_active":"2024-03-18T16:06:49.014313+0000","last_became_peered":"2024-03-18T16:06:49.014313+0000","last_unstale":"2024-03-18T16:08:47.883553+0000","last_undegraded":"2024-03-18T16:08:47.883553+0000","last_fullsized":"2024-03-18T16:08:47.883553+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:06:12.168888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'3","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883632+0000","last_change":"2024-03-18T16:06:43.719496+0000","last_active":"2024-03-18T16:08:47.883632+0000","last_peered":"2024-03-18T16:08:47.883632+0000","last_clean":"2024-03-18T16:08:47.883632+0000","last_became_active":"2024-03-18T16:06:43.719139+0000","last_became_peered":"2024-03-18T16:06:43.719139+0000","last_unstale":"2024-03-18T16:08:47.883632+0000","last_undegraded":"2024-03-18T16:08:47.883632+0000","last_fullsized":"2024-03-18T16:08:47.883632+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:10:01.250923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.885958+0000","last_change":"2024-03-18T16:06:44.737792+0000","last_active":"2024-03-18T16:08:47.885958+0000","last_peered":"2024-03-18T16:08:47.885958+0000","last_clean":"2024-03-18T16:08:47.885958+0000","last_became_active":"2024-03-18T16:06:44.736311+0000","last_became_peered":"2024-03-18T16:06:44.736311+0000","last_unstale":"2024-03-18T16:08:47.885958+0000","last_undegraded":"2024-03-18T16:08:47.885958+0000","last_fullsized":"2024-03-18T16:08:47.885958+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:43:56.110205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883609+0000","last_change":"2024-03-18T16:06:49.058893+0000","last_active":"2024-03-18T16:08:47.883609+0000","last_peered":"2024-03-18T16:08:47.883609+0000","last_clean":"2024-03-18T16:08:47.883609+0000","last_became_active":"2024-03-18T16:06:49.058477+0000","last_became_peered":"2024-03-18T16:06:49.058477+0000","last_unstale":"2024-03-18T16:08:47.883609+0000","last_undegraded":"2024-03-18T16:08:47.883609+0000","last_fullsized":"2024-03-18T16:08:47.883609+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:50:51.890348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.884563+0000","last_change":"2024-03-18T16:06:43.703073+0000","last_active":"2024-03-18T16:08:47.884563+0000","last_peered":"2024-03-18T16:08:47.884563+0000","last_clean":"2024-03-18T16:08:47.884563+0000","last_became_active":"2024-03-18T16:06:43.702708+0000","last_became_peered":"2024-03-18T16:06:43.702708+0000","last_unstale":"2024-03-18T16:08:47.884563+0000","last_undegraded":"2024-03-18T16:08:47.884563+0000","last_fullsized":"2024-03-18T16:08:47.884563+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:38:41.454458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.960699+0000","last_change":"2024-03-18T16:06:44.713622+0000","last_active":"2024-03-18T16:08:47.960699+0000","last_peered":"2024-03-18T16:08:47.960699+0000","last_clean":"2024-03-18T16:08:47.960699+0000","last_became_active":"2024-03-18T16:06:44.713160+0000","last_became_peered":"2024-03-18T16:06:44.713160+0000","last_unstale":"2024-03-18T16:08:47.960699+0000","last_undegraded":"2024-03-18T16:08:47.960699+0000","last_fullsized":"2024-03-18T16:08:47.960699+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:49:21.886543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.959868+0000","last_change":"2024-03-18T16:07:47.411958+0000","last_active":"2024-03-18T16:08:47.959868+0000","last_peered":"2024-03-18T16:08:47.959868+0000","last_clean":"2024-03-18T16:08:47.959868+0000","last_became_active":"2024-03-18T16:07:47.411433+0000","last_became_peered":"2024-03-18T16:07:47.411433+0000","last_unstale":"2024-03-18T16:08:47.959868+0000","last_undegraded":"2024-03-18T16:08:47.959868+0000","last_fullsized":"2024-03-18T16:08:47.959868+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:58:47.853897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6,7],"acting":[1,8,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'5","reported_seq":126,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.884800+0000","last_change":"2024-03-18T16:06:43.717161+0000","last_active":"2024-03-18T16:08:47.884800+0000","last_peered":"2024-03-18T16:08:47.884800+0000","last_clean":"2024-03-18T16:08:47.884800+0000","last_became_active":"2024-03-18T16:06:43.716657+0000","last_became_peered":"2024-03-18T16:06:43.716657+0000","last_unstale":"2024-03-18T16:08:47.884800+0000","last_undegraded":"2024-03-18T16:08:47.884800+0000","last_fullsized":"2024-03-18T16:08:47.884800+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:23:48.194779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":91,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.961073+0000","last_change":"2024-03-18T16:06:44.709777+0000","last_active":"2024-03-18T16:08:47.961073+0000","last_peered":"2024-03-18T16:08:47.961073+0000","last_clean":"2024-03-18T16:08:47.961073+0000","last_became_active":"2024-03-18T16:06:44.709409+0000","last_became_peered":"2024-03-18T16:06:44.709409+0000","last_unstale":"2024-03-18T16:08:47.961073+0000","last_undegraded":"2024-03-18T16:08:47.961073+0000","last_fullsized":"2024-03-18T16:08:47.961073+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:37:12.288262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.961469+0000","last_change":"2024-03-18T16:06:49.040872+0000","last_active":"2024-03-18T16:08:47.961469+0000","last_peered":"2024-03-18T16:08:47.961469+0000","last_clean":"2024-03-18T16:08:47.961469+0000","last_became_active":"2024-03-18T16:06:49.039807+0000","last_became_peered":"2024-03-18T16:06:49.039807+0000","last_unstale":"2024-03-18T16:08:47.961469+0000","last_undegraded":"2024-03-18T16:08:47.961469+0000","last_fullsized":"2024-03-18T16:08:47.961469+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:09:27.045450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"89'3","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.886202+0000","last_change":"2024-03-18T16:06:43.723675+0000","last_active":"2024-03-18T16:08:47.886202+0000","last_peered":"2024-03-18T16:08:47.886202+0000","last_clean":"2024-03-18T16:08:47.886202+0000","last_became_active":"2024-03-18T16:06:43.723334+0000","last_became_peered":"2024-03-18T16:06:43.723334+0000","last_unstale":"2024-03-18T16:08:47.886202+0000","last_undegraded":"2024-03-18T16:08:47.886202+0000","last_fullsized":"2024-03-18T16:08:47.886202+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:01:36.553362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.961035+0000","last_change":"2024-03-18T16:07:47.359134+0000","last_active":"2024-03-18T16:08:47.961035+0000","last_peered":"2024-03-18T16:08:47.961035+0000","last_clean":"2024-03-18T16:08:47.961035+0000","last_became_active":"2024-03-18T16:07:47.358692+0000","last_became_peered":"2024-03-18T16:07:47.358692+0000","last_unstale":"2024-03-18T16:08:47.961035+0000","last_undegraded":"2024-03-18T16:08:47.961035+0000","last_fullsized":"2024-03-18T16:08:47.961035+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:48:22.929034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.885495+0000","last_change":"2024-03-18T16:06:49.011981+0000","last_active":"2024-03-18T16:08:47.885495+0000","last_peered":"2024-03-18T16:08:47.885495+0000","last_clean":"2024-03-18T16:08:47.885495+0000","last_became_active":"2024-03-18T16:06:49.011621+0000","last_became_peered":"2024-03-18T16:06:49.011621+0000","last_unstale":"2024-03-18T16:08:47.885495+0000","last_undegraded":"2024-03-18T16:08:47.885495+0000","last_fullsized":"2024-03-18T16:08:47.885495+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:11:42.382721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.962644+0000","last_change":"2024-03-18T16:06:43.725404+0000","last_active":"2024-03-18T16:08:47.962644+0000","last_peered":"2024-03-18T16:08:47.962644+0000","last_clean":"2024-03-18T16:08:47.962644+0000","last_became_active":"2024-03-18T16:06:43.724117+0000","last_became_peered":"2024-03-18T16:06:43.724117+0000","last_unstale":"2024-03-18T16:08:47.962644+0000","last_undegraded":"2024-03-18T16:08:47.962644+0000","last_fullsized":"2024-03-18T16:08:47.962644+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:58:43.243070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.961253+0000","last_change":"2024-03-18T16:06:44.709979+0000","last_active":"2024-03-18T16:08:47.961253+0000","last_peered":"2024-03-18T16:08:47.961253+0000","last_clean":"2024-03-18T16:08:47.961253+0000","last_became_active":"2024-03-18T16:06:44.709591+0000","last_became_peered":"2024-03-18T16:06:44.709591+0000","last_unstale":"2024-03-18T16:08:47.961253+0000","last_undegraded":"2024-03-18T16:08:47.961253+0000","last_fullsized":"2024-03-18T16:08:47.961253+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:56:37.537248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.881778+0000","last_change":"2024-03-18T16:07:47.354636+0000","last_active":"2024-03-18T16:08:47.881778+0000","last_peered":"2024-03-18T16:08:47.881778+0000","last_clean":"2024-03-18T16:08:47.881778+0000","last_became_active":"2024-03-18T16:07:47.354219+0000","last_became_peered":"2024-03-18T16:07:47.354219+0000","last_unstale":"2024-03-18T16:08:47.881778+0000","last_undegraded":"2024-03-18T16:08:47.881778+0000","last_fullsized":"2024-03-18T16:08:47.881778+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:35:48.094404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,9],"acting":[11,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883718+0000","last_change":"2024-03-18T16:06:44.714836+0000","last_active":"2024-03-18T16:08:47.883718+0000","last_peered":"2024-03-18T16:08:47.883718+0000","last_clean":"2024-03-18T16:08:47.883718+0000","last_became_active":"2024-03-18T16:06:44.714454+0000","last_became_peered":"2024-03-18T16:06:44.714454+0000","last_unstale":"2024-03-18T16:08:47.883718+0000","last_undegraded":"2024-03-18T16:08:47.883718+0000","last_fullsized":"2024-03-18T16:08:47.883718+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:05:24.532238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883502+0000","last_change":"2024-03-18T16:06:44.733183+0000","last_active":"2024-03-18T16:08:47.883502+0000","last_peered":"2024-03-18T16:08:47.883502+0000","last_clean":"2024-03-18T16:08:47.883502+0000","last_became_active":"2024-03-18T16:06:44.732909+0000","last_became_peered":"2024-03-18T16:06:44.732909+0000","last_unstale":"2024-03-18T16:08:47.883502+0000","last_undegraded":"2024-03-18T16:08:47.883502+0000","last_fullsized":"2024-03-18T16:08:47.883502+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:28:07.636692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.961804+0000","last_change":"2024-03-18T16:06:44.731050+0000","last_active":"2024-03-18T16:08:47.961804+0000","last_peered":"2024-03-18T16:08:47.961804+0000","last_clean":"2024-03-18T16:08:47.961804+0000","last_became_active":"2024-03-18T16:06:44.730634+0000","last_became_peered":"2024-03-18T16:06:44.730634+0000","last_unstale":"2024-03-18T16:08:47.961804+0000","last_undegraded":"2024-03-18T16:08:47.961804+0000","last_fullsized":"2024-03-18T16:08:47.961804+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:07:18.619130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.961995+0000","last_change":"2024-03-18T16:06:44.735190+0000","last_active":"2024-03-18T16:08:47.961995+0000","last_peered":"2024-03-18T16:08:47.961995+0000","last_clean":"2024-03-18T16:08:47.961995+0000","last_became_active":"2024-03-18T16:06:44.734312+0000","last_became_peered":"2024-03-18T16:06:44.734312+0000","last_unstale":"2024-03-18T16:08:47.961995+0000","last_undegraded":"2024-03-18T16:08:47.961995+0000","last_fullsized":"2024-03-18T16:08:47.961995+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:59:50.209515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.881817+0000","last_change":"2024-03-18T16:06:44.705385+0000","last_active":"2024-03-18T16:08:47.881817+0000","last_peered":"2024-03-18T16:08:47.881817+0000","last_clean":"2024-03-18T16:08:47.881817+0000","last_became_active":"2024-03-18T16:06:44.705011+0000","last_became_peered":"2024-03-18T16:06:44.705011+0000","last_unstale":"2024-03-18T16:08:47.881817+0000","last_undegraded":"2024-03-18T16:08:47.881817+0000","last_fullsized":"2024-03-18T16:08:47.881817+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:50:02.152851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.885230+0000","last_change":"2024-03-18T16:06:44.709621+0000","last_active":"2024-03-18T16:08:47.885230+0000","last_peered":"2024-03-18T16:08:47.885230+0000","last_clean":"2024-03-18T16:08:47.885230+0000","last_became_active":"2024-03-18T16:06:44.709275+0000","last_became_peered":"2024-03-18T16:06:44.709275+0000","last_unstale":"2024-03-18T16:08:47.885230+0000","last_undegraded":"2024-03-18T16:08:47.885230+0000","last_fullsized":"2024-03-18T16:08:47.885230+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:41:17.346661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.886274+0000","last_change":"2024-03-18T16:06:44.719827+0000","last_active":"2024-03-18T16:08:47.886274+0000","last_peered":"2024-03-18T16:08:47.886274+0000","last_clean":"2024-03-18T16:08:47.886274+0000","last_became_active":"2024-03-18T16:06:44.719432+0000","last_became_peered":"2024-03-18T16:06:44.719432+0000","last_unstale":"2024-03-18T16:08:47.886274+0000","last_undegraded":"2024-03-18T16:08:47.886274+0000","last_fullsized":"2024-03-18T16:08:47.886274+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:13:42.344211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.885754+0000","last_change":"2024-03-18T16:06:44.714526+0000","last_active":"2024-03-18T16:08:47.885754+0000","last_peered":"2024-03-18T16:08:47.885754+0000","last_clean":"2024-03-18T16:08:47.885754+0000","last_became_active":"2024-03-18T16:06:44.714184+0000","last_became_peered":"2024-03-18T16:06:44.714184+0000","last_unstale":"2024-03-18T16:08:47.885754+0000","last_undegraded":"2024-03-18T16:08:47.885754+0000","last_fullsized":"2024-03-18T16:08:47.885754+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:31:07.917143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.884885+0000","last_change":"2024-03-18T16:06:44.730905+0000","last_active":"2024-03-18T16:08:47.884885+0000","last_peered":"2024-03-18T16:08:47.884885+0000","last_clean":"2024-03-18T16:08:47.884885+0000","last_became_active":"2024-03-18T16:06:44.729002+0000","last_became_peered":"2024-03-18T16:06:44.729002+0000","last_unstale":"2024-03-18T16:08:47.884885+0000","last_undegraded":"2024-03-18T16:08:47.884885+0000","last_fullsized":"2024-03-18T16:08:47.884885+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:49:18.867571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.960152+0000","last_change":"2024-03-18T16:06:44.727608+0000","last_active":"2024-03-18T16:08:47.960152+0000","last_peered":"2024-03-18T16:08:47.960152+0000","last_clean":"2024-03-18T16:08:47.960152+0000","last_became_active":"2024-03-18T16:06:44.727096+0000","last_became_peered":"2024-03-18T16:06:44.727096+0000","last_unstale":"2024-03-18T16:08:47.960152+0000","last_undegraded":"2024-03-18T16:08:47.960152+0000","last_fullsized":"2024-03-18T16:08:47.960152+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:08:20.767443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.962383+0000","last_change":"2024-03-18T16:06:44.731823+0000","last_active":"2024-03-18T16:08:47.962383+0000","last_peered":"2024-03-18T16:08:47.962383+0000","last_clean":"2024-03-18T16:08:47.962383+0000","last_became_active":"2024-03-18T16:06:44.731429+0000","last_became_peered":"2024-03-18T16:06:44.731429+0000","last_unstale":"2024-03-18T16:08:47.962383+0000","last_undegraded":"2024-03-18T16:08:47.962383+0000","last_fullsized":"2024-03-18T16:08:47.962383+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:36:28.752466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.881547+0000","last_change":"2024-03-18T16:06:44.714354+0000","last_active":"2024-03-18T16:08:47.881547+0000","last_peered":"2024-03-18T16:08:47.881547+0000","last_clean":"2024-03-18T16:08:47.881547+0000","last_became_active":"2024-03-18T16:06:44.713969+0000","last_became_peered":"2024-03-18T16:06:44.713969+0000","last_unstale":"2024-03-18T16:08:47.881547+0000","last_undegraded":"2024-03-18T16:08:47.881547+0000","last_fullsized":"2024-03-18T16:08:47.881547+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:18:49.502178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":68695,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":949,"num_read_kb":549,"num_write":249,"num_write_kb":143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":90,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":206085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":216,"ondisk_log_size":216,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383721,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36392,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708760,"statfs":{"total":95995035648,"available":95957770240,"internally_reserved":0,"allocated":1077248,"data_stored":520297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645356,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37064,"kb_used_data":1720,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708088,"statfs":{"total":95995035648,"available":95957082112,"internally_reserved":0,"allocated":1761280,"data_stored":1172860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808882,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28088,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1089536,"data_stored":520137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070522,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36396,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708756,"statfs":{"total":95995035648,"available":95957766144,"internally_reserved":0,"allocated":1081344,"data_stored":520493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332158,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28668,"kb_used_data":1644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716484,"statfs":{"total":95995035648,"available":95965679616,"internally_reserved":0,"allocated":1683456,"data_stored":1117153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495683,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32224,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712928,"statfs":{"total":95995035648,"available":95962038272,"internally_reserved":0,"allocated":1064960,"data_stored":515641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574943,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36448,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708704,"statfs":{"total":95995035648,"available":95957712896,"internally_reserved":0,"allocated":1130496,"data_stored":578412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738470,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28088,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1089536,"data_stored":517097,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411419,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36460,"kb_used_data":1116,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708692,"statfs":{"total":95995035648,"available":95957700608,"internally_reserved":0,"allocated":1142784,"data_stored":579096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182485,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1077248,"data_stored":515349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920848,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36392,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708760,"statfs":{"total":95995035648,"available":95957770240,"internally_reserved":0,"allocated":1073152,"data_stored":519715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757323,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36996,"kb_used_data":1652,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708156,"statfs":{"total":95995035648,"available":95957151744,"internally_reserved":0,"allocated":1691648,"data_stored":1112735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":63493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":64177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":63477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1282,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T16:08:56.330 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-18T16:08:56.330 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-18T16:08:56.330 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-18T16:08:56.331 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph health --format=json 2024-03-18T16:08:56.841 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T16:08:57.191 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:08:56 smithi039 ceph-mon[31721]: from='client.25046 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T16:08:57.191 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:08:56 smithi039 ceph-mon[31721]: pgmap v388: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 590 B/s wr, 11 op/s; 10 B/s, 1 keys/s, 0 objects/s recovering 2024-03-18T16:08:57.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:08:56 smithi196 ceph-mon[36307]: from='client.25046 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T16:08:57.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:08:56 smithi196 ceph-mon[36307]: pgmap v388: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 590 B/s wr, 11 op/s; 10 B/s, 1 keys/s, 0 objects/s recovering 2024-03-18T16:08:57.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:08:56 smithi194 ceph-mon[36200]: from='client.25046 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T16:08:57.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:08:56 smithi194 ceph-mon[36200]: pgmap v388: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 590 B/s wr, 11 op/s; 10 B/s, 1 keys/s, 0 objects/s recovering 2024-03-18T16:08:59.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:59.162+0000 7fd435600700 1 -- 172.21.15.39:0/3464850991 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd43010a590 msgr2=0x7fd43010a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:59.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:59.162+0000 7fd435600700 1 --2- 172.21.15.39:0/3464850991 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd43010a590 0x7fd43010a970 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd4200099c0 tx=0x7fd420009cd0 comp rx=0 tx=0).stop 2024-03-18T16:08:59.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:59.162+0000 7fd435600700 1 -- 172.21.15.39:0/3464850991 shutdown_connections 2024-03-18T16:08:59.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:59.162+0000 7fd435600700 1 --2- 172.21.15.39:0/3464850991 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd430104ea0 0x7fd430111360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:59.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:59.162+0000 7fd435600700 1 --2- 172.21.15.39:0/3464850991 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd43010a590 0x7fd43010a970 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:59.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:59.162+0000 7fd435600700 1 --2- 172.21.15.39:0/3464850991 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd4301044e0 0x7fd430104960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:59.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:59.162+0000 7fd435600700 1 -- 172.21.15.39:0/3464850991 >> 172.21.15.39:0/3464850991 conn(0x7fd4300fffb0 msgr2=0x7fd4301023d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:08:59.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:59.163+0000 7fd435600700 1 -- 172.21.15.39:0/3464850991 shutdown_connections 2024-03-18T16:08:59.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:59.163+0000 7fd435600700 1 -- 172.21.15.39:0/3464850991 wait complete. 2024-03-18T16:08:59.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:59.164+0000 7fd435600700 1 Processor -- start 2024-03-18T16:08:59.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:59.164+0000 7fd435600700 1 -- start start 2024-03-18T16:08:59.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:59.165+0000 7fd435600700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd4301044e0 0x7fd4301a0790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:59.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:59.165+0000 7fd435600700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd430104ea0 0x7fd4301a0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:59.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:59.165+0000 7fd435600700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd43010a590 0x7fd43019a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:59.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:59.165+0000 7fd435600700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fd4301147d0 con 0x7fd43010a590 2024-03-18T16:08:59.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:59.165+0000 7fd435600700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fd430114650 con 0x7fd4301044e0 2024-03-18T16:08:59.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:59.165+0000 7fd435600700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fd430114950 con 0x7fd430104ea0 2024-03-18T16:08:59.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:59.165+0000 7fd427e00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd430104ea0 0x7fd4301a0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:59.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:59.165+0000 7fd42f400700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd43010a590 0x7fd43019a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:59.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:59.165+0000 7fd427e00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd430104ea0 0x7fd4301a0cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:35946/0 (socket says 172.21.15.39:35946) 2024-03-18T16:08:59.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:59.165+0000 7fd427e00700 1 -- 172.21.15.39:0/2411834422 learned_addr learned my addr 172.21.15.39:0/2411834422 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:08:59.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:59.165+0000 7fd42ea00700 1 --2- 172.21.15.39:0/2411834422 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd4301044e0 0x7fd4301a0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:59.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:59.166+0000 7fd427e00700 1 -- 172.21.15.39:0/2411834422 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd4301044e0 msgr2=0x7fd4301a0790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:59.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:59.166+0000 7fd427e00700 1 --2- 172.21.15.39:0/2411834422 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd4301044e0 0x7fd4301a0790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:59.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:59.166+0000 7fd427e00700 1 -- 172.21.15.39:0/2411834422 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd43010a590 msgr2=0x7fd43019a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:59.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:59.166+0000 7fd427e00700 1 --2- 172.21.15.39:0/2411834422 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd43010a590 0x7fd43019a9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:59.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:59.166+0000 7fd427e00700 1 -- 172.21.15.39:0/2411834422 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd43019b150 con 0x7fd430104ea0 2024-03-18T16:08:59.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:59.166+0000 7fd427e00700 1 --2- 172.21.15.39:0/2411834422 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd430104ea0 0x7fd4301a0cd0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fd41800e9a0 tx=0x7fd41800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:59.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:59.166+0000 7fd427400700 1 -- 172.21.15.39:0/2411834422 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd41800c9c0 con 0x7fd430104ea0 2024-03-18T16:08:59.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:59.166+0000 7fd427400700 1 -- 172.21.15.39:0/2411834422 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd41800cb60 con 0x7fd430104ea0 2024-03-18T16:08:59.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:59.166+0000 7fd435600700 1 -- 172.21.15.39:0/2411834422 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd43019b440 con 0x7fd430104ea0 2024-03-18T16:08:59.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:59.166+0000 7fd427400700 1 -- 172.21.15.39:0/2411834422 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd418018660 con 0x7fd430104ea0 2024-03-18T16:08:59.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:59.167+0000 7fd435600700 1 -- 172.21.15.39:0/2411834422 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4301a7660 con 0x7fd430104ea0 2024-03-18T16:08:59.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:59.168+0000 7fd435600700 1 -- 172.21.15.39:0/2411834422 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd43010ca20 con 0x7fd430104ea0 2024-03-18T16:08:59.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:59.170+0000 7fd427400700 1 -- 172.21.15.39:0/2411834422 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fd41800cd10 con 0x7fd430104ea0 2024-03-18T16:08:59.172 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:59.171+0000 7fd427400700 1 --2- 172.21.15.39:0/2411834422 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd4140a2910 0x7fd4140a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:08:59.172 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:59.171+0000 7fd427400700 1 -- 172.21.15.39:0/2411834422 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fd4180cef70 con 0x7fd430104ea0 2024-03-18T16:08:59.172 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:59.171+0000 7fd42ea00700 1 --2- 172.21.15.39:0/2411834422 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd4140a2910 0x7fd4140a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:08:59.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:59.172+0000 7fd42ea00700 1 --2- 172.21.15.39:0/2411834422 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd4140a2910 0x7fd4140a4dd0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fd420000c00 tx=0x7fd420005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:08:59.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:59.175+0000 7fd427400700 1 -- 172.21.15.39:0/2411834422 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd41809bc20 con 0x7fd430104ea0 2024-03-18T16:08:59.227 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:08:58 smithi196 ceph-mon[36307]: pgmap v389: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 409 B/s wr, 10 op/s; 8 B/s, 0 keys/s, 0 objects/s recovering 2024-03-18T16:08:59.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:08:58 smithi194 ceph-mon[36200]: pgmap v389: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 409 B/s wr, 10 op/s; 8 B/s, 0 keys/s, 0 objects/s recovering 2024-03-18T16:08:59.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:08:58 smithi039 ceph-mon[31721]: pgmap v389: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 409 B/s wr, 10 op/s; 8 B/s, 0 keys/s, 0 objects/s recovering 2024-03-18T16:08:59.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:59.465+0000 7fd435600700 1 -- 172.21.15.39:0/2411834422 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fd430068f90 con 0x7fd430104ea0 2024-03-18T16:08:59.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:59.465+0000 7fd427400700 1 -- 172.21.15.39:0/2411834422 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7fd41809b520 con 0x7fd430104ea0 2024-03-18T16:08:59.467 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:08:59.467 INFO:teuthology.orchestra.run.smithi039.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-18T16:08:59.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:59.469+0000 7fd435600700 1 -- 172.21.15.39:0/2411834422 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd4140a2910 msgr2=0x7fd4140a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:59.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:59.469+0000 7fd435600700 1 --2- 172.21.15.39:0/2411834422 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd4140a2910 0x7fd4140a4dd0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fd420000c00 tx=0x7fd420005fb0 comp rx=0 tx=0).stop 2024-03-18T16:08:59.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:59.469+0000 7fd435600700 1 -- 172.21.15.39:0/2411834422 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd430104ea0 msgr2=0x7fd4301a0cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:08:59.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:59.469+0000 7fd435600700 1 --2- 172.21.15.39:0/2411834422 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd430104ea0 0x7fd4301a0cd0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fd41800e9a0 tx=0x7fd41800ecb0 comp rx=0 tx=0).stop 2024-03-18T16:08:59.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:59.469+0000 7fd435600700 1 -- 172.21.15.39:0/2411834422 shutdown_connections 2024-03-18T16:08:59.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:59.469+0000 7fd435600700 1 --2- 172.21.15.39:0/2411834422 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd4140a2910 0x7fd4140a4dd0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:59.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:59.469+0000 7fd435600700 1 --2- 172.21.15.39:0/2411834422 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd43010a590 0x7fd43019a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:59.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:59.469+0000 7fd435600700 1 --2- 172.21.15.39:0/2411834422 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd4301044e0 0x7fd4301a0790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:59.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:59.469+0000 7fd435600700 1 --2- 172.21.15.39:0/2411834422 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd430104ea0 0x7fd4301a0cd0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:08:59.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:59.469+0000 7fd435600700 1 -- 172.21.15.39:0/2411834422 >> 172.21.15.39:0/2411834422 conn(0x7fd4300fffb0 msgr2=0x7fd4301015b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:08:59.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:59.469+0000 7fd435600700 1 -- 172.21.15.39:0/2411834422 shutdown_connections 2024-03-18T16:08:59.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:08:59.469+0000 7fd435600700 1 -- 172.21.15.39:0/2411834422 wait complete. 2024-03-18T16:09:00.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:08:59 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/2411834422' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-18T16:09:00.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:08:59 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/2411834422' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-18T16:09:00.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:08:59 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/2411834422' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-18T16:09:00.457 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-18T16:09:00.457 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T16:09:00.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:00.695+0000 7fbce3200640 1 -- 172.21.15.39:0/1347995283 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbce41027c0 msgr2=0x7fbce4102c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:00.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:00.695+0000 7fbce3200640 1 --2- 172.21.15.39:0/1347995283 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbce41027c0 0x7fbce4102c40 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fbccc009860 tx=0x7fbccc02bb70 comp rx=0 tx=0).stop 2024-03-18T16:09:00.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:00.695+0000 7fbce3200640 1 -- 172.21.15.39:0/1347995283 shutdown_connections 2024-03-18T16:09:00.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:00.695+0000 7fbce3200640 1 --2- 172.21.15.39:0/1347995283 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbce4103180 0x7fbce4109a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:00.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:00.695+0000 7fbce3200640 1 --2- 172.21.15.39:0/1347995283 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbce41027c0 0x7fbce4102c40 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:00.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:00.695+0000 7fbce3200640 1 --2- 172.21.15.39:0/1347995283 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbce41015d0 0x7fbce41019f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:00.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:00.695+0000 7fbce3200640 1 -- 172.21.15.39:0/1347995283 >> 172.21.15.39:0/1347995283 conn(0x7fbce40fcb40 msgr2=0x7fbce40fef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:00.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:00.695+0000 7fbce3200640 1 -- 172.21.15.39:0/1347995283 shutdown_connections 2024-03-18T16:09:00.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:00.695+0000 7fbce3200640 1 -- 172.21.15.39:0/1347995283 wait complete. 2024-03-18T16:09:00.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:00.696+0000 7fbce3200640 1 Processor -- start 2024-03-18T16:09:00.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:00.696+0000 7fbce3200640 1 -- start start 2024-03-18T16:09:00.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:00.697+0000 7fbce3200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbce41015d0 0x7fbce4199ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:00.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:00.697+0000 7fbce3200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbce41027c0 0x7fbce4199ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:00.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:00.697+0000 7fbce3200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbce4103180 0x7fbce41a1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:00.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:00.697+0000 7fbce3200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fbce410d230 con 0x7fbce41027c0 2024-03-18T16:09:00.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:00.697+0000 7fbce3200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fbce410d0b0 con 0x7fbce41015d0 2024-03-18T16:09:00.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:00.697+0000 7fbce3200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fbce410d3b0 con 0x7fbce4103180 2024-03-18T16:09:00.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:00.697+0000 7fbcd9400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbce41027c0 0x7fbce4199ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:00.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:00.697+0000 7fbce1e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbce41015d0 0x7fbce4199ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:00.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:00.697+0000 7fbce2800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbce4103180 0x7fbce41a1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:00.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:00.697+0000 7fbce1e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbce41015d0 0x7fbce4199ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:35066/0 (socket says 172.21.15.39:35066) 2024-03-18T16:09:00.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:00.697+0000 7fbce2800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbce4103180 0x7fbce41a1060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:35962/0 (socket says 172.21.15.39:35962) 2024-03-18T16:09:00.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:00.697+0000 7fbce1e00640 1 -- 172.21.15.39:0/2098170285 learned_addr learned my addr 172.21.15.39:0/2098170285 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:00.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:00.697+0000 7fbce1e00640 1 -- 172.21.15.39:0/2098170285 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbce4103180 msgr2=0x7fbce41a1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:00.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:00.697+0000 7fbce1e00640 1 --2- 172.21.15.39:0/2098170285 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbce4103180 0x7fbce41a1060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:00.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:00.697+0000 7fbce1e00640 1 -- 172.21.15.39:0/2098170285 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbce41027c0 msgr2=0x7fbce4199ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:00.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:00.697+0000 7fbce1e00640 1 --2- 172.21.15.39:0/2098170285 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbce41027c0 0x7fbce4199ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:00.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:00.697+0000 7fbce1e00640 1 -- 172.21.15.39:0/2098170285 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbce41a16d0 con 0x7fbce41015d0 2024-03-18T16:09:00.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:00.697+0000 7fbce2800640 1 --2- 172.21.15.39:0/2098170285 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbce4103180 0x7fbce41a1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:09:00.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:00.698+0000 7fbce1e00640 1 --2- 172.21.15.39:0/2098170285 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbce41015d0 0x7fbce4199ab0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fbcd400e9f0 tx=0x7fbcd400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:00.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:00.698+0000 7fbcdb400640 1 -- 172.21.15.39:0/2098170285 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcd400ccc0 con 0x7fbce41015d0 2024-03-18T16:09:00.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:00.698+0000 7fbcdb400640 1 -- 172.21.15.39:0/2098170285 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbcd40035f0 con 0x7fbce41015d0 2024-03-18T16:09:00.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:00.698+0000 7fbce3200640 1 -- 172.21.15.39:0/2098170285 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbce41a19c0 con 0x7fbce41015d0 2024-03-18T16:09:00.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:00.698+0000 7fbcdb400640 1 -- 172.21.15.39:0/2098170285 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcd400ce60 con 0x7fbce41015d0 2024-03-18T16:09:00.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:00.698+0000 7fbce3200640 1 -- 172.21.15.39:0/2098170285 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbce41a1f30 con 0x7fbce41015d0 2024-03-18T16:09:00.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:00.699+0000 7fbce3200640 1 -- 172.21.15.39:0/2098170285 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbce41a21c0 con 0x7fbce41015d0 2024-03-18T16:09:00.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:00.701+0000 7fbcdb400640 1 -- 172.21.15.39:0/2098170285 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fbcd40037a0 con 0x7fbce41015d0 2024-03-18T16:09:00.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:00.702+0000 7fbcdb400640 1 --2- 172.21.15.39:0/2098170285 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fbcc80a2880 0x7fbcc80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:00.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:00.703+0000 7fbcdb400640 1 -- 172.21.15.39:0/2098170285 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fbcd40ceff0 con 0x7fbce41015d0 2024-03-18T16:09:00.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:00.703+0000 7fbcd9400640 1 --2- 172.21.15.39:0/2098170285 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fbcc80a2880 0x7fbcc80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:00.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:00.703+0000 7fbcd9400640 1 --2- 172.21.15.39:0/2098170285 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fbcc80a2880 0x7fbcc80a4d40 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fbccc009830 tx=0x7fbccc034ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:00.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:00.706+0000 7fbcdb400640 1 -- 172.21.15.39:0/2098170285 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fbcd409bce0 con 0x7fbce41015d0 2024-03-18T16:09:00.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:00.911+0000 7fbce3200640 1 -- 172.21.15.39:0/2098170285 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbce41a2390 con 0x7fbce41015d0 2024-03-18T16:09:00.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:00.913+0000 7fbcdb400640 1 -- 172.21.15.39:0/2098170285 <== mon.2 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24762 (secure 0 0 0) 0x7fbcd409b5e0 con 0x7fbce41015d0 2024-03-18T16:09:00.915 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:09:00.915 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":93,"fsid":"1892d8fc-e540-11ee-95c9-87774f69a715","created":"2024-03-18T15:57:36.507008+0000","modified":"2024-03-18T16:08:47.874967+0000","last_up_change":"2024-03-18T16:05:45.250629+0000","last_in_change":"2024-03-18T16:05:24.841337+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T16:01:30.048247+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T16:06:42.191791+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T16:06:43.670325+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T16:06:47.523388+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"771e0558-be62-4436-94b0-6364751911c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6803","nonce":4090743635}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6805","nonce":4090743635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6809","nonce":4090743635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6807","nonce":4090743635}]},"public_addr":"172.21.15.39:6803/4090743635","cluster_addr":"172.21.15.39:6805/4090743635","heartbeat_back_addr":"172.21.15.39:6809/4090743635","heartbeat_front_addr":"172.21.15.39:6807/4090743635","state":["exists","up"]},{"osd":1,"uuid":"0430eee1-cf34-4277-b255-c7d9f07ad19a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6801","nonce":2800329805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6803","nonce":2800329805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6807","nonce":2800329805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6805","nonce":2800329805}]},"public_addr":"172.21.15.194:6801/2800329805","cluster_addr":"172.21.15.194:6803/2800329805","heartbeat_back_addr":"172.21.15.194:6807/2800329805","heartbeat_front_addr":"172.21.15.194:6805/2800329805","state":["exists","up"]},{"osd":2,"uuid":"3a718da4-2511-4ec1-b1a8-eafd6cce0df0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6801","nonce":3935874719}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6803","nonce":3935874719}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6807","nonce":3935874719}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6805","nonce":3935874719}]},"public_addr":"172.21.15.196:6801/3935874719","cluster_addr":"172.21.15.196:6803/3935874719","heartbeat_back_addr":"172.21.15.196:6807/3935874719","heartbeat_front_addr":"172.21.15.196:6805/3935874719","state":["exists","up"]},{"osd":3,"uuid":"6f9c37f9-0ab9-4038-8919-afd4b39c4165","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6811","nonce":893345939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6813","nonce":893345939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6817","nonce":893345939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6815","nonce":893345939}]},"public_addr":"172.21.15.39:6811/893345939","cluster_addr":"172.21.15.39:6813/893345939","heartbeat_back_addr":"172.21.15.39:6817/893345939","heartbeat_front_addr":"172.21.15.39:6815/893345939","state":["exists","up"]},{"osd":4,"uuid":"e795bb0e-4791-4537-acda-1a321e19bc9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6809","nonce":800423088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6811","nonce":800423088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6815","nonce":800423088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6813","nonce":800423088}]},"public_addr":"172.21.15.194:6809/800423088","cluster_addr":"172.21.15.194:6811/800423088","heartbeat_back_addr":"172.21.15.194:6815/800423088","heartbeat_front_addr":"172.21.15.194:6813/800423088","state":["exists","up"]},{"osd":5,"uuid":"026e175e-6acb-4b4c-b235-8ccee8ae1149","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6809","nonce":2746224525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6811","nonce":2746224525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6815","nonce":2746224525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6813","nonce":2746224525}]},"public_addr":"172.21.15.196:6809/2746224525","cluster_addr":"172.21.15.196:6811/2746224525","heartbeat_back_addr":"172.21.15.196:6815/2746224525","heartbeat_front_addr":"172.21.15.196:6813/2746224525","state":["exists","up"]},{"osd":6,"uuid":"c1b183ff-383f-4c5c-9075-f76bab1c0bb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6819","nonce":1213371331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6821","nonce":1213371331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6825","nonce":1213371331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6823","nonce":1213371331}]},"public_addr":"172.21.15.39:6819/1213371331","cluster_addr":"172.21.15.39:6821/1213371331","heartbeat_back_addr":"172.21.15.39:6825/1213371331","heartbeat_front_addr":"172.21.15.39:6823/1213371331","state":["exists","up"]},{"osd":7,"uuid":"612f8697-957d-4ef4-a87f-241b36f9d13d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6817","nonce":3299162444}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6819","nonce":3299162444}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6823","nonce":3299162444}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6821","nonce":3299162444}]},"public_addr":"172.21.15.194:6817/3299162444","cluster_addr":"172.21.15.194:6819/3299162444","heartbeat_back_addr":"172.21.15.194:6823/3299162444","heartbeat_front_addr":"172.21.15.194:6821/3299162444","state":["exists","up"]},{"osd":8,"uuid":"836a9107-4d4f-4594-b8dc-0301e64f59d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6817","nonce":2456143145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6819","nonce":2456143145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6823","nonce":2456143145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6821","nonce":2456143145}]},"public_addr":"172.21.15.196:6817/2456143145","cluster_addr":"172.21.15.196:6819/2456143145","heartbeat_back_addr":"172.21.15.196:6823/2456143145","heartbeat_front_addr":"172.21.15.196:6821/2456143145","state":["exists","up"]},{"osd":9,"uuid":"4f30b960-12e3-4b16-bd28-fcc988139b58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6827","nonce":1848634433}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6829","nonce":1848634433}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6833","nonce":1848634433}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6831","nonce":1848634433}]},"public_addr":"172.21.15.39:6827/1848634433","cluster_addr":"172.21.15.39:6829/1848634433","heartbeat_back_addr":"172.21.15.39:6833/1848634433","heartbeat_front_addr":"172.21.15.39:6831/1848634433","state":["exists","up"]},{"osd":10,"uuid":"620f90e8-23e6-4bc0-8269-7219b9e1aebd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6825","nonce":3725064809}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6827","nonce":3725064809}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6831","nonce":3725064809}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6829","nonce":3725064809}]},"public_addr":"172.21.15.194:6825/3725064809","cluster_addr":"172.21.15.194:6827/3725064809","heartbeat_back_addr":"172.21.15.194:6831/3725064809","heartbeat_front_addr":"172.21.15.194:6829/3725064809","state":["exists","up"]},{"osd":11,"uuid":"867a2535-8235-4ceb-a698-eb86869d6d20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6825","nonce":2501212422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6827","nonce":2501212422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6831","nonce":2501212422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6829","nonce":2501212422}]},"public_addr":"172.21.15.196:6825/2501212422","cluster_addr":"172.21.15.196:6827/2501212422","heartbeat_back_addr":"172.21.15.196:6831/2501212422","heartbeat_front_addr":"172.21.15.196:6829/2501212422","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:00:36.931566+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:01:02.614024+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:01:26.193265+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:01:58.308833+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:02:25.049546+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:02:49.748750+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:03:23.582153+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:03:50.447321+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:04:15.812129+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:04:49.998047+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:05:17.087924+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:05:43.167637+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":4,"to":7}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.a","mappings":[{"from":5,"to":8}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.39:0/3482086140":"2024-03-19T15:58:45.340294+0000","172.21.15.39:0/3844306284":"2024-03-19T15:58:45.340294+0000","172.21.15.39:6801/1810238546":"2024-03-19T15:58:45.340294+0000","172.21.15.39:6800/1810238546":"2024-03-19T15:58:45.340294+0000","172.21.15.39:0/2528397488":"2024-03-19T15:58:10.264798+0000","172.21.15.39:0/2240981254":"2024-03-19T15:58:10.264798+0000","172.21.15.39:6800/1217503189":"2024-03-19T15:58:10.264798+0000","172.21.15.39:0/3754876370":"2024-03-19T15:58:10.264798+0000","172.21.15.39:6801/1217503189":"2024-03-19T15:58:10.264798+0000","172.21.15.39:0/1874637628":"2024-03-19T15:58:45.340294+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T16:09:00.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:00.916+0000 7fbce3200640 1 -- 172.21.15.39:0/2098170285 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fbcc80a2880 msgr2=0x7fbcc80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:00.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:00.916+0000 7fbce3200640 1 --2- 172.21.15.39:0/2098170285 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fbcc80a2880 0x7fbcc80a4d40 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fbccc009830 tx=0x7fbccc034ab0 comp rx=0 tx=0).stop 2024-03-18T16:09:00.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:00.916+0000 7fbce3200640 1 -- 172.21.15.39:0/2098170285 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbce41015d0 msgr2=0x7fbce4199ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:00.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:00.916+0000 7fbce3200640 1 --2- 172.21.15.39:0/2098170285 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbce41015d0 0x7fbce4199ab0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fbcd400e9f0 tx=0x7fbcd400eeb0 comp rx=0 tx=0).stop 2024-03-18T16:09:00.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:00.916+0000 7fbce3200640 1 -- 172.21.15.39:0/2098170285 shutdown_connections 2024-03-18T16:09:00.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:00.916+0000 7fbce3200640 1 --2- 172.21.15.39:0/2098170285 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fbcc80a2880 0x7fbcc80a4d40 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:00.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:00.916+0000 7fbce3200640 1 --2- 172.21.15.39:0/2098170285 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbce4103180 0x7fbce41a1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:00.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:00.916+0000 7fbce3200640 1 --2- 172.21.15.39:0/2098170285 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbce41027c0 0x7fbce4199ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:00.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:00.916+0000 7fbce3200640 1 --2- 172.21.15.39:0/2098170285 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbce41015d0 0x7fbce4199ab0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:00.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:00.916+0000 7fbce3200640 1 -- 172.21.15.39:0/2098170285 >> 172.21.15.39:0/2098170285 conn(0x7fbce40fcb40 msgr2=0x7fbce4073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:00.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:00.916+0000 7fbce3200640 1 -- 172.21.15.39:0/2098170285 shutdown_connections 2024-03-18T16:09:00.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:00.916+0000 7fbce3200640 1 -- 172.21.15.39:0/2098170285 wait complete. 2024-03-18T16:09:00.940 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T16:01:30.048247+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T16:06:42.191791+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T16:06:43.670325+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T16:06:47.523388+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T16:09:00.940 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T16:09:01.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.150+0000 7fbbed600640 1 -- 172.21.15.39:0/3535926106 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbbe81010d0 msgr2=0x7fbbe8101950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:01.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.150+0000 7fbbed600640 1 --2- 172.21.15.39:0/3535926106 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbbe81010d0 0x7fbbe8101950 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fbbe0009900 tx=0x7fbbe002bc30 comp rx=0 tx=0).stop 2024-03-18T16:09:01.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.150+0000 7fbbed600640 1 -- 172.21.15.39:0/3535926106 shutdown_connections 2024-03-18T16:09:01.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.150+0000 7fbbed600640 1 --2- 172.21.15.39:0/3535926106 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbbe81010d0 0x7fbbe8101950 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:01.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.150+0000 7fbbed600640 1 --2- 172.21.15.39:0/3535926106 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbbe8100710 0x7fbbe8100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:01.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.150+0000 7fbbed600640 1 --2- 172.21.15.39:0/3535926106 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbbe80ff9b0 0x7fbbe80ffdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:01.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.150+0000 7fbbed600640 1 -- 172.21.15.39:0/3535926106 >> 172.21.15.39:0/3535926106 conn(0x7fbbe80fb2e0 msgr2=0x7fbbe80fd720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:01.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.150+0000 7fbbed600640 1 -- 172.21.15.39:0/3535926106 shutdown_connections 2024-03-18T16:09:01.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.150+0000 7fbbed600640 1 -- 172.21.15.39:0/3535926106 wait complete. 2024-03-18T16:09:01.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.151+0000 7fbbed600640 1 Processor -- start 2024-03-18T16:09:01.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.151+0000 7fbbed600640 1 -- start start 2024-03-18T16:09:01.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.152+0000 7fbbed600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbbe80ff9b0 0x7fbbe8199b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:01.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.152+0000 7fbbed600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbbe8100710 0x7fbbe819a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:01.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.152+0000 7fbbed600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbbe81010d0 0x7fbbe81a10f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:01.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.152+0000 7fbbed600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fbbe810f5c0 con 0x7fbbe81010d0 2024-03-18T16:09:01.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.152+0000 7fbbed600640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fbbe810f440 con 0x7fbbe8100710 2024-03-18T16:09:01.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.152+0000 7fbbed600640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fbbe810f740 con 0x7fbbe80ff9b0 2024-03-18T16:09:01.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.152+0000 7fbbe7e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbbe80ff9b0 0x7fbbe8199b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:01.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.152+0000 7fbbe7e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbbe80ff9b0 0x7fbbe8199b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:35972/0 (socket says 172.21.15.39:35972) 2024-03-18T16:09:01.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.152+0000 7fbbe7e00640 1 -- 172.21.15.39:0/7483339 learned_addr learned my addr 172.21.15.39:0/7483339 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:01.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.152+0000 7fbbdf400640 1 --2- 172.21.15.39:0/7483339 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbbe8100710 0x7fbbe819a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:01.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.152+0000 7fbbecc00640 1 --2- 172.21.15.39:0/7483339 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbbe81010d0 0x7fbbe81a10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:01.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.152+0000 7fbbe7e00640 1 -- 172.21.15.39:0/7483339 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbbe8100710 msgr2=0x7fbbe819a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:01.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.152+0000 7fbbe7e00640 1 --2- 172.21.15.39:0/7483339 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbbe8100710 0x7fbbe819a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:01.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.152+0000 7fbbe7e00640 1 -- 172.21.15.39:0/7483339 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbbe81010d0 msgr2=0x7fbbe81a10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:01.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.152+0000 7fbbe7e00640 1 --2- 172.21.15.39:0/7483339 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbbe81010d0 0x7fbbe81a10f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:01.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.152+0000 7fbbe7e00640 1 -- 172.21.15.39:0/7483339 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbe81a1760 con 0x7fbbe80ff9b0 2024-03-18T16:09:01.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.153+0000 7fbbe7e00640 1 --2- 172.21.15.39:0/7483339 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbbe80ff9b0 0x7fbbe8199b40 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fbbd800d950 tx=0x7fbbd800de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:01.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.153+0000 7fbbe5600640 1 -- 172.21.15.39:0/7483339 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbd8017070 con 0x7fbbe80ff9b0 2024-03-18T16:09:01.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.153+0000 7fbbe5600640 1 -- 172.21.15.39:0/7483339 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbbd8009c30 con 0x7fbbe80ff9b0 2024-03-18T16:09:01.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.153+0000 7fbbed600640 1 -- 172.21.15.39:0/7483339 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbe81a1a50 con 0x7fbbe80ff9b0 2024-03-18T16:09:01.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.153+0000 7fbbe5600640 1 -- 172.21.15.39:0/7483339 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbd800f6b0 con 0x7fbbe80ff9b0 2024-03-18T16:09:01.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.153+0000 7fbbed600640 1 -- 172.21.15.39:0/7483339 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbe81a1ff0 con 0x7fbbe80ff9b0 2024-03-18T16:09:01.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.154+0000 7fbbed600640 1 -- 172.21.15.39:0/7483339 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbbe8068940 con 0x7fbbe80ff9b0 2024-03-18T16:09:01.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.156+0000 7fbbe5600640 1 -- 172.21.15.39:0/7483339 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fbbd80109d0 con 0x7fbbe80ff9b0 2024-03-18T16:09:01.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.157+0000 7fbbe5600640 1 --2- 172.21.15.39:0/7483339 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fbbcc0a2880 0x7fbbcc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:01.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.158+0000 7fbbe5600640 1 -- 172.21.15.39:0/7483339 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fbbd80ce990 con 0x7fbbe80ff9b0 2024-03-18T16:09:01.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.158+0000 7fbbdf400640 1 --2- 172.21.15.39:0/7483339 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fbbcc0a2880 0x7fbbcc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:01.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.159+0000 7fbbdf400640 1 --2- 172.21.15.39:0/7483339 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fbbcc0a2880 0x7fbbcc0a4d40 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fbbe819b050 tx=0x7fbbd0009720 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:01.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.166+0000 7fbbe5600640 1 -- 172.21.15.39:0/7483339 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fbbd8009de0 con 0x7fbbe80ff9b0 2024-03-18T16:09:01.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:00 smithi196 ceph-mon[36307]: pgmap v390: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 351 B/s wr, 11 op/s; 7 B/s, 0 keys/s, 0 objects/s recovering 2024-03-18T16:09:01.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:00 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/2098170285' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:09:01.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:00 smithi194 ceph-mon[36200]: pgmap v390: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 351 B/s wr, 11 op/s; 7 B/s, 0 keys/s, 0 objects/s recovering 2024-03-18T16:09:01.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:00 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/2098170285' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:09:01.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:00 smithi039 ceph-mon[31721]: pgmap v390: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 351 B/s wr, 11 op/s; 7 B/s, 0 keys/s, 0 objects/s recovering 2024-03-18T16:09:01.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:00 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/2098170285' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:09:01.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.372+0000 7fbbed600640 1 -- 172.21.15.39:0/7483339 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fbbe80623e0 con 0x7fbbe80ff9b0 2024-03-18T16:09:01.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.372+0000 7fbbe5600640 1 -- 172.21.15.39:0/7483339 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7fbbd809aff0 con 0x7fbbe80ff9b0 2024-03-18T16:09:01.374 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2024-03-18T16:09:01.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.375+0000 7fbbed600640 1 -- 172.21.15.39:0/7483339 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fbbcc0a2880 msgr2=0x7fbbcc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:01.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.375+0000 7fbbed600640 1 --2- 172.21.15.39:0/7483339 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fbbcc0a2880 0x7fbbcc0a4d40 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fbbe819b050 tx=0x7fbbd0009720 comp rx=0 tx=0).stop 2024-03-18T16:09:01.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.375+0000 7fbbed600640 1 -- 172.21.15.39:0/7483339 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbbe80ff9b0 msgr2=0x7fbbe8199b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:01.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.375+0000 7fbbed600640 1 --2- 172.21.15.39:0/7483339 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbbe80ff9b0 0x7fbbe8199b40 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fbbd800d950 tx=0x7fbbd800de10 comp rx=0 tx=0).stop 2024-03-18T16:09:01.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.375+0000 7fbbed600640 1 -- 172.21.15.39:0/7483339 shutdown_connections 2024-03-18T16:09:01.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.375+0000 7fbbed600640 1 --2- 172.21.15.39:0/7483339 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbbe8100710 0x7fbbe819a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:01.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.375+0000 7fbbed600640 1 --2- 172.21.15.39:0/7483339 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fbbcc0a2880 0x7fbbcc0a4d40 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:01.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.375+0000 7fbbed600640 1 --2- 172.21.15.39:0/7483339 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbbe81010d0 0x7fbbe81a10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:01.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.375+0000 7fbbed600640 1 --2- 172.21.15.39:0/7483339 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbbe80ff9b0 0x7fbbe8199b40 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:01.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.375+0000 7fbbed600640 1 -- 172.21.15.39:0/7483339 >> 172.21.15.39:0/7483339 conn(0x7fbbe80fb2e0 msgr2=0x7fbbe80fc930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:01.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.375+0000 7fbbed600640 1 -- 172.21.15.39:0/7483339 shutdown_connections 2024-03-18T16:09:01.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.375+0000 7fbbed600640 1 -- 172.21.15.39:0/7483339 wait complete. 2024-03-18T16:09:01.396 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T16:09:01.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.615+0000 7fdaa7400640 1 -- 172.21.15.39:0/3781919541 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdaa01000e0 msgr2=0x7fdaa0100500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:01.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.615+0000 7fdaa7400640 1 --2- 172.21.15.39:0/3781919541 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdaa01000e0 0x7fdaa0100500 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fda94009900 tx=0x7fda9402bcc0 comp rx=0 tx=0).stop 2024-03-18T16:09:01.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.615+0000 7fdaa7400640 1 -- 172.21.15.39:0/3781919541 shutdown_connections 2024-03-18T16:09:01.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.615+0000 7fdaa7400640 1 --2- 172.21.15.39:0/3781919541 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdaa0100a40 0x7fdaa0100ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:01.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.615+0000 7fdaa7400640 1 --2- 172.21.15.39:0/3781919541 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdaa01013e0 0x7fdaa0101860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:01.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.615+0000 7fdaa7400640 1 --2- 172.21.15.39:0/3781919541 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdaa01000e0 0x7fdaa0100500 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:01.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.615+0000 7fdaa7400640 1 -- 172.21.15.39:0/3781919541 >> 172.21.15.39:0/3781919541 conn(0x7fdaa00fbc60 msgr2=0x7fdaa00fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:01.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.615+0000 7fdaa7400640 1 -- 172.21.15.39:0/3781919541 shutdown_connections 2024-03-18T16:09:01.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.615+0000 7fdaa7400640 1 -- 172.21.15.39:0/3781919541 wait complete. 2024-03-18T16:09:01.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.616+0000 7fdaa7400640 1 Processor -- start 2024-03-18T16:09:01.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.616+0000 7fdaa7400640 1 -- start start 2024-03-18T16:09:01.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.617+0000 7fdaa7400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdaa01000e0 0x7fdaa0199d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:01.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.617+0000 7fdaa7400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdaa0100a40 0x7fdaa019a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:01.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.617+0000 7fdaa7400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdaa01013e0 0x7fdaa01a12a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:01.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.617+0000 7fdaa7400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fdaa010f690 con 0x7fdaa0100a40 2024-03-18T16:09:01.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.617+0000 7fdaa7400640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fdaa010f510 con 0x7fdaa01000e0 2024-03-18T16:09:01.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.617+0000 7fdaa7400640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fdaa010f810 con 0x7fdaa01013e0 2024-03-18T16:09:01.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.617+0000 7fdaa4a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdaa01000e0 0x7fdaa0199d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:01.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.617+0000 7fdaa4a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdaa01000e0 0x7fdaa0199d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:35100/0 (socket says 172.21.15.39:35100) 2024-03-18T16:09:01.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.617+0000 7fdaa4a00640 1 -- 172.21.15.39:0/56223718 learned_addr learned my addr 172.21.15.39:0/56223718 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:01.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.617+0000 7fdaa5400640 1 --2- 172.21.15.39:0/56223718 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdaa01013e0 0x7fdaa01a12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:01.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.617+0000 7fda9fe00640 1 --2- 172.21.15.39:0/56223718 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdaa0100a40 0x7fdaa019a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:01.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.617+0000 7fdaa4a00640 1 -- 172.21.15.39:0/56223718 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdaa01013e0 msgr2=0x7fdaa01a12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:01.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.617+0000 7fdaa4a00640 1 --2- 172.21.15.39:0/56223718 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdaa01013e0 0x7fdaa01a12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:01.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.617+0000 7fdaa4a00640 1 -- 172.21.15.39:0/56223718 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdaa0100a40 msgr2=0x7fdaa019a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:01.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.617+0000 7fdaa4a00640 1 --2- 172.21.15.39:0/56223718 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdaa0100a40 0x7fdaa019a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:01.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.617+0000 7fdaa4a00640 1 -- 172.21.15.39:0/56223718 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdaa01a1940 con 0x7fdaa01000e0 2024-03-18T16:09:01.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.617+0000 7fdaa5400640 1 --2- 172.21.15.39:0/56223718 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdaa01013e0 0x7fdaa01a12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:09:01.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.618+0000 7fdaa4a00640 1 --2- 172.21.15.39:0/56223718 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdaa01000e0 0x7fdaa0199d30 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fda94005ab0 tx=0x7fda9402c750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:01.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.618+0000 7fda9d600640 1 -- 172.21.15.39:0/56223718 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda94043020 con 0x7fdaa01000e0 2024-03-18T16:09:01.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.618+0000 7fdaa7400640 1 -- 172.21.15.39:0/56223718 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdaa01a1bd0 con 0x7fdaa01000e0 2024-03-18T16:09:01.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.618+0000 7fda9d600640 1 -- 172.21.15.39:0/56223718 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fda940054e0 con 0x7fdaa01000e0 2024-03-18T16:09:01.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.618+0000 7fda9d600640 1 -- 172.21.15.39:0/56223718 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda94003ca0 con 0x7fdaa01000e0 2024-03-18T16:09:01.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.618+0000 7fdaa7400640 1 -- 172.21.15.39:0/56223718 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdaa01a2120 con 0x7fdaa01000e0 2024-03-18T16:09:01.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.619+0000 7fdaa7400640 1 -- 172.21.15.39:0/56223718 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdaa0068a20 con 0x7fdaa01000e0 2024-03-18T16:09:01.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.621+0000 7fda9d600640 1 -- 172.21.15.39:0/56223718 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fda94046020 con 0x7fdaa01000e0 2024-03-18T16:09:01.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.622+0000 7fda9d600640 1 --2- 172.21.15.39:0/56223718 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fda880a2800 0x7fda880a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:01.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.623+0000 7fda9d600640 1 -- 172.21.15.39:0/56223718 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fda940f1100 con 0x7fdaa01000e0 2024-03-18T16:09:01.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.623+0000 7fda9fe00640 1 --2- 172.21.15.39:0/56223718 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fda880a2800 0x7fda880a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:01.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.623+0000 7fda9fe00640 1 --2- 172.21.15.39:0/56223718 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fda880a2800 0x7fda880a4cc0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fda8c0095f0 tx=0x7fda8c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:01.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.626+0000 7fda9d600640 1 -- 172.21.15.39:0/56223718 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fda9404d020 con 0x7fdaa01000e0 2024-03-18T16:09:01.868 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.867+0000 7fdaa7400640 1 -- 172.21.15.39:0/56223718 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fdaa00623e0 con 0x7fdaa01000e0 2024-03-18T16:09:01.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.867+0000 7fda9d600640 1 -- 172.21.15.39:0/56223718 <== mon.2 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7fda940bded0 con 0x7fdaa01000e0 2024-03-18T16:09:01.869 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 32 2024-03-18T16:09:01.873 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.871+0000 7fdaa7400640 1 -- 172.21.15.39:0/56223718 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fda880a2800 msgr2=0x7fda880a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:01.873 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.871+0000 7fdaa7400640 1 --2- 172.21.15.39:0/56223718 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fda880a2800 0x7fda880a4cc0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fda8c0095f0 tx=0x7fda8c005f90 comp rx=0 tx=0).stop 2024-03-18T16:09:01.873 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.871+0000 7fdaa7400640 1 -- 172.21.15.39:0/56223718 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdaa01000e0 msgr2=0x7fdaa0199d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:01.873 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.871+0000 7fdaa7400640 1 --2- 172.21.15.39:0/56223718 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdaa01000e0 0x7fdaa0199d30 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fda94005ab0 tx=0x7fda9402c750 comp rx=0 tx=0).stop 2024-03-18T16:09:01.873 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.871+0000 7fdaa7400640 1 -- 172.21.15.39:0/56223718 shutdown_connections 2024-03-18T16:09:01.873 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.871+0000 7fdaa7400640 1 --2- 172.21.15.39:0/56223718 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fda880a2800 0x7fda880a4cc0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:01.873 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.871+0000 7fdaa7400640 1 --2- 172.21.15.39:0/56223718 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdaa01013e0 0x7fdaa01a12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:01.873 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.871+0000 7fdaa7400640 1 --2- 172.21.15.39:0/56223718 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdaa0100a40 0x7fdaa019a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:01.873 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.871+0000 7fdaa7400640 1 --2- 172.21.15.39:0/56223718 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdaa01000e0 0x7fdaa0199d30 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:01.873 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.871+0000 7fdaa7400640 1 -- 172.21.15.39:0/56223718 >> 172.21.15.39:0/56223718 conn(0x7fdaa00fbc60 msgr2=0x7fdaa00fd3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:01.873 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.871+0000 7fdaa7400640 1 -- 172.21.15.39:0/56223718 shutdown_connections 2024-03-18T16:09:01.873 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:01.871+0000 7fdaa7400640 1 -- 172.21.15.39:0/56223718 wait complete. 2024-03-18T16:09:01.892 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T16:09:02.124 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.122+0000 7ff019200640 1 -- 172.21.15.39:0/3907659893 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff0141016e0 msgr2=0x7ff014101b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:02.124 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.122+0000 7ff019200640 1 --2- 172.21.15.39:0/3907659893 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff0141016e0 0x7ff014101b00 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ff004009900 tx=0x7ff00402bc40 comp rx=0 tx=0).stop 2024-03-18T16:09:02.124 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.122+0000 7ff019200640 1 -- 172.21.15.39:0/3907659893 shutdown_connections 2024-03-18T16:09:02.124 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.122+0000 7ff019200640 1 --2- 172.21.15.39:0/3907659893 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff014103290 0x7ff014109b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:02.124 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.122+0000 7ff019200640 1 --2- 172.21.15.39:0/3907659893 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff0141028d0 0x7ff014102d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:02.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.122+0000 7ff019200640 1 --2- 172.21.15.39:0/3907659893 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff0141016e0 0x7ff014101b00 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:02.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.122+0000 7ff019200640 1 -- 172.21.15.39:0/3907659893 >> 172.21.15.39:0/3907659893 conn(0x7ff0140fcc50 msgr2=0x7ff0140ff090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:02.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.122+0000 7ff019200640 1 -- 172.21.15.39:0/3907659893 shutdown_connections 2024-03-18T16:09:02.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.122+0000 7ff019200640 1 -- 172.21.15.39:0/3907659893 wait complete. 2024-03-18T16:09:02.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.124+0000 7ff019200640 1 Processor -- start 2024-03-18T16:09:02.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.124+0000 7ff019200640 1 -- start start 2024-03-18T16:09:02.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.124+0000 7ff019200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff0141016e0 0x7ff014199c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:02.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.125+0000 7ff019200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff0141028d0 0x7ff01419a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:02.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.125+0000 7ff019200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff014103290 0x7ff0141a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:02.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.125+0000 7ff019200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7ff01410d380 con 0x7ff0141016e0 2024-03-18T16:09:02.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.125+0000 7ff019200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ff01410d200 con 0x7ff014103290 2024-03-18T16:09:02.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.125+0000 7ff019200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7ff01410d500 con 0x7ff0141028d0 2024-03-18T16:09:02.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.125+0000 7ff013e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff014103290 0x7ff0141a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:02.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.125+0000 7ff012a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff0141028d0 0x7ff01419a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:02.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.125+0000 7ff013400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff0141016e0 0x7ff014199c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:02.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.125+0000 7ff013e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff014103290 0x7ff0141a11a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:35124/0 (socket says 172.21.15.39:35124) 2024-03-18T16:09:02.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.125+0000 7ff013e00640 1 -- 172.21.15.39:0/310943248 learned_addr learned my addr 172.21.15.39:0/310943248 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:02.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.125+0000 7ff013e00640 1 -- 172.21.15.39:0/310943248 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff0141028d0 msgr2=0x7ff01419a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:02.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.125+0000 7ff013e00640 1 --2- 172.21.15.39:0/310943248 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff0141028d0 0x7ff01419a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:02.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.125+0000 7ff013e00640 1 -- 172.21.15.39:0/310943248 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff0141016e0 msgr2=0x7ff014199c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:02.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.125+0000 7ff013e00640 1 --2- 172.21.15.39:0/310943248 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff0141016e0 0x7ff014199c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:02.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.125+0000 7ff013e00640 1 -- 172.21.15.39:0/310943248 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0141a1840 con 0x7ff014103290 2024-03-18T16:09:02.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.125+0000 7ff012a00640 1 --2- 172.21.15.39:0/310943248 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff0141028d0 0x7ff01419a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:09:02.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.126+0000 7ff013e00640 1 --2- 172.21.15.39:0/310943248 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff014103290 0x7ff0141a11a0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7ff00c00eb10 tx=0x7ff00c00c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:02.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.126+0000 7ff00be00640 1 -- 172.21.15.39:0/310943248 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff00c004b20 con 0x7ff014103290 2024-03-18T16:09:02.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.126+0000 7ff00be00640 1 -- 172.21.15.39:0/310943248 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff00c004cc0 con 0x7ff014103290 2024-03-18T16:09:02.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.126+0000 7ff019200640 1 -- 172.21.15.39:0/310943248 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0141a1b30 con 0x7ff014103290 2024-03-18T16:09:02.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.126+0000 7ff00be00640 1 -- 172.21.15.39:0/310943248 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff00c010760 con 0x7ff014103290 2024-03-18T16:09:02.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.126+0000 7ff019200640 1 -- 172.21.15.39:0/310943248 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0141a2000 con 0x7ff014103290 2024-03-18T16:09:02.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.127+0000 7ff019200640 1 -- 172.21.15.39:0/310943248 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0141a2230 con 0x7ff014103290 2024-03-18T16:09:02.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.129+0000 7ff00be00640 1 -- 172.21.15.39:0/310943248 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7ff00c005020 con 0x7ff014103290 2024-03-18T16:09:02.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.131+0000 7ff00be00640 1 --2- 172.21.15.39:0/310943248 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7feff80a2800 0x7feff80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:02.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.132+0000 7ff00be00640 1 -- 172.21.15.39:0/310943248 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7ff00c0cf3e0 con 0x7ff014103290 2024-03-18T16:09:02.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.132+0000 7ff013400640 1 --2- 172.21.15.39:0/310943248 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7feff80a2800 0x7feff80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:02.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.132+0000 7ff013400640 1 --2- 172.21.15.39:0/310943248 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7feff80a2800 0x7feff80a4cc0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7ff00402c180 tx=0x7ff00402cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:02.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.139+0000 7ff00be00640 1 -- 172.21.15.39:0/310943248 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff00c09c090 con 0x7ff014103290 2024-03-18T16:09:02.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:01 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/7483339' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T16:09:02.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:01 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/56223718' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T16:09:02.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:01 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/7483339' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T16:09:02.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:01 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/56223718' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T16:09:02.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:01 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/7483339' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T16:09:02.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:01 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/56223718' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T16:09:02.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.409+0000 7ff019200640 1 -- 172.21.15.39:0/310943248 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7ff0141a2400 con 0x7ff014103290 2024-03-18T16:09:02.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.409+0000 7ff00be00640 1 -- 172.21.15.39:0/310943248 <== mon.2 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7ff00c09b990 con 0x7ff014103290 2024-03-18T16:09:02.411 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 64 2024-03-18T16:09:02.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.412+0000 7ff019200640 1 -- 172.21.15.39:0/310943248 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7feff80a2800 msgr2=0x7feff80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:02.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.412+0000 7ff019200640 1 --2- 172.21.15.39:0/310943248 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7feff80a2800 0x7feff80a4cc0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7ff00402c180 tx=0x7ff00402cf90 comp rx=0 tx=0).stop 2024-03-18T16:09:02.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.412+0000 7ff019200640 1 -- 172.21.15.39:0/310943248 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff014103290 msgr2=0x7ff0141a11a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:02.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.412+0000 7ff019200640 1 --2- 172.21.15.39:0/310943248 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff014103290 0x7ff0141a11a0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7ff00c00eb10 tx=0x7ff00c00c2d0 comp rx=0 tx=0).stop 2024-03-18T16:09:02.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.412+0000 7ff019200640 1 -- 172.21.15.39:0/310943248 shutdown_connections 2024-03-18T16:09:02.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.412+0000 7ff019200640 1 --2- 172.21.15.39:0/310943248 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff014103290 0x7ff0141a11a0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:02.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.412+0000 7ff019200640 1 --2- 172.21.15.39:0/310943248 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7feff80a2800 0x7feff80a4cc0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:02.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.412+0000 7ff019200640 1 --2- 172.21.15.39:0/310943248 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff0141028d0 0x7ff01419a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:02.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.412+0000 7ff019200640 1 --2- 172.21.15.39:0/310943248 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff0141016e0 0x7ff014199c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:02.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.412+0000 7ff019200640 1 -- 172.21.15.39:0/310943248 >> 172.21.15.39:0/310943248 conn(0x7ff0140fcc50 msgr2=0x7ff014073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:02.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.413+0000 7ff019200640 1 -- 172.21.15.39:0/310943248 shutdown_connections 2024-03-18T16:09:02.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.413+0000 7ff019200640 1 -- 172.21.15.39:0/310943248 wait complete. 2024-03-18T16:09:02.435 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T16:09:02.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.656+0000 7f03aac00640 1 -- 172.21.15.39:0/233801150 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f03a4104a70 msgr2=0x7f03a4106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:02.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.656+0000 7f03aac00640 1 --2- 172.21.15.39:0/233801150 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f03a4104a70 0x7f03a4106e60 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f0390009900 tx=0x7f039002bc30 comp rx=0 tx=0).stop 2024-03-18T16:09:02.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.657+0000 7f03aac00640 1 -- 172.21.15.39:0/233801150 shutdown_connections 2024-03-18T16:09:02.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.657+0000 7f03aac00640 1 --2- 172.21.15.39:0/233801150 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f03a4104a70 0x7f03a4106e60 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:02.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.657+0000 7f03aac00640 1 --2- 172.21.15.39:0/233801150 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f03a41073a0 0x7f03a41097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:02.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.657+0000 7f03aac00640 1 --2- 172.21.15.39:0/233801150 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f03a4102140 0x7f03a4104530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:02.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.657+0000 7f03aac00640 1 -- 172.21.15.39:0/233801150 >> 172.21.15.39:0/233801150 conn(0x7f03a40fbb50 msgr2=0x7f03a40fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:02.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.657+0000 7f03aac00640 1 -- 172.21.15.39:0/233801150 shutdown_connections 2024-03-18T16:09:02.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.657+0000 7f03aac00640 1 -- 172.21.15.39:0/233801150 wait complete. 2024-03-18T16:09:02.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.658+0000 7f03aac00640 1 Processor -- start 2024-03-18T16:09:02.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.658+0000 7f03aac00640 1 -- start start 2024-03-18T16:09:02.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.659+0000 7f03aac00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f03a4102140 0x7f03a41979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:02.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.659+0000 7f03aac00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f03a4104a70 0x7f03a4197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:02.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.659+0000 7f03aac00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f03a41073a0 0x7f03a419efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:02.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.659+0000 7f03aac00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f03a410d3a0 con 0x7f03a4102140 2024-03-18T16:09:02.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.659+0000 7f03aac00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f03a410d220 con 0x7f03a41073a0 2024-03-18T16:09:02.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.659+0000 7f03aac00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f03a410d520 con 0x7f03a4104a70 2024-03-18T16:09:02.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.659+0000 7f03a9800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f03a4102140 0x7f03a41979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:02.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.659+0000 7f03a8e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f03a4104a70 0x7f03a4197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:02.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.659+0000 7f03a8e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f03a4104a70 0x7f03a4197f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:36008/0 (socket says 172.21.15.39:36008) 2024-03-18T16:09:02.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.659+0000 7f03a8e00640 1 -- 172.21.15.39:0/2416889769 learned_addr learned my addr 172.21.15.39:0/2416889769 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:02.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.659+0000 7f03aa200640 1 --2- 172.21.15.39:0/2416889769 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f03a41073a0 0x7f03a419efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:02.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.660+0000 7f03a8e00640 1 -- 172.21.15.39:0/2416889769 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f03a41073a0 msgr2=0x7f03a419efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:02.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.660+0000 7f03a8e00640 1 --2- 172.21.15.39:0/2416889769 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f03a41073a0 0x7f03a419efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:02.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.660+0000 7f03a8e00640 1 -- 172.21.15.39:0/2416889769 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f03a4102140 msgr2=0x7f03a41979f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:02.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.660+0000 7f03a8e00640 1 --2- 172.21.15.39:0/2416889769 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f03a4102140 0x7f03a41979f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:02.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.660+0000 7f03a8e00640 1 -- 172.21.15.39:0/2416889769 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03a419f610 con 0x7f03a4104a70 2024-03-18T16:09:02.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.660+0000 7f03aa200640 1 --2- 172.21.15.39:0/2416889769 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f03a41073a0 0x7f03a419efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:09:02.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.660+0000 7f03a8e00640 1 --2- 172.21.15.39:0/2416889769 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f03a4104a70 0x7f03a4197f30 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f0390000c00 tx=0x7f039002c6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:02.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.660+0000 7f03a2000640 1 -- 172.21.15.39:0/2416889769 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0390043020 con 0x7f03a4104a70 2024-03-18T16:09:02.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.660+0000 7f03aac00640 1 -- 172.21.15.39:0/2416889769 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03a419f8a0 con 0x7f03a4104a70 2024-03-18T16:09:02.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.660+0000 7f03a2000640 1 -- 172.21.15.39:0/2416889769 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03900054e0 con 0x7f03a4104a70 2024-03-18T16:09:02.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.660+0000 7f03a2000640 1 -- 172.21.15.39:0/2416889769 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0390003ce0 con 0x7f03a4104a70 2024-03-18T16:09:02.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.660+0000 7f03aac00640 1 -- 172.21.15.39:0/2416889769 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03a419fdc0 con 0x7f03a4104a70 2024-03-18T16:09:02.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.661+0000 7f03aac00640 1 -- 172.21.15.39:0/2416889769 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03a40fd640 con 0x7f03a4104a70 2024-03-18T16:09:02.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.663+0000 7f03a2000640 1 -- 172.21.15.39:0/2416889769 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f0390046020 con 0x7f03a4104a70 2024-03-18T16:09:02.666 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.665+0000 7f03a2000640 1 --2- 172.21.15.39:0/2416889769 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f038c0a2800 0x7f038c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:02.666 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.665+0000 7f03a2000640 1 -- 172.21.15.39:0/2416889769 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f03900f1450 con 0x7f03a4104a70 2024-03-18T16:09:02.666 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.665+0000 7f03a9800640 1 --2- 172.21.15.39:0/2416889769 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f038c0a2800 0x7f038c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:02.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.666+0000 7f03a9800640 1 --2- 172.21.15.39:0/2416889769 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f038c0a2800 0x7f038c0a4cc0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f0398009bd0 tx=0x7f0398005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:02.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.667+0000 7f03a2000640 1 -- 172.21.15.39:0/2416889769 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f039004d020 con 0x7f03a4104a70 2024-03-18T16:09:02.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.997+0000 7f03aac00640 1 -- 172.21.15.39:0/2416889769 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f03a402d340 con 0x7f03a4104a70 2024-03-18T16:09:02.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:02.998+0000 7f03a2000640 1 -- 172.21.15.39:0/2416889769 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7f03900be100 con 0x7f03a4104a70 2024-03-18T16:09:02.999 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 32 2024-03-18T16:09:03.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.001+0000 7f03aac00640 1 -- 172.21.15.39:0/2416889769 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f038c0a2800 msgr2=0x7f038c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:03.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.001+0000 7f03aac00640 1 --2- 172.21.15.39:0/2416889769 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f038c0a2800 0x7f038c0a4cc0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f0398009bd0 tx=0x7f0398005f90 comp rx=0 tx=0).stop 2024-03-18T16:09:03.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.001+0000 7f03aac00640 1 -- 172.21.15.39:0/2416889769 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f03a4104a70 msgr2=0x7f03a4197f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:03.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.001+0000 7f03aac00640 1 --2- 172.21.15.39:0/2416889769 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f03a4104a70 0x7f03a4197f30 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f0390000c00 tx=0x7f039002c6c0 comp rx=0 tx=0).stop 2024-03-18T16:09:03.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.001+0000 7f03aac00640 1 -- 172.21.15.39:0/2416889769 shutdown_connections 2024-03-18T16:09:03.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.001+0000 7f03aac00640 1 --2- 172.21.15.39:0/2416889769 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f03a41073a0 0x7f03a419efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:03.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.001+0000 7f03aac00640 1 --2- 172.21.15.39:0/2416889769 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f038c0a2800 0x7f038c0a4cc0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:03.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.001+0000 7f03aac00640 1 --2- 172.21.15.39:0/2416889769 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f03a4104a70 0x7f03a4197f30 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:03.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.001+0000 7f03aac00640 1 --2- 172.21.15.39:0/2416889769 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f03a4102140 0x7f03a41979f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:03.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.001+0000 7f03aac00640 1 -- 172.21.15.39:0/2416889769 >> 172.21.15.39:0/2416889769 conn(0x7f03a40fbb50 msgr2=0x7f03a4102a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:03.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.001+0000 7f03aac00640 1 -- 172.21.15.39:0/2416889769 shutdown_connections 2024-03-18T16:09:03.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.001+0000 7f03aac00640 1 -- 172.21.15.39:0/2416889769 wait complete. 2024-03-18T16:09:03.023 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:09:03.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:02 smithi196 ceph-mon[36307]: pgmap v391: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s; 7 B/s, 0 keys/s, 0 objects/s recovering 2024-03-18T16:09:03.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:02 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/310943248' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T16:09:03.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.238+0000 7f21b8600640 1 -- 172.21.15.39:0/2190237231 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f21b01018d0 msgr2=0x7f21b0101d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:03.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.238+0000 7f21b8600640 1 --2- 172.21.15.39:0/2190237231 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f21b01018d0 0x7f21b0101d50 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f219c009900 tx=0x7f219c02bc40 comp rx=0 tx=0).stop 2024-03-18T16:09:03.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.238+0000 7f21b8600640 1 -- 172.21.15.39:0/2190237231 shutdown_connections 2024-03-18T16:09:03.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.238+0000 7f21b8600640 1 --2- 172.21.15.39:0/2190237231 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f21b01018d0 0x7f21b0101d50 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:03.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.238+0000 7f21b8600640 1 --2- 172.21.15.39:0/2190237231 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f21b0102290 0x7f21b0108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:03.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.238+0000 7f21b8600640 1 --2- 172.21.15.39:0/2190237231 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f21b01006e0 0x7f21b0100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:03.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.238+0000 7f21b8600640 1 -- 172.21.15.39:0/2190237231 >> 172.21.15.39:0/2190237231 conn(0x7f21b00fbc70 msgr2=0x7f21b00fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:03.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.239+0000 7f21b8600640 1 -- 172.21.15.39:0/2190237231 shutdown_connections 2024-03-18T16:09:03.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.239+0000 7f21b8600640 1 -- 172.21.15.39:0/2190237231 wait complete. 2024-03-18T16:09:03.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:02 smithi194 ceph-mon[36200]: pgmap v391: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s; 7 B/s, 0 keys/s, 0 objects/s recovering 2024-03-18T16:09:03.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:02 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/310943248' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T16:09:03.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.239+0000 7f21b8600640 1 Processor -- start 2024-03-18T16:09:03.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.240+0000 7f21b8600640 1 -- start start 2024-03-18T16:09:03.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.240+0000 7f21b8600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f21b01006e0 0x7f21b0104670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:03.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.240+0000 7f21b8600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f21b01018d0 0x7f21b0102cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:03.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.240+0000 7f21b8600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f21b0102290 0x7f21b01031f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:03.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.240+0000 7f21b8600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f21b010c390 con 0x7f21b0102290 2024-03-18T16:09:03.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.240+0000 7f21b8600640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f21b010c210 con 0x7f21b01018d0 2024-03-18T16:09:03.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.240+0000 7f21b8600640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f21b010c510 con 0x7f21b01006e0 2024-03-18T16:09:03.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.240+0000 7f21b5200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f21b01018d0 0x7f21b0102cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:03.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.240+0000 7f21b5200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f21b01018d0 0x7f21b0102cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:35162/0 (socket says 172.21.15.39:35162) 2024-03-18T16:09:03.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.240+0000 7f21b5200640 1 -- 172.21.15.39:0/1375561407 learned_addr learned my addr 172.21.15.39:0/1375561407 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:03.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.240+0000 7f21b6600640 1 --2- 172.21.15.39:0/1375561407 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f21b0102290 0x7f21b01031f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:03.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.240+0000 7f21b5c00640 1 --2- 172.21.15.39:0/1375561407 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f21b01006e0 0x7f21b0104670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:03.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.241+0000 7f21b5200640 1 -- 172.21.15.39:0/1375561407 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f21b01006e0 msgr2=0x7f21b0104670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:03.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.241+0000 7f21b5200640 1 --2- 172.21.15.39:0/1375561407 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f21b01006e0 0x7f21b0104670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:03.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.241+0000 7f21b5200640 1 -- 172.21.15.39:0/1375561407 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f21b0102290 msgr2=0x7f21b01031f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:03.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.241+0000 7f21b5200640 1 --2- 172.21.15.39:0/1375561407 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f21b0102290 0x7f21b01031f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:03.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.241+0000 7f21b5200640 1 -- 172.21.15.39:0/1375561407 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21b0103a80 con 0x7f21b01018d0 2024-03-18T16:09:03.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.241+0000 7f21b5200640 1 --2- 172.21.15.39:0/1375561407 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f21b01018d0 0x7f21b0102cb0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f219c02c8f0 tx=0x7f219c005d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:03.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.241+0000 7f21ae000640 1 -- 172.21.15.39:0/1375561407 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f219c041070 con 0x7f21b01018d0 2024-03-18T16:09:03.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.241+0000 7f21ae000640 1 -- 172.21.15.39:0/1375561407 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f219c0044d0 con 0x7f21b01018d0 2024-03-18T16:09:03.245 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.241+0000 7f21b8600640 1 -- 172.21.15.39:0/1375561407 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21b01a1bb0 con 0x7f21b01018d0 2024-03-18T16:09:03.245 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.241+0000 7f21ae000640 1 -- 172.21.15.39:0/1375561407 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f219c034440 con 0x7f21b01018d0 2024-03-18T16:09:03.245 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.241+0000 7f21b8600640 1 -- 172.21.15.39:0/1375561407 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21b01a2150 con 0x7f21b01018d0 2024-03-18T16:09:03.245 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.243+0000 7f21b8600640 1 -- 172.21.15.39:0/1375561407 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21b01a2320 con 0x7f21b01018d0 2024-03-18T16:09:03.245 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.244+0000 7f21ae000640 1 -- 172.21.15.39:0/1375561407 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f219c044070 con 0x7f21b01018d0 2024-03-18T16:09:03.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.247+0000 7f21ae000640 1 --2- 172.21.15.39:0/1375561407 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f21980a2880 0x7f21980a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:03.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.247+0000 7f21ae000640 1 -- 172.21.15.39:0/1375561407 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f219c0f0930 con 0x7f21b01018d0 2024-03-18T16:09:03.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.247+0000 7f21b5c00640 1 --2- 172.21.15.39:0/1375561407 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f21980a2880 0x7f21980a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:03.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.248+0000 7f21b5c00640 1 --2- 172.21.15.39:0/1375561407 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f21980a2880 0x7f21980a4d40 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f21a4005fd0 tx=0x7f21a400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:03.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.249+0000 7f21ae000640 1 -- 172.21.15.39:0/1375561407 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f219c0bcf90 con 0x7f21b01018d0 2024-03-18T16:09:03.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:02 smithi039 ceph-mon[31721]: pgmap v391: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s; 7 B/s, 0 keys/s, 0 objects/s recovering 2024-03-18T16:09:03.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:02 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/310943248' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T16:09:03.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.525+0000 7f21b8600640 1 -- 172.21.15.39:0/1375561407 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f21b01a24f0 con 0x7f21b01018d0 2024-03-18T16:09:03.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.526+0000 7f21ae000640 1 -- 172.21.15.39:0/1375561407 <== mon.2 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11932 (secure 0 0 0) 0x7f219c0c1030 con 0x7f21b01018d0 2024-03-18T16:09:03.528 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:09:03.528 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2520226608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2520226608},{"type":"v1","addr":"172.21.15.39:6835","nonce":2520226608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/719341763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":719341763},{"type":"v1","addr":"172.21.15.39:6837","nonce":719341763}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T16:06:44.180425+0000","modified":"2024-03-18T16:07:20.994405+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24575,"mds_1":24556,"mds_2":24535,"mds_3":24524,"mds_4":14463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14445":{"gid":14445,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.39:6839/656513813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":656513813},{"type":"v1","addr":"172.21.15.39:6839","nonce":656513813}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14463":{"gid":14463,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":13,"addr":"172.21.15.39:6841/3515326042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3515326042},{"type":"v1","addr":"172.21.15.39:6841","nonce":3515326042}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6833/931350738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":931350738},{"type":"v1","addr":"172.21.15.194:6833","nonce":931350738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6833/3874209621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":3874209621},{"type":"v1","addr":"172.21.15.196:6833","nonce":3874209621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6835/188385470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":188385470},{"type":"v1","addr":"172.21.15.194:6835","nonce":188385470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6835/3949984309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":3949984309},{"type":"v1","addr":"172.21.15.196:6835","nonce":3949984309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":15,"addr":"172.21.15.194:6837/1301464246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":1301464246},{"type":"v1","addr":"172.21.15.194:6837","nonce":1301464246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":16,"addr":"172.21.15.196:6837/895247251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":895247251},{"type":"v1","addr":"172.21.15.196:6837","nonce":895247251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.196:6839/2001427268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6838","nonce":2001427268},{"type":"v1","addr":"172.21.15.196:6839","nonce":2001427268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24575":{"gid":24575,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.194:6839/274911158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":274911158},{"type":"v1","addr":"172.21.15.194:6839","nonce":274911158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24575,"qdb_cluster":[24556,24535,24575,24524,14463]},"id":1}]} 2024-03-18T16:09:03.528 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 27 2024-03-18T16:09:03.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.529+0000 7f21b8600640 1 -- 172.21.15.39:0/1375561407 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f21980a2880 msgr2=0x7f21980a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:03.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.529+0000 7f21b8600640 1 --2- 172.21.15.39:0/1375561407 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f21980a2880 0x7f21980a4d40 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f21a4005fd0 tx=0x7f21a400b410 comp rx=0 tx=0).stop 2024-03-18T16:09:03.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.529+0000 7f21b8600640 1 -- 172.21.15.39:0/1375561407 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f21b01018d0 msgr2=0x7f21b0102cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:03.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.529+0000 7f21b8600640 1 --2- 172.21.15.39:0/1375561407 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f21b01018d0 0x7f21b0102cb0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f219c02c8f0 tx=0x7f219c005d50 comp rx=0 tx=0).stop 2024-03-18T16:09:03.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.529+0000 7f21b8600640 1 -- 172.21.15.39:0/1375561407 shutdown_connections 2024-03-18T16:09:03.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.529+0000 7f21b8600640 1 --2- 172.21.15.39:0/1375561407 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f21b01018d0 0x7f21b0102cb0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:03.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.529+0000 7f21b8600640 1 --2- 172.21.15.39:0/1375561407 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f21980a2880 0x7f21980a4d40 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:03.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.529+0000 7f21b8600640 1 --2- 172.21.15.39:0/1375561407 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f21b0102290 0x7f21b01031f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:03.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.529+0000 7f21b8600640 1 --2- 172.21.15.39:0/1375561407 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f21b01006e0 0x7f21b0104670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:03.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.529+0000 7f21b8600640 1 -- 172.21.15.39:0/1375561407 >> 172.21.15.39:0/1375561407 conn(0x7f21b00fbc70 msgr2=0x7f21b00fd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:03.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.529+0000 7f21b8600640 1 -- 172.21.15.39:0/1375561407 shutdown_connections 2024-03-18T16:09:03.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.529+0000 7f21b8600640 1 -- 172.21.15.39:0/1375561407 wait complete. 2024-03-18T16:09:03.550 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T16:09:03.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.765+0000 7fee4be00640 1 -- 172.21.15.39:0/3417249387 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fee4c0ffe60 msgr2=0x7fee4c102290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:03.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.766+0000 7fee4be00640 1 --2- 172.21.15.39:0/3417249387 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fee4c0ffe60 0x7fee4c102290 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fee3c009900 tx=0x7fee3c02bc30 comp rx=0 tx=0).stop 2024-03-18T16:09:03.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.766+0000 7fee4be00640 1 -- 172.21.15.39:0/3417249387 shutdown_connections 2024-03-18T16:09:03.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.766+0000 7fee4be00640 1 --2- 172.21.15.39:0/3417249387 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fee4c105180 0x7fee4c105600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:03.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.766+0000 7fee4be00640 1 --2- 172.21.15.39:0/3417249387 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fee4c1027d0 0x7fee4c104c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:03.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.766+0000 7fee4be00640 1 --2- 172.21.15.39:0/3417249387 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fee4c0ffe60 0x7fee4c102290 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:03.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.766+0000 7fee4be00640 1 -- 172.21.15.39:0/3417249387 >> 172.21.15.39:0/3417249387 conn(0x7fee4c0fba60 msgr2=0x7fee4c0fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:03.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.766+0000 7fee4be00640 1 -- 172.21.15.39:0/3417249387 shutdown_connections 2024-03-18T16:09:03.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.766+0000 7fee4be00640 1 -- 172.21.15.39:0/3417249387 wait complete. 2024-03-18T16:09:03.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.767+0000 7fee4be00640 1 Processor -- start 2024-03-18T16:09:03.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.767+0000 7fee4be00640 1 -- start start 2024-03-18T16:09:03.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.767+0000 7fee4be00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fee4c0ffe60 0x7fee4c19df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:03.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.767+0000 7fee4be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fee4c1027d0 0x7fee4c19e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:03.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.767+0000 7fee4be00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fee4c105180 0x7fee4c1a5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:03.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.767+0000 7fee4be00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fee4c113890 con 0x7fee4c1027d0 2024-03-18T16:09:03.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.767+0000 7fee4be00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fee4c113710 con 0x7fee4c0ffe60 2024-03-18T16:09:03.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.767+0000 7fee4be00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fee4c113a10 con 0x7fee4c105180 2024-03-18T16:09:03.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.767+0000 7fee4aa00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fee4c0ffe60 0x7fee4c19df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:03.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.768+0000 7fee4aa00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fee4c0ffe60 0x7fee4c19df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:35190/0 (socket says 172.21.15.39:35190) 2024-03-18T16:09:03.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.768+0000 7fee4a000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fee4c1027d0 0x7fee4c19e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:03.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.768+0000 7fee4aa00640 1 -- 172.21.15.39:0/134203774 learned_addr learned my addr 172.21.15.39:0/134203774 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:03.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.768+0000 7fee4b400640 1 --2- 172.21.15.39:0/134203774 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fee4c105180 0x7fee4c1a5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:03.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.768+0000 7fee4aa00640 1 -- 172.21.15.39:0/134203774 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fee4c105180 msgr2=0x7fee4c1a5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:03.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.768+0000 7fee4aa00640 1 --2- 172.21.15.39:0/134203774 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fee4c105180 0x7fee4c1a5490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:03.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.768+0000 7fee4aa00640 1 -- 172.21.15.39:0/134203774 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fee4c1027d0 msgr2=0x7fee4c19e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:03.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.768+0000 7fee4aa00640 1 --2- 172.21.15.39:0/134203774 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fee4c1027d0 0x7fee4c19e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:03.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.768+0000 7fee4aa00640 1 -- 172.21.15.39:0/134203774 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee4c1a5b00 con 0x7fee4c0ffe60 2024-03-18T16:09:03.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.768+0000 7fee4aa00640 1 --2- 172.21.15.39:0/134203774 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fee4c0ffe60 0x7fee4c19df30 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fee3c000c00 tx=0x7fee3c02c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:03.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.769+0000 7fee43400640 1 -- 172.21.15.39:0/134203774 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee3c043020 con 0x7fee4c0ffe60 2024-03-18T16:09:03.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.769+0000 7fee43400640 1 -- 172.21.15.39:0/134203774 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee3c0054e0 con 0x7fee4c0ffe60 2024-03-18T16:09:03.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.769+0000 7fee4be00640 1 -- 172.21.15.39:0/134203774 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee4c1a5d90 con 0x7fee4c0ffe60 2024-03-18T16:09:03.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.769+0000 7fee43400640 1 -- 172.21.15.39:0/134203774 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee3c003bd0 con 0x7fee4c0ffe60 2024-03-18T16:09:03.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.769+0000 7fee4be00640 1 -- 172.21.15.39:0/134203774 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee4c1a6250 con 0x7fee4c0ffe60 2024-03-18T16:09:03.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.770+0000 7fee4be00640 1 -- 172.21.15.39:0/134203774 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee4c0fd550 con 0x7fee4c0ffe60 2024-03-18T16:09:03.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.772+0000 7fee43400640 1 -- 172.21.15.39:0/134203774 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fee3c046020 con 0x7fee4c0ffe60 2024-03-18T16:09:03.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.773+0000 7fee43400640 1 --2- 172.21.15.39:0/134203774 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fee300a2880 0x7fee300a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:03.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.773+0000 7fee43400640 1 -- 172.21.15.39:0/134203774 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fee3c0f0e30 con 0x7fee4c0ffe60 2024-03-18T16:09:03.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.773+0000 7fee4a000640 1 --2- 172.21.15.39:0/134203774 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fee300a2880 0x7fee300a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:03.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.774+0000 7fee4a000640 1 --2- 172.21.15.39:0/134203774 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fee300a2880 0x7fee300a4d40 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fee4c19f3f0 tx=0x7fee38005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:03.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.777+0000 7fee43400640 1 -- 172.21.15.39:0/134203774 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fee3c04d020 con 0x7fee4c0ffe60 2024-03-18T16:09:03.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.983+0000 7fee4be00640 1 -- 172.21.15.39:0/134203774 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fee4c062520 con 0x7fee4c0ffe60 2024-03-18T16:09:03.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.985+0000 7fee43400640 1 -- 172.21.15.39:0/134203774 <== mon.2 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24762 (secure 0 0 0) 0x7fee3c0bdae0 con 0x7fee4c0ffe60 2024-03-18T16:09:03.987 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:09:03.987 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":93,"fsid":"1892d8fc-e540-11ee-95c9-87774f69a715","created":"2024-03-18T15:57:36.507008+0000","modified":"2024-03-18T16:08:47.874967+0000","last_up_change":"2024-03-18T16:05:45.250629+0000","last_in_change":"2024-03-18T16:05:24.841337+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T16:01:30.048247+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T16:06:42.191791+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T16:06:43.670325+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T16:06:47.523388+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"771e0558-be62-4436-94b0-6364751911c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6803","nonce":4090743635}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6805","nonce":4090743635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6809","nonce":4090743635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6807","nonce":4090743635}]},"public_addr":"172.21.15.39:6803/4090743635","cluster_addr":"172.21.15.39:6805/4090743635","heartbeat_back_addr":"172.21.15.39:6809/4090743635","heartbeat_front_addr":"172.21.15.39:6807/4090743635","state":["exists","up"]},{"osd":1,"uuid":"0430eee1-cf34-4277-b255-c7d9f07ad19a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6801","nonce":2800329805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6803","nonce":2800329805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6807","nonce":2800329805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6805","nonce":2800329805}]},"public_addr":"172.21.15.194:6801/2800329805","cluster_addr":"172.21.15.194:6803/2800329805","heartbeat_back_addr":"172.21.15.194:6807/2800329805","heartbeat_front_addr":"172.21.15.194:6805/2800329805","state":["exists","up"]},{"osd":2,"uuid":"3a718da4-2511-4ec1-b1a8-eafd6cce0df0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6801","nonce":3935874719}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6803","nonce":3935874719}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6807","nonce":3935874719}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6805","nonce":3935874719}]},"public_addr":"172.21.15.196:6801/3935874719","cluster_addr":"172.21.15.196:6803/3935874719","heartbeat_back_addr":"172.21.15.196:6807/3935874719","heartbeat_front_addr":"172.21.15.196:6805/3935874719","state":["exists","up"]},{"osd":3,"uuid":"6f9c37f9-0ab9-4038-8919-afd4b39c4165","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6811","nonce":893345939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6813","nonce":893345939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6817","nonce":893345939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6815","nonce":893345939}]},"public_addr":"172.21.15.39:6811/893345939","cluster_addr":"172.21.15.39:6813/893345939","heartbeat_back_addr":"172.21.15.39:6817/893345939","heartbeat_front_addr":"172.21.15.39:6815/893345939","state":["exists","up"]},{"osd":4,"uuid":"e795bb0e-4791-4537-acda-1a321e19bc9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6809","nonce":800423088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6811","nonce":800423088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6815","nonce":800423088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6813","nonce":800423088}]},"public_addr":"172.21.15.194:6809/800423088","cluster_addr":"172.21.15.194:6811/800423088","heartbeat_back_addr":"172.21.15.194:6815/800423088","heartbeat_front_addr":"172.21.15.194:6813/800423088","state":["exists","up"]},{"osd":5,"uuid":"026e175e-6acb-4b4c-b235-8ccee8ae1149","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6809","nonce":2746224525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6811","nonce":2746224525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6815","nonce":2746224525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6813","nonce":2746224525}]},"public_addr":"172.21.15.196:6809/2746224525","cluster_addr":"172.21.15.196:6811/2746224525","heartbeat_back_addr":"172.21.15.196:6815/2746224525","heartbeat_front_addr":"172.21.15.196:6813/2746224525","state":["exists","up"]},{"osd":6,"uuid":"c1b183ff-383f-4c5c-9075-f76bab1c0bb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6819","nonce":1213371331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6821","nonce":1213371331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6825","nonce":1213371331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6823","nonce":1213371331}]},"public_addr":"172.21.15.39:6819/1213371331","cluster_addr":"172.21.15.39:6821/1213371331","heartbeat_back_addr":"172.21.15.39:6825/1213371331","heartbeat_front_addr":"172.21.15.39:6823/1213371331","state":["exists","up"]},{"osd":7,"uuid":"612f8697-957d-4ef4-a87f-241b36f9d13d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6817","nonce":3299162444}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6819","nonce":3299162444}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6823","nonce":3299162444}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6821","nonce":3299162444}]},"public_addr":"172.21.15.194:6817/3299162444","cluster_addr":"172.21.15.194:6819/3299162444","heartbeat_back_addr":"172.21.15.194:6823/3299162444","heartbeat_front_addr":"172.21.15.194:6821/3299162444","state":["exists","up"]},{"osd":8,"uuid":"836a9107-4d4f-4594-b8dc-0301e64f59d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6817","nonce":2456143145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6819","nonce":2456143145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6823","nonce":2456143145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6821","nonce":2456143145}]},"public_addr":"172.21.15.196:6817/2456143145","cluster_addr":"172.21.15.196:6819/2456143145","heartbeat_back_addr":"172.21.15.196:6823/2456143145","heartbeat_front_addr":"172.21.15.196:6821/2456143145","state":["exists","up"]},{"osd":9,"uuid":"4f30b960-12e3-4b16-bd28-fcc988139b58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6827","nonce":1848634433}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6829","nonce":1848634433}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6833","nonce":1848634433}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6831","nonce":1848634433}]},"public_addr":"172.21.15.39:6827/1848634433","cluster_addr":"172.21.15.39:6829/1848634433","heartbeat_back_addr":"172.21.15.39:6833/1848634433","heartbeat_front_addr":"172.21.15.39:6831/1848634433","state":["exists","up"]},{"osd":10,"uuid":"620f90e8-23e6-4bc0-8269-7219b9e1aebd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6825","nonce":3725064809}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6827","nonce":3725064809}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6831","nonce":3725064809}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6829","nonce":3725064809}]},"public_addr":"172.21.15.194:6825/3725064809","cluster_addr":"172.21.15.194:6827/3725064809","heartbeat_back_addr":"172.21.15.194:6831/3725064809","heartbeat_front_addr":"172.21.15.194:6829/3725064809","state":["exists","up"]},{"osd":11,"uuid":"867a2535-8235-4ceb-a698-eb86869d6d20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6825","nonce":2501212422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6827","nonce":2501212422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6831","nonce":2501212422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6829","nonce":2501212422}]},"public_addr":"172.21.15.196:6825/2501212422","cluster_addr":"172.21.15.196:6827/2501212422","heartbeat_back_addr":"172.21.15.196:6831/2501212422","heartbeat_front_addr":"172.21.15.196:6829/2501212422","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:00:36.931566+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:01:02.614024+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:01:26.193265+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:01:58.308833+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:02:25.049546+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:02:49.748750+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:03:23.582153+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:03:50.447321+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:04:15.812129+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:04:49.998047+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:05:17.087924+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:05:43.167637+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":4,"to":7}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.a","mappings":[{"from":5,"to":8}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.39:0/3482086140":"2024-03-19T15:58:45.340294+0000","172.21.15.39:0/3844306284":"2024-03-19T15:58:45.340294+0000","172.21.15.39:6801/1810238546":"2024-03-19T15:58:45.340294+0000","172.21.15.39:6800/1810238546":"2024-03-19T15:58:45.340294+0000","172.21.15.39:0/2528397488":"2024-03-19T15:58:10.264798+0000","172.21.15.39:0/2240981254":"2024-03-19T15:58:10.264798+0000","172.21.15.39:6800/1217503189":"2024-03-19T15:58:10.264798+0000","172.21.15.39:0/3754876370":"2024-03-19T15:58:10.264798+0000","172.21.15.39:6801/1217503189":"2024-03-19T15:58:10.264798+0000","172.21.15.39:0/1874637628":"2024-03-19T15:58:45.340294+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T16:09:03.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.988+0000 7fee4be00640 1 -- 172.21.15.39:0/134203774 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fee300a2880 msgr2=0x7fee300a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:03.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.988+0000 7fee4be00640 1 --2- 172.21.15.39:0/134203774 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fee300a2880 0x7fee300a4d40 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fee4c19f3f0 tx=0x7fee38005f90 comp rx=0 tx=0).stop 2024-03-18T16:09:03.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.988+0000 7fee4be00640 1 -- 172.21.15.39:0/134203774 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fee4c0ffe60 msgr2=0x7fee4c19df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:03.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.988+0000 7fee4be00640 1 --2- 172.21.15.39:0/134203774 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fee4c0ffe60 0x7fee4c19df30 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fee3c000c00 tx=0x7fee3c02c650 comp rx=0 tx=0).stop 2024-03-18T16:09:03.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.988+0000 7fee4be00640 1 -- 172.21.15.39:0/134203774 shutdown_connections 2024-03-18T16:09:03.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.988+0000 7fee4be00640 1 --2- 172.21.15.39:0/134203774 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fee300a2880 0x7fee300a4d40 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:03.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.988+0000 7fee4be00640 1 --2- 172.21.15.39:0/134203774 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fee4c105180 0x7fee4c1a5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:03.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.988+0000 7fee4be00640 1 --2- 172.21.15.39:0/134203774 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fee4c1027d0 0x7fee4c19e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:03.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.988+0000 7fee4be00640 1 --2- 172.21.15.39:0/134203774 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fee4c0ffe60 0x7fee4c19df30 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:03.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.988+0000 7fee4be00640 1 -- 172.21.15.39:0/134203774 >> 172.21.15.39:0/134203774 conn(0x7fee4c0fba60 msgr2=0x7fee4c103180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:03.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.989+0000 7fee4be00640 1 -- 172.21.15.39:0/134203774 shutdown_connections 2024-03-18T16:09:03.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:03.989+0000 7fee4be00640 1 -- 172.21.15.39:0/134203774 wait complete. 2024-03-18T16:09:04.010 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T16:01:30.048247+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T16:06:42.191791+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T16:06:43.670325+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T16:06:47.523388+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T16:09:04.010 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T16:09:04.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:03 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/2416889769' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T16:09:04.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:03 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/1375561407' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:09:04.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:03 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/2416889769' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T16:09:04.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:03 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/1375561407' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:09:04.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.257+0000 7fcc01600640 1 -- 172.21.15.39:0/1145402901 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcbfc077d80 msgr2=0x7fcbfc078620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:04.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.257+0000 7fcc01600640 1 --2- 172.21.15.39:0/1145402901 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcbfc077d80 0x7fcbfc078620 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fcbf4009ca0 tx=0x7fcbf402c100 comp rx=0 tx=0).stop 2024-03-18T16:09:04.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.257+0000 7fcc01600640 1 -- 172.21.15.39:0/1145402901 shutdown_connections 2024-03-18T16:09:04.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.257+0000 7fcc01600640 1 --2- 172.21.15.39:0/1145402901 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcbfc077d80 0x7fcbfc078620 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:04.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.257+0000 7fcc01600640 1 --2- 172.21.15.39:0/1145402901 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcbfc0773e0 0x7fcbfc077840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:04.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.257+0000 7fcc01600640 1 --2- 172.21.15.39:0/1145402901 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcbfc0761a0 0x7fcbfc0765c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:04.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.257+0000 7fcc01600640 1 -- 172.21.15.39:0/1145402901 >> 172.21.15.39:0/1145402901 conn(0x7fcbfc0fef90 msgr2=0x7fcbfc1013d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:04.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.257+0000 7fcc01600640 1 -- 172.21.15.39:0/1145402901 shutdown_connections 2024-03-18T16:09:04.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.258+0000 7fcc01600640 1 -- 172.21.15.39:0/1145402901 wait complete. 2024-03-18T16:09:04.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.259+0000 7fcc01600640 1 Processor -- start 2024-03-18T16:09:04.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.259+0000 7fcc01600640 1 -- start start 2024-03-18T16:09:04.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.259+0000 7fcc01600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcbfc0761a0 0x7fcbfc19e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:04.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.259+0000 7fcc01600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcbfc0773e0 0x7fcbfc19e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:04.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.260+0000 7fcc01600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcbfc077d80 0x7fcbfc1a5630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:04.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.260+0000 7fcc01600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fcbfc113ab0 con 0x7fcbfc0761a0 2024-03-18T16:09:04.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.260+0000 7fcc01600640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fcbfc113930 con 0x7fcbfc077d80 2024-03-18T16:09:04.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.260+0000 7fcc01600640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fcbfc113c30 con 0x7fcbfc0773e0 2024-03-18T16:09:04.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.260+0000 7fcbf3e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcbfc0773e0 0x7fcbfc19e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:04.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.260+0000 7fcbfb400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcbfc077d80 0x7fcbfc1a5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:04.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.260+0000 7fcbf3e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcbfc0773e0 0x7fcbfc19e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:36060/0 (socket says 172.21.15.39:36060) 2024-03-18T16:09:04.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.260+0000 7fcbf3e00640 1 -- 172.21.15.39:0/639145740 learned_addr learned my addr 172.21.15.39:0/639145740 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:04.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.260+0000 7fcbfb400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcbfc077d80 0x7fcbfc1a5630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:35206/0 (socket says 172.21.15.39:35206) 2024-03-18T16:09:04.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.260+0000 7fcbfaa00640 1 --2- 172.21.15.39:0/639145740 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcbfc0761a0 0x7fcbfc19e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:04.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.260+0000 7fcbfb400640 1 -- 172.21.15.39:0/639145740 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcbfc0773e0 msgr2=0x7fcbfc19e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:04.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.260+0000 7fcbfb400640 1 --2- 172.21.15.39:0/639145740 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcbfc0773e0 0x7fcbfc19e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:04.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.260+0000 7fcbfb400640 1 -- 172.21.15.39:0/639145740 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcbfc0761a0 msgr2=0x7fcbfc19e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:04.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.260+0000 7fcbfb400640 1 --2- 172.21.15.39:0/639145740 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcbfc0761a0 0x7fcbfc19e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:04.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.260+0000 7fcbfb400640 1 -- 172.21.15.39:0/639145740 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbfc1a5ca0 con 0x7fcbfc077d80 2024-03-18T16:09:04.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.260+0000 7fcbf3e00640 1 --2- 172.21.15.39:0/639145740 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcbfc0773e0 0x7fcbfc19e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:09:04.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.261+0000 7fcbfb400640 1 --2- 172.21.15.39:0/639145740 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcbfc077d80 0x7fcbfc1a5630 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fcbf402c640 tx=0x7fcbf4035d90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:04.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.261+0000 7fcbf3400640 1 -- 172.21.15.39:0/639145740 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbf4039040 con 0x7fcbfc077d80 2024-03-18T16:09:04.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.261+0000 7fcbf3400640 1 -- 172.21.15.39:0/639145740 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcbf4044070 con 0x7fcbfc077d80 2024-03-18T16:09:04.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.261+0000 7fcc01600640 1 -- 172.21.15.39:0/639145740 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbfc1a5f30 con 0x7fcbfc077d80 2024-03-18T16:09:04.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.261+0000 7fcbf3400640 1 -- 172.21.15.39:0/639145740 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbf40043e0 con 0x7fcbfc077d80 2024-03-18T16:09:04.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.261+0000 7fcc01600640 1 -- 172.21.15.39:0/639145740 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbfc1a63f0 con 0x7fcbfc077d80 2024-03-18T16:09:04.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.262+0000 7fcc01600640 1 -- 172.21.15.39:0/639145740 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbfc1104d0 con 0x7fcbfc077d80 2024-03-18T16:09:04.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.265+0000 7fcbf3400640 1 -- 172.21.15.39:0/639145740 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fcbf4004580 con 0x7fcbfc077d80 2024-03-18T16:09:04.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.266+0000 7fcbf3400640 1 --2- 172.21.15.39:0/639145740 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fcbe00a2880 0x7fcbe00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:04.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.266+0000 7fcbf3400640 1 -- 172.21.15.39:0/639145740 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fcbf40f1ee0 con 0x7fcbfc077d80 2024-03-18T16:09:04.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.266+0000 7fcbfaa00640 1 --2- 172.21.15.39:0/639145740 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fcbe00a2880 0x7fcbe00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:04.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.267+0000 7fcbfaa00640 1 --2- 172.21.15.39:0/639145740 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fcbe00a2880 0x7fcbe00a4d40 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fcbec009bd0 tx=0x7fcbec005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:04.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.269+0000 7fcbf3400640 1 -- 172.21.15.39:0/639145740 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fcbf40beb90 con 0x7fcbfc077d80 2024-03-18T16:09:04.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:03 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/2416889769' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T16:09:04.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:03 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/1375561407' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:09:04.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.485+0000 7fcc01600640 1 -- 172.21.15.39:0/639145740 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fcbfc02d3b0 con 0x7fcbfc077d80 2024-03-18T16:09:04.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.485+0000 7fcbf3400640 1 -- 172.21.15.39:0/639145740 <== mon.2 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7fcbf40be490 con 0x7fcbfc077d80 2024-03-18T16:09:04.487 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2024-03-18T16:09:04.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.488+0000 7fcc01600640 1 -- 172.21.15.39:0/639145740 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fcbe00a2880 msgr2=0x7fcbe00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:04.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.488+0000 7fcc01600640 1 --2- 172.21.15.39:0/639145740 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fcbe00a2880 0x7fcbe00a4d40 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fcbec009bd0 tx=0x7fcbec005f90 comp rx=0 tx=0).stop 2024-03-18T16:09:04.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.488+0000 7fcc01600640 1 -- 172.21.15.39:0/639145740 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcbfc077d80 msgr2=0x7fcbfc1a5630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:04.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.488+0000 7fcc01600640 1 --2- 172.21.15.39:0/639145740 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcbfc077d80 0x7fcbfc1a5630 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fcbf402c640 tx=0x7fcbf4035d90 comp rx=0 tx=0).stop 2024-03-18T16:09:04.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.488+0000 7fcc01600640 1 -- 172.21.15.39:0/639145740 shutdown_connections 2024-03-18T16:09:04.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.488+0000 7fcc01600640 1 --2- 172.21.15.39:0/639145740 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcbfc077d80 0x7fcbfc1a5630 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:04.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.488+0000 7fcc01600640 1 --2- 172.21.15.39:0/639145740 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fcbe00a2880 0x7fcbe00a4d40 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:04.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.488+0000 7fcc01600640 1 --2- 172.21.15.39:0/639145740 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcbfc0773e0 0x7fcbfc19e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:04.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.488+0000 7fcc01600640 1 --2- 172.21.15.39:0/639145740 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcbfc0761a0 0x7fcbfc19e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:04.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.488+0000 7fcc01600640 1 -- 172.21.15.39:0/639145740 >> 172.21.15.39:0/639145740 conn(0x7fcbfc0fef90 msgr2=0x7fcbfc100610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:04.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.489+0000 7fcc01600640 1 -- 172.21.15.39:0/639145740 shutdown_connections 2024-03-18T16:09:04.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.489+0000 7fcc01600640 1 -- 172.21.15.39:0/639145740 wait complete. 2024-03-18T16:09:04.509 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T16:09:04.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.729+0000 7f4c91000640 1 -- 172.21.15.39:0/799136967 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4c8c1016e0 msgr2=0x7f4c8c101b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:04.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.729+0000 7f4c91000640 1 --2- 172.21.15.39:0/799136967 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4c8c1016e0 0x7f4c8c101b00 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f4c7c009900 tx=0x7f4c7c02bc30 comp rx=0 tx=0).stop 2024-03-18T16:09:04.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.729+0000 7f4c91000640 1 -- 172.21.15.39:0/799136967 shutdown_connections 2024-03-18T16:09:04.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.729+0000 7f4c91000640 1 --2- 172.21.15.39:0/799136967 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c8c103290 0x7f4c8c109b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:04.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.729+0000 7f4c91000640 1 --2- 172.21.15.39:0/799136967 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4c8c1028d0 0x7f4c8c102d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:04.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.729+0000 7f4c91000640 1 --2- 172.21.15.39:0/799136967 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4c8c1016e0 0x7f4c8c101b00 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:04.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.729+0000 7f4c91000640 1 -- 172.21.15.39:0/799136967 >> 172.21.15.39:0/799136967 conn(0x7f4c8c0fcc50 msgr2=0x7f4c8c0ff090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:04.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.729+0000 7f4c91000640 1 -- 172.21.15.39:0/799136967 shutdown_connections 2024-03-18T16:09:04.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.729+0000 7f4c91000640 1 -- 172.21.15.39:0/799136967 wait complete. 2024-03-18T16:09:04.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.730+0000 7f4c91000640 1 Processor -- start 2024-03-18T16:09:04.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.731+0000 7f4c91000640 1 -- start start 2024-03-18T16:09:04.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.731+0000 7f4c91000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4c8c1016e0 0x7f4c8c199c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:04.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.731+0000 7f4c91000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4c8c1028d0 0x7f4c8c19a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:04.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.731+0000 7f4c91000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c8c103290 0x7f4c8c1a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:04.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.731+0000 7f4c91000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f4c8c10d380 con 0x7f4c8c103290 2024-03-18T16:09:04.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.731+0000 7f4c91000640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f4c8c10d200 con 0x7f4c8c1028d0 2024-03-18T16:09:04.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.731+0000 7f4c91000640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f4c8c10d500 con 0x7f4c8c1016e0 2024-03-18T16:09:04.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.731+0000 7f4c8b400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4c8c1016e0 0x7f4c8c199c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:04.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.731+0000 7f4c8b400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4c8c1016e0 0x7f4c8c199c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:36070/0 (socket says 172.21.15.39:36070) 2024-03-18T16:09:04.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.731+0000 7f4c8b400640 1 -- 172.21.15.39:0/1824334873 learned_addr learned my addr 172.21.15.39:0/1824334873 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:04.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.731+0000 7f4c8aa00640 1 --2- 172.21.15.39:0/1824334873 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4c8c1028d0 0x7f4c8c19a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:04.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.731+0000 7f4c8be00640 1 --2- 172.21.15.39:0/1824334873 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c8c103290 0x7f4c8c1a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:04.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.732+0000 7f4c8b400640 1 -- 172.21.15.39:0/1824334873 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4c8c1028d0 msgr2=0x7f4c8c19a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:04.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.732+0000 7f4c8b400640 1 --2- 172.21.15.39:0/1824334873 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4c8c1028d0 0x7f4c8c19a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:04.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.732+0000 7f4c8b400640 1 -- 172.21.15.39:0/1824334873 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c8c103290 msgr2=0x7f4c8c1a11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:04.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.732+0000 7f4c8b400640 1 --2- 172.21.15.39:0/1824334873 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c8c103290 0x7f4c8c1a11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:04.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.732+0000 7f4c8b400640 1 -- 172.21.15.39:0/1824334873 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c8c1a1840 con 0x7f4c8c1016e0 2024-03-18T16:09:04.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.732+0000 7f4c8be00640 1 --2- 172.21.15.39:0/1824334873 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c8c103290 0x7f4c8c1a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:09:04.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.732+0000 7f4c8b400640 1 --2- 172.21.15.39:0/1824334873 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4c8c1016e0 0x7f4c8c199c30 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4c7c000c00 tx=0x7f4c7c02c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:04.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.732+0000 7f4c83e00640 1 -- 172.21.15.39:0/1824334873 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c7c043020 con 0x7f4c8c1016e0 2024-03-18T16:09:04.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.732+0000 7f4c91000640 1 -- 172.21.15.39:0/1824334873 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c8c1a1ad0 con 0x7f4c8c1016e0 2024-03-18T16:09:04.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.732+0000 7f4c83e00640 1 -- 172.21.15.39:0/1824334873 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4c7c0054e0 con 0x7f4c8c1016e0 2024-03-18T16:09:04.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.732+0000 7f4c83e00640 1 -- 172.21.15.39:0/1824334873 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c7c003c40 con 0x7f4c8c1016e0 2024-03-18T16:09:04.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.732+0000 7f4c91000640 1 -- 172.21.15.39:0/1824334873 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c8c1a2020 con 0x7f4c8c1016e0 2024-03-18T16:09:04.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.734+0000 7f4c91000640 1 -- 172.21.15.39:0/1824334873 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c8c0684f0 con 0x7f4c8c1016e0 2024-03-18T16:09:04.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.735+0000 7f4c83e00640 1 -- 172.21.15.39:0/1824334873 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f4c7c046020 con 0x7f4c8c1016e0 2024-03-18T16:09:04.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.737+0000 7f4c83e00640 1 --2- 172.21.15.39:0/1824334873 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f4c700a2800 0x7f4c700a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:04.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.737+0000 7f4c83e00640 1 -- 172.21.15.39:0/1824334873 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f4c7c0f0f30 con 0x7f4c8c1016e0 2024-03-18T16:09:04.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.737+0000 7f4c8aa00640 1 --2- 172.21.15.39:0/1824334873 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f4c700a2800 0x7f4c700a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:04.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.738+0000 7f4c8aa00640 1 --2- 172.21.15.39:0/1824334873 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f4c700a2800 0x7f4c700a4cc0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f4c78009bd0 tx=0x7f4c78005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:04.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.741+0000 7f4c83e00640 1 -- 172.21.15.39:0/1824334873 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4c7c04d020 con 0x7f4c8c1016e0 2024-03-18T16:09:04.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.984+0000 7f4c91000640 1 -- 172.21.15.39:0/1824334873 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f4c8c1a28c0 con 0x7f4c8c1016e0 2024-03-18T16:09:04.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.984+0000 7f4c83e00640 1 -- 172.21.15.39:0/1824334873 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7f4c7c0bdbe0 con 0x7f4c8c1016e0 2024-03-18T16:09:04.986 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 32 2024-03-18T16:09:04.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.987+0000 7f4c91000640 1 -- 172.21.15.39:0/1824334873 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f4c700a2800 msgr2=0x7f4c700a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:04.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.987+0000 7f4c91000640 1 --2- 172.21.15.39:0/1824334873 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f4c700a2800 0x7f4c700a4cc0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f4c78009bd0 tx=0x7f4c78005f90 comp rx=0 tx=0).stop 2024-03-18T16:09:04.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.987+0000 7f4c91000640 1 -- 172.21.15.39:0/1824334873 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4c8c1016e0 msgr2=0x7f4c8c199c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:04.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.987+0000 7f4c91000640 1 --2- 172.21.15.39:0/1824334873 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4c8c1016e0 0x7f4c8c199c30 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4c7c000c00 tx=0x7f4c7c02c650 comp rx=0 tx=0).stop 2024-03-18T16:09:04.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.987+0000 7f4c91000640 1 -- 172.21.15.39:0/1824334873 shutdown_connections 2024-03-18T16:09:04.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.987+0000 7f4c91000640 1 --2- 172.21.15.39:0/1824334873 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4c8c1028d0 0x7f4c8c19a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:04.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.987+0000 7f4c91000640 1 --2- 172.21.15.39:0/1824334873 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f4c700a2800 0x7f4c700a4cc0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:04.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.987+0000 7f4c91000640 1 --2- 172.21.15.39:0/1824334873 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4c8c103290 0x7f4c8c1a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:04.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.987+0000 7f4c91000640 1 --2- 172.21.15.39:0/1824334873 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4c8c1016e0 0x7f4c8c199c30 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:04.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.987+0000 7f4c91000640 1 -- 172.21.15.39:0/1824334873 >> 172.21.15.39:0/1824334873 conn(0x7f4c8c0fcc50 msgr2=0x7f4c8c073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:04.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.988+0000 7f4c91000640 1 -- 172.21.15.39:0/1824334873 shutdown_connections 2024-03-18T16:09:04.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:04.988+0000 7f4c91000640 1 -- 172.21.15.39:0/1824334873 wait complete. 2024-03-18T16:09:05.012 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T16:09:05.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.223+0000 7f6ce5600640 1 -- 172.21.15.39:0/167801970 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6ce01021c0 msgr2=0x7f6ce01045b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:05.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.223+0000 7f6ce5600640 1 --2- 172.21.15.39:0/167801970 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6ce01021c0 0x7f6ce01045b0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f6cd0009860 tx=0x7f6cd002bb70 comp rx=0 tx=0).stop 2024-03-18T16:09:05.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.223+0000 7f6ce5600640 1 -- 172.21.15.39:0/167801970 shutdown_connections 2024-03-18T16:09:05.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.223+0000 7f6ce5600640 1 --2- 172.21.15.39:0/167801970 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6ce0107420 0x7f6ce0109810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:05.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.223+0000 7f6ce5600640 1 --2- 172.21.15.39:0/167801970 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6ce0104af0 0x7f6ce0106ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:05.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.223+0000 7f6ce5600640 1 --2- 172.21.15.39:0/167801970 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6ce01021c0 0x7f6ce01045b0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:05.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.223+0000 7f6ce5600640 1 -- 172.21.15.39:0/167801970 >> 172.21.15.39:0/167801970 conn(0x7f6ce00fbb90 msgr2=0x7f6ce00fdfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:05.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.223+0000 7f6ce5600640 1 -- 172.21.15.39:0/167801970 shutdown_connections 2024-03-18T16:09:05.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.223+0000 7f6ce5600640 1 -- 172.21.15.39:0/167801970 wait complete. 2024-03-18T16:09:05.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.225+0000 7f6ce5600640 1 Processor -- start 2024-03-18T16:09:05.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.225+0000 7f6ce5600640 1 -- start start 2024-03-18T16:09:05.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.226+0000 7f6ce5600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6ce01021c0 0x7f6ce0101b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:05.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.226+0000 7f6ce5600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6ce0104af0 0x7f6ce0100170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:05.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.226+0000 7f6ce5600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6ce0107420 0x7f6ce01006b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:05.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.226+0000 7f6ce5600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f6ce010d360 con 0x7f6ce0104af0 2024-03-18T16:09:05.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.226+0000 7f6ce5600640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f6ce010d1e0 con 0x7f6ce0107420 2024-03-18T16:09:05.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.226+0000 7f6ce5600640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f6ce010d4e0 con 0x7f6ce01021c0 2024-03-18T16:09:05.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.226+0000 7f6cdfe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6ce01021c0 0x7f6ce0101b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:05.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.226+0000 7f6cdfe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6ce01021c0 0x7f6ce0101b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:36092/0 (socket says 172.21.15.39:36092) 2024-03-18T16:09:05.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.226+0000 7f6cdfe00640 1 -- 172.21.15.39:0/109002147 learned_addr learned my addr 172.21.15.39:0/109002147 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:05.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.226+0000 7f6cdf400640 1 --2- 172.21.15.39:0/109002147 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6ce0104af0 0x7f6ce0100170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:05.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.226+0000 7f6ce4c00640 1 --2- 172.21.15.39:0/109002147 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6ce0107420 0x7f6ce01006b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:05.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:04 smithi196 ceph-mon[36307]: pgmap v392: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T16:09:05.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:04 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/134203774' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:09:05.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:04 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/639145740' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T16:09:05.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.226+0000 7f6cdfe00640 1 -- 172.21.15.39:0/109002147 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6ce0107420 msgr2=0x7f6ce01006b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:05.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.226+0000 7f6cdfe00640 1 --2- 172.21.15.39:0/109002147 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6ce0107420 0x7f6ce01006b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:05.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.226+0000 7f6cdfe00640 1 -- 172.21.15.39:0/109002147 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6ce0104af0 msgr2=0x7f6ce0100170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:05.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.226+0000 7f6cdfe00640 1 --2- 172.21.15.39:0/109002147 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6ce0104af0 0x7f6ce0100170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:05.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.226+0000 7f6cdfe00640 1 -- 172.21.15.39:0/109002147 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ce0100f40 con 0x7f6ce01021c0 2024-03-18T16:09:05.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.227+0000 7f6cdfe00640 1 --2- 172.21.15.39:0/109002147 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6ce01021c0 0x7f6ce0101b30 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f6cd00349e0 tx=0x7f6cd00042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:05.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.227+0000 7f6cdcc00640 1 -- 172.21.15.39:0/109002147 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cd0041030 con 0x7f6ce01021c0 2024-03-18T16:09:05.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.227+0000 7f6cdcc00640 1 -- 172.21.15.39:0/109002147 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6cd00347b0 con 0x7f6ce01021c0 2024-03-18T16:09:05.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.227+0000 7f6ce5600640 1 -- 172.21.15.39:0/109002147 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ce01a1ac0 con 0x7f6ce01021c0 2024-03-18T16:09:05.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.227+0000 7f6cdcc00640 1 -- 172.21.15.39:0/109002147 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cd0039040 con 0x7f6ce01021c0 2024-03-18T16:09:05.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.227+0000 7f6ce5600640 1 -- 172.21.15.39:0/109002147 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ce01a2060 con 0x7f6ce01021c0 2024-03-18T16:09:05.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.229+0000 7f6ce5600640 1 -- 172.21.15.39:0/109002147 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ce00fd680 con 0x7f6ce01021c0 2024-03-18T16:09:05.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.230+0000 7f6cdcc00640 1 -- 172.21.15.39:0/109002147 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f6cd0044070 con 0x7f6ce01021c0 2024-03-18T16:09:05.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.233+0000 7f6cdcc00640 1 --2- 172.21.15.39:0/109002147 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f6cc40a2880 0x7f6cc40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:05.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.233+0000 7f6cdcc00640 1 -- 172.21.15.39:0/109002147 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f6cd00f0420 con 0x7f6ce01021c0 2024-03-18T16:09:05.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.233+0000 7f6cdf400640 1 --2- 172.21.15.39:0/109002147 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f6cc40a2880 0x7f6cc40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:05.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.234+0000 7f6cdf400640 1 --2- 172.21.15.39:0/109002147 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f6cc40a2880 0x7f6cc40a4d40 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f6ce01018a0 tx=0x7f6ccc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:05.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.236+0000 7f6cdcc00640 1 -- 172.21.15.39:0/109002147 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6cd00bd0d0 con 0x7f6ce01021c0 2024-03-18T16:09:05.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:04 smithi194 ceph-mon[36200]: pgmap v392: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T16:09:05.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:04 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/134203774' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:09:05.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:04 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/639145740' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T16:09:05.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:04 smithi039 ceph-mon[31721]: pgmap v392: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T16:09:05.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:04 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/134203774' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:09:05.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:04 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/639145740' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T16:09:05.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.452+0000 7f6ce5600640 1 -- 172.21.15.39:0/109002147 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f6ce00623e0 con 0x7f6ce01021c0 2024-03-18T16:09:05.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.453+0000 7f6cdcc00640 1 -- 172.21.15.39:0/109002147 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7f6cd00bc9d0 con 0x7f6ce01021c0 2024-03-18T16:09:05.454 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 64 2024-03-18T16:09:05.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.455+0000 7f6ce5600640 1 -- 172.21.15.39:0/109002147 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f6cc40a2880 msgr2=0x7f6cc40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:05.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.455+0000 7f6ce5600640 1 --2- 172.21.15.39:0/109002147 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f6cc40a2880 0x7f6cc40a4d40 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f6ce01018a0 tx=0x7f6ccc009260 comp rx=0 tx=0).stop 2024-03-18T16:09:05.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.455+0000 7f6ce5600640 1 -- 172.21.15.39:0/109002147 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6ce01021c0 msgr2=0x7f6ce0101b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:05.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.455+0000 7f6ce5600640 1 --2- 172.21.15.39:0/109002147 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6ce01021c0 0x7f6ce0101b30 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f6cd00349e0 tx=0x7f6cd00042f0 comp rx=0 tx=0).stop 2024-03-18T16:09:05.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.455+0000 7f6ce5600640 1 -- 172.21.15.39:0/109002147 shutdown_connections 2024-03-18T16:09:05.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.455+0000 7f6ce5600640 1 --2- 172.21.15.39:0/109002147 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6ce0107420 0x7f6ce01006b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:05.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.455+0000 7f6ce5600640 1 --2- 172.21.15.39:0/109002147 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f6cc40a2880 0x7f6cc40a4d40 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:05.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.456+0000 7f6ce5600640 1 --2- 172.21.15.39:0/109002147 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6ce0104af0 0x7f6ce0100170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:05.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.456+0000 7f6ce5600640 1 --2- 172.21.15.39:0/109002147 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6ce01021c0 0x7f6ce0101b30 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:05.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.456+0000 7f6ce5600640 1 -- 172.21.15.39:0/109002147 >> 172.21.15.39:0/109002147 conn(0x7f6ce00fbb90 msgr2=0x7f6ce0102b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:05.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.456+0000 7f6ce5600640 1 -- 172.21.15.39:0/109002147 shutdown_connections 2024-03-18T16:09:05.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.456+0000 7f6ce5600640 1 -- 172.21.15.39:0/109002147 wait complete. 2024-03-18T16:09:05.476 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T16:09:05.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.680+0000 7f4d43200640 1 -- 172.21.15.39:0/3480259991 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4d441017d0 msgr2=0x7f4d44101c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:05.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.680+0000 7f4d43200640 1 --2- 172.21.15.39:0/3480259991 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4d441017d0 0x7f4d44101c50 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f4d30009900 tx=0x7f4d3002bc30 comp rx=0 tx=0).stop 2024-03-18T16:09:05.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.680+0000 7f4d43200640 1 -- 172.21.15.39:0/3480259991 shutdown_connections 2024-03-18T16:09:05.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.680+0000 7f4d43200640 1 --2- 172.21.15.39:0/3480259991 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4d44102190 0x7f4d44108a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:05.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.680+0000 7f4d43200640 1 --2- 172.21.15.39:0/3480259991 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4d441017d0 0x7f4d44101c50 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:05.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.680+0000 7f4d43200640 1 --2- 172.21.15.39:0/3480259991 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4d441005e0 0x7f4d44100a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:05.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.680+0000 7f4d43200640 1 -- 172.21.15.39:0/3480259991 >> 172.21.15.39:0/3480259991 conn(0x7f4d440fbb90 msgr2=0x7f4d440fdfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:05.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.680+0000 7f4d43200640 1 -- 172.21.15.39:0/3480259991 shutdown_connections 2024-03-18T16:09:05.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.680+0000 7f4d43200640 1 -- 172.21.15.39:0/3480259991 wait complete. 2024-03-18T16:09:05.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.682+0000 7f4d43200640 1 Processor -- start 2024-03-18T16:09:05.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.682+0000 7f4d43200640 1 -- start start 2024-03-18T16:09:05.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.682+0000 7f4d43200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4d441005e0 0x7f4d44197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:05.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.682+0000 7f4d43200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4d441017d0 0x7f4d44197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:05.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.682+0000 7f4d43200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4d44102190 0x7f4d4419efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:05.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.682+0000 7f4d43200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f4d4410c2a0 con 0x7f4d441017d0 2024-03-18T16:09:05.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.682+0000 7f4d43200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f4d4410c120 con 0x7f4d441005e0 2024-03-18T16:09:05.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.682+0000 7f4d43200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f4d4410c420 con 0x7f4d44102190 2024-03-18T16:09:05.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.683+0000 7f4d41e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4d441005e0 0x7f4d44197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:05.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.683+0000 7f4d41e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4d441005e0 0x7f4d44197a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:35262/0 (socket says 172.21.15.39:35262) 2024-03-18T16:09:05.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.683+0000 7f4d41e00640 1 -- 172.21.15.39:0/2766831909 learned_addr learned my addr 172.21.15.39:0/2766831909 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:05.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.683+0000 7f4d41400640 1 --2- 172.21.15.39:0/2766831909 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4d441017d0 0x7f4d44197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:05.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.683+0000 7f4d42800640 1 --2- 172.21.15.39:0/2766831909 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4d44102190 0x7f4d4419efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:05.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.683+0000 7f4d41e00640 1 -- 172.21.15.39:0/2766831909 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4d44102190 msgr2=0x7f4d4419efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:05.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.683+0000 7f4d41e00640 1 --2- 172.21.15.39:0/2766831909 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4d44102190 0x7f4d4419efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:05.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.683+0000 7f4d41e00640 1 -- 172.21.15.39:0/2766831909 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4d441017d0 msgr2=0x7f4d44197f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:05.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.683+0000 7f4d41e00640 1 --2- 172.21.15.39:0/2766831909 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4d441017d0 0x7f4d44197f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:05.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.683+0000 7f4d41e00640 1 -- 172.21.15.39:0/2766831909 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d4419f620 con 0x7f4d441005e0 2024-03-18T16:09:05.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.683+0000 7f4d41400640 1 --2- 172.21.15.39:0/2766831909 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4d441017d0 0x7f4d44197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:09:05.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.683+0000 7f4d41e00640 1 --2- 172.21.15.39:0/2766831909 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4d441005e0 0x7f4d44197a00 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f4d3400e9c0 tx=0x7f4d3400edc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:05.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.684+0000 7f4d3aa00640 1 -- 172.21.15.39:0/2766831909 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d3400ccc0 con 0x7f4d441005e0 2024-03-18T16:09:05.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.684+0000 7f4d3aa00640 1 -- 172.21.15.39:0/2766831909 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d340035f0 con 0x7f4d441005e0 2024-03-18T16:09:05.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.684+0000 7f4d3aa00640 1 -- 172.21.15.39:0/2766831909 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d3400ce60 con 0x7f4d441005e0 2024-03-18T16:09:05.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.684+0000 7f4d43200640 1 -- 172.21.15.39:0/2766831909 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d4419f910 con 0x7f4d441005e0 2024-03-18T16:09:05.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.684+0000 7f4d43200640 1 -- 172.21.15.39:0/2766831909 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d4419fde0 con 0x7f4d441005e0 2024-03-18T16:09:05.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.685+0000 7f4d43200640 1 -- 172.21.15.39:0/2766831909 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d441a0040 con 0x7f4d441005e0 2024-03-18T16:09:05.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.687+0000 7f4d3aa00640 1 -- 172.21.15.39:0/2766831909 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f4d340037a0 con 0x7f4d441005e0 2024-03-18T16:09:05.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.688+0000 7f4d3aa00640 1 --2- 172.21.15.39:0/2766831909 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f4d280a2880 0x7f4d280a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:05.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.688+0000 7f4d3aa00640 1 -- 172.21.15.39:0/2766831909 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f4d340cf1f0 con 0x7f4d441005e0 2024-03-18T16:09:05.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.688+0000 7f4d41400640 1 --2- 172.21.15.39:0/2766831909 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f4d280a2880 0x7f4d280a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:05.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.689+0000 7f4d41400640 1 --2- 172.21.15.39:0/2766831909 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f4d280a2880 0x7f4d280a4d40 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f4d3002c240 tx=0x7f4d3002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:05.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.696+0000 7f4d3aa00640 1 -- 172.21.15.39:0/2766831909 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4d3409bea0 con 0x7f4d441005e0 2024-03-18T16:09:05.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.899+0000 7f4d43200640 1 -- 172.21.15.39:0/2766831909 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f4d441a0210 con 0x7f4d441005e0 2024-03-18T16:09:05.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.899+0000 7f4d3aa00640 1 -- 172.21.15.39:0/2766831909 <== mon.2 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7f4d3409b7a0 con 0x7f4d441005e0 2024-03-18T16:09:05.901 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 32 2024-03-18T16:09:05.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.902+0000 7f4d43200640 1 -- 172.21.15.39:0/2766831909 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f4d280a2880 msgr2=0x7f4d280a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:05.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.902+0000 7f4d43200640 1 --2- 172.21.15.39:0/2766831909 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f4d280a2880 0x7f4d280a4d40 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f4d3002c240 tx=0x7f4d3002df90 comp rx=0 tx=0).stop 2024-03-18T16:09:05.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.902+0000 7f4d43200640 1 -- 172.21.15.39:0/2766831909 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4d441005e0 msgr2=0x7f4d44197a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:05.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.902+0000 7f4d43200640 1 --2- 172.21.15.39:0/2766831909 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4d441005e0 0x7f4d44197a00 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f4d3400e9c0 tx=0x7f4d3400edc0 comp rx=0 tx=0).stop 2024-03-18T16:09:05.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.902+0000 7f4d43200640 1 -- 172.21.15.39:0/2766831909 shutdown_connections 2024-03-18T16:09:05.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.902+0000 7f4d43200640 1 --2- 172.21.15.39:0/2766831909 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f4d280a2880 0x7f4d280a4d40 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:05.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.902+0000 7f4d43200640 1 --2- 172.21.15.39:0/2766831909 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4d44102190 0x7f4d4419efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:05.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.902+0000 7f4d43200640 1 --2- 172.21.15.39:0/2766831909 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4d441017d0 0x7f4d44197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:05.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.902+0000 7f4d43200640 1 --2- 172.21.15.39:0/2766831909 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4d441005e0 0x7f4d44197a00 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:05.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.902+0000 7f4d43200640 1 -- 172.21.15.39:0/2766831909 >> 172.21.15.39:0/2766831909 conn(0x7f4d440fbb90 msgr2=0x7f4d440fd5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:05.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.902+0000 7f4d43200640 1 -- 172.21.15.39:0/2766831909 shutdown_connections 2024-03-18T16:09:05.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:05.902+0000 7f4d43200640 1 -- 172.21.15.39:0/2766831909 wait complete. 2024-03-18T16:09:05.921 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:09:06.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.134+0000 7fd7b6a00640 1 -- 172.21.15.39:0/1168500498 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd7b01018c0 msgr2=0x7fd7b0101d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:06.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.134+0000 7fd7b6a00640 1 --2- 172.21.15.39:0/1168500498 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd7b01018c0 0x7fd7b0101d40 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fd798009900 tx=0x7fd79802bc30 comp rx=0 tx=0).stop 2024-03-18T16:09:06.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.134+0000 7fd7b6a00640 1 -- 172.21.15.39:0/1168500498 shutdown_connections 2024-03-18T16:09:06.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.134+0000 7fd7b6a00640 1 --2- 172.21.15.39:0/1168500498 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd7b01018c0 0x7fd7b0101d40 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:06.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.134+0000 7fd7b6a00640 1 --2- 172.21.15.39:0/1168500498 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd7b0102280 0x7fd7b0108b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:06.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.134+0000 7fd7b6a00640 1 --2- 172.21.15.39:0/1168500498 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd7b01006d0 0x7fd7b0100af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:06.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.134+0000 7fd7b6a00640 1 -- 172.21.15.39:0/1168500498 >> 172.21.15.39:0/1168500498 conn(0x7fd7b00fbc40 msgr2=0x7fd7b00fe080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:06.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.134+0000 7fd7b6a00640 1 -- 172.21.15.39:0/1168500498 shutdown_connections 2024-03-18T16:09:06.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.134+0000 7fd7b6a00640 1 -- 172.21.15.39:0/1168500498 wait complete. 2024-03-18T16:09:06.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.135+0000 7fd7b6a00640 1 Processor -- start 2024-03-18T16:09:06.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.135+0000 7fd7b6a00640 1 -- start start 2024-03-18T16:09:06.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.135+0000 7fd7b6a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd7b01006d0 0x7fd7b0197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:06.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.135+0000 7fd7b6a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd7b01018c0 0x7fd7b0197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:06.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.136+0000 7fd7b6a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd7b0102280 0x7fd7b019f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:06.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.136+0000 7fd7b6a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fd7b010d4d0 con 0x7fd7b0102280 2024-03-18T16:09:06.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.136+0000 7fd7b6a00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fd7b010d350 con 0x7fd7b01018c0 2024-03-18T16:09:06.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.136+0000 7fd7b6a00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fd7b010d650 con 0x7fd7b01006d0 2024-03-18T16:09:06.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.136+0000 7fd7afe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd7b01006d0 0x7fd7b0197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:06.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.136+0000 7fd7afe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd7b01006d0 0x7fd7b0197ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:36130/0 (socket says 172.21.15.39:36130) 2024-03-18T16:09:06.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.136+0000 7fd7afe00640 1 -- 172.21.15.39:0/4219135863 learned_addr learned my addr 172.21.15.39:0/4219135863 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:06.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.136+0000 7fd7a7e00640 1 --2- 172.21.15.39:0/4219135863 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd7b01018c0 0x7fd7b0197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:06.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.136+0000 7fd7b4a00640 1 --2- 172.21.15.39:0/4219135863 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd7b0102280 0x7fd7b019f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:06.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.136+0000 7fd7afe00640 1 -- 172.21.15.39:0/4219135863 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd7b01018c0 msgr2=0x7fd7b0197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:06.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.136+0000 7fd7afe00640 1 --2- 172.21.15.39:0/4219135863 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd7b01018c0 0x7fd7b0197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:06.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.136+0000 7fd7afe00640 1 -- 172.21.15.39:0/4219135863 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd7b0102280 msgr2=0x7fd7b019f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:06.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.136+0000 7fd7afe00640 1 --2- 172.21.15.39:0/4219135863 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd7b0102280 0x7fd7b019f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:06.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.136+0000 7fd7afe00640 1 -- 172.21.15.39:0/4219135863 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7b019f6d0 con 0x7fd7b01006d0 2024-03-18T16:09:06.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.136+0000 7fd7a7e00640 1 --2- 172.21.15.39:0/4219135863 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd7b01018c0 0x7fd7b0197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:09:06.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.137+0000 7fd7afe00640 1 --2- 172.21.15.39:0/4219135863 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd7b01006d0 0x7fd7b0197ab0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fd7a000e9c0 tx=0x7fd7a000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:06.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.137+0000 7fd7ad600640 1 -- 172.21.15.39:0/4219135863 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7a000cd80 con 0x7fd7b01006d0 2024-03-18T16:09:06.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.137+0000 7fd7ad600640 1 -- 172.21.15.39:0/4219135863 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd7a00035f0 con 0x7fd7b01006d0 2024-03-18T16:09:06.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.137+0000 7fd7b6a00640 1 -- 172.21.15.39:0/4219135863 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7b019f9c0 con 0x7fd7b01006d0 2024-03-18T16:09:06.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.137+0000 7fd7ad600640 1 -- 172.21.15.39:0/4219135863 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7a0010690 con 0x7fd7b01006d0 2024-03-18T16:09:06.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.137+0000 7fd7b6a00640 1 -- 172.21.15.39:0/4219135863 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7b019fee0 con 0x7fd7b01006d0 2024-03-18T16:09:06.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.138+0000 7fd7b6a00640 1 -- 172.21.15.39:0/4219135863 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7b01079f0 con 0x7fd7b01006d0 2024-03-18T16:09:06.141 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.140+0000 7fd7ad600640 1 -- 172.21.15.39:0/4219135863 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fd7a00037a0 con 0x7fd7b01006d0 2024-03-18T16:09:06.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.141+0000 7fd7ad600640 1 --2- 172.21.15.39:0/4219135863 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd7940a2880 0x7fd7940a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:06.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.141+0000 7fd7ad600640 1 -- 172.21.15.39:0/4219135863 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fd7a00cf0c0 con 0x7fd7b01006d0 2024-03-18T16:09:06.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.141+0000 7fd7a7e00640 1 --2- 172.21.15.39:0/4219135863 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd7940a2880 0x7fd7940a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:06.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.142+0000 7fd7a7e00640 1 --2- 172.21.15.39:0/4219135863 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd7940a2880 0x7fd7940a4d40 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fd79802c240 tx=0x7fd79802df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:06.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.145+0000 7fd7ad600640 1 -- 172.21.15.39:0/4219135863 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd7a009bdd0 con 0x7fd7b01006d0 2024-03-18T16:09:06.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:05 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/1824334873' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T16:09:06.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:05 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/109002147' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T16:09:06.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:05 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/2766831909' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T16:09:06.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:05 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/1824334873' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T16:09:06.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:05 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/109002147' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T16:09:06.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:05 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/2766831909' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T16:09:06.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:05 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/1824334873' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T16:09:06.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:05 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/109002147' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T16:09:06.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:05 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/2766831909' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T16:09:06.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.505+0000 7fd7b6a00640 1 -- 172.21.15.39:0/4219135863 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd7b00624a0 con 0x7fd7b01006d0 2024-03-18T16:09:06.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.506+0000 7fd7ad600640 1 -- 172.21.15.39:0/4219135863 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11932 (secure 0 0 0) 0x7fd7a00860e0 con 0x7fd7b01006d0 2024-03-18T16:09:06.508 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:09:06.508 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2520226608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2520226608},{"type":"v1","addr":"172.21.15.39:6835","nonce":2520226608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/719341763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":719341763},{"type":"v1","addr":"172.21.15.39:6837","nonce":719341763}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T16:06:44.180425+0000","modified":"2024-03-18T16:07:20.994405+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24575,"mds_1":24556,"mds_2":24535,"mds_3":24524,"mds_4":14463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14445":{"gid":14445,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.39:6839/656513813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":656513813},{"type":"v1","addr":"172.21.15.39:6839","nonce":656513813}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14463":{"gid":14463,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":13,"addr":"172.21.15.39:6841/3515326042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3515326042},{"type":"v1","addr":"172.21.15.39:6841","nonce":3515326042}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6833/931350738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":931350738},{"type":"v1","addr":"172.21.15.194:6833","nonce":931350738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6833/3874209621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":3874209621},{"type":"v1","addr":"172.21.15.196:6833","nonce":3874209621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6835/188385470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":188385470},{"type":"v1","addr":"172.21.15.194:6835","nonce":188385470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6835/3949984309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":3949984309},{"type":"v1","addr":"172.21.15.196:6835","nonce":3949984309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":15,"addr":"172.21.15.194:6837/1301464246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":1301464246},{"type":"v1","addr":"172.21.15.194:6837","nonce":1301464246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":16,"addr":"172.21.15.196:6837/895247251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":895247251},{"type":"v1","addr":"172.21.15.196:6837","nonce":895247251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.196:6839/2001427268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6838","nonce":2001427268},{"type":"v1","addr":"172.21.15.196:6839","nonce":2001427268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24575":{"gid":24575,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.194:6839/274911158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":274911158},{"type":"v1","addr":"172.21.15.194:6839","nonce":274911158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24575,"qdb_cluster":[24556,24535,24575,24524,14463]},"id":1}]} 2024-03-18T16:09:06.508 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 27 2024-03-18T16:09:06.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.509+0000 7fd7b6a00640 1 -- 172.21.15.39:0/4219135863 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd7940a2880 msgr2=0x7fd7940a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:06.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.509+0000 7fd7b6a00640 1 --2- 172.21.15.39:0/4219135863 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd7940a2880 0x7fd7940a4d40 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fd79802c240 tx=0x7fd79802df90 comp rx=0 tx=0).stop 2024-03-18T16:09:06.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.509+0000 7fd7b6a00640 1 -- 172.21.15.39:0/4219135863 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd7b01006d0 msgr2=0x7fd7b0197ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:06.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.509+0000 7fd7b6a00640 1 --2- 172.21.15.39:0/4219135863 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd7b01006d0 0x7fd7b0197ab0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fd7a000e9c0 tx=0x7fd7a000ee80 comp rx=0 tx=0).stop 2024-03-18T16:09:06.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.509+0000 7fd7b6a00640 1 -- 172.21.15.39:0/4219135863 shutdown_connections 2024-03-18T16:09:06.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.509+0000 7fd7b6a00640 1 --2- 172.21.15.39:0/4219135863 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd7b01018c0 0x7fd7b0197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:06.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.509+0000 7fd7b6a00640 1 --2- 172.21.15.39:0/4219135863 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd7940a2880 0x7fd7940a4d40 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:06.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.509+0000 7fd7b6a00640 1 --2- 172.21.15.39:0/4219135863 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd7b0102280 0x7fd7b019f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:06.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.509+0000 7fd7b6a00640 1 --2- 172.21.15.39:0/4219135863 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd7b01006d0 0x7fd7b0197ab0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:06.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.509+0000 7fd7b6a00640 1 -- 172.21.15.39:0/4219135863 >> 172.21.15.39:0/4219135863 conn(0x7fd7b00fbc40 msgr2=0x7fd7b0104bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:06.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.510+0000 7fd7b6a00640 1 -- 172.21.15.39:0/4219135863 shutdown_connections 2024-03-18T16:09:06.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.510+0000 7fd7b6a00640 1 -- 172.21.15.39:0/4219135863 wait complete. 2024-03-18T16:09:06.533 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T16:09:06.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.737+0000 7ff77a200640 1 -- 172.21.15.39:0/1998801706 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff7741049a0 msgr2=0x7ff774106d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:06.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.737+0000 7ff77a200640 1 --2- 172.21.15.39:0/1998801706 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff7741049a0 0x7ff774106d90 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff760009900 tx=0x7ff76002bc50 comp rx=0 tx=0).stop 2024-03-18T16:09:06.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.738+0000 7ff77a200640 1 -- 172.21.15.39:0/1998801706 shutdown_connections 2024-03-18T16:09:06.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.738+0000 7ff77a200640 1 --2- 172.21.15.39:0/1998801706 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff7741072d0 0x7ff774109720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:06.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.738+0000 7ff77a200640 1 --2- 172.21.15.39:0/1998801706 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff7741049a0 0x7ff774106d90 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:06.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.738+0000 7ff77a200640 1 --2- 172.21.15.39:0/1998801706 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff774102070 0x7ff774104460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:06.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.738+0000 7ff77a200640 1 -- 172.21.15.39:0/1998801706 >> 172.21.15.39:0/1998801706 conn(0x7ff7740fba60 msgr2=0x7ff7740fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:06.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.738+0000 7ff77a200640 1 -- 172.21.15.39:0/1998801706 shutdown_connections 2024-03-18T16:09:06.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.738+0000 7ff77a200640 1 -- 172.21.15.39:0/1998801706 wait complete. 2024-03-18T16:09:06.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.739+0000 7ff77a200640 1 Processor -- start 2024-03-18T16:09:06.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.739+0000 7ff77a200640 1 -- start start 2024-03-18T16:09:06.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.739+0000 7ff77a200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff774102070 0x7ff7741979a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:06.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.739+0000 7ff77a200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7741049a0 0x7ff774197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:06.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.739+0000 7ff77a200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff7741072d0 0x7ff77419ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:06.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.739+0000 7ff77a200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7ff77410d250 con 0x7ff7741049a0 2024-03-18T16:09:06.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.739+0000 7ff77a200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ff77410d0d0 con 0x7ff7741072d0 2024-03-18T16:09:06.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.739+0000 7ff77a200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7ff77410d3d0 con 0x7ff774102070 2024-03-18T16:09:06.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.739+0000 7ff778e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff774102070 0x7ff7741979a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:06.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.739+0000 7ff778e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff774102070 0x7ff7741979a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:36158/0 (socket says 172.21.15.39:36158) 2024-03-18T16:09:06.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.739+0000 7ff778e00640 1 -- 172.21.15.39:0/3242826704 learned_addr learned my addr 172.21.15.39:0/3242826704 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:06.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.739+0000 7ff779800640 1 --2- 172.21.15.39:0/3242826704 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff7741072d0 0x7ff77419ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:06.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.739+0000 7ff773e00640 1 --2- 172.21.15.39:0/3242826704 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7741049a0 0x7ff774197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:06.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.740+0000 7ff778e00640 1 -- 172.21.15.39:0/3242826704 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff7741072d0 msgr2=0x7ff77419ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:06.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.740+0000 7ff778e00640 1 --2- 172.21.15.39:0/3242826704 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff7741072d0 0x7ff77419ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:06.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.740+0000 7ff778e00640 1 -- 172.21.15.39:0/3242826704 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7741049a0 msgr2=0x7ff774197ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:06.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.740+0000 7ff778e00640 1 --2- 172.21.15.39:0/3242826704 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7741049a0 0x7ff774197ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:06.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.740+0000 7ff778e00640 1 -- 172.21.15.39:0/3242826704 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff77419f5c0 con 0x7ff774102070 2024-03-18T16:09:06.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.740+0000 7ff779800640 1 --2- 172.21.15.39:0/3242826704 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff7741072d0 0x7ff77419ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:09:06.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.740+0000 7ff778e00640 1 --2- 172.21.15.39:0/3242826704 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff774102070 0x7ff7741979a0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ff76800e9f0 tx=0x7ff76800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:06.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.740+0000 7ff771600640 1 -- 172.21.15.39:0/3242826704 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff76800cd80 con 0x7ff774102070 2024-03-18T16:09:06.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.740+0000 7ff77a200640 1 -- 172.21.15.39:0/3242826704 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff77419f8b0 con 0x7ff774102070 2024-03-18T16:09:06.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.740+0000 7ff771600640 1 -- 172.21.15.39:0/3242826704 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff768010910 con 0x7ff774102070 2024-03-18T16:09:06.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.740+0000 7ff771600640 1 -- 172.21.15.39:0/3242826704 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7680188f0 con 0x7ff774102070 2024-03-18T16:09:06.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.740+0000 7ff77a200640 1 -- 172.21.15.39:0/3242826704 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff77419fdd0 con 0x7ff774102070 2024-03-18T16:09:06.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.742+0000 7ff77a200640 1 -- 172.21.15.39:0/3242826704 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7740fd550 con 0x7ff774102070 2024-03-18T16:09:06.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.743+0000 7ff771600640 1 -- 172.21.15.39:0/3242826704 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7ff768010ac0 con 0x7ff774102070 2024-03-18T16:09:06.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.744+0000 7ff771600640 1 --2- 172.21.15.39:0/3242826704 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff75c0a2800 0x7ff75c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:06.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.745+0000 7ff771600640 1 -- 172.21.15.39:0/3242826704 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7ff7680cf200 con 0x7ff774102070 2024-03-18T16:09:06.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.745+0000 7ff773e00640 1 --2- 172.21.15.39:0/3242826704 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff75c0a2800 0x7ff75c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:06.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.745+0000 7ff773e00640 1 --2- 172.21.15.39:0/3242826704 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff75c0a2800 0x7ff75c0a4cc0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7ff760000c00 tx=0x7ff760005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:06.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:06.748+0000 7ff771600640 1 -- 172.21.15.39:0/3242826704 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff76809b860 con 0x7ff774102070 2024-03-18T16:09:07.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.002+0000 7ff77a200640 1 -- 172.21.15.39:0/3242826704 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff7740623e0 con 0x7ff774102070 2024-03-18T16:09:07.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.004+0000 7ff771600640 1 -- 172.21.15.39:0/3242826704 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24762 (secure 0 0 0) 0x7ff76809f900 con 0x7ff774102070 2024-03-18T16:09:07.006 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:09:07.006 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":93,"fsid":"1892d8fc-e540-11ee-95c9-87774f69a715","created":"2024-03-18T15:57:36.507008+0000","modified":"2024-03-18T16:08:47.874967+0000","last_up_change":"2024-03-18T16:05:45.250629+0000","last_in_change":"2024-03-18T16:05:24.841337+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T16:01:30.048247+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T16:06:42.191791+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T16:06:43.670325+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T16:06:47.523388+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"771e0558-be62-4436-94b0-6364751911c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6803","nonce":4090743635}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6805","nonce":4090743635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6809","nonce":4090743635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6807","nonce":4090743635}]},"public_addr":"172.21.15.39:6803/4090743635","cluster_addr":"172.21.15.39:6805/4090743635","heartbeat_back_addr":"172.21.15.39:6809/4090743635","heartbeat_front_addr":"172.21.15.39:6807/4090743635","state":["exists","up"]},{"osd":1,"uuid":"0430eee1-cf34-4277-b255-c7d9f07ad19a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6801","nonce":2800329805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6803","nonce":2800329805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6807","nonce":2800329805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6805","nonce":2800329805}]},"public_addr":"172.21.15.194:6801/2800329805","cluster_addr":"172.21.15.194:6803/2800329805","heartbeat_back_addr":"172.21.15.194:6807/2800329805","heartbeat_front_addr":"172.21.15.194:6805/2800329805","state":["exists","up"]},{"osd":2,"uuid":"3a718da4-2511-4ec1-b1a8-eafd6cce0df0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6801","nonce":3935874719}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6803","nonce":3935874719}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6807","nonce":3935874719}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6805","nonce":3935874719}]},"public_addr":"172.21.15.196:6801/3935874719","cluster_addr":"172.21.15.196:6803/3935874719","heartbeat_back_addr":"172.21.15.196:6807/3935874719","heartbeat_front_addr":"172.21.15.196:6805/3935874719","state":["exists","up"]},{"osd":3,"uuid":"6f9c37f9-0ab9-4038-8919-afd4b39c4165","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6811","nonce":893345939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6813","nonce":893345939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6817","nonce":893345939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6815","nonce":893345939}]},"public_addr":"172.21.15.39:6811/893345939","cluster_addr":"172.21.15.39:6813/893345939","heartbeat_back_addr":"172.21.15.39:6817/893345939","heartbeat_front_addr":"172.21.15.39:6815/893345939","state":["exists","up"]},{"osd":4,"uuid":"e795bb0e-4791-4537-acda-1a321e19bc9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6809","nonce":800423088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6811","nonce":800423088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6815","nonce":800423088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6813","nonce":800423088}]},"public_addr":"172.21.15.194:6809/800423088","cluster_addr":"172.21.15.194:6811/800423088","heartbeat_back_addr":"172.21.15.194:6815/800423088","heartbeat_front_addr":"172.21.15.194:6813/800423088","state":["exists","up"]},{"osd":5,"uuid":"026e175e-6acb-4b4c-b235-8ccee8ae1149","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6809","nonce":2746224525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6811","nonce":2746224525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6815","nonce":2746224525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6813","nonce":2746224525}]},"public_addr":"172.21.15.196:6809/2746224525","cluster_addr":"172.21.15.196:6811/2746224525","heartbeat_back_addr":"172.21.15.196:6815/2746224525","heartbeat_front_addr":"172.21.15.196:6813/2746224525","state":["exists","up"]},{"osd":6,"uuid":"c1b183ff-383f-4c5c-9075-f76bab1c0bb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6819","nonce":1213371331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6821","nonce":1213371331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6825","nonce":1213371331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6823","nonce":1213371331}]},"public_addr":"172.21.15.39:6819/1213371331","cluster_addr":"172.21.15.39:6821/1213371331","heartbeat_back_addr":"172.21.15.39:6825/1213371331","heartbeat_front_addr":"172.21.15.39:6823/1213371331","state":["exists","up"]},{"osd":7,"uuid":"612f8697-957d-4ef4-a87f-241b36f9d13d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6817","nonce":3299162444}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6819","nonce":3299162444}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6823","nonce":3299162444}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6821","nonce":3299162444}]},"public_addr":"172.21.15.194:6817/3299162444","cluster_addr":"172.21.15.194:6819/3299162444","heartbeat_back_addr":"172.21.15.194:6823/3299162444","heartbeat_front_addr":"172.21.15.194:6821/3299162444","state":["exists","up"]},{"osd":8,"uuid":"836a9107-4d4f-4594-b8dc-0301e64f59d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6817","nonce":2456143145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6819","nonce":2456143145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6823","nonce":2456143145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6821","nonce":2456143145}]},"public_addr":"172.21.15.196:6817/2456143145","cluster_addr":"172.21.15.196:6819/2456143145","heartbeat_back_addr":"172.21.15.196:6823/2456143145","heartbeat_front_addr":"172.21.15.196:6821/2456143145","state":["exists","up"]},{"osd":9,"uuid":"4f30b960-12e3-4b16-bd28-fcc988139b58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6827","nonce":1848634433}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6829","nonce":1848634433}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6833","nonce":1848634433}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6831","nonce":1848634433}]},"public_addr":"172.21.15.39:6827/1848634433","cluster_addr":"172.21.15.39:6829/1848634433","heartbeat_back_addr":"172.21.15.39:6833/1848634433","heartbeat_front_addr":"172.21.15.39:6831/1848634433","state":["exists","up"]},{"osd":10,"uuid":"620f90e8-23e6-4bc0-8269-7219b9e1aebd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6825","nonce":3725064809}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6827","nonce":3725064809}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6831","nonce":3725064809}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6829","nonce":3725064809}]},"public_addr":"172.21.15.194:6825/3725064809","cluster_addr":"172.21.15.194:6827/3725064809","heartbeat_back_addr":"172.21.15.194:6831/3725064809","heartbeat_front_addr":"172.21.15.194:6829/3725064809","state":["exists","up"]},{"osd":11,"uuid":"867a2535-8235-4ceb-a698-eb86869d6d20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6825","nonce":2501212422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6827","nonce":2501212422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6831","nonce":2501212422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6829","nonce":2501212422}]},"public_addr":"172.21.15.196:6825/2501212422","cluster_addr":"172.21.15.196:6827/2501212422","heartbeat_back_addr":"172.21.15.196:6831/2501212422","heartbeat_front_addr":"172.21.15.196:6829/2501212422","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:00:36.931566+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:01:02.614024+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:01:26.193265+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:01:58.308833+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:02:25.049546+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:02:49.748750+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:03:23.582153+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:03:50.447321+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:04:15.812129+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:04:49.998047+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:05:17.087924+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:05:43.167637+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":4,"to":7}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.a","mappings":[{"from":5,"to":8}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.39:0/3482086140":"2024-03-19T15:58:45.340294+0000","172.21.15.39:0/3844306284":"2024-03-19T15:58:45.340294+0000","172.21.15.39:6801/1810238546":"2024-03-19T15:58:45.340294+0000","172.21.15.39:6800/1810238546":"2024-03-19T15:58:45.340294+0000","172.21.15.39:0/2528397488":"2024-03-19T15:58:10.264798+0000","172.21.15.39:0/2240981254":"2024-03-19T15:58:10.264798+0000","172.21.15.39:6800/1217503189":"2024-03-19T15:58:10.264798+0000","172.21.15.39:0/3754876370":"2024-03-19T15:58:10.264798+0000","172.21.15.39:6801/1217503189":"2024-03-19T15:58:10.264798+0000","172.21.15.39:0/1874637628":"2024-03-19T15:58:45.340294+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T16:09:07.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.008+0000 7ff77a200640 1 -- 172.21.15.39:0/3242826704 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff75c0a2800 msgr2=0x7ff75c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:07.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.008+0000 7ff77a200640 1 --2- 172.21.15.39:0/3242826704 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff75c0a2800 0x7ff75c0a4cc0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7ff760000c00 tx=0x7ff760005fd0 comp rx=0 tx=0).stop 2024-03-18T16:09:07.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.008+0000 7ff77a200640 1 -- 172.21.15.39:0/3242826704 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff774102070 msgr2=0x7ff7741979a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:07.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.008+0000 7ff77a200640 1 --2- 172.21.15.39:0/3242826704 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff774102070 0x7ff7741979a0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ff76800e9f0 tx=0x7ff76800eeb0 comp rx=0 tx=0).stop 2024-03-18T16:09:07.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.008+0000 7ff77a200640 1 -- 172.21.15.39:0/3242826704 shutdown_connections 2024-03-18T16:09:07.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.008+0000 7ff77a200640 1 --2- 172.21.15.39:0/3242826704 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff7741072d0 0x7ff77419ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:07.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.008+0000 7ff77a200640 1 --2- 172.21.15.39:0/3242826704 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff75c0a2800 0x7ff75c0a4cc0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:07.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.008+0000 7ff77a200640 1 --2- 172.21.15.39:0/3242826704 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7741049a0 0x7ff774197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:07.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.008+0000 7ff77a200640 1 --2- 172.21.15.39:0/3242826704 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff774102070 0x7ff7741979a0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:07.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.008+0000 7ff77a200640 1 -- 172.21.15.39:0/3242826704 >> 172.21.15.39:0/3242826704 conn(0x7ff7740fba60 msgr2=0x7ff7741029b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:07.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.009+0000 7ff77a200640 1 -- 172.21.15.39:0/3242826704 shutdown_connections 2024-03-18T16:09:07.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.009+0000 7ff77a200640 1 -- 172.21.15.39:0/3242826704 wait complete. 2024-03-18T16:09:07.037 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:09:07.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:06 smithi196 ceph-mon[36307]: pgmap v393: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T16:09:07.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:06 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/4219135863' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:09:07.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:06 smithi194 ceph-mon[36200]: pgmap v393: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T16:09:07.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:06 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/4219135863' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:09:07.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.254+0000 7f44bb200640 1 -- 172.21.15.39:0/2087888453 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f44bc0ffe60 msgr2=0x7f44bc100280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:07.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.254+0000 7f44bb200640 1 --2- 172.21.15.39:0/2087888453 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f44bc0ffe60 0x7f44bc100280 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f44ac009ca0 tx=0x7f44ac02c0f0 comp rx=0 tx=0).stop 2024-03-18T16:09:07.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.254+0000 7f44bb200640 1 -- 172.21.15.39:0/2087888453 shutdown_connections 2024-03-18T16:09:07.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.254+0000 7f44bb200640 1 --2- 172.21.15.39:0/2087888453 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f44bc1007c0 0x7f44bc100c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:07.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.254+0000 7f44bb200640 1 --2- 172.21.15.39:0/2087888453 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f44bc101160 0x7f44bc1015e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:07.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.254+0000 7f44bb200640 1 --2- 172.21.15.39:0/2087888453 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f44bc0ffe60 0x7f44bc100280 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:07.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.254+0000 7f44bb200640 1 -- 172.21.15.39:0/2087888453 >> 172.21.15.39:0/2087888453 conn(0x7f44bc0fba60 msgr2=0x7f44bc0fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:07.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.254+0000 7f44bb200640 1 -- 172.21.15.39:0/2087888453 shutdown_connections 2024-03-18T16:09:07.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.254+0000 7f44bb200640 1 -- 172.21.15.39:0/2087888453 wait complete. 2024-03-18T16:09:07.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.255+0000 7f44bb200640 1 Processor -- start 2024-03-18T16:09:07.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.255+0000 7f44bb200640 1 -- start start 2024-03-18T16:09:07.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.255+0000 7f44bb200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f44bc0ffe60 0x7f44bc199a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:07.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.256+0000 7f44bb200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f44bc1007c0 0x7f44bc199fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:07.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.256+0000 7f44bb200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f44bc101160 0x7f44bc1a1010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:07.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.256+0000 7f44bb200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f44bc10f450 con 0x7f44bc0ffe60 2024-03-18T16:09:07.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.256+0000 7f44bb200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f44bc10f2d0 con 0x7f44bc1007c0 2024-03-18T16:09:07.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.256+0000 7f44bb200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f44bc10f5d0 con 0x7f44bc101160 2024-03-18T16:09:07.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.256+0000 7f44b9e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f44bc0ffe60 0x7f44bc199a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:07.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.256+0000 7f44b9e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f44bc0ffe60 0x7f44bc199a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:42420/0 (socket says 172.21.15.39:42420) 2024-03-18T16:09:07.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.256+0000 7f44b1400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f44bc1007c0 0x7f44bc199fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:07.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.256+0000 7f44b9e00640 1 -- 172.21.15.39:0/4229551780 learned_addr learned my addr 172.21.15.39:0/4229551780 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:07.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.256+0000 7f44ba800640 1 --2- 172.21.15.39:0/4229551780 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f44bc101160 0x7f44bc1a1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:07.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.256+0000 7f44b1400640 1 -- 172.21.15.39:0/4229551780 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f44bc101160 msgr2=0x7f44bc1a1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:07.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.256+0000 7f44b1400640 1 --2- 172.21.15.39:0/4229551780 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f44bc101160 0x7f44bc1a1010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:07.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.256+0000 7f44b1400640 1 -- 172.21.15.39:0/4229551780 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f44bc0ffe60 msgr2=0x7f44bc199a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:07.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.256+0000 7f44b1400640 1 --2- 172.21.15.39:0/4229551780 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f44bc0ffe60 0x7f44bc199a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:07.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.256+0000 7f44b1400640 1 -- 172.21.15.39:0/4229551780 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44bc1a1680 con 0x7f44bc1007c0 2024-03-18T16:09:07.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.257+0000 7f44b1400640 1 --2- 172.21.15.39:0/4229551780 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f44bc1007c0 0x7f44bc199fa0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f44a400e9f0 tx=0x7f44a400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:07.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.257+0000 7f44b3400640 1 -- 172.21.15.39:0/4229551780 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44a400ccc0 con 0x7f44bc1007c0 2024-03-18T16:09:07.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.257+0000 7f44b3400640 1 -- 172.21.15.39:0/4229551780 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f44a4005020 con 0x7f44bc1007c0 2024-03-18T16:09:07.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.257+0000 7f44b3400640 1 -- 172.21.15.39:0/4229551780 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44a40049e0 con 0x7f44bc1007c0 2024-03-18T16:09:07.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.257+0000 7f44bb200640 1 -- 172.21.15.39:0/4229551780 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44bc1a1970 con 0x7f44bc1007c0 2024-03-18T16:09:07.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.257+0000 7f44bb200640 1 -- 172.21.15.39:0/4229551780 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44bc1a1e70 con 0x7f44bc1007c0 2024-03-18T16:09:07.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.258+0000 7f44bb200640 1 -- 172.21.15.39:0/4229551780 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44bc10acf0 con 0x7f44bc1007c0 2024-03-18T16:09:07.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.260+0000 7f44b3400640 1 -- 172.21.15.39:0/4229551780 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f44a4005560 con 0x7f44bc1007c0 2024-03-18T16:09:07.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.261+0000 7f44b3400640 1 --2- 172.21.15.39:0/4229551780 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f44a00a2800 0x7f44a00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:07.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.261+0000 7f44b3400640 1 -- 172.21.15.39:0/4229551780 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f44a40cee40 con 0x7f44bc1007c0 2024-03-18T16:09:07.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.261+0000 7f44b9e00640 1 --2- 172.21.15.39:0/4229551780 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f44a00a2800 0x7f44a00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:07.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.262+0000 7f44b9e00640 1 --2- 172.21.15.39:0/4229551780 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f44a00a2800 0x7f44a00a4cc0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f44ac006010 tx=0x7f44ac004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:07.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.265+0000 7f44b3400640 1 -- 172.21.15.39:0/4229551780 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f44a40051d0 con 0x7f44bc1007c0 2024-03-18T16:09:07.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:06 smithi039 ceph-mon[31721]: pgmap v393: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T16:09:07.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:06 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/4219135863' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:09:07.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.498+0000 7f44bb200640 1 -- 172.21.15.39:0/4229551780 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f44bc0623e0 con 0x7f44bc1007c0 2024-03-18T16:09:07.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.499+0000 7f44b3400640 1 -- 172.21.15.39:0/4229551780 <== mon.2 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11932 (secure 0 0 0) 0x7f44a409baf0 con 0x7f44bc1007c0 2024-03-18T16:09:07.501 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:09:07.501 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2520226608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2520226608},{"type":"v1","addr":"172.21.15.39:6835","nonce":2520226608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/719341763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":719341763},{"type":"v1","addr":"172.21.15.39:6837","nonce":719341763}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T16:06:44.180425+0000","modified":"2024-03-18T16:07:20.994405+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24575,"mds_1":24556,"mds_2":24535,"mds_3":24524,"mds_4":14463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14445":{"gid":14445,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.39:6839/656513813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":656513813},{"type":"v1","addr":"172.21.15.39:6839","nonce":656513813}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14463":{"gid":14463,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":13,"addr":"172.21.15.39:6841/3515326042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3515326042},{"type":"v1","addr":"172.21.15.39:6841","nonce":3515326042}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6833/931350738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":931350738},{"type":"v1","addr":"172.21.15.194:6833","nonce":931350738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6833/3874209621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":3874209621},{"type":"v1","addr":"172.21.15.196:6833","nonce":3874209621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6835/188385470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":188385470},{"type":"v1","addr":"172.21.15.194:6835","nonce":188385470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6835/3949984309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":3949984309},{"type":"v1","addr":"172.21.15.196:6835","nonce":3949984309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":15,"addr":"172.21.15.194:6837/1301464246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":1301464246},{"type":"v1","addr":"172.21.15.194:6837","nonce":1301464246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":16,"addr":"172.21.15.196:6837/895247251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":895247251},{"type":"v1","addr":"172.21.15.196:6837","nonce":895247251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.196:6839/2001427268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6838","nonce":2001427268},{"type":"v1","addr":"172.21.15.196:6839","nonce":2001427268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24575":{"gid":24575,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.194:6839/274911158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":274911158},{"type":"v1","addr":"172.21.15.194:6839","nonce":274911158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24575,"qdb_cluster":[24556,24535,24575,24524,14463]},"id":1}]} 2024-03-18T16:09:07.501 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 27 2024-03-18T16:09:07.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.502+0000 7f44bb200640 1 -- 172.21.15.39:0/4229551780 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f44a00a2800 msgr2=0x7f44a00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:07.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.503+0000 7f44bb200640 1 --2- 172.21.15.39:0/4229551780 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f44a00a2800 0x7f44a00a4cc0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f44ac006010 tx=0x7f44ac004a40 comp rx=0 tx=0).stop 2024-03-18T16:09:07.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.503+0000 7f44bb200640 1 -- 172.21.15.39:0/4229551780 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f44bc1007c0 msgr2=0x7f44bc199fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:07.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.503+0000 7f44bb200640 1 --2- 172.21.15.39:0/4229551780 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f44bc1007c0 0x7f44bc199fa0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f44a400e9f0 tx=0x7f44a400eeb0 comp rx=0 tx=0).stop 2024-03-18T16:09:07.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.503+0000 7f44bb200640 1 -- 172.21.15.39:0/4229551780 shutdown_connections 2024-03-18T16:09:07.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.503+0000 7f44bb200640 1 --2- 172.21.15.39:0/4229551780 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f44bc1007c0 0x7f44bc199fa0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:07.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.503+0000 7f44bb200640 1 --2- 172.21.15.39:0/4229551780 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f44a00a2800 0x7f44a00a4cc0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:07.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.503+0000 7f44bb200640 1 --2- 172.21.15.39:0/4229551780 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f44bc101160 0x7f44bc1a1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:07.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.503+0000 7f44bb200640 1 --2- 172.21.15.39:0/4229551780 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f44bc0ffe60 0x7f44bc199a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:07.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.503+0000 7f44bb200640 1 -- 172.21.15.39:0/4229551780 >> 172.21.15.39:0/4229551780 conn(0x7f44bc0fba60 msgr2=0x7f44bc0fd0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:07.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.503+0000 7f44bb200640 1 -- 172.21.15.39:0/4229551780 shutdown_connections 2024-03-18T16:09:07.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.503+0000 7f44bb200640 1 -- 172.21.15.39:0/4229551780 wait complete. 2024-03-18T16:09:07.521 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-18T16:06:44.180425+0000', 'modified': '2024-03-18T16:07:20.994405+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24575, 'mds_1': 24556, 'mds_2': 24535, 'mds_3': 24524, 'mds_4': 14463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14445': {'gid': 14445, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.39:6839/656513813', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6838', 'nonce': 656513813}, {'type': 'v1', 'addr': '172.21.15.39:6839', 'nonce': 656513813}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14463': {'gid': 14463, 'name': 'a', 'rank': 4, 'incarnation': 25, 'state': 'up:active', 'state_seq': 13, 'addr': '172.21.15.39:6841/3515326042', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6840', 'nonce': 3515326042}, {'type': 'v1', 'addr': '172.21.15.39:6841', 'nonce': 3515326042}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24446': {'gid': 24446, 'name': 'k', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.194:6833/931350738', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.194:6832', 'nonce': 931350738}, {'type': 'v1', 'addr': '172.21.15.194:6833', 'nonce': 931350738}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24463': {'gid': 24463, 'name': 'l', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.196:6833/3874209621', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6832', 'nonce': 3874209621}, {'type': 'v1', 'addr': '172.21.15.196:6833', 'nonce': 3874209621}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24494': {'gid': 24494, 'name': 'e', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.194:6835/188385470', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.194:6834', 'nonce': 188385470}, {'type': 'v1', 'addr': '172.21.15.194:6835', 'nonce': 188385470}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24523': {'gid': 24523, 'name': 'f', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.196:6835/3949984309', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6834', 'nonce': 3949984309}, {'type': 'v1', 'addr': '172.21.15.196:6835', 'nonce': 3949984309}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24524': {'gid': 24524, 'name': 'h', 'rank': 3, 'incarnation': 23, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.194:6837/1301464246', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.194:6836', 'nonce': 1301464246}, {'type': 'v1', 'addr': '172.21.15.194:6837', 'nonce': 1301464246}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24535': {'gid': 24535, 'name': 'c', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.196:6837/895247251', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6836', 'nonce': 895247251}, {'type': 'v1', 'addr': '172.21.15.196:6837', 'nonce': 895247251}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24556': {'gid': 24556, 'name': 'i', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.196:6839/2001427268', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6838', 'nonce': 2001427268}, {'type': 'v1', 'addr': '172.21.15.196:6839', 'nonce': 2001427268}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24575': {'gid': 24575, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.194:6839/274911158', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.194:6838', 'nonce': 274911158}, {'type': 'v1', 'addr': '172.21.15.194:6839', 'nonce': 274911158}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24575, 'qdb_cluster': [24556, 24535, 24575, 24524, 14463]} 2024-03-18T16:09:07.521 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-03-18T16:09:07.522 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-18T16:09:07.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.735+0000 7f523ce00640 1 -- 172.21.15.39:0/2478878172 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5238102140 msgr2=0x7f5238104530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:07.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.735+0000 7f523ce00640 1 --2- 172.21.15.39:0/2478878172 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5238102140 0x7f5238104530 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f5228009900 tx=0x7f522802bc30 comp rx=0 tx=0).stop 2024-03-18T16:09:07.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.735+0000 7f523ce00640 1 -- 172.21.15.39:0/2478878172 shutdown_connections 2024-03-18T16:09:07.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.735+0000 7f523ce00640 1 --2- 172.21.15.39:0/2478878172 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f52381073a0 0x7f52381097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:07.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.735+0000 7f523ce00640 1 --2- 172.21.15.39:0/2478878172 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5238104a70 0x7f5238106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:07.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.735+0000 7f523ce00640 1 --2- 172.21.15.39:0/2478878172 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5238102140 0x7f5238104530 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:07.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.735+0000 7f523ce00640 1 -- 172.21.15.39:0/2478878172 >> 172.21.15.39:0/2478878172 conn(0x7f52380fbb50 msgr2=0x7f52380fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:07.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.735+0000 7f523ce00640 1 -- 172.21.15.39:0/2478878172 shutdown_connections 2024-03-18T16:09:07.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.735+0000 7f523ce00640 1 -- 172.21.15.39:0/2478878172 wait complete. 2024-03-18T16:09:07.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.736+0000 7f523ce00640 1 Processor -- start 2024-03-18T16:09:07.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.736+0000 7f523ce00640 1 -- start start 2024-03-18T16:09:07.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.736+0000 7f523ce00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5238102140 0x7f5238199c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:07.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.736+0000 7f523ce00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5238104a70 0x7f523819a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:07.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.736+0000 7f523ce00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f52381073a0 0x7f52381a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:07.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.736+0000 7f523ce00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f523810d360 con 0x7f5238104a70 2024-03-18T16:09:07.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.736+0000 7f523ce00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f523810d1e0 con 0x7f52381073a0 2024-03-18T16:09:07.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.736+0000 7f523ce00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f523810d4e0 con 0x7f5238102140 2024-03-18T16:09:07.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.737+0000 7f5237400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5238102140 0x7f5238199c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:07.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.737+0000 7f5237400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5238102140 0x7f5238199c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:36212/0 (socket says 172.21.15.39:36212) 2024-03-18T16:09:07.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.737+0000 7f5237400640 1 -- 172.21.15.39:0/553665277 learned_addr learned my addr 172.21.15.39:0/553665277 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:07.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.737+0000 7f5236a00640 1 --2- 172.21.15.39:0/553665277 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5238104a70 0x7f523819a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:07.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.737+0000 7f5237e00640 1 --2- 172.21.15.39:0/553665277 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f52381073a0 0x7f52381a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:07.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.737+0000 7f5237400640 1 -- 172.21.15.39:0/553665277 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f52381073a0 msgr2=0x7f52381a11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:07.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.737+0000 7f5237400640 1 --2- 172.21.15.39:0/553665277 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f52381073a0 0x7f52381a11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:07.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.737+0000 7f5237400640 1 -- 172.21.15.39:0/553665277 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5238104a70 msgr2=0x7f523819a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:07.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.737+0000 7f5237400640 1 --2- 172.21.15.39:0/553665277 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5238104a70 0x7f523819a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:07.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.737+0000 7f5237400640 1 -- 172.21.15.39:0/553665277 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52381a1840 con 0x7f5238102140 2024-03-18T16:09:07.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.737+0000 7f5237400640 1 --2- 172.21.15.39:0/553665277 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5238102140 0x7f5238199c30 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f5228009230 tx=0x7f522802c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:07.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.738+0000 7f522fe00640 1 -- 172.21.15.39:0/553665277 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5228041070 con 0x7f5238102140 2024-03-18T16:09:07.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.738+0000 7f523ce00640 1 -- 172.21.15.39:0/553665277 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52381a1ad0 con 0x7f5238102140 2024-03-18T16:09:07.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.738+0000 7f522fe00640 1 -- 172.21.15.39:0/553665277 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5228004450 con 0x7f5238102140 2024-03-18T16:09:07.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.738+0000 7f522fe00640 1 -- 172.21.15.39:0/553665277 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5228003db0 con 0x7f5238102140 2024-03-18T16:09:07.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.738+0000 7f523ce00640 1 -- 172.21.15.39:0/553665277 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52381a1fc0 con 0x7f5238102140 2024-03-18T16:09:07.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.742+0000 7f52411e9740 1 Processor -- start 2024-03-18T16:09:07.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.742+0000 7f52411e9740 1 -- start start 2024-03-18T16:09:07.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.742+0000 7f522fe00640 1 -- 172.21.15.39:0/553665277 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f5228044070 con 0x7f5238102140 2024-03-18T16:09:07.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.742+0000 7f52411e9740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5649dc346340 0x5649dc346760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:07.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.742+0000 7f52411e9740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5649dc346d30 0x5649dc34b830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:07.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.742+0000 7f52411e9740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649dc34bea0 0x5649dc34e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:07.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.742+0000 7f52411e9740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5649dc2aac50 con 0x5649dc346340 2024-03-18T16:09:07.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.742+0000 7f52411e9740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5649dc2539d0 con 0x5649dc34bea0 2024-03-18T16:09:07.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.743+0000 7f52411e9740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x5649dc2a6cf0 con 0x5649dc346d30 2024-03-18T16:09:07.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.743+0000 7f5237e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5649dc346340 0x5649dc346760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:07.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.743+0000 7f5236a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649dc34bea0 0x5649dc34e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:07.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.743+0000 7f5237400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5649dc346d30 0x5649dc34b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:07.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.743+0000 7f5236a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649dc34bea0 0x5649dc34e330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:35352/0 (socket says 172.21.15.39:35352) 2024-03-18T16:09:07.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.743+0000 7f5237e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5649dc346340 0x5649dc346760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:42442/0 (socket says 172.21.15.39:42442) 2024-03-18T16:09:07.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.743+0000 7f5236a00640 1 -- 172.21.15.39:0/3566135713 learned_addr learned my addr 172.21.15.39:0/3566135713 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:07.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.743+0000 7f5236a00640 1 -- 172.21.15.39:0/3566135713 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5649dc346d30 msgr2=0x5649dc34b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:07.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.743+0000 7f5236a00640 1 --2- 172.21.15.39:0/3566135713 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5649dc346d30 0x5649dc34b830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:07.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.743+0000 7f5236a00640 1 -- 172.21.15.39:0/3566135713 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5649dc346340 msgr2=0x5649dc346760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:07.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.743+0000 7f5236a00640 1 --2- 172.21.15.39:0/3566135713 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5649dc346340 0x5649dc346760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:07.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.743+0000 7f5236a00640 1 -- 172.21.15.39:0/3566135713 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5649dc26df10 con 0x5649dc34bea0 2024-03-18T16:09:07.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.743+0000 7f5237400640 1 --2- 172.21.15.39:0/3566135713 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5649dc346d30 0x5649dc34b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:09:07.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.743+0000 7f5236a00640 1 --2- 172.21.15.39:0/3566135713 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649dc34bea0 0x5649dc34e330 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f5224009ca0 tx=0x7f5224011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c9c02ba7a7e6b45a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:07.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.744+0000 7f522cc00640 1 -- 172.21.15.39:0/3566135713 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f522401e070 con 0x5649dc34bea0 2024-03-18T16:09:07.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.744+0000 7f522cc00640 1 -- 172.21.15.39:0/3566135713 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5224019b10 con 0x5649dc34bea0 2024-03-18T16:09:07.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.744+0000 7f522cc00640 1 -- 172.21.15.39:0/3566135713 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f522400ea30 con 0x5649dc34bea0 2024-03-18T16:09:07.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.744+0000 7f522fe00640 1 --2- 172.21.15.39:0/553665277 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f521c0a2800 0x7f521c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:07.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.744+0000 7f522fe00640 1 -- 172.21.15.39:0/553665277 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f52280f0a30 con 0x7f5238102140 2024-03-18T16:09:07.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.744+0000 7f5237e00640 1 --2- 172.21.15.39:0/553665277 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f521c0a2800 0x7f521c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:07.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.745+0000 7f52411e9740 1 -- 172.21.15.39:0/3566135713 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649dc34bea0 msgr2=0x5649dc34e330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:07.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.745+0000 7f52411e9740 1 --2- 172.21.15.39:0/3566135713 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649dc34bea0 0x5649dc34e330 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f5224009ca0 tx=0x7f5224011810 comp rx=0 tx=0).stop 2024-03-18T16:09:07.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.745+0000 7f52411e9740 1 -- 172.21.15.39:0/3566135713 shutdown_connections 2024-03-18T16:09:07.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.745+0000 7f52411e9740 1 --2- 172.21.15.39:0/3566135713 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649dc34bea0 0x5649dc34e330 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:07.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.745+0000 7f52411e9740 1 --2- 172.21.15.39:0/3566135713 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5649dc346d30 0x5649dc34b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:07.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.745+0000 7f52411e9740 1 --2- 172.21.15.39:0/3566135713 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5649dc346340 0x5649dc346760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:07.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.745+0000 7f52411e9740 1 -- 172.21.15.39:0/3566135713 >> 172.21.15.39:0/3566135713 conn(0x5649dc3418b0 msgr2=0x5649dc341cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:07.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.745+0000 7f52411e9740 1 -- 172.21.15.39:0/3566135713 shutdown_connections 2024-03-18T16:09:07.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.745+0000 7f5237e00640 1 --2- 172.21.15.39:0/553665277 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f521c0a2800 0x7f521c0a4cc0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x5649dc347590 tx=0x7f5230009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:07.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.745+0000 7f52411e9740 1 -- 172.21.15.39:0/3566135713 wait complete. 2024-03-18T16:09:07.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.745+0000 7f52411e9740 1 -- start start 2024-03-18T16:09:07.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.746+0000 7f52411e9740 1 Processor -- start 2024-03-18T16:09:07.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.746+0000 7f52411e9740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649dc346340 0x5649dc350470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:07.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.746+0000 7f52411e9740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5649dc346d30 0x5649dc3509b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:07.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.746+0000 7f52411e9740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5649dc34bea0 0x5649dc3579f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:07.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.746+0000 7f52411e9740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5649dc34ee50 con 0x5649dc346d30 2024-03-18T16:09:07.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.746+0000 7f52411e9740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5649dc266e20 con 0x5649dc346340 2024-03-18T16:09:07.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.746+0000 7f52411e9740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x5649dc34efd0 con 0x5649dc34bea0 2024-03-18T16:09:07.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.746+0000 7f52411e9740 10 client.0 _ll_register_callbacks cb 0x5649dc2667f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:09:07.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.746+0000 7f5237400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649dc346340 0x5649dc350470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:07.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.746+0000 7f5237400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649dc346340 0x5649dc350470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:35362/0 (socket says 172.21.15.39:35362) 2024-03-18T16:09:07.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.746+0000 7f5237400640 1 -- 172.21.15.39:0/2316060493 learned_addr learned my addr 172.21.15.39:0/2316060493 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:07.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.746+0000 7f5236a00640 1 --2- 172.21.15.39:0/2316060493 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5649dc346d30 0x5649dc3509b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:07.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.747+0000 7f5237e00640 1 --2- 172.21.15.39:0/2316060493 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5649dc34bea0 0x5649dc3579f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:07.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.747+0000 7f5237400640 1 -- 172.21.15.39:0/2316060493 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5649dc34bea0 msgr2=0x5649dc3579f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:07.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.747+0000 7f5237400640 1 --2- 172.21.15.39:0/2316060493 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5649dc34bea0 0x5649dc3579f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:07.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.747+0000 7f5237400640 1 -- 172.21.15.39:0/2316060493 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5649dc346d30 msgr2=0x5649dc3509b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:07.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.747+0000 7f5237400640 1 --2- 172.21.15.39:0/2316060493 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5649dc346d30 0x5649dc3509b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:07.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.747+0000 7f5237400640 1 -- 172.21.15.39:0/2316060493 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5649dbfbf2d0 con 0x5649dc346340 2024-03-18T16:09:07.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.747+0000 7f5237e00640 1 --2- 172.21.15.39:0/2316060493 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5649dc34bea0 0x5649dc3579f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:09:07.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.747+0000 7f5236a00640 1 --2- 172.21.15.39:0/2316060493 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5649dc346d30 0x5649dc3509b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:09:07.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.747+0000 7f5237400640 1 --2- 172.21.15.39:0/2316060493 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649dc346340 0x5649dc350470 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f522802c7b0 tx=0x7f52280dd110 comp rx=0 tx=0).ready entity=mon.2 client_cookie=38112e337f9c7232 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:07.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.747+0000 7f5221600640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:09:07.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.747+0000 7f5221600640 1 -- 172.21.15.39:0/2316060493 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f522804b070 con 0x5649dc346340 2024-03-18T16:09:07.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.747+0000 7f5221600640 1 -- 172.21.15.39:0/2316060493 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f52280c5350 con 0x5649dc346340 2024-03-18T16:09:07.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.747+0000 7f52411e9740 1 -- 172.21.15.39:0/2316060493 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5224019b10 con 0x5649dc346340 2024-03-18T16:09:07.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.747+0000 7f5221600640 1 -- 172.21.15.39:0/2316060493 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f52280bcb80 con 0x5649dc346340 2024-03-18T16:09:07.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.747+0000 7f52411e9740 10 client.25148 fetch_fsmap learned FSMap version 27 2024-03-18T16:09:07.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.747+0000 7f52411e9740 1 -- 172.21.15.39:0/2316060493 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5649dc0896b0 con 0x5649dc346340 2024-03-18T16:09:07.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.748+0000 7f5221600640 1 -- 172.21.15.39:0/2316060493 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5224019b10 con 0x5649dc346340 2024-03-18T16:09:07.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.748+0000 7f5221600640 1 -- 172.21.15.39:0/2316060493 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f52280dbb40 con 0x5649dc346340 2024-03-18T16:09:07.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.748+0000 7f52411e9740 10 client.25148 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T16:09:07.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.748+0000 7f52411e9740 10 client.25148 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-18T16:09:07.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.748+0000 7f52411e9740 20 client.25148 populate_metadata read hostname 'smithi039' 2024-03-18T16:09:07.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.748+0000 7f52411e9740 1 --2- 172.21.15.39:0/2316060493 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x5649dc35d1f0 0x5649dc35e660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:07.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.748+0000 7f52411e9740 4 client.25148 mds_command: new command op to 14463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:09:07.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.748+0000 7f52411e9740 1 -- 172.21.15.39:0/2316060493 --> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5649dc3082b0 con 0x5649dc35d1f0 2024-03-18T16:09:07.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.748+0000 7f5236a00640 1 --2- 172.21.15.39:0/2316060493 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x5649dc35d1f0 0x5649dc35e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:07.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.749+0000 7f5236a00640 1 --2- 172.21.15.39:0/2316060493 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x5649dc35d1f0 0x5649dc35e660 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=a1fe0d24d42c2d67 server_cookie=fb943ac592f0236e in_seq=0 out_seq=0 2024-03-18T16:09:07.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.749+0000 7f5221600640 10 client.25148 ms_handle_connect on v2:172.21.15.39:6840/3515326042 2024-03-18T16:09:07.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.750+0000 7f5221600640 1 -- 172.21.15.39:0/2316060493 <== mds.4 v2:172.21.15.39:6840/3515326042 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5649dc3082b0 con 0x5649dc35d1f0 2024-03-18T16:09:07.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.750+0000 7f5221600640 10 client.25148 handle_command_reply: tid=0 2024-03-18T16:09:07.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.750+0000 7f52411e9740 1 client.25148 shutdown 2024-03-18T16:09:07.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.751+0000 7f52411e9740 1 -- 172.21.15.39:0/2316060493 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649dc346340 msgr2=0x5649dc350470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:07.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.751+0000 7f52411e9740 1 --2- 172.21.15.39:0/2316060493 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649dc346340 0x5649dc350470 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f522802c7b0 tx=0x7f52280dd110 comp rx=0 tx=0).stop 2024-03-18T16:09:07.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.751+0000 7f52411e9740 1 -- 172.21.15.39:0/2316060493 shutdown_connections 2024-03-18T16:09:07.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.751+0000 7f52411e9740 1 --2- 172.21.15.39:0/2316060493 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x5649dc35d1f0 0x5649dc35e660 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:07.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.751+0000 7f52411e9740 1 --2- 172.21.15.39:0/2316060493 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5649dc34bea0 0x5649dc3579f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:07.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.751+0000 7f52411e9740 1 --2- 172.21.15.39:0/2316060493 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5649dc346d30 0x5649dc3509b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:07.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.751+0000 7f52411e9740 1 --2- 172.21.15.39:0/2316060493 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649dc346340 0x5649dc350470 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:07.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.751+0000 7f5221600640 0 client.25148 ms_handle_reset on v2:172.21.15.39:6840/3515326042 2024-03-18T16:09:07.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.751+0000 7f52411e9740 1 -- 172.21.15.39:0/2316060493 >> 172.21.15.39:0/2316060493 conn(0x5649dc3418b0 msgr2=0x5649dc340a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:07.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.751+0000 7f52411e9740 1 -- 172.21.15.39:0/2316060493 shutdown_connections 2024-03-18T16:09:07.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.751+0000 7f52411e9740 1 -- 172.21.15.39:0/2316060493 wait complete. 2024-03-18T16:09:07.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.751+0000 7f52411e9740 20 client.25148 trim_cache size 0 max 16384 2024-03-18T16:09:07.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.771+0000 7f52411e9740 1 Processor -- start 2024-03-18T16:09:07.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.771+0000 7f52411e9740 1 -- start start 2024-03-18T16:09:07.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.771+0000 7f52411e9740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5649dc346340 0x5649dc350080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:07.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.771+0000 7f52411e9740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649dc346d30 0x5649dc357980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:07.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.771+0000 7f52411e9740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5649dc34bea0 0x5649dc3505c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:07.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.771+0000 7f52411e9740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5649dc35c9a0 con 0x5649dc346340 2024-03-18T16:09:07.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.771+0000 7f52411e9740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5649dc35d070 con 0x5649dc346d30 2024-03-18T16:09:07.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.771+0000 7f52411e9740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x5649dc35d740 con 0x5649dc34bea0 2024-03-18T16:09:07.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.771+0000 7f5237400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5649dc346340 0x5649dc350080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:07.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.771+0000 7f5237e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5649dc34bea0 0x5649dc3505c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:07.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.771+0000 7f5237400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5649dc346340 0x5649dc350080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:42468/0 (socket says 172.21.15.39:42468) 2024-03-18T16:09:07.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.771+0000 7f5237400640 1 -- 172.21.15.39:0/3800265521 learned_addr learned my addr 172.21.15.39:0/3800265521 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:07.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.771+0000 7f5237e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5649dc34bea0 0x5649dc3505c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:36230/0 (socket says 172.21.15.39:36230) 2024-03-18T16:09:07.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.771+0000 7f5236a00640 1 --2- 172.21.15.39:0/3800265521 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649dc346d30 0x5649dc357980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:07.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.772+0000 7f5237400640 1 -- 172.21.15.39:0/3800265521 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5649dc34bea0 msgr2=0x5649dc3505c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:07.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.772+0000 7f5237400640 1 --2- 172.21.15.39:0/3800265521 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5649dc34bea0 0x5649dc3505c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:07.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.772+0000 7f5237400640 1 -- 172.21.15.39:0/3800265521 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649dc346d30 msgr2=0x5649dc357980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:07.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.772+0000 7f5237400640 1 --2- 172.21.15.39:0/3800265521 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649dc346d30 0x5649dc357980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:07.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.772+0000 7f5237400640 1 -- 172.21.15.39:0/3800265521 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5649dc342620 con 0x5649dc346340 2024-03-18T16:09:07.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.772+0000 7f5236a00640 1 --2- 172.21.15.39:0/3800265521 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649dc346d30 0x5649dc357980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:09:07.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.772+0000 7f5237400640 1 --2- 172.21.15.39:0/3800265521 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5649dc346340 0x5649dc350080 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f52280b9410 tx=0x7f52280f0d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce4690bc3c796e6b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:07.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.772+0000 7f5223e00640 1 -- 172.21.15.39:0/3800265521 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f522804b070 con 0x5649dc346340 2024-03-18T16:09:07.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.772+0000 7f5223e00640 1 -- 172.21.15.39:0/3800265521 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f52280bc7b0 con 0x5649dc346340 2024-03-18T16:09:07.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.772+0000 7f5223e00640 1 -- 172.21.15.39:0/3800265521 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52280bb180 con 0x5649dc346340 2024-03-18T16:09:07.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.773+0000 7f52411e9740 1 -- 172.21.15.39:0/3800265521 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5649dc346340 msgr2=0x5649dc350080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:07.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.773+0000 7f52411e9740 1 --2- 172.21.15.39:0/3800265521 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5649dc346340 0x5649dc350080 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f52280b9410 tx=0x7f52280f0d30 comp rx=0 tx=0).stop 2024-03-18T16:09:07.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.773+0000 7f52411e9740 1 -- 172.21.15.39:0/3800265521 shutdown_connections 2024-03-18T16:09:07.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.773+0000 7f52411e9740 1 --2- 172.21.15.39:0/3800265521 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649dc346d30 0x5649dc357980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:07.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.773+0000 7f52411e9740 1 --2- 172.21.15.39:0/3800265521 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5649dc34bea0 0x5649dc3505c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:07.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.773+0000 7f52411e9740 1 --2- 172.21.15.39:0/3800265521 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5649dc346340 0x5649dc350080 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:07.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.773+0000 7f52411e9740 1 -- 172.21.15.39:0/3800265521 >> 172.21.15.39:0/3800265521 conn(0x5649dc3418b0 msgr2=0x5649dc35bee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:07.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.773+0000 7f52411e9740 1 -- 172.21.15.39:0/3800265521 shutdown_connections 2024-03-18T16:09:07.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.773+0000 7f52411e9740 1 -- 172.21.15.39:0/3800265521 wait complete. 2024-03-18T16:09:07.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.773+0000 7f52411e9740 1 -- start start 2024-03-18T16:09:07.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.774+0000 7f52411e9740 1 Processor -- start 2024-03-18T16:09:07.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.774+0000 7f52411e9740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5649dc346340 0x5649dc3539f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:07.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.774+0000 7f52411e9740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649dc346d30 0x5649dc353f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:07.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.774+0000 7f52411e9740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5649dc34bea0 0x5649dc356e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:07.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.774+0000 7f52411e9740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5649dc35e4e0 con 0x5649dc346340 2024-03-18T16:09:07.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.774+0000 7f52411e9740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5649dc35de10 con 0x5649dc346d30 2024-03-18T16:09:07.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.774+0000 7f52411e9740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x5649dc35ebb0 con 0x5649dc34bea0 2024-03-18T16:09:07.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.774+0000 7f52411e9740 10 client.0 _ll_register_callbacks cb 0x7f5228009280 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:09:07.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.774+0000 7f5237400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5649dc346340 0x5649dc3539f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:07.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.774+0000 7f5236a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649dc346d30 0x5649dc353f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:07.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.774+0000 7f5236a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649dc346d30 0x5649dc353f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:35388/0 (socket says 172.21.15.39:35388) 2024-03-18T16:09:07.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.774+0000 7f5236a00640 1 -- 172.21.15.39:0/2957167101 learned_addr learned my addr 172.21.15.39:0/2957167101 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:07.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.774+0000 7f5237e00640 1 --2- 172.21.15.39:0/2957167101 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5649dc34bea0 0x5649dc356e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:07.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.775+0000 7f5236a00640 1 -- 172.21.15.39:0/2957167101 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5649dc34bea0 msgr2=0x5649dc356e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:07.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.775+0000 7f5236a00640 1 --2- 172.21.15.39:0/2957167101 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5649dc34bea0 0x5649dc356e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:07.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.775+0000 7f5236a00640 1 -- 172.21.15.39:0/2957167101 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5649dc346340 msgr2=0x5649dc3539f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:07.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.775+0000 7f5236a00640 1 --2- 172.21.15.39:0/2957167101 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5649dc346340 0x5649dc3539f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:07.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.775+0000 7f5236a00640 1 -- 172.21.15.39:0/2957167101 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5649dc357410 con 0x5649dc346d30 2024-03-18T16:09:07.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.775+0000 7f5236a00640 1 --2- 172.21.15.39:0/2957167101 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649dc346d30 0x5649dc353f30 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5224009bc0 tx=0x7f5224019f00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=26c2b5dd575953f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:07.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.775+0000 7f5221600640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:09:07.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.775+0000 7f5221600640 1 -- 172.21.15.39:0/2957167101 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f522401e040 con 0x5649dc346d30 2024-03-18T16:09:07.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.775+0000 7f52411e9740 1 -- 172.21.15.39:0/2957167101 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f52280bc7b0 con 0x5649dc346d30 2024-03-18T16:09:07.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.775+0000 7f5221600640 1 -- 172.21.15.39:0/2957167101 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f52240052b0 con 0x5649dc346d30 2024-03-18T16:09:07.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.775+0000 7f5221600640 1 -- 172.21.15.39:0/2957167101 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f522403c050 con 0x5649dc346d30 2024-03-18T16:09:07.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.775+0000 7f52411e9740 10 client.25154 fetch_fsmap learned FSMap version 27 2024-03-18T16:09:07.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.776+0000 7f52411e9740 1 -- 172.21.15.39:0/2957167101 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5649dc3c37f0 con 0x5649dc346d30 2024-03-18T16:09:07.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.776+0000 7f5221600640 1 -- 172.21.15.39:0/2957167101 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f52280bc7b0 con 0x5649dc346d30 2024-03-18T16:09:07.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.776+0000 7f5221600640 1 -- 172.21.15.39:0/2957167101 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f52240035f0 con 0x5649dc346d30 2024-03-18T16:09:07.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.776+0000 7f52411e9740 10 client.25154 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T16:09:07.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.776+0000 7f52411e9740 10 client.25154 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-18T16:09:07.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.776+0000 7f52411e9740 20 client.25154 populate_metadata read hostname 'smithi039' 2024-03-18T16:09:07.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.776+0000 7f52411e9740 1 --2- 172.21.15.39:0/2957167101 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x5649dc3c8200 0x5649dc3c6f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:07.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.776+0000 7f52411e9740 4 client.25154 mds_command: new command op to 14463 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T16:09:07.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.776+0000 7f52411e9740 1 -- 172.21.15.39:0/2957167101 --> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] -- command(tid 0: {"prefix": "status"}) -- 0x5649dc2b8190 con 0x5649dc3c8200 2024-03-18T16:09:07.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.776+0000 7f5237400640 1 --2- 172.21.15.39:0/2957167101 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x5649dc3c8200 0x5649dc3c6f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:07.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.777+0000 7f5237400640 1 --2- 172.21.15.39:0/2957167101 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x5649dc3c8200 0x5649dc3c6f10 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=6c15ec50764537e2 server_cookie=edb73e37fbd26a29 in_seq=0 out_seq=0 2024-03-18T16:09:07.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.777+0000 7f5221600640 10 client.25154 ms_handle_connect on v2:172.21.15.39:6840/3515326042 2024-03-18T16:09:07.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.777+0000 7f5221600640 1 -- 172.21.15.39:0/2957167101 <== mds.4 v2:172.21.15.39:6840/3515326042 1 ==== command_reply(tid 0: 0 ) ==== 8+0+321 (crc 0 0 0) 0x5649dc2b8190 con 0x5649dc3c8200 2024-03-18T16:09:07.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.777+0000 7f5221600640 10 client.25154 handle_command_reply: tid=0 2024-03-18T16:09:07.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.777+0000 7f52411e9740 1 client.25154 shutdown 2024-03-18T16:09:07.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.777+0000 7f52411e9740 1 -- 172.21.15.39:0/2957167101 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649dc346d30 msgr2=0x5649dc353f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:07.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.777+0000 7f52411e9740 1 --2- 172.21.15.39:0/2957167101 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649dc346d30 0x5649dc353f30 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f5224009bc0 tx=0x7f5224019f00 comp rx=0 tx=0).stop 2024-03-18T16:09:07.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.777+0000 7f52411e9740 1 -- 172.21.15.39:0/2957167101 shutdown_connections 2024-03-18T16:09:07.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.777+0000 7f52411e9740 1 --2- 172.21.15.39:0/2957167101 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x5649dc3c8200 0x5649dc3c6f10 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:07.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.777+0000 7f52411e9740 1 --2- 172.21.15.39:0/2957167101 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649dc346d30 0x5649dc353f30 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:07.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.777+0000 7f52411e9740 1 --2- 172.21.15.39:0/2957167101 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5649dc34bea0 0x5649dc356e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:07.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.777+0000 7f52411e9740 1 --2- 172.21.15.39:0/2957167101 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5649dc346340 0x5649dc3539f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:07.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.777+0000 7f5221600640 0 client.25154 ms_handle_reset on v2:172.21.15.39:6840/3515326042 2024-03-18T16:09:07.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.777+0000 7f52411e9740 1 -- 172.21.15.39:0/2957167101 >> 172.21.15.39:0/2957167101 conn(0x5649dc3418b0 msgr2=0x5649dc350060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:07.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.778+0000 7f52411e9740 1 -- 172.21.15.39:0/2957167101 shutdown_connections 2024-03-18T16:09:07.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.778+0000 7f52411e9740 1 -- 172.21.15.39:0/2957167101 wait complete. 2024-03-18T16:09:07.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.778+0000 7f52411e9740 20 client.25154 trim_cache size 0 max 16384 2024-03-18T16:09:07.784 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-03-18T16:09:07.784 INFO:teuthology.orchestra.run.smithi039.stdout: "cluster_fsid": "1892d8fc-e540-11ee-95c9-87774f69a715", 2024-03-18T16:09:07.784 INFO:teuthology.orchestra.run.smithi039.stdout: "whoami": 4, 2024-03-18T16:09:07.784 INFO:teuthology.orchestra.run.smithi039.stdout: "id": 14463, 2024-03-18T16:09:07.784 INFO:teuthology.orchestra.run.smithi039.stdout: "want_state": "up:active", 2024-03-18T16:09:07.784 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "up:active", 2024-03-18T16:09:07.784 INFO:teuthology.orchestra.run.smithi039.stdout: "fs_name": "cephfs", 2024-03-18T16:09:07.784 INFO:teuthology.orchestra.run.smithi039.stdout: "rank_uptime": 107.790747516, 2024-03-18T16:09:07.784 INFO:teuthology.orchestra.run.smithi039.stdout: "mdsmap_epoch": 27, 2024-03-18T16:09:07.784 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_epoch": 93, 2024-03-18T16:09:07.785 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_epoch_barrier": 89, 2024-03-18T16:09:07.785 INFO:teuthology.orchestra.run.smithi039.stdout: "uptime": 153.14031623 2024-03-18T16:09:07.785 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-03-18T16:09:07.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.778+0000 7f523ce00640 1 -- 172.21.15.39:0/553665277 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f521c0a2800 msgr2=0x7f521c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:07.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.778+0000 7f523ce00640 1 --2- 172.21.15.39:0/553665277 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f521c0a2800 0x7f521c0a4cc0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x5649dc347590 tx=0x7f5230009500 comp rx=0 tx=0).stop 2024-03-18T16:09:07.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.778+0000 7f523ce00640 1 -- 172.21.15.39:0/553665277 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5238102140 msgr2=0x7f5238199c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:07.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.778+0000 7f523ce00640 1 --2- 172.21.15.39:0/553665277 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5238102140 0x7f5238199c30 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f5228009230 tx=0x7f522802c650 comp rx=0 tx=0).stop 2024-03-18T16:09:07.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.778+0000 7f523ce00640 1 -- 172.21.15.39:0/553665277 shutdown_connections 2024-03-18T16:09:07.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.779+0000 7f523ce00640 1 --2- 172.21.15.39:0/553665277 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f52381073a0 0x7f52381a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:07.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.779+0000 7f523ce00640 1 --2- 172.21.15.39:0/553665277 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f521c0a2800 0x7f521c0a4cc0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:07.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.779+0000 7f523ce00640 1 --2- 172.21.15.39:0/553665277 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5238104a70 0x7f523819a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:07.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.779+0000 7f523ce00640 1 --2- 172.21.15.39:0/553665277 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5238102140 0x7f5238199c30 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:07.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.779+0000 7f523ce00640 1 -- 172.21.15.39:0/553665277 >> 172.21.15.39:0/553665277 conn(0x7f52380fbb50 msgr2=0x7f5238101830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:07.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.779+0000 7f523ce00640 1 -- 172.21.15.39:0/553665277 shutdown_connections 2024-03-18T16:09:07.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:07.779+0000 7f523ce00640 1 -- 172.21.15.39:0/553665277 wait complete. 2024-03-18T16:09:07.797 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-03-18T16:09:08.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.003+0000 7fe43ea00640 1 -- 172.21.15.39:0/2257120725 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe4380fffb0 msgr2=0x7fe4381003d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:08.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.003+0000 7fe43ea00640 1 --2- 172.21.15.39:0/2257120725 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe4380fffb0 0x7fe4381003d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fe42c009900 tx=0x7fe42c02bcd0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.003+0000 7fe43ea00640 1 -- 172.21.15.39:0/2257120725 shutdown_connections 2024-03-18T16:09:08.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.003+0000 7fe43ea00640 1 --2- 172.21.15.39:0/2257120725 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe4381012b0 0x7fe438101730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.004+0000 7fe43ea00640 1 --2- 172.21.15.39:0/2257120725 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe438100910 0x7fe438100d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.004+0000 7fe43ea00640 1 --2- 172.21.15.39:0/2257120725 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe4380fffb0 0x7fe4381003d0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.004+0000 7fe43ea00640 1 -- 172.21.15.39:0/2257120725 >> 172.21.15.39:0/2257120725 conn(0x7fe4380fbb50 msgr2=0x7fe4380fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:08.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.004+0000 7fe43ea00640 1 -- 172.21.15.39:0/2257120725 shutdown_connections 2024-03-18T16:09:08.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.004+0000 7fe43ea00640 1 -- 172.21.15.39:0/2257120725 wait complete. 2024-03-18T16:09:08.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.005+0000 7fe43ea00640 1 Processor -- start 2024-03-18T16:09:08.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.005+0000 7fe43ea00640 1 -- start start 2024-03-18T16:09:08.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.005+0000 7fe43ea00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe4380fffb0 0x7fe438199bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:08.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.005+0000 7fe43ea00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe438100910 0x7fe43819a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:08.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.006+0000 7fe43ea00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe4381012b0 0x7fe4381a1160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:08.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.006+0000 7fe43ea00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fe43810f560 con 0x7fe4380fffb0 2024-03-18T16:09:08.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.006+0000 7fe43ea00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fe43810f3e0 con 0x7fe4381012b0 2024-03-18T16:09:08.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.006+0000 7fe43ea00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fe43810f6e0 con 0x7fe438100910 2024-03-18T16:09:08.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.006+0000 7fe43cc00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe438100910 0x7fe43819a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:08.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.006+0000 7fe43d600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe4380fffb0 0x7fe438199bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:08.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.006+0000 7fe43cc00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe438100910 0x7fe43819a0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:36258/0 (socket says 172.21.15.39:36258) 2024-03-18T16:09:08.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.006+0000 7fe43e000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe4381012b0 0x7fe4381a1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:08.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.006+0000 7fe43cc00640 1 -- 172.21.15.39:0/719521800 learned_addr learned my addr 172.21.15.39:0/719521800 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:08.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.006+0000 7fe43cc00640 1 -- 172.21.15.39:0/719521800 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe4381012b0 msgr2=0x7fe4381a1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:08.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.006+0000 7fe43cc00640 1 --2- 172.21.15.39:0/719521800 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe4381012b0 0x7fe4381a1160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.006+0000 7fe43cc00640 1 -- 172.21.15.39:0/719521800 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe4380fffb0 msgr2=0x7fe438199bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:08.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.006+0000 7fe43cc00640 1 --2- 172.21.15.39:0/719521800 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe4380fffb0 0x7fe438199bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.006+0000 7fe43cc00640 1 -- 172.21.15.39:0/719521800 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4381a17d0 con 0x7fe438100910 2024-03-18T16:09:08.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.007+0000 7fe43cc00640 1 --2- 172.21.15.39:0/719521800 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe438100910 0x7fe43819a0f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fe42400e9f0 tx=0x7fe42400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:08.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.007+0000 7fe436000640 1 -- 172.21.15.39:0/719521800 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe42400cd80 con 0x7fe438100910 2024-03-18T16:09:08.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.007+0000 7fe436000640 1 -- 172.21.15.39:0/719521800 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe424010910 con 0x7fe438100910 2024-03-18T16:09:08.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.007+0000 7fe436000640 1 -- 172.21.15.39:0/719521800 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe424018990 con 0x7fe438100910 2024-03-18T16:09:08.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.007+0000 7fe43ea00640 1 -- 172.21.15.39:0/719521800 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4381a1ac0 con 0x7fe438100910 2024-03-18T16:09:08.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.007+0000 7fe43ea00640 1 -- 172.21.15.39:0/719521800 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4381a1fc0 con 0x7fe438100910 2024-03-18T16:09:08.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.010+0000 7fe436000640 1 -- 172.21.15.39:0/719521800 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fe424010ac0 con 0x7fe438100910 2024-03-18T16:09:08.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.011+0000 7fe436000640 1 --2- 172.21.15.39:0/719521800 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fe4200a2880 0x7fe4200a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:08.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.011+0000 7fe436000640 1 -- 172.21.15.39:0/719521800 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fe4240cf0a0 con 0x7fe438100910 2024-03-18T16:09:08.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.011+0000 7fe43d600640 1 --2- 172.21.15.39:0/719521800 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fe4200a2880 0x7fe4200a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:08.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.012+0000 7fe442e19740 1 Processor -- start 2024-03-18T16:09:08.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.012+0000 7fe442e19740 1 -- start start 2024-03-18T16:09:08.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.012+0000 7fe43d600640 1 --2- 172.21.15.39:0/719521800 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fe4200a2880 0x7fe4200a4d40 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fe42c02c2e0 tx=0x7fe42c02c920 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:08.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.012+0000 7fe442e19740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563557e0bb50 0x563557e0bf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:08.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.012+0000 7fe442e19740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563557e0c540 0x563557e11040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:08.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.012+0000 7fe442e19740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563557e116b0 0x563557e13b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:08.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.012+0000 7fe442e19740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x563557d67c50 con 0x563557e0bb50 2024-03-18T16:09:08.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.012+0000 7fe442e19740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x563557d109d0 con 0x563557e116b0 2024-03-18T16:09:08.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.012+0000 7fe442e19740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x563557d63cf0 con 0x563557e0c540 2024-03-18T16:09:08.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.013+0000 7fe43d600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563557e0bb50 0x563557e0bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:08.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.013+0000 7fe43d600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563557e0bb50 0x563557e0bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:42510/0 (socket says 172.21.15.39:42510) 2024-03-18T16:09:08.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.013+0000 7fe43d600640 1 -- 172.21.15.39:0/4122790348 learned_addr learned my addr 172.21.15.39:0/4122790348 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:08.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.013+0000 7fe43cc00640 1 --2- 172.21.15.39:0/4122790348 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563557e0c540 0x563557e11040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:08.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.013+0000 7fe43e000640 1 --2- 172.21.15.39:0/4122790348 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563557e116b0 0x563557e13b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:08.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.013+0000 7fe43cc00640 1 -- 172.21.15.39:0/4122790348 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563557e116b0 msgr2=0x563557e13b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:08.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.013+0000 7fe43cc00640 1 --2- 172.21.15.39:0/4122790348 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563557e116b0 0x563557e13b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.013+0000 7fe43cc00640 1 -- 172.21.15.39:0/4122790348 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563557e0bb50 msgr2=0x563557e0bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:08.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.013+0000 7fe43cc00640 1 --2- 172.21.15.39:0/4122790348 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563557e0bb50 0x563557e0bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.013+0000 7fe43cc00640 1 -- 172.21.15.39:0/4122790348 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563557d2af10 con 0x563557e0c540 2024-03-18T16:09:08.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.013+0000 7fe43cc00640 1 --2- 172.21.15.39:0/4122790348 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563557e0c540 0x563557e11040 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe4240cc810 tx=0x7fe42409b1e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3feab2b793502d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:08.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.014+0000 7fe42a600640 1 -- 172.21.15.39:0/4122790348 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe424099a60 con 0x563557e0c540 2024-03-18T16:09:08.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.014+0000 7fe42a600640 1 -- 172.21.15.39:0/4122790348 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe4240860b0 con 0x563557e0c540 2024-03-18T16:09:08.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.014+0000 7fe42a600640 1 -- 172.21.15.39:0/4122790348 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe42409ae10 con 0x563557e0c540 2024-03-18T16:09:08.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.014+0000 7fe442e19740 1 -- 172.21.15.39:0/4122790348 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563557e0c540 msgr2=0x563557e11040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:08.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.014+0000 7fe442e19740 1 --2- 172.21.15.39:0/4122790348 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563557e0c540 0x563557e11040 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fe4240cc810 tx=0x7fe42409b1e0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.014+0000 7fe442e19740 1 -- 172.21.15.39:0/4122790348 shutdown_connections 2024-03-18T16:09:08.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.014+0000 7fe442e19740 1 --2- 172.21.15.39:0/4122790348 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563557e116b0 0x563557e13b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.014+0000 7fe442e19740 1 --2- 172.21.15.39:0/4122790348 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563557e0c540 0x563557e11040 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.014+0000 7fe442e19740 1 --2- 172.21.15.39:0/4122790348 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563557e0bb50 0x563557e0bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.014+0000 7fe442e19740 1 -- 172.21.15.39:0/4122790348 >> 172.21.15.39:0/4122790348 conn(0x563557e070c0 msgr2=0x563557e074c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:08.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.014+0000 7fe442e19740 1 -- 172.21.15.39:0/4122790348 shutdown_connections 2024-03-18T16:09:08.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.014+0000 7fe442e19740 1 -- 172.21.15.39:0/4122790348 wait complete. 2024-03-18T16:09:08.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.015+0000 7fe442e19740 1 -- start start 2024-03-18T16:09:08.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.015+0000 7fe442e19740 1 Processor -- start 2024-03-18T16:09:08.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.015+0000 7fe442e19740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563557e0bb50 0x563557e15db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:08.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.015+0000 7fe442e19740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563557e0c540 0x563557e162f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:08.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.015+0000 7fe442e19740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563557e116b0 0x563557e1d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:08.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.015+0000 7fe442e19740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x563557e14660 con 0x563557e116b0 2024-03-18T16:09:08.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.016+0000 7fe442e19740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x563557d23e20 con 0x563557e0bb50 2024-03-18T16:09:08.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.016+0000 7fe442e19740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x563557e147e0 con 0x563557e0c540 2024-03-18T16:09:08.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.016+0000 7fe442e19740 10 client.0 _ll_register_callbacks cb 0x563557d237f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:09:08.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.016+0000 7fe43d600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563557e0bb50 0x563557e15db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:08.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.016+0000 7fe43cc00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563557e0c540 0x563557e162f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:08.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.016+0000 7fe43d600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563557e0bb50 0x563557e15db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:35428/0 (socket says 172.21.15.39:35428) 2024-03-18T16:09:08.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.016+0000 7fe43d600640 1 -- 172.21.15.39:0/3455816508 learned_addr learned my addr 172.21.15.39:0/3455816508 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:08.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.016+0000 7fe43e000640 1 --2- 172.21.15.39:0/3455816508 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563557e116b0 0x563557e1d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:08.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.016+0000 7fe43d600640 1 -- 172.21.15.39:0/3455816508 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563557e0c540 msgr2=0x563557e162f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:08.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.016+0000 7fe43d600640 1 --2- 172.21.15.39:0/3455816508 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563557e0c540 0x563557e162f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.016+0000 7fe43d600640 1 -- 172.21.15.39:0/3455816508 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563557e116b0 msgr2=0x563557e1d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:08.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.016+0000 7fe43d600640 1 --2- 172.21.15.39:0/3455816508 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563557e116b0 0x563557e1d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.016+0000 7fe43d600640 1 -- 172.21.15.39:0/3455816508 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563557a7c2d0 con 0x563557e0bb50 2024-03-18T16:09:08.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.016+0000 7fe43cc00640 1 --2- 172.21.15.39:0/3455816508 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563557e0c540 0x563557e162f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:09:08.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.017+0000 7fe43d600640 1 --2- 172.21.15.39:0/3455816508 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563557e0bb50 0x563557e15db0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fe42c009230 tx=0x7fe42c009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b16a244e99456d5d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:08.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.017+0000 7fe41ea00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:09:08.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.017+0000 7fe41ea00640 1 -- 172.21.15.39:0/3455816508 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe42c050070 con 0x563557e0bb50 2024-03-18T16:09:08.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.017+0000 7fe41ea00640 1 -- 172.21.15.39:0/3455816508 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fe42c004990 con 0x563557e0bb50 2024-03-18T16:09:08.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.017+0000 7fe442e19740 1 -- 172.21.15.39:0/3455816508 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe4240860b0 con 0x563557e0bb50 2024-03-18T16:09:08.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.017+0000 7fe41ea00640 1 -- 172.21.15.39:0/3455816508 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe42c05c050 con 0x563557e0bb50 2024-03-18T16:09:08.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.017+0000 7fe442e19740 10 client.25166 fetch_fsmap learned FSMap version 27 2024-03-18T16:09:08.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.017+0000 7fe442e19740 1 -- 172.21.15.39:0/3455816508 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x563557b466b0 con 0x563557e0bb50 2024-03-18T16:09:08.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.017+0000 7fe41ea00640 1 -- 172.21.15.39:0/3455816508 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe4240860b0 con 0x563557e0bb50 2024-03-18T16:09:08.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.018+0000 7fe41ea00640 1 -- 172.21.15.39:0/3455816508 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fe42c0037b0 con 0x563557e0bb50 2024-03-18T16:09:08.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.018+0000 7fe442e19740 10 client.25166 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T16:09:08.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.018+0000 7fe442e19740 10 client.25166 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-18T16:09:08.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.018+0000 7fe442e19740 20 client.25166 populate_metadata read hostname 'smithi039' 2024-03-18T16:09:08.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.018+0000 7fe442e19740 1 --2- 172.21.15.39:0/3455816508 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x563557e22aa0 0x563557e23f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:08.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.018+0000 7fe442e19740 4 client.25166 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:09:08.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.018+0000 7fe442e19740 1 -- 172.21.15.39:0/3455816508 --> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563557dc52b0 con 0x563557e22aa0 2024-03-18T16:09:08.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.018+0000 7fe43cc00640 1 --2- 172.21.15.39:0/3455816508 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x563557e22aa0 0x563557e23f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:08.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.019+0000 7fe43cc00640 1 --2- 172.21.15.39:0/3455816508 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x563557e22aa0 0x563557e23f10 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=10a11e94c13968c8 server_cookie=f18777e92388ae3c in_seq=0 out_seq=0 2024-03-18T16:09:08.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.019+0000 7fe41ea00640 10 client.25166 ms_handle_connect on v2:172.21.15.194:6836/1301464246 2024-03-18T16:09:08.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.020+0000 7fe41ea00640 1 -- 172.21.15.39:0/3455816508 <== mds.3 v2:172.21.15.194:6836/1301464246 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563557dc52b0 con 0x563557e22aa0 2024-03-18T16:09:08.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.020+0000 7fe41ea00640 10 client.25166 handle_command_reply: tid=0 2024-03-18T16:09:08.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.020+0000 7fe442e19740 1 client.25166 shutdown 2024-03-18T16:09:08.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.020+0000 7fe442e19740 1 -- 172.21.15.39:0/3455816508 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563557e0bb50 msgr2=0x563557e15db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:08.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.020+0000 7fe442e19740 1 --2- 172.21.15.39:0/3455816508 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563557e0bb50 0x563557e15db0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fe42c009230 tx=0x7fe42c009260 comp rx=0 tx=0).stop 2024-03-18T16:09:08.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.020+0000 7fe442e19740 1 -- 172.21.15.39:0/3455816508 shutdown_connections 2024-03-18T16:09:08.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.020+0000 7fe442e19740 1 --2- 172.21.15.39:0/3455816508 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x563557e22aa0 0x563557e23f10 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.020+0000 7fe442e19740 1 --2- 172.21.15.39:0/3455816508 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563557e116b0 0x563557e1d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.020+0000 7fe442e19740 1 --2- 172.21.15.39:0/3455816508 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563557e0c540 0x563557e162f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.020+0000 7fe41ea00640 0 client.25166 ms_handle_reset on v2:172.21.15.194:6836/1301464246 2024-03-18T16:09:08.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.020+0000 7fe442e19740 1 --2- 172.21.15.39:0/3455816508 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563557e0bb50 0x563557e15db0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.020+0000 7fe442e19740 1 -- 172.21.15.39:0/3455816508 >> 172.21.15.39:0/3455816508 conn(0x563557e070c0 msgr2=0x563557dfda70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:08.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.020+0000 7fe442e19740 1 -- 172.21.15.39:0/3455816508 shutdown_connections 2024-03-18T16:09:08.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.020+0000 7fe442e19740 1 -- 172.21.15.39:0/3455816508 wait complete. 2024-03-18T16:09:08.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.021+0000 7fe442e19740 20 client.25166 trim_cache size 0 max 16384 2024-03-18T16:09:08.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.040+0000 7fe442e19740 1 Processor -- start 2024-03-18T16:09:08.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.040+0000 7fe442e19740 1 -- start start 2024-03-18T16:09:08.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.041+0000 7fe442e19740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563557e0bb50 0x563557e23840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:08.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.041+0000 7fe442e19740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563557e0c540 0x563557e1d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:08.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.041+0000 7fe442e19740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563557e116b0 0x563557e16250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:08.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.041+0000 7fe442e19740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x563557e22250 con 0x563557e116b0 2024-03-18T16:09:08.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.041+0000 7fe442e19740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x563557e22920 con 0x563557e0c540 2024-03-18T16:09:08.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.041+0000 7fe442e19740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x563557e22ff0 con 0x563557e0bb50 2024-03-18T16:09:08.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.041+0000 7fe43d600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563557e0bb50 0x563557e23840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:08.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.041+0000 7fe43d600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563557e0bb50 0x563557e23840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:33912/0 (socket says 172.21.15.39:33912) 2024-03-18T16:09:08.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.041+0000 7fe43d600640 1 -- 172.21.15.39:0/946230020 learned_addr learned my addr 172.21.15.39:0/946230020 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:08.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.041+0000 7fe43cc00640 1 --2- 172.21.15.39:0/946230020 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563557e0c540 0x563557e1d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:08.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.041+0000 7fe43e000640 1 --2- 172.21.15.39:0/946230020 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563557e116b0 0x563557e16250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:08.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.041+0000 7fe43d600640 1 -- 172.21.15.39:0/946230020 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563557e0c540 msgr2=0x563557e1d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:08.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.041+0000 7fe43d600640 1 --2- 172.21.15.39:0/946230020 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563557e0c540 0x563557e1d250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.041+0000 7fe43d600640 1 -- 172.21.15.39:0/946230020 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563557e116b0 msgr2=0x563557e16250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:08.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.041+0000 7fe43d600640 1 --2- 172.21.15.39:0/946230020 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563557e116b0 0x563557e16250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.041+0000 7fe43d600640 1 -- 172.21.15.39:0/946230020 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563557e11400 con 0x563557e0bb50 2024-03-18T16:09:08.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.041+0000 7fe43cc00640 1 --2- 172.21.15.39:0/946230020 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563557e0c540 0x563557e1d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:09:08.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.041+0000 7fe43e000640 1 --2- 172.21.15.39:0/946230020 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563557e116b0 0x563557e16250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:09:08.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.042+0000 7fe43d600640 1 --2- 172.21.15.39:0/946230020 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563557e0bb50 0x563557e23840 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fe42c02c7b0 tx=0x7fe42c004610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=49ac3a886dbe1720 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:08.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.042+0000 7fe429c00640 1 -- 172.21.15.39:0/946230020 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe42c065070 con 0x563557e0bb50 2024-03-18T16:09:08.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.042+0000 7fe429c00640 1 -- 172.21.15.39:0/946230020 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe42c009520 con 0x563557e0bb50 2024-03-18T16:09:08.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.042+0000 7fe429c00640 1 -- 172.21.15.39:0/946230020 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe42c050040 con 0x563557e0bb50 2024-03-18T16:09:08.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.042+0000 7fe442e19740 1 -- 172.21.15.39:0/946230020 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563557e0bb50 msgr2=0x563557e23840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:08.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.042+0000 7fe442e19740 1 --2- 172.21.15.39:0/946230020 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563557e0bb50 0x563557e23840 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fe42c02c7b0 tx=0x7fe42c004610 comp rx=0 tx=0).stop 2024-03-18T16:09:08.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.042+0000 7fe442e19740 1 -- 172.21.15.39:0/946230020 shutdown_connections 2024-03-18T16:09:08.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.042+0000 7fe442e19740 1 --2- 172.21.15.39:0/946230020 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563557e0c540 0x563557e1d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.042+0000 7fe442e19740 1 --2- 172.21.15.39:0/946230020 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563557e116b0 0x563557e16250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.042+0000 7fe442e19740 1 --2- 172.21.15.39:0/946230020 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563557e0bb50 0x563557e23840 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.043+0000 7fe442e19740 1 -- 172.21.15.39:0/946230020 >> 172.21.15.39:0/946230020 conn(0x563557e070c0 msgr2=0x563557e074a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:08.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.043+0000 7fe442e19740 1 -- 172.21.15.39:0/946230020 shutdown_connections 2024-03-18T16:09:08.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.043+0000 7fe442e19740 1 -- 172.21.15.39:0/946230020 wait complete. 2024-03-18T16:09:08.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.043+0000 7fe442e19740 1 -- start start 2024-03-18T16:09:08.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.043+0000 7fe442e19740 1 Processor -- start 2024-03-18T16:09:08.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.044+0000 7fe442e19740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563557e0bb50 0x563557e1b420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:08.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.044+0000 7fe442e19740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563557e0c540 0x563557e1b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:08.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.044+0000 7fe442e19740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563557e116b0 0x563557e1bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:08.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.044+0000 7fe442e19740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x563557e23d90 con 0x563557e0bb50 2024-03-18T16:09:08.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.044+0000 7fe442e19740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x563557e236c0 con 0x563557e116b0 2024-03-18T16:09:08.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.044+0000 7fe442e19740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x563557e24460 con 0x563557e0c540 2024-03-18T16:09:08.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.044+0000 7fe442e19740 10 client.0 _ll_register_callbacks cb 0x7fe42c004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:09:08.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.044+0000 7fe43cc00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563557e0c540 0x563557e1b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:08.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.044+0000 7fe43e000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563557e116b0 0x563557e1bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:08.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.044+0000 7fe43d600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563557e0bb50 0x563557e1b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:08.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.044+0000 7fe43cc00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563557e0c540 0x563557e1b960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:33928/0 (socket says 172.21.15.39:33928) 2024-03-18T16:09:08.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.044+0000 7fe43cc00640 1 -- 172.21.15.39:0/2957315659 learned_addr learned my addr 172.21.15.39:0/2957315659 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:08.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.044+0000 7fe43cc00640 1 -- 172.21.15.39:0/2957315659 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563557e116b0 msgr2=0x563557e1bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:08.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.044+0000 7fe43cc00640 1 --2- 172.21.15.39:0/2957315659 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563557e116b0 0x563557e1bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.044+0000 7fe43cc00640 1 -- 172.21.15.39:0/2957315659 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563557e0bb50 msgr2=0x563557e1b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:08.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.044+0000 7fe43cc00640 1 --2- 172.21.15.39:0/2957315659 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563557e0bb50 0x563557e1b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.044+0000 7fe43cc00640 1 -- 172.21.15.39:0/2957315659 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563557e1c810 con 0x563557e0c540 2024-03-18T16:09:08.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.044+0000 7fe43cc00640 1 --2- 172.21.15.39:0/2957315659 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563557e0c540 0x563557e1b960 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fe424098130 tx=0x7fe424097b30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9f51bee4e2528a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:08.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.044+0000 7fe41ea00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:09:08.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.045+0000 7fe41ea00640 1 -- 172.21.15.39:0/2957315659 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4240af060 con 0x563557e0c540 2024-03-18T16:09:08.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.045+0000 7fe41ea00640 1 -- 172.21.15.39:0/2957315659 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fe424097dd0 con 0x563557e0c540 2024-03-18T16:09:08.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.045+0000 7fe442e19740 1 -- 172.21.15.39:0/2957315659 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe42c009520 con 0x563557e0c540 2024-03-18T16:09:08.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.045+0000 7fe41ea00640 1 -- 172.21.15.39:0/2957315659 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe4240860b0 con 0x563557e0c540 2024-03-18T16:09:08.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.045+0000 7fe442e19740 10 client.25189 fetch_fsmap learned FSMap version 27 2024-03-18T16:09:08.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.045+0000 7fe442e19740 1 -- 172.21.15.39:0/2957315659 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x563557e88b90 con 0x563557e0c540 2024-03-18T16:09:08.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.045+0000 7fe41ea00640 1 -- 172.21.15.39:0/2957315659 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe42c009520 con 0x563557e0c540 2024-03-18T16:09:08.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.045+0000 7fe41ea00640 1 -- 172.21.15.39:0/2957315659 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fe4240974b0 con 0x563557e0c540 2024-03-18T16:09:08.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.045+0000 7fe442e19740 10 client.25189 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T16:09:08.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.045+0000 7fe442e19740 10 client.25189 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-18T16:09:08.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.045+0000 7fe442e19740 20 client.25189 populate_metadata read hostname 'smithi039' 2024-03-18T16:09:08.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.045+0000 7fe442e19740 1 --2- 172.21.15.39:0/2957315659 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x563557e8d260 0x563557e8bf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:08.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.045+0000 7fe442e19740 4 client.25189 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T16:09:08.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.045+0000 7fe442e19740 1 -- 172.21.15.39:0/2957315659 --> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] -- command(tid 0: {"prefix": "status"}) -- 0x563557d75190 con 0x563557e8d260 2024-03-18T16:09:08.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.046+0000 7fe43d600640 1 --2- 172.21.15.39:0/2957315659 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x563557e8d260 0x563557e8bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:08.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.046+0000 7fe43d600640 1 --2- 172.21.15.39:0/2957315659 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x563557e8d260 0x563557e8bf70 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=cdc84bfafacacb20 server_cookie=a26696eef0e66036 in_seq=0 out_seq=0 2024-03-18T16:09:08.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.046+0000 7fe41ea00640 10 client.25189 ms_handle_connect on v2:172.21.15.194:6836/1301464246 2024-03-18T16:09:08.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.046+0000 7fe41ea00640 1 -- 172.21.15.39:0/2957315659 <== mds.3 v2:172.21.15.194:6836/1301464246 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x563557d75190 con 0x563557e8d260 2024-03-18T16:09:08.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.046+0000 7fe41ea00640 10 client.25189 handle_command_reply: tid=0 2024-03-18T16:09:08.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.046+0000 7fe442e19740 1 client.25189 shutdown 2024-03-18T16:09:08.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.046+0000 7fe442e19740 1 -- 172.21.15.39:0/2957315659 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563557e0c540 msgr2=0x563557e1b960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:08.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.046+0000 7fe442e19740 1 --2- 172.21.15.39:0/2957315659 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563557e0c540 0x563557e1b960 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fe424098130 tx=0x7fe424097b30 comp rx=0 tx=0).stop 2024-03-18T16:09:08.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.047+0000 7fe442e19740 1 -- 172.21.15.39:0/2957315659 shutdown_connections 2024-03-18T16:09:08.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.047+0000 7fe442e19740 1 --2- 172.21.15.39:0/2957315659 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x563557e8d260 0x563557e8bf70 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.047+0000 7fe442e19740 1 --2- 172.21.15.39:0/2957315659 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563557e116b0 0x563557e1bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.047+0000 7fe41ea00640 0 client.25189 ms_handle_reset on v2:172.21.15.194:6836/1301464246 2024-03-18T16:09:08.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.047+0000 7fe442e19740 1 --2- 172.21.15.39:0/2957315659 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563557e0c540 0x563557e1b960 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.047+0000 7fe442e19740 1 --2- 172.21.15.39:0/2957315659 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563557e0bb50 0x563557e1b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.047+0000 7fe442e19740 1 -- 172.21.15.39:0/2957315659 >> 172.21.15.39:0/2957315659 conn(0x563557e070c0 msgr2=0x563557e0a320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:08.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.047+0000 7fe442e19740 1 -- 172.21.15.39:0/2957315659 shutdown_connections 2024-03-18T16:09:08.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.047+0000 7fe442e19740 1 -- 172.21.15.39:0/2957315659 wait complete. 2024-03-18T16:09:08.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.047+0000 7fe442e19740 20 client.25189 trim_cache size 0 max 16384 2024-03-18T16:09:08.053 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-03-18T16:09:08.053 INFO:teuthology.orchestra.run.smithi039.stdout: "cluster_fsid": "1892d8fc-e540-11ee-95c9-87774f69a715", 2024-03-18T16:09:08.053 INFO:teuthology.orchestra.run.smithi039.stdout: "whoami": 3, 2024-03-18T16:09:08.053 INFO:teuthology.orchestra.run.smithi039.stdout: "id": 24524, 2024-03-18T16:09:08.053 INFO:teuthology.orchestra.run.smithi039.stdout: "want_state": "up:active", 2024-03-18T16:09:08.054 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "up:active", 2024-03-18T16:09:08.054 INFO:teuthology.orchestra.run.smithi039.stdout: "fs_name": "cephfs", 2024-03-18T16:09:08.054 INFO:teuthology.orchestra.run.smithi039.stdout: "rank_uptime": 108.814178673, 2024-03-18T16:09:08.054 INFO:teuthology.orchestra.run.smithi039.stdout: "mdsmap_epoch": 27, 2024-03-18T16:09:08.054 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_epoch": 93, 2024-03-18T16:09:08.054 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_epoch_barrier": 89, 2024-03-18T16:09:08.054 INFO:teuthology.orchestra.run.smithi039.stdout: "uptime": 164.791533356 2024-03-18T16:09:08.054 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-03-18T16:09:08.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.047+0000 7fe43ea00640 1 -- 172.21.15.39:0/719521800 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fe4200a2880 msgr2=0x7fe4200a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:08.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.047+0000 7fe43ea00640 1 --2- 172.21.15.39:0/719521800 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fe4200a2880 0x7fe4200a4d40 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fe42c02c2e0 tx=0x7fe42c02c920 comp rx=0 tx=0).stop 2024-03-18T16:09:08.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.047+0000 7fe43ea00640 1 -- 172.21.15.39:0/719521800 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe438100910 msgr2=0x7fe43819a0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:08.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.047+0000 7fe43ea00640 1 --2- 172.21.15.39:0/719521800 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe438100910 0x7fe43819a0f0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fe42400e9f0 tx=0x7fe42400eeb0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.047+0000 7fe43ea00640 1 -- 172.21.15.39:0/719521800 shutdown_connections 2024-03-18T16:09:08.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.047+0000 7fe43ea00640 1 --2- 172.21.15.39:0/719521800 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe4381012b0 0x7fe4381a1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.047+0000 7fe43ea00640 1 --2- 172.21.15.39:0/719521800 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fe4200a2880 0x7fe4200a4d40 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.047+0000 7fe43ea00640 1 --2- 172.21.15.39:0/719521800 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe438100910 0x7fe43819a0f0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.047+0000 7fe43ea00640 1 --2- 172.21.15.39:0/719521800 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe4380fffb0 0x7fe438199bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.048+0000 7fe43ea00640 1 -- 172.21.15.39:0/719521800 >> 172.21.15.39:0/719521800 conn(0x7fe4380fbb50 msgr2=0x7fe4380fd650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:08.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.048+0000 7fe43ea00640 1 -- 172.21.15.39:0/719521800 shutdown_connections 2024-03-18T16:09:08.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.048+0000 7fe43ea00640 1 -- 172.21.15.39:0/719521800 wait complete. 2024-03-18T16:09:08.063 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:07 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/3242826704' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:09:08.063 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:07 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/4229551780' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:09:08.067 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-18T16:09:08.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:07 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/3242826704' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:09:08.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:07 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/4229551780' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:09:08.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.283+0000 7f62b9600640 1 -- 172.21.15.39:0/837973996 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f62b4103270 msgr2=0x7f62b4109b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:08.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.283+0000 7f62b9600640 1 --2- 172.21.15.39:0/837973996 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f62b4103270 0x7f62b4109b00 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f62ac009ca0 tx=0x7f62ac02c0f0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.284+0000 7f62b9600640 1 -- 172.21.15.39:0/837973996 shutdown_connections 2024-03-18T16:09:08.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.284+0000 7f62b9600640 1 --2- 172.21.15.39:0/837973996 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f62b4103270 0x7f62b4109b00 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.284+0000 7f62b9600640 1 --2- 172.21.15.39:0/837973996 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f62b41028b0 0x7f62b4102d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.284+0000 7f62b9600640 1 --2- 172.21.15.39:0/837973996 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f62b41016c0 0x7f62b4101ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.284+0000 7f62b9600640 1 -- 172.21.15.39:0/837973996 >> 172.21.15.39:0/837973996 conn(0x7f62b40fcc50 msgr2=0x7f62b40ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:08.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.284+0000 7f62b9600640 1 -- 172.21.15.39:0/837973996 shutdown_connections 2024-03-18T16:09:08.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.284+0000 7f62b9600640 1 -- 172.21.15.39:0/837973996 wait complete. 2024-03-18T16:09:08.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.285+0000 7f62b9600640 1 Processor -- start 2024-03-18T16:09:08.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.285+0000 7f62b9600640 1 -- start start 2024-03-18T16:09:08.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.286+0000 7f62b9600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f62b41016c0 0x7f62b4197910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:08.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.286+0000 7f62b9600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f62b41028b0 0x7f62b4197e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:08.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.286+0000 7f62b9600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f62b4103270 0x7f62b419eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:08.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.286+0000 7f62b9600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f62b410d320 con 0x7f62b4103270 2024-03-18T16:09:08.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.286+0000 7f62b9600640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f62b410d1a0 con 0x7f62b41028b0 2024-03-18T16:09:08.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.286+0000 7f62b9600640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f62b410d4a0 con 0x7f62b41016c0 2024-03-18T16:09:08.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.286+0000 7f62b3e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f62b41016c0 0x7f62b4197910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:08.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.286+0000 7f62b3400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f62b41028b0 0x7f62b4197e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:08.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.286+0000 7f62b3e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f62b41016c0 0x7f62b4197910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:33940/0 (socket says 172.21.15.39:33940) 2024-03-18T16:09:08.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.286+0000 7f62b3e00640 1 -- 172.21.15.39:0/1513992195 learned_addr learned my addr 172.21.15.39:0/1513992195 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:08.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.286+0000 7f62b8c00640 1 --2- 172.21.15.39:0/1513992195 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f62b4103270 0x7f62b419eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:08.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.286+0000 7f62b3400640 1 -- 172.21.15.39:0/1513992195 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f62b41016c0 msgr2=0x7f62b4197910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:08.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.286+0000 7f62b3400640 1 --2- 172.21.15.39:0/1513992195 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f62b41016c0 0x7f62b4197910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.286+0000 7f62b3400640 1 -- 172.21.15.39:0/1513992195 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f62b4103270 msgr2=0x7f62b419eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:08.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.286+0000 7f62b3400640 1 --2- 172.21.15.39:0/1513992195 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f62b4103270 0x7f62b419eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.286+0000 7f62b3400640 1 -- 172.21.15.39:0/1513992195 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62b419f530 con 0x7f62b41028b0 2024-03-18T16:09:08.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.286+0000 7f62b8c00640 1 --2- 172.21.15.39:0/1513992195 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f62b4103270 0x7f62b419eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:09:08.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.286+0000 7f62b3e00640 1 --2- 172.21.15.39:0/1513992195 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f62b41016c0 0x7f62b4197910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:09:08.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.287+0000 7f62b3400640 1 --2- 172.21.15.39:0/1513992195 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f62b41028b0 0x7f62b4197e50 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f62a000e9f0 tx=0x7f62a000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:08.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.287+0000 7f62b0c00640 1 -- 172.21.15.39:0/1513992195 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a000ccc0 con 0x7f62b41028b0 2024-03-18T16:09:08.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.287+0000 7f62b0c00640 1 -- 172.21.15.39:0/1513992195 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62a0010910 con 0x7f62b41028b0 2024-03-18T16:09:08.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.287+0000 7f62b0c00640 1 -- 172.21.15.39:0/1513992195 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a0018960 con 0x7f62b41028b0 2024-03-18T16:09:08.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.287+0000 7f62b9600640 1 -- 172.21.15.39:0/1513992195 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62b419f820 con 0x7f62b41028b0 2024-03-18T16:09:08.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.287+0000 7f62b9600640 1 -- 172.21.15.39:0/1513992195 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62b419fd20 con 0x7f62b41028b0 2024-03-18T16:09:08.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.290+0000 7f62bd9dd740 1 Processor -- start 2024-03-18T16:09:08.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.290+0000 7f62bd9dd740 1 -- start start 2024-03-18T16:09:08.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.290+0000 7f62b0c00640 1 -- 172.21.15.39:0/1513992195 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f62a0010ac0 con 0x7f62b41028b0 2024-03-18T16:09:08.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.291+0000 7f62bd9dd740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557a9f96d750 0x557a9f96db70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:08.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.291+0000 7f62bd9dd740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557a9f96e140 0x557a9f972c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:08.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.291+0000 7f62bd9dd740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557a9f9732b0 0x557a9f975740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:08.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.291+0000 7f62bd9dd740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x557a9f8cdc50 con 0x557a9f9732b0 2024-03-18T16:09:08.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.291+0000 7f62bd9dd740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x557a9f8769d0 con 0x557a9f96d750 2024-03-18T16:09:08.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.291+0000 7f62bd9dd740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x557a9f8c9cf0 con 0x557a9f96e140 2024-03-18T16:09:08.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.291+0000 7f62b8c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557a9f96d750 0x557a9f96db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:08.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.291+0000 7f62b8c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557a9f96d750 0x557a9f96db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:56530/0 (socket says 172.21.15.39:56530) 2024-03-18T16:09:08.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.291+0000 7f62b8c00640 1 -- 172.21.15.39:0/647772047 learned_addr learned my addr 172.21.15.39:0/647772047 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:08.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.291+0000 7f62b3400640 1 --2- 172.21.15.39:0/647772047 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557a9f9732b0 0x557a9f975740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:08.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.291+0000 7f62b3e00640 1 --2- 172.21.15.39:0/647772047 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557a9f96e140 0x557a9f972c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:08.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.291+0000 7f62b8c00640 1 -- 172.21.15.39:0/647772047 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557a9f96e140 msgr2=0x557a9f972c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:08.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.291+0000 7f62b8c00640 1 --2- 172.21.15.39:0/647772047 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557a9f96e140 0x557a9f972c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.291+0000 7f62b8c00640 1 -- 172.21.15.39:0/647772047 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557a9f9732b0 msgr2=0x557a9f975740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:08.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.291+0000 7f62b8c00640 1 --2- 172.21.15.39:0/647772047 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557a9f9732b0 0x557a9f975740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.291+0000 7f62b8c00640 1 -- 172.21.15.39:0/647772047 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557a9f890f10 con 0x557a9f96d750 2024-03-18T16:09:08.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.292+0000 7f62b8c00640 1 --2- 172.21.15.39:0/647772047 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557a9f96d750 0x557a9f96db70 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f62ac0324c0 tx=0x7f62ac005f70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc26db75458b5ea7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:08.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.292+0000 7f62a9200640 1 -- 172.21.15.39:0/647772047 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62ac043020 con 0x557a9f96d750 2024-03-18T16:09:08.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.292+0000 7f62a9200640 1 -- 172.21.15.39:0/647772047 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62ac032c90 con 0x557a9f96d750 2024-03-18T16:09:08.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.292+0000 7f62a9200640 1 -- 172.21.15.39:0/647772047 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62ac039040 con 0x557a9f96d750 2024-03-18T16:09:08.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.292+0000 7f62b0c00640 1 --2- 172.21.15.39:0/1513992195 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f62980a2800 0x7f62980a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:08.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.293+0000 7f62b0c00640 1 -- 172.21.15.39:0/1513992195 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f62a00cf2b0 con 0x7f62b41028b0 2024-03-18T16:09:08.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.293+0000 7f62b3e00640 1 --2- 172.21.15.39:0/1513992195 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f62980a2800 0x7f62980a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:08.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.293+0000 7f62bd9dd740 1 -- 172.21.15.39:0/647772047 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557a9f96d750 msgr2=0x557a9f96db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:08.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.293+0000 7f62bd9dd740 1 --2- 172.21.15.39:0/647772047 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557a9f96d750 0x557a9f96db70 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f62ac0324c0 tx=0x7f62ac005f70 comp rx=0 tx=0).stop 2024-03-18T16:09:08.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.294+0000 7f62bd9dd740 1 -- 172.21.15.39:0/647772047 shutdown_connections 2024-03-18T16:09:08.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.294+0000 7f62bd9dd740 1 --2- 172.21.15.39:0/647772047 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557a9f9732b0 0x557a9f975740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.294+0000 7f62b3e00640 1 --2- 172.21.15.39:0/1513992195 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f62980a2800 0x7f62980a4cc0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x557a9f96e540 tx=0x7f62a4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:08.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.294+0000 7f62bd9dd740 1 --2- 172.21.15.39:0/647772047 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557a9f96e140 0x557a9f972c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.294+0000 7f62bd9dd740 1 --2- 172.21.15.39:0/647772047 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557a9f96d750 0x557a9f96db70 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.294+0000 7f62bd9dd740 1 -- 172.21.15.39:0/647772047 >> 172.21.15.39:0/647772047 conn(0x557a9f9648b0 msgr2=0x557a9f964cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:08.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.294+0000 7f62bd9dd740 1 -- 172.21.15.39:0/647772047 shutdown_connections 2024-03-18T16:09:08.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.295+0000 7f62bd9dd740 1 -- 172.21.15.39:0/647772047 wait complete. 2024-03-18T16:09:08.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.295+0000 7f62bd9dd740 1 -- start start 2024-03-18T16:09:08.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.295+0000 7f62bd9dd740 1 Processor -- start 2024-03-18T16:09:08.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.296+0000 7f62bd9dd740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557a9f96d750 0x557a9f9778a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:08.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.296+0000 7f62bd9dd740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557a9f96e140 0x557a9f977de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:08.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.296+0000 7f62bd9dd740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557a9f9732b0 0x557a9f97ee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:08.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.296+0000 7f62bd9dd740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x557a9f976200 con 0x557a9f96d750 2024-03-18T16:09:08.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.296+0000 7f62bd9dd740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x557a9f889e20 con 0x557a9f96e140 2024-03-18T16:09:08.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.296+0000 7f62bd9dd740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x557a9f976380 con 0x557a9f9732b0 2024-03-18T16:09:08.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.296+0000 7f62bd9dd740 10 client.0 _ll_register_callbacks cb 0x557a9f8897f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:09:08.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.296+0000 7f62b3400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557a9f96e140 0x557a9f977de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:08.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.296+0000 7f62b8c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557a9f9732b0 0x557a9f97ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:08.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.296+0000 7f62b3e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557a9f96d750 0x557a9f9778a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:08.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.297+0000 7f62b8c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557a9f9732b0 0x557a9f97ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:33958/0 (socket says 172.21.15.39:33958) 2024-03-18T16:09:08.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.297+0000 7f62b3400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557a9f96e140 0x557a9f977de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:56544/0 (socket says 172.21.15.39:56544) 2024-03-18T16:09:08.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.297+0000 7f62b8c00640 1 -- 172.21.15.39:0/34949776 learned_addr learned my addr 172.21.15.39:0/34949776 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:08.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.297+0000 7f62b3400640 1 -- 172.21.15.39:0/34949776 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557a9f9732b0 msgr2=0x557a9f97ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:08.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.297+0000 7f62b3400640 1 --2- 172.21.15.39:0/34949776 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557a9f9732b0 0x557a9f97ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.297+0000 7f62b3400640 1 -- 172.21.15.39:0/34949776 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557a9f96d750 msgr2=0x557a9f9778a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:08.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.297+0000 7f62b3400640 1 --2- 172.21.15.39:0/34949776 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557a9f96d750 0x557a9f9778a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.297+0000 7f62b3400640 1 -- 172.21.15.39:0/34949776 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557a9f5e22d0 con 0x557a9f96e140 2024-03-18T16:09:08.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.297+0000 7f62b8c00640 1 --2- 172.21.15.39:0/34949776 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557a9f9732b0 0x557a9f97ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:09:08.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.297+0000 7f62b3400640 1 --2- 172.21.15.39:0/34949776 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557a9f96e140 0x557a9f977de0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f62a000c6a0 tx=0x7f62a009b3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=400ccfe292164745 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:08.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.297+0000 7f629d600640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:09:08.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.297+0000 7f629d600640 1 -- 172.21.15.39:0/34949776 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a009b860 con 0x557a9f96e140 2024-03-18T16:09:08.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.298+0000 7f629d600640 1 -- 172.21.15.39:0/34949776 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f62a00860e0 con 0x557a9f96e140 2024-03-18T16:09:08.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.298+0000 7f62bd9dd740 1 -- 172.21.15.39:0/34949776 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f62ac032c90 con 0x557a9f96e140 2024-03-18T16:09:08.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.298+0000 7f629d600640 1 -- 172.21.15.39:0/34949776 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f62a009f9b0 con 0x557a9f96e140 2024-03-18T16:09:08.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.298+0000 7f62bd9dd740 10 client.25184 fetch_fsmap learned FSMap version 27 2024-03-18T16:09:08.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.298+0000 7f62bd9dd740 1 -- 172.21.15.39:0/34949776 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x557a9f6ac6b0 con 0x557a9f96e140 2024-03-18T16:09:08.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.298+0000 7f629d600640 1 -- 172.21.15.39:0/34949776 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62ac032c90 con 0x557a9f96e140 2024-03-18T16:09:08.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.298+0000 7f629d600640 1 -- 172.21.15.39:0/34949776 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f62a0018b00 con 0x557a9f96e140 2024-03-18T16:09:08.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.298+0000 7f62bd9dd740 10 client.25184 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T16:09:08.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.298+0000 7f62bd9dd740 10 client.25184 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-18T16:09:08.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.298+0000 7f62bd9dd740 20 client.25184 populate_metadata read hostname 'smithi039' 2024-03-18T16:09:08.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.298+0000 7f62bd9dd740 1 --2- 172.21.15.39:0/34949776 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x557a9f984570 0x557a9f9859e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:08.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.298+0000 7f62bd9dd740 4 client.25184 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:09:08.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.298+0000 7f62bd9dd740 1 -- 172.21.15.39:0/34949776 --> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557a9f92b2b0 con 0x557a9f984570 2024-03-18T16:09:08.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.299+0000 7f62b3e00640 1 --2- 172.21.15.39:0/34949776 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x557a9f984570 0x557a9f9859e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:08.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.299+0000 7f62b3e00640 1 --2- 172.21.15.39:0/34949776 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x557a9f984570 0x557a9f9859e0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=cac7985640a96b59 server_cookie=429acb2d76b84e92 in_seq=0 out_seq=0 2024-03-18T16:09:08.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.299+0000 7f629d600640 10 client.25184 ms_handle_connect on v2:172.21.15.196:6836/895247251 2024-03-18T16:09:08.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.300+0000 7f629d600640 1 -- 172.21.15.39:0/34949776 <== mds.2 v2:172.21.15.196:6836/895247251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557a9f92b2b0 con 0x557a9f984570 2024-03-18T16:09:08.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.300+0000 7f629d600640 10 client.25184 handle_command_reply: tid=0 2024-03-18T16:09:08.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.300+0000 7f62bd9dd740 1 client.25184 shutdown 2024-03-18T16:09:08.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.300+0000 7f62bd9dd740 1 -- 172.21.15.39:0/34949776 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557a9f96e140 msgr2=0x557a9f977de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:08.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.300+0000 7f62bd9dd740 1 --2- 172.21.15.39:0/34949776 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557a9f96e140 0x557a9f977de0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f62a000c6a0 tx=0x7f62a009b3a0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.300+0000 7f62bd9dd740 1 -- 172.21.15.39:0/34949776 shutdown_connections 2024-03-18T16:09:08.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.300+0000 7f62bd9dd740 1 --2- 172.21.15.39:0/34949776 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557a9f96e140 0x557a9f977de0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.300+0000 7f62bd9dd740 1 --2- 172.21.15.39:0/34949776 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x557a9f984570 0x557a9f9859e0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.300+0000 7f62bd9dd740 1 --2- 172.21.15.39:0/34949776 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557a9f9732b0 0x557a9f97ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.300+0000 7f62bd9dd740 1 --2- 172.21.15.39:0/34949776 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557a9f96d750 0x557a9f9778a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.300+0000 7f629d600640 0 client.25184 ms_handle_reset on v2:172.21.15.196:6836/895247251 2024-03-18T16:09:08.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.300+0000 7f62bd9dd740 1 -- 172.21.15.39:0/34949776 >> 172.21.15.39:0/34949776 conn(0x557a9f9648b0 msgr2=0x557a9f963a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:08.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.301+0000 7f62bd9dd740 1 -- 172.21.15.39:0/34949776 shutdown_connections 2024-03-18T16:09:08.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.301+0000 7f62bd9dd740 1 -- 172.21.15.39:0/34949776 wait complete. 2024-03-18T16:09:08.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.301+0000 7f62bd9dd740 20 client.25184 trim_cache size 0 max 16384 2024-03-18T16:09:08.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.321+0000 7f62bd9dd740 1 Processor -- start 2024-03-18T16:09:08.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.321+0000 7f62bd9dd740 1 -- start start 2024-03-18T16:09:08.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.321+0000 7f62bd9dd740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557a9f96d750 0x557a9f9774b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:08.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.321+0000 7f62bd9dd740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557a9f96e140 0x557a9f96cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:08.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.321+0000 7f62bd9dd740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557a9f9732b0 0x557a9f9779f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:08.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.321+0000 7f62bd9dd740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x557a9f983d20 con 0x557a9f96e140 2024-03-18T16:09:08.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.321+0000 7f62bd9dd740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x557a9f9843f0 con 0x557a9f96d750 2024-03-18T16:09:08.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.321+0000 7f62bd9dd740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x557a9f984ac0 con 0x557a9f9732b0 2024-03-18T16:09:08.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.321+0000 7f62b3e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557a9f96d750 0x557a9f9774b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:08.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.321+0000 7f62b8c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557a9f9732b0 0x557a9f9779f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:08.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.321+0000 7f62b3e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557a9f96d750 0x557a9f9774b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:56548/0 (socket says 172.21.15.39:56548) 2024-03-18T16:09:08.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.321+0000 7f62b3e00640 1 -- 172.21.15.39:0/2098301193 learned_addr learned my addr 172.21.15.39:0/2098301193 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:08.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.321+0000 7f62b3400640 1 --2- 172.21.15.39:0/2098301193 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557a9f96e140 0x557a9f96cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:08.326 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.322+0000 7f62b3e00640 1 -- 172.21.15.39:0/2098301193 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557a9f9732b0 msgr2=0x557a9f9779f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:08.326 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.322+0000 7f62b3e00640 1 --2- 172.21.15.39:0/2098301193 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557a9f9732b0 0x557a9f9779f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.326 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.322+0000 7f62b3e00640 1 -- 172.21.15.39:0/2098301193 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557a9f96e140 msgr2=0x557a9f96cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:08.326 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.322+0000 7f62b3e00640 1 --2- 172.21.15.39:0/2098301193 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557a9f96e140 0x557a9f96cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.326 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.322+0000 7f62b3e00640 1 -- 172.21.15.39:0/2098301193 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557a9f965620 con 0x557a9f96d750 2024-03-18T16:09:08.326 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.322+0000 7f62b3e00640 1 --2- 172.21.15.39:0/2098301193 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557a9f96d750 0x557a9f9774b0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f62a4009cc0 tx=0x7f62a401b000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d6350a598fa43695 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:08.326 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.322+0000 7f629fe00640 1 -- 172.21.15.39:0/2098301193 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a402b420 con 0x557a9f96d750 2024-03-18T16:09:08.326 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.322+0000 7f629fe00640 1 -- 172.21.15.39:0/2098301193 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62a4030070 con 0x557a9f96d750 2024-03-18T16:09:08.326 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.322+0000 7f629fe00640 1 -- 172.21.15.39:0/2098301193 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a401f400 con 0x557a9f96d750 2024-03-18T16:09:08.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.323+0000 7f62bd9dd740 1 -- 172.21.15.39:0/2098301193 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557a9f96d750 msgr2=0x557a9f9774b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:08.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.323+0000 7f62bd9dd740 1 --2- 172.21.15.39:0/2098301193 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557a9f96d750 0x557a9f9774b0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f62a4009cc0 tx=0x7f62a401b000 comp rx=0 tx=0).stop 2024-03-18T16:09:08.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.323+0000 7f62bd9dd740 1 -- 172.21.15.39:0/2098301193 shutdown_connections 2024-03-18T16:09:08.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.323+0000 7f62bd9dd740 1 --2- 172.21.15.39:0/2098301193 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557a9f9732b0 0x557a9f9779f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.323+0000 7f62bd9dd740 1 --2- 172.21.15.39:0/2098301193 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557a9f96e140 0x557a9f96cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.323+0000 7f62bd9dd740 1 --2- 172.21.15.39:0/2098301193 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557a9f96d750 0x557a9f9774b0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.323+0000 7f62bd9dd740 1 -- 172.21.15.39:0/2098301193 >> 172.21.15.39:0/2098301193 conn(0x557a9f9648b0 msgr2=0x557a9f9728f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:08.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.323+0000 7f62bd9dd740 1 -- 172.21.15.39:0/2098301193 shutdown_connections 2024-03-18T16:09:08.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.323+0000 7f62bd9dd740 1 -- 172.21.15.39:0/2098301193 wait complete. 2024-03-18T16:09:08.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.323+0000 7f62bd9dd740 1 -- start start 2024-03-18T16:09:08.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.324+0000 7f62bd9dd740 1 Processor -- start 2024-03-18T16:09:08.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.324+0000 7f62bd9dd740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557a9f96d750 0x557a9f978db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:08.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.324+0000 7f62bd9dd740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557a9f96e140 0x557a9f97b300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:08.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.324+0000 7f62bd9dd740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557a9f9732b0 0x557a9f97d850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:08.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.324+0000 7f62bd9dd740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x557a9f985860 con 0x557a9f96e140 2024-03-18T16:09:08.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.324+0000 7f62bd9dd740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x557a9f985190 con 0x557a9f96d750 2024-03-18T16:09:08.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.324+0000 7f62bd9dd740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x557a9f985f30 con 0x557a9f9732b0 2024-03-18T16:09:08.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.324+0000 7f62bd9dd740 10 client.0 _ll_register_callbacks cb 0x7f62a009b630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:09:08.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.324+0000 7f62b3e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557a9f96d750 0x557a9f978db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:08.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.324+0000 7f62b3e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557a9f96d750 0x557a9f978db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:56562/0 (socket says 172.21.15.39:56562) 2024-03-18T16:09:08.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.324+0000 7f62b3e00640 1 -- 172.21.15.39:0/3050528635 learned_addr learned my addr 172.21.15.39:0/3050528635 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:08.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.324+0000 7f62b3400640 1 --2- 172.21.15.39:0/3050528635 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557a9f96e140 0x557a9f97b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:08.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.324+0000 7f62b8c00640 1 --2- 172.21.15.39:0/3050528635 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557a9f9732b0 0x557a9f97d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:08.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.325+0000 7f62b3e00640 1 -- 172.21.15.39:0/3050528635 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557a9f9732b0 msgr2=0x557a9f97d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:08.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.325+0000 7f62b3e00640 1 --2- 172.21.15.39:0/3050528635 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557a9f9732b0 0x557a9f97d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.325+0000 7f62b3e00640 1 -- 172.21.15.39:0/3050528635 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557a9f96e140 msgr2=0x557a9f97b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:08.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.325+0000 7f62b3e00640 1 --2- 172.21.15.39:0/3050528635 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557a9f96e140 0x557a9f97b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.325+0000 7f62b3e00640 1 -- 172.21.15.39:0/3050528635 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557a9f97dd90 con 0x557a9f96d750 2024-03-18T16:09:08.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.325+0000 7f62b3e00640 1 --2- 172.21.15.39:0/3050528635 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557a9f96d750 0x557a9f978db0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f62a4003fd0 tx=0x7f62a4004140 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bcf10ad3763a2aa2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:08.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.325+0000 7f629d600640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:09:08.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.325+0000 7f629d600640 1 -- 172.21.15.39:0/3050528635 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62a4039070 con 0x557a9f96d750 2024-03-18T16:09:08.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.325+0000 7f629d600640 1 -- 172.21.15.39:0/3050528635 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f62a402b420 con 0x557a9f96d750 2024-03-18T16:09:08.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.325+0000 7f62bd9dd740 1 -- 172.21.15.39:0/3050528635 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f62a4030070 con 0x557a9f96d750 2024-03-18T16:09:08.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.326+0000 7f629d600640 1 -- 172.21.15.39:0/3050528635 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f62a4046050 con 0x557a9f96d750 2024-03-18T16:09:08.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.326+0000 7f62bd9dd740 10 client.25190 fetch_fsmap learned FSMap version 27 2024-03-18T16:09:08.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.326+0000 7f62bd9dd740 1 -- 172.21.15.39:0/3050528635 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x557a9f97f030 con 0x557a9f96d750 2024-03-18T16:09:08.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.326+0000 7f629d600640 1 -- 172.21.15.39:0/3050528635 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62a4030070 con 0x557a9f96d750 2024-03-18T16:09:08.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.326+0000 7f629d600640 1 -- 172.21.15.39:0/3050528635 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f62a403d780 con 0x557a9f96d750 2024-03-18T16:09:08.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.326+0000 7f62bd9dd740 10 client.25190 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T16:09:08.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.326+0000 7f62bd9dd740 10 client.25190 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-18T16:09:08.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.326+0000 7f62bd9dd740 20 client.25190 populate_metadata read hostname 'smithi039' 2024-03-18T16:09:08.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.326+0000 7f62bd9dd740 1 --2- 172.21.15.39:0/3050528635 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x557a9f9f2f00 0x557a9f9f1c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:08.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.326+0000 7f62bd9dd740 4 client.25190 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T16:09:08.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.326+0000 7f62bd9dd740 1 -- 172.21.15.39:0/3050528635 --> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] -- command(tid 0: {"prefix": "status"}) -- 0x557a9f8db190 con 0x557a9f9f2f00 2024-03-18T16:09:08.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.326+0000 7f62b3400640 1 --2- 172.21.15.39:0/3050528635 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x557a9f9f2f00 0x557a9f9f1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:08.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.327+0000 7f62b3400640 1 --2- 172.21.15.39:0/3050528635 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x557a9f9f2f00 0x557a9f9f1c10 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5e6c3b0ead8326ca server_cookie=b70250e196118e98 in_seq=0 out_seq=0 2024-03-18T16:09:08.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.327+0000 7f629d600640 10 client.25190 ms_handle_connect on v2:172.21.15.196:6836/895247251 2024-03-18T16:09:08.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.327+0000 7f629d600640 1 -- 172.21.15.39:0/3050528635 <== mds.2 v2:172.21.15.196:6836/895247251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x557a9f8db190 con 0x557a9f9f2f00 2024-03-18T16:09:08.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.327+0000 7f629d600640 10 client.25190 handle_command_reply: tid=0 2024-03-18T16:09:08.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.327+0000 7f62bd9dd740 1 client.25190 shutdown 2024-03-18T16:09:08.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.327+0000 7f62bd9dd740 1 -- 172.21.15.39:0/3050528635 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557a9f96d750 msgr2=0x557a9f978db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:08.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.327+0000 7f62bd9dd740 1 --2- 172.21.15.39:0/3050528635 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557a9f96d750 0x557a9f978db0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f62a4003fd0 tx=0x7f62a4004140 comp rx=0 tx=0).stop 2024-03-18T16:09:08.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.327+0000 7f62bd9dd740 1 -- 172.21.15.39:0/3050528635 shutdown_connections 2024-03-18T16:09:08.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.327+0000 7f62bd9dd740 1 --2- 172.21.15.39:0/3050528635 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x557a9f9f2f00 0x557a9f9f1c10 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.327+0000 7f62bd9dd740 1 --2- 172.21.15.39:0/3050528635 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557a9f9732b0 0x557a9f97d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.327+0000 7f62bd9dd740 1 --2- 172.21.15.39:0/3050528635 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557a9f96e140 0x557a9f97b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.327+0000 7f62bd9dd740 1 --2- 172.21.15.39:0/3050528635 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557a9f96d750 0x557a9f978db0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.327+0000 7f62bd9dd740 1 -- 172.21.15.39:0/3050528635 >> 172.21.15.39:0/3050528635 conn(0x557a9f9648b0 msgr2=0x557a9f9774b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:08.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.327+0000 7f629d600640 0 client.25190 ms_handle_reset on v2:172.21.15.196:6836/895247251 2024-03-18T16:09:08.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.327+0000 7f62bd9dd740 1 -- 172.21.15.39:0/3050528635 shutdown_connections 2024-03-18T16:09:08.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.327+0000 7f62bd9dd740 1 -- 172.21.15.39:0/3050528635 wait complete. 2024-03-18T16:09:08.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.328+0000 7f62bd9dd740 20 client.25190 trim_cache size 0 max 16384 2024-03-18T16:09:08.333 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-03-18T16:09:08.333 INFO:teuthology.orchestra.run.smithi039.stdout: "cluster_fsid": "1892d8fc-e540-11ee-95c9-87774f69a715", 2024-03-18T16:09:08.333 INFO:teuthology.orchestra.run.smithi039.stdout: "whoami": 2, 2024-03-18T16:09:08.334 INFO:teuthology.orchestra.run.smithi039.stdout: "id": 24535, 2024-03-18T16:09:08.334 INFO:teuthology.orchestra.run.smithi039.stdout: "want_state": "up:active", 2024-03-18T16:09:08.334 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "up:active", 2024-03-18T16:09:08.334 INFO:teuthology.orchestra.run.smithi039.stdout: "fs_name": "cephfs", 2024-03-18T16:09:08.334 INFO:teuthology.orchestra.run.smithi039.stdout: "rank_uptime": 110.11188650299999, 2024-03-18T16:09:08.334 INFO:teuthology.orchestra.run.smithi039.stdout: "mdsmap_epoch": 27, 2024-03-18T16:09:08.334 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_epoch": 93, 2024-03-18T16:09:08.334 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_epoch_barrier": 89, 2024-03-18T16:09:08.334 INFO:teuthology.orchestra.run.smithi039.stdout: "uptime": 167.900341602 2024-03-18T16:09:08.334 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-03-18T16:09:08.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.328+0000 7f62b9600640 1 -- 172.21.15.39:0/1513992195 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f62980a2800 msgr2=0x7f62980a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:08.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.328+0000 7f62b9600640 1 --2- 172.21.15.39:0/1513992195 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f62980a2800 0x7f62980a4cc0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x557a9f96e540 tx=0x7f62a4005f90 comp rx=0 tx=0).stop 2024-03-18T16:09:08.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.328+0000 7f62b9600640 1 -- 172.21.15.39:0/1513992195 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f62b41028b0 msgr2=0x7f62b4197e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:08.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.328+0000 7f62b9600640 1 --2- 172.21.15.39:0/1513992195 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f62b41028b0 0x7f62b4197e50 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f62a000e9f0 tx=0x7f62a000eeb0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.328+0000 7f62b9600640 1 -- 172.21.15.39:0/1513992195 shutdown_connections 2024-03-18T16:09:08.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.328+0000 7f62b9600640 1 --2- 172.21.15.39:0/1513992195 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f62b41028b0 0x7f62b4197e50 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.328+0000 7f62b9600640 1 --2- 172.21.15.39:0/1513992195 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f62980a2800 0x7f62980a4cc0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.328+0000 7f62b9600640 1 --2- 172.21.15.39:0/1513992195 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f62b4103270 0x7f62b419eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.328+0000 7f62b9600640 1 --2- 172.21.15.39:0/1513992195 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f62b41016c0 0x7f62b4197910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.328+0000 7f62b9600640 1 -- 172.21.15.39:0/1513992195 >> 172.21.15.39:0/1513992195 conn(0x7f62b40fcc50 msgr2=0x7f62b4105bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:08.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.328+0000 7f62b9600640 1 -- 172.21.15.39:0/1513992195 shutdown_connections 2024-03-18T16:09:08.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.329+0000 7f62b9600640 1 -- 172.21.15.39:0/1513992195 wait complete. 2024-03-18T16:09:08.347 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-18T16:09:08.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:07 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/3242826704' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:09:08.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:07 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/4229551780' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:09:08.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.568+0000 7fe0c6c00640 1 -- 172.21.15.39:0/2728609057 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe0c0074240 msgr2=0x7fe0c00746c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:08.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.568+0000 7fe0c6c00640 1 --2- 172.21.15.39:0/2728609057 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe0c0074240 0x7fe0c00746c0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fe0b0009ca0 tx=0x7fe0b002c100 comp rx=0 tx=0).stop 2024-03-18T16:09:08.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.568+0000 7fe0c6c00640 1 -- 172.21.15.39:0/2728609057 shutdown_connections 2024-03-18T16:09:08.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.568+0000 7fe0c6c00640 1 --2- 172.21.15.39:0/2728609057 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe0c01096b0 0x7fe0c010bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.568+0000 7fe0c6c00640 1 --2- 172.21.15.39:0/2728609057 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe0c0074240 0x7fe0c00746c0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.568+0000 7fe0c6c00640 1 --2- 172.21.15.39:0/2728609057 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe0c0075820 0x7fe0c0073c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.568+0000 7fe0c6c00640 1 -- 172.21.15.39:0/2728609057 >> 172.21.15.39:0/2728609057 conn(0x7fe0c00fcd60 msgr2=0x7fe0c00ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:08.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.568+0000 7fe0c6c00640 1 -- 172.21.15.39:0/2728609057 shutdown_connections 2024-03-18T16:09:08.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.568+0000 7fe0c6c00640 1 -- 172.21.15.39:0/2728609057 wait complete. 2024-03-18T16:09:08.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.569+0000 7fe0c6c00640 1 Processor -- start 2024-03-18T16:09:08.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.569+0000 7fe0c6c00640 1 -- start start 2024-03-18T16:09:08.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.569+0000 7fe0c6c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe0c0074240 0x7fe0c0199da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:08.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.569+0000 7fe0c6c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe0c0075820 0x7fe0c019a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:08.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.569+0000 7fe0c6c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe0c01096b0 0x7fe0c01a1310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:08.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.569+0000 7fe0c6c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fe0c010f630 con 0x7fe0c0075820 2024-03-18T16:09:08.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.569+0000 7fe0c6c00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fe0c010f4b0 con 0x7fe0c0074240 2024-03-18T16:09:08.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.569+0000 7fe0c6c00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fe0c010f7b0 con 0x7fe0c01096b0 2024-03-18T16:09:08.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.570+0000 7fe0bf400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe0c0075820 0x7fe0c019a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:08.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.570+0000 7fe0c4c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe0c01096b0 0x7fe0c01a1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:08.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.570+0000 7fe0bfe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe0c0074240 0x7fe0c0199da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:08.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.570+0000 7fe0c4c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe0c01096b0 0x7fe0c01a1310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:34006/0 (socket says 172.21.15.39:34006) 2024-03-18T16:09:08.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.570+0000 7fe0bfe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe0c0074240 0x7fe0c0199da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:56582/0 (socket says 172.21.15.39:56582) 2024-03-18T16:09:08.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.570+0000 7fe0bf400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe0c0075820 0x7fe0c019a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:32894/0 (socket says 172.21.15.39:32894) 2024-03-18T16:09:08.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.570+0000 7fe0c4c00640 1 -- 172.21.15.39:0/2892901510 learned_addr learned my addr 172.21.15.39:0/2892901510 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:08.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.570+0000 7fe0bfe00640 1 -- 172.21.15.39:0/2892901510 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe0c01096b0 msgr2=0x7fe0c01a1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:08.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.570+0000 7fe0bfe00640 1 --2- 172.21.15.39:0/2892901510 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe0c01096b0 0x7fe0c01a1310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.570+0000 7fe0bfe00640 1 -- 172.21.15.39:0/2892901510 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe0c0075820 msgr2=0x7fe0c019a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:08.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.570+0000 7fe0bfe00640 1 --2- 172.21.15.39:0/2892901510 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe0c0075820 0x7fe0c019a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.570+0000 7fe0bfe00640 1 -- 172.21.15.39:0/2892901510 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0c01a19b0 con 0x7fe0c0074240 2024-03-18T16:09:08.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.570+0000 7fe0bfe00640 1 --2- 172.21.15.39:0/2892901510 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe0c0074240 0x7fe0c0199da0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fe0a800e9f0 tx=0x7fe0a800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:08.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.571+0000 7fe0bcc00640 1 -- 172.21.15.39:0/2892901510 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0a800ccc0 con 0x7fe0c0074240 2024-03-18T16:09:08.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.571+0000 7fe0bcc00640 1 -- 172.21.15.39:0/2892901510 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe0a8010910 con 0x7fe0c0074240 2024-03-18T16:09:08.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.571+0000 7fe0bcc00640 1 -- 172.21.15.39:0/2892901510 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0a8018960 con 0x7fe0c0074240 2024-03-18T16:09:08.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.571+0000 7fe0c6c00640 1 -- 172.21.15.39:0/2892901510 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0c01a1ca0 con 0x7fe0c0074240 2024-03-18T16:09:08.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.571+0000 7fe0c6c00640 1 -- 172.21.15.39:0/2892901510 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0c01a2190 con 0x7fe0c0074240 2024-03-18T16:09:08.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.574+0000 7fe0c99c4740 1 Processor -- start 2024-03-18T16:09:08.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.574+0000 7fe0c99c4740 1 -- start start 2024-03-18T16:09:08.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.575+0000 7fe0c99c4740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cc6f38a160 0x55cc6f38a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:08.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.575+0000 7fe0c99c4740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc6f38ab50 0x55cc6f38f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:08.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.575+0000 7fe0c99c4740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cc6f38fcc0 0x55cc6f392150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:08.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.575+0000 7fe0c99c4740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55cc6f2eec50 con 0x55cc6f38a160 2024-03-18T16:09:08.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.575+0000 7fe0c99c4740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55cc6f2979d0 con 0x55cc6f38ab50 2024-03-18T16:09:08.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.575+0000 7fe0c99c4740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55cc6f2eacf0 con 0x55cc6f38fcc0 2024-03-18T16:09:08.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.575+0000 7fe0bf400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cc6f38fcc0 0x55cc6f392150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:08.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.575+0000 7fe0c4c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cc6f38a160 0x55cc6f38a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:08.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.575+0000 7fe0bf400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cc6f38fcc0 0x55cc6f392150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:34014/0 (socket says 172.21.15.39:34014) 2024-03-18T16:09:08.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.575+0000 7fe0bf400640 1 -- 172.21.15.39:0/84146573 learned_addr learned my addr 172.21.15.39:0/84146573 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:08.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.575+0000 7fe0bf400640 1 -- 172.21.15.39:0/84146573 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc6f38ab50 msgr2=0x55cc6f38f650 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:09:08.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.575+0000 7fe0bf400640 1 --2- 172.21.15.39:0/84146573 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc6f38ab50 0x55cc6f38f650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.575+0000 7fe0bf400640 1 -- 172.21.15.39:0/84146573 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cc6f38a160 msgr2=0x55cc6f38a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:08.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.575+0000 7fe0bf400640 1 --2- 172.21.15.39:0/84146573 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cc6f38a160 0x55cc6f38a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.575+0000 7fe0bf400640 1 -- 172.21.15.39:0/84146573 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cc6f2b1f10 con 0x55cc6f38fcc0 2024-03-18T16:09:08.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.576+0000 7fe0bf400640 1 --2- 172.21.15.39:0/84146573 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cc6f38fcc0 0x55cc6f392150 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fe0b0005e30 tx=0x7fe0b0005f60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b18cb8a5552cd3a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:08.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.576+0000 7fe0bcc00640 1 -- 172.21.15.39:0/2892901510 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fe0a8010ac0 con 0x7fe0c0074240 2024-03-18T16:09:08.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.576+0000 7fe0b5200640 1 -- 172.21.15.39:0/84146573 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0b0043070 con 0x55cc6f38fcc0 2024-03-18T16:09:08.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.576+0000 7fe0b5200640 1 -- 172.21.15.39:0/84146573 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe0b0003970 con 0x55cc6f38fcc0 2024-03-18T16:09:08.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.576+0000 7fe0b5200640 1 -- 172.21.15.39:0/84146573 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0b0039040 con 0x55cc6f38fcc0 2024-03-18T16:09:08.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.576+0000 7fe0c99c4740 1 -- 172.21.15.39:0/84146573 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cc6f38fcc0 msgr2=0x55cc6f392150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:08.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.576+0000 7fe0c99c4740 1 --2- 172.21.15.39:0/84146573 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cc6f38fcc0 0x55cc6f392150 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fe0b0005e30 tx=0x7fe0b0005f60 comp rx=0 tx=0).stop 2024-03-18T16:09:08.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.577+0000 7fe0c99c4740 1 -- 172.21.15.39:0/84146573 shutdown_connections 2024-03-18T16:09:08.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.577+0000 7fe0c99c4740 1 --2- 172.21.15.39:0/84146573 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc6f38ab50 0x55cc6f38f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.577+0000 7fe0c99c4740 1 --2- 172.21.15.39:0/84146573 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cc6f38fcc0 0x55cc6f392150 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.577+0000 7fe0c99c4740 1 --2- 172.21.15.39:0/84146573 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cc6f38a160 0x55cc6f38a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.577+0000 7fe0c99c4740 1 -- 172.21.15.39:0/84146573 >> 172.21.15.39:0/84146573 conn(0x55cc6f3856d0 msgr2=0x55cc6f385ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:08.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.577+0000 7fe0c99c4740 1 -- 172.21.15.39:0/84146573 shutdown_connections 2024-03-18T16:09:08.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.577+0000 7fe0c99c4740 1 -- 172.21.15.39:0/84146573 wait complete. 2024-03-18T16:09:08.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.577+0000 7fe0bcc00640 1 --2- 172.21.15.39:0/2892901510 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fe0a40a27b0 0x7fe0a40a4c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:08.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.577+0000 7fe0c99c4740 1 -- start start 2024-03-18T16:09:08.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.577+0000 7fe0bcc00640 1 -- 172.21.15.39:0/2892901510 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fe0a80cf1d0 con 0x7fe0c0074240 2024-03-18T16:09:08.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.578+0000 7fe0c99c4740 1 Processor -- start 2024-03-18T16:09:08.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.578+0000 7fe0c4c00640 1 --2- 172.21.15.39:0/2892901510 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fe0a40a27b0 0x7fe0a40a4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:08.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.578+0000 7fe0c99c4740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cc6f38a160 0x55cc6f3942b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:08.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.578+0000 7fe0c99c4740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc6f38ab50 0x55cc6f3947f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:08.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.578+0000 7fe0c99c4740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cc6f38fcc0 0x55cc6f39b830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:08.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.578+0000 7fe0c99c4740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55cc6f392c10 con 0x55cc6f38a160 2024-03-18T16:09:08.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.578+0000 7fe0c4c00640 1 --2- 172.21.15.39:0/2892901510 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fe0a40a27b0 0x7fe0a40a4c70 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x55cc6f38b3b0 tx=0x7fe0b800b420 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:08.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.578+0000 7fe0c99c4740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55cc6f2aae20 con 0x55cc6f38ab50 2024-03-18T16:09:08.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.578+0000 7fe0c99c4740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55cc6f392d90 con 0x55cc6f38fcc0 2024-03-18T16:09:08.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.579+0000 7fe0c99c4740 10 client.0 _ll_register_callbacks cb 0x55cc6f2aa7f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:09:08.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.579+0000 7fe0bfe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cc6f38a160 0x55cc6f3942b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:08.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.579+0000 7fe0bf400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc6f38ab50 0x55cc6f3947f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:08.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.579+0000 7fe0bfe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cc6f38a160 0x55cc6f3942b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:32916/0 (socket says 172.21.15.39:32916) 2024-03-18T16:09:08.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.579+0000 7fe0c4c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cc6f38fcc0 0x55cc6f39b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:08.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.579+0000 7fe0bfe00640 1 -- 172.21.15.39:0/4015023378 learned_addr learned my addr 172.21.15.39:0/4015023378 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:08.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.579+0000 7fe0bf400640 1 -- 172.21.15.39:0/4015023378 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cc6f38fcc0 msgr2=0x55cc6f39b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:08.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.579+0000 7fe0bf400640 1 --2- 172.21.15.39:0/4015023378 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cc6f38fcc0 0x55cc6f39b830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.579+0000 7fe0bf400640 1 -- 172.21.15.39:0/4015023378 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cc6f38a160 msgr2=0x55cc6f3942b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:08.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.579+0000 7fe0bf400640 1 --2- 172.21.15.39:0/4015023378 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cc6f38a160 0x55cc6f3942b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.579+0000 7fe0bf400640 1 -- 172.21.15.39:0/4015023378 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cc6f0032d0 con 0x55cc6f38ab50 2024-03-18T16:09:08.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.579+0000 7fe0bfe00640 1 --2- 172.21.15.39:0/4015023378 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cc6f38a160 0x55cc6f3942b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:09:08.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.579+0000 7fe0bf400640 1 --2- 172.21.15.39:0/4015023378 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc6f38ab50 0x55cc6f3947f0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fe0b002c710 tx=0x7fe0b0005bf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=174086ed3db2e2a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:08.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.579+0000 7fe0ad600640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:09:08.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.580+0000 7fe0ad600640 1 -- 172.21.15.39:0/4015023378 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0b004c030 con 0x55cc6f38ab50 2024-03-18T16:09:08.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.580+0000 7fe0ad600640 1 -- 172.21.15.39:0/4015023378 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fe0b0003d30 con 0x55cc6f38ab50 2024-03-18T16:09:08.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.580+0000 7fe0c99c4740 1 -- 172.21.15.39:0/4015023378 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe0b0003970 con 0x55cc6f38ab50 2024-03-18T16:09:08.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.580+0000 7fe0ad600640 1 -- 172.21.15.39:0/4015023378 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe0b0059050 con 0x55cc6f38ab50 2024-03-18T16:09:08.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.580+0000 7fe0c99c4740 10 client.25199 fetch_fsmap learned FSMap version 27 2024-03-18T16:09:08.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.580+0000 7fe0c99c4740 1 -- 172.21.15.39:0/4015023378 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55cc6f0cd6b0 con 0x55cc6f38ab50 2024-03-18T16:09:08.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.580+0000 7fe0ad600640 1 -- 172.21.15.39:0/4015023378 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe0b0003970 con 0x55cc6f38ab50 2024-03-18T16:09:08.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.580+0000 7fe0ad600640 1 -- 172.21.15.39:0/4015023378 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fe0b00486f0 con 0x55cc6f38ab50 2024-03-18T16:09:08.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.581+0000 7fe0c99c4740 10 client.25199 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T16:09:08.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.581+0000 7fe0c99c4740 10 client.25199 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-18T16:09:08.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.581+0000 7fe0c99c4740 20 client.25199 populate_metadata read hostname 'smithi039' 2024-03-18T16:09:08.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.581+0000 7fe0c99c4740 1 --2- 172.21.15.39:0/4015023378 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x55cc6f39fd20 0x55cc6f3a1f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:08.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.581+0000 7fe0c99c4740 4 client.25199 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:09:08.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.581+0000 7fe0c99c4740 1 -- 172.21.15.39:0/4015023378 --> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cc6f34c2b0 con 0x55cc6f39fd20 2024-03-18T16:09:08.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.581+0000 7fe0bfe00640 1 --2- 172.21.15.39:0/4015023378 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x55cc6f39fd20 0x55cc6f3a1f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:08.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.581+0000 7fe0bfe00640 1 --2- 172.21.15.39:0/4015023378 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x55cc6f39fd20 0x55cc6f3a1f10 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=656436c917026159 server_cookie=82c826aa39a67cd5 in_seq=0 out_seq=0 2024-03-18T16:09:08.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.581+0000 7fe0ad600640 10 client.25199 ms_handle_connect on v2:172.21.15.196:6838/2001427268 2024-03-18T16:09:08.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.582+0000 7fe0ad600640 1 -- 172.21.15.39:0/4015023378 <== mds.1 v2:172.21.15.196:6838/2001427268 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55cc6f34c2b0 con 0x55cc6f39fd20 2024-03-18T16:09:08.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.582+0000 7fe0ad600640 10 client.25199 handle_command_reply: tid=0 2024-03-18T16:09:08.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.582+0000 7fe0c99c4740 1 client.25199 shutdown 2024-03-18T16:09:08.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.582+0000 7fe0c99c4740 1 -- 172.21.15.39:0/4015023378 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc6f38ab50 msgr2=0x55cc6f3947f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:08.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.582+0000 7fe0c99c4740 1 --2- 172.21.15.39:0/4015023378 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc6f38ab50 0x55cc6f3947f0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fe0b002c710 tx=0x7fe0b0005bf0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.583+0000 7fe0c99c4740 1 -- 172.21.15.39:0/4015023378 shutdown_connections 2024-03-18T16:09:08.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.583+0000 7fe0c99c4740 1 --2- 172.21.15.39:0/4015023378 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x55cc6f39fd20 0x55cc6f3a1f10 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.583+0000 7fe0c99c4740 1 --2- 172.21.15.39:0/4015023378 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc6f38ab50 0x55cc6f3947f0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.583+0000 7fe0c99c4740 1 --2- 172.21.15.39:0/4015023378 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cc6f38fcc0 0x55cc6f39b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.583+0000 7fe0c99c4740 1 --2- 172.21.15.39:0/4015023378 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cc6f38a160 0x55cc6f3942b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.583+0000 7fe0ad600640 0 client.25199 ms_handle_reset on v2:172.21.15.196:6838/2001427268 2024-03-18T16:09:08.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.583+0000 7fe0c99c4740 1 -- 172.21.15.39:0/4015023378 >> 172.21.15.39:0/4015023378 conn(0x55cc6f3856d0 msgr2=0x55cc6f384a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:08.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.583+0000 7fe0c99c4740 1 -- 172.21.15.39:0/4015023378 shutdown_connections 2024-03-18T16:09:08.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.583+0000 7fe0c99c4740 1 -- 172.21.15.39:0/4015023378 wait complete. 2024-03-18T16:09:08.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.583+0000 7fe0c99c4740 20 client.25199 trim_cache size 0 max 16384 2024-03-18T16:09:08.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.603+0000 7fe0c99c4740 1 Processor -- start 2024-03-18T16:09:08.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.603+0000 7fe0c99c4740 1 -- start start 2024-03-18T16:09:08.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.603+0000 7fe0c99c4740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc6f38a160 0x55cc6f394270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:08.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.603+0000 7fe0c99c4740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cc6f38ab50 0x55cc6f39b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:08.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.603+0000 7fe0c99c4740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cc6f38fcc0 0x55cc6f3947b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:08.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.603+0000 7fe0c99c4740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55cc6f3a0250 con 0x55cc6f38ab50 2024-03-18T16:09:08.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.603+0000 7fe0c99c4740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55cc6f3a0920 con 0x55cc6f38a160 2024-03-18T16:09:08.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.603+0000 7fe0c99c4740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55cc6f3a0ff0 con 0x55cc6f38fcc0 2024-03-18T16:09:08.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.603+0000 7fe0bf400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cc6f38ab50 0x55cc6f39b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:08.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.603+0000 7fe0c4c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cc6f38fcc0 0x55cc6f3947b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:08.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.603+0000 7fe0bfe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc6f38a160 0x55cc6f394270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:08.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.603+0000 7fe0c4c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cc6f38fcc0 0x55cc6f3947b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:34034/0 (socket says 172.21.15.39:34034) 2024-03-18T16:09:08.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.603+0000 7fe0bf400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cc6f38ab50 0x55cc6f39b7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:32922/0 (socket says 172.21.15.39:32922) 2024-03-18T16:09:08.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.603+0000 7fe0c4c00640 1 -- 172.21.15.39:0/2967599135 learned_addr learned my addr 172.21.15.39:0/2967599135 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:08.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.604+0000 7fe0c4c00640 1 -- 172.21.15.39:0/2967599135 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc6f38a160 msgr2=0x55cc6f394270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:08.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.604+0000 7fe0c4c00640 1 --2- 172.21.15.39:0/2967599135 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc6f38a160 0x55cc6f394270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.604+0000 7fe0c4c00640 1 -- 172.21.15.39:0/2967599135 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cc6f38ab50 msgr2=0x55cc6f39b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:08.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.604+0000 7fe0c4c00640 1 --2- 172.21.15.39:0/2967599135 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cc6f38ab50 0x55cc6f39b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.604+0000 7fe0c4c00640 1 -- 172.21.15.39:0/2967599135 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cc6f35b750 con 0x55cc6f38fcc0 2024-03-18T16:09:08.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.604+0000 7fe0bfe00640 1 --2- 172.21.15.39:0/2967599135 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc6f38a160 0x55cc6f394270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:09:08.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.604+0000 7fe0c4c00640 1 --2- 172.21.15.39:0/2967599135 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cc6f38fcc0 0x55cc6f3947b0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe0b801e740 tx=0x7fe0b8019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=67ebc3412f1266a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:08.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.604+0000 7fe0afe00640 1 -- 172.21.15.39:0/2967599135 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0b8004050 con 0x55cc6f38fcc0 2024-03-18T16:09:08.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.605+0000 7fe0afe00640 1 -- 172.21.15.39:0/2967599135 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe0b80041f0 con 0x55cc6f38fcc0 2024-03-18T16:09:08.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.605+0000 7fe0afe00640 1 -- 172.21.15.39:0/2967599135 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0b8010a50 con 0x55cc6f38fcc0 2024-03-18T16:09:08.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.605+0000 7fe0c99c4740 1 -- 172.21.15.39:0/2967599135 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cc6f38fcc0 msgr2=0x55cc6f3947b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:08.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.605+0000 7fe0c99c4740 1 --2- 172.21.15.39:0/2967599135 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cc6f38fcc0 0x55cc6f3947b0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe0b801e740 tx=0x7fe0b8019000 comp rx=0 tx=0).stop 2024-03-18T16:09:08.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.605+0000 7fe0c99c4740 1 -- 172.21.15.39:0/2967599135 shutdown_connections 2024-03-18T16:09:08.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.605+0000 7fe0c99c4740 1 --2- 172.21.15.39:0/2967599135 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cc6f38fcc0 0x55cc6f3947b0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.605+0000 7fe0c99c4740 1 --2- 172.21.15.39:0/2967599135 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cc6f38ab50 0x55cc6f39b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.605+0000 7fe0c99c4740 1 --2- 172.21.15.39:0/2967599135 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc6f38a160 0x55cc6f394270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.605+0000 7fe0c99c4740 1 -- 172.21.15.39:0/2967599135 >> 172.21.15.39:0/2967599135 conn(0x55cc6f3856d0 msgr2=0x55cc6f385ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:08.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.605+0000 7fe0c99c4740 1 -- 172.21.15.39:0/2967599135 shutdown_connections 2024-03-18T16:09:08.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.605+0000 7fe0c99c4740 1 -- 172.21.15.39:0/2967599135 wait complete. 2024-03-18T16:09:08.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.606+0000 7fe0c99c4740 1 -- start start 2024-03-18T16:09:08.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.606+0000 7fe0c99c4740 1 Processor -- start 2024-03-18T16:09:08.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.606+0000 7fe0c99c4740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cc6f38a160 0x55cc6f3997e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:08.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.606+0000 7fe0c99c4740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cc6f38ab50 0x55cc6f4027f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:08.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.606+0000 7fe0c99c4740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc6f38fcc0 0x55cc6f404ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:08.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.606+0000 7fe0c99c4740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55cc6f3a1d90 con 0x55cc6f38a160 2024-03-18T16:09:08.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.606+0000 7fe0c99c4740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55cc6f3a16c0 con 0x55cc6f38fcc0 2024-03-18T16:09:08.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.606+0000 7fe0c99c4740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55cc6f3a2460 con 0x55cc6f38ab50 2024-03-18T16:09:08.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.606+0000 7fe0c99c4740 10 client.0 _ll_register_callbacks cb 0x7fe0b0032a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:09:08.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.606+0000 7fe0bf400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cc6f38ab50 0x55cc6f4027f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:08.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.606+0000 7fe0bfe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cc6f38a160 0x55cc6f3997e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:08.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.607+0000 7fe0bf400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cc6f38ab50 0x55cc6f4027f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:34044/0 (socket says 172.21.15.39:34044) 2024-03-18T16:09:08.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.607+0000 7fe0bf400640 1 -- 172.21.15.39:0/2701834329 learned_addr learned my addr 172.21.15.39:0/2701834329 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:08.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.607+0000 7fe0bfe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cc6f38a160 0x55cc6f3997e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:32934/0 (socket says 172.21.15.39:32934) 2024-03-18T16:09:08.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.607+0000 7fe0c4c00640 1 --2- 172.21.15.39:0/2701834329 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc6f38fcc0 0x55cc6f404ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:08.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.607+0000 7fe0bf400640 1 -- 172.21.15.39:0/2701834329 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc6f38fcc0 msgr2=0x55cc6f404ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:08.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.607+0000 7fe0bf400640 1 --2- 172.21.15.39:0/2701834329 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc6f38fcc0 0x55cc6f404ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.607+0000 7fe0bf400640 1 -- 172.21.15.39:0/2701834329 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cc6f38a160 msgr2=0x55cc6f3997e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:08.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.607+0000 7fe0bf400640 1 --2- 172.21.15.39:0/2701834329 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cc6f38a160 0x55cc6f3997e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.607+0000 7fe0bf400640 1 -- 172.21.15.39:0/2701834329 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cc6f4056d0 con 0x55cc6f38ab50 2024-03-18T16:09:08.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.607+0000 7fe0c4c00640 1 --2- 172.21.15.39:0/2701834329 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc6f38fcc0 0x55cc6f404ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:09:08.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.607+0000 7fe0bfe00640 1 --2- 172.21.15.39:0/2701834329 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cc6f38a160 0x55cc6f3997e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:09:08.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.607+0000 7fe0bf400640 1 --2- 172.21.15.39:0/2701834329 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cc6f38ab50 0x55cc6f4027f0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe0b0048cb0 tx=0x7fe0b00021d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=465e8a05b7f604de server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:08.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.607+0000 7fe0ad600640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:09:08.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.607+0000 7fe0ad600640 1 -- 172.21.15.39:0/2701834329 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0b004c070 con 0x55cc6f38ab50 2024-03-18T16:09:08.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.607+0000 7fe0ad600640 1 -- 172.21.15.39:0/2701834329 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fe0b0048650 con 0x55cc6f38ab50 2024-03-18T16:09:08.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.607+0000 7fe0c99c4740 1 -- 172.21.15.39:0/2701834329 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe0b80041f0 con 0x55cc6f38ab50 2024-03-18T16:09:08.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.608+0000 7fe0ad600640 1 -- 172.21.15.39:0/2701834329 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe0b0039070 con 0x55cc6f38ab50 2024-03-18T16:09:08.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.608+0000 7fe0c99c4740 10 client.25225 fetch_fsmap learned FSMap version 27 2024-03-18T16:09:08.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.608+0000 7fe0c99c4740 1 -- 172.21.15.39:0/2701834329 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55cc6f4069b0 con 0x55cc6f38ab50 2024-03-18T16:09:08.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.608+0000 7fe0ad600640 1 -- 172.21.15.39:0/2701834329 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe0b80041f0 con 0x55cc6f38ab50 2024-03-18T16:09:08.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.608+0000 7fe0ad600640 1 -- 172.21.15.39:0/2701834329 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fe0b0034440 con 0x55cc6f38ab50 2024-03-18T16:09:08.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.608+0000 7fe0c99c4740 10 client.25225 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T16:09:08.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.608+0000 7fe0c99c4740 10 client.25225 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-18T16:09:08.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.608+0000 7fe0c99c4740 20 client.25225 populate_metadata read hostname 'smithi039' 2024-03-18T16:09:08.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.608+0000 7fe0c99c4740 1 --2- 172.21.15.39:0/2701834329 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x55cc6f40b580 0x55cc6f40a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:08.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.608+0000 7fe0c99c4740 4 client.25225 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T16:09:08.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.608+0000 7fe0c99c4740 1 -- 172.21.15.39:0/2701834329 --> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] -- command(tid 0: {"prefix": "status"}) -- 0x55cc6f2fc190 con 0x55cc6f40b580 2024-03-18T16:09:08.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.609+0000 7fe0bfe00640 1 --2- 172.21.15.39:0/2701834329 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x55cc6f40b580 0x55cc6f40a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:08.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.609+0000 7fe0bfe00640 1 --2- 172.21.15.39:0/2701834329 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x55cc6f40b580 0x55cc6f40a290 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=bdd67d26220db08c server_cookie=b6df6ef2b25f3404 in_seq=0 out_seq=0 2024-03-18T16:09:08.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.609+0000 7fe0ad600640 10 client.25225 ms_handle_connect on v2:172.21.15.196:6838/2001427268 2024-03-18T16:09:08.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.609+0000 7fe0ad600640 1 -- 172.21.15.39:0/2701834329 <== mds.1 v2:172.21.15.196:6838/2001427268 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55cc6f2fc190 con 0x55cc6f40b580 2024-03-18T16:09:08.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.609+0000 7fe0ad600640 10 client.25225 handle_command_reply: tid=0 2024-03-18T16:09:08.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.609+0000 7fe0c99c4740 1 client.25225 shutdown 2024-03-18T16:09:08.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.609+0000 7fe0c99c4740 1 -- 172.21.15.39:0/2701834329 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cc6f38ab50 msgr2=0x55cc6f4027f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:08.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.609+0000 7fe0c99c4740 1 --2- 172.21.15.39:0/2701834329 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cc6f38ab50 0x55cc6f4027f0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe0b0048cb0 tx=0x7fe0b00021d0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.610+0000 7fe0c99c4740 1 -- 172.21.15.39:0/2701834329 shutdown_connections 2024-03-18T16:09:08.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.610+0000 7fe0c99c4740 1 --2- 172.21.15.39:0/2701834329 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x55cc6f40b580 0x55cc6f40a290 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.610+0000 7fe0c99c4740 1 --2- 172.21.15.39:0/2701834329 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc6f38fcc0 0x55cc6f404ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.610+0000 7fe0c99c4740 1 --2- 172.21.15.39:0/2701834329 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cc6f38ab50 0x55cc6f4027f0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.610+0000 7fe0c99c4740 1 --2- 172.21.15.39:0/2701834329 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cc6f38a160 0x55cc6f3997e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.610+0000 7fe0ad600640 0 client.25225 ms_handle_reset on v2:172.21.15.196:6838/2001427268 2024-03-18T16:09:08.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.610+0000 7fe0c99c4740 1 -- 172.21.15.39:0/2701834329 >> 172.21.15.39:0/2701834329 conn(0x55cc6f3856d0 msgr2=0x55cc6f394270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:08.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.610+0000 7fe0c99c4740 1 -- 172.21.15.39:0/2701834329 shutdown_connections 2024-03-18T16:09:08.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.610+0000 7fe0c99c4740 1 -- 172.21.15.39:0/2701834329 wait complete. 2024-03-18T16:09:08.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.610+0000 7fe0c99c4740 20 client.25225 trim_cache size 0 max 16384 2024-03-18T16:09:08.616 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-03-18T16:09:08.616 INFO:teuthology.orchestra.run.smithi039.stdout: "cluster_fsid": "1892d8fc-e540-11ee-95c9-87774f69a715", 2024-03-18T16:09:08.616 INFO:teuthology.orchestra.run.smithi039.stdout: "whoami": 1, 2024-03-18T16:09:08.616 INFO:teuthology.orchestra.run.smithi039.stdout: "id": 24556, 2024-03-18T16:09:08.616 INFO:teuthology.orchestra.run.smithi039.stdout: "want_state": "up:active", 2024-03-18T16:09:08.617 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "up:active", 2024-03-18T16:09:08.617 INFO:teuthology.orchestra.run.smithi039.stdout: "fs_name": "cephfs", 2024-03-18T16:09:08.617 INFO:teuthology.orchestra.run.smithi039.stdout: "rank_uptime": 111.41478669599999, 2024-03-18T16:09:08.617 INFO:teuthology.orchestra.run.smithi039.stdout: "mdsmap_epoch": 27, 2024-03-18T16:09:08.617 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_epoch": 93, 2024-03-18T16:09:08.617 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_epoch_barrier": 89, 2024-03-18T16:09:08.617 INFO:teuthology.orchestra.run.smithi039.stdout: "uptime": 162.41108528199999 2024-03-18T16:09:08.617 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-03-18T16:09:08.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.610+0000 7fe0c6c00640 1 -- 172.21.15.39:0/2892901510 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fe0a40a27b0 msgr2=0x7fe0a40a4c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:08.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.610+0000 7fe0c6c00640 1 --2- 172.21.15.39:0/2892901510 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fe0a40a27b0 0x7fe0a40a4c70 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x55cc6f38b3b0 tx=0x7fe0b800b420 comp rx=0 tx=0).stop 2024-03-18T16:09:08.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.611+0000 7fe0c6c00640 1 -- 172.21.15.39:0/2892901510 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe0c0074240 msgr2=0x7fe0c0199da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:08.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.611+0000 7fe0c6c00640 1 --2- 172.21.15.39:0/2892901510 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe0c0074240 0x7fe0c0199da0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fe0a800e9f0 tx=0x7fe0a800eeb0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.611+0000 7fe0c6c00640 1 -- 172.21.15.39:0/2892901510 shutdown_connections 2024-03-18T16:09:08.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.611+0000 7fe0c6c00640 1 --2- 172.21.15.39:0/2892901510 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fe0a40a27b0 0x7fe0a40a4c70 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.611+0000 7fe0c6c00640 1 --2- 172.21.15.39:0/2892901510 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe0c01096b0 0x7fe0c01a1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.611+0000 7fe0c6c00640 1 --2- 172.21.15.39:0/2892901510 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe0c0075820 0x7fe0c019a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.611+0000 7fe0c6c00640 1 --2- 172.21.15.39:0/2892901510 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe0c0074240 0x7fe0c0199da0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.611+0000 7fe0c6c00640 1 -- 172.21.15.39:0/2892901510 >> 172.21.15.39:0/2892901510 conn(0x7fe0c00fcd60 msgr2=0x7fe0c01055b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:08.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.611+0000 7fe0c6c00640 1 -- 172.21.15.39:0/2892901510 shutdown_connections 2024-03-18T16:09:08.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.611+0000 7fe0c6c00640 1 -- 172.21.15.39:0/2892901510 wait complete. 2024-03-18T16:09:08.630 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-18T16:09:08.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.840+0000 7fa6f7200640 1 -- 172.21.15.39:0/3538644183 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa6f8104ad0 msgr2=0x7fa6f8104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:08.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.840+0000 7fa6f7200640 1 --2- 172.21.15.39:0/3538644183 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa6f8104ad0 0x7fa6f8104f50 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fa6e0009900 tx=0x7fa6e002bc30 comp rx=0 tx=0).stop 2024-03-18T16:09:08.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.840+0000 7fa6f7200640 1 -- 172.21.15.39:0/3538644183 shutdown_connections 2024-03-18T16:09:08.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.840+0000 7fa6f7200640 1 --2- 172.21.15.39:0/3538644183 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa6f8105490 0x7fa6f810bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.840+0000 7fa6f7200640 1 --2- 172.21.15.39:0/3538644183 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa6f8104ad0 0x7fa6f8104f50 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.840+0000 7fa6f7200640 1 --2- 172.21.15.39:0/3538644183 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa6f81038e0 0x7fa6f8103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.840+0000 7fa6f7200640 1 -- 172.21.15.39:0/3538644183 >> 172.21.15.39:0/3538644183 conn(0x7fa6f80fee50 msgr2=0x7fa6f8101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:08.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.840+0000 7fa6f7200640 1 -- 172.21.15.39:0/3538644183 shutdown_connections 2024-03-18T16:09:08.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.840+0000 7fa6f7200640 1 -- 172.21.15.39:0/3538644183 wait complete. 2024-03-18T16:09:08.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.841+0000 7fa6f7200640 1 Processor -- start 2024-03-18T16:09:08.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.841+0000 7fa6f7200640 1 -- start start 2024-03-18T16:09:08.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.842+0000 7fa6f7200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa6f81038e0 0x7fa6f8199bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:08.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.842+0000 7fa6f7200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa6f8104ad0 0x7fa6f819a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:08.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.842+0000 7fa6f7200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa6f8105490 0x7fa6f81a1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:08.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.842+0000 7fa6f7200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fa6f810f590 con 0x7fa6f8104ad0 2024-03-18T16:09:08.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.842+0000 7fa6f7200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fa6f810f410 con 0x7fa6f8105490 2024-03-18T16:09:08.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.842+0000 7fa6f7200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fa6f810f710 con 0x7fa6f81038e0 2024-03-18T16:09:08.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.842+0000 7fa6f5e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa6f81038e0 0x7fa6f8199bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:08.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.842+0000 7fa6f5e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa6f81038e0 0x7fa6f8199bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:34064/0 (socket says 172.21.15.39:34064) 2024-03-18T16:09:08.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.842+0000 7fa6f5e00640 1 -- 172.21.15.39:0/3784895169 learned_addr learned my addr 172.21.15.39:0/3784895169 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:08.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.842+0000 7fa6efe00640 1 --2- 172.21.15.39:0/3784895169 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa6f8104ad0 0x7fa6f819a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:08.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.842+0000 7fa6f6800640 1 --2- 172.21.15.39:0/3784895169 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa6f8105490 0x7fa6f81a1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:08.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.843+0000 7fa6f5e00640 1 -- 172.21.15.39:0/3784895169 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa6f8105490 msgr2=0x7fa6f81a1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:08.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.843+0000 7fa6f5e00640 1 --2- 172.21.15.39:0/3784895169 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa6f8105490 0x7fa6f81a1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.843+0000 7fa6f5e00640 1 -- 172.21.15.39:0/3784895169 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa6f8104ad0 msgr2=0x7fa6f819a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:08.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.843+0000 7fa6f5e00640 1 --2- 172.21.15.39:0/3784895169 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa6f8104ad0 0x7fa6f819a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.843+0000 7fa6f5e00640 1 -- 172.21.15.39:0/3784895169 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6f81a17f0 con 0x7fa6f81038e0 2024-03-18T16:09:08.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.843+0000 7fa6f5e00640 1 --2- 172.21.15.39:0/3784895169 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa6f81038e0 0x7fa6f8199bd0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa6e800e9c0 tx=0x7fa6e800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:08.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.843+0000 7fa6eea00640 1 -- 172.21.15.39:0/3784895169 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6e800cd80 con 0x7fa6f81038e0 2024-03-18T16:09:08.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.843+0000 7fa6eea00640 1 -- 172.21.15.39:0/3784895169 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa6e80035f0 con 0x7fa6f81038e0 2024-03-18T16:09:08.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.843+0000 7fa6f7200640 1 -- 172.21.15.39:0/3784895169 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6f81a1ae0 con 0x7fa6f81038e0 2024-03-18T16:09:08.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.843+0000 7fa6eea00640 1 -- 172.21.15.39:0/3784895169 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6e8010690 con 0x7fa6f81038e0 2024-03-18T16:09:08.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.843+0000 7fa6f7200640 1 -- 172.21.15.39:0/3784895169 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6f81a2050 con 0x7fa6f81038e0 2024-03-18T16:09:08.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.846+0000 7fa6fff78740 1 Processor -- start 2024-03-18T16:09:08.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.847+0000 7fa6fff78740 1 -- start start 2024-03-18T16:09:08.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.847+0000 7fa6eea00640 1 -- 172.21.15.39:0/3784895169 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fa6e80037a0 con 0x7fa6f81038e0 2024-03-18T16:09:08.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.847+0000 7fa6fff78740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b71c72c340 0x55b71c72c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:08.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.847+0000 7fa6fff78740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b71c72cd30 0x55b71c731830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:08.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.847+0000 7fa6fff78740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b71c731ea0 0x55b71c734330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:08.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.847+0000 7fa6fff78740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55b71c690c50 con 0x55b71c731ea0 2024-03-18T16:09:08.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.847+0000 7fa6fff78740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55b71c6399d0 con 0x55b71c72c340 2024-03-18T16:09:08.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.847+0000 7fa6fff78740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55b71c68ccf0 con 0x55b71c72cd30 2024-03-18T16:09:08.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.847+0000 7fa6f5e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b71c72cd30 0x55b71c731830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:08.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.847+0000 7fa6f6800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b71c72c340 0x55b71c72c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:08.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.847+0000 7fa6f6800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b71c72c340 0x55b71c72c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:56632/0 (socket says 172.21.15.39:56632) 2024-03-18T16:09:08.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.847+0000 7fa6f6800640 1 -- 172.21.15.39:0/1617491339 learned_addr learned my addr 172.21.15.39:0/1617491339 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:08.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.847+0000 7fa6f5e00640 1 --2- 172.21.15.39:0/1617491339 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b71c72cd30 0x55b71c731830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:34080/0 (socket says 172.21.15.39:34080) 2024-03-18T16:09:08.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.847+0000 7fa6efe00640 1 --2- 172.21.15.39:0/1617491339 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b71c731ea0 0x55b71c734330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:08.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.848+0000 7fa6f6800640 1 -- 172.21.15.39:0/1617491339 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b71c72cd30 msgr2=0x55b71c731830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:08.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.848+0000 7fa6f6800640 1 --2- 172.21.15.39:0/1617491339 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b71c72cd30 0x55b71c731830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.848+0000 7fa6f6800640 1 -- 172.21.15.39:0/1617491339 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b71c731ea0 msgr2=0x55b71c734330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:08.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.848+0000 7fa6f6800640 1 --2- 172.21.15.39:0/1617491339 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b71c731ea0 0x55b71c734330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.848+0000 7fa6f6800640 1 -- 172.21.15.39:0/1617491339 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b71c653f10 con 0x55b71c72c340 2024-03-18T16:09:08.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.848+0000 7fa6f6800640 1 --2- 172.21.15.39:0/1617491339 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b71c72c340 0x55b71c72c760 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa6f0009fd0 tx=0x7fa6f00118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c87a608a92dff1d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:08.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.848+0000 7fa6e7400640 1 -- 172.21.15.39:0/1617491339 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6f0004d20 con 0x55b71c72c340 2024-03-18T16:09:08.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.848+0000 7fa6e7400640 1 -- 172.21.15.39:0/1617491339 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa6f0004050 con 0x55b71c72c340 2024-03-18T16:09:08.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.848+0000 7fa6e7400640 1 -- 172.21.15.39:0/1617491339 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6f0005260 con 0x55b71c72c340 2024-03-18T16:09:08.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.849+0000 7fa6eea00640 1 --2- 172.21.15.39:0/3784895169 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa6dc0a2880 0x7fa6dc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:08.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.849+0000 7fa6fff78740 1 -- 172.21.15.39:0/1617491339 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b71c72c340 msgr2=0x55b71c72c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:08.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.849+0000 7fa6fff78740 1 --2- 172.21.15.39:0/1617491339 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b71c72c340 0x55b71c72c760 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa6f0009fd0 tx=0x7fa6f00118d0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.849+0000 7fa6eea00640 1 -- 172.21.15.39:0/3784895169 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fa6e80cf160 con 0x7fa6f81038e0 2024-03-18T16:09:08.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.849+0000 7fa6f5e00640 1 --2- 172.21.15.39:0/3784895169 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa6dc0a2880 0x7fa6dc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:08.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.849+0000 7fa6fff78740 1 -- 172.21.15.39:0/1617491339 shutdown_connections 2024-03-18T16:09:08.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.849+0000 7fa6fff78740 1 --2- 172.21.15.39:0/1617491339 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b71c731ea0 0x55b71c734330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.849+0000 7fa6fff78740 1 --2- 172.21.15.39:0/1617491339 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b71c72cd30 0x55b71c731830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.849+0000 7fa6fff78740 1 --2- 172.21.15.39:0/1617491339 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b71c72c340 0x55b71c72c760 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.849+0000 7fa6fff78740 1 -- 172.21.15.39:0/1617491339 >> 172.21.15.39:0/1617491339 conn(0x55b71c7278b0 msgr2=0x55b71c727cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:08.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.849+0000 7fa6fff78740 1 -- 172.21.15.39:0/1617491339 shutdown_connections 2024-03-18T16:09:08.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.850+0000 7fa6fff78740 1 -- 172.21.15.39:0/1617491339 wait complete. 2024-03-18T16:09:08.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.850+0000 7fa6fff78740 1 -- start start 2024-03-18T16:09:08.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.850+0000 7fa6f5e00640 1 --2- 172.21.15.39:0/3784895169 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa6dc0a2880 0x7fa6dc0a4d40 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fa6e80980e0 tx=0x7fa6e80d2ba0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:08.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.850+0000 7fa6fff78740 1 Processor -- start 2024-03-18T16:09:08.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.851+0000 7fa6fff78740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b71c72c340 0x55b71c736430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:08.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.851+0000 7fa6fff78740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b71c72cd30 0x55b71c736970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:08.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.851+0000 7fa6fff78740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b71c731ea0 0x55b71c73d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:08.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.851+0000 7fa6fff78740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55b71c734e50 con 0x55b71c731ea0 2024-03-18T16:09:08.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.851+0000 7fa6fff78740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55b71c64ce20 con 0x55b71c72cd30 2024-03-18T16:09:08.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.851+0000 7fa6fff78740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55b71c734fd0 con 0x55b71c72c340 2024-03-18T16:09:08.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.851+0000 7fa6fff78740 10 client.0 _ll_register_callbacks cb 0x55b71c64c7f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:09:08.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.851+0000 7fa6efe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b71c72cd30 0x55b71c736970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:08.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.851+0000 7fa6f6800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b71c72c340 0x55b71c736430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:08.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.851+0000 7fa6efe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b71c72cd30 0x55b71c736970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:56644/0 (socket says 172.21.15.39:56644) 2024-03-18T16:09:08.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.851+0000 7fa6efe00640 1 -- 172.21.15.39:0/3906570789 learned_addr learned my addr 172.21.15.39:0/3906570789 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:08.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.851+0000 7fa6f6800640 1 --2- 172.21.15.39:0/3906570789 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b71c731ea0 0x55b71c73d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:08.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.851+0000 7fa6efe00640 1 -- 172.21.15.39:0/3906570789 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b71c72c340 msgr2=0x55b71c736430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:08.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.851+0000 7fa6efe00640 1 --2- 172.21.15.39:0/3906570789 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b71c72c340 0x55b71c736430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.851+0000 7fa6efe00640 1 -- 172.21.15.39:0/3906570789 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b71c731ea0 msgr2=0x55b71c73d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:08.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.851+0000 7fa6efe00640 1 --2- 172.21.15.39:0/3906570789 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b71c731ea0 0x55b71c73d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.851+0000 7fa6efe00640 1 -- 172.21.15.39:0/3906570789 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b71c3a52d0 con 0x55b71c72cd30 2024-03-18T16:09:08.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.852+0000 7fa6efe00640 1 --2- 172.21.15.39:0/3906570789 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b71c72cd30 0x55b71c736970 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fa6e0000c00 tx=0x7fa6e0004250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b7093f88a432418b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:08.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.852+0000 7fa6dbe00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:09:08.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.852+0000 7fa6dbe00640 1 -- 172.21.15.39:0/3906570789 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6e0005d50 con 0x55b71c72cd30 2024-03-18T16:09:08.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.852+0000 7fa6dbe00640 1 -- 172.21.15.39:0/3906570789 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fa6e00046c0 con 0x55b71c72cd30 2024-03-18T16:09:08.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.852+0000 7fa6fff78740 1 -- 172.21.15.39:0/3906570789 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa6f0004050 con 0x55b71c72cd30 2024-03-18T16:09:08.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.852+0000 7fa6dbe00640 1 -- 172.21.15.39:0/3906570789 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa6e00048e0 con 0x55b71c72cd30 2024-03-18T16:09:08.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.852+0000 7fa6fff78740 10 client.25217 fetch_fsmap learned FSMap version 27 2024-03-18T16:09:08.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.852+0000 7fa6fff78740 1 -- 172.21.15.39:0/3906570789 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b71c46f6b0 con 0x55b71c72cd30 2024-03-18T16:09:08.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.853+0000 7fa6dbe00640 1 -- 172.21.15.39:0/3906570789 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa6f0004050 con 0x55b71c72cd30 2024-03-18T16:09:08.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.853+0000 7fa6dbe00640 1 -- 172.21.15.39:0/3906570789 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fa6e00035f0 con 0x55b71c72cd30 2024-03-18T16:09:08.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.853+0000 7fa6fff78740 10 client.25217 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T16:09:08.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.853+0000 7fa6fff78740 10 client.25217 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-18T16:09:08.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.853+0000 7fa6fff78740 20 client.25217 populate_metadata read hostname 'smithi039' 2024-03-18T16:09:08.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.853+0000 7fa6fff78740 1 --2- 172.21.15.39:0/3906570789 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55b71c7431b0 0x55b71c744620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:08.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.853+0000 7fa6fff78740 4 client.25217 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:09:08.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.853+0000 7fa6fff78740 1 -- 172.21.15.39:0/3906570789 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b71c6ee2b0 con 0x55b71c7431b0 2024-03-18T16:09:08.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.853+0000 7fa6f6800640 1 --2- 172.21.15.39:0/3906570789 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55b71c7431b0 0x55b71c744620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:08.859 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.854+0000 7fa6f6800640 1 --2- 172.21.15.39:0/3906570789 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55b71c7431b0 0x55b71c744620 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aa52a6b907379860 server_cookie=d667a58c03afeae2 in_seq=0 out_seq=0 2024-03-18T16:09:08.860 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.854+0000 7fa6dbe00640 10 client.25217 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:09:08.860 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.855+0000 7fa6dbe00640 1 -- 172.21.15.39:0/3906570789 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b71c6ee2b0 con 0x55b71c7431b0 2024-03-18T16:09:08.860 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.855+0000 7fa6dbe00640 10 client.25217 handle_command_reply: tid=0 2024-03-18T16:09:08.860 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.855+0000 7fa6fff78740 1 client.25217 shutdown 2024-03-18T16:09:08.860 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.855+0000 7fa6fff78740 1 -- 172.21.15.39:0/3906570789 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b71c72cd30 msgr2=0x55b71c736970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:08.860 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.855+0000 7fa6fff78740 1 --2- 172.21.15.39:0/3906570789 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b71c72cd30 0x55b71c736970 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fa6e0000c00 tx=0x7fa6e0004250 comp rx=0 tx=0).stop 2024-03-18T16:09:08.860 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.855+0000 7fa6fff78740 1 -- 172.21.15.39:0/3906570789 shutdown_connections 2024-03-18T16:09:08.860 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.855+0000 7fa6fff78740 1 --2- 172.21.15.39:0/3906570789 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55b71c7431b0 0x55b71c744620 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.860 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.855+0000 7fa6fff78740 1 --2- 172.21.15.39:0/3906570789 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b71c72cd30 0x55b71c736970 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.860 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.855+0000 7fa6fff78740 1 --2- 172.21.15.39:0/3906570789 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b71c731ea0 0x55b71c73d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.860 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.855+0000 7fa6fff78740 1 --2- 172.21.15.39:0/3906570789 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b71c72c340 0x55b71c736430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.860 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.855+0000 7fa6dbe00640 0 client.25217 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:09:08.860 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.855+0000 7fa6fff78740 1 -- 172.21.15.39:0/3906570789 >> 172.21.15.39:0/3906570789 conn(0x55b71c7278b0 msgr2=0x55b71c726a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:08.860 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.855+0000 7fa6fff78740 1 -- 172.21.15.39:0/3906570789 shutdown_connections 2024-03-18T16:09:08.860 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.855+0000 7fa6fff78740 1 -- 172.21.15.39:0/3906570789 wait complete. 2024-03-18T16:09:08.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.855+0000 7fa6fff78740 20 client.25217 trim_cache size 0 max 16384 2024-03-18T16:09:08.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.875+0000 7fa6fff78740 1 Processor -- start 2024-03-18T16:09:08.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.875+0000 7fa6fff78740 1 -- start start 2024-03-18T16:09:08.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.876+0000 7fa6fff78740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b71c72c340 0x55b71c7268f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:08.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.876+0000 7fa6fff78740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b71c72cd30 0x55b71c73d970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:08.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.876+0000 7fa6fff78740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b71c731ea0 0x55b71c72d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:08.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.876+0000 7fa6fff78740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55b71c742960 con 0x55b71c72cd30 2024-03-18T16:09:08.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.876+0000 7fa6fff78740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55b71c743030 con 0x55b71c72c340 2024-03-18T16:09:08.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.876+0000 7fa6fff78740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55b71c743700 con 0x55b71c731ea0 2024-03-18T16:09:08.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.876+0000 7fa6f6800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b71c72c340 0x55b71c7268f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:08.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.876+0000 7fa6f6800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b71c72c340 0x55b71c7268f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:56660/0 (socket says 172.21.15.39:56660) 2024-03-18T16:09:08.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.876+0000 7fa6f6800640 1 -- 172.21.15.39:0/357380435 learned_addr learned my addr 172.21.15.39:0/357380435 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:08.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.876+0000 7fa6f6800640 1 --2- 172.21.15.39:0/357380435 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b71c731ea0 0x55b71c72d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:08.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.876+0000 7fa6efe00640 1 --2- 172.21.15.39:0/357380435 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b71c72cd30 0x55b71c73d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:08.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.876+0000 7fa6f6800640 1 -- 172.21.15.39:0/357380435 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b71c731ea0 msgr2=0x55b71c72d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:08.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.876+0000 7fa6f6800640 1 --2- 172.21.15.39:0/357380435 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b71c731ea0 0x55b71c72d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.876+0000 7fa6f6800640 1 -- 172.21.15.39:0/357380435 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b71c72cd30 msgr2=0x55b71c73d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:08.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.876+0000 7fa6f6800640 1 --2- 172.21.15.39:0/357380435 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b71c72cd30 0x55b71c73d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.876+0000 7fa6f6800640 1 -- 172.21.15.39:0/357380435 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b71c7273f0 con 0x55b71c72c340 2024-03-18T16:09:08.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.877+0000 7fa6efe00640 1 --2- 172.21.15.39:0/357380435 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b71c72cd30 0x55b71c73d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:09:08.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.877+0000 7fa6f6800640 1 --2- 172.21.15.39:0/357380435 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b71c72c340 0x55b71c7268f0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fa6f0003f50 tx=0x7fa6f0009500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=76a2ed4a2a8778f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:08.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.877+0000 7fa6e6a00640 1 -- 172.21.15.39:0/357380435 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6f002c240 con 0x55b71c72c340 2024-03-18T16:09:08.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.877+0000 7fa6e6a00640 1 -- 172.21.15.39:0/357380435 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa6f0004000 con 0x55b71c72c340 2024-03-18T16:09:08.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.877+0000 7fa6e6a00640 1 -- 172.21.15.39:0/357380435 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6f000f720 con 0x55b71c72c340 2024-03-18T16:09:08.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.878+0000 7fa6fff78740 1 -- 172.21.15.39:0/357380435 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b71c72c340 msgr2=0x55b71c7268f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:08.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.878+0000 7fa6fff78740 1 --2- 172.21.15.39:0/357380435 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b71c72c340 0x55b71c7268f0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fa6f0003f50 tx=0x7fa6f0009500 comp rx=0 tx=0).stop 2024-03-18T16:09:08.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.878+0000 7fa6fff78740 1 -- 172.21.15.39:0/357380435 shutdown_connections 2024-03-18T16:09:08.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.878+0000 7fa6fff78740 1 --2- 172.21.15.39:0/357380435 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b71c731ea0 0x55b71c72d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.878+0000 7fa6fff78740 1 --2- 172.21.15.39:0/357380435 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b71c72cd30 0x55b71c73d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.878+0000 7fa6fff78740 1 --2- 172.21.15.39:0/357380435 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b71c72c340 0x55b71c7268f0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.878+0000 7fa6fff78740 1 -- 172.21.15.39:0/357380435 >> 172.21.15.39:0/357380435 conn(0x55b71c7278b0 msgr2=0x55b71c7261e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:08.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.878+0000 7fa6fff78740 1 -- 172.21.15.39:0/357380435 shutdown_connections 2024-03-18T16:09:08.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.878+0000 7fa6fff78740 1 -- 172.21.15.39:0/357380435 wait complete. 2024-03-18T16:09:08.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.878+0000 7fa6fff78740 1 -- start start 2024-03-18T16:09:08.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.878+0000 7fa6fff78740 1 Processor -- start 2024-03-18T16:09:08.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.879+0000 7fa6fff78740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b71c72c340 0x55b71c739400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:08.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.879+0000 7fa6fff78740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b71c72cd30 0x55b71c739940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:08.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.879+0000 7fa6fff78740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b71c731ea0 0x55b71c73c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:08.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.879+0000 7fa6fff78740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55b71c7444a0 con 0x55b71c72cd30 2024-03-18T16:09:08.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.879+0000 7fa6fff78740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55b71c743dd0 con 0x55b71c731ea0 2024-03-18T16:09:08.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.879+0000 7fa6fff78740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55b71c744b70 con 0x55b71c72c340 2024-03-18T16:09:08.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.879+0000 7fa6fff78740 10 client.0 _ll_register_callbacks cb 0x7fa6e002cdf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:09:08.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.879+0000 7fa6f6800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b71c72c340 0x55b71c739400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:08.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.879+0000 7fa6f6800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b71c72c340 0x55b71c739400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:34104/0 (socket says 172.21.15.39:34104) 2024-03-18T16:09:08.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.879+0000 7fa6f6800640 1 -- 172.21.15.39:0/4249377965 learned_addr learned my addr 172.21.15.39:0/4249377965 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:08.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.879+0000 7fa6f6800640 1 --2- 172.21.15.39:0/4249377965 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b71c731ea0 0x55b71c73c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:08.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.879+0000 7fa6efe00640 1 --2- 172.21.15.39:0/4249377965 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b71c72cd30 0x55b71c739940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:08.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.880+0000 7fa6f6800640 1 -- 172.21.15.39:0/4249377965 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b71c731ea0 msgr2=0x55b71c73c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:08.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.880+0000 7fa6f6800640 1 --2- 172.21.15.39:0/4249377965 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b71c731ea0 0x55b71c73c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.880+0000 7fa6f6800640 1 -- 172.21.15.39:0/4249377965 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b71c72cd30 msgr2=0x55b71c739940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:08.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.880+0000 7fa6f6800640 1 --2- 172.21.15.39:0/4249377965 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b71c72cd30 0x55b71c739940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.880+0000 7fa6f6800640 1 -- 172.21.15.39:0/4249377965 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b71c73cf50 con 0x55b71c72c340 2024-03-18T16:09:08.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.880+0000 7fa6f6800640 1 --2- 172.21.15.39:0/4249377965 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b71c72c340 0x55b71c739400 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa6f0005d70 tx=0x7fa6f0013040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a4e31aaee04fb843 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:08.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.880+0000 7fa6dbe00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:09:08.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.880+0000 7fa6dbe00640 1 -- 172.21.15.39:0/4249377965 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6f0004470 con 0x55b71c72c340 2024-03-18T16:09:08.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.880+0000 7fa6dbe00640 1 -- 172.21.15.39:0/4249377965 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fa6f0004610 con 0x55b71c72c340 2024-03-18T16:09:08.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.880+0000 7fa6fff78740 1 -- 172.21.15.39:0/4249377965 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa6f0004000 con 0x55b71c72c340 2024-03-18T16:09:08.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.880+0000 7fa6dbe00640 1 -- 172.21.15.39:0/4249377965 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa6f002c040 con 0x55b71c72c340 2024-03-18T16:09:08.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.880+0000 7fa6fff78740 10 client.25243 fetch_fsmap learned FSMap version 27 2024-03-18T16:09:08.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.880+0000 7fa6fff78740 1 -- 172.21.15.39:0/4249377965 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55b71c7b1a60 con 0x55b71c72c340 2024-03-18T16:09:08.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.881+0000 7fa6dbe00640 1 -- 172.21.15.39:0/4249377965 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa6f0004000 con 0x55b71c72c340 2024-03-18T16:09:08.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.881+0000 7fa6dbe00640 1 -- 172.21.15.39:0/4249377965 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fa6f0025040 con 0x55b71c72c340 2024-03-18T16:09:08.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.881+0000 7fa6fff78740 10 client.25243 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T16:09:08.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.881+0000 7fa6fff78740 10 client.25243 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-18T16:09:08.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.881+0000 7fa6fff78740 20 client.25243 populate_metadata read hostname 'smithi039' 2024-03-18T16:09:08.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.881+0000 7fa6fff78740 1 --2- 172.21.15.39:0/4249377965 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55b71c7b5f00 0x55b71c7b4c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:08.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.881+0000 7fa6fff78740 4 client.25243 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T16:09:08.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.881+0000 7fa6fff78740 1 -- 172.21.15.39:0/4249377965 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "status"}) -- 0x55b71c69e190 con 0x55b71c7b5f00 2024-03-18T16:09:08.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.881+0000 7fa6efe00640 1 --2- 172.21.15.39:0/4249377965 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55b71c7b5f00 0x55b71c7b4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:08.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.882+0000 7fa6efe00640 1 --2- 172.21.15.39:0/4249377965 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55b71c7b5f00 0x55b71c7b4c10 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=87437aba2150fe0e server_cookie=fdc129c4b239b85f in_seq=0 out_seq=0 2024-03-18T16:09:08.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.882+0000 7fa6dbe00640 10 client.25243 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:09:08.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.882+0000 7fa6dbe00640 1 -- 172.21.15.39:0/4249377965 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55b71c69e190 con 0x55b71c7b5f00 2024-03-18T16:09:08.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.882+0000 7fa6dbe00640 10 client.25243 handle_command_reply: tid=0 2024-03-18T16:09:08.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.882+0000 7fa6fff78740 1 client.25243 shutdown 2024-03-18T16:09:08.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.882+0000 7fa6fff78740 1 -- 172.21.15.39:0/4249377965 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b71c72c340 msgr2=0x55b71c739400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:08.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.882+0000 7fa6fff78740 1 --2- 172.21.15.39:0/4249377965 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b71c72c340 0x55b71c739400 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa6f0005d70 tx=0x7fa6f0013040 comp rx=0 tx=0).stop 2024-03-18T16:09:08.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.882+0000 7fa6fff78740 1 -- 172.21.15.39:0/4249377965 shutdown_connections 2024-03-18T16:09:08.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.882+0000 7fa6fff78740 1 --2- 172.21.15.39:0/4249377965 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55b71c7b5f00 0x55b71c7b4c10 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.882+0000 7fa6fff78740 1 --2- 172.21.15.39:0/4249377965 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b71c731ea0 0x55b71c73c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.882+0000 7fa6fff78740 1 --2- 172.21.15.39:0/4249377965 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b71c72cd30 0x55b71c739940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.882+0000 7fa6dbe00640 0 client.25243 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:09:08.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.882+0000 7fa6fff78740 1 --2- 172.21.15.39:0/4249377965 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b71c72c340 0x55b71c739400 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.882+0000 7fa6fff78740 1 -- 172.21.15.39:0/4249377965 >> 172.21.15.39:0/4249377965 conn(0x55b71c7278b0 msgr2=0x55b71c726a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:08.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.882+0000 7fa6fff78740 1 -- 172.21.15.39:0/4249377965 shutdown_connections 2024-03-18T16:09:08.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.883+0000 7fa6fff78740 1 -- 172.21.15.39:0/4249377965 wait complete. 2024-03-18T16:09:08.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.883+0000 7fa6fff78740 20 client.25243 trim_cache size 0 max 16384 2024-03-18T16:09:08.889 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-03-18T16:09:08.889 INFO:teuthology.orchestra.run.smithi039.stdout: "cluster_fsid": "1892d8fc-e540-11ee-95c9-87774f69a715", 2024-03-18T16:09:08.889 INFO:teuthology.orchestra.run.smithi039.stdout: "whoami": 0, 2024-03-18T16:09:08.889 INFO:teuthology.orchestra.run.smithi039.stdout: "id": 24575, 2024-03-18T16:09:08.889 INFO:teuthology.orchestra.run.smithi039.stdout: "want_state": "up:active", 2024-03-18T16:09:08.889 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "up:active", 2024-03-18T16:09:08.889 INFO:teuthology.orchestra.run.smithi039.stdout: "fs_name": "cephfs", 2024-03-18T16:09:08.889 INFO:teuthology.orchestra.run.smithi039.stdout: "rank_uptime": 144.05824455699999, 2024-03-18T16:09:08.889 INFO:teuthology.orchestra.run.smithi039.stdout: "mdsmap_epoch": 27, 2024-03-18T16:09:08.889 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_epoch": 93, 2024-03-18T16:09:08.889 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_epoch_barrier": 89, 2024-03-18T16:09:08.890 INFO:teuthology.orchestra.run.smithi039.stdout: "uptime": 151.45735780000001 2024-03-18T16:09:08.890 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-03-18T16:09:08.890 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.883+0000 7fa6f7200640 1 -- 172.21.15.39:0/3784895169 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa6dc0a2880 msgr2=0x7fa6dc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:08.890 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.883+0000 7fa6f7200640 1 --2- 172.21.15.39:0/3784895169 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa6dc0a2880 0x7fa6dc0a4d40 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fa6e80980e0 tx=0x7fa6e80d2ba0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.883+0000 7fa6f7200640 1 -- 172.21.15.39:0/3784895169 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa6f81038e0 msgr2=0x7fa6f8199bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:08.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.883+0000 7fa6f7200640 1 --2- 172.21.15.39:0/3784895169 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa6f81038e0 0x7fa6f8199bd0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa6e800e9c0 tx=0x7fa6e800ee80 comp rx=0 tx=0).stop 2024-03-18T16:09:08.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.883+0000 7fa6f7200640 1 -- 172.21.15.39:0/3784895169 shutdown_connections 2024-03-18T16:09:08.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.884+0000 7fa6f7200640 1 --2- 172.21.15.39:0/3784895169 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa6f8105490 0x7fa6f81a1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.884+0000 7fa6f7200640 1 --2- 172.21.15.39:0/3784895169 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa6dc0a2880 0x7fa6dc0a4d40 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.884+0000 7fa6f7200640 1 --2- 172.21.15.39:0/3784895169 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa6f8104ad0 0x7fa6f819a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.884+0000 7fa6f7200640 1 --2- 172.21.15.39:0/3784895169 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa6f81038e0 0x7fa6f8199bd0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:08.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.884+0000 7fa6f7200640 1 -- 172.21.15.39:0/3784895169 >> 172.21.15.39:0/3784895169 conn(0x7fa6f80fee50 msgr2=0x7fa6f8107e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:08.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.884+0000 7fa6f7200640 1 -- 172.21.15.39:0/3784895169 shutdown_connections 2024-03-18T16:09:08.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:08.884+0000 7fa6f7200640 1 -- 172.21.15.39:0/3784895169 wait complete. 2024-03-18T16:09:08.903 INFO:tasks.cephadm:Setup complete, yielding 2024-03-18T16:09:08.903 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-18T16:09:08.913 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi039.front.sepia.ceph.com 2024-03-18T16:09:08.913 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- bash -c 'ceph orch status' 2024-03-18T16:09:09.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:08 smithi194 ceph-mon[36200]: pgmap v394: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T16:09:09.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:08 smithi039 ceph-mon[31721]: pgmap v394: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T16:09:09.356 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T16:09:09.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:08 smithi196 ceph-mon[36307]: pgmap v394: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T16:09:11.300 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:10 smithi039 ceph-mon[31721]: pgmap v395: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T16:09:11.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:10 smithi196 ceph-mon[36307]: pgmap v395: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T16:09:11.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:10 smithi194 ceph-mon[36200]: pgmap v395: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T16:09:11.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:11.916+0000 7ff72e400700 1 -- 172.21.15.39:0/2008802162 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff7281023c0 msgr2=0x7ff728102820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:11.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:11.916+0000 7ff72e400700 1 --2- 172.21.15.39:0/2008802162 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff7281023c0 0x7ff728102820 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff714009990 tx=0x7ff714009ca0 comp rx=0 tx=0).stop 2024-03-18T16:09:11.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:11.916+0000 7ff72e400700 1 -- 172.21.15.39:0/2008802162 shutdown_connections 2024-03-18T16:09:11.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:11.916+0000 7ff72e400700 1 --2- 172.21.15.39:0/2008802162 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff728102d60 0x7ff728109610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:11.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:11.916+0000 7ff72e400700 1 --2- 172.21.15.39:0/2008802162 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7281011c0 0x7ff7281015e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:11.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:11.916+0000 7ff72e400700 1 --2- 172.21.15.39:0/2008802162 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff7281023c0 0x7ff728102820 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:11.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:11.916+0000 7ff72e400700 1 -- 172.21.15.39:0/2008802162 >> 172.21.15.39:0/2008802162 conn(0x7ff7280fc760 msgr2=0x7ff7280feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:11.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:11.916+0000 7ff72e400700 1 -- 172.21.15.39:0/2008802162 shutdown_connections 2024-03-18T16:09:11.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:11.916+0000 7ff72e400700 1 -- 172.21.15.39:0/2008802162 wait complete. 2024-03-18T16:09:11.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:11.917+0000 7ff72e400700 1 Processor -- start 2024-03-18T16:09:11.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:11.918+0000 7ff72e400700 1 -- start start 2024-03-18T16:09:11.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:11.918+0000 7ff72e400700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff7281011c0 0x7ff728198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:11.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:11.918+0000 7ff72e400700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7281023c0 0x7ff728198a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:11.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:11.918+0000 7ff72e400700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff728102d60 0x7ff72819fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:11.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:11.918+0000 7ff72e400700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7ff72810ce10 con 0x7ff7281023c0 2024-03-18T16:09:11.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:11.918+0000 7ff72e400700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ff72810cc90 con 0x7ff7281011c0 2024-03-18T16:09:11.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:11.918+0000 7ff72e400700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7ff72810cf90 con 0x7ff728102d60 2024-03-18T16:09:11.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:11.919+0000 7ff726a00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7281023c0 0x7ff728198a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:11.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:11.919+0000 7ff727400700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff7281011c0 0x7ff728198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:11.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:11.919+0000 7ff727e00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff728102d60 0x7ff72819fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:11.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:11.919+0000 7ff726a00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7281023c0 0x7ff728198a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:33022/0 (socket says 172.21.15.39:33022) 2024-03-18T16:09:11.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:11.919+0000 7ff726a00700 1 -- 172.21.15.39:0/714851550 learned_addr learned my addr 172.21.15.39:0/714851550 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:11.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:11.919+0000 7ff727400700 1 -- 172.21.15.39:0/714851550 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff728102d60 msgr2=0x7ff72819fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:11.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:11.919+0000 7ff727400700 1 --2- 172.21.15.39:0/714851550 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff728102d60 0x7ff72819fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:11.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:11.919+0000 7ff727400700 1 -- 172.21.15.39:0/714851550 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7281023c0 msgr2=0x7ff728198a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:11.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:11.919+0000 7ff727400700 1 --2- 172.21.15.39:0/714851550 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7281023c0 0x7ff728198a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:11.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:11.919+0000 7ff727400700 1 -- 172.21.15.39:0/714851550 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7281036b0 con 0x7ff7281011c0 2024-03-18T16:09:11.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:11.919+0000 7ff727400700 1 --2- 172.21.15.39:0/714851550 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff7281011c0 0x7ff728198540 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ff71800d720 tx=0x7ff71800da30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:11.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:11.920+0000 7ff71fe00700 1 -- 172.21.15.39:0/714851550 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff718014030 con 0x7ff7281011c0 2024-03-18T16:09:11.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:11.920+0000 7ff71fe00700 1 -- 172.21.15.39:0/714851550 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff71800fb70 con 0x7ff7281011c0 2024-03-18T16:09:11.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:11.920+0000 7ff72e400700 1 -- 172.21.15.39:0/714851550 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7281039a0 con 0x7ff7281011c0 2024-03-18T16:09:11.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:11.920+0000 7ff71fe00700 1 -- 172.21.15.39:0/714851550 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff718010df0 con 0x7ff7281011c0 2024-03-18T16:09:11.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:11.920+0000 7ff72e400700 1 -- 172.21.15.39:0/714851550 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff728103f30 con 0x7ff7281011c0 2024-03-18T16:09:11.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:11.921+0000 7ff72e400700 1 -- 172.21.15.39:0/714851550 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff728068f90 con 0x7ff7281011c0 2024-03-18T16:09:11.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:11.923+0000 7ff71fe00700 1 -- 172.21.15.39:0/714851550 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7ff718017030 con 0x7ff7281011c0 2024-03-18T16:09:11.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:11.924+0000 7ff71fe00700 1 --2- 172.21.15.39:0/714851550 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff70c0a2910 0x7ff70c0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:11.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:11.925+0000 7ff71fe00700 1 -- 172.21.15.39:0/714851550 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7ff7180ce6e0 con 0x7ff7281011c0 2024-03-18T16:09:11.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:11.925+0000 7ff726a00700 1 --2- 172.21.15.39:0/714851550 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff70c0a2910 0x7ff70c0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:11.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:11.925+0000 7ff726a00700 1 --2- 172.21.15.39:0/714851550 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff70c0a2910 0x7ff70c0a4dd0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7ff71400b2e0 tx=0x7ff71400b8f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:11.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:11.929+0000 7ff71fe00700 1 -- 172.21.15.39:0/714851550 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff71809b390 con 0x7ff7281011c0 2024-03-18T16:09:12.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:12.176+0000 7ff72e400700 1 -- 172.21.15.39:0/714851550 --> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7ff72806af20 con 0x7ff70c0a2910 2024-03-18T16:09:12.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:12.177+0000 7ff71fe00700 1 -- 172.21.15.39:0/714851550 <== mgr.14152 v2:172.21.15.39:6800/2749214286 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7ff72806af20 con 0x7ff70c0a2910 2024-03-18T16:09:12.179 INFO:teuthology.orchestra.run.smithi039.stdout:Backend: cephadm 2024-03-18T16:09:12.179 INFO:teuthology.orchestra.run.smithi039.stdout:Available: Yes 2024-03-18T16:09:12.179 INFO:teuthology.orchestra.run.smithi039.stdout:Paused: No 2024-03-18T16:09:12.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:12.180+0000 7ff72e400700 1 -- 172.21.15.39:0/714851550 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff70c0a2910 msgr2=0x7ff70c0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:12.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:12.180+0000 7ff72e400700 1 --2- 172.21.15.39:0/714851550 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff70c0a2910 0x7ff70c0a4dd0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7ff71400b2e0 tx=0x7ff71400b8f0 comp rx=0 tx=0).stop 2024-03-18T16:09:12.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:12.180+0000 7ff72e400700 1 -- 172.21.15.39:0/714851550 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff7281011c0 msgr2=0x7ff728198540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:12.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:12.180+0000 7ff72e400700 1 --2- 172.21.15.39:0/714851550 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff7281011c0 0x7ff728198540 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ff71800d720 tx=0x7ff71800da30 comp rx=0 tx=0).stop 2024-03-18T16:09:12.183 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:12.180+0000 7ff72e400700 1 -- 172.21.15.39:0/714851550 shutdown_connections 2024-03-18T16:09:12.183 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:12.180+0000 7ff72e400700 1 --2- 172.21.15.39:0/714851550 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff70c0a2910 0x7ff70c0a4dd0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:12.183 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:12.180+0000 7ff72e400700 1 --2- 172.21.15.39:0/714851550 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff728102d60 0x7ff72819fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:12.183 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:12.180+0000 7ff72e400700 1 --2- 172.21.15.39:0/714851550 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff7281011c0 0x7ff728198540 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:12.183 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:12.180+0000 7ff72e400700 1 --2- 172.21.15.39:0/714851550 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7281023c0 0x7ff728198a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:12.183 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:12.180+0000 7ff72e400700 1 -- 172.21.15.39:0/714851550 >> 172.21.15.39:0/714851550 conn(0x7ff7280fc760 msgr2=0x7ff72806a810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:12.183 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:12.181+0000 7ff72e400700 1 -- 172.21.15.39:0/714851550 shutdown_connections 2024-03-18T16:09:12.183 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:12.181+0000 7ff72e400700 1 -- 172.21.15.39:0/714851550 wait complete. 2024-03-18T16:09:13.178 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:12 smithi039 ceph-mon[31721]: pgmap v396: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-18T16:09:13.178 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:13 smithi039 ceph-mon[31721]: from='client.25229 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:09:13.234 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-18T16:09:13.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:12 smithi196 ceph-mon[36307]: pgmap v396: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-18T16:09:13.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:12 smithi196 ceph-mon[36307]: from='client.25229 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:09:13.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:12 smithi194 ceph-mon[36200]: pgmap v396: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-18T16:09:13.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:12 smithi194 ceph-mon[36200]: from='client.25229 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:09:13.735 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T16:09:15.234 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:15 smithi039 ceph-mon[31721]: pgmap v397: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T16:09:15.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:15 smithi196 ceph-mon[36307]: pgmap v397: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T16:09:15.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:15 smithi194 ceph-mon[36200]: pgmap v397: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T16:09:16.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:16.280+0000 7f5573200700 1 -- 172.21.15.39:0/600448069 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5574076300 msgr2=0x7f5574074760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:16.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:16.280+0000 7f5573200700 1 --2- 172.21.15.39:0/600448069 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5574076300 0x7f5574074760 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f55640098f0 tx=0x7f5564009c00 comp rx=0 tx=0).stop 2024-03-18T16:09:16.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:16.280+0000 7f5573200700 1 -- 172.21.15.39:0/600448069 shutdown_connections 2024-03-18T16:09:16.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:16.280+0000 7f5573200700 1 --2- 172.21.15.39:0/600448069 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f557410a110 0x7f557410c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:16.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:16.280+0000 7f5573200700 1 --2- 172.21.15.39:0/600448069 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5574076300 0x7f5574074760 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:16.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:16.280+0000 7f5573200700 1 --2- 172.21.15.39:0/600448069 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5574074d30 0x7f55740751b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:16.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:16.280+0000 7f5573200700 1 -- 172.21.15.39:0/600448069 >> 172.21.15.39:0/600448069 conn(0x7f55740fd7e0 msgr2=0x7f55740ffc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:16.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:16.280+0000 7f5573200700 1 -- 172.21.15.39:0/600448069 shutdown_connections 2024-03-18T16:09:16.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:16.280+0000 7f5573200700 1 -- 172.21.15.39:0/600448069 wait complete. 2024-03-18T16:09:16.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:16.281+0000 7f5573200700 1 Processor -- start 2024-03-18T16:09:16.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:16.281+0000 7f5573200700 1 -- start start 2024-03-18T16:09:16.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:16.282+0000 7f5573200700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5574074d30 0x7f557419a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:16.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:16.282+0000 7f5573200700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5574076300 0x7f557419ac50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:16.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:16.282+0000 7f5573200700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f557410a110 0x7f55741a1c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:16.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:16.282+0000 7f5573200700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f5574110080 con 0x7f557410a110 2024-03-18T16:09:16.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:16.282+0000 7f5573200700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f557410ff00 con 0x7f5574074d30 2024-03-18T16:09:16.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:16.282+0000 7f5573200700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f5574110200 con 0x7f5574076300 2024-03-18T16:09:16.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:16.282+0000 7f5571e00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5574074d30 0x7f557419a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:16.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:16.282+0000 7f5571400700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5574076300 0x7f557419ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:16.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:16.282+0000 7f5571e00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5574074d30 0x7f557419a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:56704/0 (socket says 172.21.15.39:56704) 2024-03-18T16:09:16.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:16.282+0000 7f5571e00700 1 -- 172.21.15.39:0/749242642 learned_addr learned my addr 172.21.15.39:0/749242642 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:16.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:16.282+0000 7f5572800700 1 --2- 172.21.15.39:0/749242642 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f557410a110 0x7f55741a1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:16.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:16.282+0000 7f5571e00700 1 -- 172.21.15.39:0/749242642 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5574076300 msgr2=0x7f557419ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:16.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:16.282+0000 7f5571e00700 1 --2- 172.21.15.39:0/749242642 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5574076300 0x7f557419ac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:16.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:16.282+0000 7f5571e00700 1 -- 172.21.15.39:0/749242642 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f557410a110 msgr2=0x7f55741a1c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:16.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:16.282+0000 7f5571e00700 1 --2- 172.21.15.39:0/749242642 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f557410a110 0x7f55741a1c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:16.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:16.283+0000 7f5571e00700 1 -- 172.21.15.39:0/749242642 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55741a2320 con 0x7f5574074d30 2024-03-18T16:09:16.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:16.283+0000 7f5571e00700 1 --2- 172.21.15.39:0/749242642 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5574074d30 0x7f557419a710 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f556400fca0 tx=0x7f5564005250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:16.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:16.283+0000 7f556aa00700 1 -- 172.21.15.39:0/749242642 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f556400f950 con 0x7f5574074d30 2024-03-18T16:09:16.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:16.283+0000 7f556aa00700 1 -- 172.21.15.39:0/749242642 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f55640054f0 con 0x7f5574074d30 2024-03-18T16:09:16.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:16.283+0000 7f556aa00700 1 -- 172.21.15.39:0/749242642 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5564003bc0 con 0x7f5574074d30 2024-03-18T16:09:16.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:16.283+0000 7f5573200700 1 -- 172.21.15.39:0/749242642 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55741a25b0 con 0x7f5574074d30 2024-03-18T16:09:16.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:16.283+0000 7f5573200700 1 -- 172.21.15.39:0/749242642 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55741a2a60 con 0x7f5574074d30 2024-03-18T16:09:16.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:16.284+0000 7f5573200700 1 -- 172.21.15.39:0/749242642 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5574068f90 con 0x7f5574074d30 2024-03-18T16:09:16.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:16.286+0000 7f556aa00700 1 -- 172.21.15.39:0/749242642 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f556400b560 con 0x7f5574074d30 2024-03-18T16:09:16.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:16.288+0000 7f556aa00700 1 --2- 172.21.15.39:0/749242642 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f55580a2ab0 0x7f55580a4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:16.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:16.288+0000 7f556aa00700 1 -- 172.21.15.39:0/749242642 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f55640cfe10 con 0x7f5574074d30 2024-03-18T16:09:16.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:16.288+0000 7f5571400700 1 --2- 172.21.15.39:0/749242642 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f55580a2ab0 0x7f55580a4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:16.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:16.289+0000 7f5571400700 1 --2- 172.21.15.39:0/749242642 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f55580a2ab0 0x7f55580a4f70 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f5574076030 tx=0x7f5560009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:16.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:16.292+0000 7f556aa00700 1 -- 172.21.15.39:0/749242642 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f55640d3030 con 0x7f5574074d30 2024-03-18T16:09:16.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:16.561+0000 7f5573200700 1 -- 172.21.15.39:0/749242642 --> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7f5574062710 con 0x7f55580a2ab0 2024-03-18T16:09:16.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:16.588+0000 7f556aa00700 1 -- 172.21.15.39:0/749242642 <== mgr.14152 v2:172.21.15.39:6800/2749214286 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4123 (secure 0 0 0) 0x7f5574062710 con 0x7f55580a2ab0 2024-03-18T16:09:16.590 INFO:teuthology.orchestra.run.smithi039.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-18T16:09:16.590 INFO:teuthology.orchestra.run.smithi039.stdout:mds.a smithi039 running (2m) 29s ago 2m 15.9M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 26e6fb6d37c4 2024-03-18T16:09:16.590 INFO:teuthology.orchestra.run.smithi039.stdout:mds.b smithi194 running (2m) 90s ago 2m 26.3M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 13068c0a3b14 2024-03-18T16:09:16.590 INFO:teuthology.orchestra.run.smithi039.stdout:mds.c smithi196 running (2m) 90s ago 2m 16.7M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 4a10438865ac 2024-03-18T16:09:16.590 INFO:teuthology.orchestra.run.smithi039.stdout:mds.d smithi039 running (3m) 29s ago 3m 12.5M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 312ac9d705b0 2024-03-18T16:09:16.590 INFO:teuthology.orchestra.run.smithi039.stdout:mds.e smithi194 running (3m) 90s ago 3m 15.6M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 315ebcc53fe1 2024-03-18T16:09:16.590 INFO:teuthology.orchestra.run.smithi039.stdout:mds.f smithi196 running (2m) 90s ago 2m 19.1M - 19.0.0-2162-g3d78f5ba be86dc8cb22b e19ea3e4220c 2024-03-18T16:09:16.590 INFO:teuthology.orchestra.run.smithi039.stdout:mds.g smithi039 running (2m) 29s ago 2m 15.4M - 19.0.0-2162-g3d78f5ba be86dc8cb22b d8fdf2cc8f12 2024-03-18T16:09:16.590 INFO:teuthology.orchestra.run.smithi039.stdout:mds.h smithi194 running (2m) 90s ago 2m 18.1M - 19.0.0-2162-g3d78f5ba be86dc8cb22b a4ae79a1fecd 2024-03-18T16:09:16.590 INFO:teuthology.orchestra.run.smithi039.stdout:mds.i smithi196 running (2m) 90s ago 2m 17.3M - 19.0.0-2162-g3d78f5ba be86dc8cb22b d4b43a4f0cf7 2024-03-18T16:09:16.590 INFO:teuthology.orchestra.run.smithi039.stdout:mds.j smithi039 running (3m) 29s ago 3m 12.6M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 035dc69f8af4 2024-03-18T16:09:16.590 INFO:teuthology.orchestra.run.smithi039.stdout:mds.k smithi194 running (3m) 90s ago 3m 18.2M - 19.0.0-2162-g3d78f5ba be86dc8cb22b e2b93397987b 2024-03-18T16:09:16.591 INFO:teuthology.orchestra.run.smithi039.stdout:mds.l smithi196 running (3m) 90s ago 3m 18.4M - 19.0.0-2162-g3d78f5ba be86dc8cb22b cefba4accc94 2024-03-18T16:09:16.591 INFO:teuthology.orchestra.run.smithi039.stdout:mgr.x smithi039 *:9283,8765 running (11m) 29s ago 11m 515M - 19.0.0-2162-g3d78f5ba be86dc8cb22b 0ad604841d44 2024-03-18T16:09:16.591 INFO:teuthology.orchestra.run.smithi039.stdout:mgr.y smithi194 *:8443,8765 running (9m) 90s ago 9m 450M - 19.0.0-2162-g3d78f5ba be86dc8cb22b d24c93dbcadc 2024-03-18T16:09:16.591 INFO:teuthology.orchestra.run.smithi039.stdout:mgr.z smithi196 *:8443,8765 running (9m) 90s ago 9m 452M - 19.0.0-2162-g3d78f5ba be86dc8cb22b fce4512a2469 2024-03-18T16:09:16.591 INFO:teuthology.orchestra.run.smithi039.stdout:mon.a smithi039 running (11m) 29s ago 11m 48.2M 2048M 19.0.0-2162-g3d78f5ba be86dc8cb22b c8c8b05ddbd7 2024-03-18T16:09:16.591 INFO:teuthology.orchestra.run.smithi039.stdout:mon.b smithi194 running (9m) 90s ago 9m 48.3M 2048M 19.0.0-2162-g3d78f5ba be86dc8cb22b 171673f257b8 2024-03-18T16:09:16.591 INFO:teuthology.orchestra.run.smithi039.stdout:mon.c smithi196 running (9m) 90s ago 9m 41.0M 2048M 19.0.0-2162-g3d78f5ba be86dc8cb22b 83e5d1a96c46 2024-03-18T16:09:16.591 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0 smithi039 running (8m) 29s ago 8m 44.9M 4272M 19.0.0-2162-g3d78f5ba be86dc8cb22b 2bd2b2c0bfbe 2024-03-18T16:09:16.591 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1 smithi194 running (8m) 90s ago 8m 54.7M 4270M 19.0.0-2162-g3d78f5ba be86dc8cb22b 082da29464fe 2024-03-18T16:09:16.591 INFO:teuthology.orchestra.run.smithi039.stdout:osd.2 smithi196 running (7m) 90s ago 7m 54.8M 4270M 19.0.0-2162-g3d78f5ba be86dc8cb22b 2041433785f6 2024-03-18T16:09:16.591 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3 smithi039 running (7m) 29s ago 7m 43.0M 4272M 19.0.0-2162-g3d78f5ba be86dc8cb22b ba1dc46ed7d3 2024-03-18T16:09:16.591 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4 smithi194 running (6m) 90s ago 6m 50.8M 4270M 19.0.0-2162-g3d78f5ba be86dc8cb22b 9cac66029169 2024-03-18T16:09:16.591 INFO:teuthology.orchestra.run.smithi039.stdout:osd.5 smithi196 running (6m) 90s ago 6m 52.7M 4270M 19.0.0-2162-g3d78f5ba be86dc8cb22b 223e035c854b 2024-03-18T16:09:16.591 INFO:teuthology.orchestra.run.smithi039.stdout:osd.6 smithi039 running (5m) 29s ago 5m 49.1M 4272M 19.0.0-2162-g3d78f5ba be86dc8cb22b 789035e5d60f 2024-03-18T16:09:16.591 INFO:teuthology.orchestra.run.smithi039.stdout:osd.7 smithi194 running (5m) 90s ago 5m 47.0M 4270M 19.0.0-2162-g3d78f5ba be86dc8cb22b 770cf6249f6f 2024-03-18T16:09:16.592 INFO:teuthology.orchestra.run.smithi039.stdout:osd.8 smithi196 running (5m) 90s ago 5m 55.2M 4270M 19.0.0-2162-g3d78f5ba be86dc8cb22b b9c41d44520c 2024-03-18T16:09:16.592 INFO:teuthology.orchestra.run.smithi039.stdout:osd.9 smithi039 running (4m) 29s ago 4m 41.1M 4272M 19.0.0-2162-g3d78f5ba be86dc8cb22b 12c25915865a 2024-03-18T16:09:16.592 INFO:teuthology.orchestra.run.smithi039.stdout:osd.10 smithi194 running (4m) 90s ago 4m 54.4M 4270M 19.0.0-2162-g3d78f5ba be86dc8cb22b f5126ff7bc17 2024-03-18T16:09:16.592 INFO:teuthology.orchestra.run.smithi039.stdout:osd.11 smithi196 running (3m) 90s ago 3m 52.2M 4270M 19.0.0-2162-g3d78f5ba be86dc8cb22b c87aa051aa63 2024-03-18T16:09:16.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:16.591+0000 7f5573200700 1 -- 172.21.15.39:0/749242642 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f55580a2ab0 msgr2=0x7f55580a4f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:16.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:16.591+0000 7f5573200700 1 --2- 172.21.15.39:0/749242642 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f55580a2ab0 0x7f55580a4f70 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f5574076030 tx=0x7f5560009310 comp rx=0 tx=0).stop 2024-03-18T16:09:16.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:16.591+0000 7f5573200700 1 -- 172.21.15.39:0/749242642 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5574074d30 msgr2=0x7f557419a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:16.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:16.591+0000 7f5573200700 1 --2- 172.21.15.39:0/749242642 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5574074d30 0x7f557419a710 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f556400fca0 tx=0x7f5564005250 comp rx=0 tx=0).stop 2024-03-18T16:09:16.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:16.592+0000 7f5573200700 1 -- 172.21.15.39:0/749242642 shutdown_connections 2024-03-18T16:09:16.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:16.592+0000 7f5573200700 1 --2- 172.21.15.39:0/749242642 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f55580a2ab0 0x7f55580a4f70 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:16.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:16.592+0000 7f5573200700 1 --2- 172.21.15.39:0/749242642 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f557410a110 0x7f55741a1c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:16.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:16.592+0000 7f5573200700 1 --2- 172.21.15.39:0/749242642 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5574074d30 0x7f557419a710 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:16.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:16.592+0000 7f5573200700 1 --2- 172.21.15.39:0/749242642 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5574076300 0x7f557419ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:16.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:16.592+0000 7f5573200700 1 -- 172.21.15.39:0/749242642 >> 172.21.15.39:0/749242642 conn(0x7f55740fd7e0 msgr2=0x7f5574106010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:16.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:16.592+0000 7f5573200700 1 -- 172.21.15.39:0/749242642 shutdown_connections 2024-03-18T16:09:16.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:16.592+0000 7f5573200700 1 -- 172.21.15.39:0/749242642 wait complete. 2024-03-18T16:09:17.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:17 smithi039 ceph-mon[31721]: pgmap v398: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T16:09:17.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:17 smithi196 ceph-mon[36307]: pgmap v398: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T16:09:17.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:17 smithi194 ceph-mon[36200]: pgmap v398: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T16:09:17.620 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-18T16:09:18.082 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T16:09:18.288 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:18 smithi039 ceph-mon[31721]: from='client.25235 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:09:18.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:18 smithi196 ceph-mon[36307]: from='client.25235 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:09:18.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:18 smithi194 ceph-mon[36200]: from='client.25235 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:09:19.301 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:19 smithi039 ceph-mon[31721]: pgmap v399: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T16:09:19.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:19 smithi196 ceph-mon[36307]: pgmap v399: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T16:09:19.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:19 smithi194 ceph-mon[36200]: pgmap v399: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T16:09:20.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:20.588+0000 7fb71b600700 1 -- 172.21.15.39:0/1525262993 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb714103510 msgr2=0x7fb714103970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:20.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:20.588+0000 7fb71b600700 1 --2- 172.21.15.39:0/1525262993 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb714103510 0x7fb714103970 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fb7000099c0 tx=0x7fb700009cd0 comp rx=0 tx=0).stop 2024-03-18T16:09:20.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:20.589+0000 7fb71b600700 1 -- 172.21.15.39:0/1525262993 shutdown_connections 2024-03-18T16:09:20.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:20.589+0000 7fb71b600700 1 --2- 172.21.15.39:0/1525262993 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb714103eb0 0x7fb71410a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:20.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:20.589+0000 7fb71b600700 1 --2- 172.21.15.39:0/1525262993 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb714102310 0x7fb714102730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:20.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:20.589+0000 7fb71b600700 1 --2- 172.21.15.39:0/1525262993 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb714103510 0x7fb714103970 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:20.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:20.589+0000 7fb71b600700 1 -- 172.21.15.39:0/1525262993 >> 172.21.15.39:0/1525262993 conn(0x7fb7140fd870 msgr2=0x7fb7140ffcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:20.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:20.589+0000 7fb71b600700 1 -- 172.21.15.39:0/1525262993 shutdown_connections 2024-03-18T16:09:20.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:20.589+0000 7fb71b600700 1 -- 172.21.15.39:0/1525262993 wait complete. 2024-03-18T16:09:20.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:20.590+0000 7fb71b600700 1 Processor -- start 2024-03-18T16:09:20.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:20.590+0000 7fb71b600700 1 -- start start 2024-03-18T16:09:20.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:20.590+0000 7fb71b600700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb714102310 0x7fb714198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:20.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:20.590+0000 7fb71b600700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb714103510 0x7fb714198ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:20.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:20.590+0000 7fb71b600700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb714103eb0 0x7fb71419fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:20.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:20.590+0000 7fb71b600700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fb71410df30 con 0x7fb714103510 2024-03-18T16:09:20.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:20.590+0000 7fb71b600700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fb71410ddb0 con 0x7fb714102310 2024-03-18T16:09:20.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:20.590+0000 7fb71b600700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fb71410e0b0 con 0x7fb714103eb0 2024-03-18T16:09:20.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:20.591+0000 7fb718c00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb714102310 0x7fb714198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:20.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:20.591+0000 7fb718c00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb714102310 0x7fb714198570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:35344/0 (socket says 172.21.15.39:35344) 2024-03-18T16:09:20.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:20.591+0000 7fb713e00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb714103510 0x7fb714198ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:20.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:20.591+0000 7fb718c00700 1 -- 172.21.15.39:0/3654743349 learned_addr learned my addr 172.21.15.39:0/3654743349 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:20.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:20.591+0000 7fb719600700 1 --2- 172.21.15.39:0/3654743349 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb714103eb0 0x7fb71419fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:20.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:20.591+0000 7fb718c00700 1 -- 172.21.15.39:0/3654743349 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb714103eb0 msgr2=0x7fb71419fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:20.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:20.591+0000 7fb718c00700 1 --2- 172.21.15.39:0/3654743349 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb714103eb0 0x7fb71419fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:20.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:20.591+0000 7fb718c00700 1 -- 172.21.15.39:0/3654743349 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb714103510 msgr2=0x7fb714198ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:20.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:20.591+0000 7fb718c00700 1 --2- 172.21.15.39:0/3654743349 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb714103510 0x7fb714198ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:20.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:20.591+0000 7fb718c00700 1 -- 172.21.15.39:0/3654743349 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7141a01d0 con 0x7fb714102310 2024-03-18T16:09:20.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:20.592+0000 7fb718c00700 1 --2- 172.21.15.39:0/3654743349 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb714102310 0x7fb714198570 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fb70800e970 tx=0x7fb70800ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:20.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:20.592+0000 7fb711600700 1 -- 172.21.15.39:0/3654743349 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb708004220 con 0x7fb714102310 2024-03-18T16:09:20.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:20.592+0000 7fb711600700 1 -- 172.21.15.39:0/3654743349 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb7080043c0 con 0x7fb714102310 2024-03-18T16:09:20.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:20.592+0000 7fb71b600700 1 -- 172.21.15.39:0/3654743349 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7141a04c0 con 0x7fb714102310 2024-03-18T16:09:20.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:20.592+0000 7fb71b600700 1 -- 172.21.15.39:0/3654743349 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7141a0a50 con 0x7fb714102310 2024-03-18T16:09:20.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:20.592+0000 7fb711600700 1 -- 172.21.15.39:0/3654743349 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb708003df0 con 0x7fb714102310 2024-03-18T16:09:20.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:20.594+0000 7fb71b600700 1 -- 172.21.15.39:0/3654743349 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6f4005330 con 0x7fb714102310 2024-03-18T16:09:20.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:20.596+0000 7fb711600700 1 -- 172.21.15.39:0/3654743349 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fb708044910 con 0x7fb714102310 2024-03-18T16:09:20.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:20.598+0000 7fb711600700 1 --2- 172.21.15.39:0/3654743349 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb6fc0a2910 0x7fb6fc0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:20.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:20.598+0000 7fb711600700 1 -- 172.21.15.39:0/3654743349 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fb7080ceeb0 con 0x7fb714102310 2024-03-18T16:09:20.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:20.599+0000 7fb713e00700 1 --2- 172.21.15.39:0/3654743349 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb6fc0a2910 0x7fb6fc0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:20.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:20.599+0000 7fb713e00700 1 --2- 172.21.15.39:0/3654743349 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb6fc0a2910 0x7fb6fc0a4dd0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fb700000c00 tx=0x7fb700005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:20.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:20.604+0000 7fb711600700 1 -- 172.21.15.39:0/3654743349 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb7080203a0 con 0x7fb714102310 2024-03-18T16:09:20.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:20.839+0000 7fb71b600700 1 -- 172.21.15.39:0/3654743349 --> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7fb6f4000bc0 con 0x7fb6fc0a2910 2024-03-18T16:09:20.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:20.851+0000 7fb711600700 1 -- 172.21.15.39:0/3654743349 <== mgr.14152 v2:172.21.15.39:6800/2749214286 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7fb6f4000bc0 con 0x7fb6fc0a2910 2024-03-18T16:09:20.852 INFO:teuthology.orchestra.run.smithi039.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-18T16:09:20.852 INFO:teuthology.orchestra.run.smithi039.stdout:mds.all 12/12 95s ago 3m smithi039=a;smithi039=d;smithi039=g;smithi039=j;smithi194=b;smithi194=e;smithi194=h;smithi194=k;smithi196=c;smithi196=f;smithi196=i;smithi196=l;count:12 2024-03-18T16:09:20.853 INFO:teuthology.orchestra.run.smithi039.stdout:mgr 3/3 95s ago 9m smithi039=x;smithi194=y;smithi196=z;count:3 2024-03-18T16:09:20.853 INFO:teuthology.orchestra.run.smithi039.stdout:mon 3/3 95s ago 9m smithi039:172.21.15.39=a;smithi194:172.21.15.194=b;smithi196:172.21.15.196=c;count:3 2024-03-18T16:09:20.853 INFO:teuthology.orchestra.run.smithi039.stdout:osd 12 95s ago - 2024-03-18T16:09:20.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:20.853+0000 7fb71b600700 1 -- 172.21.15.39:0/3654743349 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb6fc0a2910 msgr2=0x7fb6fc0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:20.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:20.853+0000 7fb71b600700 1 --2- 172.21.15.39:0/3654743349 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb6fc0a2910 0x7fb6fc0a4dd0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fb700000c00 tx=0x7fb700005fb0 comp rx=0 tx=0).stop 2024-03-18T16:09:20.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:20.853+0000 7fb71b600700 1 -- 172.21.15.39:0/3654743349 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb714102310 msgr2=0x7fb714198570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:20.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:20.853+0000 7fb71b600700 1 --2- 172.21.15.39:0/3654743349 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb714102310 0x7fb714198570 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fb70800e970 tx=0x7fb70800ec80 comp rx=0 tx=0).stop 2024-03-18T16:09:20.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:20.854+0000 7fb71b600700 1 -- 172.21.15.39:0/3654743349 shutdown_connections 2024-03-18T16:09:20.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:20.854+0000 7fb71b600700 1 --2- 172.21.15.39:0/3654743349 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb6fc0a2910 0x7fb6fc0a4dd0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:20.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:20.854+0000 7fb71b600700 1 --2- 172.21.15.39:0/3654743349 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb714103eb0 0x7fb71419fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:20.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:20.854+0000 7fb71b600700 1 --2- 172.21.15.39:0/3654743349 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb714102310 0x7fb714198570 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:20.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:20.854+0000 7fb71b600700 1 --2- 172.21.15.39:0/3654743349 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb714103510 0x7fb714198ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:20.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:20.854+0000 7fb71b600700 1 -- 172.21.15.39:0/3654743349 >> 172.21.15.39:0/3654743349 conn(0x7fb7140fd870 msgr2=0x7fb714106810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:20.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:20.854+0000 7fb71b600700 1 -- 172.21.15.39:0/3654743349 shutdown_connections 2024-03-18T16:09:20.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:20.854+0000 7fb71b600700 1 -- 172.21.15.39:0/3654743349 wait complete. 2024-03-18T16:09:21.321 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:21 smithi039 ceph-mon[31721]: pgmap v400: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T16:09:21.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:21 smithi196 ceph-mon[36307]: pgmap v400: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T16:09:21.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:21 smithi194 ceph-mon[36200]: pgmap v400: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T16:09:21.901 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-18T16:09:22.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:22 smithi039 ceph-mon[31721]: from='client.25241 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:09:22.376 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T16:09:22.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:22 smithi196 ceph-mon[36307]: from='client.25241 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:09:22.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:22 smithi194 ceph-mon[36200]: from='client.25241 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:09:23.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:23 smithi039 ceph-mon[31721]: pgmap v401: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-18T16:09:23.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:23 smithi196 ceph-mon[36307]: pgmap v401: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-18T16:09:23.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:23 smithi194 ceph-mon[36200]: pgmap v401: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-18T16:09:24.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:24.613+0000 7fea7a200700 1 -- 172.21.15.39:0/3956670816 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fea74100b80 msgr2=0x7fea74100fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:24.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:24.613+0000 7fea7a200700 1 --2- 172.21.15.39:0/3956670816 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fea74100b80 0x7fea74100fa0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fea64009990 tx=0x7fea64009ca0 comp rx=0 tx=0).stop 2024-03-18T16:09:24.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:24.614+0000 7fea7a200700 1 -- 172.21.15.39:0/3956670816 shutdown_connections 2024-03-18T16:09:24.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:24.614+0000 7fea7a200700 1 --2- 172.21.15.39:0/3956670816 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fea74101ea0 0x7fea74102320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:24.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:24.614+0000 7fea7a200700 1 --2- 172.21.15.39:0/3956670816 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fea74100b80 0x7fea74100fa0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:24.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:24.614+0000 7fea7a200700 1 --2- 172.21.15.39:0/3956670816 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fea741014e0 0x7fea74101960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:24.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:24.614+0000 7fea7a200700 1 -- 172.21.15.39:0/3956670816 >> 172.21.15.39:0/3956670816 conn(0x7fea740fc7a0 msgr2=0x7fea740febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:24.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:24.614+0000 7fea7a200700 1 -- 172.21.15.39:0/3956670816 shutdown_connections 2024-03-18T16:09:24.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:24.614+0000 7fea7a200700 1 -- 172.21.15.39:0/3956670816 wait complete. 2024-03-18T16:09:24.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:24.615+0000 7fea7a200700 1 Processor -- start 2024-03-18T16:09:24.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:24.615+0000 7fea7a200700 1 -- start start 2024-03-18T16:09:24.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:24.615+0000 7fea7a200700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fea74100b80 0x7fea7419a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:24.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:24.615+0000 7fea7a200700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fea741014e0 0x7fea7419acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:24.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:24.615+0000 7fea7a200700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fea74101ea0 0x7fea741a1d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:24.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:24.615+0000 7fea7a200700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fea74110100 con 0x7fea741014e0 2024-03-18T16:09:24.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:24.615+0000 7fea7a200700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fea7410ff80 con 0x7fea74101ea0 2024-03-18T16:09:24.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:24.615+0000 7fea7a200700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fea74110280 con 0x7fea74100b80 2024-03-18T16:09:24.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:24.616+0000 7fea72a00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fea741014e0 0x7fea7419acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:24.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:24.616+0000 7fea73e00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fea74101ea0 0x7fea741a1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:24.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:24.616+0000 7fea73e00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fea74101ea0 0x7fea741a1d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:35368/0 (socket says 172.21.15.39:35368) 2024-03-18T16:09:24.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:24.616+0000 7fea73e00700 1 -- 172.21.15.39:0/1548214909 learned_addr learned my addr 172.21.15.39:0/1548214909 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:24.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:24.616+0000 7fea73400700 1 --2- 172.21.15.39:0/1548214909 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fea74100b80 0x7fea7419a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:24.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:24.616+0000 7fea73e00700 1 -- 172.21.15.39:0/1548214909 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fea74100b80 msgr2=0x7fea7419a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:24.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:24.616+0000 7fea73e00700 1 --2- 172.21.15.39:0/1548214909 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fea74100b80 0x7fea7419a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:24.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:24.616+0000 7fea73e00700 1 -- 172.21.15.39:0/1548214909 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fea741014e0 msgr2=0x7fea7419acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:24.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:24.616+0000 7fea73e00700 1 --2- 172.21.15.39:0/1548214909 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fea741014e0 0x7fea7419acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:24.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:24.616+0000 7fea73e00700 1 -- 172.21.15.39:0/1548214909 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea741a23d0 con 0x7fea74101ea0 2024-03-18T16:09:24.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:24.616+0000 7fea73e00700 1 --2- 172.21.15.39:0/1548214909 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fea74101ea0 0x7fea741a1d30 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fea6c00ef30 tx=0x7fea6c00c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:24.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:24.617+0000 7fea6be00700 1 -- 172.21.15.39:0/1548214909 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea6c0092e0 con 0x7fea74101ea0 2024-03-18T16:09:24.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:24.617+0000 7fea6be00700 1 -- 172.21.15.39:0/1548214909 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea6c004bb0 con 0x7fea74101ea0 2024-03-18T16:09:24.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:24.617+0000 7fea6be00700 1 -- 172.21.15.39:0/1548214909 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea6c004280 con 0x7fea74101ea0 2024-03-18T16:09:24.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:24.617+0000 7fea7a200700 1 -- 172.21.15.39:0/1548214909 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea741a2660 con 0x7fea74101ea0 2024-03-18T16:09:24.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:24.617+0000 7fea7a200700 1 -- 172.21.15.39:0/1548214909 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea741a2b50 con 0x7fea74101ea0 2024-03-18T16:09:24.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:24.618+0000 7fea7a200700 1 -- 172.21.15.39:0/1548214909 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea74068f90 con 0x7fea74101ea0 2024-03-18T16:09:24.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:24.622+0000 7fea6be00700 1 -- 172.21.15.39:0/1548214909 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fea6c01b030 con 0x7fea74101ea0 2024-03-18T16:09:24.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:24.623+0000 7fea6be00700 1 --2- 172.21.15.39:0/1548214909 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fea580a2910 0x7fea580a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:24.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:24.623+0000 7fea6be00700 1 -- 172.21.15.39:0/1548214909 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fea6c0cf1c0 con 0x7fea74101ea0 2024-03-18T16:09:24.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:24.624+0000 7fea73400700 1 --2- 172.21.15.39:0/1548214909 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fea580a2910 0x7fea580a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:24.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:24.624+0000 7fea6be00700 1 -- 172.21.15.39:0/1548214909 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fea6c022020 con 0x7fea74101ea0 2024-03-18T16:09:24.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:24.624+0000 7fea73400700 1 --2- 172.21.15.39:0/1548214909 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fea580a2910 0x7fea580a4dd0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fea6400b7b0 tx=0x7fea6400b880 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:24.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:24.860+0000 7fea7a200700 1 -- 172.21.15.39:0/1548214909 --> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7fea740627d0 con 0x7fea580a2910 2024-03-18T16:09:24.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:24.862+0000 7fea6be00700 1 -- 172.21.15.39:0/1548214909 <== mgr.14152 v2:172.21.15.39:6800/2749214286 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7fea740627d0 con 0x7fea580a2910 2024-03-18T16:09:24.865 INFO:teuthology.orchestra.run.smithi039.stdout:HOST ADDR LABELS STATUS 2024-03-18T16:09:24.865 INFO:teuthology.orchestra.run.smithi039.stdout:smithi039 172.21.15.39 2024-03-18T16:09:24.866 INFO:teuthology.orchestra.run.smithi039.stdout:smithi194 172.21.15.194 2024-03-18T16:09:24.866 INFO:teuthology.orchestra.run.smithi039.stdout:smithi196 172.21.15.196 2024-03-18T16:09:24.866 INFO:teuthology.orchestra.run.smithi039.stdout:3 hosts in cluster 2024-03-18T16:09:24.868 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:24.865+0000 7fea7a200700 1 -- 172.21.15.39:0/1548214909 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fea580a2910 msgr2=0x7fea580a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:24.868 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:24.865+0000 7fea7a200700 1 --2- 172.21.15.39:0/1548214909 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fea580a2910 0x7fea580a4dd0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fea6400b7b0 tx=0x7fea6400b880 comp rx=0 tx=0).stop 2024-03-18T16:09:24.868 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:24.865+0000 7fea7a200700 1 -- 172.21.15.39:0/1548214909 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fea74101ea0 msgr2=0x7fea741a1d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:24.868 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:24.865+0000 7fea7a200700 1 --2- 172.21.15.39:0/1548214909 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fea74101ea0 0x7fea741a1d30 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fea6c00ef30 tx=0x7fea6c00c3a0 comp rx=0 tx=0).stop 2024-03-18T16:09:24.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:24.866+0000 7fea7a200700 1 -- 172.21.15.39:0/1548214909 shutdown_connections 2024-03-18T16:09:24.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:24.866+0000 7fea7a200700 1 --2- 172.21.15.39:0/1548214909 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fea580a2910 0x7fea580a4dd0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:24.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:24.866+0000 7fea7a200700 1 --2- 172.21.15.39:0/1548214909 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fea74101ea0 0x7fea741a1d30 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:24.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:24.866+0000 7fea7a200700 1 --2- 172.21.15.39:0/1548214909 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fea74100b80 0x7fea7419a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:24.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:24.866+0000 7fea7a200700 1 --2- 172.21.15.39:0/1548214909 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fea741014e0 0x7fea7419acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:24.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:24.866+0000 7fea7a200700 1 -- 172.21.15.39:0/1548214909 >> 172.21.15.39:0/1548214909 conn(0x7fea740fc7a0 msgr2=0x7fea740fe2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:24.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:24.866+0000 7fea7a200700 1 -- 172.21.15.39:0/1548214909 shutdown_connections 2024-03-18T16:09:24.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:24.866+0000 7fea7a200700 1 -- 172.21.15.39:0/1548214909 wait complete. 2024-03-18T16:09:25.311 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:25 smithi039 ceph-mon[31721]: pgmap v402: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T16:09:25.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:25 smithi196 ceph-mon[36307]: pgmap v402: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T16:09:25.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:25 smithi194 ceph-mon[36200]: pgmap v402: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T16:09:25.920 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-18T16:09:26.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:26 smithi039 ceph-mon[31721]: from='client.25247 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:09:26.395 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T16:09:26.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:26 smithi196 ceph-mon[36307]: from='client.25247 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:09:26.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:26 smithi194 ceph-mon[36200]: from='client.25247 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:09:27.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:27 smithi039 ceph-mon[31721]: pgmap v403: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T16:09:27.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:27 smithi196 ceph-mon[36307]: pgmap v403: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T16:09:27.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:27 smithi194 ceph-mon[36200]: pgmap v403: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T16:09:28.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:28.742+0000 7f1ebce00700 1 -- 172.21.15.39:0/765119525 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1eb8103500 msgr2=0x7f1eb8103960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:28.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:28.742+0000 7f1ebce00700 1 --2- 172.21.15.39:0/765119525 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1eb8103500 0x7f1eb8103960 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f1ea4009990 tx=0x7f1ea4009ca0 comp rx=0 tx=0).stop 2024-03-18T16:09:28.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:28.743+0000 7f1ebce00700 1 -- 172.21.15.39:0/765119525 shutdown_connections 2024-03-18T16:09:28.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:28.743+0000 7f1ebce00700 1 --2- 172.21.15.39:0/765119525 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1eb8103ea0 0x7f1eb810a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:28.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:28.743+0000 7f1ebce00700 1 --2- 172.21.15.39:0/765119525 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1eb8102300 0x7f1eb8102720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:28.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:28.743+0000 7f1ebce00700 1 --2- 172.21.15.39:0/765119525 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1eb8103500 0x7f1eb8103960 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:28.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:28.743+0000 7f1ebce00700 1 -- 172.21.15.39:0/765119525 >> 172.21.15.39:0/765119525 conn(0x7f1eb80fd8e0 msgr2=0x7f1eb80ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:28.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:28.743+0000 7f1ebce00700 1 -- 172.21.15.39:0/765119525 shutdown_connections 2024-03-18T16:09:28.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:28.743+0000 7f1ebce00700 1 -- 172.21.15.39:0/765119525 wait complete. 2024-03-18T16:09:28.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:28.744+0000 7f1ebce00700 1 Processor -- start 2024-03-18T16:09:28.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:28.744+0000 7f1ebce00700 1 -- start start 2024-03-18T16:09:28.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:28.744+0000 7f1ebce00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1eb8102300 0x7f1eb819a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:28.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:28.744+0000 7f1ebce00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1eb8103500 0x7f1eb819ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:28.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:28.744+0000 7f1ebce00700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1eb8103ea0 0x7f1eb81a1d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:28.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:28.745+0000 7f1ebce00700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f1eb810df20 con 0x7f1eb8103ea0 2024-03-18T16:09:28.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:28.745+0000 7f1ebce00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f1eb810dda0 con 0x7f1eb8102300 2024-03-18T16:09:28.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:28.745+0000 7f1ebce00700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f1eb810e0a0 con 0x7f1eb8103500 2024-03-18T16:09:28.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:28.745+0000 7f1eb5e00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1eb8102300 0x7f1eb819a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:28.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:28.745+0000 7f1eb6800700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1eb8103ea0 0x7f1eb81a1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:28.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:28.745+0000 7f1eb5e00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1eb8102300 0x7f1eb819a7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:43384/0 (socket says 172.21.15.39:43384) 2024-03-18T16:09:28.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:28.745+0000 7f1eb5e00700 1 -- 172.21.15.39:0/3647258441 learned_addr learned my addr 172.21.15.39:0/3647258441 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:28.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:28.745+0000 7f1eb5400700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1eb8103500 0x7f1eb819ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:28.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:28.745+0000 7f1eb5400700 1 -- 172.21.15.39:0/3647258441 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1eb8102300 msgr2=0x7f1eb819a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:28.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:28.745+0000 7f1eb5400700 1 --2- 172.21.15.39:0/3647258441 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1eb8102300 0x7f1eb819a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:28.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:28.745+0000 7f1eb5400700 1 -- 172.21.15.39:0/3647258441 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1eb8103ea0 msgr2=0x7f1eb81a1d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:28.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:28.745+0000 7f1eb5400700 1 --2- 172.21.15.39:0/3647258441 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1eb8103ea0 0x7f1eb81a1d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:28.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:28.745+0000 7f1eb5400700 1 -- 172.21.15.39:0/3647258441 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1eb81a23b0 con 0x7f1eb8103500 2024-03-18T16:09:28.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:28.745+0000 7f1eb6800700 1 --2- 172.21.15.39:0/3647258441 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1eb8103ea0 0x7f1eb81a1d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:09:28.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:28.746+0000 7f1eb5400700 1 --2- 172.21.15.39:0/3647258441 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1eb8103500 0x7f1eb819ace0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f1ea40098e0 tx=0x7f1ea4009580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:28.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:28.746+0000 7f1eaea00700 1 -- 172.21.15.39:0/3647258441 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ea4026070 con 0x7f1eb8103500 2024-03-18T16:09:28.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:28.746+0000 7f1eaea00700 1 -- 172.21.15.39:0/3647258441 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ea400fd00 con 0x7f1eb8103500 2024-03-18T16:09:28.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:28.746+0000 7f1eaea00700 1 -- 172.21.15.39:0/3647258441 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ea4021e60 con 0x7f1eb8103500 2024-03-18T16:09:28.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:28.746+0000 7f1ebce00700 1 -- 172.21.15.39:0/3647258441 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1eb81a2640 con 0x7f1eb8103500 2024-03-18T16:09:28.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:28.746+0000 7f1ebce00700 1 -- 172.21.15.39:0/3647258441 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1eb81a2a40 con 0x7f1eb8103500 2024-03-18T16:09:28.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:28.747+0000 7f1ebce00700 1 -- 172.21.15.39:0/3647258441 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1eb8068f90 con 0x7f1eb8103500 2024-03-18T16:09:28.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:28.749+0000 7f1eaea00700 1 -- 172.21.15.39:0/3647258441 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f1ea4009d10 con 0x7f1eb8103500 2024-03-18T16:09:28.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:28.751+0000 7f1eaea00700 1 --2- 172.21.15.39:0/3647258441 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1e9c0a2910 0x7f1e9c0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:28.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:28.751+0000 7f1eaea00700 1 -- 172.21.15.39:0/3647258441 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f1ea40cfb20 con 0x7f1eb8103500 2024-03-18T16:09:28.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:28.751+0000 7f1eb5e00700 1 --2- 172.21.15.39:0/3647258441 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1e9c0a2910 0x7f1e9c0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:28.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:28.752+0000 7f1eb5e00700 1 --2- 172.21.15.39:0/3647258441 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1e9c0a2910 0x7f1e9c0a4dd0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f1ea8005950 tx=0x7f1ea8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:28.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:28.753+0000 7f1eaea00700 1 -- 172.21.15.39:0/3647258441 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1ea4013070 con 0x7f1eb8103500 2024-03-18T16:09:28.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:28.990+0000 7f1ebce00700 1 -- 172.21.15.39:0/3647258441 --> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7f1eb8075c20 con 0x7f1e9c0a2910 2024-03-18T16:09:28.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:28.994+0000 7f1eaea00700 1 -- 172.21.15.39:0/3647258441 <== mgr.14152 v2:172.21.15.39:6800/2749214286 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7f1eb8075c20 con 0x7f1e9c0a2910 2024-03-18T16:09:28.997 INFO:teuthology.orchestra.run.smithi039.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-18T16:09:28.997 INFO:teuthology.orchestra.run.smithi039.stdout:smithi039 /dev/nvme0n1 ssd INTEL_SSDPEDKE040T7_BTLE736102QX4P0KGN 3726G No 4m ago Has a FileSystem, LVM detected 2024-03-18T16:09:28.997 INFO:teuthology.orchestra.run.smithi039.stdout:smithi194 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT6240002A400BGN 372G No 4m ago Has a FileSystem, LVM detected 2024-03-18T16:09:28.997 INFO:teuthology.orchestra.run.smithi039.stdout:smithi196 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT6246000Z400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-03-18T16:09:29.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:28.998+0000 7f1ebce00700 1 -- 172.21.15.39:0/3647258441 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1e9c0a2910 msgr2=0x7f1e9c0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:29.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:28.998+0000 7f1ebce00700 1 --2- 172.21.15.39:0/3647258441 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1e9c0a2910 0x7f1e9c0a4dd0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f1ea8005950 tx=0x7f1ea8009260 comp rx=0 tx=0).stop 2024-03-18T16:09:29.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:28.998+0000 7f1ebce00700 1 -- 172.21.15.39:0/3647258441 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1eb8103500 msgr2=0x7f1eb819ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:29.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:28.998+0000 7f1ebce00700 1 --2- 172.21.15.39:0/3647258441 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1eb8103500 0x7f1eb819ace0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f1ea40098e0 tx=0x7f1ea4009580 comp rx=0 tx=0).stop 2024-03-18T16:09:29.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:28.998+0000 7f1ebce00700 1 -- 172.21.15.39:0/3647258441 shutdown_connections 2024-03-18T16:09:29.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:28.998+0000 7f1ebce00700 1 --2- 172.21.15.39:0/3647258441 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1e9c0a2910 0x7f1e9c0a4dd0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:29.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:28.998+0000 7f1ebce00700 1 --2- 172.21.15.39:0/3647258441 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1eb8103ea0 0x7f1eb81a1d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:29.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:28.998+0000 7f1ebce00700 1 --2- 172.21.15.39:0/3647258441 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1eb8102300 0x7f1eb819a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:29.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:28.998+0000 7f1ebce00700 1 --2- 172.21.15.39:0/3647258441 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1eb8103500 0x7f1eb819ace0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:29.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:28.998+0000 7f1ebce00700 1 -- 172.21.15.39:0/3647258441 >> 172.21.15.39:0/3647258441 conn(0x7f1eb80fd8e0 msgr2=0x7f1eb8074500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:29.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:28.999+0000 7f1ebce00700 1 -- 172.21.15.39:0/3647258441 shutdown_connections 2024-03-18T16:09:29.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:28.999+0000 7f1ebce00700 1 -- 172.21.15.39:0/3647258441 wait complete. 2024-03-18T16:09:29.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:29 smithi039 ceph-mon[31721]: pgmap v404: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T16:09:29.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:29 smithi196 ceph-mon[36307]: pgmap v404: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T16:09:29.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:29 smithi194 ceph-mon[36200]: pgmap v404: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T16:09:29.945 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-18T16:09:29.955 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi039.front.sepia.ceph.com 2024-03-18T16:09:29.955 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- bash -c 'ceph fs dump' 2024-03-18T16:09:30.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:30 smithi039 ceph-mon[31721]: from='client.25273 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:09:30.433 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T16:09:30.477 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:30 smithi196 ceph-mon[36307]: from='client.25273 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:09:30.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:30 smithi194 ceph-mon[36200]: from='client.25273 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-18T16:09:31.164 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:31 smithi039 ceph-mon[31721]: pgmap v405: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T16:09:31.379 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:31 smithi196 ceph-mon[36307]: pgmap v405: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T16:09:31.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:31 smithi194 ceph-mon[36200]: pgmap v405: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T16:09:32.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:32.960+0000 7f400a200700 1 -- 172.21.15.39:0/3484331703 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4004101400 msgr2=0x7f4004101880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:32.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:32.960+0000 7f400a200700 1 --2- 172.21.15.39:0/3484331703 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4004101400 0x7f4004101880 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f3ff0009990 tx=0x7f3ff0009ca0 comp rx=0 tx=0).stop 2024-03-18T16:09:32.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:32.960+0000 7f400a200700 1 -- 172.21.15.39:0/3484331703 shutdown_connections 2024-03-18T16:09:32.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:32.960+0000 7f400a200700 1 --2- 172.21.15.39:0/3484331703 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4004101dc0 0x7f4004102240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:32.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:32.960+0000 7f400a200700 1 --2- 172.21.15.39:0/3484331703 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4004100aa0 0x7f4004100ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:32.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:32.960+0000 7f400a200700 1 --2- 172.21.15.39:0/3484331703 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4004101400 0x7f4004101880 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:32.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:32.960+0000 7f400a200700 1 -- 172.21.15.39:0/3484331703 >> 172.21.15.39:0/3484331703 conn(0x7f40040fc6c0 msgr2=0x7f40040feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:32.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:32.960+0000 7f400a200700 1 -- 172.21.15.39:0/3484331703 shutdown_connections 2024-03-18T16:09:32.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:32.960+0000 7f400a200700 1 -- 172.21.15.39:0/3484331703 wait complete. 2024-03-18T16:09:32.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:32.961+0000 7f400a200700 1 Processor -- start 2024-03-18T16:09:32.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:32.961+0000 7f400a200700 1 -- start start 2024-03-18T16:09:32.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:32.962+0000 7f400a200700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4004100aa0 0x7f400419a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:32.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:32.962+0000 7f400a200700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4004101400 0x7f400419ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:32.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:32.962+0000 7f400a200700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4004101dc0 0x7f40041a1be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:32.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:32.962+0000 7f400a200700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f4004110040 con 0x7f4004101400 2024-03-18T16:09:32.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:32.962+0000 7f400a200700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f400410fec0 con 0x7f4004101dc0 2024-03-18T16:09:32.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:32.962+0000 7f400a200700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f40041101c0 con 0x7f4004100aa0 2024-03-18T16:09:32.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:32.962+0000 7f4008e00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4004100aa0 0x7f400419a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:32.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:32.962+0000 7f4008e00700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4004100aa0 0x7f400419a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:60418/0 (socket says 172.21.15.39:60418) 2024-03-18T16:09:32.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:32.962+0000 7f4008e00700 1 -- 172.21.15.39:0/1745620435 learned_addr learned my addr 172.21.15.39:0/1745620435 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:32.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:32.962+0000 7f4003e00700 1 --2- 172.21.15.39:0/1745620435 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4004101400 0x7f400419ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:32.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:32.962+0000 7f4009800700 1 --2- 172.21.15.39:0/1745620435 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4004101dc0 0x7f40041a1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:32.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:32.963+0000 7f4008e00700 1 -- 172.21.15.39:0/1745620435 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4004101dc0 msgr2=0x7f40041a1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:32.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:32.963+0000 7f4008e00700 1 --2- 172.21.15.39:0/1745620435 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4004101dc0 0x7f40041a1be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:32.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:32.963+0000 7f4008e00700 1 -- 172.21.15.39:0/1745620435 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4004101400 msgr2=0x7f400419ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:32.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:32.963+0000 7f4008e00700 1 --2- 172.21.15.39:0/1745620435 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4004101400 0x7f400419ab60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:32.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:32.963+0000 7f4008e00700 1 -- 172.21.15.39:0/1745620435 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40041a2280 con 0x7f4004100aa0 2024-03-18T16:09:32.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:32.963+0000 7f4003e00700 1 --2- 172.21.15.39:0/1745620435 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4004101400 0x7f400419ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:09:32.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:32.963+0000 7f4008e00700 1 --2- 172.21.15.39:0/1745620435 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4004100aa0 0x7f400419a620 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f3ff800e9f0 tx=0x7f3ff800ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:32.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:32.963+0000 7f4001600700 1 -- 172.21.15.39:0/1745620435 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ff800cbd0 con 0x7f4004100aa0 2024-03-18T16:09:32.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:32.963+0000 7f4001600700 1 -- 172.21.15.39:0/1745620435 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ff800cd70 con 0x7f4004100aa0 2024-03-18T16:09:32.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:32.963+0000 7f4001600700 1 -- 172.21.15.39:0/1745620435 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ff8010430 con 0x7f4004100aa0 2024-03-18T16:09:32.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:32.963+0000 7f400a200700 1 -- 172.21.15.39:0/1745620435 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40041a2570 con 0x7f4004100aa0 2024-03-18T16:09:32.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:32.963+0000 7f400a200700 1 -- 172.21.15.39:0/1745620435 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40041a2a60 con 0x7f4004100aa0 2024-03-18T16:09:32.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:32.965+0000 7f400a200700 1 -- 172.21.15.39:0/1745620435 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4004068f90 con 0x7f4004100aa0 2024-03-18T16:09:32.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:32.966+0000 7f4001600700 1 -- 172.21.15.39:0/1745620435 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f3ff8004750 con 0x7f4004100aa0 2024-03-18T16:09:32.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:32.968+0000 7f4001600700 1 --2- 172.21.15.39:0/1745620435 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3fec0a2890 0x7f3fec0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:32.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:32.968+0000 7f4001600700 1 -- 172.21.15.39:0/1745620435 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f3ff80cf100 con 0x7f4004100aa0 2024-03-18T16:09:32.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:32.968+0000 7f4003e00700 1 --2- 172.21.15.39:0/1745620435 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3fec0a2890 0x7f3fec0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:32.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:32.969+0000 7f4003e00700 1 --2- 172.21.15.39:0/1745620435 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3fec0a2890 0x7f3fec0a4d50 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f3ff000b2e0 tx=0x7f3ff001a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:32.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:32.972+0000 7f4001600700 1 -- 172.21.15.39:0/1745620435 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3ff809bdb0 con 0x7f4004100aa0 2024-03-18T16:09:33.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:33.251+0000 7f400a200700 1 -- 172.21.15.39:0/1745620435 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7f4004062ee0 con 0x7f4004100aa0 2024-03-18T16:09:33.253 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:33 smithi039 ceph-mon[31721]: pgmap v406: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-18T16:09:33.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:33.251+0000 7f4001600700 1 -- 172.21.15.39:0/1745620435 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 27 v27) ==== 76+0+3092 (secure 0 0 0) 0x7f3ff80bb030 con 0x7f4004100aa0 2024-03-18T16:09:33.255 INFO:teuthology.orchestra.run.smithi039.stdout:e27 2024-03-18T16:09:33.255 INFO:teuthology.orchestra.run.smithi039.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-18T16:09:33.255 INFO:teuthology.orchestra.run.smithi039.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-18T16:09:33.255 INFO:teuthology.orchestra.run.smithi039.stdout:legacy client fscid: 1 2024-03-18T16:09:33.255 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:09:33.255 INFO:teuthology.orchestra.run.smithi039.stdout:Filesystem 'cephfs' (1) 2024-03-18T16:09:33.255 INFO:teuthology.orchestra.run.smithi039.stdout:fs_name cephfs 2024-03-18T16:09:33.255 INFO:teuthology.orchestra.run.smithi039.stdout:epoch 27 2024-03-18T16:09:33.255 INFO:teuthology.orchestra.run.smithi039.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-18T16:09:33.255 INFO:teuthology.orchestra.run.smithi039.stdout:created 2024-03-18T16:06:44.180425+0000 2024-03-18T16:09:33.255 INFO:teuthology.orchestra.run.smithi039.stdout:modified 2024-03-18T16:07:20.994405+0000 2024-03-18T16:09:33.255 INFO:teuthology.orchestra.run.smithi039.stdout:tableserver 0 2024-03-18T16:09:33.256 INFO:teuthology.orchestra.run.smithi039.stdout:root 0 2024-03-18T16:09:33.256 INFO:teuthology.orchestra.run.smithi039.stdout:session_timeout 300 2024-03-18T16:09:33.256 INFO:teuthology.orchestra.run.smithi039.stdout:session_autoclose 300 2024-03-18T16:09:33.256 INFO:teuthology.orchestra.run.smithi039.stdout:max_file_size 1099511627776 2024-03-18T16:09:33.256 INFO:teuthology.orchestra.run.smithi039.stdout:max_xattr_size 65536 2024-03-18T16:09:33.256 INFO:teuthology.orchestra.run.smithi039.stdout:required_client_features {} 2024-03-18T16:09:33.256 INFO:teuthology.orchestra.run.smithi039.stdout:last_failure 0 2024-03-18T16:09:33.256 INFO:teuthology.orchestra.run.smithi039.stdout:last_failure_osd_epoch 0 2024-03-18T16:09:33.256 INFO:teuthology.orchestra.run.smithi039.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-03-18T16:09:33.256 INFO:teuthology.orchestra.run.smithi039.stdout:max_mds 5 2024-03-18T16:09:33.256 INFO:teuthology.orchestra.run.smithi039.stdout:in 0,1,2,3,4 2024-03-18T16:09:33.256 INFO:teuthology.orchestra.run.smithi039.stdout:up {0=24575,1=24556,2=24535,3=24524,4=14463} 2024-03-18T16:09:33.256 INFO:teuthology.orchestra.run.smithi039.stdout:failed 2024-03-18T16:09:33.257 INFO:teuthology.orchestra.run.smithi039.stdout:damaged 2024-03-18T16:09:33.257 INFO:teuthology.orchestra.run.smithi039.stdout:stopped 2024-03-18T16:09:33.257 INFO:teuthology.orchestra.run.smithi039.stdout:data_pools [3,4] 2024-03-18T16:09:33.257 INFO:teuthology.orchestra.run.smithi039.stdout:metadata_pool 2 2024-03-18T16:09:33.257 INFO:teuthology.orchestra.run.smithi039.stdout:inline_data disabled 2024-03-18T16:09:33.257 INFO:teuthology.orchestra.run.smithi039.stdout:balancer 2024-03-18T16:09:33.257 INFO:teuthology.orchestra.run.smithi039.stdout:bal_rank_mask -1 2024-03-18T16:09:33.257 INFO:teuthology.orchestra.run.smithi039.stdout:standby_count_wanted 0 2024-03-18T16:09:33.257 INFO:teuthology.orchestra.run.smithi039.stdout:qdb_cluster leader: 24575 members: 24556,24535,24575,24524,14463 2024-03-18T16:09:33.257 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.b{0:24575} state up:active seq 3 addr [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T16:09:33.257 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.f{0:24523} state up:standby-replay seq 1 addr [v2:172.21.15.196:6834/3949984309,v1:172.21.15.196:6835/3949984309] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T16:09:33.257 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.i{1:24556} state up:active seq 14 addr [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T16:09:33.257 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.e{1:24494} state up:standby-replay seq 1 addr [v2:172.21.15.194:6834/188385470,v1:172.21.15.194:6835/188385470] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T16:09:33.257 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.c{2:24535} state up:active seq 16 addr [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T16:09:33.258 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.l{2:24463} state up:standby-replay seq 1 addr [v2:172.21.15.196:6832/3874209621,v1:172.21.15.196:6833/3874209621] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T16:09:33.258 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.h{3:24524} state up:active seq 15 addr [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T16:09:33.258 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.k{3:24446} state up:standby-replay seq 1 addr [v2:172.21.15.194:6832/931350738,v1:172.21.15.194:6833/931350738] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T16:09:33.258 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.a{4:14463} state up:active seq 13 addr [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T16:09:33.258 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.g{4:14445} state up:standby-replay seq 1 addr [v2:172.21.15.39:6838/656513813,v1:172.21.15.39:6839/656513813] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T16:09:33.258 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:09:33.258 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:09:33.258 INFO:teuthology.orchestra.run.smithi039.stdout:Standby daemons: 2024-03-18T16:09:33.258 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:09:33.258 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.j{-1:14379} state up:standby seq 1 addr [v2:172.21.15.39:6834/2520226608,v1:172.21.15.39:6835/2520226608] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T16:09:33.258 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.d{-1:14391} state up:standby seq 1 addr [v2:172.21.15.39:6836/719341763,v1:172.21.15.39:6837/719341763] compat {c=[1],r=[1],i=[1fff]}] 2024-03-18T16:09:33.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:33.255+0000 7f400a200700 1 -- 172.21.15.39:0/1745620435 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3fec0a2890 msgr2=0x7f3fec0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:33.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:33.255+0000 7f400a200700 1 --2- 172.21.15.39:0/1745620435 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3fec0a2890 0x7f3fec0a4d50 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f3ff000b2e0 tx=0x7f3ff001a040 comp rx=0 tx=0).stop 2024-03-18T16:09:33.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:33.255+0000 7f400a200700 1 -- 172.21.15.39:0/1745620435 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4004100aa0 msgr2=0x7f400419a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:33.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:33.255+0000 7f400a200700 1 --2- 172.21.15.39:0/1745620435 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4004100aa0 0x7f400419a620 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f3ff800e9f0 tx=0x7f3ff800ed00 comp rx=0 tx=0).stop 2024-03-18T16:09:33.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:33.255+0000 7f400a200700 1 -- 172.21.15.39:0/1745620435 shutdown_connections 2024-03-18T16:09:33.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:33.255+0000 7f400a200700 1 --2- 172.21.15.39:0/1745620435 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3fec0a2890 0x7f3fec0a4d50 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:33.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:33.255+0000 7f400a200700 1 --2- 172.21.15.39:0/1745620435 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4004101dc0 0x7f40041a1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:33.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:33.255+0000 7f400a200700 1 --2- 172.21.15.39:0/1745620435 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4004100aa0 0x7f400419a620 secure :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f3ff800e9f0 tx=0x7f3ff800ed00 comp rx=0 tx=0).stop 2024-03-18T16:09:33.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:33.255+0000 7f400a200700 1 --2- 172.21.15.39:0/1745620435 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4004101400 0x7f400419ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:33.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:33.255+0000 7f400a200700 1 -- 172.21.15.39:0/1745620435 >> 172.21.15.39:0/1745620435 conn(0x7f40040fc6c0 msgr2=0x7f40040ffa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:33.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:33.255+0000 7f400a200700 1 -- 172.21.15.39:0/1745620435 shutdown_connections 2024-03-18T16:09:33.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:33.256+0000 7f400a200700 1 -- 172.21.15.39:0/1745620435 wait complete. 2024-03-18T16:09:33.260 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 27 2024-03-18T16:09:33.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:33 smithi196 ceph-mon[36307]: pgmap v406: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-18T16:09:33.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:33 smithi194 ceph-mon[36200]: pgmap v406: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-18T16:09:34.245 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- bash -c 'ceph osd dump' 2024-03-18T16:09:34.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:34 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/1745620435' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-18T16:09:34.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:34 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/1745620435' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-18T16:09:34.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:34 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/1745620435' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-18T16:09:34.735 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T16:09:35.217 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:35 smithi039 ceph-mon[31721]: pgmap v407: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T16:09:35.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:35 smithi196 ceph-mon[36307]: pgmap v407: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T16:09:35.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:35 smithi194 ceph-mon[36200]: pgmap v407: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T16:09:37.192 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:37 smithi039 ceph-mon[31721]: pgmap v408: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T16:09:37.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:37.252+0000 7fd7c3200700 1 -- 172.21.15.39:0/3392553679 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd7c4105660 msgr2=0x7fd7c4107a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:37.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:37.252+0000 7fd7c3200700 1 --2- 172.21.15.39:0/3392553679 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd7c4105660 0x7fd7c4107a50 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd7b0009990 tx=0x7fd7b0009ca0 comp rx=0 tx=0).stop 2024-03-18T16:09:37.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:37.252+0000 7fd7c3200700 1 -- 172.21.15.39:0/3392553679 shutdown_connections 2024-03-18T16:09:37.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:37.252+0000 7fd7c3200700 1 --2- 172.21.15.39:0/3392553679 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd7c4107f90 0x7fd7c410a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:37.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:37.252+0000 7fd7c3200700 1 --2- 172.21.15.39:0/3392553679 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd7c4102d30 0x7fd7c4105120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:37.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:37.252+0000 7fd7c3200700 1 --2- 172.21.15.39:0/3392553679 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd7c4105660 0x7fd7c4107a50 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:37.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:37.252+0000 7fd7c3200700 1 -- 172.21.15.39:0/3392553679 >> 172.21.15.39:0/3392553679 conn(0x7fd7c40fc6c0 msgr2=0x7fd7c40feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:37.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:37.252+0000 7fd7c3200700 1 -- 172.21.15.39:0/3392553679 shutdown_connections 2024-03-18T16:09:37.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:37.252+0000 7fd7c3200700 1 -- 172.21.15.39:0/3392553679 wait complete. 2024-03-18T16:09:37.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:37.254+0000 7fd7c3200700 1 Processor -- start 2024-03-18T16:09:37.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:37.254+0000 7fd7c3200700 1 -- start start 2024-03-18T16:09:37.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:37.255+0000 7fd7c3200700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd7c4102d30 0x7fd7c419a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:37.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:37.255+0000 7fd7c3200700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd7c4105660 0x7fd7c419abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:37.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:37.255+0000 7fd7c3200700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd7c4107f90 0x7fd7c41a1c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:37.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:37.255+0000 7fd7c3200700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fd7c410de50 con 0x7fd7c4105660 2024-03-18T16:09:37.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:37.255+0000 7fd7c3200700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fd7c410dcd0 con 0x7fd7c4102d30 2024-03-18T16:09:37.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:37.255+0000 7fd7c3200700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fd7c410dfd0 con 0x7fd7c4107f90 2024-03-18T16:09:37.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:37.255+0000 7fd7c1e00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd7c4102d30 0x7fd7c419a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:37.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:37.255+0000 7fd7c1e00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd7c4102d30 0x7fd7c419a690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:43422/0 (socket says 172.21.15.39:43422) 2024-03-18T16:09:37.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:37.255+0000 7fd7c1e00700 1 -- 172.21.15.39:0/4025629484 learned_addr learned my addr 172.21.15.39:0/4025629484 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:37.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:37.255+0000 7fd7c2800700 1 --2- 172.21.15.39:0/4025629484 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd7c4107f90 0x7fd7c41a1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:37.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:37.255+0000 7fd7c1400700 1 --2- 172.21.15.39:0/4025629484 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd7c4105660 0x7fd7c419abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:37.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:37.255+0000 7fd7c1e00700 1 -- 172.21.15.39:0/4025629484 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd7c4107f90 msgr2=0x7fd7c41a1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:37.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:37.255+0000 7fd7c1e00700 1 --2- 172.21.15.39:0/4025629484 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd7c4107f90 0x7fd7c41a1c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:37.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:37.255+0000 7fd7c1e00700 1 -- 172.21.15.39:0/4025629484 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd7c4105660 msgr2=0x7fd7c419abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:37.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:37.255+0000 7fd7c1e00700 1 --2- 172.21.15.39:0/4025629484 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd7c4105660 0x7fd7c419abd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:37.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:37.255+0000 7fd7c1e00700 1 -- 172.21.15.39:0/4025629484 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7c41a22f0 con 0x7fd7c4102d30 2024-03-18T16:09:37.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:37.255+0000 7fd7c1400700 1 --2- 172.21.15.39:0/4025629484 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd7c4105660 0x7fd7c419abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:09:37.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:37.255+0000 7fd7c2800700 1 --2- 172.21.15.39:0/4025629484 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd7c4107f90 0x7fd7c41a1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:09:37.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:37.256+0000 7fd7c1e00700 1 --2- 172.21.15.39:0/4025629484 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd7c4102d30 0x7fd7c419a690 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fd7b400e9a0 tx=0x7fd7b400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:37.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:37.256+0000 7fd7baa00700 1 -- 172.21.15.39:0/4025629484 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7b400cbd0 con 0x7fd7c4102d30 2024-03-18T16:09:37.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:37.256+0000 7fd7baa00700 1 -- 172.21.15.39:0/4025629484 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd7b400cd70 con 0x7fd7c4102d30 2024-03-18T16:09:37.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:37.256+0000 7fd7baa00700 1 -- 172.21.15.39:0/4025629484 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7b40187a0 con 0x7fd7c4102d30 2024-03-18T16:09:37.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:37.256+0000 7fd7c3200700 1 -- 172.21.15.39:0/4025629484 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7c41a25e0 con 0x7fd7c4102d30 2024-03-18T16:09:37.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:37.256+0000 7fd7c3200700 1 -- 172.21.15.39:0/4025629484 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7c41a2ad0 con 0x7fd7c4102d30 2024-03-18T16:09:37.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:37.257+0000 7fd7c3200700 1 -- 172.21.15.39:0/4025629484 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7c4068f90 con 0x7fd7c4102d30 2024-03-18T16:09:37.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:37.259+0000 7fd7baa00700 1 -- 172.21.15.39:0/4025629484 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fd7b40109a0 con 0x7fd7c4102d30 2024-03-18T16:09:37.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:37.260+0000 7fd7baa00700 1 --2- 172.21.15.39:0/4025629484 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd7a80a2910 0x7fd7a80a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:37.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:37.261+0000 7fd7baa00700 1 -- 172.21.15.39:0/4025629484 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fd7b40ceff0 con 0x7fd7c4102d30 2024-03-18T16:09:37.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:37.261+0000 7fd7c1400700 1 --2- 172.21.15.39:0/4025629484 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd7a80a2910 0x7fd7a80a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:37.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:37.261+0000 7fd7c1400700 1 --2- 172.21.15.39:0/4025629484 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd7a80a2910 0x7fd7a80a4dd0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fd7b000b2e0 tx=0x7fd7b0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:37.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:37.264+0000 7fd7baa00700 1 -- 172.21.15.39:0/4025629484 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd7b409bca0 con 0x7fd7c4102d30 2024-03-18T16:09:37.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:37 smithi196 ceph-mon[36307]: pgmap v408: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T16:09:37.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:37 smithi194 ceph-mon[36200]: pgmap v408: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T16:09:37.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:37.495+0000 7fd7c3200700 1 -- 172.21.15.39:0/4025629484 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7fd7c4062ee0 con 0x7fd7c4102d30 2024-03-18T16:09:37.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:37.497+0000 7fd7baa00700 1 -- 172.21.15.39:0/4025629484 <== mon.2 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v93) ==== 56+0+5659 (secure 0 0 0) 0x7fd7b409b5a0 con 0x7fd7c4102d30 2024-03-18T16:09:37.500 INFO:teuthology.orchestra.run.smithi039.stdout:epoch 93 2024-03-18T16:09:37.500 INFO:teuthology.orchestra.run.smithi039.stdout:fsid 1892d8fc-e540-11ee-95c9-87774f69a715 2024-03-18T16:09:37.500 INFO:teuthology.orchestra.run.smithi039.stdout:created 2024-03-18T15:57:36.507008+0000 2024-03-18T16:09:37.500 INFO:teuthology.orchestra.run.smithi039.stdout:modified 2024-03-18T16:08:47.874967+0000 2024-03-18T16:09:37.500 INFO:teuthology.orchestra.run.smithi039.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-03-18T16:09:37.500 INFO:teuthology.orchestra.run.smithi039.stdout:crush_version 27 2024-03-18T16:09:37.500 INFO:teuthology.orchestra.run.smithi039.stdout:full_ratio 0.95 2024-03-18T16:09:37.500 INFO:teuthology.orchestra.run.smithi039.stdout:backfillfull_ratio 0.9 2024-03-18T16:09:37.500 INFO:teuthology.orchestra.run.smithi039.stdout:nearfull_ratio 0.85 2024-03-18T16:09:37.501 INFO:teuthology.orchestra.run.smithi039.stdout:require_min_compat_client luminous 2024-03-18T16:09:37.501 INFO:teuthology.orchestra.run.smithi039.stdout:min_compat_client luminous 2024-03-18T16:09:37.501 INFO:teuthology.orchestra.run.smithi039.stdout:require_osd_release squid 2024-03-18T16:09:37.501 INFO:teuthology.orchestra.run.smithi039.stdout:stretch_mode_enabled false 2024-03-18T16:09:37.501 INFO:teuthology.orchestra.run.smithi039.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-03-18T16:09:37.501 INFO:teuthology.orchestra.run.smithi039.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 84 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-03-18T16:09:37.501 INFO:teuthology.orchestra.run.smithi039.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2024-03-18T16:09:37.501 INFO:teuthology.orchestra.run.smithi039.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 89 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-03-18T16:09:37.501 INFO:teuthology.orchestra.run.smithi039.stdout:max_osd 12 2024-03-18T16:09:37.501 INFO:teuthology.orchestra.run.smithi039.stdout:osd.0 up in weight 1 up_from 8 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.39:6802/4090743635,v1:172.21.15.39:6803/4090743635] [v2:172.21.15.39:6804/4090743635,v1:172.21.15.39:6805/4090743635] exists,up 771e0558-be62-4436-94b0-6364751911c4 2024-03-18T16:09:37.501 INFO:teuthology.orchestra.run.smithi039.stdout:osd.1 up in weight 1 up_from 13 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.194:6800/2800329805,v1:172.21.15.194:6801/2800329805] [v2:172.21.15.194:6802/2800329805,v1:172.21.15.194:6803/2800329805] exists,up 0430eee1-cf34-4277-b255-c7d9f07ad19a 2024-03-18T16:09:37.501 INFO:teuthology.orchestra.run.smithi039.stdout:osd.2 up in weight 1 up_from 18 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.196:6800/3935874719,v1:172.21.15.196:6801/3935874719] [v2:172.21.15.196:6802/3935874719,v1:172.21.15.196:6803/3935874719] exists,up 3a718da4-2511-4ec1-b1a8-eafd6cce0df0 2024-03-18T16:09:37.501 INFO:teuthology.orchestra.run.smithi039.stdout:osd.3 up in weight 1 up_from 25 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.39:6810/893345939,v1:172.21.15.39:6811/893345939] [v2:172.21.15.39:6812/893345939,v1:172.21.15.39:6813/893345939] exists,up 6f9c37f9-0ab9-4038-8919-afd4b39c4165 2024-03-18T16:09:37.501 INFO:teuthology.orchestra.run.smithi039.stdout:osd.4 up in weight 1 up_from 33 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.194:6808/800423088,v1:172.21.15.194:6809/800423088] [v2:172.21.15.194:6810/800423088,v1:172.21.15.194:6811/800423088] exists,up e795bb0e-4791-4537-acda-1a321e19bc9c 2024-03-18T16:09:37.502 INFO:teuthology.orchestra.run.smithi039.stdout:osd.5 up in weight 1 up_from 38 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.196:6808/2746224525,v1:172.21.15.196:6809/2746224525] [v2:172.21.15.196:6810/2746224525,v1:172.21.15.196:6811/2746224525] exists,up 026e175e-6acb-4b4c-b235-8ccee8ae1149 2024-03-18T16:09:37.502 INFO:teuthology.orchestra.run.smithi039.stdout:osd.6 up in weight 1 up_from 46 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.39:6818/1213371331,v1:172.21.15.39:6819/1213371331] [v2:172.21.15.39:6820/1213371331,v1:172.21.15.39:6821/1213371331] exists,up c1b183ff-383f-4c5c-9075-f76bab1c0bb3 2024-03-18T16:09:37.502 INFO:teuthology.orchestra.run.smithi039.stdout:osd.7 up in weight 1 up_from 51 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.194:6816/3299162444,v1:172.21.15.194:6817/3299162444] [v2:172.21.15.194:6818/3299162444,v1:172.21.15.194:6819/3299162444] exists,up 612f8697-957d-4ef4-a87f-241b36f9d13d 2024-03-18T16:09:37.502 INFO:teuthology.orchestra.run.smithi039.stdout:osd.8 up in weight 1 up_from 59 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.196:6816/2456143145,v1:172.21.15.196:6817/2456143145] [v2:172.21.15.196:6818/2456143145,v1:172.21.15.196:6819/2456143145] exists,up 836a9107-4d4f-4594-b8dc-0301e64f59d9 2024-03-18T16:09:37.502 INFO:teuthology.orchestra.run.smithi039.stdout:osd.9 up in weight 1 up_from 67 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.39:6826/1848634433,v1:172.21.15.39:6827/1848634433] [v2:172.21.15.39:6828/1848634433,v1:172.21.15.39:6829/1848634433] exists,up 4f30b960-12e3-4b16-bd28-fcc988139b58 2024-03-18T16:09:37.502 INFO:teuthology.orchestra.run.smithi039.stdout:osd.10 up in weight 1 up_from 72 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.194:6824/3725064809,v1:172.21.15.194:6825/3725064809] [v2:172.21.15.194:6826/3725064809,v1:172.21.15.194:6827/3725064809] exists,up 620f90e8-23e6-4bc0-8269-7219b9e1aebd 2024-03-18T16:09:37.502 INFO:teuthology.orchestra.run.smithi039.stdout:osd.11 up in weight 1 up_from 80 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.196:6824/2501212422,v1:172.21.15.196:6825/2501212422] [v2:172.21.15.196:6826/2501212422,v1:172.21.15.196:6827/2501212422] exists,up 867a2535-8235-4ceb-a698-eb86869d6d20 2024-03-18T16:09:37.502 INFO:teuthology.orchestra.run.smithi039.stdout:pg_upmap_items 2.f [8,11] 2024-03-18T16:09:37.502 INFO:teuthology.orchestra.run.smithi039.stdout:pg_upmap_items 2.13 [2,11] 2024-03-18T16:09:37.502 INFO:teuthology.orchestra.run.smithi039.stdout:pg_upmap_items 3.c [3,9] 2024-03-18T16:09:37.502 INFO:teuthology.orchestra.run.smithi039.stdout:pg_upmap_items 3.11 [4,8] 2024-03-18T16:09:37.502 INFO:teuthology.orchestra.run.smithi039.stdout:pg_upmap_items 3.1d [4,7] 2024-03-18T16:09:37.502 INFO:teuthology.orchestra.run.smithi039.stdout:pg_upmap_items 3.20 [5,9] 2024-03-18T16:09:37.503 INFO:teuthology.orchestra.run.smithi039.stdout:pg_upmap_items 3.31 [4,6] 2024-03-18T16:09:37.503 INFO:teuthology.orchestra.run.smithi039.stdout:pg_upmap_items 3.38 [3,9] 2024-03-18T16:09:37.503 INFO:teuthology.orchestra.run.smithi039.stdout:pg_upmap_items 4.5 [9,10] 2024-03-18T16:09:37.503 INFO:teuthology.orchestra.run.smithi039.stdout:pg_upmap_items 4.a [5,8] 2024-03-18T16:09:37.503 INFO:teuthology.orchestra.run.smithi039.stdout:pg_upmap_items 4.f [9,8] 2024-03-18T16:09:37.503 INFO:teuthology.orchestra.run.smithi039.stdout:pg_upmap_items 4.13 [9,8] 2024-03-18T16:09:37.503 INFO:teuthology.orchestra.run.smithi039.stdout:blocklist 172.21.15.39:0/3482086140 expires 2024-03-19T15:58:45.340294+0000 2024-03-18T16:09:37.503 INFO:teuthology.orchestra.run.smithi039.stdout:blocklist 172.21.15.39:0/3844306284 expires 2024-03-19T15:58:45.340294+0000 2024-03-18T16:09:37.503 INFO:teuthology.orchestra.run.smithi039.stdout:blocklist 172.21.15.39:6801/1810238546 expires 2024-03-19T15:58:45.340294+0000 2024-03-18T16:09:37.503 INFO:teuthology.orchestra.run.smithi039.stdout:blocklist 172.21.15.39:6800/1810238546 expires 2024-03-19T15:58:45.340294+0000 2024-03-18T16:09:37.503 INFO:teuthology.orchestra.run.smithi039.stdout:blocklist 172.21.15.39:0/2528397488 expires 2024-03-19T15:58:10.264798+0000 2024-03-18T16:09:37.503 INFO:teuthology.orchestra.run.smithi039.stdout:blocklist 172.21.15.39:0/2240981254 expires 2024-03-19T15:58:10.264798+0000 2024-03-18T16:09:37.503 INFO:teuthology.orchestra.run.smithi039.stdout:blocklist 172.21.15.39:6800/1217503189 expires 2024-03-19T15:58:10.264798+0000 2024-03-18T16:09:37.503 INFO:teuthology.orchestra.run.smithi039.stdout:blocklist 172.21.15.39:0/3754876370 expires 2024-03-19T15:58:10.264798+0000 2024-03-18T16:09:37.503 INFO:teuthology.orchestra.run.smithi039.stdout:blocklist 172.21.15.39:6801/1217503189 expires 2024-03-19T15:58:10.264798+0000 2024-03-18T16:09:37.503 INFO:teuthology.orchestra.run.smithi039.stdout:blocklist 172.21.15.39:0/1874637628 expires 2024-03-19T15:58:45.340294+0000 2024-03-18T16:09:37.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:37.500+0000 7fd7c3200700 1 -- 172.21.15.39:0/4025629484 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd7a80a2910 msgr2=0x7fd7a80a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:37.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:37.500+0000 7fd7c3200700 1 --2- 172.21.15.39:0/4025629484 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd7a80a2910 0x7fd7a80a4dd0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fd7b000b2e0 tx=0x7fd7b0005fb0 comp rx=0 tx=0).stop 2024-03-18T16:09:37.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:37.500+0000 7fd7c3200700 1 -- 172.21.15.39:0/4025629484 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd7c4102d30 msgr2=0x7fd7c419a690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:37.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:37.500+0000 7fd7c3200700 1 --2- 172.21.15.39:0/4025629484 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd7c4102d30 0x7fd7c419a690 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fd7b400e9a0 tx=0x7fd7b400ecb0 comp rx=0 tx=0).stop 2024-03-18T16:09:37.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:37.500+0000 7fd7c3200700 1 -- 172.21.15.39:0/4025629484 shutdown_connections 2024-03-18T16:09:37.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:37.500+0000 7fd7c3200700 1 --2- 172.21.15.39:0/4025629484 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd7a80a2910 0x7fd7a80a4dd0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:37.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:37.500+0000 7fd7c3200700 1 --2- 172.21.15.39:0/4025629484 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd7c4107f90 0x7fd7c41a1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:37.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:37.500+0000 7fd7c3200700 1 --2- 172.21.15.39:0/4025629484 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd7c4102d30 0x7fd7c419a690 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:37.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:37.500+0000 7fd7c3200700 1 --2- 172.21.15.39:0/4025629484 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd7c4105660 0x7fd7c419abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:37.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:37.500+0000 7fd7c3200700 1 -- 172.21.15.39:0/4025629484 >> 172.21.15.39:0/4025629484 conn(0x7fd7c40fc6c0 msgr2=0x7fd7c4102510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:37.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:37.500+0000 7fd7c3200700 1 -- 172.21.15.39:0/4025629484 shutdown_connections 2024-03-18T16:09:37.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:37.500+0000 7fd7c3200700 1 -- 172.21.15.39:0/4025629484 wait complete. 2024-03-18T16:09:38.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:38 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/4025629484' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-18T16:09:38.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:38 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/4025629484' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-18T16:09:38.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:38 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/4025629484' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-18T16:09:38.697 INFO:teuthology.run_tasks:Running task fs.ready... 2024-03-18T16:09:38.708 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T16:09:38.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:38.917+0000 7fab62800640 1 -- 172.21.15.39:0/1912919934 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fab5c074240 msgr2=0x7fab5c0746c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:38.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:38.917+0000 7fab62800640 1 --2- 172.21.15.39:0/1912919934 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fab5c074240 0x7fab5c0746c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fab48009900 tx=0x7fab4802bc40 comp rx=0 tx=0).stop 2024-03-18T16:09:38.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:38.917+0000 7fab62800640 1 -- 172.21.15.39:0/1912919934 shutdown_connections 2024-03-18T16:09:38.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:38.917+0000 7fab62800640 1 --2- 172.21.15.39:0/1912919934 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fab5c10dbb0 0x7fab5c10ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:38.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:38.917+0000 7fab62800640 1 --2- 172.21.15.39:0/1912919934 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fab5c074240 0x7fab5c0746c0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:38.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:38.917+0000 7fab62800640 1 --2- 172.21.15.39:0/1912919934 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fab5c075820 0x7fab5c073c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:38.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:38.917+0000 7fab62800640 1 -- 172.21.15.39:0/1912919934 >> 172.21.15.39:0/1912919934 conn(0x7fab5c0fcb70 msgr2=0x7fab5c0fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:38.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:38.917+0000 7fab62800640 1 -- 172.21.15.39:0/1912919934 shutdown_connections 2024-03-18T16:09:38.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:38.917+0000 7fab62800640 1 -- 172.21.15.39:0/1912919934 wait complete. 2024-03-18T16:09:38.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:38.918+0000 7fab62800640 1 Processor -- start 2024-03-18T16:09:38.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:38.918+0000 7fab62800640 1 -- start start 2024-03-18T16:09:38.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:38.918+0000 7fab62800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fab5c074240 0x7fab5c19def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:38.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:38.918+0000 7fab62800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fab5c075820 0x7fab5c19e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:38.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:38.918+0000 7fab62800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fab5c10dbb0 0x7fab5c1a54a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:38.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:38.918+0000 7fab62800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fab5c113830 con 0x7fab5c075820 2024-03-18T16:09:38.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:38.918+0000 7fab62800640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fab5c1136b0 con 0x7fab5c10dbb0 2024-03-18T16:09:38.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:38.918+0000 7fab62800640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fab5c1139b0 con 0x7fab5c074240 2024-03-18T16:09:38.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:38.919+0000 7fab61400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fab5c074240 0x7fab5c19def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:38.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:38.919+0000 7fab61400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fab5c074240 0x7fab5c19def0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:55862/0 (socket says 172.21.15.39:55862) 2024-03-18T16:09:38.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:38.919+0000 7fab60a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fab5c075820 0x7fab5c19e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:38.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:38.919+0000 7fab61400640 1 -- 172.21.15.39:0/1662441066 learned_addr learned my addr 172.21.15.39:0/1662441066 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:38.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:38.919+0000 7fab61e00640 1 --2- 172.21.15.39:0/1662441066 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fab5c10dbb0 0x7fab5c1a54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:38.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:38.919+0000 7fab61400640 1 -- 172.21.15.39:0/1662441066 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fab5c10dbb0 msgr2=0x7fab5c1a54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:38.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:38.919+0000 7fab61400640 1 --2- 172.21.15.39:0/1662441066 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fab5c10dbb0 0x7fab5c1a54a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:38.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:38.919+0000 7fab61400640 1 -- 172.21.15.39:0/1662441066 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fab5c075820 msgr2=0x7fab5c19e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:38.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:38.919+0000 7fab61400640 1 --2- 172.21.15.39:0/1662441066 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fab5c075820 0x7fab5c19e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:38.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:38.919+0000 7fab61400640 1 -- 172.21.15.39:0/1662441066 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab5c1a5b10 con 0x7fab5c074240 2024-03-18T16:09:38.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:38.919+0000 7fab61400640 1 --2- 172.21.15.39:0/1662441066 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fab5c074240 0x7fab5c19def0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fab5000e9f0 tx=0x7fab5000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:38.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:38.920+0000 7fab5a000640 1 -- 172.21.15.39:0/1662441066 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab5000cd80 con 0x7fab5c074240 2024-03-18T16:09:38.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:38.920+0000 7fab5a000640 1 -- 172.21.15.39:0/1662441066 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab50010910 con 0x7fab5c074240 2024-03-18T16:09:38.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:38.920+0000 7fab62800640 1 -- 172.21.15.39:0/1662441066 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab5c1a5e00 con 0x7fab5c074240 2024-03-18T16:09:38.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:38.920+0000 7fab5a000640 1 -- 172.21.15.39:0/1662441066 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab50018990 con 0x7fab5c074240 2024-03-18T16:09:38.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:38.920+0000 7fab62800640 1 -- 172.21.15.39:0/1662441066 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab5c1a6260 con 0x7fab5c074240 2024-03-18T16:09:38.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:38.921+0000 7fab62800640 1 -- 172.21.15.39:0/1662441066 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab5c103d80 con 0x7fab5c074240 2024-03-18T16:09:38.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:38.923+0000 7fab5a000640 1 -- 172.21.15.39:0/1662441066 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fab50010ac0 con 0x7fab5c074240 2024-03-18T16:09:38.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:38.924+0000 7fab5a000640 1 --2- 172.21.15.39:0/1662441066 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fab440a2800 0x7fab440a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:38.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:38.924+0000 7fab5a000640 1 -- 172.21.15.39:0/1662441066 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fab500cf180 con 0x7fab5c074240 2024-03-18T16:09:38.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:38.924+0000 7fab60a00640 1 --2- 172.21.15.39:0/1662441066 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fab440a2800 0x7fab440a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:38.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:38.925+0000 7fab60a00640 1 --2- 172.21.15.39:0/1662441066 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fab440a2800 0x7fab440a4cc0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fab48006010 tx=0x7fab48000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:38.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:38.928+0000 7fab5a000640 1 -- 172.21.15.39:0/1662441066 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fab5009b7e0 con 0x7fab5c074240 2024-03-18T16:09:39.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.133+0000 7fab62800640 1 -- 172.21.15.39:0/1662441066 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fab5c02d380 con 0x7fab5c074240 2024-03-18T16:09:39.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.135+0000 7fab5a000640 1 -- 172.21.15.39:0/1662441066 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24762 (secure 0 0 0) 0x7fab5009f880 con 0x7fab5c074240 2024-03-18T16:09:39.138 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:09:39.138 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":93,"fsid":"1892d8fc-e540-11ee-95c9-87774f69a715","created":"2024-03-18T15:57:36.507008+0000","modified":"2024-03-18T16:08:47.874967+0000","last_up_change":"2024-03-18T16:05:45.250629+0000","last_in_change":"2024-03-18T16:05:24.841337+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T16:01:30.048247+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T16:06:42.191791+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T16:06:43.670325+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T16:06:47.523388+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"771e0558-be62-4436-94b0-6364751911c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6803","nonce":4090743635}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6805","nonce":4090743635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6809","nonce":4090743635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6807","nonce":4090743635}]},"public_addr":"172.21.15.39:6803/4090743635","cluster_addr":"172.21.15.39:6805/4090743635","heartbeat_back_addr":"172.21.15.39:6809/4090743635","heartbeat_front_addr":"172.21.15.39:6807/4090743635","state":["exists","up"]},{"osd":1,"uuid":"0430eee1-cf34-4277-b255-c7d9f07ad19a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6801","nonce":2800329805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6803","nonce":2800329805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6807","nonce":2800329805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6805","nonce":2800329805}]},"public_addr":"172.21.15.194:6801/2800329805","cluster_addr":"172.21.15.194:6803/2800329805","heartbeat_back_addr":"172.21.15.194:6807/2800329805","heartbeat_front_addr":"172.21.15.194:6805/2800329805","state":["exists","up"]},{"osd":2,"uuid":"3a718da4-2511-4ec1-b1a8-eafd6cce0df0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6801","nonce":3935874719}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6803","nonce":3935874719}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6807","nonce":3935874719}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6805","nonce":3935874719}]},"public_addr":"172.21.15.196:6801/3935874719","cluster_addr":"172.21.15.196:6803/3935874719","heartbeat_back_addr":"172.21.15.196:6807/3935874719","heartbeat_front_addr":"172.21.15.196:6805/3935874719","state":["exists","up"]},{"osd":3,"uuid":"6f9c37f9-0ab9-4038-8919-afd4b39c4165","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6811","nonce":893345939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6813","nonce":893345939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6817","nonce":893345939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6815","nonce":893345939}]},"public_addr":"172.21.15.39:6811/893345939","cluster_addr":"172.21.15.39:6813/893345939","heartbeat_back_addr":"172.21.15.39:6817/893345939","heartbeat_front_addr":"172.21.15.39:6815/893345939","state":["exists","up"]},{"osd":4,"uuid":"e795bb0e-4791-4537-acda-1a321e19bc9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6809","nonce":800423088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6811","nonce":800423088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6815","nonce":800423088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6813","nonce":800423088}]},"public_addr":"172.21.15.194:6809/800423088","cluster_addr":"172.21.15.194:6811/800423088","heartbeat_back_addr":"172.21.15.194:6815/800423088","heartbeat_front_addr":"172.21.15.194:6813/800423088","state":["exists","up"]},{"osd":5,"uuid":"026e175e-6acb-4b4c-b235-8ccee8ae1149","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6809","nonce":2746224525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6811","nonce":2746224525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6815","nonce":2746224525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6813","nonce":2746224525}]},"public_addr":"172.21.15.196:6809/2746224525","cluster_addr":"172.21.15.196:6811/2746224525","heartbeat_back_addr":"172.21.15.196:6815/2746224525","heartbeat_front_addr":"172.21.15.196:6813/2746224525","state":["exists","up"]},{"osd":6,"uuid":"c1b183ff-383f-4c5c-9075-f76bab1c0bb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6819","nonce":1213371331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6821","nonce":1213371331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6825","nonce":1213371331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6823","nonce":1213371331}]},"public_addr":"172.21.15.39:6819/1213371331","cluster_addr":"172.21.15.39:6821/1213371331","heartbeat_back_addr":"172.21.15.39:6825/1213371331","heartbeat_front_addr":"172.21.15.39:6823/1213371331","state":["exists","up"]},{"osd":7,"uuid":"612f8697-957d-4ef4-a87f-241b36f9d13d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6817","nonce":3299162444}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6819","nonce":3299162444}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6823","nonce":3299162444}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6821","nonce":3299162444}]},"public_addr":"172.21.15.194:6817/3299162444","cluster_addr":"172.21.15.194:6819/3299162444","heartbeat_back_addr":"172.21.15.194:6823/3299162444","heartbeat_front_addr":"172.21.15.194:6821/3299162444","state":["exists","up"]},{"osd":8,"uuid":"836a9107-4d4f-4594-b8dc-0301e64f59d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6817","nonce":2456143145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6819","nonce":2456143145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6823","nonce":2456143145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6821","nonce":2456143145}]},"public_addr":"172.21.15.196:6817/2456143145","cluster_addr":"172.21.15.196:6819/2456143145","heartbeat_back_addr":"172.21.15.196:6823/2456143145","heartbeat_front_addr":"172.21.15.196:6821/2456143145","state":["exists","up"]},{"osd":9,"uuid":"4f30b960-12e3-4b16-bd28-fcc988139b58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6827","nonce":1848634433}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6829","nonce":1848634433}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6833","nonce":1848634433}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6831","nonce":1848634433}]},"public_addr":"172.21.15.39:6827/1848634433","cluster_addr":"172.21.15.39:6829/1848634433","heartbeat_back_addr":"172.21.15.39:6833/1848634433","heartbeat_front_addr":"172.21.15.39:6831/1848634433","state":["exists","up"]},{"osd":10,"uuid":"620f90e8-23e6-4bc0-8269-7219b9e1aebd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6825","nonce":3725064809}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6827","nonce":3725064809}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6831","nonce":3725064809}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6829","nonce":3725064809}]},"public_addr":"172.21.15.194:6825/3725064809","cluster_addr":"172.21.15.194:6827/3725064809","heartbeat_back_addr":"172.21.15.194:6831/3725064809","heartbeat_front_addr":"172.21.15.194:6829/3725064809","state":["exists","up"]},{"osd":11,"uuid":"867a2535-8235-4ceb-a698-eb86869d6d20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6825","nonce":2501212422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6827","nonce":2501212422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6831","nonce":2501212422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6829","nonce":2501212422}]},"public_addr":"172.21.15.196:6825/2501212422","cluster_addr":"172.21.15.196:6827/2501212422","heartbeat_back_addr":"172.21.15.196:6831/2501212422","heartbeat_front_addr":"172.21.15.196:6829/2501212422","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:00:36.931566+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:01:02.614024+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:01:26.193265+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:01:58.308833+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:02:25.049546+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:02:49.748750+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:03:23.582153+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:03:50.447321+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:04:15.812129+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:04:49.998047+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:05:17.087924+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:05:43.167637+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":4,"to":7}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.a","mappings":[{"from":5,"to":8}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.39:0/3482086140":"2024-03-19T15:58:45.340294+0000","172.21.15.39:0/3844306284":"2024-03-19T15:58:45.340294+0000","172.21.15.39:6801/1810238546":"2024-03-19T15:58:45.340294+0000","172.21.15.39:6800/1810238546":"2024-03-19T15:58:45.340294+0000","172.21.15.39:0/2528397488":"2024-03-19T15:58:10.264798+0000","172.21.15.39:0/2240981254":"2024-03-19T15:58:10.264798+0000","172.21.15.39:6800/1217503189":"2024-03-19T15:58:10.264798+0000","172.21.15.39:0/3754876370":"2024-03-19T15:58:10.264798+0000","172.21.15.39:6801/1217503189":"2024-03-19T15:58:10.264798+0000","172.21.15.39:0/1874637628":"2024-03-19T15:58:45.340294+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T16:09:39.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.139+0000 7fab62800640 1 -- 172.21.15.39:0/1662441066 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fab440a2800 msgr2=0x7fab440a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:39.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.139+0000 7fab62800640 1 --2- 172.21.15.39:0/1662441066 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fab440a2800 0x7fab440a4cc0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fab48006010 tx=0x7fab48000bc0 comp rx=0 tx=0).stop 2024-03-18T16:09:39.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.139+0000 7fab62800640 1 -- 172.21.15.39:0/1662441066 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fab5c074240 msgr2=0x7fab5c19def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:39.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.139+0000 7fab62800640 1 --2- 172.21.15.39:0/1662441066 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fab5c074240 0x7fab5c19def0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fab5000e9f0 tx=0x7fab5000eeb0 comp rx=0 tx=0).stop 2024-03-18T16:09:39.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.139+0000 7fab62800640 1 -- 172.21.15.39:0/1662441066 shutdown_connections 2024-03-18T16:09:39.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.139+0000 7fab62800640 1 --2- 172.21.15.39:0/1662441066 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fab5c10dbb0 0x7fab5c1a54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:39.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.139+0000 7fab62800640 1 --2- 172.21.15.39:0/1662441066 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fab440a2800 0x7fab440a4cc0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:39.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.139+0000 7fab62800640 1 --2- 172.21.15.39:0/1662441066 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fab5c075820 0x7fab5c19e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:39.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.139+0000 7fab62800640 1 --2- 172.21.15.39:0/1662441066 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fab5c074240 0x7fab5c19def0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:39.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.139+0000 7fab62800640 1 -- 172.21.15.39:0/1662441066 >> 172.21.15.39:0/1662441066 conn(0x7fab5c0fcb70 msgr2=0x7fab5c100f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:39.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.139+0000 7fab62800640 1 -- 172.21.15.39:0/1662441066 shutdown_connections 2024-03-18T16:09:39.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.139+0000 7fab62800640 1 -- 172.21.15.39:0/1662441066 wait complete. 2024-03-18T16:09:39.164 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T16:01:30.048247+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T16:06:42.191791+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T16:06:43.670325+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T16:06:47.523388+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T16:09:39.164 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T16:09:39.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.366+0000 7fd5d8800640 1 -- 172.21.15.39:0/501697767 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd5d0103a20 msgr2=0x7fd5d0103e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:39.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.366+0000 7fd5d8800640 1 --2- 172.21.15.39:0/501697767 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd5d0103a20 0x7fd5d0103e40 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fd5c4009900 tx=0x7fd5c402bc40 comp rx=0 tx=0).stop 2024-03-18T16:09:39.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.366+0000 7fd5d8800640 1 -- 172.21.15.39:0/501697767 shutdown_connections 2024-03-18T16:09:39.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.366+0000 7fd5d8800640 1 --2- 172.21.15.39:0/501697767 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd5d01055d0 0x7fd5d010be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:39.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.366+0000 7fd5d8800640 1 --2- 172.21.15.39:0/501697767 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5d0104c10 0x7fd5d0105090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:39.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.366+0000 7fd5d8800640 1 --2- 172.21.15.39:0/501697767 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd5d0103a20 0x7fd5d0103e40 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:39.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.366+0000 7fd5d8800640 1 -- 172.21.15.39:0/501697767 >> 172.21.15.39:0/501697767 conn(0x7fd5d00fef90 msgr2=0x7fd5d01013d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:39.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.366+0000 7fd5d8800640 1 -- 172.21.15.39:0/501697767 shutdown_connections 2024-03-18T16:09:39.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.366+0000 7fd5d8800640 1 -- 172.21.15.39:0/501697767 wait complete. 2024-03-18T16:09:39.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.367+0000 7fd5d8800640 1 Processor -- start 2024-03-18T16:09:39.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.368+0000 7fd5d8800640 1 -- start start 2024-03-18T16:09:39.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.368+0000 7fd5d8800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd5d0103a20 0x7fd5d0199d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:39.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.368+0000 7fd5d8800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd5d0104c10 0x7fd5d019a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:39.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.368+0000 7fd5d8800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5d01055d0 0x7fd5d01a12a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:39.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.368+0000 7fd5d8800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fd5d010f6c0 con 0x7fd5d01055d0 2024-03-18T16:09:39.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.368+0000 7fd5d8800640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fd5d010f540 con 0x7fd5d0103a20 2024-03-18T16:09:39.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.368+0000 7fd5d8800640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fd5d010f840 con 0x7fd5d0104c10 2024-03-18T16:09:39.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.368+0000 7fd5d5e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd5d0103a20 0x7fd5d0199d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:39.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.368+0000 7fd5d5400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd5d0104c10 0x7fd5d019a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:39.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.368+0000 7fd5d5e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd5d0103a20 0x7fd5d0199d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:38470/0 (socket says 172.21.15.39:38470) 2024-03-18T16:09:39.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.368+0000 7fd5d5e00640 1 -- 172.21.15.39:0/788884712 learned_addr learned my addr 172.21.15.39:0/788884712 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:39.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.369+0000 7fd5d6800640 1 --2- 172.21.15.39:0/788884712 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5d01055d0 0x7fd5d01a12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:39.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.369+0000 7fd5d5e00640 1 -- 172.21.15.39:0/788884712 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd5d0104c10 msgr2=0x7fd5d019a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:39.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.369+0000 7fd5d5e00640 1 --2- 172.21.15.39:0/788884712 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd5d0104c10 0x7fd5d019a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:39.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.369+0000 7fd5d5e00640 1 -- 172.21.15.39:0/788884712 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5d01055d0 msgr2=0x7fd5d01a12a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:39.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.369+0000 7fd5d5e00640 1 --2- 172.21.15.39:0/788884712 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5d01055d0 0x7fd5d01a12a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:39.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.369+0000 7fd5d5e00640 1 -- 172.21.15.39:0/788884712 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5d01a1940 con 0x7fd5d0103a20 2024-03-18T16:09:39.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.369+0000 7fd5d5400640 1 --2- 172.21.15.39:0/788884712 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd5d0104c10 0x7fd5d019a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:09:39.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.369+0000 7fd5d5e00640 1 --2- 172.21.15.39:0/788884712 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd5d0103a20 0x7fd5d0199d30 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd5c402efd0 tx=0x7fd5c402c660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:39.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.369+0000 7fd5caa00640 1 -- 172.21.15.39:0/788884712 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5c4041070 con 0x7fd5d0103a20 2024-03-18T16:09:39.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.370+0000 7fd5d8800640 1 -- 172.21.15.39:0/788884712 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5d01a1bd0 con 0x7fd5d0103a20 2024-03-18T16:09:39.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.370+0000 7fd5caa00640 1 -- 172.21.15.39:0/788884712 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd5c4004450 con 0x7fd5d0103a20 2024-03-18T16:09:39.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.370+0000 7fd5caa00640 1 -- 172.21.15.39:0/788884712 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5c4003e50 con 0x7fd5d0103a20 2024-03-18T16:09:39.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.370+0000 7fd5d8800640 1 -- 172.21.15.39:0/788884712 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5d01a2090 con 0x7fd5d0103a20 2024-03-18T16:09:39.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.371+0000 7fd5d8800640 1 -- 172.21.15.39:0/788884712 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5d010ad40 con 0x7fd5d0103a20 2024-03-18T16:09:39.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.373+0000 7fd5caa00640 1 -- 172.21.15.39:0/788884712 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fd5c4044070 con 0x7fd5d0103a20 2024-03-18T16:09:39.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.374+0000 7fd5caa00640 1 --2- 172.21.15.39:0/788884712 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd5b80a29a0 0x7fd5b80a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:39.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.374+0000 7fd5caa00640 1 -- 172.21.15.39:0/788884712 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fd5c40f0a80 con 0x7fd5d0103a20 2024-03-18T16:09:39.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.374+0000 7fd5d5400640 1 --2- 172.21.15.39:0/788884712 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd5b80a29a0 0x7fd5b80a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:39.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.375+0000 7fd5d5400640 1 --2- 172.21.15.39:0/788884712 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd5b80a29a0 0x7fd5b80a4e60 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fd5c0009fc0 tx=0x7fd5c0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:39.383 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.380+0000 7fd5caa00640 1 -- 172.21.15.39:0/788884712 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd5c40f2e20 con 0x7fd5d0103a20 2024-03-18T16:09:39.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:39 smithi196 ceph-mon[36307]: pgmap v409: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T16:09:39.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:39 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/1662441066' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:09:39.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:39 smithi194 ceph-mon[36200]: pgmap v409: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T16:09:39.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:39 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/1662441066' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:09:39.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:39 smithi039 ceph-mon[31721]: pgmap v409: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T16:09:39.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:39 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/1662441066' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:09:39.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.605+0000 7fd5d8800640 1 -- 172.21.15.39:0/788884712 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd5d0062410 con 0x7fd5d0103a20 2024-03-18T16:09:39.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.605+0000 7fd5caa00640 1 -- 172.21.15.39:0/788884712 <== mon.2 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7fd5c40bd730 con 0x7fd5d0103a20 2024-03-18T16:09:39.608 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2024-03-18T16:09:39.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.608+0000 7fd5d8800640 1 -- 172.21.15.39:0/788884712 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd5b80a29a0 msgr2=0x7fd5b80a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:39.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.608+0000 7fd5d8800640 1 --2- 172.21.15.39:0/788884712 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd5b80a29a0 0x7fd5b80a4e60 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fd5c0009fc0 tx=0x7fd5c0005f90 comp rx=0 tx=0).stop 2024-03-18T16:09:39.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.608+0000 7fd5d8800640 1 -- 172.21.15.39:0/788884712 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd5d0103a20 msgr2=0x7fd5d0199d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:39.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.608+0000 7fd5d8800640 1 --2- 172.21.15.39:0/788884712 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd5d0103a20 0x7fd5d0199d30 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd5c402efd0 tx=0x7fd5c402c660 comp rx=0 tx=0).stop 2024-03-18T16:09:39.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.608+0000 7fd5d8800640 1 -- 172.21.15.39:0/788884712 shutdown_connections 2024-03-18T16:09:39.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.608+0000 7fd5d8800640 1 --2- 172.21.15.39:0/788884712 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd5b80a29a0 0x7fd5b80a4e60 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:39.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.608+0000 7fd5d8800640 1 --2- 172.21.15.39:0/788884712 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd5d01055d0 0x7fd5d01a12a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:39.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.608+0000 7fd5d8800640 1 --2- 172.21.15.39:0/788884712 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd5d0104c10 0x7fd5d019a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:39.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.608+0000 7fd5d8800640 1 --2- 172.21.15.39:0/788884712 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd5d0103a20 0x7fd5d0199d30 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:39.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.608+0000 7fd5d8800640 1 -- 172.21.15.39:0/788884712 >> 172.21.15.39:0/788884712 conn(0x7fd5d00fef90 msgr2=0x7fd5d0107f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:39.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.608+0000 7fd5d8800640 1 -- 172.21.15.39:0/788884712 shutdown_connections 2024-03-18T16:09:39.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.608+0000 7fd5d8800640 1 -- 172.21.15.39:0/788884712 wait complete. 2024-03-18T16:09:39.632 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T16:09:39.860 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.857+0000 7f5426e00640 1 -- 172.21.15.39:0/3032897938 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f54201016e0 msgr2=0x7f5420101b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:39.860 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.857+0000 7f5426e00640 1 --2- 172.21.15.39:0/3032897938 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f54201016e0 0x7f5420101b00 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f5414009ca0 tx=0x7f541402c100 comp rx=0 tx=0).stop 2024-03-18T16:09:39.860 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.858+0000 7f5426e00640 1 -- 172.21.15.39:0/3032897938 shutdown_connections 2024-03-18T16:09:39.860 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.858+0000 7f5426e00640 1 --2- 172.21.15.39:0/3032897938 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5420103290 0x7f5420109b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:39.860 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.858+0000 7f5426e00640 1 --2- 172.21.15.39:0/3032897938 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f54201028d0 0x7f5420102d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:39.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.858+0000 7f5426e00640 1 --2- 172.21.15.39:0/3032897938 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f54201016e0 0x7f5420101b00 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:39.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.858+0000 7f5426e00640 1 -- 172.21.15.39:0/3032897938 >> 172.21.15.39:0/3032897938 conn(0x7f54200fcc50 msgr2=0x7f54200ff090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:39.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.858+0000 7f5426e00640 1 -- 172.21.15.39:0/3032897938 shutdown_connections 2024-03-18T16:09:39.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.858+0000 7f5426e00640 1 -- 172.21.15.39:0/3032897938 wait complete. 2024-03-18T16:09:39.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.859+0000 7f5426e00640 1 Processor -- start 2024-03-18T16:09:39.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.859+0000 7f5426e00640 1 -- start start 2024-03-18T16:09:39.863 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.859+0000 7f5426e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f54201016e0 0x7f5420199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:39.863 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.859+0000 7f5426e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f54201028d0 0x7f542019a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:39.863 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.859+0000 7f5426e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5420103290 0x7f54201a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:39.863 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.859+0000 7f5426e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f542010d350 con 0x7f5420103290 2024-03-18T16:09:39.863 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.859+0000 7f5426e00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f542010d1d0 con 0x7f54201016e0 2024-03-18T16:09:39.863 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.859+0000 7f5426e00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f542010d4d0 con 0x7f54201028d0 2024-03-18T16:09:39.863 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.860+0000 7f5425a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f54201016e0 0x7f5420199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:39.863 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.860+0000 7f5425a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f54201016e0 0x7f5420199c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:38482/0 (socket says 172.21.15.39:38482) 2024-03-18T16:09:39.863 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.860+0000 7f5425000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f54201028d0 0x7f542019a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:39.863 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.860+0000 7f5425a00640 1 -- 172.21.15.39:0/1250478301 learned_addr learned my addr 172.21.15.39:0/1250478301 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:39.863 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.860+0000 7f5426400640 1 --2- 172.21.15.39:0/1250478301 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5420103290 0x7f54201a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:39.863 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.860+0000 7f5425a00640 1 -- 172.21.15.39:0/1250478301 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f54201028d0 msgr2=0x7f542019a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:39.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.860+0000 7f5425a00640 1 --2- 172.21.15.39:0/1250478301 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f54201028d0 0x7f542019a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:39.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.860+0000 7f5425a00640 1 -- 172.21.15.39:0/1250478301 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5420103290 msgr2=0x7f54201a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:39.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.860+0000 7f5425a00640 1 --2- 172.21.15.39:0/1250478301 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5420103290 0x7f54201a1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:39.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.860+0000 7f5425a00640 1 -- 172.21.15.39:0/1250478301 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54201a1830 con 0x7f54201016e0 2024-03-18T16:09:39.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.860+0000 7f5425a00640 1 --2- 172.21.15.39:0/1250478301 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f54201016e0 0x7f5420199c20 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f5414000f80 tx=0x7f5414035d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:39.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.860+0000 7f541e000640 1 -- 172.21.15.39:0/1250478301 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5414039040 con 0x7f54201016e0 2024-03-18T16:09:39.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.861+0000 7f541e000640 1 -- 172.21.15.39:0/1250478301 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5414044070 con 0x7f54201016e0 2024-03-18T16:09:39.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.861+0000 7f541e000640 1 -- 172.21.15.39:0/1250478301 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54140044a0 con 0x7f54201016e0 2024-03-18T16:09:39.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.861+0000 7f5426e00640 1 -- 172.21.15.39:0/1250478301 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54201a1ac0 con 0x7f54201016e0 2024-03-18T16:09:39.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.861+0000 7f5426e00640 1 -- 172.21.15.39:0/1250478301 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54201a1fd0 con 0x7f54201016e0 2024-03-18T16:09:39.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.862+0000 7f5426e00640 1 -- 172.21.15.39:0/1250478301 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54200684f0 con 0x7f54201016e0 2024-03-18T16:09:39.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.863+0000 7f541e000640 1 -- 172.21.15.39:0/1250478301 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f5414004640 con 0x7f54201016e0 2024-03-18T16:09:39.867 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.865+0000 7f541e000640 1 --2- 172.21.15.39:0/1250478301 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f54080a2880 0x7f54080a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:39.867 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.865+0000 7f541e000640 1 -- 172.21.15.39:0/1250478301 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f54140f1400 con 0x7f54201016e0 2024-03-18T16:09:39.867 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.865+0000 7f5425000640 1 --2- 172.21.15.39:0/1250478301 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f54080a2880 0x7f54080a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:39.868 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.866+0000 7f5425000640 1 --2- 172.21.15.39:0/1250478301 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f54080a2880 0x7f54080a4d40 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f540c009bd0 tx=0x7f540c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:39.871 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:39.869+0000 7f541e000640 1 -- 172.21.15.39:0/1250478301 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f54140be0b0 con 0x7f54201016e0 2024-03-18T16:09:40.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.148+0000 7f5426e00640 1 -- 172.21.15.39:0/1250478301 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f54201a2240 con 0x7f54201016e0 2024-03-18T16:09:40.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.149+0000 7f541e000640 1 -- 172.21.15.39:0/1250478301 <== mon.2 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7f54140bd9b0 con 0x7f54201016e0 2024-03-18T16:09:40.151 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 32 2024-03-18T16:09:40.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.152+0000 7f5426e00640 1 -- 172.21.15.39:0/1250478301 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f54080a2880 msgr2=0x7f54080a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:40.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.152+0000 7f5426e00640 1 --2- 172.21.15.39:0/1250478301 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f54080a2880 0x7f54080a4d40 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f540c009bd0 tx=0x7f540c005f90 comp rx=0 tx=0).stop 2024-03-18T16:09:40.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.152+0000 7f5426e00640 1 -- 172.21.15.39:0/1250478301 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f54201016e0 msgr2=0x7f5420199c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:40.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.152+0000 7f5426e00640 1 --2- 172.21.15.39:0/1250478301 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f54201016e0 0x7f5420199c20 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f5414000f80 tx=0x7f5414035d10 comp rx=0 tx=0).stop 2024-03-18T16:09:40.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.152+0000 7f5426e00640 1 -- 172.21.15.39:0/1250478301 shutdown_connections 2024-03-18T16:09:40.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.152+0000 7f5426e00640 1 --2- 172.21.15.39:0/1250478301 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f54080a2880 0x7f54080a4d40 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:40.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.152+0000 7f5426e00640 1 --2- 172.21.15.39:0/1250478301 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5420103290 0x7f54201a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:40.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.152+0000 7f5426e00640 1 --2- 172.21.15.39:0/1250478301 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f54201028d0 0x7f542019a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:40.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.152+0000 7f5426e00640 1 --2- 172.21.15.39:0/1250478301 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f54201016e0 0x7f5420199c20 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:40.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.152+0000 7f5426e00640 1 -- 172.21.15.39:0/1250478301 >> 172.21.15.39:0/1250478301 conn(0x7f54200fcc50 msgr2=0x7f5420108070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:40.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.152+0000 7f5426e00640 1 -- 172.21.15.39:0/1250478301 shutdown_connections 2024-03-18T16:09:40.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.152+0000 7f5426e00640 1 -- 172.21.15.39:0/1250478301 wait complete. 2024-03-18T16:09:40.173 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T16:09:40.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.389+0000 7fbbfa800640 1 -- 172.21.15.39:0/464166856 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbbf4100910 msgr2=0x7fbbf4100d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:40.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.389+0000 7fbbfa800640 1 --2- 172.21.15.39:0/464166856 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbbf4100910 0x7fbbf4100d70 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fbbe0009860 tx=0x7fbbe002bc00 comp rx=0 tx=0).stop 2024-03-18T16:09:40.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.389+0000 7fbbfa800640 1 -- 172.21.15.39:0/464166856 shutdown_connections 2024-03-18T16:09:40.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.389+0000 7fbbfa800640 1 --2- 172.21.15.39:0/464166856 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbbf41012b0 0x7fbbf4101730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:40.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.389+0000 7fbbfa800640 1 --2- 172.21.15.39:0/464166856 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbbf4100910 0x7fbbf4100d70 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:40.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.389+0000 7fbbfa800640 1 --2- 172.21.15.39:0/464166856 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbbf40fffb0 0x7fbbf41003d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:40.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.389+0000 7fbbfa800640 1 -- 172.21.15.39:0/464166856 >> 172.21.15.39:0/464166856 conn(0x7fbbf40fbb50 msgr2=0x7fbbf40fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:40.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.389+0000 7fbbfa800640 1 -- 172.21.15.39:0/464166856 shutdown_connections 2024-03-18T16:09:40.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.389+0000 7fbbfa800640 1 -- 172.21.15.39:0/464166856 wait complete. 2024-03-18T16:09:40.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.390+0000 7fbbfa800640 1 Processor -- start 2024-03-18T16:09:40.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.390+0000 7fbbfa800640 1 -- start start 2024-03-18T16:09:40.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.391+0000 7fbbfa800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbbf40fffb0 0x7fbbf4199bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:40.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.391+0000 7fbbfa800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbbf4100910 0x7fbbf419a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:40.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.391+0000 7fbbfa800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbbf41012b0 0x7fbbf41a1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:40.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.391+0000 7fbbfa800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fbbf410f520 con 0x7fbbf41012b0 2024-03-18T16:09:40.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.391+0000 7fbbfa800640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fbbf410f3a0 con 0x7fbbf4100910 2024-03-18T16:09:40.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.391+0000 7fbbfa800640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fbbf410f6a0 con 0x7fbbf40fffb0 2024-03-18T16:09:40.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.391+0000 7fbbf9400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbbf40fffb0 0x7fbbf4199bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:40.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.391+0000 7fbbf8a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbbf4100910 0x7fbbf419a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:40.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.391+0000 7fbbf9400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbbf40fffb0 0x7fbbf4199bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:55942/0 (socket says 172.21.15.39:55942) 2024-03-18T16:09:40.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.391+0000 7fbbf9400640 1 -- 172.21.15.39:0/1102345503 learned_addr learned my addr 172.21.15.39:0/1102345503 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:40.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.391+0000 7fbbf9e00640 1 --2- 172.21.15.39:0/1102345503 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbbf41012b0 0x7fbbf41a1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:40.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.391+0000 7fbbf9400640 1 -- 172.21.15.39:0/1102345503 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbbf4100910 msgr2=0x7fbbf419a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:40.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.391+0000 7fbbf9400640 1 --2- 172.21.15.39:0/1102345503 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbbf4100910 0x7fbbf419a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:40.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.391+0000 7fbbf9400640 1 -- 172.21.15.39:0/1102345503 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbbf41012b0 msgr2=0x7fbbf41a1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:40.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.391+0000 7fbbf9400640 1 --2- 172.21.15.39:0/1102345503 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbbf41012b0 0x7fbbf41a1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:40.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.391+0000 7fbbf9400640 1 -- 172.21.15.39:0/1102345503 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbf41a17e0 con 0x7fbbf40fffb0 2024-03-18T16:09:40.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.391+0000 7fbbf8a00640 1 --2- 172.21.15.39:0/1102345503 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbbf4100910 0x7fbbf419a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:09:40.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.392+0000 7fbbf9400640 1 --2- 172.21.15.39:0/1102345503 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbbf40fffb0 0x7fbbf4199bc0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fbbe800e9c0 tx=0x7fbbe800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:40.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.392+0000 7fbbf2000640 1 -- 172.21.15.39:0/1102345503 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbe800cde0 con 0x7fbbf40fffb0 2024-03-18T16:09:40.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.392+0000 7fbbf2000640 1 -- 172.21.15.39:0/1102345503 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbbe80035f0 con 0x7fbbf40fffb0 2024-03-18T16:09:40.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.392+0000 7fbbf2000640 1 -- 172.21.15.39:0/1102345503 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbe8010690 con 0x7fbbf40fffb0 2024-03-18T16:09:40.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.392+0000 7fbbfa800640 1 -- 172.21.15.39:0/1102345503 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbf41a1ad0 con 0x7fbbf40fffb0 2024-03-18T16:09:40.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.392+0000 7fbbfa800640 1 -- 172.21.15.39:0/1102345503 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbf41a1fa0 con 0x7fbbf40fffb0 2024-03-18T16:09:40.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.393+0000 7fbbfa800640 1 -- 172.21.15.39:0/1102345503 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbbf410ad00 con 0x7fbbf40fffb0 2024-03-18T16:09:40.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.395+0000 7fbbf2000640 1 -- 172.21.15.39:0/1102345503 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fbbe80037a0 con 0x7fbbf40fffb0 2024-03-18T16:09:40.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.396+0000 7fbbf2000640 1 --2- 172.21.15.39:0/1102345503 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fbbdc0a2880 0x7fbbdc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:40.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.397+0000 7fbbf2000640 1 -- 172.21.15.39:0/1102345503 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fbbe80cf0a0 con 0x7fbbf40fffb0 2024-03-18T16:09:40.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.397+0000 7fbbf8a00640 1 --2- 172.21.15.39:0/1102345503 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fbbdc0a2880 0x7fbbdc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:40.400 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.398+0000 7fbbf8a00640 1 --2- 172.21.15.39:0/1102345503 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fbbdc0a2880 0x7fbbdc0a4d40 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fbbe0009830 tx=0x7fbbe0034b50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:40.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.403+0000 7fbbf2000640 1 -- 172.21.15.39:0/1102345503 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fbbe809bd50 con 0x7fbbf40fffb0 2024-03-18T16:09:40.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:40 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/788884712' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T16:09:40.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:40 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/788884712' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T16:09:40.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:40 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/788884712' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T16:09:40.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.608+0000 7fbbfa800640 1 -- 172.21.15.39:0/1102345503 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fbbf402d460 con 0x7fbbf40fffb0 2024-03-18T16:09:40.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.609+0000 7fbbf2000640 1 -- 172.21.15.39:0/1102345503 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7fbbe809b650 con 0x7fbbf40fffb0 2024-03-18T16:09:40.612 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 64 2024-03-18T16:09:40.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.612+0000 7fbbfa800640 1 -- 172.21.15.39:0/1102345503 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fbbdc0a2880 msgr2=0x7fbbdc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:40.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.612+0000 7fbbfa800640 1 --2- 172.21.15.39:0/1102345503 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fbbdc0a2880 0x7fbbdc0a4d40 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fbbe0009830 tx=0x7fbbe0034b50 comp rx=0 tx=0).stop 2024-03-18T16:09:40.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.612+0000 7fbbfa800640 1 -- 172.21.15.39:0/1102345503 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbbf40fffb0 msgr2=0x7fbbf4199bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:40.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.612+0000 7fbbfa800640 1 --2- 172.21.15.39:0/1102345503 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbbf40fffb0 0x7fbbf4199bc0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fbbe800e9c0 tx=0x7fbbe800ee80 comp rx=0 tx=0).stop 2024-03-18T16:09:40.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.612+0000 7fbbfa800640 1 -- 172.21.15.39:0/1102345503 shutdown_connections 2024-03-18T16:09:40.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.612+0000 7fbbfa800640 1 --2- 172.21.15.39:0/1102345503 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbbf4100910 0x7fbbf419a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:40.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.612+0000 7fbbfa800640 1 --2- 172.21.15.39:0/1102345503 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fbbdc0a2880 0x7fbbdc0a4d40 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:40.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.612+0000 7fbbfa800640 1 --2- 172.21.15.39:0/1102345503 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbbf41012b0 0x7fbbf41a1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:40.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.612+0000 7fbbfa800640 1 --2- 172.21.15.39:0/1102345503 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbbf40fffb0 0x7fbbf4199bc0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:40.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.612+0000 7fbbfa800640 1 -- 172.21.15.39:0/1102345503 >> 172.21.15.39:0/1102345503 conn(0x7fbbf40fbb50 msgr2=0x7fbbf40fd650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:40.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.612+0000 7fbbfa800640 1 -- 172.21.15.39:0/1102345503 shutdown_connections 2024-03-18T16:09:40.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.612+0000 7fbbfa800640 1 -- 172.21.15.39:0/1102345503 wait complete. 2024-03-18T16:09:40.633 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T16:09:40.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.843+0000 7fa70b000640 1 -- 172.21.15.39:0/2557321119 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa7041033a0 msgr2=0x7fa704109c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:40.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.843+0000 7fa70b000640 1 --2- 172.21.15.39:0/2557321119 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa7041033a0 0x7fa704109c30 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fa6fc009860 tx=0x7fa6fc02bb70 comp rx=0 tx=0).stop 2024-03-18T16:09:40.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.843+0000 7fa70b000640 1 -- 172.21.15.39:0/2557321119 shutdown_connections 2024-03-18T16:09:40.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.843+0000 7fa70b000640 1 --2- 172.21.15.39:0/2557321119 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa7041029e0 0x7fa704102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:40.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.843+0000 7fa70b000640 1 --2- 172.21.15.39:0/2557321119 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa7041033a0 0x7fa704109c30 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:40.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.843+0000 7fa70b000640 1 --2- 172.21.15.39:0/2557321119 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7041017f0 0x7fa704101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:40.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.843+0000 7fa70b000640 1 -- 172.21.15.39:0/2557321119 >> 172.21.15.39:0/2557321119 conn(0x7fa7040fcd80 msgr2=0x7fa7040ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:40.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.844+0000 7fa70b000640 1 -- 172.21.15.39:0/2557321119 shutdown_connections 2024-03-18T16:09:40.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.844+0000 7fa70b000640 1 -- 172.21.15.39:0/2557321119 wait complete. 2024-03-18T16:09:40.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.845+0000 7fa70b000640 1 Processor -- start 2024-03-18T16:09:40.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.845+0000 7fa70b000640 1 -- start start 2024-03-18T16:09:40.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.845+0000 7fa70b000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa7041017f0 0x7fa704199c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:40.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.845+0000 7fa70b000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa7041029e0 0x7fa70419a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:40.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.845+0000 7fa70b000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7041033a0 0x7fa7041a1230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:40.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.845+0000 7fa70b000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fa70410d450 con 0x7fa7041033a0 2024-03-18T16:09:40.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.845+0000 7fa70b000640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fa70410d2d0 con 0x7fa7041029e0 2024-03-18T16:09:40.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.845+0000 7fa70b000640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fa70410d5d0 con 0x7fa7041017f0 2024-03-18T16:09:40.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.846+0000 7fa703e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa7041017f0 0x7fa704199c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:40.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.846+0000 7fa703e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa7041017f0 0x7fa704199c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:55966/0 (socket says 172.21.15.39:55966) 2024-03-18T16:09:40.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.846+0000 7fa703e00640 1 -- 172.21.15.39:0/119693489 learned_addr learned my addr 172.21.15.39:0/119693489 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:40.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.846+0000 7fa6fb400640 1 --2- 172.21.15.39:0/119693489 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa7041029e0 0x7fa70419a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:40.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.846+0000 7fa709000640 1 --2- 172.21.15.39:0/119693489 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7041033a0 0x7fa7041a1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:40.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.846+0000 7fa703e00640 1 -- 172.21.15.39:0/119693489 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa7041029e0 msgr2=0x7fa70419a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:40.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.846+0000 7fa703e00640 1 --2- 172.21.15.39:0/119693489 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa7041029e0 0x7fa70419a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:40.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.846+0000 7fa703e00640 1 -- 172.21.15.39:0/119693489 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7041033a0 msgr2=0x7fa7041a1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:40.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.846+0000 7fa703e00640 1 --2- 172.21.15.39:0/119693489 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7041033a0 0x7fa7041a1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:40.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.846+0000 7fa703e00640 1 -- 172.21.15.39:0/119693489 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7041a18d0 con 0x7fa7041017f0 2024-03-18T16:09:40.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.846+0000 7fa709000640 1 --2- 172.21.15.39:0/119693489 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7041033a0 0x7fa7041a1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:09:40.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.846+0000 7fa703e00640 1 --2- 172.21.15.39:0/119693489 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa7041017f0 0x7fa704199c90 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa6f400c3a0 tx=0x7fa6f400c860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:40.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.847+0000 7fa701600640 1 -- 172.21.15.39:0/119693489 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6f4003970 con 0x7fa7041017f0 2024-03-18T16:09:40.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.847+0000 7fa701600640 1 -- 172.21.15.39:0/119693489 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa6f4004020 con 0x7fa7041017f0 2024-03-18T16:09:40.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.847+0000 7fa701600640 1 -- 172.21.15.39:0/119693489 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6f400b9f0 con 0x7fa7041017f0 2024-03-18T16:09:40.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.847+0000 7fa70b000640 1 -- 172.21.15.39:0/119693489 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7041a1bc0 con 0x7fa7041017f0 2024-03-18T16:09:40.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.847+0000 7fa70b000640 1 -- 172.21.15.39:0/119693489 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7041a2160 con 0x7fa7041017f0 2024-03-18T16:09:40.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.849+0000 7fa70b000640 1 -- 172.21.15.39:0/119693489 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7041a2330 con 0x7fa7041017f0 2024-03-18T16:09:40.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.850+0000 7fa701600640 1 -- 172.21.15.39:0/119693489 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fa6f401d020 con 0x7fa7041017f0 2024-03-18T16:09:40.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.851+0000 7fa701600640 1 --2- 172.21.15.39:0/119693489 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa6e80a2880 0x7fa6e80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:40.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.852+0000 7fa701600640 1 -- 172.21.15.39:0/119693489 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fa6f40ceeb0 con 0x7fa7041017f0 2024-03-18T16:09:40.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.852+0000 7fa6fb400640 1 --2- 172.21.15.39:0/119693489 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa6e80a2880 0x7fa6e80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:40.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.852+0000 7fa6fb400640 1 --2- 172.21.15.39:0/119693489 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa6e80a2880 0x7fa6e80a4d40 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fa6ec0095f0 tx=0x7fa6ec005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:40.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:40.859+0000 7fa701600640 1 -- 172.21.15.39:0/119693489 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa6f409bbc0 con 0x7fa7041017f0 2024-03-18T16:09:41.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.157+0000 7fa70b000640 1 -- 172.21.15.39:0/119693489 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fa7041a25a0 con 0x7fa7041017f0 2024-03-18T16:09:41.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.160+0000 7fa701600640 1 -- 172.21.15.39:0/119693489 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7fa6f409b4c0 con 0x7fa7041017f0 2024-03-18T16:09:41.162 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 32 2024-03-18T16:09:41.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.163+0000 7fa70b000640 1 -- 172.21.15.39:0/119693489 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa6e80a2880 msgr2=0x7fa6e80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:41.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.163+0000 7fa70b000640 1 --2- 172.21.15.39:0/119693489 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa6e80a2880 0x7fa6e80a4d40 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fa6ec0095f0 tx=0x7fa6ec005f90 comp rx=0 tx=0).stop 2024-03-18T16:09:41.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.163+0000 7fa70b000640 1 -- 172.21.15.39:0/119693489 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa7041017f0 msgr2=0x7fa704199c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:41.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.163+0000 7fa70b000640 1 --2- 172.21.15.39:0/119693489 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa7041017f0 0x7fa704199c90 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fa6f400c3a0 tx=0x7fa6f400c860 comp rx=0 tx=0).stop 2024-03-18T16:09:41.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.163+0000 7fa70b000640 1 -- 172.21.15.39:0/119693489 shutdown_connections 2024-03-18T16:09:41.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.163+0000 7fa70b000640 1 --2- 172.21.15.39:0/119693489 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa7041029e0 0x7fa70419a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:41.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.163+0000 7fa70b000640 1 --2- 172.21.15.39:0/119693489 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa6e80a2880 0x7fa6e80a4d40 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:41.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.163+0000 7fa70b000640 1 --2- 172.21.15.39:0/119693489 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7041033a0 0x7fa7041a1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:41.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.163+0000 7fa70b000640 1 --2- 172.21.15.39:0/119693489 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa7041017f0 0x7fa704199c90 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:41.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.163+0000 7fa70b000640 1 -- 172.21.15.39:0/119693489 >> 172.21.15.39:0/119693489 conn(0x7fa7040fcd80 msgr2=0x7fa704073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:41.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.163+0000 7fa70b000640 1 -- 172.21.15.39:0/119693489 shutdown_connections 2024-03-18T16:09:41.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.163+0000 7fa70b000640 1 -- 172.21.15.39:0/119693489 wait complete. 2024-03-18T16:09:41.186 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:09:41.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.405+0000 7fa9d8800640 1 -- 172.21.15.39:0/2863164312 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa9d01029d0 msgr2=0x7fa9d0102e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:41.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.405+0000 7fa9d8800640 1 --2- 172.21.15.39:0/2863164312 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa9d01029d0 0x7fa9d0102e50 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fa9c0009900 tx=0x7fa9c002bc40 comp rx=0 tx=0).stop 2024-03-18T16:09:41.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.405+0000 7fa9d8800640 1 -- 172.21.15.39:0/2863164312 shutdown_connections 2024-03-18T16:09:41.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.405+0000 7fa9d8800640 1 --2- 172.21.15.39:0/2863164312 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa9d0103390 0x7fa9d0109c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:41.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.405+0000 7fa9d8800640 1 --2- 172.21.15.39:0/2863164312 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa9d01029d0 0x7fa9d0102e50 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:41.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.405+0000 7fa9d8800640 1 --2- 172.21.15.39:0/2863164312 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa9d01017e0 0x7fa9d0101c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:41.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.405+0000 7fa9d8800640 1 -- 172.21.15.39:0/2863164312 >> 172.21.15.39:0/2863164312 conn(0x7fa9d00fcd50 msgr2=0x7fa9d00ff190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:41.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.405+0000 7fa9d8800640 1 -- 172.21.15.39:0/2863164312 shutdown_connections 2024-03-18T16:09:41.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.405+0000 7fa9d8800640 1 -- 172.21.15.39:0/2863164312 wait complete. 2024-03-18T16:09:41.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.406+0000 7fa9d8800640 1 Processor -- start 2024-03-18T16:09:41.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.406+0000 7fa9d8800640 1 -- start start 2024-03-18T16:09:41.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.407+0000 7fa9d8800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa9d01017e0 0x7fa9d0197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:41.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.407+0000 7fa9d8800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa9d01029d0 0x7fa9d0197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:41.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.407+0000 7fa9d8800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa9d0103390 0x7fa9d019f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:41.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.407+0000 7fa9d8800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fa9d010d490 con 0x7fa9d0103390 2024-03-18T16:09:41.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.407+0000 7fa9d8800640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fa9d010d310 con 0x7fa9d01017e0 2024-03-18T16:09:41.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.407+0000 7fa9d8800640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fa9d010d610 con 0x7fa9d01029d0 2024-03-18T16:09:41.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.407+0000 7fa9d5e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa9d01017e0 0x7fa9d0197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:41.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.407+0000 7fa9d5e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa9d01017e0 0x7fa9d0197ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:38536/0 (socket says 172.21.15.39:38536) 2024-03-18T16:09:41.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.407+0000 7fa9d5e00640 1 -- 172.21.15.39:0/68684024 learned_addr learned my addr 172.21.15.39:0/68684024 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:41.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.407+0000 7fa9d6800640 1 --2- 172.21.15.39:0/68684024 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa9d0103390 0x7fa9d019f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:41.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.407+0000 7fa9d5400640 1 --2- 172.21.15.39:0/68684024 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa9d01029d0 0x7fa9d0197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:41.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.408+0000 7fa9d5e00640 1 -- 172.21.15.39:0/68684024 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa9d01029d0 msgr2=0x7fa9d0197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:41.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.408+0000 7fa9d5e00640 1 --2- 172.21.15.39:0/68684024 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa9d01029d0 0x7fa9d0197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:41.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.408+0000 7fa9d5e00640 1 -- 172.21.15.39:0/68684024 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa9d0103390 msgr2=0x7fa9d019f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:41.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.408+0000 7fa9d5e00640 1 --2- 172.21.15.39:0/68684024 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa9d0103390 0x7fa9d019f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:41.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.408+0000 7fa9d5e00640 1 -- 172.21.15.39:0/68684024 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9d019f6d0 con 0x7fa9d01017e0 2024-03-18T16:09:41.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.408+0000 7fa9d5e00640 1 --2- 172.21.15.39:0/68684024 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa9d01017e0 0x7fa9d0197ab0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa9c400d770 tx=0x7fa9c400dc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:41.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.408+0000 7fa9caa00640 1 -- 172.21.15.39:0/68684024 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9c4017070 con 0x7fa9d01017e0 2024-03-18T16:09:41.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.408+0000 7fa9caa00640 1 -- 172.21.15.39:0/68684024 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa9c4009c30 con 0x7fa9d01017e0 2024-03-18T16:09:41.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.408+0000 7fa9d8800640 1 -- 172.21.15.39:0/68684024 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9d019f9c0 con 0x7fa9d01017e0 2024-03-18T16:09:41.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.408+0000 7fa9caa00640 1 -- 172.21.15.39:0/68684024 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9c4003dd0 con 0x7fa9d01017e0 2024-03-18T16:09:41.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.408+0000 7fa9d8800640 1 -- 172.21.15.39:0/68684024 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9d019ff30 con 0x7fa9d01017e0 2024-03-18T16:09:41.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.410+0000 7fa9d8800640 1 -- 172.21.15.39:0/68684024 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9d0108aa0 con 0x7fa9d01017e0 2024-03-18T16:09:41.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.411+0000 7fa9caa00640 1 -- 172.21.15.39:0/68684024 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fa9c4003fb0 con 0x7fa9d01017e0 2024-03-18T16:09:41.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.412+0000 7fa9caa00640 1 --2- 172.21.15.39:0/68684024 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa9b80a2880 0x7fa9b80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:41.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.413+0000 7fa9caa00640 1 -- 172.21.15.39:0/68684024 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fa9c40ce6b0 con 0x7fa9d01017e0 2024-03-18T16:09:41.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.413+0000 7fa9d5400640 1 --2- 172.21.15.39:0/68684024 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa9b80a2880 0x7fa9b80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:41.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.414+0000 7fa9d5400640 1 --2- 172.21.15.39:0/68684024 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa9b80a2880 0x7fa9b80a4d40 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fa9c002c250 tx=0x7fa9c002cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:41.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.416+0000 7fa9caa00640 1 -- 172.21.15.39:0/68684024 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa9c4009de0 con 0x7fa9d01017e0 2024-03-18T16:09:41.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:41 smithi196 ceph-mon[36307]: pgmap v410: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T16:09:41.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:41 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/1250478301' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T16:09:41.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:41 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/1102345503' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T16:09:41.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:41 smithi194 ceph-mon[36200]: pgmap v410: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T16:09:41.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:41 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/1250478301' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T16:09:41.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:41 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/1102345503' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T16:09:41.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:41 smithi039 ceph-mon[31721]: pgmap v410: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T16:09:41.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:41 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/1250478301' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T16:09:41.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:41 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/1102345503' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T16:09:41.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.693+0000 7fa9d8800640 1 -- 172.21.15.39:0/68684024 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa9d00623e0 con 0x7fa9d01017e0 2024-03-18T16:09:41.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.694+0000 7fa9caa00640 1 -- 172.21.15.39:0/68684024 <== mon.2 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11932 (secure 0 0 0) 0x7fa9c409ad10 con 0x7fa9d01017e0 2024-03-18T16:09:41.697 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:09:41.697 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2520226608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2520226608},{"type":"v1","addr":"172.21.15.39:6835","nonce":2520226608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/719341763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":719341763},{"type":"v1","addr":"172.21.15.39:6837","nonce":719341763}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T16:06:44.180425+0000","modified":"2024-03-18T16:07:20.994405+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24575,"mds_1":24556,"mds_2":24535,"mds_3":24524,"mds_4":14463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14445":{"gid":14445,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.39:6839/656513813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":656513813},{"type":"v1","addr":"172.21.15.39:6839","nonce":656513813}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14463":{"gid":14463,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":13,"addr":"172.21.15.39:6841/3515326042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3515326042},{"type":"v1","addr":"172.21.15.39:6841","nonce":3515326042}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6833/931350738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":931350738},{"type":"v1","addr":"172.21.15.194:6833","nonce":931350738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6833/3874209621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":3874209621},{"type":"v1","addr":"172.21.15.196:6833","nonce":3874209621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6835/188385470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":188385470},{"type":"v1","addr":"172.21.15.194:6835","nonce":188385470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6835/3949984309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":3949984309},{"type":"v1","addr":"172.21.15.196:6835","nonce":3949984309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":15,"addr":"172.21.15.194:6837/1301464246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":1301464246},{"type":"v1","addr":"172.21.15.194:6837","nonce":1301464246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":16,"addr":"172.21.15.196:6837/895247251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":895247251},{"type":"v1","addr":"172.21.15.196:6837","nonce":895247251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.196:6839/2001427268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6838","nonce":2001427268},{"type":"v1","addr":"172.21.15.196:6839","nonce":2001427268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24575":{"gid":24575,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.194:6839/274911158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":274911158},{"type":"v1","addr":"172.21.15.194:6839","nonce":274911158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24575,"qdb_cluster":[24556,24535,24575,24524,14463]},"id":1}]} 2024-03-18T16:09:41.698 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 27 2024-03-18T16:09:41.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.698+0000 7fa9d8800640 1 -- 172.21.15.39:0/68684024 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa9b80a2880 msgr2=0x7fa9b80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:41.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.698+0000 7fa9d8800640 1 --2- 172.21.15.39:0/68684024 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa9b80a2880 0x7fa9b80a4d40 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fa9c002c250 tx=0x7fa9c002cf90 comp rx=0 tx=0).stop 2024-03-18T16:09:41.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.698+0000 7fa9d8800640 1 -- 172.21.15.39:0/68684024 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa9d01017e0 msgr2=0x7fa9d0197ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:41.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.698+0000 7fa9d8800640 1 --2- 172.21.15.39:0/68684024 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa9d01017e0 0x7fa9d0197ab0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa9c400d770 tx=0x7fa9c400dc30 comp rx=0 tx=0).stop 2024-03-18T16:09:41.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.698+0000 7fa9d8800640 1 -- 172.21.15.39:0/68684024 shutdown_connections 2024-03-18T16:09:41.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.698+0000 7fa9d8800640 1 --2- 172.21.15.39:0/68684024 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa9b80a2880 0x7fa9b80a4d40 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:41.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.698+0000 7fa9d8800640 1 --2- 172.21.15.39:0/68684024 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa9d0103390 0x7fa9d019f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:41.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.698+0000 7fa9d8800640 1 --2- 172.21.15.39:0/68684024 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa9d01029d0 0x7fa9d0197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:41.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.698+0000 7fa9d8800640 1 --2- 172.21.15.39:0/68684024 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa9d01017e0 0x7fa9d0197ab0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:41.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.698+0000 7fa9d8800640 1 -- 172.21.15.39:0/68684024 >> 172.21.15.39:0/68684024 conn(0x7fa9d00fcd50 msgr2=0x7fa9d0105c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:41.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.698+0000 7fa9d8800640 1 -- 172.21.15.39:0/68684024 shutdown_connections 2024-03-18T16:09:41.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.698+0000 7fa9d8800640 1 -- 172.21.15.39:0/68684024 wait complete. 2024-03-18T16:09:41.722 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T16:09:41.952 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.949+0000 7fdad4a00640 1 -- 172.21.15.39:0/4272205529 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdad00ffe60 msgr2=0x7fdad0100280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:41.952 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.949+0000 7fdad4a00640 1 --2- 172.21.15.39:0/4272205529 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdad00ffe60 0x7fdad0100280 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fdac0009900 tx=0x7fdac002bc40 comp rx=0 tx=0).stop 2024-03-18T16:09:41.952 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.950+0000 7fdad4a00640 1 -- 172.21.15.39:0/4272205529 shutdown_connections 2024-03-18T16:09:41.952 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.950+0000 7fdad4a00640 1 --2- 172.21.15.39:0/4272205529 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdad01007c0 0x7fdad0100c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:41.952 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.950+0000 7fdad4a00640 1 --2- 172.21.15.39:0/4272205529 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdad0101160 0x7fdad01015e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:41.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.950+0000 7fdad4a00640 1 --2- 172.21.15.39:0/4272205529 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdad00ffe60 0x7fdad0100280 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:41.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.950+0000 7fdad4a00640 1 -- 172.21.15.39:0/4272205529 >> 172.21.15.39:0/4272205529 conn(0x7fdad00fba60 msgr2=0x7fdad00fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:41.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.950+0000 7fdad4a00640 1 -- 172.21.15.39:0/4272205529 shutdown_connections 2024-03-18T16:09:41.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.950+0000 7fdad4a00640 1 -- 172.21.15.39:0/4272205529 wait complete. 2024-03-18T16:09:41.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.951+0000 7fdad4a00640 1 Processor -- start 2024-03-18T16:09:41.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.951+0000 7fdad4a00640 1 -- start start 2024-03-18T16:09:41.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.951+0000 7fdad4a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdad00ffe60 0x7fdad0199a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:41.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.952+0000 7fdad4a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdad01007c0 0x7fdad0199fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:41.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.952+0000 7fdad4a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdad0101160 0x7fdad01a1040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:41.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.952+0000 7fdad4a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fdad010f480 con 0x7fdad01007c0 2024-03-18T16:09:41.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.952+0000 7fdad4a00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fdad010f300 con 0x7fdad0101160 2024-03-18T16:09:41.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.952+0000 7fdad4a00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fdad010f600 con 0x7fdad00ffe60 2024-03-18T16:09:41.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.952+0000 7fdace800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdad00ffe60 0x7fdad0199a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:41.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.952+0000 7fdace800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdad00ffe60 0x7fdad0199a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:56020/0 (socket says 172.21.15.39:56020) 2024-03-18T16:09:41.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.952+0000 7fdace800640 1 -- 172.21.15.39:0/335401520 learned_addr learned my addr 172.21.15.39:0/335401520 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:41.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.952+0000 7fdac7e00640 1 --2- 172.21.15.39:0/335401520 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdad01007c0 0x7fdad0199fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:41.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.952+0000 7fdacf200640 1 --2- 172.21.15.39:0/335401520 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdad0101160 0x7fdad01a1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:41.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.952+0000 7fdace800640 1 -- 172.21.15.39:0/335401520 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdad0101160 msgr2=0x7fdad01a1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:41.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.952+0000 7fdace800640 1 --2- 172.21.15.39:0/335401520 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdad0101160 0x7fdad01a1040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:41.956 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.952+0000 7fdace800640 1 -- 172.21.15.39:0/335401520 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdad01007c0 msgr2=0x7fdad0199fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:41.956 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.952+0000 7fdace800640 1 --2- 172.21.15.39:0/335401520 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdad01007c0 0x7fdad0199fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:41.956 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.952+0000 7fdace800640 1 -- 172.21.15.39:0/335401520 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdad01a16b0 con 0x7fdad00ffe60 2024-03-18T16:09:41.956 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.952+0000 7fdac7e00640 1 --2- 172.21.15.39:0/335401520 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdad01007c0 0x7fdad0199fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:09:41.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.953+0000 7fdace800640 1 --2- 172.21.15.39:0/335401520 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdad00ffe60 0x7fdad0199a90 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fdac002c8f0 tx=0x7fdac002c660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:41.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.953+0000 7fdac7400640 1 -- 172.21.15.39:0/335401520 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdac0041070 con 0x7fdad00ffe60 2024-03-18T16:09:41.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.953+0000 7fdac7400640 1 -- 172.21.15.39:0/335401520 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdac0004450 con 0x7fdad00ffe60 2024-03-18T16:09:41.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.953+0000 7fdac7400640 1 -- 172.21.15.39:0/335401520 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdac0034440 con 0x7fdad00ffe60 2024-03-18T16:09:41.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.953+0000 7fdad4a00640 1 -- 172.21.15.39:0/335401520 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdad01a1940 con 0x7fdad00ffe60 2024-03-18T16:09:41.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.953+0000 7fdad4a00640 1 -- 172.21.15.39:0/335401520 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdad01a1de0 con 0x7fdad00ffe60 2024-03-18T16:09:41.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.954+0000 7fdad4a00640 1 -- 172.21.15.39:0/335401520 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdad010acf0 con 0x7fdad00ffe60 2024-03-18T16:09:41.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.956+0000 7fdac7400640 1 -- 172.21.15.39:0/335401520 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fdac0044070 con 0x7fdad00ffe60 2024-03-18T16:09:41.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.958+0000 7fdac7400640 1 --2- 172.21.15.39:0/335401520 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fdab40a2880 0x7fdab40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:41.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.958+0000 7fdac7400640 1 -- 172.21.15.39:0/335401520 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fdac00f0c70 con 0x7fdad00ffe60 2024-03-18T16:09:41.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.958+0000 7fdac7e00640 1 --2- 172.21.15.39:0/335401520 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fdab40a2880 0x7fdab40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:41.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.959+0000 7fdac7e00640 1 --2- 172.21.15.39:0/335401520 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fdab40a2880 0x7fdab40a4d40 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fdab80095f0 tx=0x7fdab8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:41.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:41.960+0000 7fdac7400640 1 -- 172.21.15.39:0/335401520 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fdac00bd2d0 con 0x7fdad00ffe60 2024-03-18T16:09:42.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.167+0000 7fdad4a00640 1 -- 172.21.15.39:0/335401520 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdad002d380 con 0x7fdad00ffe60 2024-03-18T16:09:42.172 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.170+0000 7fdac7400640 1 -- 172.21.15.39:0/335401520 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24762 (secure 0 0 0) 0x7fdac00c1370 con 0x7fdad00ffe60 2024-03-18T16:09:42.173 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:09:42.173 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":93,"fsid":"1892d8fc-e540-11ee-95c9-87774f69a715","created":"2024-03-18T15:57:36.507008+0000","modified":"2024-03-18T16:08:47.874967+0000","last_up_change":"2024-03-18T16:05:45.250629+0000","last_in_change":"2024-03-18T16:05:24.841337+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T16:01:30.048247+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T16:06:42.191791+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T16:06:43.670325+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T16:06:47.523388+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"771e0558-be62-4436-94b0-6364751911c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6803","nonce":4090743635}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6805","nonce":4090743635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6809","nonce":4090743635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6807","nonce":4090743635}]},"public_addr":"172.21.15.39:6803/4090743635","cluster_addr":"172.21.15.39:6805/4090743635","heartbeat_back_addr":"172.21.15.39:6809/4090743635","heartbeat_front_addr":"172.21.15.39:6807/4090743635","state":["exists","up"]},{"osd":1,"uuid":"0430eee1-cf34-4277-b255-c7d9f07ad19a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6801","nonce":2800329805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6803","nonce":2800329805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6807","nonce":2800329805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6805","nonce":2800329805}]},"public_addr":"172.21.15.194:6801/2800329805","cluster_addr":"172.21.15.194:6803/2800329805","heartbeat_back_addr":"172.21.15.194:6807/2800329805","heartbeat_front_addr":"172.21.15.194:6805/2800329805","state":["exists","up"]},{"osd":2,"uuid":"3a718da4-2511-4ec1-b1a8-eafd6cce0df0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6801","nonce":3935874719}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6803","nonce":3935874719}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6807","nonce":3935874719}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6805","nonce":3935874719}]},"public_addr":"172.21.15.196:6801/3935874719","cluster_addr":"172.21.15.196:6803/3935874719","heartbeat_back_addr":"172.21.15.196:6807/3935874719","heartbeat_front_addr":"172.21.15.196:6805/3935874719","state":["exists","up"]},{"osd":3,"uuid":"6f9c37f9-0ab9-4038-8919-afd4b39c4165","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6811","nonce":893345939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6813","nonce":893345939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6817","nonce":893345939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6815","nonce":893345939}]},"public_addr":"172.21.15.39:6811/893345939","cluster_addr":"172.21.15.39:6813/893345939","heartbeat_back_addr":"172.21.15.39:6817/893345939","heartbeat_front_addr":"172.21.15.39:6815/893345939","state":["exists","up"]},{"osd":4,"uuid":"e795bb0e-4791-4537-acda-1a321e19bc9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6809","nonce":800423088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6811","nonce":800423088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6815","nonce":800423088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6813","nonce":800423088}]},"public_addr":"172.21.15.194:6809/800423088","cluster_addr":"172.21.15.194:6811/800423088","heartbeat_back_addr":"172.21.15.194:6815/800423088","heartbeat_front_addr":"172.21.15.194:6813/800423088","state":["exists","up"]},{"osd":5,"uuid":"026e175e-6acb-4b4c-b235-8ccee8ae1149","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6809","nonce":2746224525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6811","nonce":2746224525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6815","nonce":2746224525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6813","nonce":2746224525}]},"public_addr":"172.21.15.196:6809/2746224525","cluster_addr":"172.21.15.196:6811/2746224525","heartbeat_back_addr":"172.21.15.196:6815/2746224525","heartbeat_front_addr":"172.21.15.196:6813/2746224525","state":["exists","up"]},{"osd":6,"uuid":"c1b183ff-383f-4c5c-9075-f76bab1c0bb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6819","nonce":1213371331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6821","nonce":1213371331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6825","nonce":1213371331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6823","nonce":1213371331}]},"public_addr":"172.21.15.39:6819/1213371331","cluster_addr":"172.21.15.39:6821/1213371331","heartbeat_back_addr":"172.21.15.39:6825/1213371331","heartbeat_front_addr":"172.21.15.39:6823/1213371331","state":["exists","up"]},{"osd":7,"uuid":"612f8697-957d-4ef4-a87f-241b36f9d13d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6817","nonce":3299162444}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6819","nonce":3299162444}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6823","nonce":3299162444}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6821","nonce":3299162444}]},"public_addr":"172.21.15.194:6817/3299162444","cluster_addr":"172.21.15.194:6819/3299162444","heartbeat_back_addr":"172.21.15.194:6823/3299162444","heartbeat_front_addr":"172.21.15.194:6821/3299162444","state":["exists","up"]},{"osd":8,"uuid":"836a9107-4d4f-4594-b8dc-0301e64f59d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6817","nonce":2456143145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6819","nonce":2456143145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6823","nonce":2456143145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6821","nonce":2456143145}]},"public_addr":"172.21.15.196:6817/2456143145","cluster_addr":"172.21.15.196:6819/2456143145","heartbeat_back_addr":"172.21.15.196:6823/2456143145","heartbeat_front_addr":"172.21.15.196:6821/2456143145","state":["exists","up"]},{"osd":9,"uuid":"4f30b960-12e3-4b16-bd28-fcc988139b58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6827","nonce":1848634433}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6829","nonce":1848634433}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6833","nonce":1848634433}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6831","nonce":1848634433}]},"public_addr":"172.21.15.39:6827/1848634433","cluster_addr":"172.21.15.39:6829/1848634433","heartbeat_back_addr":"172.21.15.39:6833/1848634433","heartbeat_front_addr":"172.21.15.39:6831/1848634433","state":["exists","up"]},{"osd":10,"uuid":"620f90e8-23e6-4bc0-8269-7219b9e1aebd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6825","nonce":3725064809}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6827","nonce":3725064809}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6831","nonce":3725064809}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6829","nonce":3725064809}]},"public_addr":"172.21.15.194:6825/3725064809","cluster_addr":"172.21.15.194:6827/3725064809","heartbeat_back_addr":"172.21.15.194:6831/3725064809","heartbeat_front_addr":"172.21.15.194:6829/3725064809","state":["exists","up"]},{"osd":11,"uuid":"867a2535-8235-4ceb-a698-eb86869d6d20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6825","nonce":2501212422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6827","nonce":2501212422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6831","nonce":2501212422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6829","nonce":2501212422}]},"public_addr":"172.21.15.196:6825/2501212422","cluster_addr":"172.21.15.196:6827/2501212422","heartbeat_back_addr":"172.21.15.196:6831/2501212422","heartbeat_front_addr":"172.21.15.196:6829/2501212422","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:00:36.931566+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:01:02.614024+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:01:26.193265+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:01:58.308833+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:02:25.049546+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:02:49.748750+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:03:23.582153+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:03:50.447321+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:04:15.812129+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:04:49.998047+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:05:17.087924+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:05:43.167637+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":4,"to":7}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.a","mappings":[{"from":5,"to":8}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.39:0/3482086140":"2024-03-19T15:58:45.340294+0000","172.21.15.39:0/3844306284":"2024-03-19T15:58:45.340294+0000","172.21.15.39:6801/1810238546":"2024-03-19T15:58:45.340294+0000","172.21.15.39:6800/1810238546":"2024-03-19T15:58:45.340294+0000","172.21.15.39:0/2528397488":"2024-03-19T15:58:10.264798+0000","172.21.15.39:0/2240981254":"2024-03-19T15:58:10.264798+0000","172.21.15.39:6800/1217503189":"2024-03-19T15:58:10.264798+0000","172.21.15.39:0/3754876370":"2024-03-19T15:58:10.264798+0000","172.21.15.39:6801/1217503189":"2024-03-19T15:58:10.264798+0000","172.21.15.39:0/1874637628":"2024-03-19T15:58:45.340294+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T16:09:42.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.173+0000 7fdad4a00640 1 -- 172.21.15.39:0/335401520 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fdab40a2880 msgr2=0x7fdab40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:42.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.173+0000 7fdad4a00640 1 --2- 172.21.15.39:0/335401520 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fdab40a2880 0x7fdab40a4d40 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fdab80095f0 tx=0x7fdab8005f90 comp rx=0 tx=0).stop 2024-03-18T16:09:42.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.173+0000 7fdad4a00640 1 -- 172.21.15.39:0/335401520 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdad00ffe60 msgr2=0x7fdad0199a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:42.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.173+0000 7fdad4a00640 1 --2- 172.21.15.39:0/335401520 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdad00ffe60 0x7fdad0199a90 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fdac002c8f0 tx=0x7fdac002c660 comp rx=0 tx=0).stop 2024-03-18T16:09:42.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.174+0000 7fdad4a00640 1 -- 172.21.15.39:0/335401520 shutdown_connections 2024-03-18T16:09:42.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.174+0000 7fdad4a00640 1 --2- 172.21.15.39:0/335401520 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdad0101160 0x7fdad01a1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:42.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.174+0000 7fdad4a00640 1 --2- 172.21.15.39:0/335401520 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fdab40a2880 0x7fdab40a4d40 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:42.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.174+0000 7fdad4a00640 1 --2- 172.21.15.39:0/335401520 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdad01007c0 0x7fdad0199fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:42.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.174+0000 7fdad4a00640 1 --2- 172.21.15.39:0/335401520 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdad00ffe60 0x7fdad0199a90 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:42.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.174+0000 7fdad4a00640 1 -- 172.21.15.39:0/335401520 >> 172.21.15.39:0/335401520 conn(0x7fdad00fba60 msgr2=0x7fdad00fd0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:42.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.174+0000 7fdad4a00640 1 -- 172.21.15.39:0/335401520 shutdown_connections 2024-03-18T16:09:42.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.174+0000 7fdad4a00640 1 -- 172.21.15.39:0/335401520 wait complete. 2024-03-18T16:09:42.198 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T16:01:30.048247+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T16:06:42.191791+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T16:06:43.670325+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T16:06:47.523388+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T16:09:42.199 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T16:09:42.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.414+0000 7fdee0e00640 1 -- 172.21.15.39:0/1607395808 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdedc1029c0 msgr2=0x7fdedc104e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:42.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.414+0000 7fdee0e00640 1 --2- 172.21.15.39:0/1607395808 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdedc1029c0 0x7fdedc104e30 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fdec8009900 tx=0x7fdec802bc30 comp rx=0 tx=0).stop 2024-03-18T16:09:42.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.414+0000 7fdee0e00640 1 -- 172.21.15.39:0/1607395808 shutdown_connections 2024-03-18T16:09:42.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.414+0000 7fdee0e00640 1 --2- 172.21.15.39:0/1607395808 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdedc105370 0x7fdedc1057f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:42.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.414+0000 7fdee0e00640 1 --2- 172.21.15.39:0/1607395808 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdedc1029c0 0x7fdedc104e30 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:42.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.414+0000 7fdee0e00640 1 --2- 172.21.15.39:0/1607395808 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdedc100050 0x7fdedc102480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:42.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.414+0000 7fdee0e00640 1 -- 172.21.15.39:0/1607395808 >> 172.21.15.39:0/1607395808 conn(0x7fdedc0fbc50 msgr2=0x7fdedc0fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:42.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.414+0000 7fdee0e00640 1 -- 172.21.15.39:0/1607395808 shutdown_connections 2024-03-18T16:09:42.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.414+0000 7fdee0e00640 1 -- 172.21.15.39:0/1607395808 wait complete. 2024-03-18T16:09:42.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.415+0000 7fdee0e00640 1 Processor -- start 2024-03-18T16:09:42.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.416+0000 7fdee0e00640 1 -- start start 2024-03-18T16:09:42.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.416+0000 7fdee0e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdedc100050 0x7fdedc19e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:42.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.416+0000 7fdee0e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdedc1029c0 0x7fdedc19e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:42.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.416+0000 7fdee0e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdedc105370 0x7fdedc1a56b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:42.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.416+0000 7fdee0e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fdedc113a30 con 0x7fdedc100050 2024-03-18T16:09:42.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.416+0000 7fdee0e00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fdedc1138b0 con 0x7fdedc105370 2024-03-18T16:09:42.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.416+0000 7fdee0e00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fdedc113bb0 con 0x7fdedc1029c0 2024-03-18T16:09:42.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.416+0000 7fded9400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdedc1029c0 0x7fdedc19e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:42.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.416+0000 7fded9e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdedc100050 0x7fdedc19e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:42.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.416+0000 7fded9400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdedc1029c0 0x7fdedc19e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:56032/0 (socket says 172.21.15.39:56032) 2024-03-18T16:09:42.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.416+0000 7fded9400640 1 -- 172.21.15.39:0/3957238023 learned_addr learned my addr 172.21.15.39:0/3957238023 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:42.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.416+0000 7fded9e00640 1 --2- 172.21.15.39:0/3957238023 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdedc100050 0x7fdedc19e100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:37282/0 (socket says 172.21.15.39:37282) 2024-03-18T16:09:42.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.416+0000 7fdeda800640 1 --2- 172.21.15.39:0/3957238023 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdedc105370 0x7fdedc1a56b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:42.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.417+0000 7fded9400640 1 -- 172.21.15.39:0/3957238023 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdedc105370 msgr2=0x7fdedc1a56b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:42.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.417+0000 7fded9400640 1 --2- 172.21.15.39:0/3957238023 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdedc105370 0x7fdedc1a56b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:42.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.417+0000 7fded9400640 1 -- 172.21.15.39:0/3957238023 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdedc100050 msgr2=0x7fdedc19e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:42.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.417+0000 7fded9400640 1 --2- 172.21.15.39:0/3957238023 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdedc100050 0x7fdedc19e100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:42.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.417+0000 7fded9400640 1 -- 172.21.15.39:0/3957238023 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdedc1a5d20 con 0x7fdedc1029c0 2024-03-18T16:09:42.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.417+0000 7fded9e00640 1 --2- 172.21.15.39:0/3957238023 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdedc100050 0x7fdedc19e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:09:42.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.417+0000 7fdeda800640 1 --2- 172.21.15.39:0/3957238023 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdedc105370 0x7fdedc1a56b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:09:42.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.417+0000 7fded9400640 1 --2- 172.21.15.39:0/3957238023 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdedc1029c0 0x7fdedc19e640 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fdec8009230 tx=0x7fdec802c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:42.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.417+0000 7fded2a00640 1 -- 172.21.15.39:0/3957238023 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdec8041070 con 0x7fdedc1029c0 2024-03-18T16:09:42.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.418+0000 7fded2a00640 1 -- 172.21.15.39:0/3957238023 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdec8004450 con 0x7fdedc1029c0 2024-03-18T16:09:42.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.418+0000 7fded2a00640 1 -- 172.21.15.39:0/3957238023 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdec8003e20 con 0x7fdedc1029c0 2024-03-18T16:09:42.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.418+0000 7fdee0e00640 1 -- 172.21.15.39:0/3957238023 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdedc1a5fb0 con 0x7fdedc1029c0 2024-03-18T16:09:42.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.418+0000 7fdee0e00640 1 -- 172.21.15.39:0/3957238023 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdedc1a63c0 con 0x7fdedc1029c0 2024-03-18T16:09:42.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.419+0000 7fdee0e00640 1 -- 172.21.15.39:0/3957238023 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdedc0fd740 con 0x7fdedc1029c0 2024-03-18T16:09:42.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.421+0000 7fded2a00640 1 -- 172.21.15.39:0/3957238023 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fdec8044070 con 0x7fdedc1029c0 2024-03-18T16:09:42.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.422+0000 7fded2a00640 1 --2- 172.21.15.39:0/3957238023 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fdec00a2880 0x7fdec00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:42.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.422+0000 7fded2a00640 1 -- 172.21.15.39:0/3957238023 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fdec80f0c80 con 0x7fdedc1029c0 2024-03-18T16:09:42.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.423+0000 7fded9e00640 1 --2- 172.21.15.39:0/3957238023 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fdec00a2880 0x7fdec00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:42.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.423+0000 7fded9e00640 1 --2- 172.21.15.39:0/3957238023 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fdec00a2880 0x7fdec00a4d40 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fdecc009fc0 tx=0x7fdecc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:42.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.428+0000 7fded2a00640 1 -- 172.21.15.39:0/3957238023 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fdec80bd2e0 con 0x7fdedc1029c0 2024-03-18T16:09:42.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:42 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/119693489' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T16:09:42.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:42 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/68684024' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:09:42.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:42 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/119693489' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T16:09:42.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:42 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/68684024' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:09:42.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:42 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/119693489' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T16:09:42.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:42 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/68684024' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:09:42.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.671+0000 7fdee0e00640 1 -- 172.21.15.39:0/3957238023 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fdedc02d340 con 0x7fdedc1029c0 2024-03-18T16:09:42.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.671+0000 7fded2a00640 1 -- 172.21.15.39:0/3957238023 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7fdec80c1380 con 0x7fdedc1029c0 2024-03-18T16:09:42.674 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2024-03-18T16:09:42.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.674+0000 7fdee0e00640 1 -- 172.21.15.39:0/3957238023 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fdec00a2880 msgr2=0x7fdec00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:42.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.674+0000 7fdee0e00640 1 --2- 172.21.15.39:0/3957238023 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fdec00a2880 0x7fdec00a4d40 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fdecc009fc0 tx=0x7fdecc005f90 comp rx=0 tx=0).stop 2024-03-18T16:09:42.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.674+0000 7fdee0e00640 1 -- 172.21.15.39:0/3957238023 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdedc1029c0 msgr2=0x7fdedc19e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:42.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.674+0000 7fdee0e00640 1 --2- 172.21.15.39:0/3957238023 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdedc1029c0 0x7fdedc19e640 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fdec8009230 tx=0x7fdec802c650 comp rx=0 tx=0).stop 2024-03-18T16:09:42.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.674+0000 7fdee0e00640 1 -- 172.21.15.39:0/3957238023 shutdown_connections 2024-03-18T16:09:42.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.674+0000 7fdee0e00640 1 --2- 172.21.15.39:0/3957238023 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdedc105370 0x7fdedc1a56b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:42.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.674+0000 7fdee0e00640 1 --2- 172.21.15.39:0/3957238023 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fdec00a2880 0x7fdec00a4d40 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:42.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.674+0000 7fdee0e00640 1 --2- 172.21.15.39:0/3957238023 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdedc1029c0 0x7fdedc19e640 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:42.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.674+0000 7fdee0e00640 1 --2- 172.21.15.39:0/3957238023 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdedc100050 0x7fdedc19e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:42.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.674+0000 7fdee0e00640 1 -- 172.21.15.39:0/3957238023 >> 172.21.15.39:0/3957238023 conn(0x7fdedc0fbc50 msgr2=0x7fdedc1009d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:42.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.674+0000 7fdee0e00640 1 -- 172.21.15.39:0/3957238023 shutdown_connections 2024-03-18T16:09:42.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.674+0000 7fdee0e00640 1 -- 172.21.15.39:0/3957238023 wait complete. 2024-03-18T16:09:42.697 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T16:09:42.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.912+0000 7f52fec00640 1 -- 172.21.15.39:0/994861237 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f52f8104c10 msgr2=0x7f52f8105090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:42.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.912+0000 7f52fec00640 1 --2- 172.21.15.39:0/994861237 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f52f8104c10 0x7f52f8105090 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f52e0009900 tx=0x7f52e002bc50 comp rx=0 tx=0).stop 2024-03-18T16:09:42.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.912+0000 7f52fec00640 1 -- 172.21.15.39:0/994861237 shutdown_connections 2024-03-18T16:09:42.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.912+0000 7f52fec00640 1 --2- 172.21.15.39:0/994861237 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f52f8104c10 0x7f52f8105090 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:42.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.912+0000 7f52fec00640 1 --2- 172.21.15.39:0/994861237 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f52f81055d0 0x7f52f810be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:42.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.912+0000 7f52fec00640 1 --2- 172.21.15.39:0/994861237 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f52f8103a20 0x7f52f8103e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:42.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.912+0000 7f52fec00640 1 -- 172.21.15.39:0/994861237 >> 172.21.15.39:0/994861237 conn(0x7f52f80fef90 msgr2=0x7f52f81013d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:42.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.912+0000 7f52fec00640 1 -- 172.21.15.39:0/994861237 shutdown_connections 2024-03-18T16:09:42.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.912+0000 7f52fec00640 1 -- 172.21.15.39:0/994861237 wait complete. 2024-03-18T16:09:42.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.914+0000 7f52fec00640 1 Processor -- start 2024-03-18T16:09:42.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.914+0000 7f52fec00640 1 -- start start 2024-03-18T16:09:42.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.915+0000 7f52fec00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f52f8103a20 0x7f52f8199da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:42.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.915+0000 7f52fec00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f52f8104c10 0x7f52f819a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:42.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.915+0000 7f52fec00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f52f81055d0 0x7f52f81a1310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:42.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.915+0000 7f52fec00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f52f810f6d0 con 0x7f52f8103a20 2024-03-18T16:09:42.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.915+0000 7f52fec00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f52f810f550 con 0x7f52f81055d0 2024-03-18T16:09:42.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.915+0000 7f52fec00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f52f810f850 con 0x7f52f8104c10 2024-03-18T16:09:42.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.915+0000 7f52f7e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f52f8103a20 0x7f52f8199da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:42.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.915+0000 7f52efe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f52f8104c10 0x7f52f819a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:42.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.915+0000 7f52efe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f52f8104c10 0x7f52f819a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:56048/0 (socket says 172.21.15.39:56048) 2024-03-18T16:09:42.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.915+0000 7f52f7e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f52f8103a20 0x7f52f8199da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:37302/0 (socket says 172.21.15.39:37302) 2024-03-18T16:09:42.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.915+0000 7f52efe00640 1 -- 172.21.15.39:0/2124081197 learned_addr learned my addr 172.21.15.39:0/2124081197 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:42.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.915+0000 7f52fcc00640 1 --2- 172.21.15.39:0/2124081197 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f52f81055d0 0x7f52f81a1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:42.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.916+0000 7f52efe00640 1 -- 172.21.15.39:0/2124081197 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f52f81055d0 msgr2=0x7f52f81a1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:42.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.916+0000 7f52efe00640 1 --2- 172.21.15.39:0/2124081197 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f52f81055d0 0x7f52f81a1310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:42.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.916+0000 7f52efe00640 1 -- 172.21.15.39:0/2124081197 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f52f8103a20 msgr2=0x7f52f8199da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:42.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.916+0000 7f52efe00640 1 --2- 172.21.15.39:0/2124081197 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f52f8103a20 0x7f52f8199da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:42.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.916+0000 7f52efe00640 1 -- 172.21.15.39:0/2124081197 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52f81a19b0 con 0x7f52f8104c10 2024-03-18T16:09:42.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.916+0000 7f52f7e00640 1 --2- 172.21.15.39:0/2124081197 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f52f8103a20 0x7f52f8199da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:09:42.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.916+0000 7f52efe00640 1 --2- 172.21.15.39:0/2124081197 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f52f8104c10 0x7f52f819a2e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f52e0000c00 tx=0x7f52e002c670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:42.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.917+0000 7f52f5600640 1 -- 172.21.15.39:0/2124081197 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52e0041070 con 0x7f52f8104c10 2024-03-18T16:09:42.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.917+0000 7f52f5600640 1 -- 172.21.15.39:0/2124081197 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f52e0004450 con 0x7f52f8104c10 2024-03-18T16:09:42.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.917+0000 7f52fec00640 1 -- 172.21.15.39:0/2124081197 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52f81a1c40 con 0x7f52f8104c10 2024-03-18T16:09:42.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.917+0000 7f52f5600640 1 -- 172.21.15.39:0/2124081197 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52e0034440 con 0x7f52f8104c10 2024-03-18T16:09:42.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.917+0000 7f52fec00640 1 -- 172.21.15.39:0/2124081197 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52f81a2190 con 0x7f52f8104c10 2024-03-18T16:09:42.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.918+0000 7f52fec00640 1 -- 172.21.15.39:0/2124081197 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52f810ace0 con 0x7f52f8104c10 2024-03-18T16:09:42.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.920+0000 7f52f5600640 1 -- 172.21.15.39:0/2124081197 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f52e0044070 con 0x7f52f8104c10 2024-03-18T16:09:42.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.921+0000 7f52f5600640 1 --2- 172.21.15.39:0/2124081197 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f52dc0a2880 0x7f52dc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:42.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.921+0000 7f52f5600640 1 -- 172.21.15.39:0/2124081197 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f52e00f0cc0 con 0x7f52f8104c10 2024-03-18T16:09:42.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.922+0000 7f52f7e00640 1 --2- 172.21.15.39:0/2124081197 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f52dc0a2880 0x7f52dc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:42.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.922+0000 7f52f7e00640 1 --2- 172.21.15.39:0/2124081197 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f52dc0a2880 0x7f52dc0a4d40 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f52f819b270 tx=0x7f52e8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:42.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:42.923+0000 7f52f5600640 1 -- 172.21.15.39:0/2124081197 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f52e00bd320 con 0x7f52f8104c10 2024-03-18T16:09:43.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.125+0000 7f52fec00640 1 -- 172.21.15.39:0/2124081197 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f52f802d340 con 0x7f52f8104c10 2024-03-18T16:09:43.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.126+0000 7f52f5600640 1 -- 172.21.15.39:0/2124081197 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7f52e00c13c0 con 0x7f52f8104c10 2024-03-18T16:09:43.128 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 32 2024-03-18T16:09:43.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.129+0000 7f52fec00640 1 -- 172.21.15.39:0/2124081197 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f52dc0a2880 msgr2=0x7f52dc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:43.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.129+0000 7f52fec00640 1 --2- 172.21.15.39:0/2124081197 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f52dc0a2880 0x7f52dc0a4d40 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f52f819b270 tx=0x7f52e8005f90 comp rx=0 tx=0).stop 2024-03-18T16:09:43.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.129+0000 7f52fec00640 1 -- 172.21.15.39:0/2124081197 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f52f8104c10 msgr2=0x7f52f819a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:43.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.129+0000 7f52fec00640 1 --2- 172.21.15.39:0/2124081197 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f52f8104c10 0x7f52f819a2e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f52e0000c00 tx=0x7f52e002c670 comp rx=0 tx=0).stop 2024-03-18T16:09:43.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.129+0000 7f52fec00640 1 -- 172.21.15.39:0/2124081197 shutdown_connections 2024-03-18T16:09:43.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.129+0000 7f52fec00640 1 --2- 172.21.15.39:0/2124081197 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f52f81055d0 0x7f52f81a1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:43.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.129+0000 7f52fec00640 1 --2- 172.21.15.39:0/2124081197 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f52dc0a2880 0x7f52dc0a4d40 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:43.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.129+0000 7f52fec00640 1 --2- 172.21.15.39:0/2124081197 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f52f8104c10 0x7f52f819a2e0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:43.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.129+0000 7f52fec00640 1 --2- 172.21.15.39:0/2124081197 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f52f8103a20 0x7f52f8199da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:43.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.129+0000 7f52fec00640 1 -- 172.21.15.39:0/2124081197 >> 172.21.15.39:0/2124081197 conn(0x7f52f80fef90 msgr2=0x7f52f8107eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:43.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.129+0000 7f52fec00640 1 -- 172.21.15.39:0/2124081197 shutdown_connections 2024-03-18T16:09:43.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.129+0000 7f52fec00640 1 -- 172.21.15.39:0/2124081197 wait complete. 2024-03-18T16:09:43.150 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T16:09:43.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.363+0000 7ff56d800640 1 -- 172.21.15.39:0/3628740596 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff568101070 msgr2=0x7ff5681018f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:43.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.363+0000 7ff56d800640 1 --2- 172.21.15.39:0/3628740596 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff568101070 0x7ff5681018f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ff560009900 tx=0x7ff56002bc40 comp rx=0 tx=0).stop 2024-03-18T16:09:43.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.363+0000 7ff56d800640 1 -- 172.21.15.39:0/3628740596 shutdown_connections 2024-03-18T16:09:43.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.363+0000 7ff56d800640 1 --2- 172.21.15.39:0/3628740596 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff5681006b0 0x7ff568100b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:43.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.363+0000 7ff56d800640 1 --2- 172.21.15.39:0/3628740596 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff568101070 0x7ff5681018f0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:43.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.363+0000 7ff56d800640 1 --2- 172.21.15.39:0/3628740596 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff5680ff600 0x7ff5680ffa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:43.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.363+0000 7ff56d800640 1 -- 172.21.15.39:0/3628740596 >> 172.21.15.39:0/3628740596 conn(0x7ff5680fae60 msgr2=0x7ff5680fd2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:43.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.363+0000 7ff56d800640 1 -- 172.21.15.39:0/3628740596 shutdown_connections 2024-03-18T16:09:43.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.363+0000 7ff56d800640 1 -- 172.21.15.39:0/3628740596 wait complete. 2024-03-18T16:09:43.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.364+0000 7ff56d800640 1 Processor -- start 2024-03-18T16:09:43.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.365+0000 7ff56d800640 1 -- start start 2024-03-18T16:09:43.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.365+0000 7ff56d800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff5680ff600 0x7ff568199b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:43.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.365+0000 7ff56d800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff5681006b0 0x7ff56819a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:43.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.365+0000 7ff56d800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff568101070 0x7ff5681a1100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:43.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.365+0000 7ff56d800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7ff56810f590 con 0x7ff568101070 2024-03-18T16:09:43.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.365+0000 7ff56d800640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ff56810f410 con 0x7ff5680ff600 2024-03-18T16:09:43.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.365+0000 7ff56d800640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7ff56810f710 con 0x7ff5681006b0 2024-03-18T16:09:43.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.365+0000 7ff567e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff5680ff600 0x7ff568199b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:43.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.365+0000 7ff567e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff5680ff600 0x7ff568199b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:38622/0 (socket says 172.21.15.39:38622) 2024-03-18T16:09:43.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.365+0000 7ff567e00640 1 -- 172.21.15.39:0/1961811453 learned_addr learned my addr 172.21.15.39:0/1961811453 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:43.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.365+0000 7ff55f400640 1 --2- 172.21.15.39:0/1961811453 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff5681006b0 0x7ff56819a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:43.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.365+0000 7ff56ce00640 1 --2- 172.21.15.39:0/1961811453 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff568101070 0x7ff5681a1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:43.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.366+0000 7ff567e00640 1 -- 172.21.15.39:0/1961811453 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff5681006b0 msgr2=0x7ff56819a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:43.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.366+0000 7ff567e00640 1 --2- 172.21.15.39:0/1961811453 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff5681006b0 0x7ff56819a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:43.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.366+0000 7ff567e00640 1 -- 172.21.15.39:0/1961811453 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff568101070 msgr2=0x7ff5681a1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:43.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.366+0000 7ff567e00640 1 --2- 172.21.15.39:0/1961811453 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff568101070 0x7ff5681a1100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:43.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.366+0000 7ff567e00640 1 -- 172.21.15.39:0/1961811453 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5681a1770 con 0x7ff5680ff600 2024-03-18T16:09:43.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.366+0000 7ff56ce00640 1 --2- 172.21.15.39:0/1961811453 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff568101070 0x7ff5681a1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:09:43.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.366+0000 7ff567e00640 1 --2- 172.21.15.39:0/1961811453 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff5680ff600 0x7ff568199b50 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7ff55800ebc0 tx=0x7ff55800c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:43.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.366+0000 7ff565600640 1 -- 172.21.15.39:0/1961811453 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff558003b70 con 0x7ff5680ff600 2024-03-18T16:09:43.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.366+0000 7ff565600640 1 -- 172.21.15.39:0/1961811453 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff558003d10 con 0x7ff5680ff600 2024-03-18T16:09:43.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.366+0000 7ff56d800640 1 -- 172.21.15.39:0/1961811453 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5681a1a60 con 0x7ff5680ff600 2024-03-18T16:09:43.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.366+0000 7ff565600640 1 -- 172.21.15.39:0/1961811453 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff558019980 con 0x7ff5680ff600 2024-03-18T16:09:43.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.367+0000 7ff56d800640 1 -- 172.21.15.39:0/1961811453 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5681a2000 con 0x7ff5680ff600 2024-03-18T16:09:43.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.368+0000 7ff56d800640 1 -- 172.21.15.39:0/1961811453 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff56810ba60 con 0x7ff5680ff600 2024-03-18T16:09:43.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.371+0000 7ff565600640 1 -- 172.21.15.39:0/1961811453 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7ff558019b20 con 0x7ff5680ff600 2024-03-18T16:09:43.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.372+0000 7ff565600640 1 --2- 172.21.15.39:0/1961811453 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff54c0a2880 0x7ff54c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:43.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.373+0000 7ff565600640 1 -- 172.21.15.39:0/1961811453 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7ff5580cf600 con 0x7ff5680ff600 2024-03-18T16:09:43.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.373+0000 7ff55f400640 1 --2- 172.21.15.39:0/1961811453 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff54c0a2880 0x7ff54c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:43.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.373+0000 7ff55f400640 1 --2- 172.21.15.39:0/1961811453 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff54c0a2880 0x7ff54c0a4d40 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7ff5500095f0 tx=0x7ff550005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:43.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.379+0000 7ff565600640 1 -- 172.21.15.39:0/1961811453 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff55809c310 con 0x7ff5680ff600 2024-03-18T16:09:43.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:43 smithi196 ceph-mon[36307]: pgmap v411: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-18T16:09:43.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:43 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/335401520' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:09:43.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:43 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/3957238023' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T16:09:43.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:43 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/2124081197' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T16:09:43.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:43 smithi194 ceph-mon[36200]: pgmap v411: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-18T16:09:43.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:43 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/335401520' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:09:43.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:43 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/3957238023' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T16:09:43.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:43 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/2124081197' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T16:09:43.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:43 smithi039 ceph-mon[31721]: pgmap v411: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-18T16:09:43.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:43 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/335401520' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:09:43.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:43 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/3957238023' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T16:09:43.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:43 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/2124081197' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T16:09:43.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.588+0000 7ff56d800640 1 -- 172.21.15.39:0/1961811453 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7ff56802d1b0 con 0x7ff5680ff600 2024-03-18T16:09:43.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.589+0000 7ff565600640 1 -- 172.21.15.39:0/1961811453 <== mon.2 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7ff56802d1b0 con 0x7ff5680ff600 2024-03-18T16:09:43.591 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 64 2024-03-18T16:09:43.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.591+0000 7ff56d800640 1 -- 172.21.15.39:0/1961811453 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff54c0a2880 msgr2=0x7ff54c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:43.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.591+0000 7ff56d800640 1 --2- 172.21.15.39:0/1961811453 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff54c0a2880 0x7ff54c0a4d40 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7ff5500095f0 tx=0x7ff550005f90 comp rx=0 tx=0).stop 2024-03-18T16:09:43.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.591+0000 7ff56d800640 1 -- 172.21.15.39:0/1961811453 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff5680ff600 msgr2=0x7ff568199b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:43.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.591+0000 7ff56d800640 1 --2- 172.21.15.39:0/1961811453 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff5680ff600 0x7ff568199b50 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7ff55800ebc0 tx=0x7ff55800c6a0 comp rx=0 tx=0).stop 2024-03-18T16:09:43.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.591+0000 7ff56d800640 1 -- 172.21.15.39:0/1961811453 shutdown_connections 2024-03-18T16:09:43.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.591+0000 7ff56d800640 1 --2- 172.21.15.39:0/1961811453 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff54c0a2880 0x7ff54c0a4d40 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:43.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.591+0000 7ff56d800640 1 --2- 172.21.15.39:0/1961811453 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff568101070 0x7ff5681a1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:43.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.591+0000 7ff56d800640 1 --2- 172.21.15.39:0/1961811453 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff5681006b0 0x7ff56819a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:43.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.591+0000 7ff56d800640 1 --2- 172.21.15.39:0/1961811453 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff5680ff600 0x7ff568199b50 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:43.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.591+0000 7ff56d800640 1 -- 172.21.15.39:0/1961811453 >> 172.21.15.39:0/1961811453 conn(0x7ff5680fae60 msgr2=0x7ff5680fc500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:43.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.592+0000 7ff56d800640 1 -- 172.21.15.39:0/1961811453 shutdown_connections 2024-03-18T16:09:43.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.592+0000 7ff56d800640 1 -- 172.21.15.39:0/1961811453 wait complete. 2024-03-18T16:09:43.614 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T16:09:43.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.826+0000 7fe3c9600640 1 -- 172.21.15.39:0/3783043615 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe3c41006f0 msgr2=0x7fe3c4100b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:43.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.826+0000 7fe3c9600640 1 --2- 172.21.15.39:0/3783043615 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe3c41006f0 0x7fe3c4100b10 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fe3b4009900 tx=0x7fe3b402bc40 comp rx=0 tx=0).stop 2024-03-18T16:09:43.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.826+0000 7fe3c9600640 1 -- 172.21.15.39:0/3783043615 shutdown_connections 2024-03-18T16:09:43.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.826+0000 7fe3c9600640 1 --2- 172.21.15.39:0/3783043615 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe3c41022a0 0x7fe3c4108b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:43.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.826+0000 7fe3c9600640 1 --2- 172.21.15.39:0/3783043615 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe3c41018e0 0x7fe3c4101d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:43.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.826+0000 7fe3c9600640 1 --2- 172.21.15.39:0/3783043615 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe3c41006f0 0x7fe3c4100b10 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:43.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.827+0000 7fe3c9600640 1 -- 172.21.15.39:0/3783043615 >> 172.21.15.39:0/3783043615 conn(0x7fe3c40fbc80 msgr2=0x7fe3c40fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:43.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.827+0000 7fe3c9600640 1 -- 172.21.15.39:0/3783043615 shutdown_connections 2024-03-18T16:09:43.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.827+0000 7fe3c9600640 1 -- 172.21.15.39:0/3783043615 wait complete. 2024-03-18T16:09:43.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.828+0000 7fe3c9600640 1 Processor -- start 2024-03-18T16:09:43.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.828+0000 7fe3c9600640 1 -- start start 2024-03-18T16:09:43.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.829+0000 7fe3c9600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe3c41006f0 0x7fe3c4104680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:43.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.829+0000 7fe3c9600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe3c41018e0 0x7fe3c4102cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:43.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.829+0000 7fe3c9600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe3c41022a0 0x7fe3c4103200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:43.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.829+0000 7fe3c9600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fe3c410c3b0 con 0x7fe3c41022a0 2024-03-18T16:09:43.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.829+0000 7fe3c9600640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fe3c410c230 con 0x7fe3c41006f0 2024-03-18T16:09:43.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.829+0000 7fe3c9600640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fe3c410c530 con 0x7fe3c41018e0 2024-03-18T16:09:43.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.829+0000 7fe3c2000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe3c41018e0 0x7fe3c4102cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:43.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.829+0000 7fe3c3400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe3c41022a0 0x7fe3c4103200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:43.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.829+0000 7fe3c2a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe3c41006f0 0x7fe3c4104680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:43.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.829+0000 7fe3c2000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe3c41018e0 0x7fe3c4102cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:56092/0 (socket says 172.21.15.39:56092) 2024-03-18T16:09:43.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.829+0000 7fe3c2000640 1 -- 172.21.15.39:0/2657775920 learned_addr learned my addr 172.21.15.39:0/2657775920 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:43.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.830+0000 7fe3c2000640 1 -- 172.21.15.39:0/2657775920 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe3c41006f0 msgr2=0x7fe3c4104680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:43.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.830+0000 7fe3c2000640 1 --2- 172.21.15.39:0/2657775920 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe3c41006f0 0x7fe3c4104680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:43.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.830+0000 7fe3c2000640 1 -- 172.21.15.39:0/2657775920 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe3c41022a0 msgr2=0x7fe3c4103200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:43.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.830+0000 7fe3c2000640 1 --2- 172.21.15.39:0/2657775920 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe3c41022a0 0x7fe3c4103200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:43.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.830+0000 7fe3c2000640 1 -- 172.21.15.39:0/2657775920 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3c4103a90 con 0x7fe3c41018e0 2024-03-18T16:09:43.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.830+0000 7fe3c2a00640 1 --2- 172.21.15.39:0/2657775920 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe3c41006f0 0x7fe3c4104680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:09:43.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.830+0000 7fe3c2000640 1 --2- 172.21.15.39:0/2657775920 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe3c41018e0 0x7fe3c4102cc0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fe3b000e9f0 tx=0x7fe3b000edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:43.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.830+0000 7fe3bb400640 1 -- 172.21.15.39:0/2657775920 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3b000cd80 con 0x7fe3c41018e0 2024-03-18T16:09:43.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.830+0000 7fe3bb400640 1 -- 172.21.15.39:0/2657775920 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe3b0005500 con 0x7fe3c41018e0 2024-03-18T16:09:43.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.830+0000 7fe3bb400640 1 -- 172.21.15.39:0/2657775920 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3b0010690 con 0x7fe3c41018e0 2024-03-18T16:09:43.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.830+0000 7fe3c9600640 1 -- 172.21.15.39:0/2657775920 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3c41a1cb0 con 0x7fe3c41018e0 2024-03-18T16:09:43.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.830+0000 7fe3c9600640 1 -- 172.21.15.39:0/2657775920 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3c41a21b0 con 0x7fe3c41018e0 2024-03-18T16:09:43.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.831+0000 7fe3c9600640 1 -- 172.21.15.39:0/2657775920 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3c41a23e0 con 0x7fe3c41018e0 2024-03-18T16:09:43.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.835+0000 7fe3bb400640 1 -- 172.21.15.39:0/2657775920 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fe3b00046c0 con 0x7fe3c41018e0 2024-03-18T16:09:43.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.836+0000 7fe3bb400640 1 --2- 172.21.15.39:0/2657775920 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fe3a80a2800 0x7fe3a80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:43.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.836+0000 7fe3bb400640 1 -- 172.21.15.39:0/2657775920 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fe3b00cf270 con 0x7fe3c41018e0 2024-03-18T16:09:43.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.836+0000 7fe3c2a00640 1 --2- 172.21.15.39:0/2657775920 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fe3a80a2800 0x7fe3a80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:43.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.837+0000 7fe3c2a00640 1 --2- 172.21.15.39:0/2657775920 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fe3a80a2800 0x7fe3a80a4cc0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fe3b402c180 tx=0x7fe3b402cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:43.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:43.840+0000 7fe3bb400640 1 -- 172.21.15.39:0/2657775920 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe3b009bf80 con 0x7fe3c41018e0 2024-03-18T16:09:44.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.099+0000 7fe3c9600640 1 -- 172.21.15.39:0/2657775920 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fe3c41a2770 con 0x7fe3c41018e0 2024-03-18T16:09:44.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.100+0000 7fe3bb400640 1 -- 172.21.15.39:0/2657775920 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7fe3b009b880 con 0x7fe3c41018e0 2024-03-18T16:09:44.102 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 32 2024-03-18T16:09:44.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.103+0000 7fe3c9600640 1 -- 172.21.15.39:0/2657775920 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fe3a80a2800 msgr2=0x7fe3a80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:44.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.103+0000 7fe3c9600640 1 --2- 172.21.15.39:0/2657775920 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fe3a80a2800 0x7fe3a80a4cc0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fe3b402c180 tx=0x7fe3b402cf90 comp rx=0 tx=0).stop 2024-03-18T16:09:44.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.103+0000 7fe3c9600640 1 -- 172.21.15.39:0/2657775920 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe3c41018e0 msgr2=0x7fe3c4102cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:44.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.103+0000 7fe3c9600640 1 --2- 172.21.15.39:0/2657775920 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe3c41018e0 0x7fe3c4102cc0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fe3b000e9f0 tx=0x7fe3b000edf0 comp rx=0 tx=0).stop 2024-03-18T16:09:44.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.103+0000 7fe3c9600640 1 -- 172.21.15.39:0/2657775920 shutdown_connections 2024-03-18T16:09:44.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.103+0000 7fe3c9600640 1 --2- 172.21.15.39:0/2657775920 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fe3a80a2800 0x7fe3a80a4cc0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:44.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.103+0000 7fe3c9600640 1 --2- 172.21.15.39:0/2657775920 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe3c41022a0 0x7fe3c4103200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:44.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.103+0000 7fe3c9600640 1 --2- 172.21.15.39:0/2657775920 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe3c41018e0 0x7fe3c4102cc0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:44.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.103+0000 7fe3c9600640 1 --2- 172.21.15.39:0/2657775920 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe3c41006f0 0x7fe3c4104680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:44.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.103+0000 7fe3c9600640 1 -- 172.21.15.39:0/2657775920 >> 172.21.15.39:0/2657775920 conn(0x7fe3c40fbc80 msgr2=0x7fe3c40fd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:44.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.103+0000 7fe3c9600640 1 -- 172.21.15.39:0/2657775920 shutdown_connections 2024-03-18T16:09:44.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.103+0000 7fe3c9600640 1 -- 172.21.15.39:0/2657775920 wait complete. 2024-03-18T16:09:44.127 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:09:44.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.337+0000 7f0b21000640 1 -- 172.21.15.39:0/2058093757 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0b1c1049d0 msgr2=0x7f0b1c104e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:44.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.337+0000 7f0b21000640 1 --2- 172.21.15.39:0/2058093757 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0b1c1049d0 0x7f0b1c104e50 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f0b04009900 tx=0x7f0b0402bc40 comp rx=0 tx=0).stop 2024-03-18T16:09:44.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.337+0000 7f0b21000640 1 -- 172.21.15.39:0/2058093757 shutdown_connections 2024-03-18T16:09:44.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.337+0000 7f0b21000640 1 --2- 172.21.15.39:0/2058093757 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0b1c105390 0x7f0b1c10bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:44.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.337+0000 7f0b21000640 1 --2- 172.21.15.39:0/2058093757 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0b1c1049d0 0x7f0b1c104e50 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:44.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.337+0000 7f0b21000640 1 --2- 172.21.15.39:0/2058093757 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0b1c1037e0 0x7f0b1c103c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:44.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.337+0000 7f0b21000640 1 -- 172.21.15.39:0/2058093757 >> 172.21.15.39:0/2058093757 conn(0x7f0b1c0fed50 msgr2=0x7f0b1c101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:44.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.338+0000 7f0b21000640 1 -- 172.21.15.39:0/2058093757 shutdown_connections 2024-03-18T16:09:44.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.338+0000 7f0b21000640 1 -- 172.21.15.39:0/2058093757 wait complete. 2024-03-18T16:09:44.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.339+0000 7f0b21000640 1 Processor -- start 2024-03-18T16:09:44.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.339+0000 7f0b21000640 1 -- start start 2024-03-18T16:09:44.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.339+0000 7f0b21000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0b1c1037e0 0x7f0b1c199af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:44.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.339+0000 7f0b21000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0b1c1049d0 0x7f0b1c19a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:44.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.339+0000 7f0b21000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0b1c105390 0x7f0b1c1a10a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:44.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.339+0000 7f0b21000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f0b1c10f490 con 0x7f0b1c105390 2024-03-18T16:09:44.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.339+0000 7f0b21000640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f0b1c10f310 con 0x7f0b1c1049d0 2024-03-18T16:09:44.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.339+0000 7f0b21000640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f0b1c10f610 con 0x7f0b1c1037e0 2024-03-18T16:09:44.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.339+0000 7f0b1b400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0b1c1037e0 0x7f0b1c199af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:44.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.339+0000 7f0b1be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0b1c105390 0x7f0b1c1a10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:44.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.339+0000 7f0b1b400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0b1c1037e0 0x7f0b1c199af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:56102/0 (socket says 172.21.15.39:56102) 2024-03-18T16:09:44.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.339+0000 7f0b1b400640 1 -- 172.21.15.39:0/3934060100 learned_addr learned my addr 172.21.15.39:0/3934060100 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:44.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.340+0000 7f0b13e00640 1 --2- 172.21.15.39:0/3934060100 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0b1c1049d0 0x7f0b1c19a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:44.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.340+0000 7f0b1b400640 1 -- 172.21.15.39:0/3934060100 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0b1c1049d0 msgr2=0x7f0b1c19a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:44.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.340+0000 7f0b1b400640 1 --2- 172.21.15.39:0/3934060100 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0b1c1049d0 0x7f0b1c19a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:44.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.340+0000 7f0b1b400640 1 -- 172.21.15.39:0/3934060100 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0b1c105390 msgr2=0x7f0b1c1a10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:44.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.340+0000 7f0b1b400640 1 --2- 172.21.15.39:0/3934060100 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0b1c105390 0x7f0b1c1a10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:44.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.340+0000 7f0b1b400640 1 -- 172.21.15.39:0/3934060100 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b1c1a1710 con 0x7f0b1c1037e0 2024-03-18T16:09:44.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.340+0000 7f0b13e00640 1 --2- 172.21.15.39:0/3934060100 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0b1c1049d0 0x7f0b1c19a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:09:44.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.340+0000 7f0b1b400640 1 --2- 172.21.15.39:0/3934060100 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0b1c1037e0 0x7f0b1c199af0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c00d950 tx=0x7f0b0c00de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:44.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.341+0000 7f0b18c00640 1 -- 172.21.15.39:0/3934060100 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b0c017020 con 0x7f0b1c1037e0 2024-03-18T16:09:44.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.341+0000 7f0b18c00640 1 -- 172.21.15.39:0/3934060100 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0b0c009c30 con 0x7f0b1c1037e0 2024-03-18T16:09:44.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.341+0000 7f0b21000640 1 -- 172.21.15.39:0/3934060100 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b1c1a1a00 con 0x7f0b1c1037e0 2024-03-18T16:09:44.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.341+0000 7f0b18c00640 1 -- 172.21.15.39:0/3934060100 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b0c00f720 con 0x7f0b1c1037e0 2024-03-18T16:09:44.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.341+0000 7f0b21000640 1 -- 172.21.15.39:0/3934060100 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b1c1a1f20 con 0x7f0b1c1037e0 2024-03-18T16:09:44.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.342+0000 7f0b21000640 1 -- 172.21.15.39:0/3934060100 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b1c068940 con 0x7f0b1c1037e0 2024-03-18T16:09:44.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.343+0000 7f0b18c00640 1 -- 172.21.15.39:0/3934060100 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f0b0c01a020 con 0x7f0b1c1037e0 2024-03-18T16:09:44.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.345+0000 7f0b18c00640 1 --2- 172.21.15.39:0/3934060100 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f0b000a2800 0x7f0b000a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:44.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.345+0000 7f0b18c00640 1 -- 172.21.15.39:0/3934060100 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f0b0c0ced00 con 0x7f0b1c1037e0 2024-03-18T16:09:44.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.345+0000 7f0b13e00640 1 --2- 172.21.15.39:0/3934060100 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f0b000a2800 0x7f0b000a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:44.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.346+0000 7f0b13e00640 1 --2- 172.21.15.39:0/3934060100 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f0b000a2800 0x7f0b000a4cc0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f0b0402efd0 tx=0x7f0b0402cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:44.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.349+0000 7f0b18c00640 1 -- 172.21.15.39:0/3934060100 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f0b0c021020 con 0x7f0b1c1037e0 2024-03-18T16:09:44.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:44 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/1961811453' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T16:09:44.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:44 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/2657775920' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T16:09:44.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:44 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/1961811453' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T16:09:44.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:44 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/2657775920' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T16:09:44.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:44 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/1961811453' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T16:09:44.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:44 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/2657775920' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T16:09:44.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.612+0000 7f0b21000640 1 -- 172.21.15.39:0/3934060100 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0b1c0623e0 con 0x7f0b1c1037e0 2024-03-18T16:09:44.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.613+0000 7f0b18c00640 1 -- 172.21.15.39:0/3934060100 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11932 (secure 0 0 0) 0x7f0b0c009de0 con 0x7f0b1c1037e0 2024-03-18T16:09:44.615 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:09:44.615 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2520226608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2520226608},{"type":"v1","addr":"172.21.15.39:6835","nonce":2520226608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/719341763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":719341763},{"type":"v1","addr":"172.21.15.39:6837","nonce":719341763}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T16:06:44.180425+0000","modified":"2024-03-18T16:07:20.994405+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24575,"mds_1":24556,"mds_2":24535,"mds_3":24524,"mds_4":14463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14445":{"gid":14445,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.39:6839/656513813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":656513813},{"type":"v1","addr":"172.21.15.39:6839","nonce":656513813}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14463":{"gid":14463,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":13,"addr":"172.21.15.39:6841/3515326042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3515326042},{"type":"v1","addr":"172.21.15.39:6841","nonce":3515326042}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6833/931350738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":931350738},{"type":"v1","addr":"172.21.15.194:6833","nonce":931350738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6833/3874209621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":3874209621},{"type":"v1","addr":"172.21.15.196:6833","nonce":3874209621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6835/188385470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":188385470},{"type":"v1","addr":"172.21.15.194:6835","nonce":188385470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6835/3949984309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":3949984309},{"type":"v1","addr":"172.21.15.196:6835","nonce":3949984309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":15,"addr":"172.21.15.194:6837/1301464246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":1301464246},{"type":"v1","addr":"172.21.15.194:6837","nonce":1301464246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":16,"addr":"172.21.15.196:6837/895247251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":895247251},{"type":"v1","addr":"172.21.15.196:6837","nonce":895247251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.196:6839/2001427268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6838","nonce":2001427268},{"type":"v1","addr":"172.21.15.196:6839","nonce":2001427268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24575":{"gid":24575,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.194:6839/274911158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":274911158},{"type":"v1","addr":"172.21.15.194:6839","nonce":274911158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24575,"qdb_cluster":[24556,24535,24575,24524,14463]},"id":1}]} 2024-03-18T16:09:44.616 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 27 2024-03-18T16:09:44.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.617+0000 7f0b21000640 1 -- 172.21.15.39:0/3934060100 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f0b000a2800 msgr2=0x7f0b000a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:44.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.617+0000 7f0b21000640 1 --2- 172.21.15.39:0/3934060100 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f0b000a2800 0x7f0b000a4cc0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f0b0402efd0 tx=0x7f0b0402cf90 comp rx=0 tx=0).stop 2024-03-18T16:09:44.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.617+0000 7f0b21000640 1 -- 172.21.15.39:0/3934060100 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0b1c1037e0 msgr2=0x7f0b1c199af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:44.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.617+0000 7f0b21000640 1 --2- 172.21.15.39:0/3934060100 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0b1c1037e0 0x7f0b1c199af0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f0b0c00d950 tx=0x7f0b0c00de10 comp rx=0 tx=0).stop 2024-03-18T16:09:44.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.617+0000 7f0b21000640 1 -- 172.21.15.39:0/3934060100 shutdown_connections 2024-03-18T16:09:44.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.617+0000 7f0b21000640 1 --2- 172.21.15.39:0/3934060100 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0b1c1049d0 0x7f0b1c19a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:44.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.617+0000 7f0b21000640 1 --2- 172.21.15.39:0/3934060100 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f0b000a2800 0x7f0b000a4cc0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:44.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.617+0000 7f0b21000640 1 --2- 172.21.15.39:0/3934060100 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0b1c105390 0x7f0b1c1a10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:44.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.617+0000 7f0b21000640 1 --2- 172.21.15.39:0/3934060100 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0b1c1037e0 0x7f0b1c199af0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:44.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.617+0000 7f0b21000640 1 -- 172.21.15.39:0/3934060100 >> 172.21.15.39:0/3934060100 conn(0x7f0b1c0fed50 msgr2=0x7f0b1c107c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:44.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.618+0000 7f0b21000640 1 -- 172.21.15.39:0/3934060100 shutdown_connections 2024-03-18T16:09:44.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.618+0000 7f0b21000640 1 -- 172.21.15.39:0/3934060100 wait complete. 2024-03-18T16:09:44.641 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T16:09:44.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.862+0000 7f1caf600640 1 -- 172.21.15.39:0/701975871 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1ca8100700 msgr2=0x7f1ca8100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:44.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.862+0000 7f1caf600640 1 --2- 172.21.15.39:0/701975871 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1ca8100700 0x7f1ca8100b20 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c009900 tx=0x7f1c9c02bc30 comp rx=0 tx=0).stop 2024-03-18T16:09:44.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.862+0000 7f1caf600640 1 -- 172.21.15.39:0/701975871 shutdown_connections 2024-03-18T16:09:44.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.862+0000 7f1caf600640 1 --2- 172.21.15.39:0/701975871 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1ca81022b0 0x7f1ca8108b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:44.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.862+0000 7f1caf600640 1 --2- 172.21.15.39:0/701975871 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1ca81018f0 0x7f1ca8101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:44.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.862+0000 7f1caf600640 1 --2- 172.21.15.39:0/701975871 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1ca8100700 0x7f1ca8100b20 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:44.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.862+0000 7f1caf600640 1 -- 172.21.15.39:0/701975871 >> 172.21.15.39:0/701975871 conn(0x7f1ca80fbc70 msgr2=0x7f1ca80fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:44.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.862+0000 7f1caf600640 1 -- 172.21.15.39:0/701975871 shutdown_connections 2024-03-18T16:09:44.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.862+0000 7f1caf600640 1 -- 172.21.15.39:0/701975871 wait complete. 2024-03-18T16:09:44.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.864+0000 7f1caf600640 1 Processor -- start 2024-03-18T16:09:44.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.864+0000 7f1caf600640 1 -- start start 2024-03-18T16:09:44.868 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.864+0000 7f1caf600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1ca8100700 0x7f1ca8104690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:44.868 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.865+0000 7f1caf600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1ca81018f0 0x7f1ca8102cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:44.868 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.865+0000 7f1caf600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1ca81022b0 0x7f1ca8103210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:44.868 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.865+0000 7f1caf600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f1ca810c3a0 con 0x7f1ca8100700 2024-03-18T16:09:44.868 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.865+0000 7f1caf600640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f1ca810c220 con 0x7f1ca81022b0 2024-03-18T16:09:44.868 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.865+0000 7f1caf600640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f1ca810c520 con 0x7f1ca81018f0 2024-03-18T16:09:44.868 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.865+0000 7f1cacc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1ca8100700 0x7f1ca8104690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:44.868 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.865+0000 7f1ca7e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1ca81018f0 0x7f1ca8102cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:44.868 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.865+0000 7f1cacc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1ca8100700 0x7f1ca8104690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:37376/0 (socket says 172.21.15.39:37376) 2024-03-18T16:09:44.868 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.865+0000 7f1cacc00640 1 -- 172.21.15.39:0/4108733474 learned_addr learned my addr 172.21.15.39:0/4108733474 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:44.868 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.865+0000 7f1cad600640 1 --2- 172.21.15.39:0/4108733474 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1ca81022b0 0x7f1ca8103210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:44.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.865+0000 7f1ca7e00640 1 -- 172.21.15.39:0/4108733474 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1ca81022b0 msgr2=0x7f1ca8103210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:44.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.865+0000 7f1ca7e00640 1 --2- 172.21.15.39:0/4108733474 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1ca81022b0 0x7f1ca8103210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:44.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.865+0000 7f1ca7e00640 1 -- 172.21.15.39:0/4108733474 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1ca8100700 msgr2=0x7f1ca8104690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:44.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.865+0000 7f1ca7e00640 1 --2- 172.21.15.39:0/4108733474 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1ca8100700 0x7f1ca8104690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:44.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.865+0000 7f1ca7e00640 1 -- 172.21.15.39:0/4108733474 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ca8103aa0 con 0x7f1ca81018f0 2024-03-18T16:09:44.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.865+0000 7f1cacc00640 1 --2- 172.21.15.39:0/4108733474 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1ca8100700 0x7f1ca8104690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:09:44.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.866+0000 7f1ca7e00640 1 --2- 172.21.15.39:0/4108733474 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1ca81018f0 0x7f1ca8102cd0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f1c9400e9f0 tx=0x7f1c9400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:44.870 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.866+0000 7f1ca5600640 1 -- 172.21.15.39:0/4108733474 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c9400ccc0 con 0x7f1ca81018f0 2024-03-18T16:09:44.870 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.866+0000 7f1ca5600640 1 -- 172.21.15.39:0/4108733474 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c94010910 con 0x7f1ca81018f0 2024-03-18T16:09:44.870 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.866+0000 7f1ca5600640 1 -- 172.21.15.39:0/4108733474 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c94018960 con 0x7f1ca81018f0 2024-03-18T16:09:44.870 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.866+0000 7f1caf600640 1 -- 172.21.15.39:0/4108733474 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ca81a1c50 con 0x7f1ca81018f0 2024-03-18T16:09:44.870 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.866+0000 7f1caf600640 1 -- 172.21.15.39:0/4108733474 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ca81a21a0 con 0x7f1ca81018f0 2024-03-18T16:09:44.870 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.867+0000 7f1caf600640 1 -- 172.21.15.39:0/4108733474 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ca81a23d0 con 0x7f1ca81018f0 2024-03-18T16:09:44.872 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.870+0000 7f1ca5600640 1 -- 172.21.15.39:0/4108733474 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f1c94010ac0 con 0x7f1ca81018f0 2024-03-18T16:09:44.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.871+0000 7f1ca5600640 1 --2- 172.21.15.39:0/4108733474 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1c900a2880 0x7f1c900a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:44.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.872+0000 7f1ca5600640 1 -- 172.21.15.39:0/4108733474 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f1c940cf140 con 0x7f1ca81018f0 2024-03-18T16:09:44.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.872+0000 7f1cacc00640 1 --2- 172.21.15.39:0/4108733474 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1c900a2880 0x7f1c900a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:44.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.873+0000 7f1cacc00640 1 --2- 172.21.15.39:0/4108733474 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1c900a2880 0x7f1c900a4d40 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c006010 tx=0x7f1c9c02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:44.877 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:44.875+0000 7f1ca5600640 1 -- 172.21.15.39:0/4108733474 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1c9409b7a0 con 0x7f1ca81018f0 2024-03-18T16:09:45.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.079+0000 7f1caf600640 1 -- 172.21.15.39:0/4108733474 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1ca81a2760 con 0x7f1ca81018f0 2024-03-18T16:09:45.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.081+0000 7f1ca5600640 1 -- 172.21.15.39:0/4108733474 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24762 (secure 0 0 0) 0x7f1c9409f840 con 0x7f1ca81018f0 2024-03-18T16:09:45.084 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:09:45.085 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":93,"fsid":"1892d8fc-e540-11ee-95c9-87774f69a715","created":"2024-03-18T15:57:36.507008+0000","modified":"2024-03-18T16:08:47.874967+0000","last_up_change":"2024-03-18T16:05:45.250629+0000","last_in_change":"2024-03-18T16:05:24.841337+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T16:01:30.048247+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T16:06:42.191791+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T16:06:43.670325+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T16:06:47.523388+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"771e0558-be62-4436-94b0-6364751911c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6803","nonce":4090743635}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6805","nonce":4090743635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6809","nonce":4090743635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6807","nonce":4090743635}]},"public_addr":"172.21.15.39:6803/4090743635","cluster_addr":"172.21.15.39:6805/4090743635","heartbeat_back_addr":"172.21.15.39:6809/4090743635","heartbeat_front_addr":"172.21.15.39:6807/4090743635","state":["exists","up"]},{"osd":1,"uuid":"0430eee1-cf34-4277-b255-c7d9f07ad19a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6801","nonce":2800329805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6803","nonce":2800329805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6807","nonce":2800329805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6805","nonce":2800329805}]},"public_addr":"172.21.15.194:6801/2800329805","cluster_addr":"172.21.15.194:6803/2800329805","heartbeat_back_addr":"172.21.15.194:6807/2800329805","heartbeat_front_addr":"172.21.15.194:6805/2800329805","state":["exists","up"]},{"osd":2,"uuid":"3a718da4-2511-4ec1-b1a8-eafd6cce0df0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6801","nonce":3935874719}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6803","nonce":3935874719}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6807","nonce":3935874719}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6805","nonce":3935874719}]},"public_addr":"172.21.15.196:6801/3935874719","cluster_addr":"172.21.15.196:6803/3935874719","heartbeat_back_addr":"172.21.15.196:6807/3935874719","heartbeat_front_addr":"172.21.15.196:6805/3935874719","state":["exists","up"]},{"osd":3,"uuid":"6f9c37f9-0ab9-4038-8919-afd4b39c4165","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6811","nonce":893345939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6813","nonce":893345939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6817","nonce":893345939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6815","nonce":893345939}]},"public_addr":"172.21.15.39:6811/893345939","cluster_addr":"172.21.15.39:6813/893345939","heartbeat_back_addr":"172.21.15.39:6817/893345939","heartbeat_front_addr":"172.21.15.39:6815/893345939","state":["exists","up"]},{"osd":4,"uuid":"e795bb0e-4791-4537-acda-1a321e19bc9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6809","nonce":800423088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6811","nonce":800423088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6815","nonce":800423088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6813","nonce":800423088}]},"public_addr":"172.21.15.194:6809/800423088","cluster_addr":"172.21.15.194:6811/800423088","heartbeat_back_addr":"172.21.15.194:6815/800423088","heartbeat_front_addr":"172.21.15.194:6813/800423088","state":["exists","up"]},{"osd":5,"uuid":"026e175e-6acb-4b4c-b235-8ccee8ae1149","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6809","nonce":2746224525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6811","nonce":2746224525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6815","nonce":2746224525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6813","nonce":2746224525}]},"public_addr":"172.21.15.196:6809/2746224525","cluster_addr":"172.21.15.196:6811/2746224525","heartbeat_back_addr":"172.21.15.196:6815/2746224525","heartbeat_front_addr":"172.21.15.196:6813/2746224525","state":["exists","up"]},{"osd":6,"uuid":"c1b183ff-383f-4c5c-9075-f76bab1c0bb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6819","nonce":1213371331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6821","nonce":1213371331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6825","nonce":1213371331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6823","nonce":1213371331}]},"public_addr":"172.21.15.39:6819/1213371331","cluster_addr":"172.21.15.39:6821/1213371331","heartbeat_back_addr":"172.21.15.39:6825/1213371331","heartbeat_front_addr":"172.21.15.39:6823/1213371331","state":["exists","up"]},{"osd":7,"uuid":"612f8697-957d-4ef4-a87f-241b36f9d13d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6817","nonce":3299162444}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6819","nonce":3299162444}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6823","nonce":3299162444}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6821","nonce":3299162444}]},"public_addr":"172.21.15.194:6817/3299162444","cluster_addr":"172.21.15.194:6819/3299162444","heartbeat_back_addr":"172.21.15.194:6823/3299162444","heartbeat_front_addr":"172.21.15.194:6821/3299162444","state":["exists","up"]},{"osd":8,"uuid":"836a9107-4d4f-4594-b8dc-0301e64f59d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6817","nonce":2456143145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6819","nonce":2456143145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6823","nonce":2456143145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6821","nonce":2456143145}]},"public_addr":"172.21.15.196:6817/2456143145","cluster_addr":"172.21.15.196:6819/2456143145","heartbeat_back_addr":"172.21.15.196:6823/2456143145","heartbeat_front_addr":"172.21.15.196:6821/2456143145","state":["exists","up"]},{"osd":9,"uuid":"4f30b960-12e3-4b16-bd28-fcc988139b58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6827","nonce":1848634433}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6829","nonce":1848634433}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6833","nonce":1848634433}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6831","nonce":1848634433}]},"public_addr":"172.21.15.39:6827/1848634433","cluster_addr":"172.21.15.39:6829/1848634433","heartbeat_back_addr":"172.21.15.39:6833/1848634433","heartbeat_front_addr":"172.21.15.39:6831/1848634433","state":["exists","up"]},{"osd":10,"uuid":"620f90e8-23e6-4bc0-8269-7219b9e1aebd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6825","nonce":3725064809}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6827","nonce":3725064809}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6831","nonce":3725064809}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6829","nonce":3725064809}]},"public_addr":"172.21.15.194:6825/3725064809","cluster_addr":"172.21.15.194:6827/3725064809","heartbeat_back_addr":"172.21.15.194:6831/3725064809","heartbeat_front_addr":"172.21.15.194:6829/3725064809","state":["exists","up"]},{"osd":11,"uuid":"867a2535-8235-4ceb-a698-eb86869d6d20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6825","nonce":2501212422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6827","nonce":2501212422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6831","nonce":2501212422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6829","nonce":2501212422}]},"public_addr":"172.21.15.196:6825/2501212422","cluster_addr":"172.21.15.196:6827/2501212422","heartbeat_back_addr":"172.21.15.196:6831/2501212422","heartbeat_front_addr":"172.21.15.196:6829/2501212422","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:00:36.931566+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:01:02.614024+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:01:26.193265+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:01:58.308833+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:02:25.049546+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:02:49.748750+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:03:23.582153+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:03:50.447321+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:04:15.812129+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:04:49.998047+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:05:17.087924+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:05:43.167637+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":4,"to":7}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.a","mappings":[{"from":5,"to":8}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.39:0/3482086140":"2024-03-19T15:58:45.340294+0000","172.21.15.39:0/3844306284":"2024-03-19T15:58:45.340294+0000","172.21.15.39:6801/1810238546":"2024-03-19T15:58:45.340294+0000","172.21.15.39:6800/1810238546":"2024-03-19T15:58:45.340294+0000","172.21.15.39:0/2528397488":"2024-03-19T15:58:10.264798+0000","172.21.15.39:0/2240981254":"2024-03-19T15:58:10.264798+0000","172.21.15.39:6800/1217503189":"2024-03-19T15:58:10.264798+0000","172.21.15.39:0/3754876370":"2024-03-19T15:58:10.264798+0000","172.21.15.39:6801/1217503189":"2024-03-19T15:58:10.264798+0000","172.21.15.39:0/1874637628":"2024-03-19T15:58:45.340294+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T16:09:45.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.084+0000 7f1caf600640 1 -- 172.21.15.39:0/4108733474 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1c900a2880 msgr2=0x7f1c900a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:45.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.084+0000 7f1caf600640 1 --2- 172.21.15.39:0/4108733474 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1c900a2880 0x7f1c900a4d40 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f1c9c006010 tx=0x7f1c9c02df90 comp rx=0 tx=0).stop 2024-03-18T16:09:45.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.084+0000 7f1caf600640 1 -- 172.21.15.39:0/4108733474 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1ca81018f0 msgr2=0x7f1ca8102cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:45.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.084+0000 7f1caf600640 1 --2- 172.21.15.39:0/4108733474 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1ca81018f0 0x7f1ca8102cd0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f1c9400e9f0 tx=0x7f1c9400eeb0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.084+0000 7f1caf600640 1 -- 172.21.15.39:0/4108733474 shutdown_connections 2024-03-18T16:09:45.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.084+0000 7f1caf600640 1 --2- 172.21.15.39:0/4108733474 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1ca81022b0 0x7f1ca8103210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.084+0000 7f1caf600640 1 --2- 172.21.15.39:0/4108733474 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1c900a2880 0x7f1c900a4d40 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.084+0000 7f1caf600640 1 --2- 172.21.15.39:0/4108733474 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1ca81018f0 0x7f1ca8102cd0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.084+0000 7f1caf600640 1 --2- 172.21.15.39:0/4108733474 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1ca8100700 0x7f1ca8104690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.084+0000 7f1caf600640 1 -- 172.21.15.39:0/4108733474 >> 172.21.15.39:0/4108733474 conn(0x7f1ca80fbc70 msgr2=0x7f1ca80fd6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:45.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.085+0000 7f1caf600640 1 -- 172.21.15.39:0/4108733474 shutdown_connections 2024-03-18T16:09:45.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.085+0000 7f1caf600640 1 -- 172.21.15.39:0/4108733474 wait complete. 2024-03-18T16:09:45.107 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-18T16:06:44.180425+0000', 'modified': '2024-03-18T16:07:20.994405+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24575, 'mds_1': 24556, 'mds_2': 24535, 'mds_3': 24524, 'mds_4': 14463}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14445': {'gid': 14445, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.39:6839/656513813', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6838', 'nonce': 656513813}, {'type': 'v1', 'addr': '172.21.15.39:6839', 'nonce': 656513813}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_14463': {'gid': 14463, 'name': 'a', 'rank': 4, 'incarnation': 25, 'state': 'up:active', 'state_seq': 13, 'addr': '172.21.15.39:6841/3515326042', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.39:6840', 'nonce': 3515326042}, {'type': 'v1', 'addr': '172.21.15.39:6841', 'nonce': 3515326042}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24446': {'gid': 24446, 'name': 'k', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.194:6833/931350738', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.194:6832', 'nonce': 931350738}, {'type': 'v1', 'addr': '172.21.15.194:6833', 'nonce': 931350738}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24463': {'gid': 24463, 'name': 'l', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.196:6833/3874209621', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6832', 'nonce': 3874209621}, {'type': 'v1', 'addr': '172.21.15.196:6833', 'nonce': 3874209621}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24494': {'gid': 24494, 'name': 'e', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.194:6835/188385470', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.194:6834', 'nonce': 188385470}, {'type': 'v1', 'addr': '172.21.15.194:6835', 'nonce': 188385470}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24523': {'gid': 24523, 'name': 'f', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.196:6835/3949984309', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6834', 'nonce': 3949984309}, {'type': 'v1', 'addr': '172.21.15.196:6835', 'nonce': 3949984309}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24524': {'gid': 24524, 'name': 'h', 'rank': 3, 'incarnation': 23, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.194:6837/1301464246', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.194:6836', 'nonce': 1301464246}, {'type': 'v1', 'addr': '172.21.15.194:6837', 'nonce': 1301464246}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24535': {'gid': 24535, 'name': 'c', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.196:6837/895247251', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6836', 'nonce': 895247251}, {'type': 'v1', 'addr': '172.21.15.196:6837', 'nonce': 895247251}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24556': {'gid': 24556, 'name': 'i', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.196:6839/2001427268', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6838', 'nonce': 2001427268}, {'type': 'v1', 'addr': '172.21.15.196:6839', 'nonce': 2001427268}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24575': {'gid': 24575, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.194:6839/274911158', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.194:6838', 'nonce': 274911158}, {'type': 'v1', 'addr': '172.21.15.194:6839', 'nonce': 274911158}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24575, 'qdb_cluster': [24556, 24535, 24575, 24524, 14463]} 2024-03-18T16:09:45.107 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-03-18T16:09:45.107 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-18T16:09:45.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.317+0000 7f3447e00640 1 -- 172.21.15.39:0/2061032656 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f34481073a0 msgr2=0x7f34481097f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:45.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.317+0000 7f3447e00640 1 --2- 172.21.15.39:0/2061032656 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f34481073a0 0x7f34481097f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f3440009ca0 tx=0x7f344002c100 comp rx=0 tx=0).stop 2024-03-18T16:09:45.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.317+0000 7f3447e00640 1 -- 172.21.15.39:0/2061032656 shutdown_connections 2024-03-18T16:09:45.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.317+0000 7f3447e00640 1 --2- 172.21.15.39:0/2061032656 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f34481073a0 0x7f34481097f0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.317+0000 7f3447e00640 1 --2- 172.21.15.39:0/2061032656 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3448104a70 0x7f3448106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.317+0000 7f3447e00640 1 --2- 172.21.15.39:0/2061032656 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3448102140 0x7f3448104530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.317+0000 7f3447e00640 1 -- 172.21.15.39:0/2061032656 >> 172.21.15.39:0/2061032656 conn(0x7f34480fbb50 msgr2=0x7f34480fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:45.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.317+0000 7f3447e00640 1 -- 172.21.15.39:0/2061032656 shutdown_connections 2024-03-18T16:09:45.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.317+0000 7f3447e00640 1 -- 172.21.15.39:0/2061032656 wait complete. 2024-03-18T16:09:45.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.318+0000 7f3447e00640 1 Processor -- start 2024-03-18T16:09:45.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.318+0000 7f3447e00640 1 -- start start 2024-03-18T16:09:45.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.319+0000 7f3447e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3448102140 0x7f3448197940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:45.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.319+0000 7f3447e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3448104a70 0x7f3448197e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:45.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.319+0000 7f3447e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f34481073a0 0x7f344819eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:45.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.319+0000 7f3447e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f344810d340 con 0x7f34481073a0 2024-03-18T16:09:45.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.319+0000 7f3447e00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f344810d1c0 con 0x7f3448102140 2024-03-18T16:09:45.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.319+0000 7f3447e00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f344810d4c0 con 0x7f3448104a70 2024-03-18T16:09:45.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.319+0000 7f3446000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3448104a70 0x7f3448197e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:45.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.319+0000 7f3446a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3448102140 0x7f3448197940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:45.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.319+0000 7f3446a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3448102140 0x7f3448197940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:38704/0 (socket says 172.21.15.39:38704) 2024-03-18T16:09:45.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.319+0000 7f3446a00640 1 -- 172.21.15.39:0/1626257695 learned_addr learned my addr 172.21.15.39:0/1626257695 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:45.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.319+0000 7f3446000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3448104a70 0x7f3448197e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:56154/0 (socket says 172.21.15.39:56154) 2024-03-18T16:09:45.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.319+0000 7f3447400640 1 --2- 172.21.15.39:0/1626257695 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f34481073a0 0x7f344819eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:45.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.320+0000 7f3446000640 1 -- 172.21.15.39:0/1626257695 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3448102140 msgr2=0x7f3448197940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:45.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.320+0000 7f3446000640 1 --2- 172.21.15.39:0/1626257695 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3448102140 0x7f3448197940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.320+0000 7f3446000640 1 -- 172.21.15.39:0/1626257695 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f34481073a0 msgr2=0x7f344819eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:45.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.320+0000 7f3446000640 1 --2- 172.21.15.39:0/1626257695 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f34481073a0 0x7f344819eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.320+0000 7f3446000640 1 -- 172.21.15.39:0/1626257695 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f344819f560 con 0x7f3448104a70 2024-03-18T16:09:45.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.320+0000 7f3446000640 1 --2- 172.21.15.39:0/1626257695 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3448104a70 0x7f3448197e80 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f343400e9c0 tx=0x7f343400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:45.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.320+0000 7f344ca00640 1 -- 172.21.15.39:0/1626257695 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f343400ccc0 con 0x7f3448104a70 2024-03-18T16:09:45.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.320+0000 7f344ca00640 1 -- 172.21.15.39:0/1626257695 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3434010910 con 0x7f3448104a70 2024-03-18T16:09:45.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.320+0000 7f3447e00640 1 -- 172.21.15.39:0/1626257695 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f344819f850 con 0x7f3448104a70 2024-03-18T16:09:45.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.320+0000 7f344ca00640 1 -- 172.21.15.39:0/1626257695 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34340188c0 con 0x7f3448104a70 2024-03-18T16:09:45.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.320+0000 7f3447e00640 1 -- 172.21.15.39:0/1626257695 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f344819fda0 con 0x7f3448104a70 2024-03-18T16:09:45.326 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.323+0000 7f344ca00640 1 -- 172.21.15.39:0/1626257695 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f3434010ac0 con 0x7f3448104a70 2024-03-18T16:09:45.326 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.323+0000 7f3450a53740 1 Processor -- start 2024-03-18T16:09:45.326 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.324+0000 7f3450a53740 1 -- start start 2024-03-18T16:09:45.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.324+0000 7f3450a53740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bef1d72750 0x55bef1d72b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:45.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.324+0000 7f3450a53740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bef1d73140 0x55bef1d77c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:45.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.324+0000 7f3450a53740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bef1d782b0 0x55bef1d7a740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:45.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.324+0000 7f3450a53740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55bef1cd2c50 con 0x55bef1d73140 2024-03-18T16:09:45.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.324+0000 7f3450a53740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55bef1c7b9d0 con 0x55bef1d782b0 2024-03-18T16:09:45.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.324+0000 7f3450a53740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55bef1ccecf0 con 0x55bef1d72750 2024-03-18T16:09:45.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.324+0000 7f3447400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bef1d72750 0x55bef1d72b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:45.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.324+0000 7f3447400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bef1d72750 0x55bef1d72b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:56164/0 (socket says 172.21.15.39:56164) 2024-03-18T16:09:45.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.324+0000 7f3447400640 1 -- 172.21.15.39:0/1289668380 learned_addr learned my addr 172.21.15.39:0/1289668380 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:45.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.324+0000 7f3446000640 1 --2- 172.21.15.39:0/1289668380 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bef1d782b0 0x55bef1d7a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:45.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.324+0000 7f3446a00640 1 --2- 172.21.15.39:0/1289668380 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bef1d73140 0x55bef1d77c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:45.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.325+0000 7f3447400640 1 -- 172.21.15.39:0/1289668380 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bef1d782b0 msgr2=0x55bef1d7a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:45.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.325+0000 7f3447400640 1 --2- 172.21.15.39:0/1289668380 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bef1d782b0 0x55bef1d7a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.325+0000 7f3447400640 1 -- 172.21.15.39:0/1289668380 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bef1d73140 msgr2=0x55bef1d77c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:45.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.325+0000 7f3447400640 1 --2- 172.21.15.39:0/1289668380 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bef1d73140 0x55bef1d77c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.325+0000 7f3447400640 1 -- 172.21.15.39:0/1289668380 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bef1c95f10 con 0x55bef1d72750 2024-03-18T16:09:45.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.325+0000 7f344ca00640 1 --2- 172.21.15.39:0/1626257695 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f342c0a2800 0x7f342c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:45.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.325+0000 7f3447400640 1 --2- 172.21.15.39:0/1289668380 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bef1d72750 0x55bef1d72b70 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f344002c710 tx=0x7f34400323d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f4c63090ac27c87b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:45.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.325+0000 7f344ca00640 1 -- 172.21.15.39:0/1626257695 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f34340cf200 con 0x7f3448104a70 2024-03-18T16:09:45.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.325+0000 7f3446a00640 1 --2- 172.21.15.39:0/1626257695 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f342c0a2800 0x7f342c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:45.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.325+0000 7f3433e00640 1 -- 172.21.15.39:0/1289668380 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3440043070 con 0x55bef1d72750 2024-03-18T16:09:45.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.326+0000 7f3433e00640 1 -- 172.21.15.39:0/1289668380 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3440032c90 con 0x55bef1d72750 2024-03-18T16:09:45.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.326+0000 7f3433e00640 1 -- 172.21.15.39:0/1289668380 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3440039040 con 0x55bef1d72750 2024-03-18T16:09:45.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.326+0000 7f3450a53740 1 -- 172.21.15.39:0/1289668380 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bef1d72750 msgr2=0x55bef1d72b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:45.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.326+0000 7f3450a53740 1 --2- 172.21.15.39:0/1289668380 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bef1d72750 0x55bef1d72b70 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f344002c710 tx=0x7f34400323d0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.326+0000 7f3446a00640 1 --2- 172.21.15.39:0/1626257695 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f342c0a2800 0x7f342c0a4cc0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x55bef1d73540 tx=0x7f3438005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:45.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.326+0000 7f3450a53740 1 -- 172.21.15.39:0/1289668380 shutdown_connections 2024-03-18T16:09:45.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.326+0000 7f3450a53740 1 --2- 172.21.15.39:0/1289668380 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bef1d782b0 0x55bef1d7a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.326+0000 7f3450a53740 1 --2- 172.21.15.39:0/1289668380 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bef1d73140 0x55bef1d77c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.327+0000 7f3450a53740 1 --2- 172.21.15.39:0/1289668380 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bef1d72750 0x55bef1d72b70 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.327+0000 7f3450a53740 1 -- 172.21.15.39:0/1289668380 >> 172.21.15.39:0/1289668380 conn(0x55bef1d698b0 msgr2=0x55bef1d69cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:45.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.327+0000 7f3450a53740 1 -- 172.21.15.39:0/1289668380 shutdown_connections 2024-03-18T16:09:45.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.327+0000 7f3450a53740 1 -- 172.21.15.39:0/1289668380 wait complete. 2024-03-18T16:09:45.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.327+0000 7f3450a53740 1 -- start start 2024-03-18T16:09:45.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.327+0000 7f3450a53740 1 Processor -- start 2024-03-18T16:09:45.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.328+0000 7f3450a53740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bef1d72750 0x55bef1d7c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:45.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.328+0000 7f3450a53740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bef1d73140 0x55bef1d7cde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:45.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.328+0000 7f3450a53740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bef1d782b0 0x55bef1d83e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:45.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.328+0000 7f3450a53740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55bef1d7b200 con 0x55bef1d73140 2024-03-18T16:09:45.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.328+0000 7f3450a53740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55bef1c8ee20 con 0x55bef1d72750 2024-03-18T16:09:45.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.328+0000 7f3450a53740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55bef1d7b380 con 0x55bef1d782b0 2024-03-18T16:09:45.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.328+0000 7f3450a53740 10 client.0 _ll_register_callbacks cb 0x55bef1c8e7f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:09:45.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.328+0000 7f3446a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bef1d72750 0x55bef1d7c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:45.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.328+0000 7f3446a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bef1d72750 0x55bef1d7c8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:38718/0 (socket says 172.21.15.39:38718) 2024-03-18T16:09:45.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.328+0000 7f3446a00640 1 -- 172.21.15.39:0/2231324051 learned_addr learned my addr 172.21.15.39:0/2231324051 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:45.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.328+0000 7f3447400640 1 --2- 172.21.15.39:0/2231324051 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bef1d782b0 0x55bef1d83e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:45.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.328+0000 7f3446000640 1 --2- 172.21.15.39:0/2231324051 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bef1d73140 0x55bef1d7cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:45.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.328+0000 7f3446a00640 1 -- 172.21.15.39:0/2231324051 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bef1d782b0 msgr2=0x55bef1d83e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:45.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.328+0000 7f3446a00640 1 --2- 172.21.15.39:0/2231324051 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bef1d782b0 0x55bef1d83e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.328+0000 7f3446a00640 1 -- 172.21.15.39:0/2231324051 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bef1d73140 msgr2=0x55bef1d7cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:45.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.328+0000 7f3446a00640 1 --2- 172.21.15.39:0/2231324051 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bef1d73140 0x55bef1d7cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.328+0000 7f3446a00640 1 -- 172.21.15.39:0/2231324051 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bef19e72d0 con 0x55bef1d72750 2024-03-18T16:09:45.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.328+0000 7f3447400640 1 --2- 172.21.15.39:0/2231324051 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bef1d782b0 0x55bef1d83e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:09:45.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.329+0000 7f3446a00640 1 --2- 172.21.15.39:0/2231324051 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bef1d72750 0x55bef1d7c8a0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f343801e9d0 tx=0x7f3438004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd0404dc31a90c50 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:45.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.329+0000 7f3430c00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:09:45.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.329+0000 7f3430c00640 1 -- 172.21.15.39:0/2231324051 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3438004020 con 0x55bef1d72750 2024-03-18T16:09:45.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.329+0000 7f3430c00640 1 -- 172.21.15.39:0/2231324051 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f3438018030 con 0x55bef1d72750 2024-03-18T16:09:45.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.329+0000 7f3450a53740 1 -- 172.21.15.39:0/2231324051 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3440032c90 con 0x55bef1d72750 2024-03-18T16:09:45.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.329+0000 7f3430c00640 1 -- 172.21.15.39:0/2231324051 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f3438033050 con 0x55bef1d72750 2024-03-18T16:09:45.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.329+0000 7f3450a53740 10 client.25355 fetch_fsmap learned FSMap version 27 2024-03-18T16:09:45.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.329+0000 7f3450a53740 1 -- 172.21.15.39:0/2231324051 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55bef1ab16b0 con 0x55bef1d72750 2024-03-18T16:09:45.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.330+0000 7f3430c00640 1 -- 172.21.15.39:0/2231324051 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3440032c90 con 0x55bef1d72750 2024-03-18T16:09:45.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.330+0000 7f3430c00640 1 -- 172.21.15.39:0/2231324051 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f3438029420 con 0x55bef1d72750 2024-03-18T16:09:45.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.330+0000 7f3450a53740 10 client.25355 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T16:09:45.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.330+0000 7f3450a53740 10 client.25355 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-18T16:09:45.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.330+0000 7f3450a53740 20 client.25355 populate_metadata read hostname 'smithi039' 2024-03-18T16:09:45.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.330+0000 7f3450a53740 1 --2- 172.21.15.39:0/2231324051 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x55bef1d896d0 0x55bef1d8ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:45.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.330+0000 7f3450a53740 4 client.25355 mds_command: new command op to 14463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:09:45.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.330+0000 7f3450a53740 1 -- 172.21.15.39:0/2231324051 --> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bef1d302b0 con 0x55bef1d896d0 2024-03-18T16:09:45.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.330+0000 7f3446000640 1 --2- 172.21.15.39:0/2231324051 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x55bef1d896d0 0x55bef1d8ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:45.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.331+0000 7f3446000640 1 --2- 172.21.15.39:0/2231324051 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x55bef1d896d0 0x55bef1d8ab40 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=b6761ba1ae4aab2e server_cookie=1fba1fcc6a1338cf in_seq=0 out_seq=0 2024-03-18T16:09:45.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.331+0000 7f3430c00640 10 client.25355 ms_handle_connect on v2:172.21.15.39:6840/3515326042 2024-03-18T16:09:45.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.331+0000 7f3430c00640 1 -- 172.21.15.39:0/2231324051 <== mds.4 v2:172.21.15.39:6840/3515326042 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55bef1d302b0 con 0x55bef1d896d0 2024-03-18T16:09:45.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.331+0000 7f3430c00640 10 client.25355 handle_command_reply: tid=0 2024-03-18T16:09:45.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.332+0000 7f3450a53740 1 client.25355 shutdown 2024-03-18T16:09:45.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.332+0000 7f3450a53740 1 -- 172.21.15.39:0/2231324051 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bef1d72750 msgr2=0x55bef1d7c8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:45.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.332+0000 7f3450a53740 1 --2- 172.21.15.39:0/2231324051 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bef1d72750 0x55bef1d7c8a0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f343801e9d0 tx=0x7f3438004730 comp rx=0 tx=0).stop 2024-03-18T16:09:45.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.332+0000 7f3450a53740 1 -- 172.21.15.39:0/2231324051 shutdown_connections 2024-03-18T16:09:45.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.332+0000 7f3450a53740 1 --2- 172.21.15.39:0/2231324051 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x55bef1d896d0 0x55bef1d8ab40 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.332+0000 7f3450a53740 1 --2- 172.21.15.39:0/2231324051 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bef1d782b0 0x55bef1d83e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.332+0000 7f3430c00640 0 client.25355 ms_handle_reset on v2:172.21.15.39:6840/3515326042 2024-03-18T16:09:45.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.332+0000 7f3450a53740 1 --2- 172.21.15.39:0/2231324051 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bef1d73140 0x55bef1d7cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.332+0000 7f3450a53740 1 --2- 172.21.15.39:0/2231324051 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bef1d72750 0x55bef1d7c8a0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.332+0000 7f3450a53740 1 -- 172.21.15.39:0/2231324051 >> 172.21.15.39:0/2231324051 conn(0x55bef1d698b0 msgr2=0x55bef1d68a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:45.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.332+0000 7f3450a53740 1 -- 172.21.15.39:0/2231324051 shutdown_connections 2024-03-18T16:09:45.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.332+0000 7f3450a53740 1 -- 172.21.15.39:0/2231324051 wait complete. 2024-03-18T16:09:45.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.332+0000 7f3450a53740 20 client.25355 trim_cache size 0 max 16384 2024-03-18T16:09:45.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.352+0000 7f3450a53740 1 Processor -- start 2024-03-18T16:09:45.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.352+0000 7f3450a53740 1 -- start start 2024-03-18T16:09:45.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.352+0000 7f3450a53740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bef1d72750 0x55bef1d7c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:45.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.352+0000 7f3450a53740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bef1d73140 0x55bef1d83de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:45.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.352+0000 7f3450a53740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bef1d782b0 0x55bef1d7c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:45.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.352+0000 7f3450a53740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55bef1d88e80 con 0x55bef1d73140 2024-03-18T16:09:45.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.352+0000 7f3450a53740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55bef1d89550 con 0x55bef1d72750 2024-03-18T16:09:45.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.352+0000 7f3450a53740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55bef1d89c20 con 0x55bef1d782b0 2024-03-18T16:09:45.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.352+0000 7f3447400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bef1d782b0 0x55bef1d7c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:45.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.352+0000 7f3447400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bef1d782b0 0x55bef1d7c9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:56174/0 (socket says 172.21.15.39:56174) 2024-03-18T16:09:45.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.352+0000 7f3447400640 1 -- 172.21.15.39:0/4217354798 learned_addr learned my addr 172.21.15.39:0/4217354798 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:45.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.353+0000 7f3446a00640 1 --2- 172.21.15.39:0/4217354798 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bef1d72750 0x55bef1d7c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:45.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.353+0000 7f3446000640 1 --2- 172.21.15.39:0/4217354798 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bef1d73140 0x55bef1d83de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:45.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.353+0000 7f3447400640 1 -- 172.21.15.39:0/4217354798 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bef1d72750 msgr2=0x55bef1d7c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:45.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.353+0000 7f3447400640 1 --2- 172.21.15.39:0/4217354798 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bef1d72750 0x55bef1d7c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.353+0000 7f3447400640 1 -- 172.21.15.39:0/4217354798 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bef1d73140 msgr2=0x55bef1d83de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:45.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.353+0000 7f3447400640 1 --2- 172.21.15.39:0/4217354798 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bef1d73140 0x55bef1d83de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.353+0000 7f3447400640 1 -- 172.21.15.39:0/4217354798 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bef1d6a620 con 0x55bef1d782b0 2024-03-18T16:09:45.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.353+0000 7f3446000640 1 --2- 172.21.15.39:0/4217354798 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bef1d73140 0x55bef1d83de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:09:45.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.353+0000 7f3447400640 1 --2- 172.21.15.39:0/4217354798 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bef1d782b0 0x55bef1d7c9f0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f3440009ca0 tx=0x7f3440004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5b72748680b7e6c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:45.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.353+0000 7f3433400640 1 -- 172.21.15.39:0/4217354798 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3440039050 con 0x55bef1d782b0 2024-03-18T16:09:45.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.354+0000 7f3433400640 1 -- 172.21.15.39:0/4217354798 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f344004c070 con 0x55bef1d782b0 2024-03-18T16:09:45.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.354+0000 7f3433400640 1 -- 172.21.15.39:0/4217354798 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3440043040 con 0x55bef1d782b0 2024-03-18T16:09:45.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.354+0000 7f3450a53740 1 -- 172.21.15.39:0/4217354798 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bef1d782b0 msgr2=0x55bef1d7c9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:45.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.354+0000 7f3450a53740 1 --2- 172.21.15.39:0/4217354798 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bef1d782b0 0x55bef1d7c9f0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f3440009ca0 tx=0x7f3440004a40 comp rx=0 tx=0).stop 2024-03-18T16:09:45.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.354+0000 7f3450a53740 1 -- 172.21.15.39:0/4217354798 shutdown_connections 2024-03-18T16:09:45.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.354+0000 7f3450a53740 1 --2- 172.21.15.39:0/4217354798 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bef1d782b0 0x55bef1d7c9f0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.354+0000 7f3450a53740 1 --2- 172.21.15.39:0/4217354798 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bef1d73140 0x55bef1d83de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.354+0000 7f3450a53740 1 --2- 172.21.15.39:0/4217354798 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bef1d72750 0x55bef1d7c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.354+0000 7f3450a53740 1 -- 172.21.15.39:0/4217354798 >> 172.21.15.39:0/4217354798 conn(0x55bef1d698b0 msgr2=0x55bef1d77920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:45.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.354+0000 7f3450a53740 1 -- 172.21.15.39:0/4217354798 shutdown_connections 2024-03-18T16:09:45.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.354+0000 7f3450a53740 1 -- 172.21.15.39:0/4217354798 wait complete. 2024-03-18T16:09:45.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.355+0000 7f3450a53740 1 -- start start 2024-03-18T16:09:45.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.355+0000 7f3450a53740 1 Processor -- start 2024-03-18T16:09:45.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.355+0000 7f3450a53740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bef1d72750 0x55bef1d7f480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:45.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.355+0000 7f3450a53740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bef1d73140 0x55bef1d7f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:45.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.355+0000 7f3450a53740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bef1d782b0 0x55bef1d829e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:45.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.355+0000 7f3450a53740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55bef1d8a9c0 con 0x55bef1d73140 2024-03-18T16:09:45.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.355+0000 7f3450a53740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55bef1d8a2f0 con 0x55bef1d72750 2024-03-18T16:09:45.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.355+0000 7f3450a53740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55bef1d8b090 con 0x55bef1d782b0 2024-03-18T16:09:45.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.355+0000 7f3450a53740 10 client.0 _ll_register_callbacks cb 0x7f343801e890 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:09:45.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.355+0000 7f3446a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bef1d72750 0x55bef1d7f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:45.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.355+0000 7f3446a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bef1d72750 0x55bef1d7f480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:38728/0 (socket says 172.21.15.39:38728) 2024-03-18T16:09:45.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.355+0000 7f3446a00640 1 -- 172.21.15.39:0/1330321973 learned_addr learned my addr 172.21.15.39:0/1330321973 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:45.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.356+0000 7f3447400640 1 --2- 172.21.15.39:0/1330321973 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bef1d782b0 0x55bef1d829e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:45.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.356+0000 7f3446000640 1 --2- 172.21.15.39:0/1330321973 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bef1d73140 0x55bef1d7f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:45.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.356+0000 7f3446a00640 1 -- 172.21.15.39:0/1330321973 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bef1d782b0 msgr2=0x55bef1d829e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:45.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.356+0000 7f3446a00640 1 --2- 172.21.15.39:0/1330321973 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bef1d782b0 0x55bef1d829e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.356+0000 7f3446a00640 1 -- 172.21.15.39:0/1330321973 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bef1d73140 msgr2=0x55bef1d7f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:45.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.356+0000 7f3446a00640 1 --2- 172.21.15.39:0/1330321973 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bef1d73140 0x55bef1d7f9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.356+0000 7f3446a00640 1 -- 172.21.15.39:0/1330321973 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bef1d82fd0 con 0x55bef1d72750 2024-03-18T16:09:45.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.356+0000 7f3446000640 1 --2- 172.21.15.39:0/1330321973 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bef1d73140 0x55bef1d7f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:09:45.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.356+0000 7f3446a00640 1 --2- 172.21.15.39:0/1330321973 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bef1d72750 0x55bef1d7f480 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f3438000c00 tx=0x7f3438029d00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4990192e008b235a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:45.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.356+0000 7f3430c00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:09:45.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.356+0000 7f3430c00640 1 -- 172.21.15.39:0/1330321973 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f343803d030 con 0x55bef1d72750 2024-03-18T16:09:45.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.356+0000 7f3430c00640 1 -- 172.21.15.39:0/1330321973 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f3438024040 con 0x55bef1d72750 2024-03-18T16:09:45.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.356+0000 7f3450a53740 1 -- 172.21.15.39:0/1330321973 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f344004c070 con 0x55bef1d72750 2024-03-18T16:09:45.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.357+0000 7f3430c00640 1 -- 172.21.15.39:0/1330321973 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f3438047050 con 0x55bef1d72750 2024-03-18T16:09:45.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.357+0000 7f3450a53740 10 client.25361 fetch_fsmap learned FSMap version 27 2024-03-18T16:09:45.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.357+0000 7f3450a53740 1 -- 172.21.15.39:0/1330321973 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55bef1def440 con 0x55bef1d72750 2024-03-18T16:09:45.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.357+0000 7f3430c00640 1 -- 172.21.15.39:0/1330321973 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f344004c070 con 0x55bef1d72750 2024-03-18T16:09:45.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.357+0000 7f3430c00640 1 -- 172.21.15.39:0/1330321973 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f3438005310 con 0x55bef1d72750 2024-03-18T16:09:45.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.357+0000 7f3450a53740 10 client.25361 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T16:09:45.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.357+0000 7f3450a53740 10 client.25361 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-18T16:09:45.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.357+0000 7f3450a53740 20 client.25361 populate_metadata read hostname 'smithi039' 2024-03-18T16:09:45.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.357+0000 7f3450a53740 1 --2- 172.21.15.39:0/1330321973 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x55bef1df3fc0 0x55bef1df2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:45.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.357+0000 7f3450a53740 4 client.25361 mds_command: new command op to 14463 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T16:09:45.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.357+0000 7f3450a53740 1 -- 172.21.15.39:0/1330321973 --> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] -- command(tid 0: {"prefix": "status"}) -- 0x55bef1ce0190 con 0x55bef1df3fc0 2024-03-18T16:09:45.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.358+0000 7f3446000640 1 --2- 172.21.15.39:0/1330321973 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x55bef1df3fc0 0x55bef1df2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:45.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.358+0000 7f3446000640 1 --2- 172.21.15.39:0/1330321973 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x55bef1df3fc0 0x55bef1df2cd0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=92395faea08e5579 server_cookie=e5a6cf893e6ed64d in_seq=0 out_seq=0 2024-03-18T16:09:45.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.358+0000 7f3430c00640 10 client.25361 ms_handle_connect on v2:172.21.15.39:6840/3515326042 2024-03-18T16:09:45.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.359+0000 7f3430c00640 1 -- 172.21.15.39:0/1330321973 <== mds.4 v2:172.21.15.39:6840/3515326042 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55bef1ce0190 con 0x55bef1df3fc0 2024-03-18T16:09:45.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.359+0000 7f3430c00640 10 client.25361 handle_command_reply: tid=0 2024-03-18T16:09:45.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.359+0000 7f3450a53740 1 client.25361 shutdown 2024-03-18T16:09:45.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.359+0000 7f3450a53740 1 -- 172.21.15.39:0/1330321973 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bef1d72750 msgr2=0x55bef1d7f480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:45.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.359+0000 7f3450a53740 1 --2- 172.21.15.39:0/1330321973 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bef1d72750 0x55bef1d7f480 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f3438000c00 tx=0x7f3438029d00 comp rx=0 tx=0).stop 2024-03-18T16:09:45.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.359+0000 7f3450a53740 1 -- 172.21.15.39:0/1330321973 shutdown_connections 2024-03-18T16:09:45.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.359+0000 7f3450a53740 1 --2- 172.21.15.39:0/1330321973 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x55bef1df3fc0 0x55bef1df2cd0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.359+0000 7f3450a53740 1 --2- 172.21.15.39:0/1330321973 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bef1d782b0 0x55bef1d829e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.359+0000 7f3450a53740 1 --2- 172.21.15.39:0/1330321973 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bef1d73140 0x55bef1d7f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.359+0000 7f3450a53740 1 --2- 172.21.15.39:0/1330321973 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bef1d72750 0x55bef1d7f480 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.359+0000 7f3450a53740 1 -- 172.21.15.39:0/1330321973 >> 172.21.15.39:0/1330321973 conn(0x55bef1d698b0 msgr2=0x55bef1d7c490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:45.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.359+0000 7f3430c00640 0 client.25361 ms_handle_reset on v2:172.21.15.39:6840/3515326042 2024-03-18T16:09:45.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.359+0000 7f3450a53740 1 -- 172.21.15.39:0/1330321973 shutdown_connections 2024-03-18T16:09:45.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.359+0000 7f3450a53740 1 -- 172.21.15.39:0/1330321973 wait complete. 2024-03-18T16:09:45.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.359+0000 7f3450a53740 20 client.25361 trim_cache size 0 max 16384 2024-03-18T16:09:45.366 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-03-18T16:09:45.366 INFO:teuthology.orchestra.run.smithi039.stdout: "cluster_fsid": "1892d8fc-e540-11ee-95c9-87774f69a715", 2024-03-18T16:09:45.366 INFO:teuthology.orchestra.run.smithi039.stdout: "whoami": 4, 2024-03-18T16:09:45.366 INFO:teuthology.orchestra.run.smithi039.stdout: "id": 14463, 2024-03-18T16:09:45.366 INFO:teuthology.orchestra.run.smithi039.stdout: "want_state": "up:active", 2024-03-18T16:09:45.367 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "up:active", 2024-03-18T16:09:45.367 INFO:teuthology.orchestra.run.smithi039.stdout: "fs_name": "cephfs", 2024-03-18T16:09:45.367 INFO:teuthology.orchestra.run.smithi039.stdout: "rank_uptime": 145.37311340700001, 2024-03-18T16:09:45.367 INFO:teuthology.orchestra.run.smithi039.stdout: "mdsmap_epoch": 27, 2024-03-18T16:09:45.367 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_epoch": 93, 2024-03-18T16:09:45.367 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_epoch_barrier": 89, 2024-03-18T16:09:45.367 INFO:teuthology.orchestra.run.smithi039.stdout: "uptime": 190.722687267 2024-03-18T16:09:45.367 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-03-18T16:09:45.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.360+0000 7f3447e00640 1 -- 172.21.15.39:0/1626257695 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f342c0a2800 msgr2=0x7f342c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:45.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.360+0000 7f3447e00640 1 --2- 172.21.15.39:0/1626257695 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f342c0a2800 0x7f342c0a4cc0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x55bef1d73540 tx=0x7f3438005f90 comp rx=0 tx=0).stop 2024-03-18T16:09:45.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.360+0000 7f3447e00640 1 -- 172.21.15.39:0/1626257695 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3448104a70 msgr2=0x7f3448197e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:45.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.360+0000 7f3447e00640 1 --2- 172.21.15.39:0/1626257695 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3448104a70 0x7f3448197e80 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f343400e9c0 tx=0x7f343400ee80 comp rx=0 tx=0).stop 2024-03-18T16:09:45.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.360+0000 7f3447e00640 1 -- 172.21.15.39:0/1626257695 shutdown_connections 2024-03-18T16:09:45.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.360+0000 7f3447e00640 1 --2- 172.21.15.39:0/1626257695 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f342c0a2800 0x7f342c0a4cc0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.360+0000 7f3447e00640 1 --2- 172.21.15.39:0/1626257695 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f34481073a0 0x7f344819eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.360+0000 7f3447e00640 1 --2- 172.21.15.39:0/1626257695 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3448104a70 0x7f3448197e80 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.360+0000 7f3447e00640 1 --2- 172.21.15.39:0/1626257695 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3448102140 0x7f3448197940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.360+0000 7f3447e00640 1 -- 172.21.15.39:0/1626257695 >> 172.21.15.39:0/1626257695 conn(0x7f34480fbb50 msgr2=0x7f3448102a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:45.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.360+0000 7f3447e00640 1 -- 172.21.15.39:0/1626257695 shutdown_connections 2024-03-18T16:09:45.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.360+0000 7f3447e00640 1 -- 172.21.15.39:0/1626257695 wait complete. 2024-03-18T16:09:45.377 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:45 smithi039 ceph-mon[31721]: pgmap v412: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T16:09:45.377 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:45 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/3934060100' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:09:45.377 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:45 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/4108733474' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:09:45.380 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-03-18T16:09:45.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:45 smithi196 ceph-mon[36307]: pgmap v412: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T16:09:45.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:45 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/3934060100' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:09:45.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:45 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/4108733474' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:09:45.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:45 smithi194 ceph-mon[36200]: pgmap v412: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T16:09:45.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:45 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/3934060100' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:09:45.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:45 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/4108733474' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:09:45.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.586+0000 7f7694a00640 1 -- 172.21.15.39:0/154787570 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f76901016e0 msgr2=0x7f7690101b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:45.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.586+0000 7f7694a00640 1 --2- 172.21.15.39:0/154787570 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f76901016e0 0x7f7690101b00 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f7680009ca0 tx=0x7f768002c100 comp rx=0 tx=0).stop 2024-03-18T16:09:45.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.586+0000 7f7694a00640 1 -- 172.21.15.39:0/154787570 shutdown_connections 2024-03-18T16:09:45.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.586+0000 7f7694a00640 1 --2- 172.21.15.39:0/154787570 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7690103290 0x7f7690109b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.586+0000 7f7694a00640 1 --2- 172.21.15.39:0/154787570 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f76901028d0 0x7f7690102d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.586+0000 7f7694a00640 1 --2- 172.21.15.39:0/154787570 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f76901016e0 0x7f7690101b00 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.586+0000 7f7694a00640 1 -- 172.21.15.39:0/154787570 >> 172.21.15.39:0/154787570 conn(0x7f76900fcc50 msgr2=0x7f76900ff090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:45.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.586+0000 7f7694a00640 1 -- 172.21.15.39:0/154787570 shutdown_connections 2024-03-18T16:09:45.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.586+0000 7f7694a00640 1 -- 172.21.15.39:0/154787570 wait complete. 2024-03-18T16:09:45.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.587+0000 7f7694a00640 1 Processor -- start 2024-03-18T16:09:45.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.587+0000 7f7694a00640 1 -- start start 2024-03-18T16:09:45.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.587+0000 7f7694a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f76901016e0 0x7f7690075590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:45.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.587+0000 7f7694a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f76901028d0 0x7f7690073bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:45.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.587+0000 7f7694a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7690103290 0x7f7690074110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:45.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.587+0000 7f7694a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f769010d330 con 0x7f7690103290 2024-03-18T16:09:45.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.587+0000 7f7694a00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f769010d1b0 con 0x7f76901016e0 2024-03-18T16:09:45.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.587+0000 7f7694a00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f769010d4b0 con 0x7f76901028d0 2024-03-18T16:09:45.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.588+0000 7f768f400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f76901016e0 0x7f7690075590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:45.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.588+0000 7f768f400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f76901016e0 0x7f7690075590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:38740/0 (socket says 172.21.15.39:38740) 2024-03-18T16:09:45.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.588+0000 7f768f400640 1 -- 172.21.15.39:0/2508270193 learned_addr learned my addr 172.21.15.39:0/2508270193 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:45.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.588+0000 7f768fe00640 1 --2- 172.21.15.39:0/2508270193 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7690103290 0x7f7690074110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:45.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.588+0000 7f7686a00640 1 --2- 172.21.15.39:0/2508270193 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f76901028d0 0x7f7690073bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:45.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.588+0000 7f768f400640 1 -- 172.21.15.39:0/2508270193 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f76901028d0 msgr2=0x7f7690073bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:45.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.588+0000 7f768f400640 1 --2- 172.21.15.39:0/2508270193 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f76901028d0 0x7f7690073bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.588+0000 7f768f400640 1 -- 172.21.15.39:0/2508270193 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7690103290 msgr2=0x7f7690074110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:45.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.588+0000 7f768f400640 1 --2- 172.21.15.39:0/2508270193 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7690103290 0x7f7690074110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.588+0000 7f768f400640 1 -- 172.21.15.39:0/2508270193 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76900749a0 con 0x7f76901016e0 2024-03-18T16:09:45.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.589+0000 7f768f400640 1 --2- 172.21.15.39:0/2508270193 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f76901016e0 0x7f7690075590 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f7680000f80 tx=0x7f7680035d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:45.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.589+0000 7f768cc00640 1 -- 172.21.15.39:0/2508270193 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7680039040 con 0x7f76901016e0 2024-03-18T16:09:45.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.589+0000 7f768cc00640 1 -- 172.21.15.39:0/2508270193 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7680044070 con 0x7f76901016e0 2024-03-18T16:09:45.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.589+0000 7f7694a00640 1 -- 172.21.15.39:0/2508270193 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76901a1a90 con 0x7f76901016e0 2024-03-18T16:09:45.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.589+0000 7f768cc00640 1 -- 172.21.15.39:0/2508270193 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7680004330 con 0x7f76901016e0 2024-03-18T16:09:45.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.589+0000 7f7694a00640 1 -- 172.21.15.39:0/2508270193 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76901a1fb0 con 0x7f76901016e0 2024-03-18T16:09:45.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.592+0000 7f7698c83740 1 Processor -- start 2024-03-18T16:09:45.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.592+0000 7f7698c83740 1 -- start start 2024-03-18T16:09:45.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.593+0000 7f7698c83740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556653f32340 0x556653f32760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:45.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.593+0000 7f7698c83740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556653f32d30 0x556653f37830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:45.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.593+0000 7f7698c83740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556653f37ea0 0x556653f3a330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:45.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.593+0000 7f7698c83740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x556653e96c50 con 0x556653f32340 2024-03-18T16:09:45.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.593+0000 7f7698c83740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x556653e3f9d0 con 0x556653f32d30 2024-03-18T16:09:45.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.593+0000 7f7698c83740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x556653e92cf0 con 0x556653f37ea0 2024-03-18T16:09:45.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.593+0000 7f768fe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556653f32340 0x556653f32760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:45.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.593+0000 7f7686a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556653f37ea0 0x556653f3a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:45.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.593+0000 7f768fe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556653f32340 0x556653f32760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:37446/0 (socket says 172.21.15.39:37446) 2024-03-18T16:09:45.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.593+0000 7f768fe00640 1 -- 172.21.15.39:0/2651600326 learned_addr learned my addr 172.21.15.39:0/2651600326 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:45.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.593+0000 7f7686a00640 1 -- 172.21.15.39:0/2651600326 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556653f32d30 msgr2=0x556653f37830 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:09:45.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.593+0000 7f7686a00640 1 --2- 172.21.15.39:0/2651600326 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556653f32d30 0x556653f37830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.593+0000 7f7686a00640 1 -- 172.21.15.39:0/2651600326 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556653f32340 msgr2=0x556653f32760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:45.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.593+0000 7f7686a00640 1 --2- 172.21.15.39:0/2651600326 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556653f32340 0x556653f32760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.593+0000 7f768cc00640 1 -- 172.21.15.39:0/2508270193 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f76800044d0 con 0x7f76901016e0 2024-03-18T16:09:45.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.593+0000 7f7686a00640 1 -- 172.21.15.39:0/2651600326 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556653e59f10 con 0x556653f37ea0 2024-03-18T16:09:45.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.594+0000 7f7686a00640 1 --2- 172.21.15.39:0/2651600326 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556653f37ea0 0x556653f3a330 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f7678009bf0 tx=0x7f7678011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=af324ce761e77251 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:45.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.594+0000 7f767fe00640 1 -- 172.21.15.39:0/2651600326 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7678004050 con 0x556653f37ea0 2024-03-18T16:09:45.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.594+0000 7f767fe00640 1 -- 172.21.15.39:0/2651600326 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7678005500 con 0x556653f37ea0 2024-03-18T16:09:45.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.594+0000 7f767fe00640 1 -- 172.21.15.39:0/2651600326 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7678004e40 con 0x556653f37ea0 2024-03-18T16:09:45.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.595+0000 7f7698c83740 1 -- 172.21.15.39:0/2651600326 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556653f37ea0 msgr2=0x556653f3a330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:45.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.595+0000 7f7698c83740 1 --2- 172.21.15.39:0/2651600326 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556653f37ea0 0x556653f3a330 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f7678009bf0 tx=0x7f7678011810 comp rx=0 tx=0).stop 2024-03-18T16:09:45.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.595+0000 7f7698c83740 1 -- 172.21.15.39:0/2651600326 shutdown_connections 2024-03-18T16:09:45.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.595+0000 7f7698c83740 1 --2- 172.21.15.39:0/2651600326 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556653f32d30 0x556653f37830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.595+0000 7f7698c83740 1 --2- 172.21.15.39:0/2651600326 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556653f37ea0 0x556653f3a330 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.595+0000 7f7698c83740 1 --2- 172.21.15.39:0/2651600326 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556653f32340 0x556653f32760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.595+0000 7f7698c83740 1 -- 172.21.15.39:0/2651600326 >> 172.21.15.39:0/2651600326 conn(0x556653f2d8b0 msgr2=0x556653f2dcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:45.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.595+0000 7f7698c83740 1 -- 172.21.15.39:0/2651600326 shutdown_connections 2024-03-18T16:09:45.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.595+0000 7f7698c83740 1 -- 172.21.15.39:0/2651600326 wait complete. 2024-03-18T16:09:45.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.595+0000 7f7698c83740 1 -- start start 2024-03-18T16:09:45.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.595+0000 7f7698c83740 1 Processor -- start 2024-03-18T16:09:45.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.596+0000 7f7698c83740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556653f32340 0x556653f3c520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:45.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.596+0000 7f7698c83740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556653f32d30 0x556653f3ca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:45.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.596+0000 7f7698c83740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556653f37ea0 0x556653f43aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:45.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.596+0000 7f7698c83740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x556653f3ae50 con 0x556653f32340 2024-03-18T16:09:45.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.596+0000 7f7698c83740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x556653e52e20 con 0x556653f32d30 2024-03-18T16:09:45.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.596+0000 7f7698c83740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x556653f3afd0 con 0x556653f37ea0 2024-03-18T16:09:45.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.596+0000 7f768cc00640 1 --2- 172.21.15.39:0/2508270193 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f76740a2710 0x7f76740a4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:45.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.596+0000 7f7698c83740 10 client.0 _ll_register_callbacks cb 0x556653e527f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:09:45.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.596+0000 7f7686a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556653f37ea0 0x556653f43aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:45.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.596+0000 7f7686a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556653f37ea0 0x556653f43aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:56208/0 (socket says 172.21.15.39:56208) 2024-03-18T16:09:45.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.596+0000 7f7686a00640 1 -- 172.21.15.39:0/2216743596 learned_addr learned my addr 172.21.15.39:0/2216743596 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:45.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.596+0000 7f768fe00640 1 --2- 172.21.15.39:0/2216743596 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556653f32340 0x556653f3c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:45.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.596+0000 7f768f400640 1 --2- 172.21.15.39:0/2216743596 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556653f32d30 0x556653f3ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:45.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.597+0000 7f768fe00640 1 --2- 172.21.15.39:0/2508270193 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f76740a2710 0x7f76740a4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:45.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.597+0000 7f768cc00640 1 -- 172.21.15.39:0/2508270193 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f76800f13d0 con 0x7f76901016e0 2024-03-18T16:09:45.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.597+0000 7f7686a00640 1 -- 172.21.15.39:0/2216743596 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556653f32d30 msgr2=0x556653f3ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:45.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.597+0000 7f7686a00640 1 --2- 172.21.15.39:0/2216743596 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556653f32d30 0x556653f3ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.597+0000 7f7686a00640 1 -- 172.21.15.39:0/2216743596 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556653f32340 msgr2=0x556653f3c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:45.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.597+0000 7f7686a00640 1 --2- 172.21.15.39:0/2216743596 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556653f32340 0x556653f3c520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.597+0000 7f7686a00640 1 -- 172.21.15.39:0/2216743596 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556653bab2d0 con 0x556653f37ea0 2024-03-18T16:09:45.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.597+0000 7f768fe00640 1 --2- 172.21.15.39:0/2216743596 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556653f32340 0x556653f3c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:09:45.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.597+0000 7f7686a00640 1 --2- 172.21.15.39:0/2216743596 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556653f37ea0 0x556653f43aa0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f767801b040 tx=0x7f76780098b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6aae059b632de71f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:45.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.597+0000 7f767cc00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:09:45.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.597+0000 7f768fe00640 1 --2- 172.21.15.39:0/2508270193 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f76740a2710 0x7f76740a4bd0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f7688009fd0 tx=0x7f7688012040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:45.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.598+0000 7f767cc00640 1 -- 172.21.15.39:0/2216743596 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7678029070 con 0x556653f37ea0 2024-03-18T16:09:45.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.598+0000 7f767cc00640 1 -- 172.21.15.39:0/2216743596 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f7678004d80 con 0x556653f37ea0 2024-03-18T16:09:45.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.598+0000 7f7698c83740 1 -- 172.21.15.39:0/2216743596 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7678005500 con 0x556653f37ea0 2024-03-18T16:09:45.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.598+0000 7f767cc00640 1 -- 172.21.15.39:0/2216743596 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f7678035050 con 0x556653f37ea0 2024-03-18T16:09:45.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.598+0000 7f7698c83740 10 client.25393 fetch_fsmap learned FSMap version 27 2024-03-18T16:09:45.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.598+0000 7f7698c83740 1 -- 172.21.15.39:0/2216743596 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x556653c756b0 con 0x556653f37ea0 2024-03-18T16:09:45.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.598+0000 7f767cc00640 1 -- 172.21.15.39:0/2216743596 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7678005500 con 0x556653f37ea0 2024-03-18T16:09:45.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.599+0000 7f767cc00640 1 -- 172.21.15.39:0/2216743596 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f7678023430 con 0x556653f37ea0 2024-03-18T16:09:45.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.599+0000 7f7698c83740 10 client.25393 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T16:09:45.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.599+0000 7f7698c83740 10 client.25393 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-18T16:09:45.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.599+0000 7f7698c83740 20 client.25393 populate_metadata read hostname 'smithi039' 2024-03-18T16:09:45.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.599+0000 7f7698c83740 1 --2- 172.21.15.39:0/2216743596 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x556653f49010 0x556653f4a480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:45.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.599+0000 7f7698c83740 4 client.25393 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:09:45.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.599+0000 7f7698c83740 1 -- 172.21.15.39:0/2216743596 --> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556653ef42b0 con 0x556653f49010 2024-03-18T16:09:45.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.599+0000 7f768f400640 1 --2- 172.21.15.39:0/2216743596 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x556653f49010 0x556653f4a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:45.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.600+0000 7f768f400640 1 --2- 172.21.15.39:0/2216743596 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x556653f49010 0x556653f4a480 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=a163efc4cab407 server_cookie=68705cbb2ff0326d in_seq=0 out_seq=0 2024-03-18T16:09:45.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.600+0000 7f767cc00640 10 client.25393 ms_handle_connect on v2:172.21.15.194:6836/1301464246 2024-03-18T16:09:45.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.601+0000 7f767cc00640 1 -- 172.21.15.39:0/2216743596 <== mds.3 v2:172.21.15.194:6836/1301464246 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556653ef42b0 con 0x556653f49010 2024-03-18T16:09:45.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.601+0000 7f767cc00640 10 client.25393 handle_command_reply: tid=0 2024-03-18T16:09:45.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.601+0000 7f7698c83740 1 client.25393 shutdown 2024-03-18T16:09:45.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.601+0000 7f7698c83740 1 -- 172.21.15.39:0/2216743596 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556653f37ea0 msgr2=0x556653f43aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:45.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.601+0000 7f7698c83740 1 --2- 172.21.15.39:0/2216743596 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556653f37ea0 0x556653f43aa0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f767801b040 tx=0x7f76780098b0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.601+0000 7f7698c83740 1 -- 172.21.15.39:0/2216743596 shutdown_connections 2024-03-18T16:09:45.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.601+0000 7f7698c83740 1 --2- 172.21.15.39:0/2216743596 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x556653f49010 0x556653f4a480 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.601+0000 7f7698c83740 1 --2- 172.21.15.39:0/2216743596 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556653f32d30 0x556653f3ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.601+0000 7f7698c83740 1 --2- 172.21.15.39:0/2216743596 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556653f37ea0 0x556653f43aa0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.601+0000 7f7698c83740 1 --2- 172.21.15.39:0/2216743596 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556653f32340 0x556653f3c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.601+0000 7f767cc00640 0 client.25393 ms_handle_reset on v2:172.21.15.194:6836/1301464246 2024-03-18T16:09:45.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.601+0000 7f7698c83740 1 -- 172.21.15.39:0/2216743596 >> 172.21.15.39:0/2216743596 conn(0x556653f2d8b0 msgr2=0x556653f2ca70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:45.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.601+0000 7f7698c83740 1 -- 172.21.15.39:0/2216743596 shutdown_connections 2024-03-18T16:09:45.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.601+0000 7f7698c83740 1 -- 172.21.15.39:0/2216743596 wait complete. 2024-03-18T16:09:45.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.601+0000 7f7698c83740 20 client.25393 trim_cache size 0 max 16384 2024-03-18T16:09:45.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.621+0000 7f7698c83740 1 Processor -- start 2024-03-18T16:09:45.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.622+0000 7f7698c83740 1 -- start start 2024-03-18T16:09:45.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.622+0000 7f7698c83740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556653f32340 0x556653f37540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:45.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.622+0000 7f7698c83740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556653f32d30 0x556653f439c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:45.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.622+0000 7f7698c83740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556653f37ea0 0x556653f337d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:45.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.622+0000 7f7698c83740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x556653f487c0 con 0x556653f32d30 2024-03-18T16:09:45.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.622+0000 7f7698c83740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x556653f48e90 con 0x556653f32340 2024-03-18T16:09:45.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.622+0000 7f7698c83740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x556653f49560 con 0x556653f37ea0 2024-03-18T16:09:45.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.622+0000 7f7686a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556653f32d30 0x556653f439c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:45.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.622+0000 7f768f400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556653f32340 0x556653f37540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:45.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.622+0000 7f768fe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556653f37ea0 0x556653f337d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:45.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.622+0000 7f768f400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556653f32340 0x556653f37540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:38756/0 (socket says 172.21.15.39:38756) 2024-03-18T16:09:45.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.622+0000 7f768f400640 1 -- 172.21.15.39:0/1075636040 learned_addr learned my addr 172.21.15.39:0/1075636040 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:45.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.623+0000 7f768f400640 1 -- 172.21.15.39:0/1075636040 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556653f37ea0 msgr2=0x556653f337d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:45.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.623+0000 7f768f400640 1 --2- 172.21.15.39:0/1075636040 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556653f37ea0 0x556653f337d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.623+0000 7f768f400640 1 -- 172.21.15.39:0/1075636040 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556653f32d30 msgr2=0x556653f439c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:45.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.623+0000 7f768f400640 1 --2- 172.21.15.39:0/1075636040 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556653f32d30 0x556653f439c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.623+0000 7f768f400640 1 -- 172.21.15.39:0/1075636040 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556653f485f0 con 0x556653f32340 2024-03-18T16:09:45.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.623+0000 7f768f400640 1 --2- 172.21.15.39:0/1075636040 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556653f32340 0x556653f37540 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f76800ba3b0 tx=0x7f76800c1040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c71a8e64248d2e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:45.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.623+0000 7f767f400640 1 -- 172.21.15.39:0/1075636040 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76800be0e0 con 0x556653f32340 2024-03-18T16:09:45.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.623+0000 7f767f400640 1 -- 172.21.15.39:0/1075636040 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76800bd9e0 con 0x556653f32340 2024-03-18T16:09:45.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.623+0000 7f767f400640 1 -- 172.21.15.39:0/1075636040 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76800f1210 con 0x556653f32340 2024-03-18T16:09:45.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.624+0000 7f7698c83740 1 -- 172.21.15.39:0/1075636040 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556653f32340 msgr2=0x556653f37540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:45.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.624+0000 7f7698c83740 1 --2- 172.21.15.39:0/1075636040 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556653f32340 0x556653f37540 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f76800ba3b0 tx=0x7f76800c1040 comp rx=0 tx=0).stop 2024-03-18T16:09:45.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.624+0000 7f7698c83740 1 -- 172.21.15.39:0/1075636040 shutdown_connections 2024-03-18T16:09:45.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.624+0000 7f7698c83740 1 --2- 172.21.15.39:0/1075636040 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556653f37ea0 0x556653f337d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.624+0000 7f7698c83740 1 --2- 172.21.15.39:0/1075636040 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556653f32d30 0x556653f439c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.624+0000 7f7698c83740 1 --2- 172.21.15.39:0/1075636040 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556653f32340 0x556653f37540 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.624+0000 7f7698c83740 1 -- 172.21.15.39:0/1075636040 >> 172.21.15.39:0/1075636040 conn(0x556653f2d8b0 msgr2=0x556653f2dc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:45.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.624+0000 7f7698c83740 1 -- 172.21.15.39:0/1075636040 shutdown_connections 2024-03-18T16:09:45.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.624+0000 7f7698c83740 1 -- 172.21.15.39:0/1075636040 wait complete. 2024-03-18T16:09:45.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.624+0000 7f7698c83740 1 -- start start 2024-03-18T16:09:45.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.624+0000 7f7698c83740 1 Processor -- start 2024-03-18T16:09:45.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.625+0000 7f7698c83740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556653f32340 0x556653f3ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:45.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.625+0000 7f7698c83740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556653f32d30 0x556653f3f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:45.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.625+0000 7f7698c83740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556653f37ea0 0x556653f423e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:45.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.625+0000 7f7698c83740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x556653f4a300 con 0x556653f32d30 2024-03-18T16:09:45.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.625+0000 7f7698c83740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x556653f49c30 con 0x556653f37ea0 2024-03-18T16:09:45.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.625+0000 7f7698c83740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x556653f4a9d0 con 0x556653f32340 2024-03-18T16:09:45.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.625+0000 7f7698c83740 10 client.0 _ll_register_callbacks cb 0x7f76780043f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:09:45.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.625+0000 7f7686a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556653f32d30 0x556653f3f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:45.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.625+0000 7f768f400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556653f32340 0x556653f3ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:45.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.625+0000 7f768fe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556653f37ea0 0x556653f423e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:45.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.625+0000 7f768f400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556653f32340 0x556653f3ee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:56220/0 (socket says 172.21.15.39:56220) 2024-03-18T16:09:45.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.625+0000 7f768f400640 1 -- 172.21.15.39:0/2499871062 learned_addr learned my addr 172.21.15.39:0/2499871062 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:45.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.626+0000 7f768fe00640 1 -- 172.21.15.39:0/2499871062 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556653f32340 msgr2=0x556653f3ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:45.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.626+0000 7f768fe00640 1 --2- 172.21.15.39:0/2499871062 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556653f32340 0x556653f3ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.626+0000 7f768fe00640 1 -- 172.21.15.39:0/2499871062 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556653f32d30 msgr2=0x556653f3f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:45.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.626+0000 7f768fe00640 1 --2- 172.21.15.39:0/2499871062 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556653f32d30 0x556653f3f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.626+0000 7f768fe00640 1 -- 172.21.15.39:0/2499871062 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556653f429d0 con 0x556653f37ea0 2024-03-18T16:09:45.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.626+0000 7f768f400640 1 --2- 172.21.15.39:0/2499871062 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556653f32340 0x556653f3ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:09:45.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.626+0000 7f7686a00640 1 --2- 172.21.15.39:0/2499871062 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556653f32d30 0x556653f3f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:09:45.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.626+0000 7f768fe00640 1 --2- 172.21.15.39:0/2499871062 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556653f37ea0 0x556653f423e0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f7688012010 tx=0x7f7688004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=771952b2e2a54327 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:45.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.626+0000 7f767cc00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:09:45.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.626+0000 7f767cc00640 1 -- 172.21.15.39:0/2499871062 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7688019040 con 0x556653f37ea0 2024-03-18T16:09:45.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.626+0000 7f7698c83740 1 -- 172.21.15.39:0/2499871062 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f76800bd9e0 con 0x556653f37ea0 2024-03-18T16:09:45.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.626+0000 7f767cc00640 1 -- 172.21.15.39:0/2499871062 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f768801fab0 con 0x556653f37ea0 2024-03-18T16:09:45.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.627+0000 7f767cc00640 1 -- 172.21.15.39:0/2499871062 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f768801fcb0 con 0x556653f37ea0 2024-03-18T16:09:45.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.627+0000 7f7698c83740 10 client.25376 fetch_fsmap learned FSMap version 27 2024-03-18T16:09:45.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.627+0000 7f7698c83740 1 -- 172.21.15.39:0/2499871062 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x556653f43d90 con 0x556653f37ea0 2024-03-18T16:09:45.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.627+0000 7f767cc00640 1 -- 172.21.15.39:0/2499871062 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76800bd9e0 con 0x556653f37ea0 2024-03-18T16:09:45.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.627+0000 7f767cc00640 1 -- 172.21.15.39:0/2499871062 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f7688004b70 con 0x556653f37ea0 2024-03-18T16:09:45.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.627+0000 7f7698c83740 10 client.25376 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T16:09:45.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.627+0000 7f7698c83740 10 client.25376 resolve_mds: resolved name 'h' to daemon mds.h 2024-03-18T16:09:45.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.627+0000 7f7698c83740 20 client.25376 populate_metadata read hostname 'smithi039' 2024-03-18T16:09:45.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.627+0000 7f7698c83740 1 --2- 172.21.15.39:0/2499871062 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x556653fb3700 0x556653fb2410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:45.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.627+0000 7f7698c83740 4 client.25376 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T16:09:45.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.627+0000 7f7698c83740 1 -- 172.21.15.39:0/2499871062 --> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] -- command(tid 0: {"prefix": "status"}) -- 0x556653ea4190 con 0x556653fb3700 2024-03-18T16:09:45.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.627+0000 7f768f400640 1 --2- 172.21.15.39:0/2499871062 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x556653fb3700 0x556653fb2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:45.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.628+0000 7f768f400640 1 --2- 172.21.15.39:0/2499871062 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x556653fb3700 0x556653fb2410 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=b0c0c49a3c566502 server_cookie=efee91d56f9a3076 in_seq=0 out_seq=0 2024-03-18T16:09:45.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.628+0000 7f767cc00640 10 client.25376 ms_handle_connect on v2:172.21.15.194:6836/1301464246 2024-03-18T16:09:45.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.628+0000 7f767cc00640 1 -- 172.21.15.39:0/2499871062 <== mds.3 v2:172.21.15.194:6836/1301464246 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x556653ea4190 con 0x556653fb3700 2024-03-18T16:09:45.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.628+0000 7f767cc00640 10 client.25376 handle_command_reply: tid=0 2024-03-18T16:09:45.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.628+0000 7f7698c83740 1 client.25376 shutdown 2024-03-18T16:09:45.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.628+0000 7f7698c83740 1 -- 172.21.15.39:0/2499871062 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556653f37ea0 msgr2=0x556653f423e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:45.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.628+0000 7f7698c83740 1 --2- 172.21.15.39:0/2499871062 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556653f37ea0 0x556653f423e0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f7688012010 tx=0x7f7688004730 comp rx=0 tx=0).stop 2024-03-18T16:09:45.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.628+0000 7f7698c83740 1 -- 172.21.15.39:0/2499871062 shutdown_connections 2024-03-18T16:09:45.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.628+0000 7f7698c83740 1 --2- 172.21.15.39:0/2499871062 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x556653fb3700 0x556653fb2410 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.628+0000 7f7698c83740 1 --2- 172.21.15.39:0/2499871062 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556653f37ea0 0x556653f423e0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.628+0000 7f7698c83740 1 --2- 172.21.15.39:0/2499871062 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556653f32d30 0x556653f3f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.628+0000 7f7698c83740 1 --2- 172.21.15.39:0/2499871062 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556653f32340 0x556653f3ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.628+0000 7f7698c83740 1 -- 172.21.15.39:0/2499871062 >> 172.21.15.39:0/2499871062 conn(0x556653f2d8b0 msgr2=0x556653f37140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:45.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.628+0000 7f767cc00640 0 client.25376 ms_handle_reset on v2:172.21.15.194:6836/1301464246 2024-03-18T16:09:45.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.629+0000 7f7698c83740 1 -- 172.21.15.39:0/2499871062 shutdown_connections 2024-03-18T16:09:45.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.629+0000 7f7698c83740 1 -- 172.21.15.39:0/2499871062 wait complete. 2024-03-18T16:09:45.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.629+0000 7f7698c83740 20 client.25376 trim_cache size 0 max 16384 2024-03-18T16:09:45.636 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-03-18T16:09:45.636 INFO:teuthology.orchestra.run.smithi039.stdout: "cluster_fsid": "1892d8fc-e540-11ee-95c9-87774f69a715", 2024-03-18T16:09:45.636 INFO:teuthology.orchestra.run.smithi039.stdout: "whoami": 3, 2024-03-18T16:09:45.636 INFO:teuthology.orchestra.run.smithi039.stdout: "id": 24524, 2024-03-18T16:09:45.636 INFO:teuthology.orchestra.run.smithi039.stdout: "want_state": "up:active", 2024-03-18T16:09:45.636 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "up:active", 2024-03-18T16:09:45.636 INFO:teuthology.orchestra.run.smithi039.stdout: "fs_name": "cephfs", 2024-03-18T16:09:45.636 INFO:teuthology.orchestra.run.smithi039.stdout: "rank_uptime": 146.39699544000001, 2024-03-18T16:09:45.636 INFO:teuthology.orchestra.run.smithi039.stdout: "mdsmap_epoch": 27, 2024-03-18T16:09:45.636 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_epoch": 93, 2024-03-18T16:09:45.636 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_epoch_barrier": 89, 2024-03-18T16:09:45.636 INFO:teuthology.orchestra.run.smithi039.stdout: "uptime": 202.374355186 2024-03-18T16:09:45.636 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-03-18T16:09:45.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.629+0000 7f7694a00640 1 -- 172.21.15.39:0/2508270193 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f76740a2710 msgr2=0x7f76740a4bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:45.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.629+0000 7f7694a00640 1 --2- 172.21.15.39:0/2508270193 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f76740a2710 0x7f76740a4bd0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f7688009fd0 tx=0x7f7688012040 comp rx=0 tx=0).stop 2024-03-18T16:09:45.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.629+0000 7f7694a00640 1 -- 172.21.15.39:0/2508270193 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f76901016e0 msgr2=0x7f7690075590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:45.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.629+0000 7f7694a00640 1 --2- 172.21.15.39:0/2508270193 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f76901016e0 0x7f7690075590 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f7680000f80 tx=0x7f7680035d10 comp rx=0 tx=0).stop 2024-03-18T16:09:45.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.629+0000 7f7694a00640 1 -- 172.21.15.39:0/2508270193 shutdown_connections 2024-03-18T16:09:45.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.629+0000 7f7694a00640 1 --2- 172.21.15.39:0/2508270193 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f76740a2710 0x7f76740a4bd0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.629+0000 7f7694a00640 1 --2- 172.21.15.39:0/2508270193 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7690103290 0x7f7690074110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.629+0000 7f7694a00640 1 --2- 172.21.15.39:0/2508270193 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f76901028d0 0x7f7690073bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.629+0000 7f7694a00640 1 --2- 172.21.15.39:0/2508270193 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f76901016e0 0x7f7690075590 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.629+0000 7f7694a00640 1 -- 172.21.15.39:0/2508270193 >> 172.21.15.39:0/2508270193 conn(0x7f76900fcc50 msgr2=0x7f7690105c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:45.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.630+0000 7f7694a00640 1 -- 172.21.15.39:0/2508270193 shutdown_connections 2024-03-18T16:09:45.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.630+0000 7f7694a00640 1 -- 172.21.15.39:0/2508270193 wait complete. 2024-03-18T16:09:45.649 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-18T16:09:45.863 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.860+0000 7f514be00640 1 -- 172.21.15.39:0/4060612367 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f514c1073a0 msgr2=0x7f514c1097f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:45.863 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.860+0000 7f514be00640 1 --2- 172.21.15.39:0/4060612367 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f514c1073a0 0x7f514c1097f0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f5144009ca0 tx=0x7f514402c100 comp rx=0 tx=0).stop 2024-03-18T16:09:45.863 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.860+0000 7f514be00640 1 -- 172.21.15.39:0/4060612367 shutdown_connections 2024-03-18T16:09:45.863 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.860+0000 7f514be00640 1 --2- 172.21.15.39:0/4060612367 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f514c1073a0 0x7f514c1097f0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.863 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.860+0000 7f514be00640 1 --2- 172.21.15.39:0/4060612367 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f514c104a70 0x7f514c106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.863 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.860+0000 7f514be00640 1 --2- 172.21.15.39:0/4060612367 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f514c102140 0x7f514c104530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.863 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.860+0000 7f514be00640 1 -- 172.21.15.39:0/4060612367 >> 172.21.15.39:0/4060612367 conn(0x7f514c0fbb50 msgr2=0x7f514c0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:45.863 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.860+0000 7f514be00640 1 -- 172.21.15.39:0/4060612367 shutdown_connections 2024-03-18T16:09:45.863 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.860+0000 7f514be00640 1 -- 172.21.15.39:0/4060612367 wait complete. 2024-03-18T16:09:45.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.861+0000 7f514be00640 1 Processor -- start 2024-03-18T16:09:45.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.861+0000 7f514be00640 1 -- start start 2024-03-18T16:09:45.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.862+0000 7f514be00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f514c102140 0x7f514c101ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:45.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.862+0000 7f514be00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f514c104a70 0x7f514c1000f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:45.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.862+0000 7f514be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f514c1073a0 0x7f514c100630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:45.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.862+0000 7f514be00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f514c10d360 con 0x7f514c1073a0 2024-03-18T16:09:45.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.862+0000 7f514be00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f514c10d1e0 con 0x7f514c102140 2024-03-18T16:09:45.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.862+0000 7f514be00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f514c10d4e0 con 0x7f514c104a70 2024-03-18T16:09:45.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.862+0000 7f514aa00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f514c102140 0x7f514c101ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:45.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.862+0000 7f5142000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f514c104a70 0x7f514c1000f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:45.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.862+0000 7f514aa00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f514c102140 0x7f514c101ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:38786/0 (socket says 172.21.15.39:38786) 2024-03-18T16:09:45.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.862+0000 7f5142000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f514c104a70 0x7f514c1000f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:56234/0 (socket says 172.21.15.39:56234) 2024-03-18T16:09:45.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.862+0000 7f514aa00640 1 -- 172.21.15.39:0/3980898297 learned_addr learned my addr 172.21.15.39:0/3980898297 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:45.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.862+0000 7f514b400640 1 --2- 172.21.15.39:0/3980898297 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f514c1073a0 0x7f514c100630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:45.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.862+0000 7f5142000640 1 -- 172.21.15.39:0/3980898297 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f514c102140 msgr2=0x7f514c101ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:45.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.862+0000 7f5142000640 1 --2- 172.21.15.39:0/3980898297 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f514c102140 0x7f514c101ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.863+0000 7f5142000640 1 -- 172.21.15.39:0/3980898297 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f514c1073a0 msgr2=0x7f514c100630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:45.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.863+0000 7f5142000640 1 --2- 172.21.15.39:0/3980898297 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f514c1073a0 0x7f514c100630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.863+0000 7f5142000640 1 -- 172.21.15.39:0/3980898297 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f514c100ec0 con 0x7f514c104a70 2024-03-18T16:09:45.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.863+0000 7f514aa00640 1 --2- 172.21.15.39:0/3980898297 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f514c102140 0x7f514c101ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:09:45.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.863+0000 7f514b400640 1 --2- 172.21.15.39:0/3980898297 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f514c1073a0 0x7f514c100630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:09:45.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.863+0000 7f5142000640 1 --2- 172.21.15.39:0/3980898297 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f514c104a70 0x7f514c1000f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f513400e9c0 tx=0x7f513400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:45.867 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.863+0000 7f5143e00640 1 -- 172.21.15.39:0/3980898297 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f513400ccc0 con 0x7f514c104a70 2024-03-18T16:09:45.867 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.863+0000 7f5143e00640 1 -- 172.21.15.39:0/3980898297 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5134005020 con 0x7f514c104a70 2024-03-18T16:09:45.867 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.863+0000 7f514be00640 1 -- 172.21.15.39:0/3980898297 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f514c1a1a90 con 0x7f514c104a70 2024-03-18T16:09:45.867 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.863+0000 7f5143e00640 1 -- 172.21.15.39:0/3980898297 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5134004a00 con 0x7f514c104a70 2024-03-18T16:09:45.867 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.863+0000 7f514be00640 1 -- 172.21.15.39:0/3980898297 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f514c1a1f90 con 0x7f514c104a70 2024-03-18T16:09:45.868 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.866+0000 7f5154252740 1 Processor -- start 2024-03-18T16:09:45.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.866+0000 7f5154252740 1 -- start start 2024-03-18T16:09:45.870 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.867+0000 7f5154252740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d95a2a9340 0x55d95a2a9760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:45.870 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.867+0000 7f5154252740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d95a2a9d30 0x55d95a2ae830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:45.870 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.867+0000 7f5154252740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d95a2aeea0 0x55d95a2b1330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:45.870 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.867+0000 7f5154252740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55d95a20dc50 con 0x55d95a2a9d30 2024-03-18T16:09:45.870 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.867+0000 7f5154252740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55d95a1b69d0 con 0x55d95a2aeea0 2024-03-18T16:09:45.870 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.867+0000 7f5154252740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55d95a209cf0 con 0x55d95a2a9340 2024-03-18T16:09:45.870 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.867+0000 7f5143e00640 1 -- 172.21.15.39:0/3980898297 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f51340051d0 con 0x7f514c104a70 2024-03-18T16:09:45.871 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.867+0000 7f514b400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d95a2a9340 0x55d95a2a9760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:45.871 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.867+0000 7f514aa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d95a2a9d30 0x55d95a2ae830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:45.871 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.867+0000 7f514b400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d95a2a9340 0x55d95a2a9760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:56242/0 (socket says 172.21.15.39:56242) 2024-03-18T16:09:45.871 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.867+0000 7f514b400640 1 -- 172.21.15.39:0/833097519 learned_addr learned my addr 172.21.15.39:0/833097519 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:45.871 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.867+0000 7f5142000640 1 --2- 172.21.15.39:0/833097519 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d95a2aeea0 0x55d95a2b1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:45.871 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.867+0000 7f514b400640 1 -- 172.21.15.39:0/833097519 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d95a2aeea0 msgr2=0x55d95a2b1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:45.871 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.868+0000 7f514b400640 1 --2- 172.21.15.39:0/833097519 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d95a2aeea0 0x55d95a2b1330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.871 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.868+0000 7f514b400640 1 -- 172.21.15.39:0/833097519 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d95a2a9d30 msgr2=0x55d95a2ae830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:45.871 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.868+0000 7f514b400640 1 --2- 172.21.15.39:0/833097519 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d95a2a9d30 0x55d95a2ae830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.871 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.868+0000 7f514b400640 1 -- 172.21.15.39:0/833097519 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d95a1d0f10 con 0x55d95a2a9340 2024-03-18T16:09:45.871 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.868+0000 7f5142000640 1 --2- 172.21.15.39:0/833097519 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d95a2aeea0 0x55d95a2b1330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:09:45.871 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.868+0000 7f514b400640 1 --2- 172.21.15.39:0/833097519 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d95a2a9340 0x55d95a2a9760 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f514402c710 tx=0x7f51440323d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=703f3f1ecf96d758 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:45.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.868+0000 7f513b400640 1 -- 172.21.15.39:0/833097519 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5144043070 con 0x55d95a2a9340 2024-03-18T16:09:45.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.868+0000 7f513b400640 1 -- 172.21.15.39:0/833097519 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5144032c90 con 0x55d95a2a9340 2024-03-18T16:09:45.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.868+0000 7f513b400640 1 -- 172.21.15.39:0/833097519 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5144039040 con 0x55d95a2a9340 2024-03-18T16:09:45.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.869+0000 7f5154252740 1 -- 172.21.15.39:0/833097519 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d95a2a9340 msgr2=0x55d95a2a9760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:45.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.869+0000 7f5154252740 1 --2- 172.21.15.39:0/833097519 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d95a2a9340 0x55d95a2a9760 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f514402c710 tx=0x7f51440323d0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.869+0000 7f5154252740 1 -- 172.21.15.39:0/833097519 shutdown_connections 2024-03-18T16:09:45.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.869+0000 7f5154252740 1 --2- 172.21.15.39:0/833097519 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d95a2aeea0 0x55d95a2b1330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.869+0000 7f5154252740 1 --2- 172.21.15.39:0/833097519 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d95a2a9d30 0x55d95a2ae830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.869+0000 7f5154252740 1 --2- 172.21.15.39:0/833097519 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d95a2a9340 0x55d95a2a9760 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.869+0000 7f5154252740 1 -- 172.21.15.39:0/833097519 >> 172.21.15.39:0/833097519 conn(0x55d95a2a48b0 msgr2=0x55d95a2a4cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:45.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.869+0000 7f5154252740 1 -- 172.21.15.39:0/833097519 shutdown_connections 2024-03-18T16:09:45.877 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.869+0000 7f5154252740 1 -- 172.21.15.39:0/833097519 wait complete. 2024-03-18T16:09:45.877 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.869+0000 7f5143e00640 1 --2- 172.21.15.39:0/3980898297 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f51300a2710 0x7f51300a4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:45.877 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.870+0000 7f5154252740 1 -- start start 2024-03-18T16:09:45.877 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.870+0000 7f5143e00640 1 -- 172.21.15.39:0/3980898297 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f51340ceda0 con 0x7f514c104a70 2024-03-18T16:09:45.877 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.870+0000 7f514b400640 1 --2- 172.21.15.39:0/3980898297 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f51300a2710 0x7f51300a4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:45.877 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.870+0000 7f5154252740 1 Processor -- start 2024-03-18T16:09:45.877 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.870+0000 7f5154252740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d95a2a9340 0x55d95a2b3450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:45.877 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.870+0000 7f5154252740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d95a2a9d30 0x55d95a2b3990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:45.877 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.870+0000 7f5154252740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d95a2aeea0 0x55d95a2ba9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:45.877 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.870+0000 7f5154252740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55d95a2b1df0 con 0x55d95a2a9d30 2024-03-18T16:09:45.877 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.870+0000 7f5154252740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55d95a1c9e20 con 0x55d95a2a9340 2024-03-18T16:09:45.877 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.870+0000 7f5154252740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55d95a2b1f70 con 0x55d95a2aeea0 2024-03-18T16:09:45.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.871+0000 7f5154252740 10 client.0 _ll_register_callbacks cb 0x55d95a1c97f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:09:45.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.871+0000 7f514b400640 1 --2- 172.21.15.39:0/3980898297 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f51300a2710 0x7f51300a4bd0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f51440324c0 tx=0x7f5144009360 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:45.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.871+0000 7f514aa00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d95a2a9340 0x55d95a2b3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:45.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.871+0000 7f514aa00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d95a2a9340 0x55d95a2b3450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:38816/0 (socket says 172.21.15.39:38816) 2024-03-18T16:09:45.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.871+0000 7f514aa00640 1 -- 172.21.15.39:0/3238990522 learned_addr learned my addr 172.21.15.39:0/3238990522 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:45.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.871+0000 7f5142000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d95a2a9d30 0x55d95a2b3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:45.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.871+0000 7f514b400640 1 --2- 172.21.15.39:0/3238990522 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d95a2aeea0 0x55d95a2ba9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:45.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.871+0000 7f514aa00640 1 -- 172.21.15.39:0/3238990522 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d95a2aeea0 msgr2=0x55d95a2ba9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:45.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.871+0000 7f514aa00640 1 --2- 172.21.15.39:0/3238990522 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d95a2aeea0 0x55d95a2ba9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.871+0000 7f514aa00640 1 -- 172.21.15.39:0/3238990522 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d95a2a9d30 msgr2=0x55d95a2b3990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:45.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.871+0000 7f514aa00640 1 --2- 172.21.15.39:0/3238990522 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d95a2a9d30 0x55d95a2b3990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.871+0000 7f514aa00640 1 -- 172.21.15.39:0/3238990522 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d959f222d0 con 0x55d95a2a9340 2024-03-18T16:09:45.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.871+0000 7f514aa00640 1 --2- 172.21.15.39:0/3238990522 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d95a2a9340 0x55d95a2b3450 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f513c012d20 tx=0x7f513c00f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7d6741bd967f03cd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:45.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.871+0000 7f512fe00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:09:45.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.872+0000 7f512fe00640 1 -- 172.21.15.39:0/3238990522 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f513c012110 con 0x55d95a2a9340 2024-03-18T16:09:45.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.872+0000 7f512fe00640 1 -- 172.21.15.39:0/3238990522 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f513c004050 con 0x55d95a2a9340 2024-03-18T16:09:45.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.872+0000 7f5154252740 1 -- 172.21.15.39:0/3238990522 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5144032c90 con 0x55d95a2a9340 2024-03-18T16:09:45.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.872+0000 7f512fe00640 1 -- 172.21.15.39:0/3238990522 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f513c025050 con 0x55d95a2a9340 2024-03-18T16:09:45.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.872+0000 7f5154252740 10 client.25388 fetch_fsmap learned FSMap version 27 2024-03-18T16:09:45.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.872+0000 7f5154252740 1 -- 172.21.15.39:0/3238990522 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d959fec6b0 con 0x55d95a2a9340 2024-03-18T16:09:45.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.872+0000 7f512fe00640 1 -- 172.21.15.39:0/3238990522 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5144032c90 con 0x55d95a2a9340 2024-03-18T16:09:45.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.873+0000 7f512fe00640 1 -- 172.21.15.39:0/3238990522 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f513c010430 con 0x55d95a2a9340 2024-03-18T16:09:45.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.873+0000 7f5154252740 10 client.25388 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T16:09:45.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.873+0000 7f5154252740 10 client.25388 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-18T16:09:45.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.873+0000 7f5154252740 20 client.25388 populate_metadata read hostname 'smithi039' 2024-03-18T16:09:45.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.873+0000 7f5154252740 1 --2- 172.21.15.39:0/3238990522 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x55d95a2c02c0 0x55d95a2c1730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:45.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.873+0000 7f5154252740 4 client.25388 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:09:45.880 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.873+0000 7f5154252740 1 -- 172.21.15.39:0/3238990522 --> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d95a26b2b0 con 0x55d95a2c02c0 2024-03-18T16:09:45.880 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.873+0000 7f5142000640 1 --2- 172.21.15.39:0/3238990522 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x55d95a2c02c0 0x55d95a2c1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:45.880 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.874+0000 7f5142000640 1 --2- 172.21.15.39:0/3238990522 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x55d95a2c02c0 0x55d95a2c1730 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6227ac48aca29362 server_cookie=9da79da2d8206760 in_seq=0 out_seq=0 2024-03-18T16:09:45.880 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.874+0000 7f512fe00640 10 client.25388 ms_handle_connect on v2:172.21.15.196:6836/895247251 2024-03-18T16:09:45.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.874+0000 7f512fe00640 1 -- 172.21.15.39:0/3238990522 <== mds.2 v2:172.21.15.196:6836/895247251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d95a26b2b0 con 0x55d95a2c02c0 2024-03-18T16:09:45.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.874+0000 7f512fe00640 10 client.25388 handle_command_reply: tid=0 2024-03-18T16:09:45.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.874+0000 7f5154252740 1 client.25388 shutdown 2024-03-18T16:09:45.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.875+0000 7f5154252740 1 -- 172.21.15.39:0/3238990522 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d95a2a9340 msgr2=0x55d95a2b3450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:45.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.875+0000 7f5154252740 1 --2- 172.21.15.39:0/3238990522 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d95a2a9340 0x55d95a2b3450 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f513c012d20 tx=0x7f513c00f790 comp rx=0 tx=0).stop 2024-03-18T16:09:45.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.875+0000 7f5154252740 1 -- 172.21.15.39:0/3238990522 shutdown_connections 2024-03-18T16:09:45.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.875+0000 7f5154252740 1 --2- 172.21.15.39:0/3238990522 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x55d95a2c02c0 0x55d95a2c1730 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.875+0000 7f5154252740 1 --2- 172.21.15.39:0/3238990522 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d95a2aeea0 0x55d95a2ba9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.875+0000 7f5154252740 1 --2- 172.21.15.39:0/3238990522 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d95a2a9d30 0x55d95a2b3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.875+0000 7f5154252740 1 --2- 172.21.15.39:0/3238990522 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d95a2a9340 0x55d95a2b3450 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.875+0000 7f512fe00640 0 client.25388 ms_handle_reset on v2:172.21.15.196:6836/895247251 2024-03-18T16:09:45.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.875+0000 7f5154252740 1 -- 172.21.15.39:0/3238990522 >> 172.21.15.39:0/3238990522 conn(0x55d95a2a48b0 msgr2=0x55d95a2a3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:45.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.875+0000 7f5154252740 1 -- 172.21.15.39:0/3238990522 shutdown_connections 2024-03-18T16:09:45.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.875+0000 7f5154252740 1 -- 172.21.15.39:0/3238990522 wait complete. 2024-03-18T16:09:45.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.875+0000 7f5154252740 20 client.25388 trim_cache size 0 max 16384 2024-03-18T16:09:45.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.895+0000 7f5154252740 1 Processor -- start 2024-03-18T16:09:45.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.895+0000 7f5154252740 1 -- start start 2024-03-18T16:09:45.898 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.895+0000 7f5154252740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d95a2a9340 0x55d95a2b1290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:45.898 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.895+0000 7f5154252740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d95a2a9d30 0x55d95a2ba910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:45.898 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.895+0000 7f5154252740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d95a2aeea0 0x55d95a2a31e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:45.898 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.895+0000 7f5154252740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55d95a2bfa70 con 0x55d95a2aeea0 2024-03-18T16:09:45.898 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.895+0000 7f5154252740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55d95a2c0140 con 0x55d95a2a9340 2024-03-18T16:09:45.898 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.895+0000 7f5154252740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55d95a2c0810 con 0x55d95a2a9d30 2024-03-18T16:09:45.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.896+0000 7f514b400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d95a2aeea0 0x55d95a2a31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:45.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.896+0000 7f514aa00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d95a2a9340 0x55d95a2b1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:45.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.896+0000 7f5142000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d95a2a9d30 0x55d95a2ba910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:45.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.896+0000 7f514aa00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d95a2a9340 0x55d95a2b1290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:38826/0 (socket says 172.21.15.39:38826) 2024-03-18T16:09:45.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.896+0000 7f514aa00640 1 -- 172.21.15.39:0/3593237508 learned_addr learned my addr 172.21.15.39:0/3593237508 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:45.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.896+0000 7f514aa00640 1 -- 172.21.15.39:0/3593237508 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d95a2a9d30 msgr2=0x55d95a2ba910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:45.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.896+0000 7f514aa00640 1 --2- 172.21.15.39:0/3593237508 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d95a2a9d30 0x55d95a2ba910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.896+0000 7f514aa00640 1 -- 172.21.15.39:0/3593237508 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d95a2aeea0 msgr2=0x55d95a2a31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:45.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.896+0000 7f514aa00640 1 --2- 172.21.15.39:0/3593237508 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d95a2aeea0 0x55d95a2a31e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.896+0000 7f514aa00640 1 -- 172.21.15.39:0/3593237508 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d95a2c0cc0 con 0x55d95a2a9340 2024-03-18T16:09:45.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.896+0000 7f5142000640 1 --2- 172.21.15.39:0/3593237508 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d95a2a9d30 0x55d95a2ba910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:09:45.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.896+0000 7f514b400640 1 --2- 172.21.15.39:0/3593237508 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d95a2aeea0 0x55d95a2a31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:09:45.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.896+0000 7f514aa00640 1 --2- 172.21.15.39:0/3593237508 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d95a2a9340 0x55d95a2b1290 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f513c0123d0 tx=0x7f513c005e30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df17462be5a6420b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:45.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.897+0000 7f513aa00640 1 -- 172.21.15.39:0/3593237508 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f513c02e070 con 0x55d95a2a9340 2024-03-18T16:09:45.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.897+0000 7f513aa00640 1 -- 172.21.15.39:0/3593237508 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f513c0104f0 con 0x55d95a2a9340 2024-03-18T16:09:45.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.897+0000 7f513aa00640 1 -- 172.21.15.39:0/3593237508 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f513c0038e0 con 0x55d95a2a9340 2024-03-18T16:09:45.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.897+0000 7f5154252740 1 -- 172.21.15.39:0/3593237508 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d95a2a9340 msgr2=0x55d95a2b1290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:45.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.897+0000 7f5154252740 1 --2- 172.21.15.39:0/3593237508 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d95a2a9340 0x55d95a2b1290 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f513c0123d0 tx=0x7f513c005e30 comp rx=0 tx=0).stop 2024-03-18T16:09:45.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.897+0000 7f5154252740 1 -- 172.21.15.39:0/3593237508 shutdown_connections 2024-03-18T16:09:45.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.897+0000 7f5154252740 1 --2- 172.21.15.39:0/3593237508 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d95a2aeea0 0x55d95a2a31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.897+0000 7f5154252740 1 --2- 172.21.15.39:0/3593237508 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d95a2a9d30 0x55d95a2ba910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.897+0000 7f5154252740 1 --2- 172.21.15.39:0/3593237508 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d95a2a9340 0x55d95a2b1290 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.897+0000 7f5154252740 1 -- 172.21.15.39:0/3593237508 >> 172.21.15.39:0/3593237508 conn(0x55d95a2a48b0 msgr2=0x55d95a2ae510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:45.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.898+0000 7f5154252740 1 -- 172.21.15.39:0/3593237508 shutdown_connections 2024-03-18T16:09:45.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.898+0000 7f5154252740 1 -- 172.21.15.39:0/3593237508 wait complete. 2024-03-18T16:09:45.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.898+0000 7f5154252740 1 -- start start 2024-03-18T16:09:45.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.898+0000 7f5154252740 1 Processor -- start 2024-03-18T16:09:45.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.898+0000 7f5154252740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d95a2a9340 0x55d95a2b6350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:45.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.898+0000 7f5154252740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d95a2a9d30 0x55d95a2b6890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:45.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.898+0000 7f5154252740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d95a2aeea0 0x55d95a2b98b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:45.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.899+0000 7f5154252740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55d95a2c15b0 con 0x55d95a2a9340 2024-03-18T16:09:45.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.899+0000 7f5154252740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55d95a2c0ee0 con 0x55d95a2a9d30 2024-03-18T16:09:45.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.899+0000 7f5154252740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55d95a2c1c80 con 0x55d95a2aeea0 2024-03-18T16:09:45.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.899+0000 7f5154252740 10 client.0 _ll_register_callbacks cb 0x7f513c012930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:09:45.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.899+0000 7f514aa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d95a2a9340 0x55d95a2b6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:45.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.899+0000 7f514b400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d95a2aeea0 0x55d95a2b98b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:45.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.899+0000 7f5142000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d95a2a9d30 0x55d95a2b6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:45.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.899+0000 7f514b400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d95a2aeea0 0x55d95a2b98b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:56264/0 (socket says 172.21.15.39:56264) 2024-03-18T16:09:45.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.899+0000 7f514b400640 1 -- 172.21.15.39:0/3626675365 learned_addr learned my addr 172.21.15.39:0/3626675365 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:45.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.899+0000 7f514b400640 1 -- 172.21.15.39:0/3626675365 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d95a2a9d30 msgr2=0x55d95a2b6890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:45.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.899+0000 7f514b400640 1 --2- 172.21.15.39:0/3626675365 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d95a2a9d30 0x55d95a2b6890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.899+0000 7f514b400640 1 -- 172.21.15.39:0/3626675365 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d95a2a9340 msgr2=0x55d95a2b6350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:45.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.899+0000 7f514b400640 1 --2- 172.21.15.39:0/3626675365 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d95a2a9340 0x55d95a2b6350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.899+0000 7f514b400640 1 -- 172.21.15.39:0/3626675365 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d95a2b9ea0 con 0x55d95a2aeea0 2024-03-18T16:09:45.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.899+0000 7f514aa00640 1 --2- 172.21.15.39:0/3626675365 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d95a2a9340 0x55d95a2b6350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:09:45.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.900+0000 7f514b400640 1 --2- 172.21.15.39:0/3626675365 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d95a2aeea0 0x55d95a2b98b0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f514402c6e0 tx=0x7f514402c100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1edad43ae5dbeb5b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:45.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.900+0000 7f512fe00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:09:45.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.900+0000 7f512fe00640 1 -- 172.21.15.39:0/3626675365 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f514405a070 con 0x55d95a2aeea0 2024-03-18T16:09:45.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.900+0000 7f5154252740 1 -- 172.21.15.39:0/3626675365 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f513c0104f0 con 0x55d95a2aeea0 2024-03-18T16:09:45.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.900+0000 7f512fe00640 1 -- 172.21.15.39:0/3626675365 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f5144056480 con 0x55d95a2aeea0 2024-03-18T16:09:45.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.900+0000 7f512fe00640 1 -- 172.21.15.39:0/3626675365 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5144043040 con 0x55d95a2aeea0 2024-03-18T16:09:45.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.900+0000 7f5154252740 10 client.25417 fetch_fsmap learned FSMap version 27 2024-03-18T16:09:45.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.900+0000 7f5154252740 1 -- 172.21.15.39:0/3626675365 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55d95a32eb10 con 0x55d95a2aeea0 2024-03-18T16:09:45.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.900+0000 7f512fe00640 1 -- 172.21.15.39:0/3626675365 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f513c0104f0 con 0x55d95a2aeea0 2024-03-18T16:09:45.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.900+0000 7f512fe00640 1 -- 172.21.15.39:0/3626675365 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f5144053050 con 0x55d95a2aeea0 2024-03-18T16:09:45.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.901+0000 7f5154252740 10 client.25417 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T16:09:45.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.901+0000 7f5154252740 10 client.25417 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-18T16:09:45.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.901+0000 7f5154252740 20 client.25417 populate_metadata read hostname 'smithi039' 2024-03-18T16:09:45.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.901+0000 7f5154252740 1 --2- 172.21.15.39:0/3626675365 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x55d95a333340 0x55d95a32f780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:45.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.901+0000 7f5154252740 4 client.25417 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T16:09:45.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.901+0000 7f5154252740 1 -- 172.21.15.39:0/3626675365 --> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] -- command(tid 0: {"prefix": "status"}) -- 0x55d95a21b190 con 0x55d95a333340 2024-03-18T16:09:45.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.901+0000 7f514aa00640 1 --2- 172.21.15.39:0/3626675365 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x55d95a333340 0x55d95a32f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:45.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.901+0000 7f514aa00640 1 --2- 172.21.15.39:0/3626675365 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x55d95a333340 0x55d95a32f780 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4df39a08303df4de server_cookie=da25f0e1793e75f0 in_seq=0 out_seq=0 2024-03-18T16:09:45.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.901+0000 7f512fe00640 10 client.25417 ms_handle_connect on v2:172.21.15.196:6836/895247251 2024-03-18T16:09:45.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.902+0000 7f512fe00640 1 -- 172.21.15.39:0/3626675365 <== mds.2 v2:172.21.15.196:6836/895247251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55d95a21b190 con 0x55d95a333340 2024-03-18T16:09:45.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.902+0000 7f512fe00640 10 client.25417 handle_command_reply: tid=0 2024-03-18T16:09:45.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.902+0000 7f5154252740 1 client.25417 shutdown 2024-03-18T16:09:45.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.902+0000 7f5154252740 1 -- 172.21.15.39:0/3626675365 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d95a2aeea0 msgr2=0x55d95a2b98b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:45.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.902+0000 7f5154252740 1 --2- 172.21.15.39:0/3626675365 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d95a2aeea0 0x55d95a2b98b0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f514402c6e0 tx=0x7f514402c100 comp rx=0 tx=0).stop 2024-03-18T16:09:45.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.902+0000 7f5154252740 1 -- 172.21.15.39:0/3626675365 shutdown_connections 2024-03-18T16:09:45.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.902+0000 7f5154252740 1 --2- 172.21.15.39:0/3626675365 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d95a2a9d30 0x55d95a2b6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.902+0000 7f5154252740 1 --2- 172.21.15.39:0/3626675365 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x55d95a333340 0x55d95a32f780 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.902+0000 7f5154252740 1 --2- 172.21.15.39:0/3626675365 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d95a2aeea0 0x55d95a2b98b0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.902+0000 7f5154252740 1 --2- 172.21.15.39:0/3626675365 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d95a2a9340 0x55d95a2b6350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.902+0000 7f512fe00640 0 client.25417 ms_handle_reset on v2:172.21.15.196:6836/895247251 2024-03-18T16:09:45.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.902+0000 7f5154252740 1 -- 172.21.15.39:0/3626675365 >> 172.21.15.39:0/3626675365 conn(0x55d95a2a48b0 msgr2=0x55d95a2a31e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:45.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.902+0000 7f5154252740 1 -- 172.21.15.39:0/3626675365 shutdown_connections 2024-03-18T16:09:45.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.902+0000 7f5154252740 1 -- 172.21.15.39:0/3626675365 wait complete. 2024-03-18T16:09:45.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.902+0000 7f5154252740 20 client.25417 trim_cache size 0 max 16384 2024-03-18T16:09:45.909 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-03-18T16:09:45.910 INFO:teuthology.orchestra.run.smithi039.stdout: "cluster_fsid": "1892d8fc-e540-11ee-95c9-87774f69a715", 2024-03-18T16:09:45.910 INFO:teuthology.orchestra.run.smithi039.stdout: "whoami": 2, 2024-03-18T16:09:45.910 INFO:teuthology.orchestra.run.smithi039.stdout: "id": 24535, 2024-03-18T16:09:45.910 INFO:teuthology.orchestra.run.smithi039.stdout: "want_state": "up:active", 2024-03-18T16:09:45.910 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "up:active", 2024-03-18T16:09:45.910 INFO:teuthology.orchestra.run.smithi039.stdout: "fs_name": "cephfs", 2024-03-18T16:09:45.910 INFO:teuthology.orchestra.run.smithi039.stdout: "rank_uptime": 147.687559652, 2024-03-18T16:09:45.910 INFO:teuthology.orchestra.run.smithi039.stdout: "mdsmap_epoch": 27, 2024-03-18T16:09:45.910 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_epoch": 93, 2024-03-18T16:09:45.910 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_epoch_barrier": 89, 2024-03-18T16:09:45.910 INFO:teuthology.orchestra.run.smithi039.stdout: "uptime": 205.47601496499999 2024-03-18T16:09:45.910 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-03-18T16:09:45.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.903+0000 7f514be00640 1 -- 172.21.15.39:0/3980898297 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f51300a2710 msgr2=0x7f51300a4bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:45.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.903+0000 7f514be00640 1 --2- 172.21.15.39:0/3980898297 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f51300a2710 0x7f51300a4bd0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f51440324c0 tx=0x7f5144009360 comp rx=0 tx=0).stop 2024-03-18T16:09:45.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.903+0000 7f514be00640 1 -- 172.21.15.39:0/3980898297 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f514c104a70 msgr2=0x7f514c1000f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:45.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.903+0000 7f514be00640 1 --2- 172.21.15.39:0/3980898297 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f514c104a70 0x7f514c1000f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f513400e9c0 tx=0x7f513400ee80 comp rx=0 tx=0).stop 2024-03-18T16:09:45.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.903+0000 7f514be00640 1 -- 172.21.15.39:0/3980898297 shutdown_connections 2024-03-18T16:09:45.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.903+0000 7f514be00640 1 --2- 172.21.15.39:0/3980898297 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f51300a2710 0x7f51300a4bd0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.903+0000 7f514be00640 1 --2- 172.21.15.39:0/3980898297 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f514c1073a0 0x7f514c100630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.903+0000 7f514be00640 1 --2- 172.21.15.39:0/3980898297 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f514c104a70 0x7f514c1000f0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.903+0000 7f514be00640 1 --2- 172.21.15.39:0/3980898297 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f514c102140 0x7f514c101ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:45.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.903+0000 7f514be00640 1 -- 172.21.15.39:0/3980898297 >> 172.21.15.39:0/3980898297 conn(0x7f514c0fbb50 msgr2=0x7f514c102a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:45.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.903+0000 7f514be00640 1 -- 172.21.15.39:0/3980898297 shutdown_connections 2024-03-18T16:09:45.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:45.903+0000 7f514be00640 1 -- 172.21.15.39:0/3980898297 wait complete. 2024-03-18T16:09:45.924 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-03-18T16:09:46.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.128+0000 7f498d200640 1 -- 172.21.15.39:0/3588639271 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f49880761a0 msgr2=0x7f49880765c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:46.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.128+0000 7f498d200640 1 --2- 172.21.15.39:0/3588639271 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f49880761a0 0x7f49880765c0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f4978009860 tx=0x7f497802bb70 comp rx=0 tx=0).stop 2024-03-18T16:09:46.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.128+0000 7f498d200640 1 -- 172.21.15.39:0/3588639271 shutdown_connections 2024-03-18T16:09:46.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.128+0000 7f498d200640 1 --2- 172.21.15.39:0/3588639271 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4988077d80 0x7f4988078620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:46.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.128+0000 7f498d200640 1 --2- 172.21.15.39:0/3588639271 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49880773e0 0x7f4988077840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:46.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.128+0000 7f498d200640 1 --2- 172.21.15.39:0/3588639271 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f49880761a0 0x7f49880765c0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:46.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.128+0000 7f498d200640 1 -- 172.21.15.39:0/3588639271 >> 172.21.15.39:0/3588639271 conn(0x7f49880fee50 msgr2=0x7f4988101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:46.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.129+0000 7f498d200640 1 -- 172.21.15.39:0/3588639271 shutdown_connections 2024-03-18T16:09:46.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.129+0000 7f498d200640 1 -- 172.21.15.39:0/3588639271 wait complete. 2024-03-18T16:09:46.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.130+0000 7f498d200640 1 Processor -- start 2024-03-18T16:09:46.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.130+0000 7f498d200640 1 -- start start 2024-03-18T16:09:46.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.130+0000 7f498d200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49880761a0 0x7f498819dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:46.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.130+0000 7f498d200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f49880773e0 0x7f498819e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:46.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.130+0000 7f498d200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4988077d80 0x7f49881a5550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:46.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.130+0000 7f498d200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f4988113970 con 0x7f49880761a0 2024-03-18T16:09:46.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.130+0000 7f498d200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f49881137f0 con 0x7f49880773e0 2024-03-18T16:09:46.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.130+0000 7f498d200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f4988113af0 con 0x7f4988077d80 2024-03-18T16:09:46.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.131+0000 7f4987400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49880761a0 0x7f498819dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:46.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.131+0000 7f4986a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f49880773e0 0x7f498819e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:46.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.131+0000 7f4987e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4988077d80 0x7f49881a5550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:46.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.131+0000 7f4986a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f49880773e0 0x7f498819e4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:38844/0 (socket says 172.21.15.39:38844) 2024-03-18T16:09:46.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.131+0000 7f4986a00640 1 -- 172.21.15.39:0/2778941813 learned_addr learned my addr 172.21.15.39:0/2778941813 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:46.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.131+0000 7f4986a00640 1 -- 172.21.15.39:0/2778941813 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4988077d80 msgr2=0x7f49881a5550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:46.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.131+0000 7f4986a00640 1 --2- 172.21.15.39:0/2778941813 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4988077d80 0x7f49881a5550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:46.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.131+0000 7f4986a00640 1 -- 172.21.15.39:0/2778941813 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49880761a0 msgr2=0x7f498819dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:46.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.131+0000 7f4986a00640 1 --2- 172.21.15.39:0/2778941813 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49880761a0 0x7f498819dfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:46.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.131+0000 7f4986a00640 1 -- 172.21.15.39:0/2778941813 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49881a5bc0 con 0x7f49880773e0 2024-03-18T16:09:46.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.131+0000 7f4987e00640 1 --2- 172.21.15.39:0/2778941813 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4988077d80 0x7f49881a5550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:09:46.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.132+0000 7f4986a00640 1 --2- 172.21.15.39:0/2778941813 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f49880773e0 0x7f498819e4e0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f497400e9c0 tx=0x7f497400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:46.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.132+0000 7f497fe00640 1 -- 172.21.15.39:0/2778941813 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f497400ccc0 con 0x7f49880773e0 2024-03-18T16:09:46.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.132+0000 7f497fe00640 1 -- 172.21.15.39:0/2778941813 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4974005500 con 0x7f49880773e0 2024-03-18T16:09:46.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.132+0000 7f497fe00640 1 -- 172.21.15.39:0/2778941813 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4974010430 con 0x7f49880773e0 2024-03-18T16:09:46.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.132+0000 7f498d200640 1 -- 172.21.15.39:0/2778941813 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49881a5eb0 con 0x7f49880773e0 2024-03-18T16:09:46.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.132+0000 7f498d200640 1 -- 172.21.15.39:0/2778941813 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49881a6370 con 0x7f49880773e0 2024-03-18T16:09:46.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.136+0000 7f4991478740 1 Processor -- start 2024-03-18T16:09:46.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.137+0000 7f4991478740 1 -- start start 2024-03-18T16:09:46.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.137+0000 7f4991478740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ed0cb24340 0x55ed0cb24760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:46.141 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.137+0000 7f4991478740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ed0cb24d30 0x55ed0cb29830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:46.141 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.137+0000 7f4991478740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ed0cb29ea0 0x55ed0cb2c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:46.141 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.137+0000 7f4991478740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55ed0ca88c50 con 0x55ed0cb29ea0 2024-03-18T16:09:46.141 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.137+0000 7f4991478740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55ed0ca319d0 con 0x55ed0cb24340 2024-03-18T16:09:46.141 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.137+0000 7f4991478740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55ed0ca84cf0 con 0x55ed0cb24d30 2024-03-18T16:09:46.141 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.137+0000 7f4987e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ed0cb24340 0x55ed0cb24760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:46.141 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.137+0000 7f4987e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ed0cb24340 0x55ed0cb24760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:38860/0 (socket says 172.21.15.39:38860) 2024-03-18T16:09:46.141 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.137+0000 7f4987400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ed0cb24d30 0x55ed0cb29830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:46.141 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.137+0000 7f4987e00640 1 -- 172.21.15.39:0/3898510871 learned_addr learned my addr 172.21.15.39:0/3898510871 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:46.141 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.138+0000 7f497fe00640 1 -- 172.21.15.39:0/2778941813 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f49740046c0 con 0x7f49880773e0 2024-03-18T16:09:46.141 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.138+0000 7f4987e00640 1 -- 172.21.15.39:0/3898510871 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ed0cb24d30 msgr2=0x55ed0cb29830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:46.141 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.138+0000 7f4987e00640 1 --2- 172.21.15.39:0/3898510871 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ed0cb24d30 0x55ed0cb29830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:46.141 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.138+0000 7f4987e00640 1 -- 172.21.15.39:0/3898510871 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ed0cb29ea0 msgr2=0x55ed0cb2c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:46.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.138+0000 7f4986a00640 1 --2- 172.21.15.39:0/3898510871 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ed0cb29ea0 0x55ed0cb2c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:46.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.138+0000 7f4987e00640 1 --2- 172.21.15.39:0/3898510871 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ed0cb29ea0 0x55ed0cb2c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:46.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.138+0000 7f4987e00640 1 -- 172.21.15.39:0/3898510871 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ed0ca4bf10 con 0x55ed0cb24340 2024-03-18T16:09:46.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.138+0000 7f4987400640 1 --2- 172.21.15.39:0/3898510871 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ed0cb24d30 0x55ed0cb29830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:09:46.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.138+0000 7f4986a00640 1 --2- 172.21.15.39:0/3898510871 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ed0cb29ea0 0x55ed0cb2c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:09:46.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.138+0000 7f4987e00640 1 --2- 172.21.15.39:0/3898510871 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ed0cb24340 0x55ed0cb24760 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f49800117e0 tx=0x7f4980011ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb0575debae8b106 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:46.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.139+0000 7f497cc00640 1 -- 172.21.15.39:0/3898510871 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4980011310 con 0x55ed0cb24340 2024-03-18T16:09:46.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.139+0000 7f497cc00640 1 -- 172.21.15.39:0/3898510871 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4980003e50 con 0x55ed0cb24340 2024-03-18T16:09:46.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.139+0000 7f497cc00640 1 -- 172.21.15.39:0/3898510871 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49800045b0 con 0x55ed0cb24340 2024-03-18T16:09:46.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.140+0000 7f4991478740 1 -- 172.21.15.39:0/3898510871 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ed0cb24340 msgr2=0x55ed0cb24760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:46.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.140+0000 7f4991478740 1 --2- 172.21.15.39:0/3898510871 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ed0cb24340 0x55ed0cb24760 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f49800117e0 tx=0x7f4980011ca0 comp rx=0 tx=0).stop 2024-03-18T16:09:46.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.140+0000 7f497fe00640 1 --2- 172.21.15.39:0/2778941813 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f496c0a2800 0x7f496c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:46.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.140+0000 7f4991478740 1 -- 172.21.15.39:0/3898510871 shutdown_connections 2024-03-18T16:09:46.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.140+0000 7f4991478740 1 --2- 172.21.15.39:0/3898510871 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ed0cb29ea0 0x55ed0cb2c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:46.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.140+0000 7f4991478740 1 --2- 172.21.15.39:0/3898510871 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ed0cb24d30 0x55ed0cb29830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:46.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.140+0000 7f4991478740 1 --2- 172.21.15.39:0/3898510871 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ed0cb24340 0x55ed0cb24760 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:46.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.140+0000 7f4991478740 1 -- 172.21.15.39:0/3898510871 >> 172.21.15.39:0/3898510871 conn(0x55ed0cb1f8b0 msgr2=0x55ed0cb1fcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:46.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.140+0000 7f497fe00640 1 -- 172.21.15.39:0/2778941813 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f49740cf0d0 con 0x7f49880773e0 2024-03-18T16:09:46.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.140+0000 7f4987400640 1 --2- 172.21.15.39:0/2778941813 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f496c0a2800 0x7f496c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:46.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.140+0000 7f4991478740 1 -- 172.21.15.39:0/3898510871 shutdown_connections 2024-03-18T16:09:46.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.140+0000 7f4991478740 1 -- 172.21.15.39:0/3898510871 wait complete. 2024-03-18T16:09:46.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.141+0000 7f4991478740 1 -- start start 2024-03-18T16:09:46.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.141+0000 7f4987400640 1 --2- 172.21.15.39:0/2778941813 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f496c0a2800 0x7f496c0a4cc0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f4978009830 tx=0x7f4978034ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:46.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.141+0000 7f4991478740 1 Processor -- start 2024-03-18T16:09:46.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.141+0000 7f4991478740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ed0cb24340 0x55ed0cb2e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:46.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.141+0000 7f4991478740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ed0cb24d30 0x55ed0cb2ea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:46.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.141+0000 7f4991478740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ed0cb29ea0 0x55ed0cb359e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:46.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.141+0000 7f4991478740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55ed0cb2ce50 con 0x55ed0cb29ea0 2024-03-18T16:09:46.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.141+0000 7f4991478740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55ed0ca44e20 con 0x55ed0cb24340 2024-03-18T16:09:46.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.141+0000 7f4991478740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55ed0cb2cfd0 con 0x55ed0cb24d30 2024-03-18T16:09:46.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.141+0000 7f4991478740 10 client.0 _ll_register_callbacks cb 0x55ed0ca447f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:09:46.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.142+0000 7f4987400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ed0cb24340 0x55ed0cb2e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:46.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.142+0000 7f4987e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ed0cb29ea0 0x55ed0cb359e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:46.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.142+0000 7f4987400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ed0cb24340 0x55ed0cb2e4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:38866/0 (socket says 172.21.15.39:38866) 2024-03-18T16:09:46.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.142+0000 7f4987400640 1 -- 172.21.15.39:0/968807352 learned_addr learned my addr 172.21.15.39:0/968807352 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:46.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.142+0000 7f4987400640 1 -- 172.21.15.39:0/968807352 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ed0cb24d30 msgr2=0x55ed0cb2ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:46.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.142+0000 7f4987400640 1 --2- 172.21.15.39:0/968807352 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ed0cb24d30 0x55ed0cb2ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:46.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.142+0000 7f4987400640 1 -- 172.21.15.39:0/968807352 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ed0cb29ea0 msgr2=0x55ed0cb359e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:46.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.142+0000 7f4987400640 1 --2- 172.21.15.39:0/968807352 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ed0cb29ea0 0x55ed0cb359e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:46.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.142+0000 7f4987400640 1 -- 172.21.15.39:0/968807352 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ed0c79d2d0 con 0x55ed0cb24340 2024-03-18T16:09:46.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.142+0000 7f4987400640 1 --2- 172.21.15.39:0/968807352 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ed0cb24340 0x55ed0cb2e4d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f4978000c00 tx=0x7f4978005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=98f4299027db5db5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:46.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.142+0000 7f4971600640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:09:46.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.143+0000 7f4971600640 1 -- 172.21.15.39:0/968807352 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4978046040 con 0x55ed0cb24340 2024-03-18T16:09:46.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.143+0000 7f4971600640 1 -- 172.21.15.39:0/968807352 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f497802c510 con 0x55ed0cb24340 2024-03-18T16:09:46.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.143+0000 7f4991478740 1 -- 172.21.15.39:0/968807352 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4980003e50 con 0x55ed0cb24340 2024-03-18T16:09:46.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.143+0000 7f4971600640 1 -- 172.21.15.39:0/968807352 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f4978058050 con 0x55ed0cb24340 2024-03-18T16:09:46.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.143+0000 7f4991478740 10 client.25406 fetch_fsmap learned FSMap version 27 2024-03-18T16:09:46.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.143+0000 7f4991478740 1 -- 172.21.15.39:0/968807352 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ed0c8676b0 con 0x55ed0cb24340 2024-03-18T16:09:46.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.143+0000 7f4971600640 1 -- 172.21.15.39:0/968807352 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4980003e50 con 0x55ed0cb24340 2024-03-18T16:09:46.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.143+0000 7f4971600640 1 -- 172.21.15.39:0/968807352 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f49780035f0 con 0x55ed0cb24340 2024-03-18T16:09:46.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.144+0000 7f4991478740 10 client.25406 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T16:09:46.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.144+0000 7f4991478740 10 client.25406 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-18T16:09:46.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.144+0000 7f4991478740 20 client.25406 populate_metadata read hostname 'smithi039' 2024-03-18T16:09:46.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.144+0000 7f4991478740 1 --2- 172.21.15.39:0/968807352 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x55ed0cb3b270 0x55ed0cb3c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:46.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.144+0000 7f4991478740 4 client.25406 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:09:46.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.144+0000 7f4991478740 1 -- 172.21.15.39:0/968807352 --> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ed0cae62b0 con 0x55ed0cb3b270 2024-03-18T16:09:46.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.144+0000 7f4986a00640 1 --2- 172.21.15.39:0/968807352 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x55ed0cb3b270 0x55ed0cb3c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:46.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.145+0000 7f4971600640 10 client.25406 ms_handle_connect on v2:172.21.15.196:6838/2001427268 2024-03-18T16:09:46.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.145+0000 7f4986a00640 1 --2- 172.21.15.39:0/968807352 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x55ed0cb3b270 0x55ed0cb3c6e0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4643a5f56d7be823 server_cookie=5eb200f975f55c5a in_seq=0 out_seq=0 2024-03-18T16:09:46.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.145+0000 7f4971600640 1 -- 172.21.15.39:0/968807352 <== mds.1 v2:172.21.15.196:6838/2001427268 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ed0cae62b0 con 0x55ed0cb3b270 2024-03-18T16:09:46.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.145+0000 7f4971600640 10 client.25406 handle_command_reply: tid=0 2024-03-18T16:09:46.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.145+0000 7f4991478740 1 client.25406 shutdown 2024-03-18T16:09:46.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.146+0000 7f4991478740 1 -- 172.21.15.39:0/968807352 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ed0cb24340 msgr2=0x55ed0cb2e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:46.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.146+0000 7f4991478740 1 --2- 172.21.15.39:0/968807352 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ed0cb24340 0x55ed0cb2e4d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f4978000c00 tx=0x7f4978005f90 comp rx=0 tx=0).stop 2024-03-18T16:09:46.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.146+0000 7f4991478740 1 -- 172.21.15.39:0/968807352 shutdown_connections 2024-03-18T16:09:46.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.146+0000 7f4991478740 1 --2- 172.21.15.39:0/968807352 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x55ed0cb3b270 0x55ed0cb3c6e0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:46.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.146+0000 7f4991478740 1 --2- 172.21.15.39:0/968807352 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ed0cb29ea0 0x55ed0cb359e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:46.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.146+0000 7f4991478740 1 --2- 172.21.15.39:0/968807352 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ed0cb24d30 0x55ed0cb2ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:46.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.146+0000 7f4971600640 0 client.25406 ms_handle_reset on v2:172.21.15.196:6838/2001427268 2024-03-18T16:09:46.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.146+0000 7f4991478740 1 --2- 172.21.15.39:0/968807352 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ed0cb24340 0x55ed0cb2e4d0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:46.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.146+0000 7f4991478740 1 -- 172.21.15.39:0/968807352 >> 172.21.15.39:0/968807352 conn(0x55ed0cb1f8b0 msgr2=0x55ed0cb1ea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:46.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.146+0000 7f4991478740 1 -- 172.21.15.39:0/968807352 shutdown_connections 2024-03-18T16:09:46.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.146+0000 7f4991478740 1 -- 172.21.15.39:0/968807352 wait complete. 2024-03-18T16:09:46.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.146+0000 7f4991478740 20 client.25406 trim_cache size 0 max 16384 2024-03-18T16:09:46.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.166+0000 7f4991478740 1 Processor -- start 2024-03-18T16:09:46.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.166+0000 7f4991478740 1 -- start start 2024-03-18T16:09:46.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.166+0000 7f4991478740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ed0cb24340 0x55ed0cb2e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:46.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.166+0000 7f4991478740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ed0cb24d30 0x55ed0cb35970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:46.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.166+0000 7f4991478740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ed0cb29ea0 0x55ed0cb2e620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:46.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.166+0000 7f4991478740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55ed0cb3aa20 con 0x55ed0cb24d30 2024-03-18T16:09:46.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.166+0000 7f4991478740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55ed0cb3b0f0 con 0x55ed0cb24340 2024-03-18T16:09:46.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.166+0000 7f4991478740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55ed0cb3b7c0 con 0x55ed0cb29ea0 2024-03-18T16:09:46.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.167+0000 7f4986a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ed0cb24d30 0x55ed0cb35970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:46.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.167+0000 7f4987e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ed0cb29ea0 0x55ed0cb2e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:46.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.167+0000 7f4987400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ed0cb24340 0x55ed0cb2e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:46.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.167+0000 7f4987e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ed0cb29ea0 0x55ed0cb2e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:56316/0 (socket says 172.21.15.39:56316) 2024-03-18T16:09:46.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.167+0000 7f4987400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ed0cb24340 0x55ed0cb2e0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:38874/0 (socket says 172.21.15.39:38874) 2024-03-18T16:09:46.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.167+0000 7f4986a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ed0cb24d30 0x55ed0cb35970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:37554/0 (socket says 172.21.15.39:37554) 2024-03-18T16:09:46.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.167+0000 7f4987e00640 1 -- 172.21.15.39:0/1711165979 learned_addr learned my addr 172.21.15.39:0/1711165979 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:46.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.167+0000 7f4987e00640 1 -- 172.21.15.39:0/1711165979 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ed0cb24340 msgr2=0x55ed0cb2e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:46.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.167+0000 7f4987e00640 1 --2- 172.21.15.39:0/1711165979 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ed0cb24340 0x55ed0cb2e0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:46.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.167+0000 7f4987e00640 1 -- 172.21.15.39:0/1711165979 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ed0cb24d30 msgr2=0x55ed0cb35970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:46.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.167+0000 7f4987e00640 1 --2- 172.21.15.39:0/1711165979 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ed0cb24d30 0x55ed0cb35970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:46.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.167+0000 7f4987e00640 1 -- 172.21.15.39:0/1711165979 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ed0cb2d5a0 con 0x55ed0cb29ea0 2024-03-18T16:09:46.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.167+0000 7f4987400640 1 --2- 172.21.15.39:0/1711165979 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ed0cb24340 0x55ed0cb2e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:09:46.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.167+0000 7f4986a00640 1 --2- 172.21.15.39:0/1711165979 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ed0cb24d30 0x55ed0cb35970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:09:46.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.167+0000 7f4987e00640 1 --2- 172.21.15.39:0/1711165979 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ed0cb29ea0 0x55ed0cb2e620 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x55ed0cb25590 tx=0x7f498000ff50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6224e5043fdcec12 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:46.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.168+0000 7f4973e00640 1 -- 172.21.15.39:0/1711165979 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f498002a070 con 0x55ed0cb29ea0 2024-03-18T16:09:46.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.168+0000 7f4973e00640 1 -- 172.21.15.39:0/1711165979 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4980011410 con 0x55ed0cb29ea0 2024-03-18T16:09:46.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.168+0000 7f4973e00640 1 -- 172.21.15.39:0/1711165979 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49800056c0 con 0x55ed0cb29ea0 2024-03-18T16:09:46.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.168+0000 7f4991478740 1 -- 172.21.15.39:0/1711165979 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ed0cb29ea0 msgr2=0x55ed0cb2e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:46.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.168+0000 7f4991478740 1 --2- 172.21.15.39:0/1711165979 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ed0cb29ea0 0x55ed0cb2e620 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x55ed0cb25590 tx=0x7f498000ff50 comp rx=0 tx=0).stop 2024-03-18T16:09:46.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.168+0000 7f4991478740 1 -- 172.21.15.39:0/1711165979 shutdown_connections 2024-03-18T16:09:46.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.168+0000 7f4991478740 1 --2- 172.21.15.39:0/1711165979 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ed0cb29ea0 0x55ed0cb2e620 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:46.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.168+0000 7f4991478740 1 --2- 172.21.15.39:0/1711165979 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ed0cb24d30 0x55ed0cb35970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:46.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.168+0000 7f4991478740 1 --2- 172.21.15.39:0/1711165979 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ed0cb24340 0x55ed0cb2e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:46.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.168+0000 7f4991478740 1 -- 172.21.15.39:0/1711165979 >> 172.21.15.39:0/1711165979 conn(0x55ed0cb1f8b0 msgr2=0x55ed0cb39fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:46.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.168+0000 7f4991478740 1 -- 172.21.15.39:0/1711165979 shutdown_connections 2024-03-18T16:09:46.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.168+0000 7f4991478740 1 -- 172.21.15.39:0/1711165979 wait complete. 2024-03-18T16:09:46.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.169+0000 7f4991478740 1 -- start start 2024-03-18T16:09:46.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.169+0000 7f4991478740 1 Processor -- start 2024-03-18T16:09:46.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.169+0000 7f4991478740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ed0cb24340 0x55ed0cb31610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:46.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.169+0000 7f4991478740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ed0cb24d30 0x55ed0cb31b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:46.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.169+0000 7f4991478740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ed0cb29ea0 0x55ed0cb34b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:46.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.169+0000 7f4991478740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55ed0cb3c560 con 0x55ed0cb29ea0 2024-03-18T16:09:46.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.169+0000 7f4991478740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55ed0cb3be90 con 0x55ed0cb24340 2024-03-18T16:09:46.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.169+0000 7f4991478740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55ed0cb3cc30 con 0x55ed0cb24d30 2024-03-18T16:09:46.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.170+0000 7f4991478740 10 client.0 _ll_register_callbacks cb 0x7f49780096b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:09:46.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.170+0000 7f4986a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ed0cb24d30 0x55ed0cb31b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:46.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.170+0000 7f4987400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ed0cb24340 0x55ed0cb31610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:46.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.170+0000 7f4986a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ed0cb24d30 0x55ed0cb31b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:56318/0 (socket says 172.21.15.39:56318) 2024-03-18T16:09:46.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.170+0000 7f4986a00640 1 -- 172.21.15.39:0/2819645929 learned_addr learned my addr 172.21.15.39:0/2819645929 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:46.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.170+0000 7f4987400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ed0cb24340 0x55ed0cb31610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:38880/0 (socket says 172.21.15.39:38880) 2024-03-18T16:09:46.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.170+0000 7f4987e00640 1 --2- 172.21.15.39:0/2819645929 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ed0cb29ea0 0x55ed0cb34b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:46.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.170+0000 7f4986a00640 1 -- 172.21.15.39:0/2819645929 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ed0cb24340 msgr2=0x55ed0cb31610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:46.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.170+0000 7f4986a00640 1 --2- 172.21.15.39:0/2819645929 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ed0cb24340 0x55ed0cb31610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:46.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.170+0000 7f4986a00640 1 -- 172.21.15.39:0/2819645929 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ed0cb29ea0 msgr2=0x55ed0cb34b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:46.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.170+0000 7f4986a00640 1 --2- 172.21.15.39:0/2819645929 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ed0cb29ea0 0x55ed0cb34b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:46.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.170+0000 7f4986a00640 1 -- 172.21.15.39:0/2819645929 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ed0cb35110 con 0x55ed0cb24d30 2024-03-18T16:09:46.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.170+0000 7f4987e00640 1 --2- 172.21.15.39:0/2819645929 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ed0cb29ea0 0x55ed0cb34b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:09:46.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.170+0000 7f4986a00640 1 --2- 172.21.15.39:0/2819645929 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ed0cb24d30 0x55ed0cb31b50 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f497400c6a0 tx=0x7f4974009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2ef6c71edb3b1f6f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:46.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.170+0000 7f4971600640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:09:46.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.171+0000 7f4971600640 1 -- 172.21.15.39:0/2819645929 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49740b3070 con 0x55ed0cb24d30 2024-03-18T16:09:46.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.171+0000 7f4971600640 1 -- 172.21.15.39:0/2819645929 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f4974099780 con 0x55ed0cb24d30 2024-03-18T16:09:46.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.171+0000 7f4991478740 1 -- 172.21.15.39:0/2819645929 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4980011410 con 0x55ed0cb24d30 2024-03-18T16:09:46.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.171+0000 7f4971600640 1 -- 172.21.15.39:0/2819645929 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f497409b6e0 con 0x55ed0cb24d30 2024-03-18T16:09:46.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.171+0000 7f4991478740 10 client.25432 fetch_fsmap learned FSMap version 27 2024-03-18T16:09:46.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.171+0000 7f4991478740 1 -- 172.21.15.39:0/2819645929 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ed0cba15c0 con 0x55ed0cb24d30 2024-03-18T16:09:46.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.171+0000 7f4971600640 1 -- 172.21.15.39:0/2819645929 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4980011410 con 0x55ed0cb24d30 2024-03-18T16:09:46.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.171+0000 7f4971600640 1 -- 172.21.15.39:0/2819645929 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f49740a3a80 con 0x55ed0cb24d30 2024-03-18T16:09:46.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.171+0000 7f4991478740 10 client.25432 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T16:09:46.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.171+0000 7f4991478740 10 client.25432 resolve_mds: resolved name 'i' to daemon mds.i 2024-03-18T16:09:46.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.171+0000 7f4991478740 20 client.25432 populate_metadata read hostname 'smithi039' 2024-03-18T16:09:46.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.171+0000 7f4991478740 1 --2- 172.21.15.39:0/2819645929 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x55ed0cba6170 0x55ed0cba4e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:46.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.171+0000 7f4991478740 4 client.25432 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T16:09:46.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.171+0000 7f4991478740 1 -- 172.21.15.39:0/2819645929 --> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] -- command(tid 0: {"prefix": "status"}) -- 0x55ed0ca96190 con 0x55ed0cba6170 2024-03-18T16:09:46.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.172+0000 7f4987400640 1 --2- 172.21.15.39:0/2819645929 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x55ed0cba6170 0x55ed0cba4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:46.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.172+0000 7f4987400640 1 --2- 172.21.15.39:0/2819645929 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x55ed0cba6170 0x55ed0cba4e80 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f2ebb1a5540cd5a7 server_cookie=68e1b246cf794fb4 in_seq=0 out_seq=0 2024-03-18T16:09:46.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.172+0000 7f4971600640 10 client.25432 ms_handle_connect on v2:172.21.15.196:6838/2001427268 2024-03-18T16:09:46.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.172+0000 7f4971600640 1 -- 172.21.15.39:0/2819645929 <== mds.1 v2:172.21.15.196:6838/2001427268 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55ed0ca96190 con 0x55ed0cba6170 2024-03-18T16:09:46.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.172+0000 7f4971600640 10 client.25432 handle_command_reply: tid=0 2024-03-18T16:09:46.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.172+0000 7f4991478740 1 client.25432 shutdown 2024-03-18T16:09:46.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.173+0000 7f4991478740 1 -- 172.21.15.39:0/2819645929 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ed0cb24d30 msgr2=0x55ed0cb31b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:46.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.173+0000 7f4991478740 1 --2- 172.21.15.39:0/2819645929 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ed0cb24d30 0x55ed0cb31b50 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f497400c6a0 tx=0x7f4974009f90 comp rx=0 tx=0).stop 2024-03-18T16:09:46.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.173+0000 7f4991478740 1 -- 172.21.15.39:0/2819645929 shutdown_connections 2024-03-18T16:09:46.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.173+0000 7f4991478740 1 --2- 172.21.15.39:0/2819645929 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x55ed0cba6170 0x55ed0cba4e80 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:46.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.173+0000 7f4991478740 1 --2- 172.21.15.39:0/2819645929 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ed0cb29ea0 0x55ed0cb34b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:46.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.173+0000 7f4991478740 1 --2- 172.21.15.39:0/2819645929 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ed0cb24d30 0x55ed0cb31b50 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:46.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.173+0000 7f4991478740 1 --2- 172.21.15.39:0/2819645929 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ed0cb24340 0x55ed0cb31610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:46.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.173+0000 7f4971600640 0 client.25432 ms_handle_reset on v2:172.21.15.196:6838/2001427268 2024-03-18T16:09:46.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.173+0000 7f4991478740 1 -- 172.21.15.39:0/2819645929 >> 172.21.15.39:0/2819645929 conn(0x55ed0cb1f8b0 msgr2=0x55ed0cb2e0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:46.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.173+0000 7f4991478740 1 -- 172.21.15.39:0/2819645929 shutdown_connections 2024-03-18T16:09:46.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.173+0000 7f4991478740 1 -- 172.21.15.39:0/2819645929 wait complete. 2024-03-18T16:09:46.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.173+0000 7f4991478740 20 client.25432 trim_cache size 0 max 16384 2024-03-18T16:09:46.181 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-03-18T16:09:46.181 INFO:teuthology.orchestra.run.smithi039.stdout: "cluster_fsid": "1892d8fc-e540-11ee-95c9-87774f69a715", 2024-03-18T16:09:46.181 INFO:teuthology.orchestra.run.smithi039.stdout: "whoami": 1, 2024-03-18T16:09:46.181 INFO:teuthology.orchestra.run.smithi039.stdout: "id": 24556, 2024-03-18T16:09:46.181 INFO:teuthology.orchestra.run.smithi039.stdout: "want_state": "up:active", 2024-03-18T16:09:46.181 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "up:active", 2024-03-18T16:09:46.181 INFO:teuthology.orchestra.run.smithi039.stdout: "fs_name": "cephfs", 2024-03-18T16:09:46.181 INFO:teuthology.orchestra.run.smithi039.stdout: "rank_uptime": 148.978921953, 2024-03-18T16:09:46.181 INFO:teuthology.orchestra.run.smithi039.stdout: "mdsmap_epoch": 27, 2024-03-18T16:09:46.181 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_epoch": 93, 2024-03-18T16:09:46.181 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_epoch_barrier": 89, 2024-03-18T16:09:46.181 INFO:teuthology.orchestra.run.smithi039.stdout: "uptime": 199.97521562399999 2024-03-18T16:09:46.181 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-03-18T16:09:46.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.174+0000 7f498d200640 1 -- 172.21.15.39:0/2778941813 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f496c0a2800 msgr2=0x7f496c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:46.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.174+0000 7f498d200640 1 --2- 172.21.15.39:0/2778941813 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f496c0a2800 0x7f496c0a4cc0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f4978009830 tx=0x7f4978034ab0 comp rx=0 tx=0).stop 2024-03-18T16:09:46.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.174+0000 7f498d200640 1 -- 172.21.15.39:0/2778941813 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f49880773e0 msgr2=0x7f498819e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:46.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.174+0000 7f498d200640 1 --2- 172.21.15.39:0/2778941813 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f49880773e0 0x7f498819e4e0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f497400e9c0 tx=0x7f497400ee80 comp rx=0 tx=0).stop 2024-03-18T16:09:46.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.174+0000 7f498d200640 1 -- 172.21.15.39:0/2778941813 shutdown_connections 2024-03-18T16:09:46.183 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.174+0000 7f498d200640 1 --2- 172.21.15.39:0/2778941813 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f49880773e0 0x7f498819e4e0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:46.183 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.174+0000 7f498d200640 1 --2- 172.21.15.39:0/2778941813 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f496c0a2800 0x7f496c0a4cc0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:46.183 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.174+0000 7f498d200640 1 --2- 172.21.15.39:0/2778941813 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4988077d80 0x7f49881a5550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:46.183 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.174+0000 7f498d200640 1 --2- 172.21.15.39:0/2778941813 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49880761a0 0x7f498819dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:46.183 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.174+0000 7f498d200640 1 -- 172.21.15.39:0/2778941813 >> 172.21.15.39:0/2778941813 conn(0x7f49880fee50 msgr2=0x7f49881021b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:46.183 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.174+0000 7f498d200640 1 -- 172.21.15.39:0/2778941813 shutdown_connections 2024-03-18T16:09:46.183 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.174+0000 7f498d200640 1 -- 172.21.15.39:0/2778941813 wait complete. 2024-03-18T16:09:46.200 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-18T16:09:46.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.412+0000 7fadcdc00640 1 -- 172.21.15.39:0/4279595337 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fadc8102140 msgr2=0x7fadc8104530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:46.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.412+0000 7fadcdc00640 1 --2- 172.21.15.39:0/4279595337 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fadc8102140 0x7fadc8104530 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fadb8009860 tx=0x7fadb802bb70 comp rx=0 tx=0).stop 2024-03-18T16:09:46.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.412+0000 7fadcdc00640 1 -- 172.21.15.39:0/4279595337 shutdown_connections 2024-03-18T16:09:46.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.412+0000 7fadcdc00640 1 --2- 172.21.15.39:0/4279595337 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fadc8104a70 0x7fadc8106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:46.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.412+0000 7fadcdc00640 1 --2- 172.21.15.39:0/4279595337 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fadc81073a0 0x7fadc81097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:46.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.412+0000 7fadcdc00640 1 --2- 172.21.15.39:0/4279595337 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fadc8102140 0x7fadc8104530 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:46.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.412+0000 7fadcdc00640 1 -- 172.21.15.39:0/4279595337 >> 172.21.15.39:0/4279595337 conn(0x7fadc80fbb50 msgr2=0x7fadc80fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:46.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.412+0000 7fadcdc00640 1 -- 172.21.15.39:0/4279595337 shutdown_connections 2024-03-18T16:09:46.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.413+0000 7fadcdc00640 1 -- 172.21.15.39:0/4279595337 wait complete. 2024-03-18T16:09:46.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.413+0000 7fadcdc00640 1 Processor -- start 2024-03-18T16:09:46.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.414+0000 7fadcdc00640 1 -- start start 2024-03-18T16:09:46.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.414+0000 7fadcdc00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fadc8102140 0x7fadc81979d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:46.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.414+0000 7fadcdc00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fadc8104a70 0x7fadc8197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:46.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.414+0000 7fadcdc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fadc81073a0 0x7fadc819ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:46.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.414+0000 7fadcdc00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fadc810d350 con 0x7fadc81073a0 2024-03-18T16:09:46.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.414+0000 7fadcdc00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fadc810d1d0 con 0x7fadc8102140 2024-03-18T16:09:46.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.414+0000 7fadcdc00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fadc810d4d0 con 0x7fadc8104a70 2024-03-18T16:09:46.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.414+0000 7fadbfe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fadc8104a70 0x7fadc8197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:46.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.414+0000 7fadc7e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fadc8102140 0x7fadc81979d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:46.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.414+0000 7fadcd200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fadc81073a0 0x7fadc819ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:46.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.414+0000 7fadbfe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fadc8104a70 0x7fadc8197f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:56332/0 (socket says 172.21.15.39:56332) 2024-03-18T16:09:46.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.414+0000 7fadbfe00640 1 -- 172.21.15.39:0/1164547089 learned_addr learned my addr 172.21.15.39:0/1164547089 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:46.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.414+0000 7fadc7e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fadc8102140 0x7fadc81979d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:38902/0 (socket says 172.21.15.39:38902) 2024-03-18T16:09:46.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.415+0000 7fadbfe00640 1 -- 172.21.15.39:0/1164547089 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fadc8102140 msgr2=0x7fadc81979d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:46.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.415+0000 7fadbfe00640 1 --2- 172.21.15.39:0/1164547089 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fadc8102140 0x7fadc81979d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:46.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.415+0000 7fadbfe00640 1 -- 172.21.15.39:0/1164547089 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fadc81073a0 msgr2=0x7fadc819ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:46.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.415+0000 7fadbfe00640 1 --2- 172.21.15.39:0/1164547089 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fadc81073a0 0x7fadc819ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:46.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.415+0000 7fadbfe00640 1 -- 172.21.15.39:0/1164547089 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadc819f5f0 con 0x7fadc8104a70 2024-03-18T16:09:46.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.415+0000 7fadc7e00640 1 --2- 172.21.15.39:0/1164547089 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fadc8102140 0x7fadc81979d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:09:46.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.415+0000 7fadbfe00640 1 --2- 172.21.15.39:0/1164547089 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fadc8104a70 0x7fadc8197f10 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fadb000e9a0 tx=0x7fadb000ee60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:46.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.415+0000 7fadc5600640 1 -- 172.21.15.39:0/1164547089 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadb000cd80 con 0x7fadc8104a70 2024-03-18T16:09:46.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.415+0000 7fadc5600640 1 -- 172.21.15.39:0/1164547089 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fadb00035f0 con 0x7fadc8104a70 2024-03-18T16:09:46.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.415+0000 7fadcdc00640 1 -- 172.21.15.39:0/1164547089 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fadc80fff90 con 0x7fadc8104a70 2024-03-18T16:09:46.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.415+0000 7fadc5600640 1 -- 172.21.15.39:0/1164547089 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadb0010670 con 0x7fadc8104a70 2024-03-18T16:09:46.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.415+0000 7fadcdc00640 1 -- 172.21.15.39:0/1164547089 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fadc81004e0 con 0x7fadc8104a70 2024-03-18T16:09:46.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.420+0000 7fadd1e5b740 1 Processor -- start 2024-03-18T16:09:46.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.420+0000 7fadc5600640 1 -- 172.21.15.39:0/1164547089 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fadb00037a0 con 0x7fadc8104a70 2024-03-18T16:09:46.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.421+0000 7fadd1e5b740 1 -- start start 2024-03-18T16:09:46.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.421+0000 7fadd1e5b740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560810716340 0x560810716760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:46.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.421+0000 7fadd1e5b740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560810716d30 0x56081071b830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:46.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.421+0000 7fadd1e5b740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56081071bea0 0x56081071e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:46.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.421+0000 7fadd1e5b740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x56081067ac50 con 0x56081071bea0 2024-03-18T16:09:46.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.421+0000 7fadd1e5b740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5608106239d0 con 0x560810716d30 2024-03-18T16:09:46.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.421+0000 7fadd1e5b740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x560810676cf0 con 0x560810716340 2024-03-18T16:09:46.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.421+0000 7fadc7e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560810716d30 0x56081071b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:46.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.421+0000 7fadcd200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560810716340 0x560810716760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:46.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.421+0000 7fadc7e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560810716d30 0x56081071b830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:38906/0 (socket says 172.21.15.39:38906) 2024-03-18T16:09:46.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.421+0000 7fadcd200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560810716340 0x560810716760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:56346/0 (socket says 172.21.15.39:56346) 2024-03-18T16:09:46.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.421+0000 7fadc7e00640 1 -- 172.21.15.39:0/1017352991 learned_addr learned my addr 172.21.15.39:0/1017352991 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:46.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.421+0000 7fadbfe00640 1 --2- 172.21.15.39:0/1017352991 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56081071bea0 0x56081071e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:46.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.422+0000 7fadc7e00640 1 -- 172.21.15.39:0/1017352991 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560810716340 msgr2=0x560810716760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:46.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.422+0000 7fadc7e00640 1 --2- 172.21.15.39:0/1017352991 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560810716340 0x560810716760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:46.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.422+0000 7fadc7e00640 1 -- 172.21.15.39:0/1017352991 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56081071bea0 msgr2=0x56081071e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:46.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.422+0000 7fadc7e00640 1 --2- 172.21.15.39:0/1017352991 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56081071bea0 0x56081071e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:46.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.422+0000 7fadc7e00640 1 -- 172.21.15.39:0/1017352991 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56081063df10 con 0x560810716d30 2024-03-18T16:09:46.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.422+0000 7fadc7e00640 1 --2- 172.21.15.39:0/1017352991 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560810716d30 0x56081071b830 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fadb8037040 tx=0x7fadb8034ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=555e0dc04e3ad94f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:46.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.422+0000 7fadbd600640 1 -- 172.21.15.39:0/1017352991 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadb804a070 con 0x560810716d30 2024-03-18T16:09:46.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.423+0000 7fadbd600640 1 -- 172.21.15.39:0/1017352991 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fadb8004040 con 0x560810716d30 2024-03-18T16:09:46.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.423+0000 7fadbd600640 1 -- 172.21.15.39:0/1017352991 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadb802ca10 con 0x560810716d30 2024-03-18T16:09:46.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.423+0000 7fadc5600640 1 --2- 172.21.15.39:0/1164547089 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fadac0a2800 0x7fadac0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:46.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.423+0000 7fadc5600640 1 -- 172.21.15.39:0/1164547089 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fadb00cf0b0 con 0x7fadc8104a70 2024-03-18T16:09:46.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.423+0000 7fadd1e5b740 1 -- 172.21.15.39:0/1017352991 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560810716d30 msgr2=0x56081071b830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:46.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.423+0000 7fadd1e5b740 1 --2- 172.21.15.39:0/1017352991 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560810716d30 0x56081071b830 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fadb8037040 tx=0x7fadb8034ab0 comp rx=0 tx=0).stop 2024-03-18T16:09:46.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.423+0000 7fadcd200640 1 --2- 172.21.15.39:0/1164547089 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fadac0a2800 0x7fadac0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:46.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.424+0000 7fadd1e5b740 1 -- 172.21.15.39:0/1017352991 shutdown_connections 2024-03-18T16:09:46.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.424+0000 7fadd1e5b740 1 --2- 172.21.15.39:0/1017352991 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560810716d30 0x56081071b830 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:46.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.424+0000 7fadd1e5b740 1 --2- 172.21.15.39:0/1017352991 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56081071bea0 0x56081071e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:46.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.424+0000 7fadd1e5b740 1 --2- 172.21.15.39:0/1017352991 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560810716340 0x560810716760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:46.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.424+0000 7fadd1e5b740 1 -- 172.21.15.39:0/1017352991 >> 172.21.15.39:0/1017352991 conn(0x5608107118b0 msgr2=0x560810711cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:46.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.424+0000 7fadd1e5b740 1 -- 172.21.15.39:0/1017352991 shutdown_connections 2024-03-18T16:09:46.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.424+0000 7fadd1e5b740 1 -- 172.21.15.39:0/1017352991 wait complete. 2024-03-18T16:09:46.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.424+0000 7fadd1e5b740 1 -- start start 2024-03-18T16:09:46.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.424+0000 7fadcd200640 1 --2- 172.21.15.39:0/1164547089 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fadac0a2800 0x7fadac0a4cc0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x560810717590 tx=0x7fadc0009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:46.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.425+0000 7fadd1e5b740 1 Processor -- start 2024-03-18T16:09:46.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.425+0000 7fadd1e5b740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560810716340 0x560810720470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:46.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.425+0000 7fadd1e5b740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560810716d30 0x5608107209b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:46.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.425+0000 7fadd1e5b740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56081071bea0 0x5608107279f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:46.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.425+0000 7fadd1e5b740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x56081071edf0 con 0x56081071bea0 2024-03-18T16:09:46.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.425+0000 7fadd1e5b740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x560810636e20 con 0x560810716d30 2024-03-18T16:09:46.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.425+0000 7fadd1e5b740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x56081071ef70 con 0x560810716340 2024-03-18T16:09:46.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.425+0000 7fadd1e5b740 10 client.0 _ll_register_callbacks cb 0x5608106367f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:09:46.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.425+0000 7fadc7e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560810716340 0x560810720470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:46.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.425+0000 7fadc7e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560810716340 0x560810720470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:56352/0 (socket says 172.21.15.39:56352) 2024-03-18T16:09:46.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.426+0000 7fadc7e00640 1 -- 172.21.15.39:0/1659165471 learned_addr learned my addr 172.21.15.39:0/1659165471 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:46.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.426+0000 7fadbfe00640 1 --2- 172.21.15.39:0/1659165471 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560810716d30 0x5608107209b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:46.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.426+0000 7fadcd200640 1 --2- 172.21.15.39:0/1659165471 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56081071bea0 0x5608107279f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:46.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.426+0000 7fadc7e00640 1 -- 172.21.15.39:0/1659165471 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560810716d30 msgr2=0x5608107209b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:46.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.426+0000 7fadc7e00640 1 --2- 172.21.15.39:0/1659165471 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560810716d30 0x5608107209b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:46.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.426+0000 7fadc7e00640 1 -- 172.21.15.39:0/1659165471 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56081071bea0 msgr2=0x5608107279f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:46.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.426+0000 7fadc7e00640 1 --2- 172.21.15.39:0/1659165471 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56081071bea0 0x5608107279f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:46.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.426+0000 7fadc7e00640 1 -- 172.21.15.39:0/1659165471 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56081038f2d0 con 0x560810716340 2024-03-18T16:09:46.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.426+0000 7fadc7e00640 1 --2- 172.21.15.39:0/1659165471 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560810716340 0x560810720470 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fadb802c0b0 tx=0x7fadb8034670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7472864543d22048 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:46.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.426+0000 7fadb6000640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:09:46.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.426+0000 7fadb6000640 1 -- 172.21.15.39:0/1659165471 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadb804a030 con 0x560810716340 2024-03-18T16:09:46.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.426+0000 7fadb6000640 1 -- 172.21.15.39:0/1659165471 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fadb8042040 con 0x560810716340 2024-03-18T16:09:46.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.427+0000 7fadd1e5b740 1 -- 172.21.15.39:0/1659165471 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fadb8004040 con 0x560810716340 2024-03-18T16:09:46.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.427+0000 7fadb6000640 1 -- 172.21.15.39:0/1659165471 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fadb8032400 con 0x560810716340 2024-03-18T16:09:46.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.427+0000 7fadd1e5b740 10 client.25444 fetch_fsmap learned FSMap version 27 2024-03-18T16:09:46.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.427+0000 7fadd1e5b740 1 -- 172.21.15.39:0/1659165471 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5608104596b0 con 0x560810716340 2024-03-18T16:09:46.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.427+0000 7fadb6000640 1 -- 172.21.15.39:0/1659165471 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fadb8004040 con 0x560810716340 2024-03-18T16:09:46.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.427+0000 7fadb6000640 1 -- 172.21.15.39:0/1659165471 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fadb80035f0 con 0x560810716340 2024-03-18T16:09:46.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.427+0000 7fadd1e5b740 10 client.25444 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T16:09:46.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.427+0000 7fadd1e5b740 10 client.25444 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-18T16:09:46.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.427+0000 7fadd1e5b740 20 client.25444 populate_metadata read hostname 'smithi039' 2024-03-18T16:09:46.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.427+0000 7fadd1e5b740 1 --2- 172.21.15.39:0/1659165471 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x56081072bee0 0x56081072e080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:46.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.428+0000 7fadd1e5b740 4 client.25444 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:09:46.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.428+0000 7fadd1e5b740 1 -- 172.21.15.39:0/1659165471 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5608106d82b0 con 0x56081072bee0 2024-03-18T16:09:46.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.428+0000 7fadbfe00640 1 --2- 172.21.15.39:0/1659165471 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x56081072bee0 0x56081072e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:46.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.428+0000 7fadbfe00640 1 --2- 172.21.15.39:0/1659165471 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x56081072bee0 0x56081072e080 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dda910b1166d1bab server_cookie=b04efacde12eab96 in_seq=0 out_seq=0 2024-03-18T16:09:46.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.428+0000 7fadb6000640 10 client.25444 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:09:46.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.429+0000 7fadb6000640 1 -- 172.21.15.39:0/1659165471 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5608106d82b0 con 0x56081072bee0 2024-03-18T16:09:46.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.429+0000 7fadb6000640 10 client.25444 handle_command_reply: tid=0 2024-03-18T16:09:46.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.429+0000 7fadd1e5b740 1 client.25444 shutdown 2024-03-18T16:09:46.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.429+0000 7fadd1e5b740 1 -- 172.21.15.39:0/1659165471 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560810716340 msgr2=0x560810720470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:46.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.429+0000 7fadd1e5b740 1 --2- 172.21.15.39:0/1659165471 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560810716340 0x560810720470 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fadb802c0b0 tx=0x7fadb8034670 comp rx=0 tx=0).stop 2024-03-18T16:09:46.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.429+0000 7fadd1e5b740 1 -- 172.21.15.39:0/1659165471 shutdown_connections 2024-03-18T16:09:46.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.429+0000 7fadd1e5b740 1 --2- 172.21.15.39:0/1659165471 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x56081072bee0 0x56081072e080 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:46.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.429+0000 7fadd1e5b740 1 --2- 172.21.15.39:0/1659165471 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560810716d30 0x5608107209b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:46.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.429+0000 7fadd1e5b740 1 --2- 172.21.15.39:0/1659165471 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56081071bea0 0x5608107279f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:46.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.429+0000 7fadb6000640 0 client.25444 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:09:46.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.429+0000 7fadd1e5b740 1 --2- 172.21.15.39:0/1659165471 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560810716340 0x560810720470 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:46.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.429+0000 7fadd1e5b740 1 -- 172.21.15.39:0/1659165471 >> 172.21.15.39:0/1659165471 conn(0x5608107118b0 msgr2=0x560810710a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:46.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.430+0000 7fadd1e5b740 1 -- 172.21.15.39:0/1659165471 shutdown_connections 2024-03-18T16:09:46.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.430+0000 7fadd1e5b740 1 -- 172.21.15.39:0/1659165471 wait complete. 2024-03-18T16:09:46.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.430+0000 7fadd1e5b740 20 client.25444 trim_cache size 0 max 16384 2024-03-18T16:09:46.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.450+0000 7fadd1e5b740 1 Processor -- start 2024-03-18T16:09:46.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.450+0000 7fadd1e5b740 1 -- start start 2024-03-18T16:09:46.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.450+0000 7fadd1e5b740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560810716340 0x56081071b500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:46.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.450+0000 7fadd1e5b740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560810716d30 0x560810727910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:46.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.450+0000 7fadd1e5b740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56081071bea0 0x560810711c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:46.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.450+0000 7fadd1e5b740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x56081072c3c0 con 0x560810716340 2024-03-18T16:09:46.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.450+0000 7fadd1e5b740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x56081072ca90 con 0x56081071bea0 2024-03-18T16:09:46.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.450+0000 7fadd1e5b740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x56081072d160 con 0x560810716d30 2024-03-18T16:09:46.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.450+0000 7fadcd200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56081071bea0 0x560810711c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:46.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.450+0000 7fadbfe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560810716d30 0x560810727910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:46.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.450+0000 7fadcd200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56081071bea0 0x560810711c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:38936/0 (socket says 172.21.15.39:38936) 2024-03-18T16:09:46.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.450+0000 7fadbfe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560810716d30 0x560810727910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:56364/0 (socket says 172.21.15.39:56364) 2024-03-18T16:09:46.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.450+0000 7fadc7e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560810716340 0x56081071b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:46.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.450+0000 7fadcd200640 1 -- 172.21.15.39:0/2196903891 learned_addr learned my addr 172.21.15.39:0/2196903891 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:46.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.451+0000 7fadcd200640 1 -- 172.21.15.39:0/2196903891 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560810716d30 msgr2=0x560810727910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:46.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.451+0000 7fadcd200640 1 --2- 172.21.15.39:0/2196903891 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560810716d30 0x560810727910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:46.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.451+0000 7fadcd200640 1 -- 172.21.15.39:0/2196903891 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560810716340 msgr2=0x56081071b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:46.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.451+0000 7fadcd200640 1 --2- 172.21.15.39:0/2196903891 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560810716340 0x56081071b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:46.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.451+0000 7fadcd200640 1 -- 172.21.15.39:0/2196903891 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560810717110 con 0x56081071bea0 2024-03-18T16:09:46.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.451+0000 7fadbfe00640 1 --2- 172.21.15.39:0/2196903891 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560810716d30 0x560810727910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:09:46.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.451+0000 7fadcd200640 1 --2- 172.21.15.39:0/2196903891 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56081071bea0 0x560810711c90 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fadc0005f40 tx=0x7fadc0016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f9cce5480e91f12e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:46.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.451+0000 7fadbcc00640 1 -- 172.21.15.39:0/2196903891 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadc0004540 con 0x56081071bea0 2024-03-18T16:09:46.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.451+0000 7fadbcc00640 1 -- 172.21.15.39:0/2196903891 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fadc00046e0 con 0x56081071bea0 2024-03-18T16:09:46.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.451+0000 7fadbcc00640 1 -- 172.21.15.39:0/2196903891 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadc00057e0 con 0x56081071bea0 2024-03-18T16:09:46.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.452+0000 7fadd1e5b740 1 -- 172.21.15.39:0/2196903891 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56081071bea0 msgr2=0x560810711c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:46.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.452+0000 7fadd1e5b740 1 --2- 172.21.15.39:0/2196903891 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56081071bea0 0x560810711c90 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fadc0005f40 tx=0x7fadc0016000 comp rx=0 tx=0).stop 2024-03-18T16:09:46.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.452+0000 7fadd1e5b740 1 -- 172.21.15.39:0/2196903891 shutdown_connections 2024-03-18T16:09:46.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.452+0000 7fadd1e5b740 1 --2- 172.21.15.39:0/2196903891 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56081071bea0 0x560810711c90 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:46.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.452+0000 7fadd1e5b740 1 --2- 172.21.15.39:0/2196903891 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560810716d30 0x560810727910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:46.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.452+0000 7fadd1e5b740 1 --2- 172.21.15.39:0/2196903891 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560810716340 0x56081071b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:46.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.452+0000 7fadd1e5b740 1 -- 172.21.15.39:0/2196903891 >> 172.21.15.39:0/2196903891 conn(0x5608107118b0 msgr2=0x560810710a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:46.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.452+0000 7fadd1e5b740 1 -- 172.21.15.39:0/2196903891 shutdown_connections 2024-03-18T16:09:46.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.452+0000 7fadd1e5b740 1 -- 172.21.15.39:0/2196903891 wait complete. 2024-03-18T16:09:46.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.453+0000 7fadd1e5b740 1 -- start start 2024-03-18T16:09:46.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.453+0000 7fadd1e5b740 1 Processor -- start 2024-03-18T16:09:46.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.453+0000 7fadd1e5b740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560810716340 0x560810723600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:46.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.453+0000 7fadd1e5b740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560810716d30 0x560810723b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:46.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.453+0000 7fadd1e5b740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56081071bea0 0x560810726b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:46.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.453+0000 7fadd1e5b740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x56081072df00 con 0x56081071bea0 2024-03-18T16:09:46.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.453+0000 7fadd1e5b740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x56081072d830 con 0x560810716d30 2024-03-18T16:09:46.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.453+0000 7fadd1e5b740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x56081072e5d0 con 0x560810716340 2024-03-18T16:09:46.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.453+0000 7fadd1e5b740 10 client.0 _ll_register_callbacks cb 0x7fadb0005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:09:46.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.454+0000 7fadbfe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560810716d30 0x560810723b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:46.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.454+0000 7fadbfe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560810716d30 0x560810723b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:38948/0 (socket says 172.21.15.39:38948) 2024-03-18T16:09:46.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.454+0000 7fadbfe00640 1 -- 172.21.15.39:0/3987759506 learned_addr learned my addr 172.21.15.39:0/3987759506 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:46.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.454+0000 7fadcd200640 1 --2- 172.21.15.39:0/3987759506 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56081071bea0 0x560810726b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:46.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.454+0000 7fadc7e00640 1 --2- 172.21.15.39:0/3987759506 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560810716340 0x560810723600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:46.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.454+0000 7fadbfe00640 1 -- 172.21.15.39:0/3987759506 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560810716340 msgr2=0x560810723600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:46.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.454+0000 7fadbfe00640 1 --2- 172.21.15.39:0/3987759506 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560810716340 0x560810723600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:46.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.454+0000 7fadbfe00640 1 -- 172.21.15.39:0/3987759506 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56081071bea0 msgr2=0x560810726b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:09:46.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.454+0000 7fadbfe00640 1 --2- 172.21.15.39:0/3987759506 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56081071bea0 0x560810726b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:46.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.454+0000 7fadbfe00640 1 -- 172.21.15.39:0/3987759506 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560810727150 con 0x560810716d30 2024-03-18T16:09:46.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.454+0000 7fadbfe00640 1 --2- 172.21.15.39:0/3987759506 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560810716d30 0x560810723b40 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fadb0098030 tx=0x7fadb000ef90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f17f2209dd2ebebe server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:46.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.454+0000 7fadb6000640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:09:46.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.455+0000 7fadb6000640 1 -- 172.21.15.39:0/3987759506 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadb009b660 con 0x560810716d30 2024-03-18T16:09:46.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.455+0000 7fadb6000640 1 -- 172.21.15.39:0/3987759506 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fadb00cc8d0 con 0x560810716d30 2024-03-18T16:09:46.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.455+0000 7fadd1e5b740 1 -- 172.21.15.39:0/3987759506 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fadc00046e0 con 0x560810716d30 2024-03-18T16:09:46.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.455+0000 7fadb6000640 1 -- 172.21.15.39:0/3987759506 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fadb009f7b0 con 0x560810716d30 2024-03-18T16:09:46.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.455+0000 7fadd1e5b740 10 client.25430 fetch_fsmap learned FSMap version 27 2024-03-18T16:09:46.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.455+0000 7fadd1e5b740 1 -- 172.21.15.39:0/3987759506 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x560810792ff0 con 0x560810716d30 2024-03-18T16:09:46.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.455+0000 7fadb6000640 1 -- 172.21.15.39:0/3987759506 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fadc00046e0 con 0x560810716d30 2024-03-18T16:09:46.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.455+0000 7fadb6000640 1 -- 172.21.15.39:0/3987759506 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fadb00d10b0 con 0x560810716d30 2024-03-18T16:09:46.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.455+0000 7fadd1e5b740 10 client.25430 fetch_fsmap finished waiting for FSMap version 27 2024-03-18T16:09:46.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.455+0000 7fadd1e5b740 10 client.25430 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-18T16:09:46.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.455+0000 7fadd1e5b740 20 client.25430 populate_metadata read hostname 'smithi039' 2024-03-18T16:09:46.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.455+0000 7fadd1e5b740 1 --2- 172.21.15.39:0/3987759506 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x560810797b70 0x560810796880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:46.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.455+0000 7fadd1e5b740 4 client.25430 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-18T16:09:46.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.455+0000 7fadd1e5b740 1 -- 172.21.15.39:0/3987759506 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "status"}) -- 0x560810688190 con 0x560810797b70 2024-03-18T16:09:46.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.456+0000 7fadc7e00640 1 --2- 172.21.15.39:0/3987759506 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x560810797b70 0x560810796880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:46.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.456+0000 7fadc7e00640 1 --2- 172.21.15.39:0/3987759506 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x560810797b70 0x560810796880 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f6128b739ebc386e server_cookie=14cdef1f64e49499 in_seq=0 out_seq=0 2024-03-18T16:09:46.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.456+0000 7fadb6000640 10 client.25430 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:09:46.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.456+0000 7fadb6000640 1 -- 172.21.15.39:0/3987759506 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x560810688190 con 0x560810797b70 2024-03-18T16:09:46.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.456+0000 7fadb6000640 10 client.25430 handle_command_reply: tid=0 2024-03-18T16:09:46.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.456+0000 7fadd1e5b740 1 client.25430 shutdown 2024-03-18T16:09:46.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.456+0000 7fadd1e5b740 1 -- 172.21.15.39:0/3987759506 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560810716d30 msgr2=0x560810723b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:46.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.456+0000 7fadd1e5b740 1 --2- 172.21.15.39:0/3987759506 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560810716d30 0x560810723b40 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fadb0098030 tx=0x7fadb000ef90 comp rx=0 tx=0).stop 2024-03-18T16:09:46.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.456+0000 7fadd1e5b740 1 -- 172.21.15.39:0/3987759506 shutdown_connections 2024-03-18T16:09:46.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.456+0000 7fadd1e5b740 1 --2- 172.21.15.39:0/3987759506 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x560810797b70 0x560810796880 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:46.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.457+0000 7fadd1e5b740 1 --2- 172.21.15.39:0/3987759506 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560810716d30 0x560810723b40 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:46.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.457+0000 7fadb6000640 0 client.25430 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:09:46.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.457+0000 7fadd1e5b740 1 --2- 172.21.15.39:0/3987759506 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56081071bea0 0x560810726b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:46.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.457+0000 7fadd1e5b740 1 --2- 172.21.15.39:0/3987759506 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560810716340 0x560810723600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:46.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.457+0000 7fadd1e5b740 1 -- 172.21.15.39:0/3987759506 >> 172.21.15.39:0/3987759506 conn(0x5608107118b0 msgr2=0x560810710a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:46.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.457+0000 7fadd1e5b740 1 -- 172.21.15.39:0/3987759506 shutdown_connections 2024-03-18T16:09:46.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.457+0000 7fadd1e5b740 1 -- 172.21.15.39:0/3987759506 wait complete. 2024-03-18T16:09:46.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.457+0000 7fadd1e5b740 20 client.25430 trim_cache size 0 max 16384 2024-03-18T16:09:46.464 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-03-18T16:09:46.465 INFO:teuthology.orchestra.run.smithi039.stdout: "cluster_fsid": "1892d8fc-e540-11ee-95c9-87774f69a715", 2024-03-18T16:09:46.465 INFO:teuthology.orchestra.run.smithi039.stdout: "whoami": 0, 2024-03-18T16:09:46.465 INFO:teuthology.orchestra.run.smithi039.stdout: "id": 24575, 2024-03-18T16:09:46.465 INFO:teuthology.orchestra.run.smithi039.stdout: "want_state": "up:active", 2024-03-18T16:09:46.465 INFO:teuthology.orchestra.run.smithi039.stdout: "state": "up:active", 2024-03-18T16:09:46.465 INFO:teuthology.orchestra.run.smithi039.stdout: "fs_name": "cephfs", 2024-03-18T16:09:46.465 INFO:teuthology.orchestra.run.smithi039.stdout: "rank_uptime": 181.63347030700001, 2024-03-18T16:09:46.465 INFO:teuthology.orchestra.run.smithi039.stdout: "mdsmap_epoch": 27, 2024-03-18T16:09:46.465 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_epoch": 93, 2024-03-18T16:09:46.465 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_epoch_barrier": 89, 2024-03-18T16:09:46.465 INFO:teuthology.orchestra.run.smithi039.stdout: "uptime": 189.03259674899999 2024-03-18T16:09:46.465 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-03-18T16:09:46.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.457+0000 7fadcdc00640 1 -- 172.21.15.39:0/1164547089 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fadac0a2800 msgr2=0x7fadac0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:46.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.457+0000 7fadcdc00640 1 --2- 172.21.15.39:0/1164547089 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fadac0a2800 0x7fadac0a4cc0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x560810717590 tx=0x7fadc0009500 comp rx=0 tx=0).stop 2024-03-18T16:09:46.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.457+0000 7fadcdc00640 1 -- 172.21.15.39:0/1164547089 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fadc8104a70 msgr2=0x7fadc8197f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:46.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.457+0000 7fadcdc00640 1 --2- 172.21.15.39:0/1164547089 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fadc8104a70 0x7fadc8197f10 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fadb000e9a0 tx=0x7fadb000ee60 comp rx=0 tx=0).stop 2024-03-18T16:09:46.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.458+0000 7fadcdc00640 1 -- 172.21.15.39:0/1164547089 shutdown_connections 2024-03-18T16:09:46.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.458+0000 7fadcdc00640 1 --2- 172.21.15.39:0/1164547089 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fadac0a2800 0x7fadac0a4cc0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:46.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.458+0000 7fadcdc00640 1 --2- 172.21.15.39:0/1164547089 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fadc81073a0 0x7fadc819ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:46.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.458+0000 7fadcdc00640 1 --2- 172.21.15.39:0/1164547089 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fadc8104a70 0x7fadc8197f10 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:46.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.458+0000 7fadcdc00640 1 --2- 172.21.15.39:0/1164547089 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fadc8102140 0x7fadc81979d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:46.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.458+0000 7fadcdc00640 1 -- 172.21.15.39:0/1164547089 >> 172.21.15.39:0/1164547089 conn(0x7fadc80fbb50 msgr2=0x7fadc8102ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:46.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.458+0000 7fadcdc00640 1 -- 172.21.15.39:0/1164547089 shutdown_connections 2024-03-18T16:09:46.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:46.458+0000 7fadcdc00640 1 -- 172.21.15.39:0/1164547089 wait complete. 2024-03-18T16:09:46.479 INFO:teuthology.run_tasks:Running task kclient... 2024-03-18T16:09:46.492 INFO:tasks.kclient:Mounting kernel clients... 2024-03-18T16:09:46.492 INFO:tasks.kclient:config is {'client.0': {}} 2024-03-18T16:09:46.493 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T16:09:46.493 DEBUG:teuthology.orchestra.run.smithi039:> ip netns list 2024-03-18T16:09:46.540 INFO:teuthology.orchestra.run.smithi039.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-03-18T16:09:46.541 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T16:09:46.542 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-18T16:09:46.673 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T16:09:46.673 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip link delete ceph-brx 2024-03-18T16:09:46.801 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=secure', 'wsync'], 'syntax': 'v1', 'mount_subvol_num': 1} 2024-03-18T16:09:46.801 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-18T16:09:46.801 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-18T16:09:46.802 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph fs ls 2024-03-18T16:09:47.275 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T16:09:47.435 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:47 smithi039 ceph-mon[31721]: pgmap v413: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T16:09:47.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:47 smithi196 ceph-mon[36307]: pgmap v413: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T16:09:47.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:47 smithi194 ceph-mon[36200]: pgmap v413: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T16:09:49.457 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:49 smithi039 ceph-mon[31721]: pgmap v414: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T16:09:49.458 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:49 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:09:49.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:49 smithi196 ceph-mon[36307]: pgmap v414: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T16:09:49.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:49 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:09:49.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:49 smithi194 ceph-mon[36200]: pgmap v414: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T16:09:49.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:49 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:09:49.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:49.802+0000 7f1a62600700 1 -- 172.21.15.39:0/4226498257 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1a5c10a5b0 msgr2=0x7f1a5c10a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:49.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:49.802+0000 7f1a62600700 1 --2- 172.21.15.39:0/4226498257 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1a5c10a5b0 0x7f1a5c10a990 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c0099c0 tx=0x7f1a4c009cd0 comp rx=0 tx=0).stop 2024-03-18T16:09:49.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:49.802+0000 7f1a62600700 1 -- 172.21.15.39:0/4226498257 shutdown_connections 2024-03-18T16:09:49.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:49.802+0000 7f1a62600700 1 --2- 172.21.15.39:0/4226498257 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1a5c104ec0 0x7f1a5c111380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:49.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:49.802+0000 7f1a62600700 1 --2- 172.21.15.39:0/4226498257 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1a5c10a5b0 0x7f1a5c10a990 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:49.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:49.802+0000 7f1a62600700 1 --2- 172.21.15.39:0/4226498257 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1a5c104500 0x7f1a5c104980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:49.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:49.802+0000 7f1a62600700 1 -- 172.21.15.39:0/4226498257 >> 172.21.15.39:0/4226498257 conn(0x7f1a5c0fffb0 msgr2=0x7f1a5c1023d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:49.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:49.802+0000 7f1a62600700 1 -- 172.21.15.39:0/4226498257 shutdown_connections 2024-03-18T16:09:49.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:49.803+0000 7f1a62600700 1 -- 172.21.15.39:0/4226498257 wait complete. 2024-03-18T16:09:49.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:49.804+0000 7f1a62600700 1 Processor -- start 2024-03-18T16:09:49.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:49.804+0000 7f1a62600700 1 -- start start 2024-03-18T16:09:49.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:49.805+0000 7f1a62600700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1a5c104500 0x7f1a5c1a0890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:49.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:49.805+0000 7f1a62600700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1a5c104ec0 0x7f1a5c1a0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:49.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:49.805+0000 7f1a62600700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1a5c10a5b0 0x7f1a5c19aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:49.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:49.805+0000 7f1a62600700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f1a5c114850 con 0x7f1a5c10a5b0 2024-03-18T16:09:49.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:49.805+0000 7f1a62600700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f1a5c1146d0 con 0x7f1a5c104ec0 2024-03-18T16:09:49.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:49.805+0000 7f1a62600700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f1a5c1149d0 con 0x7f1a5c104500 2024-03-18T16:09:49.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:49.805+0000 7f1a5aa00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1a5c104ec0 0x7f1a5c1a0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:49.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:49.805+0000 7f1a5b400700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1a5c104500 0x7f1a5c1a0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:49.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:49.805+0000 7f1a5aa00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1a5c104ec0 0x7f1a5c1a0dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:57070/0 (socket says 172.21.15.39:57070) 2024-03-18T16:09:49.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:49.805+0000 7f1a5b400700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1a5c104500 0x7f1a5c1a0890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:52018/0 (socket says 172.21.15.39:52018) 2024-03-18T16:09:49.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:49.805+0000 7f1a5aa00700 1 -- 172.21.15.39:0/101147568 learned_addr learned my addr 172.21.15.39:0/101147568 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:49.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:49.805+0000 7f1a5be00700 1 --2- 172.21.15.39:0/101147568 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1a5c10a5b0 0x7f1a5c19aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:49.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:49.805+0000 7f1a5aa00700 1 -- 172.21.15.39:0/101147568 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1a5c104500 msgr2=0x7f1a5c1a0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:49.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:49.805+0000 7f1a5aa00700 1 --2- 172.21.15.39:0/101147568 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1a5c104500 0x7f1a5c1a0890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:49.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:49.805+0000 7f1a5aa00700 1 -- 172.21.15.39:0/101147568 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1a5c10a5b0 msgr2=0x7f1a5c19aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:49.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:49.805+0000 7f1a5aa00700 1 --2- 172.21.15.39:0/101147568 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1a5c10a5b0 0x7f1a5c19aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:49.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:49.805+0000 7f1a5aa00700 1 -- 172.21.15.39:0/101147568 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a5c19b1b0 con 0x7f1a5c104ec0 2024-03-18T16:09:49.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:49.806+0000 7f1a5aa00700 1 --2- 172.21.15.39:0/101147568 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1a5c104ec0 0x7f1a5c1a0dd0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f1a4800e9c0 tx=0x7f1a4800ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:49.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:49.806+0000 7f1a53e00700 1 -- 172.21.15.39:0/101147568 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a4800cbd0 con 0x7f1a5c104ec0 2024-03-18T16:09:49.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:49.806+0000 7f1a53e00700 1 -- 172.21.15.39:0/101147568 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1a4800cd70 con 0x7f1a5c104ec0 2024-03-18T16:09:49.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:49.806+0000 7f1a62600700 1 -- 172.21.15.39:0/101147568 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a5c19b4a0 con 0x7f1a5c104ec0 2024-03-18T16:09:49.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:49.806+0000 7f1a53e00700 1 -- 172.21.15.39:0/101147568 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a48010430 con 0x7f1a5c104ec0 2024-03-18T16:09:49.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:49.806+0000 7f1a62600700 1 -- 172.21.15.39:0/101147568 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a5c1a7760 con 0x7f1a5c104ec0 2024-03-18T16:09:49.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:49.808+0000 7f1a62600700 1 -- 172.21.15.39:0/101147568 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a5c10ca40 con 0x7f1a5c104ec0 2024-03-18T16:09:49.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:49.809+0000 7f1a53e00700 1 -- 172.21.15.39:0/101147568 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f1a48004750 con 0x7f1a5c104ec0 2024-03-18T16:09:49.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:49.811+0000 7f1a53e00700 1 --2- 172.21.15.39:0/101147568 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1a400a2910 0x7f1a400a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:49.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:49.811+0000 7f1a53e00700 1 -- 172.21.15.39:0/101147568 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f1a480cf090 con 0x7f1a5c104ec0 2024-03-18T16:09:49.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:49.811+0000 7f1a5b400700 1 --2- 172.21.15.39:0/101147568 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1a400a2910 0x7f1a400a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:49.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:49.812+0000 7f1a5b400700 1 --2- 172.21.15.39:0/101147568 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1a400a2910 0x7f1a400a4dd0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c000c00 tx=0x7f1a4c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:49.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:49.816+0000 7f1a53e00700 1 -- 172.21.15.39:0/101147568 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1a4809bd40 con 0x7f1a5c104ec0 2024-03-18T16:09:50.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:50.110+0000 7f1a62600700 1 -- 172.21.15.39:0/101147568 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f1a5c19bf80 con 0x7f1a5c104ec0 2024-03-18T16:09:50.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:50.111+0000 7f1a53e00700 1 -- 172.21.15.39:0/101147568 <== mon.2 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v27) ==== 53+0+88 (secure 0 0 0) 0x7f1a48097020 con 0x7f1a5c104ec0 2024-03-18T16:09:50.113 INFO:teuthology.orchestra.run.smithi039.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-18T16:09:50.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:50.114+0000 7f1a62600700 1 -- 172.21.15.39:0/101147568 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1a400a2910 msgr2=0x7f1a400a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:50.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:50.114+0000 7f1a62600700 1 --2- 172.21.15.39:0/101147568 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1a400a2910 0x7f1a400a4dd0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f1a4c000c00 tx=0x7f1a4c005fb0 comp rx=0 tx=0).stop 2024-03-18T16:09:50.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:50.114+0000 7f1a62600700 1 -- 172.21.15.39:0/101147568 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1a5c104ec0 msgr2=0x7f1a5c1a0dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:50.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:50.114+0000 7f1a62600700 1 --2- 172.21.15.39:0/101147568 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1a5c104ec0 0x7f1a5c1a0dd0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f1a4800e9c0 tx=0x7f1a4800ecd0 comp rx=0 tx=0).stop 2024-03-18T16:09:50.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:50.114+0000 7f1a62600700 1 -- 172.21.15.39:0/101147568 shutdown_connections 2024-03-18T16:09:50.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:50.114+0000 7f1a62600700 1 --2- 172.21.15.39:0/101147568 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1a400a2910 0x7f1a400a4dd0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:50.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:50.114+0000 7f1a62600700 1 --2- 172.21.15.39:0/101147568 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1a5c10a5b0 0x7f1a5c19aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:50.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:50.114+0000 7f1a62600700 1 --2- 172.21.15.39:0/101147568 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1a5c104500 0x7f1a5c1a0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:50.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:50.114+0000 7f1a62600700 1 --2- 172.21.15.39:0/101147568 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1a5c104ec0 0x7f1a5c1a0dd0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:50.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:50.114+0000 7f1a62600700 1 -- 172.21.15.39:0/101147568 >> 172.21.15.39:0/101147568 conn(0x7f1a5c0fffb0 msgr2=0x7f1a5c101580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:50.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:50.115+0000 7f1a62600700 1 -- 172.21.15.39:0/101147568 shutdown_connections 2024-03-18T16:09:50.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:50.115+0000 7f1a62600700 1 -- 172.21.15.39:0/101147568 wait complete. 2024-03-18T16:09:50.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:50 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:09:50.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:50 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:09:50.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:50 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:09:50.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:50 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:09:50.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:50 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:09:50.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:50 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:09:50.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:50 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/101147568' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-18T16:09:50.575 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:50 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:09:50.575 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:50 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:09:50.576 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:50 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:09:50.576 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:50 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:09:50.576 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:50 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:09:50.576 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:50 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:09:50.576 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:50 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/101147568' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-18T16:09:50.580 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:50 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:09:50.580 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:50 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:09:50.580 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:50 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:09:50.580 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:50 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:09:50.580 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:50 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:09:50.580 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:50 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:09:50.581 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:50 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/101147568' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-18T16:09:51.149 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-18T16:09:51.149 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-18T16:09:51.149 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi039.front.sepia.ceph.com 2024-03-18T16:09:51.149 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-18T16:09:51.149 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-18T16:09:51.150 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-18T16:09:51.150 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-18T16:09:51.150 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-18T16:09:51.150 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-18T16:09:51.150 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T16:09:51.150 DEBUG:teuthology.orchestra.run.smithi039:> ip addr 2024-03-18T16:09:51.194 INFO:teuthology.orchestra.run.smithi039.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-18T16:09:51.194 INFO:teuthology.orchestra.run.smithi039.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-18T16:09:51.194 INFO:teuthology.orchestra.run.smithi039.stdout: inet 127.0.0.1/8 scope host lo 2024-03-18T16:09:51.194 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-03-18T16:09:51.194 INFO:teuthology.orchestra.run.smithi039.stdout: inet6 ::1/128 scope host 2024-03-18T16:09:51.194 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-03-18T16:09:51.194 INFO:teuthology.orchestra.run.smithi039.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T16:09:51.194 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 0c:c4:7a:6b:fe:c6 brd ff:ff:ff:ff:ff:ff 2024-03-18T16:09:51.194 INFO:teuthology.orchestra.run.smithi039.stdout: altname enp7s0f0 2024-03-18T16:09:51.194 INFO:teuthology.orchestra.run.smithi039.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T16:09:51.195 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 0c:c4:7a:6b:fe:c7 brd ff:ff:ff:ff:ff:ff 2024-03-18T16:09:51.195 INFO:teuthology.orchestra.run.smithi039.stdout: altname enp7s0f1 2024-03-18T16:09:51.195 INFO:teuthology.orchestra.run.smithi039.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-18T16:09:51.195 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 0c:c4:7a:bd:13:70 brd ff:ff:ff:ff:ff:ff 2024-03-18T16:09:51.195 INFO:teuthology.orchestra.run.smithi039.stdout: altname enp1s0f0 2024-03-18T16:09:51.195 INFO:teuthology.orchestra.run.smithi039.stdout: inet 172.21.15.39/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-03-18T16:09:51.195 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft 41846sec preferred_lft 41846sec 2024-03-18T16:09:51.195 INFO:teuthology.orchestra.run.smithi039.stdout: inet6 fe80::ec4:7aff:febd:1370/64 scope link noprefixroute 2024-03-18T16:09:51.195 INFO:teuthology.orchestra.run.smithi039.stdout: valid_lft forever preferred_lft forever 2024-03-18T16:09:51.195 INFO:teuthology.orchestra.run.smithi039.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-18T16:09:51.195 INFO:teuthology.orchestra.run.smithi039.stdout: link/ether 0c:c4:7a:bd:13:71 brd ff:ff:ff:ff:ff:ff 2024-03-18T16:09:51.195 INFO:teuthology.orchestra.run.smithi039.stdout: altname enp1s0f1 2024-03-18T16:09:51.196 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-18T16:09:51.197 DEBUG:teuthology.orchestra.run.smithi039:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T16:09:51.197 DEBUG:teuthology.orchestra.run.smithi039:> set -e 2024-03-18T16:09:51.197 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip link add name ceph-brx type bridge 2024-03-18T16:09:51.197 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip addr flush dev ceph-brx 2024-03-18T16:09:51.197 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip link set ceph-brx up 2024-03-18T16:09:51.197 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-18T16:09:51.197 DEBUG:teuthology.orchestra.run.smithi039:> ') 2024-03-18T16:09:51.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:51 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T16:09:51.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T16:09:51.502 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T16:09:51.502 DEBUG:teuthology.orchestra.run.smithi039:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-18T16:09:51.577 INFO:teuthology.orchestra.run.smithi039.stdout:1 2024-03-18T16:09:51.582 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T16:09:51.583 DEBUG:teuthology.orchestra.run.smithi039:> route 2024-03-18T16:09:51.631 INFO:teuthology.orchestra.run.smithi039.stdout:Kernel IP routing table 2024-03-18T16:09:51.631 INFO:teuthology.orchestra.run.smithi039.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-18T16:09:51.631 INFO:teuthology.orchestra.run.smithi039.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-03-18T16:09:51.631 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-03-18T16:09:51.631 INFO:teuthology.orchestra.run.smithi039.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-18T16:09:51.632 DEBUG:teuthology.orchestra.run.smithi039:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T16:09:51.632 DEBUG:teuthology.orchestra.run.smithi039:> set -e 2024-03-18T16:09:51.632 DEBUG:teuthology.orchestra.run.smithi039:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-03-18T16:09:51.632 DEBUG:teuthology.orchestra.run.smithi039:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-03-18T16:09:51.632 DEBUG:teuthology.orchestra.run.smithi039:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-03-18T16:09:51.632 DEBUG:teuthology.orchestra.run.smithi039:> ') 2024-03-18T16:09:51.712 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:51 smithi039 ceph-mon[31721]: pgmap v415: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T16:09:51.712 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:51 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:09:51.712 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:51 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:09:51.713 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:51 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:09:51.713 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:51 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:09:51.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:51 smithi196 ceph-mon[36307]: pgmap v415: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T16:09:51.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:51 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:09:51.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:51 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:09:51.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:51 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:09:51.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:51 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:09:51.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:51 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T16:09:51.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:51 smithi194 ceph-mon[36200]: pgmap v415: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T16:09:51.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:51 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:09:51.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:51 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:09:51.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:51 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:09:51.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:51 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:09:51.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:51 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T16:09:51.868 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T16:09:51.868 DEBUG:teuthology.orchestra.run.smithi039:> ip netns list 2024-03-18T16:09:51.912 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T16:09:51.912 DEBUG:teuthology.orchestra.run.smithi039:> ip netns list-id 2024-03-18T16:09:51.999 DEBUG:teuthology.orchestra.run.smithi039:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T16:09:51.999 DEBUG:teuthology.orchestra.run.smithi039:> set -e 2024-03-18T16:09:51.999 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T16:09:51.999 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-18T16:09:51.999 DEBUG:teuthology.orchestra.run.smithi039:> ') 2024-03-18T16:09:52.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:52 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T16:09:52.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T16:09:52.233 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-18T16:09:52.233 DEBUG:teuthology.orchestra.run.smithi039:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T16:09:52.233 DEBUG:teuthology.orchestra.run.smithi039:> set -e 2024-03-18T16:09:52.233 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-18T16:09:52.233 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-18T16:09:52.233 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-18T16:09:52.233 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-18T16:09:52.233 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-18T16:09:52.233 DEBUG:teuthology.orchestra.run.smithi039:> ') 2024-03-18T16:09:52.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:52 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T16:09:52.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T16:09:52.565 DEBUG:teuthology.orchestra.run.smithi039:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-18T16:09:52.565 DEBUG:teuthology.orchestra.run.smithi039:> set -e 2024-03-18T16:09:52.565 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip link set brx.0 up 2024-03-18T16:09:52.565 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip link set dev brx.0 master ceph-brx 2024-03-18T16:09:52.565 DEBUG:teuthology.orchestra.run.smithi039:> ') 2024-03-18T16:09:52.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:52 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T16:09:52.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:52 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T16:09:52.789 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T16:09:52.790 DEBUG:teuthology.orchestra.run.smithi039:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-18T16:09:52.829 INFO:teuthology.orchestra.run.smithi039.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-18T16:09:52.830 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-18T16:09:52.830 DEBUG:teuthology.orchestra.run.smithi039:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-18T16:09:52.909 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-03-18T16:09:52.909 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-03-18T16:09:52.909 INFO:tasks.cephfs.kernel_mount:mounting using device: :/volumes/qa/sv_1/ba28ee63-cdd7-4699-94b4-1139b8b6c674 2024-03-18T16:09:52.909 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T16:09:52.909 DEBUG:teuthology.orchestra.run.smithi039:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/volumes/qa/sv_1/ba28ee63-cdd7-4699-94b4-1139b8b6c674 /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync 2024-03-18T16:09:53.066 INFO:teuthology.orchestra.run.smithi039.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync 2024-03-18T16:09:53.066 INFO:teuthology.orchestra.run.smithi039.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync". 2024-03-18T16:09:53.066 INFO:teuthology.orchestra.run.smithi039.stdout:invalid new device string format 2024-03-18T16:09:53.695 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:53 smithi039 ceph-mon[31721]: pgmap v416: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-18T16:09:53.695 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:53 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:09:53.695 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:53 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:09:53.718 INFO:teuthology.orchestra.run.smithi039.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync 2024-03-18T16:09:53.719 INFO:teuthology.orchestra.run.smithi039.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync". 2024-03-18T16:09:53.719 INFO:teuthology.orchestra.run.smithi039.stdout:invalid new device string format 2024-03-18T16:09:53.719 INFO:teuthology.orchestra.run.smithi039.stdout:mount.ceph: resolved to: "172.21.15.39:3300,172.21.15.194:3300,172.21.15.196:3300" 2024-03-18T16:09:53.719 INFO:teuthology.orchestra.run.smithi039.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.39:3300,172.21.15.194:3300,172.21.15.196:3300:/volumes/qa/sv_1/ba28ee63-cdd7-4699-94b4-1139b8b6c674 2024-03-18T16:09:53.719 INFO:teuthology.orchestra.run.smithi039.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,wsync,key=0,fsid=1892d8fc-e540-11ee-95c9-87774f69a715" will pass to kernel 2024-03-18T16:09:53.723 INFO:teuthology.orchestra.run.smithi039.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-18T16:09:53.723 INFO:teuthology.orchestra.run.smithi039.stdout: You just mounted a file system that supports labels which does not 2024-03-18T16:09:53.723 INFO:teuthology.orchestra.run.smithi039.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-18T16:09:53.723 INFO:teuthology.orchestra.run.smithi039.stdout: applications will generate AVC messages and not be allowed access to 2024-03-18T16:09:53.723 INFO:teuthology.orchestra.run.smithi039.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-18T16:09:53.728 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-18T16:09:53.728 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T16:09:53.728 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-18T16:09:53.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:53 smithi196 ceph-mon[36307]: pgmap v416: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-18T16:09:53.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:53 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:09:53.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:53 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:09:53.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:53 smithi194 ceph-mon[36200]: pgmap v416: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-18T16:09:53.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:53 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:09:53.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:53 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:09:53.815 DEBUG:teuthology.orchestra.run.smithi039:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-18T16:09:53.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:53 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-18T16:09:53.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:53 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T16:09:53.917 INFO:teuthology.orchestra.run.smithi039.stdout:client25459 2024-03-18T16:09:53.917 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-03-18T16:09:53.917 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T16:09:53.956 DEBUG:teuthology.orchestra.run.smithi039:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-18T16:09:54.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:54 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-18T16:09:54.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T16:09:54.100 INFO:teuthology.orchestra.run.smithi039.stdout:client25459 2024-03-18T16:09:54.101 DEBUG:teuthology.orchestra.run.smithi039:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-18T16:09:54.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:54 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-03-18T16:09:54.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T16:09:54.203 INFO:teuthology.orchestra.run.smithi039.stdout:client25459 2024-03-18T16:09:54.204 DEBUG:teuthology.orchestra.run.smithi039:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/1892d8fc-e540-11ee-95c9-87774f69a715.client25459/status') 2024-03-18T16:09:54.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:54 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-18T16:09:54.344 INFO:teuthology.orchestra.run.smithi039.stdout:instance: client.25459 (3)172.21.15.39:0/607201086 2024-03-18T16:09:54.344 INFO:teuthology.orchestra.run.smithi039.stdout:blocklisted: false 2024-03-18T16:09:54.344 INFO:teuthology.orchestra.run.smithi039.stderr:0+1 records in 2024-03-18T16:09:54.344 INFO:teuthology.orchestra.run.smithi039.stderr:0+1 records out 2024-03-18T16:09:54.344 INFO:teuthology.orchestra.run.smithi039.stderr:70 bytes copied, 5.577e-05 s, 1.3 MB/s 2024-03-18T16:09:54.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:54 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-18T16:09:54.359 INFO:teuthology.run_tasks:Running task exec... 2024-03-18T16:09:54.373 INFO:teuthology.task.exec:Executing custom commands... 2024-03-18T16:09:54.374 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi039.front.sepia.ceph.com 2024-03-18T16:09:54.374 DEBUG:teuthology.orchestra.run.smithi039:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate true' 2024-03-18T16:09:54.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:54.605+0000 7fa921800640 1 -- 172.21.15.39:0/2746752004 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa91c103860 msgr2=0x7fa91c103c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:54.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:54.605+0000 7fa921800640 1 --2- 172.21.15.39:0/2746752004 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa91c103860 0x7fa91c103c80 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fa90c009900 tx=0x7fa90c02bc40 comp rx=0 tx=0).stop 2024-03-18T16:09:54.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:54.605+0000 7fa921800640 1 -- 172.21.15.39:0/2746752004 shutdown_connections 2024-03-18T16:09:54.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:54.605+0000 7fa921800640 1 --2- 172.21.15.39:0/2746752004 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa91c105410 0x7fa91c10bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:54.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:54.605+0000 7fa921800640 1 --2- 172.21.15.39:0/2746752004 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa91c104a50 0x7fa91c104ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:54.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:54.605+0000 7fa921800640 1 --2- 172.21.15.39:0/2746752004 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa91c103860 0x7fa91c103c80 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:54.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:54.605+0000 7fa921800640 1 -- 172.21.15.39:0/2746752004 >> 172.21.15.39:0/2746752004 conn(0x7fa91c0fedf0 msgr2=0x7fa91c101210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:54.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:54.605+0000 7fa921800640 1 -- 172.21.15.39:0/2746752004 shutdown_connections 2024-03-18T16:09:54.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:54.605+0000 7fa921800640 1 -- 172.21.15.39:0/2746752004 wait complete. 2024-03-18T16:09:54.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:54.606+0000 7fa921800640 1 Processor -- start 2024-03-18T16:09:54.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:54.607+0000 7fa921800640 1 -- start start 2024-03-18T16:09:54.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:54.607+0000 7fa921800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa91c103860 0x7fa91c199b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:54.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:54.607+0000 7fa921800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa91c104a50 0x7fa91c19a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:54.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:54.607+0000 7fa921800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa91c105410 0x7fa91c1a1100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:54.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:54.607+0000 7fa921800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fa91c10f4e0 con 0x7fa91c105410 2024-03-18T16:09:54.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:54.607+0000 7fa921800640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fa91c10f360 con 0x7fa91c103860 2024-03-18T16:09:54.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:54.607+0000 7fa921800640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fa91c10f660 con 0x7fa91c104a50 2024-03-18T16:09:54.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:54.607+0000 7fa91be00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa91c103860 0x7fa91c199b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:54.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:54.607+0000 7fa91be00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa91c103860 0x7fa91c199b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:57096/0 (socket says 172.21.15.39:57096) 2024-03-18T16:09:54.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:54.607+0000 7fa91be00640 1 -- 172.21.15.39:0/3517993472 learned_addr learned my addr 172.21.15.39:0/3517993472 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:54.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:54.607+0000 7fa91b400640 1 --2- 172.21.15.39:0/3517993472 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa91c104a50 0x7fa91c19a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:54.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:54.608+0000 7fa920e00640 1 --2- 172.21.15.39:0/3517993472 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa91c105410 0x7fa91c1a1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:54.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:54.608+0000 7fa91be00640 1 -- 172.21.15.39:0/3517993472 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa91c104a50 msgr2=0x7fa91c19a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:54.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:54.608+0000 7fa91be00640 1 --2- 172.21.15.39:0/3517993472 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa91c104a50 0x7fa91c19a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:54.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:54.608+0000 7fa91be00640 1 -- 172.21.15.39:0/3517993472 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa91c105410 msgr2=0x7fa91c1a1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:54.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:54.608+0000 7fa91be00640 1 --2- 172.21.15.39:0/3517993472 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa91c105410 0x7fa91c1a1100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:54.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:54.608+0000 7fa91be00640 1 -- 172.21.15.39:0/3517993472 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa91c1a1770 con 0x7fa91c103860 2024-03-18T16:09:54.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:54.608+0000 7fa920e00640 1 --2- 172.21.15.39:0/3517993472 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa91c105410 0x7fa91c1a1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:09:54.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:54.608+0000 7fa91be00640 1 --2- 172.21.15.39:0/3517993472 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa91c103860 0x7fa91c199b50 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fa90c000c00 tx=0x7fa90c02c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:54.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:54.608+0000 7fa918c00640 1 -- 172.21.15.39:0/3517993472 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa90c041070 con 0x7fa91c103860 2024-03-18T16:09:54.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:54.608+0000 7fa918c00640 1 -- 172.21.15.39:0/3517993472 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa90c004450 con 0x7fa91c103860 2024-03-18T16:09:54.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:54.608+0000 7fa918c00640 1 -- 172.21.15.39:0/3517993472 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa90c034440 con 0x7fa91c103860 2024-03-18T16:09:54.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:54.608+0000 7fa921800640 1 -- 172.21.15.39:0/3517993472 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa91c1a1a00 con 0x7fa91c103860 2024-03-18T16:09:54.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:54.609+0000 7fa921800640 1 -- 172.21.15.39:0/3517993472 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa91c1a1ea0 con 0x7fa91c103860 2024-03-18T16:09:54.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:54.610+0000 7fa921800640 1 -- 172.21.15.39:0/3517993472 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa91c10ab80 con 0x7fa91c103860 2024-03-18T16:09:54.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:54.612+0000 7fa918c00640 1 -- 172.21.15.39:0/3517993472 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fa90c044070 con 0x7fa91c103860 2024-03-18T16:09:54.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:54.613+0000 7fa918c00640 1 --2- 172.21.15.39:0/3517993472 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa9000a2870 0x7fa9000a4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:54.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:54.613+0000 7fa918c00640 1 -- 172.21.15.39:0/3517993472 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fa90c0f0b90 con 0x7fa91c103860 2024-03-18T16:09:54.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:54.613+0000 7fa91b400640 1 --2- 172.21.15.39:0/3517993472 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa9000a2870 0x7fa9000a4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:54.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:54.614+0000 7fa91b400640 1 --2- 172.21.15.39:0/3517993472 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa9000a2870 0x7fa9000a4d30 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fa9080095f0 tx=0x7fa908005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:54.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:54.615+0000 7fa918c00640 1 -- 172.21.15.39:0/3517993472 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa90c0bd8d0 con 0x7fa91c103860 2024-03-18T16:09:54.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:54.859+0000 7fa921800640 1 -- 172.21.15.39:0/3517993472 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"} v 0) -- 0x7fa91c02d370 con 0x7fa91c103860 2024-03-18T16:09:55.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:55.407+0000 7fa918c00640 1 -- 172.21.15.39:0/3517993472 <== mon.2 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]=0 v28) ==== 117+0+0 (secure 0 0 0) 0x7fa90c0bd1d0 con 0x7fa91c103860 2024-03-18T16:09:55.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:55.410+0000 7fa921800640 1 -- 172.21.15.39:0/3517993472 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa9000a2870 msgr2=0x7fa9000a4d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:55.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:55.410+0000 7fa921800640 1 --2- 172.21.15.39:0/3517993472 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa9000a2870 0x7fa9000a4d30 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fa9080095f0 tx=0x7fa908005f90 comp rx=0 tx=0).stop 2024-03-18T16:09:55.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:55.410+0000 7fa921800640 1 -- 172.21.15.39:0/3517993472 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa91c103860 msgr2=0x7fa91c199b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:55.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:55.410+0000 7fa921800640 1 --2- 172.21.15.39:0/3517993472 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa91c103860 0x7fa91c199b50 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fa90c000c00 tx=0x7fa90c02c650 comp rx=0 tx=0).stop 2024-03-18T16:09:55.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:55.411+0000 7fa921800640 1 -- 172.21.15.39:0/3517993472 shutdown_connections 2024-03-18T16:09:55.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:55.411+0000 7fa921800640 1 --2- 172.21.15.39:0/3517993472 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa9000a2870 0x7fa9000a4d30 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:55.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:55.411+0000 7fa921800640 1 --2- 172.21.15.39:0/3517993472 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa91c105410 0x7fa91c1a1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:55.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:55.411+0000 7fa921800640 1 --2- 172.21.15.39:0/3517993472 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa91c104a50 0x7fa91c19a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:55.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:55.411+0000 7fa921800640 1 --2- 172.21.15.39:0/3517993472 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa91c103860 0x7fa91c199b50 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:55.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:55.411+0000 7fa921800640 1 -- 172.21.15.39:0/3517993472 >> 172.21.15.39:0/3517993472 conn(0x7fa91c0fedf0 msgr2=0x7fa91c107d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:55.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:55.411+0000 7fa921800640 1 -- 172.21.15.39:0/3517993472 shutdown_connections 2024-03-18T16:09:55.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:55.411+0000 7fa921800640 1 -- 172.21.15.39:0/3517993472 wait complete. 2024-03-18T16:09:55.434 INFO:teuthology.run_tasks:Running task check-counter... 2024-03-18T16:09:55.445 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-03-18T16:09:55.489 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T16:09:55.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:55.719+0000 7f2a46000640 1 -- 172.21.15.39:0/799243104 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2a40107c20 msgr2=0x7f2a40108040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:55.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:55.719+0000 7f2a46000640 1 --2- 172.21.15.39:0/799243104 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2a40107c20 0x7f2a40108040 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f2a34009900 tx=0x7f2a3402bc50 comp rx=0 tx=0).stop 2024-03-18T16:09:55.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:55.719+0000 7f2a46000640 1 -- 172.21.15.39:0/799243104 shutdown_connections 2024-03-18T16:09:55.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:55.719+0000 7f2a46000640 1 --2- 172.21.15.39:0/799243104 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2a401097d0 0x7f2a40110060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:55.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:55.719+0000 7f2a46000640 1 --2- 172.21.15.39:0/799243104 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2a40108e10 0x7f2a40109290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:55.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:55.719+0000 7f2a46000640 1 --2- 172.21.15.39:0/799243104 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2a40107c20 0x7f2a40108040 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:55.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:55.719+0000 7f2a46000640 1 -- 172.21.15.39:0/799243104 >> 172.21.15.39:0/799243104 conn(0x7f2a40076bb0 msgr2=0x7f2a40078ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:55.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:55.719+0000 7f2a46000640 1 -- 172.21.15.39:0/799243104 shutdown_connections 2024-03-18T16:09:55.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:55.719+0000 7f2a46000640 1 -- 172.21.15.39:0/799243104 wait complete. 2024-03-18T16:09:55.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:55.720+0000 7f2a46000640 1 Processor -- start 2024-03-18T16:09:55.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:55.721+0000 7f2a46000640 1 -- start start 2024-03-18T16:09:55.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:55.721+0000 7f2a46000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2a40107c20 0x7f2a4019ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:55.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:55.721+0000 7f2a46000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2a40108e10 0x7f2a4019e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:55.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:55.721+0000 7f2a46000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2a401097d0 0x7f2a401a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:55.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:55.721+0000 7f2a46000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f2a401138f0 con 0x7f2a40108e10 2024-03-18T16:09:55.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:55.721+0000 7f2a46000640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f2a40113770 con 0x7f2a40107c20 2024-03-18T16:09:55.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:55.721+0000 7f2a46000640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f2a40113a70 con 0x7f2a401097d0 2024-03-18T16:09:55.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:55.722+0000 7f2a3fe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2a40108e10 0x7f2a4019e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:55.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:55.722+0000 7f2a45600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2a401097d0 0x7f2a401a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:55.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:55.722+0000 7f2a44c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2a40107c20 0x7f2a4019ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:55.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:55.722+0000 7f2a3fe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2a40108e10 0x7f2a4019e410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:33728/0 (socket says 172.21.15.39:33728) 2024-03-18T16:09:55.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:55.722+0000 7f2a3fe00640 1 -- 172.21.15.39:0/1016345133 learned_addr learned my addr 172.21.15.39:0/1016345133 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:55.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:55.722+0000 7f2a45600640 1 -- 172.21.15.39:0/1016345133 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2a40107c20 msgr2=0x7f2a4019ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:55.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:55.722+0000 7f2a45600640 1 --2- 172.21.15.39:0/1016345133 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2a40107c20 0x7f2a4019ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:55.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:55.722+0000 7f2a45600640 1 -- 172.21.15.39:0/1016345133 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2a40108e10 msgr2=0x7f2a4019e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:55.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:55.722+0000 7f2a45600640 1 --2- 172.21.15.39:0/1016345133 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2a40108e10 0x7f2a4019e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:55.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:55.722+0000 7f2a45600640 1 -- 172.21.15.39:0/1016345133 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a401a5af0 con 0x7f2a401097d0 2024-03-18T16:09:55.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:55.722+0000 7f2a45600640 1 --2- 172.21.15.39:0/1016345133 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2a401097d0 0x7f2a401a5480 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f2a38009fd0 tx=0x7f2a3800ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:55.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:55.723+0000 7f2a3d600640 1 -- 172.21.15.39:0/1016345133 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a3800ce20 con 0x7f2a401097d0 2024-03-18T16:09:55.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:55.723+0000 7f2a3d600640 1 -- 172.21.15.39:0/1016345133 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a38005020 con 0x7f2a401097d0 2024-03-18T16:09:55.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:55.723+0000 7f2a46000640 1 -- 172.21.15.39:0/1016345133 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a401a5de0 con 0x7f2a401097d0 2024-03-18T16:09:55.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:55.723+0000 7f2a3d600640 1 -- 172.21.15.39:0/1016345133 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a38010720 con 0x7f2a401097d0 2024-03-18T16:09:55.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:55.723+0000 7f2a46000640 1 -- 172.21.15.39:0/1016345133 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a401a62f0 con 0x7f2a401097d0 2024-03-18T16:09:55.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:55.724+0000 7f2a46000640 1 -- 172.21.15.39:0/1016345133 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a4010eee0 con 0x7f2a401097d0 2024-03-18T16:09:55.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:55.726+0000 7f2a3d600640 1 -- 172.21.15.39:0/1016345133 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f2a38005540 con 0x7f2a401097d0 2024-03-18T16:09:55.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:55 smithi196 ceph-mon[36307]: pgmap v417: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T16:09:55.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:55 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:09:55.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:55 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:09:55.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:55 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:09:55.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:55 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/3517993472' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-18T16:09:55.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:55 smithi196 ceph-mon[36307]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-18T16:09:55.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:55.728+0000 7f2a3d600640 1 --2- 172.21.15.39:0/1016345133 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f2a280a2880 0x7f2a280a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:55.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:55.728+0000 7f2a3d600640 1 -- 172.21.15.39:0/1016345133 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f2a380cf340 con 0x7f2a401097d0 2024-03-18T16:09:55.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:55.728+0000 7f2a44c00640 1 --2- 172.21.15.39:0/1016345133 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f2a280a2880 0x7f2a280a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:55.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:55.729+0000 7f2a44c00640 1 --2- 172.21.15.39:0/1016345133 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f2a280a2880 0x7f2a280a4d40 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f2a3402c190 tx=0x7f2a34000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:55.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:55.732+0000 7f2a3d600640 1 -- 172.21.15.39:0/1016345133 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2a3809bff0 con 0x7f2a401097d0 2024-03-18T16:09:55.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:55 smithi194 ceph-mon[36200]: pgmap v417: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T16:09:55.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:55 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:09:55.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:55 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:09:55.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:55 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:09:55.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:55 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/3517993472' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-18T16:09:55.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:55 smithi194 ceph-mon[36200]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-18T16:09:55.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:55 smithi039 ceph-mon[31721]: pgmap v417: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T16:09:55.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:55 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:09:55.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:55 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:09:55.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:55 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:09:55.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:55 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/3517993472' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-18T16:09:55.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:55 smithi039 ceph-mon[31721]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-03-18T16:09:55.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:55.936+0000 7f2a46000640 1 -- 172.21.15.39:0/1016345133 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2a400623e0 con 0x7f2a401097d0 2024-03-18T16:09:55.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:55.938+0000 7f2a3d600640 1 -- 172.21.15.39:0/1016345133 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24762 (secure 0 0 0) 0x7f2a3809b8f0 con 0x7f2a401097d0 2024-03-18T16:09:55.939 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:09:55.940 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":93,"fsid":"1892d8fc-e540-11ee-95c9-87774f69a715","created":"2024-03-18T15:57:36.507008+0000","modified":"2024-03-18T16:08:47.874967+0000","last_up_change":"2024-03-18T16:05:45.250629+0000","last_in_change":"2024-03-18T16:05:24.841337+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T16:01:30.048247+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T16:06:42.191791+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T16:06:43.670325+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T16:06:47.523388+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"771e0558-be62-4436-94b0-6364751911c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6803","nonce":4090743635}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6805","nonce":4090743635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6809","nonce":4090743635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6807","nonce":4090743635}]},"public_addr":"172.21.15.39:6803/4090743635","cluster_addr":"172.21.15.39:6805/4090743635","heartbeat_back_addr":"172.21.15.39:6809/4090743635","heartbeat_front_addr":"172.21.15.39:6807/4090743635","state":["exists","up"]},{"osd":1,"uuid":"0430eee1-cf34-4277-b255-c7d9f07ad19a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6801","nonce":2800329805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6803","nonce":2800329805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6807","nonce":2800329805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6805","nonce":2800329805}]},"public_addr":"172.21.15.194:6801/2800329805","cluster_addr":"172.21.15.194:6803/2800329805","heartbeat_back_addr":"172.21.15.194:6807/2800329805","heartbeat_front_addr":"172.21.15.194:6805/2800329805","state":["exists","up"]},{"osd":2,"uuid":"3a718da4-2511-4ec1-b1a8-eafd6cce0df0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6801","nonce":3935874719}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6803","nonce":3935874719}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6807","nonce":3935874719}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6805","nonce":3935874719}]},"public_addr":"172.21.15.196:6801/3935874719","cluster_addr":"172.21.15.196:6803/3935874719","heartbeat_back_addr":"172.21.15.196:6807/3935874719","heartbeat_front_addr":"172.21.15.196:6805/3935874719","state":["exists","up"]},{"osd":3,"uuid":"6f9c37f9-0ab9-4038-8919-afd4b39c4165","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6811","nonce":893345939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6813","nonce":893345939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6817","nonce":893345939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6815","nonce":893345939}]},"public_addr":"172.21.15.39:6811/893345939","cluster_addr":"172.21.15.39:6813/893345939","heartbeat_back_addr":"172.21.15.39:6817/893345939","heartbeat_front_addr":"172.21.15.39:6815/893345939","state":["exists","up"]},{"osd":4,"uuid":"e795bb0e-4791-4537-acda-1a321e19bc9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6809","nonce":800423088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6811","nonce":800423088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6815","nonce":800423088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6813","nonce":800423088}]},"public_addr":"172.21.15.194:6809/800423088","cluster_addr":"172.21.15.194:6811/800423088","heartbeat_back_addr":"172.21.15.194:6815/800423088","heartbeat_front_addr":"172.21.15.194:6813/800423088","state":["exists","up"]},{"osd":5,"uuid":"026e175e-6acb-4b4c-b235-8ccee8ae1149","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6809","nonce":2746224525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6811","nonce":2746224525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6815","nonce":2746224525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6813","nonce":2746224525}]},"public_addr":"172.21.15.196:6809/2746224525","cluster_addr":"172.21.15.196:6811/2746224525","heartbeat_back_addr":"172.21.15.196:6815/2746224525","heartbeat_front_addr":"172.21.15.196:6813/2746224525","state":["exists","up"]},{"osd":6,"uuid":"c1b183ff-383f-4c5c-9075-f76bab1c0bb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6819","nonce":1213371331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6821","nonce":1213371331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6825","nonce":1213371331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6823","nonce":1213371331}]},"public_addr":"172.21.15.39:6819/1213371331","cluster_addr":"172.21.15.39:6821/1213371331","heartbeat_back_addr":"172.21.15.39:6825/1213371331","heartbeat_front_addr":"172.21.15.39:6823/1213371331","state":["exists","up"]},{"osd":7,"uuid":"612f8697-957d-4ef4-a87f-241b36f9d13d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6817","nonce":3299162444}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6819","nonce":3299162444}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6823","nonce":3299162444}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6821","nonce":3299162444}]},"public_addr":"172.21.15.194:6817/3299162444","cluster_addr":"172.21.15.194:6819/3299162444","heartbeat_back_addr":"172.21.15.194:6823/3299162444","heartbeat_front_addr":"172.21.15.194:6821/3299162444","state":["exists","up"]},{"osd":8,"uuid":"836a9107-4d4f-4594-b8dc-0301e64f59d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6817","nonce":2456143145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6819","nonce":2456143145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6823","nonce":2456143145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6821","nonce":2456143145}]},"public_addr":"172.21.15.196:6817/2456143145","cluster_addr":"172.21.15.196:6819/2456143145","heartbeat_back_addr":"172.21.15.196:6823/2456143145","heartbeat_front_addr":"172.21.15.196:6821/2456143145","state":["exists","up"]},{"osd":9,"uuid":"4f30b960-12e3-4b16-bd28-fcc988139b58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6827","nonce":1848634433}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6829","nonce":1848634433}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6833","nonce":1848634433}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6831","nonce":1848634433}]},"public_addr":"172.21.15.39:6827/1848634433","cluster_addr":"172.21.15.39:6829/1848634433","heartbeat_back_addr":"172.21.15.39:6833/1848634433","heartbeat_front_addr":"172.21.15.39:6831/1848634433","state":["exists","up"]},{"osd":10,"uuid":"620f90e8-23e6-4bc0-8269-7219b9e1aebd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6825","nonce":3725064809}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6827","nonce":3725064809}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6831","nonce":3725064809}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6829","nonce":3725064809}]},"public_addr":"172.21.15.194:6825/3725064809","cluster_addr":"172.21.15.194:6827/3725064809","heartbeat_back_addr":"172.21.15.194:6831/3725064809","heartbeat_front_addr":"172.21.15.194:6829/3725064809","state":["exists","up"]},{"osd":11,"uuid":"867a2535-8235-4ceb-a698-eb86869d6d20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6825","nonce":2501212422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6827","nonce":2501212422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6831","nonce":2501212422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6829","nonce":2501212422}]},"public_addr":"172.21.15.196:6825/2501212422","cluster_addr":"172.21.15.196:6827/2501212422","heartbeat_back_addr":"172.21.15.196:6831/2501212422","heartbeat_front_addr":"172.21.15.196:6829/2501212422","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:00:36.931566+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:01:02.614024+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:01:26.193265+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:01:58.308833+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:02:25.049546+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:02:49.748750+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:03:23.582153+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:03:50.447321+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:04:15.812129+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:04:49.998047+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:05:17.087924+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:05:43.167637+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":4,"to":7}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.a","mappings":[{"from":5,"to":8}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.39:0/3482086140":"2024-03-19T15:58:45.340294+0000","172.21.15.39:0/3844306284":"2024-03-19T15:58:45.340294+0000","172.21.15.39:6801/1810238546":"2024-03-19T15:58:45.340294+0000","172.21.15.39:6800/1810238546":"2024-03-19T15:58:45.340294+0000","172.21.15.39:0/2528397488":"2024-03-19T15:58:10.264798+0000","172.21.15.39:0/2240981254":"2024-03-19T15:58:10.264798+0000","172.21.15.39:6800/1217503189":"2024-03-19T15:58:10.264798+0000","172.21.15.39:0/3754876370":"2024-03-19T15:58:10.264798+0000","172.21.15.39:6801/1217503189":"2024-03-19T15:58:10.264798+0000","172.21.15.39:0/1874637628":"2024-03-19T15:58:45.340294+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T16:09:55.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:55.940+0000 7f2a46000640 1 -- 172.21.15.39:0/1016345133 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f2a280a2880 msgr2=0x7f2a280a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:55.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:55.940+0000 7f2a46000640 1 --2- 172.21.15.39:0/1016345133 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f2a280a2880 0x7f2a280a4d40 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f2a3402c190 tx=0x7f2a34000bc0 comp rx=0 tx=0).stop 2024-03-18T16:09:55.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:55.940+0000 7f2a46000640 1 -- 172.21.15.39:0/1016345133 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2a401097d0 msgr2=0x7f2a401a5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:55.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:55.940+0000 7f2a46000640 1 --2- 172.21.15.39:0/1016345133 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2a401097d0 0x7f2a401a5480 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f2a38009fd0 tx=0x7f2a3800ef10 comp rx=0 tx=0).stop 2024-03-18T16:09:55.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:55.940+0000 7f2a46000640 1 -- 172.21.15.39:0/1016345133 shutdown_connections 2024-03-18T16:09:55.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:55.940+0000 7f2a46000640 1 --2- 172.21.15.39:0/1016345133 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f2a280a2880 0x7f2a280a4d40 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:55.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:55.940+0000 7f2a46000640 1 --2- 172.21.15.39:0/1016345133 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2a401097d0 0x7f2a401a5480 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:55.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:55.940+0000 7f2a46000640 1 --2- 172.21.15.39:0/1016345133 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2a40108e10 0x7f2a4019e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:55.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:55.940+0000 7f2a46000640 1 --2- 172.21.15.39:0/1016345133 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2a40107c20 0x7f2a4019ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:55.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:55.940+0000 7f2a46000640 1 -- 172.21.15.39:0/1016345133 >> 172.21.15.39:0/1016345133 conn(0x7f2a40076bb0 msgr2=0x7f2a4010c0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:55.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:55.940+0000 7f2a46000640 1 -- 172.21.15.39:0/1016345133 shutdown_connections 2024-03-18T16:09:55.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:55.940+0000 7f2a46000640 1 -- 172.21.15.39:0/1016345133 wait complete. 2024-03-18T16:09:55.964 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T16:01:30.048247+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T16:06:42.191791+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T16:06:43.670325+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T16:06:47.523388+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T16:09:55.965 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T16:09:56.183 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.182+0000 7fbcce200640 1 -- 172.21.15.39:0/785196050 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbcc81028d0 msgr2=0x7fbcc8102d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:56.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.182+0000 7fbcce200640 1 --2- 172.21.15.39:0/785196050 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbcc81028d0 0x7fbcc8102d50 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fbcb4009900 tx=0x7fbcb402bc30 comp rx=0 tx=0).stop 2024-03-18T16:09:56.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.182+0000 7fbcce200640 1 -- 172.21.15.39:0/785196050 shutdown_connections 2024-03-18T16:09:56.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.182+0000 7fbcce200640 1 --2- 172.21.15.39:0/785196050 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbcc81028d0 0x7fbcc8102d50 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:56.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.182+0000 7fbcce200640 1 --2- 172.21.15.39:0/785196050 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbcc8103290 0x7fbcc8109b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:56.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.182+0000 7fbcce200640 1 --2- 172.21.15.39:0/785196050 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbcc81016e0 0x7fbcc8101b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:56.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.182+0000 7fbcce200640 1 -- 172.21.15.39:0/785196050 >> 172.21.15.39:0/785196050 conn(0x7fbcc80fcc50 msgr2=0x7fbcc80ff090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:56.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.182+0000 7fbcce200640 1 -- 172.21.15.39:0/785196050 shutdown_connections 2024-03-18T16:09:56.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.182+0000 7fbcce200640 1 -- 172.21.15.39:0/785196050 wait complete. 2024-03-18T16:09:56.185 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.183+0000 7fbcce200640 1 Processor -- start 2024-03-18T16:09:56.185 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.184+0000 7fbcce200640 1 -- start start 2024-03-18T16:09:56.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.184+0000 7fbcce200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbcc81016e0 0x7fbcc8199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:56.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.184+0000 7fbcce200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbcc81028d0 0x7fbcc819a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:56.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.184+0000 7fbcce200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbcc8103290 0x7fbcc81a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:56.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.184+0000 7fbcce200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fbcc810d3b0 con 0x7fbcc8103290 2024-03-18T16:09:56.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.184+0000 7fbcce200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fbcc810d230 con 0x7fbcc81028d0 2024-03-18T16:09:56.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.184+0000 7fbcce200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fbcc810d530 con 0x7fbcc81016e0 2024-03-18T16:09:56.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.184+0000 7fbcc7e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbcc81028d0 0x7fbcc819a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:56.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.184+0000 7fbccce00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbcc81016e0 0x7fbcc8199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:56.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.184+0000 7fbcc7e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbcc81028d0 0x7fbcc819a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:57138/0 (socket says 172.21.15.39:57138) 2024-03-18T16:09:56.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.184+0000 7fbcc7e00640 1 -- 172.21.15.39:0/2222342522 learned_addr learned my addr 172.21.15.39:0/2222342522 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:56.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.184+0000 7fbccd800640 1 --2- 172.21.15.39:0/2222342522 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbcc8103290 0x7fbcc81a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:56.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.185+0000 7fbcc7e00640 1 -- 172.21.15.39:0/2222342522 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbcc81016e0 msgr2=0x7fbcc8199c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:56.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.185+0000 7fbcc7e00640 1 --2- 172.21.15.39:0/2222342522 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbcc81016e0 0x7fbcc8199c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:56.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.185+0000 7fbcc7e00640 1 -- 172.21.15.39:0/2222342522 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbcc8103290 msgr2=0x7fbcc81a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:56.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.185+0000 7fbcc7e00640 1 --2- 172.21.15.39:0/2222342522 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbcc8103290 0x7fbcc81a1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:56.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.185+0000 7fbcc7e00640 1 -- 172.21.15.39:0/2222342522 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcc81a1830 con 0x7fbcc81028d0 2024-03-18T16:09:56.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.185+0000 7fbccce00640 1 --2- 172.21.15.39:0/2222342522 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbcc81016e0 0x7fbcc8199c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:09:56.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.185+0000 7fbcc7e00640 1 --2- 172.21.15.39:0/2222342522 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbcc81028d0 0x7fbcc819a160 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fbcb4005ab0 tx=0x7fbcb402c6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:56.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.185+0000 7fbcc5600640 1 -- 172.21.15.39:0/2222342522 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcb4043020 con 0x7fbcc81028d0 2024-03-18T16:09:56.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.185+0000 7fbcc5600640 1 -- 172.21.15.39:0/2222342522 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbcb40054e0 con 0x7fbcc81028d0 2024-03-18T16:09:56.189 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.185+0000 7fbcce200640 1 -- 172.21.15.39:0/2222342522 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbcc81a1ac0 con 0x7fbcc81028d0 2024-03-18T16:09:56.189 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.185+0000 7fbcc5600640 1 -- 172.21.15.39:0/2222342522 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcb4003ce0 con 0x7fbcc81028d0 2024-03-18T16:09:56.189 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.186+0000 7fbcce200640 1 -- 172.21.15.39:0/2222342522 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbcc81a2060 con 0x7fbcc81028d0 2024-03-18T16:09:56.189 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.187+0000 7fbcce200640 1 -- 172.21.15.39:0/2222342522 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbcc81a2230 con 0x7fbcc81028d0 2024-03-18T16:09:56.189 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.188+0000 7fbcc5600640 1 -- 172.21.15.39:0/2222342522 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fbcb4046020 con 0x7fbcc81028d0 2024-03-18T16:09:56.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.191+0000 7fbcc5600640 1 --2- 172.21.15.39:0/2222342522 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fbcb00a2880 0x7fbcb00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:56.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.191+0000 7fbcc5600640 1 -- 172.21.15.39:0/2222342522 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fbcb40f0f30 con 0x7fbcc81028d0 2024-03-18T16:09:56.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.191+0000 7fbccce00640 1 --2- 172.21.15.39:0/2222342522 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fbcb00a2880 0x7fbcb00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:56.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.192+0000 7fbccce00640 1 --2- 172.21.15.39:0/2222342522 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fbcb00a2880 0x7fbcb00a4d40 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fbcc819b0f0 tx=0x7fbcbc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:56.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.194+0000 7fbcc5600640 1 -- 172.21.15.39:0/2222342522 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fbcb404d020 con 0x7fbcc81028d0 2024-03-18T16:09:56.398 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.397+0000 7fbcce200640 1 -- 172.21.15.39:0/2222342522 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fbcc81a2400 con 0x7fbcc81028d0 2024-03-18T16:09:56.398 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.397+0000 7fbcc5600640 1 -- 172.21.15.39:0/2222342522 <== mon.2 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7fbcb40bdbe0 con 0x7fbcc81028d0 2024-03-18T16:09:56.399 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2024-03-18T16:09:56.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.400+0000 7fbcce200640 1 -- 172.21.15.39:0/2222342522 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fbcb00a2880 msgr2=0x7fbcb00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:56.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.400+0000 7fbcce200640 1 --2- 172.21.15.39:0/2222342522 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fbcb00a2880 0x7fbcb00a4d40 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fbcc819b0f0 tx=0x7fbcbc005f90 comp rx=0 tx=0).stop 2024-03-18T16:09:56.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.400+0000 7fbcce200640 1 -- 172.21.15.39:0/2222342522 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbcc81028d0 msgr2=0x7fbcc819a160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:56.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.400+0000 7fbcce200640 1 --2- 172.21.15.39:0/2222342522 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbcc81028d0 0x7fbcc819a160 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fbcb4005ab0 tx=0x7fbcb402c6c0 comp rx=0 tx=0).stop 2024-03-18T16:09:56.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.400+0000 7fbcce200640 1 -- 172.21.15.39:0/2222342522 shutdown_connections 2024-03-18T16:09:56.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.400+0000 7fbcce200640 1 --2- 172.21.15.39:0/2222342522 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbcc81028d0 0x7fbcc819a160 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:56.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.400+0000 7fbcce200640 1 --2- 172.21.15.39:0/2222342522 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fbcb00a2880 0x7fbcb00a4d40 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:56.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.400+0000 7fbcce200640 1 --2- 172.21.15.39:0/2222342522 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbcc8103290 0x7fbcc81a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:56.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.400+0000 7fbcce200640 1 --2- 172.21.15.39:0/2222342522 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbcc81016e0 0x7fbcc8199c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:56.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.400+0000 7fbcce200640 1 -- 172.21.15.39:0/2222342522 >> 172.21.15.39:0/2222342522 conn(0x7fbcc80fcc50 msgr2=0x7fbcc8073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:56.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.400+0000 7fbcce200640 1 -- 172.21.15.39:0/2222342522 shutdown_connections 2024-03-18T16:09:56.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.400+0000 7fbcce200640 1 -- 172.21.15.39:0/2222342522 wait complete. 2024-03-18T16:09:56.422 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T16:09:56.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.637+0000 7f291a800640 1 -- 172.21.15.39:0/3641015979 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2914100110 msgr2=0x7f2914100530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:56.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.637+0000 7f291a800640 1 --2- 172.21.15.39:0/3641015979 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2914100110 0x7f2914100530 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f2904009900 tx=0x7f290402bc40 comp rx=0 tx=0).stop 2024-03-18T16:09:56.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.637+0000 7f291a800640 1 -- 172.21.15.39:0/3641015979 shutdown_connections 2024-03-18T16:09:56.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.637+0000 7f291a800640 1 --2- 172.21.15.39:0/3641015979 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2914100a70 0x7f2914100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:56.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.637+0000 7f291a800640 1 --2- 172.21.15.39:0/3641015979 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2914101410 0x7f2914101890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:56.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.637+0000 7f291a800640 1 --2- 172.21.15.39:0/3641015979 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2914100110 0x7f2914100530 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:56.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.637+0000 7f291a800640 1 -- 172.21.15.39:0/3641015979 >> 172.21.15.39:0/3641015979 conn(0x7f29140fbd10 msgr2=0x7f29140fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:56.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.637+0000 7f291a800640 1 -- 172.21.15.39:0/3641015979 shutdown_connections 2024-03-18T16:09:56.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.637+0000 7f291a800640 1 -- 172.21.15.39:0/3641015979 wait complete. 2024-03-18T16:09:56.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.639+0000 7f291a800640 1 Processor -- start 2024-03-18T16:09:56.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.639+0000 7f291a800640 1 -- start start 2024-03-18T16:09:56.641 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.639+0000 7f291a800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2914100110 0x7f2914199cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:56.641 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.639+0000 7f291a800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2914100a70 0x7f291419a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:56.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.639+0000 7f291a800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2914101410 0x7f29141a1260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:56.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.639+0000 7f291a800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f291410f5d0 con 0x7f2914101410 2024-03-18T16:09:56.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.639+0000 7f291a800640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f291410f450 con 0x7f2914100110 2024-03-18T16:09:56.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.639+0000 7f291a800640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f291410f750 con 0x7f2914100a70 2024-03-18T16:09:56.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.639+0000 7f2913400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2914100110 0x7f2914199cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:56.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.640+0000 7f2913400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2914100110 0x7f2914199cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:57160/0 (socket says 172.21.15.39:57160) 2024-03-18T16:09:56.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.640+0000 7f2913400640 1 -- 172.21.15.39:0/1947967448 learned_addr learned my addr 172.21.15.39:0/1947967448 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:56.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.640+0000 7f2912a00640 1 --2- 172.21.15.39:0/1947967448 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2914100a70 0x7f291419a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:56.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.640+0000 7f2913e00640 1 --2- 172.21.15.39:0/1947967448 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2914101410 0x7f29141a1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:56.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.640+0000 7f2913400640 1 -- 172.21.15.39:0/1947967448 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2914100a70 msgr2=0x7f291419a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:56.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.640+0000 7f2913400640 1 --2- 172.21.15.39:0/1947967448 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2914100a70 0x7f291419a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:56.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.640+0000 7f2913400640 1 -- 172.21.15.39:0/1947967448 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2914101410 msgr2=0x7f29141a1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:56.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.640+0000 7f2913400640 1 --2- 172.21.15.39:0/1947967448 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2914101410 0x7f29141a1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:56.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.640+0000 7f2913400640 1 -- 172.21.15.39:0/1947967448 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29141a1900 con 0x7f2914100110 2024-03-18T16:09:56.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.640+0000 7f2913e00640 1 --2- 172.21.15.39:0/1947967448 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2914101410 0x7f29141a1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:09:56.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.640+0000 7f2913400640 1 --2- 172.21.15.39:0/1947967448 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2914100110 0x7f2914199cf0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f290402c8f0 tx=0x7f290402c690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:56.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.641+0000 7f290be00640 1 -- 172.21.15.39:0/1947967448 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2904041070 con 0x7f2914100110 2024-03-18T16:09:56.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.641+0000 7f290be00640 1 -- 172.21.15.39:0/1947967448 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2904004450 con 0x7f2914100110 2024-03-18T16:09:56.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.641+0000 7f290be00640 1 -- 172.21.15.39:0/1947967448 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2904034440 con 0x7f2914100110 2024-03-18T16:09:56.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.641+0000 7f291a800640 1 -- 172.21.15.39:0/1947967448 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29141a1b90 con 0x7f2914100110 2024-03-18T16:09:56.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.641+0000 7f291a800640 1 -- 172.21.15.39:0/1947967448 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29141a2090 con 0x7f2914100110 2024-03-18T16:09:56.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.642+0000 7f291a800640 1 -- 172.21.15.39:0/1947967448 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f291410ae60 con 0x7f2914100110 2024-03-18T16:09:56.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.645+0000 7f290be00640 1 -- 172.21.15.39:0/1947967448 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f2904044070 con 0x7f2914100110 2024-03-18T16:09:56.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.647+0000 7f290be00640 1 --2- 172.21.15.39:0/1947967448 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f28f80a2880 0x7f28f80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:56.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.647+0000 7f290be00640 1 -- 172.21.15.39:0/1947967448 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f29040f0d20 con 0x7f2914100110 2024-03-18T16:09:56.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.647+0000 7f2912a00640 1 --2- 172.21.15.39:0/1947967448 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f28f80a2880 0x7f28f80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:56.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.648+0000 7f290be00640 1 -- 172.21.15.39:0/1947967448 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f29040bda30 con 0x7f2914100110 2024-03-18T16:09:56.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.648+0000 7f2912a00640 1 --2- 172.21.15.39:0/1947967448 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f28f80a2880 0x7f28f80a4d40 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f2900009bd0 tx=0x7f2900005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:56.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:56 smithi196 ceph-mon[36307]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-03-18T16:09:56.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:56 smithi196 ceph-mon[36307]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T16:09:56.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:56 smithi196 ceph-mon[36307]: pgmap v418: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 341 B/s wr, 10 op/s 2024-03-18T16:09:56.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:56 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/1016345133' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:09:56.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:56 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/2222342522' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T16:09:56.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:56 smithi194 ceph-mon[36200]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-03-18T16:09:56.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:56 smithi194 ceph-mon[36200]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T16:09:56.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:56 smithi194 ceph-mon[36200]: pgmap v418: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 341 B/s wr, 10 op/s 2024-03-18T16:09:56.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:56 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/1016345133' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:09:56.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:56 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/2222342522' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T16:09:56.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:56 smithi039 ceph-mon[31721]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-03-18T16:09:56.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:56 smithi039 ceph-mon[31721]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T16:09:56.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:56 smithi039 ceph-mon[31721]: pgmap v418: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 341 B/s wr, 10 op/s 2024-03-18T16:09:56.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:56 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/1016345133' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:09:56.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:56 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/2222342522' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T16:09:56.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.877+0000 7f291a800640 1 -- 172.21.15.39:0/1947967448 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f291402d440 con 0x7f2914100110 2024-03-18T16:09:56.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.878+0000 7f290be00640 1 -- 172.21.15.39:0/1947967448 <== mon.2 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7f29040bd330 con 0x7f2914100110 2024-03-18T16:09:56.879 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 32 2024-03-18T16:09:56.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.880+0000 7f291a800640 1 -- 172.21.15.39:0/1947967448 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f28f80a2880 msgr2=0x7f28f80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:56.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.880+0000 7f291a800640 1 --2- 172.21.15.39:0/1947967448 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f28f80a2880 0x7f28f80a4d40 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f2900009bd0 tx=0x7f2900005f90 comp rx=0 tx=0).stop 2024-03-18T16:09:56.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.880+0000 7f291a800640 1 -- 172.21.15.39:0/1947967448 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2914100110 msgr2=0x7f2914199cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:56.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.880+0000 7f291a800640 1 --2- 172.21.15.39:0/1947967448 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2914100110 0x7f2914199cf0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f290402c8f0 tx=0x7f290402c690 comp rx=0 tx=0).stop 2024-03-18T16:09:56.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.881+0000 7f291a800640 1 -- 172.21.15.39:0/1947967448 shutdown_connections 2024-03-18T16:09:56.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.881+0000 7f291a800640 1 --2- 172.21.15.39:0/1947967448 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f28f80a2880 0x7f28f80a4d40 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:56.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.881+0000 7f291a800640 1 --2- 172.21.15.39:0/1947967448 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2914101410 0x7f29141a1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:56.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.881+0000 7f291a800640 1 --2- 172.21.15.39:0/1947967448 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2914100a70 0x7f291419a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:56.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.881+0000 7f291a800640 1 --2- 172.21.15.39:0/1947967448 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2914100110 0x7f2914199cf0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:56.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.881+0000 7f291a800640 1 -- 172.21.15.39:0/1947967448 >> 172.21.15.39:0/1947967448 conn(0x7f29140fbd10 msgr2=0x7f29140fd810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:56.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.881+0000 7f291a800640 1 -- 172.21.15.39:0/1947967448 shutdown_connections 2024-03-18T16:09:56.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:56.881+0000 7f291a800640 1 -- 172.21.15.39:0/1947967448 wait complete. 2024-03-18T16:09:56.900 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T16:09:57.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.111+0000 7fe1d3e00640 1 -- 172.21.15.39:0/3410970194 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe1d4104ad0 msgr2=0x7fe1d4104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:57.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.111+0000 7fe1d3e00640 1 --2- 172.21.15.39:0/3410970194 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe1d4104ad0 0x7fe1d4104f50 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fe1bc009860 tx=0x7fe1bc02bb90 comp rx=0 tx=0).stop 2024-03-18T16:09:57.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.111+0000 7fe1d3e00640 1 -- 172.21.15.39:0/3410970194 shutdown_connections 2024-03-18T16:09:57.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.111+0000 7fe1d3e00640 1 --2- 172.21.15.39:0/3410970194 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe1d4104ad0 0x7fe1d4104f50 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:57.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.111+0000 7fe1d3e00640 1 --2- 172.21.15.39:0/3410970194 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe1d4105490 0x7fe1d410bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:57.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.111+0000 7fe1d3e00640 1 --2- 172.21.15.39:0/3410970194 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe1d41038e0 0x7fe1d4103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:57.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.111+0000 7fe1d3e00640 1 -- 172.21.15.39:0/3410970194 >> 172.21.15.39:0/3410970194 conn(0x7fe1d40fee50 msgr2=0x7fe1d4101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:57.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.112+0000 7fe1d3e00640 1 -- 172.21.15.39:0/3410970194 shutdown_connections 2024-03-18T16:09:57.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.112+0000 7fe1d3e00640 1 -- 172.21.15.39:0/3410970194 wait complete. 2024-03-18T16:09:57.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.113+0000 7fe1d3e00640 1 Processor -- start 2024-03-18T16:09:57.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.113+0000 7fe1d3e00640 1 -- start start 2024-03-18T16:09:57.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.113+0000 7fe1d3e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe1d41038e0 0x7fe1d4199c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:57.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.113+0000 7fe1d3e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe1d4104ad0 0x7fe1d419a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:57.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.113+0000 7fe1d3e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe1d4105490 0x7fe1d41a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:57.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.113+0000 7fe1d3e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fe1d410f540 con 0x7fe1d41038e0 2024-03-18T16:09:57.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.113+0000 7fe1d3e00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fe1d410f3c0 con 0x7fe1d4104ad0 2024-03-18T16:09:57.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.113+0000 7fe1d3e00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fe1d410f6c0 con 0x7fe1d4105490 2024-03-18T16:09:57.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.113+0000 7fe1d3400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe1d4105490 0x7fe1d41a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:57.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.113+0000 7fe1d3400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe1d4105490 0x7fe1d41a11a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:52100/0 (socket says 172.21.15.39:52100) 2024-03-18T16:09:57.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.113+0000 7fe1d3400640 1 -- 172.21.15.39:0/2889095305 learned_addr learned my addr 172.21.15.39:0/2889095305 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:57.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.114+0000 7fe1ca000640 1 --2- 172.21.15.39:0/2889095305 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe1d4104ad0 0x7fe1d419a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:57.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.114+0000 7fe1d2a00640 1 --2- 172.21.15.39:0/2889095305 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe1d41038e0 0x7fe1d4199c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:57.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.114+0000 7fe1d3400640 1 -- 172.21.15.39:0/2889095305 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe1d4104ad0 msgr2=0x7fe1d419a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:57.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.114+0000 7fe1d3400640 1 --2- 172.21.15.39:0/2889095305 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe1d4104ad0 0x7fe1d419a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:57.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.114+0000 7fe1d3400640 1 -- 172.21.15.39:0/2889095305 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe1d41038e0 msgr2=0x7fe1d4199c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:57.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.114+0000 7fe1d3400640 1 --2- 172.21.15.39:0/2889095305 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe1d41038e0 0x7fe1d4199c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:57.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.114+0000 7fe1d3400640 1 -- 172.21.15.39:0/2889095305 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1d41a1840 con 0x7fe1d4105490 2024-03-18T16:09:57.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.114+0000 7fe1d2a00640 1 --2- 172.21.15.39:0/2889095305 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe1d41038e0 0x7fe1d4199c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:09:57.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.114+0000 7fe1d3400640 1 --2- 172.21.15.39:0/2889095305 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe1d4105490 0x7fe1d41a11a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fe1cc00e970 tx=0x7fe1cc00ed70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:57.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.114+0000 7fe1d8a00640 1 -- 172.21.15.39:0/2889095305 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1cc00ccc0 con 0x7fe1d4105490 2024-03-18T16:09:57.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.114+0000 7fe1d8a00640 1 -- 172.21.15.39:0/2889095305 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe1cc00ce60 con 0x7fe1d4105490 2024-03-18T16:09:57.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.114+0000 7fe1d8a00640 1 -- 172.21.15.39:0/2889095305 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1cc010490 con 0x7fe1d4105490 2024-03-18T16:09:57.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.114+0000 7fe1d3e00640 1 -- 172.21.15.39:0/2889095305 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1d41a1b30 con 0x7fe1d4105490 2024-03-18T16:09:57.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.115+0000 7fe1d3e00640 1 -- 172.21.15.39:0/2889095305 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1d41a2030 con 0x7fe1d4105490 2024-03-18T16:09:57.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.116+0000 7fe1d3e00640 1 -- 172.21.15.39:0/2889095305 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1d410ac00 con 0x7fe1d4105490 2024-03-18T16:09:57.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.118+0000 7fe1d8a00640 1 -- 172.21.15.39:0/2889095305 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fe1cc005020 con 0x7fe1d4105490 2024-03-18T16:09:57.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.119+0000 7fe1d8a00640 1 --2- 172.21.15.39:0/2889095305 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fe1b80a2800 0x7fe1b80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:57.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.119+0000 7fe1d8a00640 1 -- 172.21.15.39:0/2889095305 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fe1cc0cf420 con 0x7fe1d4105490 2024-03-18T16:09:57.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.119+0000 7fe1d2a00640 1 --2- 172.21.15.39:0/2889095305 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fe1b80a2800 0x7fe1b80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:57.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.120+0000 7fe1d2a00640 1 --2- 172.21.15.39:0/2889095305 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fe1b80a2800 0x7fe1b80a4cc0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fe1d419b100 tx=0x7fe1c40095c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:57.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.121+0000 7fe1d8a00640 1 -- 172.21.15.39:0/2889095305 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe1cc09c0d0 con 0x7fe1d4105490 2024-03-18T16:09:57.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.323+0000 7fe1d3e00640 1 -- 172.21.15.39:0/2889095305 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fe1d402d360 con 0x7fe1d4105490 2024-03-18T16:09:57.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.324+0000 7fe1d8a00640 1 -- 172.21.15.39:0/2889095305 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7fe1cc09b9d0 con 0x7fe1d4105490 2024-03-18T16:09:57.325 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 64 2024-03-18T16:09:57.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.326+0000 7fe1d3e00640 1 -- 172.21.15.39:0/2889095305 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fe1b80a2800 msgr2=0x7fe1b80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:57.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.326+0000 7fe1d3e00640 1 --2- 172.21.15.39:0/2889095305 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fe1b80a2800 0x7fe1b80a4cc0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fe1d419b100 tx=0x7fe1c40095c0 comp rx=0 tx=0).stop 2024-03-18T16:09:57.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.326+0000 7fe1d3e00640 1 -- 172.21.15.39:0/2889095305 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe1d4105490 msgr2=0x7fe1d41a11a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:57.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.326+0000 7fe1d3e00640 1 --2- 172.21.15.39:0/2889095305 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe1d4105490 0x7fe1d41a11a0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fe1cc00e970 tx=0x7fe1cc00ed70 comp rx=0 tx=0).stop 2024-03-18T16:09:57.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.326+0000 7fe1d3e00640 1 -- 172.21.15.39:0/2889095305 shutdown_connections 2024-03-18T16:09:57.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.326+0000 7fe1d3e00640 1 --2- 172.21.15.39:0/2889095305 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe1d4104ad0 0x7fe1d419a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:57.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.326+0000 7fe1d3e00640 1 --2- 172.21.15.39:0/2889095305 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fe1b80a2800 0x7fe1b80a4cc0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:57.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.326+0000 7fe1d3e00640 1 --2- 172.21.15.39:0/2889095305 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe1d4105490 0x7fe1d41a11a0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:57.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.326+0000 7fe1d3e00640 1 --2- 172.21.15.39:0/2889095305 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe1d41038e0 0x7fe1d4199c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:57.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.326+0000 7fe1d3e00640 1 -- 172.21.15.39:0/2889095305 >> 172.21.15.39:0/2889095305 conn(0x7fe1d40fee50 msgr2=0x7fe1d4107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:57.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.327+0000 7fe1d3e00640 1 -- 172.21.15.39:0/2889095305 shutdown_connections 2024-03-18T16:09:57.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.327+0000 7fe1d3e00640 1 -- 172.21.15.39:0/2889095305 wait complete. 2024-03-18T16:09:57.348 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T16:09:57.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.562+0000 7f7fa9200640 1 -- 172.21.15.39:0/2889176502 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7fa40757e0 msgr2=0x7f7fa4073c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:57.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.562+0000 7f7fa9200640 1 --2- 172.21.15.39:0/2889176502 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7fa40757e0 0x7f7fa4073c30 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f7f94009900 tx=0x7f7f9402bc30 comp rx=0 tx=0).stop 2024-03-18T16:09:57.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.562+0000 7f7fa9200640 1 -- 172.21.15.39:0/2889176502 shutdown_connections 2024-03-18T16:09:57.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.562+0000 7f7fa9200640 1 --2- 172.21.15.39:0/2889176502 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7fa4109590 0x7f7fa410ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:57.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.562+0000 7f7fa9200640 1 --2- 172.21.15.39:0/2889176502 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7fa4074200 0x7f7fa4074680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:57.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.562+0000 7f7fa9200640 1 --2- 172.21.15.39:0/2889176502 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7fa40757e0 0x7f7fa4073c30 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:57.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.562+0000 7f7fa9200640 1 -- 172.21.15.39:0/2889176502 >> 172.21.15.39:0/2889176502 conn(0x7f7fa40fcc50 msgr2=0x7f7fa40ff090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:57.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.562+0000 7f7fa9200640 1 -- 172.21.15.39:0/2889176502 shutdown_connections 2024-03-18T16:09:57.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.562+0000 7f7fa9200640 1 -- 172.21.15.39:0/2889176502 wait complete. 2024-03-18T16:09:57.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.563+0000 7f7fa9200640 1 Processor -- start 2024-03-18T16:09:57.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.563+0000 7f7fa9200640 1 -- start start 2024-03-18T16:09:57.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.563+0000 7f7fa9200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7fa4074200 0x7f7fa4199ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:57.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.563+0000 7f7fa9200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7fa40757e0 0x7f7fa419a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:57.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.563+0000 7f7fa9200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7fa4109590 0x7f7fa41a1150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:57.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.564+0000 7f7fa9200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f7fa410f570 con 0x7f7fa4074200 2024-03-18T16:09:57.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.564+0000 7f7fa9200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f7fa410f3f0 con 0x7f7fa40757e0 2024-03-18T16:09:57.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.564+0000 7f7fa9200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f7fa410f6f0 con 0x7f7fa4109590 2024-03-18T16:09:57.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.564+0000 7f7fa3400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7fa4074200 0x7f7fa4199ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:57.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.564+0000 7f7f9be00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7fa40757e0 0x7f7fa419a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:57.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.564+0000 7f7fa3e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7fa4109590 0x7f7fa41a1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:57.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.564+0000 7f7f9be00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7fa40757e0 0x7f7fa419a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:57202/0 (socket says 172.21.15.39:57202) 2024-03-18T16:09:57.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.564+0000 7f7f9be00640 1 -- 172.21.15.39:0/249056225 learned_addr learned my addr 172.21.15.39:0/249056225 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:57.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.564+0000 7f7fa3400640 1 -- 172.21.15.39:0/249056225 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7fa4109590 msgr2=0x7f7fa41a1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:57.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.564+0000 7f7fa3400640 1 --2- 172.21.15.39:0/249056225 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7fa4109590 0x7f7fa41a1150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:57.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.564+0000 7f7fa3400640 1 -- 172.21.15.39:0/249056225 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7fa40757e0 msgr2=0x7f7fa419a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:57.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.564+0000 7f7fa3400640 1 --2- 172.21.15.39:0/249056225 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7fa40757e0 0x7f7fa419a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:57.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.564+0000 7f7fa3400640 1 -- 172.21.15.39:0/249056225 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fa41a17c0 con 0x7f7fa4074200 2024-03-18T16:09:57.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.564+0000 7f7fa3400640 1 --2- 172.21.15.39:0/249056225 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7fa4074200 0x7f7fa4199ba0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f7f94036040 tx=0x7f7f9402c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:57.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.565+0000 7f7fa0c00640 1 -- 172.21.15.39:0/249056225 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f94043020 con 0x7f7fa4074200 2024-03-18T16:09:57.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.565+0000 7f7fa0c00640 1 -- 172.21.15.39:0/249056225 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f940054e0 con 0x7f7fa4074200 2024-03-18T16:09:57.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.565+0000 7f7fa0c00640 1 -- 172.21.15.39:0/249056225 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f94003c30 con 0x7f7fa4074200 2024-03-18T16:09:57.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.565+0000 7f7fa9200640 1 -- 172.21.15.39:0/249056225 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7fa41a1a50 con 0x7f7fa4074200 2024-03-18T16:09:57.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.565+0000 7f7fa9200640 1 -- 172.21.15.39:0/249056225 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7fa41a1fc0 con 0x7f7fa4074200 2024-03-18T16:09:57.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.566+0000 7f7fa9200640 1 -- 172.21.15.39:0/249056225 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7fa4108310 con 0x7f7fa4074200 2024-03-18T16:09:57.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.570+0000 7f7fa0c00640 1 -- 172.21.15.39:0/249056225 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f7f94046020 con 0x7f7fa4074200 2024-03-18T16:09:57.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.571+0000 7f7fa0c00640 1 --2- 172.21.15.39:0/249056225 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f7f880a2880 0x7f7f880a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:57.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.571+0000 7f7fa0c00640 1 -- 172.21.15.39:0/249056225 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f7f940f1430 con 0x7f7fa4074200 2024-03-18T16:09:57.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.572+0000 7f7f9be00640 1 --2- 172.21.15.39:0/249056225 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f7f880a2880 0x7f7f880a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:57.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.572+0000 7f7f9be00640 1 --2- 172.21.15.39:0/249056225 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f7f880a2880 0x7f7f880a4d40 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f7fa419b0b0 tx=0x7f7f8c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:57.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.575+0000 7f7fa0c00640 1 -- 172.21.15.39:0/249056225 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7f9404d020 con 0x7f7fa4074200 2024-03-18T16:09:57.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:57 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/1947967448' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T16:09:57.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:57 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/2889095305' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T16:09:57.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:57 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/1947967448' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T16:09:57.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:57 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/2889095305' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T16:09:57.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.785+0000 7f7fa9200640 1 -- 172.21.15.39:0/249056225 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f7fa402d340 con 0x7f7fa4074200 2024-03-18T16:09:57.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.786+0000 7f7fa0c00640 1 -- 172.21.15.39:0/249056225 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7f7f940be0e0 con 0x7f7fa4074200 2024-03-18T16:09:57.787 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 32 2024-03-18T16:09:57.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.788+0000 7f7fa9200640 1 -- 172.21.15.39:0/249056225 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f7f880a2880 msgr2=0x7f7f880a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:57.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.788+0000 7f7fa9200640 1 --2- 172.21.15.39:0/249056225 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f7f880a2880 0x7f7f880a4d40 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f7fa419b0b0 tx=0x7f7f8c005f90 comp rx=0 tx=0).stop 2024-03-18T16:09:57.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.788+0000 7f7fa9200640 1 -- 172.21.15.39:0/249056225 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7fa4074200 msgr2=0x7f7fa4199ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:57.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.788+0000 7f7fa9200640 1 --2- 172.21.15.39:0/249056225 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7fa4074200 0x7f7fa4199ba0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f7f94036040 tx=0x7f7f9402c650 comp rx=0 tx=0).stop 2024-03-18T16:09:57.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.788+0000 7f7fa9200640 1 -- 172.21.15.39:0/249056225 shutdown_connections 2024-03-18T16:09:57.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.788+0000 7f7fa9200640 1 --2- 172.21.15.39:0/249056225 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7fa40757e0 0x7f7fa419a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:57.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.788+0000 7f7fa9200640 1 --2- 172.21.15.39:0/249056225 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f7f880a2880 0x7f7f880a4d40 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:57.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.788+0000 7f7fa9200640 1 --2- 172.21.15.39:0/249056225 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7fa4109590 0x7f7fa41a1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:57.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.788+0000 7f7fa9200640 1 --2- 172.21.15.39:0/249056225 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7fa4074200 0x7f7fa4199ba0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:57.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.788+0000 7f7fa9200640 1 -- 172.21.15.39:0/249056225 >> 172.21.15.39:0/249056225 conn(0x7f7fa40fcc50 msgr2=0x7f7fa41054e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:57.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.788+0000 7f7fa9200640 1 -- 172.21.15.39:0/249056225 shutdown_connections 2024-03-18T16:09:57.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:57.788+0000 7f7fa9200640 1 -- 172.21.15.39:0/249056225 wait complete. 2024-03-18T16:09:57.806 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:57 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/1947967448' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T16:09:57.806 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:57 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/2889095305' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T16:09:57.808 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T16:09:58.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.007+0000 7f25a8c00640 1 -- 172.21.15.39:0/3706137491 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f25a41007d0 msgr2=0x7f25a4100c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:58.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.007+0000 7f25a8c00640 1 --2- 172.21.15.39:0/3706137491 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f25a41007d0 0x7f25a4100c30 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f258c009900 tx=0x7f258c02bc40 comp rx=0 tx=0).stop 2024-03-18T16:09:58.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.007+0000 7f25a8c00640 1 -- 172.21.15.39:0/3706137491 shutdown_connections 2024-03-18T16:09:58.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.007+0000 7f25a8c00640 1 --2- 172.21.15.39:0/3706137491 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f25a41007d0 0x7f25a4100c30 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:58.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.007+0000 7f25a8c00640 1 --2- 172.21.15.39:0/3706137491 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f25a4101170 0x7f25a41015f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:58.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.007+0000 7f25a8c00640 1 --2- 172.21.15.39:0/3706137491 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f25a40ffe70 0x7f25a4100290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:58.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.007+0000 7f25a8c00640 1 -- 172.21.15.39:0/3706137491 >> 172.21.15.39:0/3706137491 conn(0x7f25a40fba70 msgr2=0x7f25a40fdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:58.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.007+0000 7f25a8c00640 1 -- 172.21.15.39:0/3706137491 shutdown_connections 2024-03-18T16:09:58.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.007+0000 7f25a8c00640 1 -- 172.21.15.39:0/3706137491 wait complete. 2024-03-18T16:09:58.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.008+0000 7f25a8c00640 1 Processor -- start 2024-03-18T16:09:58.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.008+0000 7f25a8c00640 1 -- start start 2024-03-18T16:09:58.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.009+0000 7f25a8c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f25a40ffe70 0x7f25a4199b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:58.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.009+0000 7f25a8c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f25a41007d0 0x7f25a419a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:58.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.009+0000 7f25a8c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f25a4101170 0x7f25a41a10b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:58.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.009+0000 7f25a8c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f25a410f460 con 0x7f25a4101170 2024-03-18T16:09:58.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.009+0000 7f25a8c00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f25a410f2e0 con 0x7f25a41007d0 2024-03-18T16:09:58.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.009+0000 7f25a8c00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f25a410f5e0 con 0x7f25a40ffe70 2024-03-18T16:09:58.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.009+0000 7f259aa00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f25a41007d0 0x7f25a419a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:58.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.009+0000 7f25a3400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f25a40ffe70 0x7f25a4199b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:58.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.009+0000 7f25a3e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f25a4101170 0x7f25a41a10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:58.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.009+0000 7f259aa00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f25a41007d0 0x7f25a419a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:57216/0 (socket says 172.21.15.39:57216) 2024-03-18T16:09:58.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.009+0000 7f259aa00640 1 -- 172.21.15.39:0/1496602417 learned_addr learned my addr 172.21.15.39:0/1496602417 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:58.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.009+0000 7f259aa00640 1 -- 172.21.15.39:0/1496602417 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f25a40ffe70 msgr2=0x7f25a4199b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:58.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.009+0000 7f259aa00640 1 --2- 172.21.15.39:0/1496602417 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f25a40ffe70 0x7f25a4199b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:58.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.009+0000 7f259aa00640 1 -- 172.21.15.39:0/1496602417 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f25a4101170 msgr2=0x7f25a41a10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:58.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.009+0000 7f259aa00640 1 --2- 172.21.15.39:0/1496602417 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f25a4101170 0x7f25a41a10b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:58.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.009+0000 7f259aa00640 1 -- 172.21.15.39:0/1496602417 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25a41a1720 con 0x7f25a41007d0 2024-03-18T16:09:58.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.010+0000 7f259aa00640 1 --2- 172.21.15.39:0/1496602417 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f25a41007d0 0x7f25a419a090 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f258c009230 tx=0x7f258c02c660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:58.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.010+0000 7f25a0c00640 1 -- 172.21.15.39:0/1496602417 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f258c041070 con 0x7f25a41007d0 2024-03-18T16:09:58.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.010+0000 7f25a0c00640 1 -- 172.21.15.39:0/1496602417 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f258c004450 con 0x7f25a41007d0 2024-03-18T16:09:58.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.010+0000 7f25a8c00640 1 -- 172.21.15.39:0/1496602417 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25a41a19b0 con 0x7f25a41007d0 2024-03-18T16:09:58.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.010+0000 7f25a0c00640 1 -- 172.21.15.39:0/1496602417 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f258c003e00 con 0x7f25a41007d0 2024-03-18T16:09:58.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.010+0000 7f25a8c00640 1 -- 172.21.15.39:0/1496602417 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25a41a1ea0 con 0x7f25a41007d0 2024-03-18T16:09:58.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.011+0000 7f25a8c00640 1 -- 172.21.15.39:0/1496602417 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25a410abc0 con 0x7f25a41007d0 2024-03-18T16:09:58.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.013+0000 7f25a0c00640 1 -- 172.21.15.39:0/1496602417 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f258c044070 con 0x7f25a41007d0 2024-03-18T16:09:58.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.015+0000 7f25a0c00640 1 --2- 172.21.15.39:0/1496602417 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f25880a2800 0x7f25880a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:58.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.015+0000 7f25a0c00640 1 -- 172.21.15.39:0/1496602417 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f258c0f0a10 con 0x7f25a41007d0 2024-03-18T16:09:58.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.015+0000 7f25a3400640 1 --2- 172.21.15.39:0/1496602417 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f25880a2800 0x7f25880a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:58.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.016+0000 7f25a3400640 1 --2- 172.21.15.39:0/1496602417 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f25880a2800 0x7f25880a4cc0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f2594009bd0 tx=0x7f2594005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:58.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.023+0000 7f25a0c00640 1 -- 172.21.15.39:0/1496602417 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f258c0bd070 con 0x7f25a41007d0 2024-03-18T16:09:58.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.233+0000 7f25a8c00640 1 -- 172.21.15.39:0/1496602417 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f25a40623e0 con 0x7f25a41007d0 2024-03-18T16:09:58.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.235+0000 7f25a0c00640 1 -- 172.21.15.39:0/1496602417 <== mon.2 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24762 (secure 0 0 0) 0x7f258c0c1110 con 0x7f25a41007d0 2024-03-18T16:09:58.237 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:09:58.237 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":93,"fsid":"1892d8fc-e540-11ee-95c9-87774f69a715","created":"2024-03-18T15:57:36.507008+0000","modified":"2024-03-18T16:08:47.874967+0000","last_up_change":"2024-03-18T16:05:45.250629+0000","last_in_change":"2024-03-18T16:05:24.841337+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T16:01:30.048247+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T16:06:42.191791+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T16:06:43.670325+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T16:06:47.523388+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"771e0558-be62-4436-94b0-6364751911c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6803","nonce":4090743635}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6805","nonce":4090743635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6809","nonce":4090743635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6807","nonce":4090743635}]},"public_addr":"172.21.15.39:6803/4090743635","cluster_addr":"172.21.15.39:6805/4090743635","heartbeat_back_addr":"172.21.15.39:6809/4090743635","heartbeat_front_addr":"172.21.15.39:6807/4090743635","state":["exists","up"]},{"osd":1,"uuid":"0430eee1-cf34-4277-b255-c7d9f07ad19a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6801","nonce":2800329805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6803","nonce":2800329805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6807","nonce":2800329805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6805","nonce":2800329805}]},"public_addr":"172.21.15.194:6801/2800329805","cluster_addr":"172.21.15.194:6803/2800329805","heartbeat_back_addr":"172.21.15.194:6807/2800329805","heartbeat_front_addr":"172.21.15.194:6805/2800329805","state":["exists","up"]},{"osd":2,"uuid":"3a718da4-2511-4ec1-b1a8-eafd6cce0df0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6801","nonce":3935874719}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6803","nonce":3935874719}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6807","nonce":3935874719}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6805","nonce":3935874719}]},"public_addr":"172.21.15.196:6801/3935874719","cluster_addr":"172.21.15.196:6803/3935874719","heartbeat_back_addr":"172.21.15.196:6807/3935874719","heartbeat_front_addr":"172.21.15.196:6805/3935874719","state":["exists","up"]},{"osd":3,"uuid":"6f9c37f9-0ab9-4038-8919-afd4b39c4165","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6811","nonce":893345939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6813","nonce":893345939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6817","nonce":893345939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6815","nonce":893345939}]},"public_addr":"172.21.15.39:6811/893345939","cluster_addr":"172.21.15.39:6813/893345939","heartbeat_back_addr":"172.21.15.39:6817/893345939","heartbeat_front_addr":"172.21.15.39:6815/893345939","state":["exists","up"]},{"osd":4,"uuid":"e795bb0e-4791-4537-acda-1a321e19bc9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6809","nonce":800423088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6811","nonce":800423088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6815","nonce":800423088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6813","nonce":800423088}]},"public_addr":"172.21.15.194:6809/800423088","cluster_addr":"172.21.15.194:6811/800423088","heartbeat_back_addr":"172.21.15.194:6815/800423088","heartbeat_front_addr":"172.21.15.194:6813/800423088","state":["exists","up"]},{"osd":5,"uuid":"026e175e-6acb-4b4c-b235-8ccee8ae1149","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6809","nonce":2746224525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6811","nonce":2746224525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6815","nonce":2746224525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6813","nonce":2746224525}]},"public_addr":"172.21.15.196:6809/2746224525","cluster_addr":"172.21.15.196:6811/2746224525","heartbeat_back_addr":"172.21.15.196:6815/2746224525","heartbeat_front_addr":"172.21.15.196:6813/2746224525","state":["exists","up"]},{"osd":6,"uuid":"c1b183ff-383f-4c5c-9075-f76bab1c0bb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6819","nonce":1213371331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6821","nonce":1213371331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6825","nonce":1213371331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6823","nonce":1213371331}]},"public_addr":"172.21.15.39:6819/1213371331","cluster_addr":"172.21.15.39:6821/1213371331","heartbeat_back_addr":"172.21.15.39:6825/1213371331","heartbeat_front_addr":"172.21.15.39:6823/1213371331","state":["exists","up"]},{"osd":7,"uuid":"612f8697-957d-4ef4-a87f-241b36f9d13d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6817","nonce":3299162444}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6819","nonce":3299162444}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6823","nonce":3299162444}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6821","nonce":3299162444}]},"public_addr":"172.21.15.194:6817/3299162444","cluster_addr":"172.21.15.194:6819/3299162444","heartbeat_back_addr":"172.21.15.194:6823/3299162444","heartbeat_front_addr":"172.21.15.194:6821/3299162444","state":["exists","up"]},{"osd":8,"uuid":"836a9107-4d4f-4594-b8dc-0301e64f59d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6817","nonce":2456143145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6819","nonce":2456143145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6823","nonce":2456143145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6821","nonce":2456143145}]},"public_addr":"172.21.15.196:6817/2456143145","cluster_addr":"172.21.15.196:6819/2456143145","heartbeat_back_addr":"172.21.15.196:6823/2456143145","heartbeat_front_addr":"172.21.15.196:6821/2456143145","state":["exists","up"]},{"osd":9,"uuid":"4f30b960-12e3-4b16-bd28-fcc988139b58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6827","nonce":1848634433}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6829","nonce":1848634433}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6833","nonce":1848634433}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6831","nonce":1848634433}]},"public_addr":"172.21.15.39:6827/1848634433","cluster_addr":"172.21.15.39:6829/1848634433","heartbeat_back_addr":"172.21.15.39:6833/1848634433","heartbeat_front_addr":"172.21.15.39:6831/1848634433","state":["exists","up"]},{"osd":10,"uuid":"620f90e8-23e6-4bc0-8269-7219b9e1aebd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6825","nonce":3725064809}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6827","nonce":3725064809}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6831","nonce":3725064809}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6829","nonce":3725064809}]},"public_addr":"172.21.15.194:6825/3725064809","cluster_addr":"172.21.15.194:6827/3725064809","heartbeat_back_addr":"172.21.15.194:6831/3725064809","heartbeat_front_addr":"172.21.15.194:6829/3725064809","state":["exists","up"]},{"osd":11,"uuid":"867a2535-8235-4ceb-a698-eb86869d6d20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6825","nonce":2501212422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6827","nonce":2501212422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6831","nonce":2501212422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6829","nonce":2501212422}]},"public_addr":"172.21.15.196:6825/2501212422","cluster_addr":"172.21.15.196:6827/2501212422","heartbeat_back_addr":"172.21.15.196:6831/2501212422","heartbeat_front_addr":"172.21.15.196:6829/2501212422","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:00:36.931566+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:01:02.614024+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:01:26.193265+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:01:58.308833+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:02:25.049546+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:02:49.748750+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:03:23.582153+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:03:50.447321+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:04:15.812129+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:04:49.998047+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:05:17.087924+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:05:43.167637+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":4,"to":7}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.a","mappings":[{"from":5,"to":8}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.39:0/3482086140":"2024-03-19T15:58:45.340294+0000","172.21.15.39:0/3844306284":"2024-03-19T15:58:45.340294+0000","172.21.15.39:6801/1810238546":"2024-03-19T15:58:45.340294+0000","172.21.15.39:6800/1810238546":"2024-03-19T15:58:45.340294+0000","172.21.15.39:0/2528397488":"2024-03-19T15:58:10.264798+0000","172.21.15.39:0/2240981254":"2024-03-19T15:58:10.264798+0000","172.21.15.39:6800/1217503189":"2024-03-19T15:58:10.264798+0000","172.21.15.39:0/3754876370":"2024-03-19T15:58:10.264798+0000","172.21.15.39:6801/1217503189":"2024-03-19T15:58:10.264798+0000","172.21.15.39:0/1874637628":"2024-03-19T15:58:45.340294+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T16:09:58.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.238+0000 7f25a8c00640 1 -- 172.21.15.39:0/1496602417 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f25880a2800 msgr2=0x7f25880a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:58.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.238+0000 7f25a8c00640 1 --2- 172.21.15.39:0/1496602417 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f25880a2800 0x7f25880a4cc0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f2594009bd0 tx=0x7f2594005f90 comp rx=0 tx=0).stop 2024-03-18T16:09:58.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.238+0000 7f25a8c00640 1 -- 172.21.15.39:0/1496602417 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f25a41007d0 msgr2=0x7f25a419a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:58.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.238+0000 7f25a8c00640 1 --2- 172.21.15.39:0/1496602417 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f25a41007d0 0x7f25a419a090 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f258c009230 tx=0x7f258c02c660 comp rx=0 tx=0).stop 2024-03-18T16:09:58.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.238+0000 7f25a8c00640 1 -- 172.21.15.39:0/1496602417 shutdown_connections 2024-03-18T16:09:58.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.238+0000 7f25a8c00640 1 --2- 172.21.15.39:0/1496602417 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f25a41007d0 0x7f25a419a090 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:58.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.238+0000 7f25a8c00640 1 --2- 172.21.15.39:0/1496602417 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f25880a2800 0x7f25880a4cc0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:58.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.238+0000 7f25a8c00640 1 --2- 172.21.15.39:0/1496602417 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f25a4101170 0x7f25a41a10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:58.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.238+0000 7f25a8c00640 1 --2- 172.21.15.39:0/1496602417 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f25a40ffe70 0x7f25a4199b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:58.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.238+0000 7f25a8c00640 1 -- 172.21.15.39:0/1496602417 >> 172.21.15.39:0/1496602417 conn(0x7f25a40fba70 msgr2=0x7f25a40fd570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:58.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.238+0000 7f25a8c00640 1 -- 172.21.15.39:0/1496602417 shutdown_connections 2024-03-18T16:09:58.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.238+0000 7f25a8c00640 1 -- 172.21.15.39:0/1496602417 wait complete. 2024-03-18T16:09:58.259 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T16:01:30.048247+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T16:06:42.191791+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T16:06:43.670325+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T16:06:47.523388+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T16:09:58.259 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T16:09:58.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.475+0000 7f3acac00640 1 -- 172.21.15.39:0/1970514592 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3ac41016e0 msgr2=0x7f3ac4101b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:58.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.475+0000 7f3acac00640 1 --2- 172.21.15.39:0/1970514592 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3ac41016e0 0x7f3ac4101b00 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f3ab8009900 tx=0x7f3ab802bc40 comp rx=0 tx=0).stop 2024-03-18T16:09:58.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.475+0000 7f3acac00640 1 -- 172.21.15.39:0/1970514592 shutdown_connections 2024-03-18T16:09:58.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.475+0000 7f3acac00640 1 --2- 172.21.15.39:0/1970514592 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3ac4103290 0x7f3ac4109b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:58.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.475+0000 7f3acac00640 1 --2- 172.21.15.39:0/1970514592 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3ac41028d0 0x7f3ac4102d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:58.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.475+0000 7f3acac00640 1 --2- 172.21.15.39:0/1970514592 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3ac41016e0 0x7f3ac4101b00 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:58.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.475+0000 7f3acac00640 1 -- 172.21.15.39:0/1970514592 >> 172.21.15.39:0/1970514592 conn(0x7f3ac40fcc50 msgr2=0x7f3ac40ff090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:58.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.475+0000 7f3acac00640 1 -- 172.21.15.39:0/1970514592 shutdown_connections 2024-03-18T16:09:58.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.475+0000 7f3acac00640 1 -- 172.21.15.39:0/1970514592 wait complete. 2024-03-18T16:09:58.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.476+0000 7f3acac00640 1 Processor -- start 2024-03-18T16:09:58.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.476+0000 7f3acac00640 1 -- start start 2024-03-18T16:09:58.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.477+0000 7f3acac00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3ac41016e0 0x7f3ac4199be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:58.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.477+0000 7f3acac00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3ac41028d0 0x7f3ac419a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:58.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.477+0000 7f3acac00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3ac4103290 0x7f3ac41a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:58.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.477+0000 7f3acac00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f3ac410d380 con 0x7f3ac41028d0 2024-03-18T16:09:58.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.477+0000 7f3acac00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f3ac410d200 con 0x7f3ac41016e0 2024-03-18T16:09:58.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.477+0000 7f3acac00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f3ac410d500 con 0x7f3ac4103290 2024-03-18T16:09:58.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.477+0000 7f3ac8e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3ac41028d0 0x7f3ac419a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:58.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.477+0000 7f3ac9800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3ac41016e0 0x7f3ac4199be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:58.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.477+0000 7f3aca200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3ac4103290 0x7f3ac41a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:58.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.477+0000 7f3ac9800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3ac41016e0 0x7f3ac4199be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:36376/0 (socket says 172.21.15.39:36376) 2024-03-18T16:09:58.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.477+0000 7f3ac9800640 1 -- 172.21.15.39:0/3579777507 learned_addr learned my addr 172.21.15.39:0/3579777507 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:58.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.477+0000 7f3aca200640 1 -- 172.21.15.39:0/3579777507 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3ac41016e0 msgr2=0x7f3ac4199be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:58.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.477+0000 7f3aca200640 1 --2- 172.21.15.39:0/3579777507 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3ac41016e0 0x7f3ac4199be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:58.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.477+0000 7f3aca200640 1 -- 172.21.15.39:0/3579777507 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3ac41028d0 msgr2=0x7f3ac419a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:58.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.477+0000 7f3aca200640 1 --2- 172.21.15.39:0/3579777507 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3ac41028d0 0x7f3ac419a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:58.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.477+0000 7f3aca200640 1 -- 172.21.15.39:0/3579777507 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ac41a1800 con 0x7f3ac4103290 2024-03-18T16:09:58.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.478+0000 7f3aca200640 1 --2- 172.21.15.39:0/3579777507 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3ac4103290 0x7f3ac41a1190 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f3abc009fd0 tx=0x7f3abc00ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:58.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.478+0000 7f3ac2000640 1 -- 172.21.15.39:0/3579777507 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3abc00ce20 con 0x7f3ac4103290 2024-03-18T16:09:58.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.478+0000 7f3ac2000640 1 -- 172.21.15.39:0/3579777507 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3abc005020 con 0x7f3ac4103290 2024-03-18T16:09:58.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.478+0000 7f3ac2000640 1 -- 172.21.15.39:0/3579777507 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3abc010720 con 0x7f3ac4103290 2024-03-18T16:09:58.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.478+0000 7f3acac00640 1 -- 172.21.15.39:0/3579777507 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ac41a1af0 con 0x7f3ac4103290 2024-03-18T16:09:58.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.478+0000 7f3acac00640 1 -- 172.21.15.39:0/3579777507 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ac41a2040 con 0x7f3ac4103290 2024-03-18T16:09:58.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.479+0000 7f3acac00640 1 -- 172.21.15.39:0/3579777507 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ac41a2270 con 0x7f3ac4103290 2024-03-18T16:09:58.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.481+0000 7f3ac2000640 1 -- 172.21.15.39:0/3579777507 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f3abc005540 con 0x7f3ac4103290 2024-03-18T16:09:58.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.483+0000 7f3ac2000640 1 --2- 172.21.15.39:0/3579777507 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3aac0a2880 0x7f3aac0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:58.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.483+0000 7f3ac2000640 1 -- 172.21.15.39:0/3579777507 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f3abc0cf430 con 0x7f3ac4103290 2024-03-18T16:09:58.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.483+0000 7f3ac9800640 1 --2- 172.21.15.39:0/3579777507 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3aac0a2880 0x7f3aac0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:58.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.484+0000 7f3ac9800640 1 --2- 172.21.15.39:0/3579777507 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3aac0a2880 0x7f3aac0a4d40 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f3ab8005ab0 tx=0x7f3ab802cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:58.487 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.486+0000 7f3ac2000640 1 -- 172.21.15.39:0/3579777507 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3abc09c140 con 0x7f3ac4103290 2024-03-18T16:09:58.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:58 smithi039 ceph-mon[31721]: pgmap v419: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-18T16:09:58.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:58 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/249056225' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T16:09:58.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:58 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/1496602417' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:09:58.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.718+0000 7f3acac00640 1 -- 172.21.15.39:0/3579777507 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f3ac41a2440 con 0x7f3ac4103290 2024-03-18T16:09:58.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.719+0000 7f3ac2000640 1 -- 172.21.15.39:0/3579777507 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7f3abc09ba40 con 0x7f3ac4103290 2024-03-18T16:09:58.720 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2024-03-18T16:09:58.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.722+0000 7f3acac00640 1 -- 172.21.15.39:0/3579777507 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3aac0a2880 msgr2=0x7f3aac0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:58.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.722+0000 7f3acac00640 1 --2- 172.21.15.39:0/3579777507 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3aac0a2880 0x7f3aac0a4d40 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f3ab8005ab0 tx=0x7f3ab802cf90 comp rx=0 tx=0).stop 2024-03-18T16:09:58.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.722+0000 7f3acac00640 1 -- 172.21.15.39:0/3579777507 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3ac4103290 msgr2=0x7f3ac41a1190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:58.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.722+0000 7f3acac00640 1 --2- 172.21.15.39:0/3579777507 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3ac4103290 0x7f3ac41a1190 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f3abc009fd0 tx=0x7f3abc00ef10 comp rx=0 tx=0).stop 2024-03-18T16:09:58.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.722+0000 7f3acac00640 1 -- 172.21.15.39:0/3579777507 shutdown_connections 2024-03-18T16:09:58.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.722+0000 7f3acac00640 1 --2- 172.21.15.39:0/3579777507 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3aac0a2880 0x7f3aac0a4d40 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:58.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.722+0000 7f3acac00640 1 --2- 172.21.15.39:0/3579777507 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3ac4103290 0x7f3ac41a1190 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:58.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.722+0000 7f3acac00640 1 --2- 172.21.15.39:0/3579777507 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3ac41028d0 0x7f3ac419a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:58.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.722+0000 7f3acac00640 1 --2- 172.21.15.39:0/3579777507 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3ac41016e0 0x7f3ac4199be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:58.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.722+0000 7f3acac00640 1 -- 172.21.15.39:0/3579777507 >> 172.21.15.39:0/3579777507 conn(0x7f3ac40fcc50 msgr2=0x7f3ac4073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:58.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.722+0000 7f3acac00640 1 -- 172.21.15.39:0/3579777507 shutdown_connections 2024-03-18T16:09:58.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.722+0000 7f3acac00640 1 -- 172.21.15.39:0/3579777507 wait complete. 2024-03-18T16:09:58.727 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:58 smithi196 ceph-mon[36307]: pgmap v419: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-18T16:09:58.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:58 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/249056225' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T16:09:58.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:58 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/1496602417' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:09:58.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:58 smithi194 ceph-mon[36200]: pgmap v419: 129 pgs: 129 active+clean; 644 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-18T16:09:58.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:58 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/249056225' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T16:09:58.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:58 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/1496602417' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:09:58.745 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T16:09:58.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.933+0000 7f3e7aa00640 1 -- 172.21.15.39:0/2018831793 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3e740fff90 msgr2=0x7f3e741003b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:58.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.933+0000 7f3e7aa00640 1 --2- 172.21.15.39:0/2018831793 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3e740fff90 0x7f3e741003b0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f3e68009860 tx=0x7f3e6802bb70 comp rx=0 tx=0).stop 2024-03-18T16:09:58.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.933+0000 7f3e7aa00640 1 -- 172.21.15.39:0/2018831793 shutdown_connections 2024-03-18T16:09:58.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.933+0000 7f3e7aa00640 1 --2- 172.21.15.39:0/2018831793 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3e741008f0 0x7f3e74100d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:58.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.933+0000 7f3e7aa00640 1 --2- 172.21.15.39:0/2018831793 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3e74101290 0x7f3e74101710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:58.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.933+0000 7f3e7aa00640 1 --2- 172.21.15.39:0/2018831793 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3e740fff90 0x7f3e741003b0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:58.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.933+0000 7f3e7aa00640 1 -- 172.21.15.39:0/2018831793 >> 172.21.15.39:0/2018831793 conn(0x7f3e740fbb50 msgr2=0x7f3e740fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:58.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.933+0000 7f3e7aa00640 1 -- 172.21.15.39:0/2018831793 shutdown_connections 2024-03-18T16:09:58.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.933+0000 7f3e7aa00640 1 -- 172.21.15.39:0/2018831793 wait complete. 2024-03-18T16:09:58.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.934+0000 7f3e7aa00640 1 Processor -- start 2024-03-18T16:09:58.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.934+0000 7f3e7aa00640 1 -- start start 2024-03-18T16:09:58.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.935+0000 7f3e7aa00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3e740fff90 0x7f3e74199b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:58.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.935+0000 7f3e7aa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3e741008f0 0x7f3e7419a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:58.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.935+0000 7f3e7aa00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3e74101290 0x7f3e741a1140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:58.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.935+0000 7f3e7aa00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f3e7410f520 con 0x7f3e741008f0 2024-03-18T16:09:58.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.935+0000 7f3e7aa00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f3e7410f3a0 con 0x7f3e74101290 2024-03-18T16:09:58.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.935+0000 7f3e7aa00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f3e7410f6a0 con 0x7f3e740fff90 2024-03-18T16:09:58.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.935+0000 7f3e78c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3e741008f0 0x7f3e7419a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:58.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.935+0000 7f3e79600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3e740fff90 0x7f3e74199b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:58.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.935+0000 7f3e79600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3e740fff90 0x7f3e74199b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:49666/0 (socket says 172.21.15.39:49666) 2024-03-18T16:09:58.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.935+0000 7f3e7a000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3e74101290 0x7f3e741a1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:58.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.935+0000 7f3e79600640 1 -- 172.21.15.39:0/253627210 learned_addr learned my addr 172.21.15.39:0/253627210 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:58.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.935+0000 7f3e79600640 1 -- 172.21.15.39:0/253627210 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3e74101290 msgr2=0x7f3e741a1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:58.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.935+0000 7f3e79600640 1 --2- 172.21.15.39:0/253627210 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3e74101290 0x7f3e741a1140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:58.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.935+0000 7f3e79600640 1 -- 172.21.15.39:0/253627210 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3e741008f0 msgr2=0x7f3e7419a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:58.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.935+0000 7f3e79600640 1 --2- 172.21.15.39:0/253627210 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3e741008f0 0x7f3e7419a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:58.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.935+0000 7f3e79600640 1 -- 172.21.15.39:0/253627210 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e741a17b0 con 0x7f3e740fff90 2024-03-18T16:09:58.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.936+0000 7f3e79600640 1 --2- 172.21.15.39:0/253627210 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3e740fff90 0x7f3e74199b90 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f3e680349e0 tx=0x7f3e68032450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:58.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.936+0000 7f3e72000640 1 -- 172.21.15.39:0/253627210 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e68041030 con 0x7f3e740fff90 2024-03-18T16:09:58.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.936+0000 7f3e72000640 1 -- 172.21.15.39:0/253627210 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e6802c510 con 0x7f3e740fff90 2024-03-18T16:09:58.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.936+0000 7f3e72000640 1 -- 172.21.15.39:0/253627210 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e68039040 con 0x7f3e740fff90 2024-03-18T16:09:58.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.936+0000 7f3e7aa00640 1 -- 172.21.15.39:0/253627210 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e741a1a40 con 0x7f3e740fff90 2024-03-18T16:09:58.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.936+0000 7f3e7aa00640 1 -- 172.21.15.39:0/253627210 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e741a1f30 con 0x7f3e740fff90 2024-03-18T16:09:58.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.938+0000 7f3e7aa00640 1 -- 172.21.15.39:0/253627210 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e7410ae20 con 0x7f3e740fff90 2024-03-18T16:09:58.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.939+0000 7f3e72000640 1 -- 172.21.15.39:0/253627210 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f3e68044070 con 0x7f3e740fff90 2024-03-18T16:09:58.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.940+0000 7f3e72000640 1 --2- 172.21.15.39:0/253627210 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3e5c0a2800 0x7f3e5c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:58.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.940+0000 7f3e72000640 1 -- 172.21.15.39:0/253627210 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f3e680f0350 con 0x7f3e740fff90 2024-03-18T16:09:58.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.940+0000 7f3e78c00640 1 --2- 172.21.15.39:0/253627210 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3e5c0a2800 0x7f3e5c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:58.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.941+0000 7f3e78c00640 1 --2- 172.21.15.39:0/253627210 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3e5c0a2800 0x7f3e5c0a4cc0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f3e7419b0a0 tx=0x7f3e60009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:58.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:58.949+0000 7f3e72000640 1 -- 172.21.15.39:0/253627210 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3e680bd000 con 0x7f3e740fff90 2024-03-18T16:09:59.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:59.212+0000 7f3e7aa00640 1 -- 172.21.15.39:0/253627210 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f3e740623e0 con 0x7f3e740fff90 2024-03-18T16:09:59.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:59.213+0000 7f3e72000640 1 -- 172.21.15.39:0/253627210 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7f3e680bc900 con 0x7f3e740fff90 2024-03-18T16:09:59.214 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 32 2024-03-18T16:09:59.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:59.216+0000 7f3e7aa00640 1 -- 172.21.15.39:0/253627210 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3e5c0a2800 msgr2=0x7f3e5c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:59.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:59.216+0000 7f3e7aa00640 1 --2- 172.21.15.39:0/253627210 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3e5c0a2800 0x7f3e5c0a4cc0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f3e7419b0a0 tx=0x7f3e60009260 comp rx=0 tx=0).stop 2024-03-18T16:09:59.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:59.216+0000 7f3e7aa00640 1 -- 172.21.15.39:0/253627210 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3e740fff90 msgr2=0x7f3e74199b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:59.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:59.216+0000 7f3e7aa00640 1 --2- 172.21.15.39:0/253627210 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3e740fff90 0x7f3e74199b90 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f3e680349e0 tx=0x7f3e68032450 comp rx=0 tx=0).stop 2024-03-18T16:09:59.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:59.216+0000 7f3e7aa00640 1 -- 172.21.15.39:0/253627210 shutdown_connections 2024-03-18T16:09:59.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:59.216+0000 7f3e7aa00640 1 --2- 172.21.15.39:0/253627210 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3e74101290 0x7f3e741a1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:59.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:59.216+0000 7f3e7aa00640 1 --2- 172.21.15.39:0/253627210 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3e5c0a2800 0x7f3e5c0a4cc0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:59.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:59.216+0000 7f3e7aa00640 1 --2- 172.21.15.39:0/253627210 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3e741008f0 0x7f3e7419a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:59.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:59.216+0000 7f3e7aa00640 1 --2- 172.21.15.39:0/253627210 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3e740fff90 0x7f3e74199b90 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:59.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:59.216+0000 7f3e7aa00640 1 -- 172.21.15.39:0/253627210 >> 172.21.15.39:0/253627210 conn(0x7f3e740fbb50 msgr2=0x7f3e740fd290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:59.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:59.216+0000 7f3e7aa00640 1 -- 172.21.15.39:0/253627210 shutdown_connections 2024-03-18T16:09:59.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:59.216+0000 7f3e7aa00640 1 -- 172.21.15.39:0/253627210 wait complete. 2024-03-18T16:09:59.237 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T16:09:59.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:59.457+0000 7f614ea00640 1 -- 172.21.15.39:0/1592111964 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6148100090 msgr2=0x7f61481004b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:59.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:59.457+0000 7f614ea00640 1 --2- 172.21.15.39:0/1592111964 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6148100090 0x7f61481004b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f6138009900 tx=0x7f613802bc40 comp rx=0 tx=0).stop 2024-03-18T16:09:59.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:59.457+0000 7f614ea00640 1 -- 172.21.15.39:0/1592111964 shutdown_connections 2024-03-18T16:09:59.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:59.457+0000 7f614ea00640 1 --2- 172.21.15.39:0/1592111964 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6148101390 0x7f6148101810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:59.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:59.457+0000 7f614ea00640 1 --2- 172.21.15.39:0/1592111964 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f61481009f0 0x7f6148100e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:59.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:59.457+0000 7f614ea00640 1 --2- 172.21.15.39:0/1592111964 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6148100090 0x7f61481004b0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:59.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:59.457+0000 7f614ea00640 1 -- 172.21.15.39:0/1592111964 >> 172.21.15.39:0/1592111964 conn(0x7f61480fbc70 msgr2=0x7f61480fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:59.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:59.457+0000 7f614ea00640 1 -- 172.21.15.39:0/1592111964 shutdown_connections 2024-03-18T16:09:59.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:59.457+0000 7f614ea00640 1 -- 172.21.15.39:0/1592111964 wait complete. 2024-03-18T16:09:59.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:59.458+0000 7f614ea00640 1 Processor -- start 2024-03-18T16:09:59.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:59.459+0000 7f614ea00640 1 -- start start 2024-03-18T16:09:59.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:59.459+0000 7f614ea00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6148100090 0x7f6148199cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:59.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:59.459+0000 7f614ea00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f61481009f0 0x7f614819a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:59.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:59.459+0000 7f614ea00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6148101390 0x7f61481a1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:59.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:59.459+0000 7f614ea00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f614810f5e0 con 0x7f61481009f0 2024-03-18T16:09:59.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:59.459+0000 7f614ea00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f614810f460 con 0x7f6148100090 2024-03-18T16:09:59.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:59.459+0000 7f614ea00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f614810f760 con 0x7f6148101390 2024-03-18T16:09:59.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:59.459+0000 7f6147e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6148100090 0x7f6148199cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:59.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:59.459+0000 7f613fe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f61481009f0 0x7f614819a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:59.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:59.459+0000 7f614ca00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6148101390 0x7f61481a1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:59.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:59.459+0000 7f6147e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6148100090 0x7f6148199cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:36418/0 (socket says 172.21.15.39:36418) 2024-03-18T16:09:59.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:59.459+0000 7f6147e00640 1 -- 172.21.15.39:0/466280127 learned_addr learned my addr 172.21.15.39:0/466280127 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:09:59.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:59.460+0000 7f6147e00640 1 -- 172.21.15.39:0/466280127 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6148101390 msgr2=0x7f61481a1250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:59.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:59.460+0000 7f6147e00640 1 --2- 172.21.15.39:0/466280127 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6148101390 0x7f61481a1250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:59.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:59.460+0000 7f6147e00640 1 -- 172.21.15.39:0/466280127 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f61481009f0 msgr2=0x7f614819a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:59.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:59.460+0000 7f6147e00640 1 --2- 172.21.15.39:0/466280127 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f61481009f0 0x7f614819a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:59.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:59.460+0000 7f6147e00640 1 -- 172.21.15.39:0/466280127 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61481a18f0 con 0x7f6148100090 2024-03-18T16:09:59.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:59.460+0000 7f6147e00640 1 --2- 172.21.15.39:0/466280127 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6148100090 0x7f6148199cb0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f613802c8f0 tx=0x7f613802c690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:59.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:59.460+0000 7f6145600640 1 -- 172.21.15.39:0/466280127 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6138041070 con 0x7f6148100090 2024-03-18T16:09:59.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:59.460+0000 7f6145600640 1 -- 172.21.15.39:0/466280127 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6138004450 con 0x7f6148100090 2024-03-18T16:09:59.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:59.460+0000 7f6145600640 1 -- 172.21.15.39:0/466280127 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6138034440 con 0x7f6148100090 2024-03-18T16:09:59.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:59.460+0000 7f614ea00640 1 -- 172.21.15.39:0/466280127 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61481a1b80 con 0x7f6148100090 2024-03-18T16:09:59.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:59.460+0000 7f614ea00640 1 -- 172.21.15.39:0/466280127 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61481a2120 con 0x7f6148100090 2024-03-18T16:09:59.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:59.462+0000 7f614ea00640 1 -- 172.21.15.39:0/466280127 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f614810afd0 con 0x7f6148100090 2024-03-18T16:09:59.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:59.463+0000 7f6145600640 1 -- 172.21.15.39:0/466280127 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f6138044070 con 0x7f6148100090 2024-03-18T16:09:59.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:59.464+0000 7f6145600640 1 --2- 172.21.15.39:0/466280127 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f612c0a2880 0x7f612c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:09:59.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:59.464+0000 7f6145600640 1 -- 172.21.15.39:0/466280127 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f61380f0890 con 0x7f6148100090 2024-03-18T16:09:59.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:59.465+0000 7f613fe00640 1 --2- 172.21.15.39:0/466280127 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f612c0a2880 0x7f612c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:09:59.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:59.465+0000 7f613fe00640 1 --2- 172.21.15.39:0/466280127 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f612c0a2880 0x7f612c0a4d40 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f614819b1b0 tx=0x7f6130005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:09:59.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:59.468+0000 7f6145600640 1 -- 172.21.15.39:0/466280127 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f61380bcf50 con 0x7f6148100090 2024-03-18T16:09:59.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:59 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/3579777507' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T16:09:59.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:09:59 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/253627210' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T16:09:59.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:59 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/3579777507' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T16:09:59.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:09:59 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/253627210' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T16:09:59.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:59 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/3579777507' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T16:09:59.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:09:59 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/253627210' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T16:09:59.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:59.780+0000 7f614ea00640 1 -- 172.21.15.39:0/466280127 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f61480623e0 con 0x7f6148100090 2024-03-18T16:09:59.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:59.780+0000 7f6145600640 1 -- 172.21.15.39:0/466280127 <== mon.2 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7f61380c0ff0 con 0x7f6148100090 2024-03-18T16:09:59.782 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 64 2024-03-18T16:09:59.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:59.783+0000 7f614ea00640 1 -- 172.21.15.39:0/466280127 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f612c0a2880 msgr2=0x7f612c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:59.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:59.783+0000 7f614ea00640 1 --2- 172.21.15.39:0/466280127 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f612c0a2880 0x7f612c0a4d40 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f614819b1b0 tx=0x7f6130005f90 comp rx=0 tx=0).stop 2024-03-18T16:09:59.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:59.783+0000 7f614ea00640 1 -- 172.21.15.39:0/466280127 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6148100090 msgr2=0x7f6148199cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:09:59.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:59.783+0000 7f614ea00640 1 --2- 172.21.15.39:0/466280127 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6148100090 0x7f6148199cb0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f613802c8f0 tx=0x7f613802c690 comp rx=0 tx=0).stop 2024-03-18T16:09:59.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:59.783+0000 7f614ea00640 1 -- 172.21.15.39:0/466280127 shutdown_connections 2024-03-18T16:09:59.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:59.783+0000 7f614ea00640 1 --2- 172.21.15.39:0/466280127 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f612c0a2880 0x7f612c0a4d40 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:59.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:59.783+0000 7f614ea00640 1 --2- 172.21.15.39:0/466280127 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6148101390 0x7f61481a1250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:59.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:59.783+0000 7f614ea00640 1 --2- 172.21.15.39:0/466280127 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f61481009f0 0x7f614819a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:59.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:59.783+0000 7f614ea00640 1 --2- 172.21.15.39:0/466280127 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6148100090 0x7f6148199cb0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:09:59.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:59.783+0000 7f614ea00640 1 -- 172.21.15.39:0/466280127 >> 172.21.15.39:0/466280127 conn(0x7f61480fbc70 msgr2=0x7f61480fd410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:09:59.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:59.783+0000 7f614ea00640 1 -- 172.21.15.39:0/466280127 shutdown_connections 2024-03-18T16:09:59.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:09:59.783+0000 7f614ea00640 1 -- 172.21.15.39:0/466280127 wait complete. 2024-03-18T16:09:59.804 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T16:10:00.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.017+0000 7fd9c4000640 1 -- 172.21.15.39:0/3633567502 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd9bc0741c0 msgr2=0x7fd9bc074620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:00.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.017+0000 7fd9c4000640 1 --2- 172.21.15.39:0/3633567502 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd9bc0741c0 0x7fd9bc074620 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fd9a8009860 tx=0x7fd9a802bc00 comp rx=0 tx=0).stop 2024-03-18T16:10:00.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.017+0000 7fd9c4000640 1 -- 172.21.15.39:0/3633567502 shutdown_connections 2024-03-18T16:10:00.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.017+0000 7fd9c4000640 1 --2- 172.21.15.39:0/3633567502 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd9bc0741c0 0x7fd9bc074620 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:00.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.017+0000 7fd9c4000640 1 --2- 172.21.15.39:0/3633567502 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd9bc1096a0 0x7fd9bc10bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:00.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.017+0000 7fd9c4000640 1 --2- 172.21.15.39:0/3633567502 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd9bc0757a0 0x7fd9bc073bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:00.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.017+0000 7fd9c4000640 1 -- 172.21.15.39:0/3633567502 >> 172.21.15.39:0/3633567502 conn(0x7fd9bc0fcd80 msgr2=0x7fd9bc0ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:00.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.017+0000 7fd9c4000640 1 -- 172.21.15.39:0/3633567502 shutdown_connections 2024-03-18T16:10:00.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.017+0000 7fd9c4000640 1 -- 172.21.15.39:0/3633567502 wait complete. 2024-03-18T16:10:00.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.018+0000 7fd9c4000640 1 Processor -- start 2024-03-18T16:10:00.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.019+0000 7fd9c4000640 1 -- start start 2024-03-18T16:10:00.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.019+0000 7fd9c4000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd9bc0741c0 0x7fd9bc199d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:00.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.019+0000 7fd9c4000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd9bc0757a0 0x7fd9bc19a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:00.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.019+0000 7fd9c4000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd9bc1096a0 0x7fd9bc1a12b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:00.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.019+0000 7fd9c4000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fd9bc10f680 con 0x7fd9bc0757a0 2024-03-18T16:10:00.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.019+0000 7fd9c4000640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fd9bc10f500 con 0x7fd9bc1096a0 2024-03-18T16:10:00.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.019+0000 7fd9c4000640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fd9bc10f800 con 0x7fd9bc0741c0 2024-03-18T16:10:00.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.019+0000 7fd9c1600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd9bc0741c0 0x7fd9bc199d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:00.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.019+0000 7fd9c1600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd9bc0741c0 0x7fd9bc199d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:49714/0 (socket says 172.21.15.39:49714) 2024-03-18T16:10:00.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.019+0000 7fd9c1600640 1 -- 172.21.15.39:0/4294254556 learned_addr learned my addr 172.21.15.39:0/4294254556 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:10:00.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.019+0000 7fd9c2000640 1 --2- 172.21.15.39:0/4294254556 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd9bc1096a0 0x7fd9bc1a12b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:00.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.019+0000 7fd9c0c00640 1 --2- 172.21.15.39:0/4294254556 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd9bc0757a0 0x7fd9bc19a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:00.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.020+0000 7fd9c1600640 1 -- 172.21.15.39:0/4294254556 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd9bc1096a0 msgr2=0x7fd9bc1a12b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:00.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.020+0000 7fd9c1600640 1 --2- 172.21.15.39:0/4294254556 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd9bc1096a0 0x7fd9bc1a12b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:00.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.020+0000 7fd9c1600640 1 -- 172.21.15.39:0/4294254556 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd9bc0757a0 msgr2=0x7fd9bc19a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:00.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.020+0000 7fd9c1600640 1 --2- 172.21.15.39:0/4294254556 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd9bc0757a0 0x7fd9bc19a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:00.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.020+0000 7fd9c1600640 1 -- 172.21.15.39:0/4294254556 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9bc1a1950 con 0x7fd9bc0741c0 2024-03-18T16:10:00.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.020+0000 7fd9c1600640 1 --2- 172.21.15.39:0/4294254556 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd9bc0741c0 0x7fd9bc199d40 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fd9b000e9f0 tx=0x7fd9b000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:00.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.020+0000 7fd9ba000640 1 -- 172.21.15.39:0/4294254556 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9b000cd80 con 0x7fd9bc0741c0 2024-03-18T16:10:00.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.020+0000 7fd9ba000640 1 -- 172.21.15.39:0/4294254556 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd9b0010910 con 0x7fd9bc0741c0 2024-03-18T16:10:00.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.020+0000 7fd9ba000640 1 -- 172.21.15.39:0/4294254556 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9b0018990 con 0x7fd9bc0741c0 2024-03-18T16:10:00.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.020+0000 7fd9c4000640 1 -- 172.21.15.39:0/4294254556 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9bc1a1c40 con 0x7fd9bc0741c0 2024-03-18T16:10:00.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.021+0000 7fd9c4000640 1 -- 172.21.15.39:0/4294254556 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9bc1a2140 con 0x7fd9bc0741c0 2024-03-18T16:10:00.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.022+0000 7fd9c4000640 1 -- 172.21.15.39:0/4294254556 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9bc108460 con 0x7fd9bc0741c0 2024-03-18T16:10:00.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.023+0000 7fd9ba000640 1 -- 172.21.15.39:0/4294254556 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fd9b0010ac0 con 0x7fd9bc0741c0 2024-03-18T16:10:00.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.025+0000 7fd9ba000640 1 --2- 172.21.15.39:0/4294254556 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd9a40a2880 0x7fd9a40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:00.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.025+0000 7fd9ba000640 1 -- 172.21.15.39:0/4294254556 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fd9b00cf170 con 0x7fd9bc0741c0 2024-03-18T16:10:00.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.025+0000 7fd9c0c00640 1 --2- 172.21.15.39:0/4294254556 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd9a40a2880 0x7fd9a40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:00.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.026+0000 7fd9c0c00640 1 --2- 172.21.15.39:0/4294254556 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd9a40a2880 0x7fd9a40a4d40 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fd9a8009830 tx=0x7fd9a8034b50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:00.030 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.029+0000 7fd9ba000640 1 -- 172.21.15.39:0/4294254556 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd9b009b7d0 con 0x7fd9bc0741c0 2024-03-18T16:10:00.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.329+0000 7fd9c4000640 1 -- 172.21.15.39:0/4294254556 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fd9bc0623e0 con 0x7fd9bc0741c0 2024-03-18T16:10:00.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.330+0000 7fd9ba000640 1 -- 172.21.15.39:0/4294254556 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7fd9b009f870 con 0x7fd9bc0741c0 2024-03-18T16:10:00.331 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 32 2024-03-18T16:10:00.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.333+0000 7fd9c4000640 1 -- 172.21.15.39:0/4294254556 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd9a40a2880 msgr2=0x7fd9a40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:00.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.333+0000 7fd9c4000640 1 --2- 172.21.15.39:0/4294254556 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd9a40a2880 0x7fd9a40a4d40 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fd9a8009830 tx=0x7fd9a8034b50 comp rx=0 tx=0).stop 2024-03-18T16:10:00.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.333+0000 7fd9c4000640 1 -- 172.21.15.39:0/4294254556 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd9bc0741c0 msgr2=0x7fd9bc199d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:00.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.333+0000 7fd9c4000640 1 --2- 172.21.15.39:0/4294254556 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd9bc0741c0 0x7fd9bc199d40 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fd9b000e9f0 tx=0x7fd9b000eeb0 comp rx=0 tx=0).stop 2024-03-18T16:10:00.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.333+0000 7fd9c4000640 1 -- 172.21.15.39:0/4294254556 shutdown_connections 2024-03-18T16:10:00.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.333+0000 7fd9c4000640 1 --2- 172.21.15.39:0/4294254556 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd9bc1096a0 0x7fd9bc1a12b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:00.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.333+0000 7fd9c4000640 1 --2- 172.21.15.39:0/4294254556 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd9a40a2880 0x7fd9a40a4d40 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:00.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.333+0000 7fd9c4000640 1 --2- 172.21.15.39:0/4294254556 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd9bc0757a0 0x7fd9bc19a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:00.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.333+0000 7fd9c4000640 1 --2- 172.21.15.39:0/4294254556 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd9bc0741c0 0x7fd9bc199d40 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:00.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.333+0000 7fd9c4000640 1 -- 172.21.15.39:0/4294254556 >> 172.21.15.39:0/4294254556 conn(0x7fd9bc0fcd80 msgr2=0x7fd9bc105630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:00.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.333+0000 7fd9c4000640 1 -- 172.21.15.39:0/4294254556 shutdown_connections 2024-03-18T16:10:00.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.333+0000 7fd9c4000640 1 -- 172.21.15.39:0/4294254556 wait complete. 2024-03-18T16:10:00.355 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-03-18T16:10:00.355 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:10:00.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.564+0000 7fac9a800640 1 -- 172.21.15.39:0/1603913203 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fac941037e0 msgr2=0x7fac94103c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:00.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.564+0000 7fac9a800640 1 --2- 172.21.15.39:0/1603913203 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fac941037e0 0x7fac94103c00 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fac88009900 tx=0x7fac8802bc30 comp rx=0 tx=0).stop 2024-03-18T16:10:00.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.564+0000 7fac9a800640 1 -- 172.21.15.39:0/1603913203 shutdown_connections 2024-03-18T16:10:00.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.564+0000 7fac9a800640 1 --2- 172.21.15.39:0/1603913203 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fac941049d0 0x7fac94104e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:00.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.564+0000 7fac9a800640 1 --2- 172.21.15.39:0/1603913203 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fac94105390 0x7fac9410bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:00.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.564+0000 7fac9a800640 1 --2- 172.21.15.39:0/1603913203 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fac941037e0 0x7fac94103c00 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:00.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.564+0000 7fac9a800640 1 -- 172.21.15.39:0/1603913203 >> 172.21.15.39:0/1603913203 conn(0x7fac940fed50 msgr2=0x7fac94101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:00.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.564+0000 7fac9a800640 1 -- 172.21.15.39:0/1603913203 shutdown_connections 2024-03-18T16:10:00.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.564+0000 7fac9a800640 1 -- 172.21.15.39:0/1603913203 wait complete. 2024-03-18T16:10:00.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.565+0000 7fac9a800640 1 Processor -- start 2024-03-18T16:10:00.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.565+0000 7fac9a800640 1 -- start start 2024-03-18T16:10:00.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.566+0000 7fac9a800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fac941037e0 0x7fac94199b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:00.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.566+0000 7fac9a800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fac941049d0 0x7fac9419a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:00.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.566+0000 7fac9a800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fac94105390 0x7fac941a10e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:00.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.566+0000 7fac9a800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fac9410f4a0 con 0x7fac94105390 2024-03-18T16:10:00.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.566+0000 7fac9a800640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fac9410f320 con 0x7fac941049d0 2024-03-18T16:10:00.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.566+0000 7fac9a800640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fac9410f620 con 0x7fac941037e0 2024-03-18T16:10:00.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.566+0000 7fac99e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fac94105390 0x7fac941a10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:00.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.566+0000 7fac98a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fac941049d0 0x7fac9419a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:00.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.566+0000 7fac99400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fac941037e0 0x7fac94199b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:00.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.566+0000 7fac99e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fac94105390 0x7fac941a10e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:55688/0 (socket says 172.21.15.39:55688) 2024-03-18T16:10:00.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.566+0000 7fac99e00640 1 -- 172.21.15.39:0/2607097098 learned_addr learned my addr 172.21.15.39:0/2607097098 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:10:00.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.567+0000 7fac99e00640 1 -- 172.21.15.39:0/2607097098 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fac941037e0 msgr2=0x7fac94199b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:00.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.567+0000 7fac99e00640 1 --2- 172.21.15.39:0/2607097098 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fac941037e0 0x7fac94199b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:00.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.567+0000 7fac99e00640 1 -- 172.21.15.39:0/2607097098 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fac941049d0 msgr2=0x7fac9419a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:00.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.567+0000 7fac99e00640 1 --2- 172.21.15.39:0/2607097098 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fac941049d0 0x7fac9419a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:00.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.567+0000 7fac99e00640 1 -- 172.21.15.39:0/2607097098 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac941a1750 con 0x7fac94105390 2024-03-18T16:10:00.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.567+0000 7fac99e00640 1 --2- 172.21.15.39:0/2607097098 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fac94105390 0x7fac941a10e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fac8c00efd0 tx=0x7fac8c00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:00.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.567+0000 7fac92000640 1 -- 172.21.15.39:0/2607097098 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac8c003b70 con 0x7fac94105390 2024-03-18T16:10:00.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.567+0000 7fac92000640 1 -- 172.21.15.39:0/2607097098 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fac8c003d10 con 0x7fac94105390 2024-03-18T16:10:00.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.567+0000 7fac9a800640 1 -- 172.21.15.39:0/2607097098 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac941a1a40 con 0x7fac94105390 2024-03-18T16:10:00.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.567+0000 7fac92000640 1 -- 172.21.15.39:0/2607097098 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac8c0109f0 con 0x7fac94105390 2024-03-18T16:10:00.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.567+0000 7fac9a800640 1 -- 172.21.15.39:0/2607097098 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac941a1f40 con 0x7fac94105390 2024-03-18T16:10:00.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.569+0000 7fac9a800640 1 -- 172.21.15.39:0/2607097098 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac9410aba0 con 0x7fac94105390 2024-03-18T16:10:00.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.571+0000 7fac92000640 1 -- 172.21.15.39:0/2607097098 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fac8c004020 con 0x7fac94105390 2024-03-18T16:10:00.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.573+0000 7fac92000640 1 --2- 172.21.15.39:0/2607097098 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fac7c0a2800 0x7fac7c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:00.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.573+0000 7fac92000640 1 -- 172.21.15.39:0/2607097098 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fac8c0cf2c0 con 0x7fac94105390 2024-03-18T16:10:00.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.573+0000 7fac99400640 1 --2- 172.21.15.39:0/2607097098 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fac7c0a2800 0x7fac7c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:00.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.574+0000 7fac99400640 1 --2- 172.21.15.39:0/2607097098 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fac7c0a2800 0x7fac7c0a4cc0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fac8802c170 tx=0x7fac8802df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:00.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.578+0000 7fac92000640 1 -- 172.21.15.39:0/2607097098 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fac8c09bf70 con 0x7fac94105390 2024-03-18T16:10:00.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:10:00 smithi196 ceph-mon[36307]: pgmap v420: 129 pgs: 129 active+clean; 649 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 938 B/s wr, 11 op/s 2024-03-18T16:10:00.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:10:00 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/466280127' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T16:10:00.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:10:00 smithi196 ceph-mon[36307]: overall HEALTH_OK 2024-03-18T16:10:00.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:10:00 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/4294254556' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T16:10:00.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:10:00 smithi194 ceph-mon[36200]: pgmap v420: 129 pgs: 129 active+clean; 649 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 938 B/s wr, 11 op/s 2024-03-18T16:10:00.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:10:00 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/466280127' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T16:10:00.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:10:00 smithi194 ceph-mon[36200]: overall HEALTH_OK 2024-03-18T16:10:00.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:10:00 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/4294254556' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T16:10:00.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.832+0000 7fac9a800640 1 -- 172.21.15.39:0/2607097098 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fac9402d1b0 con 0x7fac94105390 2024-03-18T16:10:00.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.834+0000 7fac92000640 1 -- 172.21.15.39:0/2607097098 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11932 (secure 0 0 0) 0x7fac8c09b870 con 0x7fac94105390 2024-03-18T16:10:00.835 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:10:00.836 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2520226608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2520226608},{"type":"v1","addr":"172.21.15.39:6835","nonce":2520226608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/719341763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":719341763},{"type":"v1","addr":"172.21.15.39:6837","nonce":719341763}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T16:06:44.180425+0000","modified":"2024-03-18T16:09:54.861629+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24575,"mds_1":24556,"mds_2":24535,"mds_3":24524,"mds_4":14463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14445":{"gid":14445,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.39:6839/656513813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":656513813},{"type":"v1","addr":"172.21.15.39:6839","nonce":656513813}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14463":{"gid":14463,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":13,"addr":"172.21.15.39:6841/3515326042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3515326042},{"type":"v1","addr":"172.21.15.39:6841","nonce":3515326042}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6833/931350738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":931350738},{"type":"v1","addr":"172.21.15.194:6833","nonce":931350738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6833/3874209621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":3874209621},{"type":"v1","addr":"172.21.15.196:6833","nonce":3874209621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6835/188385470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":188385470},{"type":"v1","addr":"172.21.15.194:6835","nonce":188385470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6835/3949984309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":3949984309},{"type":"v1","addr":"172.21.15.196:6835","nonce":3949984309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":15,"addr":"172.21.15.194:6837/1301464246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":1301464246},{"type":"v1","addr":"172.21.15.194:6837","nonce":1301464246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":16,"addr":"172.21.15.196:6837/895247251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":895247251},{"type":"v1","addr":"172.21.15.196:6837","nonce":895247251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.196:6839/2001427268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6838","nonce":2001427268},{"type":"v1","addr":"172.21.15.196:6839","nonce":2001427268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24575":{"gid":24575,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.194:6839/274911158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":274911158},{"type":"v1","addr":"172.21.15.194:6839","nonce":274911158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24575,"qdb_cluster":[14463,24524,24575,24556,24535]},"id":1}]} 2024-03-18T16:10:00.836 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 28 2024-03-18T16:10:00.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:10:00 smithi039 ceph-mon[31721]: pgmap v420: 129 pgs: 129 active+clean; 649 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 938 B/s wr, 11 op/s 2024-03-18T16:10:00.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:10:00 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/466280127' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T16:10:00.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:10:00 smithi039 ceph-mon[31721]: overall HEALTH_OK 2024-03-18T16:10:00.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:10:00 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/4294254556' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T16:10:00.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.837+0000 7fac9a800640 1 -- 172.21.15.39:0/2607097098 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fac7c0a2800 msgr2=0x7fac7c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:00.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.837+0000 7fac9a800640 1 --2- 172.21.15.39:0/2607097098 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fac7c0a2800 0x7fac7c0a4cc0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fac8802c170 tx=0x7fac8802df90 comp rx=0 tx=0).stop 2024-03-18T16:10:00.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.837+0000 7fac9a800640 1 -- 172.21.15.39:0/2607097098 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fac94105390 msgr2=0x7fac941a10e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:00.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.837+0000 7fac9a800640 1 --2- 172.21.15.39:0/2607097098 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fac94105390 0x7fac941a10e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fac8c00efd0 tx=0x7fac8c00c6a0 comp rx=0 tx=0).stop 2024-03-18T16:10:00.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.837+0000 7fac9a800640 1 -- 172.21.15.39:0/2607097098 shutdown_connections 2024-03-18T16:10:00.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.837+0000 7fac9a800640 1 --2- 172.21.15.39:0/2607097098 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fac941049d0 0x7fac9419a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:00.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.837+0000 7fac9a800640 1 --2- 172.21.15.39:0/2607097098 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fac7c0a2800 0x7fac7c0a4cc0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:00.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.837+0000 7fac9a800640 1 --2- 172.21.15.39:0/2607097098 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fac94105390 0x7fac941a10e0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:00.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.837+0000 7fac9a800640 1 --2- 172.21.15.39:0/2607097098 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fac941037e0 0x7fac94199b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:00.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.837+0000 7fac9a800640 1 -- 172.21.15.39:0/2607097098 >> 172.21.15.39:0/2607097098 conn(0x7fac940fed50 msgr2=0x7fac94107d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:00.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.837+0000 7fac9a800640 1 -- 172.21.15.39:0/2607097098 shutdown_connections 2024-03-18T16:10:00.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:00.837+0000 7fac9a800640 1 -- 172.21.15.39:0/2607097098 wait complete. 2024-03-18T16:10:00.856 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-03-18T16:10:00.857 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-03-18T16:10:00.857 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T16:10:01.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.085+0000 7fc179c00640 1 -- 172.21.15.39:0/2084801884 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc174100080 msgr2=0x7fc1741004a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:01.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.085+0000 7fc179c00640 1 --2- 172.21.15.39:0/2084801884 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc174100080 0x7fc1741004a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fc164009900 tx=0x7fc16402bc30 comp rx=0 tx=0).stop 2024-03-18T16:10:01.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.085+0000 7fc179c00640 1 -- 172.21.15.39:0/2084801884 shutdown_connections 2024-03-18T16:10:01.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.085+0000 7fc179c00640 1 --2- 172.21.15.39:0/2084801884 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc174101380 0x7fc174101800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:01.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.085+0000 7fc179c00640 1 --2- 172.21.15.39:0/2084801884 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc1741009e0 0x7fc174100e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:01.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.085+0000 7fc179c00640 1 --2- 172.21.15.39:0/2084801884 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc174100080 0x7fc1741004a0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:01.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.085+0000 7fc179c00640 1 -- 172.21.15.39:0/2084801884 >> 172.21.15.39:0/2084801884 conn(0x7fc1740fbc40 msgr2=0x7fc1740fe080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:01.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.085+0000 7fc179c00640 1 -- 172.21.15.39:0/2084801884 shutdown_connections 2024-03-18T16:10:01.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.085+0000 7fc179c00640 1 -- 172.21.15.39:0/2084801884 wait complete. 2024-03-18T16:10:01.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.086+0000 7fc179c00640 1 Processor -- start 2024-03-18T16:10:01.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.086+0000 7fc179c00640 1 -- start start 2024-03-18T16:10:01.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.086+0000 7fc179c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc174100080 0x7fc174199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:01.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.086+0000 7fc179c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc1741009e0 0x7fc17419a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:01.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.086+0000 7fc179c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc174101380 0x7fc1741a11e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:01.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.086+0000 7fc179c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fc17410f580 con 0x7fc174101380 2024-03-18T16:10:01.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.086+0000 7fc179c00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fc17410f400 con 0x7fc174100080 2024-03-18T16:10:01.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.086+0000 7fc179c00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fc17410f700 con 0x7fc1741009e0 2024-03-18T16:10:01.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.087+0000 7fc172a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc174100080 0x7fc174199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:01.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.087+0000 7fc172000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc1741009e0 0x7fc17419a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:01.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.087+0000 7fc172a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc174100080 0x7fc174199c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:36468/0 (socket says 172.21.15.39:36468) 2024-03-18T16:10:01.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.087+0000 7fc173400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc174101380 0x7fc1741a11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:01.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.087+0000 7fc172a00640 1 -- 172.21.15.39:0/3352211267 learned_addr learned my addr 172.21.15.39:0/3352211267 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:10:01.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.087+0000 7fc172000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc1741009e0 0x7fc17419a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:49756/0 (socket says 172.21.15.39:49756) 2024-03-18T16:10:01.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.087+0000 7fc172a00640 1 -- 172.21.15.39:0/3352211267 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc1741009e0 msgr2=0x7fc17419a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:01.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.087+0000 7fc172a00640 1 --2- 172.21.15.39:0/3352211267 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc1741009e0 0x7fc17419a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:01.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.087+0000 7fc172a00640 1 -- 172.21.15.39:0/3352211267 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc174101380 msgr2=0x7fc1741a11e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:01.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.087+0000 7fc172a00640 1 --2- 172.21.15.39:0/3352211267 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc174101380 0x7fc1741a11e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:01.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.087+0000 7fc172a00640 1 -- 172.21.15.39:0/3352211267 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1741a1880 con 0x7fc174100080 2024-03-18T16:10:01.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.087+0000 7fc172000640 1 --2- 172.21.15.39:0/3352211267 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc1741009e0 0x7fc17419a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:10:01.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.087+0000 7fc172a00640 1 --2- 172.21.15.39:0/3352211267 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc174100080 0x7fc174199c70 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fc164009a20 tx=0x7fc16402c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:01.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.088+0000 7fc16b400640 1 -- 172.21.15.39:0/3352211267 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc164043020 con 0x7fc174100080 2024-03-18T16:10:01.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.088+0000 7fc179c00640 1 -- 172.21.15.39:0/3352211267 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1741a1b10 con 0x7fc174100080 2024-03-18T16:10:01.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.088+0000 7fc16b400640 1 -- 172.21.15.39:0/3352211267 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1640054e0 con 0x7fc174100080 2024-03-18T16:10:01.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.088+0000 7fc16b400640 1 -- 172.21.15.39:0/3352211267 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc164003ca0 con 0x7fc174100080 2024-03-18T16:10:01.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.088+0000 7fc179c00640 1 -- 172.21.15.39:0/3352211267 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1741a2060 con 0x7fc174100080 2024-03-18T16:10:01.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.090+0000 7fc179c00640 1 -- 172.21.15.39:0/3352211267 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc174068640 con 0x7fc174100080 2024-03-18T16:10:01.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.091+0000 7fc16b400640 1 -- 172.21.15.39:0/3352211267 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fc164046020 con 0x7fc174100080 2024-03-18T16:10:01.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.092+0000 7fc16b400640 1 --2- 172.21.15.39:0/3352211267 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fc1580a2800 0x7fc1580a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:01.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.092+0000 7fc16b400640 1 -- 172.21.15.39:0/3352211267 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fc1640f10d0 con 0x7fc174100080 2024-03-18T16:10:01.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.092+0000 7fc172000640 1 --2- 172.21.15.39:0/3352211267 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fc1580a2800 0x7fc1580a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:01.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.093+0000 7fc172000640 1 --2- 172.21.15.39:0/3352211267 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fc1580a2800 0x7fc1580a4cc0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fc160005d00 tx=0x7fc16000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:01.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.096+0000 7fc16b400640 1 -- 172.21.15.39:0/3352211267 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc16404d020 con 0x7fc174100080 2024-03-18T16:10:01.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.294+0000 7fc179c00640 1 -- 172.21.15.39:0/3352211267 --> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc174061c20 con 0x7fc1580a2800 2024-03-18T16:10:01.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.327+0000 7fc16b400640 1 -- 172.21.15.39:0/3352211267 <== mgr.14152 v2:172.21.15.39:6800/2749214286 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340542 (secure 0 0 0) 0x7fc174061c20 con 0x7fc1580a2800 2024-03-18T16:10:01.334 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:10:01.336 INFO:teuthology.orchestra.run.smithi039.stderr:dumped all 2024-03-18T16:10:01.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.336+0000 7fc179c00640 1 -- 172.21.15.39:0/3352211267 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fc1580a2800 msgr2=0x7fc1580a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:01.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.336+0000 7fc179c00640 1 --2- 172.21.15.39:0/3352211267 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fc1580a2800 0x7fc1580a4cc0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fc160005d00 tx=0x7fc16000b410 comp rx=0 tx=0).stop 2024-03-18T16:10:01.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.336+0000 7fc179c00640 1 -- 172.21.15.39:0/3352211267 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc174100080 msgr2=0x7fc174199c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:01.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.336+0000 7fc179c00640 1 --2- 172.21.15.39:0/3352211267 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc174100080 0x7fc174199c70 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fc164009a20 tx=0x7fc16402c650 comp rx=0 tx=0).stop 2024-03-18T16:10:01.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.336+0000 7fc179c00640 1 -- 172.21.15.39:0/3352211267 shutdown_connections 2024-03-18T16:10:01.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.336+0000 7fc179c00640 1 --2- 172.21.15.39:0/3352211267 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fc1580a2800 0x7fc1580a4cc0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:01.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.336+0000 7fc179c00640 1 --2- 172.21.15.39:0/3352211267 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc174101380 0x7fc1741a11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:01.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.336+0000 7fc179c00640 1 --2- 172.21.15.39:0/3352211267 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc1741009e0 0x7fc17419a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:01.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.336+0000 7fc179c00640 1 --2- 172.21.15.39:0/3352211267 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc174100080 0x7fc174199c70 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:01.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.336+0000 7fc179c00640 1 -- 172.21.15.39:0/3352211267 >> 172.21.15.39:0/3352211267 conn(0x7fc1740fbc40 msgr2=0x7fc1740fd3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:01.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.336+0000 7fc179c00640 1 -- 172.21.15.39:0/3352211267 shutdown_connections 2024-03-18T16:10:01.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.336+0000 7fc179c00640 1 -- 172.21.15.39:0/3352211267 wait complete. 2024-03-18T16:10:01.360 INFO:teuthology.orchestra.run.smithi039.stdout:{"pg_ready":true,"pg_map":{"version":420,"stamp":"2024-03-18T16:09:59.573392+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":664231,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1753,"num_read_kb":1005,"num_write":441,"num_write_kb":3094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":1180943,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":862,"ondisk_log_size":862,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":401264,"kb_used_data":14628,"kb_used_omap":18,"kb_used_meta":386477,"kb_avail":1124540560,"statfs":{"total":1151940427776,"available":1151529533440,"internally_reserved":0,"allocated":14979072,"data_stored":8199473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":395752763},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4934,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":133,"num_read_kb":72,"num_write":9,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005719"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.882222+0000","last_change":"2024-03-18T16:06:44.736511+0000","last_active":"2024-03-18T16:08:47.882222+0000","last_peered":"2024-03-18T16:08:47.882222+0000","last_clean":"2024-03-18T16:08:47.882222+0000","last_became_active":"2024-03-18T16:06:44.736182+0000","last_became_peered":"2024-03-18T16:06:44.736182+0000","last_unstale":"2024-03-18T16:08:47.882222+0000","last_undegraded":"2024-03-18T16:08:47.882222+0000","last_fullsized":"2024-03-18T16:08:47.882222+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:08:57.060268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.961793+0000","last_change":"2024-03-18T16:06:44.733658+0000","last_active":"2024-03-18T16:08:47.961793+0000","last_peered":"2024-03-18T16:08:47.961793+0000","last_clean":"2024-03-18T16:08:47.961793+0000","last_became_active":"2024-03-18T16:06:44.733279+0000","last_became_peered":"2024-03-18T16:06:44.733279+0000","last_unstale":"2024-03-18T16:08:47.961793+0000","last_undegraded":"2024-03-18T16:08:47.961793+0000","last_fullsized":"2024-03-18T16:08:47.961793+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:59:46.783025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.961451+0000","last_change":"2024-03-18T16:06:44.723949+0000","last_active":"2024-03-18T16:08:47.961451+0000","last_peered":"2024-03-18T16:08:47.961451+0000","last_clean":"2024-03-18T16:08:47.961451+0000","last_became_active":"2024-03-18T16:06:44.723614+0000","last_became_peered":"2024-03-18T16:06:44.723614+0000","last_unstale":"2024-03-18T16:08:47.961451+0000","last_undegraded":"2024-03-18T16:08:47.961451+0000","last_fullsized":"2024-03-18T16:08:47.961451+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:03:17.876097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883864+0000","last_change":"2024-03-18T16:06:44.711058+0000","last_active":"2024-03-18T16:08:47.883864+0000","last_peered":"2024-03-18T16:08:47.883864+0000","last_clean":"2024-03-18T16:08:47.883864+0000","last_became_active":"2024-03-18T16:06:44.708873+0000","last_became_peered":"2024-03-18T16:06:44.708873+0000","last_unstale":"2024-03-18T16:08:47.883864+0000","last_undegraded":"2024-03-18T16:08:47.883864+0000","last_fullsized":"2024-03-18T16:08:47.883864+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:58:08.182651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.884722+0000","last_change":"2024-03-18T16:06:44.721480+0000","last_active":"2024-03-18T16:08:47.884722+0000","last_peered":"2024-03-18T16:08:47.884722+0000","last_clean":"2024-03-18T16:08:47.884722+0000","last_became_active":"2024-03-18T16:06:44.721101+0000","last_became_peered":"2024-03-18T16:06:44.721101+0000","last_unstale":"2024-03-18T16:08:47.884722+0000","last_undegraded":"2024-03-18T16:08:47.884722+0000","last_fullsized":"2024-03-18T16:08:47.884722+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:19:43.341221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.879862+0000","last_change":"2024-03-18T16:06:44.716443+0000","last_active":"2024-03-18T16:08:47.879862+0000","last_peered":"2024-03-18T16:08:47.879862+0000","last_clean":"2024-03-18T16:08:47.879862+0000","last_became_active":"2024-03-18T16:06:44.715969+0000","last_became_peered":"2024-03-18T16:06:44.715969+0000","last_unstale":"2024-03-18T16:08:47.879862+0000","last_undegraded":"2024-03-18T16:08:47.879862+0000","last_fullsized":"2024-03-18T16:08:47.879862+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:57:14.714415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.960378+0000","last_change":"2024-03-18T16:06:44.711439+0000","last_active":"2024-03-18T16:08:47.960378+0000","last_peered":"2024-03-18T16:08:47.960378+0000","last_clean":"2024-03-18T16:08:47.960378+0000","last_became_active":"2024-03-18T16:06:44.711032+0000","last_became_peered":"2024-03-18T16:06:44.711032+0000","last_unstale":"2024-03-18T16:08:47.960378+0000","last_undegraded":"2024-03-18T16:08:47.960378+0000","last_fullsized":"2024-03-18T16:08:47.960378+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:19:56.516867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.881902+0000","last_change":"2024-03-18T16:07:47.358866+0000","last_active":"2024-03-18T16:08:47.881902+0000","last_peered":"2024-03-18T16:08:47.881902+0000","last_clean":"2024-03-18T16:08:47.881902+0000","last_became_active":"2024-03-18T16:07:47.357743+0000","last_became_peered":"2024-03-18T16:07:47.357743+0000","last_unstale":"2024-03-18T16:08:47.881902+0000","last_undegraded":"2024-03-18T16:08:47.881902+0000","last_fullsized":"2024-03-18T16:08:47.881902+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:13:12.361025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,10],"acting":[9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.960770+0000","last_change":"2024-03-18T16:06:44.710978+0000","last_active":"2024-03-18T16:08:47.960770+0000","last_peered":"2024-03-18T16:08:47.960770+0000","last_clean":"2024-03-18T16:08:47.960770+0000","last_became_active":"2024-03-18T16:06:44.710520+0000","last_became_peered":"2024-03-18T16:06:44.710520+0000","last_unstale":"2024-03-18T16:08:47.960770+0000","last_undegraded":"2024-03-18T16:08:47.960770+0000","last_fullsized":"2024-03-18T16:08:47.960770+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:54:53.234025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.880413+0000","last_change":"2024-03-18T16:06:44.716231+0000","last_active":"2024-03-18T16:08:47.880413+0000","last_peered":"2024-03-18T16:08:47.880413+0000","last_clean":"2024-03-18T16:08:47.880413+0000","last_became_active":"2024-03-18T16:06:44.714900+0000","last_became_peered":"2024-03-18T16:06:44.714900+0000","last_unstale":"2024-03-18T16:08:47.880413+0000","last_undegraded":"2024-03-18T16:08:47.880413+0000","last_fullsized":"2024-03-18T16:08:47.880413+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:38:54.964729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.961221+0000","last_change":"2024-03-18T16:06:44.710942+0000","last_active":"2024-03-18T16:08:47.961221+0000","last_peered":"2024-03-18T16:08:47.961221+0000","last_clean":"2024-03-18T16:08:47.961221+0000","last_became_active":"2024-03-18T16:06:44.710588+0000","last_became_peered":"2024-03-18T16:06:44.710588+0000","last_unstale":"2024-03-18T16:08:47.961221+0000","last_undegraded":"2024-03-18T16:08:47.961221+0000","last_fullsized":"2024-03-18T16:08:47.961221+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:43:19.320034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883838+0000","last_change":"2024-03-18T16:06:44.714763+0000","last_active":"2024-03-18T16:08:47.883838+0000","last_peered":"2024-03-18T16:08:47.883838+0000","last_clean":"2024-03-18T16:08:47.883838+0000","last_became_active":"2024-03-18T16:06:44.714031+0000","last_became_peered":"2024-03-18T16:06:44.714031+0000","last_unstale":"2024-03-18T16:08:47.883838+0000","last_undegraded":"2024-03-18T16:08:47.883838+0000","last_fullsized":"2024-03-18T16:08:47.883838+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:19:25.917093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.962631+0000","last_change":"2024-03-18T16:06:44.710062+0000","last_active":"2024-03-18T16:08:47.962631+0000","last_peered":"2024-03-18T16:08:47.962631+0000","last_clean":"2024-03-18T16:08:47.962631+0000","last_became_active":"2024-03-18T16:06:44.709281+0000","last_became_peered":"2024-03-18T16:06:44.709281+0000","last_unstale":"2024-03-18T16:08:47.962631+0000","last_undegraded":"2024-03-18T16:08:47.962631+0000","last_fullsized":"2024-03-18T16:08:47.962631+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:51:30.100947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.960815+0000","last_change":"2024-03-18T16:06:44.734178+0000","last_active":"2024-03-18T16:08:47.960815+0000","last_peered":"2024-03-18T16:08:47.960815+0000","last_clean":"2024-03-18T16:08:47.960815+0000","last_became_active":"2024-03-18T16:06:44.732538+0000","last_became_peered":"2024-03-18T16:06:44.732538+0000","last_unstale":"2024-03-18T16:08:47.960815+0000","last_undegraded":"2024-03-18T16:08:47.960815+0000","last_fullsized":"2024-03-18T16:08:47.960815+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:14:26.904402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.960944+0000","last_change":"2024-03-18T16:07:47.412629+0000","last_active":"2024-03-18T16:08:47.960944+0000","last_peered":"2024-03-18T16:08:47.960944+0000","last_clean":"2024-03-18T16:08:47.960944+0000","last_became_active":"2024-03-18T16:07:47.411714+0000","last_became_peered":"2024-03-18T16:07:47.411714+0000","last_unstale":"2024-03-18T16:08:47.960944+0000","last_undegraded":"2024-03-18T16:08:47.960944+0000","last_fullsized":"2024-03-18T16:08:47.960944+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:25:50.843323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.959907+0000","last_change":"2024-03-18T16:06:44.728165+0000","last_active":"2024-03-18T16:08:47.959907+0000","last_peered":"2024-03-18T16:08:47.959907+0000","last_clean":"2024-03-18T16:08:47.959907+0000","last_became_active":"2024-03-18T16:06:44.727842+0000","last_became_peered":"2024-03-18T16:06:44.727842+0000","last_unstale":"2024-03-18T16:08:47.959907+0000","last_undegraded":"2024-03-18T16:08:47.959907+0000","last_fullsized":"2024-03-18T16:08:47.959907+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:55:04.464916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.881962+0000","last_change":"2024-03-18T16:06:44.730893+0000","last_active":"2024-03-18T16:08:47.881962+0000","last_peered":"2024-03-18T16:08:47.881962+0000","last_clean":"2024-03-18T16:08:47.881962+0000","last_became_active":"2024-03-18T16:06:44.730544+0000","last_became_peered":"2024-03-18T16:06:44.730544+0000","last_unstale":"2024-03-18T16:08:47.881962+0000","last_undegraded":"2024-03-18T16:08:47.881962+0000","last_fullsized":"2024-03-18T16:08:47.881962+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:16:37.918096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.884320+0000","last_change":"2024-03-18T16:06:44.706036+0000","last_active":"2024-03-18T16:08:47.884320+0000","last_peered":"2024-03-18T16:08:47.884320+0000","last_clean":"2024-03-18T16:08:47.884320+0000","last_became_active":"2024-03-18T16:06:44.705281+0000","last_became_peered":"2024-03-18T16:06:44.705281+0000","last_unstale":"2024-03-18T16:08:47.884320+0000","last_undegraded":"2024-03-18T16:08:47.884320+0000","last_fullsized":"2024-03-18T16:08:47.884320+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:52:20.202110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.884506+0000","last_change":"2024-03-18T16:06:44.727543+0000","last_active":"2024-03-18T16:08:47.884506+0000","last_peered":"2024-03-18T16:08:47.884506+0000","last_clean":"2024-03-18T16:08:47.884506+0000","last_became_active":"2024-03-18T16:06:44.727135+0000","last_became_peered":"2024-03-18T16:06:44.727135+0000","last_unstale":"2024-03-18T16:08:47.884506+0000","last_undegraded":"2024-03-18T16:08:47.884506+0000","last_fullsized":"2024-03-18T16:08:47.884506+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:52:42.742279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883556+0000","last_change":"2024-03-18T16:06:49.028004+0000","last_active":"2024-03-18T16:08:47.883556+0000","last_peered":"2024-03-18T16:08:47.883556+0000","last_clean":"2024-03-18T16:08:47.883556+0000","last_became_active":"2024-03-18T16:06:49.027640+0000","last_became_peered":"2024-03-18T16:06:49.027640+0000","last_unstale":"2024-03-18T16:08:47.883556+0000","last_undegraded":"2024-03-18T16:08:47.883556+0000","last_fullsized":"2024-03-18T16:08:47.883556+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:23:07.926349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'13","reported_seq":202,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:09:55.306333+0000","last_change":"2024-03-18T16:06:43.709540+0000","last_active":"2024-03-18T16:09:55.306333+0000","last_peered":"2024-03-18T16:09:55.306333+0000","last_clean":"2024-03-18T16:09:55.306333+0000","last_became_active":"2024-03-18T16:06:43.709170+0000","last_became_peered":"2024-03-18T16:06:43.709170+0000","last_unstale":"2024-03-18T16:09:55.306333+0000","last_undegraded":"2024-03-18T16:09:55.306333+0000","last_fullsized":"2024-03-18T16:09:55.306333+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:12:53.934079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":157,"num_read_kb":157,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.886428+0000","last_change":"2024-03-18T16:06:44.730192+0000","last_active":"2024-03-18T16:08:47.886428+0000","last_peered":"2024-03-18T16:08:47.886428+0000","last_clean":"2024-03-18T16:08:47.886428+0000","last_became_active":"2024-03-18T16:06:44.728071+0000","last_became_peered":"2024-03-18T16:06:44.728071+0000","last_unstale":"2024-03-18T16:08:47.886428+0000","last_undegraded":"2024-03-18T16:08:47.886428+0000","last_fullsized":"2024-03-18T16:08:47.886428+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:14:59.992958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883329+0000","last_change":"2024-03-18T16:06:49.023963+0000","last_active":"2024-03-18T16:08:47.883329+0000","last_peered":"2024-03-18T16:08:47.883329+0000","last_clean":"2024-03-18T16:08:47.883329+0000","last_became_active":"2024-03-18T16:06:49.023568+0000","last_became_peered":"2024-03-18T16:06:49.023568+0000","last_unstale":"2024-03-18T16:08:47.883329+0000","last_undegraded":"2024-03-18T16:08:47.883329+0000","last_fullsized":"2024-03-18T16:08:47.883329+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:15:19.730842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.886440+0000","last_change":"2024-03-18T16:06:43.722167+0000","last_active":"2024-03-18T16:08:47.886440+0000","last_peered":"2024-03-18T16:08:47.886440+0000","last_clean":"2024-03-18T16:08:47.886440+0000","last_became_active":"2024-03-18T16:06:43.721782+0000","last_became_peered":"2024-03-18T16:06:43.721782+0000","last_unstale":"2024-03-18T16:08:47.886440+0000","last_undegraded":"2024-03-18T16:08:47.886440+0000","last_fullsized":"2024-03-18T16:08:47.886440+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:08:54.400637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.962047+0000","last_change":"2024-03-18T16:06:44.729666+0000","last_active":"2024-03-18T16:08:47.962047+0000","last_peered":"2024-03-18T16:08:47.962047+0000","last_clean":"2024-03-18T16:08:47.962047+0000","last_became_active":"2024-03-18T16:06:44.728842+0000","last_became_peered":"2024-03-18T16:06:44.728842+0000","last_unstale":"2024-03-18T16:08:47.962047+0000","last_undegraded":"2024-03-18T16:08:47.962047+0000","last_fullsized":"2024-03-18T16:08:47.962047+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:01:42.438000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883464+0000","last_change":"2024-03-18T16:06:49.031169+0000","last_active":"2024-03-18T16:08:47.883464+0000","last_peered":"2024-03-18T16:08:47.883464+0000","last_clean":"2024-03-18T16:08:47.883464+0000","last_became_active":"2024-03-18T16:06:49.030801+0000","last_became_peered":"2024-03-18T16:06:49.030801+0000","last_unstale":"2024-03-18T16:08:47.883464+0000","last_undegraded":"2024-03-18T16:08:47.883464+0000","last_fullsized":"2024-03-18T16:08:47.883464+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:10:54.880882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'12","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.885418+0000","last_change":"2024-03-18T16:06:43.707111+0000","last_active":"2024-03-18T16:08:47.885418+0000","last_peered":"2024-03-18T16:08:47.885418+0000","last_clean":"2024-03-18T16:08:47.885418+0000","last_became_active":"2024-03-18T16:06:43.706671+0000","last_became_peered":"2024-03-18T16:06:43.706671+0000","last_unstale":"2024-03-18T16:08:47.885418+0000","last_undegraded":"2024-03-18T16:08:47.885418+0000","last_fullsized":"2024-03-18T16:08:47.885418+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:18:34.295641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.960382+0000","last_change":"2024-03-18T16:06:44.722100+0000","last_active":"2024-03-18T16:08:47.960382+0000","last_peered":"2024-03-18T16:08:47.960382+0000","last_clean":"2024-03-18T16:08:47.960382+0000","last_became_active":"2024-03-18T16:06:44.721673+0000","last_became_peered":"2024-03-18T16:06:44.721673+0000","last_unstale":"2024-03-18T16:08:47.960382+0000","last_undegraded":"2024-03-18T16:08:47.960382+0000","last_fullsized":"2024-03-18T16:08:47.960382+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:41:22.624778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883382+0000","last_change":"2024-03-18T16:06:49.027966+0000","last_active":"2024-03-18T16:08:47.883382+0000","last_peered":"2024-03-18T16:08:47.883382+0000","last_clean":"2024-03-18T16:08:47.883382+0000","last_became_active":"2024-03-18T16:06:49.027567+0000","last_became_peered":"2024-03-18T16:06:49.027567+0000","last_unstale":"2024-03-18T16:08:47.883382+0000","last_undegraded":"2024-03-18T16:08:47.883382+0000","last_fullsized":"2024-03-18T16:08:47.883382+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:54:28.543019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.885092+0000","last_change":"2024-03-18T16:06:43.704023+0000","last_active":"2024-03-18T16:08:47.885092+0000","last_peered":"2024-03-18T16:08:47.885092+0000","last_clean":"2024-03-18T16:08:47.885092+0000","last_became_active":"2024-03-18T16:06:43.703690+0000","last_became_peered":"2024-03-18T16:06:43.703690+0000","last_unstale":"2024-03-18T16:08:47.885092+0000","last_undegraded":"2024-03-18T16:08:47.885092+0000","last_fullsized":"2024-03-18T16:08:47.885092+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:20:02.344346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":50,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.886144+0000","last_change":"2024-03-18T16:07:47.354361+0000","last_active":"2024-03-18T16:08:47.886144+0000","last_peered":"2024-03-18T16:08:47.886144+0000","last_clean":"2024-03-18T16:08:47.886144+0000","last_became_active":"2024-03-18T16:07:47.353964+0000","last_became_peered":"2024-03-18T16:07:47.353964+0000","last_unstale":"2024-03-18T16:08:47.886144+0000","last_undegraded":"2024-03-18T16:08:47.886144+0000","last_fullsized":"2024-03-18T16:08:47.886144+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:56:11.405218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.884343+0000","last_change":"2024-03-18T16:06:49.012310+0000","last_active":"2024-03-18T16:08:47.884343+0000","last_peered":"2024-03-18T16:08:47.884343+0000","last_clean":"2024-03-18T16:08:47.884343+0000","last_became_active":"2024-03-18T16:06:49.011844+0000","last_became_peered":"2024-03-18T16:06:49.011844+0000","last_unstale":"2024-03-18T16:08:47.884343+0000","last_undegraded":"2024-03-18T16:08:47.884343+0000","last_fullsized":"2024-03-18T16:08:47.884343+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:16:42.652205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"91'7","reported_seq":199,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:09:55.144566+0000","last_change":"2024-03-18T16:06:43.705653+0000","last_active":"2024-03-18T16:09:55.144566+0000","last_peered":"2024-03-18T16:09:55.144566+0000","last_clean":"2024-03-18T16:09:55.144566+0000","last_became_active":"2024-03-18T16:06:43.705320+0000","last_became_peered":"2024-03-18T16:06:43.705320+0000","last_unstale":"2024-03-18T16:09:55.144566+0000","last_undegraded":"2024-03-18T16:09:55.144566+0000","last_fullsized":"2024-03-18T16:09:55.144566+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:10:10.827338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":161,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.884849+0000","last_change":"2024-03-18T16:06:44.720045+0000","last_active":"2024-03-18T16:08:47.884849+0000","last_peered":"2024-03-18T16:08:47.884849+0000","last_clean":"2024-03-18T16:08:47.884849+0000","last_became_active":"2024-03-18T16:06:44.719669+0000","last_became_peered":"2024-03-18T16:06:44.719669+0000","last_unstale":"2024-03-18T16:08:47.884849+0000","last_undegraded":"2024-03-18T16:08:47.884849+0000","last_fullsized":"2024-03-18T16:08:47.884849+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:23:47.464057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.884107+0000","last_change":"2024-03-18T16:06:49.032576+0000","last_active":"2024-03-18T16:08:47.884107+0000","last_peered":"2024-03-18T16:08:47.884107+0000","last_clean":"2024-03-18T16:08:47.884107+0000","last_became_active":"2024-03-18T16:06:49.032126+0000","last_became_peered":"2024-03-18T16:06:49.032126+0000","last_unstale":"2024-03-18T16:08:47.884107+0000","last_undegraded":"2024-03-18T16:08:47.884107+0000","last_fullsized":"2024-03-18T16:08:47.884107+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:41:54.822070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.879820+0000","last_change":"2024-03-18T16:06:43.706364+0000","last_active":"2024-03-18T16:08:47.879820+0000","last_peered":"2024-03-18T16:08:47.879820+0000","last_clean":"2024-03-18T16:08:47.879820+0000","last_became_active":"2024-03-18T16:06:43.705984+0000","last_became_peered":"2024-03-18T16:06:43.705984+0000","last_unstale":"2024-03-18T16:08:47.879820+0000","last_undegraded":"2024-03-18T16:08:47.879820+0000","last_fullsized":"2024-03-18T16:08:47.879820+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:46:06.503000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.886025+0000","last_change":"2024-03-18T16:06:44.721019+0000","last_active":"2024-03-18T16:08:47.886025+0000","last_peered":"2024-03-18T16:08:47.886025+0000","last_clean":"2024-03-18T16:08:47.886025+0000","last_became_active":"2024-03-18T16:06:44.720665+0000","last_became_peered":"2024-03-18T16:06:44.720665+0000","last_unstale":"2024-03-18T16:08:47.886025+0000","last_undegraded":"2024-03-18T16:08:47.886025+0000","last_fullsized":"2024-03-18T16:08:47.886025+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:58:34.180653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.881862+0000","last_change":"2024-03-18T16:06:49.022919+0000","last_active":"2024-03-18T16:08:47.881862+0000","last_peered":"2024-03-18T16:08:47.881862+0000","last_clean":"2024-03-18T16:08:47.881862+0000","last_became_active":"2024-03-18T16:06:49.022514+0000","last_became_peered":"2024-03-18T16:06:49.022514+0000","last_unstale":"2024-03-18T16:08:47.881862+0000","last_undegraded":"2024-03-18T16:08:47.881862+0000","last_fullsized":"2024-03-18T16:08:47.881862+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:46:39.510181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.961422+0000","last_change":"2024-03-18T16:06:43.713025+0000","last_active":"2024-03-18T16:08:47.961422+0000","last_peered":"2024-03-18T16:08:47.961422+0000","last_clean":"2024-03-18T16:08:47.961422+0000","last_became_active":"2024-03-18T16:06:43.711469+0000","last_became_peered":"2024-03-18T16:06:43.711469+0000","last_unstale":"2024-03-18T16:08:47.961422+0000","last_undegraded":"2024-03-18T16:08:47.961422+0000","last_fullsized":"2024-03-18T16:08:47.961422+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:30:41.193229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883163+0000","last_change":"2024-03-18T16:06:44.723953+0000","last_active":"2024-03-18T16:08:47.883163+0000","last_peered":"2024-03-18T16:08:47.883163+0000","last_clean":"2024-03-18T16:08:47.883163+0000","last_became_active":"2024-03-18T16:06:44.723266+0000","last_became_peered":"2024-03-18T16:06:44.723266+0000","last_unstale":"2024-03-18T16:08:47.883163+0000","last_undegraded":"2024-03-18T16:08:47.883163+0000","last_fullsized":"2024-03-18T16:08:47.883163+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:41:10.067297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.961683+0000","last_change":"2024-03-18T16:06:49.039472+0000","last_active":"2024-03-18T16:08:47.961683+0000","last_peered":"2024-03-18T16:08:47.961683+0000","last_clean":"2024-03-18T16:08:47.961683+0000","last_became_active":"2024-03-18T16:06:49.038889+0000","last_became_peered":"2024-03-18T16:06:49.038889+0000","last_unstale":"2024-03-18T16:08:47.961683+0000","last_undegraded":"2024-03-18T16:08:47.961683+0000","last_fullsized":"2024-03-18T16:08:47.961683+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:03:05.345315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.885773+0000","last_change":"2024-03-18T16:06:43.705873+0000","last_active":"2024-03-18T16:08:47.885773+0000","last_peered":"2024-03-18T16:08:47.885773+0000","last_clean":"2024-03-18T16:08:47.885773+0000","last_became_active":"2024-03-18T16:06:43.705548+0000","last_became_peered":"2024-03-18T16:06:43.705548+0000","last_unstale":"2024-03-18T16:08:47.885773+0000","last_undegraded":"2024-03-18T16:08:47.885773+0000","last_fullsized":"2024-03-18T16:08:47.885773+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:52:03.790888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.882690+0000","last_change":"2024-03-18T16:06:44.718526+0000","last_active":"2024-03-18T16:08:47.882690+0000","last_peered":"2024-03-18T16:08:47.882690+0000","last_clean":"2024-03-18T16:08:47.882690+0000","last_became_active":"2024-03-18T16:06:44.717266+0000","last_became_peered":"2024-03-18T16:06:44.717266+0000","last_unstale":"2024-03-18T16:08:47.882690+0000","last_undegraded":"2024-03-18T16:08:47.882690+0000","last_fullsized":"2024-03-18T16:08:47.882690+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:55:09.340354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.885083+0000","last_change":"2024-03-18T16:06:49.021971+0000","last_active":"2024-03-18T16:08:47.885083+0000","last_peered":"2024-03-18T16:08:47.885083+0000","last_clean":"2024-03-18T16:08:47.885083+0000","last_became_active":"2024-03-18T16:06:49.021577+0000","last_became_peered":"2024-03-18T16:06:49.021577+0000","last_unstale":"2024-03-18T16:08:47.885083+0000","last_undegraded":"2024-03-18T16:08:47.885083+0000","last_fullsized":"2024-03-18T16:08:47.885083+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:47:44.768559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"93'6","reported_seq":357,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:09:59.148338+0000","last_change":"2024-03-18T16:06:43.705636+0000","last_active":"2024-03-18T16:09:59.148338+0000","last_peered":"2024-03-18T16:09:59.148338+0000","last_clean":"2024-03-18T16:09:59.148338+0000","last_became_active":"2024-03-18T16:06:43.705268+0000","last_became_peered":"2024-03-18T16:06:43.705268+0000","last_unstale":"2024-03-18T16:09:59.148338+0000","last_undegraded":"2024-03-18T16:09:59.148338+0000","last_fullsized":"2024-03-18T16:09:59.148338+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:37:31.367130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":320,"num_read_kb":320,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.960576+0000","last_change":"2024-03-18T16:06:44.712237+0000","last_active":"2024-03-18T16:08:47.960576+0000","last_peered":"2024-03-18T16:08:47.960576+0000","last_clean":"2024-03-18T16:08:47.960576+0000","last_became_active":"2024-03-18T16:06:44.711677+0000","last_became_peered":"2024-03-18T16:06:44.711677+0000","last_unstale":"2024-03-18T16:08:47.960576+0000","last_undegraded":"2024-03-18T16:08:47.960576+0000","last_fullsized":"2024-03-18T16:08:47.960576+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:16:56.326962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.882199+0000","last_change":"2024-03-18T16:06:49.034578+0000","last_active":"2024-03-18T16:08:47.882199+0000","last_peered":"2024-03-18T16:08:47.882199+0000","last_clean":"2024-03-18T16:08:47.882199+0000","last_became_active":"2024-03-18T16:06:49.031099+0000","last_became_peered":"2024-03-18T16:06:49.031099+0000","last_unstale":"2024-03-18T16:08:47.882199+0000","last_undegraded":"2024-03-18T16:08:47.882199+0000","last_fullsized":"2024-03-18T16:08:47.882199+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:11:08.042550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.961212+0000","last_change":"2024-03-18T16:06:43.708151+0000","last_active":"2024-03-18T16:08:47.961212+0000","last_peered":"2024-03-18T16:08:47.961212+0000","last_clean":"2024-03-18T16:08:47.961212+0000","last_became_active":"2024-03-18T16:06:43.707810+0000","last_became_peered":"2024-03-18T16:06:43.707810+0000","last_unstale":"2024-03-18T16:08:47.961212+0000","last_undegraded":"2024-03-18T16:08:47.961212+0000","last_fullsized":"2024-03-18T16:08:47.961212+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:30:35.745772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.962204+0000","last_change":"2024-03-18T16:06:44.729777+0000","last_active":"2024-03-18T16:08:47.962204+0000","last_peered":"2024-03-18T16:08:47.962204+0000","last_clean":"2024-03-18T16:08:47.962204+0000","last_became_active":"2024-03-18T16:06:44.729244+0000","last_became_peered":"2024-03-18T16:06:44.729244+0000","last_unstale":"2024-03-18T16:08:47.962204+0000","last_undegraded":"2024-03-18T16:08:47.962204+0000","last_fullsized":"2024-03-18T16:08:47.962204+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:52:05.420443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883653+0000","last_change":"2024-03-18T16:07:47.355131+0000","last_active":"2024-03-18T16:08:47.883653+0000","last_peered":"2024-03-18T16:08:47.883653+0000","last_clean":"2024-03-18T16:08:47.883653+0000","last_became_active":"2024-03-18T16:07:47.354608+0000","last_became_peered":"2024-03-18T16:07:47.354608+0000","last_unstale":"2024-03-18T16:08:47.883653+0000","last_undegraded":"2024-03-18T16:08:47.883653+0000","last_fullsized":"2024-03-18T16:08:47.883653+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:12:57.979168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.886234+0000","last_change":"2024-03-18T16:06:43.715280+0000","last_active":"2024-03-18T16:08:47.886234+0000","last_peered":"2024-03-18T16:08:47.886234+0000","last_clean":"2024-03-18T16:08:47.886234+0000","last_became_active":"2024-03-18T16:06:43.714936+0000","last_became_peered":"2024-03-18T16:06:43.714936+0000","last_unstale":"2024-03-18T16:08:47.886234+0000","last_undegraded":"2024-03-18T16:08:47.886234+0000","last_fullsized":"2024-03-18T16:08:47.886234+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:07:38.746932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.960603+0000","last_change":"2024-03-18T16:06:44.709004+0000","last_active":"2024-03-18T16:08:47.960603+0000","last_peered":"2024-03-18T16:08:47.960603+0000","last_clean":"2024-03-18T16:08:47.960603+0000","last_became_active":"2024-03-18T16:06:44.708574+0000","last_became_peered":"2024-03-18T16:06:44.708574+0000","last_unstale":"2024-03-18T16:08:47.960603+0000","last_undegraded":"2024-03-18T16:08:47.960603+0000","last_fullsized":"2024-03-18T16:08:47.960603+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:02:50.982110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883055+0000","last_change":"2024-03-18T16:06:49.029806+0000","last_active":"2024-03-18T16:08:47.883055+0000","last_peered":"2024-03-18T16:08:47.883055+0000","last_clean":"2024-03-18T16:08:47.883055+0000","last_became_active":"2024-03-18T16:06:49.029404+0000","last_became_peered":"2024-03-18T16:06:49.029404+0000","last_unstale":"2024-03-18T16:08:47.883055+0000","last_undegraded":"2024-03-18T16:08:47.883055+0000","last_fullsized":"2024-03-18T16:08:47.883055+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:38:37.706620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"93'40","reported_seq":402,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:09:59.149498+0000","last_change":"2024-03-18T16:06:43.702910+0000","last_active":"2024-03-18T16:09:59.149498+0000","last_peered":"2024-03-18T16:09:59.149498+0000","last_clean":"2024-03-18T16:09:59.149498+0000","last_became_active":"2024-03-18T16:06:43.702557+0000","last_became_peered":"2024-03-18T16:06:43.702557+0000","last_unstale":"2024-03-18T16:09:59.149498+0000","last_undegraded":"2024-03-18T16:09:59.149498+0000","last_fullsized":"2024-03-18T16:09:59.149498+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:55:15.362987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66365,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":331,"num_read_kb":231,"num_write":42,"num_write_kb":85,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.884961+0000","last_change":"2024-03-18T16:06:44.709075+0000","last_active":"2024-03-18T16:08:47.884961+0000","last_peered":"2024-03-18T16:08:47.884961+0000","last_clean":"2024-03-18T16:08:47.884961+0000","last_became_active":"2024-03-18T16:06:44.708366+0000","last_became_peered":"2024-03-18T16:06:44.708366+0000","last_unstale":"2024-03-18T16:08:47.884961+0000","last_undegraded":"2024-03-18T16:08:47.884961+0000","last_fullsized":"2024-03-18T16:08:47.884961+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:44:18.486385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.886444+0000","last_change":"2024-03-18T16:06:49.037380+0000","last_active":"2024-03-18T16:08:47.886444+0000","last_peered":"2024-03-18T16:08:47.886444+0000","last_clean":"2024-03-18T16:08:47.886444+0000","last_became_active":"2024-03-18T16:06:49.037021+0000","last_became_peered":"2024-03-18T16:06:49.037021+0000","last_unstale":"2024-03-18T16:08:47.886444+0000","last_undegraded":"2024-03-18T16:08:47.886444+0000","last_fullsized":"2024-03-18T16:08:47.886444+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:43:03.482631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.896395+0000","last_change":"2024-03-18T16:08:47.895964+0000","last_active":"2024-03-18T16:08:47.896395+0000","last_peered":"2024-03-18T16:08:47.896395+0000","last_clean":"2024-03-18T16:08:47.896395+0000","last_became_active":"2024-03-18T16:08:47.886729+0000","last_became_peered":"2024-03-18T16:08:47.886729+0000","last_unstale":"2024-03-18T16:08:47.896395+0000","last_undegraded":"2024-03-18T16:08:47.896395+0000","last_fullsized":"2024-03-18T16:08:47.896395+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:58:02.557326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":68,"num_keys_recovered":10,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.962072+0000","last_change":"2024-03-18T16:06:44.727204+0000","last_active":"2024-03-18T16:08:47.962072+0000","last_peered":"2024-03-18T16:08:47.962072+0000","last_clean":"2024-03-18T16:08:47.962072+0000","last_became_active":"2024-03-18T16:06:44.726775+0000","last_became_peered":"2024-03-18T16:06:44.726775+0000","last_unstale":"2024-03-18T16:08:47.962072+0000","last_undegraded":"2024-03-18T16:08:47.962072+0000","last_fullsized":"2024-03-18T16:08:47.962072+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:29:44.842972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.881484+0000","last_change":"2024-03-18T16:06:49.022508+0000","last_active":"2024-03-18T16:08:47.881484+0000","last_peered":"2024-03-18T16:08:47.881484+0000","last_clean":"2024-03-18T16:08:47.881484+0000","last_became_active":"2024-03-18T16:06:49.022017+0000","last_became_peered":"2024-03-18T16:06:49.022017+0000","last_unstale":"2024-03-18T16:08:47.881484+0000","last_undegraded":"2024-03-18T16:08:47.881484+0000","last_fullsized":"2024-03-18T16:08:47.881484+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:52:45.892880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.885883+0000","last_change":"2024-03-18T16:06:43.719618+0000","last_active":"2024-03-18T16:08:47.885883+0000","last_peered":"2024-03-18T16:08:47.885883+0000","last_clean":"2024-03-18T16:08:47.885883+0000","last_became_active":"2024-03-18T16:06:43.719161+0000","last_became_peered":"2024-03-18T16:06:43.719161+0000","last_unstale":"2024-03-18T16:08:47.885883+0000","last_undegraded":"2024-03-18T16:08:47.885883+0000","last_fullsized":"2024-03-18T16:08:47.885883+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:39:53.077878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.960915+0000","last_change":"2024-03-18T16:06:44.717829+0000","last_active":"2024-03-18T16:08:47.960915+0000","last_peered":"2024-03-18T16:08:47.960915+0000","last_clean":"2024-03-18T16:08:47.960915+0000","last_became_active":"2024-03-18T16:06:44.717473+0000","last_became_peered":"2024-03-18T16:06:44.717473+0000","last_unstale":"2024-03-18T16:08:47.960915+0000","last_undegraded":"2024-03-18T16:08:47.960915+0000","last_fullsized":"2024-03-18T16:08:47.960915+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:10:00.183613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.884360+0000","last_change":"2024-03-18T16:06:49.036766+0000","last_active":"2024-03-18T16:08:47.884360+0000","last_peered":"2024-03-18T16:08:47.884360+0000","last_clean":"2024-03-18T16:08:47.884360+0000","last_became_active":"2024-03-18T16:06:49.034617+0000","last_became_peered":"2024-03-18T16:06:49.034617+0000","last_unstale":"2024-03-18T16:08:47.884360+0000","last_undegraded":"2024-03-18T16:08:47.884360+0000","last_fullsized":"2024-03-18T16:08:47.884360+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:18:59.033781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'5","reported_seq":196,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:09:56.221145+0000","last_change":"2024-03-18T16:06:43.702452+0000","last_active":"2024-03-18T16:09:56.221145+0000","last_peered":"2024-03-18T16:09:56.221145+0000","last_clean":"2024-03-18T16:09:56.221145+0000","last_became_active":"2024-03-18T16:06:43.702141+0000","last_became_peered":"2024-03-18T16:06:43.702141+0000","last_unstale":"2024-03-18T16:09:56.221145+0000","last_undegraded":"2024-03-18T16:09:56.221145+0000","last_fullsized":"2024-03-18T16:09:56.221145+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:44:05.366101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":161,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.960175+0000","last_change":"2024-03-18T16:06:44.711403+0000","last_active":"2024-03-18T16:08:47.960175+0000","last_peered":"2024-03-18T16:08:47.960175+0000","last_clean":"2024-03-18T16:08:47.960175+0000","last_became_active":"2024-03-18T16:06:44.710654+0000","last_became_peered":"2024-03-18T16:06:44.710654+0000","last_unstale":"2024-03-18T16:08:47.960175+0000","last_undegraded":"2024-03-18T16:08:47.960175+0000","last_fullsized":"2024-03-18T16:08:47.960175+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:21:43.633274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.962799+0000","last_change":"2024-03-18T16:06:49.039192+0000","last_active":"2024-03-18T16:08:47.962799+0000","last_peered":"2024-03-18T16:08:47.962799+0000","last_clean":"2024-03-18T16:08:47.962799+0000","last_became_active":"2024-03-18T16:06:49.038421+0000","last_became_peered":"2024-03-18T16:06:49.038421+0000","last_unstale":"2024-03-18T16:08:47.962799+0000","last_undegraded":"2024-03-18T16:08:47.962799+0000","last_fullsized":"2024-03-18T16:08:47.962799+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:54:56.944209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":49,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.884583+0000","last_change":"2024-03-18T16:06:43.712523+0000","last_active":"2024-03-18T16:08:47.884583+0000","last_peered":"2024-03-18T16:08:47.884583+0000","last_clean":"2024-03-18T16:08:47.884583+0000","last_became_active":"2024-03-18T16:06:43.712151+0000","last_became_peered":"2024-03-18T16:06:43.712151+0000","last_unstale":"2024-03-18T16:08:47.884583+0000","last_undegraded":"2024-03-18T16:08:47.884583+0000","last_fullsized":"2024-03-18T16:08:47.884583+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:17:19.651293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.884896+0000","last_change":"2024-03-18T16:07:47.355115+0000","last_active":"2024-03-18T16:08:47.884896+0000","last_peered":"2024-03-18T16:08:47.884896+0000","last_clean":"2024-03-18T16:08:47.884896+0000","last_became_active":"2024-03-18T16:07:47.354744+0000","last_became_peered":"2024-03-18T16:07:47.354744+0000","last_unstale":"2024-03-18T16:08:47.884896+0000","last_undegraded":"2024-03-18T16:08:47.884896+0000","last_fullsized":"2024-03-18T16:08:47.884896+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:14:27.644891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.884782+0000","last_change":"2024-03-18T16:06:49.028943+0000","last_active":"2024-03-18T16:08:47.884782+0000","last_peered":"2024-03-18T16:08:47.884782+0000","last_clean":"2024-03-18T16:08:47.884782+0000","last_became_active":"2024-03-18T16:06:49.028453+0000","last_became_peered":"2024-03-18T16:06:49.028453+0000","last_unstale":"2024-03-18T16:08:47.884782+0000","last_undegraded":"2024-03-18T16:08:47.884782+0000","last_fullsized":"2024-03-18T16:08:47.884782+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:16:30.318649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":17,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.899433+0000","last_change":"2024-03-18T16:08:47.899048+0000","last_active":"2024-03-18T16:08:47.899433+0000","last_peered":"2024-03-18T16:08:47.899433+0000","last_clean":"2024-03-18T16:08:47.899433+0000","last_became_active":"2024-03-18T16:08:47.887283+0000","last_became_peered":"2024-03-18T16:08:47.887283+0000","last_unstale":"2024-03-18T16:08:47.899433+0000","last_undegraded":"2024-03-18T16:08:47.899433+0000","last_fullsized":"2024-03-18T16:08:47.899433+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:17:17.924210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.881500+0000","last_change":"2024-03-18T16:06:44.716284+0000","last_active":"2024-03-18T16:08:47.881500+0000","last_peered":"2024-03-18T16:08:47.881500+0000","last_clean":"2024-03-18T16:08:47.881500+0000","last_became_active":"2024-03-18T16:06:44.715911+0000","last_became_peered":"2024-03-18T16:06:44.715911+0000","last_unstale":"2024-03-18T16:08:47.881500+0000","last_undegraded":"2024-03-18T16:08:47.881500+0000","last_fullsized":"2024-03-18T16:08:47.881500+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:48:34.839514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.885009+0000","last_change":"2024-03-18T16:06:49.059061+0000","last_active":"2024-03-18T16:08:47.885009+0000","last_peered":"2024-03-18T16:08:47.885009+0000","last_clean":"2024-03-18T16:08:47.885009+0000","last_became_active":"2024-03-18T16:06:49.058619+0000","last_became_peered":"2024-03-18T16:06:49.058619+0000","last_unstale":"2024-03-18T16:08:47.885009+0000","last_undegraded":"2024-03-18T16:08:47.885009+0000","last_fullsized":"2024-03-18T16:08:47.885009+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:44:52.970785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883689+0000","last_change":"2024-03-18T16:06:43.715766+0000","last_active":"2024-03-18T16:08:47.883689+0000","last_peered":"2024-03-18T16:08:47.883689+0000","last_clean":"2024-03-18T16:08:47.883689+0000","last_became_active":"2024-03-18T16:06:43.715380+0000","last_became_peered":"2024-03-18T16:06:43.715380+0000","last_unstale":"2024-03-18T16:08:47.883689+0000","last_undegraded":"2024-03-18T16:08:47.883689+0000","last_fullsized":"2024-03-18T16:08:47.883689+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:13:44.195934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.880175+0000","last_change":"2024-03-18T16:06:44.728150+0000","last_active":"2024-03-18T16:08:47.880175+0000","last_peered":"2024-03-18T16:08:47.880175+0000","last_clean":"2024-03-18T16:08:47.880175+0000","last_became_active":"2024-03-18T16:06:44.727769+0000","last_became_peered":"2024-03-18T16:06:44.727769+0000","last_unstale":"2024-03-18T16:08:47.880175+0000","last_undegraded":"2024-03-18T16:08:47.880175+0000","last_fullsized":"2024-03-18T16:08:47.880175+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:28:16.023225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.961752+0000","last_change":"2024-03-18T16:06:49.033671+0000","last_active":"2024-03-18T16:08:47.961752+0000","last_peered":"2024-03-18T16:08:47.961752+0000","last_clean":"2024-03-18T16:08:47.961752+0000","last_became_active":"2024-03-18T16:06:49.033046+0000","last_became_peered":"2024-03-18T16:06:49.033046+0000","last_unstale":"2024-03-18T16:08:47.961752+0000","last_undegraded":"2024-03-18T16:08:47.961752+0000","last_fullsized":"2024-03-18T16:08:47.961752+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:26:37.020521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'5","reported_seq":195,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:09:56.309256+0000","last_change":"2024-03-18T16:06:43.712636+0000","last_active":"2024-03-18T16:09:56.309256+0000","last_peered":"2024-03-18T16:09:56.309256+0000","last_clean":"2024-03-18T16:09:56.309256+0000","last_became_active":"2024-03-18T16:06:43.712300+0000","last_became_peered":"2024-03-18T16:06:43.712300+0000","last_unstale":"2024-03-18T16:09:56.309256+0000","last_undegraded":"2024-03-18T16:09:56.309256+0000","last_fullsized":"2024-03-18T16:09:56.309256+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:48:20.386640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":158,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.880029+0000","last_change":"2024-03-18T16:06:44.731583+0000","last_active":"2024-03-18T16:08:47.880029+0000","last_peered":"2024-03-18T16:08:47.880029+0000","last_clean":"2024-03-18T16:08:47.880029+0000","last_became_active":"2024-03-18T16:06:44.731183+0000","last_became_peered":"2024-03-18T16:06:44.731183+0000","last_unstale":"2024-03-18T16:08:47.880029+0000","last_undegraded":"2024-03-18T16:08:47.880029+0000","last_fullsized":"2024-03-18T16:08:47.880029+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:23:05.098101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.962348+0000","last_change":"2024-03-18T16:06:49.060130+0000","last_active":"2024-03-18T16:08:47.962348+0000","last_peered":"2024-03-18T16:08:47.962348+0000","last_clean":"2024-03-18T16:08:47.962348+0000","last_became_active":"2024-03-18T16:06:49.059109+0000","last_became_peered":"2024-03-18T16:06:49.059109+0000","last_unstale":"2024-03-18T16:08:47.962348+0000","last_undegraded":"2024-03-18T16:08:47.962348+0000","last_fullsized":"2024-03-18T16:08:47.962348+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:41:40.008047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.885977+0000","last_change":"2024-03-18T16:06:44.705793+0000","last_active":"2024-03-18T16:08:47.885977+0000","last_peered":"2024-03-18T16:08:47.885977+0000","last_clean":"2024-03-18T16:08:47.885977+0000","last_became_active":"2024-03-18T16:06:44.705426+0000","last_became_peered":"2024-03-18T16:06:44.705426+0000","last_unstale":"2024-03-18T16:08:47.885977+0000","last_undegraded":"2024-03-18T16:08:47.885977+0000","last_fullsized":"2024-03-18T16:08:47.885977+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:14:31.198829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'5","reported_seq":194,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:09:55.219092+0000","last_change":"2024-03-18T16:06:43.721057+0000","last_active":"2024-03-18T16:09:55.219092+0000","last_peered":"2024-03-18T16:09:55.219092+0000","last_clean":"2024-03-18T16:09:55.219092+0000","last_became_active":"2024-03-18T16:06:43.720661+0000","last_became_peered":"2024-03-18T16:06:43.720661+0000","last_unstale":"2024-03-18T16:09:55.219092+0000","last_undegraded":"2024-03-18T16:09:55.219092+0000","last_fullsized":"2024-03-18T16:09:55.219092+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:08:18.050511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":158,"num_read_kb":158,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.961318+0000","last_change":"2024-03-18T16:07:47.361447+0000","last_active":"2024-03-18T16:08:47.961318+0000","last_peered":"2024-03-18T16:08:47.961318+0000","last_clean":"2024-03-18T16:08:47.961318+0000","last_became_active":"2024-03-18T16:07:47.353684+0000","last_became_peered":"2024-03-18T16:07:47.353684+0000","last_unstale":"2024-03-18T16:08:47.961318+0000","last_undegraded":"2024-03-18T16:08:47.961318+0000","last_fullsized":"2024-03-18T16:08:47.961318+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:59:53.025620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,10],"acting":[7,5,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.885471+0000","last_change":"2024-03-18T16:06:44.725555+0000","last_active":"2024-03-18T16:08:47.885471+0000","last_peered":"2024-03-18T16:08:47.885471+0000","last_clean":"2024-03-18T16:08:47.885471+0000","last_became_active":"2024-03-18T16:06:44.719880+0000","last_became_peered":"2024-03-18T16:06:44.719880+0000","last_unstale":"2024-03-18T16:08:47.885471+0000","last_undegraded":"2024-03-18T16:08:47.885471+0000","last_fullsized":"2024-03-18T16:08:47.885471+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:45:17.830876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883578+0000","last_change":"2024-03-18T16:06:43.700814+0000","last_active":"2024-03-18T16:08:47.883578+0000","last_peered":"2024-03-18T16:08:47.883578+0000","last_clean":"2024-03-18T16:08:47.883578+0000","last_became_active":"2024-03-18T16:06:43.698902+0000","last_became_peered":"2024-03-18T16:06:43.698902+0000","last_unstale":"2024-03-18T16:08:47.883578+0000","last_undegraded":"2024-03-18T16:08:47.883578+0000","last_fullsized":"2024-03-18T16:08:47.883578+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:20:20.100033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":354,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.960035+0000","last_change":"2024-03-18T16:05:22.379397+0000","last_active":"2024-03-18T16:08:47.960035+0000","last_peered":"2024-03-18T16:08:47.960035+0000","last_clean":"2024-03-18T16:08:47.960035+0000","last_became_active":"2024-03-18T16:05:22.378830+0000","last_became_peered":"2024-03-18T16:05:22.378830+0000","last_unstale":"2024-03-18T16:08:47.960035+0000","last_undegraded":"2024-03-18T16:08:47.960035+0000","last_fullsized":"2024-03-18T16:08:47.960035+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:01:30.048984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:01:30.048984+0000","last_clean_scrub_stamp":"2024-03-18T16:01:30.048984+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:35:45.796495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.885230+0000","last_change":"2024-03-18T16:06:49.058931+0000","last_active":"2024-03-18T16:08:47.885230+0000","last_peered":"2024-03-18T16:08:47.885230+0000","last_clean":"2024-03-18T16:08:47.885230+0000","last_became_active":"2024-03-18T16:06:49.058497+0000","last_became_peered":"2024-03-18T16:06:49.058497+0000","last_unstale":"2024-03-18T16:08:47.885230+0000","last_undegraded":"2024-03-18T16:08:47.885230+0000","last_fullsized":"2024-03-18T16:08:47.885230+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:41:51.874582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.961259+0000","last_change":"2024-03-18T16:06:44.710332+0000","last_active":"2024-03-18T16:08:47.961259+0000","last_peered":"2024-03-18T16:08:47.961259+0000","last_clean":"2024-03-18T16:08:47.961259+0000","last_became_active":"2024-03-18T16:06:44.709952+0000","last_became_peered":"2024-03-18T16:06:44.709952+0000","last_unstale":"2024-03-18T16:08:47.961259+0000","last_undegraded":"2024-03-18T16:08:47.961259+0000","last_fullsized":"2024-03-18T16:08:47.961259+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:24:26.653511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.960126+0000","last_change":"2024-03-18T16:06:43.708265+0000","last_active":"2024-03-18T16:08:47.960126+0000","last_peered":"2024-03-18T16:08:47.960126+0000","last_clean":"2024-03-18T16:08:47.960126+0000","last_became_active":"2024-03-18T16:06:43.707466+0000","last_became_peered":"2024-03-18T16:06:43.707466+0000","last_unstale":"2024-03-18T16:08:47.960126+0000","last_undegraded":"2024-03-18T16:08:47.960126+0000","last_fullsized":"2024-03-18T16:08:47.960126+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:45:40.065396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.882033+0000","last_change":"2024-03-18T16:06:49.022401+0000","last_active":"2024-03-18T16:08:47.882033+0000","last_peered":"2024-03-18T16:08:47.882033+0000","last_clean":"2024-03-18T16:08:47.882033+0000","last_became_active":"2024-03-18T16:06:49.021966+0000","last_became_peered":"2024-03-18T16:06:49.021966+0000","last_unstale":"2024-03-18T16:08:47.882033+0000","last_undegraded":"2024-03-18T16:08:47.882033+0000","last_fullsized":"2024-03-18T16:08:47.882033+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:04:12.434969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.885248+0000","last_change":"2024-03-18T16:06:43.713249+0000","last_active":"2024-03-18T16:08:47.885248+0000","last_peered":"2024-03-18T16:08:47.885248+0000","last_clean":"2024-03-18T16:08:47.885248+0000","last_became_active":"2024-03-18T16:06:43.712897+0000","last_became_peered":"2024-03-18T16:06:43.712897+0000","last_unstale":"2024-03-18T16:08:47.885248+0000","last_undegraded":"2024-03-18T16:08:47.885248+0000","last_fullsized":"2024-03-18T16:08:47.885248+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:20:04.642568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.961929+0000","last_change":"2024-03-18T16:06:44.710034+0000","last_active":"2024-03-18T16:08:47.961929+0000","last_peered":"2024-03-18T16:08:47.961929+0000","last_clean":"2024-03-18T16:08:47.961929+0000","last_became_active":"2024-03-18T16:06:44.708457+0000","last_became_peered":"2024-03-18T16:06:44.708457+0000","last_unstale":"2024-03-18T16:08:47.961929+0000","last_undegraded":"2024-03-18T16:08:47.961929+0000","last_fullsized":"2024-03-18T16:08:47.961929+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:40:00.504382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883057+0000","last_change":"2024-03-18T16:06:49.059345+0000","last_active":"2024-03-18T16:08:47.883057+0000","last_peered":"2024-03-18T16:08:47.883057+0000","last_clean":"2024-03-18T16:08:47.883057+0000","last_became_active":"2024-03-18T16:06:49.058675+0000","last_became_peered":"2024-03-18T16:06:49.058675+0000","last_unstale":"2024-03-18T16:08:47.883057+0000","last_undegraded":"2024-03-18T16:08:47.883057+0000","last_fullsized":"2024-03-18T16:08:47.883057+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:03:33.730074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"93'15","reported_seq":47,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:56.879106+0000","last_change":"2024-03-18T16:06:43.724589+0000","last_active":"2024-03-18T16:08:56.879106+0000","last_peered":"2024-03-18T16:08:56.879106+0000","last_clean":"2024-03-18T16:08:56.879106+0000","last_became_active":"2024-03-18T16:06:43.724125+0000","last_became_peered":"2024-03-18T16:06:43.724125+0000","last_unstale":"2024-03-18T16:08:56.879106+0000","last_undegraded":"2024-03-18T16:08:56.879106+0000","last_fullsized":"2024-03-18T16:08:56.879106+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:08:34.921118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.886205+0000","last_change":"2024-03-18T16:06:44.727162+0000","last_active":"2024-03-18T16:08:47.886205+0000","last_peered":"2024-03-18T16:08:47.886205+0000","last_clean":"2024-03-18T16:08:47.886205+0000","last_became_active":"2024-03-18T16:06:44.726552+0000","last_became_peered":"2024-03-18T16:06:44.726552+0000","last_unstale":"2024-03-18T16:08:47.886205+0000","last_undegraded":"2024-03-18T16:08:47.886205+0000","last_fullsized":"2024-03-18T16:08:47.886205+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:36:57.974999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883803+0000","last_change":"2024-03-18T16:06:49.008434+0000","last_active":"2024-03-18T16:08:47.883803+0000","last_peered":"2024-03-18T16:08:47.883803+0000","last_clean":"2024-03-18T16:08:47.883803+0000","last_became_active":"2024-03-18T16:06:49.007944+0000","last_became_peered":"2024-03-18T16:06:49.007944+0000","last_unstale":"2024-03-18T16:08:47.883803+0000","last_undegraded":"2024-03-18T16:08:47.883803+0000","last_fullsized":"2024-03-18T16:08:47.883803+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:01:24.450371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883928+0000","last_change":"2024-03-18T16:06:43.711797+0000","last_active":"2024-03-18T16:08:47.883928+0000","last_peered":"2024-03-18T16:08:47.883928+0000","last_clean":"2024-03-18T16:08:47.883928+0000","last_became_active":"2024-03-18T16:06:43.711493+0000","last_became_peered":"2024-03-18T16:06:43.711493+0000","last_unstale":"2024-03-18T16:08:47.883928+0000","last_undegraded":"2024-03-18T16:08:47.883928+0000","last_fullsized":"2024-03-18T16:08:47.883928+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:50:29.258166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.961906+0000","last_change":"2024-03-18T16:06:44.735910+0000","last_active":"2024-03-18T16:08:47.961906+0000","last_peered":"2024-03-18T16:08:47.961906+0000","last_clean":"2024-03-18T16:08:47.961906+0000","last_became_active":"2024-03-18T16:06:44.735504+0000","last_became_peered":"2024-03-18T16:06:44.735504+0000","last_unstale":"2024-03-18T16:08:47.961906+0000","last_undegraded":"2024-03-18T16:08:47.961906+0000","last_fullsized":"2024-03-18T16:08:47.961906+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:02:18.538217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883708+0000","last_change":"2024-03-18T16:06:49.059130+0000","last_active":"2024-03-18T16:08:47.883708+0000","last_peered":"2024-03-18T16:08:47.883708+0000","last_clean":"2024-03-18T16:08:47.883708+0000","last_became_active":"2024-03-18T16:06:49.058701+0000","last_became_peered":"2024-03-18T16:06:49.058701+0000","last_unstale":"2024-03-18T16:08:47.883708+0000","last_undegraded":"2024-03-18T16:08:47.883708+0000","last_fullsized":"2024-03-18T16:08:47.883708+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:54:52.842786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.959932+0000","last_change":"2024-03-18T16:06:43.714414+0000","last_active":"2024-03-18T16:08:47.959932+0000","last_peered":"2024-03-18T16:08:47.959932+0000","last_clean":"2024-03-18T16:08:47.959932+0000","last_became_active":"2024-03-18T16:06:43.714029+0000","last_became_peered":"2024-03-18T16:06:43.714029+0000","last_unstale":"2024-03-18T16:08:47.959932+0000","last_undegraded":"2024-03-18T16:08:47.959932+0000","last_fullsized":"2024-03-18T16:08:47.959932+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:16:39.060378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.884429+0000","last_change":"2024-03-18T16:06:44.707491+0000","last_active":"2024-03-18T16:08:47.884429+0000","last_peered":"2024-03-18T16:08:47.884429+0000","last_clean":"2024-03-18T16:08:47.884429+0000","last_became_active":"2024-03-18T16:06:44.707157+0000","last_became_peered":"2024-03-18T16:06:44.707157+0000","last_unstale":"2024-03-18T16:08:47.884429+0000","last_undegraded":"2024-03-18T16:08:47.884429+0000","last_fullsized":"2024-03-18T16:08:47.884429+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:19:08.132911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.885895+0000","last_change":"2024-03-18T16:07:47.411002+0000","last_active":"2024-03-18T16:08:47.885895+0000","last_peered":"2024-03-18T16:08:47.885895+0000","last_clean":"2024-03-18T16:08:47.885895+0000","last_became_active":"2024-03-18T16:07:47.410575+0000","last_became_peered":"2024-03-18T16:07:47.410575+0000","last_unstale":"2024-03-18T16:08:47.885895+0000","last_undegraded":"2024-03-18T16:08:47.885895+0000","last_fullsized":"2024-03-18T16:08:47.885895+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:37:25.311205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,2,5],"acting":[8,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.884825+0000","last_change":"2024-03-18T16:06:43.717755+0000","last_active":"2024-03-18T16:08:47.884825+0000","last_peered":"2024-03-18T16:08:47.884825+0000","last_clean":"2024-03-18T16:08:47.884825+0000","last_became_active":"2024-03-18T16:06:43.717278+0000","last_became_peered":"2024-03-18T16:06:43.717278+0000","last_unstale":"2024-03-18T16:08:47.884825+0000","last_undegraded":"2024-03-18T16:08:47.884825+0000","last_fullsized":"2024-03-18T16:08:47.884825+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:48:09.390659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.881317+0000","last_change":"2024-03-18T16:06:44.730858+0000","last_active":"2024-03-18T16:08:47.881317+0000","last_peered":"2024-03-18T16:08:47.881317+0000","last_clean":"2024-03-18T16:08:47.881317+0000","last_became_active":"2024-03-18T16:06:44.730500+0000","last_became_peered":"2024-03-18T16:06:44.730500+0000","last_unstale":"2024-03-18T16:08:47.881317+0000","last_undegraded":"2024-03-18T16:08:47.881317+0000","last_fullsized":"2024-03-18T16:08:47.881317+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:22:03.590834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883615+0000","last_change":"2024-03-18T16:06:49.014717+0000","last_active":"2024-03-18T16:08:47.883615+0000","last_peered":"2024-03-18T16:08:47.883615+0000","last_clean":"2024-03-18T16:08:47.883615+0000","last_became_active":"2024-03-18T16:06:49.014313+0000","last_became_peered":"2024-03-18T16:06:49.014313+0000","last_unstale":"2024-03-18T16:08:47.883615+0000","last_undegraded":"2024-03-18T16:08:47.883615+0000","last_fullsized":"2024-03-18T16:08:47.883615+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:06:12.168888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"93'4","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:53.939577+0000","last_change":"2024-03-18T16:06:43.719496+0000","last_active":"2024-03-18T16:08:53.939577+0000","last_peered":"2024-03-18T16:08:53.939577+0000","last_clean":"2024-03-18T16:08:53.939577+0000","last_became_active":"2024-03-18T16:06:43.719139+0000","last_became_peered":"2024-03-18T16:06:43.719139+0000","last_unstale":"2024-03-18T16:08:53.939577+0000","last_undegraded":"2024-03-18T16:08:53.939577+0000","last_fullsized":"2024-03-18T16:08:53.939577+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:10:01.250923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.886017+0000","last_change":"2024-03-18T16:06:44.737792+0000","last_active":"2024-03-18T16:08:47.886017+0000","last_peered":"2024-03-18T16:08:47.886017+0000","last_clean":"2024-03-18T16:08:47.886017+0000","last_became_active":"2024-03-18T16:06:44.736311+0000","last_became_peered":"2024-03-18T16:06:44.736311+0000","last_unstale":"2024-03-18T16:08:47.886017+0000","last_undegraded":"2024-03-18T16:08:47.886017+0000","last_fullsized":"2024-03-18T16:08:47.886017+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:43:56.110205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883670+0000","last_change":"2024-03-18T16:06:49.058893+0000","last_active":"2024-03-18T16:08:47.883670+0000","last_peered":"2024-03-18T16:08:47.883670+0000","last_clean":"2024-03-18T16:08:47.883670+0000","last_became_active":"2024-03-18T16:06:49.058477+0000","last_became_peered":"2024-03-18T16:06:49.058477+0000","last_unstale":"2024-03-18T16:08:47.883670+0000","last_undegraded":"2024-03-18T16:08:47.883670+0000","last_fullsized":"2024-03-18T16:08:47.883670+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:50:51.890348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.884612+0000","last_change":"2024-03-18T16:06:43.703073+0000","last_active":"2024-03-18T16:08:47.884612+0000","last_peered":"2024-03-18T16:08:47.884612+0000","last_clean":"2024-03-18T16:08:47.884612+0000","last_became_active":"2024-03-18T16:06:43.702708+0000","last_became_peered":"2024-03-18T16:06:43.702708+0000","last_unstale":"2024-03-18T16:08:47.884612+0000","last_undegraded":"2024-03-18T16:08:47.884612+0000","last_fullsized":"2024-03-18T16:08:47.884612+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:38:41.454458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.960757+0000","last_change":"2024-03-18T16:06:44.713622+0000","last_active":"2024-03-18T16:08:47.960757+0000","last_peered":"2024-03-18T16:08:47.960757+0000","last_clean":"2024-03-18T16:08:47.960757+0000","last_became_active":"2024-03-18T16:06:44.713160+0000","last_became_peered":"2024-03-18T16:06:44.713160+0000","last_unstale":"2024-03-18T16:08:47.960757+0000","last_undegraded":"2024-03-18T16:08:47.960757+0000","last_fullsized":"2024-03-18T16:08:47.960757+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:49:21.886543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.959925+0000","last_change":"2024-03-18T16:07:47.411958+0000","last_active":"2024-03-18T16:08:47.959925+0000","last_peered":"2024-03-18T16:08:47.959925+0000","last_clean":"2024-03-18T16:08:47.959925+0000","last_became_active":"2024-03-18T16:07:47.411433+0000","last_became_peered":"2024-03-18T16:07:47.411433+0000","last_unstale":"2024-03-18T16:08:47.959925+0000","last_undegraded":"2024-03-18T16:08:47.959925+0000","last_fullsized":"2024-03-18T16:08:47.959925+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:58:47.853897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6,7],"acting":[1,8,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"93'6","reported_seq":199,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:09:57.146786+0000","last_change":"2024-03-18T16:06:43.717161+0000","last_active":"2024-03-18T16:09:57.146786+0000","last_peered":"2024-03-18T16:09:57.146786+0000","last_clean":"2024-03-18T16:09:57.146786+0000","last_became_active":"2024-03-18T16:06:43.716657+0000","last_became_peered":"2024-03-18T16:06:43.716657+0000","last_unstale":"2024-03-18T16:09:57.146786+0000","last_undegraded":"2024-03-18T16:09:57.146786+0000","last_fullsized":"2024-03-18T16:09:57.146786+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:23:48.194779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1480,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":162,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.961120+0000","last_change":"2024-03-18T16:06:44.709777+0000","last_active":"2024-03-18T16:08:47.961120+0000","last_peered":"2024-03-18T16:08:47.961120+0000","last_clean":"2024-03-18T16:08:47.961120+0000","last_became_active":"2024-03-18T16:06:44.709409+0000","last_became_peered":"2024-03-18T16:06:44.709409+0000","last_unstale":"2024-03-18T16:08:47.961120+0000","last_undegraded":"2024-03-18T16:08:47.961120+0000","last_fullsized":"2024-03-18T16:08:47.961120+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:37:12.288262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.961539+0000","last_change":"2024-03-18T16:06:49.040872+0000","last_active":"2024-03-18T16:08:47.961539+0000","last_peered":"2024-03-18T16:08:47.961539+0000","last_clean":"2024-03-18T16:08:47.961539+0000","last_became_active":"2024-03-18T16:06:49.039807+0000","last_became_peered":"2024-03-18T16:06:49.039807+0000","last_unstale":"2024-03-18T16:08:47.961539+0000","last_undegraded":"2024-03-18T16:08:47.961539+0000","last_fullsized":"2024-03-18T16:08:47.961539+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:09:27.045450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"89'3","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.886254+0000","last_change":"2024-03-18T16:06:43.723675+0000","last_active":"2024-03-18T16:08:47.886254+0000","last_peered":"2024-03-18T16:08:47.886254+0000","last_clean":"2024-03-18T16:08:47.886254+0000","last_became_active":"2024-03-18T16:06:43.723334+0000","last_became_peered":"2024-03-18T16:06:43.723334+0000","last_unstale":"2024-03-18T16:08:47.886254+0000","last_undegraded":"2024-03-18T16:08:47.886254+0000","last_fullsized":"2024-03-18T16:08:47.886254+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:01:36.553362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.961094+0000","last_change":"2024-03-18T16:07:47.359134+0000","last_active":"2024-03-18T16:08:47.961094+0000","last_peered":"2024-03-18T16:08:47.961094+0000","last_clean":"2024-03-18T16:08:47.961094+0000","last_became_active":"2024-03-18T16:07:47.358692+0000","last_became_peered":"2024-03-18T16:07:47.358692+0000","last_unstale":"2024-03-18T16:08:47.961094+0000","last_undegraded":"2024-03-18T16:08:47.961094+0000","last_fullsized":"2024-03-18T16:08:47.961094+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:48:22.929034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.885552+0000","last_change":"2024-03-18T16:06:49.011981+0000","last_active":"2024-03-18T16:08:47.885552+0000","last_peered":"2024-03-18T16:08:47.885552+0000","last_clean":"2024-03-18T16:08:47.885552+0000","last_became_active":"2024-03-18T16:06:49.011621+0000","last_became_peered":"2024-03-18T16:06:49.011621+0000","last_unstale":"2024-03-18T16:08:47.885552+0000","last_undegraded":"2024-03-18T16:08:47.885552+0000","last_fullsized":"2024-03-18T16:08:47.885552+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:11:42.382721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.962696+0000","last_change":"2024-03-18T16:06:43.725404+0000","last_active":"2024-03-18T16:08:47.962696+0000","last_peered":"2024-03-18T16:08:47.962696+0000","last_clean":"2024-03-18T16:08:47.962696+0000","last_became_active":"2024-03-18T16:06:43.724117+0000","last_became_peered":"2024-03-18T16:06:43.724117+0000","last_unstale":"2024-03-18T16:08:47.962696+0000","last_undegraded":"2024-03-18T16:08:47.962696+0000","last_fullsized":"2024-03-18T16:08:47.962696+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:58:43.243070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.961302+0000","last_change":"2024-03-18T16:06:44.709979+0000","last_active":"2024-03-18T16:08:47.961302+0000","last_peered":"2024-03-18T16:08:47.961302+0000","last_clean":"2024-03-18T16:08:47.961302+0000","last_became_active":"2024-03-18T16:06:44.709591+0000","last_became_peered":"2024-03-18T16:06:44.709591+0000","last_unstale":"2024-03-18T16:08:47.961302+0000","last_undegraded":"2024-03-18T16:08:47.961302+0000","last_fullsized":"2024-03-18T16:08:47.961302+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:56:37.537248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.881778+0000","last_change":"2024-03-18T16:07:47.354636+0000","last_active":"2024-03-18T16:08:47.881778+0000","last_peered":"2024-03-18T16:08:47.881778+0000","last_clean":"2024-03-18T16:08:47.881778+0000","last_became_active":"2024-03-18T16:07:47.354219+0000","last_became_peered":"2024-03-18T16:07:47.354219+0000","last_unstale":"2024-03-18T16:08:47.881778+0000","last_undegraded":"2024-03-18T16:08:47.881778+0000","last_fullsized":"2024-03-18T16:08:47.881778+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:35:48.094404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,9],"acting":[11,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883718+0000","last_change":"2024-03-18T16:06:44.714836+0000","last_active":"2024-03-18T16:08:47.883718+0000","last_peered":"2024-03-18T16:08:47.883718+0000","last_clean":"2024-03-18T16:08:47.883718+0000","last_became_active":"2024-03-18T16:06:44.714454+0000","last_became_peered":"2024-03-18T16:06:44.714454+0000","last_unstale":"2024-03-18T16:08:47.883718+0000","last_undegraded":"2024-03-18T16:08:47.883718+0000","last_fullsized":"2024-03-18T16:08:47.883718+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:05:24.532238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883502+0000","last_change":"2024-03-18T16:06:44.733183+0000","last_active":"2024-03-18T16:08:47.883502+0000","last_peered":"2024-03-18T16:08:47.883502+0000","last_clean":"2024-03-18T16:08:47.883502+0000","last_became_active":"2024-03-18T16:06:44.732909+0000","last_became_peered":"2024-03-18T16:06:44.732909+0000","last_unstale":"2024-03-18T16:08:47.883502+0000","last_undegraded":"2024-03-18T16:08:47.883502+0000","last_fullsized":"2024-03-18T16:08:47.883502+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:28:07.636692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.961804+0000","last_change":"2024-03-18T16:06:44.731050+0000","last_active":"2024-03-18T16:08:47.961804+0000","last_peered":"2024-03-18T16:08:47.961804+0000","last_clean":"2024-03-18T16:08:47.961804+0000","last_became_active":"2024-03-18T16:06:44.730634+0000","last_became_peered":"2024-03-18T16:06:44.730634+0000","last_unstale":"2024-03-18T16:08:47.961804+0000","last_undegraded":"2024-03-18T16:08:47.961804+0000","last_fullsized":"2024-03-18T16:08:47.961804+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:07:18.619130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.962047+0000","last_change":"2024-03-18T16:06:44.735190+0000","last_active":"2024-03-18T16:08:47.962047+0000","last_peered":"2024-03-18T16:08:47.962047+0000","last_clean":"2024-03-18T16:08:47.962047+0000","last_became_active":"2024-03-18T16:06:44.734312+0000","last_became_peered":"2024-03-18T16:06:44.734312+0000","last_unstale":"2024-03-18T16:08:47.962047+0000","last_undegraded":"2024-03-18T16:08:47.962047+0000","last_fullsized":"2024-03-18T16:08:47.962047+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:59:50.209515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.881867+0000","last_change":"2024-03-18T16:06:44.705385+0000","last_active":"2024-03-18T16:08:47.881867+0000","last_peered":"2024-03-18T16:08:47.881867+0000","last_clean":"2024-03-18T16:08:47.881867+0000","last_became_active":"2024-03-18T16:06:44.705011+0000","last_became_peered":"2024-03-18T16:06:44.705011+0000","last_unstale":"2024-03-18T16:08:47.881867+0000","last_undegraded":"2024-03-18T16:08:47.881867+0000","last_fullsized":"2024-03-18T16:08:47.881867+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:50:02.152851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.885287+0000","last_change":"2024-03-18T16:06:44.709621+0000","last_active":"2024-03-18T16:08:47.885287+0000","last_peered":"2024-03-18T16:08:47.885287+0000","last_clean":"2024-03-18T16:08:47.885287+0000","last_became_active":"2024-03-18T16:06:44.709275+0000","last_became_peered":"2024-03-18T16:06:44.709275+0000","last_unstale":"2024-03-18T16:08:47.885287+0000","last_undegraded":"2024-03-18T16:08:47.885287+0000","last_fullsized":"2024-03-18T16:08:47.885287+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:41:17.346661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.886301+0000","last_change":"2024-03-18T16:06:44.719827+0000","last_active":"2024-03-18T16:08:47.886301+0000","last_peered":"2024-03-18T16:08:47.886301+0000","last_clean":"2024-03-18T16:08:47.886301+0000","last_became_active":"2024-03-18T16:06:44.719432+0000","last_became_peered":"2024-03-18T16:06:44.719432+0000","last_unstale":"2024-03-18T16:08:47.886301+0000","last_undegraded":"2024-03-18T16:08:47.886301+0000","last_fullsized":"2024-03-18T16:08:47.886301+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:13:42.344211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.885803+0000","last_change":"2024-03-18T16:06:44.714526+0000","last_active":"2024-03-18T16:08:47.885803+0000","last_peered":"2024-03-18T16:08:47.885803+0000","last_clean":"2024-03-18T16:08:47.885803+0000","last_became_active":"2024-03-18T16:06:44.714184+0000","last_became_peered":"2024-03-18T16:06:44.714184+0000","last_unstale":"2024-03-18T16:08:47.885803+0000","last_undegraded":"2024-03-18T16:08:47.885803+0000","last_fullsized":"2024-03-18T16:08:47.885803+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:31:07.917143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.884927+0000","last_change":"2024-03-18T16:06:44.730905+0000","last_active":"2024-03-18T16:08:47.884927+0000","last_peered":"2024-03-18T16:08:47.884927+0000","last_clean":"2024-03-18T16:08:47.884927+0000","last_became_active":"2024-03-18T16:06:44.729002+0000","last_became_peered":"2024-03-18T16:06:44.729002+0000","last_unstale":"2024-03-18T16:08:47.884927+0000","last_undegraded":"2024-03-18T16:08:47.884927+0000","last_fullsized":"2024-03-18T16:08:47.884927+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:49:18.867571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.960207+0000","last_change":"2024-03-18T16:06:44.727608+0000","last_active":"2024-03-18T16:08:47.960207+0000","last_peered":"2024-03-18T16:08:47.960207+0000","last_clean":"2024-03-18T16:08:47.960207+0000","last_became_active":"2024-03-18T16:06:44.727096+0000","last_became_peered":"2024-03-18T16:06:44.727096+0000","last_unstale":"2024-03-18T16:08:47.960207+0000","last_undegraded":"2024-03-18T16:08:47.960207+0000","last_fullsized":"2024-03-18T16:08:47.960207+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:08:20.767443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.962439+0000","last_change":"2024-03-18T16:06:44.731823+0000","last_active":"2024-03-18T16:08:47.962439+0000","last_peered":"2024-03-18T16:08:47.962439+0000","last_clean":"2024-03-18T16:08:47.962439+0000","last_became_active":"2024-03-18T16:06:44.731429+0000","last_became_peered":"2024-03-18T16:06:44.731429+0000","last_unstale":"2024-03-18T16:08:47.962439+0000","last_undegraded":"2024-03-18T16:08:47.962439+0000","last_fullsized":"2024-03-18T16:08:47.962439+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:36:28.752466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.881596+0000","last_change":"2024-03-18T16:06:44.714354+0000","last_active":"2024-03-18T16:08:47.881596+0000","last_peered":"2024-03-18T16:08:47.881596+0000","last_clean":"2024-03-18T16:08:47.881596+0000","last_became_active":"2024-03-18T16:06:44.713969+0000","last_became_peered":"2024-03-18T16:06:44.713969+0000","last_unstale":"2024-03-18T16:08:47.881596+0000","last_undegraded":"2024-03-18T16:08:47.881596+0000","last_fullsized":"2024-03-18T16:08:47.881596+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:18:49.502178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":73629,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1627,"num_read_kb":896,"num_write":258,"num_write_kb":154,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":90,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":581632,"data_stored":216573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":223,"ondisk_log_size":223,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383734,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36392,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708760,"statfs":{"total":95995035648,"available":95957770240,"internally_reserved":0,"allocated":1077248,"data_stored":520607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645370,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37064,"kb_used_data":1728,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708088,"statfs":{"total":95995035648,"available":95957082112,"internally_reserved":0,"allocated":1769472,"data_stored":1177484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808895,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1089536,"data_stored":520137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070535,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36392,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708760,"statfs":{"total":95995035648,"available":95957770240,"internally_reserved":0,"allocated":1081344,"data_stored":520493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332171,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28668,"kb_used_data":1644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716484,"statfs":{"total":95995035648,"available":95965679616,"internally_reserved":0,"allocated":1683456,"data_stored":1117153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495696,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32224,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712928,"statfs":{"total":95995035648,"available":95962038272,"internally_reserved":0,"allocated":1064960,"data_stored":515641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574956,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36440,"kb_used_data":1104,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708712,"statfs":{"total":95995035648,"available":95957721088,"internally_reserved":0,"allocated":1130496,"data_stored":578722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738483,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28088,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1089536,"data_stored":517407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411433,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36460,"kb_used_data":1124,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708692,"statfs":{"total":95995035648,"available":95957700608,"internally_reserved":0,"allocated":1150976,"data_stored":584030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182498,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1077248,"data_stored":515349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920861,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36392,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708760,"statfs":{"total":95995035648,"available":95957770240,"internally_reserved":0,"allocated":1073152,"data_stored":519715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757336,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36996,"kb_used_data":1652,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708156,"statfs":{"total":95995035648,"available":95957151744,"internally_reserved":0,"allocated":1691648,"data_stored":1112735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":63803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":69111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":68101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T16:10:01.366 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T16:10:01.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.625+0000 7fca12600640 1 -- 172.21.15.39:0/1600236813 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fca0c075820 msgr2=0x7fca0c073c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:01.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.625+0000 7fca12600640 1 --2- 172.21.15.39:0/1600236813 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fca0c075820 0x7fca0c073c70 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fc9fc009900 tx=0x7fc9fc02bc30 comp rx=0 tx=0).stop 2024-03-18T16:10:01.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.625+0000 7fca12600640 1 -- 172.21.15.39:0/1600236813 shutdown_connections 2024-03-18T16:10:01.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.625+0000 7fca12600640 1 --2- 172.21.15.39:0/1600236813 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fca0c1096b0 0x7fca0c10bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:01.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.625+0000 7fca12600640 1 --2- 172.21.15.39:0/1600236813 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca0c074240 0x7fca0c0746c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:01.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.625+0000 7fca12600640 1 --2- 172.21.15.39:0/1600236813 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fca0c075820 0x7fca0c073c70 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:01.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.625+0000 7fca12600640 1 -- 172.21.15.39:0/1600236813 >> 172.21.15.39:0/1600236813 conn(0x7fca0c0fcd80 msgr2=0x7fca0c0ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:01.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.625+0000 7fca12600640 1 -- 172.21.15.39:0/1600236813 shutdown_connections 2024-03-18T16:10:01.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.625+0000 7fca12600640 1 -- 172.21.15.39:0/1600236813 wait complete. 2024-03-18T16:10:01.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.626+0000 7fca12600640 1 Processor -- start 2024-03-18T16:10:01.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.626+0000 7fca12600640 1 -- start start 2024-03-18T16:10:01.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.626+0000 7fca12600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca0c074240 0x7fca0c199d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:01.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.626+0000 7fca12600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fca0c075820 0x7fca0c19a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:01.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.626+0000 7fca12600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fca0c1096b0 0x7fca0c1a12a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:01.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.626+0000 7fca12600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fca0c10f680 con 0x7fca0c074240 2024-03-18T16:10:01.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.626+0000 7fca12600640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fca0c10f500 con 0x7fca0c1096b0 2024-03-18T16:10:01.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.626+0000 7fca12600640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fca0c10f800 con 0x7fca0c075820 2024-03-18T16:10:01.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.627+0000 7fca0b400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca0c074240 0x7fca0c199d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:01.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.627+0000 7fca0aa00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fca0c075820 0x7fca0c19a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:01.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.627+0000 7fca0be00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fca0c1096b0 0x7fca0c1a12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:01.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.627+0000 7fca0b400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca0c074240 0x7fca0c199d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:55718/0 (socket says 172.21.15.39:55718) 2024-03-18T16:10:01.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.627+0000 7fca0b400640 1 -- 172.21.15.39:0/3024520638 learned_addr learned my addr 172.21.15.39:0/3024520638 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:10:01.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.627+0000 7fca0aa00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fca0c075820 0x7fca0c19a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:49788/0 (socket says 172.21.15.39:49788) 2024-03-18T16:10:01.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.627+0000 7fca0be00640 1 -- 172.21.15.39:0/3024520638 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fca0c075820 msgr2=0x7fca0c19a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:01.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.627+0000 7fca0be00640 1 --2- 172.21.15.39:0/3024520638 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fca0c075820 0x7fca0c19a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:01.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.627+0000 7fca0be00640 1 -- 172.21.15.39:0/3024520638 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca0c074240 msgr2=0x7fca0c199d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:01.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.627+0000 7fca0be00640 1 --2- 172.21.15.39:0/3024520638 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca0c074240 0x7fca0c199d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:01.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.627+0000 7fca0be00640 1 -- 172.21.15.39:0/3024520638 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca0c1a1940 con 0x7fca0c1096b0 2024-03-18T16:10:01.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.627+0000 7fca0aa00640 1 --2- 172.21.15.39:0/3024520638 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fca0c075820 0x7fca0c19a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:10:01.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.628+0000 7fca0be00640 1 --2- 172.21.15.39:0/3024520638 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fca0c1096b0 0x7fca0c1a12a0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fca0400ec50 tx=0x7fca0400c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:01.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.628+0000 7fca03e00640 1 -- 172.21.15.39:0/3024520638 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca04003b70 con 0x7fca0c1096b0 2024-03-18T16:10:01.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.628+0000 7fca03e00640 1 -- 172.21.15.39:0/3024520638 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fca04003d10 con 0x7fca0c1096b0 2024-03-18T16:10:01.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.628+0000 7fca12600640 1 -- 172.21.15.39:0/3024520638 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca0c1a1c30 con 0x7fca0c1096b0 2024-03-18T16:10:01.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.628+0000 7fca03e00640 1 -- 172.21.15.39:0/3024520638 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca04010890 con 0x7fca0c1096b0 2024-03-18T16:10:01.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.628+0000 7fca12600640 1 -- 172.21.15.39:0/3024520638 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca0c1a2180 con 0x7fca0c1096b0 2024-03-18T16:10:01.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.629+0000 7fca12600640 1 -- 172.21.15.39:0/3024520638 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca0c1083e0 con 0x7fca0c1096b0 2024-03-18T16:10:01.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.631+0000 7fca03e00640 1 -- 172.21.15.39:0/3024520638 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fca04003fc0 con 0x7fca0c1096b0 2024-03-18T16:10:01.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.632+0000 7fca03e00640 1 --2- 172.21.15.39:0/3024520638 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fc9f00a2880 0x7fc9f00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:01.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.633+0000 7fca03e00640 1 -- 172.21.15.39:0/3024520638 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fca040cf400 con 0x7fca0c1096b0 2024-03-18T16:10:01.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.633+0000 7fca0b400640 1 --2- 172.21.15.39:0/3024520638 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fc9f00a2880 0x7fc9f00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:01.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.633+0000 7fca0b400640 1 --2- 172.21.15.39:0/3024520638 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fc9f00a2880 0x7fc9f00a4d40 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fc9fc006010 tx=0x7fc9fc02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:01.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.636+0000 7fca03e00640 1 -- 172.21.15.39:0/3024520638 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fca0409c0b0 con 0x7fca0c1096b0 2024-03-18T16:10:01.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:10:01 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/2607097098' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:10:01.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:10:01 smithi196 ceph-mon[36307]: from='client.25514 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T16:10:01.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:10:01 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/2607097098' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:10:01.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:10:01 smithi194 ceph-mon[36200]: from='client.25514 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T16:10:01.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:10:01 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/2607097098' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:10:01.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:10:01 smithi039 ceph-mon[31721]: from='client.25514 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T16:10:01.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.840+0000 7fca12600640 1 -- 172.21.15.39:0/3024520638 --> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fca0c061c20 con 0x7fc9f00a2880 2024-03-18T16:10:01.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.865+0000 7fca03e00640 1 -- 172.21.15.39:0/3024520638 <== mgr.14152 v2:172.21.15.39:6800/2749214286 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340542 (secure 0 0 0) 0x7fca0c061c20 con 0x7fc9f00a2880 2024-03-18T16:10:01.872 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:10:01.873 INFO:teuthology.orchestra.run.smithi039.stderr:dumped all 2024-03-18T16:10:01.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.873+0000 7fca12600640 1 -- 172.21.15.39:0/3024520638 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fc9f00a2880 msgr2=0x7fc9f00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:01.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.873+0000 7fca12600640 1 --2- 172.21.15.39:0/3024520638 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fc9f00a2880 0x7fc9f00a4d40 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fc9fc006010 tx=0x7fc9fc02df90 comp rx=0 tx=0).stop 2024-03-18T16:10:01.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.873+0000 7fca12600640 1 -- 172.21.15.39:0/3024520638 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fca0c1096b0 msgr2=0x7fca0c1a12a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:01.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.873+0000 7fca12600640 1 --2- 172.21.15.39:0/3024520638 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fca0c1096b0 0x7fca0c1a12a0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fca0400ec50 tx=0x7fca0400c6a0 comp rx=0 tx=0).stop 2024-03-18T16:10:01.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.873+0000 7fca12600640 1 -- 172.21.15.39:0/3024520638 shutdown_connections 2024-03-18T16:10:01.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.873+0000 7fca12600640 1 --2- 172.21.15.39:0/3024520638 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fca0c1096b0 0x7fca0c1a12a0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:01.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.873+0000 7fca12600640 1 --2- 172.21.15.39:0/3024520638 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fc9f00a2880 0x7fc9f00a4d40 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:01.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.873+0000 7fca12600640 1 --2- 172.21.15.39:0/3024520638 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fca0c075820 0x7fca0c19a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:01.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.873+0000 7fca12600640 1 --2- 172.21.15.39:0/3024520638 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fca0c074240 0x7fca0c199d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:01.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.873+0000 7fca12600640 1 -- 172.21.15.39:0/3024520638 >> 172.21.15.39:0/3024520638 conn(0x7fca0c0fcd80 msgr2=0x7fca0c1055b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:01.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.873+0000 7fca12600640 1 -- 172.21.15.39:0/3024520638 shutdown_connections 2024-03-18T16:10:01.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:01.873+0000 7fca12600640 1 -- 172.21.15.39:0/3024520638 wait complete. 2024-03-18T16:10:01.897 INFO:teuthology.orchestra.run.smithi039.stdout:{"pg_ready":true,"pg_map":{"version":421,"stamp":"2024-03-18T16:10:01.574276+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":664231,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1758,"num_read_kb":1010,"num_write":441,"num_write_kb":3094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":1180943,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":862,"ondisk_log_size":862,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":401272,"kb_used_data":14636,"kb_used_omap":18,"kb_used_meta":386477,"kb_avail":1124540552,"statfs":{"total":1151940427776,"available":1151529525248,"internally_reserved":0,"allocated":14987264,"data_stored":8203787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":395752763},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4934,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":108,"num_read_kb":62,"num_write":9,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005670"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.882222+0000","last_change":"2024-03-18T16:06:44.736511+0000","last_active":"2024-03-18T16:08:47.882222+0000","last_peered":"2024-03-18T16:08:47.882222+0000","last_clean":"2024-03-18T16:08:47.882222+0000","last_became_active":"2024-03-18T16:06:44.736182+0000","last_became_peered":"2024-03-18T16:06:44.736182+0000","last_unstale":"2024-03-18T16:08:47.882222+0000","last_undegraded":"2024-03-18T16:08:47.882222+0000","last_fullsized":"2024-03-18T16:08:47.882222+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:08:57.060268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.961793+0000","last_change":"2024-03-18T16:06:44.733658+0000","last_active":"2024-03-18T16:08:47.961793+0000","last_peered":"2024-03-18T16:08:47.961793+0000","last_clean":"2024-03-18T16:08:47.961793+0000","last_became_active":"2024-03-18T16:06:44.733279+0000","last_became_peered":"2024-03-18T16:06:44.733279+0000","last_unstale":"2024-03-18T16:08:47.961793+0000","last_undegraded":"2024-03-18T16:08:47.961793+0000","last_fullsized":"2024-03-18T16:08:47.961793+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:59:46.783025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.961451+0000","last_change":"2024-03-18T16:06:44.723949+0000","last_active":"2024-03-18T16:08:47.961451+0000","last_peered":"2024-03-18T16:08:47.961451+0000","last_clean":"2024-03-18T16:08:47.961451+0000","last_became_active":"2024-03-18T16:06:44.723614+0000","last_became_peered":"2024-03-18T16:06:44.723614+0000","last_unstale":"2024-03-18T16:08:47.961451+0000","last_undegraded":"2024-03-18T16:08:47.961451+0000","last_fullsized":"2024-03-18T16:08:47.961451+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:03:17.876097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883864+0000","last_change":"2024-03-18T16:06:44.711058+0000","last_active":"2024-03-18T16:08:47.883864+0000","last_peered":"2024-03-18T16:08:47.883864+0000","last_clean":"2024-03-18T16:08:47.883864+0000","last_became_active":"2024-03-18T16:06:44.708873+0000","last_became_peered":"2024-03-18T16:06:44.708873+0000","last_unstale":"2024-03-18T16:08:47.883864+0000","last_undegraded":"2024-03-18T16:08:47.883864+0000","last_fullsized":"2024-03-18T16:08:47.883864+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:58:08.182651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.884722+0000","last_change":"2024-03-18T16:06:44.721480+0000","last_active":"2024-03-18T16:08:47.884722+0000","last_peered":"2024-03-18T16:08:47.884722+0000","last_clean":"2024-03-18T16:08:47.884722+0000","last_became_active":"2024-03-18T16:06:44.721101+0000","last_became_peered":"2024-03-18T16:06:44.721101+0000","last_unstale":"2024-03-18T16:08:47.884722+0000","last_undegraded":"2024-03-18T16:08:47.884722+0000","last_fullsized":"2024-03-18T16:08:47.884722+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:19:43.341221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.879862+0000","last_change":"2024-03-18T16:06:44.716443+0000","last_active":"2024-03-18T16:08:47.879862+0000","last_peered":"2024-03-18T16:08:47.879862+0000","last_clean":"2024-03-18T16:08:47.879862+0000","last_became_active":"2024-03-18T16:06:44.715969+0000","last_became_peered":"2024-03-18T16:06:44.715969+0000","last_unstale":"2024-03-18T16:08:47.879862+0000","last_undegraded":"2024-03-18T16:08:47.879862+0000","last_fullsized":"2024-03-18T16:08:47.879862+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:57:14.714415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.960378+0000","last_change":"2024-03-18T16:06:44.711439+0000","last_active":"2024-03-18T16:08:47.960378+0000","last_peered":"2024-03-18T16:08:47.960378+0000","last_clean":"2024-03-18T16:08:47.960378+0000","last_became_active":"2024-03-18T16:06:44.711032+0000","last_became_peered":"2024-03-18T16:06:44.711032+0000","last_unstale":"2024-03-18T16:08:47.960378+0000","last_undegraded":"2024-03-18T16:08:47.960378+0000","last_fullsized":"2024-03-18T16:08:47.960378+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:19:56.516867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.881902+0000","last_change":"2024-03-18T16:07:47.358866+0000","last_active":"2024-03-18T16:08:47.881902+0000","last_peered":"2024-03-18T16:08:47.881902+0000","last_clean":"2024-03-18T16:08:47.881902+0000","last_became_active":"2024-03-18T16:07:47.357743+0000","last_became_peered":"2024-03-18T16:07:47.357743+0000","last_unstale":"2024-03-18T16:08:47.881902+0000","last_undegraded":"2024-03-18T16:08:47.881902+0000","last_fullsized":"2024-03-18T16:08:47.881902+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:13:12.361025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,10],"acting":[9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.960770+0000","last_change":"2024-03-18T16:06:44.710978+0000","last_active":"2024-03-18T16:08:47.960770+0000","last_peered":"2024-03-18T16:08:47.960770+0000","last_clean":"2024-03-18T16:08:47.960770+0000","last_became_active":"2024-03-18T16:06:44.710520+0000","last_became_peered":"2024-03-18T16:06:44.710520+0000","last_unstale":"2024-03-18T16:08:47.960770+0000","last_undegraded":"2024-03-18T16:08:47.960770+0000","last_fullsized":"2024-03-18T16:08:47.960770+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:54:53.234025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.880413+0000","last_change":"2024-03-18T16:06:44.716231+0000","last_active":"2024-03-18T16:08:47.880413+0000","last_peered":"2024-03-18T16:08:47.880413+0000","last_clean":"2024-03-18T16:08:47.880413+0000","last_became_active":"2024-03-18T16:06:44.714900+0000","last_became_peered":"2024-03-18T16:06:44.714900+0000","last_unstale":"2024-03-18T16:08:47.880413+0000","last_undegraded":"2024-03-18T16:08:47.880413+0000","last_fullsized":"2024-03-18T16:08:47.880413+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:38:54.964729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.961221+0000","last_change":"2024-03-18T16:06:44.710942+0000","last_active":"2024-03-18T16:08:47.961221+0000","last_peered":"2024-03-18T16:08:47.961221+0000","last_clean":"2024-03-18T16:08:47.961221+0000","last_became_active":"2024-03-18T16:06:44.710588+0000","last_became_peered":"2024-03-18T16:06:44.710588+0000","last_unstale":"2024-03-18T16:08:47.961221+0000","last_undegraded":"2024-03-18T16:08:47.961221+0000","last_fullsized":"2024-03-18T16:08:47.961221+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:43:19.320034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883838+0000","last_change":"2024-03-18T16:06:44.714763+0000","last_active":"2024-03-18T16:08:47.883838+0000","last_peered":"2024-03-18T16:08:47.883838+0000","last_clean":"2024-03-18T16:08:47.883838+0000","last_became_active":"2024-03-18T16:06:44.714031+0000","last_became_peered":"2024-03-18T16:06:44.714031+0000","last_unstale":"2024-03-18T16:08:47.883838+0000","last_undegraded":"2024-03-18T16:08:47.883838+0000","last_fullsized":"2024-03-18T16:08:47.883838+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:19:25.917093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.962631+0000","last_change":"2024-03-18T16:06:44.710062+0000","last_active":"2024-03-18T16:08:47.962631+0000","last_peered":"2024-03-18T16:08:47.962631+0000","last_clean":"2024-03-18T16:08:47.962631+0000","last_became_active":"2024-03-18T16:06:44.709281+0000","last_became_peered":"2024-03-18T16:06:44.709281+0000","last_unstale":"2024-03-18T16:08:47.962631+0000","last_undegraded":"2024-03-18T16:08:47.962631+0000","last_fullsized":"2024-03-18T16:08:47.962631+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:51:30.100947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.960815+0000","last_change":"2024-03-18T16:06:44.734178+0000","last_active":"2024-03-18T16:08:47.960815+0000","last_peered":"2024-03-18T16:08:47.960815+0000","last_clean":"2024-03-18T16:08:47.960815+0000","last_became_active":"2024-03-18T16:06:44.732538+0000","last_became_peered":"2024-03-18T16:06:44.732538+0000","last_unstale":"2024-03-18T16:08:47.960815+0000","last_undegraded":"2024-03-18T16:08:47.960815+0000","last_fullsized":"2024-03-18T16:08:47.960815+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:14:26.904402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.960944+0000","last_change":"2024-03-18T16:07:47.412629+0000","last_active":"2024-03-18T16:08:47.960944+0000","last_peered":"2024-03-18T16:08:47.960944+0000","last_clean":"2024-03-18T16:08:47.960944+0000","last_became_active":"2024-03-18T16:07:47.411714+0000","last_became_peered":"2024-03-18T16:07:47.411714+0000","last_unstale":"2024-03-18T16:08:47.960944+0000","last_undegraded":"2024-03-18T16:08:47.960944+0000","last_fullsized":"2024-03-18T16:08:47.960944+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:25:50.843323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.959907+0000","last_change":"2024-03-18T16:06:44.728165+0000","last_active":"2024-03-18T16:08:47.959907+0000","last_peered":"2024-03-18T16:08:47.959907+0000","last_clean":"2024-03-18T16:08:47.959907+0000","last_became_active":"2024-03-18T16:06:44.727842+0000","last_became_peered":"2024-03-18T16:06:44.727842+0000","last_unstale":"2024-03-18T16:08:47.959907+0000","last_undegraded":"2024-03-18T16:08:47.959907+0000","last_fullsized":"2024-03-18T16:08:47.959907+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:55:04.464916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.881962+0000","last_change":"2024-03-18T16:06:44.730893+0000","last_active":"2024-03-18T16:08:47.881962+0000","last_peered":"2024-03-18T16:08:47.881962+0000","last_clean":"2024-03-18T16:08:47.881962+0000","last_became_active":"2024-03-18T16:06:44.730544+0000","last_became_peered":"2024-03-18T16:06:44.730544+0000","last_unstale":"2024-03-18T16:08:47.881962+0000","last_undegraded":"2024-03-18T16:08:47.881962+0000","last_fullsized":"2024-03-18T16:08:47.881962+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:16:37.918096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.884320+0000","last_change":"2024-03-18T16:06:44.706036+0000","last_active":"2024-03-18T16:08:47.884320+0000","last_peered":"2024-03-18T16:08:47.884320+0000","last_clean":"2024-03-18T16:08:47.884320+0000","last_became_active":"2024-03-18T16:06:44.705281+0000","last_became_peered":"2024-03-18T16:06:44.705281+0000","last_unstale":"2024-03-18T16:08:47.884320+0000","last_undegraded":"2024-03-18T16:08:47.884320+0000","last_fullsized":"2024-03-18T16:08:47.884320+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:52:20.202110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.884506+0000","last_change":"2024-03-18T16:06:44.727543+0000","last_active":"2024-03-18T16:08:47.884506+0000","last_peered":"2024-03-18T16:08:47.884506+0000","last_clean":"2024-03-18T16:08:47.884506+0000","last_became_active":"2024-03-18T16:06:44.727135+0000","last_became_peered":"2024-03-18T16:06:44.727135+0000","last_unstale":"2024-03-18T16:08:47.884506+0000","last_undegraded":"2024-03-18T16:08:47.884506+0000","last_fullsized":"2024-03-18T16:08:47.884506+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:52:42.742279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883556+0000","last_change":"2024-03-18T16:06:49.028004+0000","last_active":"2024-03-18T16:08:47.883556+0000","last_peered":"2024-03-18T16:08:47.883556+0000","last_clean":"2024-03-18T16:08:47.883556+0000","last_became_active":"2024-03-18T16:06:49.027640+0000","last_became_peered":"2024-03-18T16:06:49.027640+0000","last_unstale":"2024-03-18T16:08:47.883556+0000","last_undegraded":"2024-03-18T16:08:47.883556+0000","last_fullsized":"2024-03-18T16:08:47.883556+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:23:07.926349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'13","reported_seq":202,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:09:55.306333+0000","last_change":"2024-03-18T16:06:43.709540+0000","last_active":"2024-03-18T16:09:55.306333+0000","last_peered":"2024-03-18T16:09:55.306333+0000","last_clean":"2024-03-18T16:09:55.306333+0000","last_became_active":"2024-03-18T16:06:43.709170+0000","last_became_peered":"2024-03-18T16:06:43.709170+0000","last_unstale":"2024-03-18T16:09:55.306333+0000","last_undegraded":"2024-03-18T16:09:55.306333+0000","last_fullsized":"2024-03-18T16:09:55.306333+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:12:53.934079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":157,"num_read_kb":157,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.886428+0000","last_change":"2024-03-18T16:06:44.730192+0000","last_active":"2024-03-18T16:08:47.886428+0000","last_peered":"2024-03-18T16:08:47.886428+0000","last_clean":"2024-03-18T16:08:47.886428+0000","last_became_active":"2024-03-18T16:06:44.728071+0000","last_became_peered":"2024-03-18T16:06:44.728071+0000","last_unstale":"2024-03-18T16:08:47.886428+0000","last_undegraded":"2024-03-18T16:08:47.886428+0000","last_fullsized":"2024-03-18T16:08:47.886428+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:14:59.992958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883329+0000","last_change":"2024-03-18T16:06:49.023963+0000","last_active":"2024-03-18T16:08:47.883329+0000","last_peered":"2024-03-18T16:08:47.883329+0000","last_clean":"2024-03-18T16:08:47.883329+0000","last_became_active":"2024-03-18T16:06:49.023568+0000","last_became_peered":"2024-03-18T16:06:49.023568+0000","last_unstale":"2024-03-18T16:08:47.883329+0000","last_undegraded":"2024-03-18T16:08:47.883329+0000","last_fullsized":"2024-03-18T16:08:47.883329+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:15:19.730842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.886440+0000","last_change":"2024-03-18T16:06:43.722167+0000","last_active":"2024-03-18T16:08:47.886440+0000","last_peered":"2024-03-18T16:08:47.886440+0000","last_clean":"2024-03-18T16:08:47.886440+0000","last_became_active":"2024-03-18T16:06:43.721782+0000","last_became_peered":"2024-03-18T16:06:43.721782+0000","last_unstale":"2024-03-18T16:08:47.886440+0000","last_undegraded":"2024-03-18T16:08:47.886440+0000","last_fullsized":"2024-03-18T16:08:47.886440+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:08:54.400637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.962047+0000","last_change":"2024-03-18T16:06:44.729666+0000","last_active":"2024-03-18T16:08:47.962047+0000","last_peered":"2024-03-18T16:08:47.962047+0000","last_clean":"2024-03-18T16:08:47.962047+0000","last_became_active":"2024-03-18T16:06:44.728842+0000","last_became_peered":"2024-03-18T16:06:44.728842+0000","last_unstale":"2024-03-18T16:08:47.962047+0000","last_undegraded":"2024-03-18T16:08:47.962047+0000","last_fullsized":"2024-03-18T16:08:47.962047+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:01:42.438000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883464+0000","last_change":"2024-03-18T16:06:49.031169+0000","last_active":"2024-03-18T16:08:47.883464+0000","last_peered":"2024-03-18T16:08:47.883464+0000","last_clean":"2024-03-18T16:08:47.883464+0000","last_became_active":"2024-03-18T16:06:49.030801+0000","last_became_peered":"2024-03-18T16:06:49.030801+0000","last_unstale":"2024-03-18T16:08:47.883464+0000","last_undegraded":"2024-03-18T16:08:47.883464+0000","last_fullsized":"2024-03-18T16:08:47.883464+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:10:54.880882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'12","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.885418+0000","last_change":"2024-03-18T16:06:43.707111+0000","last_active":"2024-03-18T16:08:47.885418+0000","last_peered":"2024-03-18T16:08:47.885418+0000","last_clean":"2024-03-18T16:08:47.885418+0000","last_became_active":"2024-03-18T16:06:43.706671+0000","last_became_peered":"2024-03-18T16:06:43.706671+0000","last_unstale":"2024-03-18T16:08:47.885418+0000","last_undegraded":"2024-03-18T16:08:47.885418+0000","last_fullsized":"2024-03-18T16:08:47.885418+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:18:34.295641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.960382+0000","last_change":"2024-03-18T16:06:44.722100+0000","last_active":"2024-03-18T16:08:47.960382+0000","last_peered":"2024-03-18T16:08:47.960382+0000","last_clean":"2024-03-18T16:08:47.960382+0000","last_became_active":"2024-03-18T16:06:44.721673+0000","last_became_peered":"2024-03-18T16:06:44.721673+0000","last_unstale":"2024-03-18T16:08:47.960382+0000","last_undegraded":"2024-03-18T16:08:47.960382+0000","last_fullsized":"2024-03-18T16:08:47.960382+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:41:22.624778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883382+0000","last_change":"2024-03-18T16:06:49.027966+0000","last_active":"2024-03-18T16:08:47.883382+0000","last_peered":"2024-03-18T16:08:47.883382+0000","last_clean":"2024-03-18T16:08:47.883382+0000","last_became_active":"2024-03-18T16:06:49.027567+0000","last_became_peered":"2024-03-18T16:06:49.027567+0000","last_unstale":"2024-03-18T16:08:47.883382+0000","last_undegraded":"2024-03-18T16:08:47.883382+0000","last_fullsized":"2024-03-18T16:08:47.883382+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:54:28.543019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.885092+0000","last_change":"2024-03-18T16:06:43.704023+0000","last_active":"2024-03-18T16:08:47.885092+0000","last_peered":"2024-03-18T16:08:47.885092+0000","last_clean":"2024-03-18T16:08:47.885092+0000","last_became_active":"2024-03-18T16:06:43.703690+0000","last_became_peered":"2024-03-18T16:06:43.703690+0000","last_unstale":"2024-03-18T16:08:47.885092+0000","last_undegraded":"2024-03-18T16:08:47.885092+0000","last_fullsized":"2024-03-18T16:08:47.885092+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:20:02.344346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":50,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.886144+0000","last_change":"2024-03-18T16:07:47.354361+0000","last_active":"2024-03-18T16:08:47.886144+0000","last_peered":"2024-03-18T16:08:47.886144+0000","last_clean":"2024-03-18T16:08:47.886144+0000","last_became_active":"2024-03-18T16:07:47.353964+0000","last_became_peered":"2024-03-18T16:07:47.353964+0000","last_unstale":"2024-03-18T16:08:47.886144+0000","last_undegraded":"2024-03-18T16:08:47.886144+0000","last_fullsized":"2024-03-18T16:08:47.886144+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:56:11.405218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.884343+0000","last_change":"2024-03-18T16:06:49.012310+0000","last_active":"2024-03-18T16:08:47.884343+0000","last_peered":"2024-03-18T16:08:47.884343+0000","last_clean":"2024-03-18T16:08:47.884343+0000","last_became_active":"2024-03-18T16:06:49.011844+0000","last_became_peered":"2024-03-18T16:06:49.011844+0000","last_unstale":"2024-03-18T16:08:47.884343+0000","last_undegraded":"2024-03-18T16:08:47.884343+0000","last_fullsized":"2024-03-18T16:08:47.884343+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:16:42.652205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"91'7","reported_seq":199,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:09:55.144566+0000","last_change":"2024-03-18T16:06:43.705653+0000","last_active":"2024-03-18T16:09:55.144566+0000","last_peered":"2024-03-18T16:09:55.144566+0000","last_clean":"2024-03-18T16:09:55.144566+0000","last_became_active":"2024-03-18T16:06:43.705320+0000","last_became_peered":"2024-03-18T16:06:43.705320+0000","last_unstale":"2024-03-18T16:09:55.144566+0000","last_undegraded":"2024-03-18T16:09:55.144566+0000","last_fullsized":"2024-03-18T16:09:55.144566+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:10:10.827338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":161,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.884849+0000","last_change":"2024-03-18T16:06:44.720045+0000","last_active":"2024-03-18T16:08:47.884849+0000","last_peered":"2024-03-18T16:08:47.884849+0000","last_clean":"2024-03-18T16:08:47.884849+0000","last_became_active":"2024-03-18T16:06:44.719669+0000","last_became_peered":"2024-03-18T16:06:44.719669+0000","last_unstale":"2024-03-18T16:08:47.884849+0000","last_undegraded":"2024-03-18T16:08:47.884849+0000","last_fullsized":"2024-03-18T16:08:47.884849+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:23:47.464057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.884107+0000","last_change":"2024-03-18T16:06:49.032576+0000","last_active":"2024-03-18T16:08:47.884107+0000","last_peered":"2024-03-18T16:08:47.884107+0000","last_clean":"2024-03-18T16:08:47.884107+0000","last_became_active":"2024-03-18T16:06:49.032126+0000","last_became_peered":"2024-03-18T16:06:49.032126+0000","last_unstale":"2024-03-18T16:08:47.884107+0000","last_undegraded":"2024-03-18T16:08:47.884107+0000","last_fullsized":"2024-03-18T16:08:47.884107+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:41:54.822070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.879820+0000","last_change":"2024-03-18T16:06:43.706364+0000","last_active":"2024-03-18T16:08:47.879820+0000","last_peered":"2024-03-18T16:08:47.879820+0000","last_clean":"2024-03-18T16:08:47.879820+0000","last_became_active":"2024-03-18T16:06:43.705984+0000","last_became_peered":"2024-03-18T16:06:43.705984+0000","last_unstale":"2024-03-18T16:08:47.879820+0000","last_undegraded":"2024-03-18T16:08:47.879820+0000","last_fullsized":"2024-03-18T16:08:47.879820+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:46:06.503000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.886025+0000","last_change":"2024-03-18T16:06:44.721019+0000","last_active":"2024-03-18T16:08:47.886025+0000","last_peered":"2024-03-18T16:08:47.886025+0000","last_clean":"2024-03-18T16:08:47.886025+0000","last_became_active":"2024-03-18T16:06:44.720665+0000","last_became_peered":"2024-03-18T16:06:44.720665+0000","last_unstale":"2024-03-18T16:08:47.886025+0000","last_undegraded":"2024-03-18T16:08:47.886025+0000","last_fullsized":"2024-03-18T16:08:47.886025+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:58:34.180653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.881862+0000","last_change":"2024-03-18T16:06:49.022919+0000","last_active":"2024-03-18T16:08:47.881862+0000","last_peered":"2024-03-18T16:08:47.881862+0000","last_clean":"2024-03-18T16:08:47.881862+0000","last_became_active":"2024-03-18T16:06:49.022514+0000","last_became_peered":"2024-03-18T16:06:49.022514+0000","last_unstale":"2024-03-18T16:08:47.881862+0000","last_undegraded":"2024-03-18T16:08:47.881862+0000","last_fullsized":"2024-03-18T16:08:47.881862+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:46:39.510181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.961422+0000","last_change":"2024-03-18T16:06:43.713025+0000","last_active":"2024-03-18T16:08:47.961422+0000","last_peered":"2024-03-18T16:08:47.961422+0000","last_clean":"2024-03-18T16:08:47.961422+0000","last_became_active":"2024-03-18T16:06:43.711469+0000","last_became_peered":"2024-03-18T16:06:43.711469+0000","last_unstale":"2024-03-18T16:08:47.961422+0000","last_undegraded":"2024-03-18T16:08:47.961422+0000","last_fullsized":"2024-03-18T16:08:47.961422+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:30:41.193229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883163+0000","last_change":"2024-03-18T16:06:44.723953+0000","last_active":"2024-03-18T16:08:47.883163+0000","last_peered":"2024-03-18T16:08:47.883163+0000","last_clean":"2024-03-18T16:08:47.883163+0000","last_became_active":"2024-03-18T16:06:44.723266+0000","last_became_peered":"2024-03-18T16:06:44.723266+0000","last_unstale":"2024-03-18T16:08:47.883163+0000","last_undegraded":"2024-03-18T16:08:47.883163+0000","last_fullsized":"2024-03-18T16:08:47.883163+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:41:10.067297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.961683+0000","last_change":"2024-03-18T16:06:49.039472+0000","last_active":"2024-03-18T16:08:47.961683+0000","last_peered":"2024-03-18T16:08:47.961683+0000","last_clean":"2024-03-18T16:08:47.961683+0000","last_became_active":"2024-03-18T16:06:49.038889+0000","last_became_peered":"2024-03-18T16:06:49.038889+0000","last_unstale":"2024-03-18T16:08:47.961683+0000","last_undegraded":"2024-03-18T16:08:47.961683+0000","last_fullsized":"2024-03-18T16:08:47.961683+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:03:05.345315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.885773+0000","last_change":"2024-03-18T16:06:43.705873+0000","last_active":"2024-03-18T16:08:47.885773+0000","last_peered":"2024-03-18T16:08:47.885773+0000","last_clean":"2024-03-18T16:08:47.885773+0000","last_became_active":"2024-03-18T16:06:43.705548+0000","last_became_peered":"2024-03-18T16:06:43.705548+0000","last_unstale":"2024-03-18T16:08:47.885773+0000","last_undegraded":"2024-03-18T16:08:47.885773+0000","last_fullsized":"2024-03-18T16:08:47.885773+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:52:03.790888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.882690+0000","last_change":"2024-03-18T16:06:44.718526+0000","last_active":"2024-03-18T16:08:47.882690+0000","last_peered":"2024-03-18T16:08:47.882690+0000","last_clean":"2024-03-18T16:08:47.882690+0000","last_became_active":"2024-03-18T16:06:44.717266+0000","last_became_peered":"2024-03-18T16:06:44.717266+0000","last_unstale":"2024-03-18T16:08:47.882690+0000","last_undegraded":"2024-03-18T16:08:47.882690+0000","last_fullsized":"2024-03-18T16:08:47.882690+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:55:09.340354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.885083+0000","last_change":"2024-03-18T16:06:49.021971+0000","last_active":"2024-03-18T16:08:47.885083+0000","last_peered":"2024-03-18T16:08:47.885083+0000","last_clean":"2024-03-18T16:08:47.885083+0000","last_became_active":"2024-03-18T16:06:49.021577+0000","last_became_peered":"2024-03-18T16:06:49.021577+0000","last_unstale":"2024-03-18T16:08:47.885083+0000","last_undegraded":"2024-03-18T16:08:47.885083+0000","last_fullsized":"2024-03-18T16:08:47.885083+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:47:44.768559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"93'6","reported_seq":357,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:09:59.148338+0000","last_change":"2024-03-18T16:06:43.705636+0000","last_active":"2024-03-18T16:09:59.148338+0000","last_peered":"2024-03-18T16:09:59.148338+0000","last_clean":"2024-03-18T16:09:59.148338+0000","last_became_active":"2024-03-18T16:06:43.705268+0000","last_became_peered":"2024-03-18T16:06:43.705268+0000","last_unstale":"2024-03-18T16:09:59.148338+0000","last_undegraded":"2024-03-18T16:09:59.148338+0000","last_fullsized":"2024-03-18T16:09:59.148338+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:37:31.367130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":320,"num_read_kb":320,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.960576+0000","last_change":"2024-03-18T16:06:44.712237+0000","last_active":"2024-03-18T16:08:47.960576+0000","last_peered":"2024-03-18T16:08:47.960576+0000","last_clean":"2024-03-18T16:08:47.960576+0000","last_became_active":"2024-03-18T16:06:44.711677+0000","last_became_peered":"2024-03-18T16:06:44.711677+0000","last_unstale":"2024-03-18T16:08:47.960576+0000","last_undegraded":"2024-03-18T16:08:47.960576+0000","last_fullsized":"2024-03-18T16:08:47.960576+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:16:56.326962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.882199+0000","last_change":"2024-03-18T16:06:49.034578+0000","last_active":"2024-03-18T16:08:47.882199+0000","last_peered":"2024-03-18T16:08:47.882199+0000","last_clean":"2024-03-18T16:08:47.882199+0000","last_became_active":"2024-03-18T16:06:49.031099+0000","last_became_peered":"2024-03-18T16:06:49.031099+0000","last_unstale":"2024-03-18T16:08:47.882199+0000","last_undegraded":"2024-03-18T16:08:47.882199+0000","last_fullsized":"2024-03-18T16:08:47.882199+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:11:08.042550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.961212+0000","last_change":"2024-03-18T16:06:43.708151+0000","last_active":"2024-03-18T16:08:47.961212+0000","last_peered":"2024-03-18T16:08:47.961212+0000","last_clean":"2024-03-18T16:08:47.961212+0000","last_became_active":"2024-03-18T16:06:43.707810+0000","last_became_peered":"2024-03-18T16:06:43.707810+0000","last_unstale":"2024-03-18T16:08:47.961212+0000","last_undegraded":"2024-03-18T16:08:47.961212+0000","last_fullsized":"2024-03-18T16:08:47.961212+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:30:35.745772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.962204+0000","last_change":"2024-03-18T16:06:44.729777+0000","last_active":"2024-03-18T16:08:47.962204+0000","last_peered":"2024-03-18T16:08:47.962204+0000","last_clean":"2024-03-18T16:08:47.962204+0000","last_became_active":"2024-03-18T16:06:44.729244+0000","last_became_peered":"2024-03-18T16:06:44.729244+0000","last_unstale":"2024-03-18T16:08:47.962204+0000","last_undegraded":"2024-03-18T16:08:47.962204+0000","last_fullsized":"2024-03-18T16:08:47.962204+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:52:05.420443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883653+0000","last_change":"2024-03-18T16:07:47.355131+0000","last_active":"2024-03-18T16:08:47.883653+0000","last_peered":"2024-03-18T16:08:47.883653+0000","last_clean":"2024-03-18T16:08:47.883653+0000","last_became_active":"2024-03-18T16:07:47.354608+0000","last_became_peered":"2024-03-18T16:07:47.354608+0000","last_unstale":"2024-03-18T16:08:47.883653+0000","last_undegraded":"2024-03-18T16:08:47.883653+0000","last_fullsized":"2024-03-18T16:08:47.883653+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:12:57.979168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.886234+0000","last_change":"2024-03-18T16:06:43.715280+0000","last_active":"2024-03-18T16:08:47.886234+0000","last_peered":"2024-03-18T16:08:47.886234+0000","last_clean":"2024-03-18T16:08:47.886234+0000","last_became_active":"2024-03-18T16:06:43.714936+0000","last_became_peered":"2024-03-18T16:06:43.714936+0000","last_unstale":"2024-03-18T16:08:47.886234+0000","last_undegraded":"2024-03-18T16:08:47.886234+0000","last_fullsized":"2024-03-18T16:08:47.886234+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:07:38.746932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.960603+0000","last_change":"2024-03-18T16:06:44.709004+0000","last_active":"2024-03-18T16:08:47.960603+0000","last_peered":"2024-03-18T16:08:47.960603+0000","last_clean":"2024-03-18T16:08:47.960603+0000","last_became_active":"2024-03-18T16:06:44.708574+0000","last_became_peered":"2024-03-18T16:06:44.708574+0000","last_unstale":"2024-03-18T16:08:47.960603+0000","last_undegraded":"2024-03-18T16:08:47.960603+0000","last_fullsized":"2024-03-18T16:08:47.960603+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:02:50.982110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883055+0000","last_change":"2024-03-18T16:06:49.029806+0000","last_active":"2024-03-18T16:08:47.883055+0000","last_peered":"2024-03-18T16:08:47.883055+0000","last_clean":"2024-03-18T16:08:47.883055+0000","last_became_active":"2024-03-18T16:06:49.029404+0000","last_became_peered":"2024-03-18T16:06:49.029404+0000","last_unstale":"2024-03-18T16:08:47.883055+0000","last_undegraded":"2024-03-18T16:08:47.883055+0000","last_fullsized":"2024-03-18T16:08:47.883055+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:38:37.706620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"93'40","reported_seq":402,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:09:59.149498+0000","last_change":"2024-03-18T16:06:43.702910+0000","last_active":"2024-03-18T16:09:59.149498+0000","last_peered":"2024-03-18T16:09:59.149498+0000","last_clean":"2024-03-18T16:09:59.149498+0000","last_became_active":"2024-03-18T16:06:43.702557+0000","last_became_peered":"2024-03-18T16:06:43.702557+0000","last_unstale":"2024-03-18T16:09:59.149498+0000","last_undegraded":"2024-03-18T16:09:59.149498+0000","last_fullsized":"2024-03-18T16:09:59.149498+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:55:15.362987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66365,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":331,"num_read_kb":231,"num_write":42,"num_write_kb":85,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.884961+0000","last_change":"2024-03-18T16:06:44.709075+0000","last_active":"2024-03-18T16:08:47.884961+0000","last_peered":"2024-03-18T16:08:47.884961+0000","last_clean":"2024-03-18T16:08:47.884961+0000","last_became_active":"2024-03-18T16:06:44.708366+0000","last_became_peered":"2024-03-18T16:06:44.708366+0000","last_unstale":"2024-03-18T16:08:47.884961+0000","last_undegraded":"2024-03-18T16:08:47.884961+0000","last_fullsized":"2024-03-18T16:08:47.884961+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:44:18.486385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.886444+0000","last_change":"2024-03-18T16:06:49.037380+0000","last_active":"2024-03-18T16:08:47.886444+0000","last_peered":"2024-03-18T16:08:47.886444+0000","last_clean":"2024-03-18T16:08:47.886444+0000","last_became_active":"2024-03-18T16:06:49.037021+0000","last_became_peered":"2024-03-18T16:06:49.037021+0000","last_unstale":"2024-03-18T16:08:47.886444+0000","last_undegraded":"2024-03-18T16:08:47.886444+0000","last_fullsized":"2024-03-18T16:08:47.886444+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:43:03.482631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.896395+0000","last_change":"2024-03-18T16:08:47.895964+0000","last_active":"2024-03-18T16:08:47.896395+0000","last_peered":"2024-03-18T16:08:47.896395+0000","last_clean":"2024-03-18T16:08:47.896395+0000","last_became_active":"2024-03-18T16:08:47.886729+0000","last_became_peered":"2024-03-18T16:08:47.886729+0000","last_unstale":"2024-03-18T16:08:47.896395+0000","last_undegraded":"2024-03-18T16:08:47.896395+0000","last_fullsized":"2024-03-18T16:08:47.896395+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:58:02.557326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":68,"num_keys_recovered":10,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.962072+0000","last_change":"2024-03-18T16:06:44.727204+0000","last_active":"2024-03-18T16:08:47.962072+0000","last_peered":"2024-03-18T16:08:47.962072+0000","last_clean":"2024-03-18T16:08:47.962072+0000","last_became_active":"2024-03-18T16:06:44.726775+0000","last_became_peered":"2024-03-18T16:06:44.726775+0000","last_unstale":"2024-03-18T16:08:47.962072+0000","last_undegraded":"2024-03-18T16:08:47.962072+0000","last_fullsized":"2024-03-18T16:08:47.962072+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:29:44.842972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.881484+0000","last_change":"2024-03-18T16:06:49.022508+0000","last_active":"2024-03-18T16:08:47.881484+0000","last_peered":"2024-03-18T16:08:47.881484+0000","last_clean":"2024-03-18T16:08:47.881484+0000","last_became_active":"2024-03-18T16:06:49.022017+0000","last_became_peered":"2024-03-18T16:06:49.022017+0000","last_unstale":"2024-03-18T16:08:47.881484+0000","last_undegraded":"2024-03-18T16:08:47.881484+0000","last_fullsized":"2024-03-18T16:08:47.881484+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:52:45.892880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.885883+0000","last_change":"2024-03-18T16:06:43.719618+0000","last_active":"2024-03-18T16:08:47.885883+0000","last_peered":"2024-03-18T16:08:47.885883+0000","last_clean":"2024-03-18T16:08:47.885883+0000","last_became_active":"2024-03-18T16:06:43.719161+0000","last_became_peered":"2024-03-18T16:06:43.719161+0000","last_unstale":"2024-03-18T16:08:47.885883+0000","last_undegraded":"2024-03-18T16:08:47.885883+0000","last_fullsized":"2024-03-18T16:08:47.885883+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:39:53.077878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.960915+0000","last_change":"2024-03-18T16:06:44.717829+0000","last_active":"2024-03-18T16:08:47.960915+0000","last_peered":"2024-03-18T16:08:47.960915+0000","last_clean":"2024-03-18T16:08:47.960915+0000","last_became_active":"2024-03-18T16:06:44.717473+0000","last_became_peered":"2024-03-18T16:06:44.717473+0000","last_unstale":"2024-03-18T16:08:47.960915+0000","last_undegraded":"2024-03-18T16:08:47.960915+0000","last_fullsized":"2024-03-18T16:08:47.960915+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:10:00.183613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.884360+0000","last_change":"2024-03-18T16:06:49.036766+0000","last_active":"2024-03-18T16:08:47.884360+0000","last_peered":"2024-03-18T16:08:47.884360+0000","last_clean":"2024-03-18T16:08:47.884360+0000","last_became_active":"2024-03-18T16:06:49.034617+0000","last_became_peered":"2024-03-18T16:06:49.034617+0000","last_unstale":"2024-03-18T16:08:47.884360+0000","last_undegraded":"2024-03-18T16:08:47.884360+0000","last_fullsized":"2024-03-18T16:08:47.884360+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:18:59.033781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'5","reported_seq":196,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:09:56.221145+0000","last_change":"2024-03-18T16:06:43.702452+0000","last_active":"2024-03-18T16:09:56.221145+0000","last_peered":"2024-03-18T16:09:56.221145+0000","last_clean":"2024-03-18T16:09:56.221145+0000","last_became_active":"2024-03-18T16:06:43.702141+0000","last_became_peered":"2024-03-18T16:06:43.702141+0000","last_unstale":"2024-03-18T16:09:56.221145+0000","last_undegraded":"2024-03-18T16:09:56.221145+0000","last_fullsized":"2024-03-18T16:09:56.221145+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:44:05.366101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":161,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.960175+0000","last_change":"2024-03-18T16:06:44.711403+0000","last_active":"2024-03-18T16:08:47.960175+0000","last_peered":"2024-03-18T16:08:47.960175+0000","last_clean":"2024-03-18T16:08:47.960175+0000","last_became_active":"2024-03-18T16:06:44.710654+0000","last_became_peered":"2024-03-18T16:06:44.710654+0000","last_unstale":"2024-03-18T16:08:47.960175+0000","last_undegraded":"2024-03-18T16:08:47.960175+0000","last_fullsized":"2024-03-18T16:08:47.960175+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:21:43.633274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.962799+0000","last_change":"2024-03-18T16:06:49.039192+0000","last_active":"2024-03-18T16:08:47.962799+0000","last_peered":"2024-03-18T16:08:47.962799+0000","last_clean":"2024-03-18T16:08:47.962799+0000","last_became_active":"2024-03-18T16:06:49.038421+0000","last_became_peered":"2024-03-18T16:06:49.038421+0000","last_unstale":"2024-03-18T16:08:47.962799+0000","last_undegraded":"2024-03-18T16:08:47.962799+0000","last_fullsized":"2024-03-18T16:08:47.962799+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:54:56.944209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":49,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.884583+0000","last_change":"2024-03-18T16:06:43.712523+0000","last_active":"2024-03-18T16:08:47.884583+0000","last_peered":"2024-03-18T16:08:47.884583+0000","last_clean":"2024-03-18T16:08:47.884583+0000","last_became_active":"2024-03-18T16:06:43.712151+0000","last_became_peered":"2024-03-18T16:06:43.712151+0000","last_unstale":"2024-03-18T16:08:47.884583+0000","last_undegraded":"2024-03-18T16:08:47.884583+0000","last_fullsized":"2024-03-18T16:08:47.884583+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:17:19.651293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.884896+0000","last_change":"2024-03-18T16:07:47.355115+0000","last_active":"2024-03-18T16:08:47.884896+0000","last_peered":"2024-03-18T16:08:47.884896+0000","last_clean":"2024-03-18T16:08:47.884896+0000","last_became_active":"2024-03-18T16:07:47.354744+0000","last_became_peered":"2024-03-18T16:07:47.354744+0000","last_unstale":"2024-03-18T16:08:47.884896+0000","last_undegraded":"2024-03-18T16:08:47.884896+0000","last_fullsized":"2024-03-18T16:08:47.884896+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:14:27.644891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.884782+0000","last_change":"2024-03-18T16:06:49.028943+0000","last_active":"2024-03-18T16:08:47.884782+0000","last_peered":"2024-03-18T16:08:47.884782+0000","last_clean":"2024-03-18T16:08:47.884782+0000","last_became_active":"2024-03-18T16:06:49.028453+0000","last_became_peered":"2024-03-18T16:06:49.028453+0000","last_unstale":"2024-03-18T16:08:47.884782+0000","last_undegraded":"2024-03-18T16:08:47.884782+0000","last_fullsized":"2024-03-18T16:08:47.884782+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:16:30.318649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":17,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.899433+0000","last_change":"2024-03-18T16:08:47.899048+0000","last_active":"2024-03-18T16:08:47.899433+0000","last_peered":"2024-03-18T16:08:47.899433+0000","last_clean":"2024-03-18T16:08:47.899433+0000","last_became_active":"2024-03-18T16:08:47.887283+0000","last_became_peered":"2024-03-18T16:08:47.887283+0000","last_unstale":"2024-03-18T16:08:47.899433+0000","last_undegraded":"2024-03-18T16:08:47.899433+0000","last_fullsized":"2024-03-18T16:08:47.899433+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:17:17.924210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.881500+0000","last_change":"2024-03-18T16:06:44.716284+0000","last_active":"2024-03-18T16:08:47.881500+0000","last_peered":"2024-03-18T16:08:47.881500+0000","last_clean":"2024-03-18T16:08:47.881500+0000","last_became_active":"2024-03-18T16:06:44.715911+0000","last_became_peered":"2024-03-18T16:06:44.715911+0000","last_unstale":"2024-03-18T16:08:47.881500+0000","last_undegraded":"2024-03-18T16:08:47.881500+0000","last_fullsized":"2024-03-18T16:08:47.881500+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:48:34.839514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.885009+0000","last_change":"2024-03-18T16:06:49.059061+0000","last_active":"2024-03-18T16:08:47.885009+0000","last_peered":"2024-03-18T16:08:47.885009+0000","last_clean":"2024-03-18T16:08:47.885009+0000","last_became_active":"2024-03-18T16:06:49.058619+0000","last_became_peered":"2024-03-18T16:06:49.058619+0000","last_unstale":"2024-03-18T16:08:47.885009+0000","last_undegraded":"2024-03-18T16:08:47.885009+0000","last_fullsized":"2024-03-18T16:08:47.885009+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:44:52.970785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883689+0000","last_change":"2024-03-18T16:06:43.715766+0000","last_active":"2024-03-18T16:08:47.883689+0000","last_peered":"2024-03-18T16:08:47.883689+0000","last_clean":"2024-03-18T16:08:47.883689+0000","last_became_active":"2024-03-18T16:06:43.715380+0000","last_became_peered":"2024-03-18T16:06:43.715380+0000","last_unstale":"2024-03-18T16:08:47.883689+0000","last_undegraded":"2024-03-18T16:08:47.883689+0000","last_fullsized":"2024-03-18T16:08:47.883689+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:13:44.195934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.880175+0000","last_change":"2024-03-18T16:06:44.728150+0000","last_active":"2024-03-18T16:08:47.880175+0000","last_peered":"2024-03-18T16:08:47.880175+0000","last_clean":"2024-03-18T16:08:47.880175+0000","last_became_active":"2024-03-18T16:06:44.727769+0000","last_became_peered":"2024-03-18T16:06:44.727769+0000","last_unstale":"2024-03-18T16:08:47.880175+0000","last_undegraded":"2024-03-18T16:08:47.880175+0000","last_fullsized":"2024-03-18T16:08:47.880175+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:28:16.023225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.961752+0000","last_change":"2024-03-18T16:06:49.033671+0000","last_active":"2024-03-18T16:08:47.961752+0000","last_peered":"2024-03-18T16:08:47.961752+0000","last_clean":"2024-03-18T16:08:47.961752+0000","last_became_active":"2024-03-18T16:06:49.033046+0000","last_became_peered":"2024-03-18T16:06:49.033046+0000","last_unstale":"2024-03-18T16:08:47.961752+0000","last_undegraded":"2024-03-18T16:08:47.961752+0000","last_fullsized":"2024-03-18T16:08:47.961752+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:26:37.020521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'5","reported_seq":195,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:09:56.309256+0000","last_change":"2024-03-18T16:06:43.712636+0000","last_active":"2024-03-18T16:09:56.309256+0000","last_peered":"2024-03-18T16:09:56.309256+0000","last_clean":"2024-03-18T16:09:56.309256+0000","last_became_active":"2024-03-18T16:06:43.712300+0000","last_became_peered":"2024-03-18T16:06:43.712300+0000","last_unstale":"2024-03-18T16:09:56.309256+0000","last_undegraded":"2024-03-18T16:09:56.309256+0000","last_fullsized":"2024-03-18T16:09:56.309256+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:48:20.386640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":158,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.880029+0000","last_change":"2024-03-18T16:06:44.731583+0000","last_active":"2024-03-18T16:08:47.880029+0000","last_peered":"2024-03-18T16:08:47.880029+0000","last_clean":"2024-03-18T16:08:47.880029+0000","last_became_active":"2024-03-18T16:06:44.731183+0000","last_became_peered":"2024-03-18T16:06:44.731183+0000","last_unstale":"2024-03-18T16:08:47.880029+0000","last_undegraded":"2024-03-18T16:08:47.880029+0000","last_fullsized":"2024-03-18T16:08:47.880029+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:23:05.098101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.962348+0000","last_change":"2024-03-18T16:06:49.060130+0000","last_active":"2024-03-18T16:08:47.962348+0000","last_peered":"2024-03-18T16:08:47.962348+0000","last_clean":"2024-03-18T16:08:47.962348+0000","last_became_active":"2024-03-18T16:06:49.059109+0000","last_became_peered":"2024-03-18T16:06:49.059109+0000","last_unstale":"2024-03-18T16:08:47.962348+0000","last_undegraded":"2024-03-18T16:08:47.962348+0000","last_fullsized":"2024-03-18T16:08:47.962348+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:41:40.008047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.885977+0000","last_change":"2024-03-18T16:06:44.705793+0000","last_active":"2024-03-18T16:08:47.885977+0000","last_peered":"2024-03-18T16:08:47.885977+0000","last_clean":"2024-03-18T16:08:47.885977+0000","last_became_active":"2024-03-18T16:06:44.705426+0000","last_became_peered":"2024-03-18T16:06:44.705426+0000","last_unstale":"2024-03-18T16:08:47.885977+0000","last_undegraded":"2024-03-18T16:08:47.885977+0000","last_fullsized":"2024-03-18T16:08:47.885977+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:14:31.198829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'5","reported_seq":199,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:10:00.226377+0000","last_change":"2024-03-18T16:06:43.721057+0000","last_active":"2024-03-18T16:10:00.226377+0000","last_peered":"2024-03-18T16:10:00.226377+0000","last_clean":"2024-03-18T16:10:00.226377+0000","last_became_active":"2024-03-18T16:06:43.720661+0000","last_became_peered":"2024-03-18T16:06:43.720661+0000","last_unstale":"2024-03-18T16:10:00.226377+0000","last_undegraded":"2024-03-18T16:10:00.226377+0000","last_fullsized":"2024-03-18T16:10:00.226377+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:08:18.050511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":163,"num_read_kb":163,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.961318+0000","last_change":"2024-03-18T16:07:47.361447+0000","last_active":"2024-03-18T16:08:47.961318+0000","last_peered":"2024-03-18T16:08:47.961318+0000","last_clean":"2024-03-18T16:08:47.961318+0000","last_became_active":"2024-03-18T16:07:47.353684+0000","last_became_peered":"2024-03-18T16:07:47.353684+0000","last_unstale":"2024-03-18T16:08:47.961318+0000","last_undegraded":"2024-03-18T16:08:47.961318+0000","last_fullsized":"2024-03-18T16:08:47.961318+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:59:53.025620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,10],"acting":[7,5,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.885471+0000","last_change":"2024-03-18T16:06:44.725555+0000","last_active":"2024-03-18T16:08:47.885471+0000","last_peered":"2024-03-18T16:08:47.885471+0000","last_clean":"2024-03-18T16:08:47.885471+0000","last_became_active":"2024-03-18T16:06:44.719880+0000","last_became_peered":"2024-03-18T16:06:44.719880+0000","last_unstale":"2024-03-18T16:08:47.885471+0000","last_undegraded":"2024-03-18T16:08:47.885471+0000","last_fullsized":"2024-03-18T16:08:47.885471+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:45:17.830876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883578+0000","last_change":"2024-03-18T16:06:43.700814+0000","last_active":"2024-03-18T16:08:47.883578+0000","last_peered":"2024-03-18T16:08:47.883578+0000","last_clean":"2024-03-18T16:08:47.883578+0000","last_became_active":"2024-03-18T16:06:43.698902+0000","last_became_peered":"2024-03-18T16:06:43.698902+0000","last_unstale":"2024-03-18T16:08:47.883578+0000","last_undegraded":"2024-03-18T16:08:47.883578+0000","last_fullsized":"2024-03-18T16:08:47.883578+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:20:20.100033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":354,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.960035+0000","last_change":"2024-03-18T16:05:22.379397+0000","last_active":"2024-03-18T16:08:47.960035+0000","last_peered":"2024-03-18T16:08:47.960035+0000","last_clean":"2024-03-18T16:08:47.960035+0000","last_became_active":"2024-03-18T16:05:22.378830+0000","last_became_peered":"2024-03-18T16:05:22.378830+0000","last_unstale":"2024-03-18T16:08:47.960035+0000","last_undegraded":"2024-03-18T16:08:47.960035+0000","last_fullsized":"2024-03-18T16:08:47.960035+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:01:30.048984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:01:30.048984+0000","last_clean_scrub_stamp":"2024-03-18T16:01:30.048984+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:35:45.796495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.885230+0000","last_change":"2024-03-18T16:06:49.058931+0000","last_active":"2024-03-18T16:08:47.885230+0000","last_peered":"2024-03-18T16:08:47.885230+0000","last_clean":"2024-03-18T16:08:47.885230+0000","last_became_active":"2024-03-18T16:06:49.058497+0000","last_became_peered":"2024-03-18T16:06:49.058497+0000","last_unstale":"2024-03-18T16:08:47.885230+0000","last_undegraded":"2024-03-18T16:08:47.885230+0000","last_fullsized":"2024-03-18T16:08:47.885230+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:41:51.874582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.961259+0000","last_change":"2024-03-18T16:06:44.710332+0000","last_active":"2024-03-18T16:08:47.961259+0000","last_peered":"2024-03-18T16:08:47.961259+0000","last_clean":"2024-03-18T16:08:47.961259+0000","last_became_active":"2024-03-18T16:06:44.709952+0000","last_became_peered":"2024-03-18T16:06:44.709952+0000","last_unstale":"2024-03-18T16:08:47.961259+0000","last_undegraded":"2024-03-18T16:08:47.961259+0000","last_fullsized":"2024-03-18T16:08:47.961259+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:24:26.653511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.960126+0000","last_change":"2024-03-18T16:06:43.708265+0000","last_active":"2024-03-18T16:08:47.960126+0000","last_peered":"2024-03-18T16:08:47.960126+0000","last_clean":"2024-03-18T16:08:47.960126+0000","last_became_active":"2024-03-18T16:06:43.707466+0000","last_became_peered":"2024-03-18T16:06:43.707466+0000","last_unstale":"2024-03-18T16:08:47.960126+0000","last_undegraded":"2024-03-18T16:08:47.960126+0000","last_fullsized":"2024-03-18T16:08:47.960126+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:45:40.065396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.882033+0000","last_change":"2024-03-18T16:06:49.022401+0000","last_active":"2024-03-18T16:08:47.882033+0000","last_peered":"2024-03-18T16:08:47.882033+0000","last_clean":"2024-03-18T16:08:47.882033+0000","last_became_active":"2024-03-18T16:06:49.021966+0000","last_became_peered":"2024-03-18T16:06:49.021966+0000","last_unstale":"2024-03-18T16:08:47.882033+0000","last_undegraded":"2024-03-18T16:08:47.882033+0000","last_fullsized":"2024-03-18T16:08:47.882033+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:04:12.434969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.885248+0000","last_change":"2024-03-18T16:06:43.713249+0000","last_active":"2024-03-18T16:08:47.885248+0000","last_peered":"2024-03-18T16:08:47.885248+0000","last_clean":"2024-03-18T16:08:47.885248+0000","last_became_active":"2024-03-18T16:06:43.712897+0000","last_became_peered":"2024-03-18T16:06:43.712897+0000","last_unstale":"2024-03-18T16:08:47.885248+0000","last_undegraded":"2024-03-18T16:08:47.885248+0000","last_fullsized":"2024-03-18T16:08:47.885248+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:20:04.642568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.961929+0000","last_change":"2024-03-18T16:06:44.710034+0000","last_active":"2024-03-18T16:08:47.961929+0000","last_peered":"2024-03-18T16:08:47.961929+0000","last_clean":"2024-03-18T16:08:47.961929+0000","last_became_active":"2024-03-18T16:06:44.708457+0000","last_became_peered":"2024-03-18T16:06:44.708457+0000","last_unstale":"2024-03-18T16:08:47.961929+0000","last_undegraded":"2024-03-18T16:08:47.961929+0000","last_fullsized":"2024-03-18T16:08:47.961929+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:40:00.504382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883057+0000","last_change":"2024-03-18T16:06:49.059345+0000","last_active":"2024-03-18T16:08:47.883057+0000","last_peered":"2024-03-18T16:08:47.883057+0000","last_clean":"2024-03-18T16:08:47.883057+0000","last_became_active":"2024-03-18T16:06:49.058675+0000","last_became_peered":"2024-03-18T16:06:49.058675+0000","last_unstale":"2024-03-18T16:08:47.883057+0000","last_undegraded":"2024-03-18T16:08:47.883057+0000","last_fullsized":"2024-03-18T16:08:47.883057+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:03:33.730074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"93'15","reported_seq":47,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:56.879106+0000","last_change":"2024-03-18T16:06:43.724589+0000","last_active":"2024-03-18T16:08:56.879106+0000","last_peered":"2024-03-18T16:08:56.879106+0000","last_clean":"2024-03-18T16:08:56.879106+0000","last_became_active":"2024-03-18T16:06:43.724125+0000","last_became_peered":"2024-03-18T16:06:43.724125+0000","last_unstale":"2024-03-18T16:08:56.879106+0000","last_undegraded":"2024-03-18T16:08:56.879106+0000","last_fullsized":"2024-03-18T16:08:56.879106+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:08:34.921118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.886205+0000","last_change":"2024-03-18T16:06:44.727162+0000","last_active":"2024-03-18T16:08:47.886205+0000","last_peered":"2024-03-18T16:08:47.886205+0000","last_clean":"2024-03-18T16:08:47.886205+0000","last_became_active":"2024-03-18T16:06:44.726552+0000","last_became_peered":"2024-03-18T16:06:44.726552+0000","last_unstale":"2024-03-18T16:08:47.886205+0000","last_undegraded":"2024-03-18T16:08:47.886205+0000","last_fullsized":"2024-03-18T16:08:47.886205+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:36:57.974999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883803+0000","last_change":"2024-03-18T16:06:49.008434+0000","last_active":"2024-03-18T16:08:47.883803+0000","last_peered":"2024-03-18T16:08:47.883803+0000","last_clean":"2024-03-18T16:08:47.883803+0000","last_became_active":"2024-03-18T16:06:49.007944+0000","last_became_peered":"2024-03-18T16:06:49.007944+0000","last_unstale":"2024-03-18T16:08:47.883803+0000","last_undegraded":"2024-03-18T16:08:47.883803+0000","last_fullsized":"2024-03-18T16:08:47.883803+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:01:24.450371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883928+0000","last_change":"2024-03-18T16:06:43.711797+0000","last_active":"2024-03-18T16:08:47.883928+0000","last_peered":"2024-03-18T16:08:47.883928+0000","last_clean":"2024-03-18T16:08:47.883928+0000","last_became_active":"2024-03-18T16:06:43.711493+0000","last_became_peered":"2024-03-18T16:06:43.711493+0000","last_unstale":"2024-03-18T16:08:47.883928+0000","last_undegraded":"2024-03-18T16:08:47.883928+0000","last_fullsized":"2024-03-18T16:08:47.883928+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:50:29.258166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.961906+0000","last_change":"2024-03-18T16:06:44.735910+0000","last_active":"2024-03-18T16:08:47.961906+0000","last_peered":"2024-03-18T16:08:47.961906+0000","last_clean":"2024-03-18T16:08:47.961906+0000","last_became_active":"2024-03-18T16:06:44.735504+0000","last_became_peered":"2024-03-18T16:06:44.735504+0000","last_unstale":"2024-03-18T16:08:47.961906+0000","last_undegraded":"2024-03-18T16:08:47.961906+0000","last_fullsized":"2024-03-18T16:08:47.961906+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:02:18.538217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883708+0000","last_change":"2024-03-18T16:06:49.059130+0000","last_active":"2024-03-18T16:08:47.883708+0000","last_peered":"2024-03-18T16:08:47.883708+0000","last_clean":"2024-03-18T16:08:47.883708+0000","last_became_active":"2024-03-18T16:06:49.058701+0000","last_became_peered":"2024-03-18T16:06:49.058701+0000","last_unstale":"2024-03-18T16:08:47.883708+0000","last_undegraded":"2024-03-18T16:08:47.883708+0000","last_fullsized":"2024-03-18T16:08:47.883708+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:54:52.842786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.959932+0000","last_change":"2024-03-18T16:06:43.714414+0000","last_active":"2024-03-18T16:08:47.959932+0000","last_peered":"2024-03-18T16:08:47.959932+0000","last_clean":"2024-03-18T16:08:47.959932+0000","last_became_active":"2024-03-18T16:06:43.714029+0000","last_became_peered":"2024-03-18T16:06:43.714029+0000","last_unstale":"2024-03-18T16:08:47.959932+0000","last_undegraded":"2024-03-18T16:08:47.959932+0000","last_fullsized":"2024-03-18T16:08:47.959932+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:16:39.060378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.884429+0000","last_change":"2024-03-18T16:06:44.707491+0000","last_active":"2024-03-18T16:08:47.884429+0000","last_peered":"2024-03-18T16:08:47.884429+0000","last_clean":"2024-03-18T16:08:47.884429+0000","last_became_active":"2024-03-18T16:06:44.707157+0000","last_became_peered":"2024-03-18T16:06:44.707157+0000","last_unstale":"2024-03-18T16:08:47.884429+0000","last_undegraded":"2024-03-18T16:08:47.884429+0000","last_fullsized":"2024-03-18T16:08:47.884429+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:19:08.132911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.885895+0000","last_change":"2024-03-18T16:07:47.411002+0000","last_active":"2024-03-18T16:08:47.885895+0000","last_peered":"2024-03-18T16:08:47.885895+0000","last_clean":"2024-03-18T16:08:47.885895+0000","last_became_active":"2024-03-18T16:07:47.410575+0000","last_became_peered":"2024-03-18T16:07:47.410575+0000","last_unstale":"2024-03-18T16:08:47.885895+0000","last_undegraded":"2024-03-18T16:08:47.885895+0000","last_fullsized":"2024-03-18T16:08:47.885895+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:37:25.311205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,2,5],"acting":[8,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.884825+0000","last_change":"2024-03-18T16:06:43.717755+0000","last_active":"2024-03-18T16:08:47.884825+0000","last_peered":"2024-03-18T16:08:47.884825+0000","last_clean":"2024-03-18T16:08:47.884825+0000","last_became_active":"2024-03-18T16:06:43.717278+0000","last_became_peered":"2024-03-18T16:06:43.717278+0000","last_unstale":"2024-03-18T16:08:47.884825+0000","last_undegraded":"2024-03-18T16:08:47.884825+0000","last_fullsized":"2024-03-18T16:08:47.884825+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:48:09.390659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.881317+0000","last_change":"2024-03-18T16:06:44.730858+0000","last_active":"2024-03-18T16:08:47.881317+0000","last_peered":"2024-03-18T16:08:47.881317+0000","last_clean":"2024-03-18T16:08:47.881317+0000","last_became_active":"2024-03-18T16:06:44.730500+0000","last_became_peered":"2024-03-18T16:06:44.730500+0000","last_unstale":"2024-03-18T16:08:47.881317+0000","last_undegraded":"2024-03-18T16:08:47.881317+0000","last_fullsized":"2024-03-18T16:08:47.881317+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:22:03.590834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883615+0000","last_change":"2024-03-18T16:06:49.014717+0000","last_active":"2024-03-18T16:08:47.883615+0000","last_peered":"2024-03-18T16:08:47.883615+0000","last_clean":"2024-03-18T16:08:47.883615+0000","last_became_active":"2024-03-18T16:06:49.014313+0000","last_became_peered":"2024-03-18T16:06:49.014313+0000","last_unstale":"2024-03-18T16:08:47.883615+0000","last_undegraded":"2024-03-18T16:08:47.883615+0000","last_fullsized":"2024-03-18T16:08:47.883615+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:06:12.168888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"93'4","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:53.939577+0000","last_change":"2024-03-18T16:06:43.719496+0000","last_active":"2024-03-18T16:08:53.939577+0000","last_peered":"2024-03-18T16:08:53.939577+0000","last_clean":"2024-03-18T16:08:53.939577+0000","last_became_active":"2024-03-18T16:06:43.719139+0000","last_became_peered":"2024-03-18T16:06:43.719139+0000","last_unstale":"2024-03-18T16:08:53.939577+0000","last_undegraded":"2024-03-18T16:08:53.939577+0000","last_fullsized":"2024-03-18T16:08:53.939577+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:10:01.250923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.886017+0000","last_change":"2024-03-18T16:06:44.737792+0000","last_active":"2024-03-18T16:08:47.886017+0000","last_peered":"2024-03-18T16:08:47.886017+0000","last_clean":"2024-03-18T16:08:47.886017+0000","last_became_active":"2024-03-18T16:06:44.736311+0000","last_became_peered":"2024-03-18T16:06:44.736311+0000","last_unstale":"2024-03-18T16:08:47.886017+0000","last_undegraded":"2024-03-18T16:08:47.886017+0000","last_fullsized":"2024-03-18T16:08:47.886017+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:43:56.110205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883670+0000","last_change":"2024-03-18T16:06:49.058893+0000","last_active":"2024-03-18T16:08:47.883670+0000","last_peered":"2024-03-18T16:08:47.883670+0000","last_clean":"2024-03-18T16:08:47.883670+0000","last_became_active":"2024-03-18T16:06:49.058477+0000","last_became_peered":"2024-03-18T16:06:49.058477+0000","last_unstale":"2024-03-18T16:08:47.883670+0000","last_undegraded":"2024-03-18T16:08:47.883670+0000","last_fullsized":"2024-03-18T16:08:47.883670+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:50:51.890348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.884612+0000","last_change":"2024-03-18T16:06:43.703073+0000","last_active":"2024-03-18T16:08:47.884612+0000","last_peered":"2024-03-18T16:08:47.884612+0000","last_clean":"2024-03-18T16:08:47.884612+0000","last_became_active":"2024-03-18T16:06:43.702708+0000","last_became_peered":"2024-03-18T16:06:43.702708+0000","last_unstale":"2024-03-18T16:08:47.884612+0000","last_undegraded":"2024-03-18T16:08:47.884612+0000","last_fullsized":"2024-03-18T16:08:47.884612+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:38:41.454458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.960757+0000","last_change":"2024-03-18T16:06:44.713622+0000","last_active":"2024-03-18T16:08:47.960757+0000","last_peered":"2024-03-18T16:08:47.960757+0000","last_clean":"2024-03-18T16:08:47.960757+0000","last_became_active":"2024-03-18T16:06:44.713160+0000","last_became_peered":"2024-03-18T16:06:44.713160+0000","last_unstale":"2024-03-18T16:08:47.960757+0000","last_undegraded":"2024-03-18T16:08:47.960757+0000","last_fullsized":"2024-03-18T16:08:47.960757+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:49:21.886543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.959925+0000","last_change":"2024-03-18T16:07:47.411958+0000","last_active":"2024-03-18T16:08:47.959925+0000","last_peered":"2024-03-18T16:08:47.959925+0000","last_clean":"2024-03-18T16:08:47.959925+0000","last_became_active":"2024-03-18T16:07:47.411433+0000","last_became_peered":"2024-03-18T16:07:47.411433+0000","last_unstale":"2024-03-18T16:08:47.959925+0000","last_undegraded":"2024-03-18T16:08:47.959925+0000","last_fullsized":"2024-03-18T16:08:47.959925+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:58:47.853897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6,7],"acting":[1,8,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"93'6","reported_seq":199,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:09:57.146786+0000","last_change":"2024-03-18T16:06:43.717161+0000","last_active":"2024-03-18T16:09:57.146786+0000","last_peered":"2024-03-18T16:09:57.146786+0000","last_clean":"2024-03-18T16:09:57.146786+0000","last_became_active":"2024-03-18T16:06:43.716657+0000","last_became_peered":"2024-03-18T16:06:43.716657+0000","last_unstale":"2024-03-18T16:09:57.146786+0000","last_undegraded":"2024-03-18T16:09:57.146786+0000","last_fullsized":"2024-03-18T16:09:57.146786+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:23:48.194779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1480,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":162,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.961120+0000","last_change":"2024-03-18T16:06:44.709777+0000","last_active":"2024-03-18T16:08:47.961120+0000","last_peered":"2024-03-18T16:08:47.961120+0000","last_clean":"2024-03-18T16:08:47.961120+0000","last_became_active":"2024-03-18T16:06:44.709409+0000","last_became_peered":"2024-03-18T16:06:44.709409+0000","last_unstale":"2024-03-18T16:08:47.961120+0000","last_undegraded":"2024-03-18T16:08:47.961120+0000","last_fullsized":"2024-03-18T16:08:47.961120+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:37:12.288262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.961539+0000","last_change":"2024-03-18T16:06:49.040872+0000","last_active":"2024-03-18T16:08:47.961539+0000","last_peered":"2024-03-18T16:08:47.961539+0000","last_clean":"2024-03-18T16:08:47.961539+0000","last_became_active":"2024-03-18T16:06:49.039807+0000","last_became_peered":"2024-03-18T16:06:49.039807+0000","last_unstale":"2024-03-18T16:08:47.961539+0000","last_undegraded":"2024-03-18T16:08:47.961539+0000","last_fullsized":"2024-03-18T16:08:47.961539+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:09:27.045450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"89'3","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.886254+0000","last_change":"2024-03-18T16:06:43.723675+0000","last_active":"2024-03-18T16:08:47.886254+0000","last_peered":"2024-03-18T16:08:47.886254+0000","last_clean":"2024-03-18T16:08:47.886254+0000","last_became_active":"2024-03-18T16:06:43.723334+0000","last_became_peered":"2024-03-18T16:06:43.723334+0000","last_unstale":"2024-03-18T16:08:47.886254+0000","last_undegraded":"2024-03-18T16:08:47.886254+0000","last_fullsized":"2024-03-18T16:08:47.886254+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:01:36.553362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.961094+0000","last_change":"2024-03-18T16:07:47.359134+0000","last_active":"2024-03-18T16:08:47.961094+0000","last_peered":"2024-03-18T16:08:47.961094+0000","last_clean":"2024-03-18T16:08:47.961094+0000","last_became_active":"2024-03-18T16:07:47.358692+0000","last_became_peered":"2024-03-18T16:07:47.358692+0000","last_unstale":"2024-03-18T16:08:47.961094+0000","last_undegraded":"2024-03-18T16:08:47.961094+0000","last_fullsized":"2024-03-18T16:08:47.961094+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:48:22.929034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.885552+0000","last_change":"2024-03-18T16:06:49.011981+0000","last_active":"2024-03-18T16:08:47.885552+0000","last_peered":"2024-03-18T16:08:47.885552+0000","last_clean":"2024-03-18T16:08:47.885552+0000","last_became_active":"2024-03-18T16:06:49.011621+0000","last_became_peered":"2024-03-18T16:06:49.011621+0000","last_unstale":"2024-03-18T16:08:47.885552+0000","last_undegraded":"2024-03-18T16:08:47.885552+0000","last_fullsized":"2024-03-18T16:08:47.885552+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:11:42.382721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.962696+0000","last_change":"2024-03-18T16:06:43.725404+0000","last_active":"2024-03-18T16:08:47.962696+0000","last_peered":"2024-03-18T16:08:47.962696+0000","last_clean":"2024-03-18T16:08:47.962696+0000","last_became_active":"2024-03-18T16:06:43.724117+0000","last_became_peered":"2024-03-18T16:06:43.724117+0000","last_unstale":"2024-03-18T16:08:47.962696+0000","last_undegraded":"2024-03-18T16:08:47.962696+0000","last_fullsized":"2024-03-18T16:08:47.962696+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:58:43.243070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.961302+0000","last_change":"2024-03-18T16:06:44.709979+0000","last_active":"2024-03-18T16:08:47.961302+0000","last_peered":"2024-03-18T16:08:47.961302+0000","last_clean":"2024-03-18T16:08:47.961302+0000","last_became_active":"2024-03-18T16:06:44.709591+0000","last_became_peered":"2024-03-18T16:06:44.709591+0000","last_unstale":"2024-03-18T16:08:47.961302+0000","last_undegraded":"2024-03-18T16:08:47.961302+0000","last_fullsized":"2024-03-18T16:08:47.961302+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:56:37.537248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.881778+0000","last_change":"2024-03-18T16:07:47.354636+0000","last_active":"2024-03-18T16:08:47.881778+0000","last_peered":"2024-03-18T16:08:47.881778+0000","last_clean":"2024-03-18T16:08:47.881778+0000","last_became_active":"2024-03-18T16:07:47.354219+0000","last_became_peered":"2024-03-18T16:07:47.354219+0000","last_unstale":"2024-03-18T16:08:47.881778+0000","last_undegraded":"2024-03-18T16:08:47.881778+0000","last_fullsized":"2024-03-18T16:08:47.881778+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:35:48.094404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,9],"acting":[11,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883718+0000","last_change":"2024-03-18T16:06:44.714836+0000","last_active":"2024-03-18T16:08:47.883718+0000","last_peered":"2024-03-18T16:08:47.883718+0000","last_clean":"2024-03-18T16:08:47.883718+0000","last_became_active":"2024-03-18T16:06:44.714454+0000","last_became_peered":"2024-03-18T16:06:44.714454+0000","last_unstale":"2024-03-18T16:08:47.883718+0000","last_undegraded":"2024-03-18T16:08:47.883718+0000","last_fullsized":"2024-03-18T16:08:47.883718+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:05:24.532238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883502+0000","last_change":"2024-03-18T16:06:44.733183+0000","last_active":"2024-03-18T16:08:47.883502+0000","last_peered":"2024-03-18T16:08:47.883502+0000","last_clean":"2024-03-18T16:08:47.883502+0000","last_became_active":"2024-03-18T16:06:44.732909+0000","last_became_peered":"2024-03-18T16:06:44.732909+0000","last_unstale":"2024-03-18T16:08:47.883502+0000","last_undegraded":"2024-03-18T16:08:47.883502+0000","last_fullsized":"2024-03-18T16:08:47.883502+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:28:07.636692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.961804+0000","last_change":"2024-03-18T16:06:44.731050+0000","last_active":"2024-03-18T16:08:47.961804+0000","last_peered":"2024-03-18T16:08:47.961804+0000","last_clean":"2024-03-18T16:08:47.961804+0000","last_became_active":"2024-03-18T16:06:44.730634+0000","last_became_peered":"2024-03-18T16:06:44.730634+0000","last_unstale":"2024-03-18T16:08:47.961804+0000","last_undegraded":"2024-03-18T16:08:47.961804+0000","last_fullsized":"2024-03-18T16:08:47.961804+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:07:18.619130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.962047+0000","last_change":"2024-03-18T16:06:44.735190+0000","last_active":"2024-03-18T16:08:47.962047+0000","last_peered":"2024-03-18T16:08:47.962047+0000","last_clean":"2024-03-18T16:08:47.962047+0000","last_became_active":"2024-03-18T16:06:44.734312+0000","last_became_peered":"2024-03-18T16:06:44.734312+0000","last_unstale":"2024-03-18T16:08:47.962047+0000","last_undegraded":"2024-03-18T16:08:47.962047+0000","last_fullsized":"2024-03-18T16:08:47.962047+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:59:50.209515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.881867+0000","last_change":"2024-03-18T16:06:44.705385+0000","last_active":"2024-03-18T16:08:47.881867+0000","last_peered":"2024-03-18T16:08:47.881867+0000","last_clean":"2024-03-18T16:08:47.881867+0000","last_became_active":"2024-03-18T16:06:44.705011+0000","last_became_peered":"2024-03-18T16:06:44.705011+0000","last_unstale":"2024-03-18T16:08:47.881867+0000","last_undegraded":"2024-03-18T16:08:47.881867+0000","last_fullsized":"2024-03-18T16:08:47.881867+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:50:02.152851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.885287+0000","last_change":"2024-03-18T16:06:44.709621+0000","last_active":"2024-03-18T16:08:47.885287+0000","last_peered":"2024-03-18T16:08:47.885287+0000","last_clean":"2024-03-18T16:08:47.885287+0000","last_became_active":"2024-03-18T16:06:44.709275+0000","last_became_peered":"2024-03-18T16:06:44.709275+0000","last_unstale":"2024-03-18T16:08:47.885287+0000","last_undegraded":"2024-03-18T16:08:47.885287+0000","last_fullsized":"2024-03-18T16:08:47.885287+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:41:17.346661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.886301+0000","last_change":"2024-03-18T16:06:44.719827+0000","last_active":"2024-03-18T16:08:47.886301+0000","last_peered":"2024-03-18T16:08:47.886301+0000","last_clean":"2024-03-18T16:08:47.886301+0000","last_became_active":"2024-03-18T16:06:44.719432+0000","last_became_peered":"2024-03-18T16:06:44.719432+0000","last_unstale":"2024-03-18T16:08:47.886301+0000","last_undegraded":"2024-03-18T16:08:47.886301+0000","last_fullsized":"2024-03-18T16:08:47.886301+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:13:42.344211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.885803+0000","last_change":"2024-03-18T16:06:44.714526+0000","last_active":"2024-03-18T16:08:47.885803+0000","last_peered":"2024-03-18T16:08:47.885803+0000","last_clean":"2024-03-18T16:08:47.885803+0000","last_became_active":"2024-03-18T16:06:44.714184+0000","last_became_peered":"2024-03-18T16:06:44.714184+0000","last_unstale":"2024-03-18T16:08:47.885803+0000","last_undegraded":"2024-03-18T16:08:47.885803+0000","last_fullsized":"2024-03-18T16:08:47.885803+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:31:07.917143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.884927+0000","last_change":"2024-03-18T16:06:44.730905+0000","last_active":"2024-03-18T16:08:47.884927+0000","last_peered":"2024-03-18T16:08:47.884927+0000","last_clean":"2024-03-18T16:08:47.884927+0000","last_became_active":"2024-03-18T16:06:44.729002+0000","last_became_peered":"2024-03-18T16:06:44.729002+0000","last_unstale":"2024-03-18T16:08:47.884927+0000","last_undegraded":"2024-03-18T16:08:47.884927+0000","last_fullsized":"2024-03-18T16:08:47.884927+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:49:18.867571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.960207+0000","last_change":"2024-03-18T16:06:44.727608+0000","last_active":"2024-03-18T16:08:47.960207+0000","last_peered":"2024-03-18T16:08:47.960207+0000","last_clean":"2024-03-18T16:08:47.960207+0000","last_became_active":"2024-03-18T16:06:44.727096+0000","last_became_peered":"2024-03-18T16:06:44.727096+0000","last_unstale":"2024-03-18T16:08:47.960207+0000","last_undegraded":"2024-03-18T16:08:47.960207+0000","last_fullsized":"2024-03-18T16:08:47.960207+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:08:20.767443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.962439+0000","last_change":"2024-03-18T16:06:44.731823+0000","last_active":"2024-03-18T16:08:47.962439+0000","last_peered":"2024-03-18T16:08:47.962439+0000","last_clean":"2024-03-18T16:08:47.962439+0000","last_became_active":"2024-03-18T16:06:44.731429+0000","last_became_peered":"2024-03-18T16:06:44.731429+0000","last_unstale":"2024-03-18T16:08:47.962439+0000","last_undegraded":"2024-03-18T16:08:47.962439+0000","last_fullsized":"2024-03-18T16:08:47.962439+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:36:28.752466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.881596+0000","last_change":"2024-03-18T16:06:44.714354+0000","last_active":"2024-03-18T16:08:47.881596+0000","last_peered":"2024-03-18T16:08:47.881596+0000","last_clean":"2024-03-18T16:08:47.881596+0000","last_became_active":"2024-03-18T16:06:44.713969+0000","last_became_peered":"2024-03-18T16:06:44.713969+0000","last_unstale":"2024-03-18T16:08:47.881596+0000","last_undegraded":"2024-03-18T16:08:47.881596+0000","last_fullsized":"2024-03-18T16:08:47.881596+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:18:49.502178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":73629,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1632,"num_read_kb":901,"num_write":258,"num_write_kb":154,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":90,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":589824,"data_stored":220887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":223,"ondisk_log_size":223,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383735,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36392,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708760,"statfs":{"total":95995035648,"available":95957770240,"internally_reserved":0,"allocated":1077248,"data_stored":520607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645370,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37064,"kb_used_data":1728,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708088,"statfs":{"total":95995035648,"available":95957082112,"internally_reserved":0,"allocated":1769472,"data_stored":1177484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808895,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1089536,"data_stored":520137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070535,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36392,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708760,"statfs":{"total":95995035648,"available":95957770240,"internally_reserved":0,"allocated":1081344,"data_stored":520493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332171,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28668,"kb_used_data":1644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716484,"statfs":{"total":95995035648,"available":95965679616,"internally_reserved":0,"allocated":1683456,"data_stored":1117153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495697,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32224,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712928,"statfs":{"total":95995035648,"available":95962038272,"internally_reserved":0,"allocated":1064960,"data_stored":515641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574957,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36448,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708704,"statfs":{"total":95995035648,"available":95957712896,"internally_reserved":0,"allocated":1138688,"data_stored":583036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738483,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28088,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1089536,"data_stored":517407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411433,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36460,"kb_used_data":1124,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708692,"statfs":{"total":95995035648,"available":95957700608,"internally_reserved":0,"allocated":1150976,"data_stored":584030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182499,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1077248,"data_stored":515349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920861,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36392,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708760,"statfs":{"total":95995035648,"available":95957770240,"internally_reserved":0,"allocated":1073152,"data_stored":519715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757336,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36996,"kb_used_data":1652,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708156,"statfs":{"total":95995035648,"available":95957151744,"internally_reserved":0,"allocated":1691648,"data_stored":1112735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":68117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":69111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":68101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T16:10:01.902 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-03-18T16:10:01.902 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-18T16:10:02.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.158+0000 7f1bfbe00640 1 -- 172.21.15.39:0/4035050503 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1bfc1015f0 msgr2=0x7f1bfc101a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:02.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.158+0000 7f1bfbe00640 1 --2- 172.21.15.39:0/4035050503 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1bfc1015f0 0x7f1bfc101a10 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f1bec009900 tx=0x7f1bec02bc30 comp rx=0 tx=0).stop 2024-03-18T16:10:02.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.158+0000 7f1bfbe00640 1 -- 172.21.15.39:0/4035050503 shutdown_connections 2024-03-18T16:10:02.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.158+0000 7f1bfbe00640 1 --2- 172.21.15.39:0/4035050503 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1bfc1031a0 0x7f1bfc109a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:02.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.158+0000 7f1bfbe00640 1 --2- 172.21.15.39:0/4035050503 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1bfc1027e0 0x7f1bfc102c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:02.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.158+0000 7f1bfbe00640 1 --2- 172.21.15.39:0/4035050503 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1bfc1015f0 0x7f1bfc101a10 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:02.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.158+0000 7f1bfbe00640 1 -- 172.21.15.39:0/4035050503 >> 172.21.15.39:0/4035050503 conn(0x7f1bfc0fcb40 msgr2=0x7f1bfc0fefa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:02.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.159+0000 7f1bfbe00640 1 -- 172.21.15.39:0/4035050503 shutdown_connections 2024-03-18T16:10:02.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.159+0000 7f1bfbe00640 1 -- 172.21.15.39:0/4035050503 wait complete. 2024-03-18T16:10:02.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.160+0000 7f1bfbe00640 1 Processor -- start 2024-03-18T16:10:02.162 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.160+0000 7f1bfbe00640 1 -- start start 2024-03-18T16:10:02.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.161+0000 7f1bfbe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1bfc1015f0 0x7f1bfc199b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:02.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.161+0000 7f1bfbe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1bfc1027e0 0x7f1bfc19a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:02.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.161+0000 7f1bfbe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1bfc1031a0 0x7f1bfc1a10f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:02.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.161+0000 7f1bfbe00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f1bfc074ec0 con 0x7f1bfc1031a0 2024-03-18T16:10:02.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.161+0000 7f1bfbe00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f1bfc074d40 con 0x7f1bfc1027e0 2024-03-18T16:10:02.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.161+0000 7f1bfbe00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f1bfc075040 con 0x7f1bfc1015f0 2024-03-18T16:10:02.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.161+0000 7f1bfa000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1bfc1027e0 0x7f1bfc19a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:02.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.161+0000 7f1bfaa00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1bfc1015f0 0x7f1bfc199b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:02.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.161+0000 7f1bfa000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1bfc1027e0 0x7f1bfc19a080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:36488/0 (socket says 172.21.15.39:36488) 2024-03-18T16:10:02.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.161+0000 7f1bfaa00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1bfc1015f0 0x7f1bfc199b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:49810/0 (socket says 172.21.15.39:49810) 2024-03-18T16:10:02.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.161+0000 7f1bfa000640 1 -- 172.21.15.39:0/173704248 learned_addr learned my addr 172.21.15.39:0/173704248 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:10:02.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.161+0000 7f1bfb400640 1 --2- 172.21.15.39:0/173704248 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1bfc1031a0 0x7f1bfc1a10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:02.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.162+0000 7f1bfaa00640 1 -- 172.21.15.39:0/173704248 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1bfc1027e0 msgr2=0x7f1bfc19a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:02.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.162+0000 7f1bfaa00640 1 --2- 172.21.15.39:0/173704248 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1bfc1027e0 0x7f1bfc19a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:02.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.162+0000 7f1bfaa00640 1 -- 172.21.15.39:0/173704248 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1bfc1031a0 msgr2=0x7f1bfc1a10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:02.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.162+0000 7f1bfaa00640 1 --2- 172.21.15.39:0/173704248 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1bfc1031a0 0x7f1bfc1a10f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:02.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.162+0000 7f1bfaa00640 1 -- 172.21.15.39:0/173704248 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bfc1a1760 con 0x7f1bfc1015f0 2024-03-18T16:10:02.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.162+0000 7f1bfa000640 1 --2- 172.21.15.39:0/173704248 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1bfc1027e0 0x7f1bfc19a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:10:02.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.162+0000 7f1bfaa00640 1 --2- 172.21.15.39:0/173704248 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1bfc1015f0 0x7f1bfc199b40 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f1bec009230 tx=0x7f1bec02c5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:02.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.163+0000 7f1bf3400640 1 -- 172.21.15.39:0/173704248 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bec043020 con 0x7f1bfc1015f0 2024-03-18T16:10:02.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.163+0000 7f1bf3400640 1 -- 172.21.15.39:0/173704248 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1bec0054e0 con 0x7f1bfc1015f0 2024-03-18T16:10:02.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.163+0000 7f1bfbe00640 1 -- 172.21.15.39:0/173704248 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bfc1a19f0 con 0x7f1bfc1015f0 2024-03-18T16:10:02.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.163+0000 7f1bf3400640 1 -- 172.21.15.39:0/173704248 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bec003b60 con 0x7f1bfc1015f0 2024-03-18T16:10:02.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.163+0000 7f1bfbe00640 1 -- 172.21.15.39:0/173704248 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bfc1a1f60 con 0x7f1bfc1015f0 2024-03-18T16:10:02.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.164+0000 7f1bfbe00640 1 -- 172.21.15.39:0/173704248 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1bfc108130 con 0x7f1bfc1015f0 2024-03-18T16:10:02.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.166+0000 7f1bf3400640 1 -- 172.21.15.39:0/173704248 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f1bec046020 con 0x7f1bfc1015f0 2024-03-18T16:10:02.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.167+0000 7f1bf3400640 1 --2- 172.21.15.39:0/173704248 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1be00a2880 0x7f1be00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:02.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.167+0000 7f1bf3400640 1 -- 172.21.15.39:0/173704248 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f1bec0f0da0 con 0x7f1bfc1015f0 2024-03-18T16:10:02.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.167+0000 7f1bfa000640 1 --2- 172.21.15.39:0/173704248 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1be00a2880 0x7f1be00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:02.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.168+0000 7f1bfa000640 1 --2- 172.21.15.39:0/173704248 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1be00a2880 0x7f1be00a4d40 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc19b050 tx=0x7f1be8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:02.172 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.171+0000 7f1bf3400640 1 -- 172.21.15.39:0/173704248 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1bec04d020 con 0x7f1bfc1015f0 2024-03-18T16:10:02.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.375+0000 7f1bfbe00640 1 -- 172.21.15.39:0/173704248 --> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1bfc061c20 con 0x7f1be00a2880 2024-03-18T16:10:02.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.394+0000 7f1bf3400640 1 -- 172.21.15.39:0/173704248 <== mgr.14152 v2:172.21.15.39:6800/2749214286 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340542 (secure 0 0 0) 0x7f1bfc061c20 con 0x7f1be00a2880 2024-03-18T16:10:02.400 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:10:02.402 INFO:teuthology.orchestra.run.smithi039.stderr:dumped all 2024-03-18T16:10:02.403 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.399+0000 7f1bfbe00640 1 -- 172.21.15.39:0/173704248 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1be00a2880 msgr2=0x7f1be00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:02.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.399+0000 7f1bfbe00640 1 --2- 172.21.15.39:0/173704248 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1be00a2880 0x7f1be00a4d40 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f1bfc19b050 tx=0x7f1be8005f90 comp rx=0 tx=0).stop 2024-03-18T16:10:02.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.400+0000 7f1bfbe00640 1 -- 172.21.15.39:0/173704248 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1bfc1015f0 msgr2=0x7f1bfc199b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:02.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.400+0000 7f1bfbe00640 1 --2- 172.21.15.39:0/173704248 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1bfc1015f0 0x7f1bfc199b40 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f1bec009230 tx=0x7f1bec02c5d0 comp rx=0 tx=0).stop 2024-03-18T16:10:02.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.400+0000 7f1bfbe00640 1 -- 172.21.15.39:0/173704248 shutdown_connections 2024-03-18T16:10:02.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.400+0000 7f1bfbe00640 1 --2- 172.21.15.39:0/173704248 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1bfc1027e0 0x7f1bfc19a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:02.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.400+0000 7f1bfbe00640 1 --2- 172.21.15.39:0/173704248 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1be00a2880 0x7f1be00a4d40 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:02.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.400+0000 7f1bfbe00640 1 --2- 172.21.15.39:0/173704248 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1bfc1031a0 0x7f1bfc1a10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:02.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.400+0000 7f1bfbe00640 1 --2- 172.21.15.39:0/173704248 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1bfc1015f0 0x7f1bfc199b40 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:02.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.400+0000 7f1bfbe00640 1 -- 172.21.15.39:0/173704248 >> 172.21.15.39:0/173704248 conn(0x7f1bfc0fcb40 msgr2=0x7f1bfc0fe6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:02.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.400+0000 7f1bfbe00640 1 -- 172.21.15.39:0/173704248 shutdown_connections 2024-03-18T16:10:02.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.400+0000 7f1bfbe00640 1 -- 172.21.15.39:0/173704248 wait complete. 2024-03-18T16:10:02.420 INFO:teuthology.orchestra.run.smithi039.stdout:{"pg_ready":true,"pg_map":{"version":421,"stamp":"2024-03-18T16:10:01.574276+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":664231,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1758,"num_read_kb":1010,"num_write":441,"num_write_kb":3094,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":1180943,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":862,"ondisk_log_size":862,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":401272,"kb_used_data":14636,"kb_used_omap":18,"kb_used_meta":386477,"kb_avail":1124540552,"statfs":{"total":1151940427776,"available":1151529525248,"internally_reserved":0,"allocated":14987264,"data_stored":8203787,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":395752763},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4934,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":108,"num_read_kb":62,"num_write":9,"num_write_kb":11,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005670"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.882222+0000","last_change":"2024-03-18T16:06:44.736511+0000","last_active":"2024-03-18T16:08:47.882222+0000","last_peered":"2024-03-18T16:08:47.882222+0000","last_clean":"2024-03-18T16:08:47.882222+0000","last_became_active":"2024-03-18T16:06:44.736182+0000","last_became_peered":"2024-03-18T16:06:44.736182+0000","last_unstale":"2024-03-18T16:08:47.882222+0000","last_undegraded":"2024-03-18T16:08:47.882222+0000","last_fullsized":"2024-03-18T16:08:47.882222+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:08:57.060268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.961793+0000","last_change":"2024-03-18T16:06:44.733658+0000","last_active":"2024-03-18T16:08:47.961793+0000","last_peered":"2024-03-18T16:08:47.961793+0000","last_clean":"2024-03-18T16:08:47.961793+0000","last_became_active":"2024-03-18T16:06:44.733279+0000","last_became_peered":"2024-03-18T16:06:44.733279+0000","last_unstale":"2024-03-18T16:08:47.961793+0000","last_undegraded":"2024-03-18T16:08:47.961793+0000","last_fullsized":"2024-03-18T16:08:47.961793+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:59:46.783025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.961451+0000","last_change":"2024-03-18T16:06:44.723949+0000","last_active":"2024-03-18T16:08:47.961451+0000","last_peered":"2024-03-18T16:08:47.961451+0000","last_clean":"2024-03-18T16:08:47.961451+0000","last_became_active":"2024-03-18T16:06:44.723614+0000","last_became_peered":"2024-03-18T16:06:44.723614+0000","last_unstale":"2024-03-18T16:08:47.961451+0000","last_undegraded":"2024-03-18T16:08:47.961451+0000","last_fullsized":"2024-03-18T16:08:47.961451+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:03:17.876097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883864+0000","last_change":"2024-03-18T16:06:44.711058+0000","last_active":"2024-03-18T16:08:47.883864+0000","last_peered":"2024-03-18T16:08:47.883864+0000","last_clean":"2024-03-18T16:08:47.883864+0000","last_became_active":"2024-03-18T16:06:44.708873+0000","last_became_peered":"2024-03-18T16:06:44.708873+0000","last_unstale":"2024-03-18T16:08:47.883864+0000","last_undegraded":"2024-03-18T16:08:47.883864+0000","last_fullsized":"2024-03-18T16:08:47.883864+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:58:08.182651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.884722+0000","last_change":"2024-03-18T16:06:44.721480+0000","last_active":"2024-03-18T16:08:47.884722+0000","last_peered":"2024-03-18T16:08:47.884722+0000","last_clean":"2024-03-18T16:08:47.884722+0000","last_became_active":"2024-03-18T16:06:44.721101+0000","last_became_peered":"2024-03-18T16:06:44.721101+0000","last_unstale":"2024-03-18T16:08:47.884722+0000","last_undegraded":"2024-03-18T16:08:47.884722+0000","last_fullsized":"2024-03-18T16:08:47.884722+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:19:43.341221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.879862+0000","last_change":"2024-03-18T16:06:44.716443+0000","last_active":"2024-03-18T16:08:47.879862+0000","last_peered":"2024-03-18T16:08:47.879862+0000","last_clean":"2024-03-18T16:08:47.879862+0000","last_became_active":"2024-03-18T16:06:44.715969+0000","last_became_peered":"2024-03-18T16:06:44.715969+0000","last_unstale":"2024-03-18T16:08:47.879862+0000","last_undegraded":"2024-03-18T16:08:47.879862+0000","last_fullsized":"2024-03-18T16:08:47.879862+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:57:14.714415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.960378+0000","last_change":"2024-03-18T16:06:44.711439+0000","last_active":"2024-03-18T16:08:47.960378+0000","last_peered":"2024-03-18T16:08:47.960378+0000","last_clean":"2024-03-18T16:08:47.960378+0000","last_became_active":"2024-03-18T16:06:44.711032+0000","last_became_peered":"2024-03-18T16:06:44.711032+0000","last_unstale":"2024-03-18T16:08:47.960378+0000","last_undegraded":"2024-03-18T16:08:47.960378+0000","last_fullsized":"2024-03-18T16:08:47.960378+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:19:56.516867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.881902+0000","last_change":"2024-03-18T16:07:47.358866+0000","last_active":"2024-03-18T16:08:47.881902+0000","last_peered":"2024-03-18T16:08:47.881902+0000","last_clean":"2024-03-18T16:08:47.881902+0000","last_became_active":"2024-03-18T16:07:47.357743+0000","last_became_peered":"2024-03-18T16:07:47.357743+0000","last_unstale":"2024-03-18T16:08:47.881902+0000","last_undegraded":"2024-03-18T16:08:47.881902+0000","last_fullsized":"2024-03-18T16:08:47.881902+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:13:12.361025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,10],"acting":[9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.960770+0000","last_change":"2024-03-18T16:06:44.710978+0000","last_active":"2024-03-18T16:08:47.960770+0000","last_peered":"2024-03-18T16:08:47.960770+0000","last_clean":"2024-03-18T16:08:47.960770+0000","last_became_active":"2024-03-18T16:06:44.710520+0000","last_became_peered":"2024-03-18T16:06:44.710520+0000","last_unstale":"2024-03-18T16:08:47.960770+0000","last_undegraded":"2024-03-18T16:08:47.960770+0000","last_fullsized":"2024-03-18T16:08:47.960770+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:54:53.234025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.880413+0000","last_change":"2024-03-18T16:06:44.716231+0000","last_active":"2024-03-18T16:08:47.880413+0000","last_peered":"2024-03-18T16:08:47.880413+0000","last_clean":"2024-03-18T16:08:47.880413+0000","last_became_active":"2024-03-18T16:06:44.714900+0000","last_became_peered":"2024-03-18T16:06:44.714900+0000","last_unstale":"2024-03-18T16:08:47.880413+0000","last_undegraded":"2024-03-18T16:08:47.880413+0000","last_fullsized":"2024-03-18T16:08:47.880413+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:38:54.964729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.961221+0000","last_change":"2024-03-18T16:06:44.710942+0000","last_active":"2024-03-18T16:08:47.961221+0000","last_peered":"2024-03-18T16:08:47.961221+0000","last_clean":"2024-03-18T16:08:47.961221+0000","last_became_active":"2024-03-18T16:06:44.710588+0000","last_became_peered":"2024-03-18T16:06:44.710588+0000","last_unstale":"2024-03-18T16:08:47.961221+0000","last_undegraded":"2024-03-18T16:08:47.961221+0000","last_fullsized":"2024-03-18T16:08:47.961221+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:43:19.320034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883838+0000","last_change":"2024-03-18T16:06:44.714763+0000","last_active":"2024-03-18T16:08:47.883838+0000","last_peered":"2024-03-18T16:08:47.883838+0000","last_clean":"2024-03-18T16:08:47.883838+0000","last_became_active":"2024-03-18T16:06:44.714031+0000","last_became_peered":"2024-03-18T16:06:44.714031+0000","last_unstale":"2024-03-18T16:08:47.883838+0000","last_undegraded":"2024-03-18T16:08:47.883838+0000","last_fullsized":"2024-03-18T16:08:47.883838+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:19:25.917093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.962631+0000","last_change":"2024-03-18T16:06:44.710062+0000","last_active":"2024-03-18T16:08:47.962631+0000","last_peered":"2024-03-18T16:08:47.962631+0000","last_clean":"2024-03-18T16:08:47.962631+0000","last_became_active":"2024-03-18T16:06:44.709281+0000","last_became_peered":"2024-03-18T16:06:44.709281+0000","last_unstale":"2024-03-18T16:08:47.962631+0000","last_undegraded":"2024-03-18T16:08:47.962631+0000","last_fullsized":"2024-03-18T16:08:47.962631+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:51:30.100947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.960815+0000","last_change":"2024-03-18T16:06:44.734178+0000","last_active":"2024-03-18T16:08:47.960815+0000","last_peered":"2024-03-18T16:08:47.960815+0000","last_clean":"2024-03-18T16:08:47.960815+0000","last_became_active":"2024-03-18T16:06:44.732538+0000","last_became_peered":"2024-03-18T16:06:44.732538+0000","last_unstale":"2024-03-18T16:08:47.960815+0000","last_undegraded":"2024-03-18T16:08:47.960815+0000","last_fullsized":"2024-03-18T16:08:47.960815+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:14:26.904402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.960944+0000","last_change":"2024-03-18T16:07:47.412629+0000","last_active":"2024-03-18T16:08:47.960944+0000","last_peered":"2024-03-18T16:08:47.960944+0000","last_clean":"2024-03-18T16:08:47.960944+0000","last_became_active":"2024-03-18T16:07:47.411714+0000","last_became_peered":"2024-03-18T16:07:47.411714+0000","last_unstale":"2024-03-18T16:08:47.960944+0000","last_undegraded":"2024-03-18T16:08:47.960944+0000","last_fullsized":"2024-03-18T16:08:47.960944+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:25:50.843323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.959907+0000","last_change":"2024-03-18T16:06:44.728165+0000","last_active":"2024-03-18T16:08:47.959907+0000","last_peered":"2024-03-18T16:08:47.959907+0000","last_clean":"2024-03-18T16:08:47.959907+0000","last_became_active":"2024-03-18T16:06:44.727842+0000","last_became_peered":"2024-03-18T16:06:44.727842+0000","last_unstale":"2024-03-18T16:08:47.959907+0000","last_undegraded":"2024-03-18T16:08:47.959907+0000","last_fullsized":"2024-03-18T16:08:47.959907+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:55:04.464916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.881962+0000","last_change":"2024-03-18T16:06:44.730893+0000","last_active":"2024-03-18T16:08:47.881962+0000","last_peered":"2024-03-18T16:08:47.881962+0000","last_clean":"2024-03-18T16:08:47.881962+0000","last_became_active":"2024-03-18T16:06:44.730544+0000","last_became_peered":"2024-03-18T16:06:44.730544+0000","last_unstale":"2024-03-18T16:08:47.881962+0000","last_undegraded":"2024-03-18T16:08:47.881962+0000","last_fullsized":"2024-03-18T16:08:47.881962+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:16:37.918096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.884320+0000","last_change":"2024-03-18T16:06:44.706036+0000","last_active":"2024-03-18T16:08:47.884320+0000","last_peered":"2024-03-18T16:08:47.884320+0000","last_clean":"2024-03-18T16:08:47.884320+0000","last_became_active":"2024-03-18T16:06:44.705281+0000","last_became_peered":"2024-03-18T16:06:44.705281+0000","last_unstale":"2024-03-18T16:08:47.884320+0000","last_undegraded":"2024-03-18T16:08:47.884320+0000","last_fullsized":"2024-03-18T16:08:47.884320+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:52:20.202110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.884506+0000","last_change":"2024-03-18T16:06:44.727543+0000","last_active":"2024-03-18T16:08:47.884506+0000","last_peered":"2024-03-18T16:08:47.884506+0000","last_clean":"2024-03-18T16:08:47.884506+0000","last_became_active":"2024-03-18T16:06:44.727135+0000","last_became_peered":"2024-03-18T16:06:44.727135+0000","last_unstale":"2024-03-18T16:08:47.884506+0000","last_undegraded":"2024-03-18T16:08:47.884506+0000","last_fullsized":"2024-03-18T16:08:47.884506+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:52:42.742279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883556+0000","last_change":"2024-03-18T16:06:49.028004+0000","last_active":"2024-03-18T16:08:47.883556+0000","last_peered":"2024-03-18T16:08:47.883556+0000","last_clean":"2024-03-18T16:08:47.883556+0000","last_became_active":"2024-03-18T16:06:49.027640+0000","last_became_peered":"2024-03-18T16:06:49.027640+0000","last_unstale":"2024-03-18T16:08:47.883556+0000","last_undegraded":"2024-03-18T16:08:47.883556+0000","last_fullsized":"2024-03-18T16:08:47.883556+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:23:07.926349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'13","reported_seq":202,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:09:55.306333+0000","last_change":"2024-03-18T16:06:43.709540+0000","last_active":"2024-03-18T16:09:55.306333+0000","last_peered":"2024-03-18T16:09:55.306333+0000","last_clean":"2024-03-18T16:09:55.306333+0000","last_became_active":"2024-03-18T16:06:43.709170+0000","last_became_peered":"2024-03-18T16:06:43.709170+0000","last_unstale":"2024-03-18T16:09:55.306333+0000","last_undegraded":"2024-03-18T16:09:55.306333+0000","last_fullsized":"2024-03-18T16:09:55.306333+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:12:53.934079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":157,"num_read_kb":157,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.886428+0000","last_change":"2024-03-18T16:06:44.730192+0000","last_active":"2024-03-18T16:08:47.886428+0000","last_peered":"2024-03-18T16:08:47.886428+0000","last_clean":"2024-03-18T16:08:47.886428+0000","last_became_active":"2024-03-18T16:06:44.728071+0000","last_became_peered":"2024-03-18T16:06:44.728071+0000","last_unstale":"2024-03-18T16:08:47.886428+0000","last_undegraded":"2024-03-18T16:08:47.886428+0000","last_fullsized":"2024-03-18T16:08:47.886428+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:14:59.992958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883329+0000","last_change":"2024-03-18T16:06:49.023963+0000","last_active":"2024-03-18T16:08:47.883329+0000","last_peered":"2024-03-18T16:08:47.883329+0000","last_clean":"2024-03-18T16:08:47.883329+0000","last_became_active":"2024-03-18T16:06:49.023568+0000","last_became_peered":"2024-03-18T16:06:49.023568+0000","last_unstale":"2024-03-18T16:08:47.883329+0000","last_undegraded":"2024-03-18T16:08:47.883329+0000","last_fullsized":"2024-03-18T16:08:47.883329+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:15:19.730842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.886440+0000","last_change":"2024-03-18T16:06:43.722167+0000","last_active":"2024-03-18T16:08:47.886440+0000","last_peered":"2024-03-18T16:08:47.886440+0000","last_clean":"2024-03-18T16:08:47.886440+0000","last_became_active":"2024-03-18T16:06:43.721782+0000","last_became_peered":"2024-03-18T16:06:43.721782+0000","last_unstale":"2024-03-18T16:08:47.886440+0000","last_undegraded":"2024-03-18T16:08:47.886440+0000","last_fullsized":"2024-03-18T16:08:47.886440+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:08:54.400637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.962047+0000","last_change":"2024-03-18T16:06:44.729666+0000","last_active":"2024-03-18T16:08:47.962047+0000","last_peered":"2024-03-18T16:08:47.962047+0000","last_clean":"2024-03-18T16:08:47.962047+0000","last_became_active":"2024-03-18T16:06:44.728842+0000","last_became_peered":"2024-03-18T16:06:44.728842+0000","last_unstale":"2024-03-18T16:08:47.962047+0000","last_undegraded":"2024-03-18T16:08:47.962047+0000","last_fullsized":"2024-03-18T16:08:47.962047+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:01:42.438000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883464+0000","last_change":"2024-03-18T16:06:49.031169+0000","last_active":"2024-03-18T16:08:47.883464+0000","last_peered":"2024-03-18T16:08:47.883464+0000","last_clean":"2024-03-18T16:08:47.883464+0000","last_became_active":"2024-03-18T16:06:49.030801+0000","last_became_peered":"2024-03-18T16:06:49.030801+0000","last_unstale":"2024-03-18T16:08:47.883464+0000","last_undegraded":"2024-03-18T16:08:47.883464+0000","last_fullsized":"2024-03-18T16:08:47.883464+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:10:54.880882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'12","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.885418+0000","last_change":"2024-03-18T16:06:43.707111+0000","last_active":"2024-03-18T16:08:47.885418+0000","last_peered":"2024-03-18T16:08:47.885418+0000","last_clean":"2024-03-18T16:08:47.885418+0000","last_became_active":"2024-03-18T16:06:43.706671+0000","last_became_peered":"2024-03-18T16:06:43.706671+0000","last_unstale":"2024-03-18T16:08:47.885418+0000","last_undegraded":"2024-03-18T16:08:47.885418+0000","last_fullsized":"2024-03-18T16:08:47.885418+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:18:34.295641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.960382+0000","last_change":"2024-03-18T16:06:44.722100+0000","last_active":"2024-03-18T16:08:47.960382+0000","last_peered":"2024-03-18T16:08:47.960382+0000","last_clean":"2024-03-18T16:08:47.960382+0000","last_became_active":"2024-03-18T16:06:44.721673+0000","last_became_peered":"2024-03-18T16:06:44.721673+0000","last_unstale":"2024-03-18T16:08:47.960382+0000","last_undegraded":"2024-03-18T16:08:47.960382+0000","last_fullsized":"2024-03-18T16:08:47.960382+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:41:22.624778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883382+0000","last_change":"2024-03-18T16:06:49.027966+0000","last_active":"2024-03-18T16:08:47.883382+0000","last_peered":"2024-03-18T16:08:47.883382+0000","last_clean":"2024-03-18T16:08:47.883382+0000","last_became_active":"2024-03-18T16:06:49.027567+0000","last_became_peered":"2024-03-18T16:06:49.027567+0000","last_unstale":"2024-03-18T16:08:47.883382+0000","last_undegraded":"2024-03-18T16:08:47.883382+0000","last_fullsized":"2024-03-18T16:08:47.883382+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:54:28.543019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"89'5","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.885092+0000","last_change":"2024-03-18T16:06:43.704023+0000","last_active":"2024-03-18T16:08:47.885092+0000","last_peered":"2024-03-18T16:08:47.885092+0000","last_clean":"2024-03-18T16:08:47.885092+0000","last_became_active":"2024-03-18T16:06:43.703690+0000","last_became_peered":"2024-03-18T16:06:43.703690+0000","last_unstale":"2024-03-18T16:08:47.885092+0000","last_undegraded":"2024-03-18T16:08:47.885092+0000","last_fullsized":"2024-03-18T16:08:47.885092+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:20:02.344346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":50,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.886144+0000","last_change":"2024-03-18T16:07:47.354361+0000","last_active":"2024-03-18T16:08:47.886144+0000","last_peered":"2024-03-18T16:08:47.886144+0000","last_clean":"2024-03-18T16:08:47.886144+0000","last_became_active":"2024-03-18T16:07:47.353964+0000","last_became_peered":"2024-03-18T16:07:47.353964+0000","last_unstale":"2024-03-18T16:08:47.886144+0000","last_undegraded":"2024-03-18T16:08:47.886144+0000","last_fullsized":"2024-03-18T16:08:47.886144+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:56:11.405218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.884343+0000","last_change":"2024-03-18T16:06:49.012310+0000","last_active":"2024-03-18T16:08:47.884343+0000","last_peered":"2024-03-18T16:08:47.884343+0000","last_clean":"2024-03-18T16:08:47.884343+0000","last_became_active":"2024-03-18T16:06:49.011844+0000","last_became_peered":"2024-03-18T16:06:49.011844+0000","last_unstale":"2024-03-18T16:08:47.884343+0000","last_undegraded":"2024-03-18T16:08:47.884343+0000","last_fullsized":"2024-03-18T16:08:47.884343+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:16:42.652205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"91'7","reported_seq":199,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:09:55.144566+0000","last_change":"2024-03-18T16:06:43.705653+0000","last_active":"2024-03-18T16:09:55.144566+0000","last_peered":"2024-03-18T16:09:55.144566+0000","last_clean":"2024-03-18T16:09:55.144566+0000","last_became_active":"2024-03-18T16:06:43.705320+0000","last_became_peered":"2024-03-18T16:06:43.705320+0000","last_unstale":"2024-03-18T16:09:55.144566+0000","last_undegraded":"2024-03-18T16:09:55.144566+0000","last_fullsized":"2024-03-18T16:09:55.144566+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:10:10.827338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":161,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.884849+0000","last_change":"2024-03-18T16:06:44.720045+0000","last_active":"2024-03-18T16:08:47.884849+0000","last_peered":"2024-03-18T16:08:47.884849+0000","last_clean":"2024-03-18T16:08:47.884849+0000","last_became_active":"2024-03-18T16:06:44.719669+0000","last_became_peered":"2024-03-18T16:06:44.719669+0000","last_unstale":"2024-03-18T16:08:47.884849+0000","last_undegraded":"2024-03-18T16:08:47.884849+0000","last_fullsized":"2024-03-18T16:08:47.884849+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:23:47.464057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.884107+0000","last_change":"2024-03-18T16:06:49.032576+0000","last_active":"2024-03-18T16:08:47.884107+0000","last_peered":"2024-03-18T16:08:47.884107+0000","last_clean":"2024-03-18T16:08:47.884107+0000","last_became_active":"2024-03-18T16:06:49.032126+0000","last_became_peered":"2024-03-18T16:06:49.032126+0000","last_unstale":"2024-03-18T16:08:47.884107+0000","last_undegraded":"2024-03-18T16:08:47.884107+0000","last_fullsized":"2024-03-18T16:08:47.884107+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:41:54.822070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'7","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.879820+0000","last_change":"2024-03-18T16:06:43.706364+0000","last_active":"2024-03-18T16:08:47.879820+0000","last_peered":"2024-03-18T16:08:47.879820+0000","last_clean":"2024-03-18T16:08:47.879820+0000","last_became_active":"2024-03-18T16:06:43.705984+0000","last_became_peered":"2024-03-18T16:06:43.705984+0000","last_unstale":"2024-03-18T16:08:47.879820+0000","last_undegraded":"2024-03-18T16:08:47.879820+0000","last_fullsized":"2024-03-18T16:08:47.879820+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:46:06.503000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.886025+0000","last_change":"2024-03-18T16:06:44.721019+0000","last_active":"2024-03-18T16:08:47.886025+0000","last_peered":"2024-03-18T16:08:47.886025+0000","last_clean":"2024-03-18T16:08:47.886025+0000","last_became_active":"2024-03-18T16:06:44.720665+0000","last_became_peered":"2024-03-18T16:06:44.720665+0000","last_unstale":"2024-03-18T16:08:47.886025+0000","last_undegraded":"2024-03-18T16:08:47.886025+0000","last_fullsized":"2024-03-18T16:08:47.886025+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:58:34.180653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.881862+0000","last_change":"2024-03-18T16:06:49.022919+0000","last_active":"2024-03-18T16:08:47.881862+0000","last_peered":"2024-03-18T16:08:47.881862+0000","last_clean":"2024-03-18T16:08:47.881862+0000","last_became_active":"2024-03-18T16:06:49.022514+0000","last_became_peered":"2024-03-18T16:06:49.022514+0000","last_unstale":"2024-03-18T16:08:47.881862+0000","last_undegraded":"2024-03-18T16:08:47.881862+0000","last_fullsized":"2024-03-18T16:08:47.881862+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:46:39.510181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'3","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.961422+0000","last_change":"2024-03-18T16:06:43.713025+0000","last_active":"2024-03-18T16:08:47.961422+0000","last_peered":"2024-03-18T16:08:47.961422+0000","last_clean":"2024-03-18T16:08:47.961422+0000","last_became_active":"2024-03-18T16:06:43.711469+0000","last_became_peered":"2024-03-18T16:06:43.711469+0000","last_unstale":"2024-03-18T16:08:47.961422+0000","last_undegraded":"2024-03-18T16:08:47.961422+0000","last_fullsized":"2024-03-18T16:08:47.961422+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:30:41.193229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883163+0000","last_change":"2024-03-18T16:06:44.723953+0000","last_active":"2024-03-18T16:08:47.883163+0000","last_peered":"2024-03-18T16:08:47.883163+0000","last_clean":"2024-03-18T16:08:47.883163+0000","last_became_active":"2024-03-18T16:06:44.723266+0000","last_became_peered":"2024-03-18T16:06:44.723266+0000","last_unstale":"2024-03-18T16:08:47.883163+0000","last_undegraded":"2024-03-18T16:08:47.883163+0000","last_fullsized":"2024-03-18T16:08:47.883163+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:41:10.067297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.961683+0000","last_change":"2024-03-18T16:06:49.039472+0000","last_active":"2024-03-18T16:08:47.961683+0000","last_peered":"2024-03-18T16:08:47.961683+0000","last_clean":"2024-03-18T16:08:47.961683+0000","last_became_active":"2024-03-18T16:06:49.038889+0000","last_became_peered":"2024-03-18T16:06:49.038889+0000","last_unstale":"2024-03-18T16:08:47.961683+0000","last_undegraded":"2024-03-18T16:08:47.961683+0000","last_fullsized":"2024-03-18T16:08:47.961683+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:03:05.345315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'3","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.885773+0000","last_change":"2024-03-18T16:06:43.705873+0000","last_active":"2024-03-18T16:08:47.885773+0000","last_peered":"2024-03-18T16:08:47.885773+0000","last_clean":"2024-03-18T16:08:47.885773+0000","last_became_active":"2024-03-18T16:06:43.705548+0000","last_became_peered":"2024-03-18T16:06:43.705548+0000","last_unstale":"2024-03-18T16:08:47.885773+0000","last_undegraded":"2024-03-18T16:08:47.885773+0000","last_fullsized":"2024-03-18T16:08:47.885773+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:52:03.790888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.882690+0000","last_change":"2024-03-18T16:06:44.718526+0000","last_active":"2024-03-18T16:08:47.882690+0000","last_peered":"2024-03-18T16:08:47.882690+0000","last_clean":"2024-03-18T16:08:47.882690+0000","last_became_active":"2024-03-18T16:06:44.717266+0000","last_became_peered":"2024-03-18T16:06:44.717266+0000","last_unstale":"2024-03-18T16:08:47.882690+0000","last_undegraded":"2024-03-18T16:08:47.882690+0000","last_fullsized":"2024-03-18T16:08:47.882690+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:55:09.340354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.885083+0000","last_change":"2024-03-18T16:06:49.021971+0000","last_active":"2024-03-18T16:08:47.885083+0000","last_peered":"2024-03-18T16:08:47.885083+0000","last_clean":"2024-03-18T16:08:47.885083+0000","last_became_active":"2024-03-18T16:06:49.021577+0000","last_became_peered":"2024-03-18T16:06:49.021577+0000","last_unstale":"2024-03-18T16:08:47.885083+0000","last_undegraded":"2024-03-18T16:08:47.885083+0000","last_fullsized":"2024-03-18T16:08:47.885083+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:47:44.768559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"93'6","reported_seq":357,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:09:59.148338+0000","last_change":"2024-03-18T16:06:43.705636+0000","last_active":"2024-03-18T16:09:59.148338+0000","last_peered":"2024-03-18T16:09:59.148338+0000","last_clean":"2024-03-18T16:09:59.148338+0000","last_became_active":"2024-03-18T16:06:43.705268+0000","last_became_peered":"2024-03-18T16:06:43.705268+0000","last_unstale":"2024-03-18T16:09:59.148338+0000","last_undegraded":"2024-03-18T16:09:59.148338+0000","last_fullsized":"2024-03-18T16:09:59.148338+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:37:31.367130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":320,"num_read_kb":320,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.960576+0000","last_change":"2024-03-18T16:06:44.712237+0000","last_active":"2024-03-18T16:08:47.960576+0000","last_peered":"2024-03-18T16:08:47.960576+0000","last_clean":"2024-03-18T16:08:47.960576+0000","last_became_active":"2024-03-18T16:06:44.711677+0000","last_became_peered":"2024-03-18T16:06:44.711677+0000","last_unstale":"2024-03-18T16:08:47.960576+0000","last_undegraded":"2024-03-18T16:08:47.960576+0000","last_fullsized":"2024-03-18T16:08:47.960576+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:16:56.326962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.882199+0000","last_change":"2024-03-18T16:06:49.034578+0000","last_active":"2024-03-18T16:08:47.882199+0000","last_peered":"2024-03-18T16:08:47.882199+0000","last_clean":"2024-03-18T16:08:47.882199+0000","last_became_active":"2024-03-18T16:06:49.031099+0000","last_became_peered":"2024-03-18T16:06:49.031099+0000","last_unstale":"2024-03-18T16:08:47.882199+0000","last_undegraded":"2024-03-18T16:08:47.882199+0000","last_fullsized":"2024-03-18T16:08:47.882199+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:11:08.042550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'4","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.961212+0000","last_change":"2024-03-18T16:06:43.708151+0000","last_active":"2024-03-18T16:08:47.961212+0000","last_peered":"2024-03-18T16:08:47.961212+0000","last_clean":"2024-03-18T16:08:47.961212+0000","last_became_active":"2024-03-18T16:06:43.707810+0000","last_became_peered":"2024-03-18T16:06:43.707810+0000","last_unstale":"2024-03-18T16:08:47.961212+0000","last_undegraded":"2024-03-18T16:08:47.961212+0000","last_fullsized":"2024-03-18T16:08:47.961212+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:30:35.745772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.962204+0000","last_change":"2024-03-18T16:06:44.729777+0000","last_active":"2024-03-18T16:08:47.962204+0000","last_peered":"2024-03-18T16:08:47.962204+0000","last_clean":"2024-03-18T16:08:47.962204+0000","last_became_active":"2024-03-18T16:06:44.729244+0000","last_became_peered":"2024-03-18T16:06:44.729244+0000","last_unstale":"2024-03-18T16:08:47.962204+0000","last_undegraded":"2024-03-18T16:08:47.962204+0000","last_fullsized":"2024-03-18T16:08:47.962204+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:52:05.420443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883653+0000","last_change":"2024-03-18T16:07:47.355131+0000","last_active":"2024-03-18T16:08:47.883653+0000","last_peered":"2024-03-18T16:08:47.883653+0000","last_clean":"2024-03-18T16:08:47.883653+0000","last_became_active":"2024-03-18T16:07:47.354608+0000","last_became_peered":"2024-03-18T16:07:47.354608+0000","last_unstale":"2024-03-18T16:08:47.883653+0000","last_undegraded":"2024-03-18T16:08:47.883653+0000","last_fullsized":"2024-03-18T16:08:47.883653+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:12:57.979168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'5","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.886234+0000","last_change":"2024-03-18T16:06:43.715280+0000","last_active":"2024-03-18T16:08:47.886234+0000","last_peered":"2024-03-18T16:08:47.886234+0000","last_clean":"2024-03-18T16:08:47.886234+0000","last_became_active":"2024-03-18T16:06:43.714936+0000","last_became_peered":"2024-03-18T16:06:43.714936+0000","last_unstale":"2024-03-18T16:08:47.886234+0000","last_undegraded":"2024-03-18T16:08:47.886234+0000","last_fullsized":"2024-03-18T16:08:47.886234+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:07:38.746932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.960603+0000","last_change":"2024-03-18T16:06:44.709004+0000","last_active":"2024-03-18T16:08:47.960603+0000","last_peered":"2024-03-18T16:08:47.960603+0000","last_clean":"2024-03-18T16:08:47.960603+0000","last_became_active":"2024-03-18T16:06:44.708574+0000","last_became_peered":"2024-03-18T16:06:44.708574+0000","last_unstale":"2024-03-18T16:08:47.960603+0000","last_undegraded":"2024-03-18T16:08:47.960603+0000","last_fullsized":"2024-03-18T16:08:47.960603+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:02:50.982110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883055+0000","last_change":"2024-03-18T16:06:49.029806+0000","last_active":"2024-03-18T16:08:47.883055+0000","last_peered":"2024-03-18T16:08:47.883055+0000","last_clean":"2024-03-18T16:08:47.883055+0000","last_became_active":"2024-03-18T16:06:49.029404+0000","last_became_peered":"2024-03-18T16:06:49.029404+0000","last_unstale":"2024-03-18T16:08:47.883055+0000","last_undegraded":"2024-03-18T16:08:47.883055+0000","last_fullsized":"2024-03-18T16:08:47.883055+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:38:37.706620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"93'40","reported_seq":402,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:09:59.149498+0000","last_change":"2024-03-18T16:06:43.702910+0000","last_active":"2024-03-18T16:09:59.149498+0000","last_peered":"2024-03-18T16:09:59.149498+0000","last_clean":"2024-03-18T16:09:59.149498+0000","last_became_active":"2024-03-18T16:06:43.702557+0000","last_became_peered":"2024-03-18T16:06:43.702557+0000","last_unstale":"2024-03-18T16:09:59.149498+0000","last_undegraded":"2024-03-18T16:09:59.149498+0000","last_fullsized":"2024-03-18T16:09:59.149498+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:55:15.362987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":66365,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":331,"num_read_kb":231,"num_write":42,"num_write_kb":85,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.884961+0000","last_change":"2024-03-18T16:06:44.709075+0000","last_active":"2024-03-18T16:08:47.884961+0000","last_peered":"2024-03-18T16:08:47.884961+0000","last_clean":"2024-03-18T16:08:47.884961+0000","last_became_active":"2024-03-18T16:06:44.708366+0000","last_became_peered":"2024-03-18T16:06:44.708366+0000","last_unstale":"2024-03-18T16:08:47.884961+0000","last_undegraded":"2024-03-18T16:08:47.884961+0000","last_fullsized":"2024-03-18T16:08:47.884961+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:44:18.486385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.886444+0000","last_change":"2024-03-18T16:06:49.037380+0000","last_active":"2024-03-18T16:08:47.886444+0000","last_peered":"2024-03-18T16:08:47.886444+0000","last_clean":"2024-03-18T16:08:47.886444+0000","last_became_active":"2024-03-18T16:06:49.037021+0000","last_became_peered":"2024-03-18T16:06:49.037021+0000","last_unstale":"2024-03-18T16:08:47.886444+0000","last_undegraded":"2024-03-18T16:08:47.886444+0000","last_fullsized":"2024-03-18T16:08:47.886444+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:43:03.482631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'9","reported_seq":25,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.896395+0000","last_change":"2024-03-18T16:08:47.895964+0000","last_active":"2024-03-18T16:08:47.896395+0000","last_peered":"2024-03-18T16:08:47.896395+0000","last_clean":"2024-03-18T16:08:47.896395+0000","last_became_active":"2024-03-18T16:08:47.886729+0000","last_became_peered":"2024-03-18T16:08:47.886729+0000","last_unstale":"2024-03-18T16:08:47.896395+0000","last_undegraded":"2024-03-18T16:08:47.896395+0000","last_fullsized":"2024-03-18T16:08:47.896395+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:58:02.557326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":68,"num_keys_recovered":10,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.962072+0000","last_change":"2024-03-18T16:06:44.727204+0000","last_active":"2024-03-18T16:08:47.962072+0000","last_peered":"2024-03-18T16:08:47.962072+0000","last_clean":"2024-03-18T16:08:47.962072+0000","last_became_active":"2024-03-18T16:06:44.726775+0000","last_became_peered":"2024-03-18T16:06:44.726775+0000","last_unstale":"2024-03-18T16:08:47.962072+0000","last_undegraded":"2024-03-18T16:08:47.962072+0000","last_fullsized":"2024-03-18T16:08:47.962072+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:29:44.842972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.881484+0000","last_change":"2024-03-18T16:06:49.022508+0000","last_active":"2024-03-18T16:08:47.881484+0000","last_peered":"2024-03-18T16:08:47.881484+0000","last_clean":"2024-03-18T16:08:47.881484+0000","last_became_active":"2024-03-18T16:06:49.022017+0000","last_became_peered":"2024-03-18T16:06:49.022017+0000","last_unstale":"2024-03-18T16:08:47.881484+0000","last_undegraded":"2024-03-18T16:08:47.881484+0000","last_fullsized":"2024-03-18T16:08:47.881484+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:52:45.892880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'5","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.885883+0000","last_change":"2024-03-18T16:06:43.719618+0000","last_active":"2024-03-18T16:08:47.885883+0000","last_peered":"2024-03-18T16:08:47.885883+0000","last_clean":"2024-03-18T16:08:47.885883+0000","last_became_active":"2024-03-18T16:06:43.719161+0000","last_became_peered":"2024-03-18T16:06:43.719161+0000","last_unstale":"2024-03-18T16:08:47.885883+0000","last_undegraded":"2024-03-18T16:08:47.885883+0000","last_fullsized":"2024-03-18T16:08:47.885883+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:39:53.077878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.960915+0000","last_change":"2024-03-18T16:06:44.717829+0000","last_active":"2024-03-18T16:08:47.960915+0000","last_peered":"2024-03-18T16:08:47.960915+0000","last_clean":"2024-03-18T16:08:47.960915+0000","last_became_active":"2024-03-18T16:06:44.717473+0000","last_became_peered":"2024-03-18T16:06:44.717473+0000","last_unstale":"2024-03-18T16:08:47.960915+0000","last_undegraded":"2024-03-18T16:08:47.960915+0000","last_fullsized":"2024-03-18T16:08:47.960915+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:10:00.183613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.884360+0000","last_change":"2024-03-18T16:06:49.036766+0000","last_active":"2024-03-18T16:08:47.884360+0000","last_peered":"2024-03-18T16:08:47.884360+0000","last_clean":"2024-03-18T16:08:47.884360+0000","last_became_active":"2024-03-18T16:06:49.034617+0000","last_became_peered":"2024-03-18T16:06:49.034617+0000","last_unstale":"2024-03-18T16:08:47.884360+0000","last_undegraded":"2024-03-18T16:08:47.884360+0000","last_fullsized":"2024-03-18T16:08:47.884360+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:18:59.033781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"89'5","reported_seq":196,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:09:56.221145+0000","last_change":"2024-03-18T16:06:43.702452+0000","last_active":"2024-03-18T16:09:56.221145+0000","last_peered":"2024-03-18T16:09:56.221145+0000","last_clean":"2024-03-18T16:09:56.221145+0000","last_became_active":"2024-03-18T16:06:43.702141+0000","last_became_peered":"2024-03-18T16:06:43.702141+0000","last_unstale":"2024-03-18T16:09:56.221145+0000","last_undegraded":"2024-03-18T16:09:56.221145+0000","last_fullsized":"2024-03-18T16:09:56.221145+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:44:05.366101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":161,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.960175+0000","last_change":"2024-03-18T16:06:44.711403+0000","last_active":"2024-03-18T16:08:47.960175+0000","last_peered":"2024-03-18T16:08:47.960175+0000","last_clean":"2024-03-18T16:08:47.960175+0000","last_became_active":"2024-03-18T16:06:44.710654+0000","last_became_peered":"2024-03-18T16:06:44.710654+0000","last_unstale":"2024-03-18T16:08:47.960175+0000","last_undegraded":"2024-03-18T16:08:47.960175+0000","last_fullsized":"2024-03-18T16:08:47.960175+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:21:43.633274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.962799+0000","last_change":"2024-03-18T16:06:49.039192+0000","last_active":"2024-03-18T16:08:47.962799+0000","last_peered":"2024-03-18T16:08:47.962799+0000","last_clean":"2024-03-18T16:08:47.962799+0000","last_became_active":"2024-03-18T16:06:49.038421+0000","last_became_peered":"2024-03-18T16:06:49.038421+0000","last_unstale":"2024-03-18T16:08:47.962799+0000","last_undegraded":"2024-03-18T16:08:47.962799+0000","last_fullsized":"2024-03-18T16:08:47.962799+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T00:54:56.944209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'17","reported_seq":49,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.884583+0000","last_change":"2024-03-18T16:06:43.712523+0000","last_active":"2024-03-18T16:08:47.884583+0000","last_peered":"2024-03-18T16:08:47.884583+0000","last_clean":"2024-03-18T16:08:47.884583+0000","last_became_active":"2024-03-18T16:06:43.712151+0000","last_became_peered":"2024-03-18T16:06:43.712151+0000","last_unstale":"2024-03-18T16:08:47.884583+0000","last_undegraded":"2024-03-18T16:08:47.884583+0000","last_fullsized":"2024-03-18T16:08:47.884583+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:17:19.651293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.884896+0000","last_change":"2024-03-18T16:07:47.355115+0000","last_active":"2024-03-18T16:08:47.884896+0000","last_peered":"2024-03-18T16:08:47.884896+0000","last_clean":"2024-03-18T16:08:47.884896+0000","last_became_active":"2024-03-18T16:07:47.354744+0000","last_became_peered":"2024-03-18T16:07:47.354744+0000","last_unstale":"2024-03-18T16:08:47.884896+0000","last_undegraded":"2024-03-18T16:08:47.884896+0000","last_fullsized":"2024-03-18T16:08:47.884896+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:14:27.644891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.884782+0000","last_change":"2024-03-18T16:06:49.028943+0000","last_active":"2024-03-18T16:08:47.884782+0000","last_peered":"2024-03-18T16:08:47.884782+0000","last_clean":"2024-03-18T16:08:47.884782+0000","last_became_active":"2024-03-18T16:06:49.028453+0000","last_became_peered":"2024-03-18T16:06:49.028453+0000","last_unstale":"2024-03-18T16:08:47.884782+0000","last_undegraded":"2024-03-18T16:08:47.884782+0000","last_fullsized":"2024-03-18T16:08:47.884782+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:16:30.318649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":17,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.899433+0000","last_change":"2024-03-18T16:08:47.899048+0000","last_active":"2024-03-18T16:08:47.899433+0000","last_peered":"2024-03-18T16:08:47.899433+0000","last_clean":"2024-03-18T16:08:47.899433+0000","last_became_active":"2024-03-18T16:08:47.887283+0000","last_became_peered":"2024-03-18T16:08:47.887283+0000","last_unstale":"2024-03-18T16:08:47.899433+0000","last_undegraded":"2024-03-18T16:08:47.899433+0000","last_fullsized":"2024-03-18T16:08:47.899433+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:17:17.924210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.881500+0000","last_change":"2024-03-18T16:06:44.716284+0000","last_active":"2024-03-18T16:08:47.881500+0000","last_peered":"2024-03-18T16:08:47.881500+0000","last_clean":"2024-03-18T16:08:47.881500+0000","last_became_active":"2024-03-18T16:06:44.715911+0000","last_became_peered":"2024-03-18T16:06:44.715911+0000","last_unstale":"2024-03-18T16:08:47.881500+0000","last_undegraded":"2024-03-18T16:08:47.881500+0000","last_fullsized":"2024-03-18T16:08:47.881500+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:48:34.839514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.885009+0000","last_change":"2024-03-18T16:06:49.059061+0000","last_active":"2024-03-18T16:08:47.885009+0000","last_peered":"2024-03-18T16:08:47.885009+0000","last_clean":"2024-03-18T16:08:47.885009+0000","last_became_active":"2024-03-18T16:06:49.058619+0000","last_became_peered":"2024-03-18T16:06:49.058619+0000","last_unstale":"2024-03-18T16:08:47.885009+0000","last_undegraded":"2024-03-18T16:08:47.885009+0000","last_fullsized":"2024-03-18T16:08:47.885009+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:44:52.970785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"89'1","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883689+0000","last_change":"2024-03-18T16:06:43.715766+0000","last_active":"2024-03-18T16:08:47.883689+0000","last_peered":"2024-03-18T16:08:47.883689+0000","last_clean":"2024-03-18T16:08:47.883689+0000","last_became_active":"2024-03-18T16:06:43.715380+0000","last_became_peered":"2024-03-18T16:06:43.715380+0000","last_unstale":"2024-03-18T16:08:47.883689+0000","last_undegraded":"2024-03-18T16:08:47.883689+0000","last_fullsized":"2024-03-18T16:08:47.883689+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:13:44.195934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.880175+0000","last_change":"2024-03-18T16:06:44.728150+0000","last_active":"2024-03-18T16:08:47.880175+0000","last_peered":"2024-03-18T16:08:47.880175+0000","last_clean":"2024-03-18T16:08:47.880175+0000","last_became_active":"2024-03-18T16:06:44.727769+0000","last_became_peered":"2024-03-18T16:06:44.727769+0000","last_unstale":"2024-03-18T16:08:47.880175+0000","last_undegraded":"2024-03-18T16:08:47.880175+0000","last_fullsized":"2024-03-18T16:08:47.880175+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:28:16.023225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.961752+0000","last_change":"2024-03-18T16:06:49.033671+0000","last_active":"2024-03-18T16:08:47.961752+0000","last_peered":"2024-03-18T16:08:47.961752+0000","last_clean":"2024-03-18T16:08:47.961752+0000","last_became_active":"2024-03-18T16:06:49.033046+0000","last_became_peered":"2024-03-18T16:06:49.033046+0000","last_unstale":"2024-03-18T16:08:47.961752+0000","last_undegraded":"2024-03-18T16:08:47.961752+0000","last_fullsized":"2024-03-18T16:08:47.961752+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:26:37.020521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'5","reported_seq":195,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:09:56.309256+0000","last_change":"2024-03-18T16:06:43.712636+0000","last_active":"2024-03-18T16:09:56.309256+0000","last_peered":"2024-03-18T16:09:56.309256+0000","last_clean":"2024-03-18T16:09:56.309256+0000","last_became_active":"2024-03-18T16:06:43.712300+0000","last_became_peered":"2024-03-18T16:06:43.712300+0000","last_unstale":"2024-03-18T16:09:56.309256+0000","last_undegraded":"2024-03-18T16:09:56.309256+0000","last_fullsized":"2024-03-18T16:09:56.309256+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:48:20.386640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":158,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.880029+0000","last_change":"2024-03-18T16:06:44.731583+0000","last_active":"2024-03-18T16:08:47.880029+0000","last_peered":"2024-03-18T16:08:47.880029+0000","last_clean":"2024-03-18T16:08:47.880029+0000","last_became_active":"2024-03-18T16:06:44.731183+0000","last_became_peered":"2024-03-18T16:06:44.731183+0000","last_unstale":"2024-03-18T16:08:47.880029+0000","last_undegraded":"2024-03-18T16:08:47.880029+0000","last_fullsized":"2024-03-18T16:08:47.880029+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:23:05.098101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.962348+0000","last_change":"2024-03-18T16:06:49.060130+0000","last_active":"2024-03-18T16:08:47.962348+0000","last_peered":"2024-03-18T16:08:47.962348+0000","last_clean":"2024-03-18T16:08:47.962348+0000","last_became_active":"2024-03-18T16:06:49.059109+0000","last_became_peered":"2024-03-18T16:06:49.059109+0000","last_unstale":"2024-03-18T16:08:47.962348+0000","last_undegraded":"2024-03-18T16:08:47.962348+0000","last_fullsized":"2024-03-18T16:08:47.962348+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:41:40.008047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.885977+0000","last_change":"2024-03-18T16:06:44.705793+0000","last_active":"2024-03-18T16:08:47.885977+0000","last_peered":"2024-03-18T16:08:47.885977+0000","last_clean":"2024-03-18T16:08:47.885977+0000","last_became_active":"2024-03-18T16:06:44.705426+0000","last_became_peered":"2024-03-18T16:06:44.705426+0000","last_unstale":"2024-03-18T16:08:47.885977+0000","last_undegraded":"2024-03-18T16:08:47.885977+0000","last_fullsized":"2024-03-18T16:08:47.885977+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:14:31.198829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'5","reported_seq":199,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:10:00.226377+0000","last_change":"2024-03-18T16:06:43.721057+0000","last_active":"2024-03-18T16:10:00.226377+0000","last_peered":"2024-03-18T16:10:00.226377+0000","last_clean":"2024-03-18T16:10:00.226377+0000","last_became_active":"2024-03-18T16:06:43.720661+0000","last_became_peered":"2024-03-18T16:06:43.720661+0000","last_unstale":"2024-03-18T16:10:00.226377+0000","last_undegraded":"2024-03-18T16:10:00.226377+0000","last_fullsized":"2024-03-18T16:10:00.226377+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:08:18.050511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":163,"num_read_kb":163,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.961318+0000","last_change":"2024-03-18T16:07:47.361447+0000","last_active":"2024-03-18T16:08:47.961318+0000","last_peered":"2024-03-18T16:08:47.961318+0000","last_clean":"2024-03-18T16:08:47.961318+0000","last_became_active":"2024-03-18T16:07:47.353684+0000","last_became_peered":"2024-03-18T16:07:47.353684+0000","last_unstale":"2024-03-18T16:08:47.961318+0000","last_undegraded":"2024-03-18T16:08:47.961318+0000","last_fullsized":"2024-03-18T16:08:47.961318+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:59:53.025620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,10],"acting":[7,5,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.885471+0000","last_change":"2024-03-18T16:06:44.725555+0000","last_active":"2024-03-18T16:08:47.885471+0000","last_peered":"2024-03-18T16:08:47.885471+0000","last_clean":"2024-03-18T16:08:47.885471+0000","last_became_active":"2024-03-18T16:06:44.719880+0000","last_became_peered":"2024-03-18T16:06:44.719880+0000","last_unstale":"2024-03-18T16:08:47.885471+0000","last_undegraded":"2024-03-18T16:08:47.885471+0000","last_fullsized":"2024-03-18T16:08:47.885471+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:45:17.830876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'6","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883578+0000","last_change":"2024-03-18T16:06:43.700814+0000","last_active":"2024-03-18T16:08:47.883578+0000","last_peered":"2024-03-18T16:08:47.883578+0000","last_clean":"2024-03-18T16:08:47.883578+0000","last_became_active":"2024-03-18T16:06:43.698902+0000","last_became_peered":"2024-03-18T16:06:43.698902+0000","last_unstale":"2024-03-18T16:08:47.883578+0000","last_undegraded":"2024-03-18T16:08:47.883578+0000","last_fullsized":"2024-03-18T16:08:47.883578+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:20:20.100033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":354,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.960035+0000","last_change":"2024-03-18T16:05:22.379397+0000","last_active":"2024-03-18T16:08:47.960035+0000","last_peered":"2024-03-18T16:08:47.960035+0000","last_clean":"2024-03-18T16:08:47.960035+0000","last_became_active":"2024-03-18T16:05:22.378830+0000","last_became_peered":"2024-03-18T16:05:22.378830+0000","last_unstale":"2024-03-18T16:08:47.960035+0000","last_undegraded":"2024-03-18T16:08:47.960035+0000","last_fullsized":"2024-03-18T16:08:47.960035+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:01:30.048984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:01:30.048984+0000","last_clean_scrub_stamp":"2024-03-18T16:01:30.048984+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:35:45.796495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.885230+0000","last_change":"2024-03-18T16:06:49.058931+0000","last_active":"2024-03-18T16:08:47.885230+0000","last_peered":"2024-03-18T16:08:47.885230+0000","last_clean":"2024-03-18T16:08:47.885230+0000","last_became_active":"2024-03-18T16:06:49.058497+0000","last_became_peered":"2024-03-18T16:06:49.058497+0000","last_unstale":"2024-03-18T16:08:47.885230+0000","last_undegraded":"2024-03-18T16:08:47.885230+0000","last_fullsized":"2024-03-18T16:08:47.885230+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:41:51.874582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.961259+0000","last_change":"2024-03-18T16:06:44.710332+0000","last_active":"2024-03-18T16:08:47.961259+0000","last_peered":"2024-03-18T16:08:47.961259+0000","last_clean":"2024-03-18T16:08:47.961259+0000","last_became_active":"2024-03-18T16:06:44.709952+0000","last_became_peered":"2024-03-18T16:06:44.709952+0000","last_unstale":"2024-03-18T16:08:47.961259+0000","last_undegraded":"2024-03-18T16:08:47.961259+0000","last_fullsized":"2024-03-18T16:08:47.961259+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:24:26.653511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'4","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.960126+0000","last_change":"2024-03-18T16:06:43.708265+0000","last_active":"2024-03-18T16:08:47.960126+0000","last_peered":"2024-03-18T16:08:47.960126+0000","last_clean":"2024-03-18T16:08:47.960126+0000","last_became_active":"2024-03-18T16:06:43.707466+0000","last_became_peered":"2024-03-18T16:06:43.707466+0000","last_unstale":"2024-03-18T16:08:47.960126+0000","last_undegraded":"2024-03-18T16:08:47.960126+0000","last_fullsized":"2024-03-18T16:08:47.960126+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:45:40.065396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.882033+0000","last_change":"2024-03-18T16:06:49.022401+0000","last_active":"2024-03-18T16:08:47.882033+0000","last_peered":"2024-03-18T16:08:47.882033+0000","last_clean":"2024-03-18T16:08:47.882033+0000","last_became_active":"2024-03-18T16:06:49.021966+0000","last_became_peered":"2024-03-18T16:06:49.021966+0000","last_unstale":"2024-03-18T16:08:47.882033+0000","last_undegraded":"2024-03-18T16:08:47.882033+0000","last_fullsized":"2024-03-18T16:08:47.882033+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:04:12.434969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'4","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.885248+0000","last_change":"2024-03-18T16:06:43.713249+0000","last_active":"2024-03-18T16:08:47.885248+0000","last_peered":"2024-03-18T16:08:47.885248+0000","last_clean":"2024-03-18T16:08:47.885248+0000","last_became_active":"2024-03-18T16:06:43.712897+0000","last_became_peered":"2024-03-18T16:06:43.712897+0000","last_unstale":"2024-03-18T16:08:47.885248+0000","last_undegraded":"2024-03-18T16:08:47.885248+0000","last_fullsized":"2024-03-18T16:08:47.885248+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:20:04.642568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.961929+0000","last_change":"2024-03-18T16:06:44.710034+0000","last_active":"2024-03-18T16:08:47.961929+0000","last_peered":"2024-03-18T16:08:47.961929+0000","last_clean":"2024-03-18T16:08:47.961929+0000","last_became_active":"2024-03-18T16:06:44.708457+0000","last_became_peered":"2024-03-18T16:06:44.708457+0000","last_unstale":"2024-03-18T16:08:47.961929+0000","last_undegraded":"2024-03-18T16:08:47.961929+0000","last_fullsized":"2024-03-18T16:08:47.961929+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:40:00.504382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883057+0000","last_change":"2024-03-18T16:06:49.059345+0000","last_active":"2024-03-18T16:08:47.883057+0000","last_peered":"2024-03-18T16:08:47.883057+0000","last_clean":"2024-03-18T16:08:47.883057+0000","last_became_active":"2024-03-18T16:06:49.058675+0000","last_became_peered":"2024-03-18T16:06:49.058675+0000","last_unstale":"2024-03-18T16:08:47.883057+0000","last_undegraded":"2024-03-18T16:08:47.883057+0000","last_fullsized":"2024-03-18T16:08:47.883057+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:03:33.730074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"93'15","reported_seq":47,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:56.879106+0000","last_change":"2024-03-18T16:06:43.724589+0000","last_active":"2024-03-18T16:08:56.879106+0000","last_peered":"2024-03-18T16:08:56.879106+0000","last_clean":"2024-03-18T16:08:56.879106+0000","last_became_active":"2024-03-18T16:06:43.724125+0000","last_became_peered":"2024-03-18T16:06:43.724125+0000","last_unstale":"2024-03-18T16:08:56.879106+0000","last_undegraded":"2024-03-18T16:08:56.879106+0000","last_fullsized":"2024-03-18T16:08:56.879106+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:08:34.921118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.886205+0000","last_change":"2024-03-18T16:06:44.727162+0000","last_active":"2024-03-18T16:08:47.886205+0000","last_peered":"2024-03-18T16:08:47.886205+0000","last_clean":"2024-03-18T16:08:47.886205+0000","last_became_active":"2024-03-18T16:06:44.726552+0000","last_became_peered":"2024-03-18T16:06:44.726552+0000","last_unstale":"2024-03-18T16:08:47.886205+0000","last_undegraded":"2024-03-18T16:08:47.886205+0000","last_fullsized":"2024-03-18T16:08:47.886205+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:36:57.974999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883803+0000","last_change":"2024-03-18T16:06:49.008434+0000","last_active":"2024-03-18T16:08:47.883803+0000","last_peered":"2024-03-18T16:08:47.883803+0000","last_clean":"2024-03-18T16:08:47.883803+0000","last_became_active":"2024-03-18T16:06:49.007944+0000","last_became_peered":"2024-03-18T16:06:49.007944+0000","last_unstale":"2024-03-18T16:08:47.883803+0000","last_undegraded":"2024-03-18T16:08:47.883803+0000","last_fullsized":"2024-03-18T16:08:47.883803+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:01:24.450371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'7","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883928+0000","last_change":"2024-03-18T16:06:43.711797+0000","last_active":"2024-03-18T16:08:47.883928+0000","last_peered":"2024-03-18T16:08:47.883928+0000","last_clean":"2024-03-18T16:08:47.883928+0000","last_became_active":"2024-03-18T16:06:43.711493+0000","last_became_peered":"2024-03-18T16:06:43.711493+0000","last_unstale":"2024-03-18T16:08:47.883928+0000","last_undegraded":"2024-03-18T16:08:47.883928+0000","last_fullsized":"2024-03-18T16:08:47.883928+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:50:29.258166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.961906+0000","last_change":"2024-03-18T16:06:44.735910+0000","last_active":"2024-03-18T16:08:47.961906+0000","last_peered":"2024-03-18T16:08:47.961906+0000","last_clean":"2024-03-18T16:08:47.961906+0000","last_became_active":"2024-03-18T16:06:44.735504+0000","last_became_peered":"2024-03-18T16:06:44.735504+0000","last_unstale":"2024-03-18T16:08:47.961906+0000","last_undegraded":"2024-03-18T16:08:47.961906+0000","last_fullsized":"2024-03-18T16:08:47.961906+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:02:18.538217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883708+0000","last_change":"2024-03-18T16:06:49.059130+0000","last_active":"2024-03-18T16:08:47.883708+0000","last_peered":"2024-03-18T16:08:47.883708+0000","last_clean":"2024-03-18T16:08:47.883708+0000","last_became_active":"2024-03-18T16:06:49.058701+0000","last_became_peered":"2024-03-18T16:06:49.058701+0000","last_unstale":"2024-03-18T16:08:47.883708+0000","last_undegraded":"2024-03-18T16:08:47.883708+0000","last_fullsized":"2024-03-18T16:08:47.883708+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:54:52.842786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.959932+0000","last_change":"2024-03-18T16:06:43.714414+0000","last_active":"2024-03-18T16:08:47.959932+0000","last_peered":"2024-03-18T16:08:47.959932+0000","last_clean":"2024-03-18T16:08:47.959932+0000","last_became_active":"2024-03-18T16:06:43.714029+0000","last_became_peered":"2024-03-18T16:06:43.714029+0000","last_unstale":"2024-03-18T16:08:47.959932+0000","last_undegraded":"2024-03-18T16:08:47.959932+0000","last_fullsized":"2024-03-18T16:08:47.959932+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:16:39.060378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.884429+0000","last_change":"2024-03-18T16:06:44.707491+0000","last_active":"2024-03-18T16:08:47.884429+0000","last_peered":"2024-03-18T16:08:47.884429+0000","last_clean":"2024-03-18T16:08:47.884429+0000","last_became_active":"2024-03-18T16:06:44.707157+0000","last_became_peered":"2024-03-18T16:06:44.707157+0000","last_unstale":"2024-03-18T16:08:47.884429+0000","last_undegraded":"2024-03-18T16:08:47.884429+0000","last_fullsized":"2024-03-18T16:08:47.884429+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:19:08.132911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.885895+0000","last_change":"2024-03-18T16:07:47.411002+0000","last_active":"2024-03-18T16:08:47.885895+0000","last_peered":"2024-03-18T16:08:47.885895+0000","last_clean":"2024-03-18T16:08:47.885895+0000","last_became_active":"2024-03-18T16:07:47.410575+0000","last_became_peered":"2024-03-18T16:07:47.410575+0000","last_unstale":"2024-03-18T16:08:47.885895+0000","last_undegraded":"2024-03-18T16:08:47.885895+0000","last_fullsized":"2024-03-18T16:08:47.885895+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:37:25.311205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,2,5],"acting":[8,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"89'7","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.884825+0000","last_change":"2024-03-18T16:06:43.717755+0000","last_active":"2024-03-18T16:08:47.884825+0000","last_peered":"2024-03-18T16:08:47.884825+0000","last_clean":"2024-03-18T16:08:47.884825+0000","last_became_active":"2024-03-18T16:06:43.717278+0000","last_became_peered":"2024-03-18T16:06:43.717278+0000","last_unstale":"2024-03-18T16:08:47.884825+0000","last_undegraded":"2024-03-18T16:08:47.884825+0000","last_fullsized":"2024-03-18T16:08:47.884825+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:48:09.390659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.881317+0000","last_change":"2024-03-18T16:06:44.730858+0000","last_active":"2024-03-18T16:08:47.881317+0000","last_peered":"2024-03-18T16:08:47.881317+0000","last_clean":"2024-03-18T16:08:47.881317+0000","last_became_active":"2024-03-18T16:06:44.730500+0000","last_became_peered":"2024-03-18T16:06:44.730500+0000","last_unstale":"2024-03-18T16:08:47.881317+0000","last_undegraded":"2024-03-18T16:08:47.881317+0000","last_fullsized":"2024-03-18T16:08:47.881317+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:22:03.590834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883615+0000","last_change":"2024-03-18T16:06:49.014717+0000","last_active":"2024-03-18T16:08:47.883615+0000","last_peered":"2024-03-18T16:08:47.883615+0000","last_clean":"2024-03-18T16:08:47.883615+0000","last_became_active":"2024-03-18T16:06:49.014313+0000","last_became_peered":"2024-03-18T16:06:49.014313+0000","last_unstale":"2024-03-18T16:08:47.883615+0000","last_undegraded":"2024-03-18T16:08:47.883615+0000","last_fullsized":"2024-03-18T16:08:47.883615+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:06:12.168888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"93'4","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:53.939577+0000","last_change":"2024-03-18T16:06:43.719496+0000","last_active":"2024-03-18T16:08:53.939577+0000","last_peered":"2024-03-18T16:08:53.939577+0000","last_clean":"2024-03-18T16:08:53.939577+0000","last_became_active":"2024-03-18T16:06:43.719139+0000","last_became_peered":"2024-03-18T16:06:43.719139+0000","last_unstale":"2024-03-18T16:08:53.939577+0000","last_undegraded":"2024-03-18T16:08:53.939577+0000","last_fullsized":"2024-03-18T16:08:53.939577+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:10:01.250923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.886017+0000","last_change":"2024-03-18T16:06:44.737792+0000","last_active":"2024-03-18T16:08:47.886017+0000","last_peered":"2024-03-18T16:08:47.886017+0000","last_clean":"2024-03-18T16:08:47.886017+0000","last_became_active":"2024-03-18T16:06:44.736311+0000","last_became_peered":"2024-03-18T16:06:44.736311+0000","last_unstale":"2024-03-18T16:08:47.886017+0000","last_undegraded":"2024-03-18T16:08:47.886017+0000","last_fullsized":"2024-03-18T16:08:47.886017+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:43:56.110205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883670+0000","last_change":"2024-03-18T16:06:49.058893+0000","last_active":"2024-03-18T16:08:47.883670+0000","last_peered":"2024-03-18T16:08:47.883670+0000","last_clean":"2024-03-18T16:08:47.883670+0000","last_became_active":"2024-03-18T16:06:49.058477+0000","last_became_peered":"2024-03-18T16:06:49.058477+0000","last_unstale":"2024-03-18T16:08:47.883670+0000","last_undegraded":"2024-03-18T16:08:47.883670+0000","last_fullsized":"2024-03-18T16:08:47.883670+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:50:51.890348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.884612+0000","last_change":"2024-03-18T16:06:43.703073+0000","last_active":"2024-03-18T16:08:47.884612+0000","last_peered":"2024-03-18T16:08:47.884612+0000","last_clean":"2024-03-18T16:08:47.884612+0000","last_became_active":"2024-03-18T16:06:43.702708+0000","last_became_peered":"2024-03-18T16:06:43.702708+0000","last_unstale":"2024-03-18T16:08:47.884612+0000","last_undegraded":"2024-03-18T16:08:47.884612+0000","last_fullsized":"2024-03-18T16:08:47.884612+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T23:38:41.454458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.960757+0000","last_change":"2024-03-18T16:06:44.713622+0000","last_active":"2024-03-18T16:08:47.960757+0000","last_peered":"2024-03-18T16:08:47.960757+0000","last_clean":"2024-03-18T16:08:47.960757+0000","last_became_active":"2024-03-18T16:06:44.713160+0000","last_became_peered":"2024-03-18T16:06:44.713160+0000","last_unstale":"2024-03-18T16:08:47.960757+0000","last_undegraded":"2024-03-18T16:08:47.960757+0000","last_fullsized":"2024-03-18T16:08:47.960757+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:49:21.886543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.959925+0000","last_change":"2024-03-18T16:07:47.411958+0000","last_active":"2024-03-18T16:08:47.959925+0000","last_peered":"2024-03-18T16:08:47.959925+0000","last_clean":"2024-03-18T16:08:47.959925+0000","last_became_active":"2024-03-18T16:07:47.411433+0000","last_became_peered":"2024-03-18T16:07:47.411433+0000","last_unstale":"2024-03-18T16:08:47.959925+0000","last_undegraded":"2024-03-18T16:08:47.959925+0000","last_fullsized":"2024-03-18T16:08:47.959925+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T02:58:47.853897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6,7],"acting":[1,8,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"93'6","reported_seq":199,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:09:57.146786+0000","last_change":"2024-03-18T16:06:43.717161+0000","last_active":"2024-03-18T16:09:57.146786+0000","last_peered":"2024-03-18T16:09:57.146786+0000","last_clean":"2024-03-18T16:09:57.146786+0000","last_became_active":"2024-03-18T16:06:43.716657+0000","last_became_peered":"2024-03-18T16:06:43.716657+0000","last_unstale":"2024-03-18T16:09:57.146786+0000","last_undegraded":"2024-03-18T16:09:57.146786+0000","last_fullsized":"2024-03-18T16:09:57.146786+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:23:48.194779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1480,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":162,"num_read_kb":3,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.961120+0000","last_change":"2024-03-18T16:06:44.709777+0000","last_active":"2024-03-18T16:08:47.961120+0000","last_peered":"2024-03-18T16:08:47.961120+0000","last_clean":"2024-03-18T16:08:47.961120+0000","last_became_active":"2024-03-18T16:06:44.709409+0000","last_became_peered":"2024-03-18T16:06:44.709409+0000","last_unstale":"2024-03-18T16:08:47.961120+0000","last_undegraded":"2024-03-18T16:08:47.961120+0000","last_fullsized":"2024-03-18T16:08:47.961120+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T17:37:12.288262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.961539+0000","last_change":"2024-03-18T16:06:49.040872+0000","last_active":"2024-03-18T16:08:47.961539+0000","last_peered":"2024-03-18T16:08:47.961539+0000","last_clean":"2024-03-18T16:08:47.961539+0000","last_became_active":"2024-03-18T16:06:49.039807+0000","last_became_peered":"2024-03-18T16:06:49.039807+0000","last_unstale":"2024-03-18T16:08:47.961539+0000","last_undegraded":"2024-03-18T16:08:47.961539+0000","last_fullsized":"2024-03-18T16:08:47.961539+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:09:27.045450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"89'3","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.886254+0000","last_change":"2024-03-18T16:06:43.723675+0000","last_active":"2024-03-18T16:08:47.886254+0000","last_peered":"2024-03-18T16:08:47.886254+0000","last_clean":"2024-03-18T16:08:47.886254+0000","last_became_active":"2024-03-18T16:06:43.723334+0000","last_became_peered":"2024-03-18T16:06:43.723334+0000","last_unstale":"2024-03-18T16:08:47.886254+0000","last_undegraded":"2024-03-18T16:08:47.886254+0000","last_fullsized":"2024-03-18T16:08:47.886254+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T03:01:36.553362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.961094+0000","last_change":"2024-03-18T16:07:47.359134+0000","last_active":"2024-03-18T16:08:47.961094+0000","last_peered":"2024-03-18T16:08:47.961094+0000","last_clean":"2024-03-18T16:08:47.961094+0000","last_became_active":"2024-03-18T16:07:47.358692+0000","last_became_peered":"2024-03-18T16:07:47.358692+0000","last_unstale":"2024-03-18T16:08:47.961094+0000","last_undegraded":"2024-03-18T16:08:47.961094+0000","last_fullsized":"2024-03-18T16:08:47.961094+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:48:22.929034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.885552+0000","last_change":"2024-03-18T16:06:49.011981+0000","last_active":"2024-03-18T16:08:47.885552+0000","last_peered":"2024-03-18T16:08:47.885552+0000","last_clean":"2024-03-18T16:08:47.885552+0000","last_became_active":"2024-03-18T16:06:49.011621+0000","last_became_peered":"2024-03-18T16:06:49.011621+0000","last_unstale":"2024-03-18T16:08:47.885552+0000","last_undegraded":"2024-03-18T16:08:47.885552+0000","last_fullsized":"2024-03-18T16:08:47.885552+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:47.977111+0000","last_clean_scrub_stamp":"2024-03-18T16:06:47.977111+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:11:42.382721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"89'5","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.962696+0000","last_change":"2024-03-18T16:06:43.725404+0000","last_active":"2024-03-18T16:08:47.962696+0000","last_peered":"2024-03-18T16:08:47.962696+0000","last_clean":"2024-03-18T16:08:47.962696+0000","last_became_active":"2024-03-18T16:06:43.724117+0000","last_became_peered":"2024-03-18T16:06:43.724117+0000","last_unstale":"2024-03-18T16:08:47.962696+0000","last_undegraded":"2024-03-18T16:08:47.962696+0000","last_fullsized":"2024-03-18T16:08:47.962696+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:42.600031+0000","last_clean_scrub_stamp":"2024-03-18T16:06:42.600031+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:58:43.243070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.961302+0000","last_change":"2024-03-18T16:06:44.709979+0000","last_active":"2024-03-18T16:08:47.961302+0000","last_peered":"2024-03-18T16:08:47.961302+0000","last_clean":"2024-03-18T16:08:47.961302+0000","last_became_active":"2024-03-18T16:06:44.709591+0000","last_became_peered":"2024-03-18T16:06:44.709591+0000","last_unstale":"2024-03-18T16:08:47.961302+0000","last_undegraded":"2024-03-18T16:08:47.961302+0000","last_fullsized":"2024-03-18T16:08:47.961302+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:56:37.537248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.881778+0000","last_change":"2024-03-18T16:07:47.354636+0000","last_active":"2024-03-18T16:08:47.881778+0000","last_peered":"2024-03-18T16:08:47.881778+0000","last_clean":"2024-03-18T16:08:47.881778+0000","last_became_active":"2024-03-18T16:07:47.354219+0000","last_became_peered":"2024-03-18T16:07:47.354219+0000","last_unstale":"2024-03-18T16:08:47.881778+0000","last_undegraded":"2024-03-18T16:08:47.881778+0000","last_fullsized":"2024-03-18T16:08:47.881778+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:35:48.094404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,9],"acting":[11,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883718+0000","last_change":"2024-03-18T16:06:44.714836+0000","last_active":"2024-03-18T16:08:47.883718+0000","last_peered":"2024-03-18T16:08:47.883718+0000","last_clean":"2024-03-18T16:08:47.883718+0000","last_became_active":"2024-03-18T16:06:44.714454+0000","last_became_peered":"2024-03-18T16:06:44.714454+0000","last_unstale":"2024-03-18T16:08:47.883718+0000","last_undegraded":"2024-03-18T16:08:47.883718+0000","last_fullsized":"2024-03-18T16:08:47.883718+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T21:05:24.532238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.883502+0000","last_change":"2024-03-18T16:06:44.733183+0000","last_active":"2024-03-18T16:08:47.883502+0000","last_peered":"2024-03-18T16:08:47.883502+0000","last_clean":"2024-03-18T16:08:47.883502+0000","last_became_active":"2024-03-18T16:06:44.732909+0000","last_became_peered":"2024-03-18T16:06:44.732909+0000","last_unstale":"2024-03-18T16:08:47.883502+0000","last_undegraded":"2024-03-18T16:08:47.883502+0000","last_fullsized":"2024-03-18T16:08:47.883502+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:28:07.636692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.961804+0000","last_change":"2024-03-18T16:06:44.731050+0000","last_active":"2024-03-18T16:08:47.961804+0000","last_peered":"2024-03-18T16:08:47.961804+0000","last_clean":"2024-03-18T16:08:47.961804+0000","last_became_active":"2024-03-18T16:06:44.730634+0000","last_became_peered":"2024-03-18T16:06:44.730634+0000","last_unstale":"2024-03-18T16:08:47.961804+0000","last_undegraded":"2024-03-18T16:08:47.961804+0000","last_fullsized":"2024-03-18T16:08:47.961804+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:07:18.619130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.962047+0000","last_change":"2024-03-18T16:06:44.735190+0000","last_active":"2024-03-18T16:08:47.962047+0000","last_peered":"2024-03-18T16:08:47.962047+0000","last_clean":"2024-03-18T16:08:47.962047+0000","last_became_active":"2024-03-18T16:06:44.734312+0000","last_became_peered":"2024-03-18T16:06:44.734312+0000","last_unstale":"2024-03-18T16:08:47.962047+0000","last_undegraded":"2024-03-18T16:08:47.962047+0000","last_fullsized":"2024-03-18T16:08:47.962047+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T16:59:50.209515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.881867+0000","last_change":"2024-03-18T16:06:44.705385+0000","last_active":"2024-03-18T16:08:47.881867+0000","last_peered":"2024-03-18T16:08:47.881867+0000","last_clean":"2024-03-18T16:08:47.881867+0000","last_became_active":"2024-03-18T16:06:44.705011+0000","last_became_peered":"2024-03-18T16:06:44.705011+0000","last_unstale":"2024-03-18T16:08:47.881867+0000","last_undegraded":"2024-03-18T16:08:47.881867+0000","last_fullsized":"2024-03-18T16:08:47.881867+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:50:02.152851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.885287+0000","last_change":"2024-03-18T16:06:44.709621+0000","last_active":"2024-03-18T16:08:47.885287+0000","last_peered":"2024-03-18T16:08:47.885287+0000","last_clean":"2024-03-18T16:08:47.885287+0000","last_became_active":"2024-03-18T16:06:44.709275+0000","last_became_peered":"2024-03-18T16:06:44.709275+0000","last_unstale":"2024-03-18T16:08:47.885287+0000","last_undegraded":"2024-03-18T16:08:47.885287+0000","last_fullsized":"2024-03-18T16:08:47.885287+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:41:17.346661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.886301+0000","last_change":"2024-03-18T16:06:44.719827+0000","last_active":"2024-03-18T16:08:47.886301+0000","last_peered":"2024-03-18T16:08:47.886301+0000","last_clean":"2024-03-18T16:08:47.886301+0000","last_became_active":"2024-03-18T16:06:44.719432+0000","last_became_peered":"2024-03-18T16:06:44.719432+0000","last_unstale":"2024-03-18T16:08:47.886301+0000","last_undegraded":"2024-03-18T16:08:47.886301+0000","last_fullsized":"2024-03-18T16:08:47.886301+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T22:13:42.344211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.885803+0000","last_change":"2024-03-18T16:06:44.714526+0000","last_active":"2024-03-18T16:08:47.885803+0000","last_peered":"2024-03-18T16:08:47.885803+0000","last_clean":"2024-03-18T16:08:47.885803+0000","last_became_active":"2024-03-18T16:06:44.714184+0000","last_became_peered":"2024-03-18T16:06:44.714184+0000","last_unstale":"2024-03-18T16:08:47.885803+0000","last_undegraded":"2024-03-18T16:08:47.885803+0000","last_fullsized":"2024-03-18T16:08:47.885803+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:31:07.917143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.884927+0000","last_change":"2024-03-18T16:06:44.730905+0000","last_active":"2024-03-18T16:08:47.884927+0000","last_peered":"2024-03-18T16:08:47.884927+0000","last_clean":"2024-03-18T16:08:47.884927+0000","last_became_active":"2024-03-18T16:06:44.729002+0000","last_became_peered":"2024-03-18T16:06:44.729002+0000","last_unstale":"2024-03-18T16:08:47.884927+0000","last_undegraded":"2024-03-18T16:08:47.884927+0000","last_fullsized":"2024-03-18T16:08:47.884927+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-20T01:49:18.867571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.960207+0000","last_change":"2024-03-18T16:06:44.727608+0000","last_active":"2024-03-18T16:08:47.960207+0000","last_peered":"2024-03-18T16:08:47.960207+0000","last_clean":"2024-03-18T16:08:47.960207+0000","last_became_active":"2024-03-18T16:06:44.727096+0000","last_became_peered":"2024-03-18T16:06:44.727096+0000","last_unstale":"2024-03-18T16:08:47.960207+0000","last_undegraded":"2024-03-18T16:08:47.960207+0000","last_fullsized":"2024-03-18T16:08:47.960207+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T20:08:20.767443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.962439+0000","last_change":"2024-03-18T16:06:44.731823+0000","last_active":"2024-03-18T16:08:47.962439+0000","last_peered":"2024-03-18T16:08:47.962439+0000","last_clean":"2024-03-18T16:08:47.962439+0000","last_became_active":"2024-03-18T16:06:44.731429+0000","last_became_peered":"2024-03-18T16:06:44.731429+0000","last_unstale":"2024-03-18T16:08:47.962439+0000","last_undegraded":"2024-03-18T16:08:47.962439+0000","last_fullsized":"2024-03-18T16:08:47.962439+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T19:36:28.752466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-18T16:08:47.881596+0000","last_change":"2024-03-18T16:06:44.714354+0000","last_active":"2024-03-18T16:08:47.881596+0000","last_peered":"2024-03-18T16:08:47.881596+0000","last_clean":"2024-03-18T16:08:47.881596+0000","last_became_active":"2024-03-18T16:06:44.713969+0000","last_became_peered":"2024-03-18T16:06:44.713969+0000","last_unstale":"2024-03-18T16:08:47.881596+0000","last_undegraded":"2024-03-18T16:08:47.881596+0000","last_fullsized":"2024-03-18T16:08:47.881596+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-18T16:06:43.671356+0000","last_clean_scrub_stamp":"2024-03-18T16:06:43.671356+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-19T18:18:49.502178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":117,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":73629,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1632,"num_read_kb":901,"num_write":258,"num_write_kb":154,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":90,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":589824,"data_stored":220887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":223,"ondisk_log_size":223,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383735,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36392,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708760,"statfs":{"total":95995035648,"available":95957770240,"internally_reserved":0,"allocated":1077248,"data_stored":520607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645370,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":37064,"kb_used_data":1728,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708088,"statfs":{"total":95995035648,"available":95957082112,"internally_reserved":0,"allocated":1769472,"data_stored":1177484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808895,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1089536,"data_stored":520137,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070535,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36392,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708760,"statfs":{"total":95995035648,"available":95957770240,"internally_reserved":0,"allocated":1081344,"data_stored":520493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332171,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28668,"kb_used_data":1644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716484,"statfs":{"total":95995035648,"available":95965679616,"internally_reserved":0,"allocated":1683456,"data_stored":1117153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495697,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32224,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712928,"statfs":{"total":95995035648,"available":95962038272,"internally_reserved":0,"allocated":1064960,"data_stored":515641,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574957,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36448,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708704,"statfs":{"total":95995035648,"available":95957712896,"internally_reserved":0,"allocated":1138688,"data_stored":583036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738483,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28088,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1089536,"data_stored":517407,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411433,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36460,"kb_used_data":1124,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708692,"statfs":{"total":95995035648,"available":95957700608,"internally_reserved":0,"allocated":1150976,"data_stored":584030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182499,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28068,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717084,"statfs":{"total":95995035648,"available":95966294016,"internally_reserved":0,"allocated":1077248,"data_stored":515349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920861,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36392,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708760,"statfs":{"total":95995035648,"available":95957770240,"internally_reserved":0,"allocated":1073152,"data_stored":519715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757336,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36996,"kb_used_data":1652,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708156,"statfs":{"total":95995035648,"available":95957151744,"internally_reserved":0,"allocated":1691648,"data_stored":1112735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":68117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":69111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":68101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-18T16:10:02.425 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T16:10:02.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.680+0000 7f9366000640 1 -- 172.21.15.39:0/2832173128 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9360074240 msgr2=0x7f93600746c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:02.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.680+0000 7f9366000640 1 --2- 172.21.15.39:0/2832173128 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9360074240 0x7f93600746c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f9348009930 tx=0x7f934802bc20 comp rx=0 tx=0).stop 2024-03-18T16:10:02.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.680+0000 7f9366000640 1 -- 172.21.15.39:0/2832173128 shutdown_connections 2024-03-18T16:10:02.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.680+0000 7f9366000640 1 --2- 172.21.15.39:0/2832173128 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f936010ddc0 0x7f93601101b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:02.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.680+0000 7f9366000640 1 --2- 172.21.15.39:0/2832173128 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9360074240 0x7f93600746c0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:02.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.680+0000 7f9366000640 1 --2- 172.21.15.39:0/2832173128 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9360075820 0x7f9360073c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:02.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.680+0000 7f9366000640 1 -- 172.21.15.39:0/2832173128 >> 172.21.15.39:0/2832173128 conn(0x7f93600fcd80 msgr2=0x7f93600ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:02.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.680+0000 7f9366000640 1 -- 172.21.15.39:0/2832173128 shutdown_connections 2024-03-18T16:10:02.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.680+0000 7f9366000640 1 -- 172.21.15.39:0/2832173128 wait complete. 2024-03-18T16:10:02.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.681+0000 7f9366000640 1 Processor -- start 2024-03-18T16:10:02.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.681+0000 7f9366000640 1 -- start start 2024-03-18T16:10:02.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.682+0000 7f9366000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9360074240 0x7f936019e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:02.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.682+0000 7f9366000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9360075820 0x7f936019e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:02.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.682+0000 7f9366000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f936010ddc0 0x7f93601a5710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:02.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.682+0000 7f9366000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f9360113a40 con 0x7f9360075820 2024-03-18T16:10:02.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.682+0000 7f9366000640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f93601138c0 con 0x7f936010ddc0 2024-03-18T16:10:02.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.682+0000 7f9366000640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f9360113bc0 con 0x7f9360074240 2024-03-18T16:10:02.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.682+0000 7f935f400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9360074240 0x7f936019e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:02.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.682+0000 7f9357e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9360075820 0x7f936019e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:02.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.682+0000 7f935fe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f936010ddc0 0x7f93601a5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:02.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.682+0000 7f9357e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9360075820 0x7f936019e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:55732/0 (socket says 172.21.15.39:55732) 2024-03-18T16:10:02.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.682+0000 7f935fe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f936010ddc0 0x7f93601a5710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:36510/0 (socket says 172.21.15.39:36510) 2024-03-18T16:10:02.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.682+0000 7f935f400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9360074240 0x7f936019e160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:49826/0 (socket says 172.21.15.39:49826) 2024-03-18T16:10:02.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.682+0000 7f9357e00640 1 -- 172.21.15.39:0/4020376708 learned_addr learned my addr 172.21.15.39:0/4020376708 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:10:02.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.683+0000 7f935f400640 1 -- 172.21.15.39:0/4020376708 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f936010ddc0 msgr2=0x7f93601a5710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:02.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.683+0000 7f935f400640 1 --2- 172.21.15.39:0/4020376708 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f936010ddc0 0x7f93601a5710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:02.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.683+0000 7f935f400640 1 -- 172.21.15.39:0/4020376708 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9360075820 msgr2=0x7f936019e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:02.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.683+0000 7f935f400640 1 --2- 172.21.15.39:0/4020376708 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9360075820 0x7f936019e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:02.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.683+0000 7f935f400640 1 -- 172.21.15.39:0/4020376708 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93601a5d80 con 0x7f9360074240 2024-03-18T16:10:02.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.683+0000 7f935fe00640 1 --2- 172.21.15.39:0/4020376708 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f936010ddc0 0x7f93601a5710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:10:02.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.683+0000 7f935f400640 1 --2- 172.21.15.39:0/4020376708 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9360074240 0x7f936019e160 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f935000e9f0 tx=0x7f935000edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:02.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.683+0000 7f935cc00640 1 -- 172.21.15.39:0/4020376708 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f935000ccc0 con 0x7f9360074240 2024-03-18T16:10:02.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.683+0000 7f935cc00640 1 -- 172.21.15.39:0/4020376708 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f93500035f0 con 0x7f9360074240 2024-03-18T16:10:02.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.683+0000 7f935cc00640 1 -- 172.21.15.39:0/4020376708 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f935000ce60 con 0x7f9360074240 2024-03-18T16:10:02.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.683+0000 7f9366000640 1 -- 172.21.15.39:0/4020376708 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93601a6070 con 0x7f9360074240 2024-03-18T16:10:02.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.683+0000 7f9366000640 1 -- 172.21.15.39:0/4020376708 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93601a6570 con 0x7f9360074240 2024-03-18T16:10:02.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.685+0000 7f9366000640 1 -- 172.21.15.39:0/4020376708 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9360103f90 con 0x7f9360074240 2024-03-18T16:10:02.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.686+0000 7f935cc00640 1 -- 172.21.15.39:0/4020376708 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f93500037a0 con 0x7f9360074240 2024-03-18T16:10:02.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.688+0000 7f935cc00640 1 --2- 172.21.15.39:0/4020376708 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f93440a2880 0x7f93440a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:02.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.688+0000 7f935cc00640 1 -- 172.21.15.39:0/4020376708 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f93500cf090 con 0x7f9360074240 2024-03-18T16:10:02.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.688+0000 7f9357e00640 1 --2- 172.21.15.39:0/4020376708 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f93440a2880 0x7f93440a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:02.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.689+0000 7f9357e00640 1 --2- 172.21.15.39:0/4020376708 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f93440a2880 0x7f93440a4d40 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f93480096c0 tx=0x7f934802c600 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:02.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.692+0000 7f935cc00640 1 -- 172.21.15.39:0/4020376708 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f935009bd40 con 0x7f9360074240 2024-03-18T16:10:02.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:10:02 smithi196 ceph-mon[36307]: pgmap v421: 129 pgs: 129 active+clean; 649 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 938 B/s wr, 9 op/s 2024-03-18T16:10:02.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:10:02 smithi196 ceph-mon[36307]: from='client.25520 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T16:10:02.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:10:02 smithi196 ceph-mon[36307]: from='client.25549 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T16:10:02.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:10:02 smithi194 ceph-mon[36200]: pgmap v421: 129 pgs: 129 active+clean; 649 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 938 B/s wr, 9 op/s 2024-03-18T16:10:02.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:10:02 smithi194 ceph-mon[36200]: from='client.25520 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T16:10:02.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:10:02 smithi194 ceph-mon[36200]: from='client.25549 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T16:10:02.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:10:02 smithi039 ceph-mon[31721]: pgmap v421: 129 pgs: 129 active+clean; 649 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 938 B/s wr, 9 op/s 2024-03-18T16:10:02.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:10:02 smithi039 ceph-mon[31721]: from='client.25520 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T16:10:02.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:10:02 smithi039 ceph-mon[31721]: from='client.25549 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-18T16:10:02.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.909+0000 7f9366000640 1 -- 172.21.15.39:0/4020376708 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f93600623e0 con 0x7f9360074240 2024-03-18T16:10:02.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.910+0000 7f935cc00640 1 -- 172.21.15.39:0/4020376708 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24762 (secure 0 0 0) 0x7f9350097030 con 0x7f9360074240 2024-03-18T16:10:02.912 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:10:02.912 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":93,"fsid":"1892d8fc-e540-11ee-95c9-87774f69a715","created":"2024-03-18T15:57:36.507008+0000","modified":"2024-03-18T16:08:47.874967+0000","last_up_change":"2024-03-18T16:05:45.250629+0000","last_in_change":"2024-03-18T16:05:24.841337+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T16:01:30.048247+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T16:06:42.191791+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T16:06:43.670325+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T16:06:47.523388+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"771e0558-be62-4436-94b0-6364751911c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6803","nonce":4090743635}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6805","nonce":4090743635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6809","nonce":4090743635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6807","nonce":4090743635}]},"public_addr":"172.21.15.39:6803/4090743635","cluster_addr":"172.21.15.39:6805/4090743635","heartbeat_back_addr":"172.21.15.39:6809/4090743635","heartbeat_front_addr":"172.21.15.39:6807/4090743635","state":["exists","up"]},{"osd":1,"uuid":"0430eee1-cf34-4277-b255-c7d9f07ad19a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6801","nonce":2800329805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6803","nonce":2800329805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6807","nonce":2800329805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6805","nonce":2800329805}]},"public_addr":"172.21.15.194:6801/2800329805","cluster_addr":"172.21.15.194:6803/2800329805","heartbeat_back_addr":"172.21.15.194:6807/2800329805","heartbeat_front_addr":"172.21.15.194:6805/2800329805","state":["exists","up"]},{"osd":2,"uuid":"3a718da4-2511-4ec1-b1a8-eafd6cce0df0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6801","nonce":3935874719}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6803","nonce":3935874719}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6807","nonce":3935874719}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6805","nonce":3935874719}]},"public_addr":"172.21.15.196:6801/3935874719","cluster_addr":"172.21.15.196:6803/3935874719","heartbeat_back_addr":"172.21.15.196:6807/3935874719","heartbeat_front_addr":"172.21.15.196:6805/3935874719","state":["exists","up"]},{"osd":3,"uuid":"6f9c37f9-0ab9-4038-8919-afd4b39c4165","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6811","nonce":893345939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6813","nonce":893345939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6817","nonce":893345939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6815","nonce":893345939}]},"public_addr":"172.21.15.39:6811/893345939","cluster_addr":"172.21.15.39:6813/893345939","heartbeat_back_addr":"172.21.15.39:6817/893345939","heartbeat_front_addr":"172.21.15.39:6815/893345939","state":["exists","up"]},{"osd":4,"uuid":"e795bb0e-4791-4537-acda-1a321e19bc9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6809","nonce":800423088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6811","nonce":800423088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6815","nonce":800423088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6813","nonce":800423088}]},"public_addr":"172.21.15.194:6809/800423088","cluster_addr":"172.21.15.194:6811/800423088","heartbeat_back_addr":"172.21.15.194:6815/800423088","heartbeat_front_addr":"172.21.15.194:6813/800423088","state":["exists","up"]},{"osd":5,"uuid":"026e175e-6acb-4b4c-b235-8ccee8ae1149","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6809","nonce":2746224525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6811","nonce":2746224525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6815","nonce":2746224525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6813","nonce":2746224525}]},"public_addr":"172.21.15.196:6809/2746224525","cluster_addr":"172.21.15.196:6811/2746224525","heartbeat_back_addr":"172.21.15.196:6815/2746224525","heartbeat_front_addr":"172.21.15.196:6813/2746224525","state":["exists","up"]},{"osd":6,"uuid":"c1b183ff-383f-4c5c-9075-f76bab1c0bb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6819","nonce":1213371331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6821","nonce":1213371331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6825","nonce":1213371331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6823","nonce":1213371331}]},"public_addr":"172.21.15.39:6819/1213371331","cluster_addr":"172.21.15.39:6821/1213371331","heartbeat_back_addr":"172.21.15.39:6825/1213371331","heartbeat_front_addr":"172.21.15.39:6823/1213371331","state":["exists","up"]},{"osd":7,"uuid":"612f8697-957d-4ef4-a87f-241b36f9d13d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6817","nonce":3299162444}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6819","nonce":3299162444}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6823","nonce":3299162444}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6821","nonce":3299162444}]},"public_addr":"172.21.15.194:6817/3299162444","cluster_addr":"172.21.15.194:6819/3299162444","heartbeat_back_addr":"172.21.15.194:6823/3299162444","heartbeat_front_addr":"172.21.15.194:6821/3299162444","state":["exists","up"]},{"osd":8,"uuid":"836a9107-4d4f-4594-b8dc-0301e64f59d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6817","nonce":2456143145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6819","nonce":2456143145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6823","nonce":2456143145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6821","nonce":2456143145}]},"public_addr":"172.21.15.196:6817/2456143145","cluster_addr":"172.21.15.196:6819/2456143145","heartbeat_back_addr":"172.21.15.196:6823/2456143145","heartbeat_front_addr":"172.21.15.196:6821/2456143145","state":["exists","up"]},{"osd":9,"uuid":"4f30b960-12e3-4b16-bd28-fcc988139b58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6827","nonce":1848634433}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6829","nonce":1848634433}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6833","nonce":1848634433}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6831","nonce":1848634433}]},"public_addr":"172.21.15.39:6827/1848634433","cluster_addr":"172.21.15.39:6829/1848634433","heartbeat_back_addr":"172.21.15.39:6833/1848634433","heartbeat_front_addr":"172.21.15.39:6831/1848634433","state":["exists","up"]},{"osd":10,"uuid":"620f90e8-23e6-4bc0-8269-7219b9e1aebd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6825","nonce":3725064809}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6827","nonce":3725064809}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6831","nonce":3725064809}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6829","nonce":3725064809}]},"public_addr":"172.21.15.194:6825/3725064809","cluster_addr":"172.21.15.194:6827/3725064809","heartbeat_back_addr":"172.21.15.194:6831/3725064809","heartbeat_front_addr":"172.21.15.194:6829/3725064809","state":["exists","up"]},{"osd":11,"uuid":"867a2535-8235-4ceb-a698-eb86869d6d20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6825","nonce":2501212422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6827","nonce":2501212422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6831","nonce":2501212422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6829","nonce":2501212422}]},"public_addr":"172.21.15.196:6825/2501212422","cluster_addr":"172.21.15.196:6827/2501212422","heartbeat_back_addr":"172.21.15.196:6831/2501212422","heartbeat_front_addr":"172.21.15.196:6829/2501212422","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:00:36.931566+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:01:02.614024+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:01:26.193265+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:01:58.308833+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:02:25.049546+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:02:49.748750+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:03:23.582153+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:03:50.447321+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:04:15.812129+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:04:49.998047+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:05:17.087924+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:05:43.167637+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":4,"to":7}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.a","mappings":[{"from":5,"to":8}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.39:0/3482086140":"2024-03-19T15:58:45.340294+0000","172.21.15.39:0/3844306284":"2024-03-19T15:58:45.340294+0000","172.21.15.39:6801/1810238546":"2024-03-19T15:58:45.340294+0000","172.21.15.39:6800/1810238546":"2024-03-19T15:58:45.340294+0000","172.21.15.39:0/2528397488":"2024-03-19T15:58:10.264798+0000","172.21.15.39:0/2240981254":"2024-03-19T15:58:10.264798+0000","172.21.15.39:6800/1217503189":"2024-03-19T15:58:10.264798+0000","172.21.15.39:0/3754876370":"2024-03-19T15:58:10.264798+0000","172.21.15.39:6801/1217503189":"2024-03-19T15:58:10.264798+0000","172.21.15.39:0/1874637628":"2024-03-19T15:58:45.340294+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T16:10:02.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.913+0000 7f9366000640 1 -- 172.21.15.39:0/4020376708 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f93440a2880 msgr2=0x7f93440a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:02.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.913+0000 7f9366000640 1 --2- 172.21.15.39:0/4020376708 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f93440a2880 0x7f93440a4d40 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f93480096c0 tx=0x7f934802c600 comp rx=0 tx=0).stop 2024-03-18T16:10:02.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.913+0000 7f9366000640 1 -- 172.21.15.39:0/4020376708 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9360074240 msgr2=0x7f936019e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:02.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.913+0000 7f9366000640 1 --2- 172.21.15.39:0/4020376708 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9360074240 0x7f936019e160 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f935000e9f0 tx=0x7f935000edf0 comp rx=0 tx=0).stop 2024-03-18T16:10:02.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.913+0000 7f9366000640 1 -- 172.21.15.39:0/4020376708 shutdown_connections 2024-03-18T16:10:02.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.913+0000 7f9366000640 1 --2- 172.21.15.39:0/4020376708 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f936010ddc0 0x7f93601a5710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:02.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.913+0000 7f9366000640 1 --2- 172.21.15.39:0/4020376708 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f93440a2880 0x7f93440a4d40 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:02.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.913+0000 7f9366000640 1 --2- 172.21.15.39:0/4020376708 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9360075820 0x7f936019e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:02.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.913+0000 7f9366000640 1 --2- 172.21.15.39:0/4020376708 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9360074240 0x7f936019e160 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:02.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.913+0000 7f9366000640 1 -- 172.21.15.39:0/4020376708 >> 172.21.15.39:0/4020376708 conn(0x7f93600fcd80 msgr2=0x7f9360101160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:02.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.913+0000 7f9366000640 1 -- 172.21.15.39:0/4020376708 shutdown_connections 2024-03-18T16:10:02.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:02.913+0000 7f9366000640 1 -- 172.21.15.39:0/4020376708 wait complete. 2024-03-18T16:10:02.933 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T16:01:30.048247+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T16:06:42.191791+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T16:06:43.670325+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T16:06:47.523388+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T16:10:02.933 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T16:10:03.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.126+0000 7fbd0b200640 1 -- 172.21.15.39:0/1932004058 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbd0c105490 msgr2=0x7fbd0c10bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:03.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.126+0000 7fbd0b200640 1 --2- 172.21.15.39:0/1932004058 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbd0c105490 0x7fbd0c10bd20 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fbd04009ca0 tx=0x7fbd0402c100 comp rx=0 tx=0).stop 2024-03-18T16:10:03.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.126+0000 7fbd0b200640 1 -- 172.21.15.39:0/1932004058 shutdown_connections 2024-03-18T16:10:03.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.126+0000 7fbd0b200640 1 --2- 172.21.15.39:0/1932004058 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbd0c104ad0 0x7fbd0c104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:03.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.126+0000 7fbd0b200640 1 --2- 172.21.15.39:0/1932004058 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbd0c105490 0x7fbd0c10bd20 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:03.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.126+0000 7fbd0b200640 1 --2- 172.21.15.39:0/1932004058 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbd0c1038e0 0x7fbd0c103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:03.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.126+0000 7fbd0b200640 1 -- 172.21.15.39:0/1932004058 >> 172.21.15.39:0/1932004058 conn(0x7fbd0c0fee50 msgr2=0x7fbd0c101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:03.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.126+0000 7fbd0b200640 1 -- 172.21.15.39:0/1932004058 shutdown_connections 2024-03-18T16:10:03.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.126+0000 7fbd0b200640 1 -- 172.21.15.39:0/1932004058 wait complete. 2024-03-18T16:10:03.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.128+0000 7fbd0b200640 1 Processor -- start 2024-03-18T16:10:03.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.128+0000 7fbd0b200640 1 -- start start 2024-03-18T16:10:03.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.128+0000 7fbd0b200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbd0c1038e0 0x7fbd0c199b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:03.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.128+0000 7fbd0b200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbd0c104ad0 0x7fbd0c19a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:03.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.128+0000 7fbd0b200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbd0c105490 0x7fbd0c1a10e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:03.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.128+0000 7fbd0b200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fbd0c10f540 con 0x7fbd0c104ad0 2024-03-18T16:10:03.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.128+0000 7fbd0b200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fbd0c10f3c0 con 0x7fbd0c105490 2024-03-18T16:10:03.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.128+0000 7fbd0b200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fbd0c10f6c0 con 0x7fbd0c1038e0 2024-03-18T16:10:03.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.129+0000 7fbd09e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbd0c1038e0 0x7fbd0c199b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:03.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.129+0000 7fbd09e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbd0c1038e0 0x7fbd0c199b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:49838/0 (socket says 172.21.15.39:49838) 2024-03-18T16:10:03.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.129+0000 7fbd09e00640 1 -- 172.21.15.39:0/2991706350 learned_addr learned my addr 172.21.15.39:0/2991706350 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:10:03.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.129+0000 7fbd09400640 1 --2- 172.21.15.39:0/2991706350 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbd0c104ad0 0x7fbd0c19a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:03.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.129+0000 7fbd0a800640 1 --2- 172.21.15.39:0/2991706350 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbd0c105490 0x7fbd0c1a10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:03.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.129+0000 7fbd09e00640 1 -- 172.21.15.39:0/2991706350 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbd0c105490 msgr2=0x7fbd0c1a10e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:03.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.129+0000 7fbd09e00640 1 --2- 172.21.15.39:0/2991706350 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbd0c105490 0x7fbd0c1a10e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:03.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.129+0000 7fbd09e00640 1 -- 172.21.15.39:0/2991706350 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbd0c104ad0 msgr2=0x7fbd0c19a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:03.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.129+0000 7fbd09e00640 1 --2- 172.21.15.39:0/2991706350 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbd0c104ad0 0x7fbd0c19a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:03.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.129+0000 7fbd09e00640 1 -- 172.21.15.39:0/2991706350 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd0c1a1750 con 0x7fbd0c1038e0 2024-03-18T16:10:03.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.129+0000 7fbd0a800640 1 --2- 172.21.15.39:0/2991706350 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbd0c105490 0x7fbd0c1a10e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:10:03.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.129+0000 7fbd09e00640 1 --2- 172.21.15.39:0/2991706350 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbd0c1038e0 0x7fbd0c199b30 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fbcfc00d950 tx=0x7fbcfc00de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:03.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.130+0000 7fbd02a00640 1 -- 172.21.15.39:0/2991706350 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcfc017020 con 0x7fbd0c1038e0 2024-03-18T16:10:03.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.130+0000 7fbd02a00640 1 -- 172.21.15.39:0/2991706350 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbcfc009c30 con 0x7fbd0c1038e0 2024-03-18T16:10:03.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.130+0000 7fbd02a00640 1 -- 172.21.15.39:0/2991706350 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcfc00f720 con 0x7fbd0c1038e0 2024-03-18T16:10:03.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.130+0000 7fbd0b200640 1 -- 172.21.15.39:0/2991706350 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd0c1a1a40 con 0x7fbd0c1038e0 2024-03-18T16:10:03.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.130+0000 7fbd0b200640 1 -- 172.21.15.39:0/2991706350 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd0c1a1fe0 con 0x7fbd0c1038e0 2024-03-18T16:10:03.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.131+0000 7fbd0b200640 1 -- 172.21.15.39:0/2991706350 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd0c10ac00 con 0x7fbd0c1038e0 2024-03-18T16:10:03.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.133+0000 7fbd02a00640 1 -- 172.21.15.39:0/2991706350 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fbcfc01a020 con 0x7fbd0c1038e0 2024-03-18T16:10:03.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.135+0000 7fbd02a00640 1 --2- 172.21.15.39:0/2991706350 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fbcf00a2880 0x7fbcf00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:03.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.136+0000 7fbd02a00640 1 -- 172.21.15.39:0/2991706350 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fbcfc0cec90 con 0x7fbd0c1038e0 2024-03-18T16:10:03.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.136+0000 7fbd09400640 1 --2- 172.21.15.39:0/2991706350 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fbcf00a2880 0x7fbcf00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:03.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.136+0000 7fbd09400640 1 --2- 172.21.15.39:0/2991706350 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fbcf00a2880 0x7fbcf00a4d40 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fbd0c19b040 tx=0x7fbcf8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:03.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.143+0000 7fbd02a00640 1 -- 172.21.15.39:0/2991706350 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fbcfc021020 con 0x7fbd0c1038e0 2024-03-18T16:10:03.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.429+0000 7fbd0b200640 1 -- 172.21.15.39:0/2991706350 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fbd0c02d1b0 con 0x7fbd0c1038e0 2024-03-18T16:10:03.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.430+0000 7fbd02a00640 1 -- 172.21.15.39:0/2991706350 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7fbd0c02d1b0 con 0x7fbd0c1038e0 2024-03-18T16:10:03.432 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2024-03-18T16:10:03.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.433+0000 7fbd0b200640 1 -- 172.21.15.39:0/2991706350 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fbcf00a2880 msgr2=0x7fbcf00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:03.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.433+0000 7fbd0b200640 1 --2- 172.21.15.39:0/2991706350 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fbcf00a2880 0x7fbcf00a4d40 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fbd0c19b040 tx=0x7fbcf8005f90 comp rx=0 tx=0).stop 2024-03-18T16:10:03.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.433+0000 7fbd0b200640 1 -- 172.21.15.39:0/2991706350 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbd0c1038e0 msgr2=0x7fbd0c199b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:03.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.433+0000 7fbd0b200640 1 --2- 172.21.15.39:0/2991706350 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbd0c1038e0 0x7fbd0c199b30 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fbcfc00d950 tx=0x7fbcfc00de10 comp rx=0 tx=0).stop 2024-03-18T16:10:03.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.433+0000 7fbd0b200640 1 -- 172.21.15.39:0/2991706350 shutdown_connections 2024-03-18T16:10:03.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.433+0000 7fbd0b200640 1 --2- 172.21.15.39:0/2991706350 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbd0c105490 0x7fbd0c1a10e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:03.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.433+0000 7fbd0b200640 1 --2- 172.21.15.39:0/2991706350 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fbcf00a2880 0x7fbcf00a4d40 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:03.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.433+0000 7fbd0b200640 1 --2- 172.21.15.39:0/2991706350 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbd0c104ad0 0x7fbd0c19a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:03.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.433+0000 7fbd0b200640 1 --2- 172.21.15.39:0/2991706350 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbd0c1038e0 0x7fbd0c199b30 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:03.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.433+0000 7fbd0b200640 1 -- 172.21.15.39:0/2991706350 >> 172.21.15.39:0/2991706350 conn(0x7fbd0c0fee50 msgr2=0x7fbd0c107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:03.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.433+0000 7fbd0b200640 1 -- 172.21.15.39:0/2991706350 shutdown_connections 2024-03-18T16:10:03.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.433+0000 7fbd0b200640 1 -- 172.21.15.39:0/2991706350 wait complete. 2024-03-18T16:10:03.454 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T16:10:03.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.688+0000 7f4e6e200640 1 -- 172.21.15.39:0/1761454393 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4e681095a0 msgr2=0x7f4e6810ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:03.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.688+0000 7f4e6e200640 1 --2- 172.21.15.39:0/1761454393 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4e681095a0 0x7f4e6810ba30 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f4e60009ca0 tx=0x7f4e6002c100 comp rx=0 tx=0).stop 2024-03-18T16:10:03.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.688+0000 7f4e6e200640 1 -- 172.21.15.39:0/1761454393 shutdown_connections 2024-03-18T16:10:03.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.688+0000 7f4e6e200640 1 --2- 172.21.15.39:0/1761454393 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4e681095a0 0x7f4e6810ba30 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:03.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.688+0000 7f4e6e200640 1 --2- 172.21.15.39:0/1761454393 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4e68074240 0x7f4e680746c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:03.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.688+0000 7f4e6e200640 1 --2- 172.21.15.39:0/1761454393 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4e68075820 0x7f4e68073c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:03.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.688+0000 7f4e6e200640 1 -- 172.21.15.39:0/1761454393 >> 172.21.15.39:0/1761454393 conn(0x7f4e680fcc50 msgr2=0x7f4e680ff090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:03.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.688+0000 7f4e6e200640 1 -- 172.21.15.39:0/1761454393 shutdown_connections 2024-03-18T16:10:03.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.688+0000 7f4e6e200640 1 -- 172.21.15.39:0/1761454393 wait complete. 2024-03-18T16:10:03.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.689+0000 7f4e6e200640 1 Processor -- start 2024-03-18T16:10:03.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.690+0000 7f4e6e200640 1 -- start start 2024-03-18T16:10:03.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.690+0000 7f4e6e200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4e68074240 0x7f4e68199b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:03.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.690+0000 7f4e6e200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4e68075820 0x7f4e6819a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:03.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.690+0000 7f4e6e200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4e681095a0 0x7f4e681a1110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:03.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.690+0000 7f4e6e200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f4e6810f520 con 0x7f4e68074240 2024-03-18T16:10:03.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.690+0000 7f4e6e200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f4e6810f3a0 con 0x7f4e68075820 2024-03-18T16:10:03.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.690+0000 7f4e6e200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f4e6810f6a0 con 0x7f4e681095a0 2024-03-18T16:10:03.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.690+0000 7f4e6ce00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4e68074240 0x7f4e68199b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:03.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.690+0000 7f4e67e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4e68075820 0x7f4e6819a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:03.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.690+0000 7f4e6d800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4e681095a0 0x7f4e681a1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:03.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.690+0000 7f4e6ce00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4e68074240 0x7f4e68199b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:55762/0 (socket says 172.21.15.39:55762) 2024-03-18T16:10:03.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.690+0000 7f4e67e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4e68075820 0x7f4e6819a0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:36538/0 (socket says 172.21.15.39:36538) 2024-03-18T16:10:03.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.690+0000 7f4e6ce00640 1 -- 172.21.15.39:0/3384750275 learned_addr learned my addr 172.21.15.39:0/3384750275 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:10:03.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.690+0000 7f4e6d800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4e681095a0 0x7f4e681a1110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:49862/0 (socket says 172.21.15.39:49862) 2024-03-18T16:10:03.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.691+0000 7f4e6d800640 1 -- 172.21.15.39:0/3384750275 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4e68075820 msgr2=0x7f4e6819a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:03.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.691+0000 7f4e6d800640 1 --2- 172.21.15.39:0/3384750275 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4e68075820 0x7f4e6819a0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:03.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.691+0000 7f4e6d800640 1 -- 172.21.15.39:0/3384750275 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4e68074240 msgr2=0x7f4e68199b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:03.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.691+0000 7f4e6d800640 1 --2- 172.21.15.39:0/3384750275 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4e68074240 0x7f4e68199b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:03.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.691+0000 7f4e6d800640 1 -- 172.21.15.39:0/3384750275 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e681a1780 con 0x7f4e681095a0 2024-03-18T16:10:03.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.691+0000 7f4e6d800640 1 --2- 172.21.15.39:0/3384750275 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4e681095a0 0x7f4e681a1110 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f4e6002c640 tx=0x7f4e60035d90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:03.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.691+0000 7f4e65600640 1 -- 172.21.15.39:0/3384750275 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e60039040 con 0x7f4e681095a0 2024-03-18T16:10:03.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.691+0000 7f4e65600640 1 -- 172.21.15.39:0/3384750275 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e60044070 con 0x7f4e681095a0 2024-03-18T16:10:03.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.691+0000 7f4e6e200640 1 -- 172.21.15.39:0/3384750275 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e681a1a10 con 0x7f4e681095a0 2024-03-18T16:10:03.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.691+0000 7f4e65600640 1 -- 172.21.15.39:0/3384750275 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e600044a0 con 0x7f4e681095a0 2024-03-18T16:10:03.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.692+0000 7f4e6e200640 1 -- 172.21.15.39:0/3384750275 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e681a1f20 con 0x7f4e681095a0 2024-03-18T16:10:03.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.693+0000 7f4e6e200640 1 -- 172.21.15.39:0/3384750275 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e681082d0 con 0x7f4e681095a0 2024-03-18T16:10:03.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.695+0000 7f4e65600640 1 -- 172.21.15.39:0/3384750275 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f4e60004640 con 0x7f4e681095a0 2024-03-18T16:10:03.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.696+0000 7f4e65600640 1 --2- 172.21.15.39:0/3384750275 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f4e4c0a2880 0x7f4e4c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:03.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.696+0000 7f4e65600640 1 -- 172.21.15.39:0/3384750275 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f4e600f15f0 con 0x7f4e681095a0 2024-03-18T16:10:03.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.696+0000 7f4e6ce00640 1 --2- 172.21.15.39:0/3384750275 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f4e4c0a2880 0x7f4e4c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:03.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.697+0000 7f4e6ce00640 1 --2- 172.21.15.39:0/3384750275 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f4e4c0a2880 0x7f4e4c0a4d40 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f4e58005fd0 tx=0x7f4e5800b480 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:03.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.698+0000 7f4e65600640 1 -- 172.21.15.39:0/3384750275 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4e600be2a0 con 0x7f4e681095a0 2024-03-18T16:10:03.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:10:03 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/4020376708' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:10:03.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:10:03 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/2991706350' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T16:10:03.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:10:03 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/4020376708' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:10:03.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:10:03 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/2991706350' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T16:10:03.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:10:03 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/4020376708' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:10:03.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:10:03 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/2991706350' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T16:10:03.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.912+0000 7f4e6e200640 1 -- 172.21.15.39:0/3384750275 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f4e6802d360 con 0x7f4e681095a0 2024-03-18T16:10:03.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.912+0000 7f4e65600640 1 -- 172.21.15.39:0/3384750275 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7f4e600098b0 con 0x7f4e681095a0 2024-03-18T16:10:03.914 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 32 2024-03-18T16:10:03.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.916+0000 7f4e6e200640 1 -- 172.21.15.39:0/3384750275 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f4e4c0a2880 msgr2=0x7f4e4c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:03.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.916+0000 7f4e6e200640 1 --2- 172.21.15.39:0/3384750275 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f4e4c0a2880 0x7f4e4c0a4d40 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f4e58005fd0 tx=0x7f4e5800b480 comp rx=0 tx=0).stop 2024-03-18T16:10:03.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.916+0000 7f4e6e200640 1 -- 172.21.15.39:0/3384750275 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4e681095a0 msgr2=0x7f4e681a1110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:03.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.916+0000 7f4e6e200640 1 --2- 172.21.15.39:0/3384750275 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4e681095a0 0x7f4e681a1110 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f4e6002c640 tx=0x7f4e60035d90 comp rx=0 tx=0).stop 2024-03-18T16:10:03.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.916+0000 7f4e6e200640 1 -- 172.21.15.39:0/3384750275 shutdown_connections 2024-03-18T16:10:03.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.916+0000 7f4e6e200640 1 --2- 172.21.15.39:0/3384750275 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4e68075820 0x7f4e6819a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:03.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.916+0000 7f4e6e200640 1 --2- 172.21.15.39:0/3384750275 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f4e4c0a2880 0x7f4e4c0a4d40 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:03.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.916+0000 7f4e6e200640 1 --2- 172.21.15.39:0/3384750275 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4e681095a0 0x7f4e681a1110 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:03.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.916+0000 7f4e6e200640 1 --2- 172.21.15.39:0/3384750275 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4e68074240 0x7f4e68199b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:03.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.916+0000 7f4e6e200640 1 -- 172.21.15.39:0/3384750275 >> 172.21.15.39:0/3384750275 conn(0x7f4e680fcc50 msgr2=0x7f4e681054a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:03.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.917+0000 7f4e6e200640 1 -- 172.21.15.39:0/3384750275 shutdown_connections 2024-03-18T16:10:03.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:03.917+0000 7f4e6e200640 1 -- 172.21.15.39:0/3384750275 wait complete. 2024-03-18T16:10:03.940 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T16:10:04.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.165+0000 7f8a26200640 1 -- 172.21.15.39:0/4250579425 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8a201017f0 msgr2=0x7f8a20101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:04.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.165+0000 7f8a26200640 1 --2- 172.21.15.39:0/4250579425 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8a201017f0 0x7f8a20101c10 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f8a10009900 tx=0x7f8a1002bc30 comp rx=0 tx=0).stop 2024-03-18T16:10:04.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.165+0000 7f8a26200640 1 -- 172.21.15.39:0/4250579425 shutdown_connections 2024-03-18T16:10:04.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.165+0000 7f8a26200640 1 --2- 172.21.15.39:0/4250579425 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8a201033a0 0x7f8a20109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:04.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.165+0000 7f8a26200640 1 --2- 172.21.15.39:0/4250579425 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a201029e0 0x7f8a20102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:04.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.165+0000 7f8a26200640 1 --2- 172.21.15.39:0/4250579425 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8a201017f0 0x7f8a20101c10 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:04.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.165+0000 7f8a26200640 1 -- 172.21.15.39:0/4250579425 >> 172.21.15.39:0/4250579425 conn(0x7f8a200fcd80 msgr2=0x7f8a200ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:04.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.165+0000 7f8a26200640 1 -- 172.21.15.39:0/4250579425 shutdown_connections 2024-03-18T16:10:04.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.165+0000 7f8a26200640 1 -- 172.21.15.39:0/4250579425 wait complete. 2024-03-18T16:10:04.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.166+0000 7f8a26200640 1 Processor -- start 2024-03-18T16:10:04.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.166+0000 7f8a26200640 1 -- start start 2024-03-18T16:10:04.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.167+0000 7f8a26200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a201017f0 0x7f8a20199d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:04.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.167+0000 7f8a26200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8a201029e0 0x7f8a2019a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:04.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.167+0000 7f8a26200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8a201033a0 0x7f8a201a1320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:04.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.167+0000 7f8a26200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f8a2010d4c0 con 0x7f8a201017f0 2024-03-18T16:10:04.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.167+0000 7f8a26200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f8a2010d340 con 0x7f8a201033a0 2024-03-18T16:10:04.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.167+0000 7f8a26200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f8a2010d640 con 0x7f8a201029e0 2024-03-18T16:10:04.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.167+0000 7f8a16a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8a201029e0 0x7f8a2019a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:04.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.167+0000 7f8a1fe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8a201033a0 0x7f8a201a1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:04.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.167+0000 7f8a1f400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a201017f0 0x7f8a20199d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:04.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.167+0000 7f8a16a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8a201029e0 0x7f8a2019a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:49892/0 (socket says 172.21.15.39:49892) 2024-03-18T16:10:04.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.167+0000 7f8a16a00640 1 -- 172.21.15.39:0/1831844190 learned_addr learned my addr 172.21.15.39:0/1831844190 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:10:04.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.167+0000 7f8a16a00640 1 -- 172.21.15.39:0/1831844190 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8a201033a0 msgr2=0x7f8a201a1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:04.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.167+0000 7f8a16a00640 1 --2- 172.21.15.39:0/1831844190 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8a201033a0 0x7f8a201a1320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:04.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.167+0000 7f8a16a00640 1 -- 172.21.15.39:0/1831844190 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a201017f0 msgr2=0x7f8a20199d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:04.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.167+0000 7f8a16a00640 1 --2- 172.21.15.39:0/1831844190 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a201017f0 0x7f8a20199d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:04.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.167+0000 7f8a16a00640 1 -- 172.21.15.39:0/1831844190 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a201a1990 con 0x7f8a201029e0 2024-03-18T16:10:04.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.167+0000 7f8a1fe00640 1 --2- 172.21.15.39:0/1831844190 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8a201033a0 0x7f8a201a1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:10:04.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.168+0000 7f8a16a00640 1 --2- 172.21.15.39:0/1831844190 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8a201029e0 0x7f8a2019a2b0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f8a0800e9f0 tx=0x7f8a0800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:04.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.168+0000 7f8a1cc00640 1 -- 172.21.15.39:0/1831844190 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a0800cd80 con 0x7f8a201029e0 2024-03-18T16:10:04.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.168+0000 7f8a1cc00640 1 -- 172.21.15.39:0/1831844190 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a08010910 con 0x7f8a201029e0 2024-03-18T16:10:04.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.168+0000 7f8a26200640 1 -- 172.21.15.39:0/1831844190 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a201a1c80 con 0x7f8a201029e0 2024-03-18T16:10:04.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.168+0000 7f8a1cc00640 1 -- 172.21.15.39:0/1831844190 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a08018990 con 0x7f8a201029e0 2024-03-18T16:10:04.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.168+0000 7f8a26200640 1 -- 172.21.15.39:0/1831844190 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a201a21d0 con 0x7f8a201029e0 2024-03-18T16:10:04.172 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.169+0000 7f8a26200640 1 -- 172.21.15.39:0/1831844190 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a201a2400 con 0x7f8a201029e0 2024-03-18T16:10:04.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.173+0000 7f8a1cc00640 1 -- 172.21.15.39:0/1831844190 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f8a08010ac0 con 0x7f8a201029e0 2024-03-18T16:10:04.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.174+0000 7f8a1cc00640 1 --2- 172.21.15.39:0/1831844190 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f8a040a2880 0x7f8a040a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:04.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.175+0000 7f8a1cc00640 1 -- 172.21.15.39:0/1831844190 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f8a080cf250 con 0x7f8a201029e0 2024-03-18T16:10:04.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.175+0000 7f8a1f400640 1 --2- 172.21.15.39:0/1831844190 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f8a040a2880 0x7f8a040a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:04.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.175+0000 7f8a1cc00640 1 -- 172.21.15.39:0/1831844190 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8a0809b8b0 con 0x7f8a201029e0 2024-03-18T16:10:04.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.176+0000 7f8a1f400640 1 --2- 172.21.15.39:0/1831844190 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f8a040a2880 0x7f8a040a4d40 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f8a1002c170 tx=0x7f8a1002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:04.383 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.381+0000 7f8a26200640 1 -- 172.21.15.39:0/1831844190 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f8a201a26f0 con 0x7f8a201029e0 2024-03-18T16:10:04.383 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.382+0000 7f8a1cc00640 1 -- 172.21.15.39:0/1831844190 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7f8a080cce20 con 0x7f8a201029e0 2024-03-18T16:10:04.383 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 64 2024-03-18T16:10:04.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.385+0000 7f8a26200640 1 -- 172.21.15.39:0/1831844190 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f8a040a2880 msgr2=0x7f8a040a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:04.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.385+0000 7f8a26200640 1 --2- 172.21.15.39:0/1831844190 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f8a040a2880 0x7f8a040a4d40 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f8a1002c170 tx=0x7f8a1002df90 comp rx=0 tx=0).stop 2024-03-18T16:10:04.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.385+0000 7f8a26200640 1 -- 172.21.15.39:0/1831844190 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8a201029e0 msgr2=0x7f8a2019a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:04.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.385+0000 7f8a26200640 1 --2- 172.21.15.39:0/1831844190 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8a201029e0 0x7f8a2019a2b0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f8a0800e9f0 tx=0x7f8a0800eeb0 comp rx=0 tx=0).stop 2024-03-18T16:10:04.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.385+0000 7f8a26200640 1 -- 172.21.15.39:0/1831844190 shutdown_connections 2024-03-18T16:10:04.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.385+0000 7f8a26200640 1 --2- 172.21.15.39:0/1831844190 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8a201033a0 0x7f8a201a1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:04.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.385+0000 7f8a26200640 1 --2- 172.21.15.39:0/1831844190 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f8a040a2880 0x7f8a040a4d40 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:04.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.385+0000 7f8a26200640 1 --2- 172.21.15.39:0/1831844190 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8a201029e0 0x7f8a2019a2b0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:04.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.385+0000 7f8a26200640 1 --2- 172.21.15.39:0/1831844190 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a201017f0 0x7f8a20199d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:04.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.385+0000 7f8a26200640 1 -- 172.21.15.39:0/1831844190 >> 172.21.15.39:0/1831844190 conn(0x7f8a200fcd80 msgr2=0x7f8a20073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:04.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.385+0000 7f8a26200640 1 -- 172.21.15.39:0/1831844190 shutdown_connections 2024-03-18T16:10:04.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.385+0000 7f8a26200640 1 -- 172.21.15.39:0/1831844190 wait complete. 2024-03-18T16:10:04.409 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T16:10:04.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.656+0000 7fd391800640 1 -- 172.21.15.39:0/1091163197 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd38c1028d0 msgr2=0x7fd38c102d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:04.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.656+0000 7fd391800640 1 --2- 172.21.15.39:0/1091163197 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd38c1028d0 0x7fd38c102d50 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fd378009900 tx=0x7fd37802bc30 comp rx=0 tx=0).stop 2024-03-18T16:10:04.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.656+0000 7fd391800640 1 -- 172.21.15.39:0/1091163197 shutdown_connections 2024-03-18T16:10:04.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.656+0000 7fd391800640 1 --2- 172.21.15.39:0/1091163197 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd38c1028d0 0x7fd38c102d50 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:04.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.656+0000 7fd391800640 1 --2- 172.21.15.39:0/1091163197 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd38c103290 0x7fd38c109b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:04.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.656+0000 7fd391800640 1 --2- 172.21.15.39:0/1091163197 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd38c1016e0 0x7fd38c101b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:04.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.656+0000 7fd391800640 1 -- 172.21.15.39:0/1091163197 >> 172.21.15.39:0/1091163197 conn(0x7fd38c0fcc50 msgr2=0x7fd38c0ff090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:04.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.656+0000 7fd391800640 1 -- 172.21.15.39:0/1091163197 shutdown_connections 2024-03-18T16:10:04.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.656+0000 7fd391800640 1 -- 172.21.15.39:0/1091163197 wait complete. 2024-03-18T16:10:04.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.657+0000 7fd391800640 1 Processor -- start 2024-03-18T16:10:04.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.658+0000 7fd391800640 1 -- start start 2024-03-18T16:10:04.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.658+0000 7fd391800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd38c1016e0 0x7fd38c075590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:04.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.658+0000 7fd391800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd38c1028d0 0x7fd38c073bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:04.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.658+0000 7fd391800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd38c103290 0x7fd38c074110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:04.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.658+0000 7fd391800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fd38c10d360 con 0x7fd38c1016e0 2024-03-18T16:10:04.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.658+0000 7fd391800640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fd38c10d1e0 con 0x7fd38c1028d0 2024-03-18T16:10:04.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.658+0000 7fd391800640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fd38c10d4e0 con 0x7fd38c103290 2024-03-18T16:10:04.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.658+0000 7fd38be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd38c1016e0 0x7fd38c075590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:04.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.658+0000 7fd38b400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd38c1028d0 0x7fd38c073bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:04.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.658+0000 7fd38be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd38c1016e0 0x7fd38c075590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:55794/0 (socket says 172.21.15.39:55794) 2024-03-18T16:10:04.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.658+0000 7fd38be00640 1 -- 172.21.15.39:0/2991740050 learned_addr learned my addr 172.21.15.39:0/2991740050 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:10:04.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.658+0000 7fd38b400640 1 --2- 172.21.15.39:0/2991740050 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd38c1028d0 0x7fd38c073bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:36566/0 (socket says 172.21.15.39:36566) 2024-03-18T16:10:04.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.659+0000 7fd390e00640 1 --2- 172.21.15.39:0/2991740050 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd38c103290 0x7fd38c074110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:04.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.659+0000 7fd38b400640 1 -- 172.21.15.39:0/2991740050 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd38c103290 msgr2=0x7fd38c074110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:04.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.659+0000 7fd38b400640 1 --2- 172.21.15.39:0/2991740050 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd38c103290 0x7fd38c074110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:04.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.659+0000 7fd38b400640 1 -- 172.21.15.39:0/2991740050 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd38c1016e0 msgr2=0x7fd38c075590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:04.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.659+0000 7fd38b400640 1 --2- 172.21.15.39:0/2991740050 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd38c1016e0 0x7fd38c075590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:04.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.659+0000 7fd38b400640 1 -- 172.21.15.39:0/2991740050 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd38c0749a0 con 0x7fd38c1028d0 2024-03-18T16:10:04.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.659+0000 7fd38b400640 1 --2- 172.21.15.39:0/2991740050 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd38c1028d0 0x7fd38c073bd0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fd378009230 tx=0x7fd37802c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:04.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.659+0000 7fd388c00640 1 -- 172.21.15.39:0/2991740050 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd378041070 con 0x7fd38c1028d0 2024-03-18T16:10:04.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.660+0000 7fd388c00640 1 -- 172.21.15.39:0/2991740050 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd378004450 con 0x7fd38c1028d0 2024-03-18T16:10:04.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.660+0000 7fd391800640 1 -- 172.21.15.39:0/2991740050 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd38c1a1aa0 con 0x7fd38c1028d0 2024-03-18T16:10:04.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.660+0000 7fd388c00640 1 -- 172.21.15.39:0/2991740050 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd378003df0 con 0x7fd38c1028d0 2024-03-18T16:10:04.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.660+0000 7fd391800640 1 -- 172.21.15.39:0/2991740050 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd38c1a2040 con 0x7fd38c1028d0 2024-03-18T16:10:04.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.661+0000 7fd391800640 1 -- 172.21.15.39:0/2991740050 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd38c108a00 con 0x7fd38c1028d0 2024-03-18T16:10:04.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.662+0000 7fd388c00640 1 -- 172.21.15.39:0/2991740050 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fd378044070 con 0x7fd38c1028d0 2024-03-18T16:10:04.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.664+0000 7fd388c00640 1 --2- 172.21.15.39:0/2991740050 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd3700a2880 0x7fd3700a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:04.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.664+0000 7fd388c00640 1 -- 172.21.15.39:0/2991740050 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fd3780f07f0 con 0x7fd38c1028d0 2024-03-18T16:10:04.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.664+0000 7fd38be00640 1 --2- 172.21.15.39:0/2991740050 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd3700a2880 0x7fd3700a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:04.666 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.665+0000 7fd38be00640 1 --2- 172.21.15.39:0/2991740050 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd3700a2880 0x7fd3700a4d40 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fd37c0095f0 tx=0x7fd37c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:04.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.668+0000 7fd388c00640 1 -- 172.21.15.39:0/2991740050 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd3780bce50 con 0x7fd38c1028d0 2024-03-18T16:10:04.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:10:04 smithi196 ceph-mon[36307]: pgmap v422: 129 pgs: 129 active+clean; 649 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 938 B/s wr, 10 op/s 2024-03-18T16:10:04.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:10:04 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/3384750275' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T16:10:04.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:10:04 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/1831844190' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T16:10:04.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:10:04 smithi194 ceph-mon[36200]: pgmap v422: 129 pgs: 129 active+clean; 649 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 938 B/s wr, 10 op/s 2024-03-18T16:10:04.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:10:04 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/3384750275' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T16:10:04.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:10:04 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/1831844190' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T16:10:04.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:10:04 smithi039 ceph-mon[31721]: pgmap v422: 129 pgs: 129 active+clean; 649 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 938 B/s wr, 10 op/s 2024-03-18T16:10:04.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:10:04 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/3384750275' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T16:10:04.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:10:04 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/1831844190' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T16:10:04.873 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.872+0000 7fd391800640 1 -- 172.21.15.39:0/2991740050 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fd38c0623e0 con 0x7fd38c1028d0 2024-03-18T16:10:04.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.873+0000 7fd388c00640 1 -- 172.21.15.39:0/2991740050 <== mon.2 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7fd3780c0ef0 con 0x7fd38c1028d0 2024-03-18T16:10:04.875 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 32 2024-03-18T16:10:04.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.876+0000 7fd391800640 1 -- 172.21.15.39:0/2991740050 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd3700a2880 msgr2=0x7fd3700a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:04.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.876+0000 7fd391800640 1 --2- 172.21.15.39:0/2991740050 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd3700a2880 0x7fd3700a4d40 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fd37c0095f0 tx=0x7fd37c005f90 comp rx=0 tx=0).stop 2024-03-18T16:10:04.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.876+0000 7fd391800640 1 -- 172.21.15.39:0/2991740050 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd38c1028d0 msgr2=0x7fd38c073bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:04.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.876+0000 7fd391800640 1 --2- 172.21.15.39:0/2991740050 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd38c1028d0 0x7fd38c073bd0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fd378009230 tx=0x7fd37802c650 comp rx=0 tx=0).stop 2024-03-18T16:10:04.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.876+0000 7fd391800640 1 -- 172.21.15.39:0/2991740050 shutdown_connections 2024-03-18T16:10:04.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.876+0000 7fd391800640 1 --2- 172.21.15.39:0/2991740050 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd38c1028d0 0x7fd38c073bd0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:04.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.876+0000 7fd391800640 1 --2- 172.21.15.39:0/2991740050 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd3700a2880 0x7fd3700a4d40 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:04.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.876+0000 7fd391800640 1 --2- 172.21.15.39:0/2991740050 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd38c103290 0x7fd38c074110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:04.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.876+0000 7fd391800640 1 --2- 172.21.15.39:0/2991740050 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd38c1016e0 0x7fd38c075590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:04.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.876+0000 7fd391800640 1 -- 172.21.15.39:0/2991740050 >> 172.21.15.39:0/2991740050 conn(0x7fd38c0fcc50 msgr2=0x7fd38c105bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:04.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.876+0000 7fd391800640 1 -- 172.21.15.39:0/2991740050 shutdown_connections 2024-03-18T16:10:04.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:04.876+0000 7fd391800640 1 -- 172.21.15.39:0/2991740050 wait complete. 2024-03-18T16:10:04.899 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:10:05.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.100+0000 7fad38c00640 1 -- 172.21.15.39:0/1084130649 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fad34101630 msgr2=0x7fad34101a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:05.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.100+0000 7fad38c00640 1 --2- 172.21.15.39:0/1084130649 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fad34101630 0x7fad34101a50 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fad24009900 tx=0x7fad2402bc50 comp rx=0 tx=0).stop 2024-03-18T16:10:05.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.100+0000 7fad38c00640 1 -- 172.21.15.39:0/1084130649 shutdown_connections 2024-03-18T16:10:05.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.100+0000 7fad38c00640 1 --2- 172.21.15.39:0/1084130649 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fad34102820 0x7fad34102ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:05.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.100+0000 7fad38c00640 1 --2- 172.21.15.39:0/1084130649 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fad341031e0 0x7fad34109a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:05.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.100+0000 7fad38c00640 1 --2- 172.21.15.39:0/1084130649 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fad34101630 0x7fad34101a50 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:05.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.100+0000 7fad38c00640 1 -- 172.21.15.39:0/1084130649 >> 172.21.15.39:0/1084130649 conn(0x7fad340fcbc0 msgr2=0x7fad340fefe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:05.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.100+0000 7fad38c00640 1 -- 172.21.15.39:0/1084130649 shutdown_connections 2024-03-18T16:10:05.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.100+0000 7fad38c00640 1 -- 172.21.15.39:0/1084130649 wait complete. 2024-03-18T16:10:05.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.101+0000 7fad38c00640 1 Processor -- start 2024-03-18T16:10:05.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.101+0000 7fad38c00640 1 -- start start 2024-03-18T16:10:05.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.101+0000 7fad38c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fad34101630 0x7fad34199ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:05.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.101+0000 7fad38c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fad34102820 0x7fad3419a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:05.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.101+0000 7fad38c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fad341031e0 0x7fad341a1090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:05.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.101+0000 7fad38c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fad3410d300 con 0x7fad34101630 2024-03-18T16:10:05.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.101+0000 7fad38c00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fad3410d180 con 0x7fad34102820 2024-03-18T16:10:05.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.101+0000 7fad38c00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fad3410d480 con 0x7fad341031e0 2024-03-18T16:10:05.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.102+0000 7fad32a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fad34102820 0x7fad3419a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:05.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.102+0000 7fad33400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fad34101630 0x7fad34199ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:05.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.102+0000 7fad32a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fad34102820 0x7fad3419a020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:36586/0 (socket says 172.21.15.39:36586) 2024-03-18T16:10:05.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.102+0000 7fad32a00640 1 -- 172.21.15.39:0/59049223 learned_addr learned my addr 172.21.15.39:0/59049223 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:10:05.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.102+0000 7fad33e00640 1 --2- 172.21.15.39:0/59049223 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fad341031e0 0x7fad341a1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:05.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.102+0000 7fad32a00640 1 -- 172.21.15.39:0/59049223 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fad341031e0 msgr2=0x7fad341a1090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:05.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.102+0000 7fad32a00640 1 --2- 172.21.15.39:0/59049223 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fad341031e0 0x7fad341a1090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:05.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.102+0000 7fad32a00640 1 -- 172.21.15.39:0/59049223 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fad34101630 msgr2=0x7fad34199ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:05.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.102+0000 7fad32a00640 1 --2- 172.21.15.39:0/59049223 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fad34101630 0x7fad34199ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:05.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.102+0000 7fad32a00640 1 -- 172.21.15.39:0/59049223 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad341a1700 con 0x7fad34102820 2024-03-18T16:10:05.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.102+0000 7fad32a00640 1 --2- 172.21.15.39:0/59049223 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fad34102820 0x7fad3419a020 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fad2000ea60 tx=0x7fad2000ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:05.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.103+0000 7fad2be00640 1 -- 172.21.15.39:0/59049223 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad2000cde0 con 0x7fad34102820 2024-03-18T16:10:05.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.103+0000 7fad2be00640 1 -- 172.21.15.39:0/59049223 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad200035f0 con 0x7fad34102820 2024-03-18T16:10:05.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.103+0000 7fad2be00640 1 -- 172.21.15.39:0/59049223 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad20010670 con 0x7fad34102820 2024-03-18T16:10:05.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.103+0000 7fad38c00640 1 -- 172.21.15.39:0/59049223 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad341a19f0 con 0x7fad34102820 2024-03-18T16:10:05.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.103+0000 7fad38c00640 1 -- 172.21.15.39:0/59049223 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad341a1ef0 con 0x7fad34102820 2024-03-18T16:10:05.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.104+0000 7fad38c00640 1 -- 172.21.15.39:0/59049223 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad341a2180 con 0x7fad34102820 2024-03-18T16:10:05.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.107+0000 7fad2be00640 1 -- 172.21.15.39:0/59049223 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fad200037a0 con 0x7fad34102820 2024-03-18T16:10:05.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.108+0000 7fad2be00640 1 --2- 172.21.15.39:0/59049223 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fad180a2880 0x7fad180a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:05.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.108+0000 7fad2be00640 1 -- 172.21.15.39:0/59049223 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fad200cf090 con 0x7fad34102820 2024-03-18T16:10:05.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.108+0000 7fad33400640 1 --2- 172.21.15.39:0/59049223 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fad180a2880 0x7fad180a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:05.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.109+0000 7fad33400640 1 --2- 172.21.15.39:0/59049223 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fad180a2880 0x7fad180a4d40 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fad2402c190 tx=0x7fad24000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:05.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.109+0000 7fad2be00640 1 -- 172.21.15.39:0/59049223 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fad2009bd40 con 0x7fad34102820 2024-03-18T16:10:05.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.354+0000 7fad38c00640 1 -- 172.21.15.39:0/59049223 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fad341a2620 con 0x7fad34102820 2024-03-18T16:10:05.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.355+0000 7fad2be00640 1 -- 172.21.15.39:0/59049223 <== mon.2 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11932 (secure 0 0 0) 0x7fad20097030 con 0x7fad34102820 2024-03-18T16:10:05.357 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:10:05.357 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2520226608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2520226608},{"type":"v1","addr":"172.21.15.39:6835","nonce":2520226608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/719341763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":719341763},{"type":"v1","addr":"172.21.15.39:6837","nonce":719341763}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T16:06:44.180425+0000","modified":"2024-03-18T16:09:54.861629+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24575,"mds_1":24556,"mds_2":24535,"mds_3":24524,"mds_4":14463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14445":{"gid":14445,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.39:6839/656513813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":656513813},{"type":"v1","addr":"172.21.15.39:6839","nonce":656513813}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14463":{"gid":14463,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":13,"addr":"172.21.15.39:6841/3515326042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3515326042},{"type":"v1","addr":"172.21.15.39:6841","nonce":3515326042}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6833/931350738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":931350738},{"type":"v1","addr":"172.21.15.194:6833","nonce":931350738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6833/3874209621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":3874209621},{"type":"v1","addr":"172.21.15.196:6833","nonce":3874209621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6835/188385470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":188385470},{"type":"v1","addr":"172.21.15.194:6835","nonce":188385470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6835/3949984309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":3949984309},{"type":"v1","addr":"172.21.15.196:6835","nonce":3949984309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":15,"addr":"172.21.15.194:6837/1301464246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":1301464246},{"type":"v1","addr":"172.21.15.194:6837","nonce":1301464246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":16,"addr":"172.21.15.196:6837/895247251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":895247251},{"type":"v1","addr":"172.21.15.196:6837","nonce":895247251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.196:6839/2001427268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6838","nonce":2001427268},{"type":"v1","addr":"172.21.15.196:6839","nonce":2001427268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24575":{"gid":24575,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.194:6839/274911158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":274911158},{"type":"v1","addr":"172.21.15.194:6839","nonce":274911158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24575,"qdb_cluster":[14463,24524,24575,24556,24535]},"id":1}]} 2024-03-18T16:10:05.357 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 28 2024-03-18T16:10:05.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.358+0000 7fad38c00640 1 -- 172.21.15.39:0/59049223 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fad180a2880 msgr2=0x7fad180a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:05.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.358+0000 7fad38c00640 1 --2- 172.21.15.39:0/59049223 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fad180a2880 0x7fad180a4d40 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fad2402c190 tx=0x7fad24000bc0 comp rx=0 tx=0).stop 2024-03-18T16:10:05.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.358+0000 7fad38c00640 1 -- 172.21.15.39:0/59049223 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fad34102820 msgr2=0x7fad3419a020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:05.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.358+0000 7fad38c00640 1 --2- 172.21.15.39:0/59049223 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fad34102820 0x7fad3419a020 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fad2000ea60 tx=0x7fad2000ef20 comp rx=0 tx=0).stop 2024-03-18T16:10:05.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.358+0000 7fad38c00640 1 -- 172.21.15.39:0/59049223 shutdown_connections 2024-03-18T16:10:05.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.358+0000 7fad38c00640 1 --2- 172.21.15.39:0/59049223 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fad34102820 0x7fad3419a020 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:05.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.358+0000 7fad38c00640 1 --2- 172.21.15.39:0/59049223 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fad180a2880 0x7fad180a4d40 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:05.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.358+0000 7fad38c00640 1 --2- 172.21.15.39:0/59049223 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fad341031e0 0x7fad341a1090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:05.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.358+0000 7fad38c00640 1 --2- 172.21.15.39:0/59049223 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fad34101630 0x7fad34199ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:05.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.358+0000 7fad38c00640 1 -- 172.21.15.39:0/59049223 >> 172.21.15.39:0/59049223 conn(0x7fad340fcbc0 msgr2=0x7fad34073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:05.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.358+0000 7fad38c00640 1 -- 172.21.15.39:0/59049223 shutdown_connections 2024-03-18T16:10:05.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.358+0000 7fad38c00640 1 -- 172.21.15.39:0/59049223 wait complete. 2024-03-18T16:10:05.388 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T16:10:05.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.621+0000 7fc1a1e00640 1 -- 172.21.15.39:0/2236855047 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc19c0761a0 msgr2=0x7fc19c0765c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:05.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.621+0000 7fc1a1e00640 1 --2- 172.21.15.39:0/2236855047 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc19c0761a0 0x7fc19c0765c0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fc190009900 tx=0x7fc19002bc30 comp rx=0 tx=0).stop 2024-03-18T16:10:05.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.621+0000 7fc1a1e00640 1 -- 172.21.15.39:0/2236855047 shutdown_connections 2024-03-18T16:10:05.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.621+0000 7fc1a1e00640 1 --2- 172.21.15.39:0/2236855047 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc19c077d80 0x7fc19c078620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:05.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.621+0000 7fc1a1e00640 1 --2- 172.21.15.39:0/2236855047 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc19c0773e0 0x7fc19c077840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:05.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.621+0000 7fc1a1e00640 1 --2- 172.21.15.39:0/2236855047 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc19c0761a0 0x7fc19c0765c0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:05.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.621+0000 7fc1a1e00640 1 -- 172.21.15.39:0/2236855047 >> 172.21.15.39:0/2236855047 conn(0x7fc19c0fed50 msgr2=0x7fc19c101190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:05.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.622+0000 7fc1a1e00640 1 -- 172.21.15.39:0/2236855047 shutdown_connections 2024-03-18T16:10:05.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.622+0000 7fc1a1e00640 1 -- 172.21.15.39:0/2236855047 wait complete. 2024-03-18T16:10:05.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.623+0000 7fc1a1e00640 1 Processor -- start 2024-03-18T16:10:05.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.623+0000 7fc1a1e00640 1 -- start start 2024-03-18T16:10:05.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.623+0000 7fc1a1e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc19c0761a0 0x7fc19c19deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:05.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.623+0000 7fc1a1e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc19c0773e0 0x7fc19c19e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:05.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.623+0000 7fc1a1e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc19c077d80 0x7fc19c1a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:05.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.623+0000 7fc1a1e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fc19c113830 con 0x7fc19c077d80 2024-03-18T16:10:05.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.623+0000 7fc1a1e00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fc19c1136b0 con 0x7fc19c0761a0 2024-03-18T16:10:05.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.624+0000 7fc1a1e00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fc19c1139b0 con 0x7fc19c0773e0 2024-03-18T16:10:05.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.624+0000 7fc19be00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc19c0773e0 0x7fc19c19e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:05.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.624+0000 7fc1a0a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc19c0761a0 0x7fc19c19deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:05.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.624+0000 7fc19be00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc19c0773e0 0x7fc19c19e3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:49960/0 (socket says 172.21.15.39:49960) 2024-03-18T16:10:05.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.624+0000 7fc19be00640 1 -- 172.21.15.39:0/3055690167 learned_addr learned my addr 172.21.15.39:0/3055690167 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:10:05.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.624+0000 7fc1a1400640 1 --2- 172.21.15.39:0/3055690167 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc19c077d80 0x7fc19c1a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:05.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.624+0000 7fc19be00640 1 -- 172.21.15.39:0/3055690167 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc19c0761a0 msgr2=0x7fc19c19deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:05.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.624+0000 7fc19be00640 1 --2- 172.21.15.39:0/3055690167 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc19c0761a0 0x7fc19c19deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:05.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.624+0000 7fc19be00640 1 -- 172.21.15.39:0/3055690167 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc19c077d80 msgr2=0x7fc19c1a5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:05.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.624+0000 7fc19be00640 1 --2- 172.21.15.39:0/3055690167 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc19c077d80 0x7fc19c1a5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:05.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.624+0000 7fc19be00640 1 -- 172.21.15.39:0/3055690167 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc19c1a5ad0 con 0x7fc19c0773e0 2024-03-18T16:10:05.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.624+0000 7fc1a1400640 1 --2- 172.21.15.39:0/3055690167 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc19c077d80 0x7fc19c1a5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:10:05.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.624+0000 7fc19be00640 1 --2- 172.21.15.39:0/3055690167 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc19c0773e0 0x7fc19c19e3f0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fc18800ea60 tx=0x7fc18800ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:05.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.625+0000 7fc199600640 1 -- 172.21.15.39:0/3055690167 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc18800cde0 con 0x7fc19c0773e0 2024-03-18T16:10:05.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.625+0000 7fc199600640 1 -- 172.21.15.39:0/3055690167 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1880035f0 con 0x7fc19c0773e0 2024-03-18T16:10:05.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.625+0000 7fc199600640 1 -- 172.21.15.39:0/3055690167 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc188010670 con 0x7fc19c0773e0 2024-03-18T16:10:05.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.625+0000 7fc1a1e00640 1 -- 172.21.15.39:0/3055690167 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc19c1a5dc0 con 0x7fc19c0773e0 2024-03-18T16:10:05.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.625+0000 7fc1a1e00640 1 -- 172.21.15.39:0/3055690167 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc19c1a6230 con 0x7fc19c0773e0 2024-03-18T16:10:05.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.626+0000 7fc1a1e00640 1 -- 172.21.15.39:0/3055690167 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc19c110150 con 0x7fc19c0773e0 2024-03-18T16:10:05.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.628+0000 7fc199600640 1 -- 172.21.15.39:0/3055690167 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fc1880037a0 con 0x7fc19c0773e0 2024-03-18T16:10:05.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.630+0000 7fc199600640 1 --2- 172.21.15.39:0/3055690167 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fc1840a2880 0x7fc1840a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:05.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.630+0000 7fc199600640 1 -- 172.21.15.39:0/3055690167 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fc1880cf140 con 0x7fc19c0773e0 2024-03-18T16:10:05.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.630+0000 7fc1a0a00640 1 --2- 172.21.15.39:0/3055690167 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fc1840a2880 0x7fc1840a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:05.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.631+0000 7fc1a0a00640 1 --2- 172.21.15.39:0/3055690167 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fc1840a2880 0x7fc1840a4d40 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fc19002c7b0 tx=0x7fc190005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:05.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.634+0000 7fc199600640 1 -- 172.21.15.39:0/3055690167 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc18809bdf0 con 0x7fc19c0773e0 2024-03-18T16:10:05.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:10:05 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/2991740050' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T16:10:05.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:10:05 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/59049223' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:10:05.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:10:05 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/2991740050' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T16:10:05.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:10:05 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/59049223' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:10:05.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:10:05 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/2991740050' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T16:10:05.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:10:05 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/59049223' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:10:05.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.842+0000 7fc1a1e00640 1 -- 172.21.15.39:0/3055690167 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc19c0624e0 con 0x7fc19c0773e0 2024-03-18T16:10:05.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.844+0000 7fc199600640 1 -- 172.21.15.39:0/3055690167 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24762 (secure 0 0 0) 0x7fc18809b6f0 con 0x7fc19c0773e0 2024-03-18T16:10:05.846 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:10:05.846 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":93,"fsid":"1892d8fc-e540-11ee-95c9-87774f69a715","created":"2024-03-18T15:57:36.507008+0000","modified":"2024-03-18T16:08:47.874967+0000","last_up_change":"2024-03-18T16:05:45.250629+0000","last_in_change":"2024-03-18T16:05:24.841337+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T16:01:30.048247+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T16:06:42.191791+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T16:06:43.670325+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T16:06:47.523388+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"771e0558-be62-4436-94b0-6364751911c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6803","nonce":4090743635}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6805","nonce":4090743635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6809","nonce":4090743635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6807","nonce":4090743635}]},"public_addr":"172.21.15.39:6803/4090743635","cluster_addr":"172.21.15.39:6805/4090743635","heartbeat_back_addr":"172.21.15.39:6809/4090743635","heartbeat_front_addr":"172.21.15.39:6807/4090743635","state":["exists","up"]},{"osd":1,"uuid":"0430eee1-cf34-4277-b255-c7d9f07ad19a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6801","nonce":2800329805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6803","nonce":2800329805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6807","nonce":2800329805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6805","nonce":2800329805}]},"public_addr":"172.21.15.194:6801/2800329805","cluster_addr":"172.21.15.194:6803/2800329805","heartbeat_back_addr":"172.21.15.194:6807/2800329805","heartbeat_front_addr":"172.21.15.194:6805/2800329805","state":["exists","up"]},{"osd":2,"uuid":"3a718da4-2511-4ec1-b1a8-eafd6cce0df0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6801","nonce":3935874719}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6803","nonce":3935874719}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6807","nonce":3935874719}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6805","nonce":3935874719}]},"public_addr":"172.21.15.196:6801/3935874719","cluster_addr":"172.21.15.196:6803/3935874719","heartbeat_back_addr":"172.21.15.196:6807/3935874719","heartbeat_front_addr":"172.21.15.196:6805/3935874719","state":["exists","up"]},{"osd":3,"uuid":"6f9c37f9-0ab9-4038-8919-afd4b39c4165","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6811","nonce":893345939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6813","nonce":893345939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6817","nonce":893345939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6815","nonce":893345939}]},"public_addr":"172.21.15.39:6811/893345939","cluster_addr":"172.21.15.39:6813/893345939","heartbeat_back_addr":"172.21.15.39:6817/893345939","heartbeat_front_addr":"172.21.15.39:6815/893345939","state":["exists","up"]},{"osd":4,"uuid":"e795bb0e-4791-4537-acda-1a321e19bc9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6809","nonce":800423088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6811","nonce":800423088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6815","nonce":800423088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6813","nonce":800423088}]},"public_addr":"172.21.15.194:6809/800423088","cluster_addr":"172.21.15.194:6811/800423088","heartbeat_back_addr":"172.21.15.194:6815/800423088","heartbeat_front_addr":"172.21.15.194:6813/800423088","state":["exists","up"]},{"osd":5,"uuid":"026e175e-6acb-4b4c-b235-8ccee8ae1149","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6809","nonce":2746224525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6811","nonce":2746224525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6815","nonce":2746224525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6813","nonce":2746224525}]},"public_addr":"172.21.15.196:6809/2746224525","cluster_addr":"172.21.15.196:6811/2746224525","heartbeat_back_addr":"172.21.15.196:6815/2746224525","heartbeat_front_addr":"172.21.15.196:6813/2746224525","state":["exists","up"]},{"osd":6,"uuid":"c1b183ff-383f-4c5c-9075-f76bab1c0bb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6819","nonce":1213371331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6821","nonce":1213371331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6825","nonce":1213371331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6823","nonce":1213371331}]},"public_addr":"172.21.15.39:6819/1213371331","cluster_addr":"172.21.15.39:6821/1213371331","heartbeat_back_addr":"172.21.15.39:6825/1213371331","heartbeat_front_addr":"172.21.15.39:6823/1213371331","state":["exists","up"]},{"osd":7,"uuid":"612f8697-957d-4ef4-a87f-241b36f9d13d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6817","nonce":3299162444}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6819","nonce":3299162444}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6823","nonce":3299162444}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6821","nonce":3299162444}]},"public_addr":"172.21.15.194:6817/3299162444","cluster_addr":"172.21.15.194:6819/3299162444","heartbeat_back_addr":"172.21.15.194:6823/3299162444","heartbeat_front_addr":"172.21.15.194:6821/3299162444","state":["exists","up"]},{"osd":8,"uuid":"836a9107-4d4f-4594-b8dc-0301e64f59d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6817","nonce":2456143145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6819","nonce":2456143145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6823","nonce":2456143145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6821","nonce":2456143145}]},"public_addr":"172.21.15.196:6817/2456143145","cluster_addr":"172.21.15.196:6819/2456143145","heartbeat_back_addr":"172.21.15.196:6823/2456143145","heartbeat_front_addr":"172.21.15.196:6821/2456143145","state":["exists","up"]},{"osd":9,"uuid":"4f30b960-12e3-4b16-bd28-fcc988139b58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6827","nonce":1848634433}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6829","nonce":1848634433}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6833","nonce":1848634433}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6831","nonce":1848634433}]},"public_addr":"172.21.15.39:6827/1848634433","cluster_addr":"172.21.15.39:6829/1848634433","heartbeat_back_addr":"172.21.15.39:6833/1848634433","heartbeat_front_addr":"172.21.15.39:6831/1848634433","state":["exists","up"]},{"osd":10,"uuid":"620f90e8-23e6-4bc0-8269-7219b9e1aebd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6825","nonce":3725064809}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6827","nonce":3725064809}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6831","nonce":3725064809}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6829","nonce":3725064809}]},"public_addr":"172.21.15.194:6825/3725064809","cluster_addr":"172.21.15.194:6827/3725064809","heartbeat_back_addr":"172.21.15.194:6831/3725064809","heartbeat_front_addr":"172.21.15.194:6829/3725064809","state":["exists","up"]},{"osd":11,"uuid":"867a2535-8235-4ceb-a698-eb86869d6d20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6825","nonce":2501212422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6827","nonce":2501212422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6831","nonce":2501212422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6829","nonce":2501212422}]},"public_addr":"172.21.15.196:6825/2501212422","cluster_addr":"172.21.15.196:6827/2501212422","heartbeat_back_addr":"172.21.15.196:6831/2501212422","heartbeat_front_addr":"172.21.15.196:6829/2501212422","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:00:36.931566+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:01:02.614024+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:01:26.193265+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:01:58.308833+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:02:25.049546+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:02:49.748750+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:03:23.582153+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:03:50.447321+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:04:15.812129+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:04:49.998047+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:05:17.087924+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:05:43.167637+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":4,"to":7}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.a","mappings":[{"from":5,"to":8}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.39:0/3482086140":"2024-03-19T15:58:45.340294+0000","172.21.15.39:0/3844306284":"2024-03-19T15:58:45.340294+0000","172.21.15.39:6801/1810238546":"2024-03-19T15:58:45.340294+0000","172.21.15.39:6800/1810238546":"2024-03-19T15:58:45.340294+0000","172.21.15.39:0/2528397488":"2024-03-19T15:58:10.264798+0000","172.21.15.39:0/2240981254":"2024-03-19T15:58:10.264798+0000","172.21.15.39:6800/1217503189":"2024-03-19T15:58:10.264798+0000","172.21.15.39:0/3754876370":"2024-03-19T15:58:10.264798+0000","172.21.15.39:6801/1217503189":"2024-03-19T15:58:10.264798+0000","172.21.15.39:0/1874637628":"2024-03-19T15:58:45.340294+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T16:10:05.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.847+0000 7fc1a1e00640 1 -- 172.21.15.39:0/3055690167 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fc1840a2880 msgr2=0x7fc1840a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:05.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.847+0000 7fc1a1e00640 1 --2- 172.21.15.39:0/3055690167 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fc1840a2880 0x7fc1840a4d40 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fc19002c7b0 tx=0x7fc190005fd0 comp rx=0 tx=0).stop 2024-03-18T16:10:05.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.847+0000 7fc1a1e00640 1 -- 172.21.15.39:0/3055690167 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc19c0773e0 msgr2=0x7fc19c19e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:05.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.847+0000 7fc1a1e00640 1 --2- 172.21.15.39:0/3055690167 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc19c0773e0 0x7fc19c19e3f0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fc18800ea60 tx=0x7fc18800ef20 comp rx=0 tx=0).stop 2024-03-18T16:10:05.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.847+0000 7fc1a1e00640 1 -- 172.21.15.39:0/3055690167 shutdown_connections 2024-03-18T16:10:05.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.847+0000 7fc1a1e00640 1 --2- 172.21.15.39:0/3055690167 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fc1840a2880 0x7fc1840a4d40 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:05.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.847+0000 7fc1a1e00640 1 --2- 172.21.15.39:0/3055690167 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc19c077d80 0x7fc19c1a5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:05.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.847+0000 7fc1a1e00640 1 --2- 172.21.15.39:0/3055690167 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc19c0773e0 0x7fc19c19e3f0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:05.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.847+0000 7fc1a1e00640 1 --2- 172.21.15.39:0/3055690167 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc19c0761a0 0x7fc19c19deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:05.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.847+0000 7fc1a1e00640 1 -- 172.21.15.39:0/3055690167 >> 172.21.15.39:0/3055690167 conn(0x7fc19c0fed50 msgr2=0x7fc19c100850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:05.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.847+0000 7fc1a1e00640 1 -- 172.21.15.39:0/3055690167 shutdown_connections 2024-03-18T16:10:05.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:05.847+0000 7fc1a1e00640 1 -- 172.21.15.39:0/3055690167 wait complete. 2024-03-18T16:10:05.868 DEBUG:tasks.fwd_scrub:Yielding 2024-03-18T16:10:05.868 INFO:teuthology.run_tasks:Running task workunit... 2024-03-18T16:10:05.880 INFO:tasks.workunit:Pulling workunits from ref 4e19f704afd9957ea657ab858f9ecf76401fef6f 2024-03-18T16:10:05.881 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-18T16:10:05.881 INFO:tasks.workunit:timeout=3h 2024-03-18T16:10:05.881 INFO:tasks.workunit:cleanup=True 2024-03-18T16:10:05.881 DEBUG:teuthology.orchestra.run.smithi039:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-18T16:10:05.882 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-03-18T16:10:05.882 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-18T16:10:05.882 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T16:10:05.882 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-18T16:10:05.926 INFO:teuthology.orchestra.run.smithi039.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-03-18T16:10:05.927 INFO:teuthology.orchestra.run.smithi039.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-03-18T16:10:05.927 INFO:teuthology.orchestra.run.smithi039.stdout:Device: 76h/118d Inode: 1099511628283 Links: 2 2024-03-18T16:10:05.927 INFO:teuthology.orchestra.run.smithi039.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-18T16:10:05.927 INFO:teuthology.orchestra.run.smithi039.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-03-18T16:10:05.927 INFO:teuthology.orchestra.run.smithi039.stdout:Access: 2024-03-18 16:07:21.549606349 +0000 2024-03-18T16:10:05.927 INFO:teuthology.orchestra.run.smithi039.stdout:Modify: 2024-03-18 16:07:21.549606349 +0000 2024-03-18T16:10:05.927 INFO:teuthology.orchestra.run.smithi039.stdout:Change: 2024-03-18 16:09:53.803395871 +0000 2024-03-18T16:10:05.927 INFO:teuthology.orchestra.run.smithi039.stdout: Birth: 2024-03-18 16:07:21.549606349 +0000 2024-03-18T16:10:05.927 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-03-18T16:10:05.927 DEBUG:teuthology.orchestra.run.smithi039:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-03-18T16:10:06.067 DEBUG:teuthology.orchestra.run.smithi039:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/batrick/ceph /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 4e19f704afd9957ea657ab858f9ecf76401fef6f 2024-03-18T16:10:06.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.088+0000 7f4088400640 1 -- 172.21.15.39:0/1142763695 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4080104b90 msgr2=0x7f4080106f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:06.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.088+0000 7f4088400640 1 --2- 172.21.15.39:0/1142763695 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4080104b90 0x7f4080106f80 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f406c009900 tx=0x7f406c02bc30 comp rx=0 tx=0).stop 2024-03-18T16:10:06.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.088+0000 7f4088400640 1 -- 172.21.15.39:0/1142763695 shutdown_connections 2024-03-18T16:10:06.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.088+0000 7f4088400640 1 --2- 172.21.15.39:0/1142763695 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f40801074c0 0x7f4080109910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:06.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.088+0000 7f4088400640 1 --2- 172.21.15.39:0/1142763695 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4080104b90 0x7f4080106f80 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:06.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.088+0000 7f4088400640 1 --2- 172.21.15.39:0/1142763695 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4080102260 0x7f4080104650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:06.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.088+0000 7f4088400640 1 -- 172.21.15.39:0/1142763695 >> 172.21.15.39:0/1142763695 conn(0x7f40800fbc70 msgr2=0x7f40800fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:06.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.088+0000 7f4088400640 1 -- 172.21.15.39:0/1142763695 shutdown_connections 2024-03-18T16:10:06.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.088+0000 7f4088400640 1 -- 172.21.15.39:0/1142763695 wait complete. 2024-03-18T16:10:06.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.089+0000 7f4088400640 1 Processor -- start 2024-03-18T16:10:06.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.089+0000 7f4088400640 1 -- start start 2024-03-18T16:10:06.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.089+0000 7f4088400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4080102260 0x7f4080101bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:06.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.089+0000 7f4088400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4080104b90 0x7f4080100210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:06.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.089+0000 7f4088400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f40801074c0 0x7f4080100750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:06.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.089+0000 7f4088400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f408010d4a0 con 0x7f4080104b90 2024-03-18T16:10:06.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.089+0000 7f4088400640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f408010d320 con 0x7f40801074c0 2024-03-18T16:10:06.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.089+0000 7f4088400640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f408010d620 con 0x7f4080102260 2024-03-18T16:10:06.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.090+0000 7f4085a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4080102260 0x7f4080101bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:06.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.090+0000 7f4085a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4080102260 0x7f4080101bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:49984/0 (socket says 172.21.15.39:49984) 2024-03-18T16:10:06.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.090+0000 7f4085a00640 1 -- 172.21.15.39:0/1377832252 learned_addr learned my addr 172.21.15.39:0/1377832252 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:10:06.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.090+0000 7f4086400640 1 --2- 172.21.15.39:0/1377832252 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f40801074c0 0x7f4080100750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:06.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.090+0000 7f4085000640 1 --2- 172.21.15.39:0/1377832252 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4080104b90 0x7f4080100210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:06.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.090+0000 7f4085a00640 1 -- 172.21.15.39:0/1377832252 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f40801074c0 msgr2=0x7f4080100750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:06.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.090+0000 7f4085a00640 1 --2- 172.21.15.39:0/1377832252 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f40801074c0 0x7f4080100750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:06.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.090+0000 7f4085a00640 1 -- 172.21.15.39:0/1377832252 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4080104b90 msgr2=0x7f4080100210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:06.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.090+0000 7f4085a00640 1 --2- 172.21.15.39:0/1377832252 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4080104b90 0x7f4080100210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:06.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.090+0000 7f4085a00640 1 -- 172.21.15.39:0/1377832252 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4080100fe0 con 0x7f4080102260 2024-03-18T16:10:06.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.090+0000 7f4086400640 1 --2- 172.21.15.39:0/1377832252 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f40801074c0 0x7f4080100750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:10:06.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.091+0000 7f4085a00640 1 --2- 172.21.15.39:0/1377832252 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4080102260 0x7f4080101bd0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f407400e9f0 tx=0x7f407400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:06.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.091+0000 7f407e000640 1 -- 172.21.15.39:0/1377832252 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f407400cd80 con 0x7f4080102260 2024-03-18T16:10:06.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.091+0000 7f407e000640 1 -- 172.21.15.39:0/1377832252 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4074010910 con 0x7f4080102260 2024-03-18T16:10:06.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.091+0000 7f407e000640 1 -- 172.21.15.39:0/1377832252 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4074018990 con 0x7f4080102260 2024-03-18T16:10:06.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.091+0000 7f4088400640 1 -- 172.21.15.39:0/1377832252 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40801a1c60 con 0x7f4080102260 2024-03-18T16:10:06.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.091+0000 7f4088400640 1 -- 172.21.15.39:0/1377832252 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40801a2130 con 0x7f4080102260 2024-03-18T16:10:06.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.094+0000 7f407e000640 1 -- 172.21.15.39:0/1377832252 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f4074010ac0 con 0x7f4080102260 2024-03-18T16:10:06.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.095+0000 7f408b17e740 1 Processor -- start 2024-03-18T16:10:06.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.095+0000 7f408b17e740 1 -- start start 2024-03-18T16:10:06.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.095+0000 7f408b17e740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5630034017e0 0x563003401c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:06.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.095+0000 7f408b17e740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5630034021d0 0x563003406cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:06.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.095+0000 7f408b17e740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563003407340 0x5630034097d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:06.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.095+0000 7f408b17e740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x563003364f80 con 0x5630034021d0 2024-03-18T16:10:06.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.095+0000 7f408b17e740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x56300330dad0 con 0x5630034017e0 2024-03-18T16:10:06.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.095+0000 7f408b17e740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x563003361020 con 0x563003407340 2024-03-18T16:10:06.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.095+0000 7f4085000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563003407340 0x5630034097d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:06.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.095+0000 7f4086400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5630034017e0 0x563003401c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:06.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.095+0000 7f4085000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563003407340 0x5630034097d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:49990/0 (socket says 172.21.15.39:49990) 2024-03-18T16:10:06.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.095+0000 7f4085000640 1 -- 172.21.15.39:0/1178084674 learned_addr learned my addr 172.21.15.39:0/1178084674 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:10:06.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.096+0000 7f4085a00640 1 --2- 172.21.15.39:0/1178084674 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5630034021d0 0x563003406cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:06.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.096+0000 7f4085000640 1 -- 172.21.15.39:0/1178084674 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5630034017e0 msgr2=0x563003401c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:06.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.096+0000 7f4085000640 1 --2- 172.21.15.39:0/1178084674 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5630034017e0 0x563003401c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:06.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.096+0000 7f4085000640 1 -- 172.21.15.39:0/1178084674 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5630034021d0 msgr2=0x563003406cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:06.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.096+0000 7f4085000640 1 --2- 172.21.15.39:0/1178084674 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5630034021d0 0x563003406cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:06.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.096+0000 7f4085000640 1 -- 172.21.15.39:0/1178084674 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5630033256f0 con 0x563003407340 2024-03-18T16:10:06.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.096+0000 7f407e000640 1 --2- 172.21.15.39:0/1377832252 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f40680a2880 0x7f40680a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:06.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.096+0000 7f4085000640 1 --2- 172.21.15.39:0/1178084674 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563003407340 0x5630034097d0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f406c000c00 tx=0x7f406c009560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fbc41b9820d2d7f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:06.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.097+0000 7f407e000640 1 -- 172.21.15.39:0/1377832252 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f40740cf110 con 0x7f4080102260 2024-03-18T16:10:06.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.097+0000 7f4086400640 1 --2- 172.21.15.39:0/1377832252 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f40680a2880 0x7f40680a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:06.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.097+0000 7f4072a00640 1 -- 172.21.15.39:0/1178084674 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f406c02c980 con 0x563003407340 2024-03-18T16:10:06.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.097+0000 7f4072a00640 1 -- 172.21.15.39:0/1178084674 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f406c02cb20 con 0x563003407340 2024-03-18T16:10:06.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.097+0000 7f4072a00640 1 -- 172.21.15.39:0/1178084674 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f406c0046c0 con 0x563003407340 2024-03-18T16:10:06.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.098+0000 7f408b17e740 1 -- 172.21.15.39:0/1178084674 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563003407340 msgr2=0x5630034097d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:06.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.098+0000 7f4086400640 1 --2- 172.21.15.39:0/1377832252 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f40680a2880 0x7f40680a4d40 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x563003402a30 tx=0x7f4078009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:06.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.098+0000 7f408b17e740 1 --2- 172.21.15.39:0/1178084674 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563003407340 0x5630034097d0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f406c000c00 tx=0x7f406c009560 comp rx=0 tx=0).stop 2024-03-18T16:10:06.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.098+0000 7f408b17e740 1 -- 172.21.15.39:0/1178084674 shutdown_connections 2024-03-18T16:10:06.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.098+0000 7f408b17e740 1 --2- 172.21.15.39:0/1178084674 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563003407340 0x5630034097d0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:06.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.098+0000 7f408b17e740 1 --2- 172.21.15.39:0/1178084674 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5630034021d0 0x563003406cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:06.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.098+0000 7f408b17e740 1 --2- 172.21.15.39:0/1178084674 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5630034017e0 0x563003401c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:06.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.098+0000 7f408b17e740 1 -- 172.21.15.39:0/1178084674 >> 172.21.15.39:0/1178084674 conn(0x5630033c9930 msgr2=0x5630033fd150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:06.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.098+0000 7f408b17e740 1 -- 172.21.15.39:0/1178084674 shutdown_connections 2024-03-18T16:10:06.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.098+0000 7f408b17e740 1 -- 172.21.15.39:0/1178084674 wait complete. 2024-03-18T16:10:06.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.098+0000 7f408b17e740 1 -- start start 2024-03-18T16:10:06.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.099+0000 7f408b17e740 1 Processor -- start 2024-03-18T16:10:06.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.099+0000 7f408b17e740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5630034017e0 0x56300340bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:06.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.099+0000 7f408b17e740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5630034021d0 0x56300340bff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:06.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.099+0000 7f408b17e740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563003407340 0x563003413030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:06.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.099+0000 7f408b17e740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x56300340a290 con 0x5630034017e0 2024-03-18T16:10:06.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.099+0000 7f408b17e740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x563003320f30 con 0x563003407340 2024-03-18T16:10:06.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.099+0000 7f408b17e740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x56300340a410 con 0x5630034021d0 2024-03-18T16:10:06.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.099+0000 7f408b17e740 10 client.0 _ll_register_callbacks cb 0x56300330ebc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:10:06.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.099+0000 7f4086400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563003407340 0x563003413030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:06.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.099+0000 7f4085a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5630034017e0 0x56300340bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:06.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.099+0000 7f4086400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563003407340 0x563003413030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:36646/0 (socket says 172.21.15.39:36646) 2024-03-18T16:10:06.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.099+0000 7f4086400640 1 -- 172.21.15.39:0/867550575 learned_addr learned my addr 172.21.15.39:0/867550575 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:10:06.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.099+0000 7f4085000640 1 --2- 172.21.15.39:0/867550575 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5630034021d0 0x56300340bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:06.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.100+0000 7f4086400640 1 -- 172.21.15.39:0/867550575 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5630034021d0 msgr2=0x56300340bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:06.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.100+0000 7f4086400640 1 --2- 172.21.15.39:0/867550575 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5630034021d0 0x56300340bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:06.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.100+0000 7f4086400640 1 -- 172.21.15.39:0/867550575 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5630034017e0 msgr2=0x56300340bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:06.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.100+0000 7f4086400640 1 --2- 172.21.15.39:0/867550575 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5630034017e0 0x56300340bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:06.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.100+0000 7f4086400640 1 -- 172.21.15.39:0/867550575 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5630031417e0 con 0x563003407340 2024-03-18T16:10:06.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.100+0000 7f4086400640 1 --2- 172.21.15.39:0/867550575 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563003407340 0x563003413030 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f40780127c0 tx=0x7f4078004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=449c2dd5409a8b8b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:06.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.100+0000 7f4067400640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:10:06.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.100+0000 7f4067400640 1 -- 172.21.15.39:0/867550575 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4078011c60 con 0x563003407340 2024-03-18T16:10:06.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.101+0000 7f408b17e740 1 -- 172.21.15.39:0/867550575 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f406c02cb20 con 0x563003407340 2024-03-18T16:10:06.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.101+0000 7f4067400640 1 -- 172.21.15.39:0/867550575 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f4078011e00 con 0x563003407340 2024-03-18T16:10:06.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.101+0000 7f4067400640 1 -- 172.21.15.39:0/867550575 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f407802e520 con 0x563003407340 2024-03-18T16:10:06.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.101+0000 7f408b17e740 10 client.25580 fetch_fsmap learned FSMap version 28 2024-03-18T16:10:06.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.101+0000 7f408b17e740 1 -- 172.21.15.39:0/867550575 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5630031a1f70 con 0x563003407340 2024-03-18T16:10:06.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.101+0000 7f4067400640 1 -- 172.21.15.39:0/867550575 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f406c02cb20 con 0x563003407340 2024-03-18T16:10:06.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.101+0000 7f4067400640 1 -- 172.21.15.39:0/867550575 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f4078027070 con 0x563003407340 2024-03-18T16:10:06.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.101+0000 7f408b17e740 10 client.25580 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T16:10:06.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.101+0000 7f408b17e740 10 client.25580 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:10:06.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.101+0000 7f408b17e740 20 client.25580 populate_metadata read hostname 'smithi039' 2024-03-18T16:10:06.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.102+0000 7f408b17e740 1 --2- 172.21.15.39:0/867550575 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x563003420140 0x5630034225d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:06.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.102+0000 7f408b17e740 4 client.25580 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:10:06.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.102+0000 7f408b17e740 1 -- 172.21.15.39:0/867550575 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5630033724c0 con 0x563003420140 2024-03-18T16:10:06.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.102+0000 7f4085a00640 1 --2- 172.21.15.39:0/867550575 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x563003420140 0x5630034225d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:06.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.102+0000 7f4085a00640 1 --2- 172.21.15.39:0/867550575 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x563003420140 0x5630034225d0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f8c32cfcc8744970 server_cookie=7dd23a81f2b738e2 in_seq=0 out_seq=0 2024-03-18T16:10:06.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.102+0000 7f4067400640 10 client.25580 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:10:06.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.103+0000 7f4067400640 1 -- 172.21.15.39:0/867550575 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5630033724c0 con 0x563003420140 2024-03-18T16:10:06.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.103+0000 7f4067400640 10 client.25580 handle_command_reply: tid=0 2024-03-18T16:10:06.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.103+0000 7f408b17e740 1 client.25580 shutdown 2024-03-18T16:10:06.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.103+0000 7f408b17e740 1 -- 172.21.15.39:0/867550575 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563003407340 msgr2=0x563003413030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:06.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.103+0000 7f408b17e740 1 --2- 172.21.15.39:0/867550575 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563003407340 0x563003413030 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f40780127c0 tx=0x7f4078004ab0 comp rx=0 tx=0).stop 2024-03-18T16:10:06.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.103+0000 7f408b17e740 1 -- 172.21.15.39:0/867550575 shutdown_connections 2024-03-18T16:10:06.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.103+0000 7f408b17e740 1 --2- 172.21.15.39:0/867550575 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x563003420140 0x5630034225d0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:06.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.103+0000 7f408b17e740 1 --2- 172.21.15.39:0/867550575 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563003407340 0x563003413030 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:06.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.103+0000 7f408b17e740 1 --2- 172.21.15.39:0/867550575 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5630034021d0 0x56300340bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:06.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.104+0000 7f408b17e740 1 --2- 172.21.15.39:0/867550575 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5630034017e0 0x56300340bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:06.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.104+0000 7f408b17e740 1 -- 172.21.15.39:0/867550575 >> 172.21.15.39:0/867550575 conn(0x5630033c9930 msgr2=0x5630033fc350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:06.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.104+0000 7f4067400640 0 client.25580 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:10:06.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.104+0000 7f408b17e740 1 -- 172.21.15.39:0/867550575 shutdown_connections 2024-03-18T16:10:06.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.104+0000 7f408b17e740 1 -- 172.21.15.39:0/867550575 wait complete. 2024-03-18T16:10:06.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.104+0000 7f408b17e740 20 client.25580 trim_cache size 0 max 16384 2024-03-18T16:10:06.115 INFO:tasks.workunit.client.0.smithi039.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-18T16:10:06.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.123+0000 7f408b17e740 1 Processor -- start 2024-03-18T16:10:06.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.123+0000 7f408b17e740 1 -- start start 2024-03-18T16:10:06.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.124+0000 7f408b17e740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5630034017e0 0x563003427ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:06.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.124+0000 7f408b17e740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5630034021d0 0x563003412ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:06.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.124+0000 7f408b17e740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563003407340 0x563003400390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:06.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.124+0000 7f408b17e740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f40801ac630 con 0x563003407340 2024-03-18T16:10:06.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.124+0000 7f408b17e740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f40801ae170 con 0x5630034021d0 2024-03-18T16:10:06.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.124+0000 7f408b17e740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f40801acd00 con 0x5630034017e0 2024-03-18T16:10:06.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.124+0000 7f4085a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5630034017e0 0x563003427ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:06.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.124+0000 7f4085a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5630034017e0 0x563003427ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:50012/0 (socket says 172.21.15.39:50012) 2024-03-18T16:10:06.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.124+0000 7f4085000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5630034021d0 0x563003412ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:06.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.124+0000 7f4085a00640 1 -- 172.21.15.39:0/2121189898 learned_addr learned my addr 172.21.15.39:0/2121189898 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:10:06.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.124+0000 7f4086400640 1 --2- 172.21.15.39:0/2121189898 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563003407340 0x563003400390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:06.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.124+0000 7f4085a00640 1 -- 172.21.15.39:0/2121189898 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5630034021d0 msgr2=0x563003412ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:06.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.124+0000 7f4085a00640 1 --2- 172.21.15.39:0/2121189898 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5630034021d0 0x563003412ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:06.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.124+0000 7f4085a00640 1 -- 172.21.15.39:0/2121189898 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563003407340 msgr2=0x563003400390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:06.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.124+0000 7f4085a00640 1 --2- 172.21.15.39:0/2121189898 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563003407340 0x563003400390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:06.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.124+0000 7f4085a00640 1 -- 172.21.15.39:0/2121189898 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5630033f8980 con 0x5630034017e0 2024-03-18T16:10:06.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.124+0000 7f4085000640 1 --2- 172.21.15.39:0/2121189898 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5630034021d0 0x563003412ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:10:06.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.125+0000 7f4085a00640 1 --2- 172.21.15.39:0/2121189898 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5630034017e0 0x563003427ee0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f4074097bd0 tx=0x7f4074000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=97564f1f69726462 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:06.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.125+0000 7f4072000640 1 -- 172.21.15.39:0/2121189898 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f407409f810 con 0x5630034017e0 2024-03-18T16:10:06.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.125+0000 7f4072000640 1 -- 172.21.15.39:0/2121189898 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f40740a3e60 con 0x5630034017e0 2024-03-18T16:10:06.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.125+0000 7f4072000640 1 -- 172.21.15.39:0/2121189898 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40740d10b0 con 0x5630034017e0 2024-03-18T16:10:06.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.126+0000 7f408b17e740 1 -- 172.21.15.39:0/2121189898 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5630034017e0 msgr2=0x563003427ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:06.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.126+0000 7f408b17e740 1 --2- 172.21.15.39:0/2121189898 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5630034017e0 0x563003427ee0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f4074097bd0 tx=0x7f4074000bc0 comp rx=0 tx=0).stop 2024-03-18T16:10:06.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.126+0000 7f408b17e740 1 -- 172.21.15.39:0/2121189898 shutdown_connections 2024-03-18T16:10:06.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.126+0000 7f408b17e740 1 --2- 172.21.15.39:0/2121189898 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5630034021d0 0x563003412ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:06.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.126+0000 7f408b17e740 1 --2- 172.21.15.39:0/2121189898 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563003407340 0x563003400390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:06.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.126+0000 7f408b17e740 1 --2- 172.21.15.39:0/2121189898 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5630034017e0 0x563003427ee0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:06.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.126+0000 7f408b17e740 1 -- 172.21.15.39:0/2121189898 >> 172.21.15.39:0/2121189898 conn(0x5630033c9930 msgr2=0x5630033fccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:06.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.126+0000 7f408b17e740 1 -- 172.21.15.39:0/2121189898 shutdown_connections 2024-03-18T16:10:06.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.126+0000 7f408b17e740 1 -- 172.21.15.39:0/2121189898 wait complete. 2024-03-18T16:10:06.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.126+0000 7f408b17e740 1 -- start start 2024-03-18T16:10:06.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.127+0000 7f408b17e740 1 Processor -- start 2024-03-18T16:10:06.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.127+0000 7f408b17e740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5630034017e0 0x563003410bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:06.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.127+0000 7f408b17e740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5630034021d0 0x563003411100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:06.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.127+0000 7f408b17e740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563003407340 0x5630034120c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:06.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.127+0000 7f408b17e740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f40801adaa0 con 0x5630034021d0 2024-03-18T16:10:06.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.127+0000 7f408b17e740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f40801ad3d0 con 0x563003407340 2024-03-18T16:10:06.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.127+0000 7f408b17e740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f40801aa610 con 0x5630034017e0 2024-03-18T16:10:06.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.127+0000 7f408b17e740 10 client.0 _ll_register_callbacks cb 0x7f4078012710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:10:06.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.127+0000 7f4085a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5630034017e0 0x563003410bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:06.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.127+0000 7f4085a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5630034017e0 0x563003410bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:50028/0 (socket says 172.21.15.39:50028) 2024-03-18T16:10:06.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.127+0000 7f4085a00640 1 -- 172.21.15.39:0/3825758733 learned_addr learned my addr 172.21.15.39:0/3825758733 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:10:06.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.127+0000 7f4085000640 1 --2- 172.21.15.39:0/3825758733 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5630034021d0 0x563003411100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:06.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.127+0000 7f4086400640 1 --2- 172.21.15.39:0/3825758733 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563003407340 0x5630034120c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:06.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.128+0000 7f4085a00640 1 -- 172.21.15.39:0/3825758733 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563003407340 msgr2=0x5630034120c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:06.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.128+0000 7f4085a00640 1 --2- 172.21.15.39:0/3825758733 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563003407340 0x5630034120c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:06.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.128+0000 7f4085a00640 1 -- 172.21.15.39:0/3825758733 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5630034021d0 msgr2=0x563003411100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:06.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.128+0000 7f4085a00640 1 --2- 172.21.15.39:0/3825758733 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5630034021d0 0x563003411100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:06.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.128+0000 7f4085a00640 1 -- 172.21.15.39:0/3825758733 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563003039170 con 0x5630034017e0 2024-03-18T16:10:06.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.128+0000 7f4085a00640 1 --2- 172.21.15.39:0/3825758733 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5630034017e0 0x563003410bc0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f407409b700 tx=0x7f40740cc7c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4ac69d5cbff12589 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:06.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.128+0000 7f4067400640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:10:06.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.128+0000 7f4067400640 1 -- 172.21.15.39:0/3825758733 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4074099a70 con 0x5630034017e0 2024-03-18T16:10:06.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.129+0000 7f4067400640 1 -- 172.21.15.39:0/3825758733 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f40740cc930 con 0x5630034017e0 2024-03-18T16:10:06.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.129+0000 7f408b17e740 1 -- 172.21.15.39:0/3825758733 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f40740a3e60 con 0x5630034017e0 2024-03-18T16:10:06.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.129+0000 7f4067400640 1 -- 172.21.15.39:0/3825758733 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f40740d10e0 con 0x5630034017e0 2024-03-18T16:10:06.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.129+0000 7f408b17e740 10 client.25609 fetch_fsmap learned FSMap version 28 2024-03-18T16:10:06.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.129+0000 7f408b17e740 1 -- 172.21.15.39:0/3825758733 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x563003486800 con 0x5630034017e0 2024-03-18T16:10:06.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.130+0000 7f4067400640 1 -- 172.21.15.39:0/3825758733 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f40740a3e60 con 0x5630034017e0 2024-03-18T16:10:06.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.130+0000 7f4067400640 1 -- 172.21.15.39:0/3825758733 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f40740ccce0 con 0x5630034017e0 2024-03-18T16:10:06.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.130+0000 7f408b17e740 10 client.25609 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T16:10:06.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.130+0000 7f408b17e740 10 client.25609 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:10:06.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.130+0000 7f408b17e740 20 client.25609 populate_metadata read hostname 'smithi039' 2024-03-18T16:10:06.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.130+0000 7f408b17e740 1 --2- 172.21.15.39:0/3825758733 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5630034870f0 0x563003489510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:06.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.130+0000 7f408b17e740 4 client.25609 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-18T16:10:06.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.130+0000 7f408b17e740 1 -- 172.21.15.39:0/3825758733 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x56300336c810 con 0x5630034870f0 2024-03-18T16:10:06.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.130+0000 7f4085000640 1 --2- 172.21.15.39:0/3825758733 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5630034870f0 0x563003489510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:06.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.131+0000 7f4085000640 1 --2- 172.21.15.39:0/3825758733 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5630034870f0 0x563003489510 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ede3ae534608a324 server_cookie=4a4e58a99205b050 in_seq=0 out_seq=0 2024-03-18T16:10:06.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.131+0000 7f4067400640 10 client.25609 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:10:06.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.132+0000 7f4067400640 1 -- 172.21.15.39:0/3825758733 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x56300336c810 con 0x5630034870f0 2024-03-18T16:10:06.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.132+0000 7f4067400640 10 client.25609 handle_command_reply: tid=0 2024-03-18T16:10:06.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.132+0000 7f408b17e740 1 client.25609 shutdown 2024-03-18T16:10:06.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.132+0000 7f408b17e740 1 -- 172.21.15.39:0/3825758733 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5630034017e0 msgr2=0x563003410bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:06.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.132+0000 7f408b17e740 1 --2- 172.21.15.39:0/3825758733 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5630034017e0 0x563003410bc0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f407409b700 tx=0x7f40740cc7c0 comp rx=0 tx=0).stop 2024-03-18T16:10:06.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.132+0000 7f408b17e740 1 -- 172.21.15.39:0/3825758733 shutdown_connections 2024-03-18T16:10:06.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.132+0000 7f408b17e740 1 --2- 172.21.15.39:0/3825758733 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5630034870f0 0x563003489510 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:06.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.132+0000 7f408b17e740 1 --2- 172.21.15.39:0/3825758733 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563003407340 0x5630034120c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:06.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.132+0000 7f408b17e740 1 --2- 172.21.15.39:0/3825758733 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5630034021d0 0x563003411100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:06.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.132+0000 7f408b17e740 1 --2- 172.21.15.39:0/3825758733 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5630034017e0 0x563003410bc0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:06.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.132+0000 7f4067400640 0 client.25609 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:10:06.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.132+0000 7f408b17e740 1 -- 172.21.15.39:0/3825758733 >> 172.21.15.39:0/3825758733 conn(0x5630033c9930 msgr2=0x563003400390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:06.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.133+0000 7f408b17e740 1 -- 172.21.15.39:0/3825758733 shutdown_connections 2024-03-18T16:10:06.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.133+0000 7f408b17e740 1 -- 172.21.15.39:0/3825758733 wait complete. 2024-03-18T16:10:06.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.133+0000 7f408b17e740 20 client.25609 trim_cache size 0 max 16384 2024-03-18T16:10:06.137 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-03-18T16:10:06.138 INFO:teuthology.orchestra.run.smithi039.stdout: "return_code": 0, 2024-03-18T16:10:06.138 INFO:teuthology.orchestra.run.smithi039.stdout: "scrub_tag": "8ed4bef1-ef24-4a17-88db-4b54e8c8530c", 2024-03-18T16:10:06.138 INFO:teuthology.orchestra.run.smithi039.stdout: "mode": "asynchronous" 2024-03-18T16:10:06.138 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-03-18T16:10:06.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.133+0000 7f4088400640 1 -- 172.21.15.39:0/1377832252 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f40680a2880 msgr2=0x7f40680a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:06.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.133+0000 7f4088400640 1 --2- 172.21.15.39:0/1377832252 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f40680a2880 0x7f40680a4d40 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x563003402a30 tx=0x7f4078009500 comp rx=0 tx=0).stop 2024-03-18T16:10:06.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.133+0000 7f4088400640 1 -- 172.21.15.39:0/1377832252 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4080102260 msgr2=0x7f4080101bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:06.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.133+0000 7f4088400640 1 --2- 172.21.15.39:0/1377832252 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4080102260 0x7f4080101bd0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f407400e9f0 tx=0x7f407400eeb0 comp rx=0 tx=0).stop 2024-03-18T16:10:06.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.134+0000 7f4088400640 1 -- 172.21.15.39:0/1377832252 shutdown_connections 2024-03-18T16:10:06.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.134+0000 7f4088400640 1 --2- 172.21.15.39:0/1377832252 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f40801074c0 0x7f4080100750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:06.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.134+0000 7f4088400640 1 --2- 172.21.15.39:0/1377832252 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f40680a2880 0x7f40680a4d40 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:06.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.134+0000 7f4088400640 1 --2- 172.21.15.39:0/1377832252 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4080104b90 0x7f4080100210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:06.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.134+0000 7f4088400640 1 --2- 172.21.15.39:0/1377832252 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4080102260 0x7f4080101bd0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:06.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.134+0000 7f4088400640 1 -- 172.21.15.39:0/1377832252 >> 172.21.15.39:0/1377832252 conn(0x7f40800fbc70 msgr2=0x7f4080102ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:06.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.134+0000 7f4088400640 1 -- 172.21.15.39:0/1377832252 shutdown_connections 2024-03-18T16:10:06.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.134+0000 7f4088400640 1 -- 172.21.15.39:0/1377832252 wait complete. 2024-03-18T16:10:06.154 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T16:10:06.154 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T16:10:06.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.413+0000 7ff89b200640 1 -- 172.21.15.39:0/1551999536 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff89c105490 msgr2=0x7ff89c10bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:06.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.413+0000 7ff89b200640 1 --2- 172.21.15.39:0/1551999536 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff89c105490 0x7ff89c10bd20 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7ff894009be0 tx=0x7ff89402bef0 comp rx=0 tx=0).stop 2024-03-18T16:10:06.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.413+0000 7ff89b200640 1 -- 172.21.15.39:0/1551999536 shutdown_connections 2024-03-18T16:10:06.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.413+0000 7ff89b200640 1 --2- 172.21.15.39:0/1551999536 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff89c105490 0x7ff89c10bd20 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:06.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.413+0000 7ff89b200640 1 --2- 172.21.15.39:0/1551999536 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff89c104ad0 0x7ff89c104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:06.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.413+0000 7ff89b200640 1 --2- 172.21.15.39:0/1551999536 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff89c1038e0 0x7ff89c103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:06.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.413+0000 7ff89b200640 1 -- 172.21.15.39:0/1551999536 >> 172.21.15.39:0/1551999536 conn(0x7ff89c0fee50 msgr2=0x7ff89c101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:06.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.414+0000 7ff89b200640 1 -- 172.21.15.39:0/1551999536 shutdown_connections 2024-03-18T16:10:06.416 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.414+0000 7ff89b200640 1 -- 172.21.15.39:0/1551999536 wait complete. 2024-03-18T16:10:06.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.415+0000 7ff89b200640 1 Processor -- start 2024-03-18T16:10:06.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.415+0000 7ff89b200640 1 -- start start 2024-03-18T16:10:06.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.415+0000 7ff89b200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff89c1038e0 0x7ff89c199b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:06.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.415+0000 7ff89b200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff89c104ad0 0x7ff89c19a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:06.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.415+0000 7ff89b200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff89c105490 0x7ff89c1a10f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:06.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.415+0000 7ff89b200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7ff89c10f590 con 0x7ff89c104ad0 2024-03-18T16:10:06.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.415+0000 7ff89b200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ff89c10f410 con 0x7ff89c105490 2024-03-18T16:10:06.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.415+0000 7ff89b200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7ff89c10f710 con 0x7ff89c1038e0 2024-03-18T16:10:06.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.415+0000 7ff899400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff89c104ad0 0x7ff89c19a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:06.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.415+0000 7ff89a800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff89c105490 0x7ff89c1a10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:06.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.415+0000 7ff899e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff89c1038e0 0x7ff89c199b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:06.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.415+0000 7ff89a800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff89c105490 0x7ff89c1a10f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:36690/0 (socket says 172.21.15.39:36690) 2024-03-18T16:10:06.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.415+0000 7ff899e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff89c1038e0 0x7ff89c199b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:50048/0 (socket says 172.21.15.39:50048) 2024-03-18T16:10:06.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.415+0000 7ff899e00640 1 -- 172.21.15.39:0/2483746991 learned_addr learned my addr 172.21.15.39:0/2483746991 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:10:06.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.415+0000 7ff899400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff89c104ad0 0x7ff89c19a080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:55898/0 (socket says 172.21.15.39:55898) 2024-03-18T16:10:06.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.416+0000 7ff89a800640 1 -- 172.21.15.39:0/2483746991 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff89c1038e0 msgr2=0x7ff89c199b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:06.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.416+0000 7ff89a800640 1 --2- 172.21.15.39:0/2483746991 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff89c1038e0 0x7ff89c199b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:06.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.416+0000 7ff89a800640 1 -- 172.21.15.39:0/2483746991 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff89c104ad0 msgr2=0x7ff89c19a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:06.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.416+0000 7ff89a800640 1 --2- 172.21.15.39:0/2483746991 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff89c104ad0 0x7ff89c19a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:06.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.416+0000 7ff89a800640 1 -- 172.21.15.39:0/2483746991 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff89c1a1760 con 0x7ff89c105490 2024-03-18T16:10:06.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.416+0000 7ff899400640 1 --2- 172.21.15.39:0/2483746991 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff89c104ad0 0x7ff89c19a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:10:06.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.416+0000 7ff89a800640 1 --2- 172.21.15.39:0/2483746991 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff89c105490 0x7ff89c1a10f0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7ff89402c430 tx=0x7ff8940337b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:06.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.417+0000 7ff892a00640 1 -- 172.21.15.39:0/2483746991 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff894043020 con 0x7ff89c105490 2024-03-18T16:10:06.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.417+0000 7ff892a00640 1 -- 172.21.15.39:0/2483746991 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff89402ca60 con 0x7ff89c105490 2024-03-18T16:10:06.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.417+0000 7ff892a00640 1 -- 172.21.15.39:0/2483746991 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff894034690 con 0x7ff89c105490 2024-03-18T16:10:06.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.417+0000 7ff89b200640 1 -- 172.21.15.39:0/2483746991 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff89c1a19f0 con 0x7ff89c105490 2024-03-18T16:10:06.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.417+0000 7ff89b200640 1 -- 172.21.15.39:0/2483746991 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff89c1a1f90 con 0x7ff89c105490 2024-03-18T16:10:06.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.420+0000 7ff892a00640 1 -- 172.21.15.39:0/2483746991 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7ff894046020 con 0x7ff89c105490 2024-03-18T16:10:06.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.420+0000 7ff8a3e24740 1 Processor -- start 2024-03-18T16:10:06.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.421+0000 7ff8a3e24740 1 -- start start 2024-03-18T16:10:06.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.421+0000 7ff8a3e24740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c6cbefaa10 0x55c6cbefae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:06.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.421+0000 7ff8a3e24740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c6cbefb400 0x55c6cbefff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:06.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.421+0000 7ff8a3e24740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c6cbf00570 0x55c6cbf02a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:06.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.421+0000 7ff8a3e24740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55c6cbe5f1f0 con 0x55c6cbefaa10 2024-03-18T16:10:06.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.421+0000 7ff8a3e24740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55c6cbe07a80 con 0x55c6cbefb400 2024-03-18T16:10:06.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.421+0000 7ff8a3e24740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55c6cbe5b290 con 0x55c6cbf00570 2024-03-18T16:10:06.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.421+0000 7ff899400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c6cbefaa10 0x55c6cbefae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:06.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.421+0000 7ff899e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c6cbefb400 0x55c6cbefff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:06.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.421+0000 7ff899e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c6cbefb400 0x55c6cbefff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:36702/0 (socket says 172.21.15.39:36702) 2024-03-18T16:10:06.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.421+0000 7ff899400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c6cbefaa10 0x55c6cbefae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:55902/0 (socket says 172.21.15.39:55902) 2024-03-18T16:10:06.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.421+0000 7ff899e00640 1 -- 172.21.15.39:0/200947663 learned_addr learned my addr 172.21.15.39:0/200947663 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:10:06.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.421+0000 7ff899400640 1 --2- 172.21.15.39:0/200947663 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c6cbf00570 0x55c6cbf02a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:06.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.422+0000 7ff899e00640 1 -- 172.21.15.39:0/200947663 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c6cbf00570 msgr2=0x55c6cbf02a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:06.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.422+0000 7ff899e00640 1 --2- 172.21.15.39:0/200947663 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c6cbf00570 0x55c6cbf02a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:06.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.422+0000 7ff899e00640 1 -- 172.21.15.39:0/200947663 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c6cbefaa10 msgr2=0x55c6cbefae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:06.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.422+0000 7ff899e00640 1 --2- 172.21.15.39:0/200947663 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c6cbefaa10 0x55c6cbefae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:06.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.422+0000 7ff899e00640 1 -- 172.21.15.39:0/200947663 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c6cbc3d7d0 con 0x55c6cbefb400 2024-03-18T16:10:06.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.422+0000 7ff892a00640 1 --2- 172.21.15.39:0/2483746991 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff8800a2880 0x7ff8800a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:06.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.422+0000 7ff899e00640 1 --2- 172.21.15.39:0/200947663 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c6cbefb400 0x55c6cbefff00 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ff88c009db0 tx=0x7ff88c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a99ef9c861c888a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:06.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.422+0000 7ff892a00640 1 -- 172.21.15.39:0/2483746991 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7ff8940f1300 con 0x7ff89c105490 2024-03-18T16:10:06.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.422+0000 7ff899400640 1 --2- 172.21.15.39:0/2483746991 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff8800a2880 0x7ff8800a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:06.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.422+0000 7ff887400640 1 -- 172.21.15.39:0/200947663 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff88c004220 con 0x55c6cbefb400 2024-03-18T16:10:06.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.423+0000 7ff887400640 1 -- 172.21.15.39:0/200947663 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff88c0043c0 con 0x55c6cbefb400 2024-03-18T16:10:06.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.423+0000 7ff887400640 1 -- 172.21.15.39:0/200947663 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff88c019900 con 0x55c6cbefb400 2024-03-18T16:10:06.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.423+0000 7ff8a3e24740 1 -- 172.21.15.39:0/200947663 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c6cbefb400 msgr2=0x55c6cbefff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:06.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.423+0000 7ff899400640 1 --2- 172.21.15.39:0/2483746991 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff8800a2880 0x7ff8800a4d40 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x55c6cbefbc60 tx=0x7ff888005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:06.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.423+0000 7ff8a3e24740 1 --2- 172.21.15.39:0/200947663 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c6cbefb400 0x55c6cbefff00 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ff88c009db0 tx=0x7ff88c011810 comp rx=0 tx=0).stop 2024-03-18T16:10:06.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.424+0000 7ff8a3e24740 1 -- 172.21.15.39:0/200947663 shutdown_connections 2024-03-18T16:10:06.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.424+0000 7ff8a3e24740 1 --2- 172.21.15.39:0/200947663 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c6cbefb400 0x55c6cbefff00 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:06.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.424+0000 7ff8a3e24740 1 --2- 172.21.15.39:0/200947663 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c6cbf00570 0x55c6cbf02a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:06.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.424+0000 7ff8a3e24740 1 --2- 172.21.15.39:0/200947663 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c6cbefaa10 0x55c6cbefae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:06.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.424+0000 7ff8a3e24740 1 -- 172.21.15.39:0/200947663 >> 172.21.15.39:0/200947663 conn(0x55c6cbef5f80 msgr2=0x55c6cbef6380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:06.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.424+0000 7ff8a3e24740 1 -- 172.21.15.39:0/200947663 shutdown_connections 2024-03-18T16:10:06.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.424+0000 7ff8a3e24740 1 -- 172.21.15.39:0/200947663 wait complete. 2024-03-18T16:10:06.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.424+0000 7ff8a3e24740 1 -- start start 2024-03-18T16:10:06.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.425+0000 7ff8a3e24740 1 Processor -- start 2024-03-18T16:10:06.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.425+0000 7ff8a3e24740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c6cbefaa10 0x55c6cbf04f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:06.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.425+0000 7ff8a3e24740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c6cbefb400 0x55c6cbf05490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:06.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.425+0000 7ff8a3e24740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c6cbf00570 0x55c6cbf0c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:06.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.425+0000 7ff8a3e24740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55c6cbf03520 con 0x55c6cbefb400 2024-03-18T16:10:06.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.425+0000 7ff8a3e24740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55c6cbe1aed0 con 0x55c6cbf00570 2024-03-18T16:10:06.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.425+0000 7ff8a3e24740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55c6cbf036a0 con 0x55c6cbefaa10 2024-03-18T16:10:06.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.425+0000 7ff8a3e24740 10 client.0 _ll_register_callbacks cb 0x55c6cbe56220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:10:06.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.425+0000 7ff89a800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c6cbf00570 0x55c6cbf0c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:06.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.425+0000 7ff899e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c6cbefaa10 0x55c6cbf04f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:06.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.425+0000 7ff89a800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c6cbf00570 0x55c6cbf0c4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:36716/0 (socket says 172.21.15.39:36716) 2024-03-18T16:10:06.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.425+0000 7ff899400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c6cbefb400 0x55c6cbf05490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:06.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.426+0000 7ff89a800640 1 -- 172.21.15.39:0/1842977094 learned_addr learned my addr 172.21.15.39:0/1842977094 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:10:06.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.426+0000 7ff89a800640 1 -- 172.21.15.39:0/1842977094 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c6cbefaa10 msgr2=0x55c6cbf04f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:06.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.426+0000 7ff89a800640 1 --2- 172.21.15.39:0/1842977094 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c6cbefaa10 0x55c6cbf04f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:06.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.426+0000 7ff89a800640 1 -- 172.21.15.39:0/1842977094 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c6cbefb400 msgr2=0x55c6cbf05490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:06.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.426+0000 7ff89a800640 1 --2- 172.21.15.39:0/1842977094 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c6cbefb400 0x55c6cbf05490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:06.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.426+0000 7ff89a800640 1 -- 172.21.15.39:0/1842977094 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c6cbb733f0 con 0x55c6cbf00570 2024-03-18T16:10:06.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.426+0000 7ff899e00640 1 --2- 172.21.15.39:0/1842977094 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c6cbefaa10 0x55c6cbf04f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:10:06.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.426+0000 7ff89a800640 1 --2- 172.21.15.39:0/1842977094 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c6cbf00570 0x55c6cbf0c4d0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7ff8940c5b20 tx=0x7ff8940b9c60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fcd087ab49b7cf89 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:06.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.426+0000 7ff87be00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:10:06.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.426+0000 7ff87be00640 1 -- 172.21.15.39:0/1842977094 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8940bdfb0 con 0x55c6cbf00570 2024-03-18T16:10:06.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.426+0000 7ff87be00640 1 -- 172.21.15.39:0/1842977094 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7ff8940d6520 con 0x55c6cbf00570 2024-03-18T16:10:06.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.426+0000 7ff8a3e24740 1 -- 172.21.15.39:0/1842977094 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff88c0043c0 con 0x55c6cbf00570 2024-03-18T16:10:06.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.427+0000 7ff87be00640 1 -- 172.21.15.39:0/1842977094 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff8940db050 con 0x55c6cbf00570 2024-03-18T16:10:06.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.427+0000 7ff8a3e24740 10 client.25598 fetch_fsmap learned FSMap version 28 2024-03-18T16:10:06.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.427+0000 7ff8a3e24740 1 -- 172.21.15.39:0/1842977094 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55c6cbb330f0 con 0x55c6cbf00570 2024-03-18T16:10:06.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.427+0000 7ff87be00640 1 -- 172.21.15.39:0/1842977094 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff88c0043c0 con 0x55c6cbf00570 2024-03-18T16:10:06.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.427+0000 7ff87be00640 1 -- 172.21.15.39:0/1842977094 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7ff8940bebe0 con 0x55c6cbf00570 2024-03-18T16:10:06.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.427+0000 7ff8a3e24740 10 client.25598 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T16:10:06.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.427+0000 7ff8a3e24740 10 client.25598 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:10:06.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.427+0000 7ff8a3e24740 20 client.25598 populate_metadata read hostname 'smithi039' 2024-03-18T16:10:06.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.427+0000 7ff8a3e24740 1 --2- 172.21.15.39:0/1842977094 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55c6cbf10d50 0x55c6cbf131b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:06.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.427+0000 7ff8a3e24740 4 client.25598 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:10:06.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.427+0000 7ff8a3e24740 1 -- 172.21.15.39:0/1842977094 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c6cbe72be0 con 0x55c6cbf10d50 2024-03-18T16:10:06.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.427+0000 7ff899e00640 1 --2- 172.21.15.39:0/1842977094 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55c6cbf10d50 0x55c6cbf131b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:06.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.428+0000 7ff899e00640 1 --2- 172.21.15.39:0/1842977094 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55c6cbf10d50 0x55c6cbf131b0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2da28c7f5ee6664 server_cookie=4aba4637ebc84770 in_seq=0 out_seq=0 2024-03-18T16:10:06.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.428+0000 7ff87be00640 10 client.25598 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:10:06.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.429+0000 7ff87be00640 1 -- 172.21.15.39:0/1842977094 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c6cbe72be0 con 0x55c6cbf10d50 2024-03-18T16:10:06.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.429+0000 7ff87be00640 10 client.25598 handle_command_reply: tid=0 2024-03-18T16:10:06.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.429+0000 7ff8a3e24740 1 client.25598 shutdown 2024-03-18T16:10:06.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.429+0000 7ff8a3e24740 1 -- 172.21.15.39:0/1842977094 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c6cbf00570 msgr2=0x55c6cbf0c4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:06.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.429+0000 7ff8a3e24740 1 --2- 172.21.15.39:0/1842977094 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c6cbf00570 0x55c6cbf0c4d0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7ff8940c5b20 tx=0x7ff8940b9c60 comp rx=0 tx=0).stop 2024-03-18T16:10:06.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.429+0000 7ff8a3e24740 1 -- 172.21.15.39:0/1842977094 shutdown_connections 2024-03-18T16:10:06.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.429+0000 7ff8a3e24740 1 --2- 172.21.15.39:0/1842977094 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55c6cbf10d50 0x55c6cbf131b0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:06.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.429+0000 7ff8a3e24740 1 --2- 172.21.15.39:0/1842977094 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c6cbf00570 0x55c6cbf0c4d0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:06.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.429+0000 7ff8a3e24740 1 --2- 172.21.15.39:0/1842977094 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c6cbefb400 0x55c6cbf05490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:06.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.429+0000 7ff8a3e24740 1 --2- 172.21.15.39:0/1842977094 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c6cbefaa10 0x55c6cbf04f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:06.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.429+0000 7ff8a3e24740 1 -- 172.21.15.39:0/1842977094 >> 172.21.15.39:0/1842977094 conn(0x55c6cbef5f80 msgr2=0x55c6cbef5110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:06.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.429+0000 7ff87be00640 0 client.25598 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:10:06.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.429+0000 7ff8a3e24740 1 -- 172.21.15.39:0/1842977094 shutdown_connections 2024-03-18T16:10:06.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.429+0000 7ff8a3e24740 1 -- 172.21.15.39:0/1842977094 wait complete. 2024-03-18T16:10:06.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.429+0000 7ff8a3e24740 20 client.25598 trim_cache size 0 max 16384 2024-03-18T16:10:06.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.449+0000 7ff8a3e24740 1 Processor -- start 2024-03-18T16:10:06.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.449+0000 7ff8a3e24740 1 -- start start 2024-03-18T16:10:06.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.449+0000 7ff8a3e24740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c6cbefaa10 0x55c6cbef4880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:06.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.449+0000 7ff8a3e24740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c6cbefb400 0x55c6cbf0c490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:06.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.449+0000 7ff8a3e24740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c6cbf00570 0x55c6cbef4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:06.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.449+0000 7ff8a3e24740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7ff89c1ac3a0 con 0x55c6cbf00570 2024-03-18T16:10:06.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.449+0000 7ff8a3e24740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ff89c1adee0 con 0x55c6cbefaa10 2024-03-18T16:10:06.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.449+0000 7ff8a3e24740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7ff89c1aca70 con 0x55c6cbefb400 2024-03-18T16:10:06.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.450+0000 7ff899e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c6cbefaa10 0x55c6cbef4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:06.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.450+0000 7ff899400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c6cbefb400 0x55c6cbf0c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:06.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.450+0000 7ff899e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c6cbefaa10 0x55c6cbef4880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:36724/0 (socket says 172.21.15.39:36724) 2024-03-18T16:10:06.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.450+0000 7ff899e00640 1 -- 172.21.15.39:0/711808221 learned_addr learned my addr 172.21.15.39:0/711808221 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:10:06.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.450+0000 7ff899400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c6cbefb400 0x55c6cbf0c490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:50054/0 (socket says 172.21.15.39:50054) 2024-03-18T16:10:06.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.450+0000 7ff89a800640 1 --2- 172.21.15.39:0/711808221 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c6cbf00570 0x55c6cbef4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:06.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.450+0000 7ff899e00640 1 -- 172.21.15.39:0/711808221 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c6cbefb400 msgr2=0x55c6cbf0c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:06.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.450+0000 7ff899e00640 1 --2- 172.21.15.39:0/711808221 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c6cbefb400 0x55c6cbf0c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:06.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.450+0000 7ff899e00640 1 -- 172.21.15.39:0/711808221 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c6cbf00570 msgr2=0x55c6cbef4dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:06.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.450+0000 7ff899e00640 1 --2- 172.21.15.39:0/711808221 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c6cbf00570 0x55c6cbef4dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:06.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.450+0000 7ff899e00640 1 -- 172.21.15.39:0/711808221 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c6cbef0560 con 0x55c6cbefaa10 2024-03-18T16:10:06.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.450+0000 7ff899e00640 1 --2- 172.21.15.39:0/711808221 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c6cbefaa10 0x55c6cbef4880 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ff88c009b40 tx=0x7ff88c02bbe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9adddfd0cdad0fd0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:06.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.451+0000 7ff886a00640 1 -- 172.21.15.39:0/711808221 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff88c02bc50 con 0x55c6cbefaa10 2024-03-18T16:10:06.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.451+0000 7ff886a00640 1 -- 172.21.15.39:0/711808221 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff88c005620 con 0x55c6cbefaa10 2024-03-18T16:10:06.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.451+0000 7ff886a00640 1 -- 172.21.15.39:0/711808221 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff88c02b730 con 0x55c6cbefaa10 2024-03-18T16:10:06.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.451+0000 7ff8a3e24740 1 -- 172.21.15.39:0/711808221 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c6cbefaa10 msgr2=0x55c6cbef4880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:06.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.451+0000 7ff8a3e24740 1 --2- 172.21.15.39:0/711808221 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c6cbefaa10 0x55c6cbef4880 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ff88c009b40 tx=0x7ff88c02bbe0 comp rx=0 tx=0).stop 2024-03-18T16:10:06.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.451+0000 7ff8a3e24740 1 -- 172.21.15.39:0/711808221 shutdown_connections 2024-03-18T16:10:06.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.451+0000 7ff8a3e24740 1 --2- 172.21.15.39:0/711808221 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c6cbf00570 0x55c6cbef4dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:06.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.451+0000 7ff8a3e24740 1 --2- 172.21.15.39:0/711808221 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c6cbefb400 0x55c6cbf0c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:06.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.451+0000 7ff8a3e24740 1 --2- 172.21.15.39:0/711808221 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c6cbefaa10 0x55c6cbef4880 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:06.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.451+0000 7ff8a3e24740 1 -- 172.21.15.39:0/711808221 >> 172.21.15.39:0/711808221 conn(0x55c6cbef5f80 msgr2=0x55c6cbefc900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:06.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.452+0000 7ff8a3e24740 1 -- 172.21.15.39:0/711808221 shutdown_connections 2024-03-18T16:10:06.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.452+0000 7ff8a3e24740 1 -- 172.21.15.39:0/711808221 wait complete. 2024-03-18T16:10:06.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.452+0000 7ff8a3e24740 1 -- start start 2024-03-18T16:10:06.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.452+0000 7ff8a3e24740 1 Processor -- start 2024-03-18T16:10:06.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.453+0000 7ff8a3e24740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c6cbefaa10 0x55c6cbf09fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:06.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.453+0000 7ff8a3e24740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c6cbefb400 0x55c6cbf0a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:06.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.453+0000 7ff8a3e24740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c6cbf00570 0x55c6cbf0b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:06.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.453+0000 7ff8a3e24740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7ff89c1ad810 con 0x55c6cbefaa10 2024-03-18T16:10:06.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.453+0000 7ff8a3e24740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ff89c1ad140 con 0x55c6cbefb400 2024-03-18T16:10:06.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.453+0000 7ff8a3e24740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7ff89c1aa380 con 0x55c6cbf00570 2024-03-18T16:10:06.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.453+0000 7ff8a3e24740 10 client.0 _ll_register_callbacks cb 0x7ff8940b9450 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:10:06.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.453+0000 7ff899400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c6cbefb400 0x55c6cbf0a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:06.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.453+0000 7ff89a800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c6cbf00570 0x55c6cbf0b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:06.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.453+0000 7ff899400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c6cbefb400 0x55c6cbf0a500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:36726/0 (socket says 172.21.15.39:36726) 2024-03-18T16:10:06.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.453+0000 7ff899400640 1 -- 172.21.15.39:0/3990282043 learned_addr learned my addr 172.21.15.39:0/3990282043 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:10:06.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.453+0000 7ff899e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c6cbefaa10 0x55c6cbf09fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:06.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.453+0000 7ff899400640 1 -- 172.21.15.39:0/3990282043 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c6cbf00570 msgr2=0x55c6cbf0b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:06.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.453+0000 7ff899400640 1 --2- 172.21.15.39:0/3990282043 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c6cbf00570 0x55c6cbf0b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:06.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.453+0000 7ff899400640 1 -- 172.21.15.39:0/3990282043 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c6cbefaa10 msgr2=0x55c6cbf09fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:06.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.453+0000 7ff899400640 1 --2- 172.21.15.39:0/3990282043 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c6cbefaa10 0x55c6cbf09fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:06.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.453+0000 7ff899400640 1 -- 172.21.15.39:0/3990282043 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c6cbf0ba40 con 0x55c6cbefb400 2024-03-18T16:10:06.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.453+0000 7ff89a800640 1 --2- 172.21.15.39:0/3990282043 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c6cbf00570 0x55c6cbf0b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:10:06.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.454+0000 7ff899400640 1 --2- 172.21.15.39:0/3990282043 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c6cbefb400 0x55c6cbf0a500 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7ff8880106e0 tx=0x7ff888005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cf57d0d2df41bb2a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:06.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.454+0000 7ff87be00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:10:06.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.454+0000 7ff87be00640 1 -- 172.21.15.39:0/3990282043 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff888031020 con 0x55c6cbefb400 2024-03-18T16:10:06.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.454+0000 7ff8a3e24740 1 -- 172.21.15.39:0/3990282043 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff88c005620 con 0x55c6cbefb400 2024-03-18T16:10:06.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.454+0000 7ff87be00640 1 -- 172.21.15.39:0/3990282043 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7ff888009040 con 0x55c6cbefb400 2024-03-18T16:10:06.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.454+0000 7ff87be00640 1 -- 172.21.15.39:0/3990282043 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff88803c050 con 0x55c6cbefb400 2024-03-18T16:10:06.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.454+0000 7ff8a3e24740 10 client.25604 fetch_fsmap learned FSMap version 28 2024-03-18T16:10:06.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.454+0000 7ff8a3e24740 1 -- 172.21.15.39:0/3990282043 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55c6cbf77570 con 0x55c6cbefb400 2024-03-18T16:10:06.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.455+0000 7ff87be00640 1 -- 172.21.15.39:0/3990282043 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff88c005620 con 0x55c6cbefb400 2024-03-18T16:10:06.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.455+0000 7ff87be00640 1 -- 172.21.15.39:0/3990282043 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7ff888020810 con 0x55c6cbefb400 2024-03-18T16:10:06.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.455+0000 7ff8a3e24740 10 client.25604 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T16:10:06.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.455+0000 7ff8a3e24740 10 client.25604 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:10:06.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.455+0000 7ff8a3e24740 20 client.25604 populate_metadata read hostname 'smithi039' 2024-03-18T16:10:06.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.455+0000 7ff8a3e24740 1 --2- 172.21.15.39:0/3990282043 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55c6cbf77f70 0x55c6cbf7a390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:06.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.455+0000 7ff8a3e24740 4 client.25604 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T16:10:06.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.455+0000 7ff8a3e24740 1 -- 172.21.15.39:0/3990282043 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55c6cbe6c730 con 0x55c6cbf77f70 2024-03-18T16:10:06.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.455+0000 7ff899e00640 1 --2- 172.21.15.39:0/3990282043 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55c6cbf77f70 0x55c6cbf7a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:06.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.456+0000 7ff899e00640 1 --2- 172.21.15.39:0/3990282043 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55c6cbf77f70 0x55c6cbf7a390 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8cd5a7108a421846 server_cookie=eb016f082a3ef381 in_seq=0 out_seq=0 2024-03-18T16:10:06.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.456+0000 7ff87be00640 10 client.25604 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:10:06.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.456+0000 7ff87be00640 1 -- 172.21.15.39:0/3990282043 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55c6cbe6c730 con 0x55c6cbf77f70 2024-03-18T16:10:06.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.456+0000 7ff87be00640 10 client.25604 handle_command_reply: tid=0 2024-03-18T16:10:06.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.456+0000 7ff8a3e24740 1 client.25604 shutdown 2024-03-18T16:10:06.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.456+0000 7ff8a3e24740 1 -- 172.21.15.39:0/3990282043 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c6cbefb400 msgr2=0x55c6cbf0a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:06.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.456+0000 7ff8a3e24740 1 --2- 172.21.15.39:0/3990282043 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c6cbefb400 0x55c6cbf0a500 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7ff8880106e0 tx=0x7ff888005f90 comp rx=0 tx=0).stop 2024-03-18T16:10:06.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.456+0000 7ff8a3e24740 1 -- 172.21.15.39:0/3990282043 shutdown_connections 2024-03-18T16:10:06.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.456+0000 7ff8a3e24740 1 --2- 172.21.15.39:0/3990282043 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55c6cbf77f70 0x55c6cbf7a390 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:06.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.456+0000 7ff87be00640 0 client.25604 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:10:06.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.456+0000 7ff8a3e24740 1 --2- 172.21.15.39:0/3990282043 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c6cbefb400 0x55c6cbf0a500 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:06.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.456+0000 7ff8a3e24740 1 --2- 172.21.15.39:0/3990282043 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c6cbf00570 0x55c6cbf0b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:06.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.456+0000 7ff8a3e24740 1 --2- 172.21.15.39:0/3990282043 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c6cbefaa10 0x55c6cbf09fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:06.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.456+0000 7ff8a3e24740 1 -- 172.21.15.39:0/3990282043 >> 172.21.15.39:0/3990282043 conn(0x55c6cbef5f80 msgr2=0x55c6cbef91e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:06.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.456+0000 7ff8a3e24740 1 -- 172.21.15.39:0/3990282043 shutdown_connections 2024-03-18T16:10:06.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.456+0000 7ff8a3e24740 1 -- 172.21.15.39:0/3990282043 wait complete. 2024-03-18T16:10:06.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.456+0000 7ff8a3e24740 20 client.25604 trim_cache size 0 max 16384 2024-03-18T16:10:06.462 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-03-18T16:10:06.462 INFO:teuthology.orchestra.run.smithi039.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-18T16:10:06.462 INFO:teuthology.orchestra.run.smithi039.stdout: "scrubs": { 2024-03-18T16:10:06.462 INFO:teuthology.orchestra.run.smithi039.stdout: "8ed4bef1-ef24-4a17-88db-4b54e8c8530c": { 2024-03-18T16:10:06.462 INFO:teuthology.orchestra.run.smithi039.stdout: "path": "/", 2024-03-18T16:10:06.462 INFO:teuthology.orchestra.run.smithi039.stdout: "tag": "8ed4bef1-ef24-4a17-88db-4b54e8c8530c", 2024-03-18T16:10:06.462 INFO:teuthology.orchestra.run.smithi039.stdout: "options": "recursive,force" 2024-03-18T16:10:06.463 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:10:06.463 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:10:06.463 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-03-18T16:10:06.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.457+0000 7ff89b200640 1 -- 172.21.15.39:0/2483746991 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff8800a2880 msgr2=0x7ff8800a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:06.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.457+0000 7ff89b200640 1 --2- 172.21.15.39:0/2483746991 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff8800a2880 0x7ff8800a4d40 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x55c6cbefbc60 tx=0x7ff888005850 comp rx=0 tx=0).stop 2024-03-18T16:10:06.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.457+0000 7ff89b200640 1 -- 172.21.15.39:0/2483746991 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff89c105490 msgr2=0x7ff89c1a10f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:06.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.457+0000 7ff89b200640 1 --2- 172.21.15.39:0/2483746991 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff89c105490 0x7ff89c1a10f0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7ff89402c430 tx=0x7ff8940337b0 comp rx=0 tx=0).stop 2024-03-18T16:10:06.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.457+0000 7ff89b200640 1 -- 172.21.15.39:0/2483746991 shutdown_connections 2024-03-18T16:10:06.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.457+0000 7ff89b200640 1 --2- 172.21.15.39:0/2483746991 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff89c105490 0x7ff89c1a10f0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:06.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.457+0000 7ff89b200640 1 --2- 172.21.15.39:0/2483746991 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff8800a2880 0x7ff8800a4d40 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:06.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.457+0000 7ff89b200640 1 --2- 172.21.15.39:0/2483746991 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff89c104ad0 0x7ff89c19a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:06.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.457+0000 7ff89b200640 1 --2- 172.21.15.39:0/2483746991 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff89c1038e0 0x7ff89c199b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:06.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.457+0000 7ff89b200640 1 -- 172.21.15.39:0/2483746991 >> 172.21.15.39:0/2483746991 conn(0x7ff89c0fee50 msgr2=0x7ff89c107d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:06.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.458+0000 7ff89b200640 1 -- 172.21.15.39:0/2483746991 shutdown_connections 2024-03-18T16:10:06.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:06.458+0000 7ff89b200640 1 -- 172.21.15.39:0/2483746991 wait complete. 2024-03-18T16:10:06.476 INFO:tasks.cephfs.filesystem:scrub status for tag:8ed4bef1-ef24-4a17-88db-4b54e8c8530c - {'path': '/', 'tag': '8ed4bef1-ef24-4a17-88db-4b54e8c8530c', 'options': 'recursive,force'} 2024-03-18T16:10:06.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:10:06 smithi194 ceph-mon[36200]: pgmap v423: 129 pgs: 129 active+clean; 649 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 938 B/s wr, 11 op/s 2024-03-18T16:10:06.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:10:06 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/3055690167' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:10:06.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:10:06 smithi194 ceph-mon[36200]: scrub queued for path: / 2024-03-18T16:10:06.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:10:06 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:10:06.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:10:06 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:10:06.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:10:06 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:10:06.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:10:06 smithi039 ceph-mon[31721]: pgmap v423: 129 pgs: 129 active+clean; 649 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 938 B/s wr, 11 op/s 2024-03-18T16:10:06.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:10:06 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/3055690167' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:10:06.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:10:06 smithi039 ceph-mon[31721]: scrub queued for path: / 2024-03-18T16:10:06.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:10:06 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:10:06.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:10:06 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:10:06.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:10:06 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:10:06.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:10:06 smithi196 ceph-mon[36307]: pgmap v423: 129 pgs: 129 active+clean; 649 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 938 B/s wr, 11 op/s 2024-03-18T16:10:06.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:10:06 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/3055690167' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:10:06.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:10:06 smithi196 ceph-mon[36307]: scrub queued for path: / 2024-03-18T16:10:06.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:10:06 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:10:06.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:10:06 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:10:06.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:10:06 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:10:08.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:10:08 smithi196 ceph-mon[36307]: pgmap v424: 129 pgs: 129 active+clean; 649 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 938 B/s wr, 11 op/s 2024-03-18T16:10:08.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:10:08 smithi194 ceph-mon[36200]: pgmap v424: 129 pgs: 129 active+clean; 649 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 938 B/s wr, 11 op/s 2024-03-18T16:10:09.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:10:08 smithi039 ceph-mon[31721]: pgmap v424: 129 pgs: 129 active+clean; 649 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 938 B/s wr, 11 op/s 2024-03-18T16:10:10.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:10:10 smithi196 ceph-mon[36307]: pgmap v425: 129 pgs: 129 active+clean; 656 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.7 KiB/s wr, 12 op/s 2024-03-18T16:10:10.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:10:10 smithi194 ceph-mon[36200]: pgmap v425: 129 pgs: 129 active+clean; 656 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.7 KiB/s wr, 12 op/s 2024-03-18T16:10:11.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:10:10 smithi039 ceph-mon[31721]: pgmap v425: 129 pgs: 129 active+clean; 656 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.7 KiB/s wr, 12 op/s 2024-03-18T16:10:12.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:10:12 smithi196 ceph-mon[36307]: pgmap v426: 129 pgs: 129 active+clean; 656 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-03-18T16:10:12.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:10:12 smithi194 ceph-mon[36200]: pgmap v426: 129 pgs: 129 active+clean; 656 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-03-18T16:10:13.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:10:12 smithi039 ceph-mon[31721]: pgmap v426: 129 pgs: 129 active+clean; 656 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-03-18T16:10:14.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:10:14 smithi196 ceph-mon[36307]: pgmap v427: 129 pgs: 129 active+clean; 656 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-03-18T16:10:14.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:10:14 smithi194 ceph-mon[36200]: pgmap v427: 129 pgs: 129 active+clean; 656 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-03-18T16:10:15.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:10:14 smithi039 ceph-mon[31721]: pgmap v427: 129 pgs: 129 active+clean; 656 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-03-18T16:10:16.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:10:16 smithi196 ceph-mon[36307]: pgmap v428: 129 pgs: 129 active+clean; 656 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-03-18T16:10:16.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:10:16 smithi194 ceph-mon[36200]: pgmap v428: 129 pgs: 129 active+clean; 656 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-03-18T16:10:17.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:10:16 smithi039 ceph-mon[31721]: pgmap v428: 129 pgs: 129 active+clean; 656 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-03-18T16:10:17.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:10:17 smithi196 ceph-mon[36307]: scrub summary: idle 2024-03-18T16:10:17.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:10:17 smithi194 ceph-mon[36200]: scrub summary: idle 2024-03-18T16:10:18.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:10:17 smithi039 ceph-mon[31721]: scrub summary: idle 2024-03-18T16:10:18.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:10:18 smithi196 ceph-mon[36307]: pgmap v429: 129 pgs: 129 active+clean; 656 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-03-18T16:10:18.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:10:18 smithi194 ceph-mon[36200]: pgmap v429: 129 pgs: 129 active+clean; 656 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-03-18T16:10:19.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:10:18 smithi039 ceph-mon[31721]: pgmap v429: 129 pgs: 129 active+clean; 656 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-03-18T16:10:20.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:10:20 smithi196 ceph-mon[36307]: pgmap v430: 129 pgs: 129 active+clean; 656 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 852 B/s wr, 11 op/s 2024-03-18T16:10:20.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:10:20 smithi194 ceph-mon[36200]: pgmap v430: 129 pgs: 129 active+clean; 656 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 852 B/s wr, 11 op/s 2024-03-18T16:10:21.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:10:20 smithi039 ceph-mon[31721]: pgmap v430: 129 pgs: 129 active+clean; 656 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 852 B/s wr, 11 op/s 2024-03-18T16:10:22.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:10:22 smithi196 ceph-mon[36307]: pgmap v431: 129 pgs: 129 active+clean; 656 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-18T16:10:22.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:10:22 smithi194 ceph-mon[36200]: pgmap v431: 129 pgs: 129 active+clean; 656 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-18T16:10:23.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:10:22 smithi039 ceph-mon[31721]: pgmap v431: 129 pgs: 129 active+clean; 656 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-18T16:10:24.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:10:24 smithi196 ceph-mon[36307]: pgmap v432: 129 pgs: 129 active+clean; 656 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T16:10:24.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:10:24 smithi194 ceph-mon[36200]: pgmap v432: 129 pgs: 129 active+clean; 656 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T16:10:25.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:10:24 smithi039 ceph-mon[31721]: pgmap v432: 129 pgs: 129 active+clean; 656 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T16:10:26.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:10:26 smithi196 ceph-mon[36307]: pgmap v433: 129 pgs: 129 active+clean; 656 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T16:10:26.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:10:26 smithi194 ceph-mon[36200]: pgmap v433: 129 pgs: 129 active+clean; 656 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T16:10:27.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:10:26 smithi039 ceph-mon[31721]: pgmap v433: 129 pgs: 129 active+clean; 656 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T16:10:28.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:10:28 smithi196 ceph-mon[36307]: pgmap v434: 129 pgs: 129 active+clean; 656 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T16:10:28.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:10:28 smithi194 ceph-mon[36200]: pgmap v434: 129 pgs: 129 active+clean; 656 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T16:10:29.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:10:28 smithi039 ceph-mon[31721]: pgmap v434: 129 pgs: 129 active+clean; 656 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T16:10:30.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:10:30 smithi196 ceph-mon[36307]: pgmap v435: 129 pgs: 129 active+clean; 656 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T16:10:30.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:10:30 smithi194 ceph-mon[36200]: pgmap v435: 129 pgs: 129 active+clean; 656 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T16:10:31.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:10:30 smithi039 ceph-mon[31721]: pgmap v435: 129 pgs: 129 active+clean; 656 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T16:10:32.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:10:32 smithi196 ceph-mon[36307]: pgmap v436: 129 pgs: 129 active+clean; 656 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-03-18T16:10:32.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:10:32 smithi194 ceph-mon[36200]: pgmap v436: 129 pgs: 129 active+clean; 656 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-03-18T16:10:33.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:10:32 smithi039 ceph-mon[31721]: pgmap v436: 129 pgs: 129 active+clean; 656 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 8 op/s 2024-03-18T16:10:34.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:10:34 smithi196 ceph-mon[36307]: pgmap v437: 129 pgs: 129 active+clean; 656 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-03-18T16:10:34.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:10:34 smithi194 ceph-mon[36200]: pgmap v437: 129 pgs: 129 active+clean; 656 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-03-18T16:10:35.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:10:34 smithi039 ceph-mon[31721]: pgmap v437: 129 pgs: 129 active+clean; 656 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-03-18T16:10:36.477 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T16:10:36.477 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T16:10:36.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.708+0000 7f2ccca00640 1 -- 172.21.15.39:0/4129815853 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2cc80757a0 msgr2=0x7f2cc8073bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:36.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.708+0000 7f2ccca00640 1 --2- 172.21.15.39:0/4129815853 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2cc80757a0 0x7f2cc8073bf0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f2cb8009900 tx=0x7f2cb802bcd0 comp rx=0 tx=0).stop 2024-03-18T16:10:36.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.708+0000 7f2ccca00640 1 -- 172.21.15.39:0/4129815853 shutdown_connections 2024-03-18T16:10:36.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.708+0000 7f2ccca00640 1 --2- 172.21.15.39:0/4129815853 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2cc8109590 0x7f2cc810ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:36.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.708+0000 7f2ccca00640 1 --2- 172.21.15.39:0/4129815853 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2cc80741c0 0x7f2cc8074620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:36.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.708+0000 7f2ccca00640 1 --2- 172.21.15.39:0/4129815853 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2cc80757a0 0x7f2cc8073bf0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:36.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.708+0000 7f2ccca00640 1 -- 172.21.15.39:0/4129815853 >> 172.21.15.39:0/4129815853 conn(0x7f2cc80fcc50 msgr2=0x7f2cc80ff090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:36.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.709+0000 7f2ccca00640 1 -- 172.21.15.39:0/4129815853 shutdown_connections 2024-03-18T16:10:36.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.709+0000 7f2ccca00640 1 -- 172.21.15.39:0/4129815853 wait complete. 2024-03-18T16:10:36.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.710+0000 7f2ccca00640 1 Processor -- start 2024-03-18T16:10:36.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.710+0000 7f2ccca00640 1 -- start start 2024-03-18T16:10:36.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.710+0000 7f2ccca00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2cc80741c0 0x7f2cc8199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:36.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.710+0000 7f2ccca00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2cc80757a0 0x7f2cc819a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:36.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.710+0000 7f2ccca00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2cc8109590 0x7f2cc81a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:36.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.710+0000 7f2ccca00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f2cc810f5b0 con 0x7f2cc8109590 2024-03-18T16:10:36.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.710+0000 7f2ccca00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f2cc810f430 con 0x7f2cc80757a0 2024-03-18T16:10:36.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.710+0000 7f2ccca00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f2cc810f730 con 0x7f2cc80741c0 2024-03-18T16:10:36.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.710+0000 7f2cc7400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2cc80741c0 0x7f2cc8199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:36.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.710+0000 7f2cc7400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2cc80741c0 0x7f2cc8199c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:44942/0 (socket says 172.21.15.39:44942) 2024-03-18T16:10:36.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.710+0000 7f2cc7400640 1 -- 172.21.15.39:0/1982092463 learned_addr learned my addr 172.21.15.39:0/1982092463 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:10:36.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.711+0000 7f2cc6a00640 1 --2- 172.21.15.39:0/1982092463 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2cc80757a0 0x7f2cc819a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:36.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.711+0000 7f2cc7e00640 1 --2- 172.21.15.39:0/1982092463 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2cc8109590 0x7f2cc81a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:36.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.711+0000 7f2cc7400640 1 -- 172.21.15.39:0/1982092463 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2cc80757a0 msgr2=0x7f2cc819a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:36.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.711+0000 7f2cc7400640 1 --2- 172.21.15.39:0/1982092463 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2cc80757a0 0x7f2cc819a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:36.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.711+0000 7f2cc7400640 1 -- 172.21.15.39:0/1982092463 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2cc8109590 msgr2=0x7f2cc81a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:36.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.711+0000 7f2cc7400640 1 --2- 172.21.15.39:0/1982092463 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2cc8109590 0x7f2cc81a1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:36.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.711+0000 7f2cc7400640 1 -- 172.21.15.39:0/1982092463 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cc81a1830 con 0x7f2cc80741c0 2024-03-18T16:10:36.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.711+0000 7f2cc7e00640 1 --2- 172.21.15.39:0/1982092463 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2cc8109590 0x7f2cc81a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:10:36.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.711+0000 7f2cc7400640 1 --2- 172.21.15.39:0/1982092463 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2cc80741c0 0x7f2cc8199c20 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f2cb8005ab0 tx=0x7f2cb802c760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:36.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.712+0000 7f2cbfe00640 1 -- 172.21.15.39:0/1982092463 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cb8043020 con 0x7f2cc80741c0 2024-03-18T16:10:36.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.712+0000 7f2cbfe00640 1 -- 172.21.15.39:0/1982092463 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2cb80054e0 con 0x7f2cc80741c0 2024-03-18T16:10:36.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.712+0000 7f2ccca00640 1 -- 172.21.15.39:0/1982092463 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cc81a1ac0 con 0x7f2cc80741c0 2024-03-18T16:10:36.716 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.712+0000 7f2cbfe00640 1 -- 172.21.15.39:0/1982092463 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cb8003ca0 con 0x7f2cc80741c0 2024-03-18T16:10:36.716 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.712+0000 7f2ccca00640 1 -- 172.21.15.39:0/1982092463 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cc81a1fc0 con 0x7f2cc80741c0 2024-03-18T16:10:36.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.717+0000 7f2cd0c22740 1 Processor -- start 2024-03-18T16:10:36.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.717+0000 7f2cbfe00640 1 -- 172.21.15.39:0/1982092463 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f2cb8046020 con 0x7f2cc80741c0 2024-03-18T16:10:36.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.717+0000 7f2cd0c22740 1 -- start start 2024-03-18T16:10:36.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.717+0000 7f2cd0c22740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557f9d6aba10 0x557f9d6abe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:36.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.717+0000 7f2cd0c22740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f9d6ac400 0x557f9d6b9710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:36.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.717+0000 7f2cd0c22740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557f9d6b9d80 0x557f9d6bc210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:36.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.717+0000 7f2cd0c22740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x557f9d6101f0 con 0x557f9d6b9d80 2024-03-18T16:10:36.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.717+0000 7f2cd0c22740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x557f9d5b8a80 con 0x557f9d6ac400 2024-03-18T16:10:36.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.717+0000 7f2cd0c22740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x557f9d60c290 con 0x557f9d6aba10 2024-03-18T16:10:36.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.718+0000 7f2cc7400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f9d6ac400 0x557f9d6b9710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:36.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.718+0000 7f2cc7400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f9d6ac400 0x557f9d6b9710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:43030/0 (socket says 172.21.15.39:43030) 2024-03-18T16:10:36.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.718+0000 7f2cc7400640 1 -- 172.21.15.39:0/1342179825 learned_addr learned my addr 172.21.15.39:0/1342179825 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:10:36.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.718+0000 7f2cc7400640 1 -- 172.21.15.39:0/1342179825 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557f9d6aba10 msgr2=0x557f9d6abe30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:10:36.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.718+0000 7f2cc7e00640 1 --2- 172.21.15.39:0/1342179825 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557f9d6aba10 0x557f9d6abe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:36.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.718+0000 7f2cc7400640 1 --2- 172.21.15.39:0/1342179825 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557f9d6aba10 0x557f9d6abe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:36.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.718+0000 7f2cc7400640 1 -- 172.21.15.39:0/1342179825 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557f9d6b9d80 msgr2=0x557f9d6bc210 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:10:36.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.719+0000 7f2cc6a00640 1 --2- 172.21.15.39:0/1342179825 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557f9d6b9d80 0x557f9d6bc210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:36.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.719+0000 7f2cc7400640 1 --2- 172.21.15.39:0/1342179825 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557f9d6b9d80 0x557f9d6bc210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:36.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.719+0000 7f2cc7400640 1 -- 172.21.15.39:0/1342179825 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f9d3ee7d0 con 0x557f9d6ac400 2024-03-18T16:10:36.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.719+0000 7f2cc7e00640 1 --2- 172.21.15.39:0/1342179825 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557f9d6aba10 0x557f9d6abe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:10:36.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.719+0000 7f2cc6a00640 1 --2- 172.21.15.39:0/1342179825 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557f9d6b9d80 0x557f9d6bc210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:10:36.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.719+0000 7f2cbfe00640 1 --2- 172.21.15.39:0/1982092463 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f2cac0a2880 0x7f2cac0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:36.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.719+0000 7f2cc7400640 1 --2- 172.21.15.39:0/1342179825 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f9d6ac400 0x557f9d6b9710 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f2cb80f4f50 tx=0x7f2cb80f91a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6776fe8d4ca96e21 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:36.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.719+0000 7f2cbfe00640 1 -- 172.21.15.39:0/1982092463 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f2cb80f10c0 con 0x7f2cc80741c0 2024-03-18T16:10:36.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.720+0000 7f2cb3e00640 1 -- 172.21.15.39:0/1342179825 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cb80dd070 con 0x557f9d6ac400 2024-03-18T16:10:36.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.720+0000 7f2cb3e00640 1 -- 172.21.15.39:0/1342179825 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2cb80035f0 con 0x557f9d6ac400 2024-03-18T16:10:36.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.720+0000 7f2cb3e00640 1 -- 172.21.15.39:0/1342179825 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cb80bd1b0 con 0x557f9d6ac400 2024-03-18T16:10:36.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.720+0000 7f2cc7e00640 1 --2- 172.21.15.39:0/1982092463 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f2cac0a2880 0x7f2cac0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:36.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.721+0000 7f2cd0c22740 1 -- 172.21.15.39:0/1342179825 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f9d6ac400 msgr2=0x557f9d6b9710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:36.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.721+0000 7f2cd0c22740 1 --2- 172.21.15.39:0/1342179825 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f9d6ac400 0x557f9d6b9710 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f2cb80f4f50 tx=0x7f2cb80f91a0 comp rx=0 tx=0).stop 2024-03-18T16:10:36.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.722+0000 7f2cd0c22740 1 -- 172.21.15.39:0/1342179825 shutdown_connections 2024-03-18T16:10:36.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.722+0000 7f2cd0c22740 1 --2- 172.21.15.39:0/1342179825 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f9d6ac400 0x557f9d6b9710 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:36.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.722+0000 7f2cd0c22740 1 --2- 172.21.15.39:0/1342179825 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557f9d6b9d80 0x557f9d6bc210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:36.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.722+0000 7f2cd0c22740 1 --2- 172.21.15.39:0/1342179825 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557f9d6aba10 0x557f9d6abe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:36.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.722+0000 7f2cd0c22740 1 -- 172.21.15.39:0/1342179825 >> 172.21.15.39:0/1342179825 conn(0x557f9d6a6f80 msgr2=0x557f9d6a7380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:36.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.722+0000 7f2cc7e00640 1 --2- 172.21.15.39:0/1982092463 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f2cac0a2880 0x7f2cac0a4d40 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x557f9d6acc60 tx=0x7f2cc0009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:36.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.722+0000 7f2cd0c22740 1 -- 172.21.15.39:0/1342179825 shutdown_connections 2024-03-18T16:10:36.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.722+0000 7f2cd0c22740 1 -- 172.21.15.39:0/1342179825 wait complete. 2024-03-18T16:10:36.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.723+0000 7f2cd0c22740 1 -- start start 2024-03-18T16:10:36.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.723+0000 7f2cd0c22740 1 Processor -- start 2024-03-18T16:10:36.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.724+0000 7f2cd0c22740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f9d6aba10 0x557f9d6be3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:36.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.724+0000 7f2cd0c22740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557f9d6ac400 0x557f9d6be920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:36.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.724+0000 7f2cd0c22740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557f9d6b9d80 0x557f9d6c5960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:36.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.724+0000 7f2cc7400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f9d6aba10 0x557f9d6be3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:36.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.724+0000 7f2cc7400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f9d6aba10 0x557f9d6be3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:43034/0 (socket says 172.21.15.39:43034) 2024-03-18T16:10:36.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.724+0000 7f2cc6a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557f9d6ac400 0x557f9d6be920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:36.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.724+0000 7f2cc6a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557f9d6ac400 0x557f9d6be920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43826/0 (socket says 172.21.15.39:43826) 2024-03-18T16:10:36.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.724+0000 7f2cc7400640 1 -- 172.21.15.39:0/3749996250 learned_addr learned my addr 172.21.15.39:0/3749996250 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:10:36.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.724+0000 7f2cd0c22740 1 -- 172.21.15.39:0/3749996250 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x557f9d6bcaf0 con 0x557f9d6ac400 2024-03-18T16:10:36.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.724+0000 7f2cd0c22740 1 -- 172.21.15.39:0/3749996250 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x557f9d5cbed0 con 0x557f9d6aba10 2024-03-18T16:10:36.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.724+0000 7f2cd0c22740 1 -- 172.21.15.39:0/3749996250 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x557f9d6bcc70 con 0x557f9d6b9d80 2024-03-18T16:10:36.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.724+0000 7f2cc7e00640 1 --2- 172.21.15.39:0/3749996250 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557f9d6b9d80 0x557f9d6c5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:36.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.725+0000 7f2cd0c22740 10 client.0 _ll_register_callbacks cb 0x557f9d607220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:10:36.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.725+0000 7f2cc6a00640 1 -- 172.21.15.39:0/3749996250 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557f9d6b9d80 msgr2=0x557f9d6c5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:36.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.725+0000 7f2cc6a00640 1 --2- 172.21.15.39:0/3749996250 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557f9d6b9d80 0x557f9d6c5960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:36.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.725+0000 7f2cc6a00640 1 -- 172.21.15.39:0/3749996250 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f9d6aba10 msgr2=0x557f9d6be3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:36.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.725+0000 7f2cc6a00640 1 --2- 172.21.15.39:0/3749996250 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f9d6aba10 0x557f9d6be3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:36.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.725+0000 7f2cc6a00640 1 -- 172.21.15.39:0/3749996250 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f9d3243f0 con 0x557f9d6ac400 2024-03-18T16:10:36.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.726+0000 7f2cd0c22740 1 -- 172.21.15.39:0/3749996250 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x557f9d623be0 con 0x557f9d6ac400 2024-03-18T16:10:36.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.726+0000 7f2cc6a00640 1 --2- 172.21.15.39:0/3749996250 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557f9d6ac400 0x557f9d6be920 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f2cb4005d00 tx=0x7f2cb40117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3fcfff49e1f73b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:36.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.726+0000 7f2cc7400640 1 --2- 172.21.15.39:0/3749996250 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f9d6aba10 0x557f9d6be3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:10:36.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.726+0000 7f2cb0c00640 10 client.15489 ms_handle_connect on v2:172.21.15.39:3300/0 2024-03-18T16:10:36.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.727+0000 7f2cb0c00640 1 -- 172.21.15.39:0/3749996250 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cb4004220 con 0x557f9d6ac400 2024-03-18T16:10:36.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.727+0000 7f2cb0c00640 1 -- 172.21.15.39:0/3749996250 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f2cb40043c0 con 0x557f9d6ac400 2024-03-18T16:10:36.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.727+0000 7f2cb0c00640 1 -- 172.21.15.39:0/3749996250 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f2cb4012660 con 0x557f9d6ac400 2024-03-18T16:10:36.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.727+0000 7f2cd0c22740 10 client.15489 fetch_fsmap learned FSMap version 28 2024-03-18T16:10:36.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.727+0000 7f2cd0c22740 1 -- 172.21.15.39:0/3749996250 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({fsmap=28}) -- 0x557f9d2e40f0 con 0x557f9d6ac400 2024-03-18T16:10:36.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.757+0000 7f2cb0c00640 1 -- 172.21.15.39:0/3749996250 <== mon.0 v2:172.21.15.39:3300/0 4 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f2cb4012800 con 0x557f9d6ac400 2024-03-18T16:10:36.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.757+0000 7f2cd0c22740 10 client.15489 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T16:10:36.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.758+0000 7f2cd0c22740 10 client.15489 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:10:36.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.758+0000 7f2cd0c22740 20 client.15489 populate_metadata read hostname 'smithi039' 2024-03-18T16:10:36.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.758+0000 7f2cd0c22740 1 --2- 172.21.15.39:0/3749996250 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x557f9d6ca0a0 0x557f9d6cc4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:36.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.758+0000 7f2cd0c22740 4 client.15489 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:10:36.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.758+0000 7f2cd0c22740 1 -- 172.21.15.39:0/3749996250 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557f9d61d730 con 0x557f9d6ca0a0 2024-03-18T16:10:36.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.758+0000 7f2cc7400640 1 --2- 172.21.15.39:0/3749996250 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x557f9d6ca0a0 0x557f9d6cc4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:36.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.759+0000 7f2cc7400640 1 --2- 172.21.15.39:0/3749996250 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x557f9d6ca0a0 0x557f9d6cc4f0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e9c1a530a985536e server_cookie=8f0d011d8156c94e in_seq=0 out_seq=0 2024-03-18T16:10:36.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.759+0000 7f2cb0c00640 10 client.15489 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:10:36.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.760+0000 7f2cb0c00640 1 -- 172.21.15.39:0/3749996250 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557f9d61d730 con 0x557f9d6ca0a0 2024-03-18T16:10:36.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.760+0000 7f2cb0c00640 10 client.15489 handle_command_reply: tid=0 2024-03-18T16:10:36.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.760+0000 7f2cd0c22740 1 client.15489 shutdown 2024-03-18T16:10:36.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.761+0000 7f2cd0c22740 1 -- 172.21.15.39:0/3749996250 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557f9d6ac400 msgr2=0x557f9d6be920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:36.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.761+0000 7f2cd0c22740 1 --2- 172.21.15.39:0/3749996250 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557f9d6ac400 0x557f9d6be920 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f2cb4005d00 tx=0x7f2cb40117b0 comp rx=0 tx=0).stop 2024-03-18T16:10:36.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.761+0000 7f2cd0c22740 1 -- 172.21.15.39:0/3749996250 shutdown_connections 2024-03-18T16:10:36.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.761+0000 7f2cd0c22740 1 --2- 172.21.15.39:0/3749996250 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x557f9d6ca0a0 0x557f9d6cc4f0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:36.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.761+0000 7f2cd0c22740 1 --2- 172.21.15.39:0/3749996250 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557f9d6b9d80 0x557f9d6c5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:36.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.761+0000 7f2cd0c22740 1 --2- 172.21.15.39:0/3749996250 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557f9d6ac400 0x557f9d6be920 secure :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f2cb4005d00 tx=0x7f2cb40117b0 comp rx=0 tx=0).stop 2024-03-18T16:10:36.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.761+0000 7f2cd0c22740 1 --2- 172.21.15.39:0/3749996250 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f9d6aba10 0x557f9d6be3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:36.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.761+0000 7f2cb0c00640 0 client.15489 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:10:36.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.761+0000 7f2cd0c22740 1 -- 172.21.15.39:0/3749996250 >> 172.21.15.39:0/3749996250 conn(0x557f9d6a6f80 msgr2=0x557f9d6a6110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:36.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.764+0000 7f2cd0c22740 1 -- 172.21.15.39:0/3749996250 shutdown_connections 2024-03-18T16:10:36.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.764+0000 7f2cd0c22740 1 -- 172.21.15.39:0/3749996250 wait complete. 2024-03-18T16:10:36.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.764+0000 7f2cd0c22740 20 client.15489 trim_cache size 0 max 16384 2024-03-18T16:10:36.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.801+0000 7f2cd0c22740 1 Processor -- start 2024-03-18T16:10:36.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.801+0000 7f2cd0c22740 1 -- start start 2024-03-18T16:10:36.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.802+0000 7f2cd0c22740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557f9d6aba10 0x557f9d6a5880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:36.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.802+0000 7f2cd0c22740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f9d6b9d80 0x557f9d6c5920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:36.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.802+0000 7f2cd0c22740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557f9d6b6860 0x557f9d6a5dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:36.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.802+0000 7f2cd0c22740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f2cc81a8130 con 0x557f9d6b6860 2024-03-18T16:10:36.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.802+0000 7f2cd0c22740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f2cc81a9c70 con 0x557f9d6b9d80 2024-03-18T16:10:36.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.802+0000 7f2cd0c22740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f2cc81a8800 con 0x557f9d6aba10 2024-03-18T16:10:36.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.802+0000 7f2cc6a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f9d6b9d80 0x557f9d6c5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:36.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.802+0000 7f2cc6a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f9d6b9d80 0x557f9d6c5920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:43036/0 (socket says 172.21.15.39:43036) 2024-03-18T16:10:36.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.802+0000 7f2cc6a00640 1 -- 172.21.15.39:0/2432974132 learned_addr learned my addr 172.21.15.39:0/2432974132 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:10:36.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.802+0000 7f2cc7400640 1 --2- 172.21.15.39:0/2432974132 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557f9d6aba10 0x557f9d6a5880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:36.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.803+0000 7f2cc7e00640 1 --2- 172.21.15.39:0/2432974132 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557f9d6b6860 0x557f9d6a5dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:36.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.803+0000 7f2cc7400640 1 -- 172.21.15.39:0/2432974132 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f9d6b9d80 msgr2=0x557f9d6c5920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:36.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.803+0000 7f2cc7400640 1 --2- 172.21.15.39:0/2432974132 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f9d6b9d80 0x557f9d6c5920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:36.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.803+0000 7f2cc7400640 1 -- 172.21.15.39:0/2432974132 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557f9d6b6860 msgr2=0x557f9d6a5dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:36.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.803+0000 7f2cc7400640 1 --2- 172.21.15.39:0/2432974132 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557f9d6b6860 0x557f9d6a5dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:36.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.803+0000 7f2cc7400640 1 -- 172.21.15.39:0/2432974132 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f9d6ac7e0 con 0x557f9d6aba10 2024-03-18T16:10:36.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.803+0000 7f2cc7400640 1 --2- 172.21.15.39:0/2432974132 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557f9d6aba10 0x557f9d6a5880 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f2cb8006010 tx=0x7f2cb8005c90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=406beef0da796bb2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:36.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.803+0000 7f2cb3400640 1 -- 172.21.15.39:0/2432974132 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cb80d5020 con 0x557f9d6aba10 2024-03-18T16:10:36.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.803+0000 7f2cb3400640 1 -- 172.21.15.39:0/2432974132 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2cb8003e40 con 0x557f9d6aba10 2024-03-18T16:10:36.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.803+0000 7f2cb3400640 1 -- 172.21.15.39:0/2432974132 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cb80dd040 con 0x557f9d6aba10 2024-03-18T16:10:36.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.804+0000 7f2cd0c22740 1 -- 172.21.15.39:0/2432974132 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557f9d6aba10 msgr2=0x557f9d6a5880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:36.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.804+0000 7f2cd0c22740 1 --2- 172.21.15.39:0/2432974132 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557f9d6aba10 0x557f9d6a5880 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f2cb8006010 tx=0x7f2cb8005c90 comp rx=0 tx=0).stop 2024-03-18T16:10:36.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.805+0000 7f2cd0c22740 1 -- 172.21.15.39:0/2432974132 shutdown_connections 2024-03-18T16:10:36.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.805+0000 7f2cd0c22740 1 --2- 172.21.15.39:0/2432974132 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f9d6b9d80 0x557f9d6c5920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:36.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.805+0000 7f2cd0c22740 1 --2- 172.21.15.39:0/2432974132 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557f9d6b6860 0x557f9d6a5dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:36.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.805+0000 7f2cd0c22740 1 --2- 172.21.15.39:0/2432974132 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557f9d6aba10 0x557f9d6a5880 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:36.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.805+0000 7f2cd0c22740 1 -- 172.21.15.39:0/2432974132 >> 172.21.15.39:0/2432974132 conn(0x557f9d6a6f80 msgr2=0x557f9d6b6150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:36.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.805+0000 7f2cd0c22740 1 -- 172.21.15.39:0/2432974132 shutdown_connections 2024-03-18T16:10:36.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.805+0000 7f2cd0c22740 1 -- 172.21.15.39:0/2432974132 wait complete. 2024-03-18T16:10:36.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.805+0000 7f2cd0c22740 1 -- start start 2024-03-18T16:10:36.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.806+0000 7f2cd0c22740 1 Processor -- start 2024-03-18T16:10:36.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.806+0000 7f2cd0c22740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f9d6aba10 0x557f9d6c3350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:36.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.806+0000 7f2cd0c22740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557f9d6b9d80 0x557f9d6c3890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:36.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.806+0000 7f2cd0c22740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557f9d6c48a0 0x557f9d6c4d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:36.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.806+0000 7f2cd0c22740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f2cc81a95a0 con 0x557f9d6b9d80 2024-03-18T16:10:36.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.806+0000 7f2cd0c22740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f2cc81a8ed0 con 0x557f9d6aba10 2024-03-18T16:10:36.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.806+0000 7f2cd0c22740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f2cc81a6110 con 0x557f9d6c48a0 2024-03-18T16:10:36.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.806+0000 7f2cd0c22740 10 client.0 _ll_register_callbacks cb 0x7f2cb400bb30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:10:36.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.807+0000 7f2cc7400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f9d6aba10 0x557f9d6c3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:36.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.807+0000 7f2cc7400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f9d6aba10 0x557f9d6c3350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:43040/0 (socket says 172.21.15.39:43040) 2024-03-18T16:10:36.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.807+0000 7f2cc7400640 1 -- 172.21.15.39:0/3367171618 learned_addr learned my addr 172.21.15.39:0/3367171618 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:10:36.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.807+0000 7f2cc7e00640 1 --2- 172.21.15.39:0/3367171618 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557f9d6c48a0 0x557f9d6c4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:36.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.807+0000 7f2cc6a00640 1 --2- 172.21.15.39:0/3367171618 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557f9d6b9d80 0x557f9d6c3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:36.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.807+0000 7f2cc7e00640 1 -- 172.21.15.39:0/3367171618 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f9d6aba10 msgr2=0x557f9d6c3350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:36.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.807+0000 7f2cc7e00640 1 --2- 172.21.15.39:0/3367171618 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f9d6aba10 0x557f9d6c3350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:36.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.807+0000 7f2cc7e00640 1 -- 172.21.15.39:0/3367171618 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557f9d6b9d80 msgr2=0x557f9d6c3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:36.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.807+0000 7f2cc7e00640 1 --2- 172.21.15.39:0/3367171618 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557f9d6b9d80 0x557f9d6c3890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:36.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.807+0000 7f2cc7e00640 1 -- 172.21.15.39:0/3367171618 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f9d6c5260 con 0x557f9d6c48a0 2024-03-18T16:10:36.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.807+0000 7f2cc7400640 1 --2- 172.21.15.39:0/3367171618 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f9d6aba10 0x557f9d6c3350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:10:36.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.807+0000 7f2cc6a00640 1 --2- 172.21.15.39:0/3367171618 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557f9d6b9d80 0x557f9d6c3890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:10:36.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.808+0000 7f2cc7e00640 1 --2- 172.21.15.39:0/3367171618 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557f9d6c48a0 0x557f9d6c4d20 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f2cc0009fd0 tx=0x7f2cc0004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4c3b2a284fae3e00 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:36.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.808+0000 7f2cb0c00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:10:36.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.808+0000 7f2cb0c00640 1 -- 172.21.15.39:0/3367171618 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cc0004d20 con 0x557f9d6c48a0 2024-03-18T16:10:36.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.808+0000 7f2cb0c00640 1 -- 172.21.15.39:0/3367171618 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f2cc0005020 con 0x557f9d6c48a0 2024-03-18T16:10:36.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.808+0000 7f2cd0c22740 1 -- 172.21.15.39:0/3367171618 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x557f9d617a80 con 0x557f9d6c48a0 2024-03-18T16:10:36.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.809+0000 7f2cb0c00640 1 -- 172.21.15.39:0/3367171618 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f2cc0005220 con 0x557f9d6c48a0 2024-03-18T16:10:36.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.809+0000 7f2cd0c22740 10 client.25642 fetch_fsmap learned FSMap version 28 2024-03-18T16:10:36.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.809+0000 7f2cd0c22740 1 -- 172.21.15.39:0/3367171618 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x557f9d730ba0 con 0x557f9d6c48a0 2024-03-18T16:10:36.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.809+0000 7f2cb0c00640 1 -- 172.21.15.39:0/3367171618 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x557f9d617a80 con 0x557f9d6c48a0 2024-03-18T16:10:36.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.809+0000 7f2cb0c00640 1 -- 172.21.15.39:0/3367171618 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f2cc0028070 con 0x557f9d6c48a0 2024-03-18T16:10:36.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.809+0000 7f2cd0c22740 10 client.25642 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T16:10:36.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.810+0000 7f2cd0c22740 10 client.25642 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:10:36.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.810+0000 7f2cd0c22740 20 client.25642 populate_metadata read hostname 'smithi039' 2024-03-18T16:10:36.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.810+0000 7f2cd0c22740 1 --2- 172.21.15.39:0/3367171618 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x557f9d731540 0x557f9d733960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:36.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.810+0000 7f2cd0c22740 4 client.25642 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T16:10:36.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.810+0000 7f2cd0c22740 1 -- 172.21.15.39:0/3367171618 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "scrub status"}) -- 0x557f9d595400 con 0x557f9d731540 2024-03-18T16:10:36.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.810+0000 7f2cc7400640 1 --2- 172.21.15.39:0/3367171618 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x557f9d731540 0x557f9d733960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:36.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.811+0000 7f2cc7400640 1 --2- 172.21.15.39:0/3367171618 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x557f9d731540 0x557f9d733960 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb973ddd7e63fe52 server_cookie=d976278f52694d88 in_seq=0 out_seq=0 2024-03-18T16:10:36.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.811+0000 7f2cb0c00640 10 client.25642 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:10:36.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.811+0000 7f2cb0c00640 1 -- 172.21.15.39:0/3367171618 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x557f9d595400 con 0x557f9d731540 2024-03-18T16:10:36.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.811+0000 7f2cb0c00640 10 client.25642 handle_command_reply: tid=0 2024-03-18T16:10:36.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.811+0000 7f2cd0c22740 1 client.25642 shutdown 2024-03-18T16:10:36.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.811+0000 7f2cd0c22740 1 -- 172.21.15.39:0/3367171618 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557f9d6c48a0 msgr2=0x557f9d6c4d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:36.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.811+0000 7f2cd0c22740 1 --2- 172.21.15.39:0/3367171618 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557f9d6c48a0 0x557f9d6c4d20 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f2cc0009fd0 tx=0x7f2cc0004a40 comp rx=0 tx=0).stop 2024-03-18T16:10:36.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.811+0000 7f2cd0c22740 1 -- 172.21.15.39:0/3367171618 shutdown_connections 2024-03-18T16:10:36.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.811+0000 7f2cd0c22740 1 --2- 172.21.15.39:0/3367171618 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x557f9d731540 0x557f9d733960 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:36.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.811+0000 7f2cd0c22740 1 --2- 172.21.15.39:0/3367171618 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557f9d6c48a0 0x557f9d6c4d20 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:36.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.811+0000 7f2cd0c22740 1 --2- 172.21.15.39:0/3367171618 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557f9d6b9d80 0x557f9d6c3890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:36.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.811+0000 7f2cd0c22740 1 --2- 172.21.15.39:0/3367171618 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f9d6aba10 0x557f9d6c3350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:36.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.811+0000 7f2cb0c00640 0 client.25642 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:10:36.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.811+0000 7f2cd0c22740 1 -- 172.21.15.39:0/3367171618 >> 172.21.15.39:0/3367171618 conn(0x557f9d6a6f80 msgr2=0x557f9d6a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:36.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.812+0000 7f2cd0c22740 1 -- 172.21.15.39:0/3367171618 shutdown_connections 2024-03-18T16:10:36.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.812+0000 7f2cd0c22740 1 -- 172.21.15.39:0/3367171618 wait complete. 2024-03-18T16:10:36.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.812+0000 7f2cd0c22740 20 client.25642 trim_cache size 0 max 16384 2024-03-18T16:10:36.820 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-03-18T16:10:36.821 INFO:teuthology.orchestra.run.smithi039.stdout: "status": "no active scrubs running", 2024-03-18T16:10:36.821 INFO:teuthology.orchestra.run.smithi039.stdout: "scrubs": {} 2024-03-18T16:10:36.821 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-03-18T16:10:36.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.813+0000 7f2ccca00640 1 -- 172.21.15.39:0/1982092463 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f2cac0a2880 msgr2=0x7f2cac0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:36.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.813+0000 7f2ccca00640 1 --2- 172.21.15.39:0/1982092463 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f2cac0a2880 0x7f2cac0a4d40 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x557f9d6acc60 tx=0x7f2cc0009500 comp rx=0 tx=0).stop 2024-03-18T16:10:36.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.813+0000 7f2ccca00640 1 -- 172.21.15.39:0/1982092463 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2cc80741c0 msgr2=0x7f2cc8199c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:36.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.813+0000 7f2ccca00640 1 --2- 172.21.15.39:0/1982092463 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2cc80741c0 0x7f2cc8199c20 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f2cb8005ab0 tx=0x7f2cb802c760 comp rx=0 tx=0).stop 2024-03-18T16:10:36.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.813+0000 7f2ccca00640 1 -- 172.21.15.39:0/1982092463 shutdown_connections 2024-03-18T16:10:36.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.813+0000 7f2ccca00640 1 --2- 172.21.15.39:0/1982092463 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2cc80757a0 0x7f2cc819a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:36.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.813+0000 7f2ccca00640 1 --2- 172.21.15.39:0/1982092463 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f2cac0a2880 0x7f2cac0a4d40 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:36.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.813+0000 7f2ccca00640 1 --2- 172.21.15.39:0/1982092463 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2cc8109590 0x7f2cc81a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:36.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.813+0000 7f2ccca00640 1 --2- 172.21.15.39:0/1982092463 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2cc80741c0 0x7f2cc8199c20 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:36.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.813+0000 7f2ccca00640 1 -- 172.21.15.39:0/1982092463 >> 172.21.15.39:0/1982092463 conn(0x7f2cc80fcc50 msgr2=0x7f2cc8105520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:36.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.815+0000 7f2ccca00640 1 -- 172.21.15.39:0/1982092463 shutdown_connections 2024-03-18T16:10:36.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:36.815+0000 7f2ccca00640 1 -- 172.21.15.39:0/1982092463 wait complete. 2024-03-18T16:10:36.831 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:10:36 smithi039 ceph-mon[31721]: pgmap v438: 129 pgs: 129 active+clean; 656 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-03-18T16:10:36.835 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-18T16:10:36.835 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:10:36.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:10:36 smithi194 ceph-mon[36200]: pgmap v438: 129 pgs: 129 active+clean; 656 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-03-18T16:10:37.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.077+0000 7fbd5a800640 1 -- 172.21.15.39:0/4049385551 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbd540773e0 msgr2=0x7fbd54077840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:37.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.077+0000 7fbd5a800640 1 --2- 172.21.15.39:0/4049385551 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbd540773e0 0x7fbd54077840 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fbd40009900 tx=0x7fbd4002bc40 comp rx=0 tx=0).stop 2024-03-18T16:10:37.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.078+0000 7fbd5a800640 1 -- 172.21.15.39:0/4049385551 shutdown_connections 2024-03-18T16:10:37.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.078+0000 7fbd5a800640 1 --2- 172.21.15.39:0/4049385551 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbd54077d80 0x7fbd54078620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:37.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.078+0000 7fbd5a800640 1 --2- 172.21.15.39:0/4049385551 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbd540773e0 0x7fbd54077840 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:37.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.078+0000 7fbd5a800640 1 --2- 172.21.15.39:0/4049385551 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbd540761a0 0x7fbd540765c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:37.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.078+0000 7fbd5a800640 1 -- 172.21.15.39:0/4049385551 >> 172.21.15.39:0/4049385551 conn(0x7fbd540fef90 msgr2=0x7fbd541013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:37.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.078+0000 7fbd5a800640 1 -- 172.21.15.39:0/4049385551 shutdown_connections 2024-03-18T16:10:37.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.078+0000 7fbd5a800640 1 -- 172.21.15.39:0/4049385551 wait complete. 2024-03-18T16:10:37.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.079+0000 7fbd5a800640 1 Processor -- start 2024-03-18T16:10:37.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.079+0000 7fbd5a800640 1 -- start start 2024-03-18T16:10:37.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.079+0000 7fbd5a800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbd540761a0 0x7fbd5419e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:37.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.079+0000 7fbd5a800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbd540773e0 0x7fbd5419e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:37.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.079+0000 7fbd5a800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbd54077d80 0x7fbd541a5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:37.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.079+0000 7fbd5a800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fbd54113b10 con 0x7fbd540761a0 2024-03-18T16:10:37.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.079+0000 7fbd5a800640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fbd54113990 con 0x7fbd54077d80 2024-03-18T16:10:37.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.079+0000 7fbd5a800640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fbd54113c90 con 0x7fbd540773e0 2024-03-18T16:10:37.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.080+0000 7fbd53e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbd54077d80 0x7fbd541a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:37.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.080+0000 7fbd52a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbd540773e0 0x7fbd5419e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:37.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.080+0000 7fbd53e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbd54077d80 0x7fbd541a5730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:43056/0 (socket says 172.21.15.39:43056) 2024-03-18T16:10:37.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.080+0000 7fbd53e00640 1 -- 172.21.15.39:0/1955121901 learned_addr learned my addr 172.21.15.39:0/1955121901 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:10:37.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.080+0000 7fbd53400640 1 --2- 172.21.15.39:0/1955121901 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbd540761a0 0x7fbd5419e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:37.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.080+0000 7fbd53e00640 1 -- 172.21.15.39:0/1955121901 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbd540773e0 msgr2=0x7fbd5419e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:37.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.080+0000 7fbd53e00640 1 --2- 172.21.15.39:0/1955121901 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbd540773e0 0x7fbd5419e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:37.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.080+0000 7fbd53e00640 1 -- 172.21.15.39:0/1955121901 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbd540761a0 msgr2=0x7fbd5419e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:37.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.080+0000 7fbd53e00640 1 --2- 172.21.15.39:0/1955121901 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbd540761a0 0x7fbd5419e180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:37.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.080+0000 7fbd53e00640 1 -- 172.21.15.39:0/1955121901 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd54103560 con 0x7fbd54077d80 2024-03-18T16:10:37.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.080+0000 7fbd52a00640 1 --2- 172.21.15.39:0/1955121901 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbd540773e0 0x7fbd5419e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:10:37.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.080+0000 7fbd53400640 1 --2- 172.21.15.39:0/1955121901 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbd540761a0 0x7fbd5419e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:10:37.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.081+0000 7fbd53e00640 1 --2- 172.21.15.39:0/1955121901 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbd54077d80 0x7fbd541a5730 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fbd4c00eb10 tx=0x7fbd4c00c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:37.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.081+0000 7fbd4be00640 1 -- 172.21.15.39:0/1955121901 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd4c004b20 con 0x7fbd54077d80 2024-03-18T16:10:37.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.081+0000 7fbd4be00640 1 -- 172.21.15.39:0/1955121901 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd4c004cc0 con 0x7fbd54077d80 2024-03-18T16:10:37.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.081+0000 7fbd5a800640 1 -- 172.21.15.39:0/1955121901 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd54103850 con 0x7fbd54077d80 2024-03-18T16:10:37.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.082+0000 7fbd5a800640 1 -- 172.21.15.39:0/1955121901 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd54103d20 con 0x7fbd54077d80 2024-03-18T16:10:37.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.082+0000 7fbd4be00640 1 -- 172.21.15.39:0/1955121901 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd4c010760 con 0x7fbd54077d80 2024-03-18T16:10:37.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.083+0000 7fbd5a800640 1 -- 172.21.15.39:0/1955121901 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd541104a0 con 0x7fbd54077d80 2024-03-18T16:10:37.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.085+0000 7fbd4be00640 1 -- 172.21.15.39:0/1955121901 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fbd4c005020 con 0x7fbd54077d80 2024-03-18T16:10:37.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.086+0000 7fbd4be00640 1 --2- 172.21.15.39:0/1955121901 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fbd380a2880 0x7fbd380a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:37.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.086+0000 7fbd4be00640 1 -- 172.21.15.39:0/1955121901 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fbd4c0cf470 con 0x7fbd54077d80 2024-03-18T16:10:37.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.087+0000 7fbd53400640 1 --2- 172.21.15.39:0/1955121901 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fbd380a2880 0x7fbd380a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:37.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.088+0000 7fbd53400640 1 --2- 172.21.15.39:0/1955121901 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fbd380a2880 0x7fbd380a4d40 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fbd44009930 tx=0x7fbd44008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:37.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.090+0000 7fbd4be00640 1 -- 172.21.15.39:0/1955121901 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fbd4c09c120 con 0x7fbd54077d80 2024-03-18T16:10:37.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:10:36 smithi196 ceph-mon[36307]: pgmap v438: 129 pgs: 129 active+clean; 656 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s 2024-03-18T16:10:37.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.570+0000 7fbd5a800640 1 -- 172.21.15.39:0/1955121901 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbd540623e0 con 0x7fbd54077d80 2024-03-18T16:10:37.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.571+0000 7fbd4be00640 1 -- 172.21.15.39:0/1955121901 <== mon.2 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11932 (secure 0 0 0) 0x7fbd4c09ba20 con 0x7fbd54077d80 2024-03-18T16:10:37.574 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:10:37.574 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2520226608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2520226608},{"type":"v1","addr":"172.21.15.39:6835","nonce":2520226608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/719341763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":719341763},{"type":"v1","addr":"172.21.15.39:6837","nonce":719341763}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T16:06:44.180425+0000","modified":"2024-03-18T16:09:54.861629+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24575,"mds_1":24556,"mds_2":24535,"mds_3":24524,"mds_4":14463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14445":{"gid":14445,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.39:6839/656513813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":656513813},{"type":"v1","addr":"172.21.15.39:6839","nonce":656513813}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14463":{"gid":14463,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":13,"addr":"172.21.15.39:6841/3515326042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3515326042},{"type":"v1","addr":"172.21.15.39:6841","nonce":3515326042}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6833/931350738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":931350738},{"type":"v1","addr":"172.21.15.194:6833","nonce":931350738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6833/3874209621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":3874209621},{"type":"v1","addr":"172.21.15.196:6833","nonce":3874209621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6835/188385470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":188385470},{"type":"v1","addr":"172.21.15.194:6835","nonce":188385470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6835/3949984309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":3949984309},{"type":"v1","addr":"172.21.15.196:6835","nonce":3949984309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":15,"addr":"172.21.15.194:6837/1301464246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":1301464246},{"type":"v1","addr":"172.21.15.194:6837","nonce":1301464246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":16,"addr":"172.21.15.196:6837/895247251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":895247251},{"type":"v1","addr":"172.21.15.196:6837","nonce":895247251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.196:6839/2001427268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6838","nonce":2001427268},{"type":"v1","addr":"172.21.15.196:6839","nonce":2001427268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24575":{"gid":24575,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.194:6839/274911158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":274911158},{"type":"v1","addr":"172.21.15.194:6839","nonce":274911158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24575,"qdb_cluster":[14463,24524,24575,24556,24535]},"id":1}]} 2024-03-18T16:10:37.574 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 28 2024-03-18T16:10:37.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.574+0000 7fbd5a800640 1 -- 172.21.15.39:0/1955121901 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fbd380a2880 msgr2=0x7fbd380a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:37.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.574+0000 7fbd5a800640 1 --2- 172.21.15.39:0/1955121901 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fbd380a2880 0x7fbd380a4d40 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fbd44009930 tx=0x7fbd44008040 comp rx=0 tx=0).stop 2024-03-18T16:10:37.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.574+0000 7fbd5a800640 1 -- 172.21.15.39:0/1955121901 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbd54077d80 msgr2=0x7fbd541a5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:37.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.574+0000 7fbd5a800640 1 --2- 172.21.15.39:0/1955121901 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbd54077d80 0x7fbd541a5730 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fbd4c00eb10 tx=0x7fbd4c00c2d0 comp rx=0 tx=0).stop 2024-03-18T16:10:37.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.574+0000 7fbd5a800640 1 -- 172.21.15.39:0/1955121901 shutdown_connections 2024-03-18T16:10:37.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.574+0000 7fbd5a800640 1 --2- 172.21.15.39:0/1955121901 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbd54077d80 0x7fbd541a5730 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:37.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.574+0000 7fbd5a800640 1 --2- 172.21.15.39:0/1955121901 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fbd380a2880 0x7fbd380a4d40 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:37.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.574+0000 7fbd5a800640 1 --2- 172.21.15.39:0/1955121901 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbd540773e0 0x7fbd5419e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:37.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.574+0000 7fbd5a800640 1 --2- 172.21.15.39:0/1955121901 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbd540761a0 0x7fbd5419e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:37.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.574+0000 7fbd5a800640 1 -- 172.21.15.39:0/1955121901 >> 172.21.15.39:0/1955121901 conn(0x7fbd540fef90 msgr2=0x7fbd54100680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:37.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.574+0000 7fbd5a800640 1 -- 172.21.15.39:0/1955121901 shutdown_connections 2024-03-18T16:10:37.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.574+0000 7fbd5a800640 1 -- 172.21.15.39:0/1955121901 wait complete. 2024-03-18T16:10:37.603 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-18T16:10:37.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.808+0000 7f4843200640 1 -- 172.21.15.39:0/453671944 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f48440ffe40 msgr2=0x7f4844100260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:37.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.808+0000 7f4843200640 1 --2- 172.21.15.39:0/453671944 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f48440ffe40 0x7f4844100260 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f4834009900 tx=0x7f483402bc60 comp rx=0 tx=0).stop 2024-03-18T16:10:37.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.808+0000 7f4843200640 1 -- 172.21.15.39:0/453671944 shutdown_connections 2024-03-18T16:10:37.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.808+0000 7f4843200640 1 --2- 172.21.15.39:0/453671944 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4844101140 0x7f48441015c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:37.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.808+0000 7f4843200640 1 --2- 172.21.15.39:0/453671944 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f48441007a0 0x7f4844100c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:37.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.808+0000 7f4843200640 1 --2- 172.21.15.39:0/453671944 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f48440ffe40 0x7f4844100260 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:37.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.808+0000 7f4843200640 1 -- 172.21.15.39:0/453671944 >> 172.21.15.39:0/453671944 conn(0x7f48440fba80 msgr2=0x7f48440fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:37.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.808+0000 7f4843200640 1 -- 172.21.15.39:0/453671944 shutdown_connections 2024-03-18T16:10:37.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.808+0000 7f4843200640 1 -- 172.21.15.39:0/453671944 wait complete. 2024-03-18T16:10:37.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.809+0000 7f4843200640 1 Processor -- start 2024-03-18T16:10:37.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.809+0000 7f4843200640 1 -- start start 2024-03-18T16:10:37.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.810+0000 7f4843200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f48440ffe40 0x7f4844199a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:37.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.810+0000 7f4843200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f48441007a0 0x7f4844199f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:37.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.810+0000 7f4841e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f48440ffe40 0x7f4844199a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:37.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.810+0000 7f4841e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f48440ffe40 0x7f4844199a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:43080/0 (socket says 172.21.15.39:43080) 2024-03-18T16:10:37.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.810+0000 7f4841400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f48441007a0 0x7f4844199f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:37.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.810+0000 7f4841400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f48441007a0 0x7f4844199f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:45010/0 (socket says 172.21.15.39:45010) 2024-03-18T16:10:37.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.810+0000 7f4843200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4844101140 0x7f48441a0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:37.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.810+0000 7f4841e00640 1 -- 172.21.15.39:0/214071731 learned_addr learned my addr 172.21.15.39:0/214071731 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:10:37.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.810+0000 7f4843200640 1 -- 172.21.15.39:0/214071731 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f484410f460 con 0x7f4844101140 2024-03-18T16:10:37.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.810+0000 7f4843200640 1 -- 172.21.15.39:0/214071731 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f484410f2e0 con 0x7f48440ffe40 2024-03-18T16:10:37.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.810+0000 7f4843200640 1 -- 172.21.15.39:0/214071731 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f484410f5e0 con 0x7f48441007a0 2024-03-18T16:10:37.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.811+0000 7f4842800640 1 --2- 172.21.15.39:0/214071731 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4844101140 0x7f48441a0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:37.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.811+0000 7f4841e00640 1 -- 172.21.15.39:0/214071731 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f48441007a0 msgr2=0x7f4844199f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:37.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.811+0000 7f4841e00640 1 --2- 172.21.15.39:0/214071731 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f48441007a0 0x7f4844199f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:37.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.811+0000 7f4841e00640 1 -- 172.21.15.39:0/214071731 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4844101140 msgr2=0x7f48441a0fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:37.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.811+0000 7f4841e00640 1 --2- 172.21.15.39:0/214071731 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4844101140 0x7f48441a0fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:37.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.811+0000 7f4841e00640 1 -- 172.21.15.39:0/214071731 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48441a1640 con 0x7f48440ffe40 2024-03-18T16:10:37.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.811+0000 7f4842800640 1 --2- 172.21.15.39:0/214071731 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4844101140 0x7f48441a0fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:10:37.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.811+0000 7f4841e00640 1 --2- 172.21.15.39:0/214071731 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f48440ffe40 0x7f4844199a20 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f483402c270 tx=0x7f4834004e30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:37.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.812+0000 7f483aa00640 1 -- 172.21.15.39:0/214071731 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4834043020 con 0x7f48440ffe40 2024-03-18T16:10:37.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.812+0000 7f483aa00640 1 -- 172.21.15.39:0/214071731 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f48340054e0 con 0x7f48440ffe40 2024-03-18T16:10:37.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.812+0000 7f4843200640 1 -- 172.21.15.39:0/214071731 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48441a18d0 con 0x7f48440ffe40 2024-03-18T16:10:37.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.812+0000 7f483aa00640 1 -- 172.21.15.39:0/214071731 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4834003cc0 con 0x7f48440ffe40 2024-03-18T16:10:37.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.812+0000 7f4843200640 1 -- 172.21.15.39:0/214071731 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48441a1e70 con 0x7f48440ffe40 2024-03-18T16:10:37.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.815+0000 7f484b7f9740 1 Processor -- start 2024-03-18T16:10:37.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.817+0000 7f483aa00640 1 -- 172.21.15.39:0/214071731 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f4834046020 con 0x7f48440ffe40 2024-03-18T16:10:37.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.817+0000 7f484b7f9740 1 -- start start 2024-03-18T16:10:37.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.818+0000 7f484b7f9740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557c46dc82a0 0x557c46dc86c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:37.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.818+0000 7f484b7f9740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557c46dc8c90 0x557c46dcd790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:37.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.818+0000 7f4842800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557c46dc82a0 0x557c46dc86c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:37.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.818+0000 7f4842800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557c46dc82a0 0x557c46dc86c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:43088/0 (socket says 172.21.15.39:43088) 2024-03-18T16:10:37.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.818+0000 7f4841e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557c46dc8c90 0x557c46dcd790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:37.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.819+0000 7f4841e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557c46dc8c90 0x557c46dcd790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43898/0 (socket says 172.21.15.39:43898) 2024-03-18T16:10:37.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.819+0000 7f484b7f9740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557c46dcde00 0x557c46dd0290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:37.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.819+0000 7f4841e00640 1 -- 172.21.15.39:0/224862041 learned_addr learned my addr 172.21.15.39:0/224862041 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:10:37.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.819+0000 7f484b7f9740 1 -- 172.21.15.39:0/224862041 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x557c46d873b0 con 0x557c46dc8c90 2024-03-18T16:10:37.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.819+0000 7f484b7f9740 1 -- 172.21.15.39:0/224862041 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x557c46d24450 con 0x557c46dc82a0 2024-03-18T16:10:37.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.819+0000 7f484b7f9740 1 -- 172.21.15.39:0/224862041 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x557c46ccccb0 con 0x557c46dcde00 2024-03-18T16:10:37.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.819+0000 7f483aa00640 1 --2- 172.21.15.39:0/214071731 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f48280a2880 0x7f48280a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:37.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.820+0000 7f483aa00640 1 -- 172.21.15.39:0/214071731 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f48340f0ff0 con 0x7f48440ffe40 2024-03-18T16:10:37.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.820+0000 7f4841e00640 1 -- 172.21.15.39:0/224862041 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557c46dcde00 msgr2=0x557c46dd0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:37.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.820+0000 7f4841400640 1 --2- 172.21.15.39:0/224862041 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557c46dcde00 0x557c46dd0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:37.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.820+0000 7f4841e00640 1 --2- 172.21.15.39:0/224862041 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557c46dcde00 0x557c46dd0290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:37.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.820+0000 7f4841e00640 1 -- 172.21.15.39:0/224862041 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557c46dc82a0 msgr2=0x557c46dc86c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:37.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.820+0000 7f4841e00640 1 --2- 172.21.15.39:0/224862041 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557c46dc82a0 0x557c46dc86c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:37.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.820+0000 7f4841e00640 1 -- 172.21.15.39:0/224862041 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557c46cee1f0 con 0x557c46dc8c90 2024-03-18T16:10:37.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.820+0000 7f4842800640 1 --2- 172.21.15.39:0/224862041 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557c46dc82a0 0x557c46dc86c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:10:37.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.820+0000 7f4842800640 1 --2- 172.21.15.39:0/214071731 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f48280a2880 0x7f48280a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:37.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.820+0000 7f4841400640 1 --2- 172.21.15.39:0/224862041 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557c46dcde00 0x557c46dd0290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:10:37.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.821+0000 7f4841e00640 1 --2- 172.21.15.39:0/224862041 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557c46dc8c90 0x557c46dcd790 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f48340b9f90 tx=0x7f4834009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c513033db244262 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:37.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.821+0000 7f4842800640 1 --2- 172.21.15.39:0/214071731 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f48280a2880 0x7f48280a4d40 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x557c46dc94f0 tx=0x7f483c004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:37.825 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.821+0000 7f482f000640 1 -- 172.21.15.39:0/224862041 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48340b9860 con 0x557c46dc8c90 2024-03-18T16:10:37.825 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.821+0000 7f482f000640 1 -- 172.21.15.39:0/224862041 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4834004c60 con 0x557c46dc8c90 2024-03-18T16:10:37.825 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.821+0000 7f482f000640 1 -- 172.21.15.39:0/224862041 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4834003e60 con 0x557c46dc8c90 2024-03-18T16:10:37.825 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.822+0000 7f484b7f9740 1 -- 172.21.15.39:0/224862041 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557c46dc8c90 msgr2=0x557c46dcd790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:37.825 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.822+0000 7f484b7f9740 1 --2- 172.21.15.39:0/224862041 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557c46dc8c90 0x557c46dcd790 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f48340b9f90 tx=0x7f4834009260 comp rx=0 tx=0).stop 2024-03-18T16:10:37.825 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.822+0000 7f484b7f9740 1 -- 172.21.15.39:0/224862041 shutdown_connections 2024-03-18T16:10:37.825 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.822+0000 7f484b7f9740 1 --2- 172.21.15.39:0/224862041 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557c46dcde00 0x557c46dd0290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:37.825 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.822+0000 7f484b7f9740 1 --2- 172.21.15.39:0/224862041 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557c46dc8c90 0x557c46dcd790 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:37.825 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.822+0000 7f484b7f9740 1 --2- 172.21.15.39:0/224862041 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557c46dc82a0 0x557c46dc86c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:37.825 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.822+0000 7f484b7f9740 1 -- 172.21.15.39:0/224862041 >> 172.21.15.39:0/224862041 conn(0x557c46dbb000 msgr2=0x557c46dbb400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:37.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.823+0000 7f484b7f9740 1 -- 172.21.15.39:0/224862041 shutdown_connections 2024-03-18T16:10:37.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.823+0000 7f484b7f9740 1 -- 172.21.15.39:0/224862041 wait complete. 2024-03-18T16:10:37.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.823+0000 7f484b7f9740 1 -- start start 2024-03-18T16:10:37.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.823+0000 7f484b7f9740 1 Processor -- start 2024-03-18T16:10:37.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.824+0000 7f484b7f9740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557c46dc82a0 0x557c46dd21c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:37.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.824+0000 7f484b7f9740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557c46dc8c90 0x557c46dd2700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:37.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.824+0000 7f484b7f9740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557c46dcde00 0x557c46dd9740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:37.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.824+0000 7f484b7f9740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x557c46ce0110 con 0x557c46dc8c90 2024-03-18T16:10:37.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.824+0000 7f484b7f9740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x557c46d204f0 con 0x557c46dcde00 2024-03-18T16:10:37.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.824+0000 7f484b7f9740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x557c46d873b0 con 0x557c46dc82a0 2024-03-18T16:10:37.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.824+0000 7f484b7f9740 10 client.0 _ll_register_callbacks cb 0x557c46ccdb70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:10:37.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.824+0000 7f4842800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557c46dcde00 0x557c46dd9740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:37.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.824+0000 7f4842800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557c46dcde00 0x557c46dd9740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:43096/0 (socket says 172.21.15.39:43096) 2024-03-18T16:10:37.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.824+0000 7f4842800640 1 -- 172.21.15.39:0/3066664911 learned_addr learned my addr 172.21.15.39:0/3066664911 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:10:37.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.824+0000 7f4841e00640 1 --2- 172.21.15.39:0/3066664911 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557c46dc82a0 0x557c46dd21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:37.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.824+0000 7f4842800640 1 -- 172.21.15.39:0/3066664911 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557c46dc82a0 msgr2=0x557c46dd21c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:37.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.824+0000 7f4842800640 1 --2- 172.21.15.39:0/3066664911 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557c46dc82a0 0x557c46dd21c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:37.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.824+0000 7f4842800640 1 -- 172.21.15.39:0/3066664911 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557c46dc8c90 msgr2=0x557c46dd2700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:37.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.824+0000 7f4842800640 1 --2- 172.21.15.39:0/3066664911 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557c46dc8c90 0x557c46dd2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:37.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.824+0000 7f4842800640 1 -- 172.21.15.39:0/3066664911 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557c46a383f0 con 0x557c46dcde00 2024-03-18T16:10:37.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.825+0000 7f4841e00640 1 --2- 172.21.15.39:0/3066664911 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557c46dc82a0 0x557c46dd21c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:10:37.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.825+0000 7f4842800640 1 --2- 172.21.15.39:0/3066664911 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557c46dcde00 0x557c46dd9740 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f483c00f670 tx=0x7f483c004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b5b16cd7cffec600 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:37.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.825+0000 7f4823400640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:10:37.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.825+0000 7f4823400640 1 -- 172.21.15.39:0/3066664911 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f483c011a00 con 0x557c46dcde00 2024-03-18T16:10:37.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.825+0000 7f484b7f9740 1 -- 172.21.15.39:0/3066664911 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4834004c60 con 0x557c46dcde00 2024-03-18T16:10:37.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.825+0000 7f4823400640 1 -- 172.21.15.39:0/3066664911 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f483c011ba0 con 0x557c46dcde00 2024-03-18T16:10:37.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.825+0000 7f4823400640 1 -- 172.21.15.39:0/3066664911 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f483c02f050 con 0x557c46dcde00 2024-03-18T16:10:37.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.825+0000 7f484b7f9740 10 client.25640 fetch_fsmap learned FSMap version 28 2024-03-18T16:10:37.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.826+0000 7f484b7f9740 1 -- 172.21.15.39:0/3066664911 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x557c46b027d0 con 0x557c46dcde00 2024-03-18T16:10:37.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.826+0000 7f4823400640 1 -- 172.21.15.39:0/3066664911 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4834004c60 con 0x557c46dcde00 2024-03-18T16:10:37.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.826+0000 7f4823400640 1 -- 172.21.15.39:0/3066664911 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f483c01e420 con 0x557c46dcde00 2024-03-18T16:10:37.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.826+0000 7f484b7f9740 10 client.25640 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T16:10:37.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.826+0000 7f484b7f9740 10 client.25640 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-03-18T16:10:37.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.826+0000 7f484b7f9740 20 client.25640 populate_metadata read hostname 'smithi039' 2024-03-18T16:10:37.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.826+0000 7f484b7f9740 1 --2- 172.21.15.39:0/3066664911 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x557c46dde130 0x557c46de05f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:37.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.826+0000 7f484b7f9740 4 client.25640 mds_command: new command op to 14463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:10:37.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.826+0000 7f484b7f9740 1 -- 172.21.15.39:0/3066664911 --> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557c46d31990 con 0x557c46dde130 2024-03-18T16:10:37.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.826+0000 7f4841e00640 1 --2- 172.21.15.39:0/3066664911 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x557c46dde130 0x557c46de05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:37.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.827+0000 7f4841e00640 1 --2- 172.21.15.39:0/3066664911 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x557c46dde130 0x557c46de05f0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=96044eaf1184b102 server_cookie=d2b7dd6fff79bb02 in_seq=0 out_seq=0 2024-03-18T16:10:37.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.827+0000 7f4823400640 10 client.25640 ms_handle_connect on v2:172.21.15.39:6840/3515326042 2024-03-18T16:10:37.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.828+0000 7f4823400640 1 -- 172.21.15.39:0/3066664911 <== mds.4 v2:172.21.15.39:6840/3515326042 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557c46d31990 con 0x557c46dde130 2024-03-18T16:10:37.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.828+0000 7f4823400640 10 client.25640 handle_command_reply: tid=0 2024-03-18T16:10:37.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.828+0000 7f484b7f9740 1 client.25640 shutdown 2024-03-18T16:10:37.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.828+0000 7f484b7f9740 1 -- 172.21.15.39:0/3066664911 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557c46dcde00 msgr2=0x557c46dd9740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:37.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.828+0000 7f484b7f9740 1 --2- 172.21.15.39:0/3066664911 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557c46dcde00 0x557c46dd9740 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f483c00f670 tx=0x7f483c004ab0 comp rx=0 tx=0).stop 2024-03-18T16:10:37.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.829+0000 7f484b7f9740 1 -- 172.21.15.39:0/3066664911 shutdown_connections 2024-03-18T16:10:37.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.829+0000 7f484b7f9740 1 --2- 172.21.15.39:0/3066664911 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x557c46dde130 0x557c46de05f0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:37.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.829+0000 7f484b7f9740 1 --2- 172.21.15.39:0/3066664911 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557c46dcde00 0x557c46dd9740 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:37.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.829+0000 7f484b7f9740 1 --2- 172.21.15.39:0/3066664911 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557c46dc8c90 0x557c46dd2700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:37.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.829+0000 7f484b7f9740 1 --2- 172.21.15.39:0/3066664911 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557c46dc82a0 0x557c46dd21c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:37.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.829+0000 7f4823400640 0 client.25640 ms_handle_reset on v2:172.21.15.39:6840/3515326042 2024-03-18T16:10:37.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.829+0000 7f484b7f9740 1 -- 172.21.15.39:0/3066664911 >> 172.21.15.39:0/3066664911 conn(0x557c46dbb000 msgr2=0x557c46dba220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:37.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.829+0000 7f484b7f9740 1 -- 172.21.15.39:0/3066664911 shutdown_connections 2024-03-18T16:10:37.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.829+0000 7f484b7f9740 1 -- 172.21.15.39:0/3066664911 wait complete. 2024-03-18T16:10:37.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.829+0000 7f484b7f9740 20 client.25640 trim_cache size 0 max 16384 2024-03-18T16:10:37.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.844+0000 7f484b7f9740 1 Processor -- start 2024-03-18T16:10:37.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.846+0000 7f484b7f9740 1 -- start start 2024-03-18T16:10:37.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.846+0000 7f484b7f9740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557c46dc82a0 0x557c46db9990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:37.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.846+0000 7f484b7f9740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557c46dc8c90 0x557c46dd9700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:37.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.846+0000 7f484b7f9740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557c46dcde00 0x557c46db9ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:37.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.846+0000 7f484b7f9740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f48441a7fc0 con 0x557c46dc82a0 2024-03-18T16:10:37.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.846+0000 7f484b7f9740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f48441a9b00 con 0x557c46dc8c90 2024-03-18T16:10:37.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.846+0000 7f484b7f9740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f48441a8690 con 0x557c46dcde00 2024-03-18T16:10:37.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.847+0000 7f4842800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557c46dcde00 0x557c46db9ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:37.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.847+0000 7f4841400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557c46dc8c90 0x557c46dd9700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:37.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.847+0000 7f4842800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557c46dcde00 0x557c46db9ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:45042/0 (socket says 172.21.15.39:45042) 2024-03-18T16:10:37.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.847+0000 7f4842800640 1 -- 172.21.15.39:0/205117599 learned_addr learned my addr 172.21.15.39:0/205117599 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:10:37.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.847+0000 7f4841e00640 1 --2- 172.21.15.39:0/205117599 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557c46dc82a0 0x557c46db9990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:37.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.848+0000 7f4841e00640 1 -- 172.21.15.39:0/205117599 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557c46dcde00 msgr2=0x557c46db9ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:37.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.848+0000 7f4841e00640 1 --2- 172.21.15.39:0/205117599 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557c46dcde00 0x557c46db9ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:37.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.848+0000 7f4841e00640 1 -- 172.21.15.39:0/205117599 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557c46dc8c90 msgr2=0x557c46dd9700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:37.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.848+0000 7f4841e00640 1 --2- 172.21.15.39:0/205117599 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557c46dc8c90 0x557c46dd9700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:37.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.848+0000 7f4841e00640 1 -- 172.21.15.39:0/205117599 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557c46dbab40 con 0x557c46dc82a0 2024-03-18T16:10:37.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.848+0000 7f4841400640 1 --2- 172.21.15.39:0/205117599 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557c46dc8c90 0x557c46dd9700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:10:37.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.848+0000 7f4841e00640 1 --2- 172.21.15.39:0/205117599 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557c46dc82a0 0x557c46db9990 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x557c46dc9090 tx=0x7f4834003760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a098b1658eece9c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:37.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.848+0000 7f482e600640 1 -- 172.21.15.39:0/205117599 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48340d5070 con 0x557c46dc82a0 2024-03-18T16:10:37.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.849+0000 7f482e600640 1 -- 172.21.15.39:0/205117599 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4834004c60 con 0x557c46dc82a0 2024-03-18T16:10:37.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.849+0000 7f482e600640 1 -- 172.21.15.39:0/205117599 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48340bcf20 con 0x557c46dc82a0 2024-03-18T16:10:37.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.849+0000 7f484b7f9740 1 -- 172.21.15.39:0/205117599 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557c46dc82a0 msgr2=0x557c46db9990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:37.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.849+0000 7f484b7f9740 1 --2- 172.21.15.39:0/205117599 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557c46dc82a0 0x557c46db9990 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x557c46dc9090 tx=0x7f4834003760 comp rx=0 tx=0).stop 2024-03-18T16:10:37.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.850+0000 7f484b7f9740 1 -- 172.21.15.39:0/205117599 shutdown_connections 2024-03-18T16:10:37.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.850+0000 7f484b7f9740 1 --2- 172.21.15.39:0/205117599 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557c46dc8c90 0x557c46dd9700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:37.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.850+0000 7f484b7f9740 1 --2- 172.21.15.39:0/205117599 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557c46dcde00 0x557c46db9ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:37.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.850+0000 7f484b7f9740 1 --2- 172.21.15.39:0/205117599 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557c46dc82a0 0x557c46db9990 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:37.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.850+0000 7f484b7f9740 1 -- 172.21.15.39:0/205117599 >> 172.21.15.39:0/205117599 conn(0x557c46dbb000 msgr2=0x557c46dca050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:37.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.850+0000 7f484b7f9740 1 -- 172.21.15.39:0/205117599 shutdown_connections 2024-03-18T16:10:37.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.850+0000 7f484b7f9740 1 -- 172.21.15.39:0/205117599 wait complete. 2024-03-18T16:10:37.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.850+0000 7f484b7f9740 1 -- start start 2024-03-18T16:10:37.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.851+0000 7f484b7f9740 1 Processor -- start 2024-03-18T16:10:37.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.851+0000 7f484b7f9740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557c46dc82a0 0x557c46dd6ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:37.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.851+0000 7f484b7f9740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557c46dc8c90 0x557c46dd7400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:37.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.851+0000 7f484b7f9740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557c46dcde00 0x557c46dd8410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:37.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.851+0000 7f484b7f9740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f48441a9430 con 0x557c46dcde00 2024-03-18T16:10:37.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.851+0000 7f484b7f9740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f48441a8d60 con 0x557c46dc8c90 2024-03-18T16:10:37.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.851+0000 7f484b7f9740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f48441a5fa0 con 0x557c46dc82a0 2024-03-18T16:10:37.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.851+0000 7f484b7f9740 10 client.0 _ll_register_callbacks cb 0x7f483c009bf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:10:37.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.851+0000 7f4841e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557c46dc82a0 0x557c46dd6ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:37.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.852+0000 7f4841e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557c46dc82a0 0x557c46dd6ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:45044/0 (socket says 172.21.15.39:45044) 2024-03-18T16:10:37.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.852+0000 7f4841e00640 1 -- 172.21.15.39:0/4084067473 learned_addr learned my addr 172.21.15.39:0/4084067473 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:10:37.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.852+0000 7f4841400640 1 --2- 172.21.15.39:0/4084067473 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557c46dc8c90 0x557c46dd7400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:37.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.852+0000 7f4842800640 1 --2- 172.21.15.39:0/4084067473 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557c46dcde00 0x557c46dd8410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:37.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.852+0000 7f4841400640 1 -- 172.21.15.39:0/4084067473 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557c46dc82a0 msgr2=0x557c46dd6ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:37.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.852+0000 7f4841400640 1 --2- 172.21.15.39:0/4084067473 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557c46dc82a0 0x557c46dd6ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:37.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.852+0000 7f4841400640 1 -- 172.21.15.39:0/4084067473 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557c46dcde00 msgr2=0x557c46dd8410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:37.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.852+0000 7f4841400640 1 --2- 172.21.15.39:0/4084067473 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557c46dcde00 0x557c46dd8410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:37.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.852+0000 7f4841400640 1 -- 172.21.15.39:0/4084067473 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557c46dbbd70 con 0x557c46dc8c90 2024-03-18T16:10:37.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.852+0000 7f4842800640 1 --2- 172.21.15.39:0/4084067473 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557c46dcde00 0x557c46dd8410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:10:37.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.852+0000 7f4841e00640 1 --2- 172.21.15.39:0/4084067473 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557c46dc82a0 0x557c46dd6ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:10:37.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.853+0000 7f4841400640 1 --2- 172.21.15.39:0/4084067473 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557c46dc8c90 0x557c46dd7400 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f4830009730 tx=0x7f4830013be0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5e3808b1e8da4a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:37.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.853+0000 7f4823400640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:10:37.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.853+0000 7f4823400640 1 -- 172.21.15.39:0/4084067473 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4830011b00 con 0x557c46dc8c90 2024-03-18T16:10:37.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.853+0000 7f4823400640 1 -- 172.21.15.39:0/4084067473 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f4830011ca0 con 0x557c46dc8c90 2024-03-18T16:10:37.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.853+0000 7f484b7f9740 1 -- 172.21.15.39:0/4084067473 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4834004c60 con 0x557c46dc8c90 2024-03-18T16:10:37.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.853+0000 7f4823400640 1 -- 172.21.15.39:0/4084067473 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f4830029050 con 0x557c46dc8c90 2024-03-18T16:10:37.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.853+0000 7f484b7f9740 10 client.25646 fetch_fsmap learned FSMap version 28 2024-03-18T16:10:37.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.853+0000 7f484b7f9740 1 -- 172.21.15.39:0/4084067473 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x557c46e44530 con 0x557c46dc8c90 2024-03-18T16:10:37.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.853+0000 7f4823400640 1 -- 172.21.15.39:0/4084067473 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4834004c60 con 0x557c46dc8c90 2024-03-18T16:10:37.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.854+0000 7f4823400640 1 -- 172.21.15.39:0/4084067473 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f4830024030 con 0x557c46dc8c90 2024-03-18T16:10:37.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.854+0000 7f484b7f9740 10 client.25646 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T16:10:37.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.854+0000 7f484b7f9740 10 client.25646 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-03-18T16:10:37.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.854+0000 7f484b7f9740 20 client.25646 populate_metadata read hostname 'smithi039' 2024-03-18T16:10:37.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.854+0000 7f484b7f9740 1 --2- 172.21.15.39:0/4084067473 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x557c46e44db0 0x557c46e471a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:37.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.854+0000 7f484b7f9740 4 client.25646 mds_command: new command op to 14463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:10:37.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.854+0000 7f484b7f9740 1 -- 172.21.15.39:0/4084067473 --> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557c46d2bce0 con 0x557c46e44db0 2024-03-18T16:10:37.860 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.855+0000 7f4841e00640 1 --2- 172.21.15.39:0/4084067473 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x557c46e44db0 0x557c46e471a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:37.860 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.855+0000 7f4841e00640 1 --2- 172.21.15.39:0/4084067473 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x557c46e44db0 0x557c46e471a0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=140171e205b1e780 server_cookie=748099ad7bd1f466 in_seq=0 out_seq=0 2024-03-18T16:10:37.860 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.855+0000 7f4823400640 10 client.25646 ms_handle_connect on v2:172.21.15.39:6840/3515326042 2024-03-18T16:10:37.860 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.856+0000 7f4823400640 1 -- 172.21.15.39:0/4084067473 <== mds.4 v2:172.21.15.39:6840/3515326042 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557c46d2bce0 con 0x557c46e44db0 2024-03-18T16:10:37.860 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.856+0000 7f4823400640 10 client.25646 handle_command_reply: tid=0 2024-03-18T16:10:37.860 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.856+0000 7f484b7f9740 1 client.25646 shutdown 2024-03-18T16:10:37.860 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.856+0000 7f484b7f9740 1 -- 172.21.15.39:0/4084067473 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557c46dc8c90 msgr2=0x557c46dd7400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:37.860 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.856+0000 7f484b7f9740 1 --2- 172.21.15.39:0/4084067473 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557c46dc8c90 0x557c46dd7400 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f4830009730 tx=0x7f4830013be0 comp rx=0 tx=0).stop 2024-03-18T16:10:37.860 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.856+0000 7f484b7f9740 1 -- 172.21.15.39:0/4084067473 shutdown_connections 2024-03-18T16:10:37.860 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.856+0000 7f484b7f9740 1 --2- 172.21.15.39:0/4084067473 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x557c46e44db0 0x557c46e471a0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:37.860 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.856+0000 7f484b7f9740 1 --2- 172.21.15.39:0/4084067473 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557c46dc8c90 0x557c46dd7400 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:37.860 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.856+0000 7f484b7f9740 1 --2- 172.21.15.39:0/4084067473 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557c46dcde00 0x557c46dd8410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:37.860 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.856+0000 7f484b7f9740 1 --2- 172.21.15.39:0/4084067473 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557c46dc82a0 0x557c46dd6ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:37.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.856+0000 7f484b7f9740 1 -- 172.21.15.39:0/4084067473 >> 172.21.15.39:0/4084067473 conn(0x557c46dbb000 msgr2=0x557c46dbe260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:37.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.856+0000 7f4823400640 0 client.25646 ms_handle_reset on v2:172.21.15.39:6840/3515326042 2024-03-18T16:10:37.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.856+0000 7f484b7f9740 1 -- 172.21.15.39:0/4084067473 shutdown_connections 2024-03-18T16:10:37.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.857+0000 7f484b7f9740 1 -- 172.21.15.39:0/4084067473 wait complete. 2024-03-18T16:10:37.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.857+0000 7f484b7f9740 20 client.25646 trim_cache size 0 max 16384 2024-03-18T16:10:37.861 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:10:37.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.859+0000 7f4843200640 1 -- 172.21.15.39:0/214071731 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f48280a2880 msgr2=0x7f48280a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:37.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.859+0000 7f4843200640 1 --2- 172.21.15.39:0/214071731 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f48280a2880 0x7f48280a4d40 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x557c46dc94f0 tx=0x7f483c004a40 comp rx=0 tx=0).stop 2024-03-18T16:10:37.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.859+0000 7f4843200640 1 -- 172.21.15.39:0/214071731 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f48440ffe40 msgr2=0x7f4844199a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:37.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.859+0000 7f4843200640 1 --2- 172.21.15.39:0/214071731 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f48440ffe40 0x7f4844199a20 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f483402c270 tx=0x7f4834004e30 comp rx=0 tx=0).stop 2024-03-18T16:10:37.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.859+0000 7f4843200640 1 -- 172.21.15.39:0/214071731 shutdown_connections 2024-03-18T16:10:37.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.859+0000 7f4843200640 1 --2- 172.21.15.39:0/214071731 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f48280a2880 0x7f48280a4d40 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:37.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.859+0000 7f4843200640 1 --2- 172.21.15.39:0/214071731 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4844101140 0x7f48441a0fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:37.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.859+0000 7f4843200640 1 --2- 172.21.15.39:0/214071731 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f48441007a0 0x7f4844199f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:37.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.859+0000 7f4843200640 1 --2- 172.21.15.39:0/214071731 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f48440ffe40 0x7f4844199a20 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:37.863 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.859+0000 7f4843200640 1 -- 172.21.15.39:0/214071731 >> 172.21.15.39:0/214071731 conn(0x7f48440fba80 msgr2=0x7f48440fd580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:37.863 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.859+0000 7f4843200640 1 -- 172.21.15.39:0/214071731 shutdown_connections 2024-03-18T16:10:37.863 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:37.859+0000 7f4843200640 1 -- 172.21.15.39:0/214071731 wait complete. 2024-03-18T16:10:37.885 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-18T16:10:37.885 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:10:37 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/1955121901' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:10:37.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:10:37 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/1955121901' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:10:38.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.087+0000 7f232d800640 1 -- 172.21.15.39:0/1776532013 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f23281017e0 msgr2=0x7f2328101c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:38.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.087+0000 7f232d800640 1 --2- 172.21.15.39:0/1776532013 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f23281017e0 0x7f2328101c00 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f2318009900 tx=0x7f231802bc50 comp rx=0 tx=0).stop 2024-03-18T16:10:38.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.088+0000 7f232d800640 1 -- 172.21.15.39:0/1776532013 shutdown_connections 2024-03-18T16:10:38.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.088+0000 7f232d800640 1 --2- 172.21.15.39:0/1776532013 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2328103390 0x7f2328109c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.088+0000 7f232d800640 1 --2- 172.21.15.39:0/1776532013 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f23281029d0 0x7f2328102e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.088+0000 7f232d800640 1 --2- 172.21.15.39:0/1776532013 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f23281017e0 0x7f2328101c00 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.088+0000 7f232d800640 1 -- 172.21.15.39:0/1776532013 >> 172.21.15.39:0/1776532013 conn(0x7f23280fcd50 msgr2=0x7f23280ff190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:38.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.088+0000 7f232d800640 1 -- 172.21.15.39:0/1776532013 shutdown_connections 2024-03-18T16:10:38.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.088+0000 7f232d800640 1 -- 172.21.15.39:0/1776532013 wait complete. 2024-03-18T16:10:38.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.089+0000 7f232d800640 1 Processor -- start 2024-03-18T16:10:38.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.089+0000 7f232d800640 1 -- start start 2024-03-18T16:10:38.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.090+0000 7f232d800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f23281017e0 0x7f2328110c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:38.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.090+0000 7f232d800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f23281029d0 0x7f2328111180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:38.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.090+0000 7f232d800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2328103390 0x7f23281ae050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:38.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.090+0000 7f232d800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f23280757b0 con 0x7f2328103390 2024-03-18T16:10:38.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.090+0000 7f232d800640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f2328075630 con 0x7f23281029d0 2024-03-18T16:10:38.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.090+0000 7f232d800640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f2328075930 con 0x7f23281017e0 2024-03-18T16:10:38.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.090+0000 7f2326a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f23281017e0 0x7f2328110c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:38.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.090+0000 7f2326a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f23281017e0 0x7f2328110c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:34924/0 (socket says 172.21.15.39:34924) 2024-03-18T16:10:38.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.090+0000 7f2326a00640 1 -- 172.21.15.39:0/3208730227 learned_addr learned my addr 172.21.15.39:0/3208730227 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:10:38.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.090+0000 7f2327400640 1 --2- 172.21.15.39:0/3208730227 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2328103390 0x7f23281ae050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:38.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.090+0000 7f2326000640 1 --2- 172.21.15.39:0/3208730227 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f23281029d0 0x7f2328111180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:38.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.091+0000 7f2326a00640 1 -- 172.21.15.39:0/3208730227 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f23281029d0 msgr2=0x7f2328111180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:38.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.091+0000 7f2326a00640 1 --2- 172.21.15.39:0/3208730227 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f23281029d0 0x7f2328111180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.091+0000 7f2326a00640 1 -- 172.21.15.39:0/3208730227 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2328103390 msgr2=0x7f23281ae050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:38.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.091+0000 7f2326a00640 1 --2- 172.21.15.39:0/3208730227 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2328103390 0x7f23281ae050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.091+0000 7f2326a00640 1 -- 172.21.15.39:0/3208730227 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23281ae590 con 0x7f23281017e0 2024-03-18T16:10:38.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.091+0000 7f2327400640 1 --2- 172.21.15.39:0/3208730227 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2328103390 0x7f23281ae050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:10:38.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.092+0000 7f2326a00640 1 --2- 172.21.15.39:0/3208730227 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f23281017e0 0x7f2328110c40 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f2318005ab0 tx=0x7f231802c6e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:38.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.092+0000 7f231f400640 1 -- 172.21.15.39:0/3208730227 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2318041070 con 0x7f23281017e0 2024-03-18T16:10:38.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.092+0000 7f231f400640 1 -- 172.21.15.39:0/3208730227 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2318004450 con 0x7f23281017e0 2024-03-18T16:10:38.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.092+0000 7f231f400640 1 -- 172.21.15.39:0/3208730227 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2318034440 con 0x7f23281017e0 2024-03-18T16:10:38.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.092+0000 7f232d800640 1 -- 172.21.15.39:0/3208730227 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23281ae820 con 0x7f23281017e0 2024-03-18T16:10:38.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.092+0000 7f232d800640 1 -- 172.21.15.39:0/3208730227 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23281aed20 con 0x7f23281017e0 2024-03-18T16:10:38.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.098+0000 7f23305af740 1 Processor -- start 2024-03-18T16:10:38.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.098+0000 7f23305af740 1 -- start start 2024-03-18T16:10:38.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.098+0000 7f23305af740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b9e2a798c0 0x55b9e2a79ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:38.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.099+0000 7f23305af740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b9e2a7a2b0 0x55b9e2a7edb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:38.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.099+0000 7f23305af740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b9e2a7f420 0x55b9e2a818b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:38.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.099+0000 7f23305af740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55b9e2a413b0 con 0x55b9e2a7a2b0 2024-03-18T16:10:38.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.099+0000 7f23305af740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55b9e29de450 con 0x55b9e2a798c0 2024-03-18T16:10:38.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.099+0000 7f23305af740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55b9e2986cb0 con 0x55b9e2a7f420 2024-03-18T16:10:38.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.099+0000 7f2327400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b9e2a798c0 0x55b9e2a79ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:38.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.099+0000 7f2326000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b9e2a7f420 0x55b9e2a818b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:38.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.099+0000 7f2327400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b9e2a798c0 0x55b9e2a79ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:36078/0 (socket says 172.21.15.39:36078) 2024-03-18T16:10:38.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.099+0000 7f2327400640 1 -- 172.21.15.39:0/669465421 learned_addr learned my addr 172.21.15.39:0/669465421 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:10:38.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.099+0000 7f2326000640 1 -- 172.21.15.39:0/669465421 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b9e2a798c0 msgr2=0x55b9e2a79ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:38.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.099+0000 7f2326000640 1 --2- 172.21.15.39:0/669465421 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b9e2a798c0 0x55b9e2a79ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.100+0000 7f2326000640 1 -- 172.21.15.39:0/669465421 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b9e2a7a2b0 msgr2=0x55b9e2a7edb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:10:38.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.100+0000 7f2326000640 1 --2- 172.21.15.39:0/669465421 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b9e2a7a2b0 0x55b9e2a7edb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.100+0000 7f2326000640 1 -- 172.21.15.39:0/669465421 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b9e29a81f0 con 0x55b9e2a7f420 2024-03-18T16:10:38.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.100+0000 7f2327400640 1 --2- 172.21.15.39:0/669465421 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b9e2a798c0 0x55b9e2a79ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:10:38.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.100+0000 7f231f400640 1 -- 172.21.15.39:0/3208730227 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f2318044070 con 0x7f23281017e0 2024-03-18T16:10:38.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.100+0000 7f2326000640 1 --2- 172.21.15.39:0/669465421 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b9e2a7f420 0x55b9e2a818b0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f231400bbb0 tx=0x7f231400f730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1910d6aa5cb7a826 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:38.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.100+0000 7f2313e00640 1 -- 172.21.15.39:0/669465421 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2314015070 con 0x55b9e2a7f420 2024-03-18T16:10:38.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.100+0000 7f2313e00640 1 -- 172.21.15.39:0/669465421 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f23140035f0 con 0x55b9e2a7f420 2024-03-18T16:10:38.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.100+0000 7f2313e00640 1 -- 172.21.15.39:0/669465421 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23140116a0 con 0x55b9e2a7f420 2024-03-18T16:10:38.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.101+0000 7f23305af740 1 -- 172.21.15.39:0/669465421 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b9e2a7f420 msgr2=0x55b9e2a818b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:38.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.101+0000 7f23305af740 1 --2- 172.21.15.39:0/669465421 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b9e2a7f420 0x55b9e2a818b0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f231400bbb0 tx=0x7f231400f730 comp rx=0 tx=0).stop 2024-03-18T16:10:38.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.102+0000 7f23305af740 1 -- 172.21.15.39:0/669465421 shutdown_connections 2024-03-18T16:10:38.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.102+0000 7f23305af740 1 --2- 172.21.15.39:0/669465421 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b9e2a7f420 0x55b9e2a818b0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.102+0000 7f23305af740 1 --2- 172.21.15.39:0/669465421 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b9e2a7a2b0 0x55b9e2a7edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.102+0000 7f23305af740 1 --2- 172.21.15.39:0/669465421 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b9e2a798c0 0x55b9e2a79ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.102+0000 7f23305af740 1 -- 172.21.15.39:0/669465421 >> 172.21.15.39:0/669465421 conn(0x55b9e2a74e30 msgr2=0x55b9e2a75230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:38.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.102+0000 7f231f400640 1 --2- 172.21.15.39:0/3208730227 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f230c0a2900 0x7f230c0a4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:38.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.102+0000 7f231f400640 1 -- 172.21.15.39:0/3208730227 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f23180f05c0 con 0x7f23281017e0 2024-03-18T16:10:38.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.102+0000 7f2326000640 1 --2- 172.21.15.39:0/3208730227 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f230c0a2900 0x7f230c0a4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:38.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.103+0000 7f23305af740 1 -- 172.21.15.39:0/669465421 shutdown_connections 2024-03-18T16:10:38.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.103+0000 7f23305af740 1 -- 172.21.15.39:0/669465421 wait complete. 2024-03-18T16:10:38.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.104+0000 7f2326000640 1 --2- 172.21.15.39:0/3208730227 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f230c0a2900 0x7f230c0a4dc0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f2314003ef0 tx=0x7f2314003f50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:38.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.104+0000 7f23305af740 1 -- start start 2024-03-18T16:10:38.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.104+0000 7f23305af740 1 Processor -- start 2024-03-18T16:10:38.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.105+0000 7f23305af740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b9e2a798c0 0x55b9e2a839b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:38.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.105+0000 7f23305af740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b9e2a7a2b0 0x55b9e2a83ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:38.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.105+0000 7f23305af740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b9e2a7f420 0x55b9e2a8af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:38.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.105+0000 7f23305af740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55b9e299a110 con 0x55b9e2a798c0 2024-03-18T16:10:38.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.105+0000 7f23305af740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55b9e29da4f0 con 0x55b9e2a7f420 2024-03-18T16:10:38.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.105+0000 7f23305af740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55b9e2a823d0 con 0x55b9e2a7a2b0 2024-03-18T16:10:38.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.105+0000 7f23305af740 10 client.0 _ll_register_callbacks cb 0x55b9e2987b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:10:38.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.105+0000 7f2326000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b9e2a7a2b0 0x55b9e2a83ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:38.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.105+0000 7f2326000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b9e2a7a2b0 0x55b9e2a83ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:34948/0 (socket says 172.21.15.39:34948) 2024-03-18T16:10:38.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.105+0000 7f2326000640 1 -- 172.21.15.39:0/624580815 learned_addr learned my addr 172.21.15.39:0/624580815 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:10:38.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.105+0000 7f2326a00640 1 --2- 172.21.15.39:0/624580815 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b9e2a798c0 0x55b9e2a839b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:38.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.105+0000 7f2327400640 1 --2- 172.21.15.39:0/624580815 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b9e2a7f420 0x55b9e2a8af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:38.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.105+0000 7f2326000640 1 -- 172.21.15.39:0/624580815 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b9e2a7f420 msgr2=0x55b9e2a8af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:38.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.105+0000 7f2326000640 1 --2- 172.21.15.39:0/624580815 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b9e2a7f420 0x55b9e2a8af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.105+0000 7f2326000640 1 -- 172.21.15.39:0/624580815 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b9e2a798c0 msgr2=0x55b9e2a839b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:38.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.105+0000 7f2326000640 1 --2- 172.21.15.39:0/624580815 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b9e2a798c0 0x55b9e2a839b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.105+0000 7f2326000640 1 -- 172.21.15.39:0/624580815 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b9e26f23f0 con 0x55b9e2a7a2b0 2024-03-18T16:10:38.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.106+0000 7f2326000640 1 --2- 172.21.15.39:0/624580815 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b9e2a7a2b0 0x55b9e2a83ef0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f2314003a20 tx=0x7f231400f730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d745a2c2dc84d3c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:38.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.106+0000 7f2310c00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:10:38.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.106+0000 7f2310c00640 1 -- 172.21.15.39:0/624580815 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2314015040 con 0x55b9e2a7a2b0 2024-03-18T16:10:38.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.106+0000 7f2310c00640 1 -- 172.21.15.39:0/624580815 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f23140043d0 con 0x55b9e2a7a2b0 2024-03-18T16:10:38.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.106+0000 7f23305af740 1 -- 172.21.15.39:0/624580815 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f23140035f0 con 0x55b9e2a7a2b0 2024-03-18T16:10:38.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.107+0000 7f2310c00640 1 -- 172.21.15.39:0/624580815 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f2314011ba0 con 0x55b9e2a7a2b0 2024-03-18T16:10:38.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.107+0000 7f23305af740 10 client.25675 fetch_fsmap learned FSMap version 28 2024-03-18T16:10:38.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.107+0000 7f23305af740 1 -- 172.21.15.39:0/624580815 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b9e27bc7d0 con 0x55b9e2a7a2b0 2024-03-18T16:10:38.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.107+0000 7f2310c00640 1 -- 172.21.15.39:0/624580815 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f23140035f0 con 0x55b9e2a7a2b0 2024-03-18T16:10:38.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.108+0000 7f2310c00640 1 -- 172.21.15.39:0/624580815 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f231403c020 con 0x55b9e2a7a2b0 2024-03-18T16:10:38.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.109+0000 7f23305af740 10 client.25675 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T16:10:38.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.109+0000 7f23305af740 10 client.25675 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-18T16:10:38.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.109+0000 7f23305af740 20 client.25675 populate_metadata read hostname 'smithi039' 2024-03-18T16:10:38.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.109+0000 7f23305af740 1 --2- 172.21.15.39:0/624580815 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x55b9e2a8f750 0x55b9e2a91b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:38.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.109+0000 7f23305af740 4 client.25675 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:10:38.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.109+0000 7f23305af740 1 -- 172.21.15.39:0/624580815 --> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b9e29eb990 con 0x55b9e2a8f750 2024-03-18T16:10:38.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.110+0000 7f2326a00640 1 --2- 172.21.15.39:0/624580815 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x55b9e2a8f750 0x55b9e2a91b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:38.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.110+0000 7f2326a00640 1 --2- 172.21.15.39:0/624580815 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x55b9e2a8f750 0x55b9e2a91b80 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=93d9040b49cf7106 server_cookie=2c1650e6a9da06a6 in_seq=0 out_seq=0 2024-03-18T16:10:38.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.111+0000 7f2310c00640 10 client.25675 ms_handle_connect on v2:172.21.15.194:6836/1301464246 2024-03-18T16:10:38.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.111+0000 7f2310c00640 1 -- 172.21.15.39:0/624580815 <== mds.3 v2:172.21.15.194:6836/1301464246 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b9e29eb990 con 0x55b9e2a8f750 2024-03-18T16:10:38.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.111+0000 7f2310c00640 10 client.25675 handle_command_reply: tid=0 2024-03-18T16:10:38.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.111+0000 7f23305af740 1 client.25675 shutdown 2024-03-18T16:10:38.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.112+0000 7f23305af740 1 -- 172.21.15.39:0/624580815 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b9e2a7a2b0 msgr2=0x55b9e2a83ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:38.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.112+0000 7f23305af740 1 --2- 172.21.15.39:0/624580815 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b9e2a7a2b0 0x55b9e2a83ef0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f2314003a20 tx=0x7f231400f730 comp rx=0 tx=0).stop 2024-03-18T16:10:38.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.112+0000 7f23305af740 1 -- 172.21.15.39:0/624580815 shutdown_connections 2024-03-18T16:10:38.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.112+0000 7f23305af740 1 --2- 172.21.15.39:0/624580815 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x55b9e2a8f750 0x55b9e2a91b80 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.112+0000 7f23305af740 1 --2- 172.21.15.39:0/624580815 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b9e2a7f420 0x55b9e2a8af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.112+0000 7f23305af740 1 --2- 172.21.15.39:0/624580815 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b9e2a7a2b0 0x55b9e2a83ef0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.112+0000 7f23305af740 1 --2- 172.21.15.39:0/624580815 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b9e2a798c0 0x55b9e2a839b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.112+0000 7f23305af740 1 -- 172.21.15.39:0/624580815 >> 172.21.15.39:0/624580815 conn(0x55b9e2a74e30 msgr2=0x55b9e2a741f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:38.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.112+0000 7f2310c00640 0 client.25675 ms_handle_reset on v2:172.21.15.194:6836/1301464246 2024-03-18T16:10:38.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.112+0000 7f23305af740 1 -- 172.21.15.39:0/624580815 shutdown_connections 2024-03-18T16:10:38.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.112+0000 7f23305af740 1 -- 172.21.15.39:0/624580815 wait complete. 2024-03-18T16:10:38.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.112+0000 7f23305af740 20 client.25675 trim_cache size 0 max 16384 2024-03-18T16:10:38.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.127+0000 7f23305af740 1 Processor -- start 2024-03-18T16:10:38.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.127+0000 7f23305af740 1 -- start start 2024-03-18T16:10:38.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.127+0000 7f23305af740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b9e2a798c0 0x55b9e2a97450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:38.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.127+0000 7f23305af740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b9e2a7a2b0 0x55b9e2a8aef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:38.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.127+0000 7f23305af740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b9e2a7f420 0x55b9e2a7b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:38.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.127+0000 7f23305af740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f23281b92a0 con 0x55b9e2a7f420 2024-03-18T16:10:38.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.127+0000 7f23305af740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f23281bade0 con 0x55b9e2a7a2b0 2024-03-18T16:10:38.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.127+0000 7f23305af740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f23281b9970 con 0x55b9e2a798c0 2024-03-18T16:10:38.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.128+0000 7f2327400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b9e2a7f420 0x55b9e2a7b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:38.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.128+0000 7f2327400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b9e2a7f420 0x55b9e2a7b4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:50300/0 (socket says 172.21.15.39:50300) 2024-03-18T16:10:38.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.128+0000 7f2327400640 1 -- 172.21.15.39:0/952840217 learned_addr learned my addr 172.21.15.39:0/952840217 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:10:38.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.128+0000 7f2326a00640 1 --2- 172.21.15.39:0/952840217 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b9e2a798c0 0x55b9e2a97450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:38.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.128+0000 7f2326000640 1 --2- 172.21.15.39:0/952840217 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b9e2a7a2b0 0x55b9e2a8aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:38.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.128+0000 7f2326a00640 1 -- 172.21.15.39:0/952840217 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b9e2a7a2b0 msgr2=0x55b9e2a8aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:38.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.128+0000 7f2326a00640 1 --2- 172.21.15.39:0/952840217 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b9e2a7a2b0 0x55b9e2a8aef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.128+0000 7f2326a00640 1 -- 172.21.15.39:0/952840217 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b9e2a7f420 msgr2=0x55b9e2a7b4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:38.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.128+0000 7f2326a00640 1 --2- 172.21.15.39:0/952840217 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b9e2a7f420 0x55b9e2a7b4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.128+0000 7f2326a00640 1 -- 172.21.15.39:0/952840217 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b9e2a7e770 con 0x55b9e2a798c0 2024-03-18T16:10:38.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.129+0000 7f2326a00640 1 --2- 172.21.15.39:0/952840217 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b9e2a798c0 0x55b9e2a97450 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x55b9e2a7ad20 tx=0x7f23180035f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2bb9557afa5156b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:38.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.129+0000 7f2313400640 1 -- 172.21.15.39:0/952840217 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23180c0cc0 con 0x55b9e2a798c0 2024-03-18T16:10:38.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.129+0000 7f2313400640 1 -- 172.21.15.39:0/952840217 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f23180c5310 con 0x55b9e2a798c0 2024-03-18T16:10:38.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.129+0000 7f2313400640 1 -- 172.21.15.39:0/952840217 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2318003680 con 0x55b9e2a798c0 2024-03-18T16:10:38.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.130+0000 7f23305af740 1 -- 172.21.15.39:0/952840217 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b9e2a798c0 msgr2=0x55b9e2a97450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:38.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.130+0000 7f23305af740 1 --2- 172.21.15.39:0/952840217 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b9e2a798c0 0x55b9e2a97450 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x55b9e2a7ad20 tx=0x7f23180035f0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.131+0000 7f23305af740 1 -- 172.21.15.39:0/952840217 shutdown_connections 2024-03-18T16:10:38.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.131+0000 7f23305af740 1 --2- 172.21.15.39:0/952840217 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b9e2a7a2b0 0x55b9e2a8aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.131+0000 7f23305af740 1 --2- 172.21.15.39:0/952840217 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b9e2a7f420 0x55b9e2a7b4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.131+0000 7f23305af740 1 --2- 172.21.15.39:0/952840217 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b9e2a798c0 0x55b9e2a97450 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.131+0000 7f23305af740 1 -- 172.21.15.39:0/952840217 >> 172.21.15.39:0/952840217 conn(0x55b9e2a74e30 msgr2=0x55b9e2a839b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:38.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.131+0000 7f23305af740 1 -- 172.21.15.39:0/952840217 shutdown_connections 2024-03-18T16:10:38.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.131+0000 7f23305af740 1 -- 172.21.15.39:0/952840217 wait complete. 2024-03-18T16:10:38.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.132+0000 7f23305af740 1 -- start start 2024-03-18T16:10:38.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.132+0000 7f23305af740 1 Processor -- start 2024-03-18T16:10:38.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.133+0000 7f23305af740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b9e2a798c0 0x55b9e2a88190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:38.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.133+0000 7f23305af740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b9e2a7a2b0 0x55b9e2a886d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:38.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.134+0000 7f23305af740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b9e2a7f420 0x55b9e2a89680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:38.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.134+0000 7f23305af740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f23281ba710 con 0x55b9e2a7a2b0 2024-03-18T16:10:38.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.134+0000 7f23305af740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f23281ba040 con 0x55b9e2a7f420 2024-03-18T16:10:38.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.134+0000 7f23305af740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f23281b7280 con 0x55b9e2a798c0 2024-03-18T16:10:38.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.134+0000 7f23305af740 10 client.0 _ll_register_callbacks cb 0x7f2320009aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:10:38.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.134+0000 7f2326000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b9e2a7a2b0 0x55b9e2a886d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:38.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.134+0000 7f2326a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b9e2a798c0 0x55b9e2a88190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:38.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.134+0000 7f2326a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b9e2a798c0 0x55b9e2a88190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:34954/0 (socket says 172.21.15.39:34954) 2024-03-18T16:10:38.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.134+0000 7f2326a00640 1 -- 172.21.15.39:0/1311290356 learned_addr learned my addr 172.21.15.39:0/1311290356 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:10:38.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.134+0000 7f2327400640 1 --2- 172.21.15.39:0/1311290356 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b9e2a7f420 0x55b9e2a89680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:38.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.134+0000 7f2327400640 1 -- 172.21.15.39:0/1311290356 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b9e2a798c0 msgr2=0x55b9e2a88190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:38.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.135+0000 7f2327400640 1 --2- 172.21.15.39:0/1311290356 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b9e2a798c0 0x55b9e2a88190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.135+0000 7f2327400640 1 -- 172.21.15.39:0/1311290356 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b9e2a7a2b0 msgr2=0x55b9e2a886d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:38.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.135+0000 7f2327400640 1 --2- 172.21.15.39:0/1311290356 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b9e2a7a2b0 0x55b9e2a886d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.135+0000 7f2327400640 1 -- 172.21.15.39:0/1311290356 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b9e2a89cc0 con 0x55b9e2a7f420 2024-03-18T16:10:38.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.135+0000 7f2326a00640 1 --2- 172.21.15.39:0/1311290356 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b9e2a798c0 0x55b9e2a88190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:10:38.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.135+0000 7f2326000640 1 --2- 172.21.15.39:0/1311290356 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b9e2a7a2b0 0x55b9e2a886d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:10:38.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.135+0000 7f2327400640 1 --2- 172.21.15.39:0/1311290356 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b9e2a7f420 0x55b9e2a89680 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f2320005fd0 tx=0x7f2320010800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=94030a87d74706ae server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:38.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.135+0000 7f2310c00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:10:38.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.135+0000 7f2310c00640 1 -- 172.21.15.39:0/1311290356 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2320004b20 con 0x55b9e2a7f420 2024-03-18T16:10:38.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.135+0000 7f2310c00640 1 -- 172.21.15.39:0/1311290356 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f2320004cc0 con 0x55b9e2a7f420 2024-03-18T16:10:38.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.136+0000 7f23305af740 1 -- 172.21.15.39:0/1311290356 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f23180c5310 con 0x55b9e2a7f420 2024-03-18T16:10:38.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.136+0000 7f2310c00640 1 -- 172.21.15.39:0/1311290356 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f232000f5d0 con 0x55b9e2a7f420 2024-03-18T16:10:38.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.136+0000 7f23305af740 10 client.25664 fetch_fsmap learned FSMap version 28 2024-03-18T16:10:38.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.136+0000 7f23305af740 1 -- 172.21.15.39:0/1311290356 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b9e2a8b050 con 0x55b9e2a7f420 2024-03-18T16:10:38.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.137+0000 7f2310c00640 1 -- 172.21.15.39:0/1311290356 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f23180c5310 con 0x55b9e2a7f420 2024-03-18T16:10:38.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.137+0000 7f2310c00640 1 -- 172.21.15.39:0/1311290356 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f232001c930 con 0x55b9e2a7f420 2024-03-18T16:10:38.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.137+0000 7f23305af740 10 client.25664 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T16:10:38.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.137+0000 7f23305af740 10 client.25664 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-18T16:10:38.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.137+0000 7f23305af740 20 client.25664 populate_metadata read hostname 'smithi039' 2024-03-18T16:10:38.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.137+0000 7f23305af740 1 --2- 172.21.15.39:0/1311290356 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x55b9e2af6050 0x55b9e2af8470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:38.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.137+0000 7f23305af740 4 client.25664 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:10:38.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.137+0000 7f23305af740 1 -- 172.21.15.39:0/1311290356 --> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b9e29e5ce0 con 0x55b9e2af6050 2024-03-18T16:10:38.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.138+0000 7f2326a00640 1 --2- 172.21.15.39:0/1311290356 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x55b9e2af6050 0x55b9e2af8470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:38.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.138+0000 7f2326a00640 1 --2- 172.21.15.39:0/1311290356 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x55b9e2af6050 0x55b9e2af8470 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=92368438296d41b5 server_cookie=e4129e527469b76e in_seq=0 out_seq=0 2024-03-18T16:10:38.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.138+0000 7f2310c00640 10 client.25664 ms_handle_connect on v2:172.21.15.194:6836/1301464246 2024-03-18T16:10:38.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.138+0000 7f2310c00640 1 -- 172.21.15.39:0/1311290356 <== mds.3 v2:172.21.15.194:6836/1301464246 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b9e29e5ce0 con 0x55b9e2af6050 2024-03-18T16:10:38.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.138+0000 7f2310c00640 10 client.25664 handle_command_reply: tid=0 2024-03-18T16:10:38.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.138+0000 7f23305af740 1 client.25664 shutdown 2024-03-18T16:10:38.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.139+0000 7f23305af740 1 -- 172.21.15.39:0/1311290356 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b9e2a7f420 msgr2=0x55b9e2a89680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:38.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.139+0000 7f23305af740 1 --2- 172.21.15.39:0/1311290356 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b9e2a7f420 0x55b9e2a89680 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f2320005fd0 tx=0x7f2320010800 comp rx=0 tx=0).stop 2024-03-18T16:10:38.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.139+0000 7f23305af740 1 -- 172.21.15.39:0/1311290356 shutdown_connections 2024-03-18T16:10:38.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.139+0000 7f23305af740 1 --2- 172.21.15.39:0/1311290356 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x55b9e2af6050 0x55b9e2af8470 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.139+0000 7f2310c00640 0 client.25664 ms_handle_reset on v2:172.21.15.194:6836/1301464246 2024-03-18T16:10:38.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.139+0000 7f23305af740 1 --2- 172.21.15.39:0/1311290356 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b9e2a7f420 0x55b9e2a89680 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.139+0000 7f23305af740 1 --2- 172.21.15.39:0/1311290356 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b9e2a7a2b0 0x55b9e2a886d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.139+0000 7f23305af740 1 --2- 172.21.15.39:0/1311290356 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b9e2a798c0 0x55b9e2a88190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.139+0000 7f23305af740 1 -- 172.21.15.39:0/1311290356 >> 172.21.15.39:0/1311290356 conn(0x55b9e2a74e30 msgr2=0x55b9e2a83950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:38.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.140+0000 7f23305af740 1 -- 172.21.15.39:0/1311290356 shutdown_connections 2024-03-18T16:10:38.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.140+0000 7f23305af740 1 -- 172.21.15.39:0/1311290356 wait complete. 2024-03-18T16:10:38.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.140+0000 7f23305af740 20 client.25664 trim_cache size 0 max 16384 2024-03-18T16:10:38.144 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:10:38.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.142+0000 7f232d800640 1 -- 172.21.15.39:0/3208730227 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f230c0a2900 msgr2=0x7f230c0a4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:38.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.142+0000 7f232d800640 1 --2- 172.21.15.39:0/3208730227 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f230c0a2900 0x7f230c0a4dc0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f2314003ef0 tx=0x7f2314003f50 comp rx=0 tx=0).stop 2024-03-18T16:10:38.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.142+0000 7f232d800640 1 -- 172.21.15.39:0/3208730227 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f23281017e0 msgr2=0x7f2328110c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:38.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.142+0000 7f232d800640 1 --2- 172.21.15.39:0/3208730227 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f23281017e0 0x7f2328110c40 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f2318005ab0 tx=0x7f231802c6e0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.142+0000 7f232d800640 1 -- 172.21.15.39:0/3208730227 shutdown_connections 2024-03-18T16:10:38.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.142+0000 7f232d800640 1 --2- 172.21.15.39:0/3208730227 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f23281029d0 0x7f2328111180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.142+0000 7f232d800640 1 --2- 172.21.15.39:0/3208730227 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f230c0a2900 0x7f230c0a4dc0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.142+0000 7f232d800640 1 --2- 172.21.15.39:0/3208730227 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2328103390 0x7f23281ae050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.142+0000 7f232d800640 1 --2- 172.21.15.39:0/3208730227 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f23281017e0 0x7f2328110c40 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.142+0000 7f232d800640 1 -- 172.21.15.39:0/3208730227 >> 172.21.15.39:0/3208730227 conn(0x7f23280fcd50 msgr2=0x7f2328105cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:38.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.142+0000 7f232d800640 1 -- 172.21.15.39:0/3208730227 shutdown_connections 2024-03-18T16:10:38.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.143+0000 7f232d800640 1 -- 172.21.15.39:0/3208730227 wait complete. 2024-03-18T16:10:38.171 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-18T16:10:38.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:10:37 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/1955121901' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:10:38.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.391+0000 7f9944200640 1 -- 172.21.15.39:0/2799350969 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f993c0757c0 msgr2=0x7f993c073c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:38.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.391+0000 7f9944200640 1 --2- 172.21.15.39:0/2799350969 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f993c0757c0 0x7f993c073c10 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f9930009900 tx=0x7f993002bce0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.391+0000 7f9944200640 1 -- 172.21.15.39:0/2799350969 shutdown_connections 2024-03-18T16:10:38.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.391+0000 7f9944200640 1 --2- 172.21.15.39:0/2799350969 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f993c109670 0x7f993c10bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.391+0000 7f9944200640 1 --2- 172.21.15.39:0/2799350969 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f993c0741e0 0x7f993c074640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.391+0000 7f9944200640 1 --2- 172.21.15.39:0/2799350969 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f993c0757c0 0x7f993c073c10 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.391+0000 7f9944200640 1 -- 172.21.15.39:0/2799350969 >> 172.21.15.39:0/2799350969 conn(0x7f993c0fcd50 msgr2=0x7f993c0ff190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:38.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.391+0000 7f9944200640 1 -- 172.21.15.39:0/2799350969 shutdown_connections 2024-03-18T16:10:38.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.392+0000 7f9944200640 1 -- 172.21.15.39:0/2799350969 wait complete. 2024-03-18T16:10:38.398 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.393+0000 7f9944200640 1 Processor -- start 2024-03-18T16:10:38.398 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.393+0000 7f9944200640 1 -- start start 2024-03-18T16:10:38.398 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.393+0000 7f9944200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f993c0741e0 0x7f993c104b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:38.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.393+0000 7f9944200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f993c0757c0 0x7f993c1050a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:38.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.394+0000 7f9944200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f993c109670 0x7f993c1016d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:38.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.394+0000 7f9944200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f993c10f720 con 0x7f993c109670 2024-03-18T16:10:38.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.394+0000 7f9944200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f993c10f5a0 con 0x7f993c0741e0 2024-03-18T16:10:38.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.394+0000 7f9944200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f993c10f8a0 con 0x7f993c0757c0 2024-03-18T16:10:38.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.394+0000 7f9941800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f993c0741e0 0x7f993c104b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:38.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.394+0000 7f9941800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f993c0741e0 0x7f993c104b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:36128/0 (socket says 172.21.15.39:36128) 2024-03-18T16:10:38.400 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.394+0000 7f9941800640 1 -- 172.21.15.39:0/2817864164 learned_addr learned my addr 172.21.15.39:0/2817864164 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:10:38.400 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.394+0000 7f9940e00640 1 --2- 172.21.15.39:0/2817864164 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f993c0757c0 0x7f993c1050a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:38.400 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.394+0000 7f9942200640 1 --2- 172.21.15.39:0/2817864164 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f993c109670 0x7f993c1016d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:38.400 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.395+0000 7f9940e00640 1 -- 172.21.15.39:0/2817864164 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f993c0741e0 msgr2=0x7f993c104b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:38.400 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.395+0000 7f9940e00640 1 --2- 172.21.15.39:0/2817864164 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f993c0741e0 0x7f993c104b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.400 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.395+0000 7f9940e00640 1 -- 172.21.15.39:0/2817864164 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f993c109670 msgr2=0x7f993c1016d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:38.400 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.395+0000 7f9940e00640 1 --2- 172.21.15.39:0/2817864164 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f993c109670 0x7f993c1016d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.400 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.395+0000 7f9940e00640 1 -- 172.21.15.39:0/2817864164 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f993c101f90 con 0x7f993c0757c0 2024-03-18T16:10:38.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.395+0000 7f9940e00640 1 --2- 172.21.15.39:0/2817864164 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f993c0757c0 0x7f993c1050a0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f992800e9f0 tx=0x7f992800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:38.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.395+0000 7f9942200640 1 --2- 172.21.15.39:0/2817864164 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f993c109670 0x7f993c1016d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:10:38.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.395+0000 7f993a000640 1 -- 172.21.15.39:0/2817864164 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f992800ccc0 con 0x7f993c0757c0 2024-03-18T16:10:38.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.396+0000 7f993a000640 1 -- 172.21.15.39:0/2817864164 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9928010910 con 0x7f993c0757c0 2024-03-18T16:10:38.403 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.396+0000 7f993a000640 1 -- 172.21.15.39:0/2817864164 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99280188e0 con 0x7f993c0757c0 2024-03-18T16:10:38.403 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.397+0000 7f9944200640 1 -- 172.21.15.39:0/2817864164 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f993c199930 con 0x7f993c0757c0 2024-03-18T16:10:38.403 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.397+0000 7f9944200640 1 -- 172.21.15.39:0/2817864164 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f993c199e00 con 0x7f993c0757c0 2024-03-18T16:10:38.403 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.400+0000 7f9946fc9740 1 Processor -- start 2024-03-18T16:10:38.403 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.400+0000 7f9946fc9740 1 -- start start 2024-03-18T16:10:38.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.400+0000 7f9946fc9740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555d482f38c0 0x555d482f3ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:38.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.400+0000 7f9946fc9740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555d482f42b0 0x555d482f8db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:38.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.400+0000 7f9946fc9740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555d482f9420 0x555d482fb8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:38.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.400+0000 7f9946fc9740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x555d482bb3b0 con 0x555d482f38c0 2024-03-18T16:10:38.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.400+0000 7f9946fc9740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x555d48258450 con 0x555d482f9420 2024-03-18T16:10:38.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.400+0000 7f9946fc9740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x555d48200cb0 con 0x555d482f42b0 2024-03-18T16:10:38.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.401+0000 7f9942200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555d482f38c0 0x555d482f3ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:38.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.401+0000 7f9942200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555d482f38c0 0x555d482f3ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:50348/0 (socket says 172.21.15.39:50348) 2024-03-18T16:10:38.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.401+0000 7f9942200640 1 -- 172.21.15.39:0/903922383 learned_addr learned my addr 172.21.15.39:0/903922383 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:10:38.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.401+0000 7f9941800640 1 --2- 172.21.15.39:0/903922383 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555d482f42b0 0x555d482f8db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:38.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.402+0000 7f9942200640 1 -- 172.21.15.39:0/903922383 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555d482f42b0 msgr2=0x555d482f8db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:38.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.402+0000 7f9942200640 1 --2- 172.21.15.39:0/903922383 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555d482f42b0 0x555d482f8db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.402+0000 7f9942200640 1 -- 172.21.15.39:0/903922383 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555d482f9420 msgr2=0x555d482fb8b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:10:38.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.402+0000 7f9942200640 1 --2- 172.21.15.39:0/903922383 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555d482f9420 0x555d482fb8b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.402+0000 7f9942200640 1 -- 172.21.15.39:0/903922383 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555d482221f0 con 0x555d482f38c0 2024-03-18T16:10:38.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.402+0000 7f9941800640 1 --2- 172.21.15.39:0/903922383 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555d482f42b0 0x555d482f8db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:10:38.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.402+0000 7f9942200640 1 --2- 172.21.15.39:0/903922383 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555d482f38c0 0x555d482f3ce0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f993400c450 tx=0x7f993400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c70c190cba63a46 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:38.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.402+0000 7f993a000640 1 -- 172.21.15.39:0/2817864164 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f9928010ac0 con 0x7f993c0757c0 2024-03-18T16:10:38.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.402+0000 7f992ea00640 1 -- 172.21.15.39:0/903922383 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9934007430 con 0x555d482f38c0 2024-03-18T16:10:38.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.403+0000 7f992ea00640 1 -- 172.21.15.39:0/903922383 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9934007ab0 con 0x555d482f38c0 2024-03-18T16:10:38.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.403+0000 7f992ea00640 1 -- 172.21.15.39:0/903922383 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f993400abe0 con 0x555d482f38c0 2024-03-18T16:10:38.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.403+0000 7f9946fc9740 1 -- 172.21.15.39:0/903922383 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555d482f38c0 msgr2=0x555d482f3ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:38.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.403+0000 7f9946fc9740 1 --2- 172.21.15.39:0/903922383 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555d482f38c0 0x555d482f3ce0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f993400c450 tx=0x7f993400c910 comp rx=0 tx=0).stop 2024-03-18T16:10:38.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.404+0000 7f9946fc9740 1 -- 172.21.15.39:0/903922383 shutdown_connections 2024-03-18T16:10:38.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.404+0000 7f9946fc9740 1 --2- 172.21.15.39:0/903922383 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555d482f9420 0x555d482fb8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.404+0000 7f9946fc9740 1 --2- 172.21.15.39:0/903922383 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555d482f42b0 0x555d482f8db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.404+0000 7f9946fc9740 1 --2- 172.21.15.39:0/903922383 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555d482f38c0 0x555d482f3ce0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.404+0000 7f9946fc9740 1 -- 172.21.15.39:0/903922383 >> 172.21.15.39:0/903922383 conn(0x555d482eee30 msgr2=0x555d482ef230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:38.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.405+0000 7f9946fc9740 1 -- 172.21.15.39:0/903922383 shutdown_connections 2024-03-18T16:10:38.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.405+0000 7f993a000640 1 --2- 172.21.15.39:0/2817864164 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f99240a2980 0x7f99240a4e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:38.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.405+0000 7f9946fc9740 1 -- 172.21.15.39:0/903922383 wait complete. 2024-03-18T16:10:38.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.405+0000 7f993a000640 1 -- 172.21.15.39:0/2817864164 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f99280cf2d0 con 0x7f993c0757c0 2024-03-18T16:10:38.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.405+0000 7f9941800640 1 --2- 172.21.15.39:0/2817864164 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f99240a2980 0x7f99240a4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:38.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.405+0000 7f9946fc9740 1 -- start start 2024-03-18T16:10:38.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.406+0000 7f9946fc9740 1 Processor -- start 2024-03-18T16:10:38.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.406+0000 7f9941800640 1 --2- 172.21.15.39:0/2817864164 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f99240a2980 0x7f99240a4e40 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x555d482f4d20 tx=0x7f993002c930 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:38.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.407+0000 7f9946fc9740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555d482f38c0 0x555d482fdbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:38.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.407+0000 7f9946fc9740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555d482f42b0 0x555d482fe0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:38.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.407+0000 7f9946fc9740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555d482f9420 0x555d48305130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:38.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.407+0000 7f9946fc9740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x555d48214110 con 0x555d482f42b0 2024-03-18T16:10:38.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.407+0000 7f9946fc9740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x555d482544f0 con 0x555d482f9420 2024-03-18T16:10:38.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.407+0000 7f9946fc9740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x555d482fc3d0 con 0x555d482f38c0 2024-03-18T16:10:38.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.407+0000 7f9946fc9740 10 client.0 _ll_register_callbacks cb 0x555d48201b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:10:38.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.407+0000 7f9941800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555d482f38c0 0x555d482fdbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:38.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.407+0000 7f9941800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555d482f38c0 0x555d482fdbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:34972/0 (socket says 172.21.15.39:34972) 2024-03-18T16:10:38.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.407+0000 7f9941800640 1 -- 172.21.15.39:0/1863033795 learned_addr learned my addr 172.21.15.39:0/1863033795 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:10:38.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.407+0000 7f9940e00640 1 --2- 172.21.15.39:0/1863033795 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555d482f42b0 0x555d482fe0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:38.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.408+0000 7f9942200640 1 --2- 172.21.15.39:0/1863033795 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555d482f9420 0x555d48305130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:38.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.408+0000 7f9941800640 1 -- 172.21.15.39:0/1863033795 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555d482f9420 msgr2=0x555d48305130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:38.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.408+0000 7f9941800640 1 --2- 172.21.15.39:0/1863033795 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555d482f9420 0x555d48305130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.408+0000 7f9941800640 1 -- 172.21.15.39:0/1863033795 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555d482f42b0 msgr2=0x555d482fe0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:38.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.408+0000 7f9941800640 1 --2- 172.21.15.39:0/1863033795 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555d482f42b0 0x555d482fe0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.408+0000 7f9941800640 1 -- 172.21.15.39:0/1863033795 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555d47f6c3f0 con 0x555d482f38c0 2024-03-18T16:10:38.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.408+0000 7f9942200640 1 --2- 172.21.15.39:0/1863033795 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555d482f9420 0x555d48305130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:10:38.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.408+0000 7f9941800640 1 --2- 172.21.15.39:0/1863033795 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555d482f38c0 0x555d482fdbb0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f993002c220 tx=0x7f9930009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7e7450e113048fa4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:38.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.408+0000 7f9923400640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:10:38.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.408+0000 7f9923400640 1 -- 172.21.15.39:0/1863033795 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9930050020 con 0x555d482f38c0 2024-03-18T16:10:38.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.409+0000 7f9923400640 1 -- 172.21.15.39:0/1863033795 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f9930005410 con 0x555d482f38c0 2024-03-18T16:10:38.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.409+0000 7f9946fc9740 1 -- 172.21.15.39:0/1863033795 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9934007ab0 con 0x555d482f38c0 2024-03-18T16:10:38.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.409+0000 7f9923400640 1 -- 172.21.15.39:0/1863033795 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9930005610 con 0x555d482f38c0 2024-03-18T16:10:38.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.409+0000 7f9946fc9740 10 client.25693 fetch_fsmap learned FSMap version 28 2024-03-18T16:10:38.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.409+0000 7f9946fc9740 1 -- 172.21.15.39:0/1863033795 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x555d480367d0 con 0x555d482f38c0 2024-03-18T16:10:38.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.409+0000 7f9923400640 1 -- 172.21.15.39:0/1863033795 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9934007ab0 con 0x555d482f38c0 2024-03-18T16:10:38.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.409+0000 7f9923400640 1 -- 172.21.15.39:0/1863033795 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f993002cce0 con 0x555d482f38c0 2024-03-18T16:10:38.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.410+0000 7f9946fc9740 10 client.25693 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T16:10:38.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.410+0000 7f9946fc9740 10 client.25693 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-18T16:10:38.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.410+0000 7f9946fc9740 20 client.25693 populate_metadata read hostname 'smithi039' 2024-03-18T16:10:38.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.410+0000 7f9946fc9740 1 --2- 172.21.15.39:0/1863033795 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x555d48309ad0 0x555d4830bf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:38.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.410+0000 7f9946fc9740 4 client.25693 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:10:38.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.410+0000 7f9946fc9740 1 -- 172.21.15.39:0/1863033795 --> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555d48265990 con 0x555d48309ad0 2024-03-18T16:10:38.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.410+0000 7f9940e00640 1 --2- 172.21.15.39:0/1863033795 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x555d48309ad0 0x555d4830bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:38.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.411+0000 7f9923400640 10 client.25693 ms_handle_connect on v2:172.21.15.196:6836/895247251 2024-03-18T16:10:38.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.411+0000 7f9940e00640 1 --2- 172.21.15.39:0/1863033795 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x555d48309ad0 0x555d4830bf20 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=21005a07b82c81d1 server_cookie=b12bb48794064897 in_seq=0 out_seq=0 2024-03-18T16:10:38.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.411+0000 7f9923400640 1 -- 172.21.15.39:0/1863033795 <== mds.2 v2:172.21.15.196:6836/895247251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x555d48265990 con 0x555d48309ad0 2024-03-18T16:10:38.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.411+0000 7f9923400640 10 client.25693 handle_command_reply: tid=0 2024-03-18T16:10:38.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.412+0000 7f9946fc9740 1 client.25693 shutdown 2024-03-18T16:10:38.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.412+0000 7f9946fc9740 1 -- 172.21.15.39:0/1863033795 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555d482f38c0 msgr2=0x555d482fdbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:38.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.412+0000 7f9946fc9740 1 --2- 172.21.15.39:0/1863033795 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555d482f38c0 0x555d482fdbb0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f993002c220 tx=0x7f9930009260 comp rx=0 tx=0).stop 2024-03-18T16:10:38.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.412+0000 7f9946fc9740 1 -- 172.21.15.39:0/1863033795 shutdown_connections 2024-03-18T16:10:38.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.412+0000 7f9946fc9740 1 --2- 172.21.15.39:0/1863033795 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555d482f9420 0x555d48305130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.412+0000 7f9946fc9740 1 --2- 172.21.15.39:0/1863033795 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x555d48309ad0 0x555d4830bf20 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.412+0000 7f9946fc9740 1 --2- 172.21.15.39:0/1863033795 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555d482f42b0 0x555d482fe0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.412+0000 7f9946fc9740 1 --2- 172.21.15.39:0/1863033795 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555d482f38c0 0x555d482fdbb0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.412+0000 7f9923400640 0 client.25693 ms_handle_reset on v2:172.21.15.196:6836/895247251 2024-03-18T16:10:38.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.412+0000 7f9946fc9740 1 -- 172.21.15.39:0/1863033795 >> 172.21.15.39:0/1863033795 conn(0x555d482eee30 msgr2=0x555d482ee1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:38.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.412+0000 7f9946fc9740 1 -- 172.21.15.39:0/1863033795 shutdown_connections 2024-03-18T16:10:38.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.412+0000 7f9946fc9740 1 -- 172.21.15.39:0/1863033795 wait complete. 2024-03-18T16:10:38.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.412+0000 7f9946fc9740 20 client.25693 trim_cache size 0 max 16384 2024-03-18T16:10:38.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.441+0000 7f9946fc9740 1 Processor -- start 2024-03-18T16:10:38.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.441+0000 7f9946fc9740 1 -- start start 2024-03-18T16:10:38.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.441+0000 7f9946fc9740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555d482f38c0 0x555d482fe050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:38.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.441+0000 7f9946fc9740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555d482f42b0 0x555d483050f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:38.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.441+0000 7f9946fc9740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555d482f9420 0x555d482ee070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:38.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.441+0000 7f9946fc9740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f993c1b9290 con 0x555d482f42b0 2024-03-18T16:10:38.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.441+0000 7f9946fc9740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f993c1badd0 con 0x555d482f38c0 2024-03-18T16:10:38.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.441+0000 7f9946fc9740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f993c1b9960 con 0x555d482f9420 2024-03-18T16:10:38.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.441+0000 7f9941800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555d482f38c0 0x555d482fe050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:38.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.442+0000 7f9941800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555d482f38c0 0x555d482fe050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:36146/0 (socket says 172.21.15.39:36146) 2024-03-18T16:10:38.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.442+0000 7f9941800640 1 -- 172.21.15.39:0/901942793 learned_addr learned my addr 172.21.15.39:0/901942793 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:10:38.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.442+0000 7f9940e00640 1 --2- 172.21.15.39:0/901942793 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555d482f42b0 0x555d483050f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:38.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.442+0000 7f9942200640 1 --2- 172.21.15.39:0/901942793 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555d482f9420 0x555d482ee070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:38.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.442+0000 7f9941800640 1 -- 172.21.15.39:0/901942793 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555d482f9420 msgr2=0x555d482ee070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:38.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.442+0000 7f9941800640 1 --2- 172.21.15.39:0/901942793 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555d482f9420 0x555d482ee070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.442+0000 7f9941800640 1 -- 172.21.15.39:0/901942793 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555d482f42b0 msgr2=0x555d483050f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:38.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.442+0000 7f9941800640 1 --2- 172.21.15.39:0/901942793 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555d482f42b0 0x555d483050f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.442+0000 7f9941800640 1 -- 172.21.15.39:0/901942793 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555d482eac10 con 0x555d482f38c0 2024-03-18T16:10:38.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.442+0000 7f9940e00640 1 --2- 172.21.15.39:0/901942793 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555d482f42b0 0x555d483050f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:10:38.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.443+0000 7f9941800640 1 --2- 172.21.15.39:0/901942793 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555d482f38c0 0x555d482fe050 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f9930036040 tx=0x7f993002bce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c8c991632fad0e10 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:38.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.443+0000 7f992e000640 1 -- 172.21.15.39:0/901942793 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9930050070 con 0x555d482f38c0 2024-03-18T16:10:38.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.443+0000 7f992e000640 1 -- 172.21.15.39:0/901942793 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f993002ca30 con 0x555d482f38c0 2024-03-18T16:10:38.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.443+0000 7f992e000640 1 -- 172.21.15.39:0/901942793 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f993004c480 con 0x555d482f38c0 2024-03-18T16:10:38.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.444+0000 7f9946fc9740 1 -- 172.21.15.39:0/901942793 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555d482f38c0 msgr2=0x555d482fe050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:38.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.444+0000 7f9946fc9740 1 --2- 172.21.15.39:0/901942793 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555d482f38c0 0x555d482fe050 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f9930036040 tx=0x7f993002bce0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.444+0000 7f9946fc9740 1 -- 172.21.15.39:0/901942793 shutdown_connections 2024-03-18T16:10:38.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.444+0000 7f9946fc9740 1 --2- 172.21.15.39:0/901942793 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555d482f9420 0x555d482ee070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.444+0000 7f9946fc9740 1 --2- 172.21.15.39:0/901942793 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555d482f42b0 0x555d483050f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.444+0000 7f9946fc9740 1 --2- 172.21.15.39:0/901942793 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555d482f38c0 0x555d482fe050 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.444+0000 7f9946fc9740 1 -- 172.21.15.39:0/901942793 >> 172.21.15.39:0/901942793 conn(0x555d482eee30 msgr2=0x555d482ed960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:38.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.444+0000 7f9946fc9740 1 -- 172.21.15.39:0/901942793 shutdown_connections 2024-03-18T16:10:38.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.444+0000 7f9946fc9740 1 -- 172.21.15.39:0/901942793 wait complete. 2024-03-18T16:10:38.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.444+0000 7f9946fc9740 1 -- start start 2024-03-18T16:10:38.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.445+0000 7f9946fc9740 1 Processor -- start 2024-03-18T16:10:38.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.445+0000 7f9946fc9740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555d482f38c0 0x555d48302ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:38.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.445+0000 7f9946fc9740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555d482f42b0 0x555d48303020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:38.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.445+0000 7f9946fc9740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555d482f9420 0x555d48304030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:38.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.445+0000 7f9946fc9740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f993c1ba700 con 0x555d482f9420 2024-03-18T16:10:38.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.445+0000 7f9946fc9740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f993c1ba030 con 0x555d482f42b0 2024-03-18T16:10:38.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.445+0000 7f9946fc9740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f993c1b7270 con 0x555d482f38c0 2024-03-18T16:10:38.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.445+0000 7f9946fc9740 10 client.0 _ll_register_callbacks cb 0x7f9930043070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:10:38.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.445+0000 7f9941800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555d482f38c0 0x555d48302ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:38.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.445+0000 7f9941800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555d482f38c0 0x555d48302ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:34990/0 (socket says 172.21.15.39:34990) 2024-03-18T16:10:38.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.445+0000 7f9941800640 1 -- 172.21.15.39:0/3043018492 learned_addr learned my addr 172.21.15.39:0/3043018492 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:10:38.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.446+0000 7f9942200640 1 --2- 172.21.15.39:0/3043018492 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555d482f9420 0x555d48304030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:38.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.446+0000 7f9941800640 1 -- 172.21.15.39:0/3043018492 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555d482f42b0 msgr2=0x555d48303020 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:10:38.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.446+0000 7f9940e00640 1 --2- 172.21.15.39:0/3043018492 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555d482f42b0 0x555d48303020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:38.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.446+0000 7f9941800640 1 --2- 172.21.15.39:0/3043018492 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555d482f42b0 0x555d48303020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.446+0000 7f9941800640 1 -- 172.21.15.39:0/3043018492 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555d482f9420 msgr2=0x555d48304030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:38.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.446+0000 7f9941800640 1 --2- 172.21.15.39:0/3043018492 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555d482f9420 0x555d48304030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.446+0000 7f9941800640 1 -- 172.21.15.39:0/3043018492 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555d482fc510 con 0x555d482f38c0 2024-03-18T16:10:38.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.446+0000 7f9942200640 1 --2- 172.21.15.39:0/3043018492 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555d482f9420 0x555d48304030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:10:38.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.446+0000 7f9940e00640 1 --2- 172.21.15.39:0/3043018492 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555d482f42b0 0x555d48303020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:10:38.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.446+0000 7f9941800640 1 --2- 172.21.15.39:0/3043018492 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555d482f38c0 0x555d48302ae0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f993002c220 tx=0x7f9930009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f7b301ab339d0e4a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:38.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.446+0000 7f9923400640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:10:38.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.447+0000 7f9923400640 1 -- 172.21.15.39:0/3043018492 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9930050070 con 0x555d482f38c0 2024-03-18T16:10:38.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.447+0000 7f9923400640 1 -- 172.21.15.39:0/3043018492 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f9930003a00 con 0x555d482f38c0 2024-03-18T16:10:38.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.447+0000 7f9946fc9740 1 -- 172.21.15.39:0/3043018492 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f993002ca30 con 0x555d482f38c0 2024-03-18T16:10:38.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.447+0000 7f9923400640 1 -- 172.21.15.39:0/3043018492 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f993004c480 con 0x555d482f38c0 2024-03-18T16:10:38.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.447+0000 7f9946fc9740 10 client.25699 fetch_fsmap learned FSMap version 28 2024-03-18T16:10:38.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.447+0000 7f9946fc9740 1 -- 172.21.15.39:0/3043018492 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x555d4836fe80 con 0x555d482f38c0 2024-03-18T16:10:38.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.452+0000 7f9923400640 1 -- 172.21.15.39:0/3043018492 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f993002ca30 con 0x555d482f38c0 2024-03-18T16:10:38.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.452+0000 7f9923400640 1 -- 172.21.15.39:0/3043018492 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f99300557d0 con 0x555d482f38c0 2024-03-18T16:10:38.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.452+0000 7f9946fc9740 10 client.25699 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T16:10:38.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.452+0000 7f9946fc9740 10 client.25699 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-18T16:10:38.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.452+0000 7f9946fc9740 20 client.25699 populate_metadata read hostname 'smithi039' 2024-03-18T16:10:38.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.452+0000 7f9946fc9740 1 --2- 172.21.15.39:0/3043018492 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x555d483705b0 0x555d483729d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:38.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.452+0000 7f9946fc9740 4 client.25699 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:10:38.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.452+0000 7f9946fc9740 1 -- 172.21.15.39:0/3043018492 --> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555d4825fce0 con 0x555d483705b0 2024-03-18T16:10:38.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.452+0000 7f9940e00640 1 --2- 172.21.15.39:0/3043018492 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x555d483705b0 0x555d483729d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:38.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.453+0000 7f9923400640 10 client.25699 ms_handle_connect on v2:172.21.15.196:6836/895247251 2024-03-18T16:10:38.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.453+0000 7f9940e00640 1 --2- 172.21.15.39:0/3043018492 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x555d483705b0 0x555d483729d0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3c856a9fc42173a1 server_cookie=f0f1475d1d47012a in_seq=0 out_seq=0 2024-03-18T16:10:38.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.453+0000 7f9923400640 1 -- 172.21.15.39:0/3043018492 <== mds.2 v2:172.21.15.196:6836/895247251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555d4825fce0 con 0x555d483705b0 2024-03-18T16:10:38.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.453+0000 7f9923400640 10 client.25699 handle_command_reply: tid=0 2024-03-18T16:10:38.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.453+0000 7f9946fc9740 1 client.25699 shutdown 2024-03-18T16:10:38.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.453+0000 7f9946fc9740 1 -- 172.21.15.39:0/3043018492 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555d482f38c0 msgr2=0x555d48302ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:38.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.453+0000 7f9946fc9740 1 --2- 172.21.15.39:0/3043018492 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555d482f38c0 0x555d48302ae0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f993002c220 tx=0x7f9930009260 comp rx=0 tx=0).stop 2024-03-18T16:10:38.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.454+0000 7f9946fc9740 1 -- 172.21.15.39:0/3043018492 shutdown_connections 2024-03-18T16:10:38.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.454+0000 7f9946fc9740 1 --2- 172.21.15.39:0/3043018492 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555d482f42b0 0x555d48303020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.454+0000 7f9946fc9740 1 --2- 172.21.15.39:0/3043018492 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x555d483705b0 0x555d483729d0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.454+0000 7f9946fc9740 1 --2- 172.21.15.39:0/3043018492 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555d482f9420 0x555d48304030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.454+0000 7f9946fc9740 1 --2- 172.21.15.39:0/3043018492 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555d482f38c0 0x555d48302ae0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.454+0000 7f9946fc9740 1 -- 172.21.15.39:0/3043018492 >> 172.21.15.39:0/3043018492 conn(0x555d482eee30 msgr2=0x555d482ee1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:38.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.454+0000 7f9923400640 0 client.25699 ms_handle_reset on v2:172.21.15.196:6836/895247251 2024-03-18T16:10:38.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.454+0000 7f9946fc9740 1 -- 172.21.15.39:0/3043018492 shutdown_connections 2024-03-18T16:10:38.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.454+0000 7f9946fc9740 1 -- 172.21.15.39:0/3043018492 wait complete. 2024-03-18T16:10:38.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.454+0000 7f9946fc9740 20 client.25699 trim_cache size 0 max 16384 2024-03-18T16:10:38.461 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:10:38.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.455+0000 7f9944200640 1 -- 172.21.15.39:0/2817864164 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f99240a2980 msgr2=0x7f99240a4e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:38.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.455+0000 7f9944200640 1 --2- 172.21.15.39:0/2817864164 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f99240a2980 0x7f99240a4e40 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x555d482f4d20 tx=0x7f993002c930 comp rx=0 tx=0).stop 2024-03-18T16:10:38.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.455+0000 7f9944200640 1 -- 172.21.15.39:0/2817864164 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f993c0757c0 msgr2=0x7f993c1050a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:38.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.455+0000 7f9944200640 1 --2- 172.21.15.39:0/2817864164 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f993c0757c0 0x7f993c1050a0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f992800e9f0 tx=0x7f992800eeb0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.455+0000 7f9944200640 1 -- 172.21.15.39:0/2817864164 shutdown_connections 2024-03-18T16:10:38.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.455+0000 7f9944200640 1 --2- 172.21.15.39:0/2817864164 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f99240a2980 0x7f99240a4e40 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.455+0000 7f9944200640 1 --2- 172.21.15.39:0/2817864164 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f993c109670 0x7f993c1016d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.455+0000 7f9944200640 1 --2- 172.21.15.39:0/2817864164 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f993c0757c0 0x7f993c1050a0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.455+0000 7f9944200640 1 --2- 172.21.15.39:0/2817864164 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f993c0741e0 0x7f993c104b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.455+0000 7f9944200640 1 -- 172.21.15.39:0/2817864164 >> 172.21.15.39:0/2817864164 conn(0x7f993c0fcd50 msgr2=0x7f993c105600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:38.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.455+0000 7f9944200640 1 -- 172.21.15.39:0/2817864164 shutdown_connections 2024-03-18T16:10:38.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.455+0000 7f9944200640 1 -- 172.21.15.39:0/2817864164 wait complete. 2024-03-18T16:10:38.478 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-18T16:10:38.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.676+0000 7fbdc1200640 1 -- 172.21.15.39:0/2292955788 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbdbc1018e0 msgr2=0x7fbdbc101d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:38.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.676+0000 7fbdc1200640 1 --2- 172.21.15.39:0/2292955788 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbdbc1018e0 0x7fbdbc101d60 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fbda8009900 tx=0x7fbda802bc30 comp rx=0 tx=0).stop 2024-03-18T16:10:38.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.676+0000 7fbdc1200640 1 -- 172.21.15.39:0/2292955788 shutdown_connections 2024-03-18T16:10:38.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.676+0000 7fbdc1200640 1 --2- 172.21.15.39:0/2292955788 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbdbc1018e0 0x7fbdbc101d60 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.676+0000 7fbdc1200640 1 --2- 172.21.15.39:0/2292955788 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbdbc1022a0 0x7fbdbc108b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.676+0000 7fbdc1200640 1 --2- 172.21.15.39:0/2292955788 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbdbc1006f0 0x7fbdbc100b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.676+0000 7fbdc1200640 1 -- 172.21.15.39:0/2292955788 >> 172.21.15.39:0/2292955788 conn(0x7fbdbc0fbc40 msgr2=0x7fbdbc0fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:38.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.676+0000 7fbdc1200640 1 -- 172.21.15.39:0/2292955788 shutdown_connections 2024-03-18T16:10:38.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.676+0000 7fbdc1200640 1 -- 172.21.15.39:0/2292955788 wait complete. 2024-03-18T16:10:38.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.677+0000 7fbdc1200640 1 Processor -- start 2024-03-18T16:10:38.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.677+0000 7fbdc1200640 1 -- start start 2024-03-18T16:10:38.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.678+0000 7fbdc1200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbdbc1006f0 0x7fbdbc104630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:38.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.678+0000 7fbdc1200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbdbc1018e0 0x7fbdbc102cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:38.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.678+0000 7fbdc1200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbdbc1022a0 0x7fbdbc103200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:38.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.678+0000 7fbdc1200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fbdbc10c3c0 con 0x7fbdbc1022a0 2024-03-18T16:10:38.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.678+0000 7fbdc1200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fbdbc10c240 con 0x7fbdbc1018e0 2024-03-18T16:10:38.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.678+0000 7fbdc1200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fbdbc10c540 con 0x7fbdbc1006f0 2024-03-18T16:10:38.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.678+0000 7fbdb9e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbdbc1006f0 0x7fbdbc104630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:38.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.678+0000 7fbdb9e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbdbc1006f0 0x7fbdbc104630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:35004/0 (socket says 172.21.15.39:35004) 2024-03-18T16:10:38.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.678+0000 7fbdb9e00640 1 -- 172.21.15.39:0/1367144947 learned_addr learned my addr 172.21.15.39:0/1367144947 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:10:38.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.678+0000 7fbdb9400640 1 --2- 172.21.15.39:0/1367144947 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbdbc1018e0 0x7fbdbc102cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:38.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.678+0000 7fbdba800640 1 --2- 172.21.15.39:0/1367144947 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbdbc1022a0 0x7fbdbc103200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:38.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.679+0000 7fbdb9e00640 1 -- 172.21.15.39:0/1367144947 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbdbc1018e0 msgr2=0x7fbdbc102cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:38.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.679+0000 7fbdb9e00640 1 --2- 172.21.15.39:0/1367144947 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbdbc1018e0 0x7fbdbc102cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.679+0000 7fbdb9e00640 1 -- 172.21.15.39:0/1367144947 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbdbc1022a0 msgr2=0x7fbdbc103200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:38.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.679+0000 7fbdb9e00640 1 --2- 172.21.15.39:0/1367144947 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbdbc1022a0 0x7fbdbc103200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.679+0000 7fbdb9e00640 1 -- 172.21.15.39:0/1367144947 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdbc103ac0 con 0x7fbdbc1006f0 2024-03-18T16:10:38.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.679+0000 7fbdba800640 1 --2- 172.21.15.39:0/1367144947 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbdbc1022a0 0x7fbdbc103200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:10:38.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.679+0000 7fbdb9e00640 1 --2- 172.21.15.39:0/1367144947 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbdbc1006f0 0x7fbdbc104630 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fbdac00d770 tx=0x7fbdac00dc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:38.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.679+0000 7fbdb2a00640 1 -- 172.21.15.39:0/1367144947 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdac017070 con 0x7fbdbc1006f0 2024-03-18T16:10:38.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.679+0000 7fbdb2a00640 1 -- 172.21.15.39:0/1367144947 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbdac009c30 con 0x7fbdbc1006f0 2024-03-18T16:10:38.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.679+0000 7fbdb2a00640 1 -- 172.21.15.39:0/1367144947 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdac003d60 con 0x7fbdbc1006f0 2024-03-18T16:10:38.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.679+0000 7fbdc1200640 1 -- 172.21.15.39:0/1367144947 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbdbc1a1ce0 con 0x7fbdbc1006f0 2024-03-18T16:10:38.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.680+0000 7fbdc1200640 1 -- 172.21.15.39:0/1367144947 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbdbc1a20f0 con 0x7fbdbc1006f0 2024-03-18T16:10:38.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.685+0000 7fbdc3e20740 1 Processor -- start 2024-03-18T16:10:38.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.685+0000 7fbdc3e20740 1 -- start start 2024-03-18T16:10:38.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.685+0000 7fbdb2a00640 1 -- 172.21.15.39:0/1367144947 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fbdac003f00 con 0x7fbdbc1006f0 2024-03-18T16:10:38.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.685+0000 7fbdc3e20740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559fd7c5b0d0 0x559fd7c5b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:38.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.685+0000 7fbdc3e20740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559fd7c5bac0 0x559fd7c605c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:38.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.685+0000 7fbdc3e20740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559fd7c60c30 0x559fd7c630c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:38.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.685+0000 7fbdc3e20740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x559fd7c1a3b0 con 0x559fd7c5b0d0 2024-03-18T16:10:38.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.685+0000 7fbdc3e20740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x559fd7bb7450 con 0x559fd7c5bac0 2024-03-18T16:10:38.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.685+0000 7fbdc3e20740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x559fd7b5fcb0 con 0x559fd7c60c30 2024-03-18T16:10:38.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.686+0000 7fbdba800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559fd7c5b0d0 0x559fd7c5b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:38.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.686+0000 7fbdb9400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559fd7c60c30 0x559fd7c630c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:38.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.686+0000 7fbdb9400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559fd7c60c30 0x559fd7c630c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:35008/0 (socket says 172.21.15.39:35008) 2024-03-18T16:10:38.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.686+0000 7fbdb9400640 1 -- 172.21.15.39:0/2339744373 learned_addr learned my addr 172.21.15.39:0/2339744373 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:10:38.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.686+0000 7fbdb9e00640 1 --2- 172.21.15.39:0/2339744373 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559fd7c5bac0 0x559fd7c605c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:38.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.687+0000 7fbdb9400640 1 -- 172.21.15.39:0/2339744373 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559fd7c5bac0 msgr2=0x559fd7c605c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:38.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.687+0000 7fbdb9400640 1 --2- 172.21.15.39:0/2339744373 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559fd7c5bac0 0x559fd7c605c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.687+0000 7fbdb9400640 1 -- 172.21.15.39:0/2339744373 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559fd7c5b0d0 msgr2=0x559fd7c5b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:38.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.687+0000 7fbdb9400640 1 --2- 172.21.15.39:0/2339744373 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559fd7c5b0d0 0x559fd7c5b4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.687+0000 7fbdb9400640 1 -- 172.21.15.39:0/2339744373 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559fd7b811f0 con 0x559fd7c60c30 2024-03-18T16:10:38.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.687+0000 7fbdb2a00640 1 --2- 172.21.15.39:0/1367144947 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fbda00a2900 0x7fbda00a4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:38.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.688+0000 7fbdb9400640 1 --2- 172.21.15.39:0/2339744373 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559fd7c60c30 0x559fd7c630c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fbda802c8e0 tx=0x7fbda80095e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=58ed57852865fee8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:38.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.688+0000 7fbdb2a00640 1 -- 172.21.15.39:0/1367144947 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fbdac0ce4a0 con 0x7fbdbc1006f0 2024-03-18T16:10:38.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.688+0000 7fbda7400640 1 -- 172.21.15.39:0/2339744373 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbda8045070 con 0x559fd7c60c30 2024-03-18T16:10:38.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.688+0000 7fbda7400640 1 -- 172.21.15.39:0/2339744373 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbda8004040 con 0x559fd7c60c30 2024-03-18T16:10:38.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.688+0000 7fbda7400640 1 -- 172.21.15.39:0/2339744373 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbda8004ef0 con 0x559fd7c60c30 2024-03-18T16:10:38.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.688+0000 7fbdba800640 1 --2- 172.21.15.39:0/1367144947 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fbda00a2900 0x7fbda00a4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:38.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.689+0000 7fbdc3e20740 1 -- 172.21.15.39:0/2339744373 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559fd7c60c30 msgr2=0x559fd7c630c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:38.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.689+0000 7fbdc3e20740 1 --2- 172.21.15.39:0/2339744373 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559fd7c60c30 0x559fd7c630c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fbda802c8e0 tx=0x7fbda80095e0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.689+0000 7fbdba800640 1 --2- 172.21.15.39:0/1367144947 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fbda00a2900 0x7fbda00a4dc0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x559fd7c5c320 tx=0x7fbdb4009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:38.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.689+0000 7fbdc3e20740 1 -- 172.21.15.39:0/2339744373 shutdown_connections 2024-03-18T16:10:38.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.689+0000 7fbdc3e20740 1 --2- 172.21.15.39:0/2339744373 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559fd7c5bac0 0x559fd7c605c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.689+0000 7fbdc3e20740 1 --2- 172.21.15.39:0/2339744373 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559fd7c60c30 0x559fd7c630c0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.689+0000 7fbdc3e20740 1 --2- 172.21.15.39:0/2339744373 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559fd7c5b0d0 0x559fd7c5b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.689+0000 7fbdc3e20740 1 -- 172.21.15.39:0/2339744373 >> 172.21.15.39:0/2339744373 conn(0x559fd7c4de30 msgr2=0x559fd7c4e230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:38.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.689+0000 7fbdc3e20740 1 -- 172.21.15.39:0/2339744373 shutdown_connections 2024-03-18T16:10:38.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.689+0000 7fbdc3e20740 1 -- 172.21.15.39:0/2339744373 wait complete. 2024-03-18T16:10:38.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.690+0000 7fbdc3e20740 1 -- start start 2024-03-18T16:10:38.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.690+0000 7fbdc3e20740 1 Processor -- start 2024-03-18T16:10:38.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.691+0000 7fbdc3e20740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559fd7c5b0d0 0x559fd7c541c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:38.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.691+0000 7fbdc3e20740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559fd7c5bac0 0x559fd7c54700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:38.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.691+0000 7fbdc3e20740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559fd7c60c30 0x559fd7c59730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:38.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.691+0000 7fbdc3e20740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x559fd7b73110 con 0x559fd7c60c30 2024-03-18T16:10:38.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.691+0000 7fbdc3e20740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x559fd7bb34f0 con 0x559fd7c5b0d0 2024-03-18T16:10:38.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.691+0000 7fbdc3e20740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x559fd7c52960 con 0x559fd7c5bac0 2024-03-18T16:10:38.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.691+0000 7fbdc3e20740 10 client.0 _ll_register_callbacks cb 0x559fd7b60b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:10:38.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.691+0000 7fbdb9e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559fd7c5b0d0 0x559fd7c541c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:38.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.691+0000 7fbdb9e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559fd7c5b0d0 0x559fd7c541c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:36174/0 (socket says 172.21.15.39:36174) 2024-03-18T16:10:38.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.691+0000 7fbdb9e00640 1 -- 172.21.15.39:0/2702882578 learned_addr learned my addr 172.21.15.39:0/2702882578 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:10:38.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.691+0000 7fbdb9400640 1 --2- 172.21.15.39:0/2702882578 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559fd7c5bac0 0x559fd7c54700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:38.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.691+0000 7fbdb9e00640 1 -- 172.21.15.39:0/2702882578 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559fd7c5bac0 msgr2=0x559fd7c54700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:38.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.691+0000 7fbdba800640 1 --2- 172.21.15.39:0/2702882578 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559fd7c60c30 0x559fd7c59730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:38.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.691+0000 7fbdb9e00640 1 --2- 172.21.15.39:0/2702882578 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559fd7c5bac0 0x559fd7c54700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.691+0000 7fbdb9e00640 1 -- 172.21.15.39:0/2702882578 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559fd7c60c30 msgr2=0x559fd7c59730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:38.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.691+0000 7fbdb9e00640 1 --2- 172.21.15.39:0/2702882578 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559fd7c60c30 0x559fd7c59730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.691+0000 7fbdb9e00640 1 -- 172.21.15.39:0/2702882578 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559fd78cb3f0 con 0x559fd7c5b0d0 2024-03-18T16:10:38.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.692+0000 7fbdb9400640 1 --2- 172.21.15.39:0/2702882578 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559fd7c5bac0 0x559fd7c54700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:10:38.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.692+0000 7fbdba800640 1 --2- 172.21.15.39:0/2702882578 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559fd7c60c30 0x559fd7c59730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:10:38.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.692+0000 7fbdb9e00640 1 --2- 172.21.15.39:0/2702882578 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559fd7c5b0d0 0x559fd7c541c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fbdac096eb0 tx=0x7fbdac096dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10cacfee230737ac server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:38.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.692+0000 7fbd9be00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:10:38.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.692+0000 7fbd9be00640 1 -- 172.21.15.39:0/2702882578 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdac020040 con 0x559fd7c5b0d0 2024-03-18T16:10:38.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.692+0000 7fbdc3e20740 1 -- 172.21.15.39:0/2702882578 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbda8004040 con 0x559fd7c5b0d0 2024-03-18T16:10:38.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.692+0000 7fbd9be00640 1 -- 172.21.15.39:0/2702882578 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fbdac0970c0 con 0x559fd7c5b0d0 2024-03-18T16:10:38.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.692+0000 7fbd9be00640 1 -- 172.21.15.39:0/2702882578 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fbdac0b3050 con 0x559fd7c5b0d0 2024-03-18T16:10:38.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.692+0000 7fbdc3e20740 10 client.25682 fetch_fsmap learned FSMap version 28 2024-03-18T16:10:38.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.692+0000 7fbdc3e20740 1 -- 172.21.15.39:0/2702882578 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x559fd79957d0 con 0x559fd7c5b0d0 2024-03-18T16:10:38.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.693+0000 7fbd9be00640 1 -- 172.21.15.39:0/2702882578 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbda8004040 con 0x559fd7c5b0d0 2024-03-18T16:10:38.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.693+0000 7fbd9be00640 1 -- 172.21.15.39:0/2702882578 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fbdac023060 con 0x559fd7c5b0d0 2024-03-18T16:10:38.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.694+0000 7fbdc3e20740 10 client.25682 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T16:10:38.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.694+0000 7fbdc3e20740 10 client.25682 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T16:10:38.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.694+0000 7fbdc3e20740 20 client.25682 populate_metadata read hostname 'smithi039' 2024-03-18T16:10:38.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.694+0000 7fbdc3e20740 1 --2- 172.21.15.39:0/2702882578 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x559fd7c79ac0 0x559fd7c7beb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:38.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.694+0000 7fbdc3e20740 4 client.25682 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:10:38.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.694+0000 7fbdc3e20740 1 -- 172.21.15.39:0/2702882578 --> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559fd7bc4990 con 0x559fd7c79ac0 2024-03-18T16:10:38.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.694+0000 7fbdb9400640 1 --2- 172.21.15.39:0/2702882578 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x559fd7c79ac0 0x559fd7c7beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:38.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.694+0000 7fbdb9400640 1 --2- 172.21.15.39:0/2702882578 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x559fd7c79ac0 0x559fd7c7beb0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=89b732088d93360 server_cookie=62d26fddc0145c23 in_seq=0 out_seq=0 2024-03-18T16:10:38.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.694+0000 7fbd9be00640 10 client.25682 ms_handle_connect on v2:172.21.15.196:6838/2001427268 2024-03-18T16:10:38.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.695+0000 7fbd9be00640 1 -- 172.21.15.39:0/2702882578 <== mds.1 v2:172.21.15.196:6838/2001427268 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559fd7bc4990 con 0x559fd7c79ac0 2024-03-18T16:10:38.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.695+0000 7fbd9be00640 10 client.25682 handle_command_reply: tid=0 2024-03-18T16:10:38.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.695+0000 7fbdc3e20740 1 client.25682 shutdown 2024-03-18T16:10:38.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.695+0000 7fbdc3e20740 1 -- 172.21.15.39:0/2702882578 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559fd7c5b0d0 msgr2=0x559fd7c541c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:38.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.695+0000 7fbdc3e20740 1 --2- 172.21.15.39:0/2702882578 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559fd7c5b0d0 0x559fd7c541c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fbdac096eb0 tx=0x7fbdac096dc0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.696+0000 7fbdc3e20740 1 -- 172.21.15.39:0/2702882578 shutdown_connections 2024-03-18T16:10:38.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.696+0000 7fbdc3e20740 1 --2- 172.21.15.39:0/2702882578 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x559fd7c79ac0 0x559fd7c7beb0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.696+0000 7fbdc3e20740 1 --2- 172.21.15.39:0/2702882578 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559fd7c60c30 0x559fd7c59730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.696+0000 7fbdc3e20740 1 --2- 172.21.15.39:0/2702882578 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559fd7c5bac0 0x559fd7c54700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.696+0000 7fbdc3e20740 1 --2- 172.21.15.39:0/2702882578 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559fd7c5b0d0 0x559fd7c541c0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.696+0000 7fbdc3e20740 1 -- 172.21.15.39:0/2702882578 >> 172.21.15.39:0/2702882578 conn(0x559fd7c4de30 msgr2=0x559fd7c4d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:38.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.696+0000 7fbd9be00640 0 client.25682 ms_handle_reset on v2:172.21.15.196:6838/2001427268 2024-03-18T16:10:38.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.696+0000 7fbdc3e20740 1 -- 172.21.15.39:0/2702882578 shutdown_connections 2024-03-18T16:10:38.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.696+0000 7fbdc3e20740 1 -- 172.21.15.39:0/2702882578 wait complete. 2024-03-18T16:10:38.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.696+0000 7fbdc3e20740 20 client.25682 trim_cache size 0 max 16384 2024-03-18T16:10:38.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.710+0000 7fbdc3e20740 1 Processor -- start 2024-03-18T16:10:38.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.711+0000 7fbdc3e20740 1 -- start start 2024-03-18T16:10:38.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.711+0000 7fbdc3e20740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559fd7c5b0d0 0x559fd7c537e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:38.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.711+0000 7fbdc3e20740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559fd7c5bac0 0x559fd7c53d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:38.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.711+0000 7fbdc3e20740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559fd7c60c30 0x559fd7c54260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:38.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.711+0000 7fbdc3e20740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fbdbc1a82b0 con 0x559fd7c5b0d0 2024-03-18T16:10:38.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.711+0000 7fbdc3e20740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fbdbc1a9df0 con 0x559fd7c5bac0 2024-03-18T16:10:38.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.711+0000 7fbdc3e20740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fbdbc1a8980 con 0x559fd7c60c30 2024-03-18T16:10:38.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.711+0000 7fbdb9e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559fd7c5b0d0 0x559fd7c537e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:38.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.711+0000 7fbdb9e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559fd7c5b0d0 0x559fd7c537e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:50432/0 (socket says 172.21.15.39:50432) 2024-03-18T16:10:38.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.711+0000 7fbdb9e00640 1 -- 172.21.15.39:0/2014803664 learned_addr learned my addr 172.21.15.39:0/2014803664 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:10:38.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.711+0000 7fbdb9400640 1 --2- 172.21.15.39:0/2014803664 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559fd7c5bac0 0x559fd7c53d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:38.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.712+0000 7fbdba800640 1 --2- 172.21.15.39:0/2014803664 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559fd7c60c30 0x559fd7c54260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:38.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.712+0000 7fbdb9400640 1 -- 172.21.15.39:0/2014803664 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559fd7c60c30 msgr2=0x559fd7c54260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:38.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.712+0000 7fbdb9400640 1 --2- 172.21.15.39:0/2014803664 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559fd7c60c30 0x559fd7c54260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.712+0000 7fbdb9400640 1 -- 172.21.15.39:0/2014803664 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559fd7c5b0d0 msgr2=0x559fd7c537e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:38.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.712+0000 7fbdb9400640 1 --2- 172.21.15.39:0/2014803664 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559fd7c5b0d0 0x559fd7c537e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.712+0000 7fbdb9400640 1 -- 172.21.15.39:0/2014803664 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559fd7c49c10 con 0x559fd7c5bac0 2024-03-18T16:10:38.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.712+0000 7fbdb9e00640 1 --2- 172.21.15.39:0/2014803664 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559fd7c5b0d0 0x559fd7c537e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:10:38.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.712+0000 7fbdb9400640 1 --2- 172.21.15.39:0/2014803664 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559fd7c5bac0 0x559fd7c53d20 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fbda8005ab0 tx=0x7fbda8000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6a2482f35d922a02 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:38.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.713+0000 7fbda6a00640 1 -- 172.21.15.39:0/2014803664 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbda8045040 con 0x559fd7c5bac0 2024-03-18T16:10:38.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.713+0000 7fbda6a00640 1 -- 172.21.15.39:0/2014803664 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbda80042d0 con 0x559fd7c5bac0 2024-03-18T16:10:38.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.713+0000 7fbda6a00640 1 -- 172.21.15.39:0/2014803664 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbda8052640 con 0x559fd7c5bac0 2024-03-18T16:10:38.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.713+0000 7fbdc3e20740 1 -- 172.21.15.39:0/2014803664 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559fd7c5bac0 msgr2=0x559fd7c53d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:38.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.713+0000 7fbdc3e20740 1 --2- 172.21.15.39:0/2014803664 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559fd7c5bac0 0x559fd7c53d20 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fbda8005ab0 tx=0x7fbda8000bc0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.714+0000 7fbdc3e20740 1 -- 172.21.15.39:0/2014803664 shutdown_connections 2024-03-18T16:10:38.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.714+0000 7fbdc3e20740 1 --2- 172.21.15.39:0/2014803664 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559fd7c5bac0 0x559fd7c53d20 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.714+0000 7fbdc3e20740 1 --2- 172.21.15.39:0/2014803664 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559fd7c60c30 0x559fd7c54260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.714+0000 7fbdc3e20740 1 --2- 172.21.15.39:0/2014803664 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559fd7c5b0d0 0x559fd7c537e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.714+0000 7fbdc3e20740 1 -- 172.21.15.39:0/2014803664 >> 172.21.15.39:0/2014803664 conn(0x559fd7c4de30 msgr2=0x559fd7c4d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:38.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.714+0000 7fbdc3e20740 1 -- 172.21.15.39:0/2014803664 shutdown_connections 2024-03-18T16:10:38.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.714+0000 7fbdc3e20740 1 -- 172.21.15.39:0/2014803664 wait complete. 2024-03-18T16:10:38.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.715+0000 7fbdc3e20740 1 -- start start 2024-03-18T16:10:38.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.715+0000 7fbdc3e20740 1 Processor -- start 2024-03-18T16:10:38.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.715+0000 7fbdc3e20740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559fd7c5b0d0 0x559fd7c8a770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:38.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.715+0000 7fbdc3e20740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559fd7c5bac0 0x559fd7c8acb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:38.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.715+0000 7fbdc3e20740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559fd7c60c30 0x559fd7c8bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:38.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.715+0000 7fbdc3e20740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fbdbc1a9720 con 0x559fd7c60c30 2024-03-18T16:10:38.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.715+0000 7fbdc3e20740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fbdbc1a9050 con 0x559fd7c5bac0 2024-03-18T16:10:38.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.715+0000 7fbdc3e20740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fbdbc1a62b0 con 0x559fd7c5b0d0 2024-03-18T16:10:38.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.715+0000 7fbdc3e20740 10 client.0 _ll_register_callbacks cb 0x7fbdac00de80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:10:38.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.716+0000 7fbdb9e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559fd7c5b0d0 0x559fd7c8a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:38.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.716+0000 7fbdb9e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559fd7c5b0d0 0x559fd7c8a770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:35038/0 (socket says 172.21.15.39:35038) 2024-03-18T16:10:38.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.716+0000 7fbdb9e00640 1 -- 172.21.15.39:0/2925675115 learned_addr learned my addr 172.21.15.39:0/2925675115 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:10:38.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.716+0000 7fbdb9400640 1 --2- 172.21.15.39:0/2925675115 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559fd7c5bac0 0x559fd7c8acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:38.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.716+0000 7fbdb9e00640 1 -- 172.21.15.39:0/2925675115 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559fd7c5bac0 msgr2=0x559fd7c8acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:38.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.716+0000 7fbdb9e00640 1 --2- 172.21.15.39:0/2925675115 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559fd7c5bac0 0x559fd7c8acb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.716+0000 7fbdb9e00640 1 -- 172.21.15.39:0/2925675115 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559fd7c60c30 msgr2=0x559fd7c8bcc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:10:38.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.716+0000 7fbdba800640 1 --2- 172.21.15.39:0/2925675115 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559fd7c60c30 0x559fd7c8bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:38.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.716+0000 7fbdb9e00640 1 --2- 172.21.15.39:0/2925675115 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559fd7c60c30 0x559fd7c8bcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.716+0000 7fbdb9e00640 1 -- 172.21.15.39:0/2925675115 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559fd7c52ad0 con 0x559fd7c5b0d0 2024-03-18T16:10:38.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.716+0000 7fbdba800640 1 --2- 172.21.15.39:0/2925675115 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559fd7c60c30 0x559fd7c8bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:10:38.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.717+0000 7fbdb9e00640 1 --2- 172.21.15.39:0/2925675115 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559fd7c5b0d0 0x559fd7c8a770 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fbdac000c00 tx=0x7fbdac009de0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=af50ac107560ef4a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:38.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.717+0000 7fbd9be00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:10:38.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.717+0000 7fbd9be00640 1 -- 172.21.15.39:0/2925675115 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdac0b7050 con 0x559fd7c5b0d0 2024-03-18T16:10:38.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.717+0000 7fbdc3e20740 1 -- 172.21.15.39:0/2925675115 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbda80042d0 con 0x559fd7c5b0d0 2024-03-18T16:10:38.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.717+0000 7fbd9be00640 1 -- 172.21.15.39:0/2925675115 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fbdac09a0a0 con 0x559fd7c5b0d0 2024-03-18T16:10:38.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.717+0000 7fbd9be00640 1 -- 172.21.15.39:0/2925675115 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fbdac020040 con 0x559fd7c5b0d0 2024-03-18T16:10:38.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.717+0000 7fbdc3e20740 10 client.25717 fetch_fsmap learned FSMap version 28 2024-03-18T16:10:38.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.717+0000 7fbdc3e20740 1 -- 172.21.15.39:0/2925675115 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x559fd7cdf6f0 con 0x559fd7c5b0d0 2024-03-18T16:10:38.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.718+0000 7fbd9be00640 1 -- 172.21.15.39:0/2925675115 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbda80042d0 con 0x559fd7c5b0d0 2024-03-18T16:10:38.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.718+0000 7fbd9be00640 1 -- 172.21.15.39:0/2925675115 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fbdac0968c0 con 0x559fd7c5b0d0 2024-03-18T16:10:38.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.718+0000 7fbdc3e20740 10 client.25717 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T16:10:38.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.718+0000 7fbdc3e20740 10 client.25717 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T16:10:38.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.718+0000 7fbdc3e20740 20 client.25717 populate_metadata read hostname 'smithi039' 2024-03-18T16:10:38.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.718+0000 7fbdc3e20740 1 --2- 172.21.15.39:0/2925675115 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x559fd7cdfef0 0x559fd7ce2310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:38.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.718+0000 7fbdc3e20740 4 client.25717 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:10:38.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.718+0000 7fbdc3e20740 1 -- 172.21.15.39:0/2925675115 --> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559fd7bbece0 con 0x559fd7cdfef0 2024-03-18T16:10:38.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.718+0000 7fbdb9400640 1 --2- 172.21.15.39:0/2925675115 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x559fd7cdfef0 0x559fd7ce2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:38.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.719+0000 7fbd9be00640 10 client.25717 ms_handle_connect on v2:172.21.15.196:6838/2001427268 2024-03-18T16:10:38.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.719+0000 7fbdb9400640 1 --2- 172.21.15.39:0/2925675115 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x559fd7cdfef0 0x559fd7ce2310 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=aefdfc420d8371c4 server_cookie=2b56f71587941c78 in_seq=0 out_seq=0 2024-03-18T16:10:38.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.719+0000 7fbd9be00640 1 -- 172.21.15.39:0/2925675115 <== mds.1 v2:172.21.15.196:6838/2001427268 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559fd7bbece0 con 0x559fd7cdfef0 2024-03-18T16:10:38.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.719+0000 7fbd9be00640 10 client.25717 handle_command_reply: tid=0 2024-03-18T16:10:38.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.719+0000 7fbdc3e20740 1 client.25717 shutdown 2024-03-18T16:10:38.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.719+0000 7fbdc3e20740 1 -- 172.21.15.39:0/2925675115 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559fd7c5b0d0 msgr2=0x559fd7c8a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:38.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.719+0000 7fbdc3e20740 1 --2- 172.21.15.39:0/2925675115 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559fd7c5b0d0 0x559fd7c8a770 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fbdac000c00 tx=0x7fbdac009de0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.719+0000 7fbdc3e20740 1 -- 172.21.15.39:0/2925675115 shutdown_connections 2024-03-18T16:10:38.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.719+0000 7fbdc3e20740 1 --2- 172.21.15.39:0/2925675115 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x559fd7cdfef0 0x559fd7ce2310 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.719+0000 7fbdc3e20740 1 --2- 172.21.15.39:0/2925675115 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559fd7c5bac0 0x559fd7c8acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.719+0000 7fbdc3e20740 1 --2- 172.21.15.39:0/2925675115 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559fd7c60c30 0x559fd7c8bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.719+0000 7fbdc3e20740 1 --2- 172.21.15.39:0/2925675115 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559fd7c5b0d0 0x559fd7c8a770 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.719+0000 7fbdc3e20740 1 -- 172.21.15.39:0/2925675115 >> 172.21.15.39:0/2925675115 conn(0x559fd7c4de30 msgr2=0x559fd7c51090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:38.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.722+0000 7fbd9be00640 0 client.25717 ms_handle_reset on v2:172.21.15.196:6838/2001427268 2024-03-18T16:10:38.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.722+0000 7fbdc3e20740 1 -- 172.21.15.39:0/2925675115 shutdown_connections 2024-03-18T16:10:38.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.722+0000 7fbdc3e20740 1 -- 172.21.15.39:0/2925675115 wait complete. 2024-03-18T16:10:38.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.723+0000 7fbdc3e20740 20 client.25717 trim_cache size 0 max 16384 2024-03-18T16:10:38.730 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:10:38.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.723+0000 7fbdc1200640 1 -- 172.21.15.39:0/1367144947 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fbda00a2900 msgr2=0x7fbda00a4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:38.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.723+0000 7fbdc1200640 1 --2- 172.21.15.39:0/1367144947 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fbda00a2900 0x7fbda00a4dc0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x559fd7c5c320 tx=0x7fbdb4009500 comp rx=0 tx=0).stop 2024-03-18T16:10:38.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.723+0000 7fbdc1200640 1 -- 172.21.15.39:0/1367144947 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbdbc1006f0 msgr2=0x7fbdbc104630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:38.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.723+0000 7fbdc1200640 1 --2- 172.21.15.39:0/1367144947 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbdbc1006f0 0x7fbdbc104630 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fbdac00d770 tx=0x7fbdac00dc30 comp rx=0 tx=0).stop 2024-03-18T16:10:38.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.723+0000 7fbdc1200640 1 -- 172.21.15.39:0/1367144947 shutdown_connections 2024-03-18T16:10:38.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.723+0000 7fbdc1200640 1 --2- 172.21.15.39:0/1367144947 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbdbc1018e0 0x7fbdbc102cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.723+0000 7fbdc1200640 1 --2- 172.21.15.39:0/1367144947 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fbda00a2900 0x7fbda00a4dc0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.723+0000 7fbdc1200640 1 --2- 172.21.15.39:0/1367144947 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbdbc1022a0 0x7fbdbc103200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.723+0000 7fbdc1200640 1 --2- 172.21.15.39:0/1367144947 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbdbc1006f0 0x7fbdbc104630 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.723+0000 7fbdc1200640 1 -- 172.21.15.39:0/1367144947 >> 172.21.15.39:0/1367144947 conn(0x7fbdbc0fbc40 msgr2=0x7fbdbc0fd6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:38.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.724+0000 7fbdc1200640 1 -- 172.21.15.39:0/1367144947 shutdown_connections 2024-03-18T16:10:38.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.724+0000 7fbdc1200640 1 -- 172.21.15.39:0/1367144947 wait complete. 2024-03-18T16:10:38.744 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-18T16:10:38.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.956+0000 7fcbfa600640 1 -- 172.21.15.39:0/2265759679 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcbf41007a0 msgr2=0x7fcbf4100c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:38.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.956+0000 7fcbfa600640 1 --2- 172.21.15.39:0/2265759679 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcbf41007a0 0x7fcbf4100c00 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fcbe0009900 tx=0x7fcbe002bc40 comp rx=0 tx=0).stop 2024-03-18T16:10:38.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.956+0000 7fcbfa600640 1 -- 172.21.15.39:0/2265759679 shutdown_connections 2024-03-18T16:10:38.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.956+0000 7fcbfa600640 1 --2- 172.21.15.39:0/2265759679 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcbf41007a0 0x7fcbf4100c00 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.956+0000 7fcbfa600640 1 --2- 172.21.15.39:0/2265759679 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcbf4101140 0x7fcbf41015c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.956+0000 7fcbfa600640 1 --2- 172.21.15.39:0/2265759679 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcbf40ffe40 0x7fcbf4100260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.956+0000 7fcbfa600640 1 -- 172.21.15.39:0/2265759679 >> 172.21.15.39:0/2265759679 conn(0x7fcbf40fba80 msgr2=0x7fcbf40fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:38.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.956+0000 7fcbfa600640 1 -- 172.21.15.39:0/2265759679 shutdown_connections 2024-03-18T16:10:38.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.956+0000 7fcbfa600640 1 -- 172.21.15.39:0/2265759679 wait complete. 2024-03-18T16:10:38.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.957+0000 7fcbfa600640 1 Processor -- start 2024-03-18T16:10:38.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.957+0000 7fcbfa600640 1 -- start start 2024-03-18T16:10:38.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.957+0000 7fcbfa600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcbf40ffe40 0x7fcbf41059c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:38.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.957+0000 7fcbfa600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcbf41007a0 0x7fcbf4105f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:38.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.957+0000 7fcbfa600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcbf4101140 0x7fcbf4102530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:38.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.957+0000 7fcbfa600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fcbf410f460 con 0x7fcbf4101140 2024-03-18T16:10:38.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.957+0000 7fcbfa600640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fcbf410f2e0 con 0x7fcbf41007a0 2024-03-18T16:10:38.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.957+0000 7fcbfa600640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fcbf410f5e0 con 0x7fcbf40ffe40 2024-03-18T16:10:38.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.958+0000 7fcbf3e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcbf41007a0 0x7fcbf4105f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:38.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.958+0000 7fcbf3e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcbf41007a0 0x7fcbf4105f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:36200/0 (socket says 172.21.15.39:36200) 2024-03-18T16:10:38.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.958+0000 7fcbf3e00640 1 -- 172.21.15.39:0/4247879013 learned_addr learned my addr 172.21.15.39:0/4247879013 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:10:38.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.958+0000 7fcbf9200640 1 --2- 172.21.15.39:0/4247879013 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcbf40ffe40 0x7fcbf41059c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:38.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.958+0000 7fcbf3e00640 1 -- 172.21.15.39:0/4247879013 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcbf40ffe40 msgr2=0x7fcbf41059c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:38.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.958+0000 7fcbf3e00640 1 --2- 172.21.15.39:0/4247879013 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcbf40ffe40 0x7fcbf41059c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.958+0000 7fcbf3e00640 1 -- 172.21.15.39:0/4247879013 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcbf4101140 msgr2=0x7fcbf4102530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T16:10:38.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.959+0000 7fcbf9c00640 1 --2- 172.21.15.39:0/4247879013 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcbf4101140 0x7fcbf4102530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:38.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.960+0000 7fcbf3e00640 1 --2- 172.21.15.39:0/4247879013 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcbf4101140 0x7fcbf4102530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.960+0000 7fcbf3e00640 1 -- 172.21.15.39:0/4247879013 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbf4102df0 con 0x7fcbf41007a0 2024-03-18T16:10:38.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.960+0000 7fcbf9200640 1 --2- 172.21.15.39:0/4247879013 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcbf40ffe40 0x7fcbf41059c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:10:38.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.961+0000 7fcbf3e00640 1 --2- 172.21.15.39:0/4247879013 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcbf41007a0 0x7fcbf4105f00 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fcbe002c8f0 tx=0x7fcbe002c660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:38.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.961+0000 7fcbf9c00640 1 --2- 172.21.15.39:0/4247879013 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcbf4101140 0x7fcbf4102530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:10:38.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.961+0000 7fcbf1600640 1 -- 172.21.15.39:0/4247879013 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbe0041070 con 0x7fcbf41007a0 2024-03-18T16:10:38.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.961+0000 7fcbf1600640 1 -- 172.21.15.39:0/4247879013 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcbe0004450 con 0x7fcbf41007a0 2024-03-18T16:10:38.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.961+0000 7fcbfa600640 1 -- 172.21.15.39:0/4247879013 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbf41a5ec0 con 0x7fcbf41007a0 2024-03-18T16:10:38.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.961+0000 7fcbf1600640 1 -- 172.21.15.39:0/4247879013 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbe0034440 con 0x7fcbf41007a0 2024-03-18T16:10:38.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.961+0000 7fcbfa600640 1 -- 172.21.15.39:0/4247879013 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbf41a6270 con 0x7fcbf41007a0 2024-03-18T16:10:38.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.966+0000 7fcbfe99e740 1 Processor -- start 2024-03-18T16:10:38.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.966+0000 7fcbfe99e740 1 -- start start 2024-03-18T16:10:38.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.966+0000 7fcbf1600640 1 -- 172.21.15.39:0/4247879013 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fcbe0044070 con 0x7fcbf41007a0 2024-03-18T16:10:38.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.966+0000 7fcbfe99e740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e1ed9ada90 0x55e1ed9adeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:38.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.966+0000 7fcbfe99e740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e1ed9ae480 0x55e1ed9b2f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:38.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.966+0000 7fcbfe99e740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e1ed9b35f0 0x55e1ed9b5a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:38.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.966+0000 7fcbfe99e740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55e1ed9753b0 con 0x55e1ed9ae480 2024-03-18T16:10:38.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.966+0000 7fcbfe99e740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55e1ed912450 con 0x55e1ed9ada90 2024-03-18T16:10:38.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.966+0000 7fcbfe99e740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55e1ed8bacb0 con 0x55e1ed9b35f0 2024-03-18T16:10:38.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.967+0000 7fcbf9c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e1ed9ada90 0x55e1ed9adeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:38.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.967+0000 7fcbf9c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e1ed9ada90 0x55e1ed9adeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:36216/0 (socket says 172.21.15.39:36216) 2024-03-18T16:10:38.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.967+0000 7fcbf9c00640 1 -- 172.21.15.39:0/492187831 learned_addr learned my addr 172.21.15.39:0/492187831 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:10:38.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.967+0000 7fcbf3e00640 1 --2- 172.21.15.39:0/492187831 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e1ed9b35f0 0x55e1ed9b5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:38.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.968+0000 7fcbf3e00640 1 -- 172.21.15.39:0/492187831 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e1ed9ada90 msgr2=0x55e1ed9adeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:38.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.968+0000 7fcbf3e00640 1 --2- 172.21.15.39:0/492187831 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e1ed9ada90 0x55e1ed9adeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.968+0000 7fcbf3e00640 1 -- 172.21.15.39:0/492187831 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e1ed9ae480 msgr2=0x55e1ed9b2f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:38.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.968+0000 7fcbf3e00640 1 --2- 172.21.15.39:0/492187831 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e1ed9ae480 0x55e1ed9b2f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.968+0000 7fcbf3e00640 1 -- 172.21.15.39:0/492187831 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e1ed8dc1f0 con 0x55e1ed9b35f0 2024-03-18T16:10:38.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.968+0000 7fcbf3e00640 1 --2- 172.21.15.39:0/492187831 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e1ed9b35f0 0x55e1ed9b5a80 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fcbe00ee100 tx=0x7fcbe00f6a60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b81b5f3039a21f8f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:38.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.968+0000 7fcbf1600640 1 --2- 172.21.15.39:0/4247879013 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fcbdc0a2900 0x7fcbdc0a4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:38.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.968+0000 7fcbf1600640 1 -- 172.21.15.39:0/4247879013 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fcbe00f0a50 con 0x7fcbf41007a0 2024-03-18T16:10:38.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.969+0000 7fcbf9c00640 1 --2- 172.21.15.39:0/4247879013 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fcbdc0a2900 0x7fcbdc0a4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:38.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.969+0000 7fcbe6000640 1 -- 172.21.15.39:0/492187831 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbe00ff070 con 0x55e1ed9b35f0 2024-03-18T16:10:38.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.969+0000 7fcbe6000640 1 -- 172.21.15.39:0/492187831 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcbe00c57a0 con 0x55e1ed9b35f0 2024-03-18T16:10:38.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.969+0000 7fcbe6000640 1 -- 172.21.15.39:0/492187831 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbe00bcd70 con 0x55e1ed9b35f0 2024-03-18T16:10:38.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.970+0000 7fcbfe99e740 1 -- 172.21.15.39:0/492187831 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e1ed9b35f0 msgr2=0x55e1ed9b5a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:38.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.970+0000 7fcbfe99e740 1 --2- 172.21.15.39:0/492187831 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e1ed9b35f0 0x55e1ed9b5a80 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fcbe00ee100 tx=0x7fcbe00f6a60 comp rx=0 tx=0).stop 2024-03-18T16:10:38.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.970+0000 7fcbf9c00640 1 --2- 172.21.15.39:0/4247879013 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fcbdc0a2900 0x7fcbdc0a4dc0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x55e1ed9aece0 tx=0x7fcbec009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:38.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.970+0000 7fcbfe99e740 1 -- 172.21.15.39:0/492187831 shutdown_connections 2024-03-18T16:10:38.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.970+0000 7fcbfe99e740 1 --2- 172.21.15.39:0/492187831 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e1ed9b35f0 0x55e1ed9b5a80 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.970+0000 7fcbfe99e740 1 --2- 172.21.15.39:0/492187831 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e1ed9ae480 0x55e1ed9b2f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.970+0000 7fcbfe99e740 1 --2- 172.21.15.39:0/492187831 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e1ed9ada90 0x55e1ed9adeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.970+0000 7fcbfe99e740 1 -- 172.21.15.39:0/492187831 >> 172.21.15.39:0/492187831 conn(0x55e1ed9a9000 msgr2=0x55e1ed9a9400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:38.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.970+0000 7fcbfe99e740 1 -- 172.21.15.39:0/492187831 shutdown_connections 2024-03-18T16:10:38.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.970+0000 7fcbfe99e740 1 -- 172.21.15.39:0/492187831 wait complete. 2024-03-18T16:10:38.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.971+0000 7fcbfe99e740 1 -- start start 2024-03-18T16:10:38.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.971+0000 7fcbfe99e740 1 Processor -- start 2024-03-18T16:10:38.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.971+0000 7fcbfe99e740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e1ed9ada90 0x55e1ed9b7b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:38.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.971+0000 7fcbfe99e740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e1ed9ae480 0x55e1ed9b8050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:38.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.972+0000 7fcbfe99e740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e1ed9b35f0 0x55e1ed9bf090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:38.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.972+0000 7fcbfe99e740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55e1ed8ce110 con 0x55e1ed9b35f0 2024-03-18T16:10:38.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.972+0000 7fcbfe99e740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55e1ed90e4f0 con 0x55e1ed9ae480 2024-03-18T16:10:38.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.972+0000 7fcbfe99e740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55e1ed9b6360 con 0x55e1ed9ada90 2024-03-18T16:10:38.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.972+0000 7fcbfe99e740 10 client.0 _ll_register_callbacks cb 0x55e1ed8bbb70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:10:38.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.972+0000 7fcbf9200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e1ed9ada90 0x55e1ed9b7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:38.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.972+0000 7fcbf9200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e1ed9ada90 0x55e1ed9b7b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:35066/0 (socket says 172.21.15.39:35066) 2024-03-18T16:10:38.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.972+0000 7fcbf9200640 1 -- 172.21.15.39:0/2864726668 learned_addr learned my addr 172.21.15.39:0/2864726668 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:10:38.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.972+0000 7fcbf9c00640 1 --2- 172.21.15.39:0/2864726668 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e1ed9b35f0 0x55e1ed9bf090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:38.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.972+0000 7fcbf3e00640 1 --2- 172.21.15.39:0/2864726668 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e1ed9ae480 0x55e1ed9b8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:38.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.972+0000 7fcbf9200640 1 -- 172.21.15.39:0/2864726668 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e1ed9ae480 msgr2=0x55e1ed9b8050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:38.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.972+0000 7fcbf9200640 1 --2- 172.21.15.39:0/2864726668 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e1ed9ae480 0x55e1ed9b8050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.972+0000 7fcbf9200640 1 -- 172.21.15.39:0/2864726668 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e1ed9b35f0 msgr2=0x55e1ed9bf090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:38.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.973+0000 7fcbf9200640 1 --2- 172.21.15.39:0/2864726668 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e1ed9b35f0 0x55e1ed9bf090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.973+0000 7fcbf9200640 1 -- 172.21.15.39:0/2864726668 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e1ed6263f0 con 0x55e1ed9ada90 2024-03-18T16:10:38.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.973+0000 7fcbf9c00640 1 --2- 172.21.15.39:0/2864726668 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e1ed9b35f0 0x55e1ed9bf090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:10:38.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.973+0000 7fcbf9200640 1 --2- 172.21.15.39:0/2864726668 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e1ed9ada90 0x55e1ed9b7b10 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fcbe8011660 tx=0x7fcbe8011b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=60e9d02aa91d9770 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:38.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.973+0000 7fcbdaa00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:10:38.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.973+0000 7fcbdaa00640 1 -- 172.21.15.39:0/2864726668 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbe801f060 con 0x55e1ed9ada90 2024-03-18T16:10:38.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.973+0000 7fcbdaa00640 1 -- 172.21.15.39:0/2864726668 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fcbe800fae0 con 0x55e1ed9ada90 2024-03-18T16:10:38.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.973+0000 7fcbfe99e740 1 -- 172.21.15.39:0/2864726668 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcbe00c57a0 con 0x55e1ed9ada90 2024-03-18T16:10:38.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.974+0000 7fcbdaa00640 1 -- 172.21.15.39:0/2864726668 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fcbe8011e60 con 0x55e1ed9ada90 2024-03-18T16:10:38.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.974+0000 7fcbfe99e740 10 client.25729 fetch_fsmap learned FSMap version 28 2024-03-18T16:10:38.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.974+0000 7fcbfe99e740 1 -- 172.21.15.39:0/2864726668 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e1ed6f07d0 con 0x55e1ed9ada90 2024-03-18T16:10:38.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.975+0000 7fcbdaa00640 1 -- 172.21.15.39:0/2864726668 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcbe00c57a0 con 0x55e1ed9ada90 2024-03-18T16:10:38.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.975+0000 7fcbdaa00640 1 -- 172.21.15.39:0/2864726668 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fcbe801a440 con 0x55e1ed9ada90 2024-03-18T16:10:38.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.975+0000 7fcbfe99e740 10 client.25729 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T16:10:38.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.975+0000 7fcbfe99e740 10 client.25729 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:10:38.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.975+0000 7fcbfe99e740 20 client.25729 populate_metadata read hostname 'smithi039' 2024-03-18T16:10:38.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.975+0000 7fcbfe99e740 1 --2- 172.21.15.39:0/2864726668 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55e1ed9c3970 0x55e1ed9c5da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:38.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.975+0000 7fcbfe99e740 4 client.25729 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:10:38.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.975+0000 7fcbfe99e740 1 -- 172.21.15.39:0/2864726668 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e1ed91f990 con 0x55e1ed9c3970 2024-03-18T16:10:38.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.976+0000 7fcbf3e00640 1 --2- 172.21.15.39:0/2864726668 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55e1ed9c3970 0x55e1ed9c5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:38.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.976+0000 7fcbdaa00640 10 client.25729 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:10:38.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.976+0000 7fcbf3e00640 1 --2- 172.21.15.39:0/2864726668 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55e1ed9c3970 0x55e1ed9c5da0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ab73a1e1bb45ecbd server_cookie=90c73f6dcf0fffd9 in_seq=0 out_seq=0 2024-03-18T16:10:38.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.977+0000 7fcbdaa00640 1 -- 172.21.15.39:0/2864726668 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7fcbe00a29c0 con 0x55e1ed9c3970 2024-03-18T16:10:38.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.977+0000 7fcbdaa00640 10 client.25729 handle_command_reply: tid=0 2024-03-18T16:10:38.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.977+0000 7fcbfe99e740 1 client.25729 shutdown 2024-03-18T16:10:38.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.977+0000 7fcbfe99e740 1 -- 172.21.15.39:0/2864726668 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e1ed9ada90 msgr2=0x55e1ed9b7b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:38.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.977+0000 7fcbfe99e740 1 --2- 172.21.15.39:0/2864726668 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e1ed9ada90 0x55e1ed9b7b10 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fcbe8011660 tx=0x7fcbe8011b20 comp rx=0 tx=0).stop 2024-03-18T16:10:38.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.978+0000 7fcbfe99e740 1 -- 172.21.15.39:0/2864726668 shutdown_connections 2024-03-18T16:10:38.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.978+0000 7fcbfe99e740 1 --2- 172.21.15.39:0/2864726668 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55e1ed9c3970 0x55e1ed9c5da0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.978+0000 7fcbfe99e740 1 --2- 172.21.15.39:0/2864726668 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e1ed9ae480 0x55e1ed9b8050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.978+0000 7fcbfe99e740 1 --2- 172.21.15.39:0/2864726668 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e1ed9b35f0 0x55e1ed9bf090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.978+0000 7fcbfe99e740 1 --2- 172.21.15.39:0/2864726668 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e1ed9ada90 0x55e1ed9b7b10 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.978+0000 7fcbfe99e740 1 -- 172.21.15.39:0/2864726668 >> 172.21.15.39:0/2864726668 conn(0x55e1ed9a9000 msgr2=0x55e1ed9a8220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:38.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.978+0000 7fcbdaa00640 0 client.25729 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:10:38.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.978+0000 7fcbfe99e740 1 -- 172.21.15.39:0/2864726668 shutdown_connections 2024-03-18T16:10:38.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.978+0000 7fcbfe99e740 1 -- 172.21.15.39:0/2864726668 wait complete. 2024-03-18T16:10:38.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.978+0000 7fcbfe99e740 20 client.25729 trim_cache size 0 max 16384 2024-03-18T16:10:38.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:10:38 smithi194 ceph-mon[36200]: pgmap v439: 129 pgs: 129 active+clean; 656 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-03-18T16:10:38.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.992+0000 7fcbfe99e740 1 Processor -- start 2024-03-18T16:10:38.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.992+0000 7fcbfe99e740 1 -- start start 2024-03-18T16:10:38.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.993+0000 7fcbfe99e740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e1ed9ada90 0x55e1ed9ca6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:38.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.993+0000 7fcbfe99e740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e1ed9ae480 0x55e1ed9bf050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:38.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.993+0000 7fcbfe99e740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e1ed9b35f0 0x55e1ed9cabe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:38.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.993+0000 7fcbfe99e740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fcbf41b0870 con 0x55e1ed9ae480 2024-03-18T16:10:38.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.993+0000 7fcbfe99e740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fcbf41b23b0 con 0x55e1ed9b35f0 2024-03-18T16:10:38.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.993+0000 7fcbfe99e740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fcbf41b0f40 con 0x55e1ed9ada90 2024-03-18T16:10:38.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.993+0000 7fcbf9c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e1ed9b35f0 0x55e1ed9cabe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:38.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.993+0000 7fcbf9c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e1ed9b35f0 0x55e1ed9cabe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:36242/0 (socket says 172.21.15.39:36242) 2024-03-18T16:10:38.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.993+0000 7fcbf3e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e1ed9ae480 0x55e1ed9bf050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:38.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.993+0000 7fcbf9c00640 1 -- 172.21.15.39:0/995645020 learned_addr learned my addr 172.21.15.39:0/995645020 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:10:38.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.994+0000 7fcbf9200640 1 --2- 172.21.15.39:0/995645020 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e1ed9ada90 0x55e1ed9ca6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:38.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.994+0000 7fcbf9c00640 1 -- 172.21.15.39:0/995645020 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e1ed9ada90 msgr2=0x55e1ed9ca6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:38.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.994+0000 7fcbf9c00640 1 --2- 172.21.15.39:0/995645020 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e1ed9ada90 0x55e1ed9ca6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.994+0000 7fcbf9c00640 1 -- 172.21.15.39:0/995645020 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e1ed9ae480 msgr2=0x55e1ed9bf050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:38.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.994+0000 7fcbf9c00640 1 --2- 172.21.15.39:0/995645020 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e1ed9ae480 0x55e1ed9bf050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.994+0000 7fcbf9c00640 1 -- 172.21.15.39:0/995645020 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e1ed9a8ba0 con 0x55e1ed9b35f0 2024-03-18T16:10:38.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.994+0000 7fcbf9c00640 1 --2- 172.21.15.39:0/995645020 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e1ed9b35f0 0x55e1ed9cabe0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fcbec010790 tx=0x7fcbec015000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4df777b8abcd5b67 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:38.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.995+0000 7fcbe5600640 1 -- 172.21.15.39:0/995645020 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbec01c430 con 0x55e1ed9b35f0 2024-03-18T16:10:38.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.995+0000 7fcbe5600640 1 -- 172.21.15.39:0/995645020 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcbec01cab0 con 0x55e1ed9b35f0 2024-03-18T16:10:38.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.995+0000 7fcbe5600640 1 -- 172.21.15.39:0/995645020 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbec01fad0 con 0x55e1ed9b35f0 2024-03-18T16:10:38.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.996+0000 7fcbfe99e740 1 -- 172.21.15.39:0/995645020 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e1ed9b35f0 msgr2=0x55e1ed9cabe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:38.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.996+0000 7fcbfe99e740 1 --2- 172.21.15.39:0/995645020 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e1ed9b35f0 0x55e1ed9cabe0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fcbec010790 tx=0x7fcbec015000 comp rx=0 tx=0).stop 2024-03-18T16:10:38.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.996+0000 7fcbfe99e740 1 -- 172.21.15.39:0/995645020 shutdown_connections 2024-03-18T16:10:38.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.996+0000 7fcbfe99e740 1 --2- 172.21.15.39:0/995645020 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e1ed9b35f0 0x55e1ed9cabe0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.996+0000 7fcbfe99e740 1 --2- 172.21.15.39:0/995645020 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e1ed9ae480 0x55e1ed9bf050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.996+0000 7fcbfe99e740 1 --2- 172.21.15.39:0/995645020 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e1ed9ada90 0x55e1ed9ca6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:38.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.996+0000 7fcbfe99e740 1 -- 172.21.15.39:0/995645020 >> 172.21.15.39:0/995645020 conn(0x55e1ed9a9000 msgr2=0x55e1ed9acaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:38.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.996+0000 7fcbfe99e740 1 -- 172.21.15.39:0/995645020 shutdown_connections 2024-03-18T16:10:38.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.996+0000 7fcbfe99e740 1 -- 172.21.15.39:0/995645020 wait complete. 2024-03-18T16:10:38.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.997+0000 7fcbfe99e740 1 -- start start 2024-03-18T16:10:38.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.997+0000 7fcbfe99e740 1 Processor -- start 2024-03-18T16:10:38.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.997+0000 7fcbfe99e740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e1ed9ada90 0x55e1ed9bc620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:38.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.997+0000 7fcbfe99e740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e1ed9ae480 0x55e1ed9bcb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:39.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.997+0000 7fcbfe99e740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e1ed9b35f0 0x55e1ed9bdab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:39.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.997+0000 7fcbfe99e740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fcbf41b1ce0 con 0x55e1ed9ae480 2024-03-18T16:10:39.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.997+0000 7fcbfe99e740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fcbf41b1610 con 0x55e1ed9b35f0 2024-03-18T16:10:39.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.997+0000 7fcbfe99e740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fcbf41ae850 con 0x55e1ed9ada90 2024-03-18T16:10:39.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.998+0000 7fcbfe99e740 10 client.0 _ll_register_callbacks cb 0x7fcbe80094b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:10:39.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.998+0000 7fcbf3e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e1ed9ae480 0x55e1ed9bcb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:39.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.998+0000 7fcbf3e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e1ed9ae480 0x55e1ed9bcb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:50494/0 (socket says 172.21.15.39:50494) 2024-03-18T16:10:39.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.998+0000 7fcbf3e00640 1 -- 172.21.15.39:0/3950818121 learned_addr learned my addr 172.21.15.39:0/3950818121 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:10:39.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.998+0000 7fcbf9c00640 1 --2- 172.21.15.39:0/3950818121 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e1ed9b35f0 0x55e1ed9bdab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:39.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.998+0000 7fcbf9c00640 1 -- 172.21.15.39:0/3950818121 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e1ed9ada90 msgr2=0x55e1ed9bc620 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T16:10:39.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.998+0000 7fcbf9c00640 1 --2- 172.21.15.39:0/3950818121 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e1ed9ada90 0x55e1ed9bc620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:39.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.998+0000 7fcbf9c00640 1 -- 172.21.15.39:0/3950818121 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e1ed9ae480 msgr2=0x55e1ed9bcb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:39.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.998+0000 7fcbf9c00640 1 --2- 172.21.15.39:0/3950818121 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e1ed9ae480 0x55e1ed9bcb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:39.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.998+0000 7fcbf9c00640 1 -- 172.21.15.39:0/3950818121 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e1ed9be0a0 con 0x55e1ed9b35f0 2024-03-18T16:10:39.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.998+0000 7fcbf3e00640 1 --2- 172.21.15.39:0/3950818121 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e1ed9ae480 0x55e1ed9bcb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:10:39.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.999+0000 7fcbf9c00640 1 --2- 172.21.15.39:0/3950818121 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e1ed9b35f0 0x55e1ed9bdab0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fcbec010c50 tx=0x7fcbec01c6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=480a0ce9f786d76 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:39.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.999+0000 7fcbdaa00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:10:39.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.999+0000 7fcbdaa00640 1 -- 172.21.15.39:0/3950818121 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbec01f420 con 0x55e1ed9b35f0 2024-03-18T16:10:39.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.999+0000 7fcbdaa00640 1 -- 172.21.15.39:0/3950818121 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fcbec025040 con 0x55e1ed9b35f0 2024-03-18T16:10:39.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:38.999+0000 7fcbfe99e740 1 -- 172.21.15.39:0/3950818121 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcbec01cab0 con 0x55e1ed9b35f0 2024-03-18T16:10:39.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:39.000+0000 7fcbdaa00640 1 -- 172.21.15.39:0/3950818121 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fcbec01c8b0 con 0x55e1ed9b35f0 2024-03-18T16:10:39.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:39.000+0000 7fcbfe99e740 10 client.25703 fetch_fsmap learned FSMap version 28 2024-03-18T16:10:39.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:39.000+0000 7fcbfe99e740 1 -- 172.21.15.39:0/3950818121 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e1ed9bf3a0 con 0x55e1ed9b35f0 2024-03-18T16:10:39.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:39.000+0000 7fcbdaa00640 1 -- 172.21.15.39:0/3950818121 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcbec01cab0 con 0x55e1ed9b35f0 2024-03-18T16:10:39.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:39.000+0000 7fcbdaa00640 1 -- 172.21.15.39:0/3950818121 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fcbec0052b0 con 0x55e1ed9b35f0 2024-03-18T16:10:39.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:39.001+0000 7fcbfe99e740 10 client.25703 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T16:10:39.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:39.001+0000 7fcbfe99e740 10 client.25703 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:10:39.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:39.001+0000 7fcbfe99e740 20 client.25703 populate_metadata read hostname 'smithi039' 2024-03-18T16:10:39.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:39.001+0000 7fcbfe99e740 1 --2- 172.21.15.39:0/3950818121 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55e1eda2a3b0 0x55e1eda2c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:39.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:39.001+0000 7fcbfe99e740 4 client.25703 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:10:39.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:39.001+0000 7fcbfe99e740 1 -- 172.21.15.39:0/3950818121 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e1ed919ce0 con 0x55e1eda2a3b0 2024-03-18T16:10:39.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:39.002+0000 7fcbf9200640 1 --2- 172.21.15.39:0/3950818121 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55e1eda2a3b0 0x55e1eda2c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:39.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:39.002+0000 7fcbf9200640 1 --2- 172.21.15.39:0/3950818121 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55e1eda2a3b0 0x55e1eda2c7d0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b4374b47e38460dc server_cookie=af0cb9241c6db127 in_seq=0 out_seq=0 2024-03-18T16:10:39.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:39.002+0000 7fcbdaa00640 10 client.25703 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:10:39.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:39.002+0000 7fcbdaa00640 1 -- 172.21.15.39:0/3950818121 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e1ed919ce0 con 0x55e1eda2a3b0 2024-03-18T16:10:39.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:39.002+0000 7fcbdaa00640 10 client.25703 handle_command_reply: tid=0 2024-03-18T16:10:39.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:39.003+0000 7fcbfe99e740 1 client.25703 shutdown 2024-03-18T16:10:39.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:39.003+0000 7fcbfe99e740 1 -- 172.21.15.39:0/3950818121 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e1ed9b35f0 msgr2=0x55e1ed9bdab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:39.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:39.003+0000 7fcbfe99e740 1 --2- 172.21.15.39:0/3950818121 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e1ed9b35f0 0x55e1ed9bdab0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fcbec010c50 tx=0x7fcbec01c6c0 comp rx=0 tx=0).stop 2024-03-18T16:10:39.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:39.003+0000 7fcbfe99e740 1 -- 172.21.15.39:0/3950818121 shutdown_connections 2024-03-18T16:10:39.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:39.003+0000 7fcbfe99e740 1 --2- 172.21.15.39:0/3950818121 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55e1eda2a3b0 0x55e1eda2c7d0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:39.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:39.003+0000 7fcbfe99e740 1 --2- 172.21.15.39:0/3950818121 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e1ed9b35f0 0x55e1ed9bdab0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:39.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:39.003+0000 7fcbfe99e740 1 --2- 172.21.15.39:0/3950818121 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e1ed9ae480 0x55e1ed9bcb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:39.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:39.003+0000 7fcbdaa00640 0 client.25703 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:10:39.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:39.003+0000 7fcbfe99e740 1 --2- 172.21.15.39:0/3950818121 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e1ed9ada90 0x55e1ed9bc620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:39.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:39.003+0000 7fcbfe99e740 1 -- 172.21.15.39:0/3950818121 >> 172.21.15.39:0/3950818121 conn(0x55e1ed9a9000 msgr2=0x55e1ed9a7990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:39.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:39.003+0000 7fcbfe99e740 1 -- 172.21.15.39:0/3950818121 shutdown_connections 2024-03-18T16:10:39.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:39.003+0000 7fcbfe99e740 1 -- 172.21.15.39:0/3950818121 wait complete. 2024-03-18T16:10:39.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:39.004+0000 7fcbfe99e740 20 client.25703 trim_cache size 0 max 16384 2024-03-18T16:10:39.007 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:10:39.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:39.005+0000 7fcbfa600640 1 -- 172.21.15.39:0/4247879013 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fcbdc0a2900 msgr2=0x7fcbdc0a4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:39.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:39.005+0000 7fcbfa600640 1 --2- 172.21.15.39:0/4247879013 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fcbdc0a2900 0x7fcbdc0a4dc0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x55e1ed9aece0 tx=0x7fcbec009500 comp rx=0 tx=0).stop 2024-03-18T16:10:39.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:39.005+0000 7fcbfa600640 1 -- 172.21.15.39:0/4247879013 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcbf41007a0 msgr2=0x7fcbf4105f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:39.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:39.005+0000 7fcbfa600640 1 --2- 172.21.15.39:0/4247879013 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcbf41007a0 0x7fcbf4105f00 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fcbe002c8f0 tx=0x7fcbe002c660 comp rx=0 tx=0).stop 2024-03-18T16:10:39.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:39.005+0000 7fcbfa600640 1 -- 172.21.15.39:0/4247879013 shutdown_connections 2024-03-18T16:10:39.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:39.005+0000 7fcbfa600640 1 --2- 172.21.15.39:0/4247879013 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcbf41007a0 0x7fcbf4105f00 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:39.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:39.005+0000 7fcbfa600640 1 --2- 172.21.15.39:0/4247879013 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fcbdc0a2900 0x7fcbdc0a4dc0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:39.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:39.005+0000 7fcbfa600640 1 --2- 172.21.15.39:0/4247879013 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcbf4101140 0x7fcbf4102530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:39.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:39.005+0000 7fcbfa600640 1 --2- 172.21.15.39:0/4247879013 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcbf40ffe40 0x7fcbf41059c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:39.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:39.005+0000 7fcbfa600640 1 -- 172.21.15.39:0/4247879013 >> 172.21.15.39:0/4247879013 conn(0x7fcbf40fba80 msgr2=0x7fcbf40fd580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:39.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:39.007+0000 7fcbfa600640 1 -- 172.21.15.39:0/4247879013 shutdown_connections 2024-03-18T16:10:39.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:39.007+0000 7fcbfa600640 1 -- 172.21.15.39:0/4247879013 wait complete. 2024-03-18T16:10:39.026 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:10:38 smithi039 ceph-mon[31721]: pgmap v439: 129 pgs: 129 active+clean; 656 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-03-18T16:10:39.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:10:38 smithi196 ceph-mon[36307]: pgmap v439: 129 pgs: 129 active+clean; 656 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-03-18T16:10:40.030 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-18T16:10:40.030 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T16:10:40.030 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-18T16:10:40.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.287+0000 7fd3dee00640 1 -- 172.21.15.39:0/2611798552 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd3d8103a20 msgr2=0x7fd3d8103e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:40.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.287+0000 7fd3dee00640 1 --2- 172.21.15.39:0/2611798552 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd3d8103a20 0x7fd3d8103e40 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fd3c8009900 tx=0x7fd3c802bc30 comp rx=0 tx=0).stop 2024-03-18T16:10:40.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.287+0000 7fd3dee00640 1 -- 172.21.15.39:0/2611798552 shutdown_connections 2024-03-18T16:10:40.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.287+0000 7fd3dee00640 1 --2- 172.21.15.39:0/2611798552 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd3d81055d0 0x7fd3d810be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:40.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.287+0000 7fd3dee00640 1 --2- 172.21.15.39:0/2611798552 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd3d8104c10 0x7fd3d8105090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:40.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.287+0000 7fd3dee00640 1 --2- 172.21.15.39:0/2611798552 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd3d8103a20 0x7fd3d8103e40 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:40.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.287+0000 7fd3dee00640 1 -- 172.21.15.39:0/2611798552 >> 172.21.15.39:0/2611798552 conn(0x7fd3d80fef90 msgr2=0x7fd3d81013d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:40.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.287+0000 7fd3dee00640 1 -- 172.21.15.39:0/2611798552 shutdown_connections 2024-03-18T16:10:40.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.287+0000 7fd3dee00640 1 -- 172.21.15.39:0/2611798552 wait complete. 2024-03-18T16:10:40.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.288+0000 7fd3dee00640 1 Processor -- start 2024-03-18T16:10:40.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.288+0000 7fd3dee00640 1 -- start start 2024-03-18T16:10:40.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.289+0000 7fd3dee00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd3d8103a20 0x7fd3d8199d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:40.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.289+0000 7fd3dee00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd3d8104c10 0x7fd3d819a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:40.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.289+0000 7fd3dee00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd3d81055d0 0x7fd3d81a1310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:40.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.289+0000 7fd3dee00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fd3d810f6c0 con 0x7fd3d8103a20 2024-03-18T16:10:40.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.289+0000 7fd3dee00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fd3d810f540 con 0x7fd3d81055d0 2024-03-18T16:10:40.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.289+0000 7fd3dee00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fd3d810f840 con 0x7fd3d8104c10 2024-03-18T16:10:40.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.289+0000 7fd3d7400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd3d8104c10 0x7fd3d819a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:40.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.289+0000 7fd3d7e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd3d8103a20 0x7fd3d8199d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:40.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.289+0000 7fd3d7400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd3d8104c10 0x7fd3d819a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:35108/0 (socket says 172.21.15.39:35108) 2024-03-18T16:10:40.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.289+0000 7fd3d7400640 1 -- 172.21.15.39:0/2651358983 learned_addr learned my addr 172.21.15.39:0/2651358983 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:10:40.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.289+0000 7fd3dce00640 1 --2- 172.21.15.39:0/2651358983 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd3d81055d0 0x7fd3d81a1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:40.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.290+0000 7fd3d7400640 1 -- 172.21.15.39:0/2651358983 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd3d81055d0 msgr2=0x7fd3d81a1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:40.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.290+0000 7fd3d7400640 1 --2- 172.21.15.39:0/2651358983 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd3d81055d0 0x7fd3d81a1310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:40.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.290+0000 7fd3d7400640 1 -- 172.21.15.39:0/2651358983 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd3d8103a20 msgr2=0x7fd3d8199d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:40.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.290+0000 7fd3d7400640 1 --2- 172.21.15.39:0/2651358983 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd3d8103a20 0x7fd3d8199d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:40.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.290+0000 7fd3d7400640 1 -- 172.21.15.39:0/2651358983 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3d81a1980 con 0x7fd3d8104c10 2024-03-18T16:10:40.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.290+0000 7fd3d7400640 1 --2- 172.21.15.39:0/2651358983 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd3d8104c10 0x7fd3d819a2a0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fd3c400e9f0 tx=0x7fd3c400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:40.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.290+0000 7fd3d4c00640 1 -- 172.21.15.39:0/2651358983 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3c400cd80 con 0x7fd3d8104c10 2024-03-18T16:10:40.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.290+0000 7fd3d4c00640 1 -- 172.21.15.39:0/2651358983 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3c4022070 con 0x7fd3d8104c10 2024-03-18T16:10:40.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.290+0000 7fd3d4c00640 1 -- 172.21.15.39:0/2651358983 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3c400f3c0 con 0x7fd3d8104c10 2024-03-18T16:10:40.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.291+0000 7fd3dee00640 1 -- 172.21.15.39:0/2651358983 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3d8075c80 con 0x7fd3d8104c10 2024-03-18T16:10:40.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.291+0000 7fd3dee00640 1 -- 172.21.15.39:0/2651358983 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3d8076090 con 0x7fd3d8104c10 2024-03-18T16:10:40.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.295+0000 7fd3d4c00640 1 -- 172.21.15.39:0/2651358983 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fd3c401dae0 con 0x7fd3d8104c10 2024-03-18T16:10:40.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.296+0000 7fd3e1a39740 1 Processor -- start 2024-03-18T16:10:40.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.297+0000 7fd3e1a39740 1 -- start start 2024-03-18T16:10:40.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.297+0000 7fd3e1a39740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f8406a57e0 0x55f8406a5c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:40.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.297+0000 7fd3e1a39740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f8406a61d0 0x55f8406aacd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:40.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.297+0000 7fd3e1a39740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8406ab340 0x55f8406ad7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:40.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.297+0000 7fd3e1a39740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55f840608f80 con 0x55f8406ab340 2024-03-18T16:10:40.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.297+0000 7fd3e1a39740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55f8405b1ad0 con 0x55f8406a61d0 2024-03-18T16:10:40.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.297+0000 7fd3e1a39740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55f840605020 con 0x55f8406a57e0 2024-03-18T16:10:40.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.297+0000 7fd3d4c00640 1 --2- 172.21.15.39:0/2651358983 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd3bc0a2800 0x7fd3bc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:40.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.297+0000 7fd3d4c00640 1 -- 172.21.15.39:0/2651358983 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fd3c40d7150 con 0x7fd3d8104c10 2024-03-18T16:10:40.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.298+0000 7fd3dce00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f8406a57e0 0x55f8406a5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:40.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.298+0000 7fd3dce00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f8406a57e0 0x55f8406a5c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:35114/0 (socket says 172.21.15.39:35114) 2024-03-18T16:10:40.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.298+0000 7fd3dce00640 1 -- 172.21.15.39:0/2792333607 learned_addr learned my addr 172.21.15.39:0/2792333607 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:10:40.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.298+0000 7fd3d7e00640 1 --2- 172.21.15.39:0/2792333607 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f8406a61d0 0x55f8406aacd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:40.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.298+0000 7fd3dce00640 1 --2- 172.21.15.39:0/2651358983 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd3bc0a2800 0x7fd3bc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:40.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.298+0000 7fd3d7e00640 1 -- 172.21.15.39:0/2792333607 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f8406a57e0 msgr2=0x55f8406a5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:40.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.298+0000 7fd3d7e00640 1 --2- 172.21.15.39:0/2792333607 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f8406a57e0 0x55f8406a5c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:40.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.298+0000 7fd3d7e00640 1 -- 172.21.15.39:0/2792333607 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8406ab340 msgr2=0x55f8406ad7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:40.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.298+0000 7fd3d7e00640 1 --2- 172.21.15.39:0/2792333607 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8406ab340 0x55f8406ad7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:40.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.298+0000 7fd3d7e00640 1 -- 172.21.15.39:0/2792333607 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f8405c96f0 con 0x55f8406a61d0 2024-03-18T16:10:40.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.299+0000 7fd3d7e00640 1 --2- 172.21.15.39:0/2792333607 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f8406a61d0 0x55f8406aacd0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fd3c8009230 tx=0x7fd3c8009560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b5b2108fa3f4ff8b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:40.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.299+0000 7fd3dce00640 1 --2- 172.21.15.39:0/2651358983 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd3bc0a2800 0x7fd3bc0a4cc0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x55f8406a6a30 tx=0x7fd3d0009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:40.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.299+0000 7fd3cd600640 1 -- 172.21.15.39:0/2792333607 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3c8044020 con 0x55f8406a61d0 2024-03-18T16:10:40.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.299+0000 7fd3cd600640 1 -- 172.21.15.39:0/2792333607 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3c802cb80 con 0x55f8406a61d0 2024-03-18T16:10:40.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.299+0000 7fd3cd600640 1 -- 172.21.15.39:0/2792333607 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3c80046c0 con 0x55f8406a61d0 2024-03-18T16:10:40.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.300+0000 7fd3e1a39740 1 -- 172.21.15.39:0/2792333607 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f8406a61d0 msgr2=0x55f8406aacd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:40.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.300+0000 7fd3e1a39740 1 --2- 172.21.15.39:0/2792333607 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f8406a61d0 0x55f8406aacd0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fd3c8009230 tx=0x7fd3c8009560 comp rx=0 tx=0).stop 2024-03-18T16:10:40.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.300+0000 7fd3e1a39740 1 -- 172.21.15.39:0/2792333607 shutdown_connections 2024-03-18T16:10:40.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.300+0000 7fd3e1a39740 1 --2- 172.21.15.39:0/2792333607 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f8406a61d0 0x55f8406aacd0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:40.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.300+0000 7fd3e1a39740 1 --2- 172.21.15.39:0/2792333607 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8406ab340 0x55f8406ad7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:40.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.300+0000 7fd3e1a39740 1 --2- 172.21.15.39:0/2792333607 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f8406a57e0 0x55f8406a5c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:40.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.300+0000 7fd3e1a39740 1 -- 172.21.15.39:0/2792333607 >> 172.21.15.39:0/2792333607 conn(0x55f84066d930 msgr2=0x55f8406a1150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:40.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.301+0000 7fd3e1a39740 1 -- 172.21.15.39:0/2792333607 shutdown_connections 2024-03-18T16:10:40.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.302+0000 7fd3e1a39740 1 -- 172.21.15.39:0/2792333607 wait complete. 2024-03-18T16:10:40.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.302+0000 7fd3e1a39740 1 -- start start 2024-03-18T16:10:40.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.302+0000 7fd3e1a39740 1 Processor -- start 2024-03-18T16:10:40.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.303+0000 7fd3e1a39740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f8406a57e0 0x55f8406afb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:40.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.303+0000 7fd3e1a39740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f8406a61d0 0x55f8406b0090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:40.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.303+0000 7fd3d7e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f8406a57e0 0x55f8406afb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:40.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.303+0000 7fd3d7e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f8406a57e0 0x55f8406afb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:35124/0 (socket says 172.21.15.39:35124) 2024-03-18T16:10:40.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.303+0000 7fd3e1a39740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8406ab340 0x55f8406b70d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:40.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.303+0000 7fd3d7400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f8406a61d0 0x55f8406b0090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:40.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.303+0000 7fd3d7400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f8406a61d0 0x55f8406b0090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:36290/0 (socket says 172.21.15.39:36290) 2024-03-18T16:10:40.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.304+0000 7fd3d7e00640 1 -- 172.21.15.39:0/461236074 learned_addr learned my addr 172.21.15.39:0/461236074 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:10:40.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.304+0000 7fd3e1a39740 1 -- 172.21.15.39:0/461236074 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55f8406ae290 con 0x55f8406ab340 2024-03-18T16:10:40.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.304+0000 7fd3e1a39740 1 -- 172.21.15.39:0/461236074 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55f8405c4f30 con 0x55f8406a61d0 2024-03-18T16:10:40.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.304+0000 7fd3e1a39740 1 -- 172.21.15.39:0/461236074 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55f8406ae410 con 0x55f8406a57e0 2024-03-18T16:10:40.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.304+0000 7fd3e1a39740 10 client.0 _ll_register_callbacks cb 0x55f8405b2bc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:10:40.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.304+0000 7fd3d7e00640 1 -- 172.21.15.39:0/461236074 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f8406a61d0 msgr2=0x55f8406b0090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:40.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.304+0000 7fd3d7e00640 1 --2- 172.21.15.39:0/461236074 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f8406a61d0 0x55f8406b0090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:40.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.304+0000 7fd3d7e00640 1 -- 172.21.15.39:0/461236074 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8406ab340 msgr2=0x55f8406b70d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:10:40.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.304+0000 7fd3dce00640 1 --2- 172.21.15.39:0/461236074 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8406ab340 0x55f8406b70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:40.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.304+0000 7fd3d7e00640 1 --2- 172.21.15.39:0/461236074 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8406ab340 0x55f8406b70d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:40.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.304+0000 7fd3d7e00640 1 -- 172.21.15.39:0/461236074 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f8403e57e0 con 0x55f8406a57e0 2024-03-18T16:10:40.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.304+0000 7fd3dce00640 1 --2- 172.21.15.39:0/461236074 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8406ab340 0x55f8406b70d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:10:40.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.305+0000 7fd3d7e00640 1 --2- 172.21.15.39:0/461236074 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f8406a57e0 0x55f8406afb50 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fd3c8009900 tx=0x7fd3c8040a10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2783cc43dda20d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:40.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.305+0000 7fd3c2000640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:10:40.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.305+0000 7fd3c2000640 1 -- 172.21.15.39:0/461236074 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3c802cd30 con 0x55f8406a57e0 2024-03-18T16:10:40.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.305+0000 7fd3c2000640 1 -- 172.21.15.39:0/461236074 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fd3c8040b80 con 0x55f8406a57e0 2024-03-18T16:10:40.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.305+0000 7fd3e1a39740 1 -- 172.21.15.39:0/461236074 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd3c802cb80 con 0x55f8406a57e0 2024-03-18T16:10:40.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.306+0000 7fd3c2000640 1 -- 172.21.15.39:0/461236074 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd3c805c050 con 0x55f8406a57e0 2024-03-18T16:10:40.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.307+0000 7fd3e1a39740 10 client.25744 fetch_fsmap learned FSMap version 28 2024-03-18T16:10:40.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.307+0000 7fd3e1a39740 1 -- 172.21.15.39:0/461236074 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f840445f70 con 0x55f8406a57e0 2024-03-18T16:10:40.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.307+0000 7fd3c2000640 1 -- 172.21.15.39:0/461236074 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3c802cb80 con 0x55f8406a57e0 2024-03-18T16:10:40.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.307+0000 7fd3c2000640 1 -- 172.21.15.39:0/461236074 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fd3c8034440 con 0x55f8406a57e0 2024-03-18T16:10:40.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.307+0000 7fd3e1a39740 10 client.25744 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T16:10:40.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.307+0000 7fd3e1a39740 10 client.25744 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:10:40.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.307+0000 7fd3e1a39740 20 client.25744 populate_metadata read hostname 'smithi039' 2024-03-18T16:10:40.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.307+0000 7fd3e1a39740 1 --2- 172.21.15.39:0/461236074 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55f8406bb5c0 0x55f8406bd9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:40.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.307+0000 7fd3e1a39740 4 client.25744 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:10:40.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.307+0000 7fd3e1a39740 1 -- 172.21.15.39:0/461236074 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f8406164c0 con 0x55f8406bb5c0 2024-03-18T16:10:40.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.308+0000 7fd3d7400640 1 --2- 172.21.15.39:0/461236074 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55f8406bb5c0 0x55f8406bd9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:40.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.308+0000 7fd3d7400640 1 --2- 172.21.15.39:0/461236074 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55f8406bb5c0 0x55f8406bd9b0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7897b7dbfef27d89 server_cookie=e0c8e58fc7900ca in_seq=0 out_seq=0 2024-03-18T16:10:40.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.308+0000 7fd3c2000640 10 client.25744 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:10:40.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.309+0000 7fd3c2000640 1 -- 172.21.15.39:0/461236074 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f8406164c0 con 0x55f8406bb5c0 2024-03-18T16:10:40.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.309+0000 7fd3c2000640 10 client.25744 handle_command_reply: tid=0 2024-03-18T16:10:40.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.309+0000 7fd3e1a39740 1 client.25744 shutdown 2024-03-18T16:10:40.312 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.310+0000 7fd3e1a39740 1 -- 172.21.15.39:0/461236074 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f8406a57e0 msgr2=0x55f8406afb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:40.312 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.310+0000 7fd3e1a39740 1 --2- 172.21.15.39:0/461236074 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f8406a57e0 0x55f8406afb50 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fd3c8009900 tx=0x7fd3c8040a10 comp rx=0 tx=0).stop 2024-03-18T16:10:40.312 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.310+0000 7fd3e1a39740 1 -- 172.21.15.39:0/461236074 shutdown_connections 2024-03-18T16:10:40.312 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.310+0000 7fd3e1a39740 1 --2- 172.21.15.39:0/461236074 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55f8406bb5c0 0x55f8406bd9b0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:40.312 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.310+0000 7fd3e1a39740 1 --2- 172.21.15.39:0/461236074 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f8406a61d0 0x55f8406b0090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:40.312 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.310+0000 7fd3e1a39740 1 --2- 172.21.15.39:0/461236074 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8406ab340 0x55f8406b70d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:40.312 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.310+0000 7fd3e1a39740 1 --2- 172.21.15.39:0/461236074 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f8406a57e0 0x55f8406afb50 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:40.312 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.310+0000 7fd3e1a39740 1 -- 172.21.15.39:0/461236074 >> 172.21.15.39:0/461236074 conn(0x55f84066d930 msgr2=0x55f8406a0350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:40.312 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.310+0000 7fd3c2000640 0 client.25744 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:10:40.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.310+0000 7fd3e1a39740 1 -- 172.21.15.39:0/461236074 shutdown_connections 2024-03-18T16:10:40.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.311+0000 7fd3e1a39740 1 -- 172.21.15.39:0/461236074 wait complete. 2024-03-18T16:10:40.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.311+0000 7fd3e1a39740 20 client.25744 trim_cache size 0 max 16384 2024-03-18T16:10:40.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.349+0000 7fd3e1a39740 1 Processor -- start 2024-03-18T16:10:40.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.350+0000 7fd3e1a39740 1 -- start start 2024-03-18T16:10:40.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.350+0000 7fd3e1a39740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8406a57e0 0x55f8406af170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:40.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.350+0000 7fd3e1a39740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f8406a61d0 0x55f8406b7090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:40.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.350+0000 7fd3e1a39740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f8406ab340 0x55f8406af6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:40.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.350+0000 7fd3e1a39740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fd3d81a82f0 con 0x55f8406a57e0 2024-03-18T16:10:40.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.350+0000 7fd3e1a39740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fd3d81a9e30 con 0x55f8406ab340 2024-03-18T16:10:40.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.350+0000 7fd3e1a39740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fd3d81a89c0 con 0x55f8406a61d0 2024-03-18T16:10:40.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.350+0000 7fd3d7400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f8406a61d0 0x55f8406b7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:40.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.351+0000 7fd3d7400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f8406a61d0 0x55f8406b7090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:35134/0 (socket says 172.21.15.39:35134) 2024-03-18T16:10:40.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.351+0000 7fd3d7400640 1 -- 172.21.15.39:0/2177206478 learned_addr learned my addr 172.21.15.39:0/2177206478 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:10:40.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.351+0000 7fd3dce00640 1 --2- 172.21.15.39:0/2177206478 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f8406ab340 0x55f8406af6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:40.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.351+0000 7fd3d7e00640 1 --2- 172.21.15.39:0/2177206478 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8406a57e0 0x55f8406af170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:40.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.351+0000 7fd3d7400640 1 -- 172.21.15.39:0/2177206478 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f8406ab340 msgr2=0x55f8406af6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:40.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.351+0000 7fd3d7400640 1 --2- 172.21.15.39:0/2177206478 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f8406ab340 0x55f8406af6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:40.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.351+0000 7fd3d7400640 1 -- 172.21.15.39:0/2177206478 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8406a57e0 msgr2=0x55f8406af170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:40.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.351+0000 7fd3d7400640 1 --2- 172.21.15.39:0/2177206478 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8406a57e0 0x55f8406af170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:40.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.351+0000 7fd3d7400640 1 -- 172.21.15.39:0/2177206478 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f8402dd170 con 0x55f8406a61d0 2024-03-18T16:10:40.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.351+0000 7fd3dce00640 1 --2- 172.21.15.39:0/2177206478 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f8406ab340 0x55f8406af6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:10:40.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.351+0000 7fd3d7e00640 1 --2- 172.21.15.39:0/2177206478 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8406a57e0 0x55f8406af170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:10:40.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.351+0000 7fd3d7400640 1 --2- 172.21.15.39:0/2177206478 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f8406a61d0 0x55f8406b7090 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fd3c400c6a0 tx=0x7fd3c4010950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d435b4e4d31df73 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:40.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.351+0000 7fd3ccc00640 1 -- 172.21.15.39:0/2177206478 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3c40b7070 con 0x55f8406a61d0 2024-03-18T16:10:40.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.351+0000 7fd3ccc00640 1 -- 172.21.15.39:0/2177206478 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3c40a7850 con 0x55f8406a61d0 2024-03-18T16:10:40.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.351+0000 7fd3ccc00640 1 -- 172.21.15.39:0/2177206478 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3c40a1a90 con 0x55f8406a61d0 2024-03-18T16:10:40.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.352+0000 7fd3e1a39740 1 -- 172.21.15.39:0/2177206478 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f8406a61d0 msgr2=0x55f8406b7090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:40.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.352+0000 7fd3e1a39740 1 --2- 172.21.15.39:0/2177206478 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f8406a61d0 0x55f8406b7090 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fd3c400c6a0 tx=0x7fd3c4010950 comp rx=0 tx=0).stop 2024-03-18T16:10:40.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.352+0000 7fd3e1a39740 1 -- 172.21.15.39:0/2177206478 shutdown_connections 2024-03-18T16:10:40.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.352+0000 7fd3e1a39740 1 --2- 172.21.15.39:0/2177206478 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f8406ab340 0x55f8406af6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:40.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.352+0000 7fd3e1a39740 1 --2- 172.21.15.39:0/2177206478 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f8406a61d0 0x55f8406b7090 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:40.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.352+0000 7fd3e1a39740 1 --2- 172.21.15.39:0/2177206478 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8406a57e0 0x55f8406af170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:40.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.352+0000 7fd3e1a39740 1 -- 172.21.15.39:0/2177206478 >> 172.21.15.39:0/2177206478 conn(0x55f84066d930 msgr2=0x55f8406a0350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:40.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.352+0000 7fd3e1a39740 1 -- 172.21.15.39:0/2177206478 shutdown_connections 2024-03-18T16:10:40.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.352+0000 7fd3e1a39740 1 -- 172.21.15.39:0/2177206478 wait complete. 2024-03-18T16:10:40.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.353+0000 7fd3e1a39740 1 -- start start 2024-03-18T16:10:40.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.353+0000 7fd3e1a39740 1 Processor -- start 2024-03-18T16:10:40.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.353+0000 7fd3e1a39740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8406a57e0 0x55f8406b45e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:40.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.353+0000 7fd3e1a39740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f8406a61d0 0x55f8406b4b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:40.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.353+0000 7fd3e1a39740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f8406ab340 0x55f8406b5a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:40.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.353+0000 7fd3e1a39740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fd3d81a9760 con 0x55f8406a57e0 2024-03-18T16:10:40.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.353+0000 7fd3e1a39740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fd3d81a9090 con 0x55f8406a61d0 2024-03-18T16:10:40.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.353+0000 7fd3e1a39740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fd3d81a62d0 con 0x55f8406ab340 2024-03-18T16:10:40.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.353+0000 7fd3e1a39740 10 client.0 _ll_register_callbacks cb 0x7fd3d0011440 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:10:40.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.354+0000 7fd3d7400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f8406a61d0 0x55f8406b4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:40.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.354+0000 7fd3d7400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f8406a61d0 0x55f8406b4b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:36312/0 (socket says 172.21.15.39:36312) 2024-03-18T16:10:40.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.354+0000 7fd3d7400640 1 -- 172.21.15.39:0/4209548328 learned_addr learned my addr 172.21.15.39:0/4209548328 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:10:40.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.354+0000 7fd3d7e00640 1 --2- 172.21.15.39:0/4209548328 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8406a57e0 0x55f8406b45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:40.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.354+0000 7fd3d7400640 1 -- 172.21.15.39:0/4209548328 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f8406ab340 msgr2=0x55f8406b5a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:40.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.354+0000 7fd3d7400640 1 --2- 172.21.15.39:0/4209548328 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f8406ab340 0x55f8406b5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:40.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.354+0000 7fd3d7400640 1 -- 172.21.15.39:0/4209548328 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8406a57e0 msgr2=0x55f8406b45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:40.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.354+0000 7fd3d7400640 1 --2- 172.21.15.39:0/4209548328 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8406a57e0 0x55f8406b45e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:40.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.354+0000 7fd3d7400640 1 -- 172.21.15.39:0/4209548328 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f8406b6040 con 0x55f8406a61d0 2024-03-18T16:10:40.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.354+0000 7fd3d7e00640 1 --2- 172.21.15.39:0/4209548328 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8406a57e0 0x55f8406b45e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:10:40.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.354+0000 7fd3d7400640 1 --2- 172.21.15.39:0/4209548328 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f8406a61d0 0x55f8406b4b20 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fd3c40ab970 tx=0x7fd3c4010870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=83220d80d4c2fab2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:40.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.354+0000 7fd3c2000640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:10:40.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.355+0000 7fd3c2000640 1 -- 172.21.15.39:0/4209548328 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3c40c0020 con 0x55f8406a61d0 2024-03-18T16:10:40.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.355+0000 7fd3e1a39740 1 -- 172.21.15.39:0/4209548328 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd3c40a7850 con 0x55f8406a61d0 2024-03-18T16:10:40.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.355+0000 7fd3c2000640 1 -- 172.21.15.39:0/4209548328 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fd3c409fd70 con 0x55f8406a61d0 2024-03-18T16:10:40.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.355+0000 7fd3c2000640 1 -- 172.21.15.39:0/4209548328 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd3c40b7040 con 0x55f8406a61d0 2024-03-18T16:10:40.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.355+0000 7fd3e1a39740 10 client.25721 fetch_fsmap learned FSMap version 28 2024-03-18T16:10:40.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.355+0000 7fd3e1a39740 1 -- 172.21.15.39:0/4209548328 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f8406b7370 con 0x55f8406a61d0 2024-03-18T16:10:40.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.355+0000 7fd3c2000640 1 -- 172.21.15.39:0/4209548328 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd3c40a7850 con 0x55f8406a61d0 2024-03-18T16:10:40.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.356+0000 7fd3c2000640 1 -- 172.21.15.39:0/4209548328 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fd3c40bc290 con 0x55f8406a61d0 2024-03-18T16:10:40.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.356+0000 7fd3e1a39740 10 client.25721 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T16:10:40.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.356+0000 7fd3e1a39740 10 client.25721 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:10:40.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.356+0000 7fd3e1a39740 20 client.25721 populate_metadata read hostname 'smithi039' 2024-03-18T16:10:40.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.356+0000 7fd3e1a39740 1 --2- 172.21.15.39:0/4209548328 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55f840721e60 0x55f840724280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:40.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.356+0000 7fd3e1a39740 4 client.25721 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-18T16:10:40.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.356+0000 7fd3e1a39740 1 -- 172.21.15.39:0/4209548328 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55f840610810 con 0x55f840721e60 2024-03-18T16:10:40.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.356+0000 7fd3d7e00640 1 --2- 172.21.15.39:0/4209548328 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55f840721e60 0x55f840724280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:40.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.357+0000 7fd3d7e00640 1 --2- 172.21.15.39:0/4209548328 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55f840721e60 0x55f840724280 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8314c15c02f44a98 server_cookie=f4e0df855b65ba69 in_seq=0 out_seq=0 2024-03-18T16:10:40.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.357+0000 7fd3c2000640 10 client.25721 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:10:40.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.358+0000 7fd3c2000640 1 -- 172.21.15.39:0/4209548328 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55f840610810 con 0x55f840721e60 2024-03-18T16:10:40.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.358+0000 7fd3c2000640 10 client.25721 handle_command_reply: tid=0 2024-03-18T16:10:40.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.358+0000 7fd3e1a39740 1 client.25721 shutdown 2024-03-18T16:10:40.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.358+0000 7fd3e1a39740 1 -- 172.21.15.39:0/4209548328 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f8406a61d0 msgr2=0x55f8406b4b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:40.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.358+0000 7fd3e1a39740 1 --2- 172.21.15.39:0/4209548328 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f8406a61d0 0x55f8406b4b20 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fd3c40ab970 tx=0x7fd3c4010870 comp rx=0 tx=0).stop 2024-03-18T16:10:40.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.358+0000 7fd3e1a39740 1 -- 172.21.15.39:0/4209548328 shutdown_connections 2024-03-18T16:10:40.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.358+0000 7fd3e1a39740 1 --2- 172.21.15.39:0/4209548328 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55f840721e60 0x55f840724280 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:40.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.358+0000 7fd3e1a39740 1 --2- 172.21.15.39:0/4209548328 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f8406a61d0 0x55f8406b4b20 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:40.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.358+0000 7fd3e1a39740 1 --2- 172.21.15.39:0/4209548328 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f8406ab340 0x55f8406b5a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:40.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.358+0000 7fd3e1a39740 1 --2- 172.21.15.39:0/4209548328 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f8406a57e0 0x55f8406b45e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:40.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.358+0000 7fd3e1a39740 1 -- 172.21.15.39:0/4209548328 >> 172.21.15.39:0/4209548328 conn(0x55f84066d930 msgr2=0x55f8406a4390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:40.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.358+0000 7fd3c2000640 0 client.25721 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:10:40.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.359+0000 7fd3e1a39740 1 -- 172.21.15.39:0/4209548328 shutdown_connections 2024-03-18T16:10:40.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.359+0000 7fd3e1a39740 1 -- 172.21.15.39:0/4209548328 wait complete. 2024-03-18T16:10:40.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.359+0000 7fd3e1a39740 20 client.25721 trim_cache size 0 max 16384 2024-03-18T16:10:40.362 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-03-18T16:10:40.362 INFO:teuthology.orchestra.run.smithi039.stdout: "return_code": 0, 2024-03-18T16:10:40.362 INFO:teuthology.orchestra.run.smithi039.stdout: "scrub_tag": "d4dc04d7-8f16-41db-9e39-fd86f00c8ce3", 2024-03-18T16:10:40.362 INFO:teuthology.orchestra.run.smithi039.stdout: "mode": "asynchronous" 2024-03-18T16:10:40.362 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-03-18T16:10:40.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.361+0000 7fd3dee00640 1 -- 172.21.15.39:0/2651358983 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd3bc0a2800 msgr2=0x7fd3bc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:40.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.361+0000 7fd3dee00640 1 --2- 172.21.15.39:0/2651358983 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd3bc0a2800 0x7fd3bc0a4cc0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x55f8406a6a30 tx=0x7fd3d0009500 comp rx=0 tx=0).stop 2024-03-18T16:10:40.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.361+0000 7fd3dee00640 1 -- 172.21.15.39:0/2651358983 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd3d8104c10 msgr2=0x7fd3d819a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:40.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.361+0000 7fd3dee00640 1 --2- 172.21.15.39:0/2651358983 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd3d8104c10 0x7fd3d819a2a0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fd3c400e9f0 tx=0x7fd3c400eeb0 comp rx=0 tx=0).stop 2024-03-18T16:10:40.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.361+0000 7fd3dee00640 1 -- 172.21.15.39:0/2651358983 shutdown_connections 2024-03-18T16:10:40.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.361+0000 7fd3dee00640 1 --2- 172.21.15.39:0/2651358983 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd3d81055d0 0x7fd3d81a1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:40.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.361+0000 7fd3dee00640 1 --2- 172.21.15.39:0/2651358983 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd3bc0a2800 0x7fd3bc0a4cc0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:40.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.361+0000 7fd3dee00640 1 --2- 172.21.15.39:0/2651358983 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd3d8104c10 0x7fd3d819a2a0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:40.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.361+0000 7fd3dee00640 1 --2- 172.21.15.39:0/2651358983 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd3d8103a20 0x7fd3d8199d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:40.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.361+0000 7fd3dee00640 1 -- 172.21.15.39:0/2651358983 >> 172.21.15.39:0/2651358983 conn(0x7fd3d80fef90 msgr2=0x7fd3d8107f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:40.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.363+0000 7fd3dee00640 1 -- 172.21.15.39:0/2651358983 shutdown_connections 2024-03-18T16:10:40.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.363+0000 7fd3dee00640 1 -- 172.21.15.39:0/2651358983 wait complete. 2024-03-18T16:10:40.389 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T16:10:40.389 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T16:10:40.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.630+0000 7f647ec00640 1 -- 172.21.15.39:0/98183702 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6478104ad0 msgr2=0x7f6478104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:40.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.630+0000 7f647ec00640 1 --2- 172.21.15.39:0/98183702 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6478104ad0 0x7f6478104f50 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f6464009900 tx=0x7f646402bc30 comp rx=0 tx=0).stop 2024-03-18T16:10:40.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.630+0000 7f647ec00640 1 -- 172.21.15.39:0/98183702 shutdown_connections 2024-03-18T16:10:40.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.630+0000 7f647ec00640 1 --2- 172.21.15.39:0/98183702 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6478105490 0x7f647810bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:40.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.630+0000 7f647ec00640 1 --2- 172.21.15.39:0/98183702 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6478104ad0 0x7f6478104f50 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:40.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.630+0000 7f647ec00640 1 --2- 172.21.15.39:0/98183702 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f64781038e0 0x7f6478103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:40.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.630+0000 7f647ec00640 1 -- 172.21.15.39:0/98183702 >> 172.21.15.39:0/98183702 conn(0x7f64780fee50 msgr2=0x7f6478101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:40.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.630+0000 7f647ec00640 1 -- 172.21.15.39:0/98183702 shutdown_connections 2024-03-18T16:10:40.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.630+0000 7f647ec00640 1 -- 172.21.15.39:0/98183702 wait complete. 2024-03-18T16:10:40.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.632+0000 7f647ec00640 1 Processor -- start 2024-03-18T16:10:40.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.632+0000 7f647ec00640 1 -- start start 2024-03-18T16:10:40.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.632+0000 7f647ec00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f64781038e0 0x7f6478079590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:40.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.632+0000 7f647ec00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6478104ad0 0x7f6478079ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:40.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.632+0000 7f647ec00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6478105490 0x7f6478076100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:40.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.632+0000 7f647ec00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f647810f590 con 0x7f6478104ad0 2024-03-18T16:10:40.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.632+0000 7f647ec00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f647810f410 con 0x7f6478105490 2024-03-18T16:10:40.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.632+0000 7f647ec00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f647810f710 con 0x7f64781038e0 2024-03-18T16:10:40.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.633+0000 7f647d800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f64781038e0 0x7f6478079590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:40.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.633+0000 7f647d800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f64781038e0 0x7f6478079590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:35162/0 (socket says 172.21.15.39:35162) 2024-03-18T16:10:40.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.633+0000 7f647ce00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6478104ad0 0x7f6478079ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:40.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.633+0000 7f647d800640 1 -- 172.21.15.39:0/489535886 learned_addr learned my addr 172.21.15.39:0/489535886 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:10:40.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.633+0000 7f647e200640 1 --2- 172.21.15.39:0/489535886 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6478105490 0x7f6478076100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:40.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.633+0000 7f647d800640 1 -- 172.21.15.39:0/489535886 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6478105490 msgr2=0x7f6478076100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:40.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.633+0000 7f647d800640 1 --2- 172.21.15.39:0/489535886 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6478105490 0x7f6478076100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:40.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.633+0000 7f647d800640 1 -- 172.21.15.39:0/489535886 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6478104ad0 msgr2=0x7f6478079ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:40.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.633+0000 7f647d800640 1 --2- 172.21.15.39:0/489535886 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6478104ad0 0x7f6478079ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:40.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.633+0000 7f647d800640 1 -- 172.21.15.39:0/489535886 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64780769c0 con 0x7f64781038e0 2024-03-18T16:10:40.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.633+0000 7f647d800640 1 --2- 172.21.15.39:0/489535886 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f64781038e0 0x7f6478079590 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f646c00e9c0 tx=0x7f646c00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:40.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.634+0000 7f6476000640 1 -- 172.21.15.39:0/489535886 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f646c00cd80 con 0x7f64781038e0 2024-03-18T16:10:40.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.634+0000 7f6476000640 1 -- 172.21.15.39:0/489535886 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f646c0035f0 con 0x7f64781038e0 2024-03-18T16:10:40.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.634+0000 7f6476000640 1 -- 172.21.15.39:0/489535886 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f646c010690 con 0x7f64781038e0 2024-03-18T16:10:40.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.634+0000 7f647ec00640 1 -- 172.21.15.39:0/489535886 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64781a5ff0 con 0x7f64781038e0 2024-03-18T16:10:40.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.634+0000 7f647ec00640 1 -- 172.21.15.39:0/489535886 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64781a63a0 con 0x7f64781038e0 2024-03-18T16:10:40.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.637+0000 7f6482bfd740 1 Processor -- start 2024-03-18T16:10:40.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.637+0000 7f6482bfd740 1 -- start start 2024-03-18T16:10:40.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.637+0000 7f6482bfd740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55df1c38ca10 0x55df1c38ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:40.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.638+0000 7f6482bfd740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55df1c38d400 0x55df1c391f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:40.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.638+0000 7f6482bfd740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55df1c392570 0x55df1c394a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:40.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.638+0000 7f6482bfd740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55df1c2f11f0 con 0x55df1c38d400 2024-03-18T16:10:40.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.638+0000 7f6482bfd740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55df1c299a80 con 0x55df1c38ca10 2024-03-18T16:10:40.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.638+0000 7f6482bfd740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55df1c2ed290 con 0x55df1c392570 2024-03-18T16:10:40.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.638+0000 7f647e200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55df1c38ca10 0x55df1c38ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:40.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.638+0000 7f647e200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55df1c38ca10 0x55df1c38ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:36326/0 (socket says 172.21.15.39:36326) 2024-03-18T16:10:40.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.638+0000 7f647e200640 1 -- 172.21.15.39:0/2768710735 learned_addr learned my addr 172.21.15.39:0/2768710735 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:10:40.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.638+0000 7f647ce00640 1 --2- 172.21.15.39:0/2768710735 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55df1c392570 0x55df1c394a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:40.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.638+0000 7f647e200640 1 -- 172.21.15.39:0/2768710735 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55df1c392570 msgr2=0x55df1c394a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:40.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.638+0000 7f647e200640 1 --2- 172.21.15.39:0/2768710735 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55df1c392570 0x55df1c394a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:40.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.638+0000 7f647e200640 1 -- 172.21.15.39:0/2768710735 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55df1c38d400 msgr2=0x55df1c391f00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:10:40.641 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.638+0000 7f647e200640 1 --2- 172.21.15.39:0/2768710735 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55df1c38d400 0x55df1c391f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:40.641 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.638+0000 7f647e200640 1 -- 172.21.15.39:0/2768710735 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55df1c0cf7d0 con 0x55df1c38ca10 2024-03-18T16:10:40.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.639+0000 7f647e200640 1 --2- 172.21.15.39:0/2768710735 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55df1c38ca10 0x55df1c38ce30 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f6470009fd0 tx=0x7f64700118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d74333a59cde0ba2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:40.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.639+0000 7f646aa00640 1 -- 172.21.15.39:0/2768710735 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6470004d20 con 0x55df1c38ca10 2024-03-18T16:10:40.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.639+0000 7f646aa00640 1 -- 172.21.15.39:0/2768710735 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6470004050 con 0x55df1c38ca10 2024-03-18T16:10:40.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.639+0000 7f646aa00640 1 -- 172.21.15.39:0/2768710735 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6470005280 con 0x55df1c38ca10 2024-03-18T16:10:40.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.639+0000 7f6476000640 1 -- 172.21.15.39:0/489535886 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f646c0037a0 con 0x7f64781038e0 2024-03-18T16:10:40.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.640+0000 7f6482bfd740 1 -- 172.21.15.39:0/2768710735 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55df1c38ca10 msgr2=0x55df1c38ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:40.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.640+0000 7f6482bfd740 1 --2- 172.21.15.39:0/2768710735 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55df1c38ca10 0x55df1c38ce30 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f6470009fd0 tx=0x7f64700118d0 comp rx=0 tx=0).stop 2024-03-18T16:10:40.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.640+0000 7f6482bfd740 1 -- 172.21.15.39:0/2768710735 shutdown_connections 2024-03-18T16:10:40.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.640+0000 7f6482bfd740 1 --2- 172.21.15.39:0/2768710735 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55df1c392570 0x55df1c394a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:40.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.640+0000 7f6482bfd740 1 --2- 172.21.15.39:0/2768710735 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55df1c38d400 0x55df1c391f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:40.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.640+0000 7f6482bfd740 1 --2- 172.21.15.39:0/2768710735 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55df1c38ca10 0x55df1c38ce30 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:40.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.640+0000 7f6482bfd740 1 -- 172.21.15.39:0/2768710735 >> 172.21.15.39:0/2768710735 conn(0x55df1c387f80 msgr2=0x55df1c388380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:40.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.640+0000 7f6482bfd740 1 -- 172.21.15.39:0/2768710735 shutdown_connections 2024-03-18T16:10:40.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.640+0000 7f6482bfd740 1 -- 172.21.15.39:0/2768710735 wait complete. 2024-03-18T16:10:40.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.641+0000 7f6482bfd740 1 -- start start 2024-03-18T16:10:40.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.641+0000 7f6482bfd740 1 Processor -- start 2024-03-18T16:10:40.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.642+0000 7f6482bfd740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55df1c38ca10 0x55df1c397000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:40.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.642+0000 7f6482bfd740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55df1c38d400 0x55df1c397540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:40.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.642+0000 7f6482bfd740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55df1c392570 0x55df1c39e580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:40.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.642+0000 7f6482bfd740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55df1c395520 con 0x55df1c392570 2024-03-18T16:10:40.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.642+0000 7f6482bfd740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55df1c2aced0 con 0x55df1c38ca10 2024-03-18T16:10:40.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.642+0000 7f6482bfd740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55df1c3956a0 con 0x55df1c38d400 2024-03-18T16:10:40.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.642+0000 7f6482bfd740 10 client.0 _ll_register_callbacks cb 0x55df1c2e8220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:10:40.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.642+0000 7f6476000640 1 --2- 172.21.15.39:0/489535886 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f64600a2790 0x7f64600a4c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:40.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.642+0000 7f647d800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55df1c38d400 0x55df1c397540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:40.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.642+0000 7f647d800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55df1c38d400 0x55df1c397540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:35176/0 (socket says 172.21.15.39:35176) 2024-03-18T16:10:40.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.642+0000 7f647d800640 1 -- 172.21.15.39:0/2754712825 learned_addr learned my addr 172.21.15.39:0/2754712825 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:10:40.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.642+0000 7f647e200640 1 --2- 172.21.15.39:0/2754712825 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55df1c38ca10 0x55df1c397000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:40.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.642+0000 7f6476000640 1 -- 172.21.15.39:0/489535886 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f646c0cf1f0 con 0x7f64781038e0 2024-03-18T16:10:40.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.643+0000 7f647ce00640 1 --2- 172.21.15.39:0/2754712825 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55df1c392570 0x55df1c39e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:40.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.643+0000 7f647d800640 1 -- 172.21.15.39:0/2754712825 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55df1c38ca10 msgr2=0x55df1c397000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:40.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.643+0000 7f647d800640 1 --2- 172.21.15.39:0/2754712825 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55df1c38ca10 0x55df1c397000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:40.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.643+0000 7f647d800640 1 -- 172.21.15.39:0/2754712825 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55df1c392570 msgr2=0x55df1c39e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:40.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.643+0000 7f647d800640 1 --2- 172.21.15.39:0/2754712825 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55df1c392570 0x55df1c39e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:40.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.643+0000 7f647d800640 1 -- 172.21.15.39:0/2754712825 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55df1c0053f0 con 0x55df1c38d400 2024-03-18T16:10:40.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.643+0000 7f647ce00640 1 --2- 172.21.15.39:0/2754712825 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55df1c392570 0x55df1c39e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:10:40.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.643+0000 7f647e200640 1 --2- 172.21.15.39:0/489535886 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f64600a2790 0x7f64600a4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:40.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.643+0000 7f647d800640 1 --2- 172.21.15.39:0/2754712825 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55df1c38d400 0x55df1c397540 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f646c098170 tx=0x7f646c09b2e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6b8bd28023240f8f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:40.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.644+0000 7f645f400640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:10:40.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.644+0000 7f645f400640 1 -- 172.21.15.39:0/2754712825 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f646c0b3070 con 0x55df1c38d400 2024-03-18T16:10:40.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.644+0000 7f645f400640 1 -- 172.21.15.39:0/2754712825 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f646c0860b0 con 0x55df1c38d400 2024-03-18T16:10:40.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.644+0000 7f6482bfd740 1 -- 172.21.15.39:0/2754712825 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6470004050 con 0x55df1c38d400 2024-03-18T16:10:40.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.644+0000 7f645f400640 1 -- 172.21.15.39:0/2754712825 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f646c09b7a0 con 0x55df1c38d400 2024-03-18T16:10:40.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.644+0000 7f647e200640 1 --2- 172.21.15.39:0/489535886 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f64600a2790 0x7f64600a4c50 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f6470009fa0 tx=0x7f6470009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:40.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.644+0000 7f6482bfd740 10 client.25759 fetch_fsmap learned FSMap version 28 2024-03-18T16:10:40.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.644+0000 7f6482bfd740 1 -- 172.21.15.39:0/2754712825 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55df1bfc50f0 con 0x55df1c38d400 2024-03-18T16:10:40.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.645+0000 7f645f400640 1 -- 172.21.15.39:0/2754712825 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6470004050 con 0x55df1c38d400 2024-03-18T16:10:40.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.645+0000 7f645f400640 1 -- 172.21.15.39:0/2754712825 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f646c0afb80 con 0x55df1c38d400 2024-03-18T16:10:40.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.646+0000 7f6482bfd740 10 client.25759 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T16:10:40.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.646+0000 7f6482bfd740 10 client.25759 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:10:40.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.646+0000 7f6482bfd740 20 client.25759 populate_metadata read hostname 'smithi039' 2024-03-18T16:10:40.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.646+0000 7f6482bfd740 1 --2- 172.21.15.39:0/2754712825 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55df1c3a2db0 0x55df1c3a51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:40.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.646+0000 7f6482bfd740 4 client.25759 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:10:40.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.646+0000 7f6482bfd740 1 -- 172.21.15.39:0/2754712825 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55df1c304be0 con 0x55df1c3a2db0 2024-03-18T16:10:40.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.646+0000 7f647ce00640 1 --2- 172.21.15.39:0/2754712825 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55df1c3a2db0 0x55df1c3a51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:40.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.647+0000 7f647ce00640 1 --2- 172.21.15.39:0/2754712825 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55df1c3a2db0 0x55df1c3a51e0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9ea0931d85b54eee server_cookie=ce33eca32d1d341e in_seq=0 out_seq=0 2024-03-18T16:10:40.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.647+0000 7f645f400640 10 client.25759 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:10:40.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.647+0000 7f645f400640 1 -- 172.21.15.39:0/2754712825 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55df1c304be0 con 0x55df1c3a2db0 2024-03-18T16:10:40.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.647+0000 7f645f400640 10 client.25759 handle_command_reply: tid=0 2024-03-18T16:10:40.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.648+0000 7f6482bfd740 1 client.25759 shutdown 2024-03-18T16:10:40.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.648+0000 7f6482bfd740 1 -- 172.21.15.39:0/2754712825 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55df1c38d400 msgr2=0x55df1c397540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:40.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.648+0000 7f6482bfd740 1 --2- 172.21.15.39:0/2754712825 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55df1c38d400 0x55df1c397540 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f646c098170 tx=0x7f646c09b2e0 comp rx=0 tx=0).stop 2024-03-18T16:10:40.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.648+0000 7f6482bfd740 1 -- 172.21.15.39:0/2754712825 shutdown_connections 2024-03-18T16:10:40.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.648+0000 7f6482bfd740 1 --2- 172.21.15.39:0/2754712825 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55df1c3a2db0 0x55df1c3a51e0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:40.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.648+0000 7f6482bfd740 1 --2- 172.21.15.39:0/2754712825 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55df1c392570 0x55df1c39e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:40.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.648+0000 7f6482bfd740 1 --2- 172.21.15.39:0/2754712825 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55df1c38d400 0x55df1c397540 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:40.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.648+0000 7f6482bfd740 1 --2- 172.21.15.39:0/2754712825 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55df1c38ca10 0x55df1c397000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:40.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.648+0000 7f6482bfd740 1 -- 172.21.15.39:0/2754712825 >> 172.21.15.39:0/2754712825 conn(0x55df1c387f80 msgr2=0x55df1c387110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:40.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.648+0000 7f645f400640 0 client.25759 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:10:40.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.648+0000 7f6482bfd740 1 -- 172.21.15.39:0/2754712825 shutdown_connections 2024-03-18T16:10:40.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.648+0000 7f6482bfd740 1 -- 172.21.15.39:0/2754712825 wait complete. 2024-03-18T16:10:40.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.648+0000 7f6482bfd740 20 client.25759 trim_cache size 0 max 16384 2024-03-18T16:10:40.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.676+0000 7f6482bfd740 1 Processor -- start 2024-03-18T16:10:40.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.676+0000 7f6482bfd740 1 -- start start 2024-03-18T16:10:40.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.677+0000 7f6482bfd740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55df1c38ca10 0x55df1c395a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:40.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.677+0000 7f6482bfd740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55df1c38d400 0x55df1c39e540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:40.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.677+0000 7f6482bfd740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55df1c392570 0x55df1c395fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:40.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.677+0000 7f6482bfd740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f64781b0850 con 0x55df1c38ca10 2024-03-18T16:10:40.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.677+0000 7f6482bfd740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f64781b2390 con 0x55df1c38d400 2024-03-18T16:10:40.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.677+0000 7f6482bfd740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f64781b0f20 con 0x55df1c392570 2024-03-18T16:10:40.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.677+0000 7f647ce00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55df1c38d400 0x55df1c39e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:40.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.677+0000 7f647ce00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55df1c38d400 0x55df1c39e540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:36350/0 (socket says 172.21.15.39:36350) 2024-03-18T16:10:40.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.677+0000 7f647e200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55df1c392570 0x55df1c395fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:40.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.677+0000 7f647ce00640 1 -- 172.21.15.39:0/2432802537 learned_addr learned my addr 172.21.15.39:0/2432802537 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:10:40.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.677+0000 7f647ce00640 1 -- 172.21.15.39:0/2432802537 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55df1c392570 msgr2=0x55df1c395fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:40.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.677+0000 7f647ce00640 1 --2- 172.21.15.39:0/2432802537 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55df1c392570 0x55df1c395fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:40.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.678+0000 7f647ce00640 1 -- 172.21.15.39:0/2432802537 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55df1c38ca10 msgr2=0x55df1c395a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:40.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.678+0000 7f647ce00640 1 --2- 172.21.15.39:0/2432802537 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55df1c38ca10 0x55df1c395a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:40.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.678+0000 7f647ce00640 1 -- 172.21.15.39:0/2432802537 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55df1c38c650 con 0x55df1c38d400 2024-03-18T16:10:40.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.678+0000 7f647ce00640 1 --2- 172.21.15.39:0/2432802537 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55df1c38d400 0x55df1c39e540 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f6464005ab0 tx=0x7f6464009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5225633ee0994812 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:40.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.678+0000 7f646a000640 1 -- 172.21.15.39:0/2432802537 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f646402ca00 con 0x55df1c38d400 2024-03-18T16:10:40.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.678+0000 7f646a000640 1 -- 172.21.15.39:0/2432802537 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f646404d030 con 0x55df1c38d400 2024-03-18T16:10:40.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.678+0000 7f646a000640 1 -- 172.21.15.39:0/2432802537 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6464049330 con 0x55df1c38d400 2024-03-18T16:10:40.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.679+0000 7f6482bfd740 1 -- 172.21.15.39:0/2432802537 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55df1c38d400 msgr2=0x55df1c39e540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:40.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.679+0000 7f6482bfd740 1 --2- 172.21.15.39:0/2432802537 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55df1c38d400 0x55df1c39e540 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f6464005ab0 tx=0x7f6464009260 comp rx=0 tx=0).stop 2024-03-18T16:10:40.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.679+0000 7f6482bfd740 1 -- 172.21.15.39:0/2432802537 shutdown_connections 2024-03-18T16:10:40.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.679+0000 7f6482bfd740 1 --2- 172.21.15.39:0/2432802537 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55df1c38d400 0x55df1c39e540 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:40.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.679+0000 7f6482bfd740 1 --2- 172.21.15.39:0/2432802537 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55df1c392570 0x55df1c395fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:40.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.679+0000 7f6482bfd740 1 --2- 172.21.15.39:0/2432802537 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55df1c38ca10 0x55df1c395a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:40.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.679+0000 7f6482bfd740 1 -- 172.21.15.39:0/2432802537 >> 172.21.15.39:0/2432802537 conn(0x55df1c387f80 msgr2=0x55df1c397000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:40.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.679+0000 7f6482bfd740 1 -- 172.21.15.39:0/2432802537 shutdown_connections 2024-03-18T16:10:40.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.679+0000 7f6482bfd740 1 -- 172.21.15.39:0/2432802537 wait complete. 2024-03-18T16:10:40.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.680+0000 7f6482bfd740 1 -- start start 2024-03-18T16:10:40.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.680+0000 7f6482bfd740 1 Processor -- start 2024-03-18T16:10:40.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.681+0000 7f6482bfd740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55df1c38ca10 0x55df1c39bb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:40.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.681+0000 7f6482bfd740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55df1c38d400 0x55df1c39c050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:40.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.681+0000 7f6482bfd740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55df1c392570 0x55df1c39d060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:40.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.681+0000 7f6482bfd740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f64781b1cc0 con 0x55df1c38ca10 2024-03-18T16:10:40.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.681+0000 7f6482bfd740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f64781b15f0 con 0x55df1c38d400 2024-03-18T16:10:40.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.681+0000 7f6482bfd740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f64781ae830 con 0x55df1c392570 2024-03-18T16:10:40.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.681+0000 7f6482bfd740 10 client.0 _ll_register_callbacks cb 0x7f646c097b10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:10:40.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.681+0000 7f647ce00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55df1c38d400 0x55df1c39c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:40.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.681+0000 7f647ce00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55df1c38d400 0x55df1c39c050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:36360/0 (socket says 172.21.15.39:36360) 2024-03-18T16:10:40.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.681+0000 7f647ce00640 1 -- 172.21.15.39:0/1159562317 learned_addr learned my addr 172.21.15.39:0/1159562317 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:10:40.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.681+0000 7f647e200640 1 --2- 172.21.15.39:0/1159562317 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55df1c392570 0x55df1c39d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:40.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.682+0000 7f647d800640 1 --2- 172.21.15.39:0/1159562317 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55df1c38ca10 0x55df1c39bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:40.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.682+0000 7f647ce00640 1 -- 172.21.15.39:0/1159562317 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55df1c392570 msgr2=0x55df1c39d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:40.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.682+0000 7f647ce00640 1 --2- 172.21.15.39:0/1159562317 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55df1c392570 0x55df1c39d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:40.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.682+0000 7f647ce00640 1 -- 172.21.15.39:0/1159562317 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55df1c38ca10 msgr2=0x55df1c39bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:10:40.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.682+0000 7f647ce00640 1 --2- 172.21.15.39:0/1159562317 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55df1c38ca10 0x55df1c39bb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:40.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.682+0000 7f647ce00640 1 -- 172.21.15.39:0/1159562317 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55df1c3889f0 con 0x55df1c38d400 2024-03-18T16:10:40.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.682+0000 7f647ce00640 1 --2- 172.21.15.39:0/1159562317 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55df1c38d400 0x55df1c39c050 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f6464009230 tx=0x7f64640045c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=853d78a278825ab0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:10:40.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.682+0000 7f645f400640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:10:40.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.683+0000 7f645f400640 1 -- 172.21.15.39:0/1159562317 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f646402cc00 con 0x55df1c38d400 2024-03-18T16:10:40.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.683+0000 7f645f400640 1 -- 172.21.15.39:0/1159562317 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f6464049b10 con 0x55df1c38d400 2024-03-18T16:10:40.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.683+0000 7f6482bfd740 1 -- 172.21.15.39:0/1159562317 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f646404d030 con 0x55df1c38d400 2024-03-18T16:10:40.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.683+0000 7f645f400640 1 -- 172.21.15.39:0/1159562317 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6464049d50 con 0x55df1c38d400 2024-03-18T16:10:40.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.683+0000 7f6482bfd740 10 client.25736 fetch_fsmap learned FSMap version 28 2024-03-18T16:10:40.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.683+0000 7f6482bfd740 1 -- 172.21.15.39:0/1159562317 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55df1c39e7c0 con 0x55df1c38d400 2024-03-18T16:10:40.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.684+0000 7f645f400640 1 -- 172.21.15.39:0/1159562317 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f646404d030 con 0x55df1c38d400 2024-03-18T16:10:40.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.684+0000 7f645f400640 1 -- 172.21.15.39:0/1159562317 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f6464004eb0 con 0x55df1c38d400 2024-03-18T16:10:40.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.684+0000 7f6482bfd740 10 client.25736 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T16:10:40.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.684+0000 7f6482bfd740 10 client.25736 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:10:40.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.684+0000 7f6482bfd740 20 client.25736 populate_metadata read hostname 'smithi039' 2024-03-18T16:10:40.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.684+0000 7f6482bfd740 1 --2- 172.21.15.39:0/1159562317 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55df1c4097a0 0x55df1c40bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:10:40.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.684+0000 7f6482bfd740 4 client.25736 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T16:10:40.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.684+0000 7f6482bfd740 1 -- 172.21.15.39:0/1159562317 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55df1c2fe730 con 0x55df1c4097a0 2024-03-18T16:10:40.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.685+0000 7f647d800640 1 --2- 172.21.15.39:0/1159562317 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55df1c4097a0 0x55df1c40bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:10:40.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.685+0000 7f647d800640 1 --2- 172.21.15.39:0/1159562317 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55df1c4097a0 0x55df1c40bbc0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9781cf9d38c602fb server_cookie=74eb509336428550 in_seq=0 out_seq=0 2024-03-18T16:10:40.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.685+0000 7f645f400640 10 client.25736 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:10:40.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.685+0000 7f645f400640 1 -- 172.21.15.39:0/1159562317 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55df1c2fe730 con 0x55df1c4097a0 2024-03-18T16:10:40.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.685+0000 7f645f400640 10 client.25736 handle_command_reply: tid=0 2024-03-18T16:10:40.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.685+0000 7f6482bfd740 1 client.25736 shutdown 2024-03-18T16:10:40.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.686+0000 7f6482bfd740 1 -- 172.21.15.39:0/1159562317 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55df1c38d400 msgr2=0x55df1c39c050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:40.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.686+0000 7f6482bfd740 1 --2- 172.21.15.39:0/1159562317 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55df1c38d400 0x55df1c39c050 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f6464009230 tx=0x7f64640045c0 comp rx=0 tx=0).stop 2024-03-18T16:10:40.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.686+0000 7f6482bfd740 1 -- 172.21.15.39:0/1159562317 shutdown_connections 2024-03-18T16:10:40.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.686+0000 7f6482bfd740 1 --2- 172.21.15.39:0/1159562317 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55df1c4097a0 0x55df1c40bbc0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:40.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.686+0000 7f6482bfd740 1 --2- 172.21.15.39:0/1159562317 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55df1c38d400 0x55df1c39c050 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:40.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.686+0000 7f6482bfd740 1 --2- 172.21.15.39:0/1159562317 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55df1c392570 0x55df1c39d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:40.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.686+0000 7f6482bfd740 1 --2- 172.21.15.39:0/1159562317 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55df1c38ca10 0x55df1c39bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:40.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.686+0000 7f6482bfd740 1 -- 172.21.15.39:0/1159562317 >> 172.21.15.39:0/1159562317 conn(0x55df1c387f80 msgr2=0x55df1c395a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:40.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.686+0000 7f645f400640 0 client.25736 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:10:40.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.686+0000 7f6482bfd740 1 -- 172.21.15.39:0/1159562317 shutdown_connections 2024-03-18T16:10:40.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.686+0000 7f6482bfd740 1 -- 172.21.15.39:0/1159562317 wait complete. 2024-03-18T16:10:40.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.686+0000 7f6482bfd740 20 client.25736 trim_cache size 0 max 16384 2024-03-18T16:10:40.689 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-03-18T16:10:40.690 INFO:teuthology.orchestra.run.smithi039.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-18T16:10:40.690 INFO:teuthology.orchestra.run.smithi039.stdout: "scrubs": { 2024-03-18T16:10:40.690 INFO:teuthology.orchestra.run.smithi039.stdout: "d4dc04d7-8f16-41db-9e39-fd86f00c8ce3": { 2024-03-18T16:10:40.690 INFO:teuthology.orchestra.run.smithi039.stdout: "path": "/", 2024-03-18T16:10:40.690 INFO:teuthology.orchestra.run.smithi039.stdout: "tag": "d4dc04d7-8f16-41db-9e39-fd86f00c8ce3", 2024-03-18T16:10:40.690 INFO:teuthology.orchestra.run.smithi039.stdout: "options": "recursive,force" 2024-03-18T16:10:40.690 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:10:40.690 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:10:40.690 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-03-18T16:10:40.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.687+0000 7f647ec00640 1 -- 172.21.15.39:0/489535886 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f64600a2790 msgr2=0x7f64600a4c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:40.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.687+0000 7f647ec00640 1 --2- 172.21.15.39:0/489535886 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f64600a2790 0x7f64600a4c50 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f6470009fa0 tx=0x7f6470009500 comp rx=0 tx=0).stop 2024-03-18T16:10:40.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.687+0000 7f647ec00640 1 -- 172.21.15.39:0/489535886 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f64781038e0 msgr2=0x7f6478079590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:10:40.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.687+0000 7f647ec00640 1 --2- 172.21.15.39:0/489535886 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f64781038e0 0x7f6478079590 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f646c00e9c0 tx=0x7f646c00ee80 comp rx=0 tx=0).stop 2024-03-18T16:10:40.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.688+0000 7f647ec00640 1 -- 172.21.15.39:0/489535886 shutdown_connections 2024-03-18T16:10:40.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.688+0000 7f647ec00640 1 --2- 172.21.15.39:0/489535886 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6478105490 0x7f6478076100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:40.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.688+0000 7f647ec00640 1 --2- 172.21.15.39:0/489535886 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f64600a2790 0x7f64600a4c50 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:40.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.688+0000 7f647ec00640 1 --2- 172.21.15.39:0/489535886 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6478104ad0 0x7f6478079ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:40.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.688+0000 7f647ec00640 1 --2- 172.21.15.39:0/489535886 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f64781038e0 0x7f6478079590 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:10:40.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.688+0000 7f647ec00640 1 -- 172.21.15.39:0/489535886 >> 172.21.15.39:0/489535886 conn(0x7f64780fee50 msgr2=0x7f6478107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:10:40.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.688+0000 7f647ec00640 1 -- 172.21.15.39:0/489535886 shutdown_connections 2024-03-18T16:10:40.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:10:40.688+0000 7f647ec00640 1 -- 172.21.15.39:0/489535886 wait complete. 2024-03-18T16:10:40.710 INFO:tasks.cephfs.filesystem:scrub status for tag:d4dc04d7-8f16-41db-9e39-fd86f00c8ce3 - {'path': '/', 'tag': 'd4dc04d7-8f16-41db-9e39-fd86f00c8ce3', 'options': 'recursive,force'} 2024-03-18T16:10:41.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:10:40 smithi039 ceph-mon[31721]: pgmap v440: 129 pgs: 129 active+clean; 661 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 597 B/s wr, 10 op/s 2024-03-18T16:10:41.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:10:40 smithi039 ceph-mon[31721]: scrub queued for path: / 2024-03-18T16:10:41.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:10:40 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:10:41.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:10:40 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:10:41.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:10:40 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:10:41.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:10:40 smithi196 ceph-mon[36307]: pgmap v440: 129 pgs: 129 active+clean; 661 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 597 B/s wr, 10 op/s 2024-03-18T16:10:41.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:10:40 smithi196 ceph-mon[36307]: scrub queued for path: / 2024-03-18T16:10:41.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:10:40 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:10:41.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:10:40 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:10:41.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:10:40 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:10:41.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:10:40 smithi194 ceph-mon[36200]: pgmap v440: 129 pgs: 129 active+clean; 661 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 597 B/s wr, 10 op/s 2024-03-18T16:10:41.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:10:40 smithi194 ceph-mon[36200]: scrub queued for path: / 2024-03-18T16:10:41.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:10:40 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:10:41.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:10:40 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:10:41.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:10:40 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:10:43.042 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:10:42 smithi194 ceph-mon[36200]: pgmap v441: 129 pgs: 129 active+clean; 661 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 597 B/s wr, 8 op/s 2024-03-18T16:10:43.042 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:10:42 smithi194 ceph-mon[36200]: scrub summary: idle 2024-03-18T16:10:43.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:10:42 smithi039 ceph-mon[31721]: pgmap v441: 129 pgs: 129 active+clean; 661 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 597 B/s wr, 8 op/s 2024-03-18T16:10:43.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:10:42 smithi039 ceph-mon[31721]: scrub summary: idle 2024-03-18T16:10:43.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:10:42 smithi196 ceph-mon[36307]: pgmap v441: 129 pgs: 129 active+clean; 661 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 597 B/s wr, 8 op/s 2024-03-18T16:10:43.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:10:42 smithi196 ceph-mon[36307]: scrub summary: idle 2024-03-18T16:10:45.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:10:44 smithi039 ceph-mon[31721]: pgmap v442: 129 pgs: 129 active+clean; 661 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 682 B/s wr, 10 op/s 2024-03-18T16:10:45.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:10:44 smithi196 ceph-mon[36307]: pgmap v442: 129 pgs: 129 active+clean; 661 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 682 B/s wr, 10 op/s 2024-03-18T16:10:45.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:10:44 smithi194 ceph-mon[36200]: pgmap v442: 129 pgs: 129 active+clean; 661 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 682 B/s wr, 10 op/s 2024-03-18T16:10:47.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:10:46 smithi039 ceph-mon[31721]: pgmap v443: 129 pgs: 129 active+clean; 661 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 682 B/s wr, 10 op/s 2024-03-18T16:10:47.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:10:46 smithi196 ceph-mon[36307]: pgmap v443: 129 pgs: 129 active+clean; 661 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 682 B/s wr, 10 op/s 2024-03-18T16:10:47.285 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:10:46 smithi194 ceph-mon[36200]: pgmap v443: 129 pgs: 129 active+clean; 661 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 682 B/s wr, 10 op/s 2024-03-18T16:10:49.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:10:48 smithi039 ceph-mon[31721]: pgmap v444: 129 pgs: 129 active+clean; 661 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 682 B/s wr, 10 op/s 2024-03-18T16:10:49.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:10:48 smithi196 ceph-mon[36307]: pgmap v444: 129 pgs: 129 active+clean; 661 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 682 B/s wr, 10 op/s 2024-03-18T16:10:49.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:10:48 smithi194 ceph-mon[36200]: pgmap v444: 129 pgs: 129 active+clean; 661 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 682 B/s wr, 10 op/s 2024-03-18T16:10:51.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:10:50 smithi039 ceph-mon[31721]: pgmap v445: 129 pgs: 129 active+clean; 661 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 682 B/s wr, 11 op/s 2024-03-18T16:10:51.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:10:50 smithi196 ceph-mon[36307]: pgmap v445: 129 pgs: 129 active+clean; 661 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 682 B/s wr, 11 op/s 2024-03-18T16:10:51.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:10:50 smithi194 ceph-mon[36200]: pgmap v445: 129 pgs: 129 active+clean; 661 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 682 B/s wr, 11 op/s 2024-03-18T16:10:53.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:10:52 smithi039 ceph-mon[31721]: pgmap v446: 129 pgs: 129 active+clean; 661 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 9 op/s 2024-03-18T16:10:53.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:10:52 smithi196 ceph-mon[36307]: pgmap v446: 129 pgs: 129 active+clean; 661 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 9 op/s 2024-03-18T16:10:53.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:10:52 smithi194 ceph-mon[36200]: pgmap v446: 129 pgs: 129 active+clean; 661 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 9 op/s 2024-03-18T16:10:55.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:10:54 smithi039 ceph-mon[31721]: pgmap v447: 129 pgs: 129 active+clean; 661 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 10 op/s 2024-03-18T16:10:55.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:10:54 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:10:55.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:10:54 smithi196 ceph-mon[36307]: pgmap v447: 129 pgs: 129 active+clean; 661 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 10 op/s 2024-03-18T16:10:55.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:10:54 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:10:55.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:10:54 smithi194 ceph-mon[36200]: pgmap v447: 129 pgs: 129 active+clean; 661 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 10 op/s 2024-03-18T16:10:55.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:10:54 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:10:57.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:10:56 smithi039 ceph-mon[31721]: pgmap v448: 129 pgs: 129 active+clean; 661 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 10 op/s 2024-03-18T16:10:57.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:10:56 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:10:57.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:10:56 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:10:57.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:10:56 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:10:57.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:10:56 smithi196 ceph-mon[36307]: pgmap v448: 129 pgs: 129 active+clean; 661 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 10 op/s 2024-03-18T16:10:57.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:10:56 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:10:57.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:10:56 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:10:57.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:10:56 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:10:57.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:10:56 smithi194 ceph-mon[36200]: pgmap v448: 129 pgs: 129 active+clean; 661 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 10 op/s 2024-03-18T16:10:57.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:10:56 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:10:57.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:10:56 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:10:57.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:10:56 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:10:59.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:10:58 smithi039 ceph-mon[31721]: pgmap v449: 129 pgs: 129 active+clean; 661 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T16:10:59.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:10:58 smithi196 ceph-mon[36307]: pgmap v449: 129 pgs: 129 active+clean; 661 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T16:10:59.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:10:58 smithi194 ceph-mon[36200]: pgmap v449: 129 pgs: 129 active+clean; 661 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-18T16:11:01.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:11:00 smithi039 ceph-mon[31721]: pgmap v450: 129 pgs: 129 active+clean; 661 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T16:11:01.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:11:00 smithi196 ceph-mon[36307]: pgmap v450: 129 pgs: 129 active+clean; 661 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T16:11:01.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:11:00 smithi194 ceph-mon[36200]: pgmap v450: 129 pgs: 129 active+clean; 661 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-18T16:11:02.656 INFO:tasks.workunit.client.0.smithi039.stderr:Note: switching to '4e19f704afd9957ea657ab858f9ecf76401fef6f'. 2024-03-18T16:11:02.656 INFO:tasks.workunit.client.0.smithi039.stderr: 2024-03-18T16:11:02.656 INFO:tasks.workunit.client.0.smithi039.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-18T16:11:02.657 INFO:tasks.workunit.client.0.smithi039.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-18T16:11:02.657 INFO:tasks.workunit.client.0.smithi039.stderr:state without impacting any branches by switching back to a branch. 2024-03-18T16:11:02.657 INFO:tasks.workunit.client.0.smithi039.stderr: 2024-03-18T16:11:02.657 INFO:tasks.workunit.client.0.smithi039.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-18T16:11:02.657 INFO:tasks.workunit.client.0.smithi039.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-18T16:11:02.657 INFO:tasks.workunit.client.0.smithi039.stderr: 2024-03-18T16:11:02.657 INFO:tasks.workunit.client.0.smithi039.stderr: git switch -c 2024-03-18T16:11:02.657 INFO:tasks.workunit.client.0.smithi039.stderr: 2024-03-18T16:11:02.657 INFO:tasks.workunit.client.0.smithi039.stderr:Or undo this operation with: 2024-03-18T16:11:02.657 INFO:tasks.workunit.client.0.smithi039.stderr: 2024-03-18T16:11:02.657 INFO:tasks.workunit.client.0.smithi039.stderr: git switch - 2024-03-18T16:11:02.657 INFO:tasks.workunit.client.0.smithi039.stderr: 2024-03-18T16:11:02.657 INFO:tasks.workunit.client.0.smithi039.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-18T16:11:02.657 INFO:tasks.workunit.client.0.smithi039.stderr: 2024-03-18T16:11:02.657 INFO:tasks.workunit.client.0.smithi039.stderr:HEAD is now at 4e19f704afd ignorelist mon down 2024-03-18T16:11:02.669 DEBUG:teuthology.orchestra.run.smithi039:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-18T16:11:02.741 INFO:tasks.workunit.client.0.smithi039.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-18T16:11:02.751 INFO:tasks.workunit.client.0.smithi039.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-18T16:11:02.751 INFO:tasks.workunit.client.0.smithi039.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-18T16:11:03.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:11:02 smithi039 ceph-mon[31721]: pgmap v451: 129 pgs: 129 active+clean; 661 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-18T16:11:03.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:11:02 smithi196 ceph-mon[36307]: pgmap v451: 129 pgs: 129 active+clean; 661 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-18T16:11:03.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:11:02 smithi194 ceph-mon[36200]: pgmap v451: 129 pgs: 129 active+clean; 661 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-03-18T16:11:03.712 INFO:tasks.workunit.client.0.smithi039.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-18T16:11:03.840 INFO:tasks.workunit.client.0.smithi039.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-18T16:11:03.931 INFO:tasks.workunit.client.0.smithi039.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-18T16:11:03.937 INFO:tasks.workunit.client.0.smithi039.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-18T16:11:03.937 INFO:tasks.workunit.client.0.smithi039.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-18T16:11:04.029 INFO:tasks.workunit.client.0.smithi039.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-18T16:11:04.040 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-03-18T16:11:04.040 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-18T16:11:04.087 INFO:tasks.workunit:Running workunits matching suites/ffsb.sh on client.0... 2024-03-18T16:11:04.088 INFO:tasks.workunit:Running workunit suites/ffsb.sh... 2024-03-18T16:11:04.088 DEBUG:teuthology.orchestra.run.smithi039:workunit test suites/ffsb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4e19f704afd9957ea657ab858f9ecf76401fef6f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh 2024-03-18T16:11:04.201 INFO:tasks.workunit.client.0.smithi039.stderr:++ dirname /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh 2024-03-18T16:11:04.204 INFO:tasks.workunit.client.0.smithi039.stderr:+ mydir=/home/ubuntu/cephtest/clone.client.0/qa/workunits/suites 2024-03-18T16:11:04.204 INFO:tasks.workunit.client.0.smithi039.stderr:+ trap -- retry EXIT 2024-03-18T16:11:04.204 INFO:tasks.workunit.client.0.smithi039.stderr:+ rm -rf ffsb 2024-03-18T16:11:04.207 INFO:tasks.workunit.client.0.smithi039.stderr:+ timeout 1800 git clone https://git.ceph.com/ffsb.git --depth 1 2024-03-18T16:11:04.215 INFO:tasks.workunit.client.0.smithi039.stderr:Cloning into 'ffsb'... 2024-03-18T16:11:05.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:11:04 smithi039 ceph-mon[31721]: pgmap v452: 129 pgs: 129 active+clean; 661 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T16:11:05.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:11:04 smithi196 ceph-mon[36307]: pgmap v452: 129 pgs: 129 active+clean; 661 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T16:11:05.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:11:04 smithi194 ceph-mon[36200]: pgmap v452: 129 pgs: 129 active+clean; 661 KiB data, 392 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T16:11:05.714 INFO:tasks.workunit.client.0.smithi039.stderr:+ trap - EXIT 2024-03-18T16:11:05.714 INFO:tasks.workunit.client.0.smithi039.stderr:+ cd ffsb 2024-03-18T16:11:05.714 INFO:tasks.workunit.client.0.smithi039.stderr:+ ./configure 2024-03-18T16:11:06.019 INFO:tasks.workunit.client.0.smithi039.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2024-03-18T16:11:07.039 INFO:tasks.workunit.client.0.smithi039.stdout:checking whether build environment is sane... yes 2024-03-18T16:11:07.066 INFO:tasks.workunit.client.0.smithi039.stdout:checking for a thread-safe mkdir -p... /usr/bin/mkdir -p 2024-03-18T16:11:07.066 INFO:tasks.workunit.client.0.smithi039.stdout:checking for gawk... gawk 2024-03-18T16:11:07.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:11:06 smithi039 ceph-mon[31721]: pgmap v453: 129 pgs: 129 active+clean; 772 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9.5 KiB/s wr, 10 op/s 2024-03-18T16:11:07.102 INFO:tasks.workunit.client.0.smithi039.stdout:checking whether make sets $(MAKE)... yes 2024-03-18T16:11:07.199 INFO:tasks.workunit.client.0.smithi039.stdout:checking build system type... x86_64-unknown-linux-gnu 2024-03-18T16:11:07.199 INFO:tasks.workunit.client.0.smithi039.stdout:checking host system type... x86_64-unknown-linux-gnu 2024-03-18T16:11:07.216 INFO:tasks.workunit.client.0.smithi039.stdout:checking for style of include used by make... GNU 2024-03-18T16:11:07.222 INFO:tasks.workunit.client.0.smithi039.stdout:checking for gcc... gcc 2024-03-18T16:11:07.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:11:06 smithi196 ceph-mon[36307]: pgmap v453: 129 pgs: 129 active+clean; 772 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9.5 KiB/s wr, 10 op/s 2024-03-18T16:11:07.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:11:06 smithi194 ceph-mon[36200]: pgmap v453: 129 pgs: 129 active+clean; 772 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 9.5 KiB/s wr, 10 op/s 2024-03-18T16:11:07.341 INFO:tasks.workunit.client.0.smithi039.stdout:checking for C compiler default output file name... a.out 2024-03-18T16:11:07.349 INFO:tasks.workunit.client.0.smithi039.stdout:checking whether the C compiler works... yes 2024-03-18T16:11:07.353 INFO:tasks.workunit.client.0.smithi039.stdout:checking whether we are cross compiling... no 2024-03-18T16:11:07.438 INFO:tasks.workunit.client.0.smithi039.stdout:checking for suffix of executables... 2024-03-18T16:11:07.514 INFO:tasks.workunit.client.0.smithi039.stdout:checking for suffix of object files... o 2024-03-18T16:11:07.590 INFO:tasks.workunit.client.0.smithi039.stdout:checking whether we are using the GNU C compiler... yes 2024-03-18T16:11:07.687 INFO:tasks.workunit.client.0.smithi039.stdout:checking whether gcc accepts -g... yes 2024-03-18T16:11:07.853 INFO:tasks.workunit.client.0.smithi039.stdout:checking for gcc option to accept ISO C89... none needed 2024-03-18T16:11:08.044 INFO:tasks.workunit.client.0.smithi039.stdout:checking dependency style of gcc... gcc3 2024-03-18T16:11:08.190 INFO:tasks.workunit.client.0.smithi039.stdout:checking how to run the C preprocessor... gcc -E 2024-03-18T16:11:08.308 INFO:tasks.workunit.client.0.smithi039.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2024-03-18T16:11:08.312 INFO:tasks.workunit.client.0.smithi039.stdout:checking for egrep... /usr/bin/grep -E 2024-03-18T16:11:08.340 INFO:tasks.workunit.client.0.smithi039.stdout:checking for AIX... no 2024-03-18T16:11:08.374 INFO:tasks.workunit.client.0.smithi039.stdout:checking for gcc... (cached) gcc 2024-03-18T16:11:08.408 INFO:tasks.workunit.client.0.smithi039.stdout:checking whether we are using the GNU C compiler... (cached) yes 2024-03-18T16:11:08.411 INFO:tasks.workunit.client.0.smithi039.stdout:checking whether gcc accepts -g... (cached) yes 2024-03-18T16:11:08.412 INFO:tasks.workunit.client.0.smithi039.stdout:checking for gcc option to accept ISO C89... (cached) none needed 2024-03-18T16:11:08.412 INFO:tasks.workunit.client.0.smithi039.stdout:checking dependency style of gcc... (cached) gcc3 2024-03-18T16:11:08.549 INFO:tasks.workunit.client.0.smithi039.stdout:checking for main in -lm... yes 2024-03-18T16:11:08.665 INFO:tasks.workunit.client.0.smithi039.stdout:checking for main in -lpthread... yes 2024-03-18T16:11:09.013 INFO:tasks.workunit.client.0.smithi039.stdout:checking for ANSI C header files... yes 2024-03-18T16:11:09.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:11:08 smithi039 ceph-mon[31721]: pgmap v454: 129 pgs: 129 active+clean; 772 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-03-18T16:11:09.165 INFO:tasks.workunit.client.0.smithi039.stdout:checking for sys/wait.h that is POSIX.1 compatible... yes 2024-03-18T16:11:09.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:11:08 smithi196 ceph-mon[36307]: pgmap v454: 129 pgs: 129 active+clean; 772 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-03-18T16:11:09.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:11:08 smithi194 ceph-mon[36200]: pgmap v454: 129 pgs: 129 active+clean; 772 KiB data, 393 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 12 KiB/s wr, 11 op/s 2024-03-18T16:11:09.286 INFO:tasks.workunit.client.0.smithi039.stdout:checking for sys/types.h... yes 2024-03-18T16:11:09.481 INFO:tasks.workunit.client.0.smithi039.stdout:checking for sys/stat.h... yes 2024-03-18T16:11:09.589 INFO:tasks.workunit.client.0.smithi039.stdout:checking for stdlib.h... yes 2024-03-18T16:11:09.685 INFO:tasks.workunit.client.0.smithi039.stdout:checking for string.h... yes 2024-03-18T16:11:09.811 INFO:tasks.workunit.client.0.smithi039.stdout:checking for memory.h... yes 2024-03-18T16:11:09.917 INFO:tasks.workunit.client.0.smithi039.stdout:checking for strings.h... yes 2024-03-18T16:11:10.040 INFO:tasks.workunit.client.0.smithi039.stdout:checking for inttypes.h... yes 2024-03-18T16:11:10.140 INFO:tasks.workunit.client.0.smithi039.stdout:checking for stdint.h... yes 2024-03-18T16:11:10.243 INFO:tasks.workunit.client.0.smithi039.stdout:checking for unistd.h... yes 2024-03-18T16:11:10.383 INFO:tasks.workunit.client.0.smithi039.stdout:checking pthread.h usability... yes 2024-03-18T16:11:10.456 INFO:tasks.workunit.client.0.smithi039.stdout:checking pthread.h presence... yes 2024-03-18T16:11:10.458 INFO:tasks.workunit.client.0.smithi039.stdout:checking for pthread.h... yes 2024-03-18T16:11:10.581 INFO:tasks.workunit.client.0.smithi039.stdout:checking fcntl.h usability... yes 2024-03-18T16:11:10.637 INFO:tasks.workunit.client.0.smithi039.stdout:checking fcntl.h presence... yes 2024-03-18T16:11:10.639 INFO:tasks.workunit.client.0.smithi039.stdout:checking for fcntl.h... yes 2024-03-18T16:11:10.712 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T16:11:10.712 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T16:11:10.752 INFO:tasks.workunit.client.0.smithi039.stdout:checking limits.h usability... yes 2024-03-18T16:11:10.795 INFO:tasks.workunit.client.0.smithi039.stdout:checking limits.h presence... yes 2024-03-18T16:11:10.797 INFO:tasks.workunit.client.0.smithi039.stdout:checking for limits.h... yes 2024-03-18T16:11:10.833 INFO:tasks.workunit.client.0.smithi039.stdout:checking for stdint.h... (cached) yes 2024-03-18T16:11:10.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.936+0000 7fd6daa00640 1 -- 172.21.15.39:0/221443555 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd6d41028d0 msgr2=0x7fd6d4102d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:10.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.936+0000 7fd6daa00640 1 --2- 172.21.15.39:0/221443555 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd6d41028d0 0x7fd6d4102d50 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fd6c0009900 tx=0x7fd6c002bc40 comp rx=0 tx=0).stop 2024-03-18T16:11:10.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.936+0000 7fd6daa00640 1 -- 172.21.15.39:0/221443555 shutdown_connections 2024-03-18T16:11:10.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.936+0000 7fd6daa00640 1 --2- 172.21.15.39:0/221443555 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd6d4103290 0x7fd6d4109b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:10.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.936+0000 7fd6daa00640 1 --2- 172.21.15.39:0/221443555 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd6d41028d0 0x7fd6d4102d50 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:10.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.936+0000 7fd6daa00640 1 --2- 172.21.15.39:0/221443555 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd6d41016e0 0x7fd6d4101b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:10.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.936+0000 7fd6daa00640 1 -- 172.21.15.39:0/221443555 >> 172.21.15.39:0/221443555 conn(0x7fd6d40fcc50 msgr2=0x7fd6d40ff090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:10.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.936+0000 7fd6daa00640 1 -- 172.21.15.39:0/221443555 shutdown_connections 2024-03-18T16:11:10.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.936+0000 7fd6daa00640 1 -- 172.21.15.39:0/221443555 wait complete. 2024-03-18T16:11:10.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.937+0000 7fd6daa00640 1 Processor -- start 2024-03-18T16:11:10.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.937+0000 7fd6daa00640 1 -- start start 2024-03-18T16:11:10.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.937+0000 7fd6daa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd6d41016e0 0x7fd6d4199bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:10.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.937+0000 7fd6daa00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd6d41028d0 0x7fd6d419a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:10.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.937+0000 7fd6daa00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd6d4103290 0x7fd6d41a1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:10.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.937+0000 7fd6daa00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fd6d410d390 con 0x7fd6d41016e0 2024-03-18T16:11:10.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.937+0000 7fd6daa00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fd6d410d210 con 0x7fd6d41028d0 2024-03-18T16:11:10.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.937+0000 7fd6daa00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fd6d410d510 con 0x7fd6d4103290 2024-03-18T16:11:10.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.938+0000 7fd6d9600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd6d41016e0 0x7fd6d4199bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:10.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.938+0000 7fd6d8c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd6d41028d0 0x7fd6d419a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:10.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.938+0000 7fd6da000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd6d4103290 0x7fd6d41a1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:10.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.938+0000 7fd6d9600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd6d41016e0 0x7fd6d4199bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:34576/0 (socket says 172.21.15.39:34576) 2024-03-18T16:11:10.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.938+0000 7fd6d9600640 1 -- 172.21.15.39:0/2031008333 learned_addr learned my addr 172.21.15.39:0/2031008333 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:11:10.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.938+0000 7fd6da000640 1 -- 172.21.15.39:0/2031008333 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd6d41028d0 msgr2=0x7fd6d419a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:10.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.938+0000 7fd6da000640 1 --2- 172.21.15.39:0/2031008333 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd6d41028d0 0x7fd6d419a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:10.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.938+0000 7fd6da000640 1 -- 172.21.15.39:0/2031008333 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd6d41016e0 msgr2=0x7fd6d4199bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:10.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.938+0000 7fd6da000640 1 --2- 172.21.15.39:0/2031008333 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd6d41016e0 0x7fd6d4199bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:10.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.938+0000 7fd6da000640 1 -- 172.21.15.39:0/2031008333 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6d41a17e0 con 0x7fd6d4103290 2024-03-18T16:11:10.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.938+0000 7fd6d9600640 1 --2- 172.21.15.39:0/2031008333 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd6d41016e0 0x7fd6d4199bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:11:10.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.938+0000 7fd6d8c00640 1 --2- 172.21.15.39:0/2031008333 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd6d41028d0 0x7fd6d419a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:11:10.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.939+0000 7fd6da000640 1 --2- 172.21.15.39:0/2031008333 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd6d4103290 0x7fd6d41a1170 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc00e940 tx=0x7fd6cc00ee00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:10.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.939+0000 7fd6d2000640 1 -- 172.21.15.39:0/2031008333 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6cc004d20 con 0x7fd6d4103290 2024-03-18T16:11:10.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.939+0000 7fd6d2000640 1 -- 172.21.15.39:0/2031008333 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd6cc005020 con 0x7fd6d4103290 2024-03-18T16:11:10.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.939+0000 7fd6daa00640 1 -- 172.21.15.39:0/2031008333 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6d41a1ad0 con 0x7fd6d4103290 2024-03-18T16:11:10.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.939+0000 7fd6d2000640 1 -- 172.21.15.39:0/2031008333 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6cc0116b0 con 0x7fd6d4103290 2024-03-18T16:11:10.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.939+0000 7fd6daa00640 1 -- 172.21.15.39:0/2031008333 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6d41a1fa0 con 0x7fd6d4103290 2024-03-18T16:11:10.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.942+0000 7fd6ded77740 1 Processor -- start 2024-03-18T16:11:10.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.942+0000 7fd6d2000640 1 -- 172.21.15.39:0/2031008333 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fd6cc0054d0 con 0x7fd6d4103290 2024-03-18T16:11:10.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.943+0000 7fd6ded77740 1 -- start start 2024-03-18T16:11:10.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.943+0000 7fd6ded77740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bf711d2a10 0x55bf711d2e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:10.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.943+0000 7fd6ded77740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bf711d3400 0x55bf711d7f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:10.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.943+0000 7fd6ded77740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bf711d8570 0x55bf711daa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:10.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.943+0000 7fd6ded77740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55bf711371f0 con 0x55bf711d8570 2024-03-18T16:11:10.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.943+0000 7fd6ded77740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55bf710dfa80 con 0x55bf711d2a10 2024-03-18T16:11:10.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.943+0000 7fd6ded77740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55bf71133290 con 0x55bf711d3400 2024-03-18T16:11:10.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.943+0000 7fd6d9600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bf711d3400 0x55bf711d7f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:10.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.943+0000 7fd6d9600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bf711d3400 0x55bf711d7f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:39724/0 (socket says 172.21.15.39:39724) 2024-03-18T16:11:10.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.943+0000 7fd6d9600640 1 -- 172.21.15.39:0/2230953234 learned_addr learned my addr 172.21.15.39:0/2230953234 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:11:10.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.943+0000 7fd6d8c00640 1 --2- 172.21.15.39:0/2230953234 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bf711d2a10 0x55bf711d2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:10.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.944+0000 7fd6d9600640 1 -- 172.21.15.39:0/2230953234 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bf711d2a10 msgr2=0x55bf711d2e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:10.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.944+0000 7fd6d8c00640 1 --2- 172.21.15.39:0/2230953234 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bf711d8570 0x55bf711daa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:10.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.944+0000 7fd6d9600640 1 --2- 172.21.15.39:0/2230953234 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bf711d2a10 0x55bf711d2e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:10.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.944+0000 7fd6d9600640 1 -- 172.21.15.39:0/2230953234 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bf711d8570 msgr2=0x55bf711daa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:10.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.944+0000 7fd6d9600640 1 --2- 172.21.15.39:0/2230953234 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bf711d8570 0x55bf711daa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:10.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.944+0000 7fd6d9600640 1 -- 172.21.15.39:0/2230953234 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bf70f157d0 con 0x55bf711d3400 2024-03-18T16:11:10.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.944+0000 7fd6d8c00640 1 --2- 172.21.15.39:0/2230953234 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bf711d8570 0x55bf711daa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:11:10.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.944+0000 7fd6d9600640 1 --2- 172.21.15.39:0/2230953234 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bf711d3400 0x55bf711d7f00 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fd6c80115f0 tx=0x7fd6c8011ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c1ce2143009e3da7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:10.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.944+0000 7fd6d2000640 1 --2- 172.21.15.39:0/2031008333 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd6bc0a2800 0x7fd6bc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:10.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.944+0000 7fd6c6600640 1 -- 172.21.15.39:0/2230953234 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c800f6f0 con 0x55bf711d3400 2024-03-18T16:11:10.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.944+0000 7fd6c6600640 1 -- 172.21.15.39:0/2230953234 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd6c800fd70 con 0x55bf711d3400 2024-03-18T16:11:10.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.944+0000 7fd6c6600640 1 -- 172.21.15.39:0/2230953234 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c80183b0 con 0x55bf711d3400 2024-03-18T16:11:10.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.944+0000 7fd6d2000640 1 -- 172.21.15.39:0/2031008333 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fd6cc0ceea0 con 0x7fd6d4103290 2024-03-18T16:11:10.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.945+0000 7fd6d8c00640 1 --2- 172.21.15.39:0/2031008333 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd6bc0a2800 0x7fd6bc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:10.952 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.945+0000 7fd6ded77740 1 -- 172.21.15.39:0/2230953234 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bf711d3400 msgr2=0x55bf711d7f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:10.952 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.945+0000 7fd6ded77740 1 --2- 172.21.15.39:0/2230953234 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bf711d3400 0x55bf711d7f00 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fd6c80115f0 tx=0x7fd6c8011ab0 comp rx=0 tx=0).stop 2024-03-18T16:11:10.952 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.945+0000 7fd6ded77740 1 -- 172.21.15.39:0/2230953234 shutdown_connections 2024-03-18T16:11:10.952 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.945+0000 7fd6ded77740 1 --2- 172.21.15.39:0/2230953234 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bf711d8570 0x55bf711daa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:10.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.945+0000 7fd6ded77740 1 --2- 172.21.15.39:0/2230953234 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bf711d3400 0x55bf711d7f00 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:10.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.945+0000 7fd6ded77740 1 --2- 172.21.15.39:0/2230953234 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bf711d2a10 0x55bf711d2e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:10.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.945+0000 7fd6ded77740 1 -- 172.21.15.39:0/2230953234 >> 172.21.15.39:0/2230953234 conn(0x55bf711cdf80 msgr2=0x55bf711ce380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:10.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.945+0000 7fd6d8c00640 1 --2- 172.21.15.39:0/2031008333 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd6bc0a2800 0x7fd6bc0a4cc0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fd6c002c250 tx=0x7fd6c0000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:10.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.945+0000 7fd6ded77740 1 -- 172.21.15.39:0/2230953234 shutdown_connections 2024-03-18T16:11:10.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.945+0000 7fd6ded77740 1 -- 172.21.15.39:0/2230953234 wait complete. 2024-03-18T16:11:10.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.946+0000 7fd6ded77740 1 -- start start 2024-03-18T16:11:10.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.946+0000 7fd6ded77740 1 Processor -- start 2024-03-18T16:11:10.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.947+0000 7fd6ded77740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bf711d2a10 0x55bf711dcf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:10.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.947+0000 7fd6ded77740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bf711d3400 0x55bf711dd4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:10.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.947+0000 7fd6ded77740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bf711d8570 0x55bf711e4500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:10.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.947+0000 7fd6ded77740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55bf711db520 con 0x55bf711d8570 2024-03-18T16:11:10.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.947+0000 7fd6ded77740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55bf710f2ed0 con 0x55bf711d2a10 2024-03-18T16:11:10.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.947+0000 7fd6ded77740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55bf711db6a0 con 0x55bf711d3400 2024-03-18T16:11:10.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.947+0000 7fd6ded77740 10 client.0 _ll_register_callbacks cb 0x55bf7112e220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:11:10.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.947+0000 7fd6d9600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bf711d2a10 0x55bf711dcf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:10.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.947+0000 7fd6d8c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bf711d3400 0x55bf711dd4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:10.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.947+0000 7fd6d9600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bf711d2a10 0x55bf711dcf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:54424/0 (socket says 172.21.15.39:54424) 2024-03-18T16:11:10.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.947+0000 7fd6d9600640 1 -- 172.21.15.39:0/1433253594 learned_addr learned my addr 172.21.15.39:0/1433253594 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:11:10.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.947+0000 7fd6da000640 1 --2- 172.21.15.39:0/1433253594 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bf711d8570 0x55bf711e4500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:10.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.948+0000 7fd6d9600640 1 -- 172.21.15.39:0/1433253594 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bf711d3400 msgr2=0x55bf711dd4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:10.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.948+0000 7fd6d9600640 1 --2- 172.21.15.39:0/1433253594 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bf711d3400 0x55bf711dd4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:10.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.948+0000 7fd6d9600640 1 -- 172.21.15.39:0/1433253594 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bf711d8570 msgr2=0x55bf711e4500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:10.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.948+0000 7fd6d9600640 1 --2- 172.21.15.39:0/1433253594 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bf711d8570 0x55bf711e4500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:10.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.948+0000 7fd6d9600640 1 -- 172.21.15.39:0/1433253594 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bf70e4b3f0 con 0x55bf711d2a10 2024-03-18T16:11:10.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.948+0000 7fd6da000640 1 --2- 172.21.15.39:0/1433253594 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bf711d8570 0x55bf711e4500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:11:10.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.948+0000 7fd6d9600640 1 --2- 172.21.15.39:0/1433253594 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bf711d2a10 0x55bf711dcf80 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fd6c80115c0 tx=0x7fd6c8019e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c380c86d69cd0f88 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:10.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.948+0000 7fd6baa00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:11:10.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.948+0000 7fd6baa00640 1 -- 172.21.15.39:0/1433253594 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c800f8d0 con 0x55bf711d2a10 2024-03-18T16:11:10.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.948+0000 7fd6ded77740 1 -- 172.21.15.39:0/1433253594 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd6c800fd70 con 0x55bf711d2a10 2024-03-18T16:11:10.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.948+0000 7fd6baa00640 1 -- 172.21.15.39:0/1433253594 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fd6c8016050 con 0x55bf711d2a10 2024-03-18T16:11:10.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.949+0000 7fd6baa00640 1 -- 172.21.15.39:0/1433253594 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd6c802f050 con 0x55bf711d2a10 2024-03-18T16:11:10.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.949+0000 7fd6ded77740 10 client.25748 fetch_fsmap learned FSMap version 28 2024-03-18T16:11:10.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.949+0000 7fd6ded77740 1 -- 172.21.15.39:0/1433253594 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55bf70e0b0f0 con 0x55bf711d2a10 2024-03-18T16:11:10.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.949+0000 7fd6baa00640 1 -- 172.21.15.39:0/1433253594 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd6c800fd70 con 0x55bf711d2a10 2024-03-18T16:11:10.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.949+0000 7fd6baa00640 1 -- 172.21.15.39:0/1433253594 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fd6c8005420 con 0x55bf711d2a10 2024-03-18T16:11:10.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.949+0000 7fd6ded77740 10 client.25748 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T16:11:10.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.949+0000 7fd6ded77740 10 client.25748 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:11:10.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.949+0000 7fd6ded77740 20 client.25748 populate_metadata read hostname 'smithi039' 2024-03-18T16:11:10.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.949+0000 7fd6ded77740 1 --2- 172.21.15.39:0/1433253594 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55bf711e8a90 0x55bf711eaec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:10.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.949+0000 7fd6ded77740 4 client.25748 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:11:10.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.949+0000 7fd6ded77740 1 -- 172.21.15.39:0/1433253594 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bf7114abe0 con 0x55bf711e8a90 2024-03-18T16:11:10.956 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.950+0000 7fd6d8c00640 1 --2- 172.21.15.39:0/1433253594 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55bf711e8a90 0x55bf711eaec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:10.956 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.950+0000 7fd6d8c00640 1 --2- 172.21.15.39:0/1433253594 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55bf711e8a90 0x55bf711eaec0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e9b40db4b1131588 server_cookie=e92e59df9f671a6d in_seq=0 out_seq=0 2024-03-18T16:11:10.956 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.950+0000 7fd6baa00640 10 client.25748 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:11:10.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.951+0000 7fd6baa00640 1 -- 172.21.15.39:0/1433253594 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55bf7114abe0 con 0x55bf711e8a90 2024-03-18T16:11:10.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.951+0000 7fd6baa00640 10 client.25748 handle_command_reply: tid=0 2024-03-18T16:11:10.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.951+0000 7fd6ded77740 1 client.25748 shutdown 2024-03-18T16:11:10.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.951+0000 7fd6ded77740 1 -- 172.21.15.39:0/1433253594 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bf711d2a10 msgr2=0x55bf711dcf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:10.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.951+0000 7fd6ded77740 1 --2- 172.21.15.39:0/1433253594 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bf711d2a10 0x55bf711dcf80 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fd6c80115c0 tx=0x7fd6c8019e10 comp rx=0 tx=0).stop 2024-03-18T16:11:10.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.951+0000 7fd6ded77740 1 -- 172.21.15.39:0/1433253594 shutdown_connections 2024-03-18T16:11:10.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.951+0000 7fd6ded77740 1 --2- 172.21.15.39:0/1433253594 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55bf711e8a90 0x55bf711eaec0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:10.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.951+0000 7fd6ded77740 1 --2- 172.21.15.39:0/1433253594 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bf711d8570 0x55bf711e4500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:10.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.951+0000 7fd6ded77740 1 --2- 172.21.15.39:0/1433253594 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bf711d3400 0x55bf711dd4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:10.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.951+0000 7fd6baa00640 0 client.25748 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:11:10.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.951+0000 7fd6ded77740 1 --2- 172.21.15.39:0/1433253594 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bf711d2a10 0x55bf711dcf80 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:10.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.951+0000 7fd6ded77740 1 -- 172.21.15.39:0/1433253594 >> 172.21.15.39:0/1433253594 conn(0x55bf711cdf80 msgr2=0x55bf711cd110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:10.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.951+0000 7fd6ded77740 1 -- 172.21.15.39:0/1433253594 shutdown_connections 2024-03-18T16:11:10.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.951+0000 7fd6ded77740 1 -- 172.21.15.39:0/1433253594 wait complete. 2024-03-18T16:11:10.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.952+0000 7fd6ded77740 20 client.25748 trim_cache size 0 max 16384 2024-03-18T16:11:10.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.970+0000 7fd6ded77740 1 Processor -- start 2024-03-18T16:11:10.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.971+0000 7fd6ded77740 1 -- start start 2024-03-18T16:11:10.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.971+0000 7fd6ded77740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bf711d2a10 0x55bf711dba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:10.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.971+0000 7fd6ded77740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bf711d3400 0x55bf711e44c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:10.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.971+0000 7fd6ded77740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bf711d8570 0x55bf711dc5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:10.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.971+0000 7fd6ded77740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fd6d41ac470 con 0x55bf711d8570 2024-03-18T16:11:10.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.971+0000 7fd6ded77740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fd6d41adfb0 con 0x55bf711d3400 2024-03-18T16:11:10.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.971+0000 7fd6ded77740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fd6d41acb40 con 0x55bf711d2a10 2024-03-18T16:11:10.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.971+0000 7fd6d8c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bf711d3400 0x55bf711e44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:10.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.971+0000 7fd6d9600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bf711d2a10 0x55bf711dba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:10.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.971+0000 7fd6d8c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bf711d3400 0x55bf711e44c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:54434/0 (socket says 172.21.15.39:54434) 2024-03-18T16:11:10.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.971+0000 7fd6d8c00640 1 -- 172.21.15.39:0/3038138271 learned_addr learned my addr 172.21.15.39:0/3038138271 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:11:10.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.971+0000 7fd6da000640 1 --2- 172.21.15.39:0/3038138271 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bf711d8570 0x55bf711dc5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:10.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.972+0000 7fd6d8c00640 1 -- 172.21.15.39:0/3038138271 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bf711d2a10 msgr2=0x55bf711dba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:10.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.972+0000 7fd6d8c00640 1 --2- 172.21.15.39:0/3038138271 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bf711d2a10 0x55bf711dba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:10.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.972+0000 7fd6d8c00640 1 -- 172.21.15.39:0/3038138271 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bf711d8570 msgr2=0x55bf711dc5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:10.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.972+0000 7fd6d8c00640 1 --2- 172.21.15.39:0/3038138271 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bf711d8570 0x55bf711dc5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:10.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.972+0000 7fd6d8c00640 1 -- 172.21.15.39:0/3038138271 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bf711cee00 con 0x55bf711d3400 2024-03-18T16:11:10.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.972+0000 7fd6d9600640 1 --2- 172.21.15.39:0/3038138271 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bf711d2a10 0x55bf711dba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:11:10.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.972+0000 7fd6da000640 1 --2- 172.21.15.39:0/3038138271 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bf711d8570 0x55bf711dc5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:11:10.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.972+0000 7fd6d8c00640 1 --2- 172.21.15.39:0/3038138271 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bf711d3400 0x55bf711e44c0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fd6c002efd0 tx=0x7fd6c0041040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5e3042f5ec69e0e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:10.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.972+0000 7fd6c5c00640 1 -- 172.21.15.39:0/3038138271 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c00556e0 con 0x55bf711d3400 2024-03-18T16:11:10.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.972+0000 7fd6c5c00640 1 -- 172.21.15.39:0/3038138271 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd6c0004c60 con 0x55bf711d3400 2024-03-18T16:11:10.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.972+0000 7fd6c5c00640 1 -- 172.21.15.39:0/3038138271 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c00035f0 con 0x55bf711d3400 2024-03-18T16:11:10.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.973+0000 7fd6ded77740 1 -- 172.21.15.39:0/3038138271 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bf711d3400 msgr2=0x55bf711e44c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:10.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.973+0000 7fd6ded77740 1 --2- 172.21.15.39:0/3038138271 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bf711d3400 0x55bf711e44c0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fd6c002efd0 tx=0x7fd6c0041040 comp rx=0 tx=0).stop 2024-03-18T16:11:10.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.973+0000 7fd6ded77740 1 -- 172.21.15.39:0/3038138271 shutdown_connections 2024-03-18T16:11:10.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.973+0000 7fd6ded77740 1 --2- 172.21.15.39:0/3038138271 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bf711d3400 0x55bf711e44c0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:10.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.973+0000 7fd6ded77740 1 --2- 172.21.15.39:0/3038138271 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bf711d8570 0x55bf711dc5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:10.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.973+0000 7fd6ded77740 1 --2- 172.21.15.39:0/3038138271 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bf711d2a10 0x55bf711dba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:10.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.973+0000 7fd6ded77740 1 -- 172.21.15.39:0/3038138271 >> 172.21.15.39:0/3038138271 conn(0x55bf711cdf80 msgr2=0x55bf711cd110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:10.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.973+0000 7fd6ded77740 1 -- 172.21.15.39:0/3038138271 shutdown_connections 2024-03-18T16:11:10.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.973+0000 7fd6ded77740 1 -- 172.21.15.39:0/3038138271 wait complete. 2024-03-18T16:11:10.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.973+0000 7fd6ded77740 1 -- start start 2024-03-18T16:11:10.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.974+0000 7fd6ded77740 1 Processor -- start 2024-03-18T16:11:10.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.974+0000 7fd6ded77740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bf711d2a10 0x55bf711e15e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:10.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.974+0000 7fd6ded77740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bf711d3400 0x55bf711e1b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:10.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.974+0000 7fd6ded77740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bf711d8570 0x55bf711e2b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:10.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.974+0000 7fd6ded77740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fd6d41ad8e0 con 0x55bf711d8570 2024-03-18T16:11:10.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.974+0000 7fd6ded77740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fd6d41ad210 con 0x55bf711d2a10 2024-03-18T16:11:10.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.974+0000 7fd6ded77740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fd6d41aa450 con 0x55bf711d3400 2024-03-18T16:11:10.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.974+0000 7fd6ded77740 10 client.0 _ll_register_callbacks cb 0x7fd6c8011440 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:11:10.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.974+0000 7fd6d8c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bf711d3400 0x55bf711e1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:10.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.974+0000 7fd6d8c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bf711d3400 0x55bf711e1b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:39756/0 (socket says 172.21.15.39:39756) 2024-03-18T16:11:10.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.974+0000 7fd6d8c00640 1 -- 172.21.15.39:0/187203878 learned_addr learned my addr 172.21.15.39:0/187203878 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:11:10.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.975+0000 7fd6da000640 1 --2- 172.21.15.39:0/187203878 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bf711d8570 0x55bf711e2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:10.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.975+0000 7fd6d9600640 1 --2- 172.21.15.39:0/187203878 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bf711d2a10 0x55bf711e15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:10.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.975+0000 7fd6d8c00640 1 -- 172.21.15.39:0/187203878 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bf711d2a10 msgr2=0x55bf711e15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:10.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.975+0000 7fd6d8c00640 1 --2- 172.21.15.39:0/187203878 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bf711d2a10 0x55bf711e15e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:10.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.975+0000 7fd6d8c00640 1 -- 172.21.15.39:0/187203878 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bf711d8570 msgr2=0x55bf711e2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:10.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.975+0000 7fd6d8c00640 1 --2- 172.21.15.39:0/187203878 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bf711d8570 0x55bf711e2b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:10.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.975+0000 7fd6d8c00640 1 -- 172.21.15.39:0/187203878 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bf711e3120 con 0x55bf711d3400 2024-03-18T16:11:10.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.975+0000 7fd6da000640 1 --2- 172.21.15.39:0/187203878 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bf711d8570 0x55bf711e2b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:11:10.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.975+0000 7fd6d8c00640 1 --2- 172.21.15.39:0/187203878 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bf711d3400 0x55bf711e1b20 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fd6c0036040 tx=0x7fd6c0005790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1f7bd26af677f5e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:10.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.975+0000 7fd6baa00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:11:10.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.975+0000 7fd6baa00640 1 -- 172.21.15.39:0/187203878 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6c0063060 con 0x55bf711d3400 2024-03-18T16:11:10.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.976+0000 7fd6ded77740 1 -- 172.21.15.39:0/187203878 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd6c0004c60 con 0x55bf711d3400 2024-03-18T16:11:10.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.976+0000 7fd6baa00640 1 -- 172.21.15.39:0/187203878 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fd6c0004ef0 con 0x55bf711d3400 2024-03-18T16:11:10.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.976+0000 7fd6baa00640 1 -- 172.21.15.39:0/187203878 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fd6c006a050 con 0x55bf711d3400 2024-03-18T16:11:10.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.976+0000 7fd6ded77740 10 client.25783 fetch_fsmap learned FSMap version 28 2024-03-18T16:11:10.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.976+0000 7fd6ded77740 1 -- 172.21.15.39:0/187203878 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55bf711e4480 con 0x55bf711d3400 2024-03-18T16:11:10.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.976+0000 7fd6baa00640 1 -- 172.21.15.39:0/187203878 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd6c0004c60 con 0x55bf711d3400 2024-03-18T16:11:10.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.976+0000 7fd6baa00640 1 -- 172.21.15.39:0/187203878 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fd6c0052070 con 0x55bf711d3400 2024-03-18T16:11:10.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.976+0000 7fd6ded77740 10 client.25783 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T16:11:10.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.976+0000 7fd6ded77740 10 client.25783 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:11:10.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.976+0000 7fd6ded77740 20 client.25783 populate_metadata read hostname 'smithi039' 2024-03-18T16:11:10.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.976+0000 7fd6ded77740 1 --2- 172.21.15.39:0/187203878 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55bf7124f150 0x55bf71251570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:10.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.976+0000 7fd6ded77740 4 client.25783 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T16:11:10.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.976+0000 7fd6ded77740 1 -- 172.21.15.39:0/187203878 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55bf71144730 con 0x55bf7124f150 2024-03-18T16:11:10.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.977+0000 7fd6d9600640 1 --2- 172.21.15.39:0/187203878 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55bf7124f150 0x55bf71251570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:10.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.977+0000 7fd6d9600640 1 --2- 172.21.15.39:0/187203878 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55bf7124f150 0x55bf71251570 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a485592acb2f3bbb server_cookie=5f946dad1508e54b in_seq=0 out_seq=0 2024-03-18T16:11:10.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.977+0000 7fd6baa00640 10 client.25783 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:11:10.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.977+0000 7fd6baa00640 1 -- 172.21.15.39:0/187203878 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55bf71144730 con 0x55bf7124f150 2024-03-18T16:11:10.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.977+0000 7fd6baa00640 10 client.25783 handle_command_reply: tid=0 2024-03-18T16:11:10.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.977+0000 7fd6ded77740 1 client.25783 shutdown 2024-03-18T16:11:10.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.978+0000 7fd6ded77740 1 -- 172.21.15.39:0/187203878 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bf711d3400 msgr2=0x55bf711e1b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:10.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.978+0000 7fd6ded77740 1 --2- 172.21.15.39:0/187203878 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bf711d3400 0x55bf711e1b20 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fd6c0036040 tx=0x7fd6c0005790 comp rx=0 tx=0).stop 2024-03-18T16:11:10.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.978+0000 7fd6ded77740 1 -- 172.21.15.39:0/187203878 shutdown_connections 2024-03-18T16:11:10.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.978+0000 7fd6ded77740 1 --2- 172.21.15.39:0/187203878 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55bf7124f150 0x55bf71251570 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:10.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.978+0000 7fd6ded77740 1 --2- 172.21.15.39:0/187203878 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bf711d8570 0x55bf711e2b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:10.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.978+0000 7fd6ded77740 1 --2- 172.21.15.39:0/187203878 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bf711d3400 0x55bf711e1b20 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:10.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.978+0000 7fd6ded77740 1 --2- 172.21.15.39:0/187203878 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bf711d2a10 0x55bf711e15e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:10.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.978+0000 7fd6baa00640 0 client.25783 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:11:10.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.978+0000 7fd6ded77740 1 -- 172.21.15.39:0/187203878 >> 172.21.15.39:0/187203878 conn(0x55bf711cdf80 msgr2=0x55bf711d11e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:10.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.978+0000 7fd6ded77740 1 -- 172.21.15.39:0/187203878 shutdown_connections 2024-03-18T16:11:10.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.978+0000 7fd6ded77740 1 -- 172.21.15.39:0/187203878 wait complete. 2024-03-18T16:11:10.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.978+0000 7fd6ded77740 20 client.25783 trim_cache size 0 max 16384 2024-03-18T16:11:10.986 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-03-18T16:11:10.987 INFO:teuthology.orchestra.run.smithi039.stdout: "status": "no active scrubs running", 2024-03-18T16:11:10.987 INFO:teuthology.orchestra.run.smithi039.stdout: "scrubs": {} 2024-03-18T16:11:10.987 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-03-18T16:11:10.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.979+0000 7fd6daa00640 1 -- 172.21.15.39:0/2031008333 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd6bc0a2800 msgr2=0x7fd6bc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:10.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.979+0000 7fd6daa00640 1 --2- 172.21.15.39:0/2031008333 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd6bc0a2800 0x7fd6bc0a4cc0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fd6c002c250 tx=0x7fd6c0000bc0 comp rx=0 tx=0).stop 2024-03-18T16:11:10.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.979+0000 7fd6daa00640 1 -- 172.21.15.39:0/2031008333 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd6d4103290 msgr2=0x7fd6d41a1170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:10.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.979+0000 7fd6daa00640 1 --2- 172.21.15.39:0/2031008333 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd6d4103290 0x7fd6d41a1170 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fd6cc00e940 tx=0x7fd6cc00ee00 comp rx=0 tx=0).stop 2024-03-18T16:11:10.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.979+0000 7fd6daa00640 1 -- 172.21.15.39:0/2031008333 shutdown_connections 2024-03-18T16:11:10.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.979+0000 7fd6daa00640 1 --2- 172.21.15.39:0/2031008333 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd6d41028d0 0x7fd6d419a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:10.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.979+0000 7fd6daa00640 1 --2- 172.21.15.39:0/2031008333 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd6bc0a2800 0x7fd6bc0a4cc0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:10.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.979+0000 7fd6daa00640 1 --2- 172.21.15.39:0/2031008333 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd6d4103290 0x7fd6d41a1170 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:10.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.979+0000 7fd6daa00640 1 --2- 172.21.15.39:0/2031008333 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd6d41016e0 0x7fd6d4199bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:10.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.979+0000 7fd6daa00640 1 -- 172.21.15.39:0/2031008333 >> 172.21.15.39:0/2031008333 conn(0x7fd6d40fcc50 msgr2=0x7fd6d4073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:10.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.979+0000 7fd6daa00640 1 -- 172.21.15.39:0/2031008333 shutdown_connections 2024-03-18T16:11:10.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:10.979+0000 7fd6daa00640 1 -- 172.21.15.39:0/2031008333 wait complete. 2024-03-18T16:11:10.999 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-18T16:11:10.999 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:11:11.005 INFO:tasks.workunit.client.0.smithi039.stdout:checking sys/time.h usability... yes 2024-03-18T16:11:11.054 INFO:tasks.workunit.client.0.smithi039.stdout:checking sys/time.h presence... yes 2024-03-18T16:11:11.055 INFO:tasks.workunit.client.0.smithi039.stdout:checking for sys/time.h... yes 2024-03-18T16:11:11.075 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:11:10 smithi039 ceph-mon[31721]: pgmap v455: 129 pgs: 129 active+clean; 2.5 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 168 KiB/s wr, 19 op/s 2024-03-18T16:11:11.081 INFO:tasks.workunit.client.0.smithi039.stdout:checking for unistd.h... (cached) yes 2024-03-18T16:11:11.180 INFO:tasks.workunit.client.0.smithi039.stdout:checking sys/vfs.h usability... yes 2024-03-18T16:11:11.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.201+0000 7f551f000640 1 -- 172.21.15.39:0/2448823829 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f55181031b0 msgr2=0x7f5518109a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:11.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.201+0000 7f551f000640 1 --2- 172.21.15.39:0/2448823829 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f55181031b0 0x7f5518109a40 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f5510009be0 tx=0x7f551002bef0 comp rx=0 tx=0).stop 2024-03-18T16:11:11.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.201+0000 7f551f000640 1 -- 172.21.15.39:0/2448823829 shutdown_connections 2024-03-18T16:11:11.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.201+0000 7f551f000640 1 --2- 172.21.15.39:0/2448823829 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f55181031b0 0x7f5518109a40 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:11.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.201+0000 7f551f000640 1 --2- 172.21.15.39:0/2448823829 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f55181027f0 0x7f5518102c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:11.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.201+0000 7f551f000640 1 --2- 172.21.15.39:0/2448823829 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5518101600 0x7f5518101a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:11.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.201+0000 7f551f000640 1 -- 172.21.15.39:0/2448823829 >> 172.21.15.39:0/2448823829 conn(0x7f55180fcb70 msgr2=0x7f55180fefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:11.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.201+0000 7f551f000640 1 -- 172.21.15.39:0/2448823829 shutdown_connections 2024-03-18T16:11:11.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.201+0000 7f551f000640 1 -- 172.21.15.39:0/2448823829 wait complete. 2024-03-18T16:11:11.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.202+0000 7f551f000640 1 Processor -- start 2024-03-18T16:11:11.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.203+0000 7f551f000640 1 -- start start 2024-03-18T16:11:11.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.203+0000 7f551f000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5518101600 0x7f5518199a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:11.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.203+0000 7f551f000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f55181027f0 0x7f5518199fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:11.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.203+0000 7f551f000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f55181031b0 0x7f55181a1010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:11.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.203+0000 7f551f000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f551810d2b0 con 0x7f55181027f0 2024-03-18T16:11:11.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.203+0000 7f551f000640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f551810d130 con 0x7f55181031b0 2024-03-18T16:11:11.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.203+0000 7f551f000640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f551810d430 con 0x7f5518101600 2024-03-18T16:11:11.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.203+0000 7f551dc00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5518101600 0x7f5518199a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:11.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.203+0000 7f551dc00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5518101600 0x7f5518199a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:39772/0 (socket says 172.21.15.39:39772) 2024-03-18T16:11:11.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.203+0000 7f551dc00640 1 -- 172.21.15.39:0/3815623223 learned_addr learned my addr 172.21.15.39:0/3815623223 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:11:11.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.203+0000 7f551d200640 1 --2- 172.21.15.39:0/3815623223 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f55181027f0 0x7f5518199fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:11.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.203+0000 7f551e600640 1 --2- 172.21.15.39:0/3815623223 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f55181031b0 0x7f55181a1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:11.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.204+0000 7f551dc00640 1 -- 172.21.15.39:0/3815623223 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f55181031b0 msgr2=0x7f55181a1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:11.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.204+0000 7f551dc00640 1 --2- 172.21.15.39:0/3815623223 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f55181031b0 0x7f55181a1010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:11.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.204+0000 7f551dc00640 1 -- 172.21.15.39:0/3815623223 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f55181027f0 msgr2=0x7f5518199fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:11.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.204+0000 7f551dc00640 1 --2- 172.21.15.39:0/3815623223 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f55181027f0 0x7f5518199fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:11.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.204+0000 7f551dc00640 1 -- 172.21.15.39:0/3815623223 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55181a1680 con 0x7f5518101600 2024-03-18T16:11:11.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.204+0000 7f551d200640 1 --2- 172.21.15.39:0/3815623223 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f55181027f0 0x7f5518199fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:11:11.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.204+0000 7f551dc00640 1 --2- 172.21.15.39:0/3815623223 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5518101600 0x7f5518199a60 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f550c00e9a0 tx=0x7f550c00ee60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:11.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.204+0000 7f5516000640 1 -- 172.21.15.39:0/3815623223 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f550c00cd80 con 0x7f5518101600 2024-03-18T16:11:11.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.204+0000 7f5516000640 1 -- 172.21.15.39:0/3815623223 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f550c010910 con 0x7f5518101600 2024-03-18T16:11:11.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.204+0000 7f5516000640 1 -- 172.21.15.39:0/3815623223 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f550c018a30 con 0x7f5518101600 2024-03-18T16:11:11.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.204+0000 7f551f000640 1 -- 172.21.15.39:0/3815623223 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55181a1970 con 0x7f5518101600 2024-03-18T16:11:11.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.204+0000 7f551f000640 1 -- 172.21.15.39:0/3815623223 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55181a1e70 con 0x7f5518101600 2024-03-18T16:11:11.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.206+0000 7f551f000640 1 -- 172.21.15.39:0/3815623223 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55180684f0 con 0x7f5518101600 2024-03-18T16:11:11.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.207+0000 7f5516000640 1 -- 172.21.15.39:0/3815623223 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f550c010ac0 con 0x7f5518101600 2024-03-18T16:11:11.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.208+0000 7f5516000640 1 --2- 172.21.15.39:0/3815623223 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f55000a2800 0x7f55000a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:11.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.209+0000 7f5516000640 1 -- 172.21.15.39:0/3815623223 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f550c0cf260 con 0x7f5518101600 2024-03-18T16:11:11.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.209+0000 7f551d200640 1 --2- 172.21.15.39:0/3815623223 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f55000a2800 0x7f55000a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:11.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.209+0000 7f551d200640 1 --2- 172.21.15.39:0/3815623223 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f55000a2800 0x7f55000a4cc0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f551819af70 tx=0x7f5504005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:11.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.213+0000 7f5516000640 1 -- 172.21.15.39:0/3815623223 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f550c053740 con 0x7f5518101600 2024-03-18T16:11:11.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:11:10 smithi196 ceph-mon[36307]: pgmap v455: 129 pgs: 129 active+clean; 2.5 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 168 KiB/s wr, 19 op/s 2024-03-18T16:11:11.229 INFO:tasks.workunit.client.0.smithi039.stdout:checking sys/vfs.h presence... yes 2024-03-18T16:11:11.230 INFO:tasks.workunit.client.0.smithi039.stdout:checking for sys/vfs.h... yes 2024-03-18T16:11:11.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:11:10 smithi194 ceph-mon[36200]: pgmap v455: 129 pgs: 129 active+clean; 2.5 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 168 KiB/s wr, 19 op/s 2024-03-18T16:11:11.318 INFO:tasks.workunit.client.0.smithi039.stdout:checking sys/limits.h usability... no 2024-03-18T16:11:11.360 INFO:tasks.workunit.client.0.smithi039.stdout:checking sys/limits.h presence... no 2024-03-18T16:11:11.361 INFO:tasks.workunit.client.0.smithi039.stdout:checking for sys/limits.h... no 2024-03-18T16:11:11.435 INFO:tasks.workunit.client.0.smithi039.stdout:checking for an ANSI C-conforming const... yes 2024-03-18T16:11:11.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.459+0000 7f551f000640 1 -- 172.21.15.39:0/3815623223 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f55181a2800 con 0x7f5518101600 2024-03-18T16:11:11.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.460+0000 7f5516000640 1 -- 172.21.15.39:0/3815623223 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11932 (secure 0 0 0) 0x7f550c09bf10 con 0x7f5518101600 2024-03-18T16:11:11.463 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:11:11.463 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2520226608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2520226608},{"type":"v1","addr":"172.21.15.39:6835","nonce":2520226608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/719341763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":719341763},{"type":"v1","addr":"172.21.15.39:6837","nonce":719341763}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T16:06:44.180425+0000","modified":"2024-03-18T16:09:54.861629+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24575,"mds_1":24556,"mds_2":24535,"mds_3":24524,"mds_4":14463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14445":{"gid":14445,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.39:6839/656513813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":656513813},{"type":"v1","addr":"172.21.15.39:6839","nonce":656513813}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14463":{"gid":14463,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":13,"addr":"172.21.15.39:6841/3515326042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3515326042},{"type":"v1","addr":"172.21.15.39:6841","nonce":3515326042}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6833/931350738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":931350738},{"type":"v1","addr":"172.21.15.194:6833","nonce":931350738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6833/3874209621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":3874209621},{"type":"v1","addr":"172.21.15.196:6833","nonce":3874209621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6835/188385470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":188385470},{"type":"v1","addr":"172.21.15.194:6835","nonce":188385470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6835/3949984309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":3949984309},{"type":"v1","addr":"172.21.15.196:6835","nonce":3949984309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":15,"addr":"172.21.15.194:6837/1301464246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":1301464246},{"type":"v1","addr":"172.21.15.194:6837","nonce":1301464246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":16,"addr":"172.21.15.196:6837/895247251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":895247251},{"type":"v1","addr":"172.21.15.196:6837","nonce":895247251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.196:6839/2001427268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6838","nonce":2001427268},{"type":"v1","addr":"172.21.15.196:6839","nonce":2001427268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24575":{"gid":24575,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.194:6839/274911158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":274911158},{"type":"v1","addr":"172.21.15.194:6839","nonce":274911158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24575,"qdb_cluster":[14463,24524,24575,24556,24535]},"id":1}]} 2024-03-18T16:11:11.463 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 28 2024-03-18T16:11:11.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.463+0000 7f551f000640 1 -- 172.21.15.39:0/3815623223 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f55000a2800 msgr2=0x7f55000a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:11.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.463+0000 7f551f000640 1 --2- 172.21.15.39:0/3815623223 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f55000a2800 0x7f55000a4cc0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f551819af70 tx=0x7f5504005f90 comp rx=0 tx=0).stop 2024-03-18T16:11:11.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.463+0000 7f551f000640 1 -- 172.21.15.39:0/3815623223 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5518101600 msgr2=0x7f5518199a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:11.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.463+0000 7f551f000640 1 --2- 172.21.15.39:0/3815623223 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5518101600 0x7f5518199a60 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f550c00e9a0 tx=0x7f550c00ee60 comp rx=0 tx=0).stop 2024-03-18T16:11:11.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.463+0000 7f551f000640 1 -- 172.21.15.39:0/3815623223 shutdown_connections 2024-03-18T16:11:11.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.463+0000 7f551f000640 1 --2- 172.21.15.39:0/3815623223 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f55181031b0 0x7f55181a1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:11.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.463+0000 7f551f000640 1 --2- 172.21.15.39:0/3815623223 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f55000a2800 0x7f55000a4cc0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:11.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.463+0000 7f551f000640 1 --2- 172.21.15.39:0/3815623223 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f55181027f0 0x7f5518199fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:11.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.463+0000 7f551f000640 1 --2- 172.21.15.39:0/3815623223 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5518101600 0x7f5518199a60 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:11.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.463+0000 7f551f000640 1 -- 172.21.15.39:0/3815623223 >> 172.21.15.39:0/3815623223 conn(0x7f55180fcb70 msgr2=0x7f5518073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:11.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.463+0000 7f551f000640 1 -- 172.21.15.39:0/3815623223 shutdown_connections 2024-03-18T16:11:11.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.463+0000 7f551f000640 1 -- 172.21.15.39:0/3815623223 wait complete. 2024-03-18T16:11:11.484 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-18T16:11:11.510 INFO:tasks.workunit.client.0.smithi039.stdout:checking whether time.h and sys/time.h may both be included... yes 2024-03-18T16:11:11.512 INFO:tasks.workunit.client.0.smithi039.stdout:checking for function prototypes... yes 2024-03-18T16:11:11.576 INFO:tasks.workunit.client.0.smithi039.stdout:checking whether setvbuf arguments are reversed... no 2024-03-18T16:11:11.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.706+0000 7f436de00640 1 -- 172.21.15.39:0/1914878218 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4368074240 msgr2=0x7f43680746c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:11.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.706+0000 7f436de00640 1 --2- 172.21.15.39:0/1914878218 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4368074240 0x7f43680746c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f4354009900 tx=0x7f435402bc30 comp rx=0 tx=0).stop 2024-03-18T16:11:11.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.707+0000 7f436de00640 1 -- 172.21.15.39:0/1914878218 shutdown_connections 2024-03-18T16:11:11.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.707+0000 7f436de00640 1 --2- 172.21.15.39:0/1914878218 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4368074240 0x7f43680746c0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:11.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.707+0000 7f436de00640 1 --2- 172.21.15.39:0/1914878218 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f43681094a0 0x7f436810b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:11.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.707+0000 7f436de00640 1 --2- 172.21.15.39:0/1914878218 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4368075820 0x7f4368073c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:11.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.707+0000 7f436de00640 1 -- 172.21.15.39:0/1914878218 >> 172.21.15.39:0/1914878218 conn(0x7f43680fcb70 msgr2=0x7f43680fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:11.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.707+0000 7f436de00640 1 -- 172.21.15.39:0/1914878218 shutdown_connections 2024-03-18T16:11:11.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.707+0000 7f436de00640 1 -- 172.21.15.39:0/1914878218 wait complete. 2024-03-18T16:11:11.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.708+0000 7f436de00640 1 Processor -- start 2024-03-18T16:11:11.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.708+0000 7f436de00640 1 -- start start 2024-03-18T16:11:11.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.708+0000 7f436de00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4368074240 0x7f4368199aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:11.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.708+0000 7f436de00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4368075820 0x7f4368199fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:11.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.708+0000 7f436de00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f43681094a0 0x7f43681a1050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:11.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.708+0000 7f436de00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f436810f490 con 0x7f4368074240 2024-03-18T16:11:11.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.708+0000 7f436de00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f436810f310 con 0x7f43681094a0 2024-03-18T16:11:11.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.708+0000 7f436de00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f436810f610 con 0x7f4368075820 2024-03-18T16:11:11.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.708+0000 7f4367e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4368075820 0x7f4368199fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:11.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.708+0000 7f436d400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f43681094a0 0x7f43681a1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:11.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.708+0000 7f4367e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4368075820 0x7f4368199fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:39800/0 (socket says 172.21.15.39:39800) 2024-03-18T16:11:11.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.708+0000 7f4367e00640 1 -- 172.21.15.39:0/639234088 learned_addr learned my addr 172.21.15.39:0/639234088 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:11:11.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.708+0000 7f436ca00640 1 --2- 172.21.15.39:0/639234088 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4368074240 0x7f4368199aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:11.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.709+0000 7f4367e00640 1 -- 172.21.15.39:0/639234088 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f43681094a0 msgr2=0x7f43681a1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:11.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.709+0000 7f4367e00640 1 --2- 172.21.15.39:0/639234088 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f43681094a0 0x7f43681a1050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:11.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.709+0000 7f4367e00640 1 -- 172.21.15.39:0/639234088 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4368074240 msgr2=0x7f4368199aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:11.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.709+0000 7f4367e00640 1 --2- 172.21.15.39:0/639234088 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4368074240 0x7f4368199aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:11.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.709+0000 7f4367e00640 1 -- 172.21.15.39:0/639234088 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43681a16c0 con 0x7f4368075820 2024-03-18T16:11:11.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.709+0000 7f436d400640 1 --2- 172.21.15.39:0/639234088 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f43681094a0 0x7f43681a1050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:11:11.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.709+0000 7f4367e00640 1 --2- 172.21.15.39:0/639234088 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4368075820 0x7f4368199fe0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f4354000c00 tx=0x7f435402c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:11.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.709+0000 7f4365600640 1 -- 172.21.15.39:0/639234088 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4354043020 con 0x7f4368075820 2024-03-18T16:11:11.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.709+0000 7f4365600640 1 -- 172.21.15.39:0/639234088 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f43540054e0 con 0x7f4368075820 2024-03-18T16:11:11.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.710+0000 7f436de00640 1 -- 172.21.15.39:0/639234088 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43681a1950 con 0x7f4368075820 2024-03-18T16:11:11.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.710+0000 7f4365600640 1 -- 172.21.15.39:0/639234088 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4354003c40 con 0x7f4368075820 2024-03-18T16:11:11.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.710+0000 7f436de00640 1 -- 172.21.15.39:0/639234088 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43681a1ea0 con 0x7f4368075820 2024-03-18T16:11:11.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.713+0000 7f4365600640 1 -- 172.21.15.39:0/639234088 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f4354046020 con 0x7f4368075820 2024-03-18T16:11:11.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.713+0000 7f4372171740 1 Processor -- start 2024-03-18T16:11:11.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.713+0000 7f4372171740 1 -- start start 2024-03-18T16:11:11.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.713+0000 7f4372171740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ef94d47a90 0x55ef94d47eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:11.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.713+0000 7f4372171740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ef94d48480 0x55ef94d4cf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:11.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.713+0000 7f4372171740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ef94d4d5f0 0x55ef94d4fa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:11.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.713+0000 7f4372171740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55ef94d0f3b0 con 0x55ef94d47a90 2024-03-18T16:11:11.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.713+0000 7f4372171740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55ef94cac450 con 0x55ef94d48480 2024-03-18T16:11:11.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.713+0000 7f4372171740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55ef94c54cb0 con 0x55ef94d4d5f0 2024-03-18T16:11:11.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.714+0000 7f436ca00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ef94d48480 0x55ef94d4cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:11.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.714+0000 7f436d400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ef94d47a90 0x55ef94d47eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:11.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.714+0000 7f436ca00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ef94d48480 0x55ef94d4cf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:54478/0 (socket says 172.21.15.39:54478) 2024-03-18T16:11:11.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.714+0000 7f436ca00640 1 -- 172.21.15.39:0/3996983799 learned_addr learned my addr 172.21.15.39:0/3996983799 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:11:11.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.714+0000 7f4367e00640 1 --2- 172.21.15.39:0/3996983799 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ef94d4d5f0 0x55ef94d4fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:11.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.714+0000 7f436ca00640 1 -- 172.21.15.39:0/3996983799 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ef94d4d5f0 msgr2=0x55ef94d4fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:11.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.714+0000 7f436ca00640 1 --2- 172.21.15.39:0/3996983799 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ef94d4d5f0 0x55ef94d4fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:11.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.714+0000 7f436ca00640 1 -- 172.21.15.39:0/3996983799 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ef94d47a90 msgr2=0x55ef94d47eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:11.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.714+0000 7f436ca00640 1 --2- 172.21.15.39:0/3996983799 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ef94d47a90 0x55ef94d47eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:11.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.714+0000 7f436ca00640 1 -- 172.21.15.39:0/3996983799 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef94c761f0 con 0x55ef94d48480 2024-03-18T16:11:11.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.715+0000 7f4365600640 1 --2- 172.21.15.39:0/639234088 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f43500a2800 0x7f43500a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:11.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.715+0000 7f436ca00640 1 --2- 172.21.15.39:0/3996983799 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ef94d48480 0x55ef94d4cf80 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f435c009cf0 tx=0x7f435c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=87dadfd145cefc4f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:11.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.715+0000 7f4365600640 1 -- 172.21.15.39:0/639234088 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f43540f0e90 con 0x7f4368075820 2024-03-18T16:11:11.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.715+0000 7f4359c00640 1 -- 172.21.15.39:0/3996983799 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f435c004230 con 0x55ef94d48480 2024-03-18T16:11:11.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.715+0000 7f436d400640 1 --2- 172.21.15.39:0/639234088 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f43500a2800 0x7f43500a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:11.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.715+0000 7f4359c00640 1 -- 172.21.15.39:0/3996983799 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f435c0043d0 con 0x55ef94d48480 2024-03-18T16:11:11.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.715+0000 7f4359c00640 1 -- 172.21.15.39:0/3996983799 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f435c019960 con 0x55ef94d48480 2024-03-18T16:11:11.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.716+0000 7f436d400640 1 --2- 172.21.15.39:0/639234088 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f43500a2800 0x7f43500a4cc0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x55ef94d48ce0 tx=0x7f4360009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:11.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.716+0000 7f4372171740 1 -- 172.21.15.39:0/3996983799 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ef94d48480 msgr2=0x55ef94d4cf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:11.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.716+0000 7f4372171740 1 --2- 172.21.15.39:0/3996983799 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ef94d48480 0x55ef94d4cf80 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f435c009cf0 tx=0x7f435c011810 comp rx=0 tx=0).stop 2024-03-18T16:11:11.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.716+0000 7f4372171740 1 -- 172.21.15.39:0/3996983799 shutdown_connections 2024-03-18T16:11:11.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.716+0000 7f4372171740 1 --2- 172.21.15.39:0/3996983799 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ef94d48480 0x55ef94d4cf80 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:11.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.716+0000 7f4372171740 1 --2- 172.21.15.39:0/3996983799 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ef94d4d5f0 0x55ef94d4fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:11.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.716+0000 7f4372171740 1 --2- 172.21.15.39:0/3996983799 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ef94d47a90 0x55ef94d47eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:11.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.716+0000 7f4372171740 1 -- 172.21.15.39:0/3996983799 >> 172.21.15.39:0/3996983799 conn(0x55ef94d43000 msgr2=0x55ef94d43400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:11.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.716+0000 7f4372171740 1 -- 172.21.15.39:0/3996983799 shutdown_connections 2024-03-18T16:11:11.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.716+0000 7f4372171740 1 -- 172.21.15.39:0/3996983799 wait complete. 2024-03-18T16:11:11.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.717+0000 7f4372171740 1 -- start start 2024-03-18T16:11:11.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.717+0000 7f4372171740 1 Processor -- start 2024-03-18T16:11:11.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.718+0000 7f4372171740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ef94d47a90 0x55ef94d51d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:11.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.718+0000 7f4372171740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ef94d48480 0x55ef94d522d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:11.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.718+0000 7f4372171740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ef94d4d5f0 0x55ef94d59310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:11.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.718+0000 7f4372171740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55ef94c68110 con 0x55ef94d47a90 2024-03-18T16:11:11.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.718+0000 7f4372171740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55ef94ca84f0 con 0x55ef94d4d5f0 2024-03-18T16:11:11.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.718+0000 7f4372171740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55ef94d505a0 con 0x55ef94d48480 2024-03-18T16:11:11.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.718+0000 7f4372171740 10 client.0 _ll_register_callbacks cb 0x55ef94c55b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:11:11.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.718+0000 7f436ca00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ef94d47a90 0x55ef94d51d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:11.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.718+0000 7f436ca00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ef94d47a90 0x55ef94d51d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:34656/0 (socket says 172.21.15.39:34656) 2024-03-18T16:11:11.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.718+0000 7f4367e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ef94d48480 0x55ef94d522d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:11.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.718+0000 7f436ca00640 1 -- 172.21.15.39:0/670635867 learned_addr learned my addr 172.21.15.39:0/670635867 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:11:11.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.718+0000 7f436d400640 1 --2- 172.21.15.39:0/670635867 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ef94d4d5f0 0x55ef94d59310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:11.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.719+0000 7f4367e00640 1 -- 172.21.15.39:0/670635867 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ef94d4d5f0 msgr2=0x55ef94d59310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:11.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.719+0000 7f4367e00640 1 --2- 172.21.15.39:0/670635867 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ef94d4d5f0 0x55ef94d59310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:11.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.719+0000 7f4367e00640 1 -- 172.21.15.39:0/670635867 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ef94d47a90 msgr2=0x55ef94d51d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:11.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.719+0000 7f4367e00640 1 --2- 172.21.15.39:0/670635867 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ef94d47a90 0x55ef94d51d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:11.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.719+0000 7f4367e00640 1 -- 172.21.15.39:0/670635867 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef949c03f0 con 0x55ef94d48480 2024-03-18T16:11:11.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.719+0000 7f4367e00640 1 --2- 172.21.15.39:0/670635867 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ef94d48480 0x55ef94d522d0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f43540ee5a0 tx=0x7f43540b9760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6191865b2bd313a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:11.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.719+0000 7f434e000640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:11:11.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.719+0000 7f434e000640 1 -- 172.21.15.39:0/670635867 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4354004250 con 0x55ef94d48480 2024-03-18T16:11:11.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.719+0000 7f434e000640 1 -- 172.21.15.39:0/670635867 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f43540ee6b0 con 0x55ef94d48480 2024-03-18T16:11:11.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.719+0000 7f4372171740 1 -- 172.21.15.39:0/670635867 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f435c0043d0 con 0x55ef94d48480 2024-03-18T16:11:11.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.719+0000 7f434e000640 1 -- 172.21.15.39:0/670635867 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f43540035f0 con 0x55ef94d48480 2024-03-18T16:11:11.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.719+0000 7f4372171740 10 client.25801 fetch_fsmap learned FSMap version 28 2024-03-18T16:11:11.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.720+0000 7f4372171740 1 -- 172.21.15.39:0/670635867 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ef94a8a7d0 con 0x55ef94d48480 2024-03-18T16:11:11.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.720+0000 7f434e000640 1 -- 172.21.15.39:0/670635867 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f435c0043d0 con 0x55ef94d48480 2024-03-18T16:11:11.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.720+0000 7f434e000640 1 -- 172.21.15.39:0/670635867 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f43540c1150 con 0x55ef94d48480 2024-03-18T16:11:11.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.720+0000 7f4372171740 10 client.25801 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T16:11:11.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.720+0000 7f4372171740 10 client.25801 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-03-18T16:11:11.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.720+0000 7f4372171740 20 client.25801 populate_metadata read hostname 'smithi039' 2024-03-18T16:11:11.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.720+0000 7f4372171740 1 --2- 172.21.15.39:0/670635867 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x55ef94d5dc80 0x55ef94d60120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:11.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.720+0000 7f4372171740 4 client.25801 mds_command: new command op to 14463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:11:11.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.720+0000 7f4372171740 1 -- 172.21.15.39:0/670635867 --> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ef94cb9990 con 0x55ef94d5dc80 2024-03-18T16:11:11.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.720+0000 7f436ca00640 1 --2- 172.21.15.39:0/670635867 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x55ef94d5dc80 0x55ef94d60120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:11.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.721+0000 7f436ca00640 1 --2- 172.21.15.39:0/670635867 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x55ef94d5dc80 0x55ef94d60120 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=2053fa71516edc1f server_cookie=5d1c269fd1e8705b in_seq=0 out_seq=0 2024-03-18T16:11:11.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.721+0000 7f434e000640 10 client.25801 ms_handle_connect on v2:172.21.15.39:6840/3515326042 2024-03-18T16:11:11.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.722+0000 7f434e000640 1 -- 172.21.15.39:0/670635867 <== mds.4 v2:172.21.15.39:6840/3515326042 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ef94cb9990 con 0x55ef94d5dc80 2024-03-18T16:11:11.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.722+0000 7f434e000640 10 client.25801 handle_command_reply: tid=0 2024-03-18T16:11:11.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.722+0000 7f4372171740 1 client.25801 shutdown 2024-03-18T16:11:11.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.722+0000 7f4372171740 1 -- 172.21.15.39:0/670635867 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ef94d48480 msgr2=0x55ef94d522d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:11.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.722+0000 7f4372171740 1 --2- 172.21.15.39:0/670635867 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ef94d48480 0x55ef94d522d0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f43540ee5a0 tx=0x7f43540b9760 comp rx=0 tx=0).stop 2024-03-18T16:11:11.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.722+0000 7f4372171740 1 -- 172.21.15.39:0/670635867 shutdown_connections 2024-03-18T16:11:11.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.722+0000 7f4372171740 1 --2- 172.21.15.39:0/670635867 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x55ef94d5dc80 0x55ef94d60120 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:11.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.722+0000 7f4372171740 1 --2- 172.21.15.39:0/670635867 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ef94d4d5f0 0x55ef94d59310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:11.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.722+0000 7f4372171740 1 --2- 172.21.15.39:0/670635867 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ef94d48480 0x55ef94d522d0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:11.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.722+0000 7f4372171740 1 --2- 172.21.15.39:0/670635867 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ef94d47a90 0x55ef94d51d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:11.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.722+0000 7f434e000640 0 client.25801 ms_handle_reset on v2:172.21.15.39:6840/3515326042 2024-03-18T16:11:11.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.722+0000 7f4372171740 1 -- 172.21.15.39:0/670635867 >> 172.21.15.39:0/670635867 conn(0x55ef94d43000 msgr2=0x55ef94d42220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:11.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.723+0000 7f4372171740 1 -- 172.21.15.39:0/670635867 shutdown_connections 2024-03-18T16:11:11.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.723+0000 7f4372171740 1 -- 172.21.15.39:0/670635867 wait complete. 2024-03-18T16:11:11.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.723+0000 7f4372171740 20 client.25801 trim_cache size 0 max 16384 2024-03-18T16:11:11.729 INFO:tasks.workunit.client.0.smithi039.stdout:checking for system... yes 2024-03-18T16:11:11.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.741+0000 7f4372171740 1 Processor -- start 2024-03-18T16:11:11.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.741+0000 7f4372171740 1 -- start start 2024-03-18T16:11:11.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.741+0000 7f4372171740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ef94d47a90 0x55ef94d41990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:11.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.741+0000 7f4372171740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ef94d48480 0x55ef94d592d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:11.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.741+0000 7f4372171740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ef94d4d5f0 0x55ef94d41ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:11.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.741+0000 7f4372171740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f43681ac3b0 con 0x55ef94d47a90 2024-03-18T16:11:11.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.741+0000 7f4372171740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f43681adef0 con 0x55ef94d4d5f0 2024-03-18T16:11:11.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.741+0000 7f4372171740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f43681aca80 con 0x55ef94d48480 2024-03-18T16:11:11.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.741+0000 7f4367e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ef94d48480 0x55ef94d592d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:11.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.741+0000 7f436ca00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ef94d47a90 0x55ef94d41990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:11.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.741+0000 7f4367e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ef94d48480 0x55ef94d592d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:39826/0 (socket says 172.21.15.39:39826) 2024-03-18T16:11:11.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.742+0000 7f4367e00640 1 -- 172.21.15.39:0/1370819796 learned_addr learned my addr 172.21.15.39:0/1370819796 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:11:11.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.741+0000 7f436d400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ef94d4d5f0 0x55ef94d41ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:11.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.742+0000 7f4367e00640 1 -- 172.21.15.39:0/1370819796 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ef94d4d5f0 msgr2=0x55ef94d41ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:11.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.742+0000 7f4367e00640 1 --2- 172.21.15.39:0/1370819796 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ef94d4d5f0 0x55ef94d41ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:11.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.742+0000 7f4367e00640 1 -- 172.21.15.39:0/1370819796 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ef94d47a90 msgr2=0x55ef94d41990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:11.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.742+0000 7f4367e00640 1 --2- 172.21.15.39:0/1370819796 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ef94d47a90 0x55ef94d41990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:11.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.742+0000 7f4367e00640 1 -- 172.21.15.39:0/1370819796 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ef94d42ba0 con 0x55ef94d48480 2024-03-18T16:11:11.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.742+0000 7f4367e00640 1 --2- 172.21.15.39:0/1370819796 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ef94d48480 0x55ef94d592d0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f4354006010 tx=0x7f435402c8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5baaa70452ccf96d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:11.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.742+0000 7f4359200640 1 -- 172.21.15.39:0/1370819796 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4354004250 con 0x55ef94d48480 2024-03-18T16:11:11.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.743+0000 7f4359200640 1 -- 172.21.15.39:0/1370819796 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4354005690 con 0x55ef94d48480 2024-03-18T16:11:11.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.743+0000 7f4359200640 1 -- 172.21.15.39:0/1370819796 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43540ee6b0 con 0x55ef94d48480 2024-03-18T16:11:11.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.743+0000 7f4372171740 1 -- 172.21.15.39:0/1370819796 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ef94d48480 msgr2=0x55ef94d592d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:11.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.743+0000 7f4372171740 1 --2- 172.21.15.39:0/1370819796 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ef94d48480 0x55ef94d592d0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f4354006010 tx=0x7f435402c8a0 comp rx=0 tx=0).stop 2024-03-18T16:11:11.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.743+0000 7f4372171740 1 -- 172.21.15.39:0/1370819796 shutdown_connections 2024-03-18T16:11:11.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.743+0000 7f4372171740 1 --2- 172.21.15.39:0/1370819796 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ef94d4d5f0 0x55ef94d41ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:11.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.743+0000 7f4372171740 1 --2- 172.21.15.39:0/1370819796 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ef94d48480 0x55ef94d592d0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:11.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.743+0000 7f4372171740 1 --2- 172.21.15.39:0/1370819796 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ef94d47a90 0x55ef94d41990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:11.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.743+0000 7f4372171740 1 -- 172.21.15.39:0/1370819796 >> 172.21.15.39:0/1370819796 conn(0x55ef94d43000 msgr2=0x55ef94d433e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:11.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.744+0000 7f4372171740 1 -- 172.21.15.39:0/1370819796 shutdown_connections 2024-03-18T16:11:11.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.744+0000 7f4372171740 1 -- 172.21.15.39:0/1370819796 wait complete. 2024-03-18T16:11:11.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.744+0000 7f4372171740 1 -- start start 2024-03-18T16:11:11.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.744+0000 7f4372171740 1 Processor -- start 2024-03-18T16:11:11.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.745+0000 7f4372171740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ef94d47a90 0x55ef94d56b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:11.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.745+0000 7f4372171740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ef94d48480 0x55ef94d57080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:11.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.745+0000 7f4372171740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ef94d4d5f0 0x55ef94d57fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:11.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.745+0000 7f4372171740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f43681ad820 con 0x55ef94d47a90 2024-03-18T16:11:11.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.745+0000 7f4372171740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f43681ad150 con 0x55ef94d4d5f0 2024-03-18T16:11:11.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.745+0000 7f4372171740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f43681aa390 con 0x55ef94d48480 2024-03-18T16:11:11.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.745+0000 7f4372171740 10 client.0 _ll_register_callbacks cb 0x7f4354009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:11:11.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.745+0000 7f4367e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ef94d48480 0x55ef94d57080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:11.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.745+0000 7f436ca00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ef94d47a90 0x55ef94d56b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:11.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.745+0000 7f4367e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ef94d48480 0x55ef94d57080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:39840/0 (socket says 172.21.15.39:39840) 2024-03-18T16:11:11.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.745+0000 7f4367e00640 1 -- 172.21.15.39:0/1531853387 learned_addr learned my addr 172.21.15.39:0/1531853387 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:11:11.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.745+0000 7f436d400640 1 --2- 172.21.15.39:0/1531853387 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ef94d4d5f0 0x55ef94d57fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:11.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.745+0000 7f4367e00640 1 -- 172.21.15.39:0/1531853387 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ef94d4d5f0 msgr2=0x55ef94d57fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:11.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.745+0000 7f4367e00640 1 --2- 172.21.15.39:0/1531853387 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ef94d4d5f0 0x55ef94d57fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:11.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.745+0000 7f4367e00640 1 -- 172.21.15.39:0/1531853387 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ef94d47a90 msgr2=0x55ef94d56b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:11.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.745+0000 7f4367e00640 1 --2- 172.21.15.39:0/1531853387 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ef94d47a90 0x55ef94d56b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:11.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.746+0000 7f4367e00640 1 -- 172.21.15.39:0/1531853387 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ef94d585a0 con 0x55ef94d48480 2024-03-18T16:11:11.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.746+0000 7f4367e00640 1 --2- 172.21.15.39:0/1531853387 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ef94d48480 0x55ef94d57080 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f435402ceb0 tx=0x7f43540f4470 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e3c743444f08f96f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:11.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.746+0000 7f434e000640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:11:11.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.746+0000 7f434e000640 1 -- 172.21.15.39:0/1531853387 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4354004250 con 0x55ef94d48480 2024-03-18T16:11:11.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.746+0000 7f434e000640 1 -- 172.21.15.39:0/1531853387 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f43540ee6b0 con 0x55ef94d48480 2024-03-18T16:11:11.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.746+0000 7f4372171740 1 -- 172.21.15.39:0/1531853387 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55ef94cb3ce0 con 0x55ef94d48480 2024-03-18T16:11:11.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.746+0000 7f434e000640 1 -- 172.21.15.39:0/1531853387 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f4354004c60 con 0x55ef94d48480 2024-03-18T16:11:11.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.746+0000 7f4372171740 10 client.25807 fetch_fsmap learned FSMap version 28 2024-03-18T16:11:11.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.746+0000 7f4372171740 1 -- 172.21.15.39:0/1531853387 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ef94d585a0 con 0x55ef94d48480 2024-03-18T16:11:11.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.746+0000 7f434e000640 1 -- 172.21.15.39:0/1531853387 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55ef94cb3ce0 con 0x55ef94d48480 2024-03-18T16:11:11.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.747+0000 7f434e000640 1 -- 172.21.15.39:0/1531853387 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f43540bc890 con 0x55ef94d48480 2024-03-18T16:11:11.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.747+0000 7f4372171740 10 client.25807 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T16:11:11.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.747+0000 7f4372171740 10 client.25807 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-03-18T16:11:11.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.747+0000 7f4372171740 20 client.25807 populate_metadata read hostname 'smithi039' 2024-03-18T16:11:11.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.747+0000 7f4372171740 1 --2- 172.21.15.39:0/1531853387 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x55ef94dc45b0 0x55ef94dc69d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:11.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.747+0000 7f4372171740 4 client.25807 mds_command: new command op to 14463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:11:11.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.747+0000 7f4372171740 1 -- 172.21.15.39:0/1531853387 --> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ef94c31560 con 0x55ef94dc45b0 2024-03-18T16:11:11.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.747+0000 7f436ca00640 1 --2- 172.21.15.39:0/1531853387 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x55ef94dc45b0 0x55ef94dc69d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:11.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.748+0000 7f436ca00640 1 --2- 172.21.15.39:0/1531853387 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x55ef94dc45b0 0x55ef94dc69d0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=72c23e859226210c server_cookie=6868b0dbc0c62bf in_seq=0 out_seq=0 2024-03-18T16:11:11.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.748+0000 7f434e000640 10 client.25807 ms_handle_connect on v2:172.21.15.39:6840/3515326042 2024-03-18T16:11:11.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.748+0000 7f434e000640 1 -- 172.21.15.39:0/1531853387 <== mds.4 v2:172.21.15.39:6840/3515326042 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ef94c31560 con 0x55ef94dc45b0 2024-03-18T16:11:11.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.748+0000 7f434e000640 10 client.25807 handle_command_reply: tid=0 2024-03-18T16:11:11.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.748+0000 7f4372171740 1 client.25807 shutdown 2024-03-18T16:11:11.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.748+0000 7f4372171740 1 -- 172.21.15.39:0/1531853387 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ef94d48480 msgr2=0x55ef94d57080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:11.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.748+0000 7f4372171740 1 --2- 172.21.15.39:0/1531853387 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ef94d48480 0x55ef94d57080 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f435402ceb0 tx=0x7f43540f4470 comp rx=0 tx=0).stop 2024-03-18T16:11:11.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.748+0000 7f4372171740 1 -- 172.21.15.39:0/1531853387 shutdown_connections 2024-03-18T16:11:11.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.748+0000 7f4372171740 1 --2- 172.21.15.39:0/1531853387 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x55ef94dc45b0 0x55ef94dc69d0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:11.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.748+0000 7f4372171740 1 --2- 172.21.15.39:0/1531853387 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ef94d4d5f0 0x55ef94d57fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:11.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.748+0000 7f4372171740 1 --2- 172.21.15.39:0/1531853387 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ef94d48480 0x55ef94d57080 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:11.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.748+0000 7f4372171740 1 --2- 172.21.15.39:0/1531853387 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ef94d47a90 0x55ef94d56b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:11.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.748+0000 7f434e000640 0 client.25807 ms_handle_reset on v2:172.21.15.39:6840/3515326042 2024-03-18T16:11:11.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.748+0000 7f4372171740 1 -- 172.21.15.39:0/1531853387 >> 172.21.15.39:0/1531853387 conn(0x55ef94d43000 msgr2=0x55ef94d46260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:11.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.749+0000 7f4372171740 1 -- 172.21.15.39:0/1531853387 shutdown_connections 2024-03-18T16:11:11.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.749+0000 7f4372171740 1 -- 172.21.15.39:0/1531853387 wait complete. 2024-03-18T16:11:11.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.749+0000 7f4372171740 20 client.25807 trim_cache size 0 max 16384 2024-03-18T16:11:11.755 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:11:11.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.749+0000 7f436de00640 1 -- 172.21.15.39:0/639234088 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f43500a2800 msgr2=0x7f43500a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:11.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.749+0000 7f436de00640 1 --2- 172.21.15.39:0/639234088 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f43500a2800 0x7f43500a4cc0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x55ef94d48ce0 tx=0x7f4360009500 comp rx=0 tx=0).stop 2024-03-18T16:11:11.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.749+0000 7f436de00640 1 -- 172.21.15.39:0/639234088 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4368075820 msgr2=0x7f4368199fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:11.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.749+0000 7f436de00640 1 --2- 172.21.15.39:0/639234088 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4368075820 0x7f4368199fe0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f4354000c00 tx=0x7f435402c650 comp rx=0 tx=0).stop 2024-03-18T16:11:11.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.750+0000 7f436de00640 1 -- 172.21.15.39:0/639234088 shutdown_connections 2024-03-18T16:11:11.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.750+0000 7f436de00640 1 --2- 172.21.15.39:0/639234088 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f43681094a0 0x7f43681a1050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:11.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.750+0000 7f436de00640 1 --2- 172.21.15.39:0/639234088 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f43500a2800 0x7f43500a4cc0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:11.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.750+0000 7f436de00640 1 --2- 172.21.15.39:0/639234088 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4368075820 0x7f4368199fe0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:11.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.750+0000 7f436de00640 1 --2- 172.21.15.39:0/639234088 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4368074240 0x7f4368199aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:11.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.750+0000 7f436de00640 1 -- 172.21.15.39:0/639234088 >> 172.21.15.39:0/639234088 conn(0x7f43680fcb70 msgr2=0x7f43681053a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:11.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.750+0000 7f436de00640 1 -- 172.21.15.39:0/639234088 shutdown_connections 2024-03-18T16:11:11.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:11.750+0000 7f436de00640 1 -- 172.21.15.39:0/639234088 wait complete. 2024-03-18T16:11:11.774 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-18T16:11:11.855 INFO:tasks.workunit.client.0.smithi039.stdout:checking for gettimeofday... yes 2024-03-18T16:11:11.996 INFO:tasks.workunit.client.0.smithi039.stdout:checking for mkdir... yes 2024-03-18T16:11:12.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.032+0000 7fadd7e00640 1 -- 172.21.15.39:0/935376799 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fadd01055b0 msgr2=0x7fadd010be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:12.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.032+0000 7fadd7e00640 1 --2- 172.21.15.39:0/935376799 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fadd01055b0 0x7fadd010be40 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fadc8009ca0 tx=0x7fadc802c100 comp rx=0 tx=0).stop 2024-03-18T16:11:12.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.032+0000 7fadd7e00640 1 -- 172.21.15.39:0/935376799 shutdown_connections 2024-03-18T16:11:12.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.032+0000 7fadd7e00640 1 --2- 172.21.15.39:0/935376799 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fadd01055b0 0x7fadd010be40 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.032+0000 7fadd7e00640 1 --2- 172.21.15.39:0/935376799 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fadd0104bf0 0x7fadd0105070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.032+0000 7fadd7e00640 1 --2- 172.21.15.39:0/935376799 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fadd0103a00 0x7fadd0103e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.032+0000 7fadd7e00640 1 -- 172.21.15.39:0/935376799 >> 172.21.15.39:0/935376799 conn(0x7fadd00fef90 msgr2=0x7fadd01013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:12.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.033+0000 7fadd7e00640 1 -- 172.21.15.39:0/935376799 shutdown_connections 2024-03-18T16:11:12.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.033+0000 7fadd7e00640 1 -- 172.21.15.39:0/935376799 wait complete. 2024-03-18T16:11:12.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.034+0000 7fadd7e00640 1 Processor -- start 2024-03-18T16:11:12.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.034+0000 7fadd7e00640 1 -- start start 2024-03-18T16:11:12.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.034+0000 7fadd7e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fadd0103a00 0x7fadd0199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:12.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.034+0000 7fadd7e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fadd0104bf0 0x7fadd019a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:12.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.034+0000 7fadd7e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fadd01055b0 0x7fadd01a11e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:12.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.034+0000 7fadd7e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fadd010f660 con 0x7fadd0104bf0 2024-03-18T16:11:12.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.034+0000 7fadd7e00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fadd010f4e0 con 0x7fadd01055b0 2024-03-18T16:11:12.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.034+0000 7fadd7e00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fadd010f7e0 con 0x7fadd0103a00 2024-03-18T16:11:12.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.034+0000 7fadd5400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fadd0103a00 0x7fadd0199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:12.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.034+0000 7fadd5400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fadd0103a00 0x7fadd0199c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:39864/0 (socket says 172.21.15.39:39864) 2024-03-18T16:11:12.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.034+0000 7fadd5400640 1 -- 172.21.15.39:0/1343822256 learned_addr learned my addr 172.21.15.39:0/1343822256 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:11:12.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.034+0000 7fadd4a00640 1 --2- 172.21.15.39:0/1343822256 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fadd0104bf0 0x7fadd019a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:12.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.035+0000 7fadd5e00640 1 --2- 172.21.15.39:0/1343822256 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fadd01055b0 0x7fadd01a11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:12.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.035+0000 7fadd5400640 1 -- 172.21.15.39:0/1343822256 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fadd01055b0 msgr2=0x7fadd01a11e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:12.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.035+0000 7fadd5400640 1 --2- 172.21.15.39:0/1343822256 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fadd01055b0 0x7fadd01a11e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.035+0000 7fadd5400640 1 -- 172.21.15.39:0/1343822256 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fadd0104bf0 msgr2=0x7fadd019a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:12.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.035+0000 7fadd5400640 1 --2- 172.21.15.39:0/1343822256 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fadd0104bf0 0x7fadd019a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.035+0000 7fadd5400640 1 -- 172.21.15.39:0/1343822256 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadd01a1880 con 0x7fadd0103a00 2024-03-18T16:11:12.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.035+0000 7fadd5e00640 1 --2- 172.21.15.39:0/1343822256 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fadd01055b0 0x7fadd01a11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:11:12.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.035+0000 7fadd4a00640 1 --2- 172.21.15.39:0/1343822256 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fadd0104bf0 0x7fadd019a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:11:12.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.035+0000 7fadd5400640 1 --2- 172.21.15.39:0/1343822256 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fadd0103a00 0x7fadd0199c70 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fadc400e9c0 tx=0x7fadc400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:12.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.035+0000 7fadce000640 1 -- 172.21.15.39:0/1343822256 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadc400cd80 con 0x7fadd0103a00 2024-03-18T16:11:12.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.035+0000 7fadce000640 1 -- 172.21.15.39:0/1343822256 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fadc40035f0 con 0x7fadd0103a00 2024-03-18T16:11:12.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.035+0000 7fadce000640 1 -- 172.21.15.39:0/1343822256 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadc4010690 con 0x7fadd0103a00 2024-03-18T16:11:12.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.035+0000 7fadd7e00640 1 -- 172.21.15.39:0/1343822256 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fadd01a1b70 con 0x7fadd0103a00 2024-03-18T16:11:12.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.036+0000 7fadd7e00640 1 -- 172.21.15.39:0/1343822256 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fadd01a2070 con 0x7fadd0103a00 2024-03-18T16:11:12.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.040+0000 7fadce000640 1 -- 172.21.15.39:0/1343822256 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fadc40037a0 con 0x7fadd0103a00 2024-03-18T16:11:12.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.040+0000 7faddab88740 1 Processor -- start 2024-03-18T16:11:12.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.041+0000 7faddab88740 1 -- start start 2024-03-18T16:11:12.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.041+0000 7faddab88740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5569660eb8c0 0x5569660ebce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:12.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.041+0000 7faddab88740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5569660ec2b0 0x5569660f0db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:12.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.041+0000 7faddab88740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5569660f1420 0x5569660f38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:12.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.041+0000 7faddab88740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5569660b33b0 con 0x5569660eb8c0 2024-03-18T16:11:12.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.041+0000 7faddab88740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x556966050450 con 0x5569660ec2b0 2024-03-18T16:11:12.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.041+0000 7faddab88740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x556965ff8cb0 con 0x5569660f1420 2024-03-18T16:11:12.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.041+0000 7fadd4a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5569660f1420 0x5569660f38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:12.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.041+0000 7fadd4a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5569660f1420 0x5569660f38b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:39868/0 (socket says 172.21.15.39:39868) 2024-03-18T16:11:12.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.041+0000 7fadd4a00640 1 -- 172.21.15.39:0/2520381637 learned_addr learned my addr 172.21.15.39:0/2520381637 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:11:12.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.041+0000 7fadd5e00640 1 --2- 172.21.15.39:0/2520381637 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5569660eb8c0 0x5569660ebce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:12.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.042+0000 7fadd5400640 1 --2- 172.21.15.39:0/2520381637 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5569660ec2b0 0x5569660f0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:12.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.042+0000 7fadce000640 1 --2- 172.21.15.39:0/1343822256 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fadb80a2880 0x7fadb80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:12.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.042+0000 7fadd4a00640 1 -- 172.21.15.39:0/2520381637 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5569660ec2b0 msgr2=0x5569660f0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:12.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.042+0000 7fadd4a00640 1 --2- 172.21.15.39:0/2520381637 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5569660ec2b0 0x5569660f0db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.042+0000 7fadd4a00640 1 -- 172.21.15.39:0/2520381637 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5569660eb8c0 msgr2=0x5569660ebce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:12.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.042+0000 7fadd4a00640 1 --2- 172.21.15.39:0/2520381637 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5569660eb8c0 0x5569660ebce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.042+0000 7fadd4a00640 1 -- 172.21.15.39:0/2520381637 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55696601a1f0 con 0x5569660f1420 2024-03-18T16:11:12.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.042+0000 7fadd5e00640 1 --2- 172.21.15.39:0/2520381637 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5569660eb8c0 0x5569660ebce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:11:12.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.042+0000 7fadce000640 1 -- 172.21.15.39:0/1343822256 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fadc40cf320 con 0x7fadd0103a00 2024-03-18T16:11:12.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.042+0000 7fadd4a00640 1 --2- 172.21.15.39:0/2520381637 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5569660f1420 0x5569660f38b0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fadbc009d60 tx=0x7fadbc0117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9aab148ec0fb8888 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:12.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.043+0000 7fadc2600640 1 -- 172.21.15.39:0/2520381637 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadbc012650 con 0x5569660f1420 2024-03-18T16:11:12.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.043+0000 7fadc2600640 1 -- 172.21.15.39:0/2520381637 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fadbc009e50 con 0x5569660f1420 2024-03-18T16:11:12.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.043+0000 7fadd5e00640 1 --2- 172.21.15.39:0/1343822256 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fadb80a2880 0x7fadb80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:12.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.043+0000 7fadc2600640 1 -- 172.21.15.39:0/2520381637 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadbc00fcd0 con 0x5569660f1420 2024-03-18T16:11:12.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.044+0000 7faddab88740 1 -- 172.21.15.39:0/2520381637 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5569660f1420 msgr2=0x5569660f38b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:12.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.044+0000 7faddab88740 1 --2- 172.21.15.39:0/2520381637 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5569660f1420 0x5569660f38b0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fadbc009d60 tx=0x7fadbc0117b0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.044+0000 7fadd5e00640 1 --2- 172.21.15.39:0/1343822256 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fadb80a2880 0x7fadb80a4d40 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fadc80324c0 tx=0x7fadc802c100 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:12.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.044+0000 7faddab88740 1 -- 172.21.15.39:0/2520381637 shutdown_connections 2024-03-18T16:11:12.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.044+0000 7faddab88740 1 --2- 172.21.15.39:0/2520381637 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5569660ec2b0 0x5569660f0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.044+0000 7faddab88740 1 --2- 172.21.15.39:0/2520381637 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5569660f1420 0x5569660f38b0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.044+0000 7faddab88740 1 --2- 172.21.15.39:0/2520381637 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5569660eb8c0 0x5569660ebce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.044+0000 7faddab88740 1 -- 172.21.15.39:0/2520381637 >> 172.21.15.39:0/2520381637 conn(0x5569660e6e30 msgr2=0x5569660e7230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:12.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.044+0000 7faddab88740 1 -- 172.21.15.39:0/2520381637 shutdown_connections 2024-03-18T16:11:12.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.044+0000 7faddab88740 1 -- 172.21.15.39:0/2520381637 wait complete. 2024-03-18T16:11:12.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.045+0000 7faddab88740 1 -- start start 2024-03-18T16:11:12.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.045+0000 7faddab88740 1 Processor -- start 2024-03-18T16:11:12.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.046+0000 7faddab88740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5569660eb8c0 0x5569660f5be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:12.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.046+0000 7faddab88740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5569660ec2b0 0x5569660f6120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:12.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.046+0000 7faddab88740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5569660f1420 0x5569660fd160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:12.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.046+0000 7faddab88740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55696600c110 con 0x5569660f1420 2024-03-18T16:11:12.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.046+0000 7faddab88740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55696604c4f0 con 0x5569660eb8c0 2024-03-18T16:11:12.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.046+0000 7faddab88740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x5569660f43d0 con 0x5569660ec2b0 2024-03-18T16:11:12.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.046+0000 7faddab88740 10 client.0 _ll_register_callbacks cb 0x556965ff9b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:11:12.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.046+0000 7fadd4a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5569660ec2b0 0x5569660f6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:12.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.046+0000 7fadd4a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5569660ec2b0 0x5569660f6120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:39882/0 (socket says 172.21.15.39:39882) 2024-03-18T16:11:12.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.046+0000 7fadd4a00640 1 -- 172.21.15.39:0/1268288870 learned_addr learned my addr 172.21.15.39:0/1268288870 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:11:12.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.046+0000 7fadd5400640 1 --2- 172.21.15.39:0/1268288870 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5569660eb8c0 0x5569660f5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:12.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.046+0000 7fadd5e00640 1 --2- 172.21.15.39:0/1268288870 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5569660f1420 0x5569660fd160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:12.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.046+0000 7fadd4a00640 1 -- 172.21.15.39:0/1268288870 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5569660eb8c0 msgr2=0x5569660f5be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:12.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.046+0000 7fadd4a00640 1 --2- 172.21.15.39:0/1268288870 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5569660eb8c0 0x5569660f5be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.046+0000 7fadd4a00640 1 -- 172.21.15.39:0/1268288870 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5569660f1420 msgr2=0x5569660fd160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:12.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.046+0000 7fadd4a00640 1 --2- 172.21.15.39:0/1268288870 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5569660f1420 0x5569660fd160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.046+0000 7fadd4a00640 1 -- 172.21.15.39:0/1268288870 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556965d643f0 con 0x5569660ec2b0 2024-03-18T16:11:12.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.047+0000 7fadd4a00640 1 --2- 172.21.15.39:0/1268288870 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5569660ec2b0 0x5569660f6120 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fadbc011cb0 tx=0x7fadbc00fc80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=28798854e3e59463 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:12.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.047+0000 7fadb6a00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:11:12.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.047+0000 7fadb6a00640 1 -- 172.21.15.39:0/1268288870 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadbc028030 con 0x5569660ec2b0 2024-03-18T16:11:12.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.047+0000 7fadb6a00640 1 -- 172.21.15.39:0/1268288870 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fadbc005040 con 0x5569660ec2b0 2024-03-18T16:11:12.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.047+0000 7faddab88740 1 -- 172.21.15.39:0/1268288870 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fadbc009e50 con 0x5569660ec2b0 2024-03-18T16:11:12.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.048+0000 7fadb6a00640 1 -- 172.21.15.39:0/1268288870 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fadbc032050 con 0x5569660ec2b0 2024-03-18T16:11:12.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.048+0000 7faddab88740 10 client.25819 fetch_fsmap learned FSMap version 28 2024-03-18T16:11:12.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.048+0000 7faddab88740 1 -- 172.21.15.39:0/1268288870 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556965e2e7d0 con 0x5569660ec2b0 2024-03-18T16:11:12.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.048+0000 7fadb6a00640 1 -- 172.21.15.39:0/1268288870 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fadbc009e50 con 0x5569660ec2b0 2024-03-18T16:11:12.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.048+0000 7fadb6a00640 1 -- 172.21.15.39:0/1268288870 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fadbc012570 con 0x5569660ec2b0 2024-03-18T16:11:12.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.048+0000 7faddab88740 10 client.25819 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T16:11:12.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.048+0000 7faddab88740 10 client.25819 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-18T16:11:12.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.048+0000 7faddab88740 20 client.25819 populate_metadata read hostname 'smithi039' 2024-03-18T16:11:12.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.048+0000 7faddab88740 1 --2- 172.21.15.39:0/1268288870 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x5569661016b0 0x556966103aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:12.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.048+0000 7faddab88740 4 client.25819 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:11:12.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.048+0000 7faddab88740 1 -- 172.21.15.39:0/1268288870 --> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55696605d990 con 0x5569661016b0 2024-03-18T16:11:12.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.049+0000 7fadd5400640 1 --2- 172.21.15.39:0/1268288870 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x5569661016b0 0x556966103aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:12.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.049+0000 7fadb6a00640 10 client.25819 ms_handle_connect on v2:172.21.15.194:6836/1301464246 2024-03-18T16:11:12.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.049+0000 7fadd5400640 1 --2- 172.21.15.39:0/1268288870 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x5569661016b0 0x556966103aa0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=d181d78a00d75348 server_cookie=f36895ede0f7ec9c in_seq=0 out_seq=0 2024-03-18T16:11:12.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.050+0000 7fadb6a00640 1 -- 172.21.15.39:0/1268288870 <== mds.3 v2:172.21.15.194:6836/1301464246 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55696605d990 con 0x5569661016b0 2024-03-18T16:11:12.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.050+0000 7fadb6a00640 10 client.25819 handle_command_reply: tid=0 2024-03-18T16:11:12.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.050+0000 7faddab88740 1 client.25819 shutdown 2024-03-18T16:11:12.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.050+0000 7faddab88740 1 -- 172.21.15.39:0/1268288870 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5569660ec2b0 msgr2=0x5569660f6120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:12.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.050+0000 7faddab88740 1 --2- 172.21.15.39:0/1268288870 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5569660ec2b0 0x5569660f6120 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fadbc011cb0 tx=0x7fadbc00fc80 comp rx=0 tx=0).stop 2024-03-18T16:11:12.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.050+0000 7faddab88740 1 -- 172.21.15.39:0/1268288870 shutdown_connections 2024-03-18T16:11:12.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.050+0000 7faddab88740 1 --2- 172.21.15.39:0/1268288870 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x5569661016b0 0x556966103aa0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.050+0000 7faddab88740 1 --2- 172.21.15.39:0/1268288870 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5569660f1420 0x5569660fd160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.050+0000 7faddab88740 1 --2- 172.21.15.39:0/1268288870 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5569660ec2b0 0x5569660f6120 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.050+0000 7faddab88740 1 --2- 172.21.15.39:0/1268288870 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5569660eb8c0 0x5569660f5be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.050+0000 7fadb6a00640 0 client.25819 ms_handle_reset on v2:172.21.15.194:6836/1301464246 2024-03-18T16:11:12.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.050+0000 7faddab88740 1 -- 172.21.15.39:0/1268288870 >> 172.21.15.39:0/1268288870 conn(0x5569660e6e30 msgr2=0x5569660e61f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:12.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.051+0000 7faddab88740 1 -- 172.21.15.39:0/1268288870 shutdown_connections 2024-03-18T16:11:12.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.051+0000 7faddab88740 1 -- 172.21.15.39:0/1268288870 wait complete. 2024-03-18T16:11:12.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.051+0000 7faddab88740 20 client.25819 trim_cache size 0 max 16384 2024-03-18T16:11:12.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.072+0000 7faddab88740 1 Processor -- start 2024-03-18T16:11:12.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.072+0000 7faddab88740 1 -- start start 2024-03-18T16:11:12.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.073+0000 7faddab88740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5569660e6e30 0x5569660e6070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:12.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.073+0000 7faddab88740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5569660eb8c0 0x5569660fd120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:12.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.073+0000 7faddab88740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5569660ec2b0 0x556966108310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:12.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.073+0000 7faddab88740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fadd01ac6e0 con 0x5569660eb8c0 2024-03-18T16:11:12.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.073+0000 7faddab88740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fadd01ae220 con 0x5569660e6e30 2024-03-18T16:11:12.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.073+0000 7faddab88740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fadd01acdb0 con 0x5569660ec2b0 2024-03-18T16:11:12.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.073+0000 7fadd5e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5569660ec2b0 0x556966108310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:12.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.073+0000 7fadd5e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5569660ec2b0 0x556966108310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:39896/0 (socket says 172.21.15.39:39896) 2024-03-18T16:11:12.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.073+0000 7fadd5e00640 1 -- 172.21.15.39:0/4168407645 learned_addr learned my addr 172.21.15.39:0/4168407645 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:11:12.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.073+0000 7fadd5400640 1 --2- 172.21.15.39:0/4168407645 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5569660e6e30 0x5569660e6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:12.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.073+0000 7fadd4a00640 1 --2- 172.21.15.39:0/4168407645 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5569660eb8c0 0x5569660fd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:12.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.073+0000 7fadd5e00640 1 -- 172.21.15.39:0/4168407645 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5569660e6e30 msgr2=0x5569660e6070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:12.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.073+0000 7fadd5e00640 1 --2- 172.21.15.39:0/4168407645 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5569660e6e30 0x5569660e6070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.073+0000 7fadd5e00640 1 -- 172.21.15.39:0/4168407645 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5569660eb8c0 msgr2=0x5569660fd120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:12.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.073+0000 7fadd5e00640 1 --2- 172.21.15.39:0/4168407645 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5569660eb8c0 0x5569660fd120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.073+0000 7fadd5e00640 1 -- 172.21.15.39:0/4168407645 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5569660e2c10 con 0x5569660ec2b0 2024-03-18T16:11:12.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.074+0000 7fadd5400640 1 --2- 172.21.15.39:0/4168407645 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5569660e6e30 0x5569660e6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:11:12.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.074+0000 7fadd5e00640 1 --2- 172.21.15.39:0/4168407645 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5569660ec2b0 0x556966108310 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fadc8037040 tx=0x7fadc8005ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a77f45768b5870c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:12.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.074+0000 7fadc1c00640 1 -- 172.21.15.39:0/4168407645 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadc8053070 con 0x5569660ec2b0 2024-03-18T16:11:12.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.074+0000 7fadc1c00640 1 -- 172.21.15.39:0/4168407645 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fadc803a040 con 0x5569660ec2b0 2024-03-18T16:11:12.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.074+0000 7fadc1c00640 1 -- 172.21.15.39:0/4168407645 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadc8039040 con 0x5569660ec2b0 2024-03-18T16:11:12.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.075+0000 7faddab88740 1 -- 172.21.15.39:0/4168407645 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5569660ec2b0 msgr2=0x556966108310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:12.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.075+0000 7faddab88740 1 --2- 172.21.15.39:0/4168407645 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5569660ec2b0 0x556966108310 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fadc8037040 tx=0x7fadc8005ef0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.075+0000 7faddab88740 1 -- 172.21.15.39:0/4168407645 shutdown_connections 2024-03-18T16:11:12.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.075+0000 7faddab88740 1 --2- 172.21.15.39:0/4168407645 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5569660ec2b0 0x556966108310 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.075+0000 7faddab88740 1 --2- 172.21.15.39:0/4168407645 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5569660eb8c0 0x5569660fd120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.075+0000 7faddab88740 1 --2- 172.21.15.39:0/4168407645 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5569660e6e30 0x5569660e6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.075+0000 7faddab88740 1 -- 172.21.15.39:0/4168407645 >> 172.21.15.39:0/4168407645 conn(0x5569660e2460 msgr2=0x5569660e5960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:12.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.076+0000 7faddab88740 1 -- 172.21.15.39:0/4168407645 shutdown_connections 2024-03-18T16:11:12.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.076+0000 7faddab88740 1 -- 172.21.15.39:0/4168407645 wait complete. 2024-03-18T16:11:12.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.076+0000 7faddab88740 1 -- start start 2024-03-18T16:11:12.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.077+0000 7faddab88740 1 Processor -- start 2024-03-18T16:11:12.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.077+0000 7faddab88740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5569660e6e30 0x5569660faee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:12.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.077+0000 7faddab88740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5569660eb8c0 0x5569660fb420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:12.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.077+0000 7faddab88740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5569660ec2b0 0x5569660fc430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:12.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.077+0000 7faddab88740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fadd01adb50 con 0x5569660e6e30 2024-03-18T16:11:12.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.077+0000 7faddab88740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fadd01ad480 con 0x5569660eb8c0 2024-03-18T16:11:12.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.077+0000 7faddab88740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fadd01aa6c0 con 0x5569660ec2b0 2024-03-18T16:11:12.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.077+0000 7faddab88740 10 client.0 _ll_register_callbacks cb 0x7fadc8037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:11:12.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.077+0000 7fadd4a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5569660eb8c0 0x5569660fb420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:12.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.077+0000 7fadd5e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5569660ec2b0 0x5569660fc430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:12.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.077+0000 7fadd4a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5569660eb8c0 0x5569660fb420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:54566/0 (socket says 172.21.15.39:54566) 2024-03-18T16:11:12.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.077+0000 7fadd4a00640 1 -- 172.21.15.39:0/3426572389 learned_addr learned my addr 172.21.15.39:0/3426572389 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:11:12.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.078+0000 7fadd5400640 1 --2- 172.21.15.39:0/3426572389 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5569660e6e30 0x5569660faee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:12.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.078+0000 7fadd4a00640 1 -- 172.21.15.39:0/3426572389 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5569660ec2b0 msgr2=0x5569660fc430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:12.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.078+0000 7fadd4a00640 1 --2- 172.21.15.39:0/3426572389 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5569660ec2b0 0x5569660fc430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.078+0000 7fadd4a00640 1 -- 172.21.15.39:0/3426572389 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5569660e6e30 msgr2=0x5569660faee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:12.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.078+0000 7fadd4a00640 1 --2- 172.21.15.39:0/3426572389 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5569660e6e30 0x5569660faee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.078+0000 7fadd4a00640 1 -- 172.21.15.39:0/3426572389 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5569660e7ba0 con 0x5569660eb8c0 2024-03-18T16:11:12.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.078+0000 7fadd5e00640 1 --2- 172.21.15.39:0/3426572389 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5569660ec2b0 0x5569660fc430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:11:12.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.078+0000 7fadd4a00640 1 --2- 172.21.15.39:0/3426572389 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5569660eb8c0 0x5569660fb420 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fadbc0129c0 tx=0x7fadbc009a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78457de753859f08 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:12.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.078+0000 7fadb6a00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:11:12.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.078+0000 7fadb6a00640 1 -- 172.21.15.39:0/3426572389 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadbc028040 con 0x5569660eb8c0 2024-03-18T16:11:12.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.079+0000 7fadb6a00640 1 -- 172.21.15.39:0/3426572389 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fadbc005450 con 0x5569660eb8c0 2024-03-18T16:11:12.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.079+0000 7faddab88740 1 -- 172.21.15.39:0/3426572389 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fadc803a040 con 0x5569660eb8c0 2024-03-18T16:11:12.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.079+0000 7fadb6a00640 1 -- 172.21.15.39:0/3426572389 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fadbc012db0 con 0x5569660eb8c0 2024-03-18T16:11:12.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.079+0000 7faddab88740 10 client.25793 fetch_fsmap learned FSMap version 28 2024-03-18T16:11:12.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.079+0000 7faddab88740 1 -- 172.21.15.39:0/3426572389 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556966167e20 con 0x5569660eb8c0 2024-03-18T16:11:12.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.079+0000 7fadb6a00640 1 -- 172.21.15.39:0/3426572389 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fadc803a040 con 0x5569660eb8c0 2024-03-18T16:11:12.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.079+0000 7fadb6a00640 1 -- 172.21.15.39:0/3426572389 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fadbc004bd0 con 0x5569660eb8c0 2024-03-18T16:11:12.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.079+0000 7faddab88740 10 client.25793 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T16:11:12.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.079+0000 7faddab88740 10 client.25793 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-18T16:11:12.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.079+0000 7faddab88740 20 client.25793 populate_metadata read hostname 'smithi039' 2024-03-18T16:11:12.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.079+0000 7faddab88740 1 --2- 172.21.15.39:0/3426572389 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x5569660f1420 0x55696616a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:12.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.079+0000 7faddab88740 4 client.25793 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:11:12.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.079+0000 7faddab88740 1 -- 172.21.15.39:0/3426572389 --> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556966057ce0 con 0x5569660f1420 2024-03-18T16:11:12.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.080+0000 7fadd5400640 1 --2- 172.21.15.39:0/3426572389 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x5569660f1420 0x55696616a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:12.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.080+0000 7fadd5400640 1 --2- 172.21.15.39:0/3426572389 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x5569660f1420 0x55696616a830 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=9e39cd0950a38f17 server_cookie=1162ee16a66ea97d in_seq=0 out_seq=0 2024-03-18T16:11:12.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.080+0000 7fadb6a00640 10 client.25793 ms_handle_connect on v2:172.21.15.194:6836/1301464246 2024-03-18T16:11:12.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.081+0000 7fadb6a00640 1 -- 172.21.15.39:0/3426572389 <== mds.3 v2:172.21.15.194:6836/1301464246 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556966057ce0 con 0x5569660f1420 2024-03-18T16:11:12.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.081+0000 7fadb6a00640 10 client.25793 handle_command_reply: tid=0 2024-03-18T16:11:12.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.081+0000 7faddab88740 1 client.25793 shutdown 2024-03-18T16:11:12.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.081+0000 7faddab88740 1 -- 172.21.15.39:0/3426572389 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5569660eb8c0 msgr2=0x5569660fb420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:12.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.081+0000 7faddab88740 1 --2- 172.21.15.39:0/3426572389 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5569660eb8c0 0x5569660fb420 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fadbc0129c0 tx=0x7fadbc009a40 comp rx=0 tx=0).stop 2024-03-18T16:11:12.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.081+0000 7faddab88740 1 -- 172.21.15.39:0/3426572389 shutdown_connections 2024-03-18T16:11:12.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.081+0000 7faddab88740 1 --2- 172.21.15.39:0/3426572389 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x5569660f1420 0x55696616a830 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.081+0000 7faddab88740 1 --2- 172.21.15.39:0/3426572389 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5569660eb8c0 0x5569660fb420 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.081+0000 7faddab88740 1 --2- 172.21.15.39:0/3426572389 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5569660ec2b0 0x5569660fc430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.081+0000 7faddab88740 1 --2- 172.21.15.39:0/3426572389 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5569660e6e30 0x5569660faee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.081+0000 7faddab88740 1 -- 172.21.15.39:0/3426572389 >> 172.21.15.39:0/3426572389 conn(0x5569660e2460 msgr2=0x5569660e61f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:12.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.081+0000 7fadb6a00640 0 client.25793 ms_handle_reset on v2:172.21.15.194:6836/1301464246 2024-03-18T16:11:12.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.081+0000 7faddab88740 1 -- 172.21.15.39:0/3426572389 shutdown_connections 2024-03-18T16:11:12.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.081+0000 7faddab88740 1 -- 172.21.15.39:0/3426572389 wait complete. 2024-03-18T16:11:12.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.081+0000 7faddab88740 20 client.25793 trim_cache size 0 max 16384 2024-03-18T16:11:12.088 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:11:12.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.082+0000 7fadd7e00640 1 -- 172.21.15.39:0/1343822256 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fadb80a2880 msgr2=0x7fadb80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:12.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.082+0000 7fadd7e00640 1 --2- 172.21.15.39:0/1343822256 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fadb80a2880 0x7fadb80a4d40 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fadc80324c0 tx=0x7fadc802c100 comp rx=0 tx=0).stop 2024-03-18T16:11:12.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.082+0000 7fadd7e00640 1 -- 172.21.15.39:0/1343822256 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fadd0103a00 msgr2=0x7fadd0199c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:12.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.082+0000 7fadd7e00640 1 --2- 172.21.15.39:0/1343822256 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fadd0103a00 0x7fadd0199c70 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fadc400e9c0 tx=0x7fadc400ee80 comp rx=0 tx=0).stop 2024-03-18T16:11:12.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.082+0000 7fadd7e00640 1 -- 172.21.15.39:0/1343822256 shutdown_connections 2024-03-18T16:11:12.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.082+0000 7fadd7e00640 1 --2- 172.21.15.39:0/1343822256 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fadd01055b0 0x7fadd01a11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.082+0000 7fadd7e00640 1 --2- 172.21.15.39:0/1343822256 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fadb80a2880 0x7fadb80a4d40 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.082+0000 7fadd7e00640 1 --2- 172.21.15.39:0/1343822256 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fadd0104bf0 0x7fadd019a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.082+0000 7fadd7e00640 1 --2- 172.21.15.39:0/1343822256 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fadd0103a00 0x7fadd0199c70 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.082+0000 7fadd7e00640 1 -- 172.21.15.39:0/1343822256 >> 172.21.15.39:0/1343822256 conn(0x7fadd00fef90 msgr2=0x7fadd0107e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:12.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.082+0000 7fadd7e00640 1 -- 172.21.15.39:0/1343822256 shutdown_connections 2024-03-18T16:11:12.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.082+0000 7fadd7e00640 1 -- 172.21.15.39:0/1343822256 wait complete. 2024-03-18T16:11:12.090 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:11:11 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/3815623223' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:11:12.102 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-18T16:11:12.128 INFO:tasks.workunit.client.0.smithi039.stdout:checking for strerror... yes 2024-03-18T16:11:12.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:11:11 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/3815623223' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:11:12.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:11:11 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/3815623223' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:11:12.261 INFO:tasks.workunit.client.0.smithi039.stdout:checking for open64... yes 2024-03-18T16:11:12.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.327+0000 7f9a93a00640 1 -- 172.21.15.39:0/3055639666 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9a8c102a50 msgr2=0x7f9a8c102ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:12.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.327+0000 7f9a93a00640 1 --2- 172.21.15.39:0/3055639666 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9a8c102a50 0x7f9a8c102ed0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f9a78009900 tx=0x7f9a7802bc30 comp rx=0 tx=0).stop 2024-03-18T16:11:12.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.327+0000 7f9a93a00640 1 -- 172.21.15.39:0/3055639666 shutdown_connections 2024-03-18T16:11:12.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.327+0000 7f9a93a00640 1 --2- 172.21.15.39:0/3055639666 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9a8c102a50 0x7f9a8c102ed0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.327+0000 7f9a93a00640 1 --2- 172.21.15.39:0/3055639666 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9a8c103410 0x7f9a8c109ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.327+0000 7f9a93a00640 1 --2- 172.21.15.39:0/3055639666 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9a8c101860 0x7f9a8c101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.327+0000 7f9a93a00640 1 -- 172.21.15.39:0/3055639666 >> 172.21.15.39:0/3055639666 conn(0x7f9a8c0fcdb0 msgr2=0x7f9a8c0ff210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:12.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.327+0000 7f9a93a00640 1 -- 172.21.15.39:0/3055639666 shutdown_connections 2024-03-18T16:11:12.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.327+0000 7f9a93a00640 1 -- 172.21.15.39:0/3055639666 wait complete. 2024-03-18T16:11:12.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.328+0000 7f9a93a00640 1 Processor -- start 2024-03-18T16:11:12.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.328+0000 7f9a93a00640 1 -- start start 2024-03-18T16:11:12.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.328+0000 7f9a93a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9a8c101860 0x7f9a8c199cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:12.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.328+0000 7f9a93a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9a8c102a50 0x7f9a8c19a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:12.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.328+0000 7f9a93a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9a8c103410 0x7f9a8c1a1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:12.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.328+0000 7f9a93a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f9a8c10d450 con 0x7f9a8c102a50 2024-03-18T16:11:12.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.328+0000 7f9a93a00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f9a8c10d2d0 con 0x7f9a8c101860 2024-03-18T16:11:12.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.328+0000 7f9a93a00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f9a8c10d5d0 con 0x7f9a8c103410 2024-03-18T16:11:12.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.329+0000 7f9a91000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9a8c101860 0x7f9a8c199cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:12.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.329+0000 7f9a91000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9a8c101860 0x7f9a8c199cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:54592/0 (socket says 172.21.15.39:54592) 2024-03-18T16:11:12.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.329+0000 7f9a8be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9a8c102a50 0x7f9a8c19a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:12.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.329+0000 7f9a91a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9a8c103410 0x7f9a8c1a1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:12.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.329+0000 7f9a91000640 1 -- 172.21.15.39:0/12399331 learned_addr learned my addr 172.21.15.39:0/12399331 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:11:12.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.329+0000 7f9a91000640 1 -- 172.21.15.39:0/12399331 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9a8c103410 msgr2=0x7f9a8c1a1250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:12.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.329+0000 7f9a91000640 1 --2- 172.21.15.39:0/12399331 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9a8c103410 0x7f9a8c1a1250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.329+0000 7f9a91000640 1 -- 172.21.15.39:0/12399331 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9a8c102a50 msgr2=0x7f9a8c19a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:12.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.329+0000 7f9a91000640 1 --2- 172.21.15.39:0/12399331 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9a8c102a50 0x7f9a8c19a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.329+0000 7f9a91000640 1 -- 172.21.15.39:0/12399331 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a8c1a18f0 con 0x7f9a8c101860 2024-03-18T16:11:12.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.329+0000 7f9a91000640 1 --2- 172.21.15.39:0/12399331 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9a8c101860 0x7f9a8c199cb0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f9a8000e9c0 tx=0x7f9a8000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:12.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.330+0000 7f9a89600640 1 -- 172.21.15.39:0/12399331 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a8000cd80 con 0x7f9a8c101860 2024-03-18T16:11:12.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.330+0000 7f9a89600640 1 -- 172.21.15.39:0/12399331 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a800035f0 con 0x7f9a8c101860 2024-03-18T16:11:12.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.330+0000 7f9a93a00640 1 -- 172.21.15.39:0/12399331 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a8c1a1be0 con 0x7f9a8c101860 2024-03-18T16:11:12.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.330+0000 7f9a89600640 1 -- 172.21.15.39:0/12399331 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a80010690 con 0x7f9a8c101860 2024-03-18T16:11:12.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.330+0000 7f9a93a00640 1 -- 172.21.15.39:0/12399331 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a8c1a2150 con 0x7f9a8c101860 2024-03-18T16:11:12.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.333+0000 7f9a89600640 1 -- 172.21.15.39:0/12399331 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f9a800037a0 con 0x7f9a8c101860 2024-03-18T16:11:12.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.333+0000 7f9a96837740 1 Processor -- start 2024-03-18T16:11:12.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.333+0000 7f9a96837740 1 -- start start 2024-03-18T16:11:12.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.334+0000 7f9a96837740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e5700368c0 0x55e570036ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:12.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.334+0000 7f9a96837740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e5700372b0 0x55e57003bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:12.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.334+0000 7f9a96837740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e57003c420 0x55e57003e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:12.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.334+0000 7f9a96837740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55e56fffe3b0 con 0x55e5700372b0 2024-03-18T16:11:12.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.334+0000 7f9a96837740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55e56ff9b450 con 0x55e57003c420 2024-03-18T16:11:12.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.334+0000 7f9a96837740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55e56ff43cb0 con 0x55e5700368c0 2024-03-18T16:11:12.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.334+0000 7f9a91a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e5700368c0 0x55e570036ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:12.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.334+0000 7f9a91a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e5700368c0 0x55e570036ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:39932/0 (socket says 172.21.15.39:39932) 2024-03-18T16:11:12.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.334+0000 7f9a91a00640 1 -- 172.21.15.39:0/839988578 learned_addr learned my addr 172.21.15.39:0/839988578 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:11:12.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.334+0000 7f9a8be00640 1 --2- 172.21.15.39:0/839988578 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e57003c420 0x55e57003e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:12.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.334+0000 7f9a91000640 1 --2- 172.21.15.39:0/839988578 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e5700372b0 0x55e57003bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:12.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.334+0000 7f9a91a00640 1 -- 172.21.15.39:0/839988578 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e57003c420 msgr2=0x55e57003e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:12.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.334+0000 7f9a91a00640 1 --2- 172.21.15.39:0/839988578 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e57003c420 0x55e57003e8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.334+0000 7f9a91a00640 1 -- 172.21.15.39:0/839988578 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e5700372b0 msgr2=0x55e57003bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:12.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.334+0000 7f9a91a00640 1 --2- 172.21.15.39:0/839988578 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e5700372b0 0x55e57003bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.334+0000 7f9a91a00640 1 -- 172.21.15.39:0/839988578 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e56ff651f0 con 0x55e5700368c0 2024-03-18T16:11:12.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.335+0000 7f9a91a00640 1 --2- 172.21.15.39:0/839988578 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e5700368c0 0x55e570036ce0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f9a84009fd0 tx=0x7f9a840118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=23222a43a3816959 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:12.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.335+0000 7f9a7e000640 1 -- 172.21.15.39:0/839988578 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a84004d20 con 0x55e5700368c0 2024-03-18T16:11:12.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.335+0000 7f9a7e000640 1 -- 172.21.15.39:0/839988578 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a84004050 con 0x55e5700368c0 2024-03-18T16:11:12.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.335+0000 7f9a7e000640 1 -- 172.21.15.39:0/839988578 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a84005260 con 0x55e5700368c0 2024-03-18T16:11:12.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.335+0000 7f9a89600640 1 --2- 172.21.15.39:0/12399331 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f9a740a2880 0x7f9a740a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:12.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.335+0000 7f9a89600640 1 -- 172.21.15.39:0/12399331 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f9a800cf160 con 0x7f9a8c101860 2024-03-18T16:11:12.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.336+0000 7f9a91000640 1 --2- 172.21.15.39:0/12399331 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f9a740a2880 0x7f9a740a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:12.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.336+0000 7f9a96837740 1 -- 172.21.15.39:0/839988578 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e5700368c0 msgr2=0x55e570036ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:12.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.336+0000 7f9a96837740 1 --2- 172.21.15.39:0/839988578 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e5700368c0 0x55e570036ce0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f9a84009fd0 tx=0x7f9a840118d0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.336+0000 7f9a96837740 1 -- 172.21.15.39:0/839988578 shutdown_connections 2024-03-18T16:11:12.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.336+0000 7f9a96837740 1 --2- 172.21.15.39:0/839988578 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e57003c420 0x55e57003e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.336+0000 7f9a96837740 1 --2- 172.21.15.39:0/839988578 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e5700372b0 0x55e57003bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.336+0000 7f9a96837740 1 --2- 172.21.15.39:0/839988578 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e5700368c0 0x55e570036ce0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.336+0000 7f9a96837740 1 -- 172.21.15.39:0/839988578 >> 172.21.15.39:0/839988578 conn(0x55e570031e30 msgr2=0x55e570032230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:12.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.336+0000 7f9a96837740 1 -- 172.21.15.39:0/839988578 shutdown_connections 2024-03-18T16:11:12.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.336+0000 7f9a96837740 1 -- 172.21.15.39:0/839988578 wait complete. 2024-03-18T16:11:12.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.336+0000 7f9a91000640 1 --2- 172.21.15.39:0/12399331 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f9a740a2880 0x7f9a740a4d40 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f9a80097b70 tx=0x7f9a80000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:12.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.337+0000 7f9a96837740 1 -- start start 2024-03-18T16:11:12.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.337+0000 7f9a96837740 1 Processor -- start 2024-03-18T16:11:12.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.338+0000 7f9a96837740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e5700368c0 0x55e570040b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:12.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.338+0000 7f9a96837740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e5700372b0 0x55e5700410d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:12.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.338+0000 7f9a96837740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e57003c420 0x55e570048110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:12.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.338+0000 7f9a96837740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55e56ff57110 con 0x55e5700368c0 2024-03-18T16:11:12.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.338+0000 7f9a96837740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55e56ff974f0 con 0x55e57003c420 2024-03-18T16:11:12.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.338+0000 7f9a96837740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55e57003f3d0 con 0x55e5700372b0 2024-03-18T16:11:12.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.338+0000 7f9a96837740 10 client.0 _ll_register_callbacks cb 0x55e56ff44b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:11:12.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.338+0000 7f9a91a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e5700368c0 0x55e570040b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:12.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.338+0000 7f9a8be00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e5700372b0 0x55e5700410d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:12.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.338+0000 7f9a91a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e5700368c0 0x55e570040b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:34736/0 (socket says 172.21.15.39:34736) 2024-03-18T16:11:12.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.338+0000 7f9a91a00640 1 -- 172.21.15.39:0/3454500512 learned_addr learned my addr 172.21.15.39:0/3454500512 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:11:12.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.338+0000 7f9a91a00640 1 --2- 172.21.15.39:0/3454500512 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e57003c420 0x55e570048110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:12.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.338+0000 7f9a8be00640 1 -- 172.21.15.39:0/3454500512 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e57003c420 msgr2=0x55e570048110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:12.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.338+0000 7f9a8be00640 1 --2- 172.21.15.39:0/3454500512 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e57003c420 0x55e570048110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.338+0000 7f9a8be00640 1 -- 172.21.15.39:0/3454500512 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e5700368c0 msgr2=0x55e570040b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:12.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.338+0000 7f9a8be00640 1 --2- 172.21.15.39:0/3454500512 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e5700368c0 0x55e570040b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.338+0000 7f9a8be00640 1 -- 172.21.15.39:0/3454500512 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e56fcaf3f0 con 0x55e5700372b0 2024-03-18T16:11:12.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.339+0000 7f9a8be00640 1 --2- 172.21.15.39:0/3454500512 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e5700372b0 0x55e5700410d0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f9a78009230 tx=0x7f9a78004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=65eca7eb10577bb9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:12.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.339+0000 7f9a72a00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:11:12.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.339+0000 7f9a72a00640 1 -- 172.21.15.39:0/3454500512 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a78005d50 con 0x55e5700372b0 2024-03-18T16:11:12.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.339+0000 7f9a96837740 1 -- 172.21.15.39:0/3454500512 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9a84004050 con 0x55e5700372b0 2024-03-18T16:11:12.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.339+0000 7f9a72a00640 1 -- 172.21.15.39:0/3454500512 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f9a78005230 con 0x55e5700372b0 2024-03-18T16:11:12.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.339+0000 7f9a72a00640 1 -- 172.21.15.39:0/3454500512 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9a78005430 con 0x55e5700372b0 2024-03-18T16:11:12.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.339+0000 7f9a96837740 10 client.25837 fetch_fsmap learned FSMap version 28 2024-03-18T16:11:12.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.339+0000 7f9a96837740 1 -- 172.21.15.39:0/3454500512 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e56fd797d0 con 0x55e5700372b0 2024-03-18T16:11:12.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.340+0000 7f9a72a00640 1 -- 172.21.15.39:0/3454500512 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a84004050 con 0x55e5700372b0 2024-03-18T16:11:12.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.340+0000 7f9a72a00640 1 -- 172.21.15.39:0/3454500512 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f9a780039e0 con 0x55e5700372b0 2024-03-18T16:11:12.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.340+0000 7f9a96837740 10 client.25837 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T16:11:12.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.340+0000 7f9a96837740 10 client.25837 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-18T16:11:12.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.340+0000 7f9a96837740 20 client.25837 populate_metadata read hostname 'smithi039' 2024-03-18T16:11:12.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.340+0000 7f9a96837740 1 --2- 172.21.15.39:0/3454500512 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x55e57004c9e0 0x55e57004ee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:12.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.340+0000 7f9a96837740 4 client.25837 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:11:12.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.340+0000 7f9a96837740 1 -- 172.21.15.39:0/3454500512 --> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e56ffa8990 con 0x55e57004c9e0 2024-03-18T16:11:12.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.341+0000 7f9a91a00640 1 --2- 172.21.15.39:0/3454500512 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x55e57004c9e0 0x55e57004ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:12.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.341+0000 7f9a91a00640 1 --2- 172.21.15.39:0/3454500512 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x55e57004c9e0 0x55e57004ee10 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8dfbb5655897bd06 server_cookie=c5f3e454065a57d2 in_seq=0 out_seq=0 2024-03-18T16:11:12.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.341+0000 7f9a72a00640 10 client.25837 ms_handle_connect on v2:172.21.15.196:6836/895247251 2024-03-18T16:11:12.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.342+0000 7f9a72a00640 1 -- 172.21.15.39:0/3454500512 <== mds.2 v2:172.21.15.196:6836/895247251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e56ffa8990 con 0x55e57004c9e0 2024-03-18T16:11:12.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.342+0000 7f9a72a00640 10 client.25837 handle_command_reply: tid=0 2024-03-18T16:11:12.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.342+0000 7f9a96837740 1 client.25837 shutdown 2024-03-18T16:11:12.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.342+0000 7f9a96837740 1 -- 172.21.15.39:0/3454500512 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e5700372b0 msgr2=0x55e5700410d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:12.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.342+0000 7f9a96837740 1 --2- 172.21.15.39:0/3454500512 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e5700372b0 0x55e5700410d0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f9a78009230 tx=0x7f9a78004270 comp rx=0 tx=0).stop 2024-03-18T16:11:12.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.342+0000 7f9a96837740 1 -- 172.21.15.39:0/3454500512 shutdown_connections 2024-03-18T16:11:12.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.342+0000 7f9a96837740 1 --2- 172.21.15.39:0/3454500512 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e57003c420 0x55e570048110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.342+0000 7f9a96837740 1 --2- 172.21.15.39:0/3454500512 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x55e57004c9e0 0x55e57004ee10 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.342+0000 7f9a96837740 1 --2- 172.21.15.39:0/3454500512 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e5700372b0 0x55e5700410d0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.342+0000 7f9a96837740 1 --2- 172.21.15.39:0/3454500512 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e5700368c0 0x55e570040b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.342+0000 7f9a96837740 1 -- 172.21.15.39:0/3454500512 >> 172.21.15.39:0/3454500512 conn(0x55e570031e30 msgr2=0x55e5700311f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:12.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.342+0000 7f9a72a00640 0 client.25837 ms_handle_reset on v2:172.21.15.196:6836/895247251 2024-03-18T16:11:12.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.343+0000 7f9a96837740 1 -- 172.21.15.39:0/3454500512 shutdown_connections 2024-03-18T16:11:12.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.343+0000 7f9a96837740 1 -- 172.21.15.39:0/3454500512 wait complete. 2024-03-18T16:11:12.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.343+0000 7f9a96837740 20 client.25837 trim_cache size 0 max 16384 2024-03-18T16:11:12.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.359+0000 7f9a96837740 1 Processor -- start 2024-03-18T16:11:12.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.360+0000 7f9a96837740 1 -- start start 2024-03-18T16:11:12.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.360+0000 7f9a96837740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e5700368c0 0x55e570038670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:12.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.360+0000 7f9a96837740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e5700372b0 0x55e5700480d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:12.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.360+0000 7f9a96837740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e57003c420 0x55e570038bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:12.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.360+0000 7f9a96837740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f9a8c1ac660 con 0x55e57003c420 2024-03-18T16:11:12.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.360+0000 7f9a96837740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f9a8c1ae1a0 con 0x55e5700368c0 2024-03-18T16:11:12.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.360+0000 7f9a96837740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f9a8c1acd30 con 0x55e5700372b0 2024-03-18T16:11:12.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.360+0000 7f9a8be00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e5700372b0 0x55e5700480d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:12.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.360+0000 7f9a91a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e5700368c0 0x55e570038670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:12.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.360+0000 7f9a8be00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e5700372b0 0x55e5700480d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:39946/0 (socket says 172.21.15.39:39946) 2024-03-18T16:11:12.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.360+0000 7f9a8be00640 1 -- 172.21.15.39:0/270263568 learned_addr learned my addr 172.21.15.39:0/270263568 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:11:12.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.360+0000 7f9a91a00640 1 --2- 172.21.15.39:0/270263568 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e57003c420 0x55e570038bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:12.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.361+0000 7f9a8be00640 1 -- 172.21.15.39:0/270263568 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e5700368c0 msgr2=0x55e570038670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:12.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.361+0000 7f9a8be00640 1 --2- 172.21.15.39:0/270263568 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e5700368c0 0x55e570038670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.361+0000 7f9a8be00640 1 -- 172.21.15.39:0/270263568 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e57003c420 msgr2=0x55e570038bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:12.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.361+0000 7f9a8be00640 1 --2- 172.21.15.39:0/270263568 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e57003c420 0x55e570038bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.361+0000 7f9a8be00640 1 -- 172.21.15.39:0/270263568 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e570041b80 con 0x55e5700372b0 2024-03-18T16:11:12.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.361+0000 7f9a91a00640 1 --2- 172.21.15.39:0/270263568 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e57003c420 0x55e570038bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:11:12.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.361+0000 7f9a8be00640 1 --2- 172.21.15.39:0/270263568 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e5700372b0 0x55e5700480d0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f9a78009850 tx=0x7f9a7802c5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a7c5aa91256e3a32 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:12.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.361+0000 7f9a7d600640 1 -- 172.21.15.39:0/270263568 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a78058070 con 0x55e5700372b0 2024-03-18T16:11:12.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.361+0000 7f9a7d600640 1 -- 172.21.15.39:0/270263568 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a78009520 con 0x55e5700372b0 2024-03-18T16:11:12.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.361+0000 7f9a7d600640 1 -- 172.21.15.39:0/270263568 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a780035f0 con 0x55e5700372b0 2024-03-18T16:11:12.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.362+0000 7f9a96837740 1 -- 172.21.15.39:0/270263568 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e5700372b0 msgr2=0x55e5700480d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:12.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.362+0000 7f9a96837740 1 --2- 172.21.15.39:0/270263568 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e5700372b0 0x55e5700480d0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f9a78009850 tx=0x7f9a7802c5d0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.362+0000 7f9a96837740 1 -- 172.21.15.39:0/270263568 shutdown_connections 2024-03-18T16:11:12.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.362+0000 7f9a96837740 1 --2- 172.21.15.39:0/270263568 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e57003c420 0x55e570038bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.362+0000 7f9a96837740 1 --2- 172.21.15.39:0/270263568 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e5700372b0 0x55e5700480d0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.362+0000 7f9a96837740 1 --2- 172.21.15.39:0/270263568 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e5700368c0 0x55e570038670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.362+0000 7f9a96837740 1 -- 172.21.15.39:0/270263568 >> 172.21.15.39:0/270263568 conn(0x55e570031e30 msgr2=0x55e57002d080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:12.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.363+0000 7f9a96837740 1 -- 172.21.15.39:0/270263568 shutdown_connections 2024-03-18T16:11:12.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.363+0000 7f9a96837740 1 -- 172.21.15.39:0/270263568 wait complete. 2024-03-18T16:11:12.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.363+0000 7f9a96837740 1 -- start start 2024-03-18T16:11:12.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.363+0000 7f9a96837740 1 Processor -- start 2024-03-18T16:11:12.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.364+0000 7f9a96837740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e5700368c0 0x55e5700455c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:12.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.364+0000 7f9a96837740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e5700372b0 0x55e570045b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:12.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.364+0000 7f9a96837740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e57003c420 0x55e570046ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:12.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.364+0000 7f9a96837740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f9a8c1adad0 con 0x55e5700372b0 2024-03-18T16:11:12.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.364+0000 7f9a96837740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f9a8c1ad400 con 0x55e57003c420 2024-03-18T16:11:12.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.364+0000 7f9a96837740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f9a8c1aa640 con 0x55e5700368c0 2024-03-18T16:11:12.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.364+0000 7f9a96837740 10 client.0 _ll_register_callbacks cb 0x7f9a78009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:11:12.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.364+0000 7f9a91a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e5700368c0 0x55e5700455c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:12.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.364+0000 7f9a91a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e5700368c0 0x55e5700455c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:39948/0 (socket says 172.21.15.39:39948) 2024-03-18T16:11:12.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.364+0000 7f9a91a00640 1 -- 172.21.15.39:0/1332758319 learned_addr learned my addr 172.21.15.39:0/1332758319 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:11:12.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.364+0000 7f9a91a00640 1 --2- 172.21.15.39:0/1332758319 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e57003c420 0x55e570046ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:12.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.364+0000 7f9a8be00640 1 --2- 172.21.15.39:0/1332758319 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e5700372b0 0x55e570045b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:12.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.365+0000 7f9a91a00640 1 -- 172.21.15.39:0/1332758319 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e57003c420 msgr2=0x55e570046ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:12.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.365+0000 7f9a91a00640 1 --2- 172.21.15.39:0/1332758319 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e57003c420 0x55e570046ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.365+0000 7f9a91a00640 1 -- 172.21.15.39:0/1332758319 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e5700372b0 msgr2=0x55e570045b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:12.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.365+0000 7f9a91a00640 1 --2- 172.21.15.39:0/1332758319 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e5700372b0 0x55e570045b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.365+0000 7f9a91a00640 1 -- 172.21.15.39:0/1332758319 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e5700470b0 con 0x55e5700368c0 2024-03-18T16:11:12.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.365+0000 7f9a91a00640 1 --2- 172.21.15.39:0/1332758319 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e5700368c0 0x55e5700455c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f9a84000f80 tx=0x7f9a840118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dd9b979ead93a4c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:12.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.365+0000 7f9a72a00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:11:12.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.365+0000 7f9a72a00640 1 -- 172.21.15.39:0/1332758319 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a8402d690 con 0x55e5700368c0 2024-03-18T16:11:12.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.365+0000 7f9a72a00640 1 -- 172.21.15.39:0/1332758319 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f9a84003ec0 con 0x55e5700368c0 2024-03-18T16:11:12.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.365+0000 7f9a96837740 1 -- 172.21.15.39:0/1332758319 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9a78009520 con 0x55e5700368c0 2024-03-18T16:11:12.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.366+0000 7f9a72a00640 1 -- 172.21.15.39:0/1332758319 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9a84041050 con 0x55e5700368c0 2024-03-18T16:11:12.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.366+0000 7f9a96837740 10 client.25843 fetch_fsmap learned FSMap version 28 2024-03-18T16:11:12.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.366+0000 7f9a96837740 1 -- 172.21.15.39:0/1332758319 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55e570048420 con 0x55e5700368c0 2024-03-18T16:11:12.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.366+0000 7f9a72a00640 1 -- 172.21.15.39:0/1332758319 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9a78009520 con 0x55e5700368c0 2024-03-18T16:11:12.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.366+0000 7f9a72a00640 1 -- 172.21.15.39:0/1332758319 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f9a84025040 con 0x55e5700368c0 2024-03-18T16:11:12.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.366+0000 7f9a96837740 10 client.25843 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T16:11:12.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.366+0000 7f9a96837740 10 client.25843 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-18T16:11:12.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.366+0000 7f9a96837740 20 client.25843 populate_metadata read hostname 'smithi039' 2024-03-18T16:11:12.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.366+0000 7f9a96837740 1 --2- 172.21.15.39:0/1332758319 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x55e5700b3340 0x55e5700b5760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:12.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.366+0000 7f9a96837740 4 client.25843 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:11:12.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.366+0000 7f9a96837740 1 -- 172.21.15.39:0/1332758319 --> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e56ffa2ce0 con 0x55e5700b3340 2024-03-18T16:11:12.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.367+0000 7f9a8be00640 1 --2- 172.21.15.39:0/1332758319 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x55e5700b3340 0x55e5700b5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:12.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.367+0000 7f9a72a00640 10 client.25843 ms_handle_connect on v2:172.21.15.196:6836/895247251 2024-03-18T16:11:12.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.367+0000 7f9a8be00640 1 --2- 172.21.15.39:0/1332758319 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x55e5700b3340 0x55e5700b5760 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b463624900088a53 server_cookie=da5a54add774f4c3 in_seq=0 out_seq=0 2024-03-18T16:11:12.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.367+0000 7f9a72a00640 1 -- 172.21.15.39:0/1332758319 <== mds.2 v2:172.21.15.196:6836/895247251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e56ffa2ce0 con 0x55e5700b3340 2024-03-18T16:11:12.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.367+0000 7f9a72a00640 10 client.25843 handle_command_reply: tid=0 2024-03-18T16:11:12.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.367+0000 7f9a96837740 1 client.25843 shutdown 2024-03-18T16:11:12.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.368+0000 7f9a96837740 1 -- 172.21.15.39:0/1332758319 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e5700368c0 msgr2=0x55e5700455c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:12.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.368+0000 7f9a96837740 1 --2- 172.21.15.39:0/1332758319 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e5700368c0 0x55e5700455c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f9a84000f80 tx=0x7f9a840118d0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.368+0000 7f9a96837740 1 -- 172.21.15.39:0/1332758319 shutdown_connections 2024-03-18T16:11:12.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.368+0000 7f9a96837740 1 --2- 172.21.15.39:0/1332758319 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e57003c420 0x55e570046ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.368+0000 7f9a96837740 1 --2- 172.21.15.39:0/1332758319 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x55e5700b3340 0x55e5700b5760 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.368+0000 7f9a96837740 1 --2- 172.21.15.39:0/1332758319 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e5700372b0 0x55e570045b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.368+0000 7f9a96837740 1 --2- 172.21.15.39:0/1332758319 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e5700368c0 0x55e5700455c0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.368+0000 7f9a96837740 1 -- 172.21.15.39:0/1332758319 >> 172.21.15.39:0/1332758319 conn(0x55e570031e30 msgr2=0x55e570040b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:12.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.368+0000 7f9a72a00640 0 client.25843 ms_handle_reset on v2:172.21.15.196:6836/895247251 2024-03-18T16:11:12.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.368+0000 7f9a96837740 1 -- 172.21.15.39:0/1332758319 shutdown_connections 2024-03-18T16:11:12.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.368+0000 7f9a96837740 1 -- 172.21.15.39:0/1332758319 wait complete. 2024-03-18T16:11:12.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.368+0000 7f9a96837740 20 client.25843 trim_cache size 0 max 16384 2024-03-18T16:11:12.375 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:11:12.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.369+0000 7f9a93a00640 1 -- 172.21.15.39:0/12399331 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f9a740a2880 msgr2=0x7f9a740a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:12.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.369+0000 7f9a93a00640 1 --2- 172.21.15.39:0/12399331 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f9a740a2880 0x7f9a740a4d40 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f9a80097b70 tx=0x7f9a80000bc0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.369+0000 7f9a93a00640 1 -- 172.21.15.39:0/12399331 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9a8c101860 msgr2=0x7f9a8c199cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:12.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.369+0000 7f9a93a00640 1 --2- 172.21.15.39:0/12399331 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9a8c101860 0x7f9a8c199cb0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f9a8000e9c0 tx=0x7f9a8000ee80 comp rx=0 tx=0).stop 2024-03-18T16:11:12.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.369+0000 7f9a93a00640 1 -- 172.21.15.39:0/12399331 shutdown_connections 2024-03-18T16:11:12.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.369+0000 7f9a93a00640 1 --2- 172.21.15.39:0/12399331 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f9a740a2880 0x7f9a740a4d40 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.369+0000 7f9a93a00640 1 --2- 172.21.15.39:0/12399331 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9a8c103410 0x7f9a8c1a1250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.369+0000 7f9a93a00640 1 --2- 172.21.15.39:0/12399331 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9a8c102a50 0x7f9a8c19a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.369+0000 7f9a93a00640 1 --2- 172.21.15.39:0/12399331 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9a8c101860 0x7f9a8c199cb0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.369+0000 7f9a93a00640 1 -- 172.21.15.39:0/12399331 >> 172.21.15.39:0/12399331 conn(0x7f9a8c0fcdb0 msgr2=0x7f9a8c073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:12.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.369+0000 7f9a93a00640 1 -- 172.21.15.39:0/12399331 shutdown_connections 2024-03-18T16:11:12.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.369+0000 7f9a93a00640 1 -- 172.21.15.39:0/12399331 wait complete. 2024-03-18T16:11:12.390 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-18T16:11:12.391 INFO:tasks.workunit.client.0.smithi039.stdout:checking for stat64... yes 2024-03-18T16:11:12.527 INFO:tasks.workunit.client.0.smithi039.stdout:checking for fseeko64... yes 2024-03-18T16:11:12.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.610+0000 7f6b50800640 1 -- 172.21.15.39:0/3519993899 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6b48100a10 msgr2=0x7f6b48100e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:12.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.610+0000 7f6b50800640 1 --2- 172.21.15.39:0/3519993899 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6b48100a10 0x7f6b48100e70 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f6b38009900 tx=0x7f6b3802bc40 comp rx=0 tx=0).stop 2024-03-18T16:11:12.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.610+0000 7f6b50800640 1 -- 172.21.15.39:0/3519993899 shutdown_connections 2024-03-18T16:11:12.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.610+0000 7f6b50800640 1 --2- 172.21.15.39:0/3519993899 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6b481013b0 0x7f6b48101830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.610+0000 7f6b50800640 1 --2- 172.21.15.39:0/3519993899 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6b48100a10 0x7f6b48100e70 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.610+0000 7f6b50800640 1 --2- 172.21.15.39:0/3519993899 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6b481000b0 0x7f6b481004d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.610+0000 7f6b50800640 1 -- 172.21.15.39:0/3519993899 >> 172.21.15.39:0/3519993899 conn(0x7f6b480fbc70 msgr2=0x7f6b480fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:12.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.611+0000 7f6b50800640 1 -- 172.21.15.39:0/3519993899 shutdown_connections 2024-03-18T16:11:12.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.611+0000 7f6b50800640 1 -- 172.21.15.39:0/3519993899 wait complete. 2024-03-18T16:11:12.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.612+0000 7f6b50800640 1 Processor -- start 2024-03-18T16:11:12.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.612+0000 7f6b50800640 1 -- start start 2024-03-18T16:11:12.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.612+0000 7f6b50800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6b481000b0 0x7f6b48199cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:12.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.612+0000 7f6b50800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6b48100a10 0x7f6b4819a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:12.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.612+0000 7f6b50800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6b481013b0 0x7f6b481a1260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:12.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.612+0000 7f6b50800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f6b4810f620 con 0x7f6b481013b0 2024-03-18T16:11:12.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.612+0000 7f6b50800640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f6b4810f4a0 con 0x7f6b48100a10 2024-03-18T16:11:12.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.612+0000 7f6b50800640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f6b4810f7a0 con 0x7f6b481000b0 2024-03-18T16:11:12.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.613+0000 7f6b4d400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6b48100a10 0x7f6b4819a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:12.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.613+0000 7f6b4de00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6b481000b0 0x7f6b48199cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:12.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.613+0000 7f6b4d400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6b48100a10 0x7f6b4819a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:54648/0 (socket says 172.21.15.39:54648) 2024-03-18T16:11:12.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.613+0000 7f6b4d400640 1 -- 172.21.15.39:0/3707577542 learned_addr learned my addr 172.21.15.39:0/3707577542 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:11:12.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.613+0000 7f6b4e800640 1 --2- 172.21.15.39:0/3707577542 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6b481013b0 0x7f6b481a1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:12.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.613+0000 7f6b4d400640 1 -- 172.21.15.39:0/3707577542 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6b481000b0 msgr2=0x7f6b48199cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:12.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.613+0000 7f6b4d400640 1 --2- 172.21.15.39:0/3707577542 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6b481000b0 0x7f6b48199cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.613+0000 7f6b4d400640 1 -- 172.21.15.39:0/3707577542 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6b481013b0 msgr2=0x7f6b481a1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:12.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.613+0000 7f6b4d400640 1 --2- 172.21.15.39:0/3707577542 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6b481013b0 0x7f6b481a1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.613+0000 7f6b4d400640 1 -- 172.21.15.39:0/3707577542 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b481a1900 con 0x7f6b48100a10 2024-03-18T16:11:12.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.613+0000 7f6b4e800640 1 --2- 172.21.15.39:0/3707577542 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6b481013b0 0x7f6b481a1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:11:12.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.613+0000 7f6b4de00640 1 --2- 172.21.15.39:0/3707577542 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6b481000b0 0x7f6b48199cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:11:12.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.613+0000 7f6b4d400640 1 --2- 172.21.15.39:0/3707577542 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6b48100a10 0x7f6b4819a230 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f6b38009230 tx=0x7f6b3802c660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:12.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.614+0000 7f6b42a00640 1 -- 172.21.15.39:0/3707577542 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b38041070 con 0x7f6b48100a10 2024-03-18T16:11:12.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.614+0000 7f6b42a00640 1 -- 172.21.15.39:0/3707577542 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b38004450 con 0x7f6b48100a10 2024-03-18T16:11:12.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.614+0000 7f6b42a00640 1 -- 172.21.15.39:0/3707577542 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b38003e00 con 0x7f6b48100a10 2024-03-18T16:11:12.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.614+0000 7f6b50800640 1 -- 172.21.15.39:0/3707577542 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b481a1b90 con 0x7f6b48100a10 2024-03-18T16:11:12.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.614+0000 7f6b50800640 1 -- 172.21.15.39:0/3707577542 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b481a2030 con 0x7f6b48100a10 2024-03-18T16:11:12.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.617+0000 7f6b42a00640 1 -- 172.21.15.39:0/3707577542 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f6b38044070 con 0x7f6b48100a10 2024-03-18T16:11:12.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.618+0000 7f6b42a00640 1 --2- 172.21.15.39:0/3707577542 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f6b300a2880 0x7f6b300a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:12.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.619+0000 7f6b42a00640 1 -- 172.21.15.39:0/3707577542 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f6b380f0ad0 con 0x7f6b48100a10 2024-03-18T16:11:12.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.619+0000 7f6b535ee740 1 Processor -- start 2024-03-18T16:11:12.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.619+0000 7f6b4de00640 1 --2- 172.21.15.39:0/3707577542 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f6b300a2880 0x7f6b300a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:12.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.619+0000 7f6b535ee740 1 -- start start 2024-03-18T16:11:12.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.619+0000 7f6b535ee740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5642724ad8c0 0x5642724adce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:12.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.619+0000 7f6b535ee740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5642724ae2b0 0x5642724b2db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:12.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.619+0000 7f6b535ee740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5642724b3420 0x5642724b58b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:12.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.619+0000 7f6b535ee740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5642724753b0 con 0x5642724ae2b0 2024-03-18T16:11:12.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.619+0000 7f6b535ee740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x564272412450 con 0x5642724ad8c0 2024-03-18T16:11:12.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.619+0000 7f6b535ee740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x5642723bacb0 con 0x5642724b3420 2024-03-18T16:11:12.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.620+0000 7f6b4d400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5642724ae2b0 0x5642724b2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:12.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.620+0000 7f6b4e800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5642724b3420 0x5642724b58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:12.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.620+0000 7f6b4de00640 1 --2- 172.21.15.39:0/3707577542 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f6b300a2880 0x7f6b300a4d40 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f6b4819b1c0 tx=0x7f6b3c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:12.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.620+0000 7f6b4de00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5642724ad8c0 0x5642724adce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:12.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.620+0000 7f6b4e800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5642724b3420 0x5642724b58b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:39960/0 (socket says 172.21.15.39:39960) 2024-03-18T16:11:12.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.620+0000 7f6b4e800640 1 -- 172.21.15.39:0/882842822 learned_addr learned my addr 172.21.15.39:0/882842822 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:11:12.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.620+0000 7f6b4e800640 1 -- 172.21.15.39:0/882842822 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5642724ad8c0 msgr2=0x5642724adce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:12.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.620+0000 7f6b4e800640 1 --2- 172.21.15.39:0/882842822 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5642724ad8c0 0x5642724adce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.620+0000 7f6b4e800640 1 -- 172.21.15.39:0/882842822 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5642724ae2b0 msgr2=0x5642724b2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:12.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.620+0000 7f6b4e800640 1 --2- 172.21.15.39:0/882842822 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5642724ae2b0 0x5642724b2db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.620+0000 7f6b4e800640 1 -- 172.21.15.39:0/882842822 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5642723dc1f0 con 0x5642724b3420 2024-03-18T16:11:12.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.620+0000 7f6b4e800640 1 --2- 172.21.15.39:0/882842822 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5642724b3420 0x5642724b58b0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f6b44009fd0 tx=0x7f6b440108e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=88bd0ba6c39de96e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:12.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.621+0000 7f6b37000640 1 -- 172.21.15.39:0/882842822 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b44011760 con 0x5642724b3420 2024-03-18T16:11:12.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.621+0000 7f6b37000640 1 -- 172.21.15.39:0/882842822 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b44011de0 con 0x5642724b3420 2024-03-18T16:11:12.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.621+0000 7f6b37000640 1 -- 172.21.15.39:0/882842822 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b4400e3c0 con 0x5642724b3420 2024-03-18T16:11:12.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.622+0000 7f6b535ee740 1 -- 172.21.15.39:0/882842822 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5642724b3420 msgr2=0x5642724b58b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:12.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.622+0000 7f6b535ee740 1 --2- 172.21.15.39:0/882842822 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5642724b3420 0x5642724b58b0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f6b44009fd0 tx=0x7f6b440108e0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.622+0000 7f6b535ee740 1 -- 172.21.15.39:0/882842822 shutdown_connections 2024-03-18T16:11:12.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.622+0000 7f6b535ee740 1 --2- 172.21.15.39:0/882842822 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5642724b3420 0x5642724b58b0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.622+0000 7f6b535ee740 1 --2- 172.21.15.39:0/882842822 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5642724ae2b0 0x5642724b2db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.622+0000 7f6b535ee740 1 --2- 172.21.15.39:0/882842822 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5642724ad8c0 0x5642724adce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.622+0000 7f6b535ee740 1 -- 172.21.15.39:0/882842822 >> 172.21.15.39:0/882842822 conn(0x5642724a8e30 msgr2=0x5642724a9230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:12.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.622+0000 7f6b535ee740 1 -- 172.21.15.39:0/882842822 shutdown_connections 2024-03-18T16:11:12.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.622+0000 7f6b535ee740 1 -- 172.21.15.39:0/882842822 wait complete. 2024-03-18T16:11:12.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.622+0000 7f6b535ee740 1 -- start start 2024-03-18T16:11:12.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.623+0000 7f6b535ee740 1 Processor -- start 2024-03-18T16:11:12.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.623+0000 7f6b535ee740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5642724ad8c0 0x5642724b7b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:12.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.623+0000 7f6b535ee740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5642724ae2b0 0x5642724b80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:12.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.623+0000 7f6b535ee740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5642724b3420 0x5642724bf0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:12.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.623+0000 7f6b535ee740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5642723ce110 con 0x5642724ad8c0 2024-03-18T16:11:12.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.624+0000 7f6b535ee740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x56427240e4f0 con 0x5642724b3420 2024-03-18T16:11:12.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.624+0000 7f6b535ee740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x5642724b63d0 con 0x5642724ae2b0 2024-03-18T16:11:12.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.624+0000 7f6b4d400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5642724ae2b0 0x5642724b80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:12.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.624+0000 7f6b4e800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5642724b3420 0x5642724bf0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:12.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.624+0000 7f6b4d400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5642724ae2b0 0x5642724b80a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:39976/0 (socket says 172.21.15.39:39976) 2024-03-18T16:11:12.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.624+0000 7f6b4de00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5642724ad8c0 0x5642724b7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:12.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.624+0000 7f6b4d400640 1 -- 172.21.15.39:0/3598420490 learned_addr learned my addr 172.21.15.39:0/3598420490 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:11:12.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.624+0000 7f6b535ee740 10 client.0 _ll_register_callbacks cb 0x5642723bbb70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:11:12.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.624+0000 7f6b4d400640 1 -- 172.21.15.39:0/3598420490 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5642724b3420 msgr2=0x5642724bf0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:12.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.624+0000 7f6b4d400640 1 --2- 172.21.15.39:0/3598420490 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5642724b3420 0x5642724bf0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.624+0000 7f6b4d400640 1 -- 172.21.15.39:0/3598420490 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5642724ad8c0 msgr2=0x5642724b7b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:12.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.624+0000 7f6b4d400640 1 --2- 172.21.15.39:0/3598420490 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5642724ad8c0 0x5642724b7b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.624+0000 7f6b4d400640 1 -- 172.21.15.39:0/3598420490 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5642721263f0 con 0x5642724ae2b0 2024-03-18T16:11:12.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.624+0000 7f6b535ee740 1 -- 172.21.15.39:0/3598420490 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6b44011de0 con 0x5642724ae2b0 2024-03-18T16:11:12.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.624+0000 7f6b4d400640 1 --2- 172.21.15.39:0/3598420490 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5642724ae2b0 0x5642724b80a0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f6b38006010 tx=0x7f6b380bc9e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=601619a21582f9a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:12.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.625+0000 7f6b2b400640 10 client.25855 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:11:12.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.625+0000 7f6b2b400640 1 -- 172.21.15.39:0/3598420490 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b3804b040 con 0x5642724ae2b0 2024-03-18T16:11:12.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.625+0000 7f6b2b400640 1 -- 172.21.15.39:0/3598420490 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f6b380c5390 con 0x5642724ae2b0 2024-03-18T16:11:12.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.625+0000 7f6b2b400640 1 -- 172.21.15.39:0/3598420490 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6b380c5590 con 0x5642724ae2b0 2024-03-18T16:11:12.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.625+0000 7f6b535ee740 10 client.25855 fetch_fsmap learned FSMap version 28 2024-03-18T16:11:12.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.625+0000 7f6b535ee740 1 -- 172.21.15.39:0/3598420490 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({fsmap=28}) -- 0x5642721f07d0 con 0x5642724ae2b0 2024-03-18T16:11:12.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.625+0000 7f6b2b400640 1 -- 172.21.15.39:0/3598420490 <== mon.1 v2:172.21.15.196:3300/0 4 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f6b380dbb80 con 0x5642724ae2b0 2024-03-18T16:11:12.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.626+0000 7f6b535ee740 10 client.25855 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T16:11:12.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.626+0000 7f6b535ee740 10 client.25855 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T16:11:12.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.626+0000 7f6b535ee740 20 client.25855 populate_metadata read hostname 'smithi039' 2024-03-18T16:11:12.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.626+0000 7f6b535ee740 1 --2- 172.21.15.39:0/3598420490 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x5642724c37e0 0x5642724c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:12.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.626+0000 7f6b535ee740 4 client.25855 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:11:12.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.626+0000 7f6b535ee740 1 -- 172.21.15.39:0/3598420490 --> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56427241f990 con 0x5642724c37e0 2024-03-18T16:11:12.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.626+0000 7f6b4de00640 1 --2- 172.21.15.39:0/3598420490 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x5642724c37e0 0x5642724c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:12.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.626+0000 7f6b4de00640 1 --2- 172.21.15.39:0/3598420490 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x5642724c37e0 0x5642724c5ca0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1aa855c7588e3f2a server_cookie=4930179bc3fcb71e in_seq=0 out_seq=0 2024-03-18T16:11:12.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.626+0000 7f6b2b400640 10 client.25855 ms_handle_connect on v2:172.21.15.196:6838/2001427268 2024-03-18T16:11:12.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.627+0000 7f6b2b400640 1 -- 172.21.15.39:0/3598420490 <== mds.1 v2:172.21.15.196:6838/2001427268 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56427241f990 con 0x5642724c37e0 2024-03-18T16:11:12.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.627+0000 7f6b2b400640 10 client.25855 handle_command_reply: tid=0 2024-03-18T16:11:12.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.627+0000 7f6b535ee740 1 client.25855 shutdown 2024-03-18T16:11:12.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.627+0000 7f6b535ee740 1 -- 172.21.15.39:0/3598420490 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5642724ae2b0 msgr2=0x5642724b80a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:12.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.627+0000 7f6b535ee740 1 --2- 172.21.15.39:0/3598420490 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5642724ae2b0 0x5642724b80a0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f6b38006010 tx=0x7f6b380bc9e0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.628+0000 7f6b535ee740 1 -- 172.21.15.39:0/3598420490 shutdown_connections 2024-03-18T16:11:12.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.628+0000 7f6b535ee740 1 --2- 172.21.15.39:0/3598420490 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x5642724c37e0 0x5642724c5ca0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.628+0000 7f6b535ee740 1 --2- 172.21.15.39:0/3598420490 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5642724b3420 0x5642724bf0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.628+0000 7f6b535ee740 1 --2- 172.21.15.39:0/3598420490 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5642724ae2b0 0x5642724b80a0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.628+0000 7f6b2b400640 0 client.25855 ms_handle_reset on v2:172.21.15.196:6838/2001427268 2024-03-18T16:11:12.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.628+0000 7f6b535ee740 1 --2- 172.21.15.39:0/3598420490 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5642724ad8c0 0x5642724b7b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.628+0000 7f6b535ee740 1 -- 172.21.15.39:0/3598420490 >> 172.21.15.39:0/3598420490 conn(0x5642724a8e30 msgr2=0x5642724a81f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:12.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.628+0000 7f6b535ee740 1 -- 172.21.15.39:0/3598420490 shutdown_connections 2024-03-18T16:11:12.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.628+0000 7f6b535ee740 1 -- 172.21.15.39:0/3598420490 wait complete. 2024-03-18T16:11:12.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.628+0000 7f6b535ee740 20 client.25855 trim_cache size 0 max 16384 2024-03-18T16:11:12.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.650+0000 7f6b535ee740 1 Processor -- start 2024-03-18T16:11:12.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.650+0000 7f6b535ee740 1 -- start start 2024-03-18T16:11:12.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.650+0000 7f6b535ee740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5642724ad8c0 0x5642724a8070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:12.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.650+0000 7f6b535ee740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5642724ae2b0 0x5642724bf0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:12.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.650+0000 7f6b535ee740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5642724b3420 0x5642724ca5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:12.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.650+0000 7f6b535ee740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f6b481ac650 con 0x5642724ae2b0 2024-03-18T16:11:12.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.650+0000 7f6b535ee740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f6b481ae190 con 0x5642724b3420 2024-03-18T16:11:12.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.650+0000 7f6b535ee740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f6b481acd20 con 0x5642724ad8c0 2024-03-18T16:11:12.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.650+0000 7f6b4de00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5642724ad8c0 0x5642724a8070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:12.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.650+0000 7f6b4e800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5642724b3420 0x5642724ca5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:12.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.651+0000 7f6b4de00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5642724ad8c0 0x5642724a8070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:39990/0 (socket says 172.21.15.39:39990) 2024-03-18T16:11:12.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.651+0000 7f6b4de00640 1 -- 172.21.15.39:0/919124715 learned_addr learned my addr 172.21.15.39:0/919124715 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:11:12.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.651+0000 7f6b4d400640 1 --2- 172.21.15.39:0/919124715 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5642724ae2b0 0x5642724bf0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:12.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.651+0000 7f6b4de00640 1 -- 172.21.15.39:0/919124715 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5642724b3420 msgr2=0x5642724ca5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:12.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.651+0000 7f6b4de00640 1 --2- 172.21.15.39:0/919124715 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5642724b3420 0x5642724ca5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.651+0000 7f6b4de00640 1 -- 172.21.15.39:0/919124715 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5642724ae2b0 msgr2=0x5642724bf0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:12.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.651+0000 7f6b4de00640 1 --2- 172.21.15.39:0/919124715 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5642724ae2b0 0x5642724bf0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.651+0000 7f6b4de00640 1 -- 172.21.15.39:0/919124715 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5642724c6320 con 0x5642724ad8c0 2024-03-18T16:11:12.656 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.651+0000 7f6b4e800640 1 --2- 172.21.15.39:0/919124715 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5642724b3420 0x5642724ca5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:11:12.656 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.651+0000 7f6b4de00640 1 --2- 172.21.15.39:0/919124715 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5642724ad8c0 0x5642724a8070 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f6b3c0095f0 tx=0x7f6b3c014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6b039578448752b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:12.656 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.651+0000 7f6b36600640 1 -- 172.21.15.39:0/919124715 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b3c003e00 con 0x5642724ad8c0 2024-03-18T16:11:12.656 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.652+0000 7f6b36600640 1 -- 172.21.15.39:0/919124715 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b3c005020 con 0x5642724ad8c0 2024-03-18T16:11:12.656 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.652+0000 7f6b36600640 1 -- 172.21.15.39:0/919124715 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b3c0049e0 con 0x5642724ad8c0 2024-03-18T16:11:12.656 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.652+0000 7f6b535ee740 1 -- 172.21.15.39:0/919124715 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5642724ad8c0 msgr2=0x5642724a8070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:12.656 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.652+0000 7f6b535ee740 1 --2- 172.21.15.39:0/919124715 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5642724ad8c0 0x5642724a8070 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f6b3c0095f0 tx=0x7f6b3c014000 comp rx=0 tx=0).stop 2024-03-18T16:11:12.656 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.652+0000 7f6b535ee740 1 -- 172.21.15.39:0/919124715 shutdown_connections 2024-03-18T16:11:12.656 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.652+0000 7f6b535ee740 1 --2- 172.21.15.39:0/919124715 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5642724b3420 0x5642724ca5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.656 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.652+0000 7f6b535ee740 1 --2- 172.21.15.39:0/919124715 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5642724ae2b0 0x5642724bf0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.656 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.652+0000 7f6b535ee740 1 --2- 172.21.15.39:0/919124715 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5642724ad8c0 0x5642724a8070 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.656 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.652+0000 7f6b535ee740 1 -- 172.21.15.39:0/919124715 >> 172.21.15.39:0/919124715 conn(0x5642724a8e30 msgr2=0x5642724a7960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:12.656 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.653+0000 7f6b535ee740 1 -- 172.21.15.39:0/919124715 shutdown_connections 2024-03-18T16:11:12.656 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.653+0000 7f6b535ee740 1 -- 172.21.15.39:0/919124715 wait complete. 2024-03-18T16:11:12.656 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.653+0000 7f6b535ee740 1 -- start start 2024-03-18T16:11:12.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.653+0000 7f6b535ee740 1 Processor -- start 2024-03-18T16:11:12.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.654+0000 7f6b535ee740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5642724ad8c0 0x5642724bcce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:12.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.654+0000 7f6b535ee740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5642724ae2b0 0x5642724bd220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:12.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.654+0000 7f6b535ee740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5642724b3420 0x5642724be230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:12.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.654+0000 7f6b535ee740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f6b481adac0 con 0x5642724b3420 2024-03-18T16:11:12.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.654+0000 7f6b535ee740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f6b481ad3f0 con 0x5642724ad8c0 2024-03-18T16:11:12.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.654+0000 7f6b535ee740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f6b481aa630 con 0x5642724ae2b0 2024-03-18T16:11:12.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.654+0000 7f6b535ee740 10 client.0 _ll_register_callbacks cb 0x7f6b380dd0e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:11:12.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.654+0000 7f6b4d400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5642724ae2b0 0x5642724bd220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:12.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.654+0000 7f6b4de00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5642724ad8c0 0x5642724bcce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:12.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.654+0000 7f6b4d400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5642724ae2b0 0x5642724bd220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:39996/0 (socket says 172.21.15.39:39996) 2024-03-18T16:11:12.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.654+0000 7f6b4d400640 1 -- 172.21.15.39:0/2465691611 learned_addr learned my addr 172.21.15.39:0/2465691611 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:11:12.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.654+0000 7f6b4e800640 1 --2- 172.21.15.39:0/2465691611 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5642724b3420 0x5642724be230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:12.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.654+0000 7f6b4d400640 1 -- 172.21.15.39:0/2465691611 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5642724ad8c0 msgr2=0x5642724bcce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:12.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.654+0000 7f6b4d400640 1 --2- 172.21.15.39:0/2465691611 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5642724ad8c0 0x5642724bcce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.654+0000 7f6b4d400640 1 -- 172.21.15.39:0/2465691611 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5642724b3420 msgr2=0x5642724be230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:12.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.654+0000 7f6b4d400640 1 --2- 172.21.15.39:0/2465691611 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5642724b3420 0x5642724be230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.654+0000 7f6b4d400640 1 -- 172.21.15.39:0/2465691611 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5642724a4c10 con 0x5642724ae2b0 2024-03-18T16:11:12.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.654+0000 7f6b4de00640 1 --2- 172.21.15.39:0/2465691611 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5642724ad8c0 0x5642724bcce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:11:12.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.654+0000 7f6b4e800640 1 --2- 172.21.15.39:0/2465691611 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5642724b3420 0x5642724be230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:11:12.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.655+0000 7f6b4d400640 1 --2- 172.21.15.39:0/2465691611 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5642724ae2b0 0x5642724bd220 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f6b3802c7c0 tx=0x7f6b38009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=977e4228e65c81ca server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:12.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.655+0000 7f6b2b400640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:11:12.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.655+0000 7f6b2b400640 1 -- 172.21.15.39:0/2465691611 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b3804b070 con 0x5642724ae2b0 2024-03-18T16:11:12.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.655+0000 7f6b2b400640 1 -- 172.21.15.39:0/2465691611 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f6b380c5390 con 0x5642724ae2b0 2024-03-18T16:11:12.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.655+0000 7f6b535ee740 1 -- 172.21.15.39:0/2465691611 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6b3c005020 con 0x5642724ae2b0 2024-03-18T16:11:12.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.655+0000 7f6b2b400640 1 -- 172.21.15.39:0/2465691611 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6b380a0060 con 0x5642724ae2b0 2024-03-18T16:11:12.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.655+0000 7f6b535ee740 10 client.25861 fetch_fsmap learned FSMap version 28 2024-03-18T16:11:12.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.655+0000 7f6b535ee740 1 -- 172.21.15.39:0/2465691611 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x564272529fd0 con 0x5642724ae2b0 2024-03-18T16:11:12.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.656+0000 7f6b2b400640 1 -- 172.21.15.39:0/2465691611 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b3c005020 con 0x5642724ae2b0 2024-03-18T16:11:12.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.656+0000 7f6b2b400640 1 -- 172.21.15.39:0/2465691611 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f6b380a6120 con 0x5642724ae2b0 2024-03-18T16:11:12.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.656+0000 7f6b535ee740 10 client.25861 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T16:11:12.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.656+0000 7f6b535ee740 10 client.25861 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T16:11:12.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.656+0000 7f6b535ee740 20 client.25861 populate_metadata read hostname 'smithi039' 2024-03-18T16:11:12.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.656+0000 7f6b535ee740 1 --2- 172.21.15.39:0/2465691611 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x56427252a790 0x56427252cb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:12.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.656+0000 7f6b535ee740 4 client.25861 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:11:12.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.656+0000 7f6b535ee740 1 -- 172.21.15.39:0/2465691611 --> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564272419ce0 con 0x56427252a790 2024-03-18T16:11:12.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.656+0000 7f6b4de00640 1 --2- 172.21.15.39:0/2465691611 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x56427252a790 0x56427252cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:12.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.657+0000 7f6b4de00640 1 --2- 172.21.15.39:0/2465691611 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x56427252a790 0x56427252cb80 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=740a033aaa8d9dae server_cookie=c2856dc801c26bd5 in_seq=0 out_seq=0 2024-03-18T16:11:12.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.657+0000 7f6b2b400640 10 client.25861 ms_handle_connect on v2:172.21.15.196:6838/2001427268 2024-03-18T16:11:12.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.657+0000 7f6b2b400640 1 -- 172.21.15.39:0/2465691611 <== mds.1 v2:172.21.15.196:6838/2001427268 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564272419ce0 con 0x56427252a790 2024-03-18T16:11:12.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.657+0000 7f6b2b400640 10 client.25861 handle_command_reply: tid=0 2024-03-18T16:11:12.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.657+0000 7f6b535ee740 1 client.25861 shutdown 2024-03-18T16:11:12.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.657+0000 7f6b535ee740 1 -- 172.21.15.39:0/2465691611 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5642724ae2b0 msgr2=0x5642724bd220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:12.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.657+0000 7f6b535ee740 1 --2- 172.21.15.39:0/2465691611 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5642724ae2b0 0x5642724bd220 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f6b3802c7c0 tx=0x7f6b38009260 comp rx=0 tx=0).stop 2024-03-18T16:11:12.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.657+0000 7f6b535ee740 1 -- 172.21.15.39:0/2465691611 shutdown_connections 2024-03-18T16:11:12.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.657+0000 7f6b535ee740 1 --2- 172.21.15.39:0/2465691611 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x56427252a790 0x56427252cb80 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.657+0000 7f6b535ee740 1 --2- 172.21.15.39:0/2465691611 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5642724b3420 0x5642724be230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.657+0000 7f6b535ee740 1 --2- 172.21.15.39:0/2465691611 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5642724ae2b0 0x5642724bd220 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.657+0000 7f6b535ee740 1 --2- 172.21.15.39:0/2465691611 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5642724ad8c0 0x5642724bcce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.657+0000 7f6b2b400640 0 client.25861 ms_handle_reset on v2:172.21.15.196:6838/2001427268 2024-03-18T16:11:12.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.657+0000 7f6b535ee740 1 -- 172.21.15.39:0/2465691611 >> 172.21.15.39:0/2465691611 conn(0x5642724a8e30 msgr2=0x5642724ac920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:12.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.657+0000 7f6b535ee740 1 -- 172.21.15.39:0/2465691611 shutdown_connections 2024-03-18T16:11:12.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.657+0000 7f6b535ee740 1 -- 172.21.15.39:0/2465691611 wait complete. 2024-03-18T16:11:12.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.658+0000 7f6b535ee740 20 client.25861 trim_cache size 0 max 16384 2024-03-18T16:11:12.664 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:11:12.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.658+0000 7f6b50800640 1 -- 172.21.15.39:0/3707577542 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f6b300a2880 msgr2=0x7f6b300a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:12.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.658+0000 7f6b50800640 1 --2- 172.21.15.39:0/3707577542 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f6b300a2880 0x7f6b300a4d40 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f6b4819b1c0 tx=0x7f6b3c005f90 comp rx=0 tx=0).stop 2024-03-18T16:11:12.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.658+0000 7f6b50800640 1 -- 172.21.15.39:0/3707577542 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6b48100a10 msgr2=0x7f6b4819a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:12.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.658+0000 7f6b50800640 1 --2- 172.21.15.39:0/3707577542 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6b48100a10 0x7f6b4819a230 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f6b38009230 tx=0x7f6b3802c660 comp rx=0 tx=0).stop 2024-03-18T16:11:12.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.658+0000 7f6b50800640 1 -- 172.21.15.39:0/3707577542 shutdown_connections 2024-03-18T16:11:12.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.658+0000 7f6b50800640 1 --2- 172.21.15.39:0/3707577542 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6b48100a10 0x7f6b4819a230 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.658+0000 7f6b50800640 1 --2- 172.21.15.39:0/3707577542 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f6b300a2880 0x7f6b300a4d40 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.658+0000 7f6b50800640 1 --2- 172.21.15.39:0/3707577542 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6b481013b0 0x7f6b481a1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.658+0000 7f6b50800640 1 --2- 172.21.15.39:0/3707577542 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6b481000b0 0x7f6b48199cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.658+0000 7f6b50800640 1 -- 172.21.15.39:0/3707577542 >> 172.21.15.39:0/3707577542 conn(0x7f6b480fbc70 msgr2=0x7f6b480fd370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:12.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.659+0000 7f6b50800640 1 -- 172.21.15.39:0/3707577542 shutdown_connections 2024-03-18T16:11:12.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.659+0000 7f6b50800640 1 -- 172.21.15.39:0/3707577542 wait complete. 2024-03-18T16:11:12.666 INFO:tasks.workunit.client.0.smithi039.stdout:checking for lrand48_r... yes 2024-03-18T16:11:12.680 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-18T16:11:12.788 INFO:tasks.workunit.client.0.smithi039.stdout:checking for srand48_r... yes 2024-03-18T16:11:12.837 INFO:tasks.workunit.client.0.smithi039.stdout:configure: creating ./config.status 2024-03-18T16:11:12.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.889+0000 7f6d01200640 1 -- 172.21.15.39:0/2857830173 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6cfc1031d0 msgr2=0x7f6cfc109a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:12.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.889+0000 7f6d01200640 1 --2- 172.21.15.39:0/2857830173 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6cfc1031d0 0x7f6cfc109a60 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f6cf4009ca0 tx=0x7f6cf402c100 comp rx=0 tx=0).stop 2024-03-18T16:11:12.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.889+0000 7f6d01200640 1 -- 172.21.15.39:0/2857830173 shutdown_connections 2024-03-18T16:11:12.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.889+0000 7f6d01200640 1 --2- 172.21.15.39:0/2857830173 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6cfc102810 0x7f6cfc102c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.889+0000 7f6d01200640 1 --2- 172.21.15.39:0/2857830173 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6cfc1031d0 0x7f6cfc109a60 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.889+0000 7f6d01200640 1 --2- 172.21.15.39:0/2857830173 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6cfc101620 0x7f6cfc101a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.889+0000 7f6d01200640 1 -- 172.21.15.39:0/2857830173 >> 172.21.15.39:0/2857830173 conn(0x7f6cfc0fcb90 msgr2=0x7f6cfc0fefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:12.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.889+0000 7f6d01200640 1 -- 172.21.15.39:0/2857830173 shutdown_connections 2024-03-18T16:11:12.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.889+0000 7f6d01200640 1 -- 172.21.15.39:0/2857830173 wait complete. 2024-03-18T16:11:12.894 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.890+0000 7f6d01200640 1 Processor -- start 2024-03-18T16:11:12.894 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.890+0000 7f6d01200640 1 -- start start 2024-03-18T16:11:12.894 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.891+0000 7f6d01200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6cfc101620 0x7f6cfc1978d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:12.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.891+0000 7f6d01200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6cfc102810 0x7f6cfc197e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:12.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.891+0000 7f6d01200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6cfc1031d0 0x7f6cfc19ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:12.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.891+0000 7f6d01200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f6cfc10d2b0 con 0x7f6cfc1031d0 2024-03-18T16:11:12.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.891+0000 7f6d01200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f6cfc10d130 con 0x7f6cfc101620 2024-03-18T16:11:12.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.891+0000 7f6d01200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f6cfc10d430 con 0x7f6cfc102810 2024-03-18T16:11:12.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.891+0000 7f6cfb400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6cfc101620 0x7f6cfc1978d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:12.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.891+0000 7f6cfb400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6cfc101620 0x7f6cfc1978d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:54718/0 (socket says 172.21.15.39:54718) 2024-03-18T16:11:12.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.891+0000 7f6cfb400640 1 -- 172.21.15.39:0/2822379174 learned_addr learned my addr 172.21.15.39:0/2822379174 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:11:12.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.891+0000 7f6cfbe00640 1 --2- 172.21.15.39:0/2822379174 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6cfc1031d0 0x7f6cfc19ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:12.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.891+0000 7f6cfaa00640 1 --2- 172.21.15.39:0/2822379174 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6cfc102810 0x7f6cfc197e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:12.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.891+0000 7f6cfb400640 1 -- 172.21.15.39:0/2822379174 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6cfc102810 msgr2=0x7f6cfc197e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:12.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.891+0000 7f6cfb400640 1 --2- 172.21.15.39:0/2822379174 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6cfc102810 0x7f6cfc197e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.891+0000 7f6cfb400640 1 -- 172.21.15.39:0/2822379174 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6cfc1031d0 msgr2=0x7f6cfc19ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:12.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.891+0000 7f6cfb400640 1 --2- 172.21.15.39:0/2822379174 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6cfc1031d0 0x7f6cfc19ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.892+0000 7f6cfb400640 1 -- 172.21.15.39:0/2822379174 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6cfc19f4f0 con 0x7f6cfc101620 2024-03-18T16:11:12.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.892+0000 7f6cfbe00640 1 --2- 172.21.15.39:0/2822379174 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6cfc1031d0 0x7f6cfc19ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:11:12.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.892+0000 7f6cfb400640 1 --2- 172.21.15.39:0/2822379174 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6cfc101620 0x7f6cfc1978d0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f6cec00e9c0 tx=0x7f6cec00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:12.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.892+0000 7f6cf3e00640 1 -- 172.21.15.39:0/2822379174 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cec00cd80 con 0x7f6cfc101620 2024-03-18T16:11:12.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.892+0000 7f6cf3e00640 1 -- 172.21.15.39:0/2822379174 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6cec010910 con 0x7f6cfc101620 2024-03-18T16:11:12.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.892+0000 7f6cf3e00640 1 -- 172.21.15.39:0/2822379174 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cec018920 con 0x7f6cfc101620 2024-03-18T16:11:12.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.892+0000 7f6d01200640 1 -- 172.21.15.39:0/2822379174 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6cfc19f7e0 con 0x7f6cfc101620 2024-03-18T16:11:12.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.892+0000 7f6d01200640 1 -- 172.21.15.39:0/2822379174 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6cfc19fd80 con 0x7f6cfc101620 2024-03-18T16:11:12.898 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.896+0000 7f6d055b7740 1 Processor -- start 2024-03-18T16:11:12.898 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.896+0000 7f6d055b7740 1 -- start start 2024-03-18T16:11:12.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.896+0000 7f6cf3e00640 1 -- 172.21.15.39:0/2822379174 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f6cec01f020 con 0x7f6cfc101620 2024-03-18T16:11:12.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.896+0000 7f6d055b7740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f25a9cba90 0x55f25a9cbeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:12.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.896+0000 7f6d055b7740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f25a9cc480 0x55f25a9d0f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:12.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.896+0000 7f6d055b7740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f25a9d15f0 0x55f25a9d3a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:12.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.897+0000 7f6d055b7740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55f25a9933b0 con 0x55f25a9cba90 2024-03-18T16:11:12.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.897+0000 7f6d055b7740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55f25a930450 con 0x55f25a9d15f0 2024-03-18T16:11:12.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.897+0000 7f6d055b7740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55f25a8d8cb0 con 0x55f25a9cc480 2024-03-18T16:11:12.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.897+0000 7f6cfb400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f25a9cc480 0x55f25a9d0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:12.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.897+0000 7f6cfbe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f25a9cba90 0x55f25a9cbeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:12.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.897+0000 7f6cfb400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f25a9cc480 0x55f25a9d0f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:40020/0 (socket says 172.21.15.39:40020) 2024-03-18T16:11:12.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.897+0000 7f6cfaa00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f25a9d15f0 0x55f25a9d3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:12.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.897+0000 7f6cfb400640 1 -- 172.21.15.39:0/3548709823 learned_addr learned my addr 172.21.15.39:0/3548709823 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:11:12.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.897+0000 7f6cfb400640 1 -- 172.21.15.39:0/3548709823 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f25a9d15f0 msgr2=0x55f25a9d3a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:12.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.897+0000 7f6cfb400640 1 --2- 172.21.15.39:0/3548709823 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f25a9d15f0 0x55f25a9d3a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.897+0000 7f6cfb400640 1 -- 172.21.15.39:0/3548709823 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f25a9cba90 msgr2=0x55f25a9cbeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:12.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.897+0000 7f6cfb400640 1 --2- 172.21.15.39:0/3548709823 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f25a9cba90 0x55f25a9cbeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.898+0000 7f6cfb400640 1 -- 172.21.15.39:0/3548709823 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f25a8fa1f0 con 0x55f25a9cc480 2024-03-18T16:11:12.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.898+0000 7f6cfaa00640 1 --2- 172.21.15.39:0/3548709823 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f25a9d15f0 0x55f25a9d3a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:11:12.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.898+0000 7f6cfbe00640 1 --2- 172.21.15.39:0/3548709823 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f25a9cba90 0x55f25a9cbeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:11:12.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.898+0000 7f6cfb400640 1 --2- 172.21.15.39:0/3548709823 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f25a9cc480 0x55f25a9d0f80 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f6cec0d7290 tx=0x7f6cec0d7380 comp rx=0 tx=0).ready entity=mon.1 client_cookie=254576bbd1d6ee3a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:12.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.898+0000 7f6cf0c00640 1 -- 172.21.15.39:0/3548709823 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cec0e2070 con 0x55f25a9cc480 2024-03-18T16:11:12.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.898+0000 7f6cf3e00640 1 --2- 172.21.15.39:0/2822379174 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f6ce00a2880 0x7f6ce00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:12.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.898+0000 7f6cf0c00640 1 -- 172.21.15.39:0/3548709823 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6cec0d12e0 con 0x55f25a9cc480 2024-03-18T16:11:12.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.898+0000 7f6cf0c00640 1 -- 172.21.15.39:0/3548709823 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cec0dd660 con 0x55f25a9cc480 2024-03-18T16:11:12.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.899+0000 7f6cf3e00640 1 -- 172.21.15.39:0/2822379174 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f6cec0cf080 con 0x7f6cfc101620 2024-03-18T16:11:12.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.899+0000 7f6d055b7740 1 -- 172.21.15.39:0/3548709823 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f25a9cc480 msgr2=0x55f25a9d0f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:12.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.899+0000 7f6d055b7740 1 --2- 172.21.15.39:0/3548709823 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f25a9cc480 0x55f25a9d0f80 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f6cec0d7290 tx=0x7f6cec0d7380 comp rx=0 tx=0).stop 2024-03-18T16:11:12.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.899+0000 7f6cfbe00640 1 --2- 172.21.15.39:0/2822379174 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f6ce00a2880 0x7f6ce00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:12.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.899+0000 7f6d055b7740 1 -- 172.21.15.39:0/3548709823 shutdown_connections 2024-03-18T16:11:12.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.899+0000 7f6d055b7740 1 --2- 172.21.15.39:0/3548709823 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f25a9d15f0 0x55f25a9d3a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.899+0000 7f6d055b7740 1 --2- 172.21.15.39:0/3548709823 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f25a9cc480 0x55f25a9d0f80 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.899+0000 7f6d055b7740 1 --2- 172.21.15.39:0/3548709823 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f25a9cba90 0x55f25a9cbeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.899+0000 7f6d055b7740 1 -- 172.21.15.39:0/3548709823 >> 172.21.15.39:0/3548709823 conn(0x55f25a9c7000 msgr2=0x55f25a9c7400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:12.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.899+0000 7f6d055b7740 1 -- 172.21.15.39:0/3548709823 shutdown_connections 2024-03-18T16:11:12.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.900+0000 7f6d055b7740 1 -- 172.21.15.39:0/3548709823 wait complete. 2024-03-18T16:11:12.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.900+0000 7f6d055b7740 1 -- start start 2024-03-18T16:11:12.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.900+0000 7f6cfbe00640 1 --2- 172.21.15.39:0/2822379174 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f6ce00a2880 0x7f6ce00a4d40 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f6cf402c640 tx=0x7f6cf4032590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:12.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.900+0000 7f6d055b7740 1 Processor -- start 2024-03-18T16:11:12.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.901+0000 7f6d055b7740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f25a9cba90 0x55f25a9d5d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:12.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.901+0000 7f6d055b7740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f25a9cc480 0x55f25a9d62c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:12.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.901+0000 7f6d055b7740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f25a9d15f0 0x55f25a9dd2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:12.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.901+0000 7f6d055b7740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55f25a8ec110 con 0x55f25a9cc480 2024-03-18T16:11:12.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.901+0000 7f6d055b7740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55f25a92c4f0 con 0x55f25a9cba90 2024-03-18T16:11:12.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.901+0000 7f6d055b7740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55f25a9d45a0 con 0x55f25a9d15f0 2024-03-18T16:11:12.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.901+0000 7f6d055b7740 10 client.0 _ll_register_callbacks cb 0x55f25a8d9b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:11:12.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.901+0000 7f6cfaa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f25a9cc480 0x55f25a9d62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:12.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.901+0000 7f6cfb400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f25a9cba90 0x55f25a9d5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:12.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.901+0000 7f6cfaa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f25a9cc480 0x55f25a9d62c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:34828/0 (socket says 172.21.15.39:34828) 2024-03-18T16:11:12.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.901+0000 7f6cfaa00640 1 -- 172.21.15.39:0/1970914052 learned_addr learned my addr 172.21.15.39:0/1970914052 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:11:12.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.901+0000 7f6cfbe00640 1 --2- 172.21.15.39:0/1970914052 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f25a9d15f0 0x55f25a9dd2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:12.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.902+0000 7f6cfb400640 1 -- 172.21.15.39:0/1970914052 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f25a9d15f0 msgr2=0x55f25a9dd2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:12.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.902+0000 7f6cfb400640 1 --2- 172.21.15.39:0/1970914052 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f25a9d15f0 0x55f25a9dd2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.902+0000 7f6cfb400640 1 -- 172.21.15.39:0/1970914052 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f25a9cc480 msgr2=0x55f25a9d62c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:12.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.902+0000 7f6cfb400640 1 --2- 172.21.15.39:0/1970914052 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f25a9cc480 0x55f25a9d62c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.902+0000 7f6cfb400640 1 -- 172.21.15.39:0/1970914052 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f25a6443f0 con 0x55f25a9cba90 2024-03-18T16:11:12.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.902+0000 7f6cfaa00640 1 --2- 172.21.15.39:0/1970914052 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f25a9cc480 0x55f25a9d62c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:11:12.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.902+0000 7f6cfb400640 1 --2- 172.21.15.39:0/1970914052 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f25a9cba90 0x55f25a9d5d80 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f6cec00c6a0 tx=0x7f6cec009700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9ed7ff51f219322e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:12.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.902+0000 7f6ce5600640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:11:12.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.902+0000 7f6ce5600640 1 -- 172.21.15.39:0/1970914052 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cec0978d0 con 0x55f25a9cba90 2024-03-18T16:11:12.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.902+0000 7f6ce5600640 1 -- 172.21.15.39:0/1970914052 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f6cec097ca0 con 0x55f25a9cba90 2024-03-18T16:11:12.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.902+0000 7f6d055b7740 1 -- 172.21.15.39:0/1970914052 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6cec0d12e0 con 0x55f25a9cba90 2024-03-18T16:11:12.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.902+0000 7f6ce5600640 1 -- 172.21.15.39:0/1970914052 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6cec09af90 con 0x55f25a9cba90 2024-03-18T16:11:12.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.902+0000 7f6d055b7740 10 client.25841 fetch_fsmap learned FSMap version 28 2024-03-18T16:11:12.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.903+0000 7f6d055b7740 1 -- 172.21.15.39:0/1970914052 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f25a70e7d0 con 0x55f25a9cba90 2024-03-18T16:11:12.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.903+0000 7f6ce5600640 1 -- 172.21.15.39:0/1970914052 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6cec0d12e0 con 0x55f25a9cba90 2024-03-18T16:11:12.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.903+0000 7f6ce5600640 1 -- 172.21.15.39:0/1970914052 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f6cec09b890 con 0x55f25a9cba90 2024-03-18T16:11:12.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.903+0000 7f6d055b7740 10 client.25841 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T16:11:12.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.903+0000 7f6d055b7740 10 client.25841 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:11:12.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.903+0000 7f6d055b7740 20 client.25841 populate_metadata read hostname 'smithi039' 2024-03-18T16:11:12.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.903+0000 7f6d055b7740 1 --2- 172.21.15.39:0/1970914052 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55f25a9e17d0 0x55f25a9e3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:12.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.903+0000 7f6d055b7740 4 client.25841 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:11:12.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.903+0000 7f6d055b7740 1 -- 172.21.15.39:0/1970914052 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f25a93d990 con 0x55f25a9e17d0 2024-03-18T16:11:12.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.903+0000 7f6cfaa00640 1 --2- 172.21.15.39:0/1970914052 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55f25a9e17d0 0x55f25a9e3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:12.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.904+0000 7f6cfaa00640 1 --2- 172.21.15.39:0/1970914052 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55f25a9e17d0 0x55f25a9e3bc0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=def29e419920fa42 server_cookie=7f2fdd4a4214e1be in_seq=0 out_seq=0 2024-03-18T16:11:12.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.904+0000 7f6ce5600640 10 client.25841 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:11:12.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.905+0000 7f6ce5600640 1 -- 172.21.15.39:0/1970914052 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f25a93d990 con 0x55f25a9e17d0 2024-03-18T16:11:12.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.905+0000 7f6ce5600640 10 client.25841 handle_command_reply: tid=0 2024-03-18T16:11:12.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.905+0000 7f6d055b7740 1 client.25841 shutdown 2024-03-18T16:11:12.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.905+0000 7f6d055b7740 1 -- 172.21.15.39:0/1970914052 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f25a9cba90 msgr2=0x55f25a9d5d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:12.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.905+0000 7f6d055b7740 1 --2- 172.21.15.39:0/1970914052 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f25a9cba90 0x55f25a9d5d80 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f6cec00c6a0 tx=0x7f6cec009700 comp rx=0 tx=0).stop 2024-03-18T16:11:12.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.905+0000 7f6d055b7740 1 -- 172.21.15.39:0/1970914052 shutdown_connections 2024-03-18T16:11:12.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.905+0000 7f6d055b7740 1 --2- 172.21.15.39:0/1970914052 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55f25a9e17d0 0x55f25a9e3bc0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.905+0000 7f6d055b7740 1 --2- 172.21.15.39:0/1970914052 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f25a9d15f0 0x55f25a9dd2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.905+0000 7f6d055b7740 1 --2- 172.21.15.39:0/1970914052 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f25a9cc480 0x55f25a9d62c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.905+0000 7f6d055b7740 1 --2- 172.21.15.39:0/1970914052 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f25a9cba90 0x55f25a9d5d80 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.905+0000 7f6d055b7740 1 -- 172.21.15.39:0/1970914052 >> 172.21.15.39:0/1970914052 conn(0x55f25a9c7000 msgr2=0x55f25a9c6220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:12.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.905+0000 7f6ce5600640 0 client.25841 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:11:12.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.906+0000 7f6d055b7740 1 -- 172.21.15.39:0/1970914052 shutdown_connections 2024-03-18T16:11:12.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.906+0000 7f6d055b7740 1 -- 172.21.15.39:0/1970914052 wait complete. 2024-03-18T16:11:12.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.906+0000 7f6d055b7740 20 client.25841 trim_cache size 0 max 16384 2024-03-18T16:11:12.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.921+0000 7f6d055b7740 1 Processor -- start 2024-03-18T16:11:12.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.921+0000 7f6d055b7740 1 -- start start 2024-03-18T16:11:12.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.921+0000 7f6d055b7740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f25a9cba90 0x55f25a9ca970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:12.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.921+0000 7f6d055b7740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f25a9cc480 0x55f25a9dd280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:12.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.922+0000 7f6d055b7740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f25a9d15f0 0x55f25a9e82a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:12.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.922+0000 7f6d055b7740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f6cfc1ac4c0 con 0x55f25a9cc480 2024-03-18T16:11:12.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.922+0000 7f6d055b7740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f6cfc1ae000 con 0x55f25a9d15f0 2024-03-18T16:11:12.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.922+0000 7f6d055b7740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f6cfc1acb90 con 0x55f25a9cba90 2024-03-18T16:11:12.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.922+0000 7f6cfb400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f25a9cba90 0x55f25a9ca970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:12.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.922+0000 7f6cfaa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f25a9cc480 0x55f25a9dd280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:12.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.922+0000 7f6cfb400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f25a9cba90 0x55f25a9ca970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:40036/0 (socket says 172.21.15.39:40036) 2024-03-18T16:11:12.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.922+0000 7f6cfbe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f25a9d15f0 0x55f25a9e82a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:12.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.922+0000 7f6cfaa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f25a9cc480 0x55f25a9dd280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:34830/0 (socket says 172.21.15.39:34830) 2024-03-18T16:11:12.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.922+0000 7f6cfb400640 1 -- 172.21.15.39:0/3654381436 learned_addr learned my addr 172.21.15.39:0/3654381436 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:11:12.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.922+0000 7f6cfb400640 1 -- 172.21.15.39:0/3654381436 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f25a9d15f0 msgr2=0x55f25a9e82a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:12.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.922+0000 7f6cfb400640 1 --2- 172.21.15.39:0/3654381436 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f25a9d15f0 0x55f25a9e82a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.922+0000 7f6cfb400640 1 -- 172.21.15.39:0/3654381436 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f25a9cc480 msgr2=0x55f25a9dd280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:12.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.922+0000 7f6cfb400640 1 --2- 172.21.15.39:0/3654381436 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f25a9cc480 0x55f25a9dd280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.922+0000 7f6cfb400640 1 -- 172.21.15.39:0/3654381436 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f25a9cc860 con 0x55f25a9cba90 2024-03-18T16:11:12.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.922+0000 7f6cfbe00640 1 --2- 172.21.15.39:0/3654381436 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f25a9d15f0 0x55f25a9e82a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:11:12.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.923+0000 7f6cfb400640 1 --2- 172.21.15.39:0/3654381436 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f25a9cba90 0x55f25a9ca970 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f6cec09bc90 tx=0x7f6cec0972d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be9a32b300871171 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:12.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.923+0000 7f6ce7e00640 1 -- 172.21.15.39:0/3654381436 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cec09b170 con 0x55f25a9cba90 2024-03-18T16:11:12.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.923+0000 7f6ce7e00640 1 -- 172.21.15.39:0/3654381436 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6cec086070 con 0x55f25a9cba90 2024-03-18T16:11:12.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.923+0000 7f6ce7e00640 1 -- 172.21.15.39:0/3654381436 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cec09b400 con 0x55f25a9cba90 2024-03-18T16:11:12.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.924+0000 7f6d055b7740 1 -- 172.21.15.39:0/3654381436 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f25a9cba90 msgr2=0x55f25a9ca970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:12.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.924+0000 7f6d055b7740 1 --2- 172.21.15.39:0/3654381436 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f25a9cba90 0x55f25a9ca970 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f6cec09bc90 tx=0x7f6cec0972d0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.924+0000 7f6d055b7740 1 -- 172.21.15.39:0/3654381436 shutdown_connections 2024-03-18T16:11:12.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.924+0000 7f6d055b7740 1 --2- 172.21.15.39:0/3654381436 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f25a9d15f0 0x55f25a9e82a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.924+0000 7f6d055b7740 1 --2- 172.21.15.39:0/3654381436 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f25a9cc480 0x55f25a9dd280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.924+0000 7f6d055b7740 1 --2- 172.21.15.39:0/3654381436 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f25a9cba90 0x55f25a9ca970 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.924+0000 7f6d055b7740 1 -- 172.21.15.39:0/3654381436 >> 172.21.15.39:0/3654381436 conn(0x55f25a9c7000 msgr2=0x55f25a9ca260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:12.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.924+0000 7f6d055b7740 1 -- 172.21.15.39:0/3654381436 shutdown_connections 2024-03-18T16:11:12.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.924+0000 7f6d055b7740 1 -- 172.21.15.39:0/3654381436 wait complete. 2024-03-18T16:11:12.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.924+0000 7f6d055b7740 1 -- start start 2024-03-18T16:11:12.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.925+0000 7f6d055b7740 1 Processor -- start 2024-03-18T16:11:12.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.925+0000 7f6d055b7740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f25a9cba90 0x55f25a9dcb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:12.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.925+0000 7f6d055b7740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f25a9cc480 0x55f25a9dd0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:12.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.925+0000 7f6d055b7740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f25a9d15f0 0x55f25a9db180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:12.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.925+0000 7f6d055b7740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f6cfc1ad930 con 0x55f25a9d15f0 2024-03-18T16:11:12.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.925+0000 7f6d055b7740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f6cfc1ad260 con 0x55f25a9cc480 2024-03-18T16:11:12.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.925+0000 7f6d055b7740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f6cfc1aa4a0 con 0x55f25a9cba90 2024-03-18T16:11:12.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.925+0000 7f6d055b7740 10 client.0 _ll_register_callbacks cb 0x7f6cec0e0070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:11:12.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.926+0000 7f6cfaa00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f25a9cc480 0x55f25a9dd0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:12.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.926+0000 7f6cfbe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f25a9d15f0 0x55f25a9db180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:12.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.926+0000 7f6cfbe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f25a9d15f0 0x55f25a9db180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:34842/0 (socket says 172.21.15.39:34842) 2024-03-18T16:11:12.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.926+0000 7f6cfaa00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f25a9cc480 0x55f25a9dd0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:54756/0 (socket says 172.21.15.39:54756) 2024-03-18T16:11:12.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.926+0000 7f6cfbe00640 1 -- 172.21.15.39:0/2325920092 learned_addr learned my addr 172.21.15.39:0/2325920092 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:11:12.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.926+0000 7f6cfb400640 1 --2- 172.21.15.39:0/2325920092 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f25a9cba90 0x55f25a9dcb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:12.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.926+0000 7f6cfaa00640 1 -- 172.21.15.39:0/2325920092 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f25a9cba90 msgr2=0x55f25a9dcb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:12.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.926+0000 7f6cfaa00640 1 --2- 172.21.15.39:0/2325920092 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f25a9cba90 0x55f25a9dcb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.926+0000 7f6cfaa00640 1 -- 172.21.15.39:0/2325920092 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f25a9d15f0 msgr2=0x55f25a9db180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:12.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.926+0000 7f6cfaa00640 1 --2- 172.21.15.39:0/2325920092 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f25a9d15f0 0x55f25a9db180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.926+0000 7f6cfaa00640 1 -- 172.21.15.39:0/2325920092 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f25a9db8c0 con 0x55f25a9cc480 2024-03-18T16:11:12.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.926+0000 7f6cfbe00640 1 --2- 172.21.15.39:0/2325920092 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f25a9d15f0 0x55f25a9db180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:11:12.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.926+0000 7f6cfaa00640 1 --2- 172.21.15.39:0/2325920092 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f25a9cc480 0x55f25a9dd0b0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x55f25a8d8cd0 tx=0x7f6ce80124f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5add947a2708feb7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:12.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.926+0000 7f6ce5600640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:11:12.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.927+0000 7f6ce5600640 1 -- 172.21.15.39:0/2325920092 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ce8004460 con 0x55f25a9cc480 2024-03-18T16:11:12.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.927+0000 7f6ce5600640 1 -- 172.21.15.39:0/2325920092 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f6ce8020070 con 0x55f25a9cc480 2024-03-18T16:11:12.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.927+0000 7f6d055b7740 1 -- 172.21.15.39:0/2325920092 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6cec086070 con 0x55f25a9cc480 2024-03-18T16:11:12.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.927+0000 7f6ce5600640 1 -- 172.21.15.39:0/2325920092 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6ce8030050 con 0x55f25a9cc480 2024-03-18T16:11:12.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.927+0000 7f6d055b7740 10 client.25847 fetch_fsmap learned FSMap version 28 2024-03-18T16:11:12.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.927+0000 7f6d055b7740 1 -- 172.21.15.39:0/2325920092 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f25aa483f0 con 0x55f25a9cc480 2024-03-18T16:11:12.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.927+0000 7f6ce5600640 1 -- 172.21.15.39:0/2325920092 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6cec086070 con 0x55f25a9cc480 2024-03-18T16:11:12.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.928+0000 7f6ce5600640 1 -- 172.21.15.39:0/2325920092 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f6ce8003c00 con 0x55f25a9cc480 2024-03-18T16:11:12.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.928+0000 7f6d055b7740 10 client.25847 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T16:11:12.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.928+0000 7f6d055b7740 10 client.25847 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:11:12.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.928+0000 7f6d055b7740 20 client.25847 populate_metadata read hostname 'smithi039' 2024-03-18T16:11:12.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.928+0000 7f6d055b7740 1 --2- 172.21.15.39:0/2325920092 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55f25aa48cf0 0x55f25aa4b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:12.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.928+0000 7f6d055b7740 4 client.25847 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:11:12.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.928+0000 7f6d055b7740 1 -- 172.21.15.39:0/2325920092 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f25a937ce0 con 0x55f25aa48cf0 2024-03-18T16:11:12.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.929+0000 7f6cfb400640 1 --2- 172.21.15.39:0/2325920092 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55f25aa48cf0 0x55f25aa4b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:12.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.929+0000 7f6cfb400640 1 --2- 172.21.15.39:0/2325920092 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55f25aa48cf0 0x55f25aa4b0e0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e1de695e9a398aa3 server_cookie=21004054aafe7b13 in_seq=0 out_seq=0 2024-03-18T16:11:12.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.929+0000 7f6ce5600640 10 client.25847 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:11:12.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.929+0000 7f6ce5600640 1 -- 172.21.15.39:0/2325920092 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f25a937ce0 con 0x55f25aa48cf0 2024-03-18T16:11:12.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.929+0000 7f6ce5600640 10 client.25847 handle_command_reply: tid=0 2024-03-18T16:11:12.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.929+0000 7f6d055b7740 1 client.25847 shutdown 2024-03-18T16:11:12.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.930+0000 7f6d055b7740 1 -- 172.21.15.39:0/2325920092 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f25a9cc480 msgr2=0x55f25a9dd0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:12.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.930+0000 7f6d055b7740 1 --2- 172.21.15.39:0/2325920092 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f25a9cc480 0x55f25a9dd0b0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x55f25a8d8cd0 tx=0x7f6ce80124f0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.930+0000 7f6d055b7740 1 -- 172.21.15.39:0/2325920092 shutdown_connections 2024-03-18T16:11:12.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.930+0000 7f6d055b7740 1 --2- 172.21.15.39:0/2325920092 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55f25aa48cf0 0x55f25aa4b0e0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.930+0000 7f6d055b7740 1 --2- 172.21.15.39:0/2325920092 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f25a9cc480 0x55f25a9dd0b0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.930+0000 7f6d055b7740 1 --2- 172.21.15.39:0/2325920092 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f25a9d15f0 0x55f25a9db180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.930+0000 7f6d055b7740 1 --2- 172.21.15.39:0/2325920092 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f25a9cba90 0x55f25a9dcb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.930+0000 7f6d055b7740 1 -- 172.21.15.39:0/2325920092 >> 172.21.15.39:0/2325920092 conn(0x55f25a9c7000 msgr2=0x55f25a9caaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:12.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.930+0000 7f6ce5600640 0 client.25847 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:11:12.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.930+0000 7f6d055b7740 1 -- 172.21.15.39:0/2325920092 shutdown_connections 2024-03-18T16:11:12.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.930+0000 7f6d055b7740 1 -- 172.21.15.39:0/2325920092 wait complete. 2024-03-18T16:11:12.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.931+0000 7f6d055b7740 20 client.25847 trim_cache size 0 max 16384 2024-03-18T16:11:12.936 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:11:12.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.931+0000 7f6d01200640 1 -- 172.21.15.39:0/2822379174 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f6ce00a2880 msgr2=0x7f6ce00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:12.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.931+0000 7f6d01200640 1 --2- 172.21.15.39:0/2822379174 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f6ce00a2880 0x7f6ce00a4d40 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f6cf402c640 tx=0x7f6cf4032590 comp rx=0 tx=0).stop 2024-03-18T16:11:12.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.931+0000 7f6d01200640 1 -- 172.21.15.39:0/2822379174 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6cfc101620 msgr2=0x7f6cfc1978d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:12.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.931+0000 7f6d01200640 1 --2- 172.21.15.39:0/2822379174 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6cfc101620 0x7f6cfc1978d0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f6cec00e9c0 tx=0x7f6cec00ee80 comp rx=0 tx=0).stop 2024-03-18T16:11:12.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.932+0000 7f6d01200640 1 -- 172.21.15.39:0/2822379174 shutdown_connections 2024-03-18T16:11:12.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.932+0000 7f6d01200640 1 --2- 172.21.15.39:0/2822379174 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f6ce00a2880 0x7f6ce00a4d40 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.932+0000 7f6d01200640 1 --2- 172.21.15.39:0/2822379174 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6cfc1031d0 0x7f6cfc19ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.932+0000 7f6d01200640 1 --2- 172.21.15.39:0/2822379174 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6cfc102810 0x7f6cfc197e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.932+0000 7f6d01200640 1 --2- 172.21.15.39:0/2822379174 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6cfc101620 0x7f6cfc1978d0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:12.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.932+0000 7f6d01200640 1 -- 172.21.15.39:0/2822379174 >> 172.21.15.39:0/2822379174 conn(0x7f6cfc0fcb90 msgr2=0x7f6cfc105b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:12.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.932+0000 7f6d01200640 1 -- 172.21.15.39:0/2822379174 shutdown_connections 2024-03-18T16:11:12.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:12.932+0000 7f6d01200640 1 -- 172.21.15.39:0/2822379174 wait complete. 2024-03-18T16:11:12.952 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:11:12 smithi039 ceph-mon[31721]: pgmap v456: 129 pgs: 129 active+clean; 2.7 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 189 KiB/s wr, 24 op/s 2024-03-18T16:11:13.092 INFO:tasks.workunit.client.0.smithi039.stdout:config.status: creating Makefile 2024-03-18T16:11:13.130 INFO:tasks.workunit.client.0.smithi039.stdout:config.status: creating config.h 2024-03-18T16:11:13.141 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:11:12 smithi194 ceph-mon[36200]: pgmap v456: 129 pgs: 129 active+clean; 2.7 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 189 KiB/s wr, 24 op/s 2024-03-18T16:11:13.175 INFO:tasks.workunit.client.0.smithi039.stdout:config.status: executing depfiles commands 2024-03-18T16:11:13.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:11:12 smithi196 ceph-mon[36307]: pgmap v456: 129 pgs: 129 active+clean; 2.7 MiB data, 399 MiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 189 KiB/s wr, 24 op/s 2024-03-18T16:11:13.306 INFO:tasks.workunit.client.0.smithi039.stdout:config.status: executing default commands 2024-03-18T16:11:13.401 INFO:tasks.workunit.client.0.smithi039.stderr:+ make 2024-03-18T16:11:13.414 INFO:tasks.workunit.client.0.smithi039.stdout:cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run aclocal-1.10 2024-03-18T16:11:13.423 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing: line 46: aclocal-1.10: command not found 2024-03-18T16:11:13.426 INFO:tasks.workunit.client.0.smithi039.stderr:WARNING: `aclocal-1.10' is missing on your system. You should only need it if 2024-03-18T16:11:13.426 INFO:tasks.workunit.client.0.smithi039.stderr: you modified `acinclude.m4' or `configure.in'. You might want 2024-03-18T16:11:13.426 INFO:tasks.workunit.client.0.smithi039.stderr: to install the `Automake' and `Perl' packages. Grab them from 2024-03-18T16:11:13.426 INFO:tasks.workunit.client.0.smithi039.stderr: any GNU archive site. 2024-03-18T16:11:13.433 INFO:tasks.workunit.client.0.smithi039.stdout: cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run automake-1.10 --gnu 2024-03-18T16:11:13.437 INFO:tasks.workunit.client.0.smithi039.stderr:/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing: line 46: automake-1.10: command not found 2024-03-18T16:11:13.440 INFO:tasks.workunit.client.0.smithi039.stderr:WARNING: `automake-1.10' is missing on your system. You should only need it if 2024-03-18T16:11:13.440 INFO:tasks.workunit.client.0.smithi039.stderr: you modified `Makefile.am', `acinclude.m4' or `configure.in'. 2024-03-18T16:11:13.440 INFO:tasks.workunit.client.0.smithi039.stderr: You might want to install the `Automake' and `Perl' packages. 2024-03-18T16:11:13.440 INFO:tasks.workunit.client.0.smithi039.stderr: Grab them from any GNU archive site. 2024-03-18T16:11:13.535 INFO:tasks.workunit.client.0.smithi039.stdout:cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run autoconf 2024-03-18T16:11:13.955 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-18T16:11:13.955 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T16:11:13.956 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-18T16:11:14.146 INFO:tasks.workunit.client.0.smithi039.stderr:aclocal.m4:16: warning: this file was generated for autoconf 2.61. 2024-03-18T16:11:14.146 INFO:tasks.workunit.client.0.smithi039.stderr:You have another version of autoconf. It may work, but is not guaranteed to. 2024-03-18T16:11:14.146 INFO:tasks.workunit.client.0.smithi039.stderr:If you have problems, you may need to regenerate the build system entirely. 2024-03-18T16:11:14.146 INFO:tasks.workunit.client.0.smithi039.stderr:To do so, use the procedure documented by the package, typically `autoreconf'. 2024-03-18T16:11:14.186 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.184+0000 7f90f6600640 1 -- 172.21.15.39:0/2900140011 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f90f01021a0 msgr2=0x7f90f0104590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:14.186 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.184+0000 7f90f6600640 1 --2- 172.21.15.39:0/2900140011 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f90f01021a0 0x7f90f0104590 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f90e4009900 tx=0x7f90e402bc40 comp rx=0 tx=0).stop 2024-03-18T16:11:14.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.184+0000 7f90f6600640 1 -- 172.21.15.39:0/2900140011 shutdown_connections 2024-03-18T16:11:14.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.184+0000 7f90f6600640 1 --2- 172.21.15.39:0/2900140011 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f90f0107400 0x7f90f01097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:14.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.184+0000 7f90f6600640 1 --2- 172.21.15.39:0/2900140011 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f90f0104ad0 0x7f90f0106ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:14.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.184+0000 7f90f6600640 1 --2- 172.21.15.39:0/2900140011 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f90f01021a0 0x7f90f0104590 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:14.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.184+0000 7f90f6600640 1 -- 172.21.15.39:0/2900140011 >> 172.21.15.39:0/2900140011 conn(0x7f90f00fbb50 msgr2=0x7f90f00fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:14.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.184+0000 7f90f6600640 1 -- 172.21.15.39:0/2900140011 shutdown_connections 2024-03-18T16:11:14.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.184+0000 7f90f6600640 1 -- 172.21.15.39:0/2900140011 wait complete. 2024-03-18T16:11:14.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.185+0000 7f90f6600640 1 Processor -- start 2024-03-18T16:11:14.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.186+0000 7f90f6600640 1 -- start start 2024-03-18T16:11:14.189 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.186+0000 7f90f6600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f90f01021a0 0x7f90f0101b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:14.189 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.186+0000 7f90f6600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f90f0104ad0 0x7f90f0100150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:14.189 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.186+0000 7f90f6600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f90f0107400 0x7f90f0100690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:14.189 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.186+0000 7f90f6600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f90f010d380 con 0x7f90f01021a0 2024-03-18T16:11:14.189 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.186+0000 7f90f6600640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f90f010d200 con 0x7f90f0104ad0 2024-03-18T16:11:14.189 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.186+0000 7f90f6600640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f90f010d500 con 0x7f90f0107400 2024-03-18T16:11:14.189 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.186+0000 7f90efe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f90f0104ad0 0x7f90f0100150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:14.189 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.186+0000 7f90f5c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f90f0107400 0x7f90f0100690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:14.189 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.186+0000 7f90f5200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f90f01021a0 0x7f90f0101b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:14.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.186+0000 7f90efe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f90f0104ad0 0x7f90f0100150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:54770/0 (socket says 172.21.15.39:54770) 2024-03-18T16:11:14.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.186+0000 7f90efe00640 1 -- 172.21.15.39:0/2385217961 learned_addr learned my addr 172.21.15.39:0/2385217961 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:11:14.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.187+0000 7f90efe00640 1 -- 172.21.15.39:0/2385217961 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f90f0107400 msgr2=0x7f90f0100690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:14.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.187+0000 7f90efe00640 1 --2- 172.21.15.39:0/2385217961 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f90f0107400 0x7f90f0100690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:14.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.187+0000 7f90efe00640 1 -- 172.21.15.39:0/2385217961 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f90f01021a0 msgr2=0x7f90f0101b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:14.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.187+0000 7f90efe00640 1 --2- 172.21.15.39:0/2385217961 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f90f01021a0 0x7f90f0101b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:14.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.187+0000 7f90efe00640 1 -- 172.21.15.39:0/2385217961 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90f0100f20 con 0x7f90f0104ad0 2024-03-18T16:11:14.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.187+0000 7f90f5c00640 1 --2- 172.21.15.39:0/2385217961 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f90f0107400 0x7f90f0100690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:11:14.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.187+0000 7f90efe00640 1 --2- 172.21.15.39:0/2385217961 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f90f0104ad0 0x7f90f0100150 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f90dc00e9c0 tx=0x7f90dc00edc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:14.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.187+0000 7f90ed600640 1 -- 172.21.15.39:0/2385217961 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90dc00ccc0 con 0x7f90f0104ad0 2024-03-18T16:11:14.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.187+0000 7f90f6600640 1 -- 172.21.15.39:0/2385217961 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90f01a1af0 con 0x7f90f0104ad0 2024-03-18T16:11:14.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.187+0000 7f90ed600640 1 -- 172.21.15.39:0/2385217961 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f90dc005500 con 0x7f90f0104ad0 2024-03-18T16:11:14.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.188+0000 7f90ed600640 1 -- 172.21.15.39:0/2385217961 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90dc010430 con 0x7f90f0104ad0 2024-03-18T16:11:14.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.188+0000 7f90f6600640 1 -- 172.21.15.39:0/2385217961 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90f01a2040 con 0x7f90f0104ad0 2024-03-18T16:11:14.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.192+0000 7f90ed600640 1 -- 172.21.15.39:0/2385217961 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f90dc005000 con 0x7f90f0104ad0 2024-03-18T16:11:14.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.192+0000 7f90faa01740 1 Processor -- start 2024-03-18T16:11:14.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.193+0000 7f90faa01740 1 -- start start 2024-03-18T16:11:14.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.193+0000 7f90faa01740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ebc2691980 0x55ebc2691da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:14.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.193+0000 7f90faa01740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ebc2692370 0x55ebc2696e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:14.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.193+0000 7f90faa01740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ebc26974e0 0x55ebc2699970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:14.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.193+0000 7f90faa01740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55ebc25f4f80 con 0x55ebc2691980 2024-03-18T16:11:14.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.193+0000 7f90faa01740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55ebc259dad0 con 0x55ebc26974e0 2024-03-18T16:11:14.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.193+0000 7f90faa01740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55ebc25f1020 con 0x55ebc2692370 2024-03-18T16:11:14.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.193+0000 7f90f5200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ebc2692370 0x55ebc2696e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:14.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.193+0000 7f90ed600640 1 --2- 172.21.15.39:0/2385217961 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f90d80a2800 0x7f90d80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:14.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.193+0000 7f90f5200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ebc2692370 0x55ebc2696e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:40082/0 (socket says 172.21.15.39:40082) 2024-03-18T16:11:14.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.193+0000 7f90f5200640 1 -- 172.21.15.39:0/2268140638 learned_addr learned my addr 172.21.15.39:0/2268140638 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:11:14.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.193+0000 7f90f5c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ebc2691980 0x55ebc2691da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:14.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.193+0000 7f90efe00640 1 --2- 172.21.15.39:0/2268140638 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ebc26974e0 0x55ebc2699970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:14.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.194+0000 7f90ed600640 1 -- 172.21.15.39:0/2385217961 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f90dc0cf2a0 con 0x7f90f0104ad0 2024-03-18T16:11:14.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.194+0000 7f90f5200640 1 -- 172.21.15.39:0/2268140638 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ebc26974e0 msgr2=0x55ebc2699970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:14.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.194+0000 7f90f5200640 1 --2- 172.21.15.39:0/2268140638 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ebc26974e0 0x55ebc2699970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:14.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.194+0000 7f90f5200640 1 -- 172.21.15.39:0/2268140638 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ebc2691980 msgr2=0x55ebc2691da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:14.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.194+0000 7f90f5200640 1 --2- 172.21.15.39:0/2268140638 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ebc2691980 0x55ebc2691da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:14.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.194+0000 7f90f5200640 1 -- 172.21.15.39:0/2268140638 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ebc25b56f0 con 0x55ebc2692370 2024-03-18T16:11:14.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.194+0000 7f90efe00640 1 --2- 172.21.15.39:0/2268140638 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ebc26974e0 0x55ebc2699970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:11:14.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.194+0000 7f90f5c00640 1 --2- 172.21.15.39:0/2385217961 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f90d80a2800 0x7f90d80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:14.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.194+0000 7f90f5200640 1 --2- 172.21.15.39:0/2268140638 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ebc2692370 0x55ebc2696e70 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f90e4005ab0 tx=0x7f90e4000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3078897ab13221ed server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:14.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.195+0000 7f90e2000640 1 -- 172.21.15.39:0/2268140638 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90e4042070 con 0x55ebc2692370 2024-03-18T16:11:14.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.195+0000 7f90e2000640 1 -- 172.21.15.39:0/2268140638 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f90e402cb90 con 0x55ebc2692370 2024-03-18T16:11:14.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.195+0000 7f90e2000640 1 -- 172.21.15.39:0/2268140638 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90e4005130 con 0x55ebc2692370 2024-03-18T16:11:14.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.196+0000 7f90faa01740 1 -- 172.21.15.39:0/2268140638 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ebc2692370 msgr2=0x55ebc2696e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:14.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.196+0000 7f90f5c00640 1 --2- 172.21.15.39:0/2385217961 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f90d80a2800 0x7f90d80a4cc0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x55ebc2692bd0 tx=0x7f90e8004ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:14.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.196+0000 7f90faa01740 1 --2- 172.21.15.39:0/2268140638 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ebc2692370 0x55ebc2696e70 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f90e4005ab0 tx=0x7f90e4000bc0 comp rx=0 tx=0).stop 2024-03-18T16:11:14.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.196+0000 7f90faa01740 1 -- 172.21.15.39:0/2268140638 shutdown_connections 2024-03-18T16:11:14.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.196+0000 7f90faa01740 1 --2- 172.21.15.39:0/2268140638 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ebc26974e0 0x55ebc2699970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:14.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.196+0000 7f90faa01740 1 --2- 172.21.15.39:0/2268140638 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ebc2692370 0x55ebc2696e70 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:14.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.196+0000 7f90faa01740 1 --2- 172.21.15.39:0/2268140638 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ebc2691980 0x55ebc2691da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:14.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.196+0000 7f90faa01740 1 -- 172.21.15.39:0/2268140638 >> 172.21.15.39:0/2268140638 conn(0x55ebc2659930 msgr2=0x55ebc268d2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:14.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.196+0000 7f90faa01740 1 -- 172.21.15.39:0/2268140638 shutdown_connections 2024-03-18T16:11:14.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.196+0000 7f90faa01740 1 -- 172.21.15.39:0/2268140638 wait complete. 2024-03-18T16:11:14.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.196+0000 7f90faa01740 1 -- start start 2024-03-18T16:11:14.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.197+0000 7f90faa01740 1 Processor -- start 2024-03-18T16:11:14.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.197+0000 7f90faa01740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ebc2691980 0x55ebc269bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:14.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.197+0000 7f90faa01740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ebc2692370 0x55ebc269c190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:14.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.197+0000 7f90faa01740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ebc26974e0 0x55ebc26a31d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:14.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.197+0000 7f90faa01740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55ebc269a430 con 0x55ebc2692370 2024-03-18T16:11:14.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.198+0000 7f90faa01740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55ebc25b0f30 con 0x55ebc26974e0 2024-03-18T16:11:14.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.198+0000 7f90faa01740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55ebc269a5b0 con 0x55ebc2691980 2024-03-18T16:11:14.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.198+0000 7f90faa01740 10 client.0 _ll_register_callbacks cb 0x55ebc259ebc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:11:14.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.198+0000 7f90f5200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ebc2691980 0x55ebc269bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:14.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.198+0000 7f90f5200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ebc2691980 0x55ebc269bc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:40096/0 (socket says 172.21.15.39:40096) 2024-03-18T16:11:14.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.198+0000 7f90f5200640 1 -- 172.21.15.39:0/484382963 learned_addr learned my addr 172.21.15.39:0/484382963 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:11:14.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.198+0000 7f90efe00640 1 --2- 172.21.15.39:0/484382963 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ebc2692370 0x55ebc269c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:14.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.198+0000 7f90f5c00640 1 --2- 172.21.15.39:0/484382963 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ebc26974e0 0x55ebc26a31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:14.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.198+0000 7f90f5200640 1 -- 172.21.15.39:0/484382963 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ebc26974e0 msgr2=0x55ebc26a31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:14.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.198+0000 7f90f5200640 1 --2- 172.21.15.39:0/484382963 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ebc26974e0 0x55ebc26a31d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:14.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.198+0000 7f90f5200640 1 -- 172.21.15.39:0/484382963 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ebc2692370 msgr2=0x55ebc269c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:14.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.198+0000 7f90f5200640 1 --2- 172.21.15.39:0/484382963 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ebc2692370 0x55ebc269c190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:14.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.198+0000 7f90f5200640 1 -- 172.21.15.39:0/484382963 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ebc23d17e0 con 0x55ebc2691980 2024-03-18T16:11:14.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.198+0000 7f90efe00640 1 --2- 172.21.15.39:0/484382963 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ebc2692370 0x55ebc269c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:11:14.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.199+0000 7f90f5200640 1 --2- 172.21.15.39:0/484382963 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ebc2691980 0x55ebc269bc50 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f90e402c250 tx=0x7f90e4004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6dcef7d3a6a69684 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:14.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.199+0000 7f90d6a00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:11:14.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.199+0000 7f90d6a00640 1 -- 172.21.15.39:0/484382963 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90e4042070 con 0x55ebc2691980 2024-03-18T16:11:14.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.199+0000 7f90d6a00640 1 -- 172.21.15.39:0/484382963 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f90e4004ed0 con 0x55ebc2691980 2024-03-18T16:11:14.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.199+0000 7f90faa01740 1 -- 172.21.15.39:0/484382963 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f90e402cb90 con 0x55ebc2691980 2024-03-18T16:11:14.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.199+0000 7f90d6a00640 1 -- 172.21.15.39:0/484382963 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f90e4058050 con 0x55ebc2691980 2024-03-18T16:11:14.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.199+0000 7f90faa01740 10 client.25891 fetch_fsmap learned FSMap version 28 2024-03-18T16:11:14.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.199+0000 7f90faa01740 1 -- 172.21.15.39:0/484382963 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ebc2431f70 con 0x55ebc2691980 2024-03-18T16:11:14.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.200+0000 7f90d6a00640 1 -- 172.21.15.39:0/484382963 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f90e402cb90 con 0x55ebc2691980 2024-03-18T16:11:14.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.200+0000 7f90d6a00640 1 -- 172.21.15.39:0/484382963 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f90e4003b90 con 0x55ebc2691980 2024-03-18T16:11:14.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.200+0000 7f90faa01740 10 client.25891 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T16:11:14.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.200+0000 7f90faa01740 10 client.25891 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:11:14.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.200+0000 7f90faa01740 20 client.25891 populate_metadata read hostname 'smithi039' 2024-03-18T16:11:14.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.200+0000 7f90faa01740 1 --2- 172.21.15.39:0/484382963 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55ebc26a76c0 0x55ebc26a9ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:14.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.200+0000 7f90faa01740 4 client.25891 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:11:14.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.200+0000 7f90faa01740 1 -- 172.21.15.39:0/484382963 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ebc26024c0 con 0x55ebc26a76c0 2024-03-18T16:11:14.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.200+0000 7f90efe00640 1 --2- 172.21.15.39:0/484382963 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55ebc26a76c0 0x55ebc26a9ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:14.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.201+0000 7f90efe00640 1 --2- 172.21.15.39:0/484382963 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55ebc26a76c0 0x55ebc26a9ab0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a6cbe4648cd22ad5 server_cookie=a7e05a585519ec1 in_seq=0 out_seq=0 2024-03-18T16:11:14.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.201+0000 7f90d6a00640 10 client.25891 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:11:14.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.201+0000 7f90d6a00640 1 -- 172.21.15.39:0/484382963 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ebc26024c0 con 0x55ebc26a76c0 2024-03-18T16:11:14.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.201+0000 7f90d6a00640 10 client.25891 handle_command_reply: tid=0 2024-03-18T16:11:14.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.201+0000 7f90faa01740 1 client.25891 shutdown 2024-03-18T16:11:14.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.202+0000 7f90faa01740 1 -- 172.21.15.39:0/484382963 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ebc2691980 msgr2=0x55ebc269bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:14.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.202+0000 7f90faa01740 1 --2- 172.21.15.39:0/484382963 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ebc2691980 0x55ebc269bc50 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f90e402c250 tx=0x7f90e4004270 comp rx=0 tx=0).stop 2024-03-18T16:11:14.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.202+0000 7f90faa01740 1 -- 172.21.15.39:0/484382963 shutdown_connections 2024-03-18T16:11:14.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.202+0000 7f90faa01740 1 --2- 172.21.15.39:0/484382963 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55ebc26a76c0 0x55ebc26a9ab0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:14.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.202+0000 7f90faa01740 1 --2- 172.21.15.39:0/484382963 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ebc26974e0 0x55ebc26a31d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:14.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.202+0000 7f90faa01740 1 --2- 172.21.15.39:0/484382963 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ebc2692370 0x55ebc269c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:14.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.202+0000 7f90d6a00640 0 client.25891 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:11:14.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.202+0000 7f90faa01740 1 --2- 172.21.15.39:0/484382963 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ebc2691980 0x55ebc269bc50 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:14.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.202+0000 7f90faa01740 1 -- 172.21.15.39:0/484382963 >> 172.21.15.39:0/484382963 conn(0x55ebc2659930 msgr2=0x55ebc268c570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:14.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.202+0000 7f90faa01740 1 -- 172.21.15.39:0/484382963 shutdown_connections 2024-03-18T16:11:14.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.202+0000 7f90faa01740 1 -- 172.21.15.39:0/484382963 wait complete. 2024-03-18T16:11:14.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.202+0000 7f90faa01740 20 client.25891 trim_cache size 0 max 16384 2024-03-18T16:11:14.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.226+0000 7f90faa01740 1 Processor -- start 2024-03-18T16:11:14.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.226+0000 7f90faa01740 1 -- start start 2024-03-18T16:11:14.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.227+0000 7f90faa01740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ebc2659930 0x55ebc268c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:14.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.227+0000 7f90faa01740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ebc2691980 0x55ebc26a3190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:14.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.227+0000 7f90faa01740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ebc2692370 0x55ebc269b270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:14.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.227+0000 7f90faa01740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f90f01ac370 con 0x55ebc2692370 2024-03-18T16:11:14.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.227+0000 7f90faa01740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f90f01adeb0 con 0x55ebc2659930 2024-03-18T16:11:14.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.227+0000 7f90faa01740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f90f01aca40 con 0x55ebc2691980 2024-03-18T16:11:14.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.227+0000 7f90efe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ebc2691980 0x55ebc26a3190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:14.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.227+0000 7f90f5200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ebc2659930 0x55ebc268c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:14.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.227+0000 7f90efe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ebc2691980 0x55ebc26a3190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:40112/0 (socket says 172.21.15.39:40112) 2024-03-18T16:11:14.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.227+0000 7f90efe00640 1 -- 172.21.15.39:0/622271891 learned_addr learned my addr 172.21.15.39:0/622271891 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:11:14.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.227+0000 7f90f5c00640 1 --2- 172.21.15.39:0/622271891 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ebc2692370 0x55ebc269b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:14.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.227+0000 7f90efe00640 1 -- 172.21.15.39:0/622271891 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ebc2659930 msgr2=0x55ebc268c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:14.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.227+0000 7f90efe00640 1 --2- 172.21.15.39:0/622271891 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ebc2659930 0x55ebc268c3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:14.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.227+0000 7f90efe00640 1 -- 172.21.15.39:0/622271891 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ebc2692370 msgr2=0x55ebc269b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:14.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.227+0000 7f90efe00640 1 --2- 172.21.15.39:0/622271891 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ebc2692370 0x55ebc269b270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:14.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.227+0000 7f90efe00640 1 -- 172.21.15.39:0/622271891 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ebc22c9170 con 0x55ebc2691980 2024-03-18T16:11:14.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.227+0000 7f90f5c00640 1 --2- 172.21.15.39:0/622271891 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ebc2692370 0x55ebc269b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:11:14.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.228+0000 7f90efe00640 1 --2- 172.21.15.39:0/622271891 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ebc2691980 0x55ebc26a3190 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f90dc0981d0 tx=0x7f90dc086040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1cc3f740982c1af3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:14.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.228+0000 7f90e1600640 1 -- 172.21.15.39:0/622271891 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90dc09cb80 con 0x55ebc2691980 2024-03-18T16:11:14.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.228+0000 7f90e1600640 1 -- 172.21.15.39:0/622271891 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f90dc09bf50 con 0x55ebc2691980 2024-03-18T16:11:14.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.228+0000 7f90e1600640 1 -- 172.21.15.39:0/622271891 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90dc0ccac0 con 0x55ebc2691980 2024-03-18T16:11:14.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.229+0000 7f90faa01740 1 -- 172.21.15.39:0/622271891 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ebc2691980 msgr2=0x55ebc26a3190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:14.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.229+0000 7f90faa01740 1 --2- 172.21.15.39:0/622271891 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ebc2691980 0x55ebc26a3190 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f90dc0981d0 tx=0x7f90dc086040 comp rx=0 tx=0).stop 2024-03-18T16:11:14.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.229+0000 7f90faa01740 1 -- 172.21.15.39:0/622271891 shutdown_connections 2024-03-18T16:11:14.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.229+0000 7f90faa01740 1 --2- 172.21.15.39:0/622271891 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ebc2692370 0x55ebc269b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:14.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.229+0000 7f90faa01740 1 --2- 172.21.15.39:0/622271891 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ebc2691980 0x55ebc26a3190 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:14.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.229+0000 7f90faa01740 1 --2- 172.21.15.39:0/622271891 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ebc2659930 0x55ebc268c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:14.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.229+0000 7f90faa01740 1 -- 172.21.15.39:0/622271891 >> 172.21.15.39:0/622271891 conn(0x55ebc2692f70 msgr2=0x55ebc268bce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:14.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.229+0000 7f90faa01740 1 -- 172.21.15.39:0/622271891 shutdown_connections 2024-03-18T16:11:14.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.229+0000 7f90faa01740 1 -- 172.21.15.39:0/622271891 wait complete. 2024-03-18T16:11:14.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.229+0000 7f90faa01740 1 -- start start 2024-03-18T16:11:14.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.229+0000 7f90faa01740 1 Processor -- start 2024-03-18T16:11:14.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.230+0000 7f90faa01740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ebc2659930 0x55ebc26a0900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:14.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.230+0000 7f90faa01740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ebc2691980 0x55ebc26a0e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:14.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.230+0000 7f90faa01740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ebc2692370 0x55ebc26a1e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:14.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.230+0000 7f90faa01740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f90f01ad7e0 con 0x55ebc2692370 2024-03-18T16:11:14.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.230+0000 7f90faa01740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f90f01ad110 con 0x55ebc2691980 2024-03-18T16:11:14.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.230+0000 7f90faa01740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f90f01aa350 con 0x55ebc2659930 2024-03-18T16:11:14.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.230+0000 7f90faa01740 10 client.0 _ll_register_callbacks cb 0x7f90dc09b6a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:11:14.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.230+0000 7f90f5200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ebc2659930 0x55ebc26a0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:14.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.230+0000 7f90efe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ebc2691980 0x55ebc26a0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:14.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.230+0000 7f90f5200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ebc2659930 0x55ebc26a0900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:40124/0 (socket says 172.21.15.39:40124) 2024-03-18T16:11:14.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.230+0000 7f90f5200640 1 -- 172.21.15.39:0/2439357557 learned_addr learned my addr 172.21.15.39:0/2439357557 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:11:14.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.230+0000 7f90f5c00640 1 --2- 172.21.15.39:0/2439357557 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ebc2692370 0x55ebc26a1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:14.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.231+0000 7f90f5200640 1 -- 172.21.15.39:0/2439357557 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ebc2691980 msgr2=0x55ebc26a0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:14.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.231+0000 7f90f5200640 1 --2- 172.21.15.39:0/2439357557 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ebc2691980 0x55ebc26a0e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:14.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.231+0000 7f90f5200640 1 -- 172.21.15.39:0/2439357557 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ebc2692370 msgr2=0x55ebc26a1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:14.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.231+0000 7f90f5200640 1 --2- 172.21.15.39:0/2439357557 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ebc2692370 0x55ebc26a1e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:14.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.231+0000 7f90f5200640 1 -- 172.21.15.39:0/2439357557 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ebc26a2440 con 0x55ebc2659930 2024-03-18T16:11:14.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.231+0000 7f90f5c00640 1 --2- 172.21.15.39:0/2439357557 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ebc2692370 0x55ebc26a1e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:11:14.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.231+0000 7f90f5200640 1 --2- 172.21.15.39:0/2439357557 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ebc2659930 0x55ebc26a0900 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f90e4000c00 tx=0x7f90e4004c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c28ad464c6d42a97 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:14.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.231+0000 7f90d6a00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:11:14.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.231+0000 7f90d6a00640 1 -- 172.21.15.39:0/2439357557 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90e4061060 con 0x55ebc2659930 2024-03-18T16:11:14.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.231+0000 7f90d6a00640 1 -- 172.21.15.39:0/2439357557 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f90e40525b0 con 0x55ebc2659930 2024-03-18T16:11:14.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.231+0000 7f90faa01740 1 -- 172.21.15.39:0/2439357557 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f90dc09bf50 con 0x55ebc2659930 2024-03-18T16:11:14.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.232+0000 7f90d6a00640 1 -- 172.21.15.39:0/2439357557 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f90e4068050 con 0x55ebc2659930 2024-03-18T16:11:14.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.232+0000 7f90faa01740 10 client.25897 fetch_fsmap learned FSMap version 28 2024-03-18T16:11:14.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.232+0000 7f90faa01740 1 -- 172.21.15.39:0/2439357557 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ebc270d9f0 con 0x55ebc2659930 2024-03-18T16:11:14.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.232+0000 7f90d6a00640 1 -- 172.21.15.39:0/2439357557 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f90dc09bf50 con 0x55ebc2659930 2024-03-18T16:11:14.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.232+0000 7f90d6a00640 1 -- 172.21.15.39:0/2439357557 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f90e40527d0 con 0x55ebc2659930 2024-03-18T16:11:14.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.232+0000 7f90faa01740 10 client.25897 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T16:11:14.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.232+0000 7f90faa01740 10 client.25897 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:11:14.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.232+0000 7f90faa01740 20 client.25897 populate_metadata read hostname 'smithi039' 2024-03-18T16:11:14.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.232+0000 7f90faa01740 1 --2- 172.21.15.39:0/2439357557 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55ebc26974e0 0x55ebc2710320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:14.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.232+0000 7f90faa01740 4 client.25897 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-18T16:11:14.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.232+0000 7f90faa01740 1 -- 172.21.15.39:0/2439357557 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55ebc25fc810 con 0x55ebc26974e0 2024-03-18T16:11:14.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.232+0000 7f90efe00640 1 --2- 172.21.15.39:0/2439357557 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55ebc26974e0 0x55ebc2710320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:14.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.233+0000 7f90efe00640 1 --2- 172.21.15.39:0/2439357557 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55ebc26974e0 0x55ebc2710320 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=703f4370d1ce0d0c server_cookie=2235e4487765bcd9 in_seq=0 out_seq=0 2024-03-18T16:11:14.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.233+0000 7f90d6a00640 10 client.25897 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:11:14.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.234+0000 7f90d6a00640 1 -- 172.21.15.39:0/2439357557 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55ebc25fc810 con 0x55ebc26974e0 2024-03-18T16:11:14.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.234+0000 7f90d6a00640 10 client.25897 handle_command_reply: tid=0 2024-03-18T16:11:14.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.234+0000 7f90faa01740 1 client.25897 shutdown 2024-03-18T16:11:14.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.234+0000 7f90faa01740 1 -- 172.21.15.39:0/2439357557 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ebc2659930 msgr2=0x55ebc26a0900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:14.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.234+0000 7f90faa01740 1 --2- 172.21.15.39:0/2439357557 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ebc2659930 0x55ebc26a0900 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f90e4000c00 tx=0x7f90e4004c00 comp rx=0 tx=0).stop 2024-03-18T16:11:14.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.235+0000 7f90faa01740 1 -- 172.21.15.39:0/2439357557 shutdown_connections 2024-03-18T16:11:14.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.235+0000 7f90faa01740 1 --2- 172.21.15.39:0/2439357557 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55ebc26974e0 0x55ebc2710320 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:14.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.235+0000 7f90faa01740 1 --2- 172.21.15.39:0/2439357557 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ebc2691980 0x55ebc26a0e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:14.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.235+0000 7f90faa01740 1 --2- 172.21.15.39:0/2439357557 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ebc2692370 0x55ebc26a1e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:14.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.235+0000 7f90faa01740 1 --2- 172.21.15.39:0/2439357557 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ebc2659930 0x55ebc26a0900 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:14.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.235+0000 7f90faa01740 1 -- 172.21.15.39:0/2439357557 >> 172.21.15.39:0/2439357557 conn(0x55ebc2692f70 msgr2=0x55ebc2690dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:14.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.235+0000 7f90d6a00640 0 client.25897 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:11:14.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.235+0000 7f90faa01740 1 -- 172.21.15.39:0/2439357557 shutdown_connections 2024-03-18T16:11:14.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.235+0000 7f90faa01740 1 -- 172.21.15.39:0/2439357557 wait complete. 2024-03-18T16:11:14.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.235+0000 7f90faa01740 20 client.25897 trim_cache size 0 max 16384 2024-03-18T16:11:14.240 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-03-18T16:11:14.240 INFO:teuthology.orchestra.run.smithi039.stdout: "return_code": 0, 2024-03-18T16:11:14.240 INFO:teuthology.orchestra.run.smithi039.stdout: "scrub_tag": "ed05154a-4a21-4471-b382-740ffddd52ab", 2024-03-18T16:11:14.240 INFO:teuthology.orchestra.run.smithi039.stdout: "mode": "asynchronous" 2024-03-18T16:11:14.240 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-03-18T16:11:14.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.236+0000 7f90f6600640 1 -- 172.21.15.39:0/2385217961 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f90d80a2800 msgr2=0x7f90d80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:14.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.236+0000 7f90f6600640 1 --2- 172.21.15.39:0/2385217961 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f90d80a2800 0x7f90d80a4cc0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x55ebc2692bd0 tx=0x7f90e8004ab0 comp rx=0 tx=0).stop 2024-03-18T16:11:14.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.236+0000 7f90f6600640 1 -- 172.21.15.39:0/2385217961 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f90f0104ad0 msgr2=0x7f90f0100150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:14.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.236+0000 7f90f6600640 1 --2- 172.21.15.39:0/2385217961 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f90f0104ad0 0x7f90f0100150 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f90dc00e9c0 tx=0x7f90dc00edc0 comp rx=0 tx=0).stop 2024-03-18T16:11:14.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.236+0000 7f90f6600640 1 -- 172.21.15.39:0/2385217961 shutdown_connections 2024-03-18T16:11:14.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.236+0000 7f90f6600640 1 --2- 172.21.15.39:0/2385217961 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f90f0104ad0 0x7f90f0100150 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:14.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.236+0000 7f90f6600640 1 --2- 172.21.15.39:0/2385217961 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f90d80a2800 0x7f90d80a4cc0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:14.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.236+0000 7f90f6600640 1 --2- 172.21.15.39:0/2385217961 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f90f0107400 0x7f90f0100690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:14.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.236+0000 7f90f6600640 1 --2- 172.21.15.39:0/2385217961 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f90f01021a0 0x7f90f0101b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:14.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.236+0000 7f90f6600640 1 -- 172.21.15.39:0/2385217961 >> 172.21.15.39:0/2385217961 conn(0x7f90f00fbb50 msgr2=0x7f90f0102b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:14.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.236+0000 7f90f6600640 1 -- 172.21.15.39:0/2385217961 shutdown_connections 2024-03-18T16:11:14.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.236+0000 7f90f6600640 1 -- 172.21.15.39:0/2385217961 wait complete. 2024-03-18T16:11:14.259 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T16:11:14.259 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T16:11:14.320 INFO:tasks.workunit.client.0.smithi039.stdout:/bin/sh ./config.status --recheck 2024-03-18T16:11:14.459 INFO:tasks.workunit.client.0.smithi039.stdout:running CONFIG_SHELL=/bin/sh /bin/sh ./configure --no-create --no-recursion 2024-03-18T16:11:14.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.461+0000 7f9cbe000640 1 -- 172.21.15.39:0/1897840325 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9cb8104c10 msgr2=0x7f9cb8105090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:14.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.461+0000 7f9cbe000640 1 --2- 172.21.15.39:0/1897840325 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9cb8104c10 0x7f9cb8105090 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f9ca0009860 tx=0x7f9ca002bb70 comp rx=0 tx=0).stop 2024-03-18T16:11:14.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.461+0000 7f9cbe000640 1 -- 172.21.15.39:0/1897840325 shutdown_connections 2024-03-18T16:11:14.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.461+0000 7f9cbe000640 1 --2- 172.21.15.39:0/1897840325 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9cb81055d0 0x7f9cb810be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:14.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.461+0000 7f9cbe000640 1 --2- 172.21.15.39:0/1897840325 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9cb8104c10 0x7f9cb8105090 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:14.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.461+0000 7f9cbe000640 1 --2- 172.21.15.39:0/1897840325 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9cb8103a20 0x7f9cb8103e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:14.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.461+0000 7f9cbe000640 1 -- 172.21.15.39:0/1897840325 >> 172.21.15.39:0/1897840325 conn(0x7f9cb80fef90 msgr2=0x7f9cb81013d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:14.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.462+0000 7f9cbe000640 1 -- 172.21.15.39:0/1897840325 shutdown_connections 2024-03-18T16:11:14.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.462+0000 7f9cbe000640 1 -- 172.21.15.39:0/1897840325 wait complete. 2024-03-18T16:11:14.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.463+0000 7f9cbe000640 1 Processor -- start 2024-03-18T16:11:14.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.463+0000 7f9cbe000640 1 -- start start 2024-03-18T16:11:14.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.463+0000 7f9cbe000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9cb8103a20 0x7f9cb8199d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:14.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.463+0000 7f9cbe000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9cb8104c10 0x7f9cb819a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:14.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.463+0000 7f9cbe000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9cb81055d0 0x7f9cb81a1280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:14.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.463+0000 7f9cbe000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f9cb810f680 con 0x7f9cb81055d0 2024-03-18T16:11:14.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.463+0000 7f9cbe000640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f9cb810f500 con 0x7f9cb8104c10 2024-03-18T16:11:14.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.463+0000 7f9cbe000640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f9cb810f800 con 0x7f9cb8103a20 2024-03-18T16:11:14.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.463+0000 7f9cafe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9cb8104c10 0x7f9cb819a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:14.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.463+0000 7f9cb7400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9cb8103a20 0x7f9cb8199d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:14.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.463+0000 7f9cafe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9cb8104c10 0x7f9cb819a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:54824/0 (socket says 172.21.15.39:54824) 2024-03-18T16:11:14.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.463+0000 7f9cb7400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9cb8103a20 0x7f9cb8199d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:40142/0 (socket says 172.21.15.39:40142) 2024-03-18T16:11:14.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.463+0000 7f9cafe00640 1 -- 172.21.15.39:0/1005720201 learned_addr learned my addr 172.21.15.39:0/1005720201 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:11:14.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.463+0000 7f9cb7e00640 1 --2- 172.21.15.39:0/1005720201 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9cb81055d0 0x7f9cb81a1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:14.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.464+0000 7f9cafe00640 1 -- 172.21.15.39:0/1005720201 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9cb8103a20 msgr2=0x7f9cb8199d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:14.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.464+0000 7f9cafe00640 1 --2- 172.21.15.39:0/1005720201 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9cb8103a20 0x7f9cb8199d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:14.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.464+0000 7f9cafe00640 1 -- 172.21.15.39:0/1005720201 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9cb81055d0 msgr2=0x7f9cb81a1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:14.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.464+0000 7f9cafe00640 1 --2- 172.21.15.39:0/1005720201 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9cb81055d0 0x7f9cb81a1280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:14.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.464+0000 7f9cafe00640 1 -- 172.21.15.39:0/1005720201 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cb81a1920 con 0x7f9cb8104c10 2024-03-18T16:11:14.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.464+0000 7f9cafe00640 1 --2- 172.21.15.39:0/1005720201 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9cb8104c10 0x7f9cb819a250 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f9ca0005940 tx=0x7f9ca0032470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:14.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.465+0000 7f9cb4c00640 1 -- 172.21.15.39:0/1005720201 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ca0041020 con 0x7f9cb8104c10 2024-03-18T16:11:14.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.465+0000 7f9cb4c00640 1 -- 172.21.15.39:0/1005720201 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ca002c9f0 con 0x7f9cb8104c10 2024-03-18T16:11:14.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.465+0000 7f9cbe000640 1 -- 172.21.15.39:0/1005720201 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9cb81a1bb0 con 0x7f9cb8104c10 2024-03-18T16:11:14.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.465+0000 7f9cb4c00640 1 -- 172.21.15.39:0/1005720201 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ca0039040 con 0x7f9cb8104c10 2024-03-18T16:11:14.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.465+0000 7f9cbe000640 1 -- 172.21.15.39:0/1005720201 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9cb81a20a0 con 0x7f9cb8104c10 2024-03-18T16:11:14.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.468+0000 7f9cc0da1740 1 Processor -- start 2024-03-18T16:11:14.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.468+0000 7f9cc0da1740 1 -- start start 2024-03-18T16:11:14.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.469+0000 7f9cc0da1740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55db278d2810 0x55db278d2c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:14.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.469+0000 7f9cc0da1740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55db278d3200 0x55db278d7d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:14.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.469+0000 7f9cc0da1740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55db278d8370 0x55db278da800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:14.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.469+0000 7f9cc0da1740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55db278371f0 con 0x55db278d2810 2024-03-18T16:11:14.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.469+0000 7f9cc0da1740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55db277dfa80 con 0x55db278d3200 2024-03-18T16:11:14.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.469+0000 7f9cc0da1740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55db27833290 con 0x55db278d8370 2024-03-18T16:11:14.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.469+0000 7f9cb7e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55db278d2810 0x55db278d2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:14.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.469+0000 7f9cb7400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55db278d3200 0x55db278d7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:14.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.469+0000 7f9cb7400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55db278d3200 0x55db278d7d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:54838/0 (socket says 172.21.15.39:54838) 2024-03-18T16:11:14.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.469+0000 7f9cb7e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55db278d2810 0x55db278d2c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:34894/0 (socket says 172.21.15.39:34894) 2024-03-18T16:11:14.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.469+0000 7f9cb7400640 1 -- 172.21.15.39:0/1404620008 learned_addr learned my addr 172.21.15.39:0/1404620008 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:11:14.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.469+0000 7f9cb7400640 1 -- 172.21.15.39:0/1404620008 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55db278d8370 msgr2=0x55db278da800 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:11:14.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.469+0000 7f9cb7400640 1 --2- 172.21.15.39:0/1404620008 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55db278d8370 0x55db278da800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:14.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.469+0000 7f9cb4c00640 1 -- 172.21.15.39:0/1005720201 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f9ca0044020 con 0x7f9cb8104c10 2024-03-18T16:11:14.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.469+0000 7f9cb7400640 1 -- 172.21.15.39:0/1404620008 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55db278d2810 msgr2=0x55db278d2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:14.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.469+0000 7f9cb7400640 1 --2- 172.21.15.39:0/1404620008 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55db278d2810 0x55db278d2c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:14.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.469+0000 7f9cb7400640 1 -- 172.21.15.39:0/1404620008 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55db276157d0 con 0x55db278d3200 2024-03-18T16:11:14.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.469+0000 7f9cb7e00640 1 --2- 172.21.15.39:0/1404620008 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55db278d2810 0x55db278d2c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:11:14.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.470+0000 7f9cb7400640 1 --2- 172.21.15.39:0/1404620008 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55db278d3200 0x55db278d7d00 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f9ca8009c60 tx=0x7f9ca8011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f64fa7fef601db80 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:14.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.470+0000 7f9ca7e00640 1 -- 172.21.15.39:0/1404620008 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ca8004220 con 0x55db278d3200 2024-03-18T16:11:14.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.470+0000 7f9ca7e00640 1 -- 172.21.15.39:0/1404620008 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ca80043c0 con 0x55db278d3200 2024-03-18T16:11:14.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.470+0000 7f9ca7e00640 1 -- 172.21.15.39:0/1404620008 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ca8004e10 con 0x55db278d3200 2024-03-18T16:11:14.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.471+0000 7f9cc0da1740 1 -- 172.21.15.39:0/1404620008 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55db278d3200 msgr2=0x55db278d7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:14.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.471+0000 7f9cc0da1740 1 --2- 172.21.15.39:0/1404620008 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55db278d3200 0x55db278d7d00 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f9ca8009c60 tx=0x7f9ca8011810 comp rx=0 tx=0).stop 2024-03-18T16:11:14.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.471+0000 7f9cb4c00640 1 --2- 172.21.15.39:0/1005720201 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f9c9c0a2800 0x7f9c9c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:14.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.471+0000 7f9cb4c00640 1 -- 172.21.15.39:0/1005720201 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f9ca00f0820 con 0x7f9cb8104c10 2024-03-18T16:11:14.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.471+0000 7f9cc0da1740 1 -- 172.21.15.39:0/1404620008 shutdown_connections 2024-03-18T16:11:14.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.471+0000 7f9cc0da1740 1 --2- 172.21.15.39:0/1404620008 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55db278d3200 0x55db278d7d00 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:14.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.471+0000 7f9cc0da1740 1 --2- 172.21.15.39:0/1404620008 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55db278d8370 0x55db278da800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:14.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.471+0000 7f9cc0da1740 1 --2- 172.21.15.39:0/1404620008 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55db278d2810 0x55db278d2c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:14.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.471+0000 7f9cc0da1740 1 -- 172.21.15.39:0/1404620008 >> 172.21.15.39:0/1404620008 conn(0x55db278cdd80 msgr2=0x55db278ce180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:14.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.471+0000 7f9cb7e00640 1 --2- 172.21.15.39:0/1005720201 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f9c9c0a2800 0x7f9c9c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:14.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.471+0000 7f9cc0da1740 1 -- 172.21.15.39:0/1404620008 shutdown_connections 2024-03-18T16:11:14.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.471+0000 7f9cc0da1740 1 -- 172.21.15.39:0/1404620008 wait complete. 2024-03-18T16:11:14.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.472+0000 7f9cc0da1740 1 -- start start 2024-03-18T16:11:14.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.472+0000 7f9cb7e00640 1 --2- 172.21.15.39:0/1005720201 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f9c9c0a2800 0x7f9c9c0a4cc0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x55db278d3a60 tx=0x7f9cb0010040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:14.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.472+0000 7f9cc0da1740 1 Processor -- start 2024-03-18T16:11:14.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.473+0000 7f9cc0da1740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55db278d2810 0x55db278dcdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:14.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.473+0000 7f9cc0da1740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55db278d3200 0x55db278dd310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:14.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.473+0000 7f9cc0da1740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55db278d8370 0x55db278e4310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:14.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.473+0000 7f9cc0da1740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55db278db320 con 0x55db278d2810 2024-03-18T16:11:14.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.473+0000 7f9cc0da1740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55db277f2ed0 con 0x55db278d3200 2024-03-18T16:11:14.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.473+0000 7f9cc0da1740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55db278db4a0 con 0x55db278d8370 2024-03-18T16:11:14.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.473+0000 7f9cafe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55db278d3200 0x55db278dd310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:14.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.473+0000 7f9cc0da1740 10 client.0 _ll_register_callbacks cb 0x55db2782e220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:11:14.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.473+0000 7f9cafe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55db278d3200 0x55db278dd310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:54846/0 (socket says 172.21.15.39:54846) 2024-03-18T16:11:14.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.473+0000 7f9cafe00640 1 -- 172.21.15.39:0/1755090515 learned_addr learned my addr 172.21.15.39:0/1755090515 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:11:14.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.473+0000 7f9cb7400640 1 --2- 172.21.15.39:0/1755090515 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55db278d2810 0x55db278dcdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:14.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.473+0000 7f9cb7e00640 1 --2- 172.21.15.39:0/1755090515 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55db278d8370 0x55db278e4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:14.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.473+0000 7f9cafe00640 1 -- 172.21.15.39:0/1755090515 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55db278d8370 msgr2=0x55db278e4310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:14.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.473+0000 7f9cafe00640 1 --2- 172.21.15.39:0/1755090515 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55db278d8370 0x55db278e4310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:14.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.473+0000 7f9cafe00640 1 -- 172.21.15.39:0/1755090515 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55db278d2810 msgr2=0x55db278dcdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:14.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.473+0000 7f9cafe00640 1 --2- 172.21.15.39:0/1755090515 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55db278d2810 0x55db278dcdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:14.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.473+0000 7f9cafe00640 1 -- 172.21.15.39:0/1755090515 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55db2754b3f0 con 0x55db278d3200 2024-03-18T16:11:14.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.474+0000 7f9cb7e00640 1 --2- 172.21.15.39:0/1755090515 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55db278d8370 0x55db278e4310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:11:14.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.474+0000 7f9cafe00640 1 --2- 172.21.15.39:0/1755090515 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55db278d3200 0x55db278dd310 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f9ca0034b10 tx=0x7f9ca00bcc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec4ecd80a1f4f3d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:14.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.474+0000 7f9ca4c00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:11:14.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.474+0000 7f9ca4c00640 1 -- 172.21.15.39:0/1755090515 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ca00bd4d0 con 0x55db278d3200 2024-03-18T16:11:14.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.474+0000 7f9ca4c00640 1 -- 172.21.15.39:0/1755090515 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f9ca00c50e0 con 0x55db278d3200 2024-03-18T16:11:14.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.474+0000 7f9cc0da1740 1 -- 172.21.15.39:0/1755090515 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9ca80043c0 con 0x55db278d3200 2024-03-18T16:11:14.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.474+0000 7f9ca4c00640 1 -- 172.21.15.39:0/1755090515 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9ca00bcdd0 con 0x55db278d3200 2024-03-18T16:11:14.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.474+0000 7f9cc0da1740 10 client.25877 fetch_fsmap learned FSMap version 28 2024-03-18T16:11:14.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.474+0000 7f9cc0da1740 1 -- 172.21.15.39:0/1755090515 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55db2750b0f0 con 0x55db278d3200 2024-03-18T16:11:14.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.475+0000 7f9ca4c00640 1 -- 172.21.15.39:0/1755090515 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ca80043c0 con 0x55db278d3200 2024-03-18T16:11:14.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.475+0000 7f9ca4c00640 1 -- 172.21.15.39:0/1755090515 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f9ca003c030 con 0x55db278d3200 2024-03-18T16:11:14.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.475+0000 7f9cc0da1740 10 client.25877 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T16:11:14.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.475+0000 7f9cc0da1740 10 client.25877 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:11:14.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.475+0000 7f9cc0da1740 20 client.25877 populate_metadata read hostname 'smithi039' 2024-03-18T16:11:14.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.475+0000 7f9cc0da1740 1 --2- 172.21.15.39:0/1755090515 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55db278e8bb0 0x55db278eafe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:14.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.475+0000 7f9cc0da1740 4 client.25877 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:11:14.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.475+0000 7f9cc0da1740 1 -- 172.21.15.39:0/1755090515 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55db2784abe0 con 0x55db278e8bb0 2024-03-18T16:11:14.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.475+0000 7f9cb7400640 1 --2- 172.21.15.39:0/1755090515 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55db278e8bb0 0x55db278eafe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:14.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.476+0000 7f9cb7400640 1 --2- 172.21.15.39:0/1755090515 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55db278e8bb0 0x55db278eafe0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=13bd22537d3cfc89 server_cookie=8f73e6bec696cee7 in_seq=0 out_seq=0 2024-03-18T16:11:14.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.476+0000 7f9ca4c00640 10 client.25877 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:11:14.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.477+0000 7f9ca4c00640 1 -- 172.21.15.39:0/1755090515 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55db2784abe0 con 0x55db278e8bb0 2024-03-18T16:11:14.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.477+0000 7f9ca4c00640 10 client.25877 handle_command_reply: tid=0 2024-03-18T16:11:14.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.477+0000 7f9cc0da1740 1 client.25877 shutdown 2024-03-18T16:11:14.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.477+0000 7f9cc0da1740 1 -- 172.21.15.39:0/1755090515 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55db278d3200 msgr2=0x55db278dd310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:14.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.477+0000 7f9cc0da1740 1 --2- 172.21.15.39:0/1755090515 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55db278d3200 0x55db278dd310 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f9ca0034b10 tx=0x7f9ca00bcc60 comp rx=0 tx=0).stop 2024-03-18T16:11:14.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.477+0000 7f9cc0da1740 1 -- 172.21.15.39:0/1755090515 shutdown_connections 2024-03-18T16:11:14.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.477+0000 7f9cc0da1740 1 --2- 172.21.15.39:0/1755090515 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55db278e8bb0 0x55db278eafe0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:14.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.477+0000 7f9cc0da1740 1 --2- 172.21.15.39:0/1755090515 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55db278d3200 0x55db278dd310 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:14.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.477+0000 7f9cc0da1740 1 --2- 172.21.15.39:0/1755090515 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55db278d8370 0x55db278e4310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:14.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.477+0000 7f9ca4c00640 0 client.25877 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:11:14.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.477+0000 7f9cc0da1740 1 --2- 172.21.15.39:0/1755090515 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55db278d2810 0x55db278dcdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:14.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.477+0000 7f9cc0da1740 1 -- 172.21.15.39:0/1755090515 >> 172.21.15.39:0/1755090515 conn(0x55db278cdd80 msgr2=0x55db278ccec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:14.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.478+0000 7f9cc0da1740 1 -- 172.21.15.39:0/1755090515 shutdown_connections 2024-03-18T16:11:14.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.478+0000 7f9cc0da1740 1 -- 172.21.15.39:0/1755090515 wait complete. 2024-03-18T16:11:14.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.478+0000 7f9cc0da1740 20 client.25877 trim_cache size 0 max 16384 2024-03-18T16:11:14.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.502+0000 7f9cc0da1740 1 Processor -- start 2024-03-18T16:11:14.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.502+0000 7f9cc0da1740 1 -- start start 2024-03-18T16:11:14.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.502+0000 7f9cc0da1740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55db278d2810 0x55db278ccd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:14.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.502+0000 7f9cc0da1740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55db278d3200 0x55db278e42d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:14.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.502+0000 7f9cc0da1740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55db278d8370 0x55db278db8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:14.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.502+0000 7f9cc0da1740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f9cb81a80e0 con 0x55db278d8370 2024-03-18T16:11:14.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.502+0000 7f9cc0da1740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f9cb81a9c20 con 0x55db278d2810 2024-03-18T16:11:14.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.502+0000 7f9cc0da1740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f9cb81a87b0 con 0x55db278d3200 2024-03-18T16:11:14.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.503+0000 7f9cafe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55db278d3200 0x55db278e42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:14.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.503+0000 7f9cb7400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55db278d2810 0x55db278ccd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:14.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.503+0000 7f9cb7400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55db278d2810 0x55db278ccd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:54852/0 (socket says 172.21.15.39:54852) 2024-03-18T16:11:14.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.503+0000 7f9cafe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55db278d3200 0x55db278e42d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:40152/0 (socket says 172.21.15.39:40152) 2024-03-18T16:11:14.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.503+0000 7f9cb7400640 1 -- 172.21.15.39:0/3295077152 learned_addr learned my addr 172.21.15.39:0/3295077152 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:11:14.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.503+0000 7f9cb7e00640 1 --2- 172.21.15.39:0/3295077152 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55db278d8370 0x55db278db8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:14.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.503+0000 7f9cb7400640 1 -- 172.21.15.39:0/3295077152 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55db278d3200 msgr2=0x55db278e42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:14.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.503+0000 7f9cb7400640 1 --2- 172.21.15.39:0/3295077152 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55db278d3200 0x55db278e42d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:14.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.503+0000 7f9cb7400640 1 -- 172.21.15.39:0/3295077152 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55db278d8370 msgr2=0x55db278db8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:14.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.503+0000 7f9cb7400640 1 --2- 172.21.15.39:0/3295077152 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55db278d8370 0x55db278db8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:14.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.503+0000 7f9cb7400640 1 -- 172.21.15.39:0/3295077152 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55db278d79f0 con 0x55db278d2810 2024-03-18T16:11:14.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.503+0000 7f9cafe00640 1 --2- 172.21.15.39:0/3295077152 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55db278d3200 0x55db278e42d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:11:14.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.503+0000 7f9cb7400640 1 --2- 172.21.15.39:0/3295077152 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55db278d2810 0x55db278ccd40 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f9ca801b040 tx=0x7f9ca8028040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd48c7b7753f70e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:14.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.504+0000 7f9ca7400640 1 -- 172.21.15.39:0/3295077152 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ca801d040 con 0x55db278d2810 2024-03-18T16:11:14.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.504+0000 7f9ca7400640 1 -- 172.21.15.39:0/3295077152 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ca8032e50 con 0x55db278d2810 2024-03-18T16:11:14.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.504+0000 7f9ca7400640 1 -- 172.21.15.39:0/3295077152 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ca8019740 con 0x55db278d2810 2024-03-18T16:11:14.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.505+0000 7f9cc0da1740 1 -- 172.21.15.39:0/3295077152 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55db278d2810 msgr2=0x55db278ccd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:14.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.505+0000 7f9cc0da1740 1 --2- 172.21.15.39:0/3295077152 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55db278d2810 0x55db278ccd40 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f9ca801b040 tx=0x7f9ca8028040 comp rx=0 tx=0).stop 2024-03-18T16:11:14.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.505+0000 7f9cc0da1740 1 -- 172.21.15.39:0/3295077152 shutdown_connections 2024-03-18T16:11:14.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.505+0000 7f9cc0da1740 1 --2- 172.21.15.39:0/3295077152 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55db278d8370 0x55db278db8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:14.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.505+0000 7f9cc0da1740 1 --2- 172.21.15.39:0/3295077152 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55db278d3200 0x55db278e42d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:14.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.505+0000 7f9cc0da1740 1 --2- 172.21.15.39:0/3295077152 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55db278d2810 0x55db278ccd40 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:14.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.505+0000 7f9cc0da1740 1 -- 172.21.15.39:0/3295077152 >> 172.21.15.39:0/3295077152 conn(0x55db278cdd80 msgr2=0x55db278cc630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:14.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.505+0000 7f9cc0da1740 1 -- 172.21.15.39:0/3295077152 shutdown_connections 2024-03-18T16:11:14.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.505+0000 7f9cc0da1740 1 -- 172.21.15.39:0/3295077152 wait complete. 2024-03-18T16:11:14.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.506+0000 7f9cc0da1740 1 -- start start 2024-03-18T16:11:14.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.506+0000 7f9cc0da1740 1 Processor -- start 2024-03-18T16:11:14.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.506+0000 7f9cc0da1740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55db278d2810 0x55db278e1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:14.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.506+0000 7f9cc0da1740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55db278d3200 0x55db278e2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:14.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.507+0000 7f9cc0da1740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55db278d8370 0x55db278e3230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:14.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.507+0000 7f9cc0da1740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f9cb81a9550 con 0x55db278d8370 2024-03-18T16:11:14.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.507+0000 7f9cc0da1740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f9cb81a8e80 con 0x55db278d3200 2024-03-18T16:11:14.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.507+0000 7f9cc0da1740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f9cb81a60c0 con 0x55db278d2810 2024-03-18T16:11:14.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.507+0000 7f9cc0da1740 10 client.0 _ll_register_callbacks cb 0x7f9ca00097b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:11:14.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.507+0000 7f9cafe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55db278d3200 0x55db278e2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:14.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.507+0000 7f9cb7400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55db278d2810 0x55db278e1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:14.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.507+0000 7f9cafe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55db278d3200 0x55db278e2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:54854/0 (socket says 172.21.15.39:54854) 2024-03-18T16:11:14.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.507+0000 7f9cb7400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55db278d2810 0x55db278e1ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:40154/0 (socket says 172.21.15.39:40154) 2024-03-18T16:11:14.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.507+0000 7f9cafe00640 1 -- 172.21.15.39:0/1681037167 learned_addr learned my addr 172.21.15.39:0/1681037167 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:11:14.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.507+0000 7f9cb7e00640 1 --2- 172.21.15.39:0/1681037167 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55db278d8370 0x55db278e3230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:14.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.507+0000 7f9cafe00640 1 -- 172.21.15.39:0/1681037167 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55db278d2810 msgr2=0x55db278e1ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:14.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.507+0000 7f9cafe00640 1 --2- 172.21.15.39:0/1681037167 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55db278d2810 0x55db278e1ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:14.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.507+0000 7f9cafe00640 1 -- 172.21.15.39:0/1681037167 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55db278d8370 msgr2=0x55db278e3230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:14.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.507+0000 7f9cafe00640 1 --2- 172.21.15.39:0/1681037167 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55db278d8370 0x55db278e3230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:14.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.507+0000 7f9cafe00640 1 -- 172.21.15.39:0/1681037167 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55db278e3820 con 0x55db278d3200 2024-03-18T16:11:14.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.507+0000 7f9cb7400640 1 --2- 172.21.15.39:0/1681037167 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55db278d2810 0x55db278e1ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:11:14.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.508+0000 7f9cafe00640 1 --2- 172.21.15.39:0/1681037167 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55db278d3200 0x55db278e2220 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f9ca00349e0 tx=0x7f9ca0032f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c2522aa4b795d8dd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:14.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.508+0000 7f9ca4c00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:11:14.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.508+0000 7f9ca4c00640 1 -- 172.21.15.39:0/1681037167 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ca00bc730 con 0x55db278d3200 2024-03-18T16:11:14.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.508+0000 7f9ca4c00640 1 -- 172.21.15.39:0/1681037167 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f9ca009bdc0 con 0x55db278d3200 2024-03-18T16:11:14.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.508+0000 7f9cc0da1740 1 -- 172.21.15.39:0/1681037167 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9ca8032e50 con 0x55db278d3200 2024-03-18T16:11:14.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.508+0000 7f9ca4c00640 1 -- 172.21.15.39:0/1681037167 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f9ca00ca050 con 0x55db278d3200 2024-03-18T16:11:14.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.508+0000 7f9cc0da1740 10 client.25883 fetch_fsmap learned FSMap version 28 2024-03-18T16:11:14.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.508+0000 7f9cc0da1740 1 -- 172.21.15.39:0/1681037167 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55db2794f210 con 0x55db278d3200 2024-03-18T16:11:14.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.508+0000 7f9ca4c00640 1 -- 172.21.15.39:0/1681037167 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ca8032e50 con 0x55db278d3200 2024-03-18T16:11:14.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.508+0000 7f9ca4c00640 1 -- 172.21.15.39:0/1681037167 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f9ca00bbc90 con 0x55db278d3200 2024-03-18T16:11:14.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.509+0000 7f9cc0da1740 10 client.25883 fetch_fsmap finished waiting for FSMap version 28 2024-03-18T16:11:14.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.509+0000 7f9cc0da1740 10 client.25883 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:11:14.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.509+0000 7f9cc0da1740 20 client.25883 populate_metadata read hostname 'smithi039' 2024-03-18T16:11:14.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.509+0000 7f9cc0da1740 1 --2- 172.21.15.39:0/1681037167 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55db2794f800 0x55db27951c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:14.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.509+0000 7f9cc0da1740 4 client.25883 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T16:11:14.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.509+0000 7f9cc0da1740 1 -- 172.21.15.39:0/1681037167 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55db27844730 con 0x55db2794f800 2024-03-18T16:11:14.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.509+0000 7f9cb7400640 1 --2- 172.21.15.39:0/1681037167 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55db2794f800 0x55db27951c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:14.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.509+0000 7f9cb7400640 1 --2- 172.21.15.39:0/1681037167 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55db2794f800 0x55db27951c20 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c574dcb1e0b9fad6 server_cookie=c9b78f55bd6b8e9d in_seq=0 out_seq=0 2024-03-18T16:11:14.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.509+0000 7f9ca4c00640 10 client.25883 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:11:14.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.510+0000 7f9ca4c00640 1 -- 172.21.15.39:0/1681037167 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55db27844730 con 0x55db2794f800 2024-03-18T16:11:14.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.510+0000 7f9ca4c00640 10 client.25883 handle_command_reply: tid=0 2024-03-18T16:11:14.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.510+0000 7f9cc0da1740 1 client.25883 shutdown 2024-03-18T16:11:14.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.510+0000 7f9cc0da1740 1 -- 172.21.15.39:0/1681037167 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55db278d3200 msgr2=0x55db278e2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:14.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.510+0000 7f9cc0da1740 1 --2- 172.21.15.39:0/1681037167 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55db278d3200 0x55db278e2220 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f9ca00349e0 tx=0x7f9ca0032f90 comp rx=0 tx=0).stop 2024-03-18T16:11:14.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.510+0000 7f9cc0da1740 1 -- 172.21.15.39:0/1681037167 shutdown_connections 2024-03-18T16:11:14.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.510+0000 7f9cc0da1740 1 --2- 172.21.15.39:0/1681037167 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55db2794f800 0x55db27951c20 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:14.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.510+0000 7f9cc0da1740 1 --2- 172.21.15.39:0/1681037167 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55db278d3200 0x55db278e2220 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:14.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.510+0000 7f9cc0da1740 1 --2- 172.21.15.39:0/1681037167 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55db278d8370 0x55db278e3230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:14.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.510+0000 7f9cc0da1740 1 --2- 172.21.15.39:0/1681037167 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55db278d2810 0x55db278e1ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:14.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.510+0000 7f9ca4c00640 0 client.25883 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:11:14.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.510+0000 7f9cc0da1740 1 -- 172.21.15.39:0/1681037167 >> 172.21.15.39:0/1681037167 conn(0x55db278cdd80 msgr2=0x55db278d0fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:14.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.510+0000 7f9cc0da1740 1 -- 172.21.15.39:0/1681037167 shutdown_connections 2024-03-18T16:11:14.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.510+0000 7f9cc0da1740 1 -- 172.21.15.39:0/1681037167 wait complete. 2024-03-18T16:11:14.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.510+0000 7f9cc0da1740 20 client.25883 trim_cache size 0 max 16384 2024-03-18T16:11:14.516 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-03-18T16:11:14.516 INFO:teuthology.orchestra.run.smithi039.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-18T16:11:14.516 INFO:teuthology.orchestra.run.smithi039.stdout: "scrubs": { 2024-03-18T16:11:14.517 INFO:teuthology.orchestra.run.smithi039.stdout: "ed05154a-4a21-4471-b382-740ffddd52ab": { 2024-03-18T16:11:14.517 INFO:teuthology.orchestra.run.smithi039.stdout: "path": "/", 2024-03-18T16:11:14.517 INFO:teuthology.orchestra.run.smithi039.stdout: "tag": "ed05154a-4a21-4471-b382-740ffddd52ab", 2024-03-18T16:11:14.517 INFO:teuthology.orchestra.run.smithi039.stdout: "options": "recursive,force" 2024-03-18T16:11:14.517 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:11:14.517 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:11:14.517 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-03-18T16:11:14.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.511+0000 7f9cbe000640 1 -- 172.21.15.39:0/1005720201 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f9c9c0a2800 msgr2=0x7f9c9c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:14.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.511+0000 7f9cbe000640 1 --2- 172.21.15.39:0/1005720201 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f9c9c0a2800 0x7f9c9c0a4cc0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x55db278d3a60 tx=0x7f9cb0010040 comp rx=0 tx=0).stop 2024-03-18T16:11:14.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.511+0000 7f9cbe000640 1 -- 172.21.15.39:0/1005720201 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9cb8104c10 msgr2=0x7f9cb819a250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:14.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.511+0000 7f9cbe000640 1 --2- 172.21.15.39:0/1005720201 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9cb8104c10 0x7f9cb819a250 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f9ca0005940 tx=0x7f9ca0032470 comp rx=0 tx=0).stop 2024-03-18T16:11:14.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.511+0000 7f9cbe000640 1 -- 172.21.15.39:0/1005720201 shutdown_connections 2024-03-18T16:11:14.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.511+0000 7f9cbe000640 1 --2- 172.21.15.39:0/1005720201 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9cb8104c10 0x7f9cb819a250 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:14.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.511+0000 7f9cbe000640 1 --2- 172.21.15.39:0/1005720201 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f9c9c0a2800 0x7f9c9c0a4cc0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:14.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.511+0000 7f9cbe000640 1 --2- 172.21.15.39:0/1005720201 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9cb81055d0 0x7f9cb81a1280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:14.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.511+0000 7f9cbe000640 1 --2- 172.21.15.39:0/1005720201 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9cb8103a20 0x7f9cb8199d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:14.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.511+0000 7f9cbe000640 1 -- 172.21.15.39:0/1005720201 >> 172.21.15.39:0/1005720201 conn(0x7f9cb80fef90 msgr2=0x7f9cb8107f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:14.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.511+0000 7f9cbe000640 1 -- 172.21.15.39:0/1005720201 shutdown_connections 2024-03-18T16:11:14.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:14.511+0000 7f9cbe000640 1 -- 172.21.15.39:0/1005720201 wait complete. 2024-03-18T16:11:14.530 INFO:tasks.cephfs.filesystem:scrub status for tag:ed05154a-4a21-4471-b382-740ffddd52ab - {'path': '/', 'tag': 'ed05154a-4a21-4471-b382-740ffddd52ab', 'options': 'recursive,force'} 2024-03-18T16:11:14.677 INFO:tasks.workunit.client.0.smithi039.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2024-03-18T16:11:15.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:11:14 smithi196 ceph-mon[36307]: pgmap v457: 129 pgs: 129 active+clean; 3.3 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 261 KiB/s wr, 39 op/s 2024-03-18T16:11:15.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:11:14 smithi196 ceph-mon[36307]: scrub queued for path: / 2024-03-18T16:11:15.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:11:14 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:11:15.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:11:14 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:11:15.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:11:14 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:11:15.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:11:14 smithi194 ceph-mon[36200]: pgmap v457: 129 pgs: 129 active+clean; 3.3 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 261 KiB/s wr, 39 op/s 2024-03-18T16:11:15.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:11:14 smithi194 ceph-mon[36200]: scrub queued for path: / 2024-03-18T16:11:15.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:11:14 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:11:15.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:11:14 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:11:15.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:11:14 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:11:15.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:11:14 smithi039 ceph-mon[31721]: pgmap v457: 129 pgs: 129 active+clean; 3.3 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 261 KiB/s wr, 39 op/s 2024-03-18T16:11:15.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:11:14 smithi039 ceph-mon[31721]: scrub queued for path: / 2024-03-18T16:11:15.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:11:14 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:11:15.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:11:14 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:11:15.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:11:14 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:11:15.698 INFO:tasks.workunit.client.0.smithi039.stdout:checking whether build environment is sane... yes 2024-03-18T16:11:15.714 INFO:tasks.workunit.client.0.smithi039.stdout:checking for a thread-safe mkdir -p... /usr/bin/mkdir -p 2024-03-18T16:11:15.715 INFO:tasks.workunit.client.0.smithi039.stdout:checking for gawk... gawk 2024-03-18T16:11:15.744 INFO:tasks.workunit.client.0.smithi039.stdout:checking whether make sets $(MAKE)... yes 2024-03-18T16:11:15.832 INFO:tasks.workunit.client.0.smithi039.stdout:checking build system type... x86_64-unknown-linux-gnu 2024-03-18T16:11:15.832 INFO:tasks.workunit.client.0.smithi039.stdout:checking host system type... x86_64-unknown-linux-gnu 2024-03-18T16:11:15.854 INFO:tasks.workunit.client.0.smithi039.stdout:checking for style of include used by make... GNU 2024-03-18T16:11:15.859 INFO:tasks.workunit.client.0.smithi039.stdout:checking for gcc... gcc 2024-03-18T16:11:16.020 INFO:tasks.workunit.client.0.smithi039.stdout:checking whether the C compiler works... yes 2024-03-18T16:11:16.020 INFO:tasks.workunit.client.0.smithi039.stdout:checking for C compiler default output file name... a.out 2024-03-18T16:11:16.099 INFO:tasks.workunit.client.0.smithi039.stdout:checking for suffix of executables... 2024-03-18T16:11:16.180 INFO:tasks.workunit.client.0.smithi039.stdout:checking whether we are cross compiling... no 2024-03-18T16:11:16.256 INFO:tasks.workunit.client.0.smithi039.stdout:checking for suffix of object files... o 2024-03-18T16:11:16.313 INFO:tasks.workunit.client.0.smithi039.stdout:checking whether we are using the GNU C compiler... yes 2024-03-18T16:11:16.369 INFO:tasks.workunit.client.0.smithi039.stdout:checking whether gcc accepts -g... yes 2024-03-18T16:11:16.476 INFO:tasks.workunit.client.0.smithi039.stdout:checking for gcc option to accept ISO C89... none needed 2024-03-18T16:11:16.667 INFO:tasks.workunit.client.0.smithi039.stdout:checking dependency style of gcc... gcc3 2024-03-18T16:11:16.773 INFO:tasks.workunit.client.0.smithi039.stdout:checking how to run the C preprocessor... gcc -E 2024-03-18T16:11:16.888 INFO:tasks.workunit.client.0.smithi039.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2024-03-18T16:11:16.894 INFO:tasks.workunit.client.0.smithi039.stdout:checking for egrep... /usr/bin/grep -E 2024-03-18T16:11:17.140 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:11:16 smithi039 ceph-mon[31721]: pgmap v458: 129 pgs: 129 active+clean; 4.6 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 374 KiB/s wr, 46 op/s 2024-03-18T16:11:17.158 INFO:tasks.workunit.client.0.smithi039.stdout:checking for ANSI C header files... yes 2024-03-18T16:11:17.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:11:16 smithi196 ceph-mon[36307]: pgmap v458: 129 pgs: 129 active+clean; 4.6 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 374 KiB/s wr, 46 op/s 2024-03-18T16:11:17.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:11:16 smithi194 ceph-mon[36200]: pgmap v458: 129 pgs: 129 active+clean; 4.6 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 374 KiB/s wr, 46 op/s 2024-03-18T16:11:17.241 INFO:tasks.workunit.client.0.smithi039.stdout:checking for sys/types.h... yes 2024-03-18T16:11:17.365 INFO:tasks.workunit.client.0.smithi039.stdout:checking for sys/stat.h... yes 2024-03-18T16:11:17.463 INFO:tasks.workunit.client.0.smithi039.stdout:checking for stdlib.h... yes 2024-03-18T16:11:17.549 INFO:tasks.workunit.client.0.smithi039.stdout:checking for string.h... yes 2024-03-18T16:11:17.647 INFO:tasks.workunit.client.0.smithi039.stdout:checking for memory.h... yes 2024-03-18T16:11:17.745 INFO:tasks.workunit.client.0.smithi039.stdout:checking for strings.h... yes 2024-03-18T16:11:17.837 INFO:tasks.workunit.client.0.smithi039.stdout:checking for inttypes.h... yes 2024-03-18T16:11:17.930 INFO:tasks.workunit.client.0.smithi039.stdout:checking for stdint.h... yes 2024-03-18T16:11:18.022 INFO:tasks.workunit.client.0.smithi039.stdout:checking for unistd.h... yes 2024-03-18T16:11:18.122 INFO:tasks.workunit.client.0.smithi039.stdout:checking minix/config.h usability... no 2024-03-18T16:11:18.176 INFO:tasks.workunit.client.0.smithi039.stdout:checking minix/config.h presence... no 2024-03-18T16:11:18.176 INFO:tasks.workunit.client.0.smithi039.stdout:checking for minix/config.h... no 2024-03-18T16:11:18.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:11:17 smithi196 ceph-mon[36307]: scrub summary: idle 2024-03-18T16:11:18.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:11:17 smithi194 ceph-mon[36200]: scrub summary: idle 2024-03-18T16:11:18.254 INFO:tasks.workunit.client.0.smithi039.stdout:checking whether it is safe to define __EXTENSIONS__... yes 2024-03-18T16:11:18.255 INFO:tasks.workunit.client.0.smithi039.stdout:checking for gcc... (cached) gcc 2024-03-18T16:11:18.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:11:17 smithi039 ceph-mon[31721]: scrub summary: idle 2024-03-18T16:11:18.338 INFO:tasks.workunit.client.0.smithi039.stdout:checking whether we are using the GNU C compiler... (cached) yes 2024-03-18T16:11:18.338 INFO:tasks.workunit.client.0.smithi039.stdout:checking whether gcc accepts -g... (cached) yes 2024-03-18T16:11:18.339 INFO:tasks.workunit.client.0.smithi039.stdout:checking for gcc option to accept ISO C89... (cached) none needed 2024-03-18T16:11:18.339 INFO:tasks.workunit.client.0.smithi039.stdout:checking dependency style of gcc... (cached) gcc3 2024-03-18T16:11:18.440 INFO:tasks.workunit.client.0.smithi039.stdout:checking for main in -lm... yes 2024-03-18T16:11:18.549 INFO:tasks.workunit.client.0.smithi039.stdout:checking for main in -lpthread... yes 2024-03-18T16:11:18.552 INFO:tasks.workunit.client.0.smithi039.stdout:checking for ANSI C header files... (cached) yes 2024-03-18T16:11:18.738 INFO:tasks.workunit.client.0.smithi039.stdout:checking for sys/wait.h that is POSIX.1 compatible... yes 2024-03-18T16:11:18.856 INFO:tasks.workunit.client.0.smithi039.stdout:checking pthread.h usability... yes 2024-03-18T16:11:18.893 INFO:tasks.workunit.client.0.smithi039.stdout:checking pthread.h presence... yes 2024-03-18T16:11:18.893 INFO:tasks.workunit.client.0.smithi039.stdout:checking for pthread.h... yes 2024-03-18T16:11:18.999 INFO:tasks.workunit.client.0.smithi039.stdout:checking fcntl.h usability... yes 2024-03-18T16:11:19.036 INFO:tasks.workunit.client.0.smithi039.stdout:checking fcntl.h presence... yes 2024-03-18T16:11:19.036 INFO:tasks.workunit.client.0.smithi039.stdout:checking for fcntl.h... yes 2024-03-18T16:11:19.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:11:18 smithi196 ceph-mon[36307]: pgmap v459: 129 pgs: 129 active+clean; 4.4 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 366 KiB/s wr, 47 op/s 2024-03-18T16:11:19.240 INFO:tasks.workunit.client.0.smithi039.stdout:checking limits.h usability... yes 2024-03-18T16:11:19.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:11:18 smithi194 ceph-mon[36200]: pgmap v459: 129 pgs: 129 active+clean; 4.4 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 366 KiB/s wr, 47 op/s 2024-03-18T16:11:19.280 INFO:tasks.workunit.client.0.smithi039.stdout:checking limits.h presence... yes 2024-03-18T16:11:19.280 INFO:tasks.workunit.client.0.smithi039.stdout:checking for limits.h... yes 2024-03-18T16:11:19.314 INFO:tasks.workunit.client.0.smithi039.stdout:checking for stdint.h... (cached) yes 2024-03-18T16:11:19.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:11:18 smithi039 ceph-mon[31721]: pgmap v459: 129 pgs: 129 active+clean; 4.4 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 366 KiB/s wr, 47 op/s 2024-03-18T16:11:19.400 INFO:tasks.workunit.client.0.smithi039.stdout:checking sys/time.h usability... yes 2024-03-18T16:11:19.438 INFO:tasks.workunit.client.0.smithi039.stdout:checking sys/time.h presence... yes 2024-03-18T16:11:19.438 INFO:tasks.workunit.client.0.smithi039.stdout:checking for sys/time.h... yes 2024-03-18T16:11:19.470 INFO:tasks.workunit.client.0.smithi039.stdout:checking for unistd.h... (cached) yes 2024-03-18T16:11:19.555 INFO:tasks.workunit.client.0.smithi039.stdout:checking sys/vfs.h usability... yes 2024-03-18T16:11:19.595 INFO:tasks.workunit.client.0.smithi039.stdout:checking sys/vfs.h presence... yes 2024-03-18T16:11:19.596 INFO:tasks.workunit.client.0.smithi039.stdout:checking for sys/vfs.h... yes 2024-03-18T16:11:19.790 INFO:tasks.workunit.client.0.smithi039.stdout:checking sys/limits.h usability... no 2024-03-18T16:11:19.840 INFO:tasks.workunit.client.0.smithi039.stdout:checking sys/limits.h presence... no 2024-03-18T16:11:19.840 INFO:tasks.workunit.client.0.smithi039.stdout:checking for sys/limits.h... no 2024-03-18T16:11:19.894 INFO:tasks.workunit.client.0.smithi039.stdout:checking for an ANSI C-conforming const... yes 2024-03-18T16:11:19.970 INFO:tasks.workunit.client.0.smithi039.stdout:checking whether time.h and sys/time.h may both be included... yes 2024-03-18T16:11:20.141 INFO:tasks.workunit.client.0.smithi039.stdout:checking for system... yes 2024-03-18T16:11:20.242 INFO:tasks.workunit.client.0.smithi039.stdout:checking for gettimeofday... yes 2024-03-18T16:11:20.354 INFO:tasks.workunit.client.0.smithi039.stdout:checking for mkdir... yes 2024-03-18T16:11:20.478 INFO:tasks.workunit.client.0.smithi039.stdout:checking for strerror... yes 2024-03-18T16:11:20.604 INFO:tasks.workunit.client.0.smithi039.stdout:checking for open64... yes 2024-03-18T16:11:20.721 INFO:tasks.workunit.client.0.smithi039.stdout:checking for stat64... yes 2024-03-18T16:11:20.842 INFO:tasks.workunit.client.0.smithi039.stdout:checking for fseeko64... yes 2024-03-18T16:11:20.961 INFO:tasks.workunit.client.0.smithi039.stdout:checking for lrand48_r... yes 2024-03-18T16:11:21.087 INFO:tasks.workunit.client.0.smithi039.stdout:checking for srand48_r... yes 2024-03-18T16:11:21.139 INFO:tasks.workunit.client.0.smithi039.stdout:configure: creating ./config.status 2024-03-18T16:11:21.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:11:20 smithi196 ceph-mon[36307]: pgmap v460: 129 pgs: 129 active+clean; 5.2 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 435 KiB/s wr, 50 op/s 2024-03-18T16:11:21.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:11:20 smithi194 ceph-mon[36200]: pgmap v460: 129 pgs: 129 active+clean; 5.2 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 435 KiB/s wr, 50 op/s 2024-03-18T16:11:21.295 INFO:tasks.workunit.client.0.smithi039.stdout: /bin/sh ./config.status 2024-03-18T16:11:21.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:11:20 smithi039 ceph-mon[31721]: pgmap v460: 129 pgs: 129 active+clean; 5.2 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 435 KiB/s wr, 50 op/s 2024-03-18T16:11:21.411 INFO:tasks.workunit.client.0.smithi039.stdout:config.status: creating Makefile 2024-03-18T16:11:21.441 INFO:tasks.workunit.client.0.smithi039.stdout:config.status: creating config.h 2024-03-18T16:11:21.479 INFO:tasks.workunit.client.0.smithi039.stdout:config.status: executing depfiles commands 2024-03-18T16:11:21.515 INFO:tasks.workunit.client.0.smithi039.stdout:config.status: executing default commands 2024-03-18T16:11:21.562 INFO:tasks.workunit.client.0.smithi039.stdout:cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run autoheader 2024-03-18T16:11:21.819 INFO:tasks.workunit.client.0.smithi039.stdout:rm -f stamp-h1 2024-03-18T16:11:21.822 INFO:tasks.workunit.client.0.smithi039.stdout:touch config.h.in 2024-03-18T16:11:21.826 INFO:tasks.workunit.client.0.smithi039.stdout:cd . && /bin/sh ./config.status config.h 2024-03-18T16:11:21.929 INFO:tasks.workunit.client.0.smithi039.stdout:config.status: creating config.h 2024-03-18T16:11:21.974 INFO:tasks.workunit.client.0.smithi039.stdout:make all-am 2024-03-18T16:11:21.983 INFO:tasks.workunit.client.0.smithi039.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb' 2024-03-18T16:11:21.983 INFO:tasks.workunit.client.0.smithi039.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT fileops.o -MD -MP -MF .deps/fileops.Tpo -c -o fileops.o fileops.c 2024-03-18T16:11:22.299 INFO:tasks.workunit.client.0.smithi039.stdout:mv -f .deps/fileops.Tpo .deps/fileops.Po 2024-03-18T16:11:22.303 INFO:tasks.workunit.client.0.smithi039.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rand.o -MD -MP -MF .deps/rand.Tpo -c -o rand.o rand.c 2024-03-18T16:11:22.580 INFO:tasks.workunit.client.0.smithi039.stdout:mv -f .deps/rand.Tpo .deps/rand.Po 2024-03-18T16:11:22.585 INFO:tasks.workunit.client.0.smithi039.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT main.o -MD -MP -MF .deps/main.Tpo -c -o main.o main.c 2024-03-18T16:11:22.722 INFO:tasks.workunit.client.0.smithi039.stdout:mv -f .deps/main.Tpo .deps/main.Po 2024-03-18T16:11:22.728 INFO:tasks.workunit.client.0.smithi039.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT fh.o -MD -MP -MF .deps/fh.Tpo -c -o fh.o fh.c 2024-03-18T16:11:22.871 INFO:tasks.workunit.client.0.smithi039.stdout:mv -f .deps/fh.Tpo .deps/fh.Po 2024-03-18T16:11:22.878 INFO:tasks.workunit.client.0.smithi039.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT filelist.o -MD -MP -MF .deps/filelist.Tpo -c -o filelist.o filelist.c 2024-03-18T16:11:23.070 INFO:tasks.workunit.client.0.smithi039.stdout:mv -f .deps/filelist.Tpo .deps/filelist.Po 2024-03-18T16:11:23.077 INFO:tasks.workunit.client.0.smithi039.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT metaops.o -MD -MP -MF .deps/metaops.Tpo -c -o metaops.o metaops.c 2024-03-18T16:11:23.167 INFO:tasks.workunit.client.0.smithi039.stdout:mv -f .deps/metaops.Tpo .deps/metaops.Po 2024-03-18T16:11:23.174 INFO:tasks.workunit.client.0.smithi039.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rwlock.o -MD -MP -MF .deps/rwlock.Tpo -c -o rwlock.o rwlock.c 2024-03-18T16:11:23.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:11:22 smithi196 ceph-mon[36307]: pgmap v461: 129 pgs: 129 active+clean; 5.3 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 308 KiB/s wr, 51 op/s 2024-03-18T16:11:23.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:11:22 smithi194 ceph-mon[36200]: pgmap v461: 129 pgs: 129 active+clean; 5.3 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 308 KiB/s wr, 51 op/s 2024-03-18T16:11:23.253 INFO:tasks.workunit.client.0.smithi039.stdout:mv -f .deps/rwlock.Tpo .deps/rwlock.Po 2024-03-18T16:11:23.258 INFO:tasks.workunit.client.0.smithi039.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT cirlist.o -MD -MP -MF .deps/cirlist.Tpo -c -o cirlist.o cirlist.c 2024-03-18T16:11:23.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:11:22 smithi039 ceph-mon[31721]: pgmap v461: 129 pgs: 129 active+clean; 5.3 MiB data, 407 MiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 308 KiB/s wr, 51 op/s 2024-03-18T16:11:23.344 INFO:tasks.workunit.client.0.smithi039.stdout:mv -f .deps/cirlist.Tpo .deps/cirlist.Po 2024-03-18T16:11:23.350 INFO:tasks.workunit.client.0.smithi039.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rbt.o -MD -MP -MF .deps/rbt.Tpo -c -o rbt.o rbt.c 2024-03-18T16:11:23.629 INFO:tasks.workunit.client.0.smithi039.stdout:mv -f .deps/rbt.Tpo .deps/rbt.Po 2024-03-18T16:11:23.635 INFO:tasks.workunit.client.0.smithi039.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_tg.o -MD -MP -MF .deps/ffsb_tg.Tpo -c -o ffsb_tg.o ffsb_tg.c 2024-03-18T16:11:23.816 INFO:tasks.workunit.client.0.smithi039.stdout:mv -f .deps/ffsb_tg.Tpo .deps/ffsb_tg.Po 2024-03-18T16:11:23.822 INFO:tasks.workunit.client.0.smithi039.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_fs.o -MD -MP -MF .deps/ffsb_fs.Tpo -c -o ffsb_fs.o ffsb_fs.c 2024-03-18T16:11:24.056 INFO:tasks.workunit.client.0.smithi039.stdout:mv -f .deps/ffsb_fs.Tpo .deps/ffsb_fs.Po 2024-03-18T16:11:24.063 INFO:tasks.workunit.client.0.smithi039.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_thread.o -MD -MP -MF .deps/ffsb_thread.Tpo -c -o ffsb_thread.o ffsb_thread.c 2024-03-18T16:11:24.180 INFO:tasks.workunit.client.0.smithi039.stdout:mv -f .deps/ffsb_thread.Tpo .deps/ffsb_thread.Po 2024-03-18T16:11:24.187 INFO:tasks.workunit.client.0.smithi039.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_op.o -MD -MP -MF .deps/ffsb_op.Tpo -c -o ffsb_op.o ffsb_op.c 2024-03-18T16:11:24.350 INFO:tasks.workunit.client.0.smithi039.stdout:mv -f .deps/ffsb_op.Tpo .deps/ffsb_op.Po 2024-03-18T16:11:24.357 INFO:tasks.workunit.client.0.smithi039.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT util.o -MD -MP -MF .deps/util.Tpo -c -o util.o util.c 2024-03-18T16:11:24.555 INFO:tasks.workunit.client.0.smithi039.stdout:mv -f .deps/util.Tpo .deps/util.Po 2024-03-18T16:11:24.562 INFO:tasks.workunit.client.0.smithi039.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT parser.o -MD -MP -MF .deps/parser.Tpo -c -o parser.o parser.c 2024-03-18T16:11:24.618 INFO:tasks.workunit.client.0.smithi039.stderr:In file included from parser.h:22, 2024-03-18T16:11:24.618 INFO:tasks.workunit.client.0.smithi039.stderr: from parser.c:25: 2024-03-18T16:11:24.618 INFO:tasks.workunit.client.0.smithi039.stderr:parser.c: In function ‘init_filesys’: 2024-03-18T16:11:24.618 INFO:tasks.workunit.client.0.smithi039.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-03-18T16:11:24.618 INFO:tasks.workunit.client.0.smithi039.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-03-18T16:11:24.618 INFO:tasks.workunit.client.0.smithi039.stderr: | ^ 2024-03-18T16:11:24.618 INFO:tasks.workunit.client.0.smithi039.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-03-18T16:11:24.618 INFO:tasks.workunit.client.0.smithi039.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-03-18T16:11:24.618 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~ 2024-03-18T16:11:24.618 INFO:tasks.workunit.client.0.smithi039.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-03-18T16:11:24.618 INFO:tasks.workunit.client.0.smithi039.stderr: 36 | container_of(ptr, type, member) 2024-03-18T16:11:24.619 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~~~~~ 2024-03-18T16:11:24.619 INFO:tasks.workunit.client.0.smithi039.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-03-18T16:11:24.619 INFO:tasks.workunit.client.0.smithi039.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-03-18T16:11:24.619 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~~~ 2024-03-18T16:11:24.619 INFO:tasks.workunit.client.0.smithi039.stderr:parser.c:934:17: note: in expansion of macro ‘list_for_each_entry’ 2024-03-18T16:11:24.619 INFO:tasks.workunit.client.0.smithi039.stderr: 934 | list_for_each_entry(tmp_list, &list_head->list, list) 2024-03-18T16:11:24.619 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-03-18T16:11:24.619 INFO:tasks.workunit.client.0.smithi039.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-03-18T16:11:24.619 INFO:tasks.workunit.client.0.smithi039.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-03-18T16:11:24.619 INFO:tasks.workunit.client.0.smithi039.stderr: | ^ 2024-03-18T16:11:24.619 INFO:tasks.workunit.client.0.smithi039.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-03-18T16:11:24.619 INFO:tasks.workunit.client.0.smithi039.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-03-18T16:11:24.619 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~ 2024-03-18T16:11:24.619 INFO:tasks.workunit.client.0.smithi039.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-03-18T16:11:24.620 INFO:tasks.workunit.client.0.smithi039.stderr: 36 | container_of(ptr, type, member) 2024-03-18T16:11:24.620 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~~~~~ 2024-03-18T16:11:24.620 INFO:tasks.workunit.client.0.smithi039.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-03-18T16:11:24.620 INFO:tasks.workunit.client.0.smithi039.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-03-18T16:11:24.620 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~~~ 2024-03-18T16:11:24.620 INFO:tasks.workunit.client.0.smithi039.stderr:parser.c:934:17: note: in expansion of macro ‘list_for_each_entry’ 2024-03-18T16:11:24.620 INFO:tasks.workunit.client.0.smithi039.stderr: 934 | list_for_each_entry(tmp_list, &list_head->list, list) 2024-03-18T16:11:24.620 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-03-18T16:11:24.620 INFO:tasks.workunit.client.0.smithi039.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-03-18T16:11:24.620 INFO:tasks.workunit.client.0.smithi039.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-03-18T16:11:24.620 INFO:tasks.workunit.client.0.smithi039.stderr: | ^ 2024-03-18T16:11:24.620 INFO:tasks.workunit.client.0.smithi039.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-03-18T16:11:24.620 INFO:tasks.workunit.client.0.smithi039.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-03-18T16:11:24.620 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~ 2024-03-18T16:11:24.620 INFO:tasks.workunit.client.0.smithi039.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-03-18T16:11:24.621 INFO:tasks.workunit.client.0.smithi039.stderr: 36 | container_of(ptr, type, member) 2024-03-18T16:11:24.621 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~~~~~ 2024-03-18T16:11:24.621 INFO:tasks.workunit.client.0.smithi039.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-03-18T16:11:24.621 INFO:tasks.workunit.client.0.smithi039.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-03-18T16:11:24.621 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~~~ 2024-03-18T16:11:24.621 INFO:tasks.workunit.client.0.smithi039.stderr:parser.c:941:17: note: in expansion of macro ‘list_for_each_entry’ 2024-03-18T16:11:24.621 INFO:tasks.workunit.client.0.smithi039.stderr: 941 | list_for_each_entry(tmp_list, &list_head->list, list) { 2024-03-18T16:11:24.621 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-03-18T16:11:24.621 INFO:tasks.workunit.client.0.smithi039.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-03-18T16:11:24.621 INFO:tasks.workunit.client.0.smithi039.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-03-18T16:11:24.621 INFO:tasks.workunit.client.0.smithi039.stderr: | ^ 2024-03-18T16:11:24.621 INFO:tasks.workunit.client.0.smithi039.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-03-18T16:11:24.621 INFO:tasks.workunit.client.0.smithi039.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-03-18T16:11:24.621 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~ 2024-03-18T16:11:24.622 INFO:tasks.workunit.client.0.smithi039.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-03-18T16:11:24.622 INFO:tasks.workunit.client.0.smithi039.stderr: 36 | container_of(ptr, type, member) 2024-03-18T16:11:24.622 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~~~~~ 2024-03-18T16:11:24.622 INFO:tasks.workunit.client.0.smithi039.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-03-18T16:11:24.622 INFO:tasks.workunit.client.0.smithi039.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-03-18T16:11:24.622 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~~~ 2024-03-18T16:11:24.622 INFO:tasks.workunit.client.0.smithi039.stderr:parser.c:941:17: note: in expansion of macro ‘list_for_each_entry’ 2024-03-18T16:11:24.622 INFO:tasks.workunit.client.0.smithi039.stderr: 941 | list_for_each_entry(tmp_list, &list_head->list, list) { 2024-03-18T16:11:24.622 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-03-18T16:11:24.622 INFO:tasks.workunit.client.0.smithi039.stderr:parser.c: In function ‘init_tg_stats’: 2024-03-18T16:11:24.622 INFO:tasks.workunit.client.0.smithi039.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-03-18T16:11:24.622 INFO:tasks.workunit.client.0.smithi039.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-03-18T16:11:24.622 INFO:tasks.workunit.client.0.smithi039.stderr: | ^ 2024-03-18T16:11:24.622 INFO:tasks.workunit.client.0.smithi039.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-03-18T16:11:24.623 INFO:tasks.workunit.client.0.smithi039.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-03-18T16:11:24.623 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~ 2024-03-18T16:11:24.623 INFO:tasks.workunit.client.0.smithi039.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-03-18T16:11:24.623 INFO:tasks.workunit.client.0.smithi039.stderr: 36 | container_of(ptr, type, member) 2024-03-18T16:11:24.623 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~~~~~ 2024-03-18T16:11:24.623 INFO:tasks.workunit.client.0.smithi039.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-03-18T16:11:24.623 INFO:tasks.workunit.client.0.smithi039.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-03-18T16:11:24.623 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~~~ 2024-03-18T16:11:24.623 INFO:tasks.workunit.client.0.smithi039.stderr:parser.c:971:41: note: in expansion of macro ‘list_for_each_entry’ 2024-03-18T16:11:24.623 INFO:tasks.workunit.client.0.smithi039.stderr: 971 | list_for_each_entry(tmp_list, 2024-03-18T16:11:24.623 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-03-18T16:11:24.623 INFO:tasks.workunit.client.0.smithi039.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-03-18T16:11:24.624 INFO:tasks.workunit.client.0.smithi039.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-03-18T16:11:24.624 INFO:tasks.workunit.client.0.smithi039.stderr: | ^ 2024-03-18T16:11:24.624 INFO:tasks.workunit.client.0.smithi039.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-03-18T16:11:24.624 INFO:tasks.workunit.client.0.smithi039.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-03-18T16:11:24.624 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~ 2024-03-18T16:11:24.624 INFO:tasks.workunit.client.0.smithi039.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-03-18T16:11:24.624 INFO:tasks.workunit.client.0.smithi039.stderr: 36 | container_of(ptr, type, member) 2024-03-18T16:11:24.624 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~~~~~ 2024-03-18T16:11:24.624 INFO:tasks.workunit.client.0.smithi039.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-03-18T16:11:24.624 INFO:tasks.workunit.client.0.smithi039.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-03-18T16:11:24.624 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~~~ 2024-03-18T16:11:24.624 INFO:tasks.workunit.client.0.smithi039.stderr:parser.c:971:41: note: in expansion of macro ‘list_for_each_entry’ 2024-03-18T16:11:24.624 INFO:tasks.workunit.client.0.smithi039.stderr: 971 | list_for_each_entry(tmp_list, 2024-03-18T16:11:24.624 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-03-18T16:11:24.625 INFO:tasks.workunit.client.0.smithi039.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-03-18T16:11:24.625 INFO:tasks.workunit.client.0.smithi039.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-03-18T16:11:24.625 INFO:tasks.workunit.client.0.smithi039.stderr: | ^ 2024-03-18T16:11:24.625 INFO:tasks.workunit.client.0.smithi039.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-03-18T16:11:24.625 INFO:tasks.workunit.client.0.smithi039.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-03-18T16:11:24.625 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~ 2024-03-18T16:11:24.625 INFO:tasks.workunit.client.0.smithi039.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-03-18T16:11:24.625 INFO:tasks.workunit.client.0.smithi039.stderr: 36 | container_of(ptr, type, member) 2024-03-18T16:11:24.625 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~~~~~ 2024-03-18T16:11:24.625 INFO:tasks.workunit.client.0.smithi039.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-03-18T16:11:24.625 INFO:tasks.workunit.client.0.smithi039.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-03-18T16:11:24.625 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~~~ 2024-03-18T16:11:24.625 INFO:tasks.workunit.client.0.smithi039.stderr:parser.c:980:41: note: in expansion of macro ‘list_for_each_entry’ 2024-03-18T16:11:24.625 INFO:tasks.workunit.client.0.smithi039.stderr: 980 | list_for_each_entry(tmp_list, 2024-03-18T16:11:24.626 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-03-18T16:11:24.626 INFO:tasks.workunit.client.0.smithi039.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-03-18T16:11:24.626 INFO:tasks.workunit.client.0.smithi039.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-03-18T16:11:24.626 INFO:tasks.workunit.client.0.smithi039.stderr: | ^ 2024-03-18T16:11:24.626 INFO:tasks.workunit.client.0.smithi039.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-03-18T16:11:24.626 INFO:tasks.workunit.client.0.smithi039.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-03-18T16:11:24.626 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~ 2024-03-18T16:11:24.626 INFO:tasks.workunit.client.0.smithi039.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-03-18T16:11:24.626 INFO:tasks.workunit.client.0.smithi039.stderr: 36 | container_of(ptr, type, member) 2024-03-18T16:11:24.626 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~~~~~ 2024-03-18T16:11:24.626 INFO:tasks.workunit.client.0.smithi039.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-03-18T16:11:24.626 INFO:tasks.workunit.client.0.smithi039.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-03-18T16:11:24.626 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~~~ 2024-03-18T16:11:24.626 INFO:tasks.workunit.client.0.smithi039.stderr:parser.c:980:41: note: in expansion of macro ‘list_for_each_entry’ 2024-03-18T16:11:24.626 INFO:tasks.workunit.client.0.smithi039.stderr: 980 | list_for_each_entry(tmp_list, 2024-03-18T16:11:24.627 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-03-18T16:11:24.987 INFO:tasks.workunit.client.0.smithi039.stderr:parser.c: In function ‘get_optstr’: 2024-03-18T16:11:24.987 INFO:tasks.workunit.client.0.smithi039.stderr:parser.c:206:23: warning: ‘strnlen’ specified bound 4095 exceeds source size 1024 [-Wstringop-overread] 2024-03-18T16:11:24.987 INFO:tasks.workunit.client.0.smithi039.stderr: 206 | len = strnlen(temp, 4095) + 1; 2024-03-18T16:11:24.987 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-03-18T16:11:24.988 INFO:tasks.workunit.client.0.smithi039.stderr:parser.c:200:14: note: source object declared here 2024-03-18T16:11:24.988 INFO:tasks.workunit.client.0.smithi039.stderr: 200 | char temp[BUFSIZE]; 2024-03-18T16:11:24.988 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~ 2024-03-18T16:11:25.050 INFO:tasks.workunit.client.0.smithi039.stderr:parser.c: In function ‘size64_convert’: 2024-03-18T16:11:25.050 INFO:tasks.workunit.client.0.smithi039.stderr:parser.c:119:9: warning: ‘memcpy’ reading 3 bytes from a region of size 2 [-Wstringop-overread] 2024-03-18T16:11:25.050 INFO:tasks.workunit.client.0.smithi039.stderr: 119 | memcpy(unit, "\0", 3); 2024-03-18T16:11:25.050 INFO:tasks.workunit.client.0.smithi039.stderr: | ^~~~~~~~~~~~~~~~~~~~~ 2024-03-18T16:11:25.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:11:24 smithi196 ceph-mon[36307]: pgmap v462: 129 pgs: 129 active+clean; 6.8 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 464 KiB/s wr, 71 op/s 2024-03-18T16:11:25.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:11:24 smithi194 ceph-mon[36200]: pgmap v462: 129 pgs: 129 active+clean; 6.8 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 464 KiB/s wr, 71 op/s 2024-03-18T16:11:25.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:11:24 smithi039 ceph-mon[31721]: pgmap v462: 129 pgs: 129 active+clean; 6.8 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 464 KiB/s wr, 71 op/s 2024-03-18T16:11:25.451 INFO:tasks.workunit.client.0.smithi039.stdout:mv -f .deps/parser.Tpo .deps/parser.Po 2024-03-18T16:11:25.456 INFO:tasks.workunit.client.0.smithi039.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_fc.o -MD -MP -MF .deps/ffsb_fc.Tpo -c -o ffsb_fc.o ffsb_fc.c 2024-03-18T16:11:25.553 INFO:tasks.workunit.client.0.smithi039.stdout:mv -f .deps/ffsb_fc.Tpo .deps/ffsb_fc.Po 2024-03-18T16:11:25.560 INFO:tasks.workunit.client.0.smithi039.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_stats.o -MD -MP -MF .deps/ffsb_stats.Tpo -c -o ffsb_stats.o ffsb_stats.c 2024-03-18T16:11:25.700 INFO:tasks.workunit.client.0.smithi039.stdout:mv -f .deps/ffsb_stats.Tpo .deps/ffsb_stats.Po 2024-03-18T16:11:25.707 INFO:tasks.workunit.client.0.smithi039.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT list.o -MD -MP -MF .deps/list.Tpo -c -o list.o list.c 2024-03-18T16:11:25.778 INFO:tasks.workunit.client.0.smithi039.stdout:mv -f .deps/list.Tpo .deps/list.Po 2024-03-18T16:11:25.785 INFO:tasks.workunit.client.0.smithi039.stdout:gcc -D_GNU_SOURCE -g -O2 -D_REENTRANT -o ffsb fileops.o rand.o main.o fh.o filelist.o metaops.o rwlock.o cirlist.o rbt.o ffsb_tg.o ffsb_fs.o ffsb_thread.o ffsb_op.o util.o parser.o ffsb_fc.o ffsb_stats.o list.o -lpthread -lm 2024-03-18T16:11:25.857 INFO:tasks.workunit.client.0.smithi039.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb' 2024-03-18T16:11:25.858 INFO:tasks.workunit.client.0.smithi039.stderr:+ cd .. 2024-03-18T16:11:25.858 INFO:tasks.workunit.client.0.smithi039.stderr:+ mkdir tmp 2024-03-18T16:11:25.864 INFO:tasks.workunit.client.0.smithi039.stderr:+ cd tmp 2024-03-18T16:11:25.865 INFO:tasks.workunit.client.0.smithi039.stderr:+ for f in $mydir/*.ffsb 2024-03-18T16:11:25.865 INFO:tasks.workunit.client.0.smithi039.stderr:+ ../ffsb/ffsb /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/random_write.32.ffsb 2024-03-18T16:11:25.867 INFO:tasks.workunit.client.0.smithi039.stdout:FFSB version 6.0-RC2 started 2024-03-18T16:11:25.867 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-03-18T16:11:25.877 INFO:tasks.workunit.client.0.smithi039.stdout:benchmark time = 300 2024-03-18T16:11:25.877 INFO:tasks.workunit.client.0.smithi039.stdout:ThreadGroup 0 2024-03-18T16:11:25.877 INFO:tasks.workunit.client.0.smithi039.stdout:================ 2024-03-18T16:11:25.877 INFO:tasks.workunit.client.0.smithi039.stdout: num_threads = 32 2024-03-18T16:11:25.877 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-03-18T16:11:25.877 INFO:tasks.workunit.client.0.smithi039.stdout: read_random = off 2024-03-18T16:11:25.877 INFO:tasks.workunit.client.0.smithi039.stdout: read_size = 0 (0B) 2024-03-18T16:11:25.877 INFO:tasks.workunit.client.0.smithi039.stdout: read_blocksize = 0 (0B) 2024-03-18T16:11:25.877 INFO:tasks.workunit.client.0.smithi039.stdout: read_skip = off 2024-03-18T16:11:25.877 INFO:tasks.workunit.client.0.smithi039.stdout: read_skipsize = 0 (0B) 2024-03-18T16:11:25.877 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-03-18T16:11:25.877 INFO:tasks.workunit.client.0.smithi039.stdout: write_random = on 2024-03-18T16:11:25.878 INFO:tasks.workunit.client.0.smithi039.stdout: write_size = 5242880 (5MB) 2024-03-18T16:11:25.878 INFO:tasks.workunit.client.0.smithi039.stdout: fsync_file = 0 2024-03-18T16:11:25.878 INFO:tasks.workunit.client.0.smithi039.stdout: write_blocksize = 4096 (4KB) 2024-03-18T16:11:25.878 INFO:tasks.workunit.client.0.smithi039.stdout: wait time = 0 2024-03-18T16:11:25.878 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-03-18T16:11:25.878 INFO:tasks.workunit.client.0.smithi039.stdout: op weights 2024-03-18T16:11:25.878 INFO:tasks.workunit.client.0.smithi039.stdout: read = 0 (0.00%) 2024-03-18T16:11:25.878 INFO:tasks.workunit.client.0.smithi039.stdout: readall = 0 (0.00%) 2024-03-18T16:11:25.878 INFO:tasks.workunit.client.0.smithi039.stdout: write = 1 (100.00%) 2024-03-18T16:11:25.878 INFO:tasks.workunit.client.0.smithi039.stdout: create = 0 (0.00%) 2024-03-18T16:11:25.878 INFO:tasks.workunit.client.0.smithi039.stdout: append = 0 (0.00%) 2024-03-18T16:11:25.878 INFO:tasks.workunit.client.0.smithi039.stdout: delete = 0 (0.00%) 2024-03-18T16:11:25.879 INFO:tasks.workunit.client.0.smithi039.stdout: metaop = 0 (0.00%) 2024-03-18T16:11:25.879 INFO:tasks.workunit.client.0.smithi039.stdout: createdir = 0 (0.00%) 2024-03-18T16:11:25.879 INFO:tasks.workunit.client.0.smithi039.stdout: stat = 0 (0.00%) 2024-03-18T16:11:25.879 INFO:tasks.workunit.client.0.smithi039.stdout: writeall = 0 (0.00%) 2024-03-18T16:11:25.879 INFO:tasks.workunit.client.0.smithi039.stdout: writeall_fsync = 0 (0.00%) 2024-03-18T16:11:25.879 INFO:tasks.workunit.client.0.smithi039.stdout: open_close = 0 (0.00%) 2024-03-18T16:11:25.879 INFO:tasks.workunit.client.0.smithi039.stdout: write_fsync = 0 (0.00%) 2024-03-18T16:11:25.879 INFO:tasks.workunit.client.0.smithi039.stdout: create_fsync = 0 (0.00%) 2024-03-18T16:11:25.879 INFO:tasks.workunit.client.0.smithi039.stdout: append_fsync = 0 (0.00%) 2024-03-18T16:11:25.879 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-03-18T16:11:25.879 INFO:tasks.workunit.client.0.smithi039.stdout:FileSystem . 2024-03-18T16:11:25.879 INFO:tasks.workunit.client.0.smithi039.stdout:========== 2024-03-18T16:11:25.879 INFO:tasks.workunit.client.0.smithi039.stdout: num_dirs = 0 2024-03-18T16:11:25.879 INFO:tasks.workunit.client.0.smithi039.stdout: starting files = 128 2024-03-18T16:11:25.880 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-03-18T16:11:25.880 INFO:tasks.workunit.client.0.smithi039.stdout: min file size = 104857600 (100MB) 2024-03-18T16:11:25.880 INFO:tasks.workunit.client.0.smithi039.stdout: max file size = 104857600 (100MB) 2024-03-18T16:11:25.880 INFO:tasks.workunit.client.0.smithi039.stdout: directio = off 2024-03-18T16:11:25.880 INFO:tasks.workunit.client.0.smithi039.stdout: alignedio = on 2024-03-18T16:11:25.880 INFO:tasks.workunit.client.0.smithi039.stdout: bufferedio = off 2024-03-18T16:11:25.880 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-03-18T16:11:25.880 INFO:tasks.workunit.client.0.smithi039.stdout: aging is off 2024-03-18T16:11:25.881 INFO:tasks.workunit.client.0.smithi039.stdout: current utilization = 0.04% 2024-03-18T16:11:25.881 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-03-18T16:11:25.881 INFO:tasks.workunit.client.0.smithi039.stdout:checking existing fs: . 2024-03-18T16:11:25.882 INFO:tasks.workunit.client.0.smithi039.stdout:recreating new fileset 2024-03-18T16:11:25.882 INFO:tasks.workunit.client.0.smithi039.stderr:opendir: No such file or directory 2024-03-18T16:11:27.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:11:26 smithi196 ceph-mon[36307]: pgmap v463: 129 pgs: 129 active+clean; 6.8 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 396 KiB/s wr, 61 op/s 2024-03-18T16:11:27.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:11:26 smithi194 ceph-mon[36200]: pgmap v463: 129 pgs: 129 active+clean; 6.8 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 396 KiB/s wr, 61 op/s 2024-03-18T16:11:27.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:11:26 smithi039 ceph-mon[31721]: pgmap v463: 129 pgs: 129 active+clean; 6.8 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 273 KiB/s rd, 396 KiB/s wr, 61 op/s 2024-03-18T16:11:29.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:11:28 smithi196 ceph-mon[36307]: pgmap v464: 129 pgs: 129 active+clean; 7.1 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 311 KiB/s wr, 59 op/s 2024-03-18T16:11:29.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:11:28 smithi194 ceph-mon[36200]: pgmap v464: 129 pgs: 129 active+clean; 7.1 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 311 KiB/s wr, 59 op/s 2024-03-18T16:11:29.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:11:28 smithi039 ceph-mon[31721]: pgmap v464: 129 pgs: 129 active+clean; 7.1 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 311 KiB/s wr, 59 op/s 2024-03-18T16:11:31.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:11:30 smithi196 ceph-mon[36307]: pgmap v465: 129 pgs: 129 active+clean; 7.1 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 310 KiB/s wr, 58 op/s 2024-03-18T16:11:31.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:11:30 smithi194 ceph-mon[36200]: pgmap v465: 129 pgs: 129 active+clean; 7.1 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 310 KiB/s wr, 58 op/s 2024-03-18T16:11:31.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:11:30 smithi039 ceph-mon[31721]: pgmap v465: 129 pgs: 129 active+clean; 7.1 MiB data, 416 MiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 310 KiB/s wr, 58 op/s 2024-03-18T16:11:33.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:11:32 smithi196 ceph-mon[36307]: pgmap v466: 129 pgs: 129 active+clean; 40 MiB data, 473 MiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 2.9 MiB/s wr, 58 op/s 2024-03-18T16:11:33.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:11:32 smithi194 ceph-mon[36200]: pgmap v466: 129 pgs: 129 active+clean; 40 MiB data, 473 MiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 2.9 MiB/s wr, 58 op/s 2024-03-18T16:11:33.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:11:32 smithi039 ceph-mon[31721]: pgmap v466: 129 pgs: 129 active+clean; 40 MiB data, 473 MiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 2.9 MiB/s wr, 58 op/s 2024-03-18T16:11:34.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:11:33 smithi196 ceph-mon[36307]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T16:11:34.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:11:33 smithi194 ceph-mon[36200]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T16:11:34.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:11:33 smithi039 ceph-mon[31721]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T16:11:35.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:11:34 smithi196 ceph-mon[36307]: pgmap v467: 129 pgs: 129 active+clean; 93 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 7.4 MiB/s wr, 69 op/s 2024-03-18T16:11:35.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:11:34 smithi194 ceph-mon[36200]: pgmap v467: 129 pgs: 129 active+clean; 93 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 7.4 MiB/s wr, 69 op/s 2024-03-18T16:11:35.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:11:34 smithi039 ceph-mon[31721]: pgmap v467: 129 pgs: 129 active+clean; 93 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 175 KiB/s rd, 7.4 MiB/s wr, 69 op/s 2024-03-18T16:11:37.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:11:36 smithi196 ceph-mon[36307]: pgmap v468: 129 pgs: 129 active+clean; 213 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 17 MiB/s wr, 49 op/s 2024-03-18T16:11:37.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:11:36 smithi194 ceph-mon[36200]: pgmap v468: 129 pgs: 129 active+clean; 213 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 17 MiB/s wr, 49 op/s 2024-03-18T16:11:37.342 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:11:36 smithi039 ceph-mon[31721]: pgmap v468: 129 pgs: 129 active+clean; 213 MiB data, 829 MiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 17 MiB/s wr, 49 op/s 2024-03-18T16:11:39.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:11:38 smithi196 ceph-mon[36307]: pgmap v469: 129 pgs: 129 active+clean; 509 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 42 MiB/s wr, 63 op/s 2024-03-18T16:11:39.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:11:38 smithi196 ceph-mon[36307]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T16:11:39.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:11:38 smithi194 ceph-mon[36200]: pgmap v469: 129 pgs: 129 active+clean; 509 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 42 MiB/s wr, 63 op/s 2024-03-18T16:11:39.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:11:38 smithi194 ceph-mon[36200]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T16:11:39.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:11:38 smithi039 ceph-mon[31721]: pgmap v469: 129 pgs: 129 active+clean; 509 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 128 KiB/s rd, 42 MiB/s wr, 63 op/s 2024-03-18T16:11:39.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:11:38 smithi039 ceph-mon[31721]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T16:11:41.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:11:40 smithi196 ceph-mon[36307]: pgmap v470: 129 pgs: 129 active+clean; 981 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 81 MiB/s wr, 69 op/s 2024-03-18T16:11:41.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:11:40 smithi194 ceph-mon[36200]: pgmap v470: 129 pgs: 129 active+clean; 981 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 81 MiB/s wr, 69 op/s 2024-03-18T16:11:41.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:11:40 smithi039 ceph-mon[31721]: pgmap v470: 129 pgs: 129 active+clean; 981 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 81 MiB/s wr, 69 op/s 2024-03-18T16:11:43.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:11:42 smithi194 ceph-mon[36200]: pgmap v471: 129 pgs: 129 active+clean; 1.3 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 111 MiB/s wr, 75 op/s 2024-03-18T16:11:43.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:11:42 smithi039 ceph-mon[31721]: pgmap v471: 129 pgs: 129 active+clean; 1.3 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 111 MiB/s wr, 75 op/s 2024-03-18T16:11:43.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:11:42 smithi196 ceph-mon[36307]: pgmap v471: 129 pgs: 129 active+clean; 1.3 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 111 MiB/s wr, 75 op/s 2024-03-18T16:11:44.531 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T16:11:44.532 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T16:11:44.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.731+0000 7fbd6de00640 1 -- 172.21.15.39:0/1331708822 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbd68074240 msgr2=0x7fbd680746c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:44.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.731+0000 7fbd6de00640 1 --2- 172.21.15.39:0/1331708822 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbd68074240 0x7fbd680746c0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fbd54009900 tx=0x7fbd5402bc30 comp rx=0 tx=0).stop 2024-03-18T16:11:44.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.731+0000 7fbd6de00640 1 -- 172.21.15.39:0/1331708822 shutdown_connections 2024-03-18T16:11:44.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.731+0000 7fbd6de00640 1 --2- 172.21.15.39:0/1331708822 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbd6810ddc0 0x7fbd681101b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:44.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.731+0000 7fbd6de00640 1 --2- 172.21.15.39:0/1331708822 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbd68074240 0x7fbd680746c0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:44.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.731+0000 7fbd6de00640 1 --2- 172.21.15.39:0/1331708822 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbd68075820 0x7fbd68073c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:44.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.732+0000 7fbd6de00640 1 -- 172.21.15.39:0/1331708822 >> 172.21.15.39:0/1331708822 conn(0x7fbd680fcd80 msgr2=0x7fbd680ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:44.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.732+0000 7fbd6de00640 1 -- 172.21.15.39:0/1331708822 shutdown_connections 2024-03-18T16:11:44.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.732+0000 7fbd6de00640 1 -- 172.21.15.39:0/1331708822 wait complete. 2024-03-18T16:11:44.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.732+0000 7fbd6de00640 1 Processor -- start 2024-03-18T16:11:44.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.733+0000 7fbd6de00640 1 -- start start 2024-03-18T16:11:44.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.733+0000 7fbd6de00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbd68074240 0x7fbd6819e150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:44.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.733+0000 7fbd6de00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbd68075820 0x7fbd6819e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:44.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.733+0000 7fbd6de00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbd6810ddc0 0x7fbd681a5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:44.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.733+0000 7fbd6de00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fbd68113a90 con 0x7fbd68074240 2024-03-18T16:11:44.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.733+0000 7fbd6de00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fbd68113910 con 0x7fbd6810ddc0 2024-03-18T16:11:44.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.733+0000 7fbd6de00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fbd68113c10 con 0x7fbd68075820 2024-03-18T16:11:44.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.733+0000 7fbd66000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbd68075820 0x7fbd6819e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:44.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.733+0000 7fbd66000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbd68075820 0x7fbd6819e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:33608/0 (socket says 172.21.15.39:33608) 2024-03-18T16:11:44.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.733+0000 7fbd66000640 1 -- 172.21.15.39:0/441564750 learned_addr learned my addr 172.21.15.39:0/441564750 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:11:44.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.733+0000 7fbd66a00640 1 --2- 172.21.15.39:0/441564750 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbd68074240 0x7fbd6819e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:44.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.733+0000 7fbd67400640 1 --2- 172.21.15.39:0/441564750 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbd6810ddc0 0x7fbd681a5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:44.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.734+0000 7fbd66a00640 1 -- 172.21.15.39:0/441564750 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbd68075820 msgr2=0x7fbd6819e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:44.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.734+0000 7fbd66a00640 1 --2- 172.21.15.39:0/441564750 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbd68075820 0x7fbd6819e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:44.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.734+0000 7fbd66a00640 1 -- 172.21.15.39:0/441564750 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbd6810ddc0 msgr2=0x7fbd681a5700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:44.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.734+0000 7fbd66a00640 1 --2- 172.21.15.39:0/441564750 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbd6810ddc0 0x7fbd681a5700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:44.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.734+0000 7fbd66a00640 1 -- 172.21.15.39:0/441564750 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd681a5d70 con 0x7fbd68074240 2024-03-18T16:11:44.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.734+0000 7fbd67400640 1 --2- 172.21.15.39:0/441564750 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbd6810ddc0 0x7fbd681a5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:11:44.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.734+0000 7fbd66000640 1 --2- 172.21.15.39:0/441564750 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbd68075820 0x7fbd6819e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:11:44.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.734+0000 7fbd66a00640 1 --2- 172.21.15.39:0/441564750 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbd68074240 0x7fbd6819e150 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fbd5800e9f0 tx=0x7fbd5800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:44.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.735+0000 7fbd6ca00640 1 -- 172.21.15.39:0/441564750 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd5800ccc0 con 0x7fbd68074240 2024-03-18T16:11:44.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.735+0000 7fbd6ca00640 1 -- 172.21.15.39:0/441564750 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd58010910 con 0x7fbd68074240 2024-03-18T16:11:44.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.735+0000 7fbd6ca00640 1 -- 172.21.15.39:0/441564750 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd58018960 con 0x7fbd68074240 2024-03-18T16:11:44.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.735+0000 7fbd6de00640 1 -- 172.21.15.39:0/441564750 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd681a6060 con 0x7fbd68074240 2024-03-18T16:11:44.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.735+0000 7fbd6de00640 1 -- 172.21.15.39:0/441564750 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd681a6580 con 0x7fbd68074240 2024-03-18T16:11:44.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.739+0000 7fbd70b89740 1 Processor -- start 2024-03-18T16:11:44.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.740+0000 7fbd70b89740 1 -- start start 2024-03-18T16:11:44.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.740+0000 7fbd70b89740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555820680810 0x555820680c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:44.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.740+0000 7fbd70b89740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555820681200 0x555820685d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:44.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.740+0000 7fbd70b89740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555820686370 0x555820688800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:44.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.740+0000 7fbd70b89740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5558205e51f0 con 0x555820680810 2024-03-18T16:11:44.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.740+0000 7fbd70b89740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55582058da80 con 0x555820686370 2024-03-18T16:11:44.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.740+0000 7fbd70b89740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x5558205e1290 con 0x555820681200 2024-03-18T16:11:44.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.741+0000 7fbd67400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555820680810 0x555820680c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:44.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.741+0000 7fbd67400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555820680810 0x555820680c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43578/0 (socket says 172.21.15.39:43578) 2024-03-18T16:11:44.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.741+0000 7fbd67400640 1 -- 172.21.15.39:0/1819115712 learned_addr learned my addr 172.21.15.39:0/1819115712 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:11:44.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.741+0000 7fbd6ca00640 1 -- 172.21.15.39:0/441564750 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fbd58010ac0 con 0x7fbd68074240 2024-03-18T16:11:44.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.741+0000 7fbd66a00640 1 --2- 172.21.15.39:0/1819115712 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555820681200 0x555820685d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:44.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.742+0000 7fbd66000640 1 --2- 172.21.15.39:0/1819115712 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555820686370 0x555820688800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:44.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.742+0000 7fbd67400640 1 -- 172.21.15.39:0/1819115712 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555820681200 msgr2=0x555820685d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:44.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.742+0000 7fbd67400640 1 --2- 172.21.15.39:0/1819115712 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555820681200 0x555820685d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:44.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.742+0000 7fbd67400640 1 -- 172.21.15.39:0/1819115712 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555820686370 msgr2=0x555820688800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:44.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.742+0000 7fbd67400640 1 --2- 172.21.15.39:0/1819115712 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555820686370 0x555820688800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:44.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.742+0000 7fbd67400640 1 -- 172.21.15.39:0/1819115712 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5558203c37d0 con 0x555820680810 2024-03-18T16:11:44.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.742+0000 7fbd67400640 1 --2- 172.21.15.39:0/1819115712 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555820680810 0x555820680c30 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fbd600117e0 tx=0x7fbd60011ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=937f686f5547a192 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:44.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.744+0000 7fbd6ca00640 1 --2- 172.21.15.39:0/441564750 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fbd4c0a2880 0x7fbd4c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:44.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.744+0000 7fbd53e00640 1 -- 172.21.15.39:0/1819115712 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd60011310 con 0x555820680810 2024-03-18T16:11:44.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.744+0000 7fbd53e00640 1 -- 172.21.15.39:0/1819115712 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd60003e50 con 0x555820680810 2024-03-18T16:11:44.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.744+0000 7fbd53e00640 1 -- 172.21.15.39:0/1819115712 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd600045f0 con 0x555820680810 2024-03-18T16:11:44.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.744+0000 7fbd6ca00640 1 -- 172.21.15.39:0/441564750 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fbd580cf3a0 con 0x7fbd68074240 2024-03-18T16:11:44.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.745+0000 7fbd70b89740 1 -- 172.21.15.39:0/1819115712 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555820680810 msgr2=0x555820680c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:44.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.745+0000 7fbd70b89740 1 --2- 172.21.15.39:0/1819115712 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555820680810 0x555820680c30 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fbd600117e0 tx=0x7fbd60011ca0 comp rx=0 tx=0).stop 2024-03-18T16:11:44.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.746+0000 7fbd70b89740 1 -- 172.21.15.39:0/1819115712 shutdown_connections 2024-03-18T16:11:44.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.746+0000 7fbd70b89740 1 --2- 172.21.15.39:0/1819115712 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555820686370 0x555820688800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:44.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.746+0000 7fbd70b89740 1 --2- 172.21.15.39:0/1819115712 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555820681200 0x555820685d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:44.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.746+0000 7fbd70b89740 1 --2- 172.21.15.39:0/1819115712 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555820680810 0x555820680c30 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:44.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.746+0000 7fbd70b89740 1 -- 172.21.15.39:0/1819115712 >> 172.21.15.39:0/1819115712 conn(0x55582067bd80 msgr2=0x55582067c180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:44.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.746+0000 7fbd66a00640 1 --2- 172.21.15.39:0/441564750 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fbd4c0a2880 0x7fbd4c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:44.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.748+0000 7fbd70b89740 1 -- 172.21.15.39:0/1819115712 shutdown_connections 2024-03-18T16:11:44.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.748+0000 7fbd66a00640 1 --2- 172.21.15.39:0/441564750 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fbd4c0a2880 0x7fbd4c0a4d40 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x555820681600 tx=0x7fbd580d2d60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:44.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.749+0000 7fbd70b89740 1 -- 172.21.15.39:0/1819115712 wait complete. 2024-03-18T16:11:44.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.750+0000 7fbd70b89740 1 -- start start 2024-03-18T16:11:44.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.750+0000 7fbd70b89740 1 Processor -- start 2024-03-18T16:11:44.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.751+0000 7fbd70b89740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555820680810 0x55582068ad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:44.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.751+0000 7fbd70b89740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555820681200 0x55582068b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:44.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.751+0000 7fbd70b89740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555820686370 0x555820692310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:44.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.751+0000 7fbd70b89740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x555820689320 con 0x555820686370 2024-03-18T16:11:44.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.751+0000 7fbd70b89740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5558205a0ed0 con 0x555820680810 2024-03-18T16:11:44.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.751+0000 7fbd70b89740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x5558206894a0 con 0x555820681200 2024-03-18T16:11:44.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.751+0000 7fbd70b89740 10 client.0 _ll_register_callbacks cb 0x5558205dc220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:11:44.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.752+0000 7fbd67400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555820680810 0x55582068ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:44.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.752+0000 7fbd67400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555820680810 0x55582068ad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:49232/0 (socket says 172.21.15.39:49232) 2024-03-18T16:11:44.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.752+0000 7fbd67400640 1 -- 172.21.15.39:0/1332311995 learned_addr learned my addr 172.21.15.39:0/1332311995 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:11:44.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.752+0000 7fbd67400640 1 --2- 172.21.15.39:0/1332311995 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555820686370 0x555820692310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:44.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.753+0000 7fbd67400640 1 -- 172.21.15.39:0/1332311995 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555820681200 msgr2=0x55582068b2d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T16:11:44.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.753+0000 7fbd67400640 1 --2- 172.21.15.39:0/1332311995 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555820681200 0x55582068b2d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:44.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.753+0000 7fbd67400640 1 -- 172.21.15.39:0/1332311995 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555820680810 msgr2=0x55582068ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:44.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.753+0000 7fbd67400640 1 --2- 172.21.15.39:0/1332311995 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555820680810 0x55582068ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:44.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.753+0000 7fbd67400640 1 -- 172.21.15.39:0/1332311995 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5558202f93f0 con 0x555820686370 2024-03-18T16:11:44.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.754+0000 7fbd67400640 1 --2- 172.21.15.39:0/1332311995 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555820686370 0x555820692310 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fbd60000f80 tx=0x7fbd60005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29e0479bdf7a3559 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:44.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.754+0000 7fbd50c00640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-03-18T16:11:44.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.755+0000 7fbd50c00640 1 -- 172.21.15.39:0/1332311995 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd6000fc20 con 0x555820686370 2024-03-18T16:11:44.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.755+0000 7fbd50c00640 1 -- 172.21.15.39:0/1332311995 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fbd60004a40 con 0x555820686370 2024-03-18T16:11:44.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.755+0000 7fbd70b89740 1 -- 172.21.15.39:0/1332311995 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbd60003e50 con 0x555820686370 2024-03-18T16:11:44.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.755+0000 7fbd50c00640 1 -- 172.21.15.39:0/1332311995 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fbd60033050 con 0x555820686370 2024-03-18T16:11:44.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.755+0000 7fbd70b89740 10 client.15735 fetch_fsmap learned FSMap version 30 2024-03-18T16:11:44.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.756+0000 7fbd70b89740 1 -- 172.21.15.39:0/1332311995 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5558202b90f0 con 0x555820686370 2024-03-18T16:11:44.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.756+0000 7fbd50c00640 1 -- 172.21.15.39:0/1332311995 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd60003e50 con 0x555820686370 2024-03-18T16:11:44.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.756+0000 7fbd50c00640 1 -- 172.21.15.39:0/1332311995 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7fbd60025020 con 0x555820686370 2024-03-18T16:11:44.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.757+0000 7fbd70b89740 10 client.15735 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T16:11:44.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.757+0000 7fbd70b89740 10 client.15735 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:11:44.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.757+0000 7fbd70b89740 20 client.15735 populate_metadata read hostname 'smithi039' 2024-03-18T16:11:44.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.757+0000 7fbd70b89740 1 --2- 172.21.15.39:0/1332311995 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5558206969b0 0x555820698dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:44.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.757+0000 7fbd70b89740 4 client.15735 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:11:44.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.757+0000 7fbd70b89740 1 -- 172.21.15.39:0/1332311995 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5558205f8be0 con 0x5558206969b0 2024-03-18T16:11:44.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.757+0000 7fbd66000640 1 --2- 172.21.15.39:0/1332311995 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5558206969b0 0x555820698dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:44.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.758+0000 7fbd66000640 1 --2- 172.21.15.39:0/1332311995 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5558206969b0 0x555820698dc0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5a16b9452b618199 server_cookie=5305f20c8a95b8df in_seq=0 out_seq=0 2024-03-18T16:11:44.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.758+0000 7fbd50c00640 10 client.15735 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:11:44.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.758+0000 7fbd50c00640 1 -- 172.21.15.39:0/1332311995 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5558205f8be0 con 0x5558206969b0 2024-03-18T16:11:44.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.758+0000 7fbd50c00640 10 client.15735 handle_command_reply: tid=0 2024-03-18T16:11:44.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.759+0000 7fbd70b89740 1 client.15735 shutdown 2024-03-18T16:11:44.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.759+0000 7fbd70b89740 1 -- 172.21.15.39:0/1332311995 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555820686370 msgr2=0x555820692310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:44.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.759+0000 7fbd70b89740 1 --2- 172.21.15.39:0/1332311995 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555820686370 0x555820692310 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fbd60000f80 tx=0x7fbd60005fd0 comp rx=0 tx=0).stop 2024-03-18T16:11:44.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.759+0000 7fbd70b89740 1 -- 172.21.15.39:0/1332311995 shutdown_connections 2024-03-18T16:11:44.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.759+0000 7fbd70b89740 1 --2- 172.21.15.39:0/1332311995 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5558206969b0 0x555820698dc0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:44.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.759+0000 7fbd70b89740 1 --2- 172.21.15.39:0/1332311995 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555820686370 0x555820692310 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:44.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.759+0000 7fbd70b89740 1 --2- 172.21.15.39:0/1332311995 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555820681200 0x55582068b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:44.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.759+0000 7fbd70b89740 1 --2- 172.21.15.39:0/1332311995 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555820680810 0x55582068ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:44.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.759+0000 7fbd70b89740 1 -- 172.21.15.39:0/1332311995 >> 172.21.15.39:0/1332311995 conn(0x55582067bd80 msgr2=0x55582067aec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:44.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.759+0000 7fbd50c00640 0 client.15735 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:11:44.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.759+0000 7fbd70b89740 1 -- 172.21.15.39:0/1332311995 shutdown_connections 2024-03-18T16:11:44.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.759+0000 7fbd70b89740 1 -- 172.21.15.39:0/1332311995 wait complete. 2024-03-18T16:11:44.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.760+0000 7fbd70b89740 20 client.15735 trim_cache size 0 max 16384 2024-03-18T16:11:44.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.790+0000 7fbd70b89740 1 Processor -- start 2024-03-18T16:11:44.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.791+0000 7fbd70b89740 1 -- start start 2024-03-18T16:11:44.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.791+0000 7fbd70b89740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555820680810 0x555820689870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:44.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.791+0000 7fbd70b89740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555820681200 0x5558206922d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:44.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.791+0000 7fbd70b89740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555820686370 0x555820682740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:44.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.791+0000 7fbd70b89740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fbd681b0b30 con 0x555820681200 2024-03-18T16:11:44.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.791+0000 7fbd70b89740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fbd681b26d0 con 0x555820686370 2024-03-18T16:11:44.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.791+0000 7fbd70b89740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fbd681b1200 con 0x555820680810 2024-03-18T16:11:44.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.792+0000 7fbd67400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555820680810 0x555820689870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:44.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.792+0000 7fbd67400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555820680810 0x555820689870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:33640/0 (socket says 172.21.15.39:33640) 2024-03-18T16:11:44.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.792+0000 7fbd67400640 1 -- 172.21.15.39:0/1434106741 learned_addr learned my addr 172.21.15.39:0/1434106741 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:11:44.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.792+0000 7fbd67400640 1 --2- 172.21.15.39:0/1434106741 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555820686370 0x555820682740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:44.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.792+0000 7fbd66000640 1 --2- 172.21.15.39:0/1434106741 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555820681200 0x5558206922d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:44.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.792+0000 7fbd67400640 1 -- 172.21.15.39:0/1434106741 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555820686370 msgr2=0x555820682740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:44.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.792+0000 7fbd67400640 1 --2- 172.21.15.39:0/1434106741 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555820686370 0x555820682740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:44.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.792+0000 7fbd67400640 1 -- 172.21.15.39:0/1434106741 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555820681200 msgr2=0x5558206922d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:44.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.792+0000 7fbd67400640 1 --2- 172.21.15.39:0/1434106741 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555820681200 0x5558206922d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:44.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.793+0000 7fbd67400640 1 -- 172.21.15.39:0/1434106741 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555820676370 con 0x555820680810 2024-03-18T16:11:44.795 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.793+0000 7fbd67400640 1 --2- 172.21.15.39:0/1434106741 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555820680810 0x555820689870 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fbd60005f80 tx=0x7fbd60005f10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=570cf26649c9e505 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:44.796 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.795+0000 7fbd53400640 1 -- 172.21.15.39:0/1434106741 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd6000fc20 con 0x555820680810 2024-03-18T16:11:44.797 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.795+0000 7fbd53400640 1 -- 172.21.15.39:0/1434106741 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd60005850 con 0x555820680810 2024-03-18T16:11:44.797 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.795+0000 7fbd53400640 1 -- 172.21.15.39:0/1434106741 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd6001ab60 con 0x555820680810 2024-03-18T16:11:44.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.798+0000 7fbd70b89740 1 -- 172.21.15.39:0/1434106741 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555820680810 msgr2=0x555820689870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:44.800 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.798+0000 7fbd70b89740 1 --2- 172.21.15.39:0/1434106741 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555820680810 0x555820689870 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fbd60005f80 tx=0x7fbd60005f10 comp rx=0 tx=0).stop 2024-03-18T16:11:44.800 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.799+0000 7fbd70b89740 1 -- 172.21.15.39:0/1434106741 shutdown_connections 2024-03-18T16:11:44.801 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.799+0000 7fbd70b89740 1 --2- 172.21.15.39:0/1434106741 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555820686370 0x555820682740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:44.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.799+0000 7fbd70b89740 1 --2- 172.21.15.39:0/1434106741 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555820681200 0x5558206922d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:44.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.799+0000 7fbd70b89740 1 --2- 172.21.15.39:0/1434106741 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555820680810 0x555820689870 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:44.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.799+0000 7fbd70b89740 1 -- 172.21.15.39:0/1434106741 >> 172.21.15.39:0/1434106741 conn(0x55582067bd80 msgr2=0x55582068ad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:44.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.800+0000 7fbd70b89740 1 -- 172.21.15.39:0/1434106741 shutdown_connections 2024-03-18T16:11:44.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.800+0000 7fbd70b89740 1 -- 172.21.15.39:0/1434106741 wait complete. 2024-03-18T16:11:44.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.801+0000 7fbd70b89740 1 -- start start 2024-03-18T16:11:44.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.801+0000 7fbd70b89740 1 Processor -- start 2024-03-18T16:11:44.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.802+0000 7fbd70b89740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555820680810 0x55582068d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:44.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.802+0000 7fbd70b89740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555820681200 0x55582068da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:44.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.803+0000 7fbd70b89740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555820686370 0x555820690a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:44.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.803+0000 7fbd70b89740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fbd681b1fa0 con 0x555820681200 2024-03-18T16:11:44.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.803+0000 7fbd70b89740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fbd681b18d0 con 0x555820680810 2024-03-18T16:11:44.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.803+0000 7fbd70b89740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fbd681aeb10 con 0x555820686370 2024-03-18T16:11:44.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.803+0000 7fbd70b89740 10 client.0 _ll_register_callbacks cb 0x7fbd580d2c30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:11:44.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.803+0000 7fbd67400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555820680810 0x55582068d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:44.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.804+0000 7fbd67400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555820680810 0x55582068d4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:49246/0 (socket says 172.21.15.39:49246) 2024-03-18T16:11:44.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.804+0000 7fbd67400640 1 -- 172.21.15.39:0/633349133 learned_addr learned my addr 172.21.15.39:0/633349133 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:11:44.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.805+0000 7fbd67400640 1 --2- 172.21.15.39:0/633349133 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555820686370 0x555820690a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:44.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.807+0000 7fbd66000640 1 --2- 172.21.15.39:0/633349133 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555820681200 0x55582068da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:44.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.808+0000 7fbd67400640 1 -- 172.21.15.39:0/633349133 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555820686370 msgr2=0x555820690a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:44.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.808+0000 7fbd67400640 1 --2- 172.21.15.39:0/633349133 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555820686370 0x555820690a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:44.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.808+0000 7fbd67400640 1 -- 172.21.15.39:0/633349133 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555820681200 msgr2=0x55582068da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:44.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.809+0000 7fbd67400640 1 --2- 172.21.15.39:0/633349133 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555820681200 0x55582068da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:44.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.809+0000 7fbd67400640 1 -- 172.21.15.39:0/633349133 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555820691010 con 0x555820680810 2024-03-18T16:11:44.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.809+0000 7fbd66000640 1 --2- 172.21.15.39:0/633349133 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555820681200 0x55582068da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:11:44.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.810+0000 7fbd67400640 1 --2- 172.21.15.39:0/633349133 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555820680810 0x55582068d4c0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fbd60011620 tx=0x7fbd6000f9a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1838366fc41b66df server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:44.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.810+0000 7fbd50c00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:11:44.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.810+0000 7fbd50c00640 1 -- 172.21.15.39:0/633349133 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd6001a440 con 0x555820680810 2024-03-18T16:11:44.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.810+0000 7fbd50c00640 1 -- 172.21.15.39:0/633349133 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fbd6001a5e0 con 0x555820680810 2024-03-18T16:11:44.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.811+0000 7fbd70b89740 1 -- 172.21.15.39:0/633349133 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbd60005850 con 0x555820680810 2024-03-18T16:11:44.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.812+0000 7fbd50c00640 1 -- 172.21.15.39:0/633349133 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fbd6001a7e0 con 0x555820680810 2024-03-18T16:11:44.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.812+0000 7fbd70b89740 10 client.25898 fetch_fsmap learned FSMap version 30 2024-03-18T16:11:44.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.812+0000 7fbd70b89740 1 -- 172.21.15.39:0/633349133 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x555820692320 con 0x555820680810 2024-03-18T16:11:44.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.813+0000 7fbd50c00640 1 -- 172.21.15.39:0/633349133 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd60005850 con 0x555820680810 2024-03-18T16:11:44.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.814+0000 7fbd50c00640 1 -- 172.21.15.39:0/633349133 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7fbd60025040 con 0x555820680810 2024-03-18T16:11:44.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.814+0000 7fbd70b89740 10 client.25898 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T16:11:44.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.814+0000 7fbd70b89740 10 client.25898 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:11:44.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.814+0000 7fbd70b89740 20 client.25898 populate_metadata read hostname 'smithi039' 2024-03-18T16:11:44.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.814+0000 7fbd70b89740 1 --2- 172.21.15.39:0/633349133 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55582070e000 0x5558207103f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:44.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.817+0000 7fbd66000640 1 --2- 172.21.15.39:0/633349133 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55582070e000 0x5558207103f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:44.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.817+0000 7fbd70b89740 4 client.25898 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T16:11:44.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.817+0000 7fbd70b89740 1 -- 172.21.15.39:0/633349133 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5558205f2730 con 0x55582070e000 2024-03-18T16:11:44.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.822+0000 7fbd50c00640 10 client.25898 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:11:44.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.823+0000 7fbd66000640 1 --2- 172.21.15.39:0/633349133 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55582070e000 0x5558207103f0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a7ede8b0c8da4df6 server_cookie=15c504721c39f6ef in_seq=0 out_seq=0 2024-03-18T16:11:44.825 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.823+0000 7fbd50c00640 1 -- 172.21.15.39:0/633349133 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5558205f2730 con 0x55582070e000 2024-03-18T16:11:44.825 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.824+0000 7fbd50c00640 10 client.25898 handle_command_reply: tid=0 2024-03-18T16:11:44.825 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.824+0000 7fbd70b89740 1 client.25898 shutdown 2024-03-18T16:11:44.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.828+0000 7fbd70b89740 1 -- 172.21.15.39:0/633349133 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555820680810 msgr2=0x55582068d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:44.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.828+0000 7fbd70b89740 1 --2- 172.21.15.39:0/633349133 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555820680810 0x55582068d4c0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fbd60011620 tx=0x7fbd6000f9a0 comp rx=0 tx=0).stop 2024-03-18T16:11:44.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.828+0000 7fbd70b89740 1 -- 172.21.15.39:0/633349133 shutdown_connections 2024-03-18T16:11:44.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.828+0000 7fbd70b89740 1 --2- 172.21.15.39:0/633349133 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55582070e000 0x5558207103f0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:44.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.829+0000 7fbd50c00640 0 client.25898 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:11:44.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.829+0000 7fbd70b89740 1 --2- 172.21.15.39:0/633349133 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555820686370 0x555820690a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:44.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.829+0000 7fbd70b89740 1 --2- 172.21.15.39:0/633349133 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555820681200 0x55582068da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:44.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.829+0000 7fbd70b89740 1 --2- 172.21.15.39:0/633349133 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555820680810 0x55582068d4c0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:44.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.829+0000 7fbd70b89740 1 -- 172.21.15.39:0/633349133 >> 172.21.15.39:0/633349133 conn(0x55582067bd80 msgr2=0x55582068ad90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:44.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.830+0000 7fbd70b89740 1 -- 172.21.15.39:0/633349133 shutdown_connections 2024-03-18T16:11:44.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.830+0000 7fbd70b89740 1 -- 172.21.15.39:0/633349133 wait complete. 2024-03-18T16:11:44.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.831+0000 7fbd70b89740 20 client.25898 trim_cache size 0 max 16384 2024-03-18T16:11:44.835 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-03-18T16:11:44.835 INFO:teuthology.orchestra.run.smithi039.stdout: "status": "no active scrubs running", 2024-03-18T16:11:44.836 INFO:teuthology.orchestra.run.smithi039.stdout: "scrubs": {} 2024-03-18T16:11:44.836 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-03-18T16:11:44.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.835+0000 7fbd6de00640 1 -- 172.21.15.39:0/441564750 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fbd4c0a2880 msgr2=0x7fbd4c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:44.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.835+0000 7fbd6de00640 1 --2- 172.21.15.39:0/441564750 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fbd4c0a2880 0x7fbd4c0a4d40 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x555820681600 tx=0x7fbd580d2d60 comp rx=0 tx=0).stop 2024-03-18T16:11:44.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.835+0000 7fbd6de00640 1 -- 172.21.15.39:0/441564750 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbd68074240 msgr2=0x7fbd6819e150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:44.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.835+0000 7fbd6de00640 1 --2- 172.21.15.39:0/441564750 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbd68074240 0x7fbd6819e150 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fbd5800e9f0 tx=0x7fbd5800eeb0 comp rx=0 tx=0).stop 2024-03-18T16:11:44.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.839+0000 7fbd6de00640 1 -- 172.21.15.39:0/441564750 shutdown_connections 2024-03-18T16:11:44.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.839+0000 7fbd6de00640 1 --2- 172.21.15.39:0/441564750 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbd6810ddc0 0x7fbd681a5700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:44.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.839+0000 7fbd6de00640 1 --2- 172.21.15.39:0/441564750 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fbd4c0a2880 0x7fbd4c0a4d40 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:44.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.839+0000 7fbd6de00640 1 --2- 172.21.15.39:0/441564750 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbd68075820 0x7fbd6819e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:44.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.839+0000 7fbd6de00640 1 --2- 172.21.15.39:0/441564750 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbd68074240 0x7fbd6819e150 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:44.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.839+0000 7fbd6de00640 1 -- 172.21.15.39:0/441564750 >> 172.21.15.39:0/441564750 conn(0x7fbd680fcd80 msgr2=0x7fbd68101160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:44.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.839+0000 7fbd6de00640 1 -- 172.21.15.39:0/441564750 shutdown_connections 2024-03-18T16:11:44.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:44.839+0000 7fbd6de00640 1 -- 172.21.15.39:0/441564750 wait complete. 2024-03-18T16:11:44.859 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-18T16:11:44.860 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:11:45.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.115+0000 7fd993a00640 1 -- 172.21.15.39:0/2814145251 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd98c109140 msgr2=0x7fd98c10fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:45.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.115+0000 7fd993a00640 1 --2- 172.21.15.39:0/2814145251 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd98c109140 0x7fd98c10fb20 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fd97c009900 tx=0x7fd97c02bc30 comp rx=0 tx=0).stop 2024-03-18T16:11:45.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.115+0000 7fd993a00640 1 -- 172.21.15.39:0/2814145251 shutdown_connections 2024-03-18T16:11:45.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.115+0000 7fd993a00640 1 --2- 172.21.15.39:0/2814145251 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd98c109140 0x7fd98c10fb20 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:45.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.115+0000 7fd993a00640 1 --2- 172.21.15.39:0/2814145251 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd98c110060 0x7fd98c112450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:45.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.115+0000 7fd993a00640 1 --2- 172.21.15.39:0/2814145251 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd98c1087e0 0x7fd98c108c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:45.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.115+0000 7fd993a00640 1 -- 172.21.15.39:0/2814145251 >> 172.21.15.39:0/2814145251 conn(0x7fd98c06e640 msgr2=0x7fd98c070a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:45.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.116+0000 7fd993a00640 1 -- 172.21.15.39:0/2814145251 shutdown_connections 2024-03-18T16:11:45.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.116+0000 7fd993a00640 1 -- 172.21.15.39:0/2814145251 wait complete. 2024-03-18T16:11:45.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.117+0000 7fd993a00640 1 Processor -- start 2024-03-18T16:11:45.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.117+0000 7fd993a00640 1 -- start start 2024-03-18T16:11:45.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.118+0000 7fd993a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd98c1087e0 0x7fd98c1a6910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:45.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.118+0000 7fd993a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd98c109140 0x7fd98c1a6e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:45.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.118+0000 7fd993a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd98c110060 0x7fd98c1adec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:45.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.118+0000 7fd993a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fd98c117ea0 con 0x7fd98c110060 2024-03-18T16:11:45.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.118+0000 7fd993a00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fd98c117d20 con 0x7fd98c1087e0 2024-03-18T16:11:45.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.118+0000 7fd993a00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fd98c118020 con 0x7fd98c109140 2024-03-18T16:11:45.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.119+0000 7fd991000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd98c1087e0 0x7fd98c1a6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:45.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.119+0000 7fd991000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd98c1087e0 0x7fd98c1a6910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:49274/0 (socket says 172.21.15.39:49274) 2024-03-18T16:11:45.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.119+0000 7fd991000640 1 -- 172.21.15.39:0/135412511 learned_addr learned my addr 172.21.15.39:0/135412511 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:11:45.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.119+0000 7fd991a00640 1 --2- 172.21.15.39:0/135412511 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd98c110060 0x7fd98c1adec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:45.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.120+0000 7fd991000640 1 -- 172.21.15.39:0/135412511 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd98c109140 msgr2=0x7fd98c1a6e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:45.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.120+0000 7fd991000640 1 --2- 172.21.15.39:0/135412511 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd98c109140 0x7fd98c1a6e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:45.124 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.120+0000 7fd991000640 1 -- 172.21.15.39:0/135412511 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd98c110060 msgr2=0x7fd98c1adec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:45.124 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.120+0000 7fd991000640 1 --2- 172.21.15.39:0/135412511 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd98c110060 0x7fd98c1adec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:45.124 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.120+0000 7fd991000640 1 -- 172.21.15.39:0/135412511 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd98c1ae530 con 0x7fd98c1087e0 2024-03-18T16:11:45.124 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.120+0000 7fd991a00640 1 --2- 172.21.15.39:0/135412511 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd98c110060 0x7fd98c1adec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:11:45.124 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.120+0000 7fd991000640 1 --2- 172.21.15.39:0/135412511 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd98c1087e0 0x7fd98c1a6910 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fd98400eab0 tx=0x7fd98400ef70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:45.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.121+0000 7fd989600640 1 -- 172.21.15.39:0/135412511 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd98400ce40 con 0x7fd98c1087e0 2024-03-18T16:11:45.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.121+0000 7fd989600640 1 -- 172.21.15.39:0/135412511 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd984011950 con 0x7fd98c1087e0 2024-03-18T16:11:45.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.121+0000 7fd989600640 1 -- 172.21.15.39:0/135412511 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd984010aa0 con 0x7fd98c1087e0 2024-03-18T16:11:45.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.122+0000 7fd993a00640 1 -- 172.21.15.39:0/135412511 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd98c1ae820 con 0x7fd98c1087e0 2024-03-18T16:11:45.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.122+0000 7fd993a00640 1 -- 172.21.15.39:0/135412511 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd98c1aecf0 con 0x7fd98c1087e0 2024-03-18T16:11:45.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.124+0000 7fd993a00640 1 -- 172.21.15.39:0/135412511 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd98c068640 con 0x7fd98c1087e0 2024-03-18T16:11:45.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.128+0000 7fd989600640 1 -- 172.21.15.39:0/135412511 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fd984011b00 con 0x7fd98c1087e0 2024-03-18T16:11:45.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.133+0000 7fd989600640 1 --2- 172.21.15.39:0/135412511 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd9700a2950 0x7fd9700a4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:45.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.134+0000 7fd98be00640 1 --2- 172.21.15.39:0/135412511 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd9700a2950 0x7fd9700a4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:45.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.134+0000 7fd989600640 1 -- 172.21.15.39:0/135412511 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fd9840cf1b0 con 0x7fd98c1087e0 2024-03-18T16:11:45.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.135+0000 7fd98be00640 1 --2- 172.21.15.39:0/135412511 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd9700a2950 0x7fd9700a4e10 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fd97c036040 tx=0x7fd97c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:45.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.136+0000 7fd989600640 1 -- 172.21.15.39:0/135412511 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd984016030 con 0x7fd98c1087e0 2024-03-18T16:11:45.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:11:44 smithi039 ceph-mon[31721]: pgmap v472: 129 pgs: 129 active+clean; 2.0 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 169 MiB/s wr, 97 op/s 2024-03-18T16:11:45.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:11:44 smithi196 ceph-mon[36307]: pgmap v472: 129 pgs: 129 active+clean; 2.0 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 169 MiB/s wr, 97 op/s 2024-03-18T16:11:45.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:11:44 smithi194 ceph-mon[36200]: pgmap v472: 129 pgs: 129 active+clean; 2.0 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 169 MiB/s wr, 97 op/s 2024-03-18T16:11:45.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.580+0000 7fd993a00640 1 -- 172.21.15.39:0/135412511 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd98c1af2e0 con 0x7fd98c1087e0 2024-03-18T16:11:45.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.581+0000 7fd989600640 1 -- 172.21.15.39:0/135412511 <== mon.2 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11935 (secure 0 0 0) 0x7fd98409bfc0 con 0x7fd98c1087e0 2024-03-18T16:11:45.582 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:11:45.583 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2520226608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2520226608},{"type":"v1","addr":"172.21.15.39:6835","nonce":2520226608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/719341763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":719341763},{"type":"v1","addr":"172.21.15.39:6837","nonce":719341763}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":30,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T16:06:44.180425+0000","modified":"2024-03-18T16:11:37.432194+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24575,"mds_1":24556,"mds_2":24535,"mds_3":24524,"mds_4":14463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14445":{"gid":14445,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.39:6839/656513813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":656513813},{"type":"v1","addr":"172.21.15.39:6839","nonce":656513813}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14463":{"gid":14463,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":13,"addr":"172.21.15.39:6841/3515326042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3515326042},{"type":"v1","addr":"172.21.15.39:6841","nonce":3515326042}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6833/931350738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":931350738},{"type":"v1","addr":"172.21.15.194:6833","nonce":931350738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6833/3874209621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":3874209621},{"type":"v1","addr":"172.21.15.196:6833","nonce":3874209621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6835/188385470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":188385470},{"type":"v1","addr":"172.21.15.194:6835","nonce":188385470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6835/3949984309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":3949984309},{"type":"v1","addr":"172.21.15.196:6835","nonce":3949984309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":15,"addr":"172.21.15.194:6837/1301464246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":1301464246},{"type":"v1","addr":"172.21.15.194:6837","nonce":1301464246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":16,"addr":"172.21.15.196:6837/895247251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":895247251},{"type":"v1","addr":"172.21.15.196:6837","nonce":895247251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.196:6839/2001427268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6838","nonce":2001427268},{"type":"v1","addr":"172.21.15.196:6839","nonce":2001427268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24575":{"gid":24575,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.194:6839/274911158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":274911158},{"type":"v1","addr":"172.21.15.194:6839","nonce":274911158}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24575,"qdb_cluster":[24524,14463,24575,24535,24556]},"id":1}]} 2024-03-18T16:11:45.583 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 30 2024-03-18T16:11:45.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.583+0000 7fd993a00640 1 -- 172.21.15.39:0/135412511 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd9700a2950 msgr2=0x7fd9700a4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:45.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.583+0000 7fd993a00640 1 --2- 172.21.15.39:0/135412511 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd9700a2950 0x7fd9700a4e10 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fd97c036040 tx=0x7fd97c000bc0 comp rx=0 tx=0).stop 2024-03-18T16:11:45.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.583+0000 7fd993a00640 1 -- 172.21.15.39:0/135412511 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd98c1087e0 msgr2=0x7fd98c1a6910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:45.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.583+0000 7fd993a00640 1 --2- 172.21.15.39:0/135412511 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd98c1087e0 0x7fd98c1a6910 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fd98400eab0 tx=0x7fd98400ef70 comp rx=0 tx=0).stop 2024-03-18T16:11:45.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.583+0000 7fd993a00640 1 -- 172.21.15.39:0/135412511 shutdown_connections 2024-03-18T16:11:45.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.583+0000 7fd993a00640 1 --2- 172.21.15.39:0/135412511 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd9700a2950 0x7fd9700a4e10 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:45.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.583+0000 7fd993a00640 1 --2- 172.21.15.39:0/135412511 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd98c110060 0x7fd98c1adec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:45.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.583+0000 7fd993a00640 1 --2- 172.21.15.39:0/135412511 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd98c109140 0x7fd98c1a6e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:45.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.583+0000 7fd993a00640 1 --2- 172.21.15.39:0/135412511 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd98c1087e0 0x7fd98c1a6910 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:45.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.583+0000 7fd993a00640 1 -- 172.21.15.39:0/135412511 >> 172.21.15.39:0/135412511 conn(0x7fd98c06e640 msgr2=0x7fd98c070320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:45.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.584+0000 7fd993a00640 1 -- 172.21.15.39:0/135412511 shutdown_connections 2024-03-18T16:11:45.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.584+0000 7fd993a00640 1 -- 172.21.15.39:0/135412511 wait complete. 2024-03-18T16:11:45.603 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-18T16:11:45.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.907+0000 7fba92200640 1 -- 172.21.15.39:0/2454212934 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fba8c1094d0 msgr2=0x7fba8c10b960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:45.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.907+0000 7fba92200640 1 --2- 172.21.15.39:0/2454212934 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fba8c1094d0 0x7fba8c10b960 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fba84009be0 tx=0x7fba8402bf80 comp rx=0 tx=0).stop 2024-03-18T16:11:45.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.907+0000 7fba92200640 1 -- 172.21.15.39:0/2454212934 shutdown_connections 2024-03-18T16:11:45.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.907+0000 7fba92200640 1 --2- 172.21.15.39:0/2454212934 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fba8c0741c0 0x7fba8c074620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:45.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.907+0000 7fba92200640 1 --2- 172.21.15.39:0/2454212934 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fba8c1094d0 0x7fba8c10b960 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:45.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.907+0000 7fba92200640 1 --2- 172.21.15.39:0/2454212934 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fba8c0757a0 0x7fba8c073bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:45.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.907+0000 7fba92200640 1 -- 172.21.15.39:0/2454212934 >> 172.21.15.39:0/2454212934 conn(0x7fba8c0fcb90 msgr2=0x7fba8c0fefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:45.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.908+0000 7fba92200640 1 -- 172.21.15.39:0/2454212934 shutdown_connections 2024-03-18T16:11:45.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.908+0000 7fba92200640 1 -- 172.21.15.39:0/2454212934 wait complete. 2024-03-18T16:11:45.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.909+0000 7fba92200640 1 Processor -- start 2024-03-18T16:11:45.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.909+0000 7fba92200640 1 -- start start 2024-03-18T16:11:45.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.909+0000 7fba92200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fba8c0741c0 0x7fba8c1049c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:45.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.909+0000 7fba92200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fba8c0757a0 0x7fba8c104f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:45.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.909+0000 7fba92200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fba8c1094d0 0x7fba8c101530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:45.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.909+0000 7fba92200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fba8c10f490 con 0x7fba8c0741c0 2024-03-18T16:11:45.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.909+0000 7fba92200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fba8c10f310 con 0x7fba8c1094d0 2024-03-18T16:11:45.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.909+0000 7fba92200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fba8c10f610 con 0x7fba8c0757a0 2024-03-18T16:11:45.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.910+0000 7fba91800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fba8c1094d0 0x7fba8c101530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:45.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.910+0000 7fba91800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fba8c1094d0 0x7fba8c101530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:49290/0 (socket says 172.21.15.39:49290) 2024-03-18T16:11:45.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.910+0000 7fba91800640 1 -- 172.21.15.39:0/1724128546 learned_addr learned my addr 172.21.15.39:0/1724128546 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:11:45.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.910+0000 7fba8be00640 1 --2- 172.21.15.39:0/1724128546 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fba8c0757a0 0x7fba8c104f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:45.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.910+0000 7fba91800640 1 -- 172.21.15.39:0/1724128546 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fba8c0757a0 msgr2=0x7fba8c104f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:45.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.910+0000 7fba91800640 1 --2- 172.21.15.39:0/1724128546 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fba8c0757a0 0x7fba8c104f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:45.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.910+0000 7fba91800640 1 -- 172.21.15.39:0/1724128546 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fba8c0741c0 msgr2=0x7fba8c1049c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:45.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.910+0000 7fba91800640 1 --2- 172.21.15.39:0/1724128546 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fba8c0741c0 0x7fba8c1049c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:45.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.910+0000 7fba91800640 1 -- 172.21.15.39:0/1724128546 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba8c101df0 con 0x7fba8c1094d0 2024-03-18T16:11:45.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.911+0000 7fba91800640 1 --2- 172.21.15.39:0/1724128546 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fba8c1094d0 0x7fba8c101530 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fba8402cc90 tx=0x7fba84004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:45.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.911+0000 7fba89600640 1 -- 172.21.15.39:0/1724128546 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba84043020 con 0x7fba8c1094d0 2024-03-18T16:11:45.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.911+0000 7fba89600640 1 -- 172.21.15.39:0/1724128546 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba840046a0 con 0x7fba8c1094d0 2024-03-18T16:11:45.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.911+0000 7fba89600640 1 -- 172.21.15.39:0/1724128546 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba840347b0 con 0x7fba8c1094d0 2024-03-18T16:11:45.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.911+0000 7fba92200640 1 -- 172.21.15.39:0/1724128546 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba8c1996c0 con 0x7fba8c1094d0 2024-03-18T16:11:45.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.912+0000 7fba92200640 1 -- 172.21.15.39:0/1724128546 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba8c199bc0 con 0x7fba8c1094d0 2024-03-18T16:11:45.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.916+0000 7fba96416740 1 Processor -- start 2024-03-18T16:11:45.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.916+0000 7fba89600640 1 -- 172.21.15.39:0/1724128546 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fba84046020 con 0x7fba8c1094d0 2024-03-18T16:11:45.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.917+0000 7fba96416740 1 -- start start 2024-03-18T16:11:45.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.917+0000 7fba96416740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56383fa35a90 0x56383fa35eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:45.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.917+0000 7fba96416740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56383fa36480 0x56383fa3af80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:45.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.917+0000 7fba96416740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56383fa3b5f0 0x56383fa3da80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:45.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.917+0000 7fba96416740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x56383f9fd3b0 con 0x56383fa35a90 2024-03-18T16:11:45.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.917+0000 7fba96416740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x56383f99a450 con 0x56383fa36480 2024-03-18T16:11:45.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.917+0000 7fba96416740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x56383f942cb0 con 0x56383fa3b5f0 2024-03-18T16:11:45.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.918+0000 7fba90e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56383fa36480 0x56383fa3af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:45.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.918+0000 7fba90e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56383fa36480 0x56383fa3af80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:49294/0 (socket says 172.21.15.39:49294) 2024-03-18T16:11:45.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.918+0000 7fba90e00640 1 -- 172.21.15.39:0/4278524324 learned_addr learned my addr 172.21.15.39:0/4278524324 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:11:45.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.918+0000 7fba90e00640 1 -- 172.21.15.39:0/4278524324 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56383fa3b5f0 msgr2=0x56383fa3da80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:11:45.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.918+0000 7fba90e00640 1 --2- 172.21.15.39:0/4278524324 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56383fa3b5f0 0x56383fa3da80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:45.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.918+0000 7fba90e00640 1 -- 172.21.15.39:0/4278524324 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56383fa35a90 msgr2=0x56383fa35eb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:11:45.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.920+0000 7fba8be00640 1 --2- 172.21.15.39:0/4278524324 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56383fa35a90 0x56383fa35eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:45.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.921+0000 7fba89600640 1 --2- 172.21.15.39:0/1724128546 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fba740a2800 0x7fba740a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:45.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.921+0000 7fba89600640 1 -- 172.21.15.39:0/1724128546 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fba840f1750 con 0x7fba8c1094d0 2024-03-18T16:11:45.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.922+0000 7fba90e00640 1 --2- 172.21.15.39:0/4278524324 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56383fa35a90 0x56383fa35eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:45.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.922+0000 7fba90e00640 1 -- 172.21.15.39:0/4278524324 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56383f9641f0 con 0x56383fa36480 2024-03-18T16:11:45.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.922+0000 7fba8be00640 1 --2- 172.21.15.39:0/4278524324 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56383fa35a90 0x56383fa35eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:11:45.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.922+0000 7fba90e00640 1 --2- 172.21.15.39:0/4278524324 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56383fa36480 0x56383fa3af80 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fba8000bb10 tx=0x7fba8000d2e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e4edf5c8f192a410 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:45.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.923+0000 7fba8be00640 1 --2- 172.21.15.39:0/1724128546 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fba740a2800 0x7fba740a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:45.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.923+0000 7fba7e000640 1 -- 172.21.15.39:0/4278524324 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba80016070 con 0x56383fa36480 2024-03-18T16:11:45.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.923+0000 7fba7e000640 1 -- 172.21.15.39:0/4278524324 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba80005500 con 0x56383fa36480 2024-03-18T16:11:45.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.923+0000 7fba7e000640 1 -- 172.21.15.39:0/4278524324 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba80010460 con 0x56383fa36480 2024-03-18T16:11:45.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.924+0000 7fba8be00640 1 --2- 172.21.15.39:0/1724128546 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fba740a2800 0x7fba740a4cc0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x56383fa36ce0 tx=0x7fba780046c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:45.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.924+0000 7fba96416740 1 -- 172.21.15.39:0/4278524324 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56383fa36480 msgr2=0x56383fa3af80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:45.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.924+0000 7fba96416740 1 --2- 172.21.15.39:0/4278524324 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56383fa36480 0x56383fa3af80 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fba8000bb10 tx=0x7fba8000d2e0 comp rx=0 tx=0).stop 2024-03-18T16:11:45.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.925+0000 7fba96416740 1 -- 172.21.15.39:0/4278524324 shutdown_connections 2024-03-18T16:11:45.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.925+0000 7fba96416740 1 --2- 172.21.15.39:0/4278524324 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56383fa36480 0x56383fa3af80 secure :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fba8000bb10 tx=0x7fba8000d2e0 comp rx=0 tx=0).stop 2024-03-18T16:11:45.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.925+0000 7fba96416740 1 --2- 172.21.15.39:0/4278524324 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56383fa3b5f0 0x56383fa3da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:45.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.925+0000 7fba96416740 1 --2- 172.21.15.39:0/4278524324 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56383fa35a90 0x56383fa35eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:45.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.925+0000 7fba96416740 1 -- 172.21.15.39:0/4278524324 >> 172.21.15.39:0/4278524324 conn(0x56383fa31000 msgr2=0x56383fa31400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:45.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.925+0000 7fba96416740 1 -- 172.21.15.39:0/4278524324 shutdown_connections 2024-03-18T16:11:45.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.925+0000 7fba96416740 1 -- 172.21.15.39:0/4278524324 wait complete. 2024-03-18T16:11:45.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.926+0000 7fba96416740 1 -- start start 2024-03-18T16:11:45.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.926+0000 7fba96416740 1 Processor -- start 2024-03-18T16:11:45.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.927+0000 7fba96416740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56383fa35a90 0x56383fa3ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:45.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.927+0000 7fba96416740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56383fa3b5f0 0x56383fa40510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:45.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.927+0000 7fba96416740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56383fa45540 0x56383fa47930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:45.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.927+0000 7fba96416740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x56383f956110 con 0x56383fa35a90 2024-03-18T16:11:45.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.927+0000 7fba96416740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x56383f9964f0 con 0x56383fa45540 2024-03-18T16:11:45.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.927+0000 7fba96416740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x56383fa3e5a0 con 0x56383fa3b5f0 2024-03-18T16:11:45.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.927+0000 7fba96416740 10 client.0 _ll_register_callbacks cb 0x56383f943b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:11:45.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.927+0000 7fba90e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56383fa35a90 0x56383fa3ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:45.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.928+0000 7fba90e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56383fa35a90 0x56383fa3ffd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43652/0 (socket says 172.21.15.39:43652) 2024-03-18T16:11:45.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.928+0000 7fba90e00640 1 -- 172.21.15.39:0/570418007 learned_addr learned my addr 172.21.15.39:0/570418007 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:11:45.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.928+0000 7fba8be00640 1 --2- 172.21.15.39:0/570418007 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56383fa3b5f0 0x56383fa40510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:45.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.928+0000 7fba91800640 1 --2- 172.21.15.39:0/570418007 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56383fa45540 0x56383fa47930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:45.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.928+0000 7fba90e00640 1 -- 172.21.15.39:0/570418007 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56383fa3b5f0 msgr2=0x56383fa40510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:45.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.928+0000 7fba90e00640 1 --2- 172.21.15.39:0/570418007 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56383fa3b5f0 0x56383fa40510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:45.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.928+0000 7fba90e00640 1 -- 172.21.15.39:0/570418007 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56383fa45540 msgr2=0x56383fa47930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:45.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.928+0000 7fba90e00640 1 --2- 172.21.15.39:0/570418007 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56383fa45540 0x56383fa47930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:45.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.928+0000 7fba90e00640 1 -- 172.21.15.39:0/570418007 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56383f6ae3f0 con 0x56383fa35a90 2024-03-18T16:11:45.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.929+0000 7fba90e00640 1 --2- 172.21.15.39:0/570418007 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56383fa35a90 0x56383fa3ffd0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x56383fa36ef0 tx=0x7fba80005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5de02a20d85c776e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:45.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.929+0000 7fba72a00640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-03-18T16:11:45.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.929+0000 7fba72a00640 1 -- 172.21.15.39:0/570418007 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba80016070 con 0x56383fa35a90 2024-03-18T16:11:45.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.929+0000 7fba72a00640 1 -- 172.21.15.39:0/570418007 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fba80005200 con 0x56383fa35a90 2024-03-18T16:11:45.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.929+0000 7fba96416740 1 -- 172.21.15.39:0/570418007 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fba80005500 con 0x56383fa35a90 2024-03-18T16:11:45.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.930+0000 7fba72a00640 1 -- 172.21.15.39:0/570418007 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fba8000dae0 con 0x56383fa35a90 2024-03-18T16:11:45.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.930+0000 7fba96416740 10 client.15750 fetch_fsmap learned FSMap version 30 2024-03-18T16:11:45.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.930+0000 7fba96416740 1 -- 172.21.15.39:0/570418007 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56383f7787d0 con 0x56383fa35a90 2024-03-18T16:11:45.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.930+0000 7fba72a00640 1 -- 172.21.15.39:0/570418007 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba80005500 con 0x56383fa35a90 2024-03-18T16:11:45.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.931+0000 7fba72a00640 1 -- 172.21.15.39:0/570418007 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7fba800038a0 con 0x56383fa35a90 2024-03-18T16:11:45.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.931+0000 7fba96416740 10 client.15750 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T16:11:45.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.931+0000 7fba96416740 10 client.15750 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-03-18T16:11:45.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.931+0000 7fba96416740 20 client.15750 populate_metadata read hostname 'smithi039' 2024-03-18T16:11:45.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.931+0000 7fba96416740 1 --2- 172.21.15.39:0/570418007 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x56383fa5cf90 0x56383fa5f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:45.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.931+0000 7fba96416740 4 client.15750 mds_command: new command op to 14463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:11:45.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.931+0000 7fba96416740 1 -- 172.21.15.39:0/570418007 --> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56383f9a7990 con 0x56383fa5cf90 2024-03-18T16:11:45.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.934+0000 7fba8be00640 1 --2- 172.21.15.39:0/570418007 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x56383fa5cf90 0x56383fa5f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:45.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.935+0000 7fba8be00640 1 --2- 172.21.15.39:0/570418007 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x56383fa5cf90 0x56383fa5f3e0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=5e29f7e2b1fdc1c server_cookie=96e1a1d3130a264d in_seq=0 out_seq=0 2024-03-18T16:11:45.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.935+0000 7fba72a00640 10 client.15750 ms_handle_connect on v2:172.21.15.39:6840/3515326042 2024-03-18T16:11:45.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.937+0000 7fba72a00640 1 -- 172.21.15.39:0/570418007 <== mds.4 v2:172.21.15.39:6840/3515326042 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56383f9a7990 con 0x56383fa5cf90 2024-03-18T16:11:45.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.937+0000 7fba72a00640 10 client.15750 handle_command_reply: tid=0 2024-03-18T16:11:45.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.937+0000 7fba96416740 1 client.15750 shutdown 2024-03-18T16:11:45.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.937+0000 7fba96416740 1 -- 172.21.15.39:0/570418007 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56383fa35a90 msgr2=0x56383fa3ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:45.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.937+0000 7fba96416740 1 --2- 172.21.15.39:0/570418007 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56383fa35a90 0x56383fa3ffd0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x56383fa36ef0 tx=0x7fba80005780 comp rx=0 tx=0).stop 2024-03-18T16:11:45.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.938+0000 7fba96416740 1 -- 172.21.15.39:0/570418007 shutdown_connections 2024-03-18T16:11:45.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.938+0000 7fba96416740 1 --2- 172.21.15.39:0/570418007 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x56383fa5cf90 0x56383fa5f3e0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:45.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.938+0000 7fba96416740 1 --2- 172.21.15.39:0/570418007 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56383fa45540 0x56383fa47930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:45.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.938+0000 7fba96416740 1 --2- 172.21.15.39:0/570418007 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56383fa3b5f0 0x56383fa40510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:45.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.938+0000 7fba96416740 1 --2- 172.21.15.39:0/570418007 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56383fa35a90 0x56383fa3ffd0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:45.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.938+0000 7fba72a00640 0 client.15750 ms_handle_reset on v2:172.21.15.39:6840/3515326042 2024-03-18T16:11:45.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.938+0000 7fba96416740 1 -- 172.21.15.39:0/570418007 >> 172.21.15.39:0/570418007 conn(0x56383fa31000 msgr2=0x56383fa30220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:45.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.939+0000 7fba96416740 1 -- 172.21.15.39:0/570418007 shutdown_connections 2024-03-18T16:11:45.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.939+0000 7fba96416740 1 -- 172.21.15.39:0/570418007 wait complete. 2024-03-18T16:11:45.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.940+0000 7fba96416740 20 client.15750 trim_cache size 0 max 16384 2024-03-18T16:11:45.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.975+0000 7fba96416740 1 Processor -- start 2024-03-18T16:11:45.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.976+0000 7fba96416740 1 -- start start 2024-03-18T16:11:45.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.976+0000 7fba96416740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56383fa3b5f0 0x56383fa3e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:45.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.976+0000 7fba96416740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56383fa45540 0x56383fa3ef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:45.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.976+0000 7fba96416740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56383fa2f990 0x56383fa2fe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:45.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.976+0000 7fba96416740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fba8c06d2e0 con 0x56383fa2f990 2024-03-18T16:11:45.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.976+0000 7fba96416740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fba8c06ee80 con 0x56383fa3b5f0 2024-03-18T16:11:45.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.976+0000 7fba96416740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fba8c06d9b0 con 0x56383fa45540 2024-03-18T16:11:45.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.977+0000 7fba8be00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56383fa45540 0x56383fa3ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:45.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.977+0000 7fba8be00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56383fa45540 0x56383fa3ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:33716/0 (socket says 172.21.15.39:33716) 2024-03-18T16:11:45.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.977+0000 7fba8be00640 1 -- 172.21.15.39:0/353552739 learned_addr learned my addr 172.21.15.39:0/353552739 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:11:45.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.977+0000 7fba90e00640 1 --2- 172.21.15.39:0/353552739 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56383fa3b5f0 0x56383fa3e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:45.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.977+0000 7fba8be00640 1 -- 172.21.15.39:0/353552739 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56383fa3b5f0 msgr2=0x56383fa3e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:45.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.977+0000 7fba8be00640 1 --2- 172.21.15.39:0/353552739 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56383fa3b5f0 0x56383fa3e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:45.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.977+0000 7fba8be00640 1 -- 172.21.15.39:0/353552739 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56383fa2f990 msgr2=0x56383fa2fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:45.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.977+0000 7fba91800640 1 --2- 172.21.15.39:0/353552739 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56383fa2f990 0x56383fa2fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:45.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.977+0000 7fba8be00640 1 --2- 172.21.15.39:0/353552739 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56383fa2f990 0x56383fa2fe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:45.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.977+0000 7fba8be00640 1 -- 172.21.15.39:0/353552739 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56383fa5fa60 con 0x56383fa45540 2024-03-18T16:11:45.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.978+0000 7fba91800640 1 --2- 172.21.15.39:0/353552739 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56383fa2f990 0x56383fa2fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:11:45.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.978+0000 7fba8be00640 1 --2- 172.21.15.39:0/353552739 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56383fa45540 0x56383fa3ef10 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fba8c102940 tx=0x7fba78015000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a91f3760dc5f5fb5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:45.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.979+0000 7fba7d600640 1 -- 172.21.15.39:0/353552739 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba7802f070 con 0x56383fa45540 2024-03-18T16:11:45.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.979+0000 7fba7d600640 1 -- 172.21.15.39:0/353552739 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba7801fde0 con 0x56383fa45540 2024-03-18T16:11:45.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.979+0000 7fba7d600640 1 -- 172.21.15.39:0/353552739 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba78012a90 con 0x56383fa45540 2024-03-18T16:11:45.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.981+0000 7fba96416740 1 -- 172.21.15.39:0/353552739 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56383fa45540 msgr2=0x56383fa3ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:45.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.981+0000 7fba96416740 1 --2- 172.21.15.39:0/353552739 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56383fa45540 0x56383fa3ef10 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fba8c102940 tx=0x7fba78015000 comp rx=0 tx=0).stop 2024-03-18T16:11:45.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.981+0000 7fba96416740 1 -- 172.21.15.39:0/353552739 shutdown_connections 2024-03-18T16:11:45.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.981+0000 7fba96416740 1 --2- 172.21.15.39:0/353552739 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56383fa2f990 0x56383fa2fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:45.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.981+0000 7fba96416740 1 --2- 172.21.15.39:0/353552739 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56383fa45540 0x56383fa3ef10 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:45.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.981+0000 7fba96416740 1 --2- 172.21.15.39:0/353552739 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56383fa3b5f0 0x56383fa3e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:45.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.981+0000 7fba96416740 1 -- 172.21.15.39:0/353552739 >> 172.21.15.39:0/353552739 conn(0x56383fa31000 msgr2=0x56383fa313e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:45.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.981+0000 7fba96416740 1 -- 172.21.15.39:0/353552739 shutdown_connections 2024-03-18T16:11:45.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.981+0000 7fba96416740 1 -- 172.21.15.39:0/353552739 wait complete. 2024-03-18T16:11:45.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.982+0000 7fba96416740 1 -- start start 2024-03-18T16:11:45.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.982+0000 7fba96416740 1 Processor -- start 2024-03-18T16:11:45.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.982+0000 7fba96416740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56383fa2f990 0x56383fa6dd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:45.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.982+0000 7fba96416740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56383fa3b5f0 0x56383fa6e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:45.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.982+0000 7fba96416740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56383fa45540 0x56383fa6f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:45.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.982+0000 7fba96416740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fba8c06e750 con 0x56383fa3b5f0 2024-03-18T16:11:45.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.982+0000 7fba96416740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fba8c06e080 con 0x56383fa2f990 2024-03-18T16:11:45.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.982+0000 7fba96416740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fba8c06b2c0 con 0x56383fa45540 2024-03-18T16:11:45.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.983+0000 7fba96416740 10 client.0 _ll_register_callbacks cb 0x7fba8000b960 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:11:45.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.983+0000 7fba8be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56383fa3b5f0 0x56383fa6e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:45.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.983+0000 7fba90e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56383fa2f990 0x56383fa6dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:45.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.983+0000 7fba90e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56383fa2f990 0x56383fa6dd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:49324/0 (socket says 172.21.15.39:49324) 2024-03-18T16:11:45.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.983+0000 7fba90e00640 1 -- 172.21.15.39:0/3669756059 learned_addr learned my addr 172.21.15.39:0/3669756059 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:11:45.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.983+0000 7fba8be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56383fa3b5f0 0x56383fa6e2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43672/0 (socket says 172.21.15.39:43672) 2024-03-18T16:11:45.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.983+0000 7fba91800640 1 --2- 172.21.15.39:0/3669756059 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56383fa45540 0x56383fa6f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:45.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.983+0000 7fba90e00640 1 -- 172.21.15.39:0/3669756059 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56383fa45540 msgr2=0x56383fa6f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:45.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.983+0000 7fba90e00640 1 --2- 172.21.15.39:0/3669756059 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56383fa45540 0x56383fa6f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:45.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.983+0000 7fba90e00640 1 -- 172.21.15.39:0/3669756059 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56383fa3b5f0 msgr2=0x56383fa6e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:45.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.983+0000 7fba90e00640 1 --2- 172.21.15.39:0/3669756059 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56383fa3b5f0 0x56383fa6e2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:45.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.984+0000 7fba90e00640 1 -- 172.21.15.39:0/3669756059 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56383fa6f8d0 con 0x56383fa2f990 2024-03-18T16:11:45.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.984+0000 7fba90e00640 1 --2- 172.21.15.39:0/3669756059 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56383fa2f990 0x56383fa6dd90 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fba80003ca0 tx=0x7fba80005fb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1504e4c114793a00 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:45.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.984+0000 7fba72a00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:11:45.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.984+0000 7fba72a00640 1 -- 172.21.15.39:0/3669756059 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba80016040 con 0x56383fa2f990 2024-03-18T16:11:45.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.984+0000 7fba72a00640 1 -- 172.21.15.39:0/3669756059 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fba800035f0 con 0x56383fa2f990 2024-03-18T16:11:45.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.984+0000 7fba96416740 1 -- 172.21.15.39:0/3669756059 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fba7801fde0 con 0x56383fa2f990 2024-03-18T16:11:45.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.985+0000 7fba72a00640 1 -- 172.21.15.39:0/3669756059 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fba8003d050 con 0x56383fa2f990 2024-03-18T16:11:45.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.985+0000 7fba96416740 10 client.25922 fetch_fsmap learned FSMap version 30 2024-03-18T16:11:45.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.985+0000 7fba96416740 1 -- 172.21.15.39:0/3669756059 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x56383fac2e40 con 0x56383fa2f990 2024-03-18T16:11:45.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.986+0000 7fba72a00640 1 -- 172.21.15.39:0/3669756059 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba7801fde0 con 0x56383fa2f990 2024-03-18T16:11:45.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.986+0000 7fba72a00640 1 -- 172.21.15.39:0/3669756059 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7fba800038c0 con 0x56383fa2f990 2024-03-18T16:11:45.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.986+0000 7fba96416740 10 client.25922 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T16:11:45.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.986+0000 7fba96416740 10 client.25922 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-03-18T16:11:45.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.986+0000 7fba96416740 20 client.25922 populate_metadata read hostname 'smithi039' 2024-03-18T16:11:45.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.986+0000 7fba96416740 1 --2- 172.21.15.39:0/3669756059 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x56383fac3790 0x56383fac5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:45.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.986+0000 7fba96416740 4 client.25922 mds_command: new command op to 14463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:11:45.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.986+0000 7fba96416740 1 -- 172.21.15.39:0/3669756059 --> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56383f9a1ce0 con 0x56383fac3790 2024-03-18T16:11:45.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.986+0000 7fba8be00640 1 --2- 172.21.15.39:0/3669756059 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x56383fac3790 0x56383fac5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:45.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.988+0000 7fba72a00640 10 client.25922 ms_handle_connect on v2:172.21.15.39:6840/3515326042 2024-03-18T16:11:45.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.988+0000 7fba8be00640 1 --2- 172.21.15.39:0/3669756059 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x56383fac3790 0x56383fac5bb0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=2f1d4f0d1ded3462 server_cookie=d0558881fd9761c4 in_seq=0 out_seq=0 2024-03-18T16:11:45.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.989+0000 7fba72a00640 1 -- 172.21.15.39:0/3669756059 <== mds.4 v2:172.21.15.39:6840/3515326042 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56383f9a1ce0 con 0x56383fac3790 2024-03-18T16:11:45.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.989+0000 7fba72a00640 10 client.25922 handle_command_reply: tid=0 2024-03-18T16:11:45.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.989+0000 7fba96416740 1 client.25922 shutdown 2024-03-18T16:11:45.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.989+0000 7fba96416740 1 -- 172.21.15.39:0/3669756059 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56383fa2f990 msgr2=0x56383fa6dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:45.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.989+0000 7fba96416740 1 --2- 172.21.15.39:0/3669756059 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56383fa2f990 0x56383fa6dd90 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fba80003ca0 tx=0x7fba80005fb0 comp rx=0 tx=0).stop 2024-03-18T16:11:45.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.989+0000 7fba96416740 1 -- 172.21.15.39:0/3669756059 shutdown_connections 2024-03-18T16:11:45.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.989+0000 7fba96416740 1 --2- 172.21.15.39:0/3669756059 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x56383fac3790 0x56383fac5bb0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:45.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.989+0000 7fba96416740 1 --2- 172.21.15.39:0/3669756059 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56383fa45540 0x56383fa6f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:45.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.989+0000 7fba96416740 1 --2- 172.21.15.39:0/3669756059 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56383fa3b5f0 0x56383fa6e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:45.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.989+0000 7fba96416740 1 --2- 172.21.15.39:0/3669756059 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56383fa2f990 0x56383fa6dd90 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:45.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.989+0000 7fba96416740 1 -- 172.21.15.39:0/3669756059 >> 172.21.15.39:0/3669756059 conn(0x56383fa31000 msgr2=0x56383fa3e9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:45.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.989+0000 7fba72a00640 0 client.25922 ms_handle_reset on v2:172.21.15.39:6840/3515326042 2024-03-18T16:11:45.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.989+0000 7fba96416740 1 -- 172.21.15.39:0/3669756059 shutdown_connections 2024-03-18T16:11:45.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.990+0000 7fba96416740 1 -- 172.21.15.39:0/3669756059 wait complete. 2024-03-18T16:11:45.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.990+0000 7fba96416740 20 client.25922 trim_cache size 0 max 16384 2024-03-18T16:11:45.994 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:11:45.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.991+0000 7fba92200640 1 -- 172.21.15.39:0/1724128546 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fba740a2800 msgr2=0x7fba740a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:45.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.991+0000 7fba92200640 1 --2- 172.21.15.39:0/1724128546 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fba740a2800 0x7fba740a4cc0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x56383fa36ce0 tx=0x7fba780046c0 comp rx=0 tx=0).stop 2024-03-18T16:11:45.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.991+0000 7fba92200640 1 -- 172.21.15.39:0/1724128546 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fba8c1094d0 msgr2=0x7fba8c101530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:45.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.991+0000 7fba92200640 1 --2- 172.21.15.39:0/1724128546 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fba8c1094d0 0x7fba8c101530 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fba8402cc90 tx=0x7fba84004270 comp rx=0 tx=0).stop 2024-03-18T16:11:45.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.991+0000 7fba92200640 1 -- 172.21.15.39:0/1724128546 shutdown_connections 2024-03-18T16:11:45.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.991+0000 7fba92200640 1 --2- 172.21.15.39:0/1724128546 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fba8c1094d0 0x7fba8c101530 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:45.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.991+0000 7fba92200640 1 --2- 172.21.15.39:0/1724128546 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fba740a2800 0x7fba740a4cc0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:45.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.991+0000 7fba92200640 1 --2- 172.21.15.39:0/1724128546 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fba8c0757a0 0x7fba8c104f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:45.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.991+0000 7fba92200640 1 --2- 172.21.15.39:0/1724128546 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fba8c0741c0 0x7fba8c1049c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:45.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.991+0000 7fba92200640 1 -- 172.21.15.39:0/1724128546 >> 172.21.15.39:0/1724128546 conn(0x7fba8c0fcb90 msgr2=0x7fba8c105460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:45.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.991+0000 7fba92200640 1 -- 172.21.15.39:0/1724128546 shutdown_connections 2024-03-18T16:11:45.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:45.991+0000 7fba92200640 1 -- 172.21.15.39:0/1724128546 wait complete. 2024-03-18T16:11:46.019 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-18T16:11:46.339 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:11:46 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/135412511' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:11:46.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.440+0000 7f3c1d800640 1 -- 172.21.15.39:0/112318680 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3c18109140 msgr2=0x7f3c1810fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:46.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.440+0000 7f3c1d800640 1 --2- 172.21.15.39:0/112318680 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3c18109140 0x7f3c1810fb20 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f3c00009900 tx=0x7f3c0002bc50 comp rx=0 tx=0).stop 2024-03-18T16:11:46.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.441+0000 7f3c1d800640 1 -- 172.21.15.39:0/112318680 shutdown_connections 2024-03-18T16:11:46.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.441+0000 7f3c1d800640 1 --2- 172.21.15.39:0/112318680 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3c18109140 0x7f3c1810fb20 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:46.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.441+0000 7f3c1d800640 1 --2- 172.21.15.39:0/112318680 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3c18110060 0x7f3c18112450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:46.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.441+0000 7f3c1d800640 1 --2- 172.21.15.39:0/112318680 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c181087e0 0x7f3c18108c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:46.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.441+0000 7f3c1d800640 1 -- 172.21.15.39:0/112318680 >> 172.21.15.39:0/112318680 conn(0x7f3c1806e9c0 msgr2=0x7f3c18070e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:46.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.441+0000 7f3c1d800640 1 -- 172.21.15.39:0/112318680 shutdown_connections 2024-03-18T16:11:46.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.441+0000 7f3c1d800640 1 -- 172.21.15.39:0/112318680 wait complete. 2024-03-18T16:11:46.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.442+0000 7f3c1d800640 1 Processor -- start 2024-03-18T16:11:46.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.442+0000 7f3c1d800640 1 -- start start 2024-03-18T16:11:46.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.443+0000 7f3c1d800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3c181087e0 0x7f3c1819e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:46.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.443+0000 7f3c1d800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3c18109140 0x7f3c1819e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:46.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.443+0000 7f3c1d800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c18110060 0x7f3c181a56a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:46.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.443+0000 7f3c1d800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f3c18113ac0 con 0x7f3c18110060 2024-03-18T16:11:46.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.443+0000 7f3c1d800640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f3c18113940 con 0x7f3c181087e0 2024-03-18T16:11:46.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.443+0000 7f3c1d800640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f3c18113c40 con 0x7f3c18109140 2024-03-18T16:11:46.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.443+0000 7f3c16a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3c181087e0 0x7f3c1819e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:46.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.443+0000 7f3c16a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3c181087e0 0x7f3c1819e190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:49340/0 (socket says 172.21.15.39:49340) 2024-03-18T16:11:46.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.443+0000 7f3c16a00640 1 -- 172.21.15.39:0/1362182200 learned_addr learned my addr 172.21.15.39:0/1362182200 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:11:46.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.443+0000 7f3c17400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c18110060 0x7f3c181a56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:46.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.443+0000 7f3c16000640 1 --2- 172.21.15.39:0/1362182200 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3c18109140 0x7f3c1819e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:46.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.444+0000 7f3c16a00640 1 -- 172.21.15.39:0/1362182200 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3c18109140 msgr2=0x7f3c1819e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:46.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.444+0000 7f3c16a00640 1 --2- 172.21.15.39:0/1362182200 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3c18109140 0x7f3c1819e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:46.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.444+0000 7f3c16a00640 1 -- 172.21.15.39:0/1362182200 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c18110060 msgr2=0x7f3c181a56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:46.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.444+0000 7f3c16a00640 1 --2- 172.21.15.39:0/1362182200 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c18110060 0x7f3c181a56a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:46.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.444+0000 7f3c16a00640 1 -- 172.21.15.39:0/1362182200 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c181a5d10 con 0x7f3c181087e0 2024-03-18T16:11:46.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.444+0000 7f3c17400640 1 --2- 172.21.15.39:0/1362182200 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c18110060 0x7f3c181a56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:11:46.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.444+0000 7f3c16000640 1 --2- 172.21.15.39:0/1362182200 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3c18109140 0x7f3c1819e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:11:46.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.444+0000 7f3c16a00640 1 --2- 172.21.15.39:0/1362182200 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3c181087e0 0x7f3c1819e190 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f3c0800ebc0 tx=0x7f3c0800c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:46.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.445+0000 7f3c0f400640 1 -- 172.21.15.39:0/1362182200 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c08010630 con 0x7f3c181087e0 2024-03-18T16:11:46.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.445+0000 7f3c1d800640 1 -- 172.21.15.39:0/1362182200 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c181a5fa0 con 0x7f3c181087e0 2024-03-18T16:11:46.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.445+0000 7f3c1d800640 1 -- 172.21.15.39:0/1362182200 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c181a64b0 con 0x7f3c181087e0 2024-03-18T16:11:46.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.445+0000 7f3c0f400640 1 -- 172.21.15.39:0/1362182200 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c08010cb0 con 0x7f3c181087e0 2024-03-18T16:11:46.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.445+0000 7f3c0f400640 1 -- 172.21.15.39:0/1362182200 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c08018bd0 con 0x7f3c181087e0 2024-03-18T16:11:46.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.449+0000 7f3c205db740 1 Processor -- start 2024-03-18T16:11:46.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.449+0000 7f3c205db740 1 -- start start 2024-03-18T16:11:46.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.450+0000 7f3c205db740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55741f1ae0d0 0x55741f1ae4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:46.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.450+0000 7f3c205db740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55741f1aeac0 0x55741f1b35c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:46.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.450+0000 7f3c205db740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55741f1b3c30 0x55741f1b60c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:46.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.450+0000 7f3c205db740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55741f16d3b0 con 0x55741f1ae0d0 2024-03-18T16:11:46.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.450+0000 7f3c205db740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55741f10a450 con 0x55741f1aeac0 2024-03-18T16:11:46.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.450+0000 7f3c205db740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55741f0b2cb0 con 0x55741f1b3c30 2024-03-18T16:11:46.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.451+0000 7f3c17400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55741f1ae0d0 0x55741f1ae4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:46.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.451+0000 7f3c17400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55741f1ae0d0 0x55741f1ae4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43706/0 (socket says 172.21.15.39:43706) 2024-03-18T16:11:46.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.451+0000 7f3c17400640 1 -- 172.21.15.39:0/3365369780 learned_addr learned my addr 172.21.15.39:0/3365369780 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:11:46.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.451+0000 7f3c16000640 1 --2- 172.21.15.39:0/3365369780 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55741f1b3c30 0x55741f1b60c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:46.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.452+0000 7f3c17400640 1 -- 172.21.15.39:0/3365369780 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55741f1b3c30 msgr2=0x55741f1b60c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:46.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.452+0000 7f3c17400640 1 --2- 172.21.15.39:0/3365369780 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55741f1b3c30 0x55741f1b60c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:46.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.452+0000 7f3c17400640 1 -- 172.21.15.39:0/3365369780 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55741f1aeac0 msgr2=0x55741f1b35c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:11:46.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.452+0000 7f3c17400640 1 --2- 172.21.15.39:0/3365369780 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55741f1aeac0 0x55741f1b35c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:46.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.452+0000 7f3c17400640 1 -- 172.21.15.39:0/3365369780 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55741f0d41f0 con 0x55741f1ae0d0 2024-03-18T16:11:46.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.453+0000 7f3c17400640 1 --2- 172.21.15.39:0/3365369780 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55741f1ae0d0 0x55741f1ae4f0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f3c10010400 tx=0x7f3c100108c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5aa4833462b32e40 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:46.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.453+0000 7f3c07e00640 1 -- 172.21.15.39:0/3365369780 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c10004230 con 0x55741f1ae0d0 2024-03-18T16:11:46.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.453+0000 7f3c07e00640 1 -- 172.21.15.39:0/3365369780 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c100043d0 con 0x55741f1ae0d0 2024-03-18T16:11:46.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.453+0000 7f3c07e00640 1 -- 172.21.15.39:0/3365369780 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c10003b00 con 0x55741f1ae0d0 2024-03-18T16:11:46.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.455+0000 7f3c0f400640 1 -- 172.21.15.39:0/1362182200 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f3c080107d0 con 0x7f3c181087e0 2024-03-18T16:11:46.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.455+0000 7f3c205db740 1 -- 172.21.15.39:0/3365369780 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55741f1ae0d0 msgr2=0x55741f1ae4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:46.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.455+0000 7f3c205db740 1 --2- 172.21.15.39:0/3365369780 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55741f1ae0d0 0x55741f1ae4f0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f3c10010400 tx=0x7f3c100108c0 comp rx=0 tx=0).stop 2024-03-18T16:11:46.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.455+0000 7f3c205db740 1 -- 172.21.15.39:0/3365369780 shutdown_connections 2024-03-18T16:11:46.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.455+0000 7f3c205db740 1 --2- 172.21.15.39:0/3365369780 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55741f1aeac0 0x55741f1b35c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:46.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.455+0000 7f3c205db740 1 --2- 172.21.15.39:0/3365369780 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55741f1b3c30 0x55741f1b60c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:46.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.455+0000 7f3c205db740 1 --2- 172.21.15.39:0/3365369780 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55741f1ae0d0 0x55741f1ae4f0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:46.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.455+0000 7f3c205db740 1 -- 172.21.15.39:0/3365369780 >> 172.21.15.39:0/3365369780 conn(0x55741f1a0e30 msgr2=0x55741f1a1230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:46.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.458+0000 7f3c205db740 1 -- 172.21.15.39:0/3365369780 shutdown_connections 2024-03-18T16:11:46.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.459+0000 7f3c0f400640 1 --2- 172.21.15.39:0/1362182200 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3bfc0a2800 0x7f3bfc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:46.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.459+0000 7f3c205db740 1 -- 172.21.15.39:0/3365369780 wait complete. 2024-03-18T16:11:46.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.459+0000 7f3c205db740 1 -- start start 2024-03-18T16:11:46.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.460+0000 7f3c205db740 1 Processor -- start 2024-03-18T16:11:46.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.460+0000 7f3c0f400640 1 -- 172.21.15.39:0/1362182200 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f3c080cf3d0 con 0x7f3c181087e0 2024-03-18T16:11:46.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.462+0000 7f3c205db740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55741f1ae0d0 0x55741f1b84c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:46.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.462+0000 7f3c205db740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55741f1aeac0 0x55741f1b8a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:46.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.462+0000 7f3c205db740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55741f1b3c30 0x55741f1bfa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:46.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.463+0000 7f3c205db740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55741f0c6110 con 0x55741f1aeac0 2024-03-18T16:11:46.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.463+0000 7f3c17400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55741f1b3c30 0x55741f1bfa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:46.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.463+0000 7f3c17400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55741f1b3c30 0x55741f1bfa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:49352/0 (socket says 172.21.15.39:49352) 2024-03-18T16:11:46.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.463+0000 7f3c17400640 1 -- 172.21.15.39:0/1407701566 learned_addr learned my addr 172.21.15.39:0/1407701566 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:11:46.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.463+0000 7f3c16000640 1 --2- 172.21.15.39:0/1362182200 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3bfc0a2800 0x7f3bfc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:46.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.464+0000 7f3c205db740 1 -- 172.21.15.39:0/1407701566 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55741f1064f0 con 0x55741f1b3c30 2024-03-18T16:11:46.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.464+0000 7f3c205db740 1 -- 172.21.15.39:0/1407701566 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55741f1b6be0 con 0x55741f1ae0d0 2024-03-18T16:11:46.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.464+0000 7f3c16a00640 1 --2- 172.21.15.39:0/1407701566 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55741f1ae0d0 0x55741f1b84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:46.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.464+0000 7f3c17400640 1 -- 172.21.15.39:0/1407701566 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55741f1ae0d0 msgr2=0x55741f1b84c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:46.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.464+0000 7f3c17400640 1 --2- 172.21.15.39:0/1407701566 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55741f1ae0d0 0x55741f1b84c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:46.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.464+0000 7f3c17400640 1 -- 172.21.15.39:0/1407701566 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55741f1aeac0 msgr2=0x55741f1b8a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:46.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.464+0000 7f3c17400640 1 --2- 172.21.15.39:0/1407701566 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55741f1aeac0 0x55741f1b8a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:46.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.464+0000 7f3c17400640 1 -- 172.21.15.39:0/1407701566 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55741ee1e3f0 con 0x55741f1b3c30 2024-03-18T16:11:46.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.464+0000 7f3c205db740 10 client.0 _ll_register_callbacks cb 0x55741f0b3b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:11:46.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.464+0000 7f3c205db740 1 -- 172.21.15.39:0/1407701566 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3c100043d0 con 0x55741f1b3c30 2024-03-18T16:11:46.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.465+0000 7f3c17400640 1 --2- 172.21.15.39:0/1407701566 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55741f1b3c30 0x55741f1bfa40 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f3c10011010 tx=0x7f3c100053b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6420541efc65dce9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:46.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.465+0000 7f3c04c00640 10 client.25931 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:11:46.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.466+0000 7f3c04c00640 1 -- 172.21.15.39:0/1407701566 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c1002a060 con 0x55741f1b3c30 2024-03-18T16:11:46.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.466+0000 7f3c04c00640 1 -- 172.21.15.39:0/1407701566 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f3c100048c0 con 0x55741f1b3c30 2024-03-18T16:11:46.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.466+0000 7f3c04c00640 1 -- 172.21.15.39:0/1407701566 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f3c10004ac0 con 0x55741f1b3c30 2024-03-18T16:11:46.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.466+0000 7f3c205db740 10 client.25931 fetch_fsmap learned FSMap version 30 2024-03-18T16:11:46.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.466+0000 7f3c205db740 1 -- 172.21.15.39:0/1407701566 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({fsmap=30}) -- 0x55741eee87d0 con 0x55741f1b3c30 2024-03-18T16:11:46.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.466+0000 7f3c16000640 1 --2- 172.21.15.39:0/1362182200 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3bfc0a2800 0x7f3bfc0a4cc0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f3c00006010 tx=0x7f3c00004cb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:46.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.467+0000 7f3c04c00640 1 -- 172.21.15.39:0/1407701566 <== mon.2 v2:172.21.15.194:3300/0 4 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f3c10004c60 con 0x55741f1b3c30 2024-03-18T16:11:46.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.468+0000 7f3c205db740 10 client.25931 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T16:11:46.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.468+0000 7f3c205db740 10 client.25931 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-18T16:11:46.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.468+0000 7f3c205db740 20 client.25931 populate_metadata read hostname 'smithi039' 2024-03-18T16:11:46.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.468+0000 7f3c205db740 1 --2- 172.21.15.39:0/1407701566 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x55741f1c3ee0 0x55741f1c62d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:46.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.468+0000 7f3c205db740 4 client.25931 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:11:46.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.468+0000 7f3c205db740 1 -- 172.21.15.39:0/1407701566 --> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55741f117990 con 0x55741f1c3ee0 2024-03-18T16:11:46.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.468+0000 7f3c16a00640 1 --2- 172.21.15.39:0/1407701566 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x55741f1c3ee0 0x55741f1c62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:46.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.469+0000 7f3c16a00640 1 --2- 172.21.15.39:0/1407701566 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x55741f1c3ee0 0x55741f1c62d0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=66e145e403e391ae server_cookie=14676afa51a3727e in_seq=0 out_seq=0 2024-03-18T16:11:46.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.469+0000 7f3c04c00640 10 client.25931 ms_handle_connect on v2:172.21.15.194:6836/1301464246 2024-03-18T16:11:46.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.470+0000 7f3c04c00640 1 -- 172.21.15.39:0/1407701566 <== mds.3 v2:172.21.15.194:6836/1301464246 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55741f117990 con 0x55741f1c3ee0 2024-03-18T16:11:46.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.470+0000 7f3c04c00640 10 client.25931 handle_command_reply: tid=0 2024-03-18T16:11:46.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.471+0000 7f3c205db740 1 client.25931 shutdown 2024-03-18T16:11:46.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.471+0000 7f3c205db740 1 -- 172.21.15.39:0/1407701566 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55741f1b3c30 msgr2=0x55741f1bfa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:46.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.471+0000 7f3c205db740 1 --2- 172.21.15.39:0/1407701566 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55741f1b3c30 0x55741f1bfa40 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f3c10011010 tx=0x7f3c100053b0 comp rx=0 tx=0).stop 2024-03-18T16:11:46.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.471+0000 7f3c205db740 1 -- 172.21.15.39:0/1407701566 shutdown_connections 2024-03-18T16:11:46.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.471+0000 7f3c205db740 1 --2- 172.21.15.39:0/1407701566 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x55741f1c3ee0 0x55741f1c62d0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:46.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.471+0000 7f3c205db740 1 --2- 172.21.15.39:0/1407701566 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55741f1b3c30 0x55741f1bfa40 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:46.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.471+0000 7f3c205db740 1 --2- 172.21.15.39:0/1407701566 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55741f1aeac0 0x55741f1b8a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:46.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.471+0000 7f3c205db740 1 --2- 172.21.15.39:0/1407701566 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55741f1ae0d0 0x55741f1b84c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:46.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.471+0000 7f3c205db740 1 -- 172.21.15.39:0/1407701566 >> 172.21.15.39:0/1407701566 conn(0x55741f1a0e30 msgr2=0x55741f1a01f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:46.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.471+0000 7f3c04c00640 0 client.25931 ms_handle_reset on v2:172.21.15.194:6836/1301464246 2024-03-18T16:11:46.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.472+0000 7f3c205db740 1 -- 172.21.15.39:0/1407701566 shutdown_connections 2024-03-18T16:11:46.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.472+0000 7f3c205db740 1 -- 172.21.15.39:0/1407701566 wait complete. 2024-03-18T16:11:46.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.472+0000 7f3c205db740 20 client.25931 trim_cache size 0 max 16384 2024-03-18T16:11:46.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:11:46 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/135412511' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:11:46.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.486+0000 7f3c205db740 1 Processor -- start 2024-03-18T16:11:46.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.487+0000 7f3c205db740 1 -- start start 2024-03-18T16:11:46.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.487+0000 7f3c205db740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55741f1ae0d0 0x55741f1b7790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:46.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.487+0000 7f3c205db740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55741f1aeac0 0x55741f1bfa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:46.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.487+0000 7f3c205db740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55741f1b3c30 0x55741f1b7cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:46.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.487+0000 7f3c205db740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f3bf400fc90 con 0x55741f1aeac0 2024-03-18T16:11:46.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.487+0000 7f3c205db740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f3bf4011830 con 0x55741f1b3c30 2024-03-18T16:11:46.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.487+0000 7f3c205db740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f3bf4010360 con 0x55741f1ae0d0 2024-03-18T16:11:46.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.487+0000 7f3c16a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55741f1ae0d0 0x55741f1b7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:46.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.487+0000 7f3c16000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55741f1aeac0 0x55741f1bfa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:46.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.487+0000 7f3c16a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55741f1ae0d0 0x55741f1b7790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:33780/0 (socket says 172.21.15.39:33780) 2024-03-18T16:11:46.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.487+0000 7f3c16a00640 1 -- 172.21.15.39:0/139730029 learned_addr learned my addr 172.21.15.39:0/139730029 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:11:46.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.487+0000 7f3c17400640 1 --2- 172.21.15.39:0/139730029 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55741f1b3c30 0x55741f1b7cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:46.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.488+0000 7f3c16a00640 1 -- 172.21.15.39:0/139730029 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55741f1b3c30 msgr2=0x55741f1b7cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:46.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.488+0000 7f3c16a00640 1 --2- 172.21.15.39:0/139730029 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55741f1b3c30 0x55741f1b7cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:46.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.488+0000 7f3c16a00640 1 -- 172.21.15.39:0/139730029 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55741f1aeac0 msgr2=0x55741f1bfa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:46.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.488+0000 7f3c16a00640 1 --2- 172.21.15.39:0/139730029 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55741f1aeac0 0x55741f1bfa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:46.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.488+0000 7f3c16a00640 1 -- 172.21.15.39:0/139730029 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55741f1b3260 con 0x55741f1ae0d0 2024-03-18T16:11:46.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.488+0000 7f3c16a00640 1 --2- 172.21.15.39:0/139730029 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55741f1ae0d0 0x55741f1b7790 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f3c0809b910 tx=0x7f3c080cc9b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6572aec30295fdac server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:46.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.488+0000 7f3c07400640 1 -- 172.21.15.39:0/139730029 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c08010e60 con 0x55741f1ae0d0 2024-03-18T16:11:46.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.488+0000 7f3c07400640 1 -- 172.21.15.39:0/139730029 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c0809fad0 con 0x55741f1ae0d0 2024-03-18T16:11:46.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.488+0000 7f3c07400640 1 -- 172.21.15.39:0/139730029 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c080ccbf0 con 0x55741f1ae0d0 2024-03-18T16:11:46.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:11:46 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/135412511' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:11:46.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.489+0000 7f3c205db740 1 -- 172.21.15.39:0/139730029 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55741f1ae0d0 msgr2=0x55741f1b7790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:46.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.489+0000 7f3c205db740 1 --2- 172.21.15.39:0/139730029 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55741f1ae0d0 0x55741f1b7790 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f3c0809b910 tx=0x7f3c080cc9b0 comp rx=0 tx=0).stop 2024-03-18T16:11:46.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.489+0000 7f3c205db740 1 -- 172.21.15.39:0/139730029 shutdown_connections 2024-03-18T16:11:46.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.489+0000 7f3c205db740 1 --2- 172.21.15.39:0/139730029 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55741f1b3c30 0x55741f1b7cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:46.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.489+0000 7f3c205db740 1 --2- 172.21.15.39:0/139730029 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55741f1aeac0 0x55741f1bfa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:46.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.489+0000 7f3c205db740 1 --2- 172.21.15.39:0/139730029 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55741f1ae0d0 0x55741f1b7790 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:46.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.489+0000 7f3c205db740 1 -- 172.21.15.39:0/139730029 >> 172.21.15.39:0/139730029 conn(0x55741f1a0e30 msgr2=0x55741f1b84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:46.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.489+0000 7f3c205db740 1 -- 172.21.15.39:0/139730029 shutdown_connections 2024-03-18T16:11:46.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.489+0000 7f3c205db740 1 -- 172.21.15.39:0/139730029 wait complete. 2024-03-18T16:11:46.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.490+0000 7f3c205db740 1 -- start start 2024-03-18T16:11:46.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.490+0000 7f3c205db740 1 Processor -- start 2024-03-18T16:11:46.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.490+0000 7f3c205db740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55741f1ae0d0 0x55741f1bcbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:46.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.490+0000 7f3c205db740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55741f1aeac0 0x55741f1bd130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:46.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.490+0000 7f3c205db740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55741f1b3c30 0x55741f1be140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:46.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.490+0000 7f3c205db740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f3bf4011100 con 0x55741f1aeac0 2024-03-18T16:11:46.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.490+0000 7f3c205db740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f3bf4010a30 con 0x55741f1ae0d0 2024-03-18T16:11:46.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.490+0000 7f3c205db740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f3bf400dc70 con 0x55741f1b3c30 2024-03-18T16:11:46.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.490+0000 7f3c205db740 10 client.0 _ll_register_callbacks cb 0x7f3c080093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:11:46.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.491+0000 7f3c16a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55741f1ae0d0 0x55741f1bcbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:46.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.491+0000 7f3c16a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55741f1ae0d0 0x55741f1bcbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:49362/0 (socket says 172.21.15.39:49362) 2024-03-18T16:11:46.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.491+0000 7f3c16a00640 1 -- 172.21.15.39:0/1776529514 learned_addr learned my addr 172.21.15.39:0/1776529514 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:11:46.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.491+0000 7f3c17400640 1 --2- 172.21.15.39:0/1776529514 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55741f1b3c30 0x55741f1be140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:46.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.491+0000 7f3c16000640 1 --2- 172.21.15.39:0/1776529514 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55741f1aeac0 0x55741f1bd130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:46.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.491+0000 7f3c16a00640 1 -- 172.21.15.39:0/1776529514 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55741f1b3c30 msgr2=0x55741f1be140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:46.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.491+0000 7f3c16a00640 1 --2- 172.21.15.39:0/1776529514 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55741f1b3c30 0x55741f1be140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:46.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.491+0000 7f3c16a00640 1 -- 172.21.15.39:0/1776529514 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55741f1aeac0 msgr2=0x55741f1bd130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:46.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.491+0000 7f3c16a00640 1 --2- 172.21.15.39:0/1776529514 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55741f1aeac0 0x55741f1bd130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:46.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.491+0000 7f3c16a00640 1 -- 172.21.15.39:0/1776529514 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55741f1be730 con 0x55741f1ae0d0 2024-03-18T16:11:46.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.491+0000 7f3c17400640 1 --2- 172.21.15.39:0/1776529514 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55741f1b3c30 0x55741f1be140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:11:46.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.492+0000 7f3c16a00640 1 --2- 172.21.15.39:0/1776529514 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55741f1ae0d0 0x55741f1bcbf0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f3c08098350 tx=0x7f3c0801f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=24786df92b16ddaf server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:46.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.492+0000 7f3c04c00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:11:46.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.492+0000 7f3c04c00640 1 -- 172.21.15.39:0/1776529514 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c080b9060 con 0x55741f1ae0d0 2024-03-18T16:11:46.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.492+0000 7f3c04c00640 1 -- 172.21.15.39:0/1776529514 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f3c08097ff0 con 0x55741f1ae0d0 2024-03-18T16:11:46.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.492+0000 7f3c205db740 1 -- 172.21.15.39:0/1776529514 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3c0809fad0 con 0x55741f1ae0d0 2024-03-18T16:11:46.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.492+0000 7f3c04c00640 1 -- 172.21.15.39:0/1776529514 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f3c080b3070 con 0x55741f1ae0d0 2024-03-18T16:11:46.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.492+0000 7f3c205db740 10 client.25937 fetch_fsmap learned FSMap version 30 2024-03-18T16:11:46.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.492+0000 7f3c205db740 1 -- 172.21.15.39:0/1776529514 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55741f1bfaa0 con 0x55741f1ae0d0 2024-03-18T16:11:46.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.492+0000 7f3c04c00640 1 -- 172.21.15.39:0/1776529514 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c0809fad0 con 0x55741f1ae0d0 2024-03-18T16:11:46.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.492+0000 7f3c04c00640 1 -- 172.21.15.39:0/1776529514 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f3c0809af40 con 0x55741f1ae0d0 2024-03-18T16:11:46.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.493+0000 7f3c205db740 10 client.25937 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T16:11:46.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.493+0000 7f3c205db740 10 client.25937 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-18T16:11:46.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.493+0000 7f3c205db740 20 client.25937 populate_metadata read hostname 'smithi039' 2024-03-18T16:11:46.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.493+0000 7f3c205db740 1 --2- 172.21.15.39:0/1776529514 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x55741f22a330 0x55741f22c750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:46.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.493+0000 7f3c205db740 4 client.25937 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:11:46.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.493+0000 7f3c205db740 1 -- 172.21.15.39:0/1776529514 --> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55741f111ce0 con 0x55741f22a330 2024-03-18T16:11:46.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.493+0000 7f3c16000640 1 --2- 172.21.15.39:0/1776529514 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x55741f22a330 0x55741f22c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:46.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.493+0000 7f3c16000640 1 --2- 172.21.15.39:0/1776529514 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x55741f22a330 0x55741f22c750 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=30698366fcbe65bb server_cookie=30622a66b8136a0e in_seq=0 out_seq=0 2024-03-18T16:11:46.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.493+0000 7f3c04c00640 10 client.25937 ms_handle_connect on v2:172.21.15.194:6836/1301464246 2024-03-18T16:11:46.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.494+0000 7f3c04c00640 1 -- 172.21.15.39:0/1776529514 <== mds.3 v2:172.21.15.194:6836/1301464246 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55741f111ce0 con 0x55741f22a330 2024-03-18T16:11:46.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.494+0000 7f3c04c00640 10 client.25937 handle_command_reply: tid=0 2024-03-18T16:11:46.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.494+0000 7f3c205db740 1 client.25937 shutdown 2024-03-18T16:11:46.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.494+0000 7f3c205db740 1 -- 172.21.15.39:0/1776529514 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55741f1ae0d0 msgr2=0x55741f1bcbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:46.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.494+0000 7f3c205db740 1 --2- 172.21.15.39:0/1776529514 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55741f1ae0d0 0x55741f1bcbf0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f3c08098350 tx=0x7f3c0801f000 comp rx=0 tx=0).stop 2024-03-18T16:11:46.499 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:11:46.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.494+0000 7f3c205db740 1 -- 172.21.15.39:0/1776529514 shutdown_connections 2024-03-18T16:11:46.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.494+0000 7f3c205db740 1 --2- 172.21.15.39:0/1776529514 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x55741f22a330 0x55741f22c750 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:46.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.494+0000 7f3c205db740 1 --2- 172.21.15.39:0/1776529514 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55741f1b3c30 0x55741f1be140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:46.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.494+0000 7f3c205db740 1 --2- 172.21.15.39:0/1776529514 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55741f1aeac0 0x55741f1bd130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:46.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.494+0000 7f3c205db740 1 --2- 172.21.15.39:0/1776529514 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55741f1ae0d0 0x55741f1bcbf0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:46.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.494+0000 7f3c04c00640 0 client.25937 ms_handle_reset on v2:172.21.15.194:6836/1301464246 2024-03-18T16:11:46.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.494+0000 7f3c205db740 1 -- 172.21.15.39:0/1776529514 >> 172.21.15.39:0/1776529514 conn(0x55741f1a0e30 msgr2=0x55741f1a4920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:46.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.494+0000 7f3c205db740 1 -- 172.21.15.39:0/1776529514 shutdown_connections 2024-03-18T16:11:46.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.494+0000 7f3c205db740 1 -- 172.21.15.39:0/1776529514 wait complete. 2024-03-18T16:11:46.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.494+0000 7f3c205db740 20 client.25937 trim_cache size 0 max 16384 2024-03-18T16:11:46.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.497+0000 7f3c1d800640 1 -- 172.21.15.39:0/1362182200 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3bfc0a2800 msgr2=0x7f3bfc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:46.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.497+0000 7f3c1d800640 1 --2- 172.21.15.39:0/1362182200 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3bfc0a2800 0x7f3bfc0a4cc0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f3c00006010 tx=0x7f3c00004cb0 comp rx=0 tx=0).stop 2024-03-18T16:11:46.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.497+0000 7f3c1d800640 1 -- 172.21.15.39:0/1362182200 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3c181087e0 msgr2=0x7f3c1819e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:46.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.497+0000 7f3c1d800640 1 --2- 172.21.15.39:0/1362182200 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3c181087e0 0x7f3c1819e190 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f3c0800ebc0 tx=0x7f3c0800c6a0 comp rx=0 tx=0).stop 2024-03-18T16:11:46.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.498+0000 7f3c1d800640 1 -- 172.21.15.39:0/1362182200 shutdown_connections 2024-03-18T16:11:46.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.498+0000 7f3c1d800640 1 --2- 172.21.15.39:0/1362182200 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3bfc0a2800 0x7f3bfc0a4cc0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:46.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.498+0000 7f3c1d800640 1 --2- 172.21.15.39:0/1362182200 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c18110060 0x7f3c181a56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:46.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.498+0000 7f3c1d800640 1 --2- 172.21.15.39:0/1362182200 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3c18109140 0x7f3c1819e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:46.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.498+0000 7f3c1d800640 1 --2- 172.21.15.39:0/1362182200 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3c181087e0 0x7f3c1819e190 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:46.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.498+0000 7f3c1d800640 1 -- 172.21.15.39:0/1362182200 >> 172.21.15.39:0/1362182200 conn(0x7f3c1806e9c0 msgr2=0x7f3c1810bb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:46.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.500+0000 7f3c1d800640 1 -- 172.21.15.39:0/1362182200 shutdown_connections 2024-03-18T16:11:46.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.500+0000 7f3c1d800640 1 -- 172.21.15.39:0/1362182200 wait complete. 2024-03-18T16:11:46.531 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-18T16:11:46.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.841+0000 7f9758400640 1 -- 172.21.15.39:0/3939531247 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f97500761a0 msgr2=0x7f97500765c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:46.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.841+0000 7f9758400640 1 --2- 172.21.15.39:0/3939531247 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f97500761a0 0x7f97500765c0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f9744009900 tx=0x7f974402bc30 comp rx=0 tx=0).stop 2024-03-18T16:11:46.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.841+0000 7f9758400640 1 -- 172.21.15.39:0/3939531247 shutdown_connections 2024-03-18T16:11:46.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.841+0000 7f9758400640 1 --2- 172.21.15.39:0/3939531247 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9750077d80 0x7f9750078620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:46.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.841+0000 7f9758400640 1 --2- 172.21.15.39:0/3939531247 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f97500773e0 0x7f9750077840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:46.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.841+0000 7f9758400640 1 --2- 172.21.15.39:0/3939531247 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f97500761a0 0x7f97500765c0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:46.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.841+0000 7f9758400640 1 -- 172.21.15.39:0/3939531247 >> 172.21.15.39:0/3939531247 conn(0x7f97500fef50 msgr2=0x7f97501013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:46.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.843+0000 7f9758400640 1 -- 172.21.15.39:0/3939531247 shutdown_connections 2024-03-18T16:11:46.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.843+0000 7f9758400640 1 -- 172.21.15.39:0/3939531247 wait complete. 2024-03-18T16:11:46.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.845+0000 7f9758400640 1 Processor -- start 2024-03-18T16:11:46.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.847+0000 7f9758400640 1 -- start start 2024-03-18T16:11:46.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.848+0000 7f9758400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f97500761a0 0x7f975019e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:46.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.848+0000 7f9758400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f97500773e0 0x7f975019e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:46.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.848+0000 7f9758400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9750077d80 0x7f97501a56d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:46.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.849+0000 7f9755000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f97500773e0 0x7f975019e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:46.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.849+0000 7f9755000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f97500773e0 0x7f975019e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:49386/0 (socket says 172.21.15.39:49386) 2024-03-18T16:11:46.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.850+0000 7f9758400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f9750113b30 con 0x7f9750077d80 2024-03-18T16:11:46.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.850+0000 7f9755000640 1 -- 172.21.15.39:0/2356113558 learned_addr learned my addr 172.21.15.39:0/2356113558 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:11:46.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.850+0000 7f9758400640 1 -- 172.21.15.39:0/2356113558 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f97501139b0 con 0x7f97500773e0 2024-03-18T16:11:46.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.850+0000 7f9758400640 1 -- 172.21.15.39:0/2356113558 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f9750113cb0 con 0x7f97500761a0 2024-03-18T16:11:46.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.851+0000 7f9755000640 1 -- 172.21.15.39:0/2356113558 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f97500761a0 msgr2=0x7f975019e120 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T16:11:46.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.851+0000 7f9755000640 1 --2- 172.21.15.39:0/2356113558 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f97500761a0 0x7f975019e120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:46.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.851+0000 7f9755000640 1 -- 172.21.15.39:0/2356113558 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9750077d80 msgr2=0x7f97501a56d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:46.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.851+0000 7f9755000640 1 --2- 172.21.15.39:0/2356113558 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9750077d80 0x7f97501a56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:46.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.851+0000 7f9755000640 1 -- 172.21.15.39:0/2356113558 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97501a5d40 con 0x7f97500773e0 2024-03-18T16:11:46.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.851+0000 7f9755000640 1 --2- 172.21.15.39:0/2356113558 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f97500773e0 0x7f975019e660 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f973c00b4a0 tx=0x7f973c00b960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:46.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.853+0000 7f974e000640 1 -- 172.21.15.39:0/2356113558 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f973c014070 con 0x7f97500773e0 2024-03-18T16:11:46.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.853+0000 7f974e000640 1 -- 172.21.15.39:0/2356113558 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f973c010af0 con 0x7f97500773e0 2024-03-18T16:11:46.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.853+0000 7f9758400640 1 -- 172.21.15.39:0/2356113558 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97501a5fd0 con 0x7f97500773e0 2024-03-18T16:11:46.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.853+0000 7f974e000640 1 -- 172.21.15.39:0/2356113558 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f973c017d40 con 0x7f97500773e0 2024-03-18T16:11:46.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.853+0000 7f9758400640 1 -- 172.21.15.39:0/2356113558 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97501a64f0 con 0x7f97500773e0 2024-03-18T16:11:46.868 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.866+0000 7f975b029740 1 Processor -- start 2024-03-18T16:11:46.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.867+0000 7f975b029740 1 -- start start 2024-03-18T16:11:46.871 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.869+0000 7f975b029740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55858f0c88c0 0x55858f0c8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:46.871 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.869+0000 7f975b029740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55858f0c92b0 0x55858f0cddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:46.871 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.869+0000 7f975b029740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55858f0ce420 0x55858f0d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:46.871 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.869+0000 7f975b029740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55858f0903b0 con 0x55858f0c88c0 2024-03-18T16:11:46.871 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.869+0000 7f975b029740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55858f02d450 con 0x55858f0ce420 2024-03-18T16:11:46.871 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.869+0000 7f975b029740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55858efd5cb0 con 0x55858f0c92b0 2024-03-18T16:11:46.872 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.869+0000 7f9755a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55858f0c92b0 0x55858f0cddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:46.872 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.869+0000 7f9755a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55858f0c92b0 0x55858f0cddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:33818/0 (socket says 172.21.15.39:33818) 2024-03-18T16:11:46.872 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.869+0000 7f9755a00640 1 -- 172.21.15.39:0/611727795 learned_addr learned my addr 172.21.15.39:0/611727795 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:11:46.872 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.870+0000 7f9755000640 1 --2- 172.21.15.39:0/611727795 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55858f0ce420 0x55858f0d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:46.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.871+0000 7f9755a00640 1 -- 172.21.15.39:0/611727795 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55858f0ce420 msgr2=0x55858f0d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:46.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.871+0000 7f9755a00640 1 --2- 172.21.15.39:0/611727795 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55858f0ce420 0x55858f0d08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:46.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.871+0000 7f9755a00640 1 -- 172.21.15.39:0/611727795 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55858f0c88c0 msgr2=0x55858f0c8ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:46.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.871+0000 7f9755a00640 1 --2- 172.21.15.39:0/611727795 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55858f0c88c0 0x55858f0c8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:46.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.871+0000 7f9755a00640 1 -- 172.21.15.39:0/611727795 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55858eff71f0 con 0x55858f0c92b0 2024-03-18T16:11:46.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.871+0000 7f9755000640 1 --2- 172.21.15.39:0/611727795 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55858f0ce420 0x55858f0d08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:11:46.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.873+0000 7f9755a00640 1 --2- 172.21.15.39:0/611727795 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55858f0c92b0 0x55858f0cddb0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f974402c7b0 tx=0x7f9744005b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dd2bfce67124817c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:46.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.874+0000 7f9742a00640 1 -- 172.21.15.39:0/611727795 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9744043020 con 0x55858f0c92b0 2024-03-18T16:11:46.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.874+0000 7f974e000640 1 -- 172.21.15.39:0/2356113558 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f973c017070 con 0x7f97500773e0 2024-03-18T16:11:46.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.875+0000 7f9742a00640 1 -- 172.21.15.39:0/611727795 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9744009520 con 0x55858f0c92b0 2024-03-18T16:11:46.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.875+0000 7f9742a00640 1 -- 172.21.15.39:0/611727795 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97440053f0 con 0x55858f0c92b0 2024-03-18T16:11:46.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.878+0000 7f975b029740 1 -- 172.21.15.39:0/611727795 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55858f0c92b0 msgr2=0x55858f0cddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:46.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.878+0000 7f975b029740 1 --2- 172.21.15.39:0/611727795 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55858f0c92b0 0x55858f0cddb0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f974402c7b0 tx=0x7f9744005b50 comp rx=0 tx=0).stop 2024-03-18T16:11:46.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.881+0000 7f975b029740 1 -- 172.21.15.39:0/611727795 shutdown_connections 2024-03-18T16:11:46.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.881+0000 7f975b029740 1 --2- 172.21.15.39:0/611727795 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55858f0ce420 0x55858f0d08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:46.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.881+0000 7f975b029740 1 --2- 172.21.15.39:0/611727795 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55858f0c92b0 0x55858f0cddb0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:46.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.881+0000 7f975b029740 1 --2- 172.21.15.39:0/611727795 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55858f0c88c0 0x55858f0c8ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:46.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.881+0000 7f975b029740 1 -- 172.21.15.39:0/611727795 >> 172.21.15.39:0/611727795 conn(0x55858f0c3e30 msgr2=0x55858f0c4230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:46.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.884+0000 7f975b029740 1 -- 172.21.15.39:0/611727795 shutdown_connections 2024-03-18T16:11:46.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.884+0000 7f975b029740 1 -- 172.21.15.39:0/611727795 wait complete. 2024-03-18T16:11:46.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.889+0000 7f974e000640 1 --2- 172.21.15.39:0/2356113558 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f97380a2910 0x7f97380a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:46.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.891+0000 7f974e000640 1 -- 172.21.15.39:0/2356113558 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f973c0ce160 con 0x7f97500773e0 2024-03-18T16:11:46.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.892+0000 7f975b029740 1 -- start start 2024-03-18T16:11:46.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.892+0000 7f975b029740 1 Processor -- start 2024-03-18T16:11:46.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.894+0000 7f975b029740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55858f0c88c0 0x55858f0d2bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:46.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.894+0000 7f975b029740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55858f0c92b0 0x55858f0d3100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:46.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.894+0000 7f9755a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55858f0c88c0 0x55858f0d2bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:46.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.894+0000 7f9755a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55858f0c88c0 0x55858f0d2bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:49396/0 (socket says 172.21.15.39:49396) 2024-03-18T16:11:46.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.895+0000 7f9755a00640 1 -- 172.21.15.39:0/173042117 learned_addr learned my addr 172.21.15.39:0/173042117 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:11:46.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.895+0000 7f975b029740 1 --2- 172.21.15.39:0/173042117 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55858f0ce420 0x55858f0da140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:46.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.895+0000 7f975b029740 1 -- 172.21.15.39:0/173042117 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55858efe9110 con 0x55858f0c92b0 2024-03-18T16:11:46.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.895+0000 7f9755000640 1 --2- 172.21.15.39:0/173042117 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55858f0c92b0 0x55858f0d3100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:46.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.896+0000 7f975b029740 1 -- 172.21.15.39:0/173042117 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55858f0294f0 con 0x55858f0c88c0 2024-03-18T16:11:46.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.896+0000 7f975b029740 1 -- 172.21.15.39:0/173042117 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55858f0d1370 con 0x55858f0ce420 2024-03-18T16:11:46.898 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.896+0000 7f975b029740 10 client.0 _ll_register_callbacks cb 0x55858efd6b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:11:46.898 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.896+0000 7f9756400640 1 --2- 172.21.15.39:0/173042117 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55858f0ce420 0x55858f0da140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:46.898 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.897+0000 7f9755a00640 1 -- 172.21.15.39:0/173042117 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55858f0ce420 msgr2=0x55858f0da140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:46.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.897+0000 7f9755a00640 1 --2- 172.21.15.39:0/173042117 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55858f0ce420 0x55858f0da140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:46.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.897+0000 7f9755a00640 1 -- 172.21.15.39:0/173042117 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55858f0c92b0 msgr2=0x55858f0d3100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:46.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.897+0000 7f9755a00640 1 --2- 172.21.15.39:0/173042117 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55858f0c92b0 0x55858f0d3100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:46.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.897+0000 7f9755a00640 1 -- 172.21.15.39:0/173042117 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55858ed413f0 con 0x55858f0c88c0 2024-03-18T16:11:46.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.897+0000 7f9756400640 1 --2- 172.21.15.39:0/173042117 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55858f0ce420 0x55858f0da140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:11:46.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.897+0000 7f9756400640 1 --2- 172.21.15.39:0/2356113558 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f97380a2910 0x7f97380a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:46.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.898+0000 7f9755a00640 1 --2- 172.21.15.39:0/173042117 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55858f0c88c0 0x55858f0d2bc0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f9744000c00 tx=0x7f9744005420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dfb09a61080ffaa0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:46.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.898+0000 7f9737400640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:11:46.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.899+0000 7f9756400640 1 --2- 172.21.15.39:0/2356113558 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f97380a2910 0x7f97380a4dd0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f9748005c40 tx=0x7f9748004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:46.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.899+0000 7f9737400640 1 -- 172.21.15.39:0/173042117 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f974404c030 con 0x55858f0c88c0 2024-03-18T16:11:46.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.900+0000 7f9737400640 1 -- 172.21.15.39:0/173042117 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f97440048d0 con 0x55858f0c88c0 2024-03-18T16:11:46.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.900+0000 7f975b029740 1 -- 172.21.15.39:0/173042117 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9744009520 con 0x55858f0c88c0 2024-03-18T16:11:46.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.900+0000 7f9737400640 1 -- 172.21.15.39:0/173042117 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f9744059050 con 0x55858f0c88c0 2024-03-18T16:11:46.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.900+0000 7f975b029740 10 client.25949 fetch_fsmap learned FSMap version 30 2024-03-18T16:11:46.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.900+0000 7f975b029740 1 -- 172.21.15.39:0/173042117 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55858ee0b7d0 con 0x55858f0c88c0 2024-03-18T16:11:46.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.900+0000 7f9737400640 1 -- 172.21.15.39:0/173042117 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9744005610 con 0x55858f0c88c0 2024-03-18T16:11:46.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.902+0000 7f9737400640 1 -- 172.21.15.39:0/173042117 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f9744048b60 con 0x55858f0c88c0 2024-03-18T16:11:46.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.902+0000 7f975b029740 10 client.25949 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T16:11:46.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.902+0000 7f975b029740 10 client.25949 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-18T16:11:46.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.902+0000 7f975b029740 20 client.25949 populate_metadata read hostname 'smithi039' 2024-03-18T16:11:46.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.902+0000 7f975b029740 1 --2- 172.21.15.39:0/173042117 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x55858f0de630 0x55858f0e0a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:46.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.902+0000 7f975b029740 4 client.25949 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:11:46.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.902+0000 7f975b029740 1 -- 172.21.15.39:0/173042117 --> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f9744009520 con 0x55858f0de630 2024-03-18T16:11:46.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.903+0000 7f9755000640 1 --2- 172.21.15.39:0/173042117 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x55858f0de630 0x55858f0e0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:46.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.904+0000 7f9755000640 1 --2- 172.21.15.39:0/173042117 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x55858f0de630 0x55858f0e0a20 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d140d7db2b36dd6f server_cookie=6192951e36af2100 in_seq=0 out_seq=0 2024-03-18T16:11:46.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.904+0000 7f9737400640 10 client.25949 ms_handle_connect on v2:172.21.15.196:6836/895247251 2024-03-18T16:11:46.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.905+0000 7f9737400640 1 -- 172.21.15.39:0/173042117 <== mds.2 v2:172.21.15.196:6836/895247251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f9744009520 con 0x55858f0de630 2024-03-18T16:11:46.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.905+0000 7f9737400640 10 client.25949 handle_command_reply: tid=0 2024-03-18T16:11:46.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.906+0000 7f975b029740 1 client.25949 shutdown 2024-03-18T16:11:46.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.908+0000 7f975b029740 1 -- 172.21.15.39:0/173042117 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55858f0c88c0 msgr2=0x55858f0d2bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:46.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.908+0000 7f975b029740 1 --2- 172.21.15.39:0/173042117 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55858f0c88c0 0x55858f0d2bc0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f9744000c00 tx=0x7f9744005420 comp rx=0 tx=0).stop 2024-03-18T16:11:46.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.908+0000 7f975b029740 1 -- 172.21.15.39:0/173042117 shutdown_connections 2024-03-18T16:11:46.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.908+0000 7f975b029740 1 --2- 172.21.15.39:0/173042117 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x55858f0de630 0x55858f0e0a20 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:46.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.908+0000 7f975b029740 1 --2- 172.21.15.39:0/173042117 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55858f0ce420 0x55858f0da140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:46.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.908+0000 7f975b029740 1 --2- 172.21.15.39:0/173042117 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55858f0c92b0 0x55858f0d3100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:46.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.908+0000 7f975b029740 1 --2- 172.21.15.39:0/173042117 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55858f0c88c0 0x55858f0d2bc0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:46.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.908+0000 7f975b029740 1 -- 172.21.15.39:0/173042117 >> 172.21.15.39:0/173042117 conn(0x55858f0c3e30 msgr2=0x55858f0c31f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:46.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.909+0000 7f9737400640 0 client.25949 ms_handle_reset on v2:172.21.15.196:6836/895247251 2024-03-18T16:11:46.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.909+0000 7f975b029740 1 -- 172.21.15.39:0/173042117 shutdown_connections 2024-03-18T16:11:46.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.910+0000 7f975b029740 1 -- 172.21.15.39:0/173042117 wait complete. 2024-03-18T16:11:46.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.910+0000 7f975b029740 20 client.25949 trim_cache size 0 max 16384 2024-03-18T16:11:46.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.954+0000 7f975b029740 1 Processor -- start 2024-03-18T16:11:46.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.955+0000 7f975b029740 1 -- start start 2024-03-18T16:11:46.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.955+0000 7f975b029740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55858f0c88c0 0x55858f0c2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:46.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.955+0000 7f975b029740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55858f0c92b0 0x55858f0c2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:46.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.955+0000 7f975b029740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55858f0ce420 0x55858f0c33e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:46.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.955+0000 7f975b029740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f97501b0b80 con 0x55858f0ce420 2024-03-18T16:11:46.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.955+0000 7f975b029740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f97501b2720 con 0x55858f0c92b0 2024-03-18T16:11:46.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.955+0000 7f975b029740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f97501b1250 con 0x55858f0c88c0 2024-03-18T16:11:46.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.956+0000 7f9756400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55858f0ce420 0x55858f0c33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:46.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.956+0000 7f9755a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55858f0c88c0 0x55858f0c2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:46.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.956+0000 7f9755a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55858f0c88c0 0x55858f0c2960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:33842/0 (socket says 172.21.15.39:33842) 2024-03-18T16:11:46.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.956+0000 7f9755a00640 1 -- 172.21.15.39:0/124307087 learned_addr learned my addr 172.21.15.39:0/124307087 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:11:46.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.956+0000 7f9755000640 1 --2- 172.21.15.39:0/124307087 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55858f0c92b0 0x55858f0c2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:46.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.957+0000 7f9755a00640 1 -- 172.21.15.39:0/124307087 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55858f0c92b0 msgr2=0x55858f0c2ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:46.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.957+0000 7f9755a00640 1 --2- 172.21.15.39:0/124307087 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55858f0c92b0 0x55858f0c2ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:46.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.957+0000 7f9755a00640 1 -- 172.21.15.39:0/124307087 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55858f0ce420 msgr2=0x55858f0c33e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:46.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.959+0000 7f9755a00640 1 --2- 172.21.15.39:0/124307087 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55858f0ce420 0x55858f0c33e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:46.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.959+0000 7f9755a00640 1 -- 172.21.15.39:0/124307087 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55858f0bfc10 con 0x55858f0c88c0 2024-03-18T16:11:46.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.959+0000 7f9755a00640 1 --2- 172.21.15.39:0/124307087 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55858f0c88c0 0x55858f0c2960 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f9744001eb0 tx=0x7f9744005380 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6ad58575250528a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:46.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.959+0000 7f9742000640 1 -- 172.21.15.39:0/124307087 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f974404c020 con 0x55858f0c88c0 2024-03-18T16:11:46.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.959+0000 7f9742000640 1 -- 172.21.15.39:0/124307087 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9744009520 con 0x55858f0c88c0 2024-03-18T16:11:46.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.960+0000 7f9742000640 1 -- 172.21.15.39:0/124307087 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9744043040 con 0x55858f0c88c0 2024-03-18T16:11:46.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.962+0000 7f975b029740 1 -- 172.21.15.39:0/124307087 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55858f0c88c0 msgr2=0x55858f0c2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:46.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.962+0000 7f975b029740 1 --2- 172.21.15.39:0/124307087 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55858f0c88c0 0x55858f0c2960 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f9744001eb0 tx=0x7f9744005380 comp rx=0 tx=0).stop 2024-03-18T16:11:46.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.963+0000 7f975b029740 1 -- 172.21.15.39:0/124307087 shutdown_connections 2024-03-18T16:11:46.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.963+0000 7f975b029740 1 --2- 172.21.15.39:0/124307087 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55858f0c92b0 0x55858f0c2ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:46.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.963+0000 7f975b029740 1 --2- 172.21.15.39:0/124307087 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55858f0ce420 0x55858f0c33e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:46.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.963+0000 7f975b029740 1 --2- 172.21.15.39:0/124307087 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55858f0c88c0 0x55858f0c2960 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:46.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.964+0000 7f975b029740 1 -- 172.21.15.39:0/124307087 >> 172.21.15.39:0/124307087 conn(0x55858f0c3e30 msgr2=0x55858f0c4210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:46.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.965+0000 7f975b029740 1 -- 172.21.15.39:0/124307087 shutdown_connections 2024-03-18T16:11:46.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.965+0000 7f975b029740 1 -- 172.21.15.39:0/124307087 wait complete. 2024-03-18T16:11:46.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.966+0000 7f975b029740 1 -- start start 2024-03-18T16:11:46.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.966+0000 7f975b029740 1 Processor -- start 2024-03-18T16:11:46.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.970+0000 7f975b029740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55858f0c88c0 0x55858f0d6fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:46.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.971+0000 7f975b029740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55858f0c92b0 0x55858f0d7520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:46.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.971+0000 7f975b029740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55858f0ce420 0x55858f0d8530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:46.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.972+0000 7f9756400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55858f0ce420 0x55858f0d8530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:46.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.972+0000 7f9756400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55858f0ce420 0x55858f0d8530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:49418/0 (socket says 172.21.15.39:49418) 2024-03-18T16:11:46.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.972+0000 7f9756400640 1 -- 172.21.15.39:0/3652637413 learned_addr learned my addr 172.21.15.39:0/3652637413 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:11:46.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.972+0000 7f975b029740 1 -- 172.21.15.39:0/3652637413 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f97501b1ff0 con 0x55858f0c88c0 2024-03-18T16:11:46.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.972+0000 7f975b029740 1 -- 172.21.15.39:0/3652637413 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f97501b1920 con 0x55858f0ce420 2024-03-18T16:11:46.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.972+0000 7f975b029740 1 -- 172.21.15.39:0/3652637413 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f97501aeb60 con 0x55858f0c92b0 2024-03-18T16:11:46.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.972+0000 7f9755000640 1 --2- 172.21.15.39:0/3652637413 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55858f0c92b0 0x55858f0d7520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:46.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.973+0000 7f9756400640 1 -- 172.21.15.39:0/3652637413 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55858f0c92b0 msgr2=0x55858f0d7520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:46.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.973+0000 7f9756400640 1 --2- 172.21.15.39:0/3652637413 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55858f0c92b0 0x55858f0d7520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:46.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.973+0000 7f9756400640 1 -- 172.21.15.39:0/3652637413 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55858f0c88c0 msgr2=0x55858f0d6fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:46.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.973+0000 7f9756400640 1 --2- 172.21.15.39:0/3652637413 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55858f0c88c0 0x55858f0d6fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:46.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.973+0000 7f9756400640 1 -- 172.21.15.39:0/3652637413 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55858f0d8b20 con 0x55858f0ce420 2024-03-18T16:11:46.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.973+0000 7f975b029740 10 client.0 _ll_register_callbacks cb 0x7f973c0d1b90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:11:46.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.973+0000 7f975b029740 1 -- 172.21.15.39:0/3652637413 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9744009520 con 0x55858f0ce420 2024-03-18T16:11:46.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.975+0000 7f9756400640 1 --2- 172.21.15.39:0/3652637413 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55858f0ce420 0x55858f0d8530 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f97480126b0 tx=0x7f9748009500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a3dd058d77bdb8ad server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:46.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.977+0000 7f9737400640 10 client.25955 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:11:46.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.977+0000 7f9737400640 1 -- 172.21.15.39:0/3652637413 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f974802d050 con 0x55858f0ce420 2024-03-18T16:11:46.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.977+0000 7f9737400640 1 -- 172.21.15.39:0/3652637413 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f9748003c40 con 0x55858f0ce420 2024-03-18T16:11:46.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.978+0000 7f9737400640 1 -- 172.21.15.39:0/3652637413 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f9748023070 con 0x55858f0ce420 2024-03-18T16:11:46.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.978+0000 7f975b029740 10 client.25955 fetch_fsmap learned FSMap version 30 2024-03-18T16:11:46.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.978+0000 7f975b029740 1 -- 172.21.15.39:0/3652637413 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({fsmap=30}) -- 0x55858f0d9de0 con 0x55858f0ce420 2024-03-18T16:11:46.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.981+0000 7f9737400640 1 -- 172.21.15.39:0/3652637413 <== mon.2 v2:172.21.15.194:3300/0 4 ==== fsmap(e 30) ==== 8370+0+0 (secure 0 0 0) 0x7f974800f040 con 0x55858f0ce420 2024-03-18T16:11:46.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.981+0000 7f975b029740 10 client.25955 fetch_fsmap finished waiting for FSMap version 30 2024-03-18T16:11:46.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.981+0000 7f975b029740 10 client.25955 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-18T16:11:46.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.981+0000 7f975b029740 20 client.25955 populate_metadata read hostname 'smithi039' 2024-03-18T16:11:46.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.981+0000 7f975b029740 1 --2- 172.21.15.39:0/3652637413 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x55858f144a60 0x55858f146ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:46.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.981+0000 7f975b029740 4 client.25955 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:11:46.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.981+0000 7f975b029740 1 -- 172.21.15.39:0/3652637413 --> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55858f03a990 con 0x55858f144a60 2024-03-18T16:11:46.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.982+0000 7f9755a00640 1 --2- 172.21.15.39:0/3652637413 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x55858f144a60 0x55858f146ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:46.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.983+0000 7f9755a00640 1 --2- 172.21.15.39:0/3652637413 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x55858f144a60 0x55858f146ec0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=dd05dee1e4c409e server_cookie=4567b1b69feb100b in_seq=0 out_seq=0 2024-03-18T16:11:46.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.983+0000 7f9737400640 10 client.25955 ms_handle_connect on v2:172.21.15.196:6836/895247251 2024-03-18T16:11:46.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.984+0000 7f9737400640 1 -- 172.21.15.39:0/3652637413 <== mds.2 v2:172.21.15.196:6836/895247251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55858f03a990 con 0x55858f144a60 2024-03-18T16:11:46.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.984+0000 7f9737400640 10 client.25955 handle_command_reply: tid=0 2024-03-18T16:11:46.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.984+0000 7f975b029740 1 client.25955 shutdown 2024-03-18T16:11:46.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.987+0000 7f975b029740 1 -- 172.21.15.39:0/3652637413 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55858f0ce420 msgr2=0x55858f0d8530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:46.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.987+0000 7f975b029740 1 --2- 172.21.15.39:0/3652637413 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55858f0ce420 0x55858f0d8530 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f97480126b0 tx=0x7f9748009500 comp rx=0 tx=0).stop 2024-03-18T16:11:46.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.987+0000 7f975b029740 1 -- 172.21.15.39:0/3652637413 shutdown_connections 2024-03-18T16:11:46.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.987+0000 7f975b029740 1 --2- 172.21.15.39:0/3652637413 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55858f0ce420 0x55858f0d8530 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:46.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.987+0000 7f975b029740 1 --2- 172.21.15.39:0/3652637413 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x55858f144a60 0x55858f146ec0 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:46.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.987+0000 7f975b029740 1 --2- 172.21.15.39:0/3652637413 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55858f0c92b0 0x55858f0d7520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:46.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.987+0000 7f975b029740 1 --2- 172.21.15.39:0/3652637413 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55858f0c88c0 0x55858f0d6fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:46.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.987+0000 7f9737400640 0 client.25955 ms_handle_reset on v2:172.21.15.196:6836/895247251 2024-03-18T16:11:46.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.987+0000 7f975b029740 1 -- 172.21.15.39:0/3652637413 >> 172.21.15.39:0/3652637413 conn(0x55858f0c3e30 msgr2=0x55858f0c2960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:46.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.988+0000 7f975b029740 1 -- 172.21.15.39:0/3652637413 shutdown_connections 2024-03-18T16:11:46.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.990+0000 7f975b029740 1 -- 172.21.15.39:0/3652637413 wait complete. 2024-03-18T16:11:46.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:46.990+0000 7f975b029740 20 client.25955 trim_cache size 0 max 16384 2024-03-18T16:11:46.996 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:11:47.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.001+0000 7f9758400640 1 -- 172.21.15.39:0/2356113558 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f97380a2910 msgr2=0x7f97380a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:47.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.001+0000 7f9758400640 1 --2- 172.21.15.39:0/2356113558 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f97380a2910 0x7f97380a4dd0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f9748005c40 tx=0x7f9748004a40 comp rx=0 tx=0).stop 2024-03-18T16:11:47.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.001+0000 7f9758400640 1 -- 172.21.15.39:0/2356113558 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f97500773e0 msgr2=0x7f975019e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:47.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.001+0000 7f9758400640 1 --2- 172.21.15.39:0/2356113558 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f97500773e0 0x7f975019e660 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f973c00b4a0 tx=0x7f973c00b960 comp rx=0 tx=0).stop 2024-03-18T16:11:47.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.003+0000 7f9758400640 1 -- 172.21.15.39:0/2356113558 shutdown_connections 2024-03-18T16:11:47.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.003+0000 7f9758400640 1 --2- 172.21.15.39:0/2356113558 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f97500773e0 0x7f975019e660 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:47.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.003+0000 7f9758400640 1 --2- 172.21.15.39:0/2356113558 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f97380a2910 0x7f97380a4dd0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:47.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.003+0000 7f9758400640 1 --2- 172.21.15.39:0/2356113558 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9750077d80 0x7f97501a56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:47.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.003+0000 7f9758400640 1 --2- 172.21.15.39:0/2356113558 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f97500761a0 0x7f975019e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:47.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.003+0000 7f9758400640 1 -- 172.21.15.39:0/2356113558 >> 172.21.15.39:0/2356113558 conn(0x7f97500fef50 msgr2=0x7f9750100a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:47.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.004+0000 7f9758400640 1 -- 172.21.15.39:0/2356113558 shutdown_connections 2024-03-18T16:11:47.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.004+0000 7f9758400640 1 -- 172.21.15.39:0/2356113558 wait complete. 2024-03-18T16:11:47.049 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-18T16:11:47.342 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:11:47 smithi039 ceph-mon[31721]: pgmap v473: 129 pgs: 129 active+clean; 2.4 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 197 MiB/s wr, 86 op/s 2024-03-18T16:11:47.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.431+0000 7fbd7a800640 1 -- 172.21.15.39:0/3078843168 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbd741027b0 msgr2=0x7fbd74104c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:47.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.431+0000 7fbd7a800640 1 --2- 172.21.15.39:0/3078843168 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbd741027b0 0x7fbd74104c20 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fbd60009900 tx=0x7fbd6002bc30 comp rx=0 tx=0).stop 2024-03-18T16:11:47.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.431+0000 7fbd7a800640 1 -- 172.21.15.39:0/3078843168 shutdown_connections 2024-03-18T16:11:47.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.431+0000 7fbd7a800640 1 --2- 172.21.15.39:0/3078843168 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbd741027b0 0x7fbd74104c20 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:47.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.431+0000 7fbd7a800640 1 --2- 172.21.15.39:0/3078843168 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbd74105160 0x7fbd741055e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:47.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.431+0000 7fbd7a800640 1 --2- 172.21.15.39:0/3078843168 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbd740ffe40 0x7fbd74102270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:47.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.431+0000 7fbd7a800640 1 -- 172.21.15.39:0/3078843168 >> 172.21.15.39:0/3078843168 conn(0x7fbd740fba80 msgr2=0x7fbd740fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:47.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.431+0000 7fbd7a800640 1 -- 172.21.15.39:0/3078843168 shutdown_connections 2024-03-18T16:11:47.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.431+0000 7fbd7a800640 1 -- 172.21.15.39:0/3078843168 wait complete. 2024-03-18T16:11:47.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.433+0000 7fbd7a800640 1 Processor -- start 2024-03-18T16:11:47.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.433+0000 7fbd7a800640 1 -- start start 2024-03-18T16:11:47.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.433+0000 7fbd7a800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbd740ffe40 0x7fbd7419ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:47.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.433+0000 7fbd7a800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbd741027b0 0x7fbd7419e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:47.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.433+0000 7fbd7a800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbd74105160 0x7fbd741a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:47.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.433+0000 7fbd7a800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fbd741138f0 con 0x7fbd74105160 2024-03-18T16:11:47.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.433+0000 7fbd7a800640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fbd74113770 con 0x7fbd740ffe40 2024-03-18T16:11:47.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.433+0000 7fbd7a800640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fbd74113a70 con 0x7fbd741027b0 2024-03-18T16:11:47.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.434+0000 7fbd79400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbd740ffe40 0x7fbd7419ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:47.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.434+0000 7fbd79400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbd740ffe40 0x7fbd7419ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:49438/0 (socket says 172.21.15.39:49438) 2024-03-18T16:11:47.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.434+0000 7fbd78a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbd741027b0 0x7fbd7419e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:47.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.434+0000 7fbd79400640 1 -- 172.21.15.39:0/930732673 learned_addr learned my addr 172.21.15.39:0/930732673 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:11:47.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.434+0000 7fbd79400640 1 -- 172.21.15.39:0/930732673 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbd741027b0 msgr2=0x7fbd7419e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:47.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.434+0000 7fbd79400640 1 --2- 172.21.15.39:0/930732673 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbd741027b0 0x7fbd7419e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:47.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.434+0000 7fbd79400640 1 -- 172.21.15.39:0/930732673 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbd74105160 msgr2=0x7fbd741a5480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:47.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.434+0000 7fbd79400640 1 --2- 172.21.15.39:0/930732673 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbd74105160 0x7fbd741a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:47.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.434+0000 7fbd79400640 1 -- 172.21.15.39:0/930732673 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd741a5af0 con 0x7fbd740ffe40 2024-03-18T16:11:47.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.435+0000 7fbd79400640 1 --2- 172.21.15.39:0/930732673 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbd740ffe40 0x7fbd7419ded0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fbd6800e9f0 tx=0x7fbd6800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:47.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.435+0000 7fbd72000640 1 -- 172.21.15.39:0/930732673 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd6800cd80 con 0x7fbd740ffe40 2024-03-18T16:11:47.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.435+0000 7fbd72000640 1 -- 172.21.15.39:0/930732673 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd68010910 con 0x7fbd740ffe40 2024-03-18T16:11:47.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.435+0000 7fbd72000640 1 -- 172.21.15.39:0/930732673 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd68018990 con 0x7fbd740ffe40 2024-03-18T16:11:47.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.435+0000 7fbd7a800640 1 -- 172.21.15.39:0/930732673 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd741a5de0 con 0x7fbd740ffe40 2024-03-18T16:11:47.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.435+0000 7fbd7a800640 1 -- 172.21.15.39:0/930732673 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd741a61f0 con 0x7fbd740ffe40 2024-03-18T16:11:47.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.440+0000 7fbd7eb6c740 1 Processor -- start 2024-03-18T16:11:47.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.440+0000 7fbd7eb6c740 1 -- start start 2024-03-18T16:11:47.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.440+0000 7fbd7eb6c740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5610d7a60a90 0x5610d7a60eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:47.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.440+0000 7fbd7eb6c740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5610d7a61480 0x5610d7a65f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:47.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.440+0000 7fbd7eb6c740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5610d7a665f0 0x5610d7a68a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:47.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.440+0000 7fbd7eb6c740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5610d7a283b0 con 0x5610d7a665f0 2024-03-18T16:11:47.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.440+0000 7fbd7eb6c740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5610d79c5450 con 0x5610d7a60a90 2024-03-18T16:11:47.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.440+0000 7fbd7eb6c740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x5610d796dcb0 con 0x5610d7a61480 2024-03-18T16:11:47.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.441+0000 7fbd79e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5610d7a60a90 0x5610d7a60eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:47.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.441+0000 7fbd79e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5610d7a60a90 0x5610d7a60eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:49440/0 (socket says 172.21.15.39:49440) 2024-03-18T16:11:47.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.441+0000 7fbd79e00640 1 -- 172.21.15.39:0/2520616892 learned_addr learned my addr 172.21.15.39:0/2520616892 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:11:47.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.441+0000 7fbd78a00640 1 --2- 172.21.15.39:0/2520616892 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5610d7a665f0 0x5610d7a68a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:47.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.442+0000 7fbd72000640 1 -- 172.21.15.39:0/930732673 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fbd68010ac0 con 0x7fbd740ffe40 2024-03-18T16:11:47.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.442+0000 7fbd78a00640 1 -- 172.21.15.39:0/2520616892 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5610d7a61480 msgr2=0x5610d7a65f80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:11:47.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.442+0000 7fbd78a00640 1 --2- 172.21.15.39:0/2520616892 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5610d7a61480 0x5610d7a65f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:47.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.442+0000 7fbd78a00640 1 -- 172.21.15.39:0/2520616892 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5610d7a60a90 msgr2=0x5610d7a60eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:47.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.442+0000 7fbd78a00640 1 --2- 172.21.15.39:0/2520616892 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5610d7a60a90 0x5610d7a60eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:47.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.442+0000 7fbd78a00640 1 -- 172.21.15.39:0/2520616892 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5610d798f1f0 con 0x5610d7a665f0 2024-03-18T16:11:47.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.442+0000 7fbd79e00640 1 --2- 172.21.15.39:0/2520616892 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5610d7a60a90 0x5610d7a60eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:11:47.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.442+0000 7fbd78a00640 1 --2- 172.21.15.39:0/2520616892 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5610d7a665f0 0x5610d7a68a80 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fbd60036040 tx=0x7fbd6002c680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d675ea6e555ced server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:47.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.443+0000 7fbd66600640 1 -- 172.21.15.39:0/2520616892 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd60045070 con 0x5610d7a665f0 2024-03-18T16:11:47.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.443+0000 7fbd66600640 1 -- 172.21.15.39:0/2520616892 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd6002ca40 con 0x5610d7a665f0 2024-03-18T16:11:47.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.443+0000 7fbd66600640 1 -- 172.21.15.39:0/2520616892 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd600046c0 con 0x5610d7a665f0 2024-03-18T16:11:47.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.443+0000 7fbd7eb6c740 1 -- 172.21.15.39:0/2520616892 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5610d7a665f0 msgr2=0x5610d7a68a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:47.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.444+0000 7fbd7eb6c740 1 --2- 172.21.15.39:0/2520616892 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5610d7a665f0 0x5610d7a68a80 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fbd60036040 tx=0x7fbd6002c680 comp rx=0 tx=0).stop 2024-03-18T16:11:47.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.444+0000 7fbd7eb6c740 1 -- 172.21.15.39:0/2520616892 shutdown_connections 2024-03-18T16:11:47.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.444+0000 7fbd7eb6c740 1 --2- 172.21.15.39:0/2520616892 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5610d7a665f0 0x5610d7a68a80 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:47.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.444+0000 7fbd7eb6c740 1 --2- 172.21.15.39:0/2520616892 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5610d7a61480 0x5610d7a65f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:47.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.444+0000 7fbd7eb6c740 1 --2- 172.21.15.39:0/2520616892 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5610d7a60a90 0x5610d7a60eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:47.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.444+0000 7fbd7eb6c740 1 -- 172.21.15.39:0/2520616892 >> 172.21.15.39:0/2520616892 conn(0x5610d7a5c000 msgr2=0x5610d7a5c400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:47.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.444+0000 7fbd7eb6c740 1 -- 172.21.15.39:0/2520616892 shutdown_connections 2024-03-18T16:11:47.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.444+0000 7fbd7eb6c740 1 -- 172.21.15.39:0/2520616892 wait complete. 2024-03-18T16:11:47.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.444+0000 7fbd72000640 1 --2- 172.21.15.39:0/930732673 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fbd5c0a27e0 0x7fbd5c0a4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:47.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.445+0000 7fbd72000640 1 -- 172.21.15.39:0/930732673 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fbd680cf130 con 0x7fbd740ffe40 2024-03-18T16:11:47.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.445+0000 7fbd7eb6c740 1 -- start start 2024-03-18T16:11:47.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.445+0000 7fbd7eb6c740 1 Processor -- start 2024-03-18T16:11:47.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.445+0000 7fbd78a00640 1 --2- 172.21.15.39:0/930732673 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fbd5c0a27e0 0x7fbd5c0a4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:47.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.446+0000 7fbd7eb6c740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5610d7a60a90 0x5610d7a6ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:47.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.446+0000 7fbd7eb6c740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5610d7a61480 0x5610d7a6b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:47.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.446+0000 7fbd7eb6c740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5610d7a665f0 0x5610d7a722c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:47.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.446+0000 7fbd7eb6c740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5610d7981110 con 0x5610d7a60a90 2024-03-18T16:11:47.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.446+0000 7fbd7eb6c740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5610d79c14f0 con 0x5610d7a665f0 2024-03-18T16:11:47.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.446+0000 7fbd7eb6c740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x5610d7a69540 con 0x5610d7a61480 2024-03-18T16:11:47.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.446+0000 7fbd7eb6c740 10 client.0 _ll_register_callbacks cb 0x5610d796eb70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:11:47.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.446+0000 7fbd78a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5610d7a61480 0x5610d7a6b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:47.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.446+0000 7fbd79400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5610d7a60a90 0x5610d7a6ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:47.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.447+0000 7fbd78a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5610d7a61480 0x5610d7a6b2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:33892/0 (socket says 172.21.15.39:33892) 2024-03-18T16:11:47.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.447+0000 7fbd78a00640 1 -- 172.21.15.39:0/3313754768 learned_addr learned my addr 172.21.15.39:0/3313754768 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:11:47.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.447+0000 7fbd79e00640 1 --2- 172.21.15.39:0/3313754768 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5610d7a665f0 0x5610d7a722c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:47.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.447+0000 7fbd79e00640 1 -- 172.21.15.39:0/3313754768 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5610d7a61480 msgr2=0x5610d7a6b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:47.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.447+0000 7fbd79e00640 1 --2- 172.21.15.39:0/3313754768 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5610d7a61480 0x5610d7a6b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:47.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.447+0000 7fbd79e00640 1 -- 172.21.15.39:0/3313754768 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5610d7a60a90 msgr2=0x5610d7a6ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:47.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.447+0000 7fbd79e00640 1 --2- 172.21.15.39:0/3313754768 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5610d7a60a90 0x5610d7a6ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:47.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.448+0000 7fbd79e00640 1 -- 172.21.15.39:0/3313754768 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5610d76d93f0 con 0x5610d7a665f0 2024-03-18T16:11:47.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.448+0000 7fbd79e00640 1 --2- 172.21.15.39:0/3313754768 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5610d7a665f0 0x5610d7a722c0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fbd6c00e640 tx=0x7fbd6c00eb00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=51ac5793cca36728 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:47.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.448+0000 7fbd78a00640 1 --2- 172.21.15.39:0/930732673 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fbd5c0a27e0 0x7fbd5c0a4ca0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fbd60036040 tx=0x7fbd6002cdd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:47.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.448+0000 7fbd5aa00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:11:47.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.449+0000 7fbd5aa00640 1 -- 172.21.15.39:0/3313754768 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd6c018020 con 0x5610d7a665f0 2024-03-18T16:11:47.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.449+0000 7fbd5aa00640 1 -- 172.21.15.39:0/3313754768 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fbd6c014810 con 0x5610d7a665f0 2024-03-18T16:11:47.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.449+0000 7fbd7eb6c740 1 -- 172.21.15.39:0/3313754768 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbd6002ca40 con 0x5610d7a665f0 2024-03-18T16:11:47.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.449+0000 7fbd5aa00640 1 -- 172.21.15.39:0/3313754768 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fbd6c014a10 con 0x5610d7a665f0 2024-03-18T16:11:47.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.449+0000 7fbd7eb6c740 10 client.25967 fetch_fsmap learned FSMap version 31 2024-03-18T16:11:47.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.449+0000 7fbd7eb6c740 1 -- 172.21.15.39:0/3313754768 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5610d77a37d0 con 0x5610d7a665f0 2024-03-18T16:11:47.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.449+0000 7fbd5aa00640 1 -- 172.21.15.39:0/3313754768 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd6002ca40 con 0x5610d7a665f0 2024-03-18T16:11:47.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.450+0000 7fbd5aa00640 1 -- 172.21.15.39:0/3313754768 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 31) ==== 8374+0+0 (secure 0 0 0) 0x7fbd6c025020 con 0x5610d7a665f0 2024-03-18T16:11:47.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.450+0000 7fbd7eb6c740 10 client.25967 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T16:11:47.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.450+0000 7fbd7eb6c740 10 client.25967 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T16:11:47.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.450+0000 7fbd7eb6c740 20 client.25967 populate_metadata read hostname 'smithi039' 2024-03-18T16:11:47.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.450+0000 7fbd7eb6c740 1 --2- 172.21.15.39:0/3313754768 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x5610d7a77130 0x5610d7a79570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:47.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.450+0000 7fbd7eb6c740 4 client.25967 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:11:47.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.450+0000 7fbd7eb6c740 1 -- 172.21.15.39:0/3313754768 --> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5610d79d2990 con 0x5610d7a77130 2024-03-18T16:11:47.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.450+0000 7fbd79400640 1 --2- 172.21.15.39:0/3313754768 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x5610d7a77130 0x5610d7a79570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:47.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.451+0000 7fbd79400640 1 --2- 172.21.15.39:0/3313754768 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x5610d7a77130 0x5610d7a79570 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c7e7a606d2eeaeeb server_cookie=e9d1fc908af4b53f in_seq=0 out_seq=0 2024-03-18T16:11:47.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.451+0000 7fbd5aa00640 10 client.25967 ms_handle_connect on v2:172.21.15.196:6838/2001427268 2024-03-18T16:11:47.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.452+0000 7fbd5aa00640 1 -- 172.21.15.39:0/3313754768 <== mds.1 v2:172.21.15.196:6838/2001427268 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5610d79d2990 con 0x5610d7a77130 2024-03-18T16:11:47.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.452+0000 7fbd5aa00640 10 client.25967 handle_command_reply: tid=0 2024-03-18T16:11:47.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.452+0000 7fbd7eb6c740 1 client.25967 shutdown 2024-03-18T16:11:47.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.452+0000 7fbd7eb6c740 1 -- 172.21.15.39:0/3313754768 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5610d7a665f0 msgr2=0x5610d7a722c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:47.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.452+0000 7fbd7eb6c740 1 --2- 172.21.15.39:0/3313754768 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5610d7a665f0 0x5610d7a722c0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fbd6c00e640 tx=0x7fbd6c00eb00 comp rx=0 tx=0).stop 2024-03-18T16:11:47.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.452+0000 7fbd7eb6c740 1 -- 172.21.15.39:0/3313754768 shutdown_connections 2024-03-18T16:11:47.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.452+0000 7fbd7eb6c740 1 --2- 172.21.15.39:0/3313754768 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x5610d7a77130 0x5610d7a79570 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:47.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.452+0000 7fbd7eb6c740 1 --2- 172.21.15.39:0/3313754768 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5610d7a665f0 0x5610d7a722c0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:47.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.452+0000 7fbd7eb6c740 1 --2- 172.21.15.39:0/3313754768 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5610d7a61480 0x5610d7a6b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:47.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.452+0000 7fbd7eb6c740 1 --2- 172.21.15.39:0/3313754768 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5610d7a60a90 0x5610d7a6ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:47.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.452+0000 7fbd5aa00640 0 client.25967 ms_handle_reset on v2:172.21.15.196:6838/2001427268 2024-03-18T16:11:47.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.452+0000 7fbd7eb6c740 1 -- 172.21.15.39:0/3313754768 >> 172.21.15.39:0/3313754768 conn(0x5610d7a5c000 msgr2=0x5610d7a5b220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:47.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.453+0000 7fbd7eb6c740 1 -- 172.21.15.39:0/3313754768 shutdown_connections 2024-03-18T16:11:47.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.453+0000 7fbd7eb6c740 1 -- 172.21.15.39:0/3313754768 wait complete. 2024-03-18T16:11:47.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.453+0000 7fbd7eb6c740 20 client.25967 trim_cache size 0 max 16384 2024-03-18T16:11:47.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:11:47 smithi196 ceph-mon[36307]: pgmap v473: 129 pgs: 129 active+clean; 2.4 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 197 MiB/s wr, 86 op/s 2024-03-18T16:11:47.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.484+0000 7fbd7eb6c740 1 Processor -- start 2024-03-18T16:11:47.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.484+0000 7fbd7eb6c740 1 -- start start 2024-03-18T16:11:47.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.485+0000 7fbd7eb6c740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5610d7a60a90 0x5610d7a5a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:47.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.485+0000 7fbd7eb6c740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5610d7a61480 0x5610d7a72280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:47.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.485+0000 7fbd7eb6c740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5610d7a665f0 0x5610d7a5aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:47.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.485+0000 7fbd7eb6c740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fbd741b0890 con 0x5610d7a61480 2024-03-18T16:11:47.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.485+0000 7fbd7eb6c740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fbd741b2460 con 0x5610d7a665f0 2024-03-18T16:11:47.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.485+0000 7fbd7eb6c740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fbd741b0f60 con 0x5610d7a60a90 2024-03-18T16:11:47.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.485+0000 7fbd79400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5610d7a60a90 0x5610d7a5a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:47.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.486+0000 7fbd79e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5610d7a665f0 0x5610d7a5aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:47.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.486+0000 7fbd78a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5610d7a61480 0x5610d7a72280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:47.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.486+0000 7fbd79e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5610d7a665f0 0x5610d7a5aed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:49464/0 (socket says 172.21.15.39:49464) 2024-03-18T16:11:47.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.486+0000 7fbd79e00640 1 -- 172.21.15.39:0/2276939188 learned_addr learned my addr 172.21.15.39:0/2276939188 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:11:47.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.486+0000 7fbd78a00640 1 -- 172.21.15.39:0/2276939188 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5610d7a60a90 msgr2=0x5610d7a5a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:47.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.486+0000 7fbd78a00640 1 --2- 172.21.15.39:0/2276939188 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5610d7a60a90 0x5610d7a5a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:47.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.486+0000 7fbd78a00640 1 -- 172.21.15.39:0/2276939188 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5610d7a665f0 msgr2=0x5610d7a5aed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:47.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.486+0000 7fbd78a00640 1 --2- 172.21.15.39:0/2276939188 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5610d7a665f0 0x5610d7a5aed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:47.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.486+0000 7fbd78a00640 1 -- 172.21.15.39:0/2276939188 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5610d7a65bd0 con 0x5610d7a61480 2024-03-18T16:11:47.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.487+0000 7fbd78a00640 1 --2- 172.21.15.39:0/2276939188 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5610d7a61480 0x5610d7a72280 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fbd60044000 tx=0x7fbd60054000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28bf776a41746fbc server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:47.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.487+0000 7fbd65c00640 1 -- 172.21.15.39:0/2276939188 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd6005b070 con 0x5610d7a61480 2024-03-18T16:11:47.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.488+0000 7fbd65c00640 1 -- 172.21.15.39:0/2276939188 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd600050c0 con 0x5610d7a61480 2024-03-18T16:11:47.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.488+0000 7fbd65c00640 1 -- 172.21.15.39:0/2276939188 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd6005b070 con 0x5610d7a61480 2024-03-18T16:11:47.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.489+0000 7fbd7eb6c740 1 -- 172.21.15.39:0/2276939188 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5610d7a61480 msgr2=0x5610d7a72280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:47.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.489+0000 7fbd7eb6c740 1 --2- 172.21.15.39:0/2276939188 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5610d7a61480 0x5610d7a72280 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fbd60044000 tx=0x7fbd60054000 comp rx=0 tx=0).stop 2024-03-18T16:11:47.493 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:11:47 smithi194 ceph-mon[36200]: pgmap v473: 129 pgs: 129 active+clean; 2.4 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 197 MiB/s wr, 86 op/s 2024-03-18T16:11:47.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.489+0000 7fbd7eb6c740 1 -- 172.21.15.39:0/2276939188 shutdown_connections 2024-03-18T16:11:47.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.489+0000 7fbd7eb6c740 1 --2- 172.21.15.39:0/2276939188 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5610d7a665f0 0x5610d7a5aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:47.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.489+0000 7fbd7eb6c740 1 --2- 172.21.15.39:0/2276939188 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5610d7a61480 0x5610d7a72280 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:47.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.489+0000 7fbd7eb6c740 1 --2- 172.21.15.39:0/2276939188 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5610d7a60a90 0x5610d7a5a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:47.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.489+0000 7fbd7eb6c740 1 -- 172.21.15.39:0/2276939188 >> 172.21.15.39:0/2276939188 conn(0x5610d7a5c000 msgr2=0x5610d7a6ad60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:47.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.490+0000 7fbd7eb6c740 1 -- 172.21.15.39:0/2276939188 shutdown_connections 2024-03-18T16:11:47.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.490+0000 7fbd7eb6c740 1 -- 172.21.15.39:0/2276939188 wait complete. 2024-03-18T16:11:47.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.490+0000 7fbd7eb6c740 1 -- start start 2024-03-18T16:11:47.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.491+0000 7fbd7eb6c740 1 Processor -- start 2024-03-18T16:11:47.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.492+0000 7fbd7eb6c740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5610d7a60a90 0x5610d7a6f810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:47.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.492+0000 7fbd7eb6c740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5610d7a61480 0x5610d7a6fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:47.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.492+0000 7fbd7eb6c740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5610d7a665f0 0x5610d7a70d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:47.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.492+0000 7fbd7eb6c740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fbd741b1d30 con 0x5610d7a60a90 2024-03-18T16:11:47.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.492+0000 7fbd7eb6c740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fbd741b1630 con 0x5610d7a665f0 2024-03-18T16:11:47.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.492+0000 7fbd7eb6c740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fbd741ae830 con 0x5610d7a61480 2024-03-18T16:11:47.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.492+0000 7fbd7eb6c740 10 client.0 _ll_register_callbacks cb 0x7fbd6c0098f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:11:47.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.492+0000 7fbd79e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5610d7a665f0 0x5610d7a70d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:47.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.492+0000 7fbd79e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5610d7a665f0 0x5610d7a70d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:49470/0 (socket says 172.21.15.39:49470) 2024-03-18T16:11:47.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.492+0000 7fbd79e00640 1 -- 172.21.15.39:0/2552848737 learned_addr learned my addr 172.21.15.39:0/2552848737 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:11:47.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.493+0000 7fbd78a00640 1 --2- 172.21.15.39:0/2552848737 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5610d7a61480 0x5610d7a6fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:47.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.493+0000 7fbd79400640 1 --2- 172.21.15.39:0/2552848737 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5610d7a60a90 0x5610d7a6f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:47.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.493+0000 7fbd79e00640 1 -- 172.21.15.39:0/2552848737 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5610d7a61480 msgr2=0x5610d7a6fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:47.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.493+0000 7fbd79e00640 1 --2- 172.21.15.39:0/2552848737 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5610d7a61480 0x5610d7a6fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:47.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.494+0000 7fbd79e00640 1 -- 172.21.15.39:0/2552848737 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5610d7a60a90 msgr2=0x5610d7a6f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:47.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.494+0000 7fbd79e00640 1 --2- 172.21.15.39:0/2552848737 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5610d7a60a90 0x5610d7a6f810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:47.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.494+0000 7fbd79e00640 1 -- 172.21.15.39:0/2552848737 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5610d7a71350 con 0x5610d7a665f0 2024-03-18T16:11:47.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.494+0000 7fbd79400640 1 --2- 172.21.15.39:0/2552848737 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5610d7a60a90 0x5610d7a6f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:11:47.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.498+0000 7fbd79e00640 1 --2- 172.21.15.39:0/2552848737 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5610d7a665f0 0x5610d7a70d60 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fbd6c01a460 tx=0x7fbd6c004b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=24a39119ca88a6c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:47.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.498+0000 7fbd5aa00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:11:47.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.499+0000 7fbd5aa00640 1 -- 172.21.15.39:0/2552848737 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd6c04f070 con 0x5610d7a665f0 2024-03-18T16:11:47.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.499+0000 7fbd7eb6c740 1 -- 172.21.15.39:0/2552848737 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbd600050c0 con 0x5610d7a665f0 2024-03-18T16:11:47.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.500+0000 7fbd5aa00640 1 -- 172.21.15.39:0/2552848737 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fbd6c013c70 con 0x5610d7a665f0 2024-03-18T16:11:47.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.500+0000 7fbd5aa00640 1 -- 172.21.15.39:0/2552848737 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fbd6c05b050 con 0x5610d7a665f0 2024-03-18T16:11:47.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.500+0000 7fbd7eb6c740 10 client.25973 fetch_fsmap learned FSMap version 31 2024-03-18T16:11:47.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.500+0000 7fbd7eb6c740 1 -- 172.21.15.39:0/2552848737 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5610d7add250 con 0x5610d7a665f0 2024-03-18T16:11:47.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.501+0000 7fbd5aa00640 1 -- 172.21.15.39:0/2552848737 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd600050c0 con 0x5610d7a665f0 2024-03-18T16:11:47.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.501+0000 7fbd5aa00640 1 -- 172.21.15.39:0/2552848737 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 31) ==== 8374+0+0 (secure 0 0 0) 0x7fbd6c025020 con 0x5610d7a665f0 2024-03-18T16:11:47.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.503+0000 7fbd7eb6c740 10 client.25973 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T16:11:47.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.503+0000 7fbd7eb6c740 10 client.25973 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T16:11:47.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.503+0000 7fbd7eb6c740 20 client.25973 populate_metadata read hostname 'smithi039' 2024-03-18T16:11:47.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.503+0000 7fbd7eb6c740 1 --2- 172.21.15.39:0/2552848737 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x5610d7addaa0 0x5610d7adfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:47.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.503+0000 7fbd7eb6c740 4 client.25973 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:11:47.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.503+0000 7fbd7eb6c740 1 -- 172.21.15.39:0/2552848737 --> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5610d79ccce0 con 0x5610d7addaa0 2024-03-18T16:11:47.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.504+0000 7fbd79400640 1 --2- 172.21.15.39:0/2552848737 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x5610d7addaa0 0x5610d7adfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:47.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.505+0000 7fbd79400640 1 --2- 172.21.15.39:0/2552848737 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x5610d7addaa0 0x5610d7adfec0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8269e4d562b775e2 server_cookie=489ae68b40c11640 in_seq=0 out_seq=0 2024-03-18T16:11:47.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.505+0000 7fbd5aa00640 10 client.25973 ms_handle_connect on v2:172.21.15.196:6838/2001427268 2024-03-18T16:11:47.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.505+0000 7fbd5aa00640 1 -- 172.21.15.39:0/2552848737 <== mds.1 v2:172.21.15.196:6838/2001427268 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5610d79ccce0 con 0x5610d7addaa0 2024-03-18T16:11:47.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.505+0000 7fbd5aa00640 10 client.25973 handle_command_reply: tid=0 2024-03-18T16:11:47.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.506+0000 7fbd7eb6c740 1 client.25973 shutdown 2024-03-18T16:11:47.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.506+0000 7fbd7eb6c740 1 -- 172.21.15.39:0/2552848737 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5610d7a665f0 msgr2=0x5610d7a70d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:47.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.506+0000 7fbd7eb6c740 1 --2- 172.21.15.39:0/2552848737 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5610d7a665f0 0x5610d7a70d60 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fbd6c01a460 tx=0x7fbd6c004b50 comp rx=0 tx=0).stop 2024-03-18T16:11:47.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.506+0000 7fbd7eb6c740 1 -- 172.21.15.39:0/2552848737 shutdown_connections 2024-03-18T16:11:47.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.506+0000 7fbd7eb6c740 1 --2- 172.21.15.39:0/2552848737 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x5610d7addaa0 0x5610d7adfec0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:47.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.506+0000 7fbd7eb6c740 1 --2- 172.21.15.39:0/2552848737 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5610d7a665f0 0x5610d7a70d60 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:47.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.506+0000 7fbd7eb6c740 1 --2- 172.21.15.39:0/2552848737 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5610d7a61480 0x5610d7a6fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:47.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.506+0000 7fbd7eb6c740 1 --2- 172.21.15.39:0/2552848737 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5610d7a60a90 0x5610d7a6f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:47.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.506+0000 7fbd7eb6c740 1 -- 172.21.15.39:0/2552848737 >> 172.21.15.39:0/2552848737 conn(0x5610d7a5c000 msgr2=0x5610d7a5f260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:47.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.506+0000 7fbd5aa00640 0 client.25973 ms_handle_reset on v2:172.21.15.196:6838/2001427268 2024-03-18T16:11:47.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.507+0000 7fbd7eb6c740 1 -- 172.21.15.39:0/2552848737 shutdown_connections 2024-03-18T16:11:47.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.507+0000 7fbd7eb6c740 1 -- 172.21.15.39:0/2552848737 wait complete. 2024-03-18T16:11:47.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.507+0000 7fbd7eb6c740 20 client.25973 trim_cache size 0 max 16384 2024-03-18T16:11:47.510 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:11:47.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.508+0000 7fbd7a800640 1 -- 172.21.15.39:0/930732673 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fbd5c0a27e0 msgr2=0x7fbd5c0a4ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:47.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.508+0000 7fbd7a800640 1 --2- 172.21.15.39:0/930732673 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fbd5c0a27e0 0x7fbd5c0a4ca0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fbd60036040 tx=0x7fbd6002cdd0 comp rx=0 tx=0).stop 2024-03-18T16:11:47.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.508+0000 7fbd7a800640 1 -- 172.21.15.39:0/930732673 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbd740ffe40 msgr2=0x7fbd7419ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:47.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.508+0000 7fbd7a800640 1 --2- 172.21.15.39:0/930732673 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbd740ffe40 0x7fbd7419ded0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fbd6800e9f0 tx=0x7fbd6800eeb0 comp rx=0 tx=0).stop 2024-03-18T16:11:47.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.508+0000 7fbd7a800640 1 -- 172.21.15.39:0/930732673 shutdown_connections 2024-03-18T16:11:47.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.508+0000 7fbd7a800640 1 --2- 172.21.15.39:0/930732673 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fbd5c0a27e0 0x7fbd5c0a4ca0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:47.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.508+0000 7fbd7a800640 1 --2- 172.21.15.39:0/930732673 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbd74105160 0x7fbd741a5480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:47.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.508+0000 7fbd7a800640 1 --2- 172.21.15.39:0/930732673 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbd741027b0 0x7fbd7419e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:47.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.508+0000 7fbd7a800640 1 --2- 172.21.15.39:0/930732673 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbd740ffe40 0x7fbd7419ded0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:47.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.508+0000 7fbd7a800640 1 -- 172.21.15.39:0/930732673 >> 172.21.15.39:0/930732673 conn(0x7fbd740fba80 msgr2=0x7fbd74100af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:47.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.508+0000 7fbd7a800640 1 -- 172.21.15.39:0/930732673 shutdown_connections 2024-03-18T16:11:47.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.508+0000 7fbd7a800640 1 -- 172.21.15.39:0/930732673 wait complete. 2024-03-18T16:11:47.529 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-18T16:11:47.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.792+0000 7ff298600640 1 -- 172.21.15.39:0/3866403868 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff290104c00 msgr2=0x7ff290105080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:47.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.792+0000 7ff298600640 1 --2- 172.21.15.39:0/3866403868 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff290104c00 0x7ff290105080 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7ff27c009900 tx=0x7ff27c02bc60 comp rx=0 tx=0).stop 2024-03-18T16:11:47.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.792+0000 7ff298600640 1 -- 172.21.15.39:0/3866403868 shutdown_connections 2024-03-18T16:11:47.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.792+0000 7ff298600640 1 --2- 172.21.15.39:0/3866403868 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff2901055c0 0x7ff29010be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:47.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.792+0000 7ff298600640 1 --2- 172.21.15.39:0/3866403868 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff290104c00 0x7ff290105080 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:47.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.792+0000 7ff298600640 1 --2- 172.21.15.39:0/3866403868 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff290103a10 0x7ff290103e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:47.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.792+0000 7ff298600640 1 -- 172.21.15.39:0/3866403868 >> 172.21.15.39:0/3866403868 conn(0x7ff2900fefa0 msgr2=0x7ff2901013c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:47.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.792+0000 7ff298600640 1 -- 172.21.15.39:0/3866403868 shutdown_connections 2024-03-18T16:11:47.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.792+0000 7ff298600640 1 -- 172.21.15.39:0/3866403868 wait complete. 2024-03-18T16:11:47.795 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.793+0000 7ff298600640 1 Processor -- start 2024-03-18T16:11:47.795 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.794+0000 7ff298600640 1 -- start start 2024-03-18T16:11:47.796 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.794+0000 7ff298600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff290103a10 0x7ff290199cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:47.796 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.794+0000 7ff298600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff290104c00 0x7ff29019a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:47.796 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.794+0000 7ff298600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff2901055c0 0x7ff2901a1260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:47.796 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.794+0000 7ff298600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7ff29010f6c0 con 0x7ff290103a10 2024-03-18T16:11:47.796 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.794+0000 7ff298600640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ff29010f540 con 0x7ff2901055c0 2024-03-18T16:11:47.797 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.794+0000 7ff298600640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7ff29010f840 con 0x7ff290104c00 2024-03-18T16:11:47.797 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.794+0000 7ff295c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff290103a10 0x7ff290199cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:47.797 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.794+0000 7ff295200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff290104c00 0x7ff29019a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:47.797 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.794+0000 7ff295c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff290103a10 0x7ff290199cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43848/0 (socket says 172.21.15.39:43848) 2024-03-18T16:11:47.797 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.794+0000 7ff295c00640 1 -- 172.21.15.39:0/2468418109 learned_addr learned my addr 172.21.15.39:0/2468418109 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:11:47.797 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.795+0000 7ff296600640 1 --2- 172.21.15.39:0/2468418109 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff2901055c0 0x7ff2901a1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:47.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.795+0000 7ff295c00640 1 -- 172.21.15.39:0/2468418109 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff290104c00 msgr2=0x7ff29019a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:47.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.795+0000 7ff295c00640 1 --2- 172.21.15.39:0/2468418109 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff290104c00 0x7ff29019a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:47.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.795+0000 7ff295c00640 1 -- 172.21.15.39:0/2468418109 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff2901055c0 msgr2=0x7ff2901a1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:47.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.795+0000 7ff295c00640 1 --2- 172.21.15.39:0/2468418109 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff2901055c0 0x7ff2901a1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:47.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.795+0000 7ff295c00640 1 -- 172.21.15.39:0/2468418109 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2901a1900 con 0x7ff290103a10 2024-03-18T16:11:47.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.796+0000 7ff295c00640 1 --2- 172.21.15.39:0/2468418109 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff290103a10 0x7ff290199cf0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7ff28400e9c0 tx=0x7ff28400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:47.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.796+0000 7ff28e000640 1 -- 172.21.15.39:0/2468418109 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff28400cd80 con 0x7ff290103a10 2024-03-18T16:11:47.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.796+0000 7ff28e000640 1 -- 172.21.15.39:0/2468418109 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff2840035f0 con 0x7ff290103a10 2024-03-18T16:11:47.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.796+0000 7ff298600640 1 -- 172.21.15.39:0/2468418109 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2901a1bf0 con 0x7ff290103a10 2024-03-18T16:11:47.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.796+0000 7ff28e000640 1 -- 172.21.15.39:0/2468418109 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff284010690 con 0x7ff290103a10 2024-03-18T16:11:47.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.797+0000 7ff298600640 1 -- 172.21.15.39:0/2468418109 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2901a2140 con 0x7ff290103a10 2024-03-18T16:11:47.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.802+0000 7ff29b27c740 1 Processor -- start 2024-03-18T16:11:47.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.802+0000 7ff29b27c740 1 -- start start 2024-03-18T16:11:47.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.802+0000 7ff29b27c740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5575cd1448c0 0x5575cd144ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:47.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.802+0000 7ff29b27c740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5575cd1452b0 0x5575cd149db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:47.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.803+0000 7ff29b27c740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5575cd14a420 0x5575cd14c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:47.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.803+0000 7ff29b27c740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5575cd10c3b0 con 0x5575cd1448c0 2024-03-18T16:11:47.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.803+0000 7ff29b27c740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5575cd0a9450 con 0x5575cd1452b0 2024-03-18T16:11:47.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.803+0000 7ff29b27c740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x5575cd051cb0 con 0x5575cd14a420 2024-03-18T16:11:47.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.803+0000 7ff295200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5575cd14a420 0x5575cd14c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:47.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.803+0000 7ff295200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5575cd14a420 0x5575cd14c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:33960/0 (socket says 172.21.15.39:33960) 2024-03-18T16:11:47.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.803+0000 7ff295200640 1 -- 172.21.15.39:0/2650479914 learned_addr learned my addr 172.21.15.39:0/2650479914 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:11:47.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.803+0000 7ff296600640 1 --2- 172.21.15.39:0/2650479914 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5575cd1448c0 0x5575cd144ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:47.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.803+0000 7ff295200640 1 -- 172.21.15.39:0/2650479914 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5575cd1452b0 msgr2=0x5575cd149db0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:11:47.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.803+0000 7ff295200640 1 --2- 172.21.15.39:0/2650479914 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5575cd1452b0 0x5575cd149db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:47.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.803+0000 7ff295200640 1 -- 172.21.15.39:0/2650479914 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5575cd1448c0 msgr2=0x5575cd144ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:47.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.804+0000 7ff295200640 1 --2- 172.21.15.39:0/2650479914 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5575cd1448c0 0x5575cd144ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:47.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.804+0000 7ff295200640 1 -- 172.21.15.39:0/2650479914 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5575cd0731f0 con 0x5575cd14a420 2024-03-18T16:11:47.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.804+0000 7ff296600640 1 --2- 172.21.15.39:0/2650479914 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5575cd1448c0 0x5575cd144ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:11:47.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.804+0000 7ff295200640 1 --2- 172.21.15.39:0/2650479914 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5575cd14a420 0x5575cd14c8b0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7ff27c005ab0 tx=0x7ff27c0095f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=88c55f4e08d4ce62 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:47.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.805+0000 7ff282a00640 1 -- 172.21.15.39:0/2650479914 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff27c02cbe0 con 0x5575cd14a420 2024-03-18T16:11:47.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.805+0000 7ff282a00640 1 -- 172.21.15.39:0/2650479914 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff27c02cd80 con 0x5575cd14a420 2024-03-18T16:11:47.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.805+0000 7ff282a00640 1 -- 172.21.15.39:0/2650479914 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff27c004920 con 0x5575cd14a420 2024-03-18T16:11:47.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.806+0000 7ff29b27c740 1 -- 172.21.15.39:0/2650479914 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5575cd14a420 msgr2=0x5575cd14c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:47.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.806+0000 7ff29b27c740 1 --2- 172.21.15.39:0/2650479914 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5575cd14a420 0x5575cd14c8b0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7ff27c005ab0 tx=0x7ff27c0095f0 comp rx=0 tx=0).stop 2024-03-18T16:11:47.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.806+0000 7ff29b27c740 1 -- 172.21.15.39:0/2650479914 shutdown_connections 2024-03-18T16:11:47.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.806+0000 7ff29b27c740 1 --2- 172.21.15.39:0/2650479914 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5575cd1452b0 0x5575cd149db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:47.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.806+0000 7ff29b27c740 1 --2- 172.21.15.39:0/2650479914 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5575cd14a420 0x5575cd14c8b0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:47.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.806+0000 7ff29b27c740 1 --2- 172.21.15.39:0/2650479914 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5575cd1448c0 0x5575cd144ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:47.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.806+0000 7ff29b27c740 1 -- 172.21.15.39:0/2650479914 >> 172.21.15.39:0/2650479914 conn(0x5575cd13fe30 msgr2=0x5575cd140230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:47.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.815+0000 7ff28e000640 1 -- 172.21.15.39:0/2468418109 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7ff2840037a0 con 0x7ff290103a10 2024-03-18T16:11:47.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.816+0000 7ff29b27c740 1 -- 172.21.15.39:0/2650479914 shutdown_connections 2024-03-18T16:11:47.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.816+0000 7ff29b27c740 1 -- 172.21.15.39:0/2650479914 wait complete. 2024-03-18T16:11:47.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.816+0000 7ff28e000640 1 --2- 172.21.15.39:0/2468418109 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff2780a2890 0x7ff2780a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:47.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.816+0000 7ff29b27c740 1 -- start start 2024-03-18T16:11:47.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.816+0000 7ff28e000640 1 -- 172.21.15.39:0/2468418109 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7ff2840cf100 con 0x7ff290103a10 2024-03-18T16:11:47.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.817+0000 7ff29b27c740 1 Processor -- start 2024-03-18T16:11:47.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.817+0000 7ff29b27c740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5575cd1448c0 0x5575cd1572c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:47.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.817+0000 7ff29b27c740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5575cd14a420 0x5575cd157800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:47.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.817+0000 7ff29b27c740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5575cd15c830 0x5575cd15ec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:47.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.817+0000 7ff29b27c740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5575cd065110 con 0x5575cd14a420 2024-03-18T16:11:47.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.817+0000 7ff29b27c740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5575cd0a54f0 con 0x5575cd1448c0 2024-03-18T16:11:47.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.817+0000 7ff29b27c740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x5575cd14d370 con 0x5575cd15c830 2024-03-18T16:11:47.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.818+0000 7ff29b27c740 10 client.0 _ll_register_callbacks cb 0x5575cd052b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:11:47.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.818+0000 7ff296600640 1 --2- 172.21.15.39:0/2468418109 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff2780a2890 0x7ff2780a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:47.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.818+0000 7ff295200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5575cd14a420 0x5575cd157800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:47.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.818+0000 7ff295200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5575cd14a420 0x5575cd157800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:43864/0 (socket says 172.21.15.39:43864) 2024-03-18T16:11:47.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.818+0000 7ff295200640 1 -- 172.21.15.39:0/3291425403 learned_addr learned my addr 172.21.15.39:0/3291425403 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:11:47.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.818+0000 7ff295c00640 1 --2- 172.21.15.39:0/3291425403 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5575cd1448c0 0x5575cd1572c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:47.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.819+0000 7ff295200640 1 -- 172.21.15.39:0/3291425403 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5575cd15c830 msgr2=0x5575cd15ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:47.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.819+0000 7ff295200640 1 --2- 172.21.15.39:0/3291425403 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5575cd15c830 0x5575cd15ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:47.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.819+0000 7ff295200640 1 -- 172.21.15.39:0/3291425403 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5575cd1448c0 msgr2=0x5575cd1572c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:47.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.819+0000 7ff295200640 1 --2- 172.21.15.39:0/3291425403 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5575cd1448c0 0x5575cd1572c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:47.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.819+0000 7ff295200640 1 -- 172.21.15.39:0/3291425403 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5575ccdbd3f0 con 0x5575cd14a420 2024-03-18T16:11:47.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.820+0000 7ff296600640 1 --2- 172.21.15.39:0/2468418109 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff2780a2890 0x7ff2780a4d50 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x5575cd145b10 tx=0x7ff288011810 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:47.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.820+0000 7ff295200640 1 --2- 172.21.15.39:0/3291425403 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5575cd14a420 0x5575cd157800 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ff27c009850 tx=0x7ff27c034a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5ef482ae91c9b6c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:47.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.820+0000 7ff277400640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-03-18T16:11:47.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.820+0000 7ff277400640 1 -- 172.21.15.39:0/3291425403 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff27c046680 con 0x5575cd14a420 2024-03-18T16:11:47.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.821+0000 7ff29b27c740 1 -- 172.21.15.39:0/3291425403 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5575cd0b6990 con 0x5575cd14a420 2024-03-18T16:11:47.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.821+0000 7ff277400640 1 -- 172.21.15.39:0/3291425403 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7ff27c046820 con 0x5575cd14a420 2024-03-18T16:11:47.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.821+0000 7ff277400640 1 -- 172.21.15.39:0/3291425403 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7ff27c046a20 con 0x5575cd14a420 2024-03-18T16:11:47.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.821+0000 7ff29b27c740 10 client.15795 fetch_fsmap learned FSMap version 31 2024-03-18T16:11:47.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.821+0000 7ff29b27c740 1 -- 172.21.15.39:0/3291425403 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5575cce877d0 con 0x5575cd14a420 2024-03-18T16:11:47.825 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.822+0000 7ff277400640 1 -- 172.21.15.39:0/3291425403 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5575cd0b6990 con 0x5575cd14a420 2024-03-18T16:11:47.825 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.822+0000 7ff277400640 1 -- 172.21.15.39:0/3291425403 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 31) ==== 8374+0+0 (secure 0 0 0) 0x7ff27c04ea60 con 0x5575cd14a420 2024-03-18T16:11:47.825 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.822+0000 7ff29b27c740 10 client.15795 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T16:11:47.825 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.822+0000 7ff29b27c740 10 client.15795 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:11:47.825 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.822+0000 7ff29b27c740 20 client.15795 populate_metadata read hostname 'smithi039' 2024-03-18T16:11:47.825 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.822+0000 7ff29b27c740 1 --2- 172.21.15.39:0/3291425403 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5575cd1636d0 0x5575cd165af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:47.825 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.822+0000 7ff29b27c740 4 client.15795 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:11:47.825 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.822+0000 7ff29b27c740 1 -- 172.21.15.39:0/3291425403 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5575cd0b0ce0 con 0x5575cd1636d0 2024-03-18T16:11:47.825 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.823+0000 7ff295c00640 1 --2- 172.21.15.39:0/3291425403 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5575cd1636d0 0x5575cd165af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:47.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.823+0000 7ff295c00640 1 --2- 172.21.15.39:0/3291425403 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5575cd1636d0 0x5575cd165af0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7618c552d0244460 server_cookie=14a337657c88515 in_seq=0 out_seq=0 2024-03-18T16:11:47.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.823+0000 7ff277400640 10 client.15795 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:11:47.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.824+0000 7ff277400640 1 -- 172.21.15.39:0/3291425403 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5575cd0b0ce0 con 0x5575cd1636d0 2024-03-18T16:11:47.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.824+0000 7ff277400640 10 client.15795 handle_command_reply: tid=0 2024-03-18T16:11:47.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.824+0000 7ff29b27c740 1 client.15795 shutdown 2024-03-18T16:11:47.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.825+0000 7ff29b27c740 1 -- 172.21.15.39:0/3291425403 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5575cd14a420 msgr2=0x5575cd157800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:47.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.825+0000 7ff29b27c740 1 --2- 172.21.15.39:0/3291425403 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5575cd14a420 0x5575cd157800 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7ff27c009850 tx=0x7ff27c034a60 comp rx=0 tx=0).stop 2024-03-18T16:11:47.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.825+0000 7ff29b27c740 1 -- 172.21.15.39:0/3291425403 shutdown_connections 2024-03-18T16:11:47.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.825+0000 7ff29b27c740 1 --2- 172.21.15.39:0/3291425403 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5575cd1636d0 0x5575cd165af0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:47.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.825+0000 7ff29b27c740 1 --2- 172.21.15.39:0/3291425403 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5575cd15c830 0x5575cd15ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:47.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.825+0000 7ff29b27c740 1 --2- 172.21.15.39:0/3291425403 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5575cd14a420 0x5575cd157800 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:47.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.825+0000 7ff29b27c740 1 --2- 172.21.15.39:0/3291425403 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5575cd1448c0 0x5575cd1572c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:47.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.825+0000 7ff29b27c740 1 -- 172.21.15.39:0/3291425403 >> 172.21.15.39:0/3291425403 conn(0x5575cd13fe30 msgr2=0x5575cd13f1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:47.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.825+0000 7ff277400640 0 client.15795 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:11:47.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.825+0000 7ff29b27c740 1 -- 172.21.15.39:0/3291425403 shutdown_connections 2024-03-18T16:11:47.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.825+0000 7ff29b27c740 1 -- 172.21.15.39:0/3291425403 wait complete. 2024-03-18T16:11:47.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.825+0000 7ff29b27c740 20 client.15795 trim_cache size 0 max 16384 2024-03-18T16:11:47.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.854+0000 7ff29b27c740 1 Processor -- start 2024-03-18T16:11:47.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.854+0000 7ff29b27c740 1 -- start start 2024-03-18T16:11:47.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.855+0000 7ff29b27c740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5575cd1448c0 0x5575cd13b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:47.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.855+0000 7ff29b27c740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5575cd14a420 0x5575cd143090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:47.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.855+0000 7ff29b27c740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5575cd15c830 0x5575cd1435d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:47.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.855+0000 7ff295200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5575cd14a420 0x5575cd143090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:47.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.855+0000 7ff295200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5575cd14a420 0x5575cd143090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:49518/0 (socket says 172.21.15.39:49518) 2024-03-18T16:11:47.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.855+0000 7ff295200640 1 -- 172.21.15.39:0/53787880 learned_addr learned my addr 172.21.15.39:0/53787880 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:11:47.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.855+0000 7ff295c00640 1 --2- 172.21.15.39:0/53787880 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5575cd1448c0 0x5575cd13b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:47.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.856+0000 7ff296600640 1 --2- 172.21.15.39:0/53787880 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5575cd15c830 0x5575cd1435d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:47.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.856+0000 7ff29b27c740 1 -- 172.21.15.39:0/53787880 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7ff2901a8200 con 0x5575cd1448c0 2024-03-18T16:11:47.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.856+0000 7ff29b27c740 1 -- 172.21.15.39:0/53787880 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ff2901a9dd0 con 0x5575cd14a420 2024-03-18T16:11:47.859 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.856+0000 7ff29b27c740 1 -- 172.21.15.39:0/53787880 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7ff2901a88d0 con 0x5575cd15c830 2024-03-18T16:11:47.859 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.856+0000 7ff295200640 1 -- 172.21.15.39:0/53787880 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5575cd15c830 msgr2=0x5575cd1435d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:47.859 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.856+0000 7ff295200640 1 --2- 172.21.15.39:0/53787880 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5575cd15c830 0x5575cd1435d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:47.859 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.856+0000 7ff295200640 1 -- 172.21.15.39:0/53787880 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5575cd1448c0 msgr2=0x5575cd13b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:47.859 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.856+0000 7ff295200640 1 --2- 172.21.15.39:0/53787880 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5575cd1448c0 0x5575cd13b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:47.859 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.856+0000 7ff295200640 1 -- 172.21.15.39:0/53787880 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5575cd13bc10 con 0x5575cd14a420 2024-03-18T16:11:47.859 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.857+0000 7ff295200640 1 --2- 172.21.15.39:0/53787880 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5575cd14a420 0x5575cd143090 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7ff27c009900 tx=0x7ff27c000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=476e0928747faf13 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:47.859 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.857+0000 7ff282000640 1 -- 172.21.15.39:0/53787880 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff27c046a20 con 0x5575cd14a420 2024-03-18T16:11:47.860 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.857+0000 7ff282000640 1 -- 172.21.15.39:0/53787880 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff27c046480 con 0x5575cd14a420 2024-03-18T16:11:47.860 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.857+0000 7ff282000640 1 -- 172.21.15.39:0/53787880 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff27c0055b0 con 0x5575cd14a420 2024-03-18T16:11:47.860 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.858+0000 7ff29b27c740 1 -- 172.21.15.39:0/53787880 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5575cd14a420 msgr2=0x5575cd143090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:47.860 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.858+0000 7ff29b27c740 1 --2- 172.21.15.39:0/53787880 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5575cd14a420 0x5575cd143090 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7ff27c009900 tx=0x7ff27c000bc0 comp rx=0 tx=0).stop 2024-03-18T16:11:47.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.859+0000 7ff29b27c740 1 -- 172.21.15.39:0/53787880 shutdown_connections 2024-03-18T16:11:47.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.859+0000 7ff29b27c740 1 --2- 172.21.15.39:0/53787880 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5575cd14a420 0x5575cd143090 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:47.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.859+0000 7ff29b27c740 1 --2- 172.21.15.39:0/53787880 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5575cd15c830 0x5575cd1435d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:47.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.859+0000 7ff29b27c740 1 --2- 172.21.15.39:0/53787880 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5575cd1448c0 0x5575cd13b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:47.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.859+0000 7ff29b27c740 1 -- 172.21.15.39:0/53787880 >> 172.21.15.39:0/53787880 conn(0x5575cd13fe30 msgr2=0x5575cd140210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:47.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.863+0000 7ff29b27c740 1 -- 172.21.15.39:0/53787880 shutdown_connections 2024-03-18T16:11:47.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.863+0000 7ff29b27c740 1 -- 172.21.15.39:0/53787880 wait complete. 2024-03-18T16:11:47.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.864+0000 7ff29b27c740 1 -- start start 2024-03-18T16:11:47.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.864+0000 7ff29b27c740 1 Processor -- start 2024-03-18T16:11:47.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.865+0000 7ff29b27c740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5575cd1448c0 0x5575cd1741f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:47.867 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.865+0000 7ff29b27c740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5575cd14a420 0x5575cd174730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:47.867 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.865+0000 7ff29b27c740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5575cd15c830 0x5575cd175740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:47.867 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.865+0000 7ff29b27c740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7ff2901a96a0 con 0x5575cd14a420 2024-03-18T16:11:47.867 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.865+0000 7ff29b27c740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ff2901a8fa0 con 0x5575cd15c830 2024-03-18T16:11:47.867 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.865+0000 7ff29b27c740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7ff2901a6180 con 0x5575cd1448c0 2024-03-18T16:11:47.867 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.865+0000 7ff29b27c740 10 client.0 _ll_register_callbacks cb 0x7ff27c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:11:47.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.868+0000 7ff296600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5575cd15c830 0x5575cd175740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:47.870 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.868+0000 7ff296600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5575cd15c830 0x5575cd175740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:49520/0 (socket says 172.21.15.39:49520) 2024-03-18T16:11:47.870 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.868+0000 7ff296600640 1 -- 172.21.15.39:0/3090126174 learned_addr learned my addr 172.21.15.39:0/3090126174 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:11:47.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.869+0000 7ff295200640 1 --2- 172.21.15.39:0/3090126174 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5575cd14a420 0x5575cd174730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:47.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.870+0000 7ff295c00640 1 --2- 172.21.15.39:0/3090126174 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5575cd1448c0 0x5575cd1741f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:47.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.870+0000 7ff295c00640 1 -- 172.21.15.39:0/3090126174 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5575cd15c830 msgr2=0x5575cd175740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:47.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.870+0000 7ff295c00640 1 --2- 172.21.15.39:0/3090126174 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5575cd15c830 0x5575cd175740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:47.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.870+0000 7ff295c00640 1 -- 172.21.15.39:0/3090126174 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5575cd14a420 msgr2=0x5575cd174730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:47.877 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.875+0000 7ff295c00640 1 --2- 172.21.15.39:0/3090126174 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5575cd14a420 0x5575cd174730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:47.877 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.875+0000 7ff295c00640 1 -- 172.21.15.39:0/3090126174 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5575cd175d30 con 0x5575cd1448c0 2024-03-18T16:11:47.877 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.875+0000 7ff295200640 1 --2- 172.21.15.39:0/3090126174 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5575cd14a420 0x5575cd174730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:11:47.877 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.875+0000 7ff295c00640 1 --2- 172.21.15.39:0/3090126174 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5575cd1448c0 0x5575cd1741f0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x5575cd145d20 tx=0x7ff28409e040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8e17db134802ab6f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:47.877 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.876+0000 7ff277400640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:11:47.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.878+0000 7ff277400640 1 -- 172.21.15.39:0/3090126174 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff28409f860 con 0x5575cd1448c0 2024-03-18T16:11:47.880 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.878+0000 7ff277400640 1 -- 172.21.15.39:0/3090126174 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7ff284097cf0 con 0x5575cd1448c0 2024-03-18T16:11:47.880 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.878+0000 7ff29b27c740 1 -- 172.21.15.39:0/3090126174 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff27c046480 con 0x5575cd1448c0 2024-03-18T16:11:47.880 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.879+0000 7ff277400640 1 -- 172.21.15.39:0/3090126174 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7ff2840a3eb0 con 0x5575cd1448c0 2024-03-18T16:11:47.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.879+0000 7ff29b27c740 10 client.26002 fetch_fsmap learned FSMap version 31 2024-03-18T16:11:47.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.879+0000 7ff29b27c740 1 -- 172.21.15.39:0/3090126174 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x5575cd1c9390 con 0x5575cd1448c0 2024-03-18T16:11:47.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.880+0000 7ff277400640 1 -- 172.21.15.39:0/3090126174 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff27c046480 con 0x5575cd1448c0 2024-03-18T16:11:47.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.882+0000 7ff277400640 1 -- 172.21.15.39:0/3090126174 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 31) ==== 8374+0+0 (secure 0 0 0) 0x7ff2840974f0 con 0x5575cd1448c0 2024-03-18T16:11:47.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.882+0000 7ff29b27c740 10 client.26002 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T16:11:47.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.882+0000 7ff29b27c740 10 client.26002 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:11:47.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.882+0000 7ff29b27c740 20 client.26002 populate_metadata read hostname 'smithi039' 2024-03-18T16:11:47.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.882+0000 7ff29b27c740 1 --2- 172.21.15.39:0/3090126174 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5575cd1c9d70 0x5575cd1cc190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:47.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.882+0000 7ff29b27c740 4 client.26002 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:11:47.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.882+0000 7ff29b27c740 1 -- 172.21.15.39:0/3090126174 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5575cd02e560 con 0x5575cd1c9d70 2024-03-18T16:11:47.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.885+0000 7ff295200640 1 --2- 172.21.15.39:0/3090126174 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5575cd1c9d70 0x5575cd1cc190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:47.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.886+0000 7ff295200640 1 --2- 172.21.15.39:0/3090126174 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5575cd1c9d70 0x5575cd1cc190 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2dc8d9ab83726056 server_cookie=c6d6433e9e5bc803 in_seq=0 out_seq=0 2024-03-18T16:11:47.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.886+0000 7ff277400640 10 client.26002 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:11:47.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.886+0000 7ff277400640 1 -- 172.21.15.39:0/3090126174 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5575cd02e560 con 0x5575cd1c9d70 2024-03-18T16:11:47.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.886+0000 7ff277400640 10 client.26002 handle_command_reply: tid=0 2024-03-18T16:11:47.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.887+0000 7ff29b27c740 1 client.26002 shutdown 2024-03-18T16:11:47.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.887+0000 7ff29b27c740 1 -- 172.21.15.39:0/3090126174 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5575cd1448c0 msgr2=0x5575cd1741f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:47.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.887+0000 7ff29b27c740 1 --2- 172.21.15.39:0/3090126174 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5575cd1448c0 0x5575cd1741f0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x5575cd145d20 tx=0x7ff28409e040 comp rx=0 tx=0).stop 2024-03-18T16:11:47.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.888+0000 7ff29b27c740 1 -- 172.21.15.39:0/3090126174 shutdown_connections 2024-03-18T16:11:47.890 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.888+0000 7ff29b27c740 1 --2- 172.21.15.39:0/3090126174 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5575cd1c9d70 0x5575cd1cc190 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:47.890 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.888+0000 7ff29b27c740 1 --2- 172.21.15.39:0/3090126174 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5575cd15c830 0x5575cd175740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:47.890 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.888+0000 7ff29b27c740 1 --2- 172.21.15.39:0/3090126174 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5575cd14a420 0x5575cd174730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:47.890 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.888+0000 7ff29b27c740 1 --2- 172.21.15.39:0/3090126174 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5575cd1448c0 0x5575cd1741f0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:47.890 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.888+0000 7ff29b27c740 1 -- 172.21.15.39:0/3090126174 >> 172.21.15.39:0/3090126174 conn(0x5575cd13fe30 msgr2=0x5575cd140210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:47.890 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.888+0000 7ff277400640 0 client.26002 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:11:47.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.889+0000 7ff29b27c740 1 -- 172.21.15.39:0/3090126174 shutdown_connections 2024-03-18T16:11:47.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.890+0000 7ff29b27c740 1 -- 172.21.15.39:0/3090126174 wait complete. 2024-03-18T16:11:47.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.890+0000 7ff29b27c740 20 client.26002 trim_cache size 0 max 16384 2024-03-18T16:11:47.892 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:11:47.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.891+0000 7ff298600640 1 -- 172.21.15.39:0/2468418109 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff2780a2890 msgr2=0x7ff2780a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:47.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.891+0000 7ff298600640 1 --2- 172.21.15.39:0/2468418109 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff2780a2890 0x7ff2780a4d50 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x5575cd145b10 tx=0x7ff288011810 comp rx=0 tx=0).stop 2024-03-18T16:11:47.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.891+0000 7ff298600640 1 -- 172.21.15.39:0/2468418109 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff290103a10 msgr2=0x7ff290199cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:47.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.891+0000 7ff298600640 1 --2- 172.21.15.39:0/2468418109 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff290103a10 0x7ff290199cf0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7ff28400e9c0 tx=0x7ff28400ee80 comp rx=0 tx=0).stop 2024-03-18T16:11:47.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.891+0000 7ff298600640 1 -- 172.21.15.39:0/2468418109 shutdown_connections 2024-03-18T16:11:47.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.891+0000 7ff298600640 1 --2- 172.21.15.39:0/2468418109 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff2901055c0 0x7ff2901a1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:47.894 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.891+0000 7ff298600640 1 --2- 172.21.15.39:0/2468418109 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff2780a2890 0x7ff2780a4d50 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:47.894 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.891+0000 7ff298600640 1 --2- 172.21.15.39:0/2468418109 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff290104c00 0x7ff29019a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:47.894 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.891+0000 7ff298600640 1 --2- 172.21.15.39:0/2468418109 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff290103a10 0x7ff290199cf0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:47.894 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.891+0000 7ff298600640 1 -- 172.21.15.39:0/2468418109 >> 172.21.15.39:0/2468418109 conn(0x7ff2900fefa0 msgr2=0x7ff290107f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:47.894 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.892+0000 7ff298600640 1 -- 172.21.15.39:0/2468418109 shutdown_connections 2024-03-18T16:11:47.894 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:47.892+0000 7ff298600640 1 -- 172.21.15.39:0/2468418109 wait complete. 2024-03-18T16:11:48.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:11:48 smithi039 ceph-mon[31721]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T16:11:48.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:11:48 smithi196 ceph-mon[36307]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T16:11:48.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:11:48 smithi194 ceph-mon[36200]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T16:11:48.922 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-18T16:11:48.922 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T16:11:48.922 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-18T16:11:49.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.268+0000 7f3dc0c00640 1 -- 172.21.15.39:0/2082614332 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3dbc078f20 msgr2=0x7f3dbc07b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:49.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.268+0000 7f3dc0c00640 1 --2- 172.21.15.39:0/2082614332 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3dbc078f20 0x7f3dbc07b3b0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f3db4009ca0 tx=0x7f3db402c550 comp rx=0 tx=0).stop 2024-03-18T16:11:49.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.269+0000 7f3dc0c00640 1 -- 172.21.15.39:0/2082614332 shutdown_connections 2024-03-18T16:11:49.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.269+0000 7f3dc0c00640 1 --2- 172.21.15.39:0/2082614332 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3dbc078f20 0x7f3dbc07b3b0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:49.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.269+0000 7f3dc0c00640 1 --2- 172.21.15.39:0/2082614332 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3dbc076570 0x7f3dbc0789e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:49.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.269+0000 7f3dc0c00640 1 --2- 172.21.15.39:0/2082614332 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3dbc073690 0x7f3dbc073ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:49.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.269+0000 7f3dc0c00640 1 -- 172.21.15.39:0/2082614332 >> 172.21.15.39:0/2082614332 conn(0x7f3dbc06e640 msgr2=0x7f3dbc070a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:49.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.269+0000 7f3dc0c00640 1 -- 172.21.15.39:0/2082614332 shutdown_connections 2024-03-18T16:11:49.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.269+0000 7f3dc0c00640 1 -- 172.21.15.39:0/2082614332 wait complete. 2024-03-18T16:11:49.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.270+0000 7f3dc0c00640 1 Processor -- start 2024-03-18T16:11:49.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.272+0000 7f3dc0c00640 1 -- start start 2024-03-18T16:11:49.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.273+0000 7f3dc0c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3dbc073690 0x7f3dbc1af720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:49.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.273+0000 7f3dc0c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3dbc076570 0x7f3dbc1afc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:49.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.273+0000 7f3dc0c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3dbc078f20 0x7f3dbc1b6670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:49.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.273+0000 7f3dc0c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f3dbc07ed60 con 0x7f3dbc073690 2024-03-18T16:11:49.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.273+0000 7f3dc0c00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f3dbc07ebe0 con 0x7f3dbc076570 2024-03-18T16:11:49.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.273+0000 7f3dc0c00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f3dbc07eee0 con 0x7f3dbc078f20 2024-03-18T16:11:49.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.273+0000 7f3dba800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3dbc078f20 0x7f3dbc1b6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:49.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.273+0000 7f3dba800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3dbc078f20 0x7f3dbc1b6670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:60106/0 (socket says 172.21.15.39:60106) 2024-03-18T16:11:49.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.273+0000 7f3dba800640 1 -- 172.21.15.39:0/106550930 learned_addr learned my addr 172.21.15.39:0/106550930 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:11:49.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.274+0000 7f3dba800640 1 -- 172.21.15.39:0/106550930 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3dbc076570 msgr2=0x7f3dbc1afc60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T16:11:49.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.274+0000 7f3dba800640 1 --2- 172.21.15.39:0/106550930 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3dbc076570 0x7f3dbc1afc60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:49.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.274+0000 7f3dba800640 1 -- 172.21.15.39:0/106550930 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3dbc073690 msgr2=0x7f3dbc1af720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T16:11:49.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.274+0000 7f3dba800640 1 --2- 172.21.15.39:0/106550930 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3dbc073690 0x7f3dbc1af720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:49.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.274+0000 7f3dba800640 1 -- 172.21.15.39:0/106550930 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dbc1b6da0 con 0x7f3dbc078f20 2024-03-18T16:11:49.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.275+0000 7f3dba800640 1 --2- 172.21.15.39:0/106550930 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3dbc078f20 0x7f3dbc1b6670 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f3db4009bf0 tx=0x7f3db4033930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:49.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.275+0000 7f3db2a00640 1 -- 172.21.15.39:0/106550930 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3db4003d80 con 0x7f3dbc078f20 2024-03-18T16:11:49.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.275+0000 7f3db2a00640 1 -- 172.21.15.39:0/106550930 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3db40319f0 con 0x7f3dbc078f20 2024-03-18T16:11:49.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.275+0000 7f3db2a00640 1 -- 172.21.15.39:0/106550930 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3db403e420 con 0x7f3dbc078f20 2024-03-18T16:11:49.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.276+0000 7f3dc0c00640 1 -- 172.21.15.39:0/106550930 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3dbc1b6f70 con 0x7f3dbc078f20 2024-03-18T16:11:49.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.276+0000 7f3dc0c00640 1 -- 172.21.15.39:0/106550930 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3dbc1b73b0 con 0x7f3dbc078f20 2024-03-18T16:11:49.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.283+0000 7f3db2a00640 1 -- 172.21.15.39:0/106550930 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f3db403b040 con 0x7f3dbc078f20 2024-03-18T16:11:49.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.289+0000 7f3db2a00640 1 --2- 172.21.15.39:0/106550930 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3da00c48e0 0x7f3da00c6da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:49.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.289+0000 7f3db2a00640 1 -- 172.21.15.39:0/106550930 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f3db40f2be0 con 0x7f3dbc078f20 2024-03-18T16:11:49.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.289+0000 7f3db9e00640 1 --2- 172.21.15.39:0/106550930 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3da00c48e0 0x7f3da00c6da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:49.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.291+0000 7f3db9e00640 1 --2- 172.21.15.39:0/106550930 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3da00c48e0 0x7f3da00c6da0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f3dac009a60 tx=0x7f3dac005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:49.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.292+0000 7f3dc39da740 1 Processor -- start 2024-03-18T16:11:49.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.295+0000 7f3dc39da740 1 -- start start 2024-03-18T16:11:49.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.296+0000 7f3dc39da740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558573ac47e0 0x558573ac4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:49.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.296+0000 7f3dc39da740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558573ac51d0 0x558573ac9cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:49.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.296+0000 7f3dc39da740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558573aca340 0x558573acc7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:49.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.296+0000 7f3dc39da740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x558573a27f80 con 0x558573ac51d0 2024-03-18T16:11:49.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.296+0000 7f3dc39da740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5585739d0ad0 con 0x558573aca340 2024-03-18T16:11:49.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.296+0000 7f3dc39da740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x558573a24020 con 0x558573ac47e0 2024-03-18T16:11:49.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.296+0000 7f3db9e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558573ac47e0 0x558573ac4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:49.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.297+0000 7f3db9e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558573ac47e0 0x558573ac4c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:60116/0 (socket says 172.21.15.39:60116) 2024-03-18T16:11:49.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.297+0000 7f3db9e00640 1 -- 172.21.15.39:0/2925647146 learned_addr learned my addr 172.21.15.39:0/2925647146 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:11:49.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.297+0000 7f3dba800640 1 --2- 172.21.15.39:0/2925647146 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558573aca340 0x558573acc7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:49.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.297+0000 7f3db9400640 1 --2- 172.21.15.39:0/2925647146 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558573ac51d0 0x558573ac9cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:49.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.298+0000 7f3db9e00640 1 -- 172.21.15.39:0/2925647146 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558573aca340 msgr2=0x558573acc7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:49.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.298+0000 7f3db9e00640 1 --2- 172.21.15.39:0/2925647146 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558573aca340 0x558573acc7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:49.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.298+0000 7f3db9e00640 1 -- 172.21.15.39:0/2925647146 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558573ac51d0 msgr2=0x558573ac9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:49.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.298+0000 7f3db9e00640 1 --2- 172.21.15.39:0/2925647146 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558573ac51d0 0x558573ac9cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:49.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.298+0000 7f3db9e00640 1 -- 172.21.15.39:0/2925647146 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5585739e86f0 con 0x558573ac47e0 2024-03-18T16:11:49.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.298+0000 7f3db9400640 1 --2- 172.21.15.39:0/2925647146 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558573ac51d0 0x558573ac9cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:11:49.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.298+0000 7f3db9e00640 1 --2- 172.21.15.39:0/2925647146 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558573ac47e0 0x558573ac4c00 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f3dac00ed60 tx=0x7f3dac011000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=17387ce59e3de44f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:49.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.299+0000 7f3da7000640 1 -- 172.21.15.39:0/2925647146 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dac020070 con 0x558573ac47e0 2024-03-18T16:11:49.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.299+0000 7f3da7000640 1 -- 172.21.15.39:0/2925647146 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3dac010e40 con 0x558573ac47e0 2024-03-18T16:11:49.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.299+0000 7f3da7000640 1 -- 172.21.15.39:0/2925647146 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dac020070 con 0x558573ac47e0 2024-03-18T16:11:49.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.300+0000 7f3dc39da740 1 -- 172.21.15.39:0/2925647146 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558573ac47e0 msgr2=0x558573ac4c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:49.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.300+0000 7f3dc39da740 1 --2- 172.21.15.39:0/2925647146 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558573ac47e0 0x558573ac4c00 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f3dac00ed60 tx=0x7f3dac011000 comp rx=0 tx=0).stop 2024-03-18T16:11:49.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.300+0000 7f3dc39da740 1 -- 172.21.15.39:0/2925647146 shutdown_connections 2024-03-18T16:11:49.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.300+0000 7f3dc39da740 1 --2- 172.21.15.39:0/2925647146 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558573aca340 0x558573acc7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:49.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.300+0000 7f3dc39da740 1 --2- 172.21.15.39:0/2925647146 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558573ac51d0 0x558573ac9cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:49.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.301+0000 7f3dc39da740 1 --2- 172.21.15.39:0/2925647146 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558573ac47e0 0x558573ac4c00 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:49.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.301+0000 7f3dc39da740 1 -- 172.21.15.39:0/2925647146 >> 172.21.15.39:0/2925647146 conn(0x558573a8c930 msgr2=0x558573ac0150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:49.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.301+0000 7f3dc39da740 1 -- 172.21.15.39:0/2925647146 shutdown_connections 2024-03-18T16:11:49.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.301+0000 7f3dc39da740 1 -- 172.21.15.39:0/2925647146 wait complete. 2024-03-18T16:11:49.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.302+0000 7f3dc39da740 1 -- start start 2024-03-18T16:11:49.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.303+0000 7f3dc39da740 1 Processor -- start 2024-03-18T16:11:49.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.306+0000 7f3dc39da740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558573ac47e0 0x558573ace940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:49.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.306+0000 7f3dc39da740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558573ac51d0 0x558573acee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:49.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.307+0000 7f3dc39da740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558573aca340 0x558573ad5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:49.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.307+0000 7f3dc39da740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x558573acd2f0 con 0x558573ac47e0 2024-03-18T16:11:49.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.307+0000 7f3dc39da740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5585739e3f30 con 0x558573aca340 2024-03-18T16:11:49.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.307+0000 7f3dc39da740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x558573acd470 con 0x558573ac51d0 2024-03-18T16:11:49.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.307+0000 7f3dc39da740 10 client.0 _ll_register_callbacks cb 0x5585739d1bc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:11:49.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.308+0000 7f3db9400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558573ac51d0 0x558573acee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:49.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.308+0000 7f3db9400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558573ac51d0 0x558573acee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:60124/0 (socket says 172.21.15.39:60124) 2024-03-18T16:11:49.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.308+0000 7f3db9400640 1 -- 172.21.15.39:0/3873705966 learned_addr learned my addr 172.21.15.39:0/3873705966 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:11:49.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.308+0000 7f3dba800640 1 --2- 172.21.15.39:0/3873705966 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558573aca340 0x558573ad5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:49.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.309+0000 7f3db9e00640 1 --2- 172.21.15.39:0/3873705966 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558573ac47e0 0x558573ace940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:49.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.309+0000 7f3dba800640 1 -- 172.21.15.39:0/3873705966 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558573ac51d0 msgr2=0x558573acee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:49.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.309+0000 7f3dba800640 1 --2- 172.21.15.39:0/3873705966 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558573ac51d0 0x558573acee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:49.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.309+0000 7f3dba800640 1 -- 172.21.15.39:0/3873705966 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558573ac47e0 msgr2=0x558573ace940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:49.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.309+0000 7f3dba800640 1 --2- 172.21.15.39:0/3873705966 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558573ac47e0 0x558573ace940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:49.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.309+0000 7f3dba800640 1 -- 172.21.15.39:0/3873705966 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5585738047e0 con 0x558573aca340 2024-03-18T16:11:49.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.309+0000 7f3db9e00640 1 --2- 172.21.15.39:0/3873705966 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558573ac47e0 0x558573ace940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:11:49.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.310+0000 7f3dba800640 1 --2- 172.21.15.39:0/3873705966 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558573aca340 0x558573ad5ec0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f3db40f0350 tx=0x7f3db40beb50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb1b1740f2becf7f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:49.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.310+0000 7f3d9b400640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:11:49.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.310+0000 7f3d9b400640 1 -- 172.21.15.39:0/3873705966 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3db40bf1f0 con 0x558573aca340 2024-03-18T16:11:49.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.314+0000 7f3d9b400640 1 -- 172.21.15.39:0/3873705966 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f3db40bb860 con 0x558573aca340 2024-03-18T16:11:49.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.314+0000 7f3dc39da740 1 -- 172.21.15.39:0/3873705966 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x558573a354c0 con 0x558573aca340 2024-03-18T16:11:49.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.315+0000 7f3d9b400640 1 -- 172.21.15.39:0/3873705966 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f3db40c3340 con 0x558573aca340 2024-03-18T16:11:49.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.315+0000 7f3dc39da740 10 client.25997 fetch_fsmap learned FSMap version 31 2024-03-18T16:11:49.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.315+0000 7f3dc39da740 1 -- 172.21.15.39:0/3873705966 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x558573864f70 con 0x558573aca340 2024-03-18T16:11:49.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.316+0000 7f3d9b400640 1 -- 172.21.15.39:0/3873705966 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x558573a354c0 con 0x558573aca340 2024-03-18T16:11:49.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.319+0000 7f3d9b400640 1 -- 172.21.15.39:0/3873705966 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 31) ==== 8374+0+0 (secure 0 0 0) 0x7f3db40c7500 con 0x558573aca340 2024-03-18T16:11:49.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.319+0000 7f3dc39da740 10 client.25997 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T16:11:49.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.319+0000 7f3dc39da740 10 client.25997 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:11:49.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.319+0000 7f3dc39da740 20 client.25997 populate_metadata read hostname 'smithi039' 2024-03-18T16:11:49.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.319+0000 7f3dc39da740 1 --2- 172.21.15.39:0/3873705966 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x558573ada750 0x558573adcbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:49.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.319+0000 7f3dc39da740 4 client.25997 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:11:49.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.319+0000 7f3dc39da740 1 -- 172.21.15.39:0/3873705966 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558573a2f810 con 0x558573ada750 2024-03-18T16:11:49.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.320+0000 7f3db9e00640 1 --2- 172.21.15.39:0/3873705966 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x558573ada750 0x558573adcbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:49.326 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.321+0000 7f3db9e00640 1 --2- 172.21.15.39:0/3873705966 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x558573ada750 0x558573adcbb0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f85cde08d07a9a7f server_cookie=3c4ee37ecd3880df in_seq=0 out_seq=0 2024-03-18T16:11:49.326 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.321+0000 7f3d9b400640 10 client.25997 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:11:49.326 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.322+0000 7f3d9b400640 1 -- 172.21.15.39:0/3873705966 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558573a2f810 con 0x558573ada750 2024-03-18T16:11:49.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.322+0000 7f3d9b400640 10 client.25997 handle_command_reply: tid=0 2024-03-18T16:11:49.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.322+0000 7f3dc39da740 1 client.25997 shutdown 2024-03-18T16:11:49.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.325+0000 7f3dc39da740 1 -- 172.21.15.39:0/3873705966 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558573aca340 msgr2=0x558573ad5ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:49.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.325+0000 7f3dc39da740 1 --2- 172.21.15.39:0/3873705966 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558573aca340 0x558573ad5ec0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f3db40f0350 tx=0x7f3db40beb50 comp rx=0 tx=0).stop 2024-03-18T16:11:49.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.325+0000 7f3dc39da740 1 -- 172.21.15.39:0/3873705966 shutdown_connections 2024-03-18T16:11:49.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.325+0000 7f3dc39da740 1 --2- 172.21.15.39:0/3873705966 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x558573ada750 0x558573adcbb0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:49.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.325+0000 7f3dc39da740 1 --2- 172.21.15.39:0/3873705966 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558573aca340 0x558573ad5ec0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:49.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.325+0000 7f3dc39da740 1 --2- 172.21.15.39:0/3873705966 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558573ac51d0 0x558573acee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:49.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.325+0000 7f3d9b400640 0 client.25997 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:11:49.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.325+0000 7f3dc39da740 1 --2- 172.21.15.39:0/3873705966 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558573ac47e0 0x558573ace940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:49.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.325+0000 7f3dc39da740 1 -- 172.21.15.39:0/3873705966 >> 172.21.15.39:0/3873705966 conn(0x558573a8c930 msgr2=0x558573abf350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:49.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.329+0000 7f3dc39da740 1 -- 172.21.15.39:0/3873705966 shutdown_connections 2024-03-18T16:11:49.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.329+0000 7f3dc39da740 1 -- 172.21.15.39:0/3873705966 wait complete. 2024-03-18T16:11:49.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.331+0000 7f3dc39da740 20 client.25997 trim_cache size 0 max 16384 2024-03-18T16:11:49.340 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:11:49 smithi039 ceph-mon[31721]: pgmap v474: 129 pgs: 129 active+clean; 2.9 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 227 MiB/s wr, 100 op/s 2024-03-18T16:11:49.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.370+0000 7f3dc39da740 1 Processor -- start 2024-03-18T16:11:49.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.370+0000 7f3dc39da740 1 -- start start 2024-03-18T16:11:49.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.371+0000 7f3dc39da740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558573ac47e0 0x558573ac0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:49.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.371+0000 7f3dc39da740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558573ac51d0 0x558573ad5e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:49.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.371+0000 7f3dc39da740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558573aca340 0x558573abeac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:49.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.371+0000 7f3dc39da740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f3dbc1c1b20 con 0x558573aca340 2024-03-18T16:11:49.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.371+0000 7f3dc39da740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f3dbc1c36f0 con 0x558573ac47e0 2024-03-18T16:11:49.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.371+0000 7f3dc39da740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f3dbc1c21f0 con 0x558573ac51d0 2024-03-18T16:11:49.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.371+0000 7f3db9400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558573ac51d0 0x558573ad5e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:49.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.371+0000 7f3db9e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558573ac47e0 0x558573ac0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:49.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.371+0000 7f3db9400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558573ac51d0 0x558573ad5e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:60138/0 (socket says 172.21.15.39:60138) 2024-03-18T16:11:49.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.371+0000 7f3db9400640 1 -- 172.21.15.39:0/1086264105 learned_addr learned my addr 172.21.15.39:0/1086264105 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:11:49.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.371+0000 7f3db9e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558573ac47e0 0x558573ac0380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:38304/0 (socket says 172.21.15.39:38304) 2024-03-18T16:11:49.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.371+0000 7f3dba800640 1 --2- 172.21.15.39:0/1086264105 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558573aca340 0x558573abeac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:49.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.372+0000 7f3db9400640 1 -- 172.21.15.39:0/1086264105 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558573ac47e0 msgr2=0x558573ac0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:49.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.372+0000 7f3db9400640 1 --2- 172.21.15.39:0/1086264105 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558573ac47e0 0x558573ac0380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:49.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.372+0000 7f3db9400640 1 -- 172.21.15.39:0/1086264105 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558573aca340 msgr2=0x558573abeac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:49.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.372+0000 7f3db9400640 1 --2- 172.21.15.39:0/1086264105 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558573aca340 0x558573abeac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:49.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.372+0000 7f3db9400640 1 -- 172.21.15.39:0/1086264105 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5585736fc170 con 0x558573ac51d0 2024-03-18T16:11:49.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.372+0000 7f3db9400640 1 --2- 172.21.15.39:0/1086264105 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558573ac51d0 0x558573ad5e80 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f3da8009b60 tx=0x7f3da80127c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=abf8a570728b219c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:49.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.372+0000 7f3da6600640 1 -- 172.21.15.39:0/1086264105 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3da8009e50 con 0x558573ac51d0 2024-03-18T16:11:49.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.373+0000 7f3da6600640 1 -- 172.21.15.39:0/1086264105 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3da8005500 con 0x558573ac51d0 2024-03-18T16:11:49.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.373+0000 7f3da6600640 1 -- 172.21.15.39:0/1086264105 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3da800e740 con 0x558573ac51d0 2024-03-18T16:11:49.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.374+0000 7f3dc39da740 1 -- 172.21.15.39:0/1086264105 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558573ac51d0 msgr2=0x558573ad5e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:49.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.374+0000 7f3dc39da740 1 --2- 172.21.15.39:0/1086264105 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558573ac51d0 0x558573ad5e80 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f3da8009b60 tx=0x7f3da80127c0 comp rx=0 tx=0).stop 2024-03-18T16:11:49.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.374+0000 7f3dc39da740 1 -- 172.21.15.39:0/1086264105 shutdown_connections 2024-03-18T16:11:49.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.374+0000 7f3dc39da740 1 --2- 172.21.15.39:0/1086264105 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558573aca340 0x558573abeac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:49.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.374+0000 7f3dc39da740 1 --2- 172.21.15.39:0/1086264105 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558573ac51d0 0x558573ad5e80 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:49.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.374+0000 7f3dc39da740 1 --2- 172.21.15.39:0/1086264105 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558573ac47e0 0x558573ac0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:49.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.374+0000 7f3dc39da740 1 -- 172.21.15.39:0/1086264105 >> 172.21.15.39:0/1086264105 conn(0x558573a8c930 msgr2=0x558573abfc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:49.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.374+0000 7f3dc39da740 1 -- 172.21.15.39:0/1086264105 shutdown_connections 2024-03-18T16:11:49.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.374+0000 7f3dc39da740 1 -- 172.21.15.39:0/1086264105 wait complete. 2024-03-18T16:11:49.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.374+0000 7f3dc39da740 1 -- start start 2024-03-18T16:11:49.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.375+0000 7f3dc39da740 1 Processor -- start 2024-03-18T16:11:49.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.375+0000 7f3dc39da740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558573ac47e0 0x558573ad3750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:49.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.375+0000 7f3dc39da740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558573ac51d0 0x558573ad3c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:49.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.375+0000 7f3dc39da740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558573aca340 0x558573ad4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:49.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.375+0000 7f3dc39da740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f3dbc1c2fc0 con 0x558573ac47e0 2024-03-18T16:11:49.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.375+0000 7f3dc39da740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f3dbc1c28c0 con 0x558573ac51d0 2024-03-18T16:11:49.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.375+0000 7f3dc39da740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f3dbc1bfac0 con 0x558573aca340 2024-03-18T16:11:49.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.375+0000 7f3dc39da740 10 client.0 _ll_register_callbacks cb 0x7f3db40048b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:11:49.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.376+0000 7f3db9400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558573ac51d0 0x558573ad3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:49.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.376+0000 7f3db9400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558573ac51d0 0x558573ad3c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:38316/0 (socket says 172.21.15.39:38316) 2024-03-18T16:11:49.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.376+0000 7f3db9e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558573ac47e0 0x558573ad3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:49.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.376+0000 7f3db9400640 1 -- 172.21.15.39:0/3250844588 learned_addr learned my addr 172.21.15.39:0/3250844588 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:11:49.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.376+0000 7f3dba800640 1 --2- 172.21.15.39:0/3250844588 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558573aca340 0x558573ad4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:49.379 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.376+0000 7f3db9400640 1 -- 172.21.15.39:0/3250844588 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558573aca340 msgr2=0x558573ad4ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:49.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.376+0000 7f3db9400640 1 --2- 172.21.15.39:0/3250844588 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558573aca340 0x558573ad4ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:49.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.376+0000 7f3db9400640 1 -- 172.21.15.39:0/3250844588 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558573ac47e0 msgr2=0x558573ad3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:49.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.376+0000 7f3db9400640 1 --2- 172.21.15.39:0/3250844588 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558573ac47e0 0x558573ad3750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:49.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.376+0000 7f3db9400640 1 -- 172.21.15.39:0/3250844588 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558573ad5290 con 0x558573ac51d0 2024-03-18T16:11:49.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.377+0000 7f3db9400640 1 --2- 172.21.15.39:0/3250844588 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558573ac51d0 0x558573ad3c90 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f3da8000c00 tx=0x7f3da8005250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b8911cc99bb6a28 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:49.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.377+0000 7f3d9b400640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:11:49.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.377+0000 7f3d9b400640 1 -- 172.21.15.39:0/3250844588 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3da801f040 con 0x558573ac51d0 2024-03-18T16:11:49.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.377+0000 7f3dc39da740 1 -- 172.21.15.39:0/3250844588 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3da8005500 con 0x558573ac51d0 2024-03-18T16:11:49.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.377+0000 7f3d9b400640 1 -- 172.21.15.39:0/3250844588 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f3da80048c0 con 0x558573ac51d0 2024-03-18T16:11:49.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.377+0000 7f3d9b400640 1 -- 172.21.15.39:0/3250844588 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7f3da8032050 con 0x558573ac51d0 2024-03-18T16:11:49.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.377+0000 7f3dc39da740 10 client.26003 fetch_fsmap learned FSMap version 31 2024-03-18T16:11:49.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.377+0000 7f3dc39da740 1 -- 172.21.15.39:0/3250844588 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x558573b40cf0 con 0x558573ac51d0 2024-03-18T16:11:49.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.378+0000 7f3d9b400640 1 -- 172.21.15.39:0/3250844588 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3da8005500 con 0x558573ac51d0 2024-03-18T16:11:49.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.378+0000 7f3d9b400640 1 -- 172.21.15.39:0/3250844588 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 31) ==== 8374+0+0 (secure 0 0 0) 0x7f3da8019070 con 0x558573ac51d0 2024-03-18T16:11:49.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.378+0000 7f3dc39da740 10 client.26003 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T16:11:49.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.378+0000 7f3dc39da740 10 client.26003 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:11:49.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.378+0000 7f3dc39da740 20 client.26003 populate_metadata read hostname 'smithi039' 2024-03-18T16:11:49.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.378+0000 7f3dc39da740 1 --2- 172.21.15.39:0/3250844588 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x558573b416c0 0x558573b43ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:49.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.378+0000 7f3dc39da740 4 client.26003 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-18T16:11:49.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.378+0000 7f3dc39da740 1 -- 172.21.15.39:0/3250844588 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5585739adaa0 con 0x558573b416c0 2024-03-18T16:11:49.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.378+0000 7f3db9e00640 1 --2- 172.21.15.39:0/3250844588 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x558573b416c0 0x558573b43ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:49.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.379+0000 7f3db9e00640 1 --2- 172.21.15.39:0/3250844588 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x558573b416c0 0x558573b43ae0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9c8c887b2a791253 server_cookie=adca8c54625e8d62 in_seq=0 out_seq=0 2024-03-18T16:11:49.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.379+0000 7f3d9b400640 10 client.26003 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:11:49.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.380+0000 7f3d9b400640 1 -- 172.21.15.39:0/3250844588 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5585739adaa0 con 0x558573b416c0 2024-03-18T16:11:49.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.380+0000 7f3d9b400640 10 client.26003 handle_command_reply: tid=0 2024-03-18T16:11:49.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.380+0000 7f3dc39da740 1 client.26003 shutdown 2024-03-18T16:11:49.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.380+0000 7f3dc39da740 1 -- 172.21.15.39:0/3250844588 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558573ac51d0 msgr2=0x558573ad3c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:49.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.380+0000 7f3dc39da740 1 --2- 172.21.15.39:0/3250844588 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558573ac51d0 0x558573ad3c90 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f3da8000c00 tx=0x7f3da8005250 comp rx=0 tx=0).stop 2024-03-18T16:11:49.383 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.381+0000 7f3dc39da740 1 -- 172.21.15.39:0/3250844588 shutdown_connections 2024-03-18T16:11:49.383 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.381+0000 7f3dc39da740 1 --2- 172.21.15.39:0/3250844588 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x558573b416c0 0x558573b43ae0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:49.383 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.381+0000 7f3dc39da740 1 --2- 172.21.15.39:0/3250844588 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558573ac51d0 0x558573ad3c90 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:49.383 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.381+0000 7f3dc39da740 1 --2- 172.21.15.39:0/3250844588 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558573aca340 0x558573ad4ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:49.383 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.381+0000 7f3d9b400640 0 client.26003 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:11:49.383 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.381+0000 7f3dc39da740 1 --2- 172.21.15.39:0/3250844588 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558573ac47e0 0x558573ad3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:49.383 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.381+0000 7f3dc39da740 1 -- 172.21.15.39:0/3250844588 >> 172.21.15.39:0/3250844588 conn(0x558573a8c930 msgr2=0x558573ac3390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:49.383 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.381+0000 7f3dc39da740 1 -- 172.21.15.39:0/3250844588 shutdown_connections 2024-03-18T16:11:49.383 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.381+0000 7f3dc39da740 1 -- 172.21.15.39:0/3250844588 wait complete. 2024-03-18T16:11:49.383 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.381+0000 7f3dc39da740 20 client.26003 trim_cache size 0 max 16384 2024-03-18T16:11:49.384 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-03-18T16:11:49.384 INFO:teuthology.orchestra.run.smithi039.stdout: "return_code": 0, 2024-03-18T16:11:49.384 INFO:teuthology.orchestra.run.smithi039.stdout: "scrub_tag": "573ad386-da48-4163-8601-7c24108c8fb0", 2024-03-18T16:11:49.384 INFO:teuthology.orchestra.run.smithi039.stdout: "mode": "asynchronous" 2024-03-18T16:11:49.384 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-03-18T16:11:49.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.382+0000 7f3dc0c00640 1 -- 172.21.15.39:0/106550930 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3da00c48e0 msgr2=0x7f3da00c6da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:49.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.382+0000 7f3dc0c00640 1 --2- 172.21.15.39:0/106550930 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3da00c48e0 0x7f3da00c6da0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f3dac009a60 tx=0x7f3dac005f90 comp rx=0 tx=0).stop 2024-03-18T16:11:49.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.382+0000 7f3dc0c00640 1 -- 172.21.15.39:0/106550930 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3dbc078f20 msgr2=0x7f3dbc1b6670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:49.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.382+0000 7f3dc0c00640 1 --2- 172.21.15.39:0/106550930 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3dbc078f20 0x7f3dbc1b6670 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f3db4009bf0 tx=0x7f3db4033930 comp rx=0 tx=0).stop 2024-03-18T16:11:49.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.382+0000 7f3dc0c00640 1 -- 172.21.15.39:0/106550930 shutdown_connections 2024-03-18T16:11:49.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.382+0000 7f3dc0c00640 1 --2- 172.21.15.39:0/106550930 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3dbc076570 0x7f3dbc1afc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:49.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.382+0000 7f3dc0c00640 1 --2- 172.21.15.39:0/106550930 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3da00c48e0 0x7f3da00c6da0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:49.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.382+0000 7f3dc0c00640 1 --2- 172.21.15.39:0/106550930 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3dbc078f20 0x7f3dbc1b6670 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:49.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.382+0000 7f3dc0c00640 1 --2- 172.21.15.39:0/106550930 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3dbc073690 0x7f3dbc1af720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:49.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.382+0000 7f3dc0c00640 1 -- 172.21.15.39:0/106550930 >> 172.21.15.39:0/106550930 conn(0x7f3dbc06e640 msgr2=0x7f3dbc077b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:49.385 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.383+0000 7f3dc0c00640 1 -- 172.21.15.39:0/106550930 shutdown_connections 2024-03-18T16:11:49.386 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.383+0000 7f3dc0c00640 1 -- 172.21.15.39:0/106550930 wait complete. 2024-03-18T16:11:49.413 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T16:11:49.413 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T16:11:49.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:11:49 smithi196 ceph-mon[36307]: pgmap v474: 129 pgs: 129 active+clean; 2.9 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 227 MiB/s wr, 100 op/s 2024-03-18T16:11:49.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:11:49 smithi194 ceph-mon[36200]: pgmap v474: 129 pgs: 129 active+clean; 2.9 GiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 227 MiB/s wr, 100 op/s 2024-03-18T16:11:49.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.723+0000 7fc669c00640 1 -- 172.21.15.39:0/1703254976 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc6641017f0 msgr2=0x7fc664101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:49.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.723+0000 7fc669c00640 1 --2- 172.21.15.39:0/1703254976 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc6641017f0 0x7fc664101c10 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fc654009900 tx=0x7fc65402bc30 comp rx=0 tx=0).stop 2024-03-18T16:11:49.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.723+0000 7fc669c00640 1 -- 172.21.15.39:0/1703254976 shutdown_connections 2024-03-18T16:11:49.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.723+0000 7fc669c00640 1 --2- 172.21.15.39:0/1703254976 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc6641029e0 0x7fc664102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:49.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.723+0000 7fc669c00640 1 --2- 172.21.15.39:0/1703254976 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc6641033a0 0x7fc664109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:49.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.723+0000 7fc669c00640 1 --2- 172.21.15.39:0/1703254976 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc6641017f0 0x7fc664101c10 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:49.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.723+0000 7fc669c00640 1 -- 172.21.15.39:0/1703254976 >> 172.21.15.39:0/1703254976 conn(0x7fc6640fcd60 msgr2=0x7fc6640ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:49.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.724+0000 7fc669c00640 1 -- 172.21.15.39:0/1703254976 shutdown_connections 2024-03-18T16:11:49.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.724+0000 7fc669c00640 1 -- 172.21.15.39:0/1703254976 wait complete. 2024-03-18T16:11:49.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.725+0000 7fc669c00640 1 Processor -- start 2024-03-18T16:11:49.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.725+0000 7fc669c00640 1 -- start start 2024-03-18T16:11:49.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.725+0000 7fc669c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc6641017f0 0x7fc664195950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:49.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.725+0000 7fc669c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc6641029e0 0x7fc664195e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:49.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.725+0000 7fc669c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc6641033a0 0x7fc66419cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:49.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.725+0000 7fc669c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fc66410b2e0 con 0x7fc6641029e0 2024-03-18T16:11:49.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.725+0000 7fc669c00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fc66410b160 con 0x7fc6641033a0 2024-03-18T16:11:49.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.725+0000 7fc669c00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fc66410b460 con 0x7fc6641017f0 2024-03-18T16:11:49.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.725+0000 7fc662a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc6641017f0 0x7fc664195950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:49.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.725+0000 7fc662a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc6641017f0 0x7fc664195950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:60166/0 (socket says 172.21.15.39:60166) 2024-03-18T16:11:49.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.725+0000 7fc662a00640 1 -- 172.21.15.39:0/735081683 learned_addr learned my addr 172.21.15.39:0/735081683 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:11:49.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.726+0000 7fc663400640 1 --2- 172.21.15.39:0/735081683 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc6641033a0 0x7fc66419cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:49.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.726+0000 7fc662a00640 1 -- 172.21.15.39:0/735081683 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc6641033a0 msgr2=0x7fc66419cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:49.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.726+0000 7fc662a00640 1 --2- 172.21.15.39:0/735081683 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc6641033a0 0x7fc66419cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:49.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.726+0000 7fc662a00640 1 -- 172.21.15.39:0/735081683 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc6641029e0 msgr2=0x7fc664195e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:49.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.726+0000 7fc662000640 1 --2- 172.21.15.39:0/735081683 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc6641029e0 0x7fc664195e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:49.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.726+0000 7fc662a00640 1 --2- 172.21.15.39:0/735081683 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc6641029e0 0x7fc664195e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:49.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.726+0000 7fc662a00640 1 -- 172.21.15.39:0/735081683 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc66419d570 con 0x7fc6641017f0 2024-03-18T16:11:49.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.726+0000 7fc662000640 1 --2- 172.21.15.39:0/735081683 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc6641029e0 0x7fc664195e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:11:49.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.726+0000 7fc663400640 1 --2- 172.21.15.39:0/735081683 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc6641033a0 0x7fc66419cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:11:49.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.726+0000 7fc662a00640 1 --2- 172.21.15.39:0/735081683 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc6641017f0 0x7fc664195950 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fc654000c00 tx=0x7fc65402c6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:49.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.727+0000 7fc65b400640 1 -- 172.21.15.39:0/735081683 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc654043020 con 0x7fc6641017f0 2024-03-18T16:11:49.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.727+0000 7fc65b400640 1 -- 172.21.15.39:0/735081683 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc6540054e0 con 0x7fc6641017f0 2024-03-18T16:11:49.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.727+0000 7fc669c00640 1 -- 172.21.15.39:0/735081683 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc66419d800 con 0x7fc6641017f0 2024-03-18T16:11:49.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.727+0000 7fc65b400640 1 -- 172.21.15.39:0/735081683 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc654003d00 con 0x7fc6641017f0 2024-03-18T16:11:49.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.727+0000 7fc669c00640 1 -- 172.21.15.39:0/735081683 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc66419dd70 con 0x7fc6641017f0 2024-03-18T16:11:49.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.731+0000 7fc66c9eb740 1 Processor -- start 2024-03-18T16:11:49.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.731+0000 7fc66c9eb740 1 -- start start 2024-03-18T16:11:49.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.731+0000 7fc66c9eb740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c0c4137810 0x55c0c4137c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:49.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.731+0000 7fc66c9eb740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c0c4138200 0x55c0c413cd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:49.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.732+0000 7fc66c9eb740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c0c413d370 0x55c0c413f800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:49.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.732+0000 7fc66c9eb740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55c0c409c1f0 con 0x55c0c4137810 2024-03-18T16:11:49.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.732+0000 7fc66c9eb740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55c0c4044a80 con 0x55c0c4138200 2024-03-18T16:11:49.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.732+0000 7fc66c9eb740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55c0c4098290 con 0x55c0c413d370 2024-03-18T16:11:49.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.732+0000 7fc662000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c0c413d370 0x55c0c413f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:49.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.732+0000 7fc662000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c0c413d370 0x55c0c413f800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:60180/0 (socket says 172.21.15.39:60180) 2024-03-18T16:11:49.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.732+0000 7fc662000640 1 -- 172.21.15.39:0/769783727 learned_addr learned my addr 172.21.15.39:0/769783727 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:11:49.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.732+0000 7fc662000640 1 -- 172.21.15.39:0/769783727 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c0c4138200 msgr2=0x55c0c413cd00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:11:49.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.732+0000 7fc662000640 1 --2- 172.21.15.39:0/769783727 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c0c4138200 0x55c0c413cd00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:49.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.732+0000 7fc662000640 1 -- 172.21.15.39:0/769783727 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c0c4137810 msgr2=0x55c0c4137c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:49.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.732+0000 7fc662000640 1 --2- 172.21.15.39:0/769783727 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c0c4137810 0x55c0c4137c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:49.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.732+0000 7fc662000640 1 -- 172.21.15.39:0/769783727 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c0c3e7a7d0 con 0x55c0c413d370 2024-03-18T16:11:49.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.733+0000 7fc662000640 1 --2- 172.21.15.39:0/769783727 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c0c413d370 0x55c0c413f800 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fc65000bed0 tx=0x7fc6500124b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=79b95f630b6700b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:49.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.733+0000 7fc64fe00640 1 -- 172.21.15.39:0/769783727 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc650005020 con 0x55c0c413d370 2024-03-18T16:11:49.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.733+0000 7fc64fe00640 1 -- 172.21.15.39:0/769783727 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc6500047a0 con 0x55c0c413d370 2024-03-18T16:11:49.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.733+0000 7fc64fe00640 1 -- 172.21.15.39:0/769783727 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6500186f0 con 0x55c0c413d370 2024-03-18T16:11:49.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.733+0000 7fc65b400640 1 -- 172.21.15.39:0/735081683 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fc654046020 con 0x7fc6641017f0 2024-03-18T16:11:49.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.734+0000 7fc66c9eb740 1 -- 172.21.15.39:0/769783727 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c0c413d370 msgr2=0x55c0c413f800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:49.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.734+0000 7fc66c9eb740 1 --2- 172.21.15.39:0/769783727 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c0c413d370 0x55c0c413f800 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fc65000bed0 tx=0x7fc6500124b0 comp rx=0 tx=0).stop 2024-03-18T16:11:49.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.735+0000 7fc66c9eb740 1 -- 172.21.15.39:0/769783727 shutdown_connections 2024-03-18T16:11:49.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.735+0000 7fc66c9eb740 1 --2- 172.21.15.39:0/769783727 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c0c4138200 0x55c0c413cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:49.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.735+0000 7fc66c9eb740 1 --2- 172.21.15.39:0/769783727 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c0c413d370 0x55c0c413f800 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:49.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.735+0000 7fc66c9eb740 1 --2- 172.21.15.39:0/769783727 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c0c4137810 0x55c0c4137c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:49.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.735+0000 7fc66c9eb740 1 -- 172.21.15.39:0/769783727 >> 172.21.15.39:0/769783727 conn(0x55c0c4132d80 msgr2=0x55c0c4133180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:49.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.735+0000 7fc66c9eb740 1 -- 172.21.15.39:0/769783727 shutdown_connections 2024-03-18T16:11:49.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.735+0000 7fc66c9eb740 1 -- 172.21.15.39:0/769783727 wait complete. 2024-03-18T16:11:49.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.735+0000 7fc65b400640 1 --2- 172.21.15.39:0/735081683 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fc6480a2810 0x7fc6480a4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:49.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.735+0000 7fc65b400640 1 -- 172.21.15.39:0/735081683 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fc6540f1040 con 0x7fc6641017f0 2024-03-18T16:11:49.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.736+0000 7fc66c9eb740 1 -- start start 2024-03-18T16:11:49.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.736+0000 7fc662000640 1 --2- 172.21.15.39:0/735081683 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fc6480a2810 0x7fc6480a4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:49.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.736+0000 7fc66c9eb740 1 Processor -- start 2024-03-18T16:11:49.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.737+0000 7fc662000640 1 --2- 172.21.15.39:0/735081683 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fc6480a2810 0x7fc6480a4cd0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x55c0c4133cf0 tx=0x7fc650012ef0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:49.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.737+0000 7fc66c9eb740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c0c4137810 0x55c0c4141d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:49.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.737+0000 7fc66c9eb740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c0c4138200 0x55c0c41422b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:49.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.737+0000 7fc66c9eb740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c0c413d370 0x55c0c4151b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:49.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.737+0000 7fc662000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c0c4138200 0x55c0c41422b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:49.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.737+0000 7fc662000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c0c4138200 0x55c0c41422b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:60182/0 (socket says 172.21.15.39:60182) 2024-03-18T16:11:49.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.741+0000 7fc662a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c0c4137810 0x55c0c4141d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:49.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.741+0000 7fc662a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c0c4137810 0x55c0c4141d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:38346/0 (socket says 172.21.15.39:38346) 2024-03-18T16:11:49.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.741+0000 7fc662000640 1 -- 172.21.15.39:0/3326705275 learned_addr learned my addr 172.21.15.39:0/3326705275 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:11:49.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.741+0000 7fc663400640 1 --2- 172.21.15.39:0/3326705275 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c0c413d370 0x55c0c4151b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:49.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.742+0000 7fc66c9eb740 1 -- 172.21.15.39:0/3326705275 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55c0c4140320 con 0x55c0c413d370 2024-03-18T16:11:49.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.742+0000 7fc66c9eb740 1 -- 172.21.15.39:0/3326705275 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55c0c4057ed0 con 0x55c0c4137810 2024-03-18T16:11:49.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.742+0000 7fc66c9eb740 1 -- 172.21.15.39:0/3326705275 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55c0c41404a0 con 0x55c0c4138200 2024-03-18T16:11:49.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.743+0000 7fc66c9eb740 10 client.0 _ll_register_callbacks cb 0x55c0c4093220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:11:49.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.743+0000 7fc662a00640 1 -- 172.21.15.39:0/3326705275 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c0c4138200 msgr2=0x55c0c41422b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:49.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.743+0000 7fc662a00640 1 --2- 172.21.15.39:0/3326705275 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c0c4138200 0x55c0c41422b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:49.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.743+0000 7fc662a00640 1 -- 172.21.15.39:0/3326705275 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c0c413d370 msgr2=0x55c0c4151b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:49.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.743+0000 7fc662a00640 1 --2- 172.21.15.39:0/3326705275 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c0c413d370 0x55c0c4151b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:49.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.743+0000 7fc662a00640 1 -- 172.21.15.39:0/3326705275 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c0c3db03f0 con 0x55c0c4137810 2024-03-18T16:11:49.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.743+0000 7fc663400640 1 --2- 172.21.15.39:0/3326705275 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c0c413d370 0x55c0c4151b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:11:49.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.744+0000 7fc662000640 1 --2- 172.21.15.39:0/3326705275 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c0c4138200 0x55c0c41422b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:11:49.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.744+0000 7fc662a00640 1 --2- 172.21.15.39:0/3326705275 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c0c4137810 0x55c0c4141d70 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x55c0c4138c70 tx=0x7fc6540bd480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3491082169163767 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:49.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.744+0000 7fc64cc00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:11:49.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.744+0000 7fc64cc00640 1 -- 172.21.15.39:0/3326705275 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6540c5b00 con 0x55c0c4137810 2024-03-18T16:11:49.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.745+0000 7fc64cc00640 1 -- 172.21.15.39:0/3326705275 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fc6540a6b80 con 0x55c0c4137810 2024-03-18T16:11:49.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.745+0000 7fc66c9eb740 1 -- 172.21.15.39:0/3326705275 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc6500047a0 con 0x55c0c4137810 2024-03-18T16:11:49.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.746+0000 7fc64cc00640 1 -- 172.21.15.39:0/3326705275 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fc654004c80 con 0x55c0c4137810 2024-03-18T16:11:49.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.746+0000 7fc66c9eb740 10 client.26009 fetch_fsmap learned FSMap version 31 2024-03-18T16:11:49.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.746+0000 7fc66c9eb740 1 -- 172.21.15.39:0/3326705275 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55c0c3d700f0 con 0x55c0c4137810 2024-03-18T16:11:49.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.746+0000 7fc64cc00640 1 -- 172.21.15.39:0/3326705275 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc6500047a0 con 0x55c0c4137810 2024-03-18T16:11:49.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.746+0000 7fc64cc00640 1 -- 172.21.15.39:0/3326705275 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 31) ==== 8374+0+0 (secure 0 0 0) 0x7fc6540bca40 con 0x55c0c4137810 2024-03-18T16:11:49.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.747+0000 7fc66c9eb740 10 client.26009 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T16:11:49.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.747+0000 7fc66c9eb740 10 client.26009 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:11:49.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.747+0000 7fc66c9eb740 20 client.26009 populate_metadata read hostname 'smithi039' 2024-03-18T16:11:49.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.747+0000 7fc66c9eb740 1 --2- 172.21.15.39:0/3326705275 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55c0c41563a0 0x55c0c4158800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:49.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.747+0000 7fc66c9eb740 4 client.26009 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:11:49.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.747+0000 7fc66c9eb740 1 -- 172.21.15.39:0/3326705275 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c0c40afbe0 con 0x55c0c41563a0 2024-03-18T16:11:49.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.747+0000 7fc662000640 1 --2- 172.21.15.39:0/3326705275 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55c0c41563a0 0x55c0c4158800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:49.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.748+0000 7fc662000640 1 --2- 172.21.15.39:0/3326705275 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55c0c41563a0 0x55c0c4158800 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb513728b4b8442 server_cookie=ed913883fbe9a7fa in_seq=0 out_seq=0 2024-03-18T16:11:49.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.748+0000 7fc64cc00640 10 client.26009 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:11:49.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.749+0000 7fc64cc00640 1 -- 172.21.15.39:0/3326705275 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c0c40afbe0 con 0x55c0c41563a0 2024-03-18T16:11:49.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.749+0000 7fc64cc00640 10 client.26009 handle_command_reply: tid=0 2024-03-18T16:11:49.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.749+0000 7fc66c9eb740 1 client.26009 shutdown 2024-03-18T16:11:49.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.749+0000 7fc66c9eb740 1 -- 172.21.15.39:0/3326705275 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c0c4137810 msgr2=0x55c0c4141d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:49.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.749+0000 7fc66c9eb740 1 --2- 172.21.15.39:0/3326705275 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c0c4137810 0x55c0c4141d70 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x55c0c4138c70 tx=0x7fc6540bd480 comp rx=0 tx=0).stop 2024-03-18T16:11:49.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.749+0000 7fc66c9eb740 1 -- 172.21.15.39:0/3326705275 shutdown_connections 2024-03-18T16:11:49.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.749+0000 7fc66c9eb740 1 --2- 172.21.15.39:0/3326705275 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55c0c41563a0 0x55c0c4158800 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:49.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.749+0000 7fc66c9eb740 1 --2- 172.21.15.39:0/3326705275 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c0c413d370 0x55c0c4151b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:49.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.749+0000 7fc66c9eb740 1 --2- 172.21.15.39:0/3326705275 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c0c4138200 0x55c0c41422b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:49.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.749+0000 7fc66c9eb740 1 --2- 172.21.15.39:0/3326705275 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c0c4137810 0x55c0c4141d70 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:49.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.749+0000 7fc64cc00640 0 client.26009 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:11:49.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.749+0000 7fc66c9eb740 1 -- 172.21.15.39:0/3326705275 >> 172.21.15.39:0/3326705275 conn(0x55c0c4132d80 msgr2=0x55c0c4131ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:49.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.749+0000 7fc66c9eb740 1 -- 172.21.15.39:0/3326705275 shutdown_connections 2024-03-18T16:11:49.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.750+0000 7fc66c9eb740 1 -- 172.21.15.39:0/3326705275 wait complete. 2024-03-18T16:11:49.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.750+0000 7fc66c9eb740 20 client.26009 trim_cache size 0 max 16384 2024-03-18T16:11:49.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.797+0000 7fc66c9eb740 1 Processor -- start 2024-03-18T16:11:49.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.797+0000 7fc66c9eb740 1 -- start start 2024-03-18T16:11:49.800 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.798+0000 7fc66c9eb740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c0c4137810 0x55c0c4131d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:49.801 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.798+0000 7fc66c9eb740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c0c4138200 0x55c0c4151b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:49.801 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.798+0000 7fc66c9eb740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c0c413d370 0x55c0c4141430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:49.801 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.798+0000 7fc66c9eb740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fc6641a8240 con 0x55c0c4137810 2024-03-18T16:11:49.801 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.798+0000 7fc66c9eb740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fc6641a9e10 con 0x55c0c413d370 2024-03-18T16:11:49.801 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.798+0000 7fc66c9eb740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fc6641a8910 con 0x55c0c4138200 2024-03-18T16:11:49.801 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.798+0000 7fc662a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c0c4137810 0x55c0c4131d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:49.801 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.798+0000 7fc662a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c0c4137810 0x55c0c4131d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:59688/0 (socket says 172.21.15.39:59688) 2024-03-18T16:11:49.801 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.798+0000 7fc662a00640 1 -- 172.21.15.39:0/361566814 learned_addr learned my addr 172.21.15.39:0/361566814 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:11:49.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.799+0000 7fc662a00640 1 -- 172.21.15.39:0/361566814 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c0c4138200 msgr2=0x55c0c4151b40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T16:11:49.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.799+0000 7fc662a00640 1 --2- 172.21.15.39:0/361566814 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c0c4138200 0x55c0c4151b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:49.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.799+0000 7fc662a00640 1 -- 172.21.15.39:0/361566814 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c0c413d370 msgr2=0x55c0c4141430 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T16:11:49.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.799+0000 7fc662a00640 1 --2- 172.21.15.39:0/361566814 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c0c413d370 0x55c0c4141430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:49.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.799+0000 7fc662a00640 1 -- 172.21.15.39:0/361566814 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c0c412d370 con 0x55c0c4137810 2024-03-18T16:11:49.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.800+0000 7fc662a00640 1 --2- 172.21.15.39:0/361566814 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c0c4137810 0x55c0c4131d40 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fc65402c8e0 tx=0x7fc6540bdd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d48d465a3bda102d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:49.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.800+0000 7fc64f400640 1 -- 172.21.15.39:0/361566814 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6540b9890 con 0x55c0c4137810 2024-03-18T16:11:49.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.800+0000 7fc64f400640 1 -- 172.21.15.39:0/361566814 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc654004c80 con 0x55c0c4137810 2024-03-18T16:11:49.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.800+0000 7fc64f400640 1 -- 172.21.15.39:0/361566814 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6540bcf50 con 0x55c0c4137810 2024-03-18T16:11:49.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.801+0000 7fc66c9eb740 1 -- 172.21.15.39:0/361566814 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c0c4137810 msgr2=0x55c0c4131d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:49.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.801+0000 7fc66c9eb740 1 --2- 172.21.15.39:0/361566814 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c0c4137810 0x55c0c4131d40 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fc65402c8e0 tx=0x7fc6540bdd30 comp rx=0 tx=0).stop 2024-03-18T16:11:49.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.801+0000 7fc66c9eb740 1 -- 172.21.15.39:0/361566814 shutdown_connections 2024-03-18T16:11:49.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.801+0000 7fc66c9eb740 1 --2- 172.21.15.39:0/361566814 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c0c413d370 0x55c0c4141430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:49.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.801+0000 7fc66c9eb740 1 --2- 172.21.15.39:0/361566814 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c0c4138200 0x55c0c4151b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:49.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.801+0000 7fc66c9eb740 1 --2- 172.21.15.39:0/361566814 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c0c4137810 0x55c0c4131d40 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:49.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.801+0000 7fc66c9eb740 1 -- 172.21.15.39:0/361566814 >> 172.21.15.39:0/361566814 conn(0x55c0c4132d80 msgr2=0x55c0c4131630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:49.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.801+0000 7fc66c9eb740 1 -- 172.21.15.39:0/361566814 shutdown_connections 2024-03-18T16:11:49.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.802+0000 7fc66c9eb740 1 -- 172.21.15.39:0/361566814 wait complete. 2024-03-18T16:11:49.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.802+0000 7fc66c9eb740 1 -- start start 2024-03-18T16:11:49.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.802+0000 7fc66c9eb740 1 Processor -- start 2024-03-18T16:11:49.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.803+0000 7fc66c9eb740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c0c4137810 0x55c0c414efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:49.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.803+0000 7fc66c9eb740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c0c4138200 0x55c0c414f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:49.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.803+0000 7fc66c9eb740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c0c413d370 0x55c0c4150420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:49.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.803+0000 7fc66c9eb740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fc6641a96e0 con 0x55c0c4138200 2024-03-18T16:11:49.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.803+0000 7fc66c9eb740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fc6641a8fe0 con 0x55c0c4137810 2024-03-18T16:11:49.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.803+0000 7fc66c9eb740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fc6641a6220 con 0x55c0c413d370 2024-03-18T16:11:49.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.803+0000 7fc66c9eb740 10 client.0 _ll_register_callbacks cb 0x7fc654005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:11:49.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.803+0000 7fc662000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c0c4138200 0x55c0c414f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:49.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.803+0000 7fc662000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c0c4138200 0x55c0c414f4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:59702/0 (socket says 172.21.15.39:59702) 2024-03-18T16:11:49.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.803+0000 7fc662000640 1 -- 172.21.15.39:0/3369756522 learned_addr learned my addr 172.21.15.39:0/3369756522 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:11:49.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.803+0000 7fc662a00640 1 --2- 172.21.15.39:0/3369756522 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c0c4137810 0x55c0c414efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:49.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.803+0000 7fc663400640 1 --2- 172.21.15.39:0/3369756522 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c0c413d370 0x55c0c4150420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:49.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.803+0000 7fc662000640 1 -- 172.21.15.39:0/3369756522 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c0c413d370 msgr2=0x55c0c4150420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:49.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.803+0000 7fc662000640 1 --2- 172.21.15.39:0/3369756522 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c0c413d370 0x55c0c4150420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:49.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.803+0000 7fc662000640 1 -- 172.21.15.39:0/3369756522 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c0c4137810 msgr2=0x55c0c414efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:11:49.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.803+0000 7fc662000640 1 --2- 172.21.15.39:0/3369756522 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c0c4137810 0x55c0c414efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:49.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.804+0000 7fc662000640 1 -- 172.21.15.39:0/3369756522 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c0c4133af0 con 0x55c0c4138200 2024-03-18T16:11:49.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.804+0000 7fc662a00640 1 --2- 172.21.15.39:0/3369756522 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c0c4137810 0x55c0c414efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:11:49.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.804+0000 7fc662000640 1 --2- 172.21.15.39:0/3369756522 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c0c4138200 0x55c0c414f4f0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fc6500129b0 tx=0x7fc6500124b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf61fe0b17567564 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:11:49.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.804+0000 7fc64cc00640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-03-18T16:11:49.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.804+0000 7fc64cc00640 1 -- 172.21.15.39:0/3369756522 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc650004780 con 0x55c0c4138200 2024-03-18T16:11:49.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.804+0000 7fc64cc00640 1 -- 172.21.15.39:0/3369756522 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fc650021b20 con 0x55c0c4138200 2024-03-18T16:11:49.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.805+0000 7fc66c9eb740 1 -- 172.21.15.39:0/3369756522 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc654004c80 con 0x55c0c4138200 2024-03-18T16:11:49.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.805+0000 7fc64cc00640 1 -- 172.21.15.39:0/3369756522 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) ==== 24+0+0 (secure 0 0 0) 0x7fc650041050 con 0x55c0c4138200 2024-03-18T16:11:49.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.805+0000 7fc66c9eb740 10 client.15819 fetch_fsmap learned FSMap version 31 2024-03-18T16:11:49.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.805+0000 7fc66c9eb740 1 -- 172.21.15.39:0/3369756522 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) -- 0x55c0c4151bc0 con 0x55c0c4138200 2024-03-18T16:11:49.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.806+0000 7fc64cc00640 1 -- 172.21.15.39:0/3369756522 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc654004c80 con 0x55c0c4138200 2024-03-18T16:11:49.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.806+0000 7fc64cc00640 1 -- 172.21.15.39:0/3369756522 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 31) ==== 8374+0+0 (secure 0 0 0) 0x7fc65002c040 con 0x55c0c4138200 2024-03-18T16:11:49.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.806+0000 7fc66c9eb740 10 client.15819 fetch_fsmap finished waiting for FSMap version 31 2024-03-18T16:11:49.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.806+0000 7fc66c9eb740 10 client.15819 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:11:49.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.806+0000 7fc66c9eb740 20 client.15819 populate_metadata read hostname 'smithi039' 2024-03-18T16:11:49.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.806+0000 7fc66c9eb740 1 --2- 172.21.15.39:0/3369756522 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55c0c41bc9f0 0x55c0c41bee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:11:49.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.806+0000 7fc66c9eb740 4 client.15819 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T16:11:49.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.806+0000 7fc66c9eb740 1 -- 172.21.15.39:0/3369756522 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55c0c40a9730 con 0x55c0c41bc9f0 2024-03-18T16:11:49.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.806+0000 7fc662a00640 1 --2- 172.21.15.39:0/3369756522 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55c0c41bc9f0 0x55c0c41bee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:11:49.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.807+0000 7fc64cc00640 10 client.15819 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:11:49.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.807+0000 7fc662a00640 1 --2- 172.21.15.39:0/3369756522 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55c0c41bc9f0 0x55c0c41bee10 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b45deedc3464a771 server_cookie=1fe19f61abc79aaf in_seq=0 out_seq=0 2024-03-18T16:11:49.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.807+0000 7fc64cc00640 1 -- 172.21.15.39:0/3369756522 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55c0c40a9730 con 0x55c0c41bc9f0 2024-03-18T16:11:49.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.807+0000 7fc64cc00640 10 client.15819 handle_command_reply: tid=0 2024-03-18T16:11:49.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.807+0000 7fc66c9eb740 1 client.15819 shutdown 2024-03-18T16:11:49.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.807+0000 7fc66c9eb740 1 -- 172.21.15.39:0/3369756522 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c0c4138200 msgr2=0x55c0c414f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:49.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.807+0000 7fc66c9eb740 1 --2- 172.21.15.39:0/3369756522 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c0c4138200 0x55c0c414f4f0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fc6500129b0 tx=0x7fc6500124b0 comp rx=0 tx=0).stop 2024-03-18T16:11:49.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.808+0000 7fc66c9eb740 1 -- 172.21.15.39:0/3369756522 shutdown_connections 2024-03-18T16:11:49.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.808+0000 7fc66c9eb740 1 --2- 172.21.15.39:0/3369756522 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55c0c41bc9f0 0x55c0c41bee10 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:49.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.808+0000 7fc66c9eb740 1 --2- 172.21.15.39:0/3369756522 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c0c413d370 0x55c0c4150420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:49.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.808+0000 7fc66c9eb740 1 --2- 172.21.15.39:0/3369756522 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c0c4138200 0x55c0c414f4f0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:49.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.808+0000 7fc66c9eb740 1 --2- 172.21.15.39:0/3369756522 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c0c4137810 0x55c0c414efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:49.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.808+0000 7fc66c9eb740 1 -- 172.21.15.39:0/3369756522 >> 172.21.15.39:0/3369756522 conn(0x55c0c4132d80 msgr2=0x55c0c4131ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:49.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.808+0000 7fc64cc00640 0 client.15819 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:11:49.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.808+0000 7fc66c9eb740 1 -- 172.21.15.39:0/3369756522 shutdown_connections 2024-03-18T16:11:49.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.808+0000 7fc66c9eb740 1 -- 172.21.15.39:0/3369756522 wait complete. 2024-03-18T16:11:49.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.808+0000 7fc66c9eb740 20 client.15819 trim_cache size 0 max 16384 2024-03-18T16:11:49.812 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-03-18T16:11:49.813 INFO:teuthology.orchestra.run.smithi039.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-18T16:11:49.813 INFO:teuthology.orchestra.run.smithi039.stdout: "scrubs": { 2024-03-18T16:11:49.813 INFO:teuthology.orchestra.run.smithi039.stdout: "573ad386-da48-4163-8601-7c24108c8fb0": { 2024-03-18T16:11:49.813 INFO:teuthology.orchestra.run.smithi039.stdout: "path": "/", 2024-03-18T16:11:49.813 INFO:teuthology.orchestra.run.smithi039.stdout: "tag": "573ad386-da48-4163-8601-7c24108c8fb0", 2024-03-18T16:11:49.813 INFO:teuthology.orchestra.run.smithi039.stdout: "options": "recursive,force" 2024-03-18T16:11:49.813 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:11:49.813 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:11:49.813 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-03-18T16:11:49.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.809+0000 7fc669c00640 1 -- 172.21.15.39:0/735081683 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fc6480a2810 msgr2=0x7fc6480a4cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:49.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.809+0000 7fc669c00640 1 --2- 172.21.15.39:0/735081683 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fc6480a2810 0x7fc6480a4cd0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x55c0c4133cf0 tx=0x7fc650012ef0 comp rx=0 tx=0).stop 2024-03-18T16:11:49.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.809+0000 7fc669c00640 1 -- 172.21.15.39:0/735081683 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc6641017f0 msgr2=0x7fc664195950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:11:49.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.809+0000 7fc669c00640 1 --2- 172.21.15.39:0/735081683 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc6641017f0 0x7fc664195950 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fc654000c00 tx=0x7fc65402c6c0 comp rx=0 tx=0).stop 2024-03-18T16:11:49.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.809+0000 7fc669c00640 1 -- 172.21.15.39:0/735081683 shutdown_connections 2024-03-18T16:11:49.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.809+0000 7fc669c00640 1 --2- 172.21.15.39:0/735081683 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc6641033a0 0x7fc66419cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:49.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.809+0000 7fc669c00640 1 --2- 172.21.15.39:0/735081683 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fc6480a2810 0x7fc6480a4cd0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:49.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.809+0000 7fc669c00640 1 --2- 172.21.15.39:0/735081683 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc6641029e0 0x7fc664195e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:11:49.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.809+0000 7fc669c00640 1 --2- 172.21.15.39:0/735081683 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc6641017f0 0x7fc664195950 secure :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fc654000c00 tx=0x7fc65402c6c0 comp rx=0 tx=0).stop 2024-03-18T16:11:49.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.809+0000 7fc669c00640 1 -- 172.21.15.39:0/735081683 >> 172.21.15.39:0/735081683 conn(0x7fc6640fcd60 msgr2=0x7fc664105d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:11:49.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.809+0000 7fc669c00640 1 -- 172.21.15.39:0/735081683 shutdown_connections 2024-03-18T16:11:49.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:11:49.809+0000 7fc669c00640 1 -- 172.21.15.39:0/735081683 wait complete. 2024-03-18T16:11:49.828 INFO:tasks.cephfs.filesystem:scrub status for tag:573ad386-da48-4163-8601-7c24108c8fb0 - {'path': '/', 'tag': '573ad386-da48-4163-8601-7c24108c8fb0', 'options': 'recursive,force'} 2024-03-18T16:11:50.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:11:50 smithi039 ceph-mon[31721]: scrub queued for path: / 2024-03-18T16:11:50.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:11:50 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:11:50.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:11:50 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:11:50.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:11:50 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:11:50.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:11:50 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:11:50.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:11:50 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:11:50.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:11:50 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:11:50.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:11:50 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:11:50.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:11:50 smithi196 ceph-mon[36307]: scrub queued for path: / 2024-03-18T16:11:50.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:11:50 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:11:50.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:11:50 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:11:50.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:11:50 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:11:50.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:11:50 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:11:50.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:11:50 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:11:50.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:11:50 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:11:50.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:11:50 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:11:50.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:11:50 smithi194 ceph-mon[36200]: scrub queued for path: / 2024-03-18T16:11:50.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:11:50 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:11:50.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:11:50 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:11:50.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:11:50 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:11:50.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:11:50 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:11:50.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:11:50 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:11:50.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:11:50 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:11:50.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:11:50 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:11:51.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:11:51 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:11:51.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:11:51 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:11:51.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:11:51 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:11:51.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:11:51 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:11:51.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:11:51 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:11:51.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:11:51 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:11:51.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:11:51 smithi039 ceph-mon[31721]: pgmap v475: 129 pgs: 129 active+clean; 3.4 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 252 MiB/s wr, 96 op/s 2024-03-18T16:11:51.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:11:51 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:11:51.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:11:51 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:11:51.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:11:51 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:11:51.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:11:51 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:11:51.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:11:51 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:11:51.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:11:51 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:11:51.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:11:51 smithi196 ceph-mon[36307]: pgmap v475: 129 pgs: 129 active+clean; 3.4 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 252 MiB/s wr, 96 op/s 2024-03-18T16:11:51.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:11:51 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:11:51.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:11:51 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:11:51.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:11:51 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:11:51.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:11:51 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:11:51.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:11:51 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:11:51.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:11:51 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:11:51.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:11:51 smithi194 ceph-mon[36200]: pgmap v475: 129 pgs: 129 active+clean; 3.4 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 252 MiB/s wr, 96 op/s 2024-03-18T16:11:53.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:11:53 smithi039 ceph-mon[31721]: pgmap v476: 129 pgs: 129 active+clean; 3.8 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 241 MiB/s wr, 90 op/s 2024-03-18T16:11:53.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:11:53 smithi039 ceph-mon[31721]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T16:11:53.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:11:53 smithi196 ceph-mon[36307]: pgmap v476: 129 pgs: 129 active+clean; 3.8 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 241 MiB/s wr, 90 op/s 2024-03-18T16:11:53.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:11:53 smithi196 ceph-mon[36307]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T16:11:53.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:11:53 smithi194 ceph-mon[36200]: pgmap v476: 129 pgs: 129 active+clean; 3.8 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 241 MiB/s wr, 90 op/s 2024-03-18T16:11:53.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:11:53 smithi194 ceph-mon[36200]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T16:11:55.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:11:55 smithi039 ceph-mon[31721]: pgmap v477: 129 pgs: 129 active+clean; 4.4 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 265 MiB/s wr, 106 op/s 2024-03-18T16:11:55.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:11:55 smithi196 ceph-mon[36307]: pgmap v477: 129 pgs: 129 active+clean; 4.4 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 265 MiB/s wr, 106 op/s 2024-03-18T16:11:55.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:11:55 smithi194 ceph-mon[36200]: pgmap v477: 129 pgs: 129 active+clean; 4.4 GiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 265 MiB/s wr, 106 op/s 2024-03-18T16:11:57.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:11:57 smithi196 ceph-mon[36307]: pgmap v478: 129 pgs: 129 active+clean; 4.7 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 228 MiB/s wr, 88 op/s 2024-03-18T16:11:57.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:11:57 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:11:57.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:11:57 smithi194 ceph-mon[36200]: pgmap v478: 129 pgs: 129 active+clean; 4.7 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 228 MiB/s wr, 88 op/s 2024-03-18T16:11:57.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:11:57 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:11:57.589 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:11:57 smithi039 ceph-mon[31721]: pgmap v478: 129 pgs: 129 active+clean; 4.7 GiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 228 MiB/s wr, 88 op/s 2024-03-18T16:11:57.589 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:11:57 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:11:58.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:11:58 smithi196 ceph-mon[36307]: scrub summary: idle 2024-03-18T16:11:58.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:11:58 smithi196 ceph-mon[36307]: scrub summary: idle 2024-03-18T16:11:58.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:11:58 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:11:58.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:11:58 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:11:58.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:11:58 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:11:58.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:11:58 smithi194 ceph-mon[36200]: scrub summary: idle 2024-03-18T16:11:58.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:11:58 smithi194 ceph-mon[36200]: scrub summary: idle 2024-03-18T16:11:58.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:11:58 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:11:58.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:11:58 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:11:58.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:11:58 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:11:58.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:11:58 smithi039 ceph-mon[31721]: scrub summary: idle 2024-03-18T16:11:58.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:11:58 smithi039 ceph-mon[31721]: scrub summary: idle 2024-03-18T16:11:58.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:11:58 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:11:58.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:11:58 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:11:58.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:11:58 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:11:59.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:11:59 smithi196 ceph-mon[36307]: scrub summary: idle 2024-03-18T16:11:59.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:11:59 smithi196 ceph-mon[36307]: scrub summary: idle 2024-03-18T16:11:59.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:11:59 smithi196 ceph-mon[36307]: pgmap v479: 129 pgs: 129 active+clean; 5.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 234 MiB/s wr, 97 op/s 2024-03-18T16:11:59.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:11:59 smithi194 ceph-mon[36200]: scrub summary: idle 2024-03-18T16:11:59.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:11:59 smithi194 ceph-mon[36200]: scrub summary: idle 2024-03-18T16:11:59.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:11:59 smithi194 ceph-mon[36200]: pgmap v479: 129 pgs: 129 active+clean; 5.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 234 MiB/s wr, 97 op/s 2024-03-18T16:11:59.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:11:59 smithi039 ceph-mon[31721]: scrub summary: idle 2024-03-18T16:11:59.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:11:59 smithi039 ceph-mon[31721]: scrub summary: idle 2024-03-18T16:11:59.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:11:59 smithi039 ceph-mon[31721]: pgmap v479: 129 pgs: 129 active+clean; 5.1 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 234 MiB/s wr, 97 op/s 2024-03-18T16:12:01.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:12:01 smithi196 ceph-mon[36307]: pgmap v480: 129 pgs: 129 active+clean; 5.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 229 MiB/s wr, 89 op/s 2024-03-18T16:12:01.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:12:01 smithi194 ceph-mon[36200]: pgmap v480: 129 pgs: 129 active+clean; 5.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 229 MiB/s wr, 89 op/s 2024-03-18T16:12:01.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:12:01 smithi039 ceph-mon[31721]: pgmap v480: 129 pgs: 129 active+clean; 5.6 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 229 MiB/s wr, 89 op/s 2024-03-18T16:12:03.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:12:03 smithi196 ceph-mon[36307]: pgmap v481: 129 pgs: 129 active+clean; 5.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 211 MiB/s wr, 81 op/s 2024-03-18T16:12:03.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:12:03 smithi194 ceph-mon[36200]: pgmap v481: 129 pgs: 129 active+clean; 5.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 211 MiB/s wr, 81 op/s 2024-03-18T16:12:03.589 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:12:03 smithi039 ceph-mon[31721]: pgmap v481: 129 pgs: 129 active+clean; 5.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 211 MiB/s wr, 81 op/s 2024-03-18T16:12:05.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:12:05 smithi196 ceph-mon[36307]: pgmap v482: 129 pgs: 129 active+clean; 6.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 253 MiB/s wr, 101 op/s 2024-03-18T16:12:05.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:12:05 smithi194 ceph-mon[36200]: pgmap v482: 129 pgs: 129 active+clean; 6.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 253 MiB/s wr, 101 op/s 2024-03-18T16:12:05.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:12:05 smithi039 ceph-mon[31721]: pgmap v482: 129 pgs: 129 active+clean; 6.8 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 253 MiB/s wr, 101 op/s 2024-03-18T16:12:07.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:12:07 smithi196 ceph-mon[36307]: pgmap v483: 129 pgs: 129 active+clean; 7.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 231 MiB/s wr, 88 op/s 2024-03-18T16:12:07.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:12:07 smithi194 ceph-mon[36200]: pgmap v483: 129 pgs: 129 active+clean; 7.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 231 MiB/s wr, 88 op/s 2024-03-18T16:12:07.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:12:07 smithi039 ceph-mon[31721]: pgmap v483: 129 pgs: 129 active+clean; 7.1 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 231 MiB/s wr, 88 op/s 2024-03-18T16:12:09.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:12:09 smithi196 ceph-mon[36307]: pgmap v484: 129 pgs: 129 active+clean; 7.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 243 MiB/s wr, 99 op/s 2024-03-18T16:12:09.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:12:09 smithi194 ceph-mon[36200]: pgmap v484: 129 pgs: 129 active+clean; 7.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 243 MiB/s wr, 99 op/s 2024-03-18T16:12:09.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:12:09 smithi039 ceph-mon[31721]: pgmap v484: 129 pgs: 129 active+clean; 7.5 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 73 KiB/s rd, 243 MiB/s wr, 99 op/s 2024-03-18T16:12:11.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:12:11 smithi196 ceph-mon[36307]: pgmap v485: 129 pgs: 129 active+clean; 8.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 241 MiB/s wr, 91 op/s 2024-03-18T16:12:11.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:12:11 smithi194 ceph-mon[36200]: pgmap v485: 129 pgs: 129 active+clean; 8.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 241 MiB/s wr, 91 op/s 2024-03-18T16:12:11.589 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:12:11 smithi039 ceph-mon[31721]: pgmap v485: 129 pgs: 129 active+clean; 8.0 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 241 MiB/s wr, 91 op/s 2024-03-18T16:12:13.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:12:13 smithi196 ceph-mon[36307]: pgmap v486: 129 pgs: 129 active+clean; 8.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 240 MiB/s wr, 88 op/s 2024-03-18T16:12:13.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:12:13 smithi194 ceph-mon[36200]: pgmap v486: 129 pgs: 129 active+clean; 8.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 240 MiB/s wr, 88 op/s 2024-03-18T16:12:13.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:12:13 smithi039 ceph-mon[31721]: pgmap v486: 129 pgs: 129 active+clean; 8.4 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 240 MiB/s wr, 88 op/s 2024-03-18T16:12:14.702 INFO:tasks.workunit.client.0.smithi039.stdout:fs setup took 48 secs 2024-03-18T16:12:15.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:12:15 smithi196 ceph-mon[36307]: pgmap v487: 129 pgs: 129 active+clean; 9.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 273 MiB/s wr, 108 op/s 2024-03-18T16:12:15.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:12:15 smithi194 ceph-mon[36200]: pgmap v487: 129 pgs: 129 active+clean; 9.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 273 MiB/s wr, 108 op/s 2024-03-18T16:12:15.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:12:15 smithi039 ceph-mon[31721]: pgmap v487: 129 pgs: 129 active+clean; 9.1 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 273 MiB/s wr, 108 op/s 2024-03-18T16:12:17.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:12:17 smithi196 ceph-mon[36307]: pgmap v488: 129 pgs: 129 active+clean; 9.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 242 MiB/s wr, 94 op/s 2024-03-18T16:12:17.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:12:17 smithi194 ceph-mon[36200]: pgmap v488: 129 pgs: 129 active+clean; 9.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 242 MiB/s wr, 94 op/s 2024-03-18T16:12:17.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:12:17 smithi039 ceph-mon[31721]: pgmap v488: 129 pgs: 129 active+clean; 9.6 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 242 MiB/s wr, 94 op/s 2024-03-18T16:12:19.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:12:19 smithi196 ceph-mon[36307]: pgmap v489: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 267 MiB/s wr, 103 op/s 2024-03-18T16:12:19.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:12:19 smithi194 ceph-mon[36200]: pgmap v489: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 267 MiB/s wr, 103 op/s 2024-03-18T16:12:19.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:12:19 smithi039 ceph-mon[31721]: pgmap v489: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 267 MiB/s wr, 103 op/s 2024-03-18T16:12:19.829 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T16:12:19.830 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T16:12:20.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.085+0000 7ff385e00640 1 -- 172.21.15.39:0/2239269137 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff380100070 msgr2=0x7ff3801024a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:20.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.085+0000 7ff385e00640 1 --2- 172.21.15.39:0/2239269137 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff380100070 0x7ff3801024a0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7ff370009900 tx=0x7ff37002bc30 comp rx=0 tx=0).stop 2024-03-18T16:12:20.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.086+0000 7ff385e00640 1 -- 172.21.15.39:0/2239269137 shutdown_connections 2024-03-18T16:12:20.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.086+0000 7ff385e00640 1 --2- 172.21.15.39:0/2239269137 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff380105390 0x7ff380105810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:20.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.086+0000 7ff385e00640 1 --2- 172.21.15.39:0/2239269137 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff3801029e0 0x7ff380104e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:20.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.086+0000 7ff385e00640 1 --2- 172.21.15.39:0/2239269137 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff380100070 0x7ff3801024a0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:20.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.086+0000 7ff385e00640 1 -- 172.21.15.39:0/2239269137 >> 172.21.15.39:0/2239269137 conn(0x7ff3800fbc70 msgr2=0x7ff3800fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:20.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.086+0000 7ff385e00640 1 -- 172.21.15.39:0/2239269137 shutdown_connections 2024-03-18T16:12:20.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.086+0000 7ff385e00640 1 -- 172.21.15.39:0/2239269137 wait complete. 2024-03-18T16:12:20.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.087+0000 7ff385e00640 1 Processor -- start 2024-03-18T16:12:20.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.087+0000 7ff385e00640 1 -- start start 2024-03-18T16:12:20.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.088+0000 7ff385e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff380100070 0x7ff380070220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:20.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.088+0000 7ff385e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff3801029e0 0x7ff380070760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:20.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.088+0000 7ff385e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff380105390 0x7ff3801ae220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:20.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.088+0000 7ff385e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7ff380113b10 con 0x7ff380105390 2024-03-18T16:12:20.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.088+0000 7ff385e00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ff380113990 con 0x7ff380100070 2024-03-18T16:12:20.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.088+0000 7ff385e00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7ff380113c90 con 0x7ff3801029e0 2024-03-18T16:12:20.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.088+0000 7ff37f400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff380105390 0x7ff3801ae220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:20.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.088+0000 7ff37ea00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff380100070 0x7ff380070220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:20.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.088+0000 7ff37ea00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff380100070 0x7ff380070220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:55030/0 (socket says 172.21.15.39:55030) 2024-03-18T16:12:20.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.088+0000 7ff37f400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff380105390 0x7ff3801ae220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:41834/0 (socket says 172.21.15.39:41834) 2024-03-18T16:12:20.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.088+0000 7ff37ea00640 1 -- 172.21.15.39:0/1025344071 learned_addr learned my addr 172.21.15.39:0/1025344071 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:12:20.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.089+0000 7ff37e000640 1 --2- 172.21.15.39:0/1025344071 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff3801029e0 0x7ff380070760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:20.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.089+0000 7ff37ea00640 1 -- 172.21.15.39:0/1025344071 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff3801029e0 msgr2=0x7ff380070760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:20.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.089+0000 7ff37ea00640 1 --2- 172.21.15.39:0/1025344071 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff3801029e0 0x7ff380070760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:20.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.089+0000 7ff37ea00640 1 -- 172.21.15.39:0/1025344071 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff380105390 msgr2=0x7ff3801ae220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:20.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.089+0000 7ff37ea00640 1 --2- 172.21.15.39:0/1025344071 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff380105390 0x7ff3801ae220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:20.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.089+0000 7ff37ea00640 1 -- 172.21.15.39:0/1025344071 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3801ae760 con 0x7ff380100070 2024-03-18T16:12:20.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.090+0000 7ff37ea00640 1 --2- 172.21.15.39:0/1025344071 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff380100070 0x7ff380070220 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7ff37002c240 tx=0x7ff3700048f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:20.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.091+0000 7ff384a00640 1 -- 172.21.15.39:0/1025344071 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff370043020 con 0x7ff380100070 2024-03-18T16:12:20.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.091+0000 7ff384a00640 1 -- 172.21.15.39:0/1025344071 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff3700054e0 con 0x7ff380100070 2024-03-18T16:12:20.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.091+0000 7ff384a00640 1 -- 172.21.15.39:0/1025344071 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff370003b20 con 0x7ff380100070 2024-03-18T16:12:20.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.091+0000 7ff385e00640 1 -- 172.21.15.39:0/1025344071 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3801ae930 con 0x7ff380100070 2024-03-18T16:12:20.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.091+0000 7ff385e00640 1 -- 172.21.15.39:0/1025344071 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3801aedd0 con 0x7ff380100070 2024-03-18T16:12:20.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.098+0000 7ff388bce740 1 Processor -- start 2024-03-18T16:12:20.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.099+0000 7ff388bce740 1 -- start start 2024-03-18T16:12:20.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.099+0000 7ff388bce740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f713bb3810 0x55f713bb3c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:20.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.100+0000 7ff388bce740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f713bb4200 0x55f713bb8d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:20.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.100+0000 7ff37f400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f713bb3810 0x55f713bb3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:20.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.100+0000 7ff37f400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f713bb3810 0x55f713bb3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:54366/0 (socket says 172.21.15.39:54366) 2024-03-18T16:12:20.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.100+0000 7ff388bce740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f713bb9370 0x55f713bbb800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:20.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.100+0000 7ff388bce740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55f713b181f0 con 0x55f713bb9370 2024-03-18T16:12:20.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.100+0000 7ff388bce740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55f713ac0a80 con 0x55f713bb4200 2024-03-18T16:12:20.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.100+0000 7ff388bce740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55f713b14290 con 0x55f713bb3810 2024-03-18T16:12:20.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.100+0000 7ff37f400640 1 -- 172.21.15.39:0/2507463902 learned_addr learned my addr 172.21.15.39:0/2507463902 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:12:20.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.100+0000 7ff37ea00640 1 --2- 172.21.15.39:0/2507463902 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f713bb4200 0x55f713bb8d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:20.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.100+0000 7ff384a00640 1 -- 172.21.15.39:0/1025344071 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7ff370046020 con 0x7ff380100070 2024-03-18T16:12:20.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.101+0000 7ff37f400640 1 -- 172.21.15.39:0/2507463902 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f713bb4200 msgr2=0x55f713bb8d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:12:20.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.101+0000 7ff37f400640 1 --2- 172.21.15.39:0/2507463902 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f713bb4200 0x55f713bb8d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:20.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.101+0000 7ff37f400640 1 -- 172.21.15.39:0/2507463902 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f713bb9370 msgr2=0x55f713bbb800 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:12:20.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.101+0000 7ff37e000640 1 --2- 172.21.15.39:0/2507463902 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f713bb9370 0x55f713bbb800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:20.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.101+0000 7ff37f400640 1 --2- 172.21.15.39:0/2507463902 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f713bb9370 0x55f713bbb800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:20.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.101+0000 7ff37f400640 1 -- 172.21.15.39:0/2507463902 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f7138f67d0 con 0x55f713bb3810 2024-03-18T16:12:20.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.101+0000 7ff37e000640 1 --2- 172.21.15.39:0/2507463902 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f713bb9370 0x55f713bbb800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:12:20.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.102+0000 7ff37f400640 1 --2- 172.21.15.39:0/2507463902 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f713bb3810 0x55f713bb3c30 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7ff378011440 tx=0x7ff378011900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=28834144e057d9cd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:20.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.102+0000 7ff36be00640 1 -- 172.21.15.39:0/2507463902 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff378003c30 con 0x55f713bb3810 2024-03-18T16:12:20.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.102+0000 7ff36be00640 1 -- 172.21.15.39:0/2507463902 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff378003dd0 con 0x55f713bb3810 2024-03-18T16:12:20.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.102+0000 7ff36be00640 1 -- 172.21.15.39:0/2507463902 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff37800fb50 con 0x55f713bb3810 2024-03-18T16:12:20.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.104+0000 7ff388bce740 1 -- 172.21.15.39:0/2507463902 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f713bb3810 msgr2=0x55f713bb3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:20.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.104+0000 7ff388bce740 1 --2- 172.21.15.39:0/2507463902 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f713bb3810 0x55f713bb3c30 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7ff378011440 tx=0x7ff378011900 comp rx=0 tx=0).stop 2024-03-18T16:12:20.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.104+0000 7ff388bce740 1 -- 172.21.15.39:0/2507463902 shutdown_connections 2024-03-18T16:12:20.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.104+0000 7ff388bce740 1 --2- 172.21.15.39:0/2507463902 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f713bb4200 0x55f713bb8d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:20.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.104+0000 7ff388bce740 1 --2- 172.21.15.39:0/2507463902 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f713bb9370 0x55f713bbb800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:20.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.104+0000 7ff388bce740 1 --2- 172.21.15.39:0/2507463902 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f713bb3810 0x55f713bb3c30 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:20.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.104+0000 7ff388bce740 1 -- 172.21.15.39:0/2507463902 >> 172.21.15.39:0/2507463902 conn(0x55f713baed80 msgr2=0x55f713baf180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:20.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.104+0000 7ff388bce740 1 -- 172.21.15.39:0/2507463902 shutdown_connections 2024-03-18T16:12:20.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.104+0000 7ff388bce740 1 -- 172.21.15.39:0/2507463902 wait complete. 2024-03-18T16:12:20.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.105+0000 7ff388bce740 1 -- start start 2024-03-18T16:12:20.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.105+0000 7ff388bce740 1 Processor -- start 2024-03-18T16:12:20.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.105+0000 7ff384a00640 1 --2- 172.21.15.39:0/1025344071 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff3640a2860 0x7ff3640a4d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:20.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.106+0000 7ff388bce740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f713bb3810 0x55f713bbde30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:20.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.106+0000 7ff388bce740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f713bb4200 0x55f713bbe370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:20.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.106+0000 7ff388bce740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f713bb9370 0x55f713bc53b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:20.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.106+0000 7ff388bce740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55f713bbc320 con 0x55f713bb3810 2024-03-18T16:12:20.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.106+0000 7ff388bce740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55f713ad3ed0 con 0x55f713bb4200 2024-03-18T16:12:20.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.106+0000 7ff388bce740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55f713bbc4a0 con 0x55f713bb9370 2024-03-18T16:12:20.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.106+0000 7ff388bce740 10 client.0 _ll_register_callbacks cb 0x55f713b0f220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:12:20.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.106+0000 7ff384a00640 1 -- 172.21.15.39:0/1025344071 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7ff3700f0e30 con 0x7ff380100070 2024-03-18T16:12:20.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.106+0000 7ff37ea00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f713bb3810 0x55f713bbde30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:20.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.107+0000 7ff37ea00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f713bb3810 0x55f713bbde30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:41852/0 (socket says 172.21.15.39:41852) 2024-03-18T16:12:20.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.107+0000 7ff37ea00640 1 -- 172.21.15.39:0/636319284 learned_addr learned my addr 172.21.15.39:0/636319284 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:12:20.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.107+0000 7ff37e000640 1 --2- 172.21.15.39:0/636319284 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f713bb4200 0x55f713bbe370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:20.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.107+0000 7ff37ea00640 1 -- 172.21.15.39:0/636319284 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f713bb9370 msgr2=0x55f713bc53b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T16:12:20.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.107+0000 7ff37ea00640 1 --2- 172.21.15.39:0/636319284 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f713bb9370 0x55f713bc53b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:20.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.107+0000 7ff37ea00640 1 -- 172.21.15.39:0/636319284 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f713bb4200 msgr2=0x55f713bbe370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:12:20.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.107+0000 7ff37ea00640 1 --2- 172.21.15.39:0/636319284 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f713bb4200 0x55f713bbe370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:20.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.107+0000 7ff37f400640 1 --2- 172.21.15.39:0/1025344071 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff3640a2860 0x7ff3640a4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:20.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.107+0000 7ff37ea00640 1 -- 172.21.15.39:0/636319284 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f71382c3f0 con 0x55f713bb3810 2024-03-18T16:12:20.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.108+0000 7ff37ea00640 1 --2- 172.21.15.39:0/636319284 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f713bb3810 0x55f713bbde30 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff3700bd0e0 tx=0x7ff370005cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9be79341114d80e6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:20.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.108+0000 7ff368c00640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-03-18T16:12:20.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.108+0000 7ff368c00640 1 -- 172.21.15.39:0/636319284 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3700b9680 con 0x55f713bb3810 2024-03-18T16:12:20.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.108+0000 7ff368c00640 1 -- 172.21.15.39:0/636319284 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7ff3700042e0 con 0x55f713bb3810 2024-03-18T16:12:20.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.108+0000 7ff37f400640 1 --2- 172.21.15.39:0/1025344071 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff3640a2860 0x7ff3640a4d20 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7ff378009df0 tx=0x7ff378005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:20.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.109+0000 7ff388bce740 1 -- 172.21.15.39:0/636319284 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff378003dd0 con 0x55f713bb3810 2024-03-18T16:12:20.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.109+0000 7ff368c00640 1 -- 172.21.15.39:0/636319284 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7ff3700bcd40 con 0x55f713bb3810 2024-03-18T16:12:20.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.109+0000 7ff388bce740 10 client.15825 fetch_fsmap learned FSMap version 32 2024-03-18T16:12:20.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.109+0000 7ff388bce740 1 -- 172.21.15.39:0/636319284 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55f7137ec0f0 con 0x55f713bb3810 2024-03-18T16:12:20.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.110+0000 7ff368c00640 1 -- 172.21.15.39:0/636319284 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff378003dd0 con 0x55f713bb3810 2024-03-18T16:12:20.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.110+0000 7ff368c00640 1 -- 172.21.15.39:0/636319284 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7ff3700bd640 con 0x55f713bb3810 2024-03-18T16:12:20.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.110+0000 7ff388bce740 10 client.15825 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T16:12:20.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.110+0000 7ff388bce740 10 client.15825 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:12:20.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.110+0000 7ff388bce740 20 client.15825 populate_metadata read hostname 'smithi039' 2024-03-18T16:12:20.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.110+0000 7ff388bce740 1 --2- 172.21.15.39:0/636319284 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55f713bc9b20 0x55f713bcbf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:20.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.111+0000 7ff388bce740 4 client.15825 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:12:20.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.111+0000 7ff388bce740 1 -- 172.21.15.39:0/636319284 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f713b2bbe0 con 0x55f713bc9b20 2024-03-18T16:12:20.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.111+0000 7ff37e000640 1 --2- 172.21.15.39:0/636319284 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55f713bc9b20 0x55f713bcbf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:20.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.111+0000 7ff37e000640 1 --2- 172.21.15.39:0/636319284 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55f713bc9b20 0x55f713bcbf80 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7ec29888384fbb3a server_cookie=14d6bfa6ea26324d in_seq=0 out_seq=0 2024-03-18T16:12:20.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.111+0000 7ff368c00640 10 client.15825 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:12:20.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.112+0000 7ff368c00640 1 -- 172.21.15.39:0/636319284 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f713b2bbe0 con 0x55f713bc9b20 2024-03-18T16:12:20.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.112+0000 7ff368c00640 10 client.15825 handle_command_reply: tid=0 2024-03-18T16:12:20.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.112+0000 7ff388bce740 1 client.15825 shutdown 2024-03-18T16:12:20.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.113+0000 7ff388bce740 1 -- 172.21.15.39:0/636319284 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f713bb3810 msgr2=0x55f713bbde30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:20.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.113+0000 7ff388bce740 1 --2- 172.21.15.39:0/636319284 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f713bb3810 0x55f713bbde30 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff3700bd0e0 tx=0x7ff370005cb0 comp rx=0 tx=0).stop 2024-03-18T16:12:20.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.113+0000 7ff388bce740 1 -- 172.21.15.39:0/636319284 shutdown_connections 2024-03-18T16:12:20.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.113+0000 7ff388bce740 1 --2- 172.21.15.39:0/636319284 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55f713bc9b20 0x55f713bcbf80 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:20.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.113+0000 7ff388bce740 1 --2- 172.21.15.39:0/636319284 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f713bb4200 0x55f713bbe370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:20.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.113+0000 7ff388bce740 1 --2- 172.21.15.39:0/636319284 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f713bb9370 0x55f713bc53b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:20.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.113+0000 7ff388bce740 1 --2- 172.21.15.39:0/636319284 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f713bb3810 0x55f713bbde30 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:20.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.113+0000 7ff388bce740 1 -- 172.21.15.39:0/636319284 >> 172.21.15.39:0/636319284 conn(0x55f713baed80 msgr2=0x55f713badec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:20.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.113+0000 7ff368c00640 0 client.15825 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:12:20.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.113+0000 7ff388bce740 1 -- 172.21.15.39:0/636319284 shutdown_connections 2024-03-18T16:12:20.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.113+0000 7ff388bce740 1 -- 172.21.15.39:0/636319284 wait complete. 2024-03-18T16:12:20.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.113+0000 7ff388bce740 20 client.15825 trim_cache size 0 max 16384 2024-03-18T16:12:20.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.182+0000 7ff388bce740 1 Processor -- start 2024-03-18T16:12:20.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.183+0000 7ff388bce740 1 -- start start 2024-03-18T16:12:20.189 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.187+0000 7ff388bce740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f713bb3810 0x55f713bb5790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:20.189 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.187+0000 7ff388bce740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f713bb4200 0x55f713bb5cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:20.189 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.187+0000 7ff388bce740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f713bb9370 0x55f713bc5340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:20.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.188+0000 7ff37e000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f713bb4200 0x55f713bb5cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:20.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.188+0000 7ff37e000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f713bb4200 0x55f713bb5cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:54390/0 (socket says 172.21.15.39:54390) 2024-03-18T16:12:20.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.188+0000 7ff37ea00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f713bb3810 0x55f713bb5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:20.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.188+0000 7ff37ea00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f713bb3810 0x55f713bb5790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:55050/0 (socket says 172.21.15.39:55050) 2024-03-18T16:12:20.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.193+0000 7ff388bce740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7ff3801b93a0 con 0x55f713bb9370 2024-03-18T16:12:20.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.193+0000 7ff37e000640 1 -- 172.21.15.39:0/154835552 learned_addr learned my addr 172.21.15.39:0/154835552 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:12:20.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.193+0000 7ff388bce740 1 -- 172.21.15.39:0/154835552 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ff3801bafa0 con 0x55f713bb3810 2024-03-18T16:12:20.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.193+0000 7ff388bce740 1 -- 172.21.15.39:0/154835552 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7ff3801b9a70 con 0x55f713bb4200 2024-03-18T16:12:20.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.193+0000 7ff37f400640 1 --2- 172.21.15.39:0/154835552 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f713bb9370 0x55f713bc5340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:20.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.195+0000 7ff37ea00640 1 -- 172.21.15.39:0/154835552 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f713bb4200 msgr2=0x55f713bb5cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:12:20.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.195+0000 7ff37ea00640 1 --2- 172.21.15.39:0/154835552 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f713bb4200 0x55f713bb5cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:20.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.195+0000 7ff37ea00640 1 -- 172.21.15.39:0/154835552 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f713bb9370 msgr2=0x55f713bc5340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:12:20.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.195+0000 7ff37ea00640 1 --2- 172.21.15.39:0/154835552 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f713bb9370 0x55f713bc5340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:20.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.195+0000 7ff37ea00640 1 -- 172.21.15.39:0/154835552 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f713ba9370 con 0x55f713bb3810 2024-03-18T16:12:20.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.196+0000 7ff37ea00640 1 --2- 172.21.15.39:0/154835552 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f713bb3810 0x55f713bb5790 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7ff3700b9840 tx=0x7ff370004250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=facecba451ffde74 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:20.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.199+0000 7ff36b400640 1 -- 172.21.15.39:0/154835552 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3700db070 con 0x55f713bb3810 2024-03-18T16:12:20.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.199+0000 7ff36b400640 1 -- 172.21.15.39:0/154835552 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff370005690 con 0x55f713bb3810 2024-03-18T16:12:20.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.199+0000 7ff36b400640 1 -- 172.21.15.39:0/154835552 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3700bcf20 con 0x55f713bb3810 2024-03-18T16:12:20.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.201+0000 7ff388bce740 1 -- 172.21.15.39:0/154835552 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f713bb3810 msgr2=0x55f713bb5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:20.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.201+0000 7ff388bce740 1 --2- 172.21.15.39:0/154835552 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f713bb3810 0x55f713bb5790 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7ff3700b9840 tx=0x7ff370004250 comp rx=0 tx=0).stop 2024-03-18T16:12:20.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.201+0000 7ff388bce740 1 -- 172.21.15.39:0/154835552 shutdown_connections 2024-03-18T16:12:20.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.201+0000 7ff388bce740 1 --2- 172.21.15.39:0/154835552 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f713bb9370 0x55f713bc5340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:20.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.201+0000 7ff388bce740 1 --2- 172.21.15.39:0/154835552 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f713bb4200 0x55f713bb5cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:20.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.201+0000 7ff388bce740 1 --2- 172.21.15.39:0/154835552 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f713bb3810 0x55f713bb5790 secure :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7ff3700b9840 tx=0x7ff370004250 comp rx=0 tx=0).stop 2024-03-18T16:12:20.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.201+0000 7ff388bce740 1 -- 172.21.15.39:0/154835552 >> 172.21.15.39:0/154835552 conn(0x55f713baed80 msgr2=0x55f713baf160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:20.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.203+0000 7ff388bce740 1 -- 172.21.15.39:0/154835552 shutdown_connections 2024-03-18T16:12:20.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.204+0000 7ff388bce740 1 -- 172.21.15.39:0/154835552 wait complete. 2024-03-18T16:12:20.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.204+0000 7ff388bce740 1 -- start start 2024-03-18T16:12:20.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.205+0000 7ff388bce740 1 Processor -- start 2024-03-18T16:12:20.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.208+0000 7ff388bce740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f713bb4200 0x55f713bc28b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:20.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.209+0000 7ff388bce740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f713bb9370 0x55f713bc2df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:20.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.209+0000 7ff388bce740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f713bc3e00 0x55f713bc4280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:20.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.210+0000 7ff37ea00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f713bb4200 0x55f713bc28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:20.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.210+0000 7ff37ea00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f713bb4200 0x55f713bc28b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:55062/0 (socket says 172.21.15.39:55062) 2024-03-18T16:12:20.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.210+0000 7ff37e000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f713bb9370 0x55f713bc2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:20.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.210+0000 7ff37e000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f713bb9370 0x55f713bc2df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:54404/0 (socket says 172.21.15.39:54404) 2024-03-18T16:12:20.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.213+0000 7ff388bce740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7ff3801ba870 con 0x55f713bc3e00 2024-03-18T16:12:20.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.213+0000 7ff388bce740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ff3801ba140 con 0x55f713bb4200 2024-03-18T16:12:20.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.213+0000 7ff388bce740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7ff3801b7380 con 0x55f713bb9370 2024-03-18T16:12:20.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.213+0000 7ff388bce740 10 client.0 _ll_register_callbacks cb 0x7ff3700b7f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:12:20.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.213+0000 7ff37ea00640 1 -- 172.21.15.39:0/763984325 learned_addr learned my addr 172.21.15.39:0/763984325 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:12:20.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.214+0000 7ff37f400640 1 --2- 172.21.15.39:0/763984325 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f713bc3e00 0x55f713bc4280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:20.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.216+0000 7ff37ea00640 1 -- 172.21.15.39:0/763984325 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f713bb9370 msgr2=0x55f713bc2df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:12:20.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.216+0000 7ff37ea00640 1 --2- 172.21.15.39:0/763984325 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f713bb9370 0x55f713bc2df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:20.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.216+0000 7ff37ea00640 1 -- 172.21.15.39:0/763984325 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f713bc3e00 msgr2=0x55f713bc4280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:12:20.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.216+0000 7ff37ea00640 1 --2- 172.21.15.39:0/763984325 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f713bc3e00 0x55f713bc4280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:20.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.217+0000 7ff37ea00640 1 -- 172.21.15.39:0/763984325 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f713bc47c0 con 0x55f713bb4200 2024-03-18T16:12:20.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.217+0000 7ff37f400640 1 --2- 172.21.15.39:0/763984325 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f713bc3e00 0x55f713bc4280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:12:20.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.219+0000 7ff37ea00640 1 --2- 172.21.15.39:0/763984325 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f713bb4200 0x55f713bc28b0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7ff3700bb690 tx=0x7ff370005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53acc70d99ceced4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:20.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.219+0000 7ff368c00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:12:20.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.221+0000 7ff368c00640 1 -- 172.21.15.39:0/763984325 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3700db040 con 0x55f713bb4200 2024-03-18T16:12:20.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.223+0000 7ff368c00640 1 -- 172.21.15.39:0/763984325 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7ff3700c10f0 con 0x55f713bb4200 2024-03-18T16:12:20.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.224+0000 7ff388bce740 1 -- 172.21.15.39:0/763984325 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f713b25730 con 0x55f713bb4200 2024-03-18T16:12:20.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.224+0000 7ff368c00640 1 -- 172.21.15.39:0/763984325 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7ff3700bd1b0 con 0x55f713bb4200 2024-03-18T16:12:20.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.225+0000 7ff388bce740 10 client.26030 fetch_fsmap learned FSMap version 32 2024-03-18T16:12:20.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.226+0000 7ff388bce740 1 -- 172.21.15.39:0/763984325 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55f713bc47c0 con 0x55f713bb4200 2024-03-18T16:12:20.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.226+0000 7ff368c00640 1 -- 172.21.15.39:0/763984325 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55f713b25730 con 0x55f713bb4200 2024-03-18T16:12:20.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.226+0000 7ff368c00640 1 -- 172.21.15.39:0/763984325 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7ff3700c56f0 con 0x55f713bb4200 2024-03-18T16:12:20.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.227+0000 7ff388bce740 10 client.26030 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T16:12:20.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.227+0000 7ff388bce740 10 client.26030 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:12:20.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.227+0000 7ff388bce740 20 client.26030 populate_metadata read hostname 'smithi039' 2024-03-18T16:12:20.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.227+0000 7ff388bce740 1 --2- 172.21.15.39:0/763984325 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55f713c30680 0x55f713c32aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:20.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.227+0000 7ff388bce740 4 client.26030 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T16:12:20.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.227+0000 7ff388bce740 1 -- 172.21.15.39:0/763984325 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f713b1fa80 con 0x55f713c30680 2024-03-18T16:12:20.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.228+0000 7ff37e000640 1 --2- 172.21.15.39:0/763984325 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55f713c30680 0x55f713c32aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:20.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.234+0000 7ff368c00640 10 client.26030 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:12:20.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.234+0000 7ff37e000640 1 --2- 172.21.15.39:0/763984325 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55f713c30680 0x55f713c32aa0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7afc041702f2ba0b server_cookie=7130228f599b2854 in_seq=0 out_seq=0 2024-03-18T16:12:20.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.234+0000 7ff368c00640 1 -- 172.21.15.39:0/763984325 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55f713b1fa80 con 0x55f713c30680 2024-03-18T16:12:20.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.234+0000 7ff368c00640 10 client.26030 handle_command_reply: tid=0 2024-03-18T16:12:20.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.234+0000 7ff388bce740 1 client.26030 shutdown 2024-03-18T16:12:20.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.235+0000 7ff388bce740 1 -- 172.21.15.39:0/763984325 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f713bb4200 msgr2=0x55f713bc28b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:20.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.235+0000 7ff388bce740 1 --2- 172.21.15.39:0/763984325 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f713bb4200 0x55f713bc28b0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7ff3700bb690 tx=0x7ff370005fd0 comp rx=0 tx=0).stop 2024-03-18T16:12:20.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.235+0000 7ff388bce740 1 -- 172.21.15.39:0/763984325 shutdown_connections 2024-03-18T16:12:20.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.235+0000 7ff388bce740 1 --2- 172.21.15.39:0/763984325 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55f713c30680 0x55f713c32aa0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:20.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.235+0000 7ff388bce740 1 --2- 172.21.15.39:0/763984325 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f713bc3e00 0x55f713bc4280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:20.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.235+0000 7ff368c00640 0 client.26030 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:12:20.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.235+0000 7ff388bce740 1 --2- 172.21.15.39:0/763984325 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f713bb9370 0x55f713bc2df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:20.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.235+0000 7ff388bce740 1 --2- 172.21.15.39:0/763984325 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f713bb4200 0x55f713bc28b0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:20.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.235+0000 7ff388bce740 1 -- 172.21.15.39:0/763984325 >> 172.21.15.39:0/763984325 conn(0x55f713baed80 msgr2=0x55f713bb5790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:20.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.236+0000 7ff388bce740 1 -- 172.21.15.39:0/763984325 shutdown_connections 2024-03-18T16:12:20.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.236+0000 7ff388bce740 1 -- 172.21.15.39:0/763984325 wait complete. 2024-03-18T16:12:20.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.236+0000 7ff388bce740 20 client.26030 trim_cache size 0 max 16384 2024-03-18T16:12:20.238 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-03-18T16:12:20.238 INFO:teuthology.orchestra.run.smithi039.stdout: "status": "no active scrubs running", 2024-03-18T16:12:20.239 INFO:teuthology.orchestra.run.smithi039.stdout: "scrubs": {} 2024-03-18T16:12:20.239 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-03-18T16:12:20.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.238+0000 7ff385e00640 1 -- 172.21.15.39:0/1025344071 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff3640a2860 msgr2=0x7ff3640a4d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:20.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.238+0000 7ff385e00640 1 --2- 172.21.15.39:0/1025344071 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff3640a2860 0x7ff3640a4d20 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7ff378009df0 tx=0x7ff378005fd0 comp rx=0 tx=0).stop 2024-03-18T16:12:20.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.238+0000 7ff385e00640 1 -- 172.21.15.39:0/1025344071 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff380100070 msgr2=0x7ff380070220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:20.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.238+0000 7ff385e00640 1 --2- 172.21.15.39:0/1025344071 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff380100070 0x7ff380070220 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7ff37002c240 tx=0x7ff3700048f0 comp rx=0 tx=0).stop 2024-03-18T16:12:20.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.238+0000 7ff385e00640 1 -- 172.21.15.39:0/1025344071 shutdown_connections 2024-03-18T16:12:20.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.238+0000 7ff385e00640 1 --2- 172.21.15.39:0/1025344071 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff3640a2860 0x7ff3640a4d20 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:20.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.238+0000 7ff385e00640 1 --2- 172.21.15.39:0/1025344071 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff380105390 0x7ff3801ae220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:20.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.238+0000 7ff385e00640 1 --2- 172.21.15.39:0/1025344071 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff3801029e0 0x7ff380070760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:20.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.239+0000 7ff385e00640 1 --2- 172.21.15.39:0/1025344071 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff380100070 0x7ff380070220 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:20.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.239+0000 7ff385e00640 1 -- 172.21.15.39:0/1025344071 >> 172.21.15.39:0/1025344071 conn(0x7ff3800fbc70 msgr2=0x7ff380103460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:20.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.239+0000 7ff385e00640 1 -- 172.21.15.39:0/1025344071 shutdown_connections 2024-03-18T16:12:20.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.239+0000 7ff385e00640 1 -- 172.21.15.39:0/1025344071 wait complete. 2024-03-18T16:12:20.263 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-18T16:12:20.264 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:12:20.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.560+0000 7f068a600640 1 -- 172.21.15.39:0/4110670250 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0684074240 msgr2=0x7f06840746c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:20.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.560+0000 7f068a600640 1 --2- 172.21.15.39:0/4110670250 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0684074240 0x7f06840746c0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f0670009930 tx=0x7f067002bc20 comp rx=0 tx=0).stop 2024-03-18T16:12:20.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.560+0000 7f068a600640 1 -- 172.21.15.39:0/4110670250 shutdown_connections 2024-03-18T16:12:20.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.560+0000 7f068a600640 1 --2- 172.21.15.39:0/4110670250 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0684074240 0x7f06840746c0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:20.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.560+0000 7f068a600640 1 --2- 172.21.15.39:0/4110670250 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f068410dbc0 0x7f068410ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:20.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.560+0000 7f068a600640 1 --2- 172.21.15.39:0/4110670250 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0684075820 0x7f0684073c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:20.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.560+0000 7f068a600640 1 -- 172.21.15.39:0/4110670250 >> 172.21.15.39:0/4110670250 conn(0x7f06840fcb40 msgr2=0x7f06840fefa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:20.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.560+0000 7f068a600640 1 -- 172.21.15.39:0/4110670250 shutdown_connections 2024-03-18T16:12:20.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.560+0000 7f068a600640 1 -- 172.21.15.39:0/4110670250 wait complete. 2024-03-18T16:12:20.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.562+0000 7f068a600640 1 Processor -- start 2024-03-18T16:12:20.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.562+0000 7f068a600640 1 -- start start 2024-03-18T16:12:20.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.562+0000 7f068a600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0684074240 0x7f068410cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:20.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.562+0000 7f068a600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0684075820 0x7f068410d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:20.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.562+0000 7f068a600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f068410dbc0 0x7f068410b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:20.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.563+0000 7f0683e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0684075820 0x7f068410d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:20.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.563+0000 7f0689200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0684074240 0x7f068410cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:20.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.563+0000 7f0683e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0684075820 0x7f068410d490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:54428/0 (socket says 172.21.15.39:54428) 2024-03-18T16:12:20.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.563+0000 7f0689c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f068410dbc0 0x7f068410b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:20.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.563+0000 7f0689c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f068410dbc0 0x7f068410b590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:55092/0 (socket says 172.21.15.39:55092) 2024-03-18T16:12:20.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.563+0000 7f0689200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0684074240 0x7f068410cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:41882/0 (socket says 172.21.15.39:41882) 2024-03-18T16:12:20.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.563+0000 7f068a600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f0684105e00 con 0x7f0684074240 2024-03-18T16:12:20.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.563+0000 7f068a600640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f0684105c80 con 0x7f068410dbc0 2024-03-18T16:12:20.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.563+0000 7f068a600640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f0684105f80 con 0x7f0684075820 2024-03-18T16:12:20.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.563+0000 7f0683e00640 1 -- 172.21.15.39:0/1382861812 learned_addr learned my addr 172.21.15.39:0/1382861812 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:12:20.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.564+0000 7f0689c00640 1 -- 172.21.15.39:0/1382861812 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0684075820 msgr2=0x7f068410d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:20.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.564+0000 7f0689c00640 1 --2- 172.21.15.39:0/1382861812 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0684075820 0x7f068410d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:20.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.564+0000 7f0689c00640 1 -- 172.21.15.39:0/1382861812 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0684074240 msgr2=0x7f068410cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:20.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.564+0000 7f0689c00640 1 --2- 172.21.15.39:0/1382861812 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0684074240 0x7f068410cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:20.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.564+0000 7f0689c00640 1 -- 172.21.15.39:0/1382861812 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f068410bd90 con 0x7f068410dbc0 2024-03-18T16:12:20.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.564+0000 7f0683e00640 1 --2- 172.21.15.39:0/1382861812 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0684075820 0x7f068410d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:12:20.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.565+0000 7f0689c00640 1 --2- 172.21.15.39:0/1382861812 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f068410dbc0 0x7f068410b590 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f067c00ca30 tx=0x7f067c00cef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:20.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.565+0000 7f0681600640 1 -- 172.21.15.39:0/1382861812 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f067c007d90 con 0x7f068410dbc0 2024-03-18T16:12:20.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.565+0000 7f0681600640 1 -- 172.21.15.39:0/1382861812 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f067c005020 con 0x7f068410dbc0 2024-03-18T16:12:20.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.565+0000 7f0681600640 1 -- 172.21.15.39:0/1382861812 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f067c00f740 con 0x7f068410dbc0 2024-03-18T16:12:20.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.565+0000 7f068a600640 1 -- 172.21.15.39:0/1382861812 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f068410c020 con 0x7f068410dbc0 2024-03-18T16:12:20.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.565+0000 7f068a600640 1 -- 172.21.15.39:0/1382861812 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0684071230 con 0x7f068410dbc0 2024-03-18T16:12:20.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.571+0000 7f0681600640 1 -- 172.21.15.39:0/1382861812 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f067c0051d0 con 0x7f068410dbc0 2024-03-18T16:12:20.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.572+0000 7f068a600640 1 -- 172.21.15.39:0/1382861812 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06840684f0 con 0x7f068410dbc0 2024-03-18T16:12:20.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.573+0000 7f0681600640 1 --2- 172.21.15.39:0/1382861812 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f06680e6910 0x7f06680e8dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:20.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.573+0000 7f0681600640 1 -- 172.21.15.39:0/1382861812 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f067c0cf370 con 0x7f068410dbc0 2024-03-18T16:12:20.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.574+0000 7f0689200640 1 --2- 172.21.15.39:0/1382861812 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f06680e6910 0x7f06680e8dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:20.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.575+0000 7f0689200640 1 --2- 172.21.15.39:0/1382861812 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f06680e6910 0x7f06680e8dd0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f0674009bd0 tx=0x7f0674005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:20.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:20.579+0000 7f0681600640 1 -- 172.21.15.39:0/1382861812 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f067c014070 con 0x7f068410dbc0 2024-03-18T16:12:21.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.129+0000 7f068a600640 1 -- 172.21.15.39:0/1382861812 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0684103d90 con 0x7f068410dbc0 2024-03-18T16:12:21.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.132+0000 7f0681600640 1 -- 172.21.15.39:0/1382861812 <== mon.2 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) ==== 94+0+11938 (secure 0 0 0) 0x7f067c09c020 con 0x7f068410dbc0 2024-03-18T16:12:21.134 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:12:21.134 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2520226608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2520226608},{"type":"v1","addr":"172.21.15.39:6835","nonce":2520226608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/719341763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":719341763},{"type":"v1","addr":"172.21.15.39:6837","nonce":719341763}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":32,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T16:06:44.180425+0000","modified":"2024-03-18T16:11:51.205704+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24575,"mds_1":24556,"mds_2":24535,"mds_3":24524,"mds_4":14463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14445":{"gid":14445,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.39:6839/656513813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":656513813},{"type":"v1","addr":"172.21.15.39:6839","nonce":656513813}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14463":{"gid":14463,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":13,"addr":"172.21.15.39:6841/3515326042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3515326042},{"type":"v1","addr":"172.21.15.39:6841","nonce":3515326042}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6833/931350738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":931350738},{"type":"v1","addr":"172.21.15.194:6833","nonce":931350738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6833/3874209621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":3874209621},{"type":"v1","addr":"172.21.15.196:6833","nonce":3874209621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6835/188385470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":188385470},{"type":"v1","addr":"172.21.15.194:6835","nonce":188385470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6835/3949984309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":3949984309},{"type":"v1","addr":"172.21.15.196:6835","nonce":3949984309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":15,"addr":"172.21.15.194:6837/1301464246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":1301464246},{"type":"v1","addr":"172.21.15.194:6837","nonce":1301464246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":16,"addr":"172.21.15.196:6837/895247251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":895247251},{"type":"v1","addr":"172.21.15.196:6837","nonce":895247251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.196:6839/2001427268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6838","nonce":2001427268},{"type":"v1","addr":"172.21.15.196:6839","nonce":2001427268}]},"join_fscid":-1,"export_targets":[0,3],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24575":{"gid":24575,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.194:6839/274911158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":274911158},{"type":"v1","addr":"172.21.15.194:6839","nonce":274911158}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24575,"qdb_cluster":[14463,24524,24575,24556,24535]},"id":1}]} 2024-03-18T16:12:21.138 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 32 2024-03-18T16:12:21.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.139+0000 7f068a600640 1 -- 172.21.15.39:0/1382861812 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f06680e6910 msgr2=0x7f06680e8dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:21.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.139+0000 7f068a600640 1 --2- 172.21.15.39:0/1382861812 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f06680e6910 0x7f06680e8dd0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f0674009bd0 tx=0x7f0674005f90 comp rx=0 tx=0).stop 2024-03-18T16:12:21.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.139+0000 7f068a600640 1 -- 172.21.15.39:0/1382861812 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f068410dbc0 msgr2=0x7f068410b590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:21.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.139+0000 7f068a600640 1 --2- 172.21.15.39:0/1382861812 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f068410dbc0 0x7f068410b590 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f067c00ca30 tx=0x7f067c00cef0 comp rx=0 tx=0).stop 2024-03-18T16:12:21.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.140+0000 7f068a600640 1 -- 172.21.15.39:0/1382861812 shutdown_connections 2024-03-18T16:12:21.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.140+0000 7f068a600640 1 --2- 172.21.15.39:0/1382861812 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f068410dbc0 0x7f068410b590 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:21.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.140+0000 7f068a600640 1 --2- 172.21.15.39:0/1382861812 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f06680e6910 0x7f06680e8dd0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:21.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.140+0000 7f068a600640 1 --2- 172.21.15.39:0/1382861812 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0684075820 0x7f068410d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:21.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.140+0000 7f068a600640 1 --2- 172.21.15.39:0/1382861812 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0684074240 0x7f068410cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:21.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.140+0000 7f068a600640 1 -- 172.21.15.39:0/1382861812 >> 172.21.15.39:0/1382861812 conn(0x7f06840fcb40 msgr2=0x7f0684100f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:21.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.141+0000 7f068a600640 1 -- 172.21.15.39:0/1382861812 shutdown_connections 2024-03-18T16:12:21.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.141+0000 7f068a600640 1 -- 172.21.15.39:0/1382861812 wait complete. 2024-03-18T16:12:21.182 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-18T16:12:21.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:12:21 smithi196 ceph-mon[36307]: pgmap v490: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 284 MiB/s wr, 100 op/s 2024-03-18T16:12:21.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:12:21 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/1382861812' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:12:21.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:12:21 smithi194 ceph-mon[36200]: pgmap v490: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 284 MiB/s wr, 100 op/s 2024-03-18T16:12:21.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:12:21 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/1382861812' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:12:21.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.565+0000 7f7ac6400640 1 -- 172.21.15.39:0/3569733954 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7ac0108f70 msgr2=0x7f7ac01093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:21.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.565+0000 7f7ac6400640 1 --2- 172.21.15.39:0/3569733954 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7ac0108f70 0x7f7ac01093f0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f7ab0009860 tx=0x7f7ab002bb70 comp rx=0 tx=0).stop 2024-03-18T16:12:21.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.565+0000 7f7ac6400640 1 -- 172.21.15.39:0/3569733954 shutdown_connections 2024-03-18T16:12:21.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.565+0000 7f7ac6400640 1 --2- 172.21.15.39:0/3569733954 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7ac0109930 0x7f7ac0110250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:21.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.565+0000 7f7ac6400640 1 --2- 172.21.15.39:0/3569733954 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7ac0108f70 0x7f7ac01093f0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:21.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.565+0000 7f7ac6400640 1 --2- 172.21.15.39:0/3569733954 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7ac0107e20 0x7f7ac0108240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:21.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.565+0000 7f7ac6400640 1 -- 172.21.15.39:0/3569733954 >> 172.21.15.39:0/3569733954 conn(0x7f7ac0076dd0 msgr2=0x7f7ac00791f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:21.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.565+0000 7f7ac6400640 1 -- 172.21.15.39:0/3569733954 shutdown_connections 2024-03-18T16:12:21.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.565+0000 7f7ac6400640 1 -- 172.21.15.39:0/3569733954 wait complete. 2024-03-18T16:12:21.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.567+0000 7f7ac6400640 1 Processor -- start 2024-03-18T16:12:21.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.567+0000 7f7ac6400640 1 -- start start 2024-03-18T16:12:21.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.567+0000 7f7ac6400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7ac0107e20 0x7f7ac019e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:21.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.567+0000 7f7ac6400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7ac0108f70 0x7f7ac019e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:21.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.567+0000 7f7ac6400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7ac0109930 0x7f7ac01a56c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:21.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.567+0000 7f7ac6400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f7ac0113aa0 con 0x7f7ac0107e20 2024-03-18T16:12:21.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.567+0000 7f7ac6400640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f7ac0113920 con 0x7f7ac0108f70 2024-03-18T16:12:21.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.567+0000 7f7ac6400640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f7ac0113c20 con 0x7f7ac0109930 2024-03-18T16:12:21.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.568+0000 7f7abfe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7ac0109930 0x7f7ac01a56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:21.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.568+0000 7f7abfe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7ac0109930 0x7f7ac01a56c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:54446/0 (socket says 172.21.15.39:54446) 2024-03-18T16:12:21.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.568+0000 7f7abfe00640 1 -- 172.21.15.39:0/1995201264 learned_addr learned my addr 172.21.15.39:0/1995201264 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:12:21.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.568+0000 7f7abf400640 1 --2- 172.21.15.39:0/1995201264 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7ac0107e20 0x7f7ac019e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:21.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.568+0000 7f7abea00640 1 --2- 172.21.15.39:0/1995201264 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7ac0108f70 0x7f7ac019e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:21.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.568+0000 7f7abfe00640 1 -- 172.21.15.39:0/1995201264 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7ac0108f70 msgr2=0x7f7ac019e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:21.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.568+0000 7f7abfe00640 1 --2- 172.21.15.39:0/1995201264 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7ac0108f70 0x7f7ac019e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:21.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.569+0000 7f7abfe00640 1 -- 172.21.15.39:0/1995201264 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7ac0107e20 msgr2=0x7f7ac019e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:21.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.569+0000 7f7abfe00640 1 --2- 172.21.15.39:0/1995201264 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7ac0107e20 0x7f7ac019e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:21.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.569+0000 7f7abfe00640 1 -- 172.21.15.39:0/1995201264 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ac007a130 con 0x7f7ac0109930 2024-03-18T16:12:21.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.569+0000 7f7abea00640 1 --2- 172.21.15.39:0/1995201264 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7ac0108f70 0x7f7ac019e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:12:21.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.570+0000 7f7abfe00640 1 --2- 172.21.15.39:0/1995201264 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7ac0109930 0x7f7ac01a56c0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f7ab800efc0 tx=0x7f7ab800c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:21.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.570+0000 7f7ab7e00640 1 -- 172.21.15.39:0/1995201264 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ab8003970 con 0x7f7ac0109930 2024-03-18T16:12:21.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.570+0000 7f7ab7e00640 1 -- 172.21.15.39:0/1995201264 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ab8004020 con 0x7f7ac0109930 2024-03-18T16:12:21.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.570+0000 7f7ab7e00640 1 -- 172.21.15.39:0/1995201264 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ab800b9f0 con 0x7f7ac0109930 2024-03-18T16:12:21.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.570+0000 7f7ac6400640 1 -- 172.21.15.39:0/1995201264 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ac007a420 con 0x7f7ac0109930 2024-03-18T16:12:21.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.570+0000 7f7ac6400640 1 -- 172.21.15.39:0/1995201264 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ac007a920 con 0x7f7ac0109930 2024-03-18T16:12:21.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.575+0000 7f7ac9048740 1 Processor -- start 2024-03-18T16:12:21.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.577+0000 7f7ab7e00640 1 -- 172.21.15.39:0/1995201264 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f7ab801d020 con 0x7f7ac0109930 2024-03-18T16:12:21.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.577+0000 7f7ac9048740 1 -- start start 2024-03-18T16:12:21.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.578+0000 7f7ac9048740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5628df2c48c0 0x5628df2c4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:21.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.578+0000 7f7ac9048740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5628df2c52b0 0x5628df2c9db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:21.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.578+0000 7f7ac9048740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5628df2ca420 0x5628df2cc8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:21.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.578+0000 7f7ac9048740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5628df28c3b0 con 0x5628df2ca420 2024-03-18T16:12:21.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.578+0000 7f7ac9048740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5628df229450 con 0x5628df2c48c0 2024-03-18T16:12:21.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.578+0000 7f7ac9048740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x5628df1d1cb0 con 0x5628df2c52b0 2024-03-18T16:12:21.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.579+0000 7f7abea00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5628df2c48c0 0x5628df2c4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:21.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.579+0000 7f7abea00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5628df2c48c0 0x5628df2c4ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:55106/0 (socket says 172.21.15.39:55106) 2024-03-18T16:12:21.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.579+0000 7f7ab7e00640 1 --2- 172.21.15.39:0/1995201264 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f7aa40a2880 0x7f7aa40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:21.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.579+0000 7f7abea00640 1 -- 172.21.15.39:0/2940104668 learned_addr learned my addr 172.21.15.39:0/2940104668 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:12:21.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.580+0000 7f7abf400640 1 --2- 172.21.15.39:0/2940104668 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5628df2c52b0 0x5628df2c9db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:21.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.580+0000 7f7abea00640 1 --2- 172.21.15.39:0/2940104668 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5628df2ca420 0x5628df2cc8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:21.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.580+0000 7f7ab7e00640 1 -- 172.21.15.39:0/1995201264 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f7ab80cede0 con 0x7f7ac0109930 2024-03-18T16:12:21.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.580+0000 7f7abfe00640 1 --2- 172.21.15.39:0/1995201264 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f7aa40a2880 0x7f7aa40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:21.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.581+0000 7f7abea00640 1 -- 172.21.15.39:0/2940104668 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5628df2c52b0 msgr2=0x5628df2c9db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:12:21.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.581+0000 7f7abfe00640 1 --2- 172.21.15.39:0/1995201264 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f7aa40a2880 0x7f7aa40a4d40 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f7ab8009a60 tx=0x7f7ab800ceb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:21.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.581+0000 7f7abea00640 1 --2- 172.21.15.39:0/2940104668 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5628df2c52b0 0x5628df2c9db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:21.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.581+0000 7f7abea00640 1 -- 172.21.15.39:0/2940104668 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5628df2c48c0 msgr2=0x5628df2c4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:12:21.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.581+0000 7f7abea00640 1 --2- 172.21.15.39:0/2940104668 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5628df2c48c0 0x5628df2c4ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:21.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.581+0000 7f7abea00640 1 -- 172.21.15.39:0/2940104668 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5628df1f31f0 con 0x5628df2ca420 2024-03-18T16:12:21.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.582+0000 7f7abf400640 1 --2- 172.21.15.39:0/2940104668 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5628df2c52b0 0x5628df2c9db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:12:21.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.582+0000 7f7abea00640 1 --2- 172.21.15.39:0/2940104668 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5628df2ca420 0x5628df2cc8b0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f7ab002c0b0 tx=0x7f7ab0000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8c6c70d6154e140 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:21.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.583+0000 7f7ab4c00640 1 -- 172.21.15.39:0/2940104668 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ab0048070 con 0x5628df2ca420 2024-03-18T16:12:21.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.583+0000 7f7ab4c00640 1 -- 172.21.15.39:0/2940104668 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ab003a040 con 0x5628df2ca420 2024-03-18T16:12:21.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.583+0000 7f7ab4c00640 1 -- 172.21.15.39:0/2940104668 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ab0039070 con 0x5628df2ca420 2024-03-18T16:12:21.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.584+0000 7f7ac9048740 1 -- 172.21.15.39:0/2940104668 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5628df2ca420 msgr2=0x5628df2cc8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:21.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.584+0000 7f7ac9048740 1 --2- 172.21.15.39:0/2940104668 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5628df2ca420 0x5628df2cc8b0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f7ab002c0b0 tx=0x7f7ab0000bc0 comp rx=0 tx=0).stop 2024-03-18T16:12:21.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.584+0000 7f7ac9048740 1 -- 172.21.15.39:0/2940104668 shutdown_connections 2024-03-18T16:12:21.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.584+0000 7f7ac9048740 1 --2- 172.21.15.39:0/2940104668 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5628df2ca420 0x5628df2cc8b0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:21.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.584+0000 7f7ac9048740 1 --2- 172.21.15.39:0/2940104668 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5628df2c52b0 0x5628df2c9db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:21.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.584+0000 7f7ac9048740 1 --2- 172.21.15.39:0/2940104668 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5628df2c48c0 0x5628df2c4ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:21.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.584+0000 7f7ac9048740 1 -- 172.21.15.39:0/2940104668 >> 172.21.15.39:0/2940104668 conn(0x5628df2bfe30 msgr2=0x5628df2c0230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:21.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.584+0000 7f7ac9048740 1 -- 172.21.15.39:0/2940104668 shutdown_connections 2024-03-18T16:12:21.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.584+0000 7f7ac9048740 1 -- 172.21.15.39:0/2940104668 wait complete. 2024-03-18T16:12:21.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.585+0000 7f7ac9048740 1 -- start start 2024-03-18T16:12:21.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.585+0000 7f7ac9048740 1 Processor -- start 2024-03-18T16:12:21.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:12:21 smithi039 ceph-mon[31721]: pgmap v490: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 284 MiB/s wr, 100 op/s 2024-03-18T16:12:21.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:12:21 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/1382861812' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:12:21.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.586+0000 7f7ac9048740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5628df2c48c0 0x5628df2ceb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:21.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.586+0000 7f7ac9048740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5628df2c52b0 0x5628df2cf0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:21.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.586+0000 7f7ac9048740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5628df2ca420 0x5628df2d6100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:21.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.586+0000 7f7ac9048740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5628df1e5110 con 0x5628df2c48c0 2024-03-18T16:12:21.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.586+0000 7f7ac9048740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5628df2254f0 con 0x5628df2c52b0 2024-03-18T16:12:21.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.586+0000 7f7ac9048740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x5628df2cd370 con 0x5628df2ca420 2024-03-18T16:12:21.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.586+0000 7f7ac9048740 10 client.0 _ll_register_callbacks cb 0x5628df1d2b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:12:21.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.586+0000 7f7abea00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5628df2c48c0 0x5628df2ceb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:21.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.586+0000 7f7abea00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5628df2c48c0 0x5628df2ceb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:41916/0 (socket says 172.21.15.39:41916) 2024-03-18T16:12:21.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.586+0000 7f7abea00640 1 -- 172.21.15.39:0/1946519167 learned_addr learned my addr 172.21.15.39:0/1946519167 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:12:21.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.587+0000 7f7abf400640 1 --2- 172.21.15.39:0/1946519167 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5628df2c52b0 0x5628df2cf0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:21.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.587+0000 7f7abf400640 1 -- 172.21.15.39:0/1946519167 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5628df2ca420 msgr2=0x5628df2d6100 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T16:12:21.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.587+0000 7f7abf400640 1 --2- 172.21.15.39:0/1946519167 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5628df2ca420 0x5628df2d6100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:21.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.587+0000 7f7abf400640 1 -- 172.21.15.39:0/1946519167 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5628df2c48c0 msgr2=0x5628df2ceb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:12:21.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.587+0000 7f7abf400640 1 --2- 172.21.15.39:0/1946519167 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5628df2c48c0 0x5628df2ceb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:21.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.587+0000 7f7abf400640 1 -- 172.21.15.39:0/1946519167 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5628def3d3f0 con 0x5628df2c52b0 2024-03-18T16:12:21.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.587+0000 7f7abea00640 1 --2- 172.21.15.39:0/1946519167 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5628df2c48c0 0x5628df2ceb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:12:21.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.588+0000 7f7abf400640 1 --2- 172.21.15.39:0/1946519167 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5628df2c52b0 0x5628df2cf0c0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x5628df2c56b0 tx=0x7f7aac012b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b83c9c63c58f2b14 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:21.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.588+0000 7f7aa9600640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:12:21.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.588+0000 7f7aa9600640 1 -- 172.21.15.39:0/1946519167 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aac004260 con 0x5628df2c52b0 2024-03-18T16:12:21.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.588+0000 7f7aa9600640 1 -- 172.21.15.39:0/1946519167 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f7aac004400 con 0x5628df2c52b0 2024-03-18T16:12:21.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.589+0000 7f7ac9048740 1 -- 172.21.15.39:0/1946519167 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7ab003a040 con 0x5628df2c52b0 2024-03-18T16:12:21.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.589+0000 7f7aa9600640 1 -- 172.21.15.39:0/1946519167 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f7aac011970 con 0x5628df2c52b0 2024-03-18T16:12:21.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.589+0000 7f7ac9048740 10 client.26045 fetch_fsmap learned FSMap version 32 2024-03-18T16:12:21.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.590+0000 7f7ac9048740 1 -- 172.21.15.39:0/1946519167 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5628df0077d0 con 0x5628df2c52b0 2024-03-18T16:12:21.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.590+0000 7f7aa9600640 1 -- 172.21.15.39:0/1946519167 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ab003a040 con 0x5628df2c52b0 2024-03-18T16:12:21.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.591+0000 7f7aa9600640 1 -- 172.21.15.39:0/1946519167 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7f7aac00f440 con 0x5628df2c52b0 2024-03-18T16:12:21.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.591+0000 7f7ac9048740 10 client.26045 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T16:12:21.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.591+0000 7f7ac9048740 10 client.26045 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-03-18T16:12:21.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.592+0000 7f7ac9048740 20 client.26045 populate_metadata read hostname 'smithi039' 2024-03-18T16:12:21.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.592+0000 7f7ac9048740 1 --2- 172.21.15.39:0/1946519167 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x5628df2da9f0 0x5628df2dce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:21.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.592+0000 7f7ac9048740 4 client.26045 mds_command: new command op to 14463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:12:21.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.592+0000 7f7ac9048740 1 -- 172.21.15.39:0/1946519167 --> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5628df236990 con 0x5628df2da9f0 2024-03-18T16:12:21.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.592+0000 7f7abea00640 1 --2- 172.21.15.39:0/1946519167 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x5628df2da9f0 0x5628df2dce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:21.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.593+0000 7f7abea00640 1 --2- 172.21.15.39:0/1946519167 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x5628df2da9f0 0x5628df2dce40 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=80b75ed28a896fec server_cookie=e627886bc2402fca in_seq=0 out_seq=0 2024-03-18T16:12:21.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.593+0000 7f7aa9600640 10 client.26045 ms_handle_connect on v2:172.21.15.39:6840/3515326042 2024-03-18T16:12:21.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.594+0000 7f7aa9600640 1 -- 172.21.15.39:0/1946519167 <== mds.4 v2:172.21.15.39:6840/3515326042 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5628df236990 con 0x5628df2da9f0 2024-03-18T16:12:21.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.594+0000 7f7aa9600640 10 client.26045 handle_command_reply: tid=0 2024-03-18T16:12:21.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.594+0000 7f7ac9048740 1 client.26045 shutdown 2024-03-18T16:12:21.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.595+0000 7f7ac9048740 1 -- 172.21.15.39:0/1946519167 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5628df2c52b0 msgr2=0x5628df2cf0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:21.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.595+0000 7f7ac9048740 1 --2- 172.21.15.39:0/1946519167 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5628df2c52b0 0x5628df2cf0c0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x5628df2c56b0 tx=0x7f7aac012b60 comp rx=0 tx=0).stop 2024-03-18T16:12:21.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.595+0000 7f7ac9048740 1 -- 172.21.15.39:0/1946519167 shutdown_connections 2024-03-18T16:12:21.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.595+0000 7f7ac9048740 1 --2- 172.21.15.39:0/1946519167 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x5628df2da9f0 0x5628df2dce40 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:21.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.595+0000 7f7ac9048740 1 --2- 172.21.15.39:0/1946519167 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5628df2c52b0 0x5628df2cf0c0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:21.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.595+0000 7f7ac9048740 1 --2- 172.21.15.39:0/1946519167 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5628df2ca420 0x5628df2d6100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:21.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.595+0000 7f7ac9048740 1 --2- 172.21.15.39:0/1946519167 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5628df2c48c0 0x5628df2ceb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:21.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.595+0000 7f7ac9048740 1 -- 172.21.15.39:0/1946519167 >> 172.21.15.39:0/1946519167 conn(0x5628df2bfe30 msgr2=0x5628df2bf1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:21.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.595+0000 7f7aa9600640 0 client.26045 ms_handle_reset on v2:172.21.15.39:6840/3515326042 2024-03-18T16:12:21.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.596+0000 7f7ac9048740 1 -- 172.21.15.39:0/1946519167 shutdown_connections 2024-03-18T16:12:21.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.596+0000 7f7ac9048740 1 -- 172.21.15.39:0/1946519167 wait complete. 2024-03-18T16:12:21.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.596+0000 7f7ac9048740 20 client.26045 trim_cache size 0 max 16384 2024-03-18T16:12:21.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.620+0000 7f7ac9048740 1 Processor -- start 2024-03-18T16:12:21.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.620+0000 7f7ac9048740 1 -- start start 2024-03-18T16:12:21.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.621+0000 7f7ac9048740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5628df2c48c0 0x5628df2ceb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:21.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.621+0000 7f7ac9048740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5628df2c52b0 0x5628df2cf0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:21.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.621+0000 7f7ac9048740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5628df2ca420 0x5628df2d60c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:21.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.621+0000 7f7ac9048740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f7ac01b9280 con 0x5628df2c52b0 2024-03-18T16:12:21.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.621+0000 7f7ac9048740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f7ac01bae80 con 0x5628df2c48c0 2024-03-18T16:12:21.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.621+0000 7f7ac9048740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f7ac01b9950 con 0x5628df2ca420 2024-03-18T16:12:21.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.621+0000 7f7abea00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5628df2c48c0 0x5628df2ceb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:21.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.621+0000 7f7abea00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5628df2c48c0 0x5628df2ceb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:55132/0 (socket says 172.21.15.39:55132) 2024-03-18T16:12:21.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.621+0000 7f7abea00640 1 -- 172.21.15.39:0/52661416 learned_addr learned my addr 172.21.15.39:0/52661416 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:12:21.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.621+0000 7f7abf400640 1 --2- 172.21.15.39:0/52661416 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5628df2c52b0 0x5628df2cf0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:21.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.621+0000 7f7abea00640 1 --2- 172.21.15.39:0/52661416 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5628df2ca420 0x5628df2d60c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:21.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.622+0000 7f7abea00640 1 -- 172.21.15.39:0/52661416 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5628df2ca420 msgr2=0x5628df2d60c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:12:21.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.622+0000 7f7abea00640 1 --2- 172.21.15.39:0/52661416 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5628df2ca420 0x5628df2d60c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:21.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.622+0000 7f7abea00640 1 -- 172.21.15.39:0/52661416 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5628df2c52b0 msgr2=0x5628df2cf0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:12:21.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.622+0000 7f7abea00640 1 --2- 172.21.15.39:0/52661416 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5628df2c52b0 0x5628df2cf0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:21.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.622+0000 7f7abea00640 1 -- 172.21.15.39:0/52661416 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5628df2bbc10 con 0x5628df2c48c0 2024-03-18T16:12:21.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.622+0000 7f7abea00640 1 --2- 172.21.15.39:0/52661416 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5628df2c48c0 0x5628df2ceb80 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f7ab0046040 tx=0x7f7ab00095b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=43a65b712ee28da7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:21.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.622+0000 7f7aabe00640 1 -- 172.21.15.39:0/52661416 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ab0039070 con 0x5628df2c48c0 2024-03-18T16:12:21.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.622+0000 7f7aabe00640 1 -- 172.21.15.39:0/52661416 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ab003a040 con 0x5628df2c48c0 2024-03-18T16:12:21.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.622+0000 7f7aabe00640 1 -- 172.21.15.39:0/52661416 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ab0048040 con 0x5628df2c48c0 2024-03-18T16:12:21.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.625+0000 7f7ac9048740 1 -- 172.21.15.39:0/52661416 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5628df2c48c0 msgr2=0x5628df2ceb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:21.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.625+0000 7f7ac9048740 1 --2- 172.21.15.39:0/52661416 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5628df2c48c0 0x5628df2ceb80 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f7ab0046040 tx=0x7f7ab00095b0 comp rx=0 tx=0).stop 2024-03-18T16:12:21.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.625+0000 7f7ac9048740 1 -- 172.21.15.39:0/52661416 shutdown_connections 2024-03-18T16:12:21.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.625+0000 7f7ac9048740 1 --2- 172.21.15.39:0/52661416 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5628df2ca420 0x5628df2d60c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:21.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.625+0000 7f7ac9048740 1 --2- 172.21.15.39:0/52661416 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5628df2c52b0 0x5628df2cf0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:21.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.625+0000 7f7ac9048740 1 --2- 172.21.15.39:0/52661416 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5628df2c48c0 0x5628df2ceb80 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:21.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.625+0000 7f7ac9048740 1 -- 172.21.15.39:0/52661416 >> 172.21.15.39:0/52661416 conn(0x5628df2bfe30 msgr2=0x5628df2c0210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:21.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.625+0000 7f7ac9048740 1 -- 172.21.15.39:0/52661416 shutdown_connections 2024-03-18T16:12:21.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.625+0000 7f7ac9048740 1 -- 172.21.15.39:0/52661416 wait complete. 2024-03-18T16:12:21.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.625+0000 7f7ac9048740 1 -- start start 2024-03-18T16:12:21.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.626+0000 7f7ac9048740 1 Processor -- start 2024-03-18T16:12:21.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.626+0000 7f7ac9048740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5628df2c48c0 0x5628df2d3170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:21.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.626+0000 7f7ac9048740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5628df2c52b0 0x5628df2d36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:21.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.626+0000 7f7ac9048740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5628df2ca420 0x5628df2d46c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:21.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.626+0000 7f7ac9048740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f7ac01ba750 con 0x5628df2c52b0 2024-03-18T16:12:21.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.626+0000 7f7ac9048740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f7ac01ba020 con 0x5628df2c48c0 2024-03-18T16:12:21.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.626+0000 7f7ac9048740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f7ac0082540 con 0x5628df2ca420 2024-03-18T16:12:21.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.626+0000 7f7ac9048740 10 client.0 _ll_register_callbacks cb 0x7f7aac009760 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:12:21.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.626+0000 7f7abea00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5628df2c48c0 0x5628df2d3170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:21.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.626+0000 7f7abea00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5628df2c48c0 0x5628df2d3170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:55144/0 (socket says 172.21.15.39:55144) 2024-03-18T16:12:21.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.626+0000 7f7abea00640 1 -- 172.21.15.39:0/3013947727 learned_addr learned my addr 172.21.15.39:0/3013947727 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:12:21.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.626+0000 7f7abf400640 1 --2- 172.21.15.39:0/3013947727 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5628df2c52b0 0x5628df2d36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:21.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.626+0000 7f7abea00640 1 --2- 172.21.15.39:0/3013947727 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5628df2ca420 0x5628df2d46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:21.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.627+0000 7f7abf400640 1 -- 172.21.15.39:0/3013947727 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5628df2ca420 msgr2=0x5628df2d46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:12:21.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.627+0000 7f7abf400640 1 --2- 172.21.15.39:0/3013947727 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5628df2ca420 0x5628df2d46c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:21.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.627+0000 7f7abf400640 1 -- 172.21.15.39:0/3013947727 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5628df2c48c0 msgr2=0x5628df2d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:12:21.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.627+0000 7f7abf400640 1 --2- 172.21.15.39:0/3013947727 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5628df2c48c0 0x5628df2d3170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:21.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.627+0000 7f7abf400640 1 -- 172.21.15.39:0/3013947727 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5628df2cdfa0 con 0x5628df2c52b0 2024-03-18T16:12:21.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.627+0000 7f7abf400640 1 --2- 172.21.15.39:0/3013947727 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5628df2c52b0 0x5628df2d36b0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f7aac00fa00 tx=0x7f7aac011b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c72c764b5371380 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:21.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.627+0000 7f7aa9600640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-03-18T16:12:21.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.628+0000 7f7aa9600640 1 -- 172.21.15.39:0/3013947727 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7aac02c070 con 0x5628df2c52b0 2024-03-18T16:12:21.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.628+0000 7f7aa9600640 1 -- 172.21.15.39:0/3013947727 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f7aac0052c0 con 0x5628df2c52b0 2024-03-18T16:12:21.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.628+0000 7f7ac9048740 1 -- 172.21.15.39:0/3013947727 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7ab003a040 con 0x5628df2c52b0 2024-03-18T16:12:21.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.629+0000 7f7aa9600640 1 -- 172.21.15.39:0/3013947727 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f7aac0054c0 con 0x5628df2c52b0 2024-03-18T16:12:21.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.629+0000 7f7ac9048740 10 client.15852 fetch_fsmap learned FSMap version 32 2024-03-18T16:12:21.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.629+0000 7f7ac9048740 1 -- 172.21.15.39:0/3013947727 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5628df2d5e50 con 0x5628df2c52b0 2024-03-18T16:12:21.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.629+0000 7f7aa9600640 1 -- 172.21.15.39:0/3013947727 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ab003a040 con 0x5628df2c52b0 2024-03-18T16:12:21.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.629+0000 7f7aa9600640 1 -- 172.21.15.39:0/3013947727 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7f7aac003c10 con 0x5628df2c52b0 2024-03-18T16:12:21.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.629+0000 7f7ac9048740 10 client.15852 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T16:12:21.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.629+0000 7f7ac9048740 10 client.15852 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-03-18T16:12:21.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.629+0000 7f7ac9048740 20 client.15852 populate_metadata read hostname 'smithi039' 2024-03-18T16:12:21.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.629+0000 7f7ac9048740 1 --2- 172.21.15.39:0/3013947727 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x5628df340f00 0x5628df343320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:21.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.629+0000 7f7ac9048740 4 client.15852 mds_command: new command op to 14463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:12:21.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.629+0000 7f7ac9048740 1 -- 172.21.15.39:0/3013947727 --> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5628df230ce0 con 0x5628df340f00 2024-03-18T16:12:21.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.631+0000 7f7abea00640 1 --2- 172.21.15.39:0/3013947727 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x5628df340f00 0x5628df343320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:21.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.633+0000 7f7abea00640 1 --2- 172.21.15.39:0/3013947727 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x5628df340f00 0x5628df343320 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=660d4b564d7abd server_cookie=bddd7d69b9364e96 in_seq=0 out_seq=0 2024-03-18T16:12:21.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.633+0000 7f7aa9600640 10 client.15852 ms_handle_connect on v2:172.21.15.39:6840/3515326042 2024-03-18T16:12:21.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.633+0000 7f7aa9600640 1 -- 172.21.15.39:0/3013947727 <== mds.4 v2:172.21.15.39:6840/3515326042 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5628df230ce0 con 0x5628df340f00 2024-03-18T16:12:21.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.633+0000 7f7aa9600640 10 client.15852 handle_command_reply: tid=0 2024-03-18T16:12:21.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.633+0000 7f7ac9048740 1 client.15852 shutdown 2024-03-18T16:12:21.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.634+0000 7f7ac9048740 1 -- 172.21.15.39:0/3013947727 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5628df2c52b0 msgr2=0x5628df2d36b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:21.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.634+0000 7f7ac9048740 1 --2- 172.21.15.39:0/3013947727 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5628df2c52b0 0x5628df2d36b0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f7aac00fa00 tx=0x7f7aac011b10 comp rx=0 tx=0).stop 2024-03-18T16:12:21.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.634+0000 7f7ac9048740 1 -- 172.21.15.39:0/3013947727 shutdown_connections 2024-03-18T16:12:21.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.634+0000 7f7ac9048740 1 --2- 172.21.15.39:0/3013947727 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x5628df340f00 0x5628df343320 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:21.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.634+0000 7f7ac9048740 1 --2- 172.21.15.39:0/3013947727 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5628df2ca420 0x5628df2d46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:21.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.634+0000 7f7ac9048740 1 --2- 172.21.15.39:0/3013947727 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5628df2c52b0 0x5628df2d36b0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:21.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.634+0000 7f7ac9048740 1 --2- 172.21.15.39:0/3013947727 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5628df2c48c0 0x5628df2d3170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:21.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.634+0000 7f7ac9048740 1 -- 172.21.15.39:0/3013947727 >> 172.21.15.39:0/3013947727 conn(0x5628df2bfe30 msgr2=0x5628df2ceb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:21.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.634+0000 7f7aa9600640 0 client.15852 ms_handle_reset on v2:172.21.15.39:6840/3515326042 2024-03-18T16:12:21.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.635+0000 7f7ac9048740 1 -- 172.21.15.39:0/3013947727 shutdown_connections 2024-03-18T16:12:21.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.635+0000 7f7ac9048740 1 -- 172.21.15.39:0/3013947727 wait complete. 2024-03-18T16:12:21.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.635+0000 7f7ac9048740 20 client.15852 trim_cache size 0 max 16384 2024-03-18T16:12:21.637 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:12:21.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.636+0000 7f7ab5600640 1 -- 172.21.15.39:0/1995201264 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f7aa40a2880 msgr2=0x7f7aa40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:21.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.636+0000 7f7ab5600640 1 --2- 172.21.15.39:0/1995201264 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f7aa40a2880 0x7f7aa40a4d40 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f7ab8009a60 tx=0x7f7ab800ceb0 comp rx=0 tx=0).stop 2024-03-18T16:12:21.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.636+0000 7f7ab5600640 1 -- 172.21.15.39:0/1995201264 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7ac0109930 msgr2=0x7f7ac01a56c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:21.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.636+0000 7f7ab5600640 1 --2- 172.21.15.39:0/1995201264 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7ac0109930 0x7f7ac01a56c0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f7ab800efc0 tx=0x7f7ab800c800 comp rx=0 tx=0).stop 2024-03-18T16:12:21.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.636+0000 7f7ab5600640 1 -- 172.21.15.39:0/1995201264 shutdown_connections 2024-03-18T16:12:21.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.636+0000 7f7ab5600640 1 --2- 172.21.15.39:0/1995201264 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7ac0108f70 0x7f7ac019e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:21.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.636+0000 7f7ab5600640 1 --2- 172.21.15.39:0/1995201264 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f7aa40a2880 0x7f7aa40a4d40 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:21.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.636+0000 7f7ab5600640 1 --2- 172.21.15.39:0/1995201264 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7ac0109930 0x7f7ac01a56c0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:21.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.636+0000 7f7ab5600640 1 --2- 172.21.15.39:0/1995201264 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7ac0107e20 0x7f7ac019e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:21.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.636+0000 7f7ab5600640 1 -- 172.21.15.39:0/1995201264 >> 172.21.15.39:0/1995201264 conn(0x7f7ac0076dd0 msgr2=0x7f7ac010c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:21.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.636+0000 7f7ab5600640 1 -- 172.21.15.39:0/1995201264 shutdown_connections 2024-03-18T16:12:21.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.636+0000 7f7ab5600640 1 -- 172.21.15.39:0/1995201264 wait complete. 2024-03-18T16:12:21.659 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-18T16:12:21.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.964+0000 7fd86c000640 1 -- 172.21.15.39:0/2394263823 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd8641055b0 msgr2=0x7fd86410be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:21.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.964+0000 7fd86c000640 1 --2- 172.21.15.39:0/2394263823 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd8641055b0 0x7fd86410be40 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fd85c009ca0 tx=0x7fd85c02c0f0 comp rx=0 tx=0).stop 2024-03-18T16:12:21.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.965+0000 7fd86c000640 1 -- 172.21.15.39:0/2394263823 shutdown_connections 2024-03-18T16:12:21.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.965+0000 7fd86c000640 1 --2- 172.21.15.39:0/2394263823 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd864104bf0 0x7fd864105070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:21.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.965+0000 7fd86c000640 1 --2- 172.21.15.39:0/2394263823 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd8641055b0 0x7fd86410be40 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:21.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.965+0000 7fd86c000640 1 --2- 172.21.15.39:0/2394263823 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd864103a00 0x7fd864103e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:21.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.965+0000 7fd86c000640 1 -- 172.21.15.39:0/2394263823 >> 172.21.15.39:0/2394263823 conn(0x7fd8640fef90 msgr2=0x7fd8641013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:21.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.965+0000 7fd86c000640 1 -- 172.21.15.39:0/2394263823 shutdown_connections 2024-03-18T16:12:21.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.965+0000 7fd86c000640 1 -- 172.21.15.39:0/2394263823 wait complete. 2024-03-18T16:12:21.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.966+0000 7fd86c000640 1 Processor -- start 2024-03-18T16:12:21.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.966+0000 7fd86c000640 1 -- start start 2024-03-18T16:12:21.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.966+0000 7fd86c000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd864103a00 0x7fd864070130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:21.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.967+0000 7fd86c000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd864104bf0 0x7fd864070670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:21.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.967+0000 7fd86c000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd8641055b0 0x7fd8641a9d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:21.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.967+0000 7fd86c000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fd86410f680 con 0x7fd864104bf0 2024-03-18T16:12:21.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.967+0000 7fd86c000640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fd86410f500 con 0x7fd864103a00 2024-03-18T16:12:21.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.967+0000 7fd86c000640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fd86410f800 con 0x7fd8641055b0 2024-03-18T16:12:21.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.967+0000 7fd86a000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd8641055b0 0x7fd8641a9d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:21.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.967+0000 7fd86a000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd8641055b0 0x7fd8641a9d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:54498/0 (socket says 172.21.15.39:54498) 2024-03-18T16:12:21.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.967+0000 7fd86a000640 1 -- 172.21.15.39:0/2284379665 learned_addr learned my addr 172.21.15.39:0/2284379665 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:12:21.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.967+0000 7fd869600640 1 --2- 172.21.15.39:0/2284379665 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd864103a00 0x7fd864070130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:21.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.967+0000 7fd86a000640 1 -- 172.21.15.39:0/2284379665 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd864103a00 msgr2=0x7fd864070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:21.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.967+0000 7fd86a000640 1 --2- 172.21.15.39:0/2284379665 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd864103a00 0x7fd864070130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:21.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.967+0000 7fd86a000640 1 -- 172.21.15.39:0/2284379665 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd864104bf0 msgr2=0x7fd864070670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:21.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.967+0000 7fd86a000640 1 --2- 172.21.15.39:0/2284379665 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd864104bf0 0x7fd864070670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:21.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.967+0000 7fd86a000640 1 -- 172.21.15.39:0/2284379665 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd864070d10 con 0x7fd8641055b0 2024-03-18T16:12:21.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.968+0000 7fd86a000640 1 --2- 172.21.15.39:0/2284379665 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd8641055b0 0x7fd8641a9d90 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fd85c005e30 tx=0x7fd85c032ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:21.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.968+0000 7fd862000640 1 -- 172.21.15.39:0/2284379665 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd85c041020 con 0x7fd8641055b0 2024-03-18T16:12:21.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.968+0000 7fd862000640 1 -- 172.21.15.39:0/2284379665 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd85c034cd0 con 0x7fd8641055b0 2024-03-18T16:12:21.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.968+0000 7fd862000640 1 -- 172.21.15.39:0/2284379665 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd85c039040 con 0x7fd8641055b0 2024-03-18T16:12:21.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.969+0000 7fd86c000640 1 -- 172.21.15.39:0/2284379665 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8641aa390 con 0x7fd8641055b0 2024-03-18T16:12:21.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.969+0000 7fd86c000640 1 -- 172.21.15.39:0/2284379665 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8641aa800 con 0x7fd8641055b0 2024-03-18T16:12:21.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.975+0000 7fd86ed7a740 1 Processor -- start 2024-03-18T16:12:21.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.975+0000 7fd862000640 1 -- 172.21.15.39:0/2284379665 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fd85c044020 con 0x7fd8641055b0 2024-03-18T16:12:21.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.975+0000 7fd86ed7a740 1 -- start start 2024-03-18T16:12:21.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.976+0000 7fd86ed7a740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ff849770d0 0x55ff849774f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:21.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.976+0000 7fd862000640 1 --2- 172.21.15.39:0/2284379665 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd84c0a2950 0x7fd84c0a4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:21.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.976+0000 7fd86ed7a740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ff84977ac0 0x55ff8497c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:21.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.976+0000 7fd862000640 1 -- 172.21.15.39:0/2284379665 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fd85c0f04b0 con 0x7fd8641055b0 2024-03-18T16:12:21.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.976+0000 7fd86ed7a740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ff8497cc30 0x55ff8497f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:21.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.977+0000 7fd86ed7a740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55ff849363b0 con 0x55ff84977ac0 2024-03-18T16:12:21.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.977+0000 7fd86ed7a740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55ff848d3450 con 0x55ff8497cc30 2024-03-18T16:12:21.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.977+0000 7fd86ed7a740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55ff8487bcb0 con 0x55ff849770d0 2024-03-18T16:12:21.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.977+0000 7fd86a000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ff8497cc30 0x55ff8497f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:21.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.977+0000 7fd86a000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ff8497cc30 0x55ff8497f0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:55164/0 (socket says 172.21.15.39:55164) 2024-03-18T16:12:21.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.977+0000 7fd869600640 1 --2- 172.21.15.39:0/2284379665 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd84c0a2950 0x7fd84c0a4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:21.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.977+0000 7fd86a000640 1 -- 172.21.15.39:0/692596736 learned_addr learned my addr 172.21.15.39:0/692596736 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:12:21.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.978+0000 7fd868c00640 1 --2- 172.21.15.39:0/692596736 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ff84977ac0 0x55ff8497c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:21.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.978+0000 7fd868c00640 1 --2- 172.21.15.39:0/692596736 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ff849770d0 0x55ff849774f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:21.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.978+0000 7fd86a000640 1 -- 172.21.15.39:0/692596736 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ff849770d0 msgr2=0x55ff849774f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:12:21.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.978+0000 7fd86a000640 1 --2- 172.21.15.39:0/692596736 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ff849770d0 0x55ff849774f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:21.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.978+0000 7fd86a000640 1 -- 172.21.15.39:0/692596736 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ff84977ac0 msgr2=0x55ff8497c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:12:21.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.978+0000 7fd86a000640 1 --2- 172.21.15.39:0/692596736 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ff84977ac0 0x55ff8497c5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:21.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.978+0000 7fd86a000640 1 -- 172.21.15.39:0/692596736 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ff8489d1f0 con 0x55ff8497cc30 2024-03-18T16:12:21.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.978+0000 7fd86a000640 1 --2- 172.21.15.39:0/692596736 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ff8497cc30 0x55ff8497f0c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fd85c0c4d30 tx=0x7fd85c034f00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=317cb82e204d7d5b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:21.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.978+0000 7fd856600640 1 -- 172.21.15.39:0/692596736 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd85c0bd1c0 con 0x55ff8497cc30 2024-03-18T16:12:21.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.979+0000 7fd856600640 1 -- 172.21.15.39:0/692596736 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd85c0bcac0 con 0x55ff8497cc30 2024-03-18T16:12:21.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.979+0000 7fd856600640 1 -- 172.21.15.39:0/692596736 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd85c0badc0 con 0x55ff8497cc30 2024-03-18T16:12:21.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.979+0000 7fd86ed7a740 1 -- 172.21.15.39:0/692596736 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ff8497cc30 msgr2=0x55ff8497f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:21.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.979+0000 7fd86ed7a740 1 --2- 172.21.15.39:0/692596736 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ff8497cc30 0x55ff8497f0c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fd85c0c4d30 tx=0x7fd85c034f00 comp rx=0 tx=0).stop 2024-03-18T16:12:21.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.980+0000 7fd86ed7a740 1 -- 172.21.15.39:0/692596736 shutdown_connections 2024-03-18T16:12:21.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.980+0000 7fd86ed7a740 1 --2- 172.21.15.39:0/692596736 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ff8497cc30 0x55ff8497f0c0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:21.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.980+0000 7fd86ed7a740 1 --2- 172.21.15.39:0/692596736 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ff84977ac0 0x55ff8497c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:21.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.980+0000 7fd86ed7a740 1 --2- 172.21.15.39:0/692596736 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ff849770d0 0x55ff849774f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:21.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.980+0000 7fd86ed7a740 1 -- 172.21.15.39:0/692596736 >> 172.21.15.39:0/692596736 conn(0x55ff84972640 msgr2=0x55ff84972a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:21.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.980+0000 7fd86ed7a740 1 -- 172.21.15.39:0/692596736 shutdown_connections 2024-03-18T16:12:21.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.980+0000 7fd869600640 1 --2- 172.21.15.39:0/2284379665 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd84c0a2950 0x7fd84c0a4e10 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fd858005d00 tx=0x7fd858005c50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:21.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.980+0000 7fd86ed7a740 1 -- 172.21.15.39:0/692596736 wait complete. 2024-03-18T16:12:21.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.981+0000 7fd86ed7a740 1 -- start start 2024-03-18T16:12:21.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.981+0000 7fd86ed7a740 1 Processor -- start 2024-03-18T16:12:21.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.981+0000 7fd86ed7a740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ff849770d0 0x55ff849810a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:21.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.982+0000 7fd86ed7a740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ff84977ac0 0x55ff849815e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:21.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.982+0000 7fd869600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ff849770d0 0x55ff849810a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:21.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.982+0000 7fd869600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ff849770d0 0x55ff849810a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:41976/0 (socket says 172.21.15.39:41976) 2024-03-18T16:12:21.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.983+0000 7fd868c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ff84977ac0 0x55ff849815e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:21.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.983+0000 7fd86ed7a740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ff8497cc30 0x55ff84988620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:21.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.983+0000 7fd86ed7a740 1 -- 172.21.15.39:0/72239595 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55ff8488f110 con 0x55ff849770d0 2024-03-18T16:12:21.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.983+0000 7fd869600640 1 -- 172.21.15.39:0/72239595 learned_addr learned my addr 172.21.15.39:0/72239595 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:12:21.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.983+0000 7fd868c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ff84977ac0 0x55ff849815e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:54506/0 (socket says 172.21.15.39:54506) 2024-03-18T16:12:21.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.983+0000 7fd86ed7a740 1 -- 172.21.15.39:0/72239595 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55ff848cf4f0 con 0x55ff8497cc30 2024-03-18T16:12:21.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.983+0000 7fd86ed7a740 1 -- 172.21.15.39:0/72239595 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55ff8497f9a0 con 0x55ff84977ac0 2024-03-18T16:12:21.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.983+0000 7fd86a000640 1 --2- 172.21.15.39:0/72239595 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ff8497cc30 0x55ff84988620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:21.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.984+0000 7fd86ed7a740 10 client.0 _ll_register_callbacks cb 0x55ff8487cb70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:12:21.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.984+0000 7fd86a000640 1 -- 172.21.15.39:0/72239595 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ff84977ac0 msgr2=0x55ff849815e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:12:21.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.984+0000 7fd86a000640 1 --2- 172.21.15.39:0/72239595 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ff84977ac0 0x55ff849815e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:21.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.984+0000 7fd86a000640 1 -- 172.21.15.39:0/72239595 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ff849770d0 msgr2=0x55ff849810a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:12:21.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.984+0000 7fd86a000640 1 --2- 172.21.15.39:0/72239595 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ff849770d0 0x55ff849810a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:21.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.984+0000 7fd86a000640 1 -- 172.21.15.39:0/72239595 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ff845e73f0 con 0x55ff8497cc30 2024-03-18T16:12:21.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.984+0000 7fd86ed7a740 1 -- 172.21.15.39:0/72239595 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd85c0bcac0 con 0x55ff8497cc30 2024-03-18T16:12:21.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.984+0000 7fd868c00640 1 --2- 172.21.15.39:0/72239595 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ff84977ac0 0x55ff849815e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:12:21.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.984+0000 7fd869600640 1 --2- 172.21.15.39:0/72239595 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ff849770d0 0x55ff849810a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:12:21.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.985+0000 7fd86a000640 1 --2- 172.21.15.39:0/72239595 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ff8497cc30 0x55ff84988620 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fd85c009430 tx=0x7fd85c034950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e37d96bd0698b0e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:21.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.985+0000 7fd84aa00640 10 client.26063 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:12:21.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.985+0000 7fd84aa00640 1 -- 172.21.15.39:0/72239595 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd85c0d9020 con 0x55ff8497cc30 2024-03-18T16:12:21.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.985+0000 7fd84aa00640 1 -- 172.21.15.39:0/72239595 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fd85c0f0220 con 0x55ff8497cc30 2024-03-18T16:12:21.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.985+0000 7fd84aa00640 1 -- 172.21.15.39:0/72239595 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fd85c0bd1c0 con 0x55ff8497cc30 2024-03-18T16:12:21.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.985+0000 7fd86ed7a740 10 client.26063 fetch_fsmap learned FSMap version 32 2024-03-18T16:12:21.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.985+0000 7fd86ed7a740 1 -- 172.21.15.39:0/72239595 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({fsmap=32}) -- 0x55ff846b17d0 con 0x55ff8497cc30 2024-03-18T16:12:21.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.986+0000 7fd84aa00640 1 -- 172.21.15.39:0/72239595 <== mon.2 v2:172.21.15.194:3300/0 4 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7fd85c03c030 con 0x55ff8497cc30 2024-03-18T16:12:21.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.986+0000 7fd86ed7a740 10 client.26063 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T16:12:21.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.986+0000 7fd86ed7a740 10 client.26063 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-18T16:12:21.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.986+0000 7fd86ed7a740 20 client.26063 populate_metadata read hostname 'smithi039' 2024-03-18T16:12:21.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.986+0000 7fd86ed7a740 1 --2- 172.21.15.39:0/72239595 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x55ff8498cbe0 0x55ff8498eff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:21.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.986+0000 7fd86ed7a740 4 client.26063 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:12:21.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.986+0000 7fd86ed7a740 1 -- 172.21.15.39:0/72239595 --> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ff848e0990 con 0x55ff8498cbe0 2024-03-18T16:12:21.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.987+0000 7fd869600640 1 --2- 172.21.15.39:0/72239595 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x55ff8498cbe0 0x55ff8498eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:21.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.987+0000 7fd84aa00640 10 client.26063 ms_handle_connect on v2:172.21.15.194:6836/1301464246 2024-03-18T16:12:21.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.987+0000 7fd869600640 1 --2- 172.21.15.39:0/72239595 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x55ff8498cbe0 0x55ff8498eff0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=39bb13abeb25723c server_cookie=9ba91ef009ba2c96 in_seq=0 out_seq=0 2024-03-18T16:12:21.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.992+0000 7fd84aa00640 1 -- 172.21.15.39:0/72239595 <== mds.3 v2:172.21.15.194:6836/1301464246 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ff848e0990 con 0x55ff8498cbe0 2024-03-18T16:12:21.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.992+0000 7fd84aa00640 10 client.26063 handle_command_reply: tid=0 2024-03-18T16:12:21.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.992+0000 7fd86ed7a740 1 client.26063 shutdown 2024-03-18T16:12:21.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.992+0000 7fd86ed7a740 1 -- 172.21.15.39:0/72239595 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ff8497cc30 msgr2=0x55ff84988620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:21.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.992+0000 7fd86ed7a740 1 --2- 172.21.15.39:0/72239595 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ff8497cc30 0x55ff84988620 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fd85c009430 tx=0x7fd85c034950 comp rx=0 tx=0).stop 2024-03-18T16:12:21.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.992+0000 7fd86ed7a740 1 -- 172.21.15.39:0/72239595 shutdown_connections 2024-03-18T16:12:21.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.992+0000 7fd86ed7a740 1 --2- 172.21.15.39:0/72239595 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x55ff8498cbe0 0x55ff8498eff0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:21.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.992+0000 7fd86ed7a740 1 --2- 172.21.15.39:0/72239595 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ff8497cc30 0x55ff84988620 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:21.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.992+0000 7fd86ed7a740 1 --2- 172.21.15.39:0/72239595 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ff84977ac0 0x55ff849815e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:21.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.992+0000 7fd86ed7a740 1 --2- 172.21.15.39:0/72239595 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ff849770d0 0x55ff849810a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:21.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.992+0000 7fd86ed7a740 1 -- 172.21.15.39:0/72239595 >> 172.21.15.39:0/72239595 conn(0x55ff84972640 msgr2=0x55ff84973890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:21.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.993+0000 7fd84aa00640 0 client.26063 ms_handle_reset on v2:172.21.15.194:6836/1301464246 2024-03-18T16:12:21.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.993+0000 7fd86ed7a740 1 -- 172.21.15.39:0/72239595 shutdown_connections 2024-03-18T16:12:21.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.993+0000 7fd86ed7a740 1 -- 172.21.15.39:0/72239595 wait complete. 2024-03-18T16:12:21.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:21.994+0000 7fd86ed7a740 20 client.26063 trim_cache size 0 max 16384 2024-03-18T16:12:22.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.047+0000 7fd86ed7a740 1 Processor -- start 2024-03-18T16:12:22.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.048+0000 7fd86ed7a740 1 -- start start 2024-03-18T16:12:22.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.050+0000 7fd86ed7a740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ff849770d0 0x55ff84978f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:22.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.053+0000 7fd869600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ff849770d0 0x55ff84978f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:22.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.056+0000 7fd86ed7a740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ff84977ac0 0x55ff849794a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:22.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.056+0000 7fd86ed7a740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ff8497cc30 0x55ff849885e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:22.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.056+0000 7fd86ed7a740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fd8641b09a0 con 0x55ff8497cc30 2024-03-18T16:12:22.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.056+0000 7fd86ed7a740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fd8641b25a0 con 0x55ff849770d0 2024-03-18T16:12:22.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.056+0000 7fd86ed7a740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fd8641b1070 con 0x55ff84977ac0 2024-03-18T16:12:22.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.058+0000 7fd86a000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ff8497cc30 0x55ff849885e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:22.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.058+0000 7fd868c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ff84977ac0 0x55ff849794a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:22.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.058+0000 7fd868c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ff84977ac0 0x55ff849794a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:54510/0 (socket says 172.21.15.39:54510) 2024-03-18T16:12:22.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.058+0000 7fd868c00640 1 -- 172.21.15.39:0/4126565452 learned_addr learned my addr 172.21.15.39:0/4126565452 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:12:22.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.061+0000 7fd868c00640 1 -- 172.21.15.39:0/4126565452 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ff849770d0 msgr2=0x55ff84978f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:12:22.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.061+0000 7fd868c00640 1 --2- 172.21.15.39:0/4126565452 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ff849770d0 0x55ff84978f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:22.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.062+0000 7fd868c00640 1 -- 172.21.15.39:0/4126565452 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ff8497cc30 msgr2=0x55ff849885e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:12:22.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.063+0000 7fd868c00640 1 --2- 172.21.15.39:0/4126565452 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ff8497cc30 0x55ff849885e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:22.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.063+0000 7fd868c00640 1 -- 172.21.15.39:0/4126565452 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ff84965c10 con 0x55ff84977ac0 2024-03-18T16:12:22.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.063+0000 7fd869600640 1 --2- 172.21.15.39:0/4126565452 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ff849770d0 0x55ff84978f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:12:22.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.063+0000 7fd86a000640 1 --2- 172.21.15.39:0/4126565452 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ff8497cc30 0x55ff849885e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:12:22.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.064+0000 7fd868c00640 1 --2- 172.21.15.39:0/4126565452 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ff84977ac0 0x55ff849794a0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x55ff84978320 tx=0x7fd850005f50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6b891d373f974d72 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:22.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.065+0000 7fd855c00640 1 -- 172.21.15.39:0/4126565452 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd850003910 con 0x55ff84977ac0 2024-03-18T16:12:22.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.066+0000 7fd855c00640 1 -- 172.21.15.39:0/4126565452 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd850012450 con 0x55ff84977ac0 2024-03-18T16:12:22.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.066+0000 7fd855c00640 1 -- 172.21.15.39:0/4126565452 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd85001c6a0 con 0x55ff84977ac0 2024-03-18T16:12:22.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.067+0000 7fd86ed7a740 1 -- 172.21.15.39:0/4126565452 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ff84977ac0 msgr2=0x55ff849794a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:22.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.068+0000 7fd86ed7a740 1 --2- 172.21.15.39:0/4126565452 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ff84977ac0 0x55ff849794a0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x55ff84978320 tx=0x7fd850005f50 comp rx=0 tx=0).stop 2024-03-18T16:12:22.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.068+0000 7fd86ed7a740 1 -- 172.21.15.39:0/4126565452 shutdown_connections 2024-03-18T16:12:22.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.069+0000 7fd86ed7a740 1 --2- 172.21.15.39:0/4126565452 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ff8497cc30 0x55ff849885e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:22.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.070+0000 7fd86ed7a740 1 --2- 172.21.15.39:0/4126565452 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ff84977ac0 0x55ff849794a0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:22.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.070+0000 7fd86ed7a740 1 --2- 172.21.15.39:0/4126565452 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ff849770d0 0x55ff84978f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:22.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.070+0000 7fd86ed7a740 1 -- 172.21.15.39:0/4126565452 >> 172.21.15.39:0/4126565452 conn(0x55ff84972640 msgr2=0x55ff849810a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:22.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.070+0000 7fd86ed7a740 1 -- 172.21.15.39:0/4126565452 shutdown_connections 2024-03-18T16:12:22.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.070+0000 7fd86ed7a740 1 -- 172.21.15.39:0/4126565452 wait complete. 2024-03-18T16:12:22.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.071+0000 7fd86ed7a740 1 -- start start 2024-03-18T16:12:22.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.072+0000 7fd86ed7a740 1 Processor -- start 2024-03-18T16:12:22.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.074+0000 7fd86ed7a740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ff849770d0 0x55ff84985790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:22.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.075+0000 7fd86ed7a740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ff84977ac0 0x55ff84985cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:22.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.075+0000 7fd86ed7a740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ff8497cc30 0x55ff84986c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:22.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.075+0000 7fd86ed7a740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fd8641b1e70 con 0x55ff849770d0 2024-03-18T16:12:22.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.075+0000 7fd86ed7a740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fd8641b1740 con 0x55ff8497cc30 2024-03-18T16:12:22.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.075+0000 7fd86ed7a740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fd8641ae980 con 0x55ff84977ac0 2024-03-18T16:12:22.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.075+0000 7fd86ed7a740 10 client.0 _ll_register_callbacks cb 0x7fd85c0b8cd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:12:22.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.076+0000 7fd86a000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ff8497cc30 0x55ff84986c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:22.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.076+0000 7fd86a000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ff8497cc30 0x55ff84986c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:55172/0 (socket says 172.21.15.39:55172) 2024-03-18T16:12:22.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.076+0000 7fd86a000640 1 -- 172.21.15.39:0/1348868126 learned_addr learned my addr 172.21.15.39:0/1348868126 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:12:22.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.076+0000 7fd868c00640 1 --2- 172.21.15.39:0/1348868126 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ff84977ac0 0x55ff84985cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:22.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.077+0000 7fd86a000640 1 -- 172.21.15.39:0/1348868126 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ff84977ac0 msgr2=0x55ff84985cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:12:22.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.077+0000 7fd86a000640 1 --2- 172.21.15.39:0/1348868126 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ff84977ac0 0x55ff84985cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:22.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.077+0000 7fd86a000640 1 -- 172.21.15.39:0/1348868126 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ff849770d0 msgr2=0x55ff84985790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:12:22.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.077+0000 7fd86a000640 1 --2- 172.21.15.39:0/1348868126 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ff849770d0 0x55ff84985790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:22.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.077+0000 7fd86a000640 1 -- 172.21.15.39:0/1348868126 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ff84987280 con 0x55ff8497cc30 2024-03-18T16:12:22.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.077+0000 7fd86a000640 1 --2- 172.21.15.39:0/1348868126 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ff8497cc30 0x55ff84986c90 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fd85c0edf90 tx=0x7fd85c02e040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=580c82b65a828d2b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:22.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.077+0000 7fd84aa00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:12:22.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.078+0000 7fd84aa00640 1 -- 172.21.15.39:0/1348868126 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd85c0d9040 con 0x55ff8497cc30 2024-03-18T16:12:22.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.078+0000 7fd84aa00640 1 -- 172.21.15.39:0/1348868126 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fd85c0db2a0 con 0x55ff8497cc30 2024-03-18T16:12:22.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.078+0000 7fd86ed7a740 1 -- 172.21.15.39:0/1348868126 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd850012450 con 0x55ff8497cc30 2024-03-18T16:12:22.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.079+0000 7fd84aa00640 1 -- 172.21.15.39:0/1348868126 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fd85c0bc600 con 0x55ff8497cc30 2024-03-18T16:12:22.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.080+0000 7fd86ed7a740 10 client.26069 fetch_fsmap learned FSMap version 32 2024-03-18T16:12:22.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.080+0000 7fd86ed7a740 1 -- 172.21.15.39:0/1348868126 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55ff84988580 con 0x55ff8497cc30 2024-03-18T16:12:22.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.081+0000 7fd84aa00640 1 -- 172.21.15.39:0/1348868126 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd85c0bcac0 con 0x55ff8497cc30 2024-03-18T16:12:22.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.082+0000 7fd84aa00640 1 -- 172.21.15.39:0/1348868126 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7fd85c0d5b80 con 0x55ff8497cc30 2024-03-18T16:12:22.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.082+0000 7fd86ed7a740 10 client.26069 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T16:12:22.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.082+0000 7fd86ed7a740 10 client.26069 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-18T16:12:22.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.082+0000 7fd86ed7a740 20 client.26069 populate_metadata read hostname 'smithi039' 2024-03-18T16:12:22.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.082+0000 7fd86ed7a740 1 --2- 172.21.15.39:0/1348868126 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x55ff849f2ec0 0x55ff849f52e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:22.084 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.083+0000 7fd86ed7a740 4 client.26069 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:12:22.085 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.083+0000 7fd86ed7a740 1 -- 172.21.15.39:0/1348868126 --> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7fd850012450 con 0x55ff849f2ec0 2024-03-18T16:12:22.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.084+0000 7fd869600640 1 --2- 172.21.15.39:0/1348868126 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x55ff849f2ec0 0x55ff849f52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:22.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.088+0000 7fd869600640 1 --2- 172.21.15.39:0/1348868126 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x55ff849f2ec0 0x55ff849f52e0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=979ab4e2fd4ad9ab server_cookie=38d58b9ab42b6967 in_seq=0 out_seq=0 2024-03-18T16:12:22.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.088+0000 7fd84aa00640 10 client.26069 ms_handle_connect on v2:172.21.15.194:6836/1301464246 2024-03-18T16:12:22.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.088+0000 7fd84aa00640 1 -- 172.21.15.39:0/1348868126 <== mds.3 v2:172.21.15.194:6836/1301464246 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7fd850012450 con 0x55ff849f2ec0 2024-03-18T16:12:22.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.088+0000 7fd84aa00640 10 client.26069 handle_command_reply: tid=0 2024-03-18T16:12:22.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.088+0000 7fd86ed7a740 1 client.26069 shutdown 2024-03-18T16:12:22.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.089+0000 7fd86ed7a740 1 -- 172.21.15.39:0/1348868126 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ff8497cc30 msgr2=0x55ff84986c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:22.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.089+0000 7fd86ed7a740 1 --2- 172.21.15.39:0/1348868126 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ff8497cc30 0x55ff84986c90 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fd85c0edf90 tx=0x7fd85c02e040 comp rx=0 tx=0).stop 2024-03-18T16:12:22.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.089+0000 7fd86ed7a740 1 -- 172.21.15.39:0/1348868126 shutdown_connections 2024-03-18T16:12:22.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.089+0000 7fd86ed7a740 1 --2- 172.21.15.39:0/1348868126 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x55ff849f2ec0 0x55ff849f52e0 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:22.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.089+0000 7fd86ed7a740 1 --2- 172.21.15.39:0/1348868126 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ff8497cc30 0x55ff84986c90 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:22.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.089+0000 7fd86ed7a740 1 --2- 172.21.15.39:0/1348868126 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ff84977ac0 0x55ff84985cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:22.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.089+0000 7fd86ed7a740 1 --2- 172.21.15.39:0/1348868126 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ff849770d0 0x55ff84985790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:22.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.089+0000 7fd86ed7a740 1 -- 172.21.15.39:0/1348868126 >> 172.21.15.39:0/1348868126 conn(0x55ff84972640 msgr2=0x55ff84980ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:22.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.089+0000 7fd84aa00640 0 client.26069 ms_handle_reset on v2:172.21.15.194:6836/1301464246 2024-03-18T16:12:22.092 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:12:22.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.090+0000 7fd86ed7a740 1 -- 172.21.15.39:0/1348868126 shutdown_connections 2024-03-18T16:12:22.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.090+0000 7fd86ed7a740 1 -- 172.21.15.39:0/1348868126 wait complete. 2024-03-18T16:12:22.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.090+0000 7fd86ed7a740 20 client.26069 trim_cache size 0 max 16384 2024-03-18T16:12:22.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.091+0000 7fd86c000640 1 -- 172.21.15.39:0/2284379665 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd84c0a2950 msgr2=0x7fd84c0a4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:22.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.091+0000 7fd86c000640 1 --2- 172.21.15.39:0/2284379665 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd84c0a2950 0x7fd84c0a4e10 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fd858005d00 tx=0x7fd858005c50 comp rx=0 tx=0).stop 2024-03-18T16:12:22.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.091+0000 7fd86c000640 1 -- 172.21.15.39:0/2284379665 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd8641055b0 msgr2=0x7fd8641a9d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:22.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.091+0000 7fd86c000640 1 --2- 172.21.15.39:0/2284379665 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd8641055b0 0x7fd8641a9d90 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fd85c005e30 tx=0x7fd85c032ec0 comp rx=0 tx=0).stop 2024-03-18T16:12:22.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.091+0000 7fd86c000640 1 -- 172.21.15.39:0/2284379665 shutdown_connections 2024-03-18T16:12:22.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.091+0000 7fd86c000640 1 --2- 172.21.15.39:0/2284379665 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd84c0a2950 0x7fd84c0a4e10 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:22.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.091+0000 7fd86c000640 1 --2- 172.21.15.39:0/2284379665 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd8641055b0 0x7fd8641a9d90 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:22.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.091+0000 7fd86c000640 1 --2- 172.21.15.39:0/2284379665 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd864104bf0 0x7fd864070670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:22.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.091+0000 7fd86c000640 1 --2- 172.21.15.39:0/2284379665 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd864103a00 0x7fd864070130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:22.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.091+0000 7fd86c000640 1 -- 172.21.15.39:0/2284379665 >> 172.21.15.39:0/2284379665 conn(0x7fd8640fef90 msgr2=0x7fd864107e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:22.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.091+0000 7fd86c000640 1 -- 172.21.15.39:0/2284379665 shutdown_connections 2024-03-18T16:12:22.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.091+0000 7fd86c000640 1 -- 172.21.15.39:0/2284379665 wait complete. 2024-03-18T16:12:22.113 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-18T16:12:22.231 INFO:tasks.workunit.client.0.smithi039.stdout:Syncing()...7 sec 2024-03-18T16:12:22.233 INFO:tasks.workunit.client.0.smithi039.stdout:Starting Actual Benchmark At: Mon Mar 18 16:12:22 2024 2024-03-18T16:12:22.233 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-03-18T16:12:22.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.724+0000 7fc773a00640 1 -- 172.21.15.39:0/1056996227 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc76c10b150 msgr2=0x7fc76c10d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:22.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.724+0000 7fc773a00640 1 --2- 172.21.15.39:0/1056996227 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc76c10b150 0x7fc76c10d5c0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fc75801a910 tx=0x7fc75803cc80 comp rx=0 tx=0).stop 2024-03-18T16:12:22.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.729+0000 7fc773a00640 1 -- 172.21.15.39:0/1056996227 shutdown_connections 2024-03-18T16:12:22.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.729+0000 7fc773a00640 1 --2- 172.21.15.39:0/1056996227 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc76c10db00 0x7fc76c10df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:22.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.729+0000 7fc773a00640 1 --2- 172.21.15.39:0/1056996227 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc76c10b150 0x7fc76c10d5c0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:22.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.729+0000 7fc773a00640 1 --2- 172.21.15.39:0/1056996227 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc76c1087e0 0x7fc76c10ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:22.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.729+0000 7fc773a00640 1 -- 172.21.15.39:0/1056996227 >> 172.21.15.39:0/1056996227 conn(0x7fc76c06e860 msgr2=0x7fc76c070ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:22.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.743+0000 7fc773a00640 1 -- 172.21.15.39:0/1056996227 shutdown_connections 2024-03-18T16:12:22.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.744+0000 7fc773a00640 1 -- 172.21.15.39:0/1056996227 wait complete. 2024-03-18T16:12:22.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.745+0000 7fc773a00640 1 Processor -- start 2024-03-18T16:12:22.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.746+0000 7fc773a00640 1 -- start start 2024-03-18T16:12:22.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.751+0000 7fc773a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc76c1087e0 0x7fc76c19df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:22.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.751+0000 7fc773a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc76c10b150 0x7fc76c19e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:22.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.751+0000 7fc773a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc76c10db00 0x7fc76c1a5520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:22.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.751+0000 7fc773a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fc76c111af0 con 0x7fc76c1087e0 2024-03-18T16:12:22.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.751+0000 7fc773a00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fc76c111970 con 0x7fc76c10db00 2024-03-18T16:12:22.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.751+0000 7fc773a00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fc76c111c70 con 0x7fc76c10b150 2024-03-18T16:12:22.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.752+0000 7fc771a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc76c10db00 0x7fc76c1a5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:22.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.752+0000 7fc771a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc76c10db00 0x7fc76c1a5520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:55200/0 (socket says 172.21.15.39:55200) 2024-03-18T16:12:22.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.752+0000 7fc771a00640 1 -- 172.21.15.39:0/2910129808 learned_addr learned my addr 172.21.15.39:0/2910129808 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:12:22.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.752+0000 7fc771000640 1 --2- 172.21.15.39:0/2910129808 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc76c1087e0 0x7fc76c19df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:22.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.753+0000 7fc771000640 1 -- 172.21.15.39:0/2910129808 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc76c10b150 msgr2=0x7fc76c19e4b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T16:12:22.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.753+0000 7fc771000640 1 --2- 172.21.15.39:0/2910129808 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc76c10b150 0x7fc76c19e4b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:22.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.753+0000 7fc771000640 1 -- 172.21.15.39:0/2910129808 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc76c10db00 msgr2=0x7fc76c1a5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:22.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.753+0000 7fc771000640 1 --2- 172.21.15.39:0/2910129808 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc76c10db00 0x7fc76c1a5520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:22.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.753+0000 7fc771000640 1 -- 172.21.15.39:0/2910129808 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc76c1a5b90 con 0x7fc76c1087e0 2024-03-18T16:12:22.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.753+0000 7fc771a00640 1 --2- 172.21.15.39:0/2910129808 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc76c10db00 0x7fc76c1a5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:12:22.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.754+0000 7fc771000640 1 --2- 172.21.15.39:0/2910129808 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc76c1087e0 0x7fc76c19df70 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc75c00b750 tx=0x7fc75c00bc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:22.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.757+0000 7fc769600640 1 -- 172.21.15.39:0/2910129808 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc75c019070 con 0x7fc76c1087e0 2024-03-18T16:12:22.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.757+0000 7fc769600640 1 -- 172.21.15.39:0/2910129808 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc75c00cc00 con 0x7fc76c1087e0 2024-03-18T16:12:22.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.757+0000 7fc769600640 1 -- 172.21.15.39:0/2910129808 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc75c012340 con 0x7fc76c1087e0 2024-03-18T16:12:22.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.761+0000 7fc773a00640 1 -- 172.21.15.39:0/2910129808 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc76c1a5e80 con 0x7fc76c1087e0 2024-03-18T16:12:22.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.761+0000 7fc773a00640 1 -- 172.21.15.39:0/2910129808 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc76c1a6310 con 0x7fc76c1087e0 2024-03-18T16:12:22.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.787+0000 7fc769600640 1 -- 172.21.15.39:0/2910129808 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fc75c00c720 con 0x7fc76c1087e0 2024-03-18T16:12:22.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.788+0000 7fc77660a740 1 Processor -- start 2024-03-18T16:12:22.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.789+0000 7fc77660a740 1 -- start start 2024-03-18T16:12:22.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.789+0000 7fc77660a740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563bdc59a8c0 0x563bdc59ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:22.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.789+0000 7fc77660a740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563bdc59b2b0 0x563bdc59fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:22.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.789+0000 7fc77660a740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563bdc5a0420 0x563bdc5a28b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:22.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.789+0000 7fc77660a740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x563bdc5623b0 con 0x563bdc5a0420 2024-03-18T16:12:22.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.789+0000 7fc77660a740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x563bdc4ff450 con 0x563bdc59b2b0 2024-03-18T16:12:22.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.789+0000 7fc77660a740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x563bdc4a7cb0 con 0x563bdc59a8c0 2024-03-18T16:12:22.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.790+0000 7fc76be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563bdc5a0420 0x563bdc5a28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:22.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.790+0000 7fc76be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563bdc5a0420 0x563bdc5a28b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:42008/0 (socket says 172.21.15.39:42008) 2024-03-18T16:12:22.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.790+0000 7fc76be00640 1 -- 172.21.15.39:0/4259388832 learned_addr learned my addr 172.21.15.39:0/4259388832 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:12:22.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.791+0000 7fc76be00640 1 -- 172.21.15.39:0/4259388832 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563bdc59a8c0 msgr2=0x563bdc59ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:12:22.795 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.791+0000 7fc771a00640 1 --2- 172.21.15.39:0/4259388832 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563bdc59a8c0 0x563bdc59ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:22.795 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.793+0000 7fc76be00640 1 --2- 172.21.15.39:0/4259388832 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563bdc59a8c0 0x563bdc59ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:22.795 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.793+0000 7fc76be00640 1 -- 172.21.15.39:0/4259388832 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563bdc59b2b0 msgr2=0x563bdc59fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:12:22.796 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.794+0000 7fc771000640 1 --2- 172.21.15.39:0/4259388832 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563bdc59b2b0 0x563bdc59fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:22.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.795+0000 7fc76be00640 1 --2- 172.21.15.39:0/4259388832 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563bdc59b2b0 0x563bdc59fdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:22.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.795+0000 7fc76be00640 1 -- 172.21.15.39:0/4259388832 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563bdc4c91f0 con 0x563bdc5a0420 2024-03-18T16:12:22.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.796+0000 7fc76be00640 1 --2- 172.21.15.39:0/4259388832 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563bdc5a0420 0x563bdc5a28b0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc758004250 tx=0x7fc758004280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d106ea78f60b680 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:22.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.802+0000 7fc771a00640 1 --2- 172.21.15.39:0/4259388832 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563bdc59a8c0 0x563bdc59ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:12:22.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.802+0000 7fc771000640 1 --2- 172.21.15.39:0/4259388832 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563bdc59b2b0 0x563bdc59fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:12:22.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.807+0000 7fc769600640 1 --2- 172.21.15.39:0/2910129808 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fc7500a2a00 0x7fc7500a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:22.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.807+0000 7fc769600640 1 -- 172.21.15.39:0/2910129808 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fc75c0cdc90 con 0x7fc76c1087e0 2024-03-18T16:12:22.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.808+0000 7fc771a00640 1 --2- 172.21.15.39:0/2910129808 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fc7500a2a00 0x7fc7500a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:22.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.808+0000 7fc762000640 1 -- 172.21.15.39:0/4259388832 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc758052020 con 0x563bdc5a0420 2024-03-18T16:12:22.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.808+0000 7fc762000640 1 -- 172.21.15.39:0/4259388832 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc75801a530 con 0x563bdc5a0420 2024-03-18T16:12:22.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.808+0000 7fc762000640 1 -- 172.21.15.39:0/4259388832 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc758003b20 con 0x563bdc5a0420 2024-03-18T16:12:22.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.809+0000 7fc771a00640 1 --2- 172.21.15.39:0/2910129808 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fc7500a2a00 0x7fc7500a4ec0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x563bdc59bb10 tx=0x7fc764009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:22.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.813+0000 7fc77660a740 1 -- 172.21.15.39:0/4259388832 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563bdc5a0420 msgr2=0x563bdc5a28b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:22.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.813+0000 7fc77660a740 1 --2- 172.21.15.39:0/4259388832 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563bdc5a0420 0x563bdc5a28b0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc758004250 tx=0x7fc758004280 comp rx=0 tx=0).stop 2024-03-18T16:12:22.825 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.823+0000 7fc77660a740 1 -- 172.21.15.39:0/4259388832 shutdown_connections 2024-03-18T16:12:22.825 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.823+0000 7fc77660a740 1 --2- 172.21.15.39:0/4259388832 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563bdc59b2b0 0x563bdc59fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:22.825 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.823+0000 7fc77660a740 1 --2- 172.21.15.39:0/4259388832 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563bdc5a0420 0x563bdc5a28b0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:22.825 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.823+0000 7fc77660a740 1 --2- 172.21.15.39:0/4259388832 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563bdc59a8c0 0x563bdc59ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:22.825 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.824+0000 7fc77660a740 1 -- 172.21.15.39:0/4259388832 >> 172.21.15.39:0/4259388832 conn(0x563bdc595e30 msgr2=0x563bdc596230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:22.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.824+0000 7fc77660a740 1 -- 172.21.15.39:0/4259388832 shutdown_connections 2024-03-18T16:12:22.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.824+0000 7fc77660a740 1 -- 172.21.15.39:0/4259388832 wait complete. 2024-03-18T16:12:22.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.825+0000 7fc77660a740 1 -- start start 2024-03-18T16:12:22.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.825+0000 7fc77660a740 1 Processor -- start 2024-03-18T16:12:22.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.826+0000 7fc77660a740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563bdc59a8c0 0x563bdc5a4b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:22.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.826+0000 7fc77660a740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563bdc59b2b0 0x563bdc5a5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:22.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.826+0000 7fc771000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563bdc59a8c0 0x563bdc5a4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:22.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.827+0000 7fc771000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563bdc59a8c0 0x563bdc5a4b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:42014/0 (socket says 172.21.15.39:42014) 2024-03-18T16:12:22.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.829+0000 7fc771000640 1 -- 172.21.15.39:0/1433662448 learned_addr learned my addr 172.21.15.39:0/1433662448 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:12:22.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.829+0000 7fc77660a740 1 --2- 172.21.15.39:0/1433662448 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563bdc5a0420 0x563bdc5ac080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:22.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.829+0000 7fc76be00640 1 --2- 172.21.15.39:0/1433662448 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563bdc59b2b0 0x563bdc5a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:22.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.829+0000 7fc77660a740 1 -- 172.21.15.39:0/1433662448 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x563bdc4bb110 con 0x563bdc59a8c0 2024-03-18T16:12:22.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.829+0000 7fc77660a740 1 -- 172.21.15.39:0/1433662448 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x563bdc4fb4f0 con 0x563bdc59b2b0 2024-03-18T16:12:22.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.829+0000 7fc77660a740 1 -- 172.21.15.39:0/1433662448 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x563bdc5a3370 con 0x563bdc5a0420 2024-03-18T16:12:22.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.830+0000 7fc77660a740 10 client.0 _ll_register_callbacks cb 0x563bdc4a8b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:12:22.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.830+0000 7fc771a00640 1 --2- 172.21.15.39:0/1433662448 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563bdc5a0420 0x563bdc5ac080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:22.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.831+0000 7fc76be00640 1 -- 172.21.15.39:0/1433662448 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563bdc5a0420 msgr2=0x563bdc5ac080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:12:22.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.831+0000 7fc76be00640 1 --2- 172.21.15.39:0/1433662448 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563bdc5a0420 0x563bdc5ac080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:22.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.831+0000 7fc76be00640 1 -- 172.21.15.39:0/1433662448 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563bdc59a8c0 msgr2=0x563bdc5a4b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:12:22.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.831+0000 7fc76be00640 1 --2- 172.21.15.39:0/1433662448 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563bdc59a8c0 0x563bdc5a4b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:22.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.831+0000 7fc76be00640 1 -- 172.21.15.39:0/1433662448 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563bdc2133f0 con 0x563bdc59b2b0 2024-03-18T16:12:22.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.831+0000 7fc771000640 1 --2- 172.21.15.39:0/1433662448 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563bdc59a8c0 0x563bdc5a4b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:12:22.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.838+0000 7fc76be00640 1 --2- 172.21.15.39:0/1433662448 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563bdc59b2b0 0x563bdc5a5040 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fc75801a840 tx=0x7fc7580047a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=16d30b3197bbe9df server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:22.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.842+0000 7fc756a00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:12:22.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.842+0000 7fc756a00640 1 -- 172.21.15.39:0/1433662448 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc758052070 con 0x563bdc59b2b0 2024-03-18T16:12:22.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.844+0000 7fc756a00640 1 -- 172.21.15.39:0/1433662448 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fc758003ab0 con 0x563bdc59b2b0 2024-03-18T16:12:22.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.854+0000 7fc77660a740 1 -- 172.21.15.39:0/1433662448 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc75801a530 con 0x563bdc59b2b0 2024-03-18T16:12:22.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.864+0000 7fc756a00640 1 -- 172.21.15.39:0/1433662448 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fc758066050 con 0x563bdc59b2b0 2024-03-18T16:12:22.867 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.864+0000 7fc77660a740 10 client.26075 fetch_fsmap learned FSMap version 32 2024-03-18T16:12:22.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.864+0000 7fc77660a740 1 -- 172.21.15.39:0/1433662448 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x563bdc2dd7d0 con 0x563bdc59b2b0 2024-03-18T16:12:22.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.868+0000 7fc756a00640 1 -- 172.21.15.39:0/1433662448 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc75801a530 con 0x563bdc59b2b0 2024-03-18T16:12:22.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.868+0000 7fc756a00640 1 -- 172.21.15.39:0/1433662448 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7fc75804e480 con 0x563bdc59b2b0 2024-03-18T16:12:22.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.875+0000 7fc77660a740 10 client.26075 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T16:12:22.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.875+0000 7fc77660a740 10 client.26075 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-18T16:12:22.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.875+0000 7fc77660a740 20 client.26075 populate_metadata read hostname 'smithi039' 2024-03-18T16:12:22.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.875+0000 7fc77660a740 1 --2- 172.21.15.39:0/1433662448 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x563bdc5b04e0 0x563bdc5b28d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:22.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.875+0000 7fc77660a740 4 client.26075 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:12:22.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.875+0000 7fc77660a740 1 -- 172.21.15.39:0/1433662448 --> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563bdc50c990 con 0x563bdc5b04e0 2024-03-18T16:12:22.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.876+0000 7fc771000640 1 --2- 172.21.15.39:0/1433662448 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x563bdc5b04e0 0x563bdc5b28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:22.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.898+0000 7fc771000640 1 --2- 172.21.15.39:0/1433662448 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x563bdc5b04e0 0x563bdc5b28d0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=413817ae8e566d70 server_cookie=8d5c91413b189c55 in_seq=0 out_seq=0 2024-03-18T16:12:22.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.898+0000 7fc756a00640 10 client.26075 ms_handle_connect on v2:172.21.15.196:6836/895247251 2024-03-18T16:12:22.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.900+0000 7fc756a00640 1 -- 172.21.15.39:0/1433662448 <== mds.2 v2:172.21.15.196:6836/895247251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563bdc50c990 con 0x563bdc5b04e0 2024-03-18T16:12:22.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.900+0000 7fc756a00640 10 client.26075 handle_command_reply: tid=0 2024-03-18T16:12:22.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.905+0000 7fc77660a740 1 client.26075 shutdown 2024-03-18T16:12:22.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.909+0000 7fc77660a740 1 -- 172.21.15.39:0/1433662448 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563bdc59b2b0 msgr2=0x563bdc5a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:22.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.909+0000 7fc77660a740 1 --2- 172.21.15.39:0/1433662448 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563bdc59b2b0 0x563bdc5a5040 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fc75801a840 tx=0x7fc7580047a0 comp rx=0 tx=0).stop 2024-03-18T16:12:22.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.909+0000 7fc77660a740 1 -- 172.21.15.39:0/1433662448 shutdown_connections 2024-03-18T16:12:22.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.909+0000 7fc77660a740 1 --2- 172.21.15.39:0/1433662448 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563bdc59b2b0 0x563bdc5a5040 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:22.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.909+0000 7fc77660a740 1 --2- 172.21.15.39:0/1433662448 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x563bdc5b04e0 0x563bdc5b28d0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:22.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.909+0000 7fc77660a740 1 --2- 172.21.15.39:0/1433662448 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563bdc5a0420 0x563bdc5ac080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:22.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.909+0000 7fc77660a740 1 --2- 172.21.15.39:0/1433662448 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563bdc59a8c0 0x563bdc5a4b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:22.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.909+0000 7fc77660a740 1 -- 172.21.15.39:0/1433662448 >> 172.21.15.39:0/1433662448 conn(0x563bdc595e30 msgr2=0x563bdc5951f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:22.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.909+0000 7fc756a00640 0 client.26075 ms_handle_reset on v2:172.21.15.196:6836/895247251 2024-03-18T16:12:22.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.910+0000 7fc77660a740 1 -- 172.21.15.39:0/1433662448 shutdown_connections 2024-03-18T16:12:22.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.910+0000 7fc77660a740 1 -- 172.21.15.39:0/1433662448 wait complete. 2024-03-18T16:12:22.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:22.912+0000 7fc77660a740 20 client.26075 trim_cache size 0 max 16384 2024-03-18T16:12:23.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.018+0000 7fc77660a740 1 Processor -- start 2024-03-18T16:12:23.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.025+0000 7fc77660a740 1 -- start start 2024-03-18T16:12:23.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.026+0000 7fc77660a740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563bdc595e30 0x563bdc594960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:23.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.026+0000 7fc77660a740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563bdc59a8c0 0x563bdc5ac040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:23.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.028+0000 7fc77660a740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563bdc59b2b0 0x563bdc594ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:23.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.028+0000 7fc77660a740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fc76c1b0980 con 0x563bdc59b2b0 2024-03-18T16:12:23.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.028+0000 7fc77660a740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fc76c1b2580 con 0x563bdc595e30 2024-03-18T16:12:23.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.028+0000 7fc77660a740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fc76c1b1050 con 0x563bdc59a8c0 2024-03-18T16:12:23.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.028+0000 7fc76be00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563bdc59a8c0 0x563bdc5ac040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:23.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.028+0000 7fc76be00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563bdc59a8c0 0x563bdc5ac040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:54582/0 (socket says 172.21.15.39:54582) 2024-03-18T16:12:23.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.028+0000 7fc76be00640 1 -- 172.21.15.39:0/872583034 learned_addr learned my addr 172.21.15.39:0/872583034 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:12:23.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.028+0000 7fc771a00640 1 --2- 172.21.15.39:0/872583034 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563bdc59b2b0 0x563bdc594ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:23.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.028+0000 7fc76be00640 1 -- 172.21.15.39:0/872583034 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563bdc595e30 msgr2=0x563bdc594960 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T16:12:23.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.028+0000 7fc76be00640 1 --2- 172.21.15.39:0/872583034 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563bdc595e30 0x563bdc594960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:23.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.028+0000 7fc76be00640 1 -- 172.21.15.39:0/872583034 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563bdc59b2b0 msgr2=0x563bdc594ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:12:23.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.029+0000 7fc76be00640 1 --2- 172.21.15.39:0/872583034 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563bdc59b2b0 0x563bdc594ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:23.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.029+0000 7fc76be00640 1 -- 172.21.15.39:0/872583034 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563bdc591c10 con 0x563bdc59a8c0 2024-03-18T16:12:23.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.029+0000 7fc76be00640 1 --2- 172.21.15.39:0/872583034 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563bdc59a8c0 0x563bdc5ac040 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fc75804e850 tx=0x7fc758005850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cc94b6643c55473e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:23.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.030+0000 7fc761600640 1 -- 172.21.15.39:0/872583034 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc758070070 con 0x563bdc59a8c0 2024-03-18T16:12:23.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.030+0000 7fc761600640 1 -- 172.21.15.39:0/872583034 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc75801a0c0 con 0x563bdc59a8c0 2024-03-18T16:12:23.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.030+0000 7fc761600640 1 -- 172.21.15.39:0/872583034 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc758052070 con 0x563bdc59a8c0 2024-03-18T16:12:23.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.034+0000 7fc77660a740 1 -- 172.21.15.39:0/872583034 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563bdc59a8c0 msgr2=0x563bdc5ac040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:23.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.035+0000 7fc77660a740 1 --2- 172.21.15.39:0/872583034 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563bdc59a8c0 0x563bdc5ac040 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fc75804e850 tx=0x7fc758005850 comp rx=0 tx=0).stop 2024-03-18T16:12:23.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.035+0000 7fc77660a740 1 -- 172.21.15.39:0/872583034 shutdown_connections 2024-03-18T16:12:23.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.035+0000 7fc77660a740 1 --2- 172.21.15.39:0/872583034 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563bdc59b2b0 0x563bdc594ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:23.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.035+0000 7fc77660a740 1 --2- 172.21.15.39:0/872583034 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563bdc59a8c0 0x563bdc5ac040 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:23.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.036+0000 7fc77660a740 1 --2- 172.21.15.39:0/872583034 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563bdc595e30 0x563bdc594960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:23.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.036+0000 7fc77660a740 1 -- 172.21.15.39:0/872583034 >> 172.21.15.39:0/872583034 conn(0x563bdc5b2c40 msgr2=0x563bdc591080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:23.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.039+0000 7fc77660a740 1 -- 172.21.15.39:0/872583034 shutdown_connections 2024-03-18T16:12:23.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.039+0000 7fc77660a740 1 -- 172.21.15.39:0/872583034 wait complete. 2024-03-18T16:12:23.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.039+0000 7fc77660a740 1 -- start start 2024-03-18T16:12:23.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.040+0000 7fc77660a740 1 Processor -- start 2024-03-18T16:12:23.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.042+0000 7fc77660a740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563bdc595e30 0x563bdc5a9c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:23.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.042+0000 7fc77660a740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563bdc59a8c0 0x563bdc5aa1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:23.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.045+0000 7fc771000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563bdc595e30 0x563bdc5a9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:23.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.045+0000 7fc77660a740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563bdc59b2b0 0x563bdc5ab1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:23.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.045+0000 7fc77660a740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fc76c1b1e50 con 0x563bdc595e30 2024-03-18T16:12:23.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.045+0000 7fc771000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563bdc595e30 0x563bdc5a9c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:42028/0 (socket says 172.21.15.39:42028) 2024-03-18T16:12:23.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.045+0000 7fc771000640 1 -- 172.21.15.39:0/3807644730 learned_addr learned my addr 172.21.15.39:0/3807644730 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:12:23.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.045+0000 7fc77660a740 1 -- 172.21.15.39:0/3807644730 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fc76c1b1720 con 0x563bdc59b2b0 2024-03-18T16:12:23.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.045+0000 7fc77660a740 1 -- 172.21.15.39:0/3807644730 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fc76c1ae960 con 0x563bdc59a8c0 2024-03-18T16:12:23.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.043+0000 7fc76be00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563bdc59a8c0 0x563bdc5aa1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:23.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.045+0000 7fc77660a740 10 client.0 _ll_register_callbacks cb 0x7fc75801a790 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:12:23.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.046+0000 7fc771000640 1 -- 172.21.15.39:0/3807644730 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563bdc59a8c0 msgr2=0x563bdc5aa1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:12:23.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.045+0000 7fc771a00640 1 --2- 172.21.15.39:0/3807644730 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563bdc59b2b0 0x563bdc5ab1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:23.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.046+0000 7fc771000640 1 --2- 172.21.15.39:0/3807644730 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563bdc59a8c0 0x563bdc5aa1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:23.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.046+0000 7fc771000640 1 -- 172.21.15.39:0/3807644730 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563bdc59b2b0 msgr2=0x563bdc5ab1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:12:23.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.046+0000 7fc771000640 1 --2- 172.21.15.39:0/3807644730 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563bdc59b2b0 0x563bdc5ab1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:23.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.046+0000 7fc771000640 1 -- 172.21.15.39:0/3807644730 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563bdc5a53b0 con 0x563bdc595e30 2024-03-18T16:12:23.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.046+0000 7fc771a00640 1 --2- 172.21.15.39:0/3807644730 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563bdc59b2b0 0x563bdc5ab1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:12:23.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.047+0000 7fc76be00640 1 --2- 172.21.15.39:0/3807644730 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563bdc59a8c0 0x563bdc5aa1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:12:23.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.047+0000 7fc771000640 1 --2- 172.21.15.39:0/3807644730 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563bdc595e30 0x563bdc5a9c80 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fc75c004790 tx=0x7fc75c0cb270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16a598fac9270270 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:23.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.047+0000 7fc756a00640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-03-18T16:12:23.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.048+0000 7fc756a00640 1 -- 172.21.15.39:0/3807644730 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc75c0a29e0 con 0x563bdc595e30 2024-03-18T16:12:23.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.048+0000 7fc756a00640 1 -- 172.21.15.39:0/3807644730 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fc75c0985d0 con 0x563bdc595e30 2024-03-18T16:12:23.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.048+0000 7fc77660a740 1 -- 172.21.15.39:0/3807644730 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc75801a0c0 con 0x563bdc595e30 2024-03-18T16:12:23.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.048+0000 7fc756a00640 1 -- 172.21.15.39:0/3807644730 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fc75c0a7050 con 0x563bdc595e30 2024-03-18T16:12:23.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.048+0000 7fc77660a740 10 client.15879 fetch_fsmap learned FSMap version 32 2024-03-18T16:12:23.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.048+0000 7fc77660a740 1 -- 172.21.15.39:0/3807644730 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x563bdc616b00 con 0x563bdc595e30 2024-03-18T16:12:23.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.056+0000 7fc756a00640 1 -- 172.21.15.39:0/3807644730 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc75801a0c0 con 0x563bdc595e30 2024-03-18T16:12:23.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.056+0000 7fc756a00640 1 -- 172.21.15.39:0/3807644730 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7fc75c09a530 con 0x563bdc595e30 2024-03-18T16:12:23.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.058+0000 7fc77660a740 10 client.15879 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T16:12:23.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.058+0000 7fc77660a740 10 client.15879 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-18T16:12:23.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.058+0000 7fc77660a740 20 client.15879 populate_metadata read hostname 'smithi039' 2024-03-18T16:12:23.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.058+0000 7fc77660a740 1 --2- 172.21.15.39:0/3807644730 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x563bdc5a0420 0x563bdc619540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:23.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.061+0000 7fc77660a740 4 client.15879 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:12:23.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.061+0000 7fc77660a740 1 -- 172.21.15.39:0/3807644730 --> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563bdc506ce0 con 0x563bdc5a0420 2024-03-18T16:12:23.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.062+0000 7fc76be00640 1 --2- 172.21.15.39:0/3807644730 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x563bdc5a0420 0x563bdc619540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:23.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.063+0000 7fc76be00640 1 --2- 172.21.15.39:0/3807644730 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x563bdc5a0420 0x563bdc619540 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f44144a901042088 server_cookie=77d8ef3ca72321d0 in_seq=0 out_seq=0 2024-03-18T16:12:23.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.070+0000 7fc756a00640 10 client.15879 ms_handle_connect on v2:172.21.15.196:6836/895247251 2024-03-18T16:12:23.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.070+0000 7fc756a00640 1 -- 172.21.15.39:0/3807644730 <== mds.2 v2:172.21.15.196:6836/895247251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563bdc506ce0 con 0x563bdc5a0420 2024-03-18T16:12:23.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.070+0000 7fc756a00640 10 client.15879 handle_command_reply: tid=0 2024-03-18T16:12:23.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.070+0000 7fc77660a740 1 client.15879 shutdown 2024-03-18T16:12:23.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.071+0000 7fc77660a740 1 -- 172.21.15.39:0/3807644730 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563bdc595e30 msgr2=0x563bdc5a9c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:23.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.071+0000 7fc77660a740 1 --2- 172.21.15.39:0/3807644730 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563bdc595e30 0x563bdc5a9c80 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fc75c004790 tx=0x7fc75c0cb270 comp rx=0 tx=0).stop 2024-03-18T16:12:23.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.072+0000 7fc77660a740 1 -- 172.21.15.39:0/3807644730 shutdown_connections 2024-03-18T16:12:23.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.072+0000 7fc77660a740 1 --2- 172.21.15.39:0/3807644730 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563bdc59b2b0 0x563bdc5ab1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:23.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.072+0000 7fc77660a740 1 --2- 172.21.15.39:0/3807644730 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x563bdc5a0420 0x563bdc619540 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:23.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.072+0000 7fc77660a740 1 --2- 172.21.15.39:0/3807644730 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563bdc59a8c0 0x563bdc5aa1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:23.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.072+0000 7fc77660a740 1 --2- 172.21.15.39:0/3807644730 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563bdc595e30 0x563bdc5a9c80 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:23.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.072+0000 7fc77660a740 1 -- 172.21.15.39:0/3807644730 >> 172.21.15.39:0/3807644730 conn(0x563bdc5b2c40 msgr2=0x563bdc594960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:23.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.072+0000 7fc756a00640 0 client.15879 ms_handle_reset on v2:172.21.15.196:6836/895247251 2024-03-18T16:12:23.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.074+0000 7fc77660a740 1 -- 172.21.15.39:0/3807644730 shutdown_connections 2024-03-18T16:12:23.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.074+0000 7fc77660a740 1 -- 172.21.15.39:0/3807644730 wait complete. 2024-03-18T16:12:23.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.075+0000 7fc77660a740 20 client.15879 trim_cache size 0 max 16384 2024-03-18T16:12:23.083 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:12:23.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.094+0000 7fc773a00640 1 -- 172.21.15.39:0/2910129808 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fc7500a2a00 msgr2=0x7fc7500a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:23.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.094+0000 7fc773a00640 1 --2- 172.21.15.39:0/2910129808 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fc7500a2a00 0x7fc7500a4ec0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x563bdc59bb10 tx=0x7fc764009500 comp rx=0 tx=0).stop 2024-03-18T16:12:23.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.094+0000 7fc773a00640 1 -- 172.21.15.39:0/2910129808 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc76c1087e0 msgr2=0x7fc76c19df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:23.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.094+0000 7fc773a00640 1 --2- 172.21.15.39:0/2910129808 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc76c1087e0 0x7fc76c19df70 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc75c00b750 tx=0x7fc75c00bc10 comp rx=0 tx=0).stop 2024-03-18T16:12:23.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.096+0000 7fc773a00640 1 -- 172.21.15.39:0/2910129808 shutdown_connections 2024-03-18T16:12:23.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.096+0000 7fc773a00640 1 --2- 172.21.15.39:0/2910129808 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc76c10db00 0x7fc76c1a5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:23.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.096+0000 7fc773a00640 1 --2- 172.21.15.39:0/2910129808 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fc7500a2a00 0x7fc7500a4ec0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:23.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.096+0000 7fc773a00640 1 --2- 172.21.15.39:0/2910129808 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc76c10b150 0x7fc76c19e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:23.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.096+0000 7fc773a00640 1 --2- 172.21.15.39:0/2910129808 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc76c1087e0 0x7fc76c19df70 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:23.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.096+0000 7fc773a00640 1 -- 172.21.15.39:0/2910129808 >> 172.21.15.39:0/2910129808 conn(0x7fc76c06e860 msgr2=0x7fc76c10b700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:23.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.096+0000 7fc773a00640 1 -- 172.21.15.39:0/2910129808 shutdown_connections 2024-03-18T16:12:23.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:23.101+0000 7fc773a00640 1 -- 172.21.15.39:0/2910129808 wait complete. 2024-03-18T16:12:23.198 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-18T16:12:23.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:12:23 smithi196 ceph-mon[36307]: pgmap v491: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 287 MiB/s wr, 98 op/s 2024-03-18T16:12:23.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:12:23 smithi194 ceph-mon[36200]: pgmap v491: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 287 MiB/s wr, 98 op/s 2024-03-18T16:12:23.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:12:23 smithi039 ceph-mon[31721]: pgmap v491: 129 pgs: 129 active+clean; 11 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 287 MiB/s wr, 98 op/s 2024-03-18T16:12:25.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.096+0000 7febff600640 1 -- 172.21.15.39:0/3509576846 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7febf810db00 msgr2=0x7febf810df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:25.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.096+0000 7febff600640 1 --2- 172.21.15.39:0/3509576846 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7febf810db00 0x7febf810df80 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7febf0009ca0 tx=0x7febf002c100 comp rx=0 tx=0).stop 2024-03-18T16:12:25.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.108+0000 7febff600640 1 -- 172.21.15.39:0/3509576846 shutdown_connections 2024-03-18T16:12:25.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.108+0000 7febff600640 1 --2- 172.21.15.39:0/3509576846 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7febf810db00 0x7febf810df80 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:25.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.108+0000 7febff600640 1 --2- 172.21.15.39:0/3509576846 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7febf810b150 0x7febf810d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:25.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.108+0000 7febff600640 1 --2- 172.21.15.39:0/3509576846 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7febf81087e0 0x7febf810ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:25.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.108+0000 7febff600640 1 -- 172.21.15.39:0/3509576846 >> 172.21.15.39:0/3509576846 conn(0x7febf806e860 msgr2=0x7febf8070ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:25.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.113+0000 7febff600640 1 -- 172.21.15.39:0/3509576846 shutdown_connections 2024-03-18T16:12:25.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.124+0000 7febff600640 1 -- 172.21.15.39:0/3509576846 wait complete. 2024-03-18T16:12:25.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.125+0000 7febff600640 1 Processor -- start 2024-03-18T16:12:25.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.125+0000 7febff600640 1 -- start start 2024-03-18T16:12:25.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.126+0000 7febff600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7febf81087e0 0x7febf8114e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:25.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.126+0000 7febff600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7febf810b150 0x7febf8115350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:25.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.126+0000 7febff600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7febf810db00 0x7febf81a59a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:25.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.126+0000 7febff600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7febf811c2a0 con 0x7febf810db00 2024-03-18T16:12:25.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.126+0000 7febff600640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7febf811c120 con 0x7febf810b150 2024-03-18T16:12:25.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.126+0000 7febff600640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7febf811c420 con 0x7febf81087e0 2024-03-18T16:12:25.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.130+0000 7febf7e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7febf810b150 0x7febf8115350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:25.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.131+0000 7febfd600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7febf810db00 0x7febf81a59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:25.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.131+0000 7febf7e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7febf810b150 0x7febf8115350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:55244/0 (socket says 172.21.15.39:55244) 2024-03-18T16:12:25.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.131+0000 7febf7e00640 1 -- 172.21.15.39:0/3007278886 learned_addr learned my addr 172.21.15.39:0/3007278886 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:12:25.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.131+0000 7febfcc00640 1 --2- 172.21.15.39:0/3007278886 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7febf81087e0 0x7febf8114e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:25.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.137+0000 7febfd600640 1 -- 172.21.15.39:0/3007278886 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7febf81087e0 msgr2=0x7febf8114e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:25.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.137+0000 7febfd600640 1 --2- 172.21.15.39:0/3007278886 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7febf81087e0 0x7febf8114e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:25.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.137+0000 7febfd600640 1 -- 172.21.15.39:0/3007278886 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7febf810b150 msgr2=0x7febf8115350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:25.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.137+0000 7febfd600640 1 --2- 172.21.15.39:0/3007278886 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7febf810b150 0x7febf8115350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:25.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.137+0000 7febfd600640 1 -- 172.21.15.39:0/3007278886 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febf81a5ee0 con 0x7febf810db00 2024-03-18T16:12:25.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.138+0000 7febfd600640 1 --2- 172.21.15.39:0/3007278886 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7febf810db00 0x7febf81a59a0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7febf0037040 tx=0x7febf0035d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:25.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.141+0000 7febf5600640 1 -- 172.21.15.39:0/3007278886 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febf0039040 con 0x7febf810db00 2024-03-18T16:12:25.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.141+0000 7febf5600640 1 -- 172.21.15.39:0/3007278886 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7febf0003970 con 0x7febf810db00 2024-03-18T16:12:25.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.141+0000 7febf5600640 1 -- 172.21.15.39:0/3007278886 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febf0004620 con 0x7febf810db00 2024-03-18T16:12:25.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.141+0000 7febff600640 1 -- 172.21.15.39:0/3007278886 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febf81a6170 con 0x7febf810db00 2024-03-18T16:12:25.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.141+0000 7febff600640 1 -- 172.21.15.39:0/3007278886 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febf81a6660 con 0x7febf810db00 2024-03-18T16:12:25.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.160+0000 7febf5600640 1 -- 172.21.15.39:0/3007278886 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7febf0003b20 con 0x7febf810db00 2024-03-18T16:12:25.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.162+0000 7febf5600640 1 --2- 172.21.15.39:0/3007278886 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7febdc0a29d0 0x7febdc0a4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:25.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.164+0000 7fec0225a740 1 Processor -- start 2024-03-18T16:12:25.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.165+0000 7febf5600640 1 -- 172.21.15.39:0/3007278886 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7febf00f0e60 con 0x7febf810db00 2024-03-18T16:12:25.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.166+0000 7fec0225a740 1 -- start start 2024-03-18T16:12:25.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.167+0000 7fec0225a740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561e6a2b58c0 0x561e6a2b5ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:25.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.167+0000 7fec0225a740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561e6a2b62b0 0x561e6a2badb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:25.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.167+0000 7fec0225a740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561e6a2bb420 0x561e6a2bd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:25.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.167+0000 7fec0225a740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x561e6a27d3b0 con 0x561e6a2bb420 2024-03-18T16:12:25.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.168+0000 7febf7e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561e6a2b62b0 0x561e6a2badb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:25.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.168+0000 7febf7e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561e6a2b62b0 0x561e6a2badb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:54608/0 (socket says 172.21.15.39:54608) 2024-03-18T16:12:25.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.168+0000 7febf7e00640 1 -- 172.21.15.39:0/2411093926 learned_addr learned my addr 172.21.15.39:0/2411093926 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:12:25.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.168+0000 7febfd600640 1 --2- 172.21.15.39:0/2411093926 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561e6a2bb420 0x561e6a2bd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:25.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.168+0000 7fec0225a740 1 -- 172.21.15.39:0/2411093926 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x561e6a21a450 con 0x561e6a2b58c0 2024-03-18T16:12:25.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.168+0000 7fec0225a740 1 -- 172.21.15.39:0/2411093926 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x561e6a1c2cb0 con 0x561e6a2b62b0 2024-03-18T16:12:25.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.174+0000 7febfd600640 1 -- 172.21.15.39:0/2411093926 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561e6a2b62b0 msgr2=0x561e6a2badb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:12:25.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.174+0000 7febfd600640 1 --2- 172.21.15.39:0/2411093926 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561e6a2b62b0 0x561e6a2badb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:25.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.174+0000 7febfd600640 1 -- 172.21.15.39:0/2411093926 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561e6a2b58c0 msgr2=0x561e6a2b5ce0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:12:25.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.174+0000 7febfd600640 1 --2- 172.21.15.39:0/2411093926 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561e6a2b58c0 0x561e6a2b5ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:25.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.174+0000 7febfd600640 1 -- 172.21.15.39:0/2411093926 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561e6a1e41f0 con 0x561e6a2bb420 2024-03-18T16:12:25.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.175+0000 7febfd600640 1 --2- 172.21.15.39:0/2411093926 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561e6a2bb420 0x561e6a2bd8b0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7febf00ee570 tx=0x7febf002c100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=520b4caaff81052f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:25.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.176+0000 7febfcc00640 1 --2- 172.21.15.39:0/3007278886 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7febdc0a29d0 0x7febdc0a4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:25.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.182+0000 7febedc00640 1 -- 172.21.15.39:0/2411093926 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febf00bd410 con 0x561e6a2bb420 2024-03-18T16:12:25.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.182+0000 7febedc00640 1 -- 172.21.15.39:0/2411093926 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7febf00c1560 con 0x561e6a2bb420 2024-03-18T16:12:25.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.182+0000 7febedc00640 1 -- 172.21.15.39:0/2411093926 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febf00b9a80 con 0x561e6a2bb420 2024-03-18T16:12:25.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.189+0000 7fec0225a740 1 -- 172.21.15.39:0/2411093926 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561e6a2bb420 msgr2=0x561e6a2bd8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:25.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.189+0000 7fec0225a740 1 --2- 172.21.15.39:0/2411093926 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561e6a2bb420 0x561e6a2bd8b0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7febf00ee570 tx=0x7febf002c100 comp rx=0 tx=0).stop 2024-03-18T16:12:25.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.189+0000 7febfcc00640 1 --2- 172.21.15.39:0/3007278886 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7febdc0a29d0 0x7febdc0a4e90 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x561e6a2b6b10 tx=0x7febe8009110 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:25.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.198+0000 7fec0225a740 1 -- 172.21.15.39:0/2411093926 shutdown_connections 2024-03-18T16:12:25.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.198+0000 7fec0225a740 1 --2- 172.21.15.39:0/2411093926 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561e6a2bb420 0x561e6a2bd8b0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:25.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.198+0000 7fec0225a740 1 --2- 172.21.15.39:0/2411093926 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561e6a2b62b0 0x561e6a2badb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:25.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.198+0000 7fec0225a740 1 --2- 172.21.15.39:0/2411093926 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561e6a2b58c0 0x561e6a2b5ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:25.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.198+0000 7fec0225a740 1 -- 172.21.15.39:0/2411093926 >> 172.21.15.39:0/2411093926 conn(0x561e6a2b0e30 msgr2=0x561e6a2b1230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:25.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.203+0000 7fec0225a740 1 -- 172.21.15.39:0/2411093926 shutdown_connections 2024-03-18T16:12:25.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.204+0000 7fec0225a740 1 -- 172.21.15.39:0/2411093926 wait complete. 2024-03-18T16:12:25.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.204+0000 7fec0225a740 1 -- start start 2024-03-18T16:12:25.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.205+0000 7fec0225a740 1 Processor -- start 2024-03-18T16:12:25.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.208+0000 7fec0225a740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561e6a2b58c0 0x561e6a2bf960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:25.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.209+0000 7febfcc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561e6a2b58c0 0x561e6a2bf960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:25.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.209+0000 7febfcc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561e6a2b58c0 0x561e6a2bf960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:42062/0 (socket says 172.21.15.39:42062) 2024-03-18T16:12:25.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.210+0000 7febfcc00640 1 -- 172.21.15.39:0/995654540 learned_addr learned my addr 172.21.15.39:0/995654540 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:12:25.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.210+0000 7fec0225a740 1 --2- 172.21.15.39:0/995654540 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561e6a2b62b0 0x561e6a2bfea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:25.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.210+0000 7fec0225a740 1 --2- 172.21.15.39:0/995654540 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561e6a2bb420 0x561e6a2c6e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:25.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.215+0000 7fec0225a740 1 -- 172.21.15.39:0/995654540 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x561e6a1d6110 con 0x561e6a2b58c0 2024-03-18T16:12:25.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.215+0000 7fec0225a740 1 -- 172.21.15.39:0/995654540 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x561e6a2164f0 con 0x561e6a2b62b0 2024-03-18T16:12:25.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.215+0000 7febfd600640 1 --2- 172.21.15.39:0/995654540 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561e6a2bb420 0x561e6a2c6e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:25.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.215+0000 7fec0225a740 1 -- 172.21.15.39:0/995654540 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x561e6a2be190 con 0x561e6a2bb420 2024-03-18T16:12:25.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.215+0000 7fec0225a740 10 client.0 _ll_register_callbacks cb 0x561e6a1c3b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:12:25.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.217+0000 7febfd600640 1 -- 172.21.15.39:0/995654540 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561e6a2b62b0 msgr2=0x561e6a2bfea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:12:25.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.211+0000 7febf7e00640 1 --2- 172.21.15.39:0/995654540 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561e6a2b62b0 0x561e6a2bfea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:25.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.224+0000 7febfd600640 1 --2- 172.21.15.39:0/995654540 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561e6a2b62b0 0x561e6a2bfea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:25.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.224+0000 7febfd600640 1 -- 172.21.15.39:0/995654540 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561e6a2b58c0 msgr2=0x561e6a2bf960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:12:25.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.224+0000 7febfd600640 1 --2- 172.21.15.39:0/995654540 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561e6a2b58c0 0x561e6a2bf960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:25.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.224+0000 7febfd600640 1 -- 172.21.15.39:0/995654540 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561e69f2e3f0 con 0x561e6a2bb420 2024-03-18T16:12:25.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.224+0000 7febfcc00640 1 --2- 172.21.15.39:0/995654540 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561e6a2b58c0 0x561e6a2bf960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:12:25.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.224+0000 7febf7e00640 1 --2- 172.21.15.39:0/995654540 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561e6a2b62b0 0x561e6a2bfea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:12:25.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.225+0000 7febfd600640 1 --2- 172.21.15.39:0/995654540 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561e6a2bb420 0x561e6a2c6e70 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7febf00c5d70 tx=0x7febf00bd500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=795b8ec5ee885bf6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:25.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.225+0000 7febe2000640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:12:25.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.238+0000 7febe2000640 1 -- 172.21.15.39:0/995654540 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febf00bb4b0 con 0x561e6a2bb420 2024-03-18T16:12:25.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.238+0000 7febe2000640 1 -- 172.21.15.39:0/995654540 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7febf00bd670 con 0x561e6a2bb420 2024-03-18T16:12:25.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.239+0000 7fec0225a740 1 -- 172.21.15.39:0/995654540 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7febf00c1560 con 0x561e6a2bb420 2024-03-18T16:12:25.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.239+0000 7febe2000640 1 -- 172.21.15.39:0/995654540 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7febf00047c0 con 0x561e6a2bb420 2024-03-18T16:12:25.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.240+0000 7fec0225a740 10 client.26101 fetch_fsmap learned FSMap version 32 2024-03-18T16:12:25.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.240+0000 7fec0225a740 1 -- 172.21.15.39:0/995654540 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x561e69ff87d0 con 0x561e6a2bb420 2024-03-18T16:12:25.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.241+0000 7febe2000640 1 -- 172.21.15.39:0/995654540 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7febf00c1330 con 0x561e6a2bb420 2024-03-18T16:12:25.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.246+0000 7febe2000640 1 -- 172.21.15.39:0/995654540 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7febf003c030 con 0x561e6a2bb420 2024-03-18T16:12:25.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.246+0000 7fec0225a740 10 client.26101 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T16:12:25.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.246+0000 7fec0225a740 10 client.26101 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T16:12:25.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.246+0000 7fec0225a740 20 client.26101 populate_metadata read hostname 'smithi039' 2024-03-18T16:12:25.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.247+0000 7fec0225a740 1 --2- 172.21.15.39:0/995654540 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x561e6a2cb500 0x561e6a2cd940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:25.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.247+0000 7fec0225a740 4 client.26101 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:12:25.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.247+0000 7fec0225a740 1 -- 172.21.15.39:0/995654540 --> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7febf00c1560 con 0x561e6a2cb500 2024-03-18T16:12:25.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.249+0000 7febfcc00640 1 --2- 172.21.15.39:0/995654540 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x561e6a2cb500 0x561e6a2cd940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:25.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.253+0000 7febfcc00640 1 --2- 172.21.15.39:0/995654540 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x561e6a2cb500 0x561e6a2cd940 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=61fb25f5d7074636 server_cookie=836c19fcce8e0ad6 in_seq=0 out_seq=0 2024-03-18T16:12:25.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.257+0000 7febe2000640 10 client.26101 ms_handle_connect on v2:172.21.15.196:6838/2001427268 2024-03-18T16:12:25.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.257+0000 7febe2000640 1 -- 172.21.15.39:0/995654540 <== mds.1 v2:172.21.15.196:6838/2001427268 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7febf00c1560 con 0x561e6a2cb500 2024-03-18T16:12:25.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.257+0000 7febe2000640 10 client.26101 handle_command_reply: tid=0 2024-03-18T16:12:25.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.257+0000 7fec0225a740 1 client.26101 shutdown 2024-03-18T16:12:25.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.271+0000 7fec0225a740 1 -- 172.21.15.39:0/995654540 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561e6a2bb420 msgr2=0x561e6a2c6e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:25.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.271+0000 7fec0225a740 1 --2- 172.21.15.39:0/995654540 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561e6a2bb420 0x561e6a2c6e70 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7febf00c5d70 tx=0x7febf00bd500 comp rx=0 tx=0).stop 2024-03-18T16:12:25.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.275+0000 7fec0225a740 1 -- 172.21.15.39:0/995654540 shutdown_connections 2024-03-18T16:12:25.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.275+0000 7fec0225a740 1 --2- 172.21.15.39:0/995654540 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x561e6a2cb500 0x561e6a2cd940 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:25.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.275+0000 7fec0225a740 1 --2- 172.21.15.39:0/995654540 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561e6a2b62b0 0x561e6a2bfea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:25.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.275+0000 7fec0225a740 1 --2- 172.21.15.39:0/995654540 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561e6a2bb420 0x561e6a2c6e70 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:25.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.275+0000 7fec0225a740 1 --2- 172.21.15.39:0/995654540 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561e6a2b58c0 0x561e6a2bf960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:25.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.275+0000 7fec0225a740 1 -- 172.21.15.39:0/995654540 >> 172.21.15.39:0/995654540 conn(0x561e6a2b0e30 msgr2=0x561e6a2b01f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:25.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.275+0000 7febe2000640 0 client.26101 ms_handle_reset on v2:172.21.15.196:6838/2001427268 2024-03-18T16:12:25.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.275+0000 7fec0225a740 1 -- 172.21.15.39:0/995654540 shutdown_connections 2024-03-18T16:12:25.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.275+0000 7fec0225a740 1 -- 172.21.15.39:0/995654540 wait complete. 2024-03-18T16:12:25.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.276+0000 7fec0225a740 20 client.26101 trim_cache size 0 max 16384 2024-03-18T16:12:25.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.313+0000 7fec0225a740 1 Processor -- start 2024-03-18T16:12:25.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.313+0000 7fec0225a740 1 -- start start 2024-03-18T16:12:25.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.314+0000 7fec0225a740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561e6a2b58c0 0x561e6a2b0070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:25.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.315+0000 7fec0225a740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561e6a2b62b0 0x561e6a2c6e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:25.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.315+0000 7fec0225a740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561e6a2bb420 0x561e6a2d2240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:25.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.315+0000 7febfcc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561e6a2b58c0 0x561e6a2b0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:25.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.315+0000 7febfcc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561e6a2b58c0 0x561e6a2b0070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:42074/0 (socket says 172.21.15.39:42074) 2024-03-18T16:12:25.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.315+0000 7febfcc00640 1 -- 172.21.15.39:0/4108507934 learned_addr learned my addr 172.21.15.39:0/4108507934 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:12:25.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.315+0000 7fec0225a740 1 -- 172.21.15.39:0/4108507934 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7febf81b0a30 con 0x561e6a2b58c0 2024-03-18T16:12:25.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.316+0000 7fec0225a740 1 -- 172.21.15.39:0/4108507934 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7febf81b2630 con 0x561e6a2bb420 2024-03-18T16:12:25.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.316+0000 7fec0225a740 1 -- 172.21.15.39:0/4108507934 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7febf81b1100 con 0x561e6a2b62b0 2024-03-18T16:12:25.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.316+0000 7febf7e00640 1 --2- 172.21.15.39:0/4108507934 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561e6a2b62b0 0x561e6a2c6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:25.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.316+0000 7febfd600640 1 --2- 172.21.15.39:0/4108507934 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561e6a2bb420 0x561e6a2d2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:25.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.317+0000 7febf7e00640 1 -- 172.21.15.39:0/4108507934 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561e6a2bb420 msgr2=0x561e6a2d2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:12:25.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.317+0000 7febf7e00640 1 --2- 172.21.15.39:0/4108507934 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561e6a2bb420 0x561e6a2d2240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:25.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.317+0000 7febf7e00640 1 -- 172.21.15.39:0/4108507934 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561e6a2b58c0 msgr2=0x561e6a2b0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:12:25.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.317+0000 7febf7e00640 1 --2- 172.21.15.39:0/4108507934 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561e6a2b58c0 0x561e6a2b0070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:25.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.317+0000 7febf7e00640 1 -- 172.21.15.39:0/4108507934 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561e6a2cdfc0 con 0x561e6a2b62b0 2024-03-18T16:12:25.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.320+0000 7febf7e00640 1 --2- 172.21.15.39:0/4108507934 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561e6a2b62b0 0x561e6a2c6e30 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x561e6a2c0dd0 tx=0x7febe400df40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e41f29816342be6b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:25.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.320+0000 7febed200640 1 -- 172.21.15.39:0/4108507934 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febe4005020 con 0x561e6a2b62b0 2024-03-18T16:12:25.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.320+0000 7febed200640 1 -- 172.21.15.39:0/4108507934 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7febe40047a0 con 0x561e6a2b62b0 2024-03-18T16:12:25.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.320+0000 7febed200640 1 -- 172.21.15.39:0/4108507934 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febe400f7d0 con 0x561e6a2b62b0 2024-03-18T16:12:25.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.321+0000 7fec0225a740 1 -- 172.21.15.39:0/4108507934 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561e6a2b62b0 msgr2=0x561e6a2c6e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:25.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.321+0000 7fec0225a740 1 --2- 172.21.15.39:0/4108507934 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561e6a2b62b0 0x561e6a2c6e30 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x561e6a2c0dd0 tx=0x7febe400df40 comp rx=0 tx=0).stop 2024-03-18T16:12:25.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.323+0000 7fec0225a740 1 -- 172.21.15.39:0/4108507934 shutdown_connections 2024-03-18T16:12:25.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.323+0000 7fec0225a740 1 --2- 172.21.15.39:0/4108507934 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561e6a2bb420 0x561e6a2d2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:25.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.323+0000 7fec0225a740 1 --2- 172.21.15.39:0/4108507934 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561e6a2b62b0 0x561e6a2c6e30 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:25.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.323+0000 7fec0225a740 1 --2- 172.21.15.39:0/4108507934 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561e6a2b58c0 0x561e6a2b0070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:25.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.323+0000 7fec0225a740 1 -- 172.21.15.39:0/4108507934 >> 172.21.15.39:0/4108507934 conn(0x561e6a2b0e30 msgr2=0x561e6a2af960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:25.326 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.324+0000 7fec0225a740 1 -- 172.21.15.39:0/4108507934 shutdown_connections 2024-03-18T16:12:25.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.325+0000 7fec0225a740 1 -- 172.21.15.39:0/4108507934 wait complete. 2024-03-18T16:12:25.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.326+0000 7fec0225a740 1 -- start start 2024-03-18T16:12:25.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.326+0000 7fec0225a740 1 Processor -- start 2024-03-18T16:12:25.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.328+0000 7fec0225a740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561e6a2b58c0 0x561e6a2c67f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:25.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.328+0000 7fec0225a740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561e6a2b62b0 0x561e6a2c6d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:25.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.328+0000 7fec0225a740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561e6a2bb420 0x561e6a2c4e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:25.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.328+0000 7febf7e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561e6a2b62b0 0x561e6a2c6d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:25.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.328+0000 7febfd600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561e6a2bb420 0x561e6a2c4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:25.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.328+0000 7febfd600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561e6a2bb420 0x561e6a2c4e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:54630/0 (socket says 172.21.15.39:54630) 2024-03-18T16:12:25.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.328+0000 7febfd600640 1 -- 172.21.15.39:0/4030484234 learned_addr learned my addr 172.21.15.39:0/4030484234 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:12:25.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.328+0000 7fec0225a740 1 -- 172.21.15.39:0/4030484234 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7febf81b1f00 con 0x561e6a2b58c0 2024-03-18T16:12:25.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.329+0000 7fec0225a740 1 -- 172.21.15.39:0/4030484234 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7febf81b17d0 con 0x561e6a2b62b0 2024-03-18T16:12:25.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.329+0000 7fec0225a740 1 -- 172.21.15.39:0/4030484234 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7febf81aea10 con 0x561e6a2bb420 2024-03-18T16:12:25.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.329+0000 7fec0225a740 10 client.0 _ll_register_callbacks cb 0x7febf00bd1e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:12:25.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.332+0000 7febfcc00640 1 --2- 172.21.15.39:0/4030484234 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561e6a2b58c0 0x561e6a2c67f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:25.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.333+0000 7febf7e00640 1 -- 172.21.15.39:0/4030484234 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561e6a2bb420 msgr2=0x561e6a2c4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:12:25.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.333+0000 7febf7e00640 1 --2- 172.21.15.39:0/4030484234 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561e6a2bb420 0x561e6a2c4e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:25.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.333+0000 7febf7e00640 1 -- 172.21.15.39:0/4030484234 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561e6a2b58c0 msgr2=0x561e6a2c67f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:12:25.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.333+0000 7febf7e00640 1 --2- 172.21.15.39:0/4030484234 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561e6a2b58c0 0x561e6a2c67f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:25.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.333+0000 7febf7e00640 1 -- 172.21.15.39:0/4030484234 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561e6a2b5500 con 0x561e6a2b62b0 2024-03-18T16:12:25.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.333+0000 7febfd600640 1 --2- 172.21.15.39:0/4030484234 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561e6a2bb420 0x561e6a2c4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:12:25.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.333+0000 7febfcc00640 1 --2- 172.21.15.39:0/4030484234 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561e6a2b58c0 0x561e6a2c67f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:12:25.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.333+0000 7febf7e00640 1 --2- 172.21.15.39:0/4030484234 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561e6a2b62b0 0x561e6a2c6d30 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x561e6a2b1f10 tx=0x7febe4004660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=22cf0287fe22fa51 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:25.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.333+0000 7febe2000640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:12:25.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.334+0000 7febe2000640 1 -- 172.21.15.39:0/4030484234 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febe4026070 con 0x561e6a2b62b0 2024-03-18T16:12:25.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.334+0000 7febe2000640 1 -- 172.21.15.39:0/4030484234 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7febe4005020 con 0x561e6a2b62b0 2024-03-18T16:12:25.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.334+0000 7fec0225a740 1 -- 172.21.15.39:0/4030484234 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7febe40047a0 con 0x561e6a2b62b0 2024-03-18T16:12:25.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.335+0000 7febe2000640 1 -- 172.21.15.39:0/4030484234 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7febe4005220 con 0x561e6a2b62b0 2024-03-18T16:12:25.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.336+0000 7fec0225a740 10 client.26087 fetch_fsmap learned FSMap version 32 2024-03-18T16:12:25.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.336+0000 7fec0225a740 1 -- 172.21.15.39:0/4030484234 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x561e6a3320a0 con 0x561e6a2b62b0 2024-03-18T16:12:25.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.336+0000 7febe2000640 1 -- 172.21.15.39:0/4030484234 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7febe40047a0 con 0x561e6a2b62b0 2024-03-18T16:12:25.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.336+0000 7febe2000640 1 -- 172.21.15.39:0/4030484234 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7febe4018070 con 0x561e6a2b62b0 2024-03-18T16:12:25.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.339+0000 7fec0225a740 10 client.26087 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T16:12:25.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.339+0000 7fec0225a740 10 client.26087 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T16:12:25.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.339+0000 7fec0225a740 20 client.26087 populate_metadata read hostname 'smithi039' 2024-03-18T16:12:25.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.339+0000 7fec0225a740 1 --2- 172.21.15.39:0/4030484234 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x561e6a3328f0 0x561e6a334d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:25.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.339+0000 7fec0225a740 4 client.26087 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:12:25.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.339+0000 7fec0225a740 1 -- 172.21.15.39:0/4030484234 --> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561e6a227990 con 0x561e6a3328f0 2024-03-18T16:12:25.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.341+0000 7febfcc00640 1 --2- 172.21.15.39:0/4030484234 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x561e6a3328f0 0x561e6a334d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:25.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.343+0000 7febfcc00640 1 --2- 172.21.15.39:0/4030484234 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x561e6a3328f0 0x561e6a334d10 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=83cf09488b9b1b34 server_cookie=167b385f8d7782e in_seq=0 out_seq=0 2024-03-18T16:12:25.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.343+0000 7febe2000640 10 client.26087 ms_handle_connect on v2:172.21.15.196:6838/2001427268 2024-03-18T16:12:25.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.354+0000 7febe2000640 1 -- 172.21.15.39:0/4030484234 <== mds.1 v2:172.21.15.196:6838/2001427268 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561e6a227990 con 0x561e6a3328f0 2024-03-18T16:12:25.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.354+0000 7febe2000640 10 client.26087 handle_command_reply: tid=0 2024-03-18T16:12:25.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.355+0000 7fec0225a740 1 client.26087 shutdown 2024-03-18T16:12:25.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.355+0000 7fec0225a740 1 -- 172.21.15.39:0/4030484234 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561e6a2b62b0 msgr2=0x561e6a2c6d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:25.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.355+0000 7fec0225a740 1 --2- 172.21.15.39:0/4030484234 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561e6a2b62b0 0x561e6a2c6d30 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x561e6a2b1f10 tx=0x7febe4004660 comp rx=0 tx=0).stop 2024-03-18T16:12:25.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.355+0000 7fec0225a740 1 -- 172.21.15.39:0/4030484234 shutdown_connections 2024-03-18T16:12:25.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.355+0000 7fec0225a740 1 --2- 172.21.15.39:0/4030484234 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x561e6a3328f0 0x561e6a334d10 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:25.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.355+0000 7fec0225a740 1 --2- 172.21.15.39:0/4030484234 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561e6a2b62b0 0x561e6a2c6d30 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:25.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.355+0000 7fec0225a740 1 --2- 172.21.15.39:0/4030484234 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561e6a2bb420 0x561e6a2c4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:25.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.355+0000 7fec0225a740 1 --2- 172.21.15.39:0/4030484234 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561e6a2b58c0 0x561e6a2c67f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:25.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.355+0000 7fec0225a740 1 -- 172.21.15.39:0/4030484234 >> 172.21.15.39:0/4030484234 conn(0x561e6a2b0e30 msgr2=0x561e6a2b01f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:25.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.355+0000 7febe2000640 0 client.26087 ms_handle_reset on v2:172.21.15.196:6838/2001427268 2024-03-18T16:12:25.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.356+0000 7fec0225a740 1 -- 172.21.15.39:0/4030484234 shutdown_connections 2024-03-18T16:12:25.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.356+0000 7fec0225a740 1 -- 172.21.15.39:0/4030484234 wait complete. 2024-03-18T16:12:25.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.356+0000 7fec0225a740 20 client.26087 trim_cache size 0 max 16384 2024-03-18T16:12:25.358 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:12:25.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.357+0000 7febff600640 1 -- 172.21.15.39:0/3007278886 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7febdc0a29d0 msgr2=0x7febdc0a4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:25.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.357+0000 7febff600640 1 --2- 172.21.15.39:0/3007278886 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7febdc0a29d0 0x7febdc0a4e90 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x561e6a2b6b10 tx=0x7febe8009110 comp rx=0 tx=0).stop 2024-03-18T16:12:25.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.357+0000 7febff600640 1 -- 172.21.15.39:0/3007278886 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7febf810db00 msgr2=0x7febf81a59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:25.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.357+0000 7febff600640 1 --2- 172.21.15.39:0/3007278886 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7febf810db00 0x7febf81a59a0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7febf0037040 tx=0x7febf0035d80 comp rx=0 tx=0).stop 2024-03-18T16:12:25.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.360+0000 7febff600640 1 -- 172.21.15.39:0/3007278886 shutdown_connections 2024-03-18T16:12:25.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.360+0000 7febff600640 1 --2- 172.21.15.39:0/3007278886 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7febf810b150 0x7febf8115350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:25.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.360+0000 7febff600640 1 --2- 172.21.15.39:0/3007278886 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7febdc0a29d0 0x7febdc0a4e90 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:25.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.360+0000 7febff600640 1 --2- 172.21.15.39:0/3007278886 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7febf810db00 0x7febf81a59a0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:25.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.360+0000 7febff600640 1 --2- 172.21.15.39:0/3007278886 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7febf81087e0 0x7febf8114e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:25.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.360+0000 7febff600640 1 -- 172.21.15.39:0/3007278886 >> 172.21.15.39:0/3007278886 conn(0x7febf806e860 msgr2=0x7febf810d310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:25.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.361+0000 7febff600640 1 -- 172.21.15.39:0/3007278886 shutdown_connections 2024-03-18T16:12:25.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.361+0000 7febff600640 1 -- 172.21.15.39:0/3007278886 wait complete. 2024-03-18T16:12:25.410 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-18T16:12:25.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:12:25 smithi196 ceph-mon[36307]: pgmap v492: 129 pgs: 129 active+clean; 12 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 327 MiB/s wr, 111 op/s 2024-03-18T16:12:25.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:12:25 smithi194 ceph-mon[36200]: pgmap v492: 129 pgs: 129 active+clean; 12 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 327 MiB/s wr, 111 op/s 2024-03-18T16:12:25.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:12:25 smithi039 ceph-mon[31721]: pgmap v492: 129 pgs: 129 active+clean; 12 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 327 MiB/s wr, 111 op/s 2024-03-18T16:12:25.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.889+0000 7f3143e00640 1 -- 172.21.15.39:0/3824818444 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f313c099190 msgr2=0x7f313c09b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:25.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.889+0000 7f3143e00640 1 --2- 172.21.15.39:0/3824818444 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f313c099190 0x7f313c09b580 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f3134009900 tx=0x7f313402bc40 comp rx=0 tx=0).stop 2024-03-18T16:12:25.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.890+0000 7f3143e00640 1 -- 172.21.15.39:0/3824818444 shutdown_connections 2024-03-18T16:12:25.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.890+0000 7f3143e00640 1 --2- 172.21.15.39:0/3824818444 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f313c09bac0 0x7f313c09deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:25.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.890+0000 7f3143e00640 1 --2- 172.21.15.39:0/3824818444 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f313c099190 0x7f313c09b580 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:25.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.890+0000 7f3143e00640 1 --2- 172.21.15.39:0/3824818444 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f313c096860 0x7f313c098c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:25.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.890+0000 7f3143e00640 1 -- 172.21.15.39:0/3824818444 >> 172.21.15.39:0/3824818444 conn(0x7f313c0901d0 msgr2=0x7f313c092630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:25.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.891+0000 7f3143e00640 1 -- 172.21.15.39:0/3824818444 shutdown_connections 2024-03-18T16:12:25.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.891+0000 7f3143e00640 1 -- 172.21.15.39:0/3824818444 wait complete. 2024-03-18T16:12:25.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.898+0000 7f3143e00640 1 Processor -- start 2024-03-18T16:12:25.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.898+0000 7f3143e00640 1 -- start start 2024-03-18T16:12:25.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.900+0000 7f3143e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f313c096860 0x7f313c12e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:25.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.901+0000 7f3143e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f313c099190 0x7f313c12e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:25.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.901+0000 7f3143e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f313c09bac0 0x7f313c1358a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:25.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.901+0000 7f3143e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f313c0a3cc0 con 0x7f313c096860 2024-03-18T16:12:25.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.901+0000 7f3143e00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f313c0a3b40 con 0x7f313c09bac0 2024-03-18T16:12:25.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.901+0000 7f3143e00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f313c0a3e40 con 0x7f313c099190 2024-03-18T16:12:25.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.901+0000 7f3143400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f313c09bac0 0x7f313c1358a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:25.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.901+0000 7f3143400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f313c09bac0 0x7f313c1358a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:55282/0 (socket says 172.21.15.39:55282) 2024-03-18T16:12:25.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.901+0000 7f3143400640 1 -- 172.21.15.39:0/1489718783 learned_addr learned my addr 172.21.15.39:0/1489718783 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:12:25.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.902+0000 7f3143400640 1 -- 172.21.15.39:0/1489718783 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f313c099190 msgr2=0x7f313c12e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:25.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.902+0000 7f3143400640 1 --2- 172.21.15.39:0/1489718783 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f313c099190 0x7f313c12e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:25.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.902+0000 7f3143400640 1 -- 172.21.15.39:0/1489718783 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f313c096860 msgr2=0x7f313c12e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:25.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.902+0000 7f3143400640 1 --2- 172.21.15.39:0/1489718783 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f313c096860 0x7f313c12e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:25.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.902+0000 7f3143400640 1 -- 172.21.15.39:0/1489718783 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f313c135f10 con 0x7f313c09bac0 2024-03-18T16:12:25.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.903+0000 7f3143400640 1 --2- 172.21.15.39:0/1489718783 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f313c09bac0 0x7f313c1358a0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f313000e970 tx=0x7f313000ed70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:25.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.904+0000 7f313b400640 1 -- 172.21.15.39:0/1489718783 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f313000ccc0 con 0x7f313c09bac0 2024-03-18T16:12:25.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.904+0000 7f313b400640 1 -- 172.21.15.39:0/1489718783 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f313000ce60 con 0x7f313c09bac0 2024-03-18T16:12:25.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.904+0000 7f313b400640 1 -- 172.21.15.39:0/1489718783 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3130010490 con 0x7f313c09bac0 2024-03-18T16:12:25.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.904+0000 7f3143e00640 1 -- 172.21.15.39:0/1489718783 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f313c136200 con 0x7f313c09bac0 2024-03-18T16:12:25.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.904+0000 7f3143e00640 1 -- 172.21.15.39:0/1489718783 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f313c136700 con 0x7f313c09bac0 2024-03-18T16:12:25.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.915+0000 7f314c390740 1 Processor -- start 2024-03-18T16:12:25.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.918+0000 7f313b400640 1 -- 172.21.15.39:0/1489718783 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f3130005000 con 0x7f313c09bac0 2024-03-18T16:12:25.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.919+0000 7f314c390740 1 -- start start 2024-03-18T16:12:25.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.919+0000 7f314c390740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8e73aca60 0x55a8e73ace80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:25.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.919+0000 7f314c390740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8e73ad450 0x55a8e73b1f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:25.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.919+0000 7f314c390740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a8e73b25c0 0x55a8e73b4a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:25.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.919+0000 7f314c390740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55a8e73743b0 con 0x55a8e73aca60 2024-03-18T16:12:25.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.919+0000 7f314c390740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55a8e7311450 con 0x55a8e73ad450 2024-03-18T16:12:25.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.919+0000 7f314c390740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55a8e72b9cb0 con 0x55a8e73b25c0 2024-03-18T16:12:25.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.921+0000 7f3142000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a8e73b25c0 0x55a8e73b4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:25.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.921+0000 7f3142000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8e73aca60 0x55a8e73ace80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:25.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.921+0000 7f3142000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8e73aca60 0x55a8e73ace80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:42118/0 (socket says 172.21.15.39:42118) 2024-03-18T16:12:25.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.921+0000 7f3142000640 1 -- 172.21.15.39:0/2373007994 learned_addr learned my addr 172.21.15.39:0/2373007994 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:12:25.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.921+0000 7f3142a00640 1 --2- 172.21.15.39:0/2373007994 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8e73ad450 0x55a8e73b1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:25.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.922+0000 7f3142000640 1 -- 172.21.15.39:0/2373007994 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a8e73b25c0 msgr2=0x55a8e73b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:12:25.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.922+0000 7f3142000640 1 --2- 172.21.15.39:0/2373007994 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a8e73b25c0 0x55a8e73b4a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:25.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.922+0000 7f3142000640 1 -- 172.21.15.39:0/2373007994 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8e73ad450 msgr2=0x55a8e73b1f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:12:25.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.922+0000 7f3142000640 1 --2- 172.21.15.39:0/2373007994 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8e73ad450 0x55a8e73b1f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:25.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.922+0000 7f3142000640 1 -- 172.21.15.39:0/2373007994 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a8e72db1f0 con 0x55a8e73aca60 2024-03-18T16:12:25.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.924+0000 7f313b400640 1 --2- 172.21.15.39:0/1489718783 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f31280a2980 0x7f31280a4e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:25.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.925+0000 7f313b400640 1 -- 172.21.15.39:0/1489718783 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f31300cf320 con 0x7f313c09bac0 2024-03-18T16:12:25.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.925+0000 7f3142000640 1 --2- 172.21.15.39:0/2373007994 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8e73aca60 0x55a8e73ace80 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f3134005ab0 tx=0x7f3134000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22fdfc38957c7906 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:25.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.926+0000 7f3142a00640 1 --2- 172.21.15.39:0/1489718783 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f31280a2980 0x7f31280a4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:25.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.930+0000 7f3142a00640 1 --2- 172.21.15.39:0/1489718783 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f31280a2980 0x7f31280a4e40 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f3144050e40 tx=0x7f3144067ae0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:25.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.930+0000 7f312f400640 1 -- 172.21.15.39:0/2373007994 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3134043030 con 0x55a8e73aca60 2024-03-18T16:12:25.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.930+0000 7f312f400640 1 -- 172.21.15.39:0/2373007994 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3134004020 con 0x55a8e73aca60 2024-03-18T16:12:25.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.930+0000 7f312f400640 1 -- 172.21.15.39:0/2373007994 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31340046c0 con 0x55a8e73aca60 2024-03-18T16:12:25.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.931+0000 7f314c390740 1 -- 172.21.15.39:0/2373007994 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8e73aca60 msgr2=0x55a8e73ace80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:25.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.931+0000 7f314c390740 1 --2- 172.21.15.39:0/2373007994 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8e73aca60 0x55a8e73ace80 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f3134005ab0 tx=0x7f3134000bc0 comp rx=0 tx=0).stop 2024-03-18T16:12:25.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.933+0000 7f314c390740 1 -- 172.21.15.39:0/2373007994 shutdown_connections 2024-03-18T16:12:25.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.933+0000 7f314c390740 1 --2- 172.21.15.39:0/2373007994 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8e73ad450 0x55a8e73b1f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:25.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.933+0000 7f314c390740 1 --2- 172.21.15.39:0/2373007994 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a8e73b25c0 0x55a8e73b4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:25.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.933+0000 7f314c390740 1 --2- 172.21.15.39:0/2373007994 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8e73aca60 0x55a8e73ace80 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:25.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.933+0000 7f314c390740 1 -- 172.21.15.39:0/2373007994 >> 172.21.15.39:0/2373007994 conn(0x55a8e73a7fd0 msgr2=0x55a8e73a83d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:25.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.934+0000 7f314c390740 1 -- 172.21.15.39:0/2373007994 shutdown_connections 2024-03-18T16:12:25.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.934+0000 7f314c390740 1 -- 172.21.15.39:0/2373007994 wait complete. 2024-03-18T16:12:25.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.934+0000 7f314c390740 1 -- start start 2024-03-18T16:12:25.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.935+0000 7f314c390740 1 Processor -- start 2024-03-18T16:12:25.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.935+0000 7f314c390740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a8e73aca60 0x55a8e73bf550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:25.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.935+0000 7f314c390740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8e73ad450 0x55a8e73bfa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:25.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.935+0000 7f314c390740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8e73b25c0 0x55a8e73c6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:25.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.935+0000 7f314c390740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55a8e72cd110 con 0x55a8e73b25c0 2024-03-18T16:12:25.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.935+0000 7f314c390740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55a8e730d4f0 con 0x55a8e73ad450 2024-03-18T16:12:25.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.935+0000 7f314c390740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55a8e73bdd20 con 0x55a8e73aca60 2024-03-18T16:12:25.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.935+0000 7f314c390740 10 client.0 _ll_register_callbacks cb 0x55a8e72bab70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:12:25.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.936+0000 7f3142000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8e73ad450 0x55a8e73bfa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:25.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.936+0000 7f3142000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8e73ad450 0x55a8e73bfa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:55302/0 (socket says 172.21.15.39:55302) 2024-03-18T16:12:25.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.936+0000 7f3142000640 1 -- 172.21.15.39:0/1311991208 learned_addr learned my addr 172.21.15.39:0/1311991208 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:12:25.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.936+0000 7f3142a00640 1 --2- 172.21.15.39:0/1311991208 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a8e73aca60 0x55a8e73bf550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:25.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.937+0000 7f3142000640 1 -- 172.21.15.39:0/1311991208 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a8e73aca60 msgr2=0x55a8e73bf550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:12:25.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.937+0000 7f3142000640 1 --2- 172.21.15.39:0/1311991208 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a8e73aca60 0x55a8e73bf550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:25.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.937+0000 7f3142000640 1 -- 172.21.15.39:0/1311991208 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8e73b25c0 msgr2=0x55a8e73c6ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:12:25.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.937+0000 7f3142000640 1 --2- 172.21.15.39:0/1311991208 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8e73b25c0 0x55a8e73c6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:25.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.937+0000 7f3142000640 1 -- 172.21.15.39:0/1311991208 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a8e70253f0 con 0x55a8e73ad450 2024-03-18T16:12:25.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.937+0000 7f3142a00640 1 --2- 172.21.15.39:0/1311991208 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a8e73aca60 0x55a8e73bf550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:12:25.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.938+0000 7f3123e00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:12:25.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.938+0000 7f3142000640 1 --2- 172.21.15.39:0/1311991208 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8e73ad450 0x55a8e73bfa90 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f3134006010 tx=0x7f31340058c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ee0f091d02063fb5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:25.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.940+0000 7f3123e00640 1 -- 172.21.15.39:0/1311991208 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3134043040 con 0x55a8e73ad450 2024-03-18T16:12:25.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.940+0000 7f3123e00640 1 -- 172.21.15.39:0/1311991208 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f313402c990 con 0x55a8e73ad450 2024-03-18T16:12:25.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.940+0000 7f314c390740 1 -- 172.21.15.39:0/1311991208 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3134004020 con 0x55a8e73ad450 2024-03-18T16:12:25.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.940+0000 7f3123e00640 1 -- 172.21.15.39:0/1311991208 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f3134055050 con 0x55a8e73ad450 2024-03-18T16:12:25.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.940+0000 7f314c390740 10 client.26099 fetch_fsmap learned FSMap version 32 2024-03-18T16:12:25.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.940+0000 7f314c390740 1 -- 172.21.15.39:0/1311991208 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55a8e70ef7d0 con 0x55a8e73ad450 2024-03-18T16:12:25.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.941+0000 7f3123e00640 1 -- 172.21.15.39:0/1311991208 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3134004020 con 0x55a8e73ad450 2024-03-18T16:12:25.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.941+0000 7f3123e00640 1 -- 172.21.15.39:0/1311991208 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7f3134003790 con 0x55a8e73ad450 2024-03-18T16:12:25.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.942+0000 7f314c390740 10 client.26099 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T16:12:25.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.942+0000 7f314c390740 10 client.26099 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:12:25.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.942+0000 7f314c390740 20 client.26099 populate_metadata read hostname 'smithi039' 2024-03-18T16:12:25.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.942+0000 7f314c390740 1 --2- 172.21.15.39:0/1311991208 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55a8e73cafc0 0x55a8e73cd420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:25.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.942+0000 7f314c390740 4 client.26099 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:12:25.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.942+0000 7f314c390740 1 -- 172.21.15.39:0/1311991208 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a8e731e990 con 0x55a8e73cafc0 2024-03-18T16:12:25.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.942+0000 7f3142a00640 1 --2- 172.21.15.39:0/1311991208 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55a8e73cafc0 0x55a8e73cd420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:25.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.943+0000 7f3123e00640 10 client.26099 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:12:25.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.943+0000 7f3142a00640 1 --2- 172.21.15.39:0/1311991208 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55a8e73cafc0 0x55a8e73cd420 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=57b8cbb66baaa6d5 server_cookie=14ebeb5945a12b7c in_seq=0 out_seq=0 2024-03-18T16:12:25.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.946+0000 7f3123e00640 1 -- 172.21.15.39:0/1311991208 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a8e731e990 con 0x55a8e73cafc0 2024-03-18T16:12:25.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.946+0000 7f3123e00640 10 client.26099 handle_command_reply: tid=0 2024-03-18T16:12:25.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.946+0000 7f314c390740 1 client.26099 shutdown 2024-03-18T16:12:25.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.947+0000 7f314c390740 1 -- 172.21.15.39:0/1311991208 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8e73ad450 msgr2=0x55a8e73bfa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:25.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.947+0000 7f314c390740 1 --2- 172.21.15.39:0/1311991208 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8e73ad450 0x55a8e73bfa90 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f3134006010 tx=0x7f31340058c0 comp rx=0 tx=0).stop 2024-03-18T16:12:25.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.947+0000 7f314c390740 1 -- 172.21.15.39:0/1311991208 shutdown_connections 2024-03-18T16:12:25.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.947+0000 7f314c390740 1 --2- 172.21.15.39:0/1311991208 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55a8e73cafc0 0x55a8e73cd420 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:25.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.947+0000 7f314c390740 1 --2- 172.21.15.39:0/1311991208 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8e73ad450 0x55a8e73bfa90 secure :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f3134006010 tx=0x7f31340058c0 comp rx=0 tx=0).stop 2024-03-18T16:12:25.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.947+0000 7f314c390740 1 --2- 172.21.15.39:0/1311991208 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8e73b25c0 0x55a8e73c6ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:25.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.947+0000 7f314c390740 1 --2- 172.21.15.39:0/1311991208 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a8e73aca60 0x55a8e73bf550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:25.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.947+0000 7f314c390740 1 -- 172.21.15.39:0/1311991208 >> 172.21.15.39:0/1311991208 conn(0x55a8e73a7fd0 msgr2=0x55a8e73a71f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:25.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.947+0000 7f3123e00640 0 client.26099 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:12:25.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.948+0000 7f314c390740 1 -- 172.21.15.39:0/1311991208 shutdown_connections 2024-03-18T16:12:25.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.948+0000 7f314c390740 1 -- 172.21.15.39:0/1311991208 wait complete. 2024-03-18T16:12:25.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.948+0000 7f314c390740 20 client.26099 trim_cache size 0 max 16384 2024-03-18T16:12:25.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.989+0000 7f314c390740 1 Processor -- start 2024-03-18T16:12:25.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.989+0000 7f314c390740 1 -- start start 2024-03-18T16:12:25.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.990+0000 7f314c390740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a8e73aca60 0x55a8e73be0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:25.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.990+0000 7f314c390740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8e73b25c0 0x55a8e73c6a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:25.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.990+0000 7f314c390740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8e73aefc0 0x55a8e73be600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:25.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.990+0000 7f314c390740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f313c13c720 con 0x55a8e73b25c0 2024-03-18T16:12:25.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.990+0000 7f314c390740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f313c13e320 con 0x55a8e73aefc0 2024-03-18T16:12:25.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.990+0000 7f314c390740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f313c13cdf0 con 0x55a8e73aca60 2024-03-18T16:12:25.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.990+0000 7f3143400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8e73aefc0 0x55a8e73be600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:25.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.990+0000 7f3143400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8e73aefc0 0x55a8e73be600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:55304/0 (socket says 172.21.15.39:55304) 2024-03-18T16:12:25.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.990+0000 7f3143400640 1 -- 172.21.15.39:0/3993066208 learned_addr learned my addr 172.21.15.39:0/3993066208 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:12:25.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.991+0000 7f3142a00640 1 --2- 172.21.15.39:0/3993066208 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a8e73aca60 0x55a8e73be0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:25.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.991+0000 7f3142000640 1 --2- 172.21.15.39:0/3993066208 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8e73b25c0 0x55a8e73c6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:25.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.991+0000 7f3143400640 1 -- 172.21.15.39:0/3993066208 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a8e73aca60 msgr2=0x55a8e73be0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:12:25.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.991+0000 7f3143400640 1 --2- 172.21.15.39:0/3993066208 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a8e73aca60 0x55a8e73be0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:25.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.991+0000 7f3143400640 1 -- 172.21.15.39:0/3993066208 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8e73b25c0 msgr2=0x55a8e73c6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:12:25.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.991+0000 7f3143400640 1 --2- 172.21.15.39:0/3993066208 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8e73b25c0 0x55a8e73c6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:25.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.991+0000 7f3143400640 1 -- 172.21.15.39:0/3993066208 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a8e73a3850 con 0x55a8e73aefc0 2024-03-18T16:12:25.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.994+0000 7f3143400640 1 --2- 172.21.15.39:0/3993066208 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8e73aefc0 0x55a8e73be600 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f3130004fd0 tx=0x7f313009b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f0bc9c471a814a98 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:25.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.994+0000 7f312ea00640 1 -- 172.21.15.39:0/3993066208 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31300cf160 con 0x55a8e73aefc0 2024-03-18T16:12:25.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.994+0000 7f312ea00640 1 -- 172.21.15.39:0/3993066208 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f313009b930 con 0x55a8e73aefc0 2024-03-18T16:12:25.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.994+0000 7f312ea00640 1 -- 172.21.15.39:0/3993066208 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31300052f0 con 0x55a8e73aefc0 2024-03-18T16:12:25.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.998+0000 7f314c390740 1 -- 172.21.15.39:0/3993066208 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8e73aefc0 msgr2=0x55a8e73be600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:26.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.998+0000 7f314c390740 1 --2- 172.21.15.39:0/3993066208 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8e73aefc0 0x55a8e73be600 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f3130004fd0 tx=0x7f313009b810 comp rx=0 tx=0).stop 2024-03-18T16:12:26.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.999+0000 7f314c390740 1 -- 172.21.15.39:0/3993066208 shutdown_connections 2024-03-18T16:12:26.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.999+0000 7f314c390740 1 --2- 172.21.15.39:0/3993066208 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8e73aefc0 0x55a8e73be600 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:26.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.999+0000 7f314c390740 1 --2- 172.21.15.39:0/3993066208 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8e73b25c0 0x55a8e73c6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:26.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.999+0000 7f314c390740 1 --2- 172.21.15.39:0/3993066208 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a8e73aca60 0x55a8e73be0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:26.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:25.999+0000 7f314c390740 1 -- 172.21.15.39:0/3993066208 >> 172.21.15.39:0/3993066208 conn(0x55a8e73a7fd0 msgr2=0x55a8e73abac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:26.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:26.000+0000 7f314c390740 1 -- 172.21.15.39:0/3993066208 shutdown_connections 2024-03-18T16:12:26.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:26.000+0000 7f314c390740 1 -- 172.21.15.39:0/3993066208 wait complete. 2024-03-18T16:12:26.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:26.000+0000 7f314c390740 1 -- start start 2024-03-18T16:12:26.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:26.001+0000 7f314c390740 1 Processor -- start 2024-03-18T16:12:26.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:26.001+0000 7f314c390740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8e73aca60 0x55a8e73c3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:26.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:26.001+0000 7f314c390740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a8e73aefc0 0x55a8e73c4470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:26.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:26.001+0000 7f314c390740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8e73b25c0 0x55a8e73c5480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:26.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:26.001+0000 7f314c390740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f313c13dbf0 con 0x55a8e73b25c0 2024-03-18T16:12:26.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:26.001+0000 7f314c390740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f313c13d4c0 con 0x55a8e73aca60 2024-03-18T16:12:26.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:26.001+0000 7f314c390740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f313c13a760 con 0x55a8e73aefc0 2024-03-18T16:12:26.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:26.001+0000 7f314c390740 10 client.0 _ll_register_callbacks cb 0x7f31440679b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:12:26.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:26.002+0000 7f3142a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8e73aca60 0x55a8e73c3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:26.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:26.002+0000 7f3142a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8e73aca60 0x55a8e73c3f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:55308/0 (socket says 172.21.15.39:55308) 2024-03-18T16:12:26.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:26.002+0000 7f3142a00640 1 -- 172.21.15.39:0/2808607399 learned_addr learned my addr 172.21.15.39:0/2808607399 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:12:26.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:26.002+0000 7f3142a00640 1 -- 172.21.15.39:0/2808607399 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a8e73aefc0 msgr2=0x55a8e73c4470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:12:26.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:26.002+0000 7f3142a00640 1 --2- 172.21.15.39:0/2808607399 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a8e73aefc0 0x55a8e73c4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:26.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:26.002+0000 7f3142a00640 1 -- 172.21.15.39:0/2808607399 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8e73b25c0 msgr2=0x55a8e73c5480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:12:26.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:26.002+0000 7f3142a00640 1 --2- 172.21.15.39:0/2808607399 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8e73b25c0 0x55a8e73c5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:26.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:26.003+0000 7f3142a00640 1 -- 172.21.15.39:0/2808607399 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a8e73c5a70 con 0x55a8e73aca60 2024-03-18T16:12:26.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:26.006+0000 7f3142a00640 1 --2- 172.21.15.39:0/2808607399 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8e73aca60 0x55a8e73c3f30 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f31440675f0 tx=0x7f314404fc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=82adbbdb2d5bafe4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:26.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:26.006+0000 7f3123e00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:12:26.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:26.006+0000 7f3123e00640 1 -- 172.21.15.39:0/2808607399 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31440a1060 con 0x55a8e73aca60 2024-03-18T16:12:26.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:26.006+0000 7f3123e00640 1 -- 172.21.15.39:0/2808607399 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f3144081660 con 0x55a8e73aca60 2024-03-18T16:12:26.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:26.009+0000 7f314c390740 1 -- 172.21.15.39:0/2808607399 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f313009b930 con 0x55a8e73aca60 2024-03-18T16:12:26.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:26.009+0000 7f3123e00640 1 -- 172.21.15.39:0/2808607399 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f3144088040 con 0x55a8e73aca60 2024-03-18T16:12:26.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:26.009+0000 7f314c390740 10 client.26105 fetch_fsmap learned FSMap version 32 2024-03-18T16:12:26.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:26.009+0000 7f314c390740 1 -- 172.21.15.39:0/2808607399 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55a8e7431370 con 0x55a8e73aca60 2024-03-18T16:12:26.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:26.009+0000 7f3123e00640 1 -- 172.21.15.39:0/2808607399 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f313009b930 con 0x55a8e73aca60 2024-03-18T16:12:26.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:26.010+0000 7f3123e00640 1 -- 172.21.15.39:0/2808607399 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 32) ==== 8378+0+0 (secure 0 0 0) 0x7f314409ca00 con 0x55a8e73aca60 2024-03-18T16:12:26.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:26.010+0000 7f314c390740 10 client.26105 fetch_fsmap finished waiting for FSMap version 32 2024-03-18T16:12:26.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:26.010+0000 7f314c390740 10 client.26105 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:12:26.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:26.010+0000 7f314c390740 20 client.26105 populate_metadata read hostname 'smithi039' 2024-03-18T16:12:26.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:26.010+0000 7f314c390740 1 --2- 172.21.15.39:0/2808607399 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55a8e7431be0 0x55a8e7434060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:26.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:26.010+0000 7f314c390740 4 client.26105 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:12:26.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:26.010+0000 7f314c390740 1 -- 172.21.15.39:0/2808607399 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a8e7318ce0 con 0x55a8e7431be0 2024-03-18T16:12:26.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:26.011+0000 7f3142000640 1 --2- 172.21.15.39:0/2808607399 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55a8e7431be0 0x55a8e7434060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:26.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:26.013+0000 7f3142000640 1 --2- 172.21.15.39:0/2808607399 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55a8e7431be0 0x55a8e7434060 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3a758d8c8d6a8f60 server_cookie=b3fed57f6eef0a1 in_seq=0 out_seq=0 2024-03-18T16:12:26.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:26.013+0000 7f3123e00640 10 client.26105 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:12:26.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:26.013+0000 7f3123e00640 1 -- 172.21.15.39:0/2808607399 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a8e7318ce0 con 0x55a8e7431be0 2024-03-18T16:12:26.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:26.013+0000 7f3123e00640 10 client.26105 handle_command_reply: tid=0 2024-03-18T16:12:26.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:26.014+0000 7f314c390740 1 client.26105 shutdown 2024-03-18T16:12:26.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:26.014+0000 7f314c390740 1 -- 172.21.15.39:0/2808607399 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8e73aca60 msgr2=0x55a8e73c3f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:26.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:26.014+0000 7f314c390740 1 --2- 172.21.15.39:0/2808607399 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8e73aca60 0x55a8e73c3f30 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f31440675f0 tx=0x7f314404fc40 comp rx=0 tx=0).stop 2024-03-18T16:12:26.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:26.015+0000 7f314c390740 1 -- 172.21.15.39:0/2808607399 shutdown_connections 2024-03-18T16:12:26.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:26.015+0000 7f314c390740 1 --2- 172.21.15.39:0/2808607399 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55a8e7431be0 0x55a8e7434060 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:26.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:26.016+0000 7f314c390740 1 --2- 172.21.15.39:0/2808607399 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8e73b25c0 0x55a8e73c5480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:26.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:26.016+0000 7f314c390740 1 --2- 172.21.15.39:0/2808607399 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a8e73aefc0 0x55a8e73c4470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:26.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:26.016+0000 7f3123e00640 0 client.26105 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:12:26.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:26.016+0000 7f314c390740 1 --2- 172.21.15.39:0/2808607399 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8e73aca60 0x55a8e73c3f30 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:26.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:26.016+0000 7f314c390740 1 -- 172.21.15.39:0/2808607399 >> 172.21.15.39:0/2808607399 conn(0x55a8e73a7fd0 msgr2=0x55a8e73a6960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:26.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:26.016+0000 7f314c390740 1 -- 172.21.15.39:0/2808607399 shutdown_connections 2024-03-18T16:12:26.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:26.017+0000 7f314c390740 1 -- 172.21.15.39:0/2808607399 wait complete. 2024-03-18T16:12:26.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:26.017+0000 7f314c390740 20 client.26105 trim_cache size 0 max 16384 2024-03-18T16:12:26.019 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:12:26.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:26.019+0000 7f3143e00640 1 -- 172.21.15.39:0/1489718783 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f31280a2980 msgr2=0x7f31280a4e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:26.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:26.019+0000 7f3143e00640 1 --2- 172.21.15.39:0/1489718783 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f31280a2980 0x7f31280a4e40 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f3144050e40 tx=0x7f3144067ae0 comp rx=0 tx=0).stop 2024-03-18T16:12:26.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:26.019+0000 7f3143e00640 1 -- 172.21.15.39:0/1489718783 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f313c09bac0 msgr2=0x7f313c1358a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:26.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:26.019+0000 7f3143e00640 1 --2- 172.21.15.39:0/1489718783 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f313c09bac0 0x7f313c1358a0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f313000e970 tx=0x7f313000ed70 comp rx=0 tx=0).stop 2024-03-18T16:12:26.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:26.019+0000 7f3143e00640 1 -- 172.21.15.39:0/1489718783 shutdown_connections 2024-03-18T16:12:26.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:26.019+0000 7f3143e00640 1 --2- 172.21.15.39:0/1489718783 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f313c09bac0 0x7f313c1358a0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:26.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:26.019+0000 7f3143e00640 1 --2- 172.21.15.39:0/1489718783 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f31280a2980 0x7f31280a4e40 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:26.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:26.019+0000 7f3143e00640 1 --2- 172.21.15.39:0/1489718783 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f313c099190 0x7f313c12e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:26.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:26.019+0000 7f3143e00640 1 --2- 172.21.15.39:0/1489718783 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f313c096860 0x7f313c12e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:26.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:26.019+0000 7f3143e00640 1 -- 172.21.15.39:0/1489718783 >> 172.21.15.39:0/1489718783 conn(0x7f313c0901d0 msgr2=0x7f313c097630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:26.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:26.019+0000 7f3143e00640 1 -- 172.21.15.39:0/1489718783 shutdown_connections 2024-03-18T16:12:26.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:26.019+0000 7f3143e00640 1 -- 172.21.15.39:0/1489718783 wait complete. 2024-03-18T16:12:27.058 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-18T16:12:27.058 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T16:12:27.058 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-18T16:12:27.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.355+0000 7fb29f200640 1 -- 172.21.15.39:0/2417943945 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb298077d80 msgr2=0x7fb298078620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:27.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.355+0000 7fb29f200640 1 --2- 172.21.15.39:0/2417943945 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb298077d80 0x7fb298078620 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fb290009ca0 tx=0x7fb29002c100 comp rx=0 tx=0).stop 2024-03-18T16:12:27.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.355+0000 7fb29f200640 1 -- 172.21.15.39:0/2417943945 shutdown_connections 2024-03-18T16:12:27.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.356+0000 7fb29f200640 1 --2- 172.21.15.39:0/2417943945 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb298077d80 0x7fb298078620 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:27.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.356+0000 7fb29f200640 1 --2- 172.21.15.39:0/2417943945 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb2980773e0 0x7fb298077840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:27.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.356+0000 7fb29f200640 1 --2- 172.21.15.39:0/2417943945 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb2980761a0 0x7fb2980765c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:27.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.356+0000 7fb29f200640 1 -- 172.21.15.39:0/2417943945 >> 172.21.15.39:0/2417943945 conn(0x7fb2980fef90 msgr2=0x7fb2981013d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:27.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.356+0000 7fb29f200640 1 -- 172.21.15.39:0/2417943945 shutdown_connections 2024-03-18T16:12:27.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.356+0000 7fb29f200640 1 -- 172.21.15.39:0/2417943945 wait complete. 2024-03-18T16:12:27.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.358+0000 7fb29f200640 1 Processor -- start 2024-03-18T16:12:27.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.359+0000 7fb29f200640 1 -- start start 2024-03-18T16:12:27.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.359+0000 7fb29f200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb2980761a0 0x7fb29819e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:27.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.359+0000 7fb29f200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb2980773e0 0x7fb29819e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:27.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.360+0000 7fb29f200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb298077d80 0x7fb2981a55b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:27.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.360+0000 7fb29f200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fb298113ab0 con 0x7fb2980761a0 2024-03-18T16:12:27.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.360+0000 7fb29f200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fb298113930 con 0x7fb298077d80 2024-03-18T16:12:27.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.360+0000 7fb29f200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fb298113c30 con 0x7fb2980773e0 2024-03-18T16:12:27.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.360+0000 7fb297400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb2980773e0 0x7fb29819e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:27.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.360+0000 7fb297e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb2980761a0 0x7fb29819e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:27.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.360+0000 7fb297e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb2980761a0 0x7fb29819e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:42152/0 (socket says 172.21.15.39:42152) 2024-03-18T16:12:27.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.360+0000 7fb297e00640 1 -- 172.21.15.39:0/70394936 learned_addr learned my addr 172.21.15.39:0/70394936 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:12:27.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.360+0000 7fb297400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb2980773e0 0x7fb29819e540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:54702/0 (socket says 172.21.15.39:54702) 2024-03-18T16:12:27.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.361+0000 7fb29d200640 1 --2- 172.21.15.39:0/70394936 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb298077d80 0x7fb2981a55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:27.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.361+0000 7fb29d200640 1 -- 172.21.15.39:0/70394936 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb2980773e0 msgr2=0x7fb29819e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:27.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.361+0000 7fb29d200640 1 --2- 172.21.15.39:0/70394936 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb2980773e0 0x7fb29819e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:27.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.361+0000 7fb29d200640 1 -- 172.21.15.39:0/70394936 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb2980761a0 msgr2=0x7fb29819e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:27.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.361+0000 7fb29d200640 1 --2- 172.21.15.39:0/70394936 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb2980761a0 0x7fb29819e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:27.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.361+0000 7fb29d200640 1 -- 172.21.15.39:0/70394936 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2981a5c20 con 0x7fb298077d80 2024-03-18T16:12:27.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.362+0000 7fb29d200640 1 --2- 172.21.15.39:0/70394936 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb298077d80 0x7fb2981a55b0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fb290037040 tx=0x7fb290035d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:27.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.365+0000 7fb294c00640 1 -- 172.21.15.39:0/70394936 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb290039040 con 0x7fb298077d80 2024-03-18T16:12:27.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.365+0000 7fb294c00640 1 -- 172.21.15.39:0/70394936 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb290035e00 con 0x7fb298077d80 2024-03-18T16:12:27.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.365+0000 7fb29f200640 1 -- 172.21.15.39:0/70394936 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2981a5eb0 con 0x7fb298077d80 2024-03-18T16:12:27.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.365+0000 7fb294c00640 1 -- 172.21.15.39:0/70394936 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2900043c0 con 0x7fb298077d80 2024-03-18T16:12:27.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.366+0000 7fb29f200640 1 -- 172.21.15.39:0/70394936 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2981a6370 con 0x7fb298077d80 2024-03-18T16:12:27.379 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.375+0000 7fb294c00640 1 -- 172.21.15.39:0/70394936 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fb290003b00 con 0x7fb298077d80 2024-03-18T16:12:27.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.377+0000 7fb2a2028740 1 Processor -- start 2024-03-18T16:12:27.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.377+0000 7fb2a2028740 1 -- start start 2024-03-18T16:12:27.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.377+0000 7fb2a2028740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fbc297b7e0 0x55fbc297bc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:27.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.378+0000 7fb2a2028740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fbc297c1d0 0x55fbc2991ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:27.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.378+0000 7fb2a2028740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fbc2992350 0x55fbc29947e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:27.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.378+0000 7fb2a2028740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55fbc28def80 con 0x55fbc2992350 2024-03-18T16:12:27.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.378+0000 7fb2a2028740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55fbc2887ad0 con 0x55fbc297b7e0 2024-03-18T16:12:27.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.378+0000 7fb2a2028740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55fbc28db020 con 0x55fbc297c1d0 2024-03-18T16:12:27.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.378+0000 7fb297e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fbc297c1d0 0x55fbc2991ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:27.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.378+0000 7fb294c00640 1 --2- 172.21.15.39:0/70394936 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb27c0a28d0 0x7fb27c0a4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:27.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.378+0000 7fb297e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fbc297c1d0 0x55fbc2991ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:54710/0 (socket says 172.21.15.39:54710) 2024-03-18T16:12:27.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.378+0000 7fb297e00640 1 -- 172.21.15.39:0/888931441 learned_addr learned my addr 172.21.15.39:0/888931441 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:12:27.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.378+0000 7fb297400640 1 --2- 172.21.15.39:0/888931441 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fbc297b7e0 0x55fbc297bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:27.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.378+0000 7fb294c00640 1 -- 172.21.15.39:0/70394936 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fb2900f0c60 con 0x7fb298077d80 2024-03-18T16:12:27.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.379+0000 7fb29d200640 1 --2- 172.21.15.39:0/70394936 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb27c0a28d0 0x7fb27c0a4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:27.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.380+0000 7fb297400640 1 --2- 172.21.15.39:0/888931441 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fbc2992350 0x55fbc29947e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:27.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.380+0000 7fb297e00640 1 -- 172.21.15.39:0/888931441 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fbc297b7e0 msgr2=0x55fbc297bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:12:27.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.380+0000 7fb297e00640 1 --2- 172.21.15.39:0/888931441 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fbc297b7e0 0x55fbc297bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:27.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.380+0000 7fb297e00640 1 -- 172.21.15.39:0/888931441 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fbc2992350 msgr2=0x55fbc29947e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:12:27.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.380+0000 7fb297e00640 1 --2- 172.21.15.39:0/888931441 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fbc2992350 0x55fbc29947e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:27.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.381+0000 7fb297e00640 1 -- 172.21.15.39:0/888931441 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fbc289f6f0 con 0x55fbc297c1d0 2024-03-18T16:12:27.382 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.381+0000 7fb297400640 1 --2- 172.21.15.39:0/888931441 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fbc2992350 0x55fbc29947e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:12:27.383 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.381+0000 7fb297e00640 1 --2- 172.21.15.39:0/888931441 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fbc297c1d0 0x55fbc2991ce0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fb28400bd80 tx=0x7fb2840107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ea813bb96aeef2cd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:27.383 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.382+0000 7fb29d200640 1 --2- 172.21.15.39:0/70394936 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb27c0a28d0 0x7fb27c0a4d90 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fb2900b96d0 tx=0x7fb2900095c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:27.383 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.382+0000 7fb28d600640 1 -- 172.21.15.39:0/888931441 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb284011640 con 0x55fbc297c1d0 2024-03-18T16:12:27.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.382+0000 7fb28d600640 1 -- 172.21.15.39:0/888931441 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb284011cc0 con 0x55fbc297c1d0 2024-03-18T16:12:27.384 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.383+0000 7fb28d600640 1 -- 172.21.15.39:0/888931441 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb28401bcf0 con 0x55fbc297c1d0 2024-03-18T16:12:27.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.386+0000 7fb2a2028740 1 -- 172.21.15.39:0/888931441 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fbc297c1d0 msgr2=0x55fbc2991ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:27.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.386+0000 7fb2a2028740 1 --2- 172.21.15.39:0/888931441 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fbc297c1d0 0x55fbc2991ce0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fb28400bd80 tx=0x7fb2840107a0 comp rx=0 tx=0).stop 2024-03-18T16:12:27.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.386+0000 7fb2a2028740 1 -- 172.21.15.39:0/888931441 shutdown_connections 2024-03-18T16:12:27.388 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.386+0000 7fb2a2028740 1 --2- 172.21.15.39:0/888931441 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fbc2992350 0x55fbc29947e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:27.388 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.386+0000 7fb2a2028740 1 --2- 172.21.15.39:0/888931441 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fbc297c1d0 0x55fbc2991ce0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:27.388 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.386+0000 7fb2a2028740 1 --2- 172.21.15.39:0/888931441 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fbc297b7e0 0x55fbc297bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:27.388 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.386+0000 7fb2a2028740 1 -- 172.21.15.39:0/888931441 >> 172.21.15.39:0/888931441 conn(0x55fbc2943930 msgr2=0x55fbc2977150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:27.388 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.387+0000 7fb2a2028740 1 -- 172.21.15.39:0/888931441 shutdown_connections 2024-03-18T16:12:27.388 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.387+0000 7fb2a2028740 1 -- 172.21.15.39:0/888931441 wait complete. 2024-03-18T16:12:27.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.387+0000 7fb2a2028740 1 -- start start 2024-03-18T16:12:27.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.388+0000 7fb2a2028740 1 Processor -- start 2024-03-18T16:12:27.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.389+0000 7fb2a2028740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fbc297b7e0 0x55fbc2996b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:27.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.389+0000 7fb2a2028740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fbc297c1d0 0x55fbc29970c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:27.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.389+0000 7fb2a2028740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fbc2992350 0x55fbc299e100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:27.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.390+0000 7fb2a2028740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55fbc2995300 con 0x55fbc2992350 2024-03-18T16:12:27.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.390+0000 7fb2a2028740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55fbc289af30 con 0x55fbc297c1d0 2024-03-18T16:12:27.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.390+0000 7fb2a2028740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55fbc2995480 con 0x55fbc297b7e0 2024-03-18T16:12:27.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.390+0000 7fb2a2028740 10 client.0 _ll_register_callbacks cb 0x55fbc2888bc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:12:27.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.390+0000 7fb297400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fbc297b7e0 0x55fbc2996b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:27.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.390+0000 7fb297400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fbc297b7e0 0x55fbc2996b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:54718/0 (socket says 172.21.15.39:54718) 2024-03-18T16:12:27.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.390+0000 7fb297400640 1 -- 172.21.15.39:0/873064880 learned_addr learned my addr 172.21.15.39:0/873064880 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:12:27.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.390+0000 7fb297e00640 1 --2- 172.21.15.39:0/873064880 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fbc297c1d0 0x55fbc29970c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:27.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.391+0000 7fb297400640 1 -- 172.21.15.39:0/873064880 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fbc297c1d0 msgr2=0x55fbc29970c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:12:27.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.391+0000 7fb297400640 1 --2- 172.21.15.39:0/873064880 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fbc297c1d0 0x55fbc29970c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:27.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.391+0000 7fb297400640 1 -- 172.21.15.39:0/873064880 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fbc2992350 msgr2=0x55fbc299e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:12:27.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.391+0000 7fb297400640 1 --2- 172.21.15.39:0/873064880 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fbc2992350 0x55fbc299e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:27.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.392+0000 7fb297400640 1 -- 172.21.15.39:0/873064880 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fbc26bb7e0 con 0x55fbc297b7e0 2024-03-18T16:12:27.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.392+0000 7fb282000640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:12:27.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.393+0000 7fb297400640 1 --2- 172.21.15.39:0/873064880 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fbc297b7e0 0x55fbc2996b80 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fb28800afe0 tx=0x7fb288011ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=70240f665e4fa267 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:27.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.394+0000 7fb282000640 1 -- 172.21.15.39:0/873064880 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb28800fe20 con 0x55fbc297b7e0 2024-03-18T16:12:27.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.394+0000 7fb282000640 1 -- 172.21.15.39:0/873064880 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fb288005020 con 0x55fbc297b7e0 2024-03-18T16:12:27.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.394+0000 7fb2a2028740 1 -- 172.21.15.39:0/873064880 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb284011cc0 con 0x55fbc297b7e0 2024-03-18T16:12:27.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.394+0000 7fb282000640 1 -- 172.21.15.39:0/873064880 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fb288005220 con 0x55fbc297b7e0 2024-03-18T16:12:27.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.395+0000 7fb2a2028740 10 client.26125 fetch_fsmap learned FSMap version 33 2024-03-18T16:12:27.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.395+0000 7fb2a2028740 1 -- 172.21.15.39:0/873064880 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55fbc271bf70 con 0x55fbc297b7e0 2024-03-18T16:12:27.398 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.396+0000 7fb282000640 1 -- 172.21.15.39:0/873064880 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb284011cc0 con 0x55fbc297b7e0 2024-03-18T16:12:27.398 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.396+0000 7fb282000640 1 -- 172.21.15.39:0/873064880 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 33) ==== 8386+0+0 (secure 0 0 0) 0x7fb28801a6a0 con 0x55fbc297b7e0 2024-03-18T16:12:27.398 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.396+0000 7fb2a2028740 10 client.26125 fetch_fsmap finished waiting for FSMap version 33 2024-03-18T16:12:27.398 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.396+0000 7fb2a2028740 10 client.26125 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:12:27.398 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.396+0000 7fb2a2028740 20 client.26125 populate_metadata read hostname 'smithi039' 2024-03-18T16:12:27.398 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.396+0000 7fb2a2028740 1 --2- 172.21.15.39:0/873064880 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55fbc29a2a00 0x55fbc29a4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:27.398 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.396+0000 7fb2a2028740 4 client.26125 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:12:27.398 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.396+0000 7fb2a2028740 1 -- 172.21.15.39:0/873064880 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fbc28ec4c0 con 0x55fbc29a2a00 2024-03-18T16:12:27.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.398+0000 7fb297e00640 1 --2- 172.21.15.39:0/873064880 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55fbc29a2a00 0x55fbc29a4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:27.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.401+0000 7fb297e00640 1 --2- 172.21.15.39:0/873064880 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55fbc29a2a00 0x55fbc29a4ea0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bfaa2725f93a94c6 server_cookie=1f6b3c9c7503fc9e in_seq=0 out_seq=0 2024-03-18T16:12:27.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.401+0000 7fb282000640 10 client.26125 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:12:27.403 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.402+0000 7fb282000640 1 -- 172.21.15.39:0/873064880 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55fbc28ec4c0 con 0x55fbc29a2a00 2024-03-18T16:12:27.403 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.402+0000 7fb282000640 10 client.26125 handle_command_reply: tid=0 2024-03-18T16:12:27.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.402+0000 7fb2a2028740 1 client.26125 shutdown 2024-03-18T16:12:27.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.403+0000 7fb2a2028740 1 -- 172.21.15.39:0/873064880 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fbc297b7e0 msgr2=0x55fbc2996b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:27.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.403+0000 7fb2a2028740 1 --2- 172.21.15.39:0/873064880 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fbc297b7e0 0x55fbc2996b80 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fb28800afe0 tx=0x7fb288011ec0 comp rx=0 tx=0).stop 2024-03-18T16:12:27.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.403+0000 7fb2a2028740 1 -- 172.21.15.39:0/873064880 shutdown_connections 2024-03-18T16:12:27.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.403+0000 7fb2a2028740 1 --2- 172.21.15.39:0/873064880 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55fbc29a2a00 0x55fbc29a4ea0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:27.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.403+0000 7fb2a2028740 1 --2- 172.21.15.39:0/873064880 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fbc297c1d0 0x55fbc29970c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:27.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.403+0000 7fb2a2028740 1 --2- 172.21.15.39:0/873064880 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fbc2992350 0x55fbc299e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:27.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.403+0000 7fb2a2028740 1 --2- 172.21.15.39:0/873064880 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fbc297b7e0 0x55fbc2996b80 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:27.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.403+0000 7fb2a2028740 1 -- 172.21.15.39:0/873064880 >> 172.21.15.39:0/873064880 conn(0x55fbc2943930 msgr2=0x55fbc2976350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:27.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.404+0000 7fb282000640 0 client.26125 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:12:27.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.404+0000 7fb2a2028740 1 -- 172.21.15.39:0/873064880 shutdown_connections 2024-03-18T16:12:27.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.404+0000 7fb2a2028740 1 -- 172.21.15.39:0/873064880 wait complete. 2024-03-18T16:12:27.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.404+0000 7fb2a2028740 20 client.26125 trim_cache size 0 max 16384 2024-03-18T16:12:27.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.447+0000 7fb2a2028740 1 Processor -- start 2024-03-18T16:12:27.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.448+0000 7fb2a2028740 1 -- start start 2024-03-18T16:12:27.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.448+0000 7fb2a2028740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fbc297b7e0 0x55fbc2997020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:27.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.448+0000 7fb2a2028740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fbc297c1d0 0x55fbc299e090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:27.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.448+0000 7fb2a2028740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fbc2992350 0x55fbc297a390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:27.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.448+0000 7fb2a2028740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fb2981a8130 con 0x55fbc2992350 2024-03-18T16:12:27.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.448+0000 7fb2a2028740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fb2981a9d90 con 0x55fbc297c1d0 2024-03-18T16:12:27.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.448+0000 7fb2a2028740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fb2981a8800 con 0x55fbc297b7e0 2024-03-18T16:12:27.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.448+0000 7fb297400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fbc297b7e0 0x55fbc2997020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:27.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.449+0000 7fb297400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fbc297b7e0 0x55fbc2997020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:54732/0 (socket says 172.21.15.39:54732) 2024-03-18T16:12:27.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.449+0000 7fb297e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fbc297c1d0 0x55fbc299e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:27.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.449+0000 7fb297e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fbc297c1d0 0x55fbc299e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:55370/0 (socket says 172.21.15.39:55370) 2024-03-18T16:12:27.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.449+0000 7fb297e00640 1 -- 172.21.15.39:0/3469234474 learned_addr learned my addr 172.21.15.39:0/3469234474 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:12:27.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.449+0000 7fb297e00640 1 -- 172.21.15.39:0/3469234474 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fbc297b7e0 msgr2=0x55fbc2997020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:12:27.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.450+0000 7fb297e00640 1 --2- 172.21.15.39:0/3469234474 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fbc297b7e0 0x55fbc2997020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:27.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.450+0000 7fb297e00640 1 -- 172.21.15.39:0/3469234474 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fbc2992350 msgr2=0x55fbc297a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:12:27.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.450+0000 7fb297e00640 1 --2- 172.21.15.39:0/3469234474 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fbc2992350 0x55fbc297a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:27.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.450+0000 7fb297e00640 1 -- 172.21.15.39:0/3469234474 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fbc25b3170 con 0x55fbc297c1d0 2024-03-18T16:12:27.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.450+0000 7fb297400640 1 --2- 172.21.15.39:0/3469234474 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fbc297b7e0 0x55fbc2997020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:12:27.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.450+0000 7fb297e00640 1 --2- 172.21.15.39:0/3469234474 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fbc297c1d0 0x55fbc299e090 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fb28400b770 tx=0x7fb284005880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f3bd8f40751e3cf server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:27.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.451+0000 7fb28cc00640 1 -- 172.21.15.39:0/3469234474 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb284005040 con 0x55fbc297c1d0 2024-03-18T16:12:27.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.451+0000 7fb28cc00640 1 -- 172.21.15.39:0/3469234474 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2840280f0 con 0x55fbc297c1d0 2024-03-18T16:12:27.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.451+0000 7fb28cc00640 1 -- 172.21.15.39:0/3469234474 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb284003850 con 0x55fbc297c1d0 2024-03-18T16:12:27.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.453+0000 7fb2a2028740 1 -- 172.21.15.39:0/3469234474 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fbc297c1d0 msgr2=0x55fbc299e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:27.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.453+0000 7fb2a2028740 1 --2- 172.21.15.39:0/3469234474 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fbc297c1d0 0x55fbc299e090 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fb28400b770 tx=0x7fb284005880 comp rx=0 tx=0).stop 2024-03-18T16:12:27.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.453+0000 7fb2a2028740 1 -- 172.21.15.39:0/3469234474 shutdown_connections 2024-03-18T16:12:27.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.453+0000 7fb2a2028740 1 --2- 172.21.15.39:0/3469234474 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fbc297c1d0 0x55fbc299e090 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:27.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.453+0000 7fb2a2028740 1 --2- 172.21.15.39:0/3469234474 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fbc2992350 0x55fbc297a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:27.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.453+0000 7fb2a2028740 1 --2- 172.21.15.39:0/3469234474 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fbc297b7e0 0x55fbc2997020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:27.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.453+0000 7fb2a2028740 1 -- 172.21.15.39:0/3469234474 >> 172.21.15.39:0/3469234474 conn(0x55fbc2943930 msgr2=0x55fbc2995f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:27.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.453+0000 7fb2a2028740 1 -- 172.21.15.39:0/3469234474 shutdown_connections 2024-03-18T16:12:27.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.453+0000 7fb2a2028740 1 -- 172.21.15.39:0/3469234474 wait complete. 2024-03-18T16:12:27.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.454+0000 7fb2a2028740 1 -- start start 2024-03-18T16:12:27.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.455+0000 7fb2a2028740 1 Processor -- start 2024-03-18T16:12:27.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.455+0000 7fb2a2028740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fbc297b7e0 0x55fbc299b330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:27.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.455+0000 7fb2a2028740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fbc297c1d0 0x55fbc299b870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:27.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.455+0000 7fb2a2028740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fbc2992350 0x55fbc299c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:27.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.455+0000 7fb2a2028740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fb2981a9660 con 0x55fbc2992350 2024-03-18T16:12:27.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.455+0000 7fb2a2028740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fb2981a8ed0 con 0x55fbc297b7e0 2024-03-18T16:12:27.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.455+0000 7fb2a2028740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fb29810b710 con 0x55fbc297c1d0 2024-03-18T16:12:27.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.455+0000 7fb2a2028740 10 client.0 _ll_register_callbacks cb 0x7fb28800a870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:12:27.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.456+0000 7fb297e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fbc297c1d0 0x55fbc299b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:27.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.456+0000 7fb297400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fbc297b7e0 0x55fbc299b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:27.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.456+0000 7fb297e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fbc297c1d0 0x55fbc299b870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:54736/0 (socket says 172.21.15.39:54736) 2024-03-18T16:12:27.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.456+0000 7fb297e00640 1 -- 172.21.15.39:0/970757198 learned_addr learned my addr 172.21.15.39:0/970757198 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:12:27.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.456+0000 7fb297e00640 1 -- 172.21.15.39:0/970757198 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fbc297b7e0 msgr2=0x55fbc299b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:12:27.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.456+0000 7fb297e00640 1 --2- 172.21.15.39:0/970757198 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fbc297b7e0 0x55fbc299b330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:27.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.456+0000 7fb297400640 1 --2- 172.21.15.39:0/970757198 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fbc2992350 0x55fbc299c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:27.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.456+0000 7fb297e00640 1 -- 172.21.15.39:0/970757198 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fbc2992350 msgr2=0x55fbc299c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:12:27.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.456+0000 7fb297e00640 1 --2- 172.21.15.39:0/970757198 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fbc2992350 0x55fbc299c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:27.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.456+0000 7fb297e00640 1 -- 172.21.15.39:0/970757198 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fbc299ce70 con 0x55fbc297c1d0 2024-03-18T16:12:27.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.457+0000 7fb297400640 1 --2- 172.21.15.39:0/970757198 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fbc2992350 0x55fbc299c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:12:27.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.457+0000 7fb297e00640 1 --2- 172.21.15.39:0/970757198 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fbc297c1d0 0x55fbc299b870 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fb284005c60 tx=0x7fb284028540 comp rx=0 tx=0).ready entity=mon.1 client_cookie=34929c4fb85ed85a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:27.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.458+0000 7fb282000640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:12:27.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.458+0000 7fb282000640 1 -- 172.21.15.39:0/970757198 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb284037060 con 0x55fbc297c1d0 2024-03-18T16:12:27.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.458+0000 7fb282000640 1 -- 172.21.15.39:0/970757198 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fb284005030 con 0x55fbc297c1d0 2024-03-18T16:12:27.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.458+0000 7fb2a2028740 1 -- 172.21.15.39:0/970757198 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb2840280f0 con 0x55fbc297c1d0 2024-03-18T16:12:27.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.458+0000 7fb282000640 1 -- 172.21.15.39:0/970757198 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fb284005270 con 0x55fbc297c1d0 2024-03-18T16:12:27.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.459+0000 7fb2a2028740 10 client.26128 fetch_fsmap learned FSMap version 33 2024-03-18T16:12:27.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.459+0000 7fb2a2028740 1 -- 172.21.15.39:0/970757198 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55fbc299e120 con 0x55fbc297c1d0 2024-03-18T16:12:27.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.460+0000 7fb282000640 1 -- 172.21.15.39:0/970757198 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2840280f0 con 0x55fbc297c1d0 2024-03-18T16:12:27.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.460+0000 7fb282000640 1 -- 172.21.15.39:0/970757198 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 33) ==== 8386+0+0 (secure 0 0 0) 0x7fb28401b5c0 con 0x55fbc297c1d0 2024-03-18T16:12:27.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.460+0000 7fb2a2028740 10 client.26128 fetch_fsmap finished waiting for FSMap version 33 2024-03-18T16:12:27.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.460+0000 7fb2a2028740 10 client.26128 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:12:27.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.460+0000 7fb2a2028740 20 client.26128 populate_metadata read hostname 'smithi039' 2024-03-18T16:12:27.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.460+0000 7fb2a2028740 1 --2- 172.21.15.39:0/970757198 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55fbc2a08f10 0x55fbc2a0b300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:27.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.460+0000 7fb2a2028740 4 client.26128 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-18T16:12:27.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.460+0000 7fb2a2028740 1 -- 172.21.15.39:0/970757198 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55fbc28e6810 con 0x55fbc2a08f10 2024-03-18T16:12:27.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.461+0000 7fb297400640 1 --2- 172.21.15.39:0/970757198 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55fbc2a08f10 0x55fbc2a0b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:27.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.462+0000 7fb297400640 1 --2- 172.21.15.39:0/970757198 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55fbc2a08f10 0x55fbc2a0b300 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e22b7edaa28034ca server_cookie=ba98604369512a9a in_seq=0 out_seq=0 2024-03-18T16:12:27.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.462+0000 7fb282000640 10 client.26128 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:12:27.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.465+0000 7fb282000640 1 -- 172.21.15.39:0/970757198 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55fbc28e6810 con 0x55fbc2a08f10 2024-03-18T16:12:27.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.465+0000 7fb282000640 10 client.26128 handle_command_reply: tid=0 2024-03-18T16:12:27.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.466+0000 7fb2a2028740 1 client.26128 shutdown 2024-03-18T16:12:27.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.466+0000 7fb2a2028740 1 -- 172.21.15.39:0/970757198 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fbc297c1d0 msgr2=0x55fbc299b870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:27.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.466+0000 7fb2a2028740 1 --2- 172.21.15.39:0/970757198 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fbc297c1d0 0x55fbc299b870 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fb284005c60 tx=0x7fb284028540 comp rx=0 tx=0).stop 2024-03-18T16:12:27.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.466+0000 7fb2a2028740 1 -- 172.21.15.39:0/970757198 shutdown_connections 2024-03-18T16:12:27.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.466+0000 7fb2a2028740 1 --2- 172.21.15.39:0/970757198 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55fbc2a08f10 0x55fbc2a0b300 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:27.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.466+0000 7fb2a2028740 1 --2- 172.21.15.39:0/970757198 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fbc2992350 0x55fbc299c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:27.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.466+0000 7fb2a2028740 1 --2- 172.21.15.39:0/970757198 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fbc297c1d0 0x55fbc299b870 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:27.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.466+0000 7fb2a2028740 1 --2- 172.21.15.39:0/970757198 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fbc297b7e0 0x55fbc299b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:27.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.467+0000 7fb282000640 0 client.26128 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:12:27.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.467+0000 7fb2a2028740 1 -- 172.21.15.39:0/970757198 >> 172.21.15.39:0/970757198 conn(0x55fbc2943930 msgr2=0x55fbc2975ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:27.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.467+0000 7fb2a2028740 1 -- 172.21.15.39:0/970757198 shutdown_connections 2024-03-18T16:12:27.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.467+0000 7fb2a2028740 1 -- 172.21.15.39:0/970757198 wait complete. 2024-03-18T16:12:27.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.469+0000 7fb2a2028740 20 client.26128 trim_cache size 0 max 16384 2024-03-18T16:12:27.471 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-03-18T16:12:27.471 INFO:teuthology.orchestra.run.smithi039.stdout: "return_code": 0, 2024-03-18T16:12:27.471 INFO:teuthology.orchestra.run.smithi039.stdout: "scrub_tag": "93d229b6-d830-40af-a394-0e1bd9a6e54f", 2024-03-18T16:12:27.471 INFO:teuthology.orchestra.run.smithi039.stdout: "mode": "asynchronous" 2024-03-18T16:12:27.471 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-03-18T16:12:27.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.471+0000 7fb29f200640 1 -- 172.21.15.39:0/70394936 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb27c0a28d0 msgr2=0x7fb27c0a4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:27.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.471+0000 7fb29f200640 1 --2- 172.21.15.39:0/70394936 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb27c0a28d0 0x7fb27c0a4d90 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fb2900b96d0 tx=0x7fb2900095c0 comp rx=0 tx=0).stop 2024-03-18T16:12:27.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.471+0000 7fb29f200640 1 -- 172.21.15.39:0/70394936 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb298077d80 msgr2=0x7fb2981a55b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:27.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.471+0000 7fb29f200640 1 --2- 172.21.15.39:0/70394936 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb298077d80 0x7fb2981a55b0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fb290037040 tx=0x7fb290035d10 comp rx=0 tx=0).stop 2024-03-18T16:12:27.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.471+0000 7fb29f200640 1 -- 172.21.15.39:0/70394936 shutdown_connections 2024-03-18T16:12:27.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.472+0000 7fb29f200640 1 --2- 172.21.15.39:0/70394936 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb298077d80 0x7fb2981a55b0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:27.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.472+0000 7fb29f200640 1 --2- 172.21.15.39:0/70394936 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb27c0a28d0 0x7fb27c0a4d90 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:27.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.472+0000 7fb29f200640 1 --2- 172.21.15.39:0/70394936 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb2980773e0 0x7fb29819e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:27.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.472+0000 7fb29f200640 1 --2- 172.21.15.39:0/70394936 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb2980761a0 0x7fb29819e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:27.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.472+0000 7fb29f200640 1 -- 172.21.15.39:0/70394936 >> 172.21.15.39:0/70394936 conn(0x7fb2980fef90 msgr2=0x7fb298100a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:27.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.473+0000 7fb29f200640 1 -- 172.21.15.39:0/70394936 shutdown_connections 2024-03-18T16:12:27.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.473+0000 7fb29f200640 1 -- 172.21.15.39:0/70394936 wait complete. 2024-03-18T16:12:27.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:12:27 smithi194 ceph-mon[36200]: pgmap v493: 129 pgs: 129 active+clean; 12 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 280 MiB/s wr, 300 op/s 2024-03-18T16:12:27.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:12:27 smithi194 ceph-mon[36200]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T16:12:27.498 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:12:27 smithi039 ceph-mon[31721]: pgmap v493: 129 pgs: 129 active+clean; 12 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 280 MiB/s wr, 300 op/s 2024-03-18T16:12:27.498 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:12:27 smithi039 ceph-mon[31721]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T16:12:27.505 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T16:12:27.505 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T16:12:27.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:12:27 smithi196 ceph-mon[36307]: pgmap v493: 129 pgs: 129 active+clean; 12 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 280 MiB/s wr, 300 op/s 2024-03-18T16:12:27.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:12:27 smithi196 ceph-mon[36307]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T16:12:27.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.819+0000 7f5b4bc00640 1 -- 172.21.15.39:0/1653414960 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5b4410b150 msgr2=0x7f5b44111f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:27.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.819+0000 7f5b4bc00640 1 --2- 172.21.15.39:0/1653414960 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5b4410b150 0x7f5b44111f20 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f5b30009860 tx=0x7f5b3002bb70 comp rx=0 tx=0).stop 2024-03-18T16:12:27.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.819+0000 7f5b4bc00640 1 -- 172.21.15.39:0/1653414960 shutdown_connections 2024-03-18T16:12:27.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.819+0000 7f5b4bc00640 1 --2- 172.21.15.39:0/1653414960 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5b4410b150 0x7f5b44111f20 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:27.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.819+0000 7f5b4bc00640 1 --2- 172.21.15.39:0/1653414960 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5b4410b5d0 0x7f5b44114470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:27.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.819+0000 7f5b4bc00640 1 --2- 172.21.15.39:0/1653414960 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5b441087e0 0x7f5b4410ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:27.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.819+0000 7f5b4bc00640 1 -- 172.21.15.39:0/1653414960 >> 172.21.15.39:0/1653414960 conn(0x7f5b4406e9e0 msgr2=0x7f5b44070e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:27.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.819+0000 7f5b4bc00640 1 -- 172.21.15.39:0/1653414960 shutdown_connections 2024-03-18T16:12:27.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.819+0000 7f5b4bc00640 1 -- 172.21.15.39:0/1653414960 wait complete. 2024-03-18T16:12:27.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.821+0000 7f5b4bc00640 1 Processor -- start 2024-03-18T16:12:27.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.821+0000 7f5b4bc00640 1 -- start start 2024-03-18T16:12:27.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.821+0000 7f5b4bc00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5b441087e0 0x7f5b4410f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:27.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.821+0000 7f5b4bc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5b4410b150 0x7f5b4410f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:27.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.821+0000 7f5b4bc00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5b4410b5d0 0x7f5b4410bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:27.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.821+0000 7f5b4bc00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f5b44117e80 con 0x7f5b4410b150 2024-03-18T16:12:27.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.821+0000 7f5b4bc00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f5b44117d00 con 0x7f5b4410b5d0 2024-03-18T16:12:27.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.821+0000 7f5b4bc00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f5b44118000 con 0x7f5b441087e0 2024-03-18T16:12:27.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.821+0000 7f5b49200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5b441087e0 0x7f5b4410f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:27.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.821+0000 7f5b49200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5b441087e0 0x7f5b4410f470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:54756/0 (socket says 172.21.15.39:54756) 2024-03-18T16:12:27.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.821+0000 7f5b49200640 1 -- 172.21.15.39:0/1755101136 learned_addr learned my addr 172.21.15.39:0/1755101136 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:12:27.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.822+0000 7f5b43e00640 1 --2- 172.21.15.39:0/1755101136 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5b4410b150 0x7f5b4410f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:27.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.822+0000 7f5b49200640 1 -- 172.21.15.39:0/1755101136 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5b4410b5d0 msgr2=0x7f5b4410bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:27.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.822+0000 7f5b49200640 1 --2- 172.21.15.39:0/1755101136 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5b4410b5d0 0x7f5b4410bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:27.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.822+0000 7f5b49200640 1 -- 172.21.15.39:0/1755101136 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5b4410b150 msgr2=0x7f5b4410f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:27.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.822+0000 7f5b49200640 1 --2- 172.21.15.39:0/1755101136 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5b4410b150 0x7f5b4410f9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:27.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.822+0000 7f5b49200640 1 -- 172.21.15.39:0/1755101136 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b4410c8a0 con 0x7f5b441087e0 2024-03-18T16:12:27.825 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.822+0000 7f5b49200640 1 --2- 172.21.15.39:0/1755101136 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5b441087e0 0x7f5b4410f470 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f5b3c00ede0 tx=0x7f5b3c00c6e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:27.825 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.822+0000 7f5b41600640 1 -- 172.21.15.39:0/1755101136 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b3c01c070 con 0x7f5b441087e0 2024-03-18T16:12:27.825 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.822+0000 7f5b41600640 1 -- 172.21.15.39:0/1755101136 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b3c00fcf0 con 0x7f5b441087e0 2024-03-18T16:12:27.825 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.822+0000 7f5b41600640 1 -- 172.21.15.39:0/1755101136 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b3c017e30 con 0x7f5b441087e0 2024-03-18T16:12:27.825 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.824+0000 7f5b4bc00640 1 -- 172.21.15.39:0/1755101136 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b441a2160 con 0x7f5b441087e0 2024-03-18T16:12:27.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.824+0000 7f5b4bc00640 1 -- 172.21.15.39:0/1755101136 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b441a25d0 con 0x7f5b441087e0 2024-03-18T16:12:27.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.829+0000 7f5b4e9bb740 1 Processor -- start 2024-03-18T16:12:27.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.830+0000 7f5b41600640 1 -- 172.21.15.39:0/1755101136 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f5b3c00f810 con 0x7f5b441087e0 2024-03-18T16:12:27.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.831+0000 7f5b4e9bb740 1 -- start start 2024-03-18T16:12:27.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.831+0000 7f5b4e9bb740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cf1f240810 0x55cf1f240c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:27.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.831+0000 7f5b4e9bb740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cf1f241200 0x55cf1f245d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:27.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.831+0000 7f5b4e9bb740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cf1f246370 0x55cf1f248800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:27.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.831+0000 7f5b4e9bb740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55cf1f1a51f0 con 0x55cf1f246370 2024-03-18T16:12:27.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.831+0000 7f5b4e9bb740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55cf1f14da80 con 0x55cf1f241200 2024-03-18T16:12:27.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.831+0000 7f5b4e9bb740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55cf1f1a1290 con 0x55cf1f240810 2024-03-18T16:12:27.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.832+0000 7f5b49c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cf1f240810 0x55cf1f240c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:27.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.832+0000 7f5b49c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cf1f240810 0x55cf1f240c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:54772/0 (socket says 172.21.15.39:54772) 2024-03-18T16:12:27.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.832+0000 7f5b49c00640 1 -- 172.21.15.39:0/555237700 learned_addr learned my addr 172.21.15.39:0/555237700 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:12:27.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.833+0000 7f5b43e00640 1 --2- 172.21.15.39:0/555237700 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cf1f246370 0x55cf1f248800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:27.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.833+0000 7f5b41600640 1 --2- 172.21.15.39:0/1755101136 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f5b280a2900 0x7f5b280a4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:27.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.833+0000 7f5b41600640 1 -- 172.21.15.39:0/1755101136 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f5b3c0cf350 con 0x7f5b441087e0 2024-03-18T16:12:27.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.834+0000 7f5b49c00640 1 -- 172.21.15.39:0/555237700 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cf1f241200 msgr2=0x55cf1f245d00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:12:27.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.834+0000 7f5b49c00640 1 --2- 172.21.15.39:0/555237700 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cf1f241200 0x55cf1f245d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:27.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.834+0000 7f5b49c00640 1 -- 172.21.15.39:0/555237700 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cf1f246370 msgr2=0x55cf1f248800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:12:27.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.834+0000 7f5b49c00640 1 --2- 172.21.15.39:0/555237700 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cf1f246370 0x55cf1f248800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:27.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.834+0000 7f5b49c00640 1 -- 172.21.15.39:0/555237700 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf1ef837d0 con 0x55cf1f240810 2024-03-18T16:12:27.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.834+0000 7f5b49c00640 1 --2- 172.21.15.39:0/1755101136 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f5b280a2900 0x7f5b280a4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:27.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.834+0000 7f5b43e00640 1 --2- 172.21.15.39:0/555237700 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cf1f246370 0x55cf1f248800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:12:27.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.834+0000 7f5b49c00640 1 --2- 172.21.15.39:0/555237700 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cf1f240810 0x55cf1f240c30 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f5b3400bbc0 tx=0x7f5b3400d6b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=816cbe820eede7e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:27.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.835+0000 7f5b3a000640 1 -- 172.21.15.39:0/555237700 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b34018070 con 0x55cf1f240810 2024-03-18T16:12:27.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.835+0000 7f5b49c00640 1 --2- 172.21.15.39:0/1755101136 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f5b280a2900 0x7f5b280a4dc0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f5b3401c000 tx=0x7f5b34011a90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:27.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.835+0000 7f5b3a000640 1 -- 172.21.15.39:0/555237700 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b34012c40 con 0x55cf1f240810 2024-03-18T16:12:27.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.835+0000 7f5b3a000640 1 -- 172.21.15.39:0/555237700 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b340113e0 con 0x55cf1f240810 2024-03-18T16:12:27.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.836+0000 7f5b4e9bb740 1 -- 172.21.15.39:0/555237700 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cf1f240810 msgr2=0x55cf1f240c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:27.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.836+0000 7f5b4e9bb740 1 --2- 172.21.15.39:0/555237700 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cf1f240810 0x55cf1f240c30 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f5b3400bbc0 tx=0x7f5b3400d6b0 comp rx=0 tx=0).stop 2024-03-18T16:12:27.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.837+0000 7f5b4e9bb740 1 -- 172.21.15.39:0/555237700 shutdown_connections 2024-03-18T16:12:27.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.837+0000 7f5b4e9bb740 1 --2- 172.21.15.39:0/555237700 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cf1f241200 0x55cf1f245d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:27.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.837+0000 7f5b4e9bb740 1 --2- 172.21.15.39:0/555237700 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cf1f246370 0x55cf1f248800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:27.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.837+0000 7f5b4e9bb740 1 --2- 172.21.15.39:0/555237700 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cf1f240810 0x55cf1f240c30 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:27.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.837+0000 7f5b4e9bb740 1 -- 172.21.15.39:0/555237700 >> 172.21.15.39:0/555237700 conn(0x55cf1f23bd80 msgr2=0x55cf1f23c180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:27.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.838+0000 7f5b4e9bb740 1 -- 172.21.15.39:0/555237700 shutdown_connections 2024-03-18T16:12:27.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.839+0000 7f5b4e9bb740 1 -- 172.21.15.39:0/555237700 wait complete. 2024-03-18T16:12:27.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.839+0000 7f5b4e9bb740 1 -- start start 2024-03-18T16:12:27.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.839+0000 7f5b4e9bb740 1 Processor -- start 2024-03-18T16:12:27.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.840+0000 7f5b4e9bb740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cf1f240810 0x55cf1f24ad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:27.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.840+0000 7f5b4e9bb740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cf1f241200 0x55cf1f24b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:27.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.840+0000 7f5b4e9bb740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cf1f246370 0x55cf1f252310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:27.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.840+0000 7f5b4e9bb740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55cf1f249320 con 0x55cf1f240810 2024-03-18T16:12:27.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.840+0000 7f5b4e9bb740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55cf1f160ed0 con 0x55cf1f246370 2024-03-18T16:12:27.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.840+0000 7f5b4e9bb740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55cf1f2494a0 con 0x55cf1f241200 2024-03-18T16:12:27.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.840+0000 7f5b4e9bb740 10 client.0 _ll_register_callbacks cb 0x55cf1f19c220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:12:27.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.841+0000 7f5b49200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cf1f240810 0x55cf1f24ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:27.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.841+0000 7f5b49200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cf1f240810 0x55cf1f24ad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:42240/0 (socket says 172.21.15.39:42240) 2024-03-18T16:12:27.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.841+0000 7f5b49200640 1 -- 172.21.15.39:0/2964515316 learned_addr learned my addr 172.21.15.39:0/2964515316 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:12:27.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.841+0000 7f5b43e00640 1 --2- 172.21.15.39:0/2964515316 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cf1f241200 0x55cf1f24b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:27.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.841+0000 7f5b49c00640 1 --2- 172.21.15.39:0/2964515316 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cf1f246370 0x55cf1f252310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:27.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.842+0000 7f5b49200640 1 -- 172.21.15.39:0/2964515316 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cf1f241200 msgr2=0x55cf1f24b2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:12:27.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.842+0000 7f5b49200640 1 --2- 172.21.15.39:0/2964515316 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cf1f241200 0x55cf1f24b2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:27.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.842+0000 7f5b49200640 1 -- 172.21.15.39:0/2964515316 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cf1f246370 msgr2=0x55cf1f252310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:12:27.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.842+0000 7f5b49200640 1 --2- 172.21.15.39:0/2964515316 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cf1f246370 0x55cf1f252310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:27.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.842+0000 7f5b49200640 1 -- 172.21.15.39:0/2964515316 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf1eeb93f0 con 0x55cf1f240810 2024-03-18T16:12:27.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.842+0000 7f5b2ea00640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-03-18T16:12:27.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.842+0000 7f5b49200640 1 --2- 172.21.15.39:0/2964515316 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cf1f240810 0x55cf1f24ad90 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5b3c0a3b70 tx=0x7f5b3c09b4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14d4d0c0ec5771cf server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:27.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.842+0000 7f5b2ea00640 1 -- 172.21.15.39:0/2964515316 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b3c09b900 con 0x55cf1f240810 2024-03-18T16:12:27.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.842+0000 7f5b43e00640 1 --2- 172.21.15.39:0/2964515316 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cf1f241200 0x55cf1f24b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:12:27.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.842+0000 7f5b2ea00640 1 -- 172.21.15.39:0/2964515316 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f5b3c0174d0 con 0x55cf1f240810 2024-03-18T16:12:27.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.843+0000 7f5b4e9bb740 1 -- 172.21.15.39:0/2964515316 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5b34012c40 con 0x55cf1f240810 2024-03-18T16:12:27.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.843+0000 7f5b2ea00640 1 -- 172.21.15.39:0/2964515316 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f5b3c09b900 con 0x55cf1f240810 2024-03-18T16:12:27.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.843+0000 7f5b4e9bb740 10 client.15912 fetch_fsmap learned FSMap version 33 2024-03-18T16:12:27.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.843+0000 7f5b4e9bb740 1 -- 172.21.15.39:0/2964515316 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55cf1ee790f0 con 0x55cf1f240810 2024-03-18T16:12:27.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.843+0000 7f5b2ea00640 1 -- 172.21.15.39:0/2964515316 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b34012c40 con 0x55cf1f240810 2024-03-18T16:12:27.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.844+0000 7f5b2ea00640 1 -- 172.21.15.39:0/2964515316 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 33) ==== 8386+0+0 (secure 0 0 0) 0x7f5b3c0afce0 con 0x55cf1f240810 2024-03-18T16:12:27.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.844+0000 7f5b4e9bb740 10 client.15912 fetch_fsmap finished waiting for FSMap version 33 2024-03-18T16:12:27.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.844+0000 7f5b4e9bb740 10 client.15912 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:12:27.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.844+0000 7f5b4e9bb740 20 client.15912 populate_metadata read hostname 'smithi039' 2024-03-18T16:12:27.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.844+0000 7f5b4e9bb740 1 --2- 172.21.15.39:0/2964515316 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55cf1f257150 0x55cf1f2595d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:27.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.844+0000 7f5b4e9bb740 4 client.15912 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:12:27.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.844+0000 7f5b4e9bb740 1 -- 172.21.15.39:0/2964515316 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cf1f1b8be0 con 0x55cf1f257150 2024-03-18T16:12:27.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.846+0000 7f5b43e00640 1 --2- 172.21.15.39:0/2964515316 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55cf1f257150 0x55cf1f2595d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:27.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.847+0000 7f5b43e00640 1 --2- 172.21.15.39:0/2964515316 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55cf1f257150 0x55cf1f2595d0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e28f2ed5425e4319 server_cookie=bd4cedc815ed24d3 in_seq=0 out_seq=0 2024-03-18T16:12:27.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.847+0000 7f5b2ea00640 10 client.15912 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:12:27.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.849+0000 7f5b2ea00640 1 -- 172.21.15.39:0/2964515316 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55cf1f1b8be0 con 0x55cf1f257150 2024-03-18T16:12:27.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.849+0000 7f5b2ea00640 10 client.15912 handle_command_reply: tid=0 2024-03-18T16:12:27.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.849+0000 7f5b4e9bb740 1 client.15912 shutdown 2024-03-18T16:12:27.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.850+0000 7f5b4e9bb740 1 -- 172.21.15.39:0/2964515316 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cf1f240810 msgr2=0x55cf1f24ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:27.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.850+0000 7f5b4e9bb740 1 --2- 172.21.15.39:0/2964515316 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cf1f240810 0x55cf1f24ad90 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5b3c0a3b70 tx=0x7f5b3c09b4c0 comp rx=0 tx=0).stop 2024-03-18T16:12:27.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.850+0000 7f5b4e9bb740 1 -- 172.21.15.39:0/2964515316 shutdown_connections 2024-03-18T16:12:27.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.850+0000 7f5b4e9bb740 1 --2- 172.21.15.39:0/2964515316 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55cf1f257150 0x55cf1f2595d0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:27.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.850+0000 7f5b2ea00640 0 client.15912 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:12:27.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.850+0000 7f5b4e9bb740 1 --2- 172.21.15.39:0/2964515316 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cf1f246370 0x55cf1f252310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:27.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.850+0000 7f5b4e9bb740 1 --2- 172.21.15.39:0/2964515316 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cf1f241200 0x55cf1f24b2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:27.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.851+0000 7f5b4e9bb740 1 --2- 172.21.15.39:0/2964515316 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cf1f240810 0x55cf1f24ad90 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:27.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.851+0000 7f5b4e9bb740 1 -- 172.21.15.39:0/2964515316 >> 172.21.15.39:0/2964515316 conn(0x55cf1f23bd80 msgr2=0x55cf1f23aec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:27.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.851+0000 7f5b4e9bb740 1 -- 172.21.15.39:0/2964515316 shutdown_connections 2024-03-18T16:12:27.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.851+0000 7f5b4e9bb740 1 -- 172.21.15.39:0/2964515316 wait complete. 2024-03-18T16:12:27.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.853+0000 7f5b4e9bb740 20 client.15912 trim_cache size 0 max 16384 2024-03-18T16:12:27.894 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.893+0000 7f5b4e9bb740 1 Processor -- start 2024-03-18T16:12:27.894 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.893+0000 7f5b4e9bb740 1 -- start start 2024-03-18T16:12:27.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.893+0000 7f5b4e9bb740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cf1f240810 0x55cf1f249840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:27.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.893+0000 7f5b4e9bb740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cf1f241200 0x55cf1f2522d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:27.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.893+0000 7f5b4e9bb740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cf1f246370 0x55cf1f24a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:27.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.893+0000 7f5b4e9bb740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f5b441c1ca0 con 0x55cf1f240810 2024-03-18T16:12:27.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.893+0000 7f5b4e9bb740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f5b441c3900 con 0x55cf1f241200 2024-03-18T16:12:27.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.893+0000 7f5b4e9bb740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f5b441c2370 con 0x55cf1f246370 2024-03-18T16:12:27.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.894+0000 7f5b43e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cf1f241200 0x55cf1f2522d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:27.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.894+0000 7f5b49c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cf1f246370 0x55cf1f24a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:27.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.894+0000 7f5b43e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cf1f241200 0x55cf1f2522d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:55416/0 (socket says 172.21.15.39:55416) 2024-03-18T16:12:27.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.894+0000 7f5b43e00640 1 -- 172.21.15.39:0/352640245 learned_addr learned my addr 172.21.15.39:0/352640245 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:12:27.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.894+0000 7f5b43e00640 1 -- 172.21.15.39:0/352640245 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cf1f246370 msgr2=0x55cf1f24a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:12:27.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.894+0000 7f5b43e00640 1 --2- 172.21.15.39:0/352640245 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cf1f246370 0x55cf1f24a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:27.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.895+0000 7f5b43e00640 1 -- 172.21.15.39:0/352640245 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cf1f240810 msgr2=0x55cf1f249840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:12:27.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.895+0000 7f5b43e00640 1 --2- 172.21.15.39:0/352640245 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cf1f240810 0x55cf1f249840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:27.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.895+0000 7f5b43e00640 1 -- 172.21.15.39:0/352640245 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf1f243030 con 0x55cf1f241200 2024-03-18T16:12:27.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.895+0000 7f5b49c00640 1 --2- 172.21.15.39:0/352640245 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cf1f246370 0x55cf1f24a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:12:27.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.896+0000 7f5b43e00640 1 --2- 172.21.15.39:0/352640245 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cf1f241200 0x55cf1f2522d0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f5b30009830 tx=0x7f5b30032460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e7d1fe3577f1060 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:27.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.898+0000 7f5b39600640 1 -- 172.21.15.39:0/352640245 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b30039040 con 0x55cf1f241200 2024-03-18T16:12:27.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.898+0000 7f5b39600640 1 -- 172.21.15.39:0/352640245 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b30004040 con 0x55cf1f241200 2024-03-18T16:12:27.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.898+0000 7f5b39600640 1 -- 172.21.15.39:0/352640245 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b30005040 con 0x55cf1f241200 2024-03-18T16:12:27.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.898+0000 7f5b4e9bb740 1 -- 172.21.15.39:0/352640245 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cf1f241200 msgr2=0x55cf1f2522d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:27.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.898+0000 7f5b4e9bb740 1 --2- 172.21.15.39:0/352640245 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cf1f241200 0x55cf1f2522d0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f5b30009830 tx=0x7f5b30032460 comp rx=0 tx=0).stop 2024-03-18T16:12:27.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.899+0000 7f5b4e9bb740 1 -- 172.21.15.39:0/352640245 shutdown_connections 2024-03-18T16:12:27.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.899+0000 7f5b4e9bb740 1 --2- 172.21.15.39:0/352640245 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cf1f241200 0x55cf1f2522d0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:27.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.899+0000 7f5b4e9bb740 1 --2- 172.21.15.39:0/352640245 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cf1f246370 0x55cf1f24a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:27.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.899+0000 7f5b4e9bb740 1 --2- 172.21.15.39:0/352640245 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cf1f240810 0x55cf1f249840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:27.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.899+0000 7f5b4e9bb740 1 -- 172.21.15.39:0/352640245 >> 172.21.15.39:0/352640245 conn(0x55cf1f23bd80 msgr2=0x55cf1f23c160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:27.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.899+0000 7f5b4e9bb740 1 -- 172.21.15.39:0/352640245 shutdown_connections 2024-03-18T16:12:27.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.899+0000 7f5b4e9bb740 1 -- 172.21.15.39:0/352640245 wait complete. 2024-03-18T16:12:27.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.900+0000 7f5b4e9bb740 1 -- start start 2024-03-18T16:12:27.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.900+0000 7f5b4e9bb740 1 Processor -- start 2024-03-18T16:12:27.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.901+0000 7f5b4e9bb740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cf1f240810 0x55cf1f24fb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:27.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.901+0000 7f5b4e9bb740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cf1f246370 0x55cf1f250040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:27.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.901+0000 7f5b4e9bb740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cf1f251050 0x55cf1f2514d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:27.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.901+0000 7f5b4e9bb740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f5b441c31d0 con 0x55cf1f240810 2024-03-18T16:12:27.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.901+0000 7f5b4e9bb740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f5b441c2a40 con 0x55cf1f251050 2024-03-18T16:12:27.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.901+0000 7f5b4e9bb740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f5b441bfc20 con 0x55cf1f246370 2024-03-18T16:12:27.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.901+0000 7f5b4e9bb740 10 client.0 _ll_register_callbacks cb 0x7f5b3c09b6d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:12:27.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.901+0000 7f5b49c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cf1f251050 0x55cf1f2514d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:27.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.901+0000 7f5b43e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cf1f246370 0x55cf1f250040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:27.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.901+0000 7f5b49c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cf1f251050 0x55cf1f2514d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:55430/0 (socket says 172.21.15.39:55430) 2024-03-18T16:12:27.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.901+0000 7f5b49c00640 1 -- 172.21.15.39:0/3146416449 learned_addr learned my addr 172.21.15.39:0/3146416449 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:12:27.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.901+0000 7f5b49200640 1 --2- 172.21.15.39:0/3146416449 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cf1f240810 0x55cf1f24fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:27.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.902+0000 7f5b43e00640 1 -- 172.21.15.39:0/3146416449 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cf1f251050 msgr2=0x55cf1f2514d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:12:27.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.902+0000 7f5b43e00640 1 --2- 172.21.15.39:0/3146416449 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cf1f251050 0x55cf1f2514d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:27.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.902+0000 7f5b43e00640 1 -- 172.21.15.39:0/3146416449 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cf1f240810 msgr2=0x55cf1f24fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:12:27.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.902+0000 7f5b43e00640 1 --2- 172.21.15.39:0/3146416449 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cf1f240810 0x55cf1f24fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:27.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.902+0000 7f5b43e00640 1 -- 172.21.15.39:0/3146416449 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf1f251a10 con 0x55cf1f246370 2024-03-18T16:12:27.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.902+0000 7f5b43e00640 1 --2- 172.21.15.39:0/3146416449 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cf1f246370 0x55cf1f250040 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x55cf1f241b80 tx=0x7f5b30035f50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f57f6fd4ad142087 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:27.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.903+0000 7f5b2ea00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:12:27.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.903+0000 7f5b2ea00640 1 -- 172.21.15.39:0/3146416449 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b30039040 con 0x55cf1f246370 2024-03-18T16:12:27.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.903+0000 7f5b2ea00640 1 -- 172.21.15.39:0/3146416449 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f5b300435d0 con 0x55cf1f246370 2024-03-18T16:12:27.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.903+0000 7f5b4e9bb740 1 -- 172.21.15.39:0/3146416449 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5b30004040 con 0x55cf1f246370 2024-03-18T16:12:27.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.903+0000 7f5b2ea00640 1 -- 172.21.15.39:0/3146416449 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7f5b30043810 con 0x55cf1f246370 2024-03-18T16:12:27.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.904+0000 7f5b4e9bb740 10 client.26143 fetch_fsmap learned FSMap version 33 2024-03-18T16:12:27.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.904+0000 7f5b4e9bb740 1 -- 172.21.15.39:0/3146416449 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55cf1f2bd9d0 con 0x55cf1f246370 2024-03-18T16:12:27.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.904+0000 7f5b2ea00640 1 -- 172.21.15.39:0/3146416449 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b30004040 con 0x55cf1f246370 2024-03-18T16:12:27.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.904+0000 7f5b2ea00640 1 -- 172.21.15.39:0/3146416449 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 33) ==== 8386+0+0 (secure 0 0 0) 0x7f5b3002c510 con 0x55cf1f246370 2024-03-18T16:12:27.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.904+0000 7f5b4e9bb740 10 client.26143 fetch_fsmap finished waiting for FSMap version 33 2024-03-18T16:12:27.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.904+0000 7f5b4e9bb740 10 client.26143 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:12:27.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.904+0000 7f5b4e9bb740 20 client.26143 populate_metadata read hostname 'smithi039' 2024-03-18T16:12:27.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.904+0000 7f5b4e9bb740 1 --2- 172.21.15.39:0/3146416449 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55cf1f2be2d0 0x55cf1f2c06f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:27.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.904+0000 7f5b4e9bb740 4 client.26143 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T16:12:27.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.904+0000 7f5b4e9bb740 1 -- 172.21.15.39:0/3146416449 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55cf1f1b2730 con 0x55cf1f2be2d0 2024-03-18T16:12:27.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.905+0000 7f5b49200640 1 --2- 172.21.15.39:0/3146416449 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55cf1f2be2d0 0x55cf1f2c06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:27.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.905+0000 7f5b2ea00640 10 client.26143 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:12:27.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.907+0000 7f5b49200640 1 --2- 172.21.15.39:0/3146416449 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55cf1f2be2d0 0x55cf1f2c06f0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=feb6f6f2c14592a7 server_cookie=6233ce4c658c32f1 in_seq=0 out_seq=0 2024-03-18T16:12:27.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.907+0000 7f5b2ea00640 1 -- 172.21.15.39:0/3146416449 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+266 (crc 0 0 0) 0x55cf1f1b2730 con 0x55cf1f2be2d0 2024-03-18T16:12:27.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.908+0000 7f5b2ea00640 10 client.26143 handle_command_reply: tid=0 2024-03-18T16:12:27.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.908+0000 7f5b4e9bb740 1 client.26143 shutdown 2024-03-18T16:12:27.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.908+0000 7f5b4e9bb740 1 -- 172.21.15.39:0/3146416449 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cf1f246370 msgr2=0x55cf1f250040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:27.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.908+0000 7f5b4e9bb740 1 --2- 172.21.15.39:0/3146416449 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cf1f246370 0x55cf1f250040 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x55cf1f241b80 tx=0x7f5b30035f50 comp rx=0 tx=0).stop 2024-03-18T16:12:27.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.909+0000 7f5b4e9bb740 1 -- 172.21.15.39:0/3146416449 shutdown_connections 2024-03-18T16:12:27.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.909+0000 7f5b4e9bb740 1 --2- 172.21.15.39:0/3146416449 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55cf1f2be2d0 0x55cf1f2c06f0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:27.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.909+0000 7f5b2ea00640 0 client.26143 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:12:27.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.909+0000 7f5b4e9bb740 1 --2- 172.21.15.39:0/3146416449 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cf1f251050 0x55cf1f2514d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:27.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.909+0000 7f5b4e9bb740 1 --2- 172.21.15.39:0/3146416449 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cf1f246370 0x55cf1f250040 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:27.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.909+0000 7f5b4e9bb740 1 --2- 172.21.15.39:0/3146416449 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cf1f240810 0x55cf1f24fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:27.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.909+0000 7f5b4e9bb740 1 -- 172.21.15.39:0/3146416449 >> 172.21.15.39:0/3146416449 conn(0x55cf1f23bd80 msgr2=0x55cf1f24a9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:27.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.909+0000 7f5b4e9bb740 1 -- 172.21.15.39:0/3146416449 shutdown_connections 2024-03-18T16:12:27.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.909+0000 7f5b4e9bb740 1 -- 172.21.15.39:0/3146416449 wait complete. 2024-03-18T16:12:27.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.910+0000 7f5b4e9bb740 20 client.26143 trim_cache size 0 max 16384 2024-03-18T16:12:27.912 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-03-18T16:12:27.913 INFO:teuthology.orchestra.run.smithi039.stdout: "status": "scrub active (70 inodes in the stack)", 2024-03-18T16:12:27.913 INFO:teuthology.orchestra.run.smithi039.stdout: "scrubs": { 2024-03-18T16:12:27.913 INFO:teuthology.orchestra.run.smithi039.stdout: "93d229b6-d830-40af-a394-0e1bd9a6e54f": { 2024-03-18T16:12:27.913 INFO:teuthology.orchestra.run.smithi039.stdout: "path": "/", 2024-03-18T16:12:27.913 INFO:teuthology.orchestra.run.smithi039.stdout: "tag": "93d229b6-d830-40af-a394-0e1bd9a6e54f", 2024-03-18T16:12:27.913 INFO:teuthology.orchestra.run.smithi039.stdout: "options": "recursive,force" 2024-03-18T16:12:27.913 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:12:27.913 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:12:27.913 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-03-18T16:12:27.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.912+0000 7f5b4bc00640 1 -- 172.21.15.39:0/1755101136 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f5b280a2900 msgr2=0x7f5b280a4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:27.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.912+0000 7f5b4bc00640 1 --2- 172.21.15.39:0/1755101136 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f5b280a2900 0x7f5b280a4dc0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f5b3401c000 tx=0x7f5b34011a90 comp rx=0 tx=0).stop 2024-03-18T16:12:27.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.912+0000 7f5b4bc00640 1 -- 172.21.15.39:0/1755101136 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5b441087e0 msgr2=0x7f5b4410f470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:27.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.912+0000 7f5b4bc00640 1 --2- 172.21.15.39:0/1755101136 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5b441087e0 0x7f5b4410f470 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f5b3c00ede0 tx=0x7f5b3c00c6e0 comp rx=0 tx=0).stop 2024-03-18T16:12:27.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.912+0000 7f5b4bc00640 1 -- 172.21.15.39:0/1755101136 shutdown_connections 2024-03-18T16:12:27.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.912+0000 7f5b4bc00640 1 --2- 172.21.15.39:0/1755101136 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5b4410b5d0 0x7f5b4410bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:27.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.912+0000 7f5b4bc00640 1 --2- 172.21.15.39:0/1755101136 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f5b280a2900 0x7f5b280a4dc0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:27.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.912+0000 7f5b4bc00640 1 --2- 172.21.15.39:0/1755101136 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5b4410b150 0x7f5b4410f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:27.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.912+0000 7f5b4bc00640 1 --2- 172.21.15.39:0/1755101136 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5b441087e0 0x7f5b4410f470 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:27.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.912+0000 7f5b4bc00640 1 -- 172.21.15.39:0/1755101136 >> 172.21.15.39:0/1755101136 conn(0x7f5b4406e9e0 msgr2=0x7f5b4406f320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:27.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.912+0000 7f5b4bc00640 1 -- 172.21.15.39:0/1755101136 shutdown_connections 2024-03-18T16:12:27.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:27.913+0000 7f5b4bc00640 1 -- 172.21.15.39:0/1755101136 wait complete. 2024-03-18T16:12:27.941 INFO:tasks.cephfs.filesystem:scrub status for tag:93d229b6-d830-40af-a394-0e1bd9a6e54f - {'path': '/', 'tag': '93d229b6-d830-40af-a394-0e1bd9a6e54f', 'options': 'recursive,force'} 2024-03-18T16:12:28.235 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:12:28 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:12:28.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:12:28 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:12:28.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:12:28 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:12:29.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:12:29 smithi194 ceph-mon[36200]: scrub queued for path: / 2024-03-18T16:12:29.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:12:29 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:12:29.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:12:29 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:12:29.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:12:29 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:12:29.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:12:29 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:12:29.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:12:29 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:12:29.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:12:29 smithi194 ceph-mon[36200]: pgmap v494: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 255 MiB/s wr, 1.08k op/s 2024-03-18T16:12:29.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:12:29 smithi039 ceph-mon[31721]: scrub queued for path: / 2024-03-18T16:12:29.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:12:29 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:12:29.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:12:29 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:12:29.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:12:29 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:12:29.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:12:29 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:12:29.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:12:29 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:12:29.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:12:29 smithi039 ceph-mon[31721]: pgmap v494: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 255 MiB/s wr, 1.08k op/s 2024-03-18T16:12:29.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:12:29 smithi196 ceph-mon[36307]: scrub queued for path: / 2024-03-18T16:12:29.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:12:29 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:12:29.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:12:29 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:12:29.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:12:29 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:12:29.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:12:29 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:12:29.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:12:29 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:12:29.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:12:29 smithi196 ceph-mon[36307]: pgmap v494: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 255 MiB/s wr, 1.08k op/s 2024-03-18T16:12:31.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:12:31 smithi039 ceph-mon[31721]: pgmap v495: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 204 MiB/s wr, 2.13k op/s 2024-03-18T16:12:31.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:12:31 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:12:31.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:12:31 smithi196 ceph-mon[36307]: pgmap v495: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 204 MiB/s wr, 2.13k op/s 2024-03-18T16:12:31.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:12:31 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:12:31.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:12:31 smithi194 ceph-mon[36200]: pgmap v495: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 183 KiB/s rd, 204 MiB/s wr, 2.13k op/s 2024-03-18T16:12:31.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:12:31 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:12:33.530 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:12:33 smithi196 ceph-mon[36307]: pgmap v496: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 156 MiB/s wr, 2.78k op/s 2024-03-18T16:12:33.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:12:33 smithi039 ceph-mon[31721]: pgmap v496: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 156 MiB/s wr, 2.78k op/s 2024-03-18T16:12:33.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:12:33 smithi194 ceph-mon[36200]: pgmap v496: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 156 MiB/s wr, 2.78k op/s 2024-03-18T16:12:35.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:12:35 smithi039 ceph-mon[31721]: pgmap v497: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 131 MiB/s wr, 4.92k op/s 2024-03-18T16:12:35.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:12:35 smithi196 ceph-mon[36307]: pgmap v497: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 131 MiB/s wr, 4.92k op/s 2024-03-18T16:12:35.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:12:35 smithi194 ceph-mon[36200]: pgmap v497: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 202 KiB/s rd, 131 MiB/s wr, 4.92k op/s 2024-03-18T16:12:37.590 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:12:37 smithi039 ceph-mon[31721]: pgmap v498: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 60 MiB/s wr, 5.64k op/s 2024-03-18T16:12:37.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:12:37 smithi196 ceph-mon[36307]: pgmap v498: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 60 MiB/s wr, 5.64k op/s 2024-03-18T16:12:37.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:12:37 smithi194 ceph-mon[36200]: pgmap v498: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 199 KiB/s rd, 60 MiB/s wr, 5.64k op/s 2024-03-18T16:12:39.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:12:39 smithi039 ceph-mon[31721]: pgmap v499: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 51 MiB/s wr, 6.76k op/s 2024-03-18T16:12:39.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:12:39 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:12:39.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:12:39 smithi196 ceph-mon[36307]: pgmap v499: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 51 MiB/s wr, 6.76k op/s 2024-03-18T16:12:39.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:12:39 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:12:39.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:12:39 smithi194 ceph-mon[36200]: pgmap v499: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 51 MiB/s wr, 6.76k op/s 2024-03-18T16:12:39.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:12:39 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:12:40.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:12:40 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:12:40.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:12:40 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:12:40.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:12:40 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:12:41.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:12:41 smithi039 ceph-mon[31721]: pgmap v500: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 48 MiB/s wr, 7.88k op/s 2024-03-18T16:12:41.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:12:41 smithi196 ceph-mon[36307]: pgmap v500: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 48 MiB/s wr, 7.88k op/s 2024-03-18T16:12:41.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:12:41 smithi194 ceph-mon[36200]: pgmap v500: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 48 MiB/s wr, 7.88k op/s 2024-03-18T16:12:42.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:12:42 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:12:42.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:12:42 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:12:42.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:12:42 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:12:43.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:12:43 smithi039 ceph-mon[31721]: pgmap v501: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 44 MiB/s wr, 7.14k op/s 2024-03-18T16:12:43.641 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:12:43 smithi194 ceph-mon[36200]: pgmap v501: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 44 MiB/s wr, 7.14k op/s 2024-03-18T16:12:43.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:12:43 smithi196 ceph-mon[36307]: pgmap v501: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 44 MiB/s wr, 7.14k op/s 2024-03-18T16:12:45.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:12:45 smithi039 ceph-mon[31721]: pgmap v502: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 57 MiB/s wr, 8.98k op/s 2024-03-18T16:12:45.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:12:45 smithi196 ceph-mon[36307]: pgmap v502: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 57 MiB/s wr, 8.98k op/s 2024-03-18T16:12:45.742 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:12:45 smithi194 ceph-mon[36200]: pgmap v502: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 130 KiB/s rd, 57 MiB/s wr, 8.98k op/s 2024-03-18T16:12:47.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:12:47 smithi039 ceph-mon[31721]: pgmap v503: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 52 MiB/s wr, 7.94k op/s 2024-03-18T16:12:47.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:12:47 smithi196 ceph-mon[36307]: pgmap v503: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 52 MiB/s wr, 7.94k op/s 2024-03-18T16:12:47.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:12:47 smithi194 ceph-mon[36200]: pgmap v503: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 52 MiB/s wr, 7.94k op/s 2024-03-18T16:12:49.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:12:49 smithi039 ceph-mon[31721]: pgmap v504: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 55 MiB/s wr, 8.39k op/s 2024-03-18T16:12:49.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:12:49 smithi196 ceph-mon[36307]: pgmap v504: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 55 MiB/s wr, 8.39k op/s 2024-03-18T16:12:49.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:12:49 smithi194 ceph-mon[36200]: pgmap v504: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 55 MiB/s wr, 8.39k op/s 2024-03-18T16:12:51.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:12:51 smithi196 ceph-mon[36307]: pgmap v505: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 62 MiB/s wr, 9.22k op/s 2024-03-18T16:12:51.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:12:51 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:12:51.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:12:51 smithi194 ceph-mon[36200]: pgmap v505: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 62 MiB/s wr, 9.22k op/s 2024-03-18T16:12:51.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:12:51 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:12:51.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:12:51 smithi039 ceph-mon[31721]: pgmap v505: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 62 MiB/s wr, 9.22k op/s 2024-03-18T16:12:51.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:12:51 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:12:53.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:12:53 smithi196 ceph-mon[36307]: pgmap v506: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 53 MiB/s wr, 7.73k op/s 2024-03-18T16:12:53.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:12:53 smithi194 ceph-mon[36200]: pgmap v506: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 53 MiB/s wr, 7.73k op/s 2024-03-18T16:12:53.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:12:53 smithi039 ceph-mon[31721]: pgmap v506: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 53 MiB/s wr, 7.73k op/s 2024-03-18T16:12:54.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:12:54 smithi196 ceph-mon[36307]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T16:12:54.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:12:54 smithi194 ceph-mon[36200]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T16:12:54.839 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:12:54 smithi039 ceph-mon[31721]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T16:12:55.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:12:55 smithi196 ceph-mon[36307]: pgmap v507: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 66 MiB/s wr, 9.49k op/s 2024-03-18T16:12:55.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:12:55 smithi194 ceph-mon[36200]: pgmap v507: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 66 MiB/s wr, 9.49k op/s 2024-03-18T16:12:55.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:12:55 smithi039 ceph-mon[31721]: pgmap v507: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 66 MiB/s wr, 9.49k op/s 2024-03-18T16:12:57.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:12:57 smithi196 ceph-mon[36307]: pgmap v508: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 56 MiB/s wr, 8.12k op/s 2024-03-18T16:12:57.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:12:57 smithi194 ceph-mon[36200]: pgmap v508: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 56 MiB/s wr, 8.12k op/s 2024-03-18T16:12:57.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:12:57 smithi039 ceph-mon[31721]: pgmap v508: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 56 MiB/s wr, 8.12k op/s 2024-03-18T16:12:57.943 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T16:12:57.943 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T16:12:58.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.251+0000 7fe22a600640 1 -- 172.21.15.39:0/1110889083 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe224101970 msgr2=0x7fe224108290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:58.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.251+0000 7fe22a600640 1 --2- 172.21.15.39:0/1110889083 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe224101970 0x7fe224108290 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fe21c009be0 tx=0x7fe21c02bef0 comp rx=0 tx=0).stop 2024-03-18T16:12:58.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.252+0000 7fe22a600640 1 -- 172.21.15.39:0/1110889083 shutdown_connections 2024-03-18T16:12:58.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.252+0000 7fe22a600640 1 --2- 172.21.15.39:0/1110889083 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe224100fb0 0x7fe224101430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:58.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.252+0000 7fe22a600640 1 --2- 172.21.15.39:0/1110889083 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe224101970 0x7fe224108290 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:58.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.252+0000 7fe22a600640 1 --2- 172.21.15.39:0/1110889083 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe2240ffe60 0x7fe224100280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:58.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.252+0000 7fe22a600640 1 -- 172.21.15.39:0/1110889083 >> 172.21.15.39:0/1110889083 conn(0x7fe2240fb3d0 msgr2=0x7fe2240fd810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:58.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.252+0000 7fe22a600640 1 -- 172.21.15.39:0/1110889083 shutdown_connections 2024-03-18T16:12:58.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.252+0000 7fe22a600640 1 -- 172.21.15.39:0/1110889083 wait complete. 2024-03-18T16:12:58.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.253+0000 7fe22a600640 1 Processor -- start 2024-03-18T16:12:58.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.253+0000 7fe22a600640 1 -- start start 2024-03-18T16:12:58.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.254+0000 7fe22a600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe2240ffe60 0x7fe22419e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:58.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.254+0000 7fe22a600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe224100fb0 0x7fe22419e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:58.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.254+0000 7fe22a600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe224101970 0x7fe2241a5650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:58.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.254+0000 7fe22a600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fe22410ad80 con 0x7fe2240ffe60 2024-03-18T16:12:58.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.254+0000 7fe22a600640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fe22410ac00 con 0x7fe224101970 2024-03-18T16:12:58.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.254+0000 7fe22a600640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fe22410af00 con 0x7fe224100fb0 2024-03-18T16:12:58.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.254+0000 7fe222a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe224100fb0 0x7fe22419e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:58.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.254+0000 7fe223e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe224101970 0x7fe2241a5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:58.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.254+0000 7fe222a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe224100fb0 0x7fe22419e5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:37330/0 (socket says 172.21.15.39:37330) 2024-03-18T16:12:58.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.254+0000 7fe222a00640 1 -- 172.21.15.39:0/1942797328 learned_addr learned my addr 172.21.15.39:0/1942797328 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:12:58.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.255+0000 7fe223400640 1 --2- 172.21.15.39:0/1942797328 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe2240ffe60 0x7fe22419e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:58.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.255+0000 7fe222a00640 1 -- 172.21.15.39:0/1942797328 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe224101970 msgr2=0x7fe2241a5650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:58.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.255+0000 7fe222a00640 1 --2- 172.21.15.39:0/1942797328 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe224101970 0x7fe2241a5650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:58.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.255+0000 7fe222a00640 1 -- 172.21.15.39:0/1942797328 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe2240ffe60 msgr2=0x7fe22419e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:58.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.255+0000 7fe222a00640 1 --2- 172.21.15.39:0/1942797328 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe2240ffe60 0x7fe22419e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:58.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.255+0000 7fe222a00640 1 -- 172.21.15.39:0/1942797328 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2241a5cc0 con 0x7fe224100fb0 2024-03-18T16:12:58.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.255+0000 7fe223400640 1 --2- 172.21.15.39:0/1942797328 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe2240ffe60 0x7fe22419e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:12:58.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.256+0000 7fe222a00640 1 --2- 172.21.15.39:0/1942797328 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe224100fb0 0x7fe22419e5e0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fe21000e9f0 tx=0x7fe21000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:58.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.256+0000 7fe21be00640 1 -- 172.21.15.39:0/1942797328 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe21000cce0 con 0x7fe224100fb0 2024-03-18T16:12:58.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.256+0000 7fe21be00640 1 -- 172.21.15.39:0/1942797328 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe2100035f0 con 0x7fe224100fb0 2024-03-18T16:12:58.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.256+0000 7fe21be00640 1 -- 172.21.15.39:0/1942797328 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe210016670 con 0x7fe224100fb0 2024-03-18T16:12:58.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.257+0000 7fe22a600640 1 -- 172.21.15.39:0/1942797328 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2241a5fb0 con 0x7fe224100fb0 2024-03-18T16:12:58.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.257+0000 7fe22a600640 1 -- 172.21.15.39:0/1942797328 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2241a6420 con 0x7fe224100fb0 2024-03-18T16:12:58.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.268+0000 7fe21be00640 1 -- 172.21.15.39:0/1942797328 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fe21001d020 con 0x7fe224100fb0 2024-03-18T16:12:58.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.270+0000 7fe21be00640 1 --2- 172.21.15.39:0/1942797328 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fe2080a28d0 0x7fe2080a4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:58.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.270+0000 7fe21be00640 1 -- 172.21.15.39:0/1942797328 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fe2100ce980 con 0x7fe224100fb0 2024-03-18T16:12:58.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.270+0000 7fe223400640 1 --2- 172.21.15.39:0/1942797328 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fe2080a28d0 0x7fe2080a4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:58.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.272+0000 7fe223400640 1 --2- 172.21.15.39:0/1942797328 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fe2080a28d0 0x7fe2080a4d90 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fe2140096c0 tx=0x7fe214009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:58.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.275+0000 7fe22d377740 1 Processor -- start 2024-03-18T16:12:58.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.275+0000 7fe22d377740 1 -- start start 2024-03-18T16:12:58.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.276+0000 7fe22d377740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cc0b175810 0x55cc0b175c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:58.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.276+0000 7fe22d377740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cc0b176200 0x55cc0b17ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:58.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.276+0000 7fe22d377740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc0b17b370 0x55cc0b17d800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:58.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.276+0000 7fe22d377740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55cc0b0da1f0 con 0x55cc0b176200 2024-03-18T16:12:58.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.276+0000 7fe22d377740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55cc0b082a80 con 0x55cc0b17b370 2024-03-18T16:12:58.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.276+0000 7fe22d377740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55cc0b0d6290 con 0x55cc0b175810 2024-03-18T16:12:58.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.276+0000 7fe223400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cc0b175810 0x55cc0b175c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:58.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.276+0000 7fe223400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cc0b175810 0x55cc0b175c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:37344/0 (socket says 172.21.15.39:37344) 2024-03-18T16:12:58.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.276+0000 7fe223400640 1 -- 172.21.15.39:0/117400793 learned_addr learned my addr 172.21.15.39:0/117400793 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:12:58.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.276+0000 7fe223e00640 1 --2- 172.21.15.39:0/117400793 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc0b17b370 0x55cc0b17d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:58.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.276+0000 7fe222a00640 1 --2- 172.21.15.39:0/117400793 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cc0b176200 0x55cc0b17ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:58.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.277+0000 7fe223400640 1 -- 172.21.15.39:0/117400793 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc0b17b370 msgr2=0x55cc0b17d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:12:58.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.277+0000 7fe223400640 1 --2- 172.21.15.39:0/117400793 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc0b17b370 0x55cc0b17d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:58.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.277+0000 7fe223400640 1 -- 172.21.15.39:0/117400793 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cc0b176200 msgr2=0x55cc0b17ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:12:58.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.277+0000 7fe223400640 1 --2- 172.21.15.39:0/117400793 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cc0b176200 0x55cc0b17ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:58.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.277+0000 7fe223400640 1 -- 172.21.15.39:0/117400793 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cc0aeb87d0 con 0x55cc0b175810 2024-03-18T16:12:58.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.277+0000 7fe222a00640 1 --2- 172.21.15.39:0/117400793 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cc0b176200 0x55cc0b17ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:12:58.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.277+0000 7fe223400640 1 --2- 172.21.15.39:0/117400793 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cc0b175810 0x55cc0b175c30 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fe21400f6b0 tx=0x7fe214014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a3aae99744f27fb8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:58.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.278+0000 7fe218c00640 1 -- 172.21.15.39:0/117400793 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe214004220 con 0x55cc0b175810 2024-03-18T16:12:58.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.278+0000 7fe218c00640 1 -- 172.21.15.39:0/117400793 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe2140043c0 con 0x55cc0b175810 2024-03-18T16:12:58.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.278+0000 7fe218c00640 1 -- 172.21.15.39:0/117400793 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2140035f0 con 0x55cc0b175810 2024-03-18T16:12:58.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.279+0000 7fe22d377740 1 -- 172.21.15.39:0/117400793 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cc0b175810 msgr2=0x55cc0b175c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:58.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.279+0000 7fe22d377740 1 --2- 172.21.15.39:0/117400793 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cc0b175810 0x55cc0b175c30 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fe21400f6b0 tx=0x7fe214014000 comp rx=0 tx=0).stop 2024-03-18T16:12:58.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.279+0000 7fe22d377740 1 -- 172.21.15.39:0/117400793 shutdown_connections 2024-03-18T16:12:58.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.279+0000 7fe22d377740 1 --2- 172.21.15.39:0/117400793 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc0b17b370 0x55cc0b17d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:58.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.279+0000 7fe22d377740 1 --2- 172.21.15.39:0/117400793 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cc0b176200 0x55cc0b17ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:58.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.279+0000 7fe22d377740 1 --2- 172.21.15.39:0/117400793 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cc0b175810 0x55cc0b175c30 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:58.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.279+0000 7fe22d377740 1 -- 172.21.15.39:0/117400793 >> 172.21.15.39:0/117400793 conn(0x55cc0b170d80 msgr2=0x55cc0b171180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:58.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.280+0000 7fe22d377740 1 -- 172.21.15.39:0/117400793 shutdown_connections 2024-03-18T16:12:58.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.280+0000 7fe22d377740 1 -- 172.21.15.39:0/117400793 wait complete. 2024-03-18T16:12:58.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.281+0000 7fe22d377740 1 -- start start 2024-03-18T16:12:58.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.281+0000 7fe22d377740 1 Processor -- start 2024-03-18T16:12:58.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.283+0000 7fe22d377740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc0b175810 0x55cc0b17fd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:58.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.283+0000 7fe22d377740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cc0b176200 0x55cc0b180280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:58.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.283+0000 7fe22d377740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cc0b17b370 0x55cc0b187280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:58.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.283+0000 7fe223400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc0b175810 0x55cc0b17fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:58.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.283+0000 7fe223400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc0b175810 0x55cc0b17fd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:60724/0 (socket says 172.21.15.39:60724) 2024-03-18T16:12:58.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.283+0000 7fe223400640 1 -- 172.21.15.39:0/3193491739 learned_addr learned my addr 172.21.15.39:0/3193491739 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:12:58.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.284+0000 7fe222a00640 1 --2- 172.21.15.39:0/3193491739 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cc0b176200 0x55cc0b180280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:58.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.284+0000 7fe223e00640 1 --2- 172.21.15.39:0/3193491739 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cc0b17b370 0x55cc0b187280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:58.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.284+0000 7fe22d377740 1 -- 172.21.15.39:0/3193491739 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55cc0b17e370 con 0x55cc0b17b370 2024-03-18T16:12:58.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.284+0000 7fe22d377740 1 -- 172.21.15.39:0/3193491739 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55cc0b095ed0 con 0x55cc0b175810 2024-03-18T16:12:58.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.284+0000 7fe22d377740 1 -- 172.21.15.39:0/3193491739 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55cc0b17e4f0 con 0x55cc0b176200 2024-03-18T16:12:58.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.284+0000 7fe22d377740 10 client.0 _ll_register_callbacks cb 0x55cc0b0d1220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:12:58.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.285+0000 7fe222a00640 1 -- 172.21.15.39:0/3193491739 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc0b175810 msgr2=0x55cc0b17fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:12:58.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.285+0000 7fe222a00640 1 --2- 172.21.15.39:0/3193491739 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc0b175810 0x55cc0b17fd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:58.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.285+0000 7fe222a00640 1 -- 172.21.15.39:0/3193491739 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cc0b17b370 msgr2=0x55cc0b187280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:12:58.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.285+0000 7fe222a00640 1 --2- 172.21.15.39:0/3193491739 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cc0b17b370 0x55cc0b187280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:58.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.285+0000 7fe222a00640 1 -- 172.21.15.39:0/3193491739 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cc0adee3f0 con 0x55cc0b176200 2024-03-18T16:12:58.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.285+0000 7fe222a00640 1 --2- 172.21.15.39:0/3193491739 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cc0b176200 0x55cc0b180280 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fe2100cc090 tx=0x7fe21009b630 comp rx=0 tx=0).ready entity=mon.1 client_cookie=264514e7e9ab6d9e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:58.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.285+0000 7fe20d600640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:12:58.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.285+0000 7fe20d600640 1 -- 172.21.15.39:0/3193491739 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2100a36d0 con 0x55cc0b176200 2024-03-18T16:12:58.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.285+0000 7fe22d377740 1 -- 172.21.15.39:0/3193491739 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe2140043c0 con 0x55cc0b176200 2024-03-18T16:12:58.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.285+0000 7fe20d600640 1 -- 172.21.15.39:0/3193491739 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fe2100037a0 con 0x55cc0b176200 2024-03-18T16:12:58.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.286+0000 7fe20d600640 1 -- 172.21.15.39:0/3193491739 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fe2100a3240 con 0x55cc0b176200 2024-03-18T16:12:58.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.286+0000 7fe22d377740 10 client.26155 fetch_fsmap learned FSMap version 34 2024-03-18T16:12:58.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.286+0000 7fe22d377740 1 -- 172.21.15.39:0/3193491739 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55cc0adae0f0 con 0x55cc0b176200 2024-03-18T16:12:58.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.286+0000 7fe20d600640 1 -- 172.21.15.39:0/3193491739 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe2140043c0 con 0x55cc0b176200 2024-03-18T16:12:58.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.286+0000 7fe20d600640 1 -- 172.21.15.39:0/3193491739 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 34) ==== 8390+0+0 (secure 0 0 0) 0x7fe21000b040 con 0x55cc0b176200 2024-03-18T16:12:58.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.287+0000 7fe22d377740 10 client.26155 fetch_fsmap finished waiting for FSMap version 34 2024-03-18T16:12:58.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.287+0000 7fe22d377740 10 client.26155 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:12:58.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.287+0000 7fe22d377740 20 client.26155 populate_metadata read hostname 'smithi039' 2024-03-18T16:12:58.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.287+0000 7fe22d377740 1 --2- 172.21.15.39:0/3193491739 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55cc0b18bb60 0x55cc0b18e020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:58.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.287+0000 7fe22d377740 4 client.26155 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:12:58.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.287+0000 7fe22d377740 1 -- 172.21.15.39:0/3193491739 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cc0b0edbe0 con 0x55cc0b18bb60 2024-03-18T16:12:58.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.287+0000 7fe223400640 1 --2- 172.21.15.39:0/3193491739 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55cc0b18bb60 0x55cc0b18e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:58.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.288+0000 7fe223400640 1 --2- 172.21.15.39:0/3193491739 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55cc0b18bb60 0x55cc0b18e020 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aa4c32f0381a453f server_cookie=80d7e3eeb80d3e25 in_seq=0 out_seq=0 2024-03-18T16:12:58.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.288+0000 7fe20d600640 10 client.26155 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:12:58.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.289+0000 7fe20d600640 1 -- 172.21.15.39:0/3193491739 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55cc0b0edbe0 con 0x55cc0b18bb60 2024-03-18T16:12:58.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.289+0000 7fe20d600640 10 client.26155 handle_command_reply: tid=0 2024-03-18T16:12:58.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.290+0000 7fe22d377740 1 client.26155 shutdown 2024-03-18T16:12:58.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.290+0000 7fe22d377740 1 -- 172.21.15.39:0/3193491739 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cc0b176200 msgr2=0x55cc0b180280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:58.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.290+0000 7fe22d377740 1 --2- 172.21.15.39:0/3193491739 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cc0b176200 0x55cc0b180280 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fe2100cc090 tx=0x7fe21009b630 comp rx=0 tx=0).stop 2024-03-18T16:12:58.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.290+0000 7fe22d377740 1 -- 172.21.15.39:0/3193491739 shutdown_connections 2024-03-18T16:12:58.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.290+0000 7fe22d377740 1 --2- 172.21.15.39:0/3193491739 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55cc0b18bb60 0x55cc0b18e020 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:58.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.290+0000 7fe22d377740 1 --2- 172.21.15.39:0/3193491739 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cc0b17b370 0x55cc0b187280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:58.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.290+0000 7fe22d377740 1 --2- 172.21.15.39:0/3193491739 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cc0b176200 0x55cc0b180280 secure :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fe2100cc090 tx=0x7fe21009b630 comp rx=0 tx=0).stop 2024-03-18T16:12:58.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.290+0000 7fe22d377740 1 --2- 172.21.15.39:0/3193491739 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc0b175810 0x55cc0b17fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:58.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.290+0000 7fe20d600640 0 client.26155 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:12:58.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.290+0000 7fe22d377740 1 -- 172.21.15.39:0/3193491739 >> 172.21.15.39:0/3193491739 conn(0x55cc0b170d80 msgr2=0x55cc0b16fec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:58.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.291+0000 7fe22d377740 1 -- 172.21.15.39:0/3193491739 shutdown_connections 2024-03-18T16:12:58.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.291+0000 7fe22d377740 1 -- 172.21.15.39:0/3193491739 wait complete. 2024-03-18T16:12:58.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.291+0000 7fe22d377740 20 client.26155 trim_cache size 0 max 16384 2024-03-18T16:12:58.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.327+0000 7fe22d377740 1 Processor -- start 2024-03-18T16:12:58.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.328+0000 7fe22d377740 1 -- start start 2024-03-18T16:12:58.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.328+0000 7fe22d377740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cc0b175810 0x55cc0b17e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:58.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.328+0000 7fe22d377740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cc0b17b370 0x55cc0b187240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:58.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.328+0000 7fe22d377740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc0b16f630 0x55cc0b16fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:58.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.328+0000 7fe22d377740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fe2241b0af0 con 0x55cc0b175810 2024-03-18T16:12:58.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.328+0000 7fe22d377740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fe2241b2780 con 0x55cc0b16f630 2024-03-18T16:12:58.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.328+0000 7fe22d377740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fe2241b11c0 con 0x55cc0b17b370 2024-03-18T16:12:58.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.329+0000 7fe223e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc0b16f630 0x55cc0b16fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:58.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.329+0000 7fe223e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc0b16f630 0x55cc0b16fac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:60740/0 (socket says 172.21.15.39:60740) 2024-03-18T16:12:58.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.329+0000 7fe223400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cc0b175810 0x55cc0b17e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:58.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.329+0000 7fe222a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cc0b17b370 0x55cc0b187240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:58.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.329+0000 7fe222a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cc0b17b370 0x55cc0b187240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:37362/0 (socket says 172.21.15.39:37362) 2024-03-18T16:12:58.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.329+0000 7fe222a00640 1 -- 172.21.15.39:0/3856445658 learned_addr learned my addr 172.21.15.39:0/3856445658 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:12:58.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.329+0000 7fe223400640 1 -- 172.21.15.39:0/3856445658 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cc0b17b370 msgr2=0x55cc0b187240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:12:58.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.329+0000 7fe223400640 1 --2- 172.21.15.39:0/3856445658 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cc0b17b370 0x55cc0b187240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:58.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.329+0000 7fe223400640 1 -- 172.21.15.39:0/3856445658 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc0b16f630 msgr2=0x55cc0b16fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:12:58.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.329+0000 7fe223400640 1 --2- 172.21.15.39:0/3856445658 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc0b16f630 0x55cc0b16fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:58.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.329+0000 7fe223400640 1 -- 172.21.15.39:0/3856445658 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cc0b170bb0 con 0x55cc0b175810 2024-03-18T16:12:58.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.330+0000 7fe223400640 1 --2- 172.21.15.39:0/3856445658 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cc0b175810 0x55cc0b17e8e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe214009690 tx=0x7fe21400f660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5ed60668ef518ee server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:58.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.330+0000 7fe20fe00640 1 -- 172.21.15.39:0/3856445658 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe21403a020 con 0x55cc0b175810 2024-03-18T16:12:58.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.331+0000 7fe20fe00640 1 -- 172.21.15.39:0/3856445658 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe21401c240 con 0x55cc0b175810 2024-03-18T16:12:58.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.331+0000 7fe20fe00640 1 -- 172.21.15.39:0/3856445658 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe21402a040 con 0x55cc0b175810 2024-03-18T16:12:58.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.331+0000 7fe22d377740 1 -- 172.21.15.39:0/3856445658 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cc0b175810 msgr2=0x55cc0b17e8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:58.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.331+0000 7fe22d377740 1 --2- 172.21.15.39:0/3856445658 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cc0b175810 0x55cc0b17e8e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe214009690 tx=0x7fe21400f660 comp rx=0 tx=0).stop 2024-03-18T16:12:58.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.332+0000 7fe22d377740 1 -- 172.21.15.39:0/3856445658 shutdown_connections 2024-03-18T16:12:58.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.332+0000 7fe22d377740 1 --2- 172.21.15.39:0/3856445658 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc0b16f630 0x55cc0b16fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:58.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.332+0000 7fe22d377740 1 --2- 172.21.15.39:0/3856445658 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cc0b17b370 0x55cc0b187240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:58.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.332+0000 7fe22d377740 1 --2- 172.21.15.39:0/3856445658 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cc0b175810 0x55cc0b17e8e0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:58.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.332+0000 7fe22d377740 1 -- 172.21.15.39:0/3856445658 >> 172.21.15.39:0/3856445658 conn(0x55cc0b170d80 msgr2=0x55cc0b17fd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:58.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.332+0000 7fe22d377740 1 -- 172.21.15.39:0/3856445658 shutdown_connections 2024-03-18T16:12:58.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.332+0000 7fe22d377740 1 -- 172.21.15.39:0/3856445658 wait complete. 2024-03-18T16:12:58.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.333+0000 7fe22d377740 1 -- start start 2024-03-18T16:12:58.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.333+0000 7fe22d377740 1 Processor -- start 2024-03-18T16:12:58.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.334+0000 7fe22d377740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cc0b16f630 0x55cc0b186d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:58.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.334+0000 7fe22d377740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cc0b175810 0x55cc0b185380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:58.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.334+0000 7fe22d377740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc0b17b370 0x55cc0b1858c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:58.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.334+0000 7fe22d377740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fe2241b2020 con 0x55cc0b175810 2024-03-18T16:12:58.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.334+0000 7fe22d377740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fe2241b1890 con 0x55cc0b17b370 2024-03-18T16:12:58.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.334+0000 7fe22d377740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fe2241aead0 con 0x55cc0b16f630 2024-03-18T16:12:58.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.334+0000 7fe22d377740 10 client.0 _ll_register_callbacks cb 0x7fe2100046c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:12:58.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.335+0000 7fe223e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc0b17b370 0x55cc0b1858c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:58.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.335+0000 7fe223e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc0b17b370 0x55cc0b1858c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:60742/0 (socket says 172.21.15.39:60742) 2024-03-18T16:12:58.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.335+0000 7fe223e00640 1 -- 172.21.15.39:0/3679594016 learned_addr learned my addr 172.21.15.39:0/3679594016 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:12:58.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.335+0000 7fe222a00640 1 --2- 172.21.15.39:0/3679594016 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cc0b175810 0x55cc0b185380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:58.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.335+0000 7fe223e00640 1 -- 172.21.15.39:0/3679594016 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cc0b16f630 msgr2=0x55cc0b186d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:12:58.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.335+0000 7fe223e00640 1 --2- 172.21.15.39:0/3679594016 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cc0b16f630 0x55cc0b186d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:58.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.335+0000 7fe223e00640 1 -- 172.21.15.39:0/3679594016 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cc0b175810 msgr2=0x55cc0b185380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:12:58.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.335+0000 7fe223e00640 1 --2- 172.21.15.39:0/3679594016 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cc0b175810 0x55cc0b185380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:58.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.335+0000 7fe223e00640 1 -- 172.21.15.39:0/3679594016 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cc0b185e00 con 0x55cc0b17b370 2024-03-18T16:12:58.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.335+0000 7fe222a00640 1 --2- 172.21.15.39:0/3679594016 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cc0b175810 0x55cc0b185380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:12:58.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.336+0000 7fe223e00640 1 --2- 172.21.15.39:0/3679594016 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc0b17b370 0x55cc0b1858c0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fe21c009be0 tx=0x7fe21c033e90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3d97b733f09b5382 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:58.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.336+0000 7fe20d600640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:12:58.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.336+0000 7fe20d600640 1 -- 172.21.15.39:0/3679594016 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe21c003b80 con 0x55cc0b17b370 2024-03-18T16:12:58.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.336+0000 7fe22d377740 1 -- 172.21.15.39:0/3679594016 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe21401c240 con 0x55cc0b17b370 2024-03-18T16:12:58.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.336+0000 7fe20d600640 1 -- 172.21.15.39:0/3679594016 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fe21c03a040 con 0x55cc0b17b370 2024-03-18T16:12:58.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.336+0000 7fe20d600640 1 -- 172.21.15.39:0/3679594016 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) ==== 24+0+0 (secure 0 0 0) 0x7fe21c003d20 con 0x55cc0b17b370 2024-03-18T16:12:58.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.337+0000 7fe22d377740 10 client.26150 fetch_fsmap learned FSMap version 34 2024-03-18T16:12:58.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.337+0000 7fe22d377740 1 -- 172.21.15.39:0/3679594016 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) -- 0x55cc0b1f2cb0 con 0x55cc0b17b370 2024-03-18T16:12:58.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.337+0000 7fe20d600640 1 -- 172.21.15.39:0/3679594016 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe21401c240 con 0x55cc0b17b370 2024-03-18T16:12:58.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.339+0000 7fe20d600640 1 -- 172.21.15.39:0/3679594016 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 34) ==== 8390+0+0 (secure 0 0 0) 0x7fe21c004200 con 0x55cc0b17b370 2024-03-18T16:12:58.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.339+0000 7fe22d377740 10 client.26150 fetch_fsmap finished waiting for FSMap version 34 2024-03-18T16:12:58.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.339+0000 7fe22d377740 10 client.26150 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:12:58.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.339+0000 7fe22d377740 20 client.26150 populate_metadata read hostname 'smithi039' 2024-03-18T16:12:58.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.339+0000 7fe22d377740 1 --2- 172.21.15.39:0/3679594016 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55cc0b1f3660 0x55cc0b1f5a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:58.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.339+0000 7fe22d377740 4 client.26150 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T16:12:58.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.339+0000 7fe22d377740 1 -- 172.21.15.39:0/3679594016 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55cc0b0e7730 con 0x55cc0b1f3660 2024-03-18T16:12:58.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.340+0000 7fe223400640 1 --2- 172.21.15.39:0/3679594016 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55cc0b1f3660 0x55cc0b1f5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:58.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.341+0000 7fe223400640 1 --2- 172.21.15.39:0/3679594016 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55cc0b1f3660 0x55cc0b1f5a80 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9896df9dd00ce5bc server_cookie=d77f9db6189b2625 in_seq=0 out_seq=0 2024-03-18T16:12:58.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.341+0000 7fe20d600640 10 client.26150 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:12:58.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.342+0000 7fe20d600640 1 -- 172.21.15.39:0/3679594016 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55cc0b0e7730 con 0x55cc0b1f3660 2024-03-18T16:12:58.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.342+0000 7fe20d600640 10 client.26150 handle_command_reply: tid=0 2024-03-18T16:12:58.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.342+0000 7fe22d377740 1 client.26150 shutdown 2024-03-18T16:12:58.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.342+0000 7fe22d377740 1 -- 172.21.15.39:0/3679594016 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc0b17b370 msgr2=0x55cc0b1858c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:58.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.342+0000 7fe22d377740 1 --2- 172.21.15.39:0/3679594016 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc0b17b370 0x55cc0b1858c0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fe21c009be0 tx=0x7fe21c033e90 comp rx=0 tx=0).stop 2024-03-18T16:12:58.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.342+0000 7fe22d377740 1 -- 172.21.15.39:0/3679594016 shutdown_connections 2024-03-18T16:12:58.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.342+0000 7fe22d377740 1 --2- 172.21.15.39:0/3679594016 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55cc0b1f3660 0x55cc0b1f5a80 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:58.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.342+0000 7fe22d377740 1 --2- 172.21.15.39:0/3679594016 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc0b17b370 0x55cc0b1858c0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:58.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.342+0000 7fe22d377740 1 --2- 172.21.15.39:0/3679594016 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cc0b175810 0x55cc0b185380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:58.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.342+0000 7fe22d377740 1 --2- 172.21.15.39:0/3679594016 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cc0b16f630 0x55cc0b186d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:58.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.342+0000 7fe20d600640 0 client.26150 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:12:58.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.342+0000 7fe22d377740 1 -- 172.21.15.39:0/3679594016 >> 172.21.15.39:0/3679594016 conn(0x55cc0b170d80 msgr2=0x55cc0b17fce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:58.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.342+0000 7fe22d377740 1 -- 172.21.15.39:0/3679594016 shutdown_connections 2024-03-18T16:12:58.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.343+0000 7fe22d377740 1 -- 172.21.15.39:0/3679594016 wait complete. 2024-03-18T16:12:58.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.343+0000 7fe22d377740 20 client.26150 trim_cache size 0 max 16384 2024-03-18T16:12:58.351 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-03-18T16:12:58.351 INFO:teuthology.orchestra.run.smithi039.stdout: "status": "no active scrubs running", 2024-03-18T16:12:58.351 INFO:teuthology.orchestra.run.smithi039.stdout: "scrubs": {} 2024-03-18T16:12:58.351 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-03-18T16:12:58.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.343+0000 7fe22a600640 1 -- 172.21.15.39:0/1942797328 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fe2080a28d0 msgr2=0x7fe2080a4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:58.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.343+0000 7fe22a600640 1 --2- 172.21.15.39:0/1942797328 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fe2080a28d0 0x7fe2080a4d90 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fe2140096c0 tx=0x7fe214009260 comp rx=0 tx=0).stop 2024-03-18T16:12:58.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.343+0000 7fe22a600640 1 -- 172.21.15.39:0/1942797328 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe224100fb0 msgr2=0x7fe22419e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:58.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.343+0000 7fe22a600640 1 --2- 172.21.15.39:0/1942797328 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe224100fb0 0x7fe22419e5e0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fe21000e9f0 tx=0x7fe21000eeb0 comp rx=0 tx=0).stop 2024-03-18T16:12:58.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.344+0000 7fe22a600640 1 -- 172.21.15.39:0/1942797328 shutdown_connections 2024-03-18T16:12:58.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.344+0000 7fe22a600640 1 --2- 172.21.15.39:0/1942797328 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe224101970 0x7fe2241a5650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:58.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.344+0000 7fe22a600640 1 --2- 172.21.15.39:0/1942797328 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fe2080a28d0 0x7fe2080a4d90 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:58.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.344+0000 7fe22a600640 1 --2- 172.21.15.39:0/1942797328 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe224100fb0 0x7fe22419e5e0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:58.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.344+0000 7fe22a600640 1 --2- 172.21.15.39:0/1942797328 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe2240ffe60 0x7fe22419e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:58.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.344+0000 7fe22a600640 1 -- 172.21.15.39:0/1942797328 >> 172.21.15.39:0/1942797328 conn(0x7fe2240fb3d0 msgr2=0x7fe22410dc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:58.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.344+0000 7fe22a600640 1 -- 172.21.15.39:0/1942797328 shutdown_connections 2024-03-18T16:12:58.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.344+0000 7fe22a600640 1 -- 172.21.15.39:0/1942797328 wait complete. 2024-03-18T16:12:58.368 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-18T16:12:58.368 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:12:58.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.696+0000 7ffb0b000640 1 -- 172.21.15.39:0/1515732432 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffb0410de80 msgr2=0x7ffb04110340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:58.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.696+0000 7ffb0b000640 1 --2- 172.21.15.39:0/1515732432 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffb0410de80 0x7ffb04110340 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7ffafc009be0 tx=0x7ffafc02bef0 comp rx=0 tx=0).stop 2024-03-18T16:12:58.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.697+0000 7ffb0b000640 1 -- 172.21.15.39:0/1515732432 shutdown_connections 2024-03-18T16:12:58.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.697+0000 7ffb0b000640 1 --2- 172.21.15.39:0/1515732432 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffb0410b380 0x7ffb0410d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:58.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.697+0000 7ffb0b000640 1 --2- 172.21.15.39:0/1515732432 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffb0410de80 0x7ffb04110340 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:58.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.697+0000 7ffb0b000640 1 --2- 172.21.15.39:0/1515732432 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffb041089c0 0x7ffb0410adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:58.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.697+0000 7ffb0b000640 1 -- 172.21.15.39:0/1515732432 >> 172.21.15.39:0/1515732432 conn(0x7ffb0406e640 msgr2=0x7ffb04070a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:58.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.697+0000 7ffb0b000640 1 -- 172.21.15.39:0/1515732432 shutdown_connections 2024-03-18T16:12:58.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.697+0000 7ffb0b000640 1 -- 172.21.15.39:0/1515732432 wait complete. 2024-03-18T16:12:58.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.699+0000 7ffb0b000640 1 Processor -- start 2024-03-18T16:12:58.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.699+0000 7ffb0b000640 1 -- start start 2024-03-18T16:12:58.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.700+0000 7ffb0b000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffb041089c0 0x7ffb0419e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:58.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.700+0000 7ffb0b000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffb0410b380 0x7ffb0419e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:58.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.700+0000 7ffb0b000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffb0410de80 0x7ffb041adf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:58.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.700+0000 7ffb0b000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7ffb04115c70 con 0x7ffb0410b380 2024-03-18T16:12:58.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.700+0000 7ffb0b000640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ffb04115af0 con 0x7ffb041089c0 2024-03-18T16:12:58.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.700+0000 7ffb0b000640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7ffb04115df0 con 0x7ffb0410de80 2024-03-18T16:12:58.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.700+0000 7ffb09000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffb0410de80 0x7ffb041adf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:58.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.700+0000 7ffb09000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffb0410de80 0x7ffb041adf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:37388/0 (socket says 172.21.15.39:37388) 2024-03-18T16:12:58.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.701+0000 7ffb03e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffb041089c0 0x7ffb0419e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:58.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.701+0000 7ffb09000640 1 -- 172.21.15.39:0/2333816666 learned_addr learned my addr 172.21.15.39:0/2333816666 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:12:58.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.701+0000 7ffb09000640 1 -- 172.21.15.39:0/2333816666 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffb041089c0 msgr2=0x7ffb0419e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:58.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.701+0000 7ffb09000640 1 --2- 172.21.15.39:0/2333816666 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffb041089c0 0x7ffb0419e0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:58.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.701+0000 7ffb09000640 1 -- 172.21.15.39:0/2333816666 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffb0410b380 msgr2=0x7ffb0419e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:58.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.701+0000 7ffb03400640 1 --2- 172.21.15.39:0/2333816666 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffb0410b380 0x7ffb0419e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:58.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.701+0000 7ffb09000640 1 --2- 172.21.15.39:0/2333816666 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffb0410b380 0x7ffb0419e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:58.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.701+0000 7ffb09000640 1 -- 172.21.15.39:0/2333816666 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb041ae4f0 con 0x7ffb0410de80 2024-03-18T16:12:58.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.701+0000 7ffb03400640 1 --2- 172.21.15.39:0/2333816666 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffb0410b380 0x7ffb0419e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:12:58.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.702+0000 7ffb09000640 1 --2- 172.21.15.39:0/2333816666 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffb0410de80 0x7ffb041adf80 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7ffafc036040 tx=0x7ffafc0337b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:58.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.702+0000 7ffb00c00640 1 -- 172.21.15.39:0/2333816666 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffafc043020 con 0x7ffb0410de80 2024-03-18T16:12:58.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.702+0000 7ffb00c00640 1 -- 172.21.15.39:0/2333816666 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffafc02ca60 con 0x7ffb0410de80 2024-03-18T16:12:58.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.702+0000 7ffb00c00640 1 -- 172.21.15.39:0/2333816666 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffafc034690 con 0x7ffb0410de80 2024-03-18T16:12:58.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.703+0000 7ffb0b000640 1 -- 172.21.15.39:0/2333816666 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb041ae780 con 0x7ffb0410de80 2024-03-18T16:12:58.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.703+0000 7ffb0b000640 1 -- 172.21.15.39:0/2333816666 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb041aebc0 con 0x7ffb0410de80 2024-03-18T16:12:58.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.705+0000 7ffb0b000640 1 -- 172.21.15.39:0/2333816666 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb041aee80 con 0x7ffb0410de80 2024-03-18T16:12:58.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.707+0000 7ffb00c00640 1 -- 172.21.15.39:0/2333816666 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7ffafc046020 con 0x7ffb0410de80 2024-03-18T16:12:58.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.711+0000 7ffb00c00640 1 --2- 172.21.15.39:0/2333816666 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ffae80a2950 0x7ffae80a4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:58.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.711+0000 7ffb00c00640 1 -- 172.21.15.39:0/2333816666 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7ffafc0f12b0 con 0x7ffb0410de80 2024-03-18T16:12:58.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.712+0000 7ffb03e00640 1 --2- 172.21.15.39:0/2333816666 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ffae80a2950 0x7ffae80a4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:58.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.714+0000 7ffb03e00640 1 --2- 172.21.15.39:0/2333816666 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ffae80a2950 0x7ffae80a4e10 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7ffb04073910 tx=0x7ffaf000c040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:58.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:58.717+0000 7ffb00c00640 1 -- 172.21.15.39:0/2333816666 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ffafc04d070 con 0x7ffb0410de80 2024-03-18T16:12:58.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:12:58 smithi196 ceph-mon[36307]: scrub summary: idle 2024-03-18T16:12:58.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:12:58 smithi196 ceph-mon[36307]: scrub summary: idle 2024-03-18T16:12:58.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:12:58 smithi196 ceph-mon[36307]: scrub summary: idle 2024-03-18T16:12:58.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:12:58 smithi196 ceph-mon[36307]: scrub summary: idle 2024-03-18T16:12:58.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:12:58 smithi196 ceph-mon[36307]: scrub summary: idle 2024-03-18T16:12:58.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:12:58 smithi194 ceph-mon[36200]: scrub summary: idle 2024-03-18T16:12:58.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:12:58 smithi194 ceph-mon[36200]: scrub summary: idle 2024-03-18T16:12:58.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:12:58 smithi194 ceph-mon[36200]: scrub summary: idle 2024-03-18T16:12:58.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:12:58 smithi194 ceph-mon[36200]: scrub summary: idle 2024-03-18T16:12:58.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:12:58 smithi194 ceph-mon[36200]: scrub summary: idle 2024-03-18T16:12:58.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:12:58 smithi039 ceph-mon[31721]: scrub summary: idle 2024-03-18T16:12:58.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:12:58 smithi039 ceph-mon[31721]: scrub summary: idle 2024-03-18T16:12:58.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:12:58 smithi039 ceph-mon[31721]: scrub summary: idle 2024-03-18T16:12:58.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:12:58 smithi039 ceph-mon[31721]: scrub summary: idle 2024-03-18T16:12:58.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:12:58 smithi039 ceph-mon[31721]: scrub summary: idle 2024-03-18T16:12:59.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.304+0000 7ffb0b000640 1 -- 172.21.15.39:0/2333816666 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffb041af210 con 0x7ffb0410de80 2024-03-18T16:12:59.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.306+0000 7ffb00c00640 1 -- 172.21.15.39:0/2333816666 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) ==== 94+0+11949 (secure 0 0 0) 0x7ffafc0bdf60 con 0x7ffb0410de80 2024-03-18T16:12:59.309 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:12:59.309 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2520226608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2520226608},{"type":"v1","addr":"172.21.15.39:6835","nonce":2520226608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/719341763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":719341763},{"type":"v1","addr":"172.21.15.39:6837","nonce":719341763}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":35,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T16:06:44.180425+0000","modified":"2024-03-18T16:12:58.267182+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24575,"mds_1":24556,"mds_2":24535,"mds_3":24524,"mds_4":14463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14445":{"gid":14445,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.39:6839/656513813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":656513813},{"type":"v1","addr":"172.21.15.39:6839","nonce":656513813}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14463":{"gid":14463,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":13,"addr":"172.21.15.39:6841/3515326042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3515326042},{"type":"v1","addr":"172.21.15.39:6841","nonce":3515326042}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6833/931350738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":931350738},{"type":"v1","addr":"172.21.15.194:6833","nonce":931350738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6833/3874209621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":3874209621},{"type":"v1","addr":"172.21.15.196:6833","nonce":3874209621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6835/188385470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":188385470},{"type":"v1","addr":"172.21.15.194:6835","nonce":188385470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6835/3949984309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":3949984309},{"type":"v1","addr":"172.21.15.196:6835","nonce":3949984309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":15,"addr":"172.21.15.194:6837/1301464246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":1301464246},{"type":"v1","addr":"172.21.15.194:6837","nonce":1301464246}]},"join_fscid":-1,"export_targets":[1,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":16,"addr":"172.21.15.196:6837/895247251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":895247251},{"type":"v1","addr":"172.21.15.196:6837","nonce":895247251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.196:6839/2001427268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6838","nonce":2001427268},{"type":"v1","addr":"172.21.15.196:6839","nonce":2001427268}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24575":{"gid":24575,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.194:6839/274911158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":274911158},{"type":"v1","addr":"172.21.15.194:6839","nonce":274911158}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24575,"qdb_cluster":[24556,24535,24575,24524,14463]},"id":1}]} 2024-03-18T16:12:59.309 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 35 2024-03-18T16:12:59.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.313+0000 7ffb0b000640 1 -- 172.21.15.39:0/2333816666 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ffae80a2950 msgr2=0x7ffae80a4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:59.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.313+0000 7ffb0b000640 1 --2- 172.21.15.39:0/2333816666 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ffae80a2950 0x7ffae80a4e10 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7ffb04073910 tx=0x7ffaf000c040 comp rx=0 tx=0).stop 2024-03-18T16:12:59.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.313+0000 7ffb0b000640 1 -- 172.21.15.39:0/2333816666 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffb0410de80 msgr2=0x7ffb041adf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:59.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.313+0000 7ffb0b000640 1 --2- 172.21.15.39:0/2333816666 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffb0410de80 0x7ffb041adf80 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7ffafc036040 tx=0x7ffafc0337b0 comp rx=0 tx=0).stop 2024-03-18T16:12:59.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.314+0000 7ffb0b000640 1 -- 172.21.15.39:0/2333816666 shutdown_connections 2024-03-18T16:12:59.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.314+0000 7ffb0b000640 1 --2- 172.21.15.39:0/2333816666 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ffae80a2950 0x7ffae80a4e10 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:59.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.314+0000 7ffb0b000640 1 --2- 172.21.15.39:0/2333816666 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffb0410de80 0x7ffb041adf80 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:59.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.314+0000 7ffb0b000640 1 --2- 172.21.15.39:0/2333816666 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ffb0410b380 0x7ffb0419e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:59.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.314+0000 7ffb0b000640 1 --2- 172.21.15.39:0/2333816666 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffb041089c0 0x7ffb0419e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:59.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.314+0000 7ffb0b000640 1 -- 172.21.15.39:0/2333816666 >> 172.21.15.39:0/2333816666 conn(0x7ffb0406e640 msgr2=0x7ffb04109f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:59.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.315+0000 7ffb0b000640 1 -- 172.21.15.39:0/2333816666 shutdown_connections 2024-03-18T16:12:59.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.315+0000 7ffb0b000640 1 -- 172.21.15.39:0/2333816666 wait complete. 2024-03-18T16:12:59.357 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-18T16:12:59.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.710+0000 7fe6a1800640 1 -- 172.21.15.39:0/3299331814 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe6940a6710 msgr2=0x7fe6940a6fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:59.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.710+0000 7fe6a1800640 1 --2- 172.21.15.39:0/3299331814 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe6940a6710 0x7fe6940a6fb0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fe688009900 tx=0x7fe68802bc40 comp rx=0 tx=0).stop 2024-03-18T16:12:59.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.710+0000 7fe6a1800640 1 -- 172.21.15.39:0/3299331814 shutdown_connections 2024-03-18T16:12:59.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.710+0000 7fe6a1800640 1 --2- 172.21.15.39:0/3299331814 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe6940a6710 0x7fe6940a6fb0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:59.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.710+0000 7fe6a1800640 1 --2- 172.21.15.39:0/3299331814 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe6940a5d70 0x7fe6940a61d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:59.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.710+0000 7fe6a1800640 1 --2- 172.21.15.39:0/3299331814 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe6940a4c40 0x7fe6940a5060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:59.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.710+0000 7fe6a1800640 1 -- 172.21.15.39:0/3299331814 >> 172.21.15.39:0/3299331814 conn(0x7fe6940a00d0 msgr2=0x7fe6940a2530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:59.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.713+0000 7fe6a1800640 1 -- 172.21.15.39:0/3299331814 shutdown_connections 2024-03-18T16:12:59.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.713+0000 7fe6a1800640 1 -- 172.21.15.39:0/3299331814 wait complete. 2024-03-18T16:12:59.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.714+0000 7fe6a1800640 1 Processor -- start 2024-03-18T16:12:59.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.715+0000 7fe6a1800640 1 -- start start 2024-03-18T16:12:59.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.715+0000 7fe6a1800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe6940a4c40 0x7fe6940abd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:59.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.715+0000 7fe6a1800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe6940a5d70 0x7fe6940ac2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:59.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.715+0000 7fe6a1800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe6940a6710 0x7fe6940a88e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:59.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.716+0000 7fe69be00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe6940a4c40 0x7fe6940abd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:59.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.716+0000 7fe69b400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe6940a5d70 0x7fe6940ac2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:59.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.716+0000 7fe69be00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe6940a4c40 0x7fe6940abd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:60798/0 (socket says 172.21.15.39:60798) 2024-03-18T16:12:59.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.716+0000 7fe69b400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe6940a5d70 0x7fe6940ac2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:37414/0 (socket says 172.21.15.39:37414) 2024-03-18T16:12:59.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.716+0000 7fe69be00640 1 -- 172.21.15.39:0/3843437861 learned_addr learned my addr 172.21.15.39:0/3843437861 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:12:59.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.716+0000 7fe6a1800640 1 -- 172.21.15.39:0/3843437861 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fe6940b4cc0 con 0x7fe6940a6710 2024-03-18T16:12:59.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.716+0000 7fe6a0e00640 1 --2- 172.21.15.39:0/3843437861 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe6940a6710 0x7fe6940a88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:59.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.716+0000 7fe6a1800640 1 -- 172.21.15.39:0/3843437861 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fe6940b4b40 con 0x7fe6940a4c40 2024-03-18T16:12:59.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.716+0000 7fe6a1800640 1 -- 172.21.15.39:0/3843437861 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fe6940b4e40 con 0x7fe6940a5d70 2024-03-18T16:12:59.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.717+0000 7fe6a0e00640 1 -- 172.21.15.39:0/3843437861 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe6940a5d70 msgr2=0x7fe6940ac2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:59.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.717+0000 7fe6a0e00640 1 --2- 172.21.15.39:0/3843437861 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe6940a5d70 0x7fe6940ac2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:59.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.717+0000 7fe6a0e00640 1 -- 172.21.15.39:0/3843437861 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe6940a4c40 msgr2=0x7fe6940abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:59.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.717+0000 7fe6a0e00640 1 --2- 172.21.15.39:0/3843437861 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe6940a4c40 0x7fe6940abd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:59.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.717+0000 7fe6a0e00640 1 -- 172.21.15.39:0/3843437861 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6940a91a0 con 0x7fe6940a6710 2024-03-18T16:12:59.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.717+0000 7fe69b400640 1 --2- 172.21.15.39:0/3843437861 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe6940a5d70 0x7fe6940ac2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:12:59.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.717+0000 7fe6a0e00640 1 --2- 172.21.15.39:0/3843437861 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe6940a6710 0x7fe6940a88e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fe688009230 tx=0x7fe6880045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:59.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.718+0000 7fe698c00640 1 -- 172.21.15.39:0/3843437861 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe688043020 con 0x7fe6940a6710 2024-03-18T16:12:59.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.718+0000 7fe6a1800640 1 -- 172.21.15.39:0/3843437861 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6940a9430 con 0x7fe6940a6710 2024-03-18T16:12:59.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.718+0000 7fe6a1800640 1 -- 172.21.15.39:0/3843437861 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe69414bac0 con 0x7fe6940a6710 2024-03-18T16:12:59.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.718+0000 7fe698c00640 1 -- 172.21.15.39:0/3843437861 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe68802c990 con 0x7fe6940a6710 2024-03-18T16:12:59.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.718+0000 7fe698c00640 1 -- 172.21.15.39:0/3843437861 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe68803e450 con 0x7fe6940a6710 2024-03-18T16:12:59.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:12:59 smithi196 ceph-mon[36307]: pgmap v509: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 58 MiB/s wr, 8.31k op/s 2024-03-18T16:12:59.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:12:59 smithi196 ceph-mon[36307]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T16:12:59.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:12:59 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:12:59.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:12:59 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/2333816666' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:12:59.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.724+0000 7fe698c00640 1 -- 172.21.15.39:0/3843437861 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fe688046020 con 0x7fe6940a6710 2024-03-18T16:12:59.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.727+0000 7fe6a5a50740 1 Processor -- start 2024-03-18T16:12:59.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.727+0000 7fe6a5a50740 1 -- start start 2024-03-18T16:12:59.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.727+0000 7fe698c00640 1 --2- 172.21.15.39:0/3843437861 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fe6840a2900 0x7fe6840a4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:59.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.727+0000 7fe698c00640 1 -- 172.21.15.39:0/3843437861 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fe6880f0c60 con 0x7fe6940a6710 2024-03-18T16:12:59.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.727+0000 7fe6a5a50740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5578d8fbea60 0x5578d8fbee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:59.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.727+0000 7fe6a5a50740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578d8fbf450 0x5578d8fc3f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:59.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.727+0000 7fe6a5a50740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5578d8fc45c0 0x5578d8fc6a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:59.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.727+0000 7fe6a5a50740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5578d8f863b0 con 0x5578d8fc45c0 2024-03-18T16:12:59.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.728+0000 7fe6a5a50740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5578d8f23450 con 0x5578d8fbea60 2024-03-18T16:12:59.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.728+0000 7fe6a5a50740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x5578d8ecbcb0 con 0x5578d8fbf450 2024-03-18T16:12:59.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.728+0000 7fe69b400640 1 --2- 172.21.15.39:0/3843437861 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fe6840a2900 0x7fe6840a4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:59.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.728+0000 7fe69be00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5578d8fbea60 0x5578d8fbee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:59.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.728+0000 7fe69be00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5578d8fbea60 0x5578d8fbee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:60814/0 (socket says 172.21.15.39:60814) 2024-03-18T16:12:59.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.728+0000 7fe69be00640 1 -- 172.21.15.39:0/2335132726 learned_addr learned my addr 172.21.15.39:0/2335132726 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:12:59.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.729+0000 7fe69b400640 1 --2- 172.21.15.39:0/2335132726 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578d8fbf450 0x5578d8fc3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:59.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.729+0000 7fe69be00640 1 -- 172.21.15.39:0/2335132726 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578d8fbf450 msgr2=0x5578d8fc3f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:12:59.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.729+0000 7fe69be00640 1 --2- 172.21.15.39:0/2335132726 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578d8fbf450 0x5578d8fc3f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:59.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.730+0000 7fe69be00640 1 -- 172.21.15.39:0/2335132726 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5578d8fc45c0 msgr2=0x5578d8fc6a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:12:59.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.730+0000 7fe69be00640 1 --2- 172.21.15.39:0/2335132726 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5578d8fc45c0 0x5578d8fc6a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:59.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.730+0000 7fe69be00640 1 -- 172.21.15.39:0/2335132726 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5578d8eed1f0 con 0x5578d8fbea60 2024-03-18T16:12:59.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.731+0000 7fe69b400640 1 --2- 172.21.15.39:0/3843437861 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fe6840a2900 0x7fe6840a4dc0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fe6900058c0 tx=0x7fe690013590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:59.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.731+0000 7fe69be00640 1 --2- 172.21.15.39:0/2335132726 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5578d8fbea60 0x5578d8fbee80 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fe69c0670a0 tx=0x7fe69c073be0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e42cadb4d1e4cd3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:59.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.732+0000 7fe68d600640 1 -- 172.21.15.39:0/2335132726 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe69c071af0 con 0x5578d8fbea60 2024-03-18T16:12:59.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.732+0000 7fe68d600640 1 -- 172.21.15.39:0/2335132726 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe69c078030 con 0x5578d8fbea60 2024-03-18T16:12:59.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.732+0000 7fe68d600640 1 -- 172.21.15.39:0/2335132726 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe69c074690 con 0x5578d8fbea60 2024-03-18T16:12:59.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.733+0000 7fe6a5a50740 1 -- 172.21.15.39:0/2335132726 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5578d8fbea60 msgr2=0x5578d8fbee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:59.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.733+0000 7fe6a5a50740 1 --2- 172.21.15.39:0/2335132726 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5578d8fbea60 0x5578d8fbee80 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fe69c0670a0 tx=0x7fe69c073be0 comp rx=0 tx=0).stop 2024-03-18T16:12:59.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.733+0000 7fe6a5a50740 1 -- 172.21.15.39:0/2335132726 shutdown_connections 2024-03-18T16:12:59.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.733+0000 7fe6a5a50740 1 --2- 172.21.15.39:0/2335132726 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5578d8fc45c0 0x5578d8fc6a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:59.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.733+0000 7fe6a5a50740 1 --2- 172.21.15.39:0/2335132726 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578d8fbf450 0x5578d8fc3f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:59.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.733+0000 7fe6a5a50740 1 --2- 172.21.15.39:0/2335132726 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5578d8fbea60 0x5578d8fbee80 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:59.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.733+0000 7fe6a5a50740 1 -- 172.21.15.39:0/2335132726 >> 172.21.15.39:0/2335132726 conn(0x5578d8fb9fd0 msgr2=0x5578d8fba3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:59.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.734+0000 7fe6a5a50740 1 -- 172.21.15.39:0/2335132726 shutdown_connections 2024-03-18T16:12:59.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.734+0000 7fe6a5a50740 1 -- 172.21.15.39:0/2335132726 wait complete. 2024-03-18T16:12:59.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.734+0000 7fe6a5a50740 1 -- start start 2024-03-18T16:12:59.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.735+0000 7fe6a5a50740 1 Processor -- start 2024-03-18T16:12:59.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.737+0000 7fe6a5a50740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5578d8fbea60 0x5578d8fc8d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:59.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.738+0000 7fe6a5a50740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5578d8fbf450 0x5578d8fc92d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:59.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:12:59 smithi194 ceph-mon[36200]: pgmap v509: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 58 MiB/s wr, 8.31k op/s 2024-03-18T16:12:59.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:12:59 smithi194 ceph-mon[36200]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T16:12:59.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:12:59 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:12:59.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:12:59 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/2333816666' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:12:59.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.738+0000 7fe6a5a50740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578d8fc45c0 0x5578d8fd0310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:59.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.738+0000 7fe6a5a50740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5578d8edf110 con 0x5578d8fbea60 2024-03-18T16:12:59.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.738+0000 7fe6a5a50740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5578d8f1f4f0 con 0x5578d8fbf450 2024-03-18T16:12:59.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.738+0000 7fe6a5a50740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x5578d8fc7570 con 0x5578d8fc45c0 2024-03-18T16:12:59.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.738+0000 7fe6a5a50740 10 client.0 _ll_register_callbacks cb 0x5578d8eccb70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:12:59.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.738+0000 7fe6a0e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578d8fc45c0 0x5578d8fd0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:59.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.738+0000 7fe6a0e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578d8fc45c0 0x5578d8fd0310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:37430/0 (socket says 172.21.15.39:37430) 2024-03-18T16:12:59.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.738+0000 7fe6a0e00640 1 -- 172.21.15.39:0/518280197 learned_addr learned my addr 172.21.15.39:0/518280197 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:12:59.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.738+0000 7fe69b400640 1 --2- 172.21.15.39:0/518280197 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5578d8fbf450 0x5578d8fc92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:59.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.738+0000 7fe69be00640 1 --2- 172.21.15.39:0/518280197 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5578d8fbea60 0x5578d8fc8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:59.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.739+0000 7fe6a0e00640 1 -- 172.21.15.39:0/518280197 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5578d8fbf450 msgr2=0x5578d8fc92d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:12:59.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.739+0000 7fe6a0e00640 1 --2- 172.21.15.39:0/518280197 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5578d8fbf450 0x5578d8fc92d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:59.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.739+0000 7fe6a0e00640 1 -- 172.21.15.39:0/518280197 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5578d8fbea60 msgr2=0x5578d8fc8d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:12:59.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.739+0000 7fe6a0e00640 1 --2- 172.21.15.39:0/518280197 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5578d8fbea60 0x5578d8fc8d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:59.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.739+0000 7fe6a0e00640 1 -- 172.21.15.39:0/518280197 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5578d8c373f0 con 0x5578d8fc45c0 2024-03-18T16:12:59.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.740+0000 7fe6a0e00640 1 --2- 172.21.15.39:0/518280197 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578d8fc45c0 0x5578d8fd0310 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fe6880b9c70 tx=0x7fe6880bccc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f5835d9db0e4b7a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:59.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.740+0000 7fe682000640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:12:59.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.741+0000 7fe682000640 1 -- 172.21.15.39:0/518280197 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6880c13f0 con 0x5578d8fc45c0 2024-03-18T16:12:59.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.741+0000 7fe682000640 1 -- 172.21.15.39:0/518280197 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fe6880dd0b0 con 0x5578d8fc45c0 2024-03-18T16:12:59.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.741+0000 7fe6a5a50740 1 -- 172.21.15.39:0/518280197 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe69c078030 con 0x5578d8fc45c0 2024-03-18T16:12:59.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.742+0000 7fe682000640 1 -- 172.21.15.39:0/518280197 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fe6880c59b0 con 0x5578d8fc45c0 2024-03-18T16:12:59.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.742+0000 7fe6a5a50740 10 client.26173 fetch_fsmap learned FSMap version 35 2024-03-18T16:12:59.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.742+0000 7fe6a5a50740 1 -- 172.21.15.39:0/518280197 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5578d8d017d0 con 0x5578d8fc45c0 2024-03-18T16:12:59.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.742+0000 7fe682000640 1 -- 172.21.15.39:0/518280197 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe69c078030 con 0x5578d8fc45c0 2024-03-18T16:12:59.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.742+0000 7fe682000640 1 -- 172.21.15.39:0/518280197 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 35) ==== 8402+0+0 (secure 0 0 0) 0x7fe6880bc6f0 con 0x5578d8fc45c0 2024-03-18T16:12:59.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.743+0000 7fe6a5a50740 10 client.26173 fetch_fsmap finished waiting for FSMap version 35 2024-03-18T16:12:59.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.743+0000 7fe6a5a50740 10 client.26173 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-03-18T16:12:59.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.743+0000 7fe6a5a50740 20 client.26173 populate_metadata read hostname 'smithi039' 2024-03-18T16:12:59.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.743+0000 7fe6a5a50740 1 --2- 172.21.15.39:0/518280197 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x5578d8fd4ba0 0x5578d8fd7000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:59.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.743+0000 7fe6a5a50740 4 client.26173 mds_command: new command op to 14463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:12:59.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.743+0000 7fe6a5a50740 1 -- 172.21.15.39:0/518280197 --> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5578d8f30990 con 0x5578d8fd4ba0 2024-03-18T16:12:59.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.745+0000 7fe69be00640 1 --2- 172.21.15.39:0/518280197 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x5578d8fd4ba0 0x5578d8fd7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:59.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.751+0000 7fe682000640 10 client.26173 ms_handle_connect on v2:172.21.15.39:6840/3515326042 2024-03-18T16:12:59.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.751+0000 7fe69be00640 1 --2- 172.21.15.39:0/518280197 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x5578d8fd4ba0 0x5578d8fd7000 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=663ad145a618bfc server_cookie=bb31f247ab6dae05 in_seq=0 out_seq=0 2024-03-18T16:12:59.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.753+0000 7fe682000640 1 -- 172.21.15.39:0/518280197 <== mds.4 v2:172.21.15.39:6840/3515326042 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5578d8f30990 con 0x5578d8fd4ba0 2024-03-18T16:12:59.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.753+0000 7fe682000640 10 client.26173 handle_command_reply: tid=0 2024-03-18T16:12:59.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.754+0000 7fe6a5a50740 1 client.26173 shutdown 2024-03-18T16:12:59.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.759+0000 7fe6a5a50740 1 -- 172.21.15.39:0/518280197 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578d8fc45c0 msgr2=0x5578d8fd0310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:59.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.759+0000 7fe6a5a50740 1 --2- 172.21.15.39:0/518280197 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578d8fc45c0 0x5578d8fd0310 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fe6880b9c70 tx=0x7fe6880bccc0 comp rx=0 tx=0).stop 2024-03-18T16:12:59.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.760+0000 7fe6a5a50740 1 -- 172.21.15.39:0/518280197 shutdown_connections 2024-03-18T16:12:59.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.760+0000 7fe6a5a50740 1 --2- 172.21.15.39:0/518280197 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x5578d8fd4ba0 0x5578d8fd7000 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:59.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.760+0000 7fe6a5a50740 1 --2- 172.21.15.39:0/518280197 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5578d8fbf450 0x5578d8fc92d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:59.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.760+0000 7fe6a5a50740 1 --2- 172.21.15.39:0/518280197 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578d8fc45c0 0x5578d8fd0310 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:59.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.760+0000 7fe6a5a50740 1 --2- 172.21.15.39:0/518280197 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5578d8fbea60 0x5578d8fc8d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:59.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.760+0000 7fe6a5a50740 1 -- 172.21.15.39:0/518280197 >> 172.21.15.39:0/518280197 conn(0x5578d8fb9fd0 msgr2=0x5578d8fb91f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:59.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.760+0000 7fe682000640 0 client.26173 ms_handle_reset on v2:172.21.15.39:6840/3515326042 2024-03-18T16:12:59.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.763+0000 7fe6a5a50740 1 -- 172.21.15.39:0/518280197 shutdown_connections 2024-03-18T16:12:59.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.764+0000 7fe6a5a50740 1 -- 172.21.15.39:0/518280197 wait complete. 2024-03-18T16:12:59.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.765+0000 7fe6a5a50740 20 client.26173 trim_cache size 0 max 16384 2024-03-18T16:12:59.797 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.794+0000 7fe6a5a50740 1 Processor -- start 2024-03-18T16:12:59.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.795+0000 7fe6a5a50740 1 -- start start 2024-03-18T16:12:59.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.796+0000 7fe6a5a50740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578d8fbea60 0x5578d8fc08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:59.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.796+0000 7fe6a5a50740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5578d8fbf450 0x5578d8fd02d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:59.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.796+0000 7fe6a5a50740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5578d8fc45c0 0x5578d8fc0df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:59.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.796+0000 7fe6a5a50740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fe67c00fc90 con 0x5578d8fc45c0 2024-03-18T16:12:59.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.796+0000 7fe6a5a50740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fe67c011950 con 0x5578d8fbf450 2024-03-18T16:12:59.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.796+0000 7fe6a5a50740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fe67c010360 con 0x5578d8fbea60 2024-03-18T16:12:59.800 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.797+0000 7fe69b400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5578d8fbf450 0x5578d8fd02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:59.800 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.798+0000 7fe69be00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578d8fbea60 0x5578d8fc08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:59.800 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.798+0000 7fe6a0e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5578d8fc45c0 0x5578d8fc0df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:59.800 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.798+0000 7fe69be00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578d8fbea60 0x5578d8fc08b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:37446/0 (socket says 172.21.15.39:37446) 2024-03-18T16:12:59.800 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.798+0000 7fe69be00640 1 -- 172.21.15.39:0/3142764341 learned_addr learned my addr 172.21.15.39:0/3142764341 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:12:59.801 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.798+0000 7fe69b400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5578d8fbf450 0x5578d8fd02d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:60834/0 (socket says 172.21.15.39:60834) 2024-03-18T16:12:59.801 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.798+0000 7fe69be00640 1 -- 172.21.15.39:0/3142764341 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5578d8fbf450 msgr2=0x5578d8fd02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:12:59.801 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.798+0000 7fe69be00640 1 --2- 172.21.15.39:0/3142764341 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5578d8fbf450 0x5578d8fd02d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:59.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.798+0000 7fe69be00640 1 -- 172.21.15.39:0/3142764341 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5578d8fc45c0 msgr2=0x5578d8fc0df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:12:59.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.798+0000 7fe69be00640 1 --2- 172.21.15.39:0/3142764341 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5578d8fc45c0 0x5578d8fc0df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:59.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.798+0000 7fe69be00640 1 -- 172.21.15.39:0/3142764341 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5578d8fbfce0 con 0x5578d8fbea60 2024-03-18T16:12:59.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.799+0000 7fe69be00640 1 --2- 172.21.15.39:0/3142764341 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578d8fbea60 0x5578d8fc08b0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fe69c050a90 tx=0x7fe69c076040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d23d274e10e6d04 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:59.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.801+0000 7fe68cc00640 1 -- 172.21.15.39:0/3142764341 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe69c08b3c0 con 0x5578d8fbea60 2024-03-18T16:12:59.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.801+0000 7fe68cc00640 1 -- 172.21.15.39:0/3142764341 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe69c078040 con 0x5578d8fbea60 2024-03-18T16:12:59.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.801+0000 7fe68cc00640 1 -- 172.21.15.39:0/3142764341 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe69c069330 con 0x5578d8fbea60 2024-03-18T16:12:59.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.803+0000 7fe6a5a50740 1 -- 172.21.15.39:0/3142764341 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578d8fbea60 msgr2=0x5578d8fc08b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:59.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.803+0000 7fe6a5a50740 1 --2- 172.21.15.39:0/3142764341 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578d8fbea60 0x5578d8fc08b0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fe69c050a90 tx=0x7fe69c076040 comp rx=0 tx=0).stop 2024-03-18T16:12:59.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.803+0000 7fe6a5a50740 1 -- 172.21.15.39:0/3142764341 shutdown_connections 2024-03-18T16:12:59.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.803+0000 7fe6a5a50740 1 --2- 172.21.15.39:0/3142764341 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5578d8fbf450 0x5578d8fd02d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:59.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.803+0000 7fe6a5a50740 1 --2- 172.21.15.39:0/3142764341 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5578d8fc45c0 0x5578d8fc0df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:59.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.803+0000 7fe6a5a50740 1 --2- 172.21.15.39:0/3142764341 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578d8fbea60 0x5578d8fc08b0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:59.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.803+0000 7fe6a5a50740 1 -- 172.21.15.39:0/3142764341 >> 172.21.15.39:0/3142764341 conn(0x5578d8fb9fd0 msgr2=0x5578d8fba3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:59.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.803+0000 7fe6a5a50740 1 -- 172.21.15.39:0/3142764341 shutdown_connections 2024-03-18T16:12:59.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.804+0000 7fe6a5a50740 1 -- 172.21.15.39:0/3142764341 wait complete. 2024-03-18T16:12:59.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.804+0000 7fe6a5a50740 1 -- start start 2024-03-18T16:12:59.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.804+0000 7fe6a5a50740 1 Processor -- start 2024-03-18T16:12:59.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.805+0000 7fe6a5a50740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5578d8fbea60 0x5578d8fcdcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:59.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.806+0000 7fe6a5a50740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5578d8fbf450 0x5578d8fce200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:59.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.806+0000 7fe6a5a50740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578d8fc45c0 0x5578d8fcf210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:59.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.806+0000 7fe6a5a50740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fe67c0111c0 con 0x5578d8fbf450 2024-03-18T16:12:59.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.806+0000 7fe6a5a50740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fe67c010a30 con 0x5578d8fbea60 2024-03-18T16:12:59.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.806+0000 7fe6a5a50740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fe67c00dc70 con 0x5578d8fc45c0 2024-03-18T16:12:59.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.806+0000 7fe6a5a50740 10 client.0 _ll_register_callbacks cb 0x7fe6880057f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:12:59.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.806+0000 7fe69be00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5578d8fbea60 0x5578d8fcdcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:59.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.807+0000 7fe69be00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5578d8fbea60 0x5578d8fcdcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:60836/0 (socket says 172.21.15.39:60836) 2024-03-18T16:12:59.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.807+0000 7fe69be00640 1 -- 172.21.15.39:0/3369934780 learned_addr learned my addr 172.21.15.39:0/3369934780 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:12:59.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.807+0000 7fe69b400640 1 --2- 172.21.15.39:0/3369934780 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5578d8fbf450 0x5578d8fce200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:59.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.807+0000 7fe6a0e00640 1 --2- 172.21.15.39:0/3369934780 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578d8fc45c0 0x5578d8fcf210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:59.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.808+0000 7fe69be00640 1 -- 172.21.15.39:0/3369934780 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578d8fc45c0 msgr2=0x5578d8fcf210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:12:59.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.808+0000 7fe69be00640 1 --2- 172.21.15.39:0/3369934780 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578d8fc45c0 0x5578d8fcf210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:59.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.808+0000 7fe69be00640 1 -- 172.21.15.39:0/3369934780 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5578d8fbf450 msgr2=0x5578d8fce200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:12:59.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.808+0000 7fe69be00640 1 --2- 172.21.15.39:0/3369934780 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5578d8fbf450 0x5578d8fce200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:59.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.808+0000 7fe69be00640 1 -- 172.21.15.39:0/3369934780 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5578d8fcf800 con 0x5578d8fbea60 2024-03-18T16:12:59.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.808+0000 7fe69b400640 1 --2- 172.21.15.39:0/3369934780 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5578d8fbf450 0x5578d8fce200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:12:59.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.808+0000 7fe6a0e00640 1 --2- 172.21.15.39:0/3369934780 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578d8fc45c0 0x5578d8fcf210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:12:59.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.809+0000 7fe69be00640 1 --2- 172.21.15.39:0/3369934780 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5578d8fbea60 0x5578d8fcdcc0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fe69c050d20 tx=0x7fe69c04ff80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1de8de9ecf3c174b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:12:59.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.809+0000 7fe682000640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:12:59.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.809+0000 7fe682000640 1 -- 172.21.15.39:0/3369934780 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe69c09b060 con 0x5578d8fbea60 2024-03-18T16:12:59.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.809+0000 7fe682000640 1 -- 172.21.15.39:0/3369934780 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fe69c0718f0 con 0x5578d8fbea60 2024-03-18T16:12:59.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.809+0000 7fe6a5a50740 1 -- 172.21.15.39:0/3369934780 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe69c078040 con 0x5578d8fbea60 2024-03-18T16:12:59.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.810+0000 7fe682000640 1 -- 172.21.15.39:0/3369934780 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fe69c071b70 con 0x5578d8fbea60 2024-03-18T16:12:59.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.810+0000 7fe6a5a50740 10 client.26174 fetch_fsmap learned FSMap version 35 2024-03-18T16:12:59.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.810+0000 7fe6a5a50740 1 -- 172.21.15.39:0/3369934780 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5578d903b230 con 0x5578d8fbea60 2024-03-18T16:12:59.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.810+0000 7fe682000640 1 -- 172.21.15.39:0/3369934780 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe69c078040 con 0x5578d8fbea60 2024-03-18T16:12:59.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.811+0000 7fe682000640 1 -- 172.21.15.39:0/3369934780 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 35) ==== 8402+0+0 (secure 0 0 0) 0x7fe69c096450 con 0x5578d8fbea60 2024-03-18T16:12:59.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.811+0000 7fe6a5a50740 10 client.26174 fetch_fsmap finished waiting for FSMap version 35 2024-03-18T16:12:59.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.811+0000 7fe6a5a50740 10 client.26174 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-03-18T16:12:59.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.811+0000 7fe6a5a50740 20 client.26174 populate_metadata read hostname 'smithi039' 2024-03-18T16:12:59.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.811+0000 7fe6a5a50740 1 --2- 172.21.15.39:0/3369934780 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x5578d903baf0 0x5578d903df10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:12:59.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.811+0000 7fe6a5a50740 4 client.26174 mds_command: new command op to 14463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:12:59.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.811+0000 7fe6a5a50740 1 -- 172.21.15.39:0/3369934780 --> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5578d8f2ace0 con 0x5578d903baf0 2024-03-18T16:12:59.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.813+0000 7fe69b400640 1 --2- 172.21.15.39:0/3369934780 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x5578d903baf0 0x5578d903df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:12:59.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.814+0000 7fe69b400640 1 --2- 172.21.15.39:0/3369934780 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x5578d903baf0 0x5578d903df10 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=4f095bb022a0e88b server_cookie=fde7864f5eab580f in_seq=0 out_seq=0 2024-03-18T16:12:59.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.815+0000 7fe682000640 10 client.26174 ms_handle_connect on v2:172.21.15.39:6840/3515326042 2024-03-18T16:12:59.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.825+0000 7fe682000640 1 -- 172.21.15.39:0/3369934780 <== mds.4 v2:172.21.15.39:6840/3515326042 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5578d8f2ace0 con 0x5578d903baf0 2024-03-18T16:12:59.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.825+0000 7fe682000640 10 client.26174 handle_command_reply: tid=0 2024-03-18T16:12:59.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.825+0000 7fe6a5a50740 1 client.26174 shutdown 2024-03-18T16:12:59.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.826+0000 7fe6a5a50740 1 -- 172.21.15.39:0/3369934780 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5578d8fbea60 msgr2=0x5578d8fcdcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:59.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.826+0000 7fe6a5a50740 1 --2- 172.21.15.39:0/3369934780 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5578d8fbea60 0x5578d8fcdcc0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fe69c050d20 tx=0x7fe69c04ff80 comp rx=0 tx=0).stop 2024-03-18T16:12:59.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.826+0000 7fe6a5a50740 1 -- 172.21.15.39:0/3369934780 shutdown_connections 2024-03-18T16:12:59.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.826+0000 7fe6a5a50740 1 --2- 172.21.15.39:0/3369934780 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x5578d903baf0 0x5578d903df10 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:59.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.826+0000 7fe6a5a50740 1 --2- 172.21.15.39:0/3369934780 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5578d8fc45c0 0x5578d8fcf210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:59.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.826+0000 7fe6a5a50740 1 --2- 172.21.15.39:0/3369934780 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5578d8fbf450 0x5578d8fce200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:59.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.826+0000 7fe6a5a50740 1 --2- 172.21.15.39:0/3369934780 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5578d8fbea60 0x5578d8fcdcc0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:59.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.826+0000 7fe682000640 0 client.26174 ms_handle_reset on v2:172.21.15.39:6840/3515326042 2024-03-18T16:12:59.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.826+0000 7fe6a5a50740 1 -- 172.21.15.39:0/3369934780 >> 172.21.15.39:0/3369934780 conn(0x5578d8fb9fd0 msgr2=0x5578d8fc08b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:59.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.827+0000 7fe6a5a50740 1 -- 172.21.15.39:0/3369934780 shutdown_connections 2024-03-18T16:12:59.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.827+0000 7fe6a5a50740 1 -- 172.21.15.39:0/3369934780 wait complete. 2024-03-18T16:12:59.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.827+0000 7fe6a5a50740 20 client.26174 trim_cache size 0 max 16384 2024-03-18T16:12:59.831 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:12:59.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.828+0000 7fe6a1800640 1 -- 172.21.15.39:0/3843437861 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fe6840a2900 msgr2=0x7fe6840a4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:59.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.828+0000 7fe6a1800640 1 --2- 172.21.15.39:0/3843437861 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fe6840a2900 0x7fe6840a4dc0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fe6900058c0 tx=0x7fe690013590 comp rx=0 tx=0).stop 2024-03-18T16:12:59.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.828+0000 7fe6a1800640 1 -- 172.21.15.39:0/3843437861 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe6940a6710 msgr2=0x7fe6940a88e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:12:59.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.828+0000 7fe6a1800640 1 --2- 172.21.15.39:0/3843437861 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe6940a6710 0x7fe6940a88e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fe688009230 tx=0x7fe6880045e0 comp rx=0 tx=0).stop 2024-03-18T16:12:59.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.829+0000 7fe6a1800640 1 -- 172.21.15.39:0/3843437861 shutdown_connections 2024-03-18T16:12:59.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.829+0000 7fe6a1800640 1 --2- 172.21.15.39:0/3843437861 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fe6840a2900 0x7fe6840a4dc0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:59.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.829+0000 7fe6a1800640 1 --2- 172.21.15.39:0/3843437861 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe6940a6710 0x7fe6940a88e0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:59.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.829+0000 7fe6a1800640 1 --2- 172.21.15.39:0/3843437861 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe6940a5d70 0x7fe6940ac2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:59.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.829+0000 7fe6a1800640 1 --2- 172.21.15.39:0/3843437861 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe6940a4c40 0x7fe6940abd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:12:59.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.829+0000 7fe6a1800640 1 -- 172.21.15.39:0/3843437861 >> 172.21.15.39:0/3843437861 conn(0x7fe6940a00d0 msgr2=0x7fe6940a1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:12:59.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.829+0000 7fe6a1800640 1 -- 172.21.15.39:0/3843437861 shutdown_connections 2024-03-18T16:12:59.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:12:59.829+0000 7fe6a1800640 1 -- 172.21.15.39:0/3843437861 wait complete. 2024-03-18T16:12:59.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:12:59 smithi039 ceph-mon[31721]: pgmap v509: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 58 MiB/s wr, 8.31k op/s 2024-03-18T16:12:59.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:12:59 smithi039 ceph-mon[31721]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T16:12:59.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:12:59 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:12:59.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:12:59 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/2333816666' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:12:59.860 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-18T16:13:00.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.293+0000 7f3761200640 1 -- 172.21.15.39:0/1281631355 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f375c072bf0 msgr2=0x7f375c073010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:00.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.293+0000 7f3761200640 1 --2- 172.21.15.39:0/1281631355 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f375c072bf0 0x7f375c073010 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f374c009900 tx=0x7f374c02bc30 comp rx=0 tx=0).stop 2024-03-18T16:13:00.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.297+0000 7f3761200640 1 -- 172.21.15.39:0/1281631355 shutdown_connections 2024-03-18T16:13:00.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.297+0000 7f3761200640 1 --2- 172.21.15.39:0/1281631355 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f375c109870 0x7f375c10e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:00.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.297+0000 7f3761200640 1 --2- 172.21.15.39:0/1281631355 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f375c108eb0 0x7f375c109330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:00.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.297+0000 7f3761200640 1 --2- 172.21.15.39:0/1281631355 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f375c072bf0 0x7f375c073010 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:00.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.297+0000 7f3761200640 1 -- 172.21.15.39:0/1281631355 >> 172.21.15.39:0/1281631355 conn(0x7f375c06e6c0 msgr2=0x7f375c070b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:00.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.299+0000 7f3761200640 1 -- 172.21.15.39:0/1281631355 shutdown_connections 2024-03-18T16:13:00.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.299+0000 7f3761200640 1 -- 172.21.15.39:0/1281631355 wait complete. 2024-03-18T16:13:00.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.301+0000 7f3761200640 1 Processor -- start 2024-03-18T16:13:00.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.302+0000 7f3761200640 1 -- start start 2024-03-18T16:13:00.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.302+0000 7f3761200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f375c072bf0 0x7f375c1a6740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:00.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.302+0000 7f3761200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f375c108eb0 0x7f375c1a6c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:00.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.302+0000 7f3761200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f375c109870 0x7f375c1adcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:00.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.302+0000 7f3761200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f375c11c1c0 con 0x7f375c072bf0 2024-03-18T16:13:00.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.302+0000 7f3761200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f375c11c040 con 0x7f375c109870 2024-03-18T16:13:00.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.302+0000 7f3761200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f375c11c340 con 0x7f375c108eb0 2024-03-18T16:13:00.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.303+0000 7f375aa00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f375c108eb0 0x7f375c1a6c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:00.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.303+0000 7f375aa00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f375c108eb0 0x7f375c1a6c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:37470/0 (socket says 172.21.15.39:37470) 2024-03-18T16:13:00.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.303+0000 7f375aa00640 1 -- 172.21.15.39:0/1376096270 learned_addr learned my addr 172.21.15.39:0/1376096270 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:13:00.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.303+0000 7f375aa00640 1 -- 172.21.15.39:0/1376096270 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f375c109870 msgr2=0x7f375c1adcf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T16:13:00.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.303+0000 7f375aa00640 1 --2- 172.21.15.39:0/1376096270 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f375c109870 0x7f375c1adcf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:00.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.303+0000 7f375aa00640 1 -- 172.21.15.39:0/1376096270 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f375c072bf0 msgr2=0x7f375c1a6740 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T16:13:00.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.304+0000 7f375aa00640 1 --2- 172.21.15.39:0/1376096270 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f375c072bf0 0x7f375c1a6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:00.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.304+0000 7f375aa00640 1 -- 172.21.15.39:0/1376096270 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f375c1ae360 con 0x7f375c108eb0 2024-03-18T16:13:00.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.305+0000 7f375aa00640 1 --2- 172.21.15.39:0/1376096270 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f375c108eb0 0x7f375c1a6c80 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f374800c920 tx=0x7f374800cde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:00.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.305+0000 7f3753e00640 1 -- 172.21.15.39:0/1376096270 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3748007c70 con 0x7f375c108eb0 2024-03-18T16:13:00.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.306+0000 7f3761200640 1 -- 172.21.15.39:0/1376096270 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f375c1ae650 con 0x7f375c108eb0 2024-03-18T16:13:00.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.306+0000 7f3761200640 1 -- 172.21.15.39:0/1376096270 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f375c1aebf0 con 0x7f375c108eb0 2024-03-18T16:13:00.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.314+0000 7f3753e00640 1 -- 172.21.15.39:0/1376096270 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3748007e10 con 0x7f375c108eb0 2024-03-18T16:13:00.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.314+0000 7f3753e00640 1 -- 172.21.15.39:0/1376096270 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3748018890 con 0x7f375c108eb0 2024-03-18T16:13:00.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.317+0000 7f3765400740 1 Processor -- start 2024-03-18T16:13:00.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.317+0000 7f3765400740 1 -- start start 2024-03-18T16:13:00.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.318+0000 7f3765400740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5558c4f6aa60 0x5558c4f6ae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:00.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.318+0000 7f3765400740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5558c4f6b450 0x5558c4f6ff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:00.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.318+0000 7f3765400740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5558c4f705c0 0x5558c4f72a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:00.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.318+0000 7f3765400740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5558c4f323b0 con 0x5558c4f6aa60 2024-03-18T16:13:00.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.318+0000 7f3765400740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5558c4ecf450 con 0x5558c4f6b450 2024-03-18T16:13:00.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.318+0000 7f3765400740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x5558c4e77cb0 con 0x5558c4f705c0 2024-03-18T16:13:00.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.319+0000 7f375be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5558c4f6aa60 0x5558c4f6ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:00.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.319+0000 7f375be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5558c4f6aa60 0x5558c4f6ae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:46258/0 (socket says 172.21.15.39:46258) 2024-03-18T16:13:00.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.319+0000 7f375be00640 1 -- 172.21.15.39:0/362699133 learned_addr learned my addr 172.21.15.39:0/362699133 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:13:00.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.320+0000 7f375be00640 1 -- 172.21.15.39:0/362699133 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5558c4f705c0 msgr2=0x5558c4f72a50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:13:00.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.320+0000 7f375be00640 1 --2- 172.21.15.39:0/362699133 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5558c4f705c0 0x5558c4f72a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:00.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.320+0000 7f375be00640 1 -- 172.21.15.39:0/362699133 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5558c4f6b450 msgr2=0x5558c4f6ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:00.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.320+0000 7f375b400640 1 --2- 172.21.15.39:0/362699133 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5558c4f6b450 0x5558c4f6ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:00.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.320+0000 7f375be00640 1 --2- 172.21.15.39:0/362699133 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5558c4f6b450 0x5558c4f6ff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:00.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.320+0000 7f375be00640 1 -- 172.21.15.39:0/362699133 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5558c4e991f0 con 0x5558c4f6aa60 2024-03-18T16:13:00.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.320+0000 7f375b400640 1 --2- 172.21.15.39:0/362699133 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5558c4f6b450 0x5558c4f6ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:13:00.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.321+0000 7f375be00640 1 --2- 172.21.15.39:0/362699133 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5558c4f6aa60 0x5558c4f6ae80 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3754009e30 tx=0x7f375400e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e98a37913c6df12 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:00.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.321+0000 7f3750c00640 1 -- 172.21.15.39:0/362699133 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3754004d20 con 0x5558c4f6aa60 2024-03-18T16:13:00.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.321+0000 7f3750c00640 1 -- 172.21.15.39:0/362699133 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f375400ee70 con 0x5558c4f6aa60 2024-03-18T16:13:00.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.321+0000 7f3750c00640 1 -- 172.21.15.39:0/362699133 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3754010820 con 0x5558c4f6aa60 2024-03-18T16:13:00.326 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.322+0000 7f3765400740 1 -- 172.21.15.39:0/362699133 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5558c4f6aa60 msgr2=0x5558c4f6ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:00.326 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.322+0000 7f3765400740 1 --2- 172.21.15.39:0/362699133 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5558c4f6aa60 0x5558c4f6ae80 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3754009e30 tx=0x7f375400e720 comp rx=0 tx=0).stop 2024-03-18T16:13:00.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.323+0000 7f3765400740 1 -- 172.21.15.39:0/362699133 shutdown_connections 2024-03-18T16:13:00.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.323+0000 7f3765400740 1 --2- 172.21.15.39:0/362699133 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5558c4f6b450 0x5558c4f6ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:00.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.323+0000 7f3765400740 1 --2- 172.21.15.39:0/362699133 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5558c4f705c0 0x5558c4f72a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:00.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.323+0000 7f3765400740 1 --2- 172.21.15.39:0/362699133 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5558c4f6aa60 0x5558c4f6ae80 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:00.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.323+0000 7f3765400740 1 -- 172.21.15.39:0/362699133 >> 172.21.15.39:0/362699133 conn(0x5558c4f65fd0 msgr2=0x5558c4f663d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:00.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.324+0000 7f3765400740 1 -- 172.21.15.39:0/362699133 shutdown_connections 2024-03-18T16:13:00.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.324+0000 7f3765400740 1 -- 172.21.15.39:0/362699133 wait complete. 2024-03-18T16:13:00.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.325+0000 7f3765400740 1 -- start start 2024-03-18T16:13:00.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.325+0000 7f3765400740 1 Processor -- start 2024-03-18T16:13:00.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.328+0000 7f3753e00640 1 -- 172.21.15.39:0/1376096270 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f3748018b30 con 0x7f375c108eb0 2024-03-18T16:13:00.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.333+0000 7f3753e00640 1 --2- 172.21.15.39:0/1376096270 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f37400a2810 0x7f37400a4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:00.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.334+0000 7f3765400740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5558c4f6aa60 0x5558c4f85e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:00.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.334+0000 7f3765400740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5558c4f6b450 0x5558c4f86350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:00.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.334+0000 7f3765400740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5558c4f705c0 0x5558c4f8d390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:00.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.334+0000 7f3765400740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5558c4e8b110 con 0x5558c4f705c0 2024-03-18T16:13:00.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.334+0000 7f3765400740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5558c4ecb4f0 con 0x5558c4f6aa60 2024-03-18T16:13:00.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.334+0000 7f3765400740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x5558c4f84580 con 0x5558c4f6b450 2024-03-18T16:13:00.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.334+0000 7f3765400740 10 client.0 _ll_register_callbacks cb 0x5558c4e78b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:13:00.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.336+0000 7f375b400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5558c4f6aa60 0x5558c4f85e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:00.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.336+0000 7f375b400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5558c4f6aa60 0x5558c4f85e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:60854/0 (socket says 172.21.15.39:60854) 2024-03-18T16:13:00.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.336+0000 7f375b400640 1 -- 172.21.15.39:0/1014251381 learned_addr learned my addr 172.21.15.39:0/1014251381 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:13:00.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.336+0000 7f375b400640 1 -- 172.21.15.39:0/1014251381 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5558c4f6b450 msgr2=0x5558c4f86350 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:13:00.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.338+0000 7f375b400640 1 --2- 172.21.15.39:0/1014251381 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5558c4f6b450 0x5558c4f86350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:00.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.338+0000 7f375b400640 1 -- 172.21.15.39:0/1014251381 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5558c4f705c0 msgr2=0x5558c4f8d390 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:13:00.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.338+0000 7f375b400640 1 --2- 172.21.15.39:0/1014251381 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5558c4f705c0 0x5558c4f8d390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:00.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.338+0000 7f375b400640 1 -- 172.21.15.39:0/1014251381 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5558c4be33f0 con 0x5558c4f6aa60 2024-03-18T16:13:00.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.339+0000 7f3753e00640 1 -- 172.21.15.39:0/1376096270 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f37480cef50 con 0x7f375c108eb0 2024-03-18T16:13:00.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.339+0000 7f375b400640 1 --2- 172.21.15.39:0/1014251381 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5558c4f6aa60 0x5558c4f85e10 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f374c036040 tx=0x7f374c004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c1b28f665d25603 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:00.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.340+0000 7f375be00640 1 --2- 172.21.15.39:0/1376096270 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f37400a2810 0x7f37400a4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:00.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.340+0000 7f3745600640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:13:00.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.341+0000 7f3745600640 1 -- 172.21.15.39:0/1014251381 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f374c043020 con 0x5558c4f6aa60 2024-03-18T16:13:00.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.341+0000 7f3765400740 1 -- 172.21.15.39:0/1014251381 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f375400ee70 con 0x5558c4f6aa60 2024-03-18T16:13:00.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.341+0000 7f3745600640 1 -- 172.21.15.39:0/1014251381 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f374c03a040 con 0x5558c4f6aa60 2024-03-18T16:13:00.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.341+0000 7f3745600640 1 -- 172.21.15.39:0/1014251381 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f374c04e050 con 0x5558c4f6aa60 2024-03-18T16:13:00.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.341+0000 7f375be00640 1 --2- 172.21.15.39:0/1376096270 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f37400a2810 0x7f37400a4cd0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f3754009c70 tx=0x7f3754005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:00.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.342+0000 7f3765400740 10 client.26180 fetch_fsmap learned FSMap version 35 2024-03-18T16:13:00.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.342+0000 7f3765400740 1 -- 172.21.15.39:0/1014251381 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5558c4cad7d0 con 0x5558c4f6aa60 2024-03-18T16:13:00.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.342+0000 7f3745600640 1 -- 172.21.15.39:0/1014251381 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f375400ee70 con 0x5558c4f6aa60 2024-03-18T16:13:00.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.344+0000 7f3745600640 1 -- 172.21.15.39:0/1014251381 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 35) ==== 8402+0+0 (secure 0 0 0) 0x7f374c0048c0 con 0x5558c4f6aa60 2024-03-18T16:13:00.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.345+0000 7f3765400740 10 client.26180 fetch_fsmap finished waiting for FSMap version 35 2024-03-18T16:13:00.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.345+0000 7f3765400740 10 client.26180 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-18T16:13:00.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.346+0000 7f3765400740 20 client.26180 populate_metadata read hostname 'smithi039' 2024-03-18T16:13:00.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.346+0000 7f3765400740 1 --2- 172.21.15.39:0/1014251381 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x5558c4f91d00 0x5558c4f94180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:00.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.346+0000 7f3765400740 4 client.26180 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:13:00.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.346+0000 7f3765400740 1 -- 172.21.15.39:0/1014251381 --> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5558c4edc990 con 0x5558c4f91d00 2024-03-18T16:13:00.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.347+0000 7f375aa00640 1 --2- 172.21.15.39:0/1014251381 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x5558c4f91d00 0x5558c4f94180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:00.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.350+0000 7f375aa00640 1 --2- 172.21.15.39:0/1014251381 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x5558c4f91d00 0x5558c4f94180 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=bf04a28016818bc6 server_cookie=9e45ddf980cb4483 in_seq=0 out_seq=0 2024-03-18T16:13:00.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.350+0000 7f3745600640 10 client.26180 ms_handle_connect on v2:172.21.15.194:6836/1301464246 2024-03-18T16:13:00.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.354+0000 7f3745600640 1 -- 172.21.15.39:0/1014251381 <== mds.3 v2:172.21.15.194:6836/1301464246 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5558c4edc990 con 0x5558c4f91d00 2024-03-18T16:13:00.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.354+0000 7f3745600640 10 client.26180 handle_command_reply: tid=0 2024-03-18T16:13:00.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.354+0000 7f3765400740 1 client.26180 shutdown 2024-03-18T16:13:00.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.354+0000 7f3765400740 1 -- 172.21.15.39:0/1014251381 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5558c4f6aa60 msgr2=0x5558c4f85e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:00.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.354+0000 7f3765400740 1 --2- 172.21.15.39:0/1014251381 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5558c4f6aa60 0x5558c4f85e10 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f374c036040 tx=0x7f374c004270 comp rx=0 tx=0).stop 2024-03-18T16:13:00.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.355+0000 7f3765400740 1 -- 172.21.15.39:0/1014251381 shutdown_connections 2024-03-18T16:13:00.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.355+0000 7f3765400740 1 --2- 172.21.15.39:0/1014251381 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x5558c4f91d00 0x5558c4f94180 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:00.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.355+0000 7f3765400740 1 --2- 172.21.15.39:0/1014251381 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5558c4f705c0 0x5558c4f8d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:00.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.355+0000 7f3765400740 1 --2- 172.21.15.39:0/1014251381 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5558c4f6b450 0x5558c4f86350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:00.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.355+0000 7f3765400740 1 --2- 172.21.15.39:0/1014251381 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5558c4f6aa60 0x5558c4f85e10 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:00.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.355+0000 7f3745600640 0 client.26180 ms_handle_reset on v2:172.21.15.194:6836/1301464246 2024-03-18T16:13:00.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.355+0000 7f3765400740 1 -- 172.21.15.39:0/1014251381 >> 172.21.15.39:0/1014251381 conn(0x5558c4f65fd0 msgr2=0x5558c4f651f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:00.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.356+0000 7f3765400740 1 -- 172.21.15.39:0/1014251381 shutdown_connections 2024-03-18T16:13:00.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.356+0000 7f3765400740 1 -- 172.21.15.39:0/1014251381 wait complete. 2024-03-18T16:13:00.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.358+0000 7f3765400740 20 client.26180 trim_cache size 0 max 16384 2024-03-18T16:13:00.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.402+0000 7f3765400740 1 Processor -- start 2024-03-18T16:13:00.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.403+0000 7f3765400740 1 -- start start 2024-03-18T16:13:00.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.404+0000 7f3765400740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5558c4f6aa60 0x5558c4f663b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:00.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.404+0000 7f3765400740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5558c4f6b450 0x5558c4f85100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:00.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.404+0000 7f3765400740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5558c4f705c0 0x5558c4f85640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:00.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.404+0000 7f3765400740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f373c00fc30 con 0x5558c4f6aa60 2024-03-18T16:13:00.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.404+0000 7f3765400740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f373c0118f0 con 0x5558c4f705c0 2024-03-18T16:13:00.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.404+0000 7f3765400740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f373c010300 con 0x5558c4f6b450 2024-03-18T16:13:00.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.404+0000 7f375b400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5558c4f6aa60 0x5558c4f663b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:00.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.405+0000 7f375b400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5558c4f6aa60 0x5558c4f663b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:46270/0 (socket says 172.21.15.39:46270) 2024-03-18T16:13:00.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.405+0000 7f375b400640 1 -- 172.21.15.39:0/4184990135 learned_addr learned my addr 172.21.15.39:0/4184990135 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:13:00.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.405+0000 7f375aa00640 1 --2- 172.21.15.39:0/4184990135 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5558c4f6b450 0x5558c4f85100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:00.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.406+0000 7f375be00640 1 --2- 172.21.15.39:0/4184990135 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5558c4f705c0 0x5558c4f85640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:00.409 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.406+0000 7f375b400640 1 -- 172.21.15.39:0/4184990135 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5558c4f6b450 msgr2=0x5558c4f85100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:00.409 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.406+0000 7f375b400640 1 --2- 172.21.15.39:0/4184990135 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5558c4f6b450 0x5558c4f85100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:00.409 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.406+0000 7f375b400640 1 -- 172.21.15.39:0/4184990135 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5558c4f705c0 msgr2=0x5558c4f85640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:00.409 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.406+0000 7f375b400640 1 --2- 172.21.15.39:0/4184990135 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5558c4f705c0 0x5558c4f85640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:00.409 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.406+0000 7f375b400640 1 -- 172.21.15.39:0/4184990135 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5558c4f6a6a0 con 0x5558c4f6aa60 2024-03-18T16:13:00.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.409+0000 7f375b400640 1 --2- 172.21.15.39:0/4184990135 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5558c4f6aa60 0x5558c4f663b0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f374c009780 tx=0x7f374c0097b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82a65901e3a3d07d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:00.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.418+0000 7f3747e00640 1 -- 172.21.15.39:0/4184990135 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f374c056020 con 0x5558c4f6aa60 2024-03-18T16:13:00.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.418+0000 7f3747e00640 1 -- 172.21.15.39:0/4184990135 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f374c009520 con 0x5558c4f6aa60 2024-03-18T16:13:00.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.419+0000 7f3765400740 1 -- 172.21.15.39:0/4184990135 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5558c4f6aa60 msgr2=0x5558c4f663b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:00.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.419+0000 7f3765400740 1 --2- 172.21.15.39:0/4184990135 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5558c4f6aa60 0x5558c4f663b0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f374c009780 tx=0x7f374c0097b0 comp rx=0 tx=0).stop 2024-03-18T16:13:00.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.420+0000 7f3765400740 1 -- 172.21.15.39:0/4184990135 shutdown_connections 2024-03-18T16:13:00.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.420+0000 7f3765400740 1 --2- 172.21.15.39:0/4184990135 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5558c4f705c0 0x5558c4f85640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:00.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.420+0000 7f3765400740 1 --2- 172.21.15.39:0/4184990135 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5558c4f6b450 0x5558c4f85100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:00.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.420+0000 7f3765400740 1 --2- 172.21.15.39:0/4184990135 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5558c4f6aa60 0x5558c4f663b0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:00.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.421+0000 7f3765400740 1 -- 172.21.15.39:0/4184990135 >> 172.21.15.39:0/4184990135 conn(0x5558c4f65fd0 msgr2=0x5558c4f69ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:00.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.423+0000 7f3765400740 1 -- 172.21.15.39:0/4184990135 shutdown_connections 2024-03-18T16:13:00.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.424+0000 7f3765400740 1 -- 172.21.15.39:0/4184990135 wait complete. 2024-03-18T16:13:00.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.425+0000 7f3765400740 1 -- start start 2024-03-18T16:13:00.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.426+0000 7f3765400740 1 Processor -- start 2024-03-18T16:13:00.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.426+0000 7f3765400740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5558c4f6aa60 0x5558c4f8a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:00.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.428+0000 7f3765400740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5558c4f6b450 0x5558c4f8ac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:00.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.428+0000 7f375b400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5558c4f6aa60 0x5558c4f8a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:00.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.428+0000 7f375b400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5558c4f6aa60 0x5558c4f8a6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:46278/0 (socket says 172.21.15.39:46278) 2024-03-18T16:13:00.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.429+0000 7f375b400640 1 -- 172.21.15.39:0/197054082 learned_addr learned my addr 172.21.15.39:0/197054082 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:13:00.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.429+0000 7f3765400740 1 --2- 172.21.15.39:0/197054082 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5558c4f705c0 0x5558c4f8bc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:00.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.429+0000 7f375be00640 1 --2- 172.21.15.39:0/197054082 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5558c4f705c0 0x5558c4f8bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:00.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.430+0000 7f3765400740 1 -- 172.21.15.39:0/197054082 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f373c011160 con 0x5558c4f6aa60 2024-03-18T16:13:00.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.430+0000 7f3765400740 1 -- 172.21.15.39:0/197054082 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f373c0109d0 con 0x5558c4f705c0 2024-03-18T16:13:00.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.430+0000 7f3765400740 1 -- 172.21.15.39:0/197054082 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f373c00dc10 con 0x5558c4f6b450 2024-03-18T16:13:00.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.430+0000 7f3765400740 10 client.0 _ll_register_callbacks cb 0x7f37540093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:13:00.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.430+0000 7f375aa00640 1 --2- 172.21.15.39:0/197054082 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5558c4f6b450 0x5558c4f8ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:00.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.430+0000 7f375b400640 1 -- 172.21.15.39:0/197054082 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5558c4f6b450 msgr2=0x5558c4f8ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:00.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.430+0000 7f375b400640 1 --2- 172.21.15.39:0/197054082 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5558c4f6b450 0x5558c4f8ac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:00.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.430+0000 7f375b400640 1 -- 172.21.15.39:0/197054082 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5558c4f705c0 msgr2=0x5558c4f8bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:00.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.430+0000 7f375b400640 1 --2- 172.21.15.39:0/197054082 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5558c4f705c0 0x5558c4f8bc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:00.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.430+0000 7f375b400640 1 -- 172.21.15.39:0/197054082 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5558c4f8c230 con 0x5558c4f6aa60 2024-03-18T16:13:00.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.431+0000 7f375b400640 1 --2- 172.21.15.39:0/197054082 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5558c4f6aa60 0x5558c4f8a6f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f374c004e70 tx=0x7f374c0048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2580b8efb60380d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:00.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.431+0000 7f3745600640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-03-18T16:13:00.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.432+0000 7f3745600640 1 -- 172.21.15.39:0/197054082 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f374c05f070 con 0x5558c4f6aa60 2024-03-18T16:13:00.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.432+0000 7f3745600640 1 -- 172.21.15.39:0/197054082 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f374c005000 con 0x5558c4f6aa60 2024-03-18T16:13:00.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.432+0000 7f3765400740 1 -- 172.21.15.39:0/197054082 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f374c009520 con 0x5558c4f6aa60 2024-03-18T16:13:00.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.434+0000 7f3745600640 1 -- 172.21.15.39:0/197054082 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f374c069050 con 0x5558c4f6aa60 2024-03-18T16:13:00.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.435+0000 7f3765400740 10 client.15960 fetch_fsmap learned FSMap version 35 2024-03-18T16:13:00.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.435+0000 7f3765400740 1 -- 172.21.15.39:0/197054082 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5558c4ff8080 con 0x5558c4f6aa60 2024-03-18T16:13:00.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.436+0000 7f3745600640 1 -- 172.21.15.39:0/197054082 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f374c009520 con 0x5558c4f6aa60 2024-03-18T16:13:00.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.436+0000 7f3745600640 1 -- 172.21.15.39:0/197054082 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 35) ==== 8402+0+0 (secure 0 0 0) 0x7f374c05b750 con 0x5558c4f6aa60 2024-03-18T16:13:00.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.436+0000 7f3765400740 10 client.15960 fetch_fsmap finished waiting for FSMap version 35 2024-03-18T16:13:00.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.436+0000 7f3765400740 10 client.15960 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-18T16:13:00.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.436+0000 7f3765400740 20 client.15960 populate_metadata read hostname 'smithi039' 2024-03-18T16:13:00.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.436+0000 7f3765400740 1 --2- 172.21.15.39:0/197054082 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x5558c4ff8930 0x5558c4ffad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:00.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.436+0000 7f3765400740 4 client.15960 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:13:00.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.436+0000 7f3765400740 1 -- 172.21.15.39:0/197054082 --> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5558c4ed6ce0 con 0x5558c4ff8930 2024-03-18T16:13:00.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.437+0000 7f375aa00640 1 --2- 172.21.15.39:0/197054082 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x5558c4ff8930 0x5558c4ffad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:00.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.440+0000 7f375aa00640 1 --2- 172.21.15.39:0/197054082 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x5558c4ff8930 0x5558c4ffad50 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=b52bdcc1d0a7dd1b server_cookie=21d1ab901166ed9d in_seq=0 out_seq=0 2024-03-18T16:13:00.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.440+0000 7f3745600640 10 client.15960 ms_handle_connect on v2:172.21.15.194:6836/1301464246 2024-03-18T16:13:00.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.440+0000 7f3745600640 1 -- 172.21.15.39:0/197054082 <== mds.3 v2:172.21.15.194:6836/1301464246 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5558c4ed6ce0 con 0x5558c4ff8930 2024-03-18T16:13:00.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.440+0000 7f3745600640 10 client.15960 handle_command_reply: tid=0 2024-03-18T16:13:00.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.440+0000 7f3765400740 1 client.15960 shutdown 2024-03-18T16:13:00.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.441+0000 7f3765400740 1 -- 172.21.15.39:0/197054082 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5558c4f6aa60 msgr2=0x5558c4f8a6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:00.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.441+0000 7f3765400740 1 --2- 172.21.15.39:0/197054082 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5558c4f6aa60 0x5558c4f8a6f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f374c004e70 tx=0x7f374c0048f0 comp rx=0 tx=0).stop 2024-03-18T16:13:00.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.441+0000 7f3765400740 1 -- 172.21.15.39:0/197054082 shutdown_connections 2024-03-18T16:13:00.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.441+0000 7f3765400740 1 --2- 172.21.15.39:0/197054082 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x5558c4ff8930 0x5558c4ffad50 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:00.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.441+0000 7f3765400740 1 --2- 172.21.15.39:0/197054082 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5558c4f705c0 0x5558c4f8bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:00.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.441+0000 7f3765400740 1 --2- 172.21.15.39:0/197054082 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5558c4f6b450 0x5558c4f8ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:00.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.441+0000 7f3765400740 1 --2- 172.21.15.39:0/197054082 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5558c4f6aa60 0x5558c4f8a6f0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:00.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.441+0000 7f3765400740 1 -- 172.21.15.39:0/197054082 >> 172.21.15.39:0/197054082 conn(0x5558c4f65fd0 msgr2=0x5558c4f69ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:00.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.441+0000 7f3745600640 0 client.15960 ms_handle_reset on v2:172.21.15.194:6836/1301464246 2024-03-18T16:13:00.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.442+0000 7f3765400740 1 -- 172.21.15.39:0/197054082 shutdown_connections 2024-03-18T16:13:00.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.443+0000 7f3765400740 1 -- 172.21.15.39:0/197054082 wait complete. 2024-03-18T16:13:00.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.443+0000 7f3765400740 20 client.15960 trim_cache size 0 max 16384 2024-03-18T16:13:00.447 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:13:00.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.447+0000 7f3761200640 1 -- 172.21.15.39:0/1376096270 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f37400a2810 msgr2=0x7f37400a4cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:00.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.447+0000 7f3761200640 1 --2- 172.21.15.39:0/1376096270 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f37400a2810 0x7f37400a4cd0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f3754009c70 tx=0x7f3754005fd0 comp rx=0 tx=0).stop 2024-03-18T16:13:00.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.447+0000 7f3761200640 1 -- 172.21.15.39:0/1376096270 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f375c108eb0 msgr2=0x7f375c1a6c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:00.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.447+0000 7f3761200640 1 --2- 172.21.15.39:0/1376096270 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f375c108eb0 0x7f375c1a6c80 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f374800c920 tx=0x7f374800cde0 comp rx=0 tx=0).stop 2024-03-18T16:13:00.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.447+0000 7f3761200640 1 -- 172.21.15.39:0/1376096270 shutdown_connections 2024-03-18T16:13:00.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.447+0000 7f3761200640 1 --2- 172.21.15.39:0/1376096270 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f375c109870 0x7f375c1adcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:00.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.447+0000 7f3761200640 1 --2- 172.21.15.39:0/1376096270 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f37400a2810 0x7f37400a4cd0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:00.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.447+0000 7f3761200640 1 --2- 172.21.15.39:0/1376096270 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f375c108eb0 0x7f375c1a6c80 secure :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f374800c920 tx=0x7f374800cde0 comp rx=0 tx=0).stop 2024-03-18T16:13:00.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.447+0000 7f3761200640 1 --2- 172.21.15.39:0/1376096270 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f375c072bf0 0x7f375c1a6740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:00.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.447+0000 7f3761200640 1 -- 172.21.15.39:0/1376096270 >> 172.21.15.39:0/1376096270 conn(0x7f375c06e6c0 msgr2=0x7f375c10a0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:00.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.450+0000 7f3761200640 1 -- 172.21.15.39:0/1376096270 shutdown_connections 2024-03-18T16:13:00.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.450+0000 7f3761200640 1 -- 172.21.15.39:0/1376096270 wait complete. 2024-03-18T16:13:00.484 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-18T16:13:00.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.824+0000 7f581da00640 1 -- 172.21.15.39:0/2830689133 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f581810b2f0 msgr2=0x7f581810d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:00.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.824+0000 7f581da00640 1 --2- 172.21.15.39:0/2830689133 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f581810b2f0 0x7f581810d6e0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f5808009900 tx=0x7f580802bc40 comp rx=0 tx=0).stop 2024-03-18T16:13:00.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.824+0000 7f581da00640 1 -- 172.21.15.39:0/2830689133 shutdown_connections 2024-03-18T16:13:00.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.824+0000 7f581da00640 1 --2- 172.21.15.39:0/2830689133 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f581810b2f0 0x7f581810d6e0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:00.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.824+0000 7f581da00640 1 --2- 172.21.15.39:0/2830689133 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f581810dd70 0x7f5818110230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:00.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.824+0000 7f581da00640 1 --2- 172.21.15.39:0/2830689133 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f58181089c0 0x7f581810adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:00.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.824+0000 7f581da00640 1 -- 172.21.15.39:0/2830689133 >> 172.21.15.39:0/2830689133 conn(0x7f581806e8b0 msgr2=0x7f5818070cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:00.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.824+0000 7f581da00640 1 -- 172.21.15.39:0/2830689133 shutdown_connections 2024-03-18T16:13:00.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.824+0000 7f581da00640 1 -- 172.21.15.39:0/2830689133 wait complete. 2024-03-18T16:13:00.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.826+0000 7f581da00640 1 Processor -- start 2024-03-18T16:13:00.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.826+0000 7f581da00640 1 -- start start 2024-03-18T16:13:00.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.826+0000 7f581da00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f58181089c0 0x7f5818079970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:00.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.826+0000 7f581da00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f581810b2f0 0x7f5818079eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:00.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.826+0000 7f581da00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f581810dd70 0x7f5818080f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:00.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.826+0000 7f581da00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f58180757e0 con 0x7f581810dd70 2024-03-18T16:13:00.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.826+0000 7f581da00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f5818075660 con 0x7f581810b2f0 2024-03-18T16:13:00.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.826+0000 7f581da00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f5818075960 con 0x7f58181089c0 2024-03-18T16:13:00.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.827+0000 7f5817400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f581810b2f0 0x7f5818079eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:00.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.827+0000 7f5817400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f581810b2f0 0x7f5818079eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:60878/0 (socket says 172.21.15.39:60878) 2024-03-18T16:13:00.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.827+0000 7f5817400640 1 -- 172.21.15.39:0/1697397377 learned_addr learned my addr 172.21.15.39:0/1697397377 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:13:00.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.827+0000 7f581d000640 1 --2- 172.21.15.39:0/1697397377 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f581810dd70 0x7f5818080f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:00.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.827+0000 7f5817e00640 1 --2- 172.21.15.39:0/1697397377 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f58181089c0 0x7f5818079970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:00.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.827+0000 7f5817400640 1 -- 172.21.15.39:0/1697397377 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f58181089c0 msgr2=0x7f5818079970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:00.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.827+0000 7f5817400640 1 --2- 172.21.15.39:0/1697397377 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f58181089c0 0x7f5818079970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:00.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.827+0000 7f5817400640 1 -- 172.21.15.39:0/1697397377 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f581810dd70 msgr2=0x7f5818080f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:00.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.827+0000 7f5817400640 1 --2- 172.21.15.39:0/1697397377 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f581810dd70 0x7f5818080f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:00.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.827+0000 7f5817400640 1 -- 172.21.15.39:0/1697397377 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5818081590 con 0x7f581810b2f0 2024-03-18T16:13:00.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.828+0000 7f5817e00640 1 --2- 172.21.15.39:0/1697397377 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f58181089c0 0x7f5818079970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:13:00.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.828+0000 7f5817400640 1 --2- 172.21.15.39:0/1697397377 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f581810b2f0 0x7f5818079eb0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f580802c250 tx=0x7f5808005d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:00.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.828+0000 7f5814c00640 1 -- 172.21.15.39:0/1697397377 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5808041070 con 0x7f581810b2f0 2024-03-18T16:13:00.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.828+0000 7f581da00640 1 -- 172.21.15.39:0/1697397377 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5818081820 con 0x7f581810b2f0 2024-03-18T16:13:00.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.828+0000 7f581da00640 1 -- 172.21.15.39:0/1697397377 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5818081d30 con 0x7f581810b2f0 2024-03-18T16:13:00.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.828+0000 7f5814c00640 1 -- 172.21.15.39:0/1697397377 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f58080044d0 con 0x7f581810b2f0 2024-03-18T16:13:00.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.828+0000 7f5814c00640 1 -- 172.21.15.39:0/1697397377 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5808034440 con 0x7f581810b2f0 2024-03-18T16:13:00.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.836+0000 7f5821e21740 1 Processor -- start 2024-03-18T16:13:00.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.837+0000 7f5821e21740 1 -- start start 2024-03-18T16:13:00.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.837+0000 7f5821e21740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557ee77efa60 0x557ee77efe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:00.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.837+0000 7f5821e21740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557ee77f0450 0x557ee77f4f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:00.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.837+0000 7f5821e21740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557ee77f55c0 0x557ee77f7a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:00.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.837+0000 7f5821e21740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x557ee77b73b0 con 0x557ee77f55c0 2024-03-18T16:13:00.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.837+0000 7f5821e21740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x557ee7754450 con 0x557ee77f0450 2024-03-18T16:13:00.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.837+0000 7f5821e21740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x557ee76fccb0 con 0x557ee77efa60 2024-03-18T16:13:00.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.838+0000 7f5817e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557ee77f0450 0x557ee77f4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:00.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.838+0000 7f5817e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557ee77f0450 0x557ee77f4f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:60884/0 (socket says 172.21.15.39:60884) 2024-03-18T16:13:00.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.838+0000 7f5817e00640 1 -- 172.21.15.39:0/1781029186 learned_addr learned my addr 172.21.15.39:0/1781029186 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:13:00.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.838+0000 7f581d000640 1 --2- 172.21.15.39:0/1781029186 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557ee77efa60 0x557ee77efe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:00.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.839+0000 7f5817e00640 1 -- 172.21.15.39:0/1781029186 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557ee77efa60 msgr2=0x557ee77efe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:00.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.839+0000 7f5817e00640 1 --2- 172.21.15.39:0/1781029186 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557ee77efa60 0x557ee77efe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:00.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.839+0000 7f5817e00640 1 -- 172.21.15.39:0/1781029186 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557ee77f55c0 msgr2=0x557ee77f7a50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:13:00.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.839+0000 7f5817e00640 1 --2- 172.21.15.39:0/1781029186 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557ee77f55c0 0x557ee77f7a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:00.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.839+0000 7f5817e00640 1 -- 172.21.15.39:0/1781029186 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557ee771e1f0 con 0x557ee77f0450 2024-03-18T16:13:00.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.840+0000 7f5817e00640 1 --2- 172.21.15.39:0/1781029186 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557ee77f0450 0x557ee77f4f50 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f5804007850 tx=0x7f5804007d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=72f84844c8c1f11f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:00.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.841+0000 7f5814c00640 1 -- 172.21.15.39:0/1697397377 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f5808044070 con 0x7f581810b2f0 2024-03-18T16:13:00.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.842+0000 7f580d600640 1 -- 172.21.15.39:0/1781029186 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f580400d950 con 0x557ee77f0450 2024-03-18T16:13:00.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.842+0000 7f580d600640 1 -- 172.21.15.39:0/1781029186 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5804018070 con 0x557ee77f0450 2024-03-18T16:13:00.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.842+0000 7f580d600640 1 -- 172.21.15.39:0/1781029186 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58040123e0 con 0x557ee77f0450 2024-03-18T16:13:00.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.843+0000 7f5814c00640 1 --2- 172.21.15.39:0/1697397377 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f57fc0a2880 0x7f57fc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:00.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.843+0000 7f5821e21740 1 -- 172.21.15.39:0/1781029186 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557ee77f0450 msgr2=0x557ee77f4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:00.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.843+0000 7f5821e21740 1 --2- 172.21.15.39:0/1781029186 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557ee77f0450 0x557ee77f4f50 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f5804007850 tx=0x7f5804007d10 comp rx=0 tx=0).stop 2024-03-18T16:13:00.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.844+0000 7f5814c00640 1 -- 172.21.15.39:0/1697397377 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f58080f0950 con 0x7f581810b2f0 2024-03-18T16:13:00.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.845+0000 7f581d000640 1 --2- 172.21.15.39:0/1697397377 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f57fc0a2880 0x7f57fc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:00.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.845+0000 7f5821e21740 1 -- 172.21.15.39:0/1781029186 shutdown_connections 2024-03-18T16:13:00.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.845+0000 7f5821e21740 1 --2- 172.21.15.39:0/1781029186 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557ee77f0450 0x557ee77f4f50 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:00.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.845+0000 7f5821e21740 1 --2- 172.21.15.39:0/1781029186 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557ee77f55c0 0x557ee77f7a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:00.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.845+0000 7f5821e21740 1 --2- 172.21.15.39:0/1781029186 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557ee77efa60 0x557ee77efe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:00.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.845+0000 7f5821e21740 1 -- 172.21.15.39:0/1781029186 >> 172.21.15.39:0/1781029186 conn(0x557ee77eafd0 msgr2=0x557ee77eb3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:00.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.846+0000 7f581d000640 1 --2- 172.21.15.39:0/1697397377 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f57fc0a2880 0x7f57fc0a4d40 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x557ee77f0cb0 tx=0x7f5810009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:00.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.846+0000 7f5821e21740 1 -- 172.21.15.39:0/1781029186 shutdown_connections 2024-03-18T16:13:00.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.846+0000 7f5821e21740 1 -- 172.21.15.39:0/1781029186 wait complete. 2024-03-18T16:13:00.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.847+0000 7f5821e21740 1 -- start start 2024-03-18T16:13:00.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.847+0000 7f5821e21740 1 Processor -- start 2024-03-18T16:13:00.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.848+0000 7f5821e21740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557ee77efa60 0x557ee77f9e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:00.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.848+0000 7f5821e21740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557ee77f0450 0x557ee77fa340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:00.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.848+0000 7f5821e21740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557ee77f55c0 0x557ee7801380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:00.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.848+0000 7f5821e21740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x557ee7710110 con 0x557ee77f0450 2024-03-18T16:13:00.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.848+0000 7f5821e21740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x557ee77504f0 con 0x557ee77efa60 2024-03-18T16:13:00.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.848+0000 7f5821e21740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x557ee77f8570 con 0x557ee77f55c0 2024-03-18T16:13:00.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.848+0000 7f5821e21740 10 client.0 _ll_register_callbacks cb 0x557ee76fdb70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:13:00.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.849+0000 7f5817e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557ee77efa60 0x557ee77f9e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:00.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.849+0000 7f5817e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557ee77efa60 0x557ee77f9e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:60896/0 (socket says 172.21.15.39:60896) 2024-03-18T16:13:00.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.849+0000 7f5817e00640 1 -- 172.21.15.39:0/3419240504 learned_addr learned my addr 172.21.15.39:0/3419240504 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:13:00.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.849+0000 7f581d000640 1 --2- 172.21.15.39:0/3419240504 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557ee77f55c0 0x557ee7801380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:00.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.849+0000 7f5817400640 1 --2- 172.21.15.39:0/3419240504 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557ee77f0450 0x557ee77fa340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:00.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.850+0000 7f5817400640 1 -- 172.21.15.39:0/3419240504 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557ee77f55c0 msgr2=0x557ee7801380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:00.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.850+0000 7f5817400640 1 --2- 172.21.15.39:0/3419240504 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557ee77f55c0 0x557ee7801380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:00.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.850+0000 7f5817400640 1 -- 172.21.15.39:0/3419240504 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557ee77efa60 msgr2=0x557ee77f9e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:00.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.850+0000 7f5817400640 1 --2- 172.21.15.39:0/3419240504 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557ee77efa60 0x557ee77f9e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:00.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.850+0000 7f5817400640 1 -- 172.21.15.39:0/3419240504 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557ee74683f0 con 0x557ee77f0450 2024-03-18T16:13:00.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.851+0000 7f5817400640 1 --2- 172.21.15.39:0/3419240504 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557ee77f0450 0x557ee77fa340 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f58080ee0c0 tx=0x7f58080b92c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9e29eccf2c83e3e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:00.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.851+0000 7f5802000640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-03-18T16:13:00.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.852+0000 7f5802000640 1 -- 172.21.15.39:0/3419240504 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f580804b040 con 0x557ee77f0450 2024-03-18T16:13:00.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.852+0000 7f5802000640 1 -- 172.21.15.39:0/3419240504 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f58080035f0 con 0x557ee77f0450 2024-03-18T16:13:00.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.853+0000 7f5821e21740 1 -- 172.21.15.39:0/3419240504 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5804018070 con 0x557ee77f0450 2024-03-18T16:13:00.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.854+0000 7f5802000640 1 -- 172.21.15.39:0/3419240504 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f58080bd660 con 0x557ee77f0450 2024-03-18T16:13:00.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.854+0000 7f5821e21740 10 client.15966 fetch_fsmap learned FSMap version 35 2024-03-18T16:13:00.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.854+0000 7f5821e21740 1 -- 172.21.15.39:0/3419240504 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x557ee75327d0 con 0x557ee77f0450 2024-03-18T16:13:00.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.858+0000 7f5802000640 1 -- 172.21.15.39:0/3419240504 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5804018070 con 0x557ee77f0450 2024-03-18T16:13:00.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.859+0000 7f5802000640 1 -- 172.21.15.39:0/3419240504 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 35) ==== 8402+0+0 (secure 0 0 0) 0x7f58080c5300 con 0x557ee77f0450 2024-03-18T16:13:00.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.859+0000 7f5821e21740 10 client.15966 fetch_fsmap finished waiting for FSMap version 35 2024-03-18T16:13:00.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.859+0000 7f5821e21740 10 client.15966 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-18T16:13:00.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.859+0000 7f5821e21740 20 client.15966 populate_metadata read hostname 'smithi039' 2024-03-18T16:13:00.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.859+0000 7f5821e21740 1 --2- 172.21.15.39:0/3419240504 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x557ee7805d40 0x557ee7808160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:00.863 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.860+0000 7f5821e21740 4 client.15966 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:13:00.863 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.860+0000 7f5821e21740 1 -- 172.21.15.39:0/3419240504 --> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557ee7761990 con 0x557ee7805d40 2024-03-18T16:13:00.863 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.860+0000 7f5817e00640 1 --2- 172.21.15.39:0/3419240504 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x557ee7805d40 0x557ee7808160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:00.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.861+0000 7f5817e00640 1 --2- 172.21.15.39:0/3419240504 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x557ee7805d40 0x557ee7808160 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=dc5ed2cab94a4237 server_cookie=894a4400b0484f4 in_seq=0 out_seq=0 2024-03-18T16:13:00.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.862+0000 7f5802000640 10 client.15966 ms_handle_connect on v2:172.21.15.196:6836/895247251 2024-03-18T16:13:00.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.863+0000 7f5802000640 1 -- 172.21.15.39:0/3419240504 <== mds.2 v2:172.21.15.196:6836/895247251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557ee7761990 con 0x557ee7805d40 2024-03-18T16:13:00.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.863+0000 7f5802000640 10 client.15966 handle_command_reply: tid=0 2024-03-18T16:13:00.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.863+0000 7f5821e21740 1 client.15966 shutdown 2024-03-18T16:13:00.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.863+0000 7f5821e21740 1 -- 172.21.15.39:0/3419240504 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557ee77f0450 msgr2=0x557ee77fa340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:00.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.863+0000 7f5821e21740 1 --2- 172.21.15.39:0/3419240504 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557ee77f0450 0x557ee77fa340 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f58080ee0c0 tx=0x7f58080b92c0 comp rx=0 tx=0).stop 2024-03-18T16:13:00.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.864+0000 7f5821e21740 1 -- 172.21.15.39:0/3419240504 shutdown_connections 2024-03-18T16:13:00.867 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.864+0000 7f5821e21740 1 --2- 172.21.15.39:0/3419240504 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x557ee7805d40 0x557ee7808160 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:00.867 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.864+0000 7f5821e21740 1 --2- 172.21.15.39:0/3419240504 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557ee77f55c0 0x557ee7801380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:00.867 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.864+0000 7f5821e21740 1 --2- 172.21.15.39:0/3419240504 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557ee77f0450 0x557ee77fa340 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:00.867 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.864+0000 7f5821e21740 1 --2- 172.21.15.39:0/3419240504 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557ee77efa60 0x557ee77f9e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:00.867 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.864+0000 7f5821e21740 1 -- 172.21.15.39:0/3419240504 >> 172.21.15.39:0/3419240504 conn(0x557ee77eafd0 msgr2=0x557ee77ea1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:00.867 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.864+0000 7f5802000640 0 client.15966 ms_handle_reset on v2:172.21.15.196:6836/895247251 2024-03-18T16:13:00.868 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.865+0000 7f5821e21740 1 -- 172.21.15.39:0/3419240504 shutdown_connections 2024-03-18T16:13:00.868 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.866+0000 7f5821e21740 1 -- 172.21.15.39:0/3419240504 wait complete. 2024-03-18T16:13:00.872 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.867+0000 7f5821e21740 20 client.15966 trim_cache size 0 max 16384 2024-03-18T16:13:00.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.934+0000 7f5821e21740 1 Processor -- start 2024-03-18T16:13:00.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.934+0000 7f5821e21740 1 -- start start 2024-03-18T16:13:00.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.936+0000 7f5821e21740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557ee77eafd0 0x557ee77f9e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:00.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.936+0000 7f5821e21740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557ee77efa60 0x557ee77fa340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:00.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.936+0000 7f5821e21740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557ee77f0450 0x557ee78012a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:00.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.936+0000 7f5821e21740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f58181c1810 con 0x557ee77efa60 2024-03-18T16:13:00.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.936+0000 7f5821e21740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f58181c34d0 con 0x557ee77eafd0 2024-03-18T16:13:00.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.936+0000 7f5821e21740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f58181c1ee0 con 0x557ee77f0450 2024-03-18T16:13:00.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.936+0000 7f581d000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557ee77f0450 0x557ee78012a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:00.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.936+0000 7f581d000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557ee77f0450 0x557ee78012a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:37542/0 (socket says 172.21.15.39:37542) 2024-03-18T16:13:00.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.937+0000 7f581d000640 1 -- 172.21.15.39:0/233545907 learned_addr learned my addr 172.21.15.39:0/233545907 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:13:00.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.937+0000 7f5817400640 1 --2- 172.21.15.39:0/233545907 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557ee77efa60 0x557ee77fa340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:00.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.937+0000 7f5817e00640 1 --2- 172.21.15.39:0/233545907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557ee77eafd0 0x557ee77f9e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:00.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.937+0000 7f581d000640 1 -- 172.21.15.39:0/233545907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557ee77eafd0 msgr2=0x557ee77f9e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:00.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.937+0000 7f581d000640 1 --2- 172.21.15.39:0/233545907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557ee77eafd0 0x557ee77f9e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:00.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.937+0000 7f581d000640 1 -- 172.21.15.39:0/233545907 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557ee77efa60 msgr2=0x557ee77fa340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:00.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.937+0000 7f581d000640 1 --2- 172.21.15.39:0/233545907 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557ee77efa60 0x557ee77fa340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:00.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.937+0000 7f581d000640 1 -- 172.21.15.39:0/233545907 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557ee77ea920 con 0x557ee77f0450 2024-03-18T16:13:00.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.937+0000 7f5817400640 1 --2- 172.21.15.39:0/233545907 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557ee77efa60 0x557ee77fa340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:13:00.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.938+0000 7f581d000640 1 --2- 172.21.15.39:0/233545907 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557ee77f0450 0x557ee78012a0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f5810003df0 tx=0x7f581001b000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ec4d89a77bde4571 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:00.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.939+0000 7f580cc00640 1 -- 172.21.15.39:0/233545907 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f581002e070 con 0x557ee77f0450 2024-03-18T16:13:00.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.939+0000 7f580cc00640 1 -- 172.21.15.39:0/233545907 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f581000fa40 con 0x557ee77f0450 2024-03-18T16:13:00.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.939+0000 7f580cc00640 1 -- 172.21.15.39:0/233545907 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f581001ecf0 con 0x557ee77f0450 2024-03-18T16:13:00.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.946+0000 7f5821e21740 1 -- 172.21.15.39:0/233545907 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557ee77f0450 msgr2=0x557ee78012a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:00.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.946+0000 7f5821e21740 1 --2- 172.21.15.39:0/233545907 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557ee77f0450 0x557ee78012a0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f5810003df0 tx=0x7f581001b000 comp rx=0 tx=0).stop 2024-03-18T16:13:00.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.947+0000 7f5821e21740 1 -- 172.21.15.39:0/233545907 shutdown_connections 2024-03-18T16:13:00.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.947+0000 7f5821e21740 1 --2- 172.21.15.39:0/233545907 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557ee77f0450 0x557ee78012a0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:00.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.947+0000 7f5821e21740 1 --2- 172.21.15.39:0/233545907 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557ee77efa60 0x557ee77fa340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:00.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.947+0000 7f5821e21740 1 --2- 172.21.15.39:0/233545907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557ee77eafd0 0x557ee77f9e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:00.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.947+0000 7f5821e21740 1 -- 172.21.15.39:0/233545907 >> 172.21.15.39:0/233545907 conn(0x557ee77f1230 msgr2=0x557ee77f8970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:00.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.954+0000 7f5821e21740 1 -- 172.21.15.39:0/233545907 shutdown_connections 2024-03-18T16:13:00.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.954+0000 7f5821e21740 1 -- 172.21.15.39:0/233545907 wait complete. 2024-03-18T16:13:00.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.955+0000 7f5821e21740 1 -- start start 2024-03-18T16:13:00.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.955+0000 7f5821e21740 1 Processor -- start 2024-03-18T16:13:00.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.956+0000 7f5821e21740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557ee77eafd0 0x557ee77fe0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:00.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.956+0000 7f5821e21740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557ee77efa60 0x557ee77fe610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:00.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.956+0000 7f5821e21740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557ee77f0450 0x557ee786a000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:00.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.956+0000 7f5821e21740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f58181c2d40 con 0x557ee77f0450 2024-03-18T16:13:00.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.956+0000 7f5821e21740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f58181c25b0 con 0x557ee77efa60 2024-03-18T16:13:00.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.956+0000 7f5821e21740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f58181bf7b0 con 0x557ee77eafd0 2024-03-18T16:13:00.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.956+0000 7f5821e21740 10 client.0 _ll_register_callbacks cb 0x7f5808009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:13:00.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.957+0000 7f5817e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557ee77eafd0 0x557ee77fe0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:00.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.957+0000 7f5817e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557ee77eafd0 0x557ee77fe0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:37550/0 (socket says 172.21.15.39:37550) 2024-03-18T16:13:00.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.957+0000 7f5817e00640 1 -- 172.21.15.39:0/1182105881 learned_addr learned my addr 172.21.15.39:0/1182105881 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:13:00.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.957+0000 7f581d000640 1 --2- 172.21.15.39:0/1182105881 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557ee77f0450 0x557ee786a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:00.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.957+0000 7f5817e00640 1 -- 172.21.15.39:0/1182105881 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557ee77efa60 msgr2=0x557ee77fe610 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T16:13:00.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.957+0000 7f5817e00640 1 --2- 172.21.15.39:0/1182105881 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557ee77efa60 0x557ee77fe610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:00.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.957+0000 7f5817e00640 1 -- 172.21.15.39:0/1182105881 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557ee77f0450 msgr2=0x557ee786a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:00.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.957+0000 7f5817e00640 1 --2- 172.21.15.39:0/1182105881 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557ee77f0450 0x557ee786a000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:00.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.957+0000 7f5817e00640 1 -- 172.21.15.39:0/1182105881 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557ee786a540 con 0x557ee77eafd0 2024-03-18T16:13:00.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.958+0000 7f5817e00640 1 --2- 172.21.15.39:0/1182105881 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557ee77eafd0 0x557ee77fe0d0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f5804006f40 tx=0x7f5804007f60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aba5db8f7d50f22 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:00.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.958+0000 7f5802000640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:13:00.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.958+0000 7f5802000640 1 -- 172.21.15.39:0/1182105881 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58040136c0 con 0x557ee77eafd0 2024-03-18T16:13:00.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.959+0000 7f5802000640 1 -- 172.21.15.39:0/1182105881 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f5804026b10 con 0x557ee77eafd0 2024-03-18T16:13:00.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.960+0000 7f5821e21740 1 -- 172.21.15.39:0/1182105881 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f581000fa40 con 0x557ee77eafd0 2024-03-18T16:13:00.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.962+0000 7f5802000640 1 -- 172.21.15.39:0/1182105881 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f5804035050 con 0x557ee77eafd0 2024-03-18T16:13:00.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.962+0000 7f5821e21740 10 client.26206 fetch_fsmap learned FSMap version 35 2024-03-18T16:13:00.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.962+0000 7f5821e21740 1 -- 172.21.15.39:0/1182105881 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x557ee786b740 con 0x557ee77eafd0 2024-03-18T16:13:00.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.963+0000 7f5802000640 1 -- 172.21.15.39:0/1182105881 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f581000fa40 con 0x557ee77eafd0 2024-03-18T16:13:00.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.964+0000 7f5802000640 1 -- 172.21.15.39:0/1182105881 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 35) ==== 8402+0+0 (secure 0 0 0) 0x7f580400dc50 con 0x557ee77eafd0 2024-03-18T16:13:00.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.964+0000 7f5821e21740 10 client.26206 fetch_fsmap finished waiting for FSMap version 35 2024-03-18T16:13:00.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.964+0000 7f5821e21740 10 client.26206 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-18T16:13:00.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.964+0000 7f5821e21740 20 client.26206 populate_metadata read hostname 'smithi039' 2024-03-18T16:13:00.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.964+0000 7f5821e21740 1 --2- 172.21.15.39:0/1182105881 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x557ee77f55c0 0x557ee786de30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:00.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.965+0000 7f5817400640 1 --2- 172.21.15.39:0/1182105881 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x557ee77f55c0 0x557ee786de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:00.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.967+0000 7f5821e21740 4 client.26206 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:13:00.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.967+0000 7f5821e21740 1 -- 172.21.15.39:0/1182105881 --> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557ee775bce0 con 0x557ee77f55c0 2024-03-18T16:13:00.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.967+0000 7f5817400640 1 --2- 172.21.15.39:0/1182105881 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x557ee77f55c0 0x557ee786de30 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=86e3ca200e492328 server_cookie=44f1dbf3b1787c34 in_seq=0 out_seq=0 2024-03-18T16:13:00.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.969+0000 7f5802000640 10 client.26206 ms_handle_connect on v2:172.21.15.196:6836/895247251 2024-03-18T16:13:00.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.969+0000 7f5802000640 1 -- 172.21.15.39:0/1182105881 <== mds.2 v2:172.21.15.196:6836/895247251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557ee775bce0 con 0x557ee77f55c0 2024-03-18T16:13:00.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.969+0000 7f5802000640 10 client.26206 handle_command_reply: tid=0 2024-03-18T16:13:00.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.969+0000 7f5821e21740 1 client.26206 shutdown 2024-03-18T16:13:00.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.969+0000 7f5821e21740 1 -- 172.21.15.39:0/1182105881 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557ee77eafd0 msgr2=0x557ee77fe0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:00.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.969+0000 7f5821e21740 1 --2- 172.21.15.39:0/1182105881 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557ee77eafd0 0x557ee77fe0d0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f5804006f40 tx=0x7f5804007f60 comp rx=0 tx=0).stop 2024-03-18T16:13:00.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.969+0000 7f5821e21740 1 -- 172.21.15.39:0/1182105881 shutdown_connections 2024-03-18T16:13:00.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.969+0000 7f5821e21740 1 --2- 172.21.15.39:0/1182105881 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557ee77efa60 0x557ee77fe610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:00.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.969+0000 7f5821e21740 1 --2- 172.21.15.39:0/1182105881 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x557ee77f55c0 0x557ee786de30 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:00.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.969+0000 7f5821e21740 1 --2- 172.21.15.39:0/1182105881 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557ee77f0450 0x557ee786a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:00.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.970+0000 7f5821e21740 1 --2- 172.21.15.39:0/1182105881 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557ee77eafd0 0x557ee77fe0d0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:00.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.970+0000 7f5821e21740 1 -- 172.21.15.39:0/1182105881 >> 172.21.15.39:0/1182105881 conn(0x557ee77f1230 msgr2=0x557ee77e9960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:00.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.973+0000 7f5802000640 0 client.26206 ms_handle_reset on v2:172.21.15.196:6836/895247251 2024-03-18T16:13:00.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.973+0000 7f5821e21740 1 -- 172.21.15.39:0/1182105881 shutdown_connections 2024-03-18T16:13:00.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.974+0000 7f5821e21740 1 -- 172.21.15.39:0/1182105881 wait complete. 2024-03-18T16:13:00.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.974+0000 7f5821e21740 20 client.26206 trim_cache size 0 max 16384 2024-03-18T16:13:00.984 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:13:00.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.982+0000 7f581da00640 1 -- 172.21.15.39:0/1697397377 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f57fc0a2880 msgr2=0x7f57fc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:00.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.982+0000 7f581da00640 1 --2- 172.21.15.39:0/1697397377 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f57fc0a2880 0x7f57fc0a4d40 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x557ee77f0cb0 tx=0x7f5810009500 comp rx=0 tx=0).stop 2024-03-18T16:13:00.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.982+0000 7f581da00640 1 -- 172.21.15.39:0/1697397377 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f581810b2f0 msgr2=0x7f5818079eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:00.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.982+0000 7f581da00640 1 --2- 172.21.15.39:0/1697397377 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f581810b2f0 0x7f5818079eb0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f580802c250 tx=0x7f5808005d50 comp rx=0 tx=0).stop 2024-03-18T16:13:00.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.983+0000 7f581da00640 1 -- 172.21.15.39:0/1697397377 shutdown_connections 2024-03-18T16:13:00.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.983+0000 7f581da00640 1 --2- 172.21.15.39:0/1697397377 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f581810b2f0 0x7f5818079eb0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:00.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.983+0000 7f581da00640 1 --2- 172.21.15.39:0/1697397377 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f57fc0a2880 0x7f57fc0a4d40 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:00.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.983+0000 7f581da00640 1 --2- 172.21.15.39:0/1697397377 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f581810dd70 0x7f5818080f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:00.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.983+0000 7f581da00640 1 --2- 172.21.15.39:0/1697397377 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f58181089c0 0x7f5818079970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:00.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.983+0000 7f581da00640 1 -- 172.21.15.39:0/1697397377 >> 172.21.15.39:0/1697397377 conn(0x7f581806e8b0 msgr2=0x7f5818083ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:00.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.983+0000 7f581da00640 1 -- 172.21.15.39:0/1697397377 shutdown_connections 2024-03-18T16:13:00.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:00.983+0000 7f581da00640 1 -- 172.21.15.39:0/1697397377 wait complete. 2024-03-18T16:13:01.015 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-18T16:13:01.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.349+0000 7ff3d4a00640 1 -- 172.21.15.39:0/2508538616 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff3d0100050 msgr2=0x7ff3d0102480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:01.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.349+0000 7ff3d4a00640 1 --2- 172.21.15.39:0/2508538616 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff3d0100050 0x7ff3d0102480 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7ff3c0009ca0 tx=0x7ff3c002c140 comp rx=0 tx=0).stop 2024-03-18T16:13:01.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.349+0000 7ff3d4a00640 1 -- 172.21.15.39:0/2508538616 shutdown_connections 2024-03-18T16:13:01.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.349+0000 7ff3d4a00640 1 --2- 172.21.15.39:0/2508538616 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff3d0105370 0x7ff3d01057f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:01.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.349+0000 7ff3d4a00640 1 --2- 172.21.15.39:0/2508538616 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff3d01029c0 0x7ff3d0104e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:01.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.349+0000 7ff3d4a00640 1 --2- 172.21.15.39:0/2508538616 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff3d0100050 0x7ff3d0102480 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:01.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.349+0000 7ff3d4a00640 1 -- 172.21.15.39:0/2508538616 >> 172.21.15.39:0/2508538616 conn(0x7ff3d00fbc50 msgr2=0x7ff3d00fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:01.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.349+0000 7ff3d4a00640 1 -- 172.21.15.39:0/2508538616 shutdown_connections 2024-03-18T16:13:01.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.351+0000 7ff3d4a00640 1 -- 172.21.15.39:0/2508538616 wait complete. 2024-03-18T16:13:01.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.352+0000 7ff3d4a00640 1 Processor -- start 2024-03-18T16:13:01.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.353+0000 7ff3d4a00640 1 -- start start 2024-03-18T16:13:01.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.354+0000 7ff3d4a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff3d01029c0 0x7ff3d019e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:01.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.354+0000 7ff3d4a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff3d0105370 0x7ff3d019e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:01.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.354+0000 7ff3d4a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff3d01a3850 0x7ff3d01a5c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:01.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.354+0000 7ff3d4a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7ff3d0113a30 con 0x7ff3d0105370 2024-03-18T16:13:01.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.354+0000 7ff3ce800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff3d01a3850 0x7ff3d01a5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:01.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.354+0000 7ff3ce800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff3d01a3850 0x7ff3d01a5c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:60928/0 (socket says 172.21.15.39:60928) 2024-03-18T16:13:01.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.354+0000 7ff3ce800640 1 -- 172.21.15.39:0/2699381481 learned_addr learned my addr 172.21.15.39:0/2699381481 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:13:01.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.354+0000 7ff3cde00640 1 --2- 172.21.15.39:0/2699381481 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff3d01029c0 0x7ff3d019e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:01.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.354+0000 7ff3cd400640 1 --2- 172.21.15.39:0/2699381481 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff3d0105370 0x7ff3d019e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:01.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.355+0000 7ff3d4a00640 1 -- 172.21.15.39:0/2699381481 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ff3d01138b0 con 0x7ff3d01a3850 2024-03-18T16:13:01.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.355+0000 7ff3d4a00640 1 -- 172.21.15.39:0/2699381481 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7ff3d0113bb0 con 0x7ff3d01029c0 2024-03-18T16:13:01.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.356+0000 7ff3ce800640 1 -- 172.21.15.39:0/2699381481 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff3d01029c0 msgr2=0x7ff3d019e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:01.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.356+0000 7ff3ce800640 1 --2- 172.21.15.39:0/2699381481 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff3d01029c0 0x7ff3d019e2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:01.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.356+0000 7ff3ce800640 1 -- 172.21.15.39:0/2699381481 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff3d0105370 msgr2=0x7ff3d019e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:01.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.356+0000 7ff3ce800640 1 --2- 172.21.15.39:0/2699381481 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff3d0105370 0x7ff3d019e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:01.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.356+0000 7ff3ce800640 1 -- 172.21.15.39:0/2699381481 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3d01a62b0 con 0x7ff3d01a3850 2024-03-18T16:13:01.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.356+0000 7ff3cde00640 1 --2- 172.21.15.39:0/2699381481 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff3d01029c0 0x7ff3d019e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:13:01.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.356+0000 7ff3cd400640 1 --2- 172.21.15.39:0/2699381481 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff3d0105370 0x7ff3d019e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:13:01.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.356+0000 7ff3ce800640 1 --2- 172.21.15.39:0/2699381481 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff3d01a3850 0x7ff3d01a5c40 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7ff3c800e970 tx=0x7ff3c800ed70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:01.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.357+0000 7ff3c6a00640 1 -- 172.21.15.39:0/2699381481 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3c800ccc0 con 0x7ff3d01a3850 2024-03-18T16:13:01.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.357+0000 7ff3c6a00640 1 -- 172.21.15.39:0/2699381481 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff3c800ce60 con 0x7ff3d01a3850 2024-03-18T16:13:01.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.357+0000 7ff3c6a00640 1 -- 172.21.15.39:0/2699381481 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3c8010490 con 0x7ff3d01a3850 2024-03-18T16:13:01.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.357+0000 7ff3d4a00640 1 -- 172.21.15.39:0/2699381481 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3d01a65a0 con 0x7ff3d01a3850 2024-03-18T16:13:01.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.357+0000 7ff3d4a00640 1 -- 172.21.15.39:0/2699381481 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3d01a6aa0 con 0x7ff3d01a3850 2024-03-18T16:13:01.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.361+0000 7ff3d77ed740 1 Processor -- start 2024-03-18T16:13:01.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.362+0000 7ff3c6a00640 1 -- 172.21.15.39:0/2699381481 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7ff3c8005020 con 0x7ff3d01a3850 2024-03-18T16:13:01.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.362+0000 7ff3d77ed740 1 -- start start 2024-03-18T16:13:01.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.362+0000 7ff3d77ed740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5603ea6f68c0 0x5603ea6f6ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:01.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.363+0000 7ff3d77ed740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5603ea6f72b0 0x5603ea6fbdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:01.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.363+0000 7ff3d77ed740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5603ea6fc420 0x5603ea6fe8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:01.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.363+0000 7ff3d77ed740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5603ea6be3b0 con 0x5603ea6fc420 2024-03-18T16:13:01.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.363+0000 7ff3d77ed740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5603ea65b450 con 0x5603ea6f72b0 2024-03-18T16:13:01.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.363+0000 7ff3d77ed740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x5603ea603cb0 con 0x5603ea6f68c0 2024-03-18T16:13:01.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.363+0000 7ff3cd400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5603ea6f68c0 0x5603ea6f6ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:01.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.363+0000 7ff3cd400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5603ea6f68c0 0x5603ea6f6ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:37584/0 (socket says 172.21.15.39:37584) 2024-03-18T16:13:01.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.363+0000 7ff3cd400640 1 -- 172.21.15.39:0/441841573 learned_addr learned my addr 172.21.15.39:0/441841573 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:13:01.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.363+0000 7ff3cd400640 1 --2- 172.21.15.39:0/441841573 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5603ea6fc420 0x5603ea6fe8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:01.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.363+0000 7ff3cd400640 1 -- 172.21.15.39:0/441841573 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5603ea6f72b0 msgr2=0x5603ea6fbdb0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T16:13:01.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.364+0000 7ff3cd400640 1 --2- 172.21.15.39:0/441841573 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5603ea6f72b0 0x5603ea6fbdb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:01.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.364+0000 7ff3cd400640 1 -- 172.21.15.39:0/441841573 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5603ea6fc420 msgr2=0x5603ea6fe8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:01.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.364+0000 7ff3cd400640 1 --2- 172.21.15.39:0/441841573 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5603ea6fc420 0x5603ea6fe8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:01.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.364+0000 7ff3cd400640 1 -- 172.21.15.39:0/441841573 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5603ea6251f0 con 0x5603ea6f68c0 2024-03-18T16:13:01.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.364+0000 7ff3cd400640 1 --2- 172.21.15.39:0/441841573 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5603ea6f68c0 0x5603ea6f6ce0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc0058c0 tx=0x7ff3bc009770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=243e3e2053c8fdb9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:01.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.364+0000 7ff3bb000640 1 -- 172.21.15.39:0/441841573 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3bc004030 con 0x5603ea6f68c0 2024-03-18T16:13:01.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.364+0000 7ff3bb000640 1 -- 172.21.15.39:0/441841573 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff3bc00fda0 con 0x5603ea6f68c0 2024-03-18T16:13:01.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.364+0000 7ff3bb000640 1 -- 172.21.15.39:0/441841573 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3bc003cc0 con 0x5603ea6f68c0 2024-03-18T16:13:01.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.365+0000 7ff3c6a00640 1 --2- 172.21.15.39:0/2699381481 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff3b40a2880 0x7ff3b40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:01.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.365+0000 7ff3c6a00640 1 -- 172.21.15.39:0/2699381481 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7ff3c80cf200 con 0x7ff3d01a3850 2024-03-18T16:13:01.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.365+0000 7ff3d77ed740 1 -- 172.21.15.39:0/441841573 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5603ea6f68c0 msgr2=0x5603ea6f6ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:01.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.365+0000 7ff3d77ed740 1 --2- 172.21.15.39:0/441841573 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5603ea6f68c0 0x5603ea6f6ce0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc0058c0 tx=0x7ff3bc009770 comp rx=0 tx=0).stop 2024-03-18T16:13:01.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.366+0000 7ff3d77ed740 1 -- 172.21.15.39:0/441841573 shutdown_connections 2024-03-18T16:13:01.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.366+0000 7ff3d77ed740 1 --2- 172.21.15.39:0/441841573 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5603ea6f72b0 0x5603ea6fbdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:01.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.366+0000 7ff3d77ed740 1 --2- 172.21.15.39:0/441841573 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5603ea6fc420 0x5603ea6fe8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:01.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.366+0000 7ff3d77ed740 1 --2- 172.21.15.39:0/441841573 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5603ea6f68c0 0x5603ea6f6ce0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:01.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.366+0000 7ff3d77ed740 1 -- 172.21.15.39:0/441841573 >> 172.21.15.39:0/441841573 conn(0x5603ea6f1e30 msgr2=0x5603ea6f2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:01.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.366+0000 7ff3d77ed740 1 -- 172.21.15.39:0/441841573 shutdown_connections 2024-03-18T16:13:01.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.366+0000 7ff3cde00640 1 --2- 172.21.15.39:0/2699381481 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff3b40a2880 0x7ff3b40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:01.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.367+0000 7ff3d77ed740 1 -- 172.21.15.39:0/441841573 wait complete. 2024-03-18T16:13:01.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.367+0000 7ff3cde00640 1 --2- 172.21.15.39:0/2699381481 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff3b40a2880 0x7ff3b40a4d40 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7ff3c0009bf0 tx=0x7ff3c002e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:01.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.368+0000 7ff3d77ed740 1 -- start start 2024-03-18T16:13:01.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.368+0000 7ff3d77ed740 1 Processor -- start 2024-03-18T16:13:01.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.369+0000 7ff3d77ed740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5603ea6f68c0 0x5603ea700bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:01.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.369+0000 7ff3d77ed740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5603ea6f72b0 0x5603ea701110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:01.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.369+0000 7ff3d77ed740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5603ea6fc420 0x5603ea708150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:01.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.369+0000 7ff3d77ed740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5603ea617110 con 0x5603ea6f68c0 2024-03-18T16:13:01.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.369+0000 7ff3d77ed740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5603ea6574f0 con 0x5603ea6f72b0 2024-03-18T16:13:01.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.369+0000 7ff3d77ed740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x5603ea6ff3d0 con 0x5603ea6fc420 2024-03-18T16:13:01.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.369+0000 7ff3d77ed740 10 client.0 _ll_register_callbacks cb 0x5603ea604b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:13:01.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.369+0000 7ff3cd400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5603ea6f72b0 0x5603ea701110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:01.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.369+0000 7ff3cd400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5603ea6f72b0 0x5603ea701110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:60954/0 (socket says 172.21.15.39:60954) 2024-03-18T16:13:01.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.369+0000 7ff3cd400640 1 -- 172.21.15.39:0/3467358568 learned_addr learned my addr 172.21.15.39:0/3467358568 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:13:01.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.369+0000 7ff3cd400640 1 -- 172.21.15.39:0/3467358568 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5603ea6fc420 msgr2=0x5603ea708150 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T16:13:01.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.369+0000 7ff3cd400640 1 --2- 172.21.15.39:0/3467358568 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5603ea6fc420 0x5603ea708150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:01.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.369+0000 7ff3cd400640 1 -- 172.21.15.39:0/3467358568 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5603ea6f68c0 msgr2=0x5603ea700bd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:13:01.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.370+0000 7ff3cde00640 1 --2- 172.21.15.39:0/3467358568 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5603ea6f68c0 0x5603ea700bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:01.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.370+0000 7ff3cd400640 1 --2- 172.21.15.39:0/3467358568 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5603ea6f68c0 0x5603ea700bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:01.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.370+0000 7ff3cd400640 1 -- 172.21.15.39:0/3467358568 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5603ea36f3f0 con 0x5603ea6f72b0 2024-03-18T16:13:01.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.370+0000 7ff3cde00640 1 --2- 172.21.15.39:0/3467358568 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5603ea6f68c0 0x5603ea700bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:13:01.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.370+0000 7ff3cd400640 1 --2- 172.21.15.39:0/3467358568 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5603ea6f72b0 0x5603ea701110 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc0094b0 tx=0x7ff3bc00ffc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=681079a883061732 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:01.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.370+0000 7ff3af400640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:13:01.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.370+0000 7ff3af400640 1 -- 172.21.15.39:0/3467358568 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3bc005450 con 0x5603ea6f72b0 2024-03-18T16:13:01.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.370+0000 7ff3af400640 1 -- 172.21.15.39:0/3467358568 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7ff3bc0055f0 con 0x5603ea6f72b0 2024-03-18T16:13:01.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.370+0000 7ff3d77ed740 1 -- 172.21.15.39:0/3467358568 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff3bc00fda0 con 0x5603ea6f72b0 2024-03-18T16:13:01.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.371+0000 7ff3af400640 1 -- 172.21.15.39:0/3467358568 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7ff3bc0046c0 con 0x5603ea6f72b0 2024-03-18T16:13:01.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.371+0000 7ff3d77ed740 10 client.26207 fetch_fsmap learned FSMap version 35 2024-03-18T16:13:01.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.371+0000 7ff3d77ed740 1 -- 172.21.15.39:0/3467358568 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5603ea4397d0 con 0x5603ea6f72b0 2024-03-18T16:13:01.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.371+0000 7ff3af400640 1 -- 172.21.15.39:0/3467358568 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff3bc00fda0 con 0x5603ea6f72b0 2024-03-18T16:13:01.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.371+0000 7ff3af400640 1 -- 172.21.15.39:0/3467358568 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 35) ==== 8402+0+0 (secure 0 0 0) 0x7ff3bc026020 con 0x5603ea6f72b0 2024-03-18T16:13:01.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.372+0000 7ff3d77ed740 10 client.26207 fetch_fsmap finished waiting for FSMap version 35 2024-03-18T16:13:01.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.372+0000 7ff3d77ed740 10 client.26207 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T16:13:01.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.372+0000 7ff3d77ed740 20 client.26207 populate_metadata read hostname 'smithi039' 2024-03-18T16:13:01.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.372+0000 7ff3d77ed740 1 --2- 172.21.15.39:0/3467358568 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x5603ea70c770 0x5603ea70ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:01.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.372+0000 7ff3d77ed740 4 client.26207 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:13:01.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.372+0000 7ff3d77ed740 1 -- 172.21.15.39:0/3467358568 --> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5603ea668990 con 0x5603ea70c770 2024-03-18T16:13:01.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.373+0000 7ff3cde00640 1 --2- 172.21.15.39:0/3467358568 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x5603ea70c770 0x5603ea70ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:01.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.373+0000 7ff3cde00640 1 --2- 172.21.15.39:0/3467358568 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x5603ea70c770 0x5603ea70ebc0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b3584ead07cc0a29 server_cookie=335eff6176256ead in_seq=0 out_seq=0 2024-03-18T16:13:01.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.373+0000 7ff3af400640 10 client.26207 ms_handle_connect on v2:172.21.15.196:6838/2001427268 2024-03-18T16:13:01.379 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.377+0000 7ff3af400640 1 -- 172.21.15.39:0/3467358568 <== mds.1 v2:172.21.15.196:6838/2001427268 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5603ea668990 con 0x5603ea70c770 2024-03-18T16:13:01.379 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.377+0000 7ff3af400640 10 client.26207 handle_command_reply: tid=0 2024-03-18T16:13:01.379 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.377+0000 7ff3d77ed740 1 client.26207 shutdown 2024-03-18T16:13:01.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.377+0000 7ff3d77ed740 1 -- 172.21.15.39:0/3467358568 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5603ea6f72b0 msgr2=0x5603ea701110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:01.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.377+0000 7ff3d77ed740 1 --2- 172.21.15.39:0/3467358568 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5603ea6f72b0 0x5603ea701110 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc0094b0 tx=0x7ff3bc00ffc0 comp rx=0 tx=0).stop 2024-03-18T16:13:01.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.378+0000 7ff3d77ed740 1 -- 172.21.15.39:0/3467358568 shutdown_connections 2024-03-18T16:13:01.380 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.378+0000 7ff3d77ed740 1 --2- 172.21.15.39:0/3467358568 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x5603ea70c770 0x5603ea70ebc0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:01.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.378+0000 7ff3af400640 0 client.26207 ms_handle_reset on v2:172.21.15.196:6838/2001427268 2024-03-18T16:13:01.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.378+0000 7ff3d77ed740 1 --2- 172.21.15.39:0/3467358568 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5603ea6f72b0 0x5603ea701110 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:01.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.378+0000 7ff3d77ed740 1 --2- 172.21.15.39:0/3467358568 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5603ea6fc420 0x5603ea708150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:01.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.378+0000 7ff3d77ed740 1 --2- 172.21.15.39:0/3467358568 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5603ea6f68c0 0x5603ea700bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:01.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.378+0000 7ff3d77ed740 1 -- 172.21.15.39:0/3467358568 >> 172.21.15.39:0/3467358568 conn(0x5603ea6f1e30 msgr2=0x5603ea6f11f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:01.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.378+0000 7ff3d77ed740 1 -- 172.21.15.39:0/3467358568 shutdown_connections 2024-03-18T16:13:01.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.378+0000 7ff3d77ed740 1 -- 172.21.15.39:0/3467358568 wait complete. 2024-03-18T16:13:01.381 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.378+0000 7ff3d77ed740 20 client.26207 trim_cache size 0 max 16384 2024-03-18T16:13:01.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.405+0000 7ff3d77ed740 1 Processor -- start 2024-03-18T16:13:01.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.405+0000 7ff3d77ed740 1 -- start start 2024-03-18T16:13:01.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.406+0000 7ff3d77ed740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5603ea6f68c0 0x5603ea6f1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:01.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.406+0000 7ff3d77ed740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5603ea6f72b0 0x5603ea708110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:01.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.406+0000 7ff3d77ed740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5603ea6fc420 0x5603ea713480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:01.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.406+0000 7ff3d77ed740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7ff3d01b1020 con 0x5603ea6fc420 2024-03-18T16:13:01.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.406+0000 7ff3d77ed740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ff3d01b2ce0 con 0x5603ea6f68c0 2024-03-18T16:13:01.409 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.406+0000 7ff3d77ed740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7ff3d01b16f0 con 0x5603ea6f72b0 2024-03-18T16:13:01.409 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.406+0000 7ff3cde00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5603ea6f68c0 0x5603ea6f1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:01.409 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.406+0000 7ff3cde00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5603ea6f68c0 0x5603ea6f1070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:60958/0 (socket says 172.21.15.39:60958) 2024-03-18T16:13:01.409 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.406+0000 7ff3cde00640 1 -- 172.21.15.39:0/1872788632 learned_addr learned my addr 172.21.15.39:0/1872788632 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:13:01.409 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.407+0000 7ff3cd400640 1 --2- 172.21.15.39:0/1872788632 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5603ea6f72b0 0x5603ea708110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:01.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.407+0000 7ff3cde00640 1 -- 172.21.15.39:0/1872788632 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5603ea6f72b0 msgr2=0x5603ea708110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:01.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.407+0000 7ff3ce800640 1 --2- 172.21.15.39:0/1872788632 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5603ea6fc420 0x5603ea713480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:01.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.409+0000 7ff3cde00640 1 --2- 172.21.15.39:0/1872788632 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5603ea6f72b0 0x5603ea708110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:01.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.409+0000 7ff3cde00640 1 -- 172.21.15.39:0/1872788632 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5603ea6fc420 msgr2=0x5603ea713480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:01.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.409+0000 7ff3cde00640 1 --2- 172.21.15.39:0/1872788632 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5603ea6fc420 0x5603ea713480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:01.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.409+0000 7ff3cde00640 1 -- 172.21.15.39:0/1872788632 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5603ea6ec670 con 0x5603ea6f68c0 2024-03-18T16:13:01.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.409+0000 7ff3ce800640 1 --2- 172.21.15.39:0/1872788632 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5603ea6fc420 0x5603ea713480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:13:01.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.409+0000 7ff3cd400640 1 --2- 172.21.15.39:0/1872788632 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5603ea6f72b0 0x5603ea708110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:13:01.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.410+0000 7ff3cde00640 1 --2- 172.21.15.39:0/1872788632 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5603ea6f68c0 0x5603ea6f1070 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7ff3d0068a20 tx=0x7ff3c003e000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb6b2025de6e809a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:01.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.410+0000 7ff3ba600640 1 -- 172.21.15.39:0/1872788632 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3c0058070 con 0x5603ea6f68c0 2024-03-18T16:13:01.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.410+0000 7ff3ba600640 1 -- 172.21.15.39:0/1872788632 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff3c0045040 con 0x5603ea6f68c0 2024-03-18T16:13:01.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.411+0000 7ff3ba600640 1 -- 172.21.15.39:0/1872788632 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3c0058070 con 0x5603ea6f68c0 2024-03-18T16:13:01.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.411+0000 7ff3d77ed740 1 -- 172.21.15.39:0/1872788632 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5603ea6f68c0 msgr2=0x5603ea6f1070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:01.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.412+0000 7ff3d77ed740 1 --2- 172.21.15.39:0/1872788632 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5603ea6f68c0 0x5603ea6f1070 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7ff3d0068a20 tx=0x7ff3c003e000 comp rx=0 tx=0).stop 2024-03-18T16:13:01.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.412+0000 7ff3d77ed740 1 -- 172.21.15.39:0/1872788632 shutdown_connections 2024-03-18T16:13:01.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.412+0000 7ff3d77ed740 1 --2- 172.21.15.39:0/1872788632 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5603ea6fc420 0x5603ea713480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:01.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.412+0000 7ff3d77ed740 1 --2- 172.21.15.39:0/1872788632 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5603ea6f72b0 0x5603ea708110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:01.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.412+0000 7ff3d77ed740 1 --2- 172.21.15.39:0/1872788632 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5603ea6f68c0 0x5603ea6f1070 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:01.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.412+0000 7ff3d77ed740 1 -- 172.21.15.39:0/1872788632 >> 172.21.15.39:0/1872788632 conn(0x5603ea6f1e30 msgr2=0x5603ea6f0960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:01.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.414+0000 7ff3d77ed740 1 -- 172.21.15.39:0/1872788632 shutdown_connections 2024-03-18T16:13:01.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.415+0000 7ff3d77ed740 1 -- 172.21.15.39:0/1872788632 wait complete. 2024-03-18T16:13:01.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.415+0000 7ff3d77ed740 1 -- start start 2024-03-18T16:13:01.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.416+0000 7ff3d77ed740 1 Processor -- start 2024-03-18T16:13:01.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.417+0000 7ff3d77ed740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5603ea6f68c0 0x5603ea705670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:01.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.417+0000 7ff3d77ed740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5603ea6f72b0 0x5603ea705bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:01.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.417+0000 7ff3d77ed740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5603ea6fc420 0x5603ea706bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:01.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.417+0000 7ff3d77ed740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7ff3d01b2550 con 0x5603ea6f68c0 2024-03-18T16:13:01.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.418+0000 7ff3d77ed740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ff3d01b1dc0 con 0x5603ea6f72b0 2024-03-18T16:13:01.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.418+0000 7ff3cde00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5603ea6f68c0 0x5603ea705670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:01.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.418+0000 7ff3cde00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5603ea6f68c0 0x5603ea705670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:46376/0 (socket says 172.21.15.39:46376) 2024-03-18T16:13:01.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.418+0000 7ff3cde00640 1 -- 172.21.15.39:0/278104057 learned_addr learned my addr 172.21.15.39:0/278104057 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:13:01.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.418+0000 7ff3ce800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5603ea6fc420 0x5603ea706bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:01.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.419+0000 7ff3cd400640 1 --2- 172.21.15.39:0/278104057 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5603ea6f72b0 0x5603ea705bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:01.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.419+0000 7ff3d77ed740 1 -- 172.21.15.39:0/278104057 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7ff3d01af000 con 0x5603ea6fc420 2024-03-18T16:13:01.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.419+0000 7ff3d77ed740 10 client.0 _ll_register_callbacks cb 0x7ff3bc0116c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:13:01.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.419+0000 7ff3ce800640 1 -- 172.21.15.39:0/278104057 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5603ea6f72b0 msgr2=0x5603ea705bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:01.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.419+0000 7ff3ce800640 1 --2- 172.21.15.39:0/278104057 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5603ea6f72b0 0x5603ea705bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:01.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.419+0000 7ff3ce800640 1 -- 172.21.15.39:0/278104057 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5603ea6f68c0 msgr2=0x5603ea705670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:01.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.419+0000 7ff3ce800640 1 --2- 172.21.15.39:0/278104057 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5603ea6f68c0 0x5603ea705670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:01.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.419+0000 7ff3ce800640 1 -- 172.21.15.39:0/278104057 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5603ea7071b0 con 0x5603ea6fc420 2024-03-18T16:13:01.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.419+0000 7ff3ce800640 1 --2- 172.21.15.39:0/278104057 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5603ea6fc420 0x5603ea706bc0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x5603ea70b490 tx=0x7ff3c809b230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=11396f585e94cc67 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:01.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.419+0000 7ff3af400640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:13:01.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.419+0000 7ff3d77ed740 1 -- 172.21.15.39:0/278104057 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff3c0045040 con 0x5603ea6fc420 2024-03-18T16:13:01.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.420+0000 7ff3af400640 1 -- 172.21.15.39:0/278104057 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3c80b9050 con 0x5603ea6fc420 2024-03-18T16:13:01.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.420+0000 7ff3af400640 1 -- 172.21.15.39:0/278104057 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7ff3c80860c0 con 0x5603ea6fc420 2024-03-18T16:13:01.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.426+0000 7ff3af400640 1 -- 172.21.15.39:0/278104057 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7ff3c80b4060 con 0x5603ea6fc420 2024-03-18T16:13:01.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.427+0000 7ff3d77ed740 10 client.26221 fetch_fsmap learned FSMap version 35 2024-03-18T16:13:01.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.427+0000 7ff3d77ed740 1 -- 172.21.15.39:0/278104057 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({fsmap=35}) -- 0x5603ea7083d0 con 0x5603ea6fc420 2024-03-18T16:13:01.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.427+0000 7ff3af400640 1 -- 172.21.15.39:0/278104057 <== mon.1 v2:172.21.15.196:3300/0 4 ==== fsmap(e 35) ==== 8402+0+0 (secure 0 0 0) 0x7ff3c8005310 con 0x5603ea6fc420 2024-03-18T16:13:01.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.427+0000 7ff3d77ed740 10 client.26221 fetch_fsmap finished waiting for FSMap version 35 2024-03-18T16:13:01.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.427+0000 7ff3d77ed740 10 client.26221 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T16:13:01.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.427+0000 7ff3d77ed740 20 client.26221 populate_metadata read hostname 'smithi039' 2024-03-18T16:13:01.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.427+0000 7ff3d77ed740 1 --2- 172.21.15.39:0/278104057 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x5603ea70c770 0x5603ea7752d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:01.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.428+0000 7ff3d77ed740 4 client.26221 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:13:01.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.428+0000 7ff3d77ed740 1 -- 172.21.15.39:0/278104057 --> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5603ea662ce0 con 0x5603ea70c770 2024-03-18T16:13:01.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.428+0000 7ff3cde00640 1 --2- 172.21.15.39:0/278104057 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x5603ea70c770 0x5603ea7752d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:01.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.433+0000 7ff3af400640 10 client.26221 ms_handle_connect on v2:172.21.15.196:6838/2001427268 2024-03-18T16:13:01.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.433+0000 7ff3cde00640 1 --2- 172.21.15.39:0/278104057 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x5603ea70c770 0x5603ea7752d0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8866280c34080eec server_cookie=45571f362028f555 in_seq=0 out_seq=0 2024-03-18T16:13:01.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.433+0000 7ff3af400640 1 -- 172.21.15.39:0/278104057 <== mds.1 v2:172.21.15.196:6838/2001427268 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5603ea662ce0 con 0x5603ea70c770 2024-03-18T16:13:01.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.433+0000 7ff3af400640 10 client.26221 handle_command_reply: tid=0 2024-03-18T16:13:01.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.433+0000 7ff3d77ed740 1 client.26221 shutdown 2024-03-18T16:13:01.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.434+0000 7ff3d77ed740 1 -- 172.21.15.39:0/278104057 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5603ea6fc420 msgr2=0x5603ea706bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:01.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.434+0000 7ff3d77ed740 1 --2- 172.21.15.39:0/278104057 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5603ea6fc420 0x5603ea706bc0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x5603ea70b490 tx=0x7ff3c809b230 comp rx=0 tx=0).stop 2024-03-18T16:13:01.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.434+0000 7ff3d77ed740 1 -- 172.21.15.39:0/278104057 shutdown_connections 2024-03-18T16:13:01.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.434+0000 7ff3d77ed740 1 --2- 172.21.15.39:0/278104057 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x5603ea70c770 0x5603ea7752d0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:01.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.434+0000 7ff3d77ed740 1 --2- 172.21.15.39:0/278104057 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5603ea6f72b0 0x5603ea705bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:01.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.434+0000 7ff3d77ed740 1 --2- 172.21.15.39:0/278104057 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5603ea6fc420 0x5603ea706bc0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:01.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.434+0000 7ff3d77ed740 1 --2- 172.21.15.39:0/278104057 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5603ea6f68c0 0x5603ea705670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:01.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.434+0000 7ff3d77ed740 1 -- 172.21.15.39:0/278104057 >> 172.21.15.39:0/278104057 conn(0x5603ea6f1e30 msgr2=0x5603ea6f11f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:01.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.435+0000 7ff3af400640 0 client.26221 ms_handle_reset on v2:172.21.15.196:6838/2001427268 2024-03-18T16:13:01.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.435+0000 7ff3d77ed740 1 -- 172.21.15.39:0/278104057 shutdown_connections 2024-03-18T16:13:01.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.435+0000 7ff3d77ed740 1 -- 172.21.15.39:0/278104057 wait complete. 2024-03-18T16:13:01.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.436+0000 7ff3d77ed740 20 client.26221 trim_cache size 0 max 16384 2024-03-18T16:13:01.445 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:13:01.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.444+0000 7ff3d4a00640 1 -- 172.21.15.39:0/2699381481 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff3b40a2880 msgr2=0x7ff3b40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:01.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.444+0000 7ff3d4a00640 1 --2- 172.21.15.39:0/2699381481 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff3b40a2880 0x7ff3b40a4d40 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7ff3c0009bf0 tx=0x7ff3c002e040 comp rx=0 tx=0).stop 2024-03-18T16:13:01.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.444+0000 7ff3d4a00640 1 -- 172.21.15.39:0/2699381481 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff3d01a3850 msgr2=0x7ff3d01a5c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:01.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.444+0000 7ff3d4a00640 1 --2- 172.21.15.39:0/2699381481 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff3d01a3850 0x7ff3d01a5c40 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7ff3c800e970 tx=0x7ff3c800ed70 comp rx=0 tx=0).stop 2024-03-18T16:13:01.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.444+0000 7ff3d4a00640 1 -- 172.21.15.39:0/2699381481 shutdown_connections 2024-03-18T16:13:01.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.444+0000 7ff3d4a00640 1 --2- 172.21.15.39:0/2699381481 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff3d01a3850 0x7ff3d01a5c40 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:01.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.444+0000 7ff3d4a00640 1 --2- 172.21.15.39:0/2699381481 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff3b40a2880 0x7ff3b40a4d40 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:01.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.444+0000 7ff3d4a00640 1 --2- 172.21.15.39:0/2699381481 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff3d0105370 0x7ff3d019e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:01.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.444+0000 7ff3d4a00640 1 --2- 172.21.15.39:0/2699381481 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff3d01029c0 0x7ff3d019e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:01.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.444+0000 7ff3d4a00640 1 -- 172.21.15.39:0/2699381481 >> 172.21.15.39:0/2699381481 conn(0x7ff3d00fbc50 msgr2=0x7ff3d0103430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:01.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.445+0000 7ff3d4a00640 1 -- 172.21.15.39:0/2699381481 shutdown_connections 2024-03-18T16:13:01.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.445+0000 7ff3d4a00640 1 -- 172.21.15.39:0/2699381481 wait complete. 2024-03-18T16:13:01.473 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-18T16:13:01.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:13:01 smithi196 ceph-mon[36307]: pgmap v510: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 62 MiB/s wr, 8.89k op/s 2024-03-18T16:13:01.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:13:01 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:13:01.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:13:01 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:13:01.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:13:01 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:13:01.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:13:01 smithi194 ceph-mon[36200]: pgmap v510: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 62 MiB/s wr, 8.89k op/s 2024-03-18T16:13:01.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:13:01 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:13:01.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:13:01 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:13:01.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:13:01 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:13:01.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.742+0000 7fc412000640 1 -- 172.21.15.39:0/3725480922 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc40c103a10 msgr2=0x7fc40c103e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:01.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.742+0000 7fc412000640 1 --2- 172.21.15.39:0/3725480922 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc40c103a10 0x7fc40c103e30 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fc3fc01a910 tx=0x7fc3fc03cc70 comp rx=0 tx=0).stop 2024-03-18T16:13:01.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.742+0000 7fc412000640 1 -- 172.21.15.39:0/3725480922 shutdown_connections 2024-03-18T16:13:01.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.742+0000 7fc412000640 1 --2- 172.21.15.39:0/3725480922 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc40c1055c0 0x7fc40c10be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:01.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.742+0000 7fc412000640 1 --2- 172.21.15.39:0/3725480922 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc40c104c00 0x7fc40c105080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:01.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.742+0000 7fc412000640 1 --2- 172.21.15.39:0/3725480922 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc40c103a10 0x7fc40c103e30 secure :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fc3fc01a910 tx=0x7fc3fc03cc70 comp rx=0 tx=0).stop 2024-03-18T16:13:01.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.742+0000 7fc412000640 1 -- 172.21.15.39:0/3725480922 >> 172.21.15.39:0/3725480922 conn(0x7fc40c0fefa0 msgr2=0x7fc40c1013c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:01.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.743+0000 7fc412000640 1 -- 172.21.15.39:0/3725480922 shutdown_connections 2024-03-18T16:13:01.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.743+0000 7fc412000640 1 -- 172.21.15.39:0/3725480922 wait complete. 2024-03-18T16:13:01.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.745+0000 7fc412000640 1 Processor -- start 2024-03-18T16:13:01.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.745+0000 7fc412000640 1 -- start start 2024-03-18T16:13:01.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.746+0000 7fc412000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc40c104c00 0x7fc40c079170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:01.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.746+0000 7fc412000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc40c1055c0 0x7fc40c079730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:01.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.746+0000 7fc412000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc40c075e40 0x7fc40c0762f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:01.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.746+0000 7fc412000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fc40c06de90 con 0x7fc40c104c00 2024-03-18T16:13:01.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.746+0000 7fc412000640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fc40c06dd10 con 0x7fc40c1055c0 2024-03-18T16:13:01.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.746+0000 7fc412000640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fc40c06e010 con 0x7fc40c075e40 2024-03-18T16:13:01.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.746+0000 7fc40be00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc40c075e40 0x7fc40c0762f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:01.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.746+0000 7fc40be00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc40c075e40 0x7fc40c0762f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:37604/0 (socket says 172.21.15.39:37604) 2024-03-18T16:13:01.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.746+0000 7fc40be00640 1 -- 172.21.15.39:0/2860438747 learned_addr learned my addr 172.21.15.39:0/2860438747 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:13:01.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.747+0000 7fc40aa00640 1 --2- 172.21.15.39:0/2860438747 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc40c1055c0 0x7fc40c079730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:01.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.747+0000 7fc40be00640 1 -- 172.21.15.39:0/2860438747 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc40c1055c0 msgr2=0x7fc40c079730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:01.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.747+0000 7fc40be00640 1 --2- 172.21.15.39:0/2860438747 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc40c1055c0 0x7fc40c079730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:01.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.747+0000 7fc40be00640 1 -- 172.21.15.39:0/2860438747 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc40c104c00 msgr2=0x7fc40c079170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:01.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.747+0000 7fc40be00640 1 --2- 172.21.15.39:0/2860438747 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc40c104c00 0x7fc40c079170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:01.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.747+0000 7fc40be00640 1 -- 172.21.15.39:0/2860438747 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc40c0768b0 con 0x7fc40c075e40 2024-03-18T16:13:01.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.747+0000 7fc40aa00640 1 --2- 172.21.15.39:0/2860438747 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc40c1055c0 0x7fc40c079730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:13:01.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.747+0000 7fc40be00640 1 --2- 172.21.15.39:0/2860438747 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc40c075e40 0x7fc40c0762f0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fc40400eb10 tx=0x7fc40400c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:01.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.748+0000 7fc403e00640 1 -- 172.21.15.39:0/2860438747 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc404004b20 con 0x7fc40c075e40 2024-03-18T16:13:01.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.748+0000 7fc403e00640 1 -- 172.21.15.39:0/2860438747 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc404004cc0 con 0x7fc40c075e40 2024-03-18T16:13:01.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.748+0000 7fc403e00640 1 -- 172.21.15.39:0/2860438747 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc404010760 con 0x7fc40c075e40 2024-03-18T16:13:01.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.748+0000 7fc412000640 1 -- 172.21.15.39:0/2860438747 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc40c1917b0 con 0x7fc40c075e40 2024-03-18T16:13:01.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.749+0000 7fc412000640 1 -- 172.21.15.39:0/2860438747 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc40c191d30 con 0x7fc40c075e40 2024-03-18T16:13:01.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.753+0000 7fc414d76740 1 Processor -- start 2024-03-18T16:13:01.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.757+0000 7fc403e00640 1 -- 172.21.15.39:0/2860438747 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fc404005020 con 0x7fc40c075e40 2024-03-18T16:13:01.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.757+0000 7fc414d76740 1 -- start start 2024-03-18T16:13:01.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.757+0000 7fc414d76740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e286c238c0 0x55e286c23ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:01.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.757+0000 7fc414d76740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e286c242b0 0x55e286c28db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:01.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.757+0000 7fc414d76740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e286c29420 0x55e286c2b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:01.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.757+0000 7fc414d76740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55e286beb3b0 con 0x55e286c242b0 2024-03-18T16:13:01.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.757+0000 7fc414d76740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55e286b88450 con 0x55e286c238c0 2024-03-18T16:13:01.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.757+0000 7fc414d76740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55e286b30cb0 con 0x55e286c29420 2024-03-18T16:13:01.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.758+0000 7fc40b400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e286c242b0 0x55e286c28db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:01.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.758+0000 7fc40aa00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e286c238c0 0x55e286c23ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:01.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.758+0000 7fc40aa00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e286c238c0 0x55e286c23ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:60988/0 (socket says 172.21.15.39:60988) 2024-03-18T16:13:01.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.758+0000 7fc40aa00640 1 -- 172.21.15.39:0/2375384679 learned_addr learned my addr 172.21.15.39:0/2375384679 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:13:01.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.758+0000 7fc40aa00640 1 --2- 172.21.15.39:0/2375384679 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e286c29420 0x55e286c2b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:01.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.759+0000 7fc40aa00640 1 -- 172.21.15.39:0/2375384679 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e286c29420 msgr2=0x55e286c2b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:01.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.759+0000 7fc40aa00640 1 --2- 172.21.15.39:0/2375384679 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e286c29420 0x55e286c2b8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:01.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.759+0000 7fc40aa00640 1 -- 172.21.15.39:0/2375384679 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e286c242b0 msgr2=0x55e286c28db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:01.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.759+0000 7fc40aa00640 1 --2- 172.21.15.39:0/2375384679 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e286c242b0 0x55e286c28db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:01.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.759+0000 7fc40aa00640 1 -- 172.21.15.39:0/2375384679 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e286b521f0 con 0x55e286c238c0 2024-03-18T16:13:01.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.759+0000 7fc40b400640 1 --2- 172.21.15.39:0/2375384679 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e286c242b0 0x55e286c28db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:13:01.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.759+0000 7fc403e00640 1 --2- 172.21.15.39:0/2860438747 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fc3f00a28d0 0x7fc3f00a4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:01.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.759+0000 7fc40aa00640 1 --2- 172.21.15.39:0/2375384679 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e286c238c0 0x55e286c23ce0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fc3f8006fd0 tx=0x7fc3f800c230 comp rx=0 tx=0).ready entity=mon.2 client_cookie=97320eb2d2003f60 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:01.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.760+0000 7fc403e00640 1 -- 172.21.15.39:0/2860438747 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fc4040cf4b0 con 0x7fc40c075e40 2024-03-18T16:13:01.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.760+0000 7fc3f7e00640 1 -- 172.21.15.39:0/2375384679 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3f800f040 con 0x55e286c238c0 2024-03-18T16:13:01.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.760+0000 7fc3f7e00640 1 -- 172.21.15.39:0/2375384679 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc3f80178a0 con 0x55e286c238c0 2024-03-18T16:13:01.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.760+0000 7fc3f7e00640 1 -- 172.21.15.39:0/2375384679 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3f8016870 con 0x55e286c238c0 2024-03-18T16:13:01.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.760+0000 7fc40b400640 1 --2- 172.21.15.39:0/2860438747 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fc3f00a28d0 0x7fc3f00a4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:01.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.761+0000 7fc414d76740 1 -- 172.21.15.39:0/2375384679 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e286c238c0 msgr2=0x55e286c23ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:01.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.761+0000 7fc414d76740 1 --2- 172.21.15.39:0/2375384679 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e286c238c0 0x55e286c23ce0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fc3f8006fd0 tx=0x7fc3f800c230 comp rx=0 tx=0).stop 2024-03-18T16:13:01.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.762+0000 7fc414d76740 1 -- 172.21.15.39:0/2375384679 shutdown_connections 2024-03-18T16:13:01.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.762+0000 7fc414d76740 1 --2- 172.21.15.39:0/2375384679 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e286c29420 0x55e286c2b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:01.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.762+0000 7fc414d76740 1 --2- 172.21.15.39:0/2375384679 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e286c242b0 0x55e286c28db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:01.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.762+0000 7fc414d76740 1 --2- 172.21.15.39:0/2375384679 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e286c238c0 0x55e286c23ce0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:01.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.762+0000 7fc414d76740 1 -- 172.21.15.39:0/2375384679 >> 172.21.15.39:0/2375384679 conn(0x55e286c1ee30 msgr2=0x55e286c1f230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:01.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.762+0000 7fc40b400640 1 --2- 172.21.15.39:0/2860438747 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fc3f00a28d0 0x7fc3f00a4d90 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x55e286c24d20 tx=0x7fc3fc000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:01.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.762+0000 7fc414d76740 1 -- 172.21.15.39:0/2375384679 shutdown_connections 2024-03-18T16:13:01.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.764+0000 7fc414d76740 1 -- 172.21.15.39:0/2375384679 wait complete. 2024-03-18T16:13:01.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.766+0000 7fc414d76740 1 -- start start 2024-03-18T16:13:01.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.767+0000 7fc414d76740 1 Processor -- start 2024-03-18T16:13:01.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.768+0000 7fc414d76740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e286c238c0 0x55e286c2db60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:01.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.768+0000 7fc414d76740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e286c242b0 0x55e286c2e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:01.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.768+0000 7fc414d76740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e286c29420 0x55e286c350e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:01.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.769+0000 7fc40b400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e286c238c0 0x55e286c2db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:01.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.769+0000 7fc40b400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e286c238c0 0x55e286c2db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:37612/0 (socket says 172.21.15.39:37612) 2024-03-18T16:13:01.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.769+0000 7fc414d76740 1 -- 172.21.15.39:0/813597255 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55e286b44110 con 0x55e286c242b0 2024-03-18T16:13:01.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.769+0000 7fc40b400640 1 -- 172.21.15.39:0/813597255 learned_addr learned my addr 172.21.15.39:0/813597255 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:13:01.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.769+0000 7fc414d76740 1 -- 172.21.15.39:0/813597255 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55e286b844f0 con 0x55e286c29420 2024-03-18T16:13:01.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.769+0000 7fc414d76740 1 -- 172.21.15.39:0/813597255 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55e286c2c3d0 con 0x55e286c238c0 2024-03-18T16:13:01.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.769+0000 7fc40be00640 1 --2- 172.21.15.39:0/813597255 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e286c29420 0x55e286c350e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:01.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.769+0000 7fc414d76740 10 client.0 _ll_register_callbacks cb 0x55e286b31b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:13:01.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.769+0000 7fc40be00640 1 -- 172.21.15.39:0/813597255 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e286c238c0 msgr2=0x55e286c2db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:01.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.769+0000 7fc40be00640 1 --2- 172.21.15.39:0/813597255 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e286c238c0 0x55e286c2db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:01.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.769+0000 7fc40be00640 1 -- 172.21.15.39:0/813597255 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e286c242b0 msgr2=0x55e286c2e0a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:13:01.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.769+0000 7fc40aa00640 1 --2- 172.21.15.39:0/813597255 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e286c242b0 0x55e286c2e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:01.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.770+0000 7fc40be00640 1 --2- 172.21.15.39:0/813597255 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e286c242b0 0x55e286c2e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:01.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.770+0000 7fc40be00640 1 -- 172.21.15.39:0/813597255 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e28689c3f0 con 0x55e286c29420 2024-03-18T16:13:01.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.770+0000 7fc414d76740 1 -- 172.21.15.39:0/813597255 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc3f80178a0 con 0x55e286c29420 2024-03-18T16:13:01.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.770+0000 7fc40aa00640 1 --2- 172.21.15.39:0/813597255 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e286c242b0 0x55e286c2e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:13:01.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.770+0000 7fc40be00640 1 --2- 172.21.15.39:0/813597255 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e286c29420 0x55e286c350e0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fc4040ccbc0 tx=0x7fc40409b480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5fbd4ad18a048cc server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:01.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.770+0000 7fc3f4c00640 10 client.26222 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:13:01.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.770+0000 7fc3f4c00640 1 -- 172.21.15.39:0/813597255 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc40409fbb0 con 0x55e286c29420 2024-03-18T16:13:01.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.770+0000 7fc3f4c00640 1 -- 172.21.15.39:0/813597255 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fc4040ba250 con 0x55e286c29420 2024-03-18T16:13:01.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.770+0000 7fc3f4c00640 1 -- 172.21.15.39:0/813597255 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fc4040a4200 con 0x55e286c29420 2024-03-18T16:13:01.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.771+0000 7fc414d76740 10 client.26222 fetch_fsmap learned FSMap version 35 2024-03-18T16:13:01.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.771+0000 7fc414d76740 1 -- 172.21.15.39:0/813597255 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({fsmap=35}) -- 0x55e2869667d0 con 0x55e286c29420 2024-03-18T16:13:01.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.771+0000 7fc3f4c00640 1 -- 172.21.15.39:0/813597255 <== mon.2 v2:172.21.15.194:3300/0 4 ==== fsmap(e 35) ==== 8402+0+0 (secure 0 0 0) 0x7fc4040a3d70 con 0x55e286c29420 2024-03-18T16:13:01.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.771+0000 7fc414d76740 10 client.26222 fetch_fsmap finished waiting for FSMap version 35 2024-03-18T16:13:01.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.771+0000 7fc414d76740 10 client.26222 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:13:01.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.771+0000 7fc414d76740 20 client.26222 populate_metadata read hostname 'smithi039' 2024-03-18T16:13:01.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.771+0000 7fc414d76740 1 --2- 172.21.15.39:0/813597255 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55e286c397d0 0x55e286c3bc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:01.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.771+0000 7fc414d76740 4 client.26222 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:13:01.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.771+0000 7fc414d76740 1 -- 172.21.15.39:0/813597255 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e286b95990 con 0x55e286c397d0 2024-03-18T16:13:01.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.775+0000 7fc40b400640 1 --2- 172.21.15.39:0/813597255 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55e286c397d0 0x55e286c3bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:01.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.776+0000 7fc40b400640 1 --2- 172.21.15.39:0/813597255 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55e286c397d0 0x55e286c3bc70 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1fed947e4daa0a78 server_cookie=eca8493d8623692a in_seq=0 out_seq=0 2024-03-18T16:13:01.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.776+0000 7fc3f4c00640 10 client.26222 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:13:01.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.777+0000 7fc3f4c00640 1 -- 172.21.15.39:0/813597255 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e286b95990 con 0x55e286c397d0 2024-03-18T16:13:01.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.777+0000 7fc3f4c00640 10 client.26222 handle_command_reply: tid=0 2024-03-18T16:13:01.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.777+0000 7fc414d76740 1 client.26222 shutdown 2024-03-18T16:13:01.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.781+0000 7fc414d76740 1 -- 172.21.15.39:0/813597255 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e286c29420 msgr2=0x55e286c350e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:01.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.781+0000 7fc414d76740 1 --2- 172.21.15.39:0/813597255 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e286c29420 0x55e286c350e0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fc4040ccbc0 tx=0x7fc40409b480 comp rx=0 tx=0).stop 2024-03-18T16:13:01.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.782+0000 7fc414d76740 1 -- 172.21.15.39:0/813597255 shutdown_connections 2024-03-18T16:13:01.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.782+0000 7fc414d76740 1 --2- 172.21.15.39:0/813597255 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55e286c397d0 0x55e286c3bc70 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:01.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.782+0000 7fc414d76740 1 --2- 172.21.15.39:0/813597255 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e286c29420 0x55e286c350e0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:01.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.782+0000 7fc414d76740 1 --2- 172.21.15.39:0/813597255 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e286c242b0 0x55e286c2e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:01.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.782+0000 7fc414d76740 1 --2- 172.21.15.39:0/813597255 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e286c238c0 0x55e286c2db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:01.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.782+0000 7fc414d76740 1 -- 172.21.15.39:0/813597255 >> 172.21.15.39:0/813597255 conn(0x55e286c1ee30 msgr2=0x55e286c1e1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:01.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.782+0000 7fc3f4c00640 0 client.26222 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:13:01.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.782+0000 7fc414d76740 1 -- 172.21.15.39:0/813597255 shutdown_connections 2024-03-18T16:13:01.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.782+0000 7fc414d76740 1 -- 172.21.15.39:0/813597255 wait complete. 2024-03-18T16:13:01.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.782+0000 7fc414d76740 20 client.26222 trim_cache size 0 max 16384 2024-03-18T16:13:01.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.807+0000 7fc414d76740 1 Processor -- start 2024-03-18T16:13:01.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.808+0000 7fc414d76740 1 -- start start 2024-03-18T16:13:01.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.809+0000 7fc414d76740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e286c238c0 0x55e286c25710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:01.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.809+0000 7fc414d76740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e286c242b0 0x55e286c350a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:01.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.809+0000 7fc414d76740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e286c29420 0x55e286c25c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:01.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.809+0000 7fc414d76740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fc40c1b1090 con 0x55e286c242b0 2024-03-18T16:13:01.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.809+0000 7fc414d76740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fc40c1b2d50 con 0x55e286c29420 2024-03-18T16:13:01.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.809+0000 7fc414d76740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fc40c1b1760 con 0x55e286c238c0 2024-03-18T16:13:01.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.809+0000 7fc40b400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e286c238c0 0x55e286c25710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:01.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.809+0000 7fc40be00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e286c29420 0x55e286c25c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:01.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.809+0000 7fc40b400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e286c238c0 0x55e286c25710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:37616/0 (socket says 172.21.15.39:37616) 2024-03-18T16:13:01.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.809+0000 7fc40b400640 1 -- 172.21.15.39:0/527560044 learned_addr learned my addr 172.21.15.39:0/527560044 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:13:01.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.809+0000 7fc40aa00640 1 --2- 172.21.15.39:0/527560044 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e286c242b0 0x55e286c350a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:01.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.810+0000 7fc40be00640 1 -- 172.21.15.39:0/527560044 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e286c238c0 msgr2=0x55e286c25710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:01.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.810+0000 7fc40be00640 1 --2- 172.21.15.39:0/527560044 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e286c238c0 0x55e286c25710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:01.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.810+0000 7fc40be00640 1 -- 172.21.15.39:0/527560044 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e286c242b0 msgr2=0x55e286c350a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:01.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.810+0000 7fc40be00640 1 --2- 172.21.15.39:0/527560044 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e286c242b0 0x55e286c350a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:01.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.810+0000 7fc40be00640 1 -- 172.21.15.39:0/527560044 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e286c2c510 con 0x55e286c29420 2024-03-18T16:13:01.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.810+0000 7fc40be00640 1 --2- 172.21.15.39:0/527560044 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e286c29420 0x55e286c25c50 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fc4040a3cd0 tx=0x7fc4040bb000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=973a4f158d554361 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:01.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.810+0000 7fc3f7400640 1 -- 172.21.15.39:0/527560044 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4040cf1a0 con 0x55e286c29420 2024-03-18T16:13:01.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.810+0000 7fc3f7400640 1 -- 172.21.15.39:0/527560044 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc3f80178a0 con 0x55e286c29420 2024-03-18T16:13:01.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.810+0000 7fc3f7400640 1 -- 172.21.15.39:0/527560044 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc404005510 con 0x55e286c29420 2024-03-18T16:13:01.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.811+0000 7fc414d76740 1 -- 172.21.15.39:0/527560044 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e286c29420 msgr2=0x55e286c25c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:01.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.811+0000 7fc414d76740 1 --2- 172.21.15.39:0/527560044 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e286c29420 0x55e286c25c50 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fc4040a3cd0 tx=0x7fc4040bb000 comp rx=0 tx=0).stop 2024-03-18T16:13:01.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.811+0000 7fc414d76740 1 -- 172.21.15.39:0/527560044 shutdown_connections 2024-03-18T16:13:01.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.811+0000 7fc414d76740 1 --2- 172.21.15.39:0/527560044 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e286c29420 0x55e286c25c50 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:01.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.811+0000 7fc414d76740 1 --2- 172.21.15.39:0/527560044 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e286c242b0 0x55e286c350a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:01.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.811+0000 7fc414d76740 1 --2- 172.21.15.39:0/527560044 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e286c238c0 0x55e286c25710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:01.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.811+0000 7fc414d76740 1 -- 172.21.15.39:0/527560044 >> 172.21.15.39:0/527560044 conn(0x55e286c1ee30 msgr2=0x55e286c1f210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:01.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.812+0000 7fc414d76740 1 -- 172.21.15.39:0/527560044 shutdown_connections 2024-03-18T16:13:01.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.812+0000 7fc414d76740 1 -- 172.21.15.39:0/527560044 wait complete. 2024-03-18T16:13:01.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.813+0000 7fc414d76740 1 -- start start 2024-03-18T16:13:01.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.813+0000 7fc414d76740 1 Processor -- start 2024-03-18T16:13:01.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.813+0000 7fc414d76740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e286c238c0 0x55e286c32330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:01.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.813+0000 7fc414d76740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e286c242b0 0x55e286c32870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:01.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.814+0000 7fc414d76740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e286c29420 0x55e286c33880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:01.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.814+0000 7fc414d76740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fc40c1b25c0 con 0x55e286c242b0 2024-03-18T16:13:01.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.814+0000 7fc414d76740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fc40c1b1e30 con 0x55e286c238c0 2024-03-18T16:13:01.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.814+0000 7fc414d76740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fc40c1af070 con 0x55e286c29420 2024-03-18T16:13:01.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.814+0000 7fc414d76740 10 client.0 _ll_register_callbacks cb 0x7fc40409b8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:13:01.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.814+0000 7fc40b400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e286c238c0 0x55e286c32330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:01.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.814+0000 7fc40b400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e286c238c0 0x55e286c32330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:32784/0 (socket says 172.21.15.39:32784) 2024-03-18T16:13:01.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.814+0000 7fc40b400640 1 -- 172.21.15.39:0/1577560071 learned_addr learned my addr 172.21.15.39:0/1577560071 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:13:01.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.814+0000 7fc40aa00640 1 --2- 172.21.15.39:0/1577560071 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e286c242b0 0x55e286c32870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:01.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.814+0000 7fc40be00640 1 --2- 172.21.15.39:0/1577560071 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e286c29420 0x55e286c33880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:01.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.815+0000 7fc40be00640 1 -- 172.21.15.39:0/1577560071 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e286c238c0 msgr2=0x55e286c32330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:01.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.815+0000 7fc40be00640 1 --2- 172.21.15.39:0/1577560071 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e286c238c0 0x55e286c32330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:01.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.815+0000 7fc40be00640 1 -- 172.21.15.39:0/1577560071 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e286c242b0 msgr2=0x55e286c32870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:01.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.815+0000 7fc40be00640 1 --2- 172.21.15.39:0/1577560071 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e286c242b0 0x55e286c32870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:01.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.815+0000 7fc40be00640 1 -- 172.21.15.39:0/1577560071 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e286c33e70 con 0x55e286c29420 2024-03-18T16:13:01.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.815+0000 7fc40aa00640 1 --2- 172.21.15.39:0/1577560071 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e286c242b0 0x55e286c32870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:13:01.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.815+0000 7fc40be00640 1 --2- 172.21.15.39:0/1577560071 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e286c29420 0x55e286c33880 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fc40409b9f0 tx=0x7fc40400efd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=449bf00aa0ea0f82 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:01.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.815+0000 7fc3f4c00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:13:01.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.816+0000 7fc3f4c00640 1 -- 172.21.15.39:0/1577560071 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4040a3ef0 con 0x55e286c29420 2024-03-18T16:13:01.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.816+0000 7fc3f4c00640 1 -- 172.21.15.39:0/1577560071 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fc40409b0b0 con 0x55e286c29420 2024-03-18T16:13:01.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.816+0000 7fc414d76740 1 -- 172.21.15.39:0/1577560071 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc3f80178a0 con 0x55e286c29420 2024-03-18T16:13:01.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.817+0000 7fc3f4c00640 1 -- 172.21.15.39:0/1577560071 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fc404016df0 con 0x55e286c29420 2024-03-18T16:13:01.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.817+0000 7fc414d76740 10 client.26239 fetch_fsmap learned FSMap version 35 2024-03-18T16:13:01.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.817+0000 7fc414d76740 1 -- 172.21.15.39:0/1577560071 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55e286c33e70 con 0x55e286c29420 2024-03-18T16:13:01.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.818+0000 7fc3f4c00640 1 -- 172.21.15.39:0/1577560071 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc3f80178a0 con 0x55e286c29420 2024-03-18T16:13:01.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.818+0000 7fc3f4c00640 1 -- 172.21.15.39:0/1577560071 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 35) ==== 8402+0+0 (secure 0 0 0) 0x7fc4040a0ad0 con 0x55e286c29420 2024-03-18T16:13:01.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.818+0000 7fc414d76740 10 client.26239 fetch_fsmap finished waiting for FSMap version 35 2024-03-18T16:13:01.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.818+0000 7fc414d76740 10 client.26239 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:13:01.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.818+0000 7fc414d76740 20 client.26239 populate_metadata read hostname 'smithi039' 2024-03-18T16:13:01.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.818+0000 7fc414d76740 1 --2- 172.21.15.39:0/1577560071 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55e286c9fcd0 0x55e286ca20f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:01.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.818+0000 7fc414d76740 4 client.26239 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:13:01.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.818+0000 7fc414d76740 1 -- 172.21.15.39:0/1577560071 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e286b8fce0 con 0x55e286c9fcd0 2024-03-18T16:13:01.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.819+0000 7fc40b400640 1 --2- 172.21.15.39:0/1577560071 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55e286c9fcd0 0x55e286ca20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:01.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.819+0000 7fc40b400640 1 --2- 172.21.15.39:0/1577560071 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55e286c9fcd0 0x55e286ca20f0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=63642922fc027b50 server_cookie=89bc6ba33b8ddef1 in_seq=0 out_seq=0 2024-03-18T16:13:01.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.820+0000 7fc3f4c00640 10 client.26239 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:13:01.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.820+0000 7fc3f4c00640 1 -- 172.21.15.39:0/1577560071 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e286b8fce0 con 0x55e286c9fcd0 2024-03-18T16:13:01.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.820+0000 7fc3f4c00640 10 client.26239 handle_command_reply: tid=0 2024-03-18T16:13:01.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.820+0000 7fc414d76740 1 client.26239 shutdown 2024-03-18T16:13:01.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.821+0000 7fc414d76740 1 -- 172.21.15.39:0/1577560071 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e286c29420 msgr2=0x55e286c33880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:01.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.821+0000 7fc414d76740 1 --2- 172.21.15.39:0/1577560071 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e286c29420 0x55e286c33880 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fc40409b9f0 tx=0x7fc40400efd0 comp rx=0 tx=0).stop 2024-03-18T16:13:01.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.821+0000 7fc414d76740 1 -- 172.21.15.39:0/1577560071 shutdown_connections 2024-03-18T16:13:01.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.821+0000 7fc414d76740 1 --2- 172.21.15.39:0/1577560071 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55e286c9fcd0 0x55e286ca20f0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:01.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.821+0000 7fc414d76740 1 --2- 172.21.15.39:0/1577560071 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e286c29420 0x55e286c33880 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:01.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.821+0000 7fc3f4c00640 0 client.26239 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:13:01.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.821+0000 7fc414d76740 1 --2- 172.21.15.39:0/1577560071 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e286c242b0 0x55e286c32870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:01.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.821+0000 7fc414d76740 1 --2- 172.21.15.39:0/1577560071 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e286c238c0 0x55e286c32330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:01.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.821+0000 7fc414d76740 1 -- 172.21.15.39:0/1577560071 >> 172.21.15.39:0/1577560071 conn(0x55e286c1ee30 msgr2=0x55e286c2db40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:01.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.822+0000 7fc414d76740 1 -- 172.21.15.39:0/1577560071 shutdown_connections 2024-03-18T16:13:01.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.822+0000 7fc414d76740 1 -- 172.21.15.39:0/1577560071 wait complete. 2024-03-18T16:13:01.825 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.823+0000 7fc414d76740 20 client.26239 trim_cache size 0 max 16384 2024-03-18T16:13:01.826 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:13:01.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.824+0000 7fc412000640 1 -- 172.21.15.39:0/2860438747 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fc3f00a28d0 msgr2=0x7fc3f00a4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:01.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.824+0000 7fc412000640 1 --2- 172.21.15.39:0/2860438747 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fc3f00a28d0 0x7fc3f00a4d90 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x55e286c24d20 tx=0x7fc3fc000bc0 comp rx=0 tx=0).stop 2024-03-18T16:13:01.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.824+0000 7fc412000640 1 -- 172.21.15.39:0/2860438747 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc40c075e40 msgr2=0x7fc40c0762f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:01.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.824+0000 7fc412000640 1 --2- 172.21.15.39:0/2860438747 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc40c075e40 0x7fc40c0762f0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fc40400eb10 tx=0x7fc40400c2d0 comp rx=0 tx=0).stop 2024-03-18T16:13:01.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.824+0000 7fc412000640 1 -- 172.21.15.39:0/2860438747 shutdown_connections 2024-03-18T16:13:01.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.824+0000 7fc412000640 1 --2- 172.21.15.39:0/2860438747 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc40c1055c0 0x7fc40c079730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:01.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.825+0000 7fc412000640 1 --2- 172.21.15.39:0/2860438747 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fc3f00a28d0 0x7fc3f00a4d90 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:01.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.825+0000 7fc412000640 1 --2- 172.21.15.39:0/2860438747 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc40c075e40 0x7fc40c0762f0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:01.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.825+0000 7fc412000640 1 --2- 172.21.15.39:0/2860438747 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc40c104c00 0x7fc40c079170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:01.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.825+0000 7fc412000640 1 -- 172.21.15.39:0/2860438747 >> 172.21.15.39:0/2860438747 conn(0x7fc40c0fefa0 msgr2=0x7fc40c070960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:01.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.825+0000 7fc412000640 1 -- 172.21.15.39:0/2860438747 shutdown_connections 2024-03-18T16:13:01.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:01.825+0000 7fc412000640 1 -- 172.21.15.39:0/2860438747 wait complete. 2024-03-18T16:13:01.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:13:01 smithi039 ceph-mon[31721]: pgmap v510: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 99 KiB/s rd, 62 MiB/s wr, 8.89k op/s 2024-03-18T16:13:01.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:13:01 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:13:01.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:13:01 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:13:01.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:13:01 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:13:02.860 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-18T16:13:02.860 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T16:13:02.860 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-18T16:13:03.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.192+0000 7f6011a00640 1 -- 172.21.15.39:0/3120536104 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f600c100060 msgr2=0x7f600c102490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:03.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.192+0000 7f6011a00640 1 --2- 172.21.15.39:0/3120536104 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f600c100060 0x7f600c102490 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f6004009ca0 tx=0x7f600402c100 comp rx=0 tx=0).stop 2024-03-18T16:13:03.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.192+0000 7f6011a00640 1 -- 172.21.15.39:0/3120536104 shutdown_connections 2024-03-18T16:13:03.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.192+0000 7f6011a00640 1 --2- 172.21.15.39:0/3120536104 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f600c105380 0x7f600c105800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:03.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.192+0000 7f6011a00640 1 --2- 172.21.15.39:0/3120536104 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f600c1029d0 0x7f600c104e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:03.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.192+0000 7f6011a00640 1 --2- 172.21.15.39:0/3120536104 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f600c100060 0x7f600c102490 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:03.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.192+0000 7f6011a00640 1 -- 172.21.15.39:0/3120536104 >> 172.21.15.39:0/3120536104 conn(0x7f600c0fbc80 msgr2=0x7f600c0fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:03.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.192+0000 7f6011a00640 1 -- 172.21.15.39:0/3120536104 shutdown_connections 2024-03-18T16:13:03.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.192+0000 7f6011a00640 1 -- 172.21.15.39:0/3120536104 wait complete. 2024-03-18T16:13:03.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.194+0000 7f6011a00640 1 Processor -- start 2024-03-18T16:13:03.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.195+0000 7f6011a00640 1 -- start start 2024-03-18T16:13:03.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.195+0000 7f6011a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f600c100060 0x7f600c19e270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:03.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.195+0000 7f6011a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f600c1029d0 0x7f600c19e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:03.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.196+0000 7f6011a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f600c105380 0x7f600c1a57d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:03.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.196+0000 7f6011a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f600c113b30 con 0x7f600c105380 2024-03-18T16:13:03.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.196+0000 7f6011a00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f600c1139b0 con 0x7f600c1029d0 2024-03-18T16:13:03.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.196+0000 7f6011a00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f600c113cb0 con 0x7f600c100060 2024-03-18T16:13:03.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.196+0000 7f600a000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f600c1029d0 0x7f600c19e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:03.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.196+0000 7f600a000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f600c1029d0 0x7f600c19e7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:32808/0 (socket says 172.21.15.39:32808) 2024-03-18T16:13:03.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.196+0000 7f600a000640 1 -- 172.21.15.39:0/1659882609 learned_addr learned my addr 172.21.15.39:0/1659882609 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:13:03.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.197+0000 7f600aa00640 1 --2- 172.21.15.39:0/1659882609 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f600c100060 0x7f600c19e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:03.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.197+0000 7f600a000640 1 -- 172.21.15.39:0/1659882609 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f600c100060 msgr2=0x7f600c19e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:03.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.197+0000 7f600b400640 1 --2- 172.21.15.39:0/1659882609 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f600c105380 0x7f600c1a57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:03.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.197+0000 7f600a000640 1 --2- 172.21.15.39:0/1659882609 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f600c100060 0x7f600c19e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:03.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.197+0000 7f600a000640 1 -- 172.21.15.39:0/1659882609 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f600c105380 msgr2=0x7f600c1a57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:03.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.197+0000 7f600a000640 1 --2- 172.21.15.39:0/1659882609 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f600c105380 0x7f600c1a57d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:03.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.197+0000 7f600a000640 1 -- 172.21.15.39:0/1659882609 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f600c1a5e40 con 0x7f600c1029d0 2024-03-18T16:13:03.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.197+0000 7f600aa00640 1 --2- 172.21.15.39:0/1659882609 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f600c100060 0x7f600c19e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:13:03.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.197+0000 7f600b400640 1 --2- 172.21.15.39:0/1659882609 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f600c105380 0x7f600c1a57d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:13:03.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.198+0000 7f600a000640 1 --2- 172.21.15.39:0/1659882609 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f600c1029d0 0x7f600c19e7b0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc00e9f0 tx=0x7f5ffc00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:03.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.198+0000 7f6003400640 1 -- 172.21.15.39:0/1659882609 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ffc00cd80 con 0x7f600c1029d0 2024-03-18T16:13:03.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.198+0000 7f6003400640 1 -- 172.21.15.39:0/1659882609 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5ffc005500 con 0x7f600c1029d0 2024-03-18T16:13:03.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.198+0000 7f6003400640 1 -- 172.21.15.39:0/1659882609 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ffc010690 con 0x7f600c1029d0 2024-03-18T16:13:03.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.198+0000 7f6011a00640 1 -- 172.21.15.39:0/1659882609 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f600c1a6130 con 0x7f600c1029d0 2024-03-18T16:13:03.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.199+0000 7f6011a00640 1 -- 172.21.15.39:0/1659882609 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f600c1a6700 con 0x7f600c1029d0 2024-03-18T16:13:03.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.205+0000 7f601463f740 1 Processor -- start 2024-03-18T16:13:03.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.206+0000 7f601463f740 1 -- start start 2024-03-18T16:13:03.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.206+0000 7f6003400640 1 -- 172.21.15.39:0/1659882609 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f5ffc0046c0 con 0x7f600c1029d0 2024-03-18T16:13:03.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.208+0000 7f601463f740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56044ee827f0 0x56044ee82c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:03.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.208+0000 7f601463f740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56044ee831e0 0x56044ee87ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:03.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.208+0000 7f601463f740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56044ee88350 0x56044ee8a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:03.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.208+0000 7f601463f740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x56044edd4f80 con 0x56044ee88350 2024-03-18T16:13:03.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.208+0000 7f601463f740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x56044ed7dad0 con 0x56044ee827f0 2024-03-18T16:13:03.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.208+0000 7f601463f740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x56044edd1020 con 0x56044ee831e0 2024-03-18T16:13:03.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.209+0000 7f600b400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56044ee827f0 0x56044ee82c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:03.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.209+0000 7f600aa00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56044ee831e0 0x56044ee87ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:03.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.209+0000 7f600b400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56044ee827f0 0x56044ee82c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:32818/0 (socket says 172.21.15.39:32818) 2024-03-18T16:13:03.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.209+0000 7f600b400640 1 -- 172.21.15.39:0/152144198 learned_addr learned my addr 172.21.15.39:0/152144198 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:13:03.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.209+0000 7f600b400640 1 -- 172.21.15.39:0/152144198 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56044ee831e0 msgr2=0x56044ee87ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:03.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.209+0000 7f600b400640 1 --2- 172.21.15.39:0/152144198 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56044ee831e0 0x56044ee87ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:03.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.209+0000 7f600b400640 1 -- 172.21.15.39:0/152144198 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56044ee88350 msgr2=0x56044ee8a7e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:13:03.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.209+0000 7f600b400640 1 --2- 172.21.15.39:0/152144198 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56044ee88350 0x56044ee8a7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:03.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.209+0000 7f600b400640 1 -- 172.21.15.39:0/152144198 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56044ed956f0 con 0x56044ee827f0 2024-03-18T16:13:03.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.209+0000 7f6003400640 1 --2- 172.21.15.39:0/1659882609 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f5ff00a2950 0x7f5ff00a4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:03.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.210+0000 7f600b400640 1 --2- 172.21.15.39:0/152144198 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56044ee827f0 0x56044ee82c10 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f5ff800a990 tx=0x7f5ff800ae50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e298f590c5e3a7d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:03.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.210+0000 7f6003400640 1 -- 172.21.15.39:0/1659882609 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f5ffc0cf280 con 0x7f600c1029d0 2024-03-18T16:13:03.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.210+0000 7f5ff7e00640 1 -- 172.21.15.39:0/152144198 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ff8015070 con 0x56044ee827f0 2024-03-18T16:13:03.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.210+0000 7f5ff7e00640 1 -- 172.21.15.39:0/152144198 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5ff800cbe0 con 0x56044ee827f0 2024-03-18T16:13:03.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.210+0000 7f5ff7e00640 1 -- 172.21.15.39:0/152144198 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ff8018690 con 0x56044ee827f0 2024-03-18T16:13:03.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.210+0000 7f600aa00640 1 --2- 172.21.15.39:0/1659882609 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f5ff00a2950 0x7f5ff00a4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:03.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.211+0000 7f601463f740 1 -- 172.21.15.39:0/152144198 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56044ee827f0 msgr2=0x56044ee82c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:03.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.211+0000 7f601463f740 1 --2- 172.21.15.39:0/152144198 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56044ee827f0 0x56044ee82c10 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f5ff800a990 tx=0x7f5ff800ae50 comp rx=0 tx=0).stop 2024-03-18T16:13:03.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.212+0000 7f601463f740 1 -- 172.21.15.39:0/152144198 shutdown_connections 2024-03-18T16:13:03.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.212+0000 7f601463f740 1 --2- 172.21.15.39:0/152144198 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56044ee88350 0x56044ee8a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:03.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.212+0000 7f601463f740 1 --2- 172.21.15.39:0/152144198 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56044ee831e0 0x56044ee87ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:03.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.212+0000 7f601463f740 1 --2- 172.21.15.39:0/152144198 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56044ee827f0 0x56044ee82c10 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:03.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.212+0000 7f601463f740 1 -- 172.21.15.39:0/152144198 >> 172.21.15.39:0/152144198 conn(0x56044ee39930 msgr2=0x56044ee6d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:03.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.212+0000 7f600aa00640 1 --2- 172.21.15.39:0/1659882609 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f5ff00a2950 0x7f5ff00a4e10 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f6004006010 tx=0x7f600402c620 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:03.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.216+0000 7f601463f740 1 -- 172.21.15.39:0/152144198 shutdown_connections 2024-03-18T16:13:03.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.218+0000 7f601463f740 1 -- 172.21.15.39:0/152144198 wait complete. 2024-03-18T16:13:03.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.218+0000 7f601463f740 1 -- start start 2024-03-18T16:13:03.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.219+0000 7f601463f740 1 Processor -- start 2024-03-18T16:13:03.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.220+0000 7f601463f740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56044ee827f0 0x56044ee8cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:03.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.220+0000 7f601463f740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56044ee831e0 0x56044ee8d180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:03.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.220+0000 7f601463f740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56044ee921b0 0x56044ee945a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:03.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.221+0000 7f601463f740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x56044ee8b300 con 0x56044ee827f0 2024-03-18T16:13:03.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.221+0000 7f601463f740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x56044ed90f30 con 0x56044ee831e0 2024-03-18T16:13:03.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.221+0000 7f601463f740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x56044ee8b480 con 0x56044ee921b0 2024-03-18T16:13:03.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.221+0000 7f601463f740 10 client.0 _ll_register_callbacks cb 0x56044ed7ebc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:13:03.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.221+0000 7f600a000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56044ee831e0 0x56044ee8d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:03.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.221+0000 7f600a000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56044ee831e0 0x56044ee8d180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:32830/0 (socket says 172.21.15.39:32830) 2024-03-18T16:13:03.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.221+0000 7f600a000640 1 -- 172.21.15.39:0/2660054254 learned_addr learned my addr 172.21.15.39:0/2660054254 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:13:03.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.221+0000 7f600b400640 1 --2- 172.21.15.39:0/2660054254 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56044ee921b0 0x56044ee945a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:03.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.221+0000 7f600aa00640 1 --2- 172.21.15.39:0/2660054254 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56044ee827f0 0x56044ee8cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:03.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.221+0000 7f600a000640 1 -- 172.21.15.39:0/2660054254 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56044ee921b0 msgr2=0x56044ee945a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:03.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.221+0000 7f600a000640 1 --2- 172.21.15.39:0/2660054254 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56044ee921b0 0x56044ee945a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:03.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.221+0000 7f600a000640 1 -- 172.21.15.39:0/2660054254 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56044ee827f0 msgr2=0x56044ee8cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:03.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.222+0000 7f600a000640 1 --2- 172.21.15.39:0/2660054254 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56044ee827f0 0x56044ee8cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:03.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.222+0000 7f600a000640 1 -- 172.21.15.39:0/2660054254 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56044ebb17e0 con 0x56044ee831e0 2024-03-18T16:13:03.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.222+0000 7f600aa00640 1 --2- 172.21.15.39:0/2660054254 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56044ee827f0 0x56044ee8cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:13:03.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.223+0000 7f600a000640 1 --2- 172.21.15.39:0/2660054254 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56044ee831e0 0x56044ee8d180 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc00c6a0 tx=0x7f5ffc09b430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4974fc4c18ee2a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:03.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.223+0000 7f5ff4c00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:13:03.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.223+0000 7f5ff4c00640 1 -- 172.21.15.39:0/2660054254 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ffc0b4060 con 0x56044ee831e0 2024-03-18T16:13:03.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.223+0000 7f5ff4c00640 1 -- 172.21.15.39:0/2660054254 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f5ffc0860c0 con 0x56044ee831e0 2024-03-18T16:13:03.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.223+0000 7f601463f740 1 -- 172.21.15.39:0/2660054254 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5ff800cbe0 con 0x56044ee831e0 2024-03-18T16:13:03.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.223+0000 7f5ff4c00640 1 -- 172.21.15.39:0/2660054254 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f5ffc0b9050 con 0x56044ee831e0 2024-03-18T16:13:03.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.223+0000 7f601463f740 10 client.26240 fetch_fsmap learned FSMap version 35 2024-03-18T16:13:03.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.223+0000 7f601463f740 1 -- 172.21.15.39:0/2660054254 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x56044ec11f70 con 0x56044ee831e0 2024-03-18T16:13:03.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.224+0000 7f5ff4c00640 1 -- 172.21.15.39:0/2660054254 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5ff800cbe0 con 0x56044ee831e0 2024-03-18T16:13:03.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.224+0000 7f5ff4c00640 1 -- 172.21.15.39:0/2660054254 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 35) ==== 8402+0+0 (secure 0 0 0) 0x7f5ffc09ba90 con 0x56044ee831e0 2024-03-18T16:13:03.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.224+0000 7f601463f740 10 client.26240 fetch_fsmap finished waiting for FSMap version 35 2024-03-18T16:13:03.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.224+0000 7f601463f740 10 client.26240 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:13:03.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.224+0000 7f601463f740 20 client.26240 populate_metadata read hostname 'smithi039' 2024-03-18T16:13:03.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.224+0000 7f601463f740 1 --2- 172.21.15.39:0/2660054254 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x56044ee98f00 0x56044ee9b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:03.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.224+0000 7f601463f740 4 client.26240 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:13:03.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.224+0000 7f601463f740 1 -- 172.21.15.39:0/2660054254 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56044ede24c0 con 0x56044ee98f00 2024-03-18T16:13:03.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.225+0000 7f600aa00640 1 --2- 172.21.15.39:0/2660054254 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x56044ee98f00 0x56044ee9b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:03.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.227+0000 7f600aa00640 1 --2- 172.21.15.39:0/2660054254 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x56044ee98f00 0x56044ee9b340 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f209ea2b09b57bd6 server_cookie=6523943d71260244 in_seq=0 out_seq=0 2024-03-18T16:13:03.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.227+0000 7f5ff4c00640 10 client.26240 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:13:03.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.228+0000 7f5ff4c00640 1 -- 172.21.15.39:0/2660054254 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56044ede24c0 con 0x56044ee98f00 2024-03-18T16:13:03.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.228+0000 7f5ff4c00640 10 client.26240 handle_command_reply: tid=0 2024-03-18T16:13:03.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.229+0000 7f601463f740 1 client.26240 shutdown 2024-03-18T16:13:03.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.229+0000 7f601463f740 1 -- 172.21.15.39:0/2660054254 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56044ee831e0 msgr2=0x56044ee8d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:03.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.229+0000 7f601463f740 1 --2- 172.21.15.39:0/2660054254 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56044ee831e0 0x56044ee8d180 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc00c6a0 tx=0x7f5ffc09b430 comp rx=0 tx=0).stop 2024-03-18T16:13:03.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.230+0000 7f601463f740 1 -- 172.21.15.39:0/2660054254 shutdown_connections 2024-03-18T16:13:03.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.230+0000 7f601463f740 1 --2- 172.21.15.39:0/2660054254 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x56044ee98f00 0x56044ee9b340 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:03.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.230+0000 7f601463f740 1 --2- 172.21.15.39:0/2660054254 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56044ee831e0 0x56044ee8d180 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:03.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.230+0000 7f601463f740 1 --2- 172.21.15.39:0/2660054254 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56044ee921b0 0x56044ee945a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:03.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.230+0000 7f601463f740 1 --2- 172.21.15.39:0/2660054254 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56044ee827f0 0x56044ee8cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:03.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.230+0000 7f601463f740 1 -- 172.21.15.39:0/2660054254 >> 172.21.15.39:0/2660054254 conn(0x56044ee39930 msgr2=0x56044ee6c350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:03.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.230+0000 7f5ff4c00640 0 client.26240 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:13:03.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.230+0000 7f601463f740 1 -- 172.21.15.39:0/2660054254 shutdown_connections 2024-03-18T16:13:03.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.230+0000 7f601463f740 1 -- 172.21.15.39:0/2660054254 wait complete. 2024-03-18T16:13:03.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.230+0000 7f601463f740 20 client.26240 trim_cache size 0 max 16384 2024-03-18T16:13:03.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.266+0000 7f601463f740 1 Processor -- start 2024-03-18T16:13:03.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.266+0000 7f601463f740 1 -- start start 2024-03-18T16:13:03.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.267+0000 7f601463f740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56044ee827f0 0x56044ee84640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:03.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.267+0000 7f601463f740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56044ee831e0 0x56044ee84b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:03.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.267+0000 7f601463f740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56044ee921b0 0x56044ee6bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:03.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.267+0000 7f601463f740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f600c1a8340 con 0x56044ee827f0 2024-03-18T16:13:03.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.267+0000 7f601463f740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f600c1aa000 con 0x56044ee831e0 2024-03-18T16:13:03.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.267+0000 7f601463f740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f600c1a8a10 con 0x56044ee921b0 2024-03-18T16:13:03.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.268+0000 7f600b400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56044ee921b0 0x56044ee6bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:03.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.268+0000 7f600aa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56044ee827f0 0x56044ee84640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:03.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.268+0000 7f600aa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56044ee827f0 0x56044ee84640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:46452/0 (socket says 172.21.15.39:46452) 2024-03-18T16:13:03.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.268+0000 7f600b400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56044ee921b0 0x56044ee6bac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:37648/0 (socket says 172.21.15.39:37648) 2024-03-18T16:13:03.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.268+0000 7f600b400640 1 -- 172.21.15.39:0/1161140180 learned_addr learned my addr 172.21.15.39:0/1161140180 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:13:03.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.270+0000 7f600aa00640 1 -- 172.21.15.39:0/1161140180 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56044ee921b0 msgr2=0x56044ee6bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:03.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.270+0000 7f600aa00640 1 --2- 172.21.15.39:0/1161140180 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56044ee921b0 0x56044ee6bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:03.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.270+0000 7f600aa00640 1 -- 172.21.15.39:0/1161140180 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56044ee831e0 msgr2=0x56044ee84b80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T16:13:03.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.270+0000 7f600a000640 1 --2- 172.21.15.39:0/1161140180 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56044ee831e0 0x56044ee84b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:03.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.270+0000 7f600aa00640 1 --2- 172.21.15.39:0/1161140180 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56044ee831e0 0x56044ee84b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:03.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.270+0000 7f600aa00640 1 -- 172.21.15.39:0/1161140180 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56044eaa9170 con 0x56044ee827f0 2024-03-18T16:13:03.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.270+0000 7f600a000640 1 --2- 172.21.15.39:0/1161140180 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56044ee831e0 0x56044ee84b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:13:03.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.271+0000 7f600aa00640 1 --2- 172.21.15.39:0/1161140180 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56044ee827f0 0x56044ee84640 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f6004009bf0 tx=0x7f6004043000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9718f22171480d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:03.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.272+0000 7f5ff7400640 1 -- 172.21.15.39:0/1161140180 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6004039040 con 0x56044ee827f0 2024-03-18T16:13:03.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.272+0000 7f5ff7400640 1 -- 172.21.15.39:0/1161140180 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6004032930 con 0x56044ee827f0 2024-03-18T16:13:03.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.272+0000 7f5ff7400640 1 -- 172.21.15.39:0/1161140180 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6004040bc0 con 0x56044ee827f0 2024-03-18T16:13:03.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.273+0000 7f601463f740 1 -- 172.21.15.39:0/1161140180 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56044ee827f0 msgr2=0x56044ee84640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:03.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.273+0000 7f601463f740 1 --2- 172.21.15.39:0/1161140180 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56044ee827f0 0x56044ee84640 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f6004009bf0 tx=0x7f6004043000 comp rx=0 tx=0).stop 2024-03-18T16:13:03.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.274+0000 7f601463f740 1 -- 172.21.15.39:0/1161140180 shutdown_connections 2024-03-18T16:13:03.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.274+0000 7f601463f740 1 --2- 172.21.15.39:0/1161140180 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56044ee831e0 0x56044ee84b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:03.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.274+0000 7f601463f740 1 --2- 172.21.15.39:0/1161140180 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56044ee921b0 0x56044ee6bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:03.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.274+0000 7f601463f740 1 --2- 172.21.15.39:0/1161140180 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56044ee827f0 0x56044ee84640 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:03.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.274+0000 7f601463f740 1 -- 172.21.15.39:0/1161140180 >> 172.21.15.39:0/1161140180 conn(0x56044ee39930 msgr2=0x56044ee8cc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:03.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.275+0000 7f601463f740 1 -- 172.21.15.39:0/1161140180 shutdown_connections 2024-03-18T16:13:03.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.275+0000 7f601463f740 1 -- 172.21.15.39:0/1161140180 wait complete. 2024-03-18T16:13:03.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.275+0000 7f601463f740 1 -- start start 2024-03-18T16:13:03.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.275+0000 7f601463f740 1 Processor -- start 2024-03-18T16:13:03.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.276+0000 7f601463f740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56044ee827f0 0x56044eea8fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:03.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.276+0000 7f601463f740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56044ee831e0 0x56044eeab520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:03.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.276+0000 7f601463f740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56044ee921b0 0x56044eeaba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:03.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.276+0000 7f601463f740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f600c1a9870 con 0x56044ee827f0 2024-03-18T16:13:03.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.276+0000 7f601463f740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f600c1a90e0 con 0x56044ee831e0 2024-03-18T16:13:03.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.276+0000 7f601463f740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f600c10e7f0 con 0x56044ee921b0 2024-03-18T16:13:03.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.276+0000 7f601463f740 10 client.0 _ll_register_callbacks cb 0x7f5ffc097b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:13:03.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.277+0000 7f600a000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56044ee831e0 0x56044eeab520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:03.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.277+0000 7f600a000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56044ee831e0 0x56044eeab520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:32840/0 (socket says 172.21.15.39:32840) 2024-03-18T16:13:03.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.277+0000 7f600a000640 1 -- 172.21.15.39:0/524038342 learned_addr learned my addr 172.21.15.39:0/524038342 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:13:03.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.277+0000 7f600b400640 1 --2- 172.21.15.39:0/524038342 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56044ee921b0 0x56044eeaba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:03.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.277+0000 7f600aa00640 1 --2- 172.21.15.39:0/524038342 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56044ee827f0 0x56044eea8fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:03.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.278+0000 7f600aa00640 1 -- 172.21.15.39:0/524038342 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56044ee921b0 msgr2=0x56044eeaba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:03.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.278+0000 7f600aa00640 1 --2- 172.21.15.39:0/524038342 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56044ee921b0 0x56044eeaba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:03.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.278+0000 7f600aa00640 1 -- 172.21.15.39:0/524038342 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56044ee831e0 msgr2=0x56044eeab520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:03.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.278+0000 7f600aa00640 1 --2- 172.21.15.39:0/524038342 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56044ee831e0 0x56044eeab520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:03.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.278+0000 7f600aa00640 1 -- 172.21.15.39:0/524038342 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56044eeac0a0 con 0x56044ee827f0 2024-03-18T16:13:03.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.279+0000 7f600aa00640 1 --2- 172.21.15.39:0/524038342 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56044ee827f0 0x56044eea8fd0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f60040324c0 tx=0x7f6004004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c9c1bede76ca4dc server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:03.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.279+0000 7f5ff4c00640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-03-18T16:13:03.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.280+0000 7f5ff4c00640 1 -- 172.21.15.39:0/524038342 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6004040940 con 0x56044ee827f0 2024-03-18T16:13:03.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.280+0000 7f5ff4c00640 1 -- 172.21.15.39:0/524038342 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f6004040ae0 con 0x56044ee827f0 2024-03-18T16:13:03.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.280+0000 7f601463f740 1 -- 172.21.15.39:0/524038342 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6004032930 con 0x56044ee827f0 2024-03-18T16:13:03.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.280+0000 7f5ff4c00640 1 -- 172.21.15.39:0/524038342 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f6004040ce0 con 0x56044ee827f0 2024-03-18T16:13:03.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.280+0000 7f601463f740 10 client.15990 fetch_fsmap learned FSMap version 35 2024-03-18T16:13:03.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.280+0000 7f601463f740 1 -- 172.21.15.39:0/524038342 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x56044ef10600 con 0x56044ee827f0 2024-03-18T16:13:03.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.281+0000 7f5ff4c00640 1 -- 172.21.15.39:0/524038342 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6004032930 con 0x56044ee827f0 2024-03-18T16:13:03.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.282+0000 7f5ff4c00640 1 -- 172.21.15.39:0/524038342 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 35) ==== 8402+0+0 (secure 0 0 0) 0x7f60040348b0 con 0x56044ee827f0 2024-03-18T16:13:03.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.282+0000 7f601463f740 10 client.15990 fetch_fsmap finished waiting for FSMap version 35 2024-03-18T16:13:03.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.282+0000 7f601463f740 10 client.15990 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:13:03.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.282+0000 7f601463f740 20 client.15990 populate_metadata read hostname 'smithi039' 2024-03-18T16:13:03.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.282+0000 7f601463f740 1 --2- 172.21.15.39:0/524038342 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x56044ef10fa0 0x56044ef133c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:03.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.282+0000 7f601463f740 4 client.15990 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-18T16:13:03.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.282+0000 7f601463f740 1 -- 172.21.15.39:0/524038342 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x56044ee8ca30 con 0x56044ef10fa0 2024-03-18T16:13:03.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.282+0000 7f600a000640 1 --2- 172.21.15.39:0/524038342 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x56044ef10fa0 0x56044ef133c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:03.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.283+0000 7f600a000640 1 --2- 172.21.15.39:0/524038342 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x56044ef10fa0 0x56044ef133c0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9bc36c4706e80cd7 server_cookie=28ef7ae0611b13af in_seq=0 out_seq=0 2024-03-18T16:13:03.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.283+0000 7f5ff4c00640 10 client.15990 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:13:03.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.283+0000 7f5ff4c00640 1 -- 172.21.15.39:0/524038342 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x56044ee8ca30 con 0x56044ef10fa0 2024-03-18T16:13:03.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.283+0000 7f5ff4c00640 10 client.15990 handle_command_reply: tid=0 2024-03-18T16:13:03.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.283+0000 7f601463f740 1 client.15990 shutdown 2024-03-18T16:13:03.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.285+0000 7f601463f740 1 -- 172.21.15.39:0/524038342 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56044ee827f0 msgr2=0x56044eea8fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:03.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.285+0000 7f601463f740 1 --2- 172.21.15.39:0/524038342 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56044ee827f0 0x56044eea8fd0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f60040324c0 tx=0x7f6004004a40 comp rx=0 tx=0).stop 2024-03-18T16:13:03.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.285+0000 7f601463f740 1 -- 172.21.15.39:0/524038342 shutdown_connections 2024-03-18T16:13:03.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.285+0000 7f601463f740 1 --2- 172.21.15.39:0/524038342 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x56044ef10fa0 0x56044ef133c0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:03.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.285+0000 7f5ff4c00640 0 client.15990 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:13:03.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.285+0000 7f601463f740 1 --2- 172.21.15.39:0/524038342 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56044ee831e0 0x56044eeab520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:03.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.285+0000 7f601463f740 1 --2- 172.21.15.39:0/524038342 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56044ee921b0 0x56044eeaba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:03.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.285+0000 7f601463f740 1 --2- 172.21.15.39:0/524038342 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56044ee827f0 0x56044eea8fd0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:03.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.285+0000 7f601463f740 1 -- 172.21.15.39:0/524038342 >> 172.21.15.39:0/524038342 conn(0x56044ee39930 msgr2=0x56044ee8cc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:03.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.285+0000 7f601463f740 1 -- 172.21.15.39:0/524038342 shutdown_connections 2024-03-18T16:13:03.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.286+0000 7f601463f740 1 -- 172.21.15.39:0/524038342 wait complete. 2024-03-18T16:13:03.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.286+0000 7f601463f740 20 client.15990 trim_cache size 0 max 16384 2024-03-18T16:13:03.289 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-03-18T16:13:03.289 INFO:teuthology.orchestra.run.smithi039.stdout: "return_code": 0, 2024-03-18T16:13:03.289 INFO:teuthology.orchestra.run.smithi039.stdout: "scrub_tag": "3a44fd75-7fa8-4677-ba45-49d68eb88da5", 2024-03-18T16:13:03.290 INFO:teuthology.orchestra.run.smithi039.stdout: "mode": "asynchronous" 2024-03-18T16:13:03.290 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-03-18T16:13:03.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.287+0000 7f6011a00640 1 -- 172.21.15.39:0/1659882609 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f5ff00a2950 msgr2=0x7f5ff00a4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:03.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.287+0000 7f6011a00640 1 --2- 172.21.15.39:0/1659882609 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f5ff00a2950 0x7f5ff00a4e10 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f6004006010 tx=0x7f600402c620 comp rx=0 tx=0).stop 2024-03-18T16:13:03.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.287+0000 7f6011a00640 1 -- 172.21.15.39:0/1659882609 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f600c1029d0 msgr2=0x7f600c19e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:03.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.287+0000 7f6011a00640 1 --2- 172.21.15.39:0/1659882609 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f600c1029d0 0x7f600c19e7b0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f5ffc00e9f0 tx=0x7f5ffc00edf0 comp rx=0 tx=0).stop 2024-03-18T16:13:03.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.288+0000 7f6011a00640 1 -- 172.21.15.39:0/1659882609 shutdown_connections 2024-03-18T16:13:03.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.288+0000 7f6011a00640 1 --2- 172.21.15.39:0/1659882609 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f600c1029d0 0x7f600c19e7b0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:03.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.288+0000 7f6011a00640 1 --2- 172.21.15.39:0/1659882609 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f5ff00a2950 0x7f5ff00a4e10 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:03.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.288+0000 7f6011a00640 1 --2- 172.21.15.39:0/1659882609 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f600c105380 0x7f600c1a57d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:03.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.288+0000 7f6011a00640 1 --2- 172.21.15.39:0/1659882609 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f600c100060 0x7f600c19e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:03.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.288+0000 7f6011a00640 1 -- 172.21.15.39:0/1659882609 >> 172.21.15.39:0/1659882609 conn(0x7f600c0fbc80 msgr2=0x7f600c1032e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:03.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.288+0000 7f6011a00640 1 -- 172.21.15.39:0/1659882609 shutdown_connections 2024-03-18T16:13:03.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.288+0000 7f6011a00640 1 -- 172.21.15.39:0/1659882609 wait complete. 2024-03-18T16:13:03.324 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T16:13:03.324 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T16:13:03.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:13:03 smithi039 ceph-mon[31721]: pgmap v511: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 53 MiB/s wr, 7.57k op/s 2024-03-18T16:13:03.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.619+0000 7f3a83200640 1 -- 172.21.15.39:0/198810438 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3a7c1087e0 msgr2=0x7f3a7c108c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:03.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.619+0000 7f3a83200640 1 --2- 172.21.15.39:0/198810438 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3a7c1087e0 0x7f3a7c108c00 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c009ca0 tx=0x7f3a6c02c100 comp rx=0 tx=0).stop 2024-03-18T16:13:03.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.620+0000 7f3a83200640 1 -- 172.21.15.39:0/198810438 shutdown_connections 2024-03-18T16:13:03.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.620+0000 7f3a83200640 1 --2- 172.21.15.39:0/198810438 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3a7c110060 0x7f3a7c112450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:03.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.620+0000 7f3a83200640 1 --2- 172.21.15.39:0/198810438 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3a7c109140 0x7f3a7c10fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:03.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.620+0000 7f3a83200640 1 --2- 172.21.15.39:0/198810438 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3a7c1087e0 0x7f3a7c108c00 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:03.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.620+0000 7f3a83200640 1 -- 172.21.15.39:0/198810438 >> 172.21.15.39:0/198810438 conn(0x7f3a7c06e640 msgr2=0x7f3a7c070a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:03.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.620+0000 7f3a83200640 1 -- 172.21.15.39:0/198810438 shutdown_connections 2024-03-18T16:13:03.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.621+0000 7f3a83200640 1 -- 172.21.15.39:0/198810438 wait complete. 2024-03-18T16:13:03.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.622+0000 7f3a83200640 1 Processor -- start 2024-03-18T16:13:03.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.622+0000 7f3a83200640 1 -- start start 2024-03-18T16:13:03.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.623+0000 7f3a83200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3a7c1087e0 0x7f3a7c118070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:03.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.623+0000 7f3a83200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3a7c109140 0x7f3a7c1185b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:03.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.623+0000 7f3a83200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3a7c110060 0x7f3a7c1b6970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:03.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.623+0000 7f3a7b400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3a7c109140 0x7f3a7c1185b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:03.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.623+0000 7f3a7b400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3a7c109140 0x7f3a7c1185b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:32866/0 (socket says 172.21.15.39:32866) 2024-03-18T16:13:03.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.623+0000 7f3a7be00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3a7c1087e0 0x7f3a7c118070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:03.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.623+0000 7f3a7be00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3a7c1087e0 0x7f3a7c118070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:37682/0 (socket says 172.21.15.39:37682) 2024-03-18T16:13:03.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.623+0000 7f3a7b400640 1 -- 172.21.15.39:0/1342950213 learned_addr learned my addr 172.21.15.39:0/1342950213 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:13:03.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.623+0000 7f3a83200640 1 -- 172.21.15.39:0/1342950213 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f3a7c11c2c0 con 0x7f3a7c110060 2024-03-18T16:13:03.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.623+0000 7f3a83200640 1 -- 172.21.15.39:0/1342950213 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f3a7c11c140 con 0x7f3a7c109140 2024-03-18T16:13:03.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.623+0000 7f3a83200640 1 -- 172.21.15.39:0/1342950213 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f3a7c11c440 con 0x7f3a7c1087e0 2024-03-18T16:13:03.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.624+0000 7f3a81200640 1 --2- 172.21.15.39:0/1342950213 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3a7c110060 0x7f3a7c1b6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:03.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.624+0000 7f3a7b400640 1 -- 172.21.15.39:0/1342950213 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3a7c1087e0 msgr2=0x7f3a7c118070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:03.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.624+0000 7f3a7b400640 1 --2- 172.21.15.39:0/1342950213 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3a7c1087e0 0x7f3a7c118070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:03.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.624+0000 7f3a7b400640 1 -- 172.21.15.39:0/1342950213 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3a7c110060 msgr2=0x7f3a7c1b6970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:03.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.624+0000 7f3a7b400640 1 --2- 172.21.15.39:0/1342950213 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3a7c110060 0x7f3a7c1b6970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:03.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.624+0000 7f3a7b400640 1 -- 172.21.15.39:0/1342950213 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a7c1b6eb0 con 0x7f3a7c109140 2024-03-18T16:13:03.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.624+0000 7f3a7be00640 1 --2- 172.21.15.39:0/1342950213 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3a7c1087e0 0x7f3a7c118070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:13:03.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.624+0000 7f3a81200640 1 --2- 172.21.15.39:0/1342950213 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3a7c110060 0x7f3a7c1b6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:13:03.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.625+0000 7f3a7b400640 1 --2- 172.21.15.39:0/1342950213 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3a7c109140 0x7f3a7c1185b0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f3a6800ea60 tx=0x7f3a6800ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:03.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.626+0000 7f3a78c00640 1 -- 172.21.15.39:0/1342950213 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a6800cde0 con 0x7f3a7c109140 2024-03-18T16:13:03.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.626+0000 7f3a78c00640 1 -- 172.21.15.39:0/1342950213 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a680035f0 con 0x7f3a7c109140 2024-03-18T16:13:03.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.626+0000 7f3a78c00640 1 -- 172.21.15.39:0/1342950213 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a68010670 con 0x7f3a7c109140 2024-03-18T16:13:03.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.626+0000 7f3a83200640 1 -- 172.21.15.39:0/1342950213 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a7c1b7080 con 0x7f3a7c109140 2024-03-18T16:13:03.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.626+0000 7f3a83200640 1 -- 172.21.15.39:0/1342950213 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a7c1b7580 con 0x7f3a7c109140 2024-03-18T16:13:03.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.632+0000 7f3a85f6c740 1 Processor -- start 2024-03-18T16:13:03.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.632+0000 7f3a85f6c740 1 -- start start 2024-03-18T16:13:03.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.632+0000 7f3a85f6c740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55782e122810 0x55782e122c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:03.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.632+0000 7f3a85f6c740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55782e123200 0x55782e127d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:03.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.632+0000 7f3a85f6c740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55782e128370 0x55782e12a800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:03.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.632+0000 7f3a85f6c740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55782e0871f0 con 0x55782e123200 2024-03-18T16:13:03.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.632+0000 7f3a85f6c740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55782e02fa80 con 0x55782e122810 2024-03-18T16:13:03.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.632+0000 7f3a85f6c740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55782e083290 con 0x55782e128370 2024-03-18T16:13:03.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.633+0000 7f3a81200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55782e122810 0x55782e122c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:03.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.633+0000 7f3a81200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55782e122810 0x55782e122c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:32878/0 (socket says 172.21.15.39:32878) 2024-03-18T16:13:03.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.633+0000 7f3a81200640 1 -- 172.21.15.39:0/2182328827 learned_addr learned my addr 172.21.15.39:0/2182328827 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:13:03.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.633+0000 7f3a81200640 1 -- 172.21.15.39:0/2182328827 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55782e128370 msgr2=0x55782e12a800 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:13:03.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.633+0000 7f3a81200640 1 --2- 172.21.15.39:0/2182328827 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55782e128370 0x55782e12a800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:03.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.633+0000 7f3a81200640 1 -- 172.21.15.39:0/2182328827 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55782e123200 msgr2=0x55782e127d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:03.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.633+0000 7f3a81200640 1 --2- 172.21.15.39:0/2182328827 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55782e123200 0x55782e127d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:03.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.633+0000 7f3a81200640 1 -- 172.21.15.39:0/2182328827 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55782de657d0 con 0x55782e122810 2024-03-18T16:13:03.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.634+0000 7f3a81200640 1 --2- 172.21.15.39:0/2182328827 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55782e122810 0x55782e122c30 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f3a74009c20 tx=0x7f3a74010740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d342099c5c0545ed server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:03.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.634+0000 7f3a71600640 1 -- 172.21.15.39:0/2182328827 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a74004030 con 0x55782e122810 2024-03-18T16:13:03.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.634+0000 7f3a71600640 1 -- 172.21.15.39:0/2182328827 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a740041d0 con 0x55782e122810 2024-03-18T16:13:03.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.634+0000 7f3a71600640 1 -- 172.21.15.39:0/2182328827 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a74012690 con 0x55782e122810 2024-03-18T16:13:03.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.634+0000 7f3a78c00640 1 -- 172.21.15.39:0/1342950213 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f3a68004000 con 0x7f3a7c109140 2024-03-18T16:13:03.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.635+0000 7f3a85f6c740 1 -- 172.21.15.39:0/2182328827 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55782e122810 msgr2=0x55782e122c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:03.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.635+0000 7f3a85f6c740 1 --2- 172.21.15.39:0/2182328827 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55782e122810 0x55782e122c30 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f3a74009c20 tx=0x7f3a74010740 comp rx=0 tx=0).stop 2024-03-18T16:13:03.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.635+0000 7f3a85f6c740 1 -- 172.21.15.39:0/2182328827 shutdown_connections 2024-03-18T16:13:03.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.635+0000 7f3a85f6c740 1 --2- 172.21.15.39:0/2182328827 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55782e128370 0x55782e12a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:03.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.635+0000 7f3a85f6c740 1 --2- 172.21.15.39:0/2182328827 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55782e123200 0x55782e127d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:03.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.635+0000 7f3a85f6c740 1 --2- 172.21.15.39:0/2182328827 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55782e122810 0x55782e122c30 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:03.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.635+0000 7f3a85f6c740 1 -- 172.21.15.39:0/2182328827 >> 172.21.15.39:0/2182328827 conn(0x55782e11dd80 msgr2=0x55782e11e180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:03.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.635+0000 7f3a85f6c740 1 -- 172.21.15.39:0/2182328827 shutdown_connections 2024-03-18T16:13:03.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.635+0000 7f3a85f6c740 1 -- 172.21.15.39:0/2182328827 wait complete. 2024-03-18T16:13:03.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.636+0000 7f3a85f6c740 1 -- start start 2024-03-18T16:13:03.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.636+0000 7f3a85f6c740 1 Processor -- start 2024-03-18T16:13:03.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.637+0000 7f3a85f6c740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55782e122810 0x55782e12cd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:03.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.637+0000 7f3a85f6c740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55782e123200 0x55782e12d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:03.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.637+0000 7f3a85f6c740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55782e128370 0x55782e134280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:03.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.637+0000 7f3a85f6c740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55782e12b320 con 0x55782e122810 2024-03-18T16:13:03.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.637+0000 7f3a85f6c740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55782e042ed0 con 0x55782e123200 2024-03-18T16:13:03.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.637+0000 7f3a85f6c740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55782e12b4a0 con 0x55782e128370 2024-03-18T16:13:03.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.637+0000 7f3a85f6c740 10 client.0 _ll_register_callbacks cb 0x55782e07e220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:13:03.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.637+0000 7f3a7b400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55782e128370 0x55782e134280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:03.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.637+0000 7f3a7b400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55782e128370 0x55782e134280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:37684/0 (socket says 172.21.15.39:37684) 2024-03-18T16:13:03.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.637+0000 7f3a7b400640 1 -- 172.21.15.39:0/1991172603 learned_addr learned my addr 172.21.15.39:0/1991172603 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:13:03.641 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.638+0000 7f3a7be00640 1 --2- 172.21.15.39:0/1991172603 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55782e123200 0x55782e12d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:03.641 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.638+0000 7f3a78c00640 1 --2- 172.21.15.39:0/1342950213 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3a600a28c0 0x7f3a600a4d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:03.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.639+0000 7f3a7b400640 1 -- 172.21.15.39:0/1991172603 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55782e123200 msgr2=0x55782e12d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:03.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.639+0000 7f3a7b400640 1 --2- 172.21.15.39:0/1991172603 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55782e123200 0x55782e12d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:03.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.639+0000 7f3a7b400640 1 -- 172.21.15.39:0/1991172603 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55782e122810 msgr2=0x55782e12cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:03.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.639+0000 7f3a7b400640 1 --2- 172.21.15.39:0/1991172603 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55782e122810 0x55782e12cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:03.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.639+0000 7f3a7b400640 1 -- 172.21.15.39:0/1991172603 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55782dd9b3f0 con 0x55782e128370 2024-03-18T16:13:03.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.639+0000 7f3a7b400640 1 --2- 172.21.15.39:0/1991172603 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55782e128370 0x55782e134280 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f3a680d7530 tx=0x7f3a680d79f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f213a04284b2252b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:03.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.640+0000 7f3a81200640 1 --2- 172.21.15.39:0/1342950213 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3a600a28c0 0x7f3a600a4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:03.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.641+0000 7f3a66000640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:13:03.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.641+0000 7f3a66000640 1 -- 172.21.15.39:0/1991172603 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a680e2060 con 0x55782e128370 2024-03-18T16:13:03.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.641+0000 7f3a66000640 1 -- 172.21.15.39:0/1991172603 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f3a680cf610 con 0x55782e128370 2024-03-18T16:13:03.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.641+0000 7f3a78c00640 1 -- 172.21.15.39:0/1342950213 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f3a680cf060 con 0x7f3a7c109140 2024-03-18T16:13:03.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.641+0000 7f3a85f6c740 1 -- 172.21.15.39:0/1991172603 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3a740041d0 con 0x55782e128370 2024-03-18T16:13:03.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.641+0000 7f3a66000640 1 -- 172.21.15.39:0/1991172603 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f3a680978b0 con 0x55782e128370 2024-03-18T16:13:03.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.641+0000 7f3a85f6c740 10 client.26266 fetch_fsmap learned FSMap version 35 2024-03-18T16:13:03.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.641+0000 7f3a85f6c740 1 -- 172.21.15.39:0/1991172603 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55782dd5b0f0 con 0x55782e128370 2024-03-18T16:13:03.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.641+0000 7f3a81200640 1 --2- 172.21.15.39:0/1342950213 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3a600a28c0 0x7f3a600a4d80 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x55782e12e1e0 tx=0x7f3a74003fe0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:03.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.642+0000 7f3a66000640 1 -- 172.21.15.39:0/1991172603 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a740041d0 con 0x55782e128370 2024-03-18T16:13:03.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.643+0000 7f3a66000640 1 -- 172.21.15.39:0/1991172603 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 35) ==== 8402+0+0 (secure 0 0 0) 0x7f3a68097c80 con 0x55782e128370 2024-03-18T16:13:03.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.643+0000 7f3a85f6c740 10 client.26266 fetch_fsmap finished waiting for FSMap version 35 2024-03-18T16:13:03.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.643+0000 7f3a85f6c740 10 client.26266 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:13:03.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.643+0000 7f3a85f6c740 20 client.26266 populate_metadata read hostname 'smithi039' 2024-03-18T16:13:03.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.643+0000 7f3a85f6c740 1 --2- 172.21.15.39:0/1991172603 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55782e138b50 0x55782e13afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:03.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.643+0000 7f3a85f6c740 4 client.26266 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:13:03.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.643+0000 7f3a85f6c740 1 -- 172.21.15.39:0/1991172603 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55782e09abe0 con 0x55782e138b50 2024-03-18T16:13:03.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.643+0000 7f3a7be00640 1 --2- 172.21.15.39:0/1991172603 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55782e138b50 0x55782e13afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:03.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.644+0000 7f3a66000640 10 client.26266 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:13:03.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.644+0000 7f3a7be00640 1 --2- 172.21.15.39:0/1991172603 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55782e138b50 0x55782e13afb0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e19cf6f889c446e4 server_cookie=9faf84f2f9c73695 in_seq=0 out_seq=0 2024-03-18T16:13:03.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.644+0000 7f3a66000640 1 -- 172.21.15.39:0/1991172603 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55782e09abe0 con 0x55782e138b50 2024-03-18T16:13:03.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.644+0000 7f3a66000640 10 client.26266 handle_command_reply: tid=0 2024-03-18T16:13:03.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.644+0000 7f3a85f6c740 1 client.26266 shutdown 2024-03-18T16:13:03.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.646+0000 7f3a85f6c740 1 -- 172.21.15.39:0/1991172603 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55782e128370 msgr2=0x55782e134280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:03.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.646+0000 7f3a85f6c740 1 --2- 172.21.15.39:0/1991172603 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55782e128370 0x55782e134280 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f3a680d7530 tx=0x7f3a680d79f0 comp rx=0 tx=0).stop 2024-03-18T16:13:03.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.647+0000 7f3a85f6c740 1 -- 172.21.15.39:0/1991172603 shutdown_connections 2024-03-18T16:13:03.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.647+0000 7f3a85f6c740 1 --2- 172.21.15.39:0/1991172603 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55782e138b50 0x55782e13afb0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:03.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.647+0000 7f3a85f6c740 1 --2- 172.21.15.39:0/1991172603 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55782e123200 0x55782e12d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:03.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.647+0000 7f3a85f6c740 1 --2- 172.21.15.39:0/1991172603 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55782e128370 0x55782e134280 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:03.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.647+0000 7f3a85f6c740 1 --2- 172.21.15.39:0/1991172603 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55782e122810 0x55782e12cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:03.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.647+0000 7f3a66000640 0 client.26266 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:13:03.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.647+0000 7f3a85f6c740 1 -- 172.21.15.39:0/1991172603 >> 172.21.15.39:0/1991172603 conn(0x55782e11dd80 msgr2=0x55782e11cec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:03.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.647+0000 7f3a85f6c740 1 -- 172.21.15.39:0/1991172603 shutdown_connections 2024-03-18T16:13:03.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.647+0000 7f3a85f6c740 1 -- 172.21.15.39:0/1991172603 wait complete. 2024-03-18T16:13:03.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.647+0000 7f3a85f6c740 20 client.26266 trim_cache size 0 max 16384 2024-03-18T16:13:03.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.679+0000 7f3a85f6c740 1 Processor -- start 2024-03-18T16:13:03.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.679+0000 7f3a85f6c740 1 -- start start 2024-03-18T16:13:03.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.679+0000 7f3a85f6c740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55782e122810 0x55782e12c340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:03.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.679+0000 7f3a85f6c740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55782e123200 0x55782e134240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:03.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.680+0000 7f3a85f6c740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55782e128370 0x55782e124700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:03.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.680+0000 7f3a85f6c740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f3a5c00fba0 con 0x55782e122810 2024-03-18T16:13:03.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.680+0000 7f3a85f6c740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f3a5c011860 con 0x55782e128370 2024-03-18T16:13:03.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.680+0000 7f3a85f6c740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f3a5c010270 con 0x55782e123200 2024-03-18T16:13:03.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.680+0000 7f3a7b400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55782e123200 0x55782e134240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:03.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.680+0000 7f3a7b400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55782e123200 0x55782e134240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:37686/0 (socket says 172.21.15.39:37686) 2024-03-18T16:13:03.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.680+0000 7f3a7b400640 1 -- 172.21.15.39:0/2032594848 learned_addr learned my addr 172.21.15.39:0/2032594848 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:13:03.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.680+0000 7f3a7be00640 1 --2- 172.21.15.39:0/2032594848 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55782e122810 0x55782e12c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:03.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.680+0000 7f3a81200640 1 --2- 172.21.15.39:0/2032594848 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55782e128370 0x55782e124700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:03.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.680+0000 7f3a7be00640 1 -- 172.21.15.39:0/2032594848 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55782e123200 msgr2=0x55782e134240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:03.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.681+0000 7f3a7be00640 1 --2- 172.21.15.39:0/2032594848 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55782e123200 0x55782e134240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:03.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.681+0000 7f3a7be00640 1 -- 172.21.15.39:0/2032594848 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55782e128370 msgr2=0x55782e124700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:03.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.681+0000 7f3a7be00640 1 --2- 172.21.15.39:0/2032594848 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55782e128370 0x55782e124700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:03.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.681+0000 7f3a7be00640 1 -- 172.21.15.39:0/2032594848 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55782e11dbb0 con 0x55782e122810 2024-03-18T16:13:03.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.681+0000 7f3a81200640 1 --2- 172.21.15.39:0/2032594848 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55782e128370 0x55782e124700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:13:03.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.682+0000 7f3a7be00640 1 --2- 172.21.15.39:0/2032594848 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55782e122810 0x55782e12c340 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c009bf0 tx=0x7f3a6c0323a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47397415fd042004 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:03.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.682+0000 7f3a70c00640 1 -- 172.21.15.39:0/2032594848 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a6c071070 con 0x55782e122810 2024-03-18T16:13:03.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.682+0000 7f3a70c00640 1 -- 172.21.15.39:0/2032594848 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a6c003970 con 0x55782e122810 2024-03-18T16:13:03.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.682+0000 7f3a70c00640 1 -- 172.21.15.39:0/2032594848 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a6c039040 con 0x55782e122810 2024-03-18T16:13:03.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.683+0000 7f3a85f6c740 1 -- 172.21.15.39:0/2032594848 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55782e122810 msgr2=0x55782e12c340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:03.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.683+0000 7f3a85f6c740 1 --2- 172.21.15.39:0/2032594848 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55782e122810 0x55782e12c340 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c009bf0 tx=0x7f3a6c0323a0 comp rx=0 tx=0).stop 2024-03-18T16:13:03.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.683+0000 7f3a85f6c740 1 -- 172.21.15.39:0/2032594848 shutdown_connections 2024-03-18T16:13:03.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.683+0000 7f3a85f6c740 1 --2- 172.21.15.39:0/2032594848 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55782e128370 0x55782e124700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:03.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.683+0000 7f3a85f6c740 1 --2- 172.21.15.39:0/2032594848 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55782e123200 0x55782e134240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:03.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.683+0000 7f3a85f6c740 1 --2- 172.21.15.39:0/2032594848 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55782e122810 0x55782e12c340 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:03.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.683+0000 7f3a85f6c740 1 -- 172.21.15.39:0/2032594848 >> 172.21.15.39:0/2032594848 conn(0x55782e11dd80 msgr2=0x55782e12cd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:03.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.683+0000 7f3a85f6c740 1 -- 172.21.15.39:0/2032594848 shutdown_connections 2024-03-18T16:13:03.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.684+0000 7f3a85f6c740 1 -- 172.21.15.39:0/2032594848 wait complete. 2024-03-18T16:13:03.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.684+0000 7f3a85f6c740 1 -- start start 2024-03-18T16:13:03.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.684+0000 7f3a85f6c740 1 Processor -- start 2024-03-18T16:13:03.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.685+0000 7f3a85f6c740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55782e122810 0x55782e1318f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:03.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.685+0000 7f3a85f6c740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55782e123200 0x55782e131e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:03.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.685+0000 7f3a85f6c740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55782e128370 0x55782e132dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:03.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.685+0000 7f3a85f6c740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f3a5c0110d0 con 0x55782e123200 2024-03-18T16:13:03.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.685+0000 7f3a85f6c740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f3a5c010940 con 0x55782e128370 2024-03-18T16:13:03.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.685+0000 7f3a85f6c740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f3a5c00db80 con 0x55782e122810 2024-03-18T16:13:03.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.685+0000 7f3a85f6c740 10 client.0 _ll_register_callbacks cb 0x7f3a680d27f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:13:03.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.685+0000 7f3a81200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55782e128370 0x55782e132dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:03.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.685+0000 7f3a7be00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55782e122810 0x55782e1318f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:03.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.685+0000 7f3a81200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55782e128370 0x55782e132dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:32902/0 (socket says 172.21.15.39:32902) 2024-03-18T16:13:03.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.685+0000 7f3a81200640 1 -- 172.21.15.39:0/3477840402 learned_addr learned my addr 172.21.15.39:0/3477840402 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:13:03.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.685+0000 7f3a7b400640 1 --2- 172.21.15.39:0/3477840402 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55782e123200 0x55782e131e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:03.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.685+0000 7f3a81200640 1 -- 172.21.15.39:0/3477840402 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55782e122810 msgr2=0x55782e1318f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:03.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.685+0000 7f3a81200640 1 --2- 172.21.15.39:0/3477840402 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55782e122810 0x55782e1318f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:03.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.685+0000 7f3a81200640 1 -- 172.21.15.39:0/3477840402 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55782e123200 msgr2=0x55782e131e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:03.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.686+0000 7f3a81200640 1 --2- 172.21.15.39:0/3477840402 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55782e123200 0x55782e131e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:03.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.686+0000 7f3a81200640 1 -- 172.21.15.39:0/3477840402 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55782e1333c0 con 0x55782e128370 2024-03-18T16:13:03.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.686+0000 7f3a7be00640 1 --2- 172.21.15.39:0/3477840402 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55782e122810 0x55782e1318f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:13:03.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.686+0000 7f3a81200640 1 --2- 172.21.15.39:0/3477840402 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55782e128370 0x55782e132dd0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f3a74009c20 tx=0x7f3a74023000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9edb679831716a13 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:03.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.686+0000 7f3a66000640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:13:03.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.686+0000 7f3a66000640 1 -- 172.21.15.39:0/3477840402 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a74038060 con 0x55782e128370 2024-03-18T16:13:03.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.686+0000 7f3a66000640 1 -- 172.21.15.39:0/3477840402 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f3a74012650 con 0x55782e128370 2024-03-18T16:13:03.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.686+0000 7f3a85f6c740 1 -- 172.21.15.39:0/3477840402 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3a6c003970 con 0x55782e128370 2024-03-18T16:13:03.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.687+0000 7f3a66000640 1 -- 172.21.15.39:0/3477840402 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f3a74005530 con 0x55782e128370 2024-03-18T16:13:03.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.687+0000 7f3a85f6c740 10 client.26261 fetch_fsmap learned FSMap version 35 2024-03-18T16:13:03.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.687+0000 7f3a85f6c740 1 -- 172.21.15.39:0/3477840402 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55782e19eeb0 con 0x55782e128370 2024-03-18T16:13:03.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.687+0000 7f3a66000640 1 -- 172.21.15.39:0/3477840402 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a6c003970 con 0x55782e128370 2024-03-18T16:13:03.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.688+0000 7f3a66000640 1 -- 172.21.15.39:0/3477840402 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 35) ==== 8402+0+0 (secure 0 0 0) 0x7f3a74025070 con 0x55782e128370 2024-03-18T16:13:03.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.688+0000 7f3a85f6c740 10 client.26261 fetch_fsmap finished waiting for FSMap version 35 2024-03-18T16:13:03.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.688+0000 7f3a85f6c740 10 client.26261 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:13:03.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.688+0000 7f3a85f6c740 20 client.26261 populate_metadata read hostname 'smithi039' 2024-03-18T16:13:03.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.688+0000 7f3a85f6c740 1 --2- 172.21.15.39:0/3477840402 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55782e19f640 0x55782e1a1a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:03.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.688+0000 7f3a85f6c740 4 client.26261 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T16:13:03.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.688+0000 7f3a85f6c740 1 -- 172.21.15.39:0/3477840402 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55782e12cb10 con 0x55782e19f640 2024-03-18T16:13:03.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.688+0000 7f3a7be00640 1 --2- 172.21.15.39:0/3477840402 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55782e19f640 0x55782e1a1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:03.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.689+0000 7f3a7be00640 1 --2- 172.21.15.39:0/3477840402 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55782e19f640 0x55782e1a1a60 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=49e814271f671aae server_cookie=516f7f6e187179ed in_seq=0 out_seq=0 2024-03-18T16:13:03.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.689+0000 7f3a66000640 10 client.26261 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:13:03.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.689+0000 7f3a66000640 1 -- 172.21.15.39:0/3477840402 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55782e12cb10 con 0x55782e19f640 2024-03-18T16:13:03.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.689+0000 7f3a66000640 10 client.26261 handle_command_reply: tid=0 2024-03-18T16:13:03.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.689+0000 7f3a85f6c740 1 client.26261 shutdown 2024-03-18T16:13:03.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.690+0000 7f3a85f6c740 1 -- 172.21.15.39:0/3477840402 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55782e128370 msgr2=0x55782e132dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:03.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.690+0000 7f3a85f6c740 1 --2- 172.21.15.39:0/3477840402 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55782e128370 0x55782e132dd0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f3a74009c20 tx=0x7f3a74023000 comp rx=0 tx=0).stop 2024-03-18T16:13:03.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.691+0000 7f3a85f6c740 1 -- 172.21.15.39:0/3477840402 shutdown_connections 2024-03-18T16:13:03.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.691+0000 7f3a85f6c740 1 --2- 172.21.15.39:0/3477840402 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55782e19f640 0x55782e1a1a60 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:03.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.691+0000 7f3a85f6c740 1 --2- 172.21.15.39:0/3477840402 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55782e128370 0x55782e132dd0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:03.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.691+0000 7f3a85f6c740 1 --2- 172.21.15.39:0/3477840402 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55782e123200 0x55782e131e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:03.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.691+0000 7f3a85f6c740 1 --2- 172.21.15.39:0/3477840402 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55782e122810 0x55782e1318f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:03.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.691+0000 7f3a66000640 0 client.26261 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:13:03.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.691+0000 7f3a85f6c740 1 -- 172.21.15.39:0/3477840402 >> 172.21.15.39:0/3477840402 conn(0x55782e11dd80 msgr2=0x55782e124700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:03.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.691+0000 7f3a85f6c740 1 -- 172.21.15.39:0/3477840402 shutdown_connections 2024-03-18T16:13:03.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.691+0000 7f3a85f6c740 1 -- 172.21.15.39:0/3477840402 wait complete. 2024-03-18T16:13:03.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.691+0000 7f3a85f6c740 20 client.26261 trim_cache size 0 max 16384 2024-03-18T16:13:03.695 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-03-18T16:13:03.695 INFO:teuthology.orchestra.run.smithi039.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-18T16:13:03.695 INFO:teuthology.orchestra.run.smithi039.stdout: "scrubs": { 2024-03-18T16:13:03.695 INFO:teuthology.orchestra.run.smithi039.stdout: "3a44fd75-7fa8-4677-ba45-49d68eb88da5": { 2024-03-18T16:13:03.695 INFO:teuthology.orchestra.run.smithi039.stdout: "path": "/", 2024-03-18T16:13:03.696 INFO:teuthology.orchestra.run.smithi039.stdout: "tag": "3a44fd75-7fa8-4677-ba45-49d68eb88da5", 2024-03-18T16:13:03.696 INFO:teuthology.orchestra.run.smithi039.stdout: "options": "recursive,force" 2024-03-18T16:13:03.696 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:13:03.696 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:13:03.696 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-03-18T16:13:03.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.692+0000 7f3a83200640 1 -- 172.21.15.39:0/1342950213 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3a600a28c0 msgr2=0x7f3a600a4d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:03.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.692+0000 7f3a83200640 1 --2- 172.21.15.39:0/1342950213 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3a600a28c0 0x7f3a600a4d80 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x55782e12e1e0 tx=0x7f3a74003fe0 comp rx=0 tx=0).stop 2024-03-18T16:13:03.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.692+0000 7f3a83200640 1 -- 172.21.15.39:0/1342950213 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3a7c109140 msgr2=0x7f3a7c1185b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:03.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.692+0000 7f3a83200640 1 --2- 172.21.15.39:0/1342950213 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3a7c109140 0x7f3a7c1185b0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f3a6800ea60 tx=0x7f3a6800ef20 comp rx=0 tx=0).stop 2024-03-18T16:13:03.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.692+0000 7f3a83200640 1 -- 172.21.15.39:0/1342950213 shutdown_connections 2024-03-18T16:13:03.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.692+0000 7f3a83200640 1 --2- 172.21.15.39:0/1342950213 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3a7c109140 0x7f3a7c1185b0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:03.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.692+0000 7f3a83200640 1 --2- 172.21.15.39:0/1342950213 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3a600a28c0 0x7f3a600a4d80 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:03.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.692+0000 7f3a83200640 1 --2- 172.21.15.39:0/1342950213 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3a7c110060 0x7f3a7c1b6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:03.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.692+0000 7f3a83200640 1 --2- 172.21.15.39:0/1342950213 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3a7c1087e0 0x7f3a7c118070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:03.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.692+0000 7f3a83200640 1 -- 172.21.15.39:0/1342950213 >> 172.21.15.39:0/1342950213 conn(0x7f3a7c06e640 msgr2=0x7f3a7c070330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:03.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.694+0000 7f3a83200640 1 -- 172.21.15.39:0/1342950213 shutdown_connections 2024-03-18T16:13:03.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:03.694+0000 7f3a83200640 1 -- 172.21.15.39:0/1342950213 wait complete. 2024-03-18T16:13:03.718 INFO:tasks.cephfs.filesystem:scrub status for tag:3a44fd75-7fa8-4677-ba45-49d68eb88da5 - {'path': '/', 'tag': '3a44fd75-7fa8-4677-ba45-49d68eb88da5', 'options': 'recursive,force'} 2024-03-18T16:13:03.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:13:03 smithi196 ceph-mon[36307]: pgmap v511: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 53 MiB/s wr, 7.57k op/s 2024-03-18T16:13:03.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:13:03 smithi194 ceph-mon[36200]: pgmap v511: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 91 KiB/s rd, 53 MiB/s wr, 7.57k op/s 2024-03-18T16:13:04.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:13:04 smithi196 ceph-mon[36307]: scrub queued for path: / 2024-03-18T16:13:04.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:13:04 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:13:04.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:13:04 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:13:04.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:13:04 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:13:04.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:13:04 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:13:04.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:13:04 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:13:04.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:13:04 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:13:04.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:13:04 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:13:04.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:13:04 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:13:04.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:13:04 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:13:04.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:13:04 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:13:04.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:13:04 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:13:04.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:13:04 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:13:04.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:13:04 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:13:04.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:13:04 smithi194 ceph-mon[36200]: scrub queued for path: / 2024-03-18T16:13:04.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:13:04 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:13:04.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:13:04 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:13:04.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:13:04 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:13:04.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:13:04 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:13:04.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:13:04 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:13:04.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:13:04 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:13:04.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:13:04 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:13:04.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:13:04 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:13:04.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:13:04 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:13:04.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:13:04 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:13:04.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:13:04 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:13:04.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:13:04 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:13:04.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:13:04 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:13:04.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:13:04 smithi039 ceph-mon[31721]: scrub queued for path: / 2024-03-18T16:13:04.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:13:04 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:13:04.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:13:04 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:13:04.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:13:04 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:13:04.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:13:04 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:13:04.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:13:04 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:13:04.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:13:04 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:13:04.838 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:13:04 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:13:04.838 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:13:04 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:13:04.838 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:13:04 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:13:04.838 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:13:04 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:13:04.838 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:13:04 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:13:04.838 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:13:04 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:13:04.838 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:13:04 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:13:05.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:13:05 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:13:05.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:13:05 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:13:05.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:13:05 smithi194 ceph-mon[36200]: pgmap v512: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 60 MiB/s wr, 8.58k op/s 2024-03-18T16:13:05.839 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:13:05 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:13:05.839 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:13:05 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:13:05.839 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:13:05 smithi039 ceph-mon[31721]: pgmap v512: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 60 MiB/s wr, 8.58k op/s 2024-03-18T16:13:05.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:13:05 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:13:05.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:13:05 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:13:05.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:13:05 smithi196 ceph-mon[36307]: pgmap v512: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 60 MiB/s wr, 8.58k op/s 2024-03-18T16:13:06.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:13:06 smithi039 ceph-mon[31721]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T16:13:06.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:13:06 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:13:06.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:13:06 smithi039 ceph-mon[31721]: pgmap v513: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 56 MiB/s wr, 7.98k op/s 2024-03-18T16:13:06.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:13:06 smithi196 ceph-mon[36307]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T16:13:06.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:13:06 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:13:06.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:13:06 smithi196 ceph-mon[36307]: pgmap v513: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 56 MiB/s wr, 7.98k op/s 2024-03-18T16:13:06.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:13:06 smithi194 ceph-mon[36200]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T16:13:06.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:13:06 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:13:06.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:13:06 smithi194 ceph-mon[36200]: pgmap v513: 129 pgs: 129 active+clean; 13 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 162 KiB/s rd, 56 MiB/s wr, 7.98k op/s 2024-03-18T16:13:07.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:13:07 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:13:07.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:13:07 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:13:07.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:13:07 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:13:08.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:13:08 smithi039 ceph-mon[31721]: pgmap v514: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 57 MiB/s wr, 8.05k op/s 2024-03-18T16:13:08.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:13:08 smithi196 ceph-mon[36307]: pgmap v514: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 57 MiB/s wr, 8.05k op/s 2024-03-18T16:13:08.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:13:08 smithi194 ceph-mon[36200]: pgmap v514: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 57 MiB/s wr, 8.05k op/s 2024-03-18T16:13:10.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:13:10 smithi194 ceph-mon[36200]: pgmap v515: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 63 MiB/s wr, 8.57k op/s 2024-03-18T16:13:11.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:13:10 smithi039 ceph-mon[31721]: pgmap v515: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 63 MiB/s wr, 8.57k op/s 2024-03-18T16:13:11.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:13:10 smithi196 ceph-mon[36307]: pgmap v515: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 150 KiB/s rd, 63 MiB/s wr, 8.57k op/s 2024-03-18T16:13:13.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:13:12 smithi039 ceph-mon[31721]: pgmap v516: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 56 MiB/s wr, 7.58k op/s 2024-03-18T16:13:13.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:13:12 smithi196 ceph-mon[36307]: pgmap v516: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 56 MiB/s wr, 7.58k op/s 2024-03-18T16:13:13.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:13:12 smithi194 ceph-mon[36200]: pgmap v516: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 56 MiB/s wr, 7.58k op/s 2024-03-18T16:13:15.041 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:13:14 smithi194 ceph-mon[36200]: pgmap v517: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 62 MiB/s wr, 8.29k op/s 2024-03-18T16:13:15.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:13:14 smithi039 ceph-mon[31721]: pgmap v517: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 62 MiB/s wr, 8.29k op/s 2024-03-18T16:13:15.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:13:14 smithi196 ceph-mon[36307]: pgmap v517: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 134 KiB/s rd, 62 MiB/s wr, 8.29k op/s 2024-03-18T16:13:17.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:13:16 smithi039 ceph-mon[31721]: pgmap v518: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 59 MiB/s wr, 7.87k op/s 2024-03-18T16:13:17.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:13:16 smithi196 ceph-mon[36307]: pgmap v518: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 59 MiB/s wr, 7.87k op/s 2024-03-18T16:13:17.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:13:16 smithi194 ceph-mon[36200]: pgmap v518: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 59 MiB/s wr, 7.87k op/s 2024-03-18T16:13:18.089 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:13:17 smithi039 ceph-mon[31721]: scrub summary: idle 2024-03-18T16:13:18.089 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:13:17 smithi039 ceph-mon[31721]: scrub summary: idle 2024-03-18T16:13:18.089 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:13:17 smithi039 ceph-mon[31721]: scrub summary: idle 2024-03-18T16:13:18.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:13:17 smithi196 ceph-mon[36307]: scrub summary: idle 2024-03-18T16:13:18.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:13:17 smithi196 ceph-mon[36307]: scrub summary: idle 2024-03-18T16:13:18.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:13:17 smithi196 ceph-mon[36307]: scrub summary: idle 2024-03-18T16:13:18.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:13:17 smithi194 ceph-mon[36200]: scrub summary: idle 2024-03-18T16:13:18.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:13:17 smithi194 ceph-mon[36200]: scrub summary: idle 2024-03-18T16:13:18.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:13:17 smithi194 ceph-mon[36200]: scrub summary: idle 2024-03-18T16:13:19.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:13:18 smithi039 ceph-mon[31721]: scrub summary: idle 2024-03-18T16:13:19.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:13:18 smithi039 ceph-mon[31721]: scrub summary: idle 2024-03-18T16:13:19.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:13:18 smithi039 ceph-mon[31721]: pgmap v519: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 61 MiB/s wr, 7.97k op/s 2024-03-18T16:13:19.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:13:18 smithi196 ceph-mon[36307]: scrub summary: idle 2024-03-18T16:13:19.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:13:18 smithi196 ceph-mon[36307]: scrub summary: idle 2024-03-18T16:13:19.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:13:18 smithi196 ceph-mon[36307]: pgmap v519: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 61 MiB/s wr, 7.97k op/s 2024-03-18T16:13:19.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:13:18 smithi194 ceph-mon[36200]: scrub summary: idle 2024-03-18T16:13:19.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:13:18 smithi194 ceph-mon[36200]: scrub summary: idle 2024-03-18T16:13:19.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:13:18 smithi194 ceph-mon[36200]: pgmap v519: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 61 MiB/s wr, 7.97k op/s 2024-03-18T16:13:21.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:13:20 smithi039 ceph-mon[31721]: pgmap v520: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 66 MiB/s wr, 8.59k op/s 2024-03-18T16:13:21.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:13:20 smithi196 ceph-mon[36307]: pgmap v520: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 66 MiB/s wr, 8.59k op/s 2024-03-18T16:13:21.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:13:20 smithi194 ceph-mon[36200]: pgmap v520: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 66 MiB/s wr, 8.59k op/s 2024-03-18T16:13:23.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:13:22 smithi039 ceph-mon[31721]: pgmap v521: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 60 MiB/s wr, 7.80k op/s 2024-03-18T16:13:23.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:13:22 smithi196 ceph-mon[36307]: pgmap v521: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 60 MiB/s wr, 7.80k op/s 2024-03-18T16:13:23.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:13:22 smithi194 ceph-mon[36200]: pgmap v521: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 60 MiB/s wr, 7.80k op/s 2024-03-18T16:13:25.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:13:24 smithi039 ceph-mon[31721]: pgmap v522: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 68 MiB/s wr, 8.81k op/s 2024-03-18T16:13:25.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:13:24 smithi196 ceph-mon[36307]: pgmap v522: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 68 MiB/s wr, 8.81k op/s 2024-03-18T16:13:25.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:13:24 smithi194 ceph-mon[36200]: pgmap v522: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 68 MiB/s wr, 8.81k op/s 2024-03-18T16:13:27.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:13:26 smithi039 ceph-mon[31721]: pgmap v523: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 65 MiB/s wr, 8.31k op/s 2024-03-18T16:13:27.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:13:26 smithi196 ceph-mon[36307]: pgmap v523: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 65 MiB/s wr, 8.31k op/s 2024-03-18T16:13:27.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:13:26 smithi194 ceph-mon[36200]: pgmap v523: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 65 MiB/s wr, 8.31k op/s 2024-03-18T16:13:29.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:13:28 smithi039 ceph-mon[31721]: pgmap v524: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 72 MiB/s wr, 9.12k op/s 2024-03-18T16:13:29.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:13:28 smithi196 ceph-mon[36307]: pgmap v524: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 72 MiB/s wr, 9.12k op/s 2024-03-18T16:13:29.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:13:28 smithi194 ceph-mon[36200]: pgmap v524: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 72 MiB/s wr, 9.12k op/s 2024-03-18T16:13:31.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:13:30 smithi196 ceph-mon[36307]: pgmap v525: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 70 MiB/s wr, 8.99k op/s 2024-03-18T16:13:31.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:13:30 smithi194 ceph-mon[36200]: pgmap v525: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 70 MiB/s wr, 8.99k op/s 2024-03-18T16:13:31.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:13:30 smithi039 ceph-mon[31721]: pgmap v525: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 70 MiB/s wr, 8.99k op/s 2024-03-18T16:13:33.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:13:32 smithi196 ceph-mon[36307]: pgmap v526: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 63 MiB/s wr, 7.94k op/s 2024-03-18T16:13:33.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:13:32 smithi194 ceph-mon[36200]: pgmap v526: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 63 MiB/s wr, 7.94k op/s 2024-03-18T16:13:33.339 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:13:32 smithi039 ceph-mon[31721]: pgmap v526: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 63 MiB/s wr, 7.94k op/s 2024-03-18T16:13:33.719 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T16:13:33.720 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T16:13:35.102 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:13:34 smithi039 ceph-mon[31721]: pgmap v527: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 72 MiB/s wr, 9.05k op/s 2024-03-18T16:13:35.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:13:34 smithi196 ceph-mon[36307]: pgmap v527: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 72 MiB/s wr, 9.05k op/s 2024-03-18T16:13:35.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:13:34 smithi194 ceph-mon[36200]: pgmap v527: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 72 MiB/s wr, 9.05k op/s 2024-03-18T16:13:35.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.279+0000 7f20af000640 1 -- 172.21.15.39:0/3037088310 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f20a8105490 msgr2=0x7f20a810bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:35.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.279+0000 7f20af000640 1 --2- 172.21.15.39:0/3037088310 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f20a8105490 0x7f20a810bd20 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f2094009860 tx=0x7f209402bb70 comp rx=0 tx=0).stop 2024-03-18T16:13:35.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.293+0000 7f20af000640 1 -- 172.21.15.39:0/3037088310 shutdown_connections 2024-03-18T16:13:35.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.293+0000 7f20af000640 1 --2- 172.21.15.39:0/3037088310 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f20a8104ad0 0x7f20a8104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:35.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.294+0000 7f20af000640 1 --2- 172.21.15.39:0/3037088310 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f20a8105490 0x7f20a810bd20 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:35.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.294+0000 7f20af000640 1 --2- 172.21.15.39:0/3037088310 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f20a81038e0 0x7f20a8103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:35.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.294+0000 7f20af000640 1 -- 172.21.15.39:0/3037088310 >> 172.21.15.39:0/3037088310 conn(0x7f20a80fee50 msgr2=0x7f20a8101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:35.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.294+0000 7f20af000640 1 -- 172.21.15.39:0/3037088310 shutdown_connections 2024-03-18T16:13:35.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.295+0000 7f20af000640 1 -- 172.21.15.39:0/3037088310 wait complete. 2024-03-18T16:13:35.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.299+0000 7f20af000640 1 Processor -- start 2024-03-18T16:13:35.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.300+0000 7f20af000640 1 -- start start 2024-03-18T16:13:35.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.301+0000 7f20af000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f20a81038e0 0x7f20a8070170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:35.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.301+0000 7f20af000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f20a8104ad0 0x7f20a80706b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:35.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.301+0000 7f20af000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f20a8105490 0x7f20a81a9ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:35.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.301+0000 7f20af000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f20a810f5d0 con 0x7f20a8104ad0 2024-03-18T16:13:35.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.301+0000 7f20af000640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f20a810f450 con 0x7f20a81038e0 2024-03-18T16:13:35.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.301+0000 7f20af000640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f20a810f750 con 0x7f20a8105490 2024-03-18T16:13:35.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.302+0000 7f20adc00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f20a81038e0 0x7f20a8070170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:35.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.302+0000 7f20adc00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f20a81038e0 0x7f20a8070170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:35292/0 (socket says 172.21.15.39:35292) 2024-03-18T16:13:35.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.302+0000 7f20adc00640 1 -- 172.21.15.39:0/208718403 learned_addr learned my addr 172.21.15.39:0/208718403 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:13:35.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.302+0000 7f20ad200640 1 --2- 172.21.15.39:0/208718403 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f20a8104ad0 0x7f20a80706b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:35.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.303+0000 7f20ae600640 1 --2- 172.21.15.39:0/208718403 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f20a8105490 0x7f20a81a9ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:35.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.305+0000 7f20adc00640 1 -- 172.21.15.39:0/208718403 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f20a8105490 msgr2=0x7f20a81a9ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:35.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.305+0000 7f20adc00640 1 --2- 172.21.15.39:0/208718403 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f20a8105490 0x7f20a81a9ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:35.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.305+0000 7f20adc00640 1 -- 172.21.15.39:0/208718403 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f20a8104ad0 msgr2=0x7f20a80706b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:35.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.305+0000 7f20adc00640 1 --2- 172.21.15.39:0/208718403 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f20a8104ad0 0x7f20a80706b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:35.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.305+0000 7f20adc00640 1 -- 172.21.15.39:0/208718403 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20a8070d50 con 0x7f20a81038e0 2024-03-18T16:13:35.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.305+0000 7f20ad200640 1 --2- 172.21.15.39:0/208718403 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f20a8104ad0 0x7f20a80706b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:13:35.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.307+0000 7f20adc00640 1 --2- 172.21.15.39:0/208718403 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f20a81038e0 0x7f20a8070170 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f209c00b450 tx=0x7f209c00b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:35.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.307+0000 7f20a6000640 1 -- 172.21.15.39:0/208718403 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f209c015020 con 0x7f20a81038e0 2024-03-18T16:13:35.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.307+0000 7f20a6000640 1 -- 172.21.15.39:0/208718403 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f209c011d70 con 0x7f20a81038e0 2024-03-18T16:13:35.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.307+0000 7f20a6000640 1 -- 172.21.15.39:0/208718403 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f209c00f340 con 0x7f20a81038e0 2024-03-18T16:13:35.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.308+0000 7f20af000640 1 -- 172.21.15.39:0/208718403 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20a81aa340 con 0x7f20a81038e0 2024-03-18T16:13:35.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.308+0000 7f20af000640 1 -- 172.21.15.39:0/208718403 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20a81aa890 con 0x7f20a81038e0 2024-03-18T16:13:35.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.315+0000 7f20a6000640 1 -- 172.21.15.39:0/208718403 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f209c018020 con 0x7f20a81038e0 2024-03-18T16:13:35.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.318+0000 7f20a6000640 1 --2- 172.21.15.39:0/208718403 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f20900a2800 0x7f20900a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:35.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.318+0000 7f20a6000640 1 -- 172.21.15.39:0/208718403 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f209c0ce900 con 0x7f20a81038e0 2024-03-18T16:13:35.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.320+0000 7f20ad200640 1 --2- 172.21.15.39:0/208718403 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f20900a2800 0x7f20900a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:35.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.322+0000 7f20ad200640 1 --2- 172.21.15.39:0/208718403 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f20900a2800 0x7f20900a4cc0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f20a0009fd0 tx=0x7f20a0009680 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:35.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.449+0000 7f20b3369740 1 Processor -- start 2024-03-18T16:13:35.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.449+0000 7f20b3369740 1 -- start start 2024-03-18T16:13:35.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.449+0000 7f20b3369740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55820b9bd220 0x55820b9bd640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:35.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.449+0000 7f20b3369740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55820b9bdc10 0x55820b9c2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:35.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.449+0000 7f20b3369740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55820b9c2d80 0x55820b9c5210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:35.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.449+0000 7f20b3369740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55820b9191f0 con 0x55820b9bdc10 2024-03-18T16:13:35.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.449+0000 7f20b3369740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55820b8c1a80 con 0x55820b9c2d80 2024-03-18T16:13:35.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.450+0000 7f20b3369740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55820b915290 con 0x55820b9bd220 2024-03-18T16:13:35.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.451+0000 7f20adc00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55820b9bd220 0x55820b9bd640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:35.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.451+0000 7f20adc00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55820b9bd220 0x55820b9bd640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:55126/0 (socket says 172.21.15.39:55126) 2024-03-18T16:13:35.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.451+0000 7f20adc00640 1 -- 172.21.15.39:0/4244033778 learned_addr learned my addr 172.21.15.39:0/4244033778 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:13:35.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.451+0000 7f20ae600640 1 --2- 172.21.15.39:0/4244033778 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55820b9c2d80 0x55820b9c5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:35.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.452+0000 7f20adc00640 1 -- 172.21.15.39:0/4244033778 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55820b9c2d80 msgr2=0x55820b9c5210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:35.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.452+0000 7f20adc00640 1 --2- 172.21.15.39:0/4244033778 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55820b9c2d80 0x55820b9c5210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:35.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.452+0000 7f20adc00640 1 -- 172.21.15.39:0/4244033778 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55820b9bdc10 msgr2=0x55820b9c2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:35.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.452+0000 7f20ad200640 1 --2- 172.21.15.39:0/4244033778 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55820b9bdc10 0x55820b9c2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:35.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.452+0000 7f20adc00640 1 --2- 172.21.15.39:0/4244033778 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55820b9bdc10 0x55820b9c2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:35.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.452+0000 7f20adc00640 1 -- 172.21.15.39:0/4244033778 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55820b6f77d0 con 0x55820b9bd220 2024-03-18T16:13:35.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.453+0000 7f20ad200640 1 --2- 172.21.15.39:0/4244033778 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55820b9bdc10 0x55820b9c2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:13:35.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.453+0000 7f20adc00640 1 --2- 172.21.15.39:0/4244033778 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55820b9bd220 0x55820b9bd640 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f209c09ae40 tx=0x7f209c097260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a324285b291cefd6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:35.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.454+0000 7f209aa00640 1 -- 172.21.15.39:0/4244033778 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f209c09f000 con 0x55820b9bd220 2024-03-18T16:13:35.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.454+0000 7f209aa00640 1 -- 172.21.15.39:0/4244033778 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f209c0a3650 con 0x55820b9bd220 2024-03-18T16:13:35.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.454+0000 7f209aa00640 1 -- 172.21.15.39:0/4244033778 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f209c09a300 con 0x55820b9bd220 2024-03-18T16:13:35.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.456+0000 7f20b3369740 1 -- 172.21.15.39:0/4244033778 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55820b9bd220 msgr2=0x55820b9bd640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:35.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.456+0000 7f20b3369740 1 --2- 172.21.15.39:0/4244033778 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55820b9bd220 0x55820b9bd640 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f209c09ae40 tx=0x7f209c097260 comp rx=0 tx=0).stop 2024-03-18T16:13:35.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.457+0000 7f20b3369740 1 -- 172.21.15.39:0/4244033778 shutdown_connections 2024-03-18T16:13:35.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.457+0000 7f20b3369740 1 --2- 172.21.15.39:0/4244033778 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55820b9c2d80 0x55820b9c5210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:35.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.457+0000 7f20b3369740 1 --2- 172.21.15.39:0/4244033778 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55820b9bdc10 0x55820b9c2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:35.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.457+0000 7f20b3369740 1 --2- 172.21.15.39:0/4244033778 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55820b9bd220 0x55820b9bd640 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:35.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.457+0000 7f20b3369740 1 -- 172.21.15.39:0/4244033778 >> 172.21.15.39:0/4244033778 conn(0x55820b9aff80 msgr2=0x55820b9b0380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:35.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.457+0000 7f20b3369740 1 -- 172.21.15.39:0/4244033778 shutdown_connections 2024-03-18T16:13:35.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.458+0000 7f20b3369740 1 -- 172.21.15.39:0/4244033778 wait complete. 2024-03-18T16:13:35.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.459+0000 7f20b3369740 1 -- start start 2024-03-18T16:13:35.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.460+0000 7f20b3369740 1 Processor -- start 2024-03-18T16:13:35.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.461+0000 7f20b3369740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55820b9bd220 0x55820b9b5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:35.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.461+0000 7f20b3369740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55820b9bdc10 0x55820b9b6510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:35.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.461+0000 7f20b3369740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55820b9c2d80 0x55820b9bb540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:35.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.461+0000 7f20b3369740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55820b9b4570 con 0x55820b9bd220 2024-03-18T16:13:35.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.461+0000 7f20b3369740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55820b8d4ed0 con 0x55820b9c2d80 2024-03-18T16:13:35.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.461+0000 7f20b3369740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55820b9b46f0 con 0x55820b9bdc10 2024-03-18T16:13:35.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.462+0000 7f20b3369740 10 client.0 _ll_register_callbacks cb 0x55820b910220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:13:35.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.463+0000 7f20ad200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55820b9bdc10 0x55820b9b6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:35.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.463+0000 7f20ad200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55820b9bdc10 0x55820b9b6510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:55140/0 (socket says 172.21.15.39:55140) 2024-03-18T16:13:35.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.463+0000 7f20ad200640 1 -- 172.21.15.39:0/4172682850 learned_addr learned my addr 172.21.15.39:0/4172682850 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:13:35.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.464+0000 7f20ae600640 1 --2- 172.21.15.39:0/4172682850 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55820b9c2d80 0x55820b9bb540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:35.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.464+0000 7f20adc00640 1 --2- 172.21.15.39:0/4172682850 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55820b9bd220 0x55820b9b5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:35.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.465+0000 7f20ae600640 1 -- 172.21.15.39:0/4172682850 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55820b9bdc10 msgr2=0x55820b9b6510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:35.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.465+0000 7f20ae600640 1 --2- 172.21.15.39:0/4172682850 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55820b9bdc10 0x55820b9b6510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:35.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.465+0000 7f20ae600640 1 -- 172.21.15.39:0/4172682850 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55820b9bd220 msgr2=0x55820b9b5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:35.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.465+0000 7f20ae600640 1 --2- 172.21.15.39:0/4172682850 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55820b9bd220 0x55820b9b5fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:35.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.465+0000 7f20ae600640 1 -- 172.21.15.39:0/4172682850 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55820b62d3f0 con 0x55820b9c2d80 2024-03-18T16:13:35.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.466+0000 7f20ad200640 1 --2- 172.21.15.39:0/4172682850 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55820b9bdc10 0x55820b9b6510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:13:35.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.466+0000 7f20adc00640 1 --2- 172.21.15.39:0/4172682850 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55820b9bd220 0x55820b9b5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:13:35.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.467+0000 7f208f400640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:13:35.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.468+0000 7f20ae600640 1 --2- 172.21.15.39:0/4172682850 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55820b9c2d80 0x55820b9bb540 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f2094009780 tx=0x7f2094034dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a87956cf4a81bed6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:35.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.468+0000 7f208f400640 1 -- 172.21.15.39:0/4172682850 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2094043020 con 0x55820b9c2d80 2024-03-18T16:13:35.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.468+0000 7f208f400640 1 -- 172.21.15.39:0/4172682850 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f2094004300 con 0x55820b9c2d80 2024-03-18T16:13:35.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.469+0000 7f20b3369740 1 -- 172.21.15.39:0/4172682850 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f209c0a3650 con 0x55820b9c2d80 2024-03-18T16:13:35.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.471+0000 7f208f400640 1 -- 172.21.15.39:0/4172682850 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f2094039040 con 0x55820b9c2d80 2024-03-18T16:13:35.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.471+0000 7f20b3369740 10 client.26273 fetch_fsmap learned FSMap version 36 2024-03-18T16:13:35.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.471+0000 7f20b3369740 1 -- 172.21.15.39:0/4172682850 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55820b5ed0f0 con 0x55820b9c2d80 2024-03-18T16:13:35.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.472+0000 7f208f400640 1 -- 172.21.15.39:0/4172682850 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f209c0a3650 con 0x55820b9c2d80 2024-03-18T16:13:35.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.472+0000 7f208f400640 1 -- 172.21.15.39:0/4172682850 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 36) ==== 8406+0+0 (secure 0 0 0) 0x7f2094003a80 con 0x55820b9c2d80 2024-03-18T16:13:35.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.473+0000 7f20b3369740 10 client.26273 fetch_fsmap finished waiting for FSMap version 36 2024-03-18T16:13:35.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.473+0000 7f20b3369740 10 client.26273 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:13:35.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.473+0000 7f20b3369740 20 client.26273 populate_metadata read hostname 'smithi039' 2024-03-18T16:13:35.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.473+0000 7f20b3369740 1 --2- 172.21.15.39:0/4172682850 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55820b9dbcf0 0x55820b9de190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:35.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.473+0000 7f20b3369740 4 client.26273 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:13:35.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.473+0000 7f20b3369740 1 -- 172.21.15.39:0/4172682850 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55820b92cbe0 con 0x55820b9dbcf0 2024-03-18T16:13:35.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.476+0000 7f20adc00640 1 --2- 172.21.15.39:0/4172682850 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55820b9dbcf0 0x55820b9de190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:35.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.477+0000 7f208f400640 10 client.26273 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:13:35.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.477+0000 7f20adc00640 1 --2- 172.21.15.39:0/4172682850 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55820b9dbcf0 0x55820b9de190 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf444cd73d159d1 server_cookie=afc869afde065e60 in_seq=0 out_seq=0 2024-03-18T16:13:35.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.479+0000 7f208f400640 1 -- 172.21.15.39:0/4172682850 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55820b92cbe0 con 0x55820b9dbcf0 2024-03-18T16:13:35.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.479+0000 7f208f400640 10 client.26273 handle_command_reply: tid=0 2024-03-18T16:13:35.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.479+0000 7f20b3369740 1 client.26273 shutdown 2024-03-18T16:13:35.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.481+0000 7f20b3369740 1 -- 172.21.15.39:0/4172682850 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55820b9c2d80 msgr2=0x55820b9bb540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:35.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.481+0000 7f20b3369740 1 --2- 172.21.15.39:0/4172682850 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55820b9c2d80 0x55820b9bb540 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f2094009780 tx=0x7f2094034dc0 comp rx=0 tx=0).stop 2024-03-18T16:13:35.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.481+0000 7f20b3369740 1 -- 172.21.15.39:0/4172682850 shutdown_connections 2024-03-18T16:13:35.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.481+0000 7f20b3369740 1 --2- 172.21.15.39:0/4172682850 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55820b9dbcf0 0x55820b9de190 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:35.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.481+0000 7f20b3369740 1 --2- 172.21.15.39:0/4172682850 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55820b9c2d80 0x55820b9bb540 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:35.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.481+0000 7f20b3369740 1 --2- 172.21.15.39:0/4172682850 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55820b9bdc10 0x55820b9b6510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:35.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.481+0000 7f20b3369740 1 --2- 172.21.15.39:0/4172682850 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55820b9bd220 0x55820b9b5fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:35.487 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.481+0000 7f20b3369740 1 -- 172.21.15.39:0/4172682850 >> 172.21.15.39:0/4172682850 conn(0x55820b9aff80 msgr2=0x55820b9af110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:35.487 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.481+0000 7f208f400640 0 client.26273 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:13:35.487 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.482+0000 7f20b3369740 1 -- 172.21.15.39:0/4172682850 shutdown_connections 2024-03-18T16:13:35.487 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.482+0000 7f20b3369740 1 -- 172.21.15.39:0/4172682850 wait complete. 2024-03-18T16:13:35.487 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.482+0000 7f20b3369740 20 client.26273 trim_cache size 0 max 16384 2024-03-18T16:13:35.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.531+0000 7f20b3369740 1 Processor -- start 2024-03-18T16:13:35.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.531+0000 7f20b3369740 1 -- start start 2024-03-18T16:13:35.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.532+0000 7f20b3369740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55820b9bd220 0x55820b9b5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:35.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.532+0000 7f20b3369740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55820b9bdc10 0x55820b9b6510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:35.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.532+0000 7f20b3369740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55820b9c2d80 0x55820b9ae880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:35.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.532+0000 7f20b3369740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f20a81b0a20 con 0x55820b9c2d80 2024-03-18T16:13:35.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.532+0000 7f20b3369740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f20a81b26e0 con 0x55820b9bdc10 2024-03-18T16:13:35.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.532+0000 7f20b3369740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f20a81b10f0 con 0x55820b9bd220 2024-03-18T16:13:35.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.533+0000 7f20adc00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55820b9bd220 0x55820b9b5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:35.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.533+0000 7f20adc00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55820b9bd220 0x55820b9b5fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:55156/0 (socket says 172.21.15.39:55156) 2024-03-18T16:13:35.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.533+0000 7f20adc00640 1 -- 172.21.15.39:0/2678120255 learned_addr learned my addr 172.21.15.39:0/2678120255 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:13:35.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.533+0000 7f20ad200640 1 --2- 172.21.15.39:0/2678120255 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55820b9bdc10 0x55820b9b6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:35.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.533+0000 7f20ae600640 1 --2- 172.21.15.39:0/2678120255 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55820b9c2d80 0x55820b9ae880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:35.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.534+0000 7f20adc00640 1 -- 172.21.15.39:0/2678120255 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55820b9bdc10 msgr2=0x55820b9b6510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:35.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.534+0000 7f20adc00640 1 --2- 172.21.15.39:0/2678120255 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55820b9bdc10 0x55820b9b6510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:35.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.534+0000 7f20adc00640 1 -- 172.21.15.39:0/2678120255 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55820b9c2d80 msgr2=0x55820b9ae880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:35.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.534+0000 7f20adc00640 1 --2- 172.21.15.39:0/2678120255 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55820b9c2d80 0x55820b9ae880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:35.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.534+0000 7f20adc00640 1 -- 172.21.15.39:0/2678120255 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55820b9bce60 con 0x55820b9bd220 2024-03-18T16:13:35.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.534+0000 7f20ae600640 1 --2- 172.21.15.39:0/2678120255 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55820b9c2d80 0x55820b9ae880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:13:35.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.534+0000 7f20ad200640 1 --2- 172.21.15.39:0/2678120255 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55820b9bdc10 0x55820b9b6510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:13:35.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.535+0000 7f20adc00640 1 --2- 172.21.15.39:0/2678120255 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55820b9bd220 0x55820b9b5fd0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f209c009fd0 tx=0x7f209c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3b2b048e9dc53e54 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:35.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.536+0000 7f209a000640 1 -- 172.21.15.39:0/2678120255 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f209c064070 con 0x55820b9bd220 2024-03-18T16:13:35.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.536+0000 7f209a000640 1 -- 172.21.15.39:0/2678120255 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55820b92cbe0 con 0x55820b9bd220 2024-03-18T16:13:35.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.536+0000 7f209a000640 1 -- 172.21.15.39:0/2678120255 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f209c097520 con 0x55820b9bd220 2024-03-18T16:13:35.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.537+0000 7f20b3369740 1 -- 172.21.15.39:0/2678120255 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55820b9bd220 msgr2=0x55820b9b5fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:35.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.537+0000 7f20b3369740 1 --2- 172.21.15.39:0/2678120255 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55820b9bd220 0x55820b9b5fd0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f209c009fd0 tx=0x7f209c0046c0 comp rx=0 tx=0).stop 2024-03-18T16:13:35.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.537+0000 7f20b3369740 1 -- 172.21.15.39:0/2678120255 shutdown_connections 2024-03-18T16:13:35.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.537+0000 7f20b3369740 1 --2- 172.21.15.39:0/2678120255 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55820b9bdc10 0x55820b9b6510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:35.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.537+0000 7f20b3369740 1 --2- 172.21.15.39:0/2678120255 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55820b9c2d80 0x55820b9ae880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:35.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.537+0000 7f20b3369740 1 --2- 172.21.15.39:0/2678120255 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55820b9bd220 0x55820b9b5fd0 secure :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f209c009fd0 tx=0x7f209c0046c0 comp rx=0 tx=0).stop 2024-03-18T16:13:35.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.537+0000 7f20b3369740 1 -- 172.21.15.39:0/2678120255 >> 172.21.15.39:0/2678120255 conn(0x55820b9aff80 msgr2=0x55820b9b4a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:35.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.538+0000 7f20b3369740 1 -- 172.21.15.39:0/2678120255 shutdown_connections 2024-03-18T16:13:35.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.538+0000 7f20b3369740 1 -- 172.21.15.39:0/2678120255 wait complete. 2024-03-18T16:13:35.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.539+0000 7f20b3369740 1 -- start start 2024-03-18T16:13:35.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.539+0000 7f20b3369740 1 Processor -- start 2024-03-18T16:13:35.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.540+0000 7f20b3369740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55820b9bdc10 0x55820b9ef1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:35.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.541+0000 7f20b3369740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55820b9c2d80 0x55820b9ed7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:35.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.541+0000 7f20b3369740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55820b9edd20 0x55820b9ee1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:35.544 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.541+0000 7f20b3369740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f20a81b1f50 con 0x55820b9edd20 2024-03-18T16:13:35.544 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.541+0000 7f20b3369740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f20a81b17c0 con 0x55820b9bdc10 2024-03-18T16:13:35.544 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.541+0000 7f20b3369740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f20a81aea00 con 0x55820b9c2d80 2024-03-18T16:13:35.544 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.541+0000 7f20b3369740 10 client.0 _ll_register_callbacks cb 0x7f2094037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:13:35.544 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.541+0000 7f20ae600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55820b9edd20 0x55820b9ee1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:35.544 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.541+0000 7f20ae600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55820b9edd20 0x55820b9ee1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:36392/0 (socket says 172.21.15.39:36392) 2024-03-18T16:13:35.545 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.541+0000 7f20ae600640 1 -- 172.21.15.39:0/2395780995 learned_addr learned my addr 172.21.15.39:0/2395780995 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:13:35.545 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.542+0000 7f20adc00640 1 --2- 172.21.15.39:0/2395780995 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55820b9bdc10 0x55820b9ef1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:35.545 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.542+0000 7f20ae600640 1 -- 172.21.15.39:0/2395780995 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55820b9c2d80 msgr2=0x55820b9ed7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:35.545 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.542+0000 7f20ae600640 1 --2- 172.21.15.39:0/2395780995 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55820b9c2d80 0x55820b9ed7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:35.545 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.543+0000 7f20ae600640 1 -- 172.21.15.39:0/2395780995 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55820b9bdc10 msgr2=0x55820b9ef1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:35.545 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.543+0000 7f20ae600640 1 --2- 172.21.15.39:0/2395780995 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55820b9bdc10 0x55820b9ef1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:35.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.543+0000 7f20ae600640 1 -- 172.21.15.39:0/2395780995 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55820ba41bf0 con 0x55820b9edd20 2024-03-18T16:13:35.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.543+0000 7f20ae600640 1 --2- 172.21.15.39:0/2395780995 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55820b9edd20 0x55820b9ee1a0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f209402c0b0 tx=0x7f2094004330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74911378eee84bca server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:35.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.543+0000 7f208f400640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-03-18T16:13:35.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.544+0000 7f208f400640 1 -- 172.21.15.39:0/2395780995 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2094055020 con 0x55820b9edd20 2024-03-18T16:13:35.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.545+0000 7f208f400640 1 -- 172.21.15.39:0/2395780995 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f209403fe00 con 0x55820b9edd20 2024-03-18T16:13:35.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.545+0000 7f20b3369740 1 -- 172.21.15.39:0/2395780995 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55820b92cbe0 con 0x55820b9edd20 2024-03-18T16:13:35.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.545+0000 7f208f400640 1 -- 172.21.15.39:0/2395780995 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f2094043070 con 0x55820b9edd20 2024-03-18T16:13:35.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.545+0000 7f20b3369740 10 client.16008 fetch_fsmap learned FSMap version 36 2024-03-18T16:13:35.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.545+0000 7f20b3369740 1 -- 172.21.15.39:0/2395780995 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55820ba42ec0 con 0x55820b9edd20 2024-03-18T16:13:35.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.546+0000 7f208f400640 1 -- 172.21.15.39:0/2395780995 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55820b92cbe0 con 0x55820b9edd20 2024-03-18T16:13:35.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.546+0000 7f208f400640 1 -- 172.21.15.39:0/2395780995 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 36) ==== 8406+0+0 (secure 0 0 0) 0x7f209402c790 con 0x55820b9edd20 2024-03-18T16:13:35.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.546+0000 7f20b3369740 10 client.16008 fetch_fsmap finished waiting for FSMap version 36 2024-03-18T16:13:35.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.547+0000 7f20b3369740 10 client.16008 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:13:35.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.547+0000 7f20b3369740 20 client.16008 populate_metadata read hostname 'smithi039' 2024-03-18T16:13:35.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.547+0000 7f20b3369740 1 --2- 172.21.15.39:0/2395780995 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55820ba43910 0x55820ba45d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:35.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.547+0000 7f20b3369740 4 client.16008 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T16:13:35.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.547+0000 7f20b3369740 1 -- 172.21.15.39:0/2395780995 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55820b926730 con 0x55820ba43910 2024-03-18T16:13:35.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.547+0000 7f20adc00640 1 --2- 172.21.15.39:0/2395780995 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55820ba43910 0x55820ba45d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:35.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.550+0000 7f20adc00640 1 --2- 172.21.15.39:0/2395780995 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55820ba43910 0x55820ba45d30 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3d2c4a4f58d8c45 server_cookie=7c6516beafde180e in_seq=0 out_seq=0 2024-03-18T16:13:35.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.550+0000 7f208f400640 10 client.16008 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:13:35.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.550+0000 7f208f400640 1 -- 172.21.15.39:0/2395780995 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55820b926730 con 0x55820ba43910 2024-03-18T16:13:35.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.550+0000 7f208f400640 10 client.16008 handle_command_reply: tid=0 2024-03-18T16:13:35.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.551+0000 7f20b3369740 1 client.16008 shutdown 2024-03-18T16:13:35.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.551+0000 7f20b3369740 1 -- 172.21.15.39:0/2395780995 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55820b9edd20 msgr2=0x55820b9ee1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:35.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.551+0000 7f20b3369740 1 --2- 172.21.15.39:0/2395780995 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55820b9edd20 0x55820b9ee1a0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f209402c0b0 tx=0x7f2094004330 comp rx=0 tx=0).stop 2024-03-18T16:13:35.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.551+0000 7f20b3369740 1 -- 172.21.15.39:0/2395780995 shutdown_connections 2024-03-18T16:13:35.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.551+0000 7f20b3369740 1 --2- 172.21.15.39:0/2395780995 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55820ba43910 0x55820ba45d30 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:35.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.551+0000 7f20b3369740 1 --2- 172.21.15.39:0/2395780995 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55820b9edd20 0x55820b9ee1a0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:35.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.551+0000 7f20b3369740 1 --2- 172.21.15.39:0/2395780995 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55820b9c2d80 0x55820b9ed7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:35.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.551+0000 7f20b3369740 1 --2- 172.21.15.39:0/2395780995 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55820b9bdc10 0x55820b9ef1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:35.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.551+0000 7f20b3369740 1 -- 172.21.15.39:0/2395780995 >> 172.21.15.39:0/2395780995 conn(0x55820b9aff80 msgr2=0x55820b9b5f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:35.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.551+0000 7f208f400640 0 client.16008 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:13:35.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.552+0000 7f20b3369740 1 -- 172.21.15.39:0/2395780995 shutdown_connections 2024-03-18T16:13:35.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.552+0000 7f20b3369740 1 -- 172.21.15.39:0/2395780995 wait complete. 2024-03-18T16:13:35.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.552+0000 7f20b3369740 20 client.16008 trim_cache size 0 max 16384 2024-03-18T16:13:35.556 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-03-18T16:13:35.556 INFO:teuthology.orchestra.run.smithi039.stdout: "status": "no active scrubs running", 2024-03-18T16:13:35.556 INFO:teuthology.orchestra.run.smithi039.stdout: "scrubs": {} 2024-03-18T16:13:35.556 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-03-18T16:13:35.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.559+0000 7f20af000640 1 -- 172.21.15.39:0/208718403 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f20900a2800 msgr2=0x7f20900a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:35.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.559+0000 7f20af000640 1 --2- 172.21.15.39:0/208718403 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f20900a2800 0x7f20900a4cc0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f20a0009fd0 tx=0x7f20a0009680 comp rx=0 tx=0).stop 2024-03-18T16:13:35.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.559+0000 7f20af000640 1 -- 172.21.15.39:0/208718403 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f20a81038e0 msgr2=0x7f20a8070170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:35.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.559+0000 7f20af000640 1 --2- 172.21.15.39:0/208718403 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f20a81038e0 0x7f20a8070170 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f209c00b450 tx=0x7f209c00b910 comp rx=0 tx=0).stop 2024-03-18T16:13:35.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.559+0000 7f20af000640 1 -- 172.21.15.39:0/208718403 shutdown_connections 2024-03-18T16:13:35.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.559+0000 7f20af000640 1 --2- 172.21.15.39:0/208718403 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f20900a2800 0x7f20900a4cc0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:35.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.559+0000 7f20af000640 1 --2- 172.21.15.39:0/208718403 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f20a8105490 0x7f20a81a9ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:35.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.559+0000 7f20af000640 1 --2- 172.21.15.39:0/208718403 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f20a8104ad0 0x7f20a80706b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:35.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.559+0000 7f20af000640 1 --2- 172.21.15.39:0/208718403 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f20a81038e0 0x7f20a8070170 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:35.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.559+0000 7f20af000640 1 -- 172.21.15.39:0/208718403 >> 172.21.15.39:0/208718403 conn(0x7f20a80fee50 msgr2=0x7f20a8107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:35.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.560+0000 7f20af000640 1 -- 172.21.15.39:0/208718403 shutdown_connections 2024-03-18T16:13:35.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:35.560+0000 7f20af000640 1 -- 172.21.15.39:0/208718403 wait complete. 2024-03-18T16:13:35.617 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-18T16:13:35.617 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:13:36.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:36.085+0000 7fbba4800640 1 -- 172.21.15.39:0/3429055044 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbb9c1087e0 msgr2=0x7fbb9c108c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:36.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:36.085+0000 7fbba4800640 1 --2- 172.21.15.39:0/3429055044 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbb9c1087e0 0x7fbb9c108c00 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fbb90009860 tx=0x7fbb9002bb70 comp rx=0 tx=0).stop 2024-03-18T16:13:36.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:36.086+0000 7fbba4800640 1 -- 172.21.15.39:0/3429055044 shutdown_connections 2024-03-18T16:13:36.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:36.086+0000 7fbba4800640 1 --2- 172.21.15.39:0/3429055044 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbb9c110060 0x7fbb9c112450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:36.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:36.086+0000 7fbba4800640 1 --2- 172.21.15.39:0/3429055044 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbb9c109140 0x7fbb9c10fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:36.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:36.086+0000 7fbba4800640 1 --2- 172.21.15.39:0/3429055044 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbb9c1087e0 0x7fbb9c108c00 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:36.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:36.086+0000 7fbba4800640 1 -- 172.21.15.39:0/3429055044 >> 172.21.15.39:0/3429055044 conn(0x7fbb9c06e640 msgr2=0x7fbb9c070a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:36.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:36.087+0000 7fbba4800640 1 -- 172.21.15.39:0/3429055044 shutdown_connections 2024-03-18T16:13:36.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:36.087+0000 7fbba4800640 1 -- 172.21.15.39:0/3429055044 wait complete. 2024-03-18T16:13:36.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:36.088+0000 7fbba4800640 1 Processor -- start 2024-03-18T16:13:36.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:36.089+0000 7fbba4800640 1 -- start start 2024-03-18T16:13:36.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:36.089+0000 7fbba4800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbb9c1087e0 0x7fbb9c1a6960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:36.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:36.089+0000 7fbba4800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbb9c109140 0x7fbb9c1a6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:36.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:36.089+0000 7fbba4800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbb9c110060 0x7fbb9c1adf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:36.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:36.089+0000 7fbba4800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fbb9c10b490 con 0x7fbb9c109140 2024-03-18T16:13:36.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:36.089+0000 7fbba4800640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fbb9c10b310 con 0x7fbb9c110060 2024-03-18T16:13:36.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:36.089+0000 7fbba4800640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fbb9c10b610 con 0x7fbb9c1087e0 2024-03-18T16:13:36.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:36.090+0000 7fbba1e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbb9c1087e0 0x7fbb9c1a6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:36.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:36.090+0000 7fbba1e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbb9c1087e0 0x7fbb9c1a6960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:55166/0 (socket says 172.21.15.39:55166) 2024-03-18T16:13:36.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:36.090+0000 7fbba1e00640 1 -- 172.21.15.39:0/2221497308 learned_addr learned my addr 172.21.15.39:0/2221497308 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:13:36.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:36.090+0000 7fbba2800640 1 --2- 172.21.15.39:0/2221497308 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbb9c110060 0x7fbb9c1adf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:36.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:36.090+0000 7fbba1400640 1 --2- 172.21.15.39:0/2221497308 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbb9c109140 0x7fbb9c1a6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:36.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:36.090+0000 7fbba1e00640 1 -- 172.21.15.39:0/2221497308 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbb9c110060 msgr2=0x7fbb9c1adf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:36.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:36.090+0000 7fbba1e00640 1 --2- 172.21.15.39:0/2221497308 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbb9c110060 0x7fbb9c1adf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:36.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:36.090+0000 7fbba1e00640 1 -- 172.21.15.39:0/2221497308 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbb9c109140 msgr2=0x7fbb9c1a6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:36.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:36.090+0000 7fbba1e00640 1 --2- 172.21.15.39:0/2221497308 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbb9c109140 0x7fbb9c1a6ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:36.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:36.091+0000 7fbba1e00640 1 -- 172.21.15.39:0/2221497308 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb9c1ae580 con 0x7fbb9c1087e0 2024-03-18T16:13:36.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:36.091+0000 7fbba1e00640 1 --2- 172.21.15.39:0/2221497308 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbb9c1087e0 0x7fbb9c1a6960 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fbb90000c00 tx=0x7fbb90032460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:36.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:36.092+0000 7fbb96a00640 1 -- 172.21.15.39:0/2221497308 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb90039040 con 0x7fbb9c1087e0 2024-03-18T16:13:36.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:36.092+0000 7fbb96a00640 1 -- 172.21.15.39:0/2221497308 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbb90042070 con 0x7fbb9c1087e0 2024-03-18T16:13:36.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:36.092+0000 7fbb96a00640 1 -- 172.21.15.39:0/2221497308 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb900046c0 con 0x7fbb9c1087e0 2024-03-18T16:13:36.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:36.092+0000 7fbba4800640 1 -- 172.21.15.39:0/2221497308 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb9c1ae810 con 0x7fbb9c1087e0 2024-03-18T16:13:36.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:36.092+0000 7fbba4800640 1 -- 172.21.15.39:0/2221497308 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb9c1aecb0 con 0x7fbb9c1087e0 2024-03-18T16:13:36.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:36.096+0000 7fbba4800640 1 -- 172.21.15.39:0/2221497308 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb9c1aefd0 con 0x7fbb9c1087e0 2024-03-18T16:13:36.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:36.102+0000 7fbb96a00640 1 -- 172.21.15.39:0/2221497308 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fbb90004860 con 0x7fbb9c1087e0 2024-03-18T16:13:36.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:36.107+0000 7fbb96a00640 1 --2- 172.21.15.39:0/2221497308 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fbb840a2a00 0x7fbb840a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:36.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:36.107+0000 7fbb96a00640 1 -- 172.21.15.39:0/2221497308 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fbb900efcf0 con 0x7fbb9c1087e0 2024-03-18T16:13:36.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:36.108+0000 7fbba1400640 1 --2- 172.21.15.39:0/2221497308 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fbb840a2a00 0x7fbb840a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:36.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:36.110+0000 7fbba1400640 1 --2- 172.21.15.39:0/2221497308 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fbb840a2a00 0x7fbb840a4ec0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fbb9c1a7e70 tx=0x7fbb8c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:36.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:36.113+0000 7fbb96a00640 1 -- 172.21.15.39:0/2221497308 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fbb9003c390 con 0x7fbb9c1087e0 2024-03-18T16:13:36.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:36.587+0000 7fbba4800640 1 -- 172.21.15.39:0/2221497308 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbb9c1af1a0 con 0x7fbb9c1087e0 2024-03-18T16:13:36.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:36.588+0000 7fbb96a00640 1 -- 172.21.15.39:0/2221497308 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+11950 (secure 0 0 0) 0x7fbb900bca20 con 0x7fbb9c1087e0 2024-03-18T16:13:36.591 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:13:36.591 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2520226608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2520226608},{"type":"v1","addr":"172.21.15.39:6835","nonce":2520226608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/719341763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":719341763},{"type":"v1","addr":"172.21.15.39:6837","nonce":719341763}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":36,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T16:06:44.180425+0000","modified":"2024-03-18T16:13:04.655685+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24575,"mds_1":24556,"mds_2":24535,"mds_3":24524,"mds_4":14463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14445":{"gid":14445,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.39:6839/656513813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":656513813},{"type":"v1","addr":"172.21.15.39:6839","nonce":656513813}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14463":{"gid":14463,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":13,"addr":"172.21.15.39:6841/3515326042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3515326042},{"type":"v1","addr":"172.21.15.39:6841","nonce":3515326042}]},"join_fscid":-1,"export_targets":[3],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6833/931350738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":931350738},{"type":"v1","addr":"172.21.15.194:6833","nonce":931350738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6833/3874209621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":3874209621},{"type":"v1","addr":"172.21.15.196:6833","nonce":3874209621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6835/188385470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":188385470},{"type":"v1","addr":"172.21.15.194:6835","nonce":188385470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6835/3949984309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":3949984309},{"type":"v1","addr":"172.21.15.196:6835","nonce":3949984309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":15,"addr":"172.21.15.194:6837/1301464246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":1301464246},{"type":"v1","addr":"172.21.15.194:6837","nonce":1301464246}]},"join_fscid":-1,"export_targets":[1,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":16,"addr":"172.21.15.196:6837/895247251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":895247251},{"type":"v1","addr":"172.21.15.196:6837","nonce":895247251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.196:6839/2001427268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6838","nonce":2001427268},{"type":"v1","addr":"172.21.15.196:6839","nonce":2001427268}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24575":{"gid":24575,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.194:6839/274911158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":274911158},{"type":"v1","addr":"172.21.15.194:6839","nonce":274911158}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24575,"qdb_cluster":[14463,24524,24575,24556,24535]},"id":1}]} 2024-03-18T16:13:36.591 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 36 2024-03-18T16:13:36.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:36.594+0000 7fbba4800640 1 -- 172.21.15.39:0/2221497308 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fbb840a2a00 msgr2=0x7fbb840a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:36.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:36.594+0000 7fbba4800640 1 --2- 172.21.15.39:0/2221497308 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fbb840a2a00 0x7fbb840a4ec0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fbb9c1a7e70 tx=0x7fbb8c005f90 comp rx=0 tx=0).stop 2024-03-18T16:13:36.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:36.594+0000 7fbba4800640 1 -- 172.21.15.39:0/2221497308 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbb9c1087e0 msgr2=0x7fbb9c1a6960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:36.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:36.594+0000 7fbba4800640 1 --2- 172.21.15.39:0/2221497308 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbb9c1087e0 0x7fbb9c1a6960 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fbb90000c00 tx=0x7fbb90032460 comp rx=0 tx=0).stop 2024-03-18T16:13:36.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:36.595+0000 7fbba4800640 1 -- 172.21.15.39:0/2221497308 shutdown_connections 2024-03-18T16:13:36.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:36.595+0000 7fbba4800640 1 --2- 172.21.15.39:0/2221497308 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbb9c110060 0x7fbb9c1adf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:36.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:36.595+0000 7fbba4800640 1 --2- 172.21.15.39:0/2221497308 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fbb840a2a00 0x7fbb840a4ec0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:36.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:36.595+0000 7fbba4800640 1 --2- 172.21.15.39:0/2221497308 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbb9c109140 0x7fbb9c1a6ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:36.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:36.595+0000 7fbba4800640 1 --2- 172.21.15.39:0/2221497308 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbb9c1087e0 0x7fbb9c1a6960 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:36.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:36.595+0000 7fbba4800640 1 -- 172.21.15.39:0/2221497308 >> 172.21.15.39:0/2221497308 conn(0x7fbb9c06e640 msgr2=0x7fbb9c070370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:36.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:36.595+0000 7fbba4800640 1 -- 172.21.15.39:0/2221497308 shutdown_connections 2024-03-18T16:13:36.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:36.595+0000 7fbba4800640 1 -- 172.21.15.39:0/2221497308 wait complete. 2024-03-18T16:13:36.630 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-18T16:13:37.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:13:36 smithi196 ceph-mon[36307]: pgmap v528: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 64 MiB/s wr, 8.24k op/s 2024-03-18T16:13:37.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:13:36 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/2221497308' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:13:37.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:13:36 smithi194 ceph-mon[36200]: pgmap v528: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 64 MiB/s wr, 8.24k op/s 2024-03-18T16:13:37.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:13:36 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/2221497308' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:13:37.340 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:13:36 smithi039 ceph-mon[31721]: pgmap v528: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 64 MiB/s wr, 8.24k op/s 2024-03-18T16:13:37.340 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:13:36 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/2221497308' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:13:38.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.929+0000 7f7e5ec00640 1 -- 172.21.15.39:0/3476953105 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7e5810db00 msgr2=0x7f7e5810df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:38.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.929+0000 7f7e5ec00640 1 --2- 172.21.15.39:0/3476953105 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7e5810db00 0x7f7e5810df80 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f7e48009900 tx=0x7f7e4802bc80 comp rx=0 tx=0).stop 2024-03-18T16:13:38.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.931+0000 7f7e5ec00640 1 -- 172.21.15.39:0/3476953105 shutdown_connections 2024-03-18T16:13:38.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.931+0000 7f7e5ec00640 1 --2- 172.21.15.39:0/3476953105 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7e5810db00 0x7f7e5810df80 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:38.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.931+0000 7f7e5ec00640 1 --2- 172.21.15.39:0/3476953105 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7e5810b150 0x7f7e5810d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:38.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.931+0000 7f7e5ec00640 1 --2- 172.21.15.39:0/3476953105 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7e581087e0 0x7f7e5810ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:38.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.931+0000 7f7e5ec00640 1 -- 172.21.15.39:0/3476953105 >> 172.21.15.39:0/3476953105 conn(0x7f7e5806e860 msgr2=0x7f7e58070ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:38.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.932+0000 7f7e5ec00640 1 -- 172.21.15.39:0/3476953105 shutdown_connections 2024-03-18T16:13:38.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.933+0000 7f7e5ec00640 1 -- 172.21.15.39:0/3476953105 wait complete. 2024-03-18T16:13:38.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.934+0000 7f7e5ec00640 1 Processor -- start 2024-03-18T16:13:38.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.936+0000 7f7e5ec00640 1 -- start start 2024-03-18T16:13:38.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.936+0000 7f7e5ec00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7e581087e0 0x7f7e58116e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:38.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.936+0000 7f7e5ec00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7e5810b150 0x7f7e581173b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:38.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.936+0000 7f7e5ec00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7e5810db00 0x7f7e581b6a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:38.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.936+0000 7f7e5ec00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f7e5811c340 con 0x7f7e5810db00 2024-03-18T16:13:38.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.936+0000 7f7e5ec00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f7e5811c1c0 con 0x7f7e5810b150 2024-03-18T16:13:38.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.936+0000 7f7e5ec00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f7e5811c4c0 con 0x7f7e581087e0 2024-03-18T16:13:38.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.937+0000 7f7e57400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7e5810b150 0x7f7e581173b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:38.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.937+0000 7f7e57400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7e5810b150 0x7f7e581173b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:59986/0 (socket says 172.21.15.39:59986) 2024-03-18T16:13:38.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.937+0000 7f7e57400640 1 -- 172.21.15.39:0/2347805924 learned_addr learned my addr 172.21.15.39:0/2347805924 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:13:38.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.937+0000 7f7e57400640 1 -- 172.21.15.39:0/2347805924 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7e581087e0 msgr2=0x7f7e58116e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:38.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.937+0000 7f7e57400640 1 --2- 172.21.15.39:0/2347805924 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7e581087e0 0x7f7e58116e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:38.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.937+0000 7f7e57400640 1 -- 172.21.15.39:0/2347805924 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7e5810db00 msgr2=0x7f7e581b6a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:38.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.937+0000 7f7e57400640 1 --2- 172.21.15.39:0/2347805924 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7e5810db00 0x7f7e581b6a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:38.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.937+0000 7f7e57400640 1 -- 172.21.15.39:0/2347805924 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e581b6f90 con 0x7f7e5810b150 2024-03-18T16:13:38.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.938+0000 7f7e57400640 1 --2- 172.21.15.39:0/2347805924 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7e5810b150 0x7f7e581173b0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f7e5000ba50 tx=0x7f7e5000bf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:38.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.938+0000 7f7e54c00640 1 -- 172.21.15.39:0/2347805924 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e50019070 con 0x7f7e5810b150 2024-03-18T16:13:38.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.938+0000 7f7e5ec00640 1 -- 172.21.15.39:0/2347805924 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e581b7160 con 0x7f7e5810b150 2024-03-18T16:13:38.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.939+0000 7f7e5ec00640 1 -- 172.21.15.39:0/2347805924 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e581b7620 con 0x7f7e5810b150 2024-03-18T16:13:38.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.939+0000 7f7e54c00640 1 -- 172.21.15.39:0/2347805924 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e5000cbe0 con 0x7f7e5810b150 2024-03-18T16:13:38.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.939+0000 7f7e54c00640 1 -- 172.21.15.39:0/2347805924 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e500076e0 con 0x7f7e5810b150 2024-03-18T16:13:38.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.948+0000 7f7e54c00640 1 -- 172.21.15.39:0/2347805924 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f7e5001c070 con 0x7f7e5810b150 2024-03-18T16:13:38.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.951+0000 7f7e61831740 1 Processor -- start 2024-03-18T16:13:38.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.956+0000 7f7e61831740 1 -- start start 2024-03-18T16:13:38.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.956+0000 7f7e61831740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f4fdd6b8c0 0x55f4fdd6bce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:38.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.956+0000 7f7e61831740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f4fdd6c2b0 0x55f4fdd70db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:38.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.956+0000 7f7e61831740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f4fdd71420 0x55f4fdd738b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:38.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.956+0000 7f7e61831740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55f4fdd333b0 con 0x55f4fdd6b8c0 2024-03-18T16:13:38.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.956+0000 7f7e61831740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55f4fdcd0450 con 0x55f4fdd6c2b0 2024-03-18T16:13:38.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.956+0000 7f7e61831740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55f4fdc78cb0 con 0x55f4fdd71420 2024-03-18T16:13:38.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.957+0000 7f7e57e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f4fdd6c2b0 0x55f4fdd70db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:38.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.957+0000 7f7e57e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f4fdd6c2b0 0x55f4fdd70db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:59992/0 (socket says 172.21.15.39:59992) 2024-03-18T16:13:38.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.957+0000 7f7e57e00640 1 -- 172.21.15.39:0/2714960091 learned_addr learned my addr 172.21.15.39:0/2714960091 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:13:38.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.957+0000 7f7e54c00640 1 --2- 172.21.15.39:0/2347805924 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f7e3c0a2880 0x7f7e3c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:38.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.957+0000 7f7e57e00640 1 -- 172.21.15.39:0/2714960091 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f4fdd71420 msgr2=0x55f4fdd738b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:38.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.957+0000 7f7e57e00640 1 --2- 172.21.15.39:0/2714960091 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f4fdd71420 0x55f4fdd738b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:38.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.957+0000 7f7e57e00640 1 -- 172.21.15.39:0/2714960091 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f4fdd6b8c0 msgr2=0x55f4fdd6bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:38.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.957+0000 7f7e57e00640 1 --2- 172.21.15.39:0/2714960091 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f4fdd6b8c0 0x55f4fdd6bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:38.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.957+0000 7f7e57e00640 1 -- 172.21.15.39:0/2714960091 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f4fdc9a1f0 con 0x55f4fdd6c2b0 2024-03-18T16:13:38.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.958+0000 7f7e54c00640 1 -- 172.21.15.39:0/2347805924 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f7e500ce840 con 0x7f7e5810b150 2024-03-18T16:13:38.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.958+0000 7f7e57e00640 1 --2- 172.21.15.39:0/2714960091 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f4fdd6c2b0 0x55f4fdd70db0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f7e44009820 tx=0x7f7e44009ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b02d5c9f2384e4dd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:38.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.959+0000 7f7e5cc00640 1 --2- 172.21.15.39:0/2347805924 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f7e3c0a2880 0x7f7e3c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:38.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.959+0000 7f7e4d200640 1 -- 172.21.15.39:0/2714960091 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e44015020 con 0x55f4fdd6c2b0 2024-03-18T16:13:38.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.959+0000 7f7e4d200640 1 -- 172.21.15.39:0/2714960091 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e44011420 con 0x55f4fdd6c2b0 2024-03-18T16:13:38.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.961+0000 7f7e61831740 1 -- 172.21.15.39:0/2714960091 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f4fdd6c2b0 msgr2=0x55f4fdd70db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:38.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.961+0000 7f7e61831740 1 --2- 172.21.15.39:0/2714960091 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f4fdd6c2b0 0x55f4fdd70db0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f7e44009820 tx=0x7f7e44009ce0 comp rx=0 tx=0).stop 2024-03-18T16:13:38.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.961+0000 7f7e5cc00640 1 --2- 172.21.15.39:0/2347805924 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f7e3c0a2880 0x7f7e3c0a4d40 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f7e48005ec0 tx=0x7f7e4802c900 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:38.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.962+0000 7f7e61831740 1 -- 172.21.15.39:0/2714960091 shutdown_connections 2024-03-18T16:13:38.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.962+0000 7f7e61831740 1 --2- 172.21.15.39:0/2714960091 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f4fdd6c2b0 0x55f4fdd70db0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:38.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.962+0000 7f7e61831740 1 --2- 172.21.15.39:0/2714960091 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f4fdd71420 0x55f4fdd738b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:38.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.962+0000 7f7e61831740 1 --2- 172.21.15.39:0/2714960091 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f4fdd6b8c0 0x55f4fdd6bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:38.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.962+0000 7f7e61831740 1 -- 172.21.15.39:0/2714960091 >> 172.21.15.39:0/2714960091 conn(0x55f4fdd66e30 msgr2=0x55f4fdd67230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:38.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.974+0000 7f7e61831740 1 -- 172.21.15.39:0/2714960091 shutdown_connections 2024-03-18T16:13:38.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.974+0000 7f7e61831740 1 -- 172.21.15.39:0/2714960091 wait complete. 2024-03-18T16:13:38.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.975+0000 7f7e61831740 1 -- start start 2024-03-18T16:13:38.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.975+0000 7f7e61831740 1 Processor -- start 2024-03-18T16:13:38.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.980+0000 7f7e61831740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f4fdd6b8c0 0x55f4fdd75c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:38.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.980+0000 7f7e61831740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f4fdd6c2b0 0x55f4fdd76160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:38.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.980+0000 7f7e61831740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f4fdd71420 0x55f4fdd7d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:38.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.981+0000 7f7e57e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f4fdd6b8c0 0x55f4fdd75c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:38.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.981+0000 7f7e57e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f4fdd6b8c0 0x55f4fdd75c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:60006/0 (socket says 172.21.15.39:60006) 2024-03-18T16:13:38.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.981+0000 7f7e61831740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55f4fdc8c110 con 0x55f4fdd71420 2024-03-18T16:13:38.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.981+0000 7f7e61831740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55f4fdccc4f0 con 0x55f4fdd6b8c0 2024-03-18T16:13:38.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.981+0000 7f7e61831740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55f4fdd74470 con 0x55f4fdd6c2b0 2024-03-18T16:13:38.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.981+0000 7f7e61831740 10 client.0 _ll_register_callbacks cb 0x55f4fdc79b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:13:38.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.981+0000 7f7e57400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f4fdd6c2b0 0x55f4fdd76160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:38.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.981+0000 7f7e57e00640 1 -- 172.21.15.39:0/85675501 learned_addr learned my addr 172.21.15.39:0/85675501 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:13:38.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.982+0000 7f7e57e00640 1 -- 172.21.15.39:0/85675501 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f4fdd6c2b0 msgr2=0x55f4fdd76160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:38.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.982+0000 7f7e57e00640 1 --2- 172.21.15.39:0/85675501 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f4fdd6c2b0 0x55f4fdd76160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:38.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.982+0000 7f7e57e00640 1 -- 172.21.15.39:0/85675501 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f4fdd71420 msgr2=0x55f4fdd7d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:38.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.982+0000 7f7e57e00640 1 --2- 172.21.15.39:0/85675501 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f4fdd71420 0x55f4fdd7d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:38.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.982+0000 7f7e57e00640 1 -- 172.21.15.39:0/85675501 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f4fd9e43f0 con 0x55f4fdd6b8c0 2024-03-18T16:13:38.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.983+0000 7f7e57e00640 1 --2- 172.21.15.39:0/85675501 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f4fdd6b8c0 0x55f4fdd75c20 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f7e44016000 tx=0x7f7e44011880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ab888dbc7609111 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:38.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.983+0000 7f7e41600640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:13:38.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.983+0000 7f7e41600640 1 -- 172.21.15.39:0/85675501 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e4400bce0 con 0x55f4fdd6b8c0 2024-03-18T16:13:38.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.983+0000 7f7e41600640 1 -- 172.21.15.39:0/85675501 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f7e44010840 con 0x55f4fdd6b8c0 2024-03-18T16:13:38.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.984+0000 7f7e61831740 1 -- 172.21.15.39:0/85675501 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7e44011420 con 0x55f4fdd6b8c0 2024-03-18T16:13:38.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.984+0000 7f7e41600640 1 -- 172.21.15.39:0/85675501 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f7e4402c050 con 0x55f4fdd6b8c0 2024-03-18T16:13:38.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.984+0000 7f7e61831740 10 client.26297 fetch_fsmap learned FSMap version 36 2024-03-18T16:13:38.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.985+0000 7f7e61831740 1 -- 172.21.15.39:0/85675501 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55f4fdaae7d0 con 0x55f4fdd6b8c0 2024-03-18T16:13:38.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.985+0000 7f7e41600640 1 -- 172.21.15.39:0/85675501 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e4400be80 con 0x55f4fdd6b8c0 2024-03-18T16:13:38.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.986+0000 7f7e41600640 1 -- 172.21.15.39:0/85675501 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 36) ==== 8406+0+0 (secure 0 0 0) 0x7f7e44021020 con 0x55f4fdd6b8c0 2024-03-18T16:13:38.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.986+0000 7f7e61831740 10 client.26297 fetch_fsmap finished waiting for FSMap version 36 2024-03-18T16:13:38.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.986+0000 7f7e61831740 10 client.26297 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-03-18T16:13:38.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.986+0000 7f7e61831740 20 client.26297 populate_metadata read hostname 'smithi039' 2024-03-18T16:13:38.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.986+0000 7f7e61831740 1 --2- 172.21.15.39:0/85675501 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x55f4fdd818f0 0x55f4fdd83d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:38.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.986+0000 7f7e61831740 4 client.26297 mds_command: new command op to 14463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:13:38.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.986+0000 7f7e61831740 1 -- 172.21.15.39:0/85675501 --> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f7e44011420 con 0x55f4fdd818f0 2024-03-18T16:13:38.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.988+0000 7f7e57400640 1 --2- 172.21.15.39:0/85675501 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x55f4fdd818f0 0x55f4fdd83d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:38.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.990+0000 7f7e57400640 1 --2- 172.21.15.39:0/85675501 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x55f4fdd818f0 0x55f4fdd83d50 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=d3e9c8b3634b483f server_cookie=5e8979372cc5d81d in_seq=0 out_seq=0 2024-03-18T16:13:38.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.990+0000 7f7e41600640 10 client.26297 ms_handle_connect on v2:172.21.15.39:6840/3515326042 2024-03-18T16:13:38.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.993+0000 7f7e41600640 1 -- 172.21.15.39:0/85675501 <== mds.4 v2:172.21.15.39:6840/3515326042 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f7e44011420 con 0x55f4fdd818f0 2024-03-18T16:13:38.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.993+0000 7f7e41600640 10 client.26297 handle_command_reply: tid=0 2024-03-18T16:13:38.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.993+0000 7f7e61831740 1 client.26297 shutdown 2024-03-18T16:13:38.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.994+0000 7f7e61831740 1 -- 172.21.15.39:0/85675501 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f4fdd6b8c0 msgr2=0x55f4fdd75c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:38.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.994+0000 7f7e61831740 1 --2- 172.21.15.39:0/85675501 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f4fdd6b8c0 0x55f4fdd75c20 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f7e44016000 tx=0x7f7e44011880 comp rx=0 tx=0).stop 2024-03-18T16:13:38.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.995+0000 7f7e61831740 1 -- 172.21.15.39:0/85675501 shutdown_connections 2024-03-18T16:13:38.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.995+0000 7f7e61831740 1 --2- 172.21.15.39:0/85675501 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x55f4fdd818f0 0x55f4fdd83d50 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:38.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.995+0000 7f7e61831740 1 --2- 172.21.15.39:0/85675501 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f4fdd71420 0x55f4fdd7d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:38.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.995+0000 7f7e61831740 1 --2- 172.21.15.39:0/85675501 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f4fdd6c2b0 0x55f4fdd76160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:38.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.995+0000 7f7e61831740 1 --2- 172.21.15.39:0/85675501 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f4fdd6b8c0 0x55f4fdd75c20 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:38.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.995+0000 7f7e61831740 1 -- 172.21.15.39:0/85675501 >> 172.21.15.39:0/85675501 conn(0x55f4fdd66e30 msgr2=0x55f4fdd661f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:38.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.995+0000 7f7e41600640 0 client.26297 ms_handle_reset on v2:172.21.15.39:6840/3515326042 2024-03-18T16:13:38.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.997+0000 7f7e61831740 1 -- 172.21.15.39:0/85675501 shutdown_connections 2024-03-18T16:13:38.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.997+0000 7f7e61831740 1 -- 172.21.15.39:0/85675501 wait complete. 2024-03-18T16:13:38.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:38.997+0000 7f7e61831740 20 client.26297 trim_cache size 0 max 16384 2024-03-18T16:13:39.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.029+0000 7f7e61831740 1 Processor -- start 2024-03-18T16:13:39.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.030+0000 7f7e61831740 1 -- start start 2024-03-18T16:13:39.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.030+0000 7f7e61831740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f4fdd6b8c0 0x55f4fdd67210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:39.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.030+0000 7f7e61831740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f4fdd6c2b0 0x55f4fdd7d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:39.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.030+0000 7f7e61831740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f4fdd71420 0x55f4fdd88610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:39.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.031+0000 7f7e61831740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f7e3800fc10 con 0x55f4fdd6c2b0 2024-03-18T16:13:39.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.031+0000 7f7e61831740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f7e380118d0 con 0x55f4fdd71420 2024-03-18T16:13:39.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.031+0000 7f7e5cc00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f4fdd71420 0x55f4fdd88610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:39.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.031+0000 7f7e5cc00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f4fdd71420 0x55f4fdd88610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:60014/0 (socket says 172.21.15.39:60014) 2024-03-18T16:13:39.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.031+0000 7f7e5cc00640 1 -- 172.21.15.39:0/3111502597 learned_addr learned my addr 172.21.15.39:0/3111502597 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:13:39.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.031+0000 7f7e61831740 1 -- 172.21.15.39:0/3111502597 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f7e380102e0 con 0x55f4fdd6b8c0 2024-03-18T16:13:39.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.031+0000 7f7e57400640 1 --2- 172.21.15.39:0/3111502597 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f4fdd6c2b0 0x55f4fdd7d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:39.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.031+0000 7f7e57e00640 1 --2- 172.21.15.39:0/3111502597 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f4fdd6b8c0 0x55f4fdd67210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:39.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.032+0000 7f7e5cc00640 1 -- 172.21.15.39:0/3111502597 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f4fdd6b8c0 msgr2=0x55f4fdd67210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:39.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.032+0000 7f7e5cc00640 1 --2- 172.21.15.39:0/3111502597 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f4fdd6b8c0 0x55f4fdd67210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:39.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.032+0000 7f7e5cc00640 1 -- 172.21.15.39:0/3111502597 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f4fdd6c2b0 msgr2=0x55f4fdd7d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:39.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.033+0000 7f7e5cc00640 1 --2- 172.21.15.39:0/3111502597 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f4fdd6c2b0 0x55f4fdd7d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:39.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.033+0000 7f7e5cc00640 1 -- 172.21.15.39:0/3111502597 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f4fdd6cb40 con 0x55f4fdd71420 2024-03-18T16:13:39.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.033+0000 7f7e57400640 1 --2- 172.21.15.39:0/3111502597 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f4fdd6c2b0 0x55f4fdd7d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:13:39.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.034+0000 7f7e5cc00640 1 --2- 172.21.15.39:0/3111502597 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f4fdd71420 0x55f4fdd88610 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f7e48009900 tx=0x7f7e48005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f0e704d95a467c9d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:39.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.035+0000 7f7e43e00640 1 -- 172.21.15.39:0/3111502597 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e4804d020 con 0x55f4fdd71420 2024-03-18T16:13:39.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.035+0000 7f7e43e00640 1 -- 172.21.15.39:0/3111502597 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e48042030 con 0x55f4fdd71420 2024-03-18T16:13:39.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.035+0000 7f7e43e00640 1 -- 172.21.15.39:0/3111502597 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e480035f0 con 0x55f4fdd71420 2024-03-18T16:13:39.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.037+0000 7f7e61831740 1 -- 172.21.15.39:0/3111502597 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f4fdd71420 msgr2=0x55f4fdd88610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:39.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.037+0000 7f7e61831740 1 --2- 172.21.15.39:0/3111502597 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f4fdd71420 0x55f4fdd88610 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f7e48009900 tx=0x7f7e48005fd0 comp rx=0 tx=0).stop 2024-03-18T16:13:39.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.037+0000 7f7e61831740 1 -- 172.21.15.39:0/3111502597 shutdown_connections 2024-03-18T16:13:39.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.037+0000 7f7e61831740 1 --2- 172.21.15.39:0/3111502597 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f4fdd71420 0x55f4fdd88610 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:39.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.038+0000 7f7e61831740 1 --2- 172.21.15.39:0/3111502597 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f4fdd6c2b0 0x55f4fdd7d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:39.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.038+0000 7f7e61831740 1 --2- 172.21.15.39:0/3111502597 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f4fdd6b8c0 0x55f4fdd67210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:39.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.038+0000 7f7e61831740 1 -- 172.21.15.39:0/3111502597 >> 172.21.15.39:0/3111502597 conn(0x55f4fdd66e30 msgr2=0x55f4fdd661f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:39.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.039+0000 7f7e61831740 1 -- 172.21.15.39:0/3111502597 shutdown_connections 2024-03-18T16:13:39.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.039+0000 7f7e61831740 1 -- 172.21.15.39:0/3111502597 wait complete. 2024-03-18T16:13:39.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.039+0000 7f7e61831740 1 -- start start 2024-03-18T16:13:39.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.040+0000 7f7e61831740 1 Processor -- start 2024-03-18T16:13:39.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.040+0000 7f7e61831740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f4fdd6b8c0 0x55f4fdd7a120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:39.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.041+0000 7f7e61831740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f4fdd6c2b0 0x55f4fdd7a660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:39.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.041+0000 7f7e61831740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f4fdd71420 0x55f4fdde5c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:39.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.041+0000 7f7e61831740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f7e38011140 con 0x55f4fdd71420 2024-03-18T16:13:39.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.041+0000 7f7e61831740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f7e380109b0 con 0x55f4fdd6c2b0 2024-03-18T16:13:39.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.041+0000 7f7e61831740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f7e3800dbf0 con 0x55f4fdd6b8c0 2024-03-18T16:13:39.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.041+0000 7f7e61831740 10 client.0 _ll_register_callbacks cb 0x7f7e44009670 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:13:39.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.041+0000 7f7e57e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f4fdd6b8c0 0x55f4fdd7a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:39.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.041+0000 7f7e5cc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f4fdd71420 0x55f4fdde5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:39.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.041+0000 7f7e57400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f4fdd6c2b0 0x55f4fdd7a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:39.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.042+0000 7f7e57400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f4fdd6c2b0 0x55f4fdd7a660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:60022/0 (socket says 172.21.15.39:60022) 2024-03-18T16:13:39.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.042+0000 7f7e57400640 1 -- 172.21.15.39:0/2254968780 learned_addr learned my addr 172.21.15.39:0/2254968780 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:13:39.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.043+0000 7f7e57400640 1 -- 172.21.15.39:0/2254968780 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f4fdd6b8c0 msgr2=0x55f4fdd7a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:39.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.043+0000 7f7e57400640 1 --2- 172.21.15.39:0/2254968780 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f4fdd6b8c0 0x55f4fdd7a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:39.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.043+0000 7f7e57400640 1 -- 172.21.15.39:0/2254968780 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f4fdd71420 msgr2=0x55f4fdde5c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:39.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.043+0000 7f7e57400640 1 --2- 172.21.15.39:0/2254968780 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f4fdd71420 0x55f4fdde5c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:39.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.043+0000 7f7e57400640 1 -- 172.21.15.39:0/2254968780 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f4fdde6180 con 0x55f4fdd6c2b0 2024-03-18T16:13:39.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.044+0000 7f7e57400640 1 --2- 172.21.15.39:0/2254968780 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f4fdd6c2b0 0x55f4fdd7a660 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f7e5009afb0 tx=0x7f7e5000e040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=741e96b4b7f7eead server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:39.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.044+0000 7f7e41600640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:13:39.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.044+0000 7f7e41600640 1 -- 172.21.15.39:0/2254968780 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e5007e060 con 0x55f4fdd6c2b0 2024-03-18T16:13:39.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.045+0000 7f7e61831740 1 -- 172.21.15.39:0/2254968780 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7e48042030 con 0x55f4fdd6c2b0 2024-03-18T16:13:39.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.045+0000 7f7e41600640 1 -- 172.21.15.39:0/2254968780 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f7e5009b050 con 0x55f4fdd6c2b0 2024-03-18T16:13:39.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.047+0000 7f7e41600640 1 -- 172.21.15.39:0/2254968780 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f7e500b9070 con 0x55f4fdd6c2b0 2024-03-18T16:13:39.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.047+0000 7f7e61831740 10 client.26303 fetch_fsmap learned FSMap version 36 2024-03-18T16:13:39.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.047+0000 7f7e61831740 1 -- 172.21.15.39:0/2254968780 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55f4fdde74b0 con 0x55f4fdd6c2b0 2024-03-18T16:13:39.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.048+0000 7f7e41600640 1 -- 172.21.15.39:0/2254968780 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e48042030 con 0x55f4fdd6c2b0 2024-03-18T16:13:39.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.048+0000 7f7e41600640 1 -- 172.21.15.39:0/2254968780 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 36) ==== 8406+0+0 (secure 0 0 0) 0x7f7e5009a240 con 0x55f4fdd6c2b0 2024-03-18T16:13:39.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.048+0000 7f7e61831740 10 client.26303 fetch_fsmap finished waiting for FSMap version 36 2024-03-18T16:13:39.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.048+0000 7f7e61831740 10 client.26303 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-03-18T16:13:39.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.048+0000 7f7e61831740 20 client.26303 populate_metadata read hostname 'smithi039' 2024-03-18T16:13:39.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.048+0000 7f7e61831740 1 --2- 172.21.15.39:0/2254968780 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x55f4fdde7e40 0x55f4fddea260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:39.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.048+0000 7f7e61831740 4 client.26303 mds_command: new command op to 14463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:13:39.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.048+0000 7f7e61831740 1 -- 172.21.15.39:0/2254968780 --> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f4fdcdd990 con 0x55f4fdde7e40 2024-03-18T16:13:39.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.049+0000 7f7e57e00640 1 --2- 172.21.15.39:0/2254968780 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x55f4fdde7e40 0x55f4fddea260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:39.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.051+0000 7f7e57e00640 1 --2- 172.21.15.39:0/2254968780 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x55f4fdde7e40 0x55f4fddea260 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=d054e9682bff163f server_cookie=bf83a6d7ba719413 in_seq=0 out_seq=0 2024-03-18T16:13:39.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.051+0000 7f7e41600640 10 client.26303 ms_handle_connect on v2:172.21.15.39:6840/3515326042 2024-03-18T16:13:39.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.052+0000 7f7e41600640 1 -- 172.21.15.39:0/2254968780 <== mds.4 v2:172.21.15.39:6840/3515326042 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f4fdcdd990 con 0x55f4fdde7e40 2024-03-18T16:13:39.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.053+0000 7f7e41600640 10 client.26303 handle_command_reply: tid=0 2024-03-18T16:13:39.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.053+0000 7f7e61831740 1 client.26303 shutdown 2024-03-18T16:13:39.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.054+0000 7f7e61831740 1 -- 172.21.15.39:0/2254968780 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f4fdd6c2b0 msgr2=0x55f4fdd7a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:39.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.054+0000 7f7e61831740 1 --2- 172.21.15.39:0/2254968780 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f4fdd6c2b0 0x55f4fdd7a660 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f7e5009afb0 tx=0x7f7e5000e040 comp rx=0 tx=0).stop 2024-03-18T16:13:39.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.054+0000 7f7e61831740 1 -- 172.21.15.39:0/2254968780 shutdown_connections 2024-03-18T16:13:39.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.054+0000 7f7e61831740 1 --2- 172.21.15.39:0/2254968780 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x55f4fdde7e40 0x55f4fddea260 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:39.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.054+0000 7f7e61831740 1 --2- 172.21.15.39:0/2254968780 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f4fdd6c2b0 0x55f4fdd7a660 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:39.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.054+0000 7f7e61831740 1 --2- 172.21.15.39:0/2254968780 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f4fdd71420 0x55f4fdde5c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:39.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.054+0000 7f7e61831740 1 --2- 172.21.15.39:0/2254968780 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f4fdd6b8c0 0x55f4fdd7a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:39.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.054+0000 7f7e61831740 1 -- 172.21.15.39:0/2254968780 >> 172.21.15.39:0/2254968780 conn(0x55f4fdd66e30 msgr2=0x55f4fdd661f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:39.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.054+0000 7f7e41600640 0 client.26303 ms_handle_reset on v2:172.21.15.39:6840/3515326042 2024-03-18T16:13:39.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.055+0000 7f7e61831740 1 -- 172.21.15.39:0/2254968780 shutdown_connections 2024-03-18T16:13:39.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.055+0000 7f7e61831740 1 -- 172.21.15.39:0/2254968780 wait complete. 2024-03-18T16:13:39.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.055+0000 7f7e61831740 20 client.26303 trim_cache size 0 max 16384 2024-03-18T16:13:39.059 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:13:39.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.061+0000 7f7e5ec00640 1 -- 172.21.15.39:0/2347805924 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f7e3c0a2880 msgr2=0x7f7e3c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:39.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.061+0000 7f7e5ec00640 1 --2- 172.21.15.39:0/2347805924 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f7e3c0a2880 0x7f7e3c0a4d40 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f7e48005ec0 tx=0x7f7e4802c900 comp rx=0 tx=0).stop 2024-03-18T16:13:39.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.061+0000 7f7e5ec00640 1 -- 172.21.15.39:0/2347805924 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7e5810b150 msgr2=0x7f7e581173b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:39.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.061+0000 7f7e5ec00640 1 --2- 172.21.15.39:0/2347805924 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7e5810b150 0x7f7e581173b0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f7e5000ba50 tx=0x7f7e5000bf10 comp rx=0 tx=0).stop 2024-03-18T16:13:39.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.062+0000 7f7e5ec00640 1 -- 172.21.15.39:0/2347805924 shutdown_connections 2024-03-18T16:13:39.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.062+0000 7f7e5ec00640 1 --2- 172.21.15.39:0/2347805924 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7e5810b150 0x7f7e581173b0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:39.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.062+0000 7f7e5ec00640 1 --2- 172.21.15.39:0/2347805924 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f7e3c0a2880 0x7f7e3c0a4d40 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:39.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.062+0000 7f7e5ec00640 1 --2- 172.21.15.39:0/2347805924 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7e5810db00 0x7f7e581b6a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:39.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.062+0000 7f7e5ec00640 1 --2- 172.21.15.39:0/2347805924 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7e581087e0 0x7f7e58116e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:39.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.062+0000 7f7e5ec00640 1 -- 172.21.15.39:0/2347805924 >> 172.21.15.39:0/2347805924 conn(0x7f7e5806e860 msgr2=0x7f7e58108eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:39.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.062+0000 7f7e5ec00640 1 -- 172.21.15.39:0/2347805924 shutdown_connections 2024-03-18T16:13:39.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.062+0000 7f7e5ec00640 1 -- 172.21.15.39:0/2347805924 wait complete. 2024-03-18T16:13:39.097 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-18T16:13:39.224 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:13:38 smithi039 ceph-mon[31721]: pgmap v529: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 69 MiB/s wr, 8.85k op/s 2024-03-18T16:13:39.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:13:38 smithi196 ceph-mon[36307]: pgmap v529: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 69 MiB/s wr, 8.85k op/s 2024-03-18T16:13:39.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:13:38 smithi194 ceph-mon[36200]: pgmap v529: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 69 MiB/s wr, 8.85k op/s 2024-03-18T16:13:39.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.447+0000 7ff0f6a00640 1 -- 172.21.15.39:0/99080437 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff0f0072d90 msgr2=0x7ff0f00731b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:39.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.447+0000 7ff0f6a00640 1 --2- 172.21.15.39:0/99080437 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff0f0072d90 0x7ff0f00731b0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7ff0e4009ca0 tx=0x7ff0e402c0f0 comp rx=0 tx=0).stop 2024-03-18T16:13:39.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.447+0000 7ff0f6a00640 1 -- 172.21.15.39:0/99080437 shutdown_connections 2024-03-18T16:13:39.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.447+0000 7ff0f6a00640 1 --2- 172.21.15.39:0/99080437 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff0f0109750 0x7ff0f010e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:39.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.447+0000 7ff0f6a00640 1 --2- 172.21.15.39:0/99080437 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff0f0108db0 0x7ff0f0109210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:39.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.447+0000 7ff0f6a00640 1 --2- 172.21.15.39:0/99080437 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff0f0072d90 0x7ff0f00731b0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:39.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.447+0000 7ff0f6a00640 1 -- 172.21.15.39:0/99080437 >> 172.21.15.39:0/99080437 conn(0x7ff0f006e600 msgr2=0x7ff0f0070a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:39.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.449+0000 7ff0f6a00640 1 -- 172.21.15.39:0/99080437 shutdown_connections 2024-03-18T16:13:39.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.449+0000 7ff0f6a00640 1 -- 172.21.15.39:0/99080437 wait complete. 2024-03-18T16:13:39.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.450+0000 7ff0f6a00640 1 Processor -- start 2024-03-18T16:13:39.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.452+0000 7ff0f6a00640 1 -- start start 2024-03-18T16:13:39.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.452+0000 7ff0f6a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff0f0072d90 0x7ff0f01a6700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:39.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.453+0000 7ff0f6a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff0f0108db0 0x7ff0f01a6c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:39.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.453+0000 7ff0f6a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff0f0109750 0x7ff0f01adcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:39.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.453+0000 7ff0f4c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff0f0108db0 0x7ff0f01a6c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:39.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.453+0000 7ff0f4c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff0f0108db0 0x7ff0f01a6c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:60048/0 (socket says 172.21.15.39:60048) 2024-03-18T16:13:39.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.453+0000 7ff0f6a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7ff0f011c0a0 con 0x7ff0f0109750 2024-03-18T16:13:39.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.453+0000 7ff0f4c00640 1 -- 172.21.15.39:0/4271297225 learned_addr learned my addr 172.21.15.39:0/4271297225 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:13:39.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.453+0000 7ff0f6a00640 1 -- 172.21.15.39:0/4271297225 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ff0f011bf20 con 0x7ff0f0108db0 2024-03-18T16:13:39.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.454+0000 7ff0f6a00640 1 -- 172.21.15.39:0/4271297225 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7ff0f011c220 con 0x7ff0f0072d90 2024-03-18T16:13:39.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.454+0000 7ff0f5600640 1 --2- 172.21.15.39:0/4271297225 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff0f0072d90 0x7ff0f01a6700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:39.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.454+0000 7ff0f6000640 1 --2- 172.21.15.39:0/4271297225 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff0f0109750 0x7ff0f01adcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:39.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.454+0000 7ff0f4c00640 1 -- 172.21.15.39:0/4271297225 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff0f0072d90 msgr2=0x7ff0f01a6700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:39.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.454+0000 7ff0f4c00640 1 --2- 172.21.15.39:0/4271297225 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff0f0072d90 0x7ff0f01a6700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:39.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.454+0000 7ff0f4c00640 1 -- 172.21.15.39:0/4271297225 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff0f0109750 msgr2=0x7ff0f01adcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:39.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.454+0000 7ff0f4c00640 1 --2- 172.21.15.39:0/4271297225 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff0f0109750 0x7ff0f01adcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:39.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.454+0000 7ff0f4c00640 1 -- 172.21.15.39:0/4271297225 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0f01ae320 con 0x7ff0f0108db0 2024-03-18T16:13:39.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.455+0000 7ff0f4c00640 1 --2- 172.21.15.39:0/4271297225 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff0f0108db0 0x7ff0f01a6c40 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7ff0dc00d950 tx=0x7ff0dc00de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:39.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.456+0000 7ff0ee000640 1 -- 172.21.15.39:0/4271297225 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0dc017070 con 0x7ff0f0108db0 2024-03-18T16:13:39.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.456+0000 7ff0ee000640 1 -- 172.21.15.39:0/4271297225 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff0dc009c30 con 0x7ff0f0108db0 2024-03-18T16:13:39.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.456+0000 7ff0f6a00640 1 -- 172.21.15.39:0/4271297225 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0f01ae610 con 0x7ff0f0108db0 2024-03-18T16:13:39.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.456+0000 7ff0f6a00640 1 -- 172.21.15.39:0/4271297225 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0f01aeb60 con 0x7ff0f0108db0 2024-03-18T16:13:39.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.457+0000 7ff0ee000640 1 -- 172.21.15.39:0/4271297225 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0dc00f6b0 con 0x7ff0f0108db0 2024-03-18T16:13:39.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.465+0000 7ff0ee000640 1 -- 172.21.15.39:0/4271297225 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7ff0dc0109d0 con 0x7ff0f0108db0 2024-03-18T16:13:39.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.465+0000 7ff0fac2d740 1 Processor -- start 2024-03-18T16:13:39.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.466+0000 7ff0fac2d740 1 -- start start 2024-03-18T16:13:39.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.467+0000 7ff0fac2d740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a7fec1fa90 0x55a7fec1feb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:39.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.467+0000 7ff0fac2d740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a7fec20480 0x55a7fec24f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:39.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.467+0000 7ff0fac2d740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a7fec255f0 0x55a7fec27a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:39.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.467+0000 7ff0fac2d740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55a7febe73b0 con 0x55a7fec1fa90 2024-03-18T16:13:39.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.467+0000 7ff0fac2d740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55a7feb84450 con 0x55a7fec20480 2024-03-18T16:13:39.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.467+0000 7ff0fac2d740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55a7feb2ccb0 con 0x55a7fec255f0 2024-03-18T16:13:39.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.468+0000 7ff0f5600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a7fec20480 0x55a7fec24f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:39.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.468+0000 7ff0f4c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a7fec255f0 0x55a7fec27a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:39.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.468+0000 7ff0f5600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a7fec20480 0x55a7fec24f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:60056/0 (socket says 172.21.15.39:60056) 2024-03-18T16:13:39.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.468+0000 7ff0f4c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a7fec255f0 0x55a7fec27a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:42336/0 (socket says 172.21.15.39:42336) 2024-03-18T16:13:39.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.468+0000 7ff0f5600640 1 -- 172.21.15.39:0/1779240929 learned_addr learned my addr 172.21.15.39:0/1779240929 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:13:39.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.468+0000 7ff0f6000640 1 --2- 172.21.15.39:0/1779240929 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a7fec1fa90 0x55a7fec1feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:39.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.468+0000 7ff0f6000640 1 -- 172.21.15.39:0/1779240929 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a7fec255f0 msgr2=0x55a7fec27a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:39.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.468+0000 7ff0f6000640 1 --2- 172.21.15.39:0/1779240929 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a7fec255f0 0x55a7fec27a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:39.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.468+0000 7ff0f6000640 1 -- 172.21.15.39:0/1779240929 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a7fec20480 msgr2=0x55a7fec24f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:39.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.468+0000 7ff0f6000640 1 --2- 172.21.15.39:0/1779240929 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a7fec20480 0x55a7fec24f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:39.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.469+0000 7ff0f6000640 1 -- 172.21.15.39:0/1779240929 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a7feb4e1f0 con 0x55a7fec1fa90 2024-03-18T16:13:39.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.469+0000 7ff0f6000640 1 --2- 172.21.15.39:0/1779240929 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a7fec1fa90 0x55a7fec1feb0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ff0e8009af0 tx=0x7ff0e80113e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=447dbd61a442ef89 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:39.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.471+0000 7ff0ee000640 1 --2- 172.21.15.39:0/4271297225 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff0d80a2900 0x7ff0d80a4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:39.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.472+0000 7ff0e2600640 1 -- 172.21.15.39:0/1779240929 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0e8011e10 con 0x55a7fec1fa90 2024-03-18T16:13:39.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.472+0000 7ff0e2600640 1 -- 172.21.15.39:0/1779240929 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff0e8012930 con 0x55a7fec1fa90 2024-03-18T16:13:39.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.472+0000 7ff0e2600640 1 -- 172.21.15.39:0/1779240929 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0e800fb30 con 0x55a7fec1fa90 2024-03-18T16:13:39.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.472+0000 7ff0f5600640 1 --2- 172.21.15.39:0/4271297225 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff0d80a2900 0x7ff0d80a4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:39.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.473+0000 7ff0fac2d740 1 -- 172.21.15.39:0/1779240929 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a7fec1fa90 msgr2=0x55a7fec1feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:39.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.473+0000 7ff0fac2d740 1 --2- 172.21.15.39:0/1779240929 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a7fec1fa90 0x55a7fec1feb0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ff0e8009af0 tx=0x7ff0e80113e0 comp rx=0 tx=0).stop 2024-03-18T16:13:39.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.473+0000 7ff0ee000640 1 -- 172.21.15.39:0/4271297225 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7ff0dc0cede0 con 0x7ff0f0108db0 2024-03-18T16:13:39.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.474+0000 7ff0f5600640 1 --2- 172.21.15.39:0/4271297225 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff0d80a2900 0x7ff0d80a4dc0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7ff0e4000f80 tx=0x7ff0e402e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:39.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.475+0000 7ff0fac2d740 1 -- 172.21.15.39:0/1779240929 shutdown_connections 2024-03-18T16:13:39.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.475+0000 7ff0fac2d740 1 --2- 172.21.15.39:0/1779240929 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a7fec20480 0x55a7fec24f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:39.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.475+0000 7ff0fac2d740 1 --2- 172.21.15.39:0/1779240929 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a7fec255f0 0x55a7fec27a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:39.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.475+0000 7ff0fac2d740 1 --2- 172.21.15.39:0/1779240929 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a7fec1fa90 0x55a7fec1feb0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:39.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.475+0000 7ff0fac2d740 1 -- 172.21.15.39:0/1779240929 >> 172.21.15.39:0/1779240929 conn(0x55a7fec1b000 msgr2=0x55a7fec1b400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:39.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.475+0000 7ff0fac2d740 1 -- 172.21.15.39:0/1779240929 shutdown_connections 2024-03-18T16:13:39.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.476+0000 7ff0fac2d740 1 -- 172.21.15.39:0/1779240929 wait complete. 2024-03-18T16:13:39.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.477+0000 7ff0fac2d740 1 -- start start 2024-03-18T16:13:39.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.478+0000 7ff0fac2d740 1 Processor -- start 2024-03-18T16:13:39.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.479+0000 7ff0fac2d740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a7fec1fa90 0x55a7fec29e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:39.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.479+0000 7ff0fac2d740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a7fec20480 0x55a7fec2a360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:39.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.479+0000 7ff0fac2d740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a7fec255f0 0x55a7fec313a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:39.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.479+0000 7ff0fac2d740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55a7feb40110 con 0x55a7fec20480 2024-03-18T16:13:39.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.479+0000 7ff0fac2d740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55a7feb804f0 con 0x55a7fec1fa90 2024-03-18T16:13:39.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.479+0000 7ff0fac2d740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55a7fec285a0 con 0x55a7fec255f0 2024-03-18T16:13:39.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.479+0000 7ff0fac2d740 10 client.0 _ll_register_callbacks cb 0x55a7feb2db70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:13:39.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.480+0000 7ff0f4c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a7fec20480 0x55a7fec2a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:39.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.480+0000 7ff0f6000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a7fec255f0 0x55a7fec313a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:39.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.480+0000 7ff0f6000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a7fec255f0 0x55a7fec313a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:42344/0 (socket says 172.21.15.39:42344) 2024-03-18T16:13:39.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.480+0000 7ff0f6000640 1 -- 172.21.15.39:0/1806378920 learned_addr learned my addr 172.21.15.39:0/1806378920 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:13:39.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.480+0000 7ff0f5600640 1 --2- 172.21.15.39:0/1806378920 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a7fec1fa90 0x55a7fec29e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:39.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.480+0000 7ff0f6000640 1 -- 172.21.15.39:0/1806378920 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a7fec1fa90 msgr2=0x55a7fec29e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:39.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.480+0000 7ff0f6000640 1 --2- 172.21.15.39:0/1806378920 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a7fec1fa90 0x55a7fec29e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:39.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.480+0000 7ff0f6000640 1 -- 172.21.15.39:0/1806378920 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a7fec20480 msgr2=0x55a7fec2a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:39.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.480+0000 7ff0f6000640 1 --2- 172.21.15.39:0/1806378920 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a7fec20480 0x55a7fec2a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:39.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.480+0000 7ff0f6000640 1 -- 172.21.15.39:0/1806378920 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a7fe8983f0 con 0x55a7fec255f0 2024-03-18T16:13:39.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.481+0000 7ff0f6000640 1 --2- 172.21.15.39:0/1806378920 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a7fec255f0 0x55a7fec313a0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7ff0e8012660 tx=0x7ff0e8012690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=65eaf1c2afa21a82 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:39.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.482+0000 7ff0d6a00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:13:39.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.483+0000 7ff0d6a00640 1 -- 172.21.15.39:0/1806378920 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0e8028070 con 0x55a7fec255f0 2024-03-18T16:13:39.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.483+0000 7ff0d6a00640 1 -- 172.21.15.39:0/1806378920 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7ff0e800f870 con 0x55a7fec255f0 2024-03-18T16:13:39.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.484+0000 7ff0fac2d740 1 -- 172.21.15.39:0/1806378920 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff0e8012930 con 0x55a7fec255f0 2024-03-18T16:13:39.487 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.484+0000 7ff0d6a00640 1 -- 172.21.15.39:0/1806378920 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7ff0e8032050 con 0x55a7fec255f0 2024-03-18T16:13:39.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.485+0000 7ff0fac2d740 10 client.26314 fetch_fsmap learned FSMap version 36 2024-03-18T16:13:39.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.485+0000 7ff0fac2d740 1 -- 172.21.15.39:0/1806378920 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55a7fe9627d0 con 0x55a7fec255f0 2024-03-18T16:13:39.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.487+0000 7ff0d6a00640 1 -- 172.21.15.39:0/1806378920 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff0e8012930 con 0x55a7fec255f0 2024-03-18T16:13:39.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.487+0000 7ff0d6a00640 1 -- 172.21.15.39:0/1806378920 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 36) ==== 8406+0+0 (secure 0 0 0) 0x7ff0e8004680 con 0x55a7fec255f0 2024-03-18T16:13:39.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.488+0000 7ff0fac2d740 10 client.26314 fetch_fsmap finished waiting for FSMap version 36 2024-03-18T16:13:39.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.488+0000 7ff0fac2d740 10 client.26314 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-18T16:13:39.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.488+0000 7ff0fac2d740 20 client.26314 populate_metadata read hostname 'smithi039' 2024-03-18T16:13:39.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.488+0000 7ff0fac2d740 1 --2- 172.21.15.39:0/1806378920 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x55a7fec359a0 0x55a7fec37d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:39.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.488+0000 7ff0fac2d740 4 client.26314 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:13:39.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.488+0000 7ff0fac2d740 1 -- 172.21.15.39:0/1806378920 --> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a7feb91990 con 0x55a7fec359a0 2024-03-18T16:13:39.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.488+0000 7ff0f5600640 1 --2- 172.21.15.39:0/1806378920 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x55a7fec359a0 0x55a7fec37d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:39.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.490+0000 7ff0f5600640 1 --2- 172.21.15.39:0/1806378920 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x55a7fec359a0 0x55a7fec37d90 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=4e922b73827f496 server_cookie=6834a2010e1f0a6c in_seq=0 out_seq=0 2024-03-18T16:13:39.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.490+0000 7ff0d6a00640 10 client.26314 ms_handle_connect on v2:172.21.15.194:6836/1301464246 2024-03-18T16:13:39.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.492+0000 7ff0d6a00640 1 -- 172.21.15.39:0/1806378920 <== mds.3 v2:172.21.15.194:6836/1301464246 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a7feb91990 con 0x55a7fec359a0 2024-03-18T16:13:39.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.492+0000 7ff0d6a00640 10 client.26314 handle_command_reply: tid=0 2024-03-18T16:13:39.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.492+0000 7ff0fac2d740 1 client.26314 shutdown 2024-03-18T16:13:39.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.494+0000 7ff0fac2d740 1 -- 172.21.15.39:0/1806378920 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a7fec255f0 msgr2=0x55a7fec313a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:39.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.494+0000 7ff0fac2d740 1 --2- 172.21.15.39:0/1806378920 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a7fec255f0 0x55a7fec313a0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7ff0e8012660 tx=0x7ff0e8012690 comp rx=0 tx=0).stop 2024-03-18T16:13:39.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.494+0000 7ff0fac2d740 1 -- 172.21.15.39:0/1806378920 shutdown_connections 2024-03-18T16:13:39.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.494+0000 7ff0fac2d740 1 --2- 172.21.15.39:0/1806378920 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x55a7fec359a0 0x55a7fec37d90 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:39.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.495+0000 7ff0fac2d740 1 --2- 172.21.15.39:0/1806378920 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a7fec255f0 0x55a7fec313a0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:39.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.495+0000 7ff0fac2d740 1 --2- 172.21.15.39:0/1806378920 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a7fec20480 0x55a7fec2a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:39.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.495+0000 7ff0fac2d740 1 --2- 172.21.15.39:0/1806378920 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a7fec1fa90 0x55a7fec29e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:39.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.495+0000 7ff0fac2d740 1 -- 172.21.15.39:0/1806378920 >> 172.21.15.39:0/1806378920 conn(0x55a7fec1b000 msgr2=0x55a7fec1a220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:39.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.495+0000 7ff0d6a00640 0 client.26314 ms_handle_reset on v2:172.21.15.194:6836/1301464246 2024-03-18T16:13:39.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.495+0000 7ff0fac2d740 1 -- 172.21.15.39:0/1806378920 shutdown_connections 2024-03-18T16:13:39.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.496+0000 7ff0fac2d740 1 -- 172.21.15.39:0/1806378920 wait complete. 2024-03-18T16:13:39.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.497+0000 7ff0fac2d740 20 client.26314 trim_cache size 0 max 16384 2024-03-18T16:13:39.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.537+0000 7ff0fac2d740 1 Processor -- start 2024-03-18T16:13:39.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.537+0000 7ff0fac2d740 1 -- start start 2024-03-18T16:13:39.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.537+0000 7ff0fac2d740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a7fec1fa90 0x55a7fec379b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:39.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.537+0000 7ff0fac2d740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a7fec20480 0x55a7fec31360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:39.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.537+0000 7ff0fac2d740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a7fec255f0 0x55a7fec37ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:39.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.537+0000 7ff0fac2d740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7ff0f01b9230 con 0x55a7fec1fa90 2024-03-18T16:13:39.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.537+0000 7ff0fac2d740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ff0f01baef0 con 0x55a7fec255f0 2024-03-18T16:13:39.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.537+0000 7ff0fac2d740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7ff0f01b9900 con 0x55a7fec20480 2024-03-18T16:13:39.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.538+0000 7ff0f4c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a7fec20480 0x55a7fec31360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:39.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.538+0000 7ff0f4c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a7fec20480 0x55a7fec31360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:42356/0 (socket says 172.21.15.39:42356) 2024-03-18T16:13:39.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.538+0000 7ff0f4c00640 1 -- 172.21.15.39:0/64525364 learned_addr learned my addr 172.21.15.39:0/64525364 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:13:39.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.538+0000 7ff0f6000640 1 --2- 172.21.15.39:0/64525364 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a7fec255f0 0x55a7fec37ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:39.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.540+0000 7ff0f5600640 1 --2- 172.21.15.39:0/64525364 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a7fec1fa90 0x55a7fec379b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:39.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.540+0000 7ff0f4c00640 1 -- 172.21.15.39:0/64525364 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a7fec255f0 msgr2=0x55a7fec37ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:39.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.540+0000 7ff0f4c00640 1 --2- 172.21.15.39:0/64525364 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a7fec255f0 0x55a7fec37ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:39.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.540+0000 7ff0f4c00640 1 -- 172.21.15.39:0/64525364 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a7fec1fa90 msgr2=0x55a7fec379b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:39.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.540+0000 7ff0f4c00640 1 --2- 172.21.15.39:0/64525364 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a7fec1fa90 0x55a7fec379b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:39.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.540+0000 7ff0f4c00640 1 -- 172.21.15.39:0/64525364 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a7fec1ba50 con 0x55a7fec20480 2024-03-18T16:13:39.544 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.541+0000 7ff0f4c00640 1 --2- 172.21.15.39:0/64525364 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a7fec20480 0x55a7fec31360 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7ff0dc0cc4f0 tx=0x7ff0dc0d2820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=74eca776e61f64 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:39.544 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.542+0000 7ff0e1c00640 1 -- 172.21.15.39:0/64525364 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0dc020070 con 0x55a7fec20480 2024-03-18T16:13:39.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.542+0000 7ff0e1c00640 1 -- 172.21.15.39:0/64525364 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff0dc009de0 con 0x55a7fec20480 2024-03-18T16:13:39.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.542+0000 7ff0e1c00640 1 -- 172.21.15.39:0/64525364 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0dc020070 con 0x55a7fec20480 2024-03-18T16:13:39.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.543+0000 7ff0fac2d740 1 -- 172.21.15.39:0/64525364 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a7fec20480 msgr2=0x55a7fec31360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:39.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.543+0000 7ff0fac2d740 1 --2- 172.21.15.39:0/64525364 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a7fec20480 0x55a7fec31360 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7ff0dc0cc4f0 tx=0x7ff0dc0d2820 comp rx=0 tx=0).stop 2024-03-18T16:13:39.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.544+0000 7ff0fac2d740 1 -- 172.21.15.39:0/64525364 shutdown_connections 2024-03-18T16:13:39.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.544+0000 7ff0fac2d740 1 --2- 172.21.15.39:0/64525364 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a7fec255f0 0x55a7fec37ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:39.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.544+0000 7ff0fac2d740 1 --2- 172.21.15.39:0/64525364 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a7fec20480 0x55a7fec31360 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:39.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.544+0000 7ff0fac2d740 1 --2- 172.21.15.39:0/64525364 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a7fec1fa90 0x55a7fec379b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:39.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.544+0000 7ff0fac2d740 1 -- 172.21.15.39:0/64525364 >> 172.21.15.39:0/64525364 conn(0x55a7fec1b000 msgr2=0x55a7fec21920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:39.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.544+0000 7ff0fac2d740 1 -- 172.21.15.39:0/64525364 shutdown_connections 2024-03-18T16:13:39.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.544+0000 7ff0fac2d740 1 -- 172.21.15.39:0/64525364 wait complete. 2024-03-18T16:13:39.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.545+0000 7ff0fac2d740 1 -- start start 2024-03-18T16:13:39.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.545+0000 7ff0fac2d740 1 Processor -- start 2024-03-18T16:13:39.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.547+0000 7ff0fac2d740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a7fec1fa90 0x55a7fec2e230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:39.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.547+0000 7ff0fac2d740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a7fec20480 0x55a7fec2e770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:39.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.547+0000 7ff0fac2d740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a7fec255f0 0x55a7fec99c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:39.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.547+0000 7ff0fac2d740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7ff0f01ba760 con 0x55a7fec255f0 2024-03-18T16:13:39.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.547+0000 7ff0fac2d740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ff0f01b9fd0 con 0x55a7fec1fa90 2024-03-18T16:13:39.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.547+0000 7ff0fac2d740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7ff0f01b7210 con 0x55a7fec20480 2024-03-18T16:13:39.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.547+0000 7ff0fac2d740 10 client.0 _ll_register_callbacks cb 0x7ff0dc0d26f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:13:39.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.548+0000 7ff0f5600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a7fec1fa90 0x55a7fec2e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:39.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.548+0000 7ff0f6000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a7fec255f0 0x55a7fec99c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:39.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.548+0000 7ff0f5600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a7fec1fa90 0x55a7fec2e230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:60074/0 (socket says 172.21.15.39:60074) 2024-03-18T16:13:39.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.548+0000 7ff0f5600640 1 -- 172.21.15.39:0/3601212381 learned_addr learned my addr 172.21.15.39:0/3601212381 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:13:39.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.548+0000 7ff0f4c00640 1 --2- 172.21.15.39:0/3601212381 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a7fec20480 0x55a7fec2e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:39.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.549+0000 7ff0f4c00640 1 -- 172.21.15.39:0/3601212381 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a7fec1fa90 msgr2=0x55a7fec2e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:39.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.549+0000 7ff0f4c00640 1 --2- 172.21.15.39:0/3601212381 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a7fec1fa90 0x55a7fec2e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:39.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.549+0000 7ff0f4c00640 1 -- 172.21.15.39:0/3601212381 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a7fec255f0 msgr2=0x55a7fec99c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:39.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.549+0000 7ff0f4c00640 1 --2- 172.21.15.39:0/3601212381 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a7fec255f0 0x55a7fec99c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:39.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.549+0000 7ff0f4c00640 1 -- 172.21.15.39:0/3601212381 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a7fec9a140 con 0x55a7fec20480 2024-03-18T16:13:39.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.549+0000 7ff0f5600640 1 --2- 172.21.15.39:0/3601212381 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a7fec1fa90 0x55a7fec2e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:13:39.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.549+0000 7ff0f6000640 1 --2- 172.21.15.39:0/3601212381 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a7fec255f0 0x55a7fec99c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:13:39.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.553+0000 7ff0f4c00640 1 --2- 172.21.15.39:0/3601212381 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a7fec20480 0x55a7fec2e770 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x55a7fec27f60 tx=0x7ff0dc0d2510 comp rx=0 tx=0).ready entity=mon.1 client_cookie=77991e9decef350 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:39.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.553+0000 7ff0d6a00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:13:39.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.553+0000 7ff0d6a00640 1 -- 172.21.15.39:0/3601212381 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0dc020050 con 0x55a7fec20480 2024-03-18T16:13:39.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.553+0000 7ff0d6a00640 1 -- 172.21.15.39:0/3601212381 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7ff0dc09f170 con 0x55a7fec20480 2024-03-18T16:13:39.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.554+0000 7ff0fac2d740 1 -- 172.21.15.39:0/3601212381 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff0dc009de0 con 0x55a7fec20480 2024-03-18T16:13:39.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.554+0000 7ff0d6a00640 1 -- 172.21.15.39:0/3601212381 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7ff0dc09ba90 con 0x55a7fec20480 2024-03-18T16:13:39.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.554+0000 7ff0fac2d740 10 client.26320 fetch_fsmap learned FSMap version 36 2024-03-18T16:13:39.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.554+0000 7ff0fac2d740 1 -- 172.21.15.39:0/3601212381 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55a7fec9b3e0 con 0x55a7fec20480 2024-03-18T16:13:39.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.555+0000 7ff0d6a00640 1 -- 172.21.15.39:0/3601212381 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff0dc009de0 con 0x55a7fec20480 2024-03-18T16:13:39.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.556+0000 7ff0d6a00640 1 -- 172.21.15.39:0/3601212381 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 36) ==== 8406+0+0 (secure 0 0 0) 0x7ff0dc0a36a0 con 0x55a7fec20480 2024-03-18T16:13:39.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.556+0000 7ff0fac2d740 10 client.26320 fetch_fsmap finished waiting for FSMap version 36 2024-03-18T16:13:39.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.556+0000 7ff0fac2d740 10 client.26320 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-18T16:13:39.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.556+0000 7ff0fac2d740 20 client.26320 populate_metadata read hostname 'smithi039' 2024-03-18T16:13:39.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.556+0000 7ff0fac2d740 1 --2- 172.21.15.39:0/3601212381 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x55a7fec9bcf0 0x55a7fec9e110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:39.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.556+0000 7ff0fac2d740 4 client.26320 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:13:39.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.556+0000 7ff0fac2d740 1 -- 172.21.15.39:0/3601212381 --> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a7feb8bce0 con 0x55a7fec9bcf0 2024-03-18T16:13:39.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.557+0000 7ff0f5600640 1 --2- 172.21.15.39:0/3601212381 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x55a7fec9bcf0 0x55a7fec9e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:39.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.557+0000 7ff0f5600640 1 --2- 172.21.15.39:0/3601212381 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x55a7fec9bcf0 0x55a7fec9e110 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=113a5465d40dafd server_cookie=72bab5e1350c73f7 in_seq=0 out_seq=0 2024-03-18T16:13:39.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.558+0000 7ff0d6a00640 10 client.26320 ms_handle_connect on v2:172.21.15.194:6836/1301464246 2024-03-18T16:13:39.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.558+0000 7ff0d6a00640 1 -- 172.21.15.39:0/3601212381 <== mds.3 v2:172.21.15.194:6836/1301464246 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a7feb8bce0 con 0x55a7fec9bcf0 2024-03-18T16:13:39.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.558+0000 7ff0d6a00640 10 client.26320 handle_command_reply: tid=0 2024-03-18T16:13:39.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.558+0000 7ff0fac2d740 1 client.26320 shutdown 2024-03-18T16:13:39.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.558+0000 7ff0fac2d740 1 -- 172.21.15.39:0/3601212381 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a7fec20480 msgr2=0x55a7fec2e770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:39.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.558+0000 7ff0fac2d740 1 --2- 172.21.15.39:0/3601212381 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a7fec20480 0x55a7fec2e770 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x55a7fec27f60 tx=0x7ff0dc0d2510 comp rx=0 tx=0).stop 2024-03-18T16:13:39.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.558+0000 7ff0fac2d740 1 -- 172.21.15.39:0/3601212381 shutdown_connections 2024-03-18T16:13:39.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.558+0000 7ff0fac2d740 1 --2- 172.21.15.39:0/3601212381 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x55a7fec9bcf0 0x55a7fec9e110 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:39.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.558+0000 7ff0fac2d740 1 --2- 172.21.15.39:0/3601212381 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a7fec255f0 0x55a7fec99c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:39.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.558+0000 7ff0fac2d740 1 --2- 172.21.15.39:0/3601212381 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a7fec20480 0x55a7fec2e770 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:39.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.558+0000 7ff0fac2d740 1 --2- 172.21.15.39:0/3601212381 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a7fec1fa90 0x55a7fec2e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:39.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.558+0000 7ff0fac2d740 1 -- 172.21.15.39:0/3601212381 >> 172.21.15.39:0/3601212381 conn(0x55a7fec1b000 msgr2=0x55a7fec379b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:39.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.559+0000 7ff0d6a00640 0 client.26320 ms_handle_reset on v2:172.21.15.194:6836/1301464246 2024-03-18T16:13:39.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.559+0000 7ff0fac2d740 1 -- 172.21.15.39:0/3601212381 shutdown_connections 2024-03-18T16:13:39.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.559+0000 7ff0fac2d740 1 -- 172.21.15.39:0/3601212381 wait complete. 2024-03-18T16:13:39.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.560+0000 7ff0fac2d740 20 client.26320 trim_cache size 0 max 16384 2024-03-18T16:13:39.563 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:13:39.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.564+0000 7ff0f6a00640 1 -- 172.21.15.39:0/4271297225 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff0d80a2900 msgr2=0x7ff0d80a4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:39.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.564+0000 7ff0f6a00640 1 --2- 172.21.15.39:0/4271297225 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff0d80a2900 0x7ff0d80a4dc0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7ff0e4000f80 tx=0x7ff0e402e040 comp rx=0 tx=0).stop 2024-03-18T16:13:39.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.564+0000 7ff0f6a00640 1 -- 172.21.15.39:0/4271297225 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff0f0108db0 msgr2=0x7ff0f01a6c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:39.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.564+0000 7ff0f6a00640 1 --2- 172.21.15.39:0/4271297225 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff0f0108db0 0x7ff0f01a6c40 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7ff0dc00d950 tx=0x7ff0dc00de10 comp rx=0 tx=0).stop 2024-03-18T16:13:39.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.564+0000 7ff0f6a00640 1 -- 172.21.15.39:0/4271297225 shutdown_connections 2024-03-18T16:13:39.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.564+0000 7ff0f6a00640 1 --2- 172.21.15.39:0/4271297225 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff0f0108db0 0x7ff0f01a6c40 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:39.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.564+0000 7ff0f6a00640 1 --2- 172.21.15.39:0/4271297225 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff0d80a2900 0x7ff0d80a4dc0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:39.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.564+0000 7ff0f6a00640 1 --2- 172.21.15.39:0/4271297225 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff0f0109750 0x7ff0f01adcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:39.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.564+0000 7ff0f6a00640 1 --2- 172.21.15.39:0/4271297225 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff0f0072d90 0x7ff0f01a6700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:39.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.564+0000 7ff0f6a00640 1 -- 172.21.15.39:0/4271297225 >> 172.21.15.39:0/4271297225 conn(0x7ff0f006e600 msgr2=0x7ff0f0070240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:39.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.565+0000 7ff0f6a00640 1 -- 172.21.15.39:0/4271297225 shutdown_connections 2024-03-18T16:13:39.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.565+0000 7ff0f6a00640 1 -- 172.21.15.39:0/4271297225 wait complete. 2024-03-18T16:13:39.601 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-18T16:13:39.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.947+0000 7f51df000640 1 -- 172.21.15.39:0/3048972984 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f51d810b5d0 msgr2=0x7f51d8114470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:39.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.947+0000 7f51df000640 1 --2- 172.21.15.39:0/3048972984 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f51d810b5d0 0x7f51d8114470 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f51c8009900 tx=0x7f51c802bc30 comp rx=0 tx=0).stop 2024-03-18T16:13:39.952 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.949+0000 7f51df000640 1 -- 172.21.15.39:0/3048972984 shutdown_connections 2024-03-18T16:13:39.952 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.949+0000 7f51df000640 1 --2- 172.21.15.39:0/3048972984 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f51d810b150 0x7f51d8111f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:39.952 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.949+0000 7f51df000640 1 --2- 172.21.15.39:0/3048972984 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f51d810b5d0 0x7f51d8114470 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:39.952 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.949+0000 7f51df000640 1 --2- 172.21.15.39:0/3048972984 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f51d81087e0 0x7f51d810ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:39.952 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.949+0000 7f51df000640 1 -- 172.21.15.39:0/3048972984 >> 172.21.15.39:0/3048972984 conn(0x7f51d806e9c0 msgr2=0x7f51d8070e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:39.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.949+0000 7f51df000640 1 -- 172.21.15.39:0/3048972984 shutdown_connections 2024-03-18T16:13:39.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.949+0000 7f51df000640 1 -- 172.21.15.39:0/3048972984 wait complete. 2024-03-18T16:13:39.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.951+0000 7f51df000640 1 Processor -- start 2024-03-18T16:13:39.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.951+0000 7f51df000640 1 -- start start 2024-03-18T16:13:39.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.951+0000 7f51df000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f51d81087e0 0x7f51d81a2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:39.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.951+0000 7f51df000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f51d810b150 0x7f51d81a2950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:39.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.952+0000 7f51df000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f51d810b5d0 0x7f51d81a99c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:39.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.952+0000 7f51df000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f51d8117e80 con 0x7f51d81087e0 2024-03-18T16:13:39.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.952+0000 7f51df000640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f51d8117d00 con 0x7f51d810b5d0 2024-03-18T16:13:39.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.952+0000 7f51df000640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f51d8118000 con 0x7f51d810b150 2024-03-18T16:13:39.956 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.953+0000 7f51d7400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f51d810b150 0x7f51d81a2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:39.956 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.953+0000 7f51d7400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f51d810b150 0x7f51d81a2950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:42392/0 (socket says 172.21.15.39:42392) 2024-03-18T16:13:39.956 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.953+0000 7f51d7400640 1 -- 172.21.15.39:0/3018588824 learned_addr learned my addr 172.21.15.39:0/3018588824 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:13:39.956 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.953+0000 7f51dd000640 1 --2- 172.21.15.39:0/3018588824 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f51d810b5d0 0x7f51d81a99c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:39.956 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.953+0000 7f51d7e00640 1 --2- 172.21.15.39:0/3018588824 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f51d81087e0 0x7f51d81a2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:39.956 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.954+0000 7f51d7400640 1 -- 172.21.15.39:0/3018588824 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f51d810b5d0 msgr2=0x7f51d81a99c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:39.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.954+0000 7f51d7400640 1 --2- 172.21.15.39:0/3018588824 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f51d810b5d0 0x7f51d81a99c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:39.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.954+0000 7f51d7400640 1 -- 172.21.15.39:0/3018588824 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f51d81087e0 msgr2=0x7f51d81a2410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:39.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.954+0000 7f51d7400640 1 --2- 172.21.15.39:0/3018588824 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f51d81087e0 0x7f51d81a2410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:39.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.954+0000 7f51d7400640 1 -- 172.21.15.39:0/3018588824 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51d81aa030 con 0x7f51d810b150 2024-03-18T16:13:39.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.954+0000 7f51d7e00640 1 --2- 172.21.15.39:0/3018588824 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f51d81087e0 0x7f51d81a2410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:13:39.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.955+0000 7f51d7400640 1 --2- 172.21.15.39:0/3018588824 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f51d810b150 0x7f51d81a2950 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f51d000eb90 tx=0x7f51d000c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:39.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.955+0000 7f51d4c00640 1 -- 172.21.15.39:0/3018588824 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51d0004d20 con 0x7f51d810b150 2024-03-18T16:13:39.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.956+0000 7f51d4c00640 1 -- 172.21.15.39:0/3018588824 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51d000ee90 con 0x7f51d810b150 2024-03-18T16:13:39.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.956+0000 7f51d4c00640 1 -- 172.21.15.39:0/3018588824 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51d0010830 con 0x7f51d810b150 2024-03-18T16:13:39.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.956+0000 7f51df000640 1 -- 172.21.15.39:0/3018588824 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51d81aa320 con 0x7f51d810b150 2024-03-18T16:13:39.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.956+0000 7f51df000640 1 -- 172.21.15.39:0/3018588824 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51d81aa820 con 0x7f51d810b150 2024-03-18T16:13:39.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.968+0000 7f51d4c00640 1 -- 172.21.15.39:0/3018588824 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f51d0005020 con 0x7f51d810b150 2024-03-18T16:13:39.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.969+0000 7f51d4c00640 1 --2- 172.21.15.39:0/3018588824 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f51bc0a2950 0x7f51bc0a4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:39.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.969+0000 7f51d4c00640 1 -- 172.21.15.39:0/3018588824 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f51d00cf490 con 0x7f51d810b150 2024-03-18T16:13:39.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.970+0000 7f51d7e00640 1 --2- 172.21.15.39:0/3018588824 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f51bc0a2950 0x7f51bc0a4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:39.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.974+0000 7f51d7e00640 1 --2- 172.21.15.39:0/3018588824 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f51bc0a2950 0x7f51bc0a4e10 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f51d806a920 tx=0x7f51c4005db0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:39.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.975+0000 7f51e1d6a740 1 Processor -- start 2024-03-18T16:13:39.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.976+0000 7f51e1d6a740 1 -- start start 2024-03-18T16:13:39.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.976+0000 7f51e1d6a740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555a810a30d0 0x555a810a34f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:39.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.976+0000 7f51e1d6a740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555a810a3ac0 0x555a810a85c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:39.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.976+0000 7f51e1d6a740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555a810a8c30 0x555a810ab0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:39.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.976+0000 7f51e1d6a740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x555a810623b0 con 0x555a810a3ac0 2024-03-18T16:13:39.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.976+0000 7f51e1d6a740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x555a80fff450 con 0x555a810a30d0 2024-03-18T16:13:39.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.976+0000 7f51e1d6a740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x555a80fa7cb0 con 0x555a810a8c30 2024-03-18T16:13:39.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.978+0000 7f51dd000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555a810a8c30 0x555a810ab0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:39.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.978+0000 7f51dd000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555a810a8c30 0x555a810ab0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:42404/0 (socket says 172.21.15.39:42404) 2024-03-18T16:13:39.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.978+0000 7f51dd000640 1 -- 172.21.15.39:0/1991719989 learned_addr learned my addr 172.21.15.39:0/1991719989 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:13:39.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.978+0000 7f51d7e00640 1 --2- 172.21.15.39:0/1991719989 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555a810a30d0 0x555a810a34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:39.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.978+0000 7f51d7400640 1 --2- 172.21.15.39:0/1991719989 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555a810a3ac0 0x555a810a85c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:39.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.979+0000 7f51d7400640 1 -- 172.21.15.39:0/1991719989 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555a810a8c30 msgr2=0x555a810ab0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:39.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.979+0000 7f51d7400640 1 --2- 172.21.15.39:0/1991719989 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555a810a8c30 0x555a810ab0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:39.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.979+0000 7f51d7400640 1 -- 172.21.15.39:0/1991719989 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555a810a30d0 msgr2=0x555a810a34f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:39.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.979+0000 7f51d7400640 1 --2- 172.21.15.39:0/1991719989 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555a810a30d0 0x555a810a34f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:39.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.979+0000 7f51d7400640 1 -- 172.21.15.39:0/1991719989 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555a80fc91f0 con 0x555a810a3ac0 2024-03-18T16:13:39.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.979+0000 7f51dd000640 1 --2- 172.21.15.39:0/1991719989 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555a810a8c30 0x555a810ab0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:13:39.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.979+0000 7f51d7e00640 1 --2- 172.21.15.39:0/1991719989 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555a810a30d0 0x555a810a34f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:13:39.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.979+0000 7f51d7400640 1 --2- 172.21.15.39:0/1991719989 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555a810a3ac0 0x555a810a85c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f51d00049d0 tx=0x7f51d009b580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f75fc94d85da0c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:39.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.979+0000 7f51cd600640 1 -- 172.21.15.39:0/1991719989 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51d009ba40 con 0x555a810a3ac0 2024-03-18T16:13:39.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.980+0000 7f51cd600640 1 -- 172.21.15.39:0/1991719989 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51d009fb90 con 0x555a810a3ac0 2024-03-18T16:13:39.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.980+0000 7f51cd600640 1 -- 172.21.15.39:0/1991719989 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51d0005640 con 0x555a810a3ac0 2024-03-18T16:13:39.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.981+0000 7f51e1d6a740 1 -- 172.21.15.39:0/1991719989 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555a810a3ac0 msgr2=0x555a810a85c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:39.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.981+0000 7f51e1d6a740 1 --2- 172.21.15.39:0/1991719989 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555a810a3ac0 0x555a810a85c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f51d00049d0 tx=0x7f51d009b580 comp rx=0 tx=0).stop 2024-03-18T16:13:39.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.981+0000 7f51e1d6a740 1 -- 172.21.15.39:0/1991719989 shutdown_connections 2024-03-18T16:13:39.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.981+0000 7f51e1d6a740 1 --2- 172.21.15.39:0/1991719989 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555a810a8c30 0x555a810ab0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:39.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.981+0000 7f51e1d6a740 1 --2- 172.21.15.39:0/1991719989 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555a810a3ac0 0x555a810a85c0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:39.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.981+0000 7f51e1d6a740 1 --2- 172.21.15.39:0/1991719989 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555a810a30d0 0x555a810a34f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:39.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.981+0000 7f51e1d6a740 1 -- 172.21.15.39:0/1991719989 >> 172.21.15.39:0/1991719989 conn(0x555a8109e640 msgr2=0x555a8109ea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:39.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.981+0000 7f51e1d6a740 1 -- 172.21.15.39:0/1991719989 shutdown_connections 2024-03-18T16:13:39.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.981+0000 7f51e1d6a740 1 -- 172.21.15.39:0/1991719989 wait complete. 2024-03-18T16:13:39.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.982+0000 7f51e1d6a740 1 -- start start 2024-03-18T16:13:39.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.982+0000 7f51e1d6a740 1 Processor -- start 2024-03-18T16:13:39.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.983+0000 7f51e1d6a740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555a810a30d0 0x555a810ad430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:39.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.983+0000 7f51e1d6a740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555a810a3ac0 0x555a810ad970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:39.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.983+0000 7f51e1d6a740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555a810a8c30 0x555a810b49b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:39.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.984+0000 7f51e1d6a740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x555a80fbb110 con 0x555a810a3ac0 2024-03-18T16:13:39.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.984+0000 7f51e1d6a740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x555a80ffb4f0 con 0x555a810a8c30 2024-03-18T16:13:39.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.984+0000 7f51e1d6a740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x555a810abbe0 con 0x555a810a30d0 2024-03-18T16:13:39.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.984+0000 7f51e1d6a740 10 client.0 _ll_register_callbacks cb 0x555a80fa8b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:13:39.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.984+0000 7f51d7e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555a810a30d0 0x555a810ad430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:39.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.984+0000 7f51dd000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555a810a8c30 0x555a810b49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:39.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.984+0000 7f51dd000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555a810a8c30 0x555a810b49b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:60098/0 (socket says 172.21.15.39:60098) 2024-03-18T16:13:39.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.984+0000 7f51d7400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555a810a3ac0 0x555a810ad970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:39.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.984+0000 7f51dd000640 1 -- 172.21.15.39:0/3219985075 learned_addr learned my addr 172.21.15.39:0/3219985075 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:13:39.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.985+0000 7f51dd000640 1 -- 172.21.15.39:0/3219985075 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555a810a30d0 msgr2=0x555a810ad430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:39.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.985+0000 7f51dd000640 1 --2- 172.21.15.39:0/3219985075 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555a810a30d0 0x555a810ad430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:39.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.985+0000 7f51dd000640 1 -- 172.21.15.39:0/3219985075 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555a810a3ac0 msgr2=0x555a810ad970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:39.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.985+0000 7f51dd000640 1 --2- 172.21.15.39:0/3219985075 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555a810a3ac0 0x555a810ad970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:39.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.985+0000 7f51dd000640 1 -- 172.21.15.39:0/3219985075 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555a80d133f0 con 0x555a810a8c30 2024-03-18T16:13:39.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.985+0000 7f51d7400640 1 --2- 172.21.15.39:0/3219985075 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555a810a3ac0 0x555a810ad970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:13:39.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.986+0000 7f51dd000640 1 --2- 172.21.15.39:0/3219985075 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555a810a8c30 0x555a810b49b0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f51c802c7b0 tx=0x7f51c8004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eee3ee2cb0c7658a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:39.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.986+0000 7f51c2000640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:13:39.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.986+0000 7f51c2000640 1 -- 172.21.15.39:0/3219985075 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51c8046070 con 0x555a810a8c30 2024-03-18T16:13:39.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.987+0000 7f51c2000640 1 -- 172.21.15.39:0/3219985075 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f51c8005250 con 0x555a810a8c30 2024-03-18T16:13:39.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.987+0000 7f51e1d6a740 1 -- 172.21.15.39:0/3219985075 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f51d009fb90 con 0x555a810a8c30 2024-03-18T16:13:39.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.987+0000 7f51c2000640 1 -- 172.21.15.39:0/3219985075 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f51c8050050 con 0x555a810a8c30 2024-03-18T16:13:39.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.988+0000 7f51e1d6a740 10 client.26330 fetch_fsmap learned FSMap version 36 2024-03-18T16:13:39.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.989+0000 7f51e1d6a740 1 -- 172.21.15.39:0/3219985075 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x555a80ddd7d0 con 0x555a810a8c30 2024-03-18T16:13:39.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.989+0000 7f51c2000640 1 -- 172.21.15.39:0/3219985075 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51d009fb90 con 0x555a810a8c30 2024-03-18T16:13:39.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.989+0000 7f51c2000640 1 -- 172.21.15.39:0/3219985075 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 36) ==== 8406+0+0 (secure 0 0 0) 0x7f51c80035f0 con 0x555a810a8c30 2024-03-18T16:13:39.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.989+0000 7f51e1d6a740 10 client.26330 fetch_fsmap finished waiting for FSMap version 36 2024-03-18T16:13:39.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.989+0000 7f51e1d6a740 10 client.26330 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-18T16:13:39.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.989+0000 7f51e1d6a740 20 client.26330 populate_metadata read hostname 'smithi039' 2024-03-18T16:13:39.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.989+0000 7f51e1d6a740 1 --2- 172.21.15.39:0/3219985075 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x555a810b9340 0x555a810bb7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:39.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.989+0000 7f51e1d6a740 4 client.26330 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:13:39.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.989+0000 7f51e1d6a740 1 -- 172.21.15.39:0/3219985075 --> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555a8100c990 con 0x555a810b9340 2024-03-18T16:13:39.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.991+0000 7f51d7e00640 1 --2- 172.21.15.39:0/3219985075 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x555a810b9340 0x555a810bb7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:39.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.996+0000 7f51c2000640 10 client.26330 ms_handle_connect on v2:172.21.15.196:6836/895247251 2024-03-18T16:13:39.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.996+0000 7f51d7e00640 1 --2- 172.21.15.39:0/3219985075 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x555a810b9340 0x555a810bb7a0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a6af0ac9c120f2e9 server_cookie=dbc6f56cd71d2a6f in_seq=0 out_seq=0 2024-03-18T16:13:40.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.997+0000 7f51c2000640 1 -- 172.21.15.39:0/3219985075 <== mds.2 v2:172.21.15.196:6836/895247251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x555a8100c990 con 0x555a810b9340 2024-03-18T16:13:40.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.997+0000 7f51c2000640 10 client.26330 handle_command_reply: tid=0 2024-03-18T16:13:40.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.997+0000 7f51e1d6a740 1 client.26330 shutdown 2024-03-18T16:13:40.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.998+0000 7f51e1d6a740 1 -- 172.21.15.39:0/3219985075 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555a810a8c30 msgr2=0x555a810b49b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:40.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.998+0000 7f51e1d6a740 1 --2- 172.21.15.39:0/3219985075 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555a810a8c30 0x555a810b49b0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f51c802c7b0 tx=0x7f51c8004270 comp rx=0 tx=0).stop 2024-03-18T16:13:40.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.998+0000 7f51e1d6a740 1 -- 172.21.15.39:0/3219985075 shutdown_connections 2024-03-18T16:13:40.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.998+0000 7f51e1d6a740 1 --2- 172.21.15.39:0/3219985075 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555a810a8c30 0x555a810b49b0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:40.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.998+0000 7f51e1d6a740 1 --2- 172.21.15.39:0/3219985075 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x555a810b9340 0x555a810bb7a0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:40.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.998+0000 7f51e1d6a740 1 --2- 172.21.15.39:0/3219985075 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555a810a3ac0 0x555a810ad970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:40.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.998+0000 7f51e1d6a740 1 --2- 172.21.15.39:0/3219985075 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555a810a30d0 0x555a810ad430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:40.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.998+0000 7f51c2000640 0 client.26330 ms_handle_reset on v2:172.21.15.196:6836/895247251 2024-03-18T16:13:40.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.998+0000 7f51e1d6a740 1 -- 172.21.15.39:0/3219985075 >> 172.21.15.39:0/3219985075 conn(0x555a8109e640 msgr2=0x555a8109f890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:40.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.999+0000 7f51e1d6a740 1 -- 172.21.15.39:0/3219985075 shutdown_connections 2024-03-18T16:13:40.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.999+0000 7f51e1d6a740 1 -- 172.21.15.39:0/3219985075 wait complete. 2024-03-18T16:13:40.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:39.999+0000 7f51e1d6a740 20 client.26330 trim_cache size 0 max 16384 2024-03-18T16:13:40.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.042+0000 7f51e1d6a740 1 Processor -- start 2024-03-18T16:13:40.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.045+0000 7f51e1d6a740 1 -- start start 2024-03-18T16:13:40.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.045+0000 7f51e1d6a740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555a810a30d0 0x555a81094960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:40.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.045+0000 7f51e1d6a740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555a810a3ac0 0x555a810b4970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:40.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.045+0000 7f51e1d6a740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555a810a8c30 0x555a81094ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:40.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.045+0000 7f51e1d6a740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f51d81b0a00 con 0x555a810a8c30 2024-03-18T16:13:40.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.045+0000 7f51e1d6a740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f51d81b26c0 con 0x555a810a3ac0 2024-03-18T16:13:40.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.045+0000 7f51e1d6a740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f51d81b10d0 con 0x555a810a30d0 2024-03-18T16:13:40.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.046+0000 7f51d7400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555a810a3ac0 0x555a810b4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:40.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.046+0000 7f51d7400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555a810a3ac0 0x555a810b4970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:60100/0 (socket says 172.21.15.39:60100) 2024-03-18T16:13:40.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.046+0000 7f51d7400640 1 -- 172.21.15.39:0/1603987082 learned_addr learned my addr 172.21.15.39:0/1603987082 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:13:40.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.046+0000 7f51d7400640 1 -- 172.21.15.39:0/1603987082 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555a810a30d0 msgr2=0x555a81094960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:40.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.046+0000 7f51d7e00640 1 --2- 172.21.15.39:0/1603987082 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555a810a30d0 0x555a81094960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:40.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.046+0000 7f51d7400640 1 --2- 172.21.15.39:0/1603987082 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555a810a30d0 0x555a81094960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:40.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.046+0000 7f51d7400640 1 -- 172.21.15.39:0/1603987082 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555a810a8c30 msgr2=0x555a81094ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:40.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.046+0000 7f51d7400640 1 --2- 172.21.15.39:0/1603987082 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555a810a8c30 0x555a81094ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:40.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.046+0000 7f51d7400640 1 -- 172.21.15.39:0/1603987082 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555a810abd20 con 0x555a810a3ac0 2024-03-18T16:13:40.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.046+0000 7f51d7e00640 1 --2- 172.21.15.39:0/1603987082 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555a810a30d0 0x555a81094960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:13:40.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.047+0000 7f51d7400640 1 --2- 172.21.15.39:0/1603987082 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555a810a3ac0 0x555a810b4970 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f51d0098410 tx=0x7f51d0005960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5676701419980111 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:40.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.047+0000 7f51ccc00640 1 -- 172.21.15.39:0/1603987082 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51d00a41e0 con 0x555a810a3ac0 2024-03-18T16:13:40.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.047+0000 7f51ccc00640 1 -- 172.21.15.39:0/1603987082 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51d00b9070 con 0x555a810a3ac0 2024-03-18T16:13:40.050 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.047+0000 7f51ccc00640 1 -- 172.21.15.39:0/1603987082 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51d00a3d50 con 0x555a810a3ac0 2024-03-18T16:13:40.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.048+0000 7f51e1d6a740 1 -- 172.21.15.39:0/1603987082 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555a810a3ac0 msgr2=0x555a810b4970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:40.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.048+0000 7f51e1d6a740 1 --2- 172.21.15.39:0/1603987082 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555a810a3ac0 0x555a810b4970 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f51d0098410 tx=0x7f51d0005960 comp rx=0 tx=0).stop 2024-03-18T16:13:40.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.048+0000 7f51e1d6a740 1 -- 172.21.15.39:0/1603987082 shutdown_connections 2024-03-18T16:13:40.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.048+0000 7f51e1d6a740 1 --2- 172.21.15.39:0/1603987082 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555a810a3ac0 0x555a810b4970 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:40.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.048+0000 7f51e1d6a740 1 --2- 172.21.15.39:0/1603987082 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555a810a8c30 0x555a81094ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:40.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.048+0000 7f51e1d6a740 1 --2- 172.21.15.39:0/1603987082 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555a810a30d0 0x555a81094960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:40.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.048+0000 7f51e1d6a740 1 -- 172.21.15.39:0/1603987082 >> 172.21.15.39:0/1603987082 conn(0x555a8109e640 msgr2=0x555a810ad430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:40.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.048+0000 7f51e1d6a740 1 -- 172.21.15.39:0/1603987082 shutdown_connections 2024-03-18T16:13:40.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.049+0000 7f51e1d6a740 1 -- 172.21.15.39:0/1603987082 wait complete. 2024-03-18T16:13:40.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.050+0000 7f51e1d6a740 1 -- start start 2024-03-18T16:13:40.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.050+0000 7f51e1d6a740 1 Processor -- start 2024-03-18T16:13:40.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.052+0000 7f51e1d6a740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555a810a30d0 0x555a810b23a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:40.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.052+0000 7f51e1d6a740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555a810a3ac0 0x555a810b28e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:40.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.052+0000 7f51e1d6a740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555a810a8c30 0x555a810b38a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:40.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.053+0000 7f51dd000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555a810a8c30 0x555a810b38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:40.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.053+0000 7f51dd000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555a810a8c30 0x555a810b38a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:42430/0 (socket says 172.21.15.39:42430) 2024-03-18T16:13:40.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.053+0000 7f51dd000640 1 -- 172.21.15.39:0/1292879496 learned_addr learned my addr 172.21.15.39:0/1292879496 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:13:40.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.053+0000 7f51e1d6a740 1 -- 172.21.15.39:0/1292879496 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f51d81b1f30 con 0x555a810a30d0 2024-03-18T16:13:40.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.053+0000 7f51e1d6a740 1 -- 172.21.15.39:0/1292879496 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f51d81b17a0 con 0x555a810a3ac0 2024-03-18T16:13:40.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.053+0000 7f51e1d6a740 1 -- 172.21.15.39:0/1292879496 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f51d81ae9e0 con 0x555a810a8c30 2024-03-18T16:13:40.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.053+0000 7f51e1d6a740 10 client.0 _ll_register_callbacks cb 0x7f51c8009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:13:40.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.054+0000 7f51d7e00640 1 --2- 172.21.15.39:0/1292879496 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555a810a30d0 0x555a810b23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:40.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.054+0000 7f51dd000640 1 -- 172.21.15.39:0/1292879496 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555a810a3ac0 msgr2=0x555a810b28e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:40.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.054+0000 7f51d7400640 1 --2- 172.21.15.39:0/1292879496 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555a810a3ac0 0x555a810b28e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:40.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.055+0000 7f51dd000640 1 --2- 172.21.15.39:0/1292879496 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555a810a3ac0 0x555a810b28e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:40.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.055+0000 7f51dd000640 1 -- 172.21.15.39:0/1292879496 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555a810a30d0 msgr2=0x555a810b23a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:40.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.055+0000 7f51dd000640 1 --2- 172.21.15.39:0/1292879496 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555a810a30d0 0x555a810b23a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:40.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.055+0000 7f51dd000640 1 -- 172.21.15.39:0/1292879496 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555a810b3e90 con 0x555a810a8c30 2024-03-18T16:13:40.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.056+0000 7f51c2000640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:13:40.058 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.056+0000 7f51dd000640 1 --2- 172.21.15.39:0/1292879496 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555a810a8c30 0x555a810b38a0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f51c8009850 tx=0x7f51c8005bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bc63522647c2410f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:40.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.056+0000 7f51c2000640 1 -- 172.21.15.39:0/1292879496 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51c8046070 con 0x555a810a8c30 2024-03-18T16:13:40.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.057+0000 7f51c2000640 1 -- 172.21.15.39:0/1292879496 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f51c80046c0 con 0x555a810a8c30 2024-03-18T16:13:40.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.057+0000 7f51e1d6a740 1 -- 172.21.15.39:0/1292879496 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f51d00b9070 con 0x555a810a8c30 2024-03-18T16:13:40.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.058+0000 7f51c2000640 1 -- 172.21.15.39:0/1292879496 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f51c80044a0 con 0x555a810a8c30 2024-03-18T16:13:40.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.058+0000 7f51e1d6a740 10 client.26332 fetch_fsmap learned FSMap version 36 2024-03-18T16:13:40.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.058+0000 7f51e1d6a740 1 -- 172.21.15.39:0/1292879496 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x555a8111f860 con 0x555a810a8c30 2024-03-18T16:13:40.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.059+0000 7f51c2000640 1 -- 172.21.15.39:0/1292879496 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51d00b9070 con 0x555a810a8c30 2024-03-18T16:13:40.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.059+0000 7f51c2000640 1 -- 172.21.15.39:0/1292879496 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 36) ==== 8406+0+0 (secure 0 0 0) 0x7f51c80404e0 con 0x555a810a8c30 2024-03-18T16:13:40.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.059+0000 7f51e1d6a740 10 client.26332 fetch_fsmap finished waiting for FSMap version 36 2024-03-18T16:13:40.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.059+0000 7f51e1d6a740 10 client.26332 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-18T16:13:40.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.059+0000 7f51e1d6a740 20 client.26332 populate_metadata read hostname 'smithi039' 2024-03-18T16:13:40.063 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.059+0000 7f51e1d6a740 1 --2- 172.21.15.39:0/1292879496 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x555a81120230 0x555a81122650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:40.063 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.059+0000 7f51e1d6a740 4 client.26332 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:13:40.063 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.059+0000 7f51e1d6a740 1 -- 172.21.15.39:0/1292879496 --> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555a81006ce0 con 0x555a81120230 2024-03-18T16:13:40.063 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.060+0000 7f51d7e00640 1 --2- 172.21.15.39:0/1292879496 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x555a81120230 0x555a81122650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:40.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.062+0000 7f51d7e00640 1 --2- 172.21.15.39:0/1292879496 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x555a81120230 0x555a81122650 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2c7b31a266033166 server_cookie=931d3ceefca44c88 in_seq=0 out_seq=0 2024-03-18T16:13:40.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.063+0000 7f51c2000640 10 client.26332 ms_handle_connect on v2:172.21.15.196:6836/895247251 2024-03-18T16:13:40.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.063+0000 7f51c2000640 1 -- 172.21.15.39:0/1292879496 <== mds.2 v2:172.21.15.196:6836/895247251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555a81006ce0 con 0x555a81120230 2024-03-18T16:13:40.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.063+0000 7f51c2000640 10 client.26332 handle_command_reply: tid=0 2024-03-18T16:13:40.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.063+0000 7f51e1d6a740 1 client.26332 shutdown 2024-03-18T16:13:40.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.063+0000 7f51e1d6a740 1 -- 172.21.15.39:0/1292879496 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555a810a8c30 msgr2=0x555a810b38a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:40.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.063+0000 7f51e1d6a740 1 --2- 172.21.15.39:0/1292879496 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555a810a8c30 0x555a810b38a0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f51c8009850 tx=0x7f51c8005bc0 comp rx=0 tx=0).stop 2024-03-18T16:13:40.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.064+0000 7f51e1d6a740 1 -- 172.21.15.39:0/1292879496 shutdown_connections 2024-03-18T16:13:40.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.064+0000 7f51e1d6a740 1 --2- 172.21.15.39:0/1292879496 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555a810a3ac0 0x555a810b28e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:40.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.064+0000 7f51e1d6a740 1 --2- 172.21.15.39:0/1292879496 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x555a81120230 0x555a81122650 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:40.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.064+0000 7f51e1d6a740 1 --2- 172.21.15.39:0/1292879496 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555a810a8c30 0x555a810b38a0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:40.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.064+0000 7f51e1d6a740 1 --2- 172.21.15.39:0/1292879496 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555a810a30d0 0x555a810b23a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:40.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.064+0000 7f51e1d6a740 1 -- 172.21.15.39:0/1292879496 >> 172.21.15.39:0/1292879496 conn(0x555a8109e640 msgr2=0x555a81094960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:40.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.064+0000 7f51c2000640 0 client.26332 ms_handle_reset on v2:172.21.15.196:6836/895247251 2024-03-18T16:13:40.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.064+0000 7f51e1d6a740 1 -- 172.21.15.39:0/1292879496 shutdown_connections 2024-03-18T16:13:40.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.064+0000 7f51e1d6a740 1 -- 172.21.15.39:0/1292879496 wait complete. 2024-03-18T16:13:40.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.064+0000 7f51e1d6a740 20 client.26332 trim_cache size 0 max 16384 2024-03-18T16:13:40.068 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:13:40.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.065+0000 7f51df000640 1 -- 172.21.15.39:0/3018588824 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f51bc0a2950 msgr2=0x7f51bc0a4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:40.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.065+0000 7f51df000640 1 --2- 172.21.15.39:0/3018588824 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f51bc0a2950 0x7f51bc0a4e10 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f51d806a920 tx=0x7f51c4005db0 comp rx=0 tx=0).stop 2024-03-18T16:13:40.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.065+0000 7f51df000640 1 -- 172.21.15.39:0/3018588824 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f51d810b150 msgr2=0x7f51d81a2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:40.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.065+0000 7f51df000640 1 --2- 172.21.15.39:0/3018588824 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f51d810b150 0x7f51d81a2950 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f51d000eb90 tx=0x7f51d000c6a0 comp rx=0 tx=0).stop 2024-03-18T16:13:40.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.066+0000 7f51df000640 1 -- 172.21.15.39:0/3018588824 shutdown_connections 2024-03-18T16:13:40.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.066+0000 7f51df000640 1 --2- 172.21.15.39:0/3018588824 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f51d810b5d0 0x7f51d81a99c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:40.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.066+0000 7f51df000640 1 --2- 172.21.15.39:0/3018588824 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f51bc0a2950 0x7f51bc0a4e10 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:40.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.066+0000 7f51df000640 1 --2- 172.21.15.39:0/3018588824 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f51d810b150 0x7f51d81a2950 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:40.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.066+0000 7f51df000640 1 --2- 172.21.15.39:0/3018588824 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f51d81087e0 0x7f51d81a2410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:40.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.066+0000 7f51df000640 1 -- 172.21.15.39:0/3018588824 >> 172.21.15.39:0/3018588824 conn(0x7f51d806e9c0 msgr2=0x7f51d806f300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:40.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.068+0000 7f51df000640 1 -- 172.21.15.39:0/3018588824 shutdown_connections 2024-03-18T16:13:40.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.068+0000 7f51df000640 1 -- 172.21.15.39:0/3018588824 wait complete. 2024-03-18T16:13:40.103 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-18T16:13:40.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.436+0000 7f5df5800640 1 -- 172.21.15.39:0/1005093773 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5df0108db0 msgr2=0x7f5df0109210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:40.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.436+0000 7f5df5800640 1 --2- 172.21.15.39:0/1005093773 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5df0108db0 0x7f5df0109210 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f5de0009900 tx=0x7f5de002bc60 comp rx=0 tx=0).stop 2024-03-18T16:13:40.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.437+0000 7f5df5800640 1 -- 172.21.15.39:0/1005093773 shutdown_connections 2024-03-18T16:13:40.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.437+0000 7f5df5800640 1 --2- 172.21.15.39:0/1005093773 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5df0109750 0x7f5df010e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:40.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.437+0000 7f5df5800640 1 --2- 172.21.15.39:0/1005093773 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5df0108db0 0x7f5df0109210 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:40.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.437+0000 7f5df5800640 1 --2- 172.21.15.39:0/1005093773 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5df0072d90 0x7f5df00731b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:40.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.437+0000 7f5df5800640 1 -- 172.21.15.39:0/1005093773 >> 172.21.15.39:0/1005093773 conn(0x7f5df006e600 msgr2=0x7f5df0070a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:40.440 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.438+0000 7f5df5800640 1 -- 172.21.15.39:0/1005093773 shutdown_connections 2024-03-18T16:13:40.440 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.438+0000 7f5df5800640 1 -- 172.21.15.39:0/1005093773 wait complete. 2024-03-18T16:13:40.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.439+0000 7f5df5800640 1 Processor -- start 2024-03-18T16:13:40.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.444+0000 7f5df5800640 1 -- start start 2024-03-18T16:13:40.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.444+0000 7f5df5800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5df0072d90 0x7f5df01a67a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:40.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.444+0000 7f5df5800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5df0108db0 0x7f5df01a6ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:40.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.445+0000 7f5df5800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5df0109750 0x7f5df01add50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:40.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.445+0000 7f5df5800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f5df011c180 con 0x7f5df0072d90 2024-03-18T16:13:40.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.445+0000 7f5df5800640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f5df011c000 con 0x7f5df0108db0 2024-03-18T16:13:40.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.445+0000 7f5df5800640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f5df011c300 con 0x7f5df0109750 2024-03-18T16:13:40.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.445+0000 7f5def400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5df0108db0 0x7f5df01a6ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:40.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.445+0000 7f5def400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5df0108db0 0x7f5df01a6ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:60120/0 (socket says 172.21.15.39:60120) 2024-03-18T16:13:40.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.445+0000 7f5def400640 1 -- 172.21.15.39:0/823428318 learned_addr learned my addr 172.21.15.39:0/823428318 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:13:40.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.445+0000 7f5defe00640 1 --2- 172.21.15.39:0/823428318 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5df0072d90 0x7f5df01a67a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:40.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.446+0000 7f5df4e00640 1 --2- 172.21.15.39:0/823428318 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5df0109750 0x7f5df01add50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:40.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.446+0000 7f5def400640 1 -- 172.21.15.39:0/823428318 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5df0109750 msgr2=0x7f5df01add50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:40.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.446+0000 7f5def400640 1 --2- 172.21.15.39:0/823428318 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5df0109750 0x7f5df01add50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:40.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.446+0000 7f5def400640 1 -- 172.21.15.39:0/823428318 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5df0072d90 msgr2=0x7f5df01a67a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:40.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.446+0000 7f5def400640 1 --2- 172.21.15.39:0/823428318 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5df0072d90 0x7f5df01a67a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:40.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.446+0000 7f5def400640 1 -- 172.21.15.39:0/823428318 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5df01ae3c0 con 0x7f5df0108db0 2024-03-18T16:13:40.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.446+0000 7f5def400640 1 --2- 172.21.15.39:0/823428318 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5df0108db0 0x7f5df01a6ce0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f5de002c270 tx=0x7f5de002c670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:40.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.447+0000 7f5decc00640 1 -- 172.21.15.39:0/823428318 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5de0041070 con 0x7f5df0108db0 2024-03-18T16:13:40.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.447+0000 7f5decc00640 1 -- 172.21.15.39:0/823428318 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5de00054e0 con 0x7f5df0108db0 2024-03-18T16:13:40.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.447+0000 7f5decc00640 1 -- 172.21.15.39:0/823428318 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5de0034440 con 0x7f5df0108db0 2024-03-18T16:13:40.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.447+0000 7f5df5800640 1 -- 172.21.15.39:0/823428318 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5df01ae650 con 0x7f5df0108db0 2024-03-18T16:13:40.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.447+0000 7f5df5800640 1 -- 172.21.15.39:0/823428318 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5df01aeaf0 con 0x7f5df0108db0 2024-03-18T16:13:40.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.454+0000 7f5decc00640 1 -- 172.21.15.39:0/823428318 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f5de0044070 con 0x7f5df0108db0 2024-03-18T16:13:40.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.458+0000 7f5decc00640 1 --2- 172.21.15.39:0/823428318 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f5dd40a2a00 0x7f5dd40a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:40.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.458+0000 7f5df9bed740 1 Processor -- start 2024-03-18T16:13:40.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.458+0000 7f5decc00640 1 -- 172.21.15.39:0/823428318 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f5de00f0bb0 con 0x7f5df0108db0 2024-03-18T16:13:40.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.459+0000 7f5df9bed740 1 -- start start 2024-03-18T16:13:40.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.460+0000 7f5df9bed740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5567bc169a90 0x5567bc169eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:40.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.460+0000 7f5df9bed740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5567bc16a480 0x5567bc16ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:40.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.460+0000 7f5df9bed740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5567bc16f5f0 0x5567bc171a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:40.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.460+0000 7f5df9bed740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5567bc1313b0 con 0x5567bc169a90 2024-03-18T16:13:40.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.460+0000 7f5df9bed740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5567bc0ce450 con 0x5567bc16f5f0 2024-03-18T16:13:40.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.460+0000 7f5df9bed740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x5567bc076cb0 con 0x5567bc16a480 2024-03-18T16:13:40.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.460+0000 7f5defe00640 1 --2- 172.21.15.39:0/823428318 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f5dd40a2a00 0x7f5dd40a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:40.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.461+0000 7f5def400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5567bc16a480 0x5567bc16ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:40.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.462+0000 7f5def400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5567bc16a480 0x5567bc16ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:42458/0 (socket says 172.21.15.39:42458) 2024-03-18T16:13:40.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.462+0000 7f5def400640 1 -- 172.21.15.39:0/3387194086 learned_addr learned my addr 172.21.15.39:0/3387194086 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:13:40.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.460+0000 7f5df4e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5567bc16f5f0 0x5567bc171a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:40.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.462+0000 7f5def400640 1 -- 172.21.15.39:0/3387194086 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5567bc16f5f0 msgr2=0x5567bc171a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:40.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.462+0000 7f5def400640 1 --2- 172.21.15.39:0/3387194086 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5567bc16f5f0 0x5567bc171a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:40.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.462+0000 7f5def400640 1 -- 172.21.15.39:0/3387194086 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5567bc169a90 msgr2=0x5567bc169eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:40.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.462+0000 7f5def400640 1 --2- 172.21.15.39:0/3387194086 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5567bc169a90 0x5567bc169eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:40.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.462+0000 7f5def400640 1 -- 172.21.15.39:0/3387194086 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5567bc0981f0 con 0x5567bc16a480 2024-03-18T16:13:40.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.463+0000 7f5df4e00640 1 --2- 172.21.15.39:0/3387194086 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5567bc16f5f0 0x5567bc171a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:13:40.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.463+0000 7f5def400640 1 --2- 172.21.15.39:0/3387194086 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5567bc16a480 0x5567bc16ef80 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f5de00b9bc0 tx=0x7f5de002bc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4f9d2d07717b32ee server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:40.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.463+0000 7f5defe00640 1 --2- 172.21.15.39:0/823428318 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f5dd40a2a00 0x7f5dd40a4ec0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f5dd8009ab0 tx=0x7f5dd8009510 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:40.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.465+0000 7f5de5600640 1 -- 172.21.15.39:0/3387194086 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5de004b070 con 0x5567bc16a480 2024-03-18T16:13:40.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.465+0000 7f5de5600640 1 -- 172.21.15.39:0/3387194086 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5de00bd8f0 con 0x5567bc16a480 2024-03-18T16:13:40.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.465+0000 7f5de5600640 1 -- 172.21.15.39:0/3387194086 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5de0005000 con 0x5567bc16a480 2024-03-18T16:13:40.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.466+0000 7f5df9bed740 1 -- 172.21.15.39:0/3387194086 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5567bc16a480 msgr2=0x5567bc16ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:40.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.466+0000 7f5df9bed740 1 --2- 172.21.15.39:0/3387194086 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5567bc16a480 0x5567bc16ef80 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f5de00b9bc0 tx=0x7f5de002bc60 comp rx=0 tx=0).stop 2024-03-18T16:13:40.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.466+0000 7f5df9bed740 1 -- 172.21.15.39:0/3387194086 shutdown_connections 2024-03-18T16:13:40.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.466+0000 7f5df9bed740 1 --2- 172.21.15.39:0/3387194086 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5567bc16f5f0 0x5567bc171a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:40.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.466+0000 7f5df9bed740 1 --2- 172.21.15.39:0/3387194086 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5567bc16a480 0x5567bc16ef80 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:40.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.466+0000 7f5df9bed740 1 --2- 172.21.15.39:0/3387194086 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5567bc169a90 0x5567bc169eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:40.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.466+0000 7f5df9bed740 1 -- 172.21.15.39:0/3387194086 >> 172.21.15.39:0/3387194086 conn(0x5567bc165000 msgr2=0x5567bc165400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:40.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.466+0000 7f5df9bed740 1 -- 172.21.15.39:0/3387194086 shutdown_connections 2024-03-18T16:13:40.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.466+0000 7f5df9bed740 1 -- 172.21.15.39:0/3387194086 wait complete. 2024-03-18T16:13:40.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.467+0000 7f5df9bed740 1 -- start start 2024-03-18T16:13:40.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.467+0000 7f5df9bed740 1 Processor -- start 2024-03-18T16:13:40.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.469+0000 7f5df9bed740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5567bc169a90 0x5567bc173ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:40.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.469+0000 7f5df9bed740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5567bc16a480 0x5567bc173ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:40.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.469+0000 7f5df9bed740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5567bc16f5f0 0x5567bc17afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:40.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.470+0000 7f5defe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5567bc169a90 0x5567bc173ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:40.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.470+0000 7f5defe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5567bc169a90 0x5567bc173ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:42472/0 (socket says 172.21.15.39:42472) 2024-03-18T16:13:40.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.470+0000 7f5defe00640 1 -- 172.21.15.39:0/2529586363 learned_addr learned my addr 172.21.15.39:0/2529586363 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:13:40.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.470+0000 7f5df4e00640 1 --2- 172.21.15.39:0/2529586363 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5567bc16f5f0 0x5567bc17afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:40.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.470+0000 7f5df9bed740 1 -- 172.21.15.39:0/2529586363 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5567bc08a110 con 0x5567bc16a480 2024-03-18T16:13:40.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.471+0000 7f5def400640 1 --2- 172.21.15.39:0/2529586363 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5567bc16a480 0x5567bc173ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:40.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.471+0000 7f5df9bed740 1 -- 172.21.15.39:0/2529586363 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5567bc0ca4f0 con 0x5567bc16f5f0 2024-03-18T16:13:40.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.471+0000 7f5df9bed740 1 -- 172.21.15.39:0/2529586363 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x5567bc172360 con 0x5567bc169a90 2024-03-18T16:13:40.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.471+0000 7f5df9bed740 10 client.0 _ll_register_callbacks cb 0x5567bc077b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:13:40.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.472+0000 7f5defe00640 1 -- 172.21.15.39:0/2529586363 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5567bc16f5f0 msgr2=0x5567bc17afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:40.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.472+0000 7f5defe00640 1 --2- 172.21.15.39:0/2529586363 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5567bc16f5f0 0x5567bc17afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:40.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.472+0000 7f5defe00640 1 -- 172.21.15.39:0/2529586363 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5567bc16a480 msgr2=0x5567bc173ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:40.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.472+0000 7f5defe00640 1 --2- 172.21.15.39:0/2529586363 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5567bc16a480 0x5567bc173ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:40.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.472+0000 7f5defe00640 1 -- 172.21.15.39:0/2529586363 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5567bbde23f0 con 0x5567bc169a90 2024-03-18T16:13:40.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.473+0000 7f5dde000640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:13:40.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.473+0000 7f5defe00640 1 --2- 172.21.15.39:0/2529586363 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5567bc169a90 0x5567bc173ab0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f5dd800a010 tx=0x7f5dd80047d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bce519e8d67e874d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:40.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.473+0000 7f5def400640 1 --2- 172.21.15.39:0/2529586363 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5567bc16a480 0x5567bc173ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:13:40.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.473+0000 7f5dde000640 1 -- 172.21.15.39:0/2529586363 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dd8024020 con 0x5567bc169a90 2024-03-18T16:13:40.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.474+0000 7f5dde000640 1 -- 172.21.15.39:0/2529586363 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f5dd8016950 con 0x5567bc169a90 2024-03-18T16:13:40.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.474+0000 7f5df9bed740 1 -- 172.21.15.39:0/2529586363 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5de00bd8f0 con 0x5567bc169a90 2024-03-18T16:13:40.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.475+0000 7f5dde000640 1 -- 172.21.15.39:0/2529586363 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f5dd8016b50 con 0x5567bc169a90 2024-03-18T16:13:40.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.475+0000 7f5df9bed740 10 client.26344 fetch_fsmap learned FSMap version 36 2024-03-18T16:13:40.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.475+0000 7f5df9bed740 1 -- 172.21.15.39:0/2529586363 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5567bbeac7d0 con 0x5567bc169a90 2024-03-18T16:13:40.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.475+0000 7f5dde000640 1 -- 172.21.15.39:0/2529586363 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5de00bd8f0 con 0x5567bc169a90 2024-03-18T16:13:40.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.478+0000 7f5dde000640 1 -- 172.21.15.39:0/2529586363 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 36) ==== 8406+0+0 (secure 0 0 0) 0x7f5dd8031020 con 0x5567bc169a90 2024-03-18T16:13:40.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.478+0000 7f5df9bed740 10 client.26344 fetch_fsmap finished waiting for FSMap version 36 2024-03-18T16:13:40.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.478+0000 7f5df9bed740 10 client.26344 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T16:13:40.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.478+0000 7f5df9bed740 20 client.26344 populate_metadata read hostname 'smithi039' 2024-03-18T16:13:40.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.478+0000 7f5df9bed740 1 --2- 172.21.15.39:0/2529586363 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x5567bc17f930 0x5567bc181da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:40.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.478+0000 7f5df9bed740 4 client.26344 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:13:40.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.478+0000 7f5df9bed740 1 -- 172.21.15.39:0/2529586363 --> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5567bc0db990 con 0x5567bc17f930 2024-03-18T16:13:40.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.478+0000 7f5def400640 1 --2- 172.21.15.39:0/2529586363 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x5567bc17f930 0x5567bc181da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:40.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.479+0000 7f5def400640 1 --2- 172.21.15.39:0/2529586363 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x5567bc17f930 0x5567bc181da0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=54702f0ad0819764 server_cookie=953fee0edb4ee1f9 in_seq=0 out_seq=0 2024-03-18T16:13:40.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.479+0000 7f5dde000640 10 client.26344 ms_handle_connect on v2:172.21.15.196:6838/2001427268 2024-03-18T16:13:40.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.480+0000 7f5dde000640 1 -- 172.21.15.39:0/2529586363 <== mds.1 v2:172.21.15.196:6838/2001427268 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5567bc0db990 con 0x5567bc17f930 2024-03-18T16:13:40.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.480+0000 7f5dde000640 10 client.26344 handle_command_reply: tid=0 2024-03-18T16:13:40.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.481+0000 7f5df9bed740 1 client.26344 shutdown 2024-03-18T16:13:40.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.482+0000 7f5df9bed740 1 -- 172.21.15.39:0/2529586363 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5567bc169a90 msgr2=0x5567bc173ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:40.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.482+0000 7f5df9bed740 1 --2- 172.21.15.39:0/2529586363 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5567bc169a90 0x5567bc173ab0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f5dd800a010 tx=0x7f5dd80047d0 comp rx=0 tx=0).stop 2024-03-18T16:13:40.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.482+0000 7f5df9bed740 1 -- 172.21.15.39:0/2529586363 shutdown_connections 2024-03-18T16:13:40.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.482+0000 7f5df9bed740 1 --2- 172.21.15.39:0/2529586363 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x5567bc17f930 0x5567bc181da0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:40.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.482+0000 7f5df9bed740 1 --2- 172.21.15.39:0/2529586363 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5567bc16f5f0 0x5567bc17afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:40.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.482+0000 7f5df9bed740 1 --2- 172.21.15.39:0/2529586363 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5567bc16a480 0x5567bc173ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:40.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.482+0000 7f5df9bed740 1 --2- 172.21.15.39:0/2529586363 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5567bc169a90 0x5567bc173ab0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:40.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.482+0000 7f5df9bed740 1 -- 172.21.15.39:0/2529586363 >> 172.21.15.39:0/2529586363 conn(0x5567bc165000 msgr2=0x5567bc164220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:40.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.482+0000 7f5dde000640 0 client.26344 ms_handle_reset on v2:172.21.15.196:6838/2001427268 2024-03-18T16:13:40.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.483+0000 7f5df9bed740 1 -- 172.21.15.39:0/2529586363 shutdown_connections 2024-03-18T16:13:40.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.483+0000 7f5df9bed740 1 -- 172.21.15.39:0/2529586363 wait complete. 2024-03-18T16:13:40.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.484+0000 7f5df9bed740 20 client.26344 trim_cache size 0 max 16384 2024-03-18T16:13:40.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.520+0000 7f5df9bed740 1 Processor -- start 2024-03-18T16:13:40.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.521+0000 7f5df9bed740 1 -- start start 2024-03-18T16:13:40.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.522+0000 7f5df9bed740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5567bc169a90 0x5567bc173dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:40.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.522+0000 7f5df9bed740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5567bc16a480 0x5567bc17af80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:40.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.522+0000 7f5df9bed740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5567bc16f5f0 0x5567bc163990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:40.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.522+0000 7f5df9bed740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f5df01b9060 con 0x5567bc16a480 2024-03-18T16:13:40.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.522+0000 7f5df9bed740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f5df01bad20 con 0x5567bc16f5f0 2024-03-18T16:13:40.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.522+0000 7f5df9bed740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f5df01b9730 con 0x5567bc169a90 2024-03-18T16:13:40.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.522+0000 7f5defe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5567bc169a90 0x5567bc173dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:40.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.522+0000 7f5defe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5567bc169a90 0x5567bc173dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:42482/0 (socket says 172.21.15.39:42482) 2024-03-18T16:13:40.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.522+0000 7f5defe00640 1 -- 172.21.15.39:0/1219687629 learned_addr learned my addr 172.21.15.39:0/1219687629 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:13:40.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.523+0000 7f5df4e00640 1 --2- 172.21.15.39:0/1219687629 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5567bc16f5f0 0x5567bc163990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:40.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.523+0000 7f5def400640 1 --2- 172.21.15.39:0/1219687629 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5567bc16a480 0x5567bc17af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:40.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.523+0000 7f5df4e00640 1 -- 172.21.15.39:0/1219687629 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5567bc169a90 msgr2=0x5567bc173dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:40.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.523+0000 7f5df4e00640 1 --2- 172.21.15.39:0/1219687629 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5567bc169a90 0x5567bc173dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:40.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.523+0000 7f5df4e00640 1 -- 172.21.15.39:0/1219687629 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5567bc16a480 msgr2=0x5567bc17af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:40.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.523+0000 7f5df4e00640 1 --2- 172.21.15.39:0/1219687629 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5567bc16a480 0x5567bc17af80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:40.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.523+0000 7f5df4e00640 1 -- 172.21.15.39:0/1219687629 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5567bc16a860 con 0x5567bc16f5f0 2024-03-18T16:13:40.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.524+0000 7f5df4e00640 1 --2- 172.21.15.39:0/1219687629 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5567bc16f5f0 0x5567bc163990 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f5de800a5c0 tx=0x7f5de80097f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d34e87a84424285 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:40.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.525+0000 7f5de4c00640 1 -- 172.21.15.39:0/1219687629 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5de8007980 con 0x5567bc16f5f0 2024-03-18T16:13:40.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.525+0000 7f5de4c00640 1 -- 172.21.15.39:0/1219687629 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5de801d070 con 0x5567bc16f5f0 2024-03-18T16:13:40.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.525+0000 7f5de4c00640 1 -- 172.21.15.39:0/1219687629 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5de8017350 con 0x5567bc16f5f0 2024-03-18T16:13:40.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.526+0000 7f5df9bed740 1 -- 172.21.15.39:0/1219687629 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5567bc16f5f0 msgr2=0x5567bc163990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:40.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.526+0000 7f5df9bed740 1 --2- 172.21.15.39:0/1219687629 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5567bc16f5f0 0x5567bc163990 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f5de800a5c0 tx=0x7f5de80097f0 comp rx=0 tx=0).stop 2024-03-18T16:13:40.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.527+0000 7f5df9bed740 1 -- 172.21.15.39:0/1219687629 shutdown_connections 2024-03-18T16:13:40.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.527+0000 7f5df9bed740 1 --2- 172.21.15.39:0/1219687629 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5567bc16f5f0 0x5567bc163990 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:40.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.527+0000 7f5df9bed740 1 --2- 172.21.15.39:0/1219687629 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5567bc16a480 0x5567bc17af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:40.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.527+0000 7f5df9bed740 1 --2- 172.21.15.39:0/1219687629 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5567bc169a90 0x5567bc173dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:40.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.527+0000 7f5df9bed740 1 -- 172.21.15.39:0/1219687629 >> 172.21.15.39:0/1219687629 conn(0x5567bc165000 msgr2=0x5567bc1736c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:40.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.528+0000 7f5df9bed740 1 -- 172.21.15.39:0/1219687629 shutdown_connections 2024-03-18T16:13:40.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.528+0000 7f5df9bed740 1 -- 172.21.15.39:0/1219687629 wait complete. 2024-03-18T16:13:40.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.528+0000 7f5df9bed740 1 -- start start 2024-03-18T16:13:40.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.529+0000 7f5df9bed740 1 Processor -- start 2024-03-18T16:13:40.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.530+0000 7f5df9bed740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5567bc169a90 0x5567bc177e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:40.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.530+0000 7f5df9bed740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5567bc16a480 0x5567bc1783c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:40.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.530+0000 7f5df9bed740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5567bc16f5f0 0x5567bc1e3cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:40.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.531+0000 7f5df9bed740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f5df01ba590 con 0x5567bc16f5f0 2024-03-18T16:13:40.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.531+0000 7f5df9bed740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f5df01b9e00 con 0x5567bc16a480 2024-03-18T16:13:40.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.531+0000 7f5df9bed740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f5df01b7040 con 0x5567bc169a90 2024-03-18T16:13:40.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.531+0000 7f5df9bed740 10 client.0 _ll_register_callbacks cb 0x7f5dd80096a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:13:40.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.531+0000 7f5def400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5567bc16a480 0x5567bc1783c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:40.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.531+0000 7f5def400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5567bc16a480 0x5567bc1783c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:60152/0 (socket says 172.21.15.39:60152) 2024-03-18T16:13:40.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.531+0000 7f5def400640 1 -- 172.21.15.39:0/283974288 learned_addr learned my addr 172.21.15.39:0/283974288 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:13:40.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.531+0000 7f5defe00640 1 --2- 172.21.15.39:0/283974288 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5567bc169a90 0x5567bc177e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:40.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.531+0000 7f5df4e00640 1 --2- 172.21.15.39:0/283974288 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5567bc16f5f0 0x5567bc1e3cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:40.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.532+0000 7f5defe00640 1 -- 172.21.15.39:0/283974288 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5567bc16a480 msgr2=0x5567bc1783c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:40.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.532+0000 7f5defe00640 1 --2- 172.21.15.39:0/283974288 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5567bc16a480 0x5567bc1783c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:40.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.532+0000 7f5defe00640 1 -- 172.21.15.39:0/283974288 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5567bc16f5f0 msgr2=0x5567bc1e3cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:40.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.532+0000 7f5defe00640 1 --2- 172.21.15.39:0/283974288 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5567bc16f5f0 0x5567bc1e3cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:40.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.532+0000 7f5defe00640 1 -- 172.21.15.39:0/283974288 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5567bc1e4210 con 0x5567bc169a90 2024-03-18T16:13:40.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.533+0000 7f5def400640 1 --2- 172.21.15.39:0/283974288 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5567bc16a480 0x5567bc1783c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:13:40.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.534+0000 7f5dde000640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:13:40.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.534+0000 7f5defe00640 1 --2- 172.21.15.39:0/283974288 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5567bc169a90 0x5567bc177e80 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x5567bc0717b0 tx=0x7f5dd8029930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3a001a1fd166ac0b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:40.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.535+0000 7f5dde000640 1 -- 172.21.15.39:0/283974288 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dd8024040 con 0x5567bc169a90 2024-03-18T16:13:40.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.535+0000 7f5dde000640 1 -- 172.21.15.39:0/283974288 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f5dd8029cd0 con 0x5567bc169a90 2024-03-18T16:13:40.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.536+0000 7f5df9bed740 1 -- 172.21.15.39:0/283974288 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5de801d070 con 0x5567bc169a90 2024-03-18T16:13:40.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.536+0000 7f5dde000640 1 -- 172.21.15.39:0/283974288 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f5dd8016960 con 0x5567bc169a90 2024-03-18T16:13:40.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.536+0000 7f5df9bed740 10 client.26350 fetch_fsmap learned FSMap version 36 2024-03-18T16:13:40.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.536+0000 7f5df9bed740 1 -- 172.21.15.39:0/283974288 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5567bc1e5540 con 0x5567bc169a90 2024-03-18T16:13:40.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.537+0000 7f5dde000640 1 -- 172.21.15.39:0/283974288 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5de801d070 con 0x5567bc169a90 2024-03-18T16:13:40.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.537+0000 7f5dde000640 1 -- 172.21.15.39:0/283974288 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 36) ==== 8406+0+0 (secure 0 0 0) 0x7f5dd8031040 con 0x5567bc169a90 2024-03-18T16:13:40.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.538+0000 7f5df9bed740 10 client.26350 fetch_fsmap finished waiting for FSMap version 36 2024-03-18T16:13:40.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.538+0000 7f5df9bed740 10 client.26350 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T16:13:40.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.538+0000 7f5df9bed740 20 client.26350 populate_metadata read hostname 'smithi039' 2024-03-18T16:13:40.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.538+0000 7f5df9bed740 1 --2- 172.21.15.39:0/283974288 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x5567bc1e5f50 0x5567bc1e8370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:40.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.538+0000 7f5df9bed740 4 client.26350 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:13:40.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.538+0000 7f5df9bed740 1 -- 172.21.15.39:0/283974288 --> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5567bc0d5ce0 con 0x5567bc1e5f50 2024-03-18T16:13:40.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.539+0000 7f5def400640 1 --2- 172.21.15.39:0/283974288 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x5567bc1e5f50 0x5567bc1e8370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:40.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.541+0000 7f5def400640 1 --2- 172.21.15.39:0/283974288 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x5567bc1e5f50 0x5567bc1e8370 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e4bad9333b2b9e7f server_cookie=8b48dd0e8c33bff in_seq=0 out_seq=0 2024-03-18T16:13:40.544 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.541+0000 7f5dde000640 10 client.26350 ms_handle_connect on v2:172.21.15.196:6838/2001427268 2024-03-18T16:13:40.544 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.541+0000 7f5dde000640 1 -- 172.21.15.39:0/283974288 <== mds.1 v2:172.21.15.196:6838/2001427268 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5567bc0d5ce0 con 0x5567bc1e5f50 2024-03-18T16:13:40.544 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.541+0000 7f5dde000640 10 client.26350 handle_command_reply: tid=0 2024-03-18T16:13:40.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.542+0000 7f5df9bed740 1 client.26350 shutdown 2024-03-18T16:13:40.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.553+0000 7f5df9bed740 1 -- 172.21.15.39:0/283974288 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5567bc169a90 msgr2=0x5567bc177e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:40.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.553+0000 7f5df9bed740 1 --2- 172.21.15.39:0/283974288 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5567bc169a90 0x5567bc177e80 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x5567bc0717b0 tx=0x7f5dd8029930 comp rx=0 tx=0).stop 2024-03-18T16:13:40.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.553+0000 7f5df9bed740 1 -- 172.21.15.39:0/283974288 shutdown_connections 2024-03-18T16:13:40.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.553+0000 7f5df9bed740 1 --2- 172.21.15.39:0/283974288 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x5567bc1e5f50 0x5567bc1e8370 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:40.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.553+0000 7f5df9bed740 1 --2- 172.21.15.39:0/283974288 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5567bc16a480 0x5567bc1783c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:40.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.553+0000 7f5df9bed740 1 --2- 172.21.15.39:0/283974288 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5567bc16f5f0 0x5567bc1e3cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:40.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.553+0000 7f5dde000640 0 client.26350 ms_handle_reset on v2:172.21.15.196:6838/2001427268 2024-03-18T16:13:40.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.553+0000 7f5df9bed740 1 --2- 172.21.15.39:0/283974288 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5567bc169a90 0x5567bc177e80 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:40.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.553+0000 7f5df9bed740 1 -- 172.21.15.39:0/283974288 >> 172.21.15.39:0/283974288 conn(0x5567bc165000 msgr2=0x5567bc168260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:40.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.554+0000 7f5df9bed740 1 -- 172.21.15.39:0/283974288 shutdown_connections 2024-03-18T16:13:40.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.554+0000 7f5df9bed740 1 -- 172.21.15.39:0/283974288 wait complete. 2024-03-18T16:13:40.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.557+0000 7f5df9bed740 20 client.26350 trim_cache size 0 max 16384 2024-03-18T16:13:40.559 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:13:40.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.558+0000 7f5df5800640 1 -- 172.21.15.39:0/823428318 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f5dd40a2a00 msgr2=0x7f5dd40a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:40.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.558+0000 7f5df5800640 1 --2- 172.21.15.39:0/823428318 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f5dd40a2a00 0x7f5dd40a4ec0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f5dd8009ab0 tx=0x7f5dd8009510 comp rx=0 tx=0).stop 2024-03-18T16:13:40.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.558+0000 7f5df5800640 1 -- 172.21.15.39:0/823428318 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5df0108db0 msgr2=0x7f5df01a6ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:40.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.558+0000 7f5df5800640 1 --2- 172.21.15.39:0/823428318 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5df0108db0 0x7f5df01a6ce0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f5de002c270 tx=0x7f5de002c670 comp rx=0 tx=0).stop 2024-03-18T16:13:40.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.558+0000 7f5df5800640 1 -- 172.21.15.39:0/823428318 shutdown_connections 2024-03-18T16:13:40.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.558+0000 7f5df5800640 1 --2- 172.21.15.39:0/823428318 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5df0108db0 0x7f5df01a6ce0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:40.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.558+0000 7f5df5800640 1 --2- 172.21.15.39:0/823428318 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f5dd40a2a00 0x7f5dd40a4ec0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:40.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.558+0000 7f5df5800640 1 --2- 172.21.15.39:0/823428318 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5df0109750 0x7f5df01add50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:40.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.558+0000 7f5df5800640 1 --2- 172.21.15.39:0/823428318 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5df0072d90 0x7f5df01a67a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:40.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.558+0000 7f5df5800640 1 -- 172.21.15.39:0/823428318 >> 172.21.15.39:0/823428318 conn(0x7f5df006e600 msgr2=0x7f5df00701e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:40.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.558+0000 7f5df5800640 1 -- 172.21.15.39:0/823428318 shutdown_connections 2024-03-18T16:13:40.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.558+0000 7f5df5800640 1 -- 172.21.15.39:0/823428318 wait complete. 2024-03-18T16:13:40.593 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-18T16:13:40.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.917+0000 7f4986200640 1 -- 172.21.15.39:0/2100032437 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4980102290 msgr2=0x7f4980108b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:40.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.917+0000 7f4986200640 1 --2- 172.21.15.39:0/2100032437 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4980102290 0x7f4980108b20 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f4970009860 tx=0x7f497002bbb0 comp rx=0 tx=0).stop 2024-03-18T16:13:40.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.918+0000 7f4986200640 1 -- 172.21.15.39:0/2100032437 shutdown_connections 2024-03-18T16:13:40.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.918+0000 7f4986200640 1 --2- 172.21.15.39:0/2100032437 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4980102290 0x7f4980108b20 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:40.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.918+0000 7f4986200640 1 --2- 172.21.15.39:0/2100032437 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f49801018d0 0x7f4980101d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:40.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.918+0000 7f4986200640 1 --2- 172.21.15.39:0/2100032437 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49801006e0 0x7f4980100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:40.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.918+0000 7f4986200640 1 -- 172.21.15.39:0/2100032437 >> 172.21.15.39:0/2100032437 conn(0x7f49800fbc50 msgr2=0x7f49800fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:40.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.918+0000 7f4986200640 1 -- 172.21.15.39:0/2100032437 shutdown_connections 2024-03-18T16:13:40.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.918+0000 7f4986200640 1 -- 172.21.15.39:0/2100032437 wait complete. 2024-03-18T16:13:40.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.919+0000 7f4986200640 1 Processor -- start 2024-03-18T16:13:40.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.921+0000 7f4986200640 1 -- start start 2024-03-18T16:13:40.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.921+0000 7f4986200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f49801006e0 0x7f4980106530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:40.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.921+0000 7f4986200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49801018d0 0x7f4980106a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:40.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.921+0000 7f4986200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4980102290 0x7f49801030a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:40.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.921+0000 7f4986200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f498010d4e0 con 0x7f49801018d0 2024-03-18T16:13:40.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.921+0000 7f4986200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f498010d360 con 0x7f4980102290 2024-03-18T16:13:40.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.921+0000 7f4986200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f498010d660 con 0x7f49801006e0 2024-03-18T16:13:40.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.921+0000 7f497f400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f49801006e0 0x7f4980106530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:40.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.922+0000 7f497f400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f49801006e0 0x7f4980106530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:42502/0 (socket says 172.21.15.39:42502) 2024-03-18T16:13:40.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.922+0000 7f497f400640 1 -- 172.21.15.39:0/1434938209 learned_addr learned my addr 172.21.15.39:0/1434938209 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:13:40.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.922+0000 7f497fe00640 1 --2- 172.21.15.39:0/1434938209 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4980102290 0x7f49801030a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:40.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.922+0000 7f497f400640 1 -- 172.21.15.39:0/1434938209 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4980102290 msgr2=0x7f49801030a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:40.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.922+0000 7f497f400640 1 --2- 172.21.15.39:0/1434938209 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4980102290 0x7f49801030a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:40.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.922+0000 7f497f400640 1 -- 172.21.15.39:0/1434938209 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49801018d0 msgr2=0x7f4980106a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:40.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.922+0000 7f497f400640 1 --2- 172.21.15.39:0/1434938209 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49801018d0 0x7f4980106a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:40.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.922+0000 7f497f400640 1 -- 172.21.15.39:0/1434938209 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4980103960 con 0x7f49801006e0 2024-03-18T16:13:40.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.923+0000 7f497f400640 1 --2- 172.21.15.39:0/1434938209 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f49801006e0 0x7f4980106530 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f496c00d770 tx=0x7f496c00dc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:40.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.923+0000 7f4977e00640 1 -- 172.21.15.39:0/1434938209 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f496c017020 con 0x7f49801006e0 2024-03-18T16:13:40.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.923+0000 7f4977e00640 1 -- 172.21.15.39:0/1434938209 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f496c004440 con 0x7f49801006e0 2024-03-18T16:13:40.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.923+0000 7f4986200640 1 -- 172.21.15.39:0/1434938209 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49801a61b0 con 0x7f49801006e0 2024-03-18T16:13:40.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.923+0000 7f4977e00640 1 -- 172.21.15.39:0/1434938209 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f496c003db0 con 0x7f49801006e0 2024-03-18T16:13:40.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.923+0000 7f4986200640 1 -- 172.21.15.39:0/1434938209 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4980199980 con 0x7f49801006e0 2024-03-18T16:13:40.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.930+0000 7f4977e00640 1 -- 172.21.15.39:0/1434938209 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f496c01a020 con 0x7f49801006e0 2024-03-18T16:13:40.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.932+0000 7f4988fe6740 1 Processor -- start 2024-03-18T16:13:40.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.934+0000 7f4977e00640 1 --2- 172.21.15.39:0/1434938209 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f49640b3960 0x7f49640b5e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:40.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.934+0000 7f4977e00640 1 -- 172.21.15.39:0/1434938209 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f496c0cddf0 con 0x7f49801006e0 2024-03-18T16:13:40.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.935+0000 7f4988fe6740 1 -- start start 2024-03-18T16:13:40.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.935+0000 7f4988fe6740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bc8b5e68c0 0x55bc8b5e6ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:40.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.935+0000 7f4988fe6740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bc8b5e72b0 0x55bc8b5ebdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:40.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.935+0000 7f4988fe6740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc8b5ec420 0x55bc8b5ee8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:40.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.935+0000 7f4988fe6740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55bc8b5ae3b0 con 0x55bc8b5ec420 2024-03-18T16:13:40.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.935+0000 7f4988fe6740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55bc8b54b450 con 0x55bc8b5e72b0 2024-03-18T16:13:40.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.935+0000 7f4988fe6740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55bc8b4f3cb0 con 0x55bc8b5e68c0 2024-03-18T16:13:40.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.936+0000 7f497ea00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bc8b5e72b0 0x55bc8b5ebdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:40.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.936+0000 7f497ea00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bc8b5e72b0 0x55bc8b5ebdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:60178/0 (socket says 172.21.15.39:60178) 2024-03-18T16:13:40.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.936+0000 7f497ea00640 1 -- 172.21.15.39:0/2121390579 learned_addr learned my addr 172.21.15.39:0/2121390579 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:13:40.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.936+0000 7f497ea00640 1 -- 172.21.15.39:0/2121390579 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bc8b5e68c0 msgr2=0x55bc8b5e6ce0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:13:40.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.936+0000 7f497fe00640 1 --2- 172.21.15.39:0/1434938209 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f49640b3960 0x7f49640b5e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:40.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.936+0000 7f497ea00640 1 --2- 172.21.15.39:0/2121390579 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bc8b5e68c0 0x55bc8b5e6ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:40.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.936+0000 7f497ea00640 1 -- 172.21.15.39:0/2121390579 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc8b5ec420 msgr2=0x55bc8b5ee8b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:13:40.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.936+0000 7f497ea00640 1 --2- 172.21.15.39:0/2121390579 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc8b5ec420 0x55bc8b5ee8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:40.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.936+0000 7f497ea00640 1 -- 172.21.15.39:0/2121390579 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bc8b5151f0 con 0x55bc8b5e72b0 2024-03-18T16:13:40.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.937+0000 7f497ea00640 1 --2- 172.21.15.39:0/2121390579 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bc8b5e72b0 0x55bc8b5ebdb0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f4978009d70 tx=0x7f497800b420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b0582cc8186a0be3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:40.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.938+0000 7f497fe00640 1 --2- 172.21.15.39:0/1434938209 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f49640b3960 0x7f49640b5e20 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f4970009830 tx=0x7f4970034ae0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:40.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.938+0000 7f496be00640 1 -- 172.21.15.39:0/2121390579 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f497800bd40 con 0x55bc8b5e72b0 2024-03-18T16:13:40.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.938+0000 7f496be00640 1 -- 172.21.15.39:0/2121390579 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4978017070 con 0x55bc8b5e72b0 2024-03-18T16:13:40.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.938+0000 7f496be00640 1 -- 172.21.15.39:0/2121390579 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f497800bd40 con 0x55bc8b5e72b0 2024-03-18T16:13:40.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.938+0000 7f4988fe6740 1 -- 172.21.15.39:0/2121390579 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bc8b5e72b0 msgr2=0x55bc8b5ebdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:40.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.938+0000 7f4988fe6740 1 --2- 172.21.15.39:0/2121390579 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bc8b5e72b0 0x55bc8b5ebdb0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f4978009d70 tx=0x7f497800b420 comp rx=0 tx=0).stop 2024-03-18T16:13:40.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.941+0000 7f4988fe6740 1 -- 172.21.15.39:0/2121390579 shutdown_connections 2024-03-18T16:13:40.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.941+0000 7f4988fe6740 1 --2- 172.21.15.39:0/2121390579 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bc8b5e72b0 0x55bc8b5ebdb0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:40.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.941+0000 7f4988fe6740 1 --2- 172.21.15.39:0/2121390579 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc8b5ec420 0x55bc8b5ee8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:40.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.941+0000 7f4988fe6740 1 --2- 172.21.15.39:0/2121390579 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bc8b5e68c0 0x55bc8b5e6ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:40.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.941+0000 7f4988fe6740 1 -- 172.21.15.39:0/2121390579 >> 172.21.15.39:0/2121390579 conn(0x55bc8b5e1e30 msgr2=0x55bc8b5e2230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:40.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.941+0000 7f4988fe6740 1 -- 172.21.15.39:0/2121390579 shutdown_connections 2024-03-18T16:13:40.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.941+0000 7f4988fe6740 1 -- 172.21.15.39:0/2121390579 wait complete. 2024-03-18T16:13:40.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.942+0000 7f4988fe6740 1 -- start start 2024-03-18T16:13:40.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.942+0000 7f4988fe6740 1 Processor -- start 2024-03-18T16:13:40.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.944+0000 7f4988fe6740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bc8b5e68c0 0x55bc8b5f0c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:40.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.944+0000 7f4988fe6740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bc8b5e72b0 0x55bc8b5f1150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:40.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.944+0000 7f4988fe6740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc8b5ec420 0x55bc8b5f8190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:40.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.944+0000 7f497ea00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bc8b5e72b0 0x55bc8b5f1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:40.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.944+0000 7f4988fe6740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55bc8b507110 con 0x55bc8b5ec420 2024-03-18T16:13:40.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.944+0000 7f497ea00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bc8b5e72b0 0x55bc8b5f1150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:60190/0 (socket says 172.21.15.39:60190) 2024-03-18T16:13:40.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.944+0000 7f497ea00640 1 -- 172.21.15.39:0/2938463251 learned_addr learned my addr 172.21.15.39:0/2938463251 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:13:40.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.944+0000 7f4988fe6740 1 -- 172.21.15.39:0/2938463251 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55bc8b5474f0 con 0x55bc8b5e72b0 2024-03-18T16:13:40.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.944+0000 7f4988fe6740 1 -- 172.21.15.39:0/2938463251 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55bc8b5ef3d0 con 0x55bc8b5e68c0 2024-03-18T16:13:40.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.945+0000 7f497fe00640 1 --2- 172.21.15.39:0/2938463251 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc8b5ec420 0x55bc8b5f8190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:40.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.945+0000 7f4988fe6740 10 client.0 _ll_register_callbacks cb 0x55bc8b4f4b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:13:40.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.945+0000 7f497fe00640 1 -- 172.21.15.39:0/2938463251 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bc8b5e68c0 msgr2=0x55bc8b5f0c10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T16:13:40.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.945+0000 7f497fe00640 1 --2- 172.21.15.39:0/2938463251 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bc8b5e68c0 0x55bc8b5f0c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:40.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.945+0000 7f497fe00640 1 -- 172.21.15.39:0/2938463251 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bc8b5e72b0 msgr2=0x55bc8b5f1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:40.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.945+0000 7f497fe00640 1 --2- 172.21.15.39:0/2938463251 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bc8b5e72b0 0x55bc8b5f1150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:40.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.945+0000 7f497fe00640 1 -- 172.21.15.39:0/2938463251 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bc8b25f3f0 con 0x55bc8b5ec420 2024-03-18T16:13:40.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.946+0000 7f497fe00640 1 --2- 172.21.15.39:0/2938463251 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc8b5ec420 0x55bc8b5f8190 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f4970009780 tx=0x7f4970005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f84a088ca7ca7094 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:40.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.946+0000 7f4968c00640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-03-18T16:13:40.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.946+0000 7f4968c00640 1 -- 172.21.15.39:0/2938463251 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4970046040 con 0x55bc8b5ec420 2024-03-18T16:13:40.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.946+0000 7f4968c00640 1 -- 172.21.15.39:0/2938463251 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f4970005040 con 0x55bc8b5ec420 2024-03-18T16:13:40.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.946+0000 7f4988fe6740 1 -- 172.21.15.39:0/2938463251 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4978017070 con 0x55bc8b5ec420 2024-03-18T16:13:40.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.949+0000 7f4968c00640 1 -- 172.21.15.39:0/2938463251 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f4970058050 con 0x55bc8b5ec420 2024-03-18T16:13:40.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.949+0000 7f4988fe6740 10 client.16053 fetch_fsmap learned FSMap version 36 2024-03-18T16:13:40.952 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.949+0000 7f4988fe6740 1 -- 172.21.15.39:0/2938463251 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55bc8b3297d0 con 0x55bc8b5ec420 2024-03-18T16:13:40.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.950+0000 7f4968c00640 1 -- 172.21.15.39:0/2938463251 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4978017070 con 0x55bc8b5ec420 2024-03-18T16:13:40.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.950+0000 7f4968c00640 1 -- 172.21.15.39:0/2938463251 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 36) ==== 8406+0+0 (secure 0 0 0) 0x7f49700046c0 con 0x55bc8b5ec420 2024-03-18T16:13:40.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.950+0000 7f4988fe6740 10 client.16053 fetch_fsmap finished waiting for FSMap version 36 2024-03-18T16:13:40.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.950+0000 7f4988fe6740 10 client.16053 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:13:40.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.950+0000 7f4988fe6740 20 client.16053 populate_metadata read hostname 'smithi039' 2024-03-18T16:13:40.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.950+0000 7f4988fe6740 1 --2- 172.21.15.39:0/2938463251 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55bc8b5fcb90 0x55bc8b5ff050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:40.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.950+0000 7f4988fe6740 4 client.16053 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:13:40.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.950+0000 7f4988fe6740 1 -- 172.21.15.39:0/2938463251 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bc8b558990 con 0x55bc8b5fcb90 2024-03-18T16:13:40.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.951+0000 7f497f400640 1 --2- 172.21.15.39:0/2938463251 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55bc8b5fcb90 0x55bc8b5ff050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:40.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.951+0000 7f4968c00640 10 client.16053 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:13:40.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.951+0000 7f497f400640 1 --2- 172.21.15.39:0/2938463251 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55bc8b5fcb90 0x55bc8b5ff050 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=21edaf5b6f70e767 server_cookie=cf4b1d91dbdba753 in_seq=0 out_seq=0 2024-03-18T16:13:40.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.952+0000 7f4968c00640 1 -- 172.21.15.39:0/2938463251 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55bc8b558990 con 0x55bc8b5fcb90 2024-03-18T16:13:40.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.952+0000 7f4968c00640 10 client.16053 handle_command_reply: tid=0 2024-03-18T16:13:40.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.952+0000 7f4988fe6740 1 client.16053 shutdown 2024-03-18T16:13:40.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.953+0000 7f4988fe6740 1 -- 172.21.15.39:0/2938463251 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc8b5ec420 msgr2=0x55bc8b5f8190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:40.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.954+0000 7f4988fe6740 1 --2- 172.21.15.39:0/2938463251 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc8b5ec420 0x55bc8b5f8190 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f4970009780 tx=0x7f4970005f90 comp rx=0 tx=0).stop 2024-03-18T16:13:40.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.955+0000 7f4988fe6740 1 -- 172.21.15.39:0/2938463251 shutdown_connections 2024-03-18T16:13:40.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.955+0000 7f4988fe6740 1 --2- 172.21.15.39:0/2938463251 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55bc8b5fcb90 0x55bc8b5ff050 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:40.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.955+0000 7f4988fe6740 1 --2- 172.21.15.39:0/2938463251 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bc8b5e72b0 0x55bc8b5f1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:40.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.955+0000 7f4988fe6740 1 --2- 172.21.15.39:0/2938463251 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc8b5ec420 0x55bc8b5f8190 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:40.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.955+0000 7f4988fe6740 1 --2- 172.21.15.39:0/2938463251 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bc8b5e68c0 0x55bc8b5f0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:40.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.955+0000 7f4988fe6740 1 -- 172.21.15.39:0/2938463251 >> 172.21.15.39:0/2938463251 conn(0x55bc8b5e1e30 msgr2=0x55bc8b5e11f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:40.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.955+0000 7f4968c00640 0 client.16053 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:13:40.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.955+0000 7f4988fe6740 1 -- 172.21.15.39:0/2938463251 shutdown_connections 2024-03-18T16:13:40.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.955+0000 7f4988fe6740 1 -- 172.21.15.39:0/2938463251 wait complete. 2024-03-18T16:13:40.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.955+0000 7f4988fe6740 20 client.16053 trim_cache size 0 max 16384 2024-03-18T16:13:40.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.988+0000 7f4988fe6740 1 Processor -- start 2024-03-18T16:13:40.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.988+0000 7f4988fe6740 1 -- start start 2024-03-18T16:13:40.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.989+0000 7f4988fe6740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bc8b5e1e30 0x55bc8b5e87b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:40.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.989+0000 7f4988fe6740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bc8b5e68c0 0x55bc8b5e8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:40.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.989+0000 7f4988fe6740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc8b5e72b0 0x55bc8b5f8150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:40.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.989+0000 7f4988fe6740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f49801a83b0 con 0x55bc8b5e72b0 2024-03-18T16:13:40.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.989+0000 7f4988fe6740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f49801aa070 con 0x55bc8b5e68c0 2024-03-18T16:13:40.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.989+0000 7f4988fe6740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f49801a8a80 con 0x55bc8b5e1e30 2024-03-18T16:13:40.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.989+0000 7f497f400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bc8b5e1e30 0x55bc8b5e87b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:40.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.989+0000 7f497f400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bc8b5e1e30 0x55bc8b5e87b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:42530/0 (socket says 172.21.15.39:42530) 2024-03-18T16:13:40.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.989+0000 7f497f400640 1 -- 172.21.15.39:0/4287425037 learned_addr learned my addr 172.21.15.39:0/4287425037 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:13:40.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.989+0000 7f497ea00640 1 --2- 172.21.15.39:0/4287425037 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bc8b5e68c0 0x55bc8b5e8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:40.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.990+0000 7f497fe00640 1 --2- 172.21.15.39:0/4287425037 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc8b5e72b0 0x55bc8b5f8150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:40.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.990+0000 7f497f400640 1 -- 172.21.15.39:0/4287425037 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bc8b5e68c0 msgr2=0x55bc8b5e8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:40.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.990+0000 7f497f400640 1 --2- 172.21.15.39:0/4287425037 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bc8b5e68c0 0x55bc8b5e8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:40.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.990+0000 7f497f400640 1 -- 172.21.15.39:0/4287425037 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc8b5e72b0 msgr2=0x55bc8b5f8150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:40.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.990+0000 7f497f400640 1 --2- 172.21.15.39:0/4287425037 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc8b5e72b0 0x55bc8b5f8150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:40.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.990+0000 7f497f400640 1 -- 172.21.15.39:0/4287425037 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bc8b5ddc10 con 0x55bc8b5e1e30 2024-03-18T16:13:40.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.991+0000 7f497f400640 1 --2- 172.21.15.39:0/4287425037 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bc8b5e1e30 0x55bc8b5e87b0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f496c09a330 tx=0x7f496c0cd040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6152a0f8ea435d67 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:40.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.991+0000 7f496b400640 1 -- 172.21.15.39:0/4287425037 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f496c0b2070 con 0x55bc8b5e1e30 2024-03-18T16:13:40.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.991+0000 7f496b400640 1 -- 172.21.15.39:0/4287425037 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f496c09aaa0 con 0x55bc8b5e1e30 2024-03-18T16:13:40.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.991+0000 7f496b400640 1 -- 172.21.15.39:0/4287425037 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f496c098730 con 0x55bc8b5e1e30 2024-03-18T16:13:40.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.993+0000 7f4988fe6740 1 -- 172.21.15.39:0/4287425037 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bc8b5e1e30 msgr2=0x55bc8b5e87b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:40.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.993+0000 7f4988fe6740 1 --2- 172.21.15.39:0/4287425037 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bc8b5e1e30 0x55bc8b5e87b0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f496c09a330 tx=0x7f496c0cd040 comp rx=0 tx=0).stop 2024-03-18T16:13:40.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.994+0000 7f4988fe6740 1 -- 172.21.15.39:0/4287425037 shutdown_connections 2024-03-18T16:13:40.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.994+0000 7f4988fe6740 1 --2- 172.21.15.39:0/4287425037 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bc8b5e68c0 0x55bc8b5e8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:40.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.994+0000 7f4988fe6740 1 --2- 172.21.15.39:0/4287425037 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc8b5e72b0 0x55bc8b5f8150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:40.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.994+0000 7f4988fe6740 1 --2- 172.21.15.39:0/4287425037 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bc8b5e1e30 0x55bc8b5e87b0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:40.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.994+0000 7f4988fe6740 1 -- 172.21.15.39:0/4287425037 >> 172.21.15.39:0/4287425037 conn(0x55bc8b5dd410 msgr2=0x55bc8b5f0c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:40.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.994+0000 7f4988fe6740 1 -- 172.21.15.39:0/4287425037 shutdown_connections 2024-03-18T16:13:40.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.995+0000 7f4988fe6740 1 -- 172.21.15.39:0/4287425037 wait complete. 2024-03-18T16:13:40.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.995+0000 7f4988fe6740 1 -- start start 2024-03-18T16:13:40.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.996+0000 7f4988fe6740 1 Processor -- start 2024-03-18T16:13:40.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.997+0000 7f4988fe6740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bc8b5e1e30 0x55bc8b5f3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:41.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.997+0000 7f4988fe6740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc8b5e68c0 0x55bc8b5f3760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:41.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.997+0000 7f4988fe6740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bc8b5e72b0 0x55bc8b5f6780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:41.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.997+0000 7f4988fe6740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f49801a98e0 con 0x55bc8b5e68c0 2024-03-18T16:13:41.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.997+0000 7f4988fe6740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f49801a9150 con 0x55bc8b5e1e30 2024-03-18T16:13:41.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.997+0000 7f4988fe6740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f49801a1ee0 con 0x55bc8b5e72b0 2024-03-18T16:13:41.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.997+0000 7f4988fe6740 10 client.0 _ll_register_callbacks cb 0x7f4970043040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:13:41.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.998+0000 7f497fe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bc8b5e72b0 0x55bc8b5f6780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:41.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.998+0000 7f497fe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bc8b5e72b0 0x55bc8b5f6780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:42540/0 (socket says 172.21.15.39:42540) 2024-03-18T16:13:41.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.998+0000 7f497fe00640 1 -- 172.21.15.39:0/386976567 learned_addr learned my addr 172.21.15.39:0/386976567 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:13:41.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.998+0000 7f497f400640 1 --2- 172.21.15.39:0/386976567 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bc8b5e1e30 0x55bc8b5f3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:41.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.999+0000 7f497ea00640 1 --2- 172.21.15.39:0/386976567 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc8b5e68c0 0x55bc8b5f3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:41.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.999+0000 7f497fe00640 1 -- 172.21.15.39:0/386976567 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bc8b5e1e30 msgr2=0x55bc8b5f3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:41.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.999+0000 7f497fe00640 1 --2- 172.21.15.39:0/386976567 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bc8b5e1e30 0x55bc8b5f3220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:41.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.999+0000 7f497fe00640 1 -- 172.21.15.39:0/386976567 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc8b5e68c0 msgr2=0x55bc8b5f3760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:41.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.999+0000 7f497fe00640 1 --2- 172.21.15.39:0/386976567 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc8b5e68c0 0x55bc8b5f3760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:41.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.999+0000 7f497fe00640 1 -- 172.21.15.39:0/386976567 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bc8b5f6d70 con 0x55bc8b5e72b0 2024-03-18T16:13:41.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:40.999+0000 7f497fe00640 1 --2- 172.21.15.39:0/386976567 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bc8b5e72b0 0x55bc8b5f6780 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f4970004bb0 tx=0x7f49700324e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=55d3d93a3767190c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:41.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:41.000+0000 7f4968c00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:13:41.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:41.000+0000 7f4968c00640 1 -- 172.21.15.39:0/386976567 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4970062060 con 0x55bc8b5e72b0 2024-03-18T16:13:41.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:41.000+0000 7f4968c00640 1 -- 172.21.15.39:0/386976567 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f49700051e0 con 0x55bc8b5e72b0 2024-03-18T16:13:41.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:41.000+0000 7f4988fe6740 1 -- 172.21.15.39:0/386976567 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f496c09aaa0 con 0x55bc8b5e72b0 2024-03-18T16:13:41.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:41.001+0000 7f4968c00640 1 -- 172.21.15.39:0/386976567 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f4970046040 con 0x55bc8b5e72b0 2024-03-18T16:13:41.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:41.001+0000 7f4988fe6740 10 client.26359 fetch_fsmap learned FSMap version 36 2024-03-18T16:13:41.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:41.001+0000 7f4988fe6740 1 -- 172.21.15.39:0/386976567 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55bc8b5f8070 con 0x55bc8b5e72b0 2024-03-18T16:13:41.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:41.002+0000 7f4968c00640 1 -- 172.21.15.39:0/386976567 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f496c09aaa0 con 0x55bc8b5e72b0 2024-03-18T16:13:41.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:41.002+0000 7f4968c00640 1 -- 172.21.15.39:0/386976567 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 36) ==== 8406+0+0 (secure 0 0 0) 0x7f49700327a0 con 0x55bc8b5e72b0 2024-03-18T16:13:41.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:41.002+0000 7f4988fe6740 10 client.26359 fetch_fsmap finished waiting for FSMap version 36 2024-03-18T16:13:41.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:41.002+0000 7f4988fe6740 10 client.26359 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:13:41.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:41.002+0000 7f4988fe6740 20 client.26359 populate_metadata read hostname 'smithi039' 2024-03-18T16:13:41.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:41.002+0000 7f4988fe6740 1 --2- 172.21.15.39:0/386976567 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55bc8b5ec420 0x55bc8b676370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:41.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:41.002+0000 7f4988fe6740 4 client.26359 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:13:41.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:41.002+0000 7f4988fe6740 1 -- 172.21.15.39:0/386976567 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55bc8b5e82c0 con 0x55bc8b5ec420 2024-03-18T16:13:41.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:41.002+0000 7f497f400640 1 --2- 172.21.15.39:0/386976567 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55bc8b5ec420 0x55bc8b676370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:41.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:41.004+0000 7f4968c00640 10 client.26359 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:13:41.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:41.005+0000 7f497f400640 1 --2- 172.21.15.39:0/386976567 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55bc8b5ec420 0x55bc8b676370 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4d4cb8029a92045 server_cookie=d220547b4897063a in_seq=0 out_seq=0 2024-03-18T16:13:41.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:41.005+0000 7f4968c00640 1 -- 172.21.15.39:0/386976567 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55bc8b5e82c0 con 0x55bc8b5ec420 2024-03-18T16:13:41.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:41.005+0000 7f4968c00640 10 client.26359 handle_command_reply: tid=0 2024-03-18T16:13:41.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:41.005+0000 7f4988fe6740 1 client.26359 shutdown 2024-03-18T16:13:41.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:41.006+0000 7f4988fe6740 1 -- 172.21.15.39:0/386976567 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bc8b5e72b0 msgr2=0x55bc8b5f6780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:41.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:41.006+0000 7f4988fe6740 1 --2- 172.21.15.39:0/386976567 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bc8b5e72b0 0x55bc8b5f6780 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f4970004bb0 tx=0x7f49700324e0 comp rx=0 tx=0).stop 2024-03-18T16:13:41.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:41.006+0000 7f4988fe6740 1 -- 172.21.15.39:0/386976567 shutdown_connections 2024-03-18T16:13:41.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:41.006+0000 7f4988fe6740 1 --2- 172.21.15.39:0/386976567 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55bc8b5ec420 0x55bc8b676370 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:41.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:41.006+0000 7f4988fe6740 1 --2- 172.21.15.39:0/386976567 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bc8b5e72b0 0x55bc8b5f6780 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:41.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:41.006+0000 7f4988fe6740 1 --2- 172.21.15.39:0/386976567 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc8b5e68c0 0x55bc8b5f3760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:41.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:41.006+0000 7f4968c00640 0 client.26359 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:13:41.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:41.006+0000 7f4988fe6740 1 --2- 172.21.15.39:0/386976567 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bc8b5e1e30 0x55bc8b5f3220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:41.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:41.006+0000 7f4988fe6740 1 -- 172.21.15.39:0/386976567 >> 172.21.15.39:0/386976567 conn(0x55bc8b5dd410 msgr2=0x55bc8b5f0bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:41.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:41.007+0000 7f4988fe6740 1 -- 172.21.15.39:0/386976567 shutdown_connections 2024-03-18T16:13:41.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:41.007+0000 7f4988fe6740 1 -- 172.21.15.39:0/386976567 wait complete. 2024-03-18T16:13:41.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:41.007+0000 7f4988fe6740 20 client.26359 trim_cache size 0 max 16384 2024-03-18T16:13:41.010 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:13:41.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:41.009+0000 7f4986200640 1 -- 172.21.15.39:0/1434938209 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f49640b3960 msgr2=0x7f49640b5e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:41.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:41.009+0000 7f4986200640 1 --2- 172.21.15.39:0/1434938209 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f49640b3960 0x7f49640b5e20 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f4970009830 tx=0x7f4970034ae0 comp rx=0 tx=0).stop 2024-03-18T16:13:41.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:41.009+0000 7f4986200640 1 -- 172.21.15.39:0/1434938209 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f49801006e0 msgr2=0x7f4980106530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:41.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:41.009+0000 7f4986200640 1 --2- 172.21.15.39:0/1434938209 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f49801006e0 0x7f4980106530 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f496c00d770 tx=0x7f496c00dc30 comp rx=0 tx=0).stop 2024-03-18T16:13:41.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:41.010+0000 7f4986200640 1 -- 172.21.15.39:0/1434938209 shutdown_connections 2024-03-18T16:13:41.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:41.010+0000 7f4986200640 1 --2- 172.21.15.39:0/1434938209 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4980102290 0x7f49801030a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:41.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:41.010+0000 7f4986200640 1 --2- 172.21.15.39:0/1434938209 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f49640b3960 0x7f49640b5e20 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:41.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:41.010+0000 7f4986200640 1 --2- 172.21.15.39:0/1434938209 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49801018d0 0x7f4980106a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:41.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:41.010+0000 7f4986200640 1 --2- 172.21.15.39:0/1434938209 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f49801006e0 0x7f4980106530 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:41.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:41.010+0000 7f4986200640 1 -- 172.21.15.39:0/1434938209 >> 172.21.15.39:0/1434938209 conn(0x7f49800fbc50 msgr2=0x7f49801094e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:41.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:41.010+0000 7f4986200640 1 -- 172.21.15.39:0/1434938209 shutdown_connections 2024-03-18T16:13:41.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:41.010+0000 7f4986200640 1 -- 172.21.15.39:0/1434938209 wait complete. 2024-03-18T16:13:41.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:13:41 smithi039 ceph-mon[31721]: pgmap v530: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 62 MiB/s wr, 7.96k op/s 2024-03-18T16:13:41.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:13:41 smithi196 ceph-mon[36307]: pgmap v530: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 62 MiB/s wr, 7.96k op/s 2024-03-18T16:13:41.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:13:41 smithi194 ceph-mon[36200]: pgmap v530: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 62 MiB/s wr, 7.96k op/s 2024-03-18T16:13:42.049 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-18T16:13:42.049 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T16:13:42.049 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-18T16:13:42.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.422+0000 7f1cee800640 1 -- 172.21.15.39:0/964088739 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1ce810dc90 msgr2=0x7f1ce8110080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:42.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.425+0000 7f1ce7e00640 1 -- 172.21.15.39:0/964088739 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cd4032340 con 0x7f1ce810dc90 2024-03-18T16:13:42.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.425+0000 7f1cee800640 1 --2- 172.21.15.39:0/964088739 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1ce810dc90 0x7f1ce8110080 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f1cd4009860 tx=0x7f1cd402bb70 comp rx=0 tx=0).stop 2024-03-18T16:13:42.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.425+0000 7f1cee800640 1 -- 172.21.15.39:0/964088739 shutdown_connections 2024-03-18T16:13:42.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.425+0000 7f1cee800640 1 --2- 172.21.15.39:0/964088739 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1ce8074240 0x7f1ce80746c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:42.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.425+0000 7f1cee800640 1 --2- 172.21.15.39:0/964088739 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1ce810dc90 0x7f1ce8110080 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:42.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.425+0000 7f1cee800640 1 --2- 172.21.15.39:0/964088739 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1ce8075820 0x7f1ce8073c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:42.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.425+0000 7f1cee800640 1 -- 172.21.15.39:0/964088739 >> 172.21.15.39:0/964088739 conn(0x7f1ce80fcc50 msgr2=0x7f1ce80ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:42.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.426+0000 7f1cee800640 1 -- 172.21.15.39:0/964088739 shutdown_connections 2024-03-18T16:13:42.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.426+0000 7f1cee800640 1 -- 172.21.15.39:0/964088739 wait complete. 2024-03-18T16:13:42.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.428+0000 7f1cee800640 1 Processor -- start 2024-03-18T16:13:42.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.428+0000 7f1cee800640 1 -- start start 2024-03-18T16:13:42.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.429+0000 7f1cee800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1ce8074240 0x7f1ce819df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:42.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.429+0000 7f1cee800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1ce8075820 0x7f1ce819e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:42.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.429+0000 7f1cee800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1ce810dc90 0x7f1ce81a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:42.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.429+0000 7f1cee800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f1ce8113960 con 0x7f1ce8075820 2024-03-18T16:13:42.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.429+0000 7f1cee800640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f1ce81137e0 con 0x7f1ce8074240 2024-03-18T16:13:42.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.429+0000 7f1cee800640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f1ce8113ae0 con 0x7f1ce810dc90 2024-03-18T16:13:42.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.429+0000 7f1cede00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1ce810dc90 0x7f1ce81a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:42.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.430+0000 7f1cede00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1ce810dc90 0x7f1ce81a5500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:42552/0 (socket says 172.21.15.39:42552) 2024-03-18T16:13:42.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.430+0000 7f1ced400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1ce8074240 0x7f1ce819df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:42.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.430+0000 7f1cede00640 1 -- 172.21.15.39:0/3652107501 learned_addr learned my addr 172.21.15.39:0/3652107501 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:13:42.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.430+0000 7f1cede00640 1 -- 172.21.15.39:0/3652107501 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1ce8074240 msgr2=0x7f1ce819df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:42.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.430+0000 7f1cede00640 1 --2- 172.21.15.39:0/3652107501 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1ce8074240 0x7f1ce819df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:42.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.430+0000 7f1cede00640 1 -- 172.21.15.39:0/3652107501 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1ce8075820 msgr2=0x7f1ce819e490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T16:13:42.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.430+0000 7f1cede00640 1 --2- 172.21.15.39:0/3652107501 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1ce8075820 0x7f1ce819e490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:42.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.430+0000 7f1cede00640 1 -- 172.21.15.39:0/3652107501 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ce81a5b70 con 0x7f1ce810dc90 2024-03-18T16:13:42.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.431+0000 7f1cede00640 1 --2- 172.21.15.39:0/3652107501 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1ce810dc90 0x7f1ce81a5500 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f1cd4034a10 tx=0x7f1cd40323b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:42.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.432+0000 7f1ce6000640 1 -- 172.21.15.39:0/3652107501 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cd4041020 con 0x7f1ce810dc90 2024-03-18T16:13:42.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.432+0000 7f1ce6000640 1 -- 172.21.15.39:0/3652107501 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1cd402c9f0 con 0x7f1ce810dc90 2024-03-18T16:13:42.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.432+0000 7f1ce6000640 1 -- 172.21.15.39:0/3652107501 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cd4039040 con 0x7f1ce810dc90 2024-03-18T16:13:42.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.432+0000 7f1cee800640 1 -- 172.21.15.39:0/3652107501 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ce81a5e00 con 0x7f1ce810dc90 2024-03-18T16:13:42.436 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.434+0000 7f1cee800640 1 -- 172.21.15.39:0/3652107501 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ce81a6210 con 0x7f1ce810dc90 2024-03-18T16:13:42.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.444+0000 7f1ce6000640 1 -- 172.21.15.39:0/3652107501 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f1cd4044020 con 0x7f1ce810dc90 2024-03-18T16:13:42.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.449+0000 7f1ce6000640 1 --2- 172.21.15.39:0/3652107501 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1cd00a28c0 0x7f1cd00a4d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:42.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.450+0000 7f1cf2b6d740 1 Processor -- start 2024-03-18T16:13:42.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.450+0000 7f1ce6000640 1 -- 172.21.15.39:0/3652107501 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f1cd40f0020 con 0x7f1ce810dc90 2024-03-18T16:13:42.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.450+0000 7f1cf2b6d740 1 -- start start 2024-03-18T16:13:42.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.451+0000 7f1cf2b6d740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e06fe019d0 0x55e06fe01df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:42.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.451+0000 7f1cf2b6d740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e06fe023c0 0x55e06fe06ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:42.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.451+0000 7f1cf2b6d740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e06fe07530 0x55e06fe099c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:42.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.451+0000 7f1cf2b6d740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55e06fd53f80 con 0x55e06fe07530 2024-03-18T16:13:42.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.451+0000 7f1cf2b6d740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55e06fcfcad0 con 0x55e06fe019d0 2024-03-18T16:13:42.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.451+0000 7f1cf2b6d740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55e06fd50020 con 0x55e06fe023c0 2024-03-18T16:13:42.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.451+0000 7f1ceca00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e06fe023c0 0x55e06fe06ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:42.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.451+0000 7f1ceca00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e06fe023c0 0x55e06fe06ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:42568/0 (socket says 172.21.15.39:42568) 2024-03-18T16:13:42.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.451+0000 7f1ceca00640 1 -- 172.21.15.39:0/560386033 learned_addr learned my addr 172.21.15.39:0/560386033 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:13:42.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.452+0000 7f1cede00640 1 --2- 172.21.15.39:0/560386033 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e06fe07530 0x55e06fe099c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:42.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.452+0000 7f1ceca00640 1 -- 172.21.15.39:0/560386033 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e06fe019d0 msgr2=0x55e06fe01df0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:13:42.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.452+0000 7f1ceca00640 1 --2- 172.21.15.39:0/560386033 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e06fe019d0 0x55e06fe01df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:42.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.452+0000 7f1ceca00640 1 -- 172.21.15.39:0/560386033 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e06fe07530 msgr2=0x55e06fe099c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:42.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.452+0000 7f1ceca00640 1 --2- 172.21.15.39:0/560386033 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e06fe07530 0x55e06fe099c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:42.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.452+0000 7f1ceca00640 1 -- 172.21.15.39:0/560386033 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e06fd146f0 con 0x55e06fe023c0 2024-03-18T16:13:42.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.452+0000 7f1ced400640 1 --2- 172.21.15.39:0/3652107501 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1cd00a28c0 0x7f1cd00a4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:42.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.452+0000 7f1ceca00640 1 --2- 172.21.15.39:0/560386033 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e06fe023c0 0x55e06fe06ec0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f1cdc009aa0 tx=0x7f1cdc009f60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c1df31b7c0f89ce5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:42.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.453+0000 7f1cda600640 1 -- 172.21.15.39:0/560386033 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cdc00ece0 con 0x55e06fe023c0 2024-03-18T16:13:42.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.453+0000 7f1cda600640 1 -- 172.21.15.39:0/560386033 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1cdc00ee80 con 0x55e06fe023c0 2024-03-18T16:13:42.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.453+0000 7f1cda600640 1 -- 172.21.15.39:0/560386033 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cdc010690 con 0x55e06fe023c0 2024-03-18T16:13:42.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.454+0000 7f1cf2b6d740 1 -- 172.21.15.39:0/560386033 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e06fe023c0 msgr2=0x55e06fe06ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:42.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.454+0000 7f1cf2b6d740 1 --2- 172.21.15.39:0/560386033 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e06fe023c0 0x55e06fe06ec0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f1cdc009aa0 tx=0x7f1cdc009f60 comp rx=0 tx=0).stop 2024-03-18T16:13:42.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.455+0000 7f1ced400640 1 --2- 172.21.15.39:0/3652107501 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1cd00a28c0 0x7f1cd00a4d80 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x55e06fe02c20 tx=0x7f1ce0009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:42.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.455+0000 7f1cf2b6d740 1 -- 172.21.15.39:0/560386033 shutdown_connections 2024-03-18T16:13:42.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.455+0000 7f1cf2b6d740 1 --2- 172.21.15.39:0/560386033 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e06fe07530 0x55e06fe099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:42.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.455+0000 7f1cf2b6d740 1 --2- 172.21.15.39:0/560386033 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e06fe023c0 0x55e06fe06ec0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:42.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.455+0000 7f1cf2b6d740 1 --2- 172.21.15.39:0/560386033 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e06fe019d0 0x55e06fe01df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:42.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.455+0000 7f1cf2b6d740 1 -- 172.21.15.39:0/560386033 >> 172.21.15.39:0/560386033 conn(0x55e06fdb8930 msgr2=0x55e06fdec330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:42.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.455+0000 7f1cf2b6d740 1 -- 172.21.15.39:0/560386033 shutdown_connections 2024-03-18T16:13:42.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.456+0000 7f1cf2b6d740 1 -- 172.21.15.39:0/560386033 wait complete. 2024-03-18T16:13:42.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.456+0000 7f1cf2b6d740 1 -- start start 2024-03-18T16:13:42.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.457+0000 7f1cf2b6d740 1 Processor -- start 2024-03-18T16:13:42.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.457+0000 7f1cf2b6d740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e06fe019d0 0x55e06fe0bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:42.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.457+0000 7f1cf2b6d740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e06fe023c0 0x55e06fe0c210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:42.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.457+0000 7f1cf2b6d740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e06fe07530 0x55e06fe13250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:42.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.458+0000 7f1ceca00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e06fe023c0 0x55e06fe0c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:42.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.458+0000 7f1ceca00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e06fe023c0 0x55e06fe0c210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:42584/0 (socket says 172.21.15.39:42584) 2024-03-18T16:13:42.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.458+0000 7f1cf2b6d740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55e06fe0a4e0 con 0x55e06fe019d0 2024-03-18T16:13:42.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.458+0000 7f1cf2b6d740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55e06fd0ff30 con 0x55e06fe07530 2024-03-18T16:13:42.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.458+0000 7f1cf2b6d740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55e06fe0a660 con 0x55e06fe023c0 2024-03-18T16:13:42.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.458+0000 7f1cede00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e06fe07530 0x55e06fe13250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:42.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.458+0000 7f1cede00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e06fe07530 0x55e06fe13250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:60236/0 (socket says 172.21.15.39:60236) 2024-03-18T16:13:42.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.458+0000 7f1cede00640 1 -- 172.21.15.39:0/3237011376 learned_addr learned my addr 172.21.15.39:0/3237011376 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:13:42.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.459+0000 7f1ced400640 1 --2- 172.21.15.39:0/3237011376 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e06fe019d0 0x55e06fe0bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:42.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.458+0000 7f1cf2b6d740 10 client.0 _ll_register_callbacks cb 0x55e06fcfdbc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:13:42.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.459+0000 7f1ceca00640 1 -- 172.21.15.39:0/3237011376 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e06fe07530 msgr2=0x55e06fe13250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:42.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.459+0000 7f1ceca00640 1 --2- 172.21.15.39:0/3237011376 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e06fe07530 0x55e06fe13250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:42.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.460+0000 7f1ceca00640 1 -- 172.21.15.39:0/3237011376 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e06fe019d0 msgr2=0x55e06fe0bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:42.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.460+0000 7f1ceca00640 1 --2- 172.21.15.39:0/3237011376 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e06fe019d0 0x55e06fe0bcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:42.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.460+0000 7f1ceca00640 1 -- 172.21.15.39:0/3237011376 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e06fb307e0 con 0x55e06fe023c0 2024-03-18T16:13:42.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.460+0000 7f1ceca00640 1 --2- 172.21.15.39:0/3237011376 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e06fe023c0 0x55e06fe0c210 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x55e06fe027c0 tx=0x7f1cdc0038a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=486e5761669d7794 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:42.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.460+0000 7f1ccea00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:13:42.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.461+0000 7f1ccea00640 1 -- 172.21.15.39:0/3237011376 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cdc01d040 con 0x55e06fe023c0 2024-03-18T16:13:42.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.462+0000 7f1ccea00640 1 -- 172.21.15.39:0/3237011376 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f1cdc005230 con 0x55e06fe023c0 2024-03-18T16:13:42.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.462+0000 7f1cf2b6d740 1 -- 172.21.15.39:0/3237011376 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55e06fd614c0 con 0x55e06fe023c0 2024-03-18T16:13:42.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.463+0000 7f1ccea00640 1 -- 172.21.15.39:0/3237011376 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f1cdc005470 con 0x55e06fe023c0 2024-03-18T16:13:42.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.463+0000 7f1cf2b6d740 10 client.26371 fetch_fsmap learned FSMap version 36 2024-03-18T16:13:42.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.463+0000 7f1cf2b6d740 1 -- 172.21.15.39:0/3237011376 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55e06fb90f70 con 0x55e06fe023c0 2024-03-18T16:13:42.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.464+0000 7f1ccea00640 1 -- 172.21.15.39:0/3237011376 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55e06fd614c0 con 0x55e06fe023c0 2024-03-18T16:13:42.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.465+0000 7f1ccea00640 1 -- 172.21.15.39:0/3237011376 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 36) ==== 8406+0+0 (secure 0 0 0) 0x7f1cdc004b30 con 0x55e06fe023c0 2024-03-18T16:13:42.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.465+0000 7f1cf2b6d740 10 client.26371 fetch_fsmap finished waiting for FSMap version 36 2024-03-18T16:13:42.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.465+0000 7f1cf2b6d740 10 client.26371 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:13:42.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.465+0000 7f1cf2b6d740 20 client.26371 populate_metadata read hostname 'smithi039' 2024-03-18T16:13:42.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.465+0000 7f1cf2b6d740 1 --2- 172.21.15.39:0/3237011376 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55e06fe177d0 0x55e06fe19c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:42.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.465+0000 7f1cf2b6d740 4 client.26371 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:13:42.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.465+0000 7f1cf2b6d740 1 -- 172.21.15.39:0/3237011376 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e06fd5b810 con 0x55e06fe177d0 2024-03-18T16:13:42.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.466+0000 7f1ced400640 1 --2- 172.21.15.39:0/3237011376 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55e06fe177d0 0x55e06fe19c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:42.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.471+0000 7f1ccea00640 10 client.26371 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:13:42.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.471+0000 7f1ced400640 1 --2- 172.21.15.39:0/3237011376 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55e06fe177d0 0x55e06fe19c10 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd2a9e997d7d3419 server_cookie=cccf3c403fbf8d12 in_seq=0 out_seq=0 2024-03-18T16:13:42.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.474+0000 7f1ccea00640 1 -- 172.21.15.39:0/3237011376 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e06fd5b810 con 0x55e06fe177d0 2024-03-18T16:13:42.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.474+0000 7f1ccea00640 10 client.26371 handle_command_reply: tid=0 2024-03-18T16:13:42.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.474+0000 7f1cf2b6d740 1 client.26371 shutdown 2024-03-18T16:13:42.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.474+0000 7f1cf2b6d740 1 -- 172.21.15.39:0/3237011376 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e06fe023c0 msgr2=0x55e06fe0c210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:42.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.474+0000 7f1cf2b6d740 1 --2- 172.21.15.39:0/3237011376 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e06fe023c0 0x55e06fe0c210 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x55e06fe027c0 tx=0x7f1cdc0038a0 comp rx=0 tx=0).stop 2024-03-18T16:13:42.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.475+0000 7f1cf2b6d740 1 -- 172.21.15.39:0/3237011376 shutdown_connections 2024-03-18T16:13:42.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.475+0000 7f1cf2b6d740 1 --2- 172.21.15.39:0/3237011376 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55e06fe177d0 0x55e06fe19c10 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:42.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.475+0000 7f1cf2b6d740 1 --2- 172.21.15.39:0/3237011376 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e06fe07530 0x55e06fe13250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:42.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.475+0000 7f1cf2b6d740 1 --2- 172.21.15.39:0/3237011376 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e06fe023c0 0x55e06fe0c210 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:42.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.475+0000 7f1cf2b6d740 1 --2- 172.21.15.39:0/3237011376 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e06fe019d0 0x55e06fe0bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:42.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.475+0000 7f1cf2b6d740 1 -- 172.21.15.39:0/3237011376 >> 172.21.15.39:0/3237011376 conn(0x55e06fdb8930 msgr2=0x55e06fdeb570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:42.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.475+0000 7f1ccea00640 0 client.26371 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:13:42.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.475+0000 7f1cf2b6d740 1 -- 172.21.15.39:0/3237011376 shutdown_connections 2024-03-18T16:13:42.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.476+0000 7f1cf2b6d740 1 -- 172.21.15.39:0/3237011376 wait complete. 2024-03-18T16:13:42.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.476+0000 7f1cf2b6d740 20 client.26371 trim_cache size 0 max 16384 2024-03-18T16:13:42.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.528+0000 7f1cf2b6d740 1 Processor -- start 2024-03-18T16:13:42.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.529+0000 7f1cf2b6d740 1 -- start start 2024-03-18T16:13:42.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.529+0000 7f1cf2b6d740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e06fdb8930 0x55e06fe036e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:42.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.529+0000 7f1cf2b6d740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e06fe019d0 0x55e06fe03c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:42.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.529+0000 7f1cf2b6d740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e06fe023c0 0x55e06fe13210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:42.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.529+0000 7f1cf2b6d740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f1ce81a7f80 con 0x55e06fe023c0 2024-03-18T16:13:42.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.529+0000 7f1cf2b6d740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f1ce81a9c40 con 0x55e06fe019d0 2024-03-18T16:13:42.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.529+0000 7f1cf2b6d740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f1ce81a8650 con 0x55e06fdb8930 2024-03-18T16:13:42.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.530+0000 7f1cede00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e06fe023c0 0x55e06fe13210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:42.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.530+0000 7f1ced400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e06fdb8930 0x55e06fe036e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:42.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.530+0000 7f1ceca00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e06fe019d0 0x55e06fe03c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:42.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.530+0000 7f1ceca00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e06fe019d0 0x55e06fe03c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:60240/0 (socket says 172.21.15.39:60240) 2024-03-18T16:13:42.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.530+0000 7f1ceca00640 1 -- 172.21.15.39:0/810434592 learned_addr learned my addr 172.21.15.39:0/810434592 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:13:42.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.531+0000 7f1ced400640 1 -- 172.21.15.39:0/810434592 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e06fe019d0 msgr2=0x55e06fe03c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:42.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.531+0000 7f1ced400640 1 --2- 172.21.15.39:0/810434592 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e06fe019d0 0x55e06fe03c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:42.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.531+0000 7f1ced400640 1 -- 172.21.15.39:0/810434592 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e06fe023c0 msgr2=0x55e06fe13210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:42.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.531+0000 7f1ced400640 1 --2- 172.21.15.39:0/810434592 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e06fe023c0 0x55e06fe13210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:42.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.531+0000 7f1ced400640 1 -- 172.21.15.39:0/810434592 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e06fa28170 con 0x55e06fdb8930 2024-03-18T16:13:42.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.532+0000 7f1ced400640 1 --2- 172.21.15.39:0/810434592 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e06fdb8930 0x55e06fe036e0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x55e06fe1a420 tx=0x7f1ce0014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=af7a7e0ce7ebd7f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:42.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.532+0000 7f1cd9c00640 1 -- 172.21.15.39:0/810434592 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ce002f070 con 0x55e06fdb8930 2024-03-18T16:13:42.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.533+0000 7f1cd9c00640 1 -- 172.21.15.39:0/810434592 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ce0004620 con 0x55e06fdb8930 2024-03-18T16:13:42.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.533+0000 7f1cd9c00640 1 -- 172.21.15.39:0/810434592 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ce001d960 con 0x55e06fdb8930 2024-03-18T16:13:42.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.534+0000 7f1cf2b6d740 1 -- 172.21.15.39:0/810434592 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e06fdb8930 msgr2=0x55e06fe036e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:42.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.534+0000 7f1cf2b6d740 1 --2- 172.21.15.39:0/810434592 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e06fdb8930 0x55e06fe036e0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x55e06fe1a420 tx=0x7f1ce0014000 comp rx=0 tx=0).stop 2024-03-18T16:13:42.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.534+0000 7f1cf2b6d740 1 -- 172.21.15.39:0/810434592 shutdown_connections 2024-03-18T16:13:42.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.534+0000 7f1cf2b6d740 1 --2- 172.21.15.39:0/810434592 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e06fe019d0 0x55e06fe03c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:42.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.534+0000 7f1cf2b6d740 1 --2- 172.21.15.39:0/810434592 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e06fe023c0 0x55e06fe13210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:42.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.534+0000 7f1cf2b6d740 1 --2- 172.21.15.39:0/810434592 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e06fdb8930 0x55e06fe036e0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:42.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.534+0000 7f1cf2b6d740 1 -- 172.21.15.39:0/810434592 >> 172.21.15.39:0/810434592 conn(0x55e06fe07070 msgr2=0x55e06fe0bcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:42.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.534+0000 7f1cf2b6d740 1 -- 172.21.15.39:0/810434592 shutdown_connections 2024-03-18T16:13:42.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.534+0000 7f1cf2b6d740 1 -- 172.21.15.39:0/810434592 wait complete. 2024-03-18T16:13:42.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.536+0000 7f1cf2b6d740 1 -- start start 2024-03-18T16:13:42.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.537+0000 7f1cf2b6d740 1 Processor -- start 2024-03-18T16:13:42.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.537+0000 7f1cf2b6d740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e06fdb8930 0x55e06fe103c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:42.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.537+0000 7f1cf2b6d740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e06fe019d0 0x55e06fe10900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:42.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.537+0000 7f1cf2b6d740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e06fe023c0 0x55e06fe11910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:42.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.537+0000 7f1cf2b6d740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f1ce81a94b0 con 0x55e06fdb8930 2024-03-18T16:13:42.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.537+0000 7f1cf2b6d740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f1ce81a8d20 con 0x55e06fe019d0 2024-03-18T16:13:42.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.537+0000 7f1cf2b6d740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f1ce810d7a0 con 0x55e06fe023c0 2024-03-18T16:13:42.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.537+0000 7f1cf2b6d740 10 client.0 _ll_register_callbacks cb 0x7f1cdc009920 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:13:42.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.538+0000 7f1ced400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e06fdb8930 0x55e06fe103c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:42.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.538+0000 7f1ceca00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e06fe019d0 0x55e06fe10900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:42.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.538+0000 7f1ceca00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e06fe019d0 0x55e06fe10900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:60248/0 (socket says 172.21.15.39:60248) 2024-03-18T16:13:42.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.538+0000 7f1ceca00640 1 -- 172.21.15.39:0/1587729876 learned_addr learned my addr 172.21.15.39:0/1587729876 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:13:42.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.538+0000 7f1cede00640 1 --2- 172.21.15.39:0/1587729876 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e06fe023c0 0x55e06fe11910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:42.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.539+0000 7f1cede00640 1 -- 172.21.15.39:0/1587729876 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e06fe019d0 msgr2=0x55e06fe10900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:42.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.539+0000 7f1cede00640 1 --2- 172.21.15.39:0/1587729876 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e06fe019d0 0x55e06fe10900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:42.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.539+0000 7f1cede00640 1 -- 172.21.15.39:0/1587729876 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e06fdb8930 msgr2=0x55e06fe103c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:42.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.540+0000 7f1cede00640 1 --2- 172.21.15.39:0/1587729876 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e06fdb8930 0x55e06fe103c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:42.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.540+0000 7f1cede00640 1 -- 172.21.15.39:0/1587729876 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e06fe11f00 con 0x55e06fe023c0 2024-03-18T16:13:42.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.540+0000 7f1ced400640 1 --2- 172.21.15.39:0/1587729876 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e06fdb8930 0x55e06fe103c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:13:42.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.541+0000 7f1cede00640 1 --2- 172.21.15.39:0/1587729876 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e06fe023c0 0x55e06fe11910 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f1cd40b8a90 tx=0x7f1cd402bb70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=65859156b1fcf6e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:42.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.541+0000 7f1ccea00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:13:42.544 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.541+0000 7f1ccea00640 1 -- 172.21.15.39:0/1587729876 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cd40da060 con 0x55e06fe023c0 2024-03-18T16:13:42.544 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.541+0000 7f1cf2b6d740 1 -- 172.21.15.39:0/1587729876 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ce0004620 con 0x55e06fe023c0 2024-03-18T16:13:42.544 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.542+0000 7f1ccea00640 1 -- 172.21.15.39:0/1587729876 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f1cd402cba0 con 0x55e06fe023c0 2024-03-18T16:13:42.545 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.543+0000 7f1ccea00640 1 -- 172.21.15.39:0/1587729876 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f1cd40cc050 con 0x55e06fe023c0 2024-03-18T16:13:42.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.544+0000 7f1cf2b6d740 10 client.26377 fetch_fsmap learned FSMap version 36 2024-03-18T16:13:42.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.544+0000 7f1cf2b6d740 1 -- 172.21.15.39:0/1587729876 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55e06fe13360 con 0x55e06fe023c0 2024-03-18T16:13:42.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.545+0000 7f1ccea00640 1 -- 172.21.15.39:0/1587729876 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ce0004620 con 0x55e06fe023c0 2024-03-18T16:13:42.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.546+0000 7f1ccea00640 1 -- 172.21.15.39:0/1587729876 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 36) ==== 8406+0+0 (secure 0 0 0) 0x7f1cd403c030 con 0x55e06fe023c0 2024-03-18T16:13:42.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.546+0000 7f1cf2b6d740 10 client.26377 fetch_fsmap finished waiting for FSMap version 36 2024-03-18T16:13:42.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.546+0000 7f1cf2b6d740 10 client.26377 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:13:42.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.546+0000 7f1cf2b6d740 20 client.26377 populate_metadata read hostname 'smithi039' 2024-03-18T16:13:42.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.546+0000 7f1cf2b6d740 1 --2- 172.21.15.39:0/1587729876 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55e06fe07530 0x55e06fe80060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:42.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.546+0000 7f1cf2b6d740 4 client.26377 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-18T16:13:42.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.546+0000 7f1cf2b6d740 1 -- 172.21.15.39:0/1587729876 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55e06fcd9aa0 con 0x55e06fe07530 2024-03-18T16:13:42.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.547+0000 7f1ced400640 1 --2- 172.21.15.39:0/1587729876 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55e06fe07530 0x55e06fe80060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:42.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.549+0000 7f1ced400640 1 --2- 172.21.15.39:0/1587729876 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55e06fe07530 0x55e06fe80060 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6f10670fff7cb4cb server_cookie=683cc488ba245c1f in_seq=0 out_seq=0 2024-03-18T16:13:42.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.549+0000 7f1ccea00640 10 client.26377 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:13:42.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.553+0000 7f1ccea00640 1 -- 172.21.15.39:0/1587729876 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55e06fcd9aa0 con 0x55e06fe07530 2024-03-18T16:13:42.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.553+0000 7f1ccea00640 10 client.26377 handle_command_reply: tid=0 2024-03-18T16:13:42.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.554+0000 7f1cf2b6d740 1 client.26377 shutdown 2024-03-18T16:13:42.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.554+0000 7f1cf2b6d740 1 -- 172.21.15.39:0/1587729876 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e06fe023c0 msgr2=0x55e06fe11910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:42.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.554+0000 7f1cf2b6d740 1 --2- 172.21.15.39:0/1587729876 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e06fe023c0 0x55e06fe11910 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f1cd40b8a90 tx=0x7f1cd402bb70 comp rx=0 tx=0).stop 2024-03-18T16:13:42.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.556+0000 7f1cf2b6d740 1 -- 172.21.15.39:0/1587729876 shutdown_connections 2024-03-18T16:13:42.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.556+0000 7f1cf2b6d740 1 --2- 172.21.15.39:0/1587729876 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55e06fe07530 0x55e06fe80060 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:42.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.556+0000 7f1cf2b6d740 1 --2- 172.21.15.39:0/1587729876 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e06fe019d0 0x55e06fe10900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:42.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.556+0000 7f1cf2b6d740 1 --2- 172.21.15.39:0/1587729876 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e06fe023c0 0x55e06fe11910 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:42.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.556+0000 7f1cf2b6d740 1 --2- 172.21.15.39:0/1587729876 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e06fdb8930 0x55e06fe103c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:42.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.556+0000 7f1ccea00640 0 client.26377 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:13:42.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.556+0000 7f1cf2b6d740 1 -- 172.21.15.39:0/1587729876 >> 172.21.15.39:0/1587729876 conn(0x55e06fe07070 msgr2=0x55e06fe0bcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:42.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.559+0000 7f1cf2b6d740 1 -- 172.21.15.39:0/1587729876 shutdown_connections 2024-03-18T16:13:42.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.559+0000 7f1cf2b6d740 1 -- 172.21.15.39:0/1587729876 wait complete. 2024-03-18T16:13:42.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.559+0000 7f1cf2b6d740 20 client.26377 trim_cache size 0 max 16384 2024-03-18T16:13:42.562 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-03-18T16:13:42.562 INFO:teuthology.orchestra.run.smithi039.stdout: "return_code": 0, 2024-03-18T16:13:42.562 INFO:teuthology.orchestra.run.smithi039.stdout: "scrub_tag": "bdbde0aa-7919-40dc-97e4-d7fb17eeb280", 2024-03-18T16:13:42.562 INFO:teuthology.orchestra.run.smithi039.stdout: "mode": "asynchronous" 2024-03-18T16:13:42.562 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-03-18T16:13:42.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.560+0000 7f1cee800640 1 -- 172.21.15.39:0/3652107501 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1cd00a28c0 msgr2=0x7f1cd00a4d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:42.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.560+0000 7f1cee800640 1 --2- 172.21.15.39:0/3652107501 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1cd00a28c0 0x7f1cd00a4d80 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x55e06fe02c20 tx=0x7f1ce0009500 comp rx=0 tx=0).stop 2024-03-18T16:13:42.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.560+0000 7f1cee800640 1 -- 172.21.15.39:0/3652107501 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1ce810dc90 msgr2=0x7f1ce81a5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:42.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.560+0000 7f1cee800640 1 --2- 172.21.15.39:0/3652107501 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1ce810dc90 0x7f1ce81a5500 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f1cd4034a10 tx=0x7f1cd40323b0 comp rx=0 tx=0).stop 2024-03-18T16:13:42.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.562+0000 7f1cee800640 1 -- 172.21.15.39:0/3652107501 shutdown_connections 2024-03-18T16:13:42.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.562+0000 7f1cee800640 1 --2- 172.21.15.39:0/3652107501 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1cd00a28c0 0x7f1cd00a4d80 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:42.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.562+0000 7f1cee800640 1 --2- 172.21.15.39:0/3652107501 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1ce810dc90 0x7f1ce81a5500 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:42.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.562+0000 7f1cee800640 1 --2- 172.21.15.39:0/3652107501 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1ce8075820 0x7f1ce819e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:42.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.562+0000 7f1cee800640 1 --2- 172.21.15.39:0/3652107501 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1ce8074240 0x7f1ce819df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:42.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.562+0000 7f1cee800640 1 -- 172.21.15.39:0/3652107501 >> 172.21.15.39:0/3652107501 conn(0x7f1ce80fcc50 msgr2=0x7f1ce8101030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:42.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.565+0000 7f1cee800640 1 -- 172.21.15.39:0/3652107501 shutdown_connections 2024-03-18T16:13:42.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:42.565+0000 7f1cee800640 1 -- 172.21.15.39:0/3652107501 wait complete. 2024-03-18T16:13:42.599 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T16:13:42.599 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T16:13:43.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:13:43 smithi196 ceph-mon[36307]: pgmap v531: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 55 MiB/s wr, 7.12k op/s 2024-03-18T16:13:43.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:13:43 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:13:43.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:13:43 smithi194 ceph-mon[36200]: pgmap v531: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 55 MiB/s wr, 7.12k op/s 2024-03-18T16:13:43.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:13:43 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:13:43.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:13:43 smithi039 ceph-mon[31721]: pgmap v531: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 55 MiB/s wr, 7.12k op/s 2024-03-18T16:13:43.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:13:43 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:13:44.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.193+0000 7f381f000640 1 -- 172.21.15.39:0/578508098 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f381810b5d0 msgr2=0x7f3818114470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:44.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.193+0000 7f381f000640 1 --2- 172.21.15.39:0/578508098 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f381810b5d0 0x7f3818114470 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f3810009ca0 tx=0x7f381002c120 comp rx=0 tx=0).stop 2024-03-18T16:13:44.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.195+0000 7f381f000640 1 -- 172.21.15.39:0/578508098 shutdown_connections 2024-03-18T16:13:44.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.195+0000 7f381f000640 1 --2- 172.21.15.39:0/578508098 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f381810b150 0x7f3818111f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:44.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.195+0000 7f381f000640 1 --2- 172.21.15.39:0/578508098 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f381810b5d0 0x7f3818114470 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:44.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.195+0000 7f381f000640 1 --2- 172.21.15.39:0/578508098 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f38181087e0 0x7f381810ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:44.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.195+0000 7f381f000640 1 -- 172.21.15.39:0/578508098 >> 172.21.15.39:0/578508098 conn(0x7f381806e9e0 msgr2=0x7f3818070e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:44.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.195+0000 7f381f000640 1 -- 172.21.15.39:0/578508098 shutdown_connections 2024-03-18T16:13:44.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.196+0000 7f381f000640 1 -- 172.21.15.39:0/578508098 wait complete. 2024-03-18T16:13:44.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.197+0000 7f381f000640 1 Processor -- start 2024-03-18T16:13:44.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.197+0000 7f381f000640 1 -- start start 2024-03-18T16:13:44.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.198+0000 7f381f000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f38181087e0 0x7f38181a24c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:44.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.198+0000 7f381f000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f381810b150 0x7f38181a2a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:44.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.198+0000 7f381f000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f381810b5d0 0x7f38181a9a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:44.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.198+0000 7f381f000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f3818117ee0 con 0x7f381810b150 2024-03-18T16:13:44.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.198+0000 7f381f000640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f3818117d60 con 0x7f381810b5d0 2024-03-18T16:13:44.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.198+0000 7f381f000640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f3818118060 con 0x7f38181087e0 2024-03-18T16:13:44.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.198+0000 7f381d000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f381810b5d0 0x7f38181a9a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:44.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.198+0000 7f381d000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f381810b5d0 0x7f38181a9a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:60254/0 (socket says 172.21.15.39:60254) 2024-03-18T16:13:44.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.198+0000 7f381d000640 1 -- 172.21.15.39:0/2632111956 learned_addr learned my addr 172.21.15.39:0/2632111956 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:13:44.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.198+0000 7f3817400640 1 --2- 172.21.15.39:0/2632111956 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f381810b150 0x7f38181a2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:44.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.198+0000 7f381d000640 1 -- 172.21.15.39:0/2632111956 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f38181087e0 msgr2=0x7f38181a24c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T16:13:44.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.198+0000 7f381d000640 1 --2- 172.21.15.39:0/2632111956 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f38181087e0 0x7f38181a24c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:44.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.198+0000 7f381d000640 1 -- 172.21.15.39:0/2632111956 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f381810b150 msgr2=0x7f38181a2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:44.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.198+0000 7f381d000640 1 --2- 172.21.15.39:0/2632111956 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f381810b150 0x7f38181a2a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:44.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.198+0000 7f381d000640 1 -- 172.21.15.39:0/2632111956 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38181aa0e0 con 0x7f381810b5d0 2024-03-18T16:13:44.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.199+0000 7f381d000640 1 --2- 172.21.15.39:0/2632111956 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f381810b5d0 0x7f38181a9a70 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f3810009bf0 tx=0x7f3810035d80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:44.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.199+0000 7f3814c00640 1 -- 172.21.15.39:0/2632111956 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3810039040 con 0x7f381810b5d0 2024-03-18T16:13:44.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.199+0000 7f3814c00640 1 -- 172.21.15.39:0/2632111956 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3810044070 con 0x7f381810b5d0 2024-03-18T16:13:44.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.199+0000 7f3814c00640 1 -- 172.21.15.39:0/2632111956 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38100046b0 con 0x7f381810b5d0 2024-03-18T16:13:44.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.199+0000 7f381f000640 1 -- 172.21.15.39:0/2632111956 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38181aa370 con 0x7f381810b5d0 2024-03-18T16:13:44.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.200+0000 7f381f000640 1 -- 172.21.15.39:0/2632111956 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38181aa930 con 0x7f381810b5d0 2024-03-18T16:13:44.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.213+0000 7f3821e2c740 1 Processor -- start 2024-03-18T16:13:44.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.214+0000 7f3821e2c740 1 -- start start 2024-03-18T16:13:44.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.214+0000 7f3821e2c740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559ed3922810 0x559ed3922c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:44.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.215+0000 7f3821e2c740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559ed3923200 0x559ed3927d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:44.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.215+0000 7f3821e2c740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559ed3928370 0x559ed392a800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:44.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.215+0000 7f3821e2c740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x559ed38871f0 con 0x559ed3928370 2024-03-18T16:13:44.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.215+0000 7f3821e2c740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x559ed382fa80 con 0x559ed3923200 2024-03-18T16:13:44.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.215+0000 7f3821e2c740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x559ed3883290 con 0x559ed3922810 2024-03-18T16:13:44.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.215+0000 7f3817e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559ed3923200 0x559ed3927d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:44.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.215+0000 7f3817e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559ed3923200 0x559ed3927d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:60260/0 (socket says 172.21.15.39:60260) 2024-03-18T16:13:44.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.216+0000 7f3817e00640 1 -- 172.21.15.39:0/3174875284 learned_addr learned my addr 172.21.15.39:0/3174875284 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:13:44.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.216+0000 7f3817e00640 1 -- 172.21.15.39:0/3174875284 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559ed3922810 msgr2=0x559ed3922c30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T16:13:44.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.216+0000 7f3817e00640 1 --2- 172.21.15.39:0/3174875284 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559ed3922810 0x559ed3922c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:44.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.216+0000 7f3817e00640 1 -- 172.21.15.39:0/3174875284 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559ed3928370 msgr2=0x559ed392a800 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:13:44.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.217+0000 7f3817e00640 1 --2- 172.21.15.39:0/3174875284 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559ed3928370 0x559ed392a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:44.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.217+0000 7f3817e00640 1 -- 172.21.15.39:0/3174875284 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559ed36657d0 con 0x559ed3923200 2024-03-18T16:13:44.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.217+0000 7f3817e00640 1 --2- 172.21.15.39:0/3174875284 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559ed3923200 0x559ed3927d00 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f380800aa10 tx=0x7f380800aed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=859b9d754090b769 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:44.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.218+0000 7f380d600640 1 -- 172.21.15.39:0/3174875284 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f380800cde0 con 0x559ed3923200 2024-03-18T16:13:44.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.219+0000 7f380d600640 1 -- 172.21.15.39:0/3174875284 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38080035f0 con 0x559ed3923200 2024-03-18T16:13:44.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.219+0000 7f380d600640 1 -- 172.21.15.39:0/3174875284 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f380800f6b0 con 0x559ed3923200 2024-03-18T16:13:44.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.220+0000 7f3821e2c740 1 -- 172.21.15.39:0/3174875284 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559ed3923200 msgr2=0x559ed3927d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:44.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.220+0000 7f3821e2c740 1 --2- 172.21.15.39:0/3174875284 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559ed3923200 0x559ed3927d00 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f380800aa10 tx=0x7f380800aed0 comp rx=0 tx=0).stop 2024-03-18T16:13:44.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.220+0000 7f3821e2c740 1 -- 172.21.15.39:0/3174875284 shutdown_connections 2024-03-18T16:13:44.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.220+0000 7f3821e2c740 1 --2- 172.21.15.39:0/3174875284 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559ed3923200 0x559ed3927d00 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:44.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.220+0000 7f3821e2c740 1 --2- 172.21.15.39:0/3174875284 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559ed3928370 0x559ed392a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:44.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.220+0000 7f3821e2c740 1 --2- 172.21.15.39:0/3174875284 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559ed3922810 0x559ed3922c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:44.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.220+0000 7f3821e2c740 1 -- 172.21.15.39:0/3174875284 >> 172.21.15.39:0/3174875284 conn(0x559ed391dd80 msgr2=0x559ed391e180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:44.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.221+0000 7f3814c00640 1 -- 172.21.15.39:0/2632111956 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f3810003970 con 0x7f381810b5d0 2024-03-18T16:13:44.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.221+0000 7f3821e2c740 1 -- 172.21.15.39:0/3174875284 shutdown_connections 2024-03-18T16:13:44.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.221+0000 7f3821e2c740 1 -- 172.21.15.39:0/3174875284 wait complete. 2024-03-18T16:13:44.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.221+0000 7f3821e2c740 1 -- start start 2024-03-18T16:13:44.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.222+0000 7f3821e2c740 1 Processor -- start 2024-03-18T16:13:44.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.223+0000 7f3821e2c740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559ed3922810 0x559ed392cde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:44.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.223+0000 7f3821e2c740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559ed3923200 0x559ed392d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:44.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.223+0000 7f3821e2c740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559ed3928370 0x559ed3934320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:44.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.223+0000 7f3821e2c740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x559ed392b320 con 0x559ed3923200 2024-03-18T16:13:44.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.223+0000 7f3821e2c740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x559ed3842ed0 con 0x559ed3928370 2024-03-18T16:13:44.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.223+0000 7f3821e2c740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x559ed392b4a0 con 0x559ed3922810 2024-03-18T16:13:44.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.223+0000 7f3821e2c740 10 client.0 _ll_register_callbacks cb 0x559ed387e220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:13:44.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.224+0000 7f3817400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559ed3922810 0x559ed392cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:44.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.224+0000 7f3817400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559ed3922810 0x559ed392cde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:42626/0 (socket says 172.21.15.39:42626) 2024-03-18T16:13:44.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.224+0000 7f3817400640 1 -- 172.21.15.39:0/3498361790 learned_addr learned my addr 172.21.15.39:0/3498361790 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:13:44.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.224+0000 7f3817400640 1 --2- 172.21.15.39:0/3498361790 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559ed3928370 0x559ed3934320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:44.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.224+0000 7f3817e00640 1 --2- 172.21.15.39:0/3498361790 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559ed3923200 0x559ed392d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:44.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.224+0000 7f3814c00640 1 --2- 172.21.15.39:0/2632111956 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f37fc0a2810 0x7f37fc0a4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:44.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.225+0000 7f3814c00640 1 -- 172.21.15.39:0/2632111956 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f38100f11b0 con 0x7f381810b5d0 2024-03-18T16:13:44.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.225+0000 7f3817400640 1 -- 172.21.15.39:0/3498361790 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559ed3928370 msgr2=0x559ed3934320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:44.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.225+0000 7f3817400640 1 --2- 172.21.15.39:0/3498361790 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559ed3928370 0x559ed3934320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:44.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.225+0000 7f3817400640 1 -- 172.21.15.39:0/3498361790 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559ed3923200 msgr2=0x559ed392d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:44.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.225+0000 7f3817400640 1 --2- 172.21.15.39:0/3498361790 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559ed3923200 0x559ed392d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:44.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.225+0000 7f3817400640 1 -- 172.21.15.39:0/3498361790 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559ed359b3f0 con 0x559ed3922810 2024-03-18T16:13:44.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.226+0000 7f3817400640 1 --2- 172.21.15.39:0/3498361790 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559ed3922810 0x559ed392cde0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f380400ee30 tx=0x7f3804006f60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f716bcb80f4aa01 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:44.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.226+0000 7f3802000640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:13:44.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.226+0000 7f3802000640 1 -- 172.21.15.39:0/3498361790 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f380400f040 con 0x559ed3922810 2024-03-18T16:13:44.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.226+0000 7f3802000640 1 -- 172.21.15.39:0/3498361790 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f3804013b10 con 0x559ed3922810 2024-03-18T16:13:44.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.226+0000 7f381d000640 1 --2- 172.21.15.39:0/2632111956 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f37fc0a2810 0x7f37fc0a4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:44.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.226+0000 7f3821e2c740 1 -- 172.21.15.39:0/3498361790 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f38080035f0 con 0x559ed3922810 2024-03-18T16:13:44.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.227+0000 7f3802000640 1 -- 172.21.15.39:0/3498361790 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f380400e7b0 con 0x559ed3922810 2024-03-18T16:13:44.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.227+0000 7f3821e2c740 10 client.26383 fetch_fsmap learned FSMap version 36 2024-03-18T16:13:44.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.227+0000 7f3821e2c740 1 -- 172.21.15.39:0/3498361790 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x559ed355b0f0 con 0x559ed3922810 2024-03-18T16:13:44.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.227+0000 7f381d000640 1 --2- 172.21.15.39:0/2632111956 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f37fc0a2810 0x7f37fc0a4cd0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f381002c660 tx=0x7f38100095c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:44.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.228+0000 7f3802000640 1 -- 172.21.15.39:0/3498361790 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38080035f0 con 0x559ed3922810 2024-03-18T16:13:44.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.229+0000 7f3802000640 1 -- 172.21.15.39:0/3498361790 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 36) ==== 8406+0+0 (secure 0 0 0) 0x7f3804020400 con 0x559ed3922810 2024-03-18T16:13:44.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.230+0000 7f3821e2c740 10 client.26383 fetch_fsmap finished waiting for FSMap version 36 2024-03-18T16:13:44.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.230+0000 7f3821e2c740 10 client.26383 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:13:44.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.230+0000 7f3821e2c740 20 client.26383 populate_metadata read hostname 'smithi039' 2024-03-18T16:13:44.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.230+0000 7f3821e2c740 1 --2- 172.21.15.39:0/3498361790 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x559ed3938bd0 0x559ed393aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:44.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.230+0000 7f3821e2c740 4 client.26383 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:13:44.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.230+0000 7f3821e2c740 1 -- 172.21.15.39:0/3498361790 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559ed389abe0 con 0x559ed3938bd0 2024-03-18T16:13:44.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.231+0000 7f3817e00640 1 --2- 172.21.15.39:0/3498361790 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x559ed3938bd0 0x559ed393aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:44.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.232+0000 7f3817e00640 1 --2- 172.21.15.39:0/3498361790 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x559ed3938bd0 0x559ed393aff0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=21bfd53eeb93457b server_cookie=ddb024d837c8e3f8 in_seq=0 out_seq=0 2024-03-18T16:13:44.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.232+0000 7f3802000640 10 client.26383 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:13:44.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.232+0000 7f3802000640 1 -- 172.21.15.39:0/3498361790 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559ed389abe0 con 0x559ed3938bd0 2024-03-18T16:13:44.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.233+0000 7f3802000640 10 client.26383 handle_command_reply: tid=0 2024-03-18T16:13:44.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.233+0000 7f3821e2c740 1 client.26383 shutdown 2024-03-18T16:13:44.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.233+0000 7f3821e2c740 1 -- 172.21.15.39:0/3498361790 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559ed3922810 msgr2=0x559ed392cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:44.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.233+0000 7f3821e2c740 1 --2- 172.21.15.39:0/3498361790 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559ed3922810 0x559ed392cde0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f380400ee30 tx=0x7f3804006f60 comp rx=0 tx=0).stop 2024-03-18T16:13:44.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.234+0000 7f3821e2c740 1 -- 172.21.15.39:0/3498361790 shutdown_connections 2024-03-18T16:13:44.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.234+0000 7f3821e2c740 1 --2- 172.21.15.39:0/3498361790 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x559ed3938bd0 0x559ed393aff0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:44.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.234+0000 7f3821e2c740 1 --2- 172.21.15.39:0/3498361790 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559ed3928370 0x559ed3934320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:44.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.234+0000 7f3821e2c740 1 --2- 172.21.15.39:0/3498361790 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559ed3923200 0x559ed392d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:44.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.234+0000 7f3821e2c740 1 --2- 172.21.15.39:0/3498361790 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559ed3922810 0x559ed392cde0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:44.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.234+0000 7f3802000640 0 client.26383 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:13:44.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.234+0000 7f3821e2c740 1 -- 172.21.15.39:0/3498361790 >> 172.21.15.39:0/3498361790 conn(0x559ed391dd80 msgr2=0x559ed391cec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:44.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.234+0000 7f3821e2c740 1 -- 172.21.15.39:0/3498361790 shutdown_connections 2024-03-18T16:13:44.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.235+0000 7f3821e2c740 1 -- 172.21.15.39:0/3498361790 wait complete. 2024-03-18T16:13:44.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.235+0000 7f3821e2c740 20 client.26383 trim_cache size 0 max 16384 2024-03-18T16:13:44.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.283+0000 7f3821e2c740 1 Processor -- start 2024-03-18T16:13:44.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.283+0000 7f3821e2c740 1 -- start start 2024-03-18T16:13:44.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.283+0000 7f3821e2c740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559ed3922810 0x559ed391cd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:44.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.283+0000 7f3821e2c740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559ed3923200 0x559ed39342e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:44.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.283+0000 7f3821e2c740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559ed3928370 0x559ed393f940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:44.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.283+0000 7f3821e2c740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f38181b4e90 con 0x559ed3922810 2024-03-18T16:13:44.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.283+0000 7f3821e2c740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f38181b6b50 con 0x559ed3928370 2024-03-18T16:13:44.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.283+0000 7f3821e2c740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f38181b5560 con 0x559ed3923200 2024-03-18T16:13:44.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.284+0000 7f3817e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559ed3923200 0x559ed39342e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:44.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.284+0000 7f3817e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559ed3923200 0x559ed39342e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:42632/0 (socket says 172.21.15.39:42632) 2024-03-18T16:13:44.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.284+0000 7f3817400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559ed3928370 0x559ed393f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:44.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.285+0000 7f3817e00640 1 -- 172.21.15.39:0/474437558 learned_addr learned my addr 172.21.15.39:0/474437558 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:13:44.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.285+0000 7f3817400640 1 --2- 172.21.15.39:0/474437558 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559ed3922810 0x559ed391cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:44.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.285+0000 7f3817e00640 1 -- 172.21.15.39:0/474437558 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559ed3928370 msgr2=0x559ed393f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:44.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.285+0000 7f3817e00640 1 --2- 172.21.15.39:0/474437558 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559ed3928370 0x559ed393f940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:44.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.285+0000 7f3817e00640 1 -- 172.21.15.39:0/474437558 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559ed3922810 msgr2=0x559ed391cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:44.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.285+0000 7f3817e00640 1 --2- 172.21.15.39:0/474437558 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559ed3922810 0x559ed391cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:44.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.285+0000 7f3817e00640 1 -- 172.21.15.39:0/474437558 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559ed3922450 con 0x559ed3923200 2024-03-18T16:13:44.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.286+0000 7f3817e00640 1 --2- 172.21.15.39:0/474437558 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559ed3923200 0x559ed39342e0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f380800a7a0 tx=0x7f38080045a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a6c1b94155ce8807 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:44.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.287+0000 7f380cc00640 1 -- 172.21.15.39:0/474437558 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3808005050 con 0x559ed3923200 2024-03-18T16:13:44.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.287+0000 7f380cc00640 1 -- 172.21.15.39:0/474437558 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3808003ea0 con 0x559ed3923200 2024-03-18T16:13:44.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.287+0000 7f380cc00640 1 -- 172.21.15.39:0/474437558 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3808003b00 con 0x559ed3923200 2024-03-18T16:13:44.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.289+0000 7f3821e2c740 1 -- 172.21.15.39:0/474437558 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559ed3923200 msgr2=0x559ed39342e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:44.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.289+0000 7f3821e2c740 1 --2- 172.21.15.39:0/474437558 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559ed3923200 0x559ed39342e0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f380800a7a0 tx=0x7f38080045a0 comp rx=0 tx=0).stop 2024-03-18T16:13:44.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.289+0000 7f3821e2c740 1 -- 172.21.15.39:0/474437558 shutdown_connections 2024-03-18T16:13:44.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.289+0000 7f3821e2c740 1 --2- 172.21.15.39:0/474437558 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559ed3928370 0x559ed393f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:44.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.289+0000 7f3821e2c740 1 --2- 172.21.15.39:0/474437558 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559ed3923200 0x559ed39342e0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:44.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.289+0000 7f3821e2c740 1 --2- 172.21.15.39:0/474437558 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559ed3922810 0x559ed391cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:44.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.289+0000 7f3821e2c740 1 -- 172.21.15.39:0/474437558 >> 172.21.15.39:0/474437558 conn(0x559ed391dd80 msgr2=0x559ed391c630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:44.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.291+0000 7f3821e2c740 1 -- 172.21.15.39:0/474437558 shutdown_connections 2024-03-18T16:13:44.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.291+0000 7f3821e2c740 1 -- 172.21.15.39:0/474437558 wait complete. 2024-03-18T16:13:44.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.291+0000 7f3821e2c740 1 -- start start 2024-03-18T16:13:44.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.292+0000 7f3821e2c740 1 Processor -- start 2024-03-18T16:13:44.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.294+0000 7f3821e2c740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559ed3922810 0x559ed3931c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:44.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.295+0000 7f3821e2c740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559ed3923200 0x559ed39321c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:44.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.295+0000 7f3821e2c740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559ed3928370 0x559ed3933160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:44.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.295+0000 7f3821e2c740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f38181b63c0 con 0x559ed3923200 2024-03-18T16:13:44.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.295+0000 7f3817400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559ed3922810 0x559ed3931c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:44.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.295+0000 7f3821e2c740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f38181b5c30 con 0x559ed3922810 2024-03-18T16:13:44.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.295+0000 7f3821e2c740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f38181b2e70 con 0x559ed3928370 2024-03-18T16:13:44.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.295+0000 7f3817400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559ed3922810 0x559ed3931c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:60286/0 (socket says 172.21.15.39:60286) 2024-03-18T16:13:44.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.295+0000 7f3817400640 1 -- 172.21.15.39:0/2222004201 learned_addr learned my addr 172.21.15.39:0/2222004201 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:13:44.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.295+0000 7f3821e2c740 10 client.0 _ll_register_callbacks cb 0x7f380400e5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:13:44.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.296+0000 7f3817400640 1 --2- 172.21.15.39:0/2222004201 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559ed3928370 0x559ed3933160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:44.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.296+0000 7f3817400640 1 -- 172.21.15.39:0/2222004201 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559ed3928370 msgr2=0x559ed3933160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:44.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.296+0000 7f3817400640 1 --2- 172.21.15.39:0/2222004201 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559ed3928370 0x559ed3933160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:44.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.296+0000 7f3817400640 1 -- 172.21.15.39:0/2222004201 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559ed3923200 msgr2=0x559ed39321c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:13:44.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.296+0000 7f3817400640 1 --2- 172.21.15.39:0/2222004201 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559ed3923200 0x559ed39321c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:44.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.296+0000 7f3817400640 1 -- 172.21.15.39:0/2222004201 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559ed3918370 con 0x559ed3922810 2024-03-18T16:13:44.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.297+0000 7f3817400640 1 --2- 172.21.15.39:0/2222004201 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559ed3922810 0x559ed3931c80 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f3804015b00 tx=0x7f380400e720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=de67f2df7b5807ba server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:13:44.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.297+0000 7f3802000640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:13:44.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.298+0000 7f3802000640 1 -- 172.21.15.39:0/2222004201 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3804013cf0 con 0x559ed3922810 2024-03-18T16:13:44.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.298+0000 7f3802000640 1 -- 172.21.15.39:0/2222004201 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f38040209f0 con 0x559ed3922810 2024-03-18T16:13:44.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.298+0000 7f3821e2c740 1 -- 172.21.15.39:0/2222004201 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3808003ea0 con 0x559ed3922810 2024-03-18T16:13:44.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.298+0000 7f3802000640 1 -- 172.21.15.39:0/2222004201 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f3804013e90 con 0x559ed3922810 2024-03-18T16:13:44.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.298+0000 7f3821e2c740 10 client.26396 fetch_fsmap learned FSMap version 36 2024-03-18T16:13:44.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.298+0000 7f3821e2c740 1 -- 172.21.15.39:0/2222004201 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x559ed39a7850 con 0x559ed3922810 2024-03-18T16:13:44.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.299+0000 7f3802000640 1 -- 172.21.15.39:0/2222004201 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3808003ea0 con 0x559ed3922810 2024-03-18T16:13:44.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.299+0000 7f3802000640 1 -- 172.21.15.39:0/2222004201 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 36) ==== 8406+0+0 (secure 0 0 0) 0x7f3804028040 con 0x559ed3922810 2024-03-18T16:13:44.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.299+0000 7f3821e2c740 10 client.26396 fetch_fsmap finished waiting for FSMap version 36 2024-03-18T16:13:44.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.299+0000 7f3821e2c740 10 client.26396 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:13:44.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.299+0000 7f3821e2c740 20 client.26396 populate_metadata read hostname 'smithi039' 2024-03-18T16:13:44.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.299+0000 7f3821e2c740 1 --2- 172.21.15.39:0/2222004201 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x559ed39a8200 0x559ed39aa620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:13:44.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.299+0000 7f3821e2c740 4 client.26396 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T16:13:44.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.299+0000 7f3821e2c740 1 -- 172.21.15.39:0/2222004201 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "scrub status"}) -- 0x559ed3894730 con 0x559ed39a8200 2024-03-18T16:13:44.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.300+0000 7f3817e00640 1 --2- 172.21.15.39:0/2222004201 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x559ed39a8200 0x559ed39aa620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:13:44.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.300+0000 7f3802000640 10 client.26396 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:13:44.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.300+0000 7f3817e00640 1 --2- 172.21.15.39:0/2222004201 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x559ed39a8200 0x559ed39aa620 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9103fc92f5689bc9 server_cookie=94ecca95381f7405 in_seq=0 out_seq=0 2024-03-18T16:13:44.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.301+0000 7f3802000640 1 -- 172.21.15.39:0/2222004201 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+265 (crc 0 0 0) 0x559ed3894730 con 0x559ed39a8200 2024-03-18T16:13:44.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.301+0000 7f3802000640 10 client.26396 handle_command_reply: tid=0 2024-03-18T16:13:44.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.301+0000 7f3821e2c740 1 client.26396 shutdown 2024-03-18T16:13:44.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.301+0000 7f3821e2c740 1 -- 172.21.15.39:0/2222004201 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559ed3922810 msgr2=0x559ed3931c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:44.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.301+0000 7f3821e2c740 1 --2- 172.21.15.39:0/2222004201 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559ed3922810 0x559ed3931c80 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f3804015b00 tx=0x7f380400e720 comp rx=0 tx=0).stop 2024-03-18T16:13:44.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.301+0000 7f3821e2c740 1 -- 172.21.15.39:0/2222004201 shutdown_connections 2024-03-18T16:13:44.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.301+0000 7f3821e2c740 1 --2- 172.21.15.39:0/2222004201 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x559ed39a8200 0x559ed39aa620 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:44.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.301+0000 7f3821e2c740 1 --2- 172.21.15.39:0/2222004201 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559ed3928370 0x559ed3933160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:44.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.302+0000 7f3821e2c740 1 --2- 172.21.15.39:0/2222004201 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559ed3923200 0x559ed39321c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:44.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.302+0000 7f3802000640 0 client.26396 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:13:44.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.302+0000 7f3821e2c740 1 --2- 172.21.15.39:0/2222004201 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559ed3922810 0x559ed3931c80 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:44.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.302+0000 7f3821e2c740 1 -- 172.21.15.39:0/2222004201 >> 172.21.15.39:0/2222004201 conn(0x559ed391dd80 msgr2=0x559ed391cec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:44.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.302+0000 7f3821e2c740 1 -- 172.21.15.39:0/2222004201 shutdown_connections 2024-03-18T16:13:44.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.302+0000 7f3821e2c740 1 -- 172.21.15.39:0/2222004201 wait complete. 2024-03-18T16:13:44.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.302+0000 7f3821e2c740 20 client.26396 trim_cache size 0 max 16384 2024-03-18T16:13:44.306 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-03-18T16:13:44.306 INFO:teuthology.orchestra.run.smithi039.stdout: "status": "scrub active (0 inodes in the stack)", 2024-03-18T16:13:44.306 INFO:teuthology.orchestra.run.smithi039.stdout: "scrubs": { 2024-03-18T16:13:44.306 INFO:teuthology.orchestra.run.smithi039.stdout: "bdbde0aa-7919-40dc-97e4-d7fb17eeb280": { 2024-03-18T16:13:44.306 INFO:teuthology.orchestra.run.smithi039.stdout: "path": "/", 2024-03-18T16:13:44.306 INFO:teuthology.orchestra.run.smithi039.stdout: "tag": "bdbde0aa-7919-40dc-97e4-d7fb17eeb280", 2024-03-18T16:13:44.306 INFO:teuthology.orchestra.run.smithi039.stdout: "options": "recursive,force" 2024-03-18T16:13:44.306 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:13:44.306 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:13:44.306 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-03-18T16:13:44.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.303+0000 7f381f000640 1 -- 172.21.15.39:0/2632111956 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f37fc0a2810 msgr2=0x7f37fc0a4cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:44.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.303+0000 7f381f000640 1 --2- 172.21.15.39:0/2632111956 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f37fc0a2810 0x7f37fc0a4cd0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f381002c660 tx=0x7f38100095c0 comp rx=0 tx=0).stop 2024-03-18T16:13:44.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.303+0000 7f381f000640 1 -- 172.21.15.39:0/2632111956 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f381810b5d0 msgr2=0x7f38181a9a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:13:44.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.303+0000 7f381f000640 1 --2- 172.21.15.39:0/2632111956 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f381810b5d0 0x7f38181a9a70 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f3810009bf0 tx=0x7f3810035d80 comp rx=0 tx=0).stop 2024-03-18T16:13:44.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.304+0000 7f381f000640 1 -- 172.21.15.39:0/2632111956 shutdown_connections 2024-03-18T16:13:44.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.304+0000 7f381f000640 1 --2- 172.21.15.39:0/2632111956 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f381810b5d0 0x7f38181a9a70 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:44.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.304+0000 7f381f000640 1 --2- 172.21.15.39:0/2632111956 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f37fc0a2810 0x7f37fc0a4cd0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:44.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.304+0000 7f381f000640 1 --2- 172.21.15.39:0/2632111956 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f381810b150 0x7f38181a2a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:44.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.304+0000 7f381f000640 1 --2- 172.21.15.39:0/2632111956 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f38181087e0 0x7f38181a24c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:13:44.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.304+0000 7f381f000640 1 -- 172.21.15.39:0/2632111956 >> 172.21.15.39:0/2632111956 conn(0x7f381806e9e0 msgr2=0x7f381806f300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:13:44.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.304+0000 7f381f000640 1 -- 172.21.15.39:0/2632111956 shutdown_connections 2024-03-18T16:13:44.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:13:44.304+0000 7f381f000640 1 -- 172.21.15.39:0/2632111956 wait complete. 2024-03-18T16:13:44.332 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:13:44 smithi039 ceph-mon[31721]: scrub queued for path: / 2024-03-18T16:13:44.332 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:13:44 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:13:44.332 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:13:44 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:13:44.332 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:13:44 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:13:44.332 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:13:44 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:13:44.332 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:13:44 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:13:44.336 INFO:tasks.cephfs.filesystem:scrub status for tag:bdbde0aa-7919-40dc-97e4-d7fb17eeb280 - {'path': '/', 'tag': 'bdbde0aa-7919-40dc-97e4-d7fb17eeb280', 'options': 'recursive,force'} 2024-03-18T16:13:44.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:13:44 smithi196 ceph-mon[36307]: scrub queued for path: / 2024-03-18T16:13:44.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:13:44 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:13:44.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:13:44 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:13:44.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:13:44 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:13:44.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:13:44 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:13:44.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:13:44 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:13:44.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:13:44 smithi194 ceph-mon[36200]: scrub queued for path: / 2024-03-18T16:13:44.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:13:44 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:13:44.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:13:44 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:13:44.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:13:44 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:13:44.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:13:44 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:13:44.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:13:44 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:13:45.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:13:45 smithi196 ceph-mon[36307]: pgmap v532: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 65 MiB/s wr, 8.44k op/s 2024-03-18T16:13:45.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:13:45 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:13:45.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:13:45 smithi194 ceph-mon[36200]: pgmap v532: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 65 MiB/s wr, 8.44k op/s 2024-03-18T16:13:45.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:13:45 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:13:45.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:13:45 smithi039 ceph-mon[31721]: pgmap v532: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 65 MiB/s wr, 8.44k op/s 2024-03-18T16:13:45.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:13:45 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:13:46.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:13:46 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:13:46.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:13:46 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:13:46.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:13:46 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:13:46.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:13:46 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:13:46.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:13:46 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:13:46.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:13:46 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:13:46.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:13:46 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:13:46.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:13:46 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:13:46.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:13:46 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:13:46.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:13:46 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:13:46.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:13:46 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:13:46.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:13:46 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:13:46.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:13:46 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:13:46.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:13:46 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:13:46.589 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:13:46 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:13:46.589 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:13:46 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:13:46.589 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:13:46 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:13:46.589 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:13:46 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:13:47.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:13:47 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:13:47.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:13:47 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:13:47.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:13:47 smithi196 ceph-mon[36307]: pgmap v533: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 56 MiB/s wr, 7.36k op/s 2024-03-18T16:13:47.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:13:47 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:13:47.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:13:47 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:13:47.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:13:47 smithi194 ceph-mon[36200]: pgmap v533: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 56 MiB/s wr, 7.36k op/s 2024-03-18T16:13:47.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:13:47 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:13:47.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:13:47 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:13:47.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:13:47 smithi039 ceph-mon[31721]: pgmap v533: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 56 MiB/s wr, 7.36k op/s 2024-03-18T16:13:49.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:13:49 smithi196 ceph-mon[36307]: pgmap v534: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 52 MiB/s wr, 6.93k op/s 2024-03-18T16:13:49.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:13:49 smithi194 ceph-mon[36200]: pgmap v534: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 52 MiB/s wr, 6.93k op/s 2024-03-18T16:13:49.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:13:49 smithi039 ceph-mon[31721]: pgmap v534: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 52 MiB/s wr, 6.93k op/s 2024-03-18T16:13:51.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:13:51 smithi196 ceph-mon[36307]: pgmap v535: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 54 MiB/s wr, 7.25k op/s 2024-03-18T16:13:51.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:13:51 smithi194 ceph-mon[36200]: pgmap v535: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 54 MiB/s wr, 7.25k op/s 2024-03-18T16:13:51.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:13:51 smithi039 ceph-mon[31721]: pgmap v535: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 54 MiB/s wr, 7.25k op/s 2024-03-18T16:13:53.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:13:53 smithi196 ceph-mon[36307]: pgmap v536: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 53 MiB/s wr, 7.23k op/s 2024-03-18T16:13:53.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:13:53 smithi194 ceph-mon[36200]: pgmap v536: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 53 MiB/s wr, 7.23k op/s 2024-03-18T16:13:53.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:13:53 smithi039 ceph-mon[31721]: pgmap v536: 129 pgs: 129 active+clean; 13 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 53 MiB/s wr, 7.23k op/s 2024-03-18T16:13:55.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:13:55 smithi196 ceph-mon[36307]: pgmap v537: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 59 MiB/s wr, 8.15k op/s 2024-03-18T16:13:55.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:13:55 smithi194 ceph-mon[36200]: pgmap v537: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 59 MiB/s wr, 8.15k op/s 2024-03-18T16:13:55.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:13:55 smithi039 ceph-mon[31721]: pgmap v537: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 59 MiB/s wr, 8.15k op/s 2024-03-18T16:13:57.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:13:57 smithi196 ceph-mon[36307]: pgmap v538: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 50 MiB/s wr, 6.99k op/s 2024-03-18T16:13:57.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:13:57 smithi194 ceph-mon[36200]: pgmap v538: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 50 MiB/s wr, 6.99k op/s 2024-03-18T16:13:57.589 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:13:57 smithi039 ceph-mon[31721]: pgmap v538: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 50 MiB/s wr, 6.99k op/s 2024-03-18T16:13:58.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:13:58 smithi194 ceph-mon[36200]: scrub summary: idle 2024-03-18T16:13:58.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:13:58 smithi194 ceph-mon[36200]: scrub summary: idle 2024-03-18T16:13:58.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:13:58 smithi194 ceph-mon[36200]: scrub summary: idle 2024-03-18T16:13:58.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:13:58 smithi039 ceph-mon[31721]: scrub summary: idle 2024-03-18T16:13:58.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:13:58 smithi039 ceph-mon[31721]: scrub summary: idle 2024-03-18T16:13:58.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:13:58 smithi039 ceph-mon[31721]: scrub summary: idle 2024-03-18T16:13:58.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:13:58 smithi196 ceph-mon[36307]: scrub summary: idle 2024-03-18T16:13:58.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:13:58 smithi196 ceph-mon[36307]: scrub summary: idle 2024-03-18T16:13:58.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:13:58 smithi196 ceph-mon[36307]: scrub summary: idle 2024-03-18T16:13:59.589 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:13:59 smithi039 ceph-mon[31721]: scrub summary: idle 2024-03-18T16:13:59.589 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:13:59 smithi039 ceph-mon[31721]: scrub summary: idle 2024-03-18T16:13:59.589 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:13:59 smithi039 ceph-mon[31721]: pgmap v539: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 50 MiB/s wr, 6.91k op/s 2024-03-18T16:13:59.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:13:59 smithi196 ceph-mon[36307]: scrub summary: idle 2024-03-18T16:13:59.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:13:59 smithi196 ceph-mon[36307]: scrub summary: idle 2024-03-18T16:13:59.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:13:59 smithi196 ceph-mon[36307]: pgmap v539: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 50 MiB/s wr, 6.91k op/s 2024-03-18T16:13:59.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:13:59 smithi194 ceph-mon[36200]: scrub summary: idle 2024-03-18T16:13:59.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:13:59 smithi194 ceph-mon[36200]: scrub summary: idle 2024-03-18T16:13:59.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:13:59 smithi194 ceph-mon[36200]: pgmap v539: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 50 MiB/s wr, 6.91k op/s 2024-03-18T16:14:01.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:14:01 smithi039 ceph-mon[31721]: pgmap v540: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 61 MiB/s wr, 8.54k op/s 2024-03-18T16:14:01.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:14:01 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:14:01.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:14:01 smithi196 ceph-mon[36307]: pgmap v540: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 61 MiB/s wr, 8.54k op/s 2024-03-18T16:14:01.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:14:01 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:14:01.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:14:01 smithi194 ceph-mon[36200]: pgmap v540: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 61 MiB/s wr, 8.54k op/s 2024-03-18T16:14:01.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:14:01 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:14:03.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:14:03 smithi196 ceph-mon[36307]: pgmap v541: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 52 MiB/s wr, 7.21k op/s 2024-03-18T16:14:03.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:14:03 smithi194 ceph-mon[36200]: pgmap v541: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 52 MiB/s wr, 7.21k op/s 2024-03-18T16:14:03.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:14:03 smithi039 ceph-mon[31721]: pgmap v541: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 52 MiB/s wr, 7.21k op/s 2024-03-18T16:14:05.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:14:05 smithi196 ceph-mon[36307]: pgmap v542: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 58 MiB/s wr, 7.99k op/s 2024-03-18T16:14:05.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:14:05 smithi194 ceph-mon[36200]: pgmap v542: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 58 MiB/s wr, 7.99k op/s 2024-03-18T16:14:05.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:14:05 smithi039 ceph-mon[31721]: pgmap v542: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 32 KiB/s rd, 58 MiB/s wr, 7.99k op/s 2024-03-18T16:14:07.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:14:07 smithi196 ceph-mon[36307]: pgmap v543: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 53 MiB/s wr, 7.39k op/s 2024-03-18T16:14:07.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:14:07 smithi194 ceph-mon[36200]: pgmap v543: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 53 MiB/s wr, 7.39k op/s 2024-03-18T16:14:07.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:14:07 smithi039 ceph-mon[31721]: pgmap v543: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 53 MiB/s wr, 7.39k op/s 2024-03-18T16:14:09.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:14:09 smithi196 ceph-mon[36307]: pgmap v544: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 52 MiB/s wr, 7.22k op/s 2024-03-18T16:14:09.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:14:09 smithi194 ceph-mon[36200]: pgmap v544: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 52 MiB/s wr, 7.22k op/s 2024-03-18T16:14:09.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:14:09 smithi039 ceph-mon[31721]: pgmap v544: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 52 MiB/s wr, 7.22k op/s 2024-03-18T16:14:11.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:14:11 smithi196 ceph-mon[36307]: pgmap v545: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 53 MiB/s wr, 7.68k op/s 2024-03-18T16:14:11.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:14:11 smithi194 ceph-mon[36200]: pgmap v545: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 53 MiB/s wr, 7.68k op/s 2024-03-18T16:14:11.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:14:11 smithi039 ceph-mon[31721]: pgmap v545: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 30 KiB/s rd, 53 MiB/s wr, 7.68k op/s 2024-03-18T16:14:13.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:14:13 smithi196 ceph-mon[36307]: pgmap v546: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 42 MiB/s wr, 6.30k op/s 2024-03-18T16:14:13.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:14:13 smithi194 ceph-mon[36200]: pgmap v546: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 42 MiB/s wr, 6.30k op/s 2024-03-18T16:14:13.838 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:14:13 smithi039 ceph-mon[31721]: pgmap v546: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 42 MiB/s wr, 6.30k op/s 2024-03-18T16:14:14.337 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T16:14:14.337 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T16:14:15.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:14:15 smithi196 ceph-mon[36307]: pgmap v547: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 44 MiB/s wr, 6.72k op/s 2024-03-18T16:14:15.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:14:15 smithi194 ceph-mon[36200]: pgmap v547: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 44 MiB/s wr, 6.72k op/s 2024-03-18T16:14:15.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:14:15 smithi039 ceph-mon[31721]: pgmap v547: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 44 MiB/s wr, 6.72k op/s 2024-03-18T16:14:17.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:14:17 smithi196 ceph-mon[36307]: pgmap v548: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 38 MiB/s wr, 6.13k op/s 2024-03-18T16:14:17.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:14:17 smithi194 ceph-mon[36200]: pgmap v548: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 38 MiB/s wr, 6.13k op/s 2024-03-18T16:14:17.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:14:17 smithi039 ceph-mon[31721]: pgmap v548: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 38 MiB/s wr, 6.13k op/s 2024-03-18T16:14:19.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:14:19 smithi196 ceph-mon[36307]: pgmap v549: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 40 MiB/s wr, 6.46k op/s 2024-03-18T16:14:19.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:14:19 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:14:19.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:14:19 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:14:19.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:14:19 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:14:19.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:14:19 smithi194 ceph-mon[36200]: pgmap v549: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 40 MiB/s wr, 6.46k op/s 2024-03-18T16:14:19.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:14:19 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:14:19.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:14:19 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:14:19.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:14:19 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:14:19.838 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:14:19 smithi039 ceph-mon[31721]: pgmap v549: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 40 MiB/s wr, 6.46k op/s 2024-03-18T16:14:19.839 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:14:19 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:14:19.839 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:14:19 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:14:19.839 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:14:19 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:14:21.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.102+0000 7f6f0d000640 1 -- 172.21.15.39:0/3264389510 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6f080ffaa0 msgr2=0x7f6f080ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:14:21.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.102+0000 7f6f0d000640 1 --2- 172.21.15.39:0/3264389510 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6f080ffaa0 0x7f6f080ffec0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f6f00009cd0 tx=0x7f6f0002c0f0 comp rx=0 tx=0).stop 2024-03-18T16:14:21.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.102+0000 7f6f0d000640 1 -- 172.21.15.39:0/3264389510 shutdown_connections 2024-03-18T16:14:21.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.102+0000 7f6f0d000640 1 --2- 172.21.15.39:0/3264389510 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6f08103430 0x7f6f08109cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:21.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.102+0000 7f6f0d000640 1 --2- 172.21.15.39:0/3264389510 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f08102a70 0x7f6f08102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:21.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.102+0000 7f6f0d000640 1 --2- 172.21.15.39:0/3264389510 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6f080ffaa0 0x7f6f080ffec0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:21.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.102+0000 7f6f0d000640 1 -- 172.21.15.39:0/3264389510 >> 172.21.15.39:0/3264389510 conn(0x7f6f080fb3d0 msgr2=0x7f6f080fd810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:14:21.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.102+0000 7f6f0d000640 1 -- 172.21.15.39:0/3264389510 shutdown_connections 2024-03-18T16:14:21.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.102+0000 7f6f0d000640 1 -- 172.21.15.39:0/3264389510 wait complete. 2024-03-18T16:14:21.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.105+0000 7f6f0d000640 1 Processor -- start 2024-03-18T16:14:21.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.106+0000 7f6f0d000640 1 -- start start 2024-03-18T16:14:21.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.107+0000 7f6f0d000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f080ffaa0 0x7f6f08195990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:21.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.107+0000 7f6f0d000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6f08102a70 0x7f6f08195ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:21.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.107+0000 7f6f0d000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6f08103430 0x7f6f081ae360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:21.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.107+0000 7f6f0d000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f6f0810d450 con 0x7f6f080ffaa0 2024-03-18T16:14:21.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.107+0000 7f6f0d000640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f6f0810d2d0 con 0x7f6f08102a70 2024-03-18T16:14:21.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.107+0000 7f6f0d000640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f6f0810d5d0 con 0x7f6f08103430 2024-03-18T16:14:21.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.107+0000 7f6f05e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f080ffaa0 0x7f6f08195990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:21.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.107+0000 7f6f05e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f080ffaa0 0x7f6f08195990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:60348/0 (socket says 172.21.15.39:60348) 2024-03-18T16:14:21.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.107+0000 7f6f05e00640 1 -- 172.21.15.39:0/395994675 learned_addr learned my addr 172.21.15.39:0/395994675 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:14:21.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.107+0000 7f6f05400640 1 --2- 172.21.15.39:0/395994675 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6f08102a70 0x7f6f08195ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:21.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.108+0000 7f6f05e00640 1 -- 172.21.15.39:0/395994675 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6f08103430 msgr2=0x7f6f081ae360 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T16:14:21.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.108+0000 7f6f05e00640 1 --2- 172.21.15.39:0/395994675 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6f08103430 0x7f6f081ae360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:21.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.108+0000 7f6f05e00640 1 -- 172.21.15.39:0/395994675 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6f08102a70 msgr2=0x7f6f08195ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:14:21.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.108+0000 7f6f05e00640 1 --2- 172.21.15.39:0/395994675 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6f08102a70 0x7f6f08195ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:21.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.108+0000 7f6f05e00640 1 -- 172.21.15.39:0/395994675 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f0819b060 con 0x7f6f080ffaa0 2024-03-18T16:14:21.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.109+0000 7f6f05e00640 1 --2- 172.21.15.39:0/395994675 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f080ffaa0 0x7f6f08195990 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f6f00009ca0 tx=0x7f6f00032ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:14:21.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.109+0000 7f6efea00640 1 -- 172.21.15.39:0/395994675 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f00039040 con 0x7f6f080ffaa0 2024-03-18T16:14:21.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.109+0000 7f6efea00640 1 -- 172.21.15.39:0/395994675 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f00044070 con 0x7f6f080ffaa0 2024-03-18T16:14:21.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.109+0000 7f6efea00640 1 -- 172.21.15.39:0/395994675 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f000042a0 con 0x7f6f080ffaa0 2024-03-18T16:14:21.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.110+0000 7f6f0d000640 1 -- 172.21.15.39:0/395994675 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f081ae8a0 con 0x7f6f080ffaa0 2024-03-18T16:14:21.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.110+0000 7f6f0d000640 1 -- 172.21.15.39:0/395994675 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f081aed10 con 0x7f6f080ffaa0 2024-03-18T16:14:21.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.117+0000 7f6efea00640 1 -- 172.21.15.39:0/395994675 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f6f00003970 con 0x7f6f080ffaa0 2024-03-18T16:14:21.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.119+0000 7f6f0fdfe740 1 Processor -- start 2024-03-18T16:14:21.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.120+0000 7f6efea00640 1 --2- 172.21.15.39:0/395994675 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f6eec0a2880 0x7f6eec0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:21.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.120+0000 7f6efea00640 1 -- 172.21.15.39:0/395994675 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f6f000f11a0 con 0x7f6f080ffaa0 2024-03-18T16:14:21.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.120+0000 7f6f0fdfe740 1 -- start start 2024-03-18T16:14:21.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.120+0000 7f6f0fdfe740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55558e4dc810 0x55558e4dcc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:21.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.120+0000 7f6f0fdfe740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55558e4dd200 0x55558e4e1d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:21.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.121+0000 7f6f0fdfe740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55558e4e2370 0x55558e4e4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:21.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.121+0000 7f6f0fdfe740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55558e4411f0 con 0x55558e4dc810 2024-03-18T16:14:21.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.121+0000 7f6f0fdfe740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55558e3e9a80 con 0x55558e4dd200 2024-03-18T16:14:21.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.121+0000 7f6f0fdfe740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55558e43d290 con 0x55558e4e2370 2024-03-18T16:14:21.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.121+0000 7f6f05400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55558e4dd200 0x55558e4e1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:21.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.121+0000 7f6f05400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55558e4dd200 0x55558e4e1d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:48770/0 (socket says 172.21.15.39:48770) 2024-03-18T16:14:21.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.121+0000 7f6f05400640 1 -- 172.21.15.39:0/3384334203 learned_addr learned my addr 172.21.15.39:0/3384334203 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:14:21.124 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.122+0000 7f6f06800640 1 --2- 172.21.15.39:0/395994675 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f6eec0a2880 0x7f6eec0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:21.124 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.122+0000 7f6f06800640 1 --2- 172.21.15.39:0/3384334203 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55558e4e2370 0x55558e4e4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:21.124 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.123+0000 7f6f05e00640 1 --2- 172.21.15.39:0/3384334203 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55558e4dc810 0x55558e4dcc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:21.124 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.123+0000 7f6f05400640 1 -- 172.21.15.39:0/3384334203 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55558e4e2370 msgr2=0x55558e4e4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:14:21.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.123+0000 7f6f05400640 1 --2- 172.21.15.39:0/3384334203 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55558e4e2370 0x55558e4e4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:21.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.123+0000 7f6f05400640 1 -- 172.21.15.39:0/3384334203 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55558e4dc810 msgr2=0x55558e4dcc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:14:21.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.123+0000 7f6f05400640 1 --2- 172.21.15.39:0/3384334203 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55558e4dc810 0x55558e4dcc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:21.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.123+0000 7f6f05400640 1 -- 172.21.15.39:0/3384334203 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55558e21f7d0 con 0x55558e4dd200 2024-03-18T16:14:21.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.123+0000 7f6f05400640 1 --2- 172.21.15.39:0/3384334203 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55558e4dd200 0x55558e4e1d00 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f6ef800bc90 tx=0x7f6ef80107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7afacb656295823b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:14:21.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.124+0000 7f6f06800640 1 --2- 172.21.15.39:0/395994675 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f6eec0a2880 0x7f6eec0a4d40 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f6ef4005fd0 tx=0x7f6ef400c520 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:14:21.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.124+0000 7f6ef3400640 1 -- 172.21.15.39:0/3384334203 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ef801c480 con 0x55558e4dd200 2024-03-18T16:14:21.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.124+0000 7f6ef3400640 1 -- 172.21.15.39:0/3384334203 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ef801cb00 con 0x55558e4dd200 2024-03-18T16:14:21.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.125+0000 7f6ef3400640 1 -- 172.21.15.39:0/3384334203 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ef8011b60 con 0x55558e4dd200 2024-03-18T16:14:21.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.126+0000 7f6f0fdfe740 1 -- 172.21.15.39:0/3384334203 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55558e4dd200 msgr2=0x55558e4e1d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:14:21.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.126+0000 7f6f0fdfe740 1 --2- 172.21.15.39:0/3384334203 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55558e4dd200 0x55558e4e1d00 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f6ef800bc90 tx=0x7f6ef80107a0 comp rx=0 tx=0).stop 2024-03-18T16:14:21.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.130+0000 7f6f0fdfe740 1 -- 172.21.15.39:0/3384334203 shutdown_connections 2024-03-18T16:14:21.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.130+0000 7f6f0fdfe740 1 --2- 172.21.15.39:0/3384334203 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55558e4dd200 0x55558e4e1d00 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:21.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.130+0000 7f6f0fdfe740 1 --2- 172.21.15.39:0/3384334203 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55558e4e2370 0x55558e4e4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:21.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.130+0000 7f6f0fdfe740 1 --2- 172.21.15.39:0/3384334203 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55558e4dc810 0x55558e4dcc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:21.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.130+0000 7f6f0fdfe740 1 -- 172.21.15.39:0/3384334203 >> 172.21.15.39:0/3384334203 conn(0x55558e4d7d80 msgr2=0x55558e4d8180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:14:21.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.131+0000 7f6f0fdfe740 1 -- 172.21.15.39:0/3384334203 shutdown_connections 2024-03-18T16:14:21.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.131+0000 7f6f0fdfe740 1 -- 172.21.15.39:0/3384334203 wait complete. 2024-03-18T16:14:21.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.132+0000 7f6f0fdfe740 1 -- start start 2024-03-18T16:14:21.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.132+0000 7f6f0fdfe740 1 Processor -- start 2024-03-18T16:14:21.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.133+0000 7f6f0fdfe740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55558e4dc810 0x55558e4f7ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:21.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.133+0000 7f6f0fdfe740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55558e4dd200 0x55558e4f8000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:21.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.133+0000 7f6f0fdfe740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55558e4e2370 0x55558e4ff040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:21.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.133+0000 7f6f05e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55558e4dc810 0x55558e4f7ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:21.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.133+0000 7f6f0fdfe740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55558e4e5370 con 0x55558e4dc810 2024-03-18T16:14:21.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.133+0000 7f6f0fdfe740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55558e3fced0 con 0x55558e4dd200 2024-03-18T16:14:21.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.133+0000 7f6f0fdfe740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55558e4e54f0 con 0x55558e4e2370 2024-03-18T16:14:21.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.133+0000 7f6f05e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55558e4dc810 0x55558e4f7ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:60352/0 (socket says 172.21.15.39:60352) 2024-03-18T16:14:21.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.134+0000 7f6f05e00640 1 -- 172.21.15.39:0/4190569303 learned_addr learned my addr 172.21.15.39:0/4190569303 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:14:21.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.134+0000 7f6f05400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55558e4dd200 0x55558e4f8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:21.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.134+0000 7f6f06800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55558e4e2370 0x55558e4ff040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:21.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.134+0000 7f6f0fdfe740 10 client.0 _ll_register_callbacks cb 0x55558e438220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:14:21.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.134+0000 7f6f06800640 1 -- 172.21.15.39:0/4190569303 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55558e4dd200 msgr2=0x55558e4f8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:14:21.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.134+0000 7f6f06800640 1 --2- 172.21.15.39:0/4190569303 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55558e4dd200 0x55558e4f8000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:21.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.134+0000 7f6f06800640 1 -- 172.21.15.39:0/4190569303 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55558e4dc810 msgr2=0x55558e4f7ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:14:21.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.134+0000 7f6f06800640 1 --2- 172.21.15.39:0/4190569303 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55558e4dc810 0x55558e4f7ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:21.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.136+0000 7f6f06800640 1 -- 172.21.15.39:0/4190569303 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55558e1553f0 con 0x55558e4e2370 2024-03-18T16:14:21.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.136+0000 7f6f05e00640 1 --2- 172.21.15.39:0/4190569303 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55558e4dc810 0x55558e4f7ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:14:21.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.136+0000 7f6f06800640 1 --2- 172.21.15.39:0/4190569303 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55558e4e2370 0x55558e4ff040 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f6ef4005d00 tx=0x7f6ef40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6be4903da5e2e131 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:14:21.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.138+0000 7f6ee7e00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:14:21.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.138+0000 7f6ee7e00640 1 -- 172.21.15.39:0/4190569303 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ef4026070 con 0x55558e4e2370 2024-03-18T16:14:21.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.138+0000 7f6ee7e00640 1 -- 172.21.15.39:0/4190569303 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f6ef4005020 con 0x55558e4e2370 2024-03-18T16:14:21.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.138+0000 7f6f0fdfe740 1 -- 172.21.15.39:0/4190569303 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6ef801cb00 con 0x55558e4e2370 2024-03-18T16:14:21.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.139+0000 7f6ee7e00640 1 -- 172.21.15.39:0/4190569303 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f6ef402f050 con 0x55558e4e2370 2024-03-18T16:14:21.141 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.139+0000 7f6f0fdfe740 10 client.26398 fetch_fsmap learned FSMap version 36 2024-03-18T16:14:21.141 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.139+0000 7f6f0fdfe740 1 -- 172.21.15.39:0/4190569303 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55558e1150f0 con 0x55558e4e2370 2024-03-18T16:14:21.141 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.140+0000 7f6ee7e00640 1 -- 172.21.15.39:0/4190569303 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ef801cb00 con 0x55558e4e2370 2024-03-18T16:14:21.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.141+0000 7f6ee7e00640 1 -- 172.21.15.39:0/4190569303 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 36) ==== 8406+0+0 (secure 0 0 0) 0x7f6ef4005220 con 0x55558e4e2370 2024-03-18T16:14:21.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.141+0000 7f6f0fdfe740 10 client.26398 fetch_fsmap finished waiting for FSMap version 36 2024-03-18T16:14:21.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.141+0000 7f6f0fdfe740 10 client.26398 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:14:21.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.142+0000 7f6f0fdfe740 20 client.26398 populate_metadata read hostname 'smithi039' 2024-03-18T16:14:21.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.142+0000 7f6f0fdfe740 1 --2- 172.21.15.39:0/4190569303 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55558e4e99d0 0x55558e4ebe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:21.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.142+0000 7f6f0fdfe740 4 client.26398 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:14:21.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.142+0000 7f6f0fdfe740 1 -- 172.21.15.39:0/4190569303 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55558e454be0 con 0x55558e4e99d0 2024-03-18T16:14:21.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.143+0000 7f6f05e00640 1 --2- 172.21.15.39:0/4190569303 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55558e4e99d0 0x55558e4ebe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:21.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.144+0000 7f6f05e00640 1 --2- 172.21.15.39:0/4190569303 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55558e4e99d0 0x55558e4ebe90 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e734c8e9d8154f4a server_cookie=78bcd34586b366ae in_seq=0 out_seq=0 2024-03-18T16:14:21.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.144+0000 7f6ee7e00640 10 client.26398 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:14:21.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.153+0000 7f6ee7e00640 1 -- 172.21.15.39:0/4190569303 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55558e454be0 con 0x55558e4e99d0 2024-03-18T16:14:21.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.153+0000 7f6ee7e00640 10 client.26398 handle_command_reply: tid=0 2024-03-18T16:14:21.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.153+0000 7f6f0fdfe740 1 client.26398 shutdown 2024-03-18T16:14:21.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.153+0000 7f6f0fdfe740 1 -- 172.21.15.39:0/4190569303 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55558e4e2370 msgr2=0x55558e4ff040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:14:21.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.153+0000 7f6f0fdfe740 1 --2- 172.21.15.39:0/4190569303 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55558e4e2370 0x55558e4ff040 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f6ef4005d00 tx=0x7f6ef40046c0 comp rx=0 tx=0).stop 2024-03-18T16:14:21.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.157+0000 7f6f0fdfe740 1 -- 172.21.15.39:0/4190569303 shutdown_connections 2024-03-18T16:14:21.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.157+0000 7f6f0fdfe740 1 --2- 172.21.15.39:0/4190569303 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55558e4e99d0 0x55558e4ebe90 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:21.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.157+0000 7f6f0fdfe740 1 --2- 172.21.15.39:0/4190569303 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55558e4dd200 0x55558e4f8000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:21.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.157+0000 7f6f0fdfe740 1 --2- 172.21.15.39:0/4190569303 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55558e4e2370 0x55558e4ff040 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:21.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.157+0000 7f6f0fdfe740 1 --2- 172.21.15.39:0/4190569303 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55558e4dc810 0x55558e4f7ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:21.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.157+0000 7f6f0fdfe740 1 -- 172.21.15.39:0/4190569303 >> 172.21.15.39:0/4190569303 conn(0x55558e4d7d80 msgr2=0x55558e4d6ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:14:21.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.157+0000 7f6ee7e00640 0 client.26398 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:14:21.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.157+0000 7f6f0fdfe740 1 -- 172.21.15.39:0/4190569303 shutdown_connections 2024-03-18T16:14:21.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.159+0000 7f6f0fdfe740 1 -- 172.21.15.39:0/4190569303 wait complete. 2024-03-18T16:14:21.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.159+0000 7f6f0fdfe740 20 client.26398 trim_cache size 0 max 16384 2024-03-18T16:14:21.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.197+0000 7f6f0fdfe740 1 Processor -- start 2024-03-18T16:14:21.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.198+0000 7f6f0fdfe740 1 -- start start 2024-03-18T16:14:21.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.198+0000 7f6f0fdfe740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55558e4dc810 0x55558e4de420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:21.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.198+0000 7f6f0fdfe740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55558e4dd200 0x55558e4de960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:21.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.198+0000 7f6f0fdfe740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55558e4e2370 0x55558e4ff000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:21.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.198+0000 7f6f0fdfe740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f6f081b0a10 con 0x55558e4dd200 2024-03-18T16:14:21.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.198+0000 7f6f0fdfe740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f6f081b26d0 con 0x55558e4e2370 2024-03-18T16:14:21.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.198+0000 7f6f0fdfe740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f6f081b10e0 con 0x55558e4dc810 2024-03-18T16:14:21.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.199+0000 7f6f06800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55558e4e2370 0x55558e4ff000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:21.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.199+0000 7f6f06800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55558e4e2370 0x55558e4ff000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:48790/0 (socket says 172.21.15.39:48790) 2024-03-18T16:14:21.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.199+0000 7f6f06800640 1 -- 172.21.15.39:0/2233062995 learned_addr learned my addr 172.21.15.39:0/2233062995 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:14:21.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.199+0000 7f6f06800640 1 -- 172.21.15.39:0/2233062995 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55558e4dc810 msgr2=0x55558e4de420 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:14:21.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.199+0000 7f6f06800640 1 --2- 172.21.15.39:0/2233062995 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55558e4dc810 0x55558e4de420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:21.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.199+0000 7f6f06800640 1 -- 172.21.15.39:0/2233062995 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55558e4dd200 msgr2=0x55558e4de960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:14:21.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.199+0000 7f6f06800640 1 --2- 172.21.15.39:0/2233062995 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55558e4dd200 0x55558e4de960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:21.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.199+0000 7f6f06800640 1 -- 172.21.15.39:0/2233062995 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55558e4e9c00 con 0x55558e4e2370 2024-03-18T16:14:21.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.201+0000 7f6f06800640 1 --2- 172.21.15.39:0/2233062995 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55558e4e2370 0x55558e4ff000 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f6ef4004690 tx=0x7f6ef4014600 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e857ffbab1501bdf server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:14:21.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.201+0000 7f6ef2a00640 1 -- 172.21.15.39:0/2233062995 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ef405a070 con 0x55558e4e2370 2024-03-18T16:14:21.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.202+0000 7f6ef2a00640 1 -- 172.21.15.39:0/2233062995 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ef4004930 con 0x55558e4e2370 2024-03-18T16:14:21.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.202+0000 7f6ef2a00640 1 -- 172.21.15.39:0/2233062995 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ef4026070 con 0x55558e4e2370 2024-03-18T16:14:21.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.203+0000 7f6f0fdfe740 1 -- 172.21.15.39:0/2233062995 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55558e4e2370 msgr2=0x55558e4ff000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:14:21.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.203+0000 7f6f0fdfe740 1 --2- 172.21.15.39:0/2233062995 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55558e4e2370 0x55558e4ff000 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f6ef4004690 tx=0x7f6ef4014600 comp rx=0 tx=0).stop 2024-03-18T16:14:21.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.203+0000 7f6f0fdfe740 1 -- 172.21.15.39:0/2233062995 shutdown_connections 2024-03-18T16:14:21.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.203+0000 7f6f0fdfe740 1 --2- 172.21.15.39:0/2233062995 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55558e4e2370 0x55558e4ff000 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:21.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.203+0000 7f6f0fdfe740 1 --2- 172.21.15.39:0/2233062995 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55558e4dd200 0x55558e4de960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:21.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.203+0000 7f6f0fdfe740 1 --2- 172.21.15.39:0/2233062995 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55558e4dc810 0x55558e4de420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:21.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.203+0000 7f6f0fdfe740 1 -- 172.21.15.39:0/2233062995 >> 172.21.15.39:0/2233062995 conn(0x55558e4d7d80 msgr2=0x55558e4d8160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:14:21.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.203+0000 7f6f0fdfe740 1 -- 172.21.15.39:0/2233062995 shutdown_connections 2024-03-18T16:14:21.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.204+0000 7f6f0fdfe740 1 -- 172.21.15.39:0/2233062995 wait complete. 2024-03-18T16:14:21.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.204+0000 7f6f0fdfe740 1 -- start start 2024-03-18T16:14:21.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.204+0000 7f6f0fdfe740 1 Processor -- start 2024-03-18T16:14:21.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.206+0000 7f6f0fdfe740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55558e4dd200 0x55558e4fe9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:21.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.206+0000 7f6f0fdfe740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55558e4e2370 0x55558e4fef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:21.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.206+0000 7f6f0fdfe740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55558e4fcff0 0x55558e4fd470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:21.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.207+0000 7f6f0fdfe740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f6f081b1f40 con 0x55558e4e2370 2024-03-18T16:14:21.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.207+0000 7f6f0fdfe740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f6f081b17b0 con 0x55558e4dd200 2024-03-18T16:14:21.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.207+0000 7f6f0fdfe740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f6f080734c0 con 0x55558e4fcff0 2024-03-18T16:14:21.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.207+0000 7f6f0fdfe740 10 client.0 _ll_register_callbacks cb 0x7f6ef400cbf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:14:21.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.207+0000 7f6f06800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55558e4fcff0 0x55558e4fd470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:21.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.207+0000 7f6f05e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55558e4dd200 0x55558e4fe9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:21.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.207+0000 7f6f05e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55558e4dd200 0x55558e4fe9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:48802/0 (socket says 172.21.15.39:48802) 2024-03-18T16:14:21.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.207+0000 7f6f05e00640 1 -- 172.21.15.39:0/1185335614 learned_addr learned my addr 172.21.15.39:0/1185335614 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:14:21.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.208+0000 7f6f05e00640 1 -- 172.21.15.39:0/1185335614 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55558e4fcff0 msgr2=0x55558e4fd470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:14:21.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.209+0000 7f6f05e00640 1 --2- 172.21.15.39:0/1185335614 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55558e4fcff0 0x55558e4fd470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:21.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.209+0000 7f6f05e00640 1 -- 172.21.15.39:0/1185335614 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55558e4e2370 msgr2=0x55558e4fef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:14:21.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.209+0000 7f6f05e00640 1 --2- 172.21.15.39:0/1185335614 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55558e4e2370 0x55558e4fef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:21.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.209+0000 7f6f05e00640 1 -- 172.21.15.39:0/1185335614 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55558e4fdca0 con 0x55558e4dd200 2024-03-18T16:14:21.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.209+0000 7f6f06800640 1 --2- 172.21.15.39:0/1185335614 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55558e4fcff0 0x55558e4fd470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:14:21.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.210+0000 7f6f05e00640 1 --2- 172.21.15.39:0/1185335614 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55558e4dd200 0x55558e4fe9e0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f6f000b9bb0 tx=0x7f6f00003d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=60926e5258c98501 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:14:21.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.210+0000 7f6ee7e00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:14:21.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.211+0000 7f6ee7e00640 1 -- 172.21.15.39:0/1185335614 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f00035d60 con 0x55558e4dd200 2024-03-18T16:14:21.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.211+0000 7f6ee7e00640 1 -- 172.21.15.39:0/1185335614 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f6f000a80b0 con 0x55558e4dd200 2024-03-18T16:14:21.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.211+0000 7f6f0fdfe740 1 -- 172.21.15.39:0/1185335614 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6ef4004930 con 0x55558e4dd200 2024-03-18T16:14:21.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.211+0000 7f6ee7e00640 1 -- 172.21.15.39:0/1185335614 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f6f000bd750 con 0x55558e4dd200 2024-03-18T16:14:21.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.211+0000 7f6f0fdfe740 10 client.26414 fetch_fsmap learned FSMap version 36 2024-03-18T16:14:21.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.211+0000 7f6f0fdfe740 1 -- 172.21.15.39:0/1185335614 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55558e57b350 con 0x55558e4dd200 2024-03-18T16:14:21.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.212+0000 7f6ee7e00640 1 -- 172.21.15.39:0/1185335614 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ef4004930 con 0x55558e4dd200 2024-03-18T16:14:21.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.212+0000 7f6ee7e00640 1 -- 172.21.15.39:0/1185335614 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 36) ==== 8406+0+0 (secure 0 0 0) 0x7f6f000a6ce0 con 0x55558e4dd200 2024-03-18T16:14:21.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.212+0000 7f6f0fdfe740 10 client.26414 fetch_fsmap finished waiting for FSMap version 36 2024-03-18T16:14:21.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.212+0000 7f6f0fdfe740 10 client.26414 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:14:21.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.212+0000 7f6f0fdfe740 20 client.26414 populate_metadata read hostname 'smithi039' 2024-03-18T16:14:21.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.212+0000 7f6f0fdfe740 1 --2- 172.21.15.39:0/1185335614 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55558e57bd40 0x55558e57e160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:21.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.212+0000 7f6f0fdfe740 4 client.26414 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T16:14:21.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.212+0000 7f6f0fdfe740 1 -- 172.21.15.39:0/1185335614 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55558e44e730 con 0x55558e57bd40 2024-03-18T16:14:21.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.212+0000 7f6f05400640 1 --2- 172.21.15.39:0/1185335614 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55558e57bd40 0x55558e57e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:21.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.213+0000 7f6f05400640 1 --2- 172.21.15.39:0/1185335614 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55558e57bd40 0x55558e57e160 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2482ebb28c7521d9 server_cookie=99b6577a3d7b6220 in_seq=0 out_seq=0 2024-03-18T16:14:21.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.213+0000 7f6ee7e00640 10 client.26414 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:14:21.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.214+0000 7f6ee7e00640 1 -- 172.21.15.39:0/1185335614 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55558e44e730 con 0x55558e57bd40 2024-03-18T16:14:21.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.214+0000 7f6ee7e00640 10 client.26414 handle_command_reply: tid=0 2024-03-18T16:14:21.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.214+0000 7f6f0fdfe740 1 client.26414 shutdown 2024-03-18T16:14:21.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.217+0000 7f6f0fdfe740 1 -- 172.21.15.39:0/1185335614 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55558e4dd200 msgr2=0x55558e4fe9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:14:21.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.217+0000 7f6f0fdfe740 1 --2- 172.21.15.39:0/1185335614 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55558e4dd200 0x55558e4fe9e0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f6f000b9bb0 tx=0x7f6f00003d10 comp rx=0 tx=0).stop 2024-03-18T16:14:21.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.218+0000 7f6f0fdfe740 1 -- 172.21.15.39:0/1185335614 shutdown_connections 2024-03-18T16:14:21.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.218+0000 7f6f0fdfe740 1 --2- 172.21.15.39:0/1185335614 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55558e57bd40 0x55558e57e160 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:21.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.218+0000 7f6f0fdfe740 1 --2- 172.21.15.39:0/1185335614 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55558e4fcff0 0x55558e4fd470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:21.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.218+0000 7f6f0fdfe740 1 --2- 172.21.15.39:0/1185335614 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55558e4e2370 0x55558e4fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:21.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.218+0000 7f6f0fdfe740 1 --2- 172.21.15.39:0/1185335614 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55558e4dd200 0x55558e4fe9e0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:21.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.218+0000 7f6f0fdfe740 1 -- 172.21.15.39:0/1185335614 >> 172.21.15.39:0/1185335614 conn(0x55558e4d7d80 msgr2=0x55558e4de420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:14:21.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.218+0000 7f6ee7e00640 0 client.26414 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:14:21.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.218+0000 7f6f0fdfe740 1 -- 172.21.15.39:0/1185335614 shutdown_connections 2024-03-18T16:14:21.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.218+0000 7f6f0fdfe740 1 -- 172.21.15.39:0/1185335614 wait complete. 2024-03-18T16:14:21.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.218+0000 7f6f0fdfe740 20 client.26414 trim_cache size 0 max 16384 2024-03-18T16:14:21.221 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-03-18T16:14:21.221 INFO:teuthology.orchestra.run.smithi039.stdout: "status": "no active scrubs running", 2024-03-18T16:14:21.221 INFO:teuthology.orchestra.run.smithi039.stdout: "scrubs": {} 2024-03-18T16:14:21.221 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-03-18T16:14:21.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.221+0000 7f6f0d000640 1 -- 172.21.15.39:0/395994675 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f6eec0a2880 msgr2=0x7f6eec0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:14:21.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.221+0000 7f6f0d000640 1 --2- 172.21.15.39:0/395994675 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f6eec0a2880 0x7f6eec0a4d40 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f6ef4005fd0 tx=0x7f6ef400c520 comp rx=0 tx=0).stop 2024-03-18T16:14:21.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.221+0000 7f6f0d000640 1 -- 172.21.15.39:0/395994675 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f080ffaa0 msgr2=0x7f6f08195990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:14:21.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.221+0000 7f6f0d000640 1 --2- 172.21.15.39:0/395994675 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f080ffaa0 0x7f6f08195990 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f6f00009ca0 tx=0x7f6f00032ec0 comp rx=0 tx=0).stop 2024-03-18T16:14:21.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.221+0000 7f6f0d000640 1 -- 172.21.15.39:0/395994675 shutdown_connections 2024-03-18T16:14:21.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.221+0000 7f6f0d000640 1 --2- 172.21.15.39:0/395994675 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6f08102a70 0x7f6f08195ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:21.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.221+0000 7f6f0d000640 1 --2- 172.21.15.39:0/395994675 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f6eec0a2880 0x7f6eec0a4d40 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:21.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.222+0000 7f6f0d000640 1 --2- 172.21.15.39:0/395994675 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6f08103430 0x7f6f081ae360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:21.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.222+0000 7f6f0d000640 1 --2- 172.21.15.39:0/395994675 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6f080ffaa0 0x7f6f08195990 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:21.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.222+0000 7f6f0d000640 1 -- 172.21.15.39:0/395994675 >> 172.21.15.39:0/395994675 conn(0x7f6f080fb3d0 msgr2=0x7f6f08103c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:14:21.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.222+0000 7f6f0d000640 1 -- 172.21.15.39:0/395994675 shutdown_connections 2024-03-18T16:14:21.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:21.222+0000 7f6f0d000640 1 -- 172.21.15.39:0/395994675 wait complete. 2024-03-18T16:14:21.259 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-18T16:14:21.260 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:14:21.593 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:14:21 smithi039 ceph-mon[31721]: pgmap v550: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 41 MiB/s wr, 6.73k op/s 2024-03-18T16:14:21.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:14:21 smithi196 ceph-mon[36307]: pgmap v550: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 41 MiB/s wr, 6.73k op/s 2024-03-18T16:14:21.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:14:21 smithi194 ceph-mon[36200]: pgmap v550: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 41 MiB/s wr, 6.73k op/s 2024-03-18T16:14:22.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:22.783+0000 7f3c98a00640 1 -- 172.21.15.39:0/4124746943 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3c9410d850 msgr2=0x7f3c94118720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:14:22.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:22.783+0000 7f3c98a00640 1 --2- 172.21.15.39:0/4124746943 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3c9410d850 0x7f3c94118720 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f3c8c009be0 tx=0x7f3c8c02bf10 comp rx=0 tx=0).stop 2024-03-18T16:14:22.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:22.785+0000 7f3c98a00640 1 -- 172.21.15.39:0/4124746943 shutdown_connections 2024-03-18T16:14:22.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:22.785+0000 7f3c98a00640 1 --2- 172.21.15.39:0/4124746943 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3c9410d850 0x7f3c94118720 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:22.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:22.785+0000 7f3c98a00640 1 --2- 172.21.15.39:0/4124746943 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c9410adc0 0x7f3c9410d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:22.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:22.785+0000 7f3c98a00640 1 --2- 172.21.15.39:0/4124746943 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3c940735a0 0x7f3c9410a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:22.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:22.785+0000 7f3c98a00640 1 -- 172.21.15.39:0/4124746943 >> 172.21.15.39:0/4124746943 conn(0x7f3c9406e420 msgr2=0x7f3c94070840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:14:22.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:22.785+0000 7f3c98a00640 1 -- 172.21.15.39:0/4124746943 shutdown_connections 2024-03-18T16:14:22.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:22.785+0000 7f3c98a00640 1 -- 172.21.15.39:0/4124746943 wait complete. 2024-03-18T16:14:22.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:22.787+0000 7f3c98a00640 1 Processor -- start 2024-03-18T16:14:22.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:22.788+0000 7f3c98a00640 1 -- start start 2024-03-18T16:14:22.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:22.789+0000 7f3c98a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3c940735a0 0x7f3c94114e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:22.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:22.789+0000 7f3c98a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c9410adc0 0x7f3c94115360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:22.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:22.789+0000 7f3c93400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3c940735a0 0x7f3c94114e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:22.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:22.789+0000 7f3c93400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3c940735a0 0x7f3c94114e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:53198/0 (socket says 172.21.15.39:53198) 2024-03-18T16:14:22.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:22.791+0000 7f3c93400640 1 -- 172.21.15.39:0/4264817942 learned_addr learned my addr 172.21.15.39:0/4264817942 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:14:22.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:22.791+0000 7f3c98a00640 1 --2- 172.21.15.39:0/4264817942 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3c9410d850 0x7f3c941b6810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:22.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:22.791+0000 7f3c98a00640 1 -- 172.21.15.39:0/4264817942 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f3c9411c100 con 0x7f3c9410adc0 2024-03-18T16:14:22.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:22.791+0000 7f3c98a00640 1 -- 172.21.15.39:0/4264817942 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f3c9411bf80 con 0x7f3c9410d850 2024-03-18T16:14:22.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:22.791+0000 7f3c98a00640 1 -- 172.21.15.39:0/4264817942 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f3c9411c280 con 0x7f3c940735a0 2024-03-18T16:14:22.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:22.791+0000 7f3c92a00640 1 --2- 172.21.15.39:0/4264817942 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c9410adc0 0x7f3c94115360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:22.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:22.791+0000 7f3c93e00640 1 --2- 172.21.15.39:0/4264817942 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3c9410d850 0x7f3c941b6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:22.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:22.792+0000 7f3c93400640 1 -- 172.21.15.39:0/4264817942 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3c9410d850 msgr2=0x7f3c941b6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:14:22.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:22.792+0000 7f3c93400640 1 --2- 172.21.15.39:0/4264817942 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3c9410d850 0x7f3c941b6810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:22.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:22.792+0000 7f3c93400640 1 -- 172.21.15.39:0/4264817942 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c9410adc0 msgr2=0x7f3c94115360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:14:22.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:22.792+0000 7f3c93400640 1 --2- 172.21.15.39:0/4264817942 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c9410adc0 0x7f3c94115360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:22.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:22.792+0000 7f3c93400640 1 -- 172.21.15.39:0/4264817942 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c941159d0 con 0x7f3c940735a0 2024-03-18T16:14:22.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:22.792+0000 7f3c93e00640 1 --2- 172.21.15.39:0/4264817942 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3c9410d850 0x7f3c941b6810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:14:22.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:22.793+0000 7f3c93400640 1 --2- 172.21.15.39:0/4264817942 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3c940735a0 0x7f3c94114e20 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f3c8400e9c0 tx=0x7f3c8400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:14:22.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:22.793+0000 7f3c8be00640 1 -- 172.21.15.39:0/4264817942 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c8400ccc0 con 0x7f3c940735a0 2024-03-18T16:14:22.796 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:22.793+0000 7f3c8be00640 1 -- 172.21.15.39:0/4264817942 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c84005500 con 0x7f3c940735a0 2024-03-18T16:14:22.796 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:22.793+0000 7f3c8be00640 1 -- 172.21.15.39:0/4264817942 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c84010430 con 0x7f3c940735a0 2024-03-18T16:14:22.796 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:22.793+0000 7f3c98a00640 1 -- 172.21.15.39:0/4264817942 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c941b6e10 con 0x7f3c940735a0 2024-03-18T16:14:22.796 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:22.793+0000 7f3c98a00640 1 -- 172.21.15.39:0/4264817942 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c941b7240 con 0x7f3c940735a0 2024-03-18T16:14:22.797 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:22.795+0000 7f3c98a00640 1 -- 172.21.15.39:0/4264817942 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c9406ff10 con 0x7f3c940735a0 2024-03-18T16:14:22.801 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:22.800+0000 7f3c8be00640 1 -- 172.21.15.39:0/4264817942 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f3c840046c0 con 0x7f3c940735a0 2024-03-18T16:14:22.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:22.804+0000 7f3c8be00640 1 --2- 172.21.15.39:0/4264817942 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3c780a29d0 0x7f3c780a4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:22.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:22.804+0000 7f3c8be00640 1 -- 172.21.15.39:0/4264817942 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f3c840cf040 con 0x7f3c940735a0 2024-03-18T16:14:22.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:22.807+0000 7f3c92a00640 1 --2- 172.21.15.39:0/4264817942 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3c780a29d0 0x7f3c780a4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:22.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:22.808+0000 7f3c8be00640 1 -- 172.21.15.39:0/4264817942 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f3c8409bcf0 con 0x7f3c940735a0 2024-03-18T16:14:22.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:22.808+0000 7f3c92a00640 1 --2- 172.21.15.39:0/4264817942 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3c780a29d0 0x7f3c780a4e90 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f3c94116330 tx=0x7f3c800090e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:14:23.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.308+0000 7f3c98a00640 1 -- 172.21.15.39:0/4264817942 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3c94062440 con 0x7f3c940735a0 2024-03-18T16:14:23.310 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:14:23.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.309+0000 7f3c8be00640 1 -- 172.21.15.39:0/4264817942 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) ==== 94+0+11950 (secure 0 0 0) 0x7f3c8409b5f0 con 0x7f3c940735a0 2024-03-18T16:14:23.311 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2520226608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2520226608},{"type":"v1","addr":"172.21.15.39:6835","nonce":2520226608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/719341763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":719341763},{"type":"v1","addr":"172.21.15.39:6837","nonce":719341763}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":36,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T16:06:44.180425+0000","modified":"2024-03-18T16:13:04.655685+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24575,"mds_1":24556,"mds_2":24535,"mds_3":24524,"mds_4":14463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14445":{"gid":14445,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.39:6839/656513813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":656513813},{"type":"v1","addr":"172.21.15.39:6839","nonce":656513813}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14463":{"gid":14463,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":13,"addr":"172.21.15.39:6841/3515326042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3515326042},{"type":"v1","addr":"172.21.15.39:6841","nonce":3515326042}]},"join_fscid":-1,"export_targets":[3],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6833/931350738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":931350738},{"type":"v1","addr":"172.21.15.194:6833","nonce":931350738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6833/3874209621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":3874209621},{"type":"v1","addr":"172.21.15.196:6833","nonce":3874209621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6835/188385470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":188385470},{"type":"v1","addr":"172.21.15.194:6835","nonce":188385470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6835/3949984309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":3949984309},{"type":"v1","addr":"172.21.15.196:6835","nonce":3949984309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":15,"addr":"172.21.15.194:6837/1301464246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":1301464246},{"type":"v1","addr":"172.21.15.194:6837","nonce":1301464246}]},"join_fscid":-1,"export_targets":[1,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":16,"addr":"172.21.15.196:6837/895247251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":895247251},{"type":"v1","addr":"172.21.15.196:6837","nonce":895247251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.196:6839/2001427268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6838","nonce":2001427268},{"type":"v1","addr":"172.21.15.196:6839","nonce":2001427268}]},"join_fscid":-1,"export_targets":[0,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24575":{"gid":24575,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.194:6839/274911158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":274911158},{"type":"v1","addr":"172.21.15.194:6839","nonce":274911158}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24575,"qdb_cluster":[14463,24524,24575,24556,24535]},"id":1}]} 2024-03-18T16:14:23.311 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 36 2024-03-18T16:14:23.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.314+0000 7f3c98a00640 1 -- 172.21.15.39:0/4264817942 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3c780a29d0 msgr2=0x7f3c780a4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:14:23.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.314+0000 7f3c98a00640 1 --2- 172.21.15.39:0/4264817942 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3c780a29d0 0x7f3c780a4e90 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f3c94116330 tx=0x7f3c800090e0 comp rx=0 tx=0).stop 2024-03-18T16:14:23.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.314+0000 7f3c98a00640 1 -- 172.21.15.39:0/4264817942 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3c940735a0 msgr2=0x7f3c94114e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:14:23.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.314+0000 7f3c98a00640 1 --2- 172.21.15.39:0/4264817942 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3c940735a0 0x7f3c94114e20 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f3c8400e9c0 tx=0x7f3c8400ee80 comp rx=0 tx=0).stop 2024-03-18T16:14:23.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.314+0000 7f3c98a00640 1 -- 172.21.15.39:0/4264817942 shutdown_connections 2024-03-18T16:14:23.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.314+0000 7f3c98a00640 1 --2- 172.21.15.39:0/4264817942 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3c9410d850 0x7f3c941b6810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:23.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.314+0000 7f3c98a00640 1 --2- 172.21.15.39:0/4264817942 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3c780a29d0 0x7f3c780a4e90 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:23.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.314+0000 7f3c98a00640 1 --2- 172.21.15.39:0/4264817942 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c9410adc0 0x7f3c94115360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:23.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.314+0000 7f3c98a00640 1 --2- 172.21.15.39:0/4264817942 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3c940735a0 0x7f3c94114e20 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:23.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.314+0000 7f3c98a00640 1 -- 172.21.15.39:0/4264817942 >> 172.21.15.39:0/4264817942 conn(0x7f3c9406e420 msgr2=0x7f3c94108d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:14:23.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.314+0000 7f3c98a00640 1 -- 172.21.15.39:0/4264817942 shutdown_connections 2024-03-18T16:14:23.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.315+0000 7f3c98a00640 1 -- 172.21.15.39:0/4264817942 wait complete. 2024-03-18T16:14:23.347 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-18T16:14:23.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.672+0000 7f6d68600640 1 -- 172.21.15.39:0/1496008171 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6d601055b0 msgr2=0x7f6d6010be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:14:23.673 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.672+0000 7f6d68600640 1 --2- 172.21.15.39:0/1496008171 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6d601055b0 0x7f6d6010be40 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f6d58009ca0 tx=0x7f6d5802c100 comp rx=0 tx=0).stop 2024-03-18T16:14:23.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.672+0000 7f6d68600640 1 -- 172.21.15.39:0/1496008171 shutdown_connections 2024-03-18T16:14:23.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.672+0000 7f6d68600640 1 --2- 172.21.15.39:0/1496008171 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6d60104bf0 0x7f6d60105070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:23.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.672+0000 7f6d68600640 1 --2- 172.21.15.39:0/1496008171 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6d601055b0 0x7f6d6010be40 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:23.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.672+0000 7f6d68600640 1 --2- 172.21.15.39:0/1496008171 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6d60103a00 0x7f6d60103e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:23.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.672+0000 7f6d68600640 1 -- 172.21.15.39:0/1496008171 >> 172.21.15.39:0/1496008171 conn(0x7f6d600fef90 msgr2=0x7f6d601013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:14:23.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.673+0000 7f6d68600640 1 -- 172.21.15.39:0/1496008171 shutdown_connections 2024-03-18T16:14:23.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.673+0000 7f6d68600640 1 -- 172.21.15.39:0/1496008171 wait complete. 2024-03-18T16:14:23.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.674+0000 7f6d68600640 1 Processor -- start 2024-03-18T16:14:23.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.674+0000 7f6d68600640 1 -- start start 2024-03-18T16:14:23.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.674+0000 7f6d68600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6d60103a00 0x7f6d60199d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:23.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.675+0000 7f6d68600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6d60104bf0 0x7f6d6019a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:23.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.676+0000 7f6d68600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6d601055b0 0x7f6d601a1310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:23.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.676+0000 7f6d68600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f6d6010f6f0 con 0x7f6d60103a00 2024-03-18T16:14:23.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.676+0000 7f6d68600640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f6d6010f570 con 0x7f6d601055b0 2024-03-18T16:14:23.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.676+0000 7f6d68600640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f6d6010f870 con 0x7f6d60104bf0 2024-03-18T16:14:23.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.676+0000 7f6d66600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6d601055b0 0x7f6d601a1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:23.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.676+0000 7f6d66600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6d601055b0 0x7f6d601a1310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:48846/0 (socket says 172.21.15.39:48846) 2024-03-18T16:14:23.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.676+0000 7f6d66600640 1 -- 172.21.15.39:0/3743490969 learned_addr learned my addr 172.21.15.39:0/3743490969 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:14:23.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.676+0000 7f6d65200640 1 --2- 172.21.15.39:0/3743490969 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6d60104bf0 0x7f6d6019a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:23.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.676+0000 7f6d65c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6d60103a00 0x7f6d60199d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:23.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.677+0000 7f6d66600640 1 -- 172.21.15.39:0/3743490969 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6d60104bf0 msgr2=0x7f6d6019a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:14:23.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.677+0000 7f6d66600640 1 --2- 172.21.15.39:0/3743490969 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6d60104bf0 0x7f6d6019a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:23.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.677+0000 7f6d66600640 1 -- 172.21.15.39:0/3743490969 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6d60103a00 msgr2=0x7f6d60199d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:14:23.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.677+0000 7f6d66600640 1 --2- 172.21.15.39:0/3743490969 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6d60103a00 0x7f6d60199d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:23.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.677+0000 7f6d66600640 1 -- 172.21.15.39:0/3743490969 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d601a1980 con 0x7f6d601055b0 2024-03-18T16:14:23.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.677+0000 7f6d66600640 1 --2- 172.21.15.39:0/3743490969 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6d601055b0 0x7f6d601a1310 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f6d58009bf0 tx=0x7f6d58035d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:14:23.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.678+0000 7f6d5e000640 1 -- 172.21.15.39:0/3743490969 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d58039040 con 0x7f6d601055b0 2024-03-18T16:14:23.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.679+0000 7f6d5e000640 1 -- 172.21.15.39:0/3743490969 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d58044070 con 0x7f6d601055b0 2024-03-18T16:14:23.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.679+0000 7f6d68600640 1 -- 172.21.15.39:0/3743490969 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d601a1c10 con 0x7f6d601055b0 2024-03-18T16:14:23.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.679+0000 7f6d68600640 1 -- 172.21.15.39:0/3743490969 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d601a2100 con 0x7f6d601055b0 2024-03-18T16:14:23.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.679+0000 7f6d5e000640 1 -- 172.21.15.39:0/3743490969 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d580044e0 con 0x7f6d601055b0 2024-03-18T16:14:23.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.685+0000 7f6d5e000640 1 -- 172.21.15.39:0/3743490969 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f6d58004680 con 0x7f6d601055b0 2024-03-18T16:14:23.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.686+0000 7f6d6b3c5740 1 Processor -- start 2024-03-18T16:14:23.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.687+0000 7f6d6b3c5740 1 -- start start 2024-03-18T16:14:23.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.687+0000 7f6d6b3c5740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557f879b10d0 0x557f879b14f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:23.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.687+0000 7f6d6b3c5740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f879b1ac0 0x557f879b65c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:23.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.687+0000 7f6d6b3c5740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557f879b6c30 0x557f879b90c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:23.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.687+0000 7f6d6b3c5740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x557f879703b0 con 0x557f879b6c30 2024-03-18T16:14:23.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.687+0000 7f6d6b3c5740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x557f8790d450 con 0x557f879b1ac0 2024-03-18T16:14:23.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.687+0000 7f6d6b3c5740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x557f878b5cb0 con 0x557f879b10d0 2024-03-18T16:14:23.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.688+0000 7f6d65200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557f879b10d0 0x557f879b14f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:23.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.688+0000 7f6d65c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f879b1ac0 0x557f879b65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:23.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.688+0000 7f6d65c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f879b1ac0 0x557f879b65c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:48856/0 (socket says 172.21.15.39:48856) 2024-03-18T16:14:23.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.688+0000 7f6d65c00640 1 -- 172.21.15.39:0/582753304 learned_addr learned my addr 172.21.15.39:0/582753304 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:14:23.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.689+0000 7f6d65c00640 1 -- 172.21.15.39:0/582753304 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557f879b10d0 msgr2=0x557f879b14f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:14:23.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.689+0000 7f6d65c00640 1 --2- 172.21.15.39:0/582753304 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557f879b10d0 0x557f879b14f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:23.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.689+0000 7f6d65c00640 1 -- 172.21.15.39:0/582753304 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557f879b6c30 msgr2=0x557f879b90c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:14:23.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.689+0000 7f6d65c00640 1 --2- 172.21.15.39:0/582753304 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557f879b6c30 0x557f879b90c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:23.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.689+0000 7f6d65c00640 1 -- 172.21.15.39:0/582753304 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f878d71f0 con 0x557f879b1ac0 2024-03-18T16:14:23.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.689+0000 7f6d5e000640 1 --2- 172.21.15.39:0/3743490969 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f6d480a2a00 0x7f6d480a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:23.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.689+0000 7f6d5e000640 1 -- 172.21.15.39:0/3743490969 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f6d580f0d60 con 0x7f6d601055b0 2024-03-18T16:14:23.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.690+0000 7f6d65200640 1 --2- 172.21.15.39:0/3743490969 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f6d480a2a00 0x7f6d480a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:23.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.690+0000 7f6d65c00640 1 --2- 172.21.15.39:0/582753304 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f879b1ac0 0x557f879b65c0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f6d5400b7f0 tx=0x7f6d5400bcb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e094cb950f470c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:14:23.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.690+0000 7f6d52a00640 1 -- 172.21.15.39:0/582753304 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d5400d950 con 0x557f879b1ac0 2024-03-18T16:14:23.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.691+0000 7f6d52a00640 1 -- 172.21.15.39:0/582753304 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d54018070 con 0x557f879b1ac0 2024-03-18T16:14:23.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.691+0000 7f6d52a00640 1 -- 172.21.15.39:0/582753304 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d540123e0 con 0x557f879b1ac0 2024-03-18T16:14:23.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.692+0000 7f6d65200640 1 --2- 172.21.15.39:0/3743490969 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f6d480a2a00 0x7f6d480a4ec0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x557f879b2320 tx=0x7f6d4c013040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:14:23.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.692+0000 7f6d6b3c5740 1 -- 172.21.15.39:0/582753304 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f879b1ac0 msgr2=0x557f879b65c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:14:23.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.692+0000 7f6d6b3c5740 1 --2- 172.21.15.39:0/582753304 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f879b1ac0 0x557f879b65c0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f6d5400b7f0 tx=0x7f6d5400bcb0 comp rx=0 tx=0).stop 2024-03-18T16:14:23.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.692+0000 7f6d6b3c5740 1 -- 172.21.15.39:0/582753304 shutdown_connections 2024-03-18T16:14:23.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.692+0000 7f6d6b3c5740 1 --2- 172.21.15.39:0/582753304 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f879b1ac0 0x557f879b65c0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:23.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.692+0000 7f6d6b3c5740 1 --2- 172.21.15.39:0/582753304 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557f879b6c30 0x557f879b90c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:23.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.692+0000 7f6d6b3c5740 1 --2- 172.21.15.39:0/582753304 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557f879b10d0 0x557f879b14f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:23.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.692+0000 7f6d6b3c5740 1 -- 172.21.15.39:0/582753304 >> 172.21.15.39:0/582753304 conn(0x557f879ac640 msgr2=0x557f879aca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:14:23.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.693+0000 7f6d6b3c5740 1 -- 172.21.15.39:0/582753304 shutdown_connections 2024-03-18T16:14:23.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.693+0000 7f6d6b3c5740 1 -- 172.21.15.39:0/582753304 wait complete. 2024-03-18T16:14:23.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.693+0000 7f6d6b3c5740 1 -- start start 2024-03-18T16:14:23.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.694+0000 7f6d6b3c5740 1 Processor -- start 2024-03-18T16:14:23.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.695+0000 7f6d6b3c5740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557f879b10d0 0x557f879bb3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:23.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.695+0000 7f6d6b3c5740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f879b1ac0 0x557f879bb8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:23.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.695+0000 7f6d65c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557f879b10d0 0x557f879bb3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:23.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.695+0000 7f6d65c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557f879b10d0 0x557f879bb3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:53234/0 (socket says 172.21.15.39:53234) 2024-03-18T16:14:23.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.695+0000 7f6d65200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f879b1ac0 0x557f879bb8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:23.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.695+0000 7f6d65200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f879b1ac0 0x557f879bb8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:48862/0 (socket says 172.21.15.39:48862) 2024-03-18T16:14:23.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.695+0000 7f6d6b3c5740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557f879b6c30 0x557f879c2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:23.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.696+0000 7f6d65200640 1 -- 172.21.15.39:0/1435143142 learned_addr learned my addr 172.21.15.39:0/1435143142 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:14:23.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.696+0000 7f6d6b3c5740 1 -- 172.21.15.39:0/1435143142 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x557f878c9110 con 0x557f879b6c30 2024-03-18T16:14:23.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.696+0000 7f6d66600640 1 --2- 172.21.15.39:0/1435143142 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557f879b6c30 0x557f879c2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:23.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.696+0000 7f6d6b3c5740 1 -- 172.21.15.39:0/1435143142 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x557f879094f0 con 0x557f879b1ac0 2024-03-18T16:14:23.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.697+0000 7f6d6b3c5740 1 -- 172.21.15.39:0/1435143142 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x557f879b9be0 con 0x557f879b10d0 2024-03-18T16:14:23.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.697+0000 7f6d6b3c5740 10 client.0 _ll_register_callbacks cb 0x557f878b6b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:14:23.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.697+0000 7f6d65c00640 1 -- 172.21.15.39:0/1435143142 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f879b1ac0 msgr2=0x557f879bb8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:14:23.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.697+0000 7f6d65c00640 1 --2- 172.21.15.39:0/1435143142 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f879b1ac0 0x557f879bb8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:23.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.697+0000 7f6d65c00640 1 -- 172.21.15.39:0/1435143142 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557f879b6c30 msgr2=0x557f879c2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:14:23.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.697+0000 7f6d65c00640 1 --2- 172.21.15.39:0/1435143142 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557f879b6c30 0x557f879c2920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:23.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.697+0000 7f6d65c00640 1 -- 172.21.15.39:0/1435143142 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f876213f0 con 0x557f879b10d0 2024-03-18T16:14:23.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.698+0000 7f6d65c00640 1 --2- 172.21.15.39:0/1435143142 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557f879b10d0 0x557f879bb3a0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x557f879ad5b0 tx=0x7f6d540136b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d6a01bcb788497f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:14:23.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.698+0000 7f6d47400640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:14:23.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.698+0000 7f6d6b3c5740 1 -- 172.21.15.39:0/1435143142 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6d54018070 con 0x557f879b10d0 2024-03-18T16:14:23.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.698+0000 7f6d47400640 1 -- 172.21.15.39:0/1435143142 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d5400dd60 con 0x557f879b10d0 2024-03-18T16:14:23.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.698+0000 7f6d47400640 1 -- 172.21.15.39:0/1435143142 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f6d5401c420 con 0x557f879b10d0 2024-03-18T16:14:23.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.699+0000 7f6d47400640 1 -- 172.21.15.39:0/1435143142 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f6d5402d050 con 0x557f879b10d0 2024-03-18T16:14:23.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.699+0000 7f6d6b3c5740 10 client.26419 fetch_fsmap learned FSMap version 36 2024-03-18T16:14:23.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.699+0000 7f6d6b3c5740 1 -- 172.21.15.39:0/1435143142 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({fsmap=36}) -- 0x557f876eb7d0 con 0x557f879b10d0 2024-03-18T16:14:23.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.700+0000 7f6d47400640 1 -- 172.21.15.39:0/1435143142 <== mon.1 v2:172.21.15.196:3300/0 4 ==== fsmap(e 36) ==== 8406+0+0 (secure 0 0 0) 0x7f6d5401c620 con 0x557f879b10d0 2024-03-18T16:14:23.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.700+0000 7f6d6b3c5740 10 client.26419 fetch_fsmap finished waiting for FSMap version 36 2024-03-18T16:14:23.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.700+0000 7f6d6b3c5740 10 client.26419 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-03-18T16:14:23.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.700+0000 7f6d6b3c5740 20 client.26419 populate_metadata read hostname 'smithi039' 2024-03-18T16:14:23.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.700+0000 7f6d6b3c5740 1 --2- 172.21.15.39:0/1435143142 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x557f879c6d20 0x557f879c9110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:23.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.700+0000 7f6d6b3c5740 4 client.26419 mds_command: new command op to 14463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:14:23.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.700+0000 7f6d6b3c5740 1 -- 172.21.15.39:0/1435143142 --> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557f8791a990 con 0x557f879c6d20 2024-03-18T16:14:23.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.702+0000 7f6d65200640 1 --2- 172.21.15.39:0/1435143142 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x557f879c6d20 0x557f879c9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:23.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.703+0000 7f6d47400640 10 client.26419 ms_handle_connect on v2:172.21.15.39:6840/3515326042 2024-03-18T16:14:23.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.703+0000 7f6d65200640 1 --2- 172.21.15.39:0/1435143142 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x557f879c6d20 0x557f879c9110 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=b246b60fb79d94af server_cookie=234653532ef83679 in_seq=0 out_seq=0 2024-03-18T16:14:23.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.706+0000 7f6d47400640 1 -- 172.21.15.39:0/1435143142 <== mds.4 v2:172.21.15.39:6840/3515326042 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557f8791a990 con 0x557f879c6d20 2024-03-18T16:14:23.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.706+0000 7f6d47400640 10 client.26419 handle_command_reply: tid=0 2024-03-18T16:14:23.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.706+0000 7f6d6b3c5740 1 client.26419 shutdown 2024-03-18T16:14:23.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.707+0000 7f6d6b3c5740 1 -- 172.21.15.39:0/1435143142 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557f879b10d0 msgr2=0x557f879bb3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:14:23.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.707+0000 7f6d6b3c5740 1 --2- 172.21.15.39:0/1435143142 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557f879b10d0 0x557f879bb3a0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x557f879ad5b0 tx=0x7f6d540136b0 comp rx=0 tx=0).stop 2024-03-18T16:14:23.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.707+0000 7f6d6b3c5740 1 -- 172.21.15.39:0/1435143142 shutdown_connections 2024-03-18T16:14:23.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.707+0000 7f6d6b3c5740 1 --2- 172.21.15.39:0/1435143142 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x557f879c6d20 0x557f879c9110 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:23.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.707+0000 7f6d6b3c5740 1 --2- 172.21.15.39:0/1435143142 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f879b1ac0 0x557f879bb8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:23.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.707+0000 7f6d6b3c5740 1 --2- 172.21.15.39:0/1435143142 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557f879b6c30 0x557f879c2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:23.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.707+0000 7f6d6b3c5740 1 --2- 172.21.15.39:0/1435143142 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557f879b10d0 0x557f879bb3a0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:23.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.707+0000 7f6d6b3c5740 1 -- 172.21.15.39:0/1435143142 >> 172.21.15.39:0/1435143142 conn(0x557f879ac640 msgr2=0x557f879ad890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:14:23.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.707+0000 7f6d47400640 0 client.26419 ms_handle_reset on v2:172.21.15.39:6840/3515326042 2024-03-18T16:14:23.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.708+0000 7f6d6b3c5740 1 -- 172.21.15.39:0/1435143142 shutdown_connections 2024-03-18T16:14:23.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.708+0000 7f6d6b3c5740 1 -- 172.21.15.39:0/1435143142 wait complete. 2024-03-18T16:14:23.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.708+0000 7f6d6b3c5740 20 client.26419 trim_cache size 0 max 16384 2024-03-18T16:14:23.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:14:23 smithi196 ceph-mon[36307]: pgmap v551: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 36 MiB/s wr, 5.83k op/s 2024-03-18T16:14:23.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:14:23 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/4264817942' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:14:23.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.735+0000 7f6d6b3c5740 1 Processor -- start 2024-03-18T16:14:23.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.735+0000 7f6d6b3c5740 1 -- start start 2024-03-18T16:14:23.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.736+0000 7f6d6b3c5740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557f879b10d0 0x557f879a31f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:23.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.736+0000 7f6d6b3c5740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f879b1ac0 0x557f879ba670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:23.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.736+0000 7f6d6b3c5740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557f879b6c30 0x557f879babb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:23.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.736+0000 7f6d65200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f879b1ac0 0x557f879ba670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:23.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.736+0000 7f6d65200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f879b1ac0 0x557f879ba670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:48870/0 (socket says 172.21.15.39:48870) 2024-03-18T16:14:23.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.736+0000 7f6d65c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557f879b10d0 0x557f879a31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:23.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.736+0000 7f6d65c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557f879b10d0 0x557f879a31f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:53250/0 (socket says 172.21.15.39:53250) 2024-03-18T16:14:23.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.737+0000 7f6d65c00640 1 -- 172.21.15.39:0/3545171450 learned_addr learned my addr 172.21.15.39:0/3545171450 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:14:23.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.737+0000 7f6d6b3c5740 1 -- 172.21.15.39:0/3545171450 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f6d4000fb40 con 0x557f879b6c30 2024-03-18T16:14:23.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.737+0000 7f6d6b3c5740 1 -- 172.21.15.39:0/3545171450 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f6d40011800 con 0x557f879b1ac0 2024-03-18T16:14:23.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.737+0000 7f6d6b3c5740 1 -- 172.21.15.39:0/3545171450 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f6d40010210 con 0x557f879b10d0 2024-03-18T16:14:23.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.737+0000 7f6d66600640 1 --2- 172.21.15.39:0/3545171450 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557f879b6c30 0x557f879babb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:23.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.738+0000 7f6d65c00640 1 -- 172.21.15.39:0/3545171450 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f879b1ac0 msgr2=0x557f879ba670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:14:23.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.738+0000 7f6d65c00640 1 --2- 172.21.15.39:0/3545171450 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f879b1ac0 0x557f879ba670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:23.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.738+0000 7f6d65c00640 1 -- 172.21.15.39:0/3545171450 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557f879b6c30 msgr2=0x557f879babb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:14:23.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.738+0000 7f6d65c00640 1 --2- 172.21.15.39:0/3545171450 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557f879b6c30 0x557f879babb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:23.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.738+0000 7f6d65c00640 1 -- 172.21.15.39:0/3545171450 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557f8799fc10 con 0x557f879b10d0 2024-03-18T16:14:23.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:14:23 smithi194 ceph-mon[36200]: pgmap v551: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 36 MiB/s wr, 5.83k op/s 2024-03-18T16:14:23.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:14:23 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/4264817942' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:14:23.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.739+0000 7f6d65c00640 1 --2- 172.21.15.39:0/3545171450 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557f879b10d0 0x557f879a31f0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f6d54004810 tx=0x7f6d54020000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=920c0b3e613e9ceb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:14:23.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.739+0000 7f6d52000640 1 -- 172.21.15.39:0/3545171450 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d54036070 con 0x557f879b10d0 2024-03-18T16:14:23.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.740+0000 7f6d52000640 1 -- 172.21.15.39:0/3545171450 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d54018070 con 0x557f879b10d0 2024-03-18T16:14:23.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.740+0000 7f6d52000640 1 -- 172.21.15.39:0/3545171450 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d54031410 con 0x557f879b10d0 2024-03-18T16:14:23.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.741+0000 7f6d6b3c5740 1 -- 172.21.15.39:0/3545171450 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557f879b10d0 msgr2=0x557f879a31f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:14:23.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.741+0000 7f6d6b3c5740 1 --2- 172.21.15.39:0/3545171450 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557f879b10d0 0x557f879a31f0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f6d54004810 tx=0x7f6d54020000 comp rx=0 tx=0).stop 2024-03-18T16:14:23.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.741+0000 7f6d6b3c5740 1 -- 172.21.15.39:0/3545171450 shutdown_connections 2024-03-18T16:14:23.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.741+0000 7f6d6b3c5740 1 --2- 172.21.15.39:0/3545171450 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f879b1ac0 0x557f879ba670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:23.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.741+0000 7f6d6b3c5740 1 --2- 172.21.15.39:0/3545171450 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557f879b6c30 0x557f879babb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:23.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.741+0000 7f6d6b3c5740 1 --2- 172.21.15.39:0/3545171450 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557f879b10d0 0x557f879a31f0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:23.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.741+0000 7f6d6b3c5740 1 -- 172.21.15.39:0/3545171450 >> 172.21.15.39:0/3545171450 conn(0x557f879ac640 msgr2=0x557f8799f080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:14:23.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.742+0000 7f6d6b3c5740 1 -- 172.21.15.39:0/3545171450 shutdown_connections 2024-03-18T16:14:23.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.742+0000 7f6d6b3c5740 1 -- 172.21.15.39:0/3545171450 wait complete. 2024-03-18T16:14:23.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.743+0000 7f6d6b3c5740 1 -- start start 2024-03-18T16:14:23.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.743+0000 7f6d6b3c5740 1 Processor -- start 2024-03-18T16:14:23.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.744+0000 7f6d6b3c5740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f879b10d0 0x557f879bfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:23.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.744+0000 7f6d6b3c5740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557f879b1ac0 0x557f879c0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:23.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.744+0000 7f6d65c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f879b10d0 0x557f879bfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:23.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.744+0000 7f6d65c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f879b10d0 0x557f879bfb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:48878/0 (socket says 172.21.15.39:48878) 2024-03-18T16:14:23.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.745+0000 7f6d65c00640 1 -- 172.21.15.39:0/3107553083 learned_addr learned my addr 172.21.15.39:0/3107553083 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:14:23.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.745+0000 7f6d6b3c5740 1 --2- 172.21.15.39:0/3107553083 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557f879b6c30 0x557f879c0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:23.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.746+0000 7f6d66600640 1 --2- 172.21.15.39:0/3107553083 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557f879b6c30 0x557f879c0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:23.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.746+0000 7f6d6b3c5740 1 -- 172.21.15.39:0/3107553083 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f6d40011070 con 0x557f879b6c30 2024-03-18T16:14:23.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.746+0000 7f6d6b3c5740 1 -- 172.21.15.39:0/3107553083 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f6d400108e0 con 0x557f879b10d0 2024-03-18T16:14:23.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.746+0000 7f6d6b3c5740 1 -- 172.21.15.39:0/3107553083 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f6d4000db20 con 0x557f879b1ac0 2024-03-18T16:14:23.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.746+0000 7f6d6b3c5740 10 client.0 _ll_register_callbacks cb 0x7f6d5400b670 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:14:23.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.746+0000 7f6d65200640 1 --2- 172.21.15.39:0/3107553083 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557f879b1ac0 0x557f879c0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:23.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.747+0000 7f6d65c00640 1 -- 172.21.15.39:0/3107553083 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557f879b1ac0 msgr2=0x557f879c0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:14:23.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.747+0000 7f6d65c00640 1 --2- 172.21.15.39:0/3107553083 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557f879b1ac0 0x557f879c0060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:23.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.747+0000 7f6d65c00640 1 -- 172.21.15.39:0/3107553083 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557f879b6c30 msgr2=0x557f879c0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:14:23.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.747+0000 7f6d65c00640 1 --2- 172.21.15.39:0/3107553083 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557f879b6c30 0x557f879c0fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:23.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.747+0000 7f6d65c00640 1 -- 172.21.15.39:0/3107553083 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557f879b6910 con 0x557f879b10d0 2024-03-18T16:14:23.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.749+0000 7f6d65c00640 1 --2- 172.21.15.39:0/3107553083 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f879b10d0 0x557f879bfb20 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f6d5400bcf0 tx=0x7f6d5401cda0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=368574368d0c367a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:14:23.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.749+0000 7f6d47400640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:14:23.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.749+0000 7f6d47400640 1 -- 172.21.15.39:0/3107553083 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d54036070 con 0x557f879b10d0 2024-03-18T16:14:23.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.749+0000 7f6d47400640 1 -- 172.21.15.39:0/3107553083 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f6d5401c420 con 0x557f879b10d0 2024-03-18T16:14:23.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.750+0000 7f6d6b3c5740 1 -- 172.21.15.39:0/3107553083 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6d54018070 con 0x557f879b10d0 2024-03-18T16:14:23.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.750+0000 7f6d47400640 1 -- 172.21.15.39:0/3107553083 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f6d5401c620 con 0x557f879b10d0 2024-03-18T16:14:23.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.750+0000 7f6d6b3c5740 10 client.26438 fetch_fsmap learned FSMap version 36 2024-03-18T16:14:23.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.750+0000 7f6d6b3c5740 1 -- 172.21.15.39:0/3107553083 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x557f879c26d0 con 0x557f879b10d0 2024-03-18T16:14:23.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.751+0000 7f6d47400640 1 -- 172.21.15.39:0/3107553083 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6d54018070 con 0x557f879b10d0 2024-03-18T16:14:23.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.751+0000 7f6d47400640 1 -- 172.21.15.39:0/3107553083 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 36) ==== 8406+0+0 (secure 0 0 0) 0x7f6d54006540 con 0x557f879b10d0 2024-03-18T16:14:23.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.751+0000 7f6d6b3c5740 10 client.26438 fetch_fsmap finished waiting for FSMap version 36 2024-03-18T16:14:23.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.751+0000 7f6d6b3c5740 10 client.26438 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-03-18T16:14:23.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.751+0000 7f6d6b3c5740 20 client.26438 populate_metadata read hostname 'smithi039' 2024-03-18T16:14:23.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.751+0000 7f6d6b3c5740 1 --2- 172.21.15.39:0/3107553083 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x557f87a2d4f0 0x557f87a2f910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:23.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.752+0000 7f6d6b3c5740 4 client.26438 mds_command: new command op to 14463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:14:23.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.752+0000 7f6d6b3c5740 1 -- 172.21.15.39:0/3107553083 --> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557f87914ce0 con 0x557f87a2d4f0 2024-03-18T16:14:23.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.752+0000 7f6d65200640 1 --2- 172.21.15.39:0/3107553083 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x557f87a2d4f0 0x557f87a2f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:23.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.754+0000 7f6d65200640 1 --2- 172.21.15.39:0/3107553083 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x557f87a2d4f0 0x557f87a2f910 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=8f46a1550efb8014 server_cookie=524c6edde6d50130 in_seq=0 out_seq=0 2024-03-18T16:14:23.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.754+0000 7f6d47400640 10 client.26438 ms_handle_connect on v2:172.21.15.39:6840/3515326042 2024-03-18T16:14:23.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.754+0000 7f6d47400640 1 -- 172.21.15.39:0/3107553083 <== mds.4 v2:172.21.15.39:6840/3515326042 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557f87914ce0 con 0x557f87a2d4f0 2024-03-18T16:14:23.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.754+0000 7f6d47400640 10 client.26438 handle_command_reply: tid=0 2024-03-18T16:14:23.756 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.754+0000 7f6d6b3c5740 1 client.26438 shutdown 2024-03-18T16:14:23.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.755+0000 7f6d6b3c5740 1 -- 172.21.15.39:0/3107553083 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f879b10d0 msgr2=0x557f879bfb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:14:23.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.755+0000 7f6d6b3c5740 1 --2- 172.21.15.39:0/3107553083 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f879b10d0 0x557f879bfb20 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f6d5400bcf0 tx=0x7f6d5401cda0 comp rx=0 tx=0).stop 2024-03-18T16:14:23.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.756+0000 7f6d6b3c5740 1 -- 172.21.15.39:0/3107553083 shutdown_connections 2024-03-18T16:14:23.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.756+0000 7f6d6b3c5740 1 --2- 172.21.15.39:0/3107553083 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x557f87a2d4f0 0x557f87a2f910 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:23.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.756+0000 7f6d47400640 0 client.26438 ms_handle_reset on v2:172.21.15.39:6840/3515326042 2024-03-18T16:14:23.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.756+0000 7f6d6b3c5740 1 --2- 172.21.15.39:0/3107553083 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557f879b6c30 0x557f879c0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:23.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.756+0000 7f6d6b3c5740 1 --2- 172.21.15.39:0/3107553083 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557f879b1ac0 0x557f879c0060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:23.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.756+0000 7f6d6b3c5740 1 --2- 172.21.15.39:0/3107553083 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557f879b10d0 0x557f879bfb20 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:23.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.756+0000 7f6d6b3c5740 1 -- 172.21.15.39:0/3107553083 >> 172.21.15.39:0/3107553083 conn(0x557f879ac640 msgr2=0x557f879af8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:14:23.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.757+0000 7f6d6b3c5740 1 -- 172.21.15.39:0/3107553083 shutdown_connections 2024-03-18T16:14:23.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.757+0000 7f6d6b3c5740 1 -- 172.21.15.39:0/3107553083 wait complete. 2024-03-18T16:14:23.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.759+0000 7f6d6b3c5740 20 client.26438 trim_cache size 0 max 16384 2024-03-18T16:14:23.760 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:14:23.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.760+0000 7f6d68600640 1 -- 172.21.15.39:0/3743490969 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f6d480a2a00 msgr2=0x7f6d480a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:14:23.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.760+0000 7f6d68600640 1 --2- 172.21.15.39:0/3743490969 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f6d480a2a00 0x7f6d480a4ec0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x557f879b2320 tx=0x7f6d4c013040 comp rx=0 tx=0).stop 2024-03-18T16:14:23.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.760+0000 7f6d68600640 1 -- 172.21.15.39:0/3743490969 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6d601055b0 msgr2=0x7f6d601a1310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:14:23.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.760+0000 7f6d68600640 1 --2- 172.21.15.39:0/3743490969 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6d601055b0 0x7f6d601a1310 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f6d58009bf0 tx=0x7f6d58035d10 comp rx=0 tx=0).stop 2024-03-18T16:14:23.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.760+0000 7f6d68600640 1 -- 172.21.15.39:0/3743490969 shutdown_connections 2024-03-18T16:14:23.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.760+0000 7f6d68600640 1 --2- 172.21.15.39:0/3743490969 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6d601055b0 0x7f6d601a1310 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:23.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.760+0000 7f6d68600640 1 --2- 172.21.15.39:0/3743490969 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f6d480a2a00 0x7f6d480a4ec0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:23.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.760+0000 7f6d68600640 1 --2- 172.21.15.39:0/3743490969 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6d60104bf0 0x7f6d6019a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:23.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.760+0000 7f6d68600640 1 --2- 172.21.15.39:0/3743490969 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6d60103a00 0x7f6d60199d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:23.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.760+0000 7f6d68600640 1 -- 172.21.15.39:0/3743490969 >> 172.21.15.39:0/3743490969 conn(0x7f6d600fef90 msgr2=0x7f6d60107ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:14:23.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.760+0000 7f6d68600640 1 -- 172.21.15.39:0/3743490969 shutdown_connections 2024-03-18T16:14:23.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:23.761+0000 7f6d68600640 1 -- 172.21.15.39:0/3743490969 wait complete. 2024-03-18T16:14:23.792 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:14:23 smithi039 ceph-mon[31721]: pgmap v551: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 36 MiB/s wr, 5.83k op/s 2024-03-18T16:14:23.793 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:14:23 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/4264817942' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:14:23.799 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-18T16:14:24.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.120+0000 7f63b1400640 1 -- 172.21.15.39:0/2225045238 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f63ac10db00 msgr2=0x7f63ac10df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:14:24.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.120+0000 7f63b1400640 1 --2- 172.21.15.39:0/2225045238 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f63ac10db00 0x7f63ac10df80 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f6398009860 tx=0x7f639802bb90 comp rx=0 tx=0).stop 2024-03-18T16:14:24.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.120+0000 7f63b1400640 1 -- 172.21.15.39:0/2225045238 shutdown_connections 2024-03-18T16:14:24.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.120+0000 7f63b1400640 1 --2- 172.21.15.39:0/2225045238 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f63ac10b150 0x7f63ac10d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.124 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.120+0000 7f63b1400640 1 --2- 172.21.15.39:0/2225045238 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f63ac10db00 0x7f63ac10df80 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.124 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.120+0000 7f63b1400640 1 --2- 172.21.15.39:0/2225045238 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f63ac1087e0 0x7f63ac10ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.124 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.120+0000 7f63b1400640 1 -- 172.21.15.39:0/2225045238 >> 172.21.15.39:0/2225045238 conn(0x7f63ac06e860 msgr2=0x7f63ac070ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:14:24.124 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.120+0000 7f63b1400640 1 -- 172.21.15.39:0/2225045238 shutdown_connections 2024-03-18T16:14:24.124 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.121+0000 7f63b1400640 1 -- 172.21.15.39:0/2225045238 wait complete. 2024-03-18T16:14:24.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.122+0000 7f63b1400640 1 Processor -- start 2024-03-18T16:14:24.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.122+0000 7f63b1400640 1 -- start start 2024-03-18T16:14:24.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.123+0000 7f63b1400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f63ac1087e0 0x7f63ac19e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:24.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.123+0000 7f63b1400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f63ac10b150 0x7f63ac19e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:24.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.123+0000 7f63b1400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f63ac10db00 0x7f63ac1a5640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:24.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.124+0000 7f63b1400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f63ac111a40 con 0x7f63ac1087e0 2024-03-18T16:14:24.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.124+0000 7f63b1400640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f63ac1118c0 con 0x7f63ac10db00 2024-03-18T16:14:24.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.124+0000 7f63b1400640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f63ac111bc0 con 0x7f63ac10b150 2024-03-18T16:14:24.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.124+0000 7f63ab200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f63ac10db00 0x7f63ac1a5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:24.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.124+0000 7f63ab200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f63ac10db00 0x7f63ac1a5640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:48900/0 (socket says 172.21.15.39:48900) 2024-03-18T16:14:24.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.124+0000 7f63ab200640 1 -- 172.21.15.39:0/813981955 learned_addr learned my addr 172.21.15.39:0/813981955 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:14:24.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.124+0000 7f63aa800640 1 --2- 172.21.15.39:0/813981955 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f63ac1087e0 0x7f63ac19e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:24.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.124+0000 7f63a9e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f63ac10b150 0x7f63ac19e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:24.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.124+0000 7f63ab200640 1 -- 172.21.15.39:0/813981955 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f63ac10b150 msgr2=0x7f63ac19e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:14:24.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.124+0000 7f63ab200640 1 --2- 172.21.15.39:0/813981955 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f63ac10b150 0x7f63ac19e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.124+0000 7f63ab200640 1 -- 172.21.15.39:0/813981955 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f63ac1087e0 msgr2=0x7f63ac19e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:14:24.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.124+0000 7f63ab200640 1 --2- 172.21.15.39:0/813981955 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f63ac1087e0 0x7f63ac19e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.124+0000 7f63ab200640 1 -- 172.21.15.39:0/813981955 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63ac1a5cb0 con 0x7f63ac10db00 2024-03-18T16:14:24.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.124+0000 7f63a9e00640 1 --2- 172.21.15.39:0/813981955 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f63ac10b150 0x7f63ac19e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:14:24.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.124+0000 7f63aa800640 1 --2- 172.21.15.39:0/813981955 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f63ac1087e0 0x7f63ac19e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:14:24.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.125+0000 7f63ab200640 1 --2- 172.21.15.39:0/813981955 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f63ac10db00 0x7f63ac1a5640 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f639802c1a0 tx=0x7f6398004250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:14:24.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.125+0000 7f63a3400640 1 -- 172.21.15.39:0/813981955 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6398041020 con 0x7f63ac10db00 2024-03-18T16:14:24.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.125+0000 7f63a3400640 1 -- 172.21.15.39:0/813981955 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f639802ca10 con 0x7f63ac10db00 2024-03-18T16:14:24.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.126+0000 7f63a3400640 1 -- 172.21.15.39:0/813981955 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6398039040 con 0x7f63ac10db00 2024-03-18T16:14:24.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.126+0000 7f63b1400640 1 -- 172.21.15.39:0/813981955 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63ac1a5f40 con 0x7f63ac10db00 2024-03-18T16:14:24.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.127+0000 7f63b1400640 1 -- 172.21.15.39:0/813981955 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63ac1a6350 con 0x7f63ac10db00 2024-03-18T16:14:24.141 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.134+0000 7f63b4012740 1 Processor -- start 2024-03-18T16:14:24.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.139+0000 7f63a3400640 1 -- 172.21.15.39:0/813981955 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f6398044020 con 0x7f63ac10db00 2024-03-18T16:14:24.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.141+0000 7f63b4012740 1 -- start start 2024-03-18T16:14:24.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.142+0000 7f63b4012740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d2c6e688c0 0x55d2c6e68ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:24.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.142+0000 7f63b4012740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d2c6e692b0 0x55d2c6e6ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:24.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.142+0000 7f63b4012740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d2c6e6e420 0x55d2c6e708b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:24.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.142+0000 7f63aa800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d2c6e692b0 0x55d2c6e6ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:24.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.143+0000 7f63b4012740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55d2c6e303b0 con 0x55d2c6e688c0 2024-03-18T16:14:24.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.143+0000 7f63b4012740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55d2c6dcd450 con 0x55d2c6e6e420 2024-03-18T16:14:24.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.143+0000 7f63aa800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d2c6e692b0 0x55d2c6e6ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:53284/0 (socket says 172.21.15.39:53284) 2024-03-18T16:14:24.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.143+0000 7f63aa800640 1 -- 172.21.15.39:0/4259696713 learned_addr learned my addr 172.21.15.39:0/4259696713 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:14:24.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.143+0000 7f63b4012740 1 -- 172.21.15.39:0/4259696713 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55d2c6d75cb0 con 0x55d2c6e692b0 2024-03-18T16:14:24.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.143+0000 7f63a9e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d2c6e688c0 0x55d2c6e68ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:24.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.143+0000 7f63a3400640 1 --2- 172.21.15.39:0/813981955 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f63900a2950 0x7f63900a4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:24.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.143+0000 7f63a3400640 1 -- 172.21.15.39:0/813981955 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f63980f01e0 con 0x7f63ac10db00 2024-03-18T16:14:24.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.145+0000 7f63aa800640 1 -- 172.21.15.39:0/4259696713 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d2c6e6e420 msgr2=0x55d2c6e708b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:14:24.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.145+0000 7f63aa800640 1 --2- 172.21.15.39:0/4259696713 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d2c6e6e420 0x55d2c6e708b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.145+0000 7f63aa800640 1 -- 172.21.15.39:0/4259696713 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d2c6e688c0 msgr2=0x55d2c6e68ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:14:24.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.145+0000 7f63aa800640 1 --2- 172.21.15.39:0/4259696713 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d2c6e688c0 0x55d2c6e68ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.145+0000 7f63aa800640 1 -- 172.21.15.39:0/4259696713 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d2c6d971f0 con 0x55d2c6e692b0 2024-03-18T16:14:24.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.145+0000 7f63a9e00640 1 --2- 172.21.15.39:0/4259696713 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d2c6e688c0 0x55d2c6e68ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:14:24.148 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.144+0000 7f63ab200640 1 --2- 172.21.15.39:0/813981955 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f63900a2950 0x7f63900a4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:24.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.145+0000 7f63aa800640 1 --2- 172.21.15.39:0/4259696713 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d2c6e692b0 0x55d2c6e6ddb0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f63a40103a0 tx=0x7f63a4010860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6dd8d5b8360ca81b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:14:24.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.146+0000 7f6397400640 1 -- 172.21.15.39:0/4259696713 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63a4016070 con 0x55d2c6e692b0 2024-03-18T16:14:24.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.146+0000 7f63ab200640 1 --2- 172.21.15.39:0/813981955 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f63900a2950 0x7f63900a4e10 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f6398009780 tx=0x7f63980045c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:14:24.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.146+0000 7f6397400640 1 -- 172.21.15.39:0/4259696713 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f63a4012bf0 con 0x55d2c6e692b0 2024-03-18T16:14:24.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.148+0000 7f6397400640 1 -- 172.21.15.39:0/4259696713 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63a401bd50 con 0x55d2c6e692b0 2024-03-18T16:14:24.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.149+0000 7f63b4012740 1 -- 172.21.15.39:0/4259696713 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d2c6e692b0 msgr2=0x55d2c6e6ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:14:24.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.149+0000 7f63b4012740 1 --2- 172.21.15.39:0/4259696713 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d2c6e692b0 0x55d2c6e6ddb0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f63a40103a0 tx=0x7f63a4010860 comp rx=0 tx=0).stop 2024-03-18T16:14:24.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.149+0000 7f63b4012740 1 -- 172.21.15.39:0/4259696713 shutdown_connections 2024-03-18T16:14:24.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.149+0000 7f63b4012740 1 --2- 172.21.15.39:0/4259696713 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d2c6e6e420 0x55d2c6e708b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.149+0000 7f63b4012740 1 --2- 172.21.15.39:0/4259696713 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d2c6e692b0 0x55d2c6e6ddb0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.149+0000 7f63b4012740 1 --2- 172.21.15.39:0/4259696713 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d2c6e688c0 0x55d2c6e68ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.150+0000 7f63b4012740 1 -- 172.21.15.39:0/4259696713 >> 172.21.15.39:0/4259696713 conn(0x55d2c6e63e30 msgr2=0x55d2c6e64230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:14:24.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.150+0000 7f63b4012740 1 -- 172.21.15.39:0/4259696713 shutdown_connections 2024-03-18T16:14:24.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.150+0000 7f63b4012740 1 -- 172.21.15.39:0/4259696713 wait complete. 2024-03-18T16:14:24.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.151+0000 7f63b4012740 1 -- start start 2024-03-18T16:14:24.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.151+0000 7f63b4012740 1 Processor -- start 2024-03-18T16:14:24.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.152+0000 7f63b4012740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d2c6e688c0 0x55d2c6e72c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:24.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.152+0000 7f63b4012740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d2c6e692b0 0x55d2c6e73140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:24.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.152+0000 7f63b4012740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d2c6e6e420 0x55d2c6e7a180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:24.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.152+0000 7f63b4012740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55d2c6d89110 con 0x55d2c6e692b0 2024-03-18T16:14:24.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.152+0000 7f63b4012740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55d2c6dc94f0 con 0x55d2c6e6e420 2024-03-18T16:14:24.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.152+0000 7f63b4012740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55d2c6e71470 con 0x55d2c6e688c0 2024-03-18T16:14:24.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.153+0000 7f63a9e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d2c6e688c0 0x55d2c6e72c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:24.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.153+0000 7f63aa800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d2c6e692b0 0x55d2c6e73140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:24.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.153+0000 7f63aa800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d2c6e692b0 0x55d2c6e73140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:60464/0 (socket says 172.21.15.39:60464) 2024-03-18T16:14:24.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.153+0000 7f63aa800640 1 -- 172.21.15.39:0/4177734351 learned_addr learned my addr 172.21.15.39:0/4177734351 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:14:24.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.153+0000 7f63a9e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d2c6e688c0 0x55d2c6e72c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:53296/0 (socket says 172.21.15.39:53296) 2024-03-18T16:14:24.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.154+0000 7f63b4012740 10 client.0 _ll_register_callbacks cb 0x55d2c6d76b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:14:24.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.154+0000 7f63a9e00640 1 --2- 172.21.15.39:0/4177734351 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d2c6e6e420 0x55d2c6e7a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:24.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.156+0000 7f63aa800640 1 -- 172.21.15.39:0/4177734351 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d2c6e688c0 msgr2=0x55d2c6e72c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:14:24.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.156+0000 7f63aa800640 1 --2- 172.21.15.39:0/4177734351 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d2c6e688c0 0x55d2c6e72c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.156+0000 7f63aa800640 1 -- 172.21.15.39:0/4177734351 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d2c6e6e420 msgr2=0x55d2c6e7a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:14:24.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.156+0000 7f63aa800640 1 --2- 172.21.15.39:0/4177734351 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d2c6e6e420 0x55d2c6e7a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.156+0000 7f63aa800640 1 -- 172.21.15.39:0/4177734351 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d2c6ae13f0 con 0x55d2c6e692b0 2024-03-18T16:14:24.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.157+0000 7f63aa800640 1 --2- 172.21.15.39:0/4177734351 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d2c6e692b0 0x55d2c6e73140 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f63a4010370 tx=0x7f63a4009d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e0e5ba4c90cafa6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:14:24.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.157+0000 7f638be00640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-03-18T16:14:24.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.157+0000 7f638be00640 1 -- 172.21.15.39:0/4177734351 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63a4016040 con 0x55d2c6e692b0 2024-03-18T16:14:24.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.157+0000 7f638be00640 1 -- 172.21.15.39:0/4177734351 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f63a4004a40 con 0x55d2c6e692b0 2024-03-18T16:14:24.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.158+0000 7f63b4012740 1 -- 172.21.15.39:0/4177734351 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f63a4012bf0 con 0x55d2c6e692b0 2024-03-18T16:14:24.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.158+0000 7f638be00640 1 -- 172.21.15.39:0/4177734351 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f63a4004c40 con 0x55d2c6e692b0 2024-03-18T16:14:24.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.158+0000 7f63b4012740 10 client.16110 fetch_fsmap learned FSMap version 36 2024-03-18T16:14:24.161 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.158+0000 7f63b4012740 1 -- 172.21.15.39:0/4177734351 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55d2c6bab7d0 con 0x55d2c6e692b0 2024-03-18T16:14:24.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.159+0000 7f638be00640 1 -- 172.21.15.39:0/4177734351 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f63a4012bf0 con 0x55d2c6e692b0 2024-03-18T16:14:24.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.160+0000 7f638be00640 1 -- 172.21.15.39:0/4177734351 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 36) ==== 8406+0+0 (secure 0 0 0) 0x7f63a400f040 con 0x55d2c6e692b0 2024-03-18T16:14:24.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.160+0000 7f63b4012740 10 client.16110 fetch_fsmap finished waiting for FSMap version 36 2024-03-18T16:14:24.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.160+0000 7f63b4012740 10 client.16110 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-18T16:14:24.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.160+0000 7f63b4012740 20 client.16110 populate_metadata read hostname 'smithi039' 2024-03-18T16:14:24.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.160+0000 7f63b4012740 1 --2- 172.21.15.39:0/4177734351 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x55d2c6e7e750 0x55d2c6e80b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:24.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.160+0000 7f63b4012740 4 client.16110 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:14:24.163 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.160+0000 7f63b4012740 1 -- 172.21.15.39:0/4177734351 --> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d2c6dda990 con 0x55d2c6e7e750 2024-03-18T16:14:24.164 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.162+0000 7f63a9e00640 1 --2- 172.21.15.39:0/4177734351 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x55d2c6e7e750 0x55d2c6e80b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:24.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.162+0000 7f63a9e00640 1 --2- 172.21.15.39:0/4177734351 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x55d2c6e7e750 0x55d2c6e80b40 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=61554b31c823bce4 server_cookie=9eca321b5e18fcdf in_seq=0 out_seq=0 2024-03-18T16:14:24.165 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.162+0000 7f638be00640 10 client.16110 ms_handle_connect on v2:172.21.15.194:6836/1301464246 2024-03-18T16:14:24.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.164+0000 7f638be00640 1 -- 172.21.15.39:0/4177734351 <== mds.3 v2:172.21.15.194:6836/1301464246 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d2c6dda990 con 0x55d2c6e7e750 2024-03-18T16:14:24.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.164+0000 7f638be00640 10 client.16110 handle_command_reply: tid=0 2024-03-18T16:14:24.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.164+0000 7f63b4012740 1 client.16110 shutdown 2024-03-18T16:14:24.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.167+0000 7f63b4012740 1 -- 172.21.15.39:0/4177734351 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d2c6e692b0 msgr2=0x55d2c6e73140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:14:24.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.167+0000 7f63b4012740 1 --2- 172.21.15.39:0/4177734351 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d2c6e692b0 0x55d2c6e73140 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f63a4010370 tx=0x7f63a4009d70 comp rx=0 tx=0).stop 2024-03-18T16:14:24.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.167+0000 7f63b4012740 1 -- 172.21.15.39:0/4177734351 shutdown_connections 2024-03-18T16:14:24.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.168+0000 7f63b4012740 1 --2- 172.21.15.39:0/4177734351 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x55d2c6e7e750 0x55d2c6e80b40 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.168+0000 7f63b4012740 1 --2- 172.21.15.39:0/4177734351 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d2c6e6e420 0x55d2c6e7a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.168+0000 7f63b4012740 1 --2- 172.21.15.39:0/4177734351 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d2c6e692b0 0x55d2c6e73140 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.168+0000 7f63b4012740 1 --2- 172.21.15.39:0/4177734351 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d2c6e688c0 0x55d2c6e72c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.168+0000 7f63b4012740 1 -- 172.21.15.39:0/4177734351 >> 172.21.15.39:0/4177734351 conn(0x55d2c6e63e30 msgr2=0x55d2c6e631f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:14:24.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.168+0000 7f638be00640 0 client.16110 ms_handle_reset on v2:172.21.15.194:6836/1301464246 2024-03-18T16:14:24.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.171+0000 7f63b4012740 1 -- 172.21.15.39:0/4177734351 shutdown_connections 2024-03-18T16:14:24.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.171+0000 7f63b4012740 1 -- 172.21.15.39:0/4177734351 wait complete. 2024-03-18T16:14:24.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.171+0000 7f63b4012740 20 client.16110 trim_cache size 0 max 16384 2024-03-18T16:14:24.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.199+0000 7f63b4012740 1 Processor -- start 2024-03-18T16:14:24.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.199+0000 7f63b4012740 1 -- start start 2024-03-18T16:14:24.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.199+0000 7f63b4012740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d2c6e688c0 0x55d2c6e69b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:24.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.199+0000 7f63b4012740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d2c6e692b0 0x55d2c6e7a140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:24.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.199+0000 7f63b4012740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d2c6e6e420 0x55d2c6e63070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:24.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.199+0000 7f63b4012740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f63ac1b0b40 con 0x55d2c6e688c0 2024-03-18T16:14:24.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.199+0000 7f63b4012740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f63ac1b2800 con 0x55d2c6e692b0 2024-03-18T16:14:24.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.199+0000 7f63b4012740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f63ac1b1210 con 0x55d2c6e6e420 2024-03-18T16:14:24.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.200+0000 7f63a9e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d2c6e6e420 0x55d2c6e63070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:24.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.200+0000 7f63aa800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d2c6e692b0 0x55d2c6e7a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:24.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.200+0000 7f63a9e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d2c6e6e420 0x55d2c6e63070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:53312/0 (socket says 172.21.15.39:53312) 2024-03-18T16:14:24.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.200+0000 7f63a9e00640 1 -- 172.21.15.39:0/4291817384 learned_addr learned my addr 172.21.15.39:0/4291817384 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:14:24.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.200+0000 7f63aa800640 1 --2- 172.21.15.39:0/4291817384 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d2c6e692b0 0x55d2c6e7a140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:48922/0 (socket says 172.21.15.39:48922) 2024-03-18T16:14:24.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.200+0000 7f63a9e00640 1 --2- 172.21.15.39:0/4291817384 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d2c6e688c0 0x55d2c6e69b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:24.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.200+0000 7f63a9e00640 1 -- 172.21.15.39:0/4291817384 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d2c6e692b0 msgr2=0x55d2c6e7a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:14:24.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.200+0000 7f63a9e00640 1 --2- 172.21.15.39:0/4291817384 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d2c6e692b0 0x55d2c6e7a140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.200+0000 7f63a9e00640 1 -- 172.21.15.39:0/4291817384 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d2c6e688c0 msgr2=0x55d2c6e69b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:14:24.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.200+0000 7f63a9e00640 1 --2- 172.21.15.39:0/4291817384 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d2c6e688c0 0x55d2c6e69b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.201+0000 7f63a9e00640 1 -- 172.21.15.39:0/4291817384 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d2c6e5fc10 con 0x55d2c6e6e420 2024-03-18T16:14:24.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.201+0000 7f63a9e00640 1 --2- 172.21.15.39:0/4291817384 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d2c6e6e420 0x55d2c6e63070 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f639c005fd0 tx=0x7f639c01aba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7534e76cf3b96cfa server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:14:24.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.201+0000 7f6396a00640 1 -- 172.21.15.39:0/4291817384 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f639c039070 con 0x55d2c6e6e420 2024-03-18T16:14:24.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.201+0000 7f6396a00640 1 -- 172.21.15.39:0/4291817384 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f639c031bc0 con 0x55d2c6e6e420 2024-03-18T16:14:24.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.201+0000 7f6396a00640 1 -- 172.21.15.39:0/4291817384 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f639c039070 con 0x55d2c6e6e420 2024-03-18T16:14:24.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.202+0000 7f63b4012740 1 -- 172.21.15.39:0/4291817384 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d2c6e6e420 msgr2=0x55d2c6e63070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:14:24.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.202+0000 7f63b4012740 1 --2- 172.21.15.39:0/4291817384 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d2c6e6e420 0x55d2c6e63070 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f639c005fd0 tx=0x7f639c01aba0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.203+0000 7f63b4012740 1 -- 172.21.15.39:0/4291817384 shutdown_connections 2024-03-18T16:14:24.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.203+0000 7f63b4012740 1 --2- 172.21.15.39:0/4291817384 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d2c6e692b0 0x55d2c6e7a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.203+0000 7f63b4012740 1 --2- 172.21.15.39:0/4291817384 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d2c6e6e420 0x55d2c6e63070 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.203+0000 7f63b4012740 1 --2- 172.21.15.39:0/4291817384 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d2c6e688c0 0x55d2c6e69b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.203+0000 7f63b4012740 1 -- 172.21.15.39:0/4291817384 >> 172.21.15.39:0/4291817384 conn(0x55d2c6e63e30 msgr2=0x55d2c6e62960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:14:24.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.203+0000 7f63b4012740 1 -- 172.21.15.39:0/4291817384 shutdown_connections 2024-03-18T16:14:24.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.203+0000 7f63b4012740 1 -- 172.21.15.39:0/4291817384 wait complete. 2024-03-18T16:14:24.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.204+0000 7f63b4012740 1 -- start start 2024-03-18T16:14:24.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.204+0000 7f63b4012740 1 Processor -- start 2024-03-18T16:14:24.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.205+0000 7f63b4012740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d2c6e688c0 0x55d2c6e79c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:24.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.205+0000 7f63b4012740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d2c6e692b0 0x55d2c6e7a170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:24.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.205+0000 7f63b4012740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d2c6e6e420 0x55d2c6e782f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:24.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.205+0000 7f63b4012740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f63ac1b2070 con 0x55d2c6e6e420 2024-03-18T16:14:24.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.205+0000 7f63b4012740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f63ac1b18e0 con 0x55d2c6e688c0 2024-03-18T16:14:24.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.205+0000 7f63b4012740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f63ac1aeb20 con 0x55d2c6e692b0 2024-03-18T16:14:24.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.205+0000 7f63b4012740 10 client.0 _ll_register_callbacks cb 0x7f63a4009e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:14:24.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.206+0000 7f63a9e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d2c6e688c0 0x55d2c6e79c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:24.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.206+0000 7f63a9e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d2c6e688c0 0x55d2c6e79c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:48928/0 (socket says 172.21.15.39:48928) 2024-03-18T16:14:24.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.206+0000 7f63a9e00640 1 -- 172.21.15.39:0/4091997476 learned_addr learned my addr 172.21.15.39:0/4091997476 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:14:24.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.206+0000 7f63a9e00640 1 -- 172.21.15.39:0/4091997476 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d2c6e692b0 msgr2=0x55d2c6e7a170 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:14:24.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.206+0000 7f63a9e00640 1 --2- 172.21.15.39:0/4091997476 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d2c6e692b0 0x55d2c6e7a170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.206+0000 7f63a9e00640 1 -- 172.21.15.39:0/4091997476 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d2c6e6e420 msgr2=0x55d2c6e782f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:14:24.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.206+0000 7f63a9e00640 1 --2- 172.21.15.39:0/4091997476 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d2c6e6e420 0x55d2c6e782f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.206+0000 7f63a9e00640 1 -- 172.21.15.39:0/4091997476 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d2c6e64ba0 con 0x55d2c6e688c0 2024-03-18T16:14:24.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.207+0000 7f638be00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:14:24.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.207+0000 7f63a9e00640 1 --2- 172.21.15.39:0/4091997476 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d2c6e688c0 0x55d2c6e79c30 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f639c005780 tx=0x7f639c01ada0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b1ef516765d84220 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:14:24.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.207+0000 7f638be00640 1 -- 172.21.15.39:0/4091997476 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f639c043060 con 0x55d2c6e688c0 2024-03-18T16:14:24.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.207+0000 7f638be00640 1 -- 172.21.15.39:0/4091997476 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f639c02c040 con 0x55d2c6e688c0 2024-03-18T16:14:24.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.207+0000 7f63b4012740 1 -- 172.21.15.39:0/4091997476 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f639c031bc0 con 0x55d2c6e688c0 2024-03-18T16:14:24.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.211+0000 7f638be00640 1 -- 172.21.15.39:0/4091997476 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f639c043060 con 0x55d2c6e688c0 2024-03-18T16:14:24.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.211+0000 7f63b4012740 10 client.26453 fetch_fsmap learned FSMap version 36 2024-03-18T16:14:24.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.211+0000 7f63b4012740 1 -- 172.21.15.39:0/4091997476 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55d2c6ee5340 con 0x55d2c6e688c0 2024-03-18T16:14:24.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.211+0000 7f638be00640 1 -- 172.21.15.39:0/4091997476 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f639c03a040 con 0x55d2c6e688c0 2024-03-18T16:14:24.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.211+0000 7f638be00640 1 -- 172.21.15.39:0/4091997476 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 36) ==== 8406+0+0 (secure 0 0 0) 0x7f639c01f040 con 0x55d2c6e688c0 2024-03-18T16:14:24.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.211+0000 7f63b4012740 10 client.26453 fetch_fsmap finished waiting for FSMap version 36 2024-03-18T16:14:24.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.211+0000 7f63b4012740 10 client.26453 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-18T16:14:24.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.211+0000 7f63b4012740 20 client.26453 populate_metadata read hostname 'smithi039' 2024-03-18T16:14:24.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.211+0000 7f63b4012740 1 --2- 172.21.15.39:0/4091997476 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x55d2c6ee5c30 0x55d2c6ee8050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:24.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.211+0000 7f63b4012740 4 client.26453 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:14:24.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.212+0000 7f63b4012740 1 -- 172.21.15.39:0/4091997476 --> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f639c031bc0 con 0x55d2c6ee5c30 2024-03-18T16:14:24.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.212+0000 7f63aa800640 1 --2- 172.21.15.39:0/4091997476 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x55d2c6ee5c30 0x55d2c6ee8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:24.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.213+0000 7f63aa800640 1 --2- 172.21.15.39:0/4091997476 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x55d2c6ee5c30 0x55d2c6ee8050 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=10bd6c794543da72 server_cookie=1509b46c0c21c273 in_seq=0 out_seq=0 2024-03-18T16:14:24.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.213+0000 7f638be00640 10 client.26453 ms_handle_connect on v2:172.21.15.194:6836/1301464246 2024-03-18T16:14:24.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.213+0000 7f638be00640 1 -- 172.21.15.39:0/4091997476 <== mds.3 v2:172.21.15.194:6836/1301464246 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f639c031bc0 con 0x55d2c6ee5c30 2024-03-18T16:14:24.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.213+0000 7f638be00640 10 client.26453 handle_command_reply: tid=0 2024-03-18T16:14:24.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.213+0000 7f63b4012740 1 client.26453 shutdown 2024-03-18T16:14:24.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.216+0000 7f63b4012740 1 -- 172.21.15.39:0/4091997476 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d2c6e688c0 msgr2=0x55d2c6e79c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:14:24.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.216+0000 7f63b4012740 1 --2- 172.21.15.39:0/4091997476 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d2c6e688c0 0x55d2c6e79c30 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f639c005780 tx=0x7f639c01ada0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.217+0000 7f63b4012740 1 -- 172.21.15.39:0/4091997476 shutdown_connections 2024-03-18T16:14:24.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.217+0000 7f63b4012740 1 --2- 172.21.15.39:0/4091997476 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x55d2c6ee5c30 0x55d2c6ee8050 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.217+0000 7f63b4012740 1 --2- 172.21.15.39:0/4091997476 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d2c6e6e420 0x55d2c6e782f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.217+0000 7f63b4012740 1 --2- 172.21.15.39:0/4091997476 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d2c6e692b0 0x55d2c6e7a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.217+0000 7f63b4012740 1 --2- 172.21.15.39:0/4091997476 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d2c6e688c0 0x55d2c6e79c30 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.217+0000 7f63b4012740 1 -- 172.21.15.39:0/4091997476 >> 172.21.15.39:0/4091997476 conn(0x55d2c6e63e30 msgr2=0x55d2c6e631f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:14:24.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.217+0000 7f638be00640 0 client.26453 ms_handle_reset on v2:172.21.15.194:6836/1301464246 2024-03-18T16:14:24.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.217+0000 7f63b4012740 1 -- 172.21.15.39:0/4091997476 shutdown_connections 2024-03-18T16:14:24.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.217+0000 7f63b4012740 1 -- 172.21.15.39:0/4091997476 wait complete. 2024-03-18T16:14:24.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.217+0000 7f63b4012740 20 client.26453 trim_cache size 0 max 16384 2024-03-18T16:14:24.221 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:14:24.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.220+0000 7f63b1400640 1 -- 172.21.15.39:0/813981955 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f63900a2950 msgr2=0x7f63900a4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:14:24.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.220+0000 7f63b1400640 1 --2- 172.21.15.39:0/813981955 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f63900a2950 0x7f63900a4e10 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f6398009780 tx=0x7f63980045c0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.220+0000 7f63b1400640 1 -- 172.21.15.39:0/813981955 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f63ac10db00 msgr2=0x7f63ac1a5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:14:24.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.220+0000 7f63b1400640 1 --2- 172.21.15.39:0/813981955 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f63ac10db00 0x7f63ac1a5640 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f639802c1a0 tx=0x7f6398004250 comp rx=0 tx=0).stop 2024-03-18T16:14:24.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.220+0000 7f63b1400640 1 -- 172.21.15.39:0/813981955 shutdown_connections 2024-03-18T16:14:24.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.220+0000 7f63b1400640 1 --2- 172.21.15.39:0/813981955 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f63ac10db00 0x7f63ac1a5640 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.220+0000 7f63b1400640 1 --2- 172.21.15.39:0/813981955 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f63900a2950 0x7f63900a4e10 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.220+0000 7f63b1400640 1 --2- 172.21.15.39:0/813981955 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f63ac10b150 0x7f63ac19e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.220+0000 7f63b1400640 1 --2- 172.21.15.39:0/813981955 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f63ac1087e0 0x7f63ac19e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.221+0000 7f63b1400640 1 -- 172.21.15.39:0/813981955 >> 172.21.15.39:0/813981955 conn(0x7f63ac06e860 msgr2=0x7f63ac10b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:14:24.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.222+0000 7f63b1400640 1 -- 172.21.15.39:0/813981955 shutdown_connections 2024-03-18T16:14:24.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.222+0000 7f63b1400640 1 -- 172.21.15.39:0/813981955 wait complete. 2024-03-18T16:14:24.251 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-18T16:14:24.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.436+0000 7fb2c1600640 1 -- 172.21.15.39:0/1479810672 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb2bc101600 msgr2=0x7fb2bc101a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:14:24.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.436+0000 7fb2c1600640 1 --2- 172.21.15.39:0/1479810672 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb2bc101600 0x7fb2bc101a20 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fb2ac009900 tx=0x7fb2ac02bc50 comp rx=0 tx=0).stop 2024-03-18T16:14:24.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.436+0000 7fb2c1600640 1 -- 172.21.15.39:0/1479810672 shutdown_connections 2024-03-18T16:14:24.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.436+0000 7fb2c1600640 1 --2- 172.21.15.39:0/1479810672 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb2bc1027f0 0x7fb2bc102c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.436+0000 7fb2c1600640 1 --2- 172.21.15.39:0/1479810672 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb2bc1031b0 0x7fb2bc109a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.436+0000 7fb2c1600640 1 --2- 172.21.15.39:0/1479810672 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb2bc101600 0x7fb2bc101a20 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.436+0000 7fb2c1600640 1 -- 172.21.15.39:0/1479810672 >> 172.21.15.39:0/1479810672 conn(0x7fb2bc0fcb90 msgr2=0x7fb2bc0fefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:14:24.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.436+0000 7fb2c1600640 1 -- 172.21.15.39:0/1479810672 shutdown_connections 2024-03-18T16:14:24.440 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.436+0000 7fb2c1600640 1 -- 172.21.15.39:0/1479810672 wait complete. 2024-03-18T16:14:24.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.439+0000 7fb2c1600640 1 Processor -- start 2024-03-18T16:14:24.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.440+0000 7fb2c1600640 1 -- start start 2024-03-18T16:14:24.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.440+0000 7fb2c1600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb2bc101600 0x7fb2bc197900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:24.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.440+0000 7fb2c1600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb2bc1027f0 0x7fb2bc197e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:24.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.440+0000 7fb2c1600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb2bc1031b0 0x7fb2bc1a12b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:24.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.440+0000 7fb2c1600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fb2bc10d2c0 con 0x7fb2bc1027f0 2024-03-18T16:14:24.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.440+0000 7fb2c1600640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fb2bc10d140 con 0x7fb2bc1031b0 2024-03-18T16:14:24.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.440+0000 7fb2c1600640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fb2bc10d440 con 0x7fb2bc101600 2024-03-18T16:14:24.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.441+0000 7fb2bbe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb2bc101600 0x7fb2bc197900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:24.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.441+0000 7fb2bbe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb2bc101600 0x7fb2bc197900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:53324/0 (socket says 172.21.15.39:53324) 2024-03-18T16:14:24.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.441+0000 7fb2bbe00640 1 -- 172.21.15.39:0/103412976 learned_addr learned my addr 172.21.15.39:0/103412976 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:14:24.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.441+0000 7fb2c0c00640 1 --2- 172.21.15.39:0/103412976 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb2bc1031b0 0x7fb2bc1a12b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:24.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.441+0000 7fb2c0c00640 1 -- 172.21.15.39:0/103412976 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb2bc101600 msgr2=0x7fb2bc197900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:14:24.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.441+0000 7fb2c0c00640 1 --2- 172.21.15.39:0/103412976 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb2bc101600 0x7fb2bc197900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.441+0000 7fb2c0c00640 1 -- 172.21.15.39:0/103412976 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb2bc1027f0 msgr2=0x7fb2bc197e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:14:24.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.441+0000 7fb2c0c00640 1 --2- 172.21.15.39:0/103412976 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb2bc1027f0 0x7fb2bc197e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.441+0000 7fb2c0c00640 1 -- 172.21.15.39:0/103412976 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2bc19cea0 con 0x7fb2bc1031b0 2024-03-18T16:14:24.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.442+0000 7fb2c0c00640 1 --2- 172.21.15.39:0/103412976 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb2bc1031b0 0x7fb2bc1a12b0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fb2b400da10 tx=0x7fb2b400ded0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:14:24.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.443+0000 7fb2b8c00640 1 -- 172.21.15.39:0/103412976 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2b400bd70 con 0x7fb2bc1031b0 2024-03-18T16:14:24.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.443+0000 7fb2b8c00640 1 -- 172.21.15.39:0/103412976 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2b4003e50 con 0x7fb2bc1031b0 2024-03-18T16:14:24.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.443+0000 7fb2b8c00640 1 -- 172.21.15.39:0/103412976 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2b40106a0 con 0x7fb2bc1031b0 2024-03-18T16:14:24.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.443+0000 7fb2c1600640 1 -- 172.21.15.39:0/103412976 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2bc19d190 con 0x7fb2bc1031b0 2024-03-18T16:14:24.446 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.443+0000 7fb2c1600640 1 -- 172.21.15.39:0/103412976 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2bc19d690 con 0x7fb2bc1031b0 2024-03-18T16:14:24.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.447+0000 7fb2c59c1740 1 Processor -- start 2024-03-18T16:14:24.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.449+0000 7fb2b8c00640 1 -- 172.21.15.39:0/103412976 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fb2b4003970 con 0x7fb2bc1031b0 2024-03-18T16:14:24.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.450+0000 7fb2c59c1740 1 -- start start 2024-03-18T16:14:24.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.452+0000 7fb2c59c1740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5572c7b49a90 0x5572c7b49eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:24.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.452+0000 7fb2c59c1740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5572c7b4a480 0x5572c7b4ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:24.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.452+0000 7fb2c59c1740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5572c7b4f5f0 0x5572c7b51a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:24.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.452+0000 7fb2c59c1740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5572c7b113b0 con 0x5572c7b49a90 2024-03-18T16:14:24.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.452+0000 7fb2c59c1740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5572c7aae450 con 0x5572c7b4a480 2024-03-18T16:14:24.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.452+0000 7fb2c59c1740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x5572c7a56cb0 con 0x5572c7b4f5f0 2024-03-18T16:14:24.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.452+0000 7fb2bbe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5572c7b4a480 0x5572c7b4ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:24.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.452+0000 7fb2b8c00640 1 --2- 172.21.15.39:0/103412976 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb2a00a2950 0x7fb2a00a4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:24.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.453+0000 7fb2bbe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5572c7b4a480 0x5572c7b4ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:48946/0 (socket says 172.21.15.39:48946) 2024-03-18T16:14:24.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.453+0000 7fb2bbe00640 1 -- 172.21.15.39:0/3494392516 learned_addr learned my addr 172.21.15.39:0/3494392516 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:14:24.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.453+0000 7fb2b8c00640 1 -- 172.21.15.39:0/103412976 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fb2b40cf0f0 con 0x7fb2bc1031b0 2024-03-18T16:14:24.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.453+0000 7fb2bbe00640 1 -- 172.21.15.39:0/3494392516 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5572c7b4f5f0 msgr2=0x5572c7b51a80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:14:24.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.453+0000 7fb2bbe00640 1 --2- 172.21.15.39:0/3494392516 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5572c7b4f5f0 0x5572c7b51a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.453+0000 7fb2bbe00640 1 -- 172.21.15.39:0/3494392516 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5572c7b49a90 msgr2=0x5572c7b49eb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:14:24.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.454+0000 7fb2bbe00640 1 --2- 172.21.15.39:0/3494392516 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5572c7b49a90 0x5572c7b49eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.454+0000 7fb2bbe00640 1 -- 172.21.15.39:0/3494392516 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5572c7a781f0 con 0x5572c7b4a480 2024-03-18T16:14:24.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.454+0000 7fb2c0c00640 1 --2- 172.21.15.39:0/103412976 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb2a00a2950 0x7fb2a00a4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:24.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.454+0000 7fb2bbe00640 1 --2- 172.21.15.39:0/3494392516 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5572c7b4a480 0x5572c7b4ef80 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fb2ac036040 tx=0x7fb2ac0095e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ed4e8b2849ed771d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:14:24.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.454+0000 7fb2c0c00640 1 --2- 172.21.15.39:0/103412976 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb2a00a2950 0x7fb2a00a4e10 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fb2b4097b00 tx=0x7fb2b4005f30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:14:24.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.456+0000 7fb2b1200640 1 -- 172.21.15.39:0/3494392516 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2ac042070 con 0x5572c7b4a480 2024-03-18T16:14:24.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.456+0000 7fb2b1200640 1 -- 172.21.15.39:0/3494392516 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2ac02cba0 con 0x5572c7b4a480 2024-03-18T16:14:24.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.456+0000 7fb2b1200640 1 -- 172.21.15.39:0/3494392516 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2ac005130 con 0x5572c7b4a480 2024-03-18T16:14:24.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.456+0000 7fb2c59c1740 1 -- 172.21.15.39:0/3494392516 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5572c7b4a480 msgr2=0x5572c7b4ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:14:24.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.456+0000 7fb2c59c1740 1 --2- 172.21.15.39:0/3494392516 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5572c7b4a480 0x5572c7b4ef80 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fb2ac036040 tx=0x7fb2ac0095e0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.458+0000 7fb2c59c1740 1 -- 172.21.15.39:0/3494392516 shutdown_connections 2024-03-18T16:14:24.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.458+0000 7fb2c59c1740 1 --2- 172.21.15.39:0/3494392516 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5572c7b4a480 0x5572c7b4ef80 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.458+0000 7fb2c59c1740 1 --2- 172.21.15.39:0/3494392516 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5572c7b4f5f0 0x5572c7b51a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.458+0000 7fb2c59c1740 1 --2- 172.21.15.39:0/3494392516 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5572c7b49a90 0x5572c7b49eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.458+0000 7fb2c59c1740 1 -- 172.21.15.39:0/3494392516 >> 172.21.15.39:0/3494392516 conn(0x5572c7b45000 msgr2=0x5572c7b45400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:14:24.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.458+0000 7fb2c59c1740 1 -- 172.21.15.39:0/3494392516 shutdown_connections 2024-03-18T16:14:24.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.458+0000 7fb2c59c1740 1 -- 172.21.15.39:0/3494392516 wait complete. 2024-03-18T16:14:24.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.459+0000 7fb2c59c1740 1 -- start start 2024-03-18T16:14:24.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.460+0000 7fb2c59c1740 1 Processor -- start 2024-03-18T16:14:24.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.461+0000 7fb2c59c1740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5572c7b49a90 0x5572c7b53df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:24.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.461+0000 7fb2c59c1740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5572c7b4a480 0x5572c7b54330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:24.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.461+0000 7fb2c59c1740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5572c7b4f5f0 0x5572c7b5b330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:24.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.461+0000 7fb2c59c1740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5572c7a6a110 con 0x5572c7b4a480 2024-03-18T16:14:24.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.461+0000 7fb2c59c1740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5572c7aaa4f0 con 0x5572c7b49a90 2024-03-18T16:14:24.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.461+0000 7fb2c59c1740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x5572c7b52540 con 0x5572c7b4f5f0 2024-03-18T16:14:24.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.461+0000 7fb2c59c1740 10 client.0 _ll_register_callbacks cb 0x5572c7a57b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:14:24.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.461+0000 7fb2bb400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5572c7b49a90 0x5572c7b53df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:24.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.461+0000 7fb2bb400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5572c7b49a90 0x5572c7b53df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:48958/0 (socket says 172.21.15.39:48958) 2024-03-18T16:14:24.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.461+0000 7fb2bb400640 1 -- 172.21.15.39:0/1896765422 learned_addr learned my addr 172.21.15.39:0/1896765422 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:14:24.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.461+0000 7fb2bb400640 1 --2- 172.21.15.39:0/1896765422 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5572c7b4f5f0 0x5572c7b5b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:24.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.462+0000 7fb2bbe00640 1 --2- 172.21.15.39:0/1896765422 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5572c7b4a480 0x5572c7b54330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:24.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.462+0000 7fb2bb400640 1 -- 172.21.15.39:0/1896765422 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5572c7b4f5f0 msgr2=0x5572c7b5b330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:14:24.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.462+0000 7fb2bb400640 1 --2- 172.21.15.39:0/1896765422 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5572c7b4f5f0 0x5572c7b5b330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.462+0000 7fb2bb400640 1 -- 172.21.15.39:0/1896765422 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5572c7b4a480 msgr2=0x5572c7b54330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:14:24.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.462+0000 7fb2bb400640 1 --2- 172.21.15.39:0/1896765422 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5572c7b4a480 0x5572c7b54330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.462+0000 7fb2bb400640 1 -- 172.21.15.39:0/1896765422 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5572c77c23f0 con 0x5572c7b49a90 2024-03-18T16:14:24.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.462+0000 7fb2bb400640 1 --2- 172.21.15.39:0/1896765422 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5572c7b49a90 0x5572c7b53df0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fb2a8013ce0 tx=0x7fb2a8009610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b360cd6a6322920 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:14:24.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.463+0000 7fb2a5600640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:14:24.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.463+0000 7fb2a5600640 1 -- 172.21.15.39:0/1896765422 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2a80130d0 con 0x5572c7b49a90 2024-03-18T16:14:24.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.463+0000 7fb2a5600640 1 -- 172.21.15.39:0/1896765422 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fb2a8004050 con 0x5572c7b49a90 2024-03-18T16:14:24.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.463+0000 7fb2c59c1740 1 -- 172.21.15.39:0/1896765422 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb2ac02cba0 con 0x5572c7b49a90 2024-03-18T16:14:24.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.464+0000 7fb2a5600640 1 -- 172.21.15.39:0/1896765422 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fb2a8013270 con 0x5572c7b49a90 2024-03-18T16:14:24.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.464+0000 7fb2c59c1740 10 client.26465 fetch_fsmap learned FSMap version 36 2024-03-18T16:14:24.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.464+0000 7fb2c59c1740 1 -- 172.21.15.39:0/1896765422 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5572c788c7d0 con 0x5572c7b49a90 2024-03-18T16:14:24.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.464+0000 7fb2a5600640 1 -- 172.21.15.39:0/1896765422 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2ac02cba0 con 0x5572c7b49a90 2024-03-18T16:14:24.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.464+0000 7fb2a5600640 1 -- 172.21.15.39:0/1896765422 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 36) ==== 8406+0+0 (secure 0 0 0) 0x7fb2a8003c60 con 0x5572c7b49a90 2024-03-18T16:14:24.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.464+0000 7fb2c59c1740 10 client.26465 fetch_fsmap finished waiting for FSMap version 36 2024-03-18T16:14:24.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.464+0000 7fb2c59c1740 10 client.26465 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-18T16:14:24.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.464+0000 7fb2c59c1740 20 client.26465 populate_metadata read hostname 'smithi039' 2024-03-18T16:14:24.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.465+0000 7fb2c59c1740 1 --2- 172.21.15.39:0/1896765422 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x5572c7b70ca0 0x5572c7b73120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:24.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.465+0000 7fb2c59c1740 4 client.26465 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:14:24.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.465+0000 7fb2c59c1740 1 -- 172.21.15.39:0/1896765422 --> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5572c7abb990 con 0x5572c7b70ca0 2024-03-18T16:14:24.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.466+0000 7fb2bbe00640 1 --2- 172.21.15.39:0/1896765422 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x5572c7b70ca0 0x5572c7b73120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:24.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.469+0000 7fb2bbe00640 1 --2- 172.21.15.39:0/1896765422 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x5572c7b70ca0 0x5572c7b73120 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2dec7ab249258181 server_cookie=93e1fba7631e68b in_seq=0 out_seq=0 2024-03-18T16:14:24.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.469+0000 7fb2a5600640 10 client.26465 ms_handle_connect on v2:172.21.15.196:6836/895247251 2024-03-18T16:14:24.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.470+0000 7fb2a5600640 1 -- 172.21.15.39:0/1896765422 <== mds.2 v2:172.21.15.196:6836/895247251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5572c7abb990 con 0x5572c7b70ca0 2024-03-18T16:14:24.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.470+0000 7fb2a5600640 10 client.26465 handle_command_reply: tid=0 2024-03-18T16:14:24.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.470+0000 7fb2c59c1740 1 client.26465 shutdown 2024-03-18T16:14:24.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.471+0000 7fb2c59c1740 1 -- 172.21.15.39:0/1896765422 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5572c7b49a90 msgr2=0x5572c7b53df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:14:24.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.471+0000 7fb2c59c1740 1 --2- 172.21.15.39:0/1896765422 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5572c7b49a90 0x5572c7b53df0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fb2a8013ce0 tx=0x7fb2a8009610 comp rx=0 tx=0).stop 2024-03-18T16:14:24.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.471+0000 7fb2c59c1740 1 -- 172.21.15.39:0/1896765422 shutdown_connections 2024-03-18T16:14:24.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.471+0000 7fb2c59c1740 1 --2- 172.21.15.39:0/1896765422 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x5572c7b70ca0 0x5572c7b73120 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.471+0000 7fb2c59c1740 1 --2- 172.21.15.39:0/1896765422 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5572c7b4f5f0 0x5572c7b5b330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.471+0000 7fb2c59c1740 1 --2- 172.21.15.39:0/1896765422 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5572c7b4a480 0x5572c7b54330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.471+0000 7fb2c59c1740 1 --2- 172.21.15.39:0/1896765422 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5572c7b49a90 0x5572c7b53df0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.471+0000 7fb2c59c1740 1 -- 172.21.15.39:0/1896765422 >> 172.21.15.39:0/1896765422 conn(0x5572c7b45000 msgr2=0x5572c7b44220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:14:24.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.471+0000 7fb2a5600640 0 client.26465 ms_handle_reset on v2:172.21.15.196:6836/895247251 2024-03-18T16:14:24.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.474+0000 7fb2c59c1740 1 -- 172.21.15.39:0/1896765422 shutdown_connections 2024-03-18T16:14:24.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.474+0000 7fb2c59c1740 1 -- 172.21.15.39:0/1896765422 wait complete. 2024-03-18T16:14:24.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.474+0000 7fb2c59c1740 20 client.26465 trim_cache size 0 max 16384 2024-03-18T16:14:24.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.500+0000 7fb2c59c1740 1 Processor -- start 2024-03-18T16:14:24.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.500+0000 7fb2c59c1740 1 -- start start 2024-03-18T16:14:24.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.500+0000 7fb2c59c1740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5572c7b49a90 0x5572c7b537d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:24.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.500+0000 7fb2c59c1740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5572c7b4a480 0x5572c7b5b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:24.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.500+0000 7fb2c59c1740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5572c7b4f5f0 0x5572c7b43990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:24.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.500+0000 7fb2c59c1740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fb29c00fc00 con 0x5572c7b4a480 2024-03-18T16:14:24.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.500+0000 7fb2c59c1740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fb29c0118c0 con 0x5572c7b4f5f0 2024-03-18T16:14:24.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.500+0000 7fb2c59c1740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fb29c0102d0 con 0x5572c7b49a90 2024-03-18T16:14:24.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.501+0000 7fb2bbe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5572c7b4a480 0x5572c7b5b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:24.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.501+0000 7fb2bbe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5572c7b4a480 0x5572c7b5b2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:60530/0 (socket says 172.21.15.39:60530) 2024-03-18T16:14:24.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.501+0000 7fb2bbe00640 1 -- 172.21.15.39:0/3371968022 learned_addr learned my addr 172.21.15.39:0/3371968022 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:14:24.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.501+0000 7fb2bb400640 1 --2- 172.21.15.39:0/3371968022 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5572c7b49a90 0x5572c7b537d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:24.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.501+0000 7fb2bb400640 1 --2- 172.21.15.39:0/3371968022 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5572c7b4f5f0 0x5572c7b43990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:24.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.501+0000 7fb2bbe00640 1 -- 172.21.15.39:0/3371968022 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5572c7b49a90 msgr2=0x5572c7b537d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:14:24.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.501+0000 7fb2bbe00640 1 --2- 172.21.15.39:0/3371968022 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5572c7b49a90 0x5572c7b537d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.501+0000 7fb2bbe00640 1 -- 172.21.15.39:0/3371968022 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5572c7b4f5f0 msgr2=0x5572c7b43990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:14:24.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.501+0000 7fb2bbe00640 1 --2- 172.21.15.39:0/3371968022 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5572c7b4f5f0 0x5572c7b43990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.501+0000 7fb2bbe00640 1 -- 172.21.15.39:0/3371968022 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5572c7b44e10 con 0x5572c7b4a480 2024-03-18T16:14:24.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.502+0000 7fb2bbe00640 1 --2- 172.21.15.39:0/3371968022 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5572c7b4a480 0x5572c7b5b2f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fb2ac005ab0 tx=0x7fb2ac005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d7fe7a2da83bace server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:14:24.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.502+0000 7fb2a7e00640 1 -- 172.21.15.39:0/3371968022 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2ac042040 con 0x5572c7b4a480 2024-03-18T16:14:24.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.502+0000 7fb2a7e00640 1 -- 172.21.15.39:0/3371968022 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2ac02cba0 con 0x5572c7b4a480 2024-03-18T16:14:24.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.502+0000 7fb2a7e00640 1 -- 172.21.15.39:0/3371968022 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2ac0046c0 con 0x5572c7b4a480 2024-03-18T16:14:24.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.503+0000 7fb2c59c1740 1 -- 172.21.15.39:0/3371968022 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5572c7b4a480 msgr2=0x5572c7b5b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:14:24.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.503+0000 7fb2c59c1740 1 --2- 172.21.15.39:0/3371968022 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5572c7b4a480 0x5572c7b5b2f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fb2ac005ab0 tx=0x7fb2ac005fd0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.503+0000 7fb2c59c1740 1 -- 172.21.15.39:0/3371968022 shutdown_connections 2024-03-18T16:14:24.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.503+0000 7fb2c59c1740 1 --2- 172.21.15.39:0/3371968022 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5572c7b4f5f0 0x5572c7b43990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.503+0000 7fb2c59c1740 1 --2- 172.21.15.39:0/3371968022 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5572c7b4a480 0x5572c7b5b2f0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.503+0000 7fb2c59c1740 1 --2- 172.21.15.39:0/3371968022 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5572c7b49a90 0x5572c7b537d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.503+0000 7fb2c59c1740 1 -- 172.21.15.39:0/3371968022 >> 172.21.15.39:0/3371968022 conn(0x5572c7b45000 msgr2=0x5572c7b530c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:14:24.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.503+0000 7fb2c59c1740 1 -- 172.21.15.39:0/3371968022 shutdown_connections 2024-03-18T16:14:24.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.503+0000 7fb2c59c1740 1 -- 172.21.15.39:0/3371968022 wait complete. 2024-03-18T16:14:24.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.504+0000 7fb2c59c1740 1 -- start start 2024-03-18T16:14:24.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.504+0000 7fb2c59c1740 1 Processor -- start 2024-03-18T16:14:24.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.505+0000 7fb2c59c1740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5572c7b49a90 0x5572c7b58ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:24.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.505+0000 7fb2c59c1740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5572c7b4a480 0x5572c7b59400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:24.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.505+0000 7fb2c59c1740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5572c7b4f5f0 0x5572c7b5a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:24.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.505+0000 7fb2c59c1740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fb29c011130 con 0x5572c7b49a90 2024-03-18T16:14:24.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.505+0000 7fb2c59c1740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fb29c0109a0 con 0x5572c7b4f5f0 2024-03-18T16:14:24.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.505+0000 7fb2c59c1740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fb29c00dbe0 con 0x5572c7b4a480 2024-03-18T16:14:24.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.505+0000 7fb2c59c1740 10 client.0 _ll_register_callbacks cb 0x7fb2a8009940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:14:24.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.505+0000 7fb2bbe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5572c7b4a480 0x5572c7b59400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:24.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.505+0000 7fb2bbe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5572c7b4a480 0x5572c7b59400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:53346/0 (socket says 172.21.15.39:53346) 2024-03-18T16:14:24.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.505+0000 7fb2bbe00640 1 -- 172.21.15.39:0/2969970639 learned_addr learned my addr 172.21.15.39:0/2969970639 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:14:24.508 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.505+0000 7fb2bb400640 1 --2- 172.21.15.39:0/2969970639 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5572c7b49a90 0x5572c7b58ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:24.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.506+0000 7fb2bbe00640 1 -- 172.21.15.39:0/2969970639 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5572c7b4f5f0 msgr2=0x5572c7b5a410 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T16:14:24.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.506+0000 7fb2bbe00640 1 --2- 172.21.15.39:0/2969970639 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5572c7b4f5f0 0x5572c7b5a410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.506+0000 7fb2bbe00640 1 -- 172.21.15.39:0/2969970639 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5572c7b49a90 msgr2=0x5572c7b58ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:14:24.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.508+0000 7fb2bbe00640 1 --2- 172.21.15.39:0/2969970639 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5572c7b49a90 0x5572c7b58ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.508+0000 7fb2bbe00640 1 -- 172.21.15.39:0/2969970639 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5572c7b5aa00 con 0x5572c7b4a480 2024-03-18T16:14:24.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.509+0000 7fb2bb400640 1 --2- 172.21.15.39:0/2969970639 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5572c7b49a90 0x5572c7b58ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:14:24.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.509+0000 7fb2bbe00640 1 --2- 172.21.15.39:0/2969970639 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5572c7b4a480 0x5572c7b59400 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fb2ac02ced0 tx=0x7fb2ac0058d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ba135194ac57a021 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:14:24.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.509+0000 7fb2a5600640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:14:24.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.509+0000 7fb2a5600640 1 -- 172.21.15.39:0/2969970639 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2ac042070 con 0x5572c7b4a480 2024-03-18T16:14:24.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.510+0000 7fb2a5600640 1 -- 172.21.15.39:0/2969970639 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fb2ac050d20 con 0x5572c7b4a480 2024-03-18T16:14:24.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.510+0000 7fb2c59c1740 1 -- 172.21.15.39:0/2969970639 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb2ac02cba0 con 0x5572c7b4a480 2024-03-18T16:14:24.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.510+0000 7fb2a5600640 1 -- 172.21.15.39:0/2969970639 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7fb2ac004490 con 0x5572c7b4a480 2024-03-18T16:14:24.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.510+0000 7fb2c59c1740 10 client.26452 fetch_fsmap learned FSMap version 36 2024-03-18T16:14:24.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.510+0000 7fb2c59c1740 1 -- 172.21.15.39:0/2969970639 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5572c7bd7540 con 0x5572c7b4a480 2024-03-18T16:14:24.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.511+0000 7fb2a5600640 1 -- 172.21.15.39:0/2969970639 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2ac02cba0 con 0x5572c7b4a480 2024-03-18T16:14:24.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.511+0000 7fb2a5600640 1 -- 172.21.15.39:0/2969970639 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 36) ==== 8406+0+0 (secure 0 0 0) 0x7fb2ac004fe0 con 0x5572c7b4a480 2024-03-18T16:14:24.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.511+0000 7fb2c59c1740 10 client.26452 fetch_fsmap finished waiting for FSMap version 36 2024-03-18T16:14:24.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.511+0000 7fb2c59c1740 10 client.26452 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-18T16:14:24.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.511+0000 7fb2c59c1740 20 client.26452 populate_metadata read hostname 'smithi039' 2024-03-18T16:14:24.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.511+0000 7fb2c59c1740 1 --2- 172.21.15.39:0/2969970639 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x5572c7bd7a60 0x5572c7bd9e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:24.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.511+0000 7fb2c59c1740 4 client.26452 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:14:24.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.511+0000 7fb2c59c1740 1 -- 172.21.15.39:0/2969970639 --> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5572c7ab5ce0 con 0x5572c7bd7a60 2024-03-18T16:14:24.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.514+0000 7fb2bb400640 1 --2- 172.21.15.39:0/2969970639 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x5572c7bd7a60 0x5572c7bd9e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:24.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.514+0000 7fb2bb400640 1 --2- 172.21.15.39:0/2969970639 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x5572c7bd7a60 0x5572c7bd9e80 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2e69d61713265a8e server_cookie=431ad0001e316b6e in_seq=0 out_seq=0 2024-03-18T16:14:24.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.514+0000 7fb2a5600640 10 client.26452 ms_handle_connect on v2:172.21.15.196:6836/895247251 2024-03-18T16:14:24.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.515+0000 7fb2a5600640 1 -- 172.21.15.39:0/2969970639 <== mds.2 v2:172.21.15.196:6836/895247251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5572c7ab5ce0 con 0x5572c7bd7a60 2024-03-18T16:14:24.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.515+0000 7fb2a5600640 10 client.26452 handle_command_reply: tid=0 2024-03-18T16:14:24.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.516+0000 7fb2c59c1740 1 client.26452 shutdown 2024-03-18T16:14:24.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.518+0000 7fb2c59c1740 1 -- 172.21.15.39:0/2969970639 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5572c7b4a480 msgr2=0x5572c7b59400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:14:24.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.518+0000 7fb2c59c1740 1 --2- 172.21.15.39:0/2969970639 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5572c7b4a480 0x5572c7b59400 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fb2ac02ced0 tx=0x7fb2ac0058d0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.518+0000 7fb2c59c1740 1 -- 172.21.15.39:0/2969970639 shutdown_connections 2024-03-18T16:14:24.522 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.518+0000 7fb2c59c1740 1 --2- 172.21.15.39:0/2969970639 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5572c7b4f5f0 0x5572c7b5a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.522 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.519+0000 7fb2c59c1740 1 --2- 172.21.15.39:0/2969970639 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x5572c7bd7a60 0x5572c7bd9e80 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.522 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.519+0000 7fb2c59c1740 1 --2- 172.21.15.39:0/2969970639 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5572c7b4a480 0x5572c7b59400 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.522 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.519+0000 7fb2c59c1740 1 --2- 172.21.15.39:0/2969970639 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5572c7b49a90 0x5572c7b58ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.522 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.519+0000 7fb2c59c1740 1 -- 172.21.15.39:0/2969970639 >> 172.21.15.39:0/2969970639 conn(0x5572c7b45000 msgr2=0x5572c7b48260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:14:24.522 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.519+0000 7fb2a5600640 0 client.26452 ms_handle_reset on v2:172.21.15.196:6836/895247251 2024-03-18T16:14:24.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.521+0000 7fb2c59c1740 1 -- 172.21.15.39:0/2969970639 shutdown_connections 2024-03-18T16:14:24.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.522+0000 7fb2c59c1740 1 -- 172.21.15.39:0/2969970639 wait complete. 2024-03-18T16:14:24.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.522+0000 7fb2c59c1740 20 client.26452 trim_cache size 0 max 16384 2024-03-18T16:14:24.527 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:14:24.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.526+0000 7fb2c1600640 1 -- 172.21.15.39:0/103412976 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb2a00a2950 msgr2=0x7fb2a00a4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:14:24.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.526+0000 7fb2c1600640 1 --2- 172.21.15.39:0/103412976 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb2a00a2950 0x7fb2a00a4e10 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fb2b4097b00 tx=0x7fb2b4005f30 comp rx=0 tx=0).stop 2024-03-18T16:14:24.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.526+0000 7fb2c1600640 1 -- 172.21.15.39:0/103412976 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb2bc1031b0 msgr2=0x7fb2bc1a12b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:14:24.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.526+0000 7fb2c1600640 1 --2- 172.21.15.39:0/103412976 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb2bc1031b0 0x7fb2bc1a12b0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fb2b400da10 tx=0x7fb2b400ded0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.526+0000 7fb2c1600640 1 -- 172.21.15.39:0/103412976 shutdown_connections 2024-03-18T16:14:24.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.526+0000 7fb2c1600640 1 --2- 172.21.15.39:0/103412976 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb2bc1031b0 0x7fb2bc1a12b0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.526+0000 7fb2c1600640 1 --2- 172.21.15.39:0/103412976 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb2a00a2950 0x7fb2a00a4e10 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.526+0000 7fb2c1600640 1 --2- 172.21.15.39:0/103412976 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb2bc1027f0 0x7fb2bc197e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.526+0000 7fb2c1600640 1 --2- 172.21.15.39:0/103412976 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb2bc101600 0x7fb2bc197900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.526+0000 7fb2c1600640 1 -- 172.21.15.39:0/103412976 >> 172.21.15.39:0/103412976 conn(0x7fb2bc0fcb90 msgr2=0x7fb2bc105bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:14:24.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.528+0000 7fb2c1600640 1 -- 172.21.15.39:0/103412976 shutdown_connections 2024-03-18T16:14:24.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.528+0000 7fb2c1600640 1 -- 172.21.15.39:0/103412976 wait complete. 2024-03-18T16:14:24.556 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-18T16:14:24.871 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.868+0000 7f0034000640 1 -- 172.21.15.39:0/1696823208 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f002c1087e0 msgr2=0x7f002c10ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:14:24.871 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.868+0000 7f0034000640 1 --2- 172.21.15.39:0/1696823208 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f002c1087e0 0x7f002c10ac10 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f0018009860 tx=0x7f001802bb70 comp rx=0 tx=0).stop 2024-03-18T16:14:24.872 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.869+0000 7f0034000640 1 -- 172.21.15.39:0/1696823208 shutdown_connections 2024-03-18T16:14:24.872 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.869+0000 7f0034000640 1 --2- 172.21.15.39:0/1696823208 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f002c10db00 0x7f002c10df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.872 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.869+0000 7f0034000640 1 --2- 172.21.15.39:0/1696823208 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f002c10b150 0x7f002c10d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.872 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.869+0000 7f0034000640 1 --2- 172.21.15.39:0/1696823208 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f002c1087e0 0x7f002c10ac10 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.872 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.869+0000 7f0034000640 1 -- 172.21.15.39:0/1696823208 >> 172.21.15.39:0/1696823208 conn(0x7f002c06e860 msgr2=0x7f002c070ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:14:24.872 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.869+0000 7f0034000640 1 -- 172.21.15.39:0/1696823208 shutdown_connections 2024-03-18T16:14:24.872 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.869+0000 7f0034000640 1 -- 172.21.15.39:0/1696823208 wait complete. 2024-03-18T16:14:24.873 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.870+0000 7f0034000640 1 Processor -- start 2024-03-18T16:14:24.873 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.871+0000 7f0034000640 1 -- start start 2024-03-18T16:14:24.873 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.871+0000 7f0034000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f002c1087e0 0x7f002c1a6890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:24.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.871+0000 7f0034000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f002c10b150 0x7f002c1a6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:24.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.871+0000 7f0034000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f002c10db00 0x7f002c1addf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:24.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.871+0000 7f0034000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f002c11c220 con 0x7f002c1087e0 2024-03-18T16:14:24.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.871+0000 7f0034000640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f002c11c0a0 con 0x7f002c10db00 2024-03-18T16:14:24.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.871+0000 7f0034000640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f002c11c3a0 con 0x7f002c10b150 2024-03-18T16:14:24.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.872+0000 7f0030c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f002c10b150 0x7f002c1a6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:24.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.872+0000 7f0030c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f002c10b150 0x7f002c1a6dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:53360/0 (socket says 172.21.15.39:53360) 2024-03-18T16:14:24.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.872+0000 7f0030c00640 1 -- 172.21.15.39:0/2599261323 learned_addr learned my addr 172.21.15.39:0/2599261323 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:14:24.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.872+0000 7f0032000640 1 --2- 172.21.15.39:0/2599261323 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f002c10db00 0x7f002c1addf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:24.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.872+0000 7f0032000640 1 -- 172.21.15.39:0/2599261323 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f002c10b150 msgr2=0x7f002c1a6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:14:24.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.872+0000 7f0032000640 1 --2- 172.21.15.39:0/2599261323 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f002c10b150 0x7f002c1a6dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.872+0000 7f0032000640 1 -- 172.21.15.39:0/2599261323 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f002c1087e0 msgr2=0x7f002c1a6890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:14:24.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.872+0000 7f0032000640 1 --2- 172.21.15.39:0/2599261323 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f002c1087e0 0x7f002c1a6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.872+0000 7f0032000640 1 -- 172.21.15.39:0/2599261323 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f002c1ae460 con 0x7f002c10db00 2024-03-18T16:14:24.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.873+0000 7f0032000640 1 --2- 172.21.15.39:0/2599261323 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f002c10db00 0x7f002c1addf0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f002000efc0 tx=0x7f002000c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:14:24.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.877+0000 7f002a000640 1 -- 172.21.15.39:0/2599261323 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0020003970 con 0x7f002c10db00 2024-03-18T16:14:24.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.877+0000 7f002a000640 1 -- 172.21.15.39:0/2599261323 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0020004020 con 0x7f002c10db00 2024-03-18T16:14:24.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.877+0000 7f002a000640 1 -- 172.21.15.39:0/2599261323 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f002000b9f0 con 0x7f002c10db00 2024-03-18T16:14:24.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.879+0000 7f0034000640 1 -- 172.21.15.39:0/2599261323 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f002c1ae750 con 0x7f002c10db00 2024-03-18T16:14:24.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.879+0000 7f0034000640 1 -- 172.21.15.39:0/2599261323 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f002c1aec20 con 0x7f002c10db00 2024-03-18T16:14:24.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.884+0000 7f002a000640 1 -- 172.21.15.39:0/2599261323 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f002001d020 con 0x7f002c10db00 2024-03-18T16:14:24.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.885+0000 7f0036db1740 1 Processor -- start 2024-03-18T16:14:24.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.886+0000 7f002a000640 1 --2- 172.21.15.39:0/2599261323 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f00140a29d0 0x7f00140a4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:24.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.886+0000 7f002a000640 1 -- 172.21.15.39:0/2599261323 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f00200cecc0 con 0x7f002c10db00 2024-03-18T16:14:24.890 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.887+0000 7f0036db1740 1 -- start start 2024-03-18T16:14:24.890 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.887+0000 7f0036db1740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bef4ffa8c0 0x55bef4fface0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:24.890 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.887+0000 7f0036db1740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bef4ffb2b0 0x55bef4fffdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:24.890 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.887+0000 7f0036db1740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bef5000420 0x55bef50028b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:24.890 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.887+0000 7f0036db1740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55bef4fc23b0 con 0x55bef5000420 2024-03-18T16:14:24.890 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.887+0000 7f0036db1740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55bef4f5f450 con 0x55bef4ffb2b0 2024-03-18T16:14:24.890 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.887+0000 7f0036db1740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55bef4f07cb0 con 0x55bef4ffa8c0 2024-03-18T16:14:24.890 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.888+0000 7f0031600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bef4ffa8c0 0x55bef4fface0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:24.890 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.888+0000 7f0030c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bef4ffb2b0 0x55bef4fffdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:24.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.888+0000 7f0030c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bef4ffb2b0 0x55bef4fffdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:49018/0 (socket says 172.21.15.39:49018) 2024-03-18T16:14:24.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.888+0000 7f0030c00640 1 -- 172.21.15.39:0/1159485270 learned_addr learned my addr 172.21.15.39:0/1159485270 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:14:24.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.888+0000 7f0032000640 1 --2- 172.21.15.39:0/1159485270 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bef5000420 0x55bef50028b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:24.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.888+0000 7f0031600640 1 -- 172.21.15.39:0/1159485270 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bef4ffb2b0 msgr2=0x55bef4fffdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:14:24.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.888+0000 7f0031600640 1 --2- 172.21.15.39:0/1159485270 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bef4ffb2b0 0x55bef4fffdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.888+0000 7f0031600640 1 -- 172.21.15.39:0/1159485270 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bef5000420 msgr2=0x55bef50028b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:14:24.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.888+0000 7f0031600640 1 --2- 172.21.15.39:0/1159485270 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bef5000420 0x55bef50028b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.888+0000 7f0031600640 1 -- 172.21.15.39:0/1159485270 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bef4f291f0 con 0x55bef4ffa8c0 2024-03-18T16:14:24.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.889+0000 7f0030c00640 1 --2- 172.21.15.39:0/2599261323 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f00140a29d0 0x7f00140a4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:24.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.889+0000 7f0031600640 1 --2- 172.21.15.39:0/1159485270 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bef4ffa8c0 0x55bef4fface0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f0018000c00 tx=0x7f0018005d20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d5d1c1e9a4dff5cf server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:14:24.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.890+0000 7f001e600640 1 -- 172.21.15.39:0/1159485270 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0018041020 con 0x55bef4ffa8c0 2024-03-18T16:14:24.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.890+0000 7f001e600640 1 -- 172.21.15.39:0/1159485270 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0018034640 con 0x55bef4ffa8c0 2024-03-18T16:14:24.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.890+0000 7f001e600640 1 -- 172.21.15.39:0/1159485270 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0018039040 con 0x55bef4ffa8c0 2024-03-18T16:14:24.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.891+0000 7f0036db1740 1 -- 172.21.15.39:0/1159485270 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bef4ffa8c0 msgr2=0x55bef4fface0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:14:24.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.891+0000 7f0030c00640 1 --2- 172.21.15.39:0/2599261323 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f00140a29d0 0x7f00140a4e90 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x55bef4ffb6b0 tx=0x7f002400a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:14:24.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.891+0000 7f0036db1740 1 --2- 172.21.15.39:0/1159485270 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bef4ffa8c0 0x55bef4fface0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f0018000c00 tx=0x7f0018005d20 comp rx=0 tx=0).stop 2024-03-18T16:14:24.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.893+0000 7f0036db1740 1 -- 172.21.15.39:0/1159485270 shutdown_connections 2024-03-18T16:14:24.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.893+0000 7f0036db1740 1 --2- 172.21.15.39:0/1159485270 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bef4ffb2b0 0x55bef4fffdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.894+0000 7f0036db1740 1 --2- 172.21.15.39:0/1159485270 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bef5000420 0x55bef50028b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.894+0000 7f0036db1740 1 --2- 172.21.15.39:0/1159485270 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bef4ffa8c0 0x55bef4fface0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.894+0000 7f0036db1740 1 -- 172.21.15.39:0/1159485270 >> 172.21.15.39:0/1159485270 conn(0x55bef4ff5e30 msgr2=0x55bef4ff6230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:14:24.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.894+0000 7f0036db1740 1 -- 172.21.15.39:0/1159485270 shutdown_connections 2024-03-18T16:14:24.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.895+0000 7f0036db1740 1 -- 172.21.15.39:0/1159485270 wait complete. 2024-03-18T16:14:24.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.896+0000 7f0036db1740 1 -- start start 2024-03-18T16:14:24.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.896+0000 7f0036db1740 1 Processor -- start 2024-03-18T16:14:24.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.897+0000 7f0036db1740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bef4ffa8c0 0x55bef5004bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:24.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.897+0000 7f0036db1740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bef4ffb2b0 0x55bef5005100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:24.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.897+0000 7f0036db1740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bef5000420 0x55bef500c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:24.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.898+0000 7f0030c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bef4ffb2b0 0x55bef5005100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:24.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.898+0000 7f0030c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bef4ffb2b0 0x55bef5005100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:49028/0 (socket says 172.21.15.39:49028) 2024-03-18T16:14:24.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.898+0000 7f0036db1740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55bef4f1b110 con 0x55bef4ffa8c0 2024-03-18T16:14:24.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.898+0000 7f0036db1740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55bef4f5b4f0 con 0x55bef4ffb2b0 2024-03-18T16:14:24.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.898+0000 7f0036db1740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55bef5003370 con 0x55bef5000420 2024-03-18T16:14:24.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.898+0000 7f0036db1740 10 client.0 _ll_register_callbacks cb 0x55bef4f08b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:14:24.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.898+0000 7f0030c00640 1 -- 172.21.15.39:0/3762239159 learned_addr learned my addr 172.21.15.39:0/3762239159 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:14:24.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.899+0000 7f0030c00640 1 -- 172.21.15.39:0/3762239159 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bef5000420 msgr2=0x55bef500c140 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:14:24.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.899+0000 7f0030c00640 1 --2- 172.21.15.39:0/3762239159 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bef5000420 0x55bef500c140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.899+0000 7f0030c00640 1 -- 172.21.15.39:0/3762239159 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bef4ffa8c0 msgr2=0x55bef5004bc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:14:24.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.899+0000 7f0031600640 1 --2- 172.21.15.39:0/3762239159 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bef4ffa8c0 0x55bef5004bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:24.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.900+0000 7f0030c00640 1 --2- 172.21.15.39:0/3762239159 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bef4ffa8c0 0x55bef5004bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.900+0000 7f0030c00640 1 -- 172.21.15.39:0/3762239159 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bef4c733f0 con 0x55bef4ffb2b0 2024-03-18T16:14:24.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.900+0000 7f0031600640 1 --2- 172.21.15.39:0/3762239159 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bef4ffa8c0 0x55bef5004bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:14:24.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.901+0000 7f0030c00640 1 --2- 172.21.15.39:0/3762239159 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bef4ffb2b0 0x55bef5005100 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f002400da10 tx=0x7f0024004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be20c01e37c9f488 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:14:24.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.901+0000 7f0012a00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:14:24.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.902+0000 7f0012a00640 1 -- 172.21.15.39:0/3762239159 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0024010840 con 0x55bef4ffb2b0 2024-03-18T16:14:24.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.903+0000 7f0012a00640 1 -- 172.21.15.39:0/3762239159 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f00240109e0 con 0x55bef4ffb2b0 2024-03-18T16:14:24.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.903+0000 7f0036db1740 1 -- 172.21.15.39:0/3762239159 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0018034640 con 0x55bef4ffb2b0 2024-03-18T16:14:24.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.903+0000 7f0012a00640 1 -- 172.21.15.39:0/3762239159 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f0024010be0 con 0x55bef4ffb2b0 2024-03-18T16:14:24.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.903+0000 7f0036db1740 10 client.26480 fetch_fsmap learned FSMap version 36 2024-03-18T16:14:24.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.904+0000 7f0036db1740 1 -- 172.21.15.39:0/3762239159 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55bef4d3d7d0 con 0x55bef4ffb2b0 2024-03-18T16:14:24.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.904+0000 7f0012a00640 1 -- 172.21.15.39:0/3762239159 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0018034640 con 0x55bef4ffb2b0 2024-03-18T16:14:24.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.904+0000 7f0012a00640 1 -- 172.21.15.39:0/3762239159 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 36) ==== 8406+0+0 (secure 0 0 0) 0x7f0024032420 con 0x55bef4ffb2b0 2024-03-18T16:14:24.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.905+0000 7f0036db1740 10 client.26480 fetch_fsmap finished waiting for FSMap version 36 2024-03-18T16:14:24.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.905+0000 7f0036db1740 10 client.26480 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T16:14:24.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.905+0000 7f0036db1740 20 client.26480 populate_metadata read hostname 'smithi039' 2024-03-18T16:14:24.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.905+0000 7f0036db1740 1 --2- 172.21.15.39:0/3762239159 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x55bef5010b10 0x55bef5012fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:24.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.905+0000 7f0036db1740 4 client.26480 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:14:24.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.905+0000 7f0036db1740 1 -- 172.21.15.39:0/3762239159 --> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bef4f6c990 con 0x55bef5010b10 2024-03-18T16:14:24.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.907+0000 7f0031600640 1 --2- 172.21.15.39:0/3762239159 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x55bef5010b10 0x55bef5012fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:24.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.908+0000 7f0012a00640 10 client.26480 ms_handle_connect on v2:172.21.15.196:6838/2001427268 2024-03-18T16:14:24.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.908+0000 7f0031600640 1 --2- 172.21.15.39:0/3762239159 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x55bef5010b10 0x55bef5012fd0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e027453763c8def5 server_cookie=27ab60d5aa17ea34 in_seq=0 out_seq=0 2024-03-18T16:14:24.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.911+0000 7f0012a00640 1 -- 172.21.15.39:0/3762239159 <== mds.1 v2:172.21.15.196:6838/2001427268 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55bef4f6c990 con 0x55bef5010b10 2024-03-18T16:14:24.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.911+0000 7f0012a00640 10 client.26480 handle_command_reply: tid=0 2024-03-18T16:14:24.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.911+0000 7f0036db1740 1 client.26480 shutdown 2024-03-18T16:14:24.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.911+0000 7f0036db1740 1 -- 172.21.15.39:0/3762239159 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bef4ffb2b0 msgr2=0x55bef5005100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:14:24.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.911+0000 7f0036db1740 1 --2- 172.21.15.39:0/3762239159 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bef4ffb2b0 0x55bef5005100 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f002400da10 tx=0x7f0024004730 comp rx=0 tx=0).stop 2024-03-18T16:14:24.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.913+0000 7f0036db1740 1 -- 172.21.15.39:0/3762239159 shutdown_connections 2024-03-18T16:14:24.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.913+0000 7f0036db1740 1 --2- 172.21.15.39:0/3762239159 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x55bef5010b10 0x55bef5012fd0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.913+0000 7f0036db1740 1 --2- 172.21.15.39:0/3762239159 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bef4ffb2b0 0x55bef5005100 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.913+0000 7f0036db1740 1 --2- 172.21.15.39:0/3762239159 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bef5000420 0x55bef500c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.913+0000 7f0036db1740 1 --2- 172.21.15.39:0/3762239159 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bef4ffa8c0 0x55bef5004bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.913+0000 7f0036db1740 1 -- 172.21.15.39:0/3762239159 >> 172.21.15.39:0/3762239159 conn(0x55bef4ff5e30 msgr2=0x55bef4ff51f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:14:24.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.914+0000 7f0012a00640 0 client.26480 ms_handle_reset on v2:172.21.15.196:6838/2001427268 2024-03-18T16:14:24.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.914+0000 7f0036db1740 1 -- 172.21.15.39:0/3762239159 shutdown_connections 2024-03-18T16:14:24.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.915+0000 7f0036db1740 1 -- 172.21.15.39:0/3762239159 wait complete. 2024-03-18T16:14:24.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.915+0000 7f0036db1740 20 client.26480 trim_cache size 0 max 16384 2024-03-18T16:14:24.956 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.954+0000 7f0036db1740 1 Processor -- start 2024-03-18T16:14:24.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.954+0000 7f0036db1740 1 -- start start 2024-03-18T16:14:24.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.954+0000 7f0036db1740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bef4ffa8c0 0x55bef4ffc750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:24.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.954+0000 7f0036db1740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bef4ffb2b0 0x55bef500c100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:24.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.955+0000 7f0036db1740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bef5000420 0x55bef4ffcc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:24.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.955+0000 7f0036db1740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f002c1b91d0 con 0x55bef5000420 2024-03-18T16:14:24.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.955+0000 7f0036db1740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f002c1bae90 con 0x55bef4ffb2b0 2024-03-18T16:14:24.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.955+0000 7f0036db1740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f002c1b98a0 con 0x55bef4ffa8c0 2024-03-18T16:14:24.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.955+0000 7f0030c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bef4ffb2b0 0x55bef500c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:24.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.955+0000 7f0030c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bef4ffb2b0 0x55bef500c100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:49036/0 (socket says 172.21.15.39:49036) 2024-03-18T16:14:24.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.955+0000 7f0030c00640 1 -- 172.21.15.39:0/3210363940 learned_addr learned my addr 172.21.15.39:0/3210363940 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:14:24.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.956+0000 7f0030c00640 1 -- 172.21.15.39:0/3210363940 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bef4ffa8c0 msgr2=0x55bef4ffc750 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:14:24.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.956+0000 7f0030c00640 1 --2- 172.21.15.39:0/3210363940 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bef4ffa8c0 0x55bef4ffc750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.956+0000 7f0030c00640 1 -- 172.21.15.39:0/3210363940 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bef5000420 msgr2=0x55bef4ffcc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:14:24.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.956+0000 7f0030c00640 1 --2- 172.21.15.39:0/3210363940 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bef5000420 0x55bef4ffcc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.956+0000 7f0030c00640 1 -- 172.21.15.39:0/3210363940 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bef4ff1c10 con 0x55bef4ffb2b0 2024-03-18T16:14:24.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.957+0000 7f0030c00640 1 --2- 172.21.15.39:0/3210363940 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bef4ffb2b0 0x55bef500c100 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f0024032890 tx=0x7f0024010b70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1da3d47ce717c330 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:14:24.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.957+0000 7f001dc00640 1 -- 172.21.15.39:0/3210363940 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0024010be0 con 0x55bef4ffb2b0 2024-03-18T16:14:24.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.958+0000 7f001dc00640 1 -- 172.21.15.39:0/3210363940 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0024023070 con 0x55bef4ffb2b0 2024-03-18T16:14:24.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.958+0000 7f001dc00640 1 -- 172.21.15.39:0/3210363940 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0024032680 con 0x55bef4ffb2b0 2024-03-18T16:14:24.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.959+0000 7f0036db1740 1 -- 172.21.15.39:0/3210363940 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bef4ffb2b0 msgr2=0x55bef500c100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:14:24.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.959+0000 7f0036db1740 1 --2- 172.21.15.39:0/3210363940 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bef4ffb2b0 0x55bef500c100 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f0024032890 tx=0x7f0024010b70 comp rx=0 tx=0).stop 2024-03-18T16:14:24.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.959+0000 7f0036db1740 1 -- 172.21.15.39:0/3210363940 shutdown_connections 2024-03-18T16:14:24.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.959+0000 7f0036db1740 1 --2- 172.21.15.39:0/3210363940 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bef4ffb2b0 0x55bef500c100 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.960+0000 7f0036db1740 1 --2- 172.21.15.39:0/3210363940 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bef5000420 0x55bef4ffcc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.960+0000 7f0036db1740 1 --2- 172.21.15.39:0/3210363940 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bef4ffa8c0 0x55bef4ffc750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.960+0000 7f0036db1740 1 -- 172.21.15.39:0/3210363940 >> 172.21.15.39:0/3210363940 conn(0x55bef4ff5e30 msgr2=0x55bef5004bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:14:24.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.960+0000 7f0036db1740 1 -- 172.21.15.39:0/3210363940 shutdown_connections 2024-03-18T16:14:24.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.960+0000 7f0036db1740 1 -- 172.21.15.39:0/3210363940 wait complete. 2024-03-18T16:14:24.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.961+0000 7f0036db1740 1 -- start start 2024-03-18T16:14:24.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.961+0000 7f0036db1740 1 Processor -- start 2024-03-18T16:14:24.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.962+0000 7f0036db1740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bef4ffa8c0 0x55bef5007220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:24.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.962+0000 7f0036db1740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bef4ffb2b0 0x55bef5007760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:24.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.962+0000 7f0036db1740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bef5000420 0x55bef500a780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:24.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.962+0000 7f0036db1740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f002c1ba700 con 0x55bef4ffb2b0 2024-03-18T16:14:24.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.962+0000 7f0036db1740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f002c1b9f70 con 0x55bef4ffa8c0 2024-03-18T16:14:24.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.962+0000 7f0036db1740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f002c1b7170 con 0x55bef5000420 2024-03-18T16:14:24.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.962+0000 7f0036db1740 10 client.0 _ll_register_callbacks cb 0x7f002400d690 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:14:24.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.963+0000 7f0032000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bef5000420 0x55bef500a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:24.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.963+0000 7f0032000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bef5000420 0x55bef500a780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:53382/0 (socket says 172.21.15.39:53382) 2024-03-18T16:14:24.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.963+0000 7f0032000640 1 -- 172.21.15.39:0/1531981318 learned_addr learned my addr 172.21.15.39:0/1531981318 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:14:24.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.963+0000 7f0031600640 1 --2- 172.21.15.39:0/1531981318 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bef4ffa8c0 0x55bef5007220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:24.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.963+0000 7f0030c00640 1 --2- 172.21.15.39:0/1531981318 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bef4ffb2b0 0x55bef5007760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:24.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.964+0000 7f0032000640 1 -- 172.21.15.39:0/1531981318 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bef4ffa8c0 msgr2=0x55bef5007220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:14:24.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.964+0000 7f0032000640 1 --2- 172.21.15.39:0/1531981318 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bef4ffa8c0 0x55bef5007220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.964+0000 7f0032000640 1 -- 172.21.15.39:0/1531981318 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bef4ffb2b0 msgr2=0x55bef5007760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:14:24.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.964+0000 7f0032000640 1 --2- 172.21.15.39:0/1531981318 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bef4ffb2b0 0x55bef5007760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.964+0000 7f0032000640 1 -- 172.21.15.39:0/1531981318 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bef500ad70 con 0x55bef5000420 2024-03-18T16:14:24.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.964+0000 7f0030c00640 1 --2- 172.21.15.39:0/1531981318 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bef4ffb2b0 0x55bef5007760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:14:24.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.964+0000 7f0032000640 1 --2- 172.21.15.39:0/1531981318 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bef5000420 0x55bef500a780 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f002000cd00 tx=0x7f002009ba30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a530e5b2c5af73bb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:14:24.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.964+0000 7f0012a00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:14:24.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.965+0000 7f0012a00640 1 -- 172.21.15.39:0/1531981318 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00200a3a10 con 0x55bef5000420 2024-03-18T16:14:24.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.965+0000 7f0012a00640 1 -- 172.21.15.39:0/1531981318 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f00200bb0c0 con 0x55bef5000420 2024-03-18T16:14:24.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.965+0000 7f0036db1740 1 -- 172.21.15.39:0/1531981318 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0024023070 con 0x55bef5000420 2024-03-18T16:14:24.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.965+0000 7f0012a00640 1 -- 172.21.15.39:0/1531981318 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f002009f170 con 0x55bef5000420 2024-03-18T16:14:24.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.965+0000 7f0036db1740 10 client.26464 fetch_fsmap learned FSMap version 36 2024-03-18T16:14:24.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.965+0000 7f0036db1740 1 -- 172.21.15.39:0/1531981318 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x55bef500c140 con 0x55bef5000420 2024-03-18T16:14:24.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.965+0000 7f0012a00640 1 -- 172.21.15.39:0/1531981318 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0024023070 con 0x55bef5000420 2024-03-18T16:14:24.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.966+0000 7f0012a00640 1 -- 172.21.15.39:0/1531981318 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 36) ==== 8406+0+0 (secure 0 0 0) 0x7f002000b040 con 0x55bef5000420 2024-03-18T16:14:24.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.966+0000 7f0036db1740 10 client.26464 fetch_fsmap finished waiting for FSMap version 36 2024-03-18T16:14:24.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.967+0000 7f0036db1740 10 client.26464 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T16:14:24.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.967+0000 7f0036db1740 20 client.26464 populate_metadata read hostname 'smithi039' 2024-03-18T16:14:24.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.967+0000 7f0036db1740 1 --2- 172.21.15.39:0/1531981318 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x55bef5088570 0x55bef508a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:24.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.967+0000 7f0036db1740 4 client.26464 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:14:24.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.967+0000 7f0036db1740 1 -- 172.21.15.39:0/1531981318 --> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55bef4f66ce0 con 0x55bef5088570 2024-03-18T16:14:24.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.967+0000 7f0031600640 1 --2- 172.21.15.39:0/1531981318 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x55bef5088570 0x55bef508a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:24.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.967+0000 7f0012a00640 10 client.26464 ms_handle_connect on v2:172.21.15.196:6838/2001427268 2024-03-18T16:14:24.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.967+0000 7f0031600640 1 --2- 172.21.15.39:0/1531981318 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x55bef5088570 0x55bef508a990 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=fe23345a79fb487a server_cookie=4ca429072873c63c in_seq=0 out_seq=0 2024-03-18T16:14:24.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.968+0000 7f0012a00640 1 -- 172.21.15.39:0/1531981318 <== mds.1 v2:172.21.15.196:6838/2001427268 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55bef4f66ce0 con 0x55bef5088570 2024-03-18T16:14:24.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.968+0000 7f0012a00640 10 client.26464 handle_command_reply: tid=0 2024-03-18T16:14:24.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.968+0000 7f0036db1740 1 client.26464 shutdown 2024-03-18T16:14:24.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.968+0000 7f0036db1740 1 -- 172.21.15.39:0/1531981318 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bef5000420 msgr2=0x55bef500a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:14:24.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.968+0000 7f0036db1740 1 --2- 172.21.15.39:0/1531981318 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bef5000420 0x55bef500a780 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f002000cd00 tx=0x7f002009ba30 comp rx=0 tx=0).stop 2024-03-18T16:14:24.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.969+0000 7f0036db1740 1 -- 172.21.15.39:0/1531981318 shutdown_connections 2024-03-18T16:14:24.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.969+0000 7f0036db1740 1 --2- 172.21.15.39:0/1531981318 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x55bef5088570 0x55bef508a990 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.969+0000 7f0012a00640 0 client.26464 ms_handle_reset on v2:172.21.15.196:6838/2001427268 2024-03-18T16:14:24.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.969+0000 7f0036db1740 1 --2- 172.21.15.39:0/1531981318 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bef5000420 0x55bef500a780 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.969+0000 7f0036db1740 1 --2- 172.21.15.39:0/1531981318 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bef4ffb2b0 0x55bef5007760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.969+0000 7f0036db1740 1 --2- 172.21.15.39:0/1531981318 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bef4ffa8c0 0x55bef5007220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.969+0000 7f0036db1740 1 -- 172.21.15.39:0/1531981318 >> 172.21.15.39:0/1531981318 conn(0x55bef4ff5e30 msgr2=0x55bef5004ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:14:24.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.969+0000 7f0036db1740 1 -- 172.21.15.39:0/1531981318 shutdown_connections 2024-03-18T16:14:24.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.969+0000 7f0036db1740 1 -- 172.21.15.39:0/1531981318 wait complete. 2024-03-18T16:14:24.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.970+0000 7f0036db1740 20 client.26464 trim_cache size 0 max 16384 2024-03-18T16:14:24.973 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:14:24.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.971+0000 7f0034000640 1 -- 172.21.15.39:0/2599261323 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f00140a29d0 msgr2=0x7f00140a4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:14:24.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.971+0000 7f0034000640 1 --2- 172.21.15.39:0/2599261323 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f00140a29d0 0x7f00140a4e90 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x55bef4ffb6b0 tx=0x7f002400a380 comp rx=0 tx=0).stop 2024-03-18T16:14:24.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.971+0000 7f0034000640 1 -- 172.21.15.39:0/2599261323 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f002c10db00 msgr2=0x7f002c1addf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:14:24.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.971+0000 7f0034000640 1 --2- 172.21.15.39:0/2599261323 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f002c10db00 0x7f002c1addf0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f002000efc0 tx=0x7f002000c800 comp rx=0 tx=0).stop 2024-03-18T16:14:24.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.972+0000 7f0034000640 1 -- 172.21.15.39:0/2599261323 shutdown_connections 2024-03-18T16:14:24.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.972+0000 7f0034000640 1 --2- 172.21.15.39:0/2599261323 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f002c10db00 0x7f002c1addf0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.972+0000 7f0034000640 1 --2- 172.21.15.39:0/2599261323 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f00140a29d0 0x7f00140a4e90 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.972+0000 7f0034000640 1 --2- 172.21.15.39:0/2599261323 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f002c10b150 0x7f002c1a6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.972+0000 7f0034000640 1 --2- 172.21.15.39:0/2599261323 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f002c1087e0 0x7f002c1a6890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:24.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.972+0000 7f0034000640 1 -- 172.21.15.39:0/2599261323 >> 172.21.15.39:0/2599261323 conn(0x7f002c06e860 msgr2=0x7f002c10d360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:14:24.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.972+0000 7f0034000640 1 -- 172.21.15.39:0/2599261323 shutdown_connections 2024-03-18T16:14:24.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:24.972+0000 7f0034000640 1 -- 172.21.15.39:0/2599261323 wait complete. 2024-03-18T16:14:25.004 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-18T16:14:25.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.301+0000 7f837c600640 1 -- 172.21.15.39:0/1590224887 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f83741087e0 msgr2=0x7f837410ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:14:25.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.301+0000 7f837c600640 1 --2- 172.21.15.39:0/1590224887 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f83741087e0 0x7f837410ac10 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f8368009900 tx=0x7f836802bc50 comp rx=0 tx=0).stop 2024-03-18T16:14:25.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.301+0000 7f837c600640 1 -- 172.21.15.39:0/1590224887 shutdown_connections 2024-03-18T16:14:25.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.301+0000 7f837c600640 1 --2- 172.21.15.39:0/1590224887 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f837410b150 0x7f837410d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:25.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.301+0000 7f837c600640 1 --2- 172.21.15.39:0/1590224887 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f837410db00 0x7f837410df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:25.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.301+0000 7f837c600640 1 --2- 172.21.15.39:0/1590224887 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f83741087e0 0x7f837410ac10 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:25.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.301+0000 7f837c600640 1 -- 172.21.15.39:0/1590224887 >> 172.21.15.39:0/1590224887 conn(0x7f837406e860 msgr2=0x7f8374070ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:14:25.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.302+0000 7f837c600640 1 -- 172.21.15.39:0/1590224887 shutdown_connections 2024-03-18T16:14:25.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.302+0000 7f837c600640 1 -- 172.21.15.39:0/1590224887 wait complete. 2024-03-18T16:14:25.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.303+0000 7f837c600640 1 Processor -- start 2024-03-18T16:14:25.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.304+0000 7f837c600640 1 -- start start 2024-03-18T16:14:25.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.304+0000 7f837c600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f83741087e0 0x7f83741182d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:25.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.304+0000 7f837c600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f837410b150 0x7f8374118810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:25.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.304+0000 7f837c600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f837410db00 0x7f83741196e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:25.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.305+0000 7f8379c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f83741087e0 0x7f83741182d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:25.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.305+0000 7f8379c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f83741087e0 0x7f83741182d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:53394/0 (socket says 172.21.15.39:53394) 2024-03-18T16:14:25.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.305+0000 7f8379c00640 1 -- 172.21.15.39:0/2896074944 learned_addr learned my addr 172.21.15.39:0/2896074944 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:14:25.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.305+0000 7f837c600640 1 -- 172.21.15.39:0/2896074944 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f8374110940 con 0x7f837410b150 2024-03-18T16:14:25.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.305+0000 7f837c600640 1 -- 172.21.15.39:0/2896074944 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f83741107c0 con 0x7f837410db00 2024-03-18T16:14:25.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.305+0000 7f837c600640 1 -- 172.21.15.39:0/2896074944 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f8374110ac0 con 0x7f83741087e0 2024-03-18T16:14:25.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.305+0000 7f837a600640 1 --2- 172.21.15.39:0/2896074944 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f837410db00 0x7f83741196e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:25.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.305+0000 7f8379200640 1 --2- 172.21.15.39:0/2896074944 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f837410b150 0x7f8374118810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:25.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.305+0000 7f8379c00640 1 -- 172.21.15.39:0/2896074944 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f837410db00 msgr2=0x7f83741196e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:14:25.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.305+0000 7f8379c00640 1 --2- 172.21.15.39:0/2896074944 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f837410db00 0x7f83741196e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:25.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.306+0000 7f8379c00640 1 -- 172.21.15.39:0/2896074944 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f837410b150 msgr2=0x7f8374118810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:14:25.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.306+0000 7f8379c00640 1 --2- 172.21.15.39:0/2896074944 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f837410b150 0x7f8374118810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:25.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.306+0000 7f8379c00640 1 -- 172.21.15.39:0/2896074944 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8374119e10 con 0x7f83741087e0 2024-03-18T16:14:25.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.306+0000 7f837a600640 1 --2- 172.21.15.39:0/2896074944 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f837410db00 0x7f83741196e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:14:25.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.306+0000 7f8379c00640 1 --2- 172.21.15.39:0/2896074944 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f83741087e0 0x7f83741182d0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f8368000c00 tx=0x7f83680053c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:14:25.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.307+0000 7f8372000640 1 -- 172.21.15.39:0/2896074944 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8368041020 con 0x7f83741087e0 2024-03-18T16:14:25.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.307+0000 7f8372000640 1 -- 172.21.15.39:0/2896074944 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f836802cad0 con 0x7f83741087e0 2024-03-18T16:14:25.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.307+0000 7f837c600640 1 -- 172.21.15.39:0/2896074944 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f837411a0a0 con 0x7f83741087e0 2024-03-18T16:14:25.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.307+0000 7f837c600640 1 -- 172.21.15.39:0/2896074944 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83741b7610 con 0x7f83741087e0 2024-03-18T16:14:25.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.307+0000 7f8372000640 1 -- 172.21.15.39:0/2896074944 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8368034d40 con 0x7f83741087e0 2024-03-18T16:14:25.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.314+0000 7f8372000640 1 -- 172.21.15.39:0/2896074944 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f8368044020 con 0x7f83741087e0 2024-03-18T16:14:25.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.314+0000 7f837f373740 1 Processor -- start 2024-03-18T16:14:25.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.315+0000 7f837f373740 1 -- start start 2024-03-18T16:14:25.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.316+0000 7f837f373740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5604e8fb48c0 0x5604e8fb4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:25.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.316+0000 7f837f373740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5604e8fb52b0 0x5604e8fb9db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:25.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.316+0000 7f837f373740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5604e8fba420 0x5604e8fbc8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:25.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.316+0000 7f837f373740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5604e8f7c3b0 con 0x5604e8fb52b0 2024-03-18T16:14:25.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.316+0000 7f837f373740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5604e8f19450 con 0x5604e8fb48c0 2024-03-18T16:14:25.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.316+0000 7f837a600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5604e8fb48c0 0x5604e8fb4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:25.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.317+0000 7f8372000640 1 --2- 172.21.15.39:0/2896074944 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f835c0a2980 0x7f835c0a4e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:25.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.317+0000 7f8372000640 1 -- 172.21.15.39:0/2896074944 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f83680f0830 con 0x7f83741087e0 2024-03-18T16:14:25.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.318+0000 7f8379c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5604e8fb52b0 0x5604e8fb9db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:25.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.318+0000 7f837a600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5604e8fb48c0 0x5604e8fb4ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:49064/0 (socket says 172.21.15.39:49064) 2024-03-18T16:14:25.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.318+0000 7f837a600640 1 -- 172.21.15.39:0/666626861 learned_addr learned my addr 172.21.15.39:0/666626861 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:14:25.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.318+0000 7f8379200640 1 --2- 172.21.15.39:0/666626861 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5604e8fba420 0x5604e8fbc8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:25.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.318+0000 7f837f373740 1 -- 172.21.15.39:0/666626861 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x5604e8ec1cb0 con 0x5604e8fba420 2024-03-18T16:14:25.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.319+0000 7f8379c00640 1 -- 172.21.15.39:0/666626861 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5604e8fba420 msgr2=0x5604e8fbc8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:14:25.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.319+0000 7f8379c00640 1 --2- 172.21.15.39:0/666626861 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5604e8fba420 0x5604e8fbc8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:25.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.319+0000 7f8379c00640 1 -- 172.21.15.39:0/666626861 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5604e8fb48c0 msgr2=0x5604e8fb4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:14:25.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.319+0000 7f8379c00640 1 --2- 172.21.15.39:0/666626861 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5604e8fb48c0 0x5604e8fb4ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:25.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.319+0000 7f8379c00640 1 -- 172.21.15.39:0/666626861 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5604e8ee31f0 con 0x5604e8fb52b0 2024-03-18T16:14:25.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.319+0000 7f837a600640 1 --2- 172.21.15.39:0/666626861 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5604e8fb48c0 0x5604e8fb4ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:14:25.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.319+0000 7f837a600640 1 --2- 172.21.15.39:0/2896074944 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f835c0a2980 0x7f835c0a4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:25.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.320+0000 7f8379c00640 1 --2- 172.21.15.39:0/666626861 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5604e8fb52b0 0x5604e8fb9db0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f836802c260 tx=0x7f836802bc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=879547a894d446d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:14:25.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.320+0000 7f8366a00640 1 -- 172.21.15.39:0/666626861 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83680baf20 con 0x5604e8fb52b0 2024-03-18T16:14:25.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.320+0000 7f8366a00640 1 -- 172.21.15.39:0/666626861 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f83680bd4e0 con 0x5604e8fb52b0 2024-03-18T16:14:25.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.320+0000 7f8366a00640 1 -- 172.21.15.39:0/666626861 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8368034440 con 0x5604e8fb52b0 2024-03-18T16:14:25.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.322+0000 7f837f373740 1 -- 172.21.15.39:0/666626861 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5604e8fb52b0 msgr2=0x5604e8fb9db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:14:25.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.322+0000 7f837a600640 1 --2- 172.21.15.39:0/2896074944 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f835c0a2980 0x7f835c0a4e40 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f836c009fd0 tx=0x7f836c004b50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:14:25.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.322+0000 7f837f373740 1 --2- 172.21.15.39:0/666626861 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5604e8fb52b0 0x5604e8fb9db0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f836802c260 tx=0x7f836802bc50 comp rx=0 tx=0).stop 2024-03-18T16:14:25.326 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.323+0000 7f837f373740 1 -- 172.21.15.39:0/666626861 shutdown_connections 2024-03-18T16:14:25.326 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.323+0000 7f837f373740 1 --2- 172.21.15.39:0/666626861 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5604e8fba420 0x5604e8fbc8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:25.326 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.323+0000 7f837f373740 1 --2- 172.21.15.39:0/666626861 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5604e8fb52b0 0x5604e8fb9db0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:25.326 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.323+0000 7f837f373740 1 --2- 172.21.15.39:0/666626861 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5604e8fb48c0 0x5604e8fb4ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:25.326 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.323+0000 7f837f373740 1 -- 172.21.15.39:0/666626861 >> 172.21.15.39:0/666626861 conn(0x5604e8fafe30 msgr2=0x5604e8fb0230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:14:25.326 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.323+0000 7f837f373740 1 -- 172.21.15.39:0/666626861 shutdown_connections 2024-03-18T16:14:25.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.323+0000 7f837f373740 1 -- 172.21.15.39:0/666626861 wait complete. 2024-03-18T16:14:25.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.324+0000 7f837f373740 1 -- start start 2024-03-18T16:14:25.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.324+0000 7f837f373740 1 Processor -- start 2024-03-18T16:14:25.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.325+0000 7f837f373740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5604e8fb48c0 0x5604e8fbe7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:25.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.325+0000 7f837f373740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5604e8fb52b0 0x5604e8fbed30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:25.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.325+0000 7f837f373740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5604e8fba420 0x5604e8fc5d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:25.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.325+0000 7f837f373740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5604e8ed5110 con 0x5604e8fb52b0 2024-03-18T16:14:25.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.325+0000 7f837f373740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5604e8f154f0 con 0x5604e8fba420 2024-03-18T16:14:25.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.325+0000 7f837f373740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x5604e8fbd190 con 0x5604e8fb48c0 2024-03-18T16:14:25.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.326+0000 7f8379c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5604e8fb48c0 0x5604e8fbe7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:25.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.326+0000 7f8379c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5604e8fb48c0 0x5604e8fbe7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:53418/0 (socket says 172.21.15.39:53418) 2024-03-18T16:14:25.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.326+0000 7f8379c00640 1 -- 172.21.15.39:0/3389927463 learned_addr learned my addr 172.21.15.39:0/3389927463 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:14:25.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.326+0000 7f837f373740 10 client.0 _ll_register_callbacks cb 0x5604e8ec2b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:14:25.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.326+0000 7f837a600640 1 --2- 172.21.15.39:0/3389927463 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5604e8fba420 0x5604e8fc5d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:25.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.326+0000 7f8379c00640 1 -- 172.21.15.39:0/3389927463 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5604e8fba420 msgr2=0x5604e8fc5d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:14:25.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.326+0000 7f8379c00640 1 --2- 172.21.15.39:0/3389927463 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5604e8fba420 0x5604e8fc5d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:25.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.326+0000 7f8379c00640 1 -- 172.21.15.39:0/3389927463 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5604e8fb52b0 msgr2=0x5604e8fbed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:14:25.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.326+0000 7f8379c00640 1 --2- 172.21.15.39:0/3389927463 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5604e8fb52b0 0x5604e8fbed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:25.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.326+0000 7f8379c00640 1 -- 172.21.15.39:0/3389927463 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5604e8c2d3f0 con 0x5604e8fb48c0 2024-03-18T16:14:25.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.327+0000 7f8379c00640 1 --2- 172.21.15.39:0/3389927463 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5604e8fb48c0 0x5604e8fbe7f0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f83680f0b20 tx=0x7f83680baf20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=276060ca33640414 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:14:25.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.327+0000 7f835b400640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:14:25.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.327+0000 7f835b400640 1 -- 172.21.15.39:0/3389927463 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83680db050 con 0x5604e8fb48c0 2024-03-18T16:14:25.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.327+0000 7f835b400640 1 -- 172.21.15.39:0/3389927463 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f8368034650 con 0x5604e8fb48c0 2024-03-18T16:14:25.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.327+0000 7f837f373740 1 -- 172.21.15.39:0/3389927463 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f83680bd4e0 con 0x5604e8fb48c0 2024-03-18T16:14:25.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.328+0000 7f835b400640 1 -- 172.21.15.39:0/3389927463 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f83680c0f30 con 0x5604e8fb48c0 2024-03-18T16:14:25.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.328+0000 7f837f373740 10 client.26476 fetch_fsmap learned FSMap version 36 2024-03-18T16:14:25.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.328+0000 7f837f373740 1 -- 172.21.15.39:0/3389927463 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5604e8cf77d0 con 0x5604e8fb48c0 2024-03-18T16:14:25.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.328+0000 7f835b400640 1 -- 172.21.15.39:0/3389927463 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f83680bd4e0 con 0x5604e8fb48c0 2024-03-18T16:14:25.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.329+0000 7f835b400640 1 -- 172.21.15.39:0/3389927463 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 36) ==== 8406+0+0 (secure 0 0 0) 0x7f83680b8b30 con 0x5604e8fb48c0 2024-03-18T16:14:25.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.329+0000 7f837f373740 10 client.26476 fetch_fsmap finished waiting for FSMap version 36 2024-03-18T16:14:25.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.329+0000 7f837f373740 10 client.26476 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:14:25.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.329+0000 7f837f373740 20 client.26476 populate_metadata read hostname 'smithi039' 2024-03-18T16:14:25.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.329+0000 7f837f373740 1 --2- 172.21.15.39:0/3389927463 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5604e8fca3f0 0x5604e8fcc890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:25.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.329+0000 7f837f373740 4 client.26476 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:14:25.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.329+0000 7f837f373740 1 -- 172.21.15.39:0/3389927463 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5604e8f26990 con 0x5604e8fca3f0 2024-03-18T16:14:25.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.329+0000 7f8379200640 1 --2- 172.21.15.39:0/3389927463 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5604e8fca3f0 0x5604e8fcc890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:25.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.330+0000 7f8379200640 1 --2- 172.21.15.39:0/3389927463 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5604e8fca3f0 0x5604e8fcc890 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=836797527e6347ae server_cookie=8d9025abaef402d3 in_seq=0 out_seq=0 2024-03-18T16:14:25.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.330+0000 7f835b400640 10 client.26476 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:14:25.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.331+0000 7f835b400640 1 -- 172.21.15.39:0/3389927463 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5604e8f26990 con 0x5604e8fca3f0 2024-03-18T16:14:25.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.331+0000 7f835b400640 10 client.26476 handle_command_reply: tid=0 2024-03-18T16:14:25.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.331+0000 7f837f373740 1 client.26476 shutdown 2024-03-18T16:14:25.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.332+0000 7f837f373740 1 -- 172.21.15.39:0/3389927463 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5604e8fb48c0 msgr2=0x5604e8fbe7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:14:25.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.332+0000 7f837f373740 1 --2- 172.21.15.39:0/3389927463 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5604e8fb48c0 0x5604e8fbe7f0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f83680f0b20 tx=0x7f83680baf20 comp rx=0 tx=0).stop 2024-03-18T16:14:25.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.332+0000 7f837f373740 1 -- 172.21.15.39:0/3389927463 shutdown_connections 2024-03-18T16:14:25.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.332+0000 7f837f373740 1 --2- 172.21.15.39:0/3389927463 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5604e8fca3f0 0x5604e8fcc890 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:25.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.332+0000 7f837f373740 1 --2- 172.21.15.39:0/3389927463 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5604e8fba420 0x5604e8fc5d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:25.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.332+0000 7f837f373740 1 --2- 172.21.15.39:0/3389927463 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5604e8fb52b0 0x5604e8fbed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:25.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.332+0000 7f837f373740 1 --2- 172.21.15.39:0/3389927463 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5604e8fb48c0 0x5604e8fbe7f0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:25.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.332+0000 7f837f373740 1 -- 172.21.15.39:0/3389927463 >> 172.21.15.39:0/3389927463 conn(0x5604e8fafe30 msgr2=0x5604e8faf1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:14:25.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.332+0000 7f835b400640 0 client.26476 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:14:25.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.333+0000 7f837f373740 1 -- 172.21.15.39:0/3389927463 shutdown_connections 2024-03-18T16:14:25.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.333+0000 7f837f373740 1 -- 172.21.15.39:0/3389927463 wait complete. 2024-03-18T16:14:25.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.333+0000 7f837f373740 20 client.26476 trim_cache size 0 max 16384 2024-03-18T16:14:25.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.358+0000 7f837f373740 1 Processor -- start 2024-03-18T16:14:25.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.358+0000 7f837f373740 1 -- start start 2024-03-18T16:14:25.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.359+0000 7f837f373740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5604e8fb48c0 0x5604e8fbec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:25.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.359+0000 7f837f373740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5604e8fb52b0 0x5604e8fbdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:25.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.359+0000 7f837f373740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5604e8fba420 0x5604e8fae960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:25.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.359+0000 7f837f373740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f835400fbf0 con 0x5604e8fb48c0 2024-03-18T16:14:25.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.359+0000 7f837f373740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f83540118b0 con 0x5604e8fba420 2024-03-18T16:14:25.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.359+0000 7f837f373740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f83540102c0 con 0x5604e8fb52b0 2024-03-18T16:14:25.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.359+0000 7f8379200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5604e8fb52b0 0x5604e8fbdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:25.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.359+0000 7f837a600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5604e8fba420 0x5604e8fae960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:25.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.359+0000 7f8379200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5604e8fb52b0 0x5604e8fbdbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:53434/0 (socket says 172.21.15.39:53434) 2024-03-18T16:14:25.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.359+0000 7f8379200640 1 -- 172.21.15.39:0/1355256779 learned_addr learned my addr 172.21.15.39:0/1355256779 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:14:25.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.359+0000 7f837a600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5604e8fba420 0x5604e8fae960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:49092/0 (socket says 172.21.15.39:49092) 2024-03-18T16:14:25.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.360+0000 7f8379c00640 1 --2- 172.21.15.39:0/1355256779 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5604e8fb48c0 0x5604e8fbec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:25.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.360+0000 7f8379200640 1 -- 172.21.15.39:0/1355256779 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5604e8fba420 msgr2=0x5604e8fae960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:14:25.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.360+0000 7f8379200640 1 --2- 172.21.15.39:0/1355256779 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5604e8fba420 0x5604e8fae960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:25.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.360+0000 7f8379200640 1 -- 172.21.15.39:0/1355256779 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5604e8fb48c0 msgr2=0x5604e8fbec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:14:25.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.360+0000 7f8379200640 1 --2- 172.21.15.39:0/1355256779 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5604e8fb48c0 0x5604e8fbec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:25.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.360+0000 7f8379200640 1 -- 172.21.15.39:0/1355256779 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5604e8fabc10 con 0x5604e8fb52b0 2024-03-18T16:14:25.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.360+0000 7f8379c00640 1 --2- 172.21.15.39:0/1355256779 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5604e8fb48c0 0x5604e8fbec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:14:25.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.360+0000 7f8379200640 1 --2- 172.21.15.39:0/1355256779 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5604e8fb52b0 0x5604e8fbdbd0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x5604e8fbfcd0 tx=0x7f8360005850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b09729ba660f41bd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:14:25.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.361+0000 7f8366000640 1 -- 172.21.15.39:0/1355256779 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f836001a3f0 con 0x5604e8fb52b0 2024-03-18T16:14:25.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.361+0000 7f8366000640 1 -- 172.21.15.39:0/1355256779 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8360013e10 con 0x5604e8fb52b0 2024-03-18T16:14:25.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.361+0000 7f8366000640 1 -- 172.21.15.39:0/1355256779 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8360011aa0 con 0x5604e8fb52b0 2024-03-18T16:14:25.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.362+0000 7f837f373740 1 -- 172.21.15.39:0/1355256779 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5604e8fb52b0 msgr2=0x5604e8fbdbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:14:25.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.362+0000 7f837f373740 1 --2- 172.21.15.39:0/1355256779 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5604e8fb52b0 0x5604e8fbdbd0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x5604e8fbfcd0 tx=0x7f8360005850 comp rx=0 tx=0).stop 2024-03-18T16:14:25.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.362+0000 7f837f373740 1 -- 172.21.15.39:0/1355256779 shutdown_connections 2024-03-18T16:14:25.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.362+0000 7f837f373740 1 --2- 172.21.15.39:0/1355256779 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5604e8fba420 0x5604e8fae960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:25.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.362+0000 7f837f373740 1 --2- 172.21.15.39:0/1355256779 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5604e8fb52b0 0x5604e8fbdbd0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:25.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.362+0000 7f837f373740 1 --2- 172.21.15.39:0/1355256779 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5604e8fb48c0 0x5604e8fbec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:25.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.362+0000 7f837f373740 1 -- 172.21.15.39:0/1355256779 >> 172.21.15.39:0/1355256779 conn(0x5604e8fafe30 msgr2=0x5604e8fbd4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:14:25.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.362+0000 7f837f373740 1 -- 172.21.15.39:0/1355256779 shutdown_connections 2024-03-18T16:14:25.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.362+0000 7f837f373740 1 -- 172.21.15.39:0/1355256779 wait complete. 2024-03-18T16:14:25.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.362+0000 7f837f373740 1 -- start start 2024-03-18T16:14:25.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.362+0000 7f837f373740 1 Processor -- start 2024-03-18T16:14:25.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.363+0000 7f837f373740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5604e8fb48c0 0x5604e8fc3180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:25.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.363+0000 7f837f373740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5604e8fb52b0 0x5604e8fc36c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:25.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.363+0000 7f837f373740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5604e8fba420 0x5604e8fc46d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:25.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.363+0000 7f837f373740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f8354011120 con 0x5604e8fba420 2024-03-18T16:14:25.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.363+0000 7f837f373740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f8354010990 con 0x5604e8fb52b0 2024-03-18T16:14:25.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.363+0000 7f837f373740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f835400dbd0 con 0x5604e8fb48c0 2024-03-18T16:14:25.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.363+0000 7f837f373740 10 client.0 _ll_register_callbacks cb 0x7f83680b8ff0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:14:25.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.363+0000 7f8379c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5604e8fb48c0 0x5604e8fc3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:25.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.363+0000 7f8379c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5604e8fb48c0 0x5604e8fc3180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:53446/0 (socket says 172.21.15.39:53446) 2024-03-18T16:14:25.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.363+0000 7f8379c00640 1 -- 172.21.15.39:0/757739704 learned_addr learned my addr 172.21.15.39:0/757739704 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:14:25.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.363+0000 7f8379200640 1 --2- 172.21.15.39:0/757739704 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5604e8fb52b0 0x5604e8fc36c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:25.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.363+0000 7f837a600640 1 --2- 172.21.15.39:0/757739704 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5604e8fba420 0x5604e8fc46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:25.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.364+0000 7f8379c00640 1 -- 172.21.15.39:0/757739704 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5604e8fb52b0 msgr2=0x5604e8fc36c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:14:25.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.364+0000 7f8379c00640 1 --2- 172.21.15.39:0/757739704 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5604e8fb52b0 0x5604e8fc36c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:25.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.364+0000 7f8379c00640 1 -- 172.21.15.39:0/757739704 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5604e8fba420 msgr2=0x5604e8fc46d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:14:25.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.364+0000 7f8379c00640 1 --2- 172.21.15.39:0/757739704 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5604e8fba420 0x5604e8fc46d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:25.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.364+0000 7f8379c00640 1 -- 172.21.15.39:0/757739704 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5604e8fc4cc0 con 0x5604e8fb48c0 2024-03-18T16:14:25.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.364+0000 7f8379c00640 1 --2- 172.21.15.39:0/757739704 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5604e8fb48c0 0x5604e8fc3180 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f8368006010 tx=0x7f83680bc800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d719ad8fd14048c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:14:25.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.364+0000 7f835b400640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:14:25.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.364+0000 7f835b400640 1 -- 172.21.15.39:0/757739704 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83680bc920 con 0x5604e8fb48c0 2024-03-18T16:14:25.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.364+0000 7f835b400640 1 -- 172.21.15.39:0/757739704 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f83680bc230 con 0x5604e8fb48c0 2024-03-18T16:14:25.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.364+0000 7f837f373740 1 -- 172.21.15.39:0/757739704 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8360013e10 con 0x5604e8fb48c0 2024-03-18T16:14:25.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.365+0000 7f835b400640 1 -- 172.21.15.39:0/757739704 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f83680bd4e0 con 0x5604e8fb48c0 2024-03-18T16:14:25.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.365+0000 7f837f373740 10 client.26482 fetch_fsmap learned FSMap version 36 2024-03-18T16:14:25.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.365+0000 7f837f373740 1 -- 172.21.15.39:0/757739704 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5604e8fc4cc0 con 0x5604e8fb48c0 2024-03-18T16:14:25.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.365+0000 7f835b400640 1 -- 172.21.15.39:0/757739704 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8360013e10 con 0x5604e8fb48c0 2024-03-18T16:14:25.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.365+0000 7f835b400640 1 -- 172.21.15.39:0/757739704 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 36) ==== 8406+0+0 (secure 0 0 0) 0x7f83680c1ea0 con 0x5604e8fb48c0 2024-03-18T16:14:25.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.365+0000 7f837f373740 10 client.26482 fetch_fsmap finished waiting for FSMap version 36 2024-03-18T16:14:25.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.365+0000 7f837f373740 10 client.26482 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:14:25.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.365+0000 7f837f373740 20 client.26482 populate_metadata read hostname 'smithi039' 2024-03-18T16:14:25.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.365+0000 7f837f373740 1 --2- 172.21.15.39:0/757739704 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5604e9030f20 0x5604e9033340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:25.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.365+0000 7f837f373740 4 client.26482 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:14:25.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.365+0000 7f837f373740 1 -- 172.21.15.39:0/757739704 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5604e8f20ce0 con 0x5604e9030f20 2024-03-18T16:14:25.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.366+0000 7f8379200640 1 --2- 172.21.15.39:0/757739704 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5604e9030f20 0x5604e9033340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:25.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.367+0000 7f8379200640 1 --2- 172.21.15.39:0/757739704 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5604e9030f20 0x5604e9033340 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e6c65bda365ad9de server_cookie=d391c4d3bfdecaca in_seq=0 out_seq=0 2024-03-18T16:14:25.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.367+0000 7f835b400640 10 client.26482 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:14:25.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.368+0000 7f835b400640 1 -- 172.21.15.39:0/757739704 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5604e8f20ce0 con 0x5604e9030f20 2024-03-18T16:14:25.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.368+0000 7f835b400640 10 client.26482 handle_command_reply: tid=0 2024-03-18T16:14:25.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.368+0000 7f837f373740 1 client.26482 shutdown 2024-03-18T16:14:25.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.368+0000 7f837f373740 1 -- 172.21.15.39:0/757739704 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5604e8fb48c0 msgr2=0x5604e8fc3180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:14:25.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.368+0000 7f837f373740 1 --2- 172.21.15.39:0/757739704 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5604e8fb48c0 0x5604e8fc3180 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f8368006010 tx=0x7f83680bc800 comp rx=0 tx=0).stop 2024-03-18T16:14:25.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.369+0000 7f837f373740 1 -- 172.21.15.39:0/757739704 shutdown_connections 2024-03-18T16:14:25.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.369+0000 7f837f373740 1 --2- 172.21.15.39:0/757739704 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5604e9030f20 0x5604e9033340 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:25.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.369+0000 7f837f373740 1 --2- 172.21.15.39:0/757739704 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5604e8fb52b0 0x5604e8fc36c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:25.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.369+0000 7f837f373740 1 --2- 172.21.15.39:0/757739704 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5604e8fba420 0x5604e8fc46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:25.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.369+0000 7f837f373740 1 --2- 172.21.15.39:0/757739704 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5604e8fb48c0 0x5604e8fc3180 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:25.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.369+0000 7f837f373740 1 -- 172.21.15.39:0/757739704 >> 172.21.15.39:0/757739704 conn(0x5604e8fafe30 msgr2=0x5604e8fbeb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:14:25.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.369+0000 7f835b400640 0 client.26482 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:14:25.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.369+0000 7f837f373740 1 -- 172.21.15.39:0/757739704 shutdown_connections 2024-03-18T16:14:25.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.369+0000 7f837f373740 1 -- 172.21.15.39:0/757739704 wait complete. 2024-03-18T16:14:25.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.369+0000 7f837f373740 20 client.26482 trim_cache size 0 max 16384 2024-03-18T16:14:25.372 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:14:25.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.370+0000 7f837c600640 1 -- 172.21.15.39:0/2896074944 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f835c0a2980 msgr2=0x7f835c0a4e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:14:25.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.370+0000 7f837c600640 1 --2- 172.21.15.39:0/2896074944 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f835c0a2980 0x7f835c0a4e40 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f836c009fd0 tx=0x7f836c004b50 comp rx=0 tx=0).stop 2024-03-18T16:14:25.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.370+0000 7f837c600640 1 -- 172.21.15.39:0/2896074944 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f83741087e0 msgr2=0x7f83741182d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:14:25.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.370+0000 7f837c600640 1 --2- 172.21.15.39:0/2896074944 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f83741087e0 0x7f83741182d0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f8368000c00 tx=0x7f83680053c0 comp rx=0 tx=0).stop 2024-03-18T16:14:25.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.370+0000 7f837c600640 1 -- 172.21.15.39:0/2896074944 shutdown_connections 2024-03-18T16:14:25.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.370+0000 7f837c600640 1 --2- 172.21.15.39:0/2896074944 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f837410db00 0x7f83741196e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:25.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.370+0000 7f837c600640 1 --2- 172.21.15.39:0/2896074944 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f835c0a2980 0x7f835c0a4e40 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:25.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.370+0000 7f837c600640 1 --2- 172.21.15.39:0/2896074944 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f837410b150 0x7f8374118810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:25.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.370+0000 7f837c600640 1 --2- 172.21.15.39:0/2896074944 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f83741087e0 0x7f83741182d0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:25.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.370+0000 7f837c600640 1 -- 172.21.15.39:0/2896074944 >> 172.21.15.39:0/2896074944 conn(0x7f837406e860 msgr2=0x7f8374108ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:14:25.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.372+0000 7f837c600640 1 -- 172.21.15.39:0/2896074944 shutdown_connections 2024-03-18T16:14:25.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:25.372+0000 7f837c600640 1 -- 172.21.15.39:0/2896074944 wait complete. 2024-03-18T16:14:25.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:14:25 smithi196 ceph-mon[36307]: pgmap v552: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 38 MiB/s wr, 6.26k op/s 2024-03-18T16:14:25.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:14:25 smithi194 ceph-mon[36200]: pgmap v552: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 38 MiB/s wr, 6.26k op/s 2024-03-18T16:14:25.839 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:14:25 smithi039 ceph-mon[31721]: pgmap v552: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 38 MiB/s wr, 6.26k op/s 2024-03-18T16:14:26.398 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-18T16:14:26.398 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T16:14:26.398 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-18T16:14:26.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.724+0000 7f621e000640 1 -- 172.21.15.39:0/4136737631 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f621806a5f0 msgr2=0x7f621810b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:14:26.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.724+0000 7f621e000640 1 --2- 172.21.15.39:0/4136737631 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f621806a5f0 0x7f621810b2f0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f6200009860 tx=0x7f620002bb70 comp rx=0 tx=0).stop 2024-03-18T16:14:26.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.724+0000 7f621e000640 1 -- 172.21.15.39:0/4136737631 shutdown_connections 2024-03-18T16:14:26.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.724+0000 7f621e000640 1 --2- 172.21.15.39:0/4136737631 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f621810b8c0 0x7f621810dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:26.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.724+0000 7f621e000640 1 --2- 172.21.15.39:0/4136737631 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f621810e320 0x7f62181107b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:26.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.724+0000 7f621e000640 1 --2- 172.21.15.39:0/4136737631 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f621806a5f0 0x7f621810b2f0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:26.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.724+0000 7f621e000640 1 -- 172.21.15.39:0/4136737631 >> 172.21.15.39:0/4136737631 conn(0x7f621806e9e0 msgr2=0x7f6218070e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:14:26.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.724+0000 7f621e000640 1 -- 172.21.15.39:0/4136737631 shutdown_connections 2024-03-18T16:14:26.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.727+0000 7f621e000640 1 -- 172.21.15.39:0/4136737631 wait complete. 2024-03-18T16:14:26.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.728+0000 7f621e000640 1 Processor -- start 2024-03-18T16:14:26.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.728+0000 7f621e000640 1 -- start start 2024-03-18T16:14:26.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.729+0000 7f621e000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f621806a5f0 0x7f62181a25f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:26.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.729+0000 7f621e000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f621810b8c0 0x7f62181a2b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:26.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.729+0000 7f621e000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f621810e320 0x7f62181a9ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:26.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.729+0000 7f6217400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f621806a5f0 0x7f62181a25f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:26.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.730+0000 7f621e000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f6218115d00 con 0x7f621806a5f0 2024-03-18T16:14:26.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.730+0000 7f621e000640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f6218115b80 con 0x7f621810e320 2024-03-18T16:14:26.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.730+0000 7f621e000640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f6218115e80 con 0x7f621810b8c0 2024-03-18T16:14:26.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.730+0000 7f6216a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f621810b8c0 0x7f62181a2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:26.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.730+0000 7f6216a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f621810b8c0 0x7f62181a2b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:53472/0 (socket says 172.21.15.39:53472) 2024-03-18T16:14:26.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.730+0000 7f6216a00640 1 -- 172.21.15.39:0/333077215 learned_addr learned my addr 172.21.15.39:0/333077215 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:14:26.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.730+0000 7f6217e00640 1 --2- 172.21.15.39:0/333077215 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f621810e320 0x7f62181a9ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:26.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.730+0000 7f6216a00640 1 -- 172.21.15.39:0/333077215 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f621810e320 msgr2=0x7f62181a9ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:14:26.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.730+0000 7f6216a00640 1 --2- 172.21.15.39:0/333077215 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f621810e320 0x7f62181a9ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:26.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.730+0000 7f6216a00640 1 -- 172.21.15.39:0/333077215 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f621806a5f0 msgr2=0x7f62181a25f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:14:26.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.730+0000 7f6216a00640 1 --2- 172.21.15.39:0/333077215 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f621806a5f0 0x7f62181a25f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:26.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.730+0000 7f6216a00640 1 -- 172.21.15.39:0/333077215 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62181aa210 con 0x7f621810b8c0 2024-03-18T16:14:26.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.731+0000 7f6216a00640 1 --2- 172.21.15.39:0/333077215 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f621810b8c0 0x7f62181a2b30 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f6208005c40 tx=0x7f62080095b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:14:26.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.731+0000 7f620fe00640 1 -- 172.21.15.39:0/333077215 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6208003970 con 0x7f621810b8c0 2024-03-18T16:14:26.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.731+0000 7f621e000640 1 -- 172.21.15.39:0/333077215 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62181aa500 con 0x7f621810b8c0 2024-03-18T16:14:26.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.731+0000 7f621e000640 1 -- 172.21.15.39:0/333077215 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62181aab20 con 0x7f621810b8c0 2024-03-18T16:14:26.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.732+0000 7f620fe00640 1 -- 172.21.15.39:0/333077215 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6208004020 con 0x7f621810b8c0 2024-03-18T16:14:26.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.732+0000 7f620fe00640 1 -- 172.21.15.39:0/333077215 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6208015670 con 0x7f621810b8c0 2024-03-18T16:14:26.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.740+0000 7f620fe00640 1 -- 172.21.15.39:0/333077215 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f620801c020 con 0x7f621810b8c0 2024-03-18T16:14:26.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.741+0000 7f6220d7b740 1 Processor -- start 2024-03-18T16:14:26.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.741+0000 7f6220d7b740 1 -- start start 2024-03-18T16:14:26.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.742+0000 7f6220d7b740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5649df2787e0 0x5649df278c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:26.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.742+0000 7f6220d7b740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5649df2791d0 0x5649df27dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:26.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.742+0000 7f6220d7b740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649df27e340 0x5649df2807d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:26.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.742+0000 7f6220d7b740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5649df1dbf80 con 0x5649df2787e0 2024-03-18T16:14:26.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.742+0000 7f6220d7b740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5649df184ad0 con 0x5649df27e340 2024-03-18T16:14:26.745 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.742+0000 7f6220d7b740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x5649df1d8020 con 0x5649df2791d0 2024-03-18T16:14:26.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.743+0000 7f6217400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5649df2791d0 0x5649df27dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:26.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.743+0000 7f6217400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5649df2791d0 0x5649df27dcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:53486/0 (socket says 172.21.15.39:53486) 2024-03-18T16:14:26.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.743+0000 7f6217400640 1 -- 172.21.15.39:0/493766328 learned_addr learned my addr 172.21.15.39:0/493766328 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:14:26.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.743+0000 7f620fe00640 1 --2- 172.21.15.39:0/333077215 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f61fc0a2880 0x7f61fc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:26.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.744+0000 7f620fe00640 1 -- 172.21.15.39:0/333077215 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f62080ce1c0 con 0x7f621810b8c0 2024-03-18T16:14:26.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.744+0000 7f6217400640 1 -- 172.21.15.39:0/493766328 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649df27e340 msgr2=0x5649df2807d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:14:26.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.744+0000 7f6217400640 1 --2- 172.21.15.39:0/493766328 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649df27e340 0x5649df2807d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:26.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.744+0000 7f6217400640 1 -- 172.21.15.39:0/493766328 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5649df2787e0 msgr2=0x5649df278c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:14:26.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.744+0000 7f6217400640 1 --2- 172.21.15.39:0/493766328 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5649df2787e0 0x5649df278c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:26.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.744+0000 7f6217e00640 1 --2- 172.21.15.39:0/333077215 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f61fc0a2880 0x7f61fc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:26.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.744+0000 7f6217400640 1 -- 172.21.15.39:0/493766328 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5649df19c6f0 con 0x5649df2791d0 2024-03-18T16:14:26.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.745+0000 7f6217400640 1 --2- 172.21.15.39:0/493766328 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5649df2791d0 0x5649df27dcd0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f62000057c0 tx=0x7f6200009530 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cb5b0f49e128aa79 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:14:26.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.746+0000 7f6207e00640 1 -- 172.21.15.39:0/493766328 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6200043020 con 0x5649df2791d0 2024-03-18T16:14:26.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.746+0000 7f6207e00640 1 -- 172.21.15.39:0/493766328 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6200034db0 con 0x5649df2791d0 2024-03-18T16:14:26.749 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.746+0000 7f6207e00640 1 -- 172.21.15.39:0/493766328 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6200039040 con 0x5649df2791d0 2024-03-18T16:14:26.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.747+0000 7f6220d7b740 1 -- 172.21.15.39:0/493766328 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5649df2791d0 msgr2=0x5649df27dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:14:26.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.747+0000 7f6220d7b740 1 --2- 172.21.15.39:0/493766328 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5649df2791d0 0x5649df27dcd0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f62000057c0 tx=0x7f6200009530 comp rx=0 tx=0).stop 2024-03-18T16:14:26.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.747+0000 7f6217e00640 1 --2- 172.21.15.39:0/333077215 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f61fc0a2880 0x7f61fc0a4d40 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x5649df279a30 tx=0x7f6210008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:14:26.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.748+0000 7f6220d7b740 1 -- 172.21.15.39:0/493766328 shutdown_connections 2024-03-18T16:14:26.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.748+0000 7f6220d7b740 1 --2- 172.21.15.39:0/493766328 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649df27e340 0x5649df2807d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:26.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.748+0000 7f6220d7b740 1 --2- 172.21.15.39:0/493766328 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5649df2791d0 0x5649df27dcd0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:26.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.748+0000 7f6220d7b740 1 --2- 172.21.15.39:0/493766328 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5649df2787e0 0x5649df278c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:26.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.748+0000 7f6220d7b740 1 -- 172.21.15.39:0/493766328 >> 172.21.15.39:0/493766328 conn(0x5649df240930 msgr2=0x5649df274150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:14:26.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.748+0000 7f6220d7b740 1 -- 172.21.15.39:0/493766328 shutdown_connections 2024-03-18T16:14:26.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.750+0000 7f6220d7b740 1 -- 172.21.15.39:0/493766328 wait complete. 2024-03-18T16:14:26.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.750+0000 7f6220d7b740 1 -- start start 2024-03-18T16:14:26.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.751+0000 7f6220d7b740 1 Processor -- start 2024-03-18T16:14:26.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.753+0000 7f6220d7b740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5649df2787e0 0x5649df282910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:26.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.753+0000 7f6220d7b740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5649df2791d0 0x5649df282e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:26.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.753+0000 7f6220d7b740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649df27e340 0x5649df289e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:26.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.753+0000 7f6220d7b740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5649df281290 con 0x5649df2791d0 2024-03-18T16:14:26.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.753+0000 7f6220d7b740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5649df197f30 con 0x5649df27e340 2024-03-18T16:14:26.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.753+0000 7f6220d7b740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x5649df281410 con 0x5649df2787e0 2024-03-18T16:14:26.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.753+0000 7f6220d7b740 10 client.0 _ll_register_callbacks cb 0x5649df185bc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:14:26.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.754+0000 7f6217400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5649df2787e0 0x5649df282910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:26.757 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.754+0000 7f6217e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649df27e340 0x5649df289e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:26.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.754+0000 7f6217e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649df27e340 0x5649df289e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:49136/0 (socket says 172.21.15.39:49136) 2024-03-18T16:14:26.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.754+0000 7f6217e00640 1 -- 172.21.15.39:0/2067906034 learned_addr learned my addr 172.21.15.39:0/2067906034 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:14:26.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.755+0000 7f6217e00640 1 -- 172.21.15.39:0/2067906034 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5649df2787e0 msgr2=0x5649df282910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:14:26.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.755+0000 7f6217e00640 1 --2- 172.21.15.39:0/2067906034 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5649df2787e0 0x5649df282910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:26.758 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.755+0000 7f6217e00640 1 -- 172.21.15.39:0/2067906034 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5649df2791d0 msgr2=0x5649df282e50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:14:26.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.755+0000 7f6217e00640 1 --2- 172.21.15.39:0/2067906034 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5649df2791d0 0x5649df282e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:26.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.756+0000 7f6217e00640 1 -- 172.21.15.39:0/2067906034 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5649defb87e0 con 0x5649df27e340 2024-03-18T16:14:26.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.756+0000 7f6217e00640 1 --2- 172.21.15.39:0/2067906034 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649df27e340 0x5649df289e90 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f6210006df0 tx=0x7f62100046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=42b9c6d3da72fead server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:14:26.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.756+0000 7f6204c00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:14:26.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.757+0000 7f6204c00640 1 -- 172.21.15.39:0/2067906034 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62100137d0 con 0x5649df27e340 2024-03-18T16:14:26.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.757+0000 7f6204c00640 1 -- 172.21.15.39:0/2067906034 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f6210013970 con 0x5649df27e340 2024-03-18T16:14:26.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.757+0000 7f6220d7b740 1 -- 172.21.15.39:0/2067906034 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6200034db0 con 0x5649df27e340 2024-03-18T16:14:26.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.764+0000 7f6204c00640 1 -- 172.21.15.39:0/2067906034 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f621002d050 con 0x5649df27e340 2024-03-18T16:14:26.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.764+0000 7f6220d7b740 10 client.26513 fetch_fsmap learned FSMap version 36 2024-03-18T16:14:26.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.764+0000 7f6220d7b740 1 -- 172.21.15.39:0/2067906034 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5649df018f70 con 0x5649df27e340 2024-03-18T16:14:26.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.765+0000 7f6204c00640 1 -- 172.21.15.39:0/2067906034 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6200034db0 con 0x5649df27e340 2024-03-18T16:14:26.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.766+0000 7f6204c00640 1 -- 172.21.15.39:0/2067906034 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 36) ==== 8406+0+0 (secure 0 0 0) 0x7f6210028030 con 0x5649df27e340 2024-03-18T16:14:26.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.766+0000 7f6220d7b740 10 client.26513 fetch_fsmap finished waiting for FSMap version 36 2024-03-18T16:14:26.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.766+0000 7f6220d7b740 10 client.26513 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:14:26.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.766+0000 7f6220d7b740 20 client.26513 populate_metadata read hostname 'smithi039' 2024-03-18T16:14:26.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.766+0000 7f6220d7b740 1 --2- 172.21.15.39:0/2067906034 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5649df28e930 0x5649df290d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:26.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.766+0000 7f6220d7b740 4 client.26513 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:14:26.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.766+0000 7f6220d7b740 1 -- 172.21.15.39:0/2067906034 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5649df1e94c0 con 0x5649df28e930 2024-03-18T16:14:26.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.767+0000 7f6217400640 1 --2- 172.21.15.39:0/2067906034 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5649df28e930 0x5649df290d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:26.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.768+0000 7f6217400640 1 --2- 172.21.15.39:0/2067906034 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5649df28e930 0x5649df290d50 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d68e0ae434a5801f server_cookie=fdcd93a70fff3b65 in_seq=0 out_seq=0 2024-03-18T16:14:26.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.768+0000 7f6204c00640 10 client.26513 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:14:26.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.771+0000 7f6204c00640 1 -- 172.21.15.39:0/2067906034 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5649df1e94c0 con 0x5649df28e930 2024-03-18T16:14:26.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.771+0000 7f6204c00640 10 client.26513 handle_command_reply: tid=0 2024-03-18T16:14:26.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.771+0000 7f6220d7b740 1 client.26513 shutdown 2024-03-18T16:14:26.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.772+0000 7f6220d7b740 1 -- 172.21.15.39:0/2067906034 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649df27e340 msgr2=0x5649df289e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:14:26.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.772+0000 7f6220d7b740 1 --2- 172.21.15.39:0/2067906034 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649df27e340 0x5649df289e90 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f6210006df0 tx=0x7f62100046c0 comp rx=0 tx=0).stop 2024-03-18T16:14:26.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.772+0000 7f6220d7b740 1 -- 172.21.15.39:0/2067906034 shutdown_connections 2024-03-18T16:14:26.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.772+0000 7f6220d7b740 1 --2- 172.21.15.39:0/2067906034 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5649df28e930 0x5649df290d50 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:26.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.772+0000 7f6220d7b740 1 --2- 172.21.15.39:0/2067906034 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649df27e340 0x5649df289e90 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:26.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.773+0000 7f6204c00640 0 client.26513 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:14:26.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.773+0000 7f6220d7b740 1 --2- 172.21.15.39:0/2067906034 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5649df2791d0 0x5649df282e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:26.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.773+0000 7f6220d7b740 1 --2- 172.21.15.39:0/2067906034 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5649df2787e0 0x5649df282910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:26.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.773+0000 7f6220d7b740 1 -- 172.21.15.39:0/2067906034 >> 172.21.15.39:0/2067906034 conn(0x5649df240930 msgr2=0x5649df273350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:14:26.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.775+0000 7f6220d7b740 1 -- 172.21.15.39:0/2067906034 shutdown_connections 2024-03-18T16:14:26.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.775+0000 7f6220d7b740 1 -- 172.21.15.39:0/2067906034 wait complete. 2024-03-18T16:14:26.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.775+0000 7f6220d7b740 20 client.26513 trim_cache size 0 max 16384 2024-03-18T16:14:26.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.817+0000 7f6220d7b740 1 Processor -- start 2024-03-18T16:14:26.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.817+0000 7f6220d7b740 1 -- start start 2024-03-18T16:14:26.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.818+0000 7f6220d7b740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649df2787e0 0x5649df282310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:26.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.818+0000 7f6220d7b740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5649df2791d0 0x5649df289e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:26.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.818+0000 7f6220d7b740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5649df27e340 0x5649df277390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:26.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.818+0000 7f6220d7b740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f61f400fba0 con 0x5649df27e340 2024-03-18T16:14:26.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.818+0000 7f6220d7b740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f61f4011860 con 0x5649df2787e0 2024-03-18T16:14:26.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.818+0000 7f6220d7b740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f61f4010270 con 0x5649df2791d0 2024-03-18T16:14:26.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.819+0000 7f6217400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649df2787e0 0x5649df282310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:26.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.819+0000 7f6217400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649df2787e0 0x5649df282310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:49152/0 (socket says 172.21.15.39:49152) 2024-03-18T16:14:26.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.819+0000 7f6217400640 1 -- 172.21.15.39:0/1670666677 learned_addr learned my addr 172.21.15.39:0/1670666677 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:14:26.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.819+0000 7f6217e00640 1 --2- 172.21.15.39:0/1670666677 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5649df27e340 0x5649df277390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:26.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.819+0000 7f6216a00640 1 --2- 172.21.15.39:0/1670666677 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5649df2791d0 0x5649df289e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:26.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.820+0000 7f6217400640 1 -- 172.21.15.39:0/1670666677 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5649df2791d0 msgr2=0x5649df289e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:14:26.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.820+0000 7f6217400640 1 --2- 172.21.15.39:0/1670666677 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5649df2791d0 0x5649df289e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:26.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.820+0000 7f6217400640 1 -- 172.21.15.39:0/1670666677 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5649df27e340 msgr2=0x5649df277390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:14:26.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.820+0000 7f6217400640 1 --2- 172.21.15.39:0/1670666677 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5649df27e340 0x5649df277390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:26.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.820+0000 7f6217400640 1 -- 172.21.15.39:0/1670666677 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5649df2910c0 con 0x5649df2787e0 2024-03-18T16:14:26.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.821+0000 7f6217400640 1 --2- 172.21.15.39:0/1670666677 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649df2787e0 0x5649df282310 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f62000057c0 tx=0x7f620002c510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=24d7e3611fc30621 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:14:26.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.821+0000 7f6207400640 1 -- 172.21.15.39:0/1670666677 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6200043040 con 0x5649df2787e0 2024-03-18T16:14:26.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.822+0000 7f6207400640 1 -- 172.21.15.39:0/1670666677 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62000347c0 con 0x5649df2787e0 2024-03-18T16:14:26.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.822+0000 7f6207400640 1 -- 172.21.15.39:0/1670666677 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6200039040 con 0x5649df2787e0 2024-03-18T16:14:26.825 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.823+0000 7f6220d7b740 1 -- 172.21.15.39:0/1670666677 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649df2787e0 msgr2=0x5649df282310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:14:26.825 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.823+0000 7f6220d7b740 1 --2- 172.21.15.39:0/1670666677 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649df2787e0 0x5649df282310 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f62000057c0 tx=0x7f620002c510 comp rx=0 tx=0).stop 2024-03-18T16:14:26.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.823+0000 7f6220d7b740 1 -- 172.21.15.39:0/1670666677 shutdown_connections 2024-03-18T16:14:26.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.823+0000 7f6220d7b740 1 --2- 172.21.15.39:0/1670666677 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5649df27e340 0x5649df277390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:26.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.823+0000 7f6220d7b740 1 --2- 172.21.15.39:0/1670666677 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5649df2791d0 0x5649df289e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:26.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.823+0000 7f6220d7b740 1 --2- 172.21.15.39:0/1670666677 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649df2787e0 0x5649df282310 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:26.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.823+0000 7f6220d7b740 1 -- 172.21.15.39:0/1670666677 >> 172.21.15.39:0/1670666677 conn(0x5649df240930 msgr2=0x5649df281c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:14:26.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.824+0000 7f6220d7b740 1 -- 172.21.15.39:0/1670666677 shutdown_connections 2024-03-18T16:14:26.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.824+0000 7f6220d7b740 1 -- 172.21.15.39:0/1670666677 wait complete. 2024-03-18T16:14:26.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.825+0000 7f6220d7b740 1 -- start start 2024-03-18T16:14:26.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.826+0000 7f6220d7b740 1 Processor -- start 2024-03-18T16:14:26.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.827+0000 7f6220d7b740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649df2787e0 0x5649df287850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:26.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.827+0000 7f6220d7b740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5649df2791d0 0x5649df287d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:26.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.827+0000 7f6220d7b740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5649df27e340 0x5649df288da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:26.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.827+0000 7f6220d7b740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f61f40110d0 con 0x5649df2791d0 2024-03-18T16:14:26.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.827+0000 7f6220d7b740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f61f4010940 con 0x5649df2787e0 2024-03-18T16:14:26.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.827+0000 7f6220d7b740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f61f400db80 con 0x5649df27e340 2024-03-18T16:14:26.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.827+0000 7f6220d7b740 10 client.0 _ll_register_callbacks cb 0x7f6210017040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:14:26.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.828+0000 7f6217400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649df2787e0 0x5649df287850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:26.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.828+0000 7f6217400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649df2787e0 0x5649df287850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:49166/0 (socket says 172.21.15.39:49166) 2024-03-18T16:14:26.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.828+0000 7f6217400640 1 -- 172.21.15.39:0/2574912977 learned_addr learned my addr 172.21.15.39:0/2574912977 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:14:26.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.828+0000 7f6217e00640 1 --2- 172.21.15.39:0/2574912977 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5649df27e340 0x5649df288da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:26.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.828+0000 7f6217e00640 1 -- 172.21.15.39:0/2574912977 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649df2787e0 msgr2=0x5649df287850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:14:26.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.828+0000 7f6217e00640 1 --2- 172.21.15.39:0/2574912977 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649df2787e0 0x5649df287850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:26.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.829+0000 7f6217e00640 1 -- 172.21.15.39:0/2574912977 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5649df2791d0 msgr2=0x5649df287d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:14:26.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.829+0000 7f6217e00640 1 --2- 172.21.15.39:0/2574912977 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5649df2791d0 0x5649df287d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:26.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.829+0000 7f6217e00640 1 -- 172.21.15.39:0/2574912977 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5649df278f70 con 0x5649df27e340 2024-03-18T16:14:26.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.829+0000 7f6217e00640 1 --2- 172.21.15.39:0/2574912977 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5649df27e340 0x5649df288da0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f6210020c50 tx=0x7f6210014e00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ddbad32cb21485f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:14:26.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.829+0000 7f6204c00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:14:26.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.830+0000 7f6204c00640 1 -- 172.21.15.39:0/2574912977 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62100135d0 con 0x5649df27e340 2024-03-18T16:14:26.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.830+0000 7f6204c00640 1 -- 172.21.15.39:0/2574912977 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f6210013770 con 0x5649df27e340 2024-03-18T16:14:26.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.831+0000 7f6220d7b740 1 -- 172.21.15.39:0/2574912977 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f62000347c0 con 0x5649df27e340 2024-03-18T16:14:26.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.832+0000 7f6204c00640 1 -- 172.21.15.39:0/2574912977 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f6210043050 con 0x5649df27e340 2024-03-18T16:14:26.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.832+0000 7f6220d7b740 10 client.26497 fetch_fsmap learned FSMap version 36 2024-03-18T16:14:26.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.832+0000 7f6220d7b740 1 -- 172.21.15.39:0/2574912977 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x5649deeb0170 con 0x5649df27e340 2024-03-18T16:14:26.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.834+0000 7f6204c00640 1 -- 172.21.15.39:0/2574912977 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f62000347c0 con 0x5649df27e340 2024-03-18T16:14:26.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.836+0000 7f6204c00640 1 -- 172.21.15.39:0/2574912977 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 36) ==== 8406+0+0 (secure 0 0 0) 0x7f6210028050 con 0x5649df27e340 2024-03-18T16:14:26.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.836+0000 7f6220d7b740 10 client.26497 fetch_fsmap finished waiting for FSMap version 36 2024-03-18T16:14:26.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.836+0000 7f6220d7b740 10 client.26497 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:14:26.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.836+0000 7f6220d7b740 20 client.26497 populate_metadata read hostname 'smithi039' 2024-03-18T16:14:26.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.836+0000 7f6220d7b740 1 --2- 172.21.15.39:0/2574912977 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5649df28e930 0x5649df2f56b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:26.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.836+0000 7f6220d7b740 4 client.26497 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-18T16:14:26.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.836+0000 7f6220d7b740 1 -- 172.21.15.39:0/2574912977 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5649df1e3810 con 0x5649df28e930 2024-03-18T16:14:26.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.837+0000 7f6217400640 1 --2- 172.21.15.39:0/2574912977 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5649df28e930 0x5649df2f56b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:26.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.840+0000 7f6217400640 1 --2- 172.21.15.39:0/2574912977 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5649df28e930 0x5649df2f56b0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6eaa5f002bde2fc2 server_cookie=b2ebcb06610fc819 in_seq=0 out_seq=0 2024-03-18T16:14:26.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.840+0000 7f6204c00640 10 client.26497 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:14:26.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.841+0000 7f6204c00640 1 -- 172.21.15.39:0/2574912977 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5649df1e3810 con 0x5649df28e930 2024-03-18T16:14:26.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.841+0000 7f6204c00640 10 client.26497 handle_command_reply: tid=0 2024-03-18T16:14:26.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.841+0000 7f6220d7b740 1 client.26497 shutdown 2024-03-18T16:14:26.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.841+0000 7f6220d7b740 1 -- 172.21.15.39:0/2574912977 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5649df27e340 msgr2=0x5649df288da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:14:26.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.841+0000 7f6220d7b740 1 --2- 172.21.15.39:0/2574912977 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5649df27e340 0x5649df288da0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f6210020c50 tx=0x7f6210014e00 comp rx=0 tx=0).stop 2024-03-18T16:14:26.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.841+0000 7f6220d7b740 1 -- 172.21.15.39:0/2574912977 shutdown_connections 2024-03-18T16:14:26.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.841+0000 7f6220d7b740 1 --2- 172.21.15.39:0/2574912977 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5649df28e930 0x5649df2f56b0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:26.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.841+0000 7f6204c00640 0 client.26497 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:14:26.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.841+0000 7f6220d7b740 1 --2- 172.21.15.39:0/2574912977 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5649df27e340 0x5649df288da0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:26.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.841+0000 7f6220d7b740 1 --2- 172.21.15.39:0/2574912977 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5649df2791d0 0x5649df287d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:26.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.841+0000 7f6220d7b740 1 --2- 172.21.15.39:0/2574912977 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5649df2787e0 0x5649df287850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:26.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.841+0000 7f6220d7b740 1 -- 172.21.15.39:0/2574912977 >> 172.21.15.39:0/2574912977 conn(0x5649df240930 msgr2=0x5649df272ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:14:26.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.841+0000 7f6220d7b740 1 -- 172.21.15.39:0/2574912977 shutdown_connections 2024-03-18T16:14:26.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.841+0000 7f6220d7b740 1 -- 172.21.15.39:0/2574912977 wait complete. 2024-03-18T16:14:26.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.842+0000 7f6220d7b740 20 client.26497 trim_cache size 0 max 16384 2024-03-18T16:14:26.846 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-03-18T16:14:26.846 INFO:teuthology.orchestra.run.smithi039.stdout: "return_code": 0, 2024-03-18T16:14:26.846 INFO:teuthology.orchestra.run.smithi039.stdout: "scrub_tag": "9e29cb97-942d-44c9-ba66-4e00ebf99f56", 2024-03-18T16:14:26.846 INFO:teuthology.orchestra.run.smithi039.stdout: "mode": "asynchronous" 2024-03-18T16:14:26.846 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-03-18T16:14:26.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.843+0000 7f621e000640 1 -- 172.21.15.39:0/333077215 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f61fc0a2880 msgr2=0x7f61fc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:14:26.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.843+0000 7f621e000640 1 --2- 172.21.15.39:0/333077215 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f61fc0a2880 0x7f61fc0a4d40 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x5649df279a30 tx=0x7f6210008040 comp rx=0 tx=0).stop 2024-03-18T16:14:26.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.843+0000 7f621e000640 1 -- 172.21.15.39:0/333077215 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f621810b8c0 msgr2=0x7f62181a2b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:14:26.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.843+0000 7f621e000640 1 --2- 172.21.15.39:0/333077215 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f621810b8c0 0x7f62181a2b30 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f6208005c40 tx=0x7f62080095b0 comp rx=0 tx=0).stop 2024-03-18T16:14:26.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.843+0000 7f621e000640 1 -- 172.21.15.39:0/333077215 shutdown_connections 2024-03-18T16:14:26.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.843+0000 7f621e000640 1 --2- 172.21.15.39:0/333077215 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f621810e320 0x7f62181a9ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:26.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.843+0000 7f621e000640 1 --2- 172.21.15.39:0/333077215 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f61fc0a2880 0x7f61fc0a4d40 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:26.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.843+0000 7f621e000640 1 --2- 172.21.15.39:0/333077215 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f621810b8c0 0x7f62181a2b30 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:26.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.843+0000 7f621e000640 1 --2- 172.21.15.39:0/333077215 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f621806a5f0 0x7f62181a25f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:26.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.843+0000 7f621e000640 1 -- 172.21.15.39:0/333077215 >> 172.21.15.39:0/333077215 conn(0x7f621806e9e0 msgr2=0x7f6218112ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:14:26.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.843+0000 7f621e000640 1 -- 172.21.15.39:0/333077215 shutdown_connections 2024-03-18T16:14:26.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:26.845+0000 7f621e000640 1 -- 172.21.15.39:0/333077215 wait complete. 2024-03-18T16:14:26.880 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T16:14:26.880 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T16:14:27.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.220+0000 7f9882200640 1 -- 172.21.15.39:0/3728127220 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f987c102080 msgr2=0x7f987c104470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:14:27.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.220+0000 7f9882200640 1 --2- 172.21.15.39:0/3728127220 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f987c102080 0x7f987c104470 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f9870009900 tx=0x7f987002bc80 comp rx=0 tx=0).stop 2024-03-18T16:14:27.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.220+0000 7f9882200640 1 -- 172.21.15.39:0/3728127220 shutdown_connections 2024-03-18T16:14:27.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.220+0000 7f9882200640 1 --2- 172.21.15.39:0/3728127220 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f987c1072e0 0x7f987c109730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:27.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.220+0000 7f9882200640 1 --2- 172.21.15.39:0/3728127220 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f987c1049b0 0x7f987c106da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:27.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.220+0000 7f9882200640 1 --2- 172.21.15.39:0/3728127220 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f987c102080 0x7f987c104470 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:27.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.220+0000 7f9882200640 1 -- 172.21.15.39:0/3728127220 >> 172.21.15.39:0/3728127220 conn(0x7f987c0fbab0 msgr2=0x7f987c0fded0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:14:27.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.221+0000 7f9882200640 1 -- 172.21.15.39:0/3728127220 shutdown_connections 2024-03-18T16:14:27.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.221+0000 7f9882200640 1 -- 172.21.15.39:0/3728127220 wait complete. 2024-03-18T16:14:27.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.222+0000 7f9882200640 1 Processor -- start 2024-03-18T16:14:27.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.222+0000 7f9882200640 1 -- start start 2024-03-18T16:14:27.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.222+0000 7f9882200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f987c102080 0x7f987c196430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:27.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.222+0000 7f9882200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f987c1049b0 0x7f987c196970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:27.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.223+0000 7f9882200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f987c1072e0 0x7f987c192fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:27.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.223+0000 7f9882200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f987c10d3e0 con 0x7f987c1072e0 2024-03-18T16:14:27.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.223+0000 7f9882200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f987c10d260 con 0x7f987c1049b0 2024-03-18T16:14:27.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.223+0000 7f9882200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f987c10d560 con 0x7f987c102080 2024-03-18T16:14:27.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.223+0000 7f987be00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f987c1049b0 0x7f987c196970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:27.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.223+0000 7f987be00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f987c1049b0 0x7f987c196970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:49178/0 (socket says 172.21.15.39:49178) 2024-03-18T16:14:27.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.223+0000 7f987be00640 1 -- 172.21.15.39:0/4215373539 learned_addr learned my addr 172.21.15.39:0/4215373539 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:14:27.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.224+0000 7f9881800640 1 --2- 172.21.15.39:0/4215373539 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f987c1072e0 0x7f987c192fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:27.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.225+0000 7f987be00640 1 -- 172.21.15.39:0/4215373539 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f987c102080 msgr2=0x7f987c196430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:14:27.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.225+0000 7f987be00640 1 --2- 172.21.15.39:0/4215373539 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f987c102080 0x7f987c196430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:27.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.225+0000 7f987be00640 1 -- 172.21.15.39:0/4215373539 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f987c1072e0 msgr2=0x7f987c192fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:14:27.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.225+0000 7f987be00640 1 --2- 172.21.15.39:0/4215373539 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f987c1072e0 0x7f987c192fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:27.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.225+0000 7f987be00640 1 -- 172.21.15.39:0/4215373539 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f987c193860 con 0x7f987c1049b0 2024-03-18T16:14:27.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.225+0000 7f9881800640 1 --2- 172.21.15.39:0/4215373539 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f987c1072e0 0x7f987c192fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:14:27.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.226+0000 7f987be00640 1 --2- 172.21.15.39:0/4215373539 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f987c1049b0 0x7f987c196970 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f986800b470 tx=0x7f986800b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:14:27.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.227+0000 7f9879600640 1 -- 172.21.15.39:0/4215373539 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9868015020 con 0x7f987c1049b0 2024-03-18T16:14:27.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.227+0000 7f9879600640 1 -- 172.21.15.39:0/4215373539 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9868011d70 con 0x7f987c1049b0 2024-03-18T16:14:27.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.227+0000 7f9879600640 1 -- 172.21.15.39:0/4215373539 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f986800f340 con 0x7f987c1049b0 2024-03-18T16:14:27.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.227+0000 7f9882200640 1 -- 172.21.15.39:0/4215373539 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f987c19b840 con 0x7f987c1049b0 2024-03-18T16:14:27.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.228+0000 7f9882200640 1 -- 172.21.15.39:0/4215373539 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f987c19bd40 con 0x7f987c1049b0 2024-03-18T16:14:27.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.237+0000 7f9886599740 1 Processor -- start 2024-03-18T16:14:27.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.237+0000 7f9879600640 1 -- 172.21.15.39:0/4215373539 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f9868018020 con 0x7f987c1049b0 2024-03-18T16:14:27.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.237+0000 7f9886599740 1 -- start start 2024-03-18T16:14:27.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.238+0000 7f9886599740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56441ad5ba30 0x56441ad5be50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:27.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.238+0000 7f9886599740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56441ad5c420 0x56441ad60f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:27.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.238+0000 7f9886599740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56441ad61590 0x56441ad63a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:27.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.238+0000 7f9886599740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x56441acc01f0 con 0x56441ad5ba30 2024-03-18T16:14:27.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.238+0000 7f9886599740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x56441ac68a80 con 0x56441ad61590 2024-03-18T16:14:27.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.238+0000 7f9886599740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x56441acbc290 con 0x56441ad5c420 2024-03-18T16:14:27.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.239+0000 7f9880e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56441ad5c420 0x56441ad60f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:27.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.239+0000 7f9881800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56441ad5ba30 0x56441ad5be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:27.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.239+0000 7f9880e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56441ad5c420 0x56441ad60f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:53520/0 (socket says 172.21.15.39:53520) 2024-03-18T16:14:27.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.240+0000 7f9879600640 1 --2- 172.21.15.39:0/4215373539 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f98640a2a00 0x7f98640a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:27.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.240+0000 7f9879600640 1 -- 172.21.15.39:0/4215373539 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f98680ce660 con 0x7f987c1049b0 2024-03-18T16:14:27.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.239+0000 7f9880e00640 1 -- 172.21.15.39:0/3883794236 learned_addr learned my addr 172.21.15.39:0/3883794236 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:14:27.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.241+0000 7f987be00640 1 --2- 172.21.15.39:0/3883794236 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56441ad61590 0x56441ad63a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:27.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.241+0000 7f987be00640 1 -- 172.21.15.39:0/3883794236 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56441ad5c420 msgr2=0x56441ad60f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:14:27.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.242+0000 7f987be00640 1 --2- 172.21.15.39:0/3883794236 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56441ad5c420 0x56441ad60f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:27.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.242+0000 7f987be00640 1 -- 172.21.15.39:0/3883794236 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56441ad5ba30 msgr2=0x56441ad5be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:14:27.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.242+0000 7f987be00640 1 --2- 172.21.15.39:0/3883794236 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56441ad5ba30 0x56441ad5be50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:27.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.242+0000 7f987be00640 1 -- 172.21.15.39:0/3883794236 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56441aa9e7d0 con 0x56441ad61590 2024-03-18T16:14:27.245 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.242+0000 7f9881800640 1 --2- 172.21.15.39:0/3883794236 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56441ad5ba30 0x56441ad5be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:14:27.245 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.242+0000 7f9881800640 1 --2- 172.21.15.39:0/4215373539 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f98640a2a00 0x7f98640a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:27.245 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.243+0000 7f987be00640 1 --2- 172.21.15.39:0/3883794236 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56441ad61590 0x56441ad63a20 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f9868006fd0 tx=0x7f986809acd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=99c847c4f0a0f73b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:14:27.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.244+0000 7f986e000640 1 -- 172.21.15.39:0/3883794236 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98680b3070 con 0x56441ad61590 2024-03-18T16:14:27.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.244+0000 7f986e000640 1 -- 172.21.15.39:0/3883794236 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f98680cbe80 con 0x56441ad61590 2024-03-18T16:14:27.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.244+0000 7f986e000640 1 -- 172.21.15.39:0/3883794236 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f986809a2a0 con 0x56441ad61590 2024-03-18T16:14:27.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.245+0000 7f9886599740 1 -- 172.21.15.39:0/3883794236 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56441ad61590 msgr2=0x56441ad63a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:14:27.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.245+0000 7f9881800640 1 --2- 172.21.15.39:0/4215373539 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f98640a2a00 0x7f98640a4ec0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f9874007fd0 tx=0x7f9874004af0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:14:27.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.245+0000 7f9886599740 1 --2- 172.21.15.39:0/3883794236 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56441ad61590 0x56441ad63a20 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f9868006fd0 tx=0x7f986809acd0 comp rx=0 tx=0).stop 2024-03-18T16:14:27.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.245+0000 7f9886599740 1 -- 172.21.15.39:0/3883794236 shutdown_connections 2024-03-18T16:14:27.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.245+0000 7f9886599740 1 --2- 172.21.15.39:0/3883794236 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56441ad61590 0x56441ad63a20 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:27.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.245+0000 7f9886599740 1 --2- 172.21.15.39:0/3883794236 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56441ad5c420 0x56441ad60f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:27.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.245+0000 7f9886599740 1 --2- 172.21.15.39:0/3883794236 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56441ad5ba30 0x56441ad5be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:27.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.245+0000 7f9886599740 1 -- 172.21.15.39:0/3883794236 >> 172.21.15.39:0/3883794236 conn(0x56441ad56fa0 msgr2=0x56441ad573a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:14:27.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.246+0000 7f9886599740 1 -- 172.21.15.39:0/3883794236 shutdown_connections 2024-03-18T16:14:27.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.246+0000 7f9886599740 1 -- 172.21.15.39:0/3883794236 wait complete. 2024-03-18T16:14:27.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.246+0000 7f9886599740 1 -- start start 2024-03-18T16:14:27.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.246+0000 7f9886599740 1 Processor -- start 2024-03-18T16:14:27.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.247+0000 7f9886599740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56441ad5ba30 0x56441ad65fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:27.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.247+0000 7f9886599740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56441ad5c420 0x56441ad664f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:27.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.247+0000 7f9886599740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56441ad61590 0x56441ad6d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:27.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.247+0000 7f9880e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56441ad5ba30 0x56441ad65fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:27.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.248+0000 7f9880e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56441ad5ba30 0x56441ad65fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:49202/0 (socket says 172.21.15.39:49202) 2024-03-18T16:14:27.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.250+0000 7f9886599740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x56441ad64590 con 0x56441ad5c420 2024-03-18T16:14:27.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.250+0000 7f9886599740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x56441ac7bed0 con 0x56441ad5ba30 2024-03-18T16:14:27.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.250+0000 7f9886599740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x56441ad64710 con 0x56441ad61590 2024-03-18T16:14:27.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.250+0000 7f9880e00640 1 -- 172.21.15.39:0/1557981998 learned_addr learned my addr 172.21.15.39:0/1557981998 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:14:27.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.250+0000 7f9886599740 10 client.0 _ll_register_callbacks cb 0x56441acb7220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:14:27.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.250+0000 7f9881800640 1 --2- 172.21.15.39:0/1557981998 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56441ad61590 0x56441ad6d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:27.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.250+0000 7f987be00640 1 --2- 172.21.15.39:0/1557981998 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56441ad5c420 0x56441ad664f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:27.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.251+0000 7f9881800640 1 -- 172.21.15.39:0/1557981998 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56441ad5ba30 msgr2=0x56441ad65fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:14:27.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.251+0000 7f9881800640 1 --2- 172.21.15.39:0/1557981998 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56441ad5ba30 0x56441ad65fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:27.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.251+0000 7f9881800640 1 -- 172.21.15.39:0/1557981998 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56441ad5c420 msgr2=0x56441ad664f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:14:27.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.251+0000 7f9881800640 1 --2- 172.21.15.39:0/1557981998 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56441ad5c420 0x56441ad664f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:27.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.251+0000 7f9881800640 1 -- 172.21.15.39:0/1557981998 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56441a9d43f0 con 0x56441ad61590 2024-03-18T16:14:27.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.251+0000 7f987be00640 1 --2- 172.21.15.39:0/1557981998 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56441ad5c420 0x56441ad664f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:14:27.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.251+0000 7f9880e00640 1 --2- 172.21.15.39:0/1557981998 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56441ad5ba30 0x56441ad65fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:14:27.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.252+0000 7f9881800640 1 --2- 172.21.15.39:0/1557981998 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56441ad61590 0x56441ad6d530 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f987400ef50 tx=0x7f98740075d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ca21be433220a831 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:14:27.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.252+0000 7f9862a00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:14:27.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.253+0000 7f9862a00640 1 -- 172.21.15.39:0/1557981998 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9874021070 con 0x56441ad61590 2024-03-18T16:14:27.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.253+0000 7f9862a00640 1 -- 172.21.15.39:0/1557981998 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f9874011d20 con 0x56441ad61590 2024-03-18T16:14:27.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.254+0000 7f9886599740 1 -- 172.21.15.39:0/1557981998 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f98680cbe80 con 0x56441ad61590 2024-03-18T16:14:27.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.254+0000 7f9862a00640 1 -- 172.21.15.39:0/1557981998 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f987402b050 con 0x56441ad61590 2024-03-18T16:14:27.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.254+0000 7f9886599740 10 client.26503 fetch_fsmap learned FSMap version 36 2024-03-18T16:14:27.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.254+0000 7f9886599740 1 -- 172.21.15.39:0/1557981998 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x56441a9940f0 con 0x56441ad61590 2024-03-18T16:14:27.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.255+0000 7f9862a00640 1 -- 172.21.15.39:0/1557981998 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f98680cbe80 con 0x56441ad61590 2024-03-18T16:14:27.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.255+0000 7f9862a00640 1 -- 172.21.15.39:0/1557981998 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 36) ==== 8406+0+0 (secure 0 0 0) 0x7f987400d070 con 0x56441ad61590 2024-03-18T16:14:27.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.255+0000 7f9886599740 10 client.26503 fetch_fsmap finished waiting for FSMap version 36 2024-03-18T16:14:27.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.255+0000 7f9886599740 10 client.26503 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:14:27.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.255+0000 7f9886599740 20 client.26503 populate_metadata read hostname 'smithi039' 2024-03-18T16:14:27.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.255+0000 7f9886599740 1 --2- 172.21.15.39:0/1557981998 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x56441ad71f60 0x56441ad743e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:27.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.255+0000 7f9886599740 4 client.26503 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:14:27.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.255+0000 7f9886599740 1 -- 172.21.15.39:0/1557981998 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56441acd3be0 con 0x56441ad71f60 2024-03-18T16:14:27.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.258+0000 7f9880e00640 1 --2- 172.21.15.39:0/1557981998 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x56441ad71f60 0x56441ad743e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:27.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.260+0000 7f9880e00640 1 --2- 172.21.15.39:0/1557981998 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x56441ad71f60 0x56441ad743e0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fa8822f895431873 server_cookie=71fd40e48b6b6cb8 in_seq=0 out_seq=0 2024-03-18T16:14:27.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.262+0000 7f9862a00640 10 client.26503 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:14:27.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.264+0000 7f9862a00640 1 -- 172.21.15.39:0/1557981998 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56441acd3be0 con 0x56441ad71f60 2024-03-18T16:14:27.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.267+0000 7f9862a00640 10 client.26503 handle_command_reply: tid=0 2024-03-18T16:14:27.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.267+0000 7f9886599740 1 client.26503 shutdown 2024-03-18T16:14:27.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.268+0000 7f9886599740 1 -- 172.21.15.39:0/1557981998 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56441ad61590 msgr2=0x56441ad6d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:14:27.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.268+0000 7f9886599740 1 --2- 172.21.15.39:0/1557981998 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56441ad61590 0x56441ad6d530 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f987400ef50 tx=0x7f98740075d0 comp rx=0 tx=0).stop 2024-03-18T16:14:27.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.268+0000 7f9886599740 1 -- 172.21.15.39:0/1557981998 shutdown_connections 2024-03-18T16:14:27.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.268+0000 7f9886599740 1 --2- 172.21.15.39:0/1557981998 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x56441ad71f60 0x56441ad743e0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:27.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.269+0000 7f9886599740 1 --2- 172.21.15.39:0/1557981998 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56441ad61590 0x56441ad6d530 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:27.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.269+0000 7f9886599740 1 --2- 172.21.15.39:0/1557981998 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56441ad5c420 0x56441ad664f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:27.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.269+0000 7f9886599740 1 --2- 172.21.15.39:0/1557981998 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56441ad5ba30 0x56441ad65fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:27.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.269+0000 7f9862a00640 0 client.26503 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:14:27.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.269+0000 7f9886599740 1 -- 172.21.15.39:0/1557981998 >> 172.21.15.39:0/1557981998 conn(0x56441ad56fa0 msgr2=0x56441ad561c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:14:27.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.269+0000 7f9886599740 1 -- 172.21.15.39:0/1557981998 shutdown_connections 2024-03-18T16:14:27.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.269+0000 7f9886599740 1 -- 172.21.15.39:0/1557981998 wait complete. 2024-03-18T16:14:27.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.269+0000 7f9886599740 20 client.26503 trim_cache size 0 max 16384 2024-03-18T16:14:27.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.328+0000 7f9886599740 1 Processor -- start 2024-03-18T16:14:27.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.328+0000 7f9886599740 1 -- start start 2024-03-18T16:14:27.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.328+0000 7f9886599740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56441ad5ba30 0x56441ad5d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:27.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.329+0000 7f9886599740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56441ad5c420 0x56441ad5de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:27.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.329+0000 7f9886599740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56441ad61590 0x56441ad6d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:27.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.329+0000 7f9886599740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f985c00fc10 con 0x56441ad61590 2024-03-18T16:14:27.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.329+0000 7f9886599740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f985c0118d0 con 0x56441ad5c420 2024-03-18T16:14:27.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.329+0000 7f9886599740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f985c0102e0 con 0x56441ad5ba30 2024-03-18T16:14:27.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.329+0000 7f987be00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56441ad5c420 0x56441ad5de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:27.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.329+0000 7f9880e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56441ad5ba30 0x56441ad5d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:27.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.329+0000 7f987be00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56441ad5c420 0x56441ad5de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:49204/0 (socket says 172.21.15.39:49204) 2024-03-18T16:14:27.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.329+0000 7f9880e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56441ad5ba30 0x56441ad5d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:53528/0 (socket says 172.21.15.39:53528) 2024-03-18T16:14:27.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.329+0000 7f987be00640 1 -- 172.21.15.39:0/84223622 learned_addr learned my addr 172.21.15.39:0/84223622 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:14:27.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.329+0000 7f987be00640 1 -- 172.21.15.39:0/84223622 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56441ad5ba30 msgr2=0x56441ad5d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:14:27.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.329+0000 7f987be00640 1 --2- 172.21.15.39:0/84223622 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56441ad5ba30 0x56441ad5d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:27.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.330+0000 7f987be00640 1 -- 172.21.15.39:0/84223622 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56441ad61590 msgr2=0x56441ad6d4f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:14:27.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.330+0000 7f987be00640 1 --2- 172.21.15.39:0/84223622 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56441ad61590 0x56441ad6d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:27.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.331+0000 7f987be00640 1 -- 172.21.15.39:0/84223622 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56441ad56dd0 con 0x56441ad5c420 2024-03-18T16:14:27.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.331+0000 7f987be00640 1 --2- 172.21.15.39:0/84223622 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56441ad5c420 0x56441ad5de00 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f986809ab50 tx=0x7f986809a750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ff7dd91b4093ba7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:14:27.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.333+0000 7f986d600640 1 -- 172.21.15.39:0/84223622 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98680b3040 con 0x56441ad5c420 2024-03-18T16:14:27.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.333+0000 7f986d600640 1 -- 172.21.15.39:0/84223622 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f98680b7070 con 0x56441ad5c420 2024-03-18T16:14:27.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.333+0000 7f986d600640 1 -- 172.21.15.39:0/84223622 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f986809a060 con 0x56441ad5c420 2024-03-18T16:14:27.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.334+0000 7f9886599740 1 -- 172.21.15.39:0/84223622 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56441ad5c420 msgr2=0x56441ad5de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:14:27.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.334+0000 7f9886599740 1 --2- 172.21.15.39:0/84223622 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56441ad5c420 0x56441ad5de00 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f986809ab50 tx=0x7f986809a750 comp rx=0 tx=0).stop 2024-03-18T16:14:27.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.334+0000 7f9886599740 1 -- 172.21.15.39:0/84223622 shutdown_connections 2024-03-18T16:14:27.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.334+0000 7f9886599740 1 --2- 172.21.15.39:0/84223622 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56441ad5c420 0x56441ad5de00 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:27.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.334+0000 7f9886599740 1 --2- 172.21.15.39:0/84223622 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56441ad61590 0x56441ad6d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:27.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.335+0000 7f9886599740 1 --2- 172.21.15.39:0/84223622 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56441ad5ba30 0x56441ad5d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:27.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.335+0000 7f9886599740 1 -- 172.21.15.39:0/84223622 >> 172.21.15.39:0/84223622 conn(0x56441ad56fa0 msgr2=0x56441ad65fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:14:27.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.335+0000 7f9886599740 1 -- 172.21.15.39:0/84223622 shutdown_connections 2024-03-18T16:14:27.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.336+0000 7f9886599740 1 -- 172.21.15.39:0/84223622 wait complete. 2024-03-18T16:14:27.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.337+0000 7f9886599740 1 -- start start 2024-03-18T16:14:27.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.337+0000 7f9886599740 1 Processor -- start 2024-03-18T16:14:27.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.340+0000 7f9886599740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56441ad5ba30 0x56441ad6aa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:27.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.340+0000 7f9886599740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56441ad5c420 0x56441ad6af80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:27.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.340+0000 7f9886599740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56441ad61590 0x56441ad6bf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:27.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.341+0000 7f9886599740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f985c011140 con 0x56441ad61590 2024-03-18T16:14:27.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.341+0000 7f9886599740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f985c0109b0 con 0x56441ad5ba30 2024-03-18T16:14:27.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.341+0000 7f9886599740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f985c00dbf0 con 0x56441ad5c420 2024-03-18T16:14:27.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.341+0000 7f9886599740 10 client.0 _ll_register_callbacks cb 0x7f987400ec00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:14:27.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.341+0000 7f9880e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56441ad5ba30 0x56441ad6aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:27.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.341+0000 7f987be00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56441ad5c420 0x56441ad6af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:27.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.341+0000 7f9880e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56441ad5ba30 0x56441ad6aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:49212/0 (socket says 172.21.15.39:49212) 2024-03-18T16:14:27.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.341+0000 7f9880e00640 1 -- 172.21.15.39:0/2753369030 learned_addr learned my addr 172.21.15.39:0/2753369030 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:14:27.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.343+0000 7f9880e00640 1 -- 172.21.15.39:0/2753369030 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56441ad5c420 msgr2=0x56441ad6af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:14:27.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.343+0000 7f9880e00640 1 --2- 172.21.15.39:0/2753369030 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56441ad5c420 0x56441ad6af80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:27.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.343+0000 7f9880e00640 1 -- 172.21.15.39:0/2753369030 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56441ad61590 msgr2=0x56441ad6bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:14:27.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.343+0000 7f9880e00640 1 --2- 172.21.15.39:0/2753369030 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56441ad61590 0x56441ad6bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:27.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.343+0000 7f9880e00640 1 -- 172.21.15.39:0/2753369030 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56441ad57dd0 con 0x56441ad5ba30 2024-03-18T16:14:27.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.345+0000 7f9880e00640 1 --2- 172.21.15.39:0/2753369030 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56441ad5ba30 0x56441ad6aa40 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f987002c800 tx=0x7f98700092d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=489082576c79c4d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:14:27.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.345+0000 7f9862a00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:14:27.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.347+0000 7f9862a00640 1 -- 172.21.15.39:0/2753369030 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f987004c070 con 0x56441ad5ba30 2024-03-18T16:14:27.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.347+0000 7f9886599740 1 -- 172.21.15.39:0/2753369030 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f98680b7070 con 0x56441ad5ba30 2024-03-18T16:14:27.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.349+0000 7f9862a00640 1 -- 172.21.15.39:0/2753369030 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f9870048d10 con 0x56441ad5ba30 2024-03-18T16:14:27.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.349+0000 7f9862a00640 1 -- 172.21.15.39:0/2753369030 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) ==== 24+0+0 (secure 0 0 0) 0x7f9870058050 con 0x56441ad5ba30 2024-03-18T16:14:27.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.349+0000 7f9886599740 10 client.26537 fetch_fsmap learned FSMap version 36 2024-03-18T16:14:27.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.349+0000 7f9886599740 1 -- 172.21.15.39:0/2753369030 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) -- 0x56441ad6d730 con 0x56441ad5ba30 2024-03-18T16:14:27.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.350+0000 7f9862a00640 1 -- 172.21.15.39:0/2753369030 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f98680b7070 con 0x56441ad5ba30 2024-03-18T16:14:27.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.350+0000 7f9862a00640 1 -- 172.21.15.39:0/2753369030 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 36) ==== 8406+0+0 (secure 0 0 0) 0x7f98700046c0 con 0x56441ad5ba30 2024-03-18T16:14:27.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.350+0000 7f9886599740 10 client.26537 fetch_fsmap finished waiting for FSMap version 36 2024-03-18T16:14:27.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.350+0000 7f9886599740 10 client.26537 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:14:27.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.350+0000 7f9886599740 20 client.26537 populate_metadata read hostname 'smithi039' 2024-03-18T16:14:27.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.350+0000 7f9886599740 1 --2- 172.21.15.39:0/2753369030 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x56441add8aa0 0x56441addaec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:14:27.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.350+0000 7f9886599740 4 client.26537 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T16:14:27.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.350+0000 7f9886599740 1 -- 172.21.15.39:0/2753369030 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56441accd730 con 0x56441add8aa0 2024-03-18T16:14:27.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.351+0000 7f987be00640 1 --2- 172.21.15.39:0/2753369030 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x56441add8aa0 0x56441addaec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:14:27.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.352+0000 7f987be00640 1 --2- 172.21.15.39:0/2753369030 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x56441add8aa0 0x56441addaec0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6515b177521a36d server_cookie=e289ba75f63e029e in_seq=0 out_seq=0 2024-03-18T16:14:27.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.355+0000 7f9862a00640 10 client.26537 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:14:27.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.355+0000 7f9862a00640 1 -- 172.21.15.39:0/2753369030 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+266 (crc 0 0 0) 0x56441accd730 con 0x56441add8aa0 2024-03-18T16:14:27.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.355+0000 7f9862a00640 10 client.26537 handle_command_reply: tid=0 2024-03-18T16:14:27.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.355+0000 7f9886599740 1 client.26537 shutdown 2024-03-18T16:14:27.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.355+0000 7f9886599740 1 -- 172.21.15.39:0/2753369030 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56441ad5ba30 msgr2=0x56441ad6aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:14:27.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.355+0000 7f9886599740 1 --2- 172.21.15.39:0/2753369030 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56441ad5ba30 0x56441ad6aa40 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f987002c800 tx=0x7f98700092d0 comp rx=0 tx=0).stop 2024-03-18T16:14:27.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.356+0000 7f9886599740 1 -- 172.21.15.39:0/2753369030 shutdown_connections 2024-03-18T16:14:27.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.356+0000 7f9886599740 1 --2- 172.21.15.39:0/2753369030 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x56441add8aa0 0x56441addaec0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:27.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.356+0000 7f9886599740 1 --2- 172.21.15.39:0/2753369030 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56441ad61590 0x56441ad6bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:27.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.356+0000 7f9886599740 1 --2- 172.21.15.39:0/2753369030 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56441ad5c420 0x56441ad6af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:27.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.356+0000 7f9886599740 1 --2- 172.21.15.39:0/2753369030 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56441ad5ba30 0x56441ad6aa40 secure :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f987002c800 tx=0x7f98700092d0 comp rx=0 tx=0).stop 2024-03-18T16:14:27.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.356+0000 7f9862a00640 0 client.26537 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:14:27.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.356+0000 7f9886599740 1 -- 172.21.15.39:0/2753369030 >> 172.21.15.39:0/2753369030 conn(0x56441ad56fa0 msgr2=0x56441ad65f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:14:27.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.357+0000 7f9886599740 1 -- 172.21.15.39:0/2753369030 shutdown_connections 2024-03-18T16:14:27.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.357+0000 7f9886599740 1 -- 172.21.15.39:0/2753369030 wait complete. 2024-03-18T16:14:27.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.358+0000 7f9886599740 20 client.26537 trim_cache size 0 max 16384 2024-03-18T16:14:27.361 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-03-18T16:14:27.362 INFO:teuthology.orchestra.run.smithi039.stdout: "status": "scrub active (20 inodes in the stack)", 2024-03-18T16:14:27.362 INFO:teuthology.orchestra.run.smithi039.stdout: "scrubs": { 2024-03-18T16:14:27.362 INFO:teuthology.orchestra.run.smithi039.stdout: "9e29cb97-942d-44c9-ba66-4e00ebf99f56": { 2024-03-18T16:14:27.362 INFO:teuthology.orchestra.run.smithi039.stdout: "path": "/", 2024-03-18T16:14:27.362 INFO:teuthology.orchestra.run.smithi039.stdout: "tag": "9e29cb97-942d-44c9-ba66-4e00ebf99f56", 2024-03-18T16:14:27.362 INFO:teuthology.orchestra.run.smithi039.stdout: "options": "recursive,force" 2024-03-18T16:14:27.362 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:14:27.362 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:14:27.362 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-03-18T16:14:27.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.360+0000 7f9882200640 1 -- 172.21.15.39:0/4215373539 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f98640a2a00 msgr2=0x7f98640a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:14:27.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.360+0000 7f9882200640 1 --2- 172.21.15.39:0/4215373539 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f98640a2a00 0x7f98640a4ec0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f9874007fd0 tx=0x7f9874004af0 comp rx=0 tx=0).stop 2024-03-18T16:14:27.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.360+0000 7f9882200640 1 -- 172.21.15.39:0/4215373539 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f987c1049b0 msgr2=0x7f987c196970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:14:27.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.360+0000 7f9882200640 1 --2- 172.21.15.39:0/4215373539 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f987c1049b0 0x7f987c196970 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f986800b470 tx=0x7f986800b930 comp rx=0 tx=0).stop 2024-03-18T16:14:27.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.360+0000 7f9882200640 1 -- 172.21.15.39:0/4215373539 shutdown_connections 2024-03-18T16:14:27.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.360+0000 7f9882200640 1 --2- 172.21.15.39:0/4215373539 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f987c1049b0 0x7f987c196970 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:27.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.360+0000 7f9882200640 1 --2- 172.21.15.39:0/4215373539 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f98640a2a00 0x7f98640a4ec0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:27.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.360+0000 7f9882200640 1 --2- 172.21.15.39:0/4215373539 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f987c1072e0 0x7f987c192fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:27.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.360+0000 7f9882200640 1 --2- 172.21.15.39:0/4215373539 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f987c102080 0x7f987c196430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:14:27.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.360+0000 7f9882200640 1 -- 172.21.15.39:0/4215373539 >> 172.21.15.39:0/4215373539 conn(0x7f987c0fbab0 msgr2=0x7f987c101710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:14:27.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.361+0000 7f9882200640 1 -- 172.21.15.39:0/4215373539 shutdown_connections 2024-03-18T16:14:27.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:14:27.361+0000 7f9882200640 1 -- 172.21.15.39:0/4215373539 wait complete. 2024-03-18T16:14:27.394 INFO:tasks.cephfs.filesystem:scrub status for tag:9e29cb97-942d-44c9-ba66-4e00ebf99f56 - {'path': '/', 'tag': '9e29cb97-942d-44c9-ba66-4e00ebf99f56', 'options': 'recursive,force'} 2024-03-18T16:14:27.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:14:27 smithi196 ceph-mon[36307]: pgmap v553: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 37 MiB/s wr, 6.09k op/s 2024-03-18T16:14:27.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:14:27 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:14:27.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:14:27 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:14:27.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:14:27 smithi194 ceph-mon[36200]: pgmap v553: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 37 MiB/s wr, 6.09k op/s 2024-03-18T16:14:27.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:14:27 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:14:27.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:14:27 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:14:27.839 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:14:27 smithi039 ceph-mon[31721]: pgmap v553: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 37 MiB/s wr, 6.09k op/s 2024-03-18T16:14:27.839 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:14:27 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:14:27.839 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:14:27 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:14:28.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:14:28 smithi196 ceph-mon[36307]: scrub queued for path: / 2024-03-18T16:14:28.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:14:28 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:14:28.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:14:28 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:14:28.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:14:28 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:14:28.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:14:28 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:14:28.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:14:28 smithi194 ceph-mon[36200]: scrub queued for path: / 2024-03-18T16:14:28.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:14:28 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:14:28.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:14:28 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:14:28.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:14:28 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:14:28.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:14:28 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:14:28.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:14:28 smithi039 ceph-mon[31721]: scrub queued for path: / 2024-03-18T16:14:28.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:14:28 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:14:28.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:14:28 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:14:28.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:14:28 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:14:28.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:14:28 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:14:29.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:14:29 smithi194 ceph-mon[36200]: pgmap v554: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 35 MiB/s wr, 5.81k op/s 2024-03-18T16:14:29.838 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:14:29 smithi039 ceph-mon[31721]: pgmap v554: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 35 MiB/s wr, 5.81k op/s 2024-03-18T16:14:29.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:14:29 smithi196 ceph-mon[36307]: pgmap v554: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 35 MiB/s wr, 5.81k op/s 2024-03-18T16:14:30.838 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:14:30 smithi039 ceph-mon[31721]: pgmap v555: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 34 MiB/s wr, 5.74k op/s 2024-03-18T16:14:30.838 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:14:30 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:14:30.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:14:30 smithi196 ceph-mon[36307]: pgmap v555: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 34 MiB/s wr, 5.74k op/s 2024-03-18T16:14:30.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:14:30 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:14:30.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:14:30 smithi194 ceph-mon[36200]: pgmap v555: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 34 MiB/s wr, 5.74k op/s 2024-03-18T16:14:30.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:14:30 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:14:31.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:14:31 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:14:31.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:14:31 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:14:31.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:14:31 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:14:31.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:14:31 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:14:31.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:14:31 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:14:31.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:14:31 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:14:31.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:14:31 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:14:31.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:14:31 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:14:31.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:14:31 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:14:32.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:14:32 smithi039 ceph-mon[31721]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T16:14:32.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:14:32 smithi039 ceph-mon[31721]: pgmap v556: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 34 MiB/s wr, 5.78k op/s 2024-03-18T16:14:32.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:14:32 smithi196 ceph-mon[36307]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T16:14:32.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:14:32 smithi196 ceph-mon[36307]: pgmap v556: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 34 MiB/s wr, 5.78k op/s 2024-03-18T16:14:32.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:14:32 smithi194 ceph-mon[36200]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T16:14:32.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:14:32 smithi194 ceph-mon[36200]: pgmap v556: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 34 MiB/s wr, 5.78k op/s 2024-03-18T16:14:35.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:14:34 smithi039 ceph-mon[31721]: pgmap v557: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 37 MiB/s wr, 6.32k op/s 2024-03-18T16:14:35.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:14:34 smithi196 ceph-mon[36307]: pgmap v557: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 37 MiB/s wr, 6.32k op/s 2024-03-18T16:14:35.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:14:34 smithi194 ceph-mon[36200]: pgmap v557: 129 pgs: 129 active+clean; 13 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 37 MiB/s wr, 6.32k op/s 2024-03-18T16:14:36.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:14:35 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:14:36.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:14:35 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:14:36.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:14:35 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:14:37.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:14:36 smithi039 ceph-mon[31721]: pgmap v558: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 38 MiB/s wr, 6.50k op/s 2024-03-18T16:14:37.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:14:36 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:14:37.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:14:36 smithi196 ceph-mon[36307]: pgmap v558: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 38 MiB/s wr, 6.50k op/s 2024-03-18T16:14:37.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:14:36 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:14:37.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:14:36 smithi194 ceph-mon[36200]: pgmap v558: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 38 MiB/s wr, 6.50k op/s 2024-03-18T16:14:37.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:14:36 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:14:38.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:14:37 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:14:38.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:14:37 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:14:38.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:14:37 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:14:38.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:14:37 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:14:38.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:14:37 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:14:38.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:14:37 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:14:38.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:14:37 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:14:38.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:14:37 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:14:38.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:14:37 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:14:39.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:14:38 smithi196 ceph-mon[36307]: pgmap v559: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 34 MiB/s wr, 5.92k op/s 2024-03-18T16:14:39.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:14:38 smithi194 ceph-mon[36200]: pgmap v559: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 34 MiB/s wr, 5.92k op/s 2024-03-18T16:14:39.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:14:38 smithi039 ceph-mon[31721]: pgmap v559: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 34 MiB/s wr, 5.92k op/s 2024-03-18T16:14:41.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:14:40 smithi196 ceph-mon[36307]: pgmap v560: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 35 MiB/s wr, 6.23k op/s 2024-03-18T16:14:41.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:14:40 smithi194 ceph-mon[36200]: pgmap v560: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 35 MiB/s wr, 6.23k op/s 2024-03-18T16:14:41.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:14:40 smithi039 ceph-mon[31721]: pgmap v560: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 35 MiB/s wr, 6.23k op/s 2024-03-18T16:14:43.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:14:42 smithi196 ceph-mon[36307]: pgmap v561: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 34 MiB/s wr, 6.08k op/s 2024-03-18T16:14:43.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:14:42 smithi194 ceph-mon[36200]: pgmap v561: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 34 MiB/s wr, 6.08k op/s 2024-03-18T16:14:43.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:14:42 smithi039 ceph-mon[31721]: pgmap v561: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 34 MiB/s wr, 6.08k op/s 2024-03-18T16:14:45.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:14:44 smithi196 ceph-mon[36307]: pgmap v562: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 29 MiB/s wr, 5.30k op/s 2024-03-18T16:14:45.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:14:44 smithi194 ceph-mon[36200]: pgmap v562: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 29 MiB/s wr, 5.30k op/s 2024-03-18T16:14:45.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:14:44 smithi039 ceph-mon[31721]: pgmap v562: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 29 MiB/s wr, 5.30k op/s 2024-03-18T16:14:47.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:14:46 smithi196 ceph-mon[36307]: pgmap v563: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 34 MiB/s wr, 6.23k op/s 2024-03-18T16:14:47.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:14:46 smithi194 ceph-mon[36200]: pgmap v563: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 34 MiB/s wr, 6.23k op/s 2024-03-18T16:14:47.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:14:46 smithi039 ceph-mon[31721]: pgmap v563: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 34 MiB/s wr, 6.23k op/s 2024-03-18T16:14:48.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:14:47 smithi196 ceph-mon[36307]: scrub summary: idle 2024-03-18T16:14:48.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:14:47 smithi196 ceph-mon[36307]: scrub summary: idle 2024-03-18T16:14:48.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:14:47 smithi196 ceph-mon[36307]: scrub summary: idle 2024-03-18T16:14:48.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:14:47 smithi194 ceph-mon[36200]: scrub summary: idle 2024-03-18T16:14:48.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:14:47 smithi194 ceph-mon[36200]: scrub summary: idle 2024-03-18T16:14:48.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:14:47 smithi194 ceph-mon[36200]: scrub summary: idle 2024-03-18T16:14:48.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:14:47 smithi039 ceph-mon[31721]: scrub summary: idle 2024-03-18T16:14:48.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:14:47 smithi039 ceph-mon[31721]: scrub summary: idle 2024-03-18T16:14:48.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:14:47 smithi039 ceph-mon[31721]: scrub summary: idle 2024-03-18T16:14:49.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:14:48 smithi196 ceph-mon[36307]: scrub summary: idle 2024-03-18T16:14:49.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:14:48 smithi196 ceph-mon[36307]: scrub summary: idle 2024-03-18T16:14:49.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:14:48 smithi196 ceph-mon[36307]: pgmap v564: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 29 MiB/s wr, 5.29k op/s 2024-03-18T16:14:49.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:14:48 smithi196 ceph-mon[36307]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T16:14:49.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:14:48 smithi194 ceph-mon[36200]: scrub summary: idle 2024-03-18T16:14:49.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:14:48 smithi194 ceph-mon[36200]: scrub summary: idle 2024-03-18T16:14:49.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:14:48 smithi194 ceph-mon[36200]: pgmap v564: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 29 MiB/s wr, 5.29k op/s 2024-03-18T16:14:49.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:14:48 smithi194 ceph-mon[36200]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T16:14:49.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:14:48 smithi039 ceph-mon[31721]: scrub summary: idle 2024-03-18T16:14:49.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:14:48 smithi039 ceph-mon[31721]: scrub summary: idle 2024-03-18T16:14:49.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:14:48 smithi039 ceph-mon[31721]: pgmap v564: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 29 MiB/s wr, 5.29k op/s 2024-03-18T16:14:49.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:14:48 smithi039 ceph-mon[31721]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T16:14:51.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:14:50 smithi196 ceph-mon[36307]: pgmap v565: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 34 MiB/s wr, 6.37k op/s 2024-03-18T16:14:51.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:14:50 smithi194 ceph-mon[36200]: pgmap v565: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 34 MiB/s wr, 6.37k op/s 2024-03-18T16:14:51.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:14:50 smithi039 ceph-mon[31721]: pgmap v565: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 34 MiB/s wr, 6.37k op/s 2024-03-18T16:14:53.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:14:52 smithi196 ceph-mon[36307]: pgmap v566: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 34 MiB/s wr, 6.33k op/s 2024-03-18T16:14:53.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:14:52 smithi196 ceph-mon[36307]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T16:14:53.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:14:52 smithi194 ceph-mon[36200]: pgmap v566: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 34 MiB/s wr, 6.33k op/s 2024-03-18T16:14:53.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:14:52 smithi194 ceph-mon[36200]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T16:14:53.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:14:52 smithi039 ceph-mon[31721]: pgmap v566: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 34 MiB/s wr, 6.33k op/s 2024-03-18T16:14:53.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:14:52 smithi039 ceph-mon[31721]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T16:14:55.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:14:54 smithi196 ceph-mon[36307]: pgmap v567: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 30 MiB/s wr, 5.72k op/s 2024-03-18T16:14:55.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:14:54 smithi194 ceph-mon[36200]: pgmap v567: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 30 MiB/s wr, 5.72k op/s 2024-03-18T16:14:55.339 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:14:54 smithi039 ceph-mon[31721]: pgmap v567: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 30 MiB/s wr, 5.72k op/s 2024-03-18T16:14:57.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:14:56 smithi196 ceph-mon[36307]: pgmap v568: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 36 MiB/s wr, 6.99k op/s 2024-03-18T16:14:57.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:14:56 smithi194 ceph-mon[36200]: pgmap v568: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 36 MiB/s wr, 6.99k op/s 2024-03-18T16:14:57.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:14:56 smithi039 ceph-mon[31721]: pgmap v568: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 36 MiB/s wr, 6.99k op/s 2024-03-18T16:14:57.396 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T16:14:57.396 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T16:14:58.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:14:57 smithi194 ceph-mon[36200]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T16:14:58.340 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:14:57 smithi039 ceph-mon[31721]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T16:14:58.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:14:57 smithi196 ceph-mon[36307]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T16:14:59.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:14:58 smithi194 ceph-mon[36200]: pgmap v569: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 29 MiB/s wr, 5.74k op/s 2024-03-18T16:14:59.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:14:58 smithi039 ceph-mon[31721]: pgmap v569: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 29 MiB/s wr, 5.74k op/s 2024-03-18T16:14:59.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:14:58 smithi196 ceph-mon[36307]: pgmap v569: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 29 MiB/s wr, 5.74k op/s 2024-03-18T16:15:00.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.096+0000 7f97cdc00640 1 -- 172.21.15.39:0/1633823835 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f97c0009420 msgr2=0x7f97c0007870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:00.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.096+0000 7f97cdc00640 1 --2- 172.21.15.39:0/1633823835 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f97c0009420 0x7f97c0007870 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f97b4009900 tx=0x7f97b402bce0 comp rx=0 tx=0).stop 2024-03-18T16:15:00.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.096+0000 7f97cdc00640 1 -- 172.21.15.39:0/1633823835 shutdown_connections 2024-03-18T16:15:00.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.096+0000 7f97cdc00640 1 --2- 172.21.15.39:0/1633823835 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f97c009dc00 0x7f97c00a00a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:00.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.096+0000 7f97cdc00640 1 --2- 172.21.15.39:0/1633823835 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f97c0007e40 0x7f97c00082a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:00.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.096+0000 7f97cdc00640 1 --2- 172.21.15.39:0/1633823835 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f97c0009420 0x7f97c0007870 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:00.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.096+0000 7f97cdc00640 1 -- 172.21.15.39:0/1633823835 >> 172.21.15.39:0/1633823835 conn(0x7f97c0091300 msgr2=0x7f97c0093720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:00.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.096+0000 7f97cdc00640 1 -- 172.21.15.39:0/1633823835 shutdown_connections 2024-03-18T16:15:00.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.096+0000 7f97cdc00640 1 -- 172.21.15.39:0/1633823835 wait complete. 2024-03-18T16:15:00.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.098+0000 7f97cdc00640 1 Processor -- start 2024-03-18T16:15:00.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.099+0000 7f97cdc00640 1 -- start start 2024-03-18T16:15:00.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.099+0000 7f97cdc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f97c0007e40 0x7f97c012e310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:00.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.099+0000 7f97cdc00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f97c0009420 0x7f97c012e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:00.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.099+0000 7f97cdc00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f97c009dc00 0x7f97c01358c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:00.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.099+0000 7f97cdc00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f97c00a3ce0 con 0x7f97c0007e40 2024-03-18T16:15:00.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.099+0000 7f97cdc00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f97c00a3b60 con 0x7f97c0009420 2024-03-18T16:15:00.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.099+0000 7f97cdc00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f97c00a3e60 con 0x7f97c009dc00 2024-03-18T16:15:00.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.100+0000 7f97cd200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f97c009dc00 0x7f97c01358c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:00.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.100+0000 7f97c7e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f97c0007e40 0x7f97c012e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:00.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.100+0000 7f97c7e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f97c0007e40 0x7f97c012e310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:36278/0 (socket says 172.21.15.39:36278) 2024-03-18T16:15:00.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.100+0000 7f97c7e00640 1 -- 172.21.15.39:0/1501191113 learned_addr learned my addr 172.21.15.39:0/1501191113 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:15:00.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.101+0000 7f97c7400640 1 --2- 172.21.15.39:0/1501191113 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f97c0009420 0x7f97c012e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:00.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.102+0000 7f97c7400640 1 -- 172.21.15.39:0/1501191113 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f97c009dc00 msgr2=0x7f97c01358c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:00.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.102+0000 7f97c7400640 1 --2- 172.21.15.39:0/1501191113 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f97c009dc00 0x7f97c01358c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:00.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.102+0000 7f97c7400640 1 -- 172.21.15.39:0/1501191113 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f97c0007e40 msgr2=0x7f97c012e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:00.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.102+0000 7f97c7400640 1 --2- 172.21.15.39:0/1501191113 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f97c0007e40 0x7f97c012e310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:00.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.102+0000 7f97c7400640 1 -- 172.21.15.39:0/1501191113 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97c0135f30 con 0x7f97c0009420 2024-03-18T16:15:00.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.102+0000 7f97c7400640 1 --2- 172.21.15.39:0/1501191113 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f97c0009420 0x7f97c012e850 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f97bc00e9f0 tx=0x7f97bc00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:00.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.102+0000 7f97cd200640 1 --2- 172.21.15.39:0/1501191113 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f97c009dc00 0x7f97c01358c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:15:00.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.102+0000 7f97c4c00640 1 -- 172.21.15.39:0/1501191113 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97bc00ccc0 con 0x7f97c0009420 2024-03-18T16:15:00.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.103+0000 7f97c4c00640 1 -- 172.21.15.39:0/1501191113 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97bc005020 con 0x7f97c0009420 2024-03-18T16:15:00.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.103+0000 7f97cdc00640 1 -- 172.21.15.39:0/1501191113 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97c0136220 con 0x7f97c0009420 2024-03-18T16:15:00.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.103+0000 7f97c4c00640 1 -- 172.21.15.39:0/1501191113 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97bc004b20 con 0x7f97c0009420 2024-03-18T16:15:00.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.103+0000 7f97cdc00640 1 -- 172.21.15.39:0/1501191113 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97c0136770 con 0x7f97c0009420 2024-03-18T16:15:00.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.109+0000 7f97d1f6a740 1 Processor -- start 2024-03-18T16:15:00.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.109+0000 7f97d1f6a740 1 -- start start 2024-03-18T16:15:00.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.110+0000 7f97d1f6a740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b0c57fc220 0x55b0c57fc640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:00.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.110+0000 7f97d1f6a740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b0c57fcc10 0x55b0c5801710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:00.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.110+0000 7f97d1f6a740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b0c5801d80 0x55b0c5804210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:00.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.110+0000 7f97d1f6a740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55b0c57581f0 con 0x55b0c57fc220 2024-03-18T16:15:00.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.110+0000 7f97d1f6a740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55b0c5700a80 con 0x55b0c5801d80 2024-03-18T16:15:00.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.110+0000 7f97d1f6a740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55b0c5754290 con 0x55b0c57fcc10 2024-03-18T16:15:00.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.111+0000 7f97c7e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b0c57fcc10 0x55b0c5801710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:00.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.111+0000 7f97c7e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b0c57fcc10 0x55b0c5801710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:57498/0 (socket says 172.21.15.39:57498) 2024-03-18T16:15:00.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.111+0000 7f97c7e00640 1 -- 172.21.15.39:0/3235338562 learned_addr learned my addr 172.21.15.39:0/3235338562 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:15:00.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.111+0000 7f97c7e00640 1 -- 172.21.15.39:0/3235338562 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b0c5801d80 msgr2=0x55b0c5804210 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:15:00.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.111+0000 7f97c7e00640 1 --2- 172.21.15.39:0/3235338562 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b0c5801d80 0x55b0c5804210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:00.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.111+0000 7f97c7e00640 1 -- 172.21.15.39:0/3235338562 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b0c57fc220 msgr2=0x55b0c57fc640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:00.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.111+0000 7f97c7e00640 1 --2- 172.21.15.39:0/3235338562 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b0c57fc220 0x55b0c57fc640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:00.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.112+0000 7f97c7e00640 1 -- 172.21.15.39:0/3235338562 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b0c55367d0 con 0x55b0c57fcc10 2024-03-18T16:15:00.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.113+0000 7f97c7e00640 1 --2- 172.21.15.39:0/3235338562 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b0c57fcc10 0x55b0c5801710 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f97b4005c40 tx=0x7f97b4005d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8091b386b8fe07cd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:00.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.115+0000 7f97c4c00640 1 -- 172.21.15.39:0/1501191113 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f97bc004cc0 con 0x7f97c0009420 2024-03-18T16:15:00.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.115+0000 7f97b9600640 1 -- 172.21.15.39:0/3235338562 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97b4043070 con 0x55b0c57fcc10 2024-03-18T16:15:00.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.115+0000 7f97b9600640 1 -- 172.21.15.39:0/3235338562 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97b402cb70 con 0x55b0c57fcc10 2024-03-18T16:15:00.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.115+0000 7f97b9600640 1 -- 172.21.15.39:0/3235338562 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97b4005260 con 0x55b0c57fcc10 2024-03-18T16:15:00.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.116+0000 7f97d1f6a740 1 -- 172.21.15.39:0/3235338562 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b0c57fcc10 msgr2=0x55b0c5801710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:00.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.116+0000 7f97d1f6a740 1 --2- 172.21.15.39:0/3235338562 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b0c57fcc10 0x55b0c5801710 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f97b4005c40 tx=0x7f97b4005d10 comp rx=0 tx=0).stop 2024-03-18T16:15:00.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.117+0000 7f97c4c00640 1 --2- 172.21.15.39:0/1501191113 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f97b00a2880 0x7f97b00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:00.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.117+0000 7f97c4c00640 1 -- 172.21.15.39:0/1501191113 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f97bc0ced60 con 0x7f97c0009420 2024-03-18T16:15:00.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.117+0000 7f97cd200640 1 --2- 172.21.15.39:0/1501191113 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f97b00a2880 0x7f97b00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:00.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.120+0000 7f97cd200640 1 --2- 172.21.15.39:0/1501191113 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f97b00a2880 0x7f97b00a4d40 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f97c8064cd0 tx=0x7f97c8075040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:00.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.120+0000 7f97d1f6a740 1 -- 172.21.15.39:0/3235338562 shutdown_connections 2024-03-18T16:15:00.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.120+0000 7f97d1f6a740 1 --2- 172.21.15.39:0/3235338562 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b0c5801d80 0x55b0c5804210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:00.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.120+0000 7f97d1f6a740 1 --2- 172.21.15.39:0/3235338562 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b0c57fcc10 0x55b0c5801710 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:00.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.120+0000 7f97d1f6a740 1 --2- 172.21.15.39:0/3235338562 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b0c57fc220 0x55b0c57fc640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:00.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.120+0000 7f97d1f6a740 1 -- 172.21.15.39:0/3235338562 >> 172.21.15.39:0/3235338562 conn(0x55b0c57f7790 msgr2=0x55b0c57f7b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:00.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.120+0000 7f97d1f6a740 1 -- 172.21.15.39:0/3235338562 shutdown_connections 2024-03-18T16:15:00.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.120+0000 7f97d1f6a740 1 -- 172.21.15.39:0/3235338562 wait complete. 2024-03-18T16:15:00.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.121+0000 7f97d1f6a740 1 -- start start 2024-03-18T16:15:00.124 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.121+0000 7f97d1f6a740 1 Processor -- start 2024-03-18T16:15:00.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.124+0000 7f97d1f6a740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b0c57fc220 0x55b0c5806590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:00.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.124+0000 7f97c7e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b0c57fc220 0x55b0c5806590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:00.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.124+0000 7f97d1f6a740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b0c57fcc10 0x55b0c5806ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:00.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.124+0000 7f97d1f6a740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b0c5801d80 0x55b0c580daa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:00.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.124+0000 7f97d1f6a740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55b0c5804cd0 con 0x55b0c57fcc10 2024-03-18T16:15:00.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.125+0000 7f97cd200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b0c5801d80 0x55b0c580daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:00.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.125+0000 7f97d1f6a740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55b0c5713ed0 con 0x55b0c57fc220 2024-03-18T16:15:00.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.125+0000 7f97cd200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b0c5801d80 0x55b0c580daa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:57500/0 (socket says 172.21.15.39:57500) 2024-03-18T16:15:00.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.125+0000 7f97cd200640 1 -- 172.21.15.39:0/2230837894 learned_addr learned my addr 172.21.15.39:0/2230837894 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:15:00.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.126+0000 7f97d1f6a740 1 -- 172.21.15.39:0/2230837894 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55b0c5804e50 con 0x55b0c5801d80 2024-03-18T16:15:00.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.126+0000 7f97d1f6a740 10 client.0 _ll_register_callbacks cb 0x55b0c574f220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:15:00.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.126+0000 7f97c7400640 1 --2- 172.21.15.39:0/2230837894 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b0c57fcc10 0x55b0c5806ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:00.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.126+0000 7f97cd200640 1 -- 172.21.15.39:0/2230837894 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b0c57fc220 msgr2=0x55b0c5806590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:00.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.126+0000 7f97cd200640 1 --2- 172.21.15.39:0/2230837894 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b0c57fc220 0x55b0c5806590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:00.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.127+0000 7f97cd200640 1 -- 172.21.15.39:0/2230837894 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b0c57fcc10 msgr2=0x55b0c5806ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:00.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.127+0000 7f97cd200640 1 --2- 172.21.15.39:0/2230837894 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b0c57fcc10 0x55b0c5806ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:00.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.127+0000 7f97cd200640 1 -- 172.21.15.39:0/2230837894 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b0c546c3f0 con 0x55b0c5801d80 2024-03-18T16:15:00.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.127+0000 7f97c7400640 1 --2- 172.21.15.39:0/2230837894 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b0c57fcc10 0x55b0c5806ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:15:00.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.127+0000 7f97ae000640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:15:00.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.127+0000 7f97cd200640 1 --2- 172.21.15.39:0/2230837894 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b0c5801d80 0x55b0c580daa0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f97c8000b60 tx=0x7f97c8067680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa1006dddc2851ce server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:00.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.128+0000 7f97ae000640 1 -- 172.21.15.39:0/2230837894 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97c8090060 con 0x55b0c5801d80 2024-03-18T16:15:00.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.128+0000 7f97ae000640 1 -- 172.21.15.39:0/2230837894 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f97c8080db0 con 0x55b0c5801d80 2024-03-18T16:15:00.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.129+0000 7f97d1f6a740 1 -- 172.21.15.39:0/2230837894 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f97b402cb70 con 0x55b0c5801d80 2024-03-18T16:15:00.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.129+0000 7f97ae000640 1 -- 172.21.15.39:0/2230837894 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f97c808b400 con 0x55b0c5801d80 2024-03-18T16:15:00.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.129+0000 7f97d1f6a740 10 client.26518 fetch_fsmap learned FSMap version 40 2024-03-18T16:15:00.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.129+0000 7f97d1f6a740 1 -- 172.21.15.39:0/2230837894 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55b0c542c0f0 con 0x55b0c5801d80 2024-03-18T16:15:00.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.129+0000 7f97ae000640 1 -- 172.21.15.39:0/2230837894 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97b402cb70 con 0x55b0c5801d80 2024-03-18T16:15:00.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.130+0000 7f97ae000640 1 -- 172.21.15.39:0/2230837894 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 40) ==== 8378+0+0 (secure 0 0 0) 0x7f97c8095720 con 0x55b0c5801d80 2024-03-18T16:15:00.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.131+0000 7f97d1f6a740 10 client.26518 fetch_fsmap finished waiting for FSMap version 40 2024-03-18T16:15:00.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.131+0000 7f97d1f6a740 10 client.26518 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:15:00.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.131+0000 7f97d1f6a740 20 client.26518 populate_metadata read hostname 'smithi039' 2024-03-18T16:15:00.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.131+0000 7f97d1f6a740 1 --2- 172.21.15.39:0/2230837894 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55b0c5812400 0x55b0c5814880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:00.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.131+0000 7f97d1f6a740 4 client.26518 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:15:00.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.131+0000 7f97d1f6a740 1 -- 172.21.15.39:0/2230837894 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b0c576bbe0 con 0x55b0c5812400 2024-03-18T16:15:00.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.131+0000 7f97c7e00640 1 --2- 172.21.15.39:0/2230837894 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55b0c5812400 0x55b0c5814880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:00.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.134+0000 7f97ae000640 10 client.26518 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:15:00.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.134+0000 7f97c7e00640 1 --2- 172.21.15.39:0/2230837894 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55b0c5812400 0x55b0c5814880 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7d99f03e147136a6 server_cookie=d97eeb585121c0ab in_seq=0 out_seq=0 2024-03-18T16:15:00.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.135+0000 7f97ae000640 1 -- 172.21.15.39:0/2230837894 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55b0c576bbe0 con 0x55b0c5812400 2024-03-18T16:15:00.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.135+0000 7f97ae000640 10 client.26518 handle_command_reply: tid=0 2024-03-18T16:15:00.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.135+0000 7f97d1f6a740 1 client.26518 shutdown 2024-03-18T16:15:00.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.136+0000 7f97d1f6a740 1 -- 172.21.15.39:0/2230837894 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b0c5801d80 msgr2=0x55b0c580daa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:00.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.136+0000 7f97d1f6a740 1 --2- 172.21.15.39:0/2230837894 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b0c5801d80 0x55b0c580daa0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f97c8000b60 tx=0x7f97c8067680 comp rx=0 tx=0).stop 2024-03-18T16:15:00.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.138+0000 7f97d1f6a740 1 -- 172.21.15.39:0/2230837894 shutdown_connections 2024-03-18T16:15:00.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.138+0000 7f97d1f6a740 1 --2- 172.21.15.39:0/2230837894 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55b0c5812400 0x55b0c5814880 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:00.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.138+0000 7f97d1f6a740 1 --2- 172.21.15.39:0/2230837894 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b0c5801d80 0x55b0c580daa0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:00.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.138+0000 7f97d1f6a740 1 --2- 172.21.15.39:0/2230837894 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b0c57fcc10 0x55b0c5806ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:00.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.138+0000 7f97ae000640 0 client.26518 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:15:00.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.138+0000 7f97d1f6a740 1 --2- 172.21.15.39:0/2230837894 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b0c57fc220 0x55b0c5806590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:00.141 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.138+0000 7f97d1f6a740 1 -- 172.21.15.39:0/2230837894 >> 172.21.15.39:0/2230837894 conn(0x55b0c57f7790 msgr2=0x55b0c57ee110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:00.141 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.139+0000 7f97d1f6a740 1 -- 172.21.15.39:0/2230837894 shutdown_connections 2024-03-18T16:15:00.141 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.140+0000 7f97d1f6a740 1 -- 172.21.15.39:0/2230837894 wait complete. 2024-03-18T16:15:00.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.140+0000 7f97d1f6a740 20 client.26518 trim_cache size 0 max 16384 2024-03-18T16:15:00.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.177+0000 7f97d1f6a740 1 Processor -- start 2024-03-18T16:15:00.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.177+0000 7f97d1f6a740 1 -- start start 2024-03-18T16:15:00.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.177+0000 7f97d1f6a740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b0c57fc220 0x55b0c57fdf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:00.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.177+0000 7f97d1f6a740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b0c57fcc10 0x55b0c57fe4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:00.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.177+0000 7f97d1f6a740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b0c5801d80 0x55b0c580da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:00.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.177+0000 7f97d1f6a740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f97c013c7b0 con 0x55b0c57fcc10 2024-03-18T16:15:00.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.177+0000 7f97d1f6a740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f97c013e320 con 0x55b0c57fc220 2024-03-18T16:15:00.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.177+0000 7f97d1f6a740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f97c013ce80 con 0x55b0c5801d80 2024-03-18T16:15:00.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.178+0000 7f97c7e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b0c57fc220 0x55b0c57fdf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:00.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.178+0000 7f97c7e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b0c57fc220 0x55b0c57fdf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:50802/0 (socket says 172.21.15.39:50802) 2024-03-18T16:15:00.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.178+0000 7f97c7e00640 1 -- 172.21.15.39:0/3422005568 learned_addr learned my addr 172.21.15.39:0/3422005568 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:15:00.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.179+0000 7f97c7400640 1 --2- 172.21.15.39:0/3422005568 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b0c57fcc10 0x55b0c57fe4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:00.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.179+0000 7f97c7e00640 1 -- 172.21.15.39:0/3422005568 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b0c5801d80 msgr2=0x55b0c580da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:00.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.179+0000 7f97c7e00640 1 --2- 172.21.15.39:0/3422005568 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b0c5801d80 0x55b0c580da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:00.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.179+0000 7f97c7e00640 1 -- 172.21.15.39:0/3422005568 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b0c57fcc10 msgr2=0x55b0c57fe4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:00.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.180+0000 7f97c7e00640 1 --2- 172.21.15.39:0/3422005568 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b0c57fcc10 0x55b0c57fe4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:00.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.180+0000 7f97c7e00640 1 -- 172.21.15.39:0/3422005568 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b0c57fc600 con 0x55b0c57fc220 2024-03-18T16:15:00.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.180+0000 7f97c7400640 1 --2- 172.21.15.39:0/3422005568 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b0c57fcc10 0x55b0c57fe4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:15:00.183 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.182+0000 7f97c7e00640 1 --2- 172.21.15.39:0/3422005568 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b0c57fc220 0x55b0c57fdf70 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f97b402c860 tx=0x7f97b4004a80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4fe17339752313a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:00.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.182+0000 7f97b8c00640 1 -- 172.21.15.39:0/3422005568 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97b4043040 con 0x55b0c57fc220 2024-03-18T16:15:00.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.182+0000 7f97b8c00640 1 -- 172.21.15.39:0/3422005568 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97b4004bb0 con 0x55b0c57fc220 2024-03-18T16:15:00.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.182+0000 7f97b8c00640 1 -- 172.21.15.39:0/3422005568 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97b4043040 con 0x55b0c57fc220 2024-03-18T16:15:00.185 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.183+0000 7f97d1f6a740 1 -- 172.21.15.39:0/3422005568 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b0c57fc220 msgr2=0x55b0c57fdf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:00.185 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.183+0000 7f97d1f6a740 1 --2- 172.21.15.39:0/3422005568 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b0c57fc220 0x55b0c57fdf70 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f97b402c860 tx=0x7f97b4004a80 comp rx=0 tx=0).stop 2024-03-18T16:15:00.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.186+0000 7f97d1f6a740 1 -- 172.21.15.39:0/3422005568 shutdown_connections 2024-03-18T16:15:00.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.186+0000 7f97d1f6a740 1 --2- 172.21.15.39:0/3422005568 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b0c5801d80 0x55b0c580da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:00.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.186+0000 7f97d1f6a740 1 --2- 172.21.15.39:0/3422005568 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b0c57fcc10 0x55b0c57fe4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:00.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.186+0000 7f97d1f6a740 1 --2- 172.21.15.39:0/3422005568 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b0c57fc220 0x55b0c57fdf70 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:00.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.186+0000 7f97d1f6a740 1 -- 172.21.15.39:0/3422005568 >> 172.21.15.39:0/3422005568 conn(0x55b0c57f7790 msgr2=0x55b0c5806590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:00.189 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.187+0000 7f97d1f6a740 1 -- 172.21.15.39:0/3422005568 shutdown_connections 2024-03-18T16:15:00.189 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.187+0000 7f97d1f6a740 1 -- 172.21.15.39:0/3422005568 wait complete. 2024-03-18T16:15:00.189 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.187+0000 7f97d1f6a740 1 -- start start 2024-03-18T16:15:00.189 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.188+0000 7f97d1f6a740 1 Processor -- start 2024-03-18T16:15:00.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.188+0000 7f97d1f6a740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b0c57fc220 0x55b0c580b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:00.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.188+0000 7f97d1f6a740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b0c57fcc10 0x55b0c580b800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:00.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.188+0000 7f97d1f6a740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b0c5801d80 0x55b0c580c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:00.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.188+0000 7f97d1f6a740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f97c013dc20 con 0x55b0c57fc220 2024-03-18T16:15:00.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.188+0000 7f97d1f6a740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f97c013d550 con 0x55b0c57fcc10 2024-03-18T16:15:00.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.188+0000 7f97d1f6a740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f97c013a7f0 con 0x55b0c5801d80 2024-03-18T16:15:00.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.188+0000 7f97d1f6a740 10 client.0 _ll_register_callbacks cb 0x7f97c804fdf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:15:00.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.189+0000 7f97cd200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b0c5801d80 0x55b0c580c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:00.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.189+0000 7f97cd200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b0c5801d80 0x55b0c580c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:57520/0 (socket says 172.21.15.39:57520) 2024-03-18T16:15:00.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.189+0000 7f97cd200640 1 -- 172.21.15.39:0/730573339 learned_addr learned my addr 172.21.15.39:0/730573339 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:15:00.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.189+0000 7f97c7400640 1 --2- 172.21.15.39:0/730573339 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b0c57fcc10 0x55b0c580b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:00.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.189+0000 7f97cd200640 1 -- 172.21.15.39:0/730573339 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b0c57fcc10 msgr2=0x55b0c580b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:00.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.189+0000 7f97cd200640 1 --2- 172.21.15.39:0/730573339 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b0c57fcc10 0x55b0c580b800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:00.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.189+0000 7f97cd200640 1 -- 172.21.15.39:0/730573339 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b0c57fc220 msgr2=0x55b0c580b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:00.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.189+0000 7f97cd200640 1 --2- 172.21.15.39:0/730573339 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b0c57fc220 0x55b0c580b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:00.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.189+0000 7f97cd200640 1 -- 172.21.15.39:0/730573339 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b0c580cd90 con 0x55b0c5801d80 2024-03-18T16:15:00.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.190+0000 7f97cd200640 1 --2- 172.21.15.39:0/730573339 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b0c5801d80 0x55b0c580c7a0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f97c8066f10 tx=0x7f97c80956d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ea717e8bd8d4dd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:00.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.190+0000 7f97ae000640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:15:00.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.190+0000 7f97ae000640 1 -- 172.21.15.39:0/730573339 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97c809a080 con 0x55b0c5801d80 2024-03-18T16:15:00.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.190+0000 7f97ae000640 1 -- 172.21.15.39:0/730573339 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f97c808b7a0 con 0x55b0c5801d80 2024-03-18T16:15:00.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.190+0000 7f97d1f6a740 1 -- 172.21.15.39:0/730573339 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f97b4004bb0 con 0x55b0c5801d80 2024-03-18T16:15:00.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.191+0000 7f97ae000640 1 -- 172.21.15.39:0/730573339 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f97c808b9a0 con 0x55b0c5801d80 2024-03-18T16:15:00.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.191+0000 7f97d1f6a740 10 client.26521 fetch_fsmap learned FSMap version 40 2024-03-18T16:15:00.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.191+0000 7f97d1f6a740 1 -- 172.21.15.39:0/730573339 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55b0c5889960 con 0x55b0c5801d80 2024-03-18T16:15:00.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.191+0000 7f97ae000640 1 -- 172.21.15.39:0/730573339 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97c808bb40 con 0x55b0c5801d80 2024-03-18T16:15:00.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.191+0000 7f97ae000640 1 -- 172.21.15.39:0/730573339 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 40) ==== 8378+0+0 (secure 0 0 0) 0x7f97c806e070 con 0x55b0c5801d80 2024-03-18T16:15:00.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.192+0000 7f97d1f6a740 10 client.26521 fetch_fsmap finished waiting for FSMap version 40 2024-03-18T16:15:00.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.192+0000 7f97d1f6a740 10 client.26521 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:15:00.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.192+0000 7f97d1f6a740 20 client.26521 populate_metadata read hostname 'smithi039' 2024-03-18T16:15:00.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.192+0000 7f97d1f6a740 1 --2- 172.21.15.39:0/730573339 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55b0c5812400 0x55b0c588c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:00.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.192+0000 7f97d1f6a740 4 client.26521 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T16:15:00.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.192+0000 7f97d1f6a740 1 -- 172.21.15.39:0/730573339 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f97b4004bb0 con 0x55b0c5812400 2024-03-18T16:15:00.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.192+0000 7f97c7e00640 1 --2- 172.21.15.39:0/730573339 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55b0c5812400 0x55b0c588c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:00.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.194+0000 7f97c7e00640 1 --2- 172.21.15.39:0/730573339 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55b0c5812400 0x55b0c588c350 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9b6ade089c853491 server_cookie=8378c367cce8c4d1 in_seq=0 out_seq=0 2024-03-18T16:15:00.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.194+0000 7f97ae000640 10 client.26521 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:15:00.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.194+0000 7f97ae000640 1 -- 172.21.15.39:0/730573339 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x7f97b4004bb0 con 0x55b0c5812400 2024-03-18T16:15:00.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.194+0000 7f97ae000640 10 client.26521 handle_command_reply: tid=0 2024-03-18T16:15:00.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.194+0000 7f97d1f6a740 1 client.26521 shutdown 2024-03-18T16:15:00.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.195+0000 7f97d1f6a740 1 -- 172.21.15.39:0/730573339 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b0c5801d80 msgr2=0x55b0c580c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:00.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.195+0000 7f97d1f6a740 1 --2- 172.21.15.39:0/730573339 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b0c5801d80 0x55b0c580c7a0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f97c8066f10 tx=0x7f97c80956d0 comp rx=0 tx=0).stop 2024-03-18T16:15:00.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.195+0000 7f97d1f6a740 1 -- 172.21.15.39:0/730573339 shutdown_connections 2024-03-18T16:15:00.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.195+0000 7f97d1f6a740 1 --2- 172.21.15.39:0/730573339 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55b0c5812400 0x55b0c588c350 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:00.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.195+0000 7f97d1f6a740 1 --2- 172.21.15.39:0/730573339 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b0c57fcc10 0x55b0c580b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:00.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.195+0000 7f97ae000640 0 client.26521 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:15:00.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.195+0000 7f97d1f6a740 1 --2- 172.21.15.39:0/730573339 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b0c5801d80 0x55b0c580c7a0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:00.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.196+0000 7f97d1f6a740 1 --2- 172.21.15.39:0/730573339 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55b0c57fc220 0x55b0c580b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:00.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.196+0000 7f97d1f6a740 1 -- 172.21.15.39:0/730573339 >> 172.21.15.39:0/730573339 conn(0x55b0c57f7790 msgr2=0x55b0c5806530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:00.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.197+0000 7f97d1f6a740 1 -- 172.21.15.39:0/730573339 shutdown_connections 2024-03-18T16:15:00.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.197+0000 7f97d1f6a740 1 -- 172.21.15.39:0/730573339 wait complete. 2024-03-18T16:15:00.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.197+0000 7f97d1f6a740 20 client.26521 trim_cache size 0 max 16384 2024-03-18T16:15:00.199 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-03-18T16:15:00.199 INFO:teuthology.orchestra.run.smithi039.stdout: "status": "no active scrubs running", 2024-03-18T16:15:00.199 INFO:teuthology.orchestra.run.smithi039.stdout: "scrubs": {} 2024-03-18T16:15:00.199 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-03-18T16:15:00.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.198+0000 7f97cdc00640 1 -- 172.21.15.39:0/1501191113 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f97b00a2880 msgr2=0x7f97b00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:00.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.198+0000 7f97cdc00640 1 --2- 172.21.15.39:0/1501191113 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f97b00a2880 0x7f97b00a4d40 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f97c8064cd0 tx=0x7f97c8075040 comp rx=0 tx=0).stop 2024-03-18T16:15:00.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.198+0000 7f97cdc00640 1 -- 172.21.15.39:0/1501191113 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f97c0009420 msgr2=0x7f97c012e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:00.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.198+0000 7f97cdc00640 1 --2- 172.21.15.39:0/1501191113 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f97c0009420 0x7f97c012e850 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f97bc00e9f0 tx=0x7f97bc00eeb0 comp rx=0 tx=0).stop 2024-03-18T16:15:00.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.198+0000 7f97cdc00640 1 -- 172.21.15.39:0/1501191113 shutdown_connections 2024-03-18T16:15:00.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.198+0000 7f97cdc00640 1 --2- 172.21.15.39:0/1501191113 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f97c0009420 0x7f97c012e850 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:00.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.198+0000 7f97cdc00640 1 --2- 172.21.15.39:0/1501191113 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f97b00a2880 0x7f97b00a4d40 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:00.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.198+0000 7f97cdc00640 1 --2- 172.21.15.39:0/1501191113 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f97c009dc00 0x7f97c01358c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:00.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.198+0000 7f97cdc00640 1 --2- 172.21.15.39:0/1501191113 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f97c0007e40 0x7f97c012e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:00.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.198+0000 7f97cdc00640 1 -- 172.21.15.39:0/1501191113 >> 172.21.15.39:0/1501191113 conn(0x7f97c0091300 msgr2=0x7f97c0099b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:00.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.199+0000 7f97cdc00640 1 -- 172.21.15.39:0/1501191113 shutdown_connections 2024-03-18T16:15:00.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.199+0000 7f97cdc00640 1 -- 172.21.15.39:0/1501191113 wait complete. 2024-03-18T16:15:00.229 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-18T16:15:00.229 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:15:00.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.541+0000 7f893dc00640 1 -- 172.21.15.39:0/517163362 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f89381055b0 msgr2=0x7f893810be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:00.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.541+0000 7f893dc00640 1 --2- 172.21.15.39:0/517163362 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f89381055b0 0x7f893810be40 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f8930009be0 tx=0x7f893002bef0 comp rx=0 tx=0).stop 2024-03-18T16:15:00.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.541+0000 7f893dc00640 1 -- 172.21.15.39:0/517163362 shutdown_connections 2024-03-18T16:15:00.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.541+0000 7f893dc00640 1 --2- 172.21.15.39:0/517163362 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f89381055b0 0x7f893810be40 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:00.544 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.541+0000 7f893dc00640 1 --2- 172.21.15.39:0/517163362 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8938104bf0 0x7f8938105070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:00.544 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.541+0000 7f893dc00640 1 --2- 172.21.15.39:0/517163362 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8938103a00 0x7f8938103e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:00.544 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.541+0000 7f893dc00640 1 -- 172.21.15.39:0/517163362 >> 172.21.15.39:0/517163362 conn(0x7f89380fef90 msgr2=0x7f89381013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:00.544 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.541+0000 7f893dc00640 1 -- 172.21.15.39:0/517163362 shutdown_connections 2024-03-18T16:15:00.544 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.541+0000 7f893dc00640 1 -- 172.21.15.39:0/517163362 wait complete. 2024-03-18T16:15:00.544 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.543+0000 7f893dc00640 1 Processor -- start 2024-03-18T16:15:00.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.545+0000 7f893dc00640 1 -- start start 2024-03-18T16:15:00.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.545+0000 7f893dc00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8938103a00 0x7f8938199c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:00.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.545+0000 7f893dc00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8938104bf0 0x7f893819a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:00.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.546+0000 7f893dc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f89381055b0 0x7f89381a11f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:00.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.546+0000 7f893dc00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f893810f6b0 con 0x7f89381055b0 2024-03-18T16:15:00.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.546+0000 7f893dc00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f893810f530 con 0x7f8938103a00 2024-03-18T16:15:00.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.546+0000 7f893dc00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f893810f830 con 0x7f8938104bf0 2024-03-18T16:15:00.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.546+0000 7f8936000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8938104bf0 0x7f893819a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:00.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.546+0000 7f8936000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8938104bf0 0x7f893819a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:57546/0 (socket says 172.21.15.39:57546) 2024-03-18T16:15:00.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.546+0000 7f8936000640 1 -- 172.21.15.39:0/1083750479 learned_addr learned my addr 172.21.15.39:0/1083750479 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:15:00.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.547+0000 7f8936000640 1 -- 172.21.15.39:0/1083750479 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8938103a00 msgr2=0x7f8938199c80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T16:15:00.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.547+0000 7f8936000640 1 --2- 172.21.15.39:0/1083750479 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8938103a00 0x7f8938199c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:00.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.547+0000 7f8936000640 1 -- 172.21.15.39:0/1083750479 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f89381055b0 msgr2=0x7f89381a11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:00.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.547+0000 7f8936000640 1 --2- 172.21.15.39:0/1083750479 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f89381055b0 0x7f89381a11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:00.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.547+0000 7f8936000640 1 -- 172.21.15.39:0/1083750479 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89381a1890 con 0x7f8938104bf0 2024-03-18T16:15:00.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.547+0000 7f8936000640 1 --2- 172.21.15.39:0/1083750479 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8938104bf0 0x7f893819a1c0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f892800ea60 tx=0x7f892800ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:00.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.548+0000 7f892f400640 1 -- 172.21.15.39:0/1083750479 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f892800cde0 con 0x7f8938104bf0 2024-03-18T16:15:00.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.548+0000 7f892f400640 1 -- 172.21.15.39:0/1083750479 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89280035f0 con 0x7f8938104bf0 2024-03-18T16:15:00.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.548+0000 7f892f400640 1 -- 172.21.15.39:0/1083750479 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8928010690 con 0x7f8938104bf0 2024-03-18T16:15:00.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.548+0000 7f893dc00640 1 -- 172.21.15.39:0/1083750479 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89381a1b80 con 0x7f8938104bf0 2024-03-18T16:15:00.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.548+0000 7f893dc00640 1 -- 172.21.15.39:0/1083750479 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89381a2120 con 0x7f8938104bf0 2024-03-18T16:15:00.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.551+0000 7f892cc00640 1 -- 172.21.15.39:0/1083750479 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89380623e0 con 0x7f8938104bf0 2024-03-18T16:15:00.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.554+0000 7f892f400640 1 -- 172.21.15.39:0/1083750479 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f89280037a0 con 0x7f8938104bf0 2024-03-18T16:15:00.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.561+0000 7f892f400640 1 --2- 172.21.15.39:0/1083750479 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f891c0a2880 0x7f891c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:00.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.562+0000 7f892f400640 1 -- 172.21.15.39:0/1083750479 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f89280cf250 con 0x7f8938104bf0 2024-03-18T16:15:00.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.563+0000 7f892f400640 1 -- 172.21.15.39:0/1083750479 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8928097de0 con 0x7f8938104bf0 2024-03-18T16:15:00.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.565+0000 7f8936a00640 1 --2- 172.21.15.39:0/1083750479 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f891c0a2880 0x7f891c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:00.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:00.567+0000 7f8936a00640 1 --2- 172.21.15.39:0/1083750479 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f891c0a2880 0x7f891c0a4d40 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f893819b150 tx=0x7f8924009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:01.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.060+0000 7f892cc00640 1 -- 172.21.15.39:0/1083750479 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8938062600 con 0x7f8938104bf0 2024-03-18T16:15:01.063 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.062+0000 7f892f400640 1 -- 172.21.15.39:0/1083750479 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) ==== 94+0+11937 (secure 0 0 0) 0x7f892809bf00 con 0x7f8938104bf0 2024-03-18T16:15:01.064 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:15:01.064 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2520226608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2520226608},{"type":"v1","addr":"172.21.15.39:6835","nonce":2520226608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/719341763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":719341763},{"type":"v1","addr":"172.21.15.39:6837","nonce":719341763}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":40,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T16:06:44.180425+0000","modified":"2024-03-18T16:14:56.219132+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24575,"mds_1":24556,"mds_2":24535,"mds_3":24524,"mds_4":14463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14445":{"gid":14445,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.39:6839/656513813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":656513813},{"type":"v1","addr":"172.21.15.39:6839","nonce":656513813}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14463":{"gid":14463,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":13,"addr":"172.21.15.39:6841/3515326042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3515326042},{"type":"v1","addr":"172.21.15.39:6841","nonce":3515326042}]},"join_fscid":-1,"export_targets":[3],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6833/931350738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":931350738},{"type":"v1","addr":"172.21.15.194:6833","nonce":931350738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6833/3874209621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":3874209621},{"type":"v1","addr":"172.21.15.196:6833","nonce":3874209621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6835/188385470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":188385470},{"type":"v1","addr":"172.21.15.194:6835","nonce":188385470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6835/3949984309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":3949984309},{"type":"v1","addr":"172.21.15.196:6835","nonce":3949984309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":15,"addr":"172.21.15.194:6837/1301464246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":1301464246},{"type":"v1","addr":"172.21.15.194:6837","nonce":1301464246}]},"join_fscid":-1,"export_targets":[1,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":16,"addr":"172.21.15.196:6837/895247251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":895247251},{"type":"v1","addr":"172.21.15.196:6837","nonce":895247251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.196:6839/2001427268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6838","nonce":2001427268},{"type":"v1","addr":"172.21.15.196:6839","nonce":2001427268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24575":{"gid":24575,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.194:6839/274911158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":274911158},{"type":"v1","addr":"172.21.15.194:6839","nonce":274911158}]},"join_fscid":-1,"export_targets":[4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24575,"qdb_cluster":[14463,24524,24575,24556,24535]},"id":1}]} 2024-03-18T16:15:01.064 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 40 2024-03-18T16:15:01.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.067+0000 7f892cc00640 1 -- 172.21.15.39:0/1083750479 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f891c0a2880 msgr2=0x7f891c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:01.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.067+0000 7f892cc00640 1 --2- 172.21.15.39:0/1083750479 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f891c0a2880 0x7f891c0a4d40 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f893819b150 tx=0x7f8924009260 comp rx=0 tx=0).stop 2024-03-18T16:15:01.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.067+0000 7f892cc00640 1 -- 172.21.15.39:0/1083750479 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8938104bf0 msgr2=0x7f893819a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:01.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.067+0000 7f892cc00640 1 --2- 172.21.15.39:0/1083750479 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8938104bf0 0x7f893819a1c0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f892800ea60 tx=0x7f892800ef20 comp rx=0 tx=0).stop 2024-03-18T16:15:01.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.067+0000 7f892cc00640 1 -- 172.21.15.39:0/1083750479 shutdown_connections 2024-03-18T16:15:01.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.067+0000 7f892cc00640 1 --2- 172.21.15.39:0/1083750479 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f891c0a2880 0x7f891c0a4d40 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:01.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.067+0000 7f892cc00640 1 --2- 172.21.15.39:0/1083750479 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f89381055b0 0x7f89381a11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:01.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.067+0000 7f892cc00640 1 --2- 172.21.15.39:0/1083750479 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8938104bf0 0x7f893819a1c0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:01.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.067+0000 7f892cc00640 1 --2- 172.21.15.39:0/1083750479 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8938103a00 0x7f8938199c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:01.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.067+0000 7f892cc00640 1 -- 172.21.15.39:0/1083750479 >> 172.21.15.39:0/1083750479 conn(0x7f89380fef90 msgr2=0x7f8938107e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:01.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.067+0000 7f892cc00640 1 -- 172.21.15.39:0/1083750479 shutdown_connections 2024-03-18T16:15:01.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.067+0000 7f892cc00640 1 -- 172.21.15.39:0/1083750479 wait complete. 2024-03-18T16:15:01.097 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-18T16:15:01.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:01 smithi039 ceph-mon[31721]: pgmap v570: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 32 MiB/s wr, 6.28k op/s 2024-03-18T16:15:01.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.388+0000 7f574e000640 1 -- 172.21.15.39:0/1741425069 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5748100a30 msgr2=0x7f5748100e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:01.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.388+0000 7f574e000640 1 --2- 172.21.15.39:0/1741425069 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5748100a30 0x7f5748100e90 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f5730009900 tx=0x7f573002bce0 comp rx=0 tx=0).stop 2024-03-18T16:15:01.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.389+0000 7f574e000640 1 -- 172.21.15.39:0/1741425069 shutdown_connections 2024-03-18T16:15:01.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.389+0000 7f574e000640 1 --2- 172.21.15.39:0/1741425069 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5748100a30 0x7f5748100e90 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:01.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.389+0000 7f574e000640 1 --2- 172.21.15.39:0/1741425069 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f57481013d0 0x7f5748101850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:01.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.389+0000 7f574e000640 1 --2- 172.21.15.39:0/1741425069 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f57481000d0 0x7f57481004f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:01.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.389+0000 7f574e000640 1 -- 172.21.15.39:0/1741425069 >> 172.21.15.39:0/1741425069 conn(0x7f57480fbc50 msgr2=0x7f57480fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:01.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.389+0000 7f574e000640 1 -- 172.21.15.39:0/1741425069 shutdown_connections 2024-03-18T16:15:01.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.389+0000 7f574e000640 1 -- 172.21.15.39:0/1741425069 wait complete. 2024-03-18T16:15:01.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.390+0000 7f574e000640 1 Processor -- start 2024-03-18T16:15:01.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.391+0000 7f574e000640 1 -- start start 2024-03-18T16:15:01.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.391+0000 7f574e000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f57481000d0 0x7f5748199d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:01.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.391+0000 7f574e000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5748100a30 0x7f574819a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:01.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.391+0000 7f574e000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f57481013d0 0x7f57481a1320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:01.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.391+0000 7f5747400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f57481000d0 0x7f5748199d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:01.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.392+0000 7f5747400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f57481000d0 0x7f5748199d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:57566/0 (socket says 172.21.15.39:57566) 2024-03-18T16:15:01.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.392+0000 7f574e000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f574810f700 con 0x7f5748100a30 2024-03-18T16:15:01.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.392+0000 7f5747400640 1 -- 172.21.15.39:0/2160740362 learned_addr learned my addr 172.21.15.39:0/2160740362 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:15:01.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.392+0000 7f574e000640 1 -- 172.21.15.39:0/2160740362 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f574810f580 con 0x7f57481013d0 2024-03-18T16:15:01.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.392+0000 7f574e000640 1 -- 172.21.15.39:0/2160740362 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f574810f880 con 0x7f57481000d0 2024-03-18T16:15:01.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.392+0000 7f5747e00640 1 --2- 172.21.15.39:0/2160740362 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f57481013d0 0x7f57481a1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:01.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.393+0000 7f5746a00640 1 --2- 172.21.15.39:0/2160740362 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5748100a30 0x7f574819a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:01.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.393+0000 7f5747e00640 1 -- 172.21.15.39:0/2160740362 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f57481000d0 msgr2=0x7f5748199d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:01.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.393+0000 7f5747e00640 1 --2- 172.21.15.39:0/2160740362 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f57481000d0 0x7f5748199d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:01.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.393+0000 7f5747e00640 1 -- 172.21.15.39:0/2160740362 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5748100a30 msgr2=0x7f574819a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:01.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.393+0000 7f5747e00640 1 --2- 172.21.15.39:0/2160740362 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5748100a30 0x7f574819a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:01.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.393+0000 7f5747e00640 1 -- 172.21.15.39:0/2160740362 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57481a1990 con 0x7f57481013d0 2024-03-18T16:15:01.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.394+0000 7f5747e00640 1 --2- 172.21.15.39:0/2160740362 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f57481013d0 0x7f57481a1320 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f574000ec50 tx=0x7f574000c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:01.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.395+0000 7f573fe00640 1 -- 172.21.15.39:0/2160740362 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5740003b70 con 0x7f57481013d0 2024-03-18T16:15:01.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.395+0000 7f573fe00640 1 -- 172.21.15.39:0/2160740362 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5740003d10 con 0x7f57481013d0 2024-03-18T16:15:01.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.395+0000 7f574e000640 1 -- 172.21.15.39:0/2160740362 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57481a1c80 con 0x7f57481013d0 2024-03-18T16:15:01.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.396+0000 7f574e000640 1 -- 172.21.15.39:0/2160740362 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57481a21a0 con 0x7f57481013d0 2024-03-18T16:15:01.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.396+0000 7f573fe00640 1 -- 172.21.15.39:0/2160740362 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5740010890 con 0x7f57481013d0 2024-03-18T16:15:01.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.401+0000 7f573fe00640 1 -- 172.21.15.39:0/2160740362 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f5740003fc0 con 0x7f57481013d0 2024-03-18T16:15:01.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.403+0000 7f5750dd8740 1 Processor -- start 2024-03-18T16:15:01.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.404+0000 7f5750dd8740 1 -- start start 2024-03-18T16:15:01.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.405+0000 7f5750dd8740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a2a6c168c0 0x55a2a6c16ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:01.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.405+0000 7f5750dd8740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a2a6c172b0 0x55a2a6c1bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:01.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.405+0000 7f5750dd8740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a2a6c1c420 0x55a2a6c1e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:01.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.405+0000 7f5750dd8740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55a2a6bde3b0 con 0x55a2a6c172b0 2024-03-18T16:15:01.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.405+0000 7f5750dd8740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55a2a6b7b450 con 0x55a2a6c1c420 2024-03-18T16:15:01.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.405+0000 7f5750dd8740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55a2a6b23cb0 con 0x55a2a6c168c0 2024-03-18T16:15:01.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.406+0000 7f5746a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a2a6c168c0 0x55a2a6c16ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:01.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.406+0000 7f5746a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a2a6c168c0 0x55a2a6c16ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:57576/0 (socket says 172.21.15.39:57576) 2024-03-18T16:15:01.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.406+0000 7f573fe00640 1 --2- 172.21.15.39:0/2160740362 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f572c0a29d0 0x7f572c0a4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:01.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.407+0000 7f573fe00640 1 -- 172.21.15.39:0/2160740362 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f57400cf210 con 0x7f57481013d0 2024-03-18T16:15:01.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.407+0000 7f5747400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a2a6c172b0 0x55a2a6c1bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:01.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.407+0000 7f5746a00640 1 -- 172.21.15.39:0/3506121340 learned_addr learned my addr 172.21.15.39:0/3506121340 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:15:01.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.407+0000 7f5746a00640 1 --2- 172.21.15.39:0/3506121340 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a2a6c1c420 0x55a2a6c1e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:01.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.408+0000 7f5746a00640 1 -- 172.21.15.39:0/3506121340 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a2a6c1c420 msgr2=0x55a2a6c1e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:01.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.408+0000 7f5746a00640 1 --2- 172.21.15.39:0/3506121340 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a2a6c1c420 0x55a2a6c1e8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:01.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.408+0000 7f5746a00640 1 -- 172.21.15.39:0/3506121340 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a2a6c172b0 msgr2=0x55a2a6c1bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:01.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.408+0000 7f5746a00640 1 --2- 172.21.15.39:0/3506121340 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a2a6c172b0 0x55a2a6c1bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:01.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.408+0000 7f5746a00640 1 -- 172.21.15.39:0/3506121340 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a2a6b451f0 con 0x55a2a6c168c0 2024-03-18T16:15:01.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.408+0000 7f5746a00640 1 --2- 172.21.15.39:0/3506121340 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a2a6c168c0 0x55a2a6c16ce0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f5730000c00 tx=0x7f5730005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cabba7e1212a0a7b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:01.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.409+0000 7f5747e00640 1 --2- 172.21.15.39:0/2160740362 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f572c0a29d0 0x7f572c0a4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:01.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.409+0000 7f5737e00640 1 -- 172.21.15.39:0/3506121340 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57300040f0 con 0x55a2a6c168c0 2024-03-18T16:15:01.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.409+0000 7f5737e00640 1 -- 172.21.15.39:0/3506121340 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f57300054e0 con 0x55a2a6c168c0 2024-03-18T16:15:01.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.409+0000 7f5737e00640 1 -- 172.21.15.39:0/3506121340 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5730004ae0 con 0x55a2a6c168c0 2024-03-18T16:15:01.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.411+0000 7f5750dd8740 1 -- 172.21.15.39:0/3506121340 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a2a6c168c0 msgr2=0x55a2a6c16ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:01.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.411+0000 7f5750dd8740 1 --2- 172.21.15.39:0/3506121340 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a2a6c168c0 0x55a2a6c16ce0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f5730000c00 tx=0x7f5730005fd0 comp rx=0 tx=0).stop 2024-03-18T16:15:01.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.411+0000 7f5747e00640 1 --2- 172.21.15.39:0/2160740362 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f572c0a29d0 0x7f572c0a4e90 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f5740097c20 tx=0x7f5740005f30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:01.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.412+0000 7f5750dd8740 1 -- 172.21.15.39:0/3506121340 shutdown_connections 2024-03-18T16:15:01.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.412+0000 7f5750dd8740 1 --2- 172.21.15.39:0/3506121340 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a2a6c1c420 0x55a2a6c1e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:01.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.412+0000 7f5750dd8740 1 --2- 172.21.15.39:0/3506121340 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a2a6c172b0 0x55a2a6c1bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:01.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.412+0000 7f5750dd8740 1 --2- 172.21.15.39:0/3506121340 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a2a6c168c0 0x55a2a6c16ce0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:01.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.412+0000 7f5750dd8740 1 -- 172.21.15.39:0/3506121340 >> 172.21.15.39:0/3506121340 conn(0x55a2a6c11e30 msgr2=0x55a2a6c12230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:01.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.412+0000 7f5750dd8740 1 -- 172.21.15.39:0/3506121340 shutdown_connections 2024-03-18T16:15:01.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.413+0000 7f5750dd8740 1 -- 172.21.15.39:0/3506121340 wait complete. 2024-03-18T16:15:01.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.414+0000 7f5750dd8740 1 -- start start 2024-03-18T16:15:01.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.414+0000 7f5750dd8740 1 Processor -- start 2024-03-18T16:15:01.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.415+0000 7f5750dd8740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a2a6c168c0 0x55a2a6c31c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:01.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.416+0000 7f5750dd8740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a2a6c172b0 0x55a2a6c32170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:01.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.416+0000 7f5750dd8740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a2a6c1c420 0x55a2a6c391b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:01.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.416+0000 7f5750dd8740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55a2a6b37110 con 0x55a2a6c1c420 2024-03-18T16:15:01.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.416+0000 7f5750dd8740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55a2a6b774f0 con 0x55a2a6c172b0 2024-03-18T16:15:01.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.416+0000 7f5750dd8740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55a2a6c30380 con 0x55a2a6c168c0 2024-03-18T16:15:01.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.416+0000 7f5750dd8740 10 client.0 _ll_register_callbacks cb 0x55a2a6b24b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:15:01.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.416+0000 7f5747400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a2a6c172b0 0x55a2a6c32170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:01.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.417+0000 7f5747400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a2a6c172b0 0x55a2a6c32170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:50862/0 (socket says 172.21.15.39:50862) 2024-03-18T16:15:01.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.417+0000 7f5747400640 1 -- 172.21.15.39:0/3218574785 learned_addr learned my addr 172.21.15.39:0/3218574785 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:15:01.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.417+0000 7f5747400640 1 -- 172.21.15.39:0/3218574785 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a2a6c168c0 msgr2=0x55a2a6c31c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:01.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.416+0000 7f5746a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a2a6c168c0 0x55a2a6c31c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:01.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.419+0000 7f5747400640 1 --2- 172.21.15.39:0/3218574785 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a2a6c168c0 0x55a2a6c31c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:01.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.419+0000 7f5747400640 1 -- 172.21.15.39:0/3218574785 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a2a6c1c420 msgr2=0x55a2a6c391b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:15:01.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.419+0000 7f5747400640 1 --2- 172.21.15.39:0/3218574785 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a2a6c1c420 0x55a2a6c391b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:01.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.419+0000 7f5747400640 1 -- 172.21.15.39:0/3218574785 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a2a688f3f0 con 0x55a2a6c172b0 2024-03-18T16:15:01.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.419+0000 7f5746a00640 1 --2- 172.21.15.39:0/3218574785 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a2a6c168c0 0x55a2a6c31c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:15:01.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.419+0000 7f5747400640 1 --2- 172.21.15.39:0/3218574785 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a2a6c172b0 0x55a2a6c32170 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f5738009f40 tx=0x7f57380116f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=49fdf69ed89df975 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:01.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.419+0000 7f5734c00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:15:01.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.420+0000 7f5734c00640 1 -- 172.21.15.39:0/3218574785 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5738019070 con 0x55a2a6c172b0 2024-03-18T16:15:01.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.420+0000 7f5734c00640 1 -- 172.21.15.39:0/3218574785 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f5738015870 con 0x55a2a6c172b0 2024-03-18T16:15:01.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.420+0000 7f5750dd8740 1 -- 172.21.15.39:0/3218574785 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f57300054e0 con 0x55a2a6c172b0 2024-03-18T16:15:01.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.421+0000 7f5734c00640 1 -- 172.21.15.39:0/3218574785 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f5738023050 con 0x55a2a6c172b0 2024-03-18T16:15:01.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.422+0000 7f5750dd8740 10 client.26564 fetch_fsmap learned FSMap version 40 2024-03-18T16:15:01.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.423+0000 7f5750dd8740 1 -- 172.21.15.39:0/3218574785 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55a2a69597d0 con 0x55a2a6c172b0 2024-03-18T16:15:01.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.423+0000 7f5734c00640 1 -- 172.21.15.39:0/3218574785 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f57300054e0 con 0x55a2a6c172b0 2024-03-18T16:15:01.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.423+0000 7f5734c00640 1 -- 172.21.15.39:0/3218574785 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 40) ==== 8378+0+0 (secure 0 0 0) 0x7f573801ec90 con 0x55a2a6c172b0 2024-03-18T16:15:01.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.424+0000 7f5750dd8740 10 client.26564 fetch_fsmap finished waiting for FSMap version 40 2024-03-18T16:15:01.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.424+0000 7f5750dd8740 10 client.26564 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-03-18T16:15:01.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.424+0000 7f5750dd8740 20 client.26564 populate_metadata read hostname 'smithi039' 2024-03-18T16:15:01.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.424+0000 7f5750dd8740 1 --2- 172.21.15.39:0/3218574785 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x55a2a6c3dc20 0x55a2a6c400a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:01.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.424+0000 7f5750dd8740 4 client.26564 mds_command: new command op to 14463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:15:01.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.424+0000 7f5750dd8740 1 -- 172.21.15.39:0/3218574785 --> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a2a6b88990 con 0x55a2a6c3dc20 2024-03-18T16:15:01.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.424+0000 7f5746a00640 1 --2- 172.21.15.39:0/3218574785 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x55a2a6c3dc20 0x55a2a6c400a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:01.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.425+0000 7f5746a00640 1 --2- 172.21.15.39:0/3218574785 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x55a2a6c3dc20 0x55a2a6c400a0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=8c9fe0c2763cdb83 server_cookie=1d97e7704957d807 in_seq=0 out_seq=0 2024-03-18T16:15:01.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.425+0000 7f5734c00640 10 client.26564 ms_handle_connect on v2:172.21.15.39:6840/3515326042 2024-03-18T16:15:01.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.427+0000 7f5734c00640 1 -- 172.21.15.39:0/3218574785 <== mds.4 v2:172.21.15.39:6840/3515326042 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a2a6b88990 con 0x55a2a6c3dc20 2024-03-18T16:15:01.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.427+0000 7f5734c00640 10 client.26564 handle_command_reply: tid=0 2024-03-18T16:15:01.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.427+0000 7f5750dd8740 1 client.26564 shutdown 2024-03-18T16:15:01.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.428+0000 7f5750dd8740 1 -- 172.21.15.39:0/3218574785 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a2a6c172b0 msgr2=0x55a2a6c32170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:01.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.428+0000 7f5750dd8740 1 --2- 172.21.15.39:0/3218574785 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a2a6c172b0 0x55a2a6c32170 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f5738009f40 tx=0x7f57380116f0 comp rx=0 tx=0).stop 2024-03-18T16:15:01.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.428+0000 7f5750dd8740 1 -- 172.21.15.39:0/3218574785 shutdown_connections 2024-03-18T16:15:01.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.428+0000 7f5750dd8740 1 --2- 172.21.15.39:0/3218574785 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x55a2a6c3dc20 0x55a2a6c400a0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:01.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.428+0000 7f5734c00640 0 client.26564 ms_handle_reset on v2:172.21.15.39:6840/3515326042 2024-03-18T16:15:01.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.428+0000 7f5750dd8740 1 --2- 172.21.15.39:0/3218574785 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a2a6c172b0 0x55a2a6c32170 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:01.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.428+0000 7f5750dd8740 1 --2- 172.21.15.39:0/3218574785 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a2a6c1c420 0x55a2a6c391b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:01.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.428+0000 7f5750dd8740 1 --2- 172.21.15.39:0/3218574785 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a2a6c168c0 0x55a2a6c31c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:01.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.428+0000 7f5750dd8740 1 -- 172.21.15.39:0/3218574785 >> 172.21.15.39:0/3218574785 conn(0x55a2a6c11e30 msgr2=0x55a2a6c111f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:01.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.429+0000 7f5750dd8740 1 -- 172.21.15.39:0/3218574785 shutdown_connections 2024-03-18T16:15:01.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.429+0000 7f5750dd8740 1 -- 172.21.15.39:0/3218574785 wait complete. 2024-03-18T16:15:01.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.429+0000 7f5750dd8740 20 client.26564 trim_cache size 0 max 16384 2024-03-18T16:15:01.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.459+0000 7f5750dd8740 1 Processor -- start 2024-03-18T16:15:01.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.460+0000 7f5750dd8740 1 -- start start 2024-03-18T16:15:01.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.460+0000 7f5750dd8740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a2a6c168c0 0x55a2a6c157a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:01.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.460+0000 7f5750dd8740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a2a6c172b0 0x55a2a6c39170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:01.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.460+0000 7f5750dd8740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a2a6c1c420 0x55a2a6c44960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:01.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.460+0000 7f5750dd8740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f572400fca0 con 0x55a2a6c172b0 2024-03-18T16:15:01.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.460+0000 7f5750dd8740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f5724011810 con 0x55a2a6c168c0 2024-03-18T16:15:01.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.460+0000 7f5750dd8740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f5724010370 con 0x55a2a6c1c420 2024-03-18T16:15:01.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.460+0000 7f5746a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a2a6c168c0 0x55a2a6c157a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:01.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.461+0000 7f5746a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a2a6c168c0 0x55a2a6c157a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:50870/0 (socket says 172.21.15.39:50870) 2024-03-18T16:15:01.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.461+0000 7f5746a00640 1 -- 172.21.15.39:0/2398902997 learned_addr learned my addr 172.21.15.39:0/2398902997 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:15:01.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.461+0000 7f5746a00640 1 --2- 172.21.15.39:0/2398902997 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a2a6c1c420 0x55a2a6c44960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:01.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.461+0000 7f5746a00640 1 -- 172.21.15.39:0/2398902997 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a2a6c1c420 msgr2=0x55a2a6c44960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:01.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.461+0000 7f5746a00640 1 --2- 172.21.15.39:0/2398902997 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a2a6c1c420 0x55a2a6c44960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:01.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.461+0000 7f5746a00640 1 -- 172.21.15.39:0/2398902997 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a2a6c172b0 msgr2=0x55a2a6c39170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:01.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.461+0000 7f5746a00640 1 --2- 172.21.15.39:0/2398902997 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a2a6c172b0 0x55a2a6c39170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:01.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.461+0000 7f5746a00640 1 -- 172.21.15.39:0/2398902997 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a2a6c0dc10 con 0x55a2a6c168c0 2024-03-18T16:15:01.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.461+0000 7f5746a00640 1 --2- 172.21.15.39:0/2398902997 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a2a6c168c0 0x55a2a6c157a0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f573002c220 tx=0x7f573002bce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=59550cb8d819bab5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:01.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.462+0000 7f5737400640 1 -- 172.21.15.39:0/2398902997 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5730004370 con 0x55a2a6c168c0 2024-03-18T16:15:01.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.462+0000 7f5737400640 1 -- 172.21.15.39:0/2398902997 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f57300054e0 con 0x55a2a6c168c0 2024-03-18T16:15:01.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.462+0000 7f5737400640 1 -- 172.21.15.39:0/2398902997 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5730004d00 con 0x55a2a6c168c0 2024-03-18T16:15:01.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.463+0000 7f5750dd8740 1 -- 172.21.15.39:0/2398902997 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a2a6c168c0 msgr2=0x55a2a6c157a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:01.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.463+0000 7f5750dd8740 1 --2- 172.21.15.39:0/2398902997 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a2a6c168c0 0x55a2a6c157a0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f573002c220 tx=0x7f573002bce0 comp rx=0 tx=0).stop 2024-03-18T16:15:01.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.463+0000 7f5750dd8740 1 -- 172.21.15.39:0/2398902997 shutdown_connections 2024-03-18T16:15:01.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.463+0000 7f5750dd8740 1 --2- 172.21.15.39:0/2398902997 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a2a6c1c420 0x55a2a6c44960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:01.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.463+0000 7f5750dd8740 1 --2- 172.21.15.39:0/2398902997 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a2a6c172b0 0x55a2a6c39170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:01.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.463+0000 7f5750dd8740 1 --2- 172.21.15.39:0/2398902997 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a2a6c168c0 0x55a2a6c157a0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:01.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.463+0000 7f5750dd8740 1 -- 172.21.15.39:0/2398902997 >> 172.21.15.39:0/2398902997 conn(0x55a2a6c11e30 msgr2=0x55a2a6c15090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:01.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.463+0000 7f5750dd8740 1 -- 172.21.15.39:0/2398902997 shutdown_connections 2024-03-18T16:15:01.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.464+0000 7f5750dd8740 1 -- 172.21.15.39:0/2398902997 wait complete. 2024-03-18T16:15:01.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.464+0000 7f5750dd8740 1 -- start start 2024-03-18T16:15:01.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.465+0000 7f5750dd8740 1 Processor -- start 2024-03-18T16:15:01.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.465+0000 7f5750dd8740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a2a6c168c0 0x55a2a6c38930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:01.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.466+0000 7f5750dd8740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a2a6c172b0 0x55a2a6c38e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:01.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.466+0000 7f5750dd8740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a2a6c1c420 0x55a2a6c353d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:01.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.466+0000 7f5750dd8740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f5724011110 con 0x55a2a6c1c420 2024-03-18T16:15:01.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.466+0000 7f5750dd8740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f5724010a40 con 0x55a2a6c168c0 2024-03-18T16:15:01.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.466+0000 7f5750dd8740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f572400dc80 con 0x55a2a6c172b0 2024-03-18T16:15:01.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.466+0000 7f5750dd8740 10 client.0 _ll_register_callbacks cb 0x7f5738009530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:15:01.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.466+0000 7f5746a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a2a6c168c0 0x55a2a6c38930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:01.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.466+0000 7f5746a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a2a6c168c0 0x55a2a6c38930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:50878/0 (socket says 172.21.15.39:50878) 2024-03-18T16:15:01.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.466+0000 7f5746a00640 1 -- 172.21.15.39:0/3545833268 learned_addr learned my addr 172.21.15.39:0/3545833268 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:15:01.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.467+0000 7f5747400640 1 --2- 172.21.15.39:0/3545833268 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a2a6c172b0 0x55a2a6c38e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:01.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.468+0000 7f5746a00640 1 --2- 172.21.15.39:0/3545833268 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a2a6c1c420 0x55a2a6c353d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:01.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.468+0000 7f5747400640 1 -- 172.21.15.39:0/3545833268 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a2a6c168c0 msgr2=0x55a2a6c38930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:01.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.468+0000 7f5747400640 1 --2- 172.21.15.39:0/3545833268 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a2a6c168c0 0x55a2a6c38930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:01.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.468+0000 7f5747400640 1 -- 172.21.15.39:0/3545833268 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a2a6c1c420 msgr2=0x55a2a6c353d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:01.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.468+0000 7f5747400640 1 --2- 172.21.15.39:0/3545833268 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a2a6c1c420 0x55a2a6c353d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:01.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.468+0000 7f5747400640 1 -- 172.21.15.39:0/3545833268 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a2a6c12ba0 con 0x55a2a6c172b0 2024-03-18T16:15:01.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.468+0000 7f5746a00640 1 --2- 172.21.15.39:0/3545833268 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a2a6c1c420 0x55a2a6c353d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:15:01.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.469+0000 7f5747400640 1 --2- 172.21.15.39:0/3545833268 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a2a6c172b0 0x55a2a6c38e70 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x55a2a6c1bd20 tx=0x7f5738011ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fde6648cca110e55 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:01.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.469+0000 7f5734c00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:15:01.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.470+0000 7f5734c00640 1 -- 172.21.15.39:0/3545833268 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5738019040 con 0x55a2a6c172b0 2024-03-18T16:15:01.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.470+0000 7f5734c00640 1 -- 172.21.15.39:0/3545833268 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f5738015670 con 0x55a2a6c172b0 2024-03-18T16:15:01.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.470+0000 7f5750dd8740 1 -- 172.21.15.39:0/3545833268 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f57300054e0 con 0x55a2a6c172b0 2024-03-18T16:15:01.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.470+0000 7f5734c00640 1 -- 172.21.15.39:0/3545833268 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f5738014eb0 con 0x55a2a6c172b0 2024-03-18T16:15:01.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.470+0000 7f5750dd8740 10 client.26542 fetch_fsmap learned FSMap version 40 2024-03-18T16:15:01.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.470+0000 7f5750dd8740 1 -- 172.21.15.39:0/3545833268 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55a2a6cb5950 con 0x55a2a6c172b0 2024-03-18T16:15:01.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.471+0000 7f5734c00640 1 -- 172.21.15.39:0/3545833268 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f57300054e0 con 0x55a2a6c172b0 2024-03-18T16:15:01.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.471+0000 7f5734c00640 1 -- 172.21.15.39:0/3545833268 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 40) ==== 8378+0+0 (secure 0 0 0) 0x7f5738028b40 con 0x55a2a6c172b0 2024-03-18T16:15:01.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.471+0000 7f5750dd8740 10 client.26542 fetch_fsmap finished waiting for FSMap version 40 2024-03-18T16:15:01.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.471+0000 7f5750dd8740 10 client.26542 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-03-18T16:15:01.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.471+0000 7f5750dd8740 20 client.26542 populate_metadata read hostname 'smithi039' 2024-03-18T16:15:01.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.471+0000 7f5750dd8740 1 --2- 172.21.15.39:0/3545833268 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x55a2a6cb62f0 0x55a2a6cb8710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:01.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.471+0000 7f5750dd8740 4 client.26542 mds_command: new command op to 14463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:15:01.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.471+0000 7f5750dd8740 1 -- 172.21.15.39:0/3545833268 --> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a2a6b82ce0 con 0x55a2a6cb62f0 2024-03-18T16:15:01.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.472+0000 7f5746a00640 1 --2- 172.21.15.39:0/3545833268 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x55a2a6cb62f0 0x55a2a6cb8710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:01.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.473+0000 7f5746a00640 1 --2- 172.21.15.39:0/3545833268 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x55a2a6cb62f0 0x55a2a6cb8710 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=d1c132337e8c4508 server_cookie=4d667f7e52dfec0f in_seq=0 out_seq=0 2024-03-18T16:15:01.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.474+0000 7f5734c00640 10 client.26542 ms_handle_connect on v2:172.21.15.39:6840/3515326042 2024-03-18T16:15:01.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.474+0000 7f5734c00640 1 -- 172.21.15.39:0/3545833268 <== mds.4 v2:172.21.15.39:6840/3515326042 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a2a6b82ce0 con 0x55a2a6cb62f0 2024-03-18T16:15:01.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.474+0000 7f5734c00640 10 client.26542 handle_command_reply: tid=0 2024-03-18T16:15:01.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.475+0000 7f5750dd8740 1 client.26542 shutdown 2024-03-18T16:15:01.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.475+0000 7f5750dd8740 1 -- 172.21.15.39:0/3545833268 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a2a6c172b0 msgr2=0x55a2a6c38e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:01.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.475+0000 7f5750dd8740 1 --2- 172.21.15.39:0/3545833268 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a2a6c172b0 0x55a2a6c38e70 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x55a2a6c1bd20 tx=0x7f5738011ec0 comp rx=0 tx=0).stop 2024-03-18T16:15:01.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.475+0000 7f5750dd8740 1 -- 172.21.15.39:0/3545833268 shutdown_connections 2024-03-18T16:15:01.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.475+0000 7f5750dd8740 1 --2- 172.21.15.39:0/3545833268 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x55a2a6cb62f0 0x55a2a6cb8710 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:01.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.475+0000 7f5734c00640 0 client.26542 ms_handle_reset on v2:172.21.15.39:6840/3515326042 2024-03-18T16:15:01.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.475+0000 7f5750dd8740 1 --2- 172.21.15.39:0/3545833268 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a2a6c1c420 0x55a2a6c353d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:01.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.476+0000 7f5750dd8740 1 --2- 172.21.15.39:0/3545833268 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a2a6c172b0 0x55a2a6c38e70 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:01.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.476+0000 7f5750dd8740 1 --2- 172.21.15.39:0/3545833268 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a2a6c168c0 0x55a2a6c38930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:01.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.476+0000 7f5750dd8740 1 -- 172.21.15.39:0/3545833268 >> 172.21.15.39:0/3545833268 conn(0x55a2a6c11e30 msgr2=0x55a2a6c15920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:01.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.476+0000 7f5750dd8740 1 -- 172.21.15.39:0/3545833268 shutdown_connections 2024-03-18T16:15:01.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.477+0000 7f5750dd8740 1 -- 172.21.15.39:0/3545833268 wait complete. 2024-03-18T16:15:01.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.477+0000 7f5750dd8740 20 client.26542 trim_cache size 0 max 16384 2024-03-18T16:15:01.480 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:15:01.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.478+0000 7f574e000640 1 -- 172.21.15.39:0/2160740362 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f572c0a29d0 msgr2=0x7f572c0a4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:01.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.478+0000 7f574e000640 1 --2- 172.21.15.39:0/2160740362 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f572c0a29d0 0x7f572c0a4e90 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f5740097c20 tx=0x7f5740005f30 comp rx=0 tx=0).stop 2024-03-18T16:15:01.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.478+0000 7f574e000640 1 -- 172.21.15.39:0/2160740362 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f57481013d0 msgr2=0x7f57481a1320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:01.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.478+0000 7f574e000640 1 --2- 172.21.15.39:0/2160740362 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f57481013d0 0x7f57481a1320 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f574000ec50 tx=0x7f574000c6a0 comp rx=0 tx=0).stop 2024-03-18T16:15:01.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.478+0000 7f574e000640 1 -- 172.21.15.39:0/2160740362 shutdown_connections 2024-03-18T16:15:01.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.478+0000 7f574e000640 1 --2- 172.21.15.39:0/2160740362 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f57481013d0 0x7f57481a1320 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:01.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.478+0000 7f574e000640 1 --2- 172.21.15.39:0/2160740362 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f572c0a29d0 0x7f572c0a4e90 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:01.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.478+0000 7f574e000640 1 --2- 172.21.15.39:0/2160740362 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5748100a30 0x7f574819a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:01.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.478+0000 7f574e000640 1 --2- 172.21.15.39:0/2160740362 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f57481000d0 0x7f5748199d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:01.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.478+0000 7f574e000640 1 -- 172.21.15.39:0/2160740362 >> 172.21.15.39:0/2160740362 conn(0x7f57480fbc50 msgr2=0x7f57480fd3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:01.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.480+0000 7f574e000640 1 -- 172.21.15.39:0/2160740362 shutdown_connections 2024-03-18T16:15:01.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.480+0000 7f574e000640 1 -- 172.21.15.39:0/2160740362 wait complete. 2024-03-18T16:15:01.482 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:01 smithi196 ceph-mon[36307]: pgmap v570: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 32 MiB/s wr, 6.28k op/s 2024-03-18T16:15:01.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:01 smithi194 ceph-mon[36200]: pgmap v570: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 32 MiB/s wr, 6.28k op/s 2024-03-18T16:15:01.511 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-18T16:15:01.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.763+0000 7f5c05a00640 1 -- 172.21.15.39:0/2494130748 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5c001015e0 msgr2=0x7f5c00101a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:01.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.763+0000 7f5c05a00640 1 --2- 172.21.15.39:0/2494130748 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5c001015e0 0x7f5c00101a00 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f5bf0009900 tx=0x7f5bf002bc30 comp rx=0 tx=0).stop 2024-03-18T16:15:01.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.763+0000 7f5c05a00640 1 -- 172.21.15.39:0/2494130748 shutdown_connections 2024-03-18T16:15:01.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.763+0000 7f5c05a00640 1 --2- 172.21.15.39:0/2494130748 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5c00103190 0x7f5c00109a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:01.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.763+0000 7f5c05a00640 1 --2- 172.21.15.39:0/2494130748 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5c001027d0 0x7f5c00102c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:01.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.763+0000 7f5c05a00640 1 --2- 172.21.15.39:0/2494130748 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5c001015e0 0x7f5c00101a00 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:01.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.763+0000 7f5c05a00640 1 -- 172.21.15.39:0/2494130748 >> 172.21.15.39:0/2494130748 conn(0x7f5c000fcb70 msgr2=0x7f5c000fef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:01.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.763+0000 7f5c05a00640 1 -- 172.21.15.39:0/2494130748 shutdown_connections 2024-03-18T16:15:01.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.763+0000 7f5c05a00640 1 -- 172.21.15.39:0/2494130748 wait complete. 2024-03-18T16:15:01.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.764+0000 7f5c05a00640 1 Processor -- start 2024-03-18T16:15:01.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.764+0000 7f5c05a00640 1 -- start start 2024-03-18T16:15:01.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.765+0000 7f5c05a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5c001015e0 0x7f5c00199b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:01.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.765+0000 7f5c05a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5c001027d0 0x7f5c0019a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:01.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.765+0000 7f5c05a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5c00103190 0x7f5c001a10b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:01.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.765+0000 7f5c05a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f5c0010d2a0 con 0x7f5c001027d0 2024-03-18T16:15:01.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.765+0000 7f5c05a00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f5c0010d120 con 0x7f5c00103190 2024-03-18T16:15:01.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.765+0000 7f5c05a00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f5c0010d420 con 0x7f5c001015e0 2024-03-18T16:15:01.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.765+0000 7f5bffe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5c001015e0 0x7f5c00199b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:01.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.765+0000 7f5bffe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5c001015e0 0x7f5c00199b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:57612/0 (socket says 172.21.15.39:57612) 2024-03-18T16:15:01.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.765+0000 7f5bffe00640 1 -- 172.21.15.39:0/1896283916 learned_addr learned my addr 172.21.15.39:0/1896283916 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:15:01.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.765+0000 7f5bff400640 1 --2- 172.21.15.39:0/1896283916 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5c001027d0 0x7f5c0019a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:01.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.765+0000 7f5c05000640 1 --2- 172.21.15.39:0/1896283916 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5c00103190 0x7f5c001a10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:01.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.765+0000 7f5bffe00640 1 -- 172.21.15.39:0/1896283916 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5c00103190 msgr2=0x7f5c001a10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:01.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.765+0000 7f5bffe00640 1 --2- 172.21.15.39:0/1896283916 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5c00103190 0x7f5c001a10b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:01.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.765+0000 7f5bffe00640 1 -- 172.21.15.39:0/1896283916 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5c001027d0 msgr2=0x7f5c0019a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:01.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.765+0000 7f5bffe00640 1 --2- 172.21.15.39:0/1896283916 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5c001027d0 0x7f5c0019a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:01.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.765+0000 7f5bffe00640 1 -- 172.21.15.39:0/1896283916 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c001a1720 con 0x7f5c001015e0 2024-03-18T16:15:01.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.765+0000 7f5bff400640 1 --2- 172.21.15.39:0/1896283916 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5c001027d0 0x7f5c0019a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:15:01.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.766+0000 7f5bffe00640 1 --2- 172.21.15.39:0/1896283916 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5c001015e0 0x7f5c00199b00 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f5bf0000c00 tx=0x7f5bf002c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:01.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.766+0000 7f5bfcc00640 1 -- 172.21.15.39:0/1896283916 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bf0043020 con 0x7f5c001015e0 2024-03-18T16:15:01.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.766+0000 7f5bfcc00640 1 -- 172.21.15.39:0/1896283916 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5bf00054e0 con 0x7f5c001015e0 2024-03-18T16:15:01.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.766+0000 7f5c05a00640 1 -- 172.21.15.39:0/1896283916 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c001a19b0 con 0x7f5c001015e0 2024-03-18T16:15:01.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.766+0000 7f5bfcc00640 1 -- 172.21.15.39:0/1896283916 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bf0003c40 con 0x7f5c001015e0 2024-03-18T16:15:01.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.766+0000 7f5c05a00640 1 -- 172.21.15.39:0/1896283916 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c001a1f00 con 0x7f5c001015e0 2024-03-18T16:15:01.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.769+0000 7f5bfcc00640 1 -- 172.21.15.39:0/1896283916 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f5bf0046020 con 0x7f5c001015e0 2024-03-18T16:15:01.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.769+0000 7f5c09d91740 1 Processor -- start 2024-03-18T16:15:01.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.769+0000 7f5c09d91740 1 -- start start 2024-03-18T16:15:01.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.769+0000 7f5c09d91740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5564a1eada90 0x5564a1eadeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:01.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.769+0000 7f5c09d91740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5564a1eae480 0x5564a1eb2f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:01.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.770+0000 7f5c09d91740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5564a1eb35f0 0x5564a1eb5a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:01.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.770+0000 7f5c09d91740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5564a1e753b0 con 0x5564a1eae480 2024-03-18T16:15:01.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.770+0000 7f5c09d91740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5564a1e12450 con 0x5564a1eb35f0 2024-03-18T16:15:01.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.770+0000 7f5c09d91740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x5564a1dbacb0 con 0x5564a1eada90 2024-03-18T16:15:01.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.770+0000 7f5c05000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5564a1eada90 0x5564a1eadeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:01.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.770+0000 7f5c05000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5564a1eada90 0x5564a1eadeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:57622/0 (socket says 172.21.15.39:57622) 2024-03-18T16:15:01.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.770+0000 7f5c05000640 1 -- 172.21.15.39:0/1236598971 learned_addr learned my addr 172.21.15.39:0/1236598971 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:15:01.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.770+0000 7f5bffe00640 1 --2- 172.21.15.39:0/1236598971 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5564a1eae480 0x5564a1eb2f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:01.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.770+0000 7f5bff400640 1 --2- 172.21.15.39:0/1236598971 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5564a1eb35f0 0x5564a1eb5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:01.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.770+0000 7f5c05000640 1 -- 172.21.15.39:0/1236598971 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5564a1eb35f0 msgr2=0x5564a1eb5a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:01.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.770+0000 7f5c05000640 1 --2- 172.21.15.39:0/1236598971 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5564a1eb35f0 0x5564a1eb5a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:01.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.770+0000 7f5c05000640 1 -- 172.21.15.39:0/1236598971 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5564a1eae480 msgr2=0x5564a1eb2f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:01.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.770+0000 7f5c05000640 1 --2- 172.21.15.39:0/1236598971 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5564a1eae480 0x5564a1eb2f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:01.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.770+0000 7f5c05000640 1 -- 172.21.15.39:0/1236598971 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5564a1ddc1f0 con 0x5564a1eada90 2024-03-18T16:15:01.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.770+0000 7f5bffe00640 1 --2- 172.21.15.39:0/1236598971 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5564a1eae480 0x5564a1eb2f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:15:01.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.770+0000 7f5c05000640 1 --2- 172.21.15.39:0/1236598971 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5564a1eada90 0x5564a1eadeb0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f5bf8009fd0 tx=0x7f5bf80118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=caace5e78fa34bcd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:01.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.771+0000 7f5bf5600640 1 -- 172.21.15.39:0/1236598971 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bf8004d20 con 0x5564a1eada90 2024-03-18T16:15:01.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.771+0000 7f5bf5600640 1 -- 172.21.15.39:0/1236598971 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5bf8004050 con 0x5564a1eada90 2024-03-18T16:15:01.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.771+0000 7f5bf5600640 1 -- 172.21.15.39:0/1236598971 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bf8005260 con 0x5564a1eada90 2024-03-18T16:15:01.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.771+0000 7f5c09d91740 1 -- 172.21.15.39:0/1236598971 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5564a1eada90 msgr2=0x5564a1eadeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:01.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.771+0000 7f5c09d91740 1 --2- 172.21.15.39:0/1236598971 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5564a1eada90 0x5564a1eadeb0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f5bf8009fd0 tx=0x7f5bf80118d0 comp rx=0 tx=0).stop 2024-03-18T16:15:01.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.771+0000 7f5c09d91740 1 -- 172.21.15.39:0/1236598971 shutdown_connections 2024-03-18T16:15:01.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.771+0000 7f5c09d91740 1 --2- 172.21.15.39:0/1236598971 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5564a1eb35f0 0x5564a1eb5a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:01.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.771+0000 7f5c09d91740 1 --2- 172.21.15.39:0/1236598971 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5564a1eae480 0x5564a1eb2f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:01.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.771+0000 7f5c09d91740 1 --2- 172.21.15.39:0/1236598971 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5564a1eada90 0x5564a1eadeb0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:01.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.771+0000 7f5c09d91740 1 -- 172.21.15.39:0/1236598971 >> 172.21.15.39:0/1236598971 conn(0x5564a1ea9000 msgr2=0x5564a1ea9400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:01.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.771+0000 7f5c09d91740 1 -- 172.21.15.39:0/1236598971 shutdown_connections 2024-03-18T16:15:01.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.771+0000 7f5c09d91740 1 -- 172.21.15.39:0/1236598971 wait complete. 2024-03-18T16:15:01.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.771+0000 7f5bfcc00640 1 --2- 172.21.15.39:0/1896283916 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f5be40a2710 0x7f5be40a4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:01.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.772+0000 7f5bfcc00640 1 -- 172.21.15.39:0/1896283916 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f5bf00f0ea0 con 0x7f5c001015e0 2024-03-18T16:15:01.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.772+0000 7f5bff400640 1 --2- 172.21.15.39:0/1896283916 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f5be40a2710 0x7f5be40a4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:01.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.772+0000 7f5c09d91740 1 -- start start 2024-03-18T16:15:01.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.772+0000 7f5c09d91740 1 Processor -- start 2024-03-18T16:15:01.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.773+0000 7f5bff400640 1 --2- 172.21.15.39:0/1896283916 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f5be40a2710 0x7f5be40a4bd0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x5564a1ea9f70 tx=0x7f5bec005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:01.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.773+0000 7f5c09d91740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5564a1eada90 0x5564a1eb7d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:01.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.773+0000 7f5c09d91740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5564a1eae480 0x5564a1eb82a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:01.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.773+0000 7f5c09d91740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5564a1eb35f0 0x5564a1ebf2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:01.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.773+0000 7f5c09d91740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5564a1dce110 con 0x5564a1eb35f0 2024-03-18T16:15:01.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.773+0000 7f5c09d91740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5564a1e0e4f0 con 0x5564a1eae480 2024-03-18T16:15:01.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.773+0000 7f5c09d91740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x5564a1eb65a0 con 0x5564a1eada90 2024-03-18T16:15:01.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.773+0000 7f5c09d91740 10 client.0 _ll_register_callbacks cb 0x5564a1dbbb70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:15:01.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.773+0000 7f5bffe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5564a1eada90 0x5564a1eb7d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:01.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.773+0000 7f5bff400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5564a1eae480 0x5564a1eb82a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:01.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.773+0000 7f5bffe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5564a1eada90 0x5564a1eb7d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:57632/0 (socket says 172.21.15.39:57632) 2024-03-18T16:15:01.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.773+0000 7f5bffe00640 1 -- 172.21.15.39:0/1329859951 learned_addr learned my addr 172.21.15.39:0/1329859951 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:15:01.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.773+0000 7f5c05000640 1 --2- 172.21.15.39:0/1329859951 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5564a1eb35f0 0x5564a1ebf2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:01.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.773+0000 7f5bffe00640 1 -- 172.21.15.39:0/1329859951 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5564a1eae480 msgr2=0x5564a1eb82a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:01.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.773+0000 7f5bffe00640 1 --2- 172.21.15.39:0/1329859951 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5564a1eae480 0x5564a1eb82a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:01.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.773+0000 7f5bffe00640 1 -- 172.21.15.39:0/1329859951 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5564a1eb35f0 msgr2=0x5564a1ebf2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:01.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.773+0000 7f5bffe00640 1 --2- 172.21.15.39:0/1329859951 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5564a1eb35f0 0x5564a1ebf2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:01.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.773+0000 7f5bffe00640 1 -- 172.21.15.39:0/1329859951 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5564a1b263f0 con 0x5564a1eada90 2024-03-18T16:15:01.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.774+0000 7f5bffe00640 1 --2- 172.21.15.39:0/1329859951 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5564a1eada90 0x5564a1eb7d60 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f5bf00b98b0 tx=0x7f5bf00b97b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e5a50455d301b8f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:01.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.774+0000 7f5bea000640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:15:01.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.774+0000 7f5bea000640 1 -- 172.21.15.39:0/1329859951 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bf0004250 con 0x5564a1eada90 2024-03-18T16:15:01.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.774+0000 7f5bea000640 1 -- 172.21.15.39:0/1329859951 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f5bf00ee6c0 con 0x5564a1eada90 2024-03-18T16:15:01.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.774+0000 7f5c09d91740 1 -- 172.21.15.39:0/1329859951 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5bf8004050 con 0x5564a1eada90 2024-03-18T16:15:01.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.774+0000 7f5bea000640 1 -- 172.21.15.39:0/1329859951 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f5bf00035f0 con 0x5564a1eada90 2024-03-18T16:15:01.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.774+0000 7f5c09d91740 10 client.26554 fetch_fsmap learned FSMap version 40 2024-03-18T16:15:01.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.774+0000 7f5c09d91740 1 -- 172.21.15.39:0/1329859951 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x5564a1bf07d0 con 0x5564a1eada90 2024-03-18T16:15:01.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.774+0000 7f5bea000640 1 -- 172.21.15.39:0/1329859951 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5bf8004050 con 0x5564a1eada90 2024-03-18T16:15:01.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.775+0000 7f5bea000640 1 -- 172.21.15.39:0/1329859951 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 40) ==== 8378+0+0 (secure 0 0 0) 0x7f5bf00c1160 con 0x5564a1eada90 2024-03-18T16:15:01.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.775+0000 7f5c09d91740 10 client.26554 fetch_fsmap finished waiting for FSMap version 40 2024-03-18T16:15:01.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.775+0000 7f5c09d91740 10 client.26554 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-18T16:15:01.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.775+0000 7f5c09d91740 20 client.26554 populate_metadata read hostname 'smithi039' 2024-03-18T16:15:01.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.775+0000 7f5c09d91740 1 --2- 172.21.15.39:0/1329859951 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x5564a1ec3af0 0x5564a1ec5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:01.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.775+0000 7f5c09d91740 4 client.26554 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:15:01.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.775+0000 7f5c09d91740 1 -- 172.21.15.39:0/1329859951 --> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5564a1e1f990 con 0x5564a1ec3af0 2024-03-18T16:15:01.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.775+0000 7f5bff400640 1 --2- 172.21.15.39:0/1329859951 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x5564a1ec3af0 0x5564a1ec5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:01.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.776+0000 7f5bff400640 1 --2- 172.21.15.39:0/1329859951 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x5564a1ec3af0 0x5564a1ec5fb0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=cf461afaca15a2a8 server_cookie=93e7cd8ddf33784e in_seq=0 out_seq=0 2024-03-18T16:15:01.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.776+0000 7f5bea000640 10 client.26554 ms_handle_connect on v2:172.21.15.194:6836/1301464246 2024-03-18T16:15:01.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.777+0000 7f5bea000640 1 -- 172.21.15.39:0/1329859951 <== mds.3 v2:172.21.15.194:6836/1301464246 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5564a1e1f990 con 0x5564a1ec3af0 2024-03-18T16:15:01.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.777+0000 7f5bea000640 10 client.26554 handle_command_reply: tid=0 2024-03-18T16:15:01.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.777+0000 7f5c09d91740 1 client.26554 shutdown 2024-03-18T16:15:01.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.777+0000 7f5c09d91740 1 -- 172.21.15.39:0/1329859951 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5564a1eada90 msgr2=0x5564a1eb7d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:01.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.777+0000 7f5c09d91740 1 --2- 172.21.15.39:0/1329859951 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5564a1eada90 0x5564a1eb7d60 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f5bf00b98b0 tx=0x7f5bf00b97b0 comp rx=0 tx=0).stop 2024-03-18T16:15:01.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.777+0000 7f5c09d91740 1 -- 172.21.15.39:0/1329859951 shutdown_connections 2024-03-18T16:15:01.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.777+0000 7f5c09d91740 1 --2- 172.21.15.39:0/1329859951 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x5564a1ec3af0 0x5564a1ec5fb0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:01.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.777+0000 7f5c09d91740 1 --2- 172.21.15.39:0/1329859951 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5564a1eae480 0x5564a1eb82a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:01.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.777+0000 7f5c09d91740 1 --2- 172.21.15.39:0/1329859951 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5564a1eb35f0 0x5564a1ebf2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:01.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.777+0000 7f5c09d91740 1 --2- 172.21.15.39:0/1329859951 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5564a1eada90 0x5564a1eb7d60 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:01.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.777+0000 7f5c09d91740 1 -- 172.21.15.39:0/1329859951 >> 172.21.15.39:0/1329859951 conn(0x5564a1ea9000 msgr2=0x5564a1ea8220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:01.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.777+0000 7f5bea000640 0 client.26554 ms_handle_reset on v2:172.21.15.194:6836/1301464246 2024-03-18T16:15:01.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.777+0000 7f5c09d91740 1 -- 172.21.15.39:0/1329859951 shutdown_connections 2024-03-18T16:15:01.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.777+0000 7f5c09d91740 1 -- 172.21.15.39:0/1329859951 wait complete. 2024-03-18T16:15:01.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.777+0000 7f5c09d91740 20 client.26554 trim_cache size 0 max 16384 2024-03-18T16:15:01.800 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.799+0000 7f5c09d91740 1 Processor -- start 2024-03-18T16:15:01.800 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.799+0000 7f5c09d91740 1 -- start start 2024-03-18T16:15:01.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.799+0000 7f5c09d91740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5564a1eada90 0x5564a1ea7990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:01.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.799+0000 7f5c09d91740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5564a1eae480 0x5564a1ebf260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:01.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.799+0000 7f5c09d91740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5564a1eb35f0 0x5564a1ea7ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:01.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.799+0000 7f5c09d91740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f5c001ac4a0 con 0x5564a1eada90 2024-03-18T16:15:01.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.799+0000 7f5c09d91740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f5c001ae010 con 0x5564a1eb35f0 2024-03-18T16:15:01.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.799+0000 7f5c09d91740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f5c001acb70 con 0x5564a1eae480 2024-03-18T16:15:01.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.799+0000 7f5bffe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5564a1eada90 0x5564a1ea7990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:01.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.799+0000 7f5bff400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5564a1eae480 0x5564a1ebf260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:01.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.799+0000 7f5bffe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5564a1eada90 0x5564a1ea7990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:36424/0 (socket says 172.21.15.39:36424) 2024-03-18T16:15:01.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.799+0000 7f5bffe00640 1 -- 172.21.15.39:0/1235912360 learned_addr learned my addr 172.21.15.39:0/1235912360 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:15:01.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.799+0000 7f5c05000640 1 --2- 172.21.15.39:0/1235912360 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5564a1eb35f0 0x5564a1ea7ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:01.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.800+0000 7f5bff400640 1 -- 172.21.15.39:0/1235912360 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5564a1eb35f0 msgr2=0x5564a1ea7ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:01.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.800+0000 7f5bff400640 1 --2- 172.21.15.39:0/1235912360 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5564a1eb35f0 0x5564a1ea7ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:01.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.800+0000 7f5bff400640 1 -- 172.21.15.39:0/1235912360 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5564a1eada90 msgr2=0x5564a1ea7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:01.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.800+0000 7f5bff400640 1 --2- 172.21.15.39:0/1235912360 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5564a1eada90 0x5564a1ea7990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:01.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.800+0000 7f5bff400640 1 -- 172.21.15.39:0/1235912360 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5564a1ec6630 con 0x5564a1eae480 2024-03-18T16:15:01.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.800+0000 7f5bff400640 1 --2- 172.21.15.39:0/1235912360 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5564a1eae480 0x5564a1ebf260 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x5564a1ec3ed0 tx=0x7f5bec022040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5237c7bccf56ce91 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:01.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.800+0000 7f5bf4c00640 1 -- 172.21.15.39:0/1235912360 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bec030070 con 0x5564a1eae480 2024-03-18T16:15:01.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.800+0000 7f5bf4c00640 1 -- 172.21.15.39:0/1235912360 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5bec02bb90 con 0x5564a1eae480 2024-03-18T16:15:01.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.800+0000 7f5bf4c00640 1 -- 172.21.15.39:0/1235912360 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bec003850 con 0x5564a1eae480 2024-03-18T16:15:01.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.801+0000 7f5c09d91740 1 -- 172.21.15.39:0/1235912360 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5564a1eae480 msgr2=0x5564a1ebf260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:01.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.801+0000 7f5c09d91740 1 --2- 172.21.15.39:0/1235912360 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5564a1eae480 0x5564a1ebf260 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x5564a1ec3ed0 tx=0x7f5bec022040 comp rx=0 tx=0).stop 2024-03-18T16:15:01.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.801+0000 7f5c09d91740 1 -- 172.21.15.39:0/1235912360 shutdown_connections 2024-03-18T16:15:01.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.801+0000 7f5c09d91740 1 --2- 172.21.15.39:0/1235912360 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5564a1eb35f0 0x5564a1ea7ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:01.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.801+0000 7f5c09d91740 1 --2- 172.21.15.39:0/1235912360 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5564a1eae480 0x5564a1ebf260 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:01.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.801+0000 7f5c09d91740 1 --2- 172.21.15.39:0/1235912360 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5564a1eada90 0x5564a1ea7990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:01.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.801+0000 7f5c09d91740 1 -- 172.21.15.39:0/1235912360 >> 172.21.15.39:0/1235912360 conn(0x5564a1ea9000 msgr2=0x5564a1eb7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:01.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.801+0000 7f5c09d91740 1 -- 172.21.15.39:0/1235912360 shutdown_connections 2024-03-18T16:15:01.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.801+0000 7f5c09d91740 1 -- 172.21.15.39:0/1235912360 wait complete. 2024-03-18T16:15:01.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.801+0000 7f5c09d91740 1 -- start start 2024-03-18T16:15:01.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.801+0000 7f5c09d91740 1 Processor -- start 2024-03-18T16:15:01.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.802+0000 7f5c09d91740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5564a1eada90 0x5564a1ebd020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:01.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.802+0000 7f5c09d91740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5564a1eae480 0x5564a1ebd560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:01.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.802+0000 7f5c09d91740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5564a1eb35f0 0x5564a1ebe520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:01.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.802+0000 7f5c09d91740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f5c001ad910 con 0x5564a1eb35f0 2024-03-18T16:15:01.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.802+0000 7f5c09d91740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f5c001ad240 con 0x5564a1eada90 2024-03-18T16:15:01.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.802+0000 7f5c09d91740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f5c001aa480 con 0x5564a1eae480 2024-03-18T16:15:01.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.802+0000 7f5c09d91740 10 client.0 _ll_register_callbacks cb 0x7f5bf00b90b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:15:01.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.802+0000 7f5bff400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5564a1eae480 0x5564a1ebd560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:01.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.802+0000 7f5bffe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5564a1eada90 0x5564a1ebd020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:01.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.802+0000 7f5c05000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5564a1eb35f0 0x5564a1ebe520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:01.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.802+0000 7f5bff400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5564a1eae480 0x5564a1ebd560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:57650/0 (socket says 172.21.15.39:57650) 2024-03-18T16:15:01.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.802+0000 7f5bff400640 1 -- 172.21.15.39:0/1537185837 learned_addr learned my addr 172.21.15.39:0/1537185837 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:15:01.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.802+0000 7f5bff400640 1 -- 172.21.15.39:0/1537185837 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5564a1eada90 msgr2=0x5564a1ebd020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:01.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.802+0000 7f5bff400640 1 --2- 172.21.15.39:0/1537185837 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5564a1eada90 0x5564a1ebd020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:01.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.802+0000 7f5bff400640 1 -- 172.21.15.39:0/1537185837 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5564a1eb35f0 msgr2=0x5564a1ebe520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:01.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.802+0000 7f5bff400640 1 --2- 172.21.15.39:0/1537185837 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5564a1eb35f0 0x5564a1ebe520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:01.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.802+0000 7f5bff400640 1 -- 172.21.15.39:0/1537185837 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5564a1ead6d0 con 0x5564a1eae480 2024-03-18T16:15:01.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.803+0000 7f5bff400640 1 --2- 172.21.15.39:0/1537185837 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5564a1eae480 0x5564a1ebd560 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f5bec019040 tx=0x7f5bec020fa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=88ea6337848afd1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:01.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.803+0000 7f5bea000640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:15:01.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.803+0000 7f5bea000640 1 -- 172.21.15.39:0/1537185837 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bec030050 con 0x5564a1eae480 2024-03-18T16:15:01.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.803+0000 7f5bea000640 1 -- 172.21.15.39:0/1537185837 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f5bec005040 con 0x5564a1eae480 2024-03-18T16:15:01.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.803+0000 7f5c09d91740 1 -- 172.21.15.39:0/1537185837 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5bec02bb90 con 0x5564a1eae480 2024-03-18T16:15:01.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.803+0000 7f5bea000640 1 -- 172.21.15.39:0/1537185837 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f5bec045050 con 0x5564a1eae480 2024-03-18T16:15:01.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.803+0000 7f5c09d91740 10 client.26560 fetch_fsmap learned FSMap version 40 2024-03-18T16:15:01.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.803+0000 7f5c09d91740 1 -- 172.21.15.39:0/1537185837 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x5564a1ea8b40 con 0x5564a1eae480 2024-03-18T16:15:01.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.803+0000 7f5bea000640 1 -- 172.21.15.39:0/1537185837 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5bec02bb90 con 0x5564a1eae480 2024-03-18T16:15:01.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.804+0000 7f5bea000640 1 -- 172.21.15.39:0/1537185837 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 40) ==== 8378+0+0 (secure 0 0 0) 0x7f5bec004c00 con 0x5564a1eae480 2024-03-18T16:15:01.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.804+0000 7f5c09d91740 10 client.26560 fetch_fsmap finished waiting for FSMap version 40 2024-03-18T16:15:01.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.804+0000 7f5c09d91740 10 client.26560 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-18T16:15:01.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.804+0000 7f5c09d91740 20 client.26560 populate_metadata read hostname 'smithi039' 2024-03-18T16:15:01.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.804+0000 7f5c09d91740 1 --2- 172.21.15.39:0/1537185837 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x5564a1f2abc0 0x5564a1f2cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:01.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.804+0000 7f5c09d91740 4 client.26560 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:15:01.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.804+0000 7f5c09d91740 1 -- 172.21.15.39:0/1537185837 --> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5564a1e19ce0 con 0x5564a1f2abc0 2024-03-18T16:15:01.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.804+0000 7f5bffe00640 1 --2- 172.21.15.39:0/1537185837 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x5564a1f2abc0 0x5564a1f2cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:01.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.804+0000 7f5bffe00640 1 --2- 172.21.15.39:0/1537185837 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x5564a1f2abc0 0x5564a1f2cfe0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=d8fb8616ee55f6b5 server_cookie=105f99a527422b24 in_seq=0 out_seq=0 2024-03-18T16:15:01.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.804+0000 7f5bea000640 10 client.26560 ms_handle_connect on v2:172.21.15.194:6836/1301464246 2024-03-18T16:15:01.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.805+0000 7f5bea000640 1 -- 172.21.15.39:0/1537185837 <== mds.3 v2:172.21.15.194:6836/1301464246 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5564a1e19ce0 con 0x5564a1f2abc0 2024-03-18T16:15:01.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.805+0000 7f5bea000640 10 client.26560 handle_command_reply: tid=0 2024-03-18T16:15:01.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.805+0000 7f5c09d91740 1 client.26560 shutdown 2024-03-18T16:15:01.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.805+0000 7f5c09d91740 1 -- 172.21.15.39:0/1537185837 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5564a1eae480 msgr2=0x5564a1ebd560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:01.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.805+0000 7f5c09d91740 1 --2- 172.21.15.39:0/1537185837 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5564a1eae480 0x5564a1ebd560 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f5bec019040 tx=0x7f5bec020fa0 comp rx=0 tx=0).stop 2024-03-18T16:15:01.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.805+0000 7f5c09d91740 1 -- 172.21.15.39:0/1537185837 shutdown_connections 2024-03-18T16:15:01.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.805+0000 7f5c09d91740 1 --2- 172.21.15.39:0/1537185837 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x5564a1f2abc0 0x5564a1f2cfe0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:01.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.805+0000 7f5c09d91740 1 --2- 172.21.15.39:0/1537185837 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5564a1eb35f0 0x5564a1ebe520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:01.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.805+0000 7f5c09d91740 1 --2- 172.21.15.39:0/1537185837 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5564a1eae480 0x5564a1ebd560 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:01.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.805+0000 7f5bea000640 0 client.26560 ms_handle_reset on v2:172.21.15.194:6836/1301464246 2024-03-18T16:15:01.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.805+0000 7f5c09d91740 1 --2- 172.21.15.39:0/1537185837 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5564a1eada90 0x5564a1ebd020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:01.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.805+0000 7f5c09d91740 1 -- 172.21.15.39:0/1537185837 >> 172.21.15.39:0/1537185837 conn(0x5564a1ea9000 msgr2=0x5564a1eac260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:01.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.805+0000 7f5c09d91740 1 -- 172.21.15.39:0/1537185837 shutdown_connections 2024-03-18T16:15:01.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.805+0000 7f5c09d91740 1 -- 172.21.15.39:0/1537185837 wait complete. 2024-03-18T16:15:01.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.805+0000 7f5c09d91740 20 client.26560 trim_cache size 0 max 16384 2024-03-18T16:15:01.811 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:15:01.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.806+0000 7f5c05a00640 1 -- 172.21.15.39:0/1896283916 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f5be40a2710 msgr2=0x7f5be40a4bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:01.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.806+0000 7f5c05a00640 1 --2- 172.21.15.39:0/1896283916 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f5be40a2710 0x7f5be40a4bd0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x5564a1ea9f70 tx=0x7f5bec005f90 comp rx=0 tx=0).stop 2024-03-18T16:15:01.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.806+0000 7f5c05a00640 1 -- 172.21.15.39:0/1896283916 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5c001015e0 msgr2=0x7f5c00199b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:01.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.806+0000 7f5c05a00640 1 --2- 172.21.15.39:0/1896283916 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5c001015e0 0x7f5c00199b00 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f5bf0000c00 tx=0x7f5bf002c650 comp rx=0 tx=0).stop 2024-03-18T16:15:01.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.806+0000 7f5c05a00640 1 -- 172.21.15.39:0/1896283916 shutdown_connections 2024-03-18T16:15:01.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.806+0000 7f5c05a00640 1 --2- 172.21.15.39:0/1896283916 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5c00103190 0x7f5c001a10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:01.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.806+0000 7f5c05a00640 1 --2- 172.21.15.39:0/1896283916 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f5be40a2710 0x7f5be40a4bd0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:01.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.806+0000 7f5c05a00640 1 --2- 172.21.15.39:0/1896283916 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5c001027d0 0x7f5c0019a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:01.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.806+0000 7f5c05a00640 1 --2- 172.21.15.39:0/1896283916 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5c001015e0 0x7f5c00199b00 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:01.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.806+0000 7f5c05a00640 1 -- 172.21.15.39:0/1896283916 >> 172.21.15.39:0/1896283916 conn(0x7f5c000fcb70 msgr2=0x7f5c00073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:01.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.806+0000 7f5c05a00640 1 -- 172.21.15.39:0/1896283916 shutdown_connections 2024-03-18T16:15:01.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:01.806+0000 7f5c05a00640 1 -- 172.21.15.39:0/1896283916 wait complete. 2024-03-18T16:15:01.824 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-18T16:15:02.089 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:02 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/1083750479' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:15:02.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.166+0000 7fc918a00640 1 -- 172.21.15.39:0/72889758 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc9141055c0 msgr2=0x7fc91410be50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:02.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.166+0000 7fc918a00640 1 --2- 172.21.15.39:0/72889758 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc9141055c0 0x7fc91410be50 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fc90c009ca0 tx=0x7fc90c02c100 comp rx=0 tx=0).stop 2024-03-18T16:15:02.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.167+0000 7fc918a00640 1 -- 172.21.15.39:0/72889758 shutdown_connections 2024-03-18T16:15:02.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.167+0000 7fc918a00640 1 --2- 172.21.15.39:0/72889758 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc914104c00 0x7fc914105080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:02.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.167+0000 7fc918a00640 1 --2- 172.21.15.39:0/72889758 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc9141055c0 0x7fc91410be50 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:02.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.167+0000 7fc918a00640 1 --2- 172.21.15.39:0/72889758 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc914103a10 0x7fc914103e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:02.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.167+0000 7fc918a00640 1 -- 172.21.15.39:0/72889758 >> 172.21.15.39:0/72889758 conn(0x7fc9140fefa0 msgr2=0x7fc9141013c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:02.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.168+0000 7fc918a00640 1 -- 172.21.15.39:0/72889758 shutdown_connections 2024-03-18T16:15:02.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.169+0000 7fc918a00640 1 -- 172.21.15.39:0/72889758 wait complete. 2024-03-18T16:15:02.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.170+0000 7fc918a00640 1 Processor -- start 2024-03-18T16:15:02.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.172+0000 7fc918a00640 1 -- start start 2024-03-18T16:15:02.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.172+0000 7fc918a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc914103a10 0x7fc914199c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:02.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.172+0000 7fc918a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc914104c00 0x7fc91419a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:02.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.172+0000 7fc918a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc9141055c0 0x7fc9141a1220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:02.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.172+0000 7fc918a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fc91410f670 con 0x7fc9141055c0 2024-03-18T16:15:02.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.172+0000 7fc918a00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fc91410f4f0 con 0x7fc914103a10 2024-03-18T16:15:02.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.172+0000 7fc918a00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fc91410f7f0 con 0x7fc914104c00 2024-03-18T16:15:02.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.173+0000 7fc911e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc914103a10 0x7fc914199c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:02.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.174+0000 7fc911e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc914103a10 0x7fc914199c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:50940/0 (socket says 172.21.15.39:50940) 2024-03-18T16:15:02.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.174+0000 7fc911e00640 1 -- 172.21.15.39:0/3269017255 learned_addr learned my addr 172.21.15.39:0/3269017255 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:15:02.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.174+0000 7fc912800640 1 --2- 172.21.15.39:0/3269017255 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc9141055c0 0x7fc9141a1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:02.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.175+0000 7fc911400640 1 --2- 172.21.15.39:0/3269017255 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc914104c00 0x7fc91419a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:02.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.175+0000 7fc911e00640 1 -- 172.21.15.39:0/3269017255 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc914104c00 msgr2=0x7fc91419a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:02.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.175+0000 7fc911e00640 1 --2- 172.21.15.39:0/3269017255 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc914104c00 0x7fc91419a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:02.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.175+0000 7fc911e00640 1 -- 172.21.15.39:0/3269017255 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc9141055c0 msgr2=0x7fc9141a1220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:02.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.175+0000 7fc911e00640 1 --2- 172.21.15.39:0/3269017255 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc9141055c0 0x7fc9141a1220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:02.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.175+0000 7fc911e00640 1 -- 172.21.15.39:0/3269017255 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9141a18c0 con 0x7fc914103a10 2024-03-18T16:15:02.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.175+0000 7fc911400640 1 --2- 172.21.15.39:0/3269017255 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc914104c00 0x7fc91419a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:15:02.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.175+0000 7fc912800640 1 --2- 172.21.15.39:0/3269017255 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc9141055c0 0x7fc9141a1220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:15:02.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.176+0000 7fc911e00640 1 --2- 172.21.15.39:0/3269017255 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc914103a10 0x7fc914199c80 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fc90400c970 tx=0x7fc90400ce30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:02.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.176+0000 7fc90aa00640 1 -- 172.21.15.39:0/3269017255 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc904004220 con 0x7fc914103a10 2024-03-18T16:15:02.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.176+0000 7fc918a00640 1 -- 172.21.15.39:0/3269017255 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc914075c80 con 0x7fc914103a10 2024-03-18T16:15:02.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.176+0000 7fc90aa00640 1 -- 172.21.15.39:0/3269017255 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc9040043c0 con 0x7fc914103a10 2024-03-18T16:15:02.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.176+0000 7fc90aa00640 1 -- 172.21.15.39:0/3269017255 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc90401d060 con 0x7fc914103a10 2024-03-18T16:15:02.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.179+0000 7fc918a00640 1 -- 172.21.15.39:0/3269017255 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc914076170 con 0x7fc914103a10 2024-03-18T16:15:02.185 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.184+0000 7fc91b76f740 1 Processor -- start 2024-03-18T16:15:02.186 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.184+0000 7fc91b76f740 1 -- start start 2024-03-18T16:15:02.186 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.184+0000 7fc91b76f740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5604df33e8c0 0x5604df33ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:02.186 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.184+0000 7fc91b76f740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5604df33f2b0 0x5604df343db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:02.186 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.184+0000 7fc91b76f740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5604df344420 0x5604df3468b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:02.186 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.185+0000 7fc91b76f740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5604df3063b0 con 0x5604df33f2b0 2024-03-18T16:15:02.186 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.185+0000 7fc91b76f740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5604df2a3450 con 0x5604df33e8c0 2024-03-18T16:15:02.186 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.185+0000 7fc91b76f740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x5604df24bcb0 con 0x5604df344420 2024-03-18T16:15:02.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.185+0000 7fc911400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5604df344420 0x5604df3468b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:02.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.185+0000 7fc911400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5604df344420 0x5604df3468b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:57674/0 (socket says 172.21.15.39:57674) 2024-03-18T16:15:02.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.185+0000 7fc911400640 1 -- 172.21.15.39:0/635415761 learned_addr learned my addr 172.21.15.39:0/635415761 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:15:02.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.185+0000 7fc912800640 1 --2- 172.21.15.39:0/635415761 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5604df33e8c0 0x5604df33ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:02.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.185+0000 7fc90aa00640 1 -- 172.21.15.39:0/3269017255 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fc9040079b0 con 0x7fc914103a10 2024-03-18T16:15:02.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.186+0000 7fc911400640 1 -- 172.21.15.39:0/635415761 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5604df33e8c0 msgr2=0x5604df33ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:02.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.186+0000 7fc911400640 1 --2- 172.21.15.39:0/635415761 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5604df33e8c0 0x5604df33ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:02.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.186+0000 7fc911400640 1 -- 172.21.15.39:0/635415761 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5604df33f2b0 msgr2=0x5604df343db0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:15:02.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.186+0000 7fc911400640 1 --2- 172.21.15.39:0/635415761 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5604df33f2b0 0x5604df343db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:02.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.186+0000 7fc911400640 1 -- 172.21.15.39:0/635415761 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5604df26d1f0 con 0x5604df344420 2024-03-18T16:15:02.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.186+0000 7fc911400640 1 --2- 172.21.15.39:0/635415761 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5604df344420 0x5604df3468b0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fc90000bd30 tx=0x7fc90000f730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d22386916c24c299 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:02.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.186+0000 7fc8ff000640 1 -- 172.21.15.39:0/635415761 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc900015070 con 0x5604df344420 2024-03-18T16:15:02.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.187+0000 7fc8ff000640 1 -- 172.21.15.39:0/635415761 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc90000be20 con 0x5604df344420 2024-03-18T16:15:02.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.187+0000 7fc8ff000640 1 -- 172.21.15.39:0/635415761 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9000107f0 con 0x5604df344420 2024-03-18T16:15:02.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.190+0000 7fc90aa00640 1 --2- 172.21.15.39:0/3269017255 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fc8f80a2a80 0x7fc8f80a4f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:02.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.190+0000 7fc90aa00640 1 -- 172.21.15.39:0/3269017255 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fc9040cde40 con 0x7fc914103a10 2024-03-18T16:15:02.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.191+0000 7fc912800640 1 --2- 172.21.15.39:0/3269017255 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fc8f80a2a80 0x7fc8f80a4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:02.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.191+0000 7fc91b76f740 1 -- 172.21.15.39:0/635415761 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5604df344420 msgr2=0x5604df3468b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:02.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.191+0000 7fc91b76f740 1 --2- 172.21.15.39:0/635415761 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5604df344420 0x5604df3468b0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fc90000bd30 tx=0x7fc90000f730 comp rx=0 tx=0).stop 2024-03-18T16:15:02.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.192+0000 7fc91b76f740 1 -- 172.21.15.39:0/635415761 shutdown_connections 2024-03-18T16:15:02.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.192+0000 7fc91b76f740 1 --2- 172.21.15.39:0/635415761 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5604df344420 0x5604df3468b0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:02.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.193+0000 7fc91b76f740 1 --2- 172.21.15.39:0/635415761 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5604df33f2b0 0x5604df343db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:02.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.193+0000 7fc91b76f740 1 --2- 172.21.15.39:0/635415761 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5604df33e8c0 0x5604df33ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:02.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.193+0000 7fc91b76f740 1 -- 172.21.15.39:0/635415761 >> 172.21.15.39:0/635415761 conn(0x5604df339e30 msgr2=0x5604df33a230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:02.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.193+0000 7fc912800640 1 --2- 172.21.15.39:0/3269017255 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fc8f80a2a80 0x7fc8f80a4f40 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fc90c02c640 tx=0x7fc90c042040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:02.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.194+0000 7fc91b76f740 1 -- 172.21.15.39:0/635415761 shutdown_connections 2024-03-18T16:15:02.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.195+0000 7fc91b76f740 1 -- 172.21.15.39:0/635415761 wait complete. 2024-03-18T16:15:02.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.195+0000 7fc91b76f740 1 -- start start 2024-03-18T16:15:02.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.195+0000 7fc91b76f740 1 Processor -- start 2024-03-18T16:15:02.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.199+0000 7fc91b76f740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5604df33e8c0 0x5604df348c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:02.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.199+0000 7fc91b76f740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5604df33f2b0 0x5604df349140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:02.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.199+0000 7fc91b76f740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5604df344420 0x5604df350180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:02.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.200+0000 7fc911400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5604df33f2b0 0x5604df349140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:02.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.200+0000 7fc911400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5604df33f2b0 0x5604df349140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:57676/0 (socket says 172.21.15.39:57676) 2024-03-18T16:15:02.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.201+0000 7fc911e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5604df33e8c0 0x5604df348c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:02.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.201+0000 7fc911e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5604df33e8c0 0x5604df348c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:36470/0 (socket says 172.21.15.39:36470) 2024-03-18T16:15:02.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.203+0000 7fc91b76f740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5604df25f110 con 0x5604df33e8c0 2024-03-18T16:15:02.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.203+0000 7fc91b76f740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5604df29f4f0 con 0x5604df344420 2024-03-18T16:15:02.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.203+0000 7fc91b76f740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x5604df3473d0 con 0x5604df33f2b0 2024-03-18T16:15:02.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.203+0000 7fc91b76f740 10 client.0 _ll_register_callbacks cb 0x5604df24cb70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:15:02.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.203+0000 7fc912800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5604df344420 0x5604df350180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:02.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.203+0000 7fc911400640 1 -- 172.21.15.39:0/709985905 learned_addr learned my addr 172.21.15.39:0/709985905 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:15:02.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.205+0000 7fc912800640 1 -- 172.21.15.39:0/709985905 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5604df33f2b0 msgr2=0x5604df349140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:02.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.205+0000 7fc912800640 1 --2- 172.21.15.39:0/709985905 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5604df33f2b0 0x5604df349140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:02.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.205+0000 7fc912800640 1 -- 172.21.15.39:0/709985905 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5604df33e8c0 msgr2=0x5604df348c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:02.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.205+0000 7fc912800640 1 --2- 172.21.15.39:0/709985905 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5604df33e8c0 0x5604df348c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:02.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.205+0000 7fc912800640 1 -- 172.21.15.39:0/709985905 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5604defb73f0 con 0x5604df344420 2024-03-18T16:15:02.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.210+0000 7fc912800640 1 --2- 172.21.15.39:0/709985905 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5604df344420 0x5604df350180 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fc90c037040 tx=0x7fc90c02c100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=68b0d63ac0bf900a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:02.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.210+0000 7fc8f3400640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:15:02.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.211+0000 7fc8f3400640 1 -- 172.21.15.39:0/709985905 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc90c05a060 con 0x5604df344420 2024-03-18T16:15:02.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.211+0000 7fc91b76f740 1 -- 172.21.15.39:0/709985905 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc90000be20 con 0x5604df344420 2024-03-18T16:15:02.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.211+0000 7fc8f3400640 1 -- 172.21.15.39:0/709985905 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fc90c0347f0 con 0x5604df344420 2024-03-18T16:15:02.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.212+0000 7fc8f3400640 1 -- 172.21.15.39:0/709985905 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fc90c061050 con 0x5604df344420 2024-03-18T16:15:02.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.212+0000 7fc91b76f740 10 client.26600 fetch_fsmap learned FSMap version 40 2024-03-18T16:15:02.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.212+0000 7fc91b76f740 1 -- 172.21.15.39:0/709985905 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x5604df0817d0 con 0x5604df344420 2024-03-18T16:15:02.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.213+0000 7fc8f3400640 1 -- 172.21.15.39:0/709985905 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc90000be20 con 0x5604df344420 2024-03-18T16:15:02.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.213+0000 7fc8f3400640 1 -- 172.21.15.39:0/709985905 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 40) ==== 8378+0+0 (secure 0 0 0) 0x7fc90c055bc0 con 0x5604df344420 2024-03-18T16:15:02.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.215+0000 7fc91b76f740 10 client.26600 fetch_fsmap finished waiting for FSMap version 40 2024-03-18T16:15:02.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.217+0000 7fc91b76f740 10 client.26600 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-18T16:15:02.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.217+0000 7fc91b76f740 20 client.26600 populate_metadata read hostname 'smithi039' 2024-03-18T16:15:02.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.217+0000 7fc91b76f740 1 --2- 172.21.15.39:0/709985905 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x5604df354a40 0x5604df356ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:02.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.217+0000 7fc91b76f740 4 client.26600 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:15:02.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.217+0000 7fc91b76f740 1 -- 172.21.15.39:0/709985905 --> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5604df2b0990 con 0x5604df354a40 2024-03-18T16:15:02.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.219+0000 7fc911e00640 1 --2- 172.21.15.39:0/709985905 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x5604df354a40 0x5604df356ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:02.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.223+0000 7fc911e00640 1 --2- 172.21.15.39:0/709985905 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x5604df354a40 0x5604df356ec0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=45f29644db05cba5 server_cookie=99e10a6aa364f25 in_seq=0 out_seq=0 2024-03-18T16:15:02.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.223+0000 7fc8f3400640 10 client.26600 ms_handle_connect on v2:172.21.15.196:6836/895247251 2024-03-18T16:15:02.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.226+0000 7fc8f3400640 1 -- 172.21.15.39:0/709985905 <== mds.2 v2:172.21.15.196:6836/895247251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5604df2b0990 con 0x5604df354a40 2024-03-18T16:15:02.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.226+0000 7fc8f3400640 10 client.26600 handle_command_reply: tid=0 2024-03-18T16:15:02.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.226+0000 7fc91b76f740 1 client.26600 shutdown 2024-03-18T16:15:02.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.235+0000 7fc91b76f740 1 -- 172.21.15.39:0/709985905 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5604df344420 msgr2=0x5604df350180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:02.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.235+0000 7fc91b76f740 1 --2- 172.21.15.39:0/709985905 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5604df344420 0x5604df350180 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fc90c037040 tx=0x7fc90c02c100 comp rx=0 tx=0).stop 2024-03-18T16:15:02.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.235+0000 7fc91b76f740 1 -- 172.21.15.39:0/709985905 shutdown_connections 2024-03-18T16:15:02.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.235+0000 7fc91b76f740 1 --2- 172.21.15.39:0/709985905 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5604df344420 0x5604df350180 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:02.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.235+0000 7fc91b76f740 1 --2- 172.21.15.39:0/709985905 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x5604df354a40 0x5604df356ec0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:02.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.235+0000 7fc91b76f740 1 --2- 172.21.15.39:0/709985905 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5604df33f2b0 0x5604df349140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:02.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.235+0000 7fc91b76f740 1 --2- 172.21.15.39:0/709985905 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5604df33e8c0 0x5604df348c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:02.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.235+0000 7fc91b76f740 1 -- 172.21.15.39:0/709985905 >> 172.21.15.39:0/709985905 conn(0x5604df339e30 msgr2=0x5604df3391f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:02.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.235+0000 7fc8f3400640 0 client.26600 ms_handle_reset on v2:172.21.15.196:6836/895247251 2024-03-18T16:15:02.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.235+0000 7fc91b76f740 1 -- 172.21.15.39:0/709985905 shutdown_connections 2024-03-18T16:15:02.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.236+0000 7fc91b76f740 1 -- 172.21.15.39:0/709985905 wait complete. 2024-03-18T16:15:02.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.236+0000 7fc91b76f740 20 client.26600 trim_cache size 0 max 16384 2024-03-18T16:15:02.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.289+0000 7fc91b76f740 1 Processor -- start 2024-03-18T16:15:02.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.290+0000 7fc91b76f740 1 -- start start 2024-03-18T16:15:02.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.290+0000 7fc91b76f740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5604df33e8c0 0x5604df340710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:02.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.290+0000 7fc91b76f740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5604df33f2b0 0x5604df350140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:02.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.290+0000 7fc91b76f740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5604df344420 0x5604df340c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:02.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.290+0000 7fc91b76f740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fc9141a8160 con 0x5604df344420 2024-03-18T16:15:02.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.290+0000 7fc91b76f740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fc9141a9cd0 con 0x5604df33f2b0 2024-03-18T16:15:02.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.290+0000 7fc91b76f740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fc9141a8830 con 0x5604df33e8c0 2024-03-18T16:15:02.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.291+0000 7fc911e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5604df33e8c0 0x5604df340710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:02.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.291+0000 7fc911e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5604df33e8c0 0x5604df340710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:57690/0 (socket says 172.21.15.39:57690) 2024-03-18T16:15:02.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.292+0000 7fc911e00640 1 -- 172.21.15.39:0/2506923921 learned_addr learned my addr 172.21.15.39:0/2506923921 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:15:02.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.291+0000 7fc911400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5604df33f2b0 0x5604df350140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:02.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.293+0000 7fc911e00640 1 -- 172.21.15.39:0/2506923921 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5604df33f2b0 msgr2=0x5604df350140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:02.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.293+0000 7fc911e00640 1 --2- 172.21.15.39:0/2506923921 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5604df33f2b0 0x5604df350140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:02.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.293+0000 7fc911e00640 1 -- 172.21.15.39:0/2506923921 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5604df344420 msgr2=0x5604df340c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:02.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.293+0000 7fc911e00640 1 --2- 172.21.15.39:0/2506923921 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5604df344420 0x5604df340c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:02.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.293+0000 7fc911e00640 1 -- 172.21.15.39:0/2506923921 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5604df335c10 con 0x5604df33e8c0 2024-03-18T16:15:02.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.295+0000 7fc911e00640 1 --2- 172.21.15.39:0/2506923921 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5604df33e8c0 0x5604df340710 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fc9040cb5b0 tx=0x7fc9040cb480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=44753fdec9949ea2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:02.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.296+0000 7fc8fe600640 1 -- 172.21.15.39:0/2506923921 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc90409a450 con 0x5604df33e8c0 2024-03-18T16:15:02.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.296+0000 7fc8fe600640 1 -- 172.21.15.39:0/2506923921 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc90409e5a0 con 0x5604df33e8c0 2024-03-18T16:15:02.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.296+0000 7fc8fe600640 1 -- 172.21.15.39:0/2506923921 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc904007ca0 con 0x5604df33e8c0 2024-03-18T16:15:02.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.298+0000 7fc91b76f740 1 -- 172.21.15.39:0/2506923921 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5604df33e8c0 msgr2=0x5604df340710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:02.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.298+0000 7fc91b76f740 1 --2- 172.21.15.39:0/2506923921 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5604df33e8c0 0x5604df340710 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fc9040cb5b0 tx=0x7fc9040cb480 comp rx=0 tx=0).stop 2024-03-18T16:15:02.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.298+0000 7fc91b76f740 1 -- 172.21.15.39:0/2506923921 shutdown_connections 2024-03-18T16:15:02.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.298+0000 7fc91b76f740 1 --2- 172.21.15.39:0/2506923921 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5604df33f2b0 0x5604df350140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:02.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.298+0000 7fc91b76f740 1 --2- 172.21.15.39:0/2506923921 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5604df344420 0x5604df340c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:02.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.298+0000 7fc91b76f740 1 --2- 172.21.15.39:0/2506923921 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5604df33e8c0 0x5604df340710 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:02.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.298+0000 7fc91b76f740 1 -- 172.21.15.39:0/2506923921 >> 172.21.15.39:0/2506923921 conn(0x5604df339e30 msgr2=0x5604df348c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:02.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.298+0000 7fc91b76f740 1 -- 172.21.15.39:0/2506923921 shutdown_connections 2024-03-18T16:15:02.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.298+0000 7fc91b76f740 1 -- 172.21.15.39:0/2506923921 wait complete. 2024-03-18T16:15:02.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.299+0000 7fc91b76f740 1 -- start start 2024-03-18T16:15:02.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.299+0000 7fc91b76f740 1 Processor -- start 2024-03-18T16:15:02.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.301+0000 7fc91b76f740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5604df33e8c0 0x5604df34b370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:02.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.301+0000 7fc91b76f740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5604df33f2b0 0x5604df34b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:02.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.301+0000 7fc91b76f740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5604df344420 0x5604df34e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:02.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.302+0000 7fc911e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5604df33e8c0 0x5604df34b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:02.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.302+0000 7fc911e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5604df33e8c0 0x5604df34b370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:57694/0 (socket says 172.21.15.39:57694) 2024-03-18T16:15:02.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.302+0000 7fc911400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5604df33f2b0 0x5604df34b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:02.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.302+0000 7fc911400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5604df33f2b0 0x5604df34b8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:50982/0 (socket says 172.21.15.39:50982) 2024-03-18T16:15:02.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.302+0000 7fc911400640 1 -- 172.21.15.39:0/1672514086 learned_addr learned my addr 172.21.15.39:0/1672514086 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:15:02.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.303+0000 7fc91b76f740 1 -- 172.21.15.39:0/1672514086 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fc9141a95d0 con 0x5604df344420 2024-03-18T16:15:02.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.303+0000 7fc91b76f740 1 -- 172.21.15.39:0/1672514086 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fc9141a8f00 con 0x5604df33f2b0 2024-03-18T16:15:02.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.303+0000 7fc91b76f740 1 -- 172.21.15.39:0/1672514086 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fc9141a6140 con 0x5604df33e8c0 2024-03-18T16:15:02.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.303+0000 7fc91b76f740 10 client.0 _ll_register_callbacks cb 0x7fc90c009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:15:02.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.303+0000 7fc912800640 1 --2- 172.21.15.39:0/1672514086 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5604df344420 0x5604df34e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:02.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.304+0000 7fc911400640 1 -- 172.21.15.39:0/1672514086 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5604df33e8c0 msgr2=0x5604df34b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:02.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.304+0000 7fc911400640 1 --2- 172.21.15.39:0/1672514086 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5604df33e8c0 0x5604df34b370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:02.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.304+0000 7fc911400640 1 -- 172.21.15.39:0/1672514086 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5604df344420 msgr2=0x5604df34e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:02.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.304+0000 7fc911400640 1 --2- 172.21.15.39:0/1672514086 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5604df344420 0x5604df34e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:02.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.304+0000 7fc911400640 1 -- 172.21.15.39:0/1672514086 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5604df34eec0 con 0x5604df33f2b0 2024-03-18T16:15:02.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.305+0000 7fc8f3400640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:15:02.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.305+0000 7fc911400640 1 --2- 172.21.15.39:0/1672514086 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5604df33f2b0 0x5604df34b8b0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fc900000c00 tx=0x7fc9000052c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=44633cd30cd0dabd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:02.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.306+0000 7fc8f3400640 1 -- 172.21.15.39:0/1672514086 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc900015040 con 0x5604df33f2b0 2024-03-18T16:15:02.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.306+0000 7fc8f3400640 1 -- 172.21.15.39:0/1672514086 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fc900004230 con 0x5604df33f2b0 2024-03-18T16:15:02.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.307+0000 7fc91b76f740 1 -- 172.21.15.39:0/1672514086 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc90409e5a0 con 0x5604df33f2b0 2024-03-18T16:15:02.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.309+0000 7fc8f3400640 1 -- 172.21.15.39:0/1672514086 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fc900004430 con 0x5604df33f2b0 2024-03-18T16:15:02.312 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.311+0000 7fc91b76f740 10 client.26606 fetch_fsmap learned FSMap version 40 2024-03-18T16:15:02.312 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.311+0000 7fc91b76f740 1 -- 172.21.15.39:0/1672514086 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x5604df3501d0 con 0x5604df33f2b0 2024-03-18T16:15:02.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.312+0000 7fc8f3400640 1 -- 172.21.15.39:0/1672514086 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc90409e5a0 con 0x5604df33f2b0 2024-03-18T16:15:02.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.313+0000 7fc8f3400640 1 -- 172.21.15.39:0/1672514086 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 40) ==== 8378+0+0 (secure 0 0 0) 0x7fc900003980 con 0x5604df33f2b0 2024-03-18T16:15:02.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.313+0000 7fc91b76f740 10 client.26606 fetch_fsmap finished waiting for FSMap version 40 2024-03-18T16:15:02.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.313+0000 7fc91b76f740 10 client.26606 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-18T16:15:02.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.313+0000 7fc91b76f740 20 client.26606 populate_metadata read hostname 'smithi039' 2024-03-18T16:15:02.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.314+0000 7fc91b76f740 1 --2- 172.21.15.39:0/1672514086 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x5604df3cbfd0 0x5604df3ce3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:02.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.314+0000 7fc91b76f740 4 client.26606 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:15:02.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.314+0000 7fc91b76f740 1 -- 172.21.15.39:0/1672514086 --> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5604df2aace0 con 0x5604df3cbfd0 2024-03-18T16:15:02.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.314+0000 7fc911e00640 1 --2- 172.21.15.39:0/1672514086 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x5604df3cbfd0 0x5604df3ce3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:02.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.315+0000 7fc911e00640 1 --2- 172.21.15.39:0/1672514086 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x5604df3cbfd0 0x5604df3ce3f0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=48e00b4436885f01 server_cookie=b1428aed4a56742b in_seq=0 out_seq=0 2024-03-18T16:15:02.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.315+0000 7fc8f3400640 10 client.26606 ms_handle_connect on v2:172.21.15.196:6836/895247251 2024-03-18T16:15:02.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.316+0000 7fc8f3400640 1 -- 172.21.15.39:0/1672514086 <== mds.2 v2:172.21.15.196:6836/895247251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5604df2aace0 con 0x5604df3cbfd0 2024-03-18T16:15:02.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.316+0000 7fc8f3400640 10 client.26606 handle_command_reply: tid=0 2024-03-18T16:15:02.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.316+0000 7fc91b76f740 1 client.26606 shutdown 2024-03-18T16:15:02.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.317+0000 7fc91b76f740 1 -- 172.21.15.39:0/1672514086 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5604df33f2b0 msgr2=0x5604df34b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:02.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.317+0000 7fc91b76f740 1 --2- 172.21.15.39:0/1672514086 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5604df33f2b0 0x5604df34b8b0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fc900000c00 tx=0x7fc9000052c0 comp rx=0 tx=0).stop 2024-03-18T16:15:02.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.317+0000 7fc91b76f740 1 -- 172.21.15.39:0/1672514086 shutdown_connections 2024-03-18T16:15:02.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.317+0000 7fc91b76f740 1 --2- 172.21.15.39:0/1672514086 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5604df33f2b0 0x5604df34b8b0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:02.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.317+0000 7fc91b76f740 1 --2- 172.21.15.39:0/1672514086 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x5604df3cbfd0 0x5604df3ce3f0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:02.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.317+0000 7fc91b76f740 1 --2- 172.21.15.39:0/1672514086 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5604df344420 0x5604df34e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:02.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.317+0000 7fc8f3400640 0 client.26606 ms_handle_reset on v2:172.21.15.196:6836/895247251 2024-03-18T16:15:02.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.317+0000 7fc91b76f740 1 --2- 172.21.15.39:0/1672514086 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5604df33e8c0 0x5604df34b370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:02.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.317+0000 7fc91b76f740 1 -- 172.21.15.39:0/1672514086 >> 172.21.15.39:0/1672514086 conn(0x5604df339e30 msgr2=0x5604df348b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:02.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.318+0000 7fc91b76f740 1 -- 172.21.15.39:0/1672514086 shutdown_connections 2024-03-18T16:15:02.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.318+0000 7fc91b76f740 1 -- 172.21.15.39:0/1672514086 wait complete. 2024-03-18T16:15:02.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.318+0000 7fc91b76f740 20 client.26606 trim_cache size 0 max 16384 2024-03-18T16:15:02.321 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:15:02.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.321+0000 7fc918a00640 1 -- 172.21.15.39:0/3269017255 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fc8f80a2a80 msgr2=0x7fc8f80a4f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:02.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.321+0000 7fc918a00640 1 --2- 172.21.15.39:0/3269017255 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fc8f80a2a80 0x7fc8f80a4f40 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fc90c02c640 tx=0x7fc90c042040 comp rx=0 tx=0).stop 2024-03-18T16:15:02.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.321+0000 7fc918a00640 1 -- 172.21.15.39:0/3269017255 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc914103a10 msgr2=0x7fc914199c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:02.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.321+0000 7fc918a00640 1 --2- 172.21.15.39:0/3269017255 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc914103a10 0x7fc914199c80 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fc90400c970 tx=0x7fc90400ce30 comp rx=0 tx=0).stop 2024-03-18T16:15:02.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.321+0000 7fc918a00640 1 -- 172.21.15.39:0/3269017255 shutdown_connections 2024-03-18T16:15:02.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.321+0000 7fc918a00640 1 --2- 172.21.15.39:0/3269017255 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fc8f80a2a80 0x7fc8f80a4f40 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:02.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.321+0000 7fc918a00640 1 --2- 172.21.15.39:0/3269017255 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc9141055c0 0x7fc9141a1220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:02.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.321+0000 7fc918a00640 1 --2- 172.21.15.39:0/3269017255 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc914104c00 0x7fc91419a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:02.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.321+0000 7fc918a00640 1 --2- 172.21.15.39:0/3269017255 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc914103a10 0x7fc914199c80 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:02.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.321+0000 7fc918a00640 1 -- 172.21.15.39:0/3269017255 >> 172.21.15.39:0/3269017255 conn(0x7fc9140fefa0 msgr2=0x7fc914107ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:02.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.322+0000 7fc918a00640 1 -- 172.21.15.39:0/3269017255 shutdown_connections 2024-03-18T16:15:02.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.323+0000 7fc918a00640 1 -- 172.21.15.39:0/3269017255 wait complete. 2024-03-18T16:15:02.354 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-18T16:15:02.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:02 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/1083750479' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:15:02.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:02 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/1083750479' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:15:02.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.648+0000 7fdfc3e00640 1 -- 172.21.15.39:0/87759051 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdfc4103190 msgr2=0x7fdfc4109a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:02.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.648+0000 7fdfc3e00640 1 --2- 172.21.15.39:0/87759051 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdfc4103190 0x7fdfc4109a20 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fdfbc009ca0 tx=0x7fdfbc02c0f0 comp rx=0 tx=0).stop 2024-03-18T16:15:02.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.648+0000 7fdfc3e00640 1 -- 172.21.15.39:0/87759051 shutdown_connections 2024-03-18T16:15:02.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.648+0000 7fdfc3e00640 1 --2- 172.21.15.39:0/87759051 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdfc4103190 0x7fdfc4109a20 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:02.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.648+0000 7fdfc3e00640 1 --2- 172.21.15.39:0/87759051 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdfc41027d0 0x7fdfc4102c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:02.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.648+0000 7fdfc3e00640 1 --2- 172.21.15.39:0/87759051 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdfc41015e0 0x7fdfc4101a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:02.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.648+0000 7fdfc3e00640 1 -- 172.21.15.39:0/87759051 >> 172.21.15.39:0/87759051 conn(0x7fdfc40fcb90 msgr2=0x7fdfc40fefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:02.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.648+0000 7fdfc3e00640 1 -- 172.21.15.39:0/87759051 shutdown_connections 2024-03-18T16:15:02.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.648+0000 7fdfc3e00640 1 -- 172.21.15.39:0/87759051 wait complete. 2024-03-18T16:15:02.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.649+0000 7fdfc3e00640 1 Processor -- start 2024-03-18T16:15:02.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.649+0000 7fdfc3e00640 1 -- start start 2024-03-18T16:15:02.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.650+0000 7fdfc3e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdfc41015e0 0x7fdfc4197870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:02.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.650+0000 7fdfc3e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdfc41027d0 0x7fdfc4197db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:02.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.650+0000 7fdfc3e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdfc4103190 0x7fdfc419ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:02.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.650+0000 7fdfc3e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fdfc410d260 con 0x7fdfc41015e0 2024-03-18T16:15:02.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.650+0000 7fdfc3e00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fdfc410d0e0 con 0x7fdfc41027d0 2024-03-18T16:15:02.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.650+0000 7fdfc3e00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fdfc410d3e0 con 0x7fdfc4103190 2024-03-18T16:15:02.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.650+0000 7fdfc2000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdfc41027d0 0x7fdfc4197db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:02.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.650+0000 7fdfc2a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdfc41015e0 0x7fdfc4197870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:02.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.650+0000 7fdfc2a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdfc41015e0 0x7fdfc4197870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:36500/0 (socket says 172.21.15.39:36500) 2024-03-18T16:15:02.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.650+0000 7fdfc2a00640 1 -- 172.21.15.39:0/3114564541 learned_addr learned my addr 172.21.15.39:0/3114564541 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:15:02.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.650+0000 7fdfc2000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdfc41027d0 0x7fdfc4197db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:51002/0 (socket says 172.21.15.39:51002) 2024-03-18T16:15:02.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.650+0000 7fdfc3400640 1 --2- 172.21.15.39:0/3114564541 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdfc4103190 0x7fdfc419ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:02.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.650+0000 7fdfc2a00640 1 -- 172.21.15.39:0/3114564541 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdfc4103190 msgr2=0x7fdfc419ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:02.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.650+0000 7fdfc2a00640 1 --2- 172.21.15.39:0/3114564541 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdfc4103190 0x7fdfc419ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:02.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.650+0000 7fdfc2a00640 1 -- 172.21.15.39:0/3114564541 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdfc41027d0 msgr2=0x7fdfc4197db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:02.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.650+0000 7fdfc2a00640 1 --2- 172.21.15.39:0/3114564541 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdfc41027d0 0x7fdfc4197db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:02.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.650+0000 7fdfc2a00640 1 -- 172.21.15.39:0/3114564541 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfc419f490 con 0x7fdfc41015e0 2024-03-18T16:15:02.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.650+0000 7fdfc3400640 1 --2- 172.21.15.39:0/3114564541 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdfc4103190 0x7fdfc419ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:15:02.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.651+0000 7fdfc2a00640 1 --2- 172.21.15.39:0/3114564541 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdfc41015e0 0x7fdfc4197870 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fdfb400e9c0 tx=0x7fdfb400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:02.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.651+0000 7fdfbb400640 1 -- 172.21.15.39:0/3114564541 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfb400cd80 con 0x7fdfc41015e0 2024-03-18T16:15:02.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.651+0000 7fdfbb400640 1 -- 172.21.15.39:0/3114564541 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdfb40035f0 con 0x7fdfc41015e0 2024-03-18T16:15:02.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.651+0000 7fdfc3e00640 1 -- 172.21.15.39:0/3114564541 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfc419f780 con 0x7fdfc41015e0 2024-03-18T16:15:02.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.651+0000 7fdfbb400640 1 -- 172.21.15.39:0/3114564541 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfb4010690 con 0x7fdfc41015e0 2024-03-18T16:15:02.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.651+0000 7fdfc3e00640 1 -- 172.21.15.39:0/3114564541 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfc419fcd0 con 0x7fdfc41015e0 2024-03-18T16:15:02.656 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.654+0000 7fdfcc1f6740 1 Processor -- start 2024-03-18T16:15:02.656 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.654+0000 7fdfbb400640 1 -- 172.21.15.39:0/3114564541 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fdfb40037a0 con 0x7fdfc41015e0 2024-03-18T16:15:02.656 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.654+0000 7fdfcc1f6740 1 -- start start 2024-03-18T16:15:02.656 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.654+0000 7fdfcc1f6740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557103c01a90 0x557103c01eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:02.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.654+0000 7fdfcc1f6740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557103c02480 0x557103c06f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:02.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.654+0000 7fdfcc1f6740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557103c075f0 0x557103c09a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:02.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.654+0000 7fdfcc1f6740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x557103bc93b0 con 0x557103c01a90 2024-03-18T16:15:02.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.654+0000 7fdfcc1f6740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x557103b66450 con 0x557103c075f0 2024-03-18T16:15:02.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.654+0000 7fdfcc1f6740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x557103b0ecb0 con 0x557103c02480 2024-03-18T16:15:02.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.655+0000 7fdfc3400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557103c01a90 0x557103c01eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:02.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.655+0000 7fdfc2a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557103c02480 0x557103c06f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:02.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.655+0000 7fdfc2000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557103c075f0 0x557103c09a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:02.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.655+0000 7fdfc3400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557103c01a90 0x557103c01eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:36504/0 (socket says 172.21.15.39:36504) 2024-03-18T16:15:02.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.655+0000 7fdfc2000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557103c075f0 0x557103c09a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:51014/0 (socket says 172.21.15.39:51014) 2024-03-18T16:15:02.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.655+0000 7fdfc3400640 1 -- 172.21.15.39:0/2522331271 learned_addr learned my addr 172.21.15.39:0/2522331271 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:15:02.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.655+0000 7fdfc3400640 1 -- 172.21.15.39:0/2522331271 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557103c02480 msgr2=0x557103c06f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:02.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.655+0000 7fdfc3400640 1 --2- 172.21.15.39:0/2522331271 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557103c02480 0x557103c06f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:02.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.655+0000 7fdfc3400640 1 -- 172.21.15.39:0/2522331271 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557103c075f0 msgr2=0x557103c09a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:02.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.655+0000 7fdfc3400640 1 --2- 172.21.15.39:0/2522331271 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557103c075f0 0x557103c09a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:02.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.655+0000 7fdfc3400640 1 -- 172.21.15.39:0/2522331271 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557103b301f0 con 0x557103c01a90 2024-03-18T16:15:02.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.655+0000 7fdfc2000640 1 --2- 172.21.15.39:0/2522331271 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557103c075f0 0x557103c09a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:15:02.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.656+0000 7fdfc3400640 1 --2- 172.21.15.39:0/2522331271 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557103c01a90 0x557103c01eb0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fdfbc02c630 tx=0x7fdfbc0323e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=822112b38a239369 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:02.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.656+0000 7fdfaf400640 1 -- 172.21.15.39:0/2522331271 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfbc039050 con 0x557103c01a90 2024-03-18T16:15:02.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.656+0000 7fdfbb400640 1 --2- 172.21.15.39:0/3114564541 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fdfa80a2800 0x7fdfa80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:02.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.656+0000 7fdfaf400640 1 -- 172.21.15.39:0/2522331271 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdfbc045070 con 0x557103c01a90 2024-03-18T16:15:02.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.656+0000 7fdfaf400640 1 -- 172.21.15.39:0/2522331271 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfbc0042a0 con 0x557103c01a90 2024-03-18T16:15:02.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.656+0000 7fdfbb400640 1 -- 172.21.15.39:0/3114564541 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fdfb40cf250 con 0x7fdfc41015e0 2024-03-18T16:15:02.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.656+0000 7fdfc2a00640 1 --2- 172.21.15.39:0/3114564541 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fdfa80a2800 0x7fdfa80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:02.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.657+0000 7fdfcc1f6740 1 -- 172.21.15.39:0/2522331271 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557103c01a90 msgr2=0x557103c01eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:02.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.657+0000 7fdfcc1f6740 1 --2- 172.21.15.39:0/2522331271 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557103c01a90 0x557103c01eb0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fdfbc02c630 tx=0x7fdfbc0323e0 comp rx=0 tx=0).stop 2024-03-18T16:15:02.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.657+0000 7fdfcc1f6740 1 -- 172.21.15.39:0/2522331271 shutdown_connections 2024-03-18T16:15:02.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.657+0000 7fdfcc1f6740 1 --2- 172.21.15.39:0/2522331271 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557103c075f0 0x557103c09a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:02.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.657+0000 7fdfcc1f6740 1 --2- 172.21.15.39:0/2522331271 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557103c02480 0x557103c06f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:02.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.657+0000 7fdfcc1f6740 1 --2- 172.21.15.39:0/2522331271 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557103c01a90 0x557103c01eb0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:02.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.657+0000 7fdfcc1f6740 1 -- 172.21.15.39:0/2522331271 >> 172.21.15.39:0/2522331271 conn(0x557103bfd000 msgr2=0x557103bfd400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:02.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.657+0000 7fdfcc1f6740 1 -- 172.21.15.39:0/2522331271 shutdown_connections 2024-03-18T16:15:02.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.657+0000 7fdfc2a00640 1 --2- 172.21.15.39:0/3114564541 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fdfa80a2800 0x7fdfa80a4cc0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fdfb40ccd30 tx=0x7fdfb40d2d30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:02.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.657+0000 7fdfcc1f6740 1 -- 172.21.15.39:0/2522331271 wait complete. 2024-03-18T16:15:02.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.657+0000 7fdfcc1f6740 1 -- start start 2024-03-18T16:15:02.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.658+0000 7fdfcc1f6740 1 Processor -- start 2024-03-18T16:15:02.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.658+0000 7fdfcc1f6740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557103c01a90 0x557103c100f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:02.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.658+0000 7fdfcc1f6740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557103c02480 0x557103c10630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:02.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.658+0000 7fdfcc1f6740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557103c075f0 0x557103c17670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:02.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.658+0000 7fdfcc1f6740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x557103b22110 con 0x557103c02480 2024-03-18T16:15:02.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.658+0000 7fdfcc1f6740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x557103b624f0 con 0x557103c075f0 2024-03-18T16:15:02.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.658+0000 7fdfcc1f6740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x557103c0a540 con 0x557103c01a90 2024-03-18T16:15:02.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.658+0000 7fdfcc1f6740 10 client.0 _ll_register_callbacks cb 0x557103b0fb70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:15:02.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.658+0000 7fdfc2000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557103c02480 0x557103c10630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:02.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.658+0000 7fdfc2000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557103c02480 0x557103c10630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:36520/0 (socket says 172.21.15.39:36520) 2024-03-18T16:15:02.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.658+0000 7fdfc2000640 1 -- 172.21.15.39:0/2422550877 learned_addr learned my addr 172.21.15.39:0/2422550877 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:15:02.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.658+0000 7fdfc3400640 1 --2- 172.21.15.39:0/2422550877 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557103c01a90 0x557103c100f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:02.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.659+0000 7fdfc2000640 1 -- 172.21.15.39:0/2422550877 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557103c01a90 msgr2=0x557103c100f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:02.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.659+0000 7fdfc2000640 1 --2- 172.21.15.39:0/2422550877 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557103c01a90 0x557103c100f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:02.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.659+0000 7fdfc2000640 1 -- 172.21.15.39:0/2422550877 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557103c075f0 msgr2=0x557103c17670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:02.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.659+0000 7fdfc2000640 1 --2- 172.21.15.39:0/2422550877 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557103c075f0 0x557103c17670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:02.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.659+0000 7fdfc2000640 1 -- 172.21.15.39:0/2422550877 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55710387a3f0 con 0x557103c02480 2024-03-18T16:15:02.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.659+0000 7fdfc3400640 1 --2- 172.21.15.39:0/2422550877 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557103c01a90 0x557103c100f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:15:02.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.659+0000 7fdfc2000640 1 --2- 172.21.15.39:0/2422550877 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557103c02480 0x557103c10630 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fdfb0012720 tx=0x7fdfb0012be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=465cb553c7977368 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:02.664 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.659+0000 7fdfa3e00640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-03-18T16:15:02.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.659+0000 7fdfa3e00640 1 -- 172.21.15.39:0/2422550877 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfb0004050 con 0x557103c02480 2024-03-18T16:15:02.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.659+0000 7fdfa3e00640 1 -- 172.21.15.39:0/2422550877 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fdfb00041f0 con 0x557103c02480 2024-03-18T16:15:02.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.659+0000 7fdfcc1f6740 1 -- 172.21.15.39:0/2422550877 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdfbc045070 con 0x557103c02480 2024-03-18T16:15:02.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.659+0000 7fdfa3e00640 1 -- 172.21.15.39:0/2422550877 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fdfb0025050 con 0x557103c02480 2024-03-18T16:15:02.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.659+0000 7fdfcc1f6740 10 client.16203 fetch_fsmap learned FSMap version 40 2024-03-18T16:15:02.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.659+0000 7fdfcc1f6740 1 -- 172.21.15.39:0/2422550877 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x5571039447d0 con 0x557103c02480 2024-03-18T16:15:02.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.660+0000 7fdfa3e00640 1 -- 172.21.15.39:0/2422550877 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdfbc045070 con 0x557103c02480 2024-03-18T16:15:02.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.660+0000 7fdfa3e00640 1 -- 172.21.15.39:0/2422550877 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 40) ==== 8378+0+0 (secure 0 0 0) 0x7fdfb0003ae0 con 0x557103c02480 2024-03-18T16:15:02.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.660+0000 7fdfcc1f6740 10 client.16203 fetch_fsmap finished waiting for FSMap version 40 2024-03-18T16:15:02.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.660+0000 7fdfcc1f6740 10 client.16203 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T16:15:02.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.660+0000 7fdfcc1f6740 20 client.16203 populate_metadata read hostname 'smithi039' 2024-03-18T16:15:02.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.660+0000 7fdfcc1f6740 1 --2- 172.21.15.39:0/2422550877 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x557103c1c0a0 0x557103c1e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:02.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.660+0000 7fdfcc1f6740 4 client.16203 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:15:02.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.660+0000 7fdfcc1f6740 1 -- 172.21.15.39:0/2422550877 --> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557103b73990 con 0x557103c1c0a0 2024-03-18T16:15:02.666 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.662+0000 7fdfc3400640 1 --2- 172.21.15.39:0/2422550877 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x557103c1c0a0 0x557103c1e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:02.666 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.665+0000 7fdfc3400640 1 --2- 172.21.15.39:0/2422550877 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x557103c1c0a0 0x557103c1e4e0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a9c05a257fbdfaa7 server_cookie=c00da36f7e966092 in_seq=0 out_seq=0 2024-03-18T16:15:02.666 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.665+0000 7fdfa3e00640 10 client.16203 ms_handle_connect on v2:172.21.15.196:6838/2001427268 2024-03-18T16:15:02.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.666+0000 7fdfa3e00640 1 -- 172.21.15.39:0/2422550877 <== mds.1 v2:172.21.15.196:6838/2001427268 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557103b73990 con 0x557103c1c0a0 2024-03-18T16:15:02.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.666+0000 7fdfa3e00640 10 client.16203 handle_command_reply: tid=0 2024-03-18T16:15:02.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.666+0000 7fdfcc1f6740 1 client.16203 shutdown 2024-03-18T16:15:02.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.666+0000 7fdfcc1f6740 1 -- 172.21.15.39:0/2422550877 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557103c02480 msgr2=0x557103c10630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:02.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.666+0000 7fdfcc1f6740 1 --2- 172.21.15.39:0/2422550877 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557103c02480 0x557103c10630 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fdfb0012720 tx=0x7fdfb0012be0 comp rx=0 tx=0).stop 2024-03-18T16:15:02.668 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.666+0000 7fdfcc1f6740 1 -- 172.21.15.39:0/2422550877 shutdown_connections 2024-03-18T16:15:02.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.666+0000 7fdfcc1f6740 1 --2- 172.21.15.39:0/2422550877 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x557103c1c0a0 0x557103c1e4e0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:02.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.666+0000 7fdfcc1f6740 1 --2- 172.21.15.39:0/2422550877 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557103c075f0 0x557103c17670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:02.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.666+0000 7fdfcc1f6740 1 --2- 172.21.15.39:0/2422550877 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557103c02480 0x557103c10630 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:02.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.666+0000 7fdfcc1f6740 1 --2- 172.21.15.39:0/2422550877 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557103c01a90 0x557103c100f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:02.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.666+0000 7fdfcc1f6740 1 -- 172.21.15.39:0/2422550877 >> 172.21.15.39:0/2422550877 conn(0x557103bfd000 msgr2=0x557103bfc220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:02.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.666+0000 7fdfa3e00640 0 client.16203 ms_handle_reset on v2:172.21.15.196:6838/2001427268 2024-03-18T16:15:02.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.667+0000 7fdfcc1f6740 1 -- 172.21.15.39:0/2422550877 shutdown_connections 2024-03-18T16:15:02.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.667+0000 7fdfcc1f6740 1 -- 172.21.15.39:0/2422550877 wait complete. 2024-03-18T16:15:02.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.667+0000 7fdfcc1f6740 20 client.16203 trim_cache size 0 max 16384 2024-03-18T16:15:02.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.681+0000 7fdfcc1f6740 1 Processor -- start 2024-03-18T16:15:02.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.681+0000 7fdfcc1f6740 1 -- start start 2024-03-18T16:15:02.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.681+0000 7fdfcc1f6740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557103c01a90 0x557103bfb990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:02.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.681+0000 7fdfcc1f6740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557103c02480 0x557103c17630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:02.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.681+0000 7fdfcc1f6740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557103c075f0 0x557103bfbed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:02.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.681+0000 7fdfcc1f6740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fdfc41a7fa0 con 0x557103c01a90 2024-03-18T16:15:02.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.681+0000 7fdfcc1f6740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fdfc41a9b10 con 0x557103c02480 2024-03-18T16:15:02.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.681+0000 7fdfcc1f6740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fdfc41a8670 con 0x557103c075f0 2024-03-18T16:15:02.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.681+0000 7fdfc3400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557103c01a90 0x557103bfb990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:02.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.682+0000 7fdfc3400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557103c01a90 0x557103bfb990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:36522/0 (socket says 172.21.15.39:36522) 2024-03-18T16:15:02.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.682+0000 7fdfc3400640 1 -- 172.21.15.39:0/1465935361 learned_addr learned my addr 172.21.15.39:0/1465935361 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:15:02.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.682+0000 7fdfc2000640 1 --2- 172.21.15.39:0/1465935361 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557103c02480 0x557103c17630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:02.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.682+0000 7fdfc3400640 1 --2- 172.21.15.39:0/1465935361 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557103c075f0 0x557103bfbed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:02.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.682+0000 7fdfc3400640 1 -- 172.21.15.39:0/1465935361 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557103c075f0 msgr2=0x557103bfbed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:02.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.682+0000 7fdfc3400640 1 --2- 172.21.15.39:0/1465935361 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557103c075f0 0x557103bfbed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:02.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.682+0000 7fdfc3400640 1 -- 172.21.15.39:0/1465935361 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557103c02480 msgr2=0x557103c17630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:02.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.682+0000 7fdfc3400640 1 --2- 172.21.15.39:0/1465935361 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557103c02480 0x557103c17630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:02.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.682+0000 7fdfc3400640 1 -- 172.21.15.39:0/1465935361 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557103c0a6c0 con 0x557103c01a90 2024-03-18T16:15:02.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.682+0000 7fdfc3400640 1 --2- 172.21.15.39:0/1465935361 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557103c01a90 0x557103bfb990 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fdfbc009ca0 tx=0x7fdfbc009a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7731f24b036e2d30 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:02.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.682+0000 7fdfaea00640 1 -- 172.21.15.39:0/1465935361 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfbc055070 con 0x557103c01a90 2024-03-18T16:15:02.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.682+0000 7fdfaea00640 1 -- 172.21.15.39:0/1465935361 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdfbc045040 con 0x557103c01a90 2024-03-18T16:15:02.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.682+0000 7fdfaea00640 1 -- 172.21.15.39:0/1465935361 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfbc0045d0 con 0x557103c01a90 2024-03-18T16:15:02.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.683+0000 7fdfcc1f6740 1 -- 172.21.15.39:0/1465935361 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557103c01a90 msgr2=0x557103bfb990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:02.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.683+0000 7fdfcc1f6740 1 --2- 172.21.15.39:0/1465935361 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557103c01a90 0x557103bfb990 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fdfbc009ca0 tx=0x7fdfbc009a20 comp rx=0 tx=0).stop 2024-03-18T16:15:02.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.683+0000 7fdfcc1f6740 1 -- 172.21.15.39:0/1465935361 shutdown_connections 2024-03-18T16:15:02.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.683+0000 7fdfcc1f6740 1 --2- 172.21.15.39:0/1465935361 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557103c02480 0x557103c17630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:02.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.683+0000 7fdfcc1f6740 1 --2- 172.21.15.39:0/1465935361 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557103c075f0 0x557103bfbed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:02.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.683+0000 7fdfcc1f6740 1 --2- 172.21.15.39:0/1465935361 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557103c01a90 0x557103bfb990 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:02.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.683+0000 7fdfcc1f6740 1 -- 172.21.15.39:0/1465935361 >> 172.21.15.39:0/1465935361 conn(0x557103bfd000 msgr2=0x557103c03840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:02.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.683+0000 7fdfcc1f6740 1 -- 172.21.15.39:0/1465935361 shutdown_connections 2024-03-18T16:15:02.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.683+0000 7fdfcc1f6740 1 -- 172.21.15.39:0/1465935361 wait complete. 2024-03-18T16:15:02.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.684+0000 7fdfcc1f6740 1 -- start start 2024-03-18T16:15:02.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.684+0000 7fdfcc1f6740 1 Processor -- start 2024-03-18T16:15:02.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.684+0000 7fdfcc1f6740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557103c01a90 0x557103c14f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:02.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.684+0000 7fdfcc1f6740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557103c02480 0x557103c15480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:02.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.684+0000 7fdfcc1f6740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557103c075f0 0x557103c16490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:02.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.684+0000 7fdfcc1f6740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fdfc41a9410 con 0x557103c075f0 2024-03-18T16:15:02.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.684+0000 7fdfcc1f6740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fdfc41a8d40 con 0x557103c01a90 2024-03-18T16:15:02.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.684+0000 7fdfcc1f6740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fdfc41a5f80 con 0x557103c02480 2024-03-18T16:15:02.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.684+0000 7fdfcc1f6740 10 client.0 _ll_register_callbacks cb 0x7fdfb0009910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:15:02.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.685+0000 7fdfc2000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557103c02480 0x557103c15480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:02.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.685+0000 7fdfc2000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557103c02480 0x557103c15480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:57762/0 (socket says 172.21.15.39:57762) 2024-03-18T16:15:02.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.685+0000 7fdfc3400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557103c075f0 0x557103c16490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:02.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.685+0000 7fdfc2000640 1 -- 172.21.15.39:0/449777837 learned_addr learned my addr 172.21.15.39:0/449777837 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:15:02.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.685+0000 7fdfc3400640 1 -- 172.21.15.39:0/449777837 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557103c02480 msgr2=0x557103c15480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:02.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.685+0000 7fdfc3400640 1 --2- 172.21.15.39:0/449777837 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557103c02480 0x557103c15480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:02.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.685+0000 7fdfc3400640 1 -- 172.21.15.39:0/449777837 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557103c01a90 msgr2=0x557103c14f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:02.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.685+0000 7fdfc3400640 1 --2- 172.21.15.39:0/449777837 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557103c01a90 0x557103c14f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:02.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.685+0000 7fdfc3400640 1 -- 172.21.15.39:0/449777837 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557103c16a80 con 0x557103c075f0 2024-03-18T16:15:02.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.685+0000 7fdfc3400640 1 --2- 172.21.15.39:0/449777837 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557103c075f0 0x557103c16490 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fdfbc02c630 tx=0x7fdfbc005f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ed30f3c69cebe42 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:02.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.685+0000 7fdfa3e00640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-03-18T16:15:02.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.686+0000 7fdfa3e00640 1 -- 172.21.15.39:0/449777837 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfbc060060 con 0x557103c075f0 2024-03-18T16:15:02.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.686+0000 7fdfa3e00640 1 -- 172.21.15.39:0/449777837 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fdfbc003970 con 0x557103c075f0 2024-03-18T16:15:02.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.686+0000 7fdfcc1f6740 1 -- 172.21.15.39:0/449777837 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdfbc045040 con 0x557103c075f0 2024-03-18T16:15:02.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.686+0000 7fdfa3e00640 1 -- 172.21.15.39:0/449777837 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fdfbc055040 con 0x557103c075f0 2024-03-18T16:15:02.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.686+0000 7fdfcc1f6740 10 client.16209 fetch_fsmap learned FSMap version 40 2024-03-18T16:15:02.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.686+0000 7fdfcc1f6740 1 -- 172.21.15.39:0/449777837 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x557103c82550 con 0x557103c075f0 2024-03-18T16:15:02.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.686+0000 7fdfa3e00640 1 -- 172.21.15.39:0/449777837 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdfbc045040 con 0x557103c075f0 2024-03-18T16:15:02.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.686+0000 7fdfa3e00640 1 -- 172.21.15.39:0/449777837 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 40) ==== 8378+0+0 (secure 0 0 0) 0x7fdfbc05b6e0 con 0x557103c075f0 2024-03-18T16:15:02.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.687+0000 7fdfcc1f6740 10 client.16209 fetch_fsmap finished waiting for FSMap version 40 2024-03-18T16:15:02.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.687+0000 7fdfcc1f6740 10 client.16209 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T16:15:02.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.687+0000 7fdfcc1f6740 20 client.16209 populate_metadata read hostname 'smithi039' 2024-03-18T16:15:02.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.687+0000 7fdfcc1f6740 1 --2- 172.21.15.39:0/449777837 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x557103c82b00 0x557103c84ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:02.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.687+0000 7fdfcc1f6740 4 client.16209 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:15:02.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.687+0000 7fdfcc1f6740 1 -- 172.21.15.39:0/449777837 --> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557103b6dce0 con 0x557103c82b00 2024-03-18T16:15:02.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.687+0000 7fdfc2000640 1 --2- 172.21.15.39:0/449777837 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x557103c82b00 0x557103c84ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:02.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.688+0000 7fdfc2000640 1 --2- 172.21.15.39:0/449777837 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x557103c82b00 0x557103c84ef0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=208b57feb64f4fbf server_cookie=9e6b4ebeb452ee41 in_seq=0 out_seq=0 2024-03-18T16:15:02.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.688+0000 7fdfa3e00640 10 client.16209 ms_handle_connect on v2:172.21.15.196:6838/2001427268 2024-03-18T16:15:02.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.688+0000 7fdfa3e00640 1 -- 172.21.15.39:0/449777837 <== mds.1 v2:172.21.15.196:6838/2001427268 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557103b6dce0 con 0x557103c82b00 2024-03-18T16:15:02.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.688+0000 7fdfa3e00640 10 client.16209 handle_command_reply: tid=0 2024-03-18T16:15:02.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.688+0000 7fdfcc1f6740 1 client.16209 shutdown 2024-03-18T16:15:02.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.688+0000 7fdfcc1f6740 1 -- 172.21.15.39:0/449777837 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557103c075f0 msgr2=0x557103c16490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:02.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.688+0000 7fdfcc1f6740 1 --2- 172.21.15.39:0/449777837 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557103c075f0 0x557103c16490 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fdfbc02c630 tx=0x7fdfbc005f80 comp rx=0 tx=0).stop 2024-03-18T16:15:02.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.688+0000 7fdfcc1f6740 1 -- 172.21.15.39:0/449777837 shutdown_connections 2024-03-18T16:15:02.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.688+0000 7fdfcc1f6740 1 --2- 172.21.15.39:0/449777837 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x557103c82b00 0x557103c84ef0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:02.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.688+0000 7fdfcc1f6740 1 --2- 172.21.15.39:0/449777837 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557103c075f0 0x557103c16490 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:02.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.688+0000 7fdfcc1f6740 1 --2- 172.21.15.39:0/449777837 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557103c02480 0x557103c15480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:02.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.688+0000 7fdfcc1f6740 1 --2- 172.21.15.39:0/449777837 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557103c01a90 0x557103c14f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:02.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.688+0000 7fdfa3e00640 0 client.16209 ms_handle_reset on v2:172.21.15.196:6838/2001427268 2024-03-18T16:15:02.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.688+0000 7fdfcc1f6740 1 -- 172.21.15.39:0/449777837 >> 172.21.15.39:0/449777837 conn(0x557103bfd000 msgr2=0x557103c00260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:02.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.688+0000 7fdfcc1f6740 1 -- 172.21.15.39:0/449777837 shutdown_connections 2024-03-18T16:15:02.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.688+0000 7fdfcc1f6740 1 -- 172.21.15.39:0/449777837 wait complete. 2024-03-18T16:15:02.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.688+0000 7fdfcc1f6740 20 client.16209 trim_cache size 0 max 16384 2024-03-18T16:15:02.694 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:15:02.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.689+0000 7fdfc3e00640 1 -- 172.21.15.39:0/3114564541 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fdfa80a2800 msgr2=0x7fdfa80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:02.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.689+0000 7fdfc3e00640 1 --2- 172.21.15.39:0/3114564541 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fdfa80a2800 0x7fdfa80a4cc0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fdfb40ccd30 tx=0x7fdfb40d2d30 comp rx=0 tx=0).stop 2024-03-18T16:15:02.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.689+0000 7fdfc3e00640 1 -- 172.21.15.39:0/3114564541 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdfc41015e0 msgr2=0x7fdfc4197870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:02.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.689+0000 7fdfc3e00640 1 --2- 172.21.15.39:0/3114564541 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdfc41015e0 0x7fdfc4197870 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fdfb400e9c0 tx=0x7fdfb400ee80 comp rx=0 tx=0).stop 2024-03-18T16:15:02.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.689+0000 7fdfc3e00640 1 -- 172.21.15.39:0/3114564541 shutdown_connections 2024-03-18T16:15:02.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.689+0000 7fdfc3e00640 1 --2- 172.21.15.39:0/3114564541 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdfc41027d0 0x7fdfc4197db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:02.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.689+0000 7fdfc3e00640 1 --2- 172.21.15.39:0/3114564541 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fdfa80a2800 0x7fdfa80a4cc0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:02.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.689+0000 7fdfc3e00640 1 --2- 172.21.15.39:0/3114564541 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdfc4103190 0x7fdfc419ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:02.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.689+0000 7fdfc3e00640 1 --2- 172.21.15.39:0/3114564541 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdfc41015e0 0x7fdfc4197870 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:02.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.689+0000 7fdfc3e00640 1 -- 172.21.15.39:0/3114564541 >> 172.21.15.39:0/3114564541 conn(0x7fdfc40fcb90 msgr2=0x7fdfc4105a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:02.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.689+0000 7fdfc3e00640 1 -- 172.21.15.39:0/3114564541 shutdown_connections 2024-03-18T16:15:02.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.689+0000 7fdfc3e00640 1 -- 172.21.15.39:0/3114564541 wait complete. 2024-03-18T16:15:02.706 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-18T16:15:02.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.971+0000 7fc208600640 1 -- 172.21.15.39:0/3415026500 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc200107570 msgr2=0x7fc200109960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:02.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.971+0000 7fc208600640 1 --2- 172.21.15.39:0/3415026500 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc200107570 0x7fc200109960 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fc1ec009900 tx=0x7fc1ec02bc30 comp rx=0 tx=0).stop 2024-03-18T16:15:02.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.972+0000 7fc208600640 1 -- 172.21.15.39:0/3415026500 shutdown_connections 2024-03-18T16:15:02.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.972+0000 7fc208600640 1 --2- 172.21.15.39:0/3415026500 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc200107570 0x7fc200109960 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:02.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.972+0000 7fc208600640 1 --2- 172.21.15.39:0/3415026500 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc200104c40 0x7fc200107030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:02.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.972+0000 7fc208600640 1 --2- 172.21.15.39:0/3415026500 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc200102310 0x7fc200104700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:02.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.972+0000 7fc208600640 1 -- 172.21.15.39:0/3415026500 >> 172.21.15.39:0/3415026500 conn(0x7fc2000fbc80 msgr2=0x7fc2000fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:02.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.974+0000 7fc208600640 1 -- 172.21.15.39:0/3415026500 shutdown_connections 2024-03-18T16:15:02.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.975+0000 7fc208600640 1 -- 172.21.15.39:0/3415026500 wait complete. 2024-03-18T16:15:02.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.976+0000 7fc208600640 1 Processor -- start 2024-03-18T16:15:02.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.976+0000 7fc208600640 1 -- start start 2024-03-18T16:15:02.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.977+0000 7fc208600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc200102310 0x7fc200101c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:02.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.977+0000 7fc208600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc200104c40 0x7fc2001002c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:02.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.977+0000 7fc208600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc200107570 0x7fc200100800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:02.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.977+0000 7fc208600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fc20010d4c0 con 0x7fc200102310 2024-03-18T16:15:02.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.977+0000 7fc208600640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fc20010d340 con 0x7fc200107570 2024-03-18T16:15:02.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.977+0000 7fc208600640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fc20010d640 con 0x7fc200104c40 2024-03-18T16:15:02.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.977+0000 7fc206600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc200107570 0x7fc200100800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:02.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.977+0000 7fc206600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc200107570 0x7fc200100800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:51052/0 (socket says 172.21.15.39:51052) 2024-03-18T16:15:02.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.977+0000 7fc206600640 1 -- 172.21.15.39:0/1128326780 learned_addr learned my addr 172.21.15.39:0/1128326780 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:15:02.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.978+0000 7fc205200640 1 --2- 172.21.15.39:0/1128326780 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc200104c40 0x7fc2001002c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:02.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.978+0000 7fc205c00640 1 --2- 172.21.15.39:0/1128326780 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc200102310 0x7fc200101c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:02.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.978+0000 7fc206600640 1 -- 172.21.15.39:0/1128326780 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc200104c40 msgr2=0x7fc2001002c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:02.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.978+0000 7fc206600640 1 --2- 172.21.15.39:0/1128326780 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc200104c40 0x7fc2001002c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:02.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.978+0000 7fc206600640 1 -- 172.21.15.39:0/1128326780 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc200102310 msgr2=0x7fc200101c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:02.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.978+0000 7fc206600640 1 --2- 172.21.15.39:0/1128326780 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc200102310 0x7fc200101c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:02.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.978+0000 7fc206600640 1 -- 172.21.15.39:0/1128326780 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc200101090 con 0x7fc200107570 2024-03-18T16:15:02.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.978+0000 7fc205c00640 1 --2- 172.21.15.39:0/1128326780 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc200102310 0x7fc200101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:15:02.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.978+0000 7fc206600640 1 --2- 172.21.15.39:0/1128326780 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc200107570 0x7fc200100800 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fc1ec02c8e0 tx=0x7fc1ec02c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:02.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.979+0000 7fc1fe000640 1 -- 172.21.15.39:0/1128326780 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1ec043020 con 0x7fc200107570 2024-03-18T16:15:02.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.979+0000 7fc208600640 1 -- 172.21.15.39:0/1128326780 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2001a1c00 con 0x7fc200107570 2024-03-18T16:15:02.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.979+0000 7fc208600640 1 -- 172.21.15.39:0/1128326780 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2001a21a0 con 0x7fc200107570 2024-03-18T16:15:02.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.979+0000 7fc1fe000640 1 -- 172.21.15.39:0/1128326780 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1ec0054e0 con 0x7fc200107570 2024-03-18T16:15:02.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.980+0000 7fc1fe000640 1 -- 172.21.15.39:0/1128326780 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1ec003ce0 con 0x7fc200107570 2024-03-18T16:15:02.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.987+0000 7fc20b3e9740 1 Processor -- start 2024-03-18T16:15:02.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.990+0000 7fc1fe000640 1 -- 172.21.15.39:0/1128326780 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fc1ec046020 con 0x7fc200107570 2024-03-18T16:15:02.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.991+0000 7fc20b3e9740 1 -- start start 2024-03-18T16:15:02.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.991+0000 7fc20b3e9740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563ea39200d0 0x563ea39204f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:02.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.991+0000 7fc20b3e9740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ea3920ac0 0x563ea39255c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:02.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.991+0000 7fc20b3e9740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563ea3925c30 0x563ea39280c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:02.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.991+0000 7fc20b3e9740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x563ea38df3b0 con 0x563ea3920ac0 2024-03-18T16:15:02.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.991+0000 7fc20b3e9740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x563ea387c450 con 0x563ea39200d0 2024-03-18T16:15:02.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.991+0000 7fc20b3e9740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x563ea3824cb0 con 0x563ea3925c30 2024-03-18T16:15:02.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.993+0000 7fc1fe000640 1 --2- 172.21.15.39:0/1128326780 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fc1e80a2880 0x7fc1e80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:02.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.993+0000 7fc205200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563ea3925c30 0x563ea39280c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:02.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.993+0000 7fc1fe000640 1 -- 172.21.15.39:0/1128326780 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fc1ec0f0ef0 con 0x7fc200107570 2024-03-18T16:15:02.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.993+0000 7fc205200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563ea3925c30 0x563ea39280c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:57790/0 (socket says 172.21.15.39:57790) 2024-03-18T16:15:02.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.993+0000 7fc205200640 1 -- 172.21.15.39:0/584969207 learned_addr learned my addr 172.21.15.39:0/584969207 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:15:02.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.994+0000 7fc205200640 1 -- 172.21.15.39:0/584969207 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563ea39200d0 msgr2=0x563ea39204f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:02.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.994+0000 7fc205200640 1 --2- 172.21.15.39:0/584969207 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563ea39200d0 0x563ea39204f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:02.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.994+0000 7fc205200640 1 -- 172.21.15.39:0/584969207 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ea3920ac0 msgr2=0x563ea39255c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:15:02.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.994+0000 7fc205200640 1 --2- 172.21.15.39:0/584969207 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ea3920ac0 0x563ea39255c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:02.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.994+0000 7fc205200640 1 -- 172.21.15.39:0/584969207 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ea38461f0 con 0x563ea3925c30 2024-03-18T16:15:02.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.994+0000 7fc206600640 1 --2- 172.21.15.39:0/1128326780 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fc1e80a2880 0x7fc1e80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:02.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.994+0000 7fc205200640 1 --2- 172.21.15.39:0/584969207 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563ea3925c30 0x563ea39280c0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fc1f4009560 tx=0x7fc1f4012820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2836ad279a6dc94 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:02.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.995+0000 7fc1f2a00640 1 -- 172.21.15.39:0/584969207 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1f401b440 con 0x563ea3925c30 2024-03-18T16:15:02.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.995+0000 7fc206600640 1 --2- 172.21.15.39:0/1128326780 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fc1e80a2880 0x7fc1e80a4d40 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fc1ec0b9e70 tx=0x7fc1ec02bc30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:02.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.995+0000 7fc1f2a00640 1 -- 172.21.15.39:0/584969207 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1f400f910 con 0x563ea3925c30 2024-03-18T16:15:02.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.995+0000 7fc1f2a00640 1 -- 172.21.15.39:0/584969207 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1f401aa20 con 0x563ea3925c30 2024-03-18T16:15:02.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.996+0000 7fc20b3e9740 1 -- 172.21.15.39:0/584969207 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563ea3925c30 msgr2=0x563ea39280c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:02.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.996+0000 7fc20b3e9740 1 --2- 172.21.15.39:0/584969207 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563ea3925c30 0x563ea39280c0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fc1f4009560 tx=0x7fc1f4012820 comp rx=0 tx=0).stop 2024-03-18T16:15:02.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.997+0000 7fc20b3e9740 1 -- 172.21.15.39:0/584969207 shutdown_connections 2024-03-18T16:15:02.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.997+0000 7fc20b3e9740 1 --2- 172.21.15.39:0/584969207 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563ea3925c30 0x563ea39280c0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:02.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.997+0000 7fc20b3e9740 1 --2- 172.21.15.39:0/584969207 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ea3920ac0 0x563ea39255c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:02.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.997+0000 7fc20b3e9740 1 --2- 172.21.15.39:0/584969207 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563ea39200d0 0x563ea39204f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:02.999 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.997+0000 7fc20b3e9740 1 -- 172.21.15.39:0/584969207 >> 172.21.15.39:0/584969207 conn(0x563ea3912e30 msgr2=0x563ea3913230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:03.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.999+0000 7fc20b3e9740 1 -- 172.21.15.39:0/584969207 shutdown_connections 2024-03-18T16:15:03.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:02.999+0000 7fc20b3e9740 1 -- 172.21.15.39:0/584969207 wait complete. 2024-03-18T16:15:03.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.000+0000 7fc20b3e9740 1 -- start start 2024-03-18T16:15:03.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.000+0000 7fc20b3e9740 1 Processor -- start 2024-03-18T16:15:03.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.001+0000 7fc20b3e9740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563ea39200d0 0x563ea3919490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:03.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.001+0000 7fc20b3e9740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ea3920ac0 0x563ea39199d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:03.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.001+0000 7fc20b3e9740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563ea391e9b0 0x563ea391ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:03.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.001+0000 7fc20b3e9740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x563ea3838110 con 0x563ea3920ac0 2024-03-18T16:15:03.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.001+0000 7fc20b3e9740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x563ea38784f0 con 0x563ea391e9b0 2024-03-18T16:15:03.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.001+0000 7fc20b3e9740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x563ea39179c0 con 0x563ea39200d0 2024-03-18T16:15:03.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.001+0000 7fc20b3e9740 10 client.0 _ll_register_callbacks cb 0x563ea3825b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:15:03.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.001+0000 7fc205c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ea3920ac0 0x563ea39199d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:03.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.002+0000 7fc205c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ea3920ac0 0x563ea39199d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:36570/0 (socket says 172.21.15.39:36570) 2024-03-18T16:15:03.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.002+0000 7fc205c00640 1 -- 172.21.15.39:0/1327568902 learned_addr learned my addr 172.21.15.39:0/1327568902 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:15:03.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.002+0000 7fc205200640 1 --2- 172.21.15.39:0/1327568902 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563ea391e9b0 0x563ea391ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:03.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.002+0000 7fc205200640 1 --2- 172.21.15.39:0/1327568902 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563ea39200d0 0x563ea3919490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:03.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.002+0000 7fc205c00640 1 -- 172.21.15.39:0/1327568902 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563ea39200d0 msgr2=0x563ea3919490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:03.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.002+0000 7fc205c00640 1 --2- 172.21.15.39:0/1327568902 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563ea39200d0 0x563ea3919490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:03.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.002+0000 7fc205c00640 1 -- 172.21.15.39:0/1327568902 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563ea391e9b0 msgr2=0x563ea391ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:03.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.002+0000 7fc205c00640 1 --2- 172.21.15.39:0/1327568902 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563ea391e9b0 0x563ea391ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:03.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.002+0000 7fc205c00640 1 -- 172.21.15.39:0/1327568902 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ea35903f0 con 0x563ea3920ac0 2024-03-18T16:15:03.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.003+0000 7fc205c00640 1 --2- 172.21.15.39:0/1327568902 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ea3920ac0 0x563ea39199d0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc1f800d7a0 tx=0x7fc1f800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b387194e0720c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:03.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.003+0000 7fc1e7400640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-03-18T16:15:03.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.004+0000 7fc1e7400640 1 -- 172.21.15.39:0/1327568902 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1f800bcb0 con 0x563ea3920ac0 2024-03-18T16:15:03.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.004+0000 7fc1e7400640 1 -- 172.21.15.39:0/1327568902 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fc1f8017420 con 0x563ea3920ac0 2024-03-18T16:15:03.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.004+0000 7fc20b3e9740 1 -- 172.21.15.39:0/1327568902 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc1f400f910 con 0x563ea3920ac0 2024-03-18T16:15:03.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.004+0000 7fc1e7400640 1 -- 172.21.15.39:0/1327568902 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fc1f8022050 con 0x563ea3920ac0 2024-03-18T16:15:03.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.004+0000 7fc20b3e9740 10 client.16215 fetch_fsmap learned FSMap version 40 2024-03-18T16:15:03.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.004+0000 7fc20b3e9740 1 -- 172.21.15.39:0/1327568902 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x563ea365a7d0 con 0x563ea3920ac0 2024-03-18T16:15:03.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.005+0000 7fc1e7400640 1 -- 172.21.15.39:0/1327568902 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1f400f910 con 0x563ea3920ac0 2024-03-18T16:15:03.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.005+0000 7fc1e7400640 1 -- 172.21.15.39:0/1327568902 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 40) ==== 8378+0+0 (secure 0 0 0) 0x7fc1f800f680 con 0x563ea3920ac0 2024-03-18T16:15:03.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.006+0000 7fc20b3e9740 10 client.16215 fetch_fsmap finished waiting for FSMap version 40 2024-03-18T16:15:03.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.006+0000 7fc20b3e9740 10 client.16215 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:15:03.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.006+0000 7fc20b3e9740 20 client.16215 populate_metadata read hostname 'smithi039' 2024-03-18T16:15:03.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.006+0000 7fc20b3e9740 1 --2- 172.21.15.39:0/1327568902 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x563ea393f180 0x563ea39415a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:03.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.006+0000 7fc20b3e9740 4 client.16215 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:15:03.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.006+0000 7fc20b3e9740 1 -- 172.21.15.39:0/1327568902 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563ea3889990 con 0x563ea393f180 2024-03-18T16:15:03.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.006+0000 7fc205200640 1 --2- 172.21.15.39:0/1327568902 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x563ea393f180 0x563ea39415a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:03.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.007+0000 7fc205200640 1 --2- 172.21.15.39:0/1327568902 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x563ea393f180 0x563ea39415a0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f99336e679abb18 server_cookie=b08af1d921eeb601 in_seq=0 out_seq=0 2024-03-18T16:15:03.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.010+0000 7fc1e7400640 10 client.16215 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:15:03.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.010+0000 7fc1e7400640 1 -- 172.21.15.39:0/1327568902 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563ea3889990 con 0x563ea393f180 2024-03-18T16:15:03.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.010+0000 7fc1e7400640 10 client.16215 handle_command_reply: tid=0 2024-03-18T16:15:03.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.010+0000 7fc20b3e9740 1 client.16215 shutdown 2024-03-18T16:15:03.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.010+0000 7fc20b3e9740 1 -- 172.21.15.39:0/1327568902 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ea3920ac0 msgr2=0x563ea39199d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:03.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.010+0000 7fc20b3e9740 1 --2- 172.21.15.39:0/1327568902 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ea3920ac0 0x563ea39199d0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc1f800d7a0 tx=0x7fc1f800dc60 comp rx=0 tx=0).stop 2024-03-18T16:15:03.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.010+0000 7fc20b3e9740 1 -- 172.21.15.39:0/1327568902 shutdown_connections 2024-03-18T16:15:03.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.010+0000 7fc20b3e9740 1 --2- 172.21.15.39:0/1327568902 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x563ea393f180 0x563ea39415a0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:03.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.010+0000 7fc20b3e9740 1 --2- 172.21.15.39:0/1327568902 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563ea391e9b0 0x563ea391ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:03.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.010+0000 7fc1e7400640 0 client.16215 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:15:03.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.010+0000 7fc20b3e9740 1 --2- 172.21.15.39:0/1327568902 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ea3920ac0 0x563ea39199d0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:03.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.010+0000 7fc20b3e9740 1 --2- 172.21.15.39:0/1327568902 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563ea39200d0 0x563ea3919490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:03.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.010+0000 7fc20b3e9740 1 -- 172.21.15.39:0/1327568902 >> 172.21.15.39:0/1327568902 conn(0x563ea3912e30 msgr2=0x563ea39121f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:03.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.011+0000 7fc20b3e9740 1 -- 172.21.15.39:0/1327568902 shutdown_connections 2024-03-18T16:15:03.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.013+0000 7fc20b3e9740 1 -- 172.21.15.39:0/1327568902 wait complete. 2024-03-18T16:15:03.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.013+0000 7fc20b3e9740 20 client.16215 trim_cache size 0 max 16384 2024-03-18T16:15:03.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.062+0000 7fc20b3e9740 1 Processor -- start 2024-03-18T16:15:03.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.062+0000 7fc20b3e9740 1 -- start start 2024-03-18T16:15:03.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.062+0000 7fc20b3e9740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ea391e9b0 0x563ea3918780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:03.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.063+0000 7fc20b3e9740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563ea39200d0 0x563ea3918cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:03.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.063+0000 7fc20b3e9740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563ea3920ac0 0x563ea3911960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:03.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.063+0000 7fc20b3e9740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fc1e000fbd0 con 0x563ea391e9b0 2024-03-18T16:15:03.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.063+0000 7fc20b3e9740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fc1e0011740 con 0x563ea3920ac0 2024-03-18T16:15:03.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.063+0000 7fc20b3e9740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fc1e00102a0 con 0x563ea39200d0 2024-03-18T16:15:03.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.063+0000 7fc205200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ea391e9b0 0x563ea3918780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:03.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.063+0000 7fc205200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ea391e9b0 0x563ea3918780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:36576/0 (socket says 172.21.15.39:36576) 2024-03-18T16:15:03.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.063+0000 7fc205200640 1 -- 172.21.15.39:0/1565538409 learned_addr learned my addr 172.21.15.39:0/1565538409 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:15:03.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.063+0000 7fc205200640 1 --2- 172.21.15.39:0/1565538409 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563ea3920ac0 0x563ea3911960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:03.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.064+0000 7fc205200640 1 -- 172.21.15.39:0/1565538409 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563ea39200d0 msgr2=0x563ea3918cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:03.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.065+0000 7fc205c00640 1 --2- 172.21.15.39:0/1565538409 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563ea39200d0 0x563ea3918cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:03.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.065+0000 7fc205200640 1 --2- 172.21.15.39:0/1565538409 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563ea39200d0 0x563ea3918cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:03.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.065+0000 7fc205200640 1 -- 172.21.15.39:0/1565538409 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563ea3920ac0 msgr2=0x563ea3911960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:03.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.065+0000 7fc205200640 1 --2- 172.21.15.39:0/1565538409 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563ea3920ac0 0x563ea3911960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:03.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.065+0000 7fc205200640 1 -- 172.21.15.39:0/1565538409 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ea3941c00 con 0x563ea391e9b0 2024-03-18T16:15:03.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.065+0000 7fc205200640 1 --2- 172.21.15.39:0/1565538409 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ea391e9b0 0x563ea3918780 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fc1f4009aa0 tx=0x7fc1f4009620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1810dcff95f6fac server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:03.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.066+0000 7fc1f2000640 1 -- 172.21.15.39:0/1565538409 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1f4020040 con 0x563ea391e9b0 2024-03-18T16:15:03.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.066+0000 7fc1f2000640 1 -- 172.21.15.39:0/1565538409 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1f400f910 con 0x563ea391e9b0 2024-03-18T16:15:03.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.066+0000 7fc1f2000640 1 -- 172.21.15.39:0/1565538409 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1f402d320 con 0x563ea391e9b0 2024-03-18T16:15:03.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.068+0000 7fc20b3e9740 1 -- 172.21.15.39:0/1565538409 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ea391e9b0 msgr2=0x563ea3918780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:03.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.068+0000 7fc20b3e9740 1 --2- 172.21.15.39:0/1565538409 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ea391e9b0 0x563ea3918780 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fc1f4009aa0 tx=0x7fc1f4009620 comp rx=0 tx=0).stop 2024-03-18T16:15:03.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.068+0000 7fc20b3e9740 1 -- 172.21.15.39:0/1565538409 shutdown_connections 2024-03-18T16:15:03.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.068+0000 7fc20b3e9740 1 --2- 172.21.15.39:0/1565538409 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563ea3920ac0 0x563ea3911960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:03.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.068+0000 7fc20b3e9740 1 --2- 172.21.15.39:0/1565538409 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563ea39200d0 0x563ea3918cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:03.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.068+0000 7fc20b3e9740 1 --2- 172.21.15.39:0/1565538409 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ea391e9b0 0x563ea3918780 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:03.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.068+0000 7fc20b3e9740 1 -- 172.21.15.39:0/1565538409 >> 172.21.15.39:0/1565538409 conn(0x563ea3912e30 msgr2=0x563ea3922170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:03.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.069+0000 7fc20b3e9740 1 -- 172.21.15.39:0/1565538409 shutdown_connections 2024-03-18T16:15:03.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.069+0000 7fc20b3e9740 1 -- 172.21.15.39:0/1565538409 wait complete. 2024-03-18T16:15:03.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.070+0000 7fc20b3e9740 1 -- start start 2024-03-18T16:15:03.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.070+0000 7fc20b3e9740 1 Processor -- start 2024-03-18T16:15:03.073 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.072+0000 7fc20b3e9740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ea391e9b0 0x563ea3950230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:03.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.072+0000 7fc20b3e9740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563ea39200d0 0x563ea3950770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:03.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.072+0000 7fc20b3e9740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563ea3920ac0 0x563ea39516e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:03.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.072+0000 7fc20b3e9740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fc1e0011040 con 0x563ea391e9b0 2024-03-18T16:15:03.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.072+0000 7fc20b3e9740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fc1e0010970 con 0x563ea3920ac0 2024-03-18T16:15:03.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.072+0000 7fc20b3e9740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fc1e000dbb0 con 0x563ea39200d0 2024-03-18T16:15:03.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.072+0000 7fc20b3e9740 10 client.0 _ll_register_callbacks cb 0x7fc1f8008470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:15:03.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.073+0000 7fc205200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ea391e9b0 0x563ea3950230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:03.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.073+0000 7fc205c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563ea39200d0 0x563ea3950770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:03.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.073+0000 7fc205c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563ea39200d0 0x563ea3950770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:57810/0 (socket says 172.21.15.39:57810) 2024-03-18T16:15:03.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.073+0000 7fc205c00640 1 -- 172.21.15.39:0/826977672 learned_addr learned my addr 172.21.15.39:0/826977672 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:15:03.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.074+0000 7fc205200640 1 --2- 172.21.15.39:0/826977672 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563ea3920ac0 0x563ea39516e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:03.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.074+0000 7fc205c00640 1 -- 172.21.15.39:0/826977672 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563ea3920ac0 msgr2=0x563ea39516e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:03.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.074+0000 7fc205c00640 1 --2- 172.21.15.39:0/826977672 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563ea3920ac0 0x563ea39516e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:03.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.074+0000 7fc205c00640 1 -- 172.21.15.39:0/826977672 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ea391e9b0 msgr2=0x563ea3950230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:03.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.074+0000 7fc205c00640 1 --2- 172.21.15.39:0/826977672 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ea391e9b0 0x563ea3950230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:03.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.074+0000 7fc205c00640 1 -- 172.21.15.39:0/826977672 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ea390ec10 con 0x563ea39200d0 2024-03-18T16:15:03.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.075+0000 7fc1e7400640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:15:03.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.075+0000 7fc205c00640 1 --2- 172.21.15.39:0/826977672 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563ea39200d0 0x563ea3950770 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fc1f800fc90 tx=0x7fc1f800bd60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=935de453a4fd1d57 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:03.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.077+0000 7fc1e7400640 1 -- 172.21.15.39:0/826977672 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1f800f830 con 0x563ea39200d0 2024-03-18T16:15:03.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.077+0000 7fc1e7400640 1 -- 172.21.15.39:0/826977672 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fc1f802f470 con 0x563ea39200d0 2024-03-18T16:15:03.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.077+0000 7fc20b3e9740 1 -- 172.21.15.39:0/826977672 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc1f400f910 con 0x563ea39200d0 2024-03-18T16:15:03.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.078+0000 7fc1e7400640 1 -- 172.21.15.39:0/826977672 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fc1f800b690 con 0x563ea39200d0 2024-03-18T16:15:03.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.078+0000 7fc20b3e9740 10 client.26605 fetch_fsmap learned FSMap version 40 2024-03-18T16:15:03.080 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.079+0000 7fc20b3e9740 1 -- 172.21.15.39:0/826977672 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x563ea39a5100 con 0x563ea39200d0 2024-03-18T16:15:03.081 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.079+0000 7fc1e7400640 1 -- 172.21.15.39:0/826977672 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc1f400f910 con 0x563ea39200d0 2024-03-18T16:15:03.081 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.080+0000 7fc1e7400640 1 -- 172.21.15.39:0/826977672 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 40) ==== 8378+0+0 (secure 0 0 0) 0x7fc1f8039480 con 0x563ea39200d0 2024-03-18T16:15:03.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.080+0000 7fc20b3e9740 10 client.26605 fetch_fsmap finished waiting for FSMap version 40 2024-03-18T16:15:03.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.080+0000 7fc20b3e9740 10 client.26605 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:15:03.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.080+0000 7fc20b3e9740 20 client.26605 populate_metadata read hostname 'smithi039' 2024-03-18T16:15:03.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.080+0000 7fc20b3e9740 1 --2- 172.21.15.39:0/826977672 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x563ea39a5ab0 0x563ea39a7ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:03.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.080+0000 7fc20b3e9740 4 client.26605 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:15:03.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.080+0000 7fc20b3e9740 1 -- 172.21.15.39:0/826977672 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563ea3883ce0 con 0x563ea39a5ab0 2024-03-18T16:15:03.083 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.081+0000 7fc205200640 1 --2- 172.21.15.39:0/826977672 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x563ea39a5ab0 0x563ea39a7ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:03.087 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.086+0000 7fc205200640 1 --2- 172.21.15.39:0/826977672 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x563ea39a5ab0 0x563ea39a7ed0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ea05d87ea37d1240 server_cookie=aa80ac712173f1b in_seq=0 out_seq=0 2024-03-18T16:15:03.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.086+0000 7fc1e7400640 10 client.26605 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:15:03.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.087+0000 7fc1e7400640 1 -- 172.21.15.39:0/826977672 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563ea3883ce0 con 0x563ea39a5ab0 2024-03-18T16:15:03.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.087+0000 7fc1e7400640 10 client.26605 handle_command_reply: tid=0 2024-03-18T16:15:03.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.087+0000 7fc20b3e9740 1 client.26605 shutdown 2024-03-18T16:15:03.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.088+0000 7fc20b3e9740 1 -- 172.21.15.39:0/826977672 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563ea39200d0 msgr2=0x563ea3950770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:03.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.088+0000 7fc20b3e9740 1 --2- 172.21.15.39:0/826977672 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563ea39200d0 0x563ea3950770 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fc1f800fc90 tx=0x7fc1f800bd60 comp rx=0 tx=0).stop 2024-03-18T16:15:03.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.088+0000 7fc20b3e9740 1 -- 172.21.15.39:0/826977672 shutdown_connections 2024-03-18T16:15:03.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.088+0000 7fc20b3e9740 1 --2- 172.21.15.39:0/826977672 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x563ea39a5ab0 0x563ea39a7ed0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:03.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.088+0000 7fc20b3e9740 1 --2- 172.21.15.39:0/826977672 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563ea3920ac0 0x563ea39516e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:03.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.088+0000 7fc20b3e9740 1 --2- 172.21.15.39:0/826977672 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563ea39200d0 0x563ea3950770 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:03.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.088+0000 7fc20b3e9740 1 --2- 172.21.15.39:0/826977672 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563ea391e9b0 0x563ea3950230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:03.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.088+0000 7fc20b3e9740 1 -- 172.21.15.39:0/826977672 >> 172.21.15.39:0/826977672 conn(0x563ea3912e30 msgr2=0x563ea3918780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:03.090 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.088+0000 7fc1e7400640 0 client.26605 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:15:03.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.089+0000 7fc20b3e9740 1 -- 172.21.15.39:0/826977672 shutdown_connections 2024-03-18T16:15:03.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.089+0000 7fc20b3e9740 1 -- 172.21.15.39:0/826977672 wait complete. 2024-03-18T16:15:03.091 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.089+0000 7fc20b3e9740 20 client.26605 trim_cache size 0 max 16384 2024-03-18T16:15:03.093 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:15:03.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.092+0000 7fc208600640 1 -- 172.21.15.39:0/1128326780 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fc1e80a2880 msgr2=0x7fc1e80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:03.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.092+0000 7fc208600640 1 --2- 172.21.15.39:0/1128326780 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fc1e80a2880 0x7fc1e80a4d40 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fc1ec0b9e70 tx=0x7fc1ec02bc30 comp rx=0 tx=0).stop 2024-03-18T16:15:03.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.092+0000 7fc208600640 1 -- 172.21.15.39:0/1128326780 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc200107570 msgr2=0x7fc200100800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:03.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.093+0000 7fc208600640 1 --2- 172.21.15.39:0/1128326780 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc200107570 0x7fc200100800 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fc1ec02c8e0 tx=0x7fc1ec02c6a0 comp rx=0 tx=0).stop 2024-03-18T16:15:03.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.093+0000 7fc208600640 1 -- 172.21.15.39:0/1128326780 shutdown_connections 2024-03-18T16:15:03.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.093+0000 7fc208600640 1 --2- 172.21.15.39:0/1128326780 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc200107570 0x7fc200100800 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:03.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.093+0000 7fc208600640 1 --2- 172.21.15.39:0/1128326780 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fc1e80a2880 0x7fc1e80a4d40 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:03.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.093+0000 7fc208600640 1 --2- 172.21.15.39:0/1128326780 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc200104c40 0x7fc2001002c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:03.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.093+0000 7fc208600640 1 --2- 172.21.15.39:0/1128326780 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc200102310 0x7fc200101c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:03.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.093+0000 7fc208600640 1 -- 172.21.15.39:0/1128326780 >> 172.21.15.39:0/1128326780 conn(0x7fc2000fbc80 msgr2=0x7fc200102c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:03.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.094+0000 7fc208600640 1 -- 172.21.15.39:0/1128326780 shutdown_connections 2024-03-18T16:15:03.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:03.094+0000 7fc208600640 1 -- 172.21.15.39:0/1128326780 wait complete. 2024-03-18T16:15:03.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:03 smithi039 ceph-mon[31721]: pgmap v571: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 30 MiB/s wr, 5.85k op/s 2024-03-18T16:15:03.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:03 smithi196 ceph-mon[36307]: pgmap v571: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 30 MiB/s wr, 5.85k op/s 2024-03-18T16:15:03.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:03 smithi194 ceph-mon[36200]: pgmap v571: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 30 MiB/s wr, 5.85k op/s 2024-03-18T16:15:04.124 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-18T16:15:04.125 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T16:15:04.125 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-18T16:15:04.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.463+0000 7fd96c800640 1 -- 172.21.15.39:0/2394470538 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd9641087e0 msgr2=0x7fd964108c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:04.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.463+0000 7fd96c800640 1 --2- 172.21.15.39:0/2394470538 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd9641087e0 0x7fd964108c00 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fd958009900 tx=0x7fd95802bc30 comp rx=0 tx=0).stop 2024-03-18T16:15:04.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.465+0000 7fd96c800640 1 -- 172.21.15.39:0/2394470538 shutdown_connections 2024-03-18T16:15:04.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.465+0000 7fd96c800640 1 --2- 172.21.15.39:0/2394470538 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd964110060 0x7fd964112450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:04.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.465+0000 7fd96c800640 1 --2- 172.21.15.39:0/2394470538 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd964109140 0x7fd96410fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:04.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.465+0000 7fd96c800640 1 --2- 172.21.15.39:0/2394470538 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd9641087e0 0x7fd964108c00 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:04.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.465+0000 7fd96c800640 1 -- 172.21.15.39:0/2394470538 >> 172.21.15.39:0/2394470538 conn(0x7fd96406e640 msgr2=0x7fd964070a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:04.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.465+0000 7fd96c800640 1 -- 172.21.15.39:0/2394470538 shutdown_connections 2024-03-18T16:15:04.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.465+0000 7fd96c800640 1 -- 172.21.15.39:0/2394470538 wait complete. 2024-03-18T16:15:04.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.467+0000 7fd96c800640 1 Processor -- start 2024-03-18T16:15:04.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.467+0000 7fd96c800640 1 -- start start 2024-03-18T16:15:04.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.468+0000 7fd96c800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd9641087e0 0x7fd9641a24f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:04.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.468+0000 7fd96c800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd964109140 0x7fd9641a2a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:04.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.468+0000 7fd96c800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd964110060 0x7fd9641a9aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:04.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.468+0000 7fd96c800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fd964115cd0 con 0x7fd964109140 2024-03-18T16:15:04.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.468+0000 7fd96c800640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fd964115b50 con 0x7fd9641087e0 2024-03-18T16:15:04.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.468+0000 7fd96c800640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fd964115e50 con 0x7fd964110060 2024-03-18T16:15:04.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.468+0000 7fd96a800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd964110060 0x7fd9641a9aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:04.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.468+0000 7fd96a800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd964110060 0x7fd9641a9aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:57838/0 (socket says 172.21.15.39:57838) 2024-03-18T16:15:04.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.468+0000 7fd96a800640 1 -- 172.21.15.39:0/86642406 learned_addr learned my addr 172.21.15.39:0/86642406 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:15:04.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.468+0000 7fd969e00640 1 --2- 172.21.15.39:0/86642406 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd9641087e0 0x7fd9641a24f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:04.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.469+0000 7fd969400640 1 --2- 172.21.15.39:0/86642406 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd964109140 0x7fd9641a2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:04.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.469+0000 7fd969e00640 1 -- 172.21.15.39:0/86642406 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd964110060 msgr2=0x7fd9641a9aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:04.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.469+0000 7fd969e00640 1 --2- 172.21.15.39:0/86642406 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd964110060 0x7fd9641a9aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:04.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.469+0000 7fd969e00640 1 -- 172.21.15.39:0/86642406 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd964109140 msgr2=0x7fd9641a2a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:04.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.469+0000 7fd969e00640 1 --2- 172.21.15.39:0/86642406 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd964109140 0x7fd9641a2a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:04.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.469+0000 7fd969e00640 1 -- 172.21.15.39:0/86642406 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9641aa110 con 0x7fd9641087e0 2024-03-18T16:15:04.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.470+0000 7fd969e00640 1 --2- 172.21.15.39:0/86642406 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd9641087e0 0x7fd9641a24f0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fd958036040 tx=0x7fd958004e30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:04.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.471+0000 7fd95ea00640 1 -- 172.21.15.39:0/86642406 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd958043020 con 0x7fd9641087e0 2024-03-18T16:15:04.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.471+0000 7fd95ea00640 1 -- 172.21.15.39:0/86642406 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd9580054e0 con 0x7fd9641087e0 2024-03-18T16:15:04.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.471+0000 7fd96c800640 1 -- 172.21.15.39:0/86642406 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9641aa3a0 con 0x7fd9641087e0 2024-03-18T16:15:04.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.471+0000 7fd95ea00640 1 -- 172.21.15.39:0/86642406 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd958003bf0 con 0x7fd9641087e0 2024-03-18T16:15:04.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.471+0000 7fd96c800640 1 -- 172.21.15.39:0/86642406 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9641aa910 con 0x7fd9641087e0 2024-03-18T16:15:04.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.480+0000 7fd95ea00640 1 -- 172.21.15.39:0/86642406 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fd958046020 con 0x7fd9641087e0 2024-03-18T16:15:04.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.480+0000 7fd96f571740 1 Processor -- start 2024-03-18T16:15:04.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.481+0000 7fd96f571740 1 -- start start 2024-03-18T16:15:04.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.482+0000 7fd96f571740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55789a36dff0 0x55789a36e410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:04.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.482+0000 7fd96a800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55789a36dff0 0x55789a36e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:04.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.482+0000 7fd96a800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55789a36dff0 0x55789a36e410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:51120/0 (socket says 172.21.15.39:51120) 2024-03-18T16:15:04.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.484+0000 7fd95ea00640 1 --2- 172.21.15.39:0/86642406 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd94c0a28d0 0x7fd94c0a4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:04.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.484+0000 7fd95ea00640 1 -- 172.21.15.39:0/86642406 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fd9580f0f90 con 0x7fd9641087e0 2024-03-18T16:15:04.487 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.485+0000 7fd969400640 1 --2- 172.21.15.39:0/86642406 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd94c0a28d0 0x7fd94c0a4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:04.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.486+0000 7fd96f571740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55789a36e9e0 0x55789a3734e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:04.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.486+0000 7fd96f571740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55789a373b50 0x55789a375fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:04.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.486+0000 7fd96f571740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55789a2c8f80 con 0x55789a373b50 2024-03-18T16:15:04.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.486+0000 7fd96f571740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55789a271ad0 con 0x55789a36dff0 2024-03-18T16:15:04.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.486+0000 7fd96f571740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55789a2c5020 con 0x55789a36e9e0 2024-03-18T16:15:04.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.487+0000 7fd96a800640 1 -- 172.21.15.39:0/3522698605 learned_addr learned my addr 172.21.15.39:0/3522698605 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:15:04.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.487+0000 7fd969e00640 1 --2- 172.21.15.39:0/3522698605 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55789a36e9e0 0x55789a3734e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:04.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.488+0000 7fd969400640 1 --2- 172.21.15.39:0/86642406 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd94c0a28d0 0x7fd94c0a4d90 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fd9641a3a00 tx=0x7fd954005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:04.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.488+0000 7fd96a800640 1 --2- 172.21.15.39:0/3522698605 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55789a373b50 0x55789a375fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:04.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.488+0000 7fd96a800640 1 -- 172.21.15.39:0/3522698605 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55789a36e9e0 msgr2=0x55789a3734e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:04.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.488+0000 7fd96a800640 1 --2- 172.21.15.39:0/3522698605 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55789a36e9e0 0x55789a3734e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:04.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.488+0000 7fd96a800640 1 -- 172.21.15.39:0/3522698605 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55789a373b50 msgr2=0x55789a375fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:04.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.488+0000 7fd96a800640 1 --2- 172.21.15.39:0/3522698605 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55789a373b50 0x55789a375fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:04.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.488+0000 7fd96a800640 1 -- 172.21.15.39:0/3522698605 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55789a2896f0 con 0x55789a36dff0 2024-03-18T16:15:04.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.490+0000 7fd96a800640 1 --2- 172.21.15.39:0/3522698605 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55789a36dff0 0x55789a36e410 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fd960012580 tx=0x7fd9600076d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ff0afb0c57cbb5b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:04.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.490+0000 7fd953000640 1 -- 172.21.15.39:0/3522698605 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd960016030 con 0x55789a36dff0 2024-03-18T16:15:04.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.490+0000 7fd953000640 1 -- 172.21.15.39:0/3522698605 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd96000e680 con 0x55789a36dff0 2024-03-18T16:15:04.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.491+0000 7fd96f571740 1 -- 172.21.15.39:0/3522698605 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55789a36dff0 msgr2=0x55789a36e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:04.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.491+0000 7fd96f571740 1 --2- 172.21.15.39:0/3522698605 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55789a36dff0 0x55789a36e410 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fd960012580 tx=0x7fd9600076d0 comp rx=0 tx=0).stop 2024-03-18T16:15:04.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.492+0000 7fd96f571740 1 -- 172.21.15.39:0/3522698605 shutdown_connections 2024-03-18T16:15:04.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.492+0000 7fd96f571740 1 --2- 172.21.15.39:0/3522698605 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55789a373b50 0x55789a375fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:04.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.492+0000 7fd96f571740 1 --2- 172.21.15.39:0/3522698605 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55789a36e9e0 0x55789a3734e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:04.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.492+0000 7fd96f571740 1 --2- 172.21.15.39:0/3522698605 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55789a36dff0 0x55789a36e410 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:04.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.492+0000 7fd96f571740 1 -- 172.21.15.39:0/3522698605 >> 172.21.15.39:0/3522698605 conn(0x55789a32d930 msgr2=0x55789a369960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:04.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.492+0000 7fd96f571740 1 -- 172.21.15.39:0/3522698605 shutdown_connections 2024-03-18T16:15:04.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.493+0000 7fd96f571740 1 -- 172.21.15.39:0/3522698605 wait complete. 2024-03-18T16:15:04.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.493+0000 7fd96f571740 1 -- start start 2024-03-18T16:15:04.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.494+0000 7fd96f571740 1 Processor -- start 2024-03-18T16:15:04.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.494+0000 7fd96f571740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55789a36dff0 0x55789a378320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:04.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.495+0000 7fd96f571740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55789a36e9e0 0x55789a378860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:04.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.495+0000 7fd96f571740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55789a373b50 0x55789a37f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:04.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.495+0000 7fd96f571740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55789a376ba0 con 0x55789a36dff0 2024-03-18T16:15:04.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.495+0000 7fd96f571740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55789a284f30 con 0x55789a36e9e0 2024-03-18T16:15:04.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.495+0000 7fd96f571740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55789a376d20 con 0x55789a373b50 2024-03-18T16:15:04.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.495+0000 7fd96f571740 10 client.0 _ll_register_callbacks cb 0x55789a272bc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:15:04.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.496+0000 7fd969400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55789a36e9e0 0x55789a378860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:04.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.496+0000 7fd969400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55789a36e9e0 0x55789a378860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:51122/0 (socket says 172.21.15.39:51122) 2024-03-18T16:15:04.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.496+0000 7fd96a800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55789a373b50 0x55789a37f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:04.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.496+0000 7fd969400640 1 -- 172.21.15.39:0/442084020 learned_addr learned my addr 172.21.15.39:0/442084020 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:15:04.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.497+0000 7fd96a800640 1 -- 172.21.15.39:0/442084020 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55789a36e9e0 msgr2=0x55789a378860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:04.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.497+0000 7fd96a800640 1 --2- 172.21.15.39:0/442084020 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55789a36e9e0 0x55789a378860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:04.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.497+0000 7fd96a800640 1 -- 172.21.15.39:0/442084020 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55789a36dff0 msgr2=0x55789a378320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:04.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.497+0000 7fd96a800640 1 --2- 172.21.15.39:0/442084020 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55789a36dff0 0x55789a378320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:04.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.497+0000 7fd96a800640 1 -- 172.21.15.39:0/442084020 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55789a0a57e0 con 0x55789a373b50 2024-03-18T16:15:04.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.498+0000 7fd96a800640 1 --2- 172.21.15.39:0/442084020 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55789a373b50 0x55789a37f8a0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x55789a36f240 tx=0x7fd9600075c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=256855c1b40b781a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:04.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.498+0000 7fd947400640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:15:04.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.498+0000 7fd947400640 1 -- 172.21.15.39:0/442084020 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd960016070 con 0x55789a373b50 2024-03-18T16:15:04.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.499+0000 7fd96f571740 1 -- 172.21.15.39:0/442084020 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd96000e680 con 0x55789a373b50 2024-03-18T16:15:04.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.500+0000 7fd947400640 1 -- 172.21.15.39:0/442084020 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fd9600056c0 con 0x55789a373b50 2024-03-18T16:15:04.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.501+0000 7fd947400640 1 -- 172.21.15.39:0/442084020 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fd960016070 con 0x55789a373b50 2024-03-18T16:15:04.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.503+0000 7fd96f571740 10 client.26614 fetch_fsmap learned FSMap version 40 2024-03-18T16:15:04.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.503+0000 7fd96f571740 1 -- 172.21.15.39:0/442084020 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55789a105f70 con 0x55789a373b50 2024-03-18T16:15:04.505 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.504+0000 7fd947400640 1 -- 172.21.15.39:0/442084020 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd96000e680 con 0x55789a373b50 2024-03-18T16:15:04.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.504+0000 7fd947400640 1 -- 172.21.15.39:0/442084020 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 40) ==== 8378+0+0 (secure 0 0 0) 0x7fd96002e9a0 con 0x55789a373b50 2024-03-18T16:15:04.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.504+0000 7fd96f571740 10 client.26614 fetch_fsmap finished waiting for FSMap version 40 2024-03-18T16:15:04.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.504+0000 7fd96f571740 10 client.26614 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:15:04.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.504+0000 7fd96f571740 20 client.26614 populate_metadata read hostname 'smithi039' 2024-03-18T16:15:04.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.504+0000 7fd96f571740 1 --2- 172.21.15.39:0/442084020 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55789a363b60 0x55789a365fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:04.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.504+0000 7fd96f571740 4 client.26614 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:15:04.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.504+0000 7fd96f571740 1 -- 172.21.15.39:0/442084020 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55789a2d64c0 con 0x55789a363b60 2024-03-18T16:15:04.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.505+0000 7fd969e00640 1 --2- 172.21.15.39:0/442084020 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55789a363b60 0x55789a365fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:04.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.506+0000 7fd969e00640 1 --2- 172.21.15.39:0/442084020 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55789a363b60 0x55789a365fe0 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=db370bb608eaf5f5 server_cookie=7c58ffd0fca3ad35 in_seq=0 out_seq=0 2024-03-18T16:15:04.507 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.506+0000 7fd947400640 10 client.26614 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:15:04.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.507+0000 7fd947400640 1 -- 172.21.15.39:0/442084020 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55789a2d64c0 con 0x55789a363b60 2024-03-18T16:15:04.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.507+0000 7fd947400640 10 client.26614 handle_command_reply: tid=0 2024-03-18T16:15:04.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.507+0000 7fd96f571740 1 client.26614 shutdown 2024-03-18T16:15:04.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.508+0000 7fd96f571740 1 -- 172.21.15.39:0/442084020 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55789a373b50 msgr2=0x55789a37f8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:04.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.508+0000 7fd96f571740 1 --2- 172.21.15.39:0/442084020 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55789a373b50 0x55789a37f8a0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x55789a36f240 tx=0x7fd9600075c0 comp rx=0 tx=0).stop 2024-03-18T16:15:04.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.508+0000 7fd96f571740 1 -- 172.21.15.39:0/442084020 shutdown_connections 2024-03-18T16:15:04.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.508+0000 7fd96f571740 1 --2- 172.21.15.39:0/442084020 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55789a363b60 0x55789a365fe0 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:04.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.508+0000 7fd96f571740 1 --2- 172.21.15.39:0/442084020 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55789a36e9e0 0x55789a378860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:04.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.508+0000 7fd96f571740 1 --2- 172.21.15.39:0/442084020 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55789a373b50 0x55789a37f8a0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:04.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.508+0000 7fd96f571740 1 --2- 172.21.15.39:0/442084020 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55789a36dff0 0x55789a378320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:04.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.508+0000 7fd96f571740 1 -- 172.21.15.39:0/442084020 >> 172.21.15.39:0/442084020 conn(0x55789a32d930 msgr2=0x55789a360350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:04.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.509+0000 7fd947400640 0 client.26614 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:15:04.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.512+0000 7fd96f571740 1 -- 172.21.15.39:0/442084020 shutdown_connections 2024-03-18T16:15:04.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.513+0000 7fd96f571740 1 -- 172.21.15.39:0/442084020 wait complete. 2024-03-18T16:15:04.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.513+0000 7fd96f571740 20 client.26614 trim_cache size 0 max 16384 2024-03-18T16:15:04.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.557+0000 7fd96f571740 1 Processor -- start 2024-03-18T16:15:04.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.557+0000 7fd96f571740 1 -- start start 2024-03-18T16:15:04.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.557+0000 7fd96f571740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55789a36dff0 0x55789a369480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:04.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.557+0000 7fd96f571740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55789a36e9e0 0x55789a37f860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:04.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.558+0000 7fd96f571740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55789a373b50 0x55789a3699c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:04.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.558+0000 7fd96f571740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fd9641b0a70 con 0x55789a373b50 2024-03-18T16:15:04.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.558+0000 7fd96f571740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fd9641b25e0 con 0x55789a36dff0 2024-03-18T16:15:04.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.558+0000 7fd96f571740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fd9641b1140 con 0x55789a36e9e0 2024-03-18T16:15:04.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.558+0000 7fd969e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55789a36dff0 0x55789a369480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:04.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.558+0000 7fd96a800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55789a373b50 0x55789a3699c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:04.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.559+0000 7fd96a800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55789a373b50 0x55789a3699c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:36618/0 (socket says 172.21.15.39:36618) 2024-03-18T16:15:04.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.559+0000 7fd96a800640 1 -- 172.21.15.39:0/1318061462 learned_addr learned my addr 172.21.15.39:0/1318061462 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:15:04.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.559+0000 7fd969400640 1 --2- 172.21.15.39:0/1318061462 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55789a36e9e0 0x55789a37f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:04.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.560+0000 7fd96a800640 1 -- 172.21.15.39:0/1318061462 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55789a36e9e0 msgr2=0x55789a37f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:04.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.560+0000 7fd96a800640 1 --2- 172.21.15.39:0/1318061462 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55789a36e9e0 0x55789a37f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:04.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.560+0000 7fd96a800640 1 -- 172.21.15.39:0/1318061462 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55789a36dff0 msgr2=0x55789a369480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:04.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.563+0000 7fd96a800640 1 --2- 172.21.15.39:0/1318061462 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55789a36dff0 0x55789a369480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:04.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.563+0000 7fd96a800640 1 -- 172.21.15.39:0/1318061462 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55789a35c980 con 0x55789a373b50 2024-03-18T16:15:04.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.563+0000 7fd969400640 1 --2- 172.21.15.39:0/1318061462 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55789a36e9e0 0x55789a37f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:15:04.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.563+0000 7fd969e00640 1 --2- 172.21.15.39:0/1318061462 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55789a36dff0 0x55789a369480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:15:04.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.563+0000 7fd96a800640 1 --2- 172.21.15.39:0/1318061462 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55789a373b50 0x55789a3699c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fd9600073c0 tx=0x7fd9600050d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efea76a75481cb26 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:04.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.564+0000 7fd952600640 1 -- 172.21.15.39:0/1318061462 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9600077a0 con 0x55789a373b50 2024-03-18T16:15:04.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.564+0000 7fd952600640 1 -- 172.21.15.39:0/1318061462 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd960033070 con 0x55789a373b50 2024-03-18T16:15:04.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.564+0000 7fd952600640 1 -- 172.21.15.39:0/1318061462 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd96001b850 con 0x55789a373b50 2024-03-18T16:15:04.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.565+0000 7fd96f571740 1 -- 172.21.15.39:0/1318061462 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55789a373b50 msgr2=0x55789a3699c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:04.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.565+0000 7fd96f571740 1 --2- 172.21.15.39:0/1318061462 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55789a373b50 0x55789a3699c0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fd9600073c0 tx=0x7fd9600050d0 comp rx=0 tx=0).stop 2024-03-18T16:15:04.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.566+0000 7fd96f571740 1 -- 172.21.15.39:0/1318061462 shutdown_connections 2024-03-18T16:15:04.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.566+0000 7fd96f571740 1 --2- 172.21.15.39:0/1318061462 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55789a373b50 0x55789a3699c0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:04.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.566+0000 7fd96f571740 1 --2- 172.21.15.39:0/1318061462 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55789a36e9e0 0x55789a37f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:04.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.566+0000 7fd96f571740 1 --2- 172.21.15.39:0/1318061462 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55789a36dff0 0x55789a369480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:04.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.566+0000 7fd96f571740 1 -- 172.21.15.39:0/1318061462 >> 172.21.15.39:0/1318061462 conn(0x55789a32d930 msgr2=0x55789a368890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:04.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.566+0000 7fd96f571740 1 -- 172.21.15.39:0/1318061462 shutdown_connections 2024-03-18T16:15:04.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.566+0000 7fd96f571740 1 -- 172.21.15.39:0/1318061462 wait complete. 2024-03-18T16:15:04.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.567+0000 7fd96f571740 1 -- start start 2024-03-18T16:15:04.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.567+0000 7fd96f571740 1 Processor -- start 2024-03-18T16:15:04.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.568+0000 7fd96f571740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55789a36dff0 0x55789a37f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:04.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.568+0000 7fd96f571740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55789a36e9e0 0x55789a37c190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:04.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.568+0000 7fd96f571740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55789a373b50 0x55789a37c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:04.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.568+0000 7fd96f571740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fd9641b1ee0 con 0x55789a373b50 2024-03-18T16:15:04.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.568+0000 7fd96f571740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fd9641b1810 con 0x55789a36dff0 2024-03-18T16:15:04.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.568+0000 7fd96f571740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fd9641aea50 con 0x55789a36e9e0 2024-03-18T16:15:04.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.568+0000 7fd96f571740 10 client.0 _ll_register_callbacks cb 0x7fd960012480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:15:04.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.569+0000 7fd969e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55789a36dff0 0x55789a37f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:04.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.569+0000 7fd969e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55789a36dff0 0x55789a37f6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:51134/0 (socket says 172.21.15.39:51134) 2024-03-18T16:15:04.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.569+0000 7fd969400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55789a36e9e0 0x55789a37c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:04.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.569+0000 7fd969400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55789a36e9e0 0x55789a37c190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:57856/0 (socket says 172.21.15.39:57856) 2024-03-18T16:15:04.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.569+0000 7fd969400640 1 -- 172.21.15.39:0/2338377839 learned_addr learned my addr 172.21.15.39:0/2338377839 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:15:04.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.569+0000 7fd96a800640 1 --2- 172.21.15.39:0/2338377839 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55789a373b50 0x55789a37c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:04.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.570+0000 7fd96a800640 1 -- 172.21.15.39:0/2338377839 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55789a36e9e0 msgr2=0x55789a37c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:04.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.570+0000 7fd96a800640 1 --2- 172.21.15.39:0/2338377839 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55789a36e9e0 0x55789a37c190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:04.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.570+0000 7fd96a800640 1 -- 172.21.15.39:0/2338377839 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55789a36dff0 msgr2=0x55789a37f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:04.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.570+0000 7fd96a800640 1 --2- 172.21.15.39:0/2338377839 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55789a36dff0 0x55789a37f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:04.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.570+0000 7fd96a800640 1 -- 172.21.15.39:0/2338377839 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557899f9d170 con 0x55789a373b50 2024-03-18T16:15:04.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.572+0000 7fd96a800640 1 --2- 172.21.15.39:0/2338377839 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55789a373b50 0x55789a37c6d0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fd960000f80 tx=0x7fd96000e680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80d22c29c95c502d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:04.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.572+0000 7fd947400640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-03-18T16:15:04.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.575+0000 7fd947400640 1 -- 172.21.15.39:0/2338377839 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd96003b070 con 0x55789a373b50 2024-03-18T16:15:04.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.575+0000 7fd947400640 1 -- 172.21.15.39:0/2338377839 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fd960005640 con 0x55789a373b50 2024-03-18T16:15:04.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.576+0000 7fd96f571740 1 -- 172.21.15.39:0/2338377839 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd960033070 con 0x55789a373b50 2024-03-18T16:15:04.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.578+0000 7fd947400640 1 -- 172.21.15.39:0/2338377839 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7fd96001b660 con 0x55789a373b50 2024-03-18T16:15:04.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.578+0000 7fd96f571740 10 client.16230 fetch_fsmap learned FSMap version 40 2024-03-18T16:15:04.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.578+0000 7fd96f571740 1 -- 172.21.15.39:0/2338377839 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x55789a3f25f0 con 0x55789a373b50 2024-03-18T16:15:04.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.583+0000 7fd947400640 1 -- 172.21.15.39:0/2338377839 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd960033070 con 0x55789a373b50 2024-03-18T16:15:04.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.583+0000 7fd947400640 1 -- 172.21.15.39:0/2338377839 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 40) ==== 8378+0+0 (secure 0 0 0) 0x7fd96003e030 con 0x55789a373b50 2024-03-18T16:15:04.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.583+0000 7fd96f571740 10 client.16230 fetch_fsmap finished waiting for FSMap version 40 2024-03-18T16:15:04.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.583+0000 7fd96f571740 10 client.16230 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:15:04.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.583+0000 7fd96f571740 20 client.16230 populate_metadata read hostname 'smithi039' 2024-03-18T16:15:04.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.583+0000 7fd96f571740 1 --2- 172.21.15.39:0/2338377839 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55789a3f2ea0 0x55789a3f52c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:04.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.583+0000 7fd96f571740 4 client.16230 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-18T16:15:04.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.583+0000 7fd96f571740 1 -- 172.21.15.39:0/2338377839 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55789a2d0810 con 0x55789a3f2ea0 2024-03-18T16:15:04.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.584+0000 7fd969e00640 1 --2- 172.21.15.39:0/2338377839 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55789a3f2ea0 0x55789a3f52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:04.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.585+0000 7fd969e00640 1 --2- 172.21.15.39:0/2338377839 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55789a3f2ea0 0x55789a3f52c0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d602fed9654b286 server_cookie=8178a34aa870a362 in_seq=0 out_seq=0 2024-03-18T16:15:04.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.585+0000 7fd947400640 10 client.16230 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:15:04.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.587+0000 7fd947400640 1 -- 172.21.15.39:0/2338377839 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55789a2d0810 con 0x55789a3f2ea0 2024-03-18T16:15:04.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.587+0000 7fd947400640 10 client.16230 handle_command_reply: tid=0 2024-03-18T16:15:04.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.587+0000 7fd96f571740 1 client.16230 shutdown 2024-03-18T16:15:04.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.588+0000 7fd96f571740 1 -- 172.21.15.39:0/2338377839 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55789a373b50 msgr2=0x55789a37c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:04.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.588+0000 7fd96f571740 1 --2- 172.21.15.39:0/2338377839 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55789a373b50 0x55789a37c6d0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fd960000f80 tx=0x7fd96000e680 comp rx=0 tx=0).stop 2024-03-18T16:15:04.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.588+0000 7fd96f571740 1 -- 172.21.15.39:0/2338377839 shutdown_connections 2024-03-18T16:15:04.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.588+0000 7fd96f571740 1 --2- 172.21.15.39:0/2338377839 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55789a3f2ea0 0x55789a3f52c0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:04.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.588+0000 7fd96f571740 1 --2- 172.21.15.39:0/2338377839 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55789a373b50 0x55789a37c6d0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:04.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.588+0000 7fd96f571740 1 --2- 172.21.15.39:0/2338377839 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55789a36e9e0 0x55789a37c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:04.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.588+0000 7fd96f571740 1 --2- 172.21.15.39:0/2338377839 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55789a36dff0 0x55789a37f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:04.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.588+0000 7fd96f571740 1 -- 172.21.15.39:0/2338377839 >> 172.21.15.39:0/2338377839 conn(0x55789a32d930 msgr2=0x55789a369480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:04.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.588+0000 7fd947400640 0 client.16230 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:15:04.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.588+0000 7fd96f571740 1 -- 172.21.15.39:0/2338377839 shutdown_connections 2024-03-18T16:15:04.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.588+0000 7fd96f571740 1 -- 172.21.15.39:0/2338377839 wait complete. 2024-03-18T16:15:04.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.590+0000 7fd96f571740 20 client.16230 trim_cache size 0 max 16384 2024-03-18T16:15:04.593 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-03-18T16:15:04.593 INFO:teuthology.orchestra.run.smithi039.stdout: "return_code": 0, 2024-03-18T16:15:04.593 INFO:teuthology.orchestra.run.smithi039.stdout: "scrub_tag": "e330d19e-36a1-4a6c-8a98-355466d06f39", 2024-03-18T16:15:04.593 INFO:teuthology.orchestra.run.smithi039.stdout: "mode": "asynchronous" 2024-03-18T16:15:04.593 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-03-18T16:15:04.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.592+0000 7fd96c800640 1 -- 172.21.15.39:0/86642406 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd94c0a28d0 msgr2=0x7fd94c0a4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:04.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.592+0000 7fd96c800640 1 --2- 172.21.15.39:0/86642406 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd94c0a28d0 0x7fd94c0a4d90 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fd9641a3a00 tx=0x7fd954005f90 comp rx=0 tx=0).stop 2024-03-18T16:15:04.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.592+0000 7fd96c800640 1 -- 172.21.15.39:0/86642406 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd9641087e0 msgr2=0x7fd9641a24f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:04.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.592+0000 7fd96c800640 1 --2- 172.21.15.39:0/86642406 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd9641087e0 0x7fd9641a24f0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fd958036040 tx=0x7fd958004e30 comp rx=0 tx=0).stop 2024-03-18T16:15:04.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.593+0000 7fd96c800640 1 -- 172.21.15.39:0/86642406 shutdown_connections 2024-03-18T16:15:04.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.593+0000 7fd96c800640 1 --2- 172.21.15.39:0/86642406 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd94c0a28d0 0x7fd94c0a4d90 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:04.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.593+0000 7fd96c800640 1 --2- 172.21.15.39:0/86642406 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd964110060 0x7fd9641a9aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:04.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.593+0000 7fd96c800640 1 --2- 172.21.15.39:0/86642406 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd964109140 0x7fd9641a2a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:04.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.593+0000 7fd96c800640 1 --2- 172.21.15.39:0/86642406 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd9641087e0 0x7fd9641a24f0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:04.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.593+0000 7fd96c800640 1 -- 172.21.15.39:0/86642406 >> 172.21.15.39:0/86642406 conn(0x7fd96406e640 msgr2=0x7fd9640702c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:04.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.593+0000 7fd96c800640 1 -- 172.21.15.39:0/86642406 shutdown_connections 2024-03-18T16:15:04.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.593+0000 7fd96c800640 1 -- 172.21.15.39:0/86642406 wait complete. 2024-03-18T16:15:04.620 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T16:15:04.620 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T16:15:04.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.947+0000 7f6447400640 1 -- 172.21.15.39:0/2025601903 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6440032990 con 0x7f64481087e0 2024-03-18T16:15:04.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.949+0000 7f644f400640 1 -- 172.21.15.39:0/2025601903 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f64481087e0 msgr2=0x7f644810ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:04.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.949+0000 7f644f400640 1 --2- 172.21.15.39:0/2025601903 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f64481087e0 0x7f644810ac10 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f6440009ca0 tx=0x7f644002c100 comp rx=0 tx=0).stop 2024-03-18T16:15:04.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.957+0000 7f644f400640 1 -- 172.21.15.39:0/2025601903 shutdown_connections 2024-03-18T16:15:04.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.957+0000 7f644f400640 1 --2- 172.21.15.39:0/2025601903 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f644810db00 0x7f644810df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:04.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.957+0000 7f644f400640 1 --2- 172.21.15.39:0/2025601903 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f644810b150 0x7f644810d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:04.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.957+0000 7f644f400640 1 --2- 172.21.15.39:0/2025601903 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f64481087e0 0x7f644810ac10 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:04.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.957+0000 7f644f400640 1 -- 172.21.15.39:0/2025601903 >> 172.21.15.39:0/2025601903 conn(0x7f644806e640 msgr2=0x7f6448070a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:04.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.957+0000 7f644f400640 1 -- 172.21.15.39:0/2025601903 shutdown_connections 2024-03-18T16:15:04.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.957+0000 7f644f400640 1 -- 172.21.15.39:0/2025601903 wait complete. 2024-03-18T16:15:04.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.958+0000 7f644f400640 1 Processor -- start 2024-03-18T16:15:04.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.959+0000 7f644f400640 1 -- start start 2024-03-18T16:15:04.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.959+0000 7f644f400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f64481087e0 0x7f64481a6990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:04.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.959+0000 7f644f400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f644810b150 0x7f64481a6ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:04.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.959+0000 7f644f400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f644810db00 0x7f64481adef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:04.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.959+0000 7f644ca00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f64481087e0 0x7f64481a6990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:04.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.960+0000 7f644f400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f644811c290 con 0x7f644810db00 2024-03-18T16:15:04.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.960+0000 7f644f400640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f644811c110 con 0x7f64481087e0 2024-03-18T16:15:04.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.960+0000 7f644f400640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f644811c410 con 0x7f644810b150 2024-03-18T16:15:04.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.960+0000 7f644d400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f644810db00 0x7f64481adef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:04.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.960+0000 7f644d400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f644810db00 0x7f64481adef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:36650/0 (socket says 172.21.15.39:36650) 2024-03-18T16:15:04.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.960+0000 7f644ca00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f64481087e0 0x7f64481a6990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:51152/0 (socket says 172.21.15.39:51152) 2024-03-18T16:15:04.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.960+0000 7f644d400640 1 -- 172.21.15.39:0/2241923112 learned_addr learned my addr 172.21.15.39:0/2241923112 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:15:04.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.961+0000 7f644d400640 1 -- 172.21.15.39:0/2241923112 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f644810b150 msgr2=0x7f64481a6ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:04.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.961+0000 7f644d400640 1 --2- 172.21.15.39:0/2241923112 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f644810b150 0x7f64481a6ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:04.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.961+0000 7f644d400640 1 -- 172.21.15.39:0/2241923112 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f64481087e0 msgr2=0x7f64481a6990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:04.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.961+0000 7f644d400640 1 --2- 172.21.15.39:0/2241923112 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f64481087e0 0x7f64481a6990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:04.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.961+0000 7f644d400640 1 -- 172.21.15.39:0/2241923112 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64481ae560 con 0x7f644810db00 2024-03-18T16:15:04.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.962+0000 7f644d400640 1 --2- 172.21.15.39:0/2241923112 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f644810db00 0x7f64481adef0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f643000e970 tx=0x7f643000ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:04.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.963+0000 7f6445600640 1 -- 172.21.15.39:0/2241923112 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f643000ccc0 con 0x7f644810db00 2024-03-18T16:15:04.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.963+0000 7f6445600640 1 -- 172.21.15.39:0/2241923112 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6430005500 con 0x7f644810db00 2024-03-18T16:15:04.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.963+0000 7f6445600640 1 -- 172.21.15.39:0/2241923112 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6430010690 con 0x7f644810db00 2024-03-18T16:15:04.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.963+0000 7f644f400640 1 -- 172.21.15.39:0/2241923112 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64481ae850 con 0x7f644810db00 2024-03-18T16:15:04.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.964+0000 7f644f400640 1 -- 172.21.15.39:0/2241923112 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64481aeda0 con 0x7f644810db00 2024-03-18T16:15:04.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.969+0000 7f64521e7740 1 Processor -- start 2024-03-18T16:15:04.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.971+0000 7f64521e7740 1 -- start start 2024-03-18T16:15:04.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.971+0000 7f6445600640 1 -- 172.21.15.39:0/2241923112 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f64300046c0 con 0x7f644810db00 2024-03-18T16:15:04.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.972+0000 7f64521e7740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5597a9dbc810 0x5597a9dbcc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:04.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.972+0000 7f64521e7740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5597a9dbd200 0x5597a9dc1d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:04.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.972+0000 7f64521e7740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5597a9dc2370 0x5597a9dc4800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:04.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.972+0000 7f64521e7740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5597a9d211f0 con 0x5597a9dbc810 2024-03-18T16:15:04.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.972+0000 7f64521e7740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5597a9cc9a80 con 0x5597a9dbd200 2024-03-18T16:15:04.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.972+0000 7f64521e7740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x5597a9d1d290 con 0x5597a9dc2370 2024-03-18T16:15:04.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.972+0000 7f644ca00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5597a9dbd200 0x5597a9dc1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:04.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.972+0000 7f644ca00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5597a9dbd200 0x5597a9dc1d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:51154/0 (socket says 172.21.15.39:51154) 2024-03-18T16:15:04.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.972+0000 7f644ca00640 1 -- 172.21.15.39:0/1125305794 learned_addr learned my addr 172.21.15.39:0/1125305794 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:15:04.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.973+0000 7f6447e00640 1 --2- 172.21.15.39:0/1125305794 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5597a9dc2370 0x5597a9dc4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:04.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.973+0000 7f644ca00640 1 -- 172.21.15.39:0/1125305794 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5597a9dc2370 msgr2=0x5597a9dc4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:04.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.973+0000 7f644ca00640 1 --2- 172.21.15.39:0/1125305794 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5597a9dc2370 0x5597a9dc4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:04.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.973+0000 7f644ca00640 1 -- 172.21.15.39:0/1125305794 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5597a9dbc810 msgr2=0x5597a9dbcc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:04.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.973+0000 7f644ca00640 1 --2- 172.21.15.39:0/1125305794 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5597a9dbc810 0x5597a9dbcc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:04.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.973+0000 7f644ca00640 1 -- 172.21.15.39:0/1125305794 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5597a9aff7d0 con 0x5597a9dbd200 2024-03-18T16:15:04.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.973+0000 7f6447e00640 1 --2- 172.21.15.39:0/1125305794 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5597a9dc2370 0x5597a9dc4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:15:04.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.974+0000 7f6445600640 1 --2- 172.21.15.39:0/2241923112 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f642c0a2900 0x7f642c0a4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:04.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.974+0000 7f6445600640 1 -- 172.21.15.39:0/2241923112 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f64300cf290 con 0x7f644810db00 2024-03-18T16:15:04.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.974+0000 7f644ca00640 1 --2- 172.21.15.39:0/1125305794 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5597a9dbd200 0x5597a9dc1d00 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f6440009ca0 tx=0x7f64400323d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=65c8e5273956e2ec server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:04.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.975+0000 7f6447e00640 1 --2- 172.21.15.39:0/2241923112 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f642c0a2900 0x7f642c0a4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:04.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.976+0000 7f643dc00640 1 -- 172.21.15.39:0/1125305794 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6440043070 con 0x5597a9dbd200 2024-03-18T16:15:04.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.976+0000 7f643dc00640 1 -- 172.21.15.39:0/1125305794 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6440003970 con 0x5597a9dbd200 2024-03-18T16:15:04.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.976+0000 7f643dc00640 1 -- 172.21.15.39:0/1125305794 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6440039040 con 0x5597a9dbd200 2024-03-18T16:15:04.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.977+0000 7f64521e7740 1 -- 172.21.15.39:0/1125305794 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5597a9dbd200 msgr2=0x5597a9dc1d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:04.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.977+0000 7f64521e7740 1 --2- 172.21.15.39:0/1125305794 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5597a9dbd200 0x5597a9dc1d00 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f6440009ca0 tx=0x7f64400323d0 comp rx=0 tx=0).stop 2024-03-18T16:15:04.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.977+0000 7f6447e00640 1 --2- 172.21.15.39:0/2241923112 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f642c0a2900 0x7f642c0a4dc0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x5597a9dbda60 tx=0x7f6438009880 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:04.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.977+0000 7f64521e7740 1 -- 172.21.15.39:0/1125305794 shutdown_connections 2024-03-18T16:15:04.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.977+0000 7f64521e7740 1 --2- 172.21.15.39:0/1125305794 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5597a9dbd200 0x5597a9dc1d00 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:04.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.977+0000 7f64521e7740 1 --2- 172.21.15.39:0/1125305794 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5597a9dc2370 0x5597a9dc4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:04.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.977+0000 7f64521e7740 1 --2- 172.21.15.39:0/1125305794 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5597a9dbc810 0x5597a9dbcc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:04.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.977+0000 7f64521e7740 1 -- 172.21.15.39:0/1125305794 >> 172.21.15.39:0/1125305794 conn(0x5597a9db7d80 msgr2=0x5597a9db8180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:04.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.977+0000 7f64521e7740 1 -- 172.21.15.39:0/1125305794 shutdown_connections 2024-03-18T16:15:04.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.978+0000 7f64521e7740 1 -- 172.21.15.39:0/1125305794 wait complete. 2024-03-18T16:15:04.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.978+0000 7f64521e7740 1 -- start start 2024-03-18T16:15:04.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.979+0000 7f64521e7740 1 Processor -- start 2024-03-18T16:15:04.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.980+0000 7f64521e7740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5597a9dbc810 0x5597a9dc6d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:04.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.980+0000 7f64521e7740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5597a9dbd200 0x5597a9dc7260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:04.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.980+0000 7f64521e7740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5597a9dc2370 0x5597a9dce260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:04.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.980+0000 7f64521e7740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5597a9dc52c0 con 0x5597a9dbc810 2024-03-18T16:15:04.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.980+0000 7f64521e7740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5597a9cdced0 con 0x5597a9dc2370 2024-03-18T16:15:04.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.980+0000 7f64521e7740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x5597a9dc5440 con 0x5597a9dbd200 2024-03-18T16:15:04.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.980+0000 7f64521e7740 10 client.0 _ll_register_callbacks cb 0x5597a9d18220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:15:04.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.981+0000 7f6447e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5597a9dbd200 0x5597a9dc7260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:04.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.981+0000 7f6447e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5597a9dbd200 0x5597a9dc7260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:57884/0 (socket says 172.21.15.39:57884) 2024-03-18T16:15:04.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.981+0000 7f6447e00640 1 -- 172.21.15.39:0/1968564257 learned_addr learned my addr 172.21.15.39:0/1968564257 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:15:04.983 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.981+0000 7f644ca00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5597a9dbc810 0x5597a9dc6d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:04.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.982+0000 7f644d400640 1 --2- 172.21.15.39:0/1968564257 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5597a9dc2370 0x5597a9dce260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:04.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.983+0000 7f6447e00640 1 -- 172.21.15.39:0/1968564257 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5597a9dc2370 msgr2=0x5597a9dce260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:04.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.983+0000 7f6447e00640 1 --2- 172.21.15.39:0/1968564257 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5597a9dc2370 0x5597a9dce260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:04.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.983+0000 7f6447e00640 1 -- 172.21.15.39:0/1968564257 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5597a9dbc810 msgr2=0x5597a9dc6d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:04.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.983+0000 7f6447e00640 1 --2- 172.21.15.39:0/1968564257 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5597a9dbc810 0x5597a9dc6d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:04.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.983+0000 7f6447e00640 1 -- 172.21.15.39:0/1968564257 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5597a9a353f0 con 0x5597a9dbd200 2024-03-18T16:15:04.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.983+0000 7f644d400640 1 --2- 172.21.15.39:0/1968564257 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5597a9dc2370 0x5597a9dce260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:15:04.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.983+0000 7f6447e00640 1 --2- 172.21.15.39:0/1968564257 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5597a9dbd200 0x5597a9dc7260 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f6438010760 tx=0x7f6438005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1aff18db5775df4b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:04.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.984+0000 7f6436000640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:15:04.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.984+0000 7f6436000640 1 -- 172.21.15.39:0/1968564257 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6438016070 con 0x5597a9dbd200 2024-03-18T16:15:04.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.984+0000 7f6436000640 1 -- 172.21.15.39:0/1968564257 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f643801bac0 con 0x5597a9dbd200 2024-03-18T16:15:04.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.984+0000 7f64521e7740 1 -- 172.21.15.39:0/1968564257 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6440003970 con 0x5597a9dbd200 2024-03-18T16:15:04.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.984+0000 7f6436000640 1 -- 172.21.15.39:0/1968564257 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f643802f050 con 0x5597a9dbd200 2024-03-18T16:15:04.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.984+0000 7f64521e7740 10 client.26626 fetch_fsmap learned FSMap version 40 2024-03-18T16:15:04.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.984+0000 7f64521e7740 1 -- 172.21.15.39:0/1968564257 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x5597a99f50f0 con 0x5597a9dbd200 2024-03-18T16:15:04.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.985+0000 7f6436000640 1 -- 172.21.15.39:0/1968564257 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6440003970 con 0x5597a9dbd200 2024-03-18T16:15:04.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.985+0000 7f6436000640 1 -- 172.21.15.39:0/1968564257 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 40) ==== 8378+0+0 (secure 0 0 0) 0x7f6438004aa0 con 0x5597a9dbd200 2024-03-18T16:15:04.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.986+0000 7f64521e7740 10 client.26626 fetch_fsmap finished waiting for FSMap version 40 2024-03-18T16:15:04.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.986+0000 7f64521e7740 10 client.26626 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:15:04.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.986+0000 7f64521e7740 20 client.26626 populate_metadata read hostname 'smithi039' 2024-03-18T16:15:04.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.986+0000 7f64521e7740 1 --2- 172.21.15.39:0/1968564257 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5597a9dd2af0 0x5597a9dd4f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:04.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.986+0000 7f64521e7740 4 client.26626 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:15:04.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.986+0000 7f64521e7740 1 -- 172.21.15.39:0/1968564257 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5597a9d34be0 con 0x5597a9dd2af0 2024-03-18T16:15:04.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.988+0000 7f644ca00640 1 --2- 172.21.15.39:0/1968564257 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5597a9dd2af0 0x5597a9dd4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:04.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.989+0000 7f644ca00640 1 --2- 172.21.15.39:0/1968564257 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5597a9dd2af0 0x5597a9dd4f60 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=975ac4ec330458a9 server_cookie=eb9118e0d7b017cf in_seq=0 out_seq=0 2024-03-18T16:15:04.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.989+0000 7f6436000640 10 client.26626 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:15:04.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.991+0000 7f6436000640 1 -- 172.21.15.39:0/1968564257 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5597a9d34be0 con 0x5597a9dd2af0 2024-03-18T16:15:04.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.991+0000 7f6436000640 10 client.26626 handle_command_reply: tid=0 2024-03-18T16:15:04.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.991+0000 7f64521e7740 1 client.26626 shutdown 2024-03-18T16:15:04.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.992+0000 7f64521e7740 1 -- 172.21.15.39:0/1968564257 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5597a9dbd200 msgr2=0x5597a9dc7260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:04.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.992+0000 7f64521e7740 1 --2- 172.21.15.39:0/1968564257 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5597a9dbd200 0x5597a9dc7260 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f6438010760 tx=0x7f6438005f90 comp rx=0 tx=0).stop 2024-03-18T16:15:04.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.992+0000 7f64521e7740 1 -- 172.21.15.39:0/1968564257 shutdown_connections 2024-03-18T16:15:04.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.992+0000 7f64521e7740 1 --2- 172.21.15.39:0/1968564257 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5597a9dd2af0 0x5597a9dd4f60 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:04.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.992+0000 7f64521e7740 1 --2- 172.21.15.39:0/1968564257 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5597a9dc2370 0x5597a9dce260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:04.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.992+0000 7f64521e7740 1 --2- 172.21.15.39:0/1968564257 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5597a9dbd200 0x5597a9dc7260 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:04.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.992+0000 7f64521e7740 1 --2- 172.21.15.39:0/1968564257 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5597a9dbc810 0x5597a9dc6d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:04.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.992+0000 7f64521e7740 1 -- 172.21.15.39:0/1968564257 >> 172.21.15.39:0/1968564257 conn(0x5597a9db7d80 msgr2=0x5597a9db6ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:04.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.992+0000 7f6436000640 0 client.26626 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:15:04.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.993+0000 7f64521e7740 1 -- 172.21.15.39:0/1968564257 shutdown_connections 2024-03-18T16:15:04.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.993+0000 7f64521e7740 1 -- 172.21.15.39:0/1968564257 wait complete. 2024-03-18T16:15:04.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:04.993+0000 7f64521e7740 20 client.26626 trim_cache size 0 max 16384 2024-03-18T16:15:05.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.030+0000 7f64521e7740 1 Processor -- start 2024-03-18T16:15:05.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.030+0000 7f64521e7740 1 -- start start 2024-03-18T16:15:05.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.031+0000 7f64521e7740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5597a9dbc810 0x5597a9db6630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:05.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.031+0000 7f64521e7740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5597a9dbd200 0x5597a9dce220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:05.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.031+0000 7f64521e7740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5597a9dc2370 0x5597a9db6b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:05.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.031+0000 7f64521e7740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f64481b9320 con 0x5597a9dc2370 2024-03-18T16:15:05.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.031+0000 7f64521e7740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f64481bae90 con 0x5597a9dbd200 2024-03-18T16:15:05.033 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.031+0000 7f64521e7740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f64481b99f0 con 0x5597a9dbc810 2024-03-18T16:15:05.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.031+0000 7f644ca00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5597a9dbc810 0x5597a9db6630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:05.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.031+0000 7f6447e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5597a9dbd200 0x5597a9dce220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:05.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.031+0000 7f6447e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5597a9dbd200 0x5597a9dce220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:51172/0 (socket says 172.21.15.39:51172) 2024-03-18T16:15:05.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.031+0000 7f6447e00640 1 -- 172.21.15.39:0/345531300 learned_addr learned my addr 172.21.15.39:0/345531300 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:15:05.034 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.032+0000 7f644d400640 1 --2- 172.21.15.39:0/345531300 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5597a9dc2370 0x5597a9db6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:05.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.033+0000 7f644d400640 1 -- 172.21.15.39:0/345531300 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5597a9dbc810 msgr2=0x5597a9db6630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:05.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.033+0000 7f644d400640 1 --2- 172.21.15.39:0/345531300 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5597a9dbc810 0x5597a9db6630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:05.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.033+0000 7f644d400640 1 -- 172.21.15.39:0/345531300 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5597a9dbd200 msgr2=0x5597a9dce220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:05.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.033+0000 7f644d400640 1 --2- 172.21.15.39:0/345531300 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5597a9dbd200 0x5597a9dce220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:05.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.033+0000 7f644d400640 1 -- 172.21.15.39:0/345531300 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5597a9db2370 con 0x5597a9dc2370 2024-03-18T16:15:05.035 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.033+0000 7f644d400640 1 --2- 172.21.15.39:0/345531300 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5597a9dc2370 0x5597a9db6b70 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f64300cc970 tx=0x7f64300096c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c18634591d9803b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:05.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.034+0000 7f643d200640 1 -- 172.21.15.39:0/345531300 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64300b3070 con 0x5597a9dc2370 2024-03-18T16:15:05.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.034+0000 7f643d200640 1 -- 172.21.15.39:0/345531300 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f643009bf40 con 0x5597a9dc2370 2024-03-18T16:15:05.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.034+0000 7f643d200640 1 -- 172.21.15.39:0/345531300 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64300cce60 con 0x5597a9dc2370 2024-03-18T16:15:05.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.035+0000 7f64521e7740 1 -- 172.21.15.39:0/345531300 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5597a9dc2370 msgr2=0x5597a9db6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:05.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.035+0000 7f64521e7740 1 --2- 172.21.15.39:0/345531300 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5597a9dc2370 0x5597a9db6b70 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f64300cc970 tx=0x7f64300096c0 comp rx=0 tx=0).stop 2024-03-18T16:15:05.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.035+0000 7f64521e7740 1 -- 172.21.15.39:0/345531300 shutdown_connections 2024-03-18T16:15:05.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.035+0000 7f64521e7740 1 --2- 172.21.15.39:0/345531300 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5597a9dbd200 0x5597a9dce220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:05.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.035+0000 7f64521e7740 1 --2- 172.21.15.39:0/345531300 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5597a9dc2370 0x5597a9db6b70 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:05.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.035+0000 7f64521e7740 1 --2- 172.21.15.39:0/345531300 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5597a9dbc810 0x5597a9db6630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:05.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.035+0000 7f64521e7740 1 -- 172.21.15.39:0/345531300 >> 172.21.15.39:0/345531300 conn(0x5597a9db7d80 msgr2=0x5597a9dbe790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:05.037 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.035+0000 7f64521e7740 1 -- 172.21.15.39:0/345531300 shutdown_connections 2024-03-18T16:15:05.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.035+0000 7f64521e7740 1 -- 172.21.15.39:0/345531300 wait complete. 2024-03-18T16:15:05.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.036+0000 7f64521e7740 1 -- start start 2024-03-18T16:15:05.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.036+0000 7f64521e7740 1 Processor -- start 2024-03-18T16:15:05.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.037+0000 7f64521e7740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5597a9dbc810 0x5597a9dc9cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:05.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.037+0000 7f64521e7740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5597a9dbd200 0x5597a9dca200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:05.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.037+0000 7f64521e7740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5597a9dc2370 0x5597a9dcd220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:05.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.037+0000 7f64521e7740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f64481ba790 con 0x5597a9dbc810 2024-03-18T16:15:05.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.037+0000 7f64521e7740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f64481ba0c0 con 0x5597a9dbd200 2024-03-18T16:15:05.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.037+0000 7f64521e7740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f64481b7300 con 0x5597a9dc2370 2024-03-18T16:15:05.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.037+0000 7f64521e7740 10 client.0 _ll_register_callbacks cb 0x7f64380096b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:15:05.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.038+0000 7f6447e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5597a9dbd200 0x5597a9dca200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:05.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.038+0000 7f6447e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5597a9dbd200 0x5597a9dca200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:51178/0 (socket says 172.21.15.39:51178) 2024-03-18T16:15:05.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.038+0000 7f644d400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5597a9dc2370 0x5597a9dcd220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:05.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.038+0000 7f6447e00640 1 -- 172.21.15.39:0/1093849497 learned_addr learned my addr 172.21.15.39:0/1093849497 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:15:05.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.038+0000 7f644ca00640 1 --2- 172.21.15.39:0/1093849497 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5597a9dbc810 0x5597a9dc9cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:05.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.038+0000 7f6447e00640 1 -- 172.21.15.39:0/1093849497 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5597a9dc2370 msgr2=0x5597a9dcd220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:05.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.038+0000 7f6447e00640 1 --2- 172.21.15.39:0/1093849497 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5597a9dc2370 0x5597a9dcd220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:05.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.038+0000 7f6447e00640 1 -- 172.21.15.39:0/1093849497 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5597a9dbc810 msgr2=0x5597a9dc9cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:05.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.038+0000 7f6447e00640 1 --2- 172.21.15.39:0/1093849497 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5597a9dbc810 0x5597a9dc9cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:05.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.038+0000 7f6447e00640 1 -- 172.21.15.39:0/1093849497 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5597a9dcd810 con 0x5597a9dbd200 2024-03-18T16:15:05.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.039+0000 7f6436000640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:15:05.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.039+0000 7f6447e00640 1 --2- 172.21.15.39:0/1093849497 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5597a9dbd200 0x5597a9dca200 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f643801c4c0 tx=0x7f6438005db0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=877bce270449e2ca server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:05.041 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.039+0000 7f6436000640 1 -- 172.21.15.39:0/1093849497 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6438039020 con 0x5597a9dbd200 2024-03-18T16:15:05.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.039+0000 7f6436000640 1 -- 172.21.15.39:0/1093849497 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f6438009040 con 0x5597a9dbd200 2024-03-18T16:15:05.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.040+0000 7f64521e7740 1 -- 172.21.15.39:0/1093849497 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f643009bf40 con 0x5597a9dbd200 2024-03-18T16:15:05.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.052+0000 7f6436000640 1 -- 172.21.15.39:0/1093849497 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) ==== 24+0+0 (secure 0 0 0) 0x7f6438039020 con 0x5597a9dbd200 2024-03-18T16:15:05.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.053+0000 7f64521e7740 10 client.26669 fetch_fsmap learned FSMap version 40 2024-03-18T16:15:05.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.053+0000 7f64521e7740 1 -- 172.21.15.39:0/1093849497 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) -- 0x5597a9e4a2d0 con 0x5597a9dbd200 2024-03-18T16:15:05.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.053+0000 7f6436000640 1 -- 172.21.15.39:0/1093849497 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6438009ac0 con 0x5597a9dbd200 2024-03-18T16:15:05.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.054+0000 7f6436000640 1 -- 172.21.15.39:0/1093849497 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 40) ==== 8378+0+0 (secure 0 0 0) 0x7f6438046020 con 0x5597a9dbd200 2024-03-18T16:15:05.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.054+0000 7f64521e7740 10 client.26669 fetch_fsmap finished waiting for FSMap version 40 2024-03-18T16:15:05.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.054+0000 7f64521e7740 10 client.26669 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:15:05.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.054+0000 7f64521e7740 20 client.26669 populate_metadata read hostname 'smithi039' 2024-03-18T16:15:05.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.054+0000 7f64521e7740 1 --2- 172.21.15.39:0/1093849497 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5597a9e4ad00 0x5597a9e4d120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:05.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.054+0000 7f64521e7740 4 client.26669 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T16:15:05.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.054+0000 7f64521e7740 1 -- 172.21.15.39:0/1093849497 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f643009bf40 con 0x5597a9e4ad00 2024-03-18T16:15:05.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.055+0000 7f644ca00640 1 --2- 172.21.15.39:0/1093849497 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5597a9e4ad00 0x5597a9e4d120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:05.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.058+0000 7f644ca00640 1 --2- 172.21.15.39:0/1093849497 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5597a9e4ad00 0x5597a9e4d120 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=22f734ce9b7bd81 server_cookie=d41a7f05deac33ca in_seq=0 out_seq=0 2024-03-18T16:15:05.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.058+0000 7f6436000640 10 client.26669 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:15:05.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.058+0000 7f6436000640 1 -- 172.21.15.39:0/1093849497 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+266 (crc 0 0 0) 0x7f643009bf40 con 0x5597a9e4ad00 2024-03-18T16:15:05.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.058+0000 7f6436000640 10 client.26669 handle_command_reply: tid=0 2024-03-18T16:15:05.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.059+0000 7f64521e7740 1 client.26669 shutdown 2024-03-18T16:15:05.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.060+0000 7f64521e7740 1 -- 172.21.15.39:0/1093849497 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5597a9dbd200 msgr2=0x5597a9dca200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:05.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.060+0000 7f64521e7740 1 --2- 172.21.15.39:0/1093849497 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5597a9dbd200 0x5597a9dca200 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f643801c4c0 tx=0x7f6438005db0 comp rx=0 tx=0).stop 2024-03-18T16:15:05.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.060+0000 7f64521e7740 1 -- 172.21.15.39:0/1093849497 shutdown_connections 2024-03-18T16:15:05.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.060+0000 7f64521e7740 1 --2- 172.21.15.39:0/1093849497 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5597a9e4ad00 0x5597a9e4d120 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:05.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.060+0000 7f64521e7740 1 --2- 172.21.15.39:0/1093849497 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5597a9dbd200 0x5597a9dca200 secure :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f643801c4c0 tx=0x7f6438005db0 comp rx=0 tx=0).stop 2024-03-18T16:15:05.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.060+0000 7f64521e7740 1 --2- 172.21.15.39:0/1093849497 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5597a9dc2370 0x5597a9dcd220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:05.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.060+0000 7f6436000640 0 client.26669 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:15:05.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.060+0000 7f64521e7740 1 --2- 172.21.15.39:0/1093849497 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5597a9dbc810 0x5597a9dc9cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:05.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.060+0000 7f64521e7740 1 -- 172.21.15.39:0/1093849497 >> 172.21.15.39:0/1093849497 conn(0x5597a9db7d80 msgr2=0x5597a9db6630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:05.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.060+0000 7f64521e7740 1 -- 172.21.15.39:0/1093849497 shutdown_connections 2024-03-18T16:15:05.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.060+0000 7f64521e7740 1 -- 172.21.15.39:0/1093849497 wait complete. 2024-03-18T16:15:05.063 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.062+0000 7f64521e7740 20 client.26669 trim_cache size 0 max 16384 2024-03-18T16:15:05.065 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-03-18T16:15:05.065 INFO:teuthology.orchestra.run.smithi039.stdout: "status": "scrub active (17 inodes in the stack)", 2024-03-18T16:15:05.065 INFO:teuthology.orchestra.run.smithi039.stdout: "scrubs": { 2024-03-18T16:15:05.065 INFO:teuthology.orchestra.run.smithi039.stdout: "e330d19e-36a1-4a6c-8a98-355466d06f39": { 2024-03-18T16:15:05.065 INFO:teuthology.orchestra.run.smithi039.stdout: "path": "/", 2024-03-18T16:15:05.065 INFO:teuthology.orchestra.run.smithi039.stdout: "tag": "e330d19e-36a1-4a6c-8a98-355466d06f39", 2024-03-18T16:15:05.066 INFO:teuthology.orchestra.run.smithi039.stdout: "options": "recursive,force" 2024-03-18T16:15:05.066 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:15:05.066 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:15:05.066 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-03-18T16:15:05.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.064+0000 7f644f400640 1 -- 172.21.15.39:0/2241923112 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f642c0a2900 msgr2=0x7f642c0a4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:05.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.064+0000 7f644f400640 1 --2- 172.21.15.39:0/2241923112 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f642c0a2900 0x7f642c0a4dc0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x5597a9dbda60 tx=0x7f6438009880 comp rx=0 tx=0).stop 2024-03-18T16:15:05.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.064+0000 7f644f400640 1 -- 172.21.15.39:0/2241923112 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f644810db00 msgr2=0x7f64481adef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:05.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.064+0000 7f644f400640 1 --2- 172.21.15.39:0/2241923112 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f644810db00 0x7f64481adef0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f643000e970 tx=0x7f643000ee30 comp rx=0 tx=0).stop 2024-03-18T16:15:05.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.065+0000 7f644f400640 1 -- 172.21.15.39:0/2241923112 shutdown_connections 2024-03-18T16:15:05.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.065+0000 7f644f400640 1 --2- 172.21.15.39:0/2241923112 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f642c0a2900 0x7f642c0a4dc0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:05.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.065+0000 7f644f400640 1 --2- 172.21.15.39:0/2241923112 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f644810db00 0x7f64481adef0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:05.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.065+0000 7f644f400640 1 --2- 172.21.15.39:0/2241923112 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f644810b150 0x7f64481a6ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:05.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.065+0000 7f644f400640 1 --2- 172.21.15.39:0/2241923112 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f64481087e0 0x7f64481a6990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:05.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.065+0000 7f644f400640 1 -- 172.21.15.39:0/2241923112 >> 172.21.15.39:0/2241923112 conn(0x7f644806e640 msgr2=0x7f644810b5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:05.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.065+0000 7f644f400640 1 -- 172.21.15.39:0/2241923112 shutdown_connections 2024-03-18T16:15:05.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:05.065+0000 7f644f400640 1 -- 172.21.15.39:0/2241923112 wait complete. 2024-03-18T16:15:05.097 INFO:tasks.cephfs.filesystem:scrub status for tag:e330d19e-36a1-4a6c-8a98-355466d06f39 - {'path': '/', 'tag': 'e330d19e-36a1-4a6c-8a98-355466d06f39', 'options': 'recursive,force'} 2024-03-18T16:15:05.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:05 smithi039 ceph-mon[31721]: pgmap v572: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 26 MiB/s wr, 5.12k op/s 2024-03-18T16:15:05.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:05 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:15:05.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:05 smithi196 ceph-mon[36307]: pgmap v572: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 26 MiB/s wr, 5.12k op/s 2024-03-18T16:15:05.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:05 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:15:05.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:05 smithi194 ceph-mon[36200]: pgmap v572: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 26 MiB/s wr, 5.12k op/s 2024-03-18T16:15:05.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:05 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:15:06.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:06 smithi039 ceph-mon[31721]: scrub queued for path: / 2024-03-18T16:15:06.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:06 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:06.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:06 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:15:06.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:06 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:15:06.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:06 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:15:06.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:06 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:15:06.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:06 smithi196 ceph-mon[36307]: scrub queued for path: / 2024-03-18T16:15:06.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:06 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:06.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:06 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:15:06.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:06 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:15:06.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:06 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:15:06.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:06 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:15:06.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:06 smithi194 ceph-mon[36200]: scrub queued for path: / 2024-03-18T16:15:06.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:06 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:06.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:06 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:15:06.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:06 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:15:06.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:06 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:15:06.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:06 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:15:07.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:07 smithi039 ceph-mon[31721]: pgmap v573: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 30 MiB/s wr, 5.86k op/s 2024-03-18T16:15:07.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:07 smithi196 ceph-mon[36307]: pgmap v573: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 30 MiB/s wr, 5.86k op/s 2024-03-18T16:15:07.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:07 smithi194 ceph-mon[36200]: pgmap v573: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 30 MiB/s wr, 5.86k op/s 2024-03-18T16:15:09.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:09 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:09.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:09 smithi039 ceph-mon[31721]: pgmap v574: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 27 MiB/s wr, 5.30k op/s 2024-03-18T16:15:09.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:09 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:09.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:09 smithi196 ceph-mon[36307]: pgmap v574: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 27 MiB/s wr, 5.30k op/s 2024-03-18T16:15:09.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:09 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:09.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:09 smithi194 ceph-mon[36200]: pgmap v574: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 27 MiB/s wr, 5.30k op/s 2024-03-18T16:15:10.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:10 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:10.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:10 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:10.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:10 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:11.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:11 smithi039 ceph-mon[31721]: pgmap v575: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 30 MiB/s wr, 5.96k op/s 2024-03-18T16:15:11.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:11 smithi196 ceph-mon[36307]: pgmap v575: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 30 MiB/s wr, 5.96k op/s 2024-03-18T16:15:11.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:11 smithi194 ceph-mon[36200]: pgmap v575: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 30 MiB/s wr, 5.96k op/s 2024-03-18T16:15:13.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:13 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:13.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:13 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:13.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:13 smithi196 ceph-mon[36307]: pgmap v576: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 29 MiB/s wr, 5.82k op/s 2024-03-18T16:15:13.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:13 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:15:13.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:13 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:13.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:13 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:15:13.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:13 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:13.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:13 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:15:13.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:13 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:13.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:13 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:15:13.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:13 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:13.480 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:13 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:15:13.480 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:13 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:13.480 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:13 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:15:13.480 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:13 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:13.480 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:13 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:15:13.480 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:13 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:13.480 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:13 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:15:13.480 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:13 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:13.480 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:13 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:15:13.480 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:13 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:13.480 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:13 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:15:13.480 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:13 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:13.480 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:13 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:15:13.480 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:13 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:13.480 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:13 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:15:13.480 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:13 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:13.481 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:13 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:15:13.481 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:13 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:13.481 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:13 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:13.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:13 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:13.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:13 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:13.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:13 smithi194 ceph-mon[36200]: pgmap v576: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 29 MiB/s wr, 5.82k op/s 2024-03-18T16:15:13.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:13 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:15:13.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:13 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:13.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:13 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:15:13.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:13 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:13.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:13 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:15:13.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:13 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:13.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:13 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:15:13.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:13 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:13.492 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:13 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:15:13.492 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:13 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:13.492 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:13 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:15:13.492 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:13 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:13.492 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:13 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:15:13.492 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:13 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:13.492 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:13 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:15:13.492 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:13 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:13.492 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:13 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:15:13.492 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:13 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:13.492 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:13 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:15:13.492 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:13 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:13.492 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:13 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:15:13.492 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:13 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:13.492 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:13 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:15:13.493 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:13 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:13.493 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:13 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:15:13.493 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:13 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:13.493 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:13 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:13.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:13 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:13.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:13 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:13.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:13 smithi039 ceph-mon[31721]: pgmap v576: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 29 MiB/s wr, 5.82k op/s 2024-03-18T16:15:13.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:13 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:15:13.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:13 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:13.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:13 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:15:13.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:13 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:13.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:13 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:15:13.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:13 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:13.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:13 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:15:13.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:13 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:13.589 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:13 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:15:13.589 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:13 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:13.589 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:13 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:15:13.589 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:13 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:13.589 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:13 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:15:13.589 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:13 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:13.589 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:13 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:15:13.589 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:13 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:13.589 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:13 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:15:13.589 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:13 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:13.589 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:13 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:15:13.589 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:13 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:13.589 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:13 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:15:13.589 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:13 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:13.589 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:13 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:15:13.589 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:13 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:13.590 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:13 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:15:13.590 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:13 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:13.590 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:13 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:15.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:15 smithi196 ceph-mon[36307]: pgmap v577: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 25 MiB/s wr, 5.08k op/s 2024-03-18T16:15:15.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:15 smithi194 ceph-mon[36200]: pgmap v577: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 25 MiB/s wr, 5.08k op/s 2024-03-18T16:15:15.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:15 smithi039 ceph-mon[31721]: pgmap v577: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 25 MiB/s wr, 5.08k op/s 2024-03-18T16:15:16.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:16 smithi196 ceph-mon[36307]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T16:15:16.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:16 smithi194 ceph-mon[36200]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T16:15:16.590 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:16 smithi039 ceph-mon[31721]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T16:15:17.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:17 smithi196 ceph-mon[36307]: pgmap v578: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 29 MiB/s wr, 5.96k op/s 2024-03-18T16:15:17.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:17 smithi194 ceph-mon[36200]: pgmap v578: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 29 MiB/s wr, 5.96k op/s 2024-03-18T16:15:17.490 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:17 smithi039 ceph-mon[31721]: pgmap v578: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 29 MiB/s wr, 5.96k op/s 2024-03-18T16:15:19.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:19 smithi196 ceph-mon[36307]: pgmap v579: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 27 MiB/s wr, 5.73k op/s 2024-03-18T16:15:19.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:19 smithi196 ceph-mon[36307]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T16:15:19.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:19 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:15:19.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:19 smithi194 ceph-mon[36200]: pgmap v579: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 27 MiB/s wr, 5.73k op/s 2024-03-18T16:15:19.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:19 smithi194 ceph-mon[36200]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T16:15:19.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:19 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:15:19.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:19 smithi039 ceph-mon[31721]: pgmap v579: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 27 MiB/s wr, 5.73k op/s 2024-03-18T16:15:19.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:19 smithi039 ceph-mon[31721]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T16:15:19.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:19 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:15:21.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:21 smithi196 ceph-mon[36307]: pgmap v580: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 25 MiB/s wr, 5.31k op/s 2024-03-18T16:15:21.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:21 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:15:21.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:21 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:15:21.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:21 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:15:21.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:21 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:15:21.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:21 smithi196 ceph-mon[36307]: Adjusting osd_memory_target on smithi194 to 174.1M 2024-03-18T16:15:21.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:21 smithi196 ceph-mon[36307]: Unable to set osd_memory_target on smithi194 to 182639923: error parsing value: Value '182639923' is below minimum 939524096 2024-03-18T16:15:21.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:21 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:15:21.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:21 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:15:21.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:21 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:15:21.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:21 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:15:21.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:21 smithi039 ceph-mon[31721]: pgmap v580: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 25 MiB/s wr, 5.31k op/s 2024-03-18T16:15:21.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:21 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:15:21.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:21 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:15:21.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:21 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:15:21.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:21 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:15:21.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:21 smithi039 ceph-mon[31721]: Adjusting osd_memory_target on smithi194 to 174.1M 2024-03-18T16:15:21.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:21 smithi039 ceph-mon[31721]: Unable to set osd_memory_target on smithi194 to 182639923: error parsing value: Value '182639923' is below minimum 939524096 2024-03-18T16:15:21.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:21 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:15:21.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:21 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:15:21.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:21 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:15:21.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:21 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:15:21.713 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:21 smithi194 ceph-mon[36200]: pgmap v580: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 KiB/s rd, 25 MiB/s wr, 5.31k op/s 2024-03-18T16:15:21.714 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:21 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:15:21.714 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:21 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:15:21.714 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:21 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:15:21.714 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:21 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:15:21.714 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:21 smithi194 ceph-mon[36200]: Adjusting osd_memory_target on smithi194 to 174.1M 2024-03-18T16:15:21.714 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:21 smithi194 ceph-mon[36200]: Unable to set osd_memory_target on smithi194 to 182639923: error parsing value: Value '182639923' is below minimum 939524096 2024-03-18T16:15:21.714 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:21 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:15:21.714 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:21 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:15:21.714 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:21 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:15:21.714 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:21 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:15:22.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:22 smithi196 ceph-mon[36307]: Adjusting osd_memory_target on smithi039 to 176.8M 2024-03-18T16:15:22.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:22 smithi196 ceph-mon[36307]: Unable to set osd_memory_target on smithi039 to 185393152: error parsing value: Value '185393152' is below minimum 939524096 2024-03-18T16:15:22.551 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:22 smithi194 ceph-mon[36200]: Adjusting osd_memory_target on smithi039 to 176.8M 2024-03-18T16:15:22.551 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:22 smithi194 ceph-mon[36200]: Unable to set osd_memory_target on smithi039 to 185393152: error parsing value: Value '185393152' is below minimum 939524096 2024-03-18T16:15:22.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:22 smithi039 ceph-mon[31721]: Adjusting osd_memory_target on smithi039 to 176.8M 2024-03-18T16:15:22.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:22 smithi039 ceph-mon[31721]: Unable to set osd_memory_target on smithi039 to 185393152: error parsing value: Value '185393152' is below minimum 939524096 2024-03-18T16:15:23.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:23 smithi196 ceph-mon[36307]: pgmap v581: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 24 MiB/s wr, 5.16k op/s 2024-03-18T16:15:23.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:23 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:15:23.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:23 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:15:23.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:23 smithi196 ceph-mon[36307]: scrub summary: idle 2024-03-18T16:15:23.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:23 smithi196 ceph-mon[36307]: scrub summary: idle 2024-03-18T16:15:23.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:23 smithi196 ceph-mon[36307]: scrub summary: idle 2024-03-18T16:15:23.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:23 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:15:23.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:23 smithi194 ceph-mon[36200]: pgmap v581: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 24 MiB/s wr, 5.16k op/s 2024-03-18T16:15:23.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:23 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:15:23.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:23 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:15:23.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:23 smithi194 ceph-mon[36200]: scrub summary: idle 2024-03-18T16:15:23.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:23 smithi194 ceph-mon[36200]: scrub summary: idle 2024-03-18T16:15:23.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:23 smithi194 ceph-mon[36200]: scrub summary: idle 2024-03-18T16:15:23.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:23 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:15:23.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:23 smithi039 ceph-mon[31721]: pgmap v581: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 24 MiB/s wr, 5.16k op/s 2024-03-18T16:15:23.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:23 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:15:23.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:23 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:15:23.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:23 smithi039 ceph-mon[31721]: scrub summary: idle 2024-03-18T16:15:23.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:23 smithi039 ceph-mon[31721]: scrub summary: idle 2024-03-18T16:15:23.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:23 smithi039 ceph-mon[31721]: scrub summary: idle 2024-03-18T16:15:23.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:23 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:15:24.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:24 smithi196 ceph-mon[36307]: scrub summary: idle 2024-03-18T16:15:24.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:24 smithi196 ceph-mon[36307]: scrub summary: idle 2024-03-18T16:15:24.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:24 smithi039 ceph-mon[31721]: scrub summary: idle 2024-03-18T16:15:24.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:24 smithi039 ceph-mon[31721]: scrub summary: idle 2024-03-18T16:15:24.760 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:24 smithi194 ceph-mon[36200]: scrub summary: idle 2024-03-18T16:15:24.760 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:24 smithi194 ceph-mon[36200]: scrub summary: idle 2024-03-18T16:15:25.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:25 smithi196 ceph-mon[36307]: pgmap v582: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 21 MiB/s wr, 4.50k op/s 2024-03-18T16:15:25.538 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:25 smithi194 ceph-mon[36200]: pgmap v582: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 21 MiB/s wr, 4.50k op/s 2024-03-18T16:15:25.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:25 smithi039 ceph-mon[31721]: pgmap v582: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 21 MiB/s wr, 4.50k op/s 2024-03-18T16:15:27.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:27 smithi196 ceph-mon[36307]: pgmap v583: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 20 MiB/s wr, 4.38k op/s 2024-03-18T16:15:27.501 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:27 smithi194 ceph-mon[36200]: pgmap v583: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 20 MiB/s wr, 4.38k op/s 2024-03-18T16:15:27.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:27 smithi039 ceph-mon[31721]: pgmap v583: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 20 MiB/s wr, 4.38k op/s 2024-03-18T16:15:29.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:29 smithi196 ceph-mon[36307]: pgmap v584: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 14 MiB/s wr, 3.15k op/s 2024-03-18T16:15:29.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:29 smithi194 ceph-mon[36200]: pgmap v584: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 14 MiB/s wr, 3.15k op/s 2024-03-18T16:15:29.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:29 smithi039 ceph-mon[31721]: pgmap v584: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 14 MiB/s wr, 3.15k op/s 2024-03-18T16:15:31.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:31 smithi196 ceph-mon[36307]: pgmap v585: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 10 MiB/s wr, 2.27k op/s 2024-03-18T16:15:31.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:31 smithi194 ceph-mon[36200]: pgmap v585: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 10 MiB/s wr, 2.27k op/s 2024-03-18T16:15:31.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:31 smithi039 ceph-mon[31721]: pgmap v585: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 10 MiB/s wr, 2.27k op/s 2024-03-18T16:15:33.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:33 smithi196 ceph-mon[36307]: pgmap v586: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 10 MiB/s wr, 2.28k op/s 2024-03-18T16:15:33.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:33 smithi194 ceph-mon[36200]: pgmap v586: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 10 MiB/s wr, 2.28k op/s 2024-03-18T16:15:33.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:33 smithi039 ceph-mon[31721]: pgmap v586: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 10 MiB/s wr, 2.28k op/s 2024-03-18T16:15:35.098 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T16:15:35.098 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T16:15:35.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.422+0000 7f2746a00640 1 -- 172.21.15.39:0/3703615611 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f274010dbc0 msgr2=0x7f2740114450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:35.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.422+0000 7f2746a00640 1 --2- 172.21.15.39:0/3703615611 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f274010dbc0 0x7f2740114450 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f2738009ca0 tx=0x7f273802c100 comp rx=0 tx=0).stop 2024-03-18T16:15:35.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.422+0000 7f2746a00640 1 -- 172.21.15.39:0/3703615611 shutdown_connections 2024-03-18T16:15:35.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.422+0000 7f2746a00640 1 --2- 172.21.15.39:0/3703615611 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f274010dbc0 0x7f2740114450 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:35.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.422+0000 7f2746a00640 1 --2- 172.21.15.39:0/3703615611 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f274010d200 0x7f274010d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:35.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.422+0000 7f2746a00640 1 --2- 172.21.15.39:0/3703615611 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f274010c010 0x7f274010c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:35.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.422+0000 7f2746a00640 1 -- 172.21.15.39:0/3703615611 >> 172.21.15.39:0/3703615611 conn(0x7f274006be90 msgr2=0x7f274006c2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:35.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.423+0000 7f2746a00640 1 -- 172.21.15.39:0/3703615611 shutdown_connections 2024-03-18T16:15:35.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.423+0000 7f2746a00640 1 -- 172.21.15.39:0/3703615611 wait complete. 2024-03-18T16:15:35.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.426+0000 7f2746a00640 1 Processor -- start 2024-03-18T16:15:35.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.426+0000 7f2746a00640 1 -- start start 2024-03-18T16:15:35.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.428+0000 7f2746a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f274010c010 0x7f27401a22a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:35.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.428+0000 7f2746a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f274010d200 0x7f27401a27e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:35.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.428+0000 7f2745600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f274010c010 0x7f27401a22a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:35.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.428+0000 7f2744c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f274010d200 0x7f27401a27e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:35.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.429+0000 7f2745600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f274010c010 0x7f27401a22a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:35634/0 (socket says 172.21.15.39:35634) 2024-03-18T16:15:35.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.429+0000 7f2746a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f274010dbc0 0x7f27401a9850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:35.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.429+0000 7f2746a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f2740117ca0 con 0x7f274010d200 2024-03-18T16:15:35.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.429+0000 7f2746a00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f2740117b20 con 0x7f274010dbc0 2024-03-18T16:15:35.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.429+0000 7f2746a00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f2740117e20 con 0x7f274010c010 2024-03-18T16:15:35.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.429+0000 7f2746000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f274010dbc0 0x7f27401a9850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:35.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.429+0000 7f2746000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f274010dbc0 0x7f27401a9850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:53498/0 (socket says 172.21.15.39:53498) 2024-03-18T16:15:35.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.429+0000 7f2746000640 1 -- 172.21.15.39:0/3751595083 learned_addr learned my addr 172.21.15.39:0/3751595083 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:15:35.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.430+0000 7f2746000640 1 -- 172.21.15.39:0/3751595083 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f274010c010 msgr2=0x7f27401a22a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:35.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.430+0000 7f2746000640 1 --2- 172.21.15.39:0/3751595083 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f274010c010 0x7f27401a22a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:35.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.430+0000 7f2746000640 1 -- 172.21.15.39:0/3751595083 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f274010d200 msgr2=0x7f27401a27e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:35.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.430+0000 7f2746000640 1 --2- 172.21.15.39:0/3751595083 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f274010d200 0x7f27401a27e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:35.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.430+0000 7f2746000640 1 -- 172.21.15.39:0/3751595083 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27401a9ec0 con 0x7f274010dbc0 2024-03-18T16:15:35.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.430+0000 7f2745600640 1 --2- 172.21.15.39:0/3751595083 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f274010c010 0x7f27401a22a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:15:35.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.430+0000 7f2744c00640 1 --2- 172.21.15.39:0/3751595083 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f274010d200 0x7f27401a27e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:15:35.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.430+0000 7f2746000640 1 --2- 172.21.15.39:0/3751595083 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f274010dbc0 0x7f27401a9850 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f273802c640 tx=0x7f2738035d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:35.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.433+0000 7f273e000640 1 -- 172.21.15.39:0/3751595083 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2738039040 con 0x7f274010dbc0 2024-03-18T16:15:35.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.433+0000 7f273e000640 1 -- 172.21.15.39:0/3751595083 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2738044070 con 0x7f274010dbc0 2024-03-18T16:15:35.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.433+0000 7f273e000640 1 -- 172.21.15.39:0/3751595083 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27380044a0 con 0x7f274010dbc0 2024-03-18T16:15:35.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.434+0000 7f2746a00640 1 -- 172.21.15.39:0/3751595083 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27401aa150 con 0x7f274010dbc0 2024-03-18T16:15:35.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.435+0000 7f2746a00640 1 -- 172.21.15.39:0/3751595083 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27401aa5c0 con 0x7f274010dbc0 2024-03-18T16:15:35.442 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.439+0000 7f274ac1c740 1 Processor -- start 2024-03-18T16:15:35.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.440+0000 7f274ac1c740 1 -- start start 2024-03-18T16:15:35.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.440+0000 7f273e000640 1 -- 172.21.15.39:0/3751595083 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f2738004640 con 0x7f274010dbc0 2024-03-18T16:15:35.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.441+0000 7f274ac1c740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5583a2155a30 0x5583a2155e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:35.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.441+0000 7f274ac1c740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5583a2156420 0x5583a215af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:35.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.441+0000 7f274ac1c740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5583a215b590 0x5583a215da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:35.443 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.441+0000 7f274ac1c740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5583a20ba1f0 con 0x5583a2155a30 2024-03-18T16:15:35.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.441+0000 7f274ac1c740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5583a2062a80 con 0x5583a2156420 2024-03-18T16:15:35.444 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.441+0000 7f274ac1c740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x5583a20b6290 con 0x5583a215b590 2024-03-18T16:15:35.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.443+0000 7f2744c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5583a2155a30 0x5583a2155e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:35.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.444+0000 7f2744c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5583a2155a30 0x5583a2155e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:49304/0 (socket says 172.21.15.39:49304) 2024-03-18T16:15:35.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.444+0000 7f2744c00640 1 -- 172.21.15.39:0/800882069 learned_addr learned my addr 172.21.15.39:0/800882069 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:15:35.447 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.444+0000 7f2745600640 1 --2- 172.21.15.39:0/800882069 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5583a2156420 0x5583a215af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:35.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.444+0000 7f2744c00640 1 --2- 172.21.15.39:0/800882069 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5583a215b590 0x5583a215da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:35.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.445+0000 7f2744c00640 1 -- 172.21.15.39:0/800882069 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5583a215b590 msgr2=0x5583a215da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:35.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.445+0000 7f2744c00640 1 --2- 172.21.15.39:0/800882069 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5583a215b590 0x5583a215da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:35.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.445+0000 7f2744c00640 1 -- 172.21.15.39:0/800882069 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5583a2156420 msgr2=0x5583a215af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:35.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.445+0000 7f2744c00640 1 --2- 172.21.15.39:0/800882069 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5583a2156420 0x5583a215af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:35.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.445+0000 7f2744c00640 1 -- 172.21.15.39:0/800882069 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5583a1e987d0 con 0x5583a2155a30 2024-03-18T16:15:35.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.446+0000 7f2745600640 1 --2- 172.21.15.39:0/800882069 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5583a2156420 0x5583a215af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:15:35.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.446+0000 7f273e000640 1 --2- 172.21.15.39:0/3751595083 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f27280a2900 0x7f27280a4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:35.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.446+0000 7f2744c00640 1 --2- 172.21.15.39:0/800882069 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5583a2155a30 0x5583a2155e50 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f2734014850 tx=0x7f273400b740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30efc3d356848b22 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:35.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.446+0000 7f273e000640 1 -- 172.21.15.39:0/3751595083 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f27380f1060 con 0x7f274010dbc0 2024-03-18T16:15:35.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.448+0000 7f2745600640 1 --2- 172.21.15.39:0/3751595083 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f27280a2900 0x7f27280a4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:35.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.448+0000 7f2732600640 1 -- 172.21.15.39:0/800882069 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f273400d940 con 0x5583a2155a30 2024-03-18T16:15:35.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.449+0000 7f2732600640 1 -- 172.21.15.39:0/800882069 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2734014300 con 0x5583a2155a30 2024-03-18T16:15:35.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.449+0000 7f2732600640 1 -- 172.21.15.39:0/800882069 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f273400fbc0 con 0x5583a2155a30 2024-03-18T16:15:35.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.450+0000 7f274ac1c740 1 -- 172.21.15.39:0/800882069 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5583a2155a30 msgr2=0x5583a2155e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:35.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.450+0000 7f274ac1c740 1 --2- 172.21.15.39:0/800882069 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5583a2155a30 0x5583a2155e50 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f2734014850 tx=0x7f273400b740 comp rx=0 tx=0).stop 2024-03-18T16:15:35.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.450+0000 7f274ac1c740 1 -- 172.21.15.39:0/800882069 shutdown_connections 2024-03-18T16:15:35.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.450+0000 7f274ac1c740 1 --2- 172.21.15.39:0/800882069 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5583a2156420 0x5583a215af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:35.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.450+0000 7f274ac1c740 1 --2- 172.21.15.39:0/800882069 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5583a215b590 0x5583a215da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:35.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.450+0000 7f274ac1c740 1 --2- 172.21.15.39:0/800882069 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5583a2155a30 0x5583a2155e50 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:35.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.450+0000 7f274ac1c740 1 -- 172.21.15.39:0/800882069 >> 172.21.15.39:0/800882069 conn(0x5583a2150fa0 msgr2=0x5583a21513a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:35.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.450+0000 7f274ac1c740 1 -- 172.21.15.39:0/800882069 shutdown_connections 2024-03-18T16:15:35.453 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.451+0000 7f2745600640 1 --2- 172.21.15.39:0/3751595083 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f27280a2900 0x7f27280a4dc0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x5583a2156820 tx=0x7f272c00b5e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:35.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.452+0000 7f274ac1c740 1 -- 172.21.15.39:0/800882069 wait complete. 2024-03-18T16:15:35.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.453+0000 7f274ac1c740 1 -- start start 2024-03-18T16:15:35.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.453+0000 7f274ac1c740 1 Processor -- start 2024-03-18T16:15:35.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.454+0000 7f274ac1c740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5583a2155a30 0x5583a21687e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:35.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.454+0000 7f274ac1c740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5583a2156420 0x5583a2168d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:35.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.454+0000 7f274ac1c740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5583a215b590 0x5583a216fd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:35.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.455+0000 7f2745600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5583a2155a30 0x5583a21687e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:35.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.455+0000 7f2745600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5583a2155a30 0x5583a21687e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:53526/0 (socket says 172.21.15.39:53526) 2024-03-18T16:15:35.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.455+0000 7f2746000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5583a215b590 0x5583a216fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:35.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.455+0000 7f2746000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5583a215b590 0x5583a216fd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:35660/0 (socket says 172.21.15.39:35660) 2024-03-18T16:15:35.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.456+0000 7f2745600640 1 -- 172.21.15.39:0/1020723221 learned_addr learned my addr 172.21.15.39:0/1020723221 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:15:35.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.456+0000 7f274ac1c740 1 -- 172.21.15.39:0/1020723221 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5583a215e540 con 0x5583a2156420 2024-03-18T16:15:35.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.456+0000 7f274ac1c740 1 -- 172.21.15.39:0/1020723221 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5583a2075ed0 con 0x5583a2155a30 2024-03-18T16:15:35.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.456+0000 7f274ac1c740 1 -- 172.21.15.39:0/1020723221 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x5583a215e6c0 con 0x5583a215b590 2024-03-18T16:15:35.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.457+0000 7f274ac1c740 10 client.0 _ll_register_callbacks cb 0x5583a20b1220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:15:35.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.457+0000 7f2746000640 1 -- 172.21.15.39:0/1020723221 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5583a2155a30 msgr2=0x5583a21687e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:35.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.457+0000 7f2746000640 1 --2- 172.21.15.39:0/1020723221 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5583a2155a30 0x5583a21687e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:35.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.457+0000 7f2746000640 1 -- 172.21.15.39:0/1020723221 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5583a2156420 msgr2=0x5583a2168d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:35.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.457+0000 7f2746000640 1 --2- 172.21.15.39:0/1020723221 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5583a2156420 0x5583a2168d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:35.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.457+0000 7f2746000640 1 -- 172.21.15.39:0/1020723221 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5583a1dce3f0 con 0x5583a215b590 2024-03-18T16:15:35.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.458+0000 7f2746000640 1 --2- 172.21.15.39:0/1020723221 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5583a215b590 0x5583a216fd60 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f2738006010 tx=0x7f27380bd4f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=891b8508b70a2623 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:35.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.458+0000 7f2726a00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:15:35.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.459+0000 7f2726a00640 1 -- 172.21.15.39:0/1020723221 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27380bdda0 con 0x5583a215b590 2024-03-18T16:15:35.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.459+0000 7f2726a00640 1 -- 172.21.15.39:0/1020723221 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f2738046310 con 0x5583a215b590 2024-03-18T16:15:35.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.459+0000 7f274ac1c740 1 -- 172.21.15.39:0/1020723221 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2734014300 con 0x5583a215b590 2024-03-18T16:15:35.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.460+0000 7f2726a00640 1 -- 172.21.15.39:0/1020723221 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f2738046530 con 0x5583a215b590 2024-03-18T16:15:35.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.460+0000 7f274ac1c740 10 client.26641 fetch_fsmap learned FSMap version 42 2024-03-18T16:15:35.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.461+0000 7f274ac1c740 1 -- 172.21.15.39:0/1020723221 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x5583a1d8e0f0 con 0x5583a215b590 2024-03-18T16:15:35.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.462+0000 7f2726a00640 1 -- 172.21.15.39:0/1020723221 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27380d5030 con 0x5583a215b590 2024-03-18T16:15:35.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.462+0000 7f2726a00640 1 -- 172.21.15.39:0/1020723221 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 42) ==== 8370+0+0 (secure 0 0 0) 0x7f273803c030 con 0x5583a215b590 2024-03-18T16:15:35.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.462+0000 7f274ac1c740 10 client.26641 fetch_fsmap finished waiting for FSMap version 42 2024-03-18T16:15:35.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.463+0000 7f274ac1c740 10 client.26641 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:15:35.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.463+0000 7f274ac1c740 20 client.26641 populate_metadata read hostname 'smithi039' 2024-03-18T16:15:35.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.463+0000 7f274ac1c740 1 --2- 172.21.15.39:0/1020723221 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5583a2174680 0x5583a2176af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:35.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.463+0000 7f274ac1c740 4 client.26641 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:15:35.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.463+0000 7f274ac1c740 1 -- 172.21.15.39:0/1020723221 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5583a20cdbe0 con 0x5583a2174680 2024-03-18T16:15:35.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.463+0000 7f2745600640 1 --2- 172.21.15.39:0/1020723221 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5583a2174680 0x5583a2176af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:35.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.464+0000 7f2745600640 1 --2- 172.21.15.39:0/1020723221 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5583a2174680 0x5583a2176af0 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=69152e6a36e15cac server_cookie=e8f24fe254ef7e68 in_seq=0 out_seq=0 2024-03-18T16:15:35.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.467+0000 7f2726a00640 10 client.26641 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:15:35.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.467+0000 7f2726a00640 1 -- 172.21.15.39:0/1020723221 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5583a20cdbe0 con 0x5583a2174680 2024-03-18T16:15:35.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.468+0000 7f2726a00640 10 client.26641 handle_command_reply: tid=0 2024-03-18T16:15:35.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.468+0000 7f274ac1c740 1 client.26641 shutdown 2024-03-18T16:15:35.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.469+0000 7f274ac1c740 1 -- 172.21.15.39:0/1020723221 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5583a215b590 msgr2=0x5583a216fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:35.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.469+0000 7f274ac1c740 1 --2- 172.21.15.39:0/1020723221 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5583a215b590 0x5583a216fd60 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f2738006010 tx=0x7f27380bd4f0 comp rx=0 tx=0).stop 2024-03-18T16:15:35.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.471+0000 7f274ac1c740 1 -- 172.21.15.39:0/1020723221 shutdown_connections 2024-03-18T16:15:35.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.471+0000 7f274ac1c740 1 --2- 172.21.15.39:0/1020723221 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5583a2174680 0x5583a2176af0 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:35.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.471+0000 7f274ac1c740 1 --2- 172.21.15.39:0/1020723221 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5583a215b590 0x5583a216fd60 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:35.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.471+0000 7f274ac1c740 1 --2- 172.21.15.39:0/1020723221 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5583a2156420 0x5583a2168d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:35.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.471+0000 7f274ac1c740 1 --2- 172.21.15.39:0/1020723221 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5583a2155a30 0x5583a21687e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:35.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.471+0000 7f274ac1c740 1 -- 172.21.15.39:0/1020723221 >> 172.21.15.39:0/1020723221 conn(0x5583a2150fa0 msgr2=0x5583a21501c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:35.474 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.471+0000 7f2726a00640 0 client.26641 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:15:35.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.471+0000 7f274ac1c740 1 -- 172.21.15.39:0/1020723221 shutdown_connections 2024-03-18T16:15:35.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.472+0000 7f274ac1c740 1 -- 172.21.15.39:0/1020723221 wait complete. 2024-03-18T16:15:35.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.472+0000 7f274ac1c740 20 client.26641 trim_cache size 0 max 16384 2024-03-18T16:15:35.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:35 smithi196 ceph-mon[36307]: pgmap v587: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 6.1 MiB/s wr, 1.37k op/s 2024-03-18T16:15:35.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:35 smithi194 ceph-mon[36200]: pgmap v587: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 6.1 MiB/s wr, 1.37k op/s 2024-03-18T16:15:35.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.510+0000 7f274ac1c740 1 Processor -- start 2024-03-18T16:15:35.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.511+0000 7f274ac1c740 1 -- start start 2024-03-18T16:15:35.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.511+0000 7f274ac1c740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5583a2155a30 0x5583a21687e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:35.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.511+0000 7f274ac1c740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5583a2156420 0x5583a2168d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:35.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.511+0000 7f274ac1c740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5583a215b590 0x5583a216fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:35.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.511+0000 7f274ac1c740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f27401b0830 con 0x5583a2156420 2024-03-18T16:15:35.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.511+0000 7f274ac1c740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f27401b2370 con 0x5583a2155a30 2024-03-18T16:15:35.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.511+0000 7f274ac1c740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f27401b0f00 con 0x5583a215b590 2024-03-18T16:15:35.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.512+0000 7f2745600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5583a2155a30 0x5583a21687e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:35.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.512+0000 7f2745600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5583a2155a30 0x5583a21687e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:53542/0 (socket says 172.21.15.39:53542) 2024-03-18T16:15:35.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.512+0000 7f2745600640 1 -- 172.21.15.39:0/1179154497 learned_addr learned my addr 172.21.15.39:0/1179154497 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:15:35.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.512+0000 7f2746000640 1 --2- 172.21.15.39:0/1179154497 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5583a215b590 0x5583a216fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:35.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.512+0000 7f2745600640 1 -- 172.21.15.39:0/1179154497 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5583a215b590 msgr2=0x5583a216fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:35.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.512+0000 7f2745600640 1 --2- 172.21.15.39:0/1179154497 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5583a215b590 0x5583a216fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:35.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.513+0000 7f2745600640 1 -- 172.21.15.39:0/1179154497 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5583a2156420 msgr2=0x5583a2168d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:35.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.513+0000 7f2745600640 1 --2- 172.21.15.39:0/1179154497 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5583a2156420 0x5583a2168d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:35.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.513+0000 7f2745600640 1 -- 172.21.15.39:0/1179154497 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5583a21515b0 con 0x5583a2155a30 2024-03-18T16:15:35.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.515+0000 7f2745600640 1 --2- 172.21.15.39:0/1179154497 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5583a2155a30 0x5583a21687e0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f272c010590 tx=0x7f272c015000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eda1ec7f0cd71f99 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:35.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.515+0000 7f2731c00640 1 -- 172.21.15.39:0/1179154497 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f272c02e070 con 0x5583a2155a30 2024-03-18T16:15:35.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.515+0000 7f2731c00640 1 -- 172.21.15.39:0/1179154497 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f272c0287b0 con 0x5583a2155a30 2024-03-18T16:15:35.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.516+0000 7f2731c00640 1 -- 172.21.15.39:0/1179154497 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f272c02e070 con 0x5583a2155a30 2024-03-18T16:15:35.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.516+0000 7f274ac1c740 1 -- 172.21.15.39:0/1179154497 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5583a2155a30 msgr2=0x5583a21687e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:35.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.516+0000 7f274ac1c740 1 --2- 172.21.15.39:0/1179154497 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5583a2155a30 0x5583a21687e0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f272c010590 tx=0x7f272c015000 comp rx=0 tx=0).stop 2024-03-18T16:15:35.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.517+0000 7f274ac1c740 1 -- 172.21.15.39:0/1179154497 shutdown_connections 2024-03-18T16:15:35.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.517+0000 7f274ac1c740 1 --2- 172.21.15.39:0/1179154497 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5583a215b590 0x5583a216fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:35.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.517+0000 7f274ac1c740 1 --2- 172.21.15.39:0/1179154497 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5583a2156420 0x5583a2168d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:35.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.517+0000 7f274ac1c740 1 --2- 172.21.15.39:0/1179154497 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5583a2155a30 0x5583a21687e0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:35.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.517+0000 7f274ac1c740 1 -- 172.21.15.39:0/1179154497 >> 172.21.15.39:0/1179154497 conn(0x5583a2150fa0 msgr2=0x5583a2156e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:35.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.517+0000 7f274ac1c740 1 -- 172.21.15.39:0/1179154497 shutdown_connections 2024-03-18T16:15:35.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.517+0000 7f274ac1c740 1 -- 172.21.15.39:0/1179154497 wait complete. 2024-03-18T16:15:35.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.518+0000 7f274ac1c740 1 -- start start 2024-03-18T16:15:35.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.518+0000 7f274ac1c740 1 Processor -- start 2024-03-18T16:15:35.522 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.519+0000 7f274ac1c740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5583a2155a30 0x5583a216d170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:35.522 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.519+0000 7f274ac1c740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5583a2156420 0x5583a216d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:35.522 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.519+0000 7f274ac1c740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5583a215b590 0x5583a216e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:35.522 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.519+0000 7f274ac1c740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f27401b1ca0 con 0x5583a215b590 2024-03-18T16:15:35.522 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.519+0000 7f274ac1c740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f27401b15d0 con 0x5583a2155a30 2024-03-18T16:15:35.522 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.519+0000 7f274ac1c740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f27401ae830 con 0x5583a2156420 2024-03-18T16:15:35.522 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.519+0000 7f274ac1c740 10 client.0 _ll_register_callbacks cb 0x7f2738032340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:15:35.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.520+0000 7f2745600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5583a2155a30 0x5583a216d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:35.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.520+0000 7f2745600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5583a2155a30 0x5583a216d170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:53556/0 (socket says 172.21.15.39:53556) 2024-03-18T16:15:35.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.520+0000 7f2745600640 1 -- 172.21.15.39:0/144641878 learned_addr learned my addr 172.21.15.39:0/144641878 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:15:35.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.520+0000 7f2744c00640 1 --2- 172.21.15.39:0/144641878 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5583a2156420 0x5583a216d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:35.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.520+0000 7f2746000640 1 --2- 172.21.15.39:0/144641878 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5583a215b590 0x5583a216e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:35.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.521+0000 7f2744c00640 1 -- 172.21.15.39:0/144641878 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5583a2155a30 msgr2=0x5583a216d170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:35.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.521+0000 7f2744c00640 1 --2- 172.21.15.39:0/144641878 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5583a2155a30 0x5583a216d170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:35.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.521+0000 7f2744c00640 1 -- 172.21.15.39:0/144641878 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5583a215b590 msgr2=0x5583a216e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:35.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.521+0000 7f2744c00640 1 --2- 172.21.15.39:0/144641878 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5583a215b590 0x5583a216e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:35.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.521+0000 7f2744c00640 1 -- 172.21.15.39:0/144641878 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5583a216ecb0 con 0x5583a2156420 2024-03-18T16:15:35.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.523+0000 7f2744c00640 1 --2- 172.21.15.39:0/144641878 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5583a2156420 0x5583a216d6b0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x5583a21713b0 tx=0x7f273400db80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c604bc13a9dbf8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:35.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.523+0000 7f2726a00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:15:35.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.525+0000 7f2726a00640 1 -- 172.21.15.39:0/144641878 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2734024020 con 0x5583a2156420 2024-03-18T16:15:35.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.525+0000 7f2726a00640 1 -- 172.21.15.39:0/144641878 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f2734012450 con 0x5583a2156420 2024-03-18T16:15:35.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.526+0000 7f274ac1c740 1 -- 172.21.15.39:0/144641878 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f272c0287b0 con 0x5583a2156420 2024-03-18T16:15:35.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.526+0000 7f2726a00640 1 -- 172.21.15.39:0/144641878 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f2734012650 con 0x5583a2156420 2024-03-18T16:15:35.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.526+0000 7f274ac1c740 10 client.26647 fetch_fsmap learned FSMap version 42 2024-03-18T16:15:35.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.527+0000 7f274ac1c740 1 -- 172.21.15.39:0/144641878 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x5583a216ffc0 con 0x5583a2156420 2024-03-18T16:15:35.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.528+0000 7f2726a00640 1 -- 172.21.15.39:0/144641878 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f272c0287b0 con 0x5583a2156420 2024-03-18T16:15:35.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.528+0000 7f2726a00640 1 -- 172.21.15.39:0/144641878 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 42) ==== 8370+0+0 (secure 0 0 0) 0x7f273400f6b0 con 0x5583a2156420 2024-03-18T16:15:35.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.530+0000 7f274ac1c740 10 client.26647 fetch_fsmap finished waiting for FSMap version 42 2024-03-18T16:15:35.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.530+0000 7f274ac1c740 10 client.26647 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:15:35.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.530+0000 7f274ac1c740 20 client.26647 populate_metadata read hostname 'smithi039' 2024-03-18T16:15:35.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.530+0000 7f274ac1c740 1 --2- 172.21.15.39:0/144641878 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5583a21dac10 0x5583a21dd000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:35.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.530+0000 7f274ac1c740 4 client.26647 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T16:15:35.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.530+0000 7f274ac1c740 1 -- 172.21.15.39:0/144641878 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5583a20c7730 con 0x5583a21dac10 2024-03-18T16:15:35.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.530+0000 7f2745600640 1 --2- 172.21.15.39:0/144641878 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5583a21dac10 0x5583a21dd000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:35.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.531+0000 7f2745600640 1 --2- 172.21.15.39:0/144641878 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5583a21dac10 0x5583a21dd000 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=870cc1ff10f7e6c8 server_cookie=6195c62659b26553 in_seq=0 out_seq=0 2024-03-18T16:15:35.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.532+0000 7f2726a00640 10 client.26647 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:15:35.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.532+0000 7f2726a00640 1 -- 172.21.15.39:0/144641878 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5583a20c7730 con 0x5583a21dac10 2024-03-18T16:15:35.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.532+0000 7f2726a00640 10 client.26647 handle_command_reply: tid=0 2024-03-18T16:15:35.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.532+0000 7f274ac1c740 1 client.26647 shutdown 2024-03-18T16:15:35.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.533+0000 7f274ac1c740 1 -- 172.21.15.39:0/144641878 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5583a2156420 msgr2=0x5583a216d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:35.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.533+0000 7f274ac1c740 1 --2- 172.21.15.39:0/144641878 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5583a2156420 0x5583a216d6b0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x5583a21713b0 tx=0x7f273400db80 comp rx=0 tx=0).stop 2024-03-18T16:15:35.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.534+0000 7f274ac1c740 1 -- 172.21.15.39:0/144641878 shutdown_connections 2024-03-18T16:15:35.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.534+0000 7f274ac1c740 1 --2- 172.21.15.39:0/144641878 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5583a21dac10 0x5583a21dd000 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:35.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.534+0000 7f274ac1c740 1 --2- 172.21.15.39:0/144641878 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5583a215b590 0x5583a216e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:35.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.534+0000 7f2726a00640 0 client.26647 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:15:35.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.534+0000 7f274ac1c740 1 --2- 172.21.15.39:0/144641878 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5583a2156420 0x5583a216d6b0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:35.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.534+0000 7f274ac1c740 1 --2- 172.21.15.39:0/144641878 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5583a2155a30 0x5583a216d170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:35.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.534+0000 7f274ac1c740 1 -- 172.21.15.39:0/144641878 >> 172.21.15.39:0/144641878 conn(0x5583a2150fa0 msgr2=0x5583a2168740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:35.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.534+0000 7f274ac1c740 1 -- 172.21.15.39:0/144641878 shutdown_connections 2024-03-18T16:15:35.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.534+0000 7f274ac1c740 1 -- 172.21.15.39:0/144641878 wait complete. 2024-03-18T16:15:35.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.535+0000 7f274ac1c740 20 client.26647 trim_cache size 0 max 16384 2024-03-18T16:15:35.539 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-03-18T16:15:35.539 INFO:teuthology.orchestra.run.smithi039.stdout: "status": "no active scrubs running", 2024-03-18T16:15:35.539 INFO:teuthology.orchestra.run.smithi039.stdout: "scrubs": {} 2024-03-18T16:15:35.539 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-03-18T16:15:35.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.537+0000 7f2746a00640 1 -- 172.21.15.39:0/3751595083 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f27280a2900 msgr2=0x7f27280a4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:35.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.537+0000 7f2746a00640 1 --2- 172.21.15.39:0/3751595083 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f27280a2900 0x7f27280a4dc0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x5583a2156820 tx=0x7f272c00b5e0 comp rx=0 tx=0).stop 2024-03-18T16:15:35.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.537+0000 7f2746a00640 1 -- 172.21.15.39:0/3751595083 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f274010dbc0 msgr2=0x7f27401a9850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:35.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.537+0000 7f2746a00640 1 --2- 172.21.15.39:0/3751595083 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f274010dbc0 0x7f27401a9850 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f273802c640 tx=0x7f2738035d10 comp rx=0 tx=0).stop 2024-03-18T16:15:35.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.537+0000 7f2746a00640 1 -- 172.21.15.39:0/3751595083 shutdown_connections 2024-03-18T16:15:35.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.537+0000 7f2746a00640 1 --2- 172.21.15.39:0/3751595083 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f274010dbc0 0x7f27401a9850 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:35.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.537+0000 7f2746a00640 1 --2- 172.21.15.39:0/3751595083 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f27280a2900 0x7f27280a4dc0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:35.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.537+0000 7f2746a00640 1 --2- 172.21.15.39:0/3751595083 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f274010d200 0x7f27401a27e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:35.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.537+0000 7f2746a00640 1 --2- 172.21.15.39:0/3751595083 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f274010c010 0x7f27401a22a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:35.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.537+0000 7f2746a00640 1 -- 172.21.15.39:0/3751595083 >> 172.21.15.39:0/3751595083 conn(0x7f274006be90 msgr2=0x7f27401104a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:35.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.538+0000 7f2746a00640 1 -- 172.21.15.39:0/3751595083 shutdown_connections 2024-03-18T16:15:35.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.538+0000 7f2746a00640 1 -- 172.21.15.39:0/3751595083 wait complete. 2024-03-18T16:15:35.564 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:35 smithi039 ceph-mon[31721]: pgmap v587: 129 pgs: 129 active+clean; 13 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 6.1 MiB/s wr, 1.37k op/s 2024-03-18T16:15:35.567 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-18T16:15:35.567 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:15:35.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.904+0000 7fe123000640 1 -- 172.21.15.39:0/3590812947 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe11c1072d0 msgr2=0x7fe11c109720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:35.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.904+0000 7fe123000640 1 --2- 172.21.15.39:0/3590812947 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe11c1072d0 0x7fe11c109720 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fe114009900 tx=0x7fe11402bc30 comp rx=0 tx=0).stop 2024-03-18T16:15:35.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.904+0000 7fe123000640 1 -- 172.21.15.39:0/3590812947 shutdown_connections 2024-03-18T16:15:35.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.904+0000 7fe123000640 1 --2- 172.21.15.39:0/3590812947 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe11c1072d0 0x7fe11c109720 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:35.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.904+0000 7fe123000640 1 --2- 172.21.15.39:0/3590812947 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe11c1049a0 0x7fe11c106d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:35.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.904+0000 7fe123000640 1 --2- 172.21.15.39:0/3590812947 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe11c102070 0x7fe11c104460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:35.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.904+0000 7fe123000640 1 -- 172.21.15.39:0/3590812947 >> 172.21.15.39:0/3590812947 conn(0x7fe11c0fba60 msgr2=0x7fe11c0fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:35.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.904+0000 7fe123000640 1 -- 172.21.15.39:0/3590812947 shutdown_connections 2024-03-18T16:15:35.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.904+0000 7fe123000640 1 -- 172.21.15.39:0/3590812947 wait complete. 2024-03-18T16:15:35.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.906+0000 7fe123000640 1 Processor -- start 2024-03-18T16:15:35.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.906+0000 7fe123000640 1 -- start start 2024-03-18T16:15:35.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.906+0000 7fe123000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe11c102070 0x7fe11c197840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:35.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.907+0000 7fe123000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe11c1049a0 0x7fe11c197d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:35.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.907+0000 7fe123000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe11c1072d0 0x7fe11c19edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:35.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.907+0000 7fe123000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fe11c10d260 con 0x7fe11c102070 2024-03-18T16:15:35.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.907+0000 7fe123000640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fe11c10d0e0 con 0x7fe11c1049a0 2024-03-18T16:15:35.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.907+0000 7fe123000640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fe11c10d3e0 con 0x7fe11c1072d0 2024-03-18T16:15:35.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.907+0000 7fe122600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe11c1072d0 0x7fe11c19edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:35.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.907+0000 7fe122600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe11c1072d0 0x7fe11c19edf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:35712/0 (socket says 172.21.15.39:35712) 2024-03-18T16:15:35.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.907+0000 7fe122600640 1 -- 172.21.15.39:0/914724837 learned_addr learned my addr 172.21.15.39:0/914724837 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:15:35.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.907+0000 7fe121c00640 1 --2- 172.21.15.39:0/914724837 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe11c102070 0x7fe11c197840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:35.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.908+0000 7fe121200640 1 --2- 172.21.15.39:0/914724837 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe11c1049a0 0x7fe11c197d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:35.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.908+0000 7fe122600640 1 -- 172.21.15.39:0/914724837 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe11c1049a0 msgr2=0x7fe11c197d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:35.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.908+0000 7fe122600640 1 --2- 172.21.15.39:0/914724837 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe11c1049a0 0x7fe11c197d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:35.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.908+0000 7fe122600640 1 -- 172.21.15.39:0/914724837 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe11c102070 msgr2=0x7fe11c197840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:35.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.908+0000 7fe122600640 1 --2- 172.21.15.39:0/914724837 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe11c102070 0x7fe11c197840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:35.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.908+0000 7fe122600640 1 -- 172.21.15.39:0/914724837 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe11c19f460 con 0x7fe11c1072d0 2024-03-18T16:15:35.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.909+0000 7fe122600640 1 --2- 172.21.15.39:0/914724837 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe11c1072d0 0x7fe11c19edf0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fe114036040 tx=0x7fe114004e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:35.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.909+0000 7fe11a000640 1 -- 172.21.15.39:0/914724837 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe114043020 con 0x7fe11c1072d0 2024-03-18T16:15:35.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.909+0000 7fe11a000640 1 -- 172.21.15.39:0/914724837 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe1140054e0 con 0x7fe11c1072d0 2024-03-18T16:15:35.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.909+0000 7fe11a000640 1 -- 172.21.15.39:0/914724837 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe114003c50 con 0x7fe11c1072d0 2024-03-18T16:15:35.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.909+0000 7fe123000640 1 -- 172.21.15.39:0/914724837 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe11c19f6f0 con 0x7fe11c1072d0 2024-03-18T16:15:35.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.910+0000 7fe123000640 1 -- 172.21.15.39:0/914724837 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe11c19fc40 con 0x7fe11c1072d0 2024-03-18T16:15:35.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.911+0000 7fe123000640 1 -- 172.21.15.39:0/914724837 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe11c0fd550 con 0x7fe11c1072d0 2024-03-18T16:15:35.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.915+0000 7fe11a000640 1 -- 172.21.15.39:0/914724837 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fe114046020 con 0x7fe11c1072d0 2024-03-18T16:15:35.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.920+0000 7fe11a000640 1 --2- 172.21.15.39:0/914724837 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fe1040a2980 0x7fe1040a4e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:35.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.921+0000 7fe11a000640 1 -- 172.21.15.39:0/914724837 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fe1140f0e40 con 0x7fe11c1072d0 2024-03-18T16:15:35.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.922+0000 7fe11a000640 1 -- 172.21.15.39:0/914724837 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fe114072030 con 0x7fe11c1072d0 2024-03-18T16:15:35.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.924+0000 7fe121c00640 1 --2- 172.21.15.39:0/914724837 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fe1040a2980 0x7fe1040a4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:35.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:35.925+0000 7fe121c00640 1 --2- 172.21.15.39:0/914724837 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fe1040a2980 0x7fe1040a4e40 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fe110005fd0 tx=0x7fe110005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:36.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.418+0000 7fe123000640 1 -- 172.21.15.39:0/914724837 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe11c02d1b0 con 0x7fe11c1072d0 2024-03-18T16:15:36.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.420+0000 7fe11a000640 1 -- 172.21.15.39:0/914724837 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) ==== 94+0+11935 (secure 0 0 0) 0x7fe1140bdcf0 con 0x7fe11c1072d0 2024-03-18T16:15:36.424 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:15:36.424 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2520226608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2520226608},{"type":"v1","addr":"172.21.15.39:6835","nonce":2520226608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/719341763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":719341763},{"type":"v1","addr":"172.21.15.39:6837","nonce":719341763}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":42,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T16:06:44.180425+0000","modified":"2024-03-18T16:15:17.441015+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24575,"mds_1":24556,"mds_2":24535,"mds_3":24524,"mds_4":14463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14445":{"gid":14445,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.39:6839/656513813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":656513813},{"type":"v1","addr":"172.21.15.39:6839","nonce":656513813}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14463":{"gid":14463,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":13,"addr":"172.21.15.39:6841/3515326042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3515326042},{"type":"v1","addr":"172.21.15.39:6841","nonce":3515326042}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6833/931350738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":931350738},{"type":"v1","addr":"172.21.15.194:6833","nonce":931350738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6833/3874209621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":3874209621},{"type":"v1","addr":"172.21.15.196:6833","nonce":3874209621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6835/188385470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":188385470},{"type":"v1","addr":"172.21.15.194:6835","nonce":188385470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6835/3949984309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":3949984309},{"type":"v1","addr":"172.21.15.196:6835","nonce":3949984309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":15,"addr":"172.21.15.194:6837/1301464246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":1301464246},{"type":"v1","addr":"172.21.15.194:6837","nonce":1301464246}]},"join_fscid":-1,"export_targets":[1,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":16,"addr":"172.21.15.196:6837/895247251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":895247251},{"type":"v1","addr":"172.21.15.196:6837","nonce":895247251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.196:6839/2001427268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6838","nonce":2001427268},{"type":"v1","addr":"172.21.15.196:6839","nonce":2001427268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24575":{"gid":24575,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.194:6839/274911158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":274911158},{"type":"v1","addr":"172.21.15.194:6839","nonce":274911158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24575,"qdb_cluster":[24524,14463,24575,24535,24556]},"id":1}]} 2024-03-18T16:15:36.425 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 42 2024-03-18T16:15:36.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.424+0000 7fe123000640 1 -- 172.21.15.39:0/914724837 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fe1040a2980 msgr2=0x7fe1040a4e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:36.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.424+0000 7fe123000640 1 --2- 172.21.15.39:0/914724837 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fe1040a2980 0x7fe1040a4e40 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fe110005fd0 tx=0x7fe110005850 comp rx=0 tx=0).stop 2024-03-18T16:15:36.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.425+0000 7fe123000640 1 -- 172.21.15.39:0/914724837 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe11c1072d0 msgr2=0x7fe11c19edf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:36.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.425+0000 7fe123000640 1 --2- 172.21.15.39:0/914724837 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe11c1072d0 0x7fe11c19edf0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fe114036040 tx=0x7fe114004e30 comp rx=0 tx=0).stop 2024-03-18T16:15:36.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.425+0000 7fe123000640 1 -- 172.21.15.39:0/914724837 shutdown_connections 2024-03-18T16:15:36.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.425+0000 7fe123000640 1 --2- 172.21.15.39:0/914724837 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe11c1049a0 0x7fe11c197d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:36.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.425+0000 7fe123000640 1 --2- 172.21.15.39:0/914724837 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fe1040a2980 0x7fe1040a4e40 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:36.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.425+0000 7fe123000640 1 --2- 172.21.15.39:0/914724837 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe11c1072d0 0x7fe11c19edf0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:36.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.425+0000 7fe123000640 1 --2- 172.21.15.39:0/914724837 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe11c102070 0x7fe11c197840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:36.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.425+0000 7fe123000640 1 -- 172.21.15.39:0/914724837 >> 172.21.15.39:0/914724837 conn(0x7fe11c0fba60 msgr2=0x7fe11c1029b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:36.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.425+0000 7fe123000640 1 -- 172.21.15.39:0/914724837 shutdown_connections 2024-03-18T16:15:36.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.425+0000 7fe123000640 1 -- 172.21.15.39:0/914724837 wait complete. 2024-03-18T16:15:36.457 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-18T16:15:36.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.783+0000 7fcd78e00640 1 -- 172.21.15.39:0/442211758 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcd74109930 msgr2=0x7fcd74110250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:36.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.783+0000 7fcd78e00640 1 --2- 172.21.15.39:0/442211758 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcd74109930 0x7fcd74110250 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fcd6c009be0 tx=0x7fcd6c02bef0 comp rx=0 tx=0).stop 2024-03-18T16:15:36.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.783+0000 7fcd78e00640 1 -- 172.21.15.39:0/442211758 shutdown_connections 2024-03-18T16:15:36.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.783+0000 7fcd78e00640 1 --2- 172.21.15.39:0/442211758 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcd74109930 0x7fcd74110250 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:36.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.783+0000 7fcd78e00640 1 --2- 172.21.15.39:0/442211758 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcd74108f70 0x7fcd741093f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:36.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.783+0000 7fcd78e00640 1 --2- 172.21.15.39:0/442211758 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcd74107e20 0x7fcd74108240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:36.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.783+0000 7fcd78e00640 1 -- 172.21.15.39:0/442211758 >> 172.21.15.39:0/442211758 conn(0x7fcd74076dd0 msgr2=0x7fcd740791f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:36.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.784+0000 7fcd78e00640 1 -- 172.21.15.39:0/442211758 shutdown_connections 2024-03-18T16:15:36.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.785+0000 7fcd78e00640 1 -- 172.21.15.39:0/442211758 wait complete. 2024-03-18T16:15:36.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.787+0000 7fcd78e00640 1 Processor -- start 2024-03-18T16:15:36.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.789+0000 7fcd78e00640 1 -- start start 2024-03-18T16:15:36.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.789+0000 7fcd78e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcd74107e20 0x7fcd7407a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:36.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.789+0000 7fcd78e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcd74108f70 0x7fcd7407ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:36.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.789+0000 7fcd78e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcd74109930 0x7fcd74081e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:36.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.789+0000 7fcd78e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fcd74113aa0 con 0x7fcd74108f70 2024-03-18T16:15:36.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.789+0000 7fcd78e00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fcd74113920 con 0x7fcd74109930 2024-03-18T16:15:36.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.789+0000 7fcd78e00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fcd74113c20 con 0x7fcd74107e20 2024-03-18T16:15:36.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.790+0000 7fcd72800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcd74109930 0x7fcd74081e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:36.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.790+0000 7fcd72800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcd74109930 0x7fcd74081e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:53568/0 (socket says 172.21.15.39:53568) 2024-03-18T16:15:36.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.790+0000 7fcd72800640 1 -- 172.21.15.39:0/175179189 learned_addr learned my addr 172.21.15.39:0/175179189 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:15:36.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.790+0000 7fcd71e00640 1 --2- 172.21.15.39:0/175179189 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcd74107e20 0x7fcd7407a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:36.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.790+0000 7fcd71400640 1 --2- 172.21.15.39:0/175179189 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcd74108f70 0x7fcd7407ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:36.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.790+0000 7fcd72800640 1 -- 172.21.15.39:0/175179189 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcd74107e20 msgr2=0x7fcd7407a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:36.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.790+0000 7fcd72800640 1 --2- 172.21.15.39:0/175179189 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcd74107e20 0x7fcd7407a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:36.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.790+0000 7fcd72800640 1 -- 172.21.15.39:0/175179189 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcd74108f70 msgr2=0x7fcd7407ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:36.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.790+0000 7fcd72800640 1 --2- 172.21.15.39:0/175179189 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcd74108f70 0x7fcd7407ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:36.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.790+0000 7fcd72800640 1 -- 172.21.15.39:0/175179189 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd74082480 con 0x7fcd74109930 2024-03-18T16:15:36.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.790+0000 7fcd71400640 1 --2- 172.21.15.39:0/175179189 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcd74108f70 0x7fcd7407ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:15:36.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.790+0000 7fcd71e00640 1 --2- 172.21.15.39:0/175179189 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcd74107e20 0x7fcd7407a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:15:36.795 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.791+0000 7fcd72800640 1 --2- 172.21.15.39:0/175179189 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcd74109930 0x7fcd74081e10 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fcd6c036040 tx=0x7fcd6c0337b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:36.795 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.791+0000 7fcd6aa00640 1 -- 172.21.15.39:0/175179189 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd6c043020 con 0x7fcd74109930 2024-03-18T16:15:36.795 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.792+0000 7fcd6aa00640 1 -- 172.21.15.39:0/175179189 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcd6c0338a0 con 0x7fcd74109930 2024-03-18T16:15:36.795 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.792+0000 7fcd78e00640 1 -- 172.21.15.39:0/175179189 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd741ae830 con 0x7fcd74109930 2024-03-18T16:15:36.795 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.792+0000 7fcd6aa00640 1 -- 172.21.15.39:0/175179189 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd6c034690 con 0x7fcd74109930 2024-03-18T16:15:36.795 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.792+0000 7fcd78e00640 1 -- 172.21.15.39:0/175179189 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd741aeca0 con 0x7fcd74109930 2024-03-18T16:15:36.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.800+0000 7fcd7bb9e740 1 Processor -- start 2024-03-18T16:15:36.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.800+0000 7fcd6aa00640 1 -- 172.21.15.39:0/175179189 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fcd6c046020 con 0x7fcd74109930 2024-03-18T16:15:36.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.800+0000 7fcd7bb9e740 1 -- start start 2024-03-18T16:15:36.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.801+0000 7fcd7bb9e740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560841e638c0 0x560841e63ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:36.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.801+0000 7fcd7bb9e740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560841e642b0 0x560841e68db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:36.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.801+0000 7fcd7bb9e740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560841e69420 0x560841e6b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:36.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.801+0000 7fcd7bb9e740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x560841e2b3b0 con 0x560841e69420 2024-03-18T16:15:36.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.801+0000 7fcd7bb9e740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x560841dc8450 con 0x560841e642b0 2024-03-18T16:15:36.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.801+0000 7fcd7bb9e740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x560841d70cb0 con 0x560841e638c0 2024-03-18T16:15:36.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.801+0000 7fcd71400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560841e69420 0x560841e6b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:36.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.802+0000 7fcd71400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560841e69420 0x560841e6b8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:49386/0 (socket says 172.21.15.39:49386) 2024-03-18T16:15:36.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.802+0000 7fcd71400640 1 -- 172.21.15.39:0/1488123975 learned_addr learned my addr 172.21.15.39:0/1488123975 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:15:36.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.802+0000 7fcd71400640 1 --2- 172.21.15.39:0/1488123975 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560841e638c0 0x560841e63ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:36.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.802+0000 7fcd6aa00640 1 --2- 172.21.15.39:0/175179189 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fcd580a2950 0x7fcd580a4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:36.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.802+0000 7fcd71400640 1 -- 172.21.15.39:0/1488123975 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560841e638c0 msgr2=0x560841e63ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:36.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.802+0000 7fcd71400640 1 --2- 172.21.15.39:0/1488123975 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560841e638c0 0x560841e63ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:36.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.802+0000 7fcd71400640 1 -- 172.21.15.39:0/1488123975 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560841e642b0 msgr2=0x560841e68db0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:15:36.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.802+0000 7fcd71400640 1 --2- 172.21.15.39:0/1488123975 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560841e642b0 0x560841e68db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:36.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.802+0000 7fcd71400640 1 -- 172.21.15.39:0/1488123975 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560841d921f0 con 0x560841e69420 2024-03-18T16:15:36.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.803+0000 7fcd6aa00640 1 -- 172.21.15.39:0/175179189 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fcd6c0f1310 con 0x7fcd74109930 2024-03-18T16:15:36.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.803+0000 7fcd71400640 1 --2- 172.21.15.39:0/1488123975 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560841e69420 0x560841e6b8b0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fcd64009460 tx=0x7fcd640095f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a65b18f8b4314e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:36.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.803+0000 7fcd72800640 1 --2- 172.21.15.39:0/175179189 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fcd580a2950 0x7fcd580a4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:36.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.803+0000 7fcd5f400640 1 -- 172.21.15.39:0/1488123975 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd640152f0 con 0x560841e69420 2024-03-18T16:15:36.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.803+0000 7fcd5f400640 1 -- 172.21.15.39:0/1488123975 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcd64005500 con 0x560841e69420 2024-03-18T16:15:36.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.803+0000 7fcd5f400640 1 -- 172.21.15.39:0/1488123975 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd6400e950 con 0x560841e69420 2024-03-18T16:15:36.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.806+0000 7fcd7bb9e740 1 -- 172.21.15.39:0/1488123975 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560841e69420 msgr2=0x560841e6b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:36.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.806+0000 7fcd7bb9e740 1 --2- 172.21.15.39:0/1488123975 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560841e69420 0x560841e6b8b0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fcd64009460 tx=0x7fcd640095f0 comp rx=0 tx=0).stop 2024-03-18T16:15:36.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.806+0000 7fcd72800640 1 --2- 172.21.15.39:0/175179189 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fcd580a2950 0x7fcd580a4e10 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fcd6c033ff0 tx=0x7fcd6c004620 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:36.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.807+0000 7fcd7bb9e740 1 -- 172.21.15.39:0/1488123975 shutdown_connections 2024-03-18T16:15:36.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.807+0000 7fcd7bb9e740 1 --2- 172.21.15.39:0/1488123975 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560841e642b0 0x560841e68db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:36.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.807+0000 7fcd7bb9e740 1 --2- 172.21.15.39:0/1488123975 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560841e69420 0x560841e6b8b0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:36.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.807+0000 7fcd7bb9e740 1 --2- 172.21.15.39:0/1488123975 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560841e638c0 0x560841e63ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:36.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.807+0000 7fcd7bb9e740 1 -- 172.21.15.39:0/1488123975 >> 172.21.15.39:0/1488123975 conn(0x560841e5ee30 msgr2=0x560841e5f230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:36.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.807+0000 7fcd7bb9e740 1 -- 172.21.15.39:0/1488123975 shutdown_connections 2024-03-18T16:15:36.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.807+0000 7fcd7bb9e740 1 -- 172.21.15.39:0/1488123975 wait complete. 2024-03-18T16:15:36.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.808+0000 7fcd7bb9e740 1 -- start start 2024-03-18T16:15:36.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.808+0000 7fcd7bb9e740 1 Processor -- start 2024-03-18T16:15:36.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.808+0000 7fcd7bb9e740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560841e638c0 0x560841e6db80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:36.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.809+0000 7fcd7bb9e740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560841e642b0 0x560841e6e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:36.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.809+0000 7fcd7bb9e740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560841e69420 0x560841e75100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:36.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.809+0000 7fcd7bb9e740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x560841d84110 con 0x560841e642b0 2024-03-18T16:15:36.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.809+0000 7fcd7bb9e740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x560841dc44f0 con 0x560841e638c0 2024-03-18T16:15:36.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.809+0000 7fcd7bb9e740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x560841e6c370 con 0x560841e69420 2024-03-18T16:15:36.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.809+0000 7fcd7bb9e740 10 client.0 _ll_register_callbacks cb 0x560841d71b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:15:36.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.809+0000 7fcd71e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560841e642b0 0x560841e6e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:36.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.809+0000 7fcd71400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560841e638c0 0x560841e6db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:36.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.809+0000 7fcd71400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560841e638c0 0x560841e6db80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:53576/0 (socket says 172.21.15.39:53576) 2024-03-18T16:15:36.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.809+0000 7fcd71e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560841e642b0 0x560841e6e0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:49400/0 (socket says 172.21.15.39:49400) 2024-03-18T16:15:36.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.809+0000 7fcd71400640 1 -- 172.21.15.39:0/2329628769 learned_addr learned my addr 172.21.15.39:0/2329628769 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:15:36.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.810+0000 7fcd71400640 1 --2- 172.21.15.39:0/2329628769 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560841e69420 0x560841e75100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:36.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.810+0000 7fcd71e00640 1 -- 172.21.15.39:0/2329628769 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560841e69420 msgr2=0x560841e75100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:36.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.810+0000 7fcd71e00640 1 --2- 172.21.15.39:0/2329628769 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560841e69420 0x560841e75100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:36.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.810+0000 7fcd71e00640 1 -- 172.21.15.39:0/2329628769 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560841e638c0 msgr2=0x560841e6db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:36.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.810+0000 7fcd71e00640 1 --2- 172.21.15.39:0/2329628769 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560841e638c0 0x560841e6db80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:36.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.810+0000 7fcd71e00640 1 -- 172.21.15.39:0/2329628769 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560841adc3f0 con 0x560841e642b0 2024-03-18T16:15:36.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.812+0000 7fcd71e00640 1 --2- 172.21.15.39:0/2329628769 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560841e642b0 0x560841e6e0c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x560841e646b0 tx=0x7fcd60011af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd7b5f9187953cd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:36.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.812+0000 7fcd53e00640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-03-18T16:15:36.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.813+0000 7fcd53e00640 1 -- 172.21.15.39:0/2329628769 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd60012920 con 0x560841e642b0 2024-03-18T16:15:36.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.813+0000 7fcd7bb9e740 1 -- 172.21.15.39:0/2329628769 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcd64005500 con 0x560841e642b0 2024-03-18T16:15:36.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.813+0000 7fcd53e00640 1 -- 172.21.15.39:0/2329628769 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fcd60012ac0 con 0x560841e642b0 2024-03-18T16:15:36.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.814+0000 7fcd53e00640 1 -- 172.21.15.39:0/2329628769 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7fcd60025050 con 0x560841e642b0 2024-03-18T16:15:36.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.814+0000 7fcd7bb9e740 10 client.16266 fetch_fsmap learned FSMap version 42 2024-03-18T16:15:36.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.814+0000 7fcd7bb9e740 1 -- 172.21.15.39:0/2329628769 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x560841ba67d0 con 0x560841e642b0 2024-03-18T16:15:36.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.814+0000 7fcd53e00640 1 -- 172.21.15.39:0/2329628769 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcd64005500 con 0x560841e642b0 2024-03-18T16:15:36.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.817+0000 7fcd53e00640 1 -- 172.21.15.39:0/2329628769 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 42) ==== 8370+0+0 (secure 0 0 0) 0x7fcd6001a3f0 con 0x560841e642b0 2024-03-18T16:15:36.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.817+0000 7fcd7bb9e740 10 client.16266 fetch_fsmap finished waiting for FSMap version 42 2024-03-18T16:15:36.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.817+0000 7fcd7bb9e740 10 client.16266 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-03-18T16:15:36.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.817+0000 7fcd7bb9e740 20 client.16266 populate_metadata read hostname 'smithi039' 2024-03-18T16:15:36.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.817+0000 7fcd7bb9e740 1 --2- 172.21.15.39:0/2329628769 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x560841e79a30 0x560841e7be60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:36.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.817+0000 7fcd7bb9e740 4 client.16266 mds_command: new command op to 14463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:15:36.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.817+0000 7fcd7bb9e740 1 -- 172.21.15.39:0/2329628769 --> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560841dd5990 con 0x560841e79a30 2024-03-18T16:15:36.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.819+0000 7fcd71400640 1 --2- 172.21.15.39:0/2329628769 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x560841e79a30 0x560841e7be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:36.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.820+0000 7fcd71400640 1 --2- 172.21.15.39:0/2329628769 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x560841e79a30 0x560841e7be60 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=44dd5b67adc3afa7 server_cookie=c6069eab211eba0b in_seq=0 out_seq=0 2024-03-18T16:15:36.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.820+0000 7fcd53e00640 10 client.16266 ms_handle_connect on v2:172.21.15.39:6840/3515326042 2024-03-18T16:15:36.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.821+0000 7fcd53e00640 1 -- 172.21.15.39:0/2329628769 <== mds.4 v2:172.21.15.39:6840/3515326042 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560841dd5990 con 0x560841e79a30 2024-03-18T16:15:36.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.821+0000 7fcd53e00640 10 client.16266 handle_command_reply: tid=0 2024-03-18T16:15:36.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.821+0000 7fcd7bb9e740 1 client.16266 shutdown 2024-03-18T16:15:36.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.823+0000 7fcd7bb9e740 1 -- 172.21.15.39:0/2329628769 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560841e642b0 msgr2=0x560841e6e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:36.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.823+0000 7fcd7bb9e740 1 --2- 172.21.15.39:0/2329628769 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560841e642b0 0x560841e6e0c0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x560841e646b0 tx=0x7fcd60011af0 comp rx=0 tx=0).stop 2024-03-18T16:15:36.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.823+0000 7fcd7bb9e740 1 -- 172.21.15.39:0/2329628769 shutdown_connections 2024-03-18T16:15:36.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.823+0000 7fcd7bb9e740 1 --2- 172.21.15.39:0/2329628769 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x560841e79a30 0x560841e7be60 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:36.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.823+0000 7fcd7bb9e740 1 --2- 172.21.15.39:0/2329628769 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560841e69420 0x560841e75100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:36.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.823+0000 7fcd53e00640 0 client.16266 ms_handle_reset on v2:172.21.15.39:6840/3515326042 2024-03-18T16:15:36.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.823+0000 7fcd7bb9e740 1 --2- 172.21.15.39:0/2329628769 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560841e642b0 0x560841e6e0c0 secure :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0x560841e646b0 tx=0x7fcd60011af0 comp rx=0 tx=0).stop 2024-03-18T16:15:36.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.823+0000 7fcd7bb9e740 1 --2- 172.21.15.39:0/2329628769 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560841e638c0 0x560841e6db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:36.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.823+0000 7fcd7bb9e740 1 -- 172.21.15.39:0/2329628769 >> 172.21.15.39:0/2329628769 conn(0x560841e5ee30 msgr2=0x560841e5e1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:36.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.823+0000 7fcd7bb9e740 1 -- 172.21.15.39:0/2329628769 shutdown_connections 2024-03-18T16:15:36.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.823+0000 7fcd7bb9e740 1 -- 172.21.15.39:0/2329628769 wait complete. 2024-03-18T16:15:36.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.825+0000 7fcd7bb9e740 20 client.16266 trim_cache size 0 max 16384 2024-03-18T16:15:36.860 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.857+0000 7fcd7bb9e740 1 Processor -- start 2024-03-18T16:15:36.860 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.857+0000 7fcd7bb9e740 1 -- start start 2024-03-18T16:15:36.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.857+0000 7fcd7bb9e740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560841e638c0 0x560841e5e070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:36.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.857+0000 7fcd7bb9e740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560841e69420 0x560841e750c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:36.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.857+0000 7fcd7bb9e740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560841e6e1f0 0x560841e80760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:36.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.857+0000 7fcd7bb9e740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fcd741b8ff0 con 0x560841e69420 2024-03-18T16:15:36.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.857+0000 7fcd7bb9e740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fcd741bab30 con 0x560841e638c0 2024-03-18T16:15:36.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.857+0000 7fcd7bb9e740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fcd741b96c0 con 0x560841e6e1f0 2024-03-18T16:15:36.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.858+0000 7fcd71400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560841e638c0 0x560841e5e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:36.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.858+0000 7fcd71400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560841e638c0 0x560841e5e070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:53578/0 (socket says 172.21.15.39:53578) 2024-03-18T16:15:36.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.858+0000 7fcd71400640 1 -- 172.21.15.39:0/2756091447 learned_addr learned my addr 172.21.15.39:0/2756091447 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:15:36.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.858+0000 7fcd71400640 1 --2- 172.21.15.39:0/2756091447 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560841e6e1f0 0x560841e80760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:36.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.858+0000 7fcd71400640 1 -- 172.21.15.39:0/2756091447 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560841e6e1f0 msgr2=0x560841e80760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:36.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.858+0000 7fcd71400640 1 --2- 172.21.15.39:0/2756091447 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560841e6e1f0 0x560841e80760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:36.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.858+0000 7fcd71400640 1 -- 172.21.15.39:0/2756091447 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560841e69420 msgr2=0x560841e750c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:36.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.858+0000 7fcd71400640 1 --2- 172.21.15.39:0/2756091447 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560841e69420 0x560841e750c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:36.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.858+0000 7fcd71400640 1 -- 172.21.15.39:0/2756091447 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560841e7c4e0 con 0x560841e638c0 2024-03-18T16:15:36.863 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.860+0000 7fcd71400640 1 --2- 172.21.15.39:0/2756091447 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560841e638c0 0x560841e5e070 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x560841e7c710 tx=0x7fcd64005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea35a2fd25ef28b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:36.863 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.860+0000 7fcd5ea00640 1 -- 172.21.15.39:0/2756091447 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd640048c0 con 0x560841e638c0 2024-03-18T16:15:36.863 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.860+0000 7fcd5ea00640 1 -- 172.21.15.39:0/2756091447 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcd64032bc0 con 0x560841e638c0 2024-03-18T16:15:36.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.860+0000 7fcd5ea00640 1 -- 172.21.15.39:0/2756091447 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd64004dc0 con 0x560841e638c0 2024-03-18T16:15:36.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.861+0000 7fcd7bb9e740 1 -- 172.21.15.39:0/2756091447 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560841e638c0 msgr2=0x560841e5e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:36.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.861+0000 7fcd7bb9e740 1 --2- 172.21.15.39:0/2756091447 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560841e638c0 0x560841e5e070 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x560841e7c710 tx=0x7fcd64005f90 comp rx=0 tx=0).stop 2024-03-18T16:15:36.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.861+0000 7fcd7bb9e740 1 -- 172.21.15.39:0/2756091447 shutdown_connections 2024-03-18T16:15:36.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.861+0000 7fcd7bb9e740 1 --2- 172.21.15.39:0/2756091447 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560841e6e1f0 0x560841e80760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:36.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.861+0000 7fcd7bb9e740 1 --2- 172.21.15.39:0/2756091447 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560841e69420 0x560841e750c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:36.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.861+0000 7fcd7bb9e740 1 --2- 172.21.15.39:0/2756091447 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560841e638c0 0x560841e5e070 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:36.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.861+0000 7fcd7bb9e740 1 -- 172.21.15.39:0/2756091447 >> 172.21.15.39:0/2756091447 conn(0x560841e5ee30 msgr2=0x560841e5d960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:36.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.862+0000 7fcd7bb9e740 1 -- 172.21.15.39:0/2756091447 shutdown_connections 2024-03-18T16:15:36.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.862+0000 7fcd7bb9e740 1 -- 172.21.15.39:0/2756091447 wait complete. 2024-03-18T16:15:36.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.863+0000 7fcd7bb9e740 1 -- start start 2024-03-18T16:15:36.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.863+0000 7fcd7bb9e740 1 Processor -- start 2024-03-18T16:15:36.867 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.864+0000 7fcd7bb9e740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560841e638c0 0x560841e72ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:36.867 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.864+0000 7fcd7bb9e740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560841e69420 0x560841e730e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:36.868 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.865+0000 7fcd7bb9e740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560841e6e1f0 0x560841e740f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:36.868 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.865+0000 7fcd71400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560841e638c0 0x560841e72ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:36.868 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.865+0000 7fcd7bb9e740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fcd741ba460 con 0x560841e6e1f0 2024-03-18T16:15:36.868 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.865+0000 7fcd71400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560841e638c0 0x560841e72ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:53584/0 (socket says 172.21.15.39:53584) 2024-03-18T16:15:36.868 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.865+0000 7fcd7bb9e740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fcd741b9d90 con 0x560841e638c0 2024-03-18T16:15:36.868 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.865+0000 7fcd71400640 1 -- 172.21.15.39:0/2787862627 learned_addr learned my addr 172.21.15.39:0/2787862627 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:15:36.868 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.865+0000 7fcd7bb9e740 1 -- 172.21.15.39:0/2787862627 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fcd741b7080 con 0x560841e69420 2024-03-18T16:15:36.868 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.865+0000 7fcd71e00640 1 --2- 172.21.15.39:0/2787862627 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560841e69420 0x560841e730e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:36.868 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.865+0000 7fcd7bb9e740 10 client.0 _ll_register_callbacks cb 0x7fcd60009540 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:15:36.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.866+0000 7fcd71e00640 1 -- 172.21.15.39:0/2787862627 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560841e638c0 msgr2=0x560841e72ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:36.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.866+0000 7fcd71e00640 1 --2- 172.21.15.39:0/2787862627 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560841e638c0 0x560841e72ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:36.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.866+0000 7fcd71e00640 1 -- 172.21.15.39:0/2787862627 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560841e6e1f0 msgr2=0x560841e740f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:36.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.866+0000 7fcd71e00640 1 --2- 172.21.15.39:0/2787862627 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560841e6e1f0 0x560841e740f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:36.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.866+0000 7fcd71e00640 1 -- 172.21.15.39:0/2787862627 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560841e5ac10 con 0x560841e69420 2024-03-18T16:15:36.870 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.866+0000 7fcd71e00640 1 --2- 172.21.15.39:0/2787862627 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560841e69420 0x560841e730e0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fcd6000e000 tx=0x7fcd60009e90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a2581c620422e7b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:36.870 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.866+0000 7fcd53e00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:15:36.870 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.867+0000 7fcd71400640 1 --2- 172.21.15.39:0/2787862627 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560841e638c0 0x560841e72ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:15:36.871 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.869+0000 7fcd53e00640 1 -- 172.21.15.39:0/2787862627 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd60005710 con 0x560841e69420 2024-03-18T16:15:36.872 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.869+0000 7fcd53e00640 1 -- 172.21.15.39:0/2787862627 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fcd6001a5f0 con 0x560841e69420 2024-03-18T16:15:36.872 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.869+0000 7fcd7bb9e740 1 -- 172.21.15.39:0/2787862627 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcd64032bc0 con 0x560841e69420 2024-03-18T16:15:36.872 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.870+0000 7fcd53e00640 1 -- 172.21.15.39:0/2787862627 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7fcd60039050 con 0x560841e69420 2024-03-18T16:15:36.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.871+0000 7fcd7bb9e740 10 client.26671 fetch_fsmap learned FSMap version 42 2024-03-18T16:15:36.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.871+0000 7fcd7bb9e740 1 -- 172.21.15.39:0/2787862627 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x560841edfe90 con 0x560841e69420 2024-03-18T16:15:36.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.871+0000 7fcd53e00640 1 -- 172.21.15.39:0/2787862627 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcd64032bc0 con 0x560841e69420 2024-03-18T16:15:36.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.871+0000 7fcd53e00640 1 -- 172.21.15.39:0/2787862627 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 42) ==== 8370+0+0 (secure 0 0 0) 0x7fcd600298f0 con 0x560841e69420 2024-03-18T16:15:36.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.871+0000 7fcd7bb9e740 10 client.26671 fetch_fsmap finished waiting for FSMap version 42 2024-03-18T16:15:36.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.871+0000 7fcd7bb9e740 10 client.26671 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-03-18T16:15:36.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.871+0000 7fcd7bb9e740 20 client.26671 populate_metadata read hostname 'smithi039' 2024-03-18T16:15:36.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.871+0000 7fcd7bb9e740 1 --2- 172.21.15.39:0/2787862627 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x560841ee0860 0x560841ee2c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:36.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.871+0000 7fcd7bb9e740 4 client.26671 mds_command: new command op to 14463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:15:36.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.871+0000 7fcd7bb9e740 1 -- 172.21.15.39:0/2787862627 --> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560841dcfce0 con 0x560841ee0860 2024-03-18T16:15:36.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.872+0000 7fcd71400640 1 --2- 172.21.15.39:0/2787862627 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x560841ee0860 0x560841ee2c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:36.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.873+0000 7fcd53e00640 10 client.26671 ms_handle_connect on v2:172.21.15.39:6840/3515326042 2024-03-18T16:15:36.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.874+0000 7fcd71400640 1 --2- 172.21.15.39:0/2787862627 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x560841ee0860 0x560841ee2c80 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=61816f8ceb80d05f server_cookie=d3abe30e24e335a8 in_seq=0 out_seq=0 2024-03-18T16:15:36.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.876+0000 7fcd53e00640 1 -- 172.21.15.39:0/2787862627 <== mds.4 v2:172.21.15.39:6840/3515326042 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560841dcfce0 con 0x560841ee0860 2024-03-18T16:15:36.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.876+0000 7fcd53e00640 10 client.26671 handle_command_reply: tid=0 2024-03-18T16:15:36.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.877+0000 7fcd7bb9e740 1 client.26671 shutdown 2024-03-18T16:15:36.880 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.878+0000 7fcd7bb9e740 1 -- 172.21.15.39:0/2787862627 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560841e69420 msgr2=0x560841e730e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:36.880 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.878+0000 7fcd7bb9e740 1 --2- 172.21.15.39:0/2787862627 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560841e69420 0x560841e730e0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fcd6000e000 tx=0x7fcd60009e90 comp rx=0 tx=0).stop 2024-03-18T16:15:36.880 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.878+0000 7fcd7bb9e740 1 -- 172.21.15.39:0/2787862627 shutdown_connections 2024-03-18T16:15:36.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.878+0000 7fcd7bb9e740 1 --2- 172.21.15.39:0/2787862627 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x560841ee0860 0x560841ee2c80 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:36.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.878+0000 7fcd53e00640 0 client.26671 ms_handle_reset on v2:172.21.15.39:6840/3515326042 2024-03-18T16:15:36.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.878+0000 7fcd7bb9e740 1 --2- 172.21.15.39:0/2787862627 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560841e6e1f0 0x560841e740f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:36.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.878+0000 7fcd7bb9e740 1 --2- 172.21.15.39:0/2787862627 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560841e69420 0x560841e730e0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:36.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.878+0000 7fcd7bb9e740 1 --2- 172.21.15.39:0/2787862627 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560841e638c0 0x560841e72ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:36.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.878+0000 7fcd7bb9e740 1 -- 172.21.15.39:0/2787862627 >> 172.21.15.39:0/2787862627 conn(0x560841e5ee30 msgr2=0x560841e5e1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:36.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.879+0000 7fcd7bb9e740 1 -- 172.21.15.39:0/2787862627 shutdown_connections 2024-03-18T16:15:36.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.879+0000 7fcd7bb9e740 1 -- 172.21.15.39:0/2787862627 wait complete. 2024-03-18T16:15:36.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.880+0000 7fcd7bb9e740 20 client.26671 trim_cache size 0 max 16384 2024-03-18T16:15:36.883 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:15:36.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.881+0000 7fcd78e00640 1 -- 172.21.15.39:0/175179189 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fcd580a2950 msgr2=0x7fcd580a4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:36.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.881+0000 7fcd78e00640 1 --2- 172.21.15.39:0/175179189 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fcd580a2950 0x7fcd580a4e10 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fcd6c033ff0 tx=0x7fcd6c004620 comp rx=0 tx=0).stop 2024-03-18T16:15:36.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.881+0000 7fcd78e00640 1 -- 172.21.15.39:0/175179189 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcd74109930 msgr2=0x7fcd74081e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:36.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.881+0000 7fcd78e00640 1 --2- 172.21.15.39:0/175179189 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcd74109930 0x7fcd74081e10 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fcd6c036040 tx=0x7fcd6c0337b0 comp rx=0 tx=0).stop 2024-03-18T16:15:36.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.881+0000 7fcd78e00640 1 -- 172.21.15.39:0/175179189 shutdown_connections 2024-03-18T16:15:36.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.881+0000 7fcd78e00640 1 --2- 172.21.15.39:0/175179189 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcd74109930 0x7fcd74081e10 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:36.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.881+0000 7fcd78e00640 1 --2- 172.21.15.39:0/175179189 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fcd580a2950 0x7fcd580a4e10 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:36.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.881+0000 7fcd78e00640 1 --2- 172.21.15.39:0/175179189 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fcd74108f70 0x7fcd7407ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:36.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.881+0000 7fcd78e00640 1 --2- 172.21.15.39:0/175179189 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcd74107e20 0x7fcd7407a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:36.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.881+0000 7fcd78e00640 1 -- 172.21.15.39:0/175179189 >> 172.21.15.39:0/175179189 conn(0x7fcd74076dd0 msgr2=0x7fcd7410c230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:36.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.882+0000 7fcd78e00640 1 -- 172.21.15.39:0/175179189 shutdown_connections 2024-03-18T16:15:36.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:36.882+0000 7fcd78e00640 1 -- 172.21.15.39:0/175179189 wait complete. 2024-03-18T16:15:36.915 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-18T16:15:37.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.224+0000 7fd653a00640 1 -- 172.21.15.39:0/3379192243 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd64c0741e0 msgr2=0x7fd64c074640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:37.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.224+0000 7fd653a00640 1 --2- 172.21.15.39:0/3379192243 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd64c0741e0 0x7fd64c074640 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fd638009930 tx=0x7fd63802bcb0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.224+0000 7fd653a00640 1 -- 172.21.15.39:0/3379192243 shutdown_connections 2024-03-18T16:15:37.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.224+0000 7fd653a00640 1 --2- 172.21.15.39:0/3379192243 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd64c10de10 0x7fd64c110200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.224+0000 7fd653a00640 1 --2- 172.21.15.39:0/3379192243 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd64c0741e0 0x7fd64c074640 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.224+0000 7fd653a00640 1 --2- 172.21.15.39:0/3379192243 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd64c0757c0 0x7fd64c073c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.224+0000 7fd653a00640 1 -- 172.21.15.39:0/3379192243 >> 172.21.15.39:0/3379192243 conn(0x7fd64c0fcd50 msgr2=0x7fd64c0ff190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:37.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.225+0000 7fd653a00640 1 -- 172.21.15.39:0/3379192243 shutdown_connections 2024-03-18T16:15:37.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.225+0000 7fd653a00640 1 -- 172.21.15.39:0/3379192243 wait complete. 2024-03-18T16:15:37.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.226+0000 7fd653a00640 1 Processor -- start 2024-03-18T16:15:37.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.226+0000 7fd653a00640 1 -- start start 2024-03-18T16:15:37.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.227+0000 7fd653a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd64c0741e0 0x7fd64c0701c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:37.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.227+0000 7fd653a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd64c0757c0 0x7fd64c070700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:37.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.227+0000 7fd653a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd64c10de10 0x7fd64c1ae1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:37.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.227+0000 7fd653a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fd64c113aa0 con 0x7fd64c10de10 2024-03-18T16:15:37.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.227+0000 7fd653a00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fd64c113920 con 0x7fd64c0741e0 2024-03-18T16:15:37.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.227+0000 7fd653a00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fd64c113c20 con 0x7fd64c0757c0 2024-03-18T16:15:37.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.227+0000 7fd651000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd64c0741e0 0x7fd64c0701c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:37.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.227+0000 7fd651000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd64c0741e0 0x7fd64c0701c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:53602/0 (socket says 172.21.15.39:53602) 2024-03-18T16:15:37.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.227+0000 7fd651000640 1 -- 172.21.15.39:0/2141771518 learned_addr learned my addr 172.21.15.39:0/2141771518 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:15:37.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.227+0000 7fd64be00640 1 --2- 172.21.15.39:0/2141771518 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd64c0757c0 0x7fd64c070700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:37.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.227+0000 7fd651a00640 1 --2- 172.21.15.39:0/2141771518 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd64c10de10 0x7fd64c1ae1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:37.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.228+0000 7fd651000640 1 -- 172.21.15.39:0/2141771518 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd64c0757c0 msgr2=0x7fd64c070700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:37.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.228+0000 7fd651000640 1 --2- 172.21.15.39:0/2141771518 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd64c0757c0 0x7fd64c070700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.228+0000 7fd651000640 1 -- 172.21.15.39:0/2141771518 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd64c10de10 msgr2=0x7fd64c1ae1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:37.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.228+0000 7fd651000640 1 --2- 172.21.15.39:0/2141771518 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd64c10de10 0x7fd64c1ae1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.228+0000 7fd651000640 1 -- 172.21.15.39:0/2141771518 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd64c1ae6f0 con 0x7fd64c0741e0 2024-03-18T16:15:37.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.228+0000 7fd651000640 1 --2- 172.21.15.39:0/2141771518 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd64c0741e0 0x7fd64c0701c0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fd64000d950 tx=0x7fd64000de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:37.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.229+0000 7fd649600640 1 -- 172.21.15.39:0/2141771518 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd640017070 con 0x7fd64c0741e0 2024-03-18T16:15:37.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.229+0000 7fd653a00640 1 -- 172.21.15.39:0/2141771518 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd64c1ae8c0 con 0x7fd64c0741e0 2024-03-18T16:15:37.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.229+0000 7fd649600640 1 -- 172.21.15.39:0/2141771518 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd640009c30 con 0x7fd64c0741e0 2024-03-18T16:15:37.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.229+0000 7fd649600640 1 -- 172.21.15.39:0/2141771518 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd64000f620 con 0x7fd64c0741e0 2024-03-18T16:15:37.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.229+0000 7fd653a00640 1 -- 172.21.15.39:0/2141771518 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd64c1aedd0 con 0x7fd64c0741e0 2024-03-18T16:15:37.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.233+0000 7fd649600640 1 -- 172.21.15.39:0/2141771518 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fd640010460 con 0x7fd64c0741e0 2024-03-18T16:15:37.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.233+0000 7fd656789740 1 Processor -- start 2024-03-18T16:15:37.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.234+0000 7fd656789740 1 -- start start 2024-03-18T16:15:37.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.235+0000 7fd656789740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563526a568c0 0x563526a56ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:37.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.235+0000 7fd656789740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563526a572b0 0x563526a5bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:37.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.235+0000 7fd656789740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563526a5c420 0x563526a5e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:37.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.235+0000 7fd656789740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x563526a1e3b0 con 0x563526a5c420 2024-03-18T16:15:37.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.235+0000 7fd656789740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5635269bb450 con 0x563526a572b0 2024-03-18T16:15:37.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.235+0000 7fd656789740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x563526963cb0 con 0x563526a568c0 2024-03-18T16:15:37.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.236+0000 7fd64be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563526a5c420 0x563526a5e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:37.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.236+0000 7fd649600640 1 --2- 172.21.15.39:0/2141771518 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd6340a2900 0x7fd6340a4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:37.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.236+0000 7fd651000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563526a572b0 0x563526a5bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:37.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.236+0000 7fd651000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563526a572b0 0x563526a5bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:53608/0 (socket says 172.21.15.39:53608) 2024-03-18T16:15:37.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.236+0000 7fd651000640 1 -- 172.21.15.39:0/1370691530 learned_addr learned my addr 172.21.15.39:0/1370691530 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:15:37.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.236+0000 7fd651a00640 1 --2- 172.21.15.39:0/1370691530 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563526a568c0 0x563526a56ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:37.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.236+0000 7fd651000640 1 -- 172.21.15.39:0/1370691530 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563526a568c0 msgr2=0x563526a56ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:37.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.236+0000 7fd651000640 1 --2- 172.21.15.39:0/1370691530 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563526a568c0 0x563526a56ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.236+0000 7fd651000640 1 -- 172.21.15.39:0/1370691530 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563526a5c420 msgr2=0x563526a5e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:37.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.236+0000 7fd651000640 1 --2- 172.21.15.39:0/1370691530 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563526a5c420 0x563526a5e8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.237+0000 7fd651000640 1 -- 172.21.15.39:0/1370691530 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5635269851f0 con 0x563526a572b0 2024-03-18T16:15:37.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.237+0000 7fd651a00640 1 --2- 172.21.15.39:0/1370691530 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563526a568c0 0x563526a56ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:15:37.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.237+0000 7fd651000640 1 --2- 172.21.15.39:0/1370691530 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563526a572b0 0x563526a5bdb0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fd640097550 tx=0x7fd6400905d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=52a8ebc9d57de1c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:37.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.237+0000 7fd651a00640 1 --2- 172.21.15.39:0/2141771518 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd6340a2900 0x7fd6340a4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:37.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.237+0000 7fd649600640 1 -- 172.21.15.39:0/2141771518 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fd6400ceb40 con 0x7fd64c0741e0 2024-03-18T16:15:37.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.238+0000 7fd63e000640 1 -- 172.21.15.39:0/1370691530 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd64006c030 con 0x563526a572b0 2024-03-18T16:15:37.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.238+0000 7fd63e000640 1 -- 172.21.15.39:0/1370691530 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd6400bb080 con 0x563526a572b0 2024-03-18T16:15:37.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.238+0000 7fd63e000640 1 -- 172.21.15.39:0/1370691530 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd64009b0f0 con 0x563526a572b0 2024-03-18T16:15:37.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.239+0000 7fd656789740 1 -- 172.21.15.39:0/1370691530 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563526a572b0 msgr2=0x563526a5bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:37.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.239+0000 7fd651a00640 1 --2- 172.21.15.39:0/2141771518 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd6340a2900 0x7fd6340a4dc0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x563526a57b10 tx=0x7fd644004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:37.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.239+0000 7fd656789740 1 --2- 172.21.15.39:0/1370691530 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563526a572b0 0x563526a5bdb0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fd640097550 tx=0x7fd6400905d0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.241+0000 7fd656789740 1 -- 172.21.15.39:0/1370691530 shutdown_connections 2024-03-18T16:15:37.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.241+0000 7fd656789740 1 --2- 172.21.15.39:0/1370691530 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563526a572b0 0x563526a5bdb0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.241+0000 7fd656789740 1 --2- 172.21.15.39:0/1370691530 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563526a5c420 0x563526a5e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.241+0000 7fd656789740 1 --2- 172.21.15.39:0/1370691530 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563526a568c0 0x563526a56ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.241+0000 7fd656789740 1 -- 172.21.15.39:0/1370691530 >> 172.21.15.39:0/1370691530 conn(0x563526a51e30 msgr2=0x563526a52230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:37.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.241+0000 7fd656789740 1 -- 172.21.15.39:0/1370691530 shutdown_connections 2024-03-18T16:15:37.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.241+0000 7fd656789740 1 -- 172.21.15.39:0/1370691530 wait complete. 2024-03-18T16:15:37.245 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.242+0000 7fd656789740 1 -- start start 2024-03-18T16:15:37.245 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.242+0000 7fd656789740 1 Processor -- start 2024-03-18T16:15:37.245 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.243+0000 7fd656789740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563526a568c0 0x563526a71b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:37.245 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.243+0000 7fd656789740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563526a572b0 0x563526a72080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:37.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.243+0000 7fd656789740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563526a5c420 0x563526a790c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:37.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.243+0000 7fd656789740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x563526977110 con 0x563526a568c0 2024-03-18T16:15:37.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.243+0000 7fd656789740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5635269b74f0 con 0x563526a5c420 2024-03-18T16:15:37.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.243+0000 7fd656789740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x563526a70380 con 0x563526a572b0 2024-03-18T16:15:37.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.243+0000 7fd656789740 10 client.0 _ll_register_callbacks cb 0x563526964b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:15:37.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.243+0000 7fd651a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563526a5c420 0x563526a790c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:37.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.243+0000 7fd651a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563526a5c420 0x563526a790c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:53620/0 (socket says 172.21.15.39:53620) 2024-03-18T16:15:37.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.243+0000 7fd651a00640 1 -- 172.21.15.39:0/4008097411 learned_addr learned my addr 172.21.15.39:0/4008097411 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:15:37.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.244+0000 7fd651a00640 1 -- 172.21.15.39:0/4008097411 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563526a572b0 msgr2=0x563526a72080 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T16:15:37.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.244+0000 7fd651a00640 1 --2- 172.21.15.39:0/4008097411 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563526a572b0 0x563526a72080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.244+0000 7fd651a00640 1 -- 172.21.15.39:0/4008097411 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563526a568c0 msgr2=0x563526a71b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:37.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.244+0000 7fd651a00640 1 --2- 172.21.15.39:0/4008097411 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563526a568c0 0x563526a71b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.244+0000 7fd651a00640 1 -- 172.21.15.39:0/4008097411 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5635266cf3f0 con 0x563526a5c420 2024-03-18T16:15:37.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.244+0000 7fd651a00640 1 --2- 172.21.15.39:0/4008097411 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563526a5c420 0x563526a790c0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fd6440128a0 tx=0x7fd644004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=117bb5145b6c2903 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:37.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.244+0000 7fd632a00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:15:37.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.245+0000 7fd632a00640 1 -- 172.21.15.39:0/4008097411 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd644022070 con 0x563526a5c420 2024-03-18T16:15:37.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.245+0000 7fd632a00640 1 -- 172.21.15.39:0/4008097411 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fd644010de0 con 0x563526a5c420 2024-03-18T16:15:37.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.245+0000 7fd656789740 1 -- 172.21.15.39:0/4008097411 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd6400bb080 con 0x563526a5c420 2024-03-18T16:15:37.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.245+0000 7fd632a00640 1 -- 172.21.15.39:0/4008097411 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7fd644027440 con 0x563526a5c420 2024-03-18T16:15:37.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.245+0000 7fd656789740 10 client.26720 fetch_fsmap learned FSMap version 42 2024-03-18T16:15:37.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.245+0000 7fd656789740 1 -- 172.21.15.39:0/4008097411 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x5635267997d0 con 0x563526a5c420 2024-03-18T16:15:37.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.246+0000 7fd632a00640 1 -- 172.21.15.39:0/4008097411 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd6400bb080 con 0x563526a5c420 2024-03-18T16:15:37.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.247+0000 7fd632a00640 1 -- 172.21.15.39:0/4008097411 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 42) ==== 8370+0+0 (secure 0 0 0) 0x7fd64401d610 con 0x563526a5c420 2024-03-18T16:15:37.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.247+0000 7fd656789740 10 client.26720 fetch_fsmap finished waiting for FSMap version 42 2024-03-18T16:15:37.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.247+0000 7fd656789740 10 client.26720 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-18T16:15:37.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.247+0000 7fd656789740 20 client.26720 populate_metadata read hostname 'smithi039' 2024-03-18T16:15:37.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.247+0000 7fd656789740 1 --2- 172.21.15.39:0/4008097411 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x563526a7d9d0 0x563526a7fe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:37.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.247+0000 7fd656789740 4 client.26720 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:15:37.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.247+0000 7fd656789740 1 -- 172.21.15.39:0/4008097411 --> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5635269c8990 con 0x563526a7d9d0 2024-03-18T16:15:37.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.248+0000 7fd651000640 1 --2- 172.21.15.39:0/4008097411 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x563526a7d9d0 0x563526a7fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:37.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.248+0000 7fd651000640 1 --2- 172.21.15.39:0/4008097411 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x563526a7d9d0 0x563526a7fe90 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=786c6a04c998f837 server_cookie=8df2214aef095dd3 in_seq=0 out_seq=0 2024-03-18T16:15:37.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.249+0000 7fd632a00640 10 client.26720 ms_handle_connect on v2:172.21.15.194:6836/1301464246 2024-03-18T16:15:37.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.249+0000 7fd632a00640 1 -- 172.21.15.39:0/4008097411 <== mds.3 v2:172.21.15.194:6836/1301464246 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5635269c8990 con 0x563526a7d9d0 2024-03-18T16:15:37.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.249+0000 7fd632a00640 10 client.26720 handle_command_reply: tid=0 2024-03-18T16:15:37.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.251+0000 7fd656789740 1 client.26720 shutdown 2024-03-18T16:15:37.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.251+0000 7fd656789740 1 -- 172.21.15.39:0/4008097411 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563526a5c420 msgr2=0x563526a790c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:37.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.251+0000 7fd656789740 1 --2- 172.21.15.39:0/4008097411 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563526a5c420 0x563526a790c0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fd6440128a0 tx=0x7fd644004ab0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.251+0000 7fd656789740 1 -- 172.21.15.39:0/4008097411 shutdown_connections 2024-03-18T16:15:37.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.251+0000 7fd656789740 1 --2- 172.21.15.39:0/4008097411 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x563526a7d9d0 0x563526a7fe90 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.251+0000 7fd656789740 1 --2- 172.21.15.39:0/4008097411 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563526a5c420 0x563526a790c0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.251+0000 7fd656789740 1 --2- 172.21.15.39:0/4008097411 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563526a572b0 0x563526a72080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.251+0000 7fd656789740 1 --2- 172.21.15.39:0/4008097411 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563526a568c0 0x563526a71b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.251+0000 7fd656789740 1 -- 172.21.15.39:0/4008097411 >> 172.21.15.39:0/4008097411 conn(0x563526a51e30 msgr2=0x563526a511f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:37.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.251+0000 7fd632a00640 0 client.26720 ms_handle_reset on v2:172.21.15.194:6836/1301464246 2024-03-18T16:15:37.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.252+0000 7fd656789740 1 -- 172.21.15.39:0/4008097411 shutdown_connections 2024-03-18T16:15:37.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.252+0000 7fd656789740 1 -- 172.21.15.39:0/4008097411 wait complete. 2024-03-18T16:15:37.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.252+0000 7fd656789740 20 client.26720 trim_cache size 0 max 16384 2024-03-18T16:15:37.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.274+0000 7fd656789740 1 Processor -- start 2024-03-18T16:15:37.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.274+0000 7fd656789740 1 -- start start 2024-03-18T16:15:37.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.274+0000 7fd656789740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563526a51e30 0x563526a7f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:37.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.274+0000 7fd656789740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563526a568c0 0x563526a79080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:37.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.275+0000 7fd656789740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563526a572b0 0x563526a50960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:37.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.275+0000 7fd656789740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fd64c1b0bc0 con 0x563526a572b0 2024-03-18T16:15:37.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.275+0000 7fd656789740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fd64c1b2700 con 0x563526a568c0 2024-03-18T16:15:37.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.275+0000 7fd656789740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fd64c1b1290 con 0x563526a51e30 2024-03-18T16:15:37.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.275+0000 7fd64be00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563526a568c0 0x563526a79080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:37.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.275+0000 7fd64be00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563526a568c0 0x563526a79080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:53630/0 (socket says 172.21.15.39:53630) 2024-03-18T16:15:37.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.275+0000 7fd64be00640 1 -- 172.21.15.39:0/159801857 learned_addr learned my addr 172.21.15.39:0/159801857 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:15:37.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.275+0000 7fd651000640 1 --2- 172.21.15.39:0/159801857 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563526a51e30 0x563526a7f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:37.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.275+0000 7fd64be00640 1 -- 172.21.15.39:0/159801857 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563526a51e30 msgr2=0x563526a7f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:37.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.275+0000 7fd64be00640 1 --2- 172.21.15.39:0/159801857 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563526a51e30 0x563526a7f9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.275+0000 7fd64be00640 1 -- 172.21.15.39:0/159801857 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563526a572b0 msgr2=0x563526a50960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:37.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.275+0000 7fd64be00640 1 --2- 172.21.15.39:0/159801857 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563526a572b0 0x563526a50960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.275+0000 7fd64be00640 1 -- 172.21.15.39:0/159801857 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563526a4dc10 con 0x563526a568c0 2024-03-18T16:15:37.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.276+0000 7fd64be00640 1 --2- 172.21.15.39:0/159801857 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563526a568c0 0x563526a79080 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fd638006010 tx=0x7fd63802c6e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e4781b2d80b7e0c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:37.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.276+0000 7fd63d600640 1 -- 172.21.15.39:0/159801857 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd638042070 con 0x563526a568c0 2024-03-18T16:15:37.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.276+0000 7fd63d600640 1 -- 172.21.15.39:0/159801857 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd63802cc00 con 0x563526a568c0 2024-03-18T16:15:37.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.276+0000 7fd63d600640 1 -- 172.21.15.39:0/159801857 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6380055e0 con 0x563526a568c0 2024-03-18T16:15:37.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.277+0000 7fd656789740 1 -- 172.21.15.39:0/159801857 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563526a568c0 msgr2=0x563526a79080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:37.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.277+0000 7fd656789740 1 --2- 172.21.15.39:0/159801857 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563526a568c0 0x563526a79080 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fd638006010 tx=0x7fd63802c6e0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.278+0000 7fd656789740 1 -- 172.21.15.39:0/159801857 shutdown_connections 2024-03-18T16:15:37.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.278+0000 7fd656789740 1 --2- 172.21.15.39:0/159801857 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563526a568c0 0x563526a79080 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.278+0000 7fd656789740 1 --2- 172.21.15.39:0/159801857 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563526a572b0 0x563526a50960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.278+0000 7fd656789740 1 --2- 172.21.15.39:0/159801857 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563526a51e30 0x563526a7f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.278+0000 7fd656789740 1 -- 172.21.15.39:0/159801857 >> 172.21.15.39:0/159801857 conn(0x563526a4d410 msgr2=0x563526a58710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:37.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.278+0000 7fd656789740 1 -- 172.21.15.39:0/159801857 shutdown_connections 2024-03-18T16:15:37.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.278+0000 7fd656789740 1 -- 172.21.15.39:0/159801857 wait complete. 2024-03-18T16:15:37.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.278+0000 7fd656789740 1 -- start start 2024-03-18T16:15:37.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.279+0000 7fd656789740 1 Processor -- start 2024-03-18T16:15:37.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.279+0000 7fd656789740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563526a51e30 0x563526a76400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:37.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.279+0000 7fd656789740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563526a568c0 0x563526a76940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:37.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.279+0000 7fd656789740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563526a572b0 0x563526a77950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:37.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.279+0000 7fd656789740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fd64c1b2030 con 0x563526a51e30 2024-03-18T16:15:37.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.279+0000 7fd656789740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fd64c1b1960 con 0x563526a568c0 2024-03-18T16:15:37.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.279+0000 7fd656789740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fd64c10da10 con 0x563526a572b0 2024-03-18T16:15:37.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.279+0000 7fd656789740 10 client.0 _ll_register_callbacks cb 0x7fd6440099b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:15:37.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.280+0000 7fd64be00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563526a568c0 0x563526a76940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:37.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.280+0000 7fd64be00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563526a568c0 0x563526a76940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:53646/0 (socket says 172.21.15.39:53646) 2024-03-18T16:15:37.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.280+0000 7fd64be00640 1 -- 172.21.15.39:0/1767337041 learned_addr learned my addr 172.21.15.39:0/1767337041 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:15:37.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.280+0000 7fd651a00640 1 --2- 172.21.15.39:0/1767337041 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563526a572b0 0x563526a77950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:37.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.280+0000 7fd651000640 1 --2- 172.21.15.39:0/1767337041 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563526a51e30 0x563526a76400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:37.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.280+0000 7fd64be00640 1 -- 172.21.15.39:0/1767337041 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563526a572b0 msgr2=0x563526a77950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:37.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.280+0000 7fd64be00640 1 --2- 172.21.15.39:0/1767337041 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563526a572b0 0x563526a77950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.280+0000 7fd64be00640 1 -- 172.21.15.39:0/1767337041 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563526a51e30 msgr2=0x563526a76400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:37.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.280+0000 7fd64be00640 1 --2- 172.21.15.39:0/1767337041 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563526a51e30 0x563526a76400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.280+0000 7fd64be00640 1 -- 172.21.15.39:0/1767337041 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563526a77f40 con 0x563526a568c0 2024-03-18T16:15:37.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.280+0000 7fd651000640 1 --2- 172.21.15.39:0/1767337041 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563526a51e30 0x563526a76400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:15:37.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.281+0000 7fd64be00640 1 --2- 172.21.15.39:0/1767337041 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563526a568c0 0x563526a76940 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fd63802c1f0 tx=0x7fd638005780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bec8ddaaef816551 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:37.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.281+0000 7fd632a00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:15:37.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.281+0000 7fd632a00640 1 -- 172.21.15.39:0/1767337041 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd638042040 con 0x563526a568c0 2024-03-18T16:15:37.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.281+0000 7fd632a00640 1 -- 172.21.15.39:0/1767337041 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fd638004c80 con 0x563526a568c0 2024-03-18T16:15:37.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.281+0000 7fd656789740 1 -- 172.21.15.39:0/1767337041 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd63802cc00 con 0x563526a568c0 2024-03-18T16:15:37.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.281+0000 7fd632a00640 1 -- 172.21.15.39:0/1767337041 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7fd638005d50 con 0x563526a568c0 2024-03-18T16:15:37.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.282+0000 7fd656789740 10 client.26726 fetch_fsmap learned FSMap version 42 2024-03-18T16:15:37.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.282+0000 7fd656789740 1 -- 172.21.15.39:0/1767337041 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x563526a79270 con 0x563526a568c0 2024-03-18T16:15:37.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.282+0000 7fd632a00640 1 -- 172.21.15.39:0/1767337041 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd63802cc00 con 0x563526a568c0 2024-03-18T16:15:37.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.282+0000 7fd632a00640 1 -- 172.21.15.39:0/1767337041 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 42) ==== 8370+0+0 (secure 0 0 0) 0x7fd6380345e0 con 0x563526a568c0 2024-03-18T16:15:37.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.282+0000 7fd656789740 10 client.26726 fetch_fsmap finished waiting for FSMap version 42 2024-03-18T16:15:37.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.282+0000 7fd656789740 10 client.26726 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-18T16:15:37.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.282+0000 7fd656789740 20 client.26726 populate_metadata read hostname 'smithi039' 2024-03-18T16:15:37.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.282+0000 7fd656789740 1 --2- 172.21.15.39:0/1767337041 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x563526a5c420 0x563526ae64e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:37.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.282+0000 7fd656789740 4 client.26726 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:15:37.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.282+0000 7fd656789740 1 -- 172.21.15.39:0/1767337041 --> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5635269c2ce0 con 0x563526a5c420 2024-03-18T16:15:37.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.283+0000 7fd651000640 1 --2- 172.21.15.39:0/1767337041 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x563526a5c420 0x563526ae64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:37.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.283+0000 7fd651000640 1 --2- 172.21.15.39:0/1767337041 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x563526a5c420 0x563526ae64e0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=cf8862f0c8115208 server_cookie=6afdc25cc7d13a3a in_seq=0 out_seq=0 2024-03-18T16:15:37.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.283+0000 7fd632a00640 10 client.26726 ms_handle_connect on v2:172.21.15.194:6836/1301464246 2024-03-18T16:15:37.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.283+0000 7fd632a00640 1 -- 172.21.15.39:0/1767337041 <== mds.3 v2:172.21.15.194:6836/1301464246 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5635269c2ce0 con 0x563526a5c420 2024-03-18T16:15:37.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.283+0000 7fd632a00640 10 client.26726 handle_command_reply: tid=0 2024-03-18T16:15:37.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.283+0000 7fd656789740 1 client.26726 shutdown 2024-03-18T16:15:37.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.284+0000 7fd656789740 1 -- 172.21.15.39:0/1767337041 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563526a568c0 msgr2=0x563526a76940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:37.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.284+0000 7fd656789740 1 --2- 172.21.15.39:0/1767337041 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563526a568c0 0x563526a76940 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fd63802c1f0 tx=0x7fd638005780 comp rx=0 tx=0).stop 2024-03-18T16:15:37.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.284+0000 7fd656789740 1 -- 172.21.15.39:0/1767337041 shutdown_connections 2024-03-18T16:15:37.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.284+0000 7fd656789740 1 --2- 172.21.15.39:0/1767337041 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x563526a5c420 0x563526ae64e0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.284+0000 7fd656789740 1 --2- 172.21.15.39:0/1767337041 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563526a568c0 0x563526a76940 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.284+0000 7fd656789740 1 --2- 172.21.15.39:0/1767337041 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563526a572b0 0x563526a77950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.284+0000 7fd656789740 1 --2- 172.21.15.39:0/1767337041 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563526a51e30 0x563526a76400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.284+0000 7fd656789740 1 -- 172.21.15.39:0/1767337041 >> 172.21.15.39:0/1767337041 conn(0x563526a4d410 msgr2=0x563526a50960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:37.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.284+0000 7fd632a00640 0 client.26726 ms_handle_reset on v2:172.21.15.194:6836/1301464246 2024-03-18T16:15:37.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.284+0000 7fd656789740 1 -- 172.21.15.39:0/1767337041 shutdown_connections 2024-03-18T16:15:37.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.284+0000 7fd656789740 1 -- 172.21.15.39:0/1767337041 wait complete. 2024-03-18T16:15:37.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.284+0000 7fd656789740 20 client.26726 trim_cache size 0 max 16384 2024-03-18T16:15:37.290 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:15:37.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.285+0000 7fd653a00640 1 -- 172.21.15.39:0/2141771518 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd6340a2900 msgr2=0x7fd6340a4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:37.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.285+0000 7fd653a00640 1 --2- 172.21.15.39:0/2141771518 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd6340a2900 0x7fd6340a4dc0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x563526a57b10 tx=0x7fd644004a40 comp rx=0 tx=0).stop 2024-03-18T16:15:37.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.285+0000 7fd653a00640 1 -- 172.21.15.39:0/2141771518 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd64c0741e0 msgr2=0x7fd64c0701c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:37.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.285+0000 7fd653a00640 1 --2- 172.21.15.39:0/2141771518 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd64c0741e0 0x7fd64c0701c0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fd64000d950 tx=0x7fd64000de10 comp rx=0 tx=0).stop 2024-03-18T16:15:37.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.286+0000 7fd653a00640 1 -- 172.21.15.39:0/2141771518 shutdown_connections 2024-03-18T16:15:37.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.286+0000 7fd653a00640 1 --2- 172.21.15.39:0/2141771518 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd6340a2900 0x7fd6340a4dc0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.286+0000 7fd653a00640 1 --2- 172.21.15.39:0/2141771518 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd64c10de10 0x7fd64c1ae1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.286+0000 7fd653a00640 1 --2- 172.21.15.39:0/2141771518 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd64c0757c0 0x7fd64c070700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.286+0000 7fd653a00640 1 --2- 172.21.15.39:0/2141771518 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd64c0741e0 0x7fd64c0701c0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.286+0000 7fd653a00640 1 -- 172.21.15.39:0/2141771518 >> 172.21.15.39:0/2141771518 conn(0x7fd64c0fcd50 msgr2=0x7fd64c101150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:37.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.286+0000 7fd653a00640 1 -- 172.21.15.39:0/2141771518 shutdown_connections 2024-03-18T16:15:37.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.286+0000 7fd653a00640 1 -- 172.21.15.39:0/2141771518 wait complete. 2024-03-18T16:15:37.311 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-18T16:15:37.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:37 smithi196 ceph-mon[36307]: pgmap v588: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 8.7 MiB/s wr, 1.97k op/s 2024-03-18T16:15:37.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:37 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/914724837' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:15:37.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:37 smithi194 ceph-mon[36200]: pgmap v588: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 8.7 MiB/s wr, 1.97k op/s 2024-03-18T16:15:37.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:37 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/914724837' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:15:37.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.561+0000 7f8ef5800640 1 -- 172.21.15.39:0/1646131334 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8ef0077d80 msgr2=0x7f8ef0078620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:37.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.561+0000 7f8ef5800640 1 --2- 172.21.15.39:0/1646131334 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8ef0077d80 0x7f8ef0078620 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f8ee8009ca0 tx=0x7f8ee802c120 comp rx=0 tx=0).stop 2024-03-18T16:15:37.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.561+0000 7f8ef5800640 1 -- 172.21.15.39:0/1646131334 shutdown_connections 2024-03-18T16:15:37.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.561+0000 7f8ef5800640 1 --2- 172.21.15.39:0/1646131334 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8ef0077d80 0x7f8ef0078620 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.561+0000 7f8ef5800640 1 --2- 172.21.15.39:0/1646131334 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8ef00773e0 0x7f8ef0077840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.561+0000 7f8ef5800640 1 --2- 172.21.15.39:0/1646131334 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8ef00761a0 0x7f8ef00765c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.561+0000 7f8ef5800640 1 -- 172.21.15.39:0/1646131334 >> 172.21.15.39:0/1646131334 conn(0x7f8ef00fefa0 msgr2=0x7f8ef01013c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:37.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.562+0000 7f8ef5800640 1 -- 172.21.15.39:0/1646131334 shutdown_connections 2024-03-18T16:15:37.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.562+0000 7f8ef5800640 1 -- 172.21.15.39:0/1646131334 wait complete. 2024-03-18T16:15:37.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.563+0000 7f8ef5800640 1 Processor -- start 2024-03-18T16:15:37.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.563+0000 7f8ef5800640 1 -- start start 2024-03-18T16:15:37.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.564+0000 7f8ef5800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8ef00761a0 0x7f8ef019e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:37.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.564+0000 7f8ef5800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8ef00773e0 0x7f8ef019e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:37.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.564+0000 7f8ef5800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8ef0077d80 0x7f8ef01a56a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:37.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.564+0000 7f8ef5800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f8ef0113ae0 con 0x7f8ef00773e0 2024-03-18T16:15:37.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.564+0000 7f8ef5800640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f8ef0113960 con 0x7f8ef00761a0 2024-03-18T16:15:37.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.564+0000 7f8ef5800640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f8ef0113c60 con 0x7f8ef0077d80 2024-03-18T16:15:37.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.564+0000 7f8eef400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8ef0077d80 0x7f8ef01a56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:37.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.564+0000 7f8eeea00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8ef00761a0 0x7f8ef019e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:37.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.564+0000 7f8eef400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8ef0077d80 0x7f8ef01a56a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:35846/0 (socket says 172.21.15.39:35846) 2024-03-18T16:15:37.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.564+0000 7f8eef400640 1 -- 172.21.15.39:0/1006692923 learned_addr learned my addr 172.21.15.39:0/1006692923 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:15:37.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.564+0000 7f8eee000640 1 --2- 172.21.15.39:0/1006692923 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8ef00773e0 0x7f8ef019e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:37.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.565+0000 7f8eeea00640 1 -- 172.21.15.39:0/1006692923 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8ef0077d80 msgr2=0x7f8ef01a56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:37.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.565+0000 7f8eeea00640 1 --2- 172.21.15.39:0/1006692923 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8ef0077d80 0x7f8ef01a56a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.565+0000 7f8eeea00640 1 -- 172.21.15.39:0/1006692923 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8ef00773e0 msgr2=0x7f8ef019e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:37.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.565+0000 7f8eeea00640 1 --2- 172.21.15.39:0/1006692923 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8ef00773e0 0x7f8ef019e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.565+0000 7f8eeea00640 1 -- 172.21.15.39:0/1006692923 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ef01a5d10 con 0x7f8ef00761a0 2024-03-18T16:15:37.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.565+0000 7f8eef400640 1 --2- 172.21.15.39:0/1006692923 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8ef0077d80 0x7f8ef01a56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:15:37.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.565+0000 7f8eee000640 1 --2- 172.21.15.39:0/1006692923 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8ef00773e0 0x7f8ef019e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:15:37.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.565+0000 7f8eeea00640 1 --2- 172.21.15.39:0/1006692923 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8ef00761a0 0x7f8ef019e0f0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f8ee0009fd0 tx=0x7f8ee000b870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:37.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.567+0000 7f8ee7400640 1 -- 172.21.15.39:0/1006692923 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ee0015020 con 0x7f8ef00761a0 2024-03-18T16:15:37.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.567+0000 7f8ee7400640 1 -- 172.21.15.39:0/1006692923 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ee0011d10 con 0x7f8ef00761a0 2024-03-18T16:15:37.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.567+0000 7f8ee7400640 1 -- 172.21.15.39:0/1006692923 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ee000f2e0 con 0x7f8ef00761a0 2024-03-18T16:15:37.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.568+0000 7f8ef5800640 1 -- 172.21.15.39:0/1006692923 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ef01a5fa0 con 0x7f8ef00761a0 2024-03-18T16:15:37.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.568+0000 7f8ef5800640 1 -- 172.21.15.39:0/1006692923 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ef01a6410 con 0x7f8ef00761a0 2024-03-18T16:15:37.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.572+0000 7f8ef8597740 1 Processor -- start 2024-03-18T16:15:37.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.572+0000 7f8ee7400640 1 -- 172.21.15.39:0/1006692923 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f8ee0018020 con 0x7f8ef00761a0 2024-03-18T16:15:37.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.573+0000 7f8ef8597740 1 -- start start 2024-03-18T16:15:37.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.573+0000 7f8ef8597740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555ce38b08c0 0x555ce38b0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:37.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.574+0000 7f8ef8597740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555ce38b12b0 0x555ce38b5db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:37.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.574+0000 7f8ef8597740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555ce38b6420 0x555ce38b88b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:37.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.574+0000 7f8ef8597740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x555ce38783b0 con 0x555ce38b12b0 2024-03-18T16:15:37.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.574+0000 7f8ef8597740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x555ce3815450 con 0x555ce38b08c0 2024-03-18T16:15:37.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.574+0000 7f8ef8597740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x555ce37bdcb0 con 0x555ce38b6420 2024-03-18T16:15:37.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.574+0000 7f8eef400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555ce38b08c0 0x555ce38b0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:37.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.574+0000 7f8eef400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555ce38b08c0 0x555ce38b0ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:53684/0 (socket says 172.21.15.39:53684) 2024-03-18T16:15:37.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.574+0000 7f8eef400640 1 -- 172.21.15.39:0/3865304862 learned_addr learned my addr 172.21.15.39:0/3865304862 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:15:37.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.574+0000 7f8ee7400640 1 --2- 172.21.15.39:0/1006692923 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f8ed40a2900 0x7f8ed40a4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:37.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.575+0000 7f8eee000640 1 --2- 172.21.15.39:0/3865304862 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555ce38b6420 0x555ce38b88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:37.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.575+0000 7f8ee7400640 1 -- 172.21.15.39:0/1006692923 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f8ee00ce4e0 con 0x7f8ef00761a0 2024-03-18T16:15:37.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.575+0000 7f8eef400640 1 --2- 172.21.15.39:0/1006692923 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f8ed40a2900 0x7f8ed40a4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:37.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.575+0000 7f8eee000640 1 -- 172.21.15.39:0/3865304862 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555ce38b08c0 msgr2=0x555ce38b0ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:37.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.575+0000 7f8eee000640 1 --2- 172.21.15.39:0/3865304862 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555ce38b08c0 0x555ce38b0ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.575+0000 7f8eee000640 1 -- 172.21.15.39:0/3865304862 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555ce38b12b0 msgr2=0x555ce38b5db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:37.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.575+0000 7f8eee000640 1 --2- 172.21.15.39:0/3865304862 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555ce38b12b0 0x555ce38b5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.575+0000 7f8eee000640 1 -- 172.21.15.39:0/3865304862 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555ce37df1f0 con 0x555ce38b6420 2024-03-18T16:15:37.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.576+0000 7f8eef400640 1 --2- 172.21.15.39:0/1006692923 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f8ed40a2900 0x7f8ed40a4dc0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x555ce38b1b10 tx=0x7f8ee802c120 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:37.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.576+0000 7f8eee000640 1 --2- 172.21.15.39:0/3865304862 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555ce38b6420 0x555ce38b88b0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f8edc005fd0 tx=0x7f8edc00f730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=50f61d10580531ea server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:37.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.578+0000 7f8edbe00640 1 -- 172.21.15.39:0/3865304862 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8edc005020 con 0x555ce38b6420 2024-03-18T16:15:37.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.578+0000 7f8edbe00640 1 -- 172.21.15.39:0/3865304862 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8edc0047a0 con 0x555ce38b6420 2024-03-18T16:15:37.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.578+0000 7f8edbe00640 1 -- 172.21.15.39:0/3865304862 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8edc011730 con 0x555ce38b6420 2024-03-18T16:15:37.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.579+0000 7f8ef8597740 1 -- 172.21.15.39:0/3865304862 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555ce38b6420 msgr2=0x555ce38b88b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:37.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.579+0000 7f8ef8597740 1 --2- 172.21.15.39:0/3865304862 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555ce38b6420 0x555ce38b88b0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f8edc005fd0 tx=0x7f8edc00f730 comp rx=0 tx=0).stop 2024-03-18T16:15:37.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.579+0000 7f8ef8597740 1 -- 172.21.15.39:0/3865304862 shutdown_connections 2024-03-18T16:15:37.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.579+0000 7f8ef8597740 1 --2- 172.21.15.39:0/3865304862 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555ce38b6420 0x555ce38b88b0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.579+0000 7f8ef8597740 1 --2- 172.21.15.39:0/3865304862 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555ce38b12b0 0x555ce38b5db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.579+0000 7f8ef8597740 1 --2- 172.21.15.39:0/3865304862 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555ce38b08c0 0x555ce38b0ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.579+0000 7f8ef8597740 1 -- 172.21.15.39:0/3865304862 >> 172.21.15.39:0/3865304862 conn(0x555ce38abe30 msgr2=0x555ce38ac230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:37.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.579+0000 7f8ef8597740 1 -- 172.21.15.39:0/3865304862 shutdown_connections 2024-03-18T16:15:37.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.579+0000 7f8ef8597740 1 -- 172.21.15.39:0/3865304862 wait complete. 2024-03-18T16:15:37.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.580+0000 7f8ef8597740 1 -- start start 2024-03-18T16:15:37.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.580+0000 7f8ef8597740 1 Processor -- start 2024-03-18T16:15:37.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.581+0000 7f8ef8597740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555ce38b08c0 0x555ce38bacd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:37.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.581+0000 7f8ef8597740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555ce38b12b0 0x555ce38bb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:37.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.581+0000 7f8ef8597740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555ce38b6420 0x555ce38c21a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:37.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.581+0000 7f8ef8597740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x555ce37d1110 con 0x555ce38b12b0 2024-03-18T16:15:37.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.582+0000 7f8ef8597740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x555ce38114f0 con 0x555ce38b08c0 2024-03-18T16:15:37.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.582+0000 7f8eeea00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555ce38b08c0 0x555ce38bacd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:37.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.582+0000 7f8ef8597740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x555ce38b93d0 con 0x555ce38b6420 2024-03-18T16:15:37.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.582+0000 7f8ef8597740 10 client.0 _ll_register_callbacks cb 0x555ce37beb70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:15:37.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.582+0000 7f8eef400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555ce38b6420 0x555ce38c21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:37.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.582+0000 7f8eef400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555ce38b6420 0x555ce38c21a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:35866/0 (socket says 172.21.15.39:35866) 2024-03-18T16:15:37.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.582+0000 7f8eef400640 1 -- 172.21.15.39:0/2790566127 learned_addr learned my addr 172.21.15.39:0/2790566127 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:15:37.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.582+0000 7f8eee000640 1 --2- 172.21.15.39:0/2790566127 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555ce38b12b0 0x555ce38bb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:37.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.583+0000 7f8eef400640 1 -- 172.21.15.39:0/2790566127 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555ce38b08c0 msgr2=0x555ce38bacd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:37.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.583+0000 7f8eef400640 1 --2- 172.21.15.39:0/2790566127 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555ce38b08c0 0x555ce38bacd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.583+0000 7f8eef400640 1 -- 172.21.15.39:0/2790566127 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555ce38b12b0 msgr2=0x555ce38bb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:37.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.583+0000 7f8eef400640 1 --2- 172.21.15.39:0/2790566127 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555ce38b12b0 0x555ce38bb210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.583+0000 7f8eef400640 1 -- 172.21.15.39:0/2790566127 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555ce35293f0 con 0x555ce38b6420 2024-03-18T16:15:37.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.583+0000 7f8eeea00640 1 --2- 172.21.15.39:0/2790566127 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555ce38b08c0 0x555ce38bacd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:15:37.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.583+0000 7f8eef400640 1 --2- 172.21.15.39:0/2790566127 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555ce38b6420 0x555ce38c21a0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f8ee8009bf0 tx=0x7f8ee80095c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d9bcd83d13412877 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:37.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.583+0000 7f8ed8c00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:15:37.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.583+0000 7f8ed8c00640 1 -- 172.21.15.39:0/2790566127 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ee8053060 con 0x555ce38b6420 2024-03-18T16:15:37.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.584+0000 7f8ed8c00640 1 -- 172.21.15.39:0/2790566127 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f8ee8034c00 con 0x555ce38b6420 2024-03-18T16:15:37.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.584+0000 7f8ef8597740 1 -- 172.21.15.39:0/2790566127 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8edc0047a0 con 0x555ce38b6420 2024-03-18T16:15:37.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:37 smithi039 ceph-mon[31721]: pgmap v588: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 8.7 MiB/s wr, 1.97k op/s 2024-03-18T16:15:37.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:37 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/914724837' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:15:37.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.584+0000 7f8ed8c00640 1 -- 172.21.15.39:0/2790566127 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f8ee805a050 con 0x555ce38b6420 2024-03-18T16:15:37.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.584+0000 7f8ef8597740 10 client.26698 fetch_fsmap learned FSMap version 42 2024-03-18T16:15:37.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.584+0000 7f8ef8597740 1 -- 172.21.15.39:0/2790566127 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x555ce35f37d0 con 0x555ce38b6420 2024-03-18T16:15:37.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.585+0000 7f8ed8c00640 1 -- 172.21.15.39:0/2790566127 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8edc0047a0 con 0x555ce38b6420 2024-03-18T16:15:37.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.585+0000 7f8ed8c00640 1 -- 172.21.15.39:0/2790566127 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 42) ==== 8370+0+0 (secure 0 0 0) 0x7f8ee804e610 con 0x555ce38b6420 2024-03-18T16:15:37.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.586+0000 7f8ef8597740 10 client.26698 fetch_fsmap finished waiting for FSMap version 42 2024-03-18T16:15:37.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.586+0000 7f8ef8597740 10 client.26698 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-18T16:15:37.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.586+0000 7f8ef8597740 20 client.26698 populate_metadata read hostname 'smithi039' 2024-03-18T16:15:37.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.586+0000 7f8ef8597740 1 --2- 172.21.15.39:0/2790566127 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x555ce38c6a10 0x555ce38c8e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:37.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.586+0000 7f8ef8597740 4 client.26698 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:15:37.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.586+0000 7f8ef8597740 1 -- 172.21.15.39:0/2790566127 --> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555ce3822990 con 0x555ce38c6a10 2024-03-18T16:15:37.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.586+0000 7f8eeea00640 1 --2- 172.21.15.39:0/2790566127 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x555ce38c6a10 0x555ce38c8e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:37.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.587+0000 7f8eeea00640 1 --2- 172.21.15.39:0/2790566127 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x555ce38c6a10 0x555ce38c8e80 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ad2719f88f9a6a22 server_cookie=1f54cd6b73a1c55d in_seq=0 out_seq=0 2024-03-18T16:15:37.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.587+0000 7f8ed8c00640 10 client.26698 ms_handle_connect on v2:172.21.15.196:6836/895247251 2024-03-18T16:15:37.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.588+0000 7f8ed8c00640 1 -- 172.21.15.39:0/2790566127 <== mds.2 v2:172.21.15.196:6836/895247251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x555ce3822990 con 0x555ce38c6a10 2024-03-18T16:15:37.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.588+0000 7f8ed8c00640 10 client.26698 handle_command_reply: tid=0 2024-03-18T16:15:37.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.588+0000 7f8ef8597740 1 client.26698 shutdown 2024-03-18T16:15:37.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.588+0000 7f8ef8597740 1 -- 172.21.15.39:0/2790566127 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555ce38b6420 msgr2=0x555ce38c21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:37.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.588+0000 7f8ef8597740 1 --2- 172.21.15.39:0/2790566127 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555ce38b6420 0x555ce38c21a0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f8ee8009bf0 tx=0x7f8ee80095c0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.588+0000 7f8ef8597740 1 -- 172.21.15.39:0/2790566127 shutdown_connections 2024-03-18T16:15:37.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.588+0000 7f8ef8597740 1 --2- 172.21.15.39:0/2790566127 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x555ce38c6a10 0x555ce38c8e80 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.588+0000 7f8ef8597740 1 --2- 172.21.15.39:0/2790566127 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555ce38b6420 0x555ce38c21a0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.588+0000 7f8ef8597740 1 --2- 172.21.15.39:0/2790566127 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555ce38b12b0 0x555ce38bb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.588+0000 7f8ef8597740 1 --2- 172.21.15.39:0/2790566127 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555ce38b08c0 0x555ce38bacd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.588+0000 7f8ef8597740 1 -- 172.21.15.39:0/2790566127 >> 172.21.15.39:0/2790566127 conn(0x555ce38abe30 msgr2=0x555ce38ab1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:37.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.588+0000 7f8ed8c00640 0 client.26698 ms_handle_reset on v2:172.21.15.196:6836/895247251 2024-03-18T16:15:37.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.589+0000 7f8ef8597740 1 -- 172.21.15.39:0/2790566127 shutdown_connections 2024-03-18T16:15:37.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.589+0000 7f8ef8597740 1 -- 172.21.15.39:0/2790566127 wait complete. 2024-03-18T16:15:37.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.589+0000 7f8ef8597740 20 client.26698 trim_cache size 0 max 16384 2024-03-18T16:15:37.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.611+0000 7f8ef8597740 1 Processor -- start 2024-03-18T16:15:37.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.611+0000 7f8ef8597740 1 -- start start 2024-03-18T16:15:37.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.611+0000 7f8ef8597740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555ce38b08c0 0x555ce38b27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:37.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.611+0000 7f8ef8597740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555ce38b12b0 0x555ce38c2160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:37.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.611+0000 7f8ef8597740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555ce38b6420 0x555ce38b2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:37.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.611+0000 7f8ef8597740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f8ef01b0b30 con 0x555ce38b6420 2024-03-18T16:15:37.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.611+0000 7f8ef8597740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f8ef01b2670 con 0x555ce38b12b0 2024-03-18T16:15:37.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.611+0000 7f8ef8597740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f8ef01b1200 con 0x555ce38b08c0 2024-03-18T16:15:37.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.612+0000 7f8eee000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555ce38b12b0 0x555ce38c2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:37.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.612+0000 7f8eee000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555ce38b12b0 0x555ce38c2160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:53696/0 (socket says 172.21.15.39:53696) 2024-03-18T16:15:37.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.612+0000 7f8eee000640 1 -- 172.21.15.39:0/449132797 learned_addr learned my addr 172.21.15.39:0/449132797 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:15:37.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.612+0000 7f8eeea00640 1 --2- 172.21.15.39:0/449132797 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555ce38b08c0 0x555ce38b27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:37.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.612+0000 7f8eef400640 1 --2- 172.21.15.39:0/449132797 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555ce38b6420 0x555ce38b2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:37.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.613+0000 7f8eef400640 1 -- 172.21.15.39:0/449132797 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555ce38b08c0 msgr2=0x555ce38b27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:37.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.613+0000 7f8eef400640 1 --2- 172.21.15.39:0/449132797 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555ce38b08c0 0x555ce38b27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.613+0000 7f8eef400640 1 -- 172.21.15.39:0/449132797 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555ce38b12b0 msgr2=0x555ce38c2160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:37.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.613+0000 7f8eef400640 1 --2- 172.21.15.39:0/449132797 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555ce38b12b0 0x555ce38c2160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.613+0000 7f8eef400640 1 -- 172.21.15.39:0/449132797 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555ce38a7c10 con 0x555ce38b6420 2024-03-18T16:15:37.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.613+0000 7f8eef400640 1 --2- 172.21.15.39:0/449132797 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555ce38b6420 0x555ce38b2cf0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f8ee804ea60 tx=0x7f8ee802c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4ea1fdb79fe44e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:37.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.614+0000 7f8edb400640 1 -- 172.21.15.39:0/449132797 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ee8053070 con 0x555ce38b6420 2024-03-18T16:15:37.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.614+0000 7f8edb400640 1 -- 172.21.15.39:0/449132797 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ee8032ab0 con 0x555ce38b6420 2024-03-18T16:15:37.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.614+0000 7f8edb400640 1 -- 172.21.15.39:0/449132797 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ee8047070 con 0x555ce38b6420 2024-03-18T16:15:37.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.615+0000 7f8ef8597740 1 -- 172.21.15.39:0/449132797 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555ce38b6420 msgr2=0x555ce38b2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:37.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.615+0000 7f8ef8597740 1 --2- 172.21.15.39:0/449132797 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555ce38b6420 0x555ce38b2cf0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f8ee804ea60 tx=0x7f8ee802c820 comp rx=0 tx=0).stop 2024-03-18T16:15:37.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.616+0000 7f8ef8597740 1 -- 172.21.15.39:0/449132797 shutdown_connections 2024-03-18T16:15:37.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.616+0000 7f8ef8597740 1 --2- 172.21.15.39:0/449132797 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555ce38b12b0 0x555ce38c2160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.616+0000 7f8ef8597740 1 --2- 172.21.15.39:0/449132797 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555ce38b6420 0x555ce38b2cf0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.616+0000 7f8ef8597740 1 --2- 172.21.15.39:0/449132797 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555ce38b08c0 0x555ce38b27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.616+0000 7f8ef8597740 1 -- 172.21.15.39:0/449132797 >> 172.21.15.39:0/449132797 conn(0x555ce38abe30 msgr2=0x555ce38b97a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:37.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.616+0000 7f8ef8597740 1 -- 172.21.15.39:0/449132797 shutdown_connections 2024-03-18T16:15:37.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.616+0000 7f8ef8597740 1 -- 172.21.15.39:0/449132797 wait complete. 2024-03-18T16:15:37.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.617+0000 7f8ef8597740 1 -- start start 2024-03-18T16:15:37.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.617+0000 7f8ef8597740 1 Processor -- start 2024-03-18T16:15:37.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.617+0000 7f8ef8597740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555ce38b08c0 0x555ce38bf6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:37.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.618+0000 7f8ef8597740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555ce38b12b0 0x555ce38bfc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:37.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.618+0000 7f8ef8597740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555ce38b6420 0x555ce38c0b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:37.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.618+0000 7f8ef8597740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f8ef01b1fa0 con 0x555ce38b08c0 2024-03-18T16:15:37.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.618+0000 7f8ef8597740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f8ef01b18d0 con 0x555ce38b12b0 2024-03-18T16:15:37.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.618+0000 7f8ef8597740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f8ef01aeb10 con 0x555ce38b6420 2024-03-18T16:15:37.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.618+0000 7f8ef8597740 10 client.0 _ll_register_callbacks cb 0x7f8ee8005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:15:37.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.618+0000 7f8eee000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555ce38b12b0 0x555ce38bfc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:37.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.618+0000 7f8eee000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555ce38b12b0 0x555ce38bfc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:53710/0 (socket says 172.21.15.39:53710) 2024-03-18T16:15:37.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.618+0000 7f8eee000640 1 -- 172.21.15.39:0/2211771009 learned_addr learned my addr 172.21.15.39:0/2211771009 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:15:37.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.619+0000 7f8eee000640 1 -- 172.21.15.39:0/2211771009 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555ce38b6420 msgr2=0x555ce38c0b60 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T16:15:37.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.619+0000 7f8eee000640 1 --2- 172.21.15.39:0/2211771009 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555ce38b6420 0x555ce38c0b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.619+0000 7f8eee000640 1 -- 172.21.15.39:0/2211771009 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555ce38b08c0 msgr2=0x555ce38bf6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:37.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.619+0000 7f8eee000640 1 --2- 172.21.15.39:0/2211771009 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555ce38b08c0 0x555ce38bf6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.619+0000 7f8eee000640 1 -- 172.21.15.39:0/2211771009 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555ce38c1150 con 0x555ce38b12b0 2024-03-18T16:15:37.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.619+0000 7f8eee000640 1 --2- 172.21.15.39:0/2211771009 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555ce38b12b0 0x555ce38bfc30 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f8edc00ba60 tx=0x7f8edc0052e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1eb429f9918f716a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:37.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.619+0000 7f8ed8c00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:15:37.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.620+0000 7f8ed8c00640 1 -- 172.21.15.39:0/2211771009 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8edc02b060 con 0x555ce38b12b0 2024-03-18T16:15:37.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.620+0000 7f8ed8c00640 1 -- 172.21.15.39:0/2211771009 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f8edc0037f0 con 0x555ce38b12b0 2024-03-18T16:15:37.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.620+0000 7f8ef8597740 1 -- 172.21.15.39:0/2211771009 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8ee8032ab0 con 0x555ce38b12b0 2024-03-18T16:15:37.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.620+0000 7f8ed8c00640 1 -- 172.21.15.39:0/2211771009 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f8edc0039f0 con 0x555ce38b12b0 2024-03-18T16:15:37.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.620+0000 7f8ef8597740 10 client.26741 fetch_fsmap learned FSMap version 42 2024-03-18T16:15:37.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.620+0000 7f8ef8597740 1 -- 172.21.15.39:0/2211771009 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x555ce38c24f0 con 0x555ce38b12b0 2024-03-18T16:15:37.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.621+0000 7f8ed8c00640 1 -- 172.21.15.39:0/2211771009 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ee8032ab0 con 0x555ce38b12b0 2024-03-18T16:15:37.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.621+0000 7f8ed8c00640 1 -- 172.21.15.39:0/2211771009 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 42) ==== 8370+0+0 (secure 0 0 0) 0x7f8edc011c20 con 0x555ce38b12b0 2024-03-18T16:15:37.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.621+0000 7f8ef8597740 10 client.26741 fetch_fsmap finished waiting for FSMap version 42 2024-03-18T16:15:37.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.621+0000 7f8ef8597740 10 client.26741 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-18T16:15:37.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.621+0000 7f8ef8597740 20 client.26741 populate_metadata read hostname 'smithi039' 2024-03-18T16:15:37.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.621+0000 7f8ef8597740 1 --2- 172.21.15.39:0/2211771009 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x555ce392d260 0x555ce392f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:37.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.621+0000 7f8ef8597740 4 client.26741 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:15:37.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.621+0000 7f8ef8597740 1 -- 172.21.15.39:0/2211771009 --> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555ce381cce0 con 0x555ce392d260 2024-03-18T16:15:37.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.622+0000 7f8eeea00640 1 --2- 172.21.15.39:0/2211771009 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x555ce392d260 0x555ce392f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:37.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.622+0000 7f8eeea00640 1 --2- 172.21.15.39:0/2211771009 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x555ce392d260 0x555ce392f680 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d844fd00516bd1f9 server_cookie=f8c740225f6e02cb in_seq=0 out_seq=0 2024-03-18T16:15:37.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.622+0000 7f8ed8c00640 10 client.26741 ms_handle_connect on v2:172.21.15.196:6836/895247251 2024-03-18T16:15:37.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.624+0000 7f8ed8c00640 1 -- 172.21.15.39:0/2211771009 <== mds.2 v2:172.21.15.196:6836/895247251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555ce381cce0 con 0x555ce392d260 2024-03-18T16:15:37.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.624+0000 7f8ed8c00640 10 client.26741 handle_command_reply: tid=0 2024-03-18T16:15:37.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.624+0000 7f8ef8597740 1 client.26741 shutdown 2024-03-18T16:15:37.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.624+0000 7f8ef8597740 1 -- 172.21.15.39:0/2211771009 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555ce38b12b0 msgr2=0x555ce38bfc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:37.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.624+0000 7f8ef8597740 1 --2- 172.21.15.39:0/2211771009 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555ce38b12b0 0x555ce38bfc30 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f8edc00ba60 tx=0x7f8edc0052e0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.625+0000 7f8ef8597740 1 -- 172.21.15.39:0/2211771009 shutdown_connections 2024-03-18T16:15:37.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.625+0000 7f8ef8597740 1 --2- 172.21.15.39:0/2211771009 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555ce38b12b0 0x555ce38bfc30 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.625+0000 7f8ef8597740 1 --2- 172.21.15.39:0/2211771009 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x555ce392d260 0x555ce392f680 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.625+0000 7f8ef8597740 1 --2- 172.21.15.39:0/2211771009 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555ce38b6420 0x555ce38c0b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.625+0000 7f8ef8597740 1 --2- 172.21.15.39:0/2211771009 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555ce38b08c0 0x555ce38bf6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.625+0000 7f8ef8597740 1 -- 172.21.15.39:0/2211771009 >> 172.21.15.39:0/2211771009 conn(0x555ce38abe30 msgr2=0x555ce38bac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:37.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.625+0000 7f8ed8c00640 0 client.26741 ms_handle_reset on v2:172.21.15.196:6836/895247251 2024-03-18T16:15:37.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.625+0000 7f8ef8597740 1 -- 172.21.15.39:0/2211771009 shutdown_connections 2024-03-18T16:15:37.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.625+0000 7f8ef8597740 1 -- 172.21.15.39:0/2211771009 wait complete. 2024-03-18T16:15:37.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.625+0000 7f8ef8597740 20 client.26741 trim_cache size 0 max 16384 2024-03-18T16:15:37.629 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:15:37.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.626+0000 7f8ef5800640 1 -- 172.21.15.39:0/1006692923 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f8ed40a2900 msgr2=0x7f8ed40a4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:37.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.626+0000 7f8ef5800640 1 --2- 172.21.15.39:0/1006692923 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f8ed40a2900 0x7f8ed40a4dc0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x555ce38b1b10 tx=0x7f8ee802c120 comp rx=0 tx=0).stop 2024-03-18T16:15:37.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.626+0000 7f8ef5800640 1 -- 172.21.15.39:0/1006692923 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8ef00761a0 msgr2=0x7f8ef019e0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:37.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.626+0000 7f8ef5800640 1 --2- 172.21.15.39:0/1006692923 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8ef00761a0 0x7f8ef019e0f0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f8ee0009fd0 tx=0x7f8ee000b870 comp rx=0 tx=0).stop 2024-03-18T16:15:37.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.626+0000 7f8ef5800640 1 -- 172.21.15.39:0/1006692923 shutdown_connections 2024-03-18T16:15:37.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.627+0000 7f8ef5800640 1 --2- 172.21.15.39:0/1006692923 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f8ed40a2900 0x7f8ed40a4dc0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.627+0000 7f8ef5800640 1 --2- 172.21.15.39:0/1006692923 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8ef0077d80 0x7f8ef01a56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.627+0000 7f8ef5800640 1 --2- 172.21.15.39:0/1006692923 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8ef00773e0 0x7f8ef019e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.627+0000 7f8ef5800640 1 --2- 172.21.15.39:0/1006692923 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8ef00761a0 0x7f8ef019e0f0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.627+0000 7f8ef5800640 1 -- 172.21.15.39:0/1006692923 >> 172.21.15.39:0/1006692923 conn(0x7f8ef00fefa0 msgr2=0x7f8ef0100aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:37.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.627+0000 7f8ef5800640 1 -- 172.21.15.39:0/1006692923 shutdown_connections 2024-03-18T16:15:37.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.627+0000 7f8ef5800640 1 -- 172.21.15.39:0/1006692923 wait complete. 2024-03-18T16:15:37.652 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-18T16:15:37.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.909+0000 7fe456400640 1 -- 172.21.15.39:0/2962411183 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe4500761a0 msgr2=0x7fe4500765c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:37.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.909+0000 7fe456400640 1 --2- 172.21.15.39:0/2962411183 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe4500761a0 0x7fe4500765c0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fe440009900 tx=0x7fe44002bc30 comp rx=0 tx=0).stop 2024-03-18T16:15:37.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.909+0000 7fe456400640 1 -- 172.21.15.39:0/2962411183 shutdown_connections 2024-03-18T16:15:37.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.909+0000 7fe456400640 1 --2- 172.21.15.39:0/2962411183 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe450077d80 0x7fe450078620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.909+0000 7fe456400640 1 --2- 172.21.15.39:0/2962411183 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe4500773e0 0x7fe450077840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.909+0000 7fe456400640 1 --2- 172.21.15.39:0/2962411183 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe4500761a0 0x7fe4500765c0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.909+0000 7fe456400640 1 -- 172.21.15.39:0/2962411183 >> 172.21.15.39:0/2962411183 conn(0x7fe4500fef90 msgr2=0x7fe4501013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:37.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.910+0000 7fe456400640 1 -- 172.21.15.39:0/2962411183 shutdown_connections 2024-03-18T16:15:37.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.910+0000 7fe456400640 1 -- 172.21.15.39:0/2962411183 wait complete. 2024-03-18T16:15:37.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.911+0000 7fe456400640 1 Processor -- start 2024-03-18T16:15:37.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.911+0000 7fe456400640 1 -- start start 2024-03-18T16:15:37.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.912+0000 7fe456400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe4500761a0 0x7fe45019e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:37.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.912+0000 7fe456400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe4500773e0 0x7fe45019e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:37.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.912+0000 7fe456400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe450077d80 0x7fe4501a5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:37.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.912+0000 7fe456400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fe450113ad0 con 0x7fe4500773e0 2024-03-18T16:15:37.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.912+0000 7fe456400640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fe450113950 con 0x7fe4500761a0 2024-03-18T16:15:37.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.912+0000 7fe456400640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fe450113c50 con 0x7fe450077d80 2024-03-18T16:15:37.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.912+0000 7fe44fe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe450077d80 0x7fe4501a5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:37.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.912+0000 7fe44fe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe450077d80 0x7fe4501a5670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:35894/0 (socket says 172.21.15.39:35894) 2024-03-18T16:15:37.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.912+0000 7fe44fe00640 1 -- 172.21.15.39:0/773184227 learned_addr learned my addr 172.21.15.39:0/773184227 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:15:37.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.912+0000 7fe44f400640 1 --2- 172.21.15.39:0/773184227 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe4500761a0 0x7fe45019e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:37.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.912+0000 7fe44ea00640 1 --2- 172.21.15.39:0/773184227 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe4500773e0 0x7fe45019e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:37.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.913+0000 7fe44f400640 1 -- 172.21.15.39:0/773184227 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe450077d80 msgr2=0x7fe4501a5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:37.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.913+0000 7fe44f400640 1 --2- 172.21.15.39:0/773184227 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe450077d80 0x7fe4501a5670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.913+0000 7fe44f400640 1 -- 172.21.15.39:0/773184227 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe4500773e0 msgr2=0x7fe45019e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:37.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.913+0000 7fe44f400640 1 --2- 172.21.15.39:0/773184227 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe4500773e0 0x7fe45019e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.913+0000 7fe44f400640 1 -- 172.21.15.39:0/773184227 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4501a5ce0 con 0x7fe4500761a0 2024-03-18T16:15:37.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.913+0000 7fe44f400640 1 --2- 172.21.15.39:0/773184227 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe4500761a0 0x7fe45019e0c0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fe440036040 tx=0x7fe440005dd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:37.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.914+0000 7fe447e00640 1 -- 172.21.15.39:0/773184227 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe440043020 con 0x7fe4500761a0 2024-03-18T16:15:37.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.914+0000 7fe447e00640 1 -- 172.21.15.39:0/773184227 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe440004410 con 0x7fe4500761a0 2024-03-18T16:15:37.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.914+0000 7fe456400640 1 -- 172.21.15.39:0/773184227 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4501a5f70 con 0x7fe4500761a0 2024-03-18T16:15:37.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.914+0000 7fe456400640 1 -- 172.21.15.39:0/773184227 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4501a63e0 con 0x7fe4500761a0 2024-03-18T16:15:37.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.914+0000 7fe447e00640 1 -- 172.21.15.39:0/773184227 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe440003df0 con 0x7fe4500761a0 2024-03-18T16:15:37.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.919+0000 7fe447e00640 1 -- 172.21.15.39:0/773184227 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fe440046020 con 0x7fe4500761a0 2024-03-18T16:15:37.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.919+0000 7fe459170740 1 Processor -- start 2024-03-18T16:15:37.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.920+0000 7fe459170740 1 -- start start 2024-03-18T16:15:37.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.921+0000 7fe459170740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f0f18478c0 0x55f0f1847ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:37.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.921+0000 7fe459170740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f0f18482b0 0x55f0f184cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:37.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.921+0000 7fe459170740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f0f184d420 0x55f0f184f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:37.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.921+0000 7fe459170740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55f0f180f3b0 con 0x55f0f18478c0 2024-03-18T16:15:37.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.921+0000 7fe459170740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55f0f17ac450 con 0x55f0f184d420 2024-03-18T16:15:37.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.921+0000 7fe459170740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55f0f1754cb0 con 0x55f0f18482b0 2024-03-18T16:15:37.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.921+0000 7fe44ea00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f0f184d420 0x55f0f184f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:37.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.921+0000 7fe44ea00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f0f184d420 0x55f0f184f8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:53738/0 (socket says 172.21.15.39:53738) 2024-03-18T16:15:37.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.921+0000 7fe44ea00640 1 -- 172.21.15.39:0/1324915648 learned_addr learned my addr 172.21.15.39:0/1324915648 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:15:37.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.921+0000 7fe44f400640 1 --2- 172.21.15.39:0/1324915648 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f0f18482b0 0x55f0f184cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:37.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.922+0000 7fe44f400640 1 -- 172.21.15.39:0/1324915648 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f0f184d420 msgr2=0x55f0f184f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:37.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.922+0000 7fe44f400640 1 --2- 172.21.15.39:0/1324915648 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f0f184d420 0x55f0f184f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.922+0000 7fe44f400640 1 -- 172.21.15.39:0/1324915648 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f0f18478c0 msgr2=0x55f0f1847ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:37.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.922+0000 7fe44f400640 1 --2- 172.21.15.39:0/1324915648 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f0f18478c0 0x55f0f1847ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.922+0000 7fe44f400640 1 -- 172.21.15.39:0/1324915648 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f0f17761f0 con 0x55f0f18482b0 2024-03-18T16:15:37.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.922+0000 7fe44ea00640 1 --2- 172.21.15.39:0/1324915648 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f0f184d420 0x55f0f184f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:15:37.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.922+0000 7fe447e00640 1 --2- 172.21.15.39:0/773184227 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fe4340a2900 0x7fe4340a4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:37.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.923+0000 7fe44f400640 1 --2- 172.21.15.39:0/1324915648 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f0f18482b0 0x55f0f184cdb0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fe4400ee560 tx=0x7fe4400f4de0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ba42526bc7672c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:37.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.923+0000 7fe444c00640 1 -- 172.21.15.39:0/1324915648 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4400b9700 con 0x55f0f18482b0 2024-03-18T16:15:37.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.923+0000 7fe44fe00640 1 --2- 172.21.15.39:0/773184227 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fe4340a2900 0x7fe4340a4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:37.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.923+0000 7fe444c00640 1 -- 172.21.15.39:0/1324915648 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe4400bdb60 con 0x55f0f18482b0 2024-03-18T16:15:37.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.923+0000 7fe444c00640 1 -- 172.21.15.39:0/1324915648 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4400bcdc0 con 0x55f0f18482b0 2024-03-18T16:15:37.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.923+0000 7fe447e00640 1 -- 172.21.15.39:0/773184227 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fe4400f0eb0 con 0x7fe4500761a0 2024-03-18T16:15:37.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.924+0000 7fe459170740 1 -- 172.21.15.39:0/1324915648 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f0f18482b0 msgr2=0x55f0f184cdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:37.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.924+0000 7fe459170740 1 --2- 172.21.15.39:0/1324915648 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f0f18482b0 0x55f0f184cdb0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fe4400ee560 tx=0x7fe4400f4de0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.924+0000 7fe459170740 1 -- 172.21.15.39:0/1324915648 shutdown_connections 2024-03-18T16:15:37.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.924+0000 7fe459170740 1 --2- 172.21.15.39:0/1324915648 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f0f184d420 0x55f0f184f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.924+0000 7fe459170740 1 --2- 172.21.15.39:0/1324915648 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f0f18482b0 0x55f0f184cdb0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.924+0000 7fe459170740 1 --2- 172.21.15.39:0/1324915648 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f0f18478c0 0x55f0f1847ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.924+0000 7fe459170740 1 -- 172.21.15.39:0/1324915648 >> 172.21.15.39:0/1324915648 conn(0x55f0f1842e30 msgr2=0x55f0f1843230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:37.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.925+0000 7fe44fe00640 1 --2- 172.21.15.39:0/773184227 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fe4340a2900 0x7fe4340a4dc0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x55f0f1848b10 tx=0x7fe448009bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:37.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.925+0000 7fe459170740 1 -- 172.21.15.39:0/1324915648 shutdown_connections 2024-03-18T16:15:37.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.925+0000 7fe459170740 1 -- 172.21.15.39:0/1324915648 wait complete. 2024-03-18T16:15:37.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.925+0000 7fe459170740 1 -- start start 2024-03-18T16:15:37.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.926+0000 7fe459170740 1 Processor -- start 2024-03-18T16:15:37.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.927+0000 7fe459170740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f0f18478c0 0x55f0f1851830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:37.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.927+0000 7fe459170740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f0f18482b0 0x55f0f1851d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:37.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.927+0000 7fe459170740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f0f184d420 0x55f0f1858db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:37.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.927+0000 7fe44f400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f0f18478c0 0x55f0f1851830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:37.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.927+0000 7fe44f400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f0f18478c0 0x55f0f1851830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:53754/0 (socket says 172.21.15.39:53754) 2024-03-18T16:15:37.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.927+0000 7fe44fe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f0f184d420 0x55f0f1858db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:37.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.927+0000 7fe44fe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f0f184d420 0x55f0f1858db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:49554/0 (socket says 172.21.15.39:49554) 2024-03-18T16:15:37.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.928+0000 7fe44ea00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f0f18482b0 0x55f0f1851d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:37.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.928+0000 7fe459170740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55f0f1768110 con 0x55f0f184d420 2024-03-18T16:15:37.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.928+0000 7fe459170740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55f0f17a84f0 con 0x55f0f18478c0 2024-03-18T16:15:37.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.928+0000 7fe459170740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55f0f1850190 con 0x55f0f18482b0 2024-03-18T16:15:37.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.928+0000 7fe44ea00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f0f18482b0 0x55f0f1851d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:35904/0 (socket says 172.21.15.39:35904) 2024-03-18T16:15:37.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.928+0000 7fe44ea00640 1 -- 172.21.15.39:0/1142950947 learned_addr learned my addr 172.21.15.39:0/1142950947 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:15:37.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.928+0000 7fe459170740 10 client.0 _ll_register_callbacks cb 0x55f0f1755b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:15:37.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.928+0000 7fe44ea00640 1 -- 172.21.15.39:0/1142950947 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f0f18478c0 msgr2=0x55f0f1851830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:37.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.928+0000 7fe44ea00640 1 --2- 172.21.15.39:0/1142950947 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f0f18478c0 0x55f0f1851830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.928+0000 7fe44ea00640 1 -- 172.21.15.39:0/1142950947 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f0f184d420 msgr2=0x55f0f1858db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:37.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.929+0000 7fe44ea00640 1 --2- 172.21.15.39:0/1142950947 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f0f184d420 0x55f0f1858db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.929+0000 7fe44ea00640 1 -- 172.21.15.39:0/1142950947 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f0f14c03f0 con 0x55f0f18482b0 2024-03-18T16:15:37.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.929+0000 7fe44ea00640 1 --2- 172.21.15.39:0/1142950947 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f0f18482b0 0x55f0f1851d70 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fe43c00ba10 tx=0x7fe43c00bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dd78c1d2501a3203 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:37.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.929+0000 7fe439600640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:15:37.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.930+0000 7fe44f400640 1 --2- 172.21.15.39:0/1142950947 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f0f18478c0 0x55f0f1851830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:15:37.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.930+0000 7fe44fe00640 1 --2- 172.21.15.39:0/1142950947 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f0f184d420 0x55f0f1858db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:15:37.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.930+0000 7fe439600640 1 -- 172.21.15.39:0/1142950947 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe43c003e20 con 0x55f0f18482b0 2024-03-18T16:15:37.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.930+0000 7fe439600640 1 -- 172.21.15.39:0/1142950947 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fe43c011440 con 0x55f0f18482b0 2024-03-18T16:15:37.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.930+0000 7fe459170740 1 -- 172.21.15.39:0/1142950947 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe4400bdb60 con 0x55f0f18482b0 2024-03-18T16:15:37.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.930+0000 7fe439600640 1 -- 172.21.15.39:0/1142950947 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7fe43c011640 con 0x55f0f18482b0 2024-03-18T16:15:37.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.930+0000 7fe459170740 10 client.26713 fetch_fsmap learned FSMap version 42 2024-03-18T16:15:37.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.930+0000 7fe459170740 1 -- 172.21.15.39:0/1142950947 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55f0f158a7d0 con 0x55f0f18482b0 2024-03-18T16:15:37.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.931+0000 7fe439600640 1 -- 172.21.15.39:0/1142950947 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe4400bdb60 con 0x55f0f18482b0 2024-03-18T16:15:37.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.931+0000 7fe439600640 1 -- 172.21.15.39:0/1142950947 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 42) ==== 8370+0+0 (secure 0 0 0) 0x7fe43c0109f0 con 0x55f0f18482b0 2024-03-18T16:15:37.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.931+0000 7fe459170740 10 client.26713 fetch_fsmap finished waiting for FSMap version 42 2024-03-18T16:15:37.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.931+0000 7fe459170740 10 client.26713 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T16:15:37.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.931+0000 7fe459170740 20 client.26713 populate_metadata read hostname 'smithi039' 2024-03-18T16:15:37.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.931+0000 7fe459170740 1 --2- 172.21.15.39:0/1142950947 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x55f0f185d6e0 0x55f0f185fba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:37.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.932+0000 7fe459170740 4 client.26713 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:15:37.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.932+0000 7fe459170740 1 -- 172.21.15.39:0/1142950947 --> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f0f17b9990 con 0x55f0f185d6e0 2024-03-18T16:15:37.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.932+0000 7fe44f400640 1 --2- 172.21.15.39:0/1142950947 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x55f0f185d6e0 0x55f0f185fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:37.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.933+0000 7fe439600640 10 client.26713 ms_handle_connect on v2:172.21.15.196:6838/2001427268 2024-03-18T16:15:37.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.933+0000 7fe44f400640 1 --2- 172.21.15.39:0/1142950947 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x55f0f185d6e0 0x55f0f185fba0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=77046f91ac63842 server_cookie=169dbf47c27b7baa in_seq=0 out_seq=0 2024-03-18T16:15:37.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.934+0000 7fe439600640 1 -- 172.21.15.39:0/1142950947 <== mds.1 v2:172.21.15.196:6838/2001427268 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f0f17b9990 con 0x55f0f185d6e0 2024-03-18T16:15:37.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.934+0000 7fe439600640 10 client.26713 handle_command_reply: tid=0 2024-03-18T16:15:37.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.934+0000 7fe459170740 1 client.26713 shutdown 2024-03-18T16:15:37.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.935+0000 7fe459170740 1 -- 172.21.15.39:0/1142950947 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f0f18482b0 msgr2=0x55f0f1851d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:37.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.935+0000 7fe459170740 1 --2- 172.21.15.39:0/1142950947 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f0f18482b0 0x55f0f1851d70 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fe43c00ba10 tx=0x7fe43c00bed0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.935+0000 7fe459170740 1 -- 172.21.15.39:0/1142950947 shutdown_connections 2024-03-18T16:15:37.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.935+0000 7fe459170740 1 --2- 172.21.15.39:0/1142950947 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x55f0f185d6e0 0x55f0f185fba0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.935+0000 7fe439600640 0 client.26713 ms_handle_reset on v2:172.21.15.196:6838/2001427268 2024-03-18T16:15:37.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.935+0000 7fe459170740 1 --2- 172.21.15.39:0/1142950947 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f0f184d420 0x55f0f1858db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.935+0000 7fe459170740 1 --2- 172.21.15.39:0/1142950947 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f0f18482b0 0x55f0f1851d70 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.935+0000 7fe459170740 1 --2- 172.21.15.39:0/1142950947 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f0f18478c0 0x55f0f1851830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.935+0000 7fe459170740 1 -- 172.21.15.39:0/1142950947 >> 172.21.15.39:0/1142950947 conn(0x55f0f1842e30 msgr2=0x55f0f18421f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:37.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.936+0000 7fe459170740 1 -- 172.21.15.39:0/1142950947 shutdown_connections 2024-03-18T16:15:37.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.936+0000 7fe459170740 1 -- 172.21.15.39:0/1142950947 wait complete. 2024-03-18T16:15:37.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.936+0000 7fe459170740 20 client.26713 trim_cache size 0 max 16384 2024-03-18T16:15:37.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.965+0000 7fe459170740 1 Processor -- start 2024-03-18T16:15:37.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.965+0000 7fe459170740 1 -- start start 2024-03-18T16:15:37.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.965+0000 7fe459170740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f0f18478c0 0x55f0f1849710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:37.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.965+0000 7fe459170740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f0f18482b0 0x55f0f1849c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:37.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.965+0000 7fe459170740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f0f184d420 0x55f0f1858d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:37.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.965+0000 7fe459170740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fe4501b0a20 con 0x55f0f18478c0 2024-03-18T16:15:37.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.965+0000 7fe459170740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fe4501b2560 con 0x55f0f184d420 2024-03-18T16:15:37.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.965+0000 7fe459170740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fe4501b10f0 con 0x55f0f18482b0 2024-03-18T16:15:37.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.966+0000 7fe44ea00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f0f18482b0 0x55f0f1849c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:37.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.966+0000 7fe44ea00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f0f18482b0 0x55f0f1849c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:35906/0 (socket says 172.21.15.39:35906) 2024-03-18T16:15:37.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.966+0000 7fe44ea00640 1 -- 172.21.15.39:0/107036395 learned_addr learned my addr 172.21.15.39:0/107036395 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:15:37.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.966+0000 7fe44fe00640 1 --2- 172.21.15.39:0/107036395 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f0f184d420 0x55f0f1858d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:37.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.966+0000 7fe44f400640 1 --2- 172.21.15.39:0/107036395 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f0f18478c0 0x55f0f1849710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:37.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.966+0000 7fe44ea00640 1 -- 172.21.15.39:0/107036395 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f0f184d420 msgr2=0x55f0f1858d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:37.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.966+0000 7fe44ea00640 1 --2- 172.21.15.39:0/107036395 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f0f184d420 0x55f0f1858d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.966+0000 7fe44ea00640 1 -- 172.21.15.39:0/107036395 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f0f18478c0 msgr2=0x55f0f1849710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:37.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.966+0000 7fe44ea00640 1 --2- 172.21.15.39:0/107036395 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f0f18478c0 0x55f0f1849710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.966+0000 7fe44ea00640 1 -- 172.21.15.39:0/107036395 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f0f183ec10 con 0x55f0f18482b0 2024-03-18T16:15:37.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.967+0000 7fe44ea00640 1 --2- 172.21.15.39:0/107036395 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f0f18482b0 0x55f0f1849c50 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fe43c000c00 tx=0x7fe43c005fb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a9e9ba2db7d7d521 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:37.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.967+0000 7fe43be00640 1 -- 172.21.15.39:0/107036395 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe43c02d020 con 0x55f0f18482b0 2024-03-18T16:15:37.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.967+0000 7fe43be00640 1 -- 172.21.15.39:0/107036395 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe43c011440 con 0x55f0f18482b0 2024-03-18T16:15:37.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.967+0000 7fe43be00640 1 -- 172.21.15.39:0/107036395 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe43c004680 con 0x55f0f18482b0 2024-03-18T16:15:37.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.968+0000 7fe459170740 1 -- 172.21.15.39:0/107036395 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f0f18482b0 msgr2=0x55f0f1849c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:37.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.968+0000 7fe459170740 1 --2- 172.21.15.39:0/107036395 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f0f18482b0 0x55f0f1849c50 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fe43c000c00 tx=0x7fe43c005fb0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.968+0000 7fe459170740 1 -- 172.21.15.39:0/107036395 shutdown_connections 2024-03-18T16:15:37.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.968+0000 7fe459170740 1 --2- 172.21.15.39:0/107036395 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f0f184d420 0x55f0f1858d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.968+0000 7fe459170740 1 --2- 172.21.15.39:0/107036395 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f0f18482b0 0x55f0f1849c50 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.968+0000 7fe459170740 1 --2- 172.21.15.39:0/107036395 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f0f18478c0 0x55f0f1849710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.968+0000 7fe459170740 1 -- 172.21.15.39:0/107036395 >> 172.21.15.39:0/107036395 conn(0x55f0f1842e30 msgr2=0x55f0f1843210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:37.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.968+0000 7fe459170740 1 -- 172.21.15.39:0/107036395 shutdown_connections 2024-03-18T16:15:37.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.968+0000 7fe459170740 1 -- 172.21.15.39:0/107036395 wait complete. 2024-03-18T16:15:37.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.969+0000 7fe459170740 1 -- start start 2024-03-18T16:15:37.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.969+0000 7fe459170740 1 Processor -- start 2024-03-18T16:15:37.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.969+0000 7fe459170740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f0f18478c0 0x55f0f1855f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:37.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.969+0000 7fe459170740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f0f18482b0 0x55f0f18564c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:37.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.969+0000 7fe459170740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f0f184d420 0x55f0f18574d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:37.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.969+0000 7fe459170740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fe4501b1e90 con 0x55f0f184d420 2024-03-18T16:15:37.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.969+0000 7fe459170740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fe4501b17c0 con 0x55f0f18482b0 2024-03-18T16:15:37.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.969+0000 7fe459170740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fe4501aea00 con 0x55f0f18478c0 2024-03-18T16:15:37.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.970+0000 7fe459170740 10 client.0 _ll_register_callbacks cb 0x7fe43c004210 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:15:37.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.970+0000 7fe44ea00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f0f18482b0 0x55f0f18564c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:37.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.970+0000 7fe44ea00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f0f18482b0 0x55f0f18564c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:53766/0 (socket says 172.21.15.39:53766) 2024-03-18T16:15:37.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.970+0000 7fe44f400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f0f18478c0 0x55f0f1855f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:37.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.970+0000 7fe44ea00640 1 -- 172.21.15.39:0/1997421671 learned_addr learned my addr 172.21.15.39:0/1997421671 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:15:37.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.970+0000 7fe44fe00640 1 --2- 172.21.15.39:0/1997421671 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f0f184d420 0x55f0f18574d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:37.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.970+0000 7fe44ea00640 1 -- 172.21.15.39:0/1997421671 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f0f18478c0 msgr2=0x55f0f1855f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:37.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.970+0000 7fe44ea00640 1 --2- 172.21.15.39:0/1997421671 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f0f18478c0 0x55f0f1855f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.970+0000 7fe44ea00640 1 -- 172.21.15.39:0/1997421671 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f0f184d420 msgr2=0x55f0f18574d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:37.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.970+0000 7fe44ea00640 1 --2- 172.21.15.39:0/1997421671 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f0f184d420 0x55f0f18574d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.970+0000 7fe44ea00640 1 -- 172.21.15.39:0/1997421671 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f0f1857ac0 con 0x55f0f18482b0 2024-03-18T16:15:37.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.970+0000 7fe44f400640 1 --2- 172.21.15.39:0/1997421671 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f0f18478c0 0x55f0f1855f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:15:37.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.970+0000 7fe44ea00640 1 --2- 172.21.15.39:0/1997421671 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f0f18482b0 0x55f0f18564c0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fe43c010ee0 tx=0x7fe43c00b400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd79daeea04465a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:37.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.970+0000 7fe439600640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:15:37.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.971+0000 7fe439600640 1 -- 172.21.15.39:0/1997421671 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe43c02d040 con 0x55f0f18482b0 2024-03-18T16:15:37.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.971+0000 7fe439600640 1 -- 172.21.15.39:0/1997421671 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fe43c005240 con 0x55f0f18482b0 2024-03-18T16:15:37.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.971+0000 7fe459170740 1 -- 172.21.15.39:0/1997421671 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe43c011440 con 0x55f0f18482b0 2024-03-18T16:15:37.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.971+0000 7fe439600640 1 -- 172.21.15.39:0/1997421671 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7fe43c03f050 con 0x55f0f18482b0 2024-03-18T16:15:37.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.971+0000 7fe459170740 10 client.26759 fetch_fsmap learned FSMap version 42 2024-03-18T16:15:37.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.971+0000 7fe459170740 1 -- 172.21.15.39:0/1997421671 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55f0f1858dc0 con 0x55f0f18482b0 2024-03-18T16:15:37.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.972+0000 7fe439600640 1 -- 172.21.15.39:0/1997421671 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe43c011440 con 0x55f0f18482b0 2024-03-18T16:15:37.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.972+0000 7fe439600640 1 -- 172.21.15.39:0/1997421671 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 42) ==== 8370+0+0 (secure 0 0 0) 0x7fe43c0045f0 con 0x55f0f18482b0 2024-03-18T16:15:37.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.972+0000 7fe459170740 10 client.26759 fetch_fsmap finished waiting for FSMap version 42 2024-03-18T16:15:37.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.972+0000 7fe459170740 10 client.26759 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T16:15:37.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.972+0000 7fe459170740 20 client.26759 populate_metadata read hostname 'smithi039' 2024-03-18T16:15:37.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.972+0000 7fe459170740 1 --2- 172.21.15.39:0/1997421671 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x55f0f185d6e0 0x55f0f18d6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:37.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.972+0000 7fe459170740 4 client.26759 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:15:37.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.972+0000 7fe459170740 1 -- 172.21.15.39:0/1997421671 --> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f0f17b3ce0 con 0x55f0f185d6e0 2024-03-18T16:15:37.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.972+0000 7fe44f400640 1 --2- 172.21.15.39:0/1997421671 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x55f0f185d6e0 0x55f0f18d6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:37.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.973+0000 7fe44f400640 1 --2- 172.21.15.39:0/1997421671 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x55f0f185d6e0 0x55f0f18d6ea0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8fd52d4529cbc406 server_cookie=6ac50cbc327b981b in_seq=0 out_seq=0 2024-03-18T16:15:37.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.973+0000 7fe439600640 10 client.26759 ms_handle_connect on v2:172.21.15.196:6838/2001427268 2024-03-18T16:15:37.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.973+0000 7fe439600640 1 -- 172.21.15.39:0/1997421671 <== mds.1 v2:172.21.15.196:6838/2001427268 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f0f17b3ce0 con 0x55f0f185d6e0 2024-03-18T16:15:37.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.973+0000 7fe439600640 10 client.26759 handle_command_reply: tid=0 2024-03-18T16:15:37.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.973+0000 7fe459170740 1 client.26759 shutdown 2024-03-18T16:15:37.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.974+0000 7fe459170740 1 -- 172.21.15.39:0/1997421671 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f0f18482b0 msgr2=0x55f0f18564c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:37.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.974+0000 7fe459170740 1 --2- 172.21.15.39:0/1997421671 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f0f18482b0 0x55f0f18564c0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fe43c010ee0 tx=0x7fe43c00b400 comp rx=0 tx=0).stop 2024-03-18T16:15:37.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.974+0000 7fe459170740 1 -- 172.21.15.39:0/1997421671 shutdown_connections 2024-03-18T16:15:37.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.974+0000 7fe459170740 1 --2- 172.21.15.39:0/1997421671 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x55f0f185d6e0 0x55f0f18d6ea0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.974+0000 7fe459170740 1 --2- 172.21.15.39:0/1997421671 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f0f18482b0 0x55f0f18564c0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.974+0000 7fe459170740 1 --2- 172.21.15.39:0/1997421671 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f0f184d420 0x55f0f18574d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.974+0000 7fe459170740 1 --2- 172.21.15.39:0/1997421671 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f0f18478c0 0x55f0f1855f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.974+0000 7fe459170740 1 -- 172.21.15.39:0/1997421671 >> 172.21.15.39:0/1997421671 conn(0x55f0f1842e30 msgr2=0x55f0f18517d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:37.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.974+0000 7fe439600640 0 client.26759 ms_handle_reset on v2:172.21.15.196:6838/2001427268 2024-03-18T16:15:37.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.974+0000 7fe459170740 1 -- 172.21.15.39:0/1997421671 shutdown_connections 2024-03-18T16:15:37.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.974+0000 7fe459170740 1 -- 172.21.15.39:0/1997421671 wait complete. 2024-03-18T16:15:37.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.974+0000 7fe459170740 20 client.26759 trim_cache size 0 max 16384 2024-03-18T16:15:37.979 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:15:37.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.975+0000 7fe456400640 1 -- 172.21.15.39:0/773184227 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fe4340a2900 msgr2=0x7fe4340a4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:37.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.975+0000 7fe456400640 1 --2- 172.21.15.39:0/773184227 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fe4340a2900 0x7fe4340a4dc0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x55f0f1848b10 tx=0x7fe448009bc0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.975+0000 7fe456400640 1 -- 172.21.15.39:0/773184227 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe4500761a0 msgr2=0x7fe45019e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:37.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.975+0000 7fe456400640 1 --2- 172.21.15.39:0/773184227 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe4500761a0 0x7fe45019e0c0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fe440036040 tx=0x7fe440005dd0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.976+0000 7fe456400640 1 -- 172.21.15.39:0/773184227 shutdown_connections 2024-03-18T16:15:37.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.976+0000 7fe456400640 1 --2- 172.21.15.39:0/773184227 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fe4340a2900 0x7fe4340a4dc0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.976+0000 7fe456400640 1 --2- 172.21.15.39:0/773184227 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe450077d80 0x7fe4501a5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.976+0000 7fe456400640 1 --2- 172.21.15.39:0/773184227 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe4500773e0 0x7fe45019e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.976+0000 7fe456400640 1 --2- 172.21.15.39:0/773184227 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe4500761a0 0x7fe45019e0c0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:37.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.976+0000 7fe456400640 1 -- 172.21.15.39:0/773184227 >> 172.21.15.39:0/773184227 conn(0x7fe4500fef90 msgr2=0x7fe450100a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:37.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.976+0000 7fe456400640 1 -- 172.21.15.39:0/773184227 shutdown_connections 2024-03-18T16:15:37.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:37.976+0000 7fe456400640 1 -- 172.21.15.39:0/773184227 wait complete. 2024-03-18T16:15:38.003 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-18T16:15:38.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.252+0000 7f96f3000640 1 -- 172.21.15.39:0/4119855930 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f96ec0ffe40 msgr2=0x7f96ec102270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:38.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.252+0000 7f96f3000640 1 --2- 172.21.15.39:0/4119855930 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f96ec0ffe40 0x7f96ec102270 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f96e0009900 tx=0x7f96e002bc30 comp rx=0 tx=0).stop 2024-03-18T16:15:38.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.252+0000 7f96f3000640 1 -- 172.21.15.39:0/4119855930 shutdown_connections 2024-03-18T16:15:38.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.252+0000 7f96f3000640 1 --2- 172.21.15.39:0/4119855930 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f96ec105160 0x7f96ec1055e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:38.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.252+0000 7f96f3000640 1 --2- 172.21.15.39:0/4119855930 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f96ec1027b0 0x7f96ec104c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:38.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.252+0000 7f96f3000640 1 --2- 172.21.15.39:0/4119855930 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f96ec0ffe40 0x7f96ec102270 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:38.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.252+0000 7f96f3000640 1 -- 172.21.15.39:0/4119855930 >> 172.21.15.39:0/4119855930 conn(0x7f96ec0fba80 msgr2=0x7f96ec0fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:38.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.253+0000 7f96f3000640 1 -- 172.21.15.39:0/4119855930 shutdown_connections 2024-03-18T16:15:38.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.253+0000 7f96f3000640 1 -- 172.21.15.39:0/4119855930 wait complete. 2024-03-18T16:15:38.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.254+0000 7f96f3000640 1 Processor -- start 2024-03-18T16:15:38.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.255+0000 7f96f3000640 1 -- start start 2024-03-18T16:15:38.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.255+0000 7f96f3000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f96ec0ffe40 0x7f96ec070140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:38.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.255+0000 7f96f3000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f96ec1027b0 0x7f96ec070680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:38.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.255+0000 7f96f3000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f96ec105160 0x7f96ec1adfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:38.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.255+0000 7f96f3000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f96ec113890 con 0x7f96ec1027b0 2024-03-18T16:15:38.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.255+0000 7f96f3000640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f96ec113710 con 0x7f96ec105160 2024-03-18T16:15:38.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.255+0000 7f96f3000640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f96ec113a10 con 0x7f96ec0ffe40 2024-03-18T16:15:38.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.255+0000 7f96f1200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f96ec1027b0 0x7f96ec070680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:38.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.256+0000 7f96f1c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f96ec0ffe40 0x7f96ec070140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:38.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.256+0000 7f96f1200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f96ec1027b0 0x7f96ec070680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:35288/0 (socket says 172.21.15.39:35288) 2024-03-18T16:15:38.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.256+0000 7f96f1200640 1 -- 172.21.15.39:0/253300630 learned_addr learned my addr 172.21.15.39:0/253300630 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:15:38.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.256+0000 7f96f2600640 1 --2- 172.21.15.39:0/253300630 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f96ec105160 0x7f96ec1adfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:38.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.256+0000 7f96f1200640 1 -- 172.21.15.39:0/253300630 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f96ec0ffe40 msgr2=0x7f96ec070140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:38.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.256+0000 7f96f1200640 1 --2- 172.21.15.39:0/253300630 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f96ec0ffe40 0x7f96ec070140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:38.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.256+0000 7f96f1200640 1 -- 172.21.15.39:0/253300630 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f96ec105160 msgr2=0x7f96ec1adfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:38.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.256+0000 7f96f1200640 1 --2- 172.21.15.39:0/253300630 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f96ec105160 0x7f96ec1adfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:38.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.256+0000 7f96f1200640 1 -- 172.21.15.39:0/253300630 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96ec070d20 con 0x7f96ec1027b0 2024-03-18T16:15:38.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.257+0000 7f96f1200640 1 --2- 172.21.15.39:0/253300630 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f96ec1027b0 0x7f96ec070680 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f96d800ea60 tx=0x7f96d800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:38.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.257+0000 7f96ea000640 1 -- 172.21.15.39:0/253300630 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96d800cde0 con 0x7f96ec1027b0 2024-03-18T16:15:38.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.257+0000 7f96ea000640 1 -- 172.21.15.39:0/253300630 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f96d80035f0 con 0x7f96ec1027b0 2024-03-18T16:15:38.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.257+0000 7f96ea000640 1 -- 172.21.15.39:0/253300630 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96d8010670 con 0x7f96ec1027b0 2024-03-18T16:15:38.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.259+0000 7f96f3000640 1 -- 172.21.15.39:0/253300630 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96ec1ae600 con 0x7f96ec1027b0 2024-03-18T16:15:38.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.260+0000 7f96f3000640 1 -- 172.21.15.39:0/253300630 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96ec1aead0 con 0x7f96ec1027b0 2024-03-18T16:15:38.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.265+0000 7f96f7274740 1 Processor -- start 2024-03-18T16:15:38.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.266+0000 7f96f7274740 1 -- start start 2024-03-18T16:15:38.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.266+0000 7f96f7274740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c695685a90 0x55c695685eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:38.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.266+0000 7f96f7274740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c695686480 0x55c69568af80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:38.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.266+0000 7f96ea000640 1 -- 172.21.15.39:0/253300630 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f96d80037a0 con 0x7f96ec1027b0 2024-03-18T16:15:38.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.266+0000 7f96f7274740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c69568b5f0 0x55c69568da80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:38.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.266+0000 7f96f7274740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55c69564d3b0 con 0x55c69568b5f0 2024-03-18T16:15:38.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.266+0000 7f96f7274740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55c6955ea450 con 0x55c695686480 2024-03-18T16:15:38.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.266+0000 7f96f7274740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55c695592cb0 con 0x55c695685a90 2024-03-18T16:15:38.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.266+0000 7f96f2600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c695685a90 0x55c695685eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:38.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.266+0000 7f96f2600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c695685a90 0x55c695685eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:45922/0 (socket says 172.21.15.39:45922) 2024-03-18T16:15:38.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.266+0000 7f96f2600640 1 -- 172.21.15.39:0/1538176395 learned_addr learned my addr 172.21.15.39:0/1538176395 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:15:38.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.266+0000 7f96f1c00640 1 --2- 172.21.15.39:0/1538176395 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c695686480 0x55c69568af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:38.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.267+0000 7f96f2600640 1 -- 172.21.15.39:0/1538176395 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c695686480 msgr2=0x55c69568af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:38.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.267+0000 7f96f2600640 1 --2- 172.21.15.39:0/1538176395 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c695686480 0x55c69568af80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:38.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.267+0000 7f96f2600640 1 -- 172.21.15.39:0/1538176395 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c69568b5f0 msgr2=0x55c69568da80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:15:38.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.267+0000 7f96f2600640 1 --2- 172.21.15.39:0/1538176395 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c69568b5f0 0x55c69568da80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:38.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.267+0000 7f96f2600640 1 -- 172.21.15.39:0/1538176395 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c6955b41f0 con 0x55c695685a90 2024-03-18T16:15:38.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.268+0000 7f96f2600640 1 --2- 172.21.15.39:0/1538176395 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c695685a90 0x55c695685eb0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f96e4010370 tx=0x7f96e4010830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=843f6839acabe051 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:38.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.268+0000 7f96ea000640 1 --2- 172.21.15.39:0/253300630 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f96d40a29d0 0x7f96d40a4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:38.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.268+0000 7f96ea000640 1 -- 172.21.15.39:0/253300630 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f96d80cf310 con 0x7f96ec1027b0 2024-03-18T16:15:38.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.268+0000 7f96dea00640 1 -- 172.21.15.39:0/1538176395 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96e4016070 con 0x55c695685a90 2024-03-18T16:15:38.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.269+0000 7f96dea00640 1 -- 172.21.15.39:0/1538176395 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f96e4004d20 con 0x55c695685a90 2024-03-18T16:15:38.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.269+0000 7f96dea00640 1 -- 172.21.15.39:0/1538176395 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96e40054c0 con 0x55c695685a90 2024-03-18T16:15:38.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.269+0000 7f96f7274740 1 -- 172.21.15.39:0/1538176395 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c695685a90 msgr2=0x55c695685eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:38.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.269+0000 7f96f7274740 1 --2- 172.21.15.39:0/1538176395 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c695685a90 0x55c695685eb0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f96e4010370 tx=0x7f96e4010830 comp rx=0 tx=0).stop 2024-03-18T16:15:38.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.270+0000 7f96f7274740 1 -- 172.21.15.39:0/1538176395 shutdown_connections 2024-03-18T16:15:38.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.270+0000 7f96f7274740 1 --2- 172.21.15.39:0/1538176395 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c695686480 0x55c69568af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:38.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.270+0000 7f96f7274740 1 --2- 172.21.15.39:0/1538176395 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c69568b5f0 0x55c69568da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:38.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.270+0000 7f96f7274740 1 --2- 172.21.15.39:0/1538176395 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c695685a90 0x55c695685eb0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:38.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.270+0000 7f96f1c00640 1 --2- 172.21.15.39:0/253300630 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f96d40a29d0 0x7f96d40a4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:38.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.270+0000 7f96f7274740 1 -- 172.21.15.39:0/1538176395 >> 172.21.15.39:0/1538176395 conn(0x55c695681000 msgr2=0x55c695681400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:38.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.270+0000 7f96f7274740 1 -- 172.21.15.39:0/1538176395 shutdown_connections 2024-03-18T16:15:38.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.271+0000 7f96f7274740 1 -- 172.21.15.39:0/1538176395 wait complete. 2024-03-18T16:15:38.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.271+0000 7f96f7274740 1 -- start start 2024-03-18T16:15:38.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.272+0000 7f96f1c00640 1 --2- 172.21.15.39:0/253300630 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f96d40a29d0 0x7f96d40a4e90 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f96e002c240 tx=0x7f96e002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:38.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.272+0000 7f96f7274740 1 Processor -- start 2024-03-18T16:15:38.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.273+0000 7f96f7274740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c695685a90 0x55c69568fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:38.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.273+0000 7f96f7274740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c695686480 0x55c6956902d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:38.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.273+0000 7f96f7274740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c69568b5f0 0x55c695697310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:38.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.273+0000 7f96f7274740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55c6955a6110 con 0x55c695686480 2024-03-18T16:15:38.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.273+0000 7f96f7274740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55c6955e64f0 con 0x55c69568b5f0 2024-03-18T16:15:38.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.273+0000 7f96f7274740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55c69568e5a0 con 0x55c695685a90 2024-03-18T16:15:38.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.273+0000 7f96f7274740 10 client.0 _ll_register_callbacks cb 0x55c695593b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:15:38.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.274+0000 7f96f1c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c695685a90 0x55c69568fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:38.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.274+0000 7f96f1c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c695685a90 0x55c69568fd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:45930/0 (socket says 172.21.15.39:45930) 2024-03-18T16:15:38.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.274+0000 7f96f1c00640 1 -- 172.21.15.39:0/2727204882 learned_addr learned my addr 172.21.15.39:0/2727204882 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:15:38.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.274+0000 7f96f1c00640 1 -- 172.21.15.39:0/2727204882 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c69568b5f0 msgr2=0x55c695697310 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:15:38.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.274+0000 7f96f1c00640 1 --2- 172.21.15.39:0/2727204882 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c69568b5f0 0x55c695697310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:38.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.274+0000 7f96f1c00640 1 -- 172.21.15.39:0/2727204882 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c695686480 msgr2=0x55c6956902d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:15:38.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.274+0000 7f96f1c00640 1 --2- 172.21.15.39:0/2727204882 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c695686480 0x55c6956902d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:38.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.274+0000 7f96f1c00640 1 -- 172.21.15.39:0/2727204882 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c6952fe3f0 con 0x55c695685a90 2024-03-18T16:15:38.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.275+0000 7f96f1c00640 1 --2- 172.21.15.39:0/2727204882 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c695685a90 0x55c69568fd90 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f96e0000c00 tx=0x7f96e0009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d6c978122eb199a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:38.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.275+0000 7f96d3400640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:15:38.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.275+0000 7f96d3400640 1 -- 172.21.15.39:0/2727204882 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96e0046070 con 0x55c695685a90 2024-03-18T16:15:38.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.275+0000 7f96d3400640 1 -- 172.21.15.39:0/2727204882 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f96e0004eb0 con 0x55c695685a90 2024-03-18T16:15:38.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.275+0000 7f96f7274740 1 -- 172.21.15.39:0/2727204882 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f96e4004d20 con 0x55c695685a90 2024-03-18T16:15:38.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.276+0000 7f96d3400640 1 -- 172.21.15.39:0/2727204882 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f96e00050b0 con 0x55c695685a90 2024-03-18T16:15:38.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.276+0000 7f96f7274740 10 client.26731 fetch_fsmap learned FSMap version 42 2024-03-18T16:15:38.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.276+0000 7f96f7274740 1 -- 172.21.15.39:0/2727204882 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55c6953c87d0 con 0x55c695685a90 2024-03-18T16:15:38.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.276+0000 7f96d3400640 1 -- 172.21.15.39:0/2727204882 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f96e4004d20 con 0x55c695685a90 2024-03-18T16:15:38.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.276+0000 7f96d3400640 1 -- 172.21.15.39:0/2727204882 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 42) ==== 8370+0+0 (secure 0 0 0) 0x7f96e0003cb0 con 0x55c695685a90 2024-03-18T16:15:38.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.277+0000 7f96f7274740 10 client.26731 fetch_fsmap finished waiting for FSMap version 42 2024-03-18T16:15:38.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.277+0000 7f96f7274740 10 client.26731 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:15:38.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.277+0000 7f96f7274740 20 client.26731 populate_metadata read hostname 'smithi039' 2024-03-18T16:15:38.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.277+0000 7f96f7274740 1 --2- 172.21.15.39:0/2727204882 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55c69569bd40 0x55c69569e200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:38.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.277+0000 7f96f7274740 4 client.26731 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:15:38.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.277+0000 7f96f7274740 1 -- 172.21.15.39:0/2727204882 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c6955f7990 con 0x55c69569bd40 2024-03-18T16:15:38.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.277+0000 7f96f1200640 1 --2- 172.21.15.39:0/2727204882 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55c69569bd40 0x55c69569e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:38.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.278+0000 7f96f1200640 1 --2- 172.21.15.39:0/2727204882 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55c69569bd40 0x55c69569e200 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2ac98a8400244558 server_cookie=39c80ada11616f95 in_seq=0 out_seq=0 2024-03-18T16:15:38.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.278+0000 7f96d3400640 10 client.26731 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:15:38.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.279+0000 7f96d3400640 1 -- 172.21.15.39:0/2727204882 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c6955f7990 con 0x55c69569bd40 2024-03-18T16:15:38.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.279+0000 7f96d3400640 10 client.26731 handle_command_reply: tid=0 2024-03-18T16:15:38.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.279+0000 7f96f7274740 1 client.26731 shutdown 2024-03-18T16:15:38.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.279+0000 7f96f7274740 1 -- 172.21.15.39:0/2727204882 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c695685a90 msgr2=0x55c69568fd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:38.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.279+0000 7f96f7274740 1 --2- 172.21.15.39:0/2727204882 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c695685a90 0x55c69568fd90 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f96e0000c00 tx=0x7f96e0009260 comp rx=0 tx=0).stop 2024-03-18T16:15:38.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.280+0000 7f96f7274740 1 -- 172.21.15.39:0/2727204882 shutdown_connections 2024-03-18T16:15:38.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.280+0000 7f96f7274740 1 --2- 172.21.15.39:0/2727204882 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55c69569bd40 0x55c69569e200 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:38.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.280+0000 7f96f7274740 1 --2- 172.21.15.39:0/2727204882 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c69568b5f0 0x55c695697310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:38.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.280+0000 7f96f7274740 1 --2- 172.21.15.39:0/2727204882 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c695686480 0x55c6956902d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:38.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.280+0000 7f96f7274740 1 --2- 172.21.15.39:0/2727204882 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c695685a90 0x55c69568fd90 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:38.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.280+0000 7f96f7274740 1 -- 172.21.15.39:0/2727204882 >> 172.21.15.39:0/2727204882 conn(0x55c695681000 msgr2=0x55c695680220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:38.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.280+0000 7f96d3400640 0 client.26731 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:15:38.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.280+0000 7f96f7274740 1 -- 172.21.15.39:0/2727204882 shutdown_connections 2024-03-18T16:15:38.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.280+0000 7f96f7274740 1 -- 172.21.15.39:0/2727204882 wait complete. 2024-03-18T16:15:38.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.280+0000 7f96f7274740 20 client.26731 trim_cache size 0 max 16384 2024-03-18T16:15:38.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.305+0000 7f96f7274740 1 Processor -- start 2024-03-18T16:15:38.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.305+0000 7f96f7274740 1 -- start start 2024-03-18T16:15:38.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.306+0000 7f96f7274740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c695685a90 0x55c695690230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:38.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.306+0000 7f96f7274740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c695686480 0x55c6956972d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:38.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.306+0000 7f96f7274740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c69568b5f0 0x55c695684970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:38.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.306+0000 7f96f7274740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f96ec1b0790 con 0x55c69568b5f0 2024-03-18T16:15:38.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.306+0000 7f96f7274740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f96ec1b22d0 con 0x55c695686480 2024-03-18T16:15:38.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.306+0000 7f96f7274740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f96ec1b0e60 con 0x55c695685a90 2024-03-18T16:15:38.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.306+0000 7f96f1c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c695685a90 0x55c695690230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:38.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.306+0000 7f96f1c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c695685a90 0x55c695690230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:45934/0 (socket says 172.21.15.39:45934) 2024-03-18T16:15:38.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.306+0000 7f96f1c00640 1 -- 172.21.15.39:0/475448041 learned_addr learned my addr 172.21.15.39:0/475448041 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:15:38.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.306+0000 7f96f1200640 1 --2- 172.21.15.39:0/475448041 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c695686480 0x55c6956972d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:38.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.306+0000 7f96f2600640 1 --2- 172.21.15.39:0/475448041 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c69568b5f0 0x55c695684970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:38.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.307+0000 7f96f1200640 1 -- 172.21.15.39:0/475448041 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c695685a90 msgr2=0x55c695690230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:38.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.307+0000 7f96f1200640 1 --2- 172.21.15.39:0/475448041 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c695685a90 0x55c695690230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:38.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.307+0000 7f96f1200640 1 -- 172.21.15.39:0/475448041 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c69568b5f0 msgr2=0x55c695684970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:38.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.307+0000 7f96f1200640 1 --2- 172.21.15.39:0/475448041 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c69568b5f0 0x55c695684970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:38.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.307+0000 7f96f1200640 1 -- 172.21.15.39:0/475448041 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c69568ab90 con 0x55c695686480 2024-03-18T16:15:38.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.307+0000 7f96f1c00640 1 --2- 172.21.15.39:0/475448041 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c695685a90 0x55c695690230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:15:38.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.307+0000 7f96f2600640 1 --2- 172.21.15.39:0/475448041 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c69568b5f0 0x55c695684970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:15:38.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.307+0000 7f96f1200640 1 --2- 172.21.15.39:0/475448041 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c695686480 0x55c6956972d0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x55c695691270 tx=0x7f96d80cc8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b502c045bb322d19 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:38.312 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.307+0000 7f96de000640 1 -- 172.21.15.39:0/475448041 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96d80ae070 con 0x55c695686480 2024-03-18T16:15:38.312 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.307+0000 7f96de000640 1 -- 172.21.15.39:0/475448041 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f96d809bfc0 con 0x55c695686480 2024-03-18T16:15:38.312 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.307+0000 7f96de000640 1 -- 172.21.15.39:0/475448041 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96d8003b30 con 0x55c695686480 2024-03-18T16:15:38.312 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.308+0000 7f96f7274740 1 -- 172.21.15.39:0/475448041 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c695686480 msgr2=0x55c6956972d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:38.312 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.308+0000 7f96f7274740 1 --2- 172.21.15.39:0/475448041 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c695686480 0x55c6956972d0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x55c695691270 tx=0x7f96d80cc8f0 comp rx=0 tx=0).stop 2024-03-18T16:15:38.312 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.308+0000 7f96f7274740 1 -- 172.21.15.39:0/475448041 shutdown_connections 2024-03-18T16:15:38.312 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.308+0000 7f96f7274740 1 --2- 172.21.15.39:0/475448041 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c695686480 0x55c6956972d0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:38.312 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.308+0000 7f96f7274740 1 --2- 172.21.15.39:0/475448041 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c69568b5f0 0x55c695684970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:38.312 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.308+0000 7f96f7274740 1 --2- 172.21.15.39:0/475448041 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c695685a90 0x55c695690230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:38.312 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.308+0000 7f96f7274740 1 -- 172.21.15.39:0/475448041 >> 172.21.15.39:0/475448041 conn(0x55c695681000 msgr2=0x55c695684260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:38.312 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.309+0000 7f96f7274740 1 -- 172.21.15.39:0/475448041 shutdown_connections 2024-03-18T16:15:38.312 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.309+0000 7f96f7274740 1 -- 172.21.15.39:0/475448041 wait complete. 2024-03-18T16:15:38.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.309+0000 7f96f7274740 1 -- start start 2024-03-18T16:15:38.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.310+0000 7f96f7274740 1 Processor -- start 2024-03-18T16:15:38.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.311+0000 7f96f7274740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c695685a90 0x55c695694cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:38.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.311+0000 7f96f7274740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c695686480 0x55c695695230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:38.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.311+0000 7f96f7274740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c69568b5f0 0x55c6956961a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:38.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.312+0000 7f96f7274740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f96ec1b1c00 con 0x55c69568b5f0 2024-03-18T16:15:38.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.312+0000 7f96f7274740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f96ec1b1530 con 0x55c695685a90 2024-03-18T16:15:38.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.312+0000 7f96f7274740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f96ec10e670 con 0x55c695686480 2024-03-18T16:15:38.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.312+0000 7f96f7274740 10 client.0 _ll_register_callbacks cb 0x7f96e002ca80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:15:38.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.312+0000 7f96f1c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c695685a90 0x55c695694cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:38.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.312+0000 7f96f1c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c695685a90 0x55c695694cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:48238/0 (socket says 172.21.15.39:48238) 2024-03-18T16:15:38.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.312+0000 7f96f1c00640 1 -- 172.21.15.39:0/3496807452 learned_addr learned my addr 172.21.15.39:0/3496807452 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:15:38.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.312+0000 7f96f1200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c695686480 0x55c695695230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:38.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.312+0000 7f96f2600640 1 --2- 172.21.15.39:0/3496807452 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c69568b5f0 0x55c6956961a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:38.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.313+0000 7f96f1c00640 1 -- 172.21.15.39:0/3496807452 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c695686480 msgr2=0x55c695695230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:38.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.313+0000 7f96f1c00640 1 --2- 172.21.15.39:0/3496807452 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c695686480 0x55c695695230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:38.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.313+0000 7f96f1c00640 1 -- 172.21.15.39:0/3496807452 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c69568b5f0 msgr2=0x55c6956961a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:38.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.313+0000 7f96f1c00640 1 --2- 172.21.15.39:0/3496807452 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c69568b5f0 0x55c6956961a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:38.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.313+0000 7f96f1c00640 1 -- 172.21.15.39:0/3496807452 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c695696790 con 0x55c695685a90 2024-03-18T16:15:38.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.313+0000 7f96f1c00640 1 --2- 172.21.15.39:0/3496807452 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c695685a90 0x55c695694cf0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f96e00346d0 tx=0x7f96e0034a00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=68fcad1ca5a153ac server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:38.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.313+0000 7f96d3400640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:15:38.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.313+0000 7f96d3400640 1 -- 172.21.15.39:0/3496807452 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96e0034d30 con 0x55c695685a90 2024-03-18T16:15:38.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.313+0000 7f96f7274740 1 -- 172.21.15.39:0/3496807452 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f96d809bfc0 con 0x55c695685a90 2024-03-18T16:15:38.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.314+0000 7f96d3400640 1 -- 172.21.15.39:0/3496807452 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f96e0005250 con 0x55c695685a90 2024-03-18T16:15:38.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.314+0000 7f96d3400640 1 -- 172.21.15.39:0/3496807452 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f96e0081050 con 0x55c695685a90 2024-03-18T16:15:38.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.314+0000 7f96f7274740 10 client.26774 fetch_fsmap learned FSMap version 42 2024-03-18T16:15:38.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.314+0000 7f96f7274740 1 -- 172.21.15.39:0/3496807452 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55c695702310 con 0x55c695685a90 2024-03-18T16:15:38.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.315+0000 7f96d3400640 1 -- 172.21.15.39:0/3496807452 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f96d809bfc0 con 0x55c695685a90 2024-03-18T16:15:38.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.315+0000 7f96d3400640 1 -- 172.21.15.39:0/3496807452 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 42) ==== 8370+0+0 (secure 0 0 0) 0x7f96e0003d70 con 0x55c695685a90 2024-03-18T16:15:38.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.315+0000 7f96f7274740 10 client.26774 fetch_fsmap finished waiting for FSMap version 42 2024-03-18T16:15:38.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.315+0000 7f96f7274740 10 client.26774 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:15:38.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.315+0000 7f96f7274740 20 client.26774 populate_metadata read hostname 'smithi039' 2024-03-18T16:15:38.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.315+0000 7f96f7274740 1 --2- 172.21.15.39:0/3496807452 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55c695702cb0 0x55c6957050d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:38.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.315+0000 7f96f7274740 4 client.26774 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:15:38.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.315+0000 7f96f7274740 1 -- 172.21.15.39:0/3496807452 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c6955f1ce0 con 0x55c695702cb0 2024-03-18T16:15:38.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.316+0000 7f96f1200640 1 --2- 172.21.15.39:0/3496807452 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55c695702cb0 0x55c6957050d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:38.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.316+0000 7f96f1200640 1 --2- 172.21.15.39:0/3496807452 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55c695702cb0 0x55c6957050d0 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b9745045a0b5d1a8 server_cookie=301249cc410daf39 in_seq=0 out_seq=0 2024-03-18T16:15:38.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.316+0000 7f96d3400640 10 client.26774 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:15:38.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.317+0000 7f96d3400640 1 -- 172.21.15.39:0/3496807452 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c6955f1ce0 con 0x55c695702cb0 2024-03-18T16:15:38.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.317+0000 7f96d3400640 10 client.26774 handle_command_reply: tid=0 2024-03-18T16:15:38.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.317+0000 7f96f7274740 1 client.26774 shutdown 2024-03-18T16:15:38.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.317+0000 7f96f7274740 1 -- 172.21.15.39:0/3496807452 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c695685a90 msgr2=0x55c695694cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:38.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.317+0000 7f96f7274740 1 --2- 172.21.15.39:0/3496807452 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c695685a90 0x55c695694cf0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f96e00346d0 tx=0x7f96e0034a00 comp rx=0 tx=0).stop 2024-03-18T16:15:38.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.317+0000 7f96f7274740 1 -- 172.21.15.39:0/3496807452 shutdown_connections 2024-03-18T16:15:38.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.317+0000 7f96f7274740 1 --2- 172.21.15.39:0/3496807452 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55c695702cb0 0x55c6957050d0 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:38.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.317+0000 7f96f7274740 1 --2- 172.21.15.39:0/3496807452 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c69568b5f0 0x55c6956961a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:38.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.317+0000 7f96f7274740 1 --2- 172.21.15.39:0/3496807452 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c695686480 0x55c695695230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:38.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.317+0000 7f96d3400640 0 client.26774 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:15:38.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.317+0000 7f96f7274740 1 --2- 172.21.15.39:0/3496807452 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c695685a90 0x55c695694cf0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:38.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.317+0000 7f96f7274740 1 -- 172.21.15.39:0/3496807452 >> 172.21.15.39:0/3496807452 conn(0x55c695681000 msgr2=0x55c69567f990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:38.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.318+0000 7f96f7274740 1 -- 172.21.15.39:0/3496807452 shutdown_connections 2024-03-18T16:15:38.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.319+0000 7f96f7274740 1 -- 172.21.15.39:0/3496807452 wait complete. 2024-03-18T16:15:38.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.320+0000 7f96f7274740 20 client.26774 trim_cache size 0 max 16384 2024-03-18T16:15:38.323 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:15:38.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.321+0000 7f96f3000640 1 -- 172.21.15.39:0/253300630 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f96d40a29d0 msgr2=0x7f96d40a4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:38.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.321+0000 7f96f3000640 1 --2- 172.21.15.39:0/253300630 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f96d40a29d0 0x7f96d40a4e90 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f96e002c240 tx=0x7f96e002df90 comp rx=0 tx=0).stop 2024-03-18T16:15:38.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.321+0000 7f96f3000640 1 -- 172.21.15.39:0/253300630 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f96ec1027b0 msgr2=0x7f96ec070680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:38.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.321+0000 7f96f3000640 1 --2- 172.21.15.39:0/253300630 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f96ec1027b0 0x7f96ec070680 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f96d800ea60 tx=0x7f96d800ef20 comp rx=0 tx=0).stop 2024-03-18T16:15:38.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.321+0000 7f96f3000640 1 -- 172.21.15.39:0/253300630 shutdown_connections 2024-03-18T16:15:38.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.321+0000 7f96f3000640 1 --2- 172.21.15.39:0/253300630 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f96ec105160 0x7f96ec1adfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:38.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.321+0000 7f96f3000640 1 --2- 172.21.15.39:0/253300630 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f96d40a29d0 0x7f96d40a4e90 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:38.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.321+0000 7f96f3000640 1 --2- 172.21.15.39:0/253300630 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f96ec1027b0 0x7f96ec070680 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:38.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.321+0000 7f96f3000640 1 --2- 172.21.15.39:0/253300630 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f96ec0ffe40 0x7f96ec070140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:38.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.321+0000 7f96f3000640 1 -- 172.21.15.39:0/253300630 >> 172.21.15.39:0/253300630 conn(0x7f96ec0fba80 msgr2=0x7f96ec103470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:38.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.322+0000 7f96f3000640 1 -- 172.21.15.39:0/253300630 shutdown_connections 2024-03-18T16:15:38.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:38.322+0000 7f96f3000640 1 -- 172.21.15.39:0/253300630 wait complete. 2024-03-18T16:15:39.349 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-18T16:15:39.349 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T16:15:39.349 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-18T16:15:39.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:39 smithi196 ceph-mon[36307]: pgmap v589: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 10 MiB/s wr, 2.30k op/s 2024-03-18T16:15:39.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:39 smithi194 ceph-mon[36200]: pgmap v589: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 10 MiB/s wr, 2.30k op/s 2024-03-18T16:15:39.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:39 smithi039 ceph-mon[31721]: pgmap v589: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 10 MiB/s wr, 2.30k op/s 2024-03-18T16:15:39.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.609+0000 7f8d48c00640 1 -- 172.21.15.39:0/3974149173 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8d441017f0 msgr2=0x7f8d44101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:39.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.609+0000 7f8d48c00640 1 --2- 172.21.15.39:0/3974149173 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8d441017f0 0x7f8d44101c10 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f8d34009900 tx=0x7f8d3402bc60 comp rx=0 tx=0).stop 2024-03-18T16:15:39.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.610+0000 7f8d48c00640 1 -- 172.21.15.39:0/3974149173 shutdown_connections 2024-03-18T16:15:39.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.610+0000 7f8d48c00640 1 --2- 172.21.15.39:0/3974149173 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8d441029e0 0x7f8d44102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:39.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.610+0000 7f8d48c00640 1 --2- 172.21.15.39:0/3974149173 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8d441033a0 0x7f8d44109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:39.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.610+0000 7f8d48c00640 1 --2- 172.21.15.39:0/3974149173 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8d441017f0 0x7f8d44101c10 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:39.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.610+0000 7f8d48c00640 1 -- 172.21.15.39:0/3974149173 >> 172.21.15.39:0/3974149173 conn(0x7f8d440fcd80 msgr2=0x7f8d440ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:39.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.610+0000 7f8d48c00640 1 -- 172.21.15.39:0/3974149173 shutdown_connections 2024-03-18T16:15:39.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.610+0000 7f8d48c00640 1 -- 172.21.15.39:0/3974149173 wait complete. 2024-03-18T16:15:39.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.612+0000 7f8d48c00640 1 Processor -- start 2024-03-18T16:15:39.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.612+0000 7f8d48c00640 1 -- start start 2024-03-18T16:15:39.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.612+0000 7f8d48c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8d441017f0 0x7f8d441959f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:39.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.612+0000 7f8d48c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8d441029e0 0x7f8d44195f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:39.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.612+0000 7f8d48c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8d441033a0 0x7f8d441ae440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:39.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.612+0000 7f8d48c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f8d4410d530 con 0x7f8d441029e0 2024-03-18T16:15:39.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.612+0000 7f8d48c00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f8d4410d3b0 con 0x7f8d441033a0 2024-03-18T16:15:39.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.612+0000 7f8d48c00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f8d4410d6b0 con 0x7f8d441017f0 2024-03-18T16:15:39.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.612+0000 7f8d41e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8d441017f0 0x7f8d441959f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:39.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.613+0000 7f8d41e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8d441017f0 0x7f8d441959f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:45962/0 (socket says 172.21.15.39:45962) 2024-03-18T16:15:39.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.613+0000 7f8d41e00640 1 -- 172.21.15.39:0/2108206198 learned_addr learned my addr 172.21.15.39:0/2108206198 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:15:39.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.613+0000 7f8d41400640 1 --2- 172.21.15.39:0/2108206198 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8d441029e0 0x7f8d44195f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:39.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.613+0000 7f8d41e00640 1 -- 172.21.15.39:0/2108206198 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8d441033a0 msgr2=0x7f8d441ae440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:39.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.613+0000 7f8d42800640 1 --2- 172.21.15.39:0/2108206198 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8d441033a0 0x7f8d441ae440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:39.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.613+0000 7f8d41e00640 1 --2- 172.21.15.39:0/2108206198 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8d441033a0 0x7f8d441ae440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:39.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.613+0000 7f8d41e00640 1 -- 172.21.15.39:0/2108206198 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8d441029e0 msgr2=0x7f8d44195f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:39.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.613+0000 7f8d41e00640 1 --2- 172.21.15.39:0/2108206198 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8d441029e0 0x7f8d44195f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:39.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.613+0000 7f8d41e00640 1 -- 172.21.15.39:0/2108206198 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d4419b0c0 con 0x7f8d441017f0 2024-03-18T16:15:39.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.613+0000 7f8d41400640 1 --2- 172.21.15.39:0/2108206198 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8d441029e0 0x7f8d44195f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:15:39.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.614+0000 7f8d41e00640 1 --2- 172.21.15.39:0/2108206198 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8d441017f0 0x7f8d441959f0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f8d3402efd0 tx=0x7f8d34005e40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:39.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.614+0000 7f8d3aa00640 1 -- 172.21.15.39:0/2108206198 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d34041070 con 0x7f8d441017f0 2024-03-18T16:15:39.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.614+0000 7f8d3aa00640 1 -- 172.21.15.39:0/2108206198 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d340054e0 con 0x7f8d441017f0 2024-03-18T16:15:39.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.614+0000 7f8d3aa00640 1 -- 172.21.15.39:0/2108206198 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d34034440 con 0x7f8d441017f0 2024-03-18T16:15:39.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.614+0000 7f8d48c00640 1 -- 172.21.15.39:0/2108206198 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d441ae980 con 0x7f8d441017f0 2024-03-18T16:15:39.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.614+0000 7f8d48c00640 1 -- 172.21.15.39:0/2108206198 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d441aed30 con 0x7f8d441017f0 2024-03-18T16:15:39.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.620+0000 7f8d3aa00640 1 -- 172.21.15.39:0/2108206198 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f8d34044070 con 0x7f8d441017f0 2024-03-18T16:15:39.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.621+0000 7f8d4b982740 1 Processor -- start 2024-03-18T16:15:39.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.621+0000 7f8d4b982740 1 -- start start 2024-03-18T16:15:39.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.622+0000 7f8d4b982740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f16013f7e0 0x55f16013fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:39.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.622+0000 7f8d4b982740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f1601401d0 0x55f160144cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:39.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.622+0000 7f8d4b982740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f160145340 0x55f1601477d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:39.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.622+0000 7f8d42800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f16013f7e0 0x55f16013fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:39.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.622+0000 7f8d4b982740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55f1600a2f80 con 0x55f16013f7e0 2024-03-18T16:15:39.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.622+0000 7f8d4b982740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55f16004bad0 con 0x55f1601401d0 2024-03-18T16:15:39.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.622+0000 7f8d42800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f16013f7e0 0x55f16013fc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:35328/0 (socket says 172.21.15.39:35328) 2024-03-18T16:15:39.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.622+0000 7f8d4b982740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55f16009f020 con 0x55f160145340 2024-03-18T16:15:39.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.622+0000 7f8d42800640 1 -- 172.21.15.39:0/3854384956 learned_addr learned my addr 172.21.15.39:0/3854384956 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:15:39.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.623+0000 7f8d41e00640 1 --2- 172.21.15.39:0/3854384956 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f1601401d0 0x55f160144cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:39.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.623+0000 7f8d42800640 1 -- 172.21.15.39:0/3854384956 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f160145340 msgr2=0x55f1601477d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:39.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.623+0000 7f8d42800640 1 --2- 172.21.15.39:0/3854384956 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f160145340 0x55f1601477d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:39.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.623+0000 7f8d42800640 1 -- 172.21.15.39:0/3854384956 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f1601401d0 msgr2=0x55f160144cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:39.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.623+0000 7f8d42800640 1 --2- 172.21.15.39:0/3854384956 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f1601401d0 0x55f160144cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:39.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.623+0000 7f8d42800640 1 -- 172.21.15.39:0/3854384956 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f1600636f0 con 0x55f16013f7e0 2024-03-18T16:15:39.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.624+0000 7f8d3aa00640 1 --2- 172.21.15.39:0/2108206198 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f8d280a2800 0x7f8d280a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:39.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.624+0000 7f8d42800640 1 --2- 172.21.15.39:0/3854384956 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f16013f7e0 0x55f16013fc00 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f8d3c009df0 tx=0x7f8d3c00f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29b40e7b3175cea1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:39.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.624+0000 7f8d3aa00640 1 -- 172.21.15.39:0/2108206198 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f8d340f0bf0 con 0x7f8d441017f0 2024-03-18T16:15:39.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.624+0000 7f8d41e00640 1 --2- 172.21.15.39:0/2108206198 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f8d280a2800 0x7f8d280a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:39.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.624+0000 7f8d2f000640 1 -- 172.21.15.39:0/3854384956 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d3c003970 con 0x55f16013f7e0 2024-03-18T16:15:39.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.624+0000 7f8d2f000640 1 -- 172.21.15.39:0/3854384956 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d3c004030 con 0x55f16013f7e0 2024-03-18T16:15:39.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.624+0000 7f8d2f000640 1 -- 172.21.15.39:0/3854384956 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d3c0118c0 con 0x55f16013f7e0 2024-03-18T16:15:39.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.625+0000 7f8d4b982740 1 -- 172.21.15.39:0/3854384956 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f16013f7e0 msgr2=0x55f16013fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:39.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.625+0000 7f8d41e00640 1 --2- 172.21.15.39:0/2108206198 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f8d280a2800 0x7f8d280a4cc0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x55f1601405d0 tx=0x7f8d340f47a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:39.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.625+0000 7f8d4b982740 1 --2- 172.21.15.39:0/3854384956 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f16013f7e0 0x55f16013fc00 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f8d3c009df0 tx=0x7f8d3c00f730 comp rx=0 tx=0).stop 2024-03-18T16:15:39.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.625+0000 7f8d4b982740 1 -- 172.21.15.39:0/3854384956 shutdown_connections 2024-03-18T16:15:39.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.625+0000 7f8d4b982740 1 --2- 172.21.15.39:0/3854384956 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f1601401d0 0x55f160144cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:39.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.625+0000 7f8d4b982740 1 --2- 172.21.15.39:0/3854384956 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f160145340 0x55f1601477d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:39.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.625+0000 7f8d4b982740 1 --2- 172.21.15.39:0/3854384956 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f16013f7e0 0x55f16013fc00 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:39.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.625+0000 7f8d4b982740 1 -- 172.21.15.39:0/3854384956 >> 172.21.15.39:0/3854384956 conn(0x55f160107930 msgr2=0x55f16013b150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:39.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.626+0000 7f8d4b982740 1 -- 172.21.15.39:0/3854384956 shutdown_connections 2024-03-18T16:15:39.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.626+0000 7f8d4b982740 1 -- 172.21.15.39:0/3854384956 wait complete. 2024-03-18T16:15:39.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.626+0000 7f8d4b982740 1 -- start start 2024-03-18T16:15:39.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.627+0000 7f8d4b982740 1 Processor -- start 2024-03-18T16:15:39.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.627+0000 7f8d4b982740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f16013f7e0 0x55f160149b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:39.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.627+0000 7f8d4b982740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f1601401d0 0x55f16014a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:39.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.627+0000 7f8d4b982740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f160145340 0x55f1601510f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:39.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.627+0000 7f8d4b982740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55f1601482f0 con 0x55f160145340 2024-03-18T16:15:39.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.627+0000 7f8d4b982740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55f16005ef30 con 0x55f1601401d0 2024-03-18T16:15:39.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.627+0000 7f8d4b982740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55f160148470 con 0x55f16013f7e0 2024-03-18T16:15:39.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.627+0000 7f8d4b982740 10 client.0 _ll_register_callbacks cb 0x55f16004cbc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:15:39.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.628+0000 7f8d41400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f1601401d0 0x55f16014a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:39.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.628+0000 7f8d41400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f1601401d0 0x55f16014a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:48282/0 (socket says 172.21.15.39:48282) 2024-03-18T16:15:39.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.628+0000 7f8d41400640 1 -- 172.21.15.39:0/3118970492 learned_addr learned my addr 172.21.15.39:0/3118970492 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:15:39.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.628+0000 7f8d41400640 1 -- 172.21.15.39:0/3118970492 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f16013f7e0 msgr2=0x55f160149b70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T16:15:39.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.628+0000 7f8d41400640 1 --2- 172.21.15.39:0/3118970492 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f16013f7e0 0x55f160149b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:39.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.628+0000 7f8d41400640 1 -- 172.21.15.39:0/3118970492 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f160145340 msgr2=0x55f1601510f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:39.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.628+0000 7f8d41400640 1 --2- 172.21.15.39:0/3118970492 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f160145340 0x55f1601510f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:39.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.628+0000 7f8d41400640 1 -- 172.21.15.39:0/3118970492 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f15fe7f7e0 con 0x55f1601401d0 2024-03-18T16:15:39.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.631+0000 7f8d41400640 1 --2- 172.21.15.39:0/3118970492 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f1601401d0 0x55f16014a0b0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f8d3000afd0 tx=0x7f8d3000c5e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=97258f3de92a8881 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:39.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.631+0000 7f8d23400640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:15:39.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.631+0000 7f8d23400640 1 -- 172.21.15.39:0/3118970492 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d30015070 con 0x55f1601401d0 2024-03-18T16:15:39.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.631+0000 7f8d23400640 1 -- 172.21.15.39:0/3118970492 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f8d30011800 con 0x55f1601401d0 2024-03-18T16:15:39.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.631+0000 7f8d4b982740 1 -- 172.21.15.39:0/3118970492 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8d3c004030 con 0x55f1601401d0 2024-03-18T16:15:39.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.632+0000 7f8d23400640 1 -- 172.21.15.39:0/3118970492 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f8d3001f050 con 0x55f1601401d0 2024-03-18T16:15:39.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.632+0000 7f8d4b982740 10 client.26780 fetch_fsmap learned FSMap version 42 2024-03-18T16:15:39.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.632+0000 7f8d4b982740 1 -- 172.21.15.39:0/3118970492 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55f15fedff70 con 0x55f1601401d0 2024-03-18T16:15:39.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.632+0000 7f8d23400640 1 -- 172.21.15.39:0/3118970492 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d3c004030 con 0x55f1601401d0 2024-03-18T16:15:39.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.633+0000 7f8d23400640 1 -- 172.21.15.39:0/3118970492 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 42) ==== 8370+0+0 (secure 0 0 0) 0x7f8d3001ab30 con 0x55f1601401d0 2024-03-18T16:15:39.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.633+0000 7f8d4b982740 10 client.26780 fetch_fsmap finished waiting for FSMap version 42 2024-03-18T16:15:39.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.633+0000 7f8d4b982740 10 client.26780 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:15:39.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.633+0000 7f8d4b982740 20 client.26780 populate_metadata read hostname 'smithi039' 2024-03-18T16:15:39.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.633+0000 7f8d4b982740 1 --2- 172.21.15.39:0/3118970492 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55f160155a90 0x55f160157f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:39.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.633+0000 7f8d4b982740 4 client.26780 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:15:39.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.633+0000 7f8d4b982740 1 -- 172.21.15.39:0/3118970492 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f1600b04c0 con 0x55f160155a90 2024-03-18T16:15:39.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.633+0000 7f8d42800640 1 --2- 172.21.15.39:0/3118970492 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55f160155a90 0x55f160157f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:39.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.634+0000 7f8d42800640 1 --2- 172.21.15.39:0/3118970492 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55f160155a90 0x55f160157f10 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cd8279b02bffa02a server_cookie=4bfd43f910ff6585 in_seq=0 out_seq=0 2024-03-18T16:15:39.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.635+0000 7f8d23400640 10 client.26780 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:15:39.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.635+0000 7f8d23400640 1 -- 172.21.15.39:0/3118970492 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f1600b04c0 con 0x55f160155a90 2024-03-18T16:15:39.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.635+0000 7f8d23400640 10 client.26780 handle_command_reply: tid=0 2024-03-18T16:15:39.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.635+0000 7f8d4b982740 1 client.26780 shutdown 2024-03-18T16:15:39.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.636+0000 7f8d4b982740 1 -- 172.21.15.39:0/3118970492 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f1601401d0 msgr2=0x55f16014a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:39.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.636+0000 7f8d4b982740 1 --2- 172.21.15.39:0/3118970492 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f1601401d0 0x55f16014a0b0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f8d3000afd0 tx=0x7f8d3000c5e0 comp rx=0 tx=0).stop 2024-03-18T16:15:39.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.636+0000 7f8d4b982740 1 -- 172.21.15.39:0/3118970492 shutdown_connections 2024-03-18T16:15:39.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.636+0000 7f8d4b982740 1 --2- 172.21.15.39:0/3118970492 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55f160155a90 0x55f160157f10 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:39.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.636+0000 7f8d4b982740 1 --2- 172.21.15.39:0/3118970492 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f1601401d0 0x55f16014a0b0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:39.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.636+0000 7f8d4b982740 1 --2- 172.21.15.39:0/3118970492 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f160145340 0x55f1601510f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:39.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.636+0000 7f8d4b982740 1 --2- 172.21.15.39:0/3118970492 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f16013f7e0 0x55f160149b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:39.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.636+0000 7f8d4b982740 1 -- 172.21.15.39:0/3118970492 >> 172.21.15.39:0/3118970492 conn(0x55f160107930 msgr2=0x55f16013a350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:39.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.636+0000 7f8d23400640 0 client.26780 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:15:39.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.636+0000 7f8d4b982740 1 -- 172.21.15.39:0/3118970492 shutdown_connections 2024-03-18T16:15:39.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.637+0000 7f8d4b982740 1 -- 172.21.15.39:0/3118970492 wait complete. 2024-03-18T16:15:39.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.637+0000 7f8d4b982740 20 client.26780 trim_cache size 0 max 16384 2024-03-18T16:15:39.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.676+0000 7f8d4b982740 1 Processor -- start 2024-03-18T16:15:39.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.677+0000 7f8d4b982740 1 -- start start 2024-03-18T16:15:39.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.677+0000 7f8d4b982740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f16013f7e0 0x55f1601414f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:39.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.677+0000 7f8d4b982740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f1601401d0 0x55f1601510b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:39.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.678+0000 7f8d4b982740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f160145340 0x55f160141a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:39.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.678+0000 7f8d4b982740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f8d441b9350 con 0x55f1601401d0 2024-03-18T16:15:39.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.678+0000 7f8d42800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f16013f7e0 0x55f1601414f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:39.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.678+0000 7f8d42800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f16013f7e0 0x55f1601414f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:48294/0 (socket says 172.21.15.39:48294) 2024-03-18T16:15:39.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.678+0000 7f8d42800640 1 -- 172.21.15.39:0/4283397798 learned_addr learned my addr 172.21.15.39:0/4283397798 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:15:39.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.678+0000 7f8d4b982740 1 -- 172.21.15.39:0/4283397798 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f8d441bae90 con 0x55f16013f7e0 2024-03-18T16:15:39.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.678+0000 7f8d4b982740 1 -- 172.21.15.39:0/4283397798 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f8d441b9a20 con 0x55f160145340 2024-03-18T16:15:39.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.678+0000 7f8d41400640 1 --2- 172.21.15.39:0/4283397798 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f1601401d0 0x55f1601510b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:39.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.680+0000 7f8d41400640 1 -- 172.21.15.39:0/4283397798 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f160145340 msgr2=0x55f160141a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:39.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.680+0000 7f8d41400640 1 --2- 172.21.15.39:0/4283397798 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f160145340 0x55f160141a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:39.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.680+0000 7f8d41400640 1 -- 172.21.15.39:0/4283397798 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f16013f7e0 msgr2=0x55f1601414f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:39.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.681+0000 7f8d41400640 1 --2- 172.21.15.39:0/4283397798 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f16013f7e0 0x55f1601414f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:39.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.681+0000 7f8d41400640 1 -- 172.21.15.39:0/4283397798 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f15fd77170 con 0x55f1601401d0 2024-03-18T16:15:39.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.681+0000 7f8d42800640 1 --2- 172.21.15.39:0/4283397798 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f16013f7e0 0x55f1601414f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:15:39.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.682+0000 7f8d41400640 1 --2- 172.21.15.39:0/4283397798 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f1601401d0 0x55f1601510b0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f8d30000ea0 tx=0x7f8d3000a220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9247fc3a1ee2f85b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:39.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.683+0000 7f8d2e600640 1 -- 172.21.15.39:0/4283397798 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d3002b020 con 0x55f1601401d0 2024-03-18T16:15:39.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.683+0000 7f8d2e600640 1 -- 172.21.15.39:0/4283397798 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d30003690 con 0x55f1601401d0 2024-03-18T16:15:39.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.683+0000 7f8d2e600640 1 -- 172.21.15.39:0/4283397798 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d30015040 con 0x55f1601401d0 2024-03-18T16:15:39.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.684+0000 7f8d4b982740 1 -- 172.21.15.39:0/4283397798 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f1601401d0 msgr2=0x55f1601510b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:39.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.684+0000 7f8d4b982740 1 --2- 172.21.15.39:0/4283397798 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f1601401d0 0x55f1601510b0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f8d30000ea0 tx=0x7f8d3000a220 comp rx=0 tx=0).stop 2024-03-18T16:15:39.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.684+0000 7f8d4b982740 1 -- 172.21.15.39:0/4283397798 shutdown_connections 2024-03-18T16:15:39.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.684+0000 7f8d4b982740 1 --2- 172.21.15.39:0/4283397798 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f160145340 0x55f160141a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:39.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.684+0000 7f8d4b982740 1 --2- 172.21.15.39:0/4283397798 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f1601401d0 0x55f1601510b0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:39.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.684+0000 7f8d4b982740 1 --2- 172.21.15.39:0/4283397798 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f16013f7e0 0x55f1601414f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:39.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.684+0000 7f8d4b982740 1 -- 172.21.15.39:0/4283397798 >> 172.21.15.39:0/4283397798 conn(0x55f160107930 msgr2=0x55f160149b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:39.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.685+0000 7f8d4b982740 1 -- 172.21.15.39:0/4283397798 shutdown_connections 2024-03-18T16:15:39.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.685+0000 7f8d4b982740 1 -- 172.21.15.39:0/4283397798 wait complete. 2024-03-18T16:15:39.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.685+0000 7f8d4b982740 1 -- start start 2024-03-18T16:15:39.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.686+0000 7f8d4b982740 1 Processor -- start 2024-03-18T16:15:39.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.686+0000 7f8d4b982740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f16013f7e0 0x55f16014e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:39.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.686+0000 7f8d4b982740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f1601401d0 0x55f16014ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:39.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.686+0000 7f8d4b982740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f160145340 0x55f16014fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:39.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.686+0000 7f8d4b982740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f8d441ba7c0 con 0x55f160145340 2024-03-18T16:15:39.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.686+0000 7f8d4b982740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f8d441ba0f0 con 0x55f16013f7e0 2024-03-18T16:15:39.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.686+0000 7f8d4b982740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f8d441b7330 con 0x55f1601401d0 2024-03-18T16:15:39.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.687+0000 7f8d4b982740 10 client.0 _ll_register_callbacks cb 0x7f8d3000ab60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:15:39.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.688+0000 7f8d41400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f1601401d0 0x55f16014ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:39.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.688+0000 7f8d41400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f1601401d0 0x55f16014ecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:46000/0 (socket says 172.21.15.39:46000) 2024-03-18T16:15:39.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.688+0000 7f8d41400640 1 -- 172.21.15.39:0/4156728005 learned_addr learned my addr 172.21.15.39:0/4156728005 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:15:39.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.688+0000 7f8d42800640 1 --2- 172.21.15.39:0/4156728005 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f16013f7e0 0x55f16014e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:39.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.688+0000 7f8d42800640 1 --2- 172.21.15.39:0/4156728005 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f160145340 0x55f16014fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:39.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.689+0000 7f8d41400640 1 -- 172.21.15.39:0/4156728005 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f16013f7e0 msgr2=0x55f16014e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:39.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.689+0000 7f8d41400640 1 --2- 172.21.15.39:0/4156728005 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f16013f7e0 0x55f16014e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:39.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.689+0000 7f8d41400640 1 -- 172.21.15.39:0/4156728005 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f160145340 msgr2=0x55f16014fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:39.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.689+0000 7f8d41400640 1 --2- 172.21.15.39:0/4156728005 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f160145340 0x55f16014fcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:39.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.689+0000 7f8d41400640 1 -- 172.21.15.39:0/4156728005 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f1601502c0 con 0x55f1601401d0 2024-03-18T16:15:39.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.689+0000 7f8d41400640 1 --2- 172.21.15.39:0/4156728005 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f1601401d0 0x55f16014ecc0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f8d3000e010 tx=0x7f8d30004230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d882039a1d704716 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:39.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.689+0000 7f8d23400640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:15:39.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.689+0000 7f8d23400640 1 -- 172.21.15.39:0/4156728005 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d30035060 con 0x55f1601401d0 2024-03-18T16:15:39.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.689+0000 7f8d4b982740 1 -- 172.21.15.39:0/4156728005 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8d30003690 con 0x55f1601401d0 2024-03-18T16:15:39.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.690+0000 7f8d23400640 1 -- 172.21.15.39:0/4156728005 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f8d30011800 con 0x55f1601401d0 2024-03-18T16:15:39.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.690+0000 7f8d23400640 1 -- 172.21.15.39:0/4156728005 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f8d3003c050 con 0x55f1601401d0 2024-03-18T16:15:39.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.690+0000 7f8d4b982740 10 client.26746 fetch_fsmap learned FSMap version 42 2024-03-18T16:15:39.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.690+0000 7f8d4b982740 1 -- 172.21.15.39:0/4156728005 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55f1601bbe50 con 0x55f1601401d0 2024-03-18T16:15:39.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.690+0000 7f8d23400640 1 -- 172.21.15.39:0/4156728005 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d30003690 con 0x55f1601401d0 2024-03-18T16:15:39.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.690+0000 7f8d23400640 1 -- 172.21.15.39:0/4156728005 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 42) ==== 8370+0+0 (secure 0 0 0) 0x7f8d3001aa60 con 0x55f1601401d0 2024-03-18T16:15:39.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.690+0000 7f8d4b982740 10 client.26746 fetch_fsmap finished waiting for FSMap version 42 2024-03-18T16:15:39.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.690+0000 7f8d4b982740 10 client.26746 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:15:39.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.690+0000 7f8d4b982740 20 client.26746 populate_metadata read hostname 'smithi039' 2024-03-18T16:15:39.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.691+0000 7f8d4b982740 1 --2- 172.21.15.39:0/4156728005 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55f1601bc7d0 0x55f1601bebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:39.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.691+0000 7f8d4b982740 4 client.26746 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-18T16:15:39.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.691+0000 7f8d4b982740 1 -- 172.21.15.39:0/4156728005 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55f1600aa810 con 0x55f1601bc7d0 2024-03-18T16:15:39.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.691+0000 7f8d42800640 1 --2- 172.21.15.39:0/4156728005 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55f1601bc7d0 0x55f1601bebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:39.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.691+0000 7f8d23400640 10 client.26746 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:15:39.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.691+0000 7f8d42800640 1 --2- 172.21.15.39:0/4156728005 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55f1601bc7d0 0x55f1601bebf0 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d6520b556f87738b server_cookie=f9bc0271659c9b9c in_seq=0 out_seq=0 2024-03-18T16:15:39.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.692+0000 7f8d23400640 1 -- 172.21.15.39:0/4156728005 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55f1600aa810 con 0x55f1601bc7d0 2024-03-18T16:15:39.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.692+0000 7f8d23400640 10 client.26746 handle_command_reply: tid=0 2024-03-18T16:15:39.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.692+0000 7f8d4b982740 1 client.26746 shutdown 2024-03-18T16:15:39.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.693+0000 7f8d4b982740 1 -- 172.21.15.39:0/4156728005 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f1601401d0 msgr2=0x55f16014ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:39.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.693+0000 7f8d4b982740 1 --2- 172.21.15.39:0/4156728005 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f1601401d0 0x55f16014ecc0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f8d3000e010 tx=0x7f8d30004230 comp rx=0 tx=0).stop 2024-03-18T16:15:39.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.693+0000 7f8d4b982740 1 -- 172.21.15.39:0/4156728005 shutdown_connections 2024-03-18T16:15:39.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.693+0000 7f8d4b982740 1 --2- 172.21.15.39:0/4156728005 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55f1601bc7d0 0x55f1601bebf0 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:39.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.693+0000 7f8d4b982740 1 --2- 172.21.15.39:0/4156728005 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f160145340 0x55f16014fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:39.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.693+0000 7f8d4b982740 1 --2- 172.21.15.39:0/4156728005 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f1601401d0 0x55f16014ecc0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:39.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.693+0000 7f8d4b982740 1 --2- 172.21.15.39:0/4156728005 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f16013f7e0 0x55f16014e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:39.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.693+0000 7f8d4b982740 1 -- 172.21.15.39:0/4156728005 >> 172.21.15.39:0/4156728005 conn(0x55f160107930 msgr2=0x55f16013ec20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:39.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.693+0000 7f8d23400640 0 client.26746 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:15:39.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.693+0000 7f8d4b982740 1 -- 172.21.15.39:0/4156728005 shutdown_connections 2024-03-18T16:15:39.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.693+0000 7f8d4b982740 1 -- 172.21.15.39:0/4156728005 wait complete. 2024-03-18T16:15:39.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.694+0000 7f8d4b982740 20 client.26746 trim_cache size 0 max 16384 2024-03-18T16:15:39.696 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-03-18T16:15:39.696 INFO:teuthology.orchestra.run.smithi039.stdout: "return_code": 0, 2024-03-18T16:15:39.696 INFO:teuthology.orchestra.run.smithi039.stdout: "scrub_tag": "9a14398f-20ed-4130-a602-b636170af3be", 2024-03-18T16:15:39.696 INFO:teuthology.orchestra.run.smithi039.stdout: "mode": "asynchronous" 2024-03-18T16:15:39.697 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-03-18T16:15:39.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.695+0000 7f8d48c00640 1 -- 172.21.15.39:0/2108206198 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f8d280a2800 msgr2=0x7f8d280a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:39.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.695+0000 7f8d48c00640 1 --2- 172.21.15.39:0/2108206198 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f8d280a2800 0x7f8d280a4cc0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x55f1601405d0 tx=0x7f8d340f47a0 comp rx=0 tx=0).stop 2024-03-18T16:15:39.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.695+0000 7f8d48c00640 1 -- 172.21.15.39:0/2108206198 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8d441017f0 msgr2=0x7f8d441959f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:39.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.695+0000 7f8d48c00640 1 --2- 172.21.15.39:0/2108206198 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8d441017f0 0x7f8d441959f0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f8d3402efd0 tx=0x7f8d34005e40 comp rx=0 tx=0).stop 2024-03-18T16:15:39.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.695+0000 7f8d48c00640 1 -- 172.21.15.39:0/2108206198 shutdown_connections 2024-03-18T16:15:39.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.695+0000 7f8d48c00640 1 --2- 172.21.15.39:0/2108206198 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8d441033a0 0x7f8d441ae440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:39.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.695+0000 7f8d48c00640 1 --2- 172.21.15.39:0/2108206198 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f8d280a2800 0x7f8d280a4cc0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:39.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.695+0000 7f8d48c00640 1 --2- 172.21.15.39:0/2108206198 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8d441029e0 0x7f8d44195f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:39.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.695+0000 7f8d48c00640 1 --2- 172.21.15.39:0/2108206198 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8d441017f0 0x7f8d441959f0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:39.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.695+0000 7f8d48c00640 1 -- 172.21.15.39:0/2108206198 >> 172.21.15.39:0/2108206198 conn(0x7f8d440fcd80 msgr2=0x7f8d44103c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:39.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.695+0000 7f8d48c00640 1 -- 172.21.15.39:0/2108206198 shutdown_connections 2024-03-18T16:15:39.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.695+0000 7f8d48c00640 1 -- 172.21.15.39:0/2108206198 wait complete. 2024-03-18T16:15:39.724 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T16:15:39.724 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T16:15:40.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.999+0000 7f04c6200640 1 -- 172.21.15.39:0/3113527163 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f04c0103a20 msgr2=0x7f04c0103e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:40.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.999+0000 7f04c6200640 1 --2- 172.21.15.39:0/3113527163 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f04c0103a20 0x7f04c0103e40 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f04b0009ca0 tx=0x7f04b002c100 comp rx=0 tx=0).stop 2024-03-18T16:15:40.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.999+0000 7f04c6200640 1 -- 172.21.15.39:0/3113527163 shutdown_connections 2024-03-18T16:15:40.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.999+0000 7f04c6200640 1 --2- 172.21.15.39:0/3113527163 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f04c01055d0 0x7f04c010be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:40.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.999+0000 7f04c6200640 1 --2- 172.21.15.39:0/3113527163 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04c0104c10 0x7f04c0105090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:40.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.999+0000 7f04c6200640 1 --2- 172.21.15.39:0/3113527163 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f04c0103a20 0x7f04c0103e40 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:40.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.999+0000 7f04c6200640 1 -- 172.21.15.39:0/3113527163 >> 172.21.15.39:0/3113527163 conn(0x7f04c00fef90 msgr2=0x7f04c01013d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:40.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.999+0000 7f04c6200640 1 -- 172.21.15.39:0/3113527163 shutdown_connections 2024-03-18T16:15:40.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:39.999+0000 7f04c6200640 1 -- 172.21.15.39:0/3113527163 wait complete. 2024-03-18T16:15:40.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.001+0000 7f04c6200640 1 Processor -- start 2024-03-18T16:15:40.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.001+0000 7f04c6200640 1 -- start start 2024-03-18T16:15:40.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.001+0000 7f04c6200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f04c0103a20 0x7f04c01958e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:40.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.001+0000 7f04c6200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f04c0104c10 0x7f04c0195e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:40.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.001+0000 7f04c6200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04c01055d0 0x7f04c019ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:40.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.001+0000 7f04c6200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f04c010f670 con 0x7f04c01055d0 2024-03-18T16:15:40.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.001+0000 7f04c6200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f04c010f4f0 con 0x7f04c0103a20 2024-03-18T16:15:40.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.001+0000 7f04c6200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f04c010f7f0 con 0x7f04c0104c10 2024-03-18T16:15:40.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.001+0000 7f04bf400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f04c0103a20 0x7f04c01958e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:40.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.002+0000 7f04bf400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f04c0103a20 0x7f04c01958e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:48316/0 (socket says 172.21.15.39:48316) 2024-03-18T16:15:40.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.002+0000 7f04bf400640 1 -- 172.21.15.39:0/154441844 learned_addr learned my addr 172.21.15.39:0/154441844 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:15:40.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.002+0000 7f04b6a00640 1 --2- 172.21.15.39:0/154441844 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f04c0104c10 0x7f04c0195e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:40.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.002+0000 7f04bf400640 1 -- 172.21.15.39:0/154441844 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f04c0104c10 msgr2=0x7f04c0195e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:40.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.002+0000 7f04bf400640 1 --2- 172.21.15.39:0/154441844 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f04c0104c10 0x7f04c0195e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:40.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.002+0000 7f04bf400640 1 -- 172.21.15.39:0/154441844 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04c01055d0 msgr2=0x7f04c019ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:40.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.002+0000 7f04bf400640 1 --2- 172.21.15.39:0/154441844 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04c01055d0 0x7f04c019ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:40.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.002+0000 7f04bf400640 1 -- 172.21.15.39:0/154441844 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04c01ae570 con 0x7f04c0103a20 2024-03-18T16:15:40.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.003+0000 7f04bf400640 1 --2- 172.21.15.39:0/154441844 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f04c0103a20 0x7f04c01958e0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f04b0000f80 tx=0x7f04b0035d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:40.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.003+0000 7f04bcc00640 1 -- 172.21.15.39:0/154441844 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04b0039040 con 0x7f04c0103a20 2024-03-18T16:15:40.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.003+0000 7f04bcc00640 1 -- 172.21.15.39:0/154441844 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f04b0044070 con 0x7f04c0103a20 2024-03-18T16:15:40.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.003+0000 7f04bcc00640 1 -- 172.21.15.39:0/154441844 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04b00044a0 con 0x7f04c0103a20 2024-03-18T16:15:40.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.003+0000 7f04c6200640 1 -- 172.21.15.39:0/154441844 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04c01ae7a0 con 0x7f04c0103a20 2024-03-18T16:15:40.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.003+0000 7f04c6200640 1 -- 172.21.15.39:0/154441844 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04c01aec10 con 0x7f04c0103a20 2024-03-18T16:15:40.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.007+0000 7f04bcc00640 1 -- 172.21.15.39:0/154441844 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f04b0004640 con 0x7f04c0103a20 2024-03-18T16:15:40.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.008+0000 7f04c8f83740 1 Processor -- start 2024-03-18T16:15:40.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.008+0000 7f04c8f83740 1 -- start start 2024-03-18T16:15:40.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.009+0000 7f04c8f83740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55adaad2a810 0x55adaad2ac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:40.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.009+0000 7f04c8f83740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55adaad2b200 0x55adaad2fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:40.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.009+0000 7f04c8f83740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55adaad30370 0x55adaad32800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:40.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.009+0000 7f04c8f83740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55adaac8f1f0 con 0x55adaad2b200 2024-03-18T16:15:40.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.009+0000 7f04c8f83740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55adaac37a80 con 0x55adaad2a810 2024-03-18T16:15:40.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.009+0000 7f04c8f83740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55adaac8b290 con 0x55adaad30370 2024-03-18T16:15:40.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.009+0000 7f04bfe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55adaad2a810 0x55adaad2ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:40.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.009+0000 7f04bfe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55adaad2a810 0x55adaad2ac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:48332/0 (socket says 172.21.15.39:48332) 2024-03-18T16:15:40.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.009+0000 7f04bfe00640 1 -- 172.21.15.39:0/2448151223 learned_addr learned my addr 172.21.15.39:0/2448151223 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:15:40.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.010+0000 7f04bcc00640 1 --2- 172.21.15.39:0/154441844 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f04a40a2900 0x7f04a40a4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:40.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.010+0000 7f04bcc00640 1 -- 172.21.15.39:0/154441844 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f04b00f1420 con 0x7f04c0103a20 2024-03-18T16:15:40.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.011+0000 7f04b6a00640 1 --2- 172.21.15.39:0/2448151223 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55adaad30370 0x55adaad32800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:40.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.011+0000 7f04bfe00640 1 -- 172.21.15.39:0/2448151223 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55adaad30370 msgr2=0x55adaad32800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:40.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.011+0000 7f04bfe00640 1 --2- 172.21.15.39:0/2448151223 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55adaad30370 0x55adaad32800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:40.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.011+0000 7f04bfe00640 1 -- 172.21.15.39:0/2448151223 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55adaad2b200 msgr2=0x55adaad2fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:40.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.011+0000 7f04bf400640 1 --2- 172.21.15.39:0/2448151223 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55adaad2b200 0x55adaad2fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:40.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.011+0000 7f04bfe00640 1 --2- 172.21.15.39:0/2448151223 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55adaad2b200 0x55adaad2fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:40.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.011+0000 7f04bfe00640 1 -- 172.21.15.39:0/2448151223 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55adaaa6d7d0 con 0x55adaad2a810 2024-03-18T16:15:40.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.011+0000 7f04bf400640 1 --2- 172.21.15.39:0/2448151223 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55adaad2b200 0x55adaad2fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:15:40.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.012+0000 7f04bfe00640 1 --2- 172.21.15.39:0/2448151223 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55adaad2a810 0x55adaad2ac30 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f04b800a9f0 tx=0x7f04b800aeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f80e420cfff98fa server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:40.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.012+0000 7f04bfe00640 1 --2- 172.21.15.39:0/154441844 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f04a40a2900 0x7f04a40a4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:40.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.013+0000 7f04afe00640 1 -- 172.21.15.39:0/2448151223 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04b800f710 con 0x55adaad2a810 2024-03-18T16:15:40.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.013+0000 7f04afe00640 1 -- 172.21.15.39:0/2448151223 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f04b800fd90 con 0x55adaad2a810 2024-03-18T16:15:40.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.013+0000 7f04afe00640 1 -- 172.21.15.39:0/2448151223 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04b8019da0 con 0x55adaad2a810 2024-03-18T16:15:40.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.013+0000 7f04bfe00640 1 --2- 172.21.15.39:0/154441844 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f04a40a2900 0x7f04a40a4dc0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f04b8019000 tx=0x7f04b801c000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:40.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.015+0000 7f04c8f83740 1 -- 172.21.15.39:0/2448151223 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55adaad2a810 msgr2=0x55adaad2ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:40.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.015+0000 7f04c8f83740 1 --2- 172.21.15.39:0/2448151223 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55adaad2a810 0x55adaad2ac30 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f04b800a9f0 tx=0x7f04b800aeb0 comp rx=0 tx=0).stop 2024-03-18T16:15:40.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.015+0000 7f04c8f83740 1 -- 172.21.15.39:0/2448151223 shutdown_connections 2024-03-18T16:15:40.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.015+0000 7f04c8f83740 1 --2- 172.21.15.39:0/2448151223 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55adaad30370 0x55adaad32800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:40.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.015+0000 7f04c8f83740 1 --2- 172.21.15.39:0/2448151223 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55adaad2b200 0x55adaad2fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:40.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.015+0000 7f04c8f83740 1 --2- 172.21.15.39:0/2448151223 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55adaad2a810 0x55adaad2ac30 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:40.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.015+0000 7f04c8f83740 1 -- 172.21.15.39:0/2448151223 >> 172.21.15.39:0/2448151223 conn(0x55adaad25d80 msgr2=0x55adaad26180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:40.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.015+0000 7f04c8f83740 1 -- 172.21.15.39:0/2448151223 shutdown_connections 2024-03-18T16:15:40.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.016+0000 7f04c8f83740 1 -- 172.21.15.39:0/2448151223 wait complete. 2024-03-18T16:15:40.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.017+0000 7f04c8f83740 1 -- start start 2024-03-18T16:15:40.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.017+0000 7f04c8f83740 1 Processor -- start 2024-03-18T16:15:40.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.018+0000 7f04c8f83740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55adaad2a810 0x55adaad34cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:40.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.018+0000 7f04c8f83740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55adaad2b200 0x55adaad35230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:40.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.018+0000 7f04c8f83740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55adaad30370 0x55adaad3c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:40.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.018+0000 7f04c8f83740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55adaad332c0 con 0x55adaad30370 2024-03-18T16:15:40.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.018+0000 7f04c8f83740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55adaac4aed0 con 0x55adaad2b200 2024-03-18T16:15:40.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.018+0000 7f04c8f83740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55adaad33440 con 0x55adaad2a810 2024-03-18T16:15:40.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.018+0000 7f04c8f83740 10 client.0 _ll_register_callbacks cb 0x55adaac86220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:15:40.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.018+0000 7f04bfe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55adaad30370 0x55adaad3c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:40.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.018+0000 7f04bf400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55adaad2a810 0x55adaad34cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:40.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.018+0000 7f04bfe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55adaad30370 0x55adaad3c260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:35386/0 (socket says 172.21.15.39:35386) 2024-03-18T16:15:40.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.018+0000 7f04bfe00640 1 -- 172.21.15.39:0/1110809665 learned_addr learned my addr 172.21.15.39:0/1110809665 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:15:40.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.019+0000 7f04bf400640 1 -- 172.21.15.39:0/1110809665 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55adaad2b200 msgr2=0x55adaad35230 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:15:40.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.019+0000 7f04bf400640 1 --2- 172.21.15.39:0/1110809665 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55adaad2b200 0x55adaad35230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:40.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.019+0000 7f04bf400640 1 -- 172.21.15.39:0/1110809665 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55adaad30370 msgr2=0x55adaad3c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:40.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.019+0000 7f04bf400640 1 --2- 172.21.15.39:0/1110809665 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55adaad30370 0x55adaad3c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:40.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.019+0000 7f04bf400640 1 -- 172.21.15.39:0/1110809665 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55adaa9a33f0 con 0x55adaad2a810 2024-03-18T16:15:40.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.019+0000 7f04bf400640 1 --2- 172.21.15.39:0/1110809665 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55adaad2a810 0x55adaad34cf0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x55adaad2bc70 tx=0x7f04b00b9610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=461a0d479854bbac server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:40.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.020+0000 7f04acc00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:15:40.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.020+0000 7f04acc00640 1 -- 172.21.15.39:0/1110809665 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04b00bd330 con 0x55adaad2a810 2024-03-18T16:15:40.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.021+0000 7f04acc00640 1 -- 172.21.15.39:0/1110809665 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f04b00f10d0 con 0x55adaad2a810 2024-03-18T16:15:40.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.021+0000 7f04c8f83740 1 -- 172.21.15.39:0/1110809665 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f04b800fd90 con 0x55adaad2a810 2024-03-18T16:15:40.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.021+0000 7f04acc00640 1 -- 172.21.15.39:0/1110809665 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f04b00be0d0 con 0x55adaad2a810 2024-03-18T16:15:40.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.021+0000 7f04c8f83740 10 client.26752 fetch_fsmap learned FSMap version 42 2024-03-18T16:15:40.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.021+0000 7f04c8f83740 1 -- 172.21.15.39:0/1110809665 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55adaa9630f0 con 0x55adaad2a810 2024-03-18T16:15:40.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.022+0000 7f04acc00640 1 -- 172.21.15.39:0/1110809665 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f04b800fd90 con 0x55adaad2a810 2024-03-18T16:15:40.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.022+0000 7f04acc00640 1 -- 172.21.15.39:0/1110809665 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 42) ==== 8370+0+0 (secure 0 0 0) 0x7f04b003c030 con 0x55adaad2a810 2024-03-18T16:15:40.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.022+0000 7f04c8f83740 10 client.26752 fetch_fsmap finished waiting for FSMap version 42 2024-03-18T16:15:40.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.022+0000 7f04c8f83740 10 client.26752 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:15:40.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.022+0000 7f04c8f83740 20 client.26752 populate_metadata read hostname 'smithi039' 2024-03-18T16:15:40.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.022+0000 7f04c8f83740 1 --2- 172.21.15.39:0/1110809665 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55adaad40af0 0x55adaad42f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:40.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.022+0000 7f04c8f83740 4 client.26752 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:15:40.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.022+0000 7f04c8f83740 1 -- 172.21.15.39:0/1110809665 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55adaaca2be0 con 0x55adaad40af0 2024-03-18T16:15:40.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.024+0000 7f04b6a00640 1 --2- 172.21.15.39:0/1110809665 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55adaad40af0 0x55adaad42f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:40.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.024+0000 7f04b6a00640 1 --2- 172.21.15.39:0/1110809665 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55adaad40af0 0x55adaad42f80 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=78475e0849c14b94 server_cookie=61ca68be581d2d79 in_seq=0 out_seq=0 2024-03-18T16:15:40.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.024+0000 7f04acc00640 10 client.26752 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:15:40.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.025+0000 7f04acc00640 1 -- 172.21.15.39:0/1110809665 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55adaaca2be0 con 0x55adaad40af0 2024-03-18T16:15:40.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.025+0000 7f04acc00640 10 client.26752 handle_command_reply: tid=0 2024-03-18T16:15:40.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.025+0000 7f04c8f83740 1 client.26752 shutdown 2024-03-18T16:15:40.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.025+0000 7f04c8f83740 1 -- 172.21.15.39:0/1110809665 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55adaad2a810 msgr2=0x55adaad34cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:40.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.026+0000 7f04c8f83740 1 --2- 172.21.15.39:0/1110809665 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55adaad2a810 0x55adaad34cf0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x55adaad2bc70 tx=0x7f04b00b9610 comp rx=0 tx=0).stop 2024-03-18T16:15:40.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.026+0000 7f04c8f83740 1 -- 172.21.15.39:0/1110809665 shutdown_connections 2024-03-18T16:15:40.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.026+0000 7f04c8f83740 1 --2- 172.21.15.39:0/1110809665 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55adaad40af0 0x55adaad42f80 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:40.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.026+0000 7f04c8f83740 1 --2- 172.21.15.39:0/1110809665 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55adaad2b200 0x55adaad35230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:40.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.026+0000 7f04c8f83740 1 --2- 172.21.15.39:0/1110809665 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55adaad30370 0x55adaad3c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:40.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.026+0000 7f04c8f83740 1 --2- 172.21.15.39:0/1110809665 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55adaad2a810 0x55adaad34cf0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:40.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.026+0000 7f04c8f83740 1 -- 172.21.15.39:0/1110809665 >> 172.21.15.39:0/1110809665 conn(0x55adaad25d80 msgr2=0x55adaad24ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:40.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.026+0000 7f04acc00640 0 client.26752 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:15:40.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.026+0000 7f04c8f83740 1 -- 172.21.15.39:0/1110809665 shutdown_connections 2024-03-18T16:15:40.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.026+0000 7f04c8f83740 1 -- 172.21.15.39:0/1110809665 wait complete. 2024-03-18T16:15:40.030 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.026+0000 7f04c8f83740 20 client.26752 trim_cache size 0 max 16384 2024-03-18T16:15:40.059 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.057+0000 7f04c8f83740 1 Processor -- start 2024-03-18T16:15:40.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.057+0000 7f04c8f83740 1 -- start start 2024-03-18T16:15:40.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.057+0000 7f04c8f83740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55adaad25d80 0x55adaad24630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:40.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.057+0000 7f04c8f83740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55adaad2a810 0x55adaad3c130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:40.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.057+0000 7f04c8f83740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55adaad2b200 0x55adaad24b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:40.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.057+0000 7f04c8f83740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f04c01b9240 con 0x55adaad25d80 2024-03-18T16:15:40.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.058+0000 7f04c8f83740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f04c01bad80 con 0x55adaad2a810 2024-03-18T16:15:40.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.058+0000 7f04c8f83740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f04c01b9910 con 0x55adaad2b200 2024-03-18T16:15:40.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.058+0000 7f04b6a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55adaad2a810 0x55adaad3c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:40.060 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.058+0000 7f04bfe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55adaad2b200 0x55adaad24b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:40.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.058+0000 7f04b6a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55adaad2a810 0x55adaad3c130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:48334/0 (socket says 172.21.15.39:48334) 2024-03-18T16:15:40.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.058+0000 7f04bfe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55adaad2b200 0x55adaad24b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:46056/0 (socket says 172.21.15.39:46056) 2024-03-18T16:15:40.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.058+0000 7f04b6a00640 1 -- 172.21.15.39:0/2285087116 learned_addr learned my addr 172.21.15.39:0/2285087116 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:15:40.061 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.058+0000 7f04bf400640 1 --2- 172.21.15.39:0/2285087116 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55adaad25d80 0x55adaad24630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:40.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.059+0000 7f04bfe00640 1 -- 172.21.15.39:0/2285087116 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55adaad2a810 msgr2=0x55adaad3c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:40.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.059+0000 7f04bfe00640 1 --2- 172.21.15.39:0/2285087116 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55adaad2a810 0x55adaad3c130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:40.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.059+0000 7f04bfe00640 1 -- 172.21.15.39:0/2285087116 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55adaad25d80 msgr2=0x55adaad24630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:40.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.059+0000 7f04bfe00640 1 --2- 172.21.15.39:0/2285087116 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55adaad25d80 0x55adaad24630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:40.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.059+0000 7f04bfe00640 1 -- 172.21.15.39:0/2285087116 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55adaad2ce20 con 0x55adaad2b200 2024-03-18T16:15:40.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.059+0000 7f04bf400640 1 --2- 172.21.15.39:0/2285087116 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55adaad25d80 0x55adaad24630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:15:40.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.059+0000 7f04bfe00640 1 --2- 172.21.15.39:0/2285087116 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55adaad2b200 0x55adaad24b70 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f04b800cf80 tx=0x7f04b8023000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3142315f9c66d9ee server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:40.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.060+0000 7f04af400640 1 -- 172.21.15.39:0/2285087116 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04b8005e50 con 0x55adaad2b200 2024-03-18T16:15:40.062 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.060+0000 7f04af400640 1 -- 172.21.15.39:0/2285087116 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f04b800cbe0 con 0x55adaad2b200 2024-03-18T16:15:40.063 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.060+0000 7f04af400640 1 -- 172.21.15.39:0/2285087116 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04b8019820 con 0x55adaad2b200 2024-03-18T16:15:40.063 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.060+0000 7f04c8f83740 1 -- 172.21.15.39:0/2285087116 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55adaad2b200 msgr2=0x55adaad24b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:40.063 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.060+0000 7f04c8f83740 1 --2- 172.21.15.39:0/2285087116 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55adaad2b200 0x55adaad24b70 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f04b800cf80 tx=0x7f04b8023000 comp rx=0 tx=0).stop 2024-03-18T16:15:40.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.061+0000 7f04c8f83740 1 -- 172.21.15.39:0/2285087116 shutdown_connections 2024-03-18T16:15:40.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.061+0000 7f04c8f83740 1 --2- 172.21.15.39:0/2285087116 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55adaad2a810 0x55adaad3c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:40.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.061+0000 7f04c8f83740 1 --2- 172.21.15.39:0/2285087116 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55adaad2b200 0x55adaad24b70 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:40.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.061+0000 7f04c8f83740 1 --2- 172.21.15.39:0/2285087116 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55adaad25d80 0x55adaad24630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:40.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.061+0000 7f04c8f83740 1 -- 172.21.15.39:0/2285087116 >> 172.21.15.39:0/2285087116 conn(0x55adaad264d0 msgr2=0x55adaad337e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:40.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.061+0000 7f04c8f83740 1 -- 172.21.15.39:0/2285087116 shutdown_connections 2024-03-18T16:15:40.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.061+0000 7f04c8f83740 1 -- 172.21.15.39:0/2285087116 wait complete. 2024-03-18T16:15:40.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.061+0000 7f04c8f83740 1 -- start start 2024-03-18T16:15:40.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.062+0000 7f04c8f83740 1 Processor -- start 2024-03-18T16:15:40.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.062+0000 7f04c8f83740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55adaad25d80 0x55adaad3a050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:40.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.062+0000 7f04c8f83740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55adaad2a810 0x55adaad3a590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:40.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.062+0000 7f04c8f83740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55adaad2b200 0x55adaad3b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:40.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.062+0000 7f04c8f83740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f04c01ba6b0 con 0x55adaad25d80 2024-03-18T16:15:40.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.062+0000 7f04c8f83740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f04c01b9fe0 con 0x55adaad2a810 2024-03-18T16:15:40.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.062+0000 7f04c8f83740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f04c01b7220 con 0x55adaad2b200 2024-03-18T16:15:40.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.062+0000 7f04c8f83740 10 client.0 _ll_register_callbacks cb 0x7f04b0035f00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:15:40.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.063+0000 7f04bfe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55adaad2b200 0x55adaad3b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:40.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.063+0000 7f04bfe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55adaad2b200 0x55adaad3b5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:46064/0 (socket says 172.21.15.39:46064) 2024-03-18T16:15:40.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.063+0000 7f04bfe00640 1 -- 172.21.15.39:0/418371554 learned_addr learned my addr 172.21.15.39:0/418371554 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:15:40.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.063+0000 7f04bf400640 1 --2- 172.21.15.39:0/418371554 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55adaad25d80 0x55adaad3a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:40.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.063+0000 7f04b6a00640 1 --2- 172.21.15.39:0/418371554 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55adaad2a810 0x55adaad3a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:40.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.063+0000 7f04bfe00640 1 -- 172.21.15.39:0/418371554 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55adaad2a810 msgr2=0x55adaad3a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:40.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.063+0000 7f04bfe00640 1 --2- 172.21.15.39:0/418371554 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55adaad2a810 0x55adaad3a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:40.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.063+0000 7f04bfe00640 1 -- 172.21.15.39:0/418371554 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55adaad25d80 msgr2=0x55adaad3a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:15:40.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.063+0000 7f04bfe00640 1 --2- 172.21.15.39:0/418371554 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55adaad25d80 0x55adaad3a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:40.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.063+0000 7f04bfe00640 1 -- 172.21.15.39:0/418371554 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55adaad3bb90 con 0x55adaad2b200 2024-03-18T16:15:40.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.064+0000 7f04bf400640 1 --2- 172.21.15.39:0/418371554 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55adaad25d80 0x55adaad3a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:15:40.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.064+0000 7f04bfe00640 1 --2- 172.21.15.39:0/418371554 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55adaad2b200 0x55adaad3b5a0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f04b8019420 tx=0x7f04b800fca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b2c7fafc03466d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:15:40.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.064+0000 7f04acc00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:15:40.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.064+0000 7f04acc00640 1 -- 172.21.15.39:0/418371554 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04b800fe10 con 0x55adaad2b200 2024-03-18T16:15:40.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.064+0000 7f04acc00640 1 -- 172.21.15.39:0/418371554 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f04b8005df0 con 0x55adaad2b200 2024-03-18T16:15:40.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.065+0000 7f04c8f83740 1 -- 172.21.15.39:0/418371554 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f04b800cbe0 con 0x55adaad2b200 2024-03-18T16:15:40.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.065+0000 7f04acc00640 1 -- 172.21.15.39:0/418371554 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f04b8005240 con 0x55adaad2b200 2024-03-18T16:15:40.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.065+0000 7f04c8f83740 10 client.26758 fetch_fsmap learned FSMap version 42 2024-03-18T16:15:40.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.065+0000 7f04c8f83740 1 -- 172.21.15.39:0/418371554 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x55adaada7470 con 0x55adaad2b200 2024-03-18T16:15:40.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.065+0000 7f04acc00640 1 -- 172.21.15.39:0/418371554 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f04b800cbe0 con 0x55adaad2b200 2024-03-18T16:15:40.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.066+0000 7f04acc00640 1 -- 172.21.15.39:0/418371554 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 42) ==== 8370+0+0 (secure 0 0 0) 0x7f04b8019720 con 0x55adaad2b200 2024-03-18T16:15:40.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.066+0000 7f04c8f83740 10 client.26758 fetch_fsmap finished waiting for FSMap version 42 2024-03-18T16:15:40.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.066+0000 7f04c8f83740 10 client.26758 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:15:40.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.066+0000 7f04c8f83740 20 client.26758 populate_metadata read hostname 'smithi039' 2024-03-18T16:15:40.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.066+0000 7f04c8f83740 1 --2- 172.21.15.39:0/418371554 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55adaad30370 0x55adaada9c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:15:40.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.066+0000 7f04c8f83740 4 client.26758 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T16:15:40.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.066+0000 7f04c8f83740 1 -- 172.21.15.39:0/418371554 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55adaac9c730 con 0x55adaad30370 2024-03-18T16:15:40.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.066+0000 7f04bf400640 1 --2- 172.21.15.39:0/418371554 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55adaad30370 0x55adaada9c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:15:40.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.067+0000 7f04bf400640 1 --2- 172.21.15.39:0/418371554 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55adaad30370 0x55adaada9c60 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dcbd6c57bd35cc5a server_cookie=7e73c3294a567073 in_seq=0 out_seq=0 2024-03-18T16:15:40.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.067+0000 7f04acc00640 10 client.26758 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:15:40.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.067+0000 7f04acc00640 1 -- 172.21.15.39:0/418371554 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55adaac9c730 con 0x55adaad30370 2024-03-18T16:15:40.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.067+0000 7f04acc00640 10 client.26758 handle_command_reply: tid=0 2024-03-18T16:15:40.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.067+0000 7f04c8f83740 1 client.26758 shutdown 2024-03-18T16:15:40.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.067+0000 7f04c8f83740 1 -- 172.21.15.39:0/418371554 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55adaad2b200 msgr2=0x55adaad3b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:40.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.067+0000 7f04c8f83740 1 --2- 172.21.15.39:0/418371554 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55adaad2b200 0x55adaad3b5a0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f04b8019420 tx=0x7f04b800fca0 comp rx=0 tx=0).stop 2024-03-18T16:15:40.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.068+0000 7f04c8f83740 1 -- 172.21.15.39:0/418371554 shutdown_connections 2024-03-18T16:15:40.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.068+0000 7f04c8f83740 1 --2- 172.21.15.39:0/418371554 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55adaad30370 0x55adaada9c60 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:40.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.068+0000 7f04c8f83740 1 --2- 172.21.15.39:0/418371554 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55adaad2a810 0x55adaad3a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:40.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.068+0000 7f04c8f83740 1 --2- 172.21.15.39:0/418371554 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55adaad2b200 0x55adaad3b5a0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:40.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.068+0000 7f04c8f83740 1 --2- 172.21.15.39:0/418371554 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55adaad25d80 0x55adaad3a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:40.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.068+0000 7f04acc00640 0 client.26758 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:15:40.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.068+0000 7f04c8f83740 1 -- 172.21.15.39:0/418371554 >> 172.21.15.39:0/418371554 conn(0x55adaad264d0 msgr2=0x55adaad24630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:40.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.068+0000 7f04c8f83740 1 -- 172.21.15.39:0/418371554 shutdown_connections 2024-03-18T16:15:40.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.068+0000 7f04c8f83740 1 -- 172.21.15.39:0/418371554 wait complete. 2024-03-18T16:15:40.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.068+0000 7f04c8f83740 20 client.26758 trim_cache size 0 max 16384 2024-03-18T16:15:40.073 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-03-18T16:15:40.073 INFO:teuthology.orchestra.run.smithi039.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-18T16:15:40.073 INFO:teuthology.orchestra.run.smithi039.stdout: "scrubs": { 2024-03-18T16:15:40.073 INFO:teuthology.orchestra.run.smithi039.stdout: "9a14398f-20ed-4130-a602-b636170af3be": { 2024-03-18T16:15:40.073 INFO:teuthology.orchestra.run.smithi039.stdout: "path": "/", 2024-03-18T16:15:40.073 INFO:teuthology.orchestra.run.smithi039.stdout: "tag": "9a14398f-20ed-4130-a602-b636170af3be", 2024-03-18T16:15:40.073 INFO:teuthology.orchestra.run.smithi039.stdout: "options": "recursive,force" 2024-03-18T16:15:40.073 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:15:40.073 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:15:40.073 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-03-18T16:15:40.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.069+0000 7f04c6200640 1 -- 172.21.15.39:0/154441844 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f04a40a2900 msgr2=0x7f04a40a4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:40.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.069+0000 7f04c6200640 1 --2- 172.21.15.39:0/154441844 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f04a40a2900 0x7f04a40a4dc0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f04b8019000 tx=0x7f04b801c000 comp rx=0 tx=0).stop 2024-03-18T16:15:40.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.069+0000 7f04c6200640 1 -- 172.21.15.39:0/154441844 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f04c0103a20 msgr2=0x7f04c01958e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:15:40.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.069+0000 7f04c6200640 1 --2- 172.21.15.39:0/154441844 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f04c0103a20 0x7f04c01958e0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f04b0000f80 tx=0x7f04b0035d10 comp rx=0 tx=0).stop 2024-03-18T16:15:40.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.069+0000 7f04c6200640 1 -- 172.21.15.39:0/154441844 shutdown_connections 2024-03-18T16:15:40.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.069+0000 7f04c6200640 1 --2- 172.21.15.39:0/154441844 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f04a40a2900 0x7f04a40a4dc0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:40.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.069+0000 7f04c6200640 1 --2- 172.21.15.39:0/154441844 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f04c01055d0 0x7f04c019ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:40.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.069+0000 7f04c6200640 1 --2- 172.21.15.39:0/154441844 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f04c0104c10 0x7f04c0195e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:40.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.069+0000 7f04c6200640 1 --2- 172.21.15.39:0/154441844 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f04c0103a20 0x7f04c01958e0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:15:40.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.069+0000 7f04c6200640 1 -- 172.21.15.39:0/154441844 >> 172.21.15.39:0/154441844 conn(0x7f04c00fef90 msgr2=0x7f04c0077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:15:40.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.070+0000 7f04c6200640 1 -- 172.21.15.39:0/154441844 shutdown_connections 2024-03-18T16:15:40.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:15:40.070+0000 7f04c6200640 1 -- 172.21.15.39:0/154441844 wait complete. 2024-03-18T16:15:40.098 INFO:tasks.cephfs.filesystem:scrub status for tag:9a14398f-20ed-4130-a602-b636170af3be - {'path': '/', 'tag': '9a14398f-20ed-4130-a602-b636170af3be', 'options': 'recursive,force'} 2024-03-18T16:15:40.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:40 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:15:40.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:40 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:15:40.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:40 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:15:41.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:41 smithi196 ceph-mon[36307]: scrub queued for path: / 2024-03-18T16:15:41.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:41 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:41.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:41 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:15:41.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:41 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:15:41.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:41 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:15:41.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:41 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:15:41.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:41 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:41.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:41 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:41.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:41 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:41.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:41 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:41.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:41 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:15:41.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:41 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:41.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:41 smithi196 ceph-mon[36307]: pgmap v590: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 11 MiB/s wr, 2.40k op/s 2024-03-18T16:15:41.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:41 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:15:41.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:41 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:41.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:41 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:15:41.480 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:41 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:41.480 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:41 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:15:41.480 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:41 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:41.480 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:41 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:15:41.480 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:41 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:41.480 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:41 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:15:41.480 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:41 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:41.480 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:41 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:15:41.480 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:41 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:41.480 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:41 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:15:41.480 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:41 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:41.480 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:41 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:15:41.480 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:41 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:41.480 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:41 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:15:41.480 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:41 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:41.480 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:41 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:15:41.481 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:41 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:41.481 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:41 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:15:41.481 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:41 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:41.481 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:41 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:41.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:41 smithi194 ceph-mon[36200]: scrub queued for path: / 2024-03-18T16:15:41.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:41 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:41.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:41 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:15:41.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:41 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:15:41.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:41 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:15:41.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:41 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:15:41.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:41 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:41.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:41 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:41.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:41 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:41.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:41 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:41.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:41 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:15:41.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:41 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:41.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:41 smithi194 ceph-mon[36200]: pgmap v590: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 11 MiB/s wr, 2.40k op/s 2024-03-18T16:15:41.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:41 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:15:41.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:41 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:41.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:41 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:15:41.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:41 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:41.492 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:41 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:15:41.492 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:41 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:41.492 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:41 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:15:41.492 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:41 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:41.492 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:41 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:15:41.492 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:41 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:41.492 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:41 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:15:41.492 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:41 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:41.492 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:41 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:15:41.492 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:41 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:41.492 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:41 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:15:41.492 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:41 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:41.492 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:41 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:15:41.492 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:41 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:41.492 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:41 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:15:41.493 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:41 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:41.493 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:41 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:15:41.493 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:41 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:41.493 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:41 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:41.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:41 smithi039 ceph-mon[31721]: scrub queued for path: / 2024-03-18T16:15:41.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:41 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:41.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:41 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:15:41.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:41 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:15:41.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:41 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:15:41.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:41 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:15:41.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:41 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:41.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:41 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:41.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:41 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:41.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:41 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:41.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:41 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:15:41.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:41 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:41.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:41 smithi039 ceph-mon[31721]: pgmap v590: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 11 MiB/s wr, 2.40k op/s 2024-03-18T16:15:41.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:41 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:15:41.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:41 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:41.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:41 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:15:41.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:41 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:41.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:41 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:15:41.589 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:41 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:41.589 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:41 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:15:41.589 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:41 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:41.589 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:41 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:15:41.589 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:41 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:41.589 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:41 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:15:41.589 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:41 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:41.589 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:41 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:15:41.589 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:41 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:41.589 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:41 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:15:41.589 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:41 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:41.589 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:41 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:15:41.589 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:41 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:41.589 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:41 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:15:41.589 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:41 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:41.590 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:41 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:15:41.590 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:41 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:41.590 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:41 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:15:43.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:43 smithi196 ceph-mon[36307]: pgmap v591: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 13 MiB/s wr, 2.83k op/s 2024-03-18T16:15:43.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:43 smithi194 ceph-mon[36200]: pgmap v591: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 13 MiB/s wr, 2.83k op/s 2024-03-18T16:15:43.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:43 smithi039 ceph-mon[31721]: pgmap v591: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 13 MiB/s wr, 2.83k op/s 2024-03-18T16:15:45.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:45 smithi194 ceph-mon[36200]: pgmap v592: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 10 MiB/s wr, 2.27k op/s 2024-03-18T16:15:45.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:45 smithi194 ceph-mon[36200]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T16:15:45.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:45 smithi039 ceph-mon[31721]: pgmap v592: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 10 MiB/s wr, 2.27k op/s 2024-03-18T16:15:45.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:45 smithi039 ceph-mon[31721]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T16:15:45.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:45 smithi196 ceph-mon[36307]: pgmap v592: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 10 MiB/s wr, 2.27k op/s 2024-03-18T16:15:45.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:45 smithi196 ceph-mon[36307]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T16:15:47.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:47 smithi194 ceph-mon[36200]: pgmap v593: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.0 MiB/s wr, 2.03k op/s 2024-03-18T16:15:47.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:47 smithi039 ceph-mon[31721]: pgmap v593: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.0 MiB/s wr, 2.03k op/s 2024-03-18T16:15:47.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:47 smithi196 ceph-mon[36307]: pgmap v593: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 9.0 MiB/s wr, 2.03k op/s 2024-03-18T16:15:49.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:49 smithi194 ceph-mon[36200]: pgmap v594: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 5.5 MiB/s wr, 1.24k op/s 2024-03-18T16:15:49.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:49 smithi039 ceph-mon[31721]: pgmap v594: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 5.5 MiB/s wr, 1.24k op/s 2024-03-18T16:15:49.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:49 smithi196 ceph-mon[36307]: pgmap v594: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 KiB/s rd, 5.5 MiB/s wr, 1.24k op/s 2024-03-18T16:15:51.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:51 smithi039 ceph-mon[31721]: pgmap v595: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 3.6 MiB/s wr, 829 op/s 2024-03-18T16:15:51.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:51 smithi196 ceph-mon[36307]: pgmap v595: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 3.6 MiB/s wr, 829 op/s 2024-03-18T16:15:51.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:51 smithi194 ceph-mon[36200]: pgmap v595: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 3.6 MiB/s wr, 829 op/s 2024-03-18T16:15:53.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:53 smithi039 ceph-mon[31721]: pgmap v596: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 3.3 MiB/s wr, 776 op/s 2024-03-18T16:15:53.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:53 smithi039 ceph-mon[31721]: scrub summary: idle 2024-03-18T16:15:53.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:53 smithi039 ceph-mon[31721]: scrub summary: idle 2024-03-18T16:15:53.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:53 smithi039 ceph-mon[31721]: scrub summary: idle 2024-03-18T16:15:53.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:53 smithi196 ceph-mon[36307]: pgmap v596: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 3.3 MiB/s wr, 776 op/s 2024-03-18T16:15:53.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:53 smithi196 ceph-mon[36307]: scrub summary: idle 2024-03-18T16:15:53.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:53 smithi196 ceph-mon[36307]: scrub summary: idle 2024-03-18T16:15:53.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:53 smithi196 ceph-mon[36307]: scrub summary: idle 2024-03-18T16:15:53.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:53 smithi194 ceph-mon[36200]: pgmap v596: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 3.3 MiB/s wr, 776 op/s 2024-03-18T16:15:53.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:53 smithi194 ceph-mon[36200]: scrub summary: idle 2024-03-18T16:15:53.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:53 smithi194 ceph-mon[36200]: scrub summary: idle 2024-03-18T16:15:53.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:53 smithi194 ceph-mon[36200]: scrub summary: idle 2024-03-18T16:15:54.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:54 smithi039 ceph-mon[31721]: scrub summary: idle 2024-03-18T16:15:54.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:54 smithi039 ceph-mon[31721]: scrub summary: idle 2024-03-18T16:15:54.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:54 smithi196 ceph-mon[36307]: scrub summary: idle 2024-03-18T16:15:54.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:54 smithi196 ceph-mon[36307]: scrub summary: idle 2024-03-18T16:15:54.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:54 smithi194 ceph-mon[36200]: scrub summary: idle 2024-03-18T16:15:54.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:54 smithi194 ceph-mon[36200]: scrub summary: idle 2024-03-18T16:15:55.589 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:55 smithi039 ceph-mon[31721]: pgmap v597: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.3 MiB/s wr, 335 op/s 2024-03-18T16:15:55.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:55 smithi196 ceph-mon[36307]: pgmap v597: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.3 MiB/s wr, 335 op/s 2024-03-18T16:15:55.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:55 smithi194 ceph-mon[36200]: pgmap v597: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.3 MiB/s wr, 335 op/s 2024-03-18T16:15:57.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:57 smithi039 ceph-mon[31721]: pgmap v598: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 1.5 MiB/s wr, 399 op/s 2024-03-18T16:15:57.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:57 smithi196 ceph-mon[36307]: pgmap v598: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 1.5 MiB/s wr, 399 op/s 2024-03-18T16:15:57.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:57 smithi194 ceph-mon[36200]: pgmap v598: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 1.5 MiB/s wr, 399 op/s 2024-03-18T16:15:59.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:15:59 smithi039 ceph-mon[31721]: pgmap v599: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.2 MiB/s wr, 328 op/s 2024-03-18T16:15:59.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:15:59 smithi196 ceph-mon[36307]: pgmap v599: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.2 MiB/s wr, 328 op/s 2024-03-18T16:15:59.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:15:59 smithi194 ceph-mon[36200]: pgmap v599: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.2 MiB/s wr, 328 op/s 2024-03-18T16:16:00.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:00 smithi039 ceph-mon[31721]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T16:16:00.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:00 smithi196 ceph-mon[36307]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T16:16:00.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:00 smithi194 ceph-mon[36200]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T16:16:01.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:01 smithi039 ceph-mon[31721]: pgmap v600: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.2 MiB/s wr, 326 op/s 2024-03-18T16:16:01.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:01 smithi196 ceph-mon[36307]: pgmap v600: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.2 MiB/s wr, 326 op/s 2024-03-18T16:16:01.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:01 smithi194 ceph-mon[36200]: pgmap v600: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.2 MiB/s wr, 326 op/s 2024-03-18T16:16:03.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:03 smithi039 ceph-mon[31721]: pgmap v601: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.5 MiB/s wr, 405 op/s 2024-03-18T16:16:03.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:03 smithi196 ceph-mon[36307]: pgmap v601: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.5 MiB/s wr, 405 op/s 2024-03-18T16:16:03.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:03 smithi194 ceph-mon[36200]: pgmap v601: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.5 MiB/s wr, 405 op/s 2024-03-18T16:16:05.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:05 smithi039 ceph-mon[31721]: pgmap v602: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.3 MiB/s wr, 333 op/s 2024-03-18T16:16:05.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:05 smithi196 ceph-mon[36307]: pgmap v602: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.3 MiB/s wr, 333 op/s 2024-03-18T16:16:05.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:05 smithi194 ceph-mon[36200]: pgmap v602: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.3 MiB/s wr, 333 op/s 2024-03-18T16:16:07.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:07 smithi039 ceph-mon[31721]: pgmap v603: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 1.5 MiB/s wr, 401 op/s 2024-03-18T16:16:07.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:07 smithi196 ceph-mon[36307]: pgmap v603: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 1.5 MiB/s wr, 401 op/s 2024-03-18T16:16:07.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:07 smithi194 ceph-mon[36200]: pgmap v603: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 1.5 MiB/s wr, 401 op/s 2024-03-18T16:16:09.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:09 smithi039 ceph-mon[31721]: pgmap v604: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.3 MiB/s wr, 335 op/s 2024-03-18T16:16:09.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:09 smithi196 ceph-mon[36307]: pgmap v604: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.3 MiB/s wr, 335 op/s 2024-03-18T16:16:09.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:09 smithi194 ceph-mon[36200]: pgmap v604: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.3 MiB/s wr, 335 op/s 2024-03-18T16:16:10.099 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T16:16:10.099 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T16:16:10.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.785+0000 7fb121600640 1 -- 172.21.15.39:0/4197927126 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb11402cc80 con 0x7fb124075820 2024-03-18T16:16:10.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.785+0000 7fb129600640 1 -- 172.21.15.39:0/4197927126 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb124075820 msgr2=0x7fb124073c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:10.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.785+0000 7fb129600640 1 --2- 172.21.15.39:0/4197927126 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb124075820 0x7fb124073c70 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fb114009930 tx=0x7fb11402bc30 comp rx=0 tx=0).stop 2024-03-18T16:16:10.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.785+0000 7fb129600640 1 -- 172.21.15.39:0/4197927126 shutdown_connections 2024-03-18T16:16:10.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.785+0000 7fb129600640 1 --2- 172.21.15.39:0/4197927126 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb12410ddc0 0x7fb1241101b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:10.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.785+0000 7fb129600640 1 --2- 172.21.15.39:0/4197927126 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb124074240 0x7fb1240746c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:10.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.785+0000 7fb129600640 1 --2- 172.21.15.39:0/4197927126 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb124075820 0x7fb124073c70 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:10.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.785+0000 7fb129600640 1 -- 172.21.15.39:0/4197927126 >> 172.21.15.39:0/4197927126 conn(0x7fb1240fcd80 msgr2=0x7fb1240ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:10.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.785+0000 7fb129600640 1 -- 172.21.15.39:0/4197927126 shutdown_connections 2024-03-18T16:16:10.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.785+0000 7fb129600640 1 -- 172.21.15.39:0/4197927126 wait complete. 2024-03-18T16:16:10.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.786+0000 7fb129600640 1 Processor -- start 2024-03-18T16:16:10.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.786+0000 7fb129600640 1 -- start start 2024-03-18T16:16:10.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.787+0000 7fb129600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb124074240 0x7fb12410b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:10.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.787+0000 7fb129600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb124075820 0x7fb12410baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:10.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.787+0000 7fb129600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb12410ddc0 0x7fb12410cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:10.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.787+0000 7fb129600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fb124105bc0 con 0x7fb124075820 2024-03-18T16:16:10.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.787+0000 7fb129600640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fb124105a40 con 0x7fb124074240 2024-03-18T16:16:10.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.787+0000 7fb122a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb124074240 0x7fb12410b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:10.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.787+0000 7fb122a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb124074240 0x7fb12410b560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:53636/0 (socket says 172.21.15.39:53636) 2024-03-18T16:16:10.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.787+0000 7fb122a00640 1 -- 172.21.15.39:0/3275419342 learned_addr learned my addr 172.21.15.39:0/3275419342 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:16:10.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.787+0000 7fb129600640 1 -- 172.21.15.39:0/3275419342 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fb124105d40 con 0x7fb12410ddc0 2024-03-18T16:16:10.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.788+0000 7fb123400640 1 --2- 172.21.15.39:0/3275419342 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb12410ddc0 0x7fb12410cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:10.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.788+0000 7fb122a00640 1 -- 172.21.15.39:0/3275419342 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb12410ddc0 msgr2=0x7fb12410cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:10.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.788+0000 7fb122a00640 1 --2- 172.21.15.39:0/3275419342 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb12410ddc0 0x7fb12410cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:10.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.788+0000 7fb122a00640 1 -- 172.21.15.39:0/3275419342 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb124075820 msgr2=0x7fb12410baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:10.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.788+0000 7fb122a00640 1 --2- 172.21.15.39:0/3275419342 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb124075820 0x7fb12410baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:10.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.788+0000 7fb122a00640 1 -- 172.21.15.39:0/3275419342 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb12410d150 con 0x7fb124074240 2024-03-18T16:16:10.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.788+0000 7fb122a00640 1 --2- 172.21.15.39:0/3275419342 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb124074240 0x7fb12410b560 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fb114000c00 tx=0x7fb114005d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:10.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.789+0000 7fb11b400640 1 -- 172.21.15.39:0/3275419342 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb114041070 con 0x7fb124074240 2024-03-18T16:16:10.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.789+0000 7fb11b400640 1 -- 172.21.15.39:0/3275419342 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb1140044d0 con 0x7fb124074240 2024-03-18T16:16:10.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.789+0000 7fb11b400640 1 -- 172.21.15.39:0/3275419342 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb114034440 con 0x7fb124074240 2024-03-18T16:16:10.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.789+0000 7fb129600640 1 -- 172.21.15.39:0/3275419342 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb12410d3e0 con 0x7fb124074240 2024-03-18T16:16:10.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.789+0000 7fb129600640 1 -- 172.21.15.39:0/3275419342 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb12410d880 con 0x7fb124074240 2024-03-18T16:16:10.796 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.793+0000 7fb12c41a740 1 Processor -- start 2024-03-18T16:16:10.796 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.793+0000 7fb11b400640 1 -- 172.21.15.39:0/3275419342 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fb114044070 con 0x7fb124074240 2024-03-18T16:16:10.797 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.794+0000 7fb12c41a740 1 -- start start 2024-03-18T16:16:10.797 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.794+0000 7fb12c41a740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8df394810 0x55a8df394c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:10.797 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.794+0000 7fb12c41a740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8df395200 0x55a8df399d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:10.797 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.794+0000 7fb12c41a740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a8df39a370 0x55a8df39c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:10.797 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.795+0000 7fb12c41a740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55a8df2f91f0 con 0x55a8df395200 2024-03-18T16:16:10.797 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.795+0000 7fb12c41a740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55a8df2a1a80 con 0x55a8df394810 2024-03-18T16:16:10.797 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.795+0000 7fb12c41a740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55a8df2f5290 con 0x55a8df39a370 2024-03-18T16:16:10.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.795+0000 7fb122000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a8df39a370 0x55a8df39c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:10.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.795+0000 7fb123400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8df394810 0x55a8df394c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:10.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.795+0000 7fb122000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a8df39a370 0x55a8df39c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:59576/0 (socket says 172.21.15.39:59576) 2024-03-18T16:16:10.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.795+0000 7fb123400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8df394810 0x55a8df394c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:53648/0 (socket says 172.21.15.39:53648) 2024-03-18T16:16:10.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.795+0000 7fb122000640 1 -- 172.21.15.39:0/34722774 learned_addr learned my addr 172.21.15.39:0/34722774 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:16:10.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.795+0000 7fb122a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8df395200 0x55a8df399d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:10.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.795+0000 7fb123400640 1 -- 172.21.15.39:0/34722774 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a8df39a370 msgr2=0x55a8df39c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:10.800 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.795+0000 7fb123400640 1 --2- 172.21.15.39:0/34722774 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a8df39a370 0x55a8df39c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:10.800 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.795+0000 7fb123400640 1 -- 172.21.15.39:0/34722774 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8df395200 msgr2=0x55a8df399d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:10.800 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.795+0000 7fb123400640 1 --2- 172.21.15.39:0/34722774 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8df395200 0x55a8df399d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:10.800 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.795+0000 7fb123400640 1 -- 172.21.15.39:0/34722774 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a8df0d77d0 con 0x55a8df394810 2024-03-18T16:16:10.800 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.795+0000 7fb122000640 1 --2- 172.21.15.39:0/34722774 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a8df39a370 0x55a8df39c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:16:10.800 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.796+0000 7fb11b400640 1 --2- 172.21.15.39:0/3275419342 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb1080a2900 0x7fb1080a4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:10.800 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.796+0000 7fb123400640 1 --2- 172.21.15.39:0/34722774 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8df394810 0x55a8df394c30 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fb11c009f00 tx=0x7fb11c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2487f10e1fa680d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:10.800 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.796+0000 7fb11b400640 1 -- 172.21.15.39:0/3275419342 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fb1140f0970 con 0x7fb124074240 2024-03-18T16:16:10.800 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.796+0000 7fb10f400640 1 -- 172.21.15.39:0/34722774 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb11c004d20 con 0x55a8df394810 2024-03-18T16:16:10.800 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.796+0000 7fb10f400640 1 -- 172.21.15.39:0/34722774 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb11c004030 con 0x55a8df394810 2024-03-18T16:16:10.800 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.796+0000 7fb10f400640 1 -- 172.21.15.39:0/34722774 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb11c005260 con 0x55a8df394810 2024-03-18T16:16:10.800 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.796+0000 7fb122a00640 1 --2- 172.21.15.39:0/3275419342 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb1080a2900 0x7fb1080a4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:10.800 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.797+0000 7fb12c41a740 1 -- 172.21.15.39:0/34722774 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8df394810 msgr2=0x55a8df394c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:10.800 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.797+0000 7fb12c41a740 1 --2- 172.21.15.39:0/34722774 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8df394810 0x55a8df394c30 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fb11c009f00 tx=0x7fb11c011810 comp rx=0 tx=0).stop 2024-03-18T16:16:10.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.797+0000 7fb12c41a740 1 -- 172.21.15.39:0/34722774 shutdown_connections 2024-03-18T16:16:10.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.797+0000 7fb12c41a740 1 --2- 172.21.15.39:0/34722774 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a8df39a370 0x55a8df39c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:10.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.797+0000 7fb12c41a740 1 --2- 172.21.15.39:0/34722774 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8df395200 0x55a8df399d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:10.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.797+0000 7fb12c41a740 1 --2- 172.21.15.39:0/34722774 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8df394810 0x55a8df394c30 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:10.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.797+0000 7fb12c41a740 1 -- 172.21.15.39:0/34722774 >> 172.21.15.39:0/34722774 conn(0x55a8df38fd80 msgr2=0x55a8df390180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:10.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.797+0000 7fb12c41a740 1 -- 172.21.15.39:0/34722774 shutdown_connections 2024-03-18T16:16:10.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.798+0000 7fb122a00640 1 --2- 172.21.15.39:0/3275419342 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb1080a2900 0x7fb1080a4dc0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fb1140bceb0 tx=0x7fb1140f6530 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:10.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.798+0000 7fb12c41a740 1 -- 172.21.15.39:0/34722774 wait complete. 2024-03-18T16:16:10.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.798+0000 7fb12c41a740 1 -- start start 2024-03-18T16:16:10.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.798+0000 7fb12c41a740 1 Processor -- start 2024-03-18T16:16:10.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.799+0000 7fb12c41a740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a8df394810 0x55a8df39ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:10.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.799+0000 7fb12c41a740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8df395200 0x55a8df39f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:10.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.799+0000 7fb12c41a740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8df39a370 0x55a8df3a6310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:10.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.799+0000 7fb12c41a740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55a8df39d320 con 0x55a8df395200 2024-03-18T16:16:10.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.799+0000 7fb12c41a740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55a8df2b4ed0 con 0x55a8df39a370 2024-03-18T16:16:10.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.799+0000 7fb12c41a740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55a8df39d4a0 con 0x55a8df394810 2024-03-18T16:16:10.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.799+0000 7fb12c41a740 10 client.0 _ll_register_callbacks cb 0x55a8df2f0220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:16:10.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.800+0000 7fb123400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a8df394810 0x55a8df39ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:10.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.800+0000 7fb123400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a8df394810 0x55a8df39ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:59578/0 (socket says 172.21.15.39:59578) 2024-03-18T16:16:10.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.800+0000 7fb123400640 1 -- 172.21.15.39:0/2271498064 learned_addr learned my addr 172.21.15.39:0/2271498064 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:16:10.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.800+0000 7fb122000640 1 --2- 172.21.15.39:0/2271498064 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8df395200 0x55a8df39f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:10.804 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.800+0000 7fb123400640 1 --2- 172.21.15.39:0/2271498064 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8df39a370 0x55a8df3a6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:10.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.800+0000 7fb123400640 1 -- 172.21.15.39:0/2271498064 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8df39a370 msgr2=0x55a8df3a6310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:10.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.800+0000 7fb123400640 1 --2- 172.21.15.39:0/2271498064 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8df39a370 0x55a8df3a6310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:10.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.800+0000 7fb123400640 1 -- 172.21.15.39:0/2271498064 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8df395200 msgr2=0x55a8df39f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:10.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.801+0000 7fb123400640 1 --2- 172.21.15.39:0/2271498064 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8df395200 0x55a8df39f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:10.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.801+0000 7fb123400640 1 -- 172.21.15.39:0/2271498064 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a8df00d3f0 con 0x55a8df394810 2024-03-18T16:16:10.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.801+0000 7fb122000640 1 --2- 172.21.15.39:0/2271498064 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8df395200 0x55a8df39f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:16:10.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.801+0000 7fb123400640 1 --2- 172.21.15.39:0/2271498064 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a8df394810 0x55a8df39ed90 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fb11c011d10 tx=0x7fb11c0047d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=38e3957267438840 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:10.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.801+0000 7fb103e00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:16:10.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.801+0000 7fb103e00640 1 -- 172.21.15.39:0/2271498064 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb11c004a40 con 0x55a8df394810 2024-03-18T16:16:10.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.802+0000 7fb103e00640 1 -- 172.21.15.39:0/2271498064 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fb11c004be0 con 0x55a8df394810 2024-03-18T16:16:10.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.802+0000 7fb12c41a740 1 -- 172.21.15.39:0/2271498064 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb11c004030 con 0x55a8df394810 2024-03-18T16:16:10.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.802+0000 7fb103e00640 1 -- 172.21.15.39:0/2271498064 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fb11c004de0 con 0x55a8df394810 2024-03-18T16:16:10.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.802+0000 7fb12c41a740 10 client.26770 fetch_fsmap learned FSMap version 44 2024-03-18T16:16:10.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.802+0000 7fb12c41a740 1 -- 172.21.15.39:0/2271498064 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x55a8defcd0f0 con 0x55a8df394810 2024-03-18T16:16:10.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.802+0000 7fb103e00640 1 -- 172.21.15.39:0/2271498064 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb11c004030 con 0x55a8df394810 2024-03-18T16:16:10.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.802+0000 7fb103e00640 1 -- 172.21.15.39:0/2271498064 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7fb11c022020 con 0x55a8df394810 2024-03-18T16:16:10.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.803+0000 7fb12c41a740 10 client.26770 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:16:10.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.803+0000 7fb12c41a740 10 client.26770 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:16:10.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.803+0000 7fb12c41a740 20 client.26770 populate_metadata read hostname 'smithi039' 2024-03-18T16:16:10.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.803+0000 7fb12c41a740 1 --2- 172.21.15.39:0/2271498064 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55a8df3aa900 0x55a8df3accf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:10.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.803+0000 7fb12c41a740 4 client.26770 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:16:10.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.803+0000 7fb12c41a740 1 -- 172.21.15.39:0/2271498064 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a8df30cbe0 con 0x55a8df3aa900 2024-03-18T16:16:10.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.803+0000 7fb122000640 1 --2- 172.21.15.39:0/2271498064 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55a8df3aa900 0x55a8df3accf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:10.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.804+0000 7fb103e00640 10 client.26770 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:16:10.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.805+0000 7fb122000640 1 --2- 172.21.15.39:0/2271498064 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55a8df3aa900 0x55a8df3accf0 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f445b2e746b5fa1e server_cookie=86182d9e22575ec3 in_seq=0 out_seq=0 2024-03-18T16:16:10.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.805+0000 7fb103e00640 1 -- 172.21.15.39:0/2271498064 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a8df30cbe0 con 0x55a8df3aa900 2024-03-18T16:16:10.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.805+0000 7fb103e00640 10 client.26770 handle_command_reply: tid=0 2024-03-18T16:16:10.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.806+0000 7fb12c41a740 1 client.26770 shutdown 2024-03-18T16:16:10.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.806+0000 7fb12c41a740 1 -- 172.21.15.39:0/2271498064 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a8df394810 msgr2=0x55a8df39ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:10.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.806+0000 7fb12c41a740 1 --2- 172.21.15.39:0/2271498064 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a8df394810 0x55a8df39ed90 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fb11c011d10 tx=0x7fb11c0047d0 comp rx=0 tx=0).stop 2024-03-18T16:16:10.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.806+0000 7fb12c41a740 1 -- 172.21.15.39:0/2271498064 shutdown_connections 2024-03-18T16:16:10.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.806+0000 7fb12c41a740 1 --2- 172.21.15.39:0/2271498064 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55a8df3aa900 0x55a8df3accf0 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:10.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.806+0000 7fb12c41a740 1 --2- 172.21.15.39:0/2271498064 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8df39a370 0x55a8df3a6310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:10.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.806+0000 7fb12c41a740 1 --2- 172.21.15.39:0/2271498064 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8df395200 0x55a8df39f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:10.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.806+0000 7fb12c41a740 1 --2- 172.21.15.39:0/2271498064 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a8df394810 0x55a8df39ed90 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:10.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.806+0000 7fb103e00640 0 client.26770 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:16:10.809 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.806+0000 7fb12c41a740 1 -- 172.21.15.39:0/2271498064 >> 172.21.15.39:0/2271498064 conn(0x55a8df38fd80 msgr2=0x55a8df38eec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:10.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.807+0000 7fb12c41a740 1 -- 172.21.15.39:0/2271498064 shutdown_connections 2024-03-18T16:16:10.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.807+0000 7fb12c41a740 1 -- 172.21.15.39:0/2271498064 wait complete. 2024-03-18T16:16:10.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.807+0000 7fb12c41a740 20 client.26770 trim_cache size 0 max 16384 2024-03-18T16:16:10.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.832+0000 7fb12c41a740 1 Processor -- start 2024-03-18T16:16:10.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.832+0000 7fb12c41a740 1 -- start start 2024-03-18T16:16:10.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.832+0000 7fb12c41a740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8df394810 0x55a8df39e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:10.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.832+0000 7fb12c41a740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a8df395200 0x55a8df3a62a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:10.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.832+0000 7fb12c41a740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8df39a370 0x55a8df39eee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:10.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.832+0000 7fb12c41a740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fb1241b9340 con 0x55a8df394810 2024-03-18T16:16:10.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.832+0000 7fb12c41a740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fb1241bae80 con 0x55a8df39a370 2024-03-18T16:16:10.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.832+0000 7fb12c41a740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fb1241b9a10 con 0x55a8df395200 2024-03-18T16:16:10.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.833+0000 7fb122000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a8df395200 0x55a8df3a62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:10.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.833+0000 7fb123400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8df394810 0x55a8df39e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:10.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.833+0000 7fb122000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a8df395200 0x55a8df3a62a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:59580/0 (socket says 172.21.15.39:59580) 2024-03-18T16:16:10.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.833+0000 7fb122000640 1 -- 172.21.15.39:0/2678552575 learned_addr learned my addr 172.21.15.39:0/2678552575 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:16:10.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.833+0000 7fb123400640 1 --2- 172.21.15.39:0/2678552575 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8df39a370 0x55a8df39eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:10.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.833+0000 7fb122000640 1 -- 172.21.15.39:0/2678552575 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8df39a370 msgr2=0x55a8df39eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:10.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.833+0000 7fb122000640 1 --2- 172.21.15.39:0/2678552575 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8df39a370 0x55a8df39eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:10.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.833+0000 7fb122000640 1 -- 172.21.15.39:0/2678552575 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8df394810 msgr2=0x55a8df39e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:10.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.833+0000 7fb122000640 1 --2- 172.21.15.39:0/2678552575 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8df394810 0x55a8df39e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:10.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.833+0000 7fb122000640 1 -- 172.21.15.39:0/2678552575 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a8df394450 con 0x55a8df395200 2024-03-18T16:16:10.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.833+0000 7fb123400640 1 --2- 172.21.15.39:0/2678552575 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8df394810 0x55a8df39e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:16:10.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.834+0000 7fb122000640 1 --2- 172.21.15.39:0/2678552575 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a8df395200 0x55a8df3a62a0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fb110005fd0 tx=0x7fb1100058f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=43dbef12a8eadf0d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:10.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.834+0000 7fb10ea00640 1 -- 172.21.15.39:0/2678552575 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb110003e20 con 0x55a8df395200 2024-03-18T16:16:10.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.834+0000 7fb10ea00640 1 -- 172.21.15.39:0/2678552575 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb110005500 con 0x55a8df395200 2024-03-18T16:16:10.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.834+0000 7fb10ea00640 1 -- 172.21.15.39:0/2678552575 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb11001d2b0 con 0x55a8df395200 2024-03-18T16:16:10.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.835+0000 7fb12c41a740 1 -- 172.21.15.39:0/2678552575 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a8df395200 msgr2=0x55a8df3a62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:10.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.835+0000 7fb12c41a740 1 --2- 172.21.15.39:0/2678552575 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a8df395200 0x55a8df3a62a0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fb110005fd0 tx=0x7fb1100058f0 comp rx=0 tx=0).stop 2024-03-18T16:16:10.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.835+0000 7fb12c41a740 1 -- 172.21.15.39:0/2678552575 shutdown_connections 2024-03-18T16:16:10.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.835+0000 7fb12c41a740 1 --2- 172.21.15.39:0/2678552575 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8df39a370 0x55a8df39eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:10.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.835+0000 7fb12c41a740 1 --2- 172.21.15.39:0/2678552575 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a8df395200 0x55a8df3a62a0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:10.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.835+0000 7fb12c41a740 1 --2- 172.21.15.39:0/2678552575 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8df394810 0x55a8df39e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:10.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.835+0000 7fb12c41a740 1 -- 172.21.15.39:0/2678552575 >> 172.21.15.39:0/2678552575 conn(0x55a8df38fd80 msgr2=0x55a8df390160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:10.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.835+0000 7fb12c41a740 1 -- 172.21.15.39:0/2678552575 shutdown_connections 2024-03-18T16:16:10.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.835+0000 7fb12c41a740 1 -- 172.21.15.39:0/2678552575 wait complete. 2024-03-18T16:16:10.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.836+0000 7fb12c41a740 1 -- start start 2024-03-18T16:16:10.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.836+0000 7fb12c41a740 1 Processor -- start 2024-03-18T16:16:10.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.836+0000 7fb12c41a740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a8df394810 0x55a8df3a37a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:10.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.836+0000 7fb12c41a740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8df395200 0x55a8df3a3ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:10.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.836+0000 7fb12c41a740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8df39a370 0x55a8df3a4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:10.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.836+0000 7fb12c41a740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fb1241ba7b0 con 0x55a8df39a370 2024-03-18T16:16:10.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.836+0000 7fb12c41a740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fb1241ba0e0 con 0x55a8df395200 2024-03-18T16:16:10.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.836+0000 7fb12c41a740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fb1241b7320 con 0x55a8df394810 2024-03-18T16:16:10.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.837+0000 7fb12c41a740 10 client.0 _ll_register_callbacks cb 0x7fb1140f6400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:16:10.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.837+0000 7fb123400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a8df394810 0x55a8df3a37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:10.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.837+0000 7fb123400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a8df394810 0x55a8df3a37a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:59584/0 (socket says 172.21.15.39:59584) 2024-03-18T16:16:10.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.837+0000 7fb123400640 1 -- 172.21.15.39:0/2539173794 learned_addr learned my addr 172.21.15.39:0/2539173794 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:16:10.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.837+0000 7fb123400640 1 --2- 172.21.15.39:0/2539173794 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8df39a370 0x55a8df3a4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:10.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.837+0000 7fb122000640 1 --2- 172.21.15.39:0/2539173794 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8df395200 0x55a8df3a3ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:10.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.838+0000 7fb123400640 1 -- 172.21.15.39:0/2539173794 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8df395200 msgr2=0x55a8df3a3ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:10.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.838+0000 7fb123400640 1 --2- 172.21.15.39:0/2539173794 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8df395200 0x55a8df3a3ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:10.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.838+0000 7fb123400640 1 -- 172.21.15.39:0/2539173794 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8df39a370 msgr2=0x55a8df3a4cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:10.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.838+0000 7fb123400640 1 --2- 172.21.15.39:0/2539173794 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8df39a370 0x55a8df3a4cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:10.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.838+0000 7fb123400640 1 -- 172.21.15.39:0/2539173794 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a8df38a370 con 0x55a8df394810 2024-03-18T16:16:10.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.838+0000 7fb123400640 1 --2- 172.21.15.39:0/2539173794 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a8df394810 0x55a8df3a37a0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fb11c003bf0 tx=0x7fb11c003b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=711b7ab641be9885 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:10.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.838+0000 7fb103e00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:16:10.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.838+0000 7fb103e00640 1 -- 172.21.15.39:0/2539173794 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb11c0044f0 con 0x55a8df394810 2024-03-18T16:16:10.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.839+0000 7fb103e00640 1 -- 172.21.15.39:0/2539173794 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fb11c004f80 con 0x55a8df394810 2024-03-18T16:16:10.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.839+0000 7fb12c41a740 1 -- 172.21.15.39:0/2539173794 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb110005500 con 0x55a8df394810 2024-03-18T16:16:10.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.839+0000 7fb103e00640 1 -- 172.21.15.39:0/2539173794 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fb11c04b050 con 0x55a8df394810 2024-03-18T16:16:10.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.839+0000 7fb12c41a740 10 client.26776 fetch_fsmap learned FSMap version 44 2024-03-18T16:16:10.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.839+0000 7fb12c41a740 1 -- 172.21.15.39:0/2539173794 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x55a8df3a6420 con 0x55a8df394810 2024-03-18T16:16:10.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.839+0000 7fb103e00640 1 -- 172.21.15.39:0/2539173794 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb110005500 con 0x55a8df394810 2024-03-18T16:16:10.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.840+0000 7fb103e00640 1 -- 172.21.15.39:0/2539173794 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7fb11c038030 con 0x55a8df394810 2024-03-18T16:16:10.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.840+0000 7fb12c41a740 10 client.26776 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:16:10.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.840+0000 7fb12c41a740 10 client.26776 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:16:10.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.840+0000 7fb12c41a740 20 client.26776 populate_metadata read hostname 'smithi039' 2024-03-18T16:16:10.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.840+0000 7fb12c41a740 1 --2- 172.21.15.39:0/2539173794 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55a8df4221e0 0x55a8df4245d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:10.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.840+0000 7fb12c41a740 4 client.26776 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T16:16:10.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.840+0000 7fb12c41a740 1 -- 172.21.15.39:0/2539173794 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a8df306730 con 0x55a8df4221e0 2024-03-18T16:16:10.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.840+0000 7fb122000640 1 --2- 172.21.15.39:0/2539173794 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55a8df4221e0 0x55a8df4245d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:10.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.841+0000 7fb122000640 1 --2- 172.21.15.39:0/2539173794 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55a8df4221e0 0x55a8df4245d0 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=34c0fb9b0089c537 server_cookie=3cec3dc2ffe56528 in_seq=0 out_seq=0 2024-03-18T16:16:10.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.841+0000 7fb103e00640 10 client.26776 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:16:10.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.841+0000 7fb103e00640 1 -- 172.21.15.39:0/2539173794 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55a8df306730 con 0x55a8df4221e0 2024-03-18T16:16:10.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.841+0000 7fb103e00640 10 client.26776 handle_command_reply: tid=0 2024-03-18T16:16:10.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.841+0000 7fb12c41a740 1 client.26776 shutdown 2024-03-18T16:16:10.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.841+0000 7fb12c41a740 1 -- 172.21.15.39:0/2539173794 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a8df394810 msgr2=0x55a8df3a37a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:10.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.841+0000 7fb12c41a740 1 --2- 172.21.15.39:0/2539173794 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a8df394810 0x55a8df3a37a0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fb11c003bf0 tx=0x7fb11c003b60 comp rx=0 tx=0).stop 2024-03-18T16:16:10.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.841+0000 7fb12c41a740 1 -- 172.21.15.39:0/2539173794 shutdown_connections 2024-03-18T16:16:10.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.841+0000 7fb12c41a740 1 --2- 172.21.15.39:0/2539173794 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55a8df4221e0 0x55a8df4245d0 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:10.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.841+0000 7fb12c41a740 1 --2- 172.21.15.39:0/2539173794 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8df395200 0x55a8df3a3ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:10.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.841+0000 7fb12c41a740 1 --2- 172.21.15.39:0/2539173794 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a8df39a370 0x55a8df3a4cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:10.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.841+0000 7fb12c41a740 1 --2- 172.21.15.39:0/2539173794 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a8df394810 0x55a8df3a37a0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:10.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.841+0000 7fb103e00640 0 client.26776 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:16:10.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.841+0000 7fb12c41a740 1 -- 172.21.15.39:0/2539173794 >> 172.21.15.39:0/2539173794 conn(0x55a8df38fd80 msgr2=0x55a8df39e9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:10.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.842+0000 7fb12c41a740 1 -- 172.21.15.39:0/2539173794 shutdown_connections 2024-03-18T16:16:10.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.842+0000 7fb12c41a740 1 -- 172.21.15.39:0/2539173794 wait complete. 2024-03-18T16:16:10.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.842+0000 7fb12c41a740 20 client.26776 trim_cache size 0 max 16384 2024-03-18T16:16:10.847 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-03-18T16:16:10.847 INFO:teuthology.orchestra.run.smithi039.stdout: "status": "no active scrubs running", 2024-03-18T16:16:10.847 INFO:teuthology.orchestra.run.smithi039.stdout: "scrubs": {} 2024-03-18T16:16:10.847 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-03-18T16:16:10.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.843+0000 7fb129600640 1 -- 172.21.15.39:0/3275419342 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb1080a2900 msgr2=0x7fb1080a4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:10.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.843+0000 7fb129600640 1 --2- 172.21.15.39:0/3275419342 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb1080a2900 0x7fb1080a4dc0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fb1140bceb0 tx=0x7fb1140f6530 comp rx=0 tx=0).stop 2024-03-18T16:16:10.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.843+0000 7fb129600640 1 -- 172.21.15.39:0/3275419342 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb124074240 msgr2=0x7fb12410b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:10.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.843+0000 7fb129600640 1 --2- 172.21.15.39:0/3275419342 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb124074240 0x7fb12410b560 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fb114000c00 tx=0x7fb114005d50 comp rx=0 tx=0).stop 2024-03-18T16:16:10.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.843+0000 7fb129600640 1 -- 172.21.15.39:0/3275419342 shutdown_connections 2024-03-18T16:16:10.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.843+0000 7fb129600640 1 --2- 172.21.15.39:0/3275419342 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb1080a2900 0x7fb1080a4dc0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:10.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.843+0000 7fb129600640 1 --2- 172.21.15.39:0/3275419342 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb12410ddc0 0x7fb12410cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:10.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.843+0000 7fb129600640 1 --2- 172.21.15.39:0/3275419342 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb124075820 0x7fb12410baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:10.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.843+0000 7fb129600640 1 --2- 172.21.15.39:0/3275419342 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb124074240 0x7fb12410b560 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:10.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.843+0000 7fb129600640 1 -- 172.21.15.39:0/3275419342 >> 172.21.15.39:0/3275419342 conn(0x7fb1240fcd80 msgr2=0x7fb124101160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:10.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.843+0000 7fb129600640 1 -- 172.21.15.39:0/3275419342 shutdown_connections 2024-03-18T16:16:10.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:10.843+0000 7fb129600640 1 -- 172.21.15.39:0/3275419342 wait complete. 2024-03-18T16:16:10.869 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-18T16:16:10.869 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:16:11.124 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.121+0000 7fa758a00640 1 -- 172.21.15.39:0/1371344778 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa7540773e0 msgr2=0x7fa754077840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:11.124 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.121+0000 7fa758a00640 1 --2- 172.21.15.39:0/1371344778 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa7540773e0 0x7fa754077840 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fa73c01a910 tx=0x7fa73c03cc70 comp rx=0 tx=0).stop 2024-03-18T16:16:11.124 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.121+0000 7fa758a00640 1 -- 172.21.15.39:0/1371344778 shutdown_connections 2024-03-18T16:16:11.124 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.122+0000 7fa758a00640 1 --2- 172.21.15.39:0/1371344778 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa754077d80 0x7fa754078620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:11.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.122+0000 7fa758a00640 1 --2- 172.21.15.39:0/1371344778 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa7540773e0 0x7fa754077840 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:11.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.122+0000 7fa758a00640 1 --2- 172.21.15.39:0/1371344778 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa7540761a0 0x7fa7540765c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:11.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.122+0000 7fa758a00640 1 -- 172.21.15.39:0/1371344778 >> 172.21.15.39:0/1371344778 conn(0x7fa7540fef90 msgr2=0x7fa7541013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:11.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.124+0000 7fa758a00640 1 -- 172.21.15.39:0/1371344778 shutdown_connections 2024-03-18T16:16:11.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.125+0000 7fa758a00640 1 -- 172.21.15.39:0/1371344778 wait complete. 2024-03-18T16:16:11.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.126+0000 7fa758a00640 1 Processor -- start 2024-03-18T16:16:11.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.126+0000 7fa758a00640 1 -- start start 2024-03-18T16:16:11.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.126+0000 7fa758a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa7540761a0 0x7fa754195810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:11.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.126+0000 7fa758a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa7540773e0 0x7fa754195d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:11.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.126+0000 7fa758a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa754077d80 0x7fa75419cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:11.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.126+0000 7fa758a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fa754105250 con 0x7fa754077d80 2024-03-18T16:16:11.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.126+0000 7fa758a00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fa7541050d0 con 0x7fa7540773e0 2024-03-18T16:16:11.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.127+0000 7fa758a00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fa7541053d0 con 0x7fa7540761a0 2024-03-18T16:16:11.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.127+0000 7fa751e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa7540761a0 0x7fa754195810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:11.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.127+0000 7fa751e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa7540761a0 0x7fa754195810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:59602/0 (socket says 172.21.15.39:59602) 2024-03-18T16:16:11.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.127+0000 7fa751e00640 1 -- 172.21.15.39:0/4291310410 learned_addr learned my addr 172.21.15.39:0/4291310410 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:16:11.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.127+0000 7fa752800640 1 --2- 172.21.15.39:0/4291310410 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa754077d80 0x7fa75419cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:11.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.127+0000 7fa751e00640 1 -- 172.21.15.39:0/4291310410 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa7540773e0 msgr2=0x7fa754195d50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-18T16:16:11.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.127+0000 7fa751e00640 1 --2- 172.21.15.39:0/4291310410 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa7540773e0 0x7fa754195d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:11.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.127+0000 7fa751e00640 1 -- 172.21.15.39:0/4291310410 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa754077d80 msgr2=0x7fa75419cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:11.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.127+0000 7fa751e00640 1 --2- 172.21.15.39:0/4291310410 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa754077d80 0x7fa75419cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:11.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.127+0000 7fa751e00640 1 -- 172.21.15.39:0/4291310410 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa75419d430 con 0x7fa7540761a0 2024-03-18T16:16:11.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.128+0000 7fa751e00640 1 --2- 172.21.15.39:0/4291310410 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa7540761a0 0x7fa754195810 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fa74400cce0 tx=0x7fa74400a620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:11.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.129+0000 7fa74aa00640 1 -- 172.21.15.39:0/4291310410 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa74400f650 con 0x7fa7540761a0 2024-03-18T16:16:11.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.129+0000 7fa74aa00640 1 -- 172.21.15.39:0/4291310410 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa74400fcd0 con 0x7fa7540761a0 2024-03-18T16:16:11.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.129+0000 7fa74aa00640 1 -- 172.21.15.39:0/4291310410 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa744018c70 con 0x7fa7540761a0 2024-03-18T16:16:11.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.129+0000 7fa758a00640 1 -- 172.21.15.39:0/4291310410 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa75419d720 con 0x7fa7540761a0 2024-03-18T16:16:11.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.129+0000 7fa758a00640 1 -- 172.21.15.39:0/4291310410 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa75419dc20 con 0x7fa7540761a0 2024-03-18T16:16:11.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.130+0000 7fa758a00640 1 -- 172.21.15.39:0/4291310410 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa75418f0f0 con 0x7fa7540761a0 2024-03-18T16:16:11.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.135+0000 7fa74aa00640 1 -- 172.21.15.39:0/4291310410 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fa74400f7f0 con 0x7fa7540761a0 2024-03-18T16:16:11.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.137+0000 7fa74aa00640 1 --2- 172.21.15.39:0/4291310410 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa7380a2950 0x7fa7380a4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:11.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.138+0000 7fa74aa00640 1 -- 172.21.15.39:0/4291310410 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fa7440cf340 con 0x7fa7540761a0 2024-03-18T16:16:11.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.138+0000 7fa74be00640 1 --2- 172.21.15.39:0/4291310410 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa7380a2950 0x7fa7380a4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:11.141 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.139+0000 7fa74be00640 1 --2- 172.21.15.39:0/4291310410 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa7380a2950 0x7fa7380a4e10 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fa73c0047e0 tx=0x7fa73c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:11.142 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.140+0000 7fa74aa00640 1 -- 172.21.15.39:0/4291310410 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa744014030 con 0x7fa7540761a0 2024-03-18T16:16:11.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.528+0000 7fa758a00640 1 -- 172.21.15.39:0/4291310410 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa75402d1b0 con 0x7fa7540761a0 2024-03-18T16:16:11.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.529+0000 7fa74aa00640 1 -- 172.21.15.39:0/4291310410 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) ==== 94+0+11932 (secure 0 0 0) 0x7fa74409c1f0 con 0x7fa7540761a0 2024-03-18T16:16:11.533 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:16:11.533 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":44,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2520226608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2520226608},{"type":"v1","addr":"172.21.15.39:6835","nonce":2520226608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/719341763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":719341763},{"type":"v1","addr":"172.21.15.39:6837","nonce":719341763}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":44,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T16:06:44.180425+0000","modified":"2024-03-18T16:15:58.268360+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24575,"mds_1":24556,"mds_2":24535,"mds_3":24524,"mds_4":14463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14445":{"gid":14445,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.39:6839/656513813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":656513813},{"type":"v1","addr":"172.21.15.39:6839","nonce":656513813}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14463":{"gid":14463,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":13,"addr":"172.21.15.39:6841/3515326042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3515326042},{"type":"v1","addr":"172.21.15.39:6841","nonce":3515326042}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6833/931350738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":931350738},{"type":"v1","addr":"172.21.15.194:6833","nonce":931350738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6833/3874209621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":3874209621},{"type":"v1","addr":"172.21.15.196:6833","nonce":3874209621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6835/188385470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":188385470},{"type":"v1","addr":"172.21.15.194:6835","nonce":188385470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6835/3949984309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":3949984309},{"type":"v1","addr":"172.21.15.196:6835","nonce":3949984309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":15,"addr":"172.21.15.194:6837/1301464246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":1301464246},{"type":"v1","addr":"172.21.15.194:6837","nonce":1301464246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":16,"addr":"172.21.15.196:6837/895247251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":895247251},{"type":"v1","addr":"172.21.15.196:6837","nonce":895247251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.196:6839/2001427268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6838","nonce":2001427268},{"type":"v1","addr":"172.21.15.196:6839","nonce":2001427268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24575":{"gid":24575,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.194:6839/274911158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":274911158},{"type":"v1","addr":"172.21.15.194:6839","nonce":274911158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24575,"qdb_cluster":[14463,24524,24575,24556,24535]},"id":1}]} 2024-03-18T16:16:11.533 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 44 2024-03-18T16:16:11.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.533+0000 7fa758a00640 1 -- 172.21.15.39:0/4291310410 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa7380a2950 msgr2=0x7fa7380a4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:11.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.533+0000 7fa758a00640 1 --2- 172.21.15.39:0/4291310410 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa7380a2950 0x7fa7380a4e10 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fa73c0047e0 tx=0x7fa73c005fd0 comp rx=0 tx=0).stop 2024-03-18T16:16:11.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.533+0000 7fa758a00640 1 -- 172.21.15.39:0/4291310410 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa7540761a0 msgr2=0x7fa754195810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:11.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.533+0000 7fa758a00640 1 --2- 172.21.15.39:0/4291310410 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa7540761a0 0x7fa754195810 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fa74400cce0 tx=0x7fa74400a620 comp rx=0 tx=0).stop 2024-03-18T16:16:11.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.534+0000 7fa758a00640 1 -- 172.21.15.39:0/4291310410 shutdown_connections 2024-03-18T16:16:11.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.534+0000 7fa758a00640 1 --2- 172.21.15.39:0/4291310410 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa7540773e0 0x7fa754195d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:11.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.534+0000 7fa758a00640 1 --2- 172.21.15.39:0/4291310410 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa7380a2950 0x7fa7380a4e10 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:11.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.534+0000 7fa758a00640 1 --2- 172.21.15.39:0/4291310410 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa754077d80 0x7fa75419cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:11.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.534+0000 7fa758a00640 1 --2- 172.21.15.39:0/4291310410 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa7540761a0 0x7fa754195810 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:11.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.534+0000 7fa758a00640 1 -- 172.21.15.39:0/4291310410 >> 172.21.15.39:0/4291310410 conn(0x7fa7540fef90 msgr2=0x7fa7541005c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:11.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.534+0000 7fa758a00640 1 -- 172.21.15.39:0/4291310410 shutdown_connections 2024-03-18T16:16:11.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.534+0000 7fa758a00640 1 -- 172.21.15.39:0/4291310410 wait complete. 2024-03-18T16:16:11.560 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-18T16:16:11.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:11 smithi039 ceph-mon[31721]: pgmap v605: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 1.3 MiB/s wr, 353 op/s 2024-03-18T16:16:11.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:11 smithi196 ceph-mon[36307]: pgmap v605: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 1.3 MiB/s wr, 353 op/s 2024-03-18T16:16:11.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:11 smithi194 ceph-mon[36200]: pgmap v605: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 1.3 MiB/s wr, 353 op/s 2024-03-18T16:16:11.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.817+0000 7f0147e00640 1 -- 172.21.15.39:0/1370703417 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f01400757c0 msgr2=0x7f0140073c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:11.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.817+0000 7f0147e00640 1 --2- 172.21.15.39:0/1370703417 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f01400757c0 0x7f0140073c10 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f0134009900 tx=0x7f013402bcf0 comp rx=0 tx=0).stop 2024-03-18T16:16:11.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.817+0000 7f0147e00640 1 -- 172.21.15.39:0/1370703417 shutdown_connections 2024-03-18T16:16:11.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.817+0000 7f0147e00640 1 --2- 172.21.15.39:0/1370703417 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0140109670 0x7f014010bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:11.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.817+0000 7f0147e00640 1 --2- 172.21.15.39:0/1370703417 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f01400741e0 0x7f0140074640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:11.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.817+0000 7f0147e00640 1 --2- 172.21.15.39:0/1370703417 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f01400757c0 0x7f0140073c10 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:11.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.817+0000 7f0147e00640 1 -- 172.21.15.39:0/1370703417 >> 172.21.15.39:0/1370703417 conn(0x7f01400fcd70 msgr2=0x7f01400ff190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:11.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.818+0000 7f0147e00640 1 -- 172.21.15.39:0/1370703417 shutdown_connections 2024-03-18T16:16:11.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.818+0000 7f0147e00640 1 -- 172.21.15.39:0/1370703417 wait complete. 2024-03-18T16:16:11.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.819+0000 7f0147e00640 1 Processor -- start 2024-03-18T16:16:11.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.819+0000 7f0147e00640 1 -- start start 2024-03-18T16:16:11.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.819+0000 7f0147e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f01400741e0 0x7f0140199d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:11.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.819+0000 7f0147e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f01400757c0 0x7f014019a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:11.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.819+0000 7f0147e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0140109670 0x7f01401a1300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:11.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.819+0000 7f0147e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f014010f720 con 0x7f0140109670 2024-03-18T16:16:11.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.819+0000 7f0147e00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f014010f5a0 con 0x7f01400757c0 2024-03-18T16:16:11.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.819+0000 7f0147e00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f014010f8a0 con 0x7f01400741e0 2024-03-18T16:16:11.823 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.819+0000 7f0145400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f01400741e0 0x7f0140199d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:11.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.820+0000 7f0145400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f01400741e0 0x7f0140199d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:59620/0 (socket says 172.21.15.39:59620) 2024-03-18T16:16:11.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.820+0000 7f0145400640 1 -- 172.21.15.39:0/2701730666 learned_addr learned my addr 172.21.15.39:0/2701730666 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:16:11.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.820+0000 7f0144a00640 1 --2- 172.21.15.39:0/2701730666 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f01400757c0 0x7f014019a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:11.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.820+0000 7f0145400640 1 -- 172.21.15.39:0/2701730666 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f01400757c0 msgr2=0x7f014019a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:11.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.820+0000 7f0145400640 1 --2- 172.21.15.39:0/2701730666 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f01400757c0 0x7f014019a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:11.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.820+0000 7f0145400640 1 -- 172.21.15.39:0/2701730666 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0140109670 msgr2=0x7f01401a1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:11.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.820+0000 7f0145400640 1 --2- 172.21.15.39:0/2701730666 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0140109670 0x7f01401a1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:11.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.820+0000 7f0145400640 1 -- 172.21.15.39:0/2701730666 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f014006b330 con 0x7f01400741e0 2024-03-18T16:16:11.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.820+0000 7f0144a00640 1 --2- 172.21.15.39:0/2701730666 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f01400757c0 0x7f014019a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:16:11.824 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.820+0000 7f0145400640 1 --2- 172.21.15.39:0/2701730666 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f01400741e0 0x7f0140199d50 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f0134000c00 tx=0x7f013402c690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:11.825 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.821+0000 7f013e000640 1 -- 172.21.15.39:0/2701730666 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0134043020 con 0x7f01400741e0 2024-03-18T16:16:11.825 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.821+0000 7f013e000640 1 -- 172.21.15.39:0/2701730666 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f01340054e0 con 0x7f01400741e0 2024-03-18T16:16:11.825 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.821+0000 7f0147e00640 1 -- 172.21.15.39:0/2701730666 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f014006b5c0 con 0x7f01400741e0 2024-03-18T16:16:11.825 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.821+0000 7f013e000640 1 -- 172.21.15.39:0/2701730666 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0134003d10 con 0x7f01400741e0 2024-03-18T16:16:11.825 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.821+0000 7f0147e00640 1 -- 172.21.15.39:0/2701730666 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f014006bac0 con 0x7f01400741e0 2024-03-18T16:16:11.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.824+0000 7f013e000640 1 -- 172.21.15.39:0/2701730666 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f0134046020 con 0x7f01400741e0 2024-03-18T16:16:11.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.826+0000 7f014aba0740 1 Processor -- start 2024-03-18T16:16:11.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.826+0000 7f014aba0740 1 -- start start 2024-03-18T16:16:11.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.827+0000 7f014aba0740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5576b72fd8c0 0x5576b72fdce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:11.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.827+0000 7f014aba0740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5576b72fe2b0 0x5576b7302db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:11.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.827+0000 7f014aba0740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5576b730bc30 0x5576b730e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:11.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.827+0000 7f014aba0740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5576b72c53b0 con 0x5576b730bc30 2024-03-18T16:16:11.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.827+0000 7f014aba0740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5576b7262450 con 0x5576b72fd8c0 2024-03-18T16:16:11.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.827+0000 7f014aba0740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x5576b720acb0 con 0x5576b72fe2b0 2024-03-18T16:16:11.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.828+0000 7f0145400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5576b72fe2b0 0x5576b7302db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:11.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.828+0000 7f0145e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5576b72fd8c0 0x5576b72fdce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:11.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.828+0000 7f0145e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5576b72fd8c0 0x5576b72fdce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:53708/0 (socket says 172.21.15.39:53708) 2024-03-18T16:16:11.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.828+0000 7f0145e00640 1 -- 172.21.15.39:0/60216896 learned_addr learned my addr 172.21.15.39:0/60216896 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:16:11.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.828+0000 7f0145400640 1 -- 172.21.15.39:0/60216896 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5576b72fd8c0 msgr2=0x5576b72fdce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:11.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.828+0000 7f0145400640 1 --2- 172.21.15.39:0/60216896 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5576b72fd8c0 0x5576b72fdce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:11.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.828+0000 7f0145400640 1 -- 172.21.15.39:0/60216896 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5576b730bc30 msgr2=0x5576b730e0c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:16:11.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.828+0000 7f0145400640 1 --2- 172.21.15.39:0/60216896 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5576b730bc30 0x5576b730e0c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:11.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.828+0000 7f0145400640 1 -- 172.21.15.39:0/60216896 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5576b722c1f0 con 0x5576b72fe2b0 2024-03-18T16:16:11.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.829+0000 7f013e000640 1 --2- 172.21.15.39:0/2701730666 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f01280a2850 0x7f01280a4d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:11.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.829+0000 7f0145400640 1 --2- 172.21.15.39:0/60216896 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5576b72fe2b0 0x5576b7302db0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f01340ee600 tx=0x7f01340f90e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3f38f11d69d88f2d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:11.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.829+0000 7f013e000640 1 -- 172.21.15.39:0/2701730666 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f01340f0ec0 con 0x7f01400741e0 2024-03-18T16:16:11.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.829+0000 7f0145e00640 1 --2- 172.21.15.39:0/2701730666 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f01280a2850 0x7f01280a4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:11.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.830+0000 7f0145e00640 1 --2- 172.21.15.39:0/2701730666 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f01280a2850 0x7f01280a4d10 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x5576b72feb10 tx=0x7f01380094f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:11.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.831+0000 7f0132600640 1 -- 172.21.15.39:0/60216896 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01340b96e0 con 0x5576b72fe2b0 2024-03-18T16:16:11.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.831+0000 7f0132600640 1 -- 172.21.15.39:0/60216896 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f01340fe070 con 0x5576b72fe2b0 2024-03-18T16:16:11.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.831+0000 7f0132600640 1 -- 172.21.15.39:0/60216896 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01340bb610 con 0x5576b72fe2b0 2024-03-18T16:16:11.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.832+0000 7f014aba0740 1 -- 172.21.15.39:0/60216896 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5576b72fe2b0 msgr2=0x5576b7302db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:11.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.832+0000 7f014aba0740 1 --2- 172.21.15.39:0/60216896 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5576b72fe2b0 0x5576b7302db0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f01340ee600 tx=0x7f01340f90e0 comp rx=0 tx=0).stop 2024-03-18T16:16:11.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.833+0000 7f014aba0740 1 -- 172.21.15.39:0/60216896 shutdown_connections 2024-03-18T16:16:11.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.833+0000 7f014aba0740 1 --2- 172.21.15.39:0/60216896 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5576b730bc30 0x5576b730e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:11.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.833+0000 7f014aba0740 1 --2- 172.21.15.39:0/60216896 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5576b72fe2b0 0x5576b7302db0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:11.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.833+0000 7f014aba0740 1 --2- 172.21.15.39:0/60216896 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5576b72fd8c0 0x5576b72fdce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:11.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.833+0000 7f014aba0740 1 -- 172.21.15.39:0/60216896 >> 172.21.15.39:0/60216896 conn(0x5576b72f8e30 msgr2=0x5576b72f9230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:11.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.833+0000 7f014aba0740 1 -- 172.21.15.39:0/60216896 shutdown_connections 2024-03-18T16:16:11.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.833+0000 7f014aba0740 1 -- 172.21.15.39:0/60216896 wait complete. 2024-03-18T16:16:11.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.833+0000 7f014aba0740 1 -- start start 2024-03-18T16:16:11.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.834+0000 7f014aba0740 1 Processor -- start 2024-03-18T16:16:11.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.834+0000 7f014aba0740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5576b72fd8c0 0x5576b7310140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:11.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.834+0000 7f014aba0740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5576b72fe2b0 0x5576b7310680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:11.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.834+0000 7f014aba0740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5576b730bc30 0x5576b73176c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:11.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.834+0000 7f014aba0740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5576b721e110 con 0x5576b72fe2b0 2024-03-18T16:16:11.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.834+0000 7f014aba0740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5576b725e4f0 con 0x5576b72fd8c0 2024-03-18T16:16:11.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.834+0000 7f014aba0740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x5576b730e9a0 con 0x5576b730bc30 2024-03-18T16:16:11.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.835+0000 7f014aba0740 10 client.0 _ll_register_callbacks cb 0x5576b720bb70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:16:11.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.835+0000 7f0144a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5576b72fe2b0 0x5576b7310680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:11.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.835+0000 7f0145400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5576b72fd8c0 0x5576b7310140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:11.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.835+0000 7f0145400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5576b72fd8c0 0x5576b7310140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:53712/0 (socket says 172.21.15.39:53712) 2024-03-18T16:16:11.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.835+0000 7f0145400640 1 -- 172.21.15.39:0/3275321224 learned_addr learned my addr 172.21.15.39:0/3275321224 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:16:11.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.835+0000 7f0145e00640 1 --2- 172.21.15.39:0/3275321224 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5576b730bc30 0x5576b73176c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:11.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.836+0000 7f0145e00640 1 -- 172.21.15.39:0/3275321224 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5576b72fd8c0 msgr2=0x5576b7310140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:11.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.836+0000 7f0145e00640 1 --2- 172.21.15.39:0/3275321224 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5576b72fd8c0 0x5576b7310140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:11.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.836+0000 7f0145e00640 1 -- 172.21.15.39:0/3275321224 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5576b72fe2b0 msgr2=0x5576b7310680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:11.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.836+0000 7f0145e00640 1 --2- 172.21.15.39:0/3275321224 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5576b72fe2b0 0x5576b7310680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:11.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.836+0000 7f0145e00640 1 -- 172.21.15.39:0/3275321224 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5576b6f763f0 con 0x5576b730bc30 2024-03-18T16:16:11.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.836+0000 7f0144a00640 1 --2- 172.21.15.39:0/3275321224 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5576b72fe2b0 0x5576b7310680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:16:11.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.836+0000 7f0145e00640 1 --2- 172.21.15.39:0/3275321224 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5576b730bc30 0x5576b73176c0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f0138017bf0 tx=0x7f0138004b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8c2ab2495d958111 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:11.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.836+0000 7f0126a00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:16:11.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.837+0000 7f0126a00640 1 -- 172.21.15.39:0/3275321224 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0138027060 con 0x5576b730bc30 2024-03-18T16:16:11.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.837+0000 7f0126a00640 1 -- 172.21.15.39:0/3275321224 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f0138015d40 con 0x5576b730bc30 2024-03-18T16:16:11.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.837+0000 7f014aba0740 1 -- 172.21.15.39:0/3275321224 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f01340fe070 con 0x5576b730bc30 2024-03-18T16:16:11.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.837+0000 7f0126a00640 1 -- 172.21.15.39:0/3275321224 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f0138007e60 con 0x5576b730bc30 2024-03-18T16:16:11.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.837+0000 7f014aba0740 10 client.26794 fetch_fsmap learned FSMap version 44 2024-03-18T16:16:11.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.837+0000 7f014aba0740 1 -- 172.21.15.39:0/3275321224 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x5576b70407d0 con 0x5576b730bc30 2024-03-18T16:16:11.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.838+0000 7f0126a00640 1 -- 172.21.15.39:0/3275321224 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f01340fe070 con 0x5576b730bc30 2024-03-18T16:16:11.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.838+0000 7f0126a00640 1 -- 172.21.15.39:0/3275321224 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7f013802a050 con 0x5576b730bc30 2024-03-18T16:16:11.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.838+0000 7f014aba0740 10 client.26794 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:16:11.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.838+0000 7f014aba0740 10 client.26794 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-03-18T16:16:11.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.838+0000 7f014aba0740 20 client.26794 populate_metadata read hostname 'smithi039' 2024-03-18T16:16:11.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.838+0000 7f014aba0740 1 --2- 172.21.15.39:0/3275321224 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x5576b731bfe0 0x5576b731e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:11.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.838+0000 7f014aba0740 4 client.26794 mds_command: new command op to 14463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:16:11.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.838+0000 7f014aba0740 1 -- 172.21.15.39:0/3275321224 --> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5576b726f990 con 0x5576b731bfe0 2024-03-18T16:16:11.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.839+0000 7f0145400640 1 --2- 172.21.15.39:0/3275321224 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x5576b731bfe0 0x5576b731e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:11.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.840+0000 7f0145400640 1 --2- 172.21.15.39:0/3275321224 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x5576b731bfe0 0x5576b731e430 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=ec5f3046893bcc79 server_cookie=7bc9e6fb19913e6c in_seq=0 out_seq=0 2024-03-18T16:16:11.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.840+0000 7f0126a00640 10 client.26794 ms_handle_connect on v2:172.21.15.39:6840/3515326042 2024-03-18T16:16:11.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.841+0000 7f0126a00640 1 -- 172.21.15.39:0/3275321224 <== mds.4 v2:172.21.15.39:6840/3515326042 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5576b726f990 con 0x5576b731bfe0 2024-03-18T16:16:11.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.842+0000 7f0126a00640 10 client.26794 handle_command_reply: tid=0 2024-03-18T16:16:11.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.842+0000 7f014aba0740 1 client.26794 shutdown 2024-03-18T16:16:11.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.843+0000 7f014aba0740 1 -- 172.21.15.39:0/3275321224 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5576b730bc30 msgr2=0x5576b73176c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:11.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.843+0000 7f014aba0740 1 --2- 172.21.15.39:0/3275321224 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5576b730bc30 0x5576b73176c0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f0138017bf0 tx=0x7f0138004b60 comp rx=0 tx=0).stop 2024-03-18T16:16:11.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.843+0000 7f014aba0740 1 -- 172.21.15.39:0/3275321224 shutdown_connections 2024-03-18T16:16:11.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.843+0000 7f014aba0740 1 --2- 172.21.15.39:0/3275321224 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x5576b731bfe0 0x5576b731e430 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:11.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.843+0000 7f014aba0740 1 --2- 172.21.15.39:0/3275321224 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5576b730bc30 0x5576b73176c0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:11.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.843+0000 7f014aba0740 1 --2- 172.21.15.39:0/3275321224 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5576b72fe2b0 0x5576b7310680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:11.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.843+0000 7f014aba0740 1 --2- 172.21.15.39:0/3275321224 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5576b72fd8c0 0x5576b7310140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:11.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.843+0000 7f014aba0740 1 -- 172.21.15.39:0/3275321224 >> 172.21.15.39:0/3275321224 conn(0x5576b72f8e30 msgr2=0x5576b72f81f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:11.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.843+0000 7f0126a00640 0 client.26794 ms_handle_reset on v2:172.21.15.39:6840/3515326042 2024-03-18T16:16:11.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.843+0000 7f014aba0740 1 -- 172.21.15.39:0/3275321224 shutdown_connections 2024-03-18T16:16:11.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.843+0000 7f014aba0740 1 -- 172.21.15.39:0/3275321224 wait complete. 2024-03-18T16:16:11.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.844+0000 7f014aba0740 20 client.26794 trim_cache size 0 max 16384 2024-03-18T16:16:11.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.872+0000 7f014aba0740 1 Processor -- start 2024-03-18T16:16:11.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.872+0000 7f014aba0740 1 -- start start 2024-03-18T16:16:11.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.872+0000 7f014aba0740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5576b72fd8c0 0x5576b72f7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:11.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.872+0000 7f014aba0740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5576b72fe2b0 0x5576b7317680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:11.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.872+0000 7f014aba0740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5576b730bc30 0x5576b72f7ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:11.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.873+0000 7f014aba0740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f0140071dd0 con 0x5576b72fd8c0 2024-03-18T16:16:11.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.873+0000 7f014aba0740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f01401b2850 con 0x5576b72fe2b0 2024-03-18T16:16:11.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.873+0000 7f014aba0740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f01400724a0 con 0x5576b730bc30 2024-03-18T16:16:11.877 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.873+0000 7f0144a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5576b72fe2b0 0x5576b7317680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:11.877 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.873+0000 7f0145e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5576b730bc30 0x5576b72f7ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:11.877 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.873+0000 7f0145e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5576b730bc30 0x5576b72f7ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:59644/0 (socket says 172.21.15.39:59644) 2024-03-18T16:16:11.877 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.873+0000 7f0144a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5576b72fe2b0 0x5576b7317680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:53728/0 (socket says 172.21.15.39:53728) 2024-03-18T16:16:11.877 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.873+0000 7f0145e00640 1 -- 172.21.15.39:0/1043536674 learned_addr learned my addr 172.21.15.39:0/1043536674 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:16:11.877 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.873+0000 7f0145400640 1 --2- 172.21.15.39:0/1043536674 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5576b72fd8c0 0x5576b72f7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:11.877 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.874+0000 7f0144a00640 1 -- 172.21.15.39:0/1043536674 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5576b730bc30 msgr2=0x5576b72f7ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:11.877 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.874+0000 7f0144a00640 1 --2- 172.21.15.39:0/1043536674 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5576b730bc30 0x5576b72f7ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:11.877 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.874+0000 7f0144a00640 1 -- 172.21.15.39:0/1043536674 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5576b72fd8c0 msgr2=0x5576b72f7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:11.877 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.874+0000 7f0144a00640 1 --2- 172.21.15.39:0/1043536674 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5576b72fd8c0 0x5576b72f7960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:11.877 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.874+0000 7f0144a00640 1 -- 172.21.15.39:0/1043536674 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5576b72f4c10 con 0x5576b72fe2b0 2024-03-18T16:16:11.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.874+0000 7f0144a00640 1 --2- 172.21.15.39:0/1043536674 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5576b72fe2b0 0x5576b7317680 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x5576b72f9da0 tx=0x7f012c009f00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d66a6deff10903cd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:11.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.875+0000 7f0131c00640 1 -- 172.21.15.39:0/1043536674 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f012c012ca0 con 0x5576b72fe2b0 2024-03-18T16:16:11.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.875+0000 7f0131c00640 1 -- 172.21.15.39:0/1043536674 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f012c012e40 con 0x5576b72fe2b0 2024-03-18T16:16:11.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.875+0000 7f0131c00640 1 -- 172.21.15.39:0/1043536674 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f012c00e640 con 0x5576b72fe2b0 2024-03-18T16:16:11.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.876+0000 7f014aba0740 1 -- 172.21.15.39:0/1043536674 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5576b72fe2b0 msgr2=0x5576b7317680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:11.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.876+0000 7f014aba0740 1 --2- 172.21.15.39:0/1043536674 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5576b72fe2b0 0x5576b7317680 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x5576b72f9da0 tx=0x7f012c009f00 comp rx=0 tx=0).stop 2024-03-18T16:16:11.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.876+0000 7f014aba0740 1 -- 172.21.15.39:0/1043536674 shutdown_connections 2024-03-18T16:16:11.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.876+0000 7f014aba0740 1 --2- 172.21.15.39:0/1043536674 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5576b72fe2b0 0x5576b7317680 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:11.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.876+0000 7f014aba0740 1 --2- 172.21.15.39:0/1043536674 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5576b730bc30 0x5576b72f7ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:11.880 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.876+0000 7f014aba0740 1 --2- 172.21.15.39:0/1043536674 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5576b72fd8c0 0x5576b72f7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:11.880 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.876+0000 7f014aba0740 1 -- 172.21.15.39:0/1043536674 >> 172.21.15.39:0/1043536674 conn(0x5576b72f8e30 msgr2=0x5576b72f9210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:11.880 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.877+0000 7f014aba0740 1 -- 172.21.15.39:0/1043536674 shutdown_connections 2024-03-18T16:16:11.880 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.877+0000 7f014aba0740 1 -- 172.21.15.39:0/1043536674 wait complete. 2024-03-18T16:16:11.880 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.877+0000 7f014aba0740 1 -- start start 2024-03-18T16:16:11.880 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.878+0000 7f014aba0740 1 Processor -- start 2024-03-18T16:16:11.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.878+0000 7f014aba0740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5576b72fd8c0 0x5576b7315050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:11.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.878+0000 7f014aba0740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5576b72fe2b0 0x5576b7315590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:11.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.878+0000 7f014aba0740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5576b730bc30 0x5576b73165a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:11.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.878+0000 7f014aba0740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f0140073240 con 0x5576b72fd8c0 2024-03-18T16:16:11.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.878+0000 7f014aba0740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f0140072b70 con 0x5576b72fe2b0 2024-03-18T16:16:11.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.878+0000 7f014aba0740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f014006fd70 con 0x5576b730bc30 2024-03-18T16:16:11.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.878+0000 7f014aba0740 10 client.0 _ll_register_callbacks cb 0x7f01380178a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:16:11.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.879+0000 7f0144a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5576b72fe2b0 0x5576b7315590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:11.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.879+0000 7f0144a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5576b72fe2b0 0x5576b7315590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:53740/0 (socket says 172.21.15.39:53740) 2024-03-18T16:16:11.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.879+0000 7f0144a00640 1 -- 172.21.15.39:0/3249759927 learned_addr learned my addr 172.21.15.39:0/3249759927 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:16:11.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.879+0000 7f0145e00640 1 --2- 172.21.15.39:0/3249759927 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5576b730bc30 0x5576b73165a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:11.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.879+0000 7f0145400640 1 --2- 172.21.15.39:0/3249759927 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5576b72fd8c0 0x5576b7315050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:11.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.879+0000 7f0144a00640 1 -- 172.21.15.39:0/3249759927 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5576b730bc30 msgr2=0x5576b73165a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:11.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.879+0000 7f0144a00640 1 --2- 172.21.15.39:0/3249759927 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5576b730bc30 0x5576b73165a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:11.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.879+0000 7f0144a00640 1 -- 172.21.15.39:0/3249759927 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5576b72fd8c0 msgr2=0x5576b7315050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:11.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.879+0000 7f0144a00640 1 --2- 172.21.15.39:0/3249759927 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5576b72fd8c0 0x5576b7315050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:11.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.879+0000 7f0144a00640 1 -- 172.21.15.39:0/3249759927 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5576b7316b90 con 0x5576b72fe2b0 2024-03-18T16:16:11.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.879+0000 7f0145400640 1 --2- 172.21.15.39:0/3249759927 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5576b72fd8c0 0x5576b7315050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:16:11.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.880+0000 7f0144a00640 1 --2- 172.21.15.39:0/3249759927 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5576b72fe2b0 0x5576b7315590 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f012c00a010 tx=0x7f012c009870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=33794cedf36ebee5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:11.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.880+0000 7f0126a00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:16:11.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.880+0000 7f0126a00640 1 -- 172.21.15.39:0/3249759927 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f012c02b060 con 0x5576b72fe2b0 2024-03-18T16:16:11.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.880+0000 7f014aba0740 1 -- 172.21.15.39:0/3249759927 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5576b72ff2c0 con 0x5576b72fe2b0 2024-03-18T16:16:11.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.880+0000 7f0126a00640 1 -- 172.21.15.39:0/3249759927 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f012c005450 con 0x5576b72fe2b0 2024-03-18T16:16:11.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.880+0000 7f0126a00640 1 -- 172.21.15.39:0/3249759927 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f012c032050 con 0x5576b72fe2b0 2024-03-18T16:16:11.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.880+0000 7f014aba0740 10 client.26834 fetch_fsmap learned FSMap version 44 2024-03-18T16:16:11.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.880+0000 7f014aba0740 1 -- 172.21.15.39:0/3249759927 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x5576b7382590 con 0x5576b72fe2b0 2024-03-18T16:16:11.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.881+0000 7f0126a00640 1 -- 172.21.15.39:0/3249759927 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5576b72ff2c0 con 0x5576b72fe2b0 2024-03-18T16:16:11.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.881+0000 7f0126a00640 1 -- 172.21.15.39:0/3249759927 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7f012c019020 con 0x5576b72fe2b0 2024-03-18T16:16:11.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.881+0000 7f014aba0740 10 client.26834 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:16:11.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.881+0000 7f014aba0740 10 client.26834 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-03-18T16:16:11.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.881+0000 7f014aba0740 20 client.26834 populate_metadata read hostname 'smithi039' 2024-03-18T16:16:11.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.881+0000 7f014aba0740 1 --2- 172.21.15.39:0/3249759927 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x5576b7382eb0 0x5576b73852d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:11.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.881+0000 7f014aba0740 4 client.26834 mds_command: new command op to 14463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:16:11.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.881+0000 7f014aba0740 1 -- 172.21.15.39:0/3249759927 --> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5576b7269ce0 con 0x5576b7382eb0 2024-03-18T16:16:11.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.882+0000 7f0145400640 1 --2- 172.21.15.39:0/3249759927 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x5576b7382eb0 0x5576b73852d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:11.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.883+0000 7f0145400640 1 --2- 172.21.15.39:0/3249759927 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x5576b7382eb0 0x5576b73852d0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=4a1193eb67c79726 server_cookie=5272845beed6b1c3 in_seq=0 out_seq=0 2024-03-18T16:16:11.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.883+0000 7f0126a00640 10 client.26834 ms_handle_connect on v2:172.21.15.39:6840/3515326042 2024-03-18T16:16:11.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.883+0000 7f0126a00640 1 -- 172.21.15.39:0/3249759927 <== mds.4 v2:172.21.15.39:6840/3515326042 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5576b7269ce0 con 0x5576b7382eb0 2024-03-18T16:16:11.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.883+0000 7f0126a00640 10 client.26834 handle_command_reply: tid=0 2024-03-18T16:16:11.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.884+0000 7f014aba0740 1 client.26834 shutdown 2024-03-18T16:16:11.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.884+0000 7f014aba0740 1 -- 172.21.15.39:0/3249759927 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5576b72fe2b0 msgr2=0x5576b7315590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:11.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.884+0000 7f014aba0740 1 --2- 172.21.15.39:0/3249759927 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5576b72fe2b0 0x5576b7315590 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f012c00a010 tx=0x7f012c009870 comp rx=0 tx=0).stop 2024-03-18T16:16:11.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.884+0000 7f014aba0740 1 -- 172.21.15.39:0/3249759927 shutdown_connections 2024-03-18T16:16:11.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.884+0000 7f014aba0740 1 --2- 172.21.15.39:0/3249759927 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x5576b7382eb0 0x5576b73852d0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:11.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.884+0000 7f014aba0740 1 --2- 172.21.15.39:0/3249759927 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5576b72fe2b0 0x5576b7315590 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:11.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.884+0000 7f014aba0740 1 --2- 172.21.15.39:0/3249759927 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5576b730bc30 0x5576b73165a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:11.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.884+0000 7f014aba0740 1 --2- 172.21.15.39:0/3249759927 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5576b72fd8c0 0x5576b7315050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:11.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.884+0000 7f014aba0740 1 -- 172.21.15.39:0/3249759927 >> 172.21.15.39:0/3249759927 conn(0x5576b72f8e30 msgr2=0x5576b72fc090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:11.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.884+0000 7f0126a00640 0 client.26834 ms_handle_reset on v2:172.21.15.39:6840/3515326042 2024-03-18T16:16:11.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.885+0000 7f014aba0740 1 -- 172.21.15.39:0/3249759927 shutdown_connections 2024-03-18T16:16:11.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.885+0000 7f014aba0740 1 -- 172.21.15.39:0/3249759927 wait complete. 2024-03-18T16:16:11.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.885+0000 7f014aba0740 20 client.26834 trim_cache size 0 max 16384 2024-03-18T16:16:11.891 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:16:11.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.889+0000 7f0147e00640 1 -- 172.21.15.39:0/2701730666 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f01280a2850 msgr2=0x7f01280a4d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:11.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.889+0000 7f0147e00640 1 --2- 172.21.15.39:0/2701730666 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f01280a2850 0x7f01280a4d10 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x5576b72feb10 tx=0x7f01380094f0 comp rx=0 tx=0).stop 2024-03-18T16:16:11.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.889+0000 7f0147e00640 1 -- 172.21.15.39:0/2701730666 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f01400741e0 msgr2=0x7f0140199d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:11.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.889+0000 7f0147e00640 1 --2- 172.21.15.39:0/2701730666 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f01400741e0 0x7f0140199d50 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f0134000c00 tx=0x7f013402c690 comp rx=0 tx=0).stop 2024-03-18T16:16:11.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.889+0000 7f0147e00640 1 -- 172.21.15.39:0/2701730666 shutdown_connections 2024-03-18T16:16:11.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.889+0000 7f0147e00640 1 --2- 172.21.15.39:0/2701730666 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f01400757c0 0x7f014019a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:11.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.889+0000 7f0147e00640 1 --2- 172.21.15.39:0/2701730666 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f01280a2850 0x7f01280a4d10 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:11.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.889+0000 7f0147e00640 1 --2- 172.21.15.39:0/2701730666 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0140109670 0x7f01401a1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:11.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.889+0000 7f0147e00640 1 --2- 172.21.15.39:0/2701730666 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f01400741e0 0x7f0140199d50 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:11.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.889+0000 7f0147e00640 1 -- 172.21.15.39:0/2701730666 >> 172.21.15.39:0/2701730666 conn(0x7f01400fcd70 msgr2=0x7f0140105600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:11.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.889+0000 7f0147e00640 1 -- 172.21.15.39:0/2701730666 shutdown_connections 2024-03-18T16:16:11.893 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:11.889+0000 7f0147e00640 1 -- 172.21.15.39:0/2701730666 wait complete. 2024-03-18T16:16:11.916 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-18T16:16:12.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.206+0000 7f642aa00640 1 -- 172.21.15.39:0/1973249382 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6424104be0 msgr2=0x7f6424105060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:12.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.206+0000 7f642aa00640 1 --2- 172.21.15.39:0/1973249382 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6424104be0 0x7f6424105060 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f6410009860 tx=0x7f641002bb70 comp rx=0 tx=0).stop 2024-03-18T16:16:12.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.206+0000 7f642aa00640 1 -- 172.21.15.39:0/1973249382 shutdown_connections 2024-03-18T16:16:12.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.206+0000 7f642aa00640 1 --2- 172.21.15.39:0/1973249382 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f64241055a0 0x7f642410be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.206+0000 7f642aa00640 1 --2- 172.21.15.39:0/1973249382 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6424104be0 0x7f6424105060 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.206+0000 7f642aa00640 1 --2- 172.21.15.39:0/1973249382 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f64241039f0 0x7f6424103e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.206+0000 7f642aa00640 1 -- 172.21.15.39:0/1973249382 >> 172.21.15.39:0/1973249382 conn(0x7f64240fefa0 msgr2=0x7f64241013c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:12.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.207+0000 7f642aa00640 1 -- 172.21.15.39:0/1973249382 shutdown_connections 2024-03-18T16:16:12.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.207+0000 7f642aa00640 1 -- 172.21.15.39:0/1973249382 wait complete. 2024-03-18T16:16:12.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.208+0000 7f642aa00640 1 Processor -- start 2024-03-18T16:16:12.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.208+0000 7f642aa00640 1 -- start start 2024-03-18T16:16:12.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.208+0000 7f642aa00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f64241039f0 0x7f64241a6950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:12.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.209+0000 7f642aa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6424104be0 0x7f64241a6e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:12.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.209+0000 7f642aa00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f64241055a0 0x7f64241adf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:12.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.209+0000 7f642aa00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f642410f6a0 con 0x7f6424104be0 2024-03-18T16:16:12.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.209+0000 7f642aa00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f642410f520 con 0x7f64241055a0 2024-03-18T16:16:12.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.209+0000 7f642aa00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f642410f820 con 0x7f64241039f0 2024-03-18T16:16:12.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.209+0000 7f6423e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f64241039f0 0x7f64241a6950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:12.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.209+0000 7f6423e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f64241039f0 0x7f64241a6950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:59672/0 (socket says 172.21.15.39:59672) 2024-03-18T16:16:12.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.209+0000 7f6423e00640 1 -- 172.21.15.39:0/1076333800 learned_addr learned my addr 172.21.15.39:0/1076333800 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:16:12.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.209+0000 7f6423400640 1 --2- 172.21.15.39:0/1076333800 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6424104be0 0x7f64241a6e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:12.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.209+0000 7f6428a00640 1 --2- 172.21.15.39:0/1076333800 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f64241055a0 0x7f64241adf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:12.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.210+0000 7f6423e00640 1 -- 172.21.15.39:0/1076333800 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f64241055a0 msgr2=0x7f64241adf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:12.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.210+0000 7f6423e00640 1 --2- 172.21.15.39:0/1076333800 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f64241055a0 0x7f64241adf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.210+0000 7f6423e00640 1 -- 172.21.15.39:0/1076333800 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6424104be0 msgr2=0x7f64241a6e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:12.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.210+0000 7f6423e00640 1 --2- 172.21.15.39:0/1076333800 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6424104be0 0x7f64241a6e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.210+0000 7f6423e00640 1 -- 172.21.15.39:0/1076333800 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f642406b330 con 0x7f64241039f0 2024-03-18T16:16:12.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.210+0000 7f6423e00640 1 --2- 172.21.15.39:0/1076333800 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f64241039f0 0x7f64241a6950 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f641400e9c0 tx=0x7f641400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:12.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.211+0000 7f6420c00640 1 -- 172.21.15.39:0/1076333800 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f641400cd80 con 0x7f64241039f0 2024-03-18T16:16:12.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.211+0000 7f6420c00640 1 -- 172.21.15.39:0/1076333800 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f64140035f0 con 0x7f64241039f0 2024-03-18T16:16:12.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.211+0000 7f6420c00640 1 -- 172.21.15.39:0/1076333800 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6414010690 con 0x7f64241039f0 2024-03-18T16:16:12.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.211+0000 7f642aa00640 1 -- 172.21.15.39:0/1076333800 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f642406b620 con 0x7f64241039f0 2024-03-18T16:16:12.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.211+0000 7f642aa00640 1 -- 172.21.15.39:0/1076333800 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f642406bb20 con 0x7f64241039f0 2024-03-18T16:16:12.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.216+0000 7f6420c00640 1 -- 172.21.15.39:0/1076333800 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f64140037a0 con 0x7f64241039f0 2024-03-18T16:16:12.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.218+0000 7f6420c00640 1 --2- 172.21.15.39:0/1076333800 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f64080a28d0 0x7f64080a4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:12.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.218+0000 7f6420c00640 1 -- 172.21.15.39:0/1076333800 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f64140cf290 con 0x7f64241039f0 2024-03-18T16:16:12.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.218+0000 7f642d668740 1 Processor -- start 2024-03-18T16:16:12.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.218+0000 7f6423400640 1 --2- 172.21.15.39:0/1076333800 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f64080a28d0 0x7f64080a4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:12.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.218+0000 7f642d668740 1 -- start start 2024-03-18T16:16:12.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.219+0000 7f642d668740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564e9e7c08c0 0x564e9e7c0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:12.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.219+0000 7f642d668740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564e9e7c12b0 0x564e9e7c5db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:12.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.219+0000 7f642d668740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564e9e7c6420 0x564e9e7c88b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:12.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.219+0000 7f642d668740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x564e9e7883b0 con 0x564e9e7c12b0 2024-03-18T16:16:12.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.219+0000 7f642d668740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x564e9e725450 con 0x564e9e7c6420 2024-03-18T16:16:12.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.219+0000 7f642d668740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x564e9e6cdcb0 con 0x564e9e7c08c0 2024-03-18T16:16:12.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.219+0000 7f6428a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564e9e7c6420 0x564e9e7c88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:12.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.219+0000 7f6423400640 1 --2- 172.21.15.39:0/1076333800 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f64080a28d0 0x7f64080a4d90 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f6410037040 tx=0x7f6410009490 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:12.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.219+0000 7f6428a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564e9e7c6420 0x564e9e7c88b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:53762/0 (socket says 172.21.15.39:53762) 2024-03-18T16:16:12.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.219+0000 7f6428a00640 1 -- 172.21.15.39:0/3636127334 learned_addr learned my addr 172.21.15.39:0/3636127334 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:16:12.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.219+0000 7f6423400640 1 --2- 172.21.15.39:0/3636127334 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564e9e7c12b0 0x564e9e7c5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:12.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.219+0000 7f6423e00640 1 --2- 172.21.15.39:0/3636127334 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564e9e7c08c0 0x564e9e7c0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:12.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.220+0000 7f6428a00640 1 -- 172.21.15.39:0/3636127334 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564e9e7c08c0 msgr2=0x564e9e7c0ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:12.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.220+0000 7f6428a00640 1 --2- 172.21.15.39:0/3636127334 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564e9e7c08c0 0x564e9e7c0ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.220+0000 7f6428a00640 1 -- 172.21.15.39:0/3636127334 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564e9e7c12b0 msgr2=0x564e9e7c5db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:12.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.220+0000 7f6428a00640 1 --2- 172.21.15.39:0/3636127334 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564e9e7c12b0 0x564e9e7c5db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.220+0000 7f6428a00640 1 -- 172.21.15.39:0/3636127334 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e9e6ef1f0 con 0x564e9e7c6420 2024-03-18T16:16:12.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.220+0000 7f6423e00640 1 --2- 172.21.15.39:0/3636127334 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564e9e7c08c0 0x564e9e7c0ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:16:12.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.220+0000 7f6428a00640 1 --2- 172.21.15.39:0/3636127334 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564e9e7c6420 0x564e9e7c88b0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f641c00f480 tx=0x7f641c00f940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=281aa04b37b9631d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:12.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.220+0000 7f6419200640 1 -- 172.21.15.39:0/3636127334 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f641c01d070 con 0x564e9e7c6420 2024-03-18T16:16:12.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.220+0000 7f6419200640 1 -- 172.21.15.39:0/3636127334 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f641c00cb80 con 0x564e9e7c6420 2024-03-18T16:16:12.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.220+0000 7f6419200640 1 -- 172.21.15.39:0/3636127334 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f641c018d90 con 0x564e9e7c6420 2024-03-18T16:16:12.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.221+0000 7f642d668740 1 -- 172.21.15.39:0/3636127334 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564e9e7c6420 msgr2=0x564e9e7c88b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:12.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.221+0000 7f642d668740 1 --2- 172.21.15.39:0/3636127334 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564e9e7c6420 0x564e9e7c88b0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f641c00f480 tx=0x7f641c00f940 comp rx=0 tx=0).stop 2024-03-18T16:16:12.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.221+0000 7f642d668740 1 -- 172.21.15.39:0/3636127334 shutdown_connections 2024-03-18T16:16:12.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.221+0000 7f642d668740 1 --2- 172.21.15.39:0/3636127334 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564e9e7c6420 0x564e9e7c88b0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.221+0000 7f642d668740 1 --2- 172.21.15.39:0/3636127334 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564e9e7c12b0 0x564e9e7c5db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.221+0000 7f642d668740 1 --2- 172.21.15.39:0/3636127334 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564e9e7c08c0 0x564e9e7c0ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.221+0000 7f642d668740 1 -- 172.21.15.39:0/3636127334 >> 172.21.15.39:0/3636127334 conn(0x564e9e7bbe30 msgr2=0x564e9e7bc230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:12.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.222+0000 7f642d668740 1 -- 172.21.15.39:0/3636127334 shutdown_connections 2024-03-18T16:16:12.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.222+0000 7f642d668740 1 -- 172.21.15.39:0/3636127334 wait complete. 2024-03-18T16:16:12.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.222+0000 7f642d668740 1 -- start start 2024-03-18T16:16:12.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.222+0000 7f642d668740 1 Processor -- start 2024-03-18T16:16:12.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.223+0000 7f642d668740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564e9e7c08c0 0x564e9e7ca980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:12.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.223+0000 7f642d668740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564e9e7c12b0 0x564e9e7caec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:12.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.223+0000 7f642d668740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564e9e7c6420 0x564e9e7d1f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:12.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.223+0000 7f642d668740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x564e9e6e1110 con 0x564e9e7c08c0 2024-03-18T16:16:12.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.223+0000 7f642d668740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x564e9e7214f0 con 0x564e9e7c12b0 2024-03-18T16:16:12.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.223+0000 7f642d668740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x564e9e7c93d0 con 0x564e9e7c6420 2024-03-18T16:16:12.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.224+0000 7f642d668740 10 client.0 _ll_register_callbacks cb 0x564e9e6ceb70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:16:12.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.224+0000 7f6423400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564e9e7c12b0 0x564e9e7caec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:12.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.224+0000 7f6423400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564e9e7c12b0 0x564e9e7caec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:53770/0 (socket says 172.21.15.39:53770) 2024-03-18T16:16:12.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.224+0000 7f6423400640 1 -- 172.21.15.39:0/3087176655 learned_addr learned my addr 172.21.15.39:0/3087176655 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:16:12.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.224+0000 7f6423e00640 1 --2- 172.21.15.39:0/3087176655 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564e9e7c08c0 0x564e9e7ca980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:12.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.224+0000 7f6428a00640 1 --2- 172.21.15.39:0/3087176655 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564e9e7c6420 0x564e9e7d1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:12.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.225+0000 7f6423e00640 1 -- 172.21.15.39:0/3087176655 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564e9e7c6420 msgr2=0x564e9e7d1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:12.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.225+0000 7f6423e00640 1 --2- 172.21.15.39:0/3087176655 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564e9e7c6420 0x564e9e7d1f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.225+0000 7f6423e00640 1 -- 172.21.15.39:0/3087176655 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564e9e7c12b0 msgr2=0x564e9e7caec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:12.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.225+0000 7f6423e00640 1 --2- 172.21.15.39:0/3087176655 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564e9e7c12b0 0x564e9e7caec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.225+0000 7f6423e00640 1 -- 172.21.15.39:0/3087176655 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e9e4393f0 con 0x564e9e7c08c0 2024-03-18T16:16:12.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.225+0000 7f6423400640 1 --2- 172.21.15.39:0/3087176655 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564e9e7c12b0 0x564e9e7caec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:16:12.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.225+0000 7f6423e00640 1 --2- 172.21.15.39:0/3087176655 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564e9e7c08c0 0x564e9e7ca980 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f641409b5a0 tx=0x7f641409b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e572b20c0446941 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:12.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.226+0000 7f640d600640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-03-18T16:16:12.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.226+0000 7f640d600640 1 -- 172.21.15.39:0/3087176655 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64140a4040 con 0x564e9e7c08c0 2024-03-18T16:16:12.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.226+0000 7f640d600640 1 -- 172.21.15.39:0/3087176655 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f6414003b30 con 0x564e9e7c08c0 2024-03-18T16:16:12.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.226+0000 7f642d668740 1 -- 172.21.15.39:0/3087176655 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f641c00cb80 con 0x564e9e7c08c0 2024-03-18T16:16:12.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.226+0000 7f640d600640 1 -- 172.21.15.39:0/3087176655 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f641409cbd0 con 0x564e9e7c08c0 2024-03-18T16:16:12.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.226+0000 7f642d668740 10 client.16368 fetch_fsmap learned FSMap version 44 2024-03-18T16:16:12.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.226+0000 7f642d668740 1 -- 172.21.15.39:0/3087176655 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x564e9e5037d0 con 0x564e9e7c08c0 2024-03-18T16:16:12.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.227+0000 7f640d600640 1 -- 172.21.15.39:0/3087176655 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f641c00cb80 con 0x564e9e7c08c0 2024-03-18T16:16:12.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.228+0000 7f640d600640 1 -- 172.21.15.39:0/3087176655 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7f641409acf0 con 0x564e9e7c08c0 2024-03-18T16:16:12.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.228+0000 7f642d668740 10 client.16368 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:16:12.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.229+0000 7f642d668740 10 client.16368 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-18T16:16:12.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.229+0000 7f642d668740 20 client.16368 populate_metadata read hostname 'smithi039' 2024-03-18T16:16:12.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.229+0000 7f642d668740 1 --2- 172.21.15.39:0/3087176655 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x564e9e7d6830 0x564e9e7d8ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:12.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.229+0000 7f642d668740 4 client.16368 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:16:12.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.229+0000 7f642d668740 1 -- 172.21.15.39:0/3087176655 --> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564e9e732990 con 0x564e9e7d6830 2024-03-18T16:16:12.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.229+0000 7f6423400640 1 --2- 172.21.15.39:0/3087176655 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x564e9e7d6830 0x564e9e7d8ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:12.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.230+0000 7f6423400640 1 --2- 172.21.15.39:0/3087176655 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x564e9e7d6830 0x564e9e7d8ca0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=f21b6cc404998722 server_cookie=d2726f8e642d930c in_seq=0 out_seq=0 2024-03-18T16:16:12.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.230+0000 7f640d600640 10 client.16368 ms_handle_connect on v2:172.21.15.194:6836/1301464246 2024-03-18T16:16:12.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.233+0000 7f640d600640 1 -- 172.21.15.39:0/3087176655 <== mds.3 v2:172.21.15.194:6836/1301464246 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564e9e732990 con 0x564e9e7d6830 2024-03-18T16:16:12.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.233+0000 7f640d600640 10 client.16368 handle_command_reply: tid=0 2024-03-18T16:16:12.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.233+0000 7f642d668740 1 client.16368 shutdown 2024-03-18T16:16:12.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.234+0000 7f642d668740 1 -- 172.21.15.39:0/3087176655 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564e9e7c08c0 msgr2=0x564e9e7ca980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:12.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.234+0000 7f642d668740 1 --2- 172.21.15.39:0/3087176655 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564e9e7c08c0 0x564e9e7ca980 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f641409b5a0 tx=0x7f641409b8a0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.235+0000 7f642d668740 1 -- 172.21.15.39:0/3087176655 shutdown_connections 2024-03-18T16:16:12.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.235+0000 7f642d668740 1 --2- 172.21.15.39:0/3087176655 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x564e9e7d6830 0x564e9e7d8ca0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.235+0000 7f642d668740 1 --2- 172.21.15.39:0/3087176655 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564e9e7c12b0 0x564e9e7caec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.235+0000 7f642d668740 1 --2- 172.21.15.39:0/3087176655 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564e9e7c6420 0x564e9e7d1f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.235+0000 7f642d668740 1 --2- 172.21.15.39:0/3087176655 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564e9e7c08c0 0x564e9e7ca980 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.235+0000 7f640d600640 0 client.16368 ms_handle_reset on v2:172.21.15.194:6836/1301464246 2024-03-18T16:16:12.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.235+0000 7f642d668740 1 -- 172.21.15.39:0/3087176655 >> 172.21.15.39:0/3087176655 conn(0x564e9e7bbe30 msgr2=0x564e9e7bb1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:12.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.236+0000 7f642d668740 1 -- 172.21.15.39:0/3087176655 shutdown_connections 2024-03-18T16:16:12.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.236+0000 7f642d668740 1 -- 172.21.15.39:0/3087176655 wait complete. 2024-03-18T16:16:12.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.236+0000 7f642d668740 20 client.16368 trim_cache size 0 max 16384 2024-03-18T16:16:12.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.259+0000 7f642d668740 1 Processor -- start 2024-03-18T16:16:12.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.259+0000 7f642d668740 1 -- start start 2024-03-18T16:16:12.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.260+0000 7f642d668740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564e9e7c08c0 0x564e9e7c9e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:12.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.260+0000 7f642d668740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564e9e7c12b0 0x564e9e7d1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:12.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.260+0000 7f642d668740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564e9e7c6420 0x564e9e7de540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:12.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.260+0000 7f642d668740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f64241c1c30 con 0x564e9e7c6420 2024-03-18T16:16:12.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.260+0000 7f642d668740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f64241c3770 con 0x564e9e7c12b0 2024-03-18T16:16:12.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.260+0000 7f642d668740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f64241c2300 con 0x564e9e7c08c0 2024-03-18T16:16:12.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.260+0000 7f6423e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564e9e7c08c0 0x564e9e7c9e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:12.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.260+0000 7f6423e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564e9e7c08c0 0x564e9e7c9e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:59712/0 (socket says 172.21.15.39:59712) 2024-03-18T16:16:12.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.260+0000 7f6423e00640 1 -- 172.21.15.39:0/3490849637 learned_addr learned my addr 172.21.15.39:0/3490849637 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:16:12.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.260+0000 7f6428a00640 1 --2- 172.21.15.39:0/3490849637 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564e9e7c6420 0x564e9e7de540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:12.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.261+0000 7f6423400640 1 --2- 172.21.15.39:0/3490849637 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564e9e7c12b0 0x564e9e7d1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:12.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.261+0000 7f6428a00640 1 -- 172.21.15.39:0/3490849637 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564e9e7c08c0 msgr2=0x564e9e7c9e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:12.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.261+0000 7f6428a00640 1 --2- 172.21.15.39:0/3490849637 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564e9e7c08c0 0x564e9e7c9e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.261+0000 7f6428a00640 1 -- 172.21.15.39:0/3490849637 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564e9e7c12b0 msgr2=0x564e9e7d1ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:12.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.261+0000 7f6428a00640 1 --2- 172.21.15.39:0/3490849637 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564e9e7c12b0 0x564e9e7d1ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.261+0000 7f6428a00640 1 -- 172.21.15.39:0/3490849637 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e9e7b7c10 con 0x564e9e7c6420 2024-03-18T16:16:12.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.262+0000 7f6428a00640 1 --2- 172.21.15.39:0/3490849637 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564e9e7c6420 0x564e9e7de540 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f641c00f450 tx=0x7f641c009c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dc7fcf1123f22ec server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:12.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.262+0000 7f640fe00640 1 -- 172.21.15.39:0/3490849637 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f641c01d040 con 0x564e9e7c6420 2024-03-18T16:16:12.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.262+0000 7f640fe00640 1 -- 172.21.15.39:0/3490849637 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f641c01e040 con 0x564e9e7c6420 2024-03-18T16:16:12.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.262+0000 7f640fe00640 1 -- 172.21.15.39:0/3490849637 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f641c00cb60 con 0x564e9e7c6420 2024-03-18T16:16:12.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.263+0000 7f642d668740 1 -- 172.21.15.39:0/3490849637 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564e9e7c6420 msgr2=0x564e9e7de540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:12.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.263+0000 7f642d668740 1 --2- 172.21.15.39:0/3490849637 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564e9e7c6420 0x564e9e7de540 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f641c00f450 tx=0x7f641c009c90 comp rx=0 tx=0).stop 2024-03-18T16:16:12.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.264+0000 7f642d668740 1 -- 172.21.15.39:0/3490849637 shutdown_connections 2024-03-18T16:16:12.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.264+0000 7f642d668740 1 --2- 172.21.15.39:0/3490849637 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564e9e7c12b0 0x564e9e7d1ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.264+0000 7f642d668740 1 --2- 172.21.15.39:0/3490849637 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564e9e7c6420 0x564e9e7de540 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.264+0000 7f642d668740 1 --2- 172.21.15.39:0/3490849637 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564e9e7c08c0 0x564e9e7c9e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.264+0000 7f642d668740 1 -- 172.21.15.39:0/3490849637 >> 172.21.15.39:0/3490849637 conn(0x564e9e7bbe30 msgr2=0x564e9e7c9770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:12.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.265+0000 7f642d668740 1 -- 172.21.15.39:0/3490849637 shutdown_connections 2024-03-18T16:16:12.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.265+0000 7f642d668740 1 -- 172.21.15.39:0/3490849637 wait complete. 2024-03-18T16:16:12.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.266+0000 7f642d668740 1 -- start start 2024-03-18T16:16:12.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.266+0000 7f642d668740 1 Processor -- start 2024-03-18T16:16:12.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.267+0000 7f642d668740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564e9e7c08c0 0x564e9e7cf5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:12.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.267+0000 7f642d668740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564e9e7c12b0 0x564e9e7cfb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:12.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.267+0000 7f642d668740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564e9e7c6420 0x564e9e7d0b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:12.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.267+0000 7f6423e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564e9e7c08c0 0x564e9e7cf5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:12.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.267+0000 7f6423e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564e9e7c08c0 0x564e9e7cf5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:59724/0 (socket says 172.21.15.39:59724) 2024-03-18T16:16:12.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.268+0000 7f6428a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564e9e7c6420 0x564e9e7d0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:12.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.268+0000 7f6428a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564e9e7c6420 0x564e9e7d0b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:53786/0 (socket says 172.21.15.39:53786) 2024-03-18T16:16:12.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.268+0000 7f642d668740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f64241c30a0 con 0x564e9e7c12b0 2024-03-18T16:16:12.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.268+0000 7f6423e00640 1 -- 172.21.15.39:0/833526981 learned_addr learned my addr 172.21.15.39:0/833526981 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:16:12.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.268+0000 7f6423400640 1 --2- 172.21.15.39:0/833526981 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564e9e7c12b0 0x564e9e7cfb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:12.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.268+0000 7f642d668740 1 -- 172.21.15.39:0/833526981 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f64241c29d0 con 0x564e9e7c6420 2024-03-18T16:16:12.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.268+0000 7f642d668740 1 -- 172.21.15.39:0/833526981 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f64241bfd60 con 0x564e9e7c08c0 2024-03-18T16:16:12.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.268+0000 7f642d668740 10 client.0 _ll_register_callbacks cb 0x7f64140046c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:16:12.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.269+0000 7f6423e00640 1 -- 172.21.15.39:0/833526981 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564e9e7c6420 msgr2=0x564e9e7d0b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:12.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.269+0000 7f6423e00640 1 --2- 172.21.15.39:0/833526981 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564e9e7c6420 0x564e9e7d0b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.269+0000 7f6423e00640 1 -- 172.21.15.39:0/833526981 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564e9e7c12b0 msgr2=0x564e9e7cfb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:12.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.269+0000 7f6423e00640 1 --2- 172.21.15.39:0/833526981 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564e9e7c12b0 0x564e9e7cfb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.269+0000 7f6423e00640 1 -- 172.21.15.39:0/833526981 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e9e7d1120 con 0x564e9e7c08c0 2024-03-18T16:16:12.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.269+0000 7f6423400640 1 --2- 172.21.15.39:0/833526981 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564e9e7c12b0 0x564e9e7cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:16:12.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.269+0000 7f6423e00640 1 --2- 172.21.15.39:0/833526981 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564e9e7c08c0 0x564e9e7cf5e0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f64140058c0 tx=0x7f641409b670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d0b31a27fa2b6e20 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:12.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.269+0000 7f640d600640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:16:12.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.269+0000 7f640d600640 1 -- 172.21.15.39:0/833526981 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64140a4040 con 0x564e9e7c08c0 2024-03-18T16:16:12.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.269+0000 7f640d600640 1 -- 172.21.15.39:0/833526981 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f64140a3db0 con 0x564e9e7c08c0 2024-03-18T16:16:12.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.269+0000 7f642d668740 1 -- 172.21.15.39:0/833526981 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f641c01e040 con 0x564e9e7c08c0 2024-03-18T16:16:12.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.270+0000 7f640d600640 1 -- 172.21.15.39:0/833526981 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f641409b200 con 0x564e9e7c08c0 2024-03-18T16:16:12.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.270+0000 7f642d668740 10 client.26818 fetch_fsmap learned FSMap version 44 2024-03-18T16:16:12.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.270+0000 7f642d668740 1 -- 172.21.15.39:0/833526981 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x564e9e83cbe0 con 0x564e9e7c08c0 2024-03-18T16:16:12.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.270+0000 7f640d600640 1 -- 172.21.15.39:0/833526981 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f641c01e040 con 0x564e9e7c08c0 2024-03-18T16:16:12.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.270+0000 7f640d600640 1 -- 172.21.15.39:0/833526981 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7f641409a7f0 con 0x564e9e7c08c0 2024-03-18T16:16:12.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.271+0000 7f642d668740 10 client.26818 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:16:12.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.271+0000 7f642d668740 10 client.26818 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-18T16:16:12.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.271+0000 7f642d668740 20 client.26818 populate_metadata read hostname 'smithi039' 2024-03-18T16:16:12.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.271+0000 7f642d668740 1 --2- 172.21.15.39:0/833526981 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x564e9e83d3a0 0x564e9e83f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:12.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.271+0000 7f642d668740 4 client.26818 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:16:12.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.271+0000 7f642d668740 1 -- 172.21.15.39:0/833526981 --> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564e9e72cce0 con 0x564e9e83d3a0 2024-03-18T16:16:12.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.271+0000 7f6423400640 1 --2- 172.21.15.39:0/833526981 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x564e9e83d3a0 0x564e9e83f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:12.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.272+0000 7f640d600640 10 client.26818 ms_handle_connect on v2:172.21.15.194:6836/1301464246 2024-03-18T16:16:12.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.272+0000 7f6423400640 1 --2- 172.21.15.39:0/833526981 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x564e9e83d3a0 0x564e9e83f7c0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=8bcc8b4e146b897d server_cookie=b76bfd7bc7f0832e in_seq=0 out_seq=0 2024-03-18T16:16:12.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.273+0000 7f640d600640 1 -- 172.21.15.39:0/833526981 <== mds.3 v2:172.21.15.194:6836/1301464246 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564e9e72cce0 con 0x564e9e83d3a0 2024-03-18T16:16:12.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.273+0000 7f640d600640 10 client.26818 handle_command_reply: tid=0 2024-03-18T16:16:12.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.273+0000 7f642d668740 1 client.26818 shutdown 2024-03-18T16:16:12.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.273+0000 7f642d668740 1 -- 172.21.15.39:0/833526981 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564e9e7c08c0 msgr2=0x564e9e7cf5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:12.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.273+0000 7f642d668740 1 --2- 172.21.15.39:0/833526981 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564e9e7c08c0 0x564e9e7cf5e0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f64140058c0 tx=0x7f641409b670 comp rx=0 tx=0).stop 2024-03-18T16:16:12.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.274+0000 7f642d668740 1 -- 172.21.15.39:0/833526981 shutdown_connections 2024-03-18T16:16:12.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.274+0000 7f642d668740 1 --2- 172.21.15.39:0/833526981 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x564e9e83d3a0 0x564e9e83f7c0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.274+0000 7f642d668740 1 --2- 172.21.15.39:0/833526981 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564e9e7c6420 0x564e9e7d0b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.274+0000 7f642d668740 1 --2- 172.21.15.39:0/833526981 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564e9e7c12b0 0x564e9e7cfb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.274+0000 7f642d668740 1 --2- 172.21.15.39:0/833526981 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564e9e7c08c0 0x564e9e7cf5e0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.274+0000 7f642d668740 1 -- 172.21.15.39:0/833526981 >> 172.21.15.39:0/833526981 conn(0x564e9e7bbe30 msgr2=0x564e9e7ca590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:12.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.274+0000 7f640d600640 0 client.26818 ms_handle_reset on v2:172.21.15.194:6836/1301464246 2024-03-18T16:16:12.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.275+0000 7f642d668740 1 -- 172.21.15.39:0/833526981 shutdown_connections 2024-03-18T16:16:12.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.275+0000 7f642d668740 1 -- 172.21.15.39:0/833526981 wait complete. 2024-03-18T16:16:12.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.275+0000 7f642d668740 20 client.26818 trim_cache size 0 max 16384 2024-03-18T16:16:12.279 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:16:12.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.276+0000 7f642aa00640 1 -- 172.21.15.39:0/1076333800 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f64080a28d0 msgr2=0x7f64080a4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:12.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.276+0000 7f642aa00640 1 --2- 172.21.15.39:0/1076333800 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f64080a28d0 0x7f64080a4d90 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f6410037040 tx=0x7f6410009490 comp rx=0 tx=0).stop 2024-03-18T16:16:12.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.276+0000 7f642aa00640 1 -- 172.21.15.39:0/1076333800 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f64241039f0 msgr2=0x7f64241a6950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:12.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.276+0000 7f642aa00640 1 --2- 172.21.15.39:0/1076333800 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f64241039f0 0x7f64241a6950 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f641400e9c0 tx=0x7f641400ee80 comp rx=0 tx=0).stop 2024-03-18T16:16:12.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.276+0000 7f642aa00640 1 -- 172.21.15.39:0/1076333800 shutdown_connections 2024-03-18T16:16:12.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.277+0000 7f642aa00640 1 --2- 172.21.15.39:0/1076333800 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f64241055a0 0x7f64241adf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.277+0000 7f642aa00640 1 --2- 172.21.15.39:0/1076333800 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f64080a28d0 0x7f64080a4d90 secure :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f6410037040 tx=0x7f6410009490 comp rx=0 tx=0).stop 2024-03-18T16:16:12.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.277+0000 7f642aa00640 1 --2- 172.21.15.39:0/1076333800 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6424104be0 0x7f64241a6e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.277+0000 7f642aa00640 1 --2- 172.21.15.39:0/1076333800 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f64241039f0 0x7f64241a6950 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.277+0000 7f642aa00640 1 -- 172.21.15.39:0/1076333800 >> 172.21.15.39:0/1076333800 conn(0x7f64240fefa0 msgr2=0x7f6424077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:12.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.278+0000 7f642aa00640 1 -- 172.21.15.39:0/1076333800 shutdown_connections 2024-03-18T16:16:12.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.278+0000 7f642aa00640 1 -- 172.21.15.39:0/1076333800 wait complete. 2024-03-18T16:16:12.305 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-18T16:16:12.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.559+0000 7f217c000640 1 -- 172.21.15.39:0/512821976 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2174104bf0 msgr2=0x7f2174105070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:12.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.559+0000 7f217c000640 1 --2- 172.21.15.39:0/512821976 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2174104bf0 0x7f2174105070 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f2160009900 tx=0x7f216002bc30 comp rx=0 tx=0).stop 2024-03-18T16:16:12.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.559+0000 7f217c000640 1 -- 172.21.15.39:0/512821976 shutdown_connections 2024-03-18T16:16:12.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.559+0000 7f217c000640 1 --2- 172.21.15.39:0/512821976 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2174104bf0 0x7f2174105070 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.559+0000 7f217c000640 1 --2- 172.21.15.39:0/512821976 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f21741055b0 0x7f217410be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.559+0000 7f217c000640 1 --2- 172.21.15.39:0/512821976 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2174103a00 0x7f2174103e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.559+0000 7f217c000640 1 -- 172.21.15.39:0/512821976 >> 172.21.15.39:0/512821976 conn(0x7f21740fef90 msgr2=0x7f21741013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:12.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.560+0000 7f217c000640 1 -- 172.21.15.39:0/512821976 shutdown_connections 2024-03-18T16:16:12.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.560+0000 7f217c000640 1 -- 172.21.15.39:0/512821976 wait complete. 2024-03-18T16:16:12.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.561+0000 7f217c000640 1 Processor -- start 2024-03-18T16:16:12.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.562+0000 7f217c000640 1 -- start start 2024-03-18T16:16:12.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.562+0000 7f217c000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2174103a00 0x7f2174070170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:12.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.562+0000 7f217c000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2174104bf0 0x7f21740706b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:12.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.562+0000 7f217c000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f21741055b0 0x7f21741a9dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:12.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.562+0000 7f217c000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f217410f6b0 con 0x7f21741055b0 2024-03-18T16:16:12.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.562+0000 7f217c000640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f217410f530 con 0x7f2174104bf0 2024-03-18T16:16:12.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.562+0000 7f217c000640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f217410f830 con 0x7f2174103a00 2024-03-18T16:16:12.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.562+0000 7f2179600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2174103a00 0x7f2174070170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:12.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.562+0000 7f2179600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2174103a00 0x7f2174070170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:59740/0 (socket says 172.21.15.39:59740) 2024-03-18T16:16:12.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.562+0000 7f2179600640 1 -- 172.21.15.39:0/3395825360 learned_addr learned my addr 172.21.15.39:0/3395825360 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:16:12.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.563+0000 7f217a000640 1 --2- 172.21.15.39:0/3395825360 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f21741055b0 0x7f21741a9dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:12.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.563+0000 7f2179600640 1 -- 172.21.15.39:0/3395825360 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2174104bf0 msgr2=0x7f21740706b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T16:16:12.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.563+0000 7f2179600640 1 --2- 172.21.15.39:0/3395825360 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2174104bf0 0x7f21740706b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.563+0000 7f2179600640 1 -- 172.21.15.39:0/3395825360 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f21741055b0 msgr2=0x7f21741a9dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:12.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.563+0000 7f2179600640 1 --2- 172.21.15.39:0/3395825360 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f21741055b0 0x7f21741a9dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.563+0000 7f2179600640 1 -- 172.21.15.39:0/3395825360 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21741aa300 con 0x7f2174103a00 2024-03-18T16:16:12.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.563+0000 7f217a000640 1 --2- 172.21.15.39:0/3395825360 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f21741055b0 0x7f21741a9dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:16:12.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.563+0000 7f2179600640 1 --2- 172.21.15.39:0/3395825360 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2174103a00 0x7f2174070170 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f216800e9c0 tx=0x7f216800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:12.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.564+0000 7f2172000640 1 -- 172.21.15.39:0/3395825360 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f216800cd80 con 0x7f2174103a00 2024-03-18T16:16:12.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.564+0000 7f2172000640 1 -- 172.21.15.39:0/3395825360 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f21680035f0 con 0x7f2174103a00 2024-03-18T16:16:12.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.564+0000 7f2172000640 1 -- 172.21.15.39:0/3395825360 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2168010690 con 0x7f2174103a00 2024-03-18T16:16:12.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.564+0000 7f217c000640 1 -- 172.21.15.39:0/3395825360 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21741aa4d0 con 0x7f2174103a00 2024-03-18T16:16:12.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.564+0000 7f217c000640 1 -- 172.21.15.39:0/3395825360 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21741aa940 con 0x7f2174103a00 2024-03-18T16:16:12.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.569+0000 7f217ed64740 1 Processor -- start 2024-03-18T16:16:12.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.569+0000 7f217ed64740 1 -- start start 2024-03-18T16:16:12.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.570+0000 7f217ed64740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc1c1800d0 0x55cc1c1804f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:12.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.570+0000 7f217ed64740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cc1c180ac0 0x55cc1c1855c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:12.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.570+0000 7f217ed64740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cc1c185c30 0x55cc1c1880c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:12.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.570+0000 7f217ed64740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55cc1c13f3b0 con 0x55cc1c185c30 2024-03-18T16:16:12.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.570+0000 7f217ed64740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55cc1c0dc450 con 0x55cc1c1800d0 2024-03-18T16:16:12.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.570+0000 7f217ed64740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55cc1c084cb0 con 0x55cc1c180ac0 2024-03-18T16:16:12.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.571+0000 7f2178c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cc1c185c30 0x55cc1c1880c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:12.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.571+0000 7f2178c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cc1c185c30 0x55cc1c1880c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:46476/0 (socket says 172.21.15.39:46476) 2024-03-18T16:16:12.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.571+0000 7f2178c00640 1 -- 172.21.15.39:0/3346163585 learned_addr learned my addr 172.21.15.39:0/3346163585 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:16:12.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.572+0000 7f2178c00640 1 -- 172.21.15.39:0/3346163585 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cc1c180ac0 msgr2=0x55cc1c1855c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:16:12.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.572+0000 7f2178c00640 1 --2- 172.21.15.39:0/3346163585 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cc1c180ac0 0x55cc1c1855c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.572+0000 7f2178c00640 1 -- 172.21.15.39:0/3346163585 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc1c1800d0 msgr2=0x55cc1c1804f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T16:16:12.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.572+0000 7f2178c00640 1 --2- 172.21.15.39:0/3346163585 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc1c1800d0 0x55cc1c1804f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.572+0000 7f2178c00640 1 -- 172.21.15.39:0/3346163585 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cc1c0a61f0 con 0x55cc1c185c30 2024-03-18T16:16:12.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.572+0000 7f2178c00640 1 --2- 172.21.15.39:0/3346163585 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cc1c185c30 0x55cc1c1880c0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f2160000c00 tx=0x7f2160005760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d44083465dd67016 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:12.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.573+0000 7f2172000640 1 -- 172.21.15.39:0/3395825360 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f21680037a0 con 0x7f2174103a00 2024-03-18T16:16:12.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.573+0000 7f2166600640 1 -- 172.21.15.39:0/3346163585 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2160042020 con 0x55cc1c185c30 2024-03-18T16:16:12.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.573+0000 7f2166600640 1 -- 172.21.15.39:0/3346163585 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2160009520 con 0x55cc1c185c30 2024-03-18T16:16:12.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.573+0000 7f2166600640 1 -- 172.21.15.39:0/3346163585 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2160003c20 con 0x55cc1c185c30 2024-03-18T16:16:12.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.574+0000 7f217ed64740 1 -- 172.21.15.39:0/3346163585 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cc1c185c30 msgr2=0x55cc1c1880c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:12.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.574+0000 7f217ed64740 1 --2- 172.21.15.39:0/3346163585 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cc1c185c30 0x55cc1c1880c0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f2160000c00 tx=0x7f2160005760 comp rx=0 tx=0).stop 2024-03-18T16:16:12.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.575+0000 7f217ed64740 1 -- 172.21.15.39:0/3346163585 shutdown_connections 2024-03-18T16:16:12.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.575+0000 7f217ed64740 1 --2- 172.21.15.39:0/3346163585 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cc1c185c30 0x55cc1c1880c0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.575+0000 7f217ed64740 1 --2- 172.21.15.39:0/3346163585 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cc1c180ac0 0x55cc1c1855c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.575+0000 7f217ed64740 1 --2- 172.21.15.39:0/3346163585 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc1c1800d0 0x55cc1c1804f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.575+0000 7f217ed64740 1 -- 172.21.15.39:0/3346163585 >> 172.21.15.39:0/3346163585 conn(0x55cc1c17b640 msgr2=0x55cc1c17ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:12.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.575+0000 7f217ed64740 1 -- 172.21.15.39:0/3346163585 shutdown_connections 2024-03-18T16:16:12.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.575+0000 7f217ed64740 1 -- 172.21.15.39:0/3346163585 wait complete. 2024-03-18T16:16:12.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.575+0000 7f2172000640 1 --2- 172.21.15.39:0/3395825360 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f215c0a2890 0x7f215c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:12.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.575+0000 7f2172000640 1 -- 172.21.15.39:0/3395825360 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f21680cf270 con 0x7f2174103a00 2024-03-18T16:16:12.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.576+0000 7f217ed64740 1 -- start start 2024-03-18T16:16:12.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.576+0000 7f217ed64740 1 Processor -- start 2024-03-18T16:16:12.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.576+0000 7f2178c00640 1 --2- 172.21.15.39:0/3395825360 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f215c0a2890 0x7f215c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:12.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.577+0000 7f217ed64740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc1c1800d0 0x55cc1c18a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:12.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.577+0000 7f217ed64740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cc1c180ac0 0x55cc1c18a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:12.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.577+0000 7f217ed64740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cc1c185c30 0x55cc1c191920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:12.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.577+0000 7f217ed64740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55cc1c098110 con 0x55cc1c180ac0 2024-03-18T16:16:12.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.577+0000 7f217ed64740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55cc1c0d84f0 con 0x55cc1c1800d0 2024-03-18T16:16:12.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.577+0000 7f217ed64740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55cc1c188b80 con 0x55cc1c185c30 2024-03-18T16:16:12.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.577+0000 7f217a000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cc1c185c30 0x55cc1c191920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:12.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.577+0000 7f217a000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cc1c185c30 0x55cc1c191920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:59746/0 (socket says 172.21.15.39:59746) 2024-03-18T16:16:12.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.577+0000 7f217a000640 1 -- 172.21.15.39:0/3234282551 learned_addr learned my addr 172.21.15.39:0/3234282551 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:16:12.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.577+0000 7f217ed64740 10 client.0 _ll_register_callbacks cb 0x55cc1c085b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:16:12.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.577+0000 7f2179600640 1 --2- 172.21.15.39:0/3234282551 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc1c1800d0 0x55cc1c18a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:12.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.578+0000 7f2179600640 1 -- 172.21.15.39:0/3234282551 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cc1c185c30 msgr2=0x55cc1c191920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:12.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.578+0000 7f2179600640 1 --2- 172.21.15.39:0/3234282551 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cc1c185c30 0x55cc1c191920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.578+0000 7f2179600640 1 -- 172.21.15.39:0/3234282551 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cc1c180ac0 msgr2=0x55cc1c18a8e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:16:12.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.578+0000 7f2179600640 1 --2- 172.21.15.39:0/3234282551 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cc1c180ac0 0x55cc1c18a8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.578+0000 7f2179600640 1 -- 172.21.15.39:0/3234282551 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cc1bdf03f0 con 0x55cc1c1800d0 2024-03-18T16:16:12.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.578+0000 7f2179600640 1 --2- 172.21.15.39:0/3234282551 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc1c1800d0 0x55cc1c18a3a0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x55cc1c181530 tx=0x7f216800fe40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b1e43506b31c6b0a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:12.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.578+0000 7f215aa00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:16:12.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.578+0000 7f215aa00640 1 -- 172.21.15.39:0/3234282551 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f216809b880 con 0x55cc1c1800d0 2024-03-18T16:16:12.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.578+0000 7f217ed64740 1 -- 172.21.15.39:0/3234282551 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2160009520 con 0x55cc1c1800d0 2024-03-18T16:16:12.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.578+0000 7f215aa00640 1 -- 172.21.15.39:0/3234282551 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f2168097ef0 con 0x55cc1c1800d0 2024-03-18T16:16:12.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.579+0000 7f215aa00640 1 -- 172.21.15.39:0/3234282551 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f216809f9d0 con 0x55cc1c1800d0 2024-03-18T16:16:12.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.579+0000 7f217ed64740 10 client.26858 fetch_fsmap learned FSMap version 44 2024-03-18T16:16:12.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.579+0000 7f217ed64740 1 -- 172.21.15.39:0/3234282551 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x55cc1beba7d0 con 0x55cc1c1800d0 2024-03-18T16:16:12.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.579+0000 7f215aa00640 1 -- 172.21.15.39:0/3234282551 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2160009520 con 0x55cc1c1800d0 2024-03-18T16:16:12.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.579+0000 7f215aa00640 1 -- 172.21.15.39:0/3234282551 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7f216809ad40 con 0x55cc1c1800d0 2024-03-18T16:16:12.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.579+0000 7f217ed64740 10 client.26858 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:16:12.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.579+0000 7f217ed64740 10 client.26858 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-18T16:16:12.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.579+0000 7f217ed64740 20 client.26858 populate_metadata read hostname 'smithi039' 2024-03-18T16:16:12.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.579+0000 7f217ed64740 1 --2- 172.21.15.39:0/3234282551 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x55cc1c176bc0 0x55cc1c179020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:12.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.579+0000 7f217ed64740 4 client.26858 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:16:12.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.579+0000 7f217ed64740 1 -- 172.21.15.39:0/3234282551 --> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cc1c0e9990 con 0x55cc1c176bc0 2024-03-18T16:16:12.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.580+0000 7f2178c00640 1 --2- 172.21.15.39:0/3234282551 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x55cc1c176bc0 0x55cc1c179020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:12.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.581+0000 7f2178c00640 1 --2- 172.21.15.39:0/3234282551 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x55cc1c176bc0 0x55cc1c179020 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e3f7a5319f22542 server_cookie=9892aed1a9de66c0 in_seq=0 out_seq=0 2024-03-18T16:16:12.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.581+0000 7f215aa00640 10 client.26858 ms_handle_connect on v2:172.21.15.196:6836/895247251 2024-03-18T16:16:12.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.581+0000 7f2178c00640 1 --2- 172.21.15.39:0/3395825360 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f215c0a2890 0x7f215c0a4d50 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x55cc1c18b880 tx=0x7f216002bc30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:12.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.582+0000 7f215aa00640 1 -- 172.21.15.39:0/3234282551 <== mds.2 v2:172.21.15.196:6836/895247251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55cc1c0e9990 con 0x55cc1c176bc0 2024-03-18T16:16:12.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.582+0000 7f215aa00640 10 client.26858 handle_command_reply: tid=0 2024-03-18T16:16:12.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.582+0000 7f217ed64740 1 client.26858 shutdown 2024-03-18T16:16:12.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.583+0000 7f217ed64740 1 -- 172.21.15.39:0/3234282551 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc1c1800d0 msgr2=0x55cc1c18a3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:12.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.583+0000 7f217ed64740 1 --2- 172.21.15.39:0/3234282551 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc1c1800d0 0x55cc1c18a3a0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x55cc1c181530 tx=0x7f216800fe40 comp rx=0 tx=0).stop 2024-03-18T16:16:12.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.583+0000 7f217ed64740 1 -- 172.21.15.39:0/3234282551 shutdown_connections 2024-03-18T16:16:12.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.583+0000 7f217ed64740 1 --2- 172.21.15.39:0/3234282551 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x55cc1c176bc0 0x55cc1c179020 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.583+0000 7f217ed64740 1 --2- 172.21.15.39:0/3234282551 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cc1c185c30 0x55cc1c191920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.583+0000 7f217ed64740 1 --2- 172.21.15.39:0/3234282551 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cc1c180ac0 0x55cc1c18a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.583+0000 7f217ed64740 1 --2- 172.21.15.39:0/3234282551 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc1c1800d0 0x55cc1c18a3a0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.583+0000 7f217ed64740 1 -- 172.21.15.39:0/3234282551 >> 172.21.15.39:0/3234282551 conn(0x55cc1c17b640 msgr2=0x55cc1c17c890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:12.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.583+0000 7f215aa00640 0 client.26858 ms_handle_reset on v2:172.21.15.196:6836/895247251 2024-03-18T16:16:12.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:12 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/4291310410' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:16:12.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.584+0000 7f217ed64740 1 -- 172.21.15.39:0/3234282551 shutdown_connections 2024-03-18T16:16:12.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.584+0000 7f217ed64740 1 -- 172.21.15.39:0/3234282551 wait complete. 2024-03-18T16:16:12.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.584+0000 7f217ed64740 20 client.26858 trim_cache size 0 max 16384 2024-03-18T16:16:12.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.604+0000 7f217ed64740 1 Processor -- start 2024-03-18T16:16:12.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.605+0000 7f217ed64740 1 -- start start 2024-03-18T16:16:12.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.605+0000 7f217ed64740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cc1c1800d0 0x55cc1c18a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:12.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.605+0000 7f217ed64740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cc1c180ac0 0x55cc1c171960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:12.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.605+0000 7f217ed64740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc1c185c30 0x55cc1c171ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:12.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.605+0000 7f217ed64740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f21741b0960 con 0x55cc1c180ac0 2024-03-18T16:16:12.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.605+0000 7f217ed64740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f21741b24a0 con 0x55cc1c185c30 2024-03-18T16:16:12.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.605+0000 7f217ed64740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f21741b1030 con 0x55cc1c1800d0 2024-03-18T16:16:12.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.605+0000 7f2179600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cc1c1800d0 0x55cc1c18a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:12.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.605+0000 7f2179600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cc1c1800d0 0x55cc1c18a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:59750/0 (socket says 172.21.15.39:59750) 2024-03-18T16:16:12.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.605+0000 7f2179600640 1 -- 172.21.15.39:0/3760916076 learned_addr learned my addr 172.21.15.39:0/3760916076 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:16:12.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.606+0000 7f217a000640 1 --2- 172.21.15.39:0/3760916076 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc1c185c30 0x55cc1c171ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:12.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.606+0000 7f2179600640 1 -- 172.21.15.39:0/3760916076 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc1c185c30 msgr2=0x55cc1c171ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:12.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.606+0000 7f2179600640 1 --2- 172.21.15.39:0/3760916076 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc1c185c30 0x55cc1c171ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.606+0000 7f2179600640 1 -- 172.21.15.39:0/3760916076 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cc1c180ac0 msgr2=0x55cc1c171960 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:16:12.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.607+0000 7f2178c00640 1 --2- 172.21.15.39:0/3760916076 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cc1c180ac0 0x55cc1c171960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:12.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.607+0000 7f2179600640 1 --2- 172.21.15.39:0/3760916076 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cc1c180ac0 0x55cc1c171960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.607+0000 7f2179600640 1 -- 172.21.15.39:0/3760916076 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cc1c16ec10 con 0x55cc1c1800d0 2024-03-18T16:16:12.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.607+0000 7f2179600640 1 --2- 172.21.15.39:0/3760916076 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cc1c1800d0 0x55cc1c18a6c0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f2168097ce0 tx=0x7f2168099d90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1aa6d38f127bb153 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:12.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.607+0000 7f2165c00640 1 -- 172.21.15.39:0/3760916076 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f216809b650 con 0x55cc1c1800d0 2024-03-18T16:16:12.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.607+0000 7f2165c00640 1 -- 172.21.15.39:0/3760916076 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f216809b880 con 0x55cc1c1800d0 2024-03-18T16:16:12.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.607+0000 7f2165c00640 1 -- 172.21.15.39:0/3760916076 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f216809cbb0 con 0x55cc1c1800d0 2024-03-18T16:16:12.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.608+0000 7f2178c00640 1 --2- 172.21.15.39:0/3760916076 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cc1c180ac0 0x55cc1c171960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:16:12.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.608+0000 7f217ed64740 1 -- 172.21.15.39:0/3760916076 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cc1c1800d0 msgr2=0x55cc1c18a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:12.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.608+0000 7f217ed64740 1 --2- 172.21.15.39:0/3760916076 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cc1c1800d0 0x55cc1c18a6c0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f2168097ce0 tx=0x7f2168099d90 comp rx=0 tx=0).stop 2024-03-18T16:16:12.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.608+0000 7f217ed64740 1 -- 172.21.15.39:0/3760916076 shutdown_connections 2024-03-18T16:16:12.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.608+0000 7f217ed64740 1 --2- 172.21.15.39:0/3760916076 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc1c185c30 0x55cc1c171ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.608+0000 7f217ed64740 1 --2- 172.21.15.39:0/3760916076 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cc1c180ac0 0x55cc1c171960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.608+0000 7f217ed64740 1 --2- 172.21.15.39:0/3760916076 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cc1c1800d0 0x55cc1c18a6c0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.608+0000 7f217ed64740 1 -- 172.21.15.39:0/3760916076 >> 172.21.15.39:0/3760916076 conn(0x55cc1c17b640 msgr2=0x55cc1c189fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:12.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.609+0000 7f217ed64740 1 -- 172.21.15.39:0/3760916076 shutdown_connections 2024-03-18T16:16:12.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.609+0000 7f217ed64740 1 -- 172.21.15.39:0/3760916076 wait complete. 2024-03-18T16:16:12.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.609+0000 7f217ed64740 1 -- start start 2024-03-18T16:16:12.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.610+0000 7f217ed64740 1 Processor -- start 2024-03-18T16:16:12.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.611+0000 7f217ed64740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cc1c1800d0 0x55cc1c190880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:12.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.611+0000 7f217ed64740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc1c180ac0 0x55cc1c18ef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:12.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.611+0000 7f217ed64740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cc1c185c30 0x55cc1c18f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:12.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.611+0000 7f217ed64740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f21741b1dd0 con 0x55cc1c185c30 2024-03-18T16:16:12.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.611+0000 7f217ed64740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f21741b1700 con 0x55cc1c180ac0 2024-03-18T16:16:12.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.611+0000 7f217ed64740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f21741ae940 con 0x55cc1c1800d0 2024-03-18T16:16:12.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.611+0000 7f217ed64740 10 client.0 _ll_register_callbacks cb 0x7f2168003e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:16:12.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.611+0000 7f2179600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cc1c1800d0 0x55cc1c190880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:12.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.611+0000 7f2178c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc1c180ac0 0x55cc1c18ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:12.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.611+0000 7f2178c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc1c180ac0 0x55cc1c18ef00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:53848/0 (socket says 172.21.15.39:53848) 2024-03-18T16:16:12.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.611+0000 7f2178c00640 1 -- 172.21.15.39:0/472801176 learned_addr learned my addr 172.21.15.39:0/472801176 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:16:12.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.611+0000 7f217a000640 1 --2- 172.21.15.39:0/472801176 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cc1c185c30 0x55cc1c18f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:12.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.612+0000 7f2179600640 1 -- 172.21.15.39:0/472801176 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc1c180ac0 msgr2=0x55cc1c18ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:12.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.612+0000 7f2179600640 1 --2- 172.21.15.39:0/472801176 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc1c180ac0 0x55cc1c18ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.612+0000 7f2179600640 1 -- 172.21.15.39:0/472801176 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cc1c185c30 msgr2=0x55cc1c18f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:12.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.612+0000 7f2179600640 1 --2- 172.21.15.39:0/472801176 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cc1c185c30 0x55cc1c18f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.612+0000 7f2179600640 1 -- 172.21.15.39:0/472801176 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cc1c17fd10 con 0x55cc1c1800d0 2024-03-18T16:16:12.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.612+0000 7f2178c00640 1 --2- 172.21.15.39:0/472801176 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc1c180ac0 0x55cc1c18ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:16:12.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.614+0000 7f2179600640 1 --2- 172.21.15.39:0/472801176 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cc1c1800d0 0x55cc1c190880 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f216809b580 tx=0x7f2168097b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=156b3a900ff6f53f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:12.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.614+0000 7f215aa00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:16:12.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.615+0000 7f215aa00640 1 -- 172.21.15.39:0/472801176 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f216809b650 con 0x55cc1c1800d0 2024-03-18T16:16:12.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.615+0000 7f215aa00640 1 -- 172.21.15.39:0/472801176 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f216809cdb0 con 0x55cc1c1800d0 2024-03-18T16:16:12.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.615+0000 7f217ed64740 1 -- 172.21.15.39:0/472801176 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f216809b880 con 0x55cc1c1800d0 2024-03-18T16:16:12.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.616+0000 7f215aa00640 1 -- 172.21.15.39:0/472801176 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f2168080050 con 0x55cc1c1800d0 2024-03-18T16:16:12.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.616+0000 7f217ed64740 10 client.26833 fetch_fsmap learned FSMap version 44 2024-03-18T16:16:12.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.616+0000 7f217ed64740 1 -- 172.21.15.39:0/472801176 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x55cc1c1f3ec0 con 0x55cc1c1800d0 2024-03-18T16:16:12.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.616+0000 7f215aa00640 1 -- 172.21.15.39:0/472801176 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f216809b880 con 0x55cc1c1800d0 2024-03-18T16:16:12.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.617+0000 7f215aa00640 1 -- 172.21.15.39:0/472801176 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7f21680bb3f0 con 0x55cc1c1800d0 2024-03-18T16:16:12.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.617+0000 7f217ed64740 10 client.26833 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:16:12.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.617+0000 7f217ed64740 10 client.26833 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-18T16:16:12.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.617+0000 7f217ed64740 20 client.26833 populate_metadata read hostname 'smithi039' 2024-03-18T16:16:12.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.617+0000 7f217ed64740 1 --2- 172.21.15.39:0/472801176 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x55cc1c1f46b0 0x55cc1c1f6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:12.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.617+0000 7f217ed64740 4 client.26833 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:16:12.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.617+0000 7f217ed64740 1 -- 172.21.15.39:0/472801176 --> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55cc1c0e3ce0 con 0x55cc1c1f46b0 2024-03-18T16:16:12.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.617+0000 7f2178c00640 1 --2- 172.21.15.39:0/472801176 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x55cc1c1f46b0 0x55cc1c1f6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:12.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.618+0000 7f2178c00640 1 --2- 172.21.15.39:0/472801176 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x55cc1c1f46b0 0x55cc1c1f6ad0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=61a5f333a56d53cb server_cookie=dfb517d1558c8d04 in_seq=0 out_seq=0 2024-03-18T16:16:12.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.618+0000 7f215aa00640 10 client.26833 ms_handle_connect on v2:172.21.15.196:6836/895247251 2024-03-18T16:16:12.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.618+0000 7f215aa00640 1 -- 172.21.15.39:0/472801176 <== mds.2 v2:172.21.15.196:6836/895247251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55cc1c0e3ce0 con 0x55cc1c1f46b0 2024-03-18T16:16:12.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.618+0000 7f215aa00640 10 client.26833 handle_command_reply: tid=0 2024-03-18T16:16:12.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.618+0000 7f217ed64740 1 client.26833 shutdown 2024-03-18T16:16:12.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.619+0000 7f217ed64740 1 -- 172.21.15.39:0/472801176 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cc1c1800d0 msgr2=0x55cc1c190880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:12.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.619+0000 7f217ed64740 1 --2- 172.21.15.39:0/472801176 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cc1c1800d0 0x55cc1c190880 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f216809b580 tx=0x7f2168097b20 comp rx=0 tx=0).stop 2024-03-18T16:16:12.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.620+0000 7f217ed64740 1 -- 172.21.15.39:0/472801176 shutdown_connections 2024-03-18T16:16:12.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.620+0000 7f217ed64740 1 --2- 172.21.15.39:0/472801176 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cc1c180ac0 0x55cc1c18ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.620+0000 7f217ed64740 1 --2- 172.21.15.39:0/472801176 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x55cc1c1f46b0 0x55cc1c1f6ad0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.620+0000 7f217ed64740 1 --2- 172.21.15.39:0/472801176 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55cc1c185c30 0x55cc1c18f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.620+0000 7f217ed64740 1 --2- 172.21.15.39:0/472801176 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55cc1c1800d0 0x55cc1c190880 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.620+0000 7f215aa00640 0 client.26833 ms_handle_reset on v2:172.21.15.196:6836/895247251 2024-03-18T16:16:12.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.620+0000 7f217ed64740 1 -- 172.21.15.39:0/472801176 >> 172.21.15.39:0/472801176 conn(0x55cc1c17b640 msgr2=0x55cc1c171960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:12.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.620+0000 7f217ed64740 1 -- 172.21.15.39:0/472801176 shutdown_connections 2024-03-18T16:16:12.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.621+0000 7f217ed64740 1 -- 172.21.15.39:0/472801176 wait complete. 2024-03-18T16:16:12.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.621+0000 7f217ed64740 20 client.26833 trim_cache size 0 max 16384 2024-03-18T16:16:12.625 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:16:12.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.622+0000 7f217c000640 1 -- 172.21.15.39:0/3395825360 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f215c0a2890 msgr2=0x7f215c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:12.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.622+0000 7f217c000640 1 --2- 172.21.15.39:0/3395825360 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f215c0a2890 0x7f215c0a4d50 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x55cc1c18b880 tx=0x7f216002bc30 comp rx=0 tx=0).stop 2024-03-18T16:16:12.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.622+0000 7f217c000640 1 -- 172.21.15.39:0/3395825360 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2174103a00 msgr2=0x7f2174070170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:12.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.622+0000 7f217c000640 1 --2- 172.21.15.39:0/3395825360 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2174103a00 0x7f2174070170 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f216800e9c0 tx=0x7f216800ee80 comp rx=0 tx=0).stop 2024-03-18T16:16:12.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.622+0000 7f217c000640 1 -- 172.21.15.39:0/3395825360 shutdown_connections 2024-03-18T16:16:12.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.622+0000 7f217c000640 1 --2- 172.21.15.39:0/3395825360 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2174104bf0 0x7f21740706b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.622+0000 7f217c000640 1 --2- 172.21.15.39:0/3395825360 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f215c0a2890 0x7f215c0a4d50 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.622+0000 7f217c000640 1 --2- 172.21.15.39:0/3395825360 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f21741055b0 0x7f21741a9dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.622+0000 7f217c000640 1 --2- 172.21.15.39:0/3395825360 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2174103a00 0x7f2174070170 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.622+0000 7f217c000640 1 -- 172.21.15.39:0/3395825360 >> 172.21.15.39:0/3395825360 conn(0x7f21740fef90 msgr2=0x7f2174107ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:12.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.623+0000 7f217c000640 1 -- 172.21.15.39:0/3395825360 shutdown_connections 2024-03-18T16:16:12.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.623+0000 7f217c000640 1 -- 172.21.15.39:0/3395825360 wait complete. 2024-03-18T16:16:12.649 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-18T16:16:12.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:12 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/4291310410' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:16:12.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:12 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/4291310410' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:16:12.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.912+0000 7fa252200640 1 -- 172.21.15.39:0/3319486426 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa24c0ffe40 msgr2=0x7fa24c100260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:12.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.912+0000 7fa252200640 1 --2- 172.21.15.39:0/3319486426 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa24c0ffe40 0x7fa24c100260 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fa240009900 tx=0x7fa24002bc30 comp rx=0 tx=0).stop 2024-03-18T16:16:12.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.912+0000 7fa252200640 1 -- 172.21.15.39:0/3319486426 shutdown_connections 2024-03-18T16:16:12.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.912+0000 7fa252200640 1 --2- 172.21.15.39:0/3319486426 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa24c101140 0x7fa24c1015c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.912+0000 7fa252200640 1 --2- 172.21.15.39:0/3319486426 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa24c1007a0 0x7fa24c100c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.912+0000 7fa252200640 1 --2- 172.21.15.39:0/3319486426 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa24c0ffe40 0x7fa24c100260 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.912+0000 7fa252200640 1 -- 172.21.15.39:0/3319486426 >> 172.21.15.39:0/3319486426 conn(0x7fa24c0fba80 msgr2=0x7fa24c0fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:12.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.913+0000 7fa252200640 1 -- 172.21.15.39:0/3319486426 shutdown_connections 2024-03-18T16:16:12.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.913+0000 7fa252200640 1 -- 172.21.15.39:0/3319486426 wait complete. 2024-03-18T16:16:12.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.914+0000 7fa252200640 1 Processor -- start 2024-03-18T16:16:12.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.915+0000 7fa252200640 1 -- start start 2024-03-18T16:16:12.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.915+0000 7fa252200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa24c0ffe40 0x7fa24c19ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:12.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.915+0000 7fa252200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa24c1007a0 0x7fa24c19e330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:12.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.915+0000 7fa252200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa24c101140 0x7fa24c1a53a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:12.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.915+0000 7fa252200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fa24c1033b0 con 0x7fa24c1007a0 2024-03-18T16:16:12.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.916+0000 7fa252200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fa24c103230 con 0x7fa24c101140 2024-03-18T16:16:12.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.916+0000 7fa252200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fa24c103530 con 0x7fa24c0ffe40 2024-03-18T16:16:12.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.916+0000 7fa251800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa24c101140 0x7fa24c1a53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:12.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.916+0000 7fa250e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa24c0ffe40 0x7fa24c19ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:12.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.916+0000 7fa250e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa24c0ffe40 0x7fa24c19ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:59762/0 (socket says 172.21.15.39:59762) 2024-03-18T16:16:12.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.916+0000 7fa251800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa24c101140 0x7fa24c1a53a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:53870/0 (socket says 172.21.15.39:53870) 2024-03-18T16:16:12.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.916+0000 7fa250e00640 1 -- 172.21.15.39:0/2470748801 learned_addr learned my addr 172.21.15.39:0/2470748801 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:16:12.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.916+0000 7fa24be00640 1 --2- 172.21.15.39:0/2470748801 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa24c1007a0 0x7fa24c19e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:12.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.916+0000 7fa250e00640 1 -- 172.21.15.39:0/2470748801 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa24c101140 msgr2=0x7fa24c1a53a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:12.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.916+0000 7fa250e00640 1 --2- 172.21.15.39:0/2470748801 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa24c101140 0x7fa24c1a53a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.916+0000 7fa250e00640 1 -- 172.21.15.39:0/2470748801 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa24c1007a0 msgr2=0x7fa24c19e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:12.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.916+0000 7fa250e00640 1 --2- 172.21.15.39:0/2470748801 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa24c1007a0 0x7fa24c19e330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.916+0000 7fa250e00640 1 -- 172.21.15.39:0/2470748801 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa24c10e230 con 0x7fa24c0ffe40 2024-03-18T16:16:12.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.916+0000 7fa251800640 1 --2- 172.21.15.39:0/2470748801 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa24c101140 0x7fa24c1a53a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello state changed while learned_addr, mark_down or replacing must be happened just now 2024-03-18T16:16:12.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.917+0000 7fa250e00640 1 --2- 172.21.15.39:0/2470748801 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa24c0ffe40 0x7fa24c19ddf0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fa24002c240 tx=0x7fa240004e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:12.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.917+0000 7fa249600640 1 -- 172.21.15.39:0/2470748801 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa240043020 con 0x7fa24c0ffe40 2024-03-18T16:16:12.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.917+0000 7fa252200640 1 -- 172.21.15.39:0/2470748801 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa24c10e4c0 con 0x7fa24c0ffe40 2024-03-18T16:16:12.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.918+0000 7fa249600640 1 -- 172.21.15.39:0/2470748801 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa2400054e0 con 0x7fa24c0ffe40 2024-03-18T16:16:12.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.918+0000 7fa249600640 1 -- 172.21.15.39:0/2470748801 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa240003d00 con 0x7fa24c0ffe40 2024-03-18T16:16:12.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.918+0000 7fa252200640 1 -- 172.21.15.39:0/2470748801 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa24c10ea60 con 0x7fa24c0ffe40 2024-03-18T16:16:12.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.922+0000 7fa2565ba740 1 Processor -- start 2024-03-18T16:16:12.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.923+0000 7fa2565ba740 1 -- start start 2024-03-18T16:16:12.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.923+0000 7fa249600640 1 -- 172.21.15.39:0/2470748801 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fa240046020 con 0x7fa24c0ffe40 2024-03-18T16:16:12.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.923+0000 7fa2565ba740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5603cdbd8a90 0x5603cdbd8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:12.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.923+0000 7fa2565ba740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5603cdbd9480 0x5603cdbddf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:12.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.923+0000 7fa2565ba740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5603cdbde5f0 0x5603cdbe0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:12.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.923+0000 7fa2565ba740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5603cdba03b0 con 0x5603cdbd9480 2024-03-18T16:16:12.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.923+0000 7fa2565ba740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5603cdb3d450 con 0x5603cdbd8a90 2024-03-18T16:16:12.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.923+0000 7fa2565ba740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x5603cdae5cb0 con 0x5603cdbde5f0 2024-03-18T16:16:12.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.924+0000 7fa251800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5603cdbd8a90 0x5603cdbd8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:12.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.924+0000 7fa24be00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5603cdbde5f0 0x5603cdbe0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:12.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.924+0000 7fa251800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5603cdbd8a90 0x5603cdbd8eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:53878/0 (socket says 172.21.15.39:53878) 2024-03-18T16:16:12.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.924+0000 7fa251800640 1 -- 172.21.15.39:0/2960862175 learned_addr learned my addr 172.21.15.39:0/2960862175 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:16:12.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.924+0000 7fa24be00640 1 -- 172.21.15.39:0/2960862175 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5603cdbd8a90 msgr2=0x5603cdbd8eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:12.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.924+0000 7fa24be00640 1 --2- 172.21.15.39:0/2960862175 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5603cdbd8a90 0x5603cdbd8eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.924+0000 7fa24be00640 1 -- 172.21.15.39:0/2960862175 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5603cdbd9480 msgr2=0x5603cdbddf80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:16:12.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.924+0000 7fa24be00640 1 --2- 172.21.15.39:0/2960862175 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5603cdbd9480 0x5603cdbddf80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.924+0000 7fa24be00640 1 -- 172.21.15.39:0/2960862175 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5603cdb071f0 con 0x5603cdbde5f0 2024-03-18T16:16:12.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.925+0000 7fa24be00640 1 --2- 172.21.15.39:0/2960862175 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5603cdbde5f0 0x5603cdbe0a80 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fa238009b60 tx=0x7fa238010740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ae6631bc3500546c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:12.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.925+0000 7fa249600640 1 --2- 172.21.15.39:0/2470748801 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa2340a2950 0x7fa2340a4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:12.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.925+0000 7fa23e000640 1 -- 172.21.15.39:0/2960862175 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa238009e50 con 0x5603cdbde5f0 2024-03-18T16:16:12.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.925+0000 7fa23e000640 1 -- 172.21.15.39:0/2960862175 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa2380035f0 con 0x5603cdbde5f0 2024-03-18T16:16:12.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.925+0000 7fa23e000640 1 -- 172.21.15.39:0/2960862175 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2380126b0 con 0x5603cdbde5f0 2024-03-18T16:16:12.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.925+0000 7fa249600640 1 -- 172.21.15.39:0/2470748801 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fa2400f1040 con 0x7fa24c0ffe40 2024-03-18T16:16:12.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.926+0000 7fa2565ba740 1 -- 172.21.15.39:0/2960862175 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5603cdbde5f0 msgr2=0x5603cdbe0a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:12.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.926+0000 7fa2565ba740 1 --2- 172.21.15.39:0/2960862175 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5603cdbde5f0 0x5603cdbe0a80 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fa238009b60 tx=0x7fa238010740 comp rx=0 tx=0).stop 2024-03-18T16:16:12.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.927+0000 7fa251800640 1 --2- 172.21.15.39:0/2470748801 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa2340a2950 0x7fa2340a4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:12.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.928+0000 7fa2565ba740 1 -- 172.21.15.39:0/2960862175 shutdown_connections 2024-03-18T16:16:12.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.928+0000 7fa2565ba740 1 --2- 172.21.15.39:0/2960862175 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5603cdbde5f0 0x5603cdbe0a80 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.928+0000 7fa2565ba740 1 --2- 172.21.15.39:0/2960862175 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5603cdbd9480 0x5603cdbddf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.928+0000 7fa2565ba740 1 --2- 172.21.15.39:0/2960862175 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5603cdbd8a90 0x5603cdbd8eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.928+0000 7fa2565ba740 1 -- 172.21.15.39:0/2960862175 >> 172.21.15.39:0/2960862175 conn(0x5603cdbd4000 msgr2=0x5603cdbd4400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:12.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.928+0000 7fa251800640 1 --2- 172.21.15.39:0/2470748801 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa2340a2950 0x7fa2340a4e10 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x5603cdbd9ce0 tx=0x7fa244008730 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:12.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.928+0000 7fa2565ba740 1 -- 172.21.15.39:0/2960862175 shutdown_connections 2024-03-18T16:16:12.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.928+0000 7fa2565ba740 1 -- 172.21.15.39:0/2960862175 wait complete. 2024-03-18T16:16:12.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.929+0000 7fa2565ba740 1 -- start start 2024-03-18T16:16:12.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.929+0000 7fa2565ba740 1 Processor -- start 2024-03-18T16:16:12.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.930+0000 7fa2565ba740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5603cdbd8a90 0x5603cdbe2e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:12.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.930+0000 7fa2565ba740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5603cdbd9480 0x5603cdbe3360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:12.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.930+0000 7fa2565ba740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5603cdbde5f0 0x5603cdbea330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:12.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.930+0000 7fa2565ba740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5603cdaf9110 con 0x5603cdbd8a90 2024-03-18T16:16:12.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.930+0000 7fa2565ba740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5603cdb394f0 con 0x5603cdbd9480 2024-03-18T16:16:12.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.930+0000 7fa2565ba740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x5603cdbe15a0 con 0x5603cdbde5f0 2024-03-18T16:16:12.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.930+0000 7fa2565ba740 10 client.0 _ll_register_callbacks cb 0x5603cdae6b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:16:12.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.931+0000 7fa24be00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5603cdbd9480 0x5603cdbe3360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:12.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.931+0000 7fa24be00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5603cdbd9480 0x5603cdbe3360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:53884/0 (socket says 172.21.15.39:53884) 2024-03-18T16:16:12.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.931+0000 7fa24be00640 1 -- 172.21.15.39:0/868234755 learned_addr learned my addr 172.21.15.39:0/868234755 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:16:12.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.931+0000 7fa251800640 1 --2- 172.21.15.39:0/868234755 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5603cdbde5f0 0x5603cdbea330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:12.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.931+0000 7fa250e00640 1 --2- 172.21.15.39:0/868234755 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5603cdbd8a90 0x5603cdbe2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:12.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.931+0000 7fa251800640 1 -- 172.21.15.39:0/868234755 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5603cdbd9480 msgr2=0x5603cdbe3360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:12.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.931+0000 7fa251800640 1 --2- 172.21.15.39:0/868234755 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5603cdbd9480 0x5603cdbe3360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.931+0000 7fa251800640 1 -- 172.21.15.39:0/868234755 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5603cdbd8a90 msgr2=0x5603cdbe2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:12.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.931+0000 7fa251800640 1 --2- 172.21.15.39:0/868234755 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5603cdbd8a90 0x5603cdbe2e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.931+0000 7fa251800640 1 -- 172.21.15.39:0/868234755 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5603cd8513f0 con 0x5603cdbde5f0 2024-03-18T16:16:12.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.932+0000 7fa251800640 1 --2- 172.21.15.39:0/868234755 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5603cdbde5f0 0x5603cdbea330 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fa244010ce0 tx=0x7fa244004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=45ad6278dd836e66 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:12.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.932+0000 7fa232a00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:16:12.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.932+0000 7fa232a00640 1 -- 172.21.15.39:0/868234755 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa24401d620 con 0x5603cdbde5f0 2024-03-18T16:16:12.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.933+0000 7fa232a00640 1 -- 172.21.15.39:0/868234755 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fa24401d7c0 con 0x5603cdbde5f0 2024-03-18T16:16:12.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.933+0000 7fa2565ba740 1 -- 172.21.15.39:0/868234755 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa2380035f0 con 0x5603cdbde5f0 2024-03-18T16:16:12.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.933+0000 7fa232a00640 1 -- 172.21.15.39:0/868234755 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fa24401d9c0 con 0x5603cdbde5f0 2024-03-18T16:16:12.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.933+0000 7fa2565ba740 10 client.26845 fetch_fsmap learned FSMap version 44 2024-03-18T16:16:12.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.933+0000 7fa2565ba740 1 -- 172.21.15.39:0/868234755 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x5603cd91b7d0 con 0x5603cdbde5f0 2024-03-18T16:16:12.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.933+0000 7fa232a00640 1 -- 172.21.15.39:0/868234755 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa2380035f0 con 0x5603cdbde5f0 2024-03-18T16:16:12.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.935+0000 7fa232a00640 1 -- 172.21.15.39:0/868234755 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7fa24401abf0 con 0x5603cdbde5f0 2024-03-18T16:16:12.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.935+0000 7fa2565ba740 10 client.26845 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:16:12.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.935+0000 7fa2565ba740 10 client.26845 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T16:16:12.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.935+0000 7fa2565ba740 20 client.26845 populate_metadata read hostname 'smithi039' 2024-03-18T16:16:12.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.935+0000 7fa2565ba740 1 --2- 172.21.15.39:0/868234755 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x5603cdbeed60 0x5603cdbf11c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:12.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.935+0000 7fa2565ba740 4 client.26845 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:16:12.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.935+0000 7fa2565ba740 1 -- 172.21.15.39:0/868234755 --> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5603cdb4a990 con 0x5603cdbeed60 2024-03-18T16:16:12.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.935+0000 7fa250e00640 1 --2- 172.21.15.39:0/868234755 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x5603cdbeed60 0x5603cdbf11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:12.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.937+0000 7fa250e00640 1 --2- 172.21.15.39:0/868234755 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x5603cdbeed60 0x5603cdbf11c0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7ee4859e1e7085b9 server_cookie=2724aefb1627be4 in_seq=0 out_seq=0 2024-03-18T16:16:12.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.937+0000 7fa232a00640 10 client.26845 ms_handle_connect on v2:172.21.15.196:6838/2001427268 2024-03-18T16:16:12.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.938+0000 7fa232a00640 1 -- 172.21.15.39:0/868234755 <== mds.1 v2:172.21.15.196:6838/2001427268 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5603cdb4a990 con 0x5603cdbeed60 2024-03-18T16:16:12.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.938+0000 7fa232a00640 10 client.26845 handle_command_reply: tid=0 2024-03-18T16:16:12.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.938+0000 7fa2565ba740 1 client.26845 shutdown 2024-03-18T16:16:12.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.938+0000 7fa2565ba740 1 -- 172.21.15.39:0/868234755 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5603cdbde5f0 msgr2=0x5603cdbea330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:12.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.938+0000 7fa2565ba740 1 --2- 172.21.15.39:0/868234755 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5603cdbde5f0 0x5603cdbea330 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fa244010ce0 tx=0x7fa244004a40 comp rx=0 tx=0).stop 2024-03-18T16:16:12.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.938+0000 7fa2565ba740 1 -- 172.21.15.39:0/868234755 shutdown_connections 2024-03-18T16:16:12.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.939+0000 7fa2565ba740 1 --2- 172.21.15.39:0/868234755 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x5603cdbeed60 0x5603cdbf11c0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.939+0000 7fa2565ba740 1 --2- 172.21.15.39:0/868234755 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5603cdbd9480 0x5603cdbe3360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.939+0000 7fa2565ba740 1 --2- 172.21.15.39:0/868234755 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5603cdbde5f0 0x5603cdbea330 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.939+0000 7fa2565ba740 1 --2- 172.21.15.39:0/868234755 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5603cdbd8a90 0x5603cdbe2e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.939+0000 7fa2565ba740 1 -- 172.21.15.39:0/868234755 >> 172.21.15.39:0/868234755 conn(0x5603cdbd4000 msgr2=0x5603cdbd3220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:12.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.939+0000 7fa232a00640 0 client.26845 ms_handle_reset on v2:172.21.15.196:6838/2001427268 2024-03-18T16:16:12.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.940+0000 7fa2565ba740 1 -- 172.21.15.39:0/868234755 shutdown_connections 2024-03-18T16:16:12.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.940+0000 7fa2565ba740 1 -- 172.21.15.39:0/868234755 wait complete. 2024-03-18T16:16:12.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.940+0000 7fa2565ba740 20 client.26845 trim_cache size 0 max 16384 2024-03-18T16:16:12.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.963+0000 7fa2565ba740 1 Processor -- start 2024-03-18T16:16:12.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.963+0000 7fa2565ba740 1 -- start start 2024-03-18T16:16:12.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.964+0000 7fa2565ba740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5603cdbd8a90 0x5603cdbda940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:12.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.964+0000 7fa2565ba740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5603cdbd9480 0x5603cdbdae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:12.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.964+0000 7fa2565ba740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5603cdbde5f0 0x5603cdbea2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:12.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.964+0000 7fa2565ba740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fa24c1b8f60 con 0x5603cdbd9480 2024-03-18T16:16:12.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.964+0000 7fa2565ba740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fa24c1baaa0 con 0x5603cdbde5f0 2024-03-18T16:16:12.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.964+0000 7fa2565ba740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fa24c1b9630 con 0x5603cdbd8a90 2024-03-18T16:16:12.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.964+0000 7fa250e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5603cdbd8a90 0x5603cdbda940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:12.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.964+0000 7fa250e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5603cdbd8a90 0x5603cdbda940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:59786/0 (socket says 172.21.15.39:59786) 2024-03-18T16:16:12.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.964+0000 7fa250e00640 1 -- 172.21.15.39:0/2703829681 learned_addr learned my addr 172.21.15.39:0/2703829681 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:16:12.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.964+0000 7fa251800640 1 --2- 172.21.15.39:0/2703829681 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5603cdbde5f0 0x5603cdbea2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:12.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.965+0000 7fa24be00640 1 --2- 172.21.15.39:0/2703829681 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5603cdbd9480 0x5603cdbdae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:12.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.965+0000 7fa251800640 1 -- 172.21.15.39:0/2703829681 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5603cdbd8a90 msgr2=0x5603cdbda940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:12.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.965+0000 7fa251800640 1 --2- 172.21.15.39:0/2703829681 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5603cdbd8a90 0x5603cdbda940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.965+0000 7fa251800640 1 -- 172.21.15.39:0/2703829681 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5603cdbd9480 msgr2=0x5603cdbdae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:12.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.965+0000 7fa251800640 1 --2- 172.21.15.39:0/2703829681 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5603cdbd9480 0x5603cdbdae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.965+0000 7fa251800640 1 -- 172.21.15.39:0/2703829681 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5603cdbe1480 con 0x5603cdbde5f0 2024-03-18T16:16:12.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.965+0000 7fa250e00640 1 --2- 172.21.15.39:0/2703829681 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5603cdbd8a90 0x5603cdbda940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:16:12.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.965+0000 7fa24be00640 1 --2- 172.21.15.39:0/2703829681 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5603cdbd9480 0x5603cdbdae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:16:12.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.966+0000 7fa251800640 1 --2- 172.21.15.39:0/2703829681 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5603cdbde5f0 0x5603cdbea2f0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fa24400e8b0 tx=0x7fa244004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9971e972a565bd2f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:12.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.966+0000 7fa23d600640 1 -- 172.21.15.39:0/2703829681 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa24403e070 con 0x5603cdbde5f0 2024-03-18T16:16:12.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.966+0000 7fa23d600640 1 -- 172.21.15.39:0/2703829681 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa24401cb50 con 0x5603cdbde5f0 2024-03-18T16:16:12.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.966+0000 7fa23d600640 1 -- 172.21.15.39:0/2703829681 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa244025ae0 con 0x5603cdbde5f0 2024-03-18T16:16:12.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.967+0000 7fa2565ba740 1 -- 172.21.15.39:0/2703829681 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5603cdbde5f0 msgr2=0x5603cdbea2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:12.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.967+0000 7fa2565ba740 1 --2- 172.21.15.39:0/2703829681 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5603cdbde5f0 0x5603cdbea2f0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fa24400e8b0 tx=0x7fa244004ab0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.967+0000 7fa2565ba740 1 -- 172.21.15.39:0/2703829681 shutdown_connections 2024-03-18T16:16:12.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.967+0000 7fa2565ba740 1 --2- 172.21.15.39:0/2703829681 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5603cdbde5f0 0x5603cdbea2f0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.967+0000 7fa2565ba740 1 --2- 172.21.15.39:0/2703829681 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5603cdbd9480 0x5603cdbdae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.967+0000 7fa2565ba740 1 --2- 172.21.15.39:0/2703829681 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5603cdbd8a90 0x5603cdbda940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.967+0000 7fa2565ba740 1 -- 172.21.15.39:0/2703829681 >> 172.21.15.39:0/2703829681 conn(0x5603cdbd4000 msgr2=0x5603cdbe2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:12.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.967+0000 7fa2565ba740 1 -- 172.21.15.39:0/2703829681 shutdown_connections 2024-03-18T16:16:12.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.967+0000 7fa2565ba740 1 -- 172.21.15.39:0/2703829681 wait complete. 2024-03-18T16:16:12.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.968+0000 7fa2565ba740 1 -- start start 2024-03-18T16:16:12.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.968+0000 7fa2565ba740 1 Processor -- start 2024-03-18T16:16:12.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.969+0000 7fa2565ba740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5603cdbd8a90 0x5603cdbe6f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:12.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.969+0000 7fa2565ba740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5603cdbd9480 0x5603cdbe74c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:12.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.969+0000 7fa2565ba740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5603cdbde5f0 0x5603cdc530b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:12.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.969+0000 7fa2565ba740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fa24c1ba3d0 con 0x5603cdbd9480 2024-03-18T16:16:12.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.969+0000 7fa2565ba740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fa24c1b9d00 con 0x5603cdbde5f0 2024-03-18T16:16:12.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.969+0000 7fa2565ba740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fa24c116650 con 0x5603cdbd8a90 2024-03-18T16:16:12.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.969+0000 7fa2565ba740 10 client.0 _ll_register_callbacks cb 0x7fa24400eb10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:16:12.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.969+0000 7fa250e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5603cdbd8a90 0x5603cdbe6f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:12.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.969+0000 7fa250e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5603cdbd8a90 0x5603cdbe6f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:59796/0 (socket says 172.21.15.39:59796) 2024-03-18T16:16:12.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.969+0000 7fa250e00640 1 -- 172.21.15.39:0/2540073212 learned_addr learned my addr 172.21.15.39:0/2540073212 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:16:12.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.969+0000 7fa251800640 1 --2- 172.21.15.39:0/2540073212 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5603cdbde5f0 0x5603cdc530b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:12.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.970+0000 7fa24be00640 1 --2- 172.21.15.39:0/2540073212 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5603cdbd9480 0x5603cdbe74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:12.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.970+0000 7fa250e00640 1 -- 172.21.15.39:0/2540073212 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5603cdbde5f0 msgr2=0x5603cdc530b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:12.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.970+0000 7fa250e00640 1 --2- 172.21.15.39:0/2540073212 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5603cdbde5f0 0x5603cdc530b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.970+0000 7fa250e00640 1 -- 172.21.15.39:0/2540073212 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5603cdbd9480 msgr2=0x5603cdbe74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:12.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.970+0000 7fa250e00640 1 --2- 172.21.15.39:0/2540073212 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5603cdbd9480 0x5603cdbe74c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.970+0000 7fa250e00640 1 -- 172.21.15.39:0/2540073212 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5603cdbea590 con 0x5603cdbd8a90 2024-03-18T16:16:12.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.970+0000 7fa24be00640 1 --2- 172.21.15.39:0/2540073212 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5603cdbd9480 0x5603cdbe74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:16:12.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.970+0000 7fa250e00640 1 --2- 172.21.15.39:0/2540073212 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5603cdbd8a90 0x5603cdbe6f80 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fa2400ba0e0 tx=0x7fa2400037d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ae5d269691aeeb0c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:12.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.970+0000 7fa232a00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:16:12.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.970+0000 7fa232a00640 1 -- 172.21.15.39:0/2540073212 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa24003c350 con 0x5603cdbd8a90 2024-03-18T16:16:12.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.971+0000 7fa232a00640 1 -- 172.21.15.39:0/2540073212 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fa2400b9d80 con 0x5603cdbd8a90 2024-03-18T16:16:12.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.971+0000 7fa2565ba740 1 -- 172.21.15.39:0/2540073212 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa24401cb50 con 0x5603cdbd8a90 2024-03-18T16:16:12.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.971+0000 7fa232a00640 1 -- 172.21.15.39:0/2540073212 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fa2400bde10 con 0x5603cdbd8a90 2024-03-18T16:16:12.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.971+0000 7fa2565ba740 10 client.26851 fetch_fsmap learned FSMap version 44 2024-03-18T16:16:12.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.971+0000 7fa2565ba740 1 -- 172.21.15.39:0/2540073212 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x5603cdc54630 con 0x5603cdbd8a90 2024-03-18T16:16:12.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.972+0000 7fa232a00640 1 -- 172.21.15.39:0/2540073212 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa24401cb50 con 0x5603cdbd8a90 2024-03-18T16:16:12.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.972+0000 7fa232a00640 1 -- 172.21.15.39:0/2540073212 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7fa2400c5a20 con 0x5603cdbd8a90 2024-03-18T16:16:12.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.972+0000 7fa2565ba740 10 client.26851 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:16:12.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.972+0000 7fa2565ba740 10 client.26851 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T16:16:12.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.972+0000 7fa2565ba740 20 client.26851 populate_metadata read hostname 'smithi039' 2024-03-18T16:16:12.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.972+0000 7fa2565ba740 1 --2- 172.21.15.39:0/2540073212 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x5603cdc54fe0 0x5603cdc57400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:12.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.973+0000 7fa2565ba740 4 client.26851 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:16:12.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.973+0000 7fa2565ba740 1 -- 172.21.15.39:0/2540073212 --> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5603cdb44ce0 con 0x5603cdc54fe0 2024-03-18T16:16:12.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.973+0000 7fa24be00640 1 --2- 172.21.15.39:0/2540073212 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x5603cdc54fe0 0x5603cdc57400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:12.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.974+0000 7fa24be00640 1 --2- 172.21.15.39:0/2540073212 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x5603cdc54fe0 0x5603cdc57400 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=16e40fc251fd6edb server_cookie=70a400557b959500 in_seq=0 out_seq=0 2024-03-18T16:16:12.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.974+0000 7fa232a00640 10 client.26851 ms_handle_connect on v2:172.21.15.196:6838/2001427268 2024-03-18T16:16:12.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.974+0000 7fa232a00640 1 -- 172.21.15.39:0/2540073212 <== mds.1 v2:172.21.15.196:6838/2001427268 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5603cdb44ce0 con 0x5603cdc54fe0 2024-03-18T16:16:12.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.974+0000 7fa232a00640 10 client.26851 handle_command_reply: tid=0 2024-03-18T16:16:12.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.974+0000 7fa2565ba740 1 client.26851 shutdown 2024-03-18T16:16:12.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.976+0000 7fa2565ba740 1 -- 172.21.15.39:0/2540073212 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5603cdbd8a90 msgr2=0x5603cdbe6f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:12.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.976+0000 7fa2565ba740 1 --2- 172.21.15.39:0/2540073212 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5603cdbd8a90 0x5603cdbe6f80 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fa2400ba0e0 tx=0x7fa2400037d0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.976+0000 7fa2565ba740 1 -- 172.21.15.39:0/2540073212 shutdown_connections 2024-03-18T16:16:12.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.976+0000 7fa2565ba740 1 --2- 172.21.15.39:0/2540073212 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x5603cdc54fe0 0x5603cdc57400 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.976+0000 7fa2565ba740 1 --2- 172.21.15.39:0/2540073212 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5603cdbde5f0 0x5603cdc530b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.976+0000 7fa2565ba740 1 --2- 172.21.15.39:0/2540073212 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5603cdbd9480 0x5603cdbe74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.976+0000 7fa2565ba740 1 --2- 172.21.15.39:0/2540073212 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5603cdbd8a90 0x5603cdbe6f80 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.976+0000 7fa2565ba740 1 -- 172.21.15.39:0/2540073212 >> 172.21.15.39:0/2540073212 conn(0x5603cdbd4000 msgr2=0x5603cdbe2d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:12.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.976+0000 7fa232a00640 0 client.26851 ms_handle_reset on v2:172.21.15.196:6838/2001427268 2024-03-18T16:16:12.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.976+0000 7fa2565ba740 1 -- 172.21.15.39:0/2540073212 shutdown_connections 2024-03-18T16:16:12.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.977+0000 7fa2565ba740 1 -- 172.21.15.39:0/2540073212 wait complete. 2024-03-18T16:16:12.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.977+0000 7fa2565ba740 20 client.26851 trim_cache size 0 max 16384 2024-03-18T16:16:12.980 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:16:12.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.978+0000 7fa252200640 1 -- 172.21.15.39:0/2470748801 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa2340a2950 msgr2=0x7fa2340a4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:12.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.978+0000 7fa252200640 1 --2- 172.21.15.39:0/2470748801 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa2340a2950 0x7fa2340a4e10 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x5603cdbd9ce0 tx=0x7fa244008730 comp rx=0 tx=0).stop 2024-03-18T16:16:12.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.978+0000 7fa252200640 1 -- 172.21.15.39:0/2470748801 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa24c0ffe40 msgr2=0x7fa24c19ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:12.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.978+0000 7fa252200640 1 --2- 172.21.15.39:0/2470748801 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa24c0ffe40 0x7fa24c19ddf0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fa24002c240 tx=0x7fa240004e30 comp rx=0 tx=0).stop 2024-03-18T16:16:12.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.978+0000 7fa252200640 1 -- 172.21.15.39:0/2470748801 shutdown_connections 2024-03-18T16:16:12.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.978+0000 7fa252200640 1 --2- 172.21.15.39:0/2470748801 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa24c101140 0x7fa24c1a53a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.978+0000 7fa252200640 1 --2- 172.21.15.39:0/2470748801 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa2340a2950 0x7fa2340a4e10 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.978+0000 7fa252200640 1 --2- 172.21.15.39:0/2470748801 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa24c1007a0 0x7fa24c19e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.978+0000 7fa252200640 1 --2- 172.21.15.39:0/2470748801 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa24c0ffe40 0x7fa24c19ddf0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:12.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.978+0000 7fa252200640 1 -- 172.21.15.39:0/2470748801 >> 172.21.15.39:0/2470748801 conn(0x7fa24c0fba80 msgr2=0x7fa24c0fd580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:12.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.979+0000 7fa252200640 1 -- 172.21.15.39:0/2470748801 shutdown_connections 2024-03-18T16:16:12.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:12.979+0000 7fa252200640 1 -- 172.21.15.39:0/2470748801 wait complete. 2024-03-18T16:16:13.006 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-18T16:16:13.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.269+0000 7fee00800640 1 -- 172.21.15.39:0/1964846736 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fedf81029c0 msgr2=0x7fedf8104e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:13.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.269+0000 7fee00800640 1 --2- 172.21.15.39:0/1964846736 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fedf81029c0 0x7fedf8104e30 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fede8009900 tx=0x7fede802bc30 comp rx=0 tx=0).stop 2024-03-18T16:16:13.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.269+0000 7fee00800640 1 -- 172.21.15.39:0/1964846736 shutdown_connections 2024-03-18T16:16:13.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.269+0000 7fee00800640 1 --2- 172.21.15.39:0/1964846736 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fedf8105370 0x7fedf81057f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:13.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.269+0000 7fee00800640 1 --2- 172.21.15.39:0/1964846736 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fedf81029c0 0x7fedf8104e30 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:13.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.269+0000 7fee00800640 1 --2- 172.21.15.39:0/1964846736 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fedf8100050 0x7fedf8102480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:13.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.269+0000 7fee00800640 1 -- 172.21.15.39:0/1964846736 >> 172.21.15.39:0/1964846736 conn(0x7fedf80fbc50 msgr2=0x7fedf80fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:13.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.269+0000 7fee00800640 1 -- 172.21.15.39:0/1964846736 shutdown_connections 2024-03-18T16:16:13.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.269+0000 7fee00800640 1 -- 172.21.15.39:0/1964846736 wait complete. 2024-03-18T16:16:13.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.270+0000 7fee00800640 1 Processor -- start 2024-03-18T16:16:13.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.271+0000 7fee00800640 1 -- start start 2024-03-18T16:16:13.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.271+0000 7fee00800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fedf8100050 0x7fedf8070120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:13.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.271+0000 7fee00800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fedf81029c0 0x7fedf8070660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:13.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.271+0000 7fee00800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fedf8105370 0x7fedf81ae190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:13.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.271+0000 7fee00800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fedf8113a80 con 0x7fedf8105370 2024-03-18T16:16:13.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.271+0000 7fee00800640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fedf8113900 con 0x7fedf8100050 2024-03-18T16:16:13.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.271+0000 7fee00800640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fedf8113c00 con 0x7fedf81029c0 2024-03-18T16:16:13.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.272+0000 7fedfd400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fedf81029c0 0x7fedf8070660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:13.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.272+0000 7fedfe800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fedf8105370 0x7fedf81ae190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:13.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.272+0000 7fedfd400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fedf81029c0 0x7fedf8070660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:59826/0 (socket says 172.21.15.39:59826) 2024-03-18T16:16:13.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.272+0000 7fedfd400640 1 -- 172.21.15.39:0/946320632 learned_addr learned my addr 172.21.15.39:0/946320632 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:16:13.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.272+0000 7fedfd400640 1 -- 172.21.15.39:0/946320632 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fedf8100050 msgr2=0x7fedf8070120 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T16:16:13.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.272+0000 7fedfd400640 1 --2- 172.21.15.39:0/946320632 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fedf8100050 0x7fedf8070120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:13.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.272+0000 7fedfd400640 1 -- 172.21.15.39:0/946320632 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fedf8105370 msgr2=0x7fedf81ae190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:13.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.272+0000 7fedfd400640 1 --2- 172.21.15.39:0/946320632 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fedf8105370 0x7fedf81ae190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:13.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.272+0000 7fedfd400640 1 -- 172.21.15.39:0/946320632 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedf8070d00 con 0x7fedf81029c0 2024-03-18T16:16:13.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.272+0000 7fedfe800640 1 --2- 172.21.15.39:0/946320632 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fedf8105370 0x7fedf81ae190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:16:13.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.273+0000 7fedfd400640 1 --2- 172.21.15.39:0/946320632 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fedf81029c0 0x7fedf8070660 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fede8005ab0 tx=0x7fede802c6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:13.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.274+0000 7fedf2a00640 1 -- 172.21.15.39:0/946320632 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fede8043020 con 0x7fedf81029c0 2024-03-18T16:16:13.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.274+0000 7fedf2a00640 1 -- 172.21.15.39:0/946320632 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fede80054e0 con 0x7fedf81029c0 2024-03-18T16:16:13.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.274+0000 7fedf2a00640 1 -- 172.21.15.39:0/946320632 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fede8003ce0 con 0x7fedf81029c0 2024-03-18T16:16:13.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.274+0000 7fee00800640 1 -- 172.21.15.39:0/946320632 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fedf81ae790 con 0x7fedf81029c0 2024-03-18T16:16:13.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.274+0000 7fee00800640 1 -- 172.21.15.39:0/946320632 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fedf81aec60 con 0x7fedf81029c0 2024-03-18T16:16:13.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.279+0000 7fedf2a00640 1 -- 172.21.15.39:0/946320632 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fede8046020 con 0x7fedf81029c0 2024-03-18T16:16:13.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.279+0000 7fee035ef740 1 Processor -- start 2024-03-18T16:16:13.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.281+0000 7fee035ef740 1 -- start start 2024-03-18T16:16:13.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.281+0000 7fedf2a00640 1 --2- 172.21.15.39:0/946320632 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fede00a2950 0x7fede00a4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:13.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.281+0000 7fee035ef740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560d7c6dc8c0 0x560d7c6dcce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:13.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.281+0000 7fee035ef740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560d7c6dd2b0 0x560d7c6e1db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:13.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.281+0000 7fee035ef740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d7c6e2420 0x560d7c6e48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:13.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.281+0000 7fee035ef740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x560d7c6a43b0 con 0x560d7c6dd2b0 2024-03-18T16:16:13.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.281+0000 7fee035ef740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x560d7c641450 con 0x560d7c6e2420 2024-03-18T16:16:13.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.281+0000 7fee035ef740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x560d7c5e9cb0 con 0x560d7c6dc8c0 2024-03-18T16:16:13.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.281+0000 7fedf2a00640 1 -- 172.21.15.39:0/946320632 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fede80f0f30 con 0x7fedf81029c0 2024-03-18T16:16:13.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.282+0000 7fedfde00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560d7c6dc8c0 0x560d7c6dcce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:13.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.282+0000 7fedfde00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560d7c6dc8c0 0x560d7c6dcce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:59838/0 (socket says 172.21.15.39:59838) 2024-03-18T16:16:13.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.282+0000 7fedfe800640 1 --2- 172.21.15.39:0/946320632 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fede00a2950 0x7fede00a4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:13.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.282+0000 7fedfde00640 1 -- 172.21.15.39:0/3679503258 learned_addr learned my addr 172.21.15.39:0/3679503258 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:16:13.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.282+0000 7fedfe800640 1 --2- 172.21.15.39:0/3679503258 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d7c6e2420 0x560d7c6e48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:13.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.282+0000 7fedfd400640 1 --2- 172.21.15.39:0/3679503258 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560d7c6dd2b0 0x560d7c6e1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:13.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.282+0000 7fedfde00640 1 -- 172.21.15.39:0/3679503258 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d7c6e2420 msgr2=0x560d7c6e48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:13.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.282+0000 7fedfde00640 1 --2- 172.21.15.39:0/3679503258 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d7c6e2420 0x560d7c6e48b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:13.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.282+0000 7fedfde00640 1 -- 172.21.15.39:0/3679503258 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560d7c6dd2b0 msgr2=0x560d7c6e1db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:13.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.282+0000 7fedfde00640 1 --2- 172.21.15.39:0/3679503258 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560d7c6dd2b0 0x560d7c6e1db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:13.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.282+0000 7fedfde00640 1 -- 172.21.15.39:0/3679503258 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d7c60b1f0 con 0x560d7c6dc8c0 2024-03-18T16:16:13.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.282+0000 7fedfd400640 1 --2- 172.21.15.39:0/3679503258 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560d7c6dd2b0 0x560d7c6e1db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:16:13.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.283+0000 7fedfde00640 1 --2- 172.21.15.39:0/3679503258 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560d7c6dc8c0 0x560d7c6dcce0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fedec009bd0 tx=0x7fedec00e720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=747ef836a5b962fe server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:13.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.283+0000 7fedfe800640 1 --2- 172.21.15.39:0/946320632 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fede00a2950 0x7fede00a4e10 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fedf4005fd0 tx=0x7fedf4009590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:13.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.283+0000 7fede7000640 1 -- 172.21.15.39:0/3679503258 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedec005020 con 0x560d7c6dc8c0 2024-03-18T16:16:13.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.283+0000 7fede7000640 1 -- 172.21.15.39:0/3679503258 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedec0047a0 con 0x560d7c6dc8c0 2024-03-18T16:16:13.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.283+0000 7fede7000640 1 -- 172.21.15.39:0/3679503258 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedec010690 con 0x560d7c6dc8c0 2024-03-18T16:16:13.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.285+0000 7fee035ef740 1 -- 172.21.15.39:0/3679503258 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560d7c6dc8c0 msgr2=0x560d7c6dcce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:13.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.285+0000 7fee035ef740 1 --2- 172.21.15.39:0/3679503258 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560d7c6dc8c0 0x560d7c6dcce0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fedec009bd0 tx=0x7fedec00e720 comp rx=0 tx=0).stop 2024-03-18T16:16:13.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.285+0000 7fee035ef740 1 -- 172.21.15.39:0/3679503258 shutdown_connections 2024-03-18T16:16:13.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.285+0000 7fee035ef740 1 --2- 172.21.15.39:0/3679503258 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d7c6e2420 0x560d7c6e48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:13.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.285+0000 7fee035ef740 1 --2- 172.21.15.39:0/3679503258 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560d7c6dd2b0 0x560d7c6e1db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:13.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.285+0000 7fee035ef740 1 --2- 172.21.15.39:0/3679503258 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560d7c6dc8c0 0x560d7c6dcce0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:13.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.285+0000 7fee035ef740 1 -- 172.21.15.39:0/3679503258 >> 172.21.15.39:0/3679503258 conn(0x560d7c6d7e30 msgr2=0x560d7c6d8230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:13.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.286+0000 7fee035ef740 1 -- 172.21.15.39:0/3679503258 shutdown_connections 2024-03-18T16:16:13.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.286+0000 7fee035ef740 1 -- 172.21.15.39:0/3679503258 wait complete. 2024-03-18T16:16:13.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.287+0000 7fee035ef740 1 -- start start 2024-03-18T16:16:13.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.287+0000 7fee035ef740 1 Processor -- start 2024-03-18T16:16:13.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.288+0000 7fee035ef740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d7c6dc8c0 0x560d7c6e6bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:13.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.288+0000 7fee035ef740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560d7c6dd2b0 0x560d7c6e70f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:13.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.288+0000 7fee035ef740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560d7c6e2420 0x560d7c6ee130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:13.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.288+0000 7fee035ef740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x560d7c5fd110 con 0x560d7c6dd2b0 2024-03-18T16:16:13.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.288+0000 7fee035ef740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x560d7c63d4f0 con 0x560d7c6dc8c0 2024-03-18T16:16:13.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.288+0000 7fee035ef740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x560d7c6e53d0 con 0x560d7c6e2420 2024-03-18T16:16:13.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.288+0000 7fee035ef740 10 client.0 _ll_register_callbacks cb 0x560d7c5eab70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:16:13.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.288+0000 7fedfde00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d7c6dc8c0 0x560d7c6e6bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:13.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.288+0000 7fedfde00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d7c6dc8c0 0x560d7c6e6bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:53930/0 (socket says 172.21.15.39:53930) 2024-03-18T16:16:13.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.288+0000 7fedfde00640 1 -- 172.21.15.39:0/1576085274 learned_addr learned my addr 172.21.15.39:0/1576085274 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:16:13.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.288+0000 7fedfe800640 1 --2- 172.21.15.39:0/1576085274 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560d7c6e2420 0x560d7c6ee130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:13.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.288+0000 7fedfd400640 1 --2- 172.21.15.39:0/1576085274 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560d7c6dd2b0 0x560d7c6e70f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:13.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.289+0000 7fedfde00640 1 -- 172.21.15.39:0/1576085274 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560d7c6e2420 msgr2=0x560d7c6ee130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:13.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.289+0000 7fedfde00640 1 --2- 172.21.15.39:0/1576085274 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560d7c6e2420 0x560d7c6ee130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:13.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.289+0000 7fedfde00640 1 -- 172.21.15.39:0/1576085274 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560d7c6dd2b0 msgr2=0x560d7c6e70f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:13.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.289+0000 7fedfde00640 1 --2- 172.21.15.39:0/1576085274 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560d7c6dd2b0 0x560d7c6e70f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:13.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.289+0000 7fedfde00640 1 -- 172.21.15.39:0/1576085274 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560d7c3553f0 con 0x560d7c6dc8c0 2024-03-18T16:16:13.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.289+0000 7fedfe800640 1 --2- 172.21.15.39:0/1576085274 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560d7c6e2420 0x560d7c6ee130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:16:13.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.289+0000 7fedfde00640 1 --2- 172.21.15.39:0/1576085274 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d7c6dc8c0 0x560d7c6e6bb0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fedec009ba0 tx=0x7fedec009950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=496a79e4352839cf server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:13.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.289+0000 7feddb400640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:16:13.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.290+0000 7feddb400640 1 -- 172.21.15.39:0/1576085274 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedec01d040 con 0x560d7c6dc8c0 2024-03-18T16:16:13.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.290+0000 7feddb400640 1 -- 172.21.15.39:0/1576085274 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fedec004050 con 0x560d7c6dc8c0 2024-03-18T16:16:13.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.290+0000 7fee035ef740 1 -- 172.21.15.39:0/1576085274 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fedec0047a0 con 0x560d7c6dc8c0 2024-03-18T16:16:13.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.290+0000 7feddb400640 1 -- 172.21.15.39:0/1576085274 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fedec030050 con 0x560d7c6dc8c0 2024-03-18T16:16:13.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.290+0000 7fee035ef740 10 client.26885 fetch_fsmap learned FSMap version 44 2024-03-18T16:16:13.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.290+0000 7fee035ef740 1 -- 172.21.15.39:0/1576085274 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x560d7c41f7d0 con 0x560d7c6dc8c0 2024-03-18T16:16:13.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.290+0000 7feddb400640 1 -- 172.21.15.39:0/1576085274 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedec0047a0 con 0x560d7c6dc8c0 2024-03-18T16:16:13.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.291+0000 7feddb400640 1 -- 172.21.15.39:0/1576085274 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7fedec003950 con 0x560d7c6dc8c0 2024-03-18T16:16:13.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.291+0000 7fee035ef740 10 client.26885 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:16:13.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.291+0000 7fee035ef740 10 client.26885 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:16:13.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.291+0000 7fee035ef740 20 client.26885 populate_metadata read hostname 'smithi039' 2024-03-18T16:16:13.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.291+0000 7fee035ef740 1 --2- 172.21.15.39:0/1576085274 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x560d7c6f2640 0x560d7c6f4a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:13.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.291+0000 7fee035ef740 4 client.26885 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:16:13.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.291+0000 7fee035ef740 1 -- 172.21.15.39:0/1576085274 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560d7c64e990 con 0x560d7c6f2640 2024-03-18T16:16:13.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.292+0000 7fedfd400640 1 --2- 172.21.15.39:0/1576085274 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x560d7c6f2640 0x560d7c6f4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:13.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.292+0000 7fedfd400640 1 --2- 172.21.15.39:0/1576085274 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x560d7c6f2640 0x560d7c6f4a30 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=349fd8f8823c9a9c server_cookie=32e4a57ab6c67e0f in_seq=0 out_seq=0 2024-03-18T16:16:13.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.292+0000 7feddb400640 10 client.26885 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:16:13.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.293+0000 7feddb400640 1 -- 172.21.15.39:0/1576085274 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560d7c64e990 con 0x560d7c6f2640 2024-03-18T16:16:13.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.293+0000 7feddb400640 10 client.26885 handle_command_reply: tid=0 2024-03-18T16:16:13.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.293+0000 7fee035ef740 1 client.26885 shutdown 2024-03-18T16:16:13.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.294+0000 7fee035ef740 1 -- 172.21.15.39:0/1576085274 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d7c6dc8c0 msgr2=0x560d7c6e6bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:13.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.294+0000 7fee035ef740 1 --2- 172.21.15.39:0/1576085274 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d7c6dc8c0 0x560d7c6e6bb0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fedec009ba0 tx=0x7fedec009950 comp rx=0 tx=0).stop 2024-03-18T16:16:13.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.294+0000 7fee035ef740 1 -- 172.21.15.39:0/1576085274 shutdown_connections 2024-03-18T16:16:13.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.294+0000 7fee035ef740 1 --2- 172.21.15.39:0/1576085274 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x560d7c6f2640 0x560d7c6f4a30 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:13.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.294+0000 7fee035ef740 1 --2- 172.21.15.39:0/1576085274 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560d7c6e2420 0x560d7c6ee130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:13.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.294+0000 7fee035ef740 1 --2- 172.21.15.39:0/1576085274 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560d7c6dd2b0 0x560d7c6e70f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:13.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.294+0000 7fee035ef740 1 --2- 172.21.15.39:0/1576085274 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d7c6dc8c0 0x560d7c6e6bb0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:13.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.294+0000 7feddb400640 0 client.26885 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:16:13.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.294+0000 7fee035ef740 1 -- 172.21.15.39:0/1576085274 >> 172.21.15.39:0/1576085274 conn(0x560d7c6d7e30 msgr2=0x560d7c6d71f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:13.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.294+0000 7fee035ef740 1 -- 172.21.15.39:0/1576085274 shutdown_connections 2024-03-18T16:16:13.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.294+0000 7fee035ef740 1 -- 172.21.15.39:0/1576085274 wait complete. 2024-03-18T16:16:13.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.295+0000 7fee035ef740 20 client.26885 trim_cache size 0 max 16384 2024-03-18T16:16:13.320 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:13 smithi039 ceph-mon[31721]: pgmap v606: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.7 MiB/s wr, 449 op/s 2024-03-18T16:16:13.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.320+0000 7fee035ef740 1 Processor -- start 2024-03-18T16:16:13.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.320+0000 7fee035ef740 1 -- start start 2024-03-18T16:16:13.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.320+0000 7fee035ef740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d7c6dc8c0 0x560d7c6db7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:13.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.320+0000 7fee035ef740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560d7c6dd2b0 0x560d7c6ee0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:13.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.320+0000 7fee035ef740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560d7c6e2420 0x560d7c6f9230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:13.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.320+0000 7fee035ef740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fedf81b92c0 con 0x560d7c6dd2b0 2024-03-18T16:16:13.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.320+0000 7fee035ef740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fedf81bae00 con 0x560d7c6dc8c0 2024-03-18T16:16:13.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.320+0000 7fee035ef740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fedf81b9990 con 0x560d7c6e2420 2024-03-18T16:16:13.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.321+0000 7fedfde00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d7c6dc8c0 0x560d7c6db7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:13.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.321+0000 7fedfde00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d7c6dc8c0 0x560d7c6db7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:53946/0 (socket says 172.21.15.39:53946) 2024-03-18T16:16:13.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.321+0000 7fedfde00640 1 -- 172.21.15.39:0/1528072064 learned_addr learned my addr 172.21.15.39:0/1528072064 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:16:13.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.321+0000 7fedfd400640 1 --2- 172.21.15.39:0/1528072064 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560d7c6dd2b0 0x560d7c6ee0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:13.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.321+0000 7fedfe800640 1 --2- 172.21.15.39:0/1528072064 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560d7c6e2420 0x560d7c6f9230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:13.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.321+0000 7fedfde00640 1 -- 172.21.15.39:0/1528072064 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560d7c6e2420 msgr2=0x560d7c6f9230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:13.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.321+0000 7fedfde00640 1 --2- 172.21.15.39:0/1528072064 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560d7c6e2420 0x560d7c6f9230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:13.326 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.321+0000 7fedfde00640 1 -- 172.21.15.39:0/1528072064 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560d7c6dd2b0 msgr2=0x560d7c6ee0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:13.326 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.321+0000 7fedfde00640 1 --2- 172.21.15.39:0/1528072064 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560d7c6dd2b0 0x560d7c6ee0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:13.326 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.321+0000 7fedfde00640 1 -- 172.21.15.39:0/1528072064 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d7c6d3c10 con 0x560d7c6dc8c0 2024-03-18T16:16:13.326 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.322+0000 7fedfde00640 1 --2- 172.21.15.39:0/1528072064 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d7c6dc8c0 0x560d7c6db7a0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fedec0092e0 tx=0x7fedec009aa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1682f05382d48b61 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:13.326 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.322+0000 7fede6600640 1 -- 172.21.15.39:0/1528072064 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedec0056a0 con 0x560d7c6dc8c0 2024-03-18T16:16:13.326 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.322+0000 7fede6600640 1 -- 172.21.15.39:0/1528072064 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedec039070 con 0x560d7c6dc8c0 2024-03-18T16:16:13.326 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.322+0000 7fede6600640 1 -- 172.21.15.39:0/1528072064 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedec005110 con 0x560d7c6dc8c0 2024-03-18T16:16:13.326 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.323+0000 7fee035ef740 1 -- 172.21.15.39:0/1528072064 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d7c6dc8c0 msgr2=0x560d7c6db7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:13.326 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.323+0000 7fee035ef740 1 --2- 172.21.15.39:0/1528072064 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d7c6dc8c0 0x560d7c6db7a0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fedec0092e0 tx=0x7fedec009aa0 comp rx=0 tx=0).stop 2024-03-18T16:16:13.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.323+0000 7fee035ef740 1 -- 172.21.15.39:0/1528072064 shutdown_connections 2024-03-18T16:16:13.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.323+0000 7fee035ef740 1 --2- 172.21.15.39:0/1528072064 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560d7c6e2420 0x560d7c6f9230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:13.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.323+0000 7fee035ef740 1 --2- 172.21.15.39:0/1528072064 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560d7c6dd2b0 0x560d7c6ee0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:13.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.323+0000 7fee035ef740 1 --2- 172.21.15.39:0/1528072064 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d7c6dc8c0 0x560d7c6db7a0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:13.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.323+0000 7fee035ef740 1 -- 172.21.15.39:0/1528072064 >> 172.21.15.39:0/1528072064 conn(0x560d7c6d7e30 msgr2=0x560d7c6db090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:13.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.324+0000 7fee035ef740 1 -- 172.21.15.39:0/1528072064 shutdown_connections 2024-03-18T16:16:13.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.324+0000 7fee035ef740 1 -- 172.21.15.39:0/1528072064 wait complete. 2024-03-18T16:16:13.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.324+0000 7fee035ef740 1 -- start start 2024-03-18T16:16:13.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.324+0000 7fee035ef740 1 Processor -- start 2024-03-18T16:16:13.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.326+0000 7fee035ef740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560d7c6dc8c0 0x560d7c6eb9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:13.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.326+0000 7fee035ef740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d7c6dd2b0 0x560d7c6ebf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:13.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.326+0000 7fee035ef740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560d7c6e2420 0x560d7c6ecf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:13.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.326+0000 7fee035ef740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fedf81ba730 con 0x560d7c6dc8c0 2024-03-18T16:16:13.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.326+0000 7fee035ef740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fedf81ba060 con 0x560d7c6dd2b0 2024-03-18T16:16:13.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.326+0000 7fee035ef740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fedf81b72a0 con 0x560d7c6e2420 2024-03-18T16:16:13.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.326+0000 7fee035ef740 10 client.0 _ll_register_callbacks cb 0x7fedf4009dd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:16:13.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.327+0000 7fedfde00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560d7c6dc8c0 0x560d7c6eb9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:13.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.327+0000 7fedfde00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560d7c6dc8c0 0x560d7c6eb9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:46608/0 (socket says 172.21.15.39:46608) 2024-03-18T16:16:13.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.327+0000 7fedfde00640 1 -- 172.21.15.39:0/1085960435 learned_addr learned my addr 172.21.15.39:0/1085960435 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:16:13.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.327+0000 7fedfe800640 1 --2- 172.21.15.39:0/1085960435 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560d7c6e2420 0x560d7c6ecf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:13.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.327+0000 7fedfd400640 1 --2- 172.21.15.39:0/1085960435 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d7c6dd2b0 0x560d7c6ebf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:13.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.327+0000 7fedfe800640 1 -- 172.21.15.39:0/1085960435 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d7c6dd2b0 msgr2=0x560d7c6ebf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:13.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.327+0000 7fedfe800640 1 --2- 172.21.15.39:0/1085960435 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d7c6dd2b0 0x560d7c6ebf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:13.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.327+0000 7fedfe800640 1 -- 172.21.15.39:0/1085960435 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560d7c6dc8c0 msgr2=0x560d7c6eb9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:13.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.327+0000 7fedfe800640 1 --2- 172.21.15.39:0/1085960435 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560d7c6dc8c0 0x560d7c6eb9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:13.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.327+0000 7fedfe800640 1 -- 172.21.15.39:0/1085960435 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560d7c6e21e0 con 0x560d7c6e2420 2024-03-18T16:16:13.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.327+0000 7fedfde00640 1 --2- 172.21.15.39:0/1085960435 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560d7c6dc8c0 0x560d7c6eb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:16:13.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.328+0000 7fedfe800640 1 --2- 172.21.15.39:0/1085960435 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560d7c6e2420 0x560d7c6ecf30 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fedf4010b70 tx=0x7fedf40094a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee03591083c555 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:13.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.328+0000 7feddb400640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:16:13.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.328+0000 7feddb400640 1 -- 172.21.15.39:0/1085960435 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedf4013810 con 0x560d7c6e2420 2024-03-18T16:16:13.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.328+0000 7feddb400640 1 -- 172.21.15.39:0/1085960435 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fedf40139b0 con 0x560d7c6e2420 2024-03-18T16:16:13.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.329+0000 7fee035ef740 1 -- 172.21.15.39:0/1085960435 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fedec039070 con 0x560d7c6e2420 2024-03-18T16:16:13.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.329+0000 7feddb400640 1 -- 172.21.15.39:0/1085960435 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fedf4039050 con 0x560d7c6e2420 2024-03-18T16:16:13.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.329+0000 7fee035ef740 10 client.26869 fetch_fsmap learned FSMap version 44 2024-03-18T16:16:13.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.329+0000 7fee035ef740 1 -- 172.21.15.39:0/1085960435 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x560d7c6d8ba0 con 0x560d7c6e2420 2024-03-18T16:16:13.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.329+0000 7feddb400640 1 -- 172.21.15.39:0/1085960435 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedec039070 con 0x560d7c6e2420 2024-03-18T16:16:13.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.329+0000 7feddb400640 1 -- 172.21.15.39:0/1085960435 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7fedf401e040 con 0x560d7c6e2420 2024-03-18T16:16:13.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.330+0000 7fee035ef740 10 client.26869 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:16:13.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.330+0000 7fee035ef740 10 client.26869 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:16:13.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.330+0000 7fee035ef740 20 client.26869 populate_metadata read hostname 'smithi039' 2024-03-18T16:16:13.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.330+0000 7fee035ef740 1 --2- 172.21.15.39:0/1085960435 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x560d7c759150 0x560d7c75b570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:13.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.330+0000 7fee035ef740 4 client.26869 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:16:13.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.330+0000 7fee035ef740 1 -- 172.21.15.39:0/1085960435 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560d7c648ce0 con 0x560d7c759150 2024-03-18T16:16:13.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.330+0000 7fedfde00640 1 --2- 172.21.15.39:0/1085960435 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x560d7c759150 0x560d7c75b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:13.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.331+0000 7feddb400640 10 client.26869 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:16:13.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.331+0000 7fedfde00640 1 --2- 172.21.15.39:0/1085960435 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x560d7c759150 0x560d7c75b570 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b84a90b76da6fc3 server_cookie=751819d3ae30b3a3 in_seq=0 out_seq=0 2024-03-18T16:16:13.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.332+0000 7feddb400640 1 -- 172.21.15.39:0/1085960435 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560d7c648ce0 con 0x560d7c759150 2024-03-18T16:16:13.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.332+0000 7feddb400640 10 client.26869 handle_command_reply: tid=0 2024-03-18T16:16:13.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.332+0000 7fee035ef740 1 client.26869 shutdown 2024-03-18T16:16:13.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.333+0000 7fee035ef740 1 -- 172.21.15.39:0/1085960435 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560d7c6e2420 msgr2=0x560d7c6ecf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:13.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.333+0000 7fee035ef740 1 --2- 172.21.15.39:0/1085960435 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560d7c6e2420 0x560d7c6ecf30 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fedf4010b70 tx=0x7fedf40094a0 comp rx=0 tx=0).stop 2024-03-18T16:16:13.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.333+0000 7fee035ef740 1 -- 172.21.15.39:0/1085960435 shutdown_connections 2024-03-18T16:16:13.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.333+0000 7fee035ef740 1 --2- 172.21.15.39:0/1085960435 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x560d7c759150 0x560d7c75b570 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:13.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.333+0000 7fee035ef740 1 --2- 172.21.15.39:0/1085960435 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d7c6dd2b0 0x560d7c6ebf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:13.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.333+0000 7fee035ef740 1 --2- 172.21.15.39:0/1085960435 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560d7c6e2420 0x560d7c6ecf30 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:13.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.333+0000 7fee035ef740 1 --2- 172.21.15.39:0/1085960435 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560d7c6dc8c0 0x560d7c6eb9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:13.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.333+0000 7fee035ef740 1 -- 172.21.15.39:0/1085960435 >> 172.21.15.39:0/1085960435 conn(0x560d7c6d7e30 msgr2=0x560d7c6db920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:13.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.333+0000 7feddb400640 0 client.26869 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:16:13.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.334+0000 7fee035ef740 1 -- 172.21.15.39:0/1085960435 shutdown_connections 2024-03-18T16:16:13.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.334+0000 7fee035ef740 1 -- 172.21.15.39:0/1085960435 wait complete. 2024-03-18T16:16:13.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.334+0000 7fee035ef740 20 client.26869 trim_cache size 0 max 16384 2024-03-18T16:16:13.338 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:16:13.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.335+0000 7fee00800640 1 -- 172.21.15.39:0/946320632 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fede00a2950 msgr2=0x7fede00a4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:13.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.335+0000 7fee00800640 1 --2- 172.21.15.39:0/946320632 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fede00a2950 0x7fede00a4e10 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fedf4005fd0 tx=0x7fedf4009590 comp rx=0 tx=0).stop 2024-03-18T16:16:13.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.335+0000 7fee00800640 1 -- 172.21.15.39:0/946320632 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fedf81029c0 msgr2=0x7fedf8070660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:13.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.335+0000 7fee00800640 1 --2- 172.21.15.39:0/946320632 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fedf81029c0 0x7fedf8070660 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fede8005ab0 tx=0x7fede802c6c0 comp rx=0 tx=0).stop 2024-03-18T16:16:13.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.336+0000 7fee00800640 1 -- 172.21.15.39:0/946320632 shutdown_connections 2024-03-18T16:16:13.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.336+0000 7fee00800640 1 --2- 172.21.15.39:0/946320632 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fede00a2950 0x7fede00a4e10 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:13.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.336+0000 7fee00800640 1 --2- 172.21.15.39:0/946320632 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fedf8105370 0x7fedf81ae190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:13.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.336+0000 7fee00800640 1 --2- 172.21.15.39:0/946320632 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fedf81029c0 0x7fedf8070660 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:13.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.336+0000 7fee00800640 1 --2- 172.21.15.39:0/946320632 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fedf8100050 0x7fedf8070120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:13.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.336+0000 7fee00800640 1 -- 172.21.15.39:0/946320632 >> 172.21.15.39:0/946320632 conn(0x7fedf80fbc50 msgr2=0x7fedf81009d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:13.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.336+0000 7fee00800640 1 -- 172.21.15.39:0/946320632 shutdown_connections 2024-03-18T16:16:13.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:13.336+0000 7fee00800640 1 -- 172.21.15.39:0/946320632 wait complete. 2024-03-18T16:16:13.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:13 smithi196 ceph-mon[36307]: pgmap v606: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.7 MiB/s wr, 449 op/s 2024-03-18T16:16:13.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:13 smithi194 ceph-mon[36200]: pgmap v606: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.7 MiB/s wr, 449 op/s 2024-03-18T16:16:14.364 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-18T16:16:14.365 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T16:16:14.365 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-18T16:16:14.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.612+0000 7f4db1a00640 1 -- 172.21.15.39:0/4165393619 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4dac1017f0 msgr2=0x7f4dac101c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:14.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.612+0000 7f4db1a00640 1 --2- 172.21.15.39:0/4165393619 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4dac1017f0 0x7f4dac101c10 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f4d9c009ca0 tx=0x7f4d9c02c0f0 comp rx=0 tx=0).stop 2024-03-18T16:16:14.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.613+0000 7f4db1a00640 1 -- 172.21.15.39:0/4165393619 shutdown_connections 2024-03-18T16:16:14.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.613+0000 7f4db1a00640 1 --2- 172.21.15.39:0/4165393619 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4dac1033a0 0x7f4dac109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:14.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.613+0000 7f4db1a00640 1 --2- 172.21.15.39:0/4165393619 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4dac1029e0 0x7f4dac102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:14.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.613+0000 7f4db1a00640 1 --2- 172.21.15.39:0/4165393619 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4dac1017f0 0x7f4dac101c10 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:14.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.613+0000 7f4db1a00640 1 -- 172.21.15.39:0/4165393619 >> 172.21.15.39:0/4165393619 conn(0x7f4dac0fcd80 msgr2=0x7f4dac0ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:14.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.614+0000 7f4db1a00640 1 -- 172.21.15.39:0/4165393619 shutdown_connections 2024-03-18T16:16:14.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.614+0000 7f4db1a00640 1 -- 172.21.15.39:0/4165393619 wait complete. 2024-03-18T16:16:14.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.615+0000 7f4db1a00640 1 Processor -- start 2024-03-18T16:16:14.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.615+0000 7f4db1a00640 1 -- start start 2024-03-18T16:16:14.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.616+0000 7f4db1a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4dac1017f0 0x7f4dac19e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:14.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.616+0000 7f4db1a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4dac1029e0 0x7f4dac19e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:14.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.616+0000 7f4db1a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4dac1033a0 0x7f4dac1a56d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:14.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.616+0000 7f4db1a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f4dac10b290 con 0x7f4dac1029e0 2024-03-18T16:16:14.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.616+0000 7f4db1a00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f4dac10b110 con 0x7f4dac1033a0 2024-03-18T16:16:14.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.616+0000 7f4db1a00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f4dac10b410 con 0x7f4dac1017f0 2024-03-18T16:16:14.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.617+0000 7f4da2000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4dac1029e0 0x7f4dac19e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:14.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.617+0000 7f4da2000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4dac1029e0 0x7f4dac19e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:46634/0 (socket says 172.21.15.39:46634) 2024-03-18T16:16:14.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.617+0000 7f4da2000640 1 -- 172.21.15.39:0/2715263784 learned_addr learned my addr 172.21.15.39:0/2715263784 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:16:14.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.617+0000 7f4dab400640 1 --2- 172.21.15.39:0/2715263784 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4dac1033a0 0x7f4dac1a56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:14.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.617+0000 7f4daaa00640 1 --2- 172.21.15.39:0/2715263784 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4dac1017f0 0x7f4dac19e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:14.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.617+0000 7f4daaa00640 1 -- 172.21.15.39:0/2715263784 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4dac1033a0 msgr2=0x7f4dac1a56d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:14.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.617+0000 7f4daaa00640 1 --2- 172.21.15.39:0/2715263784 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4dac1033a0 0x7f4dac1a56d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:14.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.617+0000 7f4daaa00640 1 -- 172.21.15.39:0/2715263784 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4dac1029e0 msgr2=0x7f4dac19e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:14.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.617+0000 7f4daaa00640 1 --2- 172.21.15.39:0/2715263784 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4dac1029e0 0x7f4dac19e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:14.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.617+0000 7f4daaa00640 1 -- 172.21.15.39:0/2715263784 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4dac1a5d40 con 0x7f4dac1017f0 2024-03-18T16:16:14.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.618+0000 7f4da2000640 1 --2- 172.21.15.39:0/2715263784 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4dac1029e0 0x7f4dac19e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:16:14.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.618+0000 7f4daaa00640 1 --2- 172.21.15.39:0/2715263784 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4dac1017f0 0x7f4dac19e120 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f4d9c037040 tx=0x7f4d9c032ea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:14.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.618+0000 7f4da3e00640 1 -- 172.21.15.39:0/2715263784 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d9c041020 con 0x7f4dac1017f0 2024-03-18T16:16:14.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.618+0000 7f4da3e00640 1 -- 172.21.15.39:0/2715263784 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d9c034cd0 con 0x7f4dac1017f0 2024-03-18T16:16:14.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.618+0000 7f4da3e00640 1 -- 172.21.15.39:0/2715263784 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d9c039040 con 0x7f4dac1017f0 2024-03-18T16:16:14.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.618+0000 7f4db1a00640 1 -- 172.21.15.39:0/2715263784 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4dac1a5fd0 con 0x7f4dac1017f0 2024-03-18T16:16:14.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.619+0000 7f4db1a00640 1 -- 172.21.15.39:0/2715263784 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4dac1a63e0 con 0x7f4dac1017f0 2024-03-18T16:16:14.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.624+0000 7f4db47cf740 1 Processor -- start 2024-03-18T16:16:14.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.625+0000 7f4db47cf740 1 -- start start 2024-03-18T16:16:14.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.625+0000 7f4da3e00640 1 -- 172.21.15.39:0/2715263784 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f4d9c044020 con 0x7f4dac1017f0 2024-03-18T16:16:14.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.626+0000 7f4db47cf740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557062a2f7e0 0x557062a2fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:14.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.626+0000 7f4db47cf740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557062a301d0 0x557062a34cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:14.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.626+0000 7f4db47cf740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557062a35340 0x557062a377d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:14.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.626+0000 7f4db47cf740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x557062992f80 con 0x557062a301d0 2024-03-18T16:16:14.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.626+0000 7f4db47cf740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55706293bad0 con 0x557062a2f7e0 2024-03-18T16:16:14.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.626+0000 7f4db47cf740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55706298f020 con 0x557062a35340 2024-03-18T16:16:14.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.626+0000 7f4dab400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557062a2f7e0 0x557062a2fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:14.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.626+0000 7f4dab400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557062a2f7e0 0x557062a2fc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:53980/0 (socket says 172.21.15.39:53980) 2024-03-18T16:16:14.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.626+0000 7f4dab400640 1 -- 172.21.15.39:0/1264353320 learned_addr learned my addr 172.21.15.39:0/1264353320 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:16:14.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.627+0000 7f4da3e00640 1 --2- 172.21.15.39:0/2715263784 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f4d900a28f0 0x7f4d900a4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:14.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.627+0000 7f4da3e00640 1 -- 172.21.15.39:0/2715263784 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f4d9c0f0b40 con 0x7f4dac1017f0 2024-03-18T16:16:14.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.628+0000 7f4dab400640 1 -- 172.21.15.39:0/1264353320 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557062a35340 msgr2=0x557062a377d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:16:14.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.628+0000 7f4dab400640 1 --2- 172.21.15.39:0/1264353320 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557062a35340 0x557062a377d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:14.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.628+0000 7f4dab400640 1 -- 172.21.15.39:0/1264353320 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557062a301d0 msgr2=0x557062a34cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:14.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.628+0000 7f4dab400640 1 --2- 172.21.15.39:0/1264353320 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557062a301d0 0x557062a34cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:14.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.628+0000 7f4dab400640 1 -- 172.21.15.39:0/1264353320 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5570629536f0 con 0x557062a2f7e0 2024-03-18T16:16:14.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.629+0000 7f4dab400640 1 --2- 172.21.15.39:0/2715263784 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f4d900a28f0 0x7f4d900a4db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:14.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.632+0000 7f4dab400640 1 --2- 172.21.15.39:0/1264353320 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557062a2f7e0 0x557062a2fc00 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f4da4009ca0 tx=0x7f4da4011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2723bf129b1c5e4b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:14.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.632+0000 7f4dab400640 1 --2- 172.21.15.39:0/2715263784 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f4d900a28f0 0x7f4d900a4db0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f4da4016000 tx=0x7f4da4018000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:14.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.632+0000 7f4d9be00640 1 -- 172.21.15.39:0/1264353320 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4da400ee20 con 0x557062a2f7e0 2024-03-18T16:16:14.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.632+0000 7f4d9be00640 1 -- 172.21.15.39:0/1264353320 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4da4009e50 con 0x557062a2f7e0 2024-03-18T16:16:14.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.632+0000 7f4d9be00640 1 -- 172.21.15.39:0/1264353320 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4da401b680 con 0x557062a2f7e0 2024-03-18T16:16:14.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.634+0000 7f4db47cf740 1 -- 172.21.15.39:0/1264353320 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557062a2f7e0 msgr2=0x557062a2fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:14.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.634+0000 7f4db47cf740 1 --2- 172.21.15.39:0/1264353320 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557062a2f7e0 0x557062a2fc00 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f4da4009ca0 tx=0x7f4da4011810 comp rx=0 tx=0).stop 2024-03-18T16:16:14.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.635+0000 7f4db47cf740 1 -- 172.21.15.39:0/1264353320 shutdown_connections 2024-03-18T16:16:14.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.635+0000 7f4db47cf740 1 --2- 172.21.15.39:0/1264353320 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557062a35340 0x557062a377d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:14.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.635+0000 7f4db47cf740 1 --2- 172.21.15.39:0/1264353320 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557062a301d0 0x557062a34cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:14.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.635+0000 7f4db47cf740 1 --2- 172.21.15.39:0/1264353320 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557062a2f7e0 0x557062a2fc00 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:14.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.635+0000 7f4db47cf740 1 -- 172.21.15.39:0/1264353320 >> 172.21.15.39:0/1264353320 conn(0x5570629f7930 msgr2=0x557062a2b150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:14.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.635+0000 7f4db47cf740 1 -- 172.21.15.39:0/1264353320 shutdown_connections 2024-03-18T16:16:14.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.635+0000 7f4db47cf740 1 -- 172.21.15.39:0/1264353320 wait complete. 2024-03-18T16:16:14.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.635+0000 7f4db47cf740 1 -- start start 2024-03-18T16:16:14.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.636+0000 7f4db47cf740 1 Processor -- start 2024-03-18T16:16:14.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.636+0000 7f4db47cf740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557062a2f7e0 0x557062a39c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:14.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.637+0000 7f4db47cf740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557062a301d0 0x557062a3a160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:14.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.637+0000 7f4db47cf740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557062a35340 0x557062a411a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:14.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.637+0000 7f4db47cf740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x557062a382f0 con 0x557062a35340 2024-03-18T16:16:14.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.637+0000 7f4db47cf740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55706294ef30 con 0x557062a301d0 2024-03-18T16:16:14.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.637+0000 7f4db47cf740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x557062a38470 con 0x557062a2f7e0 2024-03-18T16:16:14.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.637+0000 7f4da2000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557062a301d0 0x557062a3a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:14.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.637+0000 7f4da2000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557062a301d0 0x557062a3a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:53988/0 (socket says 172.21.15.39:53988) 2024-03-18T16:16:14.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.637+0000 7f4da2000640 1 -- 172.21.15.39:0/3044693313 learned_addr learned my addr 172.21.15.39:0/3044693313 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:16:14.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.637+0000 7f4daaa00640 1 --2- 172.21.15.39:0/3044693313 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557062a2f7e0 0x557062a39c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:14.641 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.638+0000 7f4dab400640 1 --2- 172.21.15.39:0/3044693313 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557062a35340 0x557062a411a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:14.641 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.638+0000 7f4db47cf740 10 client.0 _ll_register_callbacks cb 0x55706293cbc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:16:14.641 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.639+0000 7f4daaa00640 1 -- 172.21.15.39:0/3044693313 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557062a301d0 msgr2=0x557062a3a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:14.641 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.639+0000 7f4daaa00640 1 --2- 172.21.15.39:0/3044693313 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557062a301d0 0x557062a3a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:14.641 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.639+0000 7f4daaa00640 1 -- 172.21.15.39:0/3044693313 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557062a35340 msgr2=0x557062a411a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:14.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.639+0000 7f4daaa00640 1 --2- 172.21.15.39:0/3044693313 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557062a35340 0x557062a411a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:14.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.639+0000 7f4daaa00640 1 -- 172.21.15.39:0/3044693313 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55706276f7e0 con 0x557062a2f7e0 2024-03-18T16:16:14.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.639+0000 7f4daaa00640 1 --2- 172.21.15.39:0/3044693313 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557062a2f7e0 0x557062a39c20 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x557062a30c40 tx=0x7f4d9c0b94e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=54085fcc137271c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:14.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.639+0000 7f4d98c00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:16:14.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.639+0000 7f4d98c00640 1 -- 172.21.15.39:0/3044693313 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d9c0098d0 con 0x557062a2f7e0 2024-03-18T16:16:14.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.640+0000 7f4d98c00640 1 -- 172.21.15.39:0/3044693313 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f4d9c0c5400 con 0x557062a2f7e0 2024-03-18T16:16:14.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.640+0000 7f4db47cf740 1 -- 172.21.15.39:0/3044693313 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4da4009e50 con 0x557062a2f7e0 2024-03-18T16:16:14.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.640+0000 7f4d98c00640 1 -- 172.21.15.39:0/3044693313 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f4d9c0bd880 con 0x557062a2f7e0 2024-03-18T16:16:14.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.640+0000 7f4db47cf740 10 client.26878 fetch_fsmap learned FSMap version 44 2024-03-18T16:16:14.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.640+0000 7f4db47cf740 1 -- 172.21.15.39:0/3044693313 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x5570627cff70 con 0x557062a2f7e0 2024-03-18T16:16:14.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.641+0000 7f4d98c00640 1 -- 172.21.15.39:0/3044693313 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4da4009e50 con 0x557062a2f7e0 2024-03-18T16:16:14.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.641+0000 7f4d98c00640 1 -- 172.21.15.39:0/3044693313 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7f4d9c03c030 con 0x557062a2f7e0 2024-03-18T16:16:14.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.642+0000 7f4db47cf740 10 client.26878 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:16:14.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.642+0000 7f4db47cf740 10 client.26878 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:16:14.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.642+0000 7f4db47cf740 20 client.26878 populate_metadata read hostname 'smithi039' 2024-03-18T16:16:14.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.642+0000 7f4db47cf740 1 --2- 172.21.15.39:0/3044693313 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x557062a568b0 0x557062a58ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:14.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.642+0000 7f4db47cf740 4 client.26878 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:16:14.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.642+0000 7f4db47cf740 1 -- 172.21.15.39:0/3044693313 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5570629a04c0 con 0x557062a568b0 2024-03-18T16:16:14.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.642+0000 7f4da2000640 1 --2- 172.21.15.39:0/3044693313 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x557062a568b0 0x557062a58ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:14.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.643+0000 7f4da2000640 1 --2- 172.21.15.39:0/3044693313 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x557062a568b0 0x557062a58ce0 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=949cfabac9b08f0a server_cookie=33999ec686a8311b in_seq=0 out_seq=0 2024-03-18T16:16:14.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.643+0000 7f4d98c00640 10 client.26878 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:16:14.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.644+0000 7f4d98c00640 1 -- 172.21.15.39:0/3044693313 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5570629a04c0 con 0x557062a568b0 2024-03-18T16:16:14.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.644+0000 7f4d98c00640 10 client.26878 handle_command_reply: tid=0 2024-03-18T16:16:14.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.644+0000 7f4db47cf740 1 client.26878 shutdown 2024-03-18T16:16:14.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.645+0000 7f4db47cf740 1 -- 172.21.15.39:0/3044693313 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557062a2f7e0 msgr2=0x557062a39c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:14.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.645+0000 7f4db47cf740 1 --2- 172.21.15.39:0/3044693313 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557062a2f7e0 0x557062a39c20 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x557062a30c40 tx=0x7f4d9c0b94e0 comp rx=0 tx=0).stop 2024-03-18T16:16:14.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.645+0000 7f4db47cf740 1 -- 172.21.15.39:0/3044693313 shutdown_connections 2024-03-18T16:16:14.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.645+0000 7f4db47cf740 1 --2- 172.21.15.39:0/3044693313 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x557062a568b0 0x557062a58ce0 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:14.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.645+0000 7f4db47cf740 1 --2- 172.21.15.39:0/3044693313 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557062a301d0 0x557062a3a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:14.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.645+0000 7f4db47cf740 1 --2- 172.21.15.39:0/3044693313 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557062a35340 0x557062a411a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:14.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.645+0000 7f4db47cf740 1 --2- 172.21.15.39:0/3044693313 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557062a2f7e0 0x557062a39c20 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:14.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.645+0000 7f4db47cf740 1 -- 172.21.15.39:0/3044693313 >> 172.21.15.39:0/3044693313 conn(0x5570629f7930 msgr2=0x557062a2a350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:14.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.645+0000 7f4d98c00640 0 client.26878 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:16:14.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.645+0000 7f4db47cf740 1 -- 172.21.15.39:0/3044693313 shutdown_connections 2024-03-18T16:16:14.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.646+0000 7f4db47cf740 1 -- 172.21.15.39:0/3044693313 wait complete. 2024-03-18T16:16:14.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.646+0000 7f4db47cf740 20 client.26878 trim_cache size 0 max 16384 2024-03-18T16:16:14.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.681+0000 7f4db47cf740 1 Processor -- start 2024-03-18T16:16:14.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.681+0000 7f4db47cf740 1 -- start start 2024-03-18T16:16:14.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.681+0000 7f4db47cf740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557062a2f7e0 0x557062a39f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:14.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.681+0000 7f4db47cf740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557062a301d0 0x557062a41130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:14.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.681+0000 7f4db47cf740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557062a35340 0x557062a29ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:14.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.681+0000 7f4db47cf740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f4dac1a82c0 con 0x557062a35340 2024-03-18T16:16:14.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.681+0000 7f4db47cf740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f4dac1a9e00 con 0x557062a301d0 2024-03-18T16:16:14.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.681+0000 7f4db47cf740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f4dac1a8990 con 0x557062a2f7e0 2024-03-18T16:16:14.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.682+0000 7f4da2000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557062a301d0 0x557062a41130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:14.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.682+0000 7f4da2000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557062a301d0 0x557062a41130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:53994/0 (socket says 172.21.15.39:53994) 2024-03-18T16:16:14.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.682+0000 7f4da2000640 1 -- 172.21.15.39:0/3050174783 learned_addr learned my addr 172.21.15.39:0/3050174783 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:16:14.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.682+0000 7f4dab400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557062a35340 0x557062a29ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:14.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.682+0000 7f4daaa00640 1 --2- 172.21.15.39:0/3050174783 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557062a2f7e0 0x557062a39f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:14.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.682+0000 7f4da2000640 1 -- 172.21.15.39:0/3050174783 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557062a2f7e0 msgr2=0x557062a39f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:14.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.682+0000 7f4da2000640 1 --2- 172.21.15.39:0/3050174783 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557062a2f7e0 0x557062a39f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:14.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.682+0000 7f4da2000640 1 -- 172.21.15.39:0/3050174783 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557062a35340 msgr2=0x557062a29ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:14.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.682+0000 7f4da2000640 1 --2- 172.21.15.39:0/3050174783 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557062a35340 0x557062a29ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:14.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.682+0000 7f4da2000640 1 -- 172.21.15.39:0/3050174783 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557062667170 con 0x557062a301d0 2024-03-18T16:16:14.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.684+0000 7f4da2000640 1 --2- 172.21.15.39:0/3050174783 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557062a301d0 0x557062a41130 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f4d94012360 tx=0x7f4d94014b00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f2b6a0a39d16ec1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:14.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.684+0000 7f4d9b400640 1 -- 172.21.15.39:0/3050174783 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d9401dd90 con 0x557062a301d0 2024-03-18T16:16:14.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.684+0000 7f4d9b400640 1 -- 172.21.15.39:0/3050174783 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d94022070 con 0x557062a301d0 2024-03-18T16:16:14.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.684+0000 7f4d9b400640 1 -- 172.21.15.39:0/3050174783 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d94012b40 con 0x557062a301d0 2024-03-18T16:16:14.688 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.685+0000 7f4dab400640 1 --2- 172.21.15.39:0/3050174783 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557062a35340 0x557062a29ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:16:14.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.686+0000 7f4db47cf740 1 -- 172.21.15.39:0/3050174783 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557062a301d0 msgr2=0x557062a41130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:14.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.686+0000 7f4db47cf740 1 --2- 172.21.15.39:0/3050174783 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557062a301d0 0x557062a41130 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f4d94012360 tx=0x7f4d94014b00 comp rx=0 tx=0).stop 2024-03-18T16:16:14.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.686+0000 7f4db47cf740 1 -- 172.21.15.39:0/3050174783 shutdown_connections 2024-03-18T16:16:14.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.686+0000 7f4db47cf740 1 --2- 172.21.15.39:0/3050174783 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557062a301d0 0x557062a41130 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:14.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.686+0000 7f4db47cf740 1 --2- 172.21.15.39:0/3050174783 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557062a35340 0x557062a29ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:14.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.686+0000 7f4db47cf740 1 --2- 172.21.15.39:0/3050174783 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557062a2f7e0 0x557062a39f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:14.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.686+0000 7f4db47cf740 1 -- 172.21.15.39:0/3050174783 >> 172.21.15.39:0/3050174783 conn(0x5570629f7930 msgr2=0x557062a39830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:14.689 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.686+0000 7f4db47cf740 1 -- 172.21.15.39:0/3050174783 shutdown_connections 2024-03-18T16:16:14.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.686+0000 7f4db47cf740 1 -- 172.21.15.39:0/3050174783 wait complete. 2024-03-18T16:16:14.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.687+0000 7f4db47cf740 1 -- start start 2024-03-18T16:16:14.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.687+0000 7f4db47cf740 1 Processor -- start 2024-03-18T16:16:14.690 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.688+0000 7f4db47cf740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557062a2f7e0 0x557062a3c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:14.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.688+0000 7f4db47cf740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557062a301d0 0x557062a3cc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:14.691 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.688+0000 7f4db47cf740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557062a35340 0x557062a3fb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:14.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.689+0000 7f4daaa00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557062a2f7e0 0x557062a3c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:14.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.689+0000 7f4daaa00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557062a2f7e0 0x557062a3c6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:54006/0 (socket says 172.21.15.39:54006) 2024-03-18T16:16:14.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.689+0000 7f4daaa00640 1 -- 172.21.15.39:0/2466749864 learned_addr learned my addr 172.21.15.39:0/2466749864 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:16:14.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.689+0000 7f4db47cf740 1 -- 172.21.15.39:0/2466749864 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f4dac1a9730 con 0x557062a301d0 2024-03-18T16:16:14.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.689+0000 7f4db47cf740 1 -- 172.21.15.39:0/2466749864 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f4dac1a9060 con 0x557062a2f7e0 2024-03-18T16:16:14.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.689+0000 7f4db47cf740 1 -- 172.21.15.39:0/2466749864 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f4dac198cb0 con 0x557062a35340 2024-03-18T16:16:14.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.689+0000 7f4db47cf740 10 client.0 _ll_register_callbacks cb 0x7f4d9c0b8d00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:16:14.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.689+0000 7f4da2000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557062a301d0 0x557062a3cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:14.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.689+0000 7f4dab400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557062a35340 0x557062a3fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:14.693 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.691+0000 7f4daaa00640 1 -- 172.21.15.39:0/2466749864 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557062a35340 msgr2=0x557062a3fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:14.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.691+0000 7f4daaa00640 1 --2- 172.21.15.39:0/2466749864 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557062a35340 0x557062a3fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:14.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.691+0000 7f4daaa00640 1 -- 172.21.15.39:0/2466749864 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557062a301d0 msgr2=0x557062a3cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:14.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.691+0000 7f4daaa00640 1 --2- 172.21.15.39:0/2466749864 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557062a301d0 0x557062a3cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:14.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.691+0000 7f4daaa00640 1 -- 172.21.15.39:0/2466749864 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557062a40150 con 0x557062a2f7e0 2024-03-18T16:16:14.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.691+0000 7f4dab400640 1 --2- 172.21.15.39:0/2466749864 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557062a35340 0x557062a3fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:16:14.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.691+0000 7f4da2000640 1 --2- 172.21.15.39:0/2466749864 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557062a301d0 0x557062a3cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:16:14.694 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.692+0000 7f4daaa00640 1 --2- 172.21.15.39:0/2466749864 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557062a2f7e0 0x557062a3c6c0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f4d9c0bce80 tx=0x7f4d9c0b8db0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=25672c349b724e57 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:14.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.692+0000 7f4d98c00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:16:14.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.692+0000 7f4d98c00640 1 -- 172.21.15.39:0/2466749864 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d9c0bccc0 con 0x557062a2f7e0 2024-03-18T16:16:14.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.692+0000 7f4d98c00640 1 -- 172.21.15.39:0/2466749864 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f4d9c0c3290 con 0x557062a2f7e0 2024-03-18T16:16:14.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.692+0000 7f4db47cf740 1 -- 172.21.15.39:0/2466749864 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4d94022070 con 0x557062a2f7e0 2024-03-18T16:16:14.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.693+0000 7f4d98c00640 1 -- 172.21.15.39:0/2466749864 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f4d9c0dd0b0 con 0x557062a2f7e0 2024-03-18T16:16:14.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.693+0000 7f4db47cf740 10 client.26906 fetch_fsmap learned FSMap version 44 2024-03-18T16:16:14.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.693+0000 7f4db47cf740 1 -- 172.21.15.39:0/2466749864 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x557062a41500 con 0x557062a2f7e0 2024-03-18T16:16:14.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.693+0000 7f4d98c00640 1 -- 172.21.15.39:0/2466749864 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d94022070 con 0x557062a2f7e0 2024-03-18T16:16:14.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.693+0000 7f4d98c00640 1 -- 172.21.15.39:0/2466749864 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7f4d9c0b8ed0 con 0x557062a2f7e0 2024-03-18T16:16:14.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.694+0000 7f4db47cf740 10 client.26906 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:16:14.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.694+0000 7f4db47cf740 10 client.26906 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:16:14.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.694+0000 7f4db47cf740 20 client.26906 populate_metadata read hostname 'smithi039' 2024-03-18T16:16:14.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.694+0000 7f4db47cf740 1 --2- 172.21.15.39:0/2466749864 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x557062ace1d0 0x557062ad05f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:14.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.694+0000 7f4db47cf740 4 client.26906 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-18T16:16:14.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.694+0000 7f4db47cf740 1 -- 172.21.15.39:0/2466749864 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55706299a810 con 0x557062ace1d0 2024-03-18T16:16:14.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.694+0000 7f4da2000640 1 --2- 172.21.15.39:0/2466749864 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x557062ace1d0 0x557062ad05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:14.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.695+0000 7f4da2000640 1 --2- 172.21.15.39:0/2466749864 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x557062ace1d0 0x557062ad05f0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c15b16a400edce3b server_cookie=ab7f637b42bd6fa8 in_seq=0 out_seq=0 2024-03-18T16:16:14.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.695+0000 7f4d98c00640 10 client.26906 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:16:14.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.695+0000 7f4d98c00640 1 -- 172.21.15.39:0/2466749864 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55706299a810 con 0x557062ace1d0 2024-03-18T16:16:14.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.695+0000 7f4d98c00640 10 client.26906 handle_command_reply: tid=0 2024-03-18T16:16:14.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.695+0000 7f4db47cf740 1 client.26906 shutdown 2024-03-18T16:16:14.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.696+0000 7f4db47cf740 1 -- 172.21.15.39:0/2466749864 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557062a2f7e0 msgr2=0x557062a3c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:14.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.696+0000 7f4db47cf740 1 --2- 172.21.15.39:0/2466749864 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557062a2f7e0 0x557062a3c6c0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f4d9c0bce80 tx=0x7f4d9c0b8db0 comp rx=0 tx=0).stop 2024-03-18T16:16:14.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.696+0000 7f4db47cf740 1 -- 172.21.15.39:0/2466749864 shutdown_connections 2024-03-18T16:16:14.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.696+0000 7f4db47cf740 1 --2- 172.21.15.39:0/2466749864 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x557062ace1d0 0x557062ad05f0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:14.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.696+0000 7f4db47cf740 1 --2- 172.21.15.39:0/2466749864 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557062a35340 0x557062a3fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:14.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.696+0000 7f4db47cf740 1 --2- 172.21.15.39:0/2466749864 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557062a301d0 0x557062a3cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:14.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.696+0000 7f4db47cf740 1 --2- 172.21.15.39:0/2466749864 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557062a2f7e0 0x557062a3c6c0 secure :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f4d9c0bce80 tx=0x7f4d9c0b8db0 comp rx=0 tx=0).stop 2024-03-18T16:16:14.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.696+0000 7f4db47cf740 1 -- 172.21.15.39:0/2466749864 >> 172.21.15.39:0/2466749864 conn(0x5570629f7930 msgr2=0x557062a2e390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:14.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.696+0000 7f4d98c00640 0 client.26906 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:16:14.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.697+0000 7f4db47cf740 1 -- 172.21.15.39:0/2466749864 shutdown_connections 2024-03-18T16:16:14.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.697+0000 7f4db47cf740 1 -- 172.21.15.39:0/2466749864 wait complete. 2024-03-18T16:16:14.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.698+0000 7f4db47cf740 20 client.26906 trim_cache size 0 max 16384 2024-03-18T16:16:14.702 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-03-18T16:16:14.702 INFO:teuthology.orchestra.run.smithi039.stdout: "return_code": 0, 2024-03-18T16:16:14.703 INFO:teuthology.orchestra.run.smithi039.stdout: "scrub_tag": "47698cef-41a7-4063-b34e-5200783d2b8c", 2024-03-18T16:16:14.703 INFO:teuthology.orchestra.run.smithi039.stdout: "mode": "asynchronous" 2024-03-18T16:16:14.703 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-03-18T16:16:14.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.701+0000 7f4db1a00640 1 -- 172.21.15.39:0/2715263784 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f4d900a28f0 msgr2=0x7f4d900a4db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:14.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.701+0000 7f4db1a00640 1 --2- 172.21.15.39:0/2715263784 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f4d900a28f0 0x7f4d900a4db0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f4da4016000 tx=0x7f4da4018000 comp rx=0 tx=0).stop 2024-03-18T16:16:14.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.701+0000 7f4db1a00640 1 -- 172.21.15.39:0/2715263784 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4dac1017f0 msgr2=0x7f4dac19e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:14.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.701+0000 7f4db1a00640 1 --2- 172.21.15.39:0/2715263784 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4dac1017f0 0x7f4dac19e120 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f4d9c037040 tx=0x7f4d9c032ea0 comp rx=0 tx=0).stop 2024-03-18T16:16:14.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.701+0000 7f4db1a00640 1 -- 172.21.15.39:0/2715263784 shutdown_connections 2024-03-18T16:16:14.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.701+0000 7f4db1a00640 1 --2- 172.21.15.39:0/2715263784 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4dac1033a0 0x7f4dac1a56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:14.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.701+0000 7f4db1a00640 1 --2- 172.21.15.39:0/2715263784 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f4d900a28f0 0x7f4d900a4db0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:14.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.701+0000 7f4db1a00640 1 --2- 172.21.15.39:0/2715263784 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4dac1029e0 0x7f4dac19e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:14.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.701+0000 7f4db1a00640 1 --2- 172.21.15.39:0/2715263784 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4dac1017f0 0x7f4dac19e120 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:14.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.701+0000 7f4db1a00640 1 -- 172.21.15.39:0/2715263784 >> 172.21.15.39:0/2715263784 conn(0x7f4dac0fcd80 msgr2=0x7f4dac105d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:14.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.702+0000 7f4db1a00640 1 -- 172.21.15.39:0/2715263784 shutdown_connections 2024-03-18T16:16:14.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.702+0000 7f4db1a00640 1 -- 172.21.15.39:0/2715263784 wait complete. 2024-03-18T16:16:14.730 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T16:16:14.731 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T16:16:15.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.999+0000 7fb5ed000640 1 -- 172.21.15.39:0/1714329152 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb5e81055d0 msgr2=0x7fb5e810be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:15.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.999+0000 7fb5ed000640 1 --2- 172.21.15.39:0/1714329152 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb5e81055d0 0x7fb5e810be60 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fb5e0009be0 tx=0x7fb5e002bf10 comp rx=0 tx=0).stop 2024-03-18T16:16:15.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.999+0000 7fb5ed000640 1 -- 172.21.15.39:0/1714329152 shutdown_connections 2024-03-18T16:16:15.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.999+0000 7fb5ed000640 1 --2- 172.21.15.39:0/1714329152 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb5e81055d0 0x7fb5e810be60 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:15.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.999+0000 7fb5ed000640 1 --2- 172.21.15.39:0/1714329152 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb5e8104c10 0x7fb5e8105090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:15.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.999+0000 7fb5ed000640 1 --2- 172.21.15.39:0/1714329152 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb5e8103a20 0x7fb5e8103e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:15.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:14.999+0000 7fb5ed000640 1 -- 172.21.15.39:0/1714329152 >> 172.21.15.39:0/1714329152 conn(0x7fb5e80fef90 msgr2=0x7fb5e81013d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:15.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.000+0000 7fb5ed000640 1 -- 172.21.15.39:0/1714329152 shutdown_connections 2024-03-18T16:16:15.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.000+0000 7fb5ed000640 1 -- 172.21.15.39:0/1714329152 wait complete. 2024-03-18T16:16:15.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.001+0000 7fb5ed000640 1 Processor -- start 2024-03-18T16:16:15.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.002+0000 7fb5ed000640 1 -- start start 2024-03-18T16:16:15.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.002+0000 7fb5ed000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb5e8103a20 0x7fb5e8079590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:15.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.002+0000 7fb5ed000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb5e8104c10 0x7fb5e8079ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:15.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.002+0000 7fb5ed000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb5e81055d0 0x7fb5e8076100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:15.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.002+0000 7fb5ed000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fb5e810f6d0 con 0x7fb5e81055d0 2024-03-18T16:16:15.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.002+0000 7fb5ed000640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fb5e810f550 con 0x7fb5e8103a20 2024-03-18T16:16:15.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.002+0000 7fb5ed000640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fb5e810f850 con 0x7fb5e8104c10 2024-03-18T16:16:15.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.002+0000 7fb5e5e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb5e8103a20 0x7fb5e8079590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:15.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.002+0000 7fb5e5e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb5e8103a20 0x7fb5e8079590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:54018/0 (socket says 172.21.15.39:54018) 2024-03-18T16:16:15.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.002+0000 7fb5e5e00640 1 -- 172.21.15.39:0/1431992748 learned_addr learned my addr 172.21.15.39:0/1431992748 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:16:15.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.003+0000 7fb5e5400640 1 --2- 172.21.15.39:0/1431992748 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb5e8104c10 0x7fb5e8079ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:15.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.003+0000 7fb5e5e00640 1 -- 172.21.15.39:0/1431992748 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb5e8104c10 msgr2=0x7fb5e8079ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:15.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.003+0000 7fb5e5e00640 1 --2- 172.21.15.39:0/1431992748 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb5e8104c10 0x7fb5e8079ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:15.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.003+0000 7fb5e5e00640 1 -- 172.21.15.39:0/1431992748 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb5e81055d0 msgr2=0x7fb5e8076100 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T16:16:15.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.003+0000 7fb5e6800640 1 --2- 172.21.15.39:0/1431992748 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb5e81055d0 0x7fb5e8076100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:15.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.003+0000 7fb5e5e00640 1 --2- 172.21.15.39:0/1431992748 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb5e81055d0 0x7fb5e8076100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:15.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.003+0000 7fb5e5e00640 1 -- 172.21.15.39:0/1431992748 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5e80769c0 con 0x7fb5e8103a20 2024-03-18T16:16:15.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.003+0000 7fb5e5400640 1 --2- 172.21.15.39:0/1431992748 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb5e8104c10 0x7fb5e8079ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:16:15.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.003+0000 7fb5e5e00640 1 --2- 172.21.15.39:0/1431992748 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb5e8103a20 0x7fb5e8079590 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fb5d800e9c0 tx=0x7fb5d800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:15.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.004+0000 7fb5dea00640 1 -- 172.21.15.39:0/1431992748 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5d800cde0 con 0x7fb5e8103a20 2024-03-18T16:16:15.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.004+0000 7fb5ed000640 1 -- 172.21.15.39:0/1431992748 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5e81a6190 con 0x7fb5e8103a20 2024-03-18T16:16:15.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.004+0000 7fb5dea00640 1 -- 172.21.15.39:0/1431992748 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb5d8010910 con 0x7fb5e8103a20 2024-03-18T16:16:15.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.004+0000 7fb5ed000640 1 -- 172.21.15.39:0/1431992748 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5e81a65f0 con 0x7fb5e8103a20 2024-03-18T16:16:15.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.004+0000 7fb5dea00640 1 -- 172.21.15.39:0/1431992748 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5d8018a70 con 0x7fb5e8103a20 2024-03-18T16:16:15.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.009+0000 7fb5dea00640 1 -- 172.21.15.39:0/1431992748 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fb5d801f020 con 0x7fb5e8103a20 2024-03-18T16:16:15.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.009+0000 7fb5efc28740 1 Processor -- start 2024-03-18T16:16:15.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.009+0000 7fb5efc28740 1 -- start start 2024-03-18T16:16:15.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.010+0000 7fb5efc28740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562e2059a810 0x562e2059ac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:15.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.010+0000 7fb5efc28740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562e2059b200 0x562e2059fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:15.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.010+0000 7fb5efc28740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562e205a0370 0x562e205a2800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:15.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.010+0000 7fb5efc28740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x562e204ff1f0 con 0x562e205a0370 2024-03-18T16:16:15.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.010+0000 7fb5efc28740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x562e204a7a80 con 0x562e2059b200 2024-03-18T16:16:15.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.010+0000 7fb5efc28740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x562e204fb290 con 0x562e2059a810 2024-03-18T16:16:15.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.011+0000 7fb5dea00640 1 --2- 172.21.15.39:0/1431992748 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb5cc0a2900 0x7fb5cc0a4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:15.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.011+0000 7fb5dea00640 1 -- 172.21.15.39:0/1431992748 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fb5d80cf240 con 0x7fb5e8103a20 2024-03-18T16:16:15.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.012+0000 7fb5e5400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562e205a0370 0x562e205a2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:15.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.012+0000 7fb5e6800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562e2059a810 0x562e2059ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:15.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.012+0000 7fb5e5400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562e205a0370 0x562e205a2800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:46680/0 (socket says 172.21.15.39:46680) 2024-03-18T16:16:15.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.012+0000 7fb5e5400640 1 -- 172.21.15.39:0/3943252679 learned_addr learned my addr 172.21.15.39:0/3943252679 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:16:15.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.012+0000 7fb5e5e00640 1 --2- 172.21.15.39:0/3943252679 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562e2059b200 0x562e2059fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:15.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.012+0000 7fb5e6800640 1 --2- 172.21.15.39:0/1431992748 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb5cc0a2900 0x7fb5cc0a4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:15.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.013+0000 7fb5e5400640 1 -- 172.21.15.39:0/3943252679 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562e2059a810 msgr2=0x562e2059ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:15.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.013+0000 7fb5e5400640 1 --2- 172.21.15.39:0/3943252679 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562e2059a810 0x562e2059ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:15.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.013+0000 7fb5e5400640 1 -- 172.21.15.39:0/3943252679 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562e2059b200 msgr2=0x562e2059fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:15.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.013+0000 7fb5e5400640 1 --2- 172.21.15.39:0/3943252679 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562e2059b200 0x562e2059fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:15.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.013+0000 7fb5e5400640 1 -- 172.21.15.39:0/3943252679 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e202dd7d0 con 0x562e205a0370 2024-03-18T16:16:15.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.013+0000 7fb5e5400640 1 --2- 172.21.15.39:0/3943252679 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562e205a0370 0x562e205a2800 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fb5d4009c20 tx=0x7fb5d400f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ced169367162b1e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:15.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.013+0000 7fb5e6800640 1 --2- 172.21.15.39:0/1431992748 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb5cc0a2900 0x7fb5cc0a4dc0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x562e2059ba60 tx=0x7fb5e002bf10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:15.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.014+0000 7fb5d3400640 1 -- 172.21.15.39:0/3943252679 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5d4004020 con 0x562e205a0370 2024-03-18T16:16:15.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.014+0000 7fb5d3400640 1 -- 172.21.15.39:0/3943252679 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb5d40041c0 con 0x562e205a0370 2024-03-18T16:16:15.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.014+0000 7fb5d3400640 1 -- 172.21.15.39:0/3943252679 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5d4011680 con 0x562e205a0370 2024-03-18T16:16:15.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.014+0000 7fb5efc28740 1 -- 172.21.15.39:0/3943252679 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562e205a0370 msgr2=0x562e205a2800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:15.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.014+0000 7fb5efc28740 1 --2- 172.21.15.39:0/3943252679 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562e205a0370 0x562e205a2800 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fb5d4009c20 tx=0x7fb5d400f730 comp rx=0 tx=0).stop 2024-03-18T16:16:15.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.015+0000 7fb5efc28740 1 -- 172.21.15.39:0/3943252679 shutdown_connections 2024-03-18T16:16:15.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.015+0000 7fb5efc28740 1 --2- 172.21.15.39:0/3943252679 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562e2059b200 0x562e2059fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:15.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.015+0000 7fb5efc28740 1 --2- 172.21.15.39:0/3943252679 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562e205a0370 0x562e205a2800 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:15.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.015+0000 7fb5efc28740 1 --2- 172.21.15.39:0/3943252679 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562e2059a810 0x562e2059ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:15.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.015+0000 7fb5efc28740 1 -- 172.21.15.39:0/3943252679 >> 172.21.15.39:0/3943252679 conn(0x562e20595d80 msgr2=0x562e20596180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:15.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.015+0000 7fb5efc28740 1 -- 172.21.15.39:0/3943252679 shutdown_connections 2024-03-18T16:16:15.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.015+0000 7fb5efc28740 1 -- 172.21.15.39:0/3943252679 wait complete. 2024-03-18T16:16:15.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.015+0000 7fb5efc28740 1 -- start start 2024-03-18T16:16:15.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.016+0000 7fb5efc28740 1 Processor -- start 2024-03-18T16:16:15.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.016+0000 7fb5efc28740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562e2059a810 0x562e205a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:15.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.016+0000 7fb5efc28740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562e2059b200 0x562e205a5200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:15.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.016+0000 7fb5efc28740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562e205a0370 0x562e205ac230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:15.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.016+0000 7fb5efc28740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x562e205a3320 con 0x562e2059a810 2024-03-18T16:16:15.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.016+0000 7fb5efc28740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x562e204baed0 con 0x562e2059b200 2024-03-18T16:16:15.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.016+0000 7fb5efc28740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x562e205a34a0 con 0x562e205a0370 2024-03-18T16:16:15.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.016+0000 7fb5efc28740 10 client.0 _ll_register_callbacks cb 0x562e204f6220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:16:15.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.017+0000 7fb5e5e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562e2059a810 0x562e205a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:15.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.017+0000 7fb5e5e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562e2059a810 0x562e205a4cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:46682/0 (socket says 172.21.15.39:46682) 2024-03-18T16:16:15.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.017+0000 7fb5e5e00640 1 -- 172.21.15.39:0/4224917154 learned_addr learned my addr 172.21.15.39:0/4224917154 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:16:15.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.017+0000 7fb5e5400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562e2059b200 0x562e205a5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:15.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.017+0000 7fb5e6800640 1 --2- 172.21.15.39:0/4224917154 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562e205a0370 0x562e205ac230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:15.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.017+0000 7fb5e5e00640 1 -- 172.21.15.39:0/4224917154 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562e205a0370 msgr2=0x562e205ac230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:15.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.017+0000 7fb5e5e00640 1 --2- 172.21.15.39:0/4224917154 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562e205a0370 0x562e205ac230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:15.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.017+0000 7fb5e5e00640 1 -- 172.21.15.39:0/4224917154 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562e2059b200 msgr2=0x562e205a5200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:15.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.017+0000 7fb5e5e00640 1 --2- 172.21.15.39:0/4224917154 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562e2059b200 0x562e205a5200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:15.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.018+0000 7fb5e5e00640 1 -- 172.21.15.39:0/4224917154 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e202133f0 con 0x562e2059a810 2024-03-18T16:16:15.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.018+0000 7fb5e5e00640 1 --2- 172.21.15.39:0/4224917154 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562e2059a810 0x562e205a4cc0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fb5d800c6a0 tx=0x7fb5d809bfb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6eff6c82ca9bee4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:15.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.018+0000 7fb5c7e00640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-03-18T16:16:15.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.019+0000 7fb5c7e00640 1 -- 172.21.15.39:0/4224917154 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5d80a3f90 con 0x562e2059a810 2024-03-18T16:16:15.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.019+0000 7fb5c7e00640 1 -- 172.21.15.39:0/4224917154 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fb5d80a3b00 con 0x562e2059a810 2024-03-18T16:16:15.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.019+0000 7fb5efc28740 1 -- 172.21.15.39:0/4224917154 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb5d40041c0 con 0x562e2059a810 2024-03-18T16:16:15.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.019+0000 7fb5c7e00640 1 -- 172.21.15.39:0/4224917154 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fb5d80a3d00 con 0x562e2059a810 2024-03-18T16:16:15.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.019+0000 7fb5efc28740 10 client.16428 fetch_fsmap learned FSMap version 44 2024-03-18T16:16:15.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.019+0000 7fb5efc28740 1 -- 172.21.15.39:0/4224917154 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x562e201d30f0 con 0x562e2059a810 2024-03-18T16:16:15.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.020+0000 7fb5c7e00640 1 -- 172.21.15.39:0/4224917154 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb5d40041c0 con 0x562e2059a810 2024-03-18T16:16:15.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.020+0000 7fb5c7e00640 1 -- 172.21.15.39:0/4224917154 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7fb5d80bb040 con 0x562e2059a810 2024-03-18T16:16:15.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.020+0000 7fb5efc28740 10 client.16428 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:16:15.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.020+0000 7fb5efc28740 10 client.16428 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:16:15.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.020+0000 7fb5efc28740 20 client.16428 populate_metadata read hostname 'smithi039' 2024-03-18T16:16:15.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.020+0000 7fb5efc28740 1 --2- 172.21.15.39:0/4224917154 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x562e205b0b10 0x562e205b2f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:15.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.020+0000 7fb5efc28740 4 client.16428 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:16:15.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.020+0000 7fb5efc28740 1 -- 172.21.15.39:0/4224917154 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562e20512be0 con 0x562e205b0b10 2024-03-18T16:16:15.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.021+0000 7fb5e5400640 1 --2- 172.21.15.39:0/4224917154 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x562e205b0b10 0x562e205b2f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:15.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.022+0000 7fb5e5400640 1 --2- 172.21.15.39:0/4224917154 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x562e205b0b10 0x562e205b2f80 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4de4c3cf94551efa server_cookie=37334cfbb539f0c4 in_seq=0 out_seq=0 2024-03-18T16:16:15.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.022+0000 7fb5c7e00640 10 client.16428 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:16:15.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.023+0000 7fb5c7e00640 1 -- 172.21.15.39:0/4224917154 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562e20512be0 con 0x562e205b0b10 2024-03-18T16:16:15.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.023+0000 7fb5c7e00640 10 client.16428 handle_command_reply: tid=0 2024-03-18T16:16:15.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.023+0000 7fb5efc28740 1 client.16428 shutdown 2024-03-18T16:16:15.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.024+0000 7fb5efc28740 1 -- 172.21.15.39:0/4224917154 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562e2059a810 msgr2=0x562e205a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:15.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.024+0000 7fb5efc28740 1 --2- 172.21.15.39:0/4224917154 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562e2059a810 0x562e205a4cc0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fb5d800c6a0 tx=0x7fb5d809bfb0 comp rx=0 tx=0).stop 2024-03-18T16:16:15.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.024+0000 7fb5efc28740 1 -- 172.21.15.39:0/4224917154 shutdown_connections 2024-03-18T16:16:15.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.024+0000 7fb5efc28740 1 --2- 172.21.15.39:0/4224917154 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x562e205b0b10 0x562e205b2f80 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:15.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.024+0000 7fb5efc28740 1 --2- 172.21.15.39:0/4224917154 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562e2059b200 0x562e205a5200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:15.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.024+0000 7fb5efc28740 1 --2- 172.21.15.39:0/4224917154 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562e205a0370 0x562e205ac230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:15.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.024+0000 7fb5efc28740 1 --2- 172.21.15.39:0/4224917154 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562e2059a810 0x562e205a4cc0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:15.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.024+0000 7fb5efc28740 1 -- 172.21.15.39:0/4224917154 >> 172.21.15.39:0/4224917154 conn(0x562e20595d80 msgr2=0x562e20594ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:15.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.024+0000 7fb5c7e00640 0 client.16428 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:16:15.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.024+0000 7fb5efc28740 1 -- 172.21.15.39:0/4224917154 shutdown_connections 2024-03-18T16:16:15.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.025+0000 7fb5efc28740 1 -- 172.21.15.39:0/4224917154 wait complete. 2024-03-18T16:16:15.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.025+0000 7fb5efc28740 20 client.16428 trim_cache size 0 max 16384 2024-03-18T16:16:15.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.061+0000 7fb5efc28740 1 Processor -- start 2024-03-18T16:16:15.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.061+0000 7fb5efc28740 1 -- start start 2024-03-18T16:16:15.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.061+0000 7fb5efc28740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562e2059a810 0x562e205a4f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:15.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.061+0000 7fb5efc28740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562e2059b200 0x562e205ac1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:15.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.061+0000 7fb5efc28740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562e205a0370 0x562e205a3840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:15.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.061+0000 7fb5efc28740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fb5e81b0ba0 con 0x562e205a0370 2024-03-18T16:16:15.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.061+0000 7fb5efc28740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fb5e81b26e0 con 0x562e2059b200 2024-03-18T16:16:15.064 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.061+0000 7fb5efc28740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fb5e81b1270 con 0x562e2059a810 2024-03-18T16:16:15.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.061+0000 7fb5e5400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562e2059b200 0x562e205ac1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:15.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.062+0000 7fb5e5e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562e2059a810 0x562e205a4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:15.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.062+0000 7fb5e5400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562e2059b200 0x562e205ac1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:54046/0 (socket says 172.21.15.39:54046) 2024-03-18T16:16:15.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.062+0000 7fb5e5400640 1 -- 172.21.15.39:0/2852349949 learned_addr learned my addr 172.21.15.39:0/2852349949 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:16:15.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.062+0000 7fb5e6800640 1 --2- 172.21.15.39:0/2852349949 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562e205a0370 0x562e205a3840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:15.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.062+0000 7fb5e5e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562e2059a810 0x562e205a4f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:59980/0 (socket says 172.21.15.39:59980) 2024-03-18T16:16:15.065 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.062+0000 7fb5e5400640 1 -- 172.21.15.39:0/2852349949 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562e2059a810 msgr2=0x562e205a4f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:15.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.062+0000 7fb5e5400640 1 --2- 172.21.15.39:0/2852349949 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562e2059a810 0x562e205a4f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:15.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.062+0000 7fb5e5400640 1 -- 172.21.15.39:0/2852349949 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562e205a0370 msgr2=0x562e205a3840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:15.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.062+0000 7fb5e5400640 1 --2- 172.21.15.39:0/2852349949 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562e205a0370 0x562e205a3840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:15.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.062+0000 7fb5e5400640 1 -- 172.21.15.39:0/2852349949 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e20595bb0 con 0x562e2059b200 2024-03-18T16:16:15.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.062+0000 7fb5e6800640 1 --2- 172.21.15.39:0/2852349949 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562e205a0370 0x562e205a3840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:16:15.066 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.062+0000 7fb5e5e00640 1 --2- 172.21.15.39:0/2852349949 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562e2059a810 0x562e205a4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:16:15.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.065+0000 7fb5e5400640 1 --2- 172.21.15.39:0/2852349949 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562e2059b200 0x562e205ac1f0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fb5d4005940 tx=0x7fb5d40058d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=94253c44609cdcce server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:15.067 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.065+0000 7fb5d2a00640 1 -- 172.21.15.39:0/2852349949 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5d402f070 con 0x562e2059b200 2024-03-18T16:16:15.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.065+0000 7fb5d2a00640 1 -- 172.21.15.39:0/2852349949 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb5d40290d0 con 0x562e2059b200 2024-03-18T16:16:15.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.065+0000 7fb5d2a00640 1 -- 172.21.15.39:0/2852349949 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5d4003850 con 0x562e2059b200 2024-03-18T16:16:15.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.066+0000 7fb5efc28740 1 -- 172.21.15.39:0/2852349949 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562e2059b200 msgr2=0x562e205ac1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:15.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.066+0000 7fb5efc28740 1 --2- 172.21.15.39:0/2852349949 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562e2059b200 0x562e205ac1f0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fb5d4005940 tx=0x7fb5d40058d0 comp rx=0 tx=0).stop 2024-03-18T16:16:15.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.066+0000 7fb5efc28740 1 -- 172.21.15.39:0/2852349949 shutdown_connections 2024-03-18T16:16:15.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.066+0000 7fb5efc28740 1 --2- 172.21.15.39:0/2852349949 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562e2059b200 0x562e205ac1f0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:15.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.066+0000 7fb5efc28740 1 --2- 172.21.15.39:0/2852349949 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562e205a0370 0x562e205a3840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:15.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.066+0000 7fb5efc28740 1 --2- 172.21.15.39:0/2852349949 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562e2059a810 0x562e205a4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:15.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.066+0000 7fb5efc28740 1 -- 172.21.15.39:0/2852349949 >> 172.21.15.39:0/2852349949 conn(0x562e20595d80 msgr2=0x562e20594ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:15.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.067+0000 7fb5efc28740 1 -- 172.21.15.39:0/2852349949 shutdown_connections 2024-03-18T16:16:15.069 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.067+0000 7fb5efc28740 1 -- 172.21.15.39:0/2852349949 wait complete. 2024-03-18T16:16:15.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.067+0000 7fb5efc28740 1 -- start start 2024-03-18T16:16:15.070 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.068+0000 7fb5efc28740 1 Processor -- start 2024-03-18T16:16:15.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.068+0000 7fb5efc28740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562e2059a810 0x562e205a9960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:15.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.068+0000 7fb5efc28740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562e2059b200 0x562e205a9ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:15.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.068+0000 7fb5efc28740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562e205a0370 0x562e205aaeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:15.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.068+0000 7fb5efc28740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fb5e81b2010 con 0x562e2059b200 2024-03-18T16:16:15.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.068+0000 7fb5efc28740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fb5e81b1940 con 0x562e205a0370 2024-03-18T16:16:15.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.068+0000 7fb5efc28740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fb5e81aeb80 con 0x562e2059a810 2024-03-18T16:16:15.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.068+0000 7fb5efc28740 10 client.0 _ll_register_callbacks cb 0x7fb5d8097b10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:16:15.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.069+0000 7fb5e5e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562e2059a810 0x562e205a9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:15.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.069+0000 7fb5e5e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562e2059a810 0x562e205a9960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:59982/0 (socket says 172.21.15.39:59982) 2024-03-18T16:16:15.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.069+0000 7fb5e5e00640 1 -- 172.21.15.39:0/1537428537 learned_addr learned my addr 172.21.15.39:0/1537428537 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:16:15.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.069+0000 7fb5e6800640 1 --2- 172.21.15.39:0/1537428537 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562e205a0370 0x562e205aaeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:15.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.069+0000 7fb5e5400640 1 --2- 172.21.15.39:0/1537428537 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562e2059b200 0x562e205a9ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:15.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.070+0000 7fb5e5e00640 1 -- 172.21.15.39:0/1537428537 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562e205a0370 msgr2=0x562e205aaeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:15.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.070+0000 7fb5e5e00640 1 --2- 172.21.15.39:0/1537428537 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562e205a0370 0x562e205aaeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:15.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.070+0000 7fb5e5e00640 1 -- 172.21.15.39:0/1537428537 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562e2059b200 msgr2=0x562e205a9ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:15.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.070+0000 7fb5e5e00640 1 --2- 172.21.15.39:0/1537428537 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562e2059b200 0x562e205a9ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:15.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.070+0000 7fb5e5e00640 1 -- 172.21.15.39:0/1537428537 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e20590370 con 0x562e2059a810 2024-03-18T16:16:15.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.070+0000 7fb5e5400640 1 --2- 172.21.15.39:0/1537428537 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562e2059b200 0x562e205a9ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:16:15.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.070+0000 7fb5e5e00640 1 --2- 172.21.15.39:0/1537428537 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562e2059a810 0x562e205a9960 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fb5d809c0a0 tx=0x7fb5d809b270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5065ca76d408c477 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:15.074 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.070+0000 7fb5c7e00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:16:15.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.071+0000 7fb5c7e00640 1 -- 172.21.15.39:0/1537428537 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5d80a3f90 con 0x562e2059a810 2024-03-18T16:16:15.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.071+0000 7fb5efc28740 1 -- 172.21.15.39:0/1537428537 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb5d40290d0 con 0x562e2059a810 2024-03-18T16:16:15.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.071+0000 7fb5c7e00640 1 -- 172.21.15.39:0/1537428537 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fb5d80a3b00 con 0x562e2059a810 2024-03-18T16:16:15.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.071+0000 7fb5c7e00640 1 -- 172.21.15.39:0/1537428537 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fb5d8010ac0 con 0x562e2059a810 2024-03-18T16:16:15.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.071+0000 7fb5efc28740 10 client.26899 fetch_fsmap learned FSMap version 44 2024-03-18T16:16:15.075 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.071+0000 7fb5efc28740 1 -- 172.21.15.39:0/1537428537 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x562e20627e60 con 0x562e2059a810 2024-03-18T16:16:15.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.072+0000 7fb5c7e00640 1 -- 172.21.15.39:0/1537428537 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb5d40290d0 con 0x562e2059a810 2024-03-18T16:16:15.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.073+0000 7fb5c7e00640 1 -- 172.21.15.39:0/1537428537 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7fb5d80bb040 con 0x562e2059a810 2024-03-18T16:16:15.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.073+0000 7fb5efc28740 10 client.26899 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:16:15.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.073+0000 7fb5efc28740 10 client.26899 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:16:15.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.073+0000 7fb5efc28740 20 client.26899 populate_metadata read hostname 'smithi039' 2024-03-18T16:16:15.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.073+0000 7fb5efc28740 1 --2- 172.21.15.39:0/1537428537 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x562e206284c0 0x562e2062a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:15.076 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.073+0000 7fb5efc28740 4 client.26899 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T16:16:15.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.073+0000 7fb5efc28740 1 -- 172.21.15.39:0/1537428537 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "scrub status"}) -- 0x562e2050c730 con 0x562e206284c0 2024-03-18T16:16:15.077 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.074+0000 7fb5e5400640 1 --2- 172.21.15.39:0/1537428537 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x562e206284c0 0x562e2062a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:15.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.074+0000 7fb5e5400640 1 --2- 172.21.15.39:0/1537428537 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x562e206284c0 0x562e2062a8e0 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5c8c7c62a4440326 server_cookie=14c6ee14cd55e809 in_seq=0 out_seq=0 2024-03-18T16:16:15.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.074+0000 7fb5c7e00640 10 client.26899 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:16:15.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.075+0000 7fb5c7e00640 1 -- 172.21.15.39:0/1537428537 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x562e2050c730 con 0x562e206284c0 2024-03-18T16:16:15.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.075+0000 7fb5c7e00640 10 client.26899 handle_command_reply: tid=0 2024-03-18T16:16:15.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.075+0000 7fb5efc28740 1 client.26899 shutdown 2024-03-18T16:16:15.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.075+0000 7fb5efc28740 1 -- 172.21.15.39:0/1537428537 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562e2059a810 msgr2=0x562e205a9960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:15.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.075+0000 7fb5efc28740 1 --2- 172.21.15.39:0/1537428537 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562e2059a810 0x562e205a9960 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fb5d809c0a0 tx=0x7fb5d809b270 comp rx=0 tx=0).stop 2024-03-18T16:16:15.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.075+0000 7fb5efc28740 1 -- 172.21.15.39:0/1537428537 shutdown_connections 2024-03-18T16:16:15.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.075+0000 7fb5efc28740 1 --2- 172.21.15.39:0/1537428537 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x562e206284c0 0x562e2062a8e0 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:15.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.075+0000 7fb5efc28740 1 --2- 172.21.15.39:0/1537428537 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562e205a0370 0x562e205aaeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:15.078 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.075+0000 7fb5efc28740 1 --2- 172.21.15.39:0/1537428537 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562e2059b200 0x562e205a9ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:15.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.075+0000 7fb5c7e00640 0 client.26899 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:16:15.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.075+0000 7fb5efc28740 1 --2- 172.21.15.39:0/1537428537 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562e2059a810 0x562e205a9960 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:15.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.075+0000 7fb5efc28740 1 -- 172.21.15.39:0/1537428537 >> 172.21.15.39:0/1537428537 conn(0x562e20595d80 msgr2=0x562e20594ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:15.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.076+0000 7fb5efc28740 1 -- 172.21.15.39:0/1537428537 shutdown_connections 2024-03-18T16:16:15.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.076+0000 7fb5efc28740 1 -- 172.21.15.39:0/1537428537 wait complete. 2024-03-18T16:16:15.079 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.076+0000 7fb5efc28740 20 client.26899 trim_cache size 0 max 16384 2024-03-18T16:16:15.080 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-03-18T16:16:15.080 INFO:teuthology.orchestra.run.smithi039.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-18T16:16:15.080 INFO:teuthology.orchestra.run.smithi039.stdout: "scrubs": { 2024-03-18T16:16:15.080 INFO:teuthology.orchestra.run.smithi039.stdout: "47698cef-41a7-4063-b34e-5200783d2b8c": { 2024-03-18T16:16:15.080 INFO:teuthology.orchestra.run.smithi039.stdout: "path": "/", 2024-03-18T16:16:15.080 INFO:teuthology.orchestra.run.smithi039.stdout: "tag": "47698cef-41a7-4063-b34e-5200783d2b8c", 2024-03-18T16:16:15.080 INFO:teuthology.orchestra.run.smithi039.stdout: "options": "recursive,force" 2024-03-18T16:16:15.080 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:16:15.080 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:16:15.080 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-03-18T16:16:15.081 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.077+0000 7fb5ed000640 1 -- 172.21.15.39:0/1431992748 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb5cc0a2900 msgr2=0x7fb5cc0a4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:15.081 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.077+0000 7fb5ed000640 1 --2- 172.21.15.39:0/1431992748 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb5cc0a2900 0x7fb5cc0a4dc0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x562e2059ba60 tx=0x7fb5e002bf10 comp rx=0 tx=0).stop 2024-03-18T16:16:15.081 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.077+0000 7fb5ed000640 1 -- 172.21.15.39:0/1431992748 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb5e8103a20 msgr2=0x7fb5e8079590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:15.081 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.077+0000 7fb5ed000640 1 --2- 172.21.15.39:0/1431992748 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb5e8103a20 0x7fb5e8079590 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fb5d800e9c0 tx=0x7fb5d800ee80 comp rx=0 tx=0).stop 2024-03-18T16:16:15.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.077+0000 7fb5ed000640 1 -- 172.21.15.39:0/1431992748 shutdown_connections 2024-03-18T16:16:15.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.077+0000 7fb5ed000640 1 --2- 172.21.15.39:0/1431992748 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb5cc0a2900 0x7fb5cc0a4dc0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:15.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.077+0000 7fb5ed000640 1 --2- 172.21.15.39:0/1431992748 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb5e81055d0 0x7fb5e8076100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:15.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.077+0000 7fb5ed000640 1 --2- 172.21.15.39:0/1431992748 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb5e8104c10 0x7fb5e8079ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:15.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.077+0000 7fb5ed000640 1 --2- 172.21.15.39:0/1431992748 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb5e8103a20 0x7fb5e8079590 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:15.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.077+0000 7fb5ed000640 1 -- 172.21.15.39:0/1431992748 >> 172.21.15.39:0/1431992748 conn(0x7fb5e80fef90 msgr2=0x7fb5e8107e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:15.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.078+0000 7fb5ed000640 1 -- 172.21.15.39:0/1431992748 shutdown_connections 2024-03-18T16:16:15.082 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:15.079+0000 7fb5ed000640 1 -- 172.21.15.39:0/1431992748 wait complete. 2024-03-18T16:16:15.106 INFO:tasks.cephfs.filesystem:scrub status for tag:47698cef-41a7-4063-b34e-5200783d2b8c - {'path': '/', 'tag': '47698cef-41a7-4063-b34e-5200783d2b8c', 'options': 'recursive,force'} 2024-03-18T16:16:15.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:15 smithi039 ceph-mon[31721]: pgmap v607: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.4 MiB/s wr, 370 op/s 2024-03-18T16:16:15.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:15 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:16:15.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:15 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:16:15.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:15 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:15.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:15 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:16:15.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:15 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:15.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:15 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:16:15.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:15 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:15.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:15 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:16:15.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:15 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:15.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:15 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:16:15.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:15 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:15.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:15 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:16:15.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:15 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:15.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:15 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:16:15.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:15 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:15.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:15 smithi196 ceph-mon[36307]: pgmap v607: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.4 MiB/s wr, 370 op/s 2024-03-18T16:16:15.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:15 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:16:15.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:15 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:16:15.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:15 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:15.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:15 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:16:15.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:15 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:15.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:15 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:16:15.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:15 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:15.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:15 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:16:15.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:15 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:15.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:15 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:16:15.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:15 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:15.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:15 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:16:15.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:15 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:15.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:15 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:16:15.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:15 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:15.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:15 smithi194 ceph-mon[36200]: pgmap v607: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.4 MiB/s wr, 370 op/s 2024-03-18T16:16:15.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:15 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:16:15.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:15 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:16:15.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:15 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:15.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:15 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:16:15.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:15 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:15.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:15 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:16:15.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:15 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:15.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:15 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:16:15.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:15 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:15.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:15 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:16:15.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:15 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:15.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:15 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:16:15.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:15 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:15.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:15 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:16:15.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:15 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:16.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:16 smithi039 ceph-mon[31721]: scrub queued for path: / 2024-03-18T16:16:16.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:16 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:16.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:16 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:16:16.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:16 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:16:16.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:16 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:16:16.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:16 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:16.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:16 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:16:16.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:16 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:16.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:16 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:16:16.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:16 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:16.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:16 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:16.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:16 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:16.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:16 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:16:16.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:16 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:16.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:16 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:16.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:16 smithi196 ceph-mon[36307]: scrub queued for path: / 2024-03-18T16:16:16.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:16 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:16.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:16 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:16:16.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:16 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:16:16.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:16 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:16:16.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:16 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:16.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:16 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:16:16.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:16 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:16.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:16 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:16:16.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:16 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:16.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:16 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:16.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:16 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:16.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:16 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:16:16.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:16 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:16.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:16 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:16.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:16 smithi194 ceph-mon[36200]: scrub queued for path: / 2024-03-18T16:16:16.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:16 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:16.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:16 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:16:16.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:16 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:16:16.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:16 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:16:16.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:16 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:16.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:16 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:16:16.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:16 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:16.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:16 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:16:16.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:16 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:16.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:16 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:16.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:16 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:16.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:16 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:16:16.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:16 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:16.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:16 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:17.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:17 smithi039 ceph-mon[31721]: pgmap v608: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 1.6 MiB/s wr, 424 op/s 2024-03-18T16:16:17.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:17 smithi196 ceph-mon[36307]: pgmap v608: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 1.6 MiB/s wr, 424 op/s 2024-03-18T16:16:17.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:17 smithi194 ceph-mon[36200]: pgmap v608: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 1.6 MiB/s wr, 424 op/s 2024-03-18T16:16:19.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:19 smithi196 ceph-mon[36307]: pgmap v609: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.4 MiB/s wr, 358 op/s 2024-03-18T16:16:19.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:19 smithi194 ceph-mon[36200]: pgmap v609: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.4 MiB/s wr, 358 op/s 2024-03-18T16:16:19.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:19 smithi039 ceph-mon[31721]: pgmap v609: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.4 MiB/s wr, 358 op/s 2024-03-18T16:16:21.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:21 smithi196 ceph-mon[36307]: pgmap v610: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 1.4 MiB/s wr, 369 op/s 2024-03-18T16:16:21.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:21 smithi194 ceph-mon[36200]: pgmap v610: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 1.4 MiB/s wr, 369 op/s 2024-03-18T16:16:21.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:21 smithi039 ceph-mon[31721]: pgmap v610: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 1.4 MiB/s wr, 369 op/s 2024-03-18T16:16:23.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:23 smithi196 ceph-mon[36307]: pgmap v611: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.6 MiB/s wr, 426 op/s 2024-03-18T16:16:23.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:23 smithi194 ceph-mon[36200]: pgmap v611: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.6 MiB/s wr, 426 op/s 2024-03-18T16:16:23.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:23 smithi039 ceph-mon[31721]: pgmap v611: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.6 MiB/s wr, 426 op/s 2024-03-18T16:16:24.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:24 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:16:24.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:24 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:16:24.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:24 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:16:25.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:25 smithi196 ceph-mon[36307]: pgmap v612: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.2 MiB/s wr, 313 op/s 2024-03-18T16:16:25.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:25 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:16:25.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:25 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:16:25.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:25 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:16:25.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:25 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:16:25.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:25 smithi194 ceph-mon[36200]: pgmap v612: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.2 MiB/s wr, 313 op/s 2024-03-18T16:16:25.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:25 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:16:25.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:25 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:16:25.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:25 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:16:25.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:25 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:16:25.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:25 smithi039 ceph-mon[31721]: pgmap v612: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.2 MiB/s wr, 313 op/s 2024-03-18T16:16:25.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:25 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:16:25.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:25 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:16:25.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:25 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:16:25.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:25 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-18T16:16:26.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:26 smithi196 ceph-mon[36307]: Adjusting osd_memory_target on smithi196 to 174.1M 2024-03-18T16:16:26.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:26 smithi196 ceph-mon[36307]: Unable to set osd_memory_target on smithi196 to 182618419: error parsing value: Value '182618419' is below minimum 939524096 2024-03-18T16:16:26.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:26 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:16:26.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:26 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:16:26.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:26 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:16:26.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:26 smithi194 ceph-mon[36200]: Adjusting osd_memory_target on smithi196 to 174.1M 2024-03-18T16:16:26.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:26 smithi194 ceph-mon[36200]: Unable to set osd_memory_target on smithi196 to 182618419: error parsing value: Value '182618419' is below minimum 939524096 2024-03-18T16:16:26.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:26 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:16:26.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:26 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:16:26.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:26 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:16:26.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:26 smithi039 ceph-mon[31721]: Adjusting osd_memory_target on smithi196 to 174.1M 2024-03-18T16:16:26.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:26 smithi039 ceph-mon[31721]: Unable to set osd_memory_target on smithi196 to 182618419: error parsing value: Value '182618419' is below minimum 939524096 2024-03-18T16:16:26.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:26 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:16:26.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:26 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:16:26.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:26 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:16:27.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:27 smithi196 ceph-mon[36307]: pgmap v613: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.4 MiB/s wr, 379 op/s 2024-03-18T16:16:27.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:27 smithi194 ceph-mon[36200]: pgmap v613: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.4 MiB/s wr, 379 op/s 2024-03-18T16:16:27.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:27 smithi039 ceph-mon[31721]: pgmap v613: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.4 MiB/s wr, 379 op/s 2024-03-18T16:16:28.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:28 smithi196 ceph-mon[36307]: scrub summary: idle 2024-03-18T16:16:28.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:28 smithi196 ceph-mon[36307]: scrub summary: idle 2024-03-18T16:16:28.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:28 smithi196 ceph-mon[36307]: scrub summary: idle 2024-03-18T16:16:28.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:28 smithi196 ceph-mon[36307]: scrub summary: idle 2024-03-18T16:16:28.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:28 smithi196 ceph-mon[36307]: scrub summary: idle 2024-03-18T16:16:28.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:28 smithi194 ceph-mon[36200]: scrub summary: idle 2024-03-18T16:16:28.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:28 smithi194 ceph-mon[36200]: scrub summary: idle 2024-03-18T16:16:28.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:28 smithi194 ceph-mon[36200]: scrub summary: idle 2024-03-18T16:16:28.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:28 smithi194 ceph-mon[36200]: scrub summary: idle 2024-03-18T16:16:28.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:28 smithi194 ceph-mon[36200]: scrub summary: idle 2024-03-18T16:16:28.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:28 smithi039 ceph-mon[31721]: scrub summary: idle 2024-03-18T16:16:28.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:28 smithi039 ceph-mon[31721]: scrub summary: idle 2024-03-18T16:16:28.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:28 smithi039 ceph-mon[31721]: scrub summary: idle 2024-03-18T16:16:28.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:28 smithi039 ceph-mon[31721]: scrub summary: idle 2024-03-18T16:16:28.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:28 smithi039 ceph-mon[31721]: scrub summary: idle 2024-03-18T16:16:29.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:29 smithi196 ceph-mon[36307]: pgmap v614: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.2 MiB/s wr, 321 op/s 2024-03-18T16:16:29.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:29 smithi194 ceph-mon[36200]: pgmap v614: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.2 MiB/s wr, 321 op/s 2024-03-18T16:16:29.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:29 smithi039 ceph-mon[31721]: pgmap v614: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.2 MiB/s wr, 321 op/s 2024-03-18T16:16:31.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:31 smithi196 ceph-mon[36307]: pgmap v615: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 1.3 MiB/s wr, 346 op/s 2024-03-18T16:16:31.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:31 smithi194 ceph-mon[36200]: pgmap v615: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 1.3 MiB/s wr, 346 op/s 2024-03-18T16:16:31.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:31 smithi039 ceph-mon[31721]: pgmap v615: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 1.3 MiB/s wr, 346 op/s 2024-03-18T16:16:33.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:33 smithi196 ceph-mon[36307]: pgmap v616: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.5 MiB/s wr, 385 op/s 2024-03-18T16:16:33.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:33 smithi194 ceph-mon[36200]: pgmap v616: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.5 MiB/s wr, 385 op/s 2024-03-18T16:16:33.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:33 smithi039 ceph-mon[31721]: pgmap v616: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 1.5 MiB/s wr, 385 op/s 2024-03-18T16:16:35.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:35 smithi196 ceph-mon[36307]: pgmap v617: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1.2 MiB/s wr, 311 op/s 2024-03-18T16:16:35.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:35 smithi194 ceph-mon[36200]: pgmap v617: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1.2 MiB/s wr, 311 op/s 2024-03-18T16:16:35.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:35 smithi039 ceph-mon[31721]: pgmap v617: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1.2 MiB/s wr, 311 op/s 2024-03-18T16:16:37.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:37 smithi196 ceph-mon[36307]: pgmap v618: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.4 MiB/s wr, 375 op/s 2024-03-18T16:16:37.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:37 smithi194 ceph-mon[36200]: pgmap v618: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.4 MiB/s wr, 375 op/s 2024-03-18T16:16:37.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:37 smithi039 ceph-mon[31721]: pgmap v618: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.4 MiB/s wr, 375 op/s 2024-03-18T16:16:39.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:39 smithi196 ceph-mon[36307]: pgmap v619: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.2 MiB/s wr, 311 op/s 2024-03-18T16:16:39.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:39 smithi194 ceph-mon[36200]: pgmap v619: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.2 MiB/s wr, 311 op/s 2024-03-18T16:16:39.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:39 smithi039 ceph-mon[31721]: pgmap v619: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.2 MiB/s wr, 311 op/s 2024-03-18T16:16:41.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:41 smithi196 ceph-mon[36307]: pgmap v620: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.3 MiB/s wr, 344 op/s 2024-03-18T16:16:41.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:41 smithi194 ceph-mon[36200]: pgmap v620: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.3 MiB/s wr, 344 op/s 2024-03-18T16:16:41.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:41 smithi039 ceph-mon[31721]: pgmap v620: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.3 MiB/s wr, 344 op/s 2024-03-18T16:16:43.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:43 smithi196 ceph-mon[36307]: pgmap v621: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1.4 MiB/s wr, 367 op/s 2024-03-18T16:16:43.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:43 smithi194 ceph-mon[36200]: pgmap v621: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1.4 MiB/s wr, 367 op/s 2024-03-18T16:16:43.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:43 smithi039 ceph-mon[31721]: pgmap v621: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 1.4 MiB/s wr, 367 op/s 2024-03-18T16:16:45.107 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T16:16:45.107 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T16:16:45.386 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.384+0000 7f5204c00640 1 -- 172.21.15.39:0/2135669588 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f52001087e0 msgr2=0x7f520010ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:45.386 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.384+0000 7f5204c00640 1 --2- 172.21.15.39:0/2135669588 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f52001087e0 0x7f520010ac10 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f51f0009ca0 tx=0x7f51f002c140 comp rx=0 tx=0).stop 2024-03-18T16:16:45.386 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.384+0000 7f5204c00640 1 -- 172.21.15.39:0/2135669588 shutdown_connections 2024-03-18T16:16:45.386 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.384+0000 7f5204c00640 1 --2- 172.21.15.39:0/2135669588 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f520010db00 0x7f520010df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:45.386 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.384+0000 7f5204c00640 1 --2- 172.21.15.39:0/2135669588 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f520010b150 0x7f520010d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:45.386 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.384+0000 7f5204c00640 1 --2- 172.21.15.39:0/2135669588 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f52001087e0 0x7f520010ac10 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:45.386 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.384+0000 7f5204c00640 1 -- 172.21.15.39:0/2135669588 >> 172.21.15.39:0/2135669588 conn(0x7f520006f550 msgr2=0x7f5200071990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:45.386 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.384+0000 7f5204c00640 1 -- 172.21.15.39:0/2135669588 shutdown_connections 2024-03-18T16:16:45.386 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.384+0000 7f5204c00640 1 -- 172.21.15.39:0/2135669588 wait complete. 2024-03-18T16:16:45.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.385+0000 7f5204c00640 1 Processor -- start 2024-03-18T16:16:45.388 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.386+0000 7f5204c00640 1 -- start start 2024-03-18T16:16:45.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.386+0000 7f5204c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f52001087e0 0x7f52001a66c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:45.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.386+0000 7f5204c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f520010b150 0x7f52001a6c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:45.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.386+0000 7f5204c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f520010db00 0x7f52001adc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:45.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.386+0000 7f5204c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f520011c150 con 0x7f52001087e0 2024-03-18T16:16:45.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.386+0000 7f5204c00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f520011bfd0 con 0x7f520010db00 2024-03-18T16:16:45.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.386+0000 7f5204c00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f520011c2d0 con 0x7f520010b150 2024-03-18T16:16:45.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.386+0000 7f51ffe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f520010db00 0x7f52001adc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:45.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.386+0000 7f51ffe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f520010db00 0x7f52001adc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:58338/0 (socket says 172.21.15.39:58338) 2024-03-18T16:16:45.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.386+0000 7f51ffe00640 1 -- 172.21.15.39:0/1736871978 learned_addr learned my addr 172.21.15.39:0/1736871978 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:16:45.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.387+0000 7f51fea00640 1 --2- 172.21.15.39:0/1736871978 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f520010b150 0x7f52001a6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:45.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.387+0000 7f51ff400640 1 --2- 172.21.15.39:0/1736871978 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f52001087e0 0x7f52001a66c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:45.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.387+0000 7f51ffe00640 1 -- 172.21.15.39:0/1736871978 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f520010b150 msgr2=0x7f52001a6c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:45.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.387+0000 7f51ffe00640 1 --2- 172.21.15.39:0/1736871978 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f520010b150 0x7f52001a6c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:45.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.387+0000 7f51ffe00640 1 -- 172.21.15.39:0/1736871978 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f52001087e0 msgr2=0x7f52001a66c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:45.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.387+0000 7f51ffe00640 1 --2- 172.21.15.39:0/1736871978 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f52001087e0 0x7f52001a66c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:45.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.387+0000 7f51ffe00640 1 -- 172.21.15.39:0/1736871978 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f520010ef30 con 0x7f520010db00 2024-03-18T16:16:45.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.387+0000 7f51fea00640 1 --2- 172.21.15.39:0/1736871978 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f520010b150 0x7f52001a6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:16:45.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.387+0000 7f51ffe00640 1 --2- 172.21.15.39:0/1736871978 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f520010db00 0x7f52001adc70 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f51e800c970 tx=0x7f51e800cd70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:45.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.388+0000 7f51f7e00640 1 -- 172.21.15.39:0/1736871978 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e8007bb0 con 0x7f520010db00 2024-03-18T16:16:45.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.388+0000 7f51f7e00640 1 -- 172.21.15.39:0/1736871978 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51e8007d50 con 0x7f520010db00 2024-03-18T16:16:45.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.388+0000 7f51f7e00640 1 -- 172.21.15.39:0/1736871978 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51e8018770 con 0x7f520010db00 2024-03-18T16:16:45.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.388+0000 7f5204c00640 1 -- 172.21.15.39:0/1736871978 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f520010f220 con 0x7f520010db00 2024-03-18T16:16:45.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.388+0000 7f5204c00640 1 -- 172.21.15.39:0/1736871978 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f520010f720 con 0x7f520010db00 2024-03-18T16:16:45.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.392+0000 7f5208fc0740 1 Processor -- start 2024-03-18T16:16:45.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.395+0000 7f51f7e00640 1 -- 172.21.15.39:0/1736871978 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f51e8018930 con 0x7f520010db00 2024-03-18T16:16:45.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.395+0000 7f5208fc0740 1 -- start start 2024-03-18T16:16:45.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.396+0000 7f5208fc0740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fe16654a10 0x55fe16654e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:45.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.396+0000 7f5208fc0740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fe16655400 0x55fe16659f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:45.398 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.396+0000 7f5208fc0740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fe1665a570 0x55fe1665ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:45.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.396+0000 7f5208fc0740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55fe165b91f0 con 0x55fe16655400 2024-03-18T16:16:45.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.396+0000 7f5208fc0740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55fe16561a80 con 0x55fe1665a570 2024-03-18T16:16:45.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.396+0000 7f5208fc0740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55fe165b5290 con 0x55fe16654a10 2024-03-18T16:16:45.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.396+0000 7f51fea00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fe16654a10 0x55fe16654e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:45.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.396+0000 7f51fea00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fe16654a10 0x55fe16654e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:54508/0 (socket says 172.21.15.39:54508) 2024-03-18T16:16:45.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.396+0000 7f51fea00640 1 -- 172.21.15.39:0/2195531449 learned_addr learned my addr 172.21.15.39:0/2195531449 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:16:45.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.396+0000 7f51ff400640 1 --2- 172.21.15.39:0/2195531449 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fe16655400 0x55fe16659f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:45.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.397+0000 7f51fea00640 1 --2- 172.21.15.39:0/2195531449 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fe1665a570 0x55fe1665ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:45.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.397+0000 7f51ff400640 1 -- 172.21.15.39:0/2195531449 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fe16654a10 msgr2=0x55fe16654e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:45.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.397+0000 7f51ff400640 1 --2- 172.21.15.39:0/2195531449 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fe16654a10 0x55fe16654e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:45.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.397+0000 7f51ff400640 1 -- 172.21.15.39:0/2195531449 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fe1665a570 msgr2=0x55fe1665ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:45.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.397+0000 7f51ff400640 1 --2- 172.21.15.39:0/2195531449 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fe1665a570 0x55fe1665ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:45.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.397+0000 7f51ff400640 1 -- 172.21.15.39:0/2195531449 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fe163977d0 con 0x55fe16655400 2024-03-18T16:16:45.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.397+0000 7f51fea00640 1 --2- 172.21.15.39:0/2195531449 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fe16654a10 0x55fe16654e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:16:45.400 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.398+0000 7f51ff400640 1 --2- 172.21.15.39:0/2195531449 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fe16655400 0x55fe16659f00 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f51f0009bf0 tx=0x7f51f0005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2b99fb366646d07 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:45.400 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.398+0000 7f51efe00640 1 -- 172.21.15.39:0/2195531449 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51f0044020 con 0x55fe16655400 2024-03-18T16:16:45.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.399+0000 7f51f7e00640 1 --2- 172.21.15.39:0/1736871978 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f51e40a2880 0x7f51e40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:45.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.399+0000 7f51efe00640 1 -- 172.21.15.39:0/2195531449 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51f003a040 con 0x55fe16655400 2024-03-18T16:16:45.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.399+0000 7f51efe00640 1 -- 172.21.15.39:0/2195531449 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51f0039050 con 0x55fe16655400 2024-03-18T16:16:45.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.399+0000 7f51f7e00640 1 -- 172.21.15.39:0/1736871978 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f51e80cee20 con 0x7f520010db00 2024-03-18T16:16:45.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.399+0000 7f51fea00640 1 --2- 172.21.15.39:0/1736871978 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f51e40a2880 0x7f51e40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:45.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.400+0000 7f5208fc0740 1 -- 172.21.15.39:0/2195531449 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fe16655400 msgr2=0x55fe16659f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:45.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.400+0000 7f5208fc0740 1 --2- 172.21.15.39:0/2195531449 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fe16655400 0x55fe16659f00 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f51f0009bf0 tx=0x7f51f0005ef0 comp rx=0 tx=0).stop 2024-03-18T16:16:45.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.400+0000 7f5208fc0740 1 -- 172.21.15.39:0/2195531449 shutdown_connections 2024-03-18T16:16:45.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.400+0000 7f5208fc0740 1 --2- 172.21.15.39:0/2195531449 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fe1665a570 0x55fe1665ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:45.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.400+0000 7f5208fc0740 1 --2- 172.21.15.39:0/2195531449 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fe16655400 0x55fe16659f00 secure :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f51f0009bf0 tx=0x7f51f0005ef0 comp rx=0 tx=0).stop 2024-03-18T16:16:45.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.400+0000 7f5208fc0740 1 --2- 172.21.15.39:0/2195531449 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fe16654a10 0x55fe16654e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:45.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.400+0000 7f5208fc0740 1 -- 172.21.15.39:0/2195531449 >> 172.21.15.39:0/2195531449 conn(0x55fe1664ff80 msgr2=0x55fe16650380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:45.403 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.400+0000 7f5208fc0740 1 -- 172.21.15.39:0/2195531449 shutdown_connections 2024-03-18T16:16:45.403 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.400+0000 7f5208fc0740 1 -- 172.21.15.39:0/2195531449 wait complete. 2024-03-18T16:16:45.403 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.401+0000 7f51fea00640 1 --2- 172.21.15.39:0/1736871978 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f51e40a2880 0x7f51e40a4d40 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x55fe16655c60 tx=0x7f51f8004730 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:45.403 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.401+0000 7f5208fc0740 1 -- start start 2024-03-18T16:16:45.403 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.402+0000 7f5208fc0740 1 Processor -- start 2024-03-18T16:16:45.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.402+0000 7f5208fc0740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fe16654a10 0x55fe1665f110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:45.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.402+0000 7f5208fc0740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fe1665a570 0x55fe1665f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:45.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.403+0000 7f5208fc0740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fe16664680 0x55fe16666a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:45.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.403+0000 7f5208fc0740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55fe1665d4c0 con 0x55fe1665a570 2024-03-18T16:16:45.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.403+0000 7f5208fc0740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55fe16574ed0 con 0x55fe16664680 2024-03-18T16:16:45.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.403+0000 7f5208fc0740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55fe1665d640 con 0x55fe16654a10 2024-03-18T16:16:45.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.403+0000 7f5208fc0740 10 client.0 _ll_register_callbacks cb 0x55fe165b0220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:16:45.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.403+0000 7f51fea00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fe1665a570 0x55fe1665f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:45.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.403+0000 7f51ffe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fe16664680 0x55fe16666a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:45.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.403+0000 7f51ffe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fe16664680 0x55fe16666a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:58360/0 (socket says 172.21.15.39:58360) 2024-03-18T16:16:45.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.403+0000 7f51ffe00640 1 -- 172.21.15.39:0/3005967119 learned_addr learned my addr 172.21.15.39:0/3005967119 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:16:45.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.403+0000 7f51ff400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fe16654a10 0x55fe1665f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:45.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.404+0000 7f51ff400640 1 -- 172.21.15.39:0/3005967119 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fe16664680 msgr2=0x55fe16666a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:45.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.404+0000 7f51ff400640 1 --2- 172.21.15.39:0/3005967119 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fe16664680 0x55fe16666a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:45.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.404+0000 7f51ff400640 1 -- 172.21.15.39:0/3005967119 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fe1665a570 msgr2=0x55fe1665f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:45.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.404+0000 7f51ff400640 1 --2- 172.21.15.39:0/3005967119 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fe1665a570 0x55fe1665f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:45.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.404+0000 7f51ff400640 1 -- 172.21.15.39:0/3005967119 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fe162cd3f0 con 0x55fe16654a10 2024-03-18T16:16:45.406 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.404+0000 7f51ffe00640 1 --2- 172.21.15.39:0/3005967119 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fe16664680 0x55fe16666a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:16:45.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.405+0000 7f51fea00640 1 --2- 172.21.15.39:0/3005967119 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fe1665a570 0x55fe1665f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:16:45.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.405+0000 7f51ecc00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:16:45.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.405+0000 7f51ff400640 1 --2- 172.21.15.39:0/3005967119 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fe16654a10 0x55fe1665f110 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x55fe166558c0 tx=0x7f51f0009a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a2fc83e1d47d380e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:45.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.406+0000 7f51ecc00640 1 -- 172.21.15.39:0/3005967119 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51f004e020 con 0x55fe16654a10 2024-03-18T16:16:45.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.406+0000 7f51ecc00640 1 -- 172.21.15.39:0/3005967119 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f51f00326b0 con 0x55fe16654a10 2024-03-18T16:16:45.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.406+0000 7f5208fc0740 1 -- 172.21.15.39:0/3005967119 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f51f003a040 con 0x55fe16654a10 2024-03-18T16:16:45.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.406+0000 7f51ecc00640 1 -- 172.21.15.39:0/3005967119 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f51f0044040 con 0x55fe16654a10 2024-03-18T16:16:45.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.406+0000 7f5208fc0740 10 client.26911 fetch_fsmap learned FSMap version 44 2024-03-18T16:16:45.409 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.407+0000 7f5208fc0740 1 -- 172.21.15.39:0/3005967119 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x55fe1628d0f0 con 0x55fe16654a10 2024-03-18T16:16:45.409 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.407+0000 7f51ecc00640 1 -- 172.21.15.39:0/3005967119 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51f003a040 con 0x55fe16654a10 2024-03-18T16:16:45.409 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.407+0000 7f51ecc00640 1 -- 172.21.15.39:0/3005967119 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7f51f00349e0 con 0x55fe16654a10 2024-03-18T16:16:45.409 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.408+0000 7f5208fc0740 10 client.26911 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:16:45.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.408+0000 7f5208fc0740 10 client.26911 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:16:45.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.408+0000 7f5208fc0740 20 client.26911 populate_metadata read hostname 'smithi039' 2024-03-18T16:16:45.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.408+0000 7f5208fc0740 1 --2- 172.21.15.39:0/3005967119 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55fe1666af60 0x55fe1666d350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:45.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.408+0000 7f5208fc0740 4 client.26911 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:16:45.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.408+0000 7f5208fc0740 1 -- 172.21.15.39:0/3005967119 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fe165ccbe0 con 0x55fe1666af60 2024-03-18T16:16:45.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.408+0000 7f51fea00640 1 --2- 172.21.15.39:0/3005967119 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55fe1666af60 0x55fe1666d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:45.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.409+0000 7f51fea00640 1 --2- 172.21.15.39:0/3005967119 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55fe1666af60 0x55fe1666d350 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=96672a5e448300a3 server_cookie=15a7c5401c6f1c5d in_seq=0 out_seq=0 2024-03-18T16:16:45.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.409+0000 7f51ecc00640 10 client.26911 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:16:45.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.410+0000 7f51ecc00640 1 -- 172.21.15.39:0/3005967119 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55fe165ccbe0 con 0x55fe1666af60 2024-03-18T16:16:45.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.410+0000 7f51ecc00640 10 client.26911 handle_command_reply: tid=0 2024-03-18T16:16:45.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.411+0000 7f5208fc0740 1 client.26911 shutdown 2024-03-18T16:16:45.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.411+0000 7f5208fc0740 1 -- 172.21.15.39:0/3005967119 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fe16654a10 msgr2=0x55fe1665f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:45.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.411+0000 7f5208fc0740 1 --2- 172.21.15.39:0/3005967119 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fe16654a10 0x55fe1665f110 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x55fe166558c0 tx=0x7f51f0009a20 comp rx=0 tx=0).stop 2024-03-18T16:16:45.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.411+0000 7f5208fc0740 1 -- 172.21.15.39:0/3005967119 shutdown_connections 2024-03-18T16:16:45.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.411+0000 7f5208fc0740 1 --2- 172.21.15.39:0/3005967119 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55fe1666af60 0x55fe1666d350 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:45.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.411+0000 7f5208fc0740 1 --2- 172.21.15.39:0/3005967119 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fe16664680 0x55fe16666a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:45.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.411+0000 7f5208fc0740 1 --2- 172.21.15.39:0/3005967119 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fe1665a570 0x55fe1665f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:45.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.411+0000 7f5208fc0740 1 --2- 172.21.15.39:0/3005967119 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fe16654a10 0x55fe1665f110 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:45.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.411+0000 7f5208fc0740 1 -- 172.21.15.39:0/3005967119 >> 172.21.15.39:0/3005967119 conn(0x55fe1664ff80 msgr2=0x55fe1664f110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:45.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.411+0000 7f51ecc00640 0 client.26911 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:16:45.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.412+0000 7f5208fc0740 1 -- 172.21.15.39:0/3005967119 shutdown_connections 2024-03-18T16:16:45.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.412+0000 7f5208fc0740 1 -- 172.21.15.39:0/3005967119 wait complete. 2024-03-18T16:16:45.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.412+0000 7f5208fc0740 20 client.26911 trim_cache size 0 max 16384 2024-03-18T16:16:45.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.452+0000 7f5208fc0740 1 Processor -- start 2024-03-18T16:16:45.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.453+0000 7f5208fc0740 1 -- start start 2024-03-18T16:16:45.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.453+0000 7f5208fc0740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fe16654a10 0x55fe1664ef90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:45.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.453+0000 7f5208fc0740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fe1665a570 0x55fe1665e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:45.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.453+0000 7f5208fc0740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fe16664680 0x55fe1665ec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:45.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.453+0000 7f5208fc0740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f52001c17d0 con 0x55fe1665a570 2024-03-18T16:16:45.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.453+0000 7f5208fc0740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f52001c3310 con 0x55fe16654a10 2024-03-18T16:16:45.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.453+0000 7f5208fc0740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f52001c1ea0 con 0x55fe16664680 2024-03-18T16:16:45.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.454+0000 7f51ff400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fe16654a10 0x55fe1664ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:45.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.454+0000 7f51ff400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fe16654a10 0x55fe1664ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:58364/0 (socket says 172.21.15.39:58364) 2024-03-18T16:16:45.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.454+0000 7f51ff400640 1 -- 172.21.15.39:0/2561636224 learned_addr learned my addr 172.21.15.39:0/2561636224 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:16:45.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.454+0000 7f51ffe00640 1 --2- 172.21.15.39:0/2561636224 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fe16664680 0x55fe1665ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:45.456 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.454+0000 7f51ff400640 1 -- 172.21.15.39:0/2561636224 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fe16664680 msgr2=0x55fe1665ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:45.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.454+0000 7f51ff400640 1 --2- 172.21.15.39:0/2561636224 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fe16664680 0x55fe1665ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:45.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.454+0000 7f51ff400640 1 -- 172.21.15.39:0/2561636224 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fe1665a570 msgr2=0x55fe1665e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:45.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.454+0000 7f51ff400640 1 --2- 172.21.15.39:0/2561636224 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fe1665a570 0x55fe1665e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:45.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.454+0000 7f51ff400640 1 -- 172.21.15.39:0/2561636224 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fe1665fed0 con 0x55fe16654a10 2024-03-18T16:16:45.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.455+0000 7f51ffe00640 1 --2- 172.21.15.39:0/2561636224 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fe16664680 0x55fe1665ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:16:45.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.455+0000 7f51ff400640 1 --2- 172.21.15.39:0/2561636224 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fe16654a10 0x55fe1664ef90 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f51f002f020 tx=0x7f51f004a640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=72a9718dda183ae8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:45.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.455+0000 7f51ef400640 1 -- 172.21.15.39:0/2561636224 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51f0039040 con 0x55fe16654a10 2024-03-18T16:16:45.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.455+0000 7f51ef400640 1 -- 172.21.15.39:0/2561636224 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51f0005bd0 con 0x55fe16654a10 2024-03-18T16:16:45.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.455+0000 7f51ef400640 1 -- 172.21.15.39:0/2561636224 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51f004e040 con 0x55fe16654a10 2024-03-18T16:16:45.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.457+0000 7f5208fc0740 1 -- 172.21.15.39:0/2561636224 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fe16654a10 msgr2=0x55fe1664ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:45.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.457+0000 7f5208fc0740 1 --2- 172.21.15.39:0/2561636224 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fe16654a10 0x55fe1664ef90 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f51f002f020 tx=0x7f51f004a640 comp rx=0 tx=0).stop 2024-03-18T16:16:45.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.458+0000 7f5208fc0740 1 -- 172.21.15.39:0/2561636224 shutdown_connections 2024-03-18T16:16:45.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.458+0000 7f5208fc0740 1 --2- 172.21.15.39:0/2561636224 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fe16664680 0x55fe1665ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:45.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.458+0000 7f5208fc0740 1 --2- 172.21.15.39:0/2561636224 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fe1665a570 0x55fe1665e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:45.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.458+0000 7f5208fc0740 1 --2- 172.21.15.39:0/2561636224 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fe16654a10 0x55fe1664ef90 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:45.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.458+0000 7f5208fc0740 1 -- 172.21.15.39:0/2561636224 >> 172.21.15.39:0/2561636224 conn(0x55fe1664ff80 msgr2=0x55fe1664e880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:45.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.458+0000 7f5208fc0740 1 -- 172.21.15.39:0/2561636224 shutdown_connections 2024-03-18T16:16:45.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.458+0000 7f5208fc0740 1 -- 172.21.15.39:0/2561636224 wait complete. 2024-03-18T16:16:45.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.458+0000 7f5208fc0740 1 -- start start 2024-03-18T16:16:45.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.459+0000 7f5208fc0740 1 Processor -- start 2024-03-18T16:16:45.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.459+0000 7f5208fc0740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fe16654a10 0x55fe1667b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:45.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.459+0000 7f5208fc0740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fe1665a570 0x55fe1667b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:45.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.459+0000 7f5208fc0740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fe16664680 0x55fe1667c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:45.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.460+0000 7f5208fc0740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f52001c2c40 con 0x55fe16664680 2024-03-18T16:16:45.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.460+0000 7f5208fc0740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f52001c2570 con 0x55fe1665a570 2024-03-18T16:16:45.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.460+0000 7f5208fc0740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f52001172f0 con 0x55fe16654a10 2024-03-18T16:16:45.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.460+0000 7f5208fc0740 10 client.0 _ll_register_callbacks cb 0x7f51f800aad0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:16:45.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.460+0000 7f51fea00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fe1665a570 0x55fe1667b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:45.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.460+0000 7f51fea00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fe1665a570 0x55fe1667b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:58368/0 (socket says 172.21.15.39:58368) 2024-03-18T16:16:45.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.460+0000 7f51fea00640 1 -- 172.21.15.39:0/1821831443 learned_addr learned my addr 172.21.15.39:0/1821831443 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:16:45.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.460+0000 7f51ff400640 1 --2- 172.21.15.39:0/1821831443 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fe16654a10 0x55fe1667b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:45.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.461+0000 7f51ff400640 1 -- 172.21.15.39:0/1821831443 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fe1665a570 msgr2=0x55fe1667b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:45.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.461+0000 7f51ff400640 1 --2- 172.21.15.39:0/1821831443 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fe1665a570 0x55fe1667b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:45.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.461+0000 7f51ff400640 1 -- 172.21.15.39:0/1821831443 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fe16664680 msgr2=0x55fe1667c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:45.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.461+0000 7f51ff400640 1 --2- 172.21.15.39:0/1821831443 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fe16664680 0x55fe1667c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:45.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.461+0000 7f51ff400640 1 -- 172.21.15.39:0/1821831443 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fe1667cf80 con 0x55fe16654a10 2024-03-18T16:16:45.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.461+0000 7f51ecc00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:16:45.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.461+0000 7f51ff400640 1 --2- 172.21.15.39:0/1821831443 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fe16654a10 0x55fe1667b440 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x55fe16655b70 tx=0x7f51f004a490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a836463fe5a5c83c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:45.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.462+0000 7f51ecc00640 1 -- 172.21.15.39:0/1821831443 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51f0039070 con 0x55fe16654a10 2024-03-18T16:16:45.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.462+0000 7f51ecc00640 1 -- 172.21.15.39:0/1821831443 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f51f0034a30 con 0x55fe16654a10 2024-03-18T16:16:45.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.462+0000 7f5208fc0740 1 -- 172.21.15.39:0/1821831443 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f51f0005bd0 con 0x55fe16654a10 2024-03-18T16:16:45.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.464+0000 7f51ecc00640 1 -- 172.21.15.39:0/1821831443 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f51f004e040 con 0x55fe16654a10 2024-03-18T16:16:45.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.464+0000 7f5208fc0740 10 client.26917 fetch_fsmap learned FSMap version 44 2024-03-18T16:16:45.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.464+0000 7f5208fc0740 1 -- 172.21.15.39:0/1821831443 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x55fe1667e290 con 0x55fe16654a10 2024-03-18T16:16:45.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.464+0000 7f51ecc00640 1 -- 172.21.15.39:0/1821831443 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51f0005bd0 con 0x55fe16654a10 2024-03-18T16:16:45.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.464+0000 7f51ecc00640 1 -- 172.21.15.39:0/1821831443 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7f51f0040bb0 con 0x55fe16654a10 2024-03-18T16:16:45.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.464+0000 7f5208fc0740 10 client.26917 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:16:45.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.465+0000 7f5208fc0740 10 client.26917 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:16:45.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.465+0000 7f5208fc0740 20 client.26917 populate_metadata read hostname 'smithi039' 2024-03-18T16:16:45.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.465+0000 7f5208fc0740 1 --2- 172.21.15.39:0/1821831443 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55fe166d0d80 0x55fe166d31a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:45.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.465+0000 7f5208fc0740 4 client.26917 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T16:16:45.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.465+0000 7f5208fc0740 1 -- 172.21.15.39:0/1821831443 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55fe165c6730 con 0x55fe166d0d80 2024-03-18T16:16:45.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.465+0000 7f51fea00640 1 --2- 172.21.15.39:0/1821831443 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55fe166d0d80 0x55fe166d31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:45.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.465+0000 7f51fea00640 1 --2- 172.21.15.39:0/1821831443 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55fe166d0d80 0x55fe166d31a0 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=71eb2fbac76a3a27 server_cookie=5535f0b43bc54fae in_seq=0 out_seq=0 2024-03-18T16:16:45.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.465+0000 7f51ecc00640 10 client.26917 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:16:45.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.466+0000 7f51ecc00640 1 -- 172.21.15.39:0/1821831443 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55fe165c6730 con 0x55fe166d0d80 2024-03-18T16:16:45.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.466+0000 7f51ecc00640 10 client.26917 handle_command_reply: tid=0 2024-03-18T16:16:45.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.466+0000 7f5208fc0740 1 client.26917 shutdown 2024-03-18T16:16:45.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.466+0000 7f5208fc0740 1 -- 172.21.15.39:0/1821831443 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fe16654a10 msgr2=0x55fe1667b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:45.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.466+0000 7f5208fc0740 1 --2- 172.21.15.39:0/1821831443 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fe16654a10 0x55fe1667b440 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x55fe16655b70 tx=0x7f51f004a490 comp rx=0 tx=0).stop 2024-03-18T16:16:45.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.466+0000 7f5208fc0740 1 -- 172.21.15.39:0/1821831443 shutdown_connections 2024-03-18T16:16:45.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.466+0000 7f5208fc0740 1 --2- 172.21.15.39:0/1821831443 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55fe166d0d80 0x55fe166d31a0 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:45.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.466+0000 7f5208fc0740 1 --2- 172.21.15.39:0/1821831443 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fe1665a570 0x55fe1667b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:45.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.466+0000 7f5208fc0740 1 --2- 172.21.15.39:0/1821831443 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fe16664680 0x55fe1667c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:45.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.466+0000 7f5208fc0740 1 --2- 172.21.15.39:0/1821831443 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fe16654a10 0x55fe1667b440 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:45.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.466+0000 7f51ecc00640 0 client.26917 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:16:45.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.466+0000 7f5208fc0740 1 -- 172.21.15.39:0/1821831443 >> 172.21.15.39:0/1821831443 conn(0x55fe1664ff80 msgr2=0x55fe166531e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:45.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.466+0000 7f5208fc0740 1 -- 172.21.15.39:0/1821831443 shutdown_connections 2024-03-18T16:16:45.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.466+0000 7f5208fc0740 1 -- 172.21.15.39:0/1821831443 wait complete. 2024-03-18T16:16:45.469 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.466+0000 7f5208fc0740 20 client.26917 trim_cache size 0 max 16384 2024-03-18T16:16:45.470 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-03-18T16:16:45.470 INFO:teuthology.orchestra.run.smithi039.stdout: "status": "no active scrubs running", 2024-03-18T16:16:45.470 INFO:teuthology.orchestra.run.smithi039.stdout: "scrubs": {} 2024-03-18T16:16:45.470 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-03-18T16:16:45.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.467+0000 7f5204c00640 1 -- 172.21.15.39:0/1736871978 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f51e40a2880 msgr2=0x7f51e40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:45.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.467+0000 7f5204c00640 1 --2- 172.21.15.39:0/1736871978 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f51e40a2880 0x7f51e40a4d40 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x55fe16655c60 tx=0x7f51f8004730 comp rx=0 tx=0).stop 2024-03-18T16:16:45.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.467+0000 7f5204c00640 1 -- 172.21.15.39:0/1736871978 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f520010db00 msgr2=0x7f52001adc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:45.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.467+0000 7f5204c00640 1 --2- 172.21.15.39:0/1736871978 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f520010db00 0x7f52001adc70 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f51e800c970 tx=0x7f51e800cd70 comp rx=0 tx=0).stop 2024-03-18T16:16:45.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.468+0000 7f5204c00640 1 -- 172.21.15.39:0/1736871978 shutdown_connections 2024-03-18T16:16:45.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.468+0000 7f5204c00640 1 --2- 172.21.15.39:0/1736871978 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f520010db00 0x7f52001adc70 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:45.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.468+0000 7f5204c00640 1 --2- 172.21.15.39:0/1736871978 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f51e40a2880 0x7f51e40a4d40 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:45.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.468+0000 7f5204c00640 1 --2- 172.21.15.39:0/1736871978 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f520010b150 0x7f52001a6c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:45.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.468+0000 7f5204c00640 1 --2- 172.21.15.39:0/1736871978 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f52001087e0 0x7f52001a66c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:45.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.468+0000 7f5204c00640 1 -- 172.21.15.39:0/1736871978 >> 172.21.15.39:0/1736871978 conn(0x7f520006f550 msgr2=0x7f520010bb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:45.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.470+0000 7f5204c00640 1 -- 172.21.15.39:0/1736871978 shutdown_connections 2024-03-18T16:16:45.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.470+0000 7f5204c00640 1 -- 172.21.15.39:0/1736871978 wait complete. 2024-03-18T16:16:45.497 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-18T16:16:45.497 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:16:45.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:45 smithi196 ceph-mon[36307]: pgmap v622: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.2 MiB/s wr, 316 op/s 2024-03-18T16:16:45.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:45 smithi194 ceph-mon[36200]: pgmap v622: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.2 MiB/s wr, 316 op/s 2024-03-18T16:16:45.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.778+0000 7f1f4c400640 1 -- 172.21.15.39:0/4216636940 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1f441007b0 msgr2=0x7f1f44100c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:45.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.778+0000 7f1f4c400640 1 --2- 172.21.15.39:0/4216636940 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1f441007b0 0x7f1f44100c10 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f1f38009900 tx=0x7f1f3802bc30 comp rx=0 tx=0).stop 2024-03-18T16:16:45.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.778+0000 7f1f4c400640 1 -- 172.21.15.39:0/4216636940 shutdown_connections 2024-03-18T16:16:45.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.778+0000 7f1f4c400640 1 --2- 172.21.15.39:0/4216636940 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1f441007b0 0x7f1f44100c10 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:45.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.778+0000 7f1f4c400640 1 --2- 172.21.15.39:0/4216636940 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f44101150 0x7f1f44105a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:45.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.778+0000 7f1f4c400640 1 --2- 172.21.15.39:0/4216636940 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1f440ff680 0x7f1f440ffaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:45.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.778+0000 7f1f4c400640 1 -- 172.21.15.39:0/4216636940 >> 172.21.15.39:0/4216636940 conn(0x7f1f440faf80 msgr2=0x7f1f440fd3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:45.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.778+0000 7f1f4c400640 1 -- 172.21.15.39:0/4216636940 shutdown_connections 2024-03-18T16:16:45.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.778+0000 7f1f4c400640 1 -- 172.21.15.39:0/4216636940 wait complete. 2024-03-18T16:16:45.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.779+0000 7f1f4c400640 1 Processor -- start 2024-03-18T16:16:45.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.780+0000 7f1f4c400640 1 -- start start 2024-03-18T16:16:45.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.780+0000 7f1f4c400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1f440ff680 0x7f1f4419e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:45.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.780+0000 7f1f4c400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f441007b0 0x7f1f4419e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:45.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.780+0000 7f1f4c400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1f44101150 0x7f1f441a5650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:45.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.780+0000 7f1f4c400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f1f44113ac0 con 0x7f1f441007b0 2024-03-18T16:16:45.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.780+0000 7f1f4c400640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f1f44113940 con 0x7f1f44101150 2024-03-18T16:16:45.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.780+0000 7f1f4c400640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f1f44113c40 con 0x7f1f440ff680 2024-03-18T16:16:45.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.781+0000 7f1f49a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1f440ff680 0x7f1f4419e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:45.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.781+0000 7f1f49000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f441007b0 0x7f1f4419e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:45.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.781+0000 7f1f49a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1f440ff680 0x7f1f4419e0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:54572/0 (socket says 172.21.15.39:54572) 2024-03-18T16:16:45.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.781+0000 7f1f49a00640 1 -- 172.21.15.39:0/2827206144 learned_addr learned my addr 172.21.15.39:0/2827206144 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:16:45.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.781+0000 7f1f4a400640 1 --2- 172.21.15.39:0/2827206144 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1f44101150 0x7f1f441a5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:45.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.781+0000 7f1f49000640 1 -- 172.21.15.39:0/2827206144 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1f440ff680 msgr2=0x7f1f4419e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:45.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.783+0000 7f1f49000640 1 --2- 172.21.15.39:0/2827206144 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1f440ff680 0x7f1f4419e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:45.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.783+0000 7f1f49000640 1 -- 172.21.15.39:0/2827206144 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1f44101150 msgr2=0x7f1f441a5650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:45.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.783+0000 7f1f49000640 1 --2- 172.21.15.39:0/2827206144 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1f44101150 0x7f1f441a5650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:45.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.783+0000 7f1f49000640 1 -- 172.21.15.39:0/2827206144 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f441073d0 con 0x7f1f441007b0 2024-03-18T16:16:45.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.783+0000 7f1f49a00640 1 --2- 172.21.15.39:0/2827206144 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1f440ff680 0x7f1f4419e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:16:45.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.785+0000 7f1f49000640 1 --2- 172.21.15.39:0/2827206144 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f441007b0 0x7f1f4419e5e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f1f38036040 tx=0x7f1f3802c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:45.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.785+0000 7f1f42000640 1 -- 172.21.15.39:0/2827206144 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f38043020 con 0x7f1f441007b0 2024-03-18T16:16:45.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.785+0000 7f1f42000640 1 -- 172.21.15.39:0/2827206144 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f380054e0 con 0x7f1f441007b0 2024-03-18T16:16:45.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.785+0000 7f1f42000640 1 -- 172.21.15.39:0/2827206144 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f38003c30 con 0x7f1f441007b0 2024-03-18T16:16:45.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.786+0000 7f1f4c400640 1 -- 172.21.15.39:0/2827206144 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f44107660 con 0x7f1f441007b0 2024-03-18T16:16:45.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.786+0000 7f1f4c400640 1 -- 172.21.15.39:0/2827206144 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f44107b30 con 0x7f1f441007b0 2024-03-18T16:16:45.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.788+0000 7f1f4c400640 1 -- 172.21.15.39:0/2827206144 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f44068a90 con 0x7f1f441007b0 2024-03-18T16:16:45.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.792+0000 7f1f42000640 1 -- 172.21.15.39:0/2827206144 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f1f38046020 con 0x7f1f441007b0 2024-03-18T16:16:45.796 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.795+0000 7f1f42000640 1 --2- 172.21.15.39:0/2827206144 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1f2c0a2900 0x7f1f2c0a4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:45.797 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.796+0000 7f1f42000640 1 -- 172.21.15.39:0/2827206144 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f1f380f0d50 con 0x7f1f441007b0 2024-03-18T16:16:45.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.796+0000 7f1f49a00640 1 --2- 172.21.15.39:0/2827206144 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1f2c0a2900 0x7f1f2c0a4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:45.800 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.798+0000 7f1f49a00640 1 --2- 172.21.15.39:0/2827206144 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1f2c0a2900 0x7f1f2c0a4dc0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f1f30011500 tx=0x7f1f30005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:45.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:45.801+0000 7f1f42000640 1 -- 172.21.15.39:0/2827206144 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f1f3804d020 con 0x7f1f441007b0 2024-03-18T16:16:45.836 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:45 smithi039 ceph-mon[31721]: pgmap v622: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.2 MiB/s wr, 316 op/s 2024-03-18T16:16:46.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.254+0000 7f1f4c400640 1 -- 172.21.15.39:0/2827206144 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1f440623e0 con 0x7f1f441007b0 2024-03-18T16:16:46.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.256+0000 7f1f42000640 1 -- 172.21.15.39:0/2827206144 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) ==== 94+0+11932 (secure 0 0 0) 0x7f1f380bda00 con 0x7f1f441007b0 2024-03-18T16:16:46.258 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:16:46.258 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":44,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2520226608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2520226608},{"type":"v1","addr":"172.21.15.39:6835","nonce":2520226608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/719341763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":719341763},{"type":"v1","addr":"172.21.15.39:6837","nonce":719341763}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":44,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T16:06:44.180425+0000","modified":"2024-03-18T16:15:58.268360+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24575,"mds_1":24556,"mds_2":24535,"mds_3":24524,"mds_4":14463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14445":{"gid":14445,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.39:6839/656513813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":656513813},{"type":"v1","addr":"172.21.15.39:6839","nonce":656513813}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14463":{"gid":14463,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":13,"addr":"172.21.15.39:6841/3515326042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3515326042},{"type":"v1","addr":"172.21.15.39:6841","nonce":3515326042}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6833/931350738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":931350738},{"type":"v1","addr":"172.21.15.194:6833","nonce":931350738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6833/3874209621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":3874209621},{"type":"v1","addr":"172.21.15.196:6833","nonce":3874209621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6835/188385470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":188385470},{"type":"v1","addr":"172.21.15.194:6835","nonce":188385470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6835/3949984309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":3949984309},{"type":"v1","addr":"172.21.15.196:6835","nonce":3949984309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":15,"addr":"172.21.15.194:6837/1301464246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":1301464246},{"type":"v1","addr":"172.21.15.194:6837","nonce":1301464246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":16,"addr":"172.21.15.196:6837/895247251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":895247251},{"type":"v1","addr":"172.21.15.196:6837","nonce":895247251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.196:6839/2001427268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6838","nonce":2001427268},{"type":"v1","addr":"172.21.15.196:6839","nonce":2001427268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24575":{"gid":24575,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.194:6839/274911158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":274911158},{"type":"v1","addr":"172.21.15.194:6839","nonce":274911158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24575,"qdb_cluster":[14463,24524,24575,24556,24535]},"id":1}]} 2024-03-18T16:16:46.259 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 44 2024-03-18T16:16:46.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.259+0000 7f1f4c400640 1 -- 172.21.15.39:0/2827206144 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1f2c0a2900 msgr2=0x7f1f2c0a4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:46.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.259+0000 7f1f4c400640 1 --2- 172.21.15.39:0/2827206144 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1f2c0a2900 0x7f1f2c0a4dc0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f1f30011500 tx=0x7f1f30005f90 comp rx=0 tx=0).stop 2024-03-18T16:16:46.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.259+0000 7f1f4c400640 1 -- 172.21.15.39:0/2827206144 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f441007b0 msgr2=0x7f1f4419e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:46.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.259+0000 7f1f4c400640 1 --2- 172.21.15.39:0/2827206144 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f441007b0 0x7f1f4419e5e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f1f38036040 tx=0x7f1f3802c650 comp rx=0 tx=0).stop 2024-03-18T16:16:46.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.260+0000 7f1f4c400640 1 -- 172.21.15.39:0/2827206144 shutdown_connections 2024-03-18T16:16:46.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.260+0000 7f1f4c400640 1 --2- 172.21.15.39:0/2827206144 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1f44101150 0x7f1f441a5650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:46.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.260+0000 7f1f4c400640 1 --2- 172.21.15.39:0/2827206144 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1f2c0a2900 0x7f1f2c0a4dc0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:46.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.260+0000 7f1f4c400640 1 --2- 172.21.15.39:0/2827206144 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1f441007b0 0x7f1f4419e5e0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:46.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.260+0000 7f1f4c400640 1 --2- 172.21.15.39:0/2827206144 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1f440ff680 0x7f1f4419e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:46.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.260+0000 7f1f4c400640 1 -- 172.21.15.39:0/2827206144 >> 172.21.15.39:0/2827206144 conn(0x7f1f440faf80 msgr2=0x7f1f441019f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:46.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.260+0000 7f1f4c400640 1 -- 172.21.15.39:0/2827206144 shutdown_connections 2024-03-18T16:16:46.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.260+0000 7f1f4c400640 1 -- 172.21.15.39:0/2827206144 wait complete. 2024-03-18T16:16:46.289 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-18T16:16:46.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.569+0000 7f316a400640 1 -- 172.21.15.39:0/1783274666 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3164103a00 msgr2=0x7f3164103e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:46.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.569+0000 7f316a400640 1 --2- 172.21.15.39:0/1783274666 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3164103a00 0x7f3164103e20 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f3150009900 tx=0x7f315002bc30 comp rx=0 tx=0).stop 2024-03-18T16:16:46.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.569+0000 7f316a400640 1 -- 172.21.15.39:0/1783274666 shutdown_connections 2024-03-18T16:16:46.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.569+0000 7f316a400640 1 --2- 172.21.15.39:0/1783274666 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f31641055b0 0x7f316410be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:46.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.569+0000 7f316a400640 1 --2- 172.21.15.39:0/1783274666 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3164104bf0 0x7f3164105070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:46.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.569+0000 7f316a400640 1 --2- 172.21.15.39:0/1783274666 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3164103a00 0x7f3164103e20 secure :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f3150009900 tx=0x7f315002bc30 comp rx=0 tx=0).stop 2024-03-18T16:16:46.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.569+0000 7f316a400640 1 -- 172.21.15.39:0/1783274666 >> 172.21.15.39:0/1783274666 conn(0x7f31640fef50 msgr2=0x7f31641013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:46.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.569+0000 7f316a400640 1 -- 172.21.15.39:0/1783274666 shutdown_connections 2024-03-18T16:16:46.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.569+0000 7f316a400640 1 -- 172.21.15.39:0/1783274666 wait complete. 2024-03-18T16:16:46.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.572+0000 7f316a400640 1 Processor -- start 2024-03-18T16:16:46.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.572+0000 7f316a400640 1 -- start start 2024-03-18T16:16:46.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.573+0000 7f316a400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3164104bf0 0x7f316406d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:46.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.573+0000 7f316a400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f31641055b0 0x7f316406d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:46.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.573+0000 7f316a400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3164072860 0x7f3164072d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:46.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.573+0000 7f316a400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f316406bc50 con 0x7f31641055b0 2024-03-18T16:16:46.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.573+0000 7f316a400640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f316406bad0 con 0x7f3164104bf0 2024-03-18T16:16:46.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.573+0000 7f316a400640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f316406bdd0 con 0x7f3164072860 2024-03-18T16:16:46.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.573+0000 7f3163e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3164072860 0x7f3164072d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:46.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.573+0000 7f3163e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3164072860 0x7f3164072d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:54594/0 (socket says 172.21.15.39:54594) 2024-03-18T16:16:46.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.573+0000 7f3163e00640 1 -- 172.21.15.39:0/67728058 learned_addr learned my addr 172.21.15.39:0/67728058 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:16:46.575 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.573+0000 7f3163400640 1 --2- 172.21.15.39:0/67728058 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3164104bf0 0x7f316406d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:46.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.574+0000 7f3162a00640 1 --2- 172.21.15.39:0/67728058 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f31641055b0 0x7f316406d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:46.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.575+0000 7f3163400640 1 -- 172.21.15.39:0/67728058 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3164072860 msgr2=0x7f3164072d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:46.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.575+0000 7f3163400640 1 --2- 172.21.15.39:0/67728058 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3164072860 0x7f3164072d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:46.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.575+0000 7f3163400640 1 -- 172.21.15.39:0/67728058 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f31641055b0 msgr2=0x7f316406d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:46.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.575+0000 7f3163400640 1 --2- 172.21.15.39:0/67728058 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f31641055b0 0x7f316406d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:46.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.575+0000 7f3163400640 1 -- 172.21.15.39:0/67728058 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31640732d0 con 0x7f3164104bf0 2024-03-18T16:16:46.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.577+0000 7f3163400640 1 --2- 172.21.15.39:0/67728058 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3164104bf0 0x7f316406d2c0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f3164103540 tx=0x7f3150005e60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:46.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.578+0000 7f315be00640 1 -- 172.21.15.39:0/67728058 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3150043020 con 0x7f3164104bf0 2024-03-18T16:16:46.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.578+0000 7f315be00640 1 -- 172.21.15.39:0/67728058 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f31500054e0 con 0x7f3164104bf0 2024-03-18T16:16:46.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.578+0000 7f315be00640 1 -- 172.21.15.39:0/67728058 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3150003de0 con 0x7f3164104bf0 2024-03-18T16:16:46.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.578+0000 7f316a400640 1 -- 172.21.15.39:0/67728058 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3164073560 con 0x7f3164104bf0 2024-03-18T16:16:46.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.578+0000 7f316a400640 1 -- 172.21.15.39:0/67728058 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31641af370 con 0x7f3164104bf0 2024-03-18T16:16:46.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.584+0000 7f315be00640 1 -- 172.21.15.39:0/67728058 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f3150046020 con 0x7f3164104bf0 2024-03-18T16:16:46.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.585+0000 7f316d1d4740 1 Processor -- start 2024-03-18T16:16:46.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:46 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/2827206144' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:16:46.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.585+0000 7f316d1d4740 1 -- start start 2024-03-18T16:16:46.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.585+0000 7f316d1d4740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562447d278c0 0x562447d27ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:46.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.585+0000 7f316d1d4740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562447d282b0 0x562447d2cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:46.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.585+0000 7f316d1d4740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562447d2d420 0x562447d2f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:46.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.585+0000 7f316d1d4740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x562447cef3b0 con 0x562447d278c0 2024-03-18T16:16:46.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.585+0000 7f316d1d4740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x562447c8c450 con 0x562447d282b0 2024-03-18T16:16:46.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.585+0000 7f316d1d4740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x562447c34cb0 con 0x562447d2d420 2024-03-18T16:16:46.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.586+0000 7f3162a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562447d2d420 0x562447d2f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:46.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.586+0000 7f3162a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562447d2d420 0x562447d2f8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:54610/0 (socket says 172.21.15.39:54610) 2024-03-18T16:16:46.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.586+0000 7f3163e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562447d278c0 0x562447d27ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:46.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.586+0000 7f3162a00640 1 -- 172.21.15.39:0/2138408294 learned_addr learned my addr 172.21.15.39:0/2138408294 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:16:46.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.586+0000 7f3163400640 1 --2- 172.21.15.39:0/2138408294 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562447d282b0 0x562447d2cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:46.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.586+0000 7f3163e00640 1 -- 172.21.15.39:0/2138408294 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562447d2d420 msgr2=0x562447d2f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:46.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.586+0000 7f3163e00640 1 --2- 172.21.15.39:0/2138408294 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562447d2d420 0x562447d2f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:46.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.586+0000 7f3163e00640 1 -- 172.21.15.39:0/2138408294 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562447d282b0 msgr2=0x562447d2cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:46.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.586+0000 7f3163e00640 1 --2- 172.21.15.39:0/2138408294 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562447d282b0 0x562447d2cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:46.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.586+0000 7f3163e00640 1 -- 172.21.15.39:0/2138408294 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562447c561f0 con 0x562447d278c0 2024-03-18T16:16:46.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.587+0000 7f3163e00640 1 --2- 172.21.15.39:0/2138408294 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562447d278c0 0x562447d27ce0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f315c007f00 tx=0x7f315c00f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b3b10e298b76834 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:46.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.587+0000 7f315be00640 1 --2- 172.21.15.39:0/67728058 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f31480a2880 0x7f31480a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:46.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.587+0000 7f3158c00640 1 -- 172.21.15.39:0/2138408294 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f315c004d20 con 0x562447d278c0 2024-03-18T16:16:46.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.588+0000 7f315be00640 1 -- 172.21.15.39:0/67728058 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f31500f0390 con 0x7f3164104bf0 2024-03-18T16:16:46.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.588+0000 7f3158c00640 1 -- 172.21.15.39:0/2138408294 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f315c010b20 con 0x562447d278c0 2024-03-18T16:16:46.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.588+0000 7f3162a00640 1 --2- 172.21.15.39:0/2138408294 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562447d2d420 0x562447d2f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:16:46.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.589+0000 7f3163400640 1 --2- 172.21.15.39:0/67728058 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f31480a2880 0x7f31480a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:46.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.589+0000 7f316d1d4740 1 -- 172.21.15.39:0/2138408294 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562447d278c0 msgr2=0x562447d27ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:46.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.589+0000 7f3158c00640 1 -- 172.21.15.39:0/2138408294 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f315c005770 con 0x562447d278c0 2024-03-18T16:16:46.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.589+0000 7f316d1d4740 1 --2- 172.21.15.39:0/2138408294 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562447d278c0 0x562447d27ce0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f315c007f00 tx=0x7f315c00f790 comp rx=0 tx=0).stop 2024-03-18T16:16:46.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.589+0000 7f316d1d4740 1 -- 172.21.15.39:0/2138408294 shutdown_connections 2024-03-18T16:16:46.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.589+0000 7f316d1d4740 1 --2- 172.21.15.39:0/2138408294 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562447d282b0 0x562447d2cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:46.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.589+0000 7f316d1d4740 1 --2- 172.21.15.39:0/2138408294 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562447d2d420 0x562447d2f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:46.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.589+0000 7f316d1d4740 1 --2- 172.21.15.39:0/2138408294 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562447d278c0 0x562447d27ce0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:46.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.589+0000 7f316d1d4740 1 -- 172.21.15.39:0/2138408294 >> 172.21.15.39:0/2138408294 conn(0x562447d22e30 msgr2=0x562447d23230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:46.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.590+0000 7f316d1d4740 1 -- 172.21.15.39:0/2138408294 shutdown_connections 2024-03-18T16:16:46.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.590+0000 7f316d1d4740 1 -- 172.21.15.39:0/2138408294 wait complete. 2024-03-18T16:16:46.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.590+0000 7f3163400640 1 --2- 172.21.15.39:0/67728058 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f31480a2880 0x7f31480a4d40 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f31500bc8d0 tx=0x7f315002bc30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:46.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.591+0000 7f316d1d4740 1 -- start start 2024-03-18T16:16:46.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.591+0000 7f316d1d4740 1 Processor -- start 2024-03-18T16:16:46.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.592+0000 7f316d1d4740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562447d278c0 0x562447d319f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:46.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.592+0000 7f316d1d4740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562447d282b0 0x562447d31f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:46.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.592+0000 7f3163e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562447d278c0 0x562447d319f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:46.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.592+0000 7f3163e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562447d278c0 0x562447d319f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:58420/0 (socket says 172.21.15.39:58420) 2024-03-18T16:16:46.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.592+0000 7f3163e00640 1 -- 172.21.15.39:0/3566417997 learned_addr learned my addr 172.21.15.39:0/3566417997 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:16:46.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.593+0000 7f316d1d4740 1 --2- 172.21.15.39:0/3566417997 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562447d2d420 0x562447d38f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:46.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.593+0000 7f316d1d4740 1 -- 172.21.15.39:0/3566417997 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x562447c48110 con 0x562447d282b0 2024-03-18T16:16:46.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.593+0000 7f316d1d4740 1 -- 172.21.15.39:0/3566417997 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x562447c884f0 con 0x562447d278c0 2024-03-18T16:16:46.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.593+0000 7f316d1d4740 1 -- 172.21.15.39:0/3566417997 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x562447d30420 con 0x562447d2d420 2024-03-18T16:16:46.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.593+0000 7f316d1d4740 10 client.0 _ll_register_callbacks cb 0x562447c35b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:16:46.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.593+0000 7f3163e00640 1 --2- 172.21.15.39:0/3566417997 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562447d2d420 0x562447d38f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:46.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.594+0000 7f3163e00640 1 -- 172.21.15.39:0/3566417997 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562447d2d420 msgr2=0x562447d38f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:46.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.594+0000 7f3163e00640 1 --2- 172.21.15.39:0/3566417997 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562447d2d420 0x562447d38f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:46.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.594+0000 7f3163e00640 1 -- 172.21.15.39:0/3566417997 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562447d282b0 msgr2=0x562447d31f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:46.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.594+0000 7f3162a00640 1 --2- 172.21.15.39:0/3566417997 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562447d282b0 0x562447d31f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:46.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.594+0000 7f3163e00640 1 --2- 172.21.15.39:0/3566417997 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562447d282b0 0x562447d31f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:46.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.594+0000 7f3163e00640 1 -- 172.21.15.39:0/3566417997 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5624479a03f0 con 0x562447d278c0 2024-03-18T16:16:46.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.594+0000 7f3163e00640 1 --2- 172.21.15.39:0/3566417997 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562447d278c0 0x562447d319f0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f315c007ed0 tx=0x7f315c007b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1fd24dbcd09291ed server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:46.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.596+0000 7f314d600640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:16:46.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.596+0000 7f314d600640 1 -- 172.21.15.39:0/3566417997 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f315c01d040 con 0x562447d278c0 2024-03-18T16:16:46.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.596+0000 7f314d600640 1 -- 172.21.15.39:0/3566417997 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f315c003970 con 0x562447d278c0 2024-03-18T16:16:46.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.596+0000 7f316d1d4740 1 -- 172.21.15.39:0/3566417997 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f315c010b20 con 0x562447d278c0 2024-03-18T16:16:46.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.597+0000 7f314d600640 1 -- 172.21.15.39:0/3566417997 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f315c003b70 con 0x562447d278c0 2024-03-18T16:16:46.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.597+0000 7f316d1d4740 10 client.26954 fetch_fsmap learned FSMap version 44 2024-03-18T16:16:46.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.597+0000 7f316d1d4740 1 -- 172.21.15.39:0/3566417997 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x562447a6a7d0 con 0x562447d278c0 2024-03-18T16:16:46.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.597+0000 7f314d600640 1 -- 172.21.15.39:0/3566417997 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f315c010b20 con 0x562447d278c0 2024-03-18T16:16:46.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.598+0000 7f314d600640 1 -- 172.21.15.39:0/3566417997 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7f315c028440 con 0x562447d278c0 2024-03-18T16:16:46.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.598+0000 7f316d1d4740 10 client.26954 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:16:46.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.598+0000 7f316d1d4740 10 client.26954 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-03-18T16:16:46.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.598+0000 7f316d1d4740 20 client.26954 populate_metadata read hostname 'smithi039' 2024-03-18T16:16:46.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.598+0000 7f316d1d4740 1 --2- 172.21.15.39:0/3566417997 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x562447d3d5b0 0x562447d3f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:46.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.598+0000 7f316d1d4740 4 client.26954 mds_command: new command op to 14463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:16:46.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.598+0000 7f316d1d4740 1 -- 172.21.15.39:0/3566417997 --> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562447c99990 con 0x562447d3d5b0 2024-03-18T16:16:46.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.598+0000 7f3162a00640 1 --2- 172.21.15.39:0/3566417997 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x562447d3d5b0 0x562447d3f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:46.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.599+0000 7f3162a00640 1 --2- 172.21.15.39:0/3566417997 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x562447d3d5b0 0x562447d3f9a0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=a26bc15c971aee08 server_cookie=b17f8dfe948b0a53 in_seq=0 out_seq=0 2024-03-18T16:16:46.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.599+0000 7f314d600640 10 client.26954 ms_handle_connect on v2:172.21.15.39:6840/3515326042 2024-03-18T16:16:46.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.600+0000 7f314d600640 1 -- 172.21.15.39:0/3566417997 <== mds.4 v2:172.21.15.39:6840/3515326042 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562447c99990 con 0x562447d3d5b0 2024-03-18T16:16:46.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.600+0000 7f314d600640 10 client.26954 handle_command_reply: tid=0 2024-03-18T16:16:46.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.600+0000 7f316d1d4740 1 client.26954 shutdown 2024-03-18T16:16:46.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.600+0000 7f316d1d4740 1 -- 172.21.15.39:0/3566417997 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562447d278c0 msgr2=0x562447d319f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:46.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.600+0000 7f316d1d4740 1 --2- 172.21.15.39:0/3566417997 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562447d278c0 0x562447d319f0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f315c007ed0 tx=0x7f315c007b50 comp rx=0 tx=0).stop 2024-03-18T16:16:46.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.601+0000 7f316d1d4740 1 -- 172.21.15.39:0/3566417997 shutdown_connections 2024-03-18T16:16:46.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.601+0000 7f316d1d4740 1 --2- 172.21.15.39:0/3566417997 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x562447d3d5b0 0x562447d3f9a0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:46.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.601+0000 7f316d1d4740 1 --2- 172.21.15.39:0/3566417997 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562447d2d420 0x562447d38f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:46.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.601+0000 7f316d1d4740 1 --2- 172.21.15.39:0/3566417997 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562447d282b0 0x562447d31f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:46.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.601+0000 7f316d1d4740 1 --2- 172.21.15.39:0/3566417997 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562447d278c0 0x562447d319f0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:46.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.601+0000 7f316d1d4740 1 -- 172.21.15.39:0/3566417997 >> 172.21.15.39:0/3566417997 conn(0x562447d22e30 msgr2=0x562447d221f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:46.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.602+0000 7f314d600640 0 client.26954 ms_handle_reset on v2:172.21.15.39:6840/3515326042 2024-03-18T16:16:46.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.602+0000 7f316d1d4740 1 -- 172.21.15.39:0/3566417997 shutdown_connections 2024-03-18T16:16:46.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.602+0000 7f316d1d4740 1 -- 172.21.15.39:0/3566417997 wait complete. 2024-03-18T16:16:46.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.604+0000 7f316d1d4740 20 client.26954 trim_cache size 0 max 16384 2024-03-18T16:16:46.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.628+0000 7f316d1d4740 1 Processor -- start 2024-03-18T16:16:46.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.628+0000 7f316d1d4740 1 -- start start 2024-03-18T16:16:46.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.628+0000 7f316d1d4740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562447d278c0 0x562447d23210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:46.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.628+0000 7f316d1d4740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562447d282b0 0x562447d31600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:46.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.628+0000 7f316d1d4740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562447d2d420 0x562447d31b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:46.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.628+0000 7f316d1d4740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f31641b9700 con 0x562447d278c0 2024-03-18T16:16:46.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.628+0000 7f316d1d4740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f31641bb240 con 0x562447d282b0 2024-03-18T16:16:46.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.628+0000 7f316d1d4740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f31641b9dd0 con 0x562447d2d420 2024-03-18T16:16:46.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.629+0000 7f3163e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562447d2d420 0x562447d31b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:46.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.629+0000 7f3163e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562447d2d420 0x562447d31b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:54632/0 (socket says 172.21.15.39:54632) 2024-03-18T16:16:46.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.629+0000 7f3163e00640 1 -- 172.21.15.39:0/1466578347 learned_addr learned my addr 172.21.15.39:0/1466578347 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:16:46.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.629+0000 7f3162a00640 1 --2- 172.21.15.39:0/1466578347 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562447d282b0 0x562447d31600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:46.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.629+0000 7f3163e00640 1 -- 172.21.15.39:0/1466578347 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562447d282b0 msgr2=0x562447d31600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:46.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.629+0000 7f3163e00640 1 --2- 172.21.15.39:0/1466578347 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562447d282b0 0x562447d31600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:46.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.629+0000 7f3163e00640 1 -- 172.21.15.39:0/1466578347 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562447d278c0 msgr2=0x562447d23210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:46.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.629+0000 7f3163e00640 1 --2- 172.21.15.39:0/1466578347 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562447d278c0 0x562447d23210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:46.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.629+0000 7f3163e00640 1 -- 172.21.15.39:0/1466578347 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562447d28f80 con 0x562447d2d420 2024-03-18T16:16:46.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.630+0000 7f3162a00640 1 --2- 172.21.15.39:0/1466578347 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562447d282b0 0x562447d31600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:16:46.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.630+0000 7f3163e00640 1 --2- 172.21.15.39:0/1466578347 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562447d2d420 0x562447d31b40 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f315c007c70 tx=0x7f315c007920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5e8d04cd7e3a353 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:46.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.630+0000 7f314fe00640 1 -- 172.21.15.39:0/1466578347 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f315c03c070 con 0x562447d2d420 2024-03-18T16:16:46.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.630+0000 7f314fe00640 1 -- 172.21.15.39:0/1466578347 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f315c00fc90 con 0x562447d2d420 2024-03-18T16:16:46.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.630+0000 7f314fe00640 1 -- 172.21.15.39:0/1466578347 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f315c005030 con 0x562447d2d420 2024-03-18T16:16:46.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.631+0000 7f316d1d4740 1 -- 172.21.15.39:0/1466578347 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562447d2d420 msgr2=0x562447d31b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:46.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.631+0000 7f316d1d4740 1 --2- 172.21.15.39:0/1466578347 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562447d2d420 0x562447d31b40 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f315c007c70 tx=0x7f315c007920 comp rx=0 tx=0).stop 2024-03-18T16:16:46.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.631+0000 7f316d1d4740 1 -- 172.21.15.39:0/1466578347 shutdown_connections 2024-03-18T16:16:46.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.631+0000 7f316d1d4740 1 --2- 172.21.15.39:0/1466578347 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562447d282b0 0x562447d31600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:46.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.631+0000 7f316d1d4740 1 --2- 172.21.15.39:0/1466578347 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562447d2d420 0x562447d31b40 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:46.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.631+0000 7f316d1d4740 1 --2- 172.21.15.39:0/1466578347 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562447d278c0 0x562447d23210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:46.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.631+0000 7f316d1d4740 1 -- 172.21.15.39:0/1466578347 >> 172.21.15.39:0/1466578347 conn(0x562447d22e30 msgr2=0x562447d2cac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:46.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.631+0000 7f316d1d4740 1 -- 172.21.15.39:0/1466578347 shutdown_connections 2024-03-18T16:16:46.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.631+0000 7f316d1d4740 1 -- 172.21.15.39:0/1466578347 wait complete. 2024-03-18T16:16:46.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.632+0000 7f316d1d4740 1 -- start start 2024-03-18T16:16:46.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.632+0000 7f316d1d4740 1 Processor -- start 2024-03-18T16:16:46.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.632+0000 7f316d1d4740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562447d278c0 0x562447d35860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:46.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.632+0000 7f316d1d4740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562447d282b0 0x562447d35da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:46.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.632+0000 7f316d1d4740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562447d2d420 0x562447da18d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:46.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.632+0000 7f316d1d4740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f31641bab70 con 0x562447d2d420 2024-03-18T16:16:46.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.632+0000 7f316d1d4740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f31641ba4a0 con 0x562447d282b0 2024-03-18T16:16:46.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.632+0000 7f316d1d4740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f31641b76e0 con 0x562447d278c0 2024-03-18T16:16:46.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.632+0000 7f316d1d4740 10 client.0 _ll_register_callbacks cb 0x7f315c007d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:16:46.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.633+0000 7f3163e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562447d278c0 0x562447d35860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:46.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.633+0000 7f3162a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562447d282b0 0x562447d35da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:46.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.633+0000 7f3163e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562447d278c0 0x562447d35860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:54644/0 (socket says 172.21.15.39:54644) 2024-03-18T16:16:46.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.633+0000 7f3163e00640 1 -- 172.21.15.39:0/1832106395 learned_addr learned my addr 172.21.15.39:0/1832106395 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:16:46.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.633+0000 7f3163e00640 1 --2- 172.21.15.39:0/1832106395 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562447d2d420 0x562447da18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:46.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.633+0000 7f3163e00640 1 -- 172.21.15.39:0/1832106395 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562447d282b0 msgr2=0x562447d35da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:46.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.633+0000 7f3163e00640 1 --2- 172.21.15.39:0/1832106395 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562447d282b0 0x562447d35da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:46.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.633+0000 7f3163e00640 1 -- 172.21.15.39:0/1832106395 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562447d2d420 msgr2=0x562447da18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:46.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.633+0000 7f3163e00640 1 --2- 172.21.15.39:0/1832106395 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562447d2d420 0x562447da18d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:46.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.633+0000 7f3163e00640 1 -- 172.21.15.39:0/1832106395 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562447d38e70 con 0x562447d278c0 2024-03-18T16:16:46.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.633+0000 7f3162a00640 1 --2- 172.21.15.39:0/1832106395 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562447d282b0 0x562447d35da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:16:46.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.634+0000 7f3163e00640 1 --2- 172.21.15.39:0/1832106395 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562447d278c0 0x562447d35860 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f315c005ef0 tx=0x7f315c012040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e5d8cbb460f11e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:46.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.634+0000 7f314d600640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:16:46.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.634+0000 7f314d600640 1 -- 172.21.15.39:0/1832106395 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f315c03c030 con 0x562447d278c0 2024-03-18T16:16:46.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.634+0000 7f314d600640 1 -- 172.21.15.39:0/1832106395 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f315c028a00 con 0x562447d278c0 2024-03-18T16:16:46.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.634+0000 7f316d1d4740 1 -- 172.21.15.39:0/1832106395 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f315c00fc90 con 0x562447d278c0 2024-03-18T16:16:46.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.634+0000 7f314d600640 1 -- 172.21.15.39:0/1832106395 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f315c061050 con 0x562447d278c0 2024-03-18T16:16:46.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.634+0000 7f316d1d4740 10 client.26941 fetch_fsmap learned FSMap version 44 2024-03-18T16:16:46.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.635+0000 7f316d1d4740 1 -- 172.21.15.39:0/1832106395 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x562447da2de0 con 0x562447d278c0 2024-03-18T16:16:46.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.635+0000 7f314d600640 1 -- 172.21.15.39:0/1832106395 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f315c00fc90 con 0x562447d278c0 2024-03-18T16:16:46.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.635+0000 7f314d600640 1 -- 172.21.15.39:0/1832106395 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7f315c038a70 con 0x562447d278c0 2024-03-18T16:16:46.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.635+0000 7f316d1d4740 10 client.26941 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:16:46.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.635+0000 7f316d1d4740 10 client.26941 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-03-18T16:16:46.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.635+0000 7f316d1d4740 20 client.26941 populate_metadata read hostname 'smithi039' 2024-03-18T16:16:46.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.635+0000 7f316d1d4740 1 --2- 172.21.15.39:0/1832106395 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x562447da3540 0x562447da5960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:46.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.635+0000 7f316d1d4740 4 client.26941 mds_command: new command op to 14463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:16:46.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.635+0000 7f316d1d4740 1 -- 172.21.15.39:0/1832106395 --> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562447c93ce0 con 0x562447da3540 2024-03-18T16:16:46.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.636+0000 7f3162a00640 1 --2- 172.21.15.39:0/1832106395 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x562447da3540 0x562447da5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:46.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.636+0000 7f314d600640 10 client.26941 ms_handle_connect on v2:172.21.15.39:6840/3515326042 2024-03-18T16:16:46.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.636+0000 7f3162a00640 1 --2- 172.21.15.39:0/1832106395 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x562447da3540 0x562447da5960 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=854bc99e501675f1 server_cookie=88551b7f75ae88be in_seq=0 out_seq=0 2024-03-18T16:16:46.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.637+0000 7f314d600640 1 -- 172.21.15.39:0/1832106395 <== mds.4 v2:172.21.15.39:6840/3515326042 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562447c93ce0 con 0x562447da3540 2024-03-18T16:16:46.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.637+0000 7f314d600640 10 client.26941 handle_command_reply: tid=0 2024-03-18T16:16:46.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.637+0000 7f316d1d4740 1 client.26941 shutdown 2024-03-18T16:16:46.641 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.637+0000 7f316d1d4740 1 -- 172.21.15.39:0/1832106395 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562447d278c0 msgr2=0x562447d35860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:46.641 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.637+0000 7f316d1d4740 1 --2- 172.21.15.39:0/1832106395 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562447d278c0 0x562447d35860 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f315c005ef0 tx=0x7f315c012040 comp rx=0 tx=0).stop 2024-03-18T16:16:46.641 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.637+0000 7f316d1d4740 1 -- 172.21.15.39:0/1832106395 shutdown_connections 2024-03-18T16:16:46.641 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.637+0000 7f316d1d4740 1 --2- 172.21.15.39:0/1832106395 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x562447da3540 0x562447da5960 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:46.641 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.637+0000 7f316d1d4740 1 --2- 172.21.15.39:0/1832106395 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562447d282b0 0x562447d35da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:46.641 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.637+0000 7f316d1d4740 1 --2- 172.21.15.39:0/1832106395 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562447d2d420 0x562447da18d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:46.641 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.637+0000 7f316d1d4740 1 --2- 172.21.15.39:0/1832106395 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562447d278c0 0x562447d35860 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:46.641 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.638+0000 7f316d1d4740 1 -- 172.21.15.39:0/1832106395 >> 172.21.15.39:0/1832106395 conn(0x562447d22e30 msgr2=0x562447d2c690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:46.641 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.638+0000 7f314d600640 0 client.26941 ms_handle_reset on v2:172.21.15.39:6840/3515326042 2024-03-18T16:16:46.641 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.638+0000 7f316d1d4740 1 -- 172.21.15.39:0/1832106395 shutdown_connections 2024-03-18T16:16:46.641 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.638+0000 7f316d1d4740 1 -- 172.21.15.39:0/1832106395 wait complete. 2024-03-18T16:16:46.641 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.638+0000 7f316d1d4740 20 client.26941 trim_cache size 0 max 16384 2024-03-18T16:16:46.641 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:16:46.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.642+0000 7f316a400640 1 -- 172.21.15.39:0/67728058 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f31480a2880 msgr2=0x7f31480a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:46.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.642+0000 7f316a400640 1 --2- 172.21.15.39:0/67728058 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f31480a2880 0x7f31480a4d40 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f31500bc8d0 tx=0x7f315002bc30 comp rx=0 tx=0).stop 2024-03-18T16:16:46.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.642+0000 7f316a400640 1 -- 172.21.15.39:0/67728058 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3164104bf0 msgr2=0x7f316406d2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:46.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.642+0000 7f316a400640 1 --2- 172.21.15.39:0/67728058 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3164104bf0 0x7f316406d2c0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f3164103540 tx=0x7f3150005e60 comp rx=0 tx=0).stop 2024-03-18T16:16:46.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.642+0000 7f316a400640 1 -- 172.21.15.39:0/67728058 shutdown_connections 2024-03-18T16:16:46.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.642+0000 7f316a400640 1 --2- 172.21.15.39:0/67728058 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f31480a2880 0x7f31480a4d40 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:46.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.642+0000 7f316a400640 1 --2- 172.21.15.39:0/67728058 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3164072860 0x7f3164072d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:46.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.642+0000 7f316a400640 1 --2- 172.21.15.39:0/67728058 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f31641055b0 0x7f316406d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:46.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.642+0000 7f316a400640 1 --2- 172.21.15.39:0/67728058 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3164104bf0 0x7f316406d2c0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:46.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.642+0000 7f316a400640 1 -- 172.21.15.39:0/67728058 >> 172.21.15.39:0/67728058 conn(0x7f31640fef50 msgr2=0x7f3164077c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:46.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.642+0000 7f316a400640 1 -- 172.21.15.39:0/67728058 shutdown_connections 2024-03-18T16:16:46.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.643+0000 7f316a400640 1 -- 172.21.15.39:0/67728058 wait complete. 2024-03-18T16:16:46.668 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-18T16:16:46.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:46 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/2827206144' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:16:46.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:46 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/2827206144' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:16:46.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.949+0000 7f1c8e200640 1 -- 172.21.15.39:0/1965764053 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1c88108f70 msgr2=0x7f1c881093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:46.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.949+0000 7f1c8e200640 1 --2- 172.21.15.39:0/1965764053 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1c88108f70 0x7f1c881093f0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f1c74009900 tx=0x7f1c7402bc40 comp rx=0 tx=0).stop 2024-03-18T16:16:46.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.949+0000 7f1c8e200640 1 -- 172.21.15.39:0/1965764053 shutdown_connections 2024-03-18T16:16:46.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.949+0000 7f1c8e200640 1 --2- 172.21.15.39:0/1965764053 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1c88109930 0x7f1c88110250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:46.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.949+0000 7f1c8e200640 1 --2- 172.21.15.39:0/1965764053 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1c88108f70 0x7f1c881093f0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:46.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.949+0000 7f1c8e200640 1 --2- 172.21.15.39:0/1965764053 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1c88107e20 0x7f1c88108240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:46.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.949+0000 7f1c8e200640 1 -- 172.21.15.39:0/1965764053 >> 172.21.15.39:0/1965764053 conn(0x7f1c88076dd0 msgr2=0x7f1c880791f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:46.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.949+0000 7f1c8e200640 1 -- 172.21.15.39:0/1965764053 shutdown_connections 2024-03-18T16:16:46.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.949+0000 7f1c8e200640 1 -- 172.21.15.39:0/1965764053 wait complete. 2024-03-18T16:16:46.952 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.950+0000 7f1c8e200640 1 Processor -- start 2024-03-18T16:16:46.952 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.951+0000 7f1c8e200640 1 -- start start 2024-03-18T16:16:46.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.951+0000 7f1c8e200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1c88107e20 0x7f1c8807efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:46.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.952+0000 7f1c8e200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1c88108f70 0x7f1c8807f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:46.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.952+0000 7f1c8e200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1c88109930 0x7f1c881ae200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:46.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.952+0000 7f1c8e200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f1c88113af0 con 0x7f1c88107e20 2024-03-18T16:16:46.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.952+0000 7f1c8e200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f1c88113970 con 0x7f1c88109930 2024-03-18T16:16:46.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.952+0000 7f1c8e200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f1c88113c70 con 0x7f1c88108f70 2024-03-18T16:16:46.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.953+0000 7f1c87e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1c88109930 0x7f1c881ae200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:46.954 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.953+0000 7f1c87400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1c88107e20 0x7f1c8807efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:46.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.953+0000 7f1c87e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1c88109930 0x7f1c881ae200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:58450/0 (socket says 172.21.15.39:58450) 2024-03-18T16:16:46.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.953+0000 7f1c87e00640 1 -- 172.21.15.39:0/2871705591 learned_addr learned my addr 172.21.15.39:0/2871705591 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:16:46.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.953+0000 7f1c87e00640 1 -- 172.21.15.39:0/2871705591 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1c88108f70 msgr2=0x7f1c8807f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:46.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.953+0000 7f1c87e00640 1 --2- 172.21.15.39:0/2871705591 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1c88108f70 0x7f1c8807f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:46.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.953+0000 7f1c87e00640 1 -- 172.21.15.39:0/2871705591 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1c88107e20 msgr2=0x7f1c8807efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:46.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.953+0000 7f1c87e00640 1 --2- 172.21.15.39:0/2871705591 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1c88107e20 0x7f1c8807efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:46.955 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.953+0000 7f1c87e00640 1 -- 172.21.15.39:0/2871705591 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c8807fbd0 con 0x7f1c88109930 2024-03-18T16:16:46.956 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.954+0000 7f1c87e00640 1 --2- 172.21.15.39:0/2871705591 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1c88109930 0x7f1c881ae200 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f1c8000efd0 tx=0x7f1c8000c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:46.956 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.954+0000 7f1c7fe00640 1 -- 172.21.15.39:0/2871705591 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c80003b70 con 0x7f1c88109930 2024-03-18T16:16:46.956 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.955+0000 7f1c8e200640 1 -- 172.21.15.39:0/2871705591 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c881ae860 con 0x7f1c88109930 2024-03-18T16:16:46.956 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.955+0000 7f1c7fe00640 1 -- 172.21.15.39:0/2871705591 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c80003d10 con 0x7f1c88109930 2024-03-18T16:16:46.956 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.955+0000 7f1c7fe00640 1 -- 172.21.15.39:0/2871705591 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c800109f0 con 0x7f1c88109930 2024-03-18T16:16:46.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.955+0000 7f1c8e200640 1 -- 172.21.15.39:0/2871705591 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c881aedb0 con 0x7f1c88109930 2024-03-18T16:16:46.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.960+0000 7f1c90fea740 1 Processor -- start 2024-03-18T16:16:46.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.960+0000 7f1c7fe00640 1 -- 172.21.15.39:0/2871705591 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f1c80004020 con 0x7f1c88109930 2024-03-18T16:16:46.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.961+0000 7f1c90fea740 1 -- start start 2024-03-18T16:16:46.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.961+0000 7f1c90fea740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d04277c8c0 0x55d04277cce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:46.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.961+0000 7f1c90fea740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d04277d2b0 0x55d042781db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:46.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.961+0000 7f1c90fea740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d042782420 0x55d0427848b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:46.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.961+0000 7f1c90fea740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55d0427443b0 con 0x55d042782420 2024-03-18T16:16:46.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.961+0000 7f1c90fea740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55d0426e1450 con 0x55d04277d2b0 2024-03-18T16:16:46.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.961+0000 7f1c90fea740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55d042689cb0 con 0x55d04277c8c0 2024-03-18T16:16:46.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.962+0000 7f1c87400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d04277d2b0 0x55d042781db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:46.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.962+0000 7f1c87400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d04277d2b0 0x55d042781db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:58454/0 (socket says 172.21.15.39:58454) 2024-03-18T16:16:46.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.962+0000 7f1c87400640 1 -- 172.21.15.39:0/2491459953 learned_addr learned my addr 172.21.15.39:0/2491459953 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:16:46.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.962+0000 7f1c87400640 1 -- 172.21.15.39:0/2491459953 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d04277c8c0 msgr2=0x55d04277cce0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T16:16:46.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.962+0000 7f1c87400640 1 --2- 172.21.15.39:0/2491459953 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d04277c8c0 0x55d04277cce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:46.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.962+0000 7f1c87400640 1 -- 172.21.15.39:0/2491459953 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d042782420 msgr2=0x55d0427848b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:16:46.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.962+0000 7f1c86a00640 1 --2- 172.21.15.39:0/2491459953 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d042782420 0x55d0427848b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:46.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.962+0000 7f1c7fe00640 1 --2- 172.21.15.39:0/2871705591 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1c6c0a2800 0x7f1c6c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:46.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.962+0000 7f1c87400640 1 --2- 172.21.15.39:0/2491459953 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d042782420 0x55d0427848b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:46.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.962+0000 7f1c87400640 1 -- 172.21.15.39:0/2491459953 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d0426ab1f0 con 0x55d04277d2b0 2024-03-18T16:16:46.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.963+0000 7f1c7fe00640 1 -- 172.21.15.39:0/2871705591 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f1c800cf260 con 0x7f1c88109930 2024-03-18T16:16:46.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.963+0000 7f1c87400640 1 --2- 172.21.15.39:0/2491459953 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d04277d2b0 0x55d042781db0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f1c7800bd50 tx=0x7f1c780107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=22d5676545993939 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:46.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.963+0000 7f1c73e00640 1 -- 172.21.15.39:0/2491459953 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c78005220 con 0x55d04277d2b0 2024-03-18T16:16:46.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.963+0000 7f1c86a00640 1 --2- 172.21.15.39:0/2871705591 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1c6c0a2800 0x7f1c6c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:46.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.964+0000 7f1c73e00640 1 -- 172.21.15.39:0/2491459953 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c7800be40 con 0x55d04277d2b0 2024-03-18T16:16:46.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.964+0000 7f1c73e00640 1 -- 172.21.15.39:0/2491459953 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c78011860 con 0x55d04277d2b0 2024-03-18T16:16:46.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.965+0000 7f1c90fea740 1 -- 172.21.15.39:0/2491459953 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d04277d2b0 msgr2=0x55d042781db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:46.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.965+0000 7f1c90fea740 1 --2- 172.21.15.39:0/2491459953 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d04277d2b0 0x55d042781db0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f1c7800bd50 tx=0x7f1c780107a0 comp rx=0 tx=0).stop 2024-03-18T16:16:46.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.966+0000 7f1c90fea740 1 -- 172.21.15.39:0/2491459953 shutdown_connections 2024-03-18T16:16:46.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.966+0000 7f1c90fea740 1 --2- 172.21.15.39:0/2491459953 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d04277d2b0 0x55d042781db0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:46.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.966+0000 7f1c90fea740 1 --2- 172.21.15.39:0/2491459953 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d042782420 0x55d0427848b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:46.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.966+0000 7f1c90fea740 1 --2- 172.21.15.39:0/2491459953 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d04277c8c0 0x55d04277cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:46.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.966+0000 7f1c90fea740 1 -- 172.21.15.39:0/2491459953 >> 172.21.15.39:0/2491459953 conn(0x55d042777e30 msgr2=0x55d042778230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:46.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.966+0000 7f1c86a00640 1 --2- 172.21.15.39:0/2871705591 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1c6c0a2800 0x7f1c6c0a4cc0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f1c74009230 tx=0x7f1c7402e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:46.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.966+0000 7f1c90fea740 1 -- 172.21.15.39:0/2491459953 shutdown_connections 2024-03-18T16:16:46.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.966+0000 7f1c90fea740 1 -- 172.21.15.39:0/2491459953 wait complete. 2024-03-18T16:16:46.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.967+0000 7f1c90fea740 1 -- start start 2024-03-18T16:16:46.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.967+0000 7f1c90fea740 1 Processor -- start 2024-03-18T16:16:46.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.968+0000 7f1c90fea740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d04277c8c0 0x55d042786b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:46.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.968+0000 7f1c90fea740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d04277d2b0 0x55d0427870d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:46.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.968+0000 7f1c90fea740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d042782420 0x55d04278e110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:46.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.969+0000 7f1c90fea740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55d04269d110 con 0x55d042782420 2024-03-18T16:16:46.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.969+0000 7f1c87400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d04277c8c0 0x55d042786b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:46.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.969+0000 7f1c87400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d04277c8c0 0x55d042786b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:58466/0 (socket says 172.21.15.39:58466) 2024-03-18T16:16:46.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.969+0000 7f1c90fea740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55d0426dd4f0 con 0x55d04277c8c0 2024-03-18T16:16:46.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.969+0000 7f1c90fea740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55d042785420 con 0x55d04277d2b0 2024-03-18T16:16:46.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.969+0000 7f1c86a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d04277d2b0 0x55d0427870d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:46.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.969+0000 7f1c90fea740 10 client.0 _ll_register_callbacks cb 0x55d04268ab70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:16:46.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.969+0000 7f1c87e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d042782420 0x55d04278e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:46.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.969+0000 7f1c87e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d042782420 0x55d04278e110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:46766/0 (socket says 172.21.15.39:46766) 2024-03-18T16:16:46.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.969+0000 7f1c87e00640 1 -- 172.21.15.39:0/2552300441 learned_addr learned my addr 172.21.15.39:0/2552300441 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:16:46.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.970+0000 7f1c87e00640 1 -- 172.21.15.39:0/2552300441 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d04277d2b0 msgr2=0x55d0427870d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:46.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.970+0000 7f1c87e00640 1 --2- 172.21.15.39:0/2552300441 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d04277d2b0 0x55d0427870d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:46.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.970+0000 7f1c87e00640 1 -- 172.21.15.39:0/2552300441 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d04277c8c0 msgr2=0x55d042786b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:46.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.970+0000 7f1c87e00640 1 --2- 172.21.15.39:0/2552300441 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d04277c8c0 0x55d042786b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:46.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.970+0000 7f1c87e00640 1 -- 172.21.15.39:0/2552300441 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d0423f53f0 con 0x55d042782420 2024-03-18T16:16:46.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.971+0000 7f1c70c00640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-03-18T16:16:46.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.971+0000 7f1c87e00640 1 --2- 172.21.15.39:0/2552300441 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d042782420 0x55d04278e110 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f1c8009b800 tx=0x7f1c8009b470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f567b14effd0d85 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:46.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.971+0000 7f1c70c00640 1 -- 172.21.15.39:0/2552300441 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c800b4060 con 0x55d042782420 2024-03-18T16:16:46.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.971+0000 7f1c70c00640 1 -- 172.21.15.39:0/2552300441 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f1c80004620 con 0x55d042782420 2024-03-18T16:16:46.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.971+0000 7f1c90fea740 1 -- 172.21.15.39:0/2552300441 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55d0426ee990 con 0x55d042782420 2024-03-18T16:16:46.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.971+0000 7f1c70c00640 1 -- 172.21.15.39:0/2552300441 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f1c8009b870 con 0x55d042782420 2024-03-18T16:16:46.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.972+0000 7f1c90fea740 10 client.16464 fetch_fsmap learned FSMap version 44 2024-03-18T16:16:46.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.972+0000 7f1c90fea740 1 -- 172.21.15.39:0/2552300441 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x55d0424bf7d0 con 0x55d042782420 2024-03-18T16:16:46.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.972+0000 7f1c70c00640 1 -- 172.21.15.39:0/2552300441 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55d0426ee990 con 0x55d042782420 2024-03-18T16:16:46.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.972+0000 7f1c70c00640 1 -- 172.21.15.39:0/2552300441 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7f1c8001f020 con 0x55d042782420 2024-03-18T16:16:46.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.972+0000 7f1c90fea740 10 client.16464 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:16:46.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.972+0000 7f1c90fea740 10 client.16464 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-18T16:16:46.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.972+0000 7f1c90fea740 20 client.16464 populate_metadata read hostname 'smithi039' 2024-03-18T16:16:46.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.972+0000 7f1c90fea740 1 --2- 172.21.15.39:0/2552300441 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x55d042792a00 0x55d042794e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:46.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.972+0000 7f1c90fea740 4 client.16464 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:16:46.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.972+0000 7f1c90fea740 1 -- 172.21.15.39:0/2552300441 --> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d0426e8ce0 con 0x55d042792a00 2024-03-18T16:16:46.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.973+0000 7f1c87400640 1 --2- 172.21.15.39:0/2552300441 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x55d042792a00 0x55d042794e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:46.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.974+0000 7f1c87400640 1 --2- 172.21.15.39:0/2552300441 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x55d042792a00 0x55d042794e70 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=57b4ba6455a0af1d server_cookie=89f413b146a39d36 in_seq=0 out_seq=0 2024-03-18T16:16:46.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.974+0000 7f1c70c00640 10 client.16464 ms_handle_connect on v2:172.21.15.194:6836/1301464246 2024-03-18T16:16:46.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.974+0000 7f1c70c00640 1 -- 172.21.15.39:0/2552300441 <== mds.3 v2:172.21.15.194:6836/1301464246 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d0426e8ce0 con 0x55d042792a00 2024-03-18T16:16:46.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.975+0000 7f1c70c00640 10 client.16464 handle_command_reply: tid=0 2024-03-18T16:16:46.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.975+0000 7f1c90fea740 1 client.16464 shutdown 2024-03-18T16:16:46.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.976+0000 7f1c90fea740 1 -- 172.21.15.39:0/2552300441 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d042782420 msgr2=0x55d04278e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:46.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.976+0000 7f1c90fea740 1 --2- 172.21.15.39:0/2552300441 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d042782420 0x55d04278e110 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f1c8009b800 tx=0x7f1c8009b470 comp rx=0 tx=0).stop 2024-03-18T16:16:46.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.976+0000 7f1c90fea740 1 -- 172.21.15.39:0/2552300441 shutdown_connections 2024-03-18T16:16:46.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.976+0000 7f1c90fea740 1 --2- 172.21.15.39:0/2552300441 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x55d042792a00 0x55d042794e70 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:46.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.976+0000 7f1c90fea740 1 --2- 172.21.15.39:0/2552300441 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d042782420 0x55d04278e110 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:46.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.976+0000 7f1c90fea740 1 --2- 172.21.15.39:0/2552300441 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d04277d2b0 0x55d0427870d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:46.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.976+0000 7f1c90fea740 1 --2- 172.21.15.39:0/2552300441 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d04277c8c0 0x55d042786b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:46.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.976+0000 7f1c70c00640 0 client.16464 ms_handle_reset on v2:172.21.15.194:6836/1301464246 2024-03-18T16:16:46.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.976+0000 7f1c90fea740 1 -- 172.21.15.39:0/2552300441 >> 172.21.15.39:0/2552300441 conn(0x55d042777e30 msgr2=0x55d0427771f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:46.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.977+0000 7f1c90fea740 1 -- 172.21.15.39:0/2552300441 shutdown_connections 2024-03-18T16:16:46.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.977+0000 7f1c90fea740 1 -- 172.21.15.39:0/2552300441 wait complete. 2024-03-18T16:16:46.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:46.977+0000 7f1c90fea740 20 client.16464 trim_cache size 0 max 16384 2024-03-18T16:16:47.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.005+0000 7f1c90fea740 1 Processor -- start 2024-03-18T16:16:47.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.005+0000 7f1c90fea740 1 -- start start 2024-03-18T16:16:47.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.005+0000 7f1c90fea740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d04277c8c0 0x55d042776960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:47.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.005+0000 7f1c90fea740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d04277d2b0 0x55d04278e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:47.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.005+0000 7f1c90fea740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d042782420 0x55d042776ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:47.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.006+0000 7f1c90fea740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f1c881b93d0 con 0x55d04277d2b0 2024-03-18T16:16:47.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.006+0000 7f1c90fea740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f1c881baf10 con 0x55d04277c8c0 2024-03-18T16:16:47.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.006+0000 7f1c90fea740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f1c881b9aa0 con 0x55d042782420 2024-03-18T16:16:47.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.006+0000 7f1c87e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d042782420 0x55d042776ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:47.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.006+0000 7f1c87e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d042782420 0x55d042776ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:54678/0 (socket says 172.21.15.39:54678) 2024-03-18T16:16:47.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.006+0000 7f1c86a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d04277d2b0 0x55d04278e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:47.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.006+0000 7f1c87e00640 1 -- 172.21.15.39:0/1704059258 learned_addr learned my addr 172.21.15.39:0/1704059258 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:16:47.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.006+0000 7f1c87400640 1 --2- 172.21.15.39:0/1704059258 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d04277c8c0 0x55d042776960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:47.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.007+0000 7f1c87e00640 1 -- 172.21.15.39:0/1704059258 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d04277c8c0 msgr2=0x55d042776960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:47.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.007+0000 7f1c87e00640 1 --2- 172.21.15.39:0/1704059258 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d04277c8c0 0x55d042776960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:47.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.007+0000 7f1c87e00640 1 -- 172.21.15.39:0/1704059258 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d04277d2b0 msgr2=0x55d04278e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:47.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.007+0000 7f1c87e00640 1 --2- 172.21.15.39:0/1704059258 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d04277d2b0 0x55d04278e0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:47.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.007+0000 7f1c87e00640 1 -- 172.21.15.39:0/1704059258 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d042773c10 con 0x55d042782420 2024-03-18T16:16:47.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.007+0000 7f1c87e00640 1 --2- 172.21.15.39:0/1704059258 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d042782420 0x55d042776ea0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f1c80005cc0 tx=0x7f1c80099d80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ad500e9f10a255ed server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:47.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.008+0000 7f1c73400640 1 -- 172.21.15.39:0/1704059258 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c8009f9c0 con 0x55d042782420 2024-03-18T16:16:47.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.008+0000 7f1c73400640 1 -- 172.21.15.39:0/1704059258 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c80004620 con 0x55d042782420 2024-03-18T16:16:47.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.008+0000 7f1c73400640 1 -- 172.21.15.39:0/1704059258 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c8009ce00 con 0x55d042782420 2024-03-18T16:16:47.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.009+0000 7f1c90fea740 1 -- 172.21.15.39:0/1704059258 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d042782420 msgr2=0x55d042776ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:47.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.009+0000 7f1c90fea740 1 --2- 172.21.15.39:0/1704059258 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d042782420 0x55d042776ea0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f1c80005cc0 tx=0x7f1c80099d80 comp rx=0 tx=0).stop 2024-03-18T16:16:47.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.009+0000 7f1c90fea740 1 -- 172.21.15.39:0/1704059258 shutdown_connections 2024-03-18T16:16:47.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.009+0000 7f1c90fea740 1 --2- 172.21.15.39:0/1704059258 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d042782420 0x55d042776ea0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:47.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.009+0000 7f1c90fea740 1 --2- 172.21.15.39:0/1704059258 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d04277d2b0 0x55d04278e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:47.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.009+0000 7f1c90fea740 1 --2- 172.21.15.39:0/1704059258 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d04277c8c0 0x55d042776960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:47.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.009+0000 7f1c90fea740 1 -- 172.21.15.39:0/1704059258 >> 172.21.15.39:0/1704059258 conn(0x55d042777e30 msgr2=0x55d042773080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:47.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.009+0000 7f1c90fea740 1 -- 172.21.15.39:0/1704059258 shutdown_connections 2024-03-18T16:16:47.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.009+0000 7f1c90fea740 1 -- 172.21.15.39:0/1704059258 wait complete. 2024-03-18T16:16:47.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.009+0000 7f1c90fea740 1 -- start start 2024-03-18T16:16:47.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.010+0000 7f1c90fea740 1 Processor -- start 2024-03-18T16:16:47.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.011+0000 7f1c90fea740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d04277c8c0 0x55d04278bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:47.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.011+0000 7f1c90fea740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d04277d2b0 0x55d04278c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:47.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.011+0000 7f1c90fea740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d042782420 0x55d04278d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:47.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.011+0000 7f1c90fea740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f1c881ba840 con 0x55d042782420 2024-03-18T16:16:47.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.011+0000 7f1c90fea740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f1c881ba170 con 0x55d04277d2b0 2024-03-18T16:16:47.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.011+0000 7f1c90fea740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f1c881b73b0 con 0x55d04277c8c0 2024-03-18T16:16:47.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.011+0000 7f1c90fea740 10 client.0 _ll_register_callbacks cb 0x7f1c80003ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:16:47.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.011+0000 7f1c86a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d04277d2b0 0x55d04278c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:47.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.011+0000 7f1c86a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d04277d2b0 0x55d04278c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:58472/0 (socket says 172.21.15.39:58472) 2024-03-18T16:16:47.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.011+0000 7f1c86a00640 1 -- 172.21.15.39:0/363587468 learned_addr learned my addr 172.21.15.39:0/363587468 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:16:47.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.011+0000 7f1c87400640 1 --2- 172.21.15.39:0/363587468 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d04277c8c0 0x55d04278bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:47.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.012+0000 7f1c87e00640 1 --2- 172.21.15.39:0/363587468 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d042782420 0x55d04278d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:47.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.012+0000 7f1c86a00640 1 -- 172.21.15.39:0/363587468 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d04277c8c0 msgr2=0x55d04278bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:47.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.012+0000 7f1c86a00640 1 --2- 172.21.15.39:0/363587468 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d04277c8c0 0x55d04278bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:47.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.012+0000 7f1c86a00640 1 -- 172.21.15.39:0/363587468 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d042782420 msgr2=0x55d04278d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:47.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.012+0000 7f1c86a00640 1 --2- 172.21.15.39:0/363587468 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d042782420 0x55d04278d2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:47.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.012+0000 7f1c86a00640 1 -- 172.21.15.39:0/363587468 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d042785560 con 0x55d04277d2b0 2024-03-18T16:16:47.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.012+0000 7f1c87400640 1 --2- 172.21.15.39:0/363587468 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d04277c8c0 0x55d04278bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:16:47.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.012+0000 7f1c86a00640 1 --2- 172.21.15.39:0/363587468 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d04277d2b0 0x55d04278c2e0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f1c74036040 tx=0x7f1c7402cf90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e2c859f293f4d25b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:47.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.013+0000 7f1c70c00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:16:47.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.013+0000 7f1c70c00640 1 -- 172.21.15.39:0/363587468 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c74050070 con 0x55d04277d2b0 2024-03-18T16:16:47.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.013+0000 7f1c70c00640 1 -- 172.21.15.39:0/363587468 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f1c74003b40 con 0x55d04277d2b0 2024-03-18T16:16:47.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.013+0000 7f1c90fea740 1 -- 172.21.15.39:0/363587468 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1c80004620 con 0x55d04277d2b0 2024-03-18T16:16:47.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.014+0000 7f1c70c00640 1 -- 172.21.15.39:0/363587468 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f1c7405c050 con 0x55d04277d2b0 2024-03-18T16:16:47.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.014+0000 7f1c90fea740 10 client.26978 fetch_fsmap learned FSMap version 44 2024-03-18T16:16:47.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.014+0000 7f1c90fea740 1 -- 172.21.15.39:0/363587468 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x55d0427927e0 con 0x55d04277d2b0 2024-03-18T16:16:47.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.015+0000 7f1c70c00640 1 -- 172.21.15.39:0/363587468 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c80004620 con 0x55d04277d2b0 2024-03-18T16:16:47.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.015+0000 7f1c70c00640 1 -- 172.21.15.39:0/363587468 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7f1c74049040 con 0x55d04277d2b0 2024-03-18T16:16:47.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.015+0000 7f1c90fea740 10 client.26978 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:16:47.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.015+0000 7f1c90fea740 10 client.26978 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-18T16:16:47.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.015+0000 7f1c90fea740 20 client.26978 populate_metadata read hostname 'smithi039' 2024-03-18T16:16:47.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.015+0000 7f1c90fea740 1 --2- 172.21.15.39:0/363587468 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x55d0427f9930 0x55d0427fbd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:47.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.015+0000 7f1c90fea740 4 client.26978 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:16:47.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.015+0000 7f1c90fea740 1 -- 172.21.15.39:0/363587468 --> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d042666560 con 0x55d0427f9930 2024-03-18T16:16:47.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.016+0000 7f1c87400640 1 --2- 172.21.15.39:0/363587468 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x55d0427f9930 0x55d0427fbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:47.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.017+0000 7f1c87400640 1 --2- 172.21.15.39:0/363587468 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x55d0427f9930 0x55d0427fbd50 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=538824394fe54167 server_cookie=8913c8d3bf7e9fc9 in_seq=0 out_seq=0 2024-03-18T16:16:47.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.018+0000 7f1c70c00640 10 client.26978 ms_handle_connect on v2:172.21.15.194:6836/1301464246 2024-03-18T16:16:47.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.018+0000 7f1c70c00640 1 -- 172.21.15.39:0/363587468 <== mds.3 v2:172.21.15.194:6836/1301464246 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d042666560 con 0x55d0427f9930 2024-03-18T16:16:47.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.018+0000 7f1c70c00640 10 client.26978 handle_command_reply: tid=0 2024-03-18T16:16:47.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.018+0000 7f1c90fea740 1 client.26978 shutdown 2024-03-18T16:16:47.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.019+0000 7f1c90fea740 1 -- 172.21.15.39:0/363587468 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d04277d2b0 msgr2=0x55d04278c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:47.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.019+0000 7f1c90fea740 1 --2- 172.21.15.39:0/363587468 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d04277d2b0 0x55d04278c2e0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f1c74036040 tx=0x7f1c7402cf90 comp rx=0 tx=0).stop 2024-03-18T16:16:47.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.019+0000 7f1c90fea740 1 -- 172.21.15.39:0/363587468 shutdown_connections 2024-03-18T16:16:47.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.019+0000 7f1c90fea740 1 --2- 172.21.15.39:0/363587468 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x55d0427f9930 0x55d0427fbd50 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:47.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.019+0000 7f1c90fea740 1 --2- 172.21.15.39:0/363587468 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d04277d2b0 0x55d04278c2e0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:47.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.019+0000 7f1c90fea740 1 --2- 172.21.15.39:0/363587468 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d042782420 0x55d04278d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:47.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.019+0000 7f1c90fea740 1 --2- 172.21.15.39:0/363587468 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d04277c8c0 0x55d04278bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:47.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.019+0000 7f1c90fea740 1 -- 172.21.15.39:0/363587468 >> 172.21.15.39:0/363587468 conn(0x55d042777e30 msgr2=0x55d04277b090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:47.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.019+0000 7f1c70c00640 0 client.26978 ms_handle_reset on v2:172.21.15.194:6836/1301464246 2024-03-18T16:16:47.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.023+0000 7f1c90fea740 1 -- 172.21.15.39:0/363587468 shutdown_connections 2024-03-18T16:16:47.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.023+0000 7f1c90fea740 1 -- 172.21.15.39:0/363587468 wait complete. 2024-03-18T16:16:47.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.023+0000 7f1c90fea740 20 client.26978 trim_cache size 0 max 16384 2024-03-18T16:16:47.025 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:16:47.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.024+0000 7f1c8e200640 1 -- 172.21.15.39:0/2871705591 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1c6c0a2800 msgr2=0x7f1c6c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:47.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.024+0000 7f1c8e200640 1 --2- 172.21.15.39:0/2871705591 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1c6c0a2800 0x7f1c6c0a4cc0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f1c74009230 tx=0x7f1c7402e040 comp rx=0 tx=0).stop 2024-03-18T16:16:47.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.025+0000 7f1c8e200640 1 -- 172.21.15.39:0/2871705591 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1c88109930 msgr2=0x7f1c881ae200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:47.026 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.025+0000 7f1c8e200640 1 --2- 172.21.15.39:0/2871705591 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1c88109930 0x7f1c881ae200 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f1c8000efd0 tx=0x7f1c8000c6a0 comp rx=0 tx=0).stop 2024-03-18T16:16:47.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.025+0000 7f1c8e200640 1 -- 172.21.15.39:0/2871705591 shutdown_connections 2024-03-18T16:16:47.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.025+0000 7f1c8e200640 1 --2- 172.21.15.39:0/2871705591 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1c88109930 0x7f1c881ae200 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:47.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.025+0000 7f1c8e200640 1 --2- 172.21.15.39:0/2871705591 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1c6c0a2800 0x7f1c6c0a4cc0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:47.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.025+0000 7f1c8e200640 1 --2- 172.21.15.39:0/2871705591 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1c88108f70 0x7f1c8807f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:47.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.025+0000 7f1c8e200640 1 --2- 172.21.15.39:0/2871705591 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1c88107e20 0x7f1c8807efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:47.027 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.025+0000 7f1c8e200640 1 -- 172.21.15.39:0/2871705591 >> 172.21.15.39:0/2871705591 conn(0x7f1c88076dd0 msgr2=0x7f1c8810c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:47.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.026+0000 7f1c8e200640 1 -- 172.21.15.39:0/2871705591 shutdown_connections 2024-03-18T16:16:47.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.026+0000 7f1c8e200640 1 -- 172.21.15.39:0/2871705591 wait complete. 2024-03-18T16:16:47.052 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-18T16:16:47.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.333+0000 7f5d41e00640 1 -- 172.21.15.39:0/4291908654 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5d3c10b150 msgr2=0x7f5d3c10b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:47.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.333+0000 7f5d41e00640 1 --2- 172.21.15.39:0/4291908654 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5d3c10b150 0x7f5d3c10b5b0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f5d24009900 tx=0x7f5d2402bc50 comp rx=0 tx=0).stop 2024-03-18T16:16:47.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.333+0000 7f5d41e00640 1 -- 172.21.15.39:0/4291908654 shutdown_connections 2024-03-18T16:16:47.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.333+0000 7f5d41e00640 1 --2- 172.21.15.39:0/4291908654 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5d3c10b150 0x7f5d3c10b5b0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:47.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.333+0000 7f5d41e00640 1 --2- 172.21.15.39:0/4291908654 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5d3c111f20 0x7f5d3c114310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:47.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.333+0000 7f5d41e00640 1 --2- 172.21.15.39:0/4291908654 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5d3c1087e0 0x7f5d3c10ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:47.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.333+0000 7f5d41e00640 1 -- 172.21.15.39:0/4291908654 >> 172.21.15.39:0/4291908654 conn(0x7f5d3c06e860 msgr2=0x7f5d3c070ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:47.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.333+0000 7f5d41e00640 1 -- 172.21.15.39:0/4291908654 shutdown_connections 2024-03-18T16:16:47.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.334+0000 7f5d41e00640 1 -- 172.21.15.39:0/4291908654 wait complete. 2024-03-18T16:16:47.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.335+0000 7f5d41e00640 1 Processor -- start 2024-03-18T16:16:47.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.336+0000 7f5d41e00640 1 -- start start 2024-03-18T16:16:47.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.337+0000 7f5d41e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5d3c1087e0 0x7f5d3c1a6880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:47.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.337+0000 7f5d41e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5d3c10b150 0x7f5d3c1a6dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:47.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.337+0000 7f5d41e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5d3c111f20 0x7f5d3c1ade30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:47.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.337+0000 7f5d41e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f5d3c10be90 con 0x7f5d3c111f20 2024-03-18T16:16:47.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.337+0000 7f5d41e00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f5d3c10bd10 con 0x7f5d3c1087e0 2024-03-18T16:16:47.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.337+0000 7f5d41e00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f5d3c10c010 con 0x7f5d3c10b150 2024-03-18T16:16:47.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.337+0000 7f5d33e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5d3c10b150 0x7f5d3c1a6dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:47.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.337+0000 7f5d33e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5d3c10b150 0x7f5d3c1a6dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:54702/0 (socket says 172.21.15.39:54702) 2024-03-18T16:16:47.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.337+0000 7f5d33e00640 1 -- 172.21.15.39:0/2823726458 learned_addr learned my addr 172.21.15.39:0/2823726458 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:16:47.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.337+0000 7f5d3aa00640 1 --2- 172.21.15.39:0/2823726458 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5d3c1087e0 0x7f5d3c1a6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:47.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.337+0000 7f5d3b400640 1 --2- 172.21.15.39:0/2823726458 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5d3c111f20 0x7f5d3c1ade30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:47.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.338+0000 7f5d33e00640 1 -- 172.21.15.39:0/2823726458 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5d3c1087e0 msgr2=0x7f5d3c1a6880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:47.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.338+0000 7f5d33e00640 1 --2- 172.21.15.39:0/2823726458 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5d3c1087e0 0x7f5d3c1a6880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:47.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.338+0000 7f5d33e00640 1 -- 172.21.15.39:0/2823726458 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5d3c111f20 msgr2=0x7f5d3c1ade30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:47.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.338+0000 7f5d33e00640 1 --2- 172.21.15.39:0/2823726458 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5d3c111f20 0x7f5d3c1ade30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:47.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.338+0000 7f5d33e00640 1 -- 172.21.15.39:0/2823726458 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d3c1ae4a0 con 0x7f5d3c10b150 2024-03-18T16:16:47.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.339+0000 7f5d33e00640 1 --2- 172.21.15.39:0/2823726458 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5d3c10b150 0x7f5d3c1a6dc0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f5d24005ab0 tx=0x7f5d2402c6e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:47.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.339+0000 7f5d40a00640 1 -- 172.21.15.39:0/2823726458 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d24041070 con 0x7f5d3c10b150 2024-03-18T16:16:47.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.339+0000 7f5d40a00640 1 -- 172.21.15.39:0/2823726458 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d24004450 con 0x7f5d3c10b150 2024-03-18T16:16:47.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.339+0000 7f5d41e00640 1 -- 172.21.15.39:0/2823726458 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d3c1ae730 con 0x7f5d3c10b150 2024-03-18T16:16:47.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.339+0000 7f5d40a00640 1 -- 172.21.15.39:0/2823726458 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d24034440 con 0x7f5d3c10b150 2024-03-18T16:16:47.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.340+0000 7f5d41e00640 1 -- 172.21.15.39:0/2823726458 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d3c1aecd0 con 0x7f5d3c10b150 2024-03-18T16:16:47.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.345+0000 7f5d40a00640 1 -- 172.21.15.39:0/2823726458 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f5d24044070 con 0x7f5d3c10b150 2024-03-18T16:16:47.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.346+0000 7f5d44bee740 1 Processor -- start 2024-03-18T16:16:47.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.346+0000 7f5d44bee740 1 -- start start 2024-03-18T16:16:47.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.347+0000 7f5d44bee740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bb71fa88c0 0x55bb71fa8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:47.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.347+0000 7f5d44bee740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bb71fa92b0 0x55bb71faddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:47.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.347+0000 7f5d44bee740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bb71fae420 0x55bb71fb08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:47.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.347+0000 7f5d44bee740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55bb71f703b0 con 0x55bb71fa92b0 2024-03-18T16:16:47.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.347+0000 7f5d44bee740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55bb71f0d450 con 0x55bb71fae420 2024-03-18T16:16:47.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.347+0000 7f5d44bee740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55bb71eb5cb0 con 0x55bb71fa88c0 2024-03-18T16:16:47.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.347+0000 7f5d3b400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bb71fa88c0 0x55bb71fa8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:47.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.347+0000 7f5d3b400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bb71fa88c0 0x55bb71fa8ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:54710/0 (socket says 172.21.15.39:54710) 2024-03-18T16:16:47.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.347+0000 7f5d3b400640 1 -- 172.21.15.39:0/3471326720 learned_addr learned my addr 172.21.15.39:0/3471326720 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:16:47.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.348+0000 7f5d33e00640 1 --2- 172.21.15.39:0/3471326720 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bb71fae420 0x55bb71fb08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:47.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.348+0000 7f5d3aa00640 1 --2- 172.21.15.39:0/3471326720 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bb71fa92b0 0x55bb71faddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:47.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.348+0000 7f5d3b400640 1 -- 172.21.15.39:0/3471326720 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bb71fae420 msgr2=0x55bb71fb08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:47.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.348+0000 7f5d3b400640 1 --2- 172.21.15.39:0/3471326720 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bb71fae420 0x55bb71fb08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:47.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.348+0000 7f5d3b400640 1 -- 172.21.15.39:0/3471326720 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bb71fa92b0 msgr2=0x55bb71faddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:47.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.348+0000 7f5d3b400640 1 --2- 172.21.15.39:0/3471326720 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bb71fa92b0 0x55bb71faddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:47.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.348+0000 7f5d3b400640 1 -- 172.21.15.39:0/3471326720 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb71ed71f0 con 0x55bb71fa88c0 2024-03-18T16:16:47.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.349+0000 7f5d40a00640 1 --2- 172.21.15.39:0/2823726458 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f5d200a2900 0x7f5d200a4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:47.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.349+0000 7f5d3b400640 1 --2- 172.21.15.39:0/3471326720 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bb71fa88c0 0x55bb71fa8ce0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f5d34009fc0 tx=0x7f5d34010860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7646b1315d1d4e60 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:47.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.349+0000 7f5d40a00640 1 -- 172.21.15.39:0/2823726458 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f5d240f0a60 con 0x7f5d3c10b150 2024-03-18T16:16:47.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.349+0000 7f5d2be00640 1 -- 172.21.15.39:0/3471326720 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d34003b70 con 0x55bb71fa88c0 2024-03-18T16:16:47.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.350+0000 7f5d2be00640 1 -- 172.21.15.39:0/3471326720 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d34003d10 con 0x55bb71fa88c0 2024-03-18T16:16:47.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.350+0000 7f5d2be00640 1 -- 172.21.15.39:0/3471326720 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d34003b70 con 0x55bb71fa88c0 2024-03-18T16:16:47.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.351+0000 7f5d3aa00640 1 --2- 172.21.15.39:0/2823726458 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f5d200a2900 0x7f5d200a4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:47.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.351+0000 7f5d44bee740 1 -- 172.21.15.39:0/3471326720 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bb71fa88c0 msgr2=0x55bb71fa8ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:47.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.351+0000 7f5d44bee740 1 --2- 172.21.15.39:0/3471326720 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bb71fa88c0 0x55bb71fa8ce0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f5d34009fc0 tx=0x7f5d34010860 comp rx=0 tx=0).stop 2024-03-18T16:16:47.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.351+0000 7f5d44bee740 1 -- 172.21.15.39:0/3471326720 shutdown_connections 2024-03-18T16:16:47.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.351+0000 7f5d44bee740 1 --2- 172.21.15.39:0/3471326720 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bb71fae420 0x55bb71fb08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:47.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.351+0000 7f5d44bee740 1 --2- 172.21.15.39:0/3471326720 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bb71fa92b0 0x55bb71faddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:47.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.351+0000 7f5d44bee740 1 --2- 172.21.15.39:0/3471326720 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bb71fa88c0 0x55bb71fa8ce0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:47.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.351+0000 7f5d44bee740 1 -- 172.21.15.39:0/3471326720 >> 172.21.15.39:0/3471326720 conn(0x55bb71fa3e30 msgr2=0x55bb71fa4230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:47.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.352+0000 7f5d44bee740 1 -- 172.21.15.39:0/3471326720 shutdown_connections 2024-03-18T16:16:47.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.352+0000 7f5d44bee740 1 -- 172.21.15.39:0/3471326720 wait complete. 2024-03-18T16:16:47.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.352+0000 7f5d3aa00640 1 --2- 172.21.15.39:0/2823726458 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f5d200a2900 0x7f5d200a4dc0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x55bb71fa4da0 tx=0x7f5d2c00a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:47.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.352+0000 7f5d44bee740 1 -- start start 2024-03-18T16:16:47.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.353+0000 7f5d44bee740 1 Processor -- start 2024-03-18T16:16:47.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.353+0000 7f5d44bee740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bb71fa88c0 0x55bb71fb29b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:47.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.353+0000 7f5d44bee740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bb71fa92b0 0x55bb71fb2ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:47.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.354+0000 7f5d44bee740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bb71fae420 0x55bb71fb9f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:47.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.354+0000 7f5d3aa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bb71fa88c0 0x55bb71fb29b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:47.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.354+0000 7f5d33e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bb71fa92b0 0x55bb71fb2ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:47.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.354+0000 7f5d44bee740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55bb71ec9110 con 0x55bb71fa88c0 2024-03-18T16:16:47.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.354+0000 7f5d44bee740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55bb71f094f0 con 0x55bb71fae420 2024-03-18T16:16:47.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.354+0000 7f5d3aa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bb71fa88c0 0x55bb71fb29b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:46816/0 (socket says 172.21.15.39:46816) 2024-03-18T16:16:47.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.354+0000 7f5d44bee740 1 -- 172.21.15.39:0/2954396473 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55bb71fb13d0 con 0x55bb71fa92b0 2024-03-18T16:16:47.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.354+0000 7f5d3aa00640 1 -- 172.21.15.39:0/2954396473 learned_addr learned my addr 172.21.15.39:0/2954396473 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:16:47.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.354+0000 7f5d44bee740 10 client.0 _ll_register_callbacks cb 0x55bb71eb6b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:16:47.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.355+0000 7f5d3b400640 1 --2- 172.21.15.39:0/2954396473 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bb71fae420 0x55bb71fb9f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:47.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.355+0000 7f5d3aa00640 1 -- 172.21.15.39:0/2954396473 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bb71fa92b0 msgr2=0x55bb71fb2ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:47.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.355+0000 7f5d3aa00640 1 --2- 172.21.15.39:0/2954396473 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bb71fa92b0 0x55bb71fb2ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:47.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.355+0000 7f5d3aa00640 1 -- 172.21.15.39:0/2954396473 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bb71fae420 msgr2=0x55bb71fb9f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:47.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.355+0000 7f5d3aa00640 1 --2- 172.21.15.39:0/2954396473 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bb71fae420 0x55bb71fb9f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:47.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.355+0000 7f5d3aa00640 1 -- 172.21.15.39:0/2954396473 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb71c213f0 con 0x55bb71fa88c0 2024-03-18T16:16:47.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.355+0000 7f5d3aa00640 1 --2- 172.21.15.39:0/2954396473 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bb71fa88c0 0x55bb71fb29b0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f5d2c00a8a0 tx=0x7f5d2c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60f64f6111d340be server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:47.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.355+0000 7f5d28c00640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-03-18T16:16:47.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.356+0000 7f5d28c00640 1 -- 172.21.15.39:0/2954396473 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d2c01db90 con 0x55bb71fa88c0 2024-03-18T16:16:47.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.356+0000 7f5d28c00640 1 -- 172.21.15.39:0/2954396473 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f5d2c01dd30 con 0x55bb71fa88c0 2024-03-18T16:16:47.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.356+0000 7f5d44bee740 1 -- 172.21.15.39:0/2954396473 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5d34003d10 con 0x55bb71fa88c0 2024-03-18T16:16:47.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.356+0000 7f5d28c00640 1 -- 172.21.15.39:0/2954396473 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f5d2c004020 con 0x55bb71fa88c0 2024-03-18T16:16:47.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.357+0000 7f5d44bee740 10 client.16479 fetch_fsmap learned FSMap version 44 2024-03-18T16:16:47.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.357+0000 7f5d44bee740 1 -- 172.21.15.39:0/2954396473 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x55bb71ceb7d0 con 0x55bb71fa88c0 2024-03-18T16:16:47.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.357+0000 7f5d28c00640 1 -- 172.21.15.39:0/2954396473 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d34003d10 con 0x55bb71fa88c0 2024-03-18T16:16:47.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.357+0000 7f5d28c00640 1 -- 172.21.15.39:0/2954396473 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7f5d2c019040 con 0x55bb71fa88c0 2024-03-18T16:16:47.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.357+0000 7f5d44bee740 10 client.16479 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:16:47.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.357+0000 7f5d44bee740 10 client.16479 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-18T16:16:47.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.357+0000 7f5d44bee740 20 client.16479 populate_metadata read hostname 'smithi039' 2024-03-18T16:16:47.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.358+0000 7f5d44bee740 1 --2- 172.21.15.39:0/2954396473 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x55bb71fbe840 0x55bb71fc0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:47.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.358+0000 7f5d44bee740 4 client.16479 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:16:47.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.358+0000 7f5d44bee740 1 -- 172.21.15.39:0/2954396473 --> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bb71f1a990 con 0x55bb71fbe840 2024-03-18T16:16:47.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.358+0000 7f5d33e00640 1 --2- 172.21.15.39:0/2954396473 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x55bb71fbe840 0x55bb71fc0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:47.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.359+0000 7f5d33e00640 1 --2- 172.21.15.39:0/2954396473 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x55bb71fbe840 0x55bb71fc0ce0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e66bb40b38476aab server_cookie=6fe3a7afd69f43e4 in_seq=0 out_seq=0 2024-03-18T16:16:47.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.359+0000 7f5d28c00640 10 client.16479 ms_handle_connect on v2:172.21.15.196:6836/895247251 2024-03-18T16:16:47.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.360+0000 7f5d28c00640 1 -- 172.21.15.39:0/2954396473 <== mds.2 v2:172.21.15.196:6836/895247251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55bb71f1a990 con 0x55bb71fbe840 2024-03-18T16:16:47.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.360+0000 7f5d28c00640 10 client.16479 handle_command_reply: tid=0 2024-03-18T16:16:47.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.360+0000 7f5d44bee740 1 client.16479 shutdown 2024-03-18T16:16:47.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.360+0000 7f5d44bee740 1 -- 172.21.15.39:0/2954396473 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bb71fa88c0 msgr2=0x55bb71fb29b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:47.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.360+0000 7f5d44bee740 1 --2- 172.21.15.39:0/2954396473 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bb71fa88c0 0x55bb71fb29b0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f5d2c00a8a0 tx=0x7f5d2c0046c0 comp rx=0 tx=0).stop 2024-03-18T16:16:47.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.360+0000 7f5d44bee740 1 -- 172.21.15.39:0/2954396473 shutdown_connections 2024-03-18T16:16:47.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.360+0000 7f5d44bee740 1 --2- 172.21.15.39:0/2954396473 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bb71fae420 0x55bb71fb9f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:47.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.360+0000 7f5d44bee740 1 --2- 172.21.15.39:0/2954396473 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x55bb71fbe840 0x55bb71fc0ce0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:47.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.360+0000 7f5d44bee740 1 --2- 172.21.15.39:0/2954396473 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bb71fa92b0 0x55bb71fb2ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:47.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.360+0000 7f5d44bee740 1 --2- 172.21.15.39:0/2954396473 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bb71fa88c0 0x55bb71fb29b0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:47.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.360+0000 7f5d44bee740 1 -- 172.21.15.39:0/2954396473 >> 172.21.15.39:0/2954396473 conn(0x55bb71fa3e30 msgr2=0x55bb71fa31f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:47.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.360+0000 7f5d28c00640 0 client.16479 ms_handle_reset on v2:172.21.15.196:6836/895247251 2024-03-18T16:16:47.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.361+0000 7f5d44bee740 1 -- 172.21.15.39:0/2954396473 shutdown_connections 2024-03-18T16:16:47.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.361+0000 7f5d44bee740 1 -- 172.21.15.39:0/2954396473 wait complete. 2024-03-18T16:16:47.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.361+0000 7f5d44bee740 20 client.16479 trim_cache size 0 max 16384 2024-03-18T16:16:47.386 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.385+0000 7f5d44bee740 1 Processor -- start 2024-03-18T16:16:47.386 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.385+0000 7f5d44bee740 1 -- start start 2024-03-18T16:16:47.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.385+0000 7f5d44bee740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bb71fa88c0 0x55bb71fc65b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:47.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.385+0000 7f5d44bee740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bb71fa92b0 0x55bb71fa2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:47.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.385+0000 7f5d44bee740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bb71fae420 0x55bb71fa2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:47.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.385+0000 7f5d44bee740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f5d3c1c1ad0 con 0x55bb71fa88c0 2024-03-18T16:16:47.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.385+0000 7f5d44bee740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f5d3c1c3610 con 0x55bb71fae420 2024-03-18T16:16:47.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.385+0000 7f5d44bee740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f5d3c1c21a0 con 0x55bb71fa92b0 2024-03-18T16:16:47.388 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.386+0000 7f5d33e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bb71fa92b0 0x55bb71fa2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:47.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.386+0000 7f5d33e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bb71fa92b0 0x55bb71fa2960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:54736/0 (socket says 172.21.15.39:54736) 2024-03-18T16:16:47.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.386+0000 7f5d33e00640 1 -- 172.21.15.39:0/3877863524 learned_addr learned my addr 172.21.15.39:0/3877863524 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:16:47.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.386+0000 7f5d3aa00640 1 --2- 172.21.15.39:0/3877863524 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bb71fa88c0 0x55bb71fc65b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:47.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.386+0000 7f5d3b400640 1 --2- 172.21.15.39:0/3877863524 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bb71fae420 0x55bb71fa2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:47.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.386+0000 7f5d33e00640 1 -- 172.21.15.39:0/3877863524 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bb71fae420 msgr2=0x55bb71fa2ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:47.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.386+0000 7f5d33e00640 1 --2- 172.21.15.39:0/3877863524 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bb71fae420 0x55bb71fa2ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:47.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.386+0000 7f5d33e00640 1 -- 172.21.15.39:0/3877863524 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bb71fa88c0 msgr2=0x55bb71fc65b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:47.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.386+0000 7f5d33e00640 1 --2- 172.21.15.39:0/3877863524 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bb71fa88c0 0x55bb71fc65b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:47.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.386+0000 7f5d33e00640 1 -- 172.21.15.39:0/3877863524 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb71f9fc10 con 0x55bb71fa92b0 2024-03-18T16:16:47.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.386+0000 7f5d3aa00640 1 --2- 172.21.15.39:0/3877863524 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bb71fa88c0 0x55bb71fc65b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:16:47.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.387+0000 7f5d33e00640 1 --2- 172.21.15.39:0/3877863524 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bb71fa92b0 0x55bb71fa2960 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f5d2402c500 tx=0x7f5d24003a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a2606883455bf29d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:47.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.387+0000 7f5d2b400640 1 -- 172.21.15.39:0/3877863524 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d2404b040 con 0x55bb71fa92b0 2024-03-18T16:16:47.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.387+0000 7f5d2b400640 1 -- 172.21.15.39:0/3877863524 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d240c1160 con 0x55bb71fa92b0 2024-03-18T16:16:47.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.387+0000 7f5d2b400640 1 -- 172.21.15.39:0/3877863524 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d24003680 con 0x55bb71fa92b0 2024-03-18T16:16:47.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.388+0000 7f5d44bee740 1 -- 172.21.15.39:0/3877863524 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bb71fa92b0 msgr2=0x55bb71fa2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:47.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.388+0000 7f5d44bee740 1 --2- 172.21.15.39:0/3877863524 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bb71fa92b0 0x55bb71fa2960 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f5d2402c500 tx=0x7f5d24003a20 comp rx=0 tx=0).stop 2024-03-18T16:16:47.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.389+0000 7f5d44bee740 1 -- 172.21.15.39:0/3877863524 shutdown_connections 2024-03-18T16:16:47.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.389+0000 7f5d44bee740 1 --2- 172.21.15.39:0/3877863524 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bb71fae420 0x55bb71fa2ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:47.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.389+0000 7f5d44bee740 1 --2- 172.21.15.39:0/3877863524 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bb71fa92b0 0x55bb71fa2960 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:47.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.389+0000 7f5d44bee740 1 --2- 172.21.15.39:0/3877863524 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bb71fa88c0 0x55bb71fc65b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:47.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.389+0000 7f5d44bee740 1 -- 172.21.15.39:0/3877863524 >> 172.21.15.39:0/3877863524 conn(0x55bb71fa3e30 msgr2=0x55bb71fa7090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:47.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.389+0000 7f5d44bee740 1 -- 172.21.15.39:0/3877863524 shutdown_connections 2024-03-18T16:16:47.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.389+0000 7f5d44bee740 1 -- 172.21.15.39:0/3877863524 wait complete. 2024-03-18T16:16:47.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.390+0000 7f5d44bee740 1 -- start start 2024-03-18T16:16:47.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.390+0000 7f5d44bee740 1 Processor -- start 2024-03-18T16:16:47.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.391+0000 7f5d44bee740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bb71fa88c0 0x55bb71fb6af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:47.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.391+0000 7f5d44bee740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bb71fa92b0 0x55bb71fb7030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:47.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.391+0000 7f5d44bee740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bb71fae420 0x55bb72022bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:47.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.391+0000 7f5d44bee740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f5d3c1c2f40 con 0x55bb71fae420 2024-03-18T16:16:47.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.391+0000 7f5d44bee740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f5d3c1c2870 con 0x55bb71fa92b0 2024-03-18T16:16:47.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.391+0000 7f5d44bee740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f5d3c11fde0 con 0x55bb71fa88c0 2024-03-18T16:16:47.393 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.392+0000 7f5d44bee740 10 client.0 _ll_register_callbacks cb 0x7f5d2c00a470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:16:47.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.392+0000 7f5d3aa00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bb71fa88c0 0x55bb71fb6af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:47.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.392+0000 7f5d3aa00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bb71fa88c0 0x55bb71fb6af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:54740/0 (socket says 172.21.15.39:54740) 2024-03-18T16:16:47.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.392+0000 7f5d3aa00640 1 -- 172.21.15.39:0/2557974535 learned_addr learned my addr 172.21.15.39:0/2557974535 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:16:47.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.392+0000 7f5d33e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bb71fa92b0 0x55bb71fb7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:47.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.393+0000 7f5d3b400640 1 --2- 172.21.15.39:0/2557974535 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bb71fae420 0x55bb72022bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:47.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.393+0000 7f5d3aa00640 1 -- 172.21.15.39:0/2557974535 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bb71fa92b0 msgr2=0x55bb71fb7030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:47.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.393+0000 7f5d3aa00640 1 --2- 172.21.15.39:0/2557974535 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bb71fa92b0 0x55bb71fb7030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:47.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.393+0000 7f5d3aa00640 1 -- 172.21.15.39:0/2557974535 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bb71fae420 msgr2=0x55bb72022bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:47.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.393+0000 7f5d3aa00640 1 --2- 172.21.15.39:0/2557974535 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bb71fae420 0x55bb72022bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:47.395 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.393+0000 7f5d3aa00640 1 -- 172.21.15.39:0/2557974535 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb72023110 con 0x55bb71fa88c0 2024-03-18T16:16:47.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.393+0000 7f5d3b400640 1 --2- 172.21.15.39:0/2557974535 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bb71fae420 0x55bb72022bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:16:47.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.394+0000 7f5d3aa00640 1 --2- 172.21.15.39:0/2557974535 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bb71fa88c0 0x55bb71fb6af0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f5d2c006010 tx=0x7f5d2c004450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5001c1c986ad7969 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:47.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.394+0000 7f5d28c00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:16:47.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.394+0000 7f5d28c00640 1 -- 172.21.15.39:0/2557974535 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d2c03c060 con 0x55bb71fa88c0 2024-03-18T16:16:47.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.394+0000 7f5d28c00640 1 -- 172.21.15.39:0/2557974535 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f5d2c005250 con 0x55bb71fa88c0 2024-03-18T16:16:47.396 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.394+0000 7f5d44bee740 1 -- 172.21.15.39:0/2557974535 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5d240c1160 con 0x55bb71fa88c0 2024-03-18T16:16:47.398 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.396+0000 7f5d28c00640 1 -- 172.21.15.39:0/2557974535 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f5d2c005450 con 0x55bb71fa88c0 2024-03-18T16:16:47.398 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.397+0000 7f5d44bee740 10 client.26971 fetch_fsmap learned FSMap version 44 2024-03-18T16:16:47.398 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.397+0000 7f5d44bee740 1 -- 172.21.15.39:0/2557974535 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x55bb720243c0 con 0x55bb71fa88c0 2024-03-18T16:16:47.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.397+0000 7f5d28c00640 1 -- 172.21.15.39:0/2557974535 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d240c1160 con 0x55bb71fa88c0 2024-03-18T16:16:47.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.397+0000 7f5d28c00640 1 -- 172.21.15.39:0/2557974535 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7f5d2c019040 con 0x55bb71fa88c0 2024-03-18T16:16:47.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.397+0000 7f5d44bee740 10 client.26971 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:16:47.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.397+0000 7f5d44bee740 10 client.26971 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-18T16:16:47.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.397+0000 7f5d44bee740 20 client.26971 populate_metadata read hostname 'smithi039' 2024-03-18T16:16:47.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.397+0000 7f5d44bee740 1 --2- 172.21.15.39:0/2557974535 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x55bb72024da0 0x55bb720271c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:47.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.397+0000 7f5d44bee740 4 client.26971 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:16:47.399 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.397+0000 7f5d44bee740 1 -- 172.21.15.39:0/2557974535 --> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55bb71f14ce0 con 0x55bb72024da0 2024-03-18T16:16:47.400 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.398+0000 7f5d33e00640 1 --2- 172.21.15.39:0/2557974535 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x55bb72024da0 0x55bb720271c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:47.400 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.398+0000 7f5d28c00640 10 client.26971 ms_handle_connect on v2:172.21.15.196:6836/895247251 2024-03-18T16:16:47.400 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.399+0000 7f5d33e00640 1 --2- 172.21.15.39:0/2557974535 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x55bb72024da0 0x55bb720271c0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4f0c68623301aa02 server_cookie=71c9e0a332501514 in_seq=0 out_seq=0 2024-03-18T16:16:47.400 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.399+0000 7f5d28c00640 1 -- 172.21.15.39:0/2557974535 <== mds.2 v2:172.21.15.196:6836/895247251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55bb71f14ce0 con 0x55bb72024da0 2024-03-18T16:16:47.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.399+0000 7f5d28c00640 10 client.26971 handle_command_reply: tid=0 2024-03-18T16:16:47.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.399+0000 7f5d44bee740 1 client.26971 shutdown 2024-03-18T16:16:47.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.400+0000 7f5d44bee740 1 -- 172.21.15.39:0/2557974535 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bb71fa88c0 msgr2=0x55bb71fb6af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:47.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.400+0000 7f5d44bee740 1 --2- 172.21.15.39:0/2557974535 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bb71fa88c0 0x55bb71fb6af0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f5d2c006010 tx=0x7f5d2c004450 comp rx=0 tx=0).stop 2024-03-18T16:16:47.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.400+0000 7f5d44bee740 1 -- 172.21.15.39:0/2557974535 shutdown_connections 2024-03-18T16:16:47.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.400+0000 7f5d44bee740 1 --2- 172.21.15.39:0/2557974535 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bb71fa92b0 0x55bb71fb7030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:47.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.400+0000 7f5d44bee740 1 --2- 172.21.15.39:0/2557974535 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x55bb72024da0 0x55bb720271c0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:47.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.400+0000 7f5d44bee740 1 --2- 172.21.15.39:0/2557974535 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bb71fae420 0x55bb72022bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:47.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.400+0000 7f5d44bee740 1 --2- 172.21.15.39:0/2557974535 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bb71fa88c0 0x55bb71fb6af0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:47.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.400+0000 7f5d44bee740 1 -- 172.21.15.39:0/2557974535 >> 172.21.15.39:0/2557974535 conn(0x55bb71fa3e30 msgr2=0x55bb71fa2960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:47.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.400+0000 7f5d28c00640 0 client.26971 ms_handle_reset on v2:172.21.15.196:6836/895247251 2024-03-18T16:16:47.403 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.400+0000 7f5d44bee740 1 -- 172.21.15.39:0/2557974535 shutdown_connections 2024-03-18T16:16:47.403 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.400+0000 7f5d44bee740 1 -- 172.21.15.39:0/2557974535 wait complete. 2024-03-18T16:16:47.403 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:16:47.403 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.401+0000 7f5d44bee740 20 client.26971 trim_cache size 0 max 16384 2024-03-18T16:16:47.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.402+0000 7f5d41e00640 1 -- 172.21.15.39:0/2823726458 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f5d200a2900 msgr2=0x7f5d200a4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:47.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.402+0000 7f5d41e00640 1 --2- 172.21.15.39:0/2823726458 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f5d200a2900 0x7f5d200a4dc0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x55bb71fa4da0 tx=0x7f5d2c00a2d0 comp rx=0 tx=0).stop 2024-03-18T16:16:47.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.402+0000 7f5d41e00640 1 -- 172.21.15.39:0/2823726458 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5d3c10b150 msgr2=0x7f5d3c1a6dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:47.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.402+0000 7f5d41e00640 1 --2- 172.21.15.39:0/2823726458 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5d3c10b150 0x7f5d3c1a6dc0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f5d24005ab0 tx=0x7f5d2402c6e0 comp rx=0 tx=0).stop 2024-03-18T16:16:47.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.402+0000 7f5d41e00640 1 -- 172.21.15.39:0/2823726458 shutdown_connections 2024-03-18T16:16:47.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.402+0000 7f5d41e00640 1 --2- 172.21.15.39:0/2823726458 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f5d200a2900 0x7f5d200a4dc0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:47.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.402+0000 7f5d41e00640 1 --2- 172.21.15.39:0/2823726458 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5d3c111f20 0x7f5d3c1ade30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:47.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.402+0000 7f5d41e00640 1 --2- 172.21.15.39:0/2823726458 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5d3c10b150 0x7f5d3c1a6dc0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:47.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.402+0000 7f5d41e00640 1 --2- 172.21.15.39:0/2823726458 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5d3c1087e0 0x7f5d3c1a6880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:47.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.402+0000 7f5d41e00640 1 -- 172.21.15.39:0/2823726458 >> 172.21.15.39:0/2823726458 conn(0x7f5d3c06e860 msgr2=0x7f5d3c1126b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:47.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.402+0000 7f5d41e00640 1 -- 172.21.15.39:0/2823726458 shutdown_connections 2024-03-18T16:16:47.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.403+0000 7f5d41e00640 1 -- 172.21.15.39:0/2823726458 wait complete. 2024-03-18T16:16:47.427 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-18T16:16:47.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.703+0000 7fdc07400640 1 -- 172.21.15.39:0/2691206706 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdc0010b410 msgr2=0x7fdc0010d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:47.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.704+0000 7fdc07400640 1 --2- 172.21.15.39:0/2691206706 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdc0010b410 0x7fdc0010d800 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fdbec009860 tx=0x7fdbec02bb70 comp rx=0 tx=0).stop 2024-03-18T16:16:47.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.704+0000 7fdc07400640 1 -- 172.21.15.39:0/2691206706 shutdown_connections 2024-03-18T16:16:47.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.704+0000 7fdc07400640 1 --2- 172.21.15.39:0/2691206706 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdc0010b410 0x7fdc0010d800 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:47.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.704+0000 7fdc07400640 1 --2- 172.21.15.39:0/2691206706 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdc0010de90 0x7fdc00110350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:47.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.704+0000 7fdc07400640 1 --2- 172.21.15.39:0/2691206706 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdc00108a50 0x7fdc0010ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:47.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.704+0000 7fdc07400640 1 -- 172.21.15.39:0/2691206706 >> 172.21.15.39:0/2691206706 conn(0x7fdc0006e9c0 msgr2=0x7fdc00070e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:47.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.705+0000 7fdc07400640 1 -- 172.21.15.39:0/2691206706 shutdown_connections 2024-03-18T16:16:47.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.705+0000 7fdc07400640 1 -- 172.21.15.39:0/2691206706 wait complete. 2024-03-18T16:16:47.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.706+0000 7fdc07400640 1 Processor -- start 2024-03-18T16:16:47.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.706+0000 7fdc07400640 1 -- start start 2024-03-18T16:16:47.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.707+0000 7fdc07400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdc00108a50 0x7fdc0019e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:47.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.708+0000 7fdc07400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdc0010b410 0x7fdc0019e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:47.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.708+0000 7fdc07400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdc0010de90 0x7fdc001a5750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:47.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.708+0000 7fdc07400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fdc00113a80 con 0x7fdc00108a50 2024-03-18T16:16:47.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.708+0000 7fdc07400640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fdc00113900 con 0x7fdc0010de90 2024-03-18T16:16:47.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.708+0000 7fdc07400640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fdc00113c00 con 0x7fdc0010b410 2024-03-18T16:16:47.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.708+0000 7fdc05400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdc0010de90 0x7fdc001a5750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:47.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.708+0000 7fdc05400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdc0010de90 0x7fdc001a5750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:58548/0 (socket says 172.21.15.39:58548) 2024-03-18T16:16:47.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.708+0000 7fdc05400640 1 -- 172.21.15.39:0/3536759816 learned_addr learned my addr 172.21.15.39:0/3536759816 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:16:47.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.708+0000 7fdbffe00640 1 --2- 172.21.15.39:0/3536759816 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdc0010b410 0x7fdc0019e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:47.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.709+0000 7fdc05400640 1 -- 172.21.15.39:0/3536759816 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdc0010b410 msgr2=0x7fdc0019e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:47.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.709+0000 7fdc05400640 1 --2- 172.21.15.39:0/3536759816 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdc0010b410 0x7fdc0019e730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:47.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.709+0000 7fdc05400640 1 -- 172.21.15.39:0/3536759816 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdc00108a50 msgr2=0x7fdc0019e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:47.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.709+0000 7fdc05400640 1 --2- 172.21.15.39:0/3536759816 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdc00108a50 0x7fdc0019e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:47.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.709+0000 7fdc05400640 1 -- 172.21.15.39:0/3536759816 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc001a5d10 con 0x7fdc0010de90 2024-03-18T16:16:47.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.709+0000 7fdc05400640 1 --2- 172.21.15.39:0/3536759816 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdc0010de90 0x7fdc001a5750 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fdbf8009fd0 tx=0x7fdbf800b260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:47.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.710+0000 7fdbfd600640 1 -- 172.21.15.39:0/3536759816 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbf8017070 con 0x7fdc0010de90 2024-03-18T16:16:47.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.710+0000 7fdbfd600640 1 -- 172.21.15.39:0/3536759816 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbf8003e50 con 0x7fdc0010de90 2024-03-18T16:16:47.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.710+0000 7fdbfd600640 1 -- 172.21.15.39:0/3536759816 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbf80106c0 con 0x7fdc0010de90 2024-03-18T16:16:47.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.710+0000 7fdc07400640 1 -- 172.21.15.39:0/3536759816 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc001a6000 con 0x7fdc0010de90 2024-03-18T16:16:47.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.710+0000 7fdc07400640 1 -- 172.21.15.39:0/3536759816 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc001a6500 con 0x7fdc0010de90 2024-03-18T16:16:47.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.715+0000 7fdbfd600640 1 -- 172.21.15.39:0/3536759816 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fdbf8003970 con 0x7fdc0010de90 2024-03-18T16:16:47.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.717+0000 7fdc0a1b3740 1 Processor -- start 2024-03-18T16:16:47.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.717+0000 7fdc0a1b3740 1 -- start start 2024-03-18T16:16:47.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.717+0000 7fdc0a1b3740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5555dc5e30d0 0x5555dc5e34f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:47.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.717+0000 7fdc0a1b3740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5555dc5e3ac0 0x5555dc5e85c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:47.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.717+0000 7fdc0a1b3740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5555dc5e8c30 0x5555dc5eb0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:47.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.717+0000 7fdc0a1b3740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5555dc5a23b0 con 0x5555dc5e3ac0 2024-03-18T16:16:47.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.717+0000 7fdc0a1b3740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5555dc53f450 con 0x5555dc5e8c30 2024-03-18T16:16:47.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.717+0000 7fdc0a1b3740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x5555dc4e7cb0 con 0x5555dc5e30d0 2024-03-18T16:16:47.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.718+0000 7fdc04a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5555dc5e3ac0 0x5555dc5e85c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:47.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.718+0000 7fdbfd600640 1 --2- 172.21.15.39:0/3536759816 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fdbe80a2950 0x7fdbe80a4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:47.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.718+0000 7fdbfd600640 1 -- 172.21.15.39:0/3536759816 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fdbf80cec30 con 0x7fdc0010de90 2024-03-18T16:16:47.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.718+0000 7fdc05400640 1 --2- 172.21.15.39:0/3536759816 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fdbe80a2950 0x7fdbe80a4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:47.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.720+0000 7fdbffe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5555dc5e30d0 0x5555dc5e34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:47.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.720+0000 7fdbffe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5555dc5e30d0 0x5555dc5e34f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:54754/0 (socket says 172.21.15.39:54754) 2024-03-18T16:16:47.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.720+0000 7fdbffe00640 1 -- 172.21.15.39:0/1530100651 learned_addr learned my addr 172.21.15.39:0/1530100651 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:16:47.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.720+0000 7fdbffe00640 1 --2- 172.21.15.39:0/1530100651 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5555dc5e8c30 0x5555dc5eb0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:47.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.720+0000 7fdc05400640 1 --2- 172.21.15.39:0/3536759816 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fdbe80a2950 0x7fdbe80a4e10 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fdbf8097bb0 tx=0x7fdbf8004020 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:47.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.721+0000 7fdc04a00640 1 -- 172.21.15.39:0/1530100651 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5555dc5e30d0 msgr2=0x5555dc5e34f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:47.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.721+0000 7fdc04a00640 1 --2- 172.21.15.39:0/1530100651 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5555dc5e30d0 0x5555dc5e34f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:47.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.721+0000 7fdc04a00640 1 -- 172.21.15.39:0/1530100651 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5555dc5e8c30 msgr2=0x5555dc5eb0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:47.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.721+0000 7fdc04a00640 1 --2- 172.21.15.39:0/1530100651 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5555dc5e8c30 0x5555dc5eb0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:47.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.721+0000 7fdc04a00640 1 -- 172.21.15.39:0/1530100651 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5555dc5091f0 con 0x5555dc5e3ac0 2024-03-18T16:16:47.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.721+0000 7fdbffe00640 1 --2- 172.21.15.39:0/1530100651 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5555dc5e30d0 0x5555dc5e34f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:16:47.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.721+0000 7fdc04a00640 1 --2- 172.21.15.39:0/1530100651 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5555dc5e3ac0 0x5555dc5e85c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fdbf400a5f0 tx=0x7fdbf400aab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36891e47ee22540d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:47.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.722+0000 7fdbf1c00640 1 -- 172.21.15.39:0/1530100651 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbf40098c0 con 0x5555dc5e3ac0 2024-03-18T16:16:47.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.722+0000 7fdbf1c00640 1 -- 172.21.15.39:0/1530100651 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbf40047a0 con 0x5555dc5e3ac0 2024-03-18T16:16:47.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.722+0000 7fdbf1c00640 1 -- 172.21.15.39:0/1530100651 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbf40098c0 con 0x5555dc5e3ac0 2024-03-18T16:16:47.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.723+0000 7fdc0a1b3740 1 -- 172.21.15.39:0/1530100651 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5555dc5e3ac0 msgr2=0x5555dc5e85c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:47.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.723+0000 7fdc0a1b3740 1 --2- 172.21.15.39:0/1530100651 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5555dc5e3ac0 0x5555dc5e85c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fdbf400a5f0 tx=0x7fdbf400aab0 comp rx=0 tx=0).stop 2024-03-18T16:16:47.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.724+0000 7fdc0a1b3740 1 -- 172.21.15.39:0/1530100651 shutdown_connections 2024-03-18T16:16:47.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.724+0000 7fdc0a1b3740 1 --2- 172.21.15.39:0/1530100651 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5555dc5e8c30 0x5555dc5eb0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:47.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.724+0000 7fdc0a1b3740 1 --2- 172.21.15.39:0/1530100651 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5555dc5e3ac0 0x5555dc5e85c0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:47.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.724+0000 7fdc0a1b3740 1 --2- 172.21.15.39:0/1530100651 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5555dc5e30d0 0x5555dc5e34f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:47.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.724+0000 7fdc0a1b3740 1 -- 172.21.15.39:0/1530100651 >> 172.21.15.39:0/1530100651 conn(0x5555dc5d5e30 msgr2=0x5555dc5d6230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:47.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.725+0000 7fdc0a1b3740 1 -- 172.21.15.39:0/1530100651 shutdown_connections 2024-03-18T16:16:47.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.725+0000 7fdc0a1b3740 1 -- 172.21.15.39:0/1530100651 wait complete. 2024-03-18T16:16:47.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.725+0000 7fdc0a1b3740 1 -- start start 2024-03-18T16:16:47.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:47 smithi196 ceph-mon[36307]: pgmap v623: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 1.4 MiB/s wr, 381 op/s 2024-03-18T16:16:47.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.728+0000 7fdc0a1b3740 1 Processor -- start 2024-03-18T16:16:47.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.729+0000 7fdc0a1b3740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5555dc5e30d0 0x5555dc5ed310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:47.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.729+0000 7fdc0a1b3740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5555dc5e3ac0 0x5555dc5ed850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:47.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.729+0000 7fdc0a1b3740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5555dc5e8c30 0x5555dc5f4890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:47.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.729+0000 7fdc0a1b3740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5555dc4fb110 con 0x5555dc5e30d0 2024-03-18T16:16:47.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.730+0000 7fdc0a1b3740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5555dc53b4f0 con 0x5555dc5e3ac0 2024-03-18T16:16:47.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.730+0000 7fdc0a1b3740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x5555dc5ebbe0 con 0x5555dc5e8c30 2024-03-18T16:16:47.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.730+0000 7fdbffe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5555dc5e30d0 0x5555dc5ed310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:47.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.730+0000 7fdbffe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5555dc5e30d0 0x5555dc5ed310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:46858/0 (socket says 172.21.15.39:46858) 2024-03-18T16:16:47.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.730+0000 7fdbffe00640 1 -- 172.21.15.39:0/142723958 learned_addr learned my addr 172.21.15.39:0/142723958 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:16:47.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.730+0000 7fdbffe00640 1 --2- 172.21.15.39:0/142723958 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5555dc5e8c30 0x5555dc5f4890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:47.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.730+0000 7fdc04a00640 1 --2- 172.21.15.39:0/142723958 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5555dc5e3ac0 0x5555dc5ed850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:47.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.730+0000 7fdc0a1b3740 10 client.0 _ll_register_callbacks cb 0x5555dc4e8b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:16:47.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.731+0000 7fdc04a00640 1 -- 172.21.15.39:0/142723958 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5555dc5e8c30 msgr2=0x5555dc5f4890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:47.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.731+0000 7fdc04a00640 1 --2- 172.21.15.39:0/142723958 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5555dc5e8c30 0x5555dc5f4890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:47.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.731+0000 7fdc04a00640 1 -- 172.21.15.39:0/142723958 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5555dc5e30d0 msgr2=0x5555dc5ed310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:47.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.731+0000 7fdc04a00640 1 --2- 172.21.15.39:0/142723958 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5555dc5e30d0 0x5555dc5ed310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:47.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.731+0000 7fdc04a00640 1 -- 172.21.15.39:0/142723958 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5555dc2533f0 con 0x5555dc5e3ac0 2024-03-18T16:16:47.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.731+0000 7fdbffe00640 1 --2- 172.21.15.39:0/142723958 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5555dc5e30d0 0x5555dc5ed310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:16:47.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.732+0000 7fdc04a00640 1 --2- 172.21.15.39:0/142723958 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5555dc5e3ac0 0x5555dc5ed850 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x5555dc5d6da0 tx=0x7fdbf4009af0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c90bd4b52b4b947 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:47.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.732+0000 7fdbe6000640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:16:47.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.732+0000 7fdbe6000640 1 -- 172.21.15.39:0/142723958 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbf4022070 con 0x5555dc5e3ac0 2024-03-18T16:16:47.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.732+0000 7fdc0a1b3740 1 -- 172.21.15.39:0/142723958 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdbf40047a0 con 0x5555dc5e3ac0 2024-03-18T16:16:47.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.733+0000 7fdbe6000640 1 -- 172.21.15.39:0/142723958 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fdbf4004020 con 0x5555dc5e3ac0 2024-03-18T16:16:47.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.733+0000 7fdbe6000640 1 -- 172.21.15.39:0/142723958 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fdbf402e050 con 0x5555dc5e3ac0 2024-03-18T16:16:47.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.733+0000 7fdc0a1b3740 10 client.27008 fetch_fsmap learned FSMap version 44 2024-03-18T16:16:47.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.733+0000 7fdc0a1b3740 1 -- 172.21.15.39:0/142723958 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x5555dc31d7d0 con 0x5555dc5e3ac0 2024-03-18T16:16:47.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.734+0000 7fdbe6000640 1 -- 172.21.15.39:0/142723958 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbf40047a0 con 0x5555dc5e3ac0 2024-03-18T16:16:47.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.734+0000 7fdbe6000640 1 -- 172.21.15.39:0/142723958 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7fdbf4013070 con 0x5555dc5e3ac0 2024-03-18T16:16:47.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.734+0000 7fdc0a1b3740 10 client.27008 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:16:47.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.734+0000 7fdc0a1b3740 10 client.27008 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T16:16:47.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.734+0000 7fdc0a1b3740 20 client.27008 populate_metadata read hostname 'smithi039' 2024-03-18T16:16:47.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.734+0000 7fdc0a1b3740 1 --2- 172.21.15.39:0/142723958 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x5555dc5f8e10 0x5555dc5fb200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:47.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.734+0000 7fdc0a1b3740 4 client.27008 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:16:47.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.734+0000 7fdc0a1b3740 1 -- 172.21.15.39:0/142723958 --> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5555dc54c990 con 0x5555dc5f8e10 2024-03-18T16:16:47.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.735+0000 7fdbffe00640 1 --2- 172.21.15.39:0/142723958 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x5555dc5f8e10 0x5555dc5fb200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:47.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.736+0000 7fdbffe00640 1 --2- 172.21.15.39:0/142723958 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x5555dc5f8e10 0x5555dc5fb200 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5e4551b9b8dd3510 server_cookie=3f440d6c9c8b6cb4 in_seq=0 out_seq=0 2024-03-18T16:16:47.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.736+0000 7fdbe6000640 10 client.27008 ms_handle_connect on v2:172.21.15.196:6838/2001427268 2024-03-18T16:16:47.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.737+0000 7fdbe6000640 1 -- 172.21.15.39:0/142723958 <== mds.1 v2:172.21.15.196:6838/2001427268 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5555dc54c990 con 0x5555dc5f8e10 2024-03-18T16:16:47.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.737+0000 7fdbe6000640 10 client.27008 handle_command_reply: tid=0 2024-03-18T16:16:47.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:47 smithi194 ceph-mon[36200]: pgmap v623: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 1.4 MiB/s wr, 381 op/s 2024-03-18T16:16:47.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.739+0000 7fdc0a1b3740 1 client.27008 shutdown 2024-03-18T16:16:47.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.739+0000 7fdc0a1b3740 1 -- 172.21.15.39:0/142723958 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5555dc5e3ac0 msgr2=0x5555dc5ed850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:47.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.739+0000 7fdc0a1b3740 1 --2- 172.21.15.39:0/142723958 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5555dc5e3ac0 0x5555dc5ed850 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x5555dc5d6da0 tx=0x7fdbf4009af0 comp rx=0 tx=0).stop 2024-03-18T16:16:47.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.740+0000 7fdc0a1b3740 1 -- 172.21.15.39:0/142723958 shutdown_connections 2024-03-18T16:16:47.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.740+0000 7fdc0a1b3740 1 --2- 172.21.15.39:0/142723958 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x5555dc5f8e10 0x5555dc5fb200 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:47.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.740+0000 7fdc0a1b3740 1 --2- 172.21.15.39:0/142723958 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5555dc5e3ac0 0x5555dc5ed850 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:47.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.740+0000 7fdc0a1b3740 1 --2- 172.21.15.39:0/142723958 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5555dc5e8c30 0x5555dc5f4890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:47.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.740+0000 7fdc0a1b3740 1 --2- 172.21.15.39:0/142723958 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5555dc5e30d0 0x5555dc5ed310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:47.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.740+0000 7fdc0a1b3740 1 -- 172.21.15.39:0/142723958 >> 172.21.15.39:0/142723958 conn(0x5555dc5d5e30 msgr2=0x5555dc5d51f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:47.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.740+0000 7fdbe6000640 0 client.27008 ms_handle_reset on v2:172.21.15.196:6838/2001427268 2024-03-18T16:16:47.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.741+0000 7fdc0a1b3740 1 -- 172.21.15.39:0/142723958 shutdown_connections 2024-03-18T16:16:47.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.741+0000 7fdc0a1b3740 1 -- 172.21.15.39:0/142723958 wait complete. 2024-03-18T16:16:47.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.741+0000 7fdc0a1b3740 20 client.27008 trim_cache size 0 max 16384 2024-03-18T16:16:47.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.769+0000 7fdc0a1b3740 1 Processor -- start 2024-03-18T16:16:47.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.769+0000 7fdc0a1b3740 1 -- start start 2024-03-18T16:16:47.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.769+0000 7fdc0a1b3740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5555dc5e30d0 0x5555dc5d4960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:47.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.769+0000 7fdc0a1b3740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5555dc5e3ac0 0x5555dc5f4850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:47.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.769+0000 7fdc0a1b3740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5555dc5e8c30 0x5555dc5d4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:47.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.769+0000 7fdc0a1b3740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fdc001b0ae0 con 0x5555dc5e3ac0 2024-03-18T16:16:47.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.769+0000 7fdc0a1b3740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fdc001b2620 con 0x5555dc5e30d0 2024-03-18T16:16:47.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.769+0000 7fdc0a1b3740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fdc001b11b0 con 0x5555dc5e8c30 2024-03-18T16:16:47.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.770+0000 7fdbffe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5555dc5e30d0 0x5555dc5d4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:47.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.770+0000 7fdbffe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5555dc5e30d0 0x5555dc5d4960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:58574/0 (socket says 172.21.15.39:58574) 2024-03-18T16:16:47.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.770+0000 7fdbffe00640 1 -- 172.21.15.39:0/1007368465 learned_addr learned my addr 172.21.15.39:0/1007368465 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:16:47.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.770+0000 7fdbffe00640 1 --2- 172.21.15.39:0/1007368465 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5555dc5e8c30 0x5555dc5d4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:47.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.770+0000 7fdc04a00640 1 --2- 172.21.15.39:0/1007368465 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5555dc5e3ac0 0x5555dc5f4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:47.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.770+0000 7fdbffe00640 1 -- 172.21.15.39:0/1007368465 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5555dc5e8c30 msgr2=0x5555dc5d4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:47.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.770+0000 7fdbffe00640 1 --2- 172.21.15.39:0/1007368465 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5555dc5e8c30 0x5555dc5d4ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:47.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.770+0000 7fdbffe00640 1 -- 172.21.15.39:0/1007368465 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5555dc5e3ac0 msgr2=0x5555dc5f4850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:47.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.770+0000 7fdbffe00640 1 --2- 172.21.15.39:0/1007368465 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5555dc5e3ac0 0x5555dc5f4850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:47.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.770+0000 7fdbffe00640 1 -- 172.21.15.39:0/1007368465 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5555dc5d1c10 con 0x5555dc5e30d0 2024-03-18T16:16:47.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.770+0000 7fdc04a00640 1 --2- 172.21.15.39:0/1007368465 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5555dc5e3ac0 0x5555dc5f4850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:16:47.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.771+0000 7fdbffe00640 1 --2- 172.21.15.39:0/1007368465 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5555dc5e30d0 0x5555dc5d4960 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fdbec0349e0 tx=0x7fdbec02bb70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ff5535f8698d77 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:47.773 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.771+0000 7fdbf1200640 1 -- 172.21.15.39:0/1007368465 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbec039040 con 0x5555dc5e30d0 2024-03-18T16:16:47.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.772+0000 7fdbf1200640 1 -- 172.21.15.39:0/1007368465 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbec032640 con 0x5555dc5e30d0 2024-03-18T16:16:47.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.772+0000 7fdbf1200640 1 -- 172.21.15.39:0/1007368465 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbec003830 con 0x5555dc5e30d0 2024-03-18T16:16:47.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.773+0000 7fdc0a1b3740 1 -- 172.21.15.39:0/1007368465 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5555dc5e30d0 msgr2=0x5555dc5d4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:47.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.773+0000 7fdc0a1b3740 1 --2- 172.21.15.39:0/1007368465 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5555dc5e30d0 0x5555dc5d4960 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fdbec0349e0 tx=0x7fdbec02bb70 comp rx=0 tx=0).stop 2024-03-18T16:16:47.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.774+0000 7fdc0a1b3740 1 -- 172.21.15.39:0/1007368465 shutdown_connections 2024-03-18T16:16:47.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.774+0000 7fdc0a1b3740 1 --2- 172.21.15.39:0/1007368465 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5555dc5e8c30 0x5555dc5d4ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:47.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.774+0000 7fdc0a1b3740 1 --2- 172.21.15.39:0/1007368465 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5555dc5e3ac0 0x5555dc5f4850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:47.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.774+0000 7fdc0a1b3740 1 --2- 172.21.15.39:0/1007368465 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5555dc5e30d0 0x5555dc5d4960 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:47.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.774+0000 7fdc0a1b3740 1 -- 172.21.15.39:0/1007368465 >> 172.21.15.39:0/1007368465 conn(0x5555dc5d5e30 msgr2=0x5555dc5e4e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:47.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.775+0000 7fdc0a1b3740 1 -- 172.21.15.39:0/1007368465 shutdown_connections 2024-03-18T16:16:47.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.775+0000 7fdc0a1b3740 1 -- 172.21.15.39:0/1007368465 wait complete. 2024-03-18T16:16:47.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.775+0000 7fdc0a1b3740 1 -- start start 2024-03-18T16:16:47.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.775+0000 7fdc0a1b3740 1 Processor -- start 2024-03-18T16:16:47.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.777+0000 7fdc0a1b3740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5555dc5e30d0 0x5555dc5f1b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:47.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.777+0000 7fdc0a1b3740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5555dc5e3ac0 0x5555dc5f20c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:47.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.777+0000 7fdc0a1b3740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5555dc5e8c30 0x5555dc5f3080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:47.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.777+0000 7fdc0a1b3740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fdc001b1f50 con 0x5555dc5e30d0 2024-03-18T16:16:47.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.777+0000 7fdc0a1b3740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fdc001b1880 con 0x5555dc5e3ac0 2024-03-18T16:16:47.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.777+0000 7fdbffe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5555dc5e30d0 0x5555dc5f1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:47.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.777+0000 7fdc0a1b3740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fdc001aea80 con 0x5555dc5e8c30 2024-03-18T16:16:47.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.777+0000 7fdc0a1b3740 10 client.0 _ll_register_callbacks cb 0x7fdbf400a4f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:16:47.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.777+0000 7fdc04a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5555dc5e3ac0 0x5555dc5f20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:47.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.777+0000 7fdbffe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5555dc5e30d0 0x5555dc5f1b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:46870/0 (socket says 172.21.15.39:46870) 2024-03-18T16:16:47.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.777+0000 7fdbffe00640 1 -- 172.21.15.39:0/3742144594 learned_addr learned my addr 172.21.15.39:0/3742144594 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:16:47.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.778+0000 7fdc04a00640 1 -- 172.21.15.39:0/3742144594 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5555dc5e8c30 msgr2=0x5555dc5f3080 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T16:16:47.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.778+0000 7fdc04a00640 1 --2- 172.21.15.39:0/3742144594 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5555dc5e8c30 0x5555dc5f3080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:47.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.778+0000 7fdc04a00640 1 -- 172.21.15.39:0/3742144594 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5555dc5e30d0 msgr2=0x5555dc5f1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:47.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.778+0000 7fdc04a00640 1 --2- 172.21.15.39:0/3742144594 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5555dc5e30d0 0x5555dc5f1b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:47.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.778+0000 7fdc04a00640 1 -- 172.21.15.39:0/3742144594 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5555dc5f3670 con 0x5555dc5e3ac0 2024-03-18T16:16:47.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.778+0000 7fdbffe00640 1 --2- 172.21.15.39:0/3742144594 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5555dc5e30d0 0x5555dc5f1b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:16:47.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.779+0000 7fdc04a00640 1 --2- 172.21.15.39:0/3742144594 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5555dc5e3ac0 0x5555dc5f20c0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x5555dc5d6910 tx=0x7fdbf4005e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9ea3d9bfb3cc9fc5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:47.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.779+0000 7fdbe6000640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:16:47.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.780+0000 7fdbe6000640 1 -- 172.21.15.39:0/3742144594 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbf402e050 con 0x5555dc5e3ac0 2024-03-18T16:16:47.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.780+0000 7fdbe6000640 1 -- 172.21.15.39:0/3742144594 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fdbf4019030 con 0x5555dc5e3ac0 2024-03-18T16:16:47.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.780+0000 7fdc0a1b3740 1 -- 172.21.15.39:0/3742144594 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdbec032640 con 0x5555dc5e3ac0 2024-03-18T16:16:47.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.781+0000 7fdbe6000640 1 -- 172.21.15.39:0/3742144594 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fdbf40047a0 con 0x5555dc5e3ac0 2024-03-18T16:16:47.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.781+0000 7fdc0a1b3740 10 client.27014 fetch_fsmap learned FSMap version 44 2024-03-18T16:16:47.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.781+0000 7fdc0a1b3740 1 -- 172.21.15.39:0/3742144594 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x5555dc5f48e0 con 0x5555dc5e3ac0 2024-03-18T16:16:47.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.781+0000 7fdbe6000640 1 -- 172.21.15.39:0/3742144594 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdbec032640 con 0x5555dc5e3ac0 2024-03-18T16:16:47.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.781+0000 7fdbe6000640 1 -- 172.21.15.39:0/3742144594 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7fdbf4013070 con 0x5555dc5e3ac0 2024-03-18T16:16:47.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.781+0000 7fdc0a1b3740 10 client.27014 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:16:47.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.781+0000 7fdc0a1b3740 10 client.27014 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T16:16:47.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.781+0000 7fdc0a1b3740 20 client.27014 populate_metadata read hostname 'smithi039' 2024-03-18T16:16:47.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.781+0000 7fdc0a1b3740 1 --2- 172.21.15.39:0/3742144594 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x5555dc65f4a0 0x5555dc6618c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:47.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.781+0000 7fdc0a1b3740 4 client.27014 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:16:47.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.781+0000 7fdc0a1b3740 1 -- 172.21.15.39:0/3742144594 --> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5555dc5ec750 con 0x5555dc65f4a0 2024-03-18T16:16:47.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.783+0000 7fdbffe00640 1 --2- 172.21.15.39:0/3742144594 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x5555dc65f4a0 0x5555dc6618c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:47.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.783+0000 7fdbffe00640 1 --2- 172.21.15.39:0/3742144594 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x5555dc65f4a0 0x5555dc6618c0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a51243d61000d45f server_cookie=a3251d05a696e8a0 in_seq=0 out_seq=0 2024-03-18T16:16:47.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.783+0000 7fdbe6000640 10 client.27014 ms_handle_connect on v2:172.21.15.196:6838/2001427268 2024-03-18T16:16:47.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.784+0000 7fdbe6000640 1 -- 172.21.15.39:0/3742144594 <== mds.1 v2:172.21.15.196:6838/2001427268 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5555dc5ec750 con 0x5555dc65f4a0 2024-03-18T16:16:47.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.784+0000 7fdbe6000640 10 client.27014 handle_command_reply: tid=0 2024-03-18T16:16:47.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.784+0000 7fdc0a1b3740 1 client.27014 shutdown 2024-03-18T16:16:47.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.784+0000 7fdc0a1b3740 1 -- 172.21.15.39:0/3742144594 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5555dc5e3ac0 msgr2=0x5555dc5f20c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:47.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.784+0000 7fdc0a1b3740 1 --2- 172.21.15.39:0/3742144594 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5555dc5e3ac0 0x5555dc5f20c0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x5555dc5d6910 tx=0x7fdbf4005e80 comp rx=0 tx=0).stop 2024-03-18T16:16:47.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.785+0000 7fdc0a1b3740 1 -- 172.21.15.39:0/3742144594 shutdown_connections 2024-03-18T16:16:47.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.785+0000 7fdc0a1b3740 1 --2- 172.21.15.39:0/3742144594 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x5555dc65f4a0 0x5555dc6618c0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:47.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.785+0000 7fdc0a1b3740 1 --2- 172.21.15.39:0/3742144594 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5555dc5e3ac0 0x5555dc5f20c0 secure :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0x5555dc5d6910 tx=0x7fdbf4005e80 comp rx=0 tx=0).stop 2024-03-18T16:16:47.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.785+0000 7fdc0a1b3740 1 --2- 172.21.15.39:0/3742144594 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5555dc5e8c30 0x5555dc5f3080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:47.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.785+0000 7fdbe6000640 0 client.27014 ms_handle_reset on v2:172.21.15.196:6838/2001427268 2024-03-18T16:16:47.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.785+0000 7fdc0a1b3740 1 --2- 172.21.15.39:0/3742144594 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5555dc5e30d0 0x5555dc5f1b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:47.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.785+0000 7fdc0a1b3740 1 -- 172.21.15.39:0/3742144594 >> 172.21.15.39:0/3742144594 conn(0x5555dc5d5e30 msgr2=0x5555dc5d4960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:47.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.785+0000 7fdc0a1b3740 1 -- 172.21.15.39:0/3742144594 shutdown_connections 2024-03-18T16:16:47.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.785+0000 7fdc0a1b3740 1 -- 172.21.15.39:0/3742144594 wait complete. 2024-03-18T16:16:47.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.785+0000 7fdc0a1b3740 20 client.27014 trim_cache size 0 max 16384 2024-03-18T16:16:47.788 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:16:47.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.786+0000 7fdc07400640 1 -- 172.21.15.39:0/3536759816 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fdbe80a2950 msgr2=0x7fdbe80a4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:47.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.786+0000 7fdc07400640 1 --2- 172.21.15.39:0/3536759816 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fdbe80a2950 0x7fdbe80a4e10 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fdbf8097bb0 tx=0x7fdbf8004020 comp rx=0 tx=0).stop 2024-03-18T16:16:47.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.787+0000 7fdc07400640 1 -- 172.21.15.39:0/3536759816 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdc0010de90 msgr2=0x7fdc001a5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:47.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.787+0000 7fdc07400640 1 --2- 172.21.15.39:0/3536759816 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdc0010de90 0x7fdc001a5750 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fdbf8009fd0 tx=0x7fdbf800b260 comp rx=0 tx=0).stop 2024-03-18T16:16:47.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.787+0000 7fdc07400640 1 -- 172.21.15.39:0/3536759816 shutdown_connections 2024-03-18T16:16:47.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.787+0000 7fdc07400640 1 --2- 172.21.15.39:0/3536759816 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdc0010de90 0x7fdc001a5750 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:47.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.787+0000 7fdc07400640 1 --2- 172.21.15.39:0/3536759816 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fdbe80a2950 0x7fdbe80a4e10 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:47.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.787+0000 7fdc07400640 1 --2- 172.21.15.39:0/3536759816 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdc0010b410 0x7fdc0019e730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:47.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.787+0000 7fdc07400640 1 --2- 172.21.15.39:0/3536759816 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdc00108a50 0x7fdc0019e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:47.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.787+0000 7fdc07400640 1 -- 172.21.15.39:0/3536759816 >> 172.21.15.39:0/3536759816 conn(0x7fdc0006e9c0 msgr2=0x7fdc00109390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:47.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.787+0000 7fdc07400640 1 -- 172.21.15.39:0/3536759816 shutdown_connections 2024-03-18T16:16:47.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:47.787+0000 7fdc07400640 1 -- 172.21.15.39:0/3536759816 wait complete. 2024-03-18T16:16:47.811 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:47 smithi039 ceph-mon[31721]: pgmap v623: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 1.4 MiB/s wr, 381 op/s 2024-03-18T16:16:47.814 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-18T16:16:48.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.091+0000 7f9855400640 1 -- 172.21.15.39:0/2443245867 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9840032340 con 0x7f985810b410 2024-03-18T16:16:48.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.091+0000 7f985ce00640 1 -- 172.21.15.39:0/2443245867 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f985810b410 msgr2=0x7f985810d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:48.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.091+0000 7f985ce00640 1 --2- 172.21.15.39:0/2443245867 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f985810b410 0x7f985810d800 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f9840009860 tx=0x7f984002bb90 comp rx=0 tx=0).stop 2024-03-18T16:16:48.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.092+0000 7f985ce00640 1 -- 172.21.15.39:0/2443245867 shutdown_connections 2024-03-18T16:16:48.093 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.092+0000 7f985ce00640 1 --2- 172.21.15.39:0/2443245867 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f985810de90 0x7f9858110350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:48.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.092+0000 7f985ce00640 1 --2- 172.21.15.39:0/2443245867 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f985810b410 0x7f985810d800 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:48.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.092+0000 7f985ce00640 1 --2- 172.21.15.39:0/2443245867 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9858108a50 0x7f985810ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:48.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.092+0000 7f985ce00640 1 -- 172.21.15.39:0/2443245867 >> 172.21.15.39:0/2443245867 conn(0x7f985806e860 msgr2=0x7f9858070ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:48.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.092+0000 7f985ce00640 1 -- 172.21.15.39:0/2443245867 shutdown_connections 2024-03-18T16:16:48.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.092+0000 7f985ce00640 1 -- 172.21.15.39:0/2443245867 wait complete. 2024-03-18T16:16:48.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.094+0000 7f985ce00640 1 Processor -- start 2024-03-18T16:16:48.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.094+0000 7f985ce00640 1 -- start start 2024-03-18T16:16:48.096 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.095+0000 7f985ce00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9858108a50 0x7f98581a69a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:48.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.095+0000 7f985ce00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f985810b410 0x7f98581a6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:48.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.095+0000 7f985ce00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f985810de90 0x7f98581adf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:48.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.095+0000 7f985ce00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f98581130c0 con 0x7f985810de90 2024-03-18T16:16:48.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.095+0000 7f985ce00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f9858112f40 con 0x7f9858108a50 2024-03-18T16:16:48.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.095+0000 7f985ce00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f9858113240 con 0x7f985810b410 2024-03-18T16:16:48.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.095+0000 7f9855e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9858108a50 0x7f98581a69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:48.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.095+0000 7f9855e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9858108a50 0x7f98581a69a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:46208/0 (socket says 172.21.15.39:46208) 2024-03-18T16:16:48.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.095+0000 7f9855e00640 1 -- 172.21.15.39:0/42628867 learned_addr learned my addr 172.21.15.39:0/42628867 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:16:48.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.095+0000 7f984d400640 1 --2- 172.21.15.39:0/42628867 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f985810b410 0x7f98581a6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:48.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.095+0000 7f9856800640 1 --2- 172.21.15.39:0/42628867 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f985810de90 0x7f98581adf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:48.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.096+0000 7f9855e00640 1 -- 172.21.15.39:0/42628867 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f985810b410 msgr2=0x7f98581a6ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:48.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.096+0000 7f9855e00640 1 --2- 172.21.15.39:0/42628867 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f985810b410 0x7f98581a6ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:48.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.096+0000 7f9855e00640 1 -- 172.21.15.39:0/42628867 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f985810de90 msgr2=0x7f98581adf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:48.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.096+0000 7f9855e00640 1 --2- 172.21.15.39:0/42628867 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f985810de90 0x7f98581adf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:48.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.096+0000 7f9855e00640 1 -- 172.21.15.39:0/42628867 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98581ae5c0 con 0x7f9858108a50 2024-03-18T16:16:48.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.096+0000 7f984d400640 1 --2- 172.21.15.39:0/42628867 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f985810b410 0x7f98581a6ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:16:48.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.096+0000 7f9855e00640 1 --2- 172.21.15.39:0/42628867 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9858108a50 0x7f98581a69a0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f984800a6c0 tx=0x7f984800ab80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:48.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.097+0000 7f984f400640 1 -- 172.21.15.39:0/42628867 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9848017070 con 0x7f9858108a50 2024-03-18T16:16:48.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.097+0000 7f984f400640 1 -- 172.21.15.39:0/42628867 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f984800ee30 con 0x7f9858108a50 2024-03-18T16:16:48.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.097+0000 7f984f400640 1 -- 172.21.15.39:0/42628867 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98480108a0 con 0x7f9858108a50 2024-03-18T16:16:48.099 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.097+0000 7f985ce00640 1 -- 172.21.15.39:0/42628867 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98581ae8b0 con 0x7f9858108a50 2024-03-18T16:16:48.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.097+0000 7f985ce00640 1 -- 172.21.15.39:0/42628867 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98581aed80 con 0x7f9858108a50 2024-03-18T16:16:48.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.102+0000 7f984f400640 1 -- 172.21.15.39:0/42628867 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f984801a070 con 0x7f9858108a50 2024-03-18T16:16:48.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.103+0000 7f984f400640 1 --2- 172.21.15.39:0/42628867 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f983c0a28d0 0x7f983c0a4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:48.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.103+0000 7f985fba3740 1 Processor -- start 2024-03-18T16:16:48.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.104+0000 7f984f400640 1 -- 172.21.15.39:0/42628867 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f98480cea80 con 0x7f9858108a50 2024-03-18T16:16:48.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.104+0000 7f985fba3740 1 -- start start 2024-03-18T16:16:48.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.104+0000 7f984d400640 1 --2- 172.21.15.39:0/42628867 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f983c0a28d0 0x7f983c0a4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:48.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.104+0000 7f985fba3740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e47e7d18c0 0x55e47e7d1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:48.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.104+0000 7f985fba3740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e47e7d22b0 0x55e47e7d6db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:48.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.104+0000 7f985fba3740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e47e7d7420 0x55e47e7d98b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:48.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.104+0000 7f985fba3740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55e47e7993b0 con 0x55e47e7d7420 2024-03-18T16:16:48.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.104+0000 7f985fba3740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55e47e736450 con 0x55e47e7d22b0 2024-03-18T16:16:48.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.104+0000 7f985fba3740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55e47e6decb0 con 0x55e47e7d18c0 2024-03-18T16:16:48.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.105+0000 7f9856800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e47e7d7420 0x55e47e7d98b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:48.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.105+0000 7f9855e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e47e7d18c0 0x55e47e7d1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:48.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.105+0000 7f9855e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e47e7d18c0 0x55e47e7d1ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:49736/0 (socket says 172.21.15.39:49736) 2024-03-18T16:16:48.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.105+0000 7f9855e00640 1 -- 172.21.15.39:0/736042072 learned_addr learned my addr 172.21.15.39:0/736042072 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:16:48.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.105+0000 7f984d400640 1 --2- 172.21.15.39:0/736042072 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e47e7d22b0 0x55e47e7d6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:48.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.106+0000 7f9855e00640 1 -- 172.21.15.39:0/736042072 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e47e7d22b0 msgr2=0x55e47e7d6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:48.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.106+0000 7f9855e00640 1 --2- 172.21.15.39:0/736042072 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e47e7d22b0 0x55e47e7d6db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:48.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.106+0000 7f9855e00640 1 -- 172.21.15.39:0/736042072 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e47e7d7420 msgr2=0x55e47e7d98b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:48.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.106+0000 7f9855e00640 1 --2- 172.21.15.39:0/736042072 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e47e7d7420 0x55e47e7d98b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:48.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.106+0000 7f9855e00640 1 -- 172.21.15.39:0/736042072 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e47e7001f0 con 0x55e47e7d18c0 2024-03-18T16:16:48.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.106+0000 7f984d400640 1 --2- 172.21.15.39:0/736042072 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e47e7d22b0 0x55e47e7d6db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:16:48.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.106+0000 7f9856800640 1 --2- 172.21.15.39:0/736042072 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e47e7d7420 0x55e47e7d98b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:16:48.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.106+0000 7f9855e00640 1 --2- 172.21.15.39:0/736042072 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e47e7d18c0 0x55e47e7d1ce0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f9848097a00 tx=0x7f98480086f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=327d3734ab7289c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:48.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.107+0000 7f984d400640 1 --2- 172.21.15.39:0/42628867 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f983c0a28d0 0x7f983c0a4d90 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f9840009830 tx=0x7f9840009540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:48.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.107+0000 7f9847400640 1 -- 172.21.15.39:0/736042072 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9848099330 con 0x55e47e7d18c0 2024-03-18T16:16:48.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.107+0000 7f9847400640 1 -- 172.21.15.39:0/736042072 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f98480976a0 con 0x55e47e7d18c0 2024-03-18T16:16:48.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.107+0000 7f9847400640 1 -- 172.21.15.39:0/736042072 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9848095ba0 con 0x55e47e7d18c0 2024-03-18T16:16:48.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.108+0000 7f985fba3740 1 -- 172.21.15.39:0/736042072 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e47e7d18c0 msgr2=0x55e47e7d1ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:48.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.108+0000 7f985fba3740 1 --2- 172.21.15.39:0/736042072 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e47e7d18c0 0x55e47e7d1ce0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f9848097a00 tx=0x7f98480086f0 comp rx=0 tx=0).stop 2024-03-18T16:16:48.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.109+0000 7f985fba3740 1 -- 172.21.15.39:0/736042072 shutdown_connections 2024-03-18T16:16:48.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.109+0000 7f985fba3740 1 --2- 172.21.15.39:0/736042072 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e47e7d22b0 0x55e47e7d6db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:48.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.109+0000 7f985fba3740 1 --2- 172.21.15.39:0/736042072 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e47e7d7420 0x55e47e7d98b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:48.110 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.109+0000 7f985fba3740 1 --2- 172.21.15.39:0/736042072 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e47e7d18c0 0x55e47e7d1ce0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:48.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.109+0000 7f985fba3740 1 -- 172.21.15.39:0/736042072 >> 172.21.15.39:0/736042072 conn(0x55e47e7cce30 msgr2=0x55e47e7cd230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:48.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.109+0000 7f985fba3740 1 -- 172.21.15.39:0/736042072 shutdown_connections 2024-03-18T16:16:48.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.109+0000 7f985fba3740 1 -- 172.21.15.39:0/736042072 wait complete. 2024-03-18T16:16:48.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.109+0000 7f985fba3740 1 -- start start 2024-03-18T16:16:48.111 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.110+0000 7f985fba3740 1 Processor -- start 2024-03-18T16:16:48.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.110+0000 7f985fba3740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e47e7d18c0 0x55e47e7dbc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:48.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.110+0000 7f985fba3740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e47e7d22b0 0x55e47e7dc1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:48.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.110+0000 7f985fba3740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e47e7d7420 0x55e47e7e31b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:48.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.110+0000 7f985fba3740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55e47e6f2110 con 0x55e47e7d22b0 2024-03-18T16:16:48.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.110+0000 7f985fba3740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55e47e7324f0 con 0x55e47e7d18c0 2024-03-18T16:16:48.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.110+0000 7f985fba3740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55e47e7da3d0 con 0x55e47e7d7420 2024-03-18T16:16:48.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.111+0000 7f985fba3740 10 client.0 _ll_register_callbacks cb 0x55e47e6dfb70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:16:48.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.111+0000 7f9855e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e47e7d18c0 0x55e47e7dbc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:48.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.111+0000 7f9855e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e47e7d18c0 0x55e47e7dbc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:46224/0 (socket says 172.21.15.39:46224) 2024-03-18T16:16:48.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.111+0000 7f9855e00640 1 -- 172.21.15.39:0/873181087 learned_addr learned my addr 172.21.15.39:0/873181087 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:16:48.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.111+0000 7f9856800640 1 --2- 172.21.15.39:0/873181087 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e47e7d7420 0x55e47e7e31b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:48.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.112+0000 7f984d400640 1 --2- 172.21.15.39:0/873181087 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e47e7d22b0 0x55e47e7dc1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:48.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.112+0000 7f9856800640 1 -- 172.21.15.39:0/873181087 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e47e7d18c0 msgr2=0x55e47e7dbc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:48.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.112+0000 7f9856800640 1 --2- 172.21.15.39:0/873181087 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e47e7d18c0 0x55e47e7dbc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:48.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.112+0000 7f9856800640 1 -- 172.21.15.39:0/873181087 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e47e7d22b0 msgr2=0x55e47e7dc1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:48.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.112+0000 7f9856800640 1 --2- 172.21.15.39:0/873181087 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e47e7d22b0 0x55e47e7dc1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:48.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.112+0000 7f9856800640 1 -- 172.21.15.39:0/873181087 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e47e44a3f0 con 0x55e47e7d7420 2024-03-18T16:16:48.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.113+0000 7f9855e00640 1 --2- 172.21.15.39:0/873181087 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e47e7d18c0 0x55e47e7dbc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:16:48.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.113+0000 7f9856800640 1 --2- 172.21.15.39:0/873181087 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e47e7d7420 0x55e47e7e31b0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f9850005fd0 tx=0x7f985000bd90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d1e19a529d323b1f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:48.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.113+0000 7f983be00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:16:48.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.113+0000 7f983be00640 1 -- 172.21.15.39:0/873181087 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9850003ea0 con 0x55e47e7d7420 2024-03-18T16:16:48.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.113+0000 7f983be00640 1 -- 172.21.15.39:0/873181087 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f985001b670 con 0x55e47e7d7420 2024-03-18T16:16:48.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.113+0000 7f985fba3740 1 -- 172.21.15.39:0/873181087 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f98480976a0 con 0x55e47e7d7420 2024-03-18T16:16:48.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.114+0000 7f983be00640 1 -- 172.21.15.39:0/873181087 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f9850024050 con 0x55e47e7d7420 2024-03-18T16:16:48.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.114+0000 7f985fba3740 10 client.26995 fetch_fsmap learned FSMap version 44 2024-03-18T16:16:48.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.114+0000 7f985fba3740 1 -- 172.21.15.39:0/873181087 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x55e47e5147d0 con 0x55e47e7d7420 2024-03-18T16:16:48.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.114+0000 7f983be00640 1 -- 172.21.15.39:0/873181087 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f98480976a0 con 0x55e47e7d7420 2024-03-18T16:16:48.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.114+0000 7f983be00640 1 -- 172.21.15.39:0/873181087 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7f9850012a90 con 0x55e47e7d7420 2024-03-18T16:16:48.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.115+0000 7f985fba3740 10 client.26995 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:16:48.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.115+0000 7f985fba3740 10 client.26995 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:16:48.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.115+0000 7f985fba3740 20 client.26995 populate_metadata read hostname 'smithi039' 2024-03-18T16:16:48.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.115+0000 7f985fba3740 1 --2- 172.21.15.39:0/873181087 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55e47e7e7a40 0x55e47e7e9e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:48.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.115+0000 7f985fba3740 4 client.26995 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:16:48.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.115+0000 7f985fba3740 1 -- 172.21.15.39:0/873181087 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e47e743990 con 0x55e47e7e7a40 2024-03-18T16:16:48.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.115+0000 7f9855e00640 1 --2- 172.21.15.39:0/873181087 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55e47e7e7a40 0x55e47e7e9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:48.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.116+0000 7f9855e00640 1 --2- 172.21.15.39:0/873181087 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55e47e7e7a40 0x55e47e7e9e90 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=325c90a3ecc7fc5 server_cookie=bb4be365794f7926 in_seq=0 out_seq=0 2024-03-18T16:16:48.118 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.116+0000 7f983be00640 10 client.26995 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:16:48.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.117+0000 7f983be00640 1 -- 172.21.15.39:0/873181087 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e47e743990 con 0x55e47e7e7a40 2024-03-18T16:16:48.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.117+0000 7f983be00640 10 client.26995 handle_command_reply: tid=0 2024-03-18T16:16:48.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.117+0000 7f985fba3740 1 client.26995 shutdown 2024-03-18T16:16:48.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.117+0000 7f985fba3740 1 -- 172.21.15.39:0/873181087 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e47e7d7420 msgr2=0x55e47e7e31b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:48.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.117+0000 7f985fba3740 1 --2- 172.21.15.39:0/873181087 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e47e7d7420 0x55e47e7e31b0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f9850005fd0 tx=0x7f985000bd90 comp rx=0 tx=0).stop 2024-03-18T16:16:48.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.118+0000 7f985fba3740 1 -- 172.21.15.39:0/873181087 shutdown_connections 2024-03-18T16:16:48.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.118+0000 7f985fba3740 1 --2- 172.21.15.39:0/873181087 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55e47e7e7a40 0x55e47e7e9e90 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:48.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.118+0000 7f985fba3740 1 --2- 172.21.15.39:0/873181087 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e47e7d7420 0x55e47e7e31b0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:48.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.118+0000 7f985fba3740 1 --2- 172.21.15.39:0/873181087 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e47e7d22b0 0x55e47e7dc1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:48.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.118+0000 7f985fba3740 1 --2- 172.21.15.39:0/873181087 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e47e7d18c0 0x55e47e7dbc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:48.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.118+0000 7f985fba3740 1 -- 172.21.15.39:0/873181087 >> 172.21.15.39:0/873181087 conn(0x55e47e7cce30 msgr2=0x55e47e7cc1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:48.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.119+0000 7f983be00640 0 client.26995 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:16:48.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.119+0000 7f985fba3740 1 -- 172.21.15.39:0/873181087 shutdown_connections 2024-03-18T16:16:48.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.119+0000 7f985fba3740 1 -- 172.21.15.39:0/873181087 wait complete. 2024-03-18T16:16:48.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.119+0000 7f985fba3740 20 client.26995 trim_cache size 0 max 16384 2024-03-18T16:16:48.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.141+0000 7f985fba3740 1 Processor -- start 2024-03-18T16:16:48.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.141+0000 7f985fba3740 1 -- start start 2024-03-18T16:16:48.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.142+0000 7f985fba3740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e47e7d18c0 0x55e47e7dbc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:48.143 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.142+0000 7f985fba3740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e47e7d22b0 0x55e47e7e3170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:48.144 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.142+0000 7f985fba3740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e47e7d7420 0x55e47e7dc1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:48.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.142+0000 7f985fba3740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f98581b93c0 con 0x55e47e7d22b0 2024-03-18T16:16:48.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.142+0000 7f985fba3740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f98581baf00 con 0x55e47e7d7420 2024-03-18T16:16:48.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.142+0000 7f985fba3740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f98581b9a90 con 0x55e47e7d18c0 2024-03-18T16:16:48.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.142+0000 7f984d400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e47e7d22b0 0x55e47e7e3170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:48.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.142+0000 7f9856800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e47e7d7420 0x55e47e7dc1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:48.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.143+0000 7f9856800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e47e7d7420 0x55e47e7dc1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:46234/0 (socket says 172.21.15.39:46234) 2024-03-18T16:16:48.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.143+0000 7f9856800640 1 -- 172.21.15.39:0/2144841880 learned_addr learned my addr 172.21.15.39:0/2144841880 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:16:48.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.143+0000 7f9856800640 1 -- 172.21.15.39:0/2144841880 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e47e7d18c0 msgr2=0x55e47e7dbc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:48.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.143+0000 7f9856800640 1 --2- 172.21.15.39:0/2144841880 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e47e7d18c0 0x55e47e7dbc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:48.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.143+0000 7f9856800640 1 -- 172.21.15.39:0/2144841880 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e47e7d22b0 msgr2=0x55e47e7e3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:48.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.143+0000 7f9856800640 1 --2- 172.21.15.39:0/2144841880 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e47e7d22b0 0x55e47e7e3170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:48.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.143+0000 7f9856800640 1 -- 172.21.15.39:0/2144841880 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e47e7c8620 con 0x55e47e7d7420 2024-03-18T16:16:48.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.143+0000 7f984d400640 1 --2- 172.21.15.39:0/2144841880 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e47e7d22b0 0x55e47e7e3170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:16:48.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.143+0000 7f9856800640 1 --2- 172.21.15.39:0/2144841880 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e47e7d7420 0x55e47e7dc1b0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f985000c010 tx=0x7f98500044c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=73b0ae6b5a552c7d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:48.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.144+0000 7f9846a00640 1 -- 172.21.15.39:0/2144841880 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9850003ea0 con 0x55e47e7d7420 2024-03-18T16:16:48.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.144+0000 7f9846a00640 1 -- 172.21.15.39:0/2144841880 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f985002c020 con 0x55e47e7d7420 2024-03-18T16:16:48.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.144+0000 7f9846a00640 1 -- 172.21.15.39:0/2144841880 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9850012930 con 0x55e47e7d7420 2024-03-18T16:16:48.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.144+0000 7f985fba3740 1 -- 172.21.15.39:0/2144841880 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e47e7d7420 msgr2=0x55e47e7dc1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:48.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.144+0000 7f985fba3740 1 --2- 172.21.15.39:0/2144841880 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e47e7d7420 0x55e47e7dc1b0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f985000c010 tx=0x7f98500044c0 comp rx=0 tx=0).stop 2024-03-18T16:16:48.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.144+0000 7f985fba3740 1 -- 172.21.15.39:0/2144841880 shutdown_connections 2024-03-18T16:16:48.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.144+0000 7f985fba3740 1 --2- 172.21.15.39:0/2144841880 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e47e7d7420 0x55e47e7dc1b0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:48.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.144+0000 7f985fba3740 1 --2- 172.21.15.39:0/2144841880 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e47e7d22b0 0x55e47e7e3170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:48.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.144+0000 7f985fba3740 1 --2- 172.21.15.39:0/2144841880 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e47e7d18c0 0x55e47e7dbc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:48.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.144+0000 7f985fba3740 1 -- 172.21.15.39:0/2144841880 >> 172.21.15.39:0/2144841880 conn(0x55e47e7cce30 msgr2=0x55e47e7cd210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:48.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.145+0000 7f985fba3740 1 -- 172.21.15.39:0/2144841880 shutdown_connections 2024-03-18T16:16:48.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.145+0000 7f985fba3740 1 -- 172.21.15.39:0/2144841880 wait complete. 2024-03-18T16:16:48.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.145+0000 7f985fba3740 1 -- start start 2024-03-18T16:16:48.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.145+0000 7f985fba3740 1 Processor -- start 2024-03-18T16:16:48.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.146+0000 7f985fba3740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e47e7d18c0 0x55e47e7e0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:48.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.146+0000 7f985fba3740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e47e7d22b0 0x55e47e7e08c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:48.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.146+0000 7f985fba3740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e47e7d7420 0x55e47e7e18d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:48.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.146+0000 7f985fba3740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f98581ba830 con 0x55e47e7d22b0 2024-03-18T16:16:48.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.146+0000 7f985fba3740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f98581ba160 con 0x55e47e7d18c0 2024-03-18T16:16:48.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.146+0000 7f985fba3740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f98581b73a0 con 0x55e47e7d7420 2024-03-18T16:16:48.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.146+0000 7f985fba3740 10 client.0 _ll_register_callbacks cb 0x7f9850004120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:16:48.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.146+0000 7f9856800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e47e7d7420 0x55e47e7e18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:48.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.146+0000 7f9856800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e47e7d7420 0x55e47e7e18d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:49752/0 (socket says 172.21.15.39:49752) 2024-03-18T16:16:48.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.146+0000 7f9856800640 1 -- 172.21.15.39:0/2708339506 learned_addr learned my addr 172.21.15.39:0/2708339506 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:16:48.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.146+0000 7f9856800640 1 -- 172.21.15.39:0/2708339506 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e47e7d18c0 msgr2=0x55e47e7e0380 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T16:16:48.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.146+0000 7f9856800640 1 --2- 172.21.15.39:0/2708339506 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e47e7d18c0 0x55e47e7e0380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:48.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.146+0000 7f9856800640 1 -- 172.21.15.39:0/2708339506 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e47e7d22b0 msgr2=0x55e47e7e08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:48.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.146+0000 7f9856800640 1 --2- 172.21.15.39:0/2708339506 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e47e7d22b0 0x55e47e7e08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:48.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.147+0000 7f9856800640 1 -- 172.21.15.39:0/2708339506 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e47e7e1ec0 con 0x55e47e7d7420 2024-03-18T16:16:48.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.147+0000 7f9856800640 1 --2- 172.21.15.39:0/2708339506 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e47e7d7420 0x55e47e7e18d0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f9850012630 tx=0x7f985000bd90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8e7aca1984ebd590 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:48.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.147+0000 7f983be00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:16:48.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.147+0000 7f983be00640 1 -- 172.21.15.39:0/2708339506 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9850036060 con 0x55e47e7d7420 2024-03-18T16:16:48.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.147+0000 7f983be00640 1 -- 172.21.15.39:0/2708339506 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f9850021040 con 0x55e47e7d7420 2024-03-18T16:16:48.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.148+0000 7f985fba3740 1 -- 172.21.15.39:0/2708339506 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f985002c020 con 0x55e47e7d7420 2024-03-18T16:16:48.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.148+0000 7f983be00640 1 -- 172.21.15.39:0/2708339506 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f9850012b80 con 0x55e47e7d7420 2024-03-18T16:16:48.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.148+0000 7f985fba3740 10 client.26998 fetch_fsmap learned FSMap version 44 2024-03-18T16:16:48.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.148+0000 7f985fba3740 1 -- 172.21.15.39:0/2708339506 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x55e47e7e31c0 con 0x55e47e7d7420 2024-03-18T16:16:48.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.149+0000 7f983be00640 1 -- 172.21.15.39:0/2708339506 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f985002c020 con 0x55e47e7d7420 2024-03-18T16:16:48.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.149+0000 7f983be00640 1 -- 172.21.15.39:0/2708339506 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7f9850004bf0 con 0x55e47e7d7420 2024-03-18T16:16:48.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.149+0000 7f985fba3740 10 client.26998 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:16:48.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.149+0000 7f985fba3740 10 client.26998 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:16:48.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.149+0000 7f985fba3740 20 client.26998 populate_metadata read hostname 'smithi039' 2024-03-18T16:16:48.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.149+0000 7f985fba3740 1 --2- 172.21.15.39:0/2708339506 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55e47e84e1a0 0x55e47e8505c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:48.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.149+0000 7f985fba3740 4 client.26998 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:16:48.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.149+0000 7f985fba3740 1 -- 172.21.15.39:0/2708339506 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55e47e73dce0 con 0x55e47e84e1a0 2024-03-18T16:16:48.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.149+0000 7f9855e00640 1 --2- 172.21.15.39:0/2708339506 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55e47e84e1a0 0x55e47e8505c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:48.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.151+0000 7f983be00640 10 client.26998 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:16:48.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.151+0000 7f9855e00640 1 --2- 172.21.15.39:0/2708339506 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55e47e84e1a0 0x55e47e8505c0 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=62966499f4f54e3d server_cookie=6e0ed464e6facd55 in_seq=0 out_seq=0 2024-03-18T16:16:48.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.151+0000 7f983be00640 1 -- 172.21.15.39:0/2708339506 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55e47e73dce0 con 0x55e47e84e1a0 2024-03-18T16:16:48.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.151+0000 7f983be00640 10 client.26998 handle_command_reply: tid=0 2024-03-18T16:16:48.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.151+0000 7f985fba3740 1 client.26998 shutdown 2024-03-18T16:16:48.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.151+0000 7f985fba3740 1 -- 172.21.15.39:0/2708339506 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e47e7d7420 msgr2=0x55e47e7e18d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:48.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.151+0000 7f985fba3740 1 --2- 172.21.15.39:0/2708339506 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e47e7d7420 0x55e47e7e18d0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f9850012630 tx=0x7f985000bd90 comp rx=0 tx=0).stop 2024-03-18T16:16:48.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.152+0000 7f985fba3740 1 -- 172.21.15.39:0/2708339506 shutdown_connections 2024-03-18T16:16:48.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.152+0000 7f985fba3740 1 --2- 172.21.15.39:0/2708339506 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55e47e84e1a0 0x55e47e8505c0 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:48.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.152+0000 7f983be00640 0 client.26998 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:16:48.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.152+0000 7f985fba3740 1 --2- 172.21.15.39:0/2708339506 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e47e7d7420 0x55e47e7e18d0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:48.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.152+0000 7f985fba3740 1 --2- 172.21.15.39:0/2708339506 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e47e7d22b0 0x55e47e7e08c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:48.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.152+0000 7f985fba3740 1 --2- 172.21.15.39:0/2708339506 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e47e7d18c0 0x55e47e7e0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:48.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.152+0000 7f985fba3740 1 -- 172.21.15.39:0/2708339506 >> 172.21.15.39:0/2708339506 conn(0x55e47e7cce30 msgr2=0x55e47e7dbc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:48.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.152+0000 7f985fba3740 1 -- 172.21.15.39:0/2708339506 shutdown_connections 2024-03-18T16:16:48.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.152+0000 7f985fba3740 1 -- 172.21.15.39:0/2708339506 wait complete. 2024-03-18T16:16:48.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.152+0000 7f985fba3740 20 client.26998 trim_cache size 0 max 16384 2024-03-18T16:16:48.155 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:16:48.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.153+0000 7f985ce00640 1 -- 172.21.15.39:0/42628867 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f983c0a28d0 msgr2=0x7f983c0a4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:48.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.153+0000 7f985ce00640 1 --2- 172.21.15.39:0/42628867 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f983c0a28d0 0x7f983c0a4d90 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f9840009830 tx=0x7f9840009540 comp rx=0 tx=0).stop 2024-03-18T16:16:48.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.153+0000 7f985ce00640 1 -- 172.21.15.39:0/42628867 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9858108a50 msgr2=0x7f98581a69a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:48.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.153+0000 7f985ce00640 1 --2- 172.21.15.39:0/42628867 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9858108a50 0x7f98581a69a0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f984800a6c0 tx=0x7f984800ab80 comp rx=0 tx=0).stop 2024-03-18T16:16:48.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.153+0000 7f985ce00640 1 -- 172.21.15.39:0/42628867 shutdown_connections 2024-03-18T16:16:48.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.153+0000 7f985ce00640 1 --2- 172.21.15.39:0/42628867 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f983c0a28d0 0x7f983c0a4d90 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:48.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.153+0000 7f985ce00640 1 --2- 172.21.15.39:0/42628867 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f985810de90 0x7f98581adf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:48.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.153+0000 7f985ce00640 1 --2- 172.21.15.39:0/42628867 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f985810b410 0x7f98581a6ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:48.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.153+0000 7f985ce00640 1 --2- 172.21.15.39:0/42628867 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9858108a50 0x7f98581a69a0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:48.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.153+0000 7f985ce00640 1 -- 172.21.15.39:0/42628867 >> 172.21.15.39:0/42628867 conn(0x7f985806e860 msgr2=0x7f985810bec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:48.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.154+0000 7f985ce00640 1 -- 172.21.15.39:0/42628867 shutdown_connections 2024-03-18T16:16:48.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:48.154+0000 7f985ce00640 1 -- 172.21.15.39:0/42628867 wait complete. 2024-03-18T16:16:49.180 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-18T16:16:49.181 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T16:16:49.181 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-18T16:16:49.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.459+0000 7f8d7d800640 1 -- 172.21.15.39:0/2225299111 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8d78104c10 msgr2=0x7f8d78105090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:49.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.459+0000 7f8d7d800640 1 --2- 172.21.15.39:0/2225299111 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8d78104c10 0x7f8d78105090 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f8d64009900 tx=0x7f8d6402bc40 comp rx=0 tx=0).stop 2024-03-18T16:16:49.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.459+0000 7f8d7d800640 1 -- 172.21.15.39:0/2225299111 shutdown_connections 2024-03-18T16:16:49.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.459+0000 7f8d7d800640 1 --2- 172.21.15.39:0/2225299111 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8d781055d0 0x7f8d7810be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:49.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.459+0000 7f8d7d800640 1 --2- 172.21.15.39:0/2225299111 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8d78104c10 0x7f8d78105090 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:49.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.459+0000 7f8d7d800640 1 --2- 172.21.15.39:0/2225299111 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8d78103a20 0x7f8d78103e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:49.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.459+0000 7f8d7d800640 1 -- 172.21.15.39:0/2225299111 >> 172.21.15.39:0/2225299111 conn(0x7f8d780fef90 msgr2=0x7f8d781013d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:49.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.459+0000 7f8d7d800640 1 -- 172.21.15.39:0/2225299111 shutdown_connections 2024-03-18T16:16:49.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.459+0000 7f8d7d800640 1 -- 172.21.15.39:0/2225299111 wait complete. 2024-03-18T16:16:49.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.460+0000 7f8d7d800640 1 Processor -- start 2024-03-18T16:16:49.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.461+0000 7f8d7d800640 1 -- start start 2024-03-18T16:16:49.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.461+0000 7f8d7d800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8d78103a20 0x7f8d78110bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:49.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.461+0000 7f8d7d800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8d78104c10 0x7f8d78111100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:49.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.461+0000 7f8d7d800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8d781055d0 0x7f8d78114150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:49.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.461+0000 7f8d7d800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f8d78077a00 con 0x7f8d78104c10 2024-03-18T16:16:49.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.461+0000 7f8d7d800640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f8d78077880 con 0x7f8d78103a20 2024-03-18T16:16:49.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.461+0000 7f8d7d800640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f8d78077b80 con 0x7f8d781055d0 2024-03-18T16:16:49.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.461+0000 7f8d77400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8d781055d0 0x7f8d78114150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:49.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.461+0000 7f8d77400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8d781055d0 0x7f8d78114150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:49774/0 (socket says 172.21.15.39:49774) 2024-03-18T16:16:49.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.461+0000 7f8d77400640 1 -- 172.21.15.39:0/1162750350 learned_addr learned my addr 172.21.15.39:0/1162750350 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:16:49.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.462+0000 7f8d76000640 1 --2- 172.21.15.39:0/1162750350 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8d78104c10 0x7f8d78111100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:49.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.462+0000 7f8d76a00640 1 --2- 172.21.15.39:0/1162750350 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8d78103a20 0x7f8d78110bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:49.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.463+0000 7f8d76000640 1 -- 172.21.15.39:0/1162750350 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8d781055d0 msgr2=0x7f8d78114150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:49.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.463+0000 7f8d76000640 1 --2- 172.21.15.39:0/1162750350 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8d781055d0 0x7f8d78114150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:49.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.463+0000 7f8d76000640 1 -- 172.21.15.39:0/1162750350 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8d78103a20 msgr2=0x7f8d78110bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:49.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.463+0000 7f8d76000640 1 --2- 172.21.15.39:0/1162750350 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8d78103a20 0x7f8d78110bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:49.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.463+0000 7f8d76000640 1 -- 172.21.15.39:0/1162750350 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d781147c0 con 0x7f8d78104c10 2024-03-18T16:16:49.465 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.463+0000 7f8d76000640 1 --2- 172.21.15.39:0/1162750350 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8d78104c10 0x7f8d78111100 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f8d64009230 tx=0x7f8d64005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:49.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.464+0000 7f8d6f400640 1 -- 172.21.15.39:0/1162750350 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d64041070 con 0x7f8d78104c10 2024-03-18T16:16:49.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.465+0000 7f8d7d800640 1 -- 172.21.15.39:0/1162750350 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d78114a50 con 0x7f8d78104c10 2024-03-18T16:16:49.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.465+0000 7f8d6f400640 1 -- 172.21.15.39:0/1162750350 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d640054e0 con 0x7f8d78104c10 2024-03-18T16:16:49.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.465+0000 7f8d7d800640 1 -- 172.21.15.39:0/1162750350 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d78114f90 con 0x7f8d78104c10 2024-03-18T16:16:49.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.465+0000 7f8d6f400640 1 -- 172.21.15.39:0/1162750350 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d64004c80 con 0x7f8d78104c10 2024-03-18T16:16:49.470 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.468+0000 7f8d8061e740 1 Processor -- start 2024-03-18T16:16:49.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.469+0000 7f8d8061e740 1 -- start start 2024-03-18T16:16:49.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.469+0000 7f8d8061e740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55aa0fe1b7e0 0x55aa0fe1bc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:49.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.469+0000 7f8d8061e740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55aa0fe1c1d0 0x55aa0fe20cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:49.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.469+0000 7f8d8061e740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55aa0fe21340 0x55aa0fe237d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:49.471 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.469+0000 7f8d8061e740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55aa0fd7ef80 con 0x55aa0fe21340 2024-03-18T16:16:49.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.469+0000 7f8d8061e740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55aa0fd27ad0 con 0x55aa0fe1b7e0 2024-03-18T16:16:49.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.469+0000 7f8d8061e740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55aa0fd7b020 con 0x55aa0fe1c1d0 2024-03-18T16:16:49.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.469+0000 7f8d76a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55aa0fe1c1d0 0x55aa0fe20cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:49.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.470+0000 7f8d77400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55aa0fe1b7e0 0x55aa0fe1bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:49.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.470+0000 7f8d77400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55aa0fe1b7e0 0x55aa0fe1bc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:46266/0 (socket says 172.21.15.39:46266) 2024-03-18T16:16:49.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.470+0000 7f8d77400640 1 -- 172.21.15.39:0/1681848535 learned_addr learned my addr 172.21.15.39:0/1681848535 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:16:49.472 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.470+0000 7f8d76a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55aa0fe1c1d0 0x55aa0fe20cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:49788/0 (socket says 172.21.15.39:49788) 2024-03-18T16:16:49.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.470+0000 7f8d77400640 1 -- 172.21.15.39:0/1681848535 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55aa0fe1c1d0 msgr2=0x55aa0fe20cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:49.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.470+0000 7f8d77400640 1 --2- 172.21.15.39:0/1681848535 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55aa0fe1c1d0 0x55aa0fe20cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:49.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.470+0000 7f8d77400640 1 -- 172.21.15.39:0/1681848535 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55aa0fe21340 msgr2=0x55aa0fe237d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:16:49.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.470+0000 7f8d77400640 1 --2- 172.21.15.39:0/1681848535 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55aa0fe21340 0x55aa0fe237d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:49.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.470+0000 7f8d77400640 1 -- 172.21.15.39:0/1681848535 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aa0fd3f6f0 con 0x55aa0fe1b7e0 2024-03-18T16:16:49.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.471+0000 7f8d77400640 1 --2- 172.21.15.39:0/1681848535 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55aa0fe1b7e0 0x55aa0fe1bc00 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f8d70007f40 tx=0x7f8d7000f850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e4859eca6ee542a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:49.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.471+0000 7f8d63e00640 1 -- 172.21.15.39:0/1681848535 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d70004d20 con 0x55aa0fe1b7e0 2024-03-18T16:16:49.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.471+0000 7f8d63e00640 1 -- 172.21.15.39:0/1681848535 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d70010be0 con 0x55aa0fe1b7e0 2024-03-18T16:16:49.473 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.471+0000 7f8d63e00640 1 -- 172.21.15.39:0/1681848535 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d70005790 con 0x55aa0fe1b7e0 2024-03-18T16:16:49.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.473+0000 7f8d8061e740 1 -- 172.21.15.39:0/1681848535 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55aa0fe1b7e0 msgr2=0x55aa0fe1bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:49.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.473+0000 7f8d8061e740 1 --2- 172.21.15.39:0/1681848535 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55aa0fe1b7e0 0x55aa0fe1bc00 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f8d70007f40 tx=0x7f8d7000f850 comp rx=0 tx=0).stop 2024-03-18T16:16:49.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.473+0000 7f8d6f400640 1 -- 172.21.15.39:0/1162750350 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f8d64044070 con 0x7f8d78104c10 2024-03-18T16:16:49.475 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.474+0000 7f8d8061e740 1 -- 172.21.15.39:0/1681848535 shutdown_connections 2024-03-18T16:16:49.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.474+0000 7f8d8061e740 1 --2- 172.21.15.39:0/1681848535 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55aa0fe21340 0x55aa0fe237d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:49.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.474+0000 7f8d8061e740 1 --2- 172.21.15.39:0/1681848535 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55aa0fe1c1d0 0x55aa0fe20cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:49.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.474+0000 7f8d8061e740 1 --2- 172.21.15.39:0/1681848535 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55aa0fe1b7e0 0x55aa0fe1bc00 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:49.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.474+0000 7f8d8061e740 1 -- 172.21.15.39:0/1681848535 >> 172.21.15.39:0/1681848535 conn(0x55aa0fde3930 msgr2=0x55aa0fe17150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:49.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.474+0000 7f8d8061e740 1 -- 172.21.15.39:0/1681848535 shutdown_connections 2024-03-18T16:16:49.476 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.474+0000 7f8d8061e740 1 -- 172.21.15.39:0/1681848535 wait complete. 2024-03-18T16:16:49.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.475+0000 7f8d8061e740 1 -- start start 2024-03-18T16:16:49.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.475+0000 7f8d8061e740 1 Processor -- start 2024-03-18T16:16:49.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.476+0000 7f8d8061e740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55aa0fe1b7e0 0x55aa0fe25b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:49.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.476+0000 7f8d8061e740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55aa0fe1c1d0 0x55aa0fe26050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:49.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.476+0000 7f8d8061e740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55aa0fe21340 0x55aa0fe2d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:49.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.476+0000 7f8d8061e740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55aa0fe242f0 con 0x55aa0fe1b7e0 2024-03-18T16:16:49.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.476+0000 7f8d8061e740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55aa0fd3af30 con 0x55aa0fe21340 2024-03-18T16:16:49.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.476+0000 7f8d8061e740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55aa0fe24470 con 0x55aa0fe1c1d0 2024-03-18T16:16:49.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.476+0000 7f8d8061e740 10 client.0 _ll_register_callbacks cb 0x55aa0fd28bc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:16:49.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.476+0000 7f8d76a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55aa0fe1c1d0 0x55aa0fe26050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:49.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.476+0000 7f8d76a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55aa0fe1c1d0 0x55aa0fe26050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:49798/0 (socket says 172.21.15.39:49798) 2024-03-18T16:16:49.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.476+0000 7f8d76a00640 1 -- 172.21.15.39:0/2165271979 learned_addr learned my addr 172.21.15.39:0/2165271979 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:16:49.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.477+0000 7f8d77400640 1 --2- 172.21.15.39:0/2165271979 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55aa0fe1b7e0 0x55aa0fe25b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:49.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.477+0000 7f8d76000640 1 --2- 172.21.15.39:0/2165271979 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55aa0fe21340 0x55aa0fe2d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:49.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.477+0000 7f8d76a00640 1 -- 172.21.15.39:0/2165271979 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55aa0fe21340 msgr2=0x55aa0fe2d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:49.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.477+0000 7f8d76a00640 1 --2- 172.21.15.39:0/2165271979 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55aa0fe21340 0x55aa0fe2d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:49.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.478+0000 7f8d76a00640 1 -- 172.21.15.39:0/2165271979 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55aa0fe1b7e0 msgr2=0x55aa0fe25b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:49.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.478+0000 7f8d76a00640 1 --2- 172.21.15.39:0/2165271979 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55aa0fe1b7e0 0x55aa0fe25b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:49.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.478+0000 7f8d76a00640 1 -- 172.21.15.39:0/2165271979 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aa0fb5b7e0 con 0x55aa0fe1c1d0 2024-03-18T16:16:49.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.478+0000 7f8d77400640 1 --2- 172.21.15.39:0/2165271979 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55aa0fe1b7e0 0x55aa0fe25b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:16:49.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.478+0000 7f8d76000640 1 --2- 172.21.15.39:0/2165271979 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55aa0fe21340 0x55aa0fe2d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:16:49.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.479+0000 7f8d6f400640 1 --2- 172.21.15.39:0/1162750350 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f8d5c0a28b0 0x7f8d5c0a4d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:49.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.479+0000 7f8d76a00640 1 --2- 172.21.15.39:0/2165271979 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55aa0fe1c1d0 0x55aa0fe26050 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f8d6800b9d0 tx=0x7f8d6800be90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bde161f33848bad9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:49.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.479+0000 7f8d6f400640 1 -- 172.21.15.39:0/1162750350 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f8d640f0900 con 0x7f8d78104c10 2024-03-18T16:16:49.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.479+0000 7f8d60c00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:16:49.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.479+0000 7f8d60c00640 1 -- 172.21.15.39:0/2165271979 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d68003e80 con 0x55aa0fe1c1d0 2024-03-18T16:16:49.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.479+0000 7f8d60c00640 1 -- 172.21.15.39:0/2165271979 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f8d68011440 con 0x55aa0fe1c1d0 2024-03-18T16:16:49.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.480+0000 7f8d8061e740 1 -- 172.21.15.39:0/2165271979 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8d70010be0 con 0x55aa0fe1c1d0 2024-03-18T16:16:49.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.480+0000 7f8d77400640 1 --2- 172.21.15.39:0/1162750350 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f8d5c0a28b0 0x7f8d5c0a4d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:49.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.481+0000 7f8d60c00640 1 -- 172.21.15.39:0/2165271979 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f8d68011640 con 0x55aa0fe1c1d0 2024-03-18T16:16:49.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.481+0000 7f8d77400640 1 --2- 172.21.15.39:0/1162750350 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f8d5c0a28b0 0x7f8d5c0a4d70 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f8d70007cd0 tx=0x7f8d70012040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:49.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.481+0000 7f8d8061e740 10 client.27010 fetch_fsmap learned FSMap version 44 2024-03-18T16:16:49.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.481+0000 7f8d8061e740 1 -- 172.21.15.39:0/2165271979 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x55aa0fbbbf70 con 0x55aa0fe1c1d0 2024-03-18T16:16:49.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.482+0000 7f8d60c00640 1 -- 172.21.15.39:0/2165271979 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d70010be0 con 0x55aa0fe1c1d0 2024-03-18T16:16:49.483 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.482+0000 7f8d60c00640 1 -- 172.21.15.39:0/2165271979 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7f8d68025950 con 0x55aa0fe1c1d0 2024-03-18T16:16:49.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.482+0000 7f8d8061e740 10 client.27010 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:16:49.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.482+0000 7f8d8061e740 10 client.27010 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:16:49.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.482+0000 7f8d8061e740 20 client.27010 populate_metadata read hostname 'smithi039' 2024-03-18T16:16:49.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.483+0000 7f8d8061e740 1 --2- 172.21.15.39:0/2165271979 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55aa0fe31db0 0x55aa0fe34210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:49.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.483+0000 7f8d8061e740 4 client.27010 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:16:49.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.483+0000 7f8d8061e740 1 -- 172.21.15.39:0/2165271979 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55aa0fd8c4c0 con 0x55aa0fe31db0 2024-03-18T16:16:49.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.483+0000 7f8d76000640 1 --2- 172.21.15.39:0/2165271979 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55aa0fe31db0 0x55aa0fe34210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:49.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.484+0000 7f8d76000640 1 --2- 172.21.15.39:0/2165271979 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55aa0fe31db0 0x55aa0fe34210 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=12f4af19b054fef7 server_cookie=7a6d22d83c62a1e8 in_seq=0 out_seq=0 2024-03-18T16:16:49.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.484+0000 7f8d60c00640 10 client.27010 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:16:49.487 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.485+0000 7f8d60c00640 1 -- 172.21.15.39:0/2165271979 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55aa0fd8c4c0 con 0x55aa0fe31db0 2024-03-18T16:16:49.487 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.485+0000 7f8d60c00640 10 client.27010 handle_command_reply: tid=0 2024-03-18T16:16:49.487 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.485+0000 7f8d8061e740 1 client.27010 shutdown 2024-03-18T16:16:49.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.486+0000 7f8d8061e740 1 -- 172.21.15.39:0/2165271979 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55aa0fe1c1d0 msgr2=0x55aa0fe26050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:49.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.486+0000 7f8d8061e740 1 --2- 172.21.15.39:0/2165271979 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55aa0fe1c1d0 0x55aa0fe26050 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f8d6800b9d0 tx=0x7f8d6800be90 comp rx=0 tx=0).stop 2024-03-18T16:16:49.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.487+0000 7f8d8061e740 1 -- 172.21.15.39:0/2165271979 shutdown_connections 2024-03-18T16:16:49.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.487+0000 7f8d8061e740 1 --2- 172.21.15.39:0/2165271979 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55aa0fe31db0 0x55aa0fe34210 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:49.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.487+0000 7f8d8061e740 1 --2- 172.21.15.39:0/2165271979 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55aa0fe21340 0x55aa0fe2d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:49.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.487+0000 7f8d8061e740 1 --2- 172.21.15.39:0/2165271979 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55aa0fe1c1d0 0x55aa0fe26050 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:49.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.487+0000 7f8d8061e740 1 --2- 172.21.15.39:0/2165271979 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55aa0fe1b7e0 0x55aa0fe25b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:49.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.487+0000 7f8d8061e740 1 -- 172.21.15.39:0/2165271979 >> 172.21.15.39:0/2165271979 conn(0x55aa0fde3930 msgr2=0x55aa0fe16350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:49.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.487+0000 7f8d60c00640 0 client.27010 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:16:49.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.487+0000 7f8d8061e740 1 -- 172.21.15.39:0/2165271979 shutdown_connections 2024-03-18T16:16:49.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.487+0000 7f8d8061e740 1 -- 172.21.15.39:0/2165271979 wait complete. 2024-03-18T16:16:49.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.487+0000 7f8d8061e740 20 client.27010 trim_cache size 0 max 16384 2024-03-18T16:16:49.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.523+0000 7f8d8061e740 1 Processor -- start 2024-03-18T16:16:49.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.523+0000 7f8d8061e740 1 -- start start 2024-03-18T16:16:49.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.524+0000 7f8d8061e740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55aa0fe1b7e0 0x55aa0fe1d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:49.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.524+0000 7f8d8061e740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55aa0fe1c1d0 0x55aa0fe1da30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:49.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.524+0000 7f8d8061e740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55aa0fe21340 0x55aa0fe25b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:49.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.524+0000 7f8d8061e740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f8d781c1b30 con 0x55aa0fe1b7e0 2024-03-18T16:16:49.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.524+0000 7f8d8061e740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f8d781c3670 con 0x55aa0fe21340 2024-03-18T16:16:49.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.524+0000 7f8d8061e740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f8d781c2200 con 0x55aa0fe1c1d0 2024-03-18T16:16:49.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.524+0000 7f8d76000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55aa0fe1c1d0 0x55aa0fe1da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:49.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.524+0000 7f8d76a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55aa0fe1b7e0 0x55aa0fe1d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:49.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.524+0000 7f8d76000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55aa0fe1c1d0 0x55aa0fe1da30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:49810/0 (socket says 172.21.15.39:49810) 2024-03-18T16:16:49.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.524+0000 7f8d76000640 1 -- 172.21.15.39:0/3814186507 learned_addr learned my addr 172.21.15.39:0/3814186507 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:16:49.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.524+0000 7f8d77400640 1 --2- 172.21.15.39:0/3814186507 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55aa0fe21340 0x55aa0fe25b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:49.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.525+0000 7f8d76000640 1 -- 172.21.15.39:0/3814186507 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55aa0fe21340 msgr2=0x55aa0fe25b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:49.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.525+0000 7f8d76000640 1 --2- 172.21.15.39:0/3814186507 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55aa0fe21340 0x55aa0fe25b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:49.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.525+0000 7f8d76000640 1 -- 172.21.15.39:0/3814186507 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55aa0fe1b7e0 msgr2=0x55aa0fe1d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:49.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.525+0000 7f8d76000640 1 --2- 172.21.15.39:0/3814186507 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55aa0fe1b7e0 0x55aa0fe1d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:49.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.525+0000 7f8d76000640 1 -- 172.21.15.39:0/3814186507 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aa0fa53170 con 0x55aa0fe1c1d0 2024-03-18T16:16:49.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.525+0000 7f8d76a00640 1 --2- 172.21.15.39:0/3814186507 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55aa0fe1b7e0 0x55aa0fe1d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:16:49.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.525+0000 7f8d76000640 1 --2- 172.21.15.39:0/3814186507 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55aa0fe1c1d0 0x55aa0fe1da30 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f8d640b9420 tx=0x7f8d6402bc40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9ab4d7bac2ad774a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:49.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.526+0000 7f8d63400640 1 -- 172.21.15.39:0/3814186507 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d6404b040 con 0x55aa0fe1c1d0 2024-03-18T16:16:49.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.526+0000 7f8d63400640 1 -- 172.21.15.39:0/3814186507 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d640bcfc0 con 0x55aa0fe1c1d0 2024-03-18T16:16:49.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.526+0000 7f8d63400640 1 -- 172.21.15.39:0/3814186507 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d640036f0 con 0x55aa0fe1c1d0 2024-03-18T16:16:49.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.527+0000 7f8d8061e740 1 -- 172.21.15.39:0/3814186507 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55aa0fe1c1d0 msgr2=0x55aa0fe1da30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:49.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.527+0000 7f8d8061e740 1 --2- 172.21.15.39:0/3814186507 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55aa0fe1c1d0 0x55aa0fe1da30 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f8d640b9420 tx=0x7f8d6402bc40 comp rx=0 tx=0).stop 2024-03-18T16:16:49.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.528+0000 7f8d8061e740 1 -- 172.21.15.39:0/3814186507 shutdown_connections 2024-03-18T16:16:49.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.528+0000 7f8d8061e740 1 --2- 172.21.15.39:0/3814186507 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55aa0fe21340 0x55aa0fe25b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:49.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.528+0000 7f8d8061e740 1 --2- 172.21.15.39:0/3814186507 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55aa0fe1c1d0 0x55aa0fe1da30 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:49.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.528+0000 7f8d8061e740 1 --2- 172.21.15.39:0/3814186507 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55aa0fe1b7e0 0x55aa0fe1d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:49.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.528+0000 7f8d8061e740 1 -- 172.21.15.39:0/3814186507 >> 172.21.15.39:0/3814186507 conn(0x55aa0fde3930 msgr2=0x55aa0fe121b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:49.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.528+0000 7f8d8061e740 1 -- 172.21.15.39:0/3814186507 shutdown_connections 2024-03-18T16:16:49.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.528+0000 7f8d8061e740 1 -- 172.21.15.39:0/3814186507 wait complete. 2024-03-18T16:16:49.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.529+0000 7f8d8061e740 1 -- start start 2024-03-18T16:16:49.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.529+0000 7f8d8061e740 1 Processor -- start 2024-03-18T16:16:49.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.530+0000 7f8d8061e740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55aa0fe1b7e0 0x55aa0fe39030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:49.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.530+0000 7f8d8061e740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55aa0fe1c1d0 0x55aa0fe39570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:49.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.530+0000 7f8d8061e740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55aa0fe21340 0x55aa0fe96150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:49.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.530+0000 7f8d8061e740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f8d781c2fa0 con 0x55aa0fe21340 2024-03-18T16:16:49.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.530+0000 7f8d8061e740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f8d781c28d0 con 0x55aa0fe1c1d0 2024-03-18T16:16:49.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.530+0000 7f8d8061e740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f8d78072720 con 0x55aa0fe1b7e0 2024-03-18T16:16:49.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.530+0000 7f8d8061e740 10 client.0 _ll_register_callbacks cb 0x7f8d680042b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:16:49.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.530+0000 7f8d76000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55aa0fe1c1d0 0x55aa0fe39570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:49.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.530+0000 7f8d76000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55aa0fe1c1d0 0x55aa0fe39570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:46280/0 (socket says 172.21.15.39:46280) 2024-03-18T16:16:49.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.530+0000 7f8d76000640 1 -- 172.21.15.39:0/3245410589 learned_addr learned my addr 172.21.15.39:0/3245410589 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:16:49.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.531+0000 7f8d76a00640 1 --2- 172.21.15.39:0/3245410589 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55aa0fe1b7e0 0x55aa0fe39030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:49.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.531+0000 7f8d76000640 1 -- 172.21.15.39:0/3245410589 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55aa0fe1b7e0 msgr2=0x55aa0fe39030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:49.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.531+0000 7f8d76000640 1 --2- 172.21.15.39:0/3245410589 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55aa0fe1b7e0 0x55aa0fe39030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:49.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.531+0000 7f8d76000640 1 -- 172.21.15.39:0/3245410589 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55aa0fe21340 msgr2=0x55aa0fe96150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:49.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.531+0000 7f8d77400640 1 --2- 172.21.15.39:0/3245410589 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55aa0fe21340 0x55aa0fe96150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:49.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.531+0000 7f8d76000640 1 --2- 172.21.15.39:0/3245410589 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55aa0fe21340 0x55aa0fe96150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:49.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.531+0000 7f8d76000640 1 -- 172.21.15.39:0/3245410589 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aa0fe96710 con 0x55aa0fe1c1d0 2024-03-18T16:16:49.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.531+0000 7f8d77400640 1 --2- 172.21.15.39:0/3245410589 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55aa0fe21340 0x55aa0fe96150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:16:49.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.532+0000 7f8d76000640 1 --2- 172.21.15.39:0/3245410589 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55aa0fe1c1d0 0x55aa0fe39570 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f8d640ee3e0 tx=0x7f8d64000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=63579dea8d24536e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:49.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.532+0000 7f8d60c00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:16:49.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.532+0000 7f8d60c00640 1 -- 172.21.15.39:0/3245410589 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d640d9020 con 0x55aa0fe1c1d0 2024-03-18T16:16:49.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.532+0000 7f8d60c00640 1 -- 172.21.15.39:0/3245410589 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f8d640c5220 con 0x55aa0fe1c1d0 2024-03-18T16:16:49.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.532+0000 7f8d8061e740 1 -- 172.21.15.39:0/3245410589 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8d640bcfc0 con 0x55aa0fe1c1d0 2024-03-18T16:16:49.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.533+0000 7f8d60c00640 1 -- 172.21.15.39:0/3245410589 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f8d6404b040 con 0x55aa0fe1c1d0 2024-03-18T16:16:49.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.533+0000 7f8d8061e740 10 client.27041 fetch_fsmap learned FSMap version 44 2024-03-18T16:16:49.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.533+0000 7f8d8061e740 1 -- 172.21.15.39:0/3245410589 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x55aa0fe97ac0 con 0x55aa0fe1c1d0 2024-03-18T16:16:49.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.533+0000 7f8d60c00640 1 -- 172.21.15.39:0/3245410589 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d640a0020 con 0x55aa0fe1c1d0 2024-03-18T16:16:49.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.533+0000 7f8d60c00640 1 -- 172.21.15.39:0/3245410589 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7f8d640d2020 con 0x55aa0fe1c1d0 2024-03-18T16:16:49.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.533+0000 7f8d8061e740 10 client.27041 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:16:49.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.534+0000 7f8d8061e740 10 client.27041 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:16:49.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.534+0000 7f8d8061e740 20 client.27041 populate_metadata read hostname 'smithi039' 2024-03-18T16:16:49.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.534+0000 7f8d8061e740 1 --2- 172.21.15.39:0/3245410589 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55aa0fe98200 0x55aa0fe9a620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:49.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.534+0000 7f8d8061e740 4 client.27041 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-18T16:16:49.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.534+0000 7f8d8061e740 1 -- 172.21.15.39:0/3245410589 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55aa0fd86810 con 0x55aa0fe98200 2024-03-18T16:16:49.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.534+0000 7f8d76a00640 1 --2- 172.21.15.39:0/3245410589 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55aa0fe98200 0x55aa0fe9a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:49.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.538+0000 7f8d76a00640 1 --2- 172.21.15.39:0/3245410589 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55aa0fe98200 0x55aa0fe9a620 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cce1c4e141056eaf server_cookie=80fc371dd91e56e0 in_seq=0 out_seq=0 2024-03-18T16:16:49.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.538+0000 7f8d60c00640 10 client.27041 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:16:49.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.538+0000 7f8d60c00640 1 -- 172.21.15.39:0/3245410589 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55aa0fd86810 con 0x55aa0fe98200 2024-03-18T16:16:49.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.538+0000 7f8d60c00640 10 client.27041 handle_command_reply: tid=0 2024-03-18T16:16:49.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.539+0000 7f8d8061e740 1 client.27041 shutdown 2024-03-18T16:16:49.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.540+0000 7f8d8061e740 1 -- 172.21.15.39:0/3245410589 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55aa0fe1c1d0 msgr2=0x55aa0fe39570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:49.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.540+0000 7f8d8061e740 1 --2- 172.21.15.39:0/3245410589 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55aa0fe1c1d0 0x55aa0fe39570 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f8d640ee3e0 tx=0x7f8d64000bc0 comp rx=0 tx=0).stop 2024-03-18T16:16:49.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.541+0000 7f8d8061e740 1 -- 172.21.15.39:0/3245410589 shutdown_connections 2024-03-18T16:16:49.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.541+0000 7f8d8061e740 1 --2- 172.21.15.39:0/3245410589 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55aa0fe98200 0x55aa0fe9a620 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:49.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.541+0000 7f8d8061e740 1 --2- 172.21.15.39:0/3245410589 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55aa0fe1c1d0 0x55aa0fe39570 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:49.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.541+0000 7f8d8061e740 1 --2- 172.21.15.39:0/3245410589 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55aa0fe21340 0x55aa0fe96150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:49.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.541+0000 7f8d8061e740 1 --2- 172.21.15.39:0/3245410589 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55aa0fe1b7e0 0x55aa0fe39030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:49.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.541+0000 7f8d8061e740 1 -- 172.21.15.39:0/3245410589 >> 172.21.15.39:0/3245410589 conn(0x55aa0fde3930 msgr2=0x55aa0fe1d4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:49.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.541+0000 7f8d60c00640 0 client.27041 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:16:49.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.542+0000 7f8d8061e740 1 -- 172.21.15.39:0/3245410589 shutdown_connections 2024-03-18T16:16:49.544 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.542+0000 7f8d8061e740 1 -- 172.21.15.39:0/3245410589 wait complete. 2024-03-18T16:16:49.545 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.543+0000 7f8d8061e740 20 client.27041 trim_cache size 0 max 16384 2024-03-18T16:16:49.548 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-03-18T16:16:49.548 INFO:teuthology.orchestra.run.smithi039.stdout: "return_code": 0, 2024-03-18T16:16:49.548 INFO:teuthology.orchestra.run.smithi039.stdout: "scrub_tag": "b67b883a-324f-4d6c-9b71-08dc79a32883", 2024-03-18T16:16:49.548 INFO:teuthology.orchestra.run.smithi039.stdout: "mode": "asynchronous" 2024-03-18T16:16:49.548 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-03-18T16:16:49.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.547+0000 7f8d7d800640 1 -- 172.21.15.39:0/1162750350 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f8d5c0a28b0 msgr2=0x7f8d5c0a4d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:49.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.547+0000 7f8d7d800640 1 --2- 172.21.15.39:0/1162750350 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f8d5c0a28b0 0x7f8d5c0a4d70 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f8d70007cd0 tx=0x7f8d70012040 comp rx=0 tx=0).stop 2024-03-18T16:16:49.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.547+0000 7f8d7d800640 1 -- 172.21.15.39:0/1162750350 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8d78104c10 msgr2=0x7f8d78111100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:49.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.547+0000 7f8d7d800640 1 --2- 172.21.15.39:0/1162750350 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8d78104c10 0x7f8d78111100 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f8d64009230 tx=0x7f8d64005dd0 comp rx=0 tx=0).stop 2024-03-18T16:16:49.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.547+0000 7f8d7d800640 1 -- 172.21.15.39:0/1162750350 shutdown_connections 2024-03-18T16:16:49.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.547+0000 7f8d7d800640 1 --2- 172.21.15.39:0/1162750350 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f8d5c0a28b0 0x7f8d5c0a4d70 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:49.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.547+0000 7f8d7d800640 1 --2- 172.21.15.39:0/1162750350 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8d781055d0 0x7f8d78114150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:49.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.547+0000 7f8d7d800640 1 --2- 172.21.15.39:0/1162750350 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8d78104c10 0x7f8d78111100 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:49.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.547+0000 7f8d7d800640 1 --2- 172.21.15.39:0/1162750350 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8d78103a20 0x7f8d78110bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:49.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.547+0000 7f8d7d800640 1 -- 172.21.15.39:0/1162750350 >> 172.21.15.39:0/1162750350 conn(0x7f8d780fef90 msgr2=0x7f8d780ff870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:49.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.548+0000 7f8d7d800640 1 -- 172.21.15.39:0/1162750350 shutdown_connections 2024-03-18T16:16:49.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.548+0000 7f8d7d800640 1 -- 172.21.15.39:0/1162750350 wait complete. 2024-03-18T16:16:49.574 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:49 smithi039 ceph-mon[31721]: pgmap v624: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.2 MiB/s wr, 325 op/s 2024-03-18T16:16:49.578 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T16:16:49.578 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T16:16:49.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:49 smithi196 ceph-mon[36307]: pgmap v624: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.2 MiB/s wr, 325 op/s 2024-03-18T16:16:49.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:49 smithi194 ceph-mon[36200]: pgmap v624: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.2 MiB/s wr, 325 op/s 2024-03-18T16:16:49.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.850+0000 7f0ed7200640 1 -- 172.21.15.39:0/2473271883 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0ed00ffe40 msgr2=0x7f0ed0106f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:49.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.850+0000 7f0ed7200640 1 --2- 172.21.15.39:0/2473271883 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0ed00ffe40 0x7f0ed0106f90 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc009900 tx=0x7f0ebc02bc30 comp rx=0 tx=0).stop 2024-03-18T16:16:49.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.850+0000 7f0ed7200640 1 -- 172.21.15.39:0/2473271883 shutdown_connections 2024-03-18T16:16:49.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.850+0000 7f0ed7200640 1 --2- 172.21.15.39:0/2473271883 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0ed01074d0 0x7f0ed0109920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:49.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.850+0000 7f0ed7200640 1 --2- 172.21.15.39:0/2473271883 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0ed00ffe40 0x7f0ed0106f90 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:49.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.850+0000 7f0ed7200640 1 --2- 172.21.15.39:0/2473271883 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ed00ff450 0x7f0ed00ff870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:49.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.850+0000 7f0ed7200640 1 -- 172.21.15.39:0/2473271883 >> 172.21.15.39:0/2473271883 conn(0x7f0ed00fafa0 msgr2=0x7f0ed00fd3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:49.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.851+0000 7f0ed7200640 1 -- 172.21.15.39:0/2473271883 shutdown_connections 2024-03-18T16:16:49.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.851+0000 7f0ed7200640 1 -- 172.21.15.39:0/2473271883 wait complete. 2024-03-18T16:16:49.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.852+0000 7f0ed7200640 1 Processor -- start 2024-03-18T16:16:49.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.853+0000 7f0ed7200640 1 -- start start 2024-03-18T16:16:49.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.853+0000 7f0ed7200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0ed00ff450 0x7f0ed0199dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:49.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.853+0000 7f0ed7200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0ed00ffe40 0x7f0ed019a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:49.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.853+0000 7f0ed7200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ed01074d0 0x7f0ed01a1320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:49.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.853+0000 7f0ed7200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f0ed010d4c0 con 0x7f0ed01074d0 2024-03-18T16:16:49.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.853+0000 7f0ed7200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f0ed010d340 con 0x7f0ed00ffe40 2024-03-18T16:16:49.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.853+0000 7f0ed7200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f0ed010d640 con 0x7f0ed00ff450 2024-03-18T16:16:49.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.853+0000 7f0ecfe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0ed00ff450 0x7f0ed0199dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:49.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.853+0000 7f0ecfe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0ed00ff450 0x7f0ed0199dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:49830/0 (socket says 172.21.15.39:49830) 2024-03-18T16:16:49.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.854+0000 7f0ecfe00640 1 -- 172.21.15.39:0/3213708400 learned_addr learned my addr 172.21.15.39:0/3213708400 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:16:49.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.854+0000 7f0ecf400640 1 --2- 172.21.15.39:0/3213708400 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0ed00ffe40 0x7f0ed019a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:49.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.854+0000 7f0ed5200640 1 --2- 172.21.15.39:0/3213708400 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ed01074d0 0x7f0ed01a1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:49.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.854+0000 7f0ecfe00640 1 -- 172.21.15.39:0/3213708400 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0ed00ffe40 msgr2=0x7f0ed019a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:49.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.854+0000 7f0ecfe00640 1 --2- 172.21.15.39:0/3213708400 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0ed00ffe40 0x7f0ed019a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:49.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.854+0000 7f0ecfe00640 1 -- 172.21.15.39:0/3213708400 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ed01074d0 msgr2=0x7f0ed01a1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:49.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.854+0000 7f0ecfe00640 1 --2- 172.21.15.39:0/3213708400 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ed01074d0 0x7f0ed01a1320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:49.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.854+0000 7f0ecfe00640 1 -- 172.21.15.39:0/3213708400 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ed01a1990 con 0x7f0ed00ff450 2024-03-18T16:16:49.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.854+0000 7f0ecfe00640 1 --2- 172.21.15.39:0/3213708400 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0ed00ff450 0x7f0ed0199dc0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f0ec000d950 tx=0x7f0ec000de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:49.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.855+0000 7f0eccc00640 1 -- 172.21.15.39:0/3213708400 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ec0017070 con 0x7f0ed00ff450 2024-03-18T16:16:49.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.855+0000 7f0eccc00640 1 -- 172.21.15.39:0/3213708400 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ec0009c30 con 0x7f0ed00ff450 2024-03-18T16:16:49.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.855+0000 7f0eccc00640 1 -- 172.21.15.39:0/3213708400 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ec000f6b0 con 0x7f0ed00ff450 2024-03-18T16:16:49.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.855+0000 7f0ed7200640 1 -- 172.21.15.39:0/3213708400 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ed01a1c80 con 0x7f0ed00ff450 2024-03-18T16:16:49.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.855+0000 7f0ed7200640 1 -- 172.21.15.39:0/3213708400 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ed01a21a0 con 0x7f0ed00ff450 2024-03-18T16:16:49.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.860+0000 7f0ed9fdc740 1 Processor -- start 2024-03-18T16:16:49.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.860+0000 7f0ed9fdc740 1 -- start start 2024-03-18T16:16:49.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.860+0000 7f0eccc00640 1 -- 172.21.15.39:0/3213708400 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f0ec00109d0 con 0x7f0ed00ff450 2024-03-18T16:16:49.863 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.860+0000 7f0ed9fdc740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fbfc062020 0x55fbfc062440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:49.863 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.860+0000 7f0ed9fdc740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fbfc062a10 0x55fbfc067510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:49.863 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.860+0000 7f0ed9fdc740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fbfc067b80 0x55fbfc06a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:49.863 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.860+0000 7f0ed9fdc740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55fbfbfbe1f0 con 0x55fbfc067b80 2024-03-18T16:16:49.863 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.860+0000 7f0ed9fdc740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55fbfbf66a80 con 0x55fbfc062a10 2024-03-18T16:16:49.863 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.860+0000 7f0ed9fdc740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55fbfbfba290 con 0x55fbfc062020 2024-03-18T16:16:49.863 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.861+0000 7f0ed5200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fbfc062020 0x55fbfc062440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:49.863 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.861+0000 7f0ed5200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fbfc062020 0x55fbfc062440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:49834/0 (socket says 172.21.15.39:49834) 2024-03-18T16:16:49.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.861+0000 7f0ed5200640 1 -- 172.21.15.39:0/2656764101 learned_addr learned my addr 172.21.15.39:0/2656764101 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:16:49.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.861+0000 7f0ecfe00640 1 --2- 172.21.15.39:0/2656764101 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fbfc062a10 0x55fbfc067510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:49.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.861+0000 7f0ecf400640 1 --2- 172.21.15.39:0/2656764101 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fbfc067b80 0x55fbfc06a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:49.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.861+0000 7f0ed5200640 1 -- 172.21.15.39:0/2656764101 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fbfc062a10 msgr2=0x55fbfc067510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:49.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.861+0000 7f0ed5200640 1 --2- 172.21.15.39:0/2656764101 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fbfc062a10 0x55fbfc067510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:49.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.861+0000 7f0ed5200640 1 -- 172.21.15.39:0/2656764101 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fbfc067b80 msgr2=0x55fbfc06a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:49.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.861+0000 7f0ed5200640 1 --2- 172.21.15.39:0/2656764101 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fbfc067b80 0x55fbfc06a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:49.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.861+0000 7f0ed5200640 1 -- 172.21.15.39:0/2656764101 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fbfbd9c7d0 con 0x55fbfc062020 2024-03-18T16:16:49.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.861+0000 7f0ecf400640 1 --2- 172.21.15.39:0/2656764101 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fbfc067b80 0x55fbfc06a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:16:49.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.862+0000 7f0ed5200640 1 --2- 172.21.15.39:0/2656764101 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fbfc062020 0x55fbfc062440 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f0ec8009f80 tx=0x7f0ec8010860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ad9b8eb7c901e6cf server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:49.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.862+0000 7f0ec5600640 1 -- 172.21.15.39:0/2656764101 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ec8004d20 con 0x55fbfc062020 2024-03-18T16:16:49.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.862+0000 7f0ec5600640 1 -- 172.21.15.39:0/2656764101 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ec80119e0 con 0x55fbfc062020 2024-03-18T16:16:49.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.862+0000 7f0ec5600640 1 -- 172.21.15.39:0/2656764101 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ec80056d0 con 0x55fbfc062020 2024-03-18T16:16:49.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.862+0000 7f0eccc00640 1 --2- 172.21.15.39:0/3213708400 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f0eb40a2900 0x7f0eb40a4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:49.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.863+0000 7f0eccc00640 1 -- 172.21.15.39:0/3213708400 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f0ec00cea40 con 0x7f0ed00ff450 2024-03-18T16:16:49.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.863+0000 7f0ed9fdc740 1 -- 172.21.15.39:0/2656764101 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fbfc062020 msgr2=0x55fbfc062440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:49.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.863+0000 7f0ed9fdc740 1 --2- 172.21.15.39:0/2656764101 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fbfc062020 0x55fbfc062440 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f0ec8009f80 tx=0x7f0ec8010860 comp rx=0 tx=0).stop 2024-03-18T16:16:49.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.863+0000 7f0ed9fdc740 1 -- 172.21.15.39:0/2656764101 shutdown_connections 2024-03-18T16:16:49.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.863+0000 7f0ed9fdc740 1 --2- 172.21.15.39:0/2656764101 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fbfc062a10 0x55fbfc067510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:49.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.863+0000 7f0ed9fdc740 1 --2- 172.21.15.39:0/2656764101 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fbfc067b80 0x55fbfc06a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:49.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.863+0000 7f0ed9fdc740 1 --2- 172.21.15.39:0/2656764101 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fbfc062020 0x55fbfc062440 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:49.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.863+0000 7f0ecfe00640 1 --2- 172.21.15.39:0/3213708400 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f0eb40a2900 0x7f0eb40a4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:49.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.863+0000 7f0ed9fdc740 1 -- 172.21.15.39:0/2656764101 >> 172.21.15.39:0/2656764101 conn(0x55fbfc054d80 msgr2=0x55fbfc055180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:49.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.864+0000 7f0ed9fdc740 1 -- 172.21.15.39:0/2656764101 shutdown_connections 2024-03-18T16:16:49.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.864+0000 7f0ed9fdc740 1 -- 172.21.15.39:0/2656764101 wait complete. 2024-03-18T16:16:49.867 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.864+0000 7f0ecfe00640 1 --2- 172.21.15.39:0/3213708400 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f0eb40a2900 0x7f0eb40a4dc0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f0ec00979c0 tx=0x7f0ec0005880 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:49.868 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.864+0000 7f0ed9fdc740 1 -- start start 2024-03-18T16:16:49.868 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.865+0000 7f0ed9fdc740 1 Processor -- start 2024-03-18T16:16:49.868 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.865+0000 7f0ed9fdc740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fbfc062020 0x55fbfc06c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:49.868 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.865+0000 7f0ed9fdc740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fbfc062a10 0x55fbfc06ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:49.868 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.865+0000 7f0ed9fdc740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fbfc067b80 0x55fbfc073a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:49.868 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.865+0000 7f0ed9fdc740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55fbfc06ab30 con 0x55fbfc062020 2024-03-18T16:16:49.868 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.865+0000 7f0ed9fdc740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55fbfbf79ed0 con 0x55fbfc067b80 2024-03-18T16:16:49.868 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.865+0000 7f0ed9fdc740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55fbfc06acb0 con 0x55fbfc062a10 2024-03-18T16:16:49.868 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.865+0000 7f0ed9fdc740 10 client.0 _ll_register_callbacks cb 0x55fbfbfb5220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:16:49.868 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.866+0000 7f0ed5200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fbfc067b80 0x55fbfc073a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:49.868 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.866+0000 7f0ecf400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fbfc062a10 0x55fbfc06ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:49.868 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.866+0000 7f0ed5200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fbfc067b80 0x55fbfc073a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:46330/0 (socket says 172.21.15.39:46330) 2024-03-18T16:16:49.868 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.866+0000 7f0ed5200640 1 -- 172.21.15.39:0/2334001206 learned_addr learned my addr 172.21.15.39:0/2334001206 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:16:49.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.866+0000 7f0ed5200640 1 -- 172.21.15.39:0/2334001206 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fbfc062a10 msgr2=0x55fbfc06ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:49.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.866+0000 7f0ed5200640 1 --2- 172.21.15.39:0/2334001206 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fbfc062a10 0x55fbfc06ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:49.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.866+0000 7f0ed5200640 1 -- 172.21.15.39:0/2334001206 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fbfc062020 msgr2=0x55fbfc06c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:49.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.866+0000 7f0ed5200640 1 --2- 172.21.15.39:0/2334001206 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fbfc062020 0x55fbfc06c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:49.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.866+0000 7f0ed5200640 1 -- 172.21.15.39:0/2334001206 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fbfbcd23f0 con 0x55fbfc067b80 2024-03-18T16:16:49.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.867+0000 7f0ed5200640 1 --2- 172.21.15.39:0/2334001206 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fbfc067b80 0x55fbfc073a60 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f0ec8011730 tx=0x7f0ec8011760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=597b80b990938481 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:49.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.867+0000 7f0eba000640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:16:49.870 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.868+0000 7f0eba000640 1 -- 172.21.15.39:0/2334001206 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ec8004df0 con 0x55fbfc067b80 2024-03-18T16:16:49.870 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.868+0000 7f0ed9fdc740 1 -- 172.21.15.39:0/2334001206 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ec80119e0 con 0x55fbfc067b80 2024-03-18T16:16:49.870 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.868+0000 7f0eba000640 1 -- 172.21.15.39:0/2334001206 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f0ec8003970 con 0x55fbfc067b80 2024-03-18T16:16:49.872 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.870+0000 7f0eba000640 1 -- 172.21.15.39:0/2334001206 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f0ec8003b70 con 0x55fbfc067b80 2024-03-18T16:16:49.872 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.870+0000 7f0ed9fdc740 10 client.27053 fetch_fsmap learned FSMap version 44 2024-03-18T16:16:49.872 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.870+0000 7f0ed9fdc740 1 -- 172.21.15.39:0/2334001206 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x55fbfbc920f0 con 0x55fbfc067b80 2024-03-18T16:16:49.872 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.871+0000 7f0eba000640 1 -- 172.21.15.39:0/2334001206 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ec80119e0 con 0x55fbfc067b80 2024-03-18T16:16:49.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.872+0000 7f0eba000640 1 -- 172.21.15.39:0/2334001206 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7f0ec8028b10 con 0x55fbfc067b80 2024-03-18T16:16:49.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.873+0000 7f0ed9fdc740 10 client.27053 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:16:49.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.873+0000 7f0ed9fdc740 10 client.27053 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:16:49.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.873+0000 7f0ed9fdc740 20 client.27053 populate_metadata read hostname 'smithi039' 2024-03-18T16:16:49.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.873+0000 7f0ed9fdc740 1 --2- 172.21.15.39:0/2334001206 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55fbfc077ff0 0x55fbfc07a420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:49.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.873+0000 7f0ed9fdc740 4 client.27053 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:16:49.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.873+0000 7f0ed9fdc740 1 -- 172.21.15.39:0/2334001206 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fbfbfd1be0 con 0x55fbfc077ff0 2024-03-18T16:16:49.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.874+0000 7f0ecf400640 1 --2- 172.21.15.39:0/2334001206 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55fbfc077ff0 0x55fbfc07a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:49.877 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.876+0000 7f0ecf400640 1 --2- 172.21.15.39:0/2334001206 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55fbfc077ff0 0x55fbfc07a420 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1f7a4059a84509f5 server_cookie=34e2271583600642 in_seq=0 out_seq=0 2024-03-18T16:16:49.877 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.876+0000 7f0eba000640 10 client.27053 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:16:49.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.877+0000 7f0eba000640 1 -- 172.21.15.39:0/2334001206 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55fbfbfd1be0 con 0x55fbfc077ff0 2024-03-18T16:16:49.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.877+0000 7f0eba000640 10 client.27053 handle_command_reply: tid=0 2024-03-18T16:16:49.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.877+0000 7f0ed9fdc740 1 client.27053 shutdown 2024-03-18T16:16:49.880 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.878+0000 7f0ed9fdc740 1 -- 172.21.15.39:0/2334001206 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fbfc067b80 msgr2=0x55fbfc073a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:49.880 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.878+0000 7f0ed9fdc740 1 --2- 172.21.15.39:0/2334001206 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fbfc067b80 0x55fbfc073a60 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f0ec8011730 tx=0x7f0ec8011760 comp rx=0 tx=0).stop 2024-03-18T16:16:49.880 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.878+0000 7f0ed9fdc740 1 -- 172.21.15.39:0/2334001206 shutdown_connections 2024-03-18T16:16:49.880 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.878+0000 7f0ed9fdc740 1 --2- 172.21.15.39:0/2334001206 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55fbfc077ff0 0x55fbfc07a420 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:49.880 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.878+0000 7f0ed9fdc740 1 --2- 172.21.15.39:0/2334001206 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fbfc067b80 0x55fbfc073a60 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:49.880 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.878+0000 7f0ed9fdc740 1 --2- 172.21.15.39:0/2334001206 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fbfc062a10 0x55fbfc06ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:49.880 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.878+0000 7f0ed9fdc740 1 --2- 172.21.15.39:0/2334001206 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fbfc062020 0x55fbfc06c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:49.880 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.878+0000 7f0ed9fdc740 1 -- 172.21.15.39:0/2334001206 >> 172.21.15.39:0/2334001206 conn(0x55fbfc054d80 msgr2=0x55fbfc053ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:49.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.878+0000 7f0eba000640 0 client.27053 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:16:49.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.878+0000 7f0ed9fdc740 1 -- 172.21.15.39:0/2334001206 shutdown_connections 2024-03-18T16:16:49.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.878+0000 7f0ed9fdc740 1 -- 172.21.15.39:0/2334001206 wait complete. 2024-03-18T16:16:49.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.879+0000 7f0ed9fdc740 20 client.27053 trim_cache size 0 max 16384 2024-03-18T16:16:49.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.913+0000 7f0ed9fdc740 1 Processor -- start 2024-03-18T16:16:49.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.913+0000 7f0ed9fdc740 1 -- start start 2024-03-18T16:16:49.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.913+0000 7f0ed9fdc740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fbfc062020 0x55fbfc053d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:49.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.913+0000 7f0ed9fdc740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fbfc062a10 0x55fbfc073a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:49.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.913+0000 7f0ed9fdc740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fbfc067b80 0x55fbfc06b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:49.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.913+0000 7f0ed9fdc740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f0ed01a82d0 con 0x55fbfc062a10 2024-03-18T16:16:49.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.913+0000 7f0ed9fdc740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f0ed01a9e10 con 0x55fbfc062020 2024-03-18T16:16:49.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.913+0000 7f0ed9fdc740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f0ed01a89a0 con 0x55fbfc067b80 2024-03-18T16:16:49.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.914+0000 7f0ed5200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fbfc062020 0x55fbfc053d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:49.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.914+0000 7f0ed5200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fbfc062020 0x55fbfc053d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:46340/0 (socket says 172.21.15.39:46340) 2024-03-18T16:16:49.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.914+0000 7f0ed5200640 1 -- 172.21.15.39:0/1518642244 learned_addr learned my addr 172.21.15.39:0/1518642244 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:16:49.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.914+0000 7f0ed5200640 1 --2- 172.21.15.39:0/1518642244 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fbfc067b80 0x55fbfc06b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:49.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.914+0000 7f0ecf400640 1 --2- 172.21.15.39:0/1518642244 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fbfc062a10 0x55fbfc073a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:49.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.914+0000 7f0ed5200640 1 -- 172.21.15.39:0/1518642244 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fbfc067b80 msgr2=0x55fbfc06b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:49.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.914+0000 7f0ed5200640 1 --2- 172.21.15.39:0/1518642244 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fbfc067b80 0x55fbfc06b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:49.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.914+0000 7f0ed5200640 1 -- 172.21.15.39:0/1518642244 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fbfc062a10 msgr2=0x55fbfc073a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:49.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.914+0000 7f0ed5200640 1 --2- 172.21.15.39:0/1518642244 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fbfc062a10 0x55fbfc073a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:49.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.914+0000 7f0ed5200640 1 -- 172.21.15.39:0/1518642244 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fbfc054bb0 con 0x55fbfc062020 2024-03-18T16:16:49.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.915+0000 7f0ed5200640 1 --2- 172.21.15.39:0/1518642244 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fbfc062020 0x55fbfc053d40 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f0ec8010de0 tx=0x7f0ec801e040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b8999109f1dc62 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:49.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.915+0000 7f0ec4c00640 1 -- 172.21.15.39:0/1518642244 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ec8004bf0 con 0x55fbfc062020 2024-03-18T16:16:49.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.915+0000 7f0ec4c00640 1 -- 172.21.15.39:0/1518642244 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ec8020070 con 0x55fbfc062020 2024-03-18T16:16:49.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.915+0000 7f0ec4c00640 1 -- 172.21.15.39:0/1518642244 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ec80053f0 con 0x55fbfc062020 2024-03-18T16:16:49.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.916+0000 7f0ed9fdc740 1 -- 172.21.15.39:0/1518642244 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fbfc062020 msgr2=0x55fbfc053d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:49.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.916+0000 7f0ed9fdc740 1 --2- 172.21.15.39:0/1518642244 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fbfc062020 0x55fbfc053d40 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f0ec8010de0 tx=0x7f0ec801e040 comp rx=0 tx=0).stop 2024-03-18T16:16:49.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.916+0000 7f0ed9fdc740 1 -- 172.21.15.39:0/1518642244 shutdown_connections 2024-03-18T16:16:49.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.916+0000 7f0ed9fdc740 1 --2- 172.21.15.39:0/1518642244 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fbfc067b80 0x55fbfc06b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:49.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.916+0000 7f0ed9fdc740 1 --2- 172.21.15.39:0/1518642244 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fbfc062a10 0x55fbfc073a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:49.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.916+0000 7f0ed9fdc740 1 --2- 172.21.15.39:0/1518642244 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fbfc062020 0x55fbfc053d40 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:49.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.916+0000 7f0ed9fdc740 1 -- 172.21.15.39:0/1518642244 >> 172.21.15.39:0/1518642244 conn(0x55fbfc054d80 msgr2=0x55fbfc053630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:49.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.917+0000 7f0ed9fdc740 1 -- 172.21.15.39:0/1518642244 shutdown_connections 2024-03-18T16:16:49.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.917+0000 7f0ed9fdc740 1 -- 172.21.15.39:0/1518642244 wait complete. 2024-03-18T16:16:49.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.917+0000 7f0ed9fdc740 1 -- start start 2024-03-18T16:16:49.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.917+0000 7f0ed9fdc740 1 Processor -- start 2024-03-18T16:16:49.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.918+0000 7f0ed9fdc740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fbfc062020 0x55fbfc0713b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:49.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.918+0000 7f0ed9fdc740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fbfc062a10 0x55fbfc0718f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:49.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.918+0000 7f0ed9fdc740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fbfc067b80 0x55fbfc072900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:49.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.918+0000 7f0ed9fdc740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f0ed01a9740 con 0x55fbfc062a10 2024-03-18T16:16:49.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.918+0000 7f0ed9fdc740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f0ed01a9070 con 0x55fbfc062020 2024-03-18T16:16:49.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.918+0000 7f0ed9fdc740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f0ed01a62f0 con 0x55fbfc067b80 2024-03-18T16:16:49.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.918+0000 7f0ed9fdc740 10 client.0 _ll_register_callbacks cb 0x7f0ec8009e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:16:49.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.918+0000 7f0ed5200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fbfc062020 0x55fbfc0713b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:49.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.918+0000 7f0ecf400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fbfc062a10 0x55fbfc0718f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:49.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.919+0000 7f0ecf400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fbfc062a10 0x55fbfc0718f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:55050/0 (socket says 172.21.15.39:55050) 2024-03-18T16:16:49.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.919+0000 7f0ecf400640 1 -- 172.21.15.39:0/3790200668 learned_addr learned my addr 172.21.15.39:0/3790200668 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:16:49.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.919+0000 7f0ed5200640 1 --2- 172.21.15.39:0/3790200668 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fbfc067b80 0x55fbfc072900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:49.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.919+0000 7f0ecf400640 1 -- 172.21.15.39:0/3790200668 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fbfc067b80 msgr2=0x55fbfc072900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:49.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.919+0000 7f0ecf400640 1 --2- 172.21.15.39:0/3790200668 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fbfc067b80 0x55fbfc072900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:49.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.919+0000 7f0ecf400640 1 -- 172.21.15.39:0/3790200668 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fbfc062020 msgr2=0x55fbfc0713b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:16:49.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.921+0000 7f0ecf400640 1 --2- 172.21.15.39:0/3790200668 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fbfc062020 0x55fbfc0713b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:49.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.921+0000 7f0ecf400640 1 -- 172.21.15.39:0/3790200668 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fbfc0557f0 con 0x55fbfc062a10 2024-03-18T16:16:49.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.921+0000 7f0ed5200640 1 --2- 172.21.15.39:0/3790200668 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fbfc062020 0x55fbfc0713b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:16:49.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.921+0000 7f0ecf400640 1 --2- 172.21.15.39:0/3790200668 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fbfc062a10 0x55fbfc0718f0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc000c00 tx=0x7f0ebc02c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2152c54efce51b51 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:16:49.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.921+0000 7f0eba000640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-03-18T16:16:49.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.922+0000 7f0eba000640 1 -- 172.21.15.39:0/3790200668 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ebc049b10 con 0x55fbfc062a10 2024-03-18T16:16:49.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.922+0000 7f0eba000640 1 -- 172.21.15.39:0/3790200668 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f0ebc049cb0 con 0x55fbfc062a10 2024-03-18T16:16:49.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.922+0000 7f0ed9fdc740 1 -- 172.21.15.39:0/3790200668 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0ec8020070 con 0x55fbfc062a10 2024-03-18T16:16:49.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.922+0000 7f0eba000640 1 -- 172.21.15.39:0/3790200668 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f0ebc049400 con 0x55fbfc062a10 2024-03-18T16:16:49.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.922+0000 7f0ed9fdc740 10 client.16527 fetch_fsmap learned FSMap version 44 2024-03-18T16:16:49.924 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.922+0000 7f0ed9fdc740 1 -- 172.21.15.39:0/3790200668 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x55fbfc04f370 con 0x55fbfc062a10 2024-03-18T16:16:49.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.923+0000 7f0eba000640 1 -- 172.21.15.39:0/3790200668 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ec8020070 con 0x55fbfc062a10 2024-03-18T16:16:49.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.923+0000 7f0eba000640 1 -- 172.21.15.39:0/3790200668 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7f0ebc052070 con 0x55fbfc062a10 2024-03-18T16:16:49.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.923+0000 7f0ed9fdc740 10 client.16527 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:16:49.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.923+0000 7f0ed9fdc740 10 client.16527 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:16:49.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.923+0000 7f0ed9fdc740 20 client.16527 populate_metadata read hostname 'smithi039' 2024-03-18T16:16:49.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.923+0000 7f0ed9fdc740 1 --2- 172.21.15.39:0/3790200668 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55fbfc0debd0 0x55fbfc0e0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:16:49.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.923+0000 7f0ed9fdc740 4 client.16527 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T16:16:49.925 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.923+0000 7f0ed9fdc740 1 -- 172.21.15.39:0/3790200668 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55fbfbfcb730 con 0x55fbfc0debd0 2024-03-18T16:16:49.926 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.924+0000 7f0ed5200640 1 --2- 172.21.15.39:0/3790200668 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55fbfc0debd0 0x55fbfc0e0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:16:49.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.925+0000 7f0ed5200640 1 --2- 172.21.15.39:0/3790200668 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55fbfc0debd0 0x55fbfc0e0ff0 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c625808ef7a3b213 server_cookie=8f01401c7a2d7bf8 in_seq=0 out_seq=0 2024-03-18T16:16:49.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.925+0000 7f0eba000640 10 client.16527 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:16:49.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.926+0000 7f0eba000640 1 -- 172.21.15.39:0/3790200668 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55fbfbfcb730 con 0x55fbfc0debd0 2024-03-18T16:16:49.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.926+0000 7f0eba000640 10 client.16527 handle_command_reply: tid=0 2024-03-18T16:16:49.927 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.926+0000 7f0ed9fdc740 1 client.16527 shutdown 2024-03-18T16:16:49.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.926+0000 7f0ed9fdc740 1 -- 172.21.15.39:0/3790200668 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fbfc062a10 msgr2=0x55fbfc0718f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:49.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.926+0000 7f0ed9fdc740 1 --2- 172.21.15.39:0/3790200668 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fbfc062a10 0x55fbfc0718f0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc000c00 tx=0x7f0ebc02c6c0 comp rx=0 tx=0).stop 2024-03-18T16:16:49.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.926+0000 7f0ed9fdc740 1 -- 172.21.15.39:0/3790200668 shutdown_connections 2024-03-18T16:16:49.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.926+0000 7f0ed9fdc740 1 --2- 172.21.15.39:0/3790200668 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55fbfc0debd0 0x55fbfc0e0ff0 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:49.928 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.927+0000 7f0ed9fdc740 1 --2- 172.21.15.39:0/3790200668 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fbfc067b80 0x55fbfc072900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:49.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.927+0000 7f0ed9fdc740 1 --2- 172.21.15.39:0/3790200668 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fbfc062a10 0x55fbfc0718f0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:49.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.927+0000 7f0ed9fdc740 1 --2- 172.21.15.39:0/3790200668 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fbfc062020 0x55fbfc0713b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:49.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.927+0000 7f0ed9fdc740 1 -- 172.21.15.39:0/3790200668 >> 172.21.15.39:0/3790200668 conn(0x55fbfc054d80 msgr2=0x55fbfc053ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:49.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.927+0000 7f0eba000640 0 client.16527 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:16:49.929 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.927+0000 7f0ed9fdc740 1 -- 172.21.15.39:0/3790200668 shutdown_connections 2024-03-18T16:16:49.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.927+0000 7f0ed9fdc740 1 -- 172.21.15.39:0/3790200668 wait complete. 2024-03-18T16:16:49.930 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.927+0000 7f0ed9fdc740 20 client.16527 trim_cache size 0 max 16384 2024-03-18T16:16:49.930 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-03-18T16:16:49.930 INFO:teuthology.orchestra.run.smithi039.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-18T16:16:49.930 INFO:teuthology.orchestra.run.smithi039.stdout: "scrubs": { 2024-03-18T16:16:49.930 INFO:teuthology.orchestra.run.smithi039.stdout: "b67b883a-324f-4d6c-9b71-08dc79a32883": { 2024-03-18T16:16:49.930 INFO:teuthology.orchestra.run.smithi039.stdout: "path": "/", 2024-03-18T16:16:49.930 INFO:teuthology.orchestra.run.smithi039.stdout: "tag": "b67b883a-324f-4d6c-9b71-08dc79a32883", 2024-03-18T16:16:49.931 INFO:teuthology.orchestra.run.smithi039.stdout: "options": "recursive,force" 2024-03-18T16:16:49.931 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:16:49.931 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:16:49.931 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-03-18T16:16:49.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.928+0000 7f0ed7200640 1 -- 172.21.15.39:0/3213708400 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f0eb40a2900 msgr2=0x7f0eb40a4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:49.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.928+0000 7f0ed7200640 1 --2- 172.21.15.39:0/3213708400 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f0eb40a2900 0x7f0eb40a4dc0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f0ec00979c0 tx=0x7f0ec0005880 comp rx=0 tx=0).stop 2024-03-18T16:16:49.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.928+0000 7f0ed7200640 1 -- 172.21.15.39:0/3213708400 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0ed00ff450 msgr2=0x7f0ed0199dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:16:49.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.928+0000 7f0ed7200640 1 --2- 172.21.15.39:0/3213708400 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0ed00ff450 0x7f0ed0199dc0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f0ec000d950 tx=0x7f0ec000de10 comp rx=0 tx=0).stop 2024-03-18T16:16:49.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.929+0000 7f0ed7200640 1 -- 172.21.15.39:0/3213708400 shutdown_connections 2024-03-18T16:16:49.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.929+0000 7f0ed7200640 1 --2- 172.21.15.39:0/3213708400 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0ed00ffe40 0x7f0ed019a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:49.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.929+0000 7f0ed7200640 1 --2- 172.21.15.39:0/3213708400 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f0eb40a2900 0x7f0eb40a4dc0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:49.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.929+0000 7f0ed7200640 1 --2- 172.21.15.39:0/3213708400 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0ed01074d0 0x7f0ed01a1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:49.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.929+0000 7f0ed7200640 1 --2- 172.21.15.39:0/3213708400 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0ed00ff450 0x7f0ed0199dc0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:16:49.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.929+0000 7f0ed7200640 1 -- 172.21.15.39:0/3213708400 >> 172.21.15.39:0/3213708400 conn(0x7f0ed00fafa0 msgr2=0x7f0ed00fd390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:16:49.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.929+0000 7f0ed7200640 1 -- 172.21.15.39:0/3213708400 shutdown_connections 2024-03-18T16:16:49.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:16:49.929+0000 7f0ed7200640 1 -- 172.21.15.39:0/3213708400 wait complete. 2024-03-18T16:16:49.953 INFO:tasks.cephfs.filesystem:scrub status for tag:b67b883a-324f-4d6c-9b71-08dc79a32883 - {'path': '/', 'tag': 'b67b883a-324f-4d6c-9b71-08dc79a32883', 'options': 'recursive,force'} 2024-03-18T16:16:50.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:50 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:16:50.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:50 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:16:50.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:50 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:16:50.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:50 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:50.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:50 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:50.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:50 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:16:50.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:50 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:50.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:50 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:16:50.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:50 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:50.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:50 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:16:50.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:50 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:50.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:50 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:16:50.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:50 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:50.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:50 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:16:50.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:50 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:50.742 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:50 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:16:50.742 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:50 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:50.742 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:50 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:16:50.742 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:50 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:50.742 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:50 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:16:50.742 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:50 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:50.742 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:50 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:16:50.742 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:50 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:50.742 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:50 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:16:50.742 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:50 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:50.742 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:50 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:16:50.742 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:50 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:50.742 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:50 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:50.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:50 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:16:50.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:50 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:16:50.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:50 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:16:50.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:50 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:50.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:50 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:50.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:50 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:16:50.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:50 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:50.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:50 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:16:50.838 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:50 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:50.838 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:50 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:16:50.838 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:50 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:50.838 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:50 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:16:50.838 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:50 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:50.838 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:50 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:16:50.838 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:50 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:50.838 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:50 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:16:50.838 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:50 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:50.838 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:50 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:16:50.838 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:50 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:50.838 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:50 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:16:50.838 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:50 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:50.838 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:50 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:16:50.839 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:50 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:50.839 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:50 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:16:50.839 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:50 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:50.839 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:50 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:16:50.839 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:50 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:50.839 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:50 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:50.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:50 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:16:50.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:50 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:16:50.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:50 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:16:50.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:50 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:50.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:50 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:50.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:50 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:16:50.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:50 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:50.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:50 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:16:50.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:50 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:50.980 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:50 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:16:50.980 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:50 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:50.980 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:50 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:16:50.980 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:50 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:50.980 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:50 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:16:50.980 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:50 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:50.980 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:50 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:16:50.980 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:50 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:50.980 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:50 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:16:50.980 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:50 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:50.980 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:50 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:16:50.980 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:50 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:50.980 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:50 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:16:50.980 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:50 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:50.981 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:50 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:16:50.981 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:50 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:50.981 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:50 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:16:50.981 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:50 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:50.981 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:50 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:51.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:51 smithi194 ceph-mon[36200]: scrub queued for path: / 2024-03-18T16:16:51.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:51 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:51.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:51 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:16:51.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:51 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:16:51.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:51 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:51.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:51 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:51.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:51 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:16:51.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:51 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:51.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:51 smithi194 ceph-mon[36200]: pgmap v625: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 1.4 MiB/s wr, 359 op/s 2024-03-18T16:16:51.840 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:51 smithi039 ceph-mon[31721]: scrub queued for path: / 2024-03-18T16:16:51.840 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:51 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:51.840 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:51 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:16:51.840 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:51 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:16:51.840 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:51 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:51.840 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:51 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:51.840 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:51 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:16:51.840 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:51 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:51.840 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:51 smithi039 ceph-mon[31721]: pgmap v625: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 1.4 MiB/s wr, 359 op/s 2024-03-18T16:16:51.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:51 smithi196 ceph-mon[36307]: scrub queued for path: / 2024-03-18T16:16:51.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:51 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:51.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:51 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:16:51.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:51 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:16:51.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:51 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:51.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:51 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:51.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:51 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:16:51.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:51 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:16:51.979 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:51 smithi196 ceph-mon[36307]: pgmap v625: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 1.4 MiB/s wr, 359 op/s 2024-03-18T16:16:53.740 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:53 smithi194 ceph-mon[36200]: pgmap v626: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 1.5 MiB/s wr, 392 op/s 2024-03-18T16:16:53.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:53 smithi039 ceph-mon[31721]: pgmap v626: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 1.5 MiB/s wr, 392 op/s 2024-03-18T16:16:53.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:53 smithi196 ceph-mon[36307]: pgmap v626: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 1.5 MiB/s wr, 392 op/s 2024-03-18T16:16:55.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:55 smithi039 ceph-mon[31721]: pgmap v627: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 1.3 MiB/s wr, 345 op/s 2024-03-18T16:16:55.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:55 smithi196 ceph-mon[36307]: pgmap v627: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 1.3 MiB/s wr, 345 op/s 2024-03-18T16:16:55.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:55 smithi194 ceph-mon[36200]: pgmap v627: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 1.3 MiB/s wr, 345 op/s 2024-03-18T16:16:57.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:57 smithi039 ceph-mon[31721]: pgmap v628: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 1.6 MiB/s wr, 416 op/s 2024-03-18T16:16:57.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:57 smithi196 ceph-mon[36307]: pgmap v628: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 1.6 MiB/s wr, 416 op/s 2024-03-18T16:16:57.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:57 smithi194 ceph-mon[36200]: pgmap v628: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 1.6 MiB/s wr, 416 op/s 2024-03-18T16:16:59.839 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:16:59 smithi039 ceph-mon[31721]: pgmap v629: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 1.4 MiB/s wr, 359 op/s 2024-03-18T16:16:59.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:16:59 smithi196 ceph-mon[36307]: pgmap v629: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 1.4 MiB/s wr, 359 op/s 2024-03-18T16:16:59.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:16:59 smithi194 ceph-mon[36200]: pgmap v629: 129 pgs: 129 active+clean; 13 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 1.4 MiB/s wr, 359 op/s 2024-03-18T16:17:01.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:17:01 smithi039 ceph-mon[31721]: pgmap v630: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 2.3 MiB/s wr, 577 op/s 2024-03-18T16:17:01.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:17:01 smithi196 ceph-mon[36307]: pgmap v630: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 2.3 MiB/s wr, 577 op/s 2024-03-18T16:17:01.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:17:01 smithi194 ceph-mon[36200]: pgmap v630: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 2.3 MiB/s wr, 577 op/s 2024-03-18T16:17:02.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:17:02 smithi039 ceph-mon[31721]: scrub summary: idle 2024-03-18T16:17:02.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:17:02 smithi039 ceph-mon[31721]: scrub summary: idle 2024-03-18T16:17:02.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:17:02 smithi196 ceph-mon[36307]: scrub summary: idle 2024-03-18T16:17:02.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:17:02 smithi196 ceph-mon[36307]: scrub summary: idle 2024-03-18T16:17:02.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:17:02 smithi194 ceph-mon[36200]: scrub summary: idle 2024-03-18T16:17:02.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:17:02 smithi194 ceph-mon[36200]: scrub summary: idle 2024-03-18T16:17:03.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:17:03 smithi039 ceph-mon[31721]: pgmap v631: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 7.4 MiB/s wr, 1.75k op/s 2024-03-18T16:17:03.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:17:03 smithi039 ceph-mon[31721]: scrub summary: idle 2024-03-18T16:17:03.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:17:03 smithi039 ceph-mon[31721]: scrub summary: idle 2024-03-18T16:17:03.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:17:03 smithi039 ceph-mon[31721]: scrub summary: idle 2024-03-18T16:17:03.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:17:03 smithi196 ceph-mon[36307]: pgmap v631: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 7.4 MiB/s wr, 1.75k op/s 2024-03-18T16:17:03.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:17:03 smithi196 ceph-mon[36307]: scrub summary: idle 2024-03-18T16:17:03.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:17:03 smithi196 ceph-mon[36307]: scrub summary: idle 2024-03-18T16:17:03.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:17:03 smithi196 ceph-mon[36307]: scrub summary: idle 2024-03-18T16:17:03.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:17:03 smithi194 ceph-mon[36200]: pgmap v631: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 19 KiB/s rd, 7.4 MiB/s wr, 1.75k op/s 2024-03-18T16:17:03.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:17:03 smithi194 ceph-mon[36200]: scrub summary: idle 2024-03-18T16:17:03.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:17:03 smithi194 ceph-mon[36200]: scrub summary: idle 2024-03-18T16:17:03.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:17:03 smithi194 ceph-mon[36200]: scrub summary: idle 2024-03-18T16:17:05.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:17:05 smithi039 ceph-mon[31721]: pgmap v632: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 9.5 MiB/s wr, 2.22k op/s 2024-03-18T16:17:05.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:17:05 smithi196 ceph-mon[36307]: pgmap v632: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 9.5 MiB/s wr, 2.22k op/s 2024-03-18T16:17:05.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:17:05 smithi194 ceph-mon[36200]: pgmap v632: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 9.5 MiB/s wr, 2.22k op/s 2024-03-18T16:17:06.978 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:17:06 smithi196 ceph-mon[36307]: pgmap v633: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 13 MiB/s wr, 3.09k op/s 2024-03-18T16:17:06.990 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:17:06 smithi194 ceph-mon[36200]: pgmap v633: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 13 MiB/s wr, 3.09k op/s 2024-03-18T16:17:07.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:17:06 smithi039 ceph-mon[31721]: pgmap v633: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 13 MiB/s wr, 3.09k op/s 2024-03-18T16:17:09.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:17:08 smithi196 ceph-mon[36307]: pgmap v634: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 15 MiB/s wr, 3.56k op/s 2024-03-18T16:17:09.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:17:08 smithi194 ceph-mon[36200]: pgmap v634: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 15 MiB/s wr, 3.56k op/s 2024-03-18T16:17:09.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:17:08 smithi039 ceph-mon[31721]: pgmap v634: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 15 MiB/s wr, 3.56k op/s 2024-03-18T16:17:10.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:17:09 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T16:17:10.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:17:09 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T16:17:10.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:17:09 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T16:17:10.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:17:09 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:17:10.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:17:09 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:17:10.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:17:09 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:17:10.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:17:09 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:17:10.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:17:09 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T16:17:10.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:17:09 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T16:17:10.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:17:09 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T16:17:10.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:17:09 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:17:10.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:17:09 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:17:10.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:17:09 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:17:10.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:17:09 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:17:10.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:17:09 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-18T16:17:10.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:17:09 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-18T16:17:10.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:17:09 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-18T16:17:10.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:17:09 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:17:10.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:17:09 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:17:10.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:17:09 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:17:10.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:17:09 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:17:15.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:17:14 smithi196 ceph-mon[36307]: pgmap v635: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 17 MiB/s wr, 3.98k op/s 2024-03-18T16:17:15.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:17:14 smithi196 ceph-mon[36307]: pgmap v636: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 21 MiB/s wr, 4.85k op/s 2024-03-18T16:17:15.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:17:14 smithi196 ceph-mon[36307]: pgmap v637: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 18 MiB/s wr, 4.00k op/s 2024-03-18T16:17:15.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:17:14 smithi194 ceph-mon[36200]: pgmap v635: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 17 MiB/s wr, 3.98k op/s 2024-03-18T16:17:15.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:17:14 smithi194 ceph-mon[36200]: pgmap v636: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 21 MiB/s wr, 4.85k op/s 2024-03-18T16:17:15.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:17:14 smithi194 ceph-mon[36200]: pgmap v637: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 18 MiB/s wr, 4.00k op/s 2024-03-18T16:17:15.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:17:14 smithi039 ceph-mon[31721]: pgmap v635: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 17 MiB/s wr, 3.98k op/s 2024-03-18T16:17:15.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:17:14 smithi039 ceph-mon[31721]: pgmap v636: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 21 MiB/s wr, 4.85k op/s 2024-03-18T16:17:15.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:17:14 smithi039 ceph-mon[31721]: pgmap v637: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 27 KiB/s rd, 18 MiB/s wr, 4.00k op/s 2024-03-18T16:17:17.140 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:17:16 smithi039 ceph-mon[31721]: pgmap v638: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 22 MiB/s wr, 4.77k op/s 2024-03-18T16:17:17.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:17:16 smithi196 ceph-mon[36307]: pgmap v638: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 22 MiB/s wr, 4.77k op/s 2024-03-18T16:17:17.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:17:16 smithi194 ceph-mon[36200]: pgmap v638: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 22 MiB/s wr, 4.77k op/s 2024-03-18T16:17:19.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:17:18 smithi196 ceph-mon[36307]: pgmap v639: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 22 MiB/s wr, 4.80k op/s 2024-03-18T16:17:19.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:17:18 smithi194 ceph-mon[36200]: pgmap v639: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 22 MiB/s wr, 4.80k op/s 2024-03-18T16:17:19.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:17:18 smithi039 ceph-mon[31721]: pgmap v639: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 35 KiB/s rd, 22 MiB/s wr, 4.80k op/s 2024-03-18T16:17:19.954 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T16:17:19.954 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T16:17:20.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.304+0000 7f3724400640 1 -- 172.21.15.39:0/657604204 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f371c111f20 msgr2=0x7f371c114330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:20.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.304+0000 7f3724400640 1 --2- 172.21.15.39:0/657604204 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f371c111f20 0x7f371c114330 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f3714009ca0 tx=0x7f371402c100 comp rx=0 tx=0).stop 2024-03-18T16:17:20.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.304+0000 7f3724400640 1 -- 172.21.15.39:0/657604204 shutdown_connections 2024-03-18T16:17:20.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.304+0000 7f3724400640 1 --2- 172.21.15.39:0/657604204 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f371c111f20 0x7f371c114330 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:20.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.304+0000 7f3724400640 1 --2- 172.21.15.39:0/657604204 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f371c10b150 0x7f371c10b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:20.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.304+0000 7f3724400640 1 --2- 172.21.15.39:0/657604204 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f371c1087e0 0x7f371c10ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:20.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.304+0000 7f3724400640 1 -- 172.21.15.39:0/657604204 >> 172.21.15.39:0/657604204 conn(0x7f371c06e640 msgr2=0x7f371c070a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:17:20.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.305+0000 7f3724400640 1 -- 172.21.15.39:0/657604204 shutdown_connections 2024-03-18T16:17:20.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.305+0000 7f3724400640 1 -- 172.21.15.39:0/657604204 wait complete. 2024-03-18T16:17:20.312 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.308+0000 7f3724400640 1 Processor -- start 2024-03-18T16:17:20.312 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.310+0000 7f3724400640 1 -- start start 2024-03-18T16:17:20.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.310+0000 7f3724400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f371c1087e0 0x7f371c1a68e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:20.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.310+0000 7f3724400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f371c10b150 0x7f371c1a6e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:20.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.310+0000 7f3724400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f371c111f20 0x7f371c1ade90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:20.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.310+0000 7f3724400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f371c117e90 con 0x7f371c10b150 2024-03-18T16:17:20.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.310+0000 7f3724400640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f371c117d10 con 0x7f371c111f20 2024-03-18T16:17:20.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.310+0000 7f3724400640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f371c118010 con 0x7f371c1087e0 2024-03-18T16:17:20.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.310+0000 7f3721a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f371c1087e0 0x7f371c1a68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:20.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.311+0000 7f3721000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f371c10b150 0x7f371c1a6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:20.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.311+0000 7f3721a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f371c1087e0 0x7f371c1a68e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:53628/0 (socket says 172.21.15.39:53628) 2024-03-18T16:17:20.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.311+0000 7f3722400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f371c111f20 0x7f371c1ade90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:20.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.311+0000 7f3721a00640 1 -- 172.21.15.39:0/1422067437 learned_addr learned my addr 172.21.15.39:0/1422067437 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:17:20.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.311+0000 7f3722400640 1 -- 172.21.15.39:0/1422067437 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f371c1087e0 msgr2=0x7f371c1a68e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:20.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.311+0000 7f3722400640 1 --2- 172.21.15.39:0/1422067437 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f371c1087e0 0x7f371c1a68e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:20.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.311+0000 7f3722400640 1 -- 172.21.15.39:0/1422067437 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f371c10b150 msgr2=0x7f371c1a6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:20.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.311+0000 7f3722400640 1 --2- 172.21.15.39:0/1422067437 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f371c10b150 0x7f371c1a6e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:20.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.311+0000 7f3722400640 1 -- 172.21.15.39:0/1422067437 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f371c1ae500 con 0x7f371c111f20 2024-03-18T16:17:20.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.312+0000 7f3722400640 1 --2- 172.21.15.39:0/1422067437 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f371c111f20 0x7f371c1ade90 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f371402c710 tx=0x7f3714035d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:17:20.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.313+0000 7f371a000640 1 -- 172.21.15.39:0/1422067437 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3714039040 con 0x7f371c111f20 2024-03-18T16:17:20.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.313+0000 7f371a000640 1 -- 172.21.15.39:0/1422067437 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3714044070 con 0x7f371c111f20 2024-03-18T16:17:20.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.313+0000 7f371a000640 1 -- 172.21.15.39:0/1422067437 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3714004380 con 0x7f371c111f20 2024-03-18T16:17:20.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.313+0000 7f3724400640 1 -- 172.21.15.39:0/1422067437 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f371c1ae790 con 0x7f371c111f20 2024-03-18T16:17:20.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.313+0000 7f3724400640 1 -- 172.21.15.39:0/1422067437 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f371c1aec00 con 0x7f371c111f20 2024-03-18T16:17:20.326 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.323+0000 7f371a000640 1 -- 172.21.15.39:0/1422067437 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f3714004520 con 0x7f371c111f20 2024-03-18T16:17:20.326 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.323+0000 7f3727073740 1 Processor -- start 2024-03-18T16:17:20.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.325+0000 7f371a000640 1 --2- 172.21.15.39:0/1422067437 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f37040a29d0 0x7f37040a4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:20.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.325+0000 7f371a000640 1 -- 172.21.15.39:0/1422067437 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f37140f0ec0 con 0x7f371c111f20 2024-03-18T16:17:20.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.327+0000 7f3727073740 1 -- start start 2024-03-18T16:17:20.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.327+0000 7f3727073740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f551064810 0x55f551064c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:20.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.327+0000 7f3727073740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f551065200 0x55f551069d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:20.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.327+0000 7f3727073740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f55106a370 0x55f55106c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:20.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.327+0000 7f3727073740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55f550fc91f0 con 0x55f55106a370 2024-03-18T16:17:20.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.327+0000 7f3727073740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55f550f71a80 con 0x55f551065200 2024-03-18T16:17:20.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.327+0000 7f3727073740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55f550fc5290 con 0x55f551064810 2024-03-18T16:17:20.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.328+0000 7f3721a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f551064810 0x55f551064c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:20.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.328+0000 7f3721a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f551064810 0x55f551064c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:53644/0 (socket says 172.21.15.39:53644) 2024-03-18T16:17:20.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.328+0000 7f3721a00640 1 -- 172.21.15.39:0/1807542385 learned_addr learned my addr 172.21.15.39:0/1807542385 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:17:20.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.328+0000 7f3721000640 1 --2- 172.21.15.39:0/1807542385 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f551065200 0x55f551069d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:20.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.328+0000 7f3721000640 1 --2- 172.21.15.39:0/1422067437 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f37040a29d0 0x7f37040a4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:20.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.329+0000 7f3722400640 1 --2- 172.21.15.39:0/1807542385 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f55106a370 0x55f55106c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:20.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.329+0000 7f3721a00640 1 -- 172.21.15.39:0/1807542385 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f551065200 msgr2=0x55f551069d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:17:20.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.329+0000 7f3721a00640 1 --2- 172.21.15.39:0/1807542385 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f551065200 0x55f551069d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:20.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.329+0000 7f3721a00640 1 -- 172.21.15.39:0/1807542385 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f55106a370 msgr2=0x55f55106c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:17:20.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.330+0000 7f3721a00640 1 --2- 172.21.15.39:0/1807542385 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f55106a370 0x55f55106c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:20.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.330+0000 7f3721a00640 1 -- 172.21.15.39:0/1807542385 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f550da77d0 con 0x55f551064810 2024-03-18T16:17:20.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.330+0000 7f3721000640 1 --2- 172.21.15.39:0/1807542385 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f551065200 0x55f551069d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:17:20.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.330+0000 7f3722400640 1 --2- 172.21.15.39:0/1807542385 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f55106a370 0x55f55106c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:17:20.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.331+0000 7f3721a00640 1 --2- 172.21.15.39:0/1807542385 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f551064810 0x55f551064c30 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f3708009d30 tx=0x7f37080117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=59b9f39f176c2b98 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:17:20.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.331+0000 7f370ea00640 1 -- 172.21.15.39:0/1807542385 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3708012650 con 0x55f551064810 2024-03-18T16:17:20.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.331+0000 7f3721000640 1 --2- 172.21.15.39:0/1422067437 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f37040a29d0 0x7f37040a4e90 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f37100058c0 tx=0x7f3710005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:17:20.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.332+0000 7f370ea00640 1 -- 172.21.15.39:0/1807542385 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3708009e20 con 0x55f551064810 2024-03-18T16:17:20.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.332+0000 7f370ea00640 1 -- 172.21.15.39:0/1807542385 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f370800fce0 con 0x55f551064810 2024-03-18T16:17:20.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.333+0000 7f3727073740 1 -- 172.21.15.39:0/1807542385 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f551064810 msgr2=0x55f551064c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:20.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.333+0000 7f3727073740 1 --2- 172.21.15.39:0/1807542385 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f551064810 0x55f551064c30 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f3708009d30 tx=0x7f37080117b0 comp rx=0 tx=0).stop 2024-03-18T16:17:20.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.333+0000 7f3727073740 1 -- 172.21.15.39:0/1807542385 shutdown_connections 2024-03-18T16:17:20.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.333+0000 7f3727073740 1 --2- 172.21.15.39:0/1807542385 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f551065200 0x55f551069d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:20.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.333+0000 7f3727073740 1 --2- 172.21.15.39:0/1807542385 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f55106a370 0x55f55106c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:20.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.333+0000 7f3727073740 1 --2- 172.21.15.39:0/1807542385 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f551064810 0x55f551064c30 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:20.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.333+0000 7f3727073740 1 -- 172.21.15.39:0/1807542385 >> 172.21.15.39:0/1807542385 conn(0x55f55105fd80 msgr2=0x55f551060180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:17:20.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.334+0000 7f3727073740 1 -- 172.21.15.39:0/1807542385 shutdown_connections 2024-03-18T16:17:20.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.334+0000 7f3727073740 1 -- 172.21.15.39:0/1807542385 wait complete. 2024-03-18T16:17:20.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.334+0000 7f3727073740 1 -- start start 2024-03-18T16:17:20.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.335+0000 7f3727073740 1 Processor -- start 2024-03-18T16:17:20.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.336+0000 7f3727073740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f551064810 0x55f55106ed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:20.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.336+0000 7f3727073740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f551065200 0x55f55106f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:20.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.336+0000 7f3727073740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f55106a370 0x55f5510762c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:20.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.336+0000 7f3721000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f551065200 0x55f55106f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:20.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.337+0000 7f3721000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f551065200 0x55f55106f2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:41678/0 (socket says 172.21.15.39:41678) 2024-03-18T16:17:20.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.337+0000 7f3721000640 1 -- 172.21.15.39:0/2955441317 learned_addr learned my addr 172.21.15.39:0/2955441317 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:17:20.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.337+0000 7f3727073740 1 -- 172.21.15.39:0/2955441317 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55f55106d320 con 0x55f55106a370 2024-03-18T16:17:20.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.337+0000 7f3727073740 1 -- 172.21.15.39:0/2955441317 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55f550f84ed0 con 0x55f551065200 2024-03-18T16:17:20.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.337+0000 7f3722400640 1 --2- 172.21.15.39:0/2955441317 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f55106a370 0x55f5510762c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:20.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.337+0000 7f3727073740 1 -- 172.21.15.39:0/2955441317 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55f55106d4a0 con 0x55f551064810 2024-03-18T16:17:20.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.338+0000 7f3721000640 1 -- 172.21.15.39:0/2955441317 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f551064810 msgr2=0x55f55106ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:17:20.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.338+0000 7f3727073740 10 client.0 _ll_register_callbacks cb 0x55f550fc0220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:17:20.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.338+0000 7f3721a00640 1 --2- 172.21.15.39:0/2955441317 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f551064810 0x55f55106ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:20.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.338+0000 7f3721000640 1 --2- 172.21.15.39:0/2955441317 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f551064810 0x55f55106ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:20.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.338+0000 7f3721000640 1 -- 172.21.15.39:0/2955441317 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f55106a370 msgr2=0x55f5510762c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:17:20.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.338+0000 7f3721000640 1 --2- 172.21.15.39:0/2955441317 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f55106a370 0x55f5510762c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:20.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.338+0000 7f3721000640 1 -- 172.21.15.39:0/2955441317 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f550cdd3f0 con 0x55f551065200 2024-03-18T16:17:20.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.338+0000 7f3722400640 1 --2- 172.21.15.39:0/2955441317 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f55106a370 0x55f5510762c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:17:20.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.339+0000 7f3727073740 1 -- 172.21.15.39:0/2955441317 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3708009e20 con 0x55f551065200 2024-03-18T16:17:20.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.339+0000 7f3721000640 1 --2- 172.21.15.39:0/2955441317 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f551065200 0x55f55106f2c0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f371000fb20 tx=0x7f3710004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=991097303c2bee12 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:17:20.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.339+0000 7f3703400640 10 client.27071 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:17:20.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.340+0000 7f3703400640 1 -- 172.21.15.39:0/2955441317 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f371001fe40 con 0x55f551065200 2024-03-18T16:17:20.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.340+0000 7f3703400640 1 -- 172.21.15.39:0/2955441317 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f3710029470 con 0x55f551065200 2024-03-18T16:17:20.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.340+0000 7f3703400640 1 -- 172.21.15.39:0/2955441317 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f3710029670 con 0x55f551065200 2024-03-18T16:17:20.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.340+0000 7f3727073740 10 client.27071 fetch_fsmap learned FSMap version 44 2024-03-18T16:17:20.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.340+0000 7f3727073740 1 -- 172.21.15.39:0/2955441317 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({fsmap=44}) -- 0x55f550c9d0f0 con 0x55f551065200 2024-03-18T16:17:20.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.342+0000 7f3703400640 1 -- 172.21.15.39:0/2955441317 <== mon.2 v2:172.21.15.194:3300/0 4 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7f3710029810 con 0x55f551065200 2024-03-18T16:17:20.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.343+0000 7f3727073740 10 client.27071 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:17:20.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.343+0000 7f3727073740 10 client.27071 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:17:20.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.343+0000 7f3727073740 20 client.27071 populate_metadata read hostname 'smithi039' 2024-03-18T16:17:20.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.343+0000 7f3727073740 1 --2- 172.21.15.39:0/2955441317 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55f55107aef0 0x55f55107d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:20.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.343+0000 7f3727073740 4 client.27071 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:17:20.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.343+0000 7f3727073740 1 -- 172.21.15.39:0/2955441317 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f550fdcbe0 con 0x55f55107aef0 2024-03-18T16:17:20.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.343+0000 7f3721a00640 1 --2- 172.21.15.39:0/2955441317 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55f55107aef0 0x55f55107d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:20.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.344+0000 7f3721a00640 1 --2- 172.21.15.39:0/2955441317 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55f55107aef0 0x55f55107d3b0 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a32e731ff07c3c1 server_cookie=5213001bf28db6e1 in_seq=0 out_seq=0 2024-03-18T16:17:20.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.344+0000 7f3703400640 10 client.27071 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:17:20.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.347+0000 7f3703400640 1 -- 172.21.15.39:0/2955441317 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f550fdcbe0 con 0x55f55107aef0 2024-03-18T16:17:20.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.347+0000 7f3703400640 10 client.27071 handle_command_reply: tid=0 2024-03-18T16:17:20.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.347+0000 7f3727073740 1 client.27071 shutdown 2024-03-18T16:17:20.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.350+0000 7f3727073740 1 -- 172.21.15.39:0/2955441317 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f551065200 msgr2=0x55f55106f2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:20.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.350+0000 7f3727073740 1 --2- 172.21.15.39:0/2955441317 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f551065200 0x55f55106f2c0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f371000fb20 tx=0x7f3710004730 comp rx=0 tx=0).stop 2024-03-18T16:17:20.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.350+0000 7f3727073740 1 -- 172.21.15.39:0/2955441317 shutdown_connections 2024-03-18T16:17:20.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.350+0000 7f3727073740 1 --2- 172.21.15.39:0/2955441317 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55f55107aef0 0x55f55107d3b0 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:20.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.350+0000 7f3727073740 1 --2- 172.21.15.39:0/2955441317 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f551065200 0x55f55106f2c0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:20.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.350+0000 7f3727073740 1 --2- 172.21.15.39:0/2955441317 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f55106a370 0x55f5510762c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:20.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.350+0000 7f3703400640 0 client.27071 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:17:20.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.350+0000 7f3727073740 1 --2- 172.21.15.39:0/2955441317 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f551064810 0x55f55106ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:20.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.350+0000 7f3727073740 1 -- 172.21.15.39:0/2955441317 >> 172.21.15.39:0/2955441317 conn(0x55f55105fd80 msgr2=0x55f55105eec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:17:20.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.350+0000 7f3727073740 1 -- 172.21.15.39:0/2955441317 shutdown_connections 2024-03-18T16:17:20.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.350+0000 7f3727073740 1 -- 172.21.15.39:0/2955441317 wait complete. 2024-03-18T16:17:20.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.351+0000 7f3727073740 20 client.27071 trim_cache size 0 max 16384 2024-03-18T16:17:20.400 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.397+0000 7f3727073740 1 Processor -- start 2024-03-18T16:17:20.400 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.397+0000 7f3727073740 1 -- start start 2024-03-18T16:17:20.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.398+0000 7f3727073740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f551064810 0x55f55105e630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:20.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.398+0000 7f3727073740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f551065200 0x55f551076280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:20.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.398+0000 7f3727073740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f55106a370 0x55f55105eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:20.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.398+0000 7f3727073740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f371c1b91d0 con 0x55f551064810 2024-03-18T16:17:20.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.398+0000 7f3727073740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f371c1bad10 con 0x55f55106a370 2024-03-18T16:17:20.401 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.398+0000 7f3727073740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f371c1b98a0 con 0x55f551065200 2024-03-18T16:17:20.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.399+0000 7f3722400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f55106a370 0x55f55105eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:20.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.399+0000 7f3722400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f55106a370 0x55f55105eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:41680/0 (socket says 172.21.15.39:41680) 2024-03-18T16:17:20.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.399+0000 7f3722400640 1 -- 172.21.15.39:0/57265661 learned_addr learned my addr 172.21.15.39:0/57265661 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:17:20.403 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.399+0000 7f3722400640 1 -- 172.21.15.39:0/57265661 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f551065200 msgr2=0x55f551076280 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:17:20.403 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.399+0000 7f3722400640 1 --2- 172.21.15.39:0/57265661 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f551065200 0x55f551076280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:20.403 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.399+0000 7f3722400640 1 -- 172.21.15.39:0/57265661 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f551064810 msgr2=0x55f55105e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:17:20.403 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.399+0000 7f3722400640 1 --2- 172.21.15.39:0/57265661 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f551064810 0x55f55105e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:20.403 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.399+0000 7f3722400640 1 -- 172.21.15.39:0/57265661 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f55105a370 con 0x55f55106a370 2024-03-18T16:17:20.403 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.400+0000 7f3722400640 1 --2- 172.21.15.39:0/57265661 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f55106a370 0x55f55105eb70 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f37140ee5d0 tx=0x7f3714004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a5aebcaf4f12ceb9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:17:20.403 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.400+0000 7f370e000640 1 -- 172.21.15.39:0/57265661 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37140bdb70 con 0x55f55106a370 2024-03-18T16:17:20.403 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.400+0000 7f370e000640 1 -- 172.21.15.39:0/57265661 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f37140bd470 con 0x55f55106a370 2024-03-18T16:17:20.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.400+0000 7f370e000640 1 -- 172.21.15.39:0/57265661 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37140b9ae0 con 0x55f55106a370 2024-03-18T16:17:20.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.401+0000 7f3727073740 1 -- 172.21.15.39:0/57265661 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f55106a370 msgr2=0x55f55105eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:20.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.401+0000 7f3727073740 1 --2- 172.21.15.39:0/57265661 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f55106a370 0x55f55105eb70 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f37140ee5d0 tx=0x7f3714004a40 comp rx=0 tx=0).stop 2024-03-18T16:17:20.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.402+0000 7f3727073740 1 -- 172.21.15.39:0/57265661 shutdown_connections 2024-03-18T16:17:20.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.402+0000 7f3727073740 1 --2- 172.21.15.39:0/57265661 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f55106a370 0x55f55105eb70 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:20.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.402+0000 7f3727073740 1 --2- 172.21.15.39:0/57265661 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f551065200 0x55f551076280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:20.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.402+0000 7f3727073740 1 --2- 172.21.15.39:0/57265661 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f551064810 0x55f55105e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:20.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.402+0000 7f3727073740 1 -- 172.21.15.39:0/57265661 >> 172.21.15.39:0/57265661 conn(0x55f55105fd80 msgr2=0x55f551066740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:17:20.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.402+0000 7f3727073740 1 -- 172.21.15.39:0/57265661 shutdown_connections 2024-03-18T16:17:20.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.405+0000 7f3727073740 1 -- 172.21.15.39:0/57265661 wait complete. 2024-03-18T16:17:20.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.406+0000 7f3727073740 1 -- start start 2024-03-18T16:17:20.409 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.406+0000 7f3727073740 1 Processor -- start 2024-03-18T16:17:20.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.407+0000 7f3727073740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f551064810 0x55f551073870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:20.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.407+0000 7f3727073740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f551065200 0x55f551073db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:20.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.407+0000 7f3727073740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f55106a370 0x55f551074dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:20.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.407+0000 7f3727073740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f371c1ba640 con 0x55f551064810 2024-03-18T16:17:20.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.407+0000 7f3727073740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f371c1b9f70 con 0x55f55106a370 2024-03-18T16:17:20.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.407+0000 7f3727073740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f371c1b71b0 con 0x55f551065200 2024-03-18T16:17:20.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.407+0000 7f3727073740 10 client.0 _ll_register_callbacks cb 0x7f3710009a50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:17:20.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.407+0000 7f3721000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f551065200 0x55f551073db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:20.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.407+0000 7f3721000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f551065200 0x55f551073db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:53660/0 (socket says 172.21.15.39:53660) 2024-03-18T16:17:20.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.407+0000 7f3721000640 1 -- 172.21.15.39:0/1496763075 learned_addr learned my addr 172.21.15.39:0/1496763075 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:17:20.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.408+0000 7f3722400640 1 --2- 172.21.15.39:0/1496763075 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f55106a370 0x55f551074dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:20.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.408+0000 7f3721000640 1 -- 172.21.15.39:0/1496763075 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f55106a370 msgr2=0x55f551074dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:17:20.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.408+0000 7f3721000640 1 --2- 172.21.15.39:0/1496763075 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f55106a370 0x55f551074dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:20.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.408+0000 7f3721000640 1 -- 172.21.15.39:0/1496763075 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f551064810 msgr2=0x55f551073870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:17:20.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.408+0000 7f3721000640 1 --2- 172.21.15.39:0/1496763075 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f551064810 0x55f551073870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:20.412 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.408+0000 7f3721000640 1 -- 172.21.15.39:0/1496763075 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f5510753b0 con 0x55f551065200 2024-03-18T16:17:20.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.409+0000 7f3721000640 1 --2- 172.21.15.39:0/1496763075 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f551065200 0x55f551073db0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f3710005890 tx=0x7f3710013030 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d2743c13505ac080 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:17:20.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.409+0000 7f3703400640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:17:20.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.409+0000 7f3703400640 1 -- 172.21.15.39:0/1496763075 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f371000f6d0 con 0x55f551065200 2024-03-18T16:17:20.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.409+0000 7f3727073740 1 -- 172.21.15.39:0/1496763075 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f37140bd470 con 0x55f551065200 2024-03-18T16:17:20.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.409+0000 7f3703400640 1 -- 172.21.15.39:0/1496763075 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f3710029470 con 0x55f551065200 2024-03-18T16:17:20.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.409+0000 7f3703400640 1 -- 172.21.15.39:0/1496763075 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f3710010ce0 con 0x55f551065200 2024-03-18T16:17:20.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.410+0000 7f3727073740 10 client.27046 fetch_fsmap learned FSMap version 44 2024-03-18T16:17:20.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.410+0000 7f3727073740 1 -- 172.21.15.39:0/1496763075 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x55f5510e12f0 con 0x55f551065200 2024-03-18T16:17:20.417 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.411+0000 7f3703400640 1 -- 172.21.15.39:0/1496763075 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f37140bd470 con 0x55f551065200 2024-03-18T16:17:20.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.415+0000 7f3703400640 1 -- 172.21.15.39:0/1496763075 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7f3710004c70 con 0x55f551065200 2024-03-18T16:17:20.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.415+0000 7f3727073740 10 client.27046 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:17:20.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.415+0000 7f3727073740 10 client.27046 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:17:20.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.415+0000 7f3727073740 20 client.27046 populate_metadata read hostname 'smithi039' 2024-03-18T16:17:20.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.415+0000 7f3727073740 1 --2- 172.21.15.39:0/1496763075 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55f5510e1bd0 0x55f5510e3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:20.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.415+0000 7f3727073740 4 client.27046 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T16:17:20.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.415+0000 7f3727073740 1 -- 172.21.15.39:0/1496763075 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f550fd6730 con 0x55f5510e1bd0 2024-03-18T16:17:20.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.416+0000 7f3721a00640 1 --2- 172.21.15.39:0/1496763075 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55f5510e1bd0 0x55f5510e3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:20.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.417+0000 7f3721a00640 1 --2- 172.21.15.39:0/1496763075 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55f5510e1bd0 0x55f5510e3fc0 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5828eb24ec2948b9 server_cookie=69a547809d5f2141 in_seq=0 out_seq=0 2024-03-18T16:17:20.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.417+0000 7f3703400640 10 client.27046 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:17:20.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.418+0000 7f3703400640 1 -- 172.21.15.39:0/1496763075 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55f550fd6730 con 0x55f5510e1bd0 2024-03-18T16:17:20.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.418+0000 7f3703400640 10 client.27046 handle_command_reply: tid=0 2024-03-18T16:17:20.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.418+0000 7f3727073740 1 client.27046 shutdown 2024-03-18T16:17:20.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.418+0000 7f3727073740 1 -- 172.21.15.39:0/1496763075 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f551065200 msgr2=0x55f551073db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:20.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.418+0000 7f3727073740 1 --2- 172.21.15.39:0/1496763075 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f551065200 0x55f551073db0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f3710005890 tx=0x7f3710013030 comp rx=0 tx=0).stop 2024-03-18T16:17:20.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.419+0000 7f3727073740 1 -- 172.21.15.39:0/1496763075 shutdown_connections 2024-03-18T16:17:20.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.419+0000 7f3727073740 1 --2- 172.21.15.39:0/1496763075 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55f5510e1bd0 0x55f5510e3fc0 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:20.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.419+0000 7f3727073740 1 --2- 172.21.15.39:0/1496763075 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f55106a370 0x55f551074dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:20.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.419+0000 7f3727073740 1 --2- 172.21.15.39:0/1496763075 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f551065200 0x55f551073db0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:20.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.419+0000 7f3727073740 1 --2- 172.21.15.39:0/1496763075 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f551064810 0x55f551073870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:20.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.419+0000 7f3703400640 0 client.27046 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:17:20.424 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.419+0000 7f3727073740 1 -- 172.21.15.39:0/1496763075 >> 172.21.15.39:0/1496763075 conn(0x55f55105fd80 msgr2=0x55f55105e630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:17:20.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.423+0000 7f3727073740 1 -- 172.21.15.39:0/1496763075 shutdown_connections 2024-03-18T16:17:20.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.423+0000 7f3727073740 1 -- 172.21.15.39:0/1496763075 wait complete. 2024-03-18T16:17:20.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.424+0000 7f3727073740 20 client.27046 trim_cache size 0 max 16384 2024-03-18T16:17:20.426 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-03-18T16:17:20.427 INFO:teuthology.orchestra.run.smithi039.stdout: "status": "no active scrubs running", 2024-03-18T16:17:20.427 INFO:teuthology.orchestra.run.smithi039.stdout: "scrubs": {} 2024-03-18T16:17:20.427 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-03-18T16:17:20.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.425+0000 7f3724400640 1 -- 172.21.15.39:0/1422067437 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f37040a29d0 msgr2=0x7f37040a4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:20.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.425+0000 7f3724400640 1 --2- 172.21.15.39:0/1422067437 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f37040a29d0 0x7f37040a4e90 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f37100058c0 tx=0x7f3710005f90 comp rx=0 tx=0).stop 2024-03-18T16:17:20.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.425+0000 7f3724400640 1 -- 172.21.15.39:0/1422067437 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f371c111f20 msgr2=0x7f371c1ade90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:20.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.425+0000 7f3724400640 1 --2- 172.21.15.39:0/1422067437 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f371c111f20 0x7f371c1ade90 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f371402c710 tx=0x7f3714035d10 comp rx=0 tx=0).stop 2024-03-18T16:17:20.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.425+0000 7f3724400640 1 -- 172.21.15.39:0/1422067437 shutdown_connections 2024-03-18T16:17:20.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.425+0000 7f3724400640 1 --2- 172.21.15.39:0/1422067437 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f371c111f20 0x7f371c1ade90 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:20.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.425+0000 7f3724400640 1 --2- 172.21.15.39:0/1422067437 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f37040a29d0 0x7f37040a4e90 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:20.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.425+0000 7f3724400640 1 --2- 172.21.15.39:0/1422067437 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f371c10b150 0x7f371c1a6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:20.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.425+0000 7f3724400640 1 --2- 172.21.15.39:0/1422067437 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f371c1087e0 0x7f371c1a68e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:20.428 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.425+0000 7f3724400640 1 -- 172.21.15.39:0/1422067437 >> 172.21.15.39:0/1422067437 conn(0x7f371c06e640 msgr2=0x7f371c10a9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:17:20.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.426+0000 7f3724400640 1 -- 172.21.15.39:0/1422067437 shutdown_connections 2024-03-18T16:17:20.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.426+0000 7f3724400640 1 -- 172.21.15.39:0/1422067437 wait complete. 2024-03-18T16:17:20.459 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-18T16:17:20.459 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:17:20.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.781+0000 7eff45200640 1 -- 172.21.15.39:0/124007690 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7eff40104bf0 msgr2=0x7eff40105070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:20.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.781+0000 7eff45200640 1 --2- 172.21.15.39:0/124007690 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7eff40104bf0 0x7eff40105070 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7eff2c009860 tx=0x7eff2c02bb70 comp rx=0 tx=0).stop 2024-03-18T16:17:20.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.785+0000 7eff45200640 1 -- 172.21.15.39:0/124007690 shutdown_connections 2024-03-18T16:17:20.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.785+0000 7eff45200640 1 --2- 172.21.15.39:0/124007690 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7eff401055b0 0x7eff4010be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:20.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.785+0000 7eff45200640 1 --2- 172.21.15.39:0/124007690 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7eff40104bf0 0x7eff40105070 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:20.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.785+0000 7eff45200640 1 --2- 172.21.15.39:0/124007690 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7eff40103a00 0x7eff40103e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:20.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.785+0000 7eff45200640 1 -- 172.21.15.39:0/124007690 >> 172.21.15.39:0/124007690 conn(0x7eff400fef90 msgr2=0x7eff401013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:17:20.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.785+0000 7eff45200640 1 -- 172.21.15.39:0/124007690 shutdown_connections 2024-03-18T16:17:20.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.785+0000 7eff45200640 1 -- 172.21.15.39:0/124007690 wait complete. 2024-03-18T16:17:20.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.787+0000 7eff45200640 1 Processor -- start 2024-03-18T16:17:20.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.787+0000 7eff45200640 1 -- start start 2024-03-18T16:17:20.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.787+0000 7eff45200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7eff40103a00 0x7eff40199d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:20.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.788+0000 7eff45200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7eff40104bf0 0x7eff4019a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:20.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.788+0000 7eff45200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7eff401055b0 0x7eff401a1310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:20.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.788+0000 7eff45200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7eff4010f6b0 con 0x7eff40104bf0 2024-03-18T16:17:20.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.788+0000 7eff45200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7eff4010f530 con 0x7eff401055b0 2024-03-18T16:17:20.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.788+0000 7eff45200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7eff4010f830 con 0x7eff40103a00 2024-03-18T16:17:20.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.789+0000 7eff3de00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7eff40103a00 0x7eff40199d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:20.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.789+0000 7eff3de00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7eff40103a00 0x7eff40199d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:53674/0 (socket says 172.21.15.39:53674) 2024-03-18T16:17:20.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.789+0000 7eff3de00640 1 -- 172.21.15.39:0/759415094 learned_addr learned my addr 172.21.15.39:0/759415094 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:17:20.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.790+0000 7eff3e800640 1 --2- 172.21.15.39:0/759415094 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7eff401055b0 0x7eff401a1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:20.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.790+0000 7eff3d400640 1 --2- 172.21.15.39:0/759415094 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7eff40104bf0 0x7eff4019a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:20.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.791+0000 7eff3d400640 1 -- 172.21.15.39:0/759415094 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7eff40103a00 msgr2=0x7eff40199d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:20.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.791+0000 7eff3d400640 1 --2- 172.21.15.39:0/759415094 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7eff40103a00 0x7eff40199d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:20.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.791+0000 7eff3d400640 1 -- 172.21.15.39:0/759415094 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7eff401055b0 msgr2=0x7eff401a1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:20.795 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.791+0000 7eff3d400640 1 --2- 172.21.15.39:0/759415094 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7eff401055b0 0x7eff401a1310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:20.795 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.792+0000 7eff3d400640 1 -- 172.21.15.39:0/759415094 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff4006b330 con 0x7eff40104bf0 2024-03-18T16:17:20.796 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.794+0000 7eff3d400640 1 --2- 172.21.15.39:0/759415094 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7eff40104bf0 0x7eff4019a2a0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7eff2c009780 tx=0x7eff2c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:17:20.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.795+0000 7eff36a00640 1 -- 172.21.15.39:0/759415094 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff2c039040 con 0x7eff40104bf0 2024-03-18T16:17:20.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.795+0000 7eff36a00640 1 -- 172.21.15.39:0/759415094 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff2c0347b0 con 0x7eff40104bf0 2024-03-18T16:17:20.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.795+0000 7eff36a00640 1 -- 172.21.15.39:0/759415094 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff2c0046c0 con 0x7eff40104bf0 2024-03-18T16:17:20.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.795+0000 7eff45200640 1 -- 172.21.15.39:0/759415094 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff4006b5c0 con 0x7eff40104bf0 2024-03-18T16:17:20.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.795+0000 7eff45200640 1 -- 172.21.15.39:0/759415094 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff40075e60 con 0x7eff40104bf0 2024-03-18T16:17:20.800 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.797+0000 7eff45200640 1 -- 172.21.15.39:0/759415094 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff4010ada0 con 0x7eff40104bf0 2024-03-18T16:17:20.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.805+0000 7eff36a00640 1 -- 172.21.15.39:0/759415094 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7eff2c044020 con 0x7eff40104bf0 2024-03-18T16:17:20.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.807+0000 7eff36a00640 1 --2- 172.21.15.39:0/759415094 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7eff240a2a00 0x7eff240a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:20.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.811+0000 7eff36a00640 1 -- 172.21.15.39:0/759415094 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7eff2c0f0810 con 0x7eff40104bf0 2024-03-18T16:17:20.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.812+0000 7eff3de00640 1 --2- 172.21.15.39:0/759415094 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7eff240a2a00 0x7eff240a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:20.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.819+0000 7eff3de00640 1 --2- 172.21.15.39:0/759415094 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7eff240a2a00 0x7eff240a4ec0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7eff30005fd0 tx=0x7eff30005dd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:17:20.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:20.825+0000 7eff36a00640 1 -- 172.21.15.39:0/759415094 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7eff2c04b020 con 0x7eff40104bf0 2024-03-18T16:17:21.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:17:20 smithi196 ceph-mon[36307]: pgmap v640: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 22 MiB/s wr, 4.74k op/s 2024-03-18T16:17:21.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:17:20 smithi194 ceph-mon[36200]: pgmap v640: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 22 MiB/s wr, 4.74k op/s 2024-03-18T16:17:21.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.330+0000 7eff45200640 1 -- 172.21.15.39:0/759415094 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7eff4002d370 con 0x7eff40104bf0 2024-03-18T16:17:21.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.333+0000 7eff36a00640 1 -- 172.21.15.39:0/759415094 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) ==== 94+0+11932 (secure 0 0 0) 0x7eff2c0bd540 con 0x7eff40104bf0 2024-03-18T16:17:21.336 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:17:21.336 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":44,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2520226608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2520226608},{"type":"v1","addr":"172.21.15.39:6835","nonce":2520226608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/719341763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":719341763},{"type":"v1","addr":"172.21.15.39:6837","nonce":719341763}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":44,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T16:06:44.180425+0000","modified":"2024-03-18T16:15:58.268360+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24575,"mds_1":24556,"mds_2":24535,"mds_3":24524,"mds_4":14463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14445":{"gid":14445,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.39:6839/656513813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":656513813},{"type":"v1","addr":"172.21.15.39:6839","nonce":656513813}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14463":{"gid":14463,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":13,"addr":"172.21.15.39:6841/3515326042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3515326042},{"type":"v1","addr":"172.21.15.39:6841","nonce":3515326042}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6833/931350738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":931350738},{"type":"v1","addr":"172.21.15.194:6833","nonce":931350738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6833/3874209621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":3874209621},{"type":"v1","addr":"172.21.15.196:6833","nonce":3874209621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6835/188385470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":188385470},{"type":"v1","addr":"172.21.15.194:6835","nonce":188385470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6835/3949984309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":3949984309},{"type":"v1","addr":"172.21.15.196:6835","nonce":3949984309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":15,"addr":"172.21.15.194:6837/1301464246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":1301464246},{"type":"v1","addr":"172.21.15.194:6837","nonce":1301464246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":16,"addr":"172.21.15.196:6837/895247251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":895247251},{"type":"v1","addr":"172.21.15.196:6837","nonce":895247251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.196:6839/2001427268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6838","nonce":2001427268},{"type":"v1","addr":"172.21.15.196:6839","nonce":2001427268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24575":{"gid":24575,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.194:6839/274911158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":274911158},{"type":"v1","addr":"172.21.15.194:6839","nonce":274911158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24575,"qdb_cluster":[14463,24524,24575,24556,24535]},"id":1}]} 2024-03-18T16:17:21.336 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 44 2024-03-18T16:17:21.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:17:20 smithi039 ceph-mon[31721]: pgmap v640: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 29 KiB/s rd, 22 MiB/s wr, 4.74k op/s 2024-03-18T16:17:21.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.339+0000 7eff45200640 1 -- 172.21.15.39:0/759415094 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7eff240a2a00 msgr2=0x7eff240a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:21.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.339+0000 7eff45200640 1 --2- 172.21.15.39:0/759415094 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7eff240a2a00 0x7eff240a4ec0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7eff30005fd0 tx=0x7eff30005dd0 comp rx=0 tx=0).stop 2024-03-18T16:17:21.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.339+0000 7eff45200640 1 -- 172.21.15.39:0/759415094 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7eff40104bf0 msgr2=0x7eff4019a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:21.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.339+0000 7eff45200640 1 --2- 172.21.15.39:0/759415094 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7eff40104bf0 0x7eff4019a2a0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7eff2c009780 tx=0x7eff2c004270 comp rx=0 tx=0).stop 2024-03-18T16:17:21.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.339+0000 7eff45200640 1 -- 172.21.15.39:0/759415094 shutdown_connections 2024-03-18T16:17:21.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.339+0000 7eff45200640 1 --2- 172.21.15.39:0/759415094 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7eff401055b0 0x7eff401a1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:21.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.339+0000 7eff45200640 1 --2- 172.21.15.39:0/759415094 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7eff240a2a00 0x7eff240a4ec0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:21.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.339+0000 7eff45200640 1 --2- 172.21.15.39:0/759415094 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7eff40104bf0 0x7eff4019a2a0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:21.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.339+0000 7eff45200640 1 --2- 172.21.15.39:0/759415094 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7eff40103a00 0x7eff40199d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:21.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.339+0000 7eff45200640 1 -- 172.21.15.39:0/759415094 >> 172.21.15.39:0/759415094 conn(0x7eff400fef90 msgr2=0x7eff40107f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:17:21.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.339+0000 7eff45200640 1 -- 172.21.15.39:0/759415094 shutdown_connections 2024-03-18T16:17:21.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.339+0000 7eff45200640 1 -- 172.21.15.39:0/759415094 wait complete. 2024-03-18T16:17:21.372 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-18T16:17:21.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.695+0000 7fd892c00640 1 -- 172.21.15.39:0/3600842260 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd88c10b8c0 msgr2=0x7fd88c10dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:21.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.695+0000 7fd892c00640 1 --2- 172.21.15.39:0/3600842260 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd88c10b8c0 0x7fd88c10dcb0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fd874009900 tx=0x7fd87402bc40 comp rx=0 tx=0).stop 2024-03-18T16:17:21.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.697+0000 7fd892c00640 1 -- 172.21.15.39:0/3600842260 shutdown_connections 2024-03-18T16:17:21.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.697+0000 7fd892c00640 1 --2- 172.21.15.39:0/3600842260 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd88c10b8c0 0x7fd88c10dcb0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:21.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.697+0000 7fd892c00640 1 --2- 172.21.15.39:0/3600842260 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd88c10e320 0x7fd88c1107b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:21.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.697+0000 7fd892c00640 1 --2- 172.21.15.39:0/3600842260 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd88c06a600 0x7fd88c10b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:21.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.697+0000 7fd892c00640 1 -- 172.21.15.39:0/3600842260 >> 172.21.15.39:0/3600842260 conn(0x7fd88c06e9c0 msgr2=0x7fd88c070e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:17:21.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.698+0000 7fd892c00640 1 -- 172.21.15.39:0/3600842260 shutdown_connections 2024-03-18T16:17:21.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.699+0000 7fd892c00640 1 -- 172.21.15.39:0/3600842260 wait complete. 2024-03-18T16:17:21.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.700+0000 7fd892c00640 1 Processor -- start 2024-03-18T16:17:21.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.700+0000 7fd892c00640 1 -- start start 2024-03-18T16:17:21.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.701+0000 7fd892c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd88c06a600 0x7fd88c1a7110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:21.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.701+0000 7fd892c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd88c10b8c0 0x7fd88c1a7650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:21.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.701+0000 7fd892c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd88c10e320 0x7fd88c1adf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:21.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.701+0000 7fd892c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fd88c114bf0 con 0x7fd88c10b8c0 2024-03-18T16:17:21.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.701+0000 7fd892c00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fd88c114a70 con 0x7fd88c10e320 2024-03-18T16:17:21.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.701+0000 7fd892c00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fd88c114d70 con 0x7fd88c06a600 2024-03-18T16:17:21.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.701+0000 7fd883e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd88c10b8c0 0x7fd88c1a7650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:21.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.701+0000 7fd88be00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd88c06a600 0x7fd88c1a7110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:21.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.701+0000 7fd88be00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd88c06a600 0x7fd88c1a7110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:53696/0 (socket says 172.21.15.39:53696) 2024-03-18T16:17:21.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.701+0000 7fd883e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd88c10b8c0 0x7fd88c1a7650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:37300/0 (socket says 172.21.15.39:37300) 2024-03-18T16:17:21.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.701+0000 7fd88be00640 1 -- 172.21.15.39:0/3284750115 learned_addr learned my addr 172.21.15.39:0/3284750115 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:17:21.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.702+0000 7fd890c00640 1 --2- 172.21.15.39:0/3284750115 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd88c10e320 0x7fd88c1adf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:21.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.703+0000 7fd883e00640 1 -- 172.21.15.39:0/3284750115 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd88c06a600 msgr2=0x7fd88c1a7110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:21.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.703+0000 7fd883e00640 1 --2- 172.21.15.39:0/3284750115 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd88c06a600 0x7fd88c1a7110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:21.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.703+0000 7fd883e00640 1 -- 172.21.15.39:0/3284750115 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd88c10e320 msgr2=0x7fd88c1adf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:21.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.703+0000 7fd883e00640 1 --2- 172.21.15.39:0/3284750115 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd88c10e320 0x7fd88c1adf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:21.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.703+0000 7fd883e00640 1 -- 172.21.15.39:0/3284750115 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd88c1ae640 con 0x7fd88c10b8c0 2024-03-18T16:17:21.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.703+0000 7fd883e00640 1 --2- 172.21.15.39:0/3284750115 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd88c10b8c0 0x7fd88c1a7650 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fd87402c180 tx=0x7fd87402c6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:17:21.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.704+0000 7fd889600640 1 -- 172.21.15.39:0/3284750115 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd874041070 con 0x7fd88c10b8c0 2024-03-18T16:17:21.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.704+0000 7fd889600640 1 -- 172.21.15.39:0/3284750115 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd874004450 con 0x7fd88c10b8c0 2024-03-18T16:17:21.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.704+0000 7fd892c00640 1 -- 172.21.15.39:0/3284750115 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd88c1ae870 con 0x7fd88c10b8c0 2024-03-18T16:17:21.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.704+0000 7fd892c00640 1 -- 172.21.15.39:0/3284750115 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd88c1aed30 con 0x7fd88c10b8c0 2024-03-18T16:17:21.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.706+0000 7fd889600640 1 -- 172.21.15.39:0/3284750115 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd874041070 con 0x7fd88c10b8c0 2024-03-18T16:17:21.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.709+0000 7fd895879740 1 Processor -- start 2024-03-18T16:17:21.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.710+0000 7fd895879740 1 -- start start 2024-03-18T16:17:21.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.711+0000 7fd895879740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558cb281e8c0 0x558cb281ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:21.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.711+0000 7fd895879740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558cb281f2b0 0x558cb2823db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:21.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.711+0000 7fd895879740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558cb2824420 0x558cb28268b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:21.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.711+0000 7fd895879740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x558cb27e63b0 con 0x558cb281e8c0 2024-03-18T16:17:21.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.711+0000 7fd895879740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x558cb2783450 con 0x558cb2824420 2024-03-18T16:17:21.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.711+0000 7fd895879740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x558cb272bcb0 con 0x558cb281f2b0 2024-03-18T16:17:21.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.712+0000 7fd88be00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558cb281f2b0 0x558cb2823db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:21.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.712+0000 7fd88be00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558cb281f2b0 0x558cb2823db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:53702/0 (socket says 172.21.15.39:53702) 2024-03-18T16:17:21.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.712+0000 7fd88be00640 1 -- 172.21.15.39:0/1284853126 learned_addr learned my addr 172.21.15.39:0/1284853126 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:17:21.716 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.712+0000 7fd890c00640 1 --2- 172.21.15.39:0/1284853126 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558cb281e8c0 0x558cb281ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:21.716 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.713+0000 7fd88be00640 1 -- 172.21.15.39:0/1284853126 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558cb2824420 msgr2=0x558cb28268b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:17:21.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.713+0000 7fd88be00640 1 --2- 172.21.15.39:0/1284853126 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558cb2824420 0x558cb28268b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:21.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.713+0000 7fd88be00640 1 -- 172.21.15.39:0/1284853126 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558cb281e8c0 msgr2=0x558cb281ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:17:21.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.713+0000 7fd88be00640 1 --2- 172.21.15.39:0/1284853126 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558cb281e8c0 0x558cb281ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:21.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.713+0000 7fd88be00640 1 -- 172.21.15.39:0/1284853126 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558cb274d1f0 con 0x558cb281f2b0 2024-03-18T16:17:21.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.714+0000 7fd889600640 1 -- 172.21.15.39:0/3284750115 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fd874044070 con 0x7fd88c10b8c0 2024-03-18T16:17:21.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.715+0000 7fd88be00640 1 --2- 172.21.15.39:0/1284853126 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558cb281f2b0 0x558cb2823db0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fd87c009d70 tx=0x7fd87c011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6e00a418a029b1a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:17:21.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.715+0000 7fd881200640 1 -- 172.21.15.39:0/1284853126 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd87c004220 con 0x558cb281f2b0 2024-03-18T16:17:21.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.715+0000 7fd881200640 1 -- 172.21.15.39:0/1284853126 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd87c0043c0 con 0x558cb281f2b0 2024-03-18T16:17:21.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.716+0000 7fd889600640 1 --2- 172.21.15.39:0/3284750115 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd8700a2a00 0x7fd8700a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:21.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.716+0000 7fd895879740 1 -- 172.21.15.39:0/1284853126 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558cb281f2b0 msgr2=0x558cb2823db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:21.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.717+0000 7fd889600640 1 -- 172.21.15.39:0/3284750115 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fd8740f05a0 con 0x7fd88c10b8c0 2024-03-18T16:17:21.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.717+0000 7fd881200640 1 -- 172.21.15.39:0/1284853126 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd87c004220 con 0x558cb281f2b0 2024-03-18T16:17:21.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.717+0000 7fd895879740 1 --2- 172.21.15.39:0/1284853126 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558cb281f2b0 0x558cb2823db0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fd87c009d70 tx=0x7fd87c011810 comp rx=0 tx=0).stop 2024-03-18T16:17:21.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.717+0000 7fd890c00640 1 --2- 172.21.15.39:0/3284750115 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd8700a2a00 0x7fd8700a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:21.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.717+0000 7fd895879740 1 -- 172.21.15.39:0/1284853126 shutdown_connections 2024-03-18T16:17:21.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.717+0000 7fd895879740 1 --2- 172.21.15.39:0/1284853126 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558cb2824420 0x558cb28268b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:21.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.717+0000 7fd895879740 1 --2- 172.21.15.39:0/1284853126 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558cb281f2b0 0x558cb2823db0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:21.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.717+0000 7fd895879740 1 --2- 172.21.15.39:0/1284853126 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558cb281e8c0 0x558cb281ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:21.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.717+0000 7fd895879740 1 -- 172.21.15.39:0/1284853126 >> 172.21.15.39:0/1284853126 conn(0x558cb2819e30 msgr2=0x558cb281a230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:17:21.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.717+0000 7fd895879740 1 -- 172.21.15.39:0/1284853126 shutdown_connections 2024-03-18T16:17:21.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.718+0000 7fd895879740 1 -- 172.21.15.39:0/1284853126 wait complete. 2024-03-18T16:17:21.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.718+0000 7fd890c00640 1 --2- 172.21.15.39:0/3284750115 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd8700a2a00 0x7fd8700a4ec0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x558cb281fb10 tx=0x7fd88400d040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:17:21.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.718+0000 7fd895879740 1 -- start start 2024-03-18T16:17:21.722 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.719+0000 7fd895879740 1 Processor -- start 2024-03-18T16:17:21.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.720+0000 7fd895879740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558cb281e8c0 0x558cb2828b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:21.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.720+0000 7fd895879740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558cb281f2b0 0x558cb2829090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:21.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.720+0000 7fd895879740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558cb2824420 0x558cb28300d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:21.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.720+0000 7fd895879740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x558cb273f110 con 0x558cb281e8c0 2024-03-18T16:17:21.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.720+0000 7fd895879740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x558cb277f4f0 con 0x558cb2824420 2024-03-18T16:17:21.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.720+0000 7fd895879740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x558cb2827420 con 0x558cb281f2b0 2024-03-18T16:17:21.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.720+0000 7fd895879740 10 client.0 _ll_register_callbacks cb 0x558cb272cb70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:17:21.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.721+0000 7fd890c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558cb2824420 0x558cb28300d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:21.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.721+0000 7fd890c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558cb2824420 0x558cb28300d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:41738/0 (socket says 172.21.15.39:41738) 2024-03-18T16:17:21.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.721+0000 7fd890c00640 1 -- 172.21.15.39:0/1574781932 learned_addr learned my addr 172.21.15.39:0/1574781932 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:17:21.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.721+0000 7fd883e00640 1 --2- 172.21.15.39:0/1574781932 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558cb281f2b0 0x558cb2829090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:21.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.721+0000 7fd890c00640 1 -- 172.21.15.39:0/1574781932 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558cb281f2b0 msgr2=0x558cb2829090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:17:21.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.721+0000 7fd890c00640 1 --2- 172.21.15.39:0/1574781932 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558cb281f2b0 0x558cb2829090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:21.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.721+0000 7fd890c00640 1 -- 172.21.15.39:0/1574781932 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558cb281e8c0 msgr2=0x558cb2828b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:17:21.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.721+0000 7fd88be00640 1 --2- 172.21.15.39:0/1574781932 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558cb281e8c0 0x558cb2828b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:21.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.724+0000 7fd890c00640 1 --2- 172.21.15.39:0/1574781932 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558cb281e8c0 0x558cb2828b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:21.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.724+0000 7fd890c00640 1 -- 172.21.15.39:0/1574781932 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558cb24973f0 con 0x558cb2824420 2024-03-18T16:17:21.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.724+0000 7fd88be00640 1 --2- 172.21.15.39:0/1574781932 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558cb281e8c0 0x558cb2828b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:17:21.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.724+0000 7fd890c00640 1 --2- 172.21.15.39:0/1574781932 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558cb2824420 0x558cb28300d0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fd884018a90 tx=0x7fd884004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=69949a678ff94cf6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:17:21.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.724+0000 7fd879600640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:17:21.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.724+0000 7fd879600640 1 -- 172.21.15.39:0/1574781932 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd884028060 con 0x558cb2824420 2024-03-18T16:17:21.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.724+0000 7fd879600640 1 -- 172.21.15.39:0/1574781932 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fd884016d60 con 0x558cb2824420 2024-03-18T16:17:21.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.724+0000 7fd895879740 1 -- 172.21.15.39:0/1574781932 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd87c0043c0 con 0x558cb2824420 2024-03-18T16:17:21.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.725+0000 7fd879600640 1 -- 172.21.15.39:0/1574781932 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fd884023440 con 0x558cb2824420 2024-03-18T16:17:21.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.725+0000 7fd895879740 10 client.27089 fetch_fsmap learned FSMap version 44 2024-03-18T16:17:21.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.725+0000 7fd895879740 1 -- 172.21.15.39:0/1574781932 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x558cb25617d0 con 0x558cb2824420 2024-03-18T16:17:21.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.726+0000 7fd879600640 1 -- 172.21.15.39:0/1574781932 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd87c0043c0 con 0x558cb2824420 2024-03-18T16:17:21.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.726+0000 7fd879600640 1 -- 172.21.15.39:0/1574781932 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7fd88402c860 con 0x558cb2824420 2024-03-18T16:17:21.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.726+0000 7fd895879740 10 client.27089 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:17:21.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.726+0000 7fd895879740 10 client.27089 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-03-18T16:17:21.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.726+0000 7fd895879740 20 client.27089 populate_metadata read hostname 'smithi039' 2024-03-18T16:17:21.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.726+0000 7fd895879740 1 --2- 172.21.15.39:0/1574781932 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x558cb2834970 0x558cb2836e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:21.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.726+0000 7fd895879740 4 client.27089 mds_command: new command op to 14463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:17:21.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.726+0000 7fd895879740 1 -- 172.21.15.39:0/1574781932 --> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558cb2790990 con 0x558cb2834970 2024-03-18T16:17:21.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.728+0000 7fd88be00640 1 --2- 172.21.15.39:0/1574781932 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x558cb2834970 0x558cb2836e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:21.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.729+0000 7fd88be00640 1 --2- 172.21.15.39:0/1574781932 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x558cb2834970 0x558cb2836e00 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=af38936899cbd6a1 server_cookie=b048cc6fb5aae8be in_seq=0 out_seq=0 2024-03-18T16:17:21.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.730+0000 7fd879600640 10 client.27089 ms_handle_connect on v2:172.21.15.39:6840/3515326042 2024-03-18T16:17:21.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.731+0000 7fd879600640 1 -- 172.21.15.39:0/1574781932 <== mds.4 v2:172.21.15.39:6840/3515326042 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x558cb2790990 con 0x558cb2834970 2024-03-18T16:17:21.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.731+0000 7fd879600640 10 client.27089 handle_command_reply: tid=0 2024-03-18T16:17:21.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.732+0000 7fd895879740 1 client.27089 shutdown 2024-03-18T16:17:21.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.732+0000 7fd895879740 1 -- 172.21.15.39:0/1574781932 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558cb2824420 msgr2=0x558cb28300d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:21.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.732+0000 7fd895879740 1 --2- 172.21.15.39:0/1574781932 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558cb2824420 0x558cb28300d0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fd884018a90 tx=0x7fd884004a40 comp rx=0 tx=0).stop 2024-03-18T16:17:21.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.733+0000 7fd895879740 1 -- 172.21.15.39:0/1574781932 shutdown_connections 2024-03-18T16:17:21.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.733+0000 7fd895879740 1 --2- 172.21.15.39:0/1574781932 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x558cb2834970 0x558cb2836e00 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:21.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.733+0000 7fd895879740 1 --2- 172.21.15.39:0/1574781932 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558cb2824420 0x558cb28300d0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:21.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.733+0000 7fd895879740 1 --2- 172.21.15.39:0/1574781932 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558cb281f2b0 0x558cb2829090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:21.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.733+0000 7fd895879740 1 --2- 172.21.15.39:0/1574781932 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558cb281e8c0 0x558cb2828b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:21.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.733+0000 7fd895879740 1 -- 172.21.15.39:0/1574781932 >> 172.21.15.39:0/1574781932 conn(0x558cb2819e30 msgr2=0x558cb28191f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:17:21.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.733+0000 7fd879600640 0 client.27089 ms_handle_reset on v2:172.21.15.39:6840/3515326042 2024-03-18T16:17:21.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.734+0000 7fd895879740 1 -- 172.21.15.39:0/1574781932 shutdown_connections 2024-03-18T16:17:21.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.734+0000 7fd895879740 1 -- 172.21.15.39:0/1574781932 wait complete. 2024-03-18T16:17:21.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.734+0000 7fd895879740 20 client.27089 trim_cache size 0 max 16384 2024-03-18T16:17:21.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.761+0000 7fd895879740 1 Processor -- start 2024-03-18T16:17:21.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.761+0000 7fd895879740 1 -- start start 2024-03-18T16:17:21.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.761+0000 7fd895879740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558cb281e8c0 0x558cb2827840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:21.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.761+0000 7fd895879740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558cb281f2b0 0x558cb2830090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:21.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.761+0000 7fd895879740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558cb2824420 0x558cb2827d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:21.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.761+0000 7fd895879740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fd88c1b9390 con 0x558cb281e8c0 2024-03-18T16:17:21.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.761+0000 7fd895879740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fd88c1baed0 con 0x558cb2824420 2024-03-18T16:17:21.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.761+0000 7fd895879740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fd88c1b9a60 con 0x558cb281f2b0 2024-03-18T16:17:21.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.762+0000 7fd883e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558cb281f2b0 0x558cb2830090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:21.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.762+0000 7fd883e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558cb281f2b0 0x558cb2830090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:53716/0 (socket says 172.21.15.39:53716) 2024-03-18T16:17:21.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.762+0000 7fd883e00640 1 -- 172.21.15.39:0/3397630460 learned_addr learned my addr 172.21.15.39:0/3397630460 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:17:21.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.762+0000 7fd890c00640 1 --2- 172.21.15.39:0/3397630460 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558cb2824420 0x558cb2827d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:21.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.762+0000 7fd883e00640 1 -- 172.21.15.39:0/3397630460 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558cb2824420 msgr2=0x558cb2827d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:17:21.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.762+0000 7fd883e00640 1 --2- 172.21.15.39:0/3397630460 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558cb2824420 0x558cb2827d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:21.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.762+0000 7fd883e00640 1 -- 172.21.15.39:0/3397630460 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558cb281e8c0 msgr2=0x558cb2827840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:17:21.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.762+0000 7fd883e00640 1 --2- 172.21.15.39:0/3397630460 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558cb281e8c0 0x558cb2827840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:21.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.762+0000 7fd883e00640 1 -- 172.21.15.39:0/3397630460 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558cb2815c10 con 0x558cb281f2b0 2024-03-18T16:17:21.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.763+0000 7fd883e00640 1 --2- 172.21.15.39:0/3397630460 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558cb281f2b0 0x558cb2830090 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fd8740b9630 tx=0x7fd8740bc7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e715dfaafdcaa60 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:17:21.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.763+0000 7fd87be00640 1 -- 172.21.15.39:0/3397630460 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8740d5070 con 0x558cb281f2b0 2024-03-18T16:17:21.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.763+0000 7fd87be00640 1 -- 172.21.15.39:0/3397630460 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd8740c0d00 con 0x558cb281f2b0 2024-03-18T16:17:21.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.763+0000 7fd87be00640 1 -- 172.21.15.39:0/3397630460 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd87404b040 con 0x558cb281f2b0 2024-03-18T16:17:21.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.764+0000 7fd895879740 1 -- 172.21.15.39:0/3397630460 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558cb281f2b0 msgr2=0x558cb2830090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:21.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.764+0000 7fd895879740 1 --2- 172.21.15.39:0/3397630460 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558cb281f2b0 0x558cb2830090 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fd8740b9630 tx=0x7fd8740bc7a0 comp rx=0 tx=0).stop 2024-03-18T16:17:21.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.764+0000 7fd895879740 1 -- 172.21.15.39:0/3397630460 shutdown_connections 2024-03-18T16:17:21.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.764+0000 7fd895879740 1 --2- 172.21.15.39:0/3397630460 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558cb2824420 0x558cb2827d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:21.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.764+0000 7fd895879740 1 --2- 172.21.15.39:0/3397630460 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558cb281f2b0 0x558cb2830090 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:21.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.764+0000 7fd895879740 1 --2- 172.21.15.39:0/3397630460 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558cb281e8c0 0x558cb2827840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:21.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.764+0000 7fd895879740 1 -- 172.21.15.39:0/3397630460 >> 172.21.15.39:0/3397630460 conn(0x558cb2819e30 msgr2=0x558cb2823b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:17:21.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.765+0000 7fd895879740 1 -- 172.21.15.39:0/3397630460 shutdown_connections 2024-03-18T16:17:21.768 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.765+0000 7fd895879740 1 -- 172.21.15.39:0/3397630460 wait complete. 2024-03-18T16:17:21.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.766+0000 7fd895879740 1 -- start start 2024-03-18T16:17:21.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.766+0000 7fd895879740 1 Processor -- start 2024-03-18T16:17:21.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.767+0000 7fd895879740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558cb281e8c0 0x558cb282b220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:21.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.767+0000 7fd895879740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558cb2824420 0x558cb282b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:21.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.767+0000 7fd895879740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558cb282e780 0x558cb282ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:21.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.767+0000 7fd895879740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fd88c1ba800 con 0x558cb282e780 2024-03-18T16:17:21.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.767+0000 7fd895879740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fd88c1ba130 con 0x558cb2824420 2024-03-18T16:17:21.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.767+0000 7fd895879740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fd88c1b7370 con 0x558cb281e8c0 2024-03-18T16:17:21.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.767+0000 7fd895879740 10 client.0 _ll_register_callbacks cb 0x7fd884018740 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:17:21.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.768+0000 7fd890c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558cb282e780 0x558cb282ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:21.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.768+0000 7fd890c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558cb282e780 0x558cb282ec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:37340/0 (socket says 172.21.15.39:37340) 2024-03-18T16:17:21.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.768+0000 7fd88be00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558cb281e8c0 0x558cb282b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:21.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.768+0000 7fd890c00640 1 -- 172.21.15.39:0/1907425283 learned_addr learned my addr 172.21.15.39:0/1907425283 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:17:21.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.768+0000 7fd883e00640 1 --2- 172.21.15.39:0/1907425283 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558cb2824420 0x558cb282b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:21.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.769+0000 7fd883e00640 1 -- 172.21.15.39:0/1907425283 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558cb281e8c0 msgr2=0x558cb282b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:17:21.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.769+0000 7fd883e00640 1 --2- 172.21.15.39:0/1907425283 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558cb281e8c0 0x558cb282b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:21.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.769+0000 7fd883e00640 1 -- 172.21.15.39:0/1907425283 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558cb282e780 msgr2=0x558cb282ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:17:21.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.769+0000 7fd883e00640 1 --2- 172.21.15.39:0/1907425283 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558cb282e780 0x558cb282ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:21.772 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.769+0000 7fd883e00640 1 -- 172.21.15.39:0/1907425283 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558cb282f140 con 0x558cb2824420 2024-03-18T16:17:21.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.773+0000 7fd879600640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:17:21.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.773+0000 7fd883e00640 1 --2- 172.21.15.39:0/1907425283 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558cb2824420 0x558cb282b760 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fd8740c4e20 tx=0x7fd8740bc8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7da4bca31e27fae3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:17:21.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.773+0000 7fd879600640 1 -- 172.21.15.39:0/1907425283 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8740d9020 con 0x558cb2824420 2024-03-18T16:17:21.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.773+0000 7fd879600640 1 -- 172.21.15.39:0/1907425283 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fd8740db2c0 con 0x558cb2824420 2024-03-18T16:17:21.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.773+0000 7fd895879740 1 -- 172.21.15.39:0/1907425283 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd8740c0d00 con 0x558cb2824420 2024-03-18T16:17:21.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.774+0000 7fd879600640 1 -- 172.21.15.39:0/1907425283 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fd8740d5050 con 0x558cb2824420 2024-03-18T16:17:21.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.774+0000 7fd895879740 10 client.27095 fetch_fsmap learned FSMap version 44 2024-03-18T16:17:21.776 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.774+0000 7fd895879740 1 -- 172.21.15.39:0/1907425283 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x558cb2830440 con 0x558cb2824420 2024-03-18T16:17:21.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.774+0000 7fd879600640 1 -- 172.21.15.39:0/1907425283 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd8740c0d00 con 0x558cb2824420 2024-03-18T16:17:21.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.774+0000 7fd879600640 1 -- 172.21.15.39:0/1907425283 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7fd8740035f0 con 0x558cb2824420 2024-03-18T16:17:21.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.774+0000 7fd895879740 10 client.27095 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:17:21.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.774+0000 7fd895879740 10 client.27095 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-03-18T16:17:21.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.775+0000 7fd895879740 20 client.27095 populate_metadata read hostname 'smithi039' 2024-03-18T16:17:21.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.775+0000 7fd895879740 1 --2- 172.21.15.39:0/1907425283 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x558cb28ac3f0 0x558cb28ae7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:21.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.775+0000 7fd895879740 4 client.27095 mds_command: new command op to 14463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:17:21.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.775+0000 7fd895879740 1 -- 172.21.15.39:0/1907425283 --> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558cb278ace0 con 0x558cb28ac3f0 2024-03-18T16:17:21.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.776+0000 7fd88be00640 1 --2- 172.21.15.39:0/1907425283 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x558cb28ac3f0 0x558cb28ae7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:21.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.779+0000 7fd88be00640 1 --2- 172.21.15.39:0/1907425283 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x558cb28ac3f0 0x558cb28ae7e0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=4aa65426b12559cc server_cookie=d2d6f75972b9b521 in_seq=0 out_seq=0 2024-03-18T16:17:21.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.779+0000 7fd879600640 10 client.27095 ms_handle_connect on v2:172.21.15.39:6840/3515326042 2024-03-18T16:17:21.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.782+0000 7fd879600640 1 -- 172.21.15.39:0/1907425283 <== mds.4 v2:172.21.15.39:6840/3515326042 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558cb278ace0 con 0x558cb28ac3f0 2024-03-18T16:17:21.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.783+0000 7fd879600640 10 client.27095 handle_command_reply: tid=0 2024-03-18T16:17:21.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.783+0000 7fd895879740 1 client.27095 shutdown 2024-03-18T16:17:21.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.783+0000 7fd895879740 1 -- 172.21.15.39:0/1907425283 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558cb2824420 msgr2=0x558cb282b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:21.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.783+0000 7fd895879740 1 --2- 172.21.15.39:0/1907425283 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558cb2824420 0x558cb282b760 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fd8740c4e20 tx=0x7fd8740bc8d0 comp rx=0 tx=0).stop 2024-03-18T16:17:21.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.784+0000 7fd895879740 1 -- 172.21.15.39:0/1907425283 shutdown_connections 2024-03-18T16:17:21.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.784+0000 7fd895879740 1 --2- 172.21.15.39:0/1907425283 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x558cb28ac3f0 0x558cb28ae7e0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:21.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.784+0000 7fd895879740 1 --2- 172.21.15.39:0/1907425283 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558cb2824420 0x558cb282b760 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:21.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.784+0000 7fd895879740 1 --2- 172.21.15.39:0/1907425283 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558cb282e780 0x558cb282ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:21.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.784+0000 7fd895879740 1 --2- 172.21.15.39:0/1907425283 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558cb281e8c0 0x558cb282b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:21.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.784+0000 7fd895879740 1 -- 172.21.15.39:0/1907425283 >> 172.21.15.39:0/1907425283 conn(0x558cb2819e30 msgr2=0x558cb2827840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:17:21.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.784+0000 7fd879600640 0 client.27095 ms_handle_reset on v2:172.21.15.39:6840/3515326042 2024-03-18T16:17:21.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.785+0000 7fd895879740 1 -- 172.21.15.39:0/1907425283 shutdown_connections 2024-03-18T16:17:21.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.786+0000 7fd895879740 1 -- 172.21.15.39:0/1907425283 wait complete. 2024-03-18T16:17:21.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.787+0000 7fd895879740 20 client.27095 trim_cache size 0 max 16384 2024-03-18T16:17:21.790 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:17:21.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.788+0000 7fd892c00640 1 -- 172.21.15.39:0/3284750115 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd8700a2a00 msgr2=0x7fd8700a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:21.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.788+0000 7fd892c00640 1 --2- 172.21.15.39:0/3284750115 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd8700a2a00 0x7fd8700a4ec0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x558cb281fb10 tx=0x7fd88400d040 comp rx=0 tx=0).stop 2024-03-18T16:17:21.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.788+0000 7fd892c00640 1 -- 172.21.15.39:0/3284750115 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd88c10b8c0 msgr2=0x7fd88c1a7650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:21.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.788+0000 7fd892c00640 1 --2- 172.21.15.39:0/3284750115 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd88c10b8c0 0x7fd88c1a7650 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fd87402c180 tx=0x7fd87402c6d0 comp rx=0 tx=0).stop 2024-03-18T16:17:21.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.789+0000 7fd892c00640 1 -- 172.21.15.39:0/3284750115 shutdown_connections 2024-03-18T16:17:21.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.789+0000 7fd892c00640 1 --2- 172.21.15.39:0/3284750115 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd88c10e320 0x7fd88c1adf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:21.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.789+0000 7fd892c00640 1 --2- 172.21.15.39:0/3284750115 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd8700a2a00 0x7fd8700a4ec0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:21.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.789+0000 7fd892c00640 1 --2- 172.21.15.39:0/3284750115 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd88c10b8c0 0x7fd88c1a7650 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:21.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.789+0000 7fd892c00640 1 --2- 172.21.15.39:0/3284750115 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd88c06a600 0x7fd88c1a7110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:21.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.789+0000 7fd892c00640 1 -- 172.21.15.39:0/3284750115 >> 172.21.15.39:0/3284750115 conn(0x7fd88c06e9c0 msgr2=0x7fd88c1118f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:17:21.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.790+0000 7fd892c00640 1 -- 172.21.15.39:0/3284750115 shutdown_connections 2024-03-18T16:17:21.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:21.790+0000 7fd892c00640 1 -- 172.21.15.39:0/3284750115 wait complete. 2024-03-18T16:17:21.821 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-18T16:17:22.086 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:17:21 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/759415094' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:17:22.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.150+0000 7fdd0f000640 1 -- 172.21.15.39:0/721621850 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdd08104bf0 msgr2=0x7fdd08105070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:22.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.150+0000 7fdd0f000640 1 --2- 172.21.15.39:0/721621850 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdd08104bf0 0x7fdd08105070 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fdcf4009900 tx=0x7fdcf402bc40 comp rx=0 tx=0).stop 2024-03-18T16:17:22.153 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.152+0000 7fdd0f000640 1 -- 172.21.15.39:0/721621850 shutdown_connections 2024-03-18T16:17:22.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.152+0000 7fdd0f000640 1 --2- 172.21.15.39:0/721621850 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdd081055b0 0x7fdd0810be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:22.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.152+0000 7fdd0f000640 1 --2- 172.21.15.39:0/721621850 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdd08104bf0 0x7fdd08105070 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:22.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.152+0000 7fdd0f000640 1 --2- 172.21.15.39:0/721621850 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdd08103a00 0x7fdd08103e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:22.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.152+0000 7fdd0f000640 1 -- 172.21.15.39:0/721621850 >> 172.21.15.39:0/721621850 conn(0x7fdd080fef90 msgr2=0x7fdd081013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:17:22.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.152+0000 7fdd0f000640 1 -- 172.21.15.39:0/721621850 shutdown_connections 2024-03-18T16:17:22.154 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.152+0000 7fdd0f000640 1 -- 172.21.15.39:0/721621850 wait complete. 2024-03-18T16:17:22.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.153+0000 7fdd0f000640 1 Processor -- start 2024-03-18T16:17:22.155 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.154+0000 7fdd0f000640 1 -- start start 2024-03-18T16:17:22.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.154+0000 7fdd0f000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdd08103a00 0x7fdd0819df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:22.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.154+0000 7fdd0f000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdd08104bf0 0x7fdd0819e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:22.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.154+0000 7fdd0f000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdd081055b0 0x7fdd081a5500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:22.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.154+0000 7fdd0f000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fdd08076670 con 0x7fdd081055b0 2024-03-18T16:17:22.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.154+0000 7fdd0f000640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fdd080764f0 con 0x7fdd08104bf0 2024-03-18T16:17:22.156 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.154+0000 7fdd0f000640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fdd080767f0 con 0x7fdd08103a00 2024-03-18T16:17:22.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.155+0000 7fdd07e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdd08103a00 0x7fdd0819df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:22.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.155+0000 7fdd07e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdd08103a00 0x7fdd0819df50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:53752/0 (socket says 172.21.15.39:53752) 2024-03-18T16:17:22.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.155+0000 7fdd07e00640 1 -- 172.21.15.39:0/4066800230 learned_addr learned my addr 172.21.15.39:0/4066800230 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:17:22.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.155+0000 7fdd07400640 1 --2- 172.21.15.39:0/4066800230 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdd08104bf0 0x7fdd0819e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:22.157 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.156+0000 7fdd07e00640 1 -- 172.21.15.39:0/4066800230 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdd08104bf0 msgr2=0x7fdd0819e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:22.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.156+0000 7fdd07e00640 1 --2- 172.21.15.39:0/4066800230 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdd08104bf0 0x7fdd0819e490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:22.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.156+0000 7fdd07e00640 1 -- 172.21.15.39:0/4066800230 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdd081055b0 msgr2=0x7fdd081a5500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:22.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.156+0000 7fdd07e00640 1 --2- 172.21.15.39:0/4066800230 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdd081055b0 0x7fdd081a5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:22.158 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.156+0000 7fdd07e00640 1 -- 172.21.15.39:0/4066800230 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd081a5b70 con 0x7fdd08103a00 2024-03-18T16:17:22.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.157+0000 7fdd07e00640 1 --2- 172.21.15.39:0/4066800230 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdd08103a00 0x7fdd0819df50 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fdcf800e9c0 tx=0x7fdcf800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:17:22.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.157+0000 7fdd04c00640 1 -- 172.21.15.39:0/4066800230 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcf800cde0 con 0x7fdd08103a00 2024-03-18T16:17:22.159 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.157+0000 7fdd0f000640 1 -- 172.21.15.39:0/4066800230 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd081a5e60 con 0x7fdd08103a00 2024-03-18T16:17:22.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.158+0000 7fdd04c00640 1 -- 172.21.15.39:0/4066800230 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdcf8010910 con 0x7fdd08103a00 2024-03-18T16:17:22.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.158+0000 7fdd04c00640 1 -- 172.21.15.39:0/4066800230 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcf8018a10 con 0x7fdd08103a00 2024-03-18T16:17:22.160 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.158+0000 7fdd0f000640 1 -- 172.21.15.39:0/4066800230 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd081a6320 con 0x7fdd08103a00 2024-03-18T16:17:22.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.165+0000 7fdd11d85740 1 Processor -- start 2024-03-18T16:17:22.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.165+0000 7fdd11d85740 1 -- start start 2024-03-18T16:17:22.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.165+0000 7fdd11d85740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56011d8cb8c0 0x56011d8cbce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:22.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.165+0000 7fdd11d85740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56011d8cc2b0 0x56011d8d0db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:22.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.165+0000 7fdd11d85740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56011d8d1420 0x56011d8d38b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:22.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.166+0000 7fdd04c00640 1 -- 172.21.15.39:0/4066800230 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fdcf801f020 con 0x7fdd08103a00 2024-03-18T16:17:22.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.166+0000 7fdd11d85740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x56011d8933b0 con 0x56011d8d1420 2024-03-18T16:17:22.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.166+0000 7fdd11d85740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x56011d830450 con 0x56011d8cb8c0 2024-03-18T16:17:22.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.166+0000 7fdd11d85740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x56011d7d8cb0 con 0x56011d8cc2b0 2024-03-18T16:17:22.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.166+0000 7fdd0d000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56011d8cb8c0 0x56011d8cbce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:22.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.166+0000 7fdd0d000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56011d8cb8c0 0x56011d8cbce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:41784/0 (socket says 172.21.15.39:41784) 2024-03-18T16:17:22.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.166+0000 7fdd0d000640 1 -- 172.21.15.39:0/137719573 learned_addr learned my addr 172.21.15.39:0/137719573 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:17:22.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.167+0000 7fdd07e00640 1 --2- 172.21.15.39:0/137719573 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56011d8cc2b0 0x56011d8d0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:22.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.167+0000 7fdd07400640 1 --2- 172.21.15.39:0/137719573 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56011d8d1420 0x56011d8d38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:22.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.167+0000 7fdd0d000640 1 -- 172.21.15.39:0/137719573 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56011d8cc2b0 msgr2=0x56011d8d0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:17:22.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.167+0000 7fdd0d000640 1 --2- 172.21.15.39:0/137719573 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56011d8cc2b0 0x56011d8d0db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:22.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.167+0000 7fdd0d000640 1 -- 172.21.15.39:0/137719573 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56011d8d1420 msgr2=0x56011d8d38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:17:22.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.167+0000 7fdd0d000640 1 --2- 172.21.15.39:0/137719573 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56011d8d1420 0x56011d8d38b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:22.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.167+0000 7fdd0d000640 1 -- 172.21.15.39:0/137719573 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56011d7fa1f0 con 0x56011d8cb8c0 2024-03-18T16:17:22.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.168+0000 7fdd04c00640 1 --2- 172.21.15.39:0/4066800230 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fdcec0a2900 0x7fdcec0a4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:22.170 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.168+0000 7fdd0d000640 1 --2- 172.21.15.39:0/137719573 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56011d8cb8c0 0x56011d8cbce0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fdd00009d00 tx=0x7fdd0000b420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a714d5c5f56cd5cf server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:17:22.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.169+0000 7fdcfd600640 1 -- 172.21.15.39:0/137719573 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd0000bd40 con 0x56011d8cb8c0 2024-03-18T16:17:22.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.169+0000 7fdcfd600640 1 -- 172.21.15.39:0/137719573 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd00017070 con 0x56011d8cb8c0 2024-03-18T16:17:22.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.169+0000 7fdcfd600640 1 -- 172.21.15.39:0/137719573 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd000116c0 con 0x56011d8cb8c0 2024-03-18T16:17:22.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.169+0000 7fdd04c00640 1 -- 172.21.15.39:0/4066800230 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fdcf80cf100 con 0x7fdd08103a00 2024-03-18T16:17:22.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.169+0000 7fdd07e00640 1 --2- 172.21.15.39:0/4066800230 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fdcec0a2900 0x7fdcec0a4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:22.172 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.171+0000 7fdd11d85740 1 -- 172.21.15.39:0/137719573 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56011d8cb8c0 msgr2=0x56011d8cbce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:22.172 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.171+0000 7fdd11d85740 1 --2- 172.21.15.39:0/137719573 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56011d8cb8c0 0x56011d8cbce0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fdd00009d00 tx=0x7fdd0000b420 comp rx=0 tx=0).stop 2024-03-18T16:17:22.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.171+0000 7fdd11d85740 1 -- 172.21.15.39:0/137719573 shutdown_connections 2024-03-18T16:17:22.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.171+0000 7fdd07e00640 1 --2- 172.21.15.39:0/4066800230 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fdcec0a2900 0x7fdcec0a4dc0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fdcf800eff0 tx=0x7fdcf80d2ac0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:17:22.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.171+0000 7fdd11d85740 1 --2- 172.21.15.39:0/137719573 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56011d8d1420 0x56011d8d38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:22.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.172+0000 7fdd11d85740 1 --2- 172.21.15.39:0/137719573 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56011d8cc2b0 0x56011d8d0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:22.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.172+0000 7fdd11d85740 1 --2- 172.21.15.39:0/137719573 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56011d8cb8c0 0x56011d8cbce0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:22.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.172+0000 7fdd11d85740 1 -- 172.21.15.39:0/137719573 >> 172.21.15.39:0/137719573 conn(0x56011d8c6e30 msgr2=0x56011d8c7230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:17:22.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.172+0000 7fdd11d85740 1 -- 172.21.15.39:0/137719573 shutdown_connections 2024-03-18T16:17:22.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.172+0000 7fdd11d85740 1 -- 172.21.15.39:0/137719573 wait complete. 2024-03-18T16:17:22.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.173+0000 7fdd11d85740 1 -- start start 2024-03-18T16:17:22.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.173+0000 7fdd11d85740 1 Processor -- start 2024-03-18T16:17:22.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.175+0000 7fdd11d85740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56011d8cb8c0 0x56011d8d5c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:22.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.175+0000 7fdd0d000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56011d8cb8c0 0x56011d8d5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:22.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.176+0000 7fdd11d85740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56011d8cc2b0 0x56011d8d6140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:22.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.176+0000 7fdd0d000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56011d8cb8c0 0x56011d8d5c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:37374/0 (socket says 172.21.15.39:37374) 2024-03-18T16:17:22.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.176+0000 7fdd0d000640 1 -- 172.21.15.39:0/3283028981 learned_addr learned my addr 172.21.15.39:0/3283028981 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:17:22.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.176+0000 7fdd11d85740 1 --2- 172.21.15.39:0/3283028981 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56011d8d1420 0x56011d8dd110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:22.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.177+0000 7fdd07400640 1 --2- 172.21.15.39:0/3283028981 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56011d8cc2b0 0x56011d8d6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:22.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.177+0000 7fdd11d85740 1 -- 172.21.15.39:0/3283028981 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x56011d7ec110 con 0x56011d8cb8c0 2024-03-18T16:17:22.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.177+0000 7fdd11d85740 1 -- 172.21.15.39:0/3283028981 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x56011d82c4f0 con 0x56011d8d1420 2024-03-18T16:17:22.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.177+0000 7fdd11d85740 1 -- 172.21.15.39:0/3283028981 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x56011d8d43d0 con 0x56011d8cc2b0 2024-03-18T16:17:22.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.178+0000 7fdd07400640 1 -- 172.21.15.39:0/3283028981 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56011d8d1420 msgr2=0x56011d8dd110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:17:22.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.178+0000 7fdd0d000640 1 --2- 172.21.15.39:0/3283028981 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56011d8d1420 0x56011d8dd110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:22.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.178+0000 7fdd11d85740 10 client.0 _ll_register_callbacks cb 0x56011d7d9b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:17:22.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.179+0000 7fdd07400640 1 --2- 172.21.15.39:0/3283028981 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56011d8d1420 0x56011d8dd110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:22.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.179+0000 7fdd07400640 1 -- 172.21.15.39:0/3283028981 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56011d8cb8c0 msgr2=0x56011d8d5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:17:22.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.179+0000 7fdd07400640 1 --2- 172.21.15.39:0/3283028981 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56011d8cb8c0 0x56011d8d5c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:22.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.179+0000 7fdd07400640 1 -- 172.21.15.39:0/3283028981 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56011d5443f0 con 0x56011d8cc2b0 2024-03-18T16:17:22.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.180+0000 7fdd0d000640 1 --2- 172.21.15.39:0/3283028981 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56011d8d1420 0x56011d8dd110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:17:22.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.180+0000 7fdd11d85740 1 -- 172.21.15.39:0/3283028981 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdd00017070 con 0x56011d8cc2b0 2024-03-18T16:17:22.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.180+0000 7fdd07400640 1 --2- 172.21.15.39:0/3283028981 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56011d8cc2b0 0x56011d8d6140 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fdcf4009850 tx=0x7fdcf4005d90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b989876f0235222 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:17:22.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.182+0000 7fdcf2000640 10 client.27079 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:17:22.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.182+0000 7fdcf2000640 1 -- 172.21.15.39:0/3283028981 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcf4005490 con 0x56011d8cc2b0 2024-03-18T16:17:22.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.182+0000 7fdcf2000640 1 -- 172.21.15.39:0/3283028981 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fdcf40046c0 con 0x56011d8cc2b0 2024-03-18T16:17:22.184 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.183+0000 7fdcf2000640 1 -- 172.21.15.39:0/3283028981 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fdcf4004920 con 0x56011d8cc2b0 2024-03-18T16:17:22.185 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.183+0000 7fdd11d85740 10 client.27079 fetch_fsmap learned FSMap version 44 2024-03-18T16:17:22.185 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.183+0000 7fdd11d85740 1 -- 172.21.15.39:0/3283028981 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({fsmap=44}) -- 0x56011d60e7d0 con 0x56011d8cc2b0 2024-03-18T16:17:22.185 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.183+0000 7fdcf2000640 1 -- 172.21.15.39:0/3283028981 <== mon.1 v2:172.21.15.196:3300/0 4 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7fdcf4004ac0 con 0x56011d8cc2b0 2024-03-18T16:17:22.185 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.184+0000 7fdd11d85740 10 client.27079 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:17:22.185 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.184+0000 7fdd11d85740 10 client.27079 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-18T16:17:22.185 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.184+0000 7fdd11d85740 20 client.27079 populate_metadata read hostname 'smithi039' 2024-03-18T16:17:22.186 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.184+0000 7fdd11d85740 1 --2- 172.21.15.39:0/3283028981 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x56011d8e1770 0x56011d8e3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:22.186 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.184+0000 7fdd11d85740 4 client.27079 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:17:22.186 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.184+0000 7fdd11d85740 1 -- 172.21.15.39:0/3283028981 --> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56011d83d990 con 0x56011d8e1770 2024-03-18T16:17:22.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.186+0000 7fdd0d000640 1 --2- 172.21.15.39:0/3283028981 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x56011d8e1770 0x56011d8e3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:22.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.187+0000 7fdd0d000640 1 --2- 172.21.15.39:0/3283028981 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x56011d8e1770 0x56011d8e3ba0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=f1f251f3efff56d8 server_cookie=d5fa3c3e672cd18b in_seq=0 out_seq=0 2024-03-18T16:17:22.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.187+0000 7fdcf2000640 10 client.27079 ms_handle_connect on v2:172.21.15.194:6836/1301464246 2024-03-18T16:17:22.189 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.187+0000 7fdcf2000640 1 -- 172.21.15.39:0/3283028981 <== mds.3 v2:172.21.15.194:6836/1301464246 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x56011d83d990 con 0x56011d8e1770 2024-03-18T16:17:22.189 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.187+0000 7fdcf2000640 10 client.27079 handle_command_reply: tid=0 2024-03-18T16:17:22.189 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.188+0000 7fdd11d85740 1 client.27079 shutdown 2024-03-18T16:17:22.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.188+0000 7fdd11d85740 1 -- 172.21.15.39:0/3283028981 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56011d8cc2b0 msgr2=0x56011d8d6140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:22.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.188+0000 7fdd11d85740 1 --2- 172.21.15.39:0/3283028981 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56011d8cc2b0 0x56011d8d6140 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fdcf4009850 tx=0x7fdcf4005d90 comp rx=0 tx=0).stop 2024-03-18T16:17:22.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.188+0000 7fdd11d85740 1 -- 172.21.15.39:0/3283028981 shutdown_connections 2024-03-18T16:17:22.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.188+0000 7fdd11d85740 1 --2- 172.21.15.39:0/3283028981 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x56011d8e1770 0x56011d8e3ba0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:22.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.188+0000 7fdd11d85740 1 --2- 172.21.15.39:0/3283028981 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56011d8d1420 0x56011d8dd110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:22.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.188+0000 7fdd11d85740 1 --2- 172.21.15.39:0/3283028981 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56011d8cc2b0 0x56011d8d6140 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:22.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.188+0000 7fdd11d85740 1 --2- 172.21.15.39:0/3283028981 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56011d8cb8c0 0x56011d8d5c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:22.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.188+0000 7fdcf2000640 0 client.27079 ms_handle_reset on v2:172.21.15.194:6836/1301464246 2024-03-18T16:17:22.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.188+0000 7fdd11d85740 1 -- 172.21.15.39:0/3283028981 >> 172.21.15.39:0/3283028981 conn(0x56011d8c6e30 msgr2=0x56011d8c61f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:17:22.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.189+0000 7fdd11d85740 1 -- 172.21.15.39:0/3283028981 shutdown_connections 2024-03-18T16:17:22.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.189+0000 7fdd11d85740 1 -- 172.21.15.39:0/3283028981 wait complete. 2024-03-18T16:17:22.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.190+0000 7fdd11d85740 20 client.27079 trim_cache size 0 max 16384 2024-03-18T16:17:22.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.225+0000 7fdd11d85740 1 Processor -- start 2024-03-18T16:17:22.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.225+0000 7fdd11d85740 1 -- start start 2024-03-18T16:17:22.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.226+0000 7fdd11d85740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56011d8cb8c0 0x56011d8cd7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:22.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.226+0000 7fdd11d85740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56011d8cc2b0 0x56011d8dd0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:22.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.226+0000 7fdd11d85740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56011d8d1420 0x56011d8cdcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:22.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.226+0000 7fdd0d000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56011d8cb8c0 0x56011d8cd7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:22.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.226+0000 7fdd0d000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56011d8cb8c0 0x56011d8cd7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:37376/0 (socket says 172.21.15.39:37376) 2024-03-18T16:17:22.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.226+0000 7fdd11d85740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fdd081b09e0 con 0x56011d8cb8c0 2024-03-18T16:17:22.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.226+0000 7fdd11d85740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fdd081b2520 con 0x56011d8cc2b0 2024-03-18T16:17:22.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.226+0000 7fdd11d85740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fdd081b10b0 con 0x56011d8d1420 2024-03-18T16:17:22.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.227+0000 7fdd07400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56011d8cc2b0 0x56011d8dd0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:22.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:17:21 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/759415094' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:17:22.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.227+0000 7fdd07400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56011d8cc2b0 0x56011d8dd0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:41812/0 (socket says 172.21.15.39:41812) 2024-03-18T16:17:22.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.227+0000 7fdd07400640 1 -- 172.21.15.39:0/1449335350 learned_addr learned my addr 172.21.15.39:0/1449335350 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:17:22.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.227+0000 7fdd0d000640 1 --2- 172.21.15.39:0/1449335350 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56011d8d1420 0x56011d8cdcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:22.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.227+0000 7fdd07400640 1 -- 172.21.15.39:0/1449335350 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56011d8d1420 msgr2=0x56011d8cdcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:17:22.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.227+0000 7fdd07400640 1 --2- 172.21.15.39:0/1449335350 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56011d8d1420 0x56011d8cdcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:22.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.227+0000 7fdd07400640 1 -- 172.21.15.39:0/1449335350 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56011d8cb8c0 msgr2=0x56011d8cd7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:17:22.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.227+0000 7fdd07400640 1 --2- 172.21.15.39:0/1449335350 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56011d8cb8c0 0x56011d8cd7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:22.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.227+0000 7fdd07400640 1 -- 172.21.15.39:0/1449335350 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56011d8c2c10 con 0x56011d8cc2b0 2024-03-18T16:17:22.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.228+0000 7fdd07400640 1 --2- 172.21.15.39:0/1449335350 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56011d8cc2b0 0x56011d8dd0d0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fdcf4009800 tx=0x7fdcf4040000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a1c6861f5a1a4f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:17:22.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.228+0000 7fdcfcc00640 1 -- 172.21.15.39:0/1449335350 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcf4004ac0 con 0x56011d8cc2b0 2024-03-18T16:17:22.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.228+0000 7fdcfcc00640 1 -- 172.21.15.39:0/1449335350 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd00017070 con 0x56011d8cc2b0 2024-03-18T16:17:22.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.229+0000 7fdcfcc00640 1 -- 172.21.15.39:0/1449335350 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcf4049480 con 0x56011d8cc2b0 2024-03-18T16:17:22.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.233+0000 7fdd11d85740 1 -- 172.21.15.39:0/1449335350 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56011d8cc2b0 msgr2=0x56011d8dd0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:22.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.233+0000 7fdd11d85740 1 --2- 172.21.15.39:0/1449335350 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56011d8cc2b0 0x56011d8dd0d0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fdcf4009800 tx=0x7fdcf4040000 comp rx=0 tx=0).stop 2024-03-18T16:17:22.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.233+0000 7fdd11d85740 1 -- 172.21.15.39:0/1449335350 shutdown_connections 2024-03-18T16:17:22.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.233+0000 7fdd11d85740 1 --2- 172.21.15.39:0/1449335350 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56011d8cc2b0 0x56011d8dd0d0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:22.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.233+0000 7fdd11d85740 1 --2- 172.21.15.39:0/1449335350 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56011d8d1420 0x56011d8cdcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:22.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.233+0000 7fdd11d85740 1 --2- 172.21.15.39:0/1449335350 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56011d8cb8c0 0x56011d8cd7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:22.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.233+0000 7fdd11d85740 1 -- 172.21.15.39:0/1449335350 >> 172.21.15.39:0/1449335350 conn(0x56011d8c6e30 msgr2=0x56011d8c7210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:17:22.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.233+0000 7fdd11d85740 1 -- 172.21.15.39:0/1449335350 shutdown_connections 2024-03-18T16:17:22.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.233+0000 7fdd11d85740 1 -- 172.21.15.39:0/1449335350 wait complete. 2024-03-18T16:17:22.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.234+0000 7fdd11d85740 1 -- start start 2024-03-18T16:17:22.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.234+0000 7fdd11d85740 1 Processor -- start 2024-03-18T16:17:22.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.235+0000 7fdd11d85740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56011d8cb8c0 0x56011d8daaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:22.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.235+0000 7fdd11d85740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56011d8cc2b0 0x56011d8db030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:22.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.235+0000 7fdd11d85740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56011d8d1420 0x56011d8dbff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:22.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.235+0000 7fdd11d85740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fdd081b1e50 con 0x56011d8d1420 2024-03-18T16:17:22.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.235+0000 7fdd11d85740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fdd081b1780 con 0x56011d8cc2b0 2024-03-18T16:17:22.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.235+0000 7fdd11d85740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fdd081ae9c0 con 0x56011d8cb8c0 2024-03-18T16:17:22.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.235+0000 7fdd11d85740 10 client.0 _ll_register_callbacks cb 0x7fdcf8005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:17:22.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.235+0000 7fdd0d000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56011d8cb8c0 0x56011d8daaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:22.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.235+0000 7fdd0d000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56011d8cb8c0 0x56011d8daaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:53792/0 (socket says 172.21.15.39:53792) 2024-03-18T16:17:22.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.235+0000 7fdd0d000640 1 -- 172.21.15.39:0/2295787006 learned_addr learned my addr 172.21.15.39:0/2295787006 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:17:22.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.236+0000 7fdd07400640 1 --2- 172.21.15.39:0/2295787006 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56011d8cc2b0 0x56011d8db030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:22.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.237+0000 7fdd0d000640 1 -- 172.21.15.39:0/2295787006 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56011d8cc2b0 msgr2=0x56011d8db030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:17:22.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.237+0000 7fdd0d000640 1 --2- 172.21.15.39:0/2295787006 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56011d8cc2b0 0x56011d8db030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:22.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.237+0000 7fdd0d000640 1 -- 172.21.15.39:0/2295787006 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56011d8d1420 msgr2=0x56011d8dbff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:17:22.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.237+0000 7fdd0d000640 1 --2- 172.21.15.39:0/2295787006 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56011d8d1420 0x56011d8dbff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:22.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.237+0000 7fdd0d000640 1 -- 172.21.15.39:0/2295787006 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56011d8dc5e0 con 0x56011d8cb8c0 2024-03-18T16:17:22.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.237+0000 7fdd07400640 1 --2- 172.21.15.39:0/2295787006 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56011d8cc2b0 0x56011d8db030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:17:22.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:17:21 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/759415094' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:17:22.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.239+0000 7fdd0d000640 1 --2- 172.21.15.39:0/2295787006 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56011d8cb8c0 0x56011d8daaf0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fdd00009f50 tx=0x7fdd00009720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=49f178bef097f6a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:17:22.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.239+0000 7fdcf2000640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:17:22.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.240+0000 7fdcf2000640 1 -- 172.21.15.39:0/2295787006 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd0002c070 con 0x56011d8cb8c0 2024-03-18T16:17:22.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.240+0000 7fdcf2000640 1 -- 172.21.15.39:0/2295787006 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fdd00012430 con 0x56011d8cb8c0 2024-03-18T16:17:22.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.241+0000 7fdd11d85740 1 -- 172.21.15.39:0/2295787006 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdd00017070 con 0x56011d8cb8c0 2024-03-18T16:17:22.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.242+0000 7fdcf2000640 1 -- 172.21.15.39:0/2295787006 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fdd00012630 con 0x56011d8cb8c0 2024-03-18T16:17:22.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.242+0000 7fdd11d85740 10 client.27085 fetch_fsmap learned FSMap version 44 2024-03-18T16:17:22.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.242+0000 7fdd11d85740 1 -- 172.21.15.39:0/2295787006 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x56011d958e30 con 0x56011d8cb8c0 2024-03-18T16:17:22.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.242+0000 7fdcf2000640 1 -- 172.21.15.39:0/2295787006 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd00017070 con 0x56011d8cb8c0 2024-03-18T16:17:22.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.243+0000 7fdcf2000640 1 -- 172.21.15.39:0/2295787006 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7fdd00011070 con 0x56011d8cb8c0 2024-03-18T16:17:22.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.243+0000 7fdd11d85740 10 client.27085 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:17:22.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.243+0000 7fdd11d85740 10 client.27085 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-18T16:17:22.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.243+0000 7fdd11d85740 20 client.27085 populate_metadata read hostname 'smithi039' 2024-03-18T16:17:22.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.243+0000 7fdd11d85740 1 --2- 172.21.15.39:0/2295787006 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x56011d9594b0 0x56011d95b8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:22.245 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.243+0000 7fdd11d85740 4 client.27085 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:17:22.245 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.243+0000 7fdd11d85740 1 -- 172.21.15.39:0/2295787006 --> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56011d837ce0 con 0x56011d9594b0 2024-03-18T16:17:22.245 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.243+0000 7fdd07400640 1 --2- 172.21.15.39:0/2295787006 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x56011d9594b0 0x56011d95b8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:22.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.244+0000 7fdd07400640 1 --2- 172.21.15.39:0/2295787006 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x56011d9594b0 0x56011d95b8a0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=3fb353beadb1b6da server_cookie=4789a91e1f568ca6 in_seq=0 out_seq=0 2024-03-18T16:17:22.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.244+0000 7fdcf2000640 10 client.27085 ms_handle_connect on v2:172.21.15.194:6836/1301464246 2024-03-18T16:17:22.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.245+0000 7fdcf2000640 1 -- 172.21.15.39:0/2295787006 <== mds.3 v2:172.21.15.194:6836/1301464246 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56011d837ce0 con 0x56011d9594b0 2024-03-18T16:17:22.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.245+0000 7fdcf2000640 10 client.27085 handle_command_reply: tid=0 2024-03-18T16:17:22.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.245+0000 7fdd11d85740 1 client.27085 shutdown 2024-03-18T16:17:22.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.246+0000 7fdd11d85740 1 -- 172.21.15.39:0/2295787006 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56011d8cb8c0 msgr2=0x56011d8daaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:22.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.246+0000 7fdd11d85740 1 --2- 172.21.15.39:0/2295787006 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56011d8cb8c0 0x56011d8daaf0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fdd00009f50 tx=0x7fdd00009720 comp rx=0 tx=0).stop 2024-03-18T16:17:22.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.246+0000 7fdd11d85740 1 -- 172.21.15.39:0/2295787006 shutdown_connections 2024-03-18T16:17:22.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.246+0000 7fdd11d85740 1 --2- 172.21.15.39:0/2295787006 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x56011d9594b0 0x56011d95b8a0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:22.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.246+0000 7fdd11d85740 1 --2- 172.21.15.39:0/2295787006 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56011d8cc2b0 0x56011d8db030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:22.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.246+0000 7fdd11d85740 1 --2- 172.21.15.39:0/2295787006 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x56011d8d1420 0x56011d8dbff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:22.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.246+0000 7fdd11d85740 1 --2- 172.21.15.39:0/2295787006 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x56011d8cb8c0 0x56011d8daaf0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:22.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.246+0000 7fdcf2000640 0 client.27085 ms_handle_reset on v2:172.21.15.194:6836/1301464246 2024-03-18T16:17:22.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.246+0000 7fdd11d85740 1 -- 172.21.15.39:0/2295787006 >> 172.21.15.39:0/2295787006 conn(0x56011d8c6e30 msgr2=0x56011d8cd7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:17:22.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.246+0000 7fdd11d85740 1 -- 172.21.15.39:0/2295787006 shutdown_connections 2024-03-18T16:17:22.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.247+0000 7fdd11d85740 1 -- 172.21.15.39:0/2295787006 wait complete. 2024-03-18T16:17:22.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.247+0000 7fdd11d85740 20 client.27085 trim_cache size 0 max 16384 2024-03-18T16:17:22.250 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:17:22.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.248+0000 7fdd0f000640 1 -- 172.21.15.39:0/4066800230 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fdcec0a2900 msgr2=0x7fdcec0a4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:22.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.248+0000 7fdd0f000640 1 --2- 172.21.15.39:0/4066800230 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fdcec0a2900 0x7fdcec0a4dc0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fdcf800eff0 tx=0x7fdcf80d2ac0 comp rx=0 tx=0).stop 2024-03-18T16:17:22.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.248+0000 7fdd0f000640 1 -- 172.21.15.39:0/4066800230 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdd08103a00 msgr2=0x7fdd0819df50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:22.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.248+0000 7fdd0f000640 1 --2- 172.21.15.39:0/4066800230 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdd08103a00 0x7fdd0819df50 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fdcf800e9c0 tx=0x7fdcf800ee80 comp rx=0 tx=0).stop 2024-03-18T16:17:22.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.248+0000 7fdd0f000640 1 -- 172.21.15.39:0/4066800230 shutdown_connections 2024-03-18T16:17:22.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.248+0000 7fdd0f000640 1 --2- 172.21.15.39:0/4066800230 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdd08104bf0 0x7fdd0819e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:22.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.248+0000 7fdd0f000640 1 --2- 172.21.15.39:0/4066800230 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fdcec0a2900 0x7fdcec0a4dc0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:22.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.249+0000 7fdd0f000640 1 --2- 172.21.15.39:0/4066800230 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fdd081055b0 0x7fdd081a5500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:22.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.249+0000 7fdd0f000640 1 --2- 172.21.15.39:0/4066800230 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdd08103a00 0x7fdd0819df50 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:22.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.249+0000 7fdd0f000640 1 -- 172.21.15.39:0/4066800230 >> 172.21.15.39:0/4066800230 conn(0x7fdd080fef90 msgr2=0x7fdd08079050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:17:22.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.249+0000 7fdd0f000640 1 -- 172.21.15.39:0/4066800230 shutdown_connections 2024-03-18T16:17:22.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.249+0000 7fdd0f000640 1 -- 172.21.15.39:0/4066800230 wait complete. 2024-03-18T16:17:22.281 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-18T16:17:22.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.631+0000 7fb21a000640 1 -- 172.21.15.39:0/3648418733 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb214110060 msgr2=0x7fb214112450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:22.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.631+0000 7fb21a000640 1 --2- 172.21.15.39:0/3648418733 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb214110060 0x7fb214112450 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fb200009900 tx=0x7fb20002bc30 comp rx=0 tx=0).stop 2024-03-18T16:17:22.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.632+0000 7fb21a000640 1 -- 172.21.15.39:0/3648418733 shutdown_connections 2024-03-18T16:17:22.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.632+0000 7fb21a000640 1 --2- 172.21.15.39:0/3648418733 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb214110060 0x7fb214112450 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:22.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.632+0000 7fb21a000640 1 --2- 172.21.15.39:0/3648418733 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb214109140 0x7fb21410fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:22.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.632+0000 7fb21a000640 1 --2- 172.21.15.39:0/3648418733 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb2141087e0 0x7fb214108c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:22.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.632+0000 7fb21a000640 1 -- 172.21.15.39:0/3648418733 >> 172.21.15.39:0/3648418733 conn(0x7fb21406e860 msgr2=0x7fb214070ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:17:22.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.632+0000 7fb21a000640 1 -- 172.21.15.39:0/3648418733 shutdown_connections 2024-03-18T16:17:22.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.633+0000 7fb21a000640 1 -- 172.21.15.39:0/3648418733 wait complete. 2024-03-18T16:17:22.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.636+0000 7fb21a000640 1 Processor -- start 2024-03-18T16:17:22.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.637+0000 7fb21a000640 1 -- start start 2024-03-18T16:17:22.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.637+0000 7fb21a000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb2141087e0 0x7fb2141a24c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:22.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.637+0000 7fb21a000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb214109140 0x7fb2141a2a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:22.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.637+0000 7fb21a000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb214110060 0x7fb2141a9a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:22.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.637+0000 7fb21a000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fb214115cf0 con 0x7fb214109140 2024-03-18T16:17:22.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.637+0000 7fb21a000640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fb214115b70 con 0x7fb214110060 2024-03-18T16:17:22.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.637+0000 7fb21a000640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fb214115e70 con 0x7fb2141087e0 2024-03-18T16:17:22.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.637+0000 7fb213e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb214110060 0x7fb2141a9a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:22.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.638+0000 7fb213e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb214110060 0x7fb2141a9a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:41844/0 (socket says 172.21.15.39:41844) 2024-03-18T16:17:22.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.638+0000 7fb213e00640 1 -- 172.21.15.39:0/1968770229 learned_addr learned my addr 172.21.15.39:0/1968770229 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:17:22.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.638+0000 7fb212a00640 1 --2- 172.21.15.39:0/1968770229 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb214109140 0x7fb2141a2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:22.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.638+0000 7fb213400640 1 --2- 172.21.15.39:0/1968770229 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb2141087e0 0x7fb2141a24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:22.641 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.639+0000 7fb213400640 1 -- 172.21.15.39:0/1968770229 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb214110060 msgr2=0x7fb2141a9a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:22.641 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.639+0000 7fb213400640 1 --2- 172.21.15.39:0/1968770229 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb214110060 0x7fb2141a9a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:22.641 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.639+0000 7fb213400640 1 -- 172.21.15.39:0/1968770229 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb214109140 msgr2=0x7fb2141a2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:22.641 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.639+0000 7fb213400640 1 --2- 172.21.15.39:0/1968770229 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb214109140 0x7fb2141a2a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:22.641 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.639+0000 7fb213400640 1 -- 172.21.15.39:0/1968770229 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2141aa0e0 con 0x7fb2141087e0 2024-03-18T16:17:22.641 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.639+0000 7fb213e00640 1 --2- 172.21.15.39:0/1968770229 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb214110060 0x7fb2141a9a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:17:22.641 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.639+0000 7fb213400640 1 --2- 172.21.15.39:0/1968770229 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb2141087e0 0x7fb2141a24c0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fb20c00e9f0 tx=0x7fb20c00edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:17:22.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.640+0000 7fb20be00640 1 -- 172.21.15.39:0/1968770229 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb20c00cd20 con 0x7fb2141087e0 2024-03-18T16:17:22.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.640+0000 7fb20be00640 1 -- 172.21.15.39:0/1968770229 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb20c011950 con 0x7fb2141087e0 2024-03-18T16:17:22.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.640+0000 7fb20be00640 1 -- 172.21.15.39:0/1968770229 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb20c01aa30 con 0x7fb2141087e0 2024-03-18T16:17:22.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.640+0000 7fb21a000640 1 -- 172.21.15.39:0/1968770229 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb21410bb60 con 0x7fb2141087e0 2024-03-18T16:17:22.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.640+0000 7fb21a000640 1 -- 172.21.15.39:0/1968770229 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb21410c060 con 0x7fb2141087e0 2024-03-18T16:17:22.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.645+0000 7fb20be00640 1 -- 172.21.15.39:0/1968770229 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fb20c01abd0 con 0x7fb2141087e0 2024-03-18T16:17:22.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.645+0000 7fb21cd8c740 1 Processor -- start 2024-03-18T16:17:22.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.646+0000 7fb21cd8c740 1 -- start start 2024-03-18T16:17:22.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.647+0000 7fb21cd8c740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ee38da18c0 0x55ee38da1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:22.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.647+0000 7fb21cd8c740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ee38da22b0 0x55ee38da6db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:22.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.647+0000 7fb21cd8c740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ee38da7420 0x55ee38da98b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:22.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.647+0000 7fb21cd8c740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55ee38d693b0 con 0x55ee38da22b0 2024-03-18T16:17:22.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.647+0000 7fb21cd8c740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55ee38d06450 con 0x55ee38da18c0 2024-03-18T16:17:22.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.647+0000 7fb21cd8c740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55ee38caecb0 con 0x55ee38da7420 2024-03-18T16:17:22.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.648+0000 7fb213e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ee38da18c0 0x55ee38da1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:22.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.648+0000 7fb213e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ee38da18c0 0x55ee38da1ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:41860/0 (socket says 172.21.15.39:41860) 2024-03-18T16:17:22.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.648+0000 7fb213e00640 1 -- 172.21.15.39:0/3906733545 learned_addr learned my addr 172.21.15.39:0/3906733545 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:17:22.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.648+0000 7fb213e00640 1 -- 172.21.15.39:0/3906733545 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ee38da7420 msgr2=0x55ee38da98b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:17:22.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.648+0000 7fb213e00640 1 --2- 172.21.15.39:0/3906733545 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ee38da7420 0x55ee38da98b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:22.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.648+0000 7fb213e00640 1 -- 172.21.15.39:0/3906733545 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ee38da22b0 msgr2=0x55ee38da6db0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:17:22.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.648+0000 7fb213e00640 1 --2- 172.21.15.39:0/3906733545 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ee38da22b0 0x55ee38da6db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:22.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.648+0000 7fb213e00640 1 -- 172.21.15.39:0/3906733545 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee38cd01f0 con 0x55ee38da18c0 2024-03-18T16:17:22.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.649+0000 7fb20be00640 1 --2- 172.21.15.39:0/1968770229 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb1f80a2950 0x7fb1f80a4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:22.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.649+0000 7fb20be00640 1 -- 172.21.15.39:0/1968770229 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fb20c0cf1b0 con 0x7fb2141087e0 2024-03-18T16:17:22.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.650+0000 7fb213400640 1 --2- 172.21.15.39:0/1968770229 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb1f80a2950 0x7fb1f80a4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:22.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.651+0000 7fb213400640 1 --2- 172.21.15.39:0/1968770229 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb1f80a2950 0x7fb1f80a4e10 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fb20c00c640 tx=0x7fb20c01a490 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:17:22.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.652+0000 7fb213e00640 1 --2- 172.21.15.39:0/3906733545 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ee38da18c0 0x55ee38da1ce0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fb20002c7b0 tx=0x7fb20002c6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df76436c9cb3ba37 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:17:22.656 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.652+0000 7fb1ffe00640 1 -- 172.21.15.39:0/3906733545 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb200045070 con 0x55ee38da18c0 2024-03-18T16:17:22.656 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.652+0000 7fb1ffe00640 1 -- 172.21.15.39:0/3906733545 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2000046c0 con 0x55ee38da18c0 2024-03-18T16:17:22.656 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.652+0000 7fb1ffe00640 1 -- 172.21.15.39:0/3906733545 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb200005380 con 0x55ee38da18c0 2024-03-18T16:17:22.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.653+0000 7fb21cd8c740 1 -- 172.21.15.39:0/3906733545 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ee38da18c0 msgr2=0x55ee38da1ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:22.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.653+0000 7fb21cd8c740 1 --2- 172.21.15.39:0/3906733545 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ee38da18c0 0x55ee38da1ce0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fb20002c7b0 tx=0x7fb20002c6c0 comp rx=0 tx=0).stop 2024-03-18T16:17:22.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.653+0000 7fb21cd8c740 1 -- 172.21.15.39:0/3906733545 shutdown_connections 2024-03-18T16:17:22.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.653+0000 7fb21cd8c740 1 --2- 172.21.15.39:0/3906733545 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ee38da7420 0x55ee38da98b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:22.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.653+0000 7fb21cd8c740 1 --2- 172.21.15.39:0/3906733545 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ee38da22b0 0x55ee38da6db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:22.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.653+0000 7fb21cd8c740 1 --2- 172.21.15.39:0/3906733545 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ee38da18c0 0x55ee38da1ce0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:22.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.653+0000 7fb21cd8c740 1 -- 172.21.15.39:0/3906733545 >> 172.21.15.39:0/3906733545 conn(0x55ee38d9ce30 msgr2=0x55ee38d9d230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:17:22.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.653+0000 7fb21cd8c740 1 -- 172.21.15.39:0/3906733545 shutdown_connections 2024-03-18T16:17:22.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.654+0000 7fb21cd8c740 1 -- 172.21.15.39:0/3906733545 wait complete. 2024-03-18T16:17:22.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.654+0000 7fb21cd8c740 1 -- start start 2024-03-18T16:17:22.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.655+0000 7fb21cd8c740 1 Processor -- start 2024-03-18T16:17:22.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.656+0000 7fb21cd8c740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ee38da18c0 0x55ee38dabbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:22.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.656+0000 7fb21cd8c740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ee38da22b0 0x55ee38dac0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:22.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.656+0000 7fb21cd8c740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ee38da7420 0x55ee38db3130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:22.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.656+0000 7fb213e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ee38da18c0 0x55ee38dabbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:22.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.656+0000 7fb213e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ee38da18c0 0x55ee38dabbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:53818/0 (socket says 172.21.15.39:53818) 2024-03-18T16:17:22.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.656+0000 7fb213e00640 1 -- 172.21.15.39:0/1477921438 learned_addr learned my addr 172.21.15.39:0/1477921438 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:17:22.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.656+0000 7fb21cd8c740 1 -- 172.21.15.39:0/1477921438 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55ee38cc2110 con 0x55ee38da22b0 2024-03-18T16:17:22.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.656+0000 7fb21cd8c740 1 -- 172.21.15.39:0/1477921438 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55ee38d024f0 con 0x55ee38da7420 2024-03-18T16:17:22.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.656+0000 7fb21cd8c740 1 -- 172.21.15.39:0/1477921438 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55ee38daa370 con 0x55ee38da18c0 2024-03-18T16:17:22.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.656+0000 7fb213e00640 1 --2- 172.21.15.39:0/1477921438 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ee38da7420 0x55ee38db3130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:22.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.657+0000 7fb21cd8c740 10 client.0 _ll_register_callbacks cb 0x55ee38cafb70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:17:22.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.657+0000 7fb213e00640 1 -- 172.21.15.39:0/1477921438 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ee38da7420 msgr2=0x55ee38db3130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:17:22.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.657+0000 7fb213e00640 1 --2- 172.21.15.39:0/1477921438 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ee38da7420 0x55ee38db3130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:22.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.657+0000 7fb213e00640 1 -- 172.21.15.39:0/1477921438 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ee38da22b0 msgr2=0x55ee38dac0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:17:22.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.657+0000 7fb213e00640 1 --2- 172.21.15.39:0/1477921438 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ee38da22b0 0x55ee38dac0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:22.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.657+0000 7fb213e00640 1 -- 172.21.15.39:0/1477921438 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee38a1a3f0 con 0x55ee38da18c0 2024-03-18T16:17:22.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.657+0000 7fb213e00640 1 --2- 172.21.15.39:0/1477921438 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ee38da18c0 0x55ee38dabbb0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fb20002c170 tx=0x7fb200009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c76d9b48bc285982 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:17:22.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.657+0000 7fb1fcc00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:17:22.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.658+0000 7fb1fcc00640 1 -- 172.21.15.39:0/1477921438 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb200055060 con 0x55ee38da18c0 2024-03-18T16:17:22.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.658+0000 7fb1fcc00640 1 -- 172.21.15.39:0/1477921438 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fb20004aa90 con 0x55ee38da18c0 2024-03-18T16:17:22.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.658+0000 7fb21cd8c740 1 -- 172.21.15.39:0/1477921438 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb2000046c0 con 0x55ee38da18c0 2024-03-18T16:17:22.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.658+0000 7fb1fcc00640 1 -- 172.21.15.39:0/1477921438 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fb200045040 con 0x55ee38da18c0 2024-03-18T16:17:22.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.658+0000 7fb21cd8c740 10 client.27094 fetch_fsmap learned FSMap version 44 2024-03-18T16:17:22.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.658+0000 7fb21cd8c740 1 -- 172.21.15.39:0/1477921438 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x55ee38ae47d0 con 0x55ee38da18c0 2024-03-18T16:17:22.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.659+0000 7fb1fcc00640 1 -- 172.21.15.39:0/1477921438 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2000046c0 con 0x55ee38da18c0 2024-03-18T16:17:22.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.660+0000 7fb1fcc00640 1 -- 172.21.15.39:0/1477921438 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7fb20004ac90 con 0x55ee38da18c0 2024-03-18T16:17:22.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.660+0000 7fb21cd8c740 10 client.27094 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:17:22.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.660+0000 7fb21cd8c740 10 client.27094 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-18T16:17:22.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.660+0000 7fb21cd8c740 20 client.27094 populate_metadata read hostname 'smithi039' 2024-03-18T16:17:22.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.660+0000 7fb21cd8c740 1 --2- 172.21.15.39:0/1477921438 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x55ee38db7600 0x55ee38db99f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:22.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.660+0000 7fb21cd8c740 4 client.27094 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:17:22.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.660+0000 7fb21cd8c740 1 -- 172.21.15.39:0/1477921438 --> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ee38d13990 con 0x55ee38db7600 2024-03-18T16:17:22.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.660+0000 7fb212a00640 1 --2- 172.21.15.39:0/1477921438 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x55ee38db7600 0x55ee38db99f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:22.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.661+0000 7fb212a00640 1 --2- 172.21.15.39:0/1477921438 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x55ee38db7600 0x55ee38db99f0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=fab2b2c0e9244214 server_cookie=5357cf0b0655d644 in_seq=0 out_seq=0 2024-03-18T16:17:22.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.661+0000 7fb1fcc00640 10 client.27094 ms_handle_connect on v2:172.21.15.196:6836/895247251 2024-03-18T16:17:22.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.662+0000 7fb1fcc00640 1 -- 172.21.15.39:0/1477921438 <== mds.2 v2:172.21.15.196:6836/895247251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55ee38d13990 con 0x55ee38db7600 2024-03-18T16:17:22.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.662+0000 7fb1fcc00640 10 client.27094 handle_command_reply: tid=0 2024-03-18T16:17:22.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.662+0000 7fb21cd8c740 1 client.27094 shutdown 2024-03-18T16:17:22.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.663+0000 7fb21cd8c740 1 -- 172.21.15.39:0/1477921438 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ee38da18c0 msgr2=0x55ee38dabbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:22.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.663+0000 7fb21cd8c740 1 --2- 172.21.15.39:0/1477921438 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ee38da18c0 0x55ee38dabbb0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fb20002c170 tx=0x7fb200009260 comp rx=0 tx=0).stop 2024-03-18T16:17:22.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.663+0000 7fb21cd8c740 1 -- 172.21.15.39:0/1477921438 shutdown_connections 2024-03-18T16:17:22.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.663+0000 7fb21cd8c740 1 --2- 172.21.15.39:0/1477921438 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ee38da7420 0x55ee38db3130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:22.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.663+0000 7fb21cd8c740 1 --2- 172.21.15.39:0/1477921438 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x55ee38db7600 0x55ee38db99f0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:22.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.663+0000 7fb21cd8c740 1 --2- 172.21.15.39:0/1477921438 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ee38da22b0 0x55ee38dac0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:22.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.663+0000 7fb21cd8c740 1 --2- 172.21.15.39:0/1477921438 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ee38da18c0 0x55ee38dabbb0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:22.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.663+0000 7fb1fcc00640 0 client.27094 ms_handle_reset on v2:172.21.15.196:6836/895247251 2024-03-18T16:17:22.666 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.663+0000 7fb21cd8c740 1 -- 172.21.15.39:0/1477921438 >> 172.21.15.39:0/1477921438 conn(0x55ee38d9ce30 msgr2=0x55ee38d9c1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:17:22.666 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.663+0000 7fb21cd8c740 1 -- 172.21.15.39:0/1477921438 shutdown_connections 2024-03-18T16:17:22.666 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.663+0000 7fb21cd8c740 1 -- 172.21.15.39:0/1477921438 wait complete. 2024-03-18T16:17:22.666 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.664+0000 7fb21cd8c740 20 client.27094 trim_cache size 0 max 16384 2024-03-18T16:17:22.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.702+0000 7fb21cd8c740 1 Processor -- start 2024-03-18T16:17:22.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.702+0000 7fb21cd8c740 1 -- start start 2024-03-18T16:17:22.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.702+0000 7fb21cd8c740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ee38da18c0 0x55ee38d9c070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:22.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.702+0000 7fb21cd8c740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ee38da22b0 0x55ee38db30f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:22.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.702+0000 7fb21cd8c740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ee38da7420 0x55ee38dbe1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:22.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.702+0000 7fb21cd8c740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fb2141b90c0 con 0x55ee38da7420 2024-03-18T16:17:22.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.702+0000 7fb21cd8c740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fb2141bac00 con 0x55ee38da22b0 2024-03-18T16:17:22.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.702+0000 7fb21cd8c740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fb2141b9790 con 0x55ee38da18c0 2024-03-18T16:17:22.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.703+0000 7fb213e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ee38da18c0 0x55ee38d9c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:22.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.703+0000 7fb213e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ee38da18c0 0x55ee38d9c070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:53834/0 (socket says 172.21.15.39:53834) 2024-03-18T16:17:22.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.703+0000 7fb213e00640 1 -- 172.21.15.39:0/3089965073 learned_addr learned my addr 172.21.15.39:0/3089965073 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:17:22.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.703+0000 7fb213e00640 1 -- 172.21.15.39:0/3089965073 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ee38da22b0 msgr2=0x55ee38db30f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T16:17:22.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.703+0000 7fb213e00640 1 --2- 172.21.15.39:0/3089965073 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ee38da22b0 0x55ee38db30f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:22.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.703+0000 7fb213e00640 1 -- 172.21.15.39:0/3089965073 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ee38da7420 msgr2=0x55ee38dbe1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:17:22.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.703+0000 7fb213e00640 1 --2- 172.21.15.39:0/3089965073 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ee38da7420 0x55ee38dbe1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:22.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.703+0000 7fb213e00640 1 -- 172.21.15.39:0/3089965073 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee38d98c10 con 0x55ee38da18c0 2024-03-18T16:17:22.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.704+0000 7fb213e00640 1 --2- 172.21.15.39:0/3089965073 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ee38da18c0 0x55ee38d9c070 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fb2000058c0 tx=0x7fb200000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ad5cfb81b196b6f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:17:22.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.704+0000 7fb1ff400640 1 -- 172.21.15.39:0/3089965073 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb200055070 con 0x55ee38da18c0 2024-03-18T16:17:22.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.704+0000 7fb1ff400640 1 -- 172.21.15.39:0/3089965073 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2000046c0 con 0x55ee38da18c0 2024-03-18T16:17:22.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.704+0000 7fb1ff400640 1 -- 172.21.15.39:0/3089965073 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb20003f7e0 con 0x55ee38da18c0 2024-03-18T16:17:22.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.706+0000 7fb21cd8c740 1 -- 172.21.15.39:0/3089965073 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ee38da18c0 msgr2=0x55ee38d9c070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:22.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.706+0000 7fb21cd8c740 1 --2- 172.21.15.39:0/3089965073 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ee38da18c0 0x55ee38d9c070 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7fb2000058c0 tx=0x7fb200000bc0 comp rx=0 tx=0).stop 2024-03-18T16:17:22.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.706+0000 7fb21cd8c740 1 -- 172.21.15.39:0/3089965073 shutdown_connections 2024-03-18T16:17:22.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.706+0000 7fb21cd8c740 1 --2- 172.21.15.39:0/3089965073 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ee38da22b0 0x55ee38db30f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:22.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.706+0000 7fb21cd8c740 1 --2- 172.21.15.39:0/3089965073 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ee38da7420 0x55ee38dbe1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:22.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.706+0000 7fb21cd8c740 1 --2- 172.21.15.39:0/3089965073 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ee38da18c0 0x55ee38d9c070 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:22.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.706+0000 7fb21cd8c740 1 -- 172.21.15.39:0/3089965073 >> 172.21.15.39:0/3089965073 conn(0x55ee38d9ce30 msgr2=0x55ee38d9b960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:17:22.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.707+0000 7fb21cd8c740 1 -- 172.21.15.39:0/3089965073 shutdown_connections 2024-03-18T16:17:22.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.707+0000 7fb21cd8c740 1 -- 172.21.15.39:0/3089965073 wait complete. 2024-03-18T16:17:22.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.709+0000 7fb21cd8c740 1 -- start start 2024-03-18T16:17:22.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.709+0000 7fb21cd8c740 1 Processor -- start 2024-03-18T16:17:22.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.710+0000 7fb21cd8c740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ee38da18c0 0x55ee38db0a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:22.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.710+0000 7fb21cd8c740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ee38da22b0 0x55ee38db0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:22.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.710+0000 7fb21cd8c740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ee38da7420 0x55ee38db1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:22.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.710+0000 7fb21cd8c740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fb2141ba530 con 0x55ee38da7420 2024-03-18T16:17:22.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.711+0000 7fb21cd8c740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fb2141b9e60 con 0x55ee38da18c0 2024-03-18T16:17:22.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.711+0000 7fb21cd8c740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fb2141b70a0 con 0x55ee38da22b0 2024-03-18T16:17:22.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.711+0000 7fb21cd8c740 10 client.0 _ll_register_callbacks cb 0x7fb2000095f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:17:22.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.711+0000 7fb213e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ee38da18c0 0x55ee38db0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:22.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.711+0000 7fb212a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ee38da22b0 0x55ee38db0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:22.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.711+0000 7fb213e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ee38da18c0 0x55ee38db0a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:41884/0 (socket says 172.21.15.39:41884) 2024-03-18T16:17:22.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.711+0000 7fb213e00640 1 -- 172.21.15.39:0/1048617343 learned_addr learned my addr 172.21.15.39:0/1048617343 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:17:22.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.711+0000 7fb213e00640 1 --2- 172.21.15.39:0/1048617343 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ee38da7420 0x55ee38db1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:22.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.712+0000 7fb213e00640 1 -- 172.21.15.39:0/1048617343 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ee38da22b0 msgr2=0x55ee38db0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:17:22.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.712+0000 7fb213e00640 1 --2- 172.21.15.39:0/1048617343 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ee38da22b0 0x55ee38db0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:22.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.712+0000 7fb213e00640 1 -- 172.21.15.39:0/1048617343 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ee38da7420 msgr2=0x55ee38db1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:17:22.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.713+0000 7fb213e00640 1 --2- 172.21.15.39:0/1048617343 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ee38da7420 0x55ee38db1fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:22.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.713+0000 7fb213e00640 1 -- 172.21.15.39:0/1048617343 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee38da71e0 con 0x55ee38da18c0 2024-03-18T16:17:22.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.713+0000 7fb212a00640 1 --2- 172.21.15.39:0/1048617343 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ee38da22b0 0x55ee38db0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:17:22.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.714+0000 7fb213e00640 1 --2- 172.21.15.39:0/1048617343 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ee38da18c0 0x55ee38db0a50 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fb200004c00 tx=0x7fb200005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d201ce2ea10accd8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:17:22.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.714+0000 7fb1fcc00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:17:22.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.715+0000 7fb1fcc00640 1 -- 172.21.15.39:0/1048617343 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb200055070 con 0x55ee38da18c0 2024-03-18T16:17:22.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.715+0000 7fb1fcc00640 1 -- 172.21.15.39:0/1048617343 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fb20004aaa0 con 0x55ee38da18c0 2024-03-18T16:17:22.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.715+0000 7fb21cd8c740 1 -- 172.21.15.39:0/1048617343 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb2000046c0 con 0x55ee38da18c0 2024-03-18T16:17:22.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.716+0000 7fb1fcc00640 1 -- 172.21.15.39:0/1048617343 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fb20004aca0 con 0x55ee38da18c0 2024-03-18T16:17:22.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.716+0000 7fb21cd8c740 10 client.27125 fetch_fsmap learned FSMap version 44 2024-03-18T16:17:22.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.716+0000 7fb21cd8c740 1 -- 172.21.15.39:0/1048617343 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x55ee38e1d840 con 0x55ee38da18c0 2024-03-18T16:17:22.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.718+0000 7fb1fcc00640 1 -- 172.21.15.39:0/1048617343 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2000046c0 con 0x55ee38da18c0 2024-03-18T16:17:22.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.718+0000 7fb1fcc00640 1 -- 172.21.15.39:0/1048617343 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7fb20006d080 con 0x55ee38da18c0 2024-03-18T16:17:22.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.718+0000 7fb21cd8c740 10 client.27125 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:17:22.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.718+0000 7fb21cd8c740 10 client.27125 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-18T16:17:22.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.718+0000 7fb21cd8c740 20 client.27125 populate_metadata read hostname 'smithi039' 2024-03-18T16:17:22.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.718+0000 7fb21cd8c740 1 --2- 172.21.15.39:0/1048617343 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x55ee38e1e150 0x55ee38e20570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:22.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.718+0000 7fb21cd8c740 4 client.27125 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:17:22.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.718+0000 7fb21cd8c740 1 -- 172.21.15.39:0/1048617343 --> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ee38d0dce0 con 0x55ee38e1e150 2024-03-18T16:17:22.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.719+0000 7fb212a00640 1 --2- 172.21.15.39:0/1048617343 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x55ee38e1e150 0x55ee38e20570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:22.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.722+0000 7fb212a00640 1 --2- 172.21.15.39:0/1048617343 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x55ee38e1e150 0x55ee38e20570 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=44197d17ba627bbe server_cookie=5cfb163f5cfba85 in_seq=0 out_seq=0 2024-03-18T16:17:22.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.722+0000 7fb1fcc00640 10 client.27125 ms_handle_connect on v2:172.21.15.196:6836/895247251 2024-03-18T16:17:22.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.723+0000 7fb1fcc00640 1 -- 172.21.15.39:0/1048617343 <== mds.2 v2:172.21.15.196:6836/895247251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ee38d0dce0 con 0x55ee38e1e150 2024-03-18T16:17:22.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.723+0000 7fb1fcc00640 10 client.27125 handle_command_reply: tid=0 2024-03-18T16:17:22.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.723+0000 7fb21cd8c740 1 client.27125 shutdown 2024-03-18T16:17:22.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.724+0000 7fb21cd8c740 1 -- 172.21.15.39:0/1048617343 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ee38da18c0 msgr2=0x55ee38db0a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:22.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.724+0000 7fb21cd8c740 1 --2- 172.21.15.39:0/1048617343 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ee38da18c0 0x55ee38db0a50 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fb200004c00 tx=0x7fb200005fd0 comp rx=0 tx=0).stop 2024-03-18T16:17:22.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.724+0000 7fb21cd8c740 1 -- 172.21.15.39:0/1048617343 shutdown_connections 2024-03-18T16:17:22.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.724+0000 7fb21cd8c740 1 --2- 172.21.15.39:0/1048617343 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x55ee38e1e150 0x55ee38e20570 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:22.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.724+0000 7fb21cd8c740 1 --2- 172.21.15.39:0/1048617343 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55ee38da7420 0x55ee38db1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:22.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.724+0000 7fb21cd8c740 1 --2- 172.21.15.39:0/1048617343 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ee38da22b0 0x55ee38db0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:22.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.724+0000 7fb1fcc00640 0 client.27125 ms_handle_reset on v2:172.21.15.196:6836/895247251 2024-03-18T16:17:22.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.724+0000 7fb21cd8c740 1 --2- 172.21.15.39:0/1048617343 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ee38da18c0 0x55ee38db0a50 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:22.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.724+0000 7fb21cd8c740 1 -- 172.21.15.39:0/1048617343 >> 172.21.15.39:0/1048617343 conn(0x55ee38d9ce30 msgr2=0x55ee38d9c1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:17:22.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.725+0000 7fb21cd8c740 1 -- 172.21.15.39:0/1048617343 shutdown_connections 2024-03-18T16:17:22.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.725+0000 7fb21cd8c740 1 -- 172.21.15.39:0/1048617343 wait complete. 2024-03-18T16:17:22.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.725+0000 7fb21cd8c740 20 client.27125 trim_cache size 0 max 16384 2024-03-18T16:17:22.729 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:17:22.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.728+0000 7fb21a000640 1 -- 172.21.15.39:0/1968770229 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb1f80a2950 msgr2=0x7fb1f80a4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:22.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.728+0000 7fb21a000640 1 --2- 172.21.15.39:0/1968770229 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb1f80a2950 0x7fb1f80a4e10 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fb20c00c640 tx=0x7fb20c01a490 comp rx=0 tx=0).stop 2024-03-18T16:17:22.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.728+0000 7fb21a000640 1 -- 172.21.15.39:0/1968770229 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb2141087e0 msgr2=0x7fb2141a24c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:22.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.728+0000 7fb21a000640 1 --2- 172.21.15.39:0/1968770229 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb2141087e0 0x7fb2141a24c0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fb20c00e9f0 tx=0x7fb20c00edf0 comp rx=0 tx=0).stop 2024-03-18T16:17:22.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.729+0000 7fb21a000640 1 -- 172.21.15.39:0/1968770229 shutdown_connections 2024-03-18T16:17:22.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.729+0000 7fb21a000640 1 --2- 172.21.15.39:0/1968770229 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb214110060 0x7fb2141a9a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:22.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.729+0000 7fb21a000640 1 --2- 172.21.15.39:0/1968770229 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb1f80a2950 0x7fb1f80a4e10 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:22.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.729+0000 7fb21a000640 1 --2- 172.21.15.39:0/1968770229 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb214109140 0x7fb2141a2a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:22.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.729+0000 7fb21a000640 1 --2- 172.21.15.39:0/1968770229 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb2141087e0 0x7fb2141a24c0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:22.731 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.729+0000 7fb21a000640 1 -- 172.21.15.39:0/1968770229 >> 172.21.15.39:0/1968770229 conn(0x7fb21406e860 msgr2=0x7fb214070420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:17:22.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.730+0000 7fb21a000640 1 -- 172.21.15.39:0/1968770229 shutdown_connections 2024-03-18T16:17:22.732 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:22.730+0000 7fb21a000640 1 -- 172.21.15.39:0/1968770229 wait complete. 2024-03-18T16:17:22.761 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-18T16:17:23.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:17:22 smithi039 ceph-mon[31721]: pgmap v641: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 28 MiB/s wr, 5.95k op/s 2024-03-18T16:17:23.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.092+0000 7efebac00640 1 -- 172.21.15.39:0/4192736419 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efeb41087e0 msgr2=0x7efeb4108c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:23.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.092+0000 7efebac00640 1 --2- 172.21.15.39:0/4192736419 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efeb41087e0 0x7efeb4108c00 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7efea0009900 tx=0x7efea002bc40 comp rx=0 tx=0).stop 2024-03-18T16:17:23.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.095+0000 7efebac00640 1 -- 172.21.15.39:0/4192736419 shutdown_connections 2024-03-18T16:17:23.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.095+0000 7efebac00640 1 --2- 172.21.15.39:0/4192736419 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efeb4110060 0x7efeb4112450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:23.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.095+0000 7efebac00640 1 --2- 172.21.15.39:0/4192736419 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efeb4109140 0x7efeb410fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:23.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.095+0000 7efebac00640 1 --2- 172.21.15.39:0/4192736419 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efeb41087e0 0x7efeb4108c00 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:23.097 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.095+0000 7efebac00640 1 -- 172.21.15.39:0/4192736419 >> 172.21.15.39:0/4192736419 conn(0x7efeb406e860 msgr2=0x7efeb4070ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:17:23.098 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.097+0000 7efebac00640 1 -- 172.21.15.39:0/4192736419 shutdown_connections 2024-03-18T16:17:23.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.097+0000 7efebac00640 1 -- 172.21.15.39:0/4192736419 wait complete. 2024-03-18T16:17:23.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.099+0000 7efebac00640 1 Processor -- start 2024-03-18T16:17:23.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.099+0000 7efebac00640 1 -- start start 2024-03-18T16:17:23.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.100+0000 7efebac00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efeb41087e0 0x7efeb419e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:23.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.100+0000 7efebac00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efeb4109140 0x7efeb419e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:23.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.101+0000 7efeb3e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efeb41087e0 0x7efeb419e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:23.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.101+0000 7efeb3e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efeb41087e0 0x7efeb419e020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:41914/0 (socket says 172.21.15.39:41914) 2024-03-18T16:17:23.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.103+0000 7efeb3e00640 1 -- 172.21.15.39:0/3104587669 learned_addr learned my addr 172.21.15.39:0/3104587669 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:17:23.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.103+0000 7efebac00640 1 --2- 172.21.15.39:0/3104587669 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efeb4110060 0x7efeb41b6980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:23.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.103+0000 7efebac00640 1 -- 172.21.15.39:0/3104587669 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7efeb4113ac0 con 0x7efeb4109140 2024-03-18T16:17:23.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.103+0000 7efebac00640 1 -- 172.21.15.39:0/3104587669 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7efeb4113940 con 0x7efeb41087e0 2024-03-18T16:17:23.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.103+0000 7efebac00640 1 -- 172.21.15.39:0/3104587669 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7efeb4113c40 con 0x7efeb4110060 2024-03-18T16:17:23.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.104+0000 7efeb3e00640 1 -- 172.21.15.39:0/3104587669 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efeb4110060 msgr2=0x7efeb41b6980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:23.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.104+0000 7efeb3e00640 1 --2- 172.21.15.39:0/3104587669 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efeb4110060 0x7efeb41b6980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:23.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.104+0000 7efeb3e00640 1 -- 172.21.15.39:0/3104587669 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efeb4109140 msgr2=0x7efeb419e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:23.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.104+0000 7efeb3e00640 1 --2- 172.21.15.39:0/3104587669 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efeb4109140 0x7efeb419e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:23.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.104+0000 7efeb3e00640 1 -- 172.21.15.39:0/3104587669 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efeb41a16e0 con 0x7efeb41087e0 2024-03-18T16:17:23.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.104+0000 7efeb3e00640 1 --2- 172.21.15.39:0/3104587669 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efeb41087e0 0x7efeb419e020 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7efea002c8f0 tx=0x7efea0005d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:17:23.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.105+0000 7efeb0c00640 1 -- 172.21.15.39:0/3104587669 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efea0041070 con 0x7efeb41087e0 2024-03-18T16:17:23.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.105+0000 7efeb0c00640 1 -- 172.21.15.39:0/3104587669 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efea00044d0 con 0x7efeb41087e0 2024-03-18T16:17:23.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.105+0000 7efeb0c00640 1 -- 172.21.15.39:0/3104587669 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efea0034440 con 0x7efeb41087e0 2024-03-18T16:17:23.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.105+0000 7efebac00640 1 -- 172.21.15.39:0/3104587669 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efeb41b6f20 con 0x7efeb41087e0 2024-03-18T16:17:23.109 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.105+0000 7efebac00640 1 -- 172.21.15.39:0/3104587669 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efeb41b7330 con 0x7efeb41087e0 2024-03-18T16:17:23.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.115+0000 7efeb0c00640 1 -- 172.21.15.39:0/3104587669 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7efea0044070 con 0x7efeb41087e0 2024-03-18T16:17:23.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.117+0000 7efebd86f740 1 Processor -- start 2024-03-18T16:17:23.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.118+0000 7efebd86f740 1 -- start start 2024-03-18T16:17:23.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.118+0000 7efebd86f740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563a6bfae8c0 0x563a6bfaece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:23.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.118+0000 7efebd86f740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563a6bfaf2b0 0x563a6bfb3db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:23.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.118+0000 7efebd86f740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563a6bfb4420 0x563a6bfb68b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:23.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.118+0000 7efebd86f740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x563a6bf763b0 con 0x563a6bfb4420 2024-03-18T16:17:23.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.118+0000 7efebd86f740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x563a6bf13450 con 0x563a6bfae8c0 2024-03-18T16:17:23.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.118+0000 7efebd86f740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x563a6bebbcb0 con 0x563a6bfaf2b0 2024-03-18T16:17:23.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.119+0000 7efeb0c00640 1 --2- 172.21.15.39:0/3104587669 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7efe980a2930 0x7efe980a4df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:23.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.119+0000 7efeb0c00640 1 -- 172.21.15.39:0/3104587669 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7efea00f04d0 con 0x7efeb41087e0 2024-03-18T16:17:23.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.119+0000 7efeb3400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563a6bfb4420 0x563a6bfb68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:23.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.119+0000 7efeb3400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563a6bfb4420 0x563a6bfb68b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:37442/0 (socket says 172.21.15.39:37442) 2024-03-18T16:17:23.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.119+0000 7efeb3400640 1 -- 172.21.15.39:0/397454679 learned_addr learned my addr 172.21.15.39:0/397454679 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:17:23.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.120+0000 7efeb3e00640 1 --2- 172.21.15.39:0/397454679 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563a6bfaf2b0 0x563a6bfb3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:23.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.120+0000 7efeb3400640 1 -- 172.21.15.39:0/397454679 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563a6bfaf2b0 msgr2=0x563a6bfb3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:17:23.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.120+0000 7efeb3400640 1 --2- 172.21.15.39:0/397454679 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563a6bfaf2b0 0x563a6bfb3db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:23.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.120+0000 7efeb3400640 1 -- 172.21.15.39:0/397454679 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563a6bfae8c0 msgr2=0x563a6bfaece0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:17:23.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.120+0000 7efeb3400640 1 --2- 172.21.15.39:0/397454679 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563a6bfae8c0 0x563a6bfaece0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:23.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.120+0000 7efeb3400640 1 -- 172.21.15.39:0/397454679 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563a6bedd1f0 con 0x563a6bfb4420 2024-03-18T16:17:23.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.120+0000 7efeb3e00640 1 --2- 172.21.15.39:0/397454679 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563a6bfaf2b0 0x563a6bfb3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:17:23.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.121+0000 7efeb8c00640 1 --2- 172.21.15.39:0/3104587669 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7efe980a2930 0x7efe980a4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:23.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.121+0000 7efeb3400640 1 --2- 172.21.15.39:0/397454679 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563a6bfb4420 0x563a6bfb68b0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7efea400aca0 tx=0x7efea400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9ca9b81c227f9e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:17:23.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.122+0000 7efea9200640 1 -- 172.21.15.39:0/397454679 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efea4004020 con 0x563a6bfb4420 2024-03-18T16:17:23.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.122+0000 7efea9200640 1 -- 172.21.15.39:0/397454679 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efea40041c0 con 0x563a6bfb4420 2024-03-18T16:17:23.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.122+0000 7efea9200640 1 -- 172.21.15.39:0/397454679 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efea400f690 con 0x563a6bfb4420 2024-03-18T16:17:23.124 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.123+0000 7efebd86f740 1 -- 172.21.15.39:0/397454679 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563a6bfb4420 msgr2=0x563a6bfb68b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:23.124 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.123+0000 7efebd86f740 1 --2- 172.21.15.39:0/397454679 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563a6bfb4420 0x563a6bfb68b0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7efea400aca0 tx=0x7efea400c6a0 comp rx=0 tx=0).stop 2024-03-18T16:17:23.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.123+0000 7efebd86f740 1 -- 172.21.15.39:0/397454679 shutdown_connections 2024-03-18T16:17:23.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.123+0000 7efebd86f740 1 --2- 172.21.15.39:0/397454679 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563a6bfb4420 0x563a6bfb68b0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:23.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.123+0000 7efebd86f740 1 --2- 172.21.15.39:0/397454679 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563a6bfaf2b0 0x563a6bfb3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:23.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.123+0000 7efebd86f740 1 --2- 172.21.15.39:0/397454679 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563a6bfae8c0 0x563a6bfaece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:23.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.123+0000 7efebd86f740 1 -- 172.21.15.39:0/397454679 >> 172.21.15.39:0/397454679 conn(0x563a6bfa9e30 msgr2=0x563a6bfaa230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:17:23.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.124+0000 7efeb8c00640 1 --2- 172.21.15.39:0/3104587669 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7efe980a2930 0x7efe980a4df0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7efeac005c40 tx=0x7efeac005bd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:17:23.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.124+0000 7efebd86f740 1 -- 172.21.15.39:0/397454679 shutdown_connections 2024-03-18T16:17:23.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.124+0000 7efebd86f740 1 -- 172.21.15.39:0/397454679 wait complete. 2024-03-18T16:17:23.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.125+0000 7efebd86f740 1 -- start start 2024-03-18T16:17:23.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.125+0000 7efebd86f740 1 Processor -- start 2024-03-18T16:17:23.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.126+0000 7efebd86f740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563a6bfae8c0 0x563a6bfb8e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:23.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.126+0000 7efebd86f740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563a6bfaf2b0 0x563a6bfb9340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:23.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.126+0000 7efebd86f740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563a6bfbe370 0x563a6bfc0760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:23.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.126+0000 7efebd86f740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x563a6becf110 con 0x563a6bfbe370 2024-03-18T16:17:23.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.126+0000 7efebd86f740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x563a6bf0f4f0 con 0x563a6bfae8c0 2024-03-18T16:17:23.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.126+0000 7efebd86f740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x563a6bfb73d0 con 0x563a6bfaf2b0 2024-03-18T16:17:23.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.126+0000 7efebd86f740 10 client.0 _ll_register_callbacks cb 0x563a6bebcb70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:17:23.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.127+0000 7efeb3400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563a6bfaf2b0 0x563a6bfb9340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:23.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.127+0000 7efeb3400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563a6bfaf2b0 0x563a6bfb9340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:53878/0 (socket says 172.21.15.39:53878) 2024-03-18T16:17:23.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.127+0000 7efeb3400640 1 -- 172.21.15.39:0/2146835410 learned_addr learned my addr 172.21.15.39:0/2146835410 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:17:23.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.127+0000 7efeb3e00640 1 --2- 172.21.15.39:0/2146835410 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563a6bfae8c0 0x563a6bfb8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:23.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.127+0000 7efeb3400640 1 -- 172.21.15.39:0/2146835410 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563a6bfae8c0 msgr2=0x563a6bfb8e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:17:23.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.127+0000 7efeb3400640 1 --2- 172.21.15.39:0/2146835410 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563a6bfae8c0 0x563a6bfb8e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:23.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.127+0000 7efeb3400640 1 -- 172.21.15.39:0/2146835410 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563a6bfbe370 msgr2=0x563a6bfc0760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:17:23.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.127+0000 7efeb3400640 1 --2- 172.21.15.39:0/2146835410 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563a6bfbe370 0x563a6bfc0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:23.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.128+0000 7efeb3400640 1 -- 172.21.15.39:0/2146835410 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563a6bc273f0 con 0x563a6bfaf2b0 2024-03-18T16:17:23.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.128+0000 7efeb3400640 1 --2- 172.21.15.39:0/2146835410 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563a6bfaf2b0 0x563a6bfb9340 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7efea4000c00 tx=0x7efea4003c90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cb154e911b6049b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:17:23.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.130+0000 7efe9d600640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:17:23.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.130+0000 7efe9d600640 1 -- 172.21.15.39:0/2146835410 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efea4026060 con 0x563a6bfaf2b0 2024-03-18T16:17:23.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.130+0000 7efe9d600640 1 -- 172.21.15.39:0/2146835410 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7efea4004370 con 0x563a6bfaf2b0 2024-03-18T16:17:23.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.130+0000 7efebd86f740 1 -- 172.21.15.39:0/2146835410 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efea40041c0 con 0x563a6bfaf2b0 2024-03-18T16:17:23.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.130+0000 7efe9d600640 1 -- 172.21.15.39:0/2146835410 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7efea4004020 con 0x563a6bfaf2b0 2024-03-18T16:17:23.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.130+0000 7efebd86f740 10 client.27106 fetch_fsmap learned FSMap version 44 2024-03-18T16:17:23.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.130+0000 7efebd86f740 1 -- 172.21.15.39:0/2146835410 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x563a6bcf17d0 con 0x563a6bfaf2b0 2024-03-18T16:17:23.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.131+0000 7efe9d600640 1 -- 172.21.15.39:0/2146835410 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efea40041c0 con 0x563a6bfaf2b0 2024-03-18T16:17:23.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.131+0000 7efe9d600640 1 -- 172.21.15.39:0/2146835410 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7efea4021410 con 0x563a6bfaf2b0 2024-03-18T16:17:23.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.131+0000 7efebd86f740 10 client.27106 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:17:23.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.131+0000 7efebd86f740 10 client.27106 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T16:17:23.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.131+0000 7efebd86f740 20 client.27106 populate_metadata read hostname 'smithi039' 2024-03-18T16:17:23.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.131+0000 7efebd86f740 1 --2- 172.21.15.39:0/2146835410 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x563a6bfc4df0 0x563a6bfc71e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:23.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.132+0000 7efebd86f740 4 client.27106 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:17:23.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.132+0000 7efebd86f740 1 -- 172.21.15.39:0/2146835410 --> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563a6bf20990 con 0x563a6bfc4df0 2024-03-18T16:17:23.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.132+0000 7efeb3e00640 1 --2- 172.21.15.39:0/2146835410 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x563a6bfc4df0 0x563a6bfc71e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:23.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.133+0000 7efeb3e00640 1 --2- 172.21.15.39:0/2146835410 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x563a6bfc4df0 0x563a6bfc71e0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=94cf86b435edb1dd server_cookie=c708702d59dad0a7 in_seq=0 out_seq=0 2024-03-18T16:17:23.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.133+0000 7efe9d600640 10 client.27106 ms_handle_connect on v2:172.21.15.196:6838/2001427268 2024-03-18T16:17:23.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.133+0000 7efe9d600640 1 -- 172.21.15.39:0/2146835410 <== mds.1 v2:172.21.15.196:6838/2001427268 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563a6bf20990 con 0x563a6bfc4df0 2024-03-18T16:17:23.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.133+0000 7efe9d600640 10 client.27106 handle_command_reply: tid=0 2024-03-18T16:17:23.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.134+0000 7efebd86f740 1 client.27106 shutdown 2024-03-18T16:17:23.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.134+0000 7efebd86f740 1 -- 172.21.15.39:0/2146835410 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563a6bfaf2b0 msgr2=0x563a6bfb9340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:23.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.134+0000 7efebd86f740 1 --2- 172.21.15.39:0/2146835410 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563a6bfaf2b0 0x563a6bfb9340 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7efea4000c00 tx=0x7efea4003c90 comp rx=0 tx=0).stop 2024-03-18T16:17:23.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.135+0000 7efebd86f740 1 -- 172.21.15.39:0/2146835410 shutdown_connections 2024-03-18T16:17:23.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.135+0000 7efebd86f740 1 --2- 172.21.15.39:0/2146835410 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x563a6bfc4df0 0x563a6bfc71e0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:23.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.135+0000 7efebd86f740 1 --2- 172.21.15.39:0/2146835410 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563a6bfbe370 0x563a6bfc0760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:23.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.135+0000 7efebd86f740 1 --2- 172.21.15.39:0/2146835410 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563a6bfaf2b0 0x563a6bfb9340 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:23.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.135+0000 7efebd86f740 1 --2- 172.21.15.39:0/2146835410 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563a6bfae8c0 0x563a6bfb8e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:23.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.135+0000 7efe9d600640 0 client.27106 ms_handle_reset on v2:172.21.15.196:6838/2001427268 2024-03-18T16:17:23.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.135+0000 7efebd86f740 1 -- 172.21.15.39:0/2146835410 >> 172.21.15.39:0/2146835410 conn(0x563a6bfa9e30 msgr2=0x563a6bfa91f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:17:23.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.135+0000 7efebd86f740 1 -- 172.21.15.39:0/2146835410 shutdown_connections 2024-03-18T16:17:23.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.135+0000 7efebd86f740 1 -- 172.21.15.39:0/2146835410 wait complete. 2024-03-18T16:17:23.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.135+0000 7efebd86f740 20 client.27106 trim_cache size 0 max 16384 2024-03-18T16:17:23.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.165+0000 7efebd86f740 1 Processor -- start 2024-03-18T16:17:23.166 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.165+0000 7efebd86f740 1 -- start start 2024-03-18T16:17:23.167 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.165+0000 7efebd86f740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563a6bfae8c0 0x563a6bfa9070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:23.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.165+0000 7efebd86f740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563a6bfaf2b0 0x563a6bfcbaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:23.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.165+0000 7efebd86f740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563a6bfbe370 0x563a6bfcbfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:23.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.166+0000 7efeb3400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563a6bfaf2b0 0x563a6bfcbaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:23.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.166+0000 7efeb3400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563a6bfaf2b0 0x563a6bfcbaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:41932/0 (socket says 172.21.15.39:41932) 2024-03-18T16:17:23.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.166+0000 7efeb3e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563a6bfae8c0 0x563a6bfa9070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:23.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.166+0000 7efebd86f740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7efeb41c19c0 con 0x563a6bfae8c0 2024-03-18T16:17:23.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.166+0000 7efeb3e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563a6bfae8c0 0x563a6bfa9070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:37460/0 (socket says 172.21.15.39:37460) 2024-03-18T16:17:23.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.166+0000 7efeb3e00640 1 -- 172.21.15.39:0/727302100 learned_addr learned my addr 172.21.15.39:0/727302100 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:17:23.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.166+0000 7efebd86f740 1 -- 172.21.15.39:0/727302100 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7efeb41c3500 con 0x563a6bfaf2b0 2024-03-18T16:17:23.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.166+0000 7efeb8c00640 1 --2- 172.21.15.39:0/727302100 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563a6bfbe370 0x563a6bfcbfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:23.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.166+0000 7efebd86f740 1 -- 172.21.15.39:0/727302100 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7efeb41c2090 con 0x563a6bfbe370 2024-03-18T16:17:23.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.167+0000 7efeb8c00640 1 -- 172.21.15.39:0/727302100 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563a6bfaf2b0 msgr2=0x563a6bfcbaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:17:23.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.167+0000 7efeb8c00640 1 --2- 172.21.15.39:0/727302100 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563a6bfaf2b0 0x563a6bfcbaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:23.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.167+0000 7efeb8c00640 1 -- 172.21.15.39:0/727302100 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563a6bfae8c0 msgr2=0x563a6bfa9070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:17:23.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.167+0000 7efeb8c00640 1 --2- 172.21.15.39:0/727302100 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563a6bfae8c0 0x563a6bfa9070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:23.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.167+0000 7efeb8c00640 1 -- 172.21.15.39:0/727302100 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563a6bfb8bf0 con 0x563a6bfbe370 2024-03-18T16:17:23.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.167+0000 7efeb8c00640 1 --2- 172.21.15.39:0/727302100 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563a6bfbe370 0x563a6bfcbfe0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7efeac005fd0 tx=0x7efeac009500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fe6cc8a23620fd96 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:17:23.169 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.168+0000 7efeb3400640 1 --2- 172.21.15.39:0/727302100 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563a6bfaf2b0 0x563a6bfcbaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello state changed while learned_addr, mark_down or replacing must be happened just now 2024-03-18T16:17:23.172 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.170+0000 7efe9fe00640 1 -- 172.21.15.39:0/727302100 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeac021070 con 0x563a6bfbe370 2024-03-18T16:17:23.172 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.170+0000 7efe9fe00640 1 -- 172.21.15.39:0/727302100 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efeac01d940 con 0x563a6bfbe370 2024-03-18T16:17:23.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.170+0000 7efe9fe00640 1 -- 172.21.15.39:0/727302100 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeac01ca20 con 0x563a6bfbe370 2024-03-18T16:17:23.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.171+0000 7efebd86f740 1 -- 172.21.15.39:0/727302100 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563a6bfbe370 msgr2=0x563a6bfcbfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:23.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.171+0000 7efebd86f740 1 --2- 172.21.15.39:0/727302100 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563a6bfbe370 0x563a6bfcbfe0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7efeac005fd0 tx=0x7efeac009500 comp rx=0 tx=0).stop 2024-03-18T16:17:23.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.171+0000 7efebd86f740 1 -- 172.21.15.39:0/727302100 shutdown_connections 2024-03-18T16:17:23.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.171+0000 7efebd86f740 1 --2- 172.21.15.39:0/727302100 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563a6bfaf2b0 0x563a6bfcbaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:23.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.171+0000 7efebd86f740 1 --2- 172.21.15.39:0/727302100 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563a6bfbe370 0x563a6bfcbfe0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:23.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.171+0000 7efebd86f740 1 --2- 172.21.15.39:0/727302100 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563a6bfae8c0 0x563a6bfa9070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:23.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.171+0000 7efebd86f740 1 -- 172.21.15.39:0/727302100 >> 172.21.15.39:0/727302100 conn(0x563a6bfa9e30 msgr2=0x563a6bfa8960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:17:23.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.171+0000 7efebd86f740 1 -- 172.21.15.39:0/727302100 shutdown_connections 2024-03-18T16:17:23.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.171+0000 7efebd86f740 1 -- 172.21.15.39:0/727302100 wait complete. 2024-03-18T16:17:23.173 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.172+0000 7efebd86f740 1 -- start start 2024-03-18T16:17:23.174 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.172+0000 7efebd86f740 1 Processor -- start 2024-03-18T16:17:23.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.174+0000 7efebd86f740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563a6bfae8c0 0x563a6bfd8160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:23.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.175+0000 7efebd86f740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563a6bfaf2b0 0x563a6bfd6770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:23.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.175+0000 7efeb3e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563a6bfae8c0 0x563a6bfd8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:23.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.175+0000 7efebd86f740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563a6bfbe370 0x563a6bfd6cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:23.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.175+0000 7efebd86f740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7efeb41c2e30 con 0x563a6bfae8c0 2024-03-18T16:17:23.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.175+0000 7efebd86f740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7efeb41c2760 con 0x563a6bfaf2b0 2024-03-18T16:17:23.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.175+0000 7efebd86f740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7efeb41bf9a0 con 0x563a6bfbe370 2024-03-18T16:17:23.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.176+0000 7efebd86f740 10 client.0 _ll_register_callbacks cb 0x7efea400aad0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:17:23.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.176+0000 7efeb8c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563a6bfbe370 0x563a6bfd6cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:23.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.176+0000 7efeb8c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563a6bfbe370 0x563a6bfd6cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:53904/0 (socket says 172.21.15.39:53904) 2024-03-18T16:17:23.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.176+0000 7efeb8c00640 1 -- 172.21.15.39:0/2726325941 learned_addr learned my addr 172.21.15.39:0/2726325941 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:17:23.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.176+0000 7efeb8c00640 1 -- 172.21.15.39:0/2726325941 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563a6bfaf2b0 msgr2=0x563a6bfd6770 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T16:17:23.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.176+0000 7efeb8c00640 1 --2- 172.21.15.39:0/2726325941 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563a6bfaf2b0 0x563a6bfd6770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:23.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.176+0000 7efeb8c00640 1 -- 172.21.15.39:0/2726325941 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563a6bfae8c0 msgr2=0x563a6bfd8160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:17:23.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.176+0000 7efeb8c00640 1 --2- 172.21.15.39:0/2726325941 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563a6bfae8c0 0x563a6bfd8160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:23.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.176+0000 7efeb8c00640 1 -- 172.21.15.39:0/2726325941 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563a6bfd74e0 con 0x563a6bfbe370 2024-03-18T16:17:23.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.177+0000 7efe9d600640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:17:23.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.177+0000 7efeb8c00640 1 --2- 172.21.15.39:0/2726325941 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563a6bfbe370 0x563a6bfd6cb0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7efeac01c5a0 tx=0x7efeac01d690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=33813f1c06a84a39 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:17:23.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.178+0000 7efe9d600640 1 -- 172.21.15.39:0/2726325941 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeac021070 con 0x563a6bfbe370 2024-03-18T16:17:23.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.178+0000 7efe9d600640 1 -- 172.21.15.39:0/2726325941 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7efeac009040 con 0x563a6bfbe370 2024-03-18T16:17:23.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.178+0000 7efebd86f740 1 -- 172.21.15.39:0/2726325941 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efeac01d940 con 0x563a6bfbe370 2024-03-18T16:17:23.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.179+0000 7efe9d600640 1 -- 172.21.15.39:0/2726325941 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7efeac01dd00 con 0x563a6bfbe370 2024-03-18T16:17:23.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.180+0000 7efebd86f740 10 client.27112 fetch_fsmap learned FSMap version 44 2024-03-18T16:17:23.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.180+0000 7efebd86f740 1 -- 172.21.15.39:0/2726325941 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x563a6c02b9f0 con 0x563a6bfbe370 2024-03-18T16:17:23.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.180+0000 7efe9d600640 1 -- 172.21.15.39:0/2726325941 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efeac01d940 con 0x563a6bfbe370 2024-03-18T16:17:23.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.181+0000 7efe9d600640 1 -- 172.21.15.39:0/2726325941 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7efeac03a610 con 0x563a6bfbe370 2024-03-18T16:17:23.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.181+0000 7efebd86f740 10 client.27112 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:17:23.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.181+0000 7efebd86f740 10 client.27112 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T16:17:23.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.181+0000 7efebd86f740 20 client.27112 populate_metadata read hostname 'smithi039' 2024-03-18T16:17:23.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.181+0000 7efebd86f740 1 --2- 172.21.15.39:0/2726325941 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x563a6c02c280 0x563a6c02e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:23.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.184+0000 7efebd86f740 4 client.27112 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:17:23.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.184+0000 7efebd86f740 1 -- 172.21.15.39:0/2726325941 --> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563a6bf1ace0 con 0x563a6c02c280 2024-03-18T16:17:23.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.190+0000 7efeb3e00640 1 --2- 172.21.15.39:0/2726325941 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x563a6c02c280 0x563a6c02e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:23.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.192+0000 7efeb3e00640 1 --2- 172.21.15.39:0/2726325941 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x563a6c02c280 0x563a6c02e6a0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a3bf4d47b9daa252 server_cookie=2baaa15c6fe5aa54 in_seq=0 out_seq=0 2024-03-18T16:17:23.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.192+0000 7efe9d600640 10 client.27112 ms_handle_connect on v2:172.21.15.196:6838/2001427268 2024-03-18T16:17:23.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.192+0000 7efe9d600640 1 -- 172.21.15.39:0/2726325941 <== mds.1 v2:172.21.15.196:6838/2001427268 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563a6bf1ace0 con 0x563a6c02c280 2024-03-18T16:17:23.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.192+0000 7efe9d600640 10 client.27112 handle_command_reply: tid=0 2024-03-18T16:17:23.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.193+0000 7efebd86f740 1 client.27112 shutdown 2024-03-18T16:17:23.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.193+0000 7efebd86f740 1 -- 172.21.15.39:0/2726325941 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563a6bfbe370 msgr2=0x563a6bfd6cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:23.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.193+0000 7efebd86f740 1 --2- 172.21.15.39:0/2726325941 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563a6bfbe370 0x563a6bfd6cb0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7efeac01c5a0 tx=0x7efeac01d690 comp rx=0 tx=0).stop 2024-03-18T16:17:23.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.193+0000 7efebd86f740 1 -- 172.21.15.39:0/2726325941 shutdown_connections 2024-03-18T16:17:23.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.193+0000 7efebd86f740 1 --2- 172.21.15.39:0/2726325941 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x563a6c02c280 0x563a6c02e6a0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:23.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.193+0000 7efebd86f740 1 --2- 172.21.15.39:0/2726325941 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563a6bfaf2b0 0x563a6bfd6770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:23.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.193+0000 7efebd86f740 1 --2- 172.21.15.39:0/2726325941 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563a6bfbe370 0x563a6bfd6cb0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:23.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.193+0000 7efebd86f740 1 --2- 172.21.15.39:0/2726325941 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563a6bfae8c0 0x563a6bfd8160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:23.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.193+0000 7efebd86f740 1 -- 172.21.15.39:0/2726325941 >> 172.21.15.39:0/2726325941 conn(0x563a6bfa9e30 msgr2=0x563a6bfa91f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:17:23.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.193+0000 7efe9d600640 0 client.27112 ms_handle_reset on v2:172.21.15.196:6838/2001427268 2024-03-18T16:17:23.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.194+0000 7efebd86f740 1 -- 172.21.15.39:0/2726325941 shutdown_connections 2024-03-18T16:17:23.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.195+0000 7efebd86f740 1 -- 172.21.15.39:0/2726325941 wait complete. 2024-03-18T16:17:23.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.195+0000 7efebd86f740 20 client.27112 trim_cache size 0 max 16384 2024-03-18T16:17:23.197 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:17:23.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.196+0000 7efebac00640 1 -- 172.21.15.39:0/3104587669 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7efe980a2930 msgr2=0x7efe980a4df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:23.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.196+0000 7efebac00640 1 --2- 172.21.15.39:0/3104587669 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7efe980a2930 0x7efe980a4df0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7efeac005c40 tx=0x7efeac005bd0 comp rx=0 tx=0).stop 2024-03-18T16:17:23.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.196+0000 7efebac00640 1 -- 172.21.15.39:0/3104587669 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efeb41087e0 msgr2=0x7efeb419e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:23.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.196+0000 7efebac00640 1 --2- 172.21.15.39:0/3104587669 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efeb41087e0 0x7efeb419e020 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7efea002c8f0 tx=0x7efea0005d50 comp rx=0 tx=0).stop 2024-03-18T16:17:23.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.196+0000 7efebac00640 1 -- 172.21.15.39:0/3104587669 shutdown_connections 2024-03-18T16:17:23.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.196+0000 7efebac00640 1 --2- 172.21.15.39:0/3104587669 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7efe980a2930 0x7efe980a4df0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:23.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.196+0000 7efebac00640 1 --2- 172.21.15.39:0/3104587669 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efeb4110060 0x7efeb41b6980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:23.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.196+0000 7efebac00640 1 --2- 172.21.15.39:0/3104587669 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efeb4109140 0x7efeb419e560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:23.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.196+0000 7efebac00640 1 --2- 172.21.15.39:0/3104587669 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efeb41087e0 0x7efeb419e020 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:23.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.196+0000 7efebac00640 1 -- 172.21.15.39:0/3104587669 >> 172.21.15.39:0/3104587669 conn(0x7efeb406e860 msgr2=0x7efeb40703e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:17:23.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.197+0000 7efebac00640 1 -- 172.21.15.39:0/3104587669 shutdown_connections 2024-03-18T16:17:23.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.197+0000 7efebac00640 1 -- 172.21.15.39:0/3104587669 wait complete. 2024-03-18T16:17:23.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:17:22 smithi196 ceph-mon[36307]: pgmap v641: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 28 MiB/s wr, 5.95k op/s 2024-03-18T16:17:23.234 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-18T16:17:23.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:17:22 smithi194 ceph-mon[36200]: pgmap v641: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 28 MiB/s wr, 5.95k op/s 2024-03-18T16:17:23.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.555+0000 7f7720200640 1 -- 172.21.15.39:0/3226605892 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f771810b410 msgr2=0x7f771810d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:23.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.555+0000 7f7720200640 1 --2- 172.21.15.39:0/3226605892 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f771810b410 0x7f771810d800 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f770c0074e0 tx=0x7f770c02c8d0 comp rx=0 tx=0).stop 2024-03-18T16:17:23.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.555+0000 7f7720200640 1 -- 172.21.15.39:0/3226605892 shutdown_connections 2024-03-18T16:17:23.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.555+0000 7f7720200640 1 --2- 172.21.15.39:0/3226605892 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f771810b410 0x7f771810d800 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:23.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.555+0000 7f7720200640 1 --2- 172.21.15.39:0/3226605892 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f771810de90 0x7f7718110350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:23.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.555+0000 7f7720200640 1 --2- 172.21.15.39:0/3226605892 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7718108a50 0x7f771810ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:23.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.555+0000 7f7720200640 1 -- 172.21.15.39:0/3226605892 >> 172.21.15.39:0/3226605892 conn(0x7f771806e860 msgr2=0x7f7718070ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:17:23.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.556+0000 7f7720200640 1 -- 172.21.15.39:0/3226605892 shutdown_connections 2024-03-18T16:17:23.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.556+0000 7f7720200640 1 -- 172.21.15.39:0/3226605892 wait complete. 2024-03-18T16:17:23.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.558+0000 7f7720200640 1 Processor -- start 2024-03-18T16:17:23.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.559+0000 7f7720200640 1 -- start start 2024-03-18T16:17:23.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.560+0000 7f7720200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7718108a50 0x7f771819e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:23.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.560+0000 7f7720200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f771810b410 0x7f771819e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:23.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.560+0000 7f7720200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f771810de90 0x7f77181a55e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:23.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.560+0000 7f7720200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f7718113a70 con 0x7f771810b410 2024-03-18T16:17:23.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.560+0000 7f7720200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f77181138f0 con 0x7f7718108a50 2024-03-18T16:17:23.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.560+0000 7f7720200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f7718113bf0 con 0x7f771810de90 2024-03-18T16:17:23.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.560+0000 7f771d800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7718108a50 0x7f771819e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:23.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.561+0000 7f771ce00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f771810b410 0x7f771819e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:23.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.561+0000 7f771d800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7718108a50 0x7f771819e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:41946/0 (socket says 172.21.15.39:41946) 2024-03-18T16:17:23.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.561+0000 7f771d800640 1 -- 172.21.15.39:0/3288586526 learned_addr learned my addr 172.21.15.39:0/3288586526 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:17:23.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.561+0000 7f771e200640 1 --2- 172.21.15.39:0/3288586526 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f771810de90 0x7f77181a55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:23.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.562+0000 7f771e200640 1 -- 172.21.15.39:0/3288586526 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7718108a50 msgr2=0x7f771819e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:23.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.562+0000 7f771e200640 1 --2- 172.21.15.39:0/3288586526 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7718108a50 0x7f771819e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:23.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.562+0000 7f771e200640 1 -- 172.21.15.39:0/3288586526 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f771810b410 msgr2=0x7f771819e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:23.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.562+0000 7f771e200640 1 --2- 172.21.15.39:0/3288586526 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f771810b410 0x7f771819e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:23.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.562+0000 7f771e200640 1 -- 172.21.15.39:0/3288586526 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77181a5ba0 con 0x7f771810de90 2024-03-18T16:17:23.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.563+0000 7f771ce00640 1 --2- 172.21.15.39:0/3288586526 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f771810b410 0x7f771819e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:17:23.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.563+0000 7f771e200640 1 --2- 172.21.15.39:0/3288586526 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f771810de90 0x7f77181a55e0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f771000cab0 tx=0x7f771000cf70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:17:23.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.563+0000 7f7716000640 1 -- 172.21.15.39:0/3288586526 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f771000ae20 con 0x7f771810de90 2024-03-18T16:17:23.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.563+0000 7f7716000640 1 -- 172.21.15.39:0/3288586526 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f771000f930 con 0x7f771810de90 2024-03-18T16:17:23.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.563+0000 7f7716000640 1 -- 172.21.15.39:0/3288586526 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7710018b50 con 0x7f771810de90 2024-03-18T16:17:23.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.563+0000 7f7720200640 1 -- 172.21.15.39:0/3288586526 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77181a5e90 con 0x7f771810de90 2024-03-18T16:17:23.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.564+0000 7f7720200640 1 -- 172.21.15.39:0/3288586526 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77181a6300 con 0x7f771810de90 2024-03-18T16:17:23.576 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.574+0000 7f7722f69740 1 Processor -- start 2024-03-18T16:17:23.577 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.576+0000 7f7716000640 1 -- 172.21.15.39:0/3288586526 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f771000fae0 con 0x7f771810de90 2024-03-18T16:17:23.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.578+0000 7f7722f69740 1 -- start start 2024-03-18T16:17:23.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.578+0000 7f7722f69740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562a6188e0d0 0x562a6188e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:23.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.578+0000 7f7722f69740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a6188eac0 0x562a618935c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:23.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.578+0000 7f7722f69740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562a61893c30 0x562a618960c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:23.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.578+0000 7f7722f69740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x562a6184d3b0 con 0x562a61893c30 2024-03-18T16:17:23.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.578+0000 7f7722f69740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x562a617ea450 con 0x562a6188eac0 2024-03-18T16:17:23.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.578+0000 7f7722f69740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x562a61792cb0 con 0x562a6188e0d0 2024-03-18T16:17:23.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.580+0000 7f7716000640 1 --2- 172.21.15.39:0/3288586526 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f77000a2880 0x7f77000a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:23.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.580+0000 7f771d800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a6188eac0 0x562a618935c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:23.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.580+0000 7f771d800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a6188eac0 0x562a618935c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:41960/0 (socket says 172.21.15.39:41960) 2024-03-18T16:17:23.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.580+0000 7f771d800640 1 -- 172.21.15.39:0/1580123922 learned_addr learned my addr 172.21.15.39:0/1580123922 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:17:23.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.581+0000 7f771ce00640 1 --2- 172.21.15.39:0/1580123922 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562a61893c30 0x562a618960c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:23.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.581+0000 7f771ce00640 1 --2- 172.21.15.39:0/1580123922 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562a6188e0d0 0x562a6188e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:23.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.581+0000 7f771e200640 1 --2- 172.21.15.39:0/3288586526 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f77000a2880 0x7f77000a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:23.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.581+0000 7f771d800640 1 -- 172.21.15.39:0/1580123922 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562a6188e0d0 msgr2=0x562a6188e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:17:23.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.581+0000 7f771d800640 1 --2- 172.21.15.39:0/1580123922 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562a6188e0d0 0x562a6188e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:23.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.581+0000 7f771d800640 1 -- 172.21.15.39:0/1580123922 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562a61893c30 msgr2=0x562a618960c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:17:23.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.581+0000 7f771d800640 1 --2- 172.21.15.39:0/1580123922 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562a61893c30 0x562a618960c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:23.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.581+0000 7f771d800640 1 -- 172.21.15.39:0/1580123922 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a617b41f0 con 0x562a6188eac0 2024-03-18T16:17:23.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.581+0000 7f771ce00640 1 --2- 172.21.15.39:0/1580123922 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562a6188e0d0 0x562a6188e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:17:23.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.582+0000 7f771d800640 1 --2- 172.21.15.39:0/1580123922 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a6188eac0 0x562a618935c0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f770400bdd0 tx=0x7f77040107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5db491b67cc3e8f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:17:23.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.583+0000 7f771e200640 1 --2- 172.21.15.39:0/3288586526 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f77000a2880 0x7f77000a4d40 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f77100ccc30 tx=0x7f77100d2c60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:17:23.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.583+0000 7f770aa00640 1 -- 172.21.15.39:0/1580123922 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7704005220 con 0x562a6188eac0 2024-03-18T16:17:23.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.583+0000 7f7716000640 1 -- 172.21.15.39:0/3288586526 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f77100cf470 con 0x7f771810de90 2024-03-18T16:17:23.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.584+0000 7f770aa00640 1 -- 172.21.15.39:0/1580123922 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7704004c80 con 0x562a6188eac0 2024-03-18T16:17:23.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.584+0000 7f770aa00640 1 -- 172.21.15.39:0/1580123922 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7704011980 con 0x562a6188eac0 2024-03-18T16:17:23.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.586+0000 7f7722f69740 1 -- 172.21.15.39:0/1580123922 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a6188eac0 msgr2=0x562a618935c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:23.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.586+0000 7f7722f69740 1 --2- 172.21.15.39:0/1580123922 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a6188eac0 0x562a618935c0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f770400bdd0 tx=0x7f77040107a0 comp rx=0 tx=0).stop 2024-03-18T16:17:23.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.587+0000 7f7722f69740 1 -- 172.21.15.39:0/1580123922 shutdown_connections 2024-03-18T16:17:23.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.587+0000 7f7722f69740 1 --2- 172.21.15.39:0/1580123922 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a6188eac0 0x562a618935c0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:23.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.587+0000 7f7722f69740 1 --2- 172.21.15.39:0/1580123922 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562a61893c30 0x562a618960c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:23.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.587+0000 7f7722f69740 1 --2- 172.21.15.39:0/1580123922 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562a6188e0d0 0x562a6188e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:23.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.587+0000 7f7722f69740 1 -- 172.21.15.39:0/1580123922 >> 172.21.15.39:0/1580123922 conn(0x562a61880e30 msgr2=0x562a61881230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:17:23.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.587+0000 7f7722f69740 1 -- 172.21.15.39:0/1580123922 shutdown_connections 2024-03-18T16:17:23.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.587+0000 7f7722f69740 1 -- 172.21.15.39:0/1580123922 wait complete. 2024-03-18T16:17:23.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.588+0000 7f7722f69740 1 -- start start 2024-03-18T16:17:23.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.588+0000 7f7722f69740 1 Processor -- start 2024-03-18T16:17:23.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.589+0000 7f7722f69740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562a6188e0d0 0x562a61898250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:23.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.589+0000 7f7722f69740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a6188eac0 0x562a61898790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:23.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.589+0000 7f7722f69740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562a61893c30 0x562a6189f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:23.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.589+0000 7f7722f69740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x562a617a6110 con 0x562a61893c30 2024-03-18T16:17:23.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.589+0000 7f7722f69740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x562a617e64f0 con 0x562a6188eac0 2024-03-18T16:17:23.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.590+0000 7f771d800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a6188eac0 0x562a61898790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:23.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.591+0000 7f7722f69740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x562a61896be0 con 0x562a6188e0d0 2024-03-18T16:17:23.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.591+0000 7f771ce00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562a6188e0d0 0x562a61898250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:23.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.591+0000 7f771ce00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562a6188e0d0 0x562a61898250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:53922/0 (socket says 172.21.15.39:53922) 2024-03-18T16:17:23.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.591+0000 7f771ce00640 1 -- 172.21.15.39:0/3558933642 learned_addr learned my addr 172.21.15.39:0/3558933642 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:17:23.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.591+0000 7f7722f69740 10 client.0 _ll_register_callbacks cb 0x562a61793b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:17:23.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.592+0000 7f771ce00640 1 -- 172.21.15.39:0/3558933642 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a6188eac0 msgr2=0x562a61898790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:17:23.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.592+0000 7f771ce00640 1 --2- 172.21.15.39:0/3558933642 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a6188eac0 0x562a61898790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:23.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.592+0000 7f771ce00640 1 -- 172.21.15.39:0/3558933642 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562a61893c30 msgr2=0x562a6189f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:17:23.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.592+0000 7f771ce00640 1 --2- 172.21.15.39:0/3558933642 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562a61893c30 0x562a6189f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:23.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.592+0000 7f771ce00640 1 -- 172.21.15.39:0/3558933642 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a614fe3f0 con 0x562a6188e0d0 2024-03-18T16:17:23.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.592+0000 7f771d800640 1 --2- 172.21.15.39:0/3558933642 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a6188eac0 0x562a61898790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:17:23.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.594+0000 7f771ce00640 1 --2- 172.21.15.39:0/3558933642 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562a6188e0d0 0x562a61898250 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f770c0074e0 tx=0x7f770c02cf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=93f60e62ae7ef349 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:17:23.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.594+0000 7f76ff400640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:17:23.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.598+0000 7f76ff400640 1 -- 172.21.15.39:0/3558933642 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f770c045070 con 0x562a6188e0d0 2024-03-18T16:17:23.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.598+0000 7f76ff400640 1 -- 172.21.15.39:0/3558933642 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f770c00a5d0 con 0x562a6188e0d0 2024-03-18T16:17:23.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.598+0000 7f7722f69740 1 -- 172.21.15.39:0/3558933642 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7704004c80 con 0x562a6188e0d0 2024-03-18T16:17:23.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.598+0000 7f76ff400640 1 -- 172.21.15.39:0/3558933642 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f770c00a7d0 con 0x562a6188e0d0 2024-03-18T16:17:23.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.598+0000 7f7722f69740 10 client.27124 fetch_fsmap learned FSMap version 44 2024-03-18T16:17:23.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.598+0000 7f7722f69740 1 -- 172.21.15.39:0/3558933642 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x562a615c87d0 con 0x562a6188e0d0 2024-03-18T16:17:23.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.599+0000 7f76ff400640 1 -- 172.21.15.39:0/3558933642 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7704004c80 con 0x562a6188e0d0 2024-03-18T16:17:23.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.599+0000 7f76ff400640 1 -- 172.21.15.39:0/3558933642 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7f770c0054d0 con 0x562a6188e0d0 2024-03-18T16:17:23.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.599+0000 7f7722f69740 10 client.27124 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:17:23.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.599+0000 7f7722f69740 10 client.27124 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:17:23.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.600+0000 7f7722f69740 20 client.27124 populate_metadata read hostname 'smithi039' 2024-03-18T16:17:23.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.600+0000 7f7722f69740 1 --2- 172.21.15.39:0/3558933642 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x562a61886450 0x562a618888d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:23.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.600+0000 7f7722f69740 4 client.27124 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:17:23.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.600+0000 7f7722f69740 1 -- 172.21.15.39:0/3558933642 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562a617f7990 con 0x562a61886450 2024-03-18T16:17:23.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.601+0000 7f771d800640 1 --2- 172.21.15.39:0/3558933642 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x562a61886450 0x562a618888d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:23.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.602+0000 7f771d800640 1 --2- 172.21.15.39:0/3558933642 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x562a61886450 0x562a618888d0 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=260079a43a62bfaf server_cookie=e79e309be69da0d0 in_seq=0 out_seq=0 2024-03-18T16:17:23.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.602+0000 7f76ff400640 10 client.27124 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:17:23.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.603+0000 7f76ff400640 1 -- 172.21.15.39:0/3558933642 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562a617f7990 con 0x562a61886450 2024-03-18T16:17:23.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.603+0000 7f76ff400640 10 client.27124 handle_command_reply: tid=0 2024-03-18T16:17:23.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.603+0000 7f7722f69740 1 client.27124 shutdown 2024-03-18T16:17:23.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.604+0000 7f7722f69740 1 -- 172.21.15.39:0/3558933642 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562a6188e0d0 msgr2=0x562a61898250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:23.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.604+0000 7f7722f69740 1 --2- 172.21.15.39:0/3558933642 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562a6188e0d0 0x562a61898250 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f770c0074e0 tx=0x7f770c02cf50 comp rx=0 tx=0).stop 2024-03-18T16:17:23.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.604+0000 7f7722f69740 1 -- 172.21.15.39:0/3558933642 shutdown_connections 2024-03-18T16:17:23.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.604+0000 7f7722f69740 1 --2- 172.21.15.39:0/3558933642 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x562a61886450 0x562a618888d0 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:23.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.604+0000 7f7722f69740 1 --2- 172.21.15.39:0/3558933642 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a6188eac0 0x562a61898790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:23.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.604+0000 7f7722f69740 1 --2- 172.21.15.39:0/3558933642 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562a61893c30 0x562a6189f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:23.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.604+0000 7f7722f69740 1 --2- 172.21.15.39:0/3558933642 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562a6188e0d0 0x562a61898250 secure :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f770c0074e0 tx=0x7f770c02cf50 comp rx=0 tx=0).stop 2024-03-18T16:17:23.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.604+0000 7f7722f69740 1 -- 172.21.15.39:0/3558933642 >> 172.21.15.39:0/3558933642 conn(0x562a61880e30 msgr2=0x562a618801f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:17:23.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.604+0000 7f76ff400640 0 client.27124 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:17:23.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.606+0000 7f7722f69740 1 -- 172.21.15.39:0/3558933642 shutdown_connections 2024-03-18T16:17:23.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.606+0000 7f7722f69740 1 -- 172.21.15.39:0/3558933642 wait complete. 2024-03-18T16:17:23.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.606+0000 7f7722f69740 20 client.27124 trim_cache size 0 max 16384 2024-03-18T16:17:23.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.642+0000 7f7722f69740 1 Processor -- start 2024-03-18T16:17:23.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.642+0000 7f7722f69740 1 -- start start 2024-03-18T16:17:23.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.643+0000 7f7722f69740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a6188eac0 0x562a618b5ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:23.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.643+0000 7f7722f69740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562a61893c30 0x562a6189f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:23.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.643+0000 7f7722f69740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562a618b6220 0x562a61897e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:23.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.643+0000 7f7722f69740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f77181b09a0 con 0x562a61893c30 2024-03-18T16:17:23.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.643+0000 7f7722f69740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f77181b24e0 con 0x562a6188eac0 2024-03-18T16:17:23.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.643+0000 7f7722f69740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f77181b1070 con 0x562a618b6220 2024-03-18T16:17:23.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.644+0000 7f771ce00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a6188eac0 0x562a618b5ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:23.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.644+0000 7f771d800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562a61893c30 0x562a6189f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:23.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.644+0000 7f771ce00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a6188eac0 0x562a618b5ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:41976/0 (socket says 172.21.15.39:41976) 2024-03-18T16:17:23.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.644+0000 7f771d800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562a61893c30 0x562a6189f720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:37512/0 (socket says 172.21.15.39:37512) 2024-03-18T16:17:23.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.645+0000 7f771d800640 1 -- 172.21.15.39:0/2029191196 learned_addr learned my addr 172.21.15.39:0/2029191196 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:17:23.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.645+0000 7f771ce00640 1 --2- 172.21.15.39:0/2029191196 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562a618b6220 0x562a61897e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:23.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.646+0000 7f771ce00640 1 -- 172.21.15.39:0/2029191196 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562a618b6220 msgr2=0x562a61897e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:17:23.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.646+0000 7f771ce00640 1 --2- 172.21.15.39:0/2029191196 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562a618b6220 0x562a61897e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:23.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.646+0000 7f771ce00640 1 -- 172.21.15.39:0/2029191196 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562a61893c30 msgr2=0x562a6189f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:17:23.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.646+0000 7f771ce00640 1 --2- 172.21.15.39:0/2029191196 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562a61893c30 0x562a6189f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:23.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.646+0000 7f771ce00640 1 -- 172.21.15.39:0/2029191196 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a6187cc10 con 0x562a6188eac0 2024-03-18T16:17:23.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.646+0000 7f771d800640 1 --2- 172.21.15.39:0/2029191196 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562a61893c30 0x562a6189f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:17:23.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.647+0000 7f771ce00640 1 --2- 172.21.15.39:0/2029191196 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a6188eac0 0x562a618b5ce0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f770c02cee0 tx=0x7f770c000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e276cc0840ca3c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:17:23.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.647+0000 7f770a000640 1 -- 172.21.15.39:0/2029191196 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f770c045070 con 0x562a6188eac0 2024-03-18T16:17:23.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.650+0000 7f770a000640 1 -- 172.21.15.39:0/2029191196 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f770c004ba0 con 0x562a6188eac0 2024-03-18T16:17:23.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.650+0000 7f770a000640 1 -- 172.21.15.39:0/2029191196 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f770c045070 con 0x562a6188eac0 2024-03-18T16:17:23.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.651+0000 7f7722f69740 1 -- 172.21.15.39:0/2029191196 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a6188eac0 msgr2=0x562a618b5ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:23.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.651+0000 7f7722f69740 1 --2- 172.21.15.39:0/2029191196 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a6188eac0 0x562a618b5ce0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f770c02cee0 tx=0x7f770c000bc0 comp rx=0 tx=0).stop 2024-03-18T16:17:23.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.651+0000 7f7722f69740 1 -- 172.21.15.39:0/2029191196 shutdown_connections 2024-03-18T16:17:23.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.651+0000 7f7722f69740 1 --2- 172.21.15.39:0/2029191196 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562a618b6220 0x562a61897e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:23.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.651+0000 7f7722f69740 1 --2- 172.21.15.39:0/2029191196 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562a61893c30 0x562a6189f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:23.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.651+0000 7f7722f69740 1 --2- 172.21.15.39:0/2029191196 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a6188eac0 0x562a618b5ce0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:23.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.651+0000 7f7722f69740 1 -- 172.21.15.39:0/2029191196 >> 172.21.15.39:0/2029191196 conn(0x562a61880e30 msgr2=0x562a61881210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:17:23.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.651+0000 7f7722f69740 1 -- 172.21.15.39:0/2029191196 shutdown_connections 2024-03-18T16:17:23.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.651+0000 7f7722f69740 1 -- 172.21.15.39:0/2029191196 wait complete. 2024-03-18T16:17:23.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.652+0000 7f7722f69740 1 -- start start 2024-03-18T16:17:23.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.652+0000 7f7722f69740 1 Processor -- start 2024-03-18T16:17:23.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.653+0000 7f7722f69740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562a6188eac0 0x562a6189f020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:23.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.653+0000 7f7722f69740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a61893c30 0x562a6189f560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:23.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.653+0000 7f7722f69740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562a618b6220 0x562a6189d680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:23.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.653+0000 7f7722f69740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f77181b1e10 con 0x562a618b6220 2024-03-18T16:17:23.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.653+0000 7f7722f69740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f77181b1740 con 0x562a61893c30 2024-03-18T16:17:23.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.653+0000 7f7722f69740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f77181ae980 con 0x562a6188eac0 2024-03-18T16:17:23.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.653+0000 7f7722f69740 10 client.0 _ll_register_callbacks cb 0x7f770c00a240 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:17:23.656 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.654+0000 7f771ce00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562a6188eac0 0x562a6189f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:23.656 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.654+0000 7f771d800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a61893c30 0x562a6189f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:23.656 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.654+0000 7f771ce00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562a6188eac0 0x562a6189f020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:53932/0 (socket says 172.21.15.39:53932) 2024-03-18T16:17:23.656 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.654+0000 7f771ce00640 1 -- 172.21.15.39:0/4113017210 learned_addr learned my addr 172.21.15.39:0/4113017210 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:17:23.656 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.654+0000 7f771d800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a61893c30 0x562a6189f560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:41978/0 (socket says 172.21.15.39:41978) 2024-03-18T16:17:23.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.654+0000 7f771ce00640 1 --2- 172.21.15.39:0/4113017210 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562a618b6220 0x562a6189d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:23.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.655+0000 7f771d800640 1 -- 172.21.15.39:0/4113017210 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562a6188eac0 msgr2=0x562a6189f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:17:23.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.655+0000 7f771d800640 1 --2- 172.21.15.39:0/4113017210 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562a6188eac0 0x562a6189f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:23.657 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.655+0000 7f771d800640 1 -- 172.21.15.39:0/4113017210 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562a618b6220 msgr2=0x562a6189d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:17:23.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.655+0000 7f771d800640 1 --2- 172.21.15.39:0/4113017210 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562a618b6220 0x562a6189d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:23.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.655+0000 7f771d800640 1 -- 172.21.15.39:0/4113017210 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a6189ddf0 con 0x562a61893c30 2024-03-18T16:17:23.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.655+0000 7f771ce00640 1 --2- 172.21.15.39:0/4113017210 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562a618b6220 0x562a6189d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:17:23.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.656+0000 7f771d800640 1 --2- 172.21.15.39:0/4113017210 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a61893c30 0x562a6189f560 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x562a61893640 tx=0x7f7704005040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=47e4a5d6c8f603d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:17:23.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.656+0000 7f76ff400640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:17:23.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.658+0000 7f76ff400640 1 -- 172.21.15.39:0/4113017210 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f770402b160 con 0x562a61893c30 2024-03-18T16:17:23.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.659+0000 7f76ff400640 1 -- 172.21.15.39:0/4113017210 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f770402b300 con 0x562a61893c30 2024-03-18T16:17:23.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.659+0000 7f7722f69740 1 -- 172.21.15.39:0/4113017210 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f770c004ba0 con 0x562a61893c30 2024-03-18T16:17:23.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.660+0000 7f76ff400640 1 -- 172.21.15.39:0/4113017210 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f770402b500 con 0x562a61893c30 2024-03-18T16:17:23.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.660+0000 7f7722f69740 10 client.27152 fetch_fsmap learned FSMap version 44 2024-03-18T16:17:23.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.660+0000 7f7722f69740 1 -- 172.21.15.39:0/4113017210 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x562a61912dd0 con 0x562a61893c30 2024-03-18T16:17:23.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.660+0000 7f76ff400640 1 -- 172.21.15.39:0/4113017210 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f770c004ba0 con 0x562a61893c30 2024-03-18T16:17:23.662 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.661+0000 7f76ff400640 1 -- 172.21.15.39:0/4113017210 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7f77040114b0 con 0x562a61893c30 2024-03-18T16:17:23.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.661+0000 7f7722f69740 10 client.27152 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:17:23.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.661+0000 7f7722f69740 10 client.27152 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:17:23.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.661+0000 7f7722f69740 20 client.27152 populate_metadata read hostname 'smithi039' 2024-03-18T16:17:23.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.661+0000 7f7722f69740 1 --2- 172.21.15.39:0/4113017210 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x562a61913540 0x562a61915960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:23.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.661+0000 7f7722f69740 4 client.27152 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:17:23.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.661+0000 7f7722f69740 1 -- 172.21.15.39:0/4113017210 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562a617f1ce0 con 0x562a61913540 2024-03-18T16:17:23.663 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.662+0000 7f771ce00640 1 --2- 172.21.15.39:0/4113017210 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x562a61913540 0x562a61915960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:23.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.665+0000 7f771ce00640 1 --2- 172.21.15.39:0/4113017210 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x562a61913540 0x562a61915960 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cdaf7e4142592542 server_cookie=20393c4d9a67bb41 in_seq=0 out_seq=0 2024-03-18T16:17:23.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.665+0000 7f76ff400640 10 client.27152 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:17:23.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.665+0000 7f76ff400640 1 -- 172.21.15.39:0/4113017210 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562a617f1ce0 con 0x562a61913540 2024-03-18T16:17:23.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.665+0000 7f76ff400640 10 client.27152 handle_command_reply: tid=0 2024-03-18T16:17:23.667 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.666+0000 7f7722f69740 1 client.27152 shutdown 2024-03-18T16:17:23.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.668+0000 7f7722f69740 1 -- 172.21.15.39:0/4113017210 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a61893c30 msgr2=0x562a6189f560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:23.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.668+0000 7f7722f69740 1 --2- 172.21.15.39:0/4113017210 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a61893c30 0x562a6189f560 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x562a61893640 tx=0x7f7704005040 comp rx=0 tx=0).stop 2024-03-18T16:17:23.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.669+0000 7f7722f69740 1 -- 172.21.15.39:0/4113017210 shutdown_connections 2024-03-18T16:17:23.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.669+0000 7f7722f69740 1 --2- 172.21.15.39:0/4113017210 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x562a61913540 0x562a61915960 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:23.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.669+0000 7f7722f69740 1 --2- 172.21.15.39:0/4113017210 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a61893c30 0x562a6189f560 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:23.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.669+0000 7f7722f69740 1 --2- 172.21.15.39:0/4113017210 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562a618b6220 0x562a6189d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:23.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.669+0000 7f7722f69740 1 --2- 172.21.15.39:0/4113017210 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562a6188eac0 0x562a6189f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:23.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.669+0000 7f7722f69740 1 -- 172.21.15.39:0/4113017210 >> 172.21.15.39:0/4113017210 conn(0x562a61880e30 msgr2=0x562a61897e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:17:23.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.669+0000 7f76ff400640 0 client.27152 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:17:23.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.670+0000 7f7722f69740 1 -- 172.21.15.39:0/4113017210 shutdown_connections 2024-03-18T16:17:23.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.670+0000 7f7722f69740 1 -- 172.21.15.39:0/4113017210 wait complete. 2024-03-18T16:17:23.672 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.670+0000 7f7722f69740 20 client.27152 trim_cache size 0 max 16384 2024-03-18T16:17:23.673 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:17:23.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.672+0000 7f7720200640 1 -- 172.21.15.39:0/3288586526 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f77000a2880 msgr2=0x7f77000a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:23.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.672+0000 7f7720200640 1 --2- 172.21.15.39:0/3288586526 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f77000a2880 0x7f77000a4d40 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f77100ccc30 tx=0x7f77100d2c60 comp rx=0 tx=0).stop 2024-03-18T16:17:23.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.672+0000 7f7720200640 1 -- 172.21.15.39:0/3288586526 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f771810de90 msgr2=0x7f77181a55e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:23.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.672+0000 7f7720200640 1 --2- 172.21.15.39:0/3288586526 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f771810de90 0x7f77181a55e0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f771000cab0 tx=0x7f771000cf70 comp rx=0 tx=0).stop 2024-03-18T16:17:23.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.672+0000 7f7720200640 1 -- 172.21.15.39:0/3288586526 shutdown_connections 2024-03-18T16:17:23.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.672+0000 7f7720200640 1 --2- 172.21.15.39:0/3288586526 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f77000a2880 0x7f77000a4d40 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:23.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.672+0000 7f7720200640 1 --2- 172.21.15.39:0/3288586526 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f771810de90 0x7f77181a55e0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:23.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.672+0000 7f7720200640 1 --2- 172.21.15.39:0/3288586526 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f771810b410 0x7f771819e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:23.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.672+0000 7f7720200640 1 --2- 172.21.15.39:0/3288586526 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7718108a50 0x7f771819e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:23.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.672+0000 7f7720200640 1 -- 172.21.15.39:0/3288586526 >> 172.21.15.39:0/3288586526 conn(0x7f771806e860 msgr2=0x7f771810abc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:17:23.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.673+0000 7f7720200640 1 -- 172.21.15.39:0/3288586526 shutdown_connections 2024-03-18T16:17:23.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:23.673+0000 7f7720200640 1 -- 172.21.15.39:0/3288586526 wait complete. 2024-03-18T16:17:24.724 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-18T16:17:24.724 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T16:17:24.724 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-18T16:17:25.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.084+0000 7efc61000640 1 -- 172.21.15.39:0/3807526362 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efc5c1005c0 msgr2=0x7efc5c1009e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:25.086 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.084+0000 7efc61000640 1 --2- 172.21.15.39:0/3807526362 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efc5c1005c0 0x7efc5c1009e0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7efc54009ca0 tx=0x7efc5402c120 comp rx=0 tx=0).stop 2024-03-18T16:17:25.088 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.087+0000 7efc61000640 1 -- 172.21.15.39:0/3807526362 shutdown_connections 2024-03-18T16:17:25.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.087+0000 7efc61000640 1 --2- 172.21.15.39:0/3807526362 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efc5c102170 0x7efc5c108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:25.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.087+0000 7efc61000640 1 --2- 172.21.15.39:0/3807526362 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc5c1017b0 0x7efc5c101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:25.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.087+0000 7efc61000640 1 --2- 172.21.15.39:0/3807526362 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efc5c1005c0 0x7efc5c1009e0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:25.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.087+0000 7efc61000640 1 -- 172.21.15.39:0/3807526362 >> 172.21.15.39:0/3807526362 conn(0x7efc5c0fbb50 msgr2=0x7efc5c0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:17:25.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.088+0000 7efc61000640 1 -- 172.21.15.39:0/3807526362 shutdown_connections 2024-03-18T16:17:25.089 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.088+0000 7efc61000640 1 -- 172.21.15.39:0/3807526362 wait complete. 2024-03-18T16:17:25.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.091+0000 7efc61000640 1 Processor -- start 2024-03-18T16:17:25.092 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.091+0000 7efc61000640 1 -- start start 2024-03-18T16:17:25.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.091+0000 7efc61000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc5c1005c0 0x7efc5c199c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:25.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.091+0000 7efc61000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efc5c1017b0 0x7efc5c19a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:25.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.091+0000 7efc61000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efc5c102170 0x7efc5c1a11d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:25.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.091+0000 7efc61000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7efc5c0695f0 con 0x7efc5c1005c0 2024-03-18T16:17:25.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.091+0000 7efc61000640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7efc5c069470 con 0x7efc5c102170 2024-03-18T16:17:25.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.091+0000 7efc61000640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7efc5c069770 con 0x7efc5c1017b0 2024-03-18T16:17:25.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.092+0000 7efc5aa00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efc5c1017b0 0x7efc5c19a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:25.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.092+0000 7efc5be00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efc5c102170 0x7efc5c1a11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:25.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.092+0000 7efc5aa00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efc5c1017b0 0x7efc5c19a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:53950/0 (socket says 172.21.15.39:53950) 2024-03-18T16:17:25.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.092+0000 7efc5aa00640 1 -- 172.21.15.39:0/2557680705 learned_addr learned my addr 172.21.15.39:0/2557680705 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:17:25.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.093+0000 7efc5aa00640 1 -- 172.21.15.39:0/2557680705 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efc5c102170 msgr2=0x7efc5c1a11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:25.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.093+0000 7efc5aa00640 1 --2- 172.21.15.39:0/2557680705 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efc5c102170 0x7efc5c1a11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:25.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.093+0000 7efc5aa00640 1 -- 172.21.15.39:0/2557680705 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc5c1005c0 msgr2=0x7efc5c199c60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T16:17:25.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.093+0000 7efc5b400640 1 --2- 172.21.15.39:0/2557680705 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc5c1005c0 0x7efc5c199c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:25.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.093+0000 7efc5aa00640 1 --2- 172.21.15.39:0/2557680705 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc5c1005c0 0x7efc5c199c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:25.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.093+0000 7efc5aa00640 1 -- 172.21.15.39:0/2557680705 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc5c1a1870 con 0x7efc5c1017b0 2024-03-18T16:17:25.095 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.093+0000 7efc5b400640 1 --2- 172.21.15.39:0/2557680705 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc5c1005c0 0x7efc5c199c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:17:25.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.100+0000 7efc5aa00640 1 --2- 172.21.15.39:0/2557680705 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efc5c1017b0 0x7efc5c19a1a0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7efc4400ea60 tx=0x7efc4400ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:17:25.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.101+0000 7efc53e00640 1 -- 172.21.15.39:0/2557680705 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc4400cde0 con 0x7efc5c1017b0 2024-03-18T16:17:25.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.101+0000 7efc53e00640 1 -- 172.21.15.39:0/2557680705 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efc440035f0 con 0x7efc5c1017b0 2024-03-18T16:17:25.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.101+0000 7efc53e00640 1 -- 172.21.15.39:0/2557680705 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc44010670 con 0x7efc5c1017b0 2024-03-18T16:17:25.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.101+0000 7efc61000640 1 -- 172.21.15.39:0/2557680705 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc5c1a1b60 con 0x7efc5c1017b0 2024-03-18T16:17:25.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.101+0000 7efc61000640 1 -- 172.21.15.39:0/2557680705 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc5c1a20b0 con 0x7efc5c1017b0 2024-03-18T16:17:25.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.114+0000 7efc53e00640 1 -- 172.21.15.39:0/2557680705 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7efc440037a0 con 0x7efc5c1017b0 2024-03-18T16:17:25.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.119+0000 7efc53e00640 1 --2- 172.21.15.39:0/2557680705 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7efc400a2950 0x7efc400a4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:25.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.119+0000 7efc53e00640 1 -- 172.21.15.39:0/2557680705 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7efc440cf160 con 0x7efc5c1017b0 2024-03-18T16:17:25.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.122+0000 7efc653cf740 1 Processor -- start 2024-03-18T16:17:25.124 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.122+0000 7efc653cf740 1 -- start start 2024-03-18T16:17:25.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.122+0000 7efc653cf740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d583ec4190 0x55d583ec45b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:25.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.122+0000 7efc653cf740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d583ec4b80 0x55d583ec9680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:25.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.122+0000 7efc653cf740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d583ec9cf0 0x55d583ecc180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:25.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.122+0000 7efc653cf740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55d583e1ef80 con 0x55d583ec4b80 2024-03-18T16:17:25.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.122+0000 7efc653cf740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55d583dc7ad0 con 0x55d583ec4190 2024-03-18T16:17:25.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.122+0000 7efc653cf740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55d583e1b020 con 0x55d583ec9cf0 2024-03-18T16:17:25.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.122+0000 7efc5b400640 1 --2- 172.21.15.39:0/2557680705 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7efc400a2950 0x7efc400a4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:25.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.123+0000 7efc5b400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d583ec4190 0x55d583ec45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:25.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.123+0000 7efc5be00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d583ec9cf0 0x55d583ecc180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:25.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.123+0000 7efc5be00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d583ec9cf0 0x55d583ecc180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:53958/0 (socket says 172.21.15.39:53958) 2024-03-18T16:17:25.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.123+0000 7efc5be00640 1 -- 172.21.15.39:0/1972620018 learned_addr learned my addr 172.21.15.39:0/1972620018 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:17:25.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.124+0000 7efc5be00640 1 -- 172.21.15.39:0/1972620018 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d583ec4190 msgr2=0x55d583ec45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:17:25.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.124+0000 7efc5be00640 1 --2- 172.21.15.39:0/1972620018 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d583ec4190 0x55d583ec45b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:25.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.124+0000 7efc5be00640 1 -- 172.21.15.39:0/1972620018 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d583ec4b80 msgr2=0x55d583ec9680 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:17:25.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.125+0000 7efc5be00640 1 --2- 172.21.15.39:0/1972620018 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d583ec4b80 0x55d583ec9680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:25.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.125+0000 7efc5be00640 1 -- 172.21.15.39:0/1972620018 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d583ddf6f0 con 0x55d583ec9cf0 2024-03-18T16:17:25.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.125+0000 7efc5be00640 1 --2- 172.21.15.39:0/1972620018 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d583ec9cf0 0x55d583ecc180 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7efc4c0099e0 tx=0x7efc4c009ea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=738e5d4040e0b4ec server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:17:25.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.126+0000 7efc5b400640 1 --2- 172.21.15.39:0/2557680705 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7efc400a2950 0x7efc400a4e10 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x55d583ec53e0 tx=0x7efc54000f40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:17:25.128 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.126+0000 7efc50c00640 1 -- 172.21.15.39:0/1972620018 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc4c017070 con 0x55d583ec9cf0 2024-03-18T16:17:25.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.127+0000 7efc50c00640 1 -- 172.21.15.39:0/1972620018 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efc4c013470 con 0x55d583ec9cf0 2024-03-18T16:17:25.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.127+0000 7efc50c00640 1 -- 172.21.15.39:0/1972620018 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc4c012630 con 0x55d583ec9cf0 2024-03-18T16:17:25.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.128+0000 7efc653cf740 1 -- 172.21.15.39:0/1972620018 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d583ec9cf0 msgr2=0x55d583ecc180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:25.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.128+0000 7efc653cf740 1 --2- 172.21.15.39:0/1972620018 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d583ec9cf0 0x55d583ecc180 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7efc4c0099e0 tx=0x7efc4c009ea0 comp rx=0 tx=0).stop 2024-03-18T16:17:25.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.130+0000 7efc653cf740 1 -- 172.21.15.39:0/1972620018 shutdown_connections 2024-03-18T16:17:25.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.130+0000 7efc653cf740 1 --2- 172.21.15.39:0/1972620018 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d583ec9cf0 0x55d583ecc180 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:25.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.130+0000 7efc653cf740 1 --2- 172.21.15.39:0/1972620018 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d583ec4b80 0x55d583ec9680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:25.131 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.130+0000 7efc653cf740 1 --2- 172.21.15.39:0/1972620018 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d583ec4190 0x55d583ec45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:25.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.130+0000 7efc653cf740 1 -- 172.21.15.39:0/1972620018 >> 172.21.15.39:0/1972620018 conn(0x55d583e83930 msgr2=0x55d583ebfb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:17:25.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.130+0000 7efc653cf740 1 -- 172.21.15.39:0/1972620018 shutdown_connections 2024-03-18T16:17:25.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.130+0000 7efc653cf740 1 -- 172.21.15.39:0/1972620018 wait complete. 2024-03-18T16:17:25.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.131+0000 7efc653cf740 1 -- start start 2024-03-18T16:17:25.133 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.131+0000 7efc653cf740 1 Processor -- start 2024-03-18T16:17:25.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.132+0000 7efc653cf740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d583ec4190 0x55d583ece520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:25.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.132+0000 7efc653cf740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d583ec4b80 0x55d583ecea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:25.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.132+0000 7efc653cf740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d583ec9cf0 0x55d583ed5aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:25.134 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.133+0000 7efc5aa00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d583ec4b80 0x55d583ecea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:25.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.133+0000 7efc5aa00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d583ec4b80 0x55d583ecea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:53970/0 (socket says 172.21.15.39:53970) 2024-03-18T16:17:25.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.133+0000 7efc5b400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d583ec4190 0x55d583ece520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:25.135 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.134+0000 7efc653cf740 1 -- 172.21.15.39:0/3775387073 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55d583eccca0 con 0x55d583ec4190 2024-03-18T16:17:25.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.134+0000 7efc653cf740 1 -- 172.21.15.39:0/3775387073 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55d583ddaf30 con 0x55d583ec9cf0 2024-03-18T16:17:25.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.134+0000 7efc653cf740 1 -- 172.21.15.39:0/3775387073 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55d583ecce20 con 0x55d583ec4b80 2024-03-18T16:17:25.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.133+0000 7efc5aa00640 1 -- 172.21.15.39:0/3775387073 learned_addr learned my addr 172.21.15.39:0/3775387073 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:17:25.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.134+0000 7efc5be00640 1 --2- 172.21.15.39:0/3775387073 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d583ec9cf0 0x55d583ed5aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:25.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.134+0000 7efc653cf740 10 client.0 _ll_register_callbacks cb 0x55d583dc8bc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:17:25.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.134+0000 7efc5be00640 1 -- 172.21.15.39:0/3775387073 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d583ec4b80 msgr2=0x55d583ecea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:17:25.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.134+0000 7efc5be00640 1 --2- 172.21.15.39:0/3775387073 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d583ec4b80 0x55d583ecea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:25.136 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.134+0000 7efc5be00640 1 -- 172.21.15.39:0/3775387073 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d583ec4190 msgr2=0x55d583ece520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:17:25.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.134+0000 7efc5be00640 1 --2- 172.21.15.39:0/3775387073 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d583ec4190 0x55d583ece520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:25.137 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.134+0000 7efc5be00640 1 -- 172.21.15.39:0/3775387073 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d583bfb7e0 con 0x55d583ec9cf0 2024-03-18T16:17:25.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.135+0000 7efc5be00640 1 --2- 172.21.15.39:0/3775387073 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d583ec9cf0 0x55d583ed5aa0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7efc4c00a010 tx=0x7efc4c013de0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea9a3bb0bd24554c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:17:25.138 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.135+0000 7efc49600640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:17:25.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.137+0000 7efc49600640 1 -- 172.21.15.39:0/3775387073 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc4c017040 con 0x55d583ec9cf0 2024-03-18T16:17:25.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.137+0000 7efc49600640 1 -- 172.21.15.39:0/3775387073 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7efc4c01c450 con 0x55d583ec9cf0 2024-03-18T16:17:25.139 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.137+0000 7efc653cf740 1 -- 172.21.15.39:0/3775387073 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efc4c013470 con 0x55d583ec9cf0 2024-03-18T16:17:25.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.138+0000 7efc49600640 1 -- 172.21.15.39:0/3775387073 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7efc4c01c650 con 0x55d583ec9cf0 2024-03-18T16:17:25.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.138+0000 7efc653cf740 10 client.27161 fetch_fsmap learned FSMap version 44 2024-03-18T16:17:25.140 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.138+0000 7efc653cf740 1 -- 172.21.15.39:0/3775387073 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x55d583c5bf70 con 0x55d583ec9cf0 2024-03-18T16:17:25.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.140+0000 7efc49600640 1 -- 172.21.15.39:0/3775387073 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efc4c013470 con 0x55d583ec9cf0 2024-03-18T16:17:25.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.140+0000 7efc49600640 1 -- 172.21.15.39:0/3775387073 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7efc4c026a40 con 0x55d583ec9cf0 2024-03-18T16:17:25.145 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.144+0000 7efc653cf740 10 client.27161 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:17:25.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.144+0000 7efc653cf740 10 client.27161 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:17:25.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.144+0000 7efc653cf740 20 client.27161 populate_metadata read hostname 'smithi039' 2024-03-18T16:17:25.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.144+0000 7efc653cf740 1 --2- 172.21.15.39:0/3775387073 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55d583ed9f70 0x55d583edc360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:25.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.144+0000 7efc653cf740 4 client.27161 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:17:25.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.144+0000 7efc653cf740 1 -- 172.21.15.39:0/3775387073 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d583e2c4c0 con 0x55d583ed9f70 2024-03-18T16:17:25.146 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.145+0000 7efc5b400640 1 --2- 172.21.15.39:0/3775387073 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55d583ed9f70 0x55d583edc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:25.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.145+0000 7efc5b400640 1 --2- 172.21.15.39:0/3775387073 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55d583ed9f70 0x55d583edc360 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fae4468ed61b65d3 server_cookie=a97f871841010ffd in_seq=0 out_seq=0 2024-03-18T16:17:25.147 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.145+0000 7efc49600640 10 client.27161 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:17:25.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.147+0000 7efc49600640 1 -- 172.21.15.39:0/3775387073 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d583e2c4c0 con 0x55d583ed9f70 2024-03-18T16:17:25.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.147+0000 7efc49600640 10 client.27161 handle_command_reply: tid=0 2024-03-18T16:17:25.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.147+0000 7efc653cf740 1 client.27161 shutdown 2024-03-18T16:17:25.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.148+0000 7efc653cf740 1 -- 172.21.15.39:0/3775387073 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d583ec9cf0 msgr2=0x55d583ed5aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:25.149 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.148+0000 7efc653cf740 1 --2- 172.21.15.39:0/3775387073 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d583ec9cf0 0x55d583ed5aa0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7efc4c00a010 tx=0x7efc4c013de0 comp rx=0 tx=0).stop 2024-03-18T16:17:25.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.149+0000 7efc653cf740 1 -- 172.21.15.39:0/3775387073 shutdown_connections 2024-03-18T16:17:25.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.149+0000 7efc653cf740 1 --2- 172.21.15.39:0/3775387073 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55d583ed9f70 0x55d583edc360 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:25.150 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.149+0000 7efc653cf740 1 --2- 172.21.15.39:0/3775387073 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d583ec9cf0 0x55d583ed5aa0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:25.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.149+0000 7efc653cf740 1 --2- 172.21.15.39:0/3775387073 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d583ec4b80 0x55d583ecea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:25.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.149+0000 7efc653cf740 1 --2- 172.21.15.39:0/3775387073 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d583ec4190 0x55d583ece520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:25.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.149+0000 7efc653cf740 1 -- 172.21.15.39:0/3775387073 >> 172.21.15.39:0/3775387073 conn(0x55d583e83930 msgr2=0x55d583eb6570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:17:25.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.149+0000 7efc49600640 0 client.27161 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:17:25.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.149+0000 7efc653cf740 1 -- 172.21.15.39:0/3775387073 shutdown_connections 2024-03-18T16:17:25.151 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.150+0000 7efc653cf740 1 -- 172.21.15.39:0/3775387073 wait complete. 2024-03-18T16:17:25.152 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.150+0000 7efc653cf740 20 client.27161 trim_cache size 0 max 16384 2024-03-18T16:17:25.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.190+0000 7efc653cf740 1 Processor -- start 2024-03-18T16:17:25.192 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.191+0000 7efc653cf740 1 -- start start 2024-03-18T16:17:25.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.191+0000 7efc653cf740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d583e83930 0x55d583ec2d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:25.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.191+0000 7efc653cf740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d583ec4190 0x55d583ed59c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:25.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.191+0000 7efc653cf740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d583ec4b80 0x55d583ec3280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:25.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.191+0000 7efc653cf740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7efc5c1a8370 con 0x55d583ec4b80 2024-03-18T16:17:25.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.191+0000 7efc653cf740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7efc5c1a9eb0 con 0x55d583e83930 2024-03-18T16:17:25.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.191+0000 7efc653cf740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7efc5c1a8a40 con 0x55d583ec4190 2024-03-18T16:17:25.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.192+0000 7efc5b400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d583e83930 0x55d583ec2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:25.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.192+0000 7efc5b400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d583e83930 0x55d583ec2d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:42028/0 (socket says 172.21.15.39:42028) 2024-03-18T16:17:25.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.192+0000 7efc5b400640 1 -- 172.21.15.39:0/3517287376 learned_addr learned my addr 172.21.15.39:0/3517287376 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:17:25.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.193+0000 7efc5b400640 1 -- 172.21.15.39:0/3517287376 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d583ec4190 msgr2=0x55d583ed59c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:17:25.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.193+0000 7efc5b400640 1 --2- 172.21.15.39:0/3517287376 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d583ec4190 0x55d583ed59c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:25.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.193+0000 7efc5b400640 1 -- 172.21.15.39:0/3517287376 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d583ec4b80 msgr2=0x55d583ec3280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:17:25.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.193+0000 7efc5b400640 1 --2- 172.21.15.39:0/3517287376 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d583ec4b80 0x55d583ec3280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:25.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.193+0000 7efc5b400640 1 -- 172.21.15.39:0/3517287376 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d583af3170 con 0x55d583e83930 2024-03-18T16:17:25.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.195+0000 7efc5b400640 1 --2- 172.21.15.39:0/3517287376 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d583e83930 0x55d583ec2d40 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7efc5402e010 tx=0x7efc5403e000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=42dbf7fae37c2916 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:17:25.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.197+0000 7efc4be00640 1 -- 172.21.15.39:0/3517287376 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc54039030 con 0x55d583e83930 2024-03-18T16:17:25.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.197+0000 7efc4be00640 1 -- 172.21.15.39:0/3517287376 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efc54057070 con 0x55d583e83930 2024-03-18T16:17:25.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.197+0000 7efc4be00640 1 -- 172.21.15.39:0/3517287376 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc540349a0 con 0x55d583e83930 2024-03-18T16:17:25.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.198+0000 7efc653cf740 1 -- 172.21.15.39:0/3517287376 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d583e83930 msgr2=0x55d583ec2d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:25.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.198+0000 7efc653cf740 1 --2- 172.21.15.39:0/3517287376 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d583e83930 0x55d583ec2d40 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7efc5402e010 tx=0x7efc5403e000 comp rx=0 tx=0).stop 2024-03-18T16:17:25.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.199+0000 7efc653cf740 1 -- 172.21.15.39:0/3517287376 shutdown_connections 2024-03-18T16:17:25.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.199+0000 7efc653cf740 1 --2- 172.21.15.39:0/3517287376 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d583ec4b80 0x55d583ec3280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:25.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.199+0000 7efc653cf740 1 --2- 172.21.15.39:0/3517287376 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d583ec4190 0x55d583ed59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:25.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.199+0000 7efc653cf740 1 --2- 172.21.15.39:0/3517287376 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d583e83930 0x55d583ec2d40 secure :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7efc5402e010 tx=0x7efc5403e000 comp rx=0 tx=0).stop 2024-03-18T16:17:25.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.199+0000 7efc653cf740 1 -- 172.21.15.39:0/3517287376 >> 172.21.15.39:0/3517287376 conn(0x55d583ec5780 msgr2=0x55d583ece500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:17:25.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.199+0000 7efc653cf740 1 -- 172.21.15.39:0/3517287376 shutdown_connections 2024-03-18T16:17:25.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.199+0000 7efc653cf740 1 -- 172.21.15.39:0/3517287376 wait complete. 2024-03-18T16:17:25.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.200+0000 7efc653cf740 1 -- start start 2024-03-18T16:17:25.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.200+0000 7efc653cf740 1 Processor -- start 2024-03-18T16:17:25.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.202+0000 7efc653cf740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d583ec4190 0x55d583ed39e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:25.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.203+0000 7efc653cf740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d583ec4b80 0x55d583ed3f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:25.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.203+0000 7efc653cf740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d583ec9cf0 0x55d583ed4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:25.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.203+0000 7efc5b400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d583ec4190 0x55d583ed39e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:25.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.203+0000 7efc5b400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d583ec4190 0x55d583ed39e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:42034/0 (socket says 172.21.15.39:42034) 2024-03-18T16:17:25.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.203+0000 7efc5aa00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d583ec4b80 0x55d583ed3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:25.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.203+0000 7efc5aa00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d583ec4b80 0x55d583ed3f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:53984/0 (socket says 172.21.15.39:53984) 2024-03-18T16:17:25.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.204+0000 7efc5b400640 1 -- 172.21.15.39:0/2040453937 learned_addr learned my addr 172.21.15.39:0/2040453937 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:17:25.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.204+0000 7efc653cf740 1 -- 172.21.15.39:0/2040453937 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7efc5c1a97e0 con 0x55d583ec9cf0 2024-03-18T16:17:25.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.204+0000 7efc653cf740 1 -- 172.21.15.39:0/2040453937 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7efc5c1a9110 con 0x55d583ec4190 2024-03-18T16:17:25.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.204+0000 7efc653cf740 1 -- 172.21.15.39:0/2040453937 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7efc5c1a6350 con 0x55d583ec4b80 2024-03-18T16:17:25.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.204+0000 7efc653cf740 10 client.0 _ll_register_callbacks cb 0x7efc4c009860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:17:25.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.204+0000 7efc5be00640 1 --2- 172.21.15.39:0/2040453937 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d583ec9cf0 0x55d583ed4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:25.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.205+0000 7efc5aa00640 1 -- 172.21.15.39:0/2040453937 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d583ec4190 msgr2=0x55d583ed39e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:17:25.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.205+0000 7efc5aa00640 1 --2- 172.21.15.39:0/2040453937 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d583ec4190 0x55d583ed39e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:25.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.205+0000 7efc5aa00640 1 -- 172.21.15.39:0/2040453937 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d583ec9cf0 msgr2=0x55d583ed4ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:17:25.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.205+0000 7efc5aa00640 1 --2- 172.21.15.39:0/2040453937 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d583ec9cf0 0x55d583ed4ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:25.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.205+0000 7efc5aa00640 1 -- 172.21.15.39:0/2040453937 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d583ed54d0 con 0x55d583ec4b80 2024-03-18T16:17:25.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.205+0000 7efc5aa00640 1 --2- 172.21.15.39:0/2040453937 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d583ec4b80 0x55d583ed3f20 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7efc44097ae0 tx=0x7efc4409b070 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ae71eec3ba1fcf23 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:17:25.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.205+0000 7efc49600640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:17:25.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.206+0000 7efc49600640 1 -- 172.21.15.39:0/2040453937 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc440b3070 con 0x55d583ec4b80 2024-03-18T16:17:25.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.206+0000 7efc653cf740 1 -- 172.21.15.39:0/2040453937 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efc54057070 con 0x55d583ec4b80 2024-03-18T16:17:25.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.206+0000 7efc49600640 1 -- 172.21.15.39:0/2040453937 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7efc4409ab60 con 0x55d583ec4b80 2024-03-18T16:17:25.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.207+0000 7efc49600640 1 -- 172.21.15.39:0/2040453937 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7efc4409b710 con 0x55d583ec4b80 2024-03-18T16:17:25.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.207+0000 7efc653cf740 10 client.27145 fetch_fsmap learned FSMap version 44 2024-03-18T16:17:25.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.207+0000 7efc653cf740 1 -- 172.21.15.39:0/2040453937 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x55d583f40a30 con 0x55d583ec4b80 2024-03-18T16:17:25.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.207+0000 7efc49600640 1 -- 172.21.15.39:0/2040453937 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efc440a6030 con 0x55d583ec4b80 2024-03-18T16:17:25.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.207+0000 7efc49600640 1 -- 172.21.15.39:0/2040453937 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7efc44004000 con 0x55d583ec4b80 2024-03-18T16:17:25.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.208+0000 7efc653cf740 10 client.27145 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:17:25.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.208+0000 7efc653cf740 10 client.27145 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:17:25.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.208+0000 7efc653cf740 20 client.27145 populate_metadata read hostname 'smithi039' 2024-03-18T16:17:25.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.208+0000 7efc653cf740 1 --2- 172.21.15.39:0/2040453937 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55d583f41410 0x55d583f43830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:25.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.208+0000 7efc653cf740 4 client.27145 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-18T16:17:25.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.208+0000 7efc653cf740 1 -- 172.21.15.39:0/2040453937 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55d583e26810 con 0x55d583f41410 2024-03-18T16:17:25.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.209+0000 7efc5b400640 1 --2- 172.21.15.39:0/2040453937 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55d583f41410 0x55d583f43830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:25.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.214+0000 7efc5b400640 1 --2- 172.21.15.39:0/2040453937 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55d583f41410 0x55d583f43830 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=89957515c0203162 server_cookie=9410de136e868f8f in_seq=0 out_seq=0 2024-03-18T16:17:25.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.214+0000 7efc49600640 10 client.27145 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:17:25.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.215+0000 7efc49600640 1 -- 172.21.15.39:0/2040453937 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55d583e26810 con 0x55d583f41410 2024-03-18T16:17:25.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.215+0000 7efc49600640 10 client.27145 handle_command_reply: tid=0 2024-03-18T16:17:25.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.215+0000 7efc653cf740 1 client.27145 shutdown 2024-03-18T16:17:25.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.216+0000 7efc653cf740 1 -- 172.21.15.39:0/2040453937 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d583ec4b80 msgr2=0x55d583ed3f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:25.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.216+0000 7efc653cf740 1 --2- 172.21.15.39:0/2040453937 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d583ec4b80 0x55d583ed3f20 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7efc44097ae0 tx=0x7efc4409b070 comp rx=0 tx=0).stop 2024-03-18T16:17:25.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.216+0000 7efc653cf740 1 -- 172.21.15.39:0/2040453937 shutdown_connections 2024-03-18T16:17:25.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.216+0000 7efc653cf740 1 --2- 172.21.15.39:0/2040453937 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55d583f41410 0x55d583f43830 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:25.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.216+0000 7efc653cf740 1 --2- 172.21.15.39:0/2040453937 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d583ec9cf0 0x55d583ed4ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:25.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.216+0000 7efc653cf740 1 --2- 172.21.15.39:0/2040453937 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d583ec4b80 0x55d583ed3f20 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:25.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.216+0000 7efc653cf740 1 --2- 172.21.15.39:0/2040453937 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d583ec4190 0x55d583ed39e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:25.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.216+0000 7efc653cf740 1 -- 172.21.15.39:0/2040453937 >> 172.21.15.39:0/2040453937 conn(0x55d583ec5780 msgr2=0x55d583eb5ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:17:25.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.216+0000 7efc49600640 0 client.27145 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:17:25.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.216+0000 7efc653cf740 1 -- 172.21.15.39:0/2040453937 shutdown_connections 2024-03-18T16:17:25.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.217+0000 7efc653cf740 1 -- 172.21.15.39:0/2040453937 wait complete. 2024-03-18T16:17:25.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.217+0000 7efc653cf740 20 client.27145 trim_cache size 0 max 16384 2024-03-18T16:17:25.219 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-03-18T16:17:25.219 INFO:teuthology.orchestra.run.smithi039.stdout: "return_code": 0, 2024-03-18T16:17:25.219 INFO:teuthology.orchestra.run.smithi039.stdout: "scrub_tag": "8ff8eef0-aa92-4da8-a1db-f45b11e4ab8a", 2024-03-18T16:17:25.219 INFO:teuthology.orchestra.run.smithi039.stdout: "mode": "asynchronous" 2024-03-18T16:17:25.219 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-03-18T16:17:25.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.219+0000 7efc61000640 1 -- 172.21.15.39:0/2557680705 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7efc400a2950 msgr2=0x7efc400a4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:25.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.219+0000 7efc61000640 1 --2- 172.21.15.39:0/2557680705 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7efc400a2950 0x7efc400a4e10 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x55d583ec53e0 tx=0x7efc54000f40 comp rx=0 tx=0).stop 2024-03-18T16:17:25.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.219+0000 7efc61000640 1 -- 172.21.15.39:0/2557680705 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efc5c1017b0 msgr2=0x7efc5c19a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:25.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.219+0000 7efc61000640 1 --2- 172.21.15.39:0/2557680705 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efc5c1017b0 0x7efc5c19a1a0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7efc4400ea60 tx=0x7efc4400ef20 comp rx=0 tx=0).stop 2024-03-18T16:17:25.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.219+0000 7efc61000640 1 -- 172.21.15.39:0/2557680705 shutdown_connections 2024-03-18T16:17:25.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.219+0000 7efc61000640 1 --2- 172.21.15.39:0/2557680705 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efc5c102170 0x7efc5c1a11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:25.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.219+0000 7efc61000640 1 --2- 172.21.15.39:0/2557680705 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7efc400a2950 0x7efc400a4e10 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:25.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.219+0000 7efc61000640 1 --2- 172.21.15.39:0/2557680705 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efc5c1017b0 0x7efc5c19a1a0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:25.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.219+0000 7efc61000640 1 --2- 172.21.15.39:0/2557680705 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7efc5c1005c0 0x7efc5c199c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:25.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.219+0000 7efc61000640 1 -- 172.21.15.39:0/2557680705 >> 172.21.15.39:0/2557680705 conn(0x7efc5c0fbb50 msgr2=0x7efc5c0fc490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:17:25.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.219+0000 7efc61000640 1 -- 172.21.15.39:0/2557680705 shutdown_connections 2024-03-18T16:17:25.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.220+0000 7efc61000640 1 -- 172.21.15.39:0/2557680705 wait complete. 2024-03-18T16:17:25.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:17:24 smithi196 ceph-mon[36307]: pgmap v642: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 25 MiB/s wr, 5.25k op/s 2024-03-18T16:17:25.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:17:24 smithi194 ceph-mon[36200]: pgmap v642: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 25 MiB/s wr, 5.25k op/s 2024-03-18T16:17:25.250 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T16:17:25.250 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T16:17:25.251 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:17:24 smithi039 ceph-mon[31721]: pgmap v642: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 25 MiB/s wr, 5.25k op/s 2024-03-18T16:17:25.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.600+0000 7fde4ac00640 1 -- 172.21.15.39:0/1610977489 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fde44102180 msgr2=0x7fde44104570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:25.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.600+0000 7fde4ac00640 1 --2- 172.21.15.39:0/1610977489 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fde44102180 0x7fde44104570 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fde38009860 tx=0x7fde3802bb90 comp rx=0 tx=0).stop 2024-03-18T16:17:25.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.601+0000 7fde4ac00640 1 -- 172.21.15.39:0/1610977489 shutdown_connections 2024-03-18T16:17:25.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.601+0000 7fde4ac00640 1 --2- 172.21.15.39:0/1610977489 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fde441073e0 0x7fde441097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:25.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.601+0000 7fde4ac00640 1 --2- 172.21.15.39:0/1610977489 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fde44104ab0 0x7fde44106ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:25.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.601+0000 7fde4ac00640 1 --2- 172.21.15.39:0/1610977489 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fde44102180 0x7fde44104570 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:25.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.601+0000 7fde4ac00640 1 -- 172.21.15.39:0/1610977489 >> 172.21.15.39:0/1610977489 conn(0x7fde440fbb50 msgr2=0x7fde440fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:17:25.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.601+0000 7fde4ac00640 1 -- 172.21.15.39:0/1610977489 shutdown_connections 2024-03-18T16:17:25.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.601+0000 7fde4ac00640 1 -- 172.21.15.39:0/1610977489 wait complete. 2024-03-18T16:17:25.604 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.603+0000 7fde4ac00640 1 Processor -- start 2024-03-18T16:17:25.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.603+0000 7fde4ac00640 1 -- start start 2024-03-18T16:17:25.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.605+0000 7fde4ac00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fde44104ab0 0x7fde44195a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:25.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.605+0000 7fde4ac00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fde441073e0 0x7fde44195fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:25.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.605+0000 7fde4ac00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fde4419b030 0x7fde4419d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:25.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.605+0000 7fde4ac00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fde4410d3b0 con 0x7fde4419b030 2024-03-18T16:17:25.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.605+0000 7fde49800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fde44104ab0 0x7fde44195a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:25.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.605+0000 7fde49800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fde44104ab0 0x7fde44195a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:42050/0 (socket says 172.21.15.39:42050) 2024-03-18T16:17:25.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.605+0000 7fde49800640 1 -- 172.21.15.39:0/4283182558 learned_addr learned my addr 172.21.15.39:0/4283182558 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:17:25.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.605+0000 7fde48e00640 1 --2- 172.21.15.39:0/4283182558 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fde441073e0 0x7fde44195fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:25.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.605+0000 7fde4ac00640 1 -- 172.21.15.39:0/4283182558 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fde4410d230 con 0x7fde44104ab0 2024-03-18T16:17:25.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.605+0000 7fde4ac00640 1 -- 172.21.15.39:0/4283182558 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fde4410d530 con 0x7fde441073e0 2024-03-18T16:17:25.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.606+0000 7fde4a200640 1 --2- 172.21.15.39:0/4283182558 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fde4419b030 0x7fde4419d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:25.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.607+0000 7fde4a200640 1 -- 172.21.15.39:0/4283182558 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fde441073e0 msgr2=0x7fde44195fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:25.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.607+0000 7fde4a200640 1 --2- 172.21.15.39:0/4283182558 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fde441073e0 0x7fde44195fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:25.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.607+0000 7fde4a200640 1 -- 172.21.15.39:0/4283182558 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fde44104ab0 msgr2=0x7fde44195a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:25.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.607+0000 7fde4a200640 1 --2- 172.21.15.39:0/4283182558 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fde44104ab0 0x7fde44195a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:25.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.607+0000 7fde4a200640 1 -- 172.21.15.39:0/4283182558 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde4419dab0 con 0x7fde4419b030 2024-03-18T16:17:25.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.607+0000 7fde4a200640 1 --2- 172.21.15.39:0/4283182558 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fde4419b030 0x7fde4419d4f0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fde3000b410 tx=0x7fde3000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:17:25.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.608+0000 7fde42000640 1 -- 172.21.15.39:0/4283182558 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde30015020 con 0x7fde4419b030 2024-03-18T16:17:25.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.608+0000 7fde42000640 1 -- 172.21.15.39:0/4283182558 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde30011d10 con 0x7fde4419b030 2024-03-18T16:17:25.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.608+0000 7fde42000640 1 -- 172.21.15.39:0/4283182558 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde3000f2e0 con 0x7fde4419b030 2024-03-18T16:17:25.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.609+0000 7fde4ac00640 1 -- 172.21.15.39:0/4283182558 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde4419dda0 con 0x7fde4419b030 2024-03-18T16:17:25.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.609+0000 7fde4ac00640 1 -- 172.21.15.39:0/4283182558 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde4419e320 con 0x7fde4419b030 2024-03-18T16:17:25.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.613+0000 7fde4efe5740 1 Processor -- start 2024-03-18T16:17:25.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.615+0000 7fde42000640 1 -- 172.21.15.39:0/4283182558 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fde30018020 con 0x7fde4419b030 2024-03-18T16:17:25.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.618+0000 7fde4efe5740 1 -- start start 2024-03-18T16:17:25.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.618+0000 7fde4efe5740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e512ff7a10 0x55e512ff7e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:25.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.618+0000 7fde4efe5740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e512ff8400 0x55e512ffcf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:25.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.618+0000 7fde4efe5740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e512ffd570 0x55e512fffa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:25.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.618+0000 7fde4efe5740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55e512f5c1f0 con 0x55e512ff7a10 2024-03-18T16:17:25.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.618+0000 7fde4efe5740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55e512f04a80 con 0x55e512ffd570 2024-03-18T16:17:25.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.618+0000 7fde4efe5740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55e512f58290 con 0x55e512ff8400 2024-03-18T16:17:25.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.619+0000 7fde49800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e512ff8400 0x55e512ffcf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:25.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.619+0000 7fde49800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e512ff8400 0x55e512ffcf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:54016/0 (socket says 172.21.15.39:54016) 2024-03-18T16:17:25.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.619+0000 7fde49800640 1 -- 172.21.15.39:0/965937335 learned_addr learned my addr 172.21.15.39:0/965937335 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:17:25.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.619+0000 7fde42000640 1 --2- 172.21.15.39:0/4283182558 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fde2c0a2880 0x7fde2c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:25.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.619+0000 7fde42000640 1 -- 172.21.15.39:0/4283182558 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fde300ce560 con 0x7fde4419b030 2024-03-18T16:17:25.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.620+0000 7fde48e00640 1 --2- 172.21.15.39:0/965937335 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e512ff7a10 0x55e512ff7e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:25.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.620+0000 7fde4a200640 1 --2- 172.21.15.39:0/4283182558 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fde2c0a2880 0x7fde2c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:25.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.621+0000 7fde48e00640 1 --2- 172.21.15.39:0/965937335 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e512ffd570 0x55e512fffa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:25.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.621+0000 7fde49800640 1 -- 172.21.15.39:0/965937335 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e512ffd570 msgr2=0x55e512fffa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:17:25.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.621+0000 7fde49800640 1 --2- 172.21.15.39:0/965937335 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e512ffd570 0x55e512fffa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:25.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.621+0000 7fde49800640 1 -- 172.21.15.39:0/965937335 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e512ff7a10 msgr2=0x55e512ff7e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:17:25.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.622+0000 7fde4a200640 1 --2- 172.21.15.39:0/4283182558 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fde2c0a2880 0x7fde2c0a4d40 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fde3000bdd0 tx=0x7fde300047d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:17:25.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.622+0000 7fde49800640 1 --2- 172.21.15.39:0/965937335 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e512ff7a10 0x55e512ff7e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:25.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.622+0000 7fde49800640 1 -- 172.21.15.39:0/965937335 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e512d3a7d0 con 0x55e512ff8400 2024-03-18T16:17:25.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.622+0000 7fde48e00640 1 --2- 172.21.15.39:0/965937335 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e512ff7a10 0x55e512ff7e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:17:25.624 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.622+0000 7fde49800640 1 --2- 172.21.15.39:0/965937335 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e512ff8400 0x55e512ffcf00 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fde380057c0 tx=0x7fde38009520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ef01e9a0111e2f8c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:17:25.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.626+0000 7fde36a00640 1 -- 172.21.15.39:0/965937335 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde38043070 con 0x55e512ff8400 2024-03-18T16:17:25.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.626+0000 7fde36a00640 1 -- 172.21.15.39:0/965937335 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde38034e50 con 0x55e512ff8400 2024-03-18T16:17:25.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.626+0000 7fde36a00640 1 -- 172.21.15.39:0/965937335 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde38039040 con 0x55e512ff8400 2024-03-18T16:17:25.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.627+0000 7fde4efe5740 1 -- 172.21.15.39:0/965937335 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e512ff8400 msgr2=0x55e512ffcf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:25.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.627+0000 7fde4efe5740 1 --2- 172.21.15.39:0/965937335 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e512ff8400 0x55e512ffcf00 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7fde380057c0 tx=0x7fde38009520 comp rx=0 tx=0).stop 2024-03-18T16:17:25.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.628+0000 7fde4efe5740 1 -- 172.21.15.39:0/965937335 shutdown_connections 2024-03-18T16:17:25.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.629+0000 7fde4efe5740 1 --2- 172.21.15.39:0/965937335 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e512ffd570 0x55e512fffa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:25.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.629+0000 7fde4efe5740 1 --2- 172.21.15.39:0/965937335 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e512ff8400 0x55e512ffcf00 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:25.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.629+0000 7fde4efe5740 1 --2- 172.21.15.39:0/965937335 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e512ff7a10 0x55e512ff7e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:25.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.629+0000 7fde4efe5740 1 -- 172.21.15.39:0/965937335 >> 172.21.15.39:0/965937335 conn(0x55e512ff2f80 msgr2=0x55e512ff3380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:17:25.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.629+0000 7fde4efe5740 1 -- 172.21.15.39:0/965937335 shutdown_connections 2024-03-18T16:17:25.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.630+0000 7fde4efe5740 1 -- 172.21.15.39:0/965937335 wait complete. 2024-03-18T16:17:25.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.631+0000 7fde4efe5740 1 -- start start 2024-03-18T16:17:25.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.632+0000 7fde4efe5740 1 Processor -- start 2024-03-18T16:17:25.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.633+0000 7fde4efe5740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e512ff7a10 0x55e513012f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:25.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.633+0000 7fde4efe5740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e512ff8400 0x55e5130134a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:25.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.633+0000 7fde4efe5740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e512ffd570 0x55e51301a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:25.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.633+0000 7fde4efe5740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55e5130114d0 con 0x55e512ff7a10 2024-03-18T16:17:25.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.634+0000 7fde4efe5740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55e512f17ed0 con 0x55e512ffd570 2024-03-18T16:17:25.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.634+0000 7fde4efe5740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55e513011650 con 0x55e512ff8400 2024-03-18T16:17:25.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.634+0000 7fde4efe5740 10 client.0 _ll_register_callbacks cb 0x55e512f53220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:17:25.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.635+0000 7fde49800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e512ff8400 0x55e5130134a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:25.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.635+0000 7fde48e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e512ff7a10 0x55e513012f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:25.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.635+0000 7fde48e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e512ff7a10 0x55e513012f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:37592/0 (socket says 172.21.15.39:37592) 2024-03-18T16:17:25.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.635+0000 7fde48e00640 1 -- 172.21.15.39:0/4125806148 learned_addr learned my addr 172.21.15.39:0/4125806148 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:17:25.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.636+0000 7fde49800640 1 -- 172.21.15.39:0/4125806148 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e512ffd570 msgr2=0x55e51301a4e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T16:17:25.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.636+0000 7fde49800640 1 --2- 172.21.15.39:0/4125806148 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e512ffd570 0x55e51301a4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:25.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.636+0000 7fde49800640 1 -- 172.21.15.39:0/4125806148 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e512ff7a10 msgr2=0x55e513012f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:17:25.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.636+0000 7fde49800640 1 --2- 172.21.15.39:0/4125806148 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e512ff7a10 0x55e513012f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:25.639 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.636+0000 7fde49800640 1 -- 172.21.15.39:0/4125806148 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e512c703f0 con 0x55e512ff8400 2024-03-18T16:17:25.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.638+0000 7fde49800640 1 --2- 172.21.15.39:0/4125806148 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e512ff8400 0x55e5130134a0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x55e512ff3ef0 tx=0x7fde38034cf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e725e67535e63c0e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:17:25.640 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.638+0000 7fde2b400640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:17:25.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.641+0000 7fde2b400640 1 -- 172.21.15.39:0/4125806148 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde38039030 con 0x55e512ff8400 2024-03-18T16:17:25.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.641+0000 7fde2b400640 1 -- 172.21.15.39:0/4125806148 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fde38005030 con 0x55e512ff8400 2024-03-18T16:17:25.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.641+0000 7fde4efe5740 1 -- 172.21.15.39:0/4125806148 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fde38034e50 con 0x55e512ff8400 2024-03-18T16:17:25.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.642+0000 7fde2b400640 1 -- 172.21.15.39:0/4125806148 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fde38043040 con 0x55e512ff8400 2024-03-18T16:17:25.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.643+0000 7fde4efe5740 10 client.27157 fetch_fsmap learned FSMap version 44 2024-03-18T16:17:25.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.643+0000 7fde4efe5740 1 -- 172.21.15.39:0/4125806148 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x55e512c300f0 con 0x55e512ff8400 2024-03-18T16:17:25.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.643+0000 7fde2b400640 1 -- 172.21.15.39:0/4125806148 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde38034e50 con 0x55e512ff8400 2024-03-18T16:17:25.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.644+0000 7fde2b400640 1 -- 172.21.15.39:0/4125806148 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7fde38048450 con 0x55e512ff8400 2024-03-18T16:17:25.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.644+0000 7fde4efe5740 10 client.27157 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:17:25.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.644+0000 7fde4efe5740 10 client.27157 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:17:25.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.644+0000 7fde4efe5740 20 client.27157 populate_metadata read hostname 'smithi039' 2024-03-18T16:17:25.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.644+0000 7fde4efe5740 1 --2- 172.21.15.39:0/4125806148 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55e51301e9d0 0x55e513020dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:25.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.644+0000 7fde4efe5740 4 client.27157 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:17:25.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.644+0000 7fde4efe5740 1 -- 172.21.15.39:0/4125806148 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e512f6fbe0 con 0x55e51301e9d0 2024-03-18T16:17:25.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.644+0000 7fde48e00640 1 --2- 172.21.15.39:0/4125806148 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55e51301e9d0 0x55e513020dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:25.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.645+0000 7fde48e00640 1 --2- 172.21.15.39:0/4125806148 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55e51301e9d0 0x55e513020dc0 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=77a6016d08b8f37b server_cookie=a9d5d41b035bdc34 in_seq=0 out_seq=0 2024-03-18T16:17:25.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.645+0000 7fde2b400640 10 client.27157 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:17:25.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.646+0000 7fde2b400640 1 -- 172.21.15.39:0/4125806148 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e512f6fbe0 con 0x55e51301e9d0 2024-03-18T16:17:25.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.646+0000 7fde2b400640 10 client.27157 handle_command_reply: tid=0 2024-03-18T16:17:25.649 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.646+0000 7fde4efe5740 1 client.27157 shutdown 2024-03-18T16:17:25.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.649+0000 7fde4efe5740 1 -- 172.21.15.39:0/4125806148 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e512ff8400 msgr2=0x55e5130134a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:25.650 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.649+0000 7fde4efe5740 1 --2- 172.21.15.39:0/4125806148 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e512ff8400 0x55e5130134a0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x55e512ff3ef0 tx=0x7fde38034cf0 comp rx=0 tx=0).stop 2024-03-18T16:17:25.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.649+0000 7fde4efe5740 1 -- 172.21.15.39:0/4125806148 shutdown_connections 2024-03-18T16:17:25.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.649+0000 7fde4efe5740 1 --2- 172.21.15.39:0/4125806148 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55e51301e9d0 0x55e513020dc0 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:25.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.649+0000 7fde4efe5740 1 --2- 172.21.15.39:0/4125806148 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e512ffd570 0x55e51301a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:25.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.649+0000 7fde4efe5740 1 --2- 172.21.15.39:0/4125806148 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e512ff8400 0x55e5130134a0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:25.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.649+0000 7fde4efe5740 1 --2- 172.21.15.39:0/4125806148 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e512ff7a10 0x55e513012f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:25.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.649+0000 7fde2b400640 0 client.27157 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:17:25.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.649+0000 7fde4efe5740 1 -- 172.21.15.39:0/4125806148 >> 172.21.15.39:0/4125806148 conn(0x55e512ff2f80 msgr2=0x55e512ff2110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:17:25.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.650+0000 7fde4efe5740 1 -- 172.21.15.39:0/4125806148 shutdown_connections 2024-03-18T16:17:25.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.651+0000 7fde4efe5740 1 -- 172.21.15.39:0/4125806148 wait complete. 2024-03-18T16:17:25.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.651+0000 7fde4efe5740 20 client.27157 trim_cache size 0 max 16384 2024-03-18T16:17:25.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.700+0000 7fde4efe5740 1 Processor -- start 2024-03-18T16:17:25.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.701+0000 7fde4efe5740 1 -- start start 2024-03-18T16:17:25.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.701+0000 7fde4efe5740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e512ff7a10 0x55e5130119f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:25.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.701+0000 7fde4efe5740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e512ff8400 0x55e51301a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:25.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.701+0000 7fde4efe5740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e512ffd570 0x55e512ff1880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:25.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.701+0000 7fde4efe5740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fde441b0ed0 con 0x55e512ff8400 2024-03-18T16:17:25.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.701+0000 7fde4efe5740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fde441b2a10 con 0x55e512ff7a10 2024-03-18T16:17:25.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.701+0000 7fde4efe5740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fde441b15a0 con 0x55e512ffd570 2024-03-18T16:17:25.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.702+0000 7fde48e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e512ff7a10 0x55e5130119f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:25.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.702+0000 7fde48e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e512ff7a10 0x55e5130119f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:42072/0 (socket says 172.21.15.39:42072) 2024-03-18T16:17:25.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.702+0000 7fde48e00640 1 -- 172.21.15.39:0/3236199837 learned_addr learned my addr 172.21.15.39:0/3236199837 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:17:25.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.702+0000 7fde48e00640 1 --2- 172.21.15.39:0/3236199837 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e512ffd570 0x55e512ff1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:25.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.702+0000 7fde49800640 1 --2- 172.21.15.39:0/3236199837 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e512ff8400 0x55e51301a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:25.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.703+0000 7fde48e00640 1 -- 172.21.15.39:0/3236199837 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e512ffd570 msgr2=0x55e512ff1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:17:25.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.703+0000 7fde48e00640 1 --2- 172.21.15.39:0/3236199837 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e512ffd570 0x55e512ff1880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:25.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.703+0000 7fde48e00640 1 -- 172.21.15.39:0/3236199837 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e512ff8400 msgr2=0x55e51301a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:17:25.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.703+0000 7fde48e00640 1 --2- 172.21.15.39:0/3236199837 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e512ff8400 0x55e51301a4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:25.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.704+0000 7fde48e00640 1 -- 172.21.15.39:0/3236199837 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e512ff8040 con 0x55e512ff7a10 2024-03-18T16:17:25.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.705+0000 7fde48e00640 1 --2- 172.21.15.39:0/3236199837 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e512ff7a10 0x55e5130119f0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fde3c005fd0 tx=0x7fde3c004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=62b9ec21adc444 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:17:25.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.708+0000 7fde36000640 1 -- 172.21.15.39:0/3236199837 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde3c01f040 con 0x55e512ff7a10 2024-03-18T16:17:25.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.708+0000 7fde36000640 1 -- 172.21.15.39:0/3236199837 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde3c01f1e0 con 0x55e512ff7a10 2024-03-18T16:17:25.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.708+0000 7fde36000640 1 -- 172.21.15.39:0/3236199837 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde3c005760 con 0x55e512ff7a10 2024-03-18T16:17:25.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.709+0000 7fde4efe5740 1 -- 172.21.15.39:0/3236199837 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e512ff7a10 msgr2=0x55e5130119f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:25.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.709+0000 7fde4efe5740 1 --2- 172.21.15.39:0/3236199837 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e512ff7a10 0x55e5130119f0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fde3c005fd0 tx=0x7fde3c004ab0 comp rx=0 tx=0).stop 2024-03-18T16:17:25.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.709+0000 7fde4efe5740 1 -- 172.21.15.39:0/3236199837 shutdown_connections 2024-03-18T16:17:25.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.709+0000 7fde4efe5740 1 --2- 172.21.15.39:0/3236199837 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e512ffd570 0x55e512ff1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:25.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.709+0000 7fde4efe5740 1 --2- 172.21.15.39:0/3236199837 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e512ff8400 0x55e51301a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:25.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.709+0000 7fde4efe5740 1 --2- 172.21.15.39:0/3236199837 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e512ff7a10 0x55e5130119f0 secure :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fde3c005fd0 tx=0x7fde3c004ab0 comp rx=0 tx=0).stop 2024-03-18T16:17:25.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.709+0000 7fde4efe5740 1 -- 172.21.15.39:0/3236199837 >> 172.21.15.39:0/3236199837 conn(0x55e512ff2f80 msgr2=0x55e512ff9990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:17:25.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.710+0000 7fde4efe5740 1 -- 172.21.15.39:0/3236199837 shutdown_connections 2024-03-18T16:17:25.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.710+0000 7fde4efe5740 1 -- 172.21.15.39:0/3236199837 wait complete. 2024-03-18T16:17:25.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.711+0000 7fde4efe5740 1 -- start start 2024-03-18T16:17:25.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.712+0000 7fde4efe5740 1 Processor -- start 2024-03-18T16:17:25.716 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.714+0000 7fde4efe5740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e512ff8400 0x55e513017b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:25.716 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.714+0000 7fde4efe5740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e512ffd570 0x55e513018060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:25.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.715+0000 7fde4efe5740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e513019070 0x55e5130194f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:25.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.715+0000 7fde48e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e512ff8400 0x55e513017b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:25.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.715+0000 7fde48e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e512ff8400 0x55e513017b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:54046/0 (socket says 172.21.15.39:54046) 2024-03-18T16:17:25.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.715+0000 7fde4efe5740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fde441b2340 con 0x55e512ffd570 2024-03-18T16:17:25.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.716+0000 7fde4efe5740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fde441b1c70 con 0x55e513019070 2024-03-18T16:17:25.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.716+0000 7fde4efe5740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fde44072200 con 0x55e512ff8400 2024-03-18T16:17:25.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.716+0000 7fde4efe5740 10 client.0 _ll_register_callbacks cb 0x7fde380096b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:17:25.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.715+0000 7fde49800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e512ffd570 0x55e513018060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:25.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.716+0000 7fde49800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e512ffd570 0x55e513018060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:37614/0 (socket says 172.21.15.39:37614) 2024-03-18T16:17:25.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.716+0000 7fde49800640 1 -- 172.21.15.39:0/3176118631 learned_addr learned my addr 172.21.15.39:0/3176118631 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:17:25.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.717+0000 7fde48e00640 1 --2- 172.21.15.39:0/3176118631 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e513019070 0x55e5130194f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:25.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.718+0000 7fde49800640 1 -- 172.21.15.39:0/3176118631 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e512ff8400 msgr2=0x55e513017b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:17:25.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.718+0000 7fde49800640 1 --2- 172.21.15.39:0/3176118631 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e512ff8400 0x55e513017b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:25.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.718+0000 7fde49800640 1 -- 172.21.15.39:0/3176118631 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e513019070 msgr2=0x55e5130194f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:17:25.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.718+0000 7fde49800640 1 --2- 172.21.15.39:0/3176118631 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e513019070 0x55e5130194f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:25.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.718+0000 7fde49800640 1 -- 172.21.15.39:0/3176118631 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e513019a30 con 0x55e512ffd570 2024-03-18T16:17:25.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.718+0000 7fde48e00640 1 --2- 172.21.15.39:0/3176118631 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e512ff8400 0x55e513017b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:17:25.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.718+0000 7fde49800640 1 --2- 172.21.15.39:0/3176118631 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e512ffd570 0x55e513018060 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fde3802c0d0 tx=0x7fde38034940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b5c0bf702bd454d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:17:25.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.718+0000 7fde2b400640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-03-18T16:17:25.720 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.719+0000 7fde2b400640 1 -- 172.21.15.39:0/3176118631 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde38039040 con 0x55e512ffd570 2024-03-18T16:17:25.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.719+0000 7fde2b400640 1 -- 172.21.15.39:0/3176118631 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fde38048cb0 con 0x55e512ffd570 2024-03-18T16:17:25.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.719+0000 7fde4efe5740 1 -- 172.21.15.39:0/3176118631 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fde3c01f1e0 con 0x55e512ffd570 2024-03-18T16:17:25.721 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.720+0000 7fde2b400640 1 -- 172.21.15.39:0/3176118631 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fde3802c740 con 0x55e512ffd570 2024-03-18T16:17:25.723 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.722+0000 7fde4efe5740 10 client.16611 fetch_fsmap learned FSMap version 44 2024-03-18T16:17:25.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.722+0000 7fde4efe5740 1 -- 172.21.15.39:0/3176118631 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x55e513085020 con 0x55e512ffd570 2024-03-18T16:17:25.724 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.722+0000 7fde2b400640 1 -- 172.21.15.39:0/3176118631 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde3c01f1e0 con 0x55e512ffd570 2024-03-18T16:17:25.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.723+0000 7fde2b400640 1 -- 172.21.15.39:0/3176118631 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7fde3803f440 con 0x55e512ffd570 2024-03-18T16:17:25.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.723+0000 7fde4efe5740 10 client.16611 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:17:25.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.723+0000 7fde4efe5740 10 client.16611 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:17:25.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.723+0000 7fde4efe5740 20 client.16611 populate_metadata read hostname 'smithi039' 2024-03-18T16:17:25.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.723+0000 7fde4efe5740 1 --2- 172.21.15.39:0/3176118631 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55e5130858e0 0x55e513087d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:25.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.723+0000 7fde4efe5740 4 client.16611 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T16:17:25.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.723+0000 7fde4efe5740 1 -- 172.21.15.39:0/3176118631 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e512f69730 con 0x55e5130858e0 2024-03-18T16:17:25.725 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.724+0000 7fde48e00640 1 --2- 172.21.15.39:0/3176118631 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55e5130858e0 0x55e513087d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:25.726 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.725+0000 7fde48e00640 1 --2- 172.21.15.39:0/3176118631 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55e5130858e0 0x55e513087d00 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d91543b221cd5843 server_cookie=e342b664d4a1e1cc in_seq=0 out_seq=0 2024-03-18T16:17:25.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.726+0000 7fde2b400640 10 client.16611 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:17:25.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.726+0000 7fde2b400640 1 -- 172.21.15.39:0/3176118631 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+266 (crc 0 0 0) 0x55e512f69730 con 0x55e5130858e0 2024-03-18T16:17:25.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.726+0000 7fde2b400640 10 client.16611 handle_command_reply: tid=0 2024-03-18T16:17:25.727 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.726+0000 7fde4efe5740 1 client.16611 shutdown 2024-03-18T16:17:25.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.726+0000 7fde4efe5740 1 -- 172.21.15.39:0/3176118631 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e512ffd570 msgr2=0x55e513018060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:25.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.726+0000 7fde4efe5740 1 --2- 172.21.15.39:0/3176118631 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e512ffd570 0x55e513018060 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fde3802c0d0 tx=0x7fde38034940 comp rx=0 tx=0).stop 2024-03-18T16:17:25.728 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.727+0000 7fde4efe5740 1 -- 172.21.15.39:0/3176118631 shutdown_connections 2024-03-18T16:17:25.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.727+0000 7fde4efe5740 1 --2- 172.21.15.39:0/3176118631 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55e5130858e0 0x55e513087d00 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:25.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.728+0000 7fde4efe5740 1 --2- 172.21.15.39:0/3176118631 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e513019070 0x55e5130194f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:25.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.728+0000 7fde2b400640 0 client.16611 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:17:25.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.728+0000 7fde4efe5740 1 --2- 172.21.15.39:0/3176118631 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e512ffd570 0x55e513018060 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:25.729 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.728+0000 7fde4efe5740 1 --2- 172.21.15.39:0/3176118631 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e512ff8400 0x55e513017b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:25.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.728+0000 7fde4efe5740 1 -- 172.21.15.39:0/3176118631 >> 172.21.15.39:0/3176118631 conn(0x55e512ff2f80 msgr2=0x55e512ff1880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:17:25.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.728+0000 7fde4efe5740 1 -- 172.21.15.39:0/3176118631 shutdown_connections 2024-03-18T16:17:25.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.728+0000 7fde4efe5740 1 -- 172.21.15.39:0/3176118631 wait complete. 2024-03-18T16:17:25.730 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.729+0000 7fde4efe5740 20 client.16611 trim_cache size 0 max 16384 2024-03-18T16:17:25.733 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-03-18T16:17:25.733 INFO:teuthology.orchestra.run.smithi039.stdout: "status": "scrub active (25 inodes in the stack)", 2024-03-18T16:17:25.733 INFO:teuthology.orchestra.run.smithi039.stdout: "scrubs": { 2024-03-18T16:17:25.733 INFO:teuthology.orchestra.run.smithi039.stdout: "8ff8eef0-aa92-4da8-a1db-f45b11e4ab8a": { 2024-03-18T16:17:25.733 INFO:teuthology.orchestra.run.smithi039.stdout: "path": "/", 2024-03-18T16:17:25.733 INFO:teuthology.orchestra.run.smithi039.stdout: "tag": "8ff8eef0-aa92-4da8-a1db-f45b11e4ab8a", 2024-03-18T16:17:25.733 INFO:teuthology.orchestra.run.smithi039.stdout: "options": "recursive,force" 2024-03-18T16:17:25.733 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:17:25.733 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:17:25.733 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-03-18T16:17:25.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.732+0000 7fde4ac00640 1 -- 172.21.15.39:0/4283182558 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fde2c0a2880 msgr2=0x7fde2c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:25.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.732+0000 7fde4ac00640 1 --2- 172.21.15.39:0/4283182558 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fde2c0a2880 0x7fde2c0a4d40 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fde3000bdd0 tx=0x7fde300047d0 comp rx=0 tx=0).stop 2024-03-18T16:17:25.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.732+0000 7fde4ac00640 1 -- 172.21.15.39:0/4283182558 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fde4419b030 msgr2=0x7fde4419d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:25.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.732+0000 7fde4ac00640 1 --2- 172.21.15.39:0/4283182558 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fde4419b030 0x7fde4419d4f0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fde3000b410 tx=0x7fde3000b8d0 comp rx=0 tx=0).stop 2024-03-18T16:17:25.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.733+0000 7fde4ac00640 1 -- 172.21.15.39:0/4283182558 shutdown_connections 2024-03-18T16:17:25.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.733+0000 7fde4ac00640 1 --2- 172.21.15.39:0/4283182558 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fde2c0a2880 0x7fde2c0a4d40 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:25.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.733+0000 7fde4ac00640 1 --2- 172.21.15.39:0/4283182558 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fde4419b030 0x7fde4419d4f0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:25.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.733+0000 7fde4ac00640 1 --2- 172.21.15.39:0/4283182558 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fde441073e0 0x7fde44195fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:25.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.733+0000 7fde4ac00640 1 --2- 172.21.15.39:0/4283182558 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fde44104ab0 0x7fde44195a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:25.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.733+0000 7fde4ac00640 1 -- 172.21.15.39:0/4283182558 >> 172.21.15.39:0/4283182558 conn(0x7fde440fbb50 msgr2=0x7fde44101960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:17:25.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.733+0000 7fde4ac00640 1 -- 172.21.15.39:0/4283182558 shutdown_connections 2024-03-18T16:17:25.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:25.733+0000 7fde4ac00640 1 -- 172.21.15.39:0/4283182558 wait complete. 2024-03-18T16:17:25.771 INFO:tasks.cephfs.filesystem:scrub status for tag:8ff8eef0-aa92-4da8-a1db-f45b11e4ab8a - {'path': '/', 'tag': '8ff8eef0-aa92-4da8-a1db-f45b11e4ab8a', 'options': 'recursive,force'} 2024-03-18T16:17:26.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:17:25 smithi039 ceph-mon[31721]: scrub queued for path: / 2024-03-18T16:17:26.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:17:25 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:17:26.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:17:25 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:17:26.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:17:25 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:17:26.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:17:25 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:17:26.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:17:25 smithi196 ceph-mon[36307]: scrub queued for path: / 2024-03-18T16:17:26.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:17:25 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:17:26.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:17:25 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:17:26.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:17:25 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:17:26.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:17:25 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:17:26.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:17:25 smithi194 ceph-mon[36200]: scrub queued for path: / 2024-03-18T16:17:26.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:17:25 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:17:26.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:17:25 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:17:26.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:17:25 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:17:26.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:17:25 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:17:27.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:17:26 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:17:27.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:17:26 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:17:27.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:17:26 smithi196 ceph-mon[36307]: pgmap v643: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 25 MiB/s wr, 5.17k op/s 2024-03-18T16:17:27.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:17:26 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:17:27.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:17:26 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:17:27.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:17:26 smithi194 ceph-mon[36200]: pgmap v643: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 25 MiB/s wr, 5.17k op/s 2024-03-18T16:17:27.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:17:26 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:17:27.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:17:26 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:17:27.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:17:26 smithi039 ceph-mon[31721]: pgmap v643: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 25 MiB/s wr, 5.17k op/s 2024-03-18T16:17:29.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:17:28 smithi196 ceph-mon[36307]: pgmap v644: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 25 MiB/s wr, 5.20k op/s 2024-03-18T16:17:29.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:17:28 smithi194 ceph-mon[36200]: pgmap v644: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 25 MiB/s wr, 5.20k op/s 2024-03-18T16:17:29.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:17:28 smithi039 ceph-mon[31721]: pgmap v644: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 25 MiB/s wr, 5.20k op/s 2024-03-18T16:17:31.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:17:30 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:17:31.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:17:30 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:17:31.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:17:30 smithi196 ceph-mon[36307]: pgmap v645: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 24 MiB/s wr, 5.04k op/s 2024-03-18T16:17:31.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:17:30 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:17:31.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:17:30 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:17:31.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:17:30 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:17:31.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:17:30 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:17:31.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:17:30 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:17:31.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:17:30 smithi194 ceph-mon[36200]: pgmap v645: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 24 MiB/s wr, 5.04k op/s 2024-03-18T16:17:31.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:17:30 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:17:31.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:17:30 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:17:31.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:17:30 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:17:31.340 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:17:30 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:17:31.340 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:17:30 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:17:31.340 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:17:30 smithi039 ceph-mon[31721]: pgmap v645: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 24 MiB/s wr, 5.04k op/s 2024-03-18T16:17:31.340 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:17:30 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:17:31.340 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:17:30 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:17:31.340 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:17:30 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:17:32.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:17:31 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:17:32.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:17:31 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:17:32.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:17:31 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:17:33.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:17:32 smithi194 ceph-mon[36200]: pgmap v646: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 25 MiB/s wr, 5.22k op/s 2024-03-18T16:17:33.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:17:32 smithi039 ceph-mon[31721]: pgmap v646: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 25 MiB/s wr, 5.22k op/s 2024-03-18T16:17:33.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:17:32 smithi196 ceph-mon[36307]: pgmap v646: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 25 MiB/s wr, 5.22k op/s 2024-03-18T16:17:35.280 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:17:34 smithi196 ceph-mon[36307]: pgmap v647: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 22 MiB/s wr, 4.50k op/s 2024-03-18T16:17:35.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:17:34 smithi039 ceph-mon[31721]: pgmap v647: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 22 MiB/s wr, 4.50k op/s 2024-03-18T16:17:35.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:17:34 smithi194 ceph-mon[36200]: pgmap v647: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 22 MiB/s wr, 4.50k op/s 2024-03-18T16:17:37.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:17:37 smithi039 ceph-mon[31721]: pgmap v648: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 23 MiB/s wr, 4.78k op/s 2024-03-18T16:17:37.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:17:37 smithi196 ceph-mon[36307]: pgmap v648: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 23 MiB/s wr, 4.78k op/s 2024-03-18T16:17:37.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:17:37 smithi194 ceph-mon[36200]: pgmap v648: 129 pgs: 129 active+clean; 13 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 23 MiB/s wr, 4.78k op/s 2024-03-18T16:17:39.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:17:39 smithi039 ceph-mon[31721]: pgmap v649: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 26 MiB/s wr, 5.44k op/s 2024-03-18T16:17:39.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:17:39 smithi196 ceph-mon[36307]: pgmap v649: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 26 MiB/s wr, 5.44k op/s 2024-03-18T16:17:39.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:17:39 smithi194 ceph-mon[36200]: pgmap v649: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 26 MiB/s wr, 5.44k op/s 2024-03-18T16:17:41.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:17:41 smithi039 ceph-mon[31721]: pgmap v650: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 26 MiB/s wr, 5.28k op/s 2024-03-18T16:17:41.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:17:41 smithi196 ceph-mon[36307]: pgmap v650: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 26 MiB/s wr, 5.28k op/s 2024-03-18T16:17:41.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:17:41 smithi194 ceph-mon[36200]: pgmap v650: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 26 MiB/s wr, 5.28k op/s 2024-03-18T16:17:43.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:17:43 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:17:43.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:17:43 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:17:43.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:17:43 smithi039 ceph-mon[31721]: pgmap v651: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 27 MiB/s wr, 5.54k op/s 2024-03-18T16:17:43.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:17:43 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:17:43.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:17:43 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:17:43.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:17:43 smithi196 ceph-mon[36307]: pgmap v651: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 27 MiB/s wr, 5.54k op/s 2024-03-18T16:17:43.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:17:43 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:17:43.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:17:43 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:17:43.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:17:43 smithi194 ceph-mon[36200]: pgmap v651: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 27 MiB/s wr, 5.54k op/s 2024-03-18T16:17:45.292 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:17:45 smithi194 ceph-mon[36200]: pgmap v652: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 25 MiB/s wr, 5.14k op/s 2024-03-18T16:17:45.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:17:45 smithi039 ceph-mon[31721]: pgmap v652: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 25 MiB/s wr, 5.14k op/s 2024-03-18T16:17:45.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:17:45 smithi196 ceph-mon[36307]: pgmap v652: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 KiB/s rd, 25 MiB/s wr, 5.14k op/s 2024-03-18T16:17:47.339 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:17:47 smithi039 ceph-mon[31721]: pgmap v653: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 26 MiB/s wr, 5.22k op/s 2024-03-18T16:17:47.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:17:47 smithi196 ceph-mon[36307]: pgmap v653: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 26 MiB/s wr, 5.22k op/s 2024-03-18T16:17:47.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:17:47 smithi194 ceph-mon[36200]: pgmap v653: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 KiB/s rd, 26 MiB/s wr, 5.22k op/s 2024-03-18T16:17:48.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:17:48 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:17:48.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:17:48 smithi039 ceph-mon[31721]: pgmap v654: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 26 MiB/s wr, 5.44k op/s 2024-03-18T16:17:48.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:17:48 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:17:48.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:17:48 smithi196 ceph-mon[36307]: pgmap v654: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 26 MiB/s wr, 5.44k op/s 2024-03-18T16:17:48.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:17:48 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:17:48.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:17:48 smithi194 ceph-mon[36200]: pgmap v654: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 26 MiB/s wr, 5.44k op/s 2024-03-18T16:17:51.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:17:50 smithi196 ceph-mon[36307]: pgmap v655: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 24 MiB/s wr, 4.92k op/s 2024-03-18T16:17:51.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:17:50 smithi194 ceph-mon[36200]: pgmap v655: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 24 MiB/s wr, 4.92k op/s 2024-03-18T16:17:51.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:17:50 smithi039 ceph-mon[31721]: pgmap v655: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 KiB/s rd, 24 MiB/s wr, 4.92k op/s 2024-03-18T16:17:53.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:17:52 smithi196 ceph-mon[36307]: pgmap v656: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 26 MiB/s wr, 5.43k op/s 2024-03-18T16:17:53.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:17:52 smithi194 ceph-mon[36200]: pgmap v656: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 26 MiB/s wr, 5.43k op/s 2024-03-18T16:17:53.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:17:52 smithi039 ceph-mon[31721]: pgmap v656: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 26 MiB/s wr, 5.43k op/s 2024-03-18T16:17:55.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:17:54 smithi196 ceph-mon[36307]: pgmap v657: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 22 MiB/s wr, 4.77k op/s 2024-03-18T16:17:55.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:17:54 smithi194 ceph-mon[36200]: pgmap v657: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 22 MiB/s wr, 4.77k op/s 2024-03-18T16:17:55.341 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:17:54 smithi039 ceph-mon[31721]: pgmap v657: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 22 MiB/s wr, 4.77k op/s 2024-03-18T16:17:55.772 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T16:17:55.772 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T16:17:56.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.531+0000 7fac98600640 1 -- 172.21.15.39:0/1698863581 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fac901087e0 msgr2=0x7fac90108c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:56.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.531+0000 7fac98600640 1 --2- 172.21.15.39:0/1698863581 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fac901087e0 0x7fac90108c00 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fac84009900 tx=0x7fac8402bc60 comp rx=0 tx=0).stop 2024-03-18T16:17:56.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.531+0000 7fac98600640 1 -- 172.21.15.39:0/1698863581 shutdown_connections 2024-03-18T16:17:56.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.531+0000 7fac98600640 1 --2- 172.21.15.39:0/1698863581 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fac90110060 0x7fac90112450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:56.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.531+0000 7fac98600640 1 --2- 172.21.15.39:0/1698863581 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fac90109140 0x7fac9010fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:56.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.531+0000 7fac98600640 1 --2- 172.21.15.39:0/1698863581 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fac901087e0 0x7fac90108c00 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:56.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.531+0000 7fac98600640 1 -- 172.21.15.39:0/1698863581 >> 172.21.15.39:0/1698863581 conn(0x7fac9006e640 msgr2=0x7fac90070a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:17:56.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.532+0000 7fac98600640 1 -- 172.21.15.39:0/1698863581 shutdown_connections 2024-03-18T16:17:56.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.532+0000 7fac98600640 1 -- 172.21.15.39:0/1698863581 wait complete. 2024-03-18T16:17:56.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.536+0000 7fac98600640 1 Processor -- start 2024-03-18T16:17:56.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.537+0000 7fac98600640 1 -- start start 2024-03-18T16:17:56.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.537+0000 7fac98600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fac901087e0 0x7fac901a6a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:56.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.538+0000 7fac98600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fac90109140 0x7fac901a6f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:56.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.538+0000 7fac98600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fac90110060 0x7fac901ae000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:56.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.538+0000 7fac98600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fac90115d60 con 0x7fac90109140 2024-03-18T16:17:56.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.538+0000 7fac98600640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fac90115be0 con 0x7fac901087e0 2024-03-18T16:17:56.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.538+0000 7fac98600640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fac90115ee0 con 0x7fac90110060 2024-03-18T16:17:56.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.538+0000 7fac96600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fac90110060 0x7fac901ae000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:56.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.538+0000 7fac96600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fac90110060 0x7fac901ae000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:45492/0 (socket says 172.21.15.39:45492) 2024-03-18T16:17:56.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.538+0000 7fac96600640 1 -- 172.21.15.39:0/424505277 learned_addr learned my addr 172.21.15.39:0/424505277 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:17:56.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.538+0000 7fac95c00640 1 --2- 172.21.15.39:0/424505277 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fac901087e0 0x7fac901a6a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:56.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.539+0000 7fac95c00640 1 -- 172.21.15.39:0/424505277 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fac90110060 msgr2=0x7fac901ae000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:56.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.539+0000 7fac95c00640 1 --2- 172.21.15.39:0/424505277 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fac90110060 0x7fac901ae000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:56.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.539+0000 7fac95c00640 1 -- 172.21.15.39:0/424505277 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fac90109140 msgr2=0x7fac901a6f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:56.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.539+0000 7fac95c00640 1 --2- 172.21.15.39:0/424505277 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fac90109140 0x7fac901a6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:56.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.539+0000 7fac95c00640 1 -- 172.21.15.39:0/424505277 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac901ae670 con 0x7fac901087e0 2024-03-18T16:17:56.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.539+0000 7fac95c00640 1 --2- 172.21.15.39:0/424505277 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fac901087e0 0x7fac901a6a50 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fac8402c1a0 tx=0x7fac840040c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:17:56.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.540+0000 7fac8e000640 1 -- 172.21.15.39:0/424505277 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac84043020 con 0x7fac901087e0 2024-03-18T16:17:56.544 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.540+0000 7fac8e000640 1 -- 172.21.15.39:0/424505277 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fac840044c0 con 0x7fac901087e0 2024-03-18T16:17:56.544 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.540+0000 7fac8e000640 1 -- 172.21.15.39:0/424505277 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac8403e450 con 0x7fac901087e0 2024-03-18T16:17:56.544 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.540+0000 7fac98600640 1 -- 172.21.15.39:0/424505277 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac901ae900 con 0x7fac901087e0 2024-03-18T16:17:56.544 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.540+0000 7fac98600640 1 -- 172.21.15.39:0/424505277 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac901aeea0 con 0x7fac901087e0 2024-03-18T16:17:56.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.544+0000 7fac9b20a740 1 Processor -- start 2024-03-18T16:17:56.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.545+0000 7fac9b20a740 1 -- start start 2024-03-18T16:17:56.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.545+0000 7fac9b20a740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563df4b8c810 0x563df4b8cc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:56.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.546+0000 7fac9b20a740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563df4b8d200 0x563df4b91d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:56.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.546+0000 7fac9b20a740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563df4b92370 0x563df4b94800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:56.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.546+0000 7fac96600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563df4b8c810 0x563df4b8cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:56.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.546+0000 7fac9b20a740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x563df4af11f0 con 0x563df4b8c810 2024-03-18T16:17:56.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.546+0000 7fac9b20a740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x563df4a99a80 con 0x563df4b8d200 2024-03-18T16:17:56.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.546+0000 7fac9b20a740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x563df4aed290 con 0x563df4b92370 2024-03-18T16:17:56.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.546+0000 7fac96600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563df4b8c810 0x563df4b8cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51966/0 (socket says 172.21.15.39:51966) 2024-03-18T16:17:56.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.546+0000 7fac96600640 1 -- 172.21.15.39:0/3447725404 learned_addr learned my addr 172.21.15.39:0/3447725404 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:17:56.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.546+0000 7fac8e000640 1 -- 172.21.15.39:0/424505277 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fac84046020 con 0x7fac901087e0 2024-03-18T16:17:56.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.547+0000 7fac95c00640 1 --2- 172.21.15.39:0/3447725404 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563df4b8d200 0x563df4b91d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:56.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.547+0000 7fac95200640 1 --2- 172.21.15.39:0/3447725404 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563df4b92370 0x563df4b94800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:56.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.547+0000 7fac95c00640 1 -- 172.21.15.39:0/3447725404 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563df4b92370 msgr2=0x563df4b94800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:17:56.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.547+0000 7fac95c00640 1 --2- 172.21.15.39:0/3447725404 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563df4b92370 0x563df4b94800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:56.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.547+0000 7fac95c00640 1 -- 172.21.15.39:0/3447725404 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563df4b8c810 msgr2=0x563df4b8cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:17:56.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.547+0000 7fac95c00640 1 --2- 172.21.15.39:0/3447725404 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563df4b8c810 0x563df4b8cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:56.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.547+0000 7fac95c00640 1 -- 172.21.15.39:0/3447725404 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563df48cf7d0 con 0x563df4b8d200 2024-03-18T16:17:56.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.548+0000 7fac95c00640 1 --2- 172.21.15.39:0/3447725404 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563df4b8d200 0x563df4b91d00 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fac840ee7f0 tx=0x7fac840f9220 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b5883ba32ffe7b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:17:56.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.548+0000 7fac82a00640 1 -- 172.21.15.39:0/3447725404 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac840f99c0 con 0x563df4b8d200 2024-03-18T16:17:56.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.548+0000 7fac82a00640 1 -- 172.21.15.39:0/3447725404 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fac840fe030 con 0x563df4b8d200 2024-03-18T16:17:56.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.548+0000 7fac82a00640 1 -- 172.21.15.39:0/3447725404 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac84102430 con 0x563df4b8d200 2024-03-18T16:17:56.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.549+0000 7fac8e000640 1 --2- 172.21.15.39:0/424505277 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fac780a2880 0x7fac780a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:56.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.549+0000 7fac8e000640 1 -- 172.21.15.39:0/424505277 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fac840f1140 con 0x7fac901087e0 2024-03-18T16:17:56.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.549+0000 7fac9b20a740 1 -- 172.21.15.39:0/3447725404 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563df4b8d200 msgr2=0x563df4b91d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:56.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.549+0000 7fac9b20a740 1 --2- 172.21.15.39:0/3447725404 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563df4b8d200 0x563df4b91d00 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fac840ee7f0 tx=0x7fac840f9220 comp rx=0 tx=0).stop 2024-03-18T16:17:56.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.550+0000 7fac96600640 1 --2- 172.21.15.39:0/424505277 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fac780a2880 0x7fac780a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:56.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.550+0000 7fac9b20a740 1 -- 172.21.15.39:0/3447725404 shutdown_connections 2024-03-18T16:17:56.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.550+0000 7fac9b20a740 1 --2- 172.21.15.39:0/3447725404 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563df4b8d200 0x563df4b91d00 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:56.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.550+0000 7fac9b20a740 1 --2- 172.21.15.39:0/3447725404 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563df4b92370 0x563df4b94800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:56.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.550+0000 7fac9b20a740 1 --2- 172.21.15.39:0/3447725404 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563df4b8c810 0x563df4b8cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:56.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.550+0000 7fac9b20a740 1 -- 172.21.15.39:0/3447725404 >> 172.21.15.39:0/3447725404 conn(0x563df4b87d80 msgr2=0x563df4b88180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:17:56.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.550+0000 7fac9b20a740 1 -- 172.21.15.39:0/3447725404 shutdown_connections 2024-03-18T16:17:56.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.550+0000 7fac9b20a740 1 -- 172.21.15.39:0/3447725404 wait complete. 2024-03-18T16:17:56.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.551+0000 7fac9b20a740 1 -- start start 2024-03-18T16:17:56.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.551+0000 7fac9b20a740 1 Processor -- start 2024-03-18T16:17:56.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.551+0000 7fac96600640 1 --2- 172.21.15.39:0/424505277 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fac780a2880 0x7fac780a4d40 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x563df4b8da60 tx=0x7fac8800b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:17:56.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.552+0000 7fac9b20a740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563df4b8c810 0x563df4b96a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:56.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.552+0000 7fac9b20a740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563df4b8d200 0x563df4b96f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:56.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.552+0000 7fac9b20a740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563df4b92370 0x563df4b9dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:56.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.553+0000 7fac95200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563df4b8d200 0x563df4b96f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:56.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.553+0000 7fac95200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563df4b8d200 0x563df4b96f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:42796/0 (socket says 172.21.15.39:42796) 2024-03-18T16:17:56.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.553+0000 7fac96600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563df4b92370 0x563df4b9dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:56.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.553+0000 7fac96600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563df4b92370 0x563df4b9dfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:45506/0 (socket says 172.21.15.39:45506) 2024-03-18T16:17:56.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.553+0000 7fac9b20a740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x563df4b950e0 con 0x563df4b8c810 2024-03-18T16:17:56.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.553+0000 7fac9b20a740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x563df4aaced0 con 0x563df4b8d200 2024-03-18T16:17:56.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.553+0000 7fac9b20a740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x563df4b95260 con 0x563df4b92370 2024-03-18T16:17:56.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.553+0000 7fac9b20a740 10 client.0 _ll_register_callbacks cb 0x563df4ae8220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:17:56.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.553+0000 7fac95c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563df4b8c810 0x563df4b96a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:56.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.553+0000 7fac95c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563df4b8c810 0x563df4b96a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51968/0 (socket says 172.21.15.39:51968) 2024-03-18T16:17:56.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.553+0000 7fac95c00640 1 -- 172.21.15.39:0/2567937695 learned_addr learned my addr 172.21.15.39:0/2567937695 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:17:56.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.556+0000 7fac96600640 1 -- 172.21.15.39:0/2567937695 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563df4b8d200 msgr2=0x563df4b96f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:17:56.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.556+0000 7fac96600640 1 --2- 172.21.15.39:0/2567937695 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563df4b8d200 0x563df4b96f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:56.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.556+0000 7fac96600640 1 -- 172.21.15.39:0/2567937695 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563df4b8c810 msgr2=0x563df4b96a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:17:56.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.556+0000 7fac96600640 1 --2- 172.21.15.39:0/2567937695 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563df4b8c810 0x563df4b96a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:56.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.556+0000 7fac96600640 1 -- 172.21.15.39:0/2567937695 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563df48053f0 con 0x563df4b92370 2024-03-18T16:17:56.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.558+0000 7fac96600640 1 --2- 172.21.15.39:0/2567937695 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563df4b92370 0x563df4b9dfd0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fac88007070 tx=0x7fac88004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c1ded85f13267fd0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:17:56.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.558+0000 7fac77400640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:17:56.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.561+0000 7fac77400640 1 -- 172.21.15.39:0/2567937695 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac88025020 con 0x563df4b92370 2024-03-18T16:17:56.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.561+0000 7fac77400640 1 -- 172.21.15.39:0/2567937695 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fac88017de0 con 0x563df4b92370 2024-03-18T16:17:56.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.562+0000 7fac9b20a740 1 -- 172.21.15.39:0/2567937695 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fac840fe030 con 0x563df4b92370 2024-03-18T16:17:56.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.566+0000 7fac77400640 1 -- 172.21.15.39:0/2567937695 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fac8801d040 con 0x563df4b92370 2024-03-18T16:17:56.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.566+0000 7fac9b20a740 10 client.27169 fetch_fsmap learned FSMap version 44 2024-03-18T16:17:56.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.566+0000 7fac9b20a740 1 -- 172.21.15.39:0/2567937695 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x563df47c50f0 con 0x563df4b92370 2024-03-18T16:17:56.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.566+0000 7fac77400640 1 -- 172.21.15.39:0/2567937695 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fac840fe030 con 0x563df4b92370 2024-03-18T16:17:56.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.566+0000 7fac77400640 1 -- 172.21.15.39:0/2567937695 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7fac88020030 con 0x563df4b92370 2024-03-18T16:17:56.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.567+0000 7fac9b20a740 10 client.27169 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:17:56.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.567+0000 7fac9b20a740 10 client.27169 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:17:56.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.567+0000 7fac9b20a740 20 client.27169 populate_metadata read hostname 'smithi039' 2024-03-18T16:17:56.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.567+0000 7fac9b20a740 1 --2- 172.21.15.39:0/2567937695 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x563df4ba2930 0x563df4ba4d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:56.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.568+0000 7fac9b20a740 4 client.27169 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:17:56.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.568+0000 7fac9b20a740 1 -- 172.21.15.39:0/2567937695 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563df4b04be0 con 0x563df4ba2930 2024-03-18T16:17:56.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.568+0000 7fac95c00640 1 --2- 172.21.15.39:0/2567937695 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x563df4ba2930 0x563df4ba4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:56.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.568+0000 7fac95c00640 1 --2- 172.21.15.39:0/2567937695 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x563df4ba2930 0x563df4ba4d90 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e715c17fee522e2c server_cookie=799fcedfef1303f in_seq=0 out_seq=0 2024-03-18T16:17:56.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.568+0000 7fac77400640 10 client.27169 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:17:56.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.570+0000 7fac77400640 1 -- 172.21.15.39:0/2567937695 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563df4b04be0 con 0x563df4ba2930 2024-03-18T16:17:56.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.570+0000 7fac77400640 10 client.27169 handle_command_reply: tid=0 2024-03-18T16:17:56.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.570+0000 7fac9b20a740 1 client.27169 shutdown 2024-03-18T16:17:56.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.570+0000 7fac9b20a740 1 -- 172.21.15.39:0/2567937695 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563df4b92370 msgr2=0x563df4b9dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:56.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.570+0000 7fac9b20a740 1 --2- 172.21.15.39:0/2567937695 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563df4b92370 0x563df4b9dfd0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fac88007070 tx=0x7fac88004a40 comp rx=0 tx=0).stop 2024-03-18T16:17:56.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.570+0000 7fac9b20a740 1 -- 172.21.15.39:0/2567937695 shutdown_connections 2024-03-18T16:17:56.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.570+0000 7fac9b20a740 1 --2- 172.21.15.39:0/2567937695 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x563df4ba2930 0x563df4ba4d90 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:56.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.570+0000 7fac9b20a740 1 --2- 172.21.15.39:0/2567937695 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563df4b8d200 0x563df4b96f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:56.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.570+0000 7fac9b20a740 1 --2- 172.21.15.39:0/2567937695 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563df4b92370 0x563df4b9dfd0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:56.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.570+0000 7fac9b20a740 1 --2- 172.21.15.39:0/2567937695 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563df4b8c810 0x563df4b96a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:56.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.570+0000 7fac9b20a740 1 -- 172.21.15.39:0/2567937695 >> 172.21.15.39:0/2567937695 conn(0x563df4b87d80 msgr2=0x563df4b86ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:17:56.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.570+0000 7fac77400640 0 client.27169 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:17:56.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.570+0000 7fac9b20a740 1 -- 172.21.15.39:0/2567937695 shutdown_connections 2024-03-18T16:17:56.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.571+0000 7fac9b20a740 1 -- 172.21.15.39:0/2567937695 wait complete. 2024-03-18T16:17:56.574 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.571+0000 7fac9b20a740 20 client.27169 trim_cache size 0 max 16384 2024-03-18T16:17:56.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.604+0000 7fac9b20a740 1 Processor -- start 2024-03-18T16:17:56.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.604+0000 7fac9b20a740 1 -- start start 2024-03-18T16:17:56.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.604+0000 7fac9b20a740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563df4b8c810 0x563df4b86630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:56.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.604+0000 7fac9b20a740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563df4b8d200 0x563df4b9df90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:56.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.604+0000 7fac9b20a740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563df4b92370 0x563df4b86b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:56.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.605+0000 7fac9b20a740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fac901b93c0 con 0x563df4b8d200 2024-03-18T16:17:56.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.605+0000 7fac9b20a740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fac901baf00 con 0x563df4b92370 2024-03-18T16:17:56.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.605+0000 7fac9b20a740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fac901b9a90 con 0x563df4b8c810 2024-03-18T16:17:56.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.605+0000 7fac95c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563df4b8c810 0x563df4b86630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:56.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.605+0000 7fac96600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563df4b92370 0x563df4b86b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:56.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.606+0000 7fac95c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563df4b8c810 0x563df4b86630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:45520/0 (socket says 172.21.15.39:45520) 2024-03-18T16:17:56.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.606+0000 7fac96600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563df4b92370 0x563df4b86b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:42806/0 (socket says 172.21.15.39:42806) 2024-03-18T16:17:56.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.606+0000 7fac96600640 1 -- 172.21.15.39:0/2395957685 learned_addr learned my addr 172.21.15.39:0/2395957685 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:17:56.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.606+0000 7fac95200640 1 --2- 172.21.15.39:0/2395957685 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563df4b8d200 0x563df4b9df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:56.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.606+0000 7fac95c00640 1 -- 172.21.15.39:0/2395957685 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563df4b92370 msgr2=0x563df4b86b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:17:56.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.606+0000 7fac95c00640 1 --2- 172.21.15.39:0/2395957685 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563df4b92370 0x563df4b86b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:56.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.606+0000 7fac95c00640 1 -- 172.21.15.39:0/2395957685 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563df4b8d200 msgr2=0x563df4b9df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:17:56.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.607+0000 7fac95c00640 1 --2- 172.21.15.39:0/2395957685 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563df4b8d200 0x563df4b9df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:56.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.607+0000 7fac95c00640 1 -- 172.21.15.39:0/2395957685 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563df4b87bb0 con 0x563df4b8c810 2024-03-18T16:17:56.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.607+0000 7fac95200640 1 --2- 172.21.15.39:0/2395957685 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563df4b8d200 0x563df4b9df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:17:56.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.607+0000 7fac95c00640 1 --2- 172.21.15.39:0/2395957685 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563df4b8c810 0x563df4b86630 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fac840ee7c0 tx=0x7fac84005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b895a0b16c7e1256 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:17:56.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.608+0000 7fac82000640 1 -- 172.21.15.39:0/2395957685 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac840b9990 con 0x563df4b8c810 2024-03-18T16:17:56.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.608+0000 7fac82000640 1 -- 172.21.15.39:0/2395957685 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fac840fe040 con 0x563df4b8c810 2024-03-18T16:17:56.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.608+0000 7fac82000640 1 -- 172.21.15.39:0/2395957685 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac840f4c70 con 0x563df4b8c810 2024-03-18T16:17:56.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.610+0000 7fac9b20a740 1 -- 172.21.15.39:0/2395957685 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563df4b8c810 msgr2=0x563df4b86630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:56.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.610+0000 7fac9b20a740 1 --2- 172.21.15.39:0/2395957685 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563df4b8c810 0x563df4b86630 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fac840ee7c0 tx=0x7fac84005fd0 comp rx=0 tx=0).stop 2024-03-18T16:17:56.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.610+0000 7fac9b20a740 1 -- 172.21.15.39:0/2395957685 shutdown_connections 2024-03-18T16:17:56.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.610+0000 7fac9b20a740 1 --2- 172.21.15.39:0/2395957685 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563df4b92370 0x563df4b86b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:56.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.610+0000 7fac9b20a740 1 --2- 172.21.15.39:0/2395957685 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563df4b8d200 0x563df4b9df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:56.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.610+0000 7fac9b20a740 1 --2- 172.21.15.39:0/2395957685 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563df4b8c810 0x563df4b86630 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:56.613 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.610+0000 7fac9b20a740 1 -- 172.21.15.39:0/2395957685 >> 172.21.15.39:0/2395957685 conn(0x563df4b87d80 msgr2=0x563df4b8e700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:17:56.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.611+0000 7fac9b20a740 1 -- 172.21.15.39:0/2395957685 shutdown_connections 2024-03-18T16:17:56.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.611+0000 7fac9b20a740 1 -- 172.21.15.39:0/2395957685 wait complete. 2024-03-18T16:17:56.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.611+0000 7fac9b20a740 1 -- start start 2024-03-18T16:17:56.614 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.612+0000 7fac9b20a740 1 Processor -- start 2024-03-18T16:17:56.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.613+0000 7fac9b20a740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563df4b8c810 0x563df4b99860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:56.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.613+0000 7fac9b20a740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563df4b8d200 0x563df4b99da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:56.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.613+0000 7fac9b20a740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563df4b92370 0x563df4b9cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:56.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.613+0000 7fac95200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563df4b8d200 0x563df4b99da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:56.616 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.614+0000 7fac95200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563df4b8d200 0x563df4b99da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:45526/0 (socket says 172.21.15.39:45526) 2024-03-18T16:17:56.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.614+0000 7fac95c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563df4b8c810 0x563df4b99860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:56.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.614+0000 7fac95c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563df4b8c810 0x563df4b99860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51974/0 (socket says 172.21.15.39:51974) 2024-03-18T16:17:56.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.614+0000 7fac95200640 1 -- 172.21.15.39:0/2065055540 learned_addr learned my addr 172.21.15.39:0/2065055540 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:17:56.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.614+0000 7fac96600640 1 --2- 172.21.15.39:0/2065055540 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563df4b92370 0x563df4b9cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:56.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.615+0000 7fac9b20a740 1 -- 172.21.15.39:0/2065055540 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fac901ba830 con 0x563df4b8c810 2024-03-18T16:17:56.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.615+0000 7fac9b20a740 1 -- 172.21.15.39:0/2065055540 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fac901ba160 con 0x563df4b92370 2024-03-18T16:17:56.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.615+0000 7fac9b20a740 1 -- 172.21.15.39:0/2065055540 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fac901b73a0 con 0x563df4b8d200 2024-03-18T16:17:56.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.615+0000 7fac9b20a740 10 client.0 _ll_register_callbacks cb 0x7fac88016bb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:17:56.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.616+0000 7fac96600640 1 -- 172.21.15.39:0/2065055540 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563df4b8d200 msgr2=0x563df4b99da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:17:56.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.616+0000 7fac96600640 1 --2- 172.21.15.39:0/2065055540 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563df4b8d200 0x563df4b99da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:56.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.616+0000 7fac96600640 1 -- 172.21.15.39:0/2065055540 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563df4b8c810 msgr2=0x563df4b99860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:17:56.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.616+0000 7fac96600640 1 --2- 172.21.15.39:0/2065055540 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563df4b8c810 0x563df4b99860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:56.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.616+0000 7fac96600640 1 -- 172.21.15.39:0/2065055540 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563df4b9d340 con 0x563df4b92370 2024-03-18T16:17:56.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.618+0000 7fac96600640 1 --2- 172.21.15.39:0/2065055540 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563df4b92370 0x563df4b9cd50 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fac88015780 tx=0x7fac88015c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b3bd639d5ebb2996 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:17:56.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.618+0000 7fac77400640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:17:56.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.618+0000 7fac77400640 1 -- 172.21.15.39:0/2065055540 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac88025050 con 0x563df4b92370 2024-03-18T16:17:56.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.619+0000 7fac77400640 1 -- 172.21.15.39:0/2065055540 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fac88033430 con 0x563df4b92370 2024-03-18T16:17:56.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.619+0000 7fac9b20a740 1 -- 172.21.15.39:0/2065055540 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fac840fe040 con 0x563df4b92370 2024-03-18T16:17:56.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.619+0000 7fac77400640 1 -- 172.21.15.39:0/2065055540 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fac88042050 con 0x563df4b92370 2024-03-18T16:17:56.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.619+0000 7fac9b20a740 10 client.27194 fetch_fsmap learned FSMap version 44 2024-03-18T16:17:56.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.619+0000 7fac9b20a740 1 -- 172.21.15.39:0/2065055540 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x563df4c19dd0 con 0x563df4b92370 2024-03-18T16:17:56.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.620+0000 7fac77400640 1 -- 172.21.15.39:0/2065055540 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fac840fe040 con 0x563df4b92370 2024-03-18T16:17:56.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.622+0000 7fac77400640 1 -- 172.21.15.39:0/2065055540 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7fac88020040 con 0x563df4b92370 2024-03-18T16:17:56.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.622+0000 7fac9b20a740 10 client.27194 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:17:56.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.622+0000 7fac9b20a740 10 client.27194 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:17:56.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.622+0000 7fac9b20a740 20 client.27194 populate_metadata read hostname 'smithi039' 2024-03-18T16:17:56.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.622+0000 7fac9b20a740 1 --2- 172.21.15.39:0/2065055540 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x563df4c1a770 0x563df4c1cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:17:56.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.622+0000 7fac9b20a740 4 client.27194 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T16:17:56.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.622+0000 7fac9b20a740 1 -- 172.21.15.39:0/2065055540 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563df4afe730 con 0x563df4c1a770 2024-03-18T16:17:56.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.622+0000 7fac95c00640 1 --2- 172.21.15.39:0/2065055540 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x563df4c1a770 0x563df4c1cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:17:56.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.623+0000 7fac77400640 10 client.27194 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:17:56.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.623+0000 7fac95c00640 1 --2- 172.21.15.39:0/2065055540 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x563df4c1a770 0x563df4c1cb90 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dfc627a3159395ca server_cookie=abb38a4381a93f53 in_seq=0 out_seq=0 2024-03-18T16:17:56.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.624+0000 7fac77400640 1 -- 172.21.15.39:0/2065055540 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x563df4afe730 con 0x563df4c1a770 2024-03-18T16:17:56.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.624+0000 7fac77400640 10 client.27194 handle_command_reply: tid=0 2024-03-18T16:17:56.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.624+0000 7fac9b20a740 1 client.27194 shutdown 2024-03-18T16:17:56.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.624+0000 7fac9b20a740 1 -- 172.21.15.39:0/2065055540 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563df4b92370 msgr2=0x563df4b9cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:56.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.624+0000 7fac9b20a740 1 --2- 172.21.15.39:0/2065055540 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563df4b92370 0x563df4b9cd50 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fac88015780 tx=0x7fac88015c20 comp rx=0 tx=0).stop 2024-03-18T16:17:56.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.624+0000 7fac9b20a740 1 -- 172.21.15.39:0/2065055540 shutdown_connections 2024-03-18T16:17:56.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.625+0000 7fac9b20a740 1 --2- 172.21.15.39:0/2065055540 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x563df4c1a770 0x563df4c1cb90 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:56.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.625+0000 7fac77400640 0 client.27194 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:17:56.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.625+0000 7fac9b20a740 1 --2- 172.21.15.39:0/2065055540 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563df4b92370 0x563df4b9cd50 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:56.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.625+0000 7fac9b20a740 1 --2- 172.21.15.39:0/2065055540 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x563df4b8d200 0x563df4b99da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:56.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.625+0000 7fac9b20a740 1 --2- 172.21.15.39:0/2065055540 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x563df4b8c810 0x563df4b99860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:56.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.625+0000 7fac9b20a740 1 -- 172.21.15.39:0/2065055540 >> 172.21.15.39:0/2065055540 conn(0x563df4b87d80 msgr2=0x563df4b86630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:17:56.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.625+0000 7fac9b20a740 1 -- 172.21.15.39:0/2065055540 shutdown_connections 2024-03-18T16:17:56.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.626+0000 7fac9b20a740 1 -- 172.21.15.39:0/2065055540 wait complete. 2024-03-18T16:17:56.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.626+0000 7fac9b20a740 20 client.27194 trim_cache size 0 max 16384 2024-03-18T16:17:56.629 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-03-18T16:17:56.629 INFO:teuthology.orchestra.run.smithi039.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-18T16:17:56.629 INFO:teuthology.orchestra.run.smithi039.stdout: "scrubs": { 2024-03-18T16:17:56.629 INFO:teuthology.orchestra.run.smithi039.stdout: "8ff8eef0-aa92-4da8-a1db-f45b11e4ab8a": { 2024-03-18T16:17:56.629 INFO:teuthology.orchestra.run.smithi039.stdout: "path": "/", 2024-03-18T16:17:56.630 INFO:teuthology.orchestra.run.smithi039.stdout: "tag": "8ff8eef0-aa92-4da8-a1db-f45b11e4ab8a", 2024-03-18T16:17:56.630 INFO:teuthology.orchestra.run.smithi039.stdout: "options": "recursive,force" 2024-03-18T16:17:56.630 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:17:56.630 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:17:56.630 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-03-18T16:17:56.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.628+0000 7fac98600640 1 -- 172.21.15.39:0/424505277 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fac780a2880 msgr2=0x7fac780a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:56.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.628+0000 7fac98600640 1 --2- 172.21.15.39:0/424505277 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fac780a2880 0x7fac780a4d40 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x563df4b8da60 tx=0x7fac8800b040 comp rx=0 tx=0).stop 2024-03-18T16:17:56.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.628+0000 7fac98600640 1 -- 172.21.15.39:0/424505277 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fac901087e0 msgr2=0x7fac901a6a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:17:56.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.628+0000 7fac98600640 1 --2- 172.21.15.39:0/424505277 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fac901087e0 0x7fac901a6a50 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fac8402c1a0 tx=0x7fac840040c0 comp rx=0 tx=0).stop 2024-03-18T16:17:56.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.628+0000 7fac98600640 1 -- 172.21.15.39:0/424505277 shutdown_connections 2024-03-18T16:17:56.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.628+0000 7fac98600640 1 --2- 172.21.15.39:0/424505277 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fac780a2880 0x7fac780a4d40 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:56.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.628+0000 7fac98600640 1 --2- 172.21.15.39:0/424505277 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fac90110060 0x7fac901ae000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:56.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.628+0000 7fac98600640 1 --2- 172.21.15.39:0/424505277 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fac90109140 0x7fac901a6f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:56.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.628+0000 7fac98600640 1 --2- 172.21.15.39:0/424505277 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fac901087e0 0x7fac901a6a50 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:17:56.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.628+0000 7fac98600640 1 -- 172.21.15.39:0/424505277 >> 172.21.15.39:0/424505277 conn(0x7fac9006e640 msgr2=0x7fac90070290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:17:56.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.629+0000 7fac98600640 1 -- 172.21.15.39:0/424505277 shutdown_connections 2024-03-18T16:17:56.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:17:56.629+0000 7fac98600640 1 -- 172.21.15.39:0/424505277 wait complete. 2024-03-18T16:17:56.669 INFO:tasks.cephfs.filesystem:scrub status for tag:8ff8eef0-aa92-4da8-a1db-f45b11e4ab8a - {'path': '/', 'tag': '8ff8eef0-aa92-4da8-a1db-f45b11e4ab8a', 'options': 'recursive,force'} 2024-03-18T16:17:57.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:17:56 smithi039 ceph-mon[31721]: pgmap v658: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 25 MiB/s wr, 5.39k op/s 2024-03-18T16:17:57.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:17:56 smithi196 ceph-mon[36307]: pgmap v658: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 25 MiB/s wr, 5.39k op/s 2024-03-18T16:17:57.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:17:56 smithi194 ceph-mon[36200]: pgmap v658: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 25 MiB/s wr, 5.39k op/s 2024-03-18T16:17:58.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:17:57 smithi196 ceph-mon[36307]: scrub summary: idle 2024-03-18T16:17:58.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:17:57 smithi196 ceph-mon[36307]: scrub summary: idle 2024-03-18T16:17:58.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:17:57 smithi196 ceph-mon[36307]: scrub summary: idle 2024-03-18T16:17:58.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:17:57 smithi194 ceph-mon[36200]: scrub summary: idle 2024-03-18T16:17:58.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:17:57 smithi194 ceph-mon[36200]: scrub summary: idle 2024-03-18T16:17:58.268 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:17:57 smithi194 ceph-mon[36200]: scrub summary: idle 2024-03-18T16:17:58.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:17:57 smithi039 ceph-mon[31721]: scrub summary: idle 2024-03-18T16:17:58.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:17:57 smithi039 ceph-mon[31721]: scrub summary: idle 2024-03-18T16:17:58.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:17:57 smithi039 ceph-mon[31721]: scrub summary: idle 2024-03-18T16:17:59.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:17:58 smithi196 ceph-mon[36307]: scrub summary: idle 2024-03-18T16:17:59.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:17:58 smithi196 ceph-mon[36307]: scrub summary: idle 2024-03-18T16:17:59.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:17:58 smithi196 ceph-mon[36307]: pgmap v659: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 24 MiB/s wr, 5.26k op/s 2024-03-18T16:17:59.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:17:58 smithi194 ceph-mon[36200]: scrub summary: idle 2024-03-18T16:17:59.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:17:58 smithi194 ceph-mon[36200]: scrub summary: idle 2024-03-18T16:17:59.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:17:58 smithi194 ceph-mon[36200]: pgmap v659: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 24 MiB/s wr, 5.26k op/s 2024-03-18T16:17:59.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:17:58 smithi039 ceph-mon[31721]: scrub summary: idle 2024-03-18T16:17:59.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:17:58 smithi039 ceph-mon[31721]: scrub summary: idle 2024-03-18T16:17:59.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:17:58 smithi039 ceph-mon[31721]: pgmap v659: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 24 MiB/s wr, 5.26k op/s 2024-03-18T16:18:01.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:18:01 smithi039 ceph-mon[31721]: pgmap v660: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 20 MiB/s wr, 4.43k op/s 2024-03-18T16:18:01.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:18:01 smithi196 ceph-mon[36307]: pgmap v660: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 20 MiB/s wr, 4.43k op/s 2024-03-18T16:18:01.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:18:01 smithi194 ceph-mon[36200]: pgmap v660: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 20 MiB/s wr, 4.43k op/s 2024-03-18T16:18:03.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:18:03 smithi039 ceph-mon[31721]: pgmap v661: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 23 MiB/s wr, 5.13k op/s 2024-03-18T16:18:03.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:18:03 smithi196 ceph-mon[36307]: pgmap v661: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 23 MiB/s wr, 5.13k op/s 2024-03-18T16:18:03.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:18:03 smithi194 ceph-mon[36200]: pgmap v661: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 23 MiB/s wr, 5.13k op/s 2024-03-18T16:18:05.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:18:05 smithi039 ceph-mon[31721]: pgmap v662: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 19 MiB/s wr, 4.22k op/s 2024-03-18T16:18:05.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:18:05 smithi196 ceph-mon[36307]: pgmap v662: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 19 MiB/s wr, 4.22k op/s 2024-03-18T16:18:05.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:18:05 smithi194 ceph-mon[36200]: pgmap v662: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 19 MiB/s wr, 4.22k op/s 2024-03-18T16:18:07.339 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:18:07 smithi039 ceph-mon[31721]: pgmap v663: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 20 MiB/s wr, 4.41k op/s 2024-03-18T16:18:07.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:18:07 smithi196 ceph-mon[36307]: pgmap v663: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 20 MiB/s wr, 4.41k op/s 2024-03-18T16:18:07.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:18:07 smithi194 ceph-mon[36200]: pgmap v663: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 20 MiB/s wr, 4.41k op/s 2024-03-18T16:18:09.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:18:09 smithi039 ceph-mon[31721]: pgmap v664: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 22 MiB/s wr, 4.94k op/s 2024-03-18T16:18:09.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:18:09 smithi196 ceph-mon[36307]: pgmap v664: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 22 MiB/s wr, 4.94k op/s 2024-03-18T16:18:09.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:18:09 smithi194 ceph-mon[36200]: pgmap v664: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 22 MiB/s wr, 4.94k op/s 2024-03-18T16:18:10.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:18:10 smithi196 ceph-mon[36307]: pgmap v665: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 18 MiB/s wr, 4.04k op/s 2024-03-18T16:18:10.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:18:10 smithi194 ceph-mon[36200]: pgmap v665: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 18 MiB/s wr, 4.04k op/s 2024-03-18T16:18:10.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:18:10 smithi039 ceph-mon[31721]: pgmap v665: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 KiB/s rd, 18 MiB/s wr, 4.04k op/s 2024-03-18T16:18:11.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:18:11 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:18:11.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:18:11 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:18:11.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:18:11 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:18:12.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:18:12 smithi196 ceph-mon[36307]: pgmap v666: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 18 MiB/s wr, 4.04k op/s 2024-03-18T16:18:12.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:18:12 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:18:12.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:18:12 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:18:12.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:18:12 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:18:12.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:18:12 smithi194 ceph-mon[36200]: pgmap v666: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 18 MiB/s wr, 4.04k op/s 2024-03-18T16:18:12.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:18:12 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:18:12.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:18:12 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:18:12.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:18:12 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:18:12.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:18:12 smithi039 ceph-mon[31721]: pgmap v666: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 18 MiB/s wr, 4.04k op/s 2024-03-18T16:18:12.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:18:12 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:18:12.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:18:12 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:18:12.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:18:12 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:18:15.191 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:18:14 smithi194 ceph-mon[36200]: pgmap v667: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 14 MiB/s wr, 3.16k op/s 2024-03-18T16:18:15.242 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:18:14 smithi196 ceph-mon[36307]: pgmap v667: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 14 MiB/s wr, 3.16k op/s 2024-03-18T16:18:15.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:18:14 smithi039 ceph-mon[31721]: pgmap v667: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 14 MiB/s wr, 3.16k op/s 2024-03-18T16:18:17.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:18:16 smithi194 ceph-mon[36200]: pgmap v668: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 10 MiB/s wr, 2.41k op/s 2024-03-18T16:18:17.255 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:18:16 smithi196 ceph-mon[36307]: pgmap v668: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 10 MiB/s wr, 2.41k op/s 2024-03-18T16:18:17.339 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:18:16 smithi039 ceph-mon[31721]: pgmap v668: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 10 MiB/s wr, 2.41k op/s 2024-03-18T16:18:19.232 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:18:18 smithi196 ceph-mon[36307]: pgmap v669: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.6 MiB/s wr, 2.21k op/s 2024-03-18T16:18:19.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:18:18 smithi194 ceph-mon[36200]: pgmap v669: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.6 MiB/s wr, 2.21k op/s 2024-03-18T16:18:19.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:18:18 smithi039 ceph-mon[31721]: pgmap v669: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.6 MiB/s wr, 2.21k op/s 2024-03-18T16:18:21.237 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:18:20 smithi196 ceph-mon[36307]: pgmap v670: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 3.7 MiB/s wr, 859 op/s 2024-03-18T16:18:21.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:18:20 smithi194 ceph-mon[36200]: pgmap v670: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 3.7 MiB/s wr, 859 op/s 2024-03-18T16:18:21.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:18:20 smithi039 ceph-mon[31721]: pgmap v670: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 3.7 MiB/s wr, 859 op/s 2024-03-18T16:18:23.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:18:22 smithi196 ceph-mon[36307]: pgmap v671: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 4.6 MiB/s wr, 1.07k op/s 2024-03-18T16:18:23.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:18:22 smithi194 ceph-mon[36200]: pgmap v671: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 4.6 MiB/s wr, 1.07k op/s 2024-03-18T16:18:23.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:18:22 smithi039 ceph-mon[31721]: pgmap v671: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 4.6 MiB/s wr, 1.07k op/s 2024-03-18T16:18:25.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:18:24 smithi196 ceph-mon[36307]: pgmap v672: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.8 MiB/s wr, 1.34k op/s 2024-03-18T16:18:25.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:18:24 smithi194 ceph-mon[36200]: pgmap v672: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.8 MiB/s wr, 1.34k op/s 2024-03-18T16:18:25.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:18:24 smithi039 ceph-mon[31721]: pgmap v672: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.8 MiB/s wr, 1.34k op/s 2024-03-18T16:18:26.670 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T16:18:26.670 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T16:18:27.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:18:26 smithi039 ceph-mon[31721]: pgmap v673: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 5.6 MiB/s wr, 1.31k op/s 2024-03-18T16:18:27.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:18:26 smithi196 ceph-mon[36307]: pgmap v673: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 5.6 MiB/s wr, 1.31k op/s 2024-03-18T16:18:27.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:18:26 smithi194 ceph-mon[36200]: pgmap v673: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 KiB/s rd, 5.6 MiB/s wr, 1.31k op/s 2024-03-18T16:18:28.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.226+0000 7f56b7c00640 1 -- 172.21.15.39:0/1513646298 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f56b0107e00 msgr2=0x7f56b0108220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:18:28.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.226+0000 7f56b7c00640 1 --2- 172.21.15.39:0/1513646298 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f56b0107e00 0x7f56b0108220 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f56a0009900 tx=0x7f56a002bc40 comp rx=0 tx=0).stop 2024-03-18T16:18:28.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.229+0000 7f56b7c00640 1 -- 172.21.15.39:0/1513646298 shutdown_connections 2024-03-18T16:18:28.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.229+0000 7f56b7c00640 1 --2- 172.21.15.39:0/1513646298 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f56b01099b0 0x7f56b0110240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:28.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.229+0000 7f56b7c00640 1 --2- 172.21.15.39:0/1513646298 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f56b0108ff0 0x7f56b0109470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:28.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.229+0000 7f56b7c00640 1 --2- 172.21.15.39:0/1513646298 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f56b0107e00 0x7f56b0108220 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:28.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.229+0000 7f56b7c00640 1 -- 172.21.15.39:0/1513646298 >> 172.21.15.39:0/1513646298 conn(0x7f56b0076da0 msgr2=0x7f56b00791e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:18:28.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.230+0000 7f56b7c00640 1 -- 172.21.15.39:0/1513646298 shutdown_connections 2024-03-18T16:18:28.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.230+0000 7f56b7c00640 1 -- 172.21.15.39:0/1513646298 wait complete. 2024-03-18T16:18:28.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.232+0000 7f56b7c00640 1 Processor -- start 2024-03-18T16:18:28.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.232+0000 7f56b7c00640 1 -- start start 2024-03-18T16:18:28.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.232+0000 7f56b7c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f56b0107e00 0x7f56b00701a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:28.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.232+0000 7f56b7c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f56b0108ff0 0x7f56b00706e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:28.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.232+0000 7f56b7c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f56b01099b0 0x7f56b01ae1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:28.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.232+0000 7f56b7c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f56b0113aa0 con 0x7f56b01099b0 2024-03-18T16:18:28.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.232+0000 7f56b7c00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f56b0113920 con 0x7f56b0107e00 2024-03-18T16:18:28.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.232+0000 7f56b7c00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f56b0113c20 con 0x7f56b0108ff0 2024-03-18T16:18:28.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.233+0000 7f56b5200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f56b0107e00 0x7f56b00701a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:28.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.233+0000 7f56b5200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f56b0107e00 0x7f56b00701a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:58376/0 (socket says 172.21.15.39:58376) 2024-03-18T16:18:28.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.233+0000 7f56b5200640 1 -- 172.21.15.39:0/2386143758 learned_addr learned my addr 172.21.15.39:0/2386143758 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:18:28.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.233+0000 7f56a7e00640 1 --2- 172.21.15.39:0/2386143758 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f56b0108ff0 0x7f56b00706e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:28.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.233+0000 7f56b5c00640 1 --2- 172.21.15.39:0/2386143758 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f56b01099b0 0x7f56b01ae1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:28.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.233+0000 7f56b5200640 1 -- 172.21.15.39:0/2386143758 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f56b0108ff0 msgr2=0x7f56b00706e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:18:28.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.233+0000 7f56b5200640 1 --2- 172.21.15.39:0/2386143758 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f56b0108ff0 0x7f56b00706e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:28.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.233+0000 7f56b5200640 1 -- 172.21.15.39:0/2386143758 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f56b01099b0 msgr2=0x7f56b01ae1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:18:28.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.233+0000 7f56b5200640 1 --2- 172.21.15.39:0/2386143758 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f56b01099b0 0x7f56b01ae1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:28.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.233+0000 7f56b5200640 1 -- 172.21.15.39:0/2386143758 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56b01ae6f0 con 0x7f56b0107e00 2024-03-18T16:18:28.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.234+0000 7f56b5200640 1 --2- 172.21.15.39:0/2386143758 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f56b0107e00 0x7f56b00701a0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f56a002c250 tx=0x7f56a0004e30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:18:28.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.234+0000 7f56ae000640 1 -- 172.21.15.39:0/2386143758 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56a0041070 con 0x7f56b0107e00 2024-03-18T16:18:28.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.234+0000 7f56ae000640 1 -- 172.21.15.39:0/2386143758 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f56a00054e0 con 0x7f56b0107e00 2024-03-18T16:18:28.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.234+0000 7f56ae000640 1 -- 172.21.15.39:0/2386143758 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56a0034440 con 0x7f56b0107e00 2024-03-18T16:18:28.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.234+0000 7f56b7c00640 1 -- 172.21.15.39:0/2386143758 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56b01ae8c0 con 0x7f56b0107e00 2024-03-18T16:18:28.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.235+0000 7f56b7c00640 1 -- 172.21.15.39:0/2386143758 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56b01aed20 con 0x7f56b0107e00 2024-03-18T16:18:28.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.239+0000 7f56ba835740 1 Processor -- start 2024-03-18T16:18:28.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.240+0000 7f56ba835740 1 -- start start 2024-03-18T16:18:28.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.240+0000 7f56ae000640 1 -- 172.21.15.39:0/2386143758 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f56a0044070 con 0x7f56b0107e00 2024-03-18T16:18:28.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.240+0000 7f56ba835740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e13ba6f810 0x55e13ba6fc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:28.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.240+0000 7f56ba835740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e13ba70200 0x55e13ba74d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:28.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.240+0000 7f56ba835740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e13ba75370 0x55e13ba77800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:28.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.240+0000 7f56ba835740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55e13b9d41f0 con 0x55e13ba6f810 2024-03-18T16:18:28.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.240+0000 7f56ba835740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55e13b97ca80 con 0x55e13ba70200 2024-03-18T16:18:28.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.240+0000 7f56ba835740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55e13b9d0290 con 0x55e13ba75370 2024-03-18T16:18:28.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.240+0000 7f56a7e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e13ba75370 0x55e13ba77800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:28.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.240+0000 7f56b5200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e13ba70200 0x55e13ba74d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:28.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.240+0000 7f56a7e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e13ba75370 0x55e13ba77800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:49020/0 (socket says 172.21.15.39:49020) 2024-03-18T16:18:28.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.240+0000 7f56a7e00640 1 -- 172.21.15.39:0/1140321345 learned_addr learned my addr 172.21.15.39:0/1140321345 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:18:28.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.241+0000 7f56a7e00640 1 -- 172.21.15.39:0/1140321345 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e13ba70200 msgr2=0x55e13ba74d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:18:28.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.241+0000 7f56a7e00640 1 --2- 172.21.15.39:0/1140321345 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e13ba70200 0x55e13ba74d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:28.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.241+0000 7f56a7e00640 1 -- 172.21.15.39:0/1140321345 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e13ba6f810 msgr2=0x55e13ba6fc30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:18:28.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.241+0000 7f56a7e00640 1 --2- 172.21.15.39:0/1140321345 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e13ba6f810 0x55e13ba6fc30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:28.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.241+0000 7f56a7e00640 1 -- 172.21.15.39:0/1140321345 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e13b7b27d0 con 0x55e13ba75370 2024-03-18T16:18:28.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.242+0000 7f56ae000640 1 --2- 172.21.15.39:0/2386143758 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f56940a28d0 0x7f56940a4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:28.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.242+0000 7f56a7e00640 1 --2- 172.21.15.39:0/1140321345 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e13ba75370 0x55e13ba77800 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f56980103d0 tx=0x7f5698010890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=37d5d53a8fed4efb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:18:28.245 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.242+0000 7f56ae000640 1 -- 172.21.15.39:0/2386143758 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f56a00f0990 con 0x7f56b0107e00 2024-03-18T16:18:28.245 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.242+0000 7f56a6000640 1 -- 172.21.15.39:0/1140321345 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5698011700 con 0x55e13ba75370 2024-03-18T16:18:28.245 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.244+0000 7f56a6000640 1 -- 172.21.15.39:0/1140321345 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5698011d80 con 0x55e13ba75370 2024-03-18T16:18:28.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.244+0000 7f56a6000640 1 -- 172.21.15.39:0/1140321345 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f569800d340 con 0x55e13ba75370 2024-03-18T16:18:28.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.244+0000 7f56b5c00640 1 --2- 172.21.15.39:0/2386143758 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f56940a28d0 0x7f56940a4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:28.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.245+0000 7f56ba835740 1 -- 172.21.15.39:0/1140321345 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e13ba75370 msgr2=0x55e13ba77800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:18:28.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.245+0000 7f56ba835740 1 --2- 172.21.15.39:0/1140321345 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e13ba75370 0x55e13ba77800 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f56980103d0 tx=0x7f5698010890 comp rx=0 tx=0).stop 2024-03-18T16:18:28.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.245+0000 7f56b5c00640 1 --2- 172.21.15.39:0/2386143758 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f56940a28d0 0x7f56940a4d90 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f56a80058c0 tx=0x7f56a8005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:18:28.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.246+0000 7f56ba835740 1 -- 172.21.15.39:0/1140321345 shutdown_connections 2024-03-18T16:18:28.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.246+0000 7f56ba835740 1 --2- 172.21.15.39:0/1140321345 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e13ba70200 0x55e13ba74d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:28.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.246+0000 7f56ba835740 1 --2- 172.21.15.39:0/1140321345 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e13ba75370 0x55e13ba77800 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:28.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.246+0000 7f56ba835740 1 --2- 172.21.15.39:0/1140321345 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e13ba6f810 0x55e13ba6fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:28.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.246+0000 7f56ba835740 1 -- 172.21.15.39:0/1140321345 >> 172.21.15.39:0/1140321345 conn(0x55e13ba6ad80 msgr2=0x55e13ba6b180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:18:28.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.246+0000 7f56ba835740 1 -- 172.21.15.39:0/1140321345 shutdown_connections 2024-03-18T16:18:28.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.246+0000 7f56ba835740 1 -- 172.21.15.39:0/1140321345 wait complete. 2024-03-18T16:18:28.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.246+0000 7f56ba835740 1 -- start start 2024-03-18T16:18:28.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.247+0000 7f56ba835740 1 Processor -- start 2024-03-18T16:18:28.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.247+0000 7f56ba835740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e13ba6f810 0x55e13ba79d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:28.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.248+0000 7f56ba835740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e13ba70200 0x55e13ba7a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:28.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.248+0000 7f56ba835740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e13ba75370 0x55e13ba812b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:28.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.248+0000 7f56ba835740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55e13ba78320 con 0x55e13ba70200 2024-03-18T16:18:28.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.248+0000 7f56ba835740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55e13b98fed0 con 0x55e13ba6f810 2024-03-18T16:18:28.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.248+0000 7f56ba835740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55e13ba784a0 con 0x55e13ba75370 2024-03-18T16:18:28.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.248+0000 7f56ba835740 10 client.0 _ll_register_callbacks cb 0x55e13b9cb220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:18:28.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.248+0000 7f56b5200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e13ba6f810 0x55e13ba79d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:28.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.248+0000 7f56b5200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e13ba6f810 0x55e13ba79d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:58392/0 (socket says 172.21.15.39:58392) 2024-03-18T16:18:28.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.248+0000 7f56b5200640 1 -- 172.21.15.39:0/2207146495 learned_addr learned my addr 172.21.15.39:0/2207146495 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:18:28.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.248+0000 7f56a7e00640 1 --2- 172.21.15.39:0/2207146495 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e13ba70200 0x55e13ba7a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:28.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.248+0000 7f56b5c00640 1 --2- 172.21.15.39:0/2207146495 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e13ba75370 0x55e13ba812b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:28.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.249+0000 7f56b5200640 1 -- 172.21.15.39:0/2207146495 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e13ba75370 msgr2=0x55e13ba812b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:18:28.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.249+0000 7f56b5200640 1 --2- 172.21.15.39:0/2207146495 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e13ba75370 0x55e13ba812b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:28.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.249+0000 7f56b5200640 1 -- 172.21.15.39:0/2207146495 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e13ba70200 msgr2=0x55e13ba7a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:18:28.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.249+0000 7f56b5200640 1 --2- 172.21.15.39:0/2207146495 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e13ba70200 0x55e13ba7a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:28.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.249+0000 7f56b5200640 1 -- 172.21.15.39:0/2207146495 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e13b6e83f0 con 0x55e13ba6f810 2024-03-18T16:18:28.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.249+0000 7f56b5200640 1 --2- 172.21.15.39:0/2207146495 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e13ba6f810 0x55e13ba79d70 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f56a002f020 tx=0x7f56a002c6e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fde77253546b9d37 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:18:28.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.249+0000 7f569ea00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:18:28.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.249+0000 7f569ea00640 1 -- 172.21.15.39:0/2207146495 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56a004b070 con 0x55e13ba6f810 2024-03-18T16:18:28.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.250+0000 7f569ea00640 1 -- 172.21.15.39:0/2207146495 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f56a00c52b0 con 0x55e13ba6f810 2024-03-18T16:18:28.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.250+0000 7f56ba835740 1 -- 172.21.15.39:0/2207146495 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5698011d80 con 0x55e13ba6f810 2024-03-18T16:18:28.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.250+0000 7f569ea00640 1 -- 172.21.15.39:0/2207146495 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f56a00b9240 con 0x55e13ba6f810 2024-03-18T16:18:28.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.250+0000 7f56ba835740 10 client.27206 fetch_fsmap learned FSMap version 44 2024-03-18T16:18:28.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.250+0000 7f56ba835740 1 -- 172.21.15.39:0/2207146495 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x55e13b6a80f0 con 0x55e13ba6f810 2024-03-18T16:18:28.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.250+0000 7f569ea00640 1 -- 172.21.15.39:0/2207146495 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5698011d80 con 0x55e13ba6f810 2024-03-18T16:18:28.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.251+0000 7f569ea00640 1 -- 172.21.15.39:0/2207146495 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7f56a00bc3f0 con 0x55e13ba6f810 2024-03-18T16:18:28.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.251+0000 7f56ba835740 10 client.27206 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:18:28.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.251+0000 7f56ba835740 10 client.27206 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:18:28.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.251+0000 7f56ba835740 20 client.27206 populate_metadata read hostname 'smithi039' 2024-03-18T16:18:28.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.251+0000 7f56ba835740 1 --2- 172.21.15.39:0/2207146495 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55e13ba85b50 0x55e13ba87ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:28.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.251+0000 7f56ba835740 4 client.27206 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:18:28.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.251+0000 7f56ba835740 1 -- 172.21.15.39:0/2207146495 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e13b9e7be0 con 0x55e13ba85b50 2024-03-18T16:18:28.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.251+0000 7f56a7e00640 1 --2- 172.21.15.39:0/2207146495 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55e13ba85b50 0x55e13ba87ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:28.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.252+0000 7f56a7e00640 1 --2- 172.21.15.39:0/2207146495 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55e13ba85b50 0x55e13ba87ff0 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c442059003f1a56d server_cookie=9a2ee72511d92964 in_seq=0 out_seq=0 2024-03-18T16:18:28.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.252+0000 7f569ea00640 10 client.27206 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:18:28.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.253+0000 7f569ea00640 1 -- 172.21.15.39:0/2207146495 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55e13b9e7be0 con 0x55e13ba85b50 2024-03-18T16:18:28.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.253+0000 7f569ea00640 10 client.27206 handle_command_reply: tid=0 2024-03-18T16:18:28.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.254+0000 7f56ba835740 1 client.27206 shutdown 2024-03-18T16:18:28.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.254+0000 7f56ba835740 1 -- 172.21.15.39:0/2207146495 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e13ba6f810 msgr2=0x55e13ba79d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:18:28.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.254+0000 7f56ba835740 1 --2- 172.21.15.39:0/2207146495 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e13ba6f810 0x55e13ba79d70 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f56a002f020 tx=0x7f56a002c6e0 comp rx=0 tx=0).stop 2024-03-18T16:18:28.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.256+0000 7f56ba835740 1 -- 172.21.15.39:0/2207146495 shutdown_connections 2024-03-18T16:18:28.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.256+0000 7f56ba835740 1 --2- 172.21.15.39:0/2207146495 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55e13ba85b50 0x55e13ba87ff0 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:28.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.256+0000 7f56ba835740 1 --2- 172.21.15.39:0/2207146495 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e13ba75370 0x55e13ba812b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:28.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.256+0000 7f569ea00640 0 client.27206 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:18:28.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.256+0000 7f56ba835740 1 --2- 172.21.15.39:0/2207146495 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e13ba70200 0x55e13ba7a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:28.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.256+0000 7f56ba835740 1 --2- 172.21.15.39:0/2207146495 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e13ba6f810 0x55e13ba79d70 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:28.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.256+0000 7f56ba835740 1 -- 172.21.15.39:0/2207146495 >> 172.21.15.39:0/2207146495 conn(0x55e13ba6ad80 msgr2=0x55e13ba69ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:18:28.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.257+0000 7f56ba835740 1 -- 172.21.15.39:0/2207146495 shutdown_connections 2024-03-18T16:18:28.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.257+0000 7f56ba835740 1 -- 172.21.15.39:0/2207146495 wait complete. 2024-03-18T16:18:28.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.259+0000 7f56ba835740 20 client.27206 trim_cache size 0 max 16384 2024-03-18T16:18:28.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.291+0000 7f56ba835740 1 Processor -- start 2024-03-18T16:18:28.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.291+0000 7f56ba835740 1 -- start start 2024-03-18T16:18:28.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.291+0000 7f56ba835740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e13ba6f810 0x55e13ba71700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:28.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.291+0000 7f56ba835740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e13ba70200 0x55e13ba81270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:28.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.291+0000 7f56ba835740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e13ba75370 0x55e13ba71c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:28.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.291+0000 7f56ba835740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f56b01b9280 con 0x55e13ba75370 2024-03-18T16:18:28.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.291+0000 7f56ba835740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f56b01badc0 con 0x55e13ba6f810 2024-03-18T16:18:28.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.291+0000 7f56ba835740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f56b01b9950 con 0x55e13ba70200 2024-03-18T16:18:28.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.292+0000 7f56a7e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e13ba70200 0x55e13ba81270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:28.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.292+0000 7f56a7e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e13ba70200 0x55e13ba81270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:49040/0 (socket says 172.21.15.39:49040) 2024-03-18T16:18:28.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.292+0000 7f56a7e00640 1 -- 172.21.15.39:0/2570271730 learned_addr learned my addr 172.21.15.39:0/2570271730 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:18:28.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.292+0000 7f56b5200640 1 --2- 172.21.15.39:0/2570271730 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e13ba6f810 0x55e13ba71700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:28.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.293+0000 7f56b5c00640 1 --2- 172.21.15.39:0/2570271730 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e13ba75370 0x55e13ba71c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:28.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.294+0000 7f56b5200640 1 -- 172.21.15.39:0/2570271730 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e13ba70200 msgr2=0x55e13ba81270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:18:28.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.294+0000 7f56b5200640 1 --2- 172.21.15.39:0/2570271730 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e13ba70200 0x55e13ba81270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:28.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.294+0000 7f56b5200640 1 -- 172.21.15.39:0/2570271730 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e13ba75370 msgr2=0x55e13ba71c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:18:28.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.294+0000 7f56b5200640 1 --2- 172.21.15.39:0/2570271730 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e13ba75370 0x55e13ba71c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:28.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.294+0000 7f56b5200640 1 -- 172.21.15.39:0/2570271730 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e13ba785e0 con 0x55e13ba6f810 2024-03-18T16:18:28.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.294+0000 7f56b5200640 1 --2- 172.21.15.39:0/2570271730 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e13ba6f810 0x55e13ba71700 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f56a00b9400 tx=0x7f56a002bc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=49bd4e25afb71993 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:18:28.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.295+0000 7f56a5600640 1 -- 172.21.15.39:0/2570271730 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56a004b070 con 0x55e13ba6f810 2024-03-18T16:18:28.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.295+0000 7f56a5600640 1 -- 172.21.15.39:0/2570271730 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f56a0005690 con 0x55e13ba6f810 2024-03-18T16:18:28.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.295+0000 7f56a5600640 1 -- 172.21.15.39:0/2570271730 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56a00bc900 con 0x55e13ba6f810 2024-03-18T16:18:28.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.296+0000 7f56ba835740 1 -- 172.21.15.39:0/2570271730 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e13ba6f810 msgr2=0x55e13ba71700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:18:28.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.296+0000 7f56ba835740 1 --2- 172.21.15.39:0/2570271730 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e13ba6f810 0x55e13ba71700 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f56a00b9400 tx=0x7f56a002bc40 comp rx=0 tx=0).stop 2024-03-18T16:18:28.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.296+0000 7f56ba835740 1 -- 172.21.15.39:0/2570271730 shutdown_connections 2024-03-18T16:18:28.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.296+0000 7f56ba835740 1 --2- 172.21.15.39:0/2570271730 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e13ba75370 0x55e13ba71c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:28.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.296+0000 7f56ba835740 1 --2- 172.21.15.39:0/2570271730 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e13ba70200 0x55e13ba81270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:28.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.296+0000 7f56ba835740 1 --2- 172.21.15.39:0/2570271730 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e13ba6f810 0x55e13ba71700 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:28.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.296+0000 7f56ba835740 1 -- 172.21.15.39:0/2570271730 >> 172.21.15.39:0/2570271730 conn(0x55e13ba6ad80 msgr2=0x55e13ba78840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:18:28.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.297+0000 7f56ba835740 1 -- 172.21.15.39:0/2570271730 shutdown_connections 2024-03-18T16:18:28.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.297+0000 7f56ba835740 1 -- 172.21.15.39:0/2570271730 wait complete. 2024-03-18T16:18:28.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.297+0000 7f56ba835740 1 -- start start 2024-03-18T16:18:28.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.297+0000 7f56ba835740 1 Processor -- start 2024-03-18T16:18:28.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.298+0000 7f56ba835740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e13ba6f810 0x55e13ba7ec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:28.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.298+0000 7f56ba835740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e13ba70200 0x55e13ba7f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:28.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.298+0000 7f56ba835740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e13ba75370 0x55e13ba801e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:28.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.298+0000 7f56ba835740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f56b01ba6f0 con 0x55e13ba6f810 2024-03-18T16:18:28.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.298+0000 7f56ba835740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f56b01ba020 con 0x55e13ba75370 2024-03-18T16:18:28.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.298+0000 7f56ba835740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f56b01b7260 con 0x55e13ba70200 2024-03-18T16:18:28.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.298+0000 7f56ba835740 10 client.0 _ll_register_callbacks cb 0x7f56a0009850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:18:28.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.299+0000 7f56b5c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e13ba75370 0x55e13ba801e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:28.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.299+0000 7f56a7e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e13ba70200 0x55e13ba7f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:28.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.299+0000 7f56b5c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e13ba75370 0x55e13ba801e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:58416/0 (socket says 172.21.15.39:58416) 2024-03-18T16:18:28.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.299+0000 7f56b5c00640 1 -- 172.21.15.39:0/2403855044 learned_addr learned my addr 172.21.15.39:0/2403855044 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:18:28.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.299+0000 7f56b5c00640 1 -- 172.21.15.39:0/2403855044 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e13ba70200 msgr2=0x55e13ba7f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:18:28.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.299+0000 7f56b5c00640 1 --2- 172.21.15.39:0/2403855044 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e13ba70200 0x55e13ba7f1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:28.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.299+0000 7f56b5c00640 1 -- 172.21.15.39:0/2403855044 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e13ba6f810 msgr2=0x55e13ba7ec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:18:28.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.299+0000 7f56b5c00640 1 --2- 172.21.15.39:0/2403855044 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e13ba6f810 0x55e13ba7ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:28.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.299+0000 7f56b5c00640 1 -- 172.21.15.39:0/2403855044 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e13ba807d0 con 0x55e13ba75370 2024-03-18T16:18:28.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.300+0000 7f56b5c00640 1 --2- 172.21.15.39:0/2403855044 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e13ba75370 0x55e13ba801e0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f56a80117f0 tx=0x7f56a80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a5c27e867103175 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:18:28.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.300+0000 7f569ea00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:18:28.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.300+0000 7f569ea00640 1 -- 172.21.15.39:0/2403855044 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56a8010ad0 con 0x55e13ba75370 2024-03-18T16:18:28.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.300+0000 7f56ba835740 1 -- 172.21.15.39:0/2403855044 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55e13b9e1730 con 0x55e13ba75370 2024-03-18T16:18:28.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.300+0000 7f569ea00640 1 -- 172.21.15.39:0/2403855044 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f56a8010c70 con 0x55e13ba75370 2024-03-18T16:18:28.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.301+0000 7f569ea00640 1 -- 172.21.15.39:0/2403855044 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f56a8005020 con 0x55e13ba75370 2024-03-18T16:18:28.303 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.301+0000 7f56ba835740 10 client.27212 fetch_fsmap learned FSMap version 44 2024-03-18T16:18:28.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.301+0000 7f56ba835740 1 -- 172.21.15.39:0/2403855044 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x55e13baec1a0 con 0x55e13ba75370 2024-03-18T16:18:28.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.301+0000 7f569ea00640 1 -- 172.21.15.39:0/2403855044 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55e13b9e1730 con 0x55e13ba75370 2024-03-18T16:18:28.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.301+0000 7f569ea00640 1 -- 172.21.15.39:0/2403855044 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7f56a800f040 con 0x55e13ba75370 2024-03-18T16:18:28.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.301+0000 7f56ba835740 10 client.27212 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:18:28.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.301+0000 7f56ba835740 10 client.27212 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:18:28.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.301+0000 7f56ba835740 20 client.27212 populate_metadata read hostname 'smithi039' 2024-03-18T16:18:28.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.301+0000 7f56ba835740 1 --2- 172.21.15.39:0/2403855044 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55e13ba85b50 0x55e13baeeb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:28.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.301+0000 7f56ba835740 4 client.27212 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T16:18:28.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.301+0000 7f56ba835740 1 -- 172.21.15.39:0/2403855044 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e13b9dba80 con 0x55e13ba85b50 2024-03-18T16:18:28.304 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.303+0000 7f56b5200640 1 --2- 172.21.15.39:0/2403855044 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55e13ba85b50 0x55e13baeeb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:28.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.304+0000 7f56b5200640 1 --2- 172.21.15.39:0/2403855044 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55e13ba85b50 0x55e13baeeb90 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=97ee144dbf382fef server_cookie=978513f1e5af00eb in_seq=0 out_seq=0 2024-03-18T16:18:28.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.304+0000 7f569ea00640 10 client.27212 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:18:28.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.304+0000 7f569ea00640 1 -- 172.21.15.39:0/2403855044 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55e13b9dba80 con 0x55e13ba85b50 2024-03-18T16:18:28.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.304+0000 7f569ea00640 10 client.27212 handle_command_reply: tid=0 2024-03-18T16:18:28.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.304+0000 7f56ba835740 1 client.27212 shutdown 2024-03-18T16:18:28.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.305+0000 7f56ba835740 1 -- 172.21.15.39:0/2403855044 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e13ba75370 msgr2=0x55e13ba801e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:18:28.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.305+0000 7f56ba835740 1 --2- 172.21.15.39:0/2403855044 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e13ba75370 0x55e13ba801e0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f56a80117f0 tx=0x7f56a80046c0 comp rx=0 tx=0).stop 2024-03-18T16:18:28.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.305+0000 7f56ba835740 1 -- 172.21.15.39:0/2403855044 shutdown_connections 2024-03-18T16:18:28.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.305+0000 7f56ba835740 1 --2- 172.21.15.39:0/2403855044 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55e13ba85b50 0x55e13baeeb90 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:28.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.305+0000 7f56ba835740 1 --2- 172.21.15.39:0/2403855044 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e13ba75370 0x55e13ba801e0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:28.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.305+0000 7f56ba835740 1 --2- 172.21.15.39:0/2403855044 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55e13ba70200 0x55e13ba7f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:28.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.305+0000 7f56ba835740 1 --2- 172.21.15.39:0/2403855044 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55e13ba6f810 0x55e13ba7ec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:28.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.305+0000 7f56ba835740 1 -- 172.21.15.39:0/2403855044 >> 172.21.15.39:0/2403855044 conn(0x55e13ba6ad80 msgr2=0x55e13ba78840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:18:28.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.305+0000 7f569ea00640 0 client.27212 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:18:28.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.306+0000 7f56ba835740 1 -- 172.21.15.39:0/2403855044 shutdown_connections 2024-03-18T16:18:28.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.306+0000 7f56ba835740 1 -- 172.21.15.39:0/2403855044 wait complete. 2024-03-18T16:18:28.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.306+0000 7f56ba835740 20 client.27212 trim_cache size 0 max 16384 2024-03-18T16:18:28.308 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-03-18T16:18:28.308 INFO:teuthology.orchestra.run.smithi039.stdout: "status": "no active scrubs running", 2024-03-18T16:18:28.308 INFO:teuthology.orchestra.run.smithi039.stdout: "scrubs": {} 2024-03-18T16:18:28.309 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-03-18T16:18:28.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.307+0000 7f56b7c00640 1 -- 172.21.15.39:0/2386143758 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f56940a28d0 msgr2=0x7f56940a4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:18:28.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.307+0000 7f56b7c00640 1 --2- 172.21.15.39:0/2386143758 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f56940a28d0 0x7f56940a4d90 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f56a80058c0 tx=0x7f56a8005850 comp rx=0 tx=0).stop 2024-03-18T16:18:28.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.307+0000 7f56b7c00640 1 -- 172.21.15.39:0/2386143758 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f56b0107e00 msgr2=0x7f56b00701a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:18:28.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.307+0000 7f56b7c00640 1 --2- 172.21.15.39:0/2386143758 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f56b0107e00 0x7f56b00701a0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f56a002c250 tx=0x7f56a0004e30 comp rx=0 tx=0).stop 2024-03-18T16:18:28.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.307+0000 7f56b7c00640 1 -- 172.21.15.39:0/2386143758 shutdown_connections 2024-03-18T16:18:28.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.307+0000 7f56b7c00640 1 --2- 172.21.15.39:0/2386143758 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f56940a28d0 0x7f56940a4d90 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:28.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.307+0000 7f56b7c00640 1 --2- 172.21.15.39:0/2386143758 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f56b01099b0 0x7f56b01ae1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:28.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.307+0000 7f56b7c00640 1 --2- 172.21.15.39:0/2386143758 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f56b0108ff0 0x7f56b00706e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:28.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.307+0000 7f56b7c00640 1 --2- 172.21.15.39:0/2386143758 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f56b0107e00 0x7f56b00701a0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:28.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.307+0000 7f56b7c00640 1 -- 172.21.15.39:0/2386143758 >> 172.21.15.39:0/2386143758 conn(0x7f56b0076da0 msgr2=0x7f56b010c370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:18:28.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.307+0000 7f56b7c00640 1 -- 172.21.15.39:0/2386143758 shutdown_connections 2024-03-18T16:18:28.312 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.310+0000 7f56b7c00640 1 -- 172.21.15.39:0/2386143758 wait complete. 2024-03-18T16:18:28.336 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-18T16:18:28.336 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:18:28.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.604+0000 7fd31e200640 1 -- 172.21.15.39:0/2606682930 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd3181096b0 msgr2=0x7fd31810bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:18:28.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.604+0000 7fd31e200640 1 --2- 172.21.15.39:0/2606682930 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd3181096b0 0x7fd31810bb40 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fd310009ca0 tx=0x7fd31002c100 comp rx=0 tx=0).stop 2024-03-18T16:18:28.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.605+0000 7fd31e200640 1 -- 172.21.15.39:0/2606682930 shutdown_connections 2024-03-18T16:18:28.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.605+0000 7fd31e200640 1 --2- 172.21.15.39:0/2606682930 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd3181096b0 0x7fd31810bb40 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:28.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.605+0000 7fd31e200640 1 --2- 172.21.15.39:0/2606682930 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd318074240 0x7fd3180746c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:28.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.605+0000 7fd31e200640 1 --2- 172.21.15.39:0/2606682930 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd318075820 0x7fd318073c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:28.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.605+0000 7fd31e200640 1 -- 172.21.15.39:0/2606682930 >> 172.21.15.39:0/2606682930 conn(0x7fd3180fcd80 msgr2=0x7fd3180ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:18:28.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.605+0000 7fd31e200640 1 -- 172.21.15.39:0/2606682930 shutdown_connections 2024-03-18T16:18:28.607 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.605+0000 7fd31e200640 1 -- 172.21.15.39:0/2606682930 wait complete. 2024-03-18T16:18:28.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.607+0000 7fd31e200640 1 Processor -- start 2024-03-18T16:18:28.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.607+0000 7fd31e200640 1 -- start start 2024-03-18T16:18:28.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.607+0000 7fd31e200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd318074240 0x7fd318070170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:28.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.607+0000 7fd31e200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd318075820 0x7fd3180706b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:28.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.607+0000 7fd31e200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd3181096b0 0x7fd3181ae530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:28.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.608+0000 7fd31e200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fd31810f630 con 0x7fd318075820 2024-03-18T16:18:28.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.608+0000 7fd31e200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fd31810f4b0 con 0x7fd318074240 2024-03-18T16:18:28.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.608+0000 7fd31e200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fd31810f7b0 con 0x7fd3181096b0 2024-03-18T16:18:28.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.608+0000 7fd316a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd318075820 0x7fd3180706b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:28.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.608+0000 7fd317400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd318074240 0x7fd318070170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:28.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.608+0000 7fd317400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd318074240 0x7fd318070170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:58432/0 (socket says 172.21.15.39:58432) 2024-03-18T16:18:28.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.608+0000 7fd317400640 1 -- 172.21.15.39:0/1871069659 learned_addr learned my addr 172.21.15.39:0/1871069659 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:18:28.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.608+0000 7fd317e00640 1 --2- 172.21.15.39:0/1871069659 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd3181096b0 0x7fd3181ae530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:28.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.609+0000 7fd317e00640 1 -- 172.21.15.39:0/1871069659 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd318074240 msgr2=0x7fd318070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:18:28.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.609+0000 7fd317e00640 1 --2- 172.21.15.39:0/1871069659 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd318074240 0x7fd318070170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:28.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.609+0000 7fd317e00640 1 -- 172.21.15.39:0/1871069659 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd318075820 msgr2=0x7fd3180706b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:18:28.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.609+0000 7fd317e00640 1 --2- 172.21.15.39:0/1871069659 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd318075820 0x7fd3180706b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:28.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.609+0000 7fd317e00640 1 -- 172.21.15.39:0/1871069659 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3181051c0 con 0x7fd3181096b0 2024-03-18T16:18:28.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.609+0000 7fd317400640 1 --2- 172.21.15.39:0/1871069659 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd318074240 0x7fd318070170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:18:28.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.609+0000 7fd317e00640 1 --2- 172.21.15.39:0/1871069659 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd3181096b0 0x7fd3181ae530 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fd310037040 tx=0x7fd310035d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:18:28.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.610+0000 7fd30fe00640 1 -- 172.21.15.39:0/1871069659 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd310039040 con 0x7fd3181096b0 2024-03-18T16:18:28.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.610+0000 7fd31e200640 1 -- 172.21.15.39:0/1871069659 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3181aea70 con 0x7fd3181096b0 2024-03-18T16:18:28.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.610+0000 7fd31e200640 1 -- 172.21.15.39:0/1871069659 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3181aee20 con 0x7fd3181096b0 2024-03-18T16:18:28.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.610+0000 7fd30fe00640 1 -- 172.21.15.39:0/1871069659 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd310035e00 con 0x7fd3181096b0 2024-03-18T16:18:28.612 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.610+0000 7fd30fe00640 1 -- 172.21.15.39:0/1871069659 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3100044e0 con 0x7fd3181096b0 2024-03-18T16:18:28.615 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.613+0000 7fd31e200640 1 -- 172.21.15.39:0/1871069659 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2f80052c0 con 0x7fd3181096b0 2024-03-18T16:18:28.617 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.616+0000 7fd30fe00640 1 -- 172.21.15.39:0/1871069659 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fd310004680 con 0x7fd3181096b0 2024-03-18T16:18:28.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.619+0000 7fd30fe00640 1 --2- 172.21.15.39:0/1871069659 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd2fc0a2a00 0x7fd2fc0a4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:28.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.620+0000 7fd317400640 1 --2- 172.21.15.39:0/1871069659 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd2fc0a2a00 0x7fd2fc0a4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:28.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.620+0000 7fd30fe00640 1 -- 172.21.15.39:0/1871069659 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fd3100f0940 con 0x7fd3181096b0 2024-03-18T16:18:28.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.621+0000 7fd317400640 1 --2- 172.21.15.39:0/1871069659 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd2fc0a2a00 0x7fd2fc0a4ec0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fd3180716c0 tx=0x7fd30400b480 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:18:28.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:28.624+0000 7fd30fe00640 1 -- 172.21.15.39:0/1871069659 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd3100098b0 con 0x7fd3181096b0 2024-03-18T16:18:29.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.017+0000 7fd31e200640 1 -- 172.21.15.39:0/1871069659 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd2f80055b0 con 0x7fd3181096b0 2024-03-18T16:18:29.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.018+0000 7fd30fe00640 1 -- 172.21.15.39:0/1871069659 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) ==== 94+0+11932 (secure 0 0 0) 0x7fd3100bd5f0 con 0x7fd3181096b0 2024-03-18T16:18:29.021 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:18:29.022 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":44,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2520226608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2520226608},{"type":"v1","addr":"172.21.15.39:6835","nonce":2520226608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/719341763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":719341763},{"type":"v1","addr":"172.21.15.39:6837","nonce":719341763}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":44,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T16:06:44.180425+0000","modified":"2024-03-18T16:15:58.268360+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24575,"mds_1":24556,"mds_2":24535,"mds_3":24524,"mds_4":14463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14445":{"gid":14445,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.39:6839/656513813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":656513813},{"type":"v1","addr":"172.21.15.39:6839","nonce":656513813}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14463":{"gid":14463,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":13,"addr":"172.21.15.39:6841/3515326042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3515326042},{"type":"v1","addr":"172.21.15.39:6841","nonce":3515326042}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6833/931350738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":931350738},{"type":"v1","addr":"172.21.15.194:6833","nonce":931350738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6833/3874209621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":3874209621},{"type":"v1","addr":"172.21.15.196:6833","nonce":3874209621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6835/188385470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":188385470},{"type":"v1","addr":"172.21.15.194:6835","nonce":188385470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6835/3949984309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":3949984309},{"type":"v1","addr":"172.21.15.196:6835","nonce":3949984309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":15,"addr":"172.21.15.194:6837/1301464246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":1301464246},{"type":"v1","addr":"172.21.15.194:6837","nonce":1301464246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":16,"addr":"172.21.15.196:6837/895247251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":895247251},{"type":"v1","addr":"172.21.15.196:6837","nonce":895247251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.196:6839/2001427268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6838","nonce":2001427268},{"type":"v1","addr":"172.21.15.196:6839","nonce":2001427268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24575":{"gid":24575,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.194:6839/274911158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":274911158},{"type":"v1","addr":"172.21.15.194:6839","nonce":274911158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24575,"qdb_cluster":[14463,24524,24575,24556,24535]},"id":1}]} 2024-03-18T16:18:29.022 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 44 2024-03-18T16:18:29.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.022+0000 7fd31e200640 1 -- 172.21.15.39:0/1871069659 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd2fc0a2a00 msgr2=0x7fd2fc0a4ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:18:29.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.022+0000 7fd31e200640 1 --2- 172.21.15.39:0/1871069659 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd2fc0a2a00 0x7fd2fc0a4ec0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fd3180716c0 tx=0x7fd30400b480 comp rx=0 tx=0).stop 2024-03-18T16:18:29.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.022+0000 7fd31e200640 1 -- 172.21.15.39:0/1871069659 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd3181096b0 msgr2=0x7fd3181ae530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:18:29.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.022+0000 7fd31e200640 1 --2- 172.21.15.39:0/1871069659 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd3181096b0 0x7fd3181ae530 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fd310037040 tx=0x7fd310035d10 comp rx=0 tx=0).stop 2024-03-18T16:18:29.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.023+0000 7fd31e200640 1 -- 172.21.15.39:0/1871069659 shutdown_connections 2024-03-18T16:18:29.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.023+0000 7fd31e200640 1 --2- 172.21.15.39:0/1871069659 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd2fc0a2a00 0x7fd2fc0a4ec0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:29.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.023+0000 7fd31e200640 1 --2- 172.21.15.39:0/1871069659 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd3181096b0 0x7fd3181ae530 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:29.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.023+0000 7fd31e200640 1 --2- 172.21.15.39:0/1871069659 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd318075820 0x7fd3180706b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:29.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.023+0000 7fd31e200640 1 --2- 172.21.15.39:0/1871069659 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd318074240 0x7fd318070170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:29.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.023+0000 7fd31e200640 1 -- 172.21.15.39:0/1871069659 >> 172.21.15.39:0/1871069659 conn(0x7fd3180fcd80 msgr2=0x7fd3181055b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:18:29.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.023+0000 7fd31e200640 1 -- 172.21.15.39:0/1871069659 shutdown_connections 2024-03-18T16:18:29.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.023+0000 7fd31e200640 1 -- 172.21.15.39:0/1871069659 wait complete. 2024-03-18T16:18:29.051 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-18T16:18:29.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:18:28 smithi196 ceph-mon[36307]: pgmap v674: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 7.4 MiB/s wr, 1.73k op/s 2024-03-18T16:18:29.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:18:28 smithi194 ceph-mon[36200]: pgmap v674: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 7.4 MiB/s wr, 1.73k op/s 2024-03-18T16:18:29.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.306+0000 7f841a800640 1 -- 172.21.15.39:0/1223281411 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f84140761a0 msgr2=0x7f84140765c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:18:29.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.306+0000 7f841a800640 1 --2- 172.21.15.39:0/1223281411 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f84140761a0 0x7f84140765c0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f8404009900 tx=0x7f840402bc40 comp rx=0 tx=0).stop 2024-03-18T16:18:29.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.308+0000 7f841a800640 1 -- 172.21.15.39:0/1223281411 shutdown_connections 2024-03-18T16:18:29.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.308+0000 7f841a800640 1 --2- 172.21.15.39:0/1223281411 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8414077d80 0x7f8414078620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:29.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.308+0000 7f841a800640 1 --2- 172.21.15.39:0/1223281411 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f84140773e0 0x7f8414077840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:29.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.308+0000 7f841a800640 1 --2- 172.21.15.39:0/1223281411 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f84140761a0 0x7f84140765c0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:29.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.308+0000 7f841a800640 1 -- 172.21.15.39:0/1223281411 >> 172.21.15.39:0/1223281411 conn(0x7f84140fef90 msgr2=0x7f84141013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:18:29.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.308+0000 7f841a800640 1 -- 172.21.15.39:0/1223281411 shutdown_connections 2024-03-18T16:18:29.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.308+0000 7f841a800640 1 -- 172.21.15.39:0/1223281411 wait complete. 2024-03-18T16:18:29.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.310+0000 7f841a800640 1 Processor -- start 2024-03-18T16:18:29.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.310+0000 7f841a800640 1 -- start start 2024-03-18T16:18:29.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.310+0000 7f841a800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f84140761a0 0x7f841419e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:29.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.310+0000 7f841a800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f84140773e0 0x7f841419e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:29.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.311+0000 7f841a800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8414077d80 0x7f8414105440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:29.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.311+0000 7f841a800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f8414113ad0 con 0x7f84140761a0 2024-03-18T16:18:29.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.311+0000 7f841a800640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f8414113950 con 0x7f8414077d80 2024-03-18T16:18:29.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.311+0000 7f841a800640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f8414113c50 con 0x7f84140773e0 2024-03-18T16:18:29.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.311+0000 7f8413e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8414077d80 0x7f8414105440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:29.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.311+0000 7f8413e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8414077d80 0x7f8414105440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:58460/0 (socket says 172.21.15.39:58460) 2024-03-18T16:18:29.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.311+0000 7f8413e00640 1 -- 172.21.15.39:0/2968233261 learned_addr learned my addr 172.21.15.39:0/2968233261 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:18:29.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.311+0000 7f8412a00640 1 --2- 172.21.15.39:0/2968233261 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f84140773e0 0x7f841419e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:29.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.311+0000 7f8413e00640 1 -- 172.21.15.39:0/2968233261 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f84140773e0 msgr2=0x7f841419e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:18:29.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.311+0000 7f8413e00640 1 --2- 172.21.15.39:0/2968233261 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f84140773e0 0x7f841419e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:29.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.311+0000 7f8413e00640 1 -- 172.21.15.39:0/2968233261 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f84140761a0 msgr2=0x7f841419e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:18:29.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.311+0000 7f8413e00640 1 --2- 172.21.15.39:0/2968233261 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f84140761a0 0x7f841419e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:29.314 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.311+0000 7f8413e00640 1 -- 172.21.15.39:0/2968233261 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8414105ab0 con 0x7f8414077d80 2024-03-18T16:18:29.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.312+0000 7f8412a00640 1 --2- 172.21.15.39:0/2968233261 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f84140773e0 0x7f841419e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:18:29.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.312+0000 7f8413e00640 1 --2- 172.21.15.39:0/2968233261 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8414077d80 0x7f8414105440 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f840c00eb10 tx=0x7f840c00c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:18:29.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.312+0000 7f840be00640 1 -- 172.21.15.39:0/2968233261 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f840c004b20 con 0x7f8414077d80 2024-03-18T16:18:29.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.312+0000 7f840be00640 1 -- 172.21.15.39:0/2968233261 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f840c004cc0 con 0x7f8414077d80 2024-03-18T16:18:29.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.312+0000 7f840be00640 1 -- 172.21.15.39:0/2968233261 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f840c010760 con 0x7f8414077d80 2024-03-18T16:18:29.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.312+0000 7f841a800640 1 -- 172.21.15.39:0/2968233261 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8414105da0 con 0x7f8414077d80 2024-03-18T16:18:29.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.313+0000 7f841a800640 1 -- 172.21.15.39:0/2968233261 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8414106240 con 0x7f8414077d80 2024-03-18T16:18:29.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.317+0000 7f841d581740 1 Processor -- start 2024-03-18T16:18:29.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.317+0000 7f840be00640 1 -- 172.21.15.39:0/2968233261 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f840c040ba0 con 0x7f8414077d80 2024-03-18T16:18:29.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.318+0000 7f841d581740 1 -- start start 2024-03-18T16:18:29.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.318+0000 7f841d581740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561dad6698c0 0x561dad669ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:29.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.318+0000 7f841d581740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561dad66a2b0 0x561dad66edb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:29.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.319+0000 7f841d581740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561dad66f420 0x561dad6718b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:29.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.319+0000 7f841d581740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x561dad6313b0 con 0x561dad66a2b0 2024-03-18T16:18:29.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.319+0000 7f841d581740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x561dad5ce450 con 0x561dad6698c0 2024-03-18T16:18:29.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.319+0000 7f841d581740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x561dad576cb0 con 0x561dad66f420 2024-03-18T16:18:29.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.319+0000 7f8412a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561dad6698c0 0x561dad669ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:29.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.319+0000 7f8413400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561dad66a2b0 0x561dad66edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:29.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.319+0000 7f8412a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561dad6698c0 0x561dad669ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:58468/0 (socket says 172.21.15.39:58468) 2024-03-18T16:18:29.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.319+0000 7f8412a00640 1 -- 172.21.15.39:0/140976381 learned_addr learned my addr 172.21.15.39:0/140976381 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:18:29.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.319+0000 7f8412a00640 1 --2- 172.21.15.39:0/140976381 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561dad66f420 0x561dad6718b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:29.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.320+0000 7f8413400640 1 -- 172.21.15.39:0/140976381 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561dad66f420 msgr2=0x561dad6718b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:18:29.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.320+0000 7f8413400640 1 --2- 172.21.15.39:0/140976381 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561dad66f420 0x561dad6718b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:29.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.320+0000 7f8413400640 1 -- 172.21.15.39:0/140976381 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561dad6698c0 msgr2=0x561dad669ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:18:29.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.320+0000 7f8413400640 1 --2- 172.21.15.39:0/140976381 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561dad6698c0 0x561dad669ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:29.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.320+0000 7f8413400640 1 -- 172.21.15.39:0/140976381 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561dad5981f0 con 0x561dad66a2b0 2024-03-18T16:18:29.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.320+0000 7f840be00640 1 --2- 172.21.15.39:0/2968233261 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f83f80a28d0 0x7f83f80a4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:29.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.320+0000 7f840be00640 1 -- 172.21.15.39:0/2968233261 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f840c0cef00 con 0x7f8414077d80 2024-03-18T16:18:29.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.321+0000 7f8413400640 1 --2- 172.21.15.39:0/140976381 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561dad66a2b0 0x561dad66edb0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f8404009850 tx=0x7f8404000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfd7167324acbbec server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:18:29.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.321+0000 7f8412a00640 1 --2- 172.21.15.39:0/2968233261 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f83f80a28d0 0x7f83f80a4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:29.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.321+0000 7f8408c00640 1 -- 172.21.15.39:0/140976381 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8404042020 con 0x561dad66a2b0 2024-03-18T16:18:29.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.321+0000 7f8408c00640 1 -- 172.21.15.39:0/140976381 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f840402cb90 con 0x561dad66a2b0 2024-03-18T16:18:29.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.321+0000 7f8408c00640 1 -- 172.21.15.39:0/140976381 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8404042020 con 0x561dad66a2b0 2024-03-18T16:18:29.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.322+0000 7f841d581740 1 -- 172.21.15.39:0/140976381 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561dad66a2b0 msgr2=0x561dad66edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:18:29.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.322+0000 7f841d581740 1 --2- 172.21.15.39:0/140976381 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561dad66a2b0 0x561dad66edb0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f8404009850 tx=0x7f8404000bc0 comp rx=0 tx=0).stop 2024-03-18T16:18:29.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.322+0000 7f8412a00640 1 --2- 172.21.15.39:0/2968233261 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f83f80a28d0 0x7f83f80a4d90 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x561dad66ab10 tx=0x7f840000c570 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:18:29.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.323+0000 7f841d581740 1 -- 172.21.15.39:0/140976381 shutdown_connections 2024-03-18T16:18:29.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.323+0000 7f841d581740 1 --2- 172.21.15.39:0/140976381 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561dad66f420 0x561dad6718b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:29.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.323+0000 7f841d581740 1 --2- 172.21.15.39:0/140976381 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561dad66a2b0 0x561dad66edb0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:29.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.323+0000 7f841d581740 1 --2- 172.21.15.39:0/140976381 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561dad6698c0 0x561dad669ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:29.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.323+0000 7f841d581740 1 -- 172.21.15.39:0/140976381 >> 172.21.15.39:0/140976381 conn(0x561dad664e30 msgr2=0x561dad665230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:18:29.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.323+0000 7f841d581740 1 -- 172.21.15.39:0/140976381 shutdown_connections 2024-03-18T16:18:29.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.323+0000 7f841d581740 1 -- 172.21.15.39:0/140976381 wait complete. 2024-03-18T16:18:29.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.323+0000 7f841d581740 1 -- start start 2024-03-18T16:18:29.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.324+0000 7f841d581740 1 Processor -- start 2024-03-18T16:18:29.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.325+0000 7f841d581740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561dad6698c0 0x561dad684aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:29.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.325+0000 7f841d581740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561dad66a2b0 0x561dad684fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:29.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.325+0000 7f841d581740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561dad66f420 0x561dad68c020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:29.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.325+0000 7f841d581740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x561dad58a110 con 0x561dad66f420 2024-03-18T16:18:29.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.325+0000 7f841d581740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x561dad5ca4f0 con 0x561dad6698c0 2024-03-18T16:18:29.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.325+0000 7f841d581740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x561dad683380 con 0x561dad66a2b0 2024-03-18T16:18:29.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.325+0000 7f841d581740 10 client.0 _ll_register_callbacks cb 0x561dad577b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:18:29.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.325+0000 7f8413400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561dad6698c0 0x561dad684aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:29.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.326+0000 7f8413400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561dad6698c0 0x561dad684aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:58484/0 (socket says 172.21.15.39:58484) 2024-03-18T16:18:29.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.326+0000 7f8413400640 1 -- 172.21.15.39:0/2404669159 learned_addr learned my addr 172.21.15.39:0/2404669159 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:18:29.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.326+0000 7f8412a00640 1 --2- 172.21.15.39:0/2404669159 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561dad66a2b0 0x561dad684fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:29.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.326+0000 7f8413400640 1 -- 172.21.15.39:0/2404669159 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561dad66a2b0 msgr2=0x561dad684fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:18:29.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.326+0000 7f8413400640 1 --2- 172.21.15.39:0/2404669159 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561dad66a2b0 0x561dad684fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:29.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.326+0000 7f8413400640 1 -- 172.21.15.39:0/2404669159 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561dad66f420 msgr2=0x561dad68c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:18:29.329 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.326+0000 7f8413400640 1 --2- 172.21.15.39:0/2404669159 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561dad66f420 0x561dad68c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:29.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.326+0000 7f8413400640 1 -- 172.21.15.39:0/2404669159 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561dad2e23f0 con 0x561dad6698c0 2024-03-18T16:18:29.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.327+0000 7f8413400640 1 --2- 172.21.15.39:0/2404669159 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561dad6698c0 0x561dad684aa0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f8404036040 tx=0x7f840402c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dd7debf7f629915f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:18:29.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.327+0000 7f83fd600640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:18:29.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.327+0000 7f83fd600640 1 -- 172.21.15.39:0/2404669159 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8404042020 con 0x561dad6698c0 2024-03-18T16:18:29.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.328+0000 7f83fd600640 1 -- 172.21.15.39:0/2404669159 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f8404004040 con 0x561dad6698c0 2024-03-18T16:18:29.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.328+0000 7f841d581740 1 -- 172.21.15.39:0/2404669159 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f840402cb90 con 0x561dad6698c0 2024-03-18T16:18:29.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.328+0000 7f83fd600640 1 -- 172.21.15.39:0/2404669159 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f840402cd40 con 0x561dad6698c0 2024-03-18T16:18:29.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.328+0000 7f841d581740 10 client.27230 fetch_fsmap learned FSMap version 44 2024-03-18T16:18:29.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.328+0000 7f841d581740 1 -- 172.21.15.39:0/2404669159 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x561dad3ac7d0 con 0x561dad6698c0 2024-03-18T16:18:29.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.328+0000 7f83fd600640 1 -- 172.21.15.39:0/2404669159 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f840402cb90 con 0x561dad6698c0 2024-03-18T16:18:29.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.329+0000 7f83fd600640 1 -- 172.21.15.39:0/2404669159 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7f840406ac90 con 0x561dad6698c0 2024-03-18T16:18:29.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.330+0000 7f841d581740 10 client.27230 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:18:29.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.330+0000 7f841d581740 10 client.27230 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-03-18T16:18:29.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.330+0000 7f841d581740 20 client.27230 populate_metadata read hostname 'smithi039' 2024-03-18T16:18:29.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.330+0000 7f841d581740 1 --2- 172.21.15.39:0/2404669159 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x561dad673150 0x561dad6755e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:29.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.330+0000 7f841d581740 4 client.27230 mds_command: new command op to 14463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:18:29.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.330+0000 7f841d581740 1 -- 172.21.15.39:0/2404669159 --> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561dad5db990 con 0x561dad673150 2024-03-18T16:18:29.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.330+0000 7f8412a00640 1 --2- 172.21.15.39:0/2404669159 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x561dad673150 0x561dad6755e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:29.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.332+0000 7f8412a00640 1 --2- 172.21.15.39:0/2404669159 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x561dad673150 0x561dad6755e0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=9430b83a445d93f6 server_cookie=9f68c373304f08f2 in_seq=0 out_seq=0 2024-03-18T16:18:29.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.332+0000 7f83fd600640 10 client.27230 ms_handle_connect on v2:172.21.15.39:6840/3515326042 2024-03-18T16:18:29.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.333+0000 7f83fd600640 1 -- 172.21.15.39:0/2404669159 <== mds.4 v2:172.21.15.39:6840/3515326042 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x561dad5db990 con 0x561dad673150 2024-03-18T16:18:29.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.333+0000 7f83fd600640 10 client.27230 handle_command_reply: tid=0 2024-03-18T16:18:29.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.333+0000 7f841d581740 1 client.27230 shutdown 2024-03-18T16:18:29.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.334+0000 7f841d581740 1 -- 172.21.15.39:0/2404669159 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561dad6698c0 msgr2=0x561dad684aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:18:29.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.334+0000 7f841d581740 1 --2- 172.21.15.39:0/2404669159 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561dad6698c0 0x561dad684aa0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f8404036040 tx=0x7f840402c8b0 comp rx=0 tx=0).stop 2024-03-18T16:18:29.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.334+0000 7f841d581740 1 -- 172.21.15.39:0/2404669159 shutdown_connections 2024-03-18T16:18:29.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.334+0000 7f841d581740 1 --2- 172.21.15.39:0/2404669159 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x561dad673150 0x561dad6755e0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:29.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:18:28 smithi039 ceph-mon[31721]: pgmap v674: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 7.4 MiB/s wr, 1.73k op/s 2024-03-18T16:18:29.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.335+0000 7f83fd600640 0 client.27230 ms_handle_reset on v2:172.21.15.39:6840/3515326042 2024-03-18T16:18:29.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.335+0000 7f841d581740 1 --2- 172.21.15.39:0/2404669159 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561dad66f420 0x561dad68c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:29.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.335+0000 7f841d581740 1 --2- 172.21.15.39:0/2404669159 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561dad66a2b0 0x561dad684fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:29.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.335+0000 7f841d581740 1 --2- 172.21.15.39:0/2404669159 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561dad6698c0 0x561dad684aa0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:29.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.335+0000 7f841d581740 1 -- 172.21.15.39:0/2404669159 >> 172.21.15.39:0/2404669159 conn(0x561dad664e30 msgr2=0x561dad6641f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:18:29.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.335+0000 7f841d581740 1 -- 172.21.15.39:0/2404669159 shutdown_connections 2024-03-18T16:18:29.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.335+0000 7f841d581740 1 -- 172.21.15.39:0/2404669159 wait complete. 2024-03-18T16:18:29.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.335+0000 7f841d581740 20 client.27230 trim_cache size 0 max 16384 2024-03-18T16:18:29.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.357+0000 7f841d581740 1 Processor -- start 2024-03-18T16:18:29.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.357+0000 7f841d581740 1 -- start start 2024-03-18T16:18:29.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.358+0000 7f841d581740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561dad6698c0 0x561dad660080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:29.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.358+0000 7f841d581740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561dad66a2b0 0x561dad665210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:29.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.358+0000 7f841d581740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561dad66f420 0x561dad6840c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:29.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.358+0000 7f841d581740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f841406fe30 con 0x561dad6698c0 2024-03-18T16:18:29.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.358+0000 7f841d581740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f8414071970 con 0x561dad66a2b0 2024-03-18T16:18:29.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.358+0000 7f841d581740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f8414070500 con 0x561dad66f420 2024-03-18T16:18:29.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.358+0000 7f8412a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561dad66a2b0 0x561dad665210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:29.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.358+0000 7f8412a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561dad66a2b0 0x561dad665210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:58498/0 (socket says 172.21.15.39:58498) 2024-03-18T16:18:29.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.358+0000 7f8413e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561dad66f420 0x561dad6840c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:29.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.358+0000 7f8412a00640 1 -- 172.21.15.39:0/2464117107 learned_addr learned my addr 172.21.15.39:0/2464117107 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:18:29.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.359+0000 7f8413400640 1 --2- 172.21.15.39:0/2464117107 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561dad6698c0 0x561dad660080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:29.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.359+0000 7f8412a00640 1 -- 172.21.15.39:0/2464117107 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561dad66f420 msgr2=0x561dad6840c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:18:29.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.359+0000 7f8412a00640 1 --2- 172.21.15.39:0/2464117107 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561dad66f420 0x561dad6840c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:29.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.359+0000 7f8412a00640 1 -- 172.21.15.39:0/2464117107 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561dad6698c0 msgr2=0x561dad660080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:18:29.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.359+0000 7f8412a00640 1 --2- 172.21.15.39:0/2464117107 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561dad6698c0 0x561dad660080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:29.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.359+0000 7f8412a00640 1 -- 172.21.15.39:0/2464117107 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561dad660c10 con 0x561dad66a2b0 2024-03-18T16:18:29.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.359+0000 7f8413e00640 1 --2- 172.21.15.39:0/2464117107 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561dad66f420 0x561dad6840c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:18:29.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.359+0000 7f8413400640 1 --2- 172.21.15.39:0/2464117107 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561dad6698c0 0x561dad660080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:18:29.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.360+0000 7f8412a00640 1 --2- 172.21.15.39:0/2464117107 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561dad66a2b0 0x561dad665210 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f8400014fa0 tx=0x7f84000046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bcdb544b538bcf3c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:18:29.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.361+0000 7f83ffe00640 1 -- 172.21.15.39:0/2464117107 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8400012e10 con 0x561dad66a2b0 2024-03-18T16:18:29.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.361+0000 7f83ffe00640 1 -- 172.21.15.39:0/2464117107 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f840002b2d0 con 0x561dad66a2b0 2024-03-18T16:18:29.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.361+0000 7f83ffe00640 1 -- 172.21.15.39:0/2464117107 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8400003a10 con 0x561dad66a2b0 2024-03-18T16:18:29.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.362+0000 7f841d581740 1 -- 172.21.15.39:0/2464117107 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561dad66a2b0 msgr2=0x561dad665210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:18:29.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.362+0000 7f841d581740 1 --2- 172.21.15.39:0/2464117107 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561dad66a2b0 0x561dad665210 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f8400014fa0 tx=0x7f84000046c0 comp rx=0 tx=0).stop 2024-03-18T16:18:29.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.362+0000 7f841d581740 1 -- 172.21.15.39:0/2464117107 shutdown_connections 2024-03-18T16:18:29.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.362+0000 7f841d581740 1 --2- 172.21.15.39:0/2464117107 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561dad66a2b0 0x561dad665210 secure :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f8400014fa0 tx=0x7f84000046c0 comp rx=0 tx=0).stop 2024-03-18T16:18:29.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.362+0000 7f841d581740 1 --2- 172.21.15.39:0/2464117107 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561dad66f420 0x561dad6840c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:29.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.362+0000 7f841d581740 1 --2- 172.21.15.39:0/2464117107 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561dad6698c0 0x561dad660080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:29.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.362+0000 7f841d581740 1 -- 172.21.15.39:0/2464117107 >> 172.21.15.39:0/2464117107 conn(0x561dad664e30 msgr2=0x561dad6641f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:18:29.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.363+0000 7f841d581740 1 -- 172.21.15.39:0/2464117107 shutdown_connections 2024-03-18T16:18:29.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.363+0000 7f841d581740 1 -- 172.21.15.39:0/2464117107 wait complete. 2024-03-18T16:18:29.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.363+0000 7f841d581740 1 -- start start 2024-03-18T16:18:29.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.363+0000 7f841d581740 1 Processor -- start 2024-03-18T16:18:29.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.364+0000 7f841d581740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561dad6698c0 0x561dad689f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:29.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.364+0000 7f841d581740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561dad66f420 0x561dad68a470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:29.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.364+0000 7f841d581740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561dad673150 0x561dad68a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:29.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.364+0000 7f841d581740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f84140712a0 con 0x561dad6698c0 2024-03-18T16:18:29.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.364+0000 7f841d581740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f8414070bd0 con 0x561dad66f420 2024-03-18T16:18:29.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.364+0000 7f841d581740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f841406de10 con 0x561dad673150 2024-03-18T16:18:29.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.364+0000 7f841d581740 10 client.0 _ll_register_callbacks cb 0x7f840000cb10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:18:29.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.365+0000 7f8413e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561dad673150 0x561dad68a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:29.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.365+0000 7f8413e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561dad673150 0x561dad68a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:49140/0 (socket says 172.21.15.39:49140) 2024-03-18T16:18:29.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.365+0000 7f8413e00640 1 -- 172.21.15.39:0/10683620 learned_addr learned my addr 172.21.15.39:0/10683620 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:18:29.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.365+0000 7f8412a00640 1 --2- 172.21.15.39:0/10683620 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561dad66f420 0x561dad68a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:29.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.365+0000 7f8413400640 1 --2- 172.21.15.39:0/10683620 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561dad6698c0 0x561dad689f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:29.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.366+0000 7f8413e00640 1 -- 172.21.15.39:0/10683620 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561dad66f420 msgr2=0x561dad68a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:18:29.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.366+0000 7f8413e00640 1 --2- 172.21.15.39:0/10683620 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561dad66f420 0x561dad68a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:29.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.366+0000 7f8413e00640 1 -- 172.21.15.39:0/10683620 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561dad6698c0 msgr2=0x561dad689f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:18:29.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.366+0000 7f8413e00640 1 --2- 172.21.15.39:0/10683620 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561dad6698c0 0x561dad689f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:29.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.366+0000 7f8413e00640 1 -- 172.21.15.39:0/10683620 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561dad68b020 con 0x561dad673150 2024-03-18T16:18:29.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.366+0000 7f8413e00640 1 --2- 172.21.15.39:0/10683620 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561dad673150 0x561dad68a9b0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f840c0cc6c0 tx=0x7f840c09ae70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d68d3e3f4bb1e96c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:18:29.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.366+0000 7f83fd600640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:18:29.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.367+0000 7f83fd600640 1 -- 172.21.15.39:0/10683620 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f840c0b4060 con 0x561dad673150 2024-03-18T16:18:29.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.367+0000 7f83fd600640 1 -- 172.21.15.39:0/10683620 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f840c0a3820 con 0x561dad673150 2024-03-18T16:18:29.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.367+0000 7f841d581740 1 -- 172.21.15.39:0/10683620 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f840002b2d0 con 0x561dad673150 2024-03-18T16:18:29.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.367+0000 7f83fd600640 1 -- 172.21.15.39:0/10683620 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f840c0b9050 con 0x561dad673150 2024-03-18T16:18:29.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.367+0000 7f841d581740 10 client.27217 fetch_fsmap learned FSMap version 44 2024-03-18T16:18:29.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.367+0000 7f841d581740 1 -- 172.21.15.39:0/10683620 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x561dad6e56e0 con 0x561dad673150 2024-03-18T16:18:29.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.367+0000 7f83fd600640 1 -- 172.21.15.39:0/10683620 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f840002b2d0 con 0x561dad673150 2024-03-18T16:18:29.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.368+0000 7f83fd600640 1 -- 172.21.15.39:0/10683620 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7f840c09f200 con 0x561dad673150 2024-03-18T16:18:29.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.368+0000 7f841d581740 10 client.27217 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:18:29.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.368+0000 7f841d581740 10 client.27217 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-03-18T16:18:29.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.368+0000 7f841d581740 20 client.27217 populate_metadata read hostname 'smithi039' 2024-03-18T16:18:29.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.368+0000 7f841d581740 1 --2- 172.21.15.39:0/10683620 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x561dad6e5fd0 0x561dad6e83f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:29.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.368+0000 7f841d581740 4 client.27217 mds_command: new command op to 14463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:18:29.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.368+0000 7f841d581740 1 -- 172.21.15.39:0/10683620 --> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561dad5d5ce0 con 0x561dad6e5fd0 2024-03-18T16:18:29.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.368+0000 7f8413400640 1 --2- 172.21.15.39:0/10683620 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x561dad6e5fd0 0x561dad6e83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:29.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.370+0000 7f8413400640 1 --2- 172.21.15.39:0/10683620 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x561dad6e5fd0 0x561dad6e83f0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=5de9f924e0215aa0 server_cookie=31a1e5e93173668f in_seq=0 out_seq=0 2024-03-18T16:18:29.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.370+0000 7f83fd600640 10 client.27217 ms_handle_connect on v2:172.21.15.39:6840/3515326042 2024-03-18T16:18:29.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.370+0000 7f83fd600640 1 -- 172.21.15.39:0/10683620 <== mds.4 v2:172.21.15.39:6840/3515326042 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561dad5d5ce0 con 0x561dad6e5fd0 2024-03-18T16:18:29.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.370+0000 7f83fd600640 10 client.27217 handle_command_reply: tid=0 2024-03-18T16:18:29.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.370+0000 7f841d581740 1 client.27217 shutdown 2024-03-18T16:18:29.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.371+0000 7f841d581740 1 -- 172.21.15.39:0/10683620 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561dad673150 msgr2=0x561dad68a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:18:29.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.371+0000 7f841d581740 1 --2- 172.21.15.39:0/10683620 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561dad673150 0x561dad68a9b0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f840c0cc6c0 tx=0x7f840c09ae70 comp rx=0 tx=0).stop 2024-03-18T16:18:29.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.371+0000 7f841d581740 1 -- 172.21.15.39:0/10683620 shutdown_connections 2024-03-18T16:18:29.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.371+0000 7f841d581740 1 --2- 172.21.15.39:0/10683620 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x561dad6e5fd0 0x561dad6e83f0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:29.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.371+0000 7f841d581740 1 --2- 172.21.15.39:0/10683620 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561dad66f420 0x561dad68a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:29.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.371+0000 7f841d581740 1 --2- 172.21.15.39:0/10683620 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561dad673150 0x561dad68a9b0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:29.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.371+0000 7f841d581740 1 --2- 172.21.15.39:0/10683620 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561dad6698c0 0x561dad689f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:29.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.371+0000 7f841d581740 1 -- 172.21.15.39:0/10683620 >> 172.21.15.39:0/10683620 conn(0x561dad664e30 msgr2=0x561dad6641f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:18:29.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.371+0000 7f83fd600640 0 client.27217 ms_handle_reset on v2:172.21.15.39:6840/3515326042 2024-03-18T16:18:29.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.372+0000 7f841d581740 1 -- 172.21.15.39:0/10683620 shutdown_connections 2024-03-18T16:18:29.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.372+0000 7f841d581740 1 -- 172.21.15.39:0/10683620 wait complete. 2024-03-18T16:18:29.374 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.372+0000 7f841d581740 20 client.27217 trim_cache size 0 max 16384 2024-03-18T16:18:29.374 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:18:29.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.373+0000 7f841a800640 1 -- 172.21.15.39:0/2968233261 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f83f80a28d0 msgr2=0x7f83f80a4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:18:29.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.373+0000 7f841a800640 1 --2- 172.21.15.39:0/2968233261 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f83f80a28d0 0x7f83f80a4d90 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x561dad66ab10 tx=0x7f840000c570 comp rx=0 tx=0).stop 2024-03-18T16:18:29.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.373+0000 7f841a800640 1 -- 172.21.15.39:0/2968233261 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8414077d80 msgr2=0x7f8414105440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:18:29.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.373+0000 7f841a800640 1 --2- 172.21.15.39:0/2968233261 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8414077d80 0x7f8414105440 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f840c00eb10 tx=0x7f840c00c2d0 comp rx=0 tx=0).stop 2024-03-18T16:18:29.375 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.374+0000 7f841a800640 1 -- 172.21.15.39:0/2968233261 shutdown_connections 2024-03-18T16:18:29.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.374+0000 7f841a800640 1 --2- 172.21.15.39:0/2968233261 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8414077d80 0x7f8414105440 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:29.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.374+0000 7f841a800640 1 --2- 172.21.15.39:0/2968233261 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f83f80a28d0 0x7f83f80a4d90 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:29.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.374+0000 7f841a800640 1 --2- 172.21.15.39:0/2968233261 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f84140773e0 0x7f841419e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:29.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.374+0000 7f841a800640 1 --2- 172.21.15.39:0/2968233261 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f84140761a0 0x7f841419e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:29.376 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.374+0000 7f841a800640 1 -- 172.21.15.39:0/2968233261 >> 172.21.15.39:0/2968233261 conn(0x7f84140fef90 msgr2=0x7f8414100a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:18:29.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.375+0000 7f841a800640 1 -- 172.21.15.39:0/2968233261 shutdown_connections 2024-03-18T16:18:29.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.375+0000 7f841a800640 1 -- 172.21.15.39:0/2968233261 wait complete. 2024-03-18T16:18:29.400 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-18T16:18:29.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.679+0000 7f9767a00640 1 -- 172.21.15.39:0/1365183001 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f97600761a0 msgr2=0x7f97600765c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:18:29.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.679+0000 7f9767a00640 1 --2- 172.21.15.39:0/1365183001 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f97600761a0 0x7f97600765c0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f9754009860 tx=0x7f975402bb70 comp rx=0 tx=0).stop 2024-03-18T16:18:29.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.679+0000 7f9767a00640 1 -- 172.21.15.39:0/1365183001 shutdown_connections 2024-03-18T16:18:29.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.679+0000 7f9767a00640 1 --2- 172.21.15.39:0/1365183001 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9760077d80 0x7f9760078620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:29.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.679+0000 7f9767a00640 1 --2- 172.21.15.39:0/1365183001 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f97600773e0 0x7f9760077840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:29.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.679+0000 7f9767a00640 1 --2- 172.21.15.39:0/1365183001 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f97600761a0 0x7f97600765c0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:29.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.679+0000 7f9767a00640 1 -- 172.21.15.39:0/1365183001 >> 172.21.15.39:0/1365183001 conn(0x7f97600fef90 msgr2=0x7f97601013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:18:29.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.680+0000 7f9767a00640 1 -- 172.21.15.39:0/1365183001 shutdown_connections 2024-03-18T16:18:29.682 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.680+0000 7f9767a00640 1 -- 172.21.15.39:0/1365183001 wait complete. 2024-03-18T16:18:29.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.681+0000 7f9767a00640 1 Processor -- start 2024-03-18T16:18:29.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.681+0000 7f9767a00640 1 -- start start 2024-03-18T16:18:29.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.682+0000 7f9767a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f97600761a0 0x7f976019e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:29.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.682+0000 7f9767a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f97600773e0 0x7f976019e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:29.683 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.682+0000 7f9767a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9760077d80 0x7f97601a56e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:29.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.682+0000 7f9765000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f97600761a0 0x7f976019e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:29.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.682+0000 7f9767a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f9760113a90 con 0x7f97600773e0 2024-03-18T16:18:29.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.682+0000 7f9767a00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f9760113910 con 0x7f97600761a0 2024-03-18T16:18:29.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.682+0000 7f9767a00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f9760113c10 con 0x7f9760077d80 2024-03-18T16:18:29.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.682+0000 7f9765000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f97600761a0 0x7f976019e130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:58514/0 (socket says 172.21.15.39:58514) 2024-03-18T16:18:29.684 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.682+0000 7f9765000640 1 -- 172.21.15.39:0/1749290208 learned_addr learned my addr 172.21.15.39:0/1749290208 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:18:29.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.683+0000 7f9765a00640 1 --2- 172.21.15.39:0/1749290208 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9760077d80 0x7f97601a56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:29.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.683+0000 7f9765000640 1 -- 172.21.15.39:0/1749290208 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9760077d80 msgr2=0x7f97601a56e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:18:29.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.683+0000 7f9765000640 1 --2- 172.21.15.39:0/1749290208 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9760077d80 0x7f97601a56e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:29.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.683+0000 7f9765000640 1 -- 172.21.15.39:0/1749290208 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f97600773e0 msgr2=0x7f976019e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:18:29.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.683+0000 7f9765000640 1 --2- 172.21.15.39:0/1749290208 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f97600773e0 0x7f976019e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:29.685 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.683+0000 7f9765000640 1 -- 172.21.15.39:0/1749290208 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97601a5d50 con 0x7f97600761a0 2024-03-18T16:18:29.686 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.684+0000 7f9765000640 1 --2- 172.21.15.39:0/1749290208 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f97600761a0 0x7f976019e130 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f9754000c00 tx=0x7f9754032460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:18:29.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.685+0000 7f975d600640 1 -- 172.21.15.39:0/1749290208 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9754039040 con 0x7f97600761a0 2024-03-18T16:18:29.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.685+0000 7f9767a00640 1 -- 172.21.15.39:0/1749290208 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97601a5fe0 con 0x7f97600761a0 2024-03-18T16:18:29.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.685+0000 7f9767a00640 1 -- 172.21.15.39:0/1749290208 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97601a64a0 con 0x7f97600761a0 2024-03-18T16:18:29.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.685+0000 7f975d600640 1 -- 172.21.15.39:0/1749290208 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9754042070 con 0x7f97600761a0 2024-03-18T16:18:29.687 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.685+0000 7f975d600640 1 -- 172.21.15.39:0/1749290208 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97540046c0 con 0x7f97600761a0 2024-03-18T16:18:29.692 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.690+0000 7f976a781740 1 Processor -- start 2024-03-18T16:18:29.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.691+0000 7f976a781740 1 -- start start 2024-03-18T16:18:29.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.691+0000 7f976a781740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55602276d8c0 0x55602276dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:29.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.691+0000 7f976a781740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55602276e2b0 0x556022772db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:29.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.691+0000 7f976a781740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556022773420 0x5560227758b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:29.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.691+0000 7f976a781740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5560227353b0 con 0x55602276d8c0 2024-03-18T16:18:29.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.691+0000 7f976a781740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5560226d2450 con 0x556022773420 2024-03-18T16:18:29.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.691+0000 7f976a781740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55602267acb0 con 0x55602276e2b0 2024-03-18T16:18:29.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.692+0000 7f975fe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556022773420 0x5560227758b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:29.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.692+0000 7f975fe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556022773420 0x5560227758b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:58520/0 (socket says 172.21.15.39:58520) 2024-03-18T16:18:29.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.692+0000 7f975fe00640 1 -- 172.21.15.39:0/1918304744 learned_addr learned my addr 172.21.15.39:0/1918304744 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:18:29.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.693+0000 7f975fe00640 1 -- 172.21.15.39:0/1918304744 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55602276e2b0 msgr2=0x556022772db0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:18:29.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.693+0000 7f975fe00640 1 --2- 172.21.15.39:0/1918304744 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55602276e2b0 0x556022772db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:29.695 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.693+0000 7f975fe00640 1 -- 172.21.15.39:0/1918304744 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55602276d8c0 msgr2=0x55602276dce0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:18:29.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.693+0000 7f975fe00640 1 --2- 172.21.15.39:0/1918304744 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55602276d8c0 0x55602276dce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:29.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.693+0000 7f975fe00640 1 -- 172.21.15.39:0/1918304744 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55602269c1f0 con 0x556022773420 2024-03-18T16:18:29.696 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.693+0000 7f975fe00640 1 --2- 172.21.15.39:0/1918304744 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556022773420 0x5560227758b0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f974c009be0 tx=0x7f974c00e720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b29f7b8f3e5f60d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:18:29.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.694+0000 7f9752000640 1 -- 172.21.15.39:0/1918304744 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f974c005020 con 0x556022773420 2024-03-18T16:18:29.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.694+0000 7f9752000640 1 -- 172.21.15.39:0/1918304744 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f974c0047a0 con 0x556022773420 2024-03-18T16:18:29.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.694+0000 7f9752000640 1 -- 172.21.15.39:0/1918304744 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f974c005020 con 0x556022773420 2024-03-18T16:18:29.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.695+0000 7f975d600640 1 -- 172.21.15.39:0/1749290208 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f9754004860 con 0x7f97600761a0 2024-03-18T16:18:29.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.695+0000 7f976a781740 1 -- 172.21.15.39:0/1918304744 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556022773420 msgr2=0x5560227758b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:18:29.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.695+0000 7f976a781740 1 --2- 172.21.15.39:0/1918304744 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556022773420 0x5560227758b0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f974c009be0 tx=0x7f974c00e720 comp rx=0 tx=0).stop 2024-03-18T16:18:29.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.696+0000 7f976a781740 1 -- 172.21.15.39:0/1918304744 shutdown_connections 2024-03-18T16:18:29.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.696+0000 7f976a781740 1 --2- 172.21.15.39:0/1918304744 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556022773420 0x5560227758b0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:29.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.696+0000 7f976a781740 1 --2- 172.21.15.39:0/1918304744 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55602276e2b0 0x556022772db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:29.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.696+0000 7f976a781740 1 --2- 172.21.15.39:0/1918304744 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55602276d8c0 0x55602276dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:29.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.696+0000 7f976a781740 1 -- 172.21.15.39:0/1918304744 >> 172.21.15.39:0/1918304744 conn(0x556022768e30 msgr2=0x556022769230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:18:29.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.697+0000 7f976a781740 1 -- 172.21.15.39:0/1918304744 shutdown_connections 2024-03-18T16:18:29.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.697+0000 7f976a781740 1 -- 172.21.15.39:0/1918304744 wait complete. 2024-03-18T16:18:29.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.697+0000 7f976a781740 1 -- start start 2024-03-18T16:18:29.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.698+0000 7f975d600640 1 --2- 172.21.15.39:0/1749290208 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f97480a2860 0x7f97480a4d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:29.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.698+0000 7f976a781740 1 Processor -- start 2024-03-18T16:18:29.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.699+0000 7f976a781740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55602276d8c0 0x556022788a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:29.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.699+0000 7f976a781740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55602276e2b0 0x556022788f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:29.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.699+0000 7f9765a00640 1 --2- 172.21.15.39:0/1749290208 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f97480a2860 0x7f97480a4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:29.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.699+0000 7f976a781740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556022773420 0x55602278ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:29.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.699+0000 7f976a781740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55602268e110 con 0x55602276e2b0 2024-03-18T16:18:29.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.699+0000 7f976a781740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5560226ce4f0 con 0x55602276d8c0 2024-03-18T16:18:29.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.699+0000 7f976a781740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x5560227873e0 con 0x556022773420 2024-03-18T16:18:29.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.700+0000 7f975d600640 1 -- 172.21.15.39:0/1749290208 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f97540f05a0 con 0x7f97600761a0 2024-03-18T16:18:29.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.700+0000 7f9765000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55602276d8c0 0x556022788a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:29.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.700+0000 7f9765000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55602276d8c0 0x556022788a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:58528/0 (socket says 172.21.15.39:58528) 2024-03-18T16:18:29.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.700+0000 7f9765000640 1 -- 172.21.15.39:0/3440972133 learned_addr learned my addr 172.21.15.39:0/3440972133 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:18:29.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.699+0000 7f976a781740 10 client.0 _ll_register_callbacks cb 0x55602267bb70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:18:29.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.700+0000 7f9765a00640 1 --2- 172.21.15.39:0/3440972133 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556022773420 0x55602278ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:29.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.701+0000 7f975fe00640 1 --2- 172.21.15.39:0/3440972133 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55602276e2b0 0x556022788f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:29.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.701+0000 7f9765000640 1 -- 172.21.15.39:0/3440972133 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556022773420 msgr2=0x55602278ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:18:29.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.701+0000 7f9765000640 1 --2- 172.21.15.39:0/3440972133 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556022773420 0x55602278ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:29.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.701+0000 7f9765000640 1 -- 172.21.15.39:0/3440972133 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55602276e2b0 msgr2=0x556022788f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:18:29.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.701+0000 7f9765000640 1 --2- 172.21.15.39:0/3440972133 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55602276e2b0 0x556022788f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:29.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.701+0000 7f9765000640 1 -- 172.21.15.39:0/3440972133 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5560223e63f0 con 0x55602276d8c0 2024-03-18T16:18:29.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.702+0000 7f9765000640 1 --2- 172.21.15.39:0/3440972133 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55602276d8c0 0x556022788a20 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f97540324d0 tx=0x7f97540bc750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=618f93dfac2dc96c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:18:29.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.702+0000 7f9746a00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:18:29.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.703+0000 7f9746a00640 1 -- 172.21.15.39:0/3440972133 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97540ef070 con 0x55602276d8c0 2024-03-18T16:18:29.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.703+0000 7f9746a00640 1 -- 172.21.15.39:0/3440972133 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f97540312a0 con 0x55602276d8c0 2024-03-18T16:18:29.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.703+0000 7f9765a00640 1 --2- 172.21.15.39:0/1749290208 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f97480a2860 0x7f97480a4d20 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x55602276eb10 tx=0x7f9758009df0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:18:29.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.703+0000 7f976a781740 1 -- 172.21.15.39:0/3440972133 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f974c0047a0 con 0x55602276d8c0 2024-03-18T16:18:29.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.704+0000 7f9746a00640 1 -- 172.21.15.39:0/3440972133 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f97540314a0 con 0x55602276d8c0 2024-03-18T16:18:29.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.704+0000 7f976a781740 10 client.27248 fetch_fsmap learned FSMap version 44 2024-03-18T16:18:29.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.704+0000 7f976a781740 1 -- 172.21.15.39:0/3440972133 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x5560224b07d0 con 0x55602276d8c0 2024-03-18T16:18:29.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.704+0000 7f9746a00640 1 -- 172.21.15.39:0/3440972133 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f974c0047a0 con 0x55602276d8c0 2024-03-18T16:18:29.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.704+0000 7f9746a00640 1 -- 172.21.15.39:0/3440972133 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7f975403c070 con 0x55602276d8c0 2024-03-18T16:18:29.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.704+0000 7f976a781740 10 client.27248 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:18:29.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.704+0000 7f976a781740 10 client.27248 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-18T16:18:29.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.704+0000 7f976a781740 20 client.27248 populate_metadata read hostname 'smithi039' 2024-03-18T16:18:29.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.704+0000 7f976a781740 1 --2- 172.21.15.39:0/3440972133 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x556022794850 0x556022796c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:29.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.704+0000 7f976a781740 4 client.27248 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:18:29.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.704+0000 7f976a781740 1 -- 172.21.15.39:0/3440972133 --> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5560226df990 con 0x556022794850 2024-03-18T16:18:29.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.705+0000 7f975fe00640 1 --2- 172.21.15.39:0/3440972133 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x556022794850 0x556022796c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:29.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.707+0000 7f9746a00640 10 client.27248 ms_handle_connect on v2:172.21.15.194:6836/1301464246 2024-03-18T16:18:29.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.707+0000 7f975fe00640 1 --2- 172.21.15.39:0/3440972133 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x556022794850 0x556022796c90 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=658d6b3c38c263c1 server_cookie=fd3fc25ad4e69536 in_seq=0 out_seq=0 2024-03-18T16:18:29.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.708+0000 7f9746a00640 1 -- 172.21.15.39:0/3440972133 <== mds.3 v2:172.21.15.194:6836/1301464246 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5560226df990 con 0x556022794850 2024-03-18T16:18:29.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.708+0000 7f9746a00640 10 client.27248 handle_command_reply: tid=0 2024-03-18T16:18:29.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.709+0000 7f976a781740 1 client.27248 shutdown 2024-03-18T16:18:29.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.709+0000 7f976a781740 1 -- 172.21.15.39:0/3440972133 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55602276d8c0 msgr2=0x556022788a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:18:29.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.709+0000 7f976a781740 1 --2- 172.21.15.39:0/3440972133 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55602276d8c0 0x556022788a20 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f97540324d0 tx=0x7f97540bc750 comp rx=0 tx=0).stop 2024-03-18T16:18:29.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.709+0000 7f976a781740 1 -- 172.21.15.39:0/3440972133 shutdown_connections 2024-03-18T16:18:29.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.709+0000 7f976a781740 1 --2- 172.21.15.39:0/3440972133 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x556022794850 0x556022796c90 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:29.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.710+0000 7f976a781740 1 --2- 172.21.15.39:0/3440972133 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556022773420 0x55602278ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:29.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.710+0000 7f976a781740 1 --2- 172.21.15.39:0/3440972133 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55602276e2b0 0x556022788f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:29.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.710+0000 7f976a781740 1 --2- 172.21.15.39:0/3440972133 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55602276d8c0 0x556022788a20 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:29.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.710+0000 7f976a781740 1 -- 172.21.15.39:0/3440972133 >> 172.21.15.39:0/3440972133 conn(0x556022768e30 msgr2=0x5560227681f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:18:29.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.710+0000 7f9746a00640 0 client.27248 ms_handle_reset on v2:172.21.15.194:6836/1301464246 2024-03-18T16:18:29.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.710+0000 7f976a781740 1 -- 172.21.15.39:0/3440972133 shutdown_connections 2024-03-18T16:18:29.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.710+0000 7f976a781740 1 -- 172.21.15.39:0/3440972133 wait complete. 2024-03-18T16:18:29.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.710+0000 7f976a781740 20 client.27248 trim_cache size 0 max 16384 2024-03-18T16:18:29.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.731+0000 7f976a781740 1 Processor -- start 2024-03-18T16:18:29.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.731+0000 7f976a781740 1 -- start start 2024-03-18T16:18:29.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.731+0000 7f976a781740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55602276d8c0 0x556022772b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:29.733 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.731+0000 7f976a781740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55602276e2b0 0x55602279c560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:29.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.732+0000 7f976a781740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556022773420 0x556022787810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:29.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.732+0000 7f976a781740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f974000fa90 con 0x556022773420 2024-03-18T16:18:29.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.732+0000 7f976a781740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f97400115d0 con 0x55602276d8c0 2024-03-18T16:18:29.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.732+0000 7f976a781740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f9740010160 con 0x55602276e2b0 2024-03-18T16:18:29.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.732+0000 7f9765000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55602276d8c0 0x556022772b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:29.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.732+0000 7f975fe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55602276e2b0 0x55602279c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:29.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.732+0000 7f975fe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55602276e2b0 0x55602279c560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:49172/0 (socket says 172.21.15.39:49172) 2024-03-18T16:18:29.734 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.732+0000 7f975fe00640 1 -- 172.21.15.39:0/1473096030 learned_addr learned my addr 172.21.15.39:0/1473096030 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:18:29.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.732+0000 7f975fe00640 1 -- 172.21.15.39:0/1473096030 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55602276d8c0 msgr2=0x556022772b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:18:29.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.732+0000 7f975fe00640 1 --2- 172.21.15.39:0/1473096030 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55602276d8c0 0x556022772b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:29.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.732+0000 7f975fe00640 1 -- 172.21.15.39:0/1473096030 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556022773420 msgr2=0x556022787810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:18:29.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.732+0000 7f975fe00640 1 --2- 172.21.15.39:0/1473096030 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556022773420 0x556022787810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:29.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.732+0000 7f975fe00640 1 -- 172.21.15.39:0/1473096030 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556022764c10 con 0x55602276e2b0 2024-03-18T16:18:29.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.733+0000 7f975fe00640 1 --2- 172.21.15.39:0/1473096030 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55602276e2b0 0x55602279c560 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x556022769da0 tx=0x7f974c000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3382617d1b93dd44 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:18:29.735 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.733+0000 7f9751600640 1 -- 172.21.15.39:0/1473096030 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f974c02e070 con 0x55602276e2b0 2024-03-18T16:18:29.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.733+0000 7f9751600640 1 -- 172.21.15.39:0/1473096030 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f974c028070 con 0x55602276e2b0 2024-03-18T16:18:29.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.733+0000 7f9751600640 1 -- 172.21.15.39:0/1473096030 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f974c01d040 con 0x55602276e2b0 2024-03-18T16:18:29.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.734+0000 7f976a781740 1 -- 172.21.15.39:0/1473096030 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55602276e2b0 msgr2=0x55602279c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:18:29.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.734+0000 7f976a781740 1 --2- 172.21.15.39:0/1473096030 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55602276e2b0 0x55602279c560 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x556022769da0 tx=0x7f974c000bc0 comp rx=0 tx=0).stop 2024-03-18T16:18:29.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.734+0000 7f976a781740 1 -- 172.21.15.39:0/1473096030 shutdown_connections 2024-03-18T16:18:29.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.734+0000 7f976a781740 1 --2- 172.21.15.39:0/1473096030 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556022773420 0x556022787810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:29.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.734+0000 7f976a781740 1 --2- 172.21.15.39:0/1473096030 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55602276e2b0 0x55602279c560 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:29.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.734+0000 7f976a781740 1 --2- 172.21.15.39:0/1473096030 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55602276d8c0 0x556022772b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:29.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.734+0000 7f976a781740 1 -- 172.21.15.39:0/1473096030 >> 172.21.15.39:0/1473096030 conn(0x556022768e30 msgr2=0x556022764080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:18:29.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.734+0000 7f976a781740 1 -- 172.21.15.39:0/1473096030 shutdown_connections 2024-03-18T16:18:29.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.735+0000 7f976a781740 1 -- 172.21.15.39:0/1473096030 wait complete. 2024-03-18T16:18:29.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.735+0000 7f976a781740 1 -- start start 2024-03-18T16:18:29.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.735+0000 7f976a781740 1 Processor -- start 2024-03-18T16:18:29.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.736+0000 7f976a781740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55602276d8c0 0x55602278d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:29.737 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.736+0000 7f976a781740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55602276e2b0 0x55602278d640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:29.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.736+0000 7f976a781740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556022773420 0x55602278e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:29.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.736+0000 7f976a781740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f9740010f00 con 0x55602276d8c0 2024-03-18T16:18:29.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.736+0000 7f976a781740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f9740010830 con 0x55602276e2b0 2024-03-18T16:18:29.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.736+0000 7f976a781740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f974000da70 con 0x556022773420 2024-03-18T16:18:29.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.736+0000 7f976a781740 10 client.0 _ll_register_callbacks cb 0x7f97540b8dd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:18:29.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.736+0000 7f9765a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556022773420 0x55602278e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:29.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.736+0000 7f9765a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556022773420 0x55602278e650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:49186/0 (socket says 172.21.15.39:49186) 2024-03-18T16:18:29.738 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.736+0000 7f9765a00640 1 -- 172.21.15.39:0/2374168514 learned_addr learned my addr 172.21.15.39:0/2374168514 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:18:29.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.736+0000 7f975fe00640 1 --2- 172.21.15.39:0/2374168514 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55602276e2b0 0x55602278d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:29.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.736+0000 7f9765000640 1 --2- 172.21.15.39:0/2374168514 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55602276d8c0 0x55602278d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:29.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.737+0000 7f975fe00640 1 -- 172.21.15.39:0/2374168514 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556022773420 msgr2=0x55602278e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:18:29.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.737+0000 7f975fe00640 1 --2- 172.21.15.39:0/2374168514 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556022773420 0x55602278e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:29.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.737+0000 7f975fe00640 1 -- 172.21.15.39:0/2374168514 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55602276d8c0 msgr2=0x55602278d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:18:29.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.737+0000 7f975fe00640 1 --2- 172.21.15.39:0/2374168514 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55602276d8c0 0x55602278d100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:29.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.737+0000 7f975fe00640 1 -- 172.21.15.39:0/2374168514 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55602276d500 con 0x55602276e2b0 2024-03-18T16:18:29.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.737+0000 7f9765a00640 1 --2- 172.21.15.39:0/2374168514 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556022773420 0x55602278e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:18:29.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.737+0000 7f9765000640 1 --2- 172.21.15.39:0/2374168514 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55602276d8c0 0x55602278d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:18:29.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.737+0000 7f975fe00640 1 --2- 172.21.15.39:0/2374168514 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55602276e2b0 0x55602278d640 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f974c00ec20 tx=0x7f974c005d30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=611cbde7c77e0746 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:18:29.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.737+0000 7f9746a00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:18:29.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.738+0000 7f9746a00640 1 -- 172.21.15.39:0/2374168514 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f974c036060 con 0x55602276e2b0 2024-03-18T16:18:29.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.738+0000 7f9746a00640 1 -- 172.21.15.39:0/2374168514 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f974c009530 con 0x55602276e2b0 2024-03-18T16:18:29.740 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.738+0000 7f976a781740 1 -- 172.21.15.39:0/2374168514 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f974c028070 con 0x55602276e2b0 2024-03-18T16:18:29.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.739+0000 7f9746a00640 1 -- 172.21.15.39:0/2374168514 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f974c03d050 con 0x55602276e2b0 2024-03-18T16:18:29.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.739+0000 7f976a781740 10 client.27254 fetch_fsmap learned FSMap version 44 2024-03-18T16:18:29.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.739+0000 7f976a781740 1 -- 172.21.15.39:0/2374168514 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x55602278fd70 con 0x55602276e2b0 2024-03-18T16:18:29.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.739+0000 7f9746a00640 1 -- 172.21.15.39:0/2374168514 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f974c028070 con 0x55602276e2b0 2024-03-18T16:18:29.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.739+0000 7f9746a00640 1 -- 172.21.15.39:0/2374168514 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7f974c005310 con 0x55602276e2b0 2024-03-18T16:18:29.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.739+0000 7f976a781740 10 client.27254 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:18:29.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.739+0000 7f976a781740 10 client.27254 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-18T16:18:29.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.739+0000 7f976a781740 20 client.27254 populate_metadata read hostname 'smithi039' 2024-03-18T16:18:29.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.739+0000 7f976a781740 1 --2- 172.21.15.39:0/2374168514 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x5560227faed0 0x5560227fd2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:29.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.739+0000 7f976a781740 4 client.27254 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:18:29.741 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.739+0000 7f976a781740 1 -- 172.21.15.39:0/2374168514 --> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5560226d9ce0 con 0x5560227faed0 2024-03-18T16:18:29.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.740+0000 7f9765000640 1 --2- 172.21.15.39:0/2374168514 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x5560227faed0 0x5560227fd2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:29.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.741+0000 7f9765000640 1 --2- 172.21.15.39:0/2374168514 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x5560227faed0 0x5560227fd2f0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=d00cb60e24d92ff1 server_cookie=23fb52b801fab965 in_seq=0 out_seq=0 2024-03-18T16:18:29.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.741+0000 7f9746a00640 10 client.27254 ms_handle_connect on v2:172.21.15.194:6836/1301464246 2024-03-18T16:18:29.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.741+0000 7f9746a00640 1 -- 172.21.15.39:0/2374168514 <== mds.3 v2:172.21.15.194:6836/1301464246 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5560226d9ce0 con 0x5560227faed0 2024-03-18T16:18:29.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.741+0000 7f9746a00640 10 client.27254 handle_command_reply: tid=0 2024-03-18T16:18:29.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.741+0000 7f976a781740 1 client.27254 shutdown 2024-03-18T16:18:29.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.741+0000 7f976a781740 1 -- 172.21.15.39:0/2374168514 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55602276e2b0 msgr2=0x55602278d640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:18:29.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.741+0000 7f976a781740 1 --2- 172.21.15.39:0/2374168514 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55602276e2b0 0x55602278d640 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f974c00ec20 tx=0x7f974c005d30 comp rx=0 tx=0).stop 2024-03-18T16:18:29.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.742+0000 7f976a781740 1 -- 172.21.15.39:0/2374168514 shutdown_connections 2024-03-18T16:18:29.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.742+0000 7f976a781740 1 --2- 172.21.15.39:0/2374168514 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x5560227faed0 0x5560227fd2f0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:29.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.742+0000 7f976a781740 1 --2- 172.21.15.39:0/2374168514 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55602276e2b0 0x55602278d640 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:29.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.742+0000 7f976a781740 1 --2- 172.21.15.39:0/2374168514 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556022773420 0x55602278e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:29.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.742+0000 7f976a781740 1 --2- 172.21.15.39:0/2374168514 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55602276d8c0 0x55602278d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:29.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.742+0000 7f976a781740 1 -- 172.21.15.39:0/2374168514 >> 172.21.15.39:0/2374168514 conn(0x556022768e30 msgr2=0x5560227726f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:18:29.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.742+0000 7f9746a00640 0 client.27254 ms_handle_reset on v2:172.21.15.194:6836/1301464246 2024-03-18T16:18:29.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.742+0000 7f976a781740 1 -- 172.21.15.39:0/2374168514 shutdown_connections 2024-03-18T16:18:29.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.742+0000 7f976a781740 1 -- 172.21.15.39:0/2374168514 wait complete. 2024-03-18T16:18:29.744 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.742+0000 7f976a781740 20 client.27254 trim_cache size 0 max 16384 2024-03-18T16:18:29.745 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:18:29.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.745+0000 7f9767a00640 1 -- 172.21.15.39:0/1749290208 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f97480a2860 msgr2=0x7f97480a4d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:18:29.746 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.745+0000 7f9767a00640 1 --2- 172.21.15.39:0/1749290208 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f97480a2860 0x7f97480a4d20 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x55602276eb10 tx=0x7f9758009df0 comp rx=0 tx=0).stop 2024-03-18T16:18:29.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.745+0000 7f9767a00640 1 -- 172.21.15.39:0/1749290208 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f97600761a0 msgr2=0x7f976019e130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:18:29.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.745+0000 7f9767a00640 1 --2- 172.21.15.39:0/1749290208 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f97600761a0 0x7f976019e130 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f9754000c00 tx=0x7f9754032460 comp rx=0 tx=0).stop 2024-03-18T16:18:29.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.745+0000 7f9767a00640 1 -- 172.21.15.39:0/1749290208 shutdown_connections 2024-03-18T16:18:29.747 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.745+0000 7f9767a00640 1 --2- 172.21.15.39:0/1749290208 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f97480a2860 0x7f97480a4d20 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:29.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.745+0000 7f9767a00640 1 --2- 172.21.15.39:0/1749290208 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9760077d80 0x7f97601a56e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:29.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.745+0000 7f9767a00640 1 --2- 172.21.15.39:0/1749290208 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f97600773e0 0x7f976019e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:29.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.745+0000 7f9767a00640 1 --2- 172.21.15.39:0/1749290208 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f97600761a0 0x7f976019e130 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:29.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.745+0000 7f9767a00640 1 -- 172.21.15.39:0/1749290208 >> 172.21.15.39:0/1749290208 conn(0x7f97600fef90 msgr2=0x7f9760100a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:18:29.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.746+0000 7f9767a00640 1 -- 172.21.15.39:0/1749290208 shutdown_connections 2024-03-18T16:18:29.748 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:29.746+0000 7f9767a00640 1 -- 172.21.15.39:0/1749290208 wait complete. 2024-03-18T16:18:29.769 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-18T16:18:30.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.048+0000 7f23eaa00640 1 -- 172.21.15.39:0/2001975878 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f23e41016e0 msgr2=0x7f23e4101b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:18:30.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.048+0000 7f23eaa00640 1 --2- 172.21.15.39:0/2001975878 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f23e41016e0 0x7f23e4101b00 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f23d8009900 tx=0x7f23d802bc40 comp rx=0 tx=0).stop 2024-03-18T16:18:30.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.048+0000 7f23eaa00640 1 -- 172.21.15.39:0/2001975878 shutdown_connections 2024-03-18T16:18:30.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.048+0000 7f23eaa00640 1 --2- 172.21.15.39:0/2001975878 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f23e4103290 0x7f23e4109b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.048+0000 7f23eaa00640 1 --2- 172.21.15.39:0/2001975878 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f23e41028d0 0x7f23e4102d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.048+0000 7f23eaa00640 1 --2- 172.21.15.39:0/2001975878 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f23e41016e0 0x7f23e4101b00 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.094 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.048+0000 7f23eaa00640 1 -- 172.21.15.39:0/2001975878 >> 172.21.15.39:0/2001975878 conn(0x7f23e40fcc50 msgr2=0x7f23e40ff090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:18:30.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.048+0000 7f23eaa00640 1 -- 172.21.15.39:0/2001975878 shutdown_connections 2024-03-18T16:18:30.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.048+0000 7f23eaa00640 1 -- 172.21.15.39:0/2001975878 wait complete. 2024-03-18T16:18:30.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.050+0000 7f23eaa00640 1 Processor -- start 2024-03-18T16:18:30.100 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.051+0000 7f23eaa00640 1 -- start start 2024-03-18T16:18:30.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.051+0000 7f23eaa00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f23e41016e0 0x7f23e4192e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:30.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.051+0000 7f23eaa00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f23e41028d0 0x7f23e4193380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:30.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.051+0000 7f23eaa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f23e4103290 0x7f23e419a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:30.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.051+0000 7f23eaa00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f23e410d360 con 0x7f23e4103290 2024-03-18T16:18:30.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.051+0000 7f23eaa00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f23e410d1e0 con 0x7f23e41028d0 2024-03-18T16:18:30.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.051+0000 7f23eaa00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f23e410d4e0 con 0x7f23e41016e0 2024-03-18T16:18:30.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.052+0000 7f23e9600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f23e41016e0 0x7f23e4192e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:30.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.052+0000 7f23e8c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f23e41028d0 0x7f23e4193380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:30.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.052+0000 7f23e8c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f23e41028d0 0x7f23e4193380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:58546/0 (socket says 172.21.15.39:58546) 2024-03-18T16:18:30.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.052+0000 7f23e8c00640 1 -- 172.21.15.39:0/921529472 learned_addr learned my addr 172.21.15.39:0/921529472 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:18:30.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.052+0000 7f23e8c00640 1 -- 172.21.15.39:0/921529472 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f23e41016e0 msgr2=0x7f23e4192e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:18:30.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.052+0000 7f23e8c00640 1 --2- 172.21.15.39:0/921529472 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f23e41016e0 0x7f23e4192e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.052+0000 7f23e8c00640 1 -- 172.21.15.39:0/921529472 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f23e4103290 msgr2=0x7f23e419a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:18:30.101 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.052+0000 7f23e8c00640 1 --2- 172.21.15.39:0/921529472 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f23e4103290 0x7f23e419a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.052+0000 7f23e8c00640 1 -- 172.21.15.39:0/921529472 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23e419aa60 con 0x7f23e41028d0 2024-03-18T16:18:30.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.053+0000 7f23e8c00640 1 --2- 172.21.15.39:0/921529472 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f23e41028d0 0x7f23e4193380 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f23d000e9f0 tx=0x7f23d000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:18:30.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.053+0000 7f23e2000640 1 -- 172.21.15.39:0/921529472 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23d000cd80 con 0x7f23e41028d0 2024-03-18T16:18:30.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.053+0000 7f23e2000640 1 -- 172.21.15.39:0/921529472 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f23d0010910 con 0x7f23e41028d0 2024-03-18T16:18:30.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.053+0000 7f23e2000640 1 -- 172.21.15.39:0/921529472 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23d0018990 con 0x7f23e41028d0 2024-03-18T16:18:30.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.054+0000 7f23eaa00640 1 -- 172.21.15.39:0/921529472 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23e419ad50 con 0x7f23e41028d0 2024-03-18T16:18:30.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.054+0000 7f23eaa00640 1 -- 172.21.15.39:0/921529472 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23e41aeba0 con 0x7f23e41028d0 2024-03-18T16:18:30.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.059+0000 7f23eedff740 1 Processor -- start 2024-03-18T16:18:30.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.059+0000 7f23eedff740 1 -- start start 2024-03-18T16:18:30.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.060+0000 7f23eedff740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5555aacbba70 0x5555aacbbe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:30.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.060+0000 7f23eedff740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5555aacbc460 0x5555aacc0f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:30.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.060+0000 7f23eedff740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5555aacc15d0 0x5555aacc3a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:30.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.060+0000 7f23eedff740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5555aac723b0 con 0x5555aacbc460 2024-03-18T16:18:30.102 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.060+0000 7f23eedff740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5555aac0f450 con 0x5555aacc15d0 2024-03-18T16:18:30.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.060+0000 7f23eedff740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x5555aabb7cb0 con 0x5555aacbba70 2024-03-18T16:18:30.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.060+0000 7f23ea000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5555aacbba70 0x5555aacbbe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:30.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.060+0000 7f23ea000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5555aacbba70 0x5555aacbbe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:49210/0 (socket says 172.21.15.39:49210) 2024-03-18T16:18:30.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.060+0000 7f23ea000640 1 -- 172.21.15.39:0/4208120796 learned_addr learned my addr 172.21.15.39:0/4208120796 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:18:30.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.061+0000 7f23ea000640 1 -- 172.21.15.39:0/4208120796 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5555aacc15d0 msgr2=0x5555aacc3a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:18:30.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.061+0000 7f23e9600640 1 --2- 172.21.15.39:0/4208120796 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5555aacbc460 0x5555aacc0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:30.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.061+0000 7f23e8c00640 1 --2- 172.21.15.39:0/4208120796 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5555aacc15d0 0x5555aacc3a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:30.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.061+0000 7f23ea000640 1 --2- 172.21.15.39:0/4208120796 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5555aacc15d0 0x5555aacc3a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.061+0000 7f23ea000640 1 -- 172.21.15.39:0/4208120796 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5555aacbc460 msgr2=0x5555aacc0f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:18:30.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.061+0000 7f23ea000640 1 --2- 172.21.15.39:0/4208120796 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5555aacbc460 0x5555aacc0f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.061+0000 7f23ea000640 1 -- 172.21.15.39:0/4208120796 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5555aabd91f0 con 0x5555aacbba70 2024-03-18T16:18:30.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.061+0000 7f23e9600640 1 --2- 172.21.15.39:0/4208120796 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5555aacbc460 0x5555aacc0f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:18:30.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.062+0000 7f23e2000640 1 -- 172.21.15.39:0/921529472 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f23d0018b30 con 0x7f23e41028d0 2024-03-18T16:18:30.103 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.062+0000 7f23ea000640 1 --2- 172.21.15.39:0/4208120796 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5555aacbba70 0x5555aacbbe90 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f23dc009ea0 tx=0x7f23dc009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=416f4d35deaffbae server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:18:30.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.062+0000 7f23e8c00640 1 --2- 172.21.15.39:0/4208120796 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5555aacc15d0 0x5555aacc3a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:18:30.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.062+0000 7f23d6600640 1 -- 172.21.15.39:0/4208120796 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23dc004d20 con 0x5555aacbba70 2024-03-18T16:18:30.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.062+0000 7f23d6600640 1 -- 172.21.15.39:0/4208120796 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f23dc00ee60 con 0x5555aacbba70 2024-03-18T16:18:30.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.062+0000 7f23d6600640 1 -- 172.21.15.39:0/4208120796 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23dc0108f0 con 0x5555aacbba70 2024-03-18T16:18:30.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.063+0000 7f23eedff740 1 -- 172.21.15.39:0/4208120796 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5555aacbba70 msgr2=0x5555aacbbe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:18:30.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.064+0000 7f23eedff740 1 --2- 172.21.15.39:0/4208120796 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5555aacbba70 0x5555aacbbe90 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f23dc009ea0 tx=0x7f23dc009f90 comp rx=0 tx=0).stop 2024-03-18T16:18:30.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.064+0000 7f23e2000640 1 --2- 172.21.15.39:0/921529472 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f23cc0a29d0 0x7f23cc0a4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:30.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.065+0000 7f23e2000640 1 -- 172.21.15.39:0/921529472 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f23d00cf190 con 0x7f23e41028d0 2024-03-18T16:18:30.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.066+0000 7f23eedff740 1 -- 172.21.15.39:0/4208120796 shutdown_connections 2024-03-18T16:18:30.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.066+0000 7f23eedff740 1 --2- 172.21.15.39:0/4208120796 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5555aacc15d0 0x5555aacc3a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.066+0000 7f23eedff740 1 --2- 172.21.15.39:0/4208120796 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5555aacbc460 0x5555aacc0f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.066+0000 7f23eedff740 1 --2- 172.21.15.39:0/4208120796 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5555aacbba70 0x5555aacbbe90 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.066+0000 7f23eedff740 1 -- 172.21.15.39:0/4208120796 >> 172.21.15.39:0/4208120796 conn(0x5555aaca5fd0 msgr2=0x5555aaca63d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:18:30.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.067+0000 7f23ea000640 1 --2- 172.21.15.39:0/921529472 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f23cc0a29d0 0x7f23cc0a4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:30.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.067+0000 7f23eedff740 1 -- 172.21.15.39:0/4208120796 shutdown_connections 2024-03-18T16:18:30.104 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.067+0000 7f23eedff740 1 -- 172.21.15.39:0/4208120796 wait complete. 2024-03-18T16:18:30.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.068+0000 7f23eedff740 1 -- start start 2024-03-18T16:18:30.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.069+0000 7f23eedff740 1 Processor -- start 2024-03-18T16:18:30.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.069+0000 7f23ea000640 1 --2- 172.21.15.39:0/921529472 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f23cc0a29d0 0x7f23cc0a4e90 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f23dc009ce0 tx=0x7f23dc009200 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:18:30.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.070+0000 7f23eedff740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5555aacbba70 0x5555aacc5c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:30.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.070+0000 7f23eedff740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5555aacbc460 0x5555aacc6190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:30.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.070+0000 7f23eedff740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5555aacc15d0 0x5555aaccd1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:30.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.071+0000 7f23eedff740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5555aabcb110 con 0x5555aacc15d0 2024-03-18T16:18:30.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.071+0000 7f23eedff740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5555aac0b4f0 con 0x5555aacbba70 2024-03-18T16:18:30.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.071+0000 7f23eedff740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x5555aacc4580 con 0x5555aacbc460 2024-03-18T16:18:30.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.071+0000 7f23eedff740 10 client.0 _ll_register_callbacks cb 0x5555aabb8b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:18:30.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.071+0000 7f23e8c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5555aacbc460 0x5555aacc6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:30.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.071+0000 7f23e9600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5555aacbba70 0x5555aacc5c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:30.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.071+0000 7f23e8c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5555aacbc460 0x5555aacc6190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:49220/0 (socket says 172.21.15.39:49220) 2024-03-18T16:18:30.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.072+0000 7f23e8c00640 1 -- 172.21.15.39:0/1305589395 learned_addr learned my addr 172.21.15.39:0/1305589395 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:18:30.105 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.072+0000 7f23ea000640 1 --2- 172.21.15.39:0/1305589395 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5555aacc15d0 0x5555aaccd1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:30.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.072+0000 7f23e8c00640 1 -- 172.21.15.39:0/1305589395 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5555aacbba70 msgr2=0x5555aacc5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:18:30.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.072+0000 7f23e8c00640 1 --2- 172.21.15.39:0/1305589395 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5555aacbba70 0x5555aacc5c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.072+0000 7f23e8c00640 1 -- 172.21.15.39:0/1305589395 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5555aacc15d0 msgr2=0x5555aaccd1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:18:30.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.073+0000 7f23e8c00640 1 --2- 172.21.15.39:0/1305589395 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5555aacc15d0 0x5555aaccd1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.073+0000 7f23e8c00640 1 -- 172.21.15.39:0/1305589395 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5555aa9233f0 con 0x5555aacbc460 2024-03-18T16:18:30.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.073+0000 7f23ea000640 1 --2- 172.21.15.39:0/1305589395 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5555aacc15d0 0x5555aaccd1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:18:30.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.074+0000 7f23e8c00640 1 --2- 172.21.15.39:0/1305589395 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5555aacbc460 0x5555aacc6190 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f23d009b6d0 tx=0x7f23d009b160 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7cd205150d5140ff server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:18:30.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.074+0000 7f23caa00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:18:30.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.074+0000 7f23caa00640 1 -- 172.21.15.39:0/1305589395 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23d00a3ee0 con 0x5555aacbc460 2024-03-18T16:18:30.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.074+0000 7f23caa00640 1 -- 172.21.15.39:0/1305589395 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f23d0097db0 con 0x5555aacbc460 2024-03-18T16:18:30.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.074+0000 7f23eedff740 1 -- 172.21.15.39:0/1305589395 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f23dc00ee60 con 0x5555aacbc460 2024-03-18T16:18:30.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.075+0000 7f23caa00640 1 -- 172.21.15.39:0/1305589395 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f23d009ab90 con 0x5555aacbc460 2024-03-18T16:18:30.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.075+0000 7f23eedff740 10 client.27244 fetch_fsmap learned FSMap version 44 2024-03-18T16:18:30.106 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.075+0000 7f23eedff740 1 -- 172.21.15.39:0/1305589395 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x5555aa9ed7d0 con 0x5555aacbc460 2024-03-18T16:18:30.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.075+0000 7f23caa00640 1 -- 172.21.15.39:0/1305589395 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f23dc00ee60 con 0x5555aacbc460 2024-03-18T16:18:30.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.075+0000 7f23caa00640 1 -- 172.21.15.39:0/1305589395 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7f23d00105d0 con 0x5555aacbc460 2024-03-18T16:18:30.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.075+0000 7f23eedff740 10 client.27244 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:18:30.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.075+0000 7f23eedff740 10 client.27244 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-18T16:18:30.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.075+0000 7f23eedff740 20 client.27244 populate_metadata read hostname 'smithi039' 2024-03-18T16:18:30.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.075+0000 7f23eedff740 1 --2- 172.21.15.39:0/1305589395 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x5555aacd1a90 0x5555aacd3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:30.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.075+0000 7f23eedff740 4 client.27244 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:18:30.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.075+0000 7f23eedff740 1 -- 172.21.15.39:0/1305589395 --> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5555aac1c990 con 0x5555aacd1a90 2024-03-18T16:18:30.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.076+0000 7f23e9600640 1 --2- 172.21.15.39:0/1305589395 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x5555aacd1a90 0x5555aacd3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:30.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.077+0000 7f23e9600640 1 --2- 172.21.15.39:0/1305589395 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x5555aacd1a90 0x5555aacd3ec0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6eb01e1c9e530385 server_cookie=7b788000da574a2f in_seq=0 out_seq=0 2024-03-18T16:18:30.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.077+0000 7f23caa00640 10 client.27244 ms_handle_connect on v2:172.21.15.196:6836/895247251 2024-03-18T16:18:30.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.078+0000 7f23caa00640 1 -- 172.21.15.39:0/1305589395 <== mds.2 v2:172.21.15.196:6836/895247251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5555aac1c990 con 0x5555aacd1a90 2024-03-18T16:18:30.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.078+0000 7f23caa00640 10 client.27244 handle_command_reply: tid=0 2024-03-18T16:18:30.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.078+0000 7f23eedff740 1 client.27244 shutdown 2024-03-18T16:18:30.107 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.079+0000 7f23eedff740 1 -- 172.21.15.39:0/1305589395 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5555aacbc460 msgr2=0x5555aacc6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:18:30.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.079+0000 7f23eedff740 1 --2- 172.21.15.39:0/1305589395 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5555aacbc460 0x5555aacc6190 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f23d009b6d0 tx=0x7f23d009b160 comp rx=0 tx=0).stop 2024-03-18T16:18:30.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.079+0000 7f23eedff740 1 -- 172.21.15.39:0/1305589395 shutdown_connections 2024-03-18T16:18:30.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.079+0000 7f23eedff740 1 --2- 172.21.15.39:0/1305589395 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x5555aacd1a90 0x5555aacd3ec0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.079+0000 7f23eedff740 1 --2- 172.21.15.39:0/1305589395 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5555aacc15d0 0x5555aaccd1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.079+0000 7f23eedff740 1 --2- 172.21.15.39:0/1305589395 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5555aacbc460 0x5555aacc6190 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.079+0000 7f23eedff740 1 --2- 172.21.15.39:0/1305589395 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5555aacbba70 0x5555aacc5c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.079+0000 7f23eedff740 1 -- 172.21.15.39:0/1305589395 >> 172.21.15.39:0/1305589395 conn(0x5555aaca5fd0 msgr2=0x5555aaca51f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:18:30.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.079+0000 7f23caa00640 0 client.27244 ms_handle_reset on v2:172.21.15.196:6836/895247251 2024-03-18T16:18:30.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.080+0000 7f23eedff740 1 -- 172.21.15.39:0/1305589395 shutdown_connections 2024-03-18T16:18:30.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.080+0000 7f23eedff740 1 -- 172.21.15.39:0/1305589395 wait complete. 2024-03-18T16:18:30.108 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.081+0000 7f23eedff740 20 client.27244 trim_cache size 0 max 16384 2024-03-18T16:18:30.108 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:18:29 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/1871069659' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:18:30.112 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.110+0000 7f23eedff740 1 Processor -- start 2024-03-18T16:18:30.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.110+0000 7f23eedff740 1 -- start start 2024-03-18T16:18:30.113 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.111+0000 7f23eedff740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5555aacbba70 0x5555aaca4960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:30.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.111+0000 7f23eedff740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5555aacbc460 0x5555aaccd190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:30.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.111+0000 7f23eedff740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5555aacc15d0 0x5555aaca4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:30.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.111+0000 7f23eedff740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f23e41b0970 con 0x5555aacbc460 2024-03-18T16:18:30.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.111+0000 7f23eedff740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f23e41b24b0 con 0x5555aacbba70 2024-03-18T16:18:30.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.111+0000 7f23eedff740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f23e41b1040 con 0x5555aacc15d0 2024-03-18T16:18:30.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.111+0000 7f23ea000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5555aacc15d0 0x5555aaca4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:30.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.111+0000 7f23ea000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5555aacc15d0 0x5555aaca4ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:49228/0 (socket says 172.21.15.39:49228) 2024-03-18T16:18:30.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.111+0000 7f23ea000640 1 -- 172.21.15.39:0/1996647770 learned_addr learned my addr 172.21.15.39:0/1996647770 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:18:30.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.111+0000 7f23e9600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5555aacbba70 0x5555aaca4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:30.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.112+0000 7f23e8c00640 1 --2- 172.21.15.39:0/1996647770 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5555aacbc460 0x5555aaccd190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:30.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.112+0000 7f23ea000640 1 -- 172.21.15.39:0/1996647770 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5555aacbba70 msgr2=0x5555aaca4960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:18:30.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.112+0000 7f23ea000640 1 --2- 172.21.15.39:0/1996647770 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5555aacbba70 0x5555aaca4960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.112+0000 7f23ea000640 1 -- 172.21.15.39:0/1996647770 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5555aacbc460 msgr2=0x5555aaccd190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:18:30.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.112+0000 7f23ea000640 1 --2- 172.21.15.39:0/1996647770 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5555aacbc460 0x5555aaccd190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.114 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.112+0000 7f23ea000640 1 -- 172.21.15.39:0/1996647770 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5555aacbc840 con 0x5555aacc15d0 2024-03-18T16:18:30.115 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.112+0000 7f23e8c00640 1 --2- 172.21.15.39:0/1996647770 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5555aacbc460 0x5555aaccd190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:18:30.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.112+0000 7f23ea000640 1 --2- 172.21.15.39:0/1996647770 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5555aacc15d0 0x5555aaca4ea0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f23dc00ebb0 tx=0x7f23dc021000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cb4bf5a022a94145 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:18:30.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.113+0000 7f23d5c00640 1 -- 172.21.15.39:0/1996647770 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23dc034070 con 0x5555aacc15d0 2024-03-18T16:18:30.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.113+0000 7f23d5c00640 1 -- 172.21.15.39:0/1996647770 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f23dc02b040 con 0x5555aacc15d0 2024-03-18T16:18:30.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.113+0000 7f23d5c00640 1 -- 172.21.15.39:0/1996647770 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23dc02a040 con 0x5555aacc15d0 2024-03-18T16:18:30.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.114+0000 7f23eedff740 1 -- 172.21.15.39:0/1996647770 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5555aacc15d0 msgr2=0x5555aaca4ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:18:30.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.114+0000 7f23eedff740 1 --2- 172.21.15.39:0/1996647770 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5555aacc15d0 0x5555aaca4ea0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f23dc00ebb0 tx=0x7f23dc021000 comp rx=0 tx=0).stop 2024-03-18T16:18:30.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.114+0000 7f23eedff740 1 -- 172.21.15.39:0/1996647770 shutdown_connections 2024-03-18T16:18:30.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.114+0000 7f23eedff740 1 --2- 172.21.15.39:0/1996647770 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5555aacc15d0 0x5555aaca4ea0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.114+0000 7f23eedff740 1 --2- 172.21.15.39:0/1996647770 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5555aacbc460 0x5555aaccd190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.114+0000 7f23eedff740 1 --2- 172.21.15.39:0/1996647770 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5555aacbba70 0x5555aaca4960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.114+0000 7f23eedff740 1 -- 172.21.15.39:0/1996647770 >> 172.21.15.39:0/1996647770 conn(0x5555aaca5fd0 msgr2=0x5555aacbd820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:18:30.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.114+0000 7f23eedff740 1 -- 172.21.15.39:0/1996647770 shutdown_connections 2024-03-18T16:18:30.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.114+0000 7f23eedff740 1 -- 172.21.15.39:0/1996647770 wait complete. 2024-03-18T16:18:30.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.114+0000 7f23eedff740 1 -- start start 2024-03-18T16:18:30.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.115+0000 7f23eedff740 1 Processor -- start 2024-03-18T16:18:30.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.115+0000 7f23eedff740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5555aacbba70 0x5555aacca470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:30.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.115+0000 7f23eedff740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5555aacbc460 0x5555aacca9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:30.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.115+0000 7f23eedff740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5555aacc15d0 0x5555aaccb920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:30.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.116+0000 7f23e9600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5555aacbba70 0x5555aacca470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:30.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.116+0000 7f23e9600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5555aacbba70 0x5555aacca470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:49232/0 (socket says 172.21.15.39:49232) 2024-03-18T16:18:30.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.116+0000 7f23e9600640 1 -- 172.21.15.39:0/2778905340 learned_addr learned my addr 172.21.15.39:0/2778905340 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:18:30.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.116+0000 7f23eedff740 1 -- 172.21.15.39:0/2778905340 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f23e41b1de0 con 0x5555aacbc460 2024-03-18T16:18:30.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.116+0000 7f23eedff740 1 -- 172.21.15.39:0/2778905340 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f23e41b1710 con 0x5555aacc15d0 2024-03-18T16:18:30.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.116+0000 7f23eedff740 1 -- 172.21.15.39:0/2778905340 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f23e40734a0 con 0x5555aacbba70 2024-03-18T16:18:30.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.116+0000 7f23eedff740 10 client.0 _ll_register_callbacks cb 0x7f23d009b5e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:18:30.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.116+0000 7f23ea000640 1 --2- 172.21.15.39:0/2778905340 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5555aacc15d0 0x5555aaccb920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:30.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.116+0000 7f23e8c00640 1 --2- 172.21.15.39:0/2778905340 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5555aacbc460 0x5555aacca9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:30.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.116+0000 7f23e9600640 1 -- 172.21.15.39:0/2778905340 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5555aacc15d0 msgr2=0x5555aaccb920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:18:30.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.116+0000 7f23e9600640 1 --2- 172.21.15.39:0/2778905340 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5555aacc15d0 0x5555aaccb920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.116+0000 7f23e9600640 1 -- 172.21.15.39:0/2778905340 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5555aacbc460 msgr2=0x5555aacca9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:18:30.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.116+0000 7f23e9600640 1 --2- 172.21.15.39:0/2778905340 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5555aacbc460 0x5555aacca9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.119 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.117+0000 7f23e9600640 1 -- 172.21.15.39:0/2778905340 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5555aaccbf10 con 0x5555aacbba70 2024-03-18T16:18:30.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.117+0000 7f23e8c00640 1 --2- 172.21.15.39:0/2778905340 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5555aacbc460 0x5555aacca9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:18:30.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.117+0000 7f23e9600640 1 --2- 172.21.15.39:0/2778905340 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5555aacbba70 0x5555aacca470 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f23d8009900 tx=0x7f23d8005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=da3d328b66916f2c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:18:30.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.117+0000 7f23caa00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:18:30.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.118+0000 7f23caa00640 1 -- 172.21.15.39:0/2778905340 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23d8005220 con 0x5555aacbba70 2024-03-18T16:18:30.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.118+0000 7f23caa00640 1 -- 172.21.15.39:0/2778905340 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f23d803a040 con 0x5555aacbba70 2024-03-18T16:18:30.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.118+0000 7f23eedff740 1 -- 172.21.15.39:0/2778905340 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f23dc02b040 con 0x5555aacbba70 2024-03-18T16:18:30.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.120+0000 7f23caa00640 1 -- 172.21.15.39:0/2778905340 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f23d805a050 con 0x5555aacbba70 2024-03-18T16:18:30.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.120+0000 7f23eedff740 10 client.27250 fetch_fsmap learned FSMap version 44 2024-03-18T16:18:30.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.120+0000 7f23eedff740 1 -- 172.21.15.39:0/2778905340 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x5555aaccd200 con 0x5555aacbba70 2024-03-18T16:18:30.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.120+0000 7f23caa00640 1 -- 172.21.15.39:0/2778905340 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f23dc02b040 con 0x5555aacbba70 2024-03-18T16:18:30.122 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.121+0000 7f23caa00640 1 -- 172.21.15.39:0/2778905340 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7f23d80046c0 con 0x5555aacbba70 2024-03-18T16:18:30.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.121+0000 7f23eedff740 10 client.27250 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:18:30.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.121+0000 7f23eedff740 10 client.27250 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-18T16:18:30.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.121+0000 7f23eedff740 20 client.27250 populate_metadata read hostname 'smithi039' 2024-03-18T16:18:30.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.121+0000 7f23eedff740 1 --2- 172.21.15.39:0/2778905340 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x5555aad38310 0x5555aad3a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:30.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.121+0000 7f23eedff740 4 client.27250 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:18:30.123 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.121+0000 7f23eedff740 1 -- 172.21.15.39:0/2778905340 --> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5555aac16ce0 con 0x5555aad38310 2024-03-18T16:18:30.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.122+0000 7f23e8c00640 1 --2- 172.21.15.39:0/2778905340 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x5555aad38310 0x5555aad3a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:30.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.123+0000 7f23e8c00640 1 --2- 172.21.15.39:0/2778905340 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x5555aad38310 0x5555aad3a730 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ebeb11f47c27124e server_cookie=4ef2e9fa9ff4297d in_seq=0 out_seq=0 2024-03-18T16:18:30.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.123+0000 7f23caa00640 10 client.27250 ms_handle_connect on v2:172.21.15.196:6836/895247251 2024-03-18T16:18:30.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.123+0000 7f23caa00640 1 -- 172.21.15.39:0/2778905340 <== mds.2 v2:172.21.15.196:6836/895247251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5555aac16ce0 con 0x5555aad38310 2024-03-18T16:18:30.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.123+0000 7f23caa00640 10 client.27250 handle_command_reply: tid=0 2024-03-18T16:18:30.125 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.123+0000 7f23eedff740 1 client.27250 shutdown 2024-03-18T16:18:30.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.124+0000 7f23eedff740 1 -- 172.21.15.39:0/2778905340 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5555aacbba70 msgr2=0x5555aacca470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:18:30.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.124+0000 7f23eedff740 1 --2- 172.21.15.39:0/2778905340 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5555aacbba70 0x5555aacca470 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f23d8009900 tx=0x7f23d8005fd0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.124+0000 7f23eedff740 1 -- 172.21.15.39:0/2778905340 shutdown_connections 2024-03-18T16:18:30.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.124+0000 7f23eedff740 1 --2- 172.21.15.39:0/2778905340 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5555aacc15d0 0x5555aaccb920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.124+0000 7f23eedff740 1 --2- 172.21.15.39:0/2778905340 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x5555aad38310 0x5555aad3a730 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.124+0000 7f23eedff740 1 --2- 172.21.15.39:0/2778905340 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5555aacbc460 0x5555aacca9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.124+0000 7f23eedff740 1 --2- 172.21.15.39:0/2778905340 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5555aacbba70 0x5555aacca470 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.124+0000 7f23caa00640 0 client.27250 ms_handle_reset on v2:172.21.15.196:6836/895247251 2024-03-18T16:18:30.126 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.124+0000 7f23eedff740 1 -- 172.21.15.39:0/2778905340 >> 172.21.15.39:0/2778905340 conn(0x5555aaca5fd0 msgr2=0x5555aaca9230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:18:30.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.125+0000 7f23eedff740 1 -- 172.21.15.39:0/2778905340 shutdown_connections 2024-03-18T16:18:30.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.125+0000 7f23eedff740 1 -- 172.21.15.39:0/2778905340 wait complete. 2024-03-18T16:18:30.127 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.125+0000 7f23eedff740 20 client.27250 trim_cache size 0 max 16384 2024-03-18T16:18:30.129 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:18:30.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.127+0000 7f23eaa00640 1 -- 172.21.15.39:0/921529472 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f23cc0a29d0 msgr2=0x7f23cc0a4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:18:30.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.127+0000 7f23eaa00640 1 --2- 172.21.15.39:0/921529472 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f23cc0a29d0 0x7f23cc0a4e90 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f23dc009ce0 tx=0x7f23dc009200 comp rx=0 tx=0).stop 2024-03-18T16:18:30.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.127+0000 7f23eaa00640 1 -- 172.21.15.39:0/921529472 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f23e41028d0 msgr2=0x7f23e4193380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:18:30.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.127+0000 7f23eaa00640 1 --2- 172.21.15.39:0/921529472 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f23e41028d0 0x7f23e4193380 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f23d000e9f0 tx=0x7f23d000eeb0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.128+0000 7f23eaa00640 1 -- 172.21.15.39:0/921529472 shutdown_connections 2024-03-18T16:18:30.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.128+0000 7f23eaa00640 1 --2- 172.21.15.39:0/921529472 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f23e41028d0 0x7f23e4193380 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.129 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.128+0000 7f23eaa00640 1 --2- 172.21.15.39:0/921529472 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f23cc0a29d0 0x7f23cc0a4e90 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.128+0000 7f23eaa00640 1 --2- 172.21.15.39:0/921529472 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f23e4103290 0x7f23e419a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.128+0000 7f23eaa00640 1 --2- 172.21.15.39:0/921529472 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f23e41016e0 0x7f23e4192e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.130 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.128+0000 7f23eaa00640 1 -- 172.21.15.39:0/921529472 >> 172.21.15.39:0/921529472 conn(0x7f23e40fcc50 msgr2=0x7f23e4103bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:18:30.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.130+0000 7f23eaa00640 1 -- 172.21.15.39:0/921529472 shutdown_connections 2024-03-18T16:18:30.132 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.131+0000 7f23eaa00640 1 -- 172.21.15.39:0/921529472 wait complete. 2024-03-18T16:18:30.156 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-18T16:18:30.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:18:29 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/1871069659' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:18:30.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:18:29 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/1871069659' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:18:30.432 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.430+0000 7f3586e00640 1 -- 172.21.15.39:0/1847787716 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f35800ffdd0 msgr2=0x7f3580106fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:18:30.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.431+0000 7f3586e00640 1 --2- 172.21.15.39:0/1847787716 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f35800ffdd0 0x7f3580106fb0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f356c009900 tx=0x7f356c02bc30 comp rx=0 tx=0).stop 2024-03-18T16:18:30.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.431+0000 7f3586e00640 1 -- 172.21.15.39:0/1847787716 shutdown_connections 2024-03-18T16:18:30.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.431+0000 7f3586e00640 1 --2- 172.21.15.39:0/1847787716 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f35801074f0 0x7f35801098e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.431+0000 7f3586e00640 1 --2- 172.21.15.39:0/1847787716 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f35800ffdd0 0x7f3580106fb0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.431+0000 7f3586e00640 1 --2- 172.21.15.39:0/1847787716 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f35800ff3e0 0x7f35800ff7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.431+0000 7f3586e00640 1 -- 172.21.15.39:0/1847787716 >> 172.21.15.39:0/1847787716 conn(0x7f35800fafc0 msgr2=0x7f35800fd3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:18:30.433 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.431+0000 7f3586e00640 1 -- 172.21.15.39:0/1847787716 shutdown_connections 2024-03-18T16:18:30.434 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.432+0000 7f3586e00640 1 -- 172.21.15.39:0/1847787716 wait complete. 2024-03-18T16:18:30.435 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.433+0000 7f3586e00640 1 Processor -- start 2024-03-18T16:18:30.437 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.435+0000 7f3586e00640 1 -- start start 2024-03-18T16:18:30.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.436+0000 7f3586e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f35800ff3e0 0x7f3580110500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:30.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.436+0000 7f3586e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f35800ffdd0 0x7f3580110a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:30.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.436+0000 7f3586e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f35801074f0 0x7f358010d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:30.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.436+0000 7f3586e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f3580101680 con 0x7f35800ffdd0 2024-03-18T16:18:30.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.436+0000 7f3586e00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f3580101500 con 0x7f35800ff3e0 2024-03-18T16:18:30.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.436+0000 7f3586e00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f3580101800 con 0x7f35801074f0 2024-03-18T16:18:30.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.436+0000 7f3584e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f35801074f0 0x7f358010d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:30.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.436+0000 7f3584e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f35801074f0 0x7f358010d070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:49248/0 (socket says 172.21.15.39:49248) 2024-03-18T16:18:30.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.436+0000 7f3584e00640 1 -- 172.21.15.39:0/2495643835 learned_addr learned my addr 172.21.15.39:0/2495643835 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:18:30.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.436+0000 7f357fe00640 1 --2- 172.21.15.39:0/2495643835 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f35800ff3e0 0x7f3580110500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:30.438 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.436+0000 7f357f400640 1 --2- 172.21.15.39:0/2495643835 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f35800ffdd0 0x7f3580110a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:30.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.436+0000 7f3584e00640 1 -- 172.21.15.39:0/2495643835 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f35800ff3e0 msgr2=0x7f3580110500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:18:30.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.437+0000 7f3584e00640 1 --2- 172.21.15.39:0/2495643835 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f35800ff3e0 0x7f3580110500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.437+0000 7f3584e00640 1 -- 172.21.15.39:0/2495643835 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f35800ffdd0 msgr2=0x7f3580110a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:18:30.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.437+0000 7f3584e00640 1 --2- 172.21.15.39:0/2495643835 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f35800ffdd0 0x7f3580110a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.437+0000 7f3584e00640 1 -- 172.21.15.39:0/2495643835 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f358010d930 con 0x7f35801074f0 2024-03-18T16:18:30.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.437+0000 7f357f400640 1 --2- 172.21.15.39:0/2495643835 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f35800ffdd0 0x7f3580110a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:18:30.439 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.437+0000 7f3584e00640 1 --2- 172.21.15.39:0/2495643835 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f35801074f0 0x7f358010d070 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f357800efd0 tx=0x7f357800c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:18:30.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.438+0000 7f357cc00640 1 -- 172.21.15.39:0/2495643835 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3578003b70 con 0x7f35801074f0 2024-03-18T16:18:30.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.438+0000 7f357cc00640 1 -- 172.21.15.39:0/2495643835 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3578003d10 con 0x7f35801074f0 2024-03-18T16:18:30.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.438+0000 7f357cc00640 1 -- 172.21.15.39:0/2495643835 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35780109f0 con 0x7f35801074f0 2024-03-18T16:18:30.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.438+0000 7f3586e00640 1 -- 172.21.15.39:0/2495643835 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35801a6110 con 0x7f35801074f0 2024-03-18T16:18:30.441 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.438+0000 7f3586e00640 1 -- 172.21.15.39:0/2495643835 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35801a64c0 con 0x7f35801074f0 2024-03-18T16:18:30.445 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.444+0000 7f357cc00640 1 -- 172.21.15.39:0/2495643835 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f3578004020 con 0x7f35801074f0 2024-03-18T16:18:30.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.446+0000 7f357cc00640 1 --2- 172.21.15.39:0/2495643835 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f35640a2950 0x7f35640a4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:30.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.446+0000 7f357cc00640 1 -- 172.21.15.39:0/2495643835 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f35780cf2f0 con 0x7f35801074f0 2024-03-18T16:18:30.448 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.446+0000 7f3589a49740 1 Processor -- start 2024-03-18T16:18:30.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.448+0000 7f3589a49740 1 -- start start 2024-03-18T16:18:30.449 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.448+0000 7f3589a49740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55586bad98c0 0x55586bad9ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:30.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.448+0000 7f3589a49740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55586bada2b0 0x55586badedb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:30.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.448+0000 7f3589a49740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55586badf420 0x55586bae18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:30.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.448+0000 7f3589a49740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55586baa13b0 con 0x55586bad98c0 2024-03-18T16:18:30.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.448+0000 7f3589a49740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55586ba3e450 con 0x55586bada2b0 2024-03-18T16:18:30.450 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.448+0000 7f3589a49740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55586b9e6cb0 con 0x55586badf420 2024-03-18T16:18:30.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.448+0000 7f357fe00640 1 --2- 172.21.15.39:0/2495643835 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f35640a2950 0x7f35640a4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:30.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.449+0000 7f357f400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55586bada2b0 0x55586badedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:30.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.449+0000 7f3584e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55586badf420 0x55586bae18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:30.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.449+0000 7f357f400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55586bada2b0 0x55586badedb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:58616/0 (socket says 172.21.15.39:58616) 2024-03-18T16:18:30.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.449+0000 7f357f400640 1 -- 172.21.15.39:0/1702084658 learned_addr learned my addr 172.21.15.39:0/1702084658 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:18:30.451 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.449+0000 7f357f400640 1 -- 172.21.15.39:0/1702084658 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55586badf420 msgr2=0x55586bae18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:18:30.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.449+0000 7f357f400640 1 --2- 172.21.15.39:0/1702084658 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55586badf420 0x55586bae18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.449+0000 7f357f400640 1 -- 172.21.15.39:0/1702084658 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55586bad98c0 msgr2=0x55586bad9ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:18:30.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.449+0000 7f357f400640 1 --2- 172.21.15.39:0/1702084658 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55586bad98c0 0x55586bad9ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.449+0000 7f357f400640 1 -- 172.21.15.39:0/1702084658 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55586ba081f0 con 0x55586bada2b0 2024-03-18T16:18:30.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.449+0000 7f3584e00640 1 --2- 172.21.15.39:0/1702084658 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55586badf420 0x55586bae18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:18:30.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.449+0000 7f357f400640 1 --2- 172.21.15.39:0/1702084658 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55586bada2b0 0x55586badedb0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f356c000c00 tx=0x7f356c02c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4052c5f9f33d2f32 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:18:30.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.450+0000 7f3575600640 1 -- 172.21.15.39:0/1702084658 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f356c044020 con 0x55586bada2b0 2024-03-18T16:18:30.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.450+0000 7f3575600640 1 -- 172.21.15.39:0/1702084658 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f356c02cb80 con 0x55586bada2b0 2024-03-18T16:18:30.452 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.450+0000 7f3575600640 1 -- 172.21.15.39:0/1702084658 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f356c004920 con 0x55586bada2b0 2024-03-18T16:18:30.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.451+0000 7f3589a49740 1 -- 172.21.15.39:0/1702084658 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55586bada2b0 msgr2=0x55586badedb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:18:30.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.451+0000 7f3589a49740 1 --2- 172.21.15.39:0/1702084658 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55586bada2b0 0x55586badedb0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f356c000c00 tx=0x7f356c02c650 comp rx=0 tx=0).stop 2024-03-18T16:18:30.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.452+0000 7f3589a49740 1 -- 172.21.15.39:0/1702084658 shutdown_connections 2024-03-18T16:18:30.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.452+0000 7f3589a49740 1 --2- 172.21.15.39:0/1702084658 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55586bada2b0 0x55586badedb0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.454 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.452+0000 7f3589a49740 1 --2- 172.21.15.39:0/1702084658 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55586badf420 0x55586bae18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.452+0000 7f3589a49740 1 --2- 172.21.15.39:0/1702084658 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55586bad98c0 0x55586bad9ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.452+0000 7f3589a49740 1 -- 172.21.15.39:0/1702084658 >> 172.21.15.39:0/1702084658 conn(0x55586bad4e30 msgr2=0x55586bad5230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:18:30.455 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.452+0000 7f357fe00640 1 --2- 172.21.15.39:0/2495643835 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f35640a2950 0x7f35640a4e10 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x55586badab10 tx=0x7f35700099c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:18:30.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.454+0000 7f3589a49740 1 -- 172.21.15.39:0/1702084658 shutdown_connections 2024-03-18T16:18:30.457 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.456+0000 7f3589a49740 1 -- 172.21.15.39:0/1702084658 wait complete. 2024-03-18T16:18:30.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.456+0000 7f3589a49740 1 -- start start 2024-03-18T16:18:30.458 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.457+0000 7f3589a49740 1 Processor -- start 2024-03-18T16:18:30.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.457+0000 7f3589a49740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55586bad98c0 0x55586bae3b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:30.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.457+0000 7f3589a49740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55586bada2b0 0x55586bae40b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:30.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.457+0000 7f3589a49740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55586badf420 0x55586baeb0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:30.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.457+0000 7f3589a49740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55586b9fa110 con 0x55586bada2b0 2024-03-18T16:18:30.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.457+0000 7f3589a49740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55586ba3a4f0 con 0x55586badf420 2024-03-18T16:18:30.459 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.457+0000 7f3589a49740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55586bae2370 con 0x55586bad98c0 2024-03-18T16:18:30.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.458+0000 7f3589a49740 10 client.0 _ll_register_callbacks cb 0x55586b9e7b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:18:30.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.458+0000 7f357fe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55586bad98c0 0x55586bae3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:30.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.458+0000 7f3584e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55586badf420 0x55586baeb0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:30.460 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.458+0000 7f357fe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55586bad98c0 0x55586bae3b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:49268/0 (socket says 172.21.15.39:49268) 2024-03-18T16:18:30.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.458+0000 7f357fe00640 1 -- 172.21.15.39:0/1311822698 learned_addr learned my addr 172.21.15.39:0/1311822698 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:18:30.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.458+0000 7f357f400640 1 --2- 172.21.15.39:0/1311822698 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55586bada2b0 0x55586bae40b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:30.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.459+0000 7f3584e00640 1 -- 172.21.15.39:0/1311822698 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55586bad98c0 msgr2=0x55586bae3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:18:30.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.459+0000 7f3584e00640 1 --2- 172.21.15.39:0/1311822698 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55586bad98c0 0x55586bae3b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.459+0000 7f3584e00640 1 -- 172.21.15.39:0/1311822698 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55586bada2b0 msgr2=0x55586bae40b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:18:30.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.459+0000 7f3584e00640 1 --2- 172.21.15.39:0/1311822698 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55586bada2b0 0x55586bae40b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.459+0000 7f3584e00640 1 -- 172.21.15.39:0/1311822698 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55586b7523f0 con 0x55586badf420 2024-03-18T16:18:30.461 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.459+0000 7f3584e00640 1 --2- 172.21.15.39:0/1311822698 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55586badf420 0x55586baeb0f0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f357800cba0 tx=0x7f3578009820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e1fe5241f79f0561 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:18:30.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.459+0000 7f356a000640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:18:30.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.459+0000 7f356a000640 1 -- 172.21.15.39:0/1311822698 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f357809b8a0 con 0x55586badf420 2024-03-18T16:18:30.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.459+0000 7f356a000640 1 -- 172.21.15.39:0/1311822698 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f3578004310 con 0x55586badf420 2024-03-18T16:18:30.462 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.459+0000 7f3589a49740 1 -- 172.21.15.39:0/1311822698 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f356c02cb80 con 0x55586badf420 2024-03-18T16:18:30.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.460+0000 7f356a000640 1 -- 172.21.15.39:0/1311822698 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f357809f9f0 con 0x55586badf420 2024-03-18T16:18:30.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.460+0000 7f3589a49740 10 client.27281 fetch_fsmap learned FSMap version 44 2024-03-18T16:18:30.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.460+0000 7f3589a49740 1 -- 172.21.15.39:0/1311822698 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x55586b81c7d0 con 0x55586badf420 2024-03-18T16:18:30.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.460+0000 7f356a000640 1 -- 172.21.15.39:0/1311822698 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f356c02cb80 con 0x55586badf420 2024-03-18T16:18:30.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.460+0000 7f356a000640 1 -- 172.21.15.39:0/1311822698 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7f357801f020 con 0x55586badf420 2024-03-18T16:18:30.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.460+0000 7f3589a49740 10 client.27281 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:18:30.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.460+0000 7f3589a49740 10 client.27281 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T16:18:30.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.461+0000 7f3589a49740 20 client.27281 populate_metadata read hostname 'smithi039' 2024-03-18T16:18:30.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.461+0000 7f3589a49740 1 --2- 172.21.15.39:0/1311822698 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x55586baefa30 0x55586baf1e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:30.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.461+0000 7f3589a49740 4 client.27281 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:18:30.463 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.461+0000 7f3589a49740 1 -- 172.21.15.39:0/1311822698 --> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55586ba4b990 con 0x55586baefa30 2024-03-18T16:18:30.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.461+0000 7f357fe00640 1 --2- 172.21.15.39:0/1311822698 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x55586baefa30 0x55586baf1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:30.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.463+0000 7f357fe00640 1 --2- 172.21.15.39:0/1311822698 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x55586baefa30 0x55586baf1e40 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9facdc219c9f7729 server_cookie=32e2bc303bd7ef04 in_seq=0 out_seq=0 2024-03-18T16:18:30.464 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.463+0000 7f356a000640 10 client.27281 ms_handle_connect on v2:172.21.15.196:6838/2001427268 2024-03-18T16:18:30.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.464+0000 7f356a000640 1 -- 172.21.15.39:0/1311822698 <== mds.1 v2:172.21.15.196:6838/2001427268 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55586ba4b990 con 0x55586baefa30 2024-03-18T16:18:30.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.464+0000 7f356a000640 10 client.27281 handle_command_reply: tid=0 2024-03-18T16:18:30.466 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.464+0000 7f3589a49740 1 client.27281 shutdown 2024-03-18T16:18:30.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.465+0000 7f3589a49740 1 -- 172.21.15.39:0/1311822698 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55586badf420 msgr2=0x55586baeb0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:18:30.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.465+0000 7f3589a49740 1 --2- 172.21.15.39:0/1311822698 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55586badf420 0x55586baeb0f0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f357800cba0 tx=0x7f3578009820 comp rx=0 tx=0).stop 2024-03-18T16:18:30.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.465+0000 7f3589a49740 1 -- 172.21.15.39:0/1311822698 shutdown_connections 2024-03-18T16:18:30.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.465+0000 7f3589a49740 1 --2- 172.21.15.39:0/1311822698 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x55586baefa30 0x55586baf1e40 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.465+0000 7f3589a49740 1 --2- 172.21.15.39:0/1311822698 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55586badf420 0x55586baeb0f0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.465+0000 7f3589a49740 1 --2- 172.21.15.39:0/1311822698 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55586bada2b0 0x55586bae40b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.465+0000 7f356a000640 0 client.27281 ms_handle_reset on v2:172.21.15.196:6838/2001427268 2024-03-18T16:18:30.467 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.465+0000 7f3589a49740 1 --2- 172.21.15.39:0/1311822698 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55586bad98c0 0x55586bae3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.465+0000 7f3589a49740 1 -- 172.21.15.39:0/1311822698 >> 172.21.15.39:0/1311822698 conn(0x55586bad4e30 msgr2=0x55586bad41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:18:30.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.465+0000 7f3589a49740 1 -- 172.21.15.39:0/1311822698 shutdown_connections 2024-03-18T16:18:30.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.465+0000 7f3589a49740 1 -- 172.21.15.39:0/1311822698 wait complete. 2024-03-18T16:18:30.468 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.466+0000 7f3589a49740 20 client.27281 trim_cache size 0 max 16384 2024-03-18T16:18:30.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.485+0000 7f3589a49740 1 Processor -- start 2024-03-18T16:18:30.487 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.485+0000 7f3589a49740 1 -- start start 2024-03-18T16:18:30.487 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.485+0000 7f3589a49740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55586bad98c0 0x55586bad4070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:30.487 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.485+0000 7f3589a49740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55586bada2b0 0x55586baeb0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:30.487 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.485+0000 7f3589a49740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55586badf420 0x55586baf6700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:30.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.485+0000 7f3589a49740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f35801b09c0 con 0x55586bada2b0 2024-03-18T16:18:30.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.485+0000 7f3589a49740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f35801b2500 con 0x55586badf420 2024-03-18T16:18:30.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.485+0000 7f3589a49740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f35801b1090 con 0x55586bad98c0 2024-03-18T16:18:30.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.486+0000 7f357fe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55586bad98c0 0x55586bad4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:30.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.486+0000 7f357fe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55586bad98c0 0x55586bad4070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:49282/0 (socket says 172.21.15.39:49282) 2024-03-18T16:18:30.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.486+0000 7f357fe00640 1 -- 172.21.15.39:0/3879586540 learned_addr learned my addr 172.21.15.39:0/3879586540 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:18:30.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.486+0000 7f3584e00640 1 --2- 172.21.15.39:0/3879586540 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55586badf420 0x55586baf6700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:30.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.486+0000 7f357f400640 1 --2- 172.21.15.39:0/3879586540 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55586bada2b0 0x55586baeb0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:30.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.487+0000 7f357f400640 1 -- 172.21.15.39:0/3879586540 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55586bad98c0 msgr2=0x55586bad4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:18:30.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.487+0000 7f357f400640 1 --2- 172.21.15.39:0/3879586540 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55586bad98c0 0x55586bad4070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.487+0000 7f357f400640 1 -- 172.21.15.39:0/3879586540 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55586badf420 msgr2=0x55586baf6700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:18:30.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.487+0000 7f357f400640 1 --2- 172.21.15.39:0/3879586540 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55586badf420 0x55586baf6700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.487+0000 7f357f400640 1 -- 172.21.15.39:0/3879586540 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55586bad0c10 con 0x55586bada2b0 2024-03-18T16:18:30.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.487+0000 7f3584e00640 1 --2- 172.21.15.39:0/3879586540 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55586badf420 0x55586baf6700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:18:30.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.488+0000 7f357f400640 1 --2- 172.21.15.39:0/3879586540 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55586bada2b0 0x55586baeb0b0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x55586bad5da0 tx=0x7f356c005c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f13d6496a63be097 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:18:30.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.488+0000 7f3574c00640 1 -- 172.21.15.39:0/3879586540 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f356c04d070 con 0x55586bada2b0 2024-03-18T16:18:30.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.488+0000 7f3574c00640 1 -- 172.21.15.39:0/3879586540 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f356c005040 con 0x55586bada2b0 2024-03-18T16:18:30.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.488+0000 7f3574c00640 1 -- 172.21.15.39:0/3879586540 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f356c044040 con 0x55586bada2b0 2024-03-18T16:18:30.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.489+0000 7f3589a49740 1 -- 172.21.15.39:0/3879586540 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55586bada2b0 msgr2=0x55586baeb0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:18:30.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.489+0000 7f3589a49740 1 --2- 172.21.15.39:0/3879586540 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55586bada2b0 0x55586baeb0b0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x55586bad5da0 tx=0x7f356c005c00 comp rx=0 tx=0).stop 2024-03-18T16:18:30.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.491+0000 7f3589a49740 1 -- 172.21.15.39:0/3879586540 shutdown_connections 2024-03-18T16:18:30.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.491+0000 7f3589a49740 1 --2- 172.21.15.39:0/3879586540 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55586badf420 0x55586baf6700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.491+0000 7f3589a49740 1 --2- 172.21.15.39:0/3879586540 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55586bada2b0 0x55586baeb0b0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.491+0000 7f3589a49740 1 --2- 172.21.15.39:0/3879586540 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55586bad98c0 0x55586bad4070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.491+0000 7f3589a49740 1 -- 172.21.15.39:0/3879586540 >> 172.21.15.39:0/3879586540 conn(0x55586bad4e30 msgr2=0x55586bad3960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:18:30.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.492+0000 7f3589a49740 1 -- 172.21.15.39:0/3879586540 shutdown_connections 2024-03-18T16:18:30.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.492+0000 7f3589a49740 1 -- 172.21.15.39:0/3879586540 wait complete. 2024-03-18T16:18:30.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.492+0000 7f3589a49740 1 -- start start 2024-03-18T16:18:30.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.493+0000 7f3589a49740 1 Processor -- start 2024-03-18T16:18:30.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.494+0000 7f3589a49740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55586bad98c0 0x55586bae89b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:30.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.494+0000 7f3589a49740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55586bada2b0 0x55586bae8ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:30.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.494+0000 7f3589a49740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55586badf420 0x55586bae9f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:30.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.494+0000 7f357f400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55586bada2b0 0x55586bae8ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:30.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.494+0000 7f357f400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55586bada2b0 0x55586bae8ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:58644/0 (socket says 172.21.15.39:58644) 2024-03-18T16:18:30.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.494+0000 7f357f400640 1 -- 172.21.15.39:0/1063802389 learned_addr learned my addr 172.21.15.39:0/1063802389 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:18:30.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.495+0000 7f3589a49740 1 -- 172.21.15.39:0/1063802389 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f35801b1e30 con 0x55586badf420 2024-03-18T16:18:30.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.495+0000 7f3589a49740 1 -- 172.21.15.39:0/1063802389 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f35801b1760 con 0x55586bada2b0 2024-03-18T16:18:30.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.495+0000 7f3589a49740 1 -- 172.21.15.39:0/1063802389 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f35801ae9a0 con 0x55586bad98c0 2024-03-18T16:18:30.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.495+0000 7f3584e00640 1 --2- 172.21.15.39:0/1063802389 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55586badf420 0x55586bae9f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:30.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.495+0000 7f3589a49740 10 client.0 _ll_register_callbacks cb 0x7f3578009b40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:18:30.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.495+0000 7f357f400640 1 -- 172.21.15.39:0/1063802389 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55586bad98c0 msgr2=0x55586bae89b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:18:30.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.495+0000 7f357f400640 1 --2- 172.21.15.39:0/1063802389 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55586bad98c0 0x55586bae89b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.495+0000 7f357f400640 1 -- 172.21.15.39:0/1063802389 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55586badf420 msgr2=0x55586bae9f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:18:30.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.495+0000 7f357f400640 1 --2- 172.21.15.39:0/1063802389 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55586badf420 0x55586bae9f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.495+0000 7f357f400640 1 -- 172.21.15.39:0/1063802389 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55586bae24b0 con 0x55586bada2b0 2024-03-18T16:18:30.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.496+0000 7f357f400640 1 --2- 172.21.15.39:0/1063802389 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55586bada2b0 0x55586bae8ef0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f356c000c00 tx=0x7f356c0058b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec4390cadea33e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:18:30.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.496+0000 7f356a000640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:18:30.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.496+0000 7f356a000640 1 -- 172.21.15.39:0/1063802389 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f356c044070 con 0x55586bada2b0 2024-03-18T16:18:30.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.496+0000 7f356a000640 1 -- 172.21.15.39:0/1063802389 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f356c004c50 con 0x55586bada2b0 2024-03-18T16:18:30.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.496+0000 7f3589a49740 1 -- 172.21.15.39:0/1063802389 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f356c005040 con 0x55586bada2b0 2024-03-18T16:18:30.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.497+0000 7f356a000640 1 -- 172.21.15.39:0/1063802389 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f356c04d040 con 0x55586bada2b0 2024-03-18T16:18:30.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.497+0000 7f3589a49740 10 client.27287 fetch_fsmap learned FSMap version 44 2024-03-18T16:18:30.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.497+0000 7f3589a49740 1 -- 172.21.15.39:0/1063802389 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x55586bb55da0 con 0x55586bada2b0 2024-03-18T16:18:30.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.498+0000 7f356a000640 1 -- 172.21.15.39:0/1063802389 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f356c005040 con 0x55586bada2b0 2024-03-18T16:18:30.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.498+0000 7f356a000640 1 -- 172.21.15.39:0/1063802389 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7f356c052450 con 0x55586bada2b0 2024-03-18T16:18:30.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.498+0000 7f3589a49740 10 client.27287 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:18:30.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.498+0000 7f3589a49740 10 client.27287 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T16:18:30.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.498+0000 7f3589a49740 20 client.27287 populate_metadata read hostname 'smithi039' 2024-03-18T16:18:30.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.498+0000 7f3589a49740 1 --2- 172.21.15.39:0/1063802389 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x55586bb56200 0x55586bb58620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:30.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.498+0000 7f3589a49740 4 client.27287 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:18:30.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.498+0000 7f3589a49740 1 -- 172.21.15.39:0/1063802389 --> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55586ba45ce0 con 0x55586bb56200 2024-03-18T16:18:30.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.498+0000 7f357fe00640 1 --2- 172.21.15.39:0/1063802389 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x55586bb56200 0x55586bb58620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:30.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.499+0000 7f356a000640 10 client.27287 ms_handle_connect on v2:172.21.15.196:6838/2001427268 2024-03-18T16:18:30.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.499+0000 7f357fe00640 1 --2- 172.21.15.39:0/1063802389 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x55586bb56200 0x55586bb58620 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=585b4977c0fd05eb server_cookie=7240ab5a1e625374 in_seq=0 out_seq=0 2024-03-18T16:18:30.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.500+0000 7f356a000640 1 -- 172.21.15.39:0/1063802389 <== mds.1 v2:172.21.15.196:6838/2001427268 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55586ba45ce0 con 0x55586bb56200 2024-03-18T16:18:30.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.500+0000 7f356a000640 10 client.27287 handle_command_reply: tid=0 2024-03-18T16:18:30.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.500+0000 7f3589a49740 1 client.27287 shutdown 2024-03-18T16:18:30.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.500+0000 7f3589a49740 1 -- 172.21.15.39:0/1063802389 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55586bada2b0 msgr2=0x55586bae8ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:18:30.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.500+0000 7f3589a49740 1 --2- 172.21.15.39:0/1063802389 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55586bada2b0 0x55586bae8ef0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f356c000c00 tx=0x7f356c0058b0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.500+0000 7f3589a49740 1 -- 172.21.15.39:0/1063802389 shutdown_connections 2024-03-18T16:18:30.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.500+0000 7f3589a49740 1 --2- 172.21.15.39:0/1063802389 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x55586bb56200 0x55586bb58620 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.500+0000 7f3589a49740 1 --2- 172.21.15.39:0/1063802389 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55586bada2b0 0x55586bae8ef0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.500+0000 7f3589a49740 1 --2- 172.21.15.39:0/1063802389 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55586badf420 0x55586bae9f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.500+0000 7f3589a49740 1 --2- 172.21.15.39:0/1063802389 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55586bad98c0 0x55586bae89b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.500+0000 7f3589a49740 1 -- 172.21.15.39:0/1063802389 >> 172.21.15.39:0/1063802389 conn(0x55586bad4e30 msgr2=0x55586bad41f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:18:30.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.500+0000 7f356a000640 0 client.27287 ms_handle_reset on v2:172.21.15.196:6838/2001427268 2024-03-18T16:18:30.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.501+0000 7f3589a49740 1 -- 172.21.15.39:0/1063802389 shutdown_connections 2024-03-18T16:18:30.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.501+0000 7f3589a49740 1 -- 172.21.15.39:0/1063802389 wait complete. 2024-03-18T16:18:30.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.501+0000 7f3589a49740 20 client.27287 trim_cache size 0 max 16384 2024-03-18T16:18:30.504 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:18:30.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.503+0000 7f3586e00640 1 -- 172.21.15.39:0/2495643835 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f35640a2950 msgr2=0x7f35640a4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:18:30.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.503+0000 7f3586e00640 1 --2- 172.21.15.39:0/2495643835 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f35640a2950 0x7f35640a4e10 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x55586badab10 tx=0x7f35700099c0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.503+0000 7f3586e00640 1 -- 172.21.15.39:0/2495643835 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f35801074f0 msgr2=0x7f358010d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:18:30.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.503+0000 7f3586e00640 1 --2- 172.21.15.39:0/2495643835 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f35801074f0 0x7f358010d070 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f357800efd0 tx=0x7f357800c6a0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.504+0000 7f3586e00640 1 -- 172.21.15.39:0/2495643835 shutdown_connections 2024-03-18T16:18:30.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.504+0000 7f3586e00640 1 --2- 172.21.15.39:0/2495643835 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f35640a2950 0x7f35640a4e10 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.504+0000 7f3586e00640 1 --2- 172.21.15.39:0/2495643835 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f35801074f0 0x7f358010d070 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.504+0000 7f3586e00640 1 --2- 172.21.15.39:0/2495643835 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f35800ffdd0 0x7f3580110a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.504+0000 7f3586e00640 1 --2- 172.21.15.39:0/2495643835 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f35800ff3e0 0x7f3580110500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.504+0000 7f3586e00640 1 -- 172.21.15.39:0/2495643835 >> 172.21.15.39:0/2495643835 conn(0x7f35800fafc0 msgr2=0x7f3580102050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:18:30.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.504+0000 7f3586e00640 1 -- 172.21.15.39:0/2495643835 shutdown_connections 2024-03-18T16:18:30.506 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.504+0000 7f3586e00640 1 -- 172.21.15.39:0/2495643835 wait complete. 2024-03-18T16:18:30.534 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-18T16:18:30.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.815+0000 7f6204000640 1 -- 172.21.15.39:0/2477575949 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f61fc109930 msgr2=0x7f61fc110250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:18:30.816 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.815+0000 7f6204000640 1 --2- 172.21.15.39:0/2477575949 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f61fc109930 0x7f61fc110250 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f61f4009be0 tx=0x7f61f402bef0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.815+0000 7f6204000640 1 -- 172.21.15.39:0/2477575949 shutdown_connections 2024-03-18T16:18:30.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.815+0000 7f6204000640 1 --2- 172.21.15.39:0/2477575949 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f61fc109930 0x7f61fc110250 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.815+0000 7f6204000640 1 --2- 172.21.15.39:0/2477575949 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f61fc108f70 0x7f61fc1093f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.815+0000 7f6204000640 1 --2- 172.21.15.39:0/2477575949 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f61fc107e20 0x7f61fc108240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.815+0000 7f6204000640 1 -- 172.21.15.39:0/2477575949 >> 172.21.15.39:0/2477575949 conn(0x7f61fc076dd0 msgr2=0x7f61fc0791f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:18:30.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.816+0000 7f6204000640 1 -- 172.21.15.39:0/2477575949 shutdown_connections 2024-03-18T16:18:30.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.816+0000 7f6204000640 1 -- 172.21.15.39:0/2477575949 wait complete. 2024-03-18T16:18:30.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.817+0000 7f6204000640 1 Processor -- start 2024-03-18T16:18:30.818 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.817+0000 7f6204000640 1 -- start start 2024-03-18T16:18:30.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.817+0000 7f6204000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f61fc107e20 0x7f61fc19e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:30.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.817+0000 7f6204000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f61fc108f70 0x7f61fc19e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:30.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.817+0000 7f6204000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f61fc109930 0x7f61fc1a5620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:30.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.817+0000 7f6204000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f61fc113af0 con 0x7f61fc108f70 2024-03-18T16:18:30.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.818+0000 7f6204000640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f61fc113970 con 0x7f61fc107e20 2024-03-18T16:18:30.819 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.818+0000 7f6204000640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f61fc113c70 con 0x7f61fc109930 2024-03-18T16:18:30.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.818+0000 7f6201600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f61fc107e20 0x7f61fc19e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:30.820 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.818+0000 7f6201600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f61fc107e20 0x7f61fc19e070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:58652/0 (socket says 172.21.15.39:58652) 2024-03-18T16:18:30.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.818+0000 7f6201600640 1 -- 172.21.15.39:0/2447157156 learned_addr learned my addr 172.21.15.39:0/2447157156 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:18:30.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.818+0000 7f6200c00640 1 --2- 172.21.15.39:0/2447157156 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f61fc108f70 0x7f61fc19e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:30.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.819+0000 7f6201600640 1 -- 172.21.15.39:0/2447157156 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f61fc109930 msgr2=0x7f61fc1a5620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:18:30.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.819+0000 7f6201600640 1 --2- 172.21.15.39:0/2447157156 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f61fc109930 0x7f61fc1a5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.819+0000 7f6201600640 1 -- 172.21.15.39:0/2447157156 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f61fc108f70 msgr2=0x7f61fc19e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:18:30.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.819+0000 7f6201600640 1 --2- 172.21.15.39:0/2447157156 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f61fc108f70 0x7f61fc19e5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.819+0000 7f6201600640 1 -- 172.21.15.39:0/2447157156 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61fc1a5c90 con 0x7f61fc107e20 2024-03-18T16:18:30.821 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.819+0000 7f6201600640 1 --2- 172.21.15.39:0/2447157156 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f61fc107e20 0x7f61fc19e070 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f61f000d950 tx=0x7f61f000de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:18:30.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.819+0000 7f61fa000640 1 -- 172.21.15.39:0/2447157156 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61f0017070 con 0x7f61fc107e20 2024-03-18T16:18:30.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.819+0000 7f61fa000640 1 -- 172.21.15.39:0/2447157156 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f61f0009c30 con 0x7f61fc107e20 2024-03-18T16:18:30.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.819+0000 7f61fa000640 1 -- 172.21.15.39:0/2447157156 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61f000f690 con 0x7f61fc107e20 2024-03-18T16:18:30.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.819+0000 7f6204000640 1 -- 172.21.15.39:0/2447157156 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61fc1a5f80 con 0x7f61fc107e20 2024-03-18T16:18:30.822 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.820+0000 7f6204000640 1 -- 172.21.15.39:0/2447157156 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61fc1a63f0 con 0x7f61fc107e20 2024-03-18T16:18:30.826 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.825+0000 7f6206c50740 1 Processor -- start 2024-03-18T16:18:30.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.825+0000 7f6206c50740 1 -- start start 2024-03-18T16:18:30.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.826+0000 7f6206c50740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557d7b9138c0 0x557d7b913ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:30.827 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.826+0000 7f6206c50740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557d7b9142b0 0x557d7b918db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:30.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.826+0000 7f6206c50740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557d7b919420 0x557d7b91b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:30.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.826+0000 7f6206c50740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x557d7b8db3b0 con 0x557d7b9142b0 2024-03-18T16:18:30.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.826+0000 7f6206c50740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x557d7b878450 con 0x557d7b9138c0 2024-03-18T16:18:30.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.826+0000 7f6206c50740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x557d7b820cb0 con 0x557d7b919420 2024-03-18T16:18:30.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.826+0000 7f6202000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557d7b9138c0 0x557d7b913ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:30.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.826+0000 7f6202000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557d7b9138c0 0x557d7b913ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:58660/0 (socket says 172.21.15.39:58660) 2024-03-18T16:18:30.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.826+0000 7f6202000640 1 -- 172.21.15.39:0/422527403 learned_addr learned my addr 172.21.15.39:0/422527403 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:18:30.828 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.826+0000 7f6200c00640 1 --2- 172.21.15.39:0/422527403 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557d7b919420 0x557d7b91b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:30.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.827+0000 7f6202000640 1 -- 172.21.15.39:0/422527403 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557d7b919420 msgr2=0x557d7b91b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:18:30.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.827+0000 7f6202000640 1 --2- 172.21.15.39:0/422527403 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557d7b919420 0x557d7b91b8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.827+0000 7f6202000640 1 -- 172.21.15.39:0/422527403 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557d7b9142b0 msgr2=0x557d7b918db0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:18:30.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.827+0000 7f6202000640 1 --2- 172.21.15.39:0/422527403 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557d7b9142b0 0x557d7b918db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.829 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.827+0000 7f6202000640 1 -- 172.21.15.39:0/422527403 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557d7b8421f0 con 0x557d7b9138c0 2024-03-18T16:18:30.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.827+0000 7f61fa000640 1 -- 172.21.15.39:0/2447157156 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f61f00109d0 con 0x7f61fc107e20 2024-03-18T16:18:30.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.828+0000 7f6202000640 1 --2- 172.21.15.39:0/422527403 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557d7b9138c0 0x557d7b913ce0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f61f4005ee0 tx=0x7f61f40092a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d257771b8f3ee8b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:18:30.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.828+0000 7f61ee600640 1 -- 172.21.15.39:0/422527403 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61f4042020 con 0x557d7b9138c0 2024-03-18T16:18:30.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.828+0000 7f61ee600640 1 -- 172.21.15.39:0/422527403 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f61f4033580 con 0x557d7b9138c0 2024-03-18T16:18:30.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.828+0000 7f61ee600640 1 -- 172.21.15.39:0/422527403 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61f4003e60 con 0x557d7b9138c0 2024-03-18T16:18:30.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.829+0000 7f6206c50740 1 -- 172.21.15.39:0/422527403 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557d7b9138c0 msgr2=0x557d7b913ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:18:30.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.829+0000 7f6206c50740 1 --2- 172.21.15.39:0/422527403 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557d7b9138c0 0x557d7b913ce0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f61f4005ee0 tx=0x7f61f40092a0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.829+0000 7f6206c50740 1 -- 172.21.15.39:0/422527403 shutdown_connections 2024-03-18T16:18:30.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.829+0000 7f6206c50740 1 --2- 172.21.15.39:0/422527403 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557d7b919420 0x557d7b91b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.829+0000 7f6206c50740 1 --2- 172.21.15.39:0/422527403 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557d7b9142b0 0x557d7b918db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.829+0000 7f6206c50740 1 --2- 172.21.15.39:0/422527403 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557d7b9138c0 0x557d7b913ce0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.829+0000 7f6206c50740 1 -- 172.21.15.39:0/422527403 >> 172.21.15.39:0/422527403 conn(0x557d7b90ee30 msgr2=0x557d7b90f230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:18:30.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.830+0000 7f6206c50740 1 -- 172.21.15.39:0/422527403 shutdown_connections 2024-03-18T16:18:30.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.830+0000 7f6206c50740 1 -- 172.21.15.39:0/422527403 wait complete. 2024-03-18T16:18:30.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.830+0000 7f61fa000640 1 --2- 172.21.15.39:0/2447157156 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f61e40a2790 0x7f61e40a4c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:30.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.830+0000 7f61fa000640 1 -- 172.21.15.39:0/2447157156 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f61f00ceb80 con 0x7f61fc107e20 2024-03-18T16:18:30.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.830+0000 7f6206c50740 1 -- start start 2024-03-18T16:18:30.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.830+0000 7f6200c00640 1 --2- 172.21.15.39:0/2447157156 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f61e40a2790 0x7f61e40a4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:30.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.831+0000 7f6206c50740 1 Processor -- start 2024-03-18T16:18:30.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.831+0000 7f6206c50740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557d7b9138c0 0x557d7b91dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:30.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.831+0000 7f6206c50740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557d7b9142b0 0x557d7b91e110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:30.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.831+0000 7f6206c50740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557d7b919420 0x557d7b925150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:30.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.831+0000 7f6206c50740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x557d7b834110 con 0x557d7b919420 2024-03-18T16:18:30.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.831+0000 7f6206c50740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x557d7b8744f0 con 0x557d7b9138c0 2024-03-18T16:18:30.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.831+0000 7f6206c50740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x557d7b91c370 con 0x557d7b9142b0 2024-03-18T16:18:30.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.831+0000 7f6200c00640 1 --2- 172.21.15.39:0/2447157156 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f61e40a2790 0x7f61e40a4c50 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x557d7b90fda0 tx=0x7f61e8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:18:30.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.832+0000 7f6206c50740 10 client.0 _ll_register_callbacks cb 0x557d7b821b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:18:30.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.832+0000 7f6201600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557d7b9138c0 0x557d7b91dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:30.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.832+0000 7f6201600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557d7b9138c0 0x557d7b91dbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:58674/0 (socket says 172.21.15.39:58674) 2024-03-18T16:18:30.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.832+0000 7f6201600640 1 -- 172.21.15.39:0/4228040979 learned_addr learned my addr 172.21.15.39:0/4228040979 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:18:30.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.832+0000 7f6202000640 1 --2- 172.21.15.39:0/4228040979 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557d7b919420 0x557d7b925150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:30.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.832+0000 7f6200c00640 1 --2- 172.21.15.39:0/4228040979 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557d7b9142b0 0x557d7b91e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:30.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.832+0000 7f6201600640 1 -- 172.21.15.39:0/4228040979 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557d7b9142b0 msgr2=0x557d7b91e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:18:30.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.832+0000 7f6201600640 1 --2- 172.21.15.39:0/4228040979 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557d7b9142b0 0x557d7b91e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.832+0000 7f6201600640 1 -- 172.21.15.39:0/4228040979 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557d7b919420 msgr2=0x557d7b925150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:18:30.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.832+0000 7f6201600640 1 --2- 172.21.15.39:0/4228040979 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557d7b919420 0x557d7b925150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.832+0000 7f6201600640 1 -- 172.21.15.39:0/4228040979 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557d7b58c3f0 con 0x557d7b9138c0 2024-03-18T16:18:30.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.832+0000 7f6202000640 1 --2- 172.21.15.39:0/4228040979 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557d7b919420 0x557d7b925150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:18:30.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.833+0000 7f6201600640 1 --2- 172.21.15.39:0/4228040979 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557d7b9138c0 0x557d7b91dbd0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f61f0009610 tx=0x7f61f009ab90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fdd9d24dd7988b12 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:18:30.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.833+0000 7f61e2a00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:18:30.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.833+0000 7f61e2a00640 1 -- 172.21.15.39:0/4228040979 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61f0020040 con 0x557d7b9138c0 2024-03-18T16:18:30.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.833+0000 7f61e2a00640 1 -- 172.21.15.39:0/4228040979 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f61f00977a0 con 0x557d7b9138c0 2024-03-18T16:18:30.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.833+0000 7f6206c50740 1 -- 172.21.15.39:0/4228040979 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f61f4033580 con 0x557d7b9138c0 2024-03-18T16:18:30.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.833+0000 7f61e2a00640 1 -- 172.21.15.39:0/4228040979 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f61f009f280 con 0x557d7b9138c0 2024-03-18T16:18:30.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.833+0000 7f6206c50740 10 client.27299 fetch_fsmap learned FSMap version 44 2024-03-18T16:18:30.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.833+0000 7f6206c50740 1 -- 172.21.15.39:0/4228040979 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x557d7b6567d0 con 0x557d7b9138c0 2024-03-18T16:18:30.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.834+0000 7f61e2a00640 1 -- 172.21.15.39:0/4228040979 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f61f4033580 con 0x557d7b9138c0 2024-03-18T16:18:30.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.834+0000 7f61e2a00640 1 -- 172.21.15.39:0/4228040979 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7f61f009a0f0 con 0x557d7b9138c0 2024-03-18T16:18:30.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.834+0000 7f6206c50740 10 client.27299 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:18:30.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.834+0000 7f6206c50740 10 client.27299 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:18:30.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.834+0000 7f6206c50740 20 client.27299 populate_metadata read hostname 'smithi039' 2024-03-18T16:18:30.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.835+0000 7f6206c50740 1 --2- 172.21.15.39:0/4228040979 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x557d7b929980 0x557d7b92bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:30.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.835+0000 7f6206c50740 4 client.27299 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:18:30.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.835+0000 7f6206c50740 1 -- 172.21.15.39:0/4228040979 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557d7b885990 con 0x557d7b929980 2024-03-18T16:18:30.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.836+0000 7f6200c00640 1 --2- 172.21.15.39:0/4228040979 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x557d7b929980 0x557d7b92bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:30.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.837+0000 7f6200c00640 1 --2- 172.21.15.39:0/4228040979 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x557d7b929980 0x557d7b92bdb0 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5d6f5e6cff8478f2 server_cookie=a03298ebb2e36816 in_seq=0 out_seq=0 2024-03-18T16:18:30.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.837+0000 7f61e2a00640 10 client.27299 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:18:30.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.840+0000 7f61e2a00640 1 -- 172.21.15.39:0/4228040979 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x557d7b885990 con 0x557d7b929980 2024-03-18T16:18:30.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.840+0000 7f61e2a00640 10 client.27299 handle_command_reply: tid=0 2024-03-18T16:18:30.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.840+0000 7f6206c50740 1 client.27299 shutdown 2024-03-18T16:18:30.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.840+0000 7f6206c50740 1 -- 172.21.15.39:0/4228040979 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557d7b9138c0 msgr2=0x557d7b91dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:18:30.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.840+0000 7f6206c50740 1 --2- 172.21.15.39:0/4228040979 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557d7b9138c0 0x557d7b91dbd0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f61f0009610 tx=0x7f61f009ab90 comp rx=0 tx=0).stop 2024-03-18T16:18:30.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.841+0000 7f6206c50740 1 -- 172.21.15.39:0/4228040979 shutdown_connections 2024-03-18T16:18:30.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.841+0000 7f6206c50740 1 --2- 172.21.15.39:0/4228040979 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x557d7b929980 0x557d7b92bdb0 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.841+0000 7f6206c50740 1 --2- 172.21.15.39:0/4228040979 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557d7b919420 0x557d7b925150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.841+0000 7f6206c50740 1 --2- 172.21.15.39:0/4228040979 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557d7b9142b0 0x557d7b91e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.841+0000 7f6206c50740 1 --2- 172.21.15.39:0/4228040979 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557d7b9138c0 0x557d7b91dbd0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.841+0000 7f6206c50740 1 -- 172.21.15.39:0/4228040979 >> 172.21.15.39:0/4228040979 conn(0x557d7b90ee30 msgr2=0x557d7b90e1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:18:30.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.841+0000 7f61e2a00640 0 client.27299 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:18:30.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.841+0000 7f6206c50740 1 -- 172.21.15.39:0/4228040979 shutdown_connections 2024-03-18T16:18:30.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.841+0000 7f6206c50740 1 -- 172.21.15.39:0/4228040979 wait complete. 2024-03-18T16:18:30.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.841+0000 7f6206c50740 20 client.27299 trim_cache size 0 max 16384 2024-03-18T16:18:30.867 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.865+0000 7f6206c50740 1 Processor -- start 2024-03-18T16:18:30.868 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.866+0000 7f6206c50740 1 -- start start 2024-03-18T16:18:30.868 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.867+0000 7f6206c50740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557d7b9138c0 0x557d7b91e070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:30.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.867+0000 7f6206c50740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557d7b9142b0 0x557d7b9250e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:30.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.868+0000 7f6206c50740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557d7b919420 0x557d7b90e070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:30.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.868+0000 7f6206c50740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f61fc1b0a50 con 0x557d7b9138c0 2024-03-18T16:18:30.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.868+0000 7f6206c50740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f61fc1b2590 con 0x557d7b919420 2024-03-18T16:18:30.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.868+0000 7f6206c50740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f61fc1b1120 con 0x557d7b9142b0 2024-03-18T16:18:30.870 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.868+0000 7f6201600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557d7b9138c0 0x557d7b91e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:30.870 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.868+0000 7f6200c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557d7b9142b0 0x557d7b9250e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:30.870 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.868+0000 7f6201600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557d7b9138c0 0x557d7b91e070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:48560/0 (socket says 172.21.15.39:48560) 2024-03-18T16:18:30.870 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.868+0000 7f6200c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557d7b9142b0 0x557d7b9250e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:49328/0 (socket says 172.21.15.39:49328) 2024-03-18T16:18:30.871 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.868+0000 7f6200c00640 1 -- 172.21.15.39:0/4132365742 learned_addr learned my addr 172.21.15.39:0/4132365742 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:18:30.871 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.868+0000 7f6202000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557d7b919420 0x557d7b90e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:30.871 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.869+0000 7f6200c00640 1 -- 172.21.15.39:0/4132365742 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557d7b919420 msgr2=0x557d7b90e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:18:30.871 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.869+0000 7f6200c00640 1 --2- 172.21.15.39:0/4132365742 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557d7b919420 0x557d7b90e070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.871 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.869+0000 7f6200c00640 1 -- 172.21.15.39:0/4132365742 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557d7b9138c0 msgr2=0x557d7b91e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:18:30.871 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.869+0000 7f6200c00640 1 --2- 172.21.15.39:0/4132365742 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557d7b9138c0 0x557d7b91e070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.871 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.869+0000 7f6200c00640 1 -- 172.21.15.39:0/4132365742 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557d7b90ac10 con 0x557d7b9142b0 2024-03-18T16:18:30.872 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.869+0000 7f6200c00640 1 --2- 172.21.15.39:0/4132365742 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557d7b9142b0 0x557d7b9250e0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f61fc107b40 tx=0x7f61e8016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=447baa4a13e4643c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:18:30.872 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.870+0000 7f61edc00640 1 -- 172.21.15.39:0/4132365742 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61e801fd00 con 0x557d7b9142b0 2024-03-18T16:18:30.872 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.870+0000 7f6201600640 1 --2- 172.21.15.39:0/4132365742 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557d7b9138c0 0x557d7b91e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:18:30.872 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.870+0000 7f61edc00640 1 -- 172.21.15.39:0/4132365742 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f61e802f070 con 0x557d7b9142b0 2024-03-18T16:18:30.872 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.870+0000 7f61edc00640 1 -- 172.21.15.39:0/4132365742 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61e801cdf0 con 0x557d7b9142b0 2024-03-18T16:18:30.872 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.871+0000 7f6206c50740 1 -- 172.21.15.39:0/4132365742 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557d7b9142b0 msgr2=0x557d7b9250e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:18:30.872 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.871+0000 7f6206c50740 1 --2- 172.21.15.39:0/4132365742 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557d7b9142b0 0x557d7b9250e0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f61fc107b40 tx=0x7f61e8016000 comp rx=0 tx=0).stop 2024-03-18T16:18:30.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.872+0000 7f6206c50740 1 -- 172.21.15.39:0/4132365742 shutdown_connections 2024-03-18T16:18:30.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.872+0000 7f6206c50740 1 --2- 172.21.15.39:0/4132365742 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557d7b919420 0x557d7b90e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.872+0000 7f6206c50740 1 --2- 172.21.15.39:0/4132365742 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557d7b9142b0 0x557d7b9250e0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.872+0000 7f6206c50740 1 --2- 172.21.15.39:0/4132365742 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557d7b9138c0 0x557d7b91e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.872+0000 7f6206c50740 1 -- 172.21.15.39:0/4132365742 >> 172.21.15.39:0/4132365742 conn(0x557d7b90ee30 msgr2=0x557d7b90d960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:18:30.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.874+0000 7f6206c50740 1 -- 172.21.15.39:0/4132365742 shutdown_connections 2024-03-18T16:18:30.875 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.874+0000 7f6206c50740 1 -- 172.21.15.39:0/4132365742 wait complete. 2024-03-18T16:18:30.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.874+0000 7f6206c50740 1 -- start start 2024-03-18T16:18:30.876 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.875+0000 7f6206c50740 1 Processor -- start 2024-03-18T16:18:30.877 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.875+0000 7f6206c50740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557d7b9138c0 0x557d7b9228b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:30.877 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.875+0000 7f6206c50740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557d7b9142b0 0x557d7b922df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:30.877 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.875+0000 7f6206c50740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557d7b919420 0x557d7b923d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:30.877 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.875+0000 7f6206c50740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f61fc1b1ec0 con 0x557d7b9142b0 2024-03-18T16:18:30.877 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.875+0000 7f6206c50740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f61fc1b17f0 con 0x557d7b9138c0 2024-03-18T16:18:30.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.875+0000 7f6206c50740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f61fc1aea30 con 0x557d7b919420 2024-03-18T16:18:30.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.876+0000 7f6206c50740 10 client.0 _ll_register_callbacks cb 0x7f61f009ac70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:18:30.878 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.876+0000 7f6200c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557d7b9142b0 0x557d7b922df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:30.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.876+0000 7f6200c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557d7b9142b0 0x557d7b922df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:48576/0 (socket says 172.21.15.39:48576) 2024-03-18T16:18:30.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.876+0000 7f6200c00640 1 -- 172.21.15.39:0/1861406557 learned_addr learned my addr 172.21.15.39:0/1861406557 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:18:30.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.876+0000 7f6202000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557d7b919420 0x557d7b923d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:30.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.877+0000 7f6200c00640 1 -- 172.21.15.39:0/1861406557 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557d7b919420 msgr2=0x557d7b923d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:18:30.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.877+0000 7f6200c00640 1 --2- 172.21.15.39:0/1861406557 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557d7b919420 0x557d7b923d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.877+0000 7f6200c00640 1 -- 172.21.15.39:0/1861406557 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557d7b9138c0 msgr2=0x557d7b9228b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T16:18:30.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.877+0000 7f6200c00640 1 --2- 172.21.15.39:0/1861406557 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557d7b9138c0 0x557d7b9228b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.877+0000 7f6200c00640 1 -- 172.21.15.39:0/1861406557 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557d7b924380 con 0x557d7b9142b0 2024-03-18T16:18:30.880 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.877+0000 7f6200c00640 1 --2- 172.21.15.39:0/1861406557 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557d7b9142b0 0x557d7b922df0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f61e8021010 tx=0x7f61e8029c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac687b754d634cdb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:18:30.880 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.878+0000 7f61e2a00640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-03-18T16:18:30.880 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.878+0000 7f61e2a00640 1 -- 172.21.15.39:0/1861406557 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61e801cb70 con 0x557d7b9142b0 2024-03-18T16:18:30.880 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.878+0000 7f6206c50740 1 -- 172.21.15.39:0/1861406557 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f61e802f070 con 0x557d7b9142b0 2024-03-18T16:18:30.880 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.878+0000 7f61e2a00640 1 -- 172.21.15.39:0/1861406557 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f61e801c450 con 0x557d7b9142b0 2024-03-18T16:18:30.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.879+0000 7f61e2a00640 1 -- 172.21.15.39:0/1861406557 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f61e801cb70 con 0x557d7b9142b0 2024-03-18T16:18:30.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.879+0000 7f6206c50740 10 client.16680 fetch_fsmap learned FSMap version 44 2024-03-18T16:18:30.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.879+0000 7f6206c50740 1 -- 172.21.15.39:0/1861406557 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x557d7b9a0d10 con 0x557d7b9142b0 2024-03-18T16:18:30.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.880+0000 7f61e2a00640 1 -- 172.21.15.39:0/1861406557 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f61e802f070 con 0x557d7b9142b0 2024-03-18T16:18:30.882 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.880+0000 7f61e2a00640 1 -- 172.21.15.39:0/1861406557 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7f61e801a040 con 0x557d7b9142b0 2024-03-18T16:18:30.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.881+0000 7f6206c50740 10 client.16680 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:18:30.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.881+0000 7f6206c50740 10 client.16680 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:18:30.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.881+0000 7f6206c50740 20 client.16680 populate_metadata read hostname 'smithi039' 2024-03-18T16:18:30.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.881+0000 7f6206c50740 1 --2- 172.21.15.39:0/1861406557 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x557d7b9a15f0 0x557d7b9a3a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:30.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.881+0000 7f6206c50740 4 client.16680 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:18:30.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.881+0000 7f6206c50740 1 -- 172.21.15.39:0/1861406557 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557d7b87fce0 con 0x557d7b9a15f0 2024-03-18T16:18:30.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.881+0000 7f6201600640 1 --2- 172.21.15.39:0/1861406557 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x557d7b9a15f0 0x557d7b9a3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:30.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.882+0000 7f6201600640 1 --2- 172.21.15.39:0/1861406557 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x557d7b9a15f0 0x557d7b9a3a10 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d5304fef58e214e6 server_cookie=e5e4e05b12fb2941 in_seq=0 out_seq=0 2024-03-18T16:18:30.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.882+0000 7f61e2a00640 10 client.16680 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:18:30.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.883+0000 7f61e2a00640 1 -- 172.21.15.39:0/1861406557 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557d7b87fce0 con 0x557d7b9a15f0 2024-03-18T16:18:30.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.883+0000 7f61e2a00640 10 client.16680 handle_command_reply: tid=0 2024-03-18T16:18:30.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.883+0000 7f6206c50740 1 client.16680 shutdown 2024-03-18T16:18:30.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.884+0000 7f6206c50740 1 -- 172.21.15.39:0/1861406557 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557d7b9142b0 msgr2=0x557d7b922df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:18:30.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.884+0000 7f6206c50740 1 --2- 172.21.15.39:0/1861406557 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557d7b9142b0 0x557d7b922df0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f61e8021010 tx=0x7f61e8029c10 comp rx=0 tx=0).stop 2024-03-18T16:18:30.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.884+0000 7f6206c50740 1 -- 172.21.15.39:0/1861406557 shutdown_connections 2024-03-18T16:18:30.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.884+0000 7f6206c50740 1 --2- 172.21.15.39:0/1861406557 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x557d7b9a15f0 0x557d7b9a3a10 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.884+0000 7f6206c50740 1 --2- 172.21.15.39:0/1861406557 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557d7b919420 0x557d7b923d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.884+0000 7f6206c50740 1 --2- 172.21.15.39:0/1861406557 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x557d7b9142b0 0x557d7b922df0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.884+0000 7f6206c50740 1 --2- 172.21.15.39:0/1861406557 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557d7b9138c0 0x557d7b9228b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.884+0000 7f6206c50740 1 -- 172.21.15.39:0/1861406557 >> 172.21.15.39:0/1861406557 conn(0x557d7b90ee30 msgr2=0x557d7b912920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:18:30.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.884+0000 7f61e2a00640 0 client.16680 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:18:30.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.884+0000 7f6206c50740 1 -- 172.21.15.39:0/1861406557 shutdown_connections 2024-03-18T16:18:30.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.885+0000 7f6206c50740 1 -- 172.21.15.39:0/1861406557 wait complete. 2024-03-18T16:18:30.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.885+0000 7f6206c50740 20 client.16680 trim_cache size 0 max 16384 2024-03-18T16:18:30.887 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:18:30.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.887+0000 7f6204000640 1 -- 172.21.15.39:0/2447157156 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f61e40a2790 msgr2=0x7f61e40a4c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:18:30.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.887+0000 7f6204000640 1 --2- 172.21.15.39:0/2447157156 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f61e40a2790 0x7f61e40a4c50 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x557d7b90fda0 tx=0x7f61e8009260 comp rx=0 tx=0).stop 2024-03-18T16:18:30.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.887+0000 7f6204000640 1 -- 172.21.15.39:0/2447157156 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f61fc107e20 msgr2=0x7f61fc19e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:18:30.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.887+0000 7f6204000640 1 --2- 172.21.15.39:0/2447157156 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f61fc107e20 0x7f61fc19e070 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f61f000d950 tx=0x7f61f000de10 comp rx=0 tx=0).stop 2024-03-18T16:18:30.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.887+0000 7f6204000640 1 -- 172.21.15.39:0/2447157156 shutdown_connections 2024-03-18T16:18:30.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.887+0000 7f6204000640 1 --2- 172.21.15.39:0/2447157156 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f61e40a2790 0x7f61e40a4c50 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.887+0000 7f6204000640 1 --2- 172.21.15.39:0/2447157156 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f61fc109930 0x7f61fc1a5620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.887+0000 7f6204000640 1 --2- 172.21.15.39:0/2447157156 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f61fc108f70 0x7f61fc19e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.887+0000 7f6204000640 1 --2- 172.21.15.39:0/2447157156 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f61fc107e20 0x7f61fc19e070 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:30.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.887+0000 7f6204000640 1 -- 172.21.15.39:0/2447157156 >> 172.21.15.39:0/2447157156 conn(0x7f61fc076dd0 msgr2=0x7f61fc10c230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:18:30.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.887+0000 7f6204000640 1 -- 172.21.15.39:0/2447157156 shutdown_connections 2024-03-18T16:18:30.889 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:30.887+0000 7f6204000640 1 -- 172.21.15.39:0/2447157156 wait complete. 2024-03-18T16:18:31.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:18:30 smithi196 ceph-mon[36307]: pgmap v675: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 6.1 MiB/s wr, 1.42k op/s 2024-03-18T16:18:31.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:18:30 smithi194 ceph-mon[36200]: pgmap v675: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 6.1 MiB/s wr, 1.42k op/s 2024-03-18T16:18:31.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:18:30 smithi039 ceph-mon[31721]: pgmap v675: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 KiB/s rd, 6.1 MiB/s wr, 1.42k op/s 2024-03-18T16:18:31.917 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-18T16:18:31.917 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T16:18:31.918 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-18T16:18:32.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.199+0000 7fd1e9600640 1 -- 172.21.15.39:0/3286755977 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1e4108ff0 msgr2=0x7fd1e4109470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:18:32.201 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.199+0000 7fd1e9600640 1 --2- 172.21.15.39:0/3286755977 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1e4108ff0 0x7fd1e4109470 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fd1d4009900 tx=0x7fd1d402bc30 comp rx=0 tx=0).stop 2024-03-18T16:18:32.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.200+0000 7fd1e9600640 1 -- 172.21.15.39:0/3286755977 shutdown_connections 2024-03-18T16:18:32.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.200+0000 7fd1e9600640 1 --2- 172.21.15.39:0/3286755977 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd1e41099b0 0x7fd1e4110240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:32.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.200+0000 7fd1e9600640 1 --2- 172.21.15.39:0/3286755977 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1e4108ff0 0x7fd1e4109470 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:32.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.200+0000 7fd1e9600640 1 --2- 172.21.15.39:0/3286755977 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd1e4107e00 0x7fd1e4108220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:32.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.200+0000 7fd1e9600640 1 -- 172.21.15.39:0/3286755977 >> 172.21.15.39:0/3286755977 conn(0x7fd1e4076da0 msgr2=0x7fd1e40791e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:18:32.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.201+0000 7fd1e9600640 1 -- 172.21.15.39:0/3286755977 shutdown_connections 2024-03-18T16:18:32.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.202+0000 7fd1e9600640 1 -- 172.21.15.39:0/3286755977 wait complete. 2024-03-18T16:18:32.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.203+0000 7fd1e9600640 1 Processor -- start 2024-03-18T16:18:32.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.203+0000 7fd1e9600640 1 -- start start 2024-03-18T16:18:32.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.204+0000 7fd1e9600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd1e4107e00 0x7fd1e419e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:32.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.204+0000 7fd1e9600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd1e4108ff0 0x7fd1e419e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:32.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.204+0000 7fd1e9600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1e41099b0 0x7fd1e41a56d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:32.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.204+0000 7fd1e9600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fd1e4113ab0 con 0x7fd1e41099b0 2024-03-18T16:18:32.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.204+0000 7fd1e9600640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fd1e4113930 con 0x7fd1e4107e00 2024-03-18T16:18:32.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.204+0000 7fd1e9600640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fd1e4113c30 con 0x7fd1e4108ff0 2024-03-18T16:18:32.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.204+0000 7fd1e2a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd1e4107e00 0x7fd1e419e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:32.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.204+0000 7fd1e2a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd1e4107e00 0x7fd1e419e120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:58694/0 (socket says 172.21.15.39:58694) 2024-03-18T16:18:32.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.204+0000 7fd1e2a00640 1 -- 172.21.15.39:0/3125627821 learned_addr learned my addr 172.21.15.39:0/3125627821 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:18:32.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.205+0000 7fd1e2000640 1 --2- 172.21.15.39:0/3125627821 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd1e4108ff0 0x7fd1e419e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:32.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.205+0000 7fd1e2a00640 1 -- 172.21.15.39:0/3125627821 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd1e4108ff0 msgr2=0x7fd1e419e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:18:32.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.205+0000 7fd1e2a00640 1 --2- 172.21.15.39:0/3125627821 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd1e4108ff0 0x7fd1e419e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:32.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.205+0000 7fd1e2a00640 1 -- 172.21.15.39:0/3125627821 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1e41099b0 msgr2=0x7fd1e41a56d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-18T16:18:32.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.206+0000 7fd1e2a00640 1 --2- 172.21.15.39:0/3125627821 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1e41099b0 0x7fd1e41a56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:32.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.206+0000 7fd1e2a00640 1 -- 172.21.15.39:0/3125627821 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1e407a0b0 con 0x7fd1e4107e00 2024-03-18T16:18:32.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.206+0000 7fd1e2a00640 1 --2- 172.21.15.39:0/3125627821 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd1e4107e00 0x7fd1e419e120 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc00d770 tx=0x7fd1cc00dc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:18:32.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.207+0000 7fd1db400640 1 -- 172.21.15.39:0/3125627821 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1cc017070 con 0x7fd1e4107e00 2024-03-18T16:18:32.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.207+0000 7fd1db400640 1 -- 172.21.15.39:0/3125627821 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1cc009c30 con 0x7fd1e4107e00 2024-03-18T16:18:32.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.207+0000 7fd1db400640 1 -- 172.21.15.39:0/3125627821 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1cc003dd0 con 0x7fd1e4107e00 2024-03-18T16:18:32.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.207+0000 7fd1e9600640 1 -- 172.21.15.39:0/3125627821 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1e407a3a0 con 0x7fd1e4107e00 2024-03-18T16:18:32.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.207+0000 7fd1e9600640 1 -- 172.21.15.39:0/3125627821 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1e407a8c0 con 0x7fd1e4107e00 2024-03-18T16:18:32.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.212+0000 7fd1db400640 1 -- 172.21.15.39:0/3125627821 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fd1cc003f90 con 0x7fd1e4107e00 2024-03-18T16:18:32.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.215+0000 7fd1db400640 1 --2- 172.21.15.39:0/3125627821 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd1c80a28d0 0x7fd1c80a4d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:32.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.216+0000 7fd1db400640 1 -- 172.21.15.39:0/3125627821 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fd1cc0ce700 con 0x7fd1e4107e00 2024-03-18T16:18:32.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.216+0000 7fd1ec367740 1 Processor -- start 2024-03-18T16:18:32.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.216+0000 7fd1e2000640 1 --2- 172.21.15.39:0/3125627821 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd1c80a28d0 0x7fd1c80a4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:32.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.216+0000 7fd1ec367740 1 -- start start 2024-03-18T16:18:32.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.217+0000 7fd1ec367740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55986692e7e0 0x55986692ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:32.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.217+0000 7fd1ec367740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55986692f1d0 0x559866933cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:32.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.217+0000 7fd1ec367740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559866934340 0x5598669367d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:32.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.217+0000 7fd1ec367740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x559866891f80 con 0x55986692e7e0 2024-03-18T16:18:32.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.217+0000 7fd1ec367740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55986683aad0 con 0x55986692f1d0 2024-03-18T16:18:32.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.217+0000 7fd1ec367740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55986688e020 con 0x559866934340 2024-03-18T16:18:32.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.217+0000 7fd1e2a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55986692e7e0 0x55986692ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:32.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.217+0000 7fd1e3400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559866934340 0x5598669367d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:32.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.217+0000 7fd1e3400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559866934340 0x5598669367d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:49374/0 (socket says 172.21.15.39:49374) 2024-03-18T16:18:32.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.217+0000 7fd1e3400640 1 -- 172.21.15.39:0/328609915 learned_addr learned my addr 172.21.15.39:0/328609915 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:18:32.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.217+0000 7fd1e2000640 1 --2- 172.21.15.39:0/328609915 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55986692f1d0 0x559866933cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:32.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.218+0000 7fd1e3400640 1 -- 172.21.15.39:0/328609915 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55986692f1d0 msgr2=0x559866933cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:18:32.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.218+0000 7fd1e3400640 1 --2- 172.21.15.39:0/328609915 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55986692f1d0 0x559866933cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:32.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.218+0000 7fd1e3400640 1 -- 172.21.15.39:0/328609915 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55986692e7e0 msgr2=0x55986692ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:18:32.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.218+0000 7fd1e3400640 1 --2- 172.21.15.39:0/328609915 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55986692e7e0 0x55986692ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:32.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.218+0000 7fd1e3400640 1 -- 172.21.15.39:0/328609915 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5598668526f0 con 0x559866934340 2024-03-18T16:18:32.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.218+0000 7fd1e2a00640 1 --2- 172.21.15.39:0/328609915 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55986692e7e0 0x55986692ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:18:32.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.218+0000 7fd1e2000640 1 --2- 172.21.15.39:0/3125627821 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd1c80a28d0 0x7fd1c80a4d90 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fd1d402c240 tx=0x7fd1d402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:18:32.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.218+0000 7fd1e3400640 1 --2- 172.21.15.39:0/328609915 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559866934340 0x5598669367d0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc005c40 tx=0x7fd1dc009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8d974a4a58c1f5d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:18:32.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.219+0000 7fd1d3400640 1 -- 172.21.15.39:0/328609915 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1dc003b70 con 0x559866934340 2024-03-18T16:18:32.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.219+0000 7fd1d3400640 1 -- 172.21.15.39:0/328609915 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1dc00ee60 con 0x559866934340 2024-03-18T16:18:32.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.219+0000 7fd1d3400640 1 -- 172.21.15.39:0/328609915 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1dc0108f0 con 0x559866934340 2024-03-18T16:18:32.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.219+0000 7fd1ec367740 1 -- 172.21.15.39:0/328609915 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559866934340 msgr2=0x5598669367d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:18:32.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.219+0000 7fd1ec367740 1 --2- 172.21.15.39:0/328609915 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559866934340 0x5598669367d0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fd1dc005c40 tx=0x7fd1dc009f90 comp rx=0 tx=0).stop 2024-03-18T16:18:32.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.220+0000 7fd1ec367740 1 -- 172.21.15.39:0/328609915 shutdown_connections 2024-03-18T16:18:32.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.220+0000 7fd1ec367740 1 --2- 172.21.15.39:0/328609915 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55986692f1d0 0x559866933cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:32.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.220+0000 7fd1ec367740 1 --2- 172.21.15.39:0/328609915 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559866934340 0x5598669367d0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:32.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.220+0000 7fd1ec367740 1 --2- 172.21.15.39:0/328609915 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55986692e7e0 0x55986692ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:32.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.220+0000 7fd1ec367740 1 -- 172.21.15.39:0/328609915 >> 172.21.15.39:0/328609915 conn(0x5598668f6930 msgr2=0x55986692a150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:18:32.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.220+0000 7fd1ec367740 1 -- 172.21.15.39:0/328609915 shutdown_connections 2024-03-18T16:18:32.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.220+0000 7fd1ec367740 1 -- 172.21.15.39:0/328609915 wait complete. 2024-03-18T16:18:32.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.220+0000 7fd1ec367740 1 -- start start 2024-03-18T16:18:32.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.221+0000 7fd1ec367740 1 Processor -- start 2024-03-18T16:18:32.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.221+0000 7fd1ec367740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55986692e7e0 0x559866938b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:32.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.221+0000 7fd1ec367740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55986692f1d0 0x559866939050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:32.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.221+0000 7fd1ec367740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559866934340 0x559866940090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:32.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.221+0000 7fd1ec367740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5598669372f0 con 0x55986692f1d0 2024-03-18T16:18:32.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.221+0000 7fd1ec367740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55986684df30 con 0x55986692e7e0 2024-03-18T16:18:32.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.221+0000 7fd1ec367740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x559866937470 con 0x559866934340 2024-03-18T16:18:32.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.222+0000 7fd1ec367740 10 client.0 _ll_register_callbacks cb 0x55986683bbc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:18:32.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.222+0000 7fd1e2a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55986692e7e0 0x559866938b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:32.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.222+0000 7fd1e2a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55986692e7e0 0x559866938b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:58718/0 (socket says 172.21.15.39:58718) 2024-03-18T16:18:32.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.222+0000 7fd1e2a00640 1 -- 172.21.15.39:0/357396155 learned_addr learned my addr 172.21.15.39:0/357396155 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:18:32.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.222+0000 7fd1e3400640 1 --2- 172.21.15.39:0/357396155 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559866934340 0x559866940090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:32.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.222+0000 7fd1e2000640 1 --2- 172.21.15.39:0/357396155 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55986692f1d0 0x559866939050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:32.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.224+0000 7fd1e2a00640 1 -- 172.21.15.39:0/357396155 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559866934340 msgr2=0x559866940090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:18:32.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.224+0000 7fd1e2a00640 1 --2- 172.21.15.39:0/357396155 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559866934340 0x559866940090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:32.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.224+0000 7fd1e2a00640 1 -- 172.21.15.39:0/357396155 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55986692f1d0 msgr2=0x559866939050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:18:32.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.224+0000 7fd1e2a00640 1 --2- 172.21.15.39:0/357396155 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55986692f1d0 0x559866939050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:32.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.224+0000 7fd1e2a00640 1 -- 172.21.15.39:0/357396155 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55986666e7e0 con 0x55986692e7e0 2024-03-18T16:18:32.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.224+0000 7fd1e2a00640 1 --2- 172.21.15.39:0/357396155 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55986692e7e0 0x559866938b10 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc09aca0 tx=0x7fd1cc09b410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff43796fd0a94259 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:18:32.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.224+0000 7fd1c7e00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:18:32.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.225+0000 7fd1c7e00640 1 -- 172.21.15.39:0/357396155 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1cc020050 con 0x55986692e7e0 2024-03-18T16:18:32.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.225+0000 7fd1ec367740 1 -- 172.21.15.39:0/357396155 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd1dc00ee60 con 0x55986692e7e0 2024-03-18T16:18:32.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.225+0000 7fd1c7e00640 1 -- 172.21.15.39:0/357396155 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fd1cc0ce410 con 0x55986692e7e0 2024-03-18T16:18:32.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.226+0000 7fd1c7e00640 1 -- 172.21.15.39:0/357396155 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fd1cc09ad10 con 0x55986692e7e0 2024-03-18T16:18:32.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.226+0000 7fd1ec367740 10 client.27314 fetch_fsmap learned FSMap version 44 2024-03-18T16:18:32.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.227+0000 7fd1ec367740 1 -- 172.21.15.39:0/357396155 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x5598666cef70 con 0x55986692e7e0 2024-03-18T16:18:32.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.228+0000 7fd1c7e00640 1 -- 172.21.15.39:0/357396155 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1dc00ee60 con 0x55986692e7e0 2024-03-18T16:18:32.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.228+0000 7fd1c7e00640 1 -- 172.21.15.39:0/357396155 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7fd1cc079ca0 con 0x55986692e7e0 2024-03-18T16:18:32.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.229+0000 7fd1ec367740 10 client.27314 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:18:32.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.229+0000 7fd1ec367740 10 client.27314 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:18:32.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.229+0000 7fd1ec367740 20 client.27314 populate_metadata read hostname 'smithi039' 2024-03-18T16:18:32.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.229+0000 7fd1ec367740 1 --2- 172.21.15.39:0/357396155 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x559866944970 0x559866946de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:32.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.229+0000 7fd1ec367740 4 client.27314 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:18:32.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.229+0000 7fd1ec367740 1 -- 172.21.15.39:0/357396155 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55986689f4c0 con 0x559866944970 2024-03-18T16:18:32.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.229+0000 7fd1e2000640 1 --2- 172.21.15.39:0/357396155 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x559866944970 0x559866946de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:32.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.230+0000 7fd1e2000640 1 --2- 172.21.15.39:0/357396155 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x559866944970 0x559866946de0 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=50f812a85942f7dc server_cookie=a9924ee462ac76ae in_seq=0 out_seq=0 2024-03-18T16:18:32.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.230+0000 7fd1c7e00640 10 client.27314 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:18:32.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.231+0000 7fd1c7e00640 1 -- 172.21.15.39:0/357396155 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55986689f4c0 con 0x559866944970 2024-03-18T16:18:32.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.231+0000 7fd1c7e00640 10 client.27314 handle_command_reply: tid=0 2024-03-18T16:18:32.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.232+0000 7fd1ec367740 1 client.27314 shutdown 2024-03-18T16:18:32.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.232+0000 7fd1ec367740 1 -- 172.21.15.39:0/357396155 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55986692e7e0 msgr2=0x559866938b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:18:32.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.232+0000 7fd1ec367740 1 --2- 172.21.15.39:0/357396155 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55986692e7e0 0x559866938b10 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc09aca0 tx=0x7fd1cc09b410 comp rx=0 tx=0).stop 2024-03-18T16:18:32.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.232+0000 7fd1ec367740 1 -- 172.21.15.39:0/357396155 shutdown_connections 2024-03-18T16:18:32.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.232+0000 7fd1ec367740 1 --2- 172.21.15.39:0/357396155 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x559866944970 0x559866946de0 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:32.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.232+0000 7fd1ec367740 1 --2- 172.21.15.39:0/357396155 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559866934340 0x559866940090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:32.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.232+0000 7fd1ec367740 1 --2- 172.21.15.39:0/357396155 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55986692f1d0 0x559866939050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:32.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.232+0000 7fd1ec367740 1 --2- 172.21.15.39:0/357396155 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55986692e7e0 0x559866938b10 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:32.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.232+0000 7fd1ec367740 1 -- 172.21.15.39:0/357396155 >> 172.21.15.39:0/357396155 conn(0x5598668f6930 msgr2=0x559866929350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:18:32.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.232+0000 7fd1c7e00640 0 client.27314 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:18:32.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.232+0000 7fd1ec367740 1 -- 172.21.15.39:0/357396155 shutdown_connections 2024-03-18T16:18:32.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.233+0000 7fd1ec367740 1 -- 172.21.15.39:0/357396155 wait complete. 2024-03-18T16:18:32.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.233+0000 7fd1ec367740 20 client.27314 trim_cache size 0 max 16384 2024-03-18T16:18:32.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.281+0000 7fd1ec367740 1 Processor -- start 2024-03-18T16:18:32.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.281+0000 7fd1ec367740 1 -- start start 2024-03-18T16:18:32.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.281+0000 7fd1ec367740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55986692f1d0 0x55986692daa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:32.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.281+0000 7fd1ec367740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559866934340 0x559866940050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:32.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.281+0000 7fd1ec367740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559866938720 0x559866938bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:32.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.281+0000 7fd1ec367740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fd1e41b9250 con 0x559866934340 2024-03-18T16:18:32.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.281+0000 7fd1ec367740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fd1e41bad90 con 0x559866938720 2024-03-18T16:18:32.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.281+0000 7fd1ec367740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fd1e41b9920 con 0x55986692f1d0 2024-03-18T16:18:32.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.282+0000 7fd1e2a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55986692f1d0 0x55986692daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:32.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.282+0000 7fd1e2a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55986692f1d0 0x55986692daa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:49392/0 (socket says 172.21.15.39:49392) 2024-03-18T16:18:32.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.282+0000 7fd1e2a00640 1 -- 172.21.15.39:0/2778817672 learned_addr learned my addr 172.21.15.39:0/2778817672 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:18:32.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.282+0000 7fd1e3400640 1 --2- 172.21.15.39:0/2778817672 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559866938720 0x559866938bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:32.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.282+0000 7fd1e2000640 1 --2- 172.21.15.39:0/2778817672 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559866934340 0x559866940050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:32.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.283+0000 7fd1e2a00640 1 -- 172.21.15.39:0/2778817672 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559866938720 msgr2=0x559866938bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:18:32.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.283+0000 7fd1e2a00640 1 --2- 172.21.15.39:0/2778817672 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559866938720 0x559866938bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:32.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.283+0000 7fd1e2a00640 1 -- 172.21.15.39:0/2778817672 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559866934340 msgr2=0x559866940050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:18:32.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.283+0000 7fd1e2a00640 1 --2- 172.21.15.39:0/2778817672 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559866934340 0x559866940050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:32.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.283+0000 7fd1e2a00640 1 -- 172.21.15.39:0/2778817672 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559866566170 con 0x55986692f1d0 2024-03-18T16:18:32.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.283+0000 7fd1e2000640 1 --2- 172.21.15.39:0/2778817672 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559866934340 0x559866940050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:18:32.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.283+0000 7fd1e2a00640 1 --2- 172.21.15.39:0/2778817672 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55986692f1d0 0x55986692daa0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc097170 tx=0x7fd1cc099220 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4cca60d3420d4b5b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:18:32.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.283+0000 7fd1d2a00640 1 -- 172.21.15.39:0/2778817672 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1cc020050 con 0x55986692f1d0 2024-03-18T16:18:32.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.283+0000 7fd1d2a00640 1 -- 172.21.15.39:0/2778817672 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1cc0a34b0 con 0x55986692f1d0 2024-03-18T16:18:32.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.283+0000 7fd1d2a00640 1 -- 172.21.15.39:0/2778817672 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1cc09ee60 con 0x55986692f1d0 2024-03-18T16:18:32.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.285+0000 7fd1ec367740 1 -- 172.21.15.39:0/2778817672 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55986692f1d0 msgr2=0x55986692daa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:18:32.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.285+0000 7fd1ec367740 1 --2- 172.21.15.39:0/2778817672 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55986692f1d0 0x55986692daa0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc097170 tx=0x7fd1cc099220 comp rx=0 tx=0).stop 2024-03-18T16:18:32.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.285+0000 7fd1ec367740 1 -- 172.21.15.39:0/2778817672 shutdown_connections 2024-03-18T16:18:32.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.285+0000 7fd1ec367740 1 --2- 172.21.15.39:0/2778817672 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559866938720 0x559866938bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:32.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.285+0000 7fd1ec367740 1 --2- 172.21.15.39:0/2778817672 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559866934340 0x559866940050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:32.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.285+0000 7fd1ec367740 1 --2- 172.21.15.39:0/2778817672 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55986692f1d0 0x55986692daa0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:32.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.285+0000 7fd1ec367740 1 -- 172.21.15.39:0/2778817672 >> 172.21.15.39:0/2778817672 conn(0x5598668f6930 msgr2=0x55986692d390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:18:32.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.286+0000 7fd1ec367740 1 -- 172.21.15.39:0/2778817672 shutdown_connections 2024-03-18T16:18:32.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.286+0000 7fd1ec367740 1 -- 172.21.15.39:0/2778817672 wait complete. 2024-03-18T16:18:32.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.287+0000 7fd1ec367740 1 -- start start 2024-03-18T16:18:32.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.287+0000 7fd1ec367740 1 Processor -- start 2024-03-18T16:18:32.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.288+0000 7fd1ec367740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55986692f1d0 0x55986693de90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:32.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.288+0000 7fd1ec367740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559866934340 0x55986693e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:32.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.288+0000 7fd1ec367740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559866938720 0x55986693f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:32.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.288+0000 7fd1ec367740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fd1e41ba6c0 con 0x559866938720 2024-03-18T16:18:32.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.288+0000 7fd1ec367740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fd1e41b9ff0 con 0x559866934340 2024-03-18T16:18:32.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.288+0000 7fd1ec367740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fd1e4082490 con 0x55986692f1d0 2024-03-18T16:18:32.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.288+0000 7fd1ec367740 10 client.0 _ll_register_callbacks cb 0x7fd1cc00de80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:18:32.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.289+0000 7fd1e2a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55986692f1d0 0x55986693de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:32.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.289+0000 7fd1e2a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55986692f1d0 0x55986693de90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:49406/0 (socket says 172.21.15.39:49406) 2024-03-18T16:18:32.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.289+0000 7fd1e2a00640 1 -- 172.21.15.39:0/3833803060 learned_addr learned my addr 172.21.15.39:0/3833803060 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:18:32.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.289+0000 7fd1e2000640 1 --2- 172.21.15.39:0/3833803060 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559866934340 0x55986693e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:32.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.289+0000 7fd1e3400640 1 --2- 172.21.15.39:0/3833803060 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559866938720 0x55986693f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:32.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.289+0000 7fd1e2a00640 1 -- 172.21.15.39:0/3833803060 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559866934340 msgr2=0x55986693e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:18:32.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.289+0000 7fd1e2a00640 1 --2- 172.21.15.39:0/3833803060 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559866934340 0x55986693e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:32.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.289+0000 7fd1e2a00640 1 -- 172.21.15.39:0/3833803060 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559866938720 msgr2=0x55986693f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:18:32.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.289+0000 7fd1e2a00640 1 --2- 172.21.15.39:0/3833803060 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559866938720 0x55986693f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:32.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.289+0000 7fd1e2a00640 1 -- 172.21.15.39:0/3833803060 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55986693f9d0 con 0x55986692f1d0 2024-03-18T16:18:32.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.289+0000 7fd1e2000640 1 --2- 172.21.15.39:0/3833803060 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559866934340 0x55986693e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:18:32.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.289+0000 7fd1e3400640 1 --2- 172.21.15.39:0/3833803060 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559866938720 0x55986693f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:18:32.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.290+0000 7fd1e2a00640 1 --2- 172.21.15.39:0/3833803060 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55986692f1d0 0x55986693de90 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc0a2f80 tx=0x7fd1cc0975b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f65f6a4c8afdae77 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:18:32.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.290+0000 7fd1c7e00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:18:32.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.290+0000 7fd1c7e00640 1 -- 172.21.15.39:0/3833803060 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1cc020030 con 0x55986692f1d0 2024-03-18T16:18:32.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.290+0000 7fd1c7e00640 1 -- 172.21.15.39:0/3833803060 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fd1cc079b80 con 0x55986692f1d0 2024-03-18T16:18:32.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.290+0000 7fd1ec367740 1 -- 172.21.15.39:0/3833803060 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd1cc0a34b0 con 0x55986692f1d0 2024-03-18T16:18:32.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.291+0000 7fd1c7e00640 1 -- 172.21.15.39:0/3833803060 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fd1cc09ee60 con 0x55986692f1d0 2024-03-18T16:18:32.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.291+0000 7fd1ec367740 10 client.27298 fetch_fsmap learned FSMap version 44 2024-03-18T16:18:32.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.291+0000 7fd1ec367740 1 -- 172.21.15.39:0/3833803060 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x55986693f9d0 con 0x55986692f1d0 2024-03-18T16:18:32.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.291+0000 7fd1c7e00640 1 -- 172.21.15.39:0/3833803060 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1cc0a34b0 con 0x55986692f1d0 2024-03-18T16:18:32.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.292+0000 7fd1c7e00640 1 -- 172.21.15.39:0/3833803060 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7fd1cc09fd80 con 0x55986692f1d0 2024-03-18T16:18:32.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.292+0000 7fd1ec367740 10 client.27298 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:18:32.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.292+0000 7fd1ec367740 10 client.27298 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:18:32.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.292+0000 7fd1ec367740 20 client.27298 populate_metadata read hostname 'smithi039' 2024-03-18T16:18:32.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.292+0000 7fd1ec367740 1 --2- 172.21.15.39:0/3833803060 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5598669ab9c0 0x5598669adde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:32.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.293+0000 7fd1ec367740 4 client.27298 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-18T16:18:32.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.293+0000 7fd1ec367740 1 -- 172.21.15.39:0/3833803060 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x559866899810 con 0x5598669ab9c0 2024-03-18T16:18:32.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.295+0000 7fd1e2000640 1 --2- 172.21.15.39:0/3833803060 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5598669ab9c0 0x5598669adde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:32.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.295+0000 7fd1e2000640 1 --2- 172.21.15.39:0/3833803060 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5598669ab9c0 0x5598669adde0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd3301a60a99df5e server_cookie=a25da867dc425827 in_seq=0 out_seq=0 2024-03-18T16:18:32.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.295+0000 7fd1c7e00640 10 client.27298 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:18:32.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.296+0000 7fd1c7e00640 1 -- 172.21.15.39:0/3833803060 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x559866899810 con 0x5598669ab9c0 2024-03-18T16:18:32.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.296+0000 7fd1c7e00640 10 client.27298 handle_command_reply: tid=0 2024-03-18T16:18:32.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.296+0000 7fd1ec367740 1 client.27298 shutdown 2024-03-18T16:18:32.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.297+0000 7fd1ec367740 1 -- 172.21.15.39:0/3833803060 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55986692f1d0 msgr2=0x55986693de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:18:32.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.297+0000 7fd1ec367740 1 --2- 172.21.15.39:0/3833803060 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55986692f1d0 0x55986693de90 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc0a2f80 tx=0x7fd1cc0975b0 comp rx=0 tx=0).stop 2024-03-18T16:18:32.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.297+0000 7fd1ec367740 1 -- 172.21.15.39:0/3833803060 shutdown_connections 2024-03-18T16:18:32.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.297+0000 7fd1ec367740 1 --2- 172.21.15.39:0/3833803060 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5598669ab9c0 0x5598669adde0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:32.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.297+0000 7fd1ec367740 1 --2- 172.21.15.39:0/3833803060 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559866934340 0x55986693e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:32.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.297+0000 7fd1ec367740 1 --2- 172.21.15.39:0/3833803060 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559866938720 0x55986693f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:32.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.297+0000 7fd1c7e00640 0 client.27298 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:18:32.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.297+0000 7fd1ec367740 1 --2- 172.21.15.39:0/3833803060 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55986692f1d0 0x55986693de90 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:32.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.297+0000 7fd1ec367740 1 -- 172.21.15.39:0/3833803060 >> 172.21.15.39:0/3833803060 conn(0x5598668f6930 msgr2=0x55986692d390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:18:32.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.297+0000 7fd1ec367740 1 -- 172.21.15.39:0/3833803060 shutdown_connections 2024-03-18T16:18:32.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.297+0000 7fd1ec367740 1 -- 172.21.15.39:0/3833803060 wait complete. 2024-03-18T16:18:32.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.298+0000 7fd1ec367740 20 client.27298 trim_cache size 0 max 16384 2024-03-18T16:18:32.300 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-03-18T16:18:32.300 INFO:teuthology.orchestra.run.smithi039.stdout: "return_code": 0, 2024-03-18T16:18:32.300 INFO:teuthology.orchestra.run.smithi039.stdout: "scrub_tag": "d7db728c-b784-4575-bb83-48c352cec3bf", 2024-03-18T16:18:32.300 INFO:teuthology.orchestra.run.smithi039.stdout: "mode": "asynchronous" 2024-03-18T16:18:32.300 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-03-18T16:18:32.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.299+0000 7fd1e9600640 1 -- 172.21.15.39:0/3125627821 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd1c80a28d0 msgr2=0x7fd1c80a4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:18:32.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.299+0000 7fd1e9600640 1 --2- 172.21.15.39:0/3125627821 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd1c80a28d0 0x7fd1c80a4d90 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fd1d402c240 tx=0x7fd1d402df90 comp rx=0 tx=0).stop 2024-03-18T16:18:32.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.299+0000 7fd1e9600640 1 -- 172.21.15.39:0/3125627821 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd1e4107e00 msgr2=0x7fd1e419e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:18:32.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.299+0000 7fd1e9600640 1 --2- 172.21.15.39:0/3125627821 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd1e4107e00 0x7fd1e419e120 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fd1cc00d770 tx=0x7fd1cc00dc30 comp rx=0 tx=0).stop 2024-03-18T16:18:32.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.299+0000 7fd1e9600640 1 -- 172.21.15.39:0/3125627821 shutdown_connections 2024-03-18T16:18:32.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.299+0000 7fd1e9600640 1 --2- 172.21.15.39:0/3125627821 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd1c80a28d0 0x7fd1c80a4d90 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:32.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.299+0000 7fd1e9600640 1 --2- 172.21.15.39:0/3125627821 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd1e41099b0 0x7fd1e41a56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:32.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.299+0000 7fd1e9600640 1 --2- 172.21.15.39:0/3125627821 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd1e4108ff0 0x7fd1e419e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:32.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.299+0000 7fd1e9600640 1 --2- 172.21.15.39:0/3125627821 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd1e4107e00 0x7fd1e419e120 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:32.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.299+0000 7fd1e9600640 1 -- 172.21.15.39:0/3125627821 >> 172.21.15.39:0/3125627821 conn(0x7fd1e4076da0 msgr2=0x7fd1e410c2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:18:32.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.299+0000 7fd1e9600640 1 -- 172.21.15.39:0/3125627821 shutdown_connections 2024-03-18T16:18:32.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.300+0000 7fd1e9600640 1 -- 172.21.15.39:0/3125627821 wait complete. 2024-03-18T16:18:32.329 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T16:18:32.329 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T16:18:32.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.624+0000 7f620da00640 1 -- 172.21.15.39:0/1864331707 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f62081099b0 msgr2=0x7f6208110240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:18:32.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.624+0000 7f620da00640 1 --2- 172.21.15.39:0/1864331707 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f62081099b0 0x7f6208110240 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f6200009ca0 tx=0x7f620002c120 comp rx=0 tx=0).stop 2024-03-18T16:18:32.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.624+0000 7f620da00640 1 -- 172.21.15.39:0/1864331707 shutdown_connections 2024-03-18T16:18:32.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.624+0000 7f620da00640 1 --2- 172.21.15.39:0/1864331707 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f62081099b0 0x7f6208110240 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:32.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.624+0000 7f620da00640 1 --2- 172.21.15.39:0/1864331707 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6208108ff0 0x7f6208109470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:32.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.624+0000 7f620da00640 1 --2- 172.21.15.39:0/1864331707 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6208107e00 0x7f6208108220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:32.626 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.624+0000 7f620da00640 1 -- 172.21.15.39:0/1864331707 >> 172.21.15.39:0/1864331707 conn(0x7f6208076da0 msgr2=0x7f62080791e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:18:32.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.625+0000 7f620da00640 1 -- 172.21.15.39:0/1864331707 shutdown_connections 2024-03-18T16:18:32.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.625+0000 7f620da00640 1 -- 172.21.15.39:0/1864331707 wait complete. 2024-03-18T16:18:32.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.627+0000 7f620da00640 1 Processor -- start 2024-03-18T16:18:32.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.627+0000 7f620da00640 1 -- start start 2024-03-18T16:18:32.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.627+0000 7f620da00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6208107e00 0x7f620819e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:32.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.627+0000 7f620da00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6208108ff0 0x7f620819e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:32.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.627+0000 7f620da00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f62081099b0 0x7f62081a5630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:32.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.627+0000 7f620da00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f6208113a60 con 0x7f62081099b0 2024-03-18T16:18:32.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.627+0000 7f620da00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f62081138e0 con 0x7f6208107e00 2024-03-18T16:18:32.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.627+0000 7f620da00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f6208113be0 con 0x7f6208108ff0 2024-03-18T16:18:32.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.628+0000 7f6206a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6208107e00 0x7f620819e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:32.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.628+0000 7f6206a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6208107e00 0x7f620819e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:58752/0 (socket says 172.21.15.39:58752) 2024-03-18T16:18:32.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.628+0000 7f6206a00640 1 -- 172.21.15.39:0/3462858938 learned_addr learned my addr 172.21.15.39:0/3462858938 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:18:32.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.628+0000 7f6207400640 1 --2- 172.21.15.39:0/3462858938 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f62081099b0 0x7f62081a5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:32.630 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.628+0000 7f6206000640 1 --2- 172.21.15.39:0/3462858938 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6208108ff0 0x7f620819e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:32.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.630+0000 7f6206a00640 1 -- 172.21.15.39:0/3462858938 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6208108ff0 msgr2=0x7f620819e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:18:32.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.630+0000 7f6206a00640 1 --2- 172.21.15.39:0/3462858938 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6208108ff0 0x7f620819e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:32.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.630+0000 7f6206a00640 1 -- 172.21.15.39:0/3462858938 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f62081099b0 msgr2=0x7f62081a5630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:18:32.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.630+0000 7f6206a00640 1 --2- 172.21.15.39:0/3462858938 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f62081099b0 0x7f62081a5630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:32.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.630+0000 7f6206a00640 1 -- 172.21.15.39:0/3462858938 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62081a5ca0 con 0x7f6208107e00 2024-03-18T16:18:32.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.630+0000 7f6206a00640 1 --2- 172.21.15.39:0/3462858938 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6208107e00 0x7f620819e080 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f61f400e9c0 tx=0x7f61f400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:18:32.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.630+0000 7f6207400640 1 --2- 172.21.15.39:0/3462858938 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f62081099b0 0x7f62081a5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:18:32.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.630+0000 7f6206000640 1 --2- 172.21.15.39:0/3462858938 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6208108ff0 0x7f620819e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:18:32.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.630+0000 7f61ff400640 1 -- 172.21.15.39:0/3462858938 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61f400cd80 con 0x7f6208107e00 2024-03-18T16:18:32.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.630+0000 7f61ff400640 1 -- 172.21.15.39:0/3462858938 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f61f40035f0 con 0x7f6208107e00 2024-03-18T16:18:32.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.630+0000 7f620da00640 1 -- 172.21.15.39:0/3462858938 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62081a5f90 con 0x7f6208107e00 2024-03-18T16:18:32.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.631+0000 7f620da00640 1 -- 172.21.15.39:0/3462858938 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62081a6450 con 0x7f6208107e00 2024-03-18T16:18:32.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.631+0000 7f61ff400640 1 -- 172.21.15.39:0/3462858938 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61f4010690 con 0x7f6208107e00 2024-03-18T16:18:32.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.636+0000 7f6210793740 1 Processor -- start 2024-03-18T16:18:32.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.636+0000 7f6210793740 1 -- start start 2024-03-18T16:18:32.674 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.637+0000 7f6210793740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559c71ff1810 0x559c71ff1c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:32.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.637+0000 7f6210793740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559c71ff2200 0x559c71ff6d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:32.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.637+0000 7f6210793740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559c71ff7370 0x559c71ff9800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:32.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.637+0000 7f6210793740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x559c71f561f0 con 0x559c71ff7370 2024-03-18T16:18:32.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.637+0000 7f6210793740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x559c71efea80 con 0x559c71ff2200 2024-03-18T16:18:32.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.637+0000 7f6210793740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x559c71f52290 con 0x559c71ff1810 2024-03-18T16:18:32.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.638+0000 7f6207400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559c71ff1810 0x559c71ff1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:32.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.638+0000 7f6207400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559c71ff1810 0x559c71ff1c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:49430/0 (socket says 172.21.15.39:49430) 2024-03-18T16:18:32.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.638+0000 7f6207400640 1 -- 172.21.15.39:0/106339626 learned_addr learned my addr 172.21.15.39:0/106339626 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:18:32.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.639+0000 7f6206000640 1 --2- 172.21.15.39:0/106339626 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559c71ff7370 0x559c71ff9800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:32.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.639+0000 7f6207400640 1 -- 172.21.15.39:0/106339626 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559c71ff2200 msgr2=0x559c71ff6d00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:18:32.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.639+0000 7f6207400640 1 --2- 172.21.15.39:0/106339626 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559c71ff2200 0x559c71ff6d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:32.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.639+0000 7f6207400640 1 -- 172.21.15.39:0/106339626 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559c71ff7370 msgr2=0x559c71ff9800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:18:32.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.639+0000 7f6207400640 1 --2- 172.21.15.39:0/106339626 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559c71ff7370 0x559c71ff9800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:32.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.639+0000 7f6207400640 1 -- 172.21.15.39:0/106339626 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559c71d347d0 con 0x559c71ff1810 2024-03-18T16:18:32.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.639+0000 7f6206000640 1 --2- 172.21.15.39:0/106339626 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559c71ff7370 0x559c71ff9800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:18:32.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.639+0000 7f61ff400640 1 -- 172.21.15.39:0/3462858938 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f61f4010830 con 0x7f6208107e00 2024-03-18T16:18:32.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.640+0000 7f6207400640 1 --2- 172.21.15.39:0/106339626 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559c71ff1810 0x559c71ff1c30 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f6200032610 tx=0x7f62000323c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=db444791d25fd79b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:18:32.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.640+0000 7f61f3e00640 1 -- 172.21.15.39:0/106339626 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6200039040 con 0x559c71ff1810 2024-03-18T16:18:32.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.640+0000 7f61f3e00640 1 -- 172.21.15.39:0/106339626 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6200032cb0 con 0x559c71ff1810 2024-03-18T16:18:32.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.640+0000 7f61f3e00640 1 -- 172.21.15.39:0/106339626 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f620003fe20 con 0x559c71ff1810 2024-03-18T16:18:32.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.642+0000 7f61ff400640 1 --2- 172.21.15.39:0/3462858938 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f61ec0a2880 0x7f61ec0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:32.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.642+0000 7f61ff400640 1 -- 172.21.15.39:0/3462858938 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f61f40cf1d0 con 0x7f6208107e00 2024-03-18T16:18:32.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.642+0000 7f6210793740 1 -- 172.21.15.39:0/106339626 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559c71ff1810 msgr2=0x559c71ff1c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:18:32.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.642+0000 7f6210793740 1 --2- 172.21.15.39:0/106339626 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559c71ff1810 0x559c71ff1c30 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f6200032610 tx=0x7f62000323c0 comp rx=0 tx=0).stop 2024-03-18T16:18:32.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.643+0000 7f6210793740 1 -- 172.21.15.39:0/106339626 shutdown_connections 2024-03-18T16:18:32.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.643+0000 7f6210793740 1 --2- 172.21.15.39:0/106339626 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559c71ff2200 0x559c71ff6d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:32.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.643+0000 7f6210793740 1 --2- 172.21.15.39:0/106339626 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559c71ff7370 0x559c71ff9800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:32.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.643+0000 7f6210793740 1 --2- 172.21.15.39:0/106339626 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559c71ff1810 0x559c71ff1c30 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:32.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.643+0000 7f6210793740 1 -- 172.21.15.39:0/106339626 >> 172.21.15.39:0/106339626 conn(0x559c71fecd80 msgr2=0x559c71fed180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:18:32.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.643+0000 7f6210793740 1 -- 172.21.15.39:0/106339626 shutdown_connections 2024-03-18T16:18:32.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.644+0000 7f6210793740 1 -- 172.21.15.39:0/106339626 wait complete. 2024-03-18T16:18:32.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.644+0000 7f6210793740 1 -- start start 2024-03-18T16:18:32.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.645+0000 7f6210793740 1 Processor -- start 2024-03-18T16:18:32.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.645+0000 7f6206a00640 1 --2- 172.21.15.39:0/3462858938 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f61ec0a2880 0x7f61ec0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:32.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.646+0000 7f6210793740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559c71ff1810 0x559c71ffbd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:32.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.646+0000 7f6206a00640 1 --2- 172.21.15.39:0/3462858938 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f61ec0a2880 0x7f61ec0a4d40 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f61f40a39f0 tx=0x7f61f4020000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:18:32.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.646+0000 7f6207400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559c71ff1810 0x559c71ffbd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:32.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.647+0000 7f6210793740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559c71ff2200 0x559c71ffc2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:32.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.647+0000 7f6207400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559c71ff1810 0x559c71ffbd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:48676/0 (socket says 172.21.15.39:48676) 2024-03-18T16:18:32.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.647+0000 7f6210793740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559c71ff7370 0x559c72003310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:32.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.647+0000 7f6207400640 1 -- 172.21.15.39:0/1262296742 learned_addr learned my addr 172.21.15.39:0/1262296742 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:18:32.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.647+0000 7f6210793740 1 -- 172.21.15.39:0/1262296742 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x559c71ffa2c0 con 0x559c71ff1810 2024-03-18T16:18:32.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.647+0000 7f6210793740 1 -- 172.21.15.39:0/1262296742 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x559c71f11ed0 con 0x559c71ff7370 2024-03-18T16:18:32.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.647+0000 7f6210793740 1 -- 172.21.15.39:0/1262296742 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x559c71ffa440 con 0x559c71ff2200 2024-03-18T16:18:32.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.647+0000 7f6210793740 10 client.0 _ll_register_callbacks cb 0x559c71f4d220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:18:32.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.647+0000 7f6207400640 1 --2- 172.21.15.39:0/1262296742 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559c71ff7370 0x559c72003310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:32.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.647+0000 7f6206000640 1 --2- 172.21.15.39:0/1262296742 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559c71ff2200 0x559c71ffc2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:32.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.648+0000 7f6207400640 1 -- 172.21.15.39:0/1262296742 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559c71ff2200 msgr2=0x559c71ffc2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:18:32.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.648+0000 7f6207400640 1 --2- 172.21.15.39:0/1262296742 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559c71ff2200 0x559c71ffc2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:32.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.648+0000 7f6207400640 1 -- 172.21.15.39:0/1262296742 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559c71ff7370 msgr2=0x559c72003310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:18:32.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.648+0000 7f6207400640 1 --2- 172.21.15.39:0/1262296742 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559c71ff7370 0x559c72003310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:32.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.648+0000 7f6207400640 1 -- 172.21.15.39:0/1262296742 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559c71c6a3f0 con 0x559c71ff1810 2024-03-18T16:18:32.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.648+0000 7f6207400640 1 --2- 172.21.15.39:0/1262296742 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559c71ff1810 0x559c71ffbd90 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f6200005e30 tx=0x7f6200032e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3db3194e7befddf server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:18:32.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.648+0000 7f61f0c00640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-03-18T16:18:32.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.648+0000 7f6206000640 1 --2- 172.21.15.39:0/1262296742 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559c71ff2200 0x559c71ffc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:18:32.678 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.649+0000 7f61f0c00640 1 -- 172.21.15.39:0/1262296742 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6200039040 con 0x559c71ff1810 2024-03-18T16:18:32.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.649+0000 7f61f0c00640 1 -- 172.21.15.39:0/1262296742 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f620003fd60 con 0x559c71ff1810 2024-03-18T16:18:32.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.650+0000 7f6210793740 1 -- 172.21.15.39:0/1262296742 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6200032cb0 con 0x559c71ff1810 2024-03-18T16:18:32.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.650+0000 7f61f0c00640 1 -- 172.21.15.39:0/1262296742 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f620004b480 con 0x559c71ff1810 2024-03-18T16:18:32.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.651+0000 7f6210793740 10 client.16701 fetch_fsmap learned FSMap version 44 2024-03-18T16:18:32.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.651+0000 7f6210793740 1 -- 172.21.15.39:0/1262296742 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x559c71c2a0f0 con 0x559c71ff1810 2024-03-18T16:18:32.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.651+0000 7f61f0c00640 1 -- 172.21.15.39:0/1262296742 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6200032cb0 con 0x559c71ff1810 2024-03-18T16:18:32.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.652+0000 7f61f0c00640 1 -- 172.21.15.39:0/1262296742 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7f62000498d0 con 0x559c71ff1810 2024-03-18T16:18:32.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.652+0000 7f6210793740 10 client.16701 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:18:32.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.653+0000 7f6210793740 10 client.16701 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:18:32.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.653+0000 7f6210793740 20 client.16701 populate_metadata read hostname 'smithi039' 2024-03-18T16:18:32.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.653+0000 7f6210793740 1 --2- 172.21.15.39:0/1262296742 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x559c72007720 0x559c72009b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:32.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.653+0000 7f6210793740 4 client.16701 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:18:32.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.653+0000 7f6210793740 1 -- 172.21.15.39:0/1262296742 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559c71f69be0 con 0x559c72007720 2024-03-18T16:18:32.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.654+0000 7f6206000640 1 --2- 172.21.15.39:0/1262296742 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x559c72007720 0x559c72009b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:32.679 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.655+0000 7f6206000640 1 --2- 172.21.15.39:0/1262296742 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x559c72007720 0x559c72009b10 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b27f21eca71e9825 server_cookie=fe13b8910929a551 in_seq=0 out_seq=0 2024-03-18T16:18:32.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.655+0000 7f61f0c00640 10 client.16701 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:18:32.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.658+0000 7f61f0c00640 1 -- 172.21.15.39:0/1262296742 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559c71f69be0 con 0x559c72007720 2024-03-18T16:18:32.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.658+0000 7f61f0c00640 10 client.16701 handle_command_reply: tid=0 2024-03-18T16:18:32.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.658+0000 7f6210793740 1 client.16701 shutdown 2024-03-18T16:18:32.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.659+0000 7f6210793740 1 -- 172.21.15.39:0/1262296742 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559c71ff1810 msgr2=0x559c71ffbd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:18:32.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.659+0000 7f6210793740 1 --2- 172.21.15.39:0/1262296742 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559c71ff1810 0x559c71ffbd90 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f6200005e30 tx=0x7f6200032e60 comp rx=0 tx=0).stop 2024-03-18T16:18:32.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.659+0000 7f6210793740 1 -- 172.21.15.39:0/1262296742 shutdown_connections 2024-03-18T16:18:32.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.659+0000 7f6210793740 1 --2- 172.21.15.39:0/1262296742 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x559c72007720 0x559c72009b10 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:32.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.659+0000 7f6210793740 1 --2- 172.21.15.39:0/1262296742 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559c71ff7370 0x559c72003310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:32.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.660+0000 7f6210793740 1 --2- 172.21.15.39:0/1262296742 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559c71ff2200 0x559c71ffc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:32.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.660+0000 7f6210793740 1 --2- 172.21.15.39:0/1262296742 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559c71ff1810 0x559c71ffbd90 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:32.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.659+0000 7f61f0c00640 0 client.16701 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:18:32.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.660+0000 7f6210793740 1 -- 172.21.15.39:0/1262296742 >> 172.21.15.39:0/1262296742 conn(0x559c71fecd80 msgr2=0x559c71febec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:18:32.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.660+0000 7f6210793740 1 -- 172.21.15.39:0/1262296742 shutdown_connections 2024-03-18T16:18:32.680 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.660+0000 7f6210793740 1 -- 172.21.15.39:0/1262296742 wait complete. 2024-03-18T16:18:32.681 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.661+0000 7f6210793740 20 client.16701 trim_cache size 0 max 16384 2024-03-18T16:18:32.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.697+0000 7f6210793740 1 Processor -- start 2024-03-18T16:18:32.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.697+0000 7f6210793740 1 -- start start 2024-03-18T16:18:32.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.697+0000 7f6210793740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559c71ff1810 0x559c71ffa870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:32.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.697+0000 7f6210793740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559c71ff2200 0x559c71ffadb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:32.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.697+0000 7f6210793740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559c71ff7370 0x559c720032a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:32.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.697+0000 7f6210793740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f61e800fbd0 con 0x559c71ff2200 2024-03-18T16:18:32.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.697+0000 7f6210793740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f61e8011710 con 0x559c71ff7370 2024-03-18T16:18:32.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.697+0000 7f6210793740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f61e80102a0 con 0x559c71ff1810 2024-03-18T16:18:32.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.698+0000 7f6207400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559c71ff1810 0x559c71ffa870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:32.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.698+0000 7f6207400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559c71ff1810 0x559c71ffa870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:49450/0 (socket says 172.21.15.39:49450) 2024-03-18T16:18:32.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.698+0000 7f6207400640 1 -- 172.21.15.39:0/4055732275 learned_addr learned my addr 172.21.15.39:0/4055732275 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:18:32.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.698+0000 7f6207400640 1 --2- 172.21.15.39:0/4055732275 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559c71ff7370 0x559c720032a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:32.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.698+0000 7f6206000640 1 --2- 172.21.15.39:0/4055732275 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559c71ff2200 0x559c71ffadb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:32.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.698+0000 7f6207400640 1 -- 172.21.15.39:0/4055732275 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559c71ff7370 msgr2=0x559c720032a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:18:32.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.698+0000 7f6207400640 1 --2- 172.21.15.39:0/4055732275 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559c71ff7370 0x559c720032a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:32.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.698+0000 7f6207400640 1 -- 172.21.15.39:0/4055732275 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559c71ff2200 msgr2=0x559c71ffadb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:18:32.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.698+0000 7f6207400640 1 --2- 172.21.15.39:0/4055732275 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559c71ff2200 0x559c71ffadb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:32.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.698+0000 7f6207400640 1 -- 172.21.15.39:0/4055732275 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559c71fe7370 con 0x559c71ff1810 2024-03-18T16:18:32.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.699+0000 7f6207400640 1 --2- 172.21.15.39:0/4055732275 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559c71ff1810 0x559c71ffa870 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f6200009430 tx=0x7f62000325b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9b7b2f87faf0c057 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:18:32.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.699+0000 7f61f3400640 1 -- 172.21.15.39:0/4055732275 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6200039040 con 0x559c71ff1810 2024-03-18T16:18:32.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.699+0000 7f61f3400640 1 -- 172.21.15.39:0/4055732275 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6200032cb0 con 0x559c71ff1810 2024-03-18T16:18:32.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.699+0000 7f61f3400640 1 -- 172.21.15.39:0/4055732275 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62000498e0 con 0x559c71ff1810 2024-03-18T16:18:32.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.700+0000 7f6210793740 1 -- 172.21.15.39:0/4055732275 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559c71ff1810 msgr2=0x559c71ffa870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:18:32.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.700+0000 7f6210793740 1 --2- 172.21.15.39:0/4055732275 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559c71ff1810 0x559c71ffa870 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f6200009430 tx=0x7f62000325b0 comp rx=0 tx=0).stop 2024-03-18T16:18:32.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.701+0000 7f6210793740 1 -- 172.21.15.39:0/4055732275 shutdown_connections 2024-03-18T16:18:32.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.701+0000 7f6210793740 1 --2- 172.21.15.39:0/4055732275 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559c71ff7370 0x559c720032a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:32.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.701+0000 7f6210793740 1 --2- 172.21.15.39:0/4055732275 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559c71ff2200 0x559c71ffadb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:32.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.701+0000 7f6210793740 1 --2- 172.21.15.39:0/4055732275 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559c71ff1810 0x559c71ffa870 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:32.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.701+0000 7f6210793740 1 -- 172.21.15.39:0/4055732275 >> 172.21.15.39:0/4055732275 conn(0x559c71fecd80 msgr2=0x559c71ff2ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:18:32.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.702+0000 7f6210793740 1 -- 172.21.15.39:0/4055732275 shutdown_connections 2024-03-18T16:18:32.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.702+0000 7f6210793740 1 -- 172.21.15.39:0/4055732275 wait complete. 2024-03-18T16:18:32.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.702+0000 7f6210793740 1 -- start start 2024-03-18T16:18:32.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.703+0000 7f6210793740 1 Processor -- start 2024-03-18T16:18:32.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.703+0000 7f6210793740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559c71ff1810 0x559c71ffe680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:32.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.704+0000 7f6210793740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559c71ff2200 0x559c71ffebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:32.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.704+0000 7f6210793740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559c71ff7370 0x559c72001be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:32.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.704+0000 7f6207400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559c71ff1810 0x559c71ffe680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:32.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.704+0000 7f6207400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559c71ff1810 0x559c71ffe680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:49466/0 (socket says 172.21.15.39:49466) 2024-03-18T16:18:32.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.704+0000 7f6207400640 1 -- 172.21.15.39:0/3377958035 learned_addr learned my addr 172.21.15.39:0/3377958035 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:18:32.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.704+0000 7f6210793740 1 -- 172.21.15.39:0/3377958035 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f61e8011040 con 0x559c71ff2200 2024-03-18T16:18:32.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.704+0000 7f6210793740 1 -- 172.21.15.39:0/3377958035 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f61e8010970 con 0x559c71ff7370 2024-03-18T16:18:32.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.704+0000 7f6210793740 1 -- 172.21.15.39:0/3377958035 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f61e800dbb0 con 0x559c71ff1810 2024-03-18T16:18:32.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.704+0000 7f6210793740 10 client.0 _ll_register_callbacks cb 0x7f61f8005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:18:32.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.705+0000 7f6206000640 1 --2- 172.21.15.39:0/3377958035 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559c71ff2200 0x559c71ffebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:32.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.705+0000 7f6207400640 1 --2- 172.21.15.39:0/3377958035 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559c71ff7370 0x559c72001be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:32.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.705+0000 7f6207400640 1 -- 172.21.15.39:0/3377958035 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559c71ff7370 msgr2=0x559c72001be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:18:32.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.705+0000 7f6207400640 1 --2- 172.21.15.39:0/3377958035 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559c71ff7370 0x559c72001be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:32.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.705+0000 7f6207400640 1 -- 172.21.15.39:0/3377958035 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559c71ff2200 msgr2=0x559c71ffebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:18:32.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.705+0000 7f6207400640 1 --2- 172.21.15.39:0/3377958035 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559c71ff2200 0x559c71ffebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:32.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.705+0000 7f6207400640 1 -- 172.21.15.39:0/3377958035 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559c720021d0 con 0x559c71ff1810 2024-03-18T16:18:32.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.706+0000 7f6207400640 1 --2- 172.21.15.39:0/3377958035 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559c71ff1810 0x559c71ffe680 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f6200005d50 tx=0x7f6200009bb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aabfe9f490347a83 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:18:32.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.706+0000 7f61f0c00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:18:32.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.707+0000 7f61f0c00640 1 -- 172.21.15.39:0/3377958035 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6200039070 con 0x559c71ff1810 2024-03-18T16:18:32.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.707+0000 7f61f0c00640 1 -- 172.21.15.39:0/3377958035 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f620003fa60 con 0x559c71ff1810 2024-03-18T16:18:32.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.708+0000 7f6210793740 1 -- 172.21.15.39:0/3377958035 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6200032cb0 con 0x559c71ff1810 2024-03-18T16:18:32.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.708+0000 7f61f0c00640 1 -- 172.21.15.39:0/3377958035 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f6200032e60 con 0x559c71ff1810 2024-03-18T16:18:32.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.708+0000 7f6210793740 10 client.27316 fetch_fsmap learned FSMap version 44 2024-03-18T16:18:32.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.708+0000 7f6210793740 1 -- 172.21.15.39:0/3377958035 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x559c720034e0 con 0x559c71ff1810 2024-03-18T16:18:32.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.708+0000 7f61f0c00640 1 -- 172.21.15.39:0/3377958035 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6200032cb0 con 0x559c71ff1810 2024-03-18T16:18:32.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.709+0000 7f61f0c00640 1 -- 172.21.15.39:0/3377958035 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7f620005f030 con 0x559c71ff1810 2024-03-18T16:18:32.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.709+0000 7f6210793740 10 client.27316 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:18:32.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.709+0000 7f6210793740 10 client.27316 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:18:32.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.709+0000 7f6210793740 20 client.27316 populate_metadata read hostname 'smithi039' 2024-03-18T16:18:32.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.709+0000 7f6210793740 1 --2- 172.21.15.39:0/3377958035 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x559c7207ee90 0x559c720812b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:18:32.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.709+0000 7f6210793740 4 client.27316 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T16:18:32.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.709+0000 7f6210793740 1 -- 172.21.15.39:0/3377958035 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "scrub status"}) -- 0x559c71f63730 con 0x559c7207ee90 2024-03-18T16:18:32.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.710+0000 7f6206000640 1 --2- 172.21.15.39:0/3377958035 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x559c7207ee90 0x559c720812b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:18:32.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.711+0000 7f6206000640 1 --2- 172.21.15.39:0/3377958035 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x559c7207ee90 0x559c720812b0 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=36f1af5c8e99c1e5 server_cookie=32a94dbb47b821f7 in_seq=0 out_seq=0 2024-03-18T16:18:32.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.711+0000 7f61f0c00640 10 client.27316 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:18:32.713 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.712+0000 7f61f0c00640 1 -- 172.21.15.39:0/3377958035 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x559c71f63730 con 0x559c7207ee90 2024-03-18T16:18:32.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.712+0000 7f61f0c00640 10 client.27316 handle_command_reply: tid=0 2024-03-18T16:18:32.714 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.712+0000 7f6210793740 1 client.27316 shutdown 2024-03-18T16:18:32.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.712+0000 7f6210793740 1 -- 172.21.15.39:0/3377958035 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559c71ff1810 msgr2=0x559c71ffe680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:18:32.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.712+0000 7f6210793740 1 --2- 172.21.15.39:0/3377958035 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559c71ff1810 0x559c71ffe680 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f6200005d50 tx=0x7f6200009bb0 comp rx=0 tx=0).stop 2024-03-18T16:18:32.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.713+0000 7f6210793740 1 -- 172.21.15.39:0/3377958035 shutdown_connections 2024-03-18T16:18:32.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.713+0000 7f6210793740 1 --2- 172.21.15.39:0/3377958035 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x559c7207ee90 0x559c720812b0 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:32.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.713+0000 7f6210793740 1 --2- 172.21.15.39:0/3377958035 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559c71ff7370 0x559c72001be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:32.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.713+0000 7f6210793740 1 --2- 172.21.15.39:0/3377958035 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559c71ff2200 0x559c71ffebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:32.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.713+0000 7f6210793740 1 --2- 172.21.15.39:0/3377958035 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559c71ff1810 0x559c71ffe680 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:32.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.713+0000 7f6210793740 1 -- 172.21.15.39:0/3377958035 >> 172.21.15.39:0/3377958035 conn(0x559c71fecd80 msgr2=0x559c71ff37a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:18:32.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.713+0000 7f61f0c00640 0 client.27316 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:18:32.715 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.713+0000 7f6210793740 1 -- 172.21.15.39:0/3377958035 shutdown_connections 2024-03-18T16:18:32.716 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.714+0000 7f6210793740 1 -- 172.21.15.39:0/3377958035 wait complete. 2024-03-18T16:18:32.716 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.715+0000 7f6210793740 20 client.27316 trim_cache size 0 max 16384 2024-03-18T16:18:32.716 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-03-18T16:18:32.717 INFO:teuthology.orchestra.run.smithi039.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-18T16:18:32.717 INFO:teuthology.orchestra.run.smithi039.stdout: "scrubs": { 2024-03-18T16:18:32.717 INFO:teuthology.orchestra.run.smithi039.stdout: "d7db728c-b784-4575-bb83-48c352cec3bf": { 2024-03-18T16:18:32.717 INFO:teuthology.orchestra.run.smithi039.stdout: "path": "/", 2024-03-18T16:18:32.717 INFO:teuthology.orchestra.run.smithi039.stdout: "tag": "d7db728c-b784-4575-bb83-48c352cec3bf", 2024-03-18T16:18:32.717 INFO:teuthology.orchestra.run.smithi039.stdout: "options": "recursive,force" 2024-03-18T16:18:32.717 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:18:32.717 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:18:32.717 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-03-18T16:18:32.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.716+0000 7f620da00640 1 -- 172.21.15.39:0/3462858938 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f61ec0a2880 msgr2=0x7f61ec0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:18:32.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.716+0000 7f620da00640 1 --2- 172.21.15.39:0/3462858938 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f61ec0a2880 0x7f61ec0a4d40 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f61f40a39f0 tx=0x7f61f4020000 comp rx=0 tx=0).stop 2024-03-18T16:18:32.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.716+0000 7f620da00640 1 -- 172.21.15.39:0/3462858938 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6208107e00 msgr2=0x7f620819e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:18:32.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.716+0000 7f620da00640 1 --2- 172.21.15.39:0/3462858938 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6208107e00 0x7f620819e080 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f61f400e9c0 tx=0x7f61f400ee80 comp rx=0 tx=0).stop 2024-03-18T16:18:32.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.716+0000 7f620da00640 1 -- 172.21.15.39:0/3462858938 shutdown_connections 2024-03-18T16:18:32.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.716+0000 7f620da00640 1 --2- 172.21.15.39:0/3462858938 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f61ec0a2880 0x7f61ec0a4d40 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:32.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.716+0000 7f620da00640 1 --2- 172.21.15.39:0/3462858938 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f62081099b0 0x7f62081a5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:32.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.716+0000 7f620da00640 1 --2- 172.21.15.39:0/3462858938 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6208108ff0 0x7f620819e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:32.718 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.716+0000 7f620da00640 1 --2- 172.21.15.39:0/3462858938 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6208107e00 0x7f620819e080 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:18:32.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.716+0000 7f620da00640 1 -- 172.21.15.39:0/3462858938 >> 172.21.15.39:0/3462858938 conn(0x7f6208076da0 msgr2=0x7f620810c2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:18:32.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.716+0000 7f620da00640 1 -- 172.21.15.39:0/3462858938 shutdown_connections 2024-03-18T16:18:32.719 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:18:32.716+0000 7f620da00640 1 -- 172.21.15.39:0/3462858938 wait complete. 2024-03-18T16:18:32.743 INFO:tasks.cephfs.filesystem:scrub status for tag:d7db728c-b784-4575-bb83-48c352cec3bf - {'path': '/', 'tag': 'd7db728c-b784-4575-bb83-48c352cec3bf', 'options': 'recursive,force'} 2024-03-18T16:18:33.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:18:32 smithi039 ceph-mon[31721]: pgmap v676: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 6.0 MiB/s wr, 1.42k op/s 2024-03-18T16:18:33.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:18:32 smithi039 ceph-mon[31721]: scrub queued for path: / 2024-03-18T16:18:33.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:18:32 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:18:33.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:18:32 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:18:33.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:18:32 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:18:33.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:18:32 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:18:33.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:18:32 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:18:33.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:18:32 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:18:33.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:18:32 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:18:33.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:18:32 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:18:33.087 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:18:32 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:18:33.088 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:18:32 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:18:33.088 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:18:32 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:18:33.088 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:18:32 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:18:33.088 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:18:32 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:18:33.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:18:32 smithi196 ceph-mon[36307]: pgmap v676: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 6.0 MiB/s wr, 1.42k op/s 2024-03-18T16:18:33.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:18:32 smithi196 ceph-mon[36307]: scrub queued for path: / 2024-03-18T16:18:33.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:18:32 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:18:33.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:18:32 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:18:33.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:18:32 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:18:33.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:18:32 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:18:33.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:18:32 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:18:33.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:18:32 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:18:33.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:18:32 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:18:33.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:18:32 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:18:33.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:18:32 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:18:33.230 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:18:32 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:18:33.230 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:18:32 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:18:33.230 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:18:32 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:18:33.230 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:18:32 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:18:33.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:18:32 smithi194 ceph-mon[36200]: pgmap v676: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 6.0 MiB/s wr, 1.42k op/s 2024-03-18T16:18:33.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:18:32 smithi194 ceph-mon[36200]: scrub queued for path: / 2024-03-18T16:18:33.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:18:32 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:18:33.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:18:32 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:18:33.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:18:32 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:18:33.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:18:32 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:18:33.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:18:32 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:18:33.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:18:32 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:18:33.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:18:32 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:18:33.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:18:32 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:18:33.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:18:32 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:18:33.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:18:32 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:18:33.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:18:32 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:18:33.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:18:32 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:18:33.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:18:32 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:18:34.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:18:33 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:18:34.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:18:33 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:18:34.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:18:33 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:18:34.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:18:33 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:18:34.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:18:33 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:18:34.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:18:33 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:18:34.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:18:33 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:18:34.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:18:33 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:18:34.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:18:33 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:18:34.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:18:33 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:18:34.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:18:33 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:18:34.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:18:33 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:18:34.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:18:33 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:18:34.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:18:33 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:18:34.230 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:18:33 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:18:34.230 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:18:33 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:18:34.230 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:18:33 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:18:34.230 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:18:33 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:18:34.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:18:33 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:18:34.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:18:33 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:18:34.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:18:33 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:18:34.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:18:33 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:18:34.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:18:33 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:18:34.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:18:33 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:18:34.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:18:33 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:18:34.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:18:33 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:18:34.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:18:33 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:18:34.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:18:33 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:18:34.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:18:33 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:18:34.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:18:33 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:18:34.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:18:33 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:18:34.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:18:33 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:18:34.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:18:33 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:18:34.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:18:33 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:18:34.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:18:33 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:18:34.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:18:33 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:18:34.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:18:33 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:18:34.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:18:33 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:18:34.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:18:33 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:18:34.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:18:33 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:18:34.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:18:33 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:18:34.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:18:33 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:18:34.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:18:33 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:18:34.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:18:33 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:18:34.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:18:33 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:18:34.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:18:33 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:18:34.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:18:33 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:18:34.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:18:33 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:18:34.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:18:33 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:18:34.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:18:33 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:18:34.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:18:33 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:18:34.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:18:33 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:18:34.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:18:33 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:18:34.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:18:33 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:18:35.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:18:34 smithi194 ceph-mon[36200]: pgmap v677: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 5.3 MiB/s wr, 1.26k op/s 2024-03-18T16:18:35.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:18:34 smithi039 ceph-mon[31721]: pgmap v677: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 5.3 MiB/s wr, 1.26k op/s 2024-03-18T16:18:35.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:18:34 smithi196 ceph-mon[36307]: pgmap v677: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 5.3 MiB/s wr, 1.26k op/s 2024-03-18T16:18:37.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:18:36 smithi194 ceph-mon[36200]: pgmap v678: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 3.9 MiB/s wr, 940 op/s 2024-03-18T16:18:37.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:18:36 smithi039 ceph-mon[31721]: pgmap v678: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 3.9 MiB/s wr, 940 op/s 2024-03-18T16:18:37.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:18:36 smithi196 ceph-mon[36307]: pgmap v678: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 3.9 MiB/s wr, 940 op/s 2024-03-18T16:18:39.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:18:38 smithi194 ceph-mon[36200]: pgmap v679: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 3.2 MiB/s wr, 777 op/s 2024-03-18T16:18:39.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:18:38 smithi039 ceph-mon[31721]: pgmap v679: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 3.2 MiB/s wr, 777 op/s 2024-03-18T16:18:39.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:18:38 smithi196 ceph-mon[36307]: pgmap v679: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 3.2 MiB/s wr, 777 op/s 2024-03-18T16:18:41.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:18:40 smithi039 ceph-mon[31721]: pgmap v680: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 1.3 MiB/s wr, 347 op/s 2024-03-18T16:18:41.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:18:40 smithi196 ceph-mon[36307]: pgmap v680: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 1.3 MiB/s wr, 347 op/s 2024-03-18T16:18:41.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:18:40 smithi194 ceph-mon[36200]: pgmap v680: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 1.3 MiB/s wr, 347 op/s 2024-03-18T16:18:43.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:18:42 smithi039 ceph-mon[31721]: pgmap v681: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.4 MiB/s wr, 359 op/s 2024-03-18T16:18:43.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:18:42 smithi039 ceph-mon[31721]: scrub summary: idle 2024-03-18T16:18:43.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:18:42 smithi039 ceph-mon[31721]: scrub summary: idle 2024-03-18T16:18:43.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:18:42 smithi039 ceph-mon[31721]: scrub summary: idle 2024-03-18T16:18:43.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:18:42 smithi196 ceph-mon[36307]: pgmap v681: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.4 MiB/s wr, 359 op/s 2024-03-18T16:18:43.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:18:42 smithi196 ceph-mon[36307]: scrub summary: idle 2024-03-18T16:18:43.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:18:42 smithi196 ceph-mon[36307]: scrub summary: idle 2024-03-18T16:18:43.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:18:42 smithi196 ceph-mon[36307]: scrub summary: idle 2024-03-18T16:18:43.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:18:42 smithi194 ceph-mon[36200]: pgmap v681: 129 pgs: 129 active+clean; 13 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.4 MiB/s wr, 359 op/s 2024-03-18T16:18:43.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:18:42 smithi194 ceph-mon[36200]: scrub summary: idle 2024-03-18T16:18:43.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:18:42 smithi194 ceph-mon[36200]: scrub summary: idle 2024-03-18T16:18:43.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:18:42 smithi194 ceph-mon[36200]: scrub summary: idle 2024-03-18T16:18:44.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:18:43 smithi039 ceph-mon[31721]: scrub summary: idle 2024-03-18T16:18:44.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:18:43 smithi039 ceph-mon[31721]: scrub summary: idle 2024-03-18T16:18:44.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:18:43 smithi196 ceph-mon[36307]: scrub summary: idle 2024-03-18T16:18:44.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:18:43 smithi196 ceph-mon[36307]: scrub summary: idle 2024-03-18T16:18:44.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:18:43 smithi194 ceph-mon[36200]: scrub summary: idle 2024-03-18T16:18:44.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:18:43 smithi194 ceph-mon[36200]: scrub summary: idle 2024-03-18T16:18:45.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:18:45 smithi039 ceph-mon[31721]: pgmap v682: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 1.4 MiB/s wr, 361 op/s 2024-03-18T16:18:45.391 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:18:45 smithi194 ceph-mon[36200]: pgmap v682: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 1.4 MiB/s wr, 361 op/s 2024-03-18T16:18:45.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:18:45 smithi196 ceph-mon[36307]: pgmap v682: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 1.4 MiB/s wr, 361 op/s 2024-03-18T16:18:47.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:18:47 smithi039 ceph-mon[31721]: pgmap v683: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.2 MiB/s wr, 328 op/s 2024-03-18T16:18:47.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:18:47 smithi196 ceph-mon[36307]: pgmap v683: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.2 MiB/s wr, 328 op/s 2024-03-18T16:18:47.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:18:47 smithi194 ceph-mon[36200]: pgmap v683: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.2 MiB/s wr, 328 op/s 2024-03-18T16:18:49.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:18:49 smithi039 ceph-mon[31721]: pgmap v684: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.3 MiB/s wr, 345 op/s 2024-03-18T16:18:49.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:18:49 smithi196 ceph-mon[36307]: pgmap v684: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.3 MiB/s wr, 345 op/s 2024-03-18T16:18:49.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:18:49 smithi194 ceph-mon[36200]: pgmap v684: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.3 MiB/s wr, 345 op/s 2024-03-18T16:18:51.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:18:51 smithi039 ceph-mon[31721]: pgmap v685: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.2 MiB/s wr, 304 op/s 2024-03-18T16:18:51.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:18:51 smithi196 ceph-mon[36307]: pgmap v685: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.2 MiB/s wr, 304 op/s 2024-03-18T16:18:51.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:18:51 smithi194 ceph-mon[36200]: pgmap v685: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.2 MiB/s wr, 304 op/s 2024-03-18T16:18:53.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:18:53 smithi039 ceph-mon[31721]: pgmap v686: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.3 MiB/s wr, 342 op/s 2024-03-18T16:18:53.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:18:53 smithi196 ceph-mon[36307]: pgmap v686: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.3 MiB/s wr, 342 op/s 2024-03-18T16:18:53.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:18:53 smithi194 ceph-mon[36200]: pgmap v686: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.3 MiB/s wr, 342 op/s 2024-03-18T16:18:55.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:18:55 smithi039 ceph-mon[31721]: pgmap v687: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.2 MiB/s wr, 319 op/s 2024-03-18T16:18:55.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:18:55 smithi196 ceph-mon[36307]: pgmap v687: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.2 MiB/s wr, 319 op/s 2024-03-18T16:18:55.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:18:55 smithi194 ceph-mon[36200]: pgmap v687: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.2 MiB/s wr, 319 op/s 2024-03-18T16:18:57.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:18:57 smithi039 ceph-mon[31721]: pgmap v688: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 1.0 MiB/s wr, 260 op/s 2024-03-18T16:18:57.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:18:57 smithi196 ceph-mon[36307]: pgmap v688: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 1.0 MiB/s wr, 260 op/s 2024-03-18T16:18:57.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:18:57 smithi194 ceph-mon[36200]: pgmap v688: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 1.0 MiB/s wr, 260 op/s 2024-03-18T16:18:59.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:18:59 smithi039 ceph-mon[31721]: pgmap v689: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.0 MiB/s wr, 268 op/s 2024-03-18T16:18:59.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:18:59 smithi196 ceph-mon[36307]: pgmap v689: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.0 MiB/s wr, 268 op/s 2024-03-18T16:18:59.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:18:59 smithi194 ceph-mon[36200]: pgmap v689: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.0 MiB/s wr, 268 op/s 2024-03-18T16:19:01.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:01 smithi039 ceph-mon[31721]: pgmap v690: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 890 KiB/s wr, 224 op/s 2024-03-18T16:19:01.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:01 smithi196 ceph-mon[36307]: pgmap v690: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 890 KiB/s wr, 224 op/s 2024-03-18T16:19:01.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:01 smithi194 ceph-mon[36200]: pgmap v690: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 890 KiB/s wr, 224 op/s 2024-03-18T16:19:02.744 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T16:19:02.744 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T16:19:02.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.983+0000 7ff06cc00640 1 -- 172.21.15.39:0/355061653 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff0681018d0 msgr2=0x7ff068101d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:02.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.983+0000 7ff06cc00640 1 --2- 172.21.15.39:0/355061653 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff0681018d0 0x7ff068101d50 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7ff050009900 tx=0x7ff05002bc60 comp rx=0 tx=0).stop 2024-03-18T16:19:02.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.984+0000 7ff06cc00640 1 -- 172.21.15.39:0/355061653 shutdown_connections 2024-03-18T16:19:02.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.984+0000 7ff06cc00640 1 --2- 172.21.15.39:0/355061653 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff0681018d0 0x7ff068101d50 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:02.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.984+0000 7ff06cc00640 1 --2- 172.21.15.39:0/355061653 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff068102290 0x7ff068108b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:02.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.984+0000 7ff06cc00640 1 --2- 172.21.15.39:0/355061653 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff0681006e0 0x7ff068100b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:02.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.984+0000 7ff06cc00640 1 -- 172.21.15.39:0/355061653 >> 172.21.15.39:0/355061653 conn(0x7ff0680fbc70 msgr2=0x7ff0680fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:02.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.984+0000 7ff06cc00640 1 -- 172.21.15.39:0/355061653 shutdown_connections 2024-03-18T16:19:02.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.984+0000 7ff06cc00640 1 -- 172.21.15.39:0/355061653 wait complete. 2024-03-18T16:19:02.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.985+0000 7ff06cc00640 1 Processor -- start 2024-03-18T16:19:02.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.986+0000 7ff06cc00640 1 -- start start 2024-03-18T16:19:02.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.986+0000 7ff06cc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff0681006e0 0x7ff06819e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:02.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.986+0000 7ff06cc00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff0681018d0 0x7ff06819e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:02.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.986+0000 7ff06cc00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff068102290 0x7ff0681a56e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:02.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.986+0000 7ff06cc00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7ff06810aca0 con 0x7ff0681006e0 2024-03-18T16:19:02.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.986+0000 7ff06cc00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ff06810ab20 con 0x7ff068102290 2024-03-18T16:19:02.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.986+0000 7ff06cc00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7ff06810ae20 con 0x7ff0681018d0 2024-03-18T16:19:02.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.987+0000 7ff065400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff0681018d0 0x7ff06819e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:02.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.987+0000 7ff065400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff0681018d0 0x7ff06819e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:59320/0 (socket says 172.21.15.39:59320) 2024-03-18T16:19:02.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.987+0000 7ff065400640 1 -- 172.21.15.39:0/1914045526 learned_addr learned my addr 172.21.15.39:0/1914045526 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:19:02.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.987+0000 7ff065e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff0681006e0 0x7ff06819e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:02.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.987+0000 7ff066800640 1 --2- 172.21.15.39:0/1914045526 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff068102290 0x7ff0681a56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:02.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.987+0000 7ff065400640 1 -- 172.21.15.39:0/1914045526 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff068102290 msgr2=0x7ff0681a56e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:02.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.987+0000 7ff065400640 1 --2- 172.21.15.39:0/1914045526 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff068102290 0x7ff0681a56e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:02.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.987+0000 7ff065400640 1 -- 172.21.15.39:0/1914045526 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff0681006e0 msgr2=0x7ff06819e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:02.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.987+0000 7ff065400640 1 --2- 172.21.15.39:0/1914045526 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff0681006e0 0x7ff06819e130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:02.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.987+0000 7ff065400640 1 -- 172.21.15.39:0/1914045526 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0681a5d50 con 0x7ff0681018d0 2024-03-18T16:19:02.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.987+0000 7ff066800640 1 --2- 172.21.15.39:0/1914045526 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff068102290 0x7ff0681a56e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:19:02.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.988+0000 7ff065400640 1 --2- 172.21.15.39:0/1914045526 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff0681018d0 0x7ff06819e670 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7ff05002c8d0 tx=0x7ff050004e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:02.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.988+0000 7ff05ea00640 1 -- 172.21.15.39:0/1914045526 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff050043020 con 0x7ff0681018d0 2024-03-18T16:19:02.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.988+0000 7ff05ea00640 1 -- 172.21.15.39:0/1914045526 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff0500054e0 con 0x7ff0681018d0 2024-03-18T16:19:02.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.988+0000 7ff06cc00640 1 -- 172.21.15.39:0/1914045526 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0681a5fe0 con 0x7ff0681018d0 2024-03-18T16:19:02.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.988+0000 7ff05ea00640 1 -- 172.21.15.39:0/1914045526 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff050003ce0 con 0x7ff0681018d0 2024-03-18T16:19:02.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.988+0000 7ff06cc00640 1 -- 172.21.15.39:0/1914045526 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0681a6430 con 0x7ff0681018d0 2024-03-18T16:19:02.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.993+0000 7ff06f9e1740 1 Processor -- start 2024-03-18T16:19:02.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.993+0000 7ff05ea00640 1 -- 172.21.15.39:0/1914045526 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7ff050046020 con 0x7ff0681018d0 2024-03-18T16:19:02.996 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.993+0000 7ff06f9e1740 1 -- start start 2024-03-18T16:19:02.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.993+0000 7ff06f9e1740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5653a1484810 0x5653a1484c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:02.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.994+0000 7ff06f9e1740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5653a1485200 0x5653a1489d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:02.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.994+0000 7ff06f9e1740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5653a148a370 0x5653a148c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:02.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.994+0000 7ff06f9e1740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5653a13e91f0 con 0x5653a1484810 2024-03-18T16:19:02.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.994+0000 7ff06f9e1740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5653a1391a80 con 0x5653a148a370 2024-03-18T16:19:02.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.994+0000 7ff06f9e1740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x5653a13e5290 con 0x5653a1485200 2024-03-18T16:19:02.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.994+0000 7ff065e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5653a1485200 0x5653a1489d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:02.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.994+0000 7ff065e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5653a1485200 0x5653a1489d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:59330/0 (socket says 172.21.15.39:59330) 2024-03-18T16:19:02.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.994+0000 7ff065e00640 1 -- 172.21.15.39:0/4101459666 learned_addr learned my addr 172.21.15.39:0/4101459666 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:19:02.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.994+0000 7ff066800640 1 --2- 172.21.15.39:0/4101459666 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5653a1484810 0x5653a1484c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:02.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.994+0000 7ff065400640 1 --2- 172.21.15.39:0/4101459666 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5653a148a370 0x5653a148c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:02.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.994+0000 7ff065e00640 1 -- 172.21.15.39:0/4101459666 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5653a148a370 msgr2=0x5653a148c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:02.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.994+0000 7ff065e00640 1 --2- 172.21.15.39:0/4101459666 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5653a148a370 0x5653a148c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:02.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.994+0000 7ff065e00640 1 -- 172.21.15.39:0/4101459666 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5653a1484810 msgr2=0x5653a1484c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:02.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.994+0000 7ff065e00640 1 --2- 172.21.15.39:0/4101459666 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5653a1484810 0x5653a1484c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:02.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.994+0000 7ff065e00640 1 -- 172.21.15.39:0/4101459666 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5653a11c77d0 con 0x5653a1485200 2024-03-18T16:19:02.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.994+0000 7ff065400640 1 --2- 172.21.15.39:0/4101459666 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5653a148a370 0x5653a148c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:19:02.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.994+0000 7ff066800640 1 --2- 172.21.15.39:0/4101459666 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5653a1484810 0x5653a1484c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:19:03.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.995+0000 7ff065e00640 1 --2- 172.21.15.39:0/4101459666 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5653a1485200 0x5653a1489d00 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7ff058009ca0 tx=0x7ff058011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aeb90c4eb2cc7a9a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:03.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.995+0000 7ff057000640 1 -- 172.21.15.39:0/4101459666 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff058004220 con 0x5653a1485200 2024-03-18T16:19:03.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.995+0000 7ff057000640 1 -- 172.21.15.39:0/4101459666 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff0580043c0 con 0x5653a1485200 2024-03-18T16:19:03.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.995+0000 7ff057000640 1 -- 172.21.15.39:0/4101459666 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0580198d0 con 0x5653a1485200 2024-03-18T16:19:03.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.995+0000 7ff05ea00640 1 --2- 172.21.15.39:0/1914045526 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff04c0a2860 0x7ff04c0a4d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:03.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.995+0000 7ff05ea00640 1 -- 172.21.15.39:0/1914045526 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7ff0500f0f50 con 0x7ff0681018d0 2024-03-18T16:19:03.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.996+0000 7ff066800640 1 --2- 172.21.15.39:0/1914045526 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff04c0a2860 0x7ff04c0a4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:03.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.996+0000 7ff06f9e1740 1 -- 172.21.15.39:0/4101459666 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5653a1485200 msgr2=0x5653a1489d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:03.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.996+0000 7ff06f9e1740 1 --2- 172.21.15.39:0/4101459666 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5653a1485200 0x5653a1489d00 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7ff058009ca0 tx=0x7ff058011810 comp rx=0 tx=0).stop 2024-03-18T16:19:03.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.996+0000 7ff06f9e1740 1 -- 172.21.15.39:0/4101459666 shutdown_connections 2024-03-18T16:19:03.000 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.996+0000 7ff06f9e1740 1 --2- 172.21.15.39:0/4101459666 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5653a148a370 0x5653a148c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:03.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.996+0000 7ff06f9e1740 1 --2- 172.21.15.39:0/4101459666 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5653a1485200 0x5653a1489d00 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:03.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.996+0000 7ff06f9e1740 1 --2- 172.21.15.39:0/4101459666 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5653a1484810 0x5653a1484c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:03.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.996+0000 7ff06f9e1740 1 -- 172.21.15.39:0/4101459666 >> 172.21.15.39:0/4101459666 conn(0x5653a147fd80 msgr2=0x5653a1480180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:03.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.997+0000 7ff06f9e1740 1 -- 172.21.15.39:0/4101459666 shutdown_connections 2024-03-18T16:19:03.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.997+0000 7ff066800640 1 --2- 172.21.15.39:0/1914045526 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff04c0a2860 0x7ff04c0a4d20 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x5653a1485a60 tx=0x7ff060009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:03.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.997+0000 7ff06f9e1740 1 -- 172.21.15.39:0/4101459666 wait complete. 2024-03-18T16:19:03.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.997+0000 7ff06f9e1740 1 -- start start 2024-03-18T16:19:03.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.998+0000 7ff06f9e1740 1 Processor -- start 2024-03-18T16:19:03.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.998+0000 7ff06f9e1740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5653a1484810 0x5653a148ed60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:03.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.998+0000 7ff06f9e1740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5653a1485200 0x5653a148f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:03.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.998+0000 7ff06f9e1740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5653a148a370 0x5653a1496260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:03.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.998+0000 7ff06f9e1740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5653a148d320 con 0x5653a1484810 2024-03-18T16:19:03.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.998+0000 7ff06f9e1740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5653a13a4ed0 con 0x5653a1485200 2024-03-18T16:19:03.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.998+0000 7ff06f9e1740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x5653a148d4a0 con 0x5653a148a370 2024-03-18T16:19:03.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.999+0000 7ff06f9e1740 10 client.0 _ll_register_callbacks cb 0x5653a13e0220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:19:03.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.999+0000 7ff065e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5653a1484810 0x5653a148ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:03.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.999+0000 7ff065400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5653a1485200 0x5653a148f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:03.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.999+0000 7ff065400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5653a1485200 0x5653a148f2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:34898/0 (socket says 172.21.15.39:34898) 2024-03-18T16:19:03.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.999+0000 7ff065400640 1 -- 172.21.15.39:0/758106375 learned_addr learned my addr 172.21.15.39:0/758106375 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:19:03.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.999+0000 7ff066800640 1 --2- 172.21.15.39:0/758106375 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5653a148a370 0x5653a1496260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:03.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.999+0000 7ff065400640 1 -- 172.21.15.39:0/758106375 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5653a148a370 msgr2=0x5653a1496260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:03.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.999+0000 7ff065400640 1 --2- 172.21.15.39:0/758106375 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5653a148a370 0x5653a1496260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:03.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.999+0000 7ff065400640 1 -- 172.21.15.39:0/758106375 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5653a1484810 msgr2=0x5653a148ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:03.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.999+0000 7ff065400640 1 --2- 172.21.15.39:0/758106375 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5653a1484810 0x5653a148ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:03.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:02.999+0000 7ff065400640 1 -- 172.21.15.39:0/758106375 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5653a10fd3f0 con 0x5653a1485200 2024-03-18T16:19:03.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.000+0000 7ff065400640 1 --2- 172.21.15.39:0/758106375 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5653a1485200 0x5653a148f2a0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7ff0500b9f30 tx=0x7ff0500b9200 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3daf96972ed1d0de server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:03.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.000+0000 7ff04b400640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:19:03.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.000+0000 7ff04b400640 1 -- 172.21.15.39:0/758106375 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0500bcec0 con 0x5653a1485200 2024-03-18T16:19:03.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.000+0000 7ff04b400640 1 -- 172.21.15.39:0/758106375 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7ff0500bb860 con 0x5653a1485200 2024-03-18T16:19:03.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.000+0000 7ff06f9e1740 1 -- 172.21.15.39:0/758106375 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff0580043c0 con 0x5653a1485200 2024-03-18T16:19:03.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.000+0000 7ff04b400640 1 -- 172.21.15.39:0/758106375 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7ff0500bdc60 con 0x5653a1485200 2024-03-18T16:19:03.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.000+0000 7ff06f9e1740 10 client.27347 fetch_fsmap learned FSMap version 44 2024-03-18T16:19:03.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.000+0000 7ff06f9e1740 1 -- 172.21.15.39:0/758106375 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x5653a10bd0f0 con 0x5653a1485200 2024-03-18T16:19:03.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.001+0000 7ff04b400640 1 -- 172.21.15.39:0/758106375 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff0580043c0 con 0x5653a1485200 2024-03-18T16:19:03.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.001+0000 7ff04b400640 1 -- 172.21.15.39:0/758106375 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7ff0500f30d0 con 0x5653a1485200 2024-03-18T16:19:03.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.002+0000 7ff06f9e1740 10 client.27347 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:19:03.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.002+0000 7ff06f9e1740 10 client.27347 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:19:03.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.002+0000 7ff06f9e1740 20 client.27347 populate_metadata read hostname 'smithi039' 2024-03-18T16:19:03.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.002+0000 7ff06f9e1740 1 --2- 172.21.15.39:0/758106375 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5653a149abd0 0x5653a149d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:03.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.002+0000 7ff06f9e1740 4 client.27347 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:19:03.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.002+0000 7ff06f9e1740 1 -- 172.21.15.39:0/758106375 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5653a13fcbe0 con 0x5653a149abd0 2024-03-18T16:19:03.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.002+0000 7ff065e00640 1 --2- 172.21.15.39:0/758106375 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5653a149abd0 0x5653a149d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:03.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.003+0000 7ff065e00640 1 --2- 172.21.15.39:0/758106375 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5653a149abd0 0x5653a149d020 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=107299566597e907 server_cookie=59c4b7899cca0e92 in_seq=0 out_seq=0 2024-03-18T16:19:03.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.003+0000 7ff04b400640 10 client.27347 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:19:03.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.004+0000 7ff04b400640 1 -- 172.21.15.39:0/758106375 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5653a13fcbe0 con 0x5653a149abd0 2024-03-18T16:19:03.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.004+0000 7ff04b400640 10 client.27347 handle_command_reply: tid=0 2024-03-18T16:19:03.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.004+0000 7ff06f9e1740 1 client.27347 shutdown 2024-03-18T16:19:03.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.004+0000 7ff06f9e1740 1 -- 172.21.15.39:0/758106375 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5653a1485200 msgr2=0x5653a148f2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:03.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.004+0000 7ff06f9e1740 1 --2- 172.21.15.39:0/758106375 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5653a1485200 0x5653a148f2a0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7ff0500b9f30 tx=0x7ff0500b9200 comp rx=0 tx=0).stop 2024-03-18T16:19:03.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.004+0000 7ff06f9e1740 1 -- 172.21.15.39:0/758106375 shutdown_connections 2024-03-18T16:19:03.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.004+0000 7ff06f9e1740 1 --2- 172.21.15.39:0/758106375 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5653a149abd0 0x5653a149d020 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:03.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.004+0000 7ff06f9e1740 1 --2- 172.21.15.39:0/758106375 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5653a1485200 0x5653a148f2a0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:03.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.004+0000 7ff06f9e1740 1 --2- 172.21.15.39:0/758106375 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5653a148a370 0x5653a1496260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:03.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.004+0000 7ff06f9e1740 1 --2- 172.21.15.39:0/758106375 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5653a1484810 0x5653a148ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:03.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.004+0000 7ff06f9e1740 1 -- 172.21.15.39:0/758106375 >> 172.21.15.39:0/758106375 conn(0x5653a147fd80 msgr2=0x5653a147eec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:03.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.004+0000 7ff04b400640 0 client.27347 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:19:03.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.005+0000 7ff06f9e1740 1 -- 172.21.15.39:0/758106375 shutdown_connections 2024-03-18T16:19:03.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.005+0000 7ff06f9e1740 1 -- 172.21.15.39:0/758106375 wait complete. 2024-03-18T16:19:03.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.005+0000 7ff06f9e1740 20 client.27347 trim_cache size 0 max 16384 2024-03-18T16:19:03.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.039+0000 7ff06f9e1740 1 Processor -- start 2024-03-18T16:19:03.042 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.040+0000 7ff06f9e1740 1 -- start start 2024-03-18T16:19:03.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.040+0000 7ff06f9e1740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5653a1484810 0x5653a147ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:03.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.040+0000 7ff06f9e1740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5653a1485200 0x5653a1496220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:03.043 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.040+0000 7ff06f9e1740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5653a148a370 0x5653a148d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:03.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.040+0000 7ff06f9e1740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7ff0681b09d0 con 0x5653a148a370 2024-03-18T16:19:03.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.040+0000 7ff06f9e1740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ff0681b2510 con 0x5653a1485200 2024-03-18T16:19:03.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.040+0000 7ff06f9e1740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7ff0681b10a0 con 0x5653a1484810 2024-03-18T16:19:03.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.041+0000 7ff065400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5653a1485200 0x5653a1496220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:03.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.041+0000 7ff065400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5653a1485200 0x5653a1496220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:34902/0 (socket says 172.21.15.39:34902) 2024-03-18T16:19:03.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.041+0000 7ff065400640 1 -- 172.21.15.39:0/3163603596 learned_addr learned my addr 172.21.15.39:0/3163603596 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:19:03.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.041+0000 7ff065400640 1 -- 172.21.15.39:0/3163603596 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5653a1484810 msgr2=0x5653a147ed40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:19:03.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.041+0000 7ff065400640 1 --2- 172.21.15.39:0/3163603596 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5653a1484810 0x5653a147ed40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:03.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.041+0000 7ff065400640 1 -- 172.21.15.39:0/3163603596 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5653a148a370 msgr2=0x5653a148d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:03.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.041+0000 7ff065400640 1 --2- 172.21.15.39:0/3163603596 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5653a148a370 0x5653a148d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:03.044 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.041+0000 7ff065400640 1 -- 172.21.15.39:0/3163603596 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5653a147a370 con 0x5653a1485200 2024-03-18T16:19:03.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.041+0000 7ff065400640 1 --2- 172.21.15.39:0/3163603596 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5653a1485200 0x5653a1496220 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7ff0500bd260 tx=0x7ff0500c5870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a803abc872135e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:03.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.042+0000 7ff056600640 1 -- 172.21.15.39:0/3163603596 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0500bcec0 con 0x5653a1485200 2024-03-18T16:19:03.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.042+0000 7ff056600640 1 -- 172.21.15.39:0/3163603596 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff0500c16b0 con 0x5653a1485200 2024-03-18T16:19:03.045 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.042+0000 7ff056600640 1 -- 172.21.15.39:0/3163603596 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0500bd330 con 0x5653a1485200 2024-03-18T16:19:03.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.043+0000 7ff06f9e1740 1 -- 172.21.15.39:0/3163603596 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5653a1485200 msgr2=0x5653a1496220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:03.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.043+0000 7ff06f9e1740 1 --2- 172.21.15.39:0/3163603596 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5653a1485200 0x5653a1496220 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7ff0500bd260 tx=0x7ff0500c5870 comp rx=0 tx=0).stop 2024-03-18T16:19:03.046 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.043+0000 7ff06f9e1740 1 -- 172.21.15.39:0/3163603596 shutdown_connections 2024-03-18T16:19:03.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.043+0000 7ff06f9e1740 1 --2- 172.21.15.39:0/3163603596 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5653a1485200 0x5653a1496220 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:03.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.043+0000 7ff06f9e1740 1 --2- 172.21.15.39:0/3163603596 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5653a148a370 0x5653a148d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:03.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.043+0000 7ff06f9e1740 1 --2- 172.21.15.39:0/3163603596 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5653a1484810 0x5653a147ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:03.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.043+0000 7ff06f9e1740 1 -- 172.21.15.39:0/3163603596 >> 172.21.15.39:0/3163603596 conn(0x5653a147fd80 msgr2=0x5653a147e630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:03.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.043+0000 7ff06f9e1740 1 -- 172.21.15.39:0/3163603596 shutdown_connections 2024-03-18T16:19:03.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.043+0000 7ff06f9e1740 1 -- 172.21.15.39:0/3163603596 wait complete. 2024-03-18T16:19:03.047 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.044+0000 7ff06f9e1740 1 -- start start 2024-03-18T16:19:03.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.044+0000 7ff06f9e1740 1 Processor -- start 2024-03-18T16:19:03.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.045+0000 7ff06f9e1740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5653a1484810 0x5653a14937e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:03.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.045+0000 7ff06f9e1740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5653a1485200 0x5653a1493d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:03.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.045+0000 7ff06f9e1740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5653a148a370 0x5653a1494d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:03.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.045+0000 7ff06f9e1740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7ff0681b1e40 con 0x5653a148a370 2024-03-18T16:19:03.048 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.045+0000 7ff06f9e1740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ff0681b1770 con 0x5653a1485200 2024-03-18T16:19:03.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.045+0000 7ff06f9e1740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7ff0681ae9b0 con 0x5653a1484810 2024-03-18T16:19:03.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.045+0000 7ff06f9e1740 10 client.0 _ll_register_callbacks cb 0x7ff050005690 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:19:03.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.045+0000 7ff065400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5653a1485200 0x5653a1493d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:03.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.045+0000 7ff065400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5653a1485200 0x5653a1493d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:34904/0 (socket says 172.21.15.39:34904) 2024-03-18T16:19:03.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.045+0000 7ff065400640 1 -- 172.21.15.39:0/2927579796 learned_addr learned my addr 172.21.15.39:0/2927579796 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:19:03.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.045+0000 7ff066800640 1 --2- 172.21.15.39:0/2927579796 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5653a148a370 0x5653a1494d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:03.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.045+0000 7ff065e00640 1 --2- 172.21.15.39:0/2927579796 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5653a1484810 0x5653a14937e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:03.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.046+0000 7ff065400640 1 -- 172.21.15.39:0/2927579796 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5653a1484810 msgr2=0x5653a14937e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:03.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.046+0000 7ff065400640 1 --2- 172.21.15.39:0/2927579796 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5653a1484810 0x5653a14937e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:03.049 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.046+0000 7ff065400640 1 -- 172.21.15.39:0/2927579796 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5653a148a370 msgr2=0x5653a1494d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:03.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.046+0000 7ff065400640 1 --2- 172.21.15.39:0/2927579796 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5653a148a370 0x5653a1494d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:03.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.046+0000 7ff065400640 1 -- 172.21.15.39:0/2927579796 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5653a14807f0 con 0x5653a1485200 2024-03-18T16:19:03.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.046+0000 7ff066800640 1 --2- 172.21.15.39:0/2927579796 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5653a148a370 0x5653a1494d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:19:03.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.047+0000 7ff065400640 1 --2- 172.21.15.39:0/2927579796 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5653a1485200 0x5653a1493d20 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7ff0500bd4f0 tx=0x7ff0500f4470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=974f5aef587b3e7b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:03.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.047+0000 7ff04b400640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:19:03.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.047+0000 7ff04b400640 1 -- 172.21.15.39:0/2927579796 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0500bd330 con 0x5653a1485200 2024-03-18T16:19:03.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.047+0000 7ff04b400640 1 -- 172.21.15.39:0/2927579796 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7ff0500b9c40 con 0x5653a1485200 2024-03-18T16:19:03.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.047+0000 7ff06f9e1740 1 -- 172.21.15.39:0/2927579796 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff0500c16b0 con 0x5653a1485200 2024-03-18T16:19:03.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.048+0000 7ff04b400640 1 -- 172.21.15.39:0/2927579796 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7ff0500ca050 con 0x5653a1485200 2024-03-18T16:19:03.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.048+0000 7ff06f9e1740 10 client.27353 fetch_fsmap learned FSMap version 44 2024-03-18T16:19:03.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.048+0000 7ff06f9e1740 1 -- 172.21.15.39:0/2927579796 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x5653a14807f0 con 0x5653a1485200 2024-03-18T16:19:03.051 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.048+0000 7ff04b400640 1 -- 172.21.15.39:0/2927579796 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff0500c16b0 con 0x5653a1485200 2024-03-18T16:19:03.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.048+0000 7ff04b400640 1 -- 172.21.15.39:0/2927579796 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7ff0500b92b0 con 0x5653a1485200 2024-03-18T16:19:03.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.048+0000 7ff06f9e1740 10 client.27353 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:19:03.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.048+0000 7ff06f9e1740 10 client.27353 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:19:03.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.048+0000 7ff06f9e1740 20 client.27353 populate_metadata read hostname 'smithi039' 2024-03-18T16:19:03.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.048+0000 7ff06f9e1740 1 --2- 172.21.15.39:0/2927579796 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5653a1501240 0x5653a1503630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:03.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.048+0000 7ff06f9e1740 4 client.27353 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T16:19:03.052 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.048+0000 7ff06f9e1740 1 -- 172.21.15.39:0/2927579796 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5653a13f6730 con 0x5653a1501240 2024-03-18T16:19:03.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.049+0000 7ff065e00640 1 --2- 172.21.15.39:0/2927579796 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5653a1501240 0x5653a1503630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:03.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.049+0000 7ff065e00640 1 --2- 172.21.15.39:0/2927579796 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5653a1501240 0x5653a1503630 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fbfd4feb7721edf server_cookie=7bf9ffc8c4299284 in_seq=0 out_seq=0 2024-03-18T16:19:03.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.049+0000 7ff04b400640 10 client.27353 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:19:03.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.050+0000 7ff04b400640 1 -- 172.21.15.39:0/2927579796 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5653a13f6730 con 0x5653a1501240 2024-03-18T16:19:03.053 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.050+0000 7ff04b400640 10 client.27353 handle_command_reply: tid=0 2024-03-18T16:19:03.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.050+0000 7ff06f9e1740 1 client.27353 shutdown 2024-03-18T16:19:03.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.050+0000 7ff06f9e1740 1 -- 172.21.15.39:0/2927579796 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5653a1485200 msgr2=0x5653a1493d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:03.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.050+0000 7ff06f9e1740 1 --2- 172.21.15.39:0/2927579796 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5653a1485200 0x5653a1493d20 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7ff0500bd4f0 tx=0x7ff0500f4470 comp rx=0 tx=0).stop 2024-03-18T16:19:03.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.050+0000 7ff06f9e1740 1 -- 172.21.15.39:0/2927579796 shutdown_connections 2024-03-18T16:19:03.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.050+0000 7ff06f9e1740 1 --2- 172.21.15.39:0/2927579796 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5653a1501240 0x5653a1503630 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:03.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.050+0000 7ff06f9e1740 1 --2- 172.21.15.39:0/2927579796 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5653a1485200 0x5653a1493d20 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:03.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.050+0000 7ff06f9e1740 1 --2- 172.21.15.39:0/2927579796 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5653a148a370 0x5653a1494d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:03.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.050+0000 7ff04b400640 0 client.27353 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:19:03.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.050+0000 7ff06f9e1740 1 --2- 172.21.15.39:0/2927579796 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5653a1484810 0x5653a14937e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:03.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.050+0000 7ff06f9e1740 1 -- 172.21.15.39:0/2927579796 >> 172.21.15.39:0/2927579796 conn(0x5653a147fd80 msgr2=0x5653a147eec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:03.054 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.050+0000 7ff06f9e1740 1 -- 172.21.15.39:0/2927579796 shutdown_connections 2024-03-18T16:19:03.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.051+0000 7ff06f9e1740 1 -- 172.21.15.39:0/2927579796 wait complete. 2024-03-18T16:19:03.055 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.052+0000 7ff06f9e1740 20 client.27353 trim_cache size 0 max 16384 2024-03-18T16:19:03.055 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-03-18T16:19:03.055 INFO:teuthology.orchestra.run.smithi039.stdout: "status": "no active scrubs running", 2024-03-18T16:19:03.056 INFO:teuthology.orchestra.run.smithi039.stdout: "scrubs": {} 2024-03-18T16:19:03.056 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-03-18T16:19:03.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.053+0000 7ff06cc00640 1 -- 172.21.15.39:0/1914045526 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff04c0a2860 msgr2=0x7ff04c0a4d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:03.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.053+0000 7ff06cc00640 1 --2- 172.21.15.39:0/1914045526 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff04c0a2860 0x7ff04c0a4d20 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x5653a1485a60 tx=0x7ff060009500 comp rx=0 tx=0).stop 2024-03-18T16:19:03.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.053+0000 7ff06cc00640 1 -- 172.21.15.39:0/1914045526 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff0681018d0 msgr2=0x7ff06819e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:03.056 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.053+0000 7ff06cc00640 1 --2- 172.21.15.39:0/1914045526 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff0681018d0 0x7ff06819e670 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7ff05002c8d0 tx=0x7ff050004e30 comp rx=0 tx=0).stop 2024-03-18T16:19:03.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.053+0000 7ff06cc00640 1 -- 172.21.15.39:0/1914045526 shutdown_connections 2024-03-18T16:19:03.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.053+0000 7ff06cc00640 1 --2- 172.21.15.39:0/1914045526 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff068102290 0x7ff0681a56e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:03.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.053+0000 7ff06cc00640 1 --2- 172.21.15.39:0/1914045526 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff04c0a2860 0x7ff04c0a4d20 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:03.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.053+0000 7ff06cc00640 1 --2- 172.21.15.39:0/1914045526 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff0681018d0 0x7ff06819e670 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:03.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.053+0000 7ff06cc00640 1 --2- 172.21.15.39:0/1914045526 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff0681006e0 0x7ff06819e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:03.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.053+0000 7ff06cc00640 1 -- 172.21.15.39:0/1914045526 >> 172.21.15.39:0/1914045526 conn(0x7ff0680fbc70 msgr2=0x7ff068102af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:03.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.054+0000 7ff06cc00640 1 -- 172.21.15.39:0/1914045526 shutdown_connections 2024-03-18T16:19:03.057 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.054+0000 7ff06cc00640 1 -- 172.21.15.39:0/1914045526 wait complete. 2024-03-18T16:19:03.078 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:03 smithi039 ceph-mon[31721]: pgmap v691: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 863 KiB/s wr, 219 op/s 2024-03-18T16:19:03.083 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-18T16:19:03.083 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:19:03.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.317+0000 7ff7fc800640 1 -- 172.21.15.39:0/2212410337 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff7f40773e0 msgr2=0x7ff7f4077840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:03.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.317+0000 7ff7fc800640 1 --2- 172.21.15.39:0/2212410337 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff7f40773e0 0x7ff7f4077840 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7ff7e4009860 tx=0x7ff7e402bb70 comp rx=0 tx=0).stop 2024-03-18T16:19:03.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.317+0000 7ff7fc800640 1 -- 172.21.15.39:0/2212410337 shutdown_connections 2024-03-18T16:19:03.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.317+0000 7ff7fc800640 1 --2- 172.21.15.39:0/2212410337 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff7f40773e0 0x7ff7f4077840 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:03.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.317+0000 7ff7fc800640 1 --2- 172.21.15.39:0/2212410337 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7f4077d80 0x7ff7f4078620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:03.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.317+0000 7ff7fc800640 1 --2- 172.21.15.39:0/2212410337 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff7f40761a0 0x7ff7f40765c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:03.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.317+0000 7ff7fc800640 1 -- 172.21.15.39:0/2212410337 >> 172.21.15.39:0/2212410337 conn(0x7ff7f40fef90 msgr2=0x7ff7f41013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:03.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.317+0000 7ff7fc800640 1 -- 172.21.15.39:0/2212410337 shutdown_connections 2024-03-18T16:19:03.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.317+0000 7ff7fc800640 1 -- 172.21.15.39:0/2212410337 wait complete. 2024-03-18T16:19:03.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.319+0000 7ff7fc800640 1 Processor -- start 2024-03-18T16:19:03.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.319+0000 7ff7fc800640 1 -- start start 2024-03-18T16:19:03.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.319+0000 7ff7fc800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff7f40761a0 0x7ff7f419e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:03.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.320+0000 7ff7fc800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7f40773e0 0x7ff7f419e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:03.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.320+0000 7ff7f9e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff7f40761a0 0x7ff7f419e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:03.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.320+0000 7ff7f9e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff7f40761a0 0x7ff7f419e0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:59354/0 (socket says 172.21.15.39:59354) 2024-03-18T16:19:03.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.320+0000 7ff7fc800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff7f4077d80 0x7ff7f41a5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:03.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.320+0000 7ff7fc800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7ff7f4113a50 con 0x7ff7f40773e0 2024-03-18T16:19:03.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.320+0000 7ff7fc800640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ff7f41138d0 con 0x7ff7f4077d80 2024-03-18T16:19:03.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.320+0000 7ff7fc800640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7ff7f4113bd0 con 0x7ff7f40761a0 2024-03-18T16:19:03.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.320+0000 7ff7f9e00640 1 -- 172.21.15.39:0/4210042527 learned_addr learned my addr 172.21.15.39:0/4210042527 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:19:03.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.320+0000 7ff7f9400640 1 --2- 172.21.15.39:0/4210042527 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7f40773e0 0x7ff7f419e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:03.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.320+0000 7ff7fa800640 1 --2- 172.21.15.39:0/4210042527 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff7f4077d80 0x7ff7f41a5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:03.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.320+0000 7ff7f9e00640 1 -- 172.21.15.39:0/4210042527 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff7f4077d80 msgr2=0x7ff7f41a5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:03.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.320+0000 7ff7f9e00640 1 --2- 172.21.15.39:0/4210042527 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff7f4077d80 0x7ff7f41a5670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:03.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.320+0000 7ff7f9e00640 1 -- 172.21.15.39:0/4210042527 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7f40773e0 msgr2=0x7ff7f419e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:03.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.320+0000 7ff7f9e00640 1 --2- 172.21.15.39:0/4210042527 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7f40773e0 0x7ff7f419e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:03.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.321+0000 7ff7f9e00640 1 -- 172.21.15.39:0/4210042527 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7f41a5ce0 con 0x7ff7f40761a0 2024-03-18T16:19:03.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.321+0000 7ff7f9400640 1 --2- 172.21.15.39:0/4210042527 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7f40773e0 0x7ff7f419e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:19:03.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.321+0000 7ff7fa800640 1 --2- 172.21.15.39:0/4210042527 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff7f4077d80 0x7ff7f41a5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:19:03.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.321+0000 7ff7f9e00640 1 --2- 172.21.15.39:0/4210042527 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff7f40761a0 0x7ff7f419e0c0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7ff7e800e9c0 tx=0x7ff7e800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:03.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.322+0000 7ff7eea00640 1 -- 172.21.15.39:0/4210042527 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7e800ccc0 con 0x7ff7f40761a0 2024-03-18T16:19:03.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.322+0000 7ff7eea00640 1 -- 172.21.15.39:0/4210042527 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff7e8005500 con 0x7ff7f40761a0 2024-03-18T16:19:03.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.322+0000 7ff7fc800640 1 -- 172.21.15.39:0/4210042527 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7f41a5fd0 con 0x7ff7f40761a0 2024-03-18T16:19:03.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.322+0000 7ff7fc800640 1 -- 172.21.15.39:0/4210042527 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7f41a6430 con 0x7ff7f40761a0 2024-03-18T16:19:03.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.322+0000 7ff7eea00640 1 -- 172.21.15.39:0/4210042527 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7e8010430 con 0x7ff7f40761a0 2024-03-18T16:19:03.326 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.323+0000 7ff7fc800640 1 -- 172.21.15.39:0/4210042527 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7d80052c0 con 0x7ff7f40761a0 2024-03-18T16:19:03.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.325+0000 7ff7eea00640 1 -- 172.21.15.39:0/4210042527 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7ff7e8005020 con 0x7ff7f40761a0 2024-03-18T16:19:03.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.327+0000 7ff7eea00640 1 --2- 172.21.15.39:0/4210042527 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff7dc0a29d0 0x7ff7dc0a4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:03.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.327+0000 7ff7eea00640 1 -- 172.21.15.39:0/4210042527 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7ff7e80cf0c0 con 0x7ff7f40761a0 2024-03-18T16:19:03.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.328+0000 7ff7f9400640 1 --2- 172.21.15.39:0/4210042527 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff7dc0a29d0 0x7ff7dc0a4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:03.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.329+0000 7ff7f9400640 1 --2- 172.21.15.39:0/4210042527 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff7dc0a29d0 0x7ff7dc0a4e90 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7ff7e4009830 tx=0x7ff7e4005d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:03.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.332+0000 7ff7eea00640 1 -- 172.21.15.39:0/4210042527 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7ff7e809bd70 con 0x7ff7f40761a0 2024-03-18T16:19:03.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:03 smithi196 ceph-mon[36307]: pgmap v691: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 863 KiB/s wr, 219 op/s 2024-03-18T16:19:03.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:03 smithi194 ceph-mon[36200]: pgmap v691: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 863 KiB/s wr, 219 op/s 2024-03-18T16:19:03.699 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.696+0000 7ff7fc800640 1 -- 172.21.15.39:0/4210042527 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff7d80055b0 con 0x7ff7f40761a0 2024-03-18T16:19:03.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.697+0000 7ff7eea00640 1 -- 172.21.15.39:0/4210042527 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) ==== 94+0+11932 (secure 0 0 0) 0x7ff7e809b670 con 0x7ff7f40761a0 2024-03-18T16:19:03.700 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:19:03.700 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":44,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2520226608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2520226608},{"type":"v1","addr":"172.21.15.39:6835","nonce":2520226608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/719341763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":719341763},{"type":"v1","addr":"172.21.15.39:6837","nonce":719341763}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":44,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T16:06:44.180425+0000","modified":"2024-03-18T16:15:58.268360+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24575,"mds_1":24556,"mds_2":24535,"mds_3":24524,"mds_4":14463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14445":{"gid":14445,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.39:6839/656513813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":656513813},{"type":"v1","addr":"172.21.15.39:6839","nonce":656513813}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14463":{"gid":14463,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":13,"addr":"172.21.15.39:6841/3515326042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3515326042},{"type":"v1","addr":"172.21.15.39:6841","nonce":3515326042}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6833/931350738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":931350738},{"type":"v1","addr":"172.21.15.194:6833","nonce":931350738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6833/3874209621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":3874209621},{"type":"v1","addr":"172.21.15.196:6833","nonce":3874209621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6835/188385470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":188385470},{"type":"v1","addr":"172.21.15.194:6835","nonce":188385470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6835/3949984309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":3949984309},{"type":"v1","addr":"172.21.15.196:6835","nonce":3949984309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":15,"addr":"172.21.15.194:6837/1301464246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":1301464246},{"type":"v1","addr":"172.21.15.194:6837","nonce":1301464246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":16,"addr":"172.21.15.196:6837/895247251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":895247251},{"type":"v1","addr":"172.21.15.196:6837","nonce":895247251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.196:6839/2001427268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6838","nonce":2001427268},{"type":"v1","addr":"172.21.15.196:6839","nonce":2001427268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24575":{"gid":24575,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.194:6839/274911158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":274911158},{"type":"v1","addr":"172.21.15.194:6839","nonce":274911158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24575,"qdb_cluster":[14463,24524,24575,24556,24535]},"id":1}]} 2024-03-18T16:19:03.701 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 44 2024-03-18T16:19:03.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.700+0000 7ff7fc800640 1 -- 172.21.15.39:0/4210042527 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff7dc0a29d0 msgr2=0x7ff7dc0a4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:03.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.700+0000 7ff7fc800640 1 --2- 172.21.15.39:0/4210042527 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff7dc0a29d0 0x7ff7dc0a4e90 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7ff7e4009830 tx=0x7ff7e4005d20 comp rx=0 tx=0).stop 2024-03-18T16:19:03.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.700+0000 7ff7fc800640 1 -- 172.21.15.39:0/4210042527 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff7f40761a0 msgr2=0x7ff7f419e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:03.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.700+0000 7ff7fc800640 1 --2- 172.21.15.39:0/4210042527 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff7f40761a0 0x7ff7f419e0c0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7ff7e800e9c0 tx=0x7ff7e800ee80 comp rx=0 tx=0).stop 2024-03-18T16:19:03.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.701+0000 7ff7fc800640 1 -- 172.21.15.39:0/4210042527 shutdown_connections 2024-03-18T16:19:03.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.701+0000 7ff7fc800640 1 --2- 172.21.15.39:0/4210042527 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff7f4077d80 0x7ff7f41a5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:03.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.701+0000 7ff7fc800640 1 --2- 172.21.15.39:0/4210042527 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7ff7dc0a29d0 0x7ff7dc0a4e90 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:03.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.701+0000 7ff7fc800640 1 --2- 172.21.15.39:0/4210042527 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff7f40773e0 0x7ff7f419e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:03.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.701+0000 7ff7fc800640 1 --2- 172.21.15.39:0/4210042527 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff7f40761a0 0x7ff7f419e0c0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:03.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.701+0000 7ff7fc800640 1 -- 172.21.15.39:0/4210042527 >> 172.21.15.39:0/4210042527 conn(0x7ff7f40fef90 msgr2=0x7ff7f4100690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:03.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.701+0000 7ff7fc800640 1 -- 172.21.15.39:0/4210042527 shutdown_connections 2024-03-18T16:19:03.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.701+0000 7ff7fc800640 1 -- 172.21.15.39:0/4210042527 wait complete. 2024-03-18T16:19:03.726 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-18T16:19:03.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.959+0000 7fb0c9e00640 1 -- 172.21.15.39:0/460129057 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb0c40757a0 msgr2=0x7fb0c4073bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:03.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.959+0000 7fb0c9e00640 1 --2- 172.21.15.39:0/460129057 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb0c40757a0 0x7fb0c4073bf0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fb0b4009900 tx=0x7fb0b402bcc0 comp rx=0 tx=0).stop 2024-03-18T16:19:03.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.959+0000 7fb0c9e00640 1 -- 172.21.15.39:0/460129057 shutdown_connections 2024-03-18T16:19:03.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.959+0000 7fb0c9e00640 1 --2- 172.21.15.39:0/460129057 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0c4109690 0x7fb0c410bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:03.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.959+0000 7fb0c9e00640 1 --2- 172.21.15.39:0/460129057 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb0c40741c0 0x7fb0c4074620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:03.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.959+0000 7fb0c9e00640 1 --2- 172.21.15.39:0/460129057 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb0c40757a0 0x7fb0c4073bf0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:03.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.959+0000 7fb0c9e00640 1 -- 172.21.15.39:0/460129057 >> 172.21.15.39:0/460129057 conn(0x7fb0c40fcd50 msgr2=0x7fb0c40ff190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:03.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.959+0000 7fb0c9e00640 1 -- 172.21.15.39:0/460129057 shutdown_connections 2024-03-18T16:19:03.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.959+0000 7fb0c9e00640 1 -- 172.21.15.39:0/460129057 wait complete. 2024-03-18T16:19:03.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.960+0000 7fb0c9e00640 1 Processor -- start 2024-03-18T16:19:03.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.960+0000 7fb0c9e00640 1 -- start start 2024-03-18T16:19:03.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.960+0000 7fb0c9e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb0c40741c0 0x7fb0c4199c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:03.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.960+0000 7fb0c9e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0c40757a0 0x7fb0c419a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:03.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.960+0000 7fb0c9e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb0c4109690 0x7fb0c41a1230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:03.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.960+0000 7fb0c9e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fb0c410f640 con 0x7fb0c40757a0 2024-03-18T16:19:03.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.960+0000 7fb0c9e00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fb0c410f4c0 con 0x7fb0c4109690 2024-03-18T16:19:03.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.960+0000 7fb0c9e00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fb0c410f7c0 con 0x7fb0c40741c0 2024-03-18T16:19:03.964 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.961+0000 7fb0c2000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0c40757a0 0x7fb0c419a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:03.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.961+0000 7fb0c2000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0c40757a0 0x7fb0c419a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:48972/0 (socket says 172.21.15.39:48972) 2024-03-18T16:19:03.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.961+0000 7fb0c2000640 1 -- 172.21.15.39:0/3934511904 learned_addr learned my addr 172.21.15.39:0/3934511904 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:19:03.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.961+0000 7fb0c2a00640 1 --2- 172.21.15.39:0/3934511904 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb0c40741c0 0x7fb0c4199c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:03.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.961+0000 7fb0c3400640 1 --2- 172.21.15.39:0/3934511904 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb0c4109690 0x7fb0c41a1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:03.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.961+0000 7fb0c2000640 1 -- 172.21.15.39:0/3934511904 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb0c40741c0 msgr2=0x7fb0c4199c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:03.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.961+0000 7fb0c2000640 1 --2- 172.21.15.39:0/3934511904 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb0c40741c0 0x7fb0c4199c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:03.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.961+0000 7fb0c2000640 1 -- 172.21.15.39:0/3934511904 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb0c4109690 msgr2=0x7fb0c41a1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:03.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.961+0000 7fb0c2000640 1 --2- 172.21.15.39:0/3934511904 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb0c4109690 0x7fb0c41a1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:03.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.961+0000 7fb0c2000640 1 -- 172.21.15.39:0/3934511904 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0c41a18d0 con 0x7fb0c40757a0 2024-03-18T16:19:03.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.962+0000 7fb0c2000640 1 --2- 172.21.15.39:0/3934511904 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0c40757a0 0x7fb0c419a1d0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fb0b000e9f0 tx=0x7fb0b000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:03.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.962+0000 7fb0c8a00640 1 -- 172.21.15.39:0/3934511904 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0b000cd80 con 0x7fb0c40757a0 2024-03-18T16:19:03.965 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.962+0000 7fb0c8a00640 1 -- 172.21.15.39:0/3934511904 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0b0010910 con 0x7fb0c40757a0 2024-03-18T16:19:03.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.962+0000 7fb0c8a00640 1 -- 172.21.15.39:0/3934511904 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0b0018990 con 0x7fb0c40757a0 2024-03-18T16:19:03.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.962+0000 7fb0c9e00640 1 -- 172.21.15.39:0/3934511904 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0c41a1bc0 con 0x7fb0c40757a0 2024-03-18T16:19:03.966 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.962+0000 7fb0c9e00640 1 -- 172.21.15.39:0/3934511904 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0c41a2130 con 0x7fb0c40757a0 2024-03-18T16:19:03.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.966+0000 7fb0c8a00640 1 -- 172.21.15.39:0/3934511904 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fb0b0010ac0 con 0x7fb0c40757a0 2024-03-18T16:19:03.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.966+0000 7fb0cca51740 1 Processor -- start 2024-03-18T16:19:03.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.967+0000 7fb0c8a00640 1 --2- 172.21.15.39:0/3934511904 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb0a80a2880 0x7fb0a80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:03.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.967+0000 7fb0c8a00640 1 -- 172.21.15.39:0/3934511904 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fb0b00cf1b0 con 0x7fb0c40757a0 2024-03-18T16:19:03.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.968+0000 7fb0cca51740 1 -- start start 2024-03-18T16:19:03.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.968+0000 7fb0cca51740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55abb8ea08c0 0x55abb8ea0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:03.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.968+0000 7fb0cca51740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55abb8ea12b0 0x55abb8ea5db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:03.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.968+0000 7fb0c3400640 1 --2- 172.21.15.39:0/3934511904 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb0a80a2880 0x7fb0a80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:03.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.968+0000 7fb0cca51740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55abb8ea6420 0x55abb8ea88b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:03.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.969+0000 7fb0cca51740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55abb8e683b0 con 0x55abb8ea08c0 2024-03-18T16:19:03.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.969+0000 7fb0cca51740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55abb8e05450 con 0x55abb8ea12b0 2024-03-18T16:19:03.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.969+0000 7fb0cca51740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55abb8dadcb0 con 0x55abb8ea6420 2024-03-18T16:19:03.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.969+0000 7fb0c2000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55abb8ea12b0 0x55abb8ea5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:03.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.969+0000 7fb0c2000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55abb8ea12b0 0x55abb8ea5db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:34950/0 (socket says 172.21.15.39:34950) 2024-03-18T16:19:03.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.969+0000 7fb0c2000640 1 -- 172.21.15.39:0/868852081 learned_addr learned my addr 172.21.15.39:0/868852081 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:19:03.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.969+0000 7fb0c3400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55abb8ea6420 0x55abb8ea88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:03.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.969+0000 7fb0c2a00640 1 --2- 172.21.15.39:0/868852081 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55abb8ea08c0 0x55abb8ea0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:03.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.970+0000 7fb0c2000640 1 -- 172.21.15.39:0/868852081 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55abb8ea6420 msgr2=0x55abb8ea88b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:03.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.970+0000 7fb0c2000640 1 --2- 172.21.15.39:0/868852081 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55abb8ea6420 0x55abb8ea88b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:03.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.970+0000 7fb0c2000640 1 -- 172.21.15.39:0/868852081 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55abb8ea08c0 msgr2=0x55abb8ea0ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:03.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.970+0000 7fb0c2000640 1 --2- 172.21.15.39:0/868852081 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55abb8ea08c0 0x55abb8ea0ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:03.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.970+0000 7fb0c2000640 1 -- 172.21.15.39:0/868852081 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55abb8dcf1f0 con 0x55abb8ea12b0 2024-03-18T16:19:03.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.970+0000 7fb0c2000640 1 --2- 172.21.15.39:0/868852081 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55abb8ea12b0 0x55abb8ea5db0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fb0b00a39d0 tx=0x7fb0b009bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f204ef907c0e234 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:03.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.970+0000 7fb0c3400640 1 --2- 172.21.15.39:0/3934511904 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb0a80a2880 0x7fb0a80a4d40 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc005c40 tx=0x7fb0bc009df0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:03.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.970+0000 7fb0afe00640 1 -- 172.21.15.39:0/868852081 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0b00a3f00 con 0x55abb8ea12b0 2024-03-18T16:19:03.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.970+0000 7fb0afe00640 1 -- 172.21.15.39:0/868852081 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0b0099a60 con 0x55abb8ea12b0 2024-03-18T16:19:03.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.970+0000 7fb0afe00640 1 -- 172.21.15.39:0/868852081 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0b0018b30 con 0x55abb8ea12b0 2024-03-18T16:19:03.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.971+0000 7fb0cca51740 1 -- 172.21.15.39:0/868852081 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55abb8ea12b0 msgr2=0x55abb8ea5db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:03.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.971+0000 7fb0cca51740 1 --2- 172.21.15.39:0/868852081 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55abb8ea12b0 0x55abb8ea5db0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fb0b00a39d0 tx=0x7fb0b009bf20 comp rx=0 tx=0).stop 2024-03-18T16:19:03.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.972+0000 7fb0cca51740 1 -- 172.21.15.39:0/868852081 shutdown_connections 2024-03-18T16:19:03.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.972+0000 7fb0cca51740 1 --2- 172.21.15.39:0/868852081 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55abb8ea12b0 0x55abb8ea5db0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:03.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.972+0000 7fb0cca51740 1 --2- 172.21.15.39:0/868852081 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55abb8ea6420 0x55abb8ea88b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:03.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.972+0000 7fb0cca51740 1 --2- 172.21.15.39:0/868852081 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55abb8ea08c0 0x55abb8ea0ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:03.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.972+0000 7fb0cca51740 1 -- 172.21.15.39:0/868852081 >> 172.21.15.39:0/868852081 conn(0x55abb8e9be30 msgr2=0x55abb8e9c230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:03.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.972+0000 7fb0cca51740 1 -- 172.21.15.39:0/868852081 shutdown_connections 2024-03-18T16:19:03.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.972+0000 7fb0cca51740 1 -- 172.21.15.39:0/868852081 wait complete. 2024-03-18T16:19:03.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.973+0000 7fb0cca51740 1 -- start start 2024-03-18T16:19:03.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.973+0000 7fb0cca51740 1 Processor -- start 2024-03-18T16:19:03.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.974+0000 7fb0cca51740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55abb8ea08c0 0x55abb8eb33f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:03.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.974+0000 7fb0cca51740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55abb8ea12b0 0x55abb8eb3930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:03.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.974+0000 7fb0cca51740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55abb8ea6420 0x55abb8eba970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:03.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.974+0000 7fb0cca51740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55abb8dc1110 con 0x55abb8ea08c0 2024-03-18T16:19:03.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.974+0000 7fb0cca51740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55abb8e014f0 con 0x55abb8ea12b0 2024-03-18T16:19:03.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.974+0000 7fb0cca51740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55abb8ea93d0 con 0x55abb8ea6420 2024-03-18T16:19:03.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.974+0000 7fb0cca51740 10 client.0 _ll_register_callbacks cb 0x55abb8daeb70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:19:03.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.974+0000 7fb0c3400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55abb8ea6420 0x55abb8eba970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:03.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.974+0000 7fb0c3400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55abb8ea6420 0x55abb8eba970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:59390/0 (socket says 172.21.15.39:59390) 2024-03-18T16:19:03.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.974+0000 7fb0c3400640 1 -- 172.21.15.39:0/2346382435 learned_addr learned my addr 172.21.15.39:0/2346382435 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:19:03.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.974+0000 7fb0c3400640 1 -- 172.21.15.39:0/2346382435 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55abb8ea12b0 msgr2=0x55abb8eb3930 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T16:19:03.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.974+0000 7fb0c2000640 1 --2- 172.21.15.39:0/2346382435 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55abb8ea12b0 0x55abb8eb3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:03.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.974+0000 7fb0c3400640 1 --2- 172.21.15.39:0/2346382435 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55abb8ea12b0 0x55abb8eb3930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:03.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.975+0000 7fb0c3400640 1 -- 172.21.15.39:0/2346382435 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55abb8ea08c0 msgr2=0x55abb8eb33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:03.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.975+0000 7fb0c3400640 1 --2- 172.21.15.39:0/2346382435 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55abb8ea08c0 0x55abb8eb33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:03.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.975+0000 7fb0c3400640 1 -- 172.21.15.39:0/2346382435 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55abb8b193f0 con 0x55abb8ea6420 2024-03-18T16:19:03.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.975+0000 7fb0c2000640 1 --2- 172.21.15.39:0/2346382435 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55abb8ea12b0 0x55abb8eb3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:19:03.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.975+0000 7fb0c3400640 1 --2- 172.21.15.39:0/2346382435 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55abb8ea6420 0x55abb8eba970 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc003ba0 tx=0x7fb0bc004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7bbc0669fba3dfce server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:03.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.975+0000 7fb0acc00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:19:03.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.975+0000 7fb0acc00640 1 -- 172.21.15.39:0/2346382435 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0bc0047c0 con 0x55abb8ea6420 2024-03-18T16:19:03.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.975+0000 7fb0acc00640 1 -- 172.21.15.39:0/2346382435 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fb0bc028470 con 0x55abb8ea6420 2024-03-18T16:19:03.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.975+0000 7fb0cca51740 1 -- 172.21.15.39:0/2346382435 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb0b0099a60 con 0x55abb8ea6420 2024-03-18T16:19:03.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.976+0000 7fb0acc00640 1 -- 172.21.15.39:0/2346382435 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fb0bc028670 con 0x55abb8ea6420 2024-03-18T16:19:03.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.976+0000 7fb0cca51740 10 client.27349 fetch_fsmap learned FSMap version 44 2024-03-18T16:19:03.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.976+0000 7fb0cca51740 1 -- 172.21.15.39:0/2346382435 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x55abb8be37d0 con 0x55abb8ea6420 2024-03-18T16:19:03.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.976+0000 7fb0acc00640 1 -- 172.21.15.39:0/2346382435 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0b0099a60 con 0x55abb8ea6420 2024-03-18T16:19:03.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.977+0000 7fb0acc00640 1 -- 172.21.15.39:0/2346382435 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7fb0bc01e070 con 0x55abb8ea6420 2024-03-18T16:19:03.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.977+0000 7fb0cca51740 10 client.27349 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:19:03.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.977+0000 7fb0cca51740 10 client.27349 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-03-18T16:19:03.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.977+0000 7fb0cca51740 20 client.27349 populate_metadata read hostname 'smithi039' 2024-03-18T16:19:03.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.977+0000 7fb0cca51740 1 --2- 172.21.15.39:0/2346382435 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x55abb8ebf340 0x55abb8ec1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:03.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.977+0000 7fb0cca51740 4 client.27349 mds_command: new command op to 14463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:19:03.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.977+0000 7fb0cca51740 1 -- 172.21.15.39:0/2346382435 --> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55abb8e12990 con 0x55abb8ebf340 2024-03-18T16:19:03.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.978+0000 7fb0c2a00640 1 --2- 172.21.15.39:0/2346382435 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x55abb8ebf340 0x55abb8ec1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:03.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.980+0000 7fb0c2a00640 1 --2- 172.21.15.39:0/2346382435 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x55abb8ebf340 0x55abb8ec1790 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=fedff5074723823e server_cookie=b3cc5015060caab0 in_seq=0 out_seq=0 2024-03-18T16:19:03.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.980+0000 7fb0acc00640 10 client.27349 ms_handle_connect on v2:172.21.15.39:6840/3515326042 2024-03-18T16:19:03.984 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.981+0000 7fb0acc00640 1 -- 172.21.15.39:0/2346382435 <== mds.4 v2:172.21.15.39:6840/3515326042 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55abb8e12990 con 0x55abb8ebf340 2024-03-18T16:19:03.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.982+0000 7fb0acc00640 10 client.27349 handle_command_reply: tid=0 2024-03-18T16:19:03.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.982+0000 7fb0cca51740 1 client.27349 shutdown 2024-03-18T16:19:03.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.982+0000 7fb0cca51740 1 -- 172.21.15.39:0/2346382435 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55abb8ea6420 msgr2=0x55abb8eba970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:03.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.982+0000 7fb0cca51740 1 --2- 172.21.15.39:0/2346382435 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55abb8ea6420 0x55abb8eba970 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc003ba0 tx=0x7fb0bc004ab0 comp rx=0 tx=0).stop 2024-03-18T16:19:03.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.982+0000 7fb0cca51740 1 -- 172.21.15.39:0/2346382435 shutdown_connections 2024-03-18T16:19:03.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.982+0000 7fb0cca51740 1 --2- 172.21.15.39:0/2346382435 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x55abb8ebf340 0x55abb8ec1790 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:03.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.982+0000 7fb0cca51740 1 --2- 172.21.15.39:0/2346382435 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55abb8ea12b0 0x55abb8eb3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:03.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.982+0000 7fb0cca51740 1 --2- 172.21.15.39:0/2346382435 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55abb8ea6420 0x55abb8eba970 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:03.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.982+0000 7fb0cca51740 1 --2- 172.21.15.39:0/2346382435 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55abb8ea08c0 0x55abb8eb33f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:03.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.982+0000 7fb0acc00640 0 client.27349 ms_handle_reset on v2:172.21.15.39:6840/3515326042 2024-03-18T16:19:03.985 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.982+0000 7fb0cca51740 1 -- 172.21.15.39:0/2346382435 >> 172.21.15.39:0/2346382435 conn(0x55abb8e9be30 msgr2=0x55abb8e9b1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:03.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.982+0000 7fb0cca51740 1 -- 172.21.15.39:0/2346382435 shutdown_connections 2024-03-18T16:19:03.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.982+0000 7fb0cca51740 1 -- 172.21.15.39:0/2346382435 wait complete. 2024-03-18T16:19:03.986 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:03.982+0000 7fb0cca51740 20 client.27349 trim_cache size 0 max 16384 2024-03-18T16:19:04.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.003+0000 7fb0cca51740 1 Processor -- start 2024-03-18T16:19:04.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.003+0000 7fb0cca51740 1 -- start start 2024-03-18T16:19:04.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.003+0000 7fb0cca51740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55abb8ea08c0 0x55abb8eb33f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:04.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.003+0000 7fb0cca51740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55abb8ea12b0 0x55abb8eb3930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:04.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.003+0000 7fb0cca51740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55abb8ea6420 0x55abb8eba930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:04.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.003+0000 7fb0cca51740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fb0c41a8240 con 0x55abb8ea6420 2024-03-18T16:19:04.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.003+0000 7fb0cca51740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fb0c41a9d80 con 0x55abb8ea08c0 2024-03-18T16:19:04.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.003+0000 7fb0cca51740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fb0c41a8910 con 0x55abb8ea12b0 2024-03-18T16:19:04.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.004+0000 7fb0c2a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55abb8ea08c0 0x55abb8eb33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:04.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.004+0000 7fb0c2000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55abb8ea12b0 0x55abb8eb3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:04.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.004+0000 7fb0c2a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55abb8ea08c0 0x55abb8eb33f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:34974/0 (socket says 172.21.15.39:34974) 2024-03-18T16:19:04.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.004+0000 7fb0c2a00640 1 -- 172.21.15.39:0/552568326 learned_addr learned my addr 172.21.15.39:0/552568326 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:19:04.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.004+0000 7fb0c2a00640 1 -- 172.21.15.39:0/552568326 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55abb8ea12b0 msgr2=0x55abb8eb3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:04.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.004+0000 7fb0c2a00640 1 --2- 172.21.15.39:0/552568326 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55abb8ea12b0 0x55abb8eb3930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.004+0000 7fb0c2a00640 1 -- 172.21.15.39:0/552568326 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55abb8ea6420 msgr2=0x55abb8eba930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:04.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.004+0000 7fb0c3400640 1 --2- 172.21.15.39:0/552568326 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55abb8ea6420 0x55abb8eba930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:04.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.004+0000 7fb0c2a00640 1 --2- 172.21.15.39:0/552568326 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55abb8ea6420 0x55abb8eba930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.004+0000 7fb0c2a00640 1 -- 172.21.15.39:0/552568326 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55abb8ea92b0 con 0x55abb8ea08c0 2024-03-18T16:19:04.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.004+0000 7fb0c3400640 1 --2- 172.21.15.39:0/552568326 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55abb8ea6420 0x55abb8eba930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:19:04.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.004+0000 7fb0c2a00640 1 --2- 172.21.15.39:0/552568326 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55abb8ea08c0 0x55abb8eb33f0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fb0b4036040 tx=0x7fb0b4005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b6a436e8d393e2e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:04.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.005+0000 7fb0af400640 1 -- 172.21.15.39:0/552568326 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0b4049ab0 con 0x55abb8ea08c0 2024-03-18T16:19:04.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.005+0000 7fb0af400640 1 -- 172.21.15.39:0/552568326 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0b4049c50 con 0x55abb8ea08c0 2024-03-18T16:19:04.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.005+0000 7fb0af400640 1 -- 172.21.15.39:0/552568326 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0b4004290 con 0x55abb8ea08c0 2024-03-18T16:19:04.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.005+0000 7fb0cca51740 1 -- 172.21.15.39:0/552568326 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55abb8ea08c0 msgr2=0x55abb8eb33f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:04.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.005+0000 7fb0cca51740 1 --2- 172.21.15.39:0/552568326 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55abb8ea08c0 0x55abb8eb33f0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fb0b4036040 tx=0x7fb0b4005fd0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.005+0000 7fb0cca51740 1 -- 172.21.15.39:0/552568326 shutdown_connections 2024-03-18T16:19:04.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.005+0000 7fb0cca51740 1 --2- 172.21.15.39:0/552568326 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55abb8ea6420 0x55abb8eba930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.005+0000 7fb0cca51740 1 --2- 172.21.15.39:0/552568326 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55abb8ea12b0 0x55abb8eb3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.005+0000 7fb0cca51740 1 --2- 172.21.15.39:0/552568326 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55abb8ea08c0 0x55abb8eb33f0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.005+0000 7fb0cca51740 1 -- 172.21.15.39:0/552568326 >> 172.21.15.39:0/552568326 conn(0x55abb8e9be30 msgr2=0x55abb8e97080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:04.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.006+0000 7fb0cca51740 1 -- 172.21.15.39:0/552568326 shutdown_connections 2024-03-18T16:19:04.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.006+0000 7fb0cca51740 1 -- 172.21.15.39:0/552568326 wait complete. 2024-03-18T16:19:04.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.006+0000 7fb0cca51740 1 -- start start 2024-03-18T16:19:04.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.006+0000 7fb0cca51740 1 Processor -- start 2024-03-18T16:19:04.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.007+0000 7fb0cca51740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55abb8ea12b0 0x55abb8eb7d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:04.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.007+0000 7fb0cca51740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55abb8ea6420 0x55abb8eb8250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:04.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.007+0000 7fb0cca51740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55abb8eb9260 0x55abb8eb96e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:04.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.007+0000 7fb0cca51740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fb0c41a96b0 con 0x55abb8ea12b0 2024-03-18T16:19:04.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.007+0000 7fb0cca51740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fb0c41a8fe0 con 0x55abb8ea6420 2024-03-18T16:19:04.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.007+0000 7fb0cca51740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fb0c41a6220 con 0x55abb8eb9260 2024-03-18T16:19:04.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.007+0000 7fb0cca51740 10 client.0 _ll_register_callbacks cb 0x7fb0bc0099a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:19:04.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.007+0000 7fb0c2000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55abb8ea6420 0x55abb8eb8250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:04.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.007+0000 7fb0c2a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55abb8ea12b0 0x55abb8eb7d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:04.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.007+0000 7fb0c2000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55abb8ea6420 0x55abb8eb8250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:34980/0 (socket says 172.21.15.39:34980) 2024-03-18T16:19:04.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.007+0000 7fb0c2000640 1 -- 172.21.15.39:0/3153296760 learned_addr learned my addr 172.21.15.39:0/3153296760 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:19:04.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.007+0000 7fb0c3400640 1 --2- 172.21.15.39:0/3153296760 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55abb8eb9260 0x55abb8eb96e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:04.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.007+0000 7fb0c2000640 1 -- 172.21.15.39:0/3153296760 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55abb8eb9260 msgr2=0x55abb8eb96e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:04.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.007+0000 7fb0c2000640 1 --2- 172.21.15.39:0/3153296760 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55abb8eb9260 0x55abb8eb96e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.007+0000 7fb0c2000640 1 -- 172.21.15.39:0/3153296760 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55abb8ea12b0 msgr2=0x55abb8eb7d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:04.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.007+0000 7fb0c2000640 1 --2- 172.21.15.39:0/3153296760 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55abb8ea12b0 0x55abb8eb7d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.007+0000 7fb0c2000640 1 -- 172.21.15.39:0/3153296760 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55abb8eb9c20 con 0x55abb8ea6420 2024-03-18T16:19:04.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.007+0000 7fb0c2a00640 1 --2- 172.21.15.39:0/3153296760 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55abb8ea12b0 0x55abb8eb7d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:19:04.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.008+0000 7fb0c2000640 1 --2- 172.21.15.39:0/3153296760 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55abb8ea6420 0x55abb8eb8250 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fb0b009b6f0 tx=0x7fb0b0009310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=197f3153f5029a16 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:04.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.008+0000 7fb0acc00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:19:04.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.008+0000 7fb0acc00640 1 -- 172.21.15.39:0/3153296760 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0b00b3020 con 0x55abb8ea6420 2024-03-18T16:19:04.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.008+0000 7fb0acc00640 1 -- 172.21.15.39:0/3153296760 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fb0b0097dd0 con 0x55abb8ea6420 2024-03-18T16:19:04.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.008+0000 7fb0cca51740 1 -- 172.21.15.39:0/3153296760 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb0b4049c50 con 0x55abb8ea6420 2024-03-18T16:19:04.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.008+0000 7fb0acc00640 1 -- 172.21.15.39:0/3153296760 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fb0b00a3f00 con 0x55abb8ea6420 2024-03-18T16:19:04.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.008+0000 7fb0cca51740 10 client.27371 fetch_fsmap learned FSMap version 44 2024-03-18T16:19:04.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.008+0000 7fb0cca51740 1 -- 172.21.15.39:0/3153296760 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x55abb8f257a0 con 0x55abb8ea6420 2024-03-18T16:19:04.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.008+0000 7fb0acc00640 1 -- 172.21.15.39:0/3153296760 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0b4049c50 con 0x55abb8ea6420 2024-03-18T16:19:04.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.009+0000 7fb0acc00640 1 -- 172.21.15.39:0/3153296760 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7fb0b0018c50 con 0x55abb8ea6420 2024-03-18T16:19:04.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.009+0000 7fb0cca51740 10 client.27371 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:19:04.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.009+0000 7fb0cca51740 10 client.27371 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-03-18T16:19:04.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.009+0000 7fb0cca51740 20 client.27371 populate_metadata read hostname 'smithi039' 2024-03-18T16:19:04.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.009+0000 7fb0cca51740 1 --2- 172.21.15.39:0/3153296760 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x55abb8f25c40 0x55abb8f28030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:04.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.009+0000 7fb0cca51740 4 client.27371 mds_command: new command op to 14463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:19:04.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.009+0000 7fb0cca51740 1 -- 172.21.15.39:0/3153296760 --> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55abb8e0cce0 con 0x55abb8f25c40 2024-03-18T16:19:04.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.009+0000 7fb0c2a00640 1 --2- 172.21.15.39:0/3153296760 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x55abb8f25c40 0x55abb8f28030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:04.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.010+0000 7fb0acc00640 10 client.27371 ms_handle_connect on v2:172.21.15.39:6840/3515326042 2024-03-18T16:19:04.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.010+0000 7fb0c2a00640 1 --2- 172.21.15.39:0/3153296760 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x55abb8f25c40 0x55abb8f28030 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=3b527d3c5beee923 server_cookie=d7baee2403e8b02e in_seq=0 out_seq=0 2024-03-18T16:19:04.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.011+0000 7fb0acc00640 1 -- 172.21.15.39:0/3153296760 <== mds.4 v2:172.21.15.39:6840/3515326042 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55abb8e0cce0 con 0x55abb8f25c40 2024-03-18T16:19:04.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.011+0000 7fb0acc00640 10 client.27371 handle_command_reply: tid=0 2024-03-18T16:19:04.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.011+0000 7fb0cca51740 1 client.27371 shutdown 2024-03-18T16:19:04.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.011+0000 7fb0cca51740 1 -- 172.21.15.39:0/3153296760 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55abb8ea6420 msgr2=0x55abb8eb8250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:04.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.011+0000 7fb0cca51740 1 --2- 172.21.15.39:0/3153296760 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55abb8ea6420 0x55abb8eb8250 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fb0b009b6f0 tx=0x7fb0b0009310 comp rx=0 tx=0).stop 2024-03-18T16:19:04.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.011+0000 7fb0cca51740 1 -- 172.21.15.39:0/3153296760 shutdown_connections 2024-03-18T16:19:04.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.011+0000 7fb0cca51740 1 --2- 172.21.15.39:0/3153296760 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x55abb8f25c40 0x55abb8f28030 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.011+0000 7fb0cca51740 1 --2- 172.21.15.39:0/3153296760 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55abb8ea6420 0x55abb8eb8250 secure :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fb0b009b6f0 tx=0x7fb0b0009310 comp rx=0 tx=0).stop 2024-03-18T16:19:04.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.011+0000 7fb0acc00640 0 client.27371 ms_handle_reset on v2:172.21.15.39:6840/3515326042 2024-03-18T16:19:04.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.011+0000 7fb0cca51740 1 --2- 172.21.15.39:0/3153296760 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55abb8eb9260 0x55abb8eb96e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.011+0000 7fb0cca51740 1 --2- 172.21.15.39:0/3153296760 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55abb8ea12b0 0x55abb8eb7d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.011+0000 7fb0cca51740 1 -- 172.21.15.39:0/3153296760 >> 172.21.15.39:0/3153296760 conn(0x55abb8e9be30 msgr2=0x55abb8eb33f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:04.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.012+0000 7fb0cca51740 1 -- 172.21.15.39:0/3153296760 shutdown_connections 2024-03-18T16:19:04.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.012+0000 7fb0cca51740 1 -- 172.21.15.39:0/3153296760 wait complete. 2024-03-18T16:19:04.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.012+0000 7fb0cca51740 20 client.27371 trim_cache size 0 max 16384 2024-03-18T16:19:04.017 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:19:04.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.013+0000 7fb0c9e00640 1 -- 172.21.15.39:0/3934511904 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb0a80a2880 msgr2=0x7fb0a80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:04.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.013+0000 7fb0c9e00640 1 --2- 172.21.15.39:0/3934511904 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb0a80a2880 0x7fb0a80a4d40 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fb0bc005c40 tx=0x7fb0bc009df0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.013+0000 7fb0c9e00640 1 -- 172.21.15.39:0/3934511904 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0c40757a0 msgr2=0x7fb0c419a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:04.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.013+0000 7fb0c9e00640 1 --2- 172.21.15.39:0/3934511904 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0c40757a0 0x7fb0c419a1d0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fb0b000e9f0 tx=0x7fb0b000eeb0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.013+0000 7fb0c9e00640 1 -- 172.21.15.39:0/3934511904 shutdown_connections 2024-03-18T16:19:04.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.013+0000 7fb0c9e00640 1 --2- 172.21.15.39:0/3934511904 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb0c4109690 0x7fb0c41a1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.013+0000 7fb0c9e00640 1 --2- 172.21.15.39:0/3934511904 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb0a80a2880 0x7fb0a80a4d40 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.013+0000 7fb0c9e00640 1 --2- 172.21.15.39:0/3934511904 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0c40757a0 0x7fb0c419a1d0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.013+0000 7fb0c9e00640 1 --2- 172.21.15.39:0/3934511904 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb0c40741c0 0x7fb0c4199c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.013+0000 7fb0c9e00640 1 -- 172.21.15.39:0/3934511904 >> 172.21.15.39:0/3934511904 conn(0x7fb0c40fcd50 msgr2=0x7fb0c4105620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:04.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.013+0000 7fb0c9e00640 1 -- 172.21.15.39:0/3934511904 shutdown_connections 2024-03-18T16:19:04.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.013+0000 7fb0c9e00640 1 -- 172.21.15.39:0/3934511904 wait complete. 2024-03-18T16:19:04.038 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-18T16:19:04.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.273+0000 7f7b54e00640 1 -- 172.21.15.39:0/3425970717 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7b501029d0 msgr2=0x7f7b50102e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:04.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.273+0000 7f7b54e00640 1 --2- 172.21.15.39:0/3425970717 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7b501029d0 0x7f7b50102e50 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f7b38009900 tx=0x7f7b3802bc30 comp rx=0 tx=0).stop 2024-03-18T16:19:04.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.273+0000 7f7b54e00640 1 -- 172.21.15.39:0/3425970717 shutdown_connections 2024-03-18T16:19:04.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.273+0000 7f7b54e00640 1 --2- 172.21.15.39:0/3425970717 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7b50103390 0x7f7b50109c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.273+0000 7f7b54e00640 1 --2- 172.21.15.39:0/3425970717 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7b501029d0 0x7f7b50102e50 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.273+0000 7f7b54e00640 1 --2- 172.21.15.39:0/3425970717 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7b501017e0 0x7f7b50101c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.273+0000 7f7b54e00640 1 -- 172.21.15.39:0/3425970717 >> 172.21.15.39:0/3425970717 conn(0x7f7b500fcd50 msgr2=0x7f7b500ff190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:04.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.273+0000 7f7b54e00640 1 -- 172.21.15.39:0/3425970717 shutdown_connections 2024-03-18T16:19:04.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.273+0000 7f7b54e00640 1 -- 172.21.15.39:0/3425970717 wait complete. 2024-03-18T16:19:04.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.274+0000 7f7b54e00640 1 Processor -- start 2024-03-18T16:19:04.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.275+0000 7f7b54e00640 1 -- start start 2024-03-18T16:19:04.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.275+0000 7f7b54e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7b501017e0 0x7f7b50110d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:04.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.275+0000 7f7b54e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7b501029d0 0x7f7b50111280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:04.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.275+0000 7f7b54e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7b50103390 0x7f7b501162e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:04.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.275+0000 7f7b54e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f7b5010b280 con 0x7f7b501017e0 2024-03-18T16:19:04.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.275+0000 7f7b54e00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f7b5010b400 con 0x7f7b50103390 2024-03-18T16:19:04.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.275+0000 7f7b54e00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f7b5010b580 con 0x7f7b501029d0 2024-03-18T16:19:04.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.275+0000 7f7b4de00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7b501017e0 0x7f7b50110d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:04.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.275+0000 7f7b4de00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7b501017e0 0x7f7b50110d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:49024/0 (socket says 172.21.15.39:49024) 2024-03-18T16:19:04.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.275+0000 7f7b4de00640 1 -- 172.21.15.39:0/2015034070 learned_addr learned my addr 172.21.15.39:0/2015034070 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:19:04.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.276+0000 7f7b4e800640 1 --2- 172.21.15.39:0/2015034070 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7b50103390 0x7f7b501162e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:04.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.276+0000 7f7b45400640 1 --2- 172.21.15.39:0/2015034070 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7b501029d0 0x7f7b50111280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:04.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.276+0000 7f7b45400640 1 -- 172.21.15.39:0/2015034070 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7b50103390 msgr2=0x7f7b501162e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:04.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.276+0000 7f7b45400640 1 --2- 172.21.15.39:0/2015034070 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7b50103390 0x7f7b501162e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.276+0000 7f7b45400640 1 -- 172.21.15.39:0/2015034070 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7b501017e0 msgr2=0x7f7b50110d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:04.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.276+0000 7f7b45400640 1 --2- 172.21.15.39:0/2015034070 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7b501017e0 0x7f7b50110d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.276+0000 7f7b45400640 1 -- 172.21.15.39:0/2015034070 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b50116950 con 0x7f7b501029d0 2024-03-18T16:19:04.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.276+0000 7f7b4de00640 1 --2- 172.21.15.39:0/2015034070 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7b501017e0 0x7f7b50110d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:19:04.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.276+0000 7f7b4e800640 1 --2- 172.21.15.39:0/2015034070 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7b50103390 0x7f7b501162e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:19:04.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.277+0000 7f7b45400640 1 --2- 172.21.15.39:0/2015034070 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7b501029d0 0x7f7b50111280 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f7b38036040 tx=0x7f7b3802c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:04.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.277+0000 7f7b47400640 1 -- 172.21.15.39:0/2015034070 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b38043020 con 0x7f7b501029d0 2024-03-18T16:19:04.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.277+0000 7f7b47400640 1 -- 172.21.15.39:0/2015034070 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b380054e0 con 0x7f7b501029d0 2024-03-18T16:19:04.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.277+0000 7f7b54e00640 1 -- 172.21.15.39:0/2015034070 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b50116be0 con 0x7f7b501029d0 2024-03-18T16:19:04.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.277+0000 7f7b47400640 1 -- 172.21.15.39:0/2015034070 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b38003b30 con 0x7f7b501029d0 2024-03-18T16:19:04.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.278+0000 7f7b54e00640 1 -- 172.21.15.39:0/2015034070 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b50117150 con 0x7f7b501029d0 2024-03-18T16:19:04.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.283+0000 7f7b47400640 1 -- 172.21.15.39:0/2015034070 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f7b38046020 con 0x7f7b501029d0 2024-03-18T16:19:04.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.283+0000 7f7b57a60740 1 Processor -- start 2024-03-18T16:19:04.286 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.284+0000 7f7b57a60740 1 -- start start 2024-03-18T16:19:04.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.284+0000 7f7b57a60740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55deed72f8c0 0x55deed72fce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:04.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.284+0000 7f7b57a60740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55deed7302b0 0x55deed734db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:04.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.284+0000 7f7b57a60740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55deed735420 0x55deed7378b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:04.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.284+0000 7f7b57a60740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55deed6f73b0 con 0x55deed72f8c0 2024-03-18T16:19:04.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.284+0000 7f7b57a60740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55deed694450 con 0x55deed735420 2024-03-18T16:19:04.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.284+0000 7f7b57a60740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55deed63ccb0 con 0x55deed7302b0 2024-03-18T16:19:04.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.284+0000 7f7b4de00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55deed7302b0 0x55deed734db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:04.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.284+0000 7f7b4e800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55deed72f8c0 0x55deed72fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:04.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.284+0000 7f7b4de00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55deed7302b0 0x55deed734db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:59424/0 (socket says 172.21.15.39:59424) 2024-03-18T16:19:04.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.284+0000 7f7b4de00640 1 -- 172.21.15.39:0/2517134791 learned_addr learned my addr 172.21.15.39:0/2517134791 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:19:04.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.285+0000 7f7b45400640 1 --2- 172.21.15.39:0/2517134791 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55deed735420 0x55deed7378b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:04.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.285+0000 7f7b4de00640 1 -- 172.21.15.39:0/2517134791 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55deed735420 msgr2=0x55deed7378b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:04.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.285+0000 7f7b4de00640 1 --2- 172.21.15.39:0/2517134791 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55deed735420 0x55deed7378b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.285+0000 7f7b4de00640 1 -- 172.21.15.39:0/2517134791 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55deed72f8c0 msgr2=0x55deed72fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:04.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.285+0000 7f7b4de00640 1 --2- 172.21.15.39:0/2517134791 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55deed72f8c0 0x55deed72fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.285+0000 7f7b4de00640 1 -- 172.21.15.39:0/2517134791 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55deed65e1f0 con 0x55deed7302b0 2024-03-18T16:19:04.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.285+0000 7f7b4de00640 1 --2- 172.21.15.39:0/2517134791 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55deed7302b0 0x55deed734db0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f7b40011900 tx=0x7f7b40011dc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be7f624ac0219b46 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:04.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.285+0000 7f7b47400640 1 --2- 172.21.15.39:0/2015034070 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f7b340a2880 0x7f7b340a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:04.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.286+0000 7f7b47400640 1 -- 172.21.15.39:0/2015034070 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f7b380f0880 con 0x7f7b501029d0 2024-03-18T16:19:04.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.286+0000 7f7b4e800640 1 --2- 172.21.15.39:0/2015034070 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f7b340a2880 0x7f7b340a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:04.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.286+0000 7f7b3f400640 1 -- 172.21.15.39:0/2517134791 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b400112f0 con 0x55deed7302b0 2024-03-18T16:19:04.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.286+0000 7f7b3f400640 1 -- 172.21.15.39:0/2517134791 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b40010430 con 0x55deed7302b0 2024-03-18T16:19:04.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.286+0000 7f7b3f400640 1 -- 172.21.15.39:0/2517134791 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b40019640 con 0x55deed7302b0 2024-03-18T16:19:04.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.287+0000 7f7b57a60740 1 -- 172.21.15.39:0/2517134791 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55deed7302b0 msgr2=0x55deed734db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:04.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.287+0000 7f7b57a60740 1 --2- 172.21.15.39:0/2517134791 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55deed7302b0 0x55deed734db0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f7b40011900 tx=0x7f7b40011dc0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.287+0000 7f7b4e800640 1 --2- 172.21.15.39:0/2015034070 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f7b340a2880 0x7f7b340a4d40 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x55deed730b10 tx=0x7f7b48005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:04.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.287+0000 7f7b57a60740 1 -- 172.21.15.39:0/2517134791 shutdown_connections 2024-03-18T16:19:04.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.287+0000 7f7b57a60740 1 --2- 172.21.15.39:0/2517134791 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55deed735420 0x55deed7378b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.287+0000 7f7b57a60740 1 --2- 172.21.15.39:0/2517134791 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55deed7302b0 0x55deed734db0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.287+0000 7f7b57a60740 1 --2- 172.21.15.39:0/2517134791 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55deed72f8c0 0x55deed72fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.287+0000 7f7b57a60740 1 -- 172.21.15.39:0/2517134791 >> 172.21.15.39:0/2517134791 conn(0x55deed72ae30 msgr2=0x55deed72b230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:04.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.288+0000 7f7b57a60740 1 -- 172.21.15.39:0/2517134791 shutdown_connections 2024-03-18T16:19:04.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.288+0000 7f7b57a60740 1 -- 172.21.15.39:0/2517134791 wait complete. 2024-03-18T16:19:04.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.288+0000 7f7b57a60740 1 -- start start 2024-03-18T16:19:04.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.289+0000 7f7b57a60740 1 Processor -- start 2024-03-18T16:19:04.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.289+0000 7f7b57a60740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55deed72f8c0 0x55deed739b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:04.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.289+0000 7f7b57a60740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55deed7302b0 0x55deed73a050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:04.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.289+0000 7f7b57a60740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55deed735420 0x55deed741090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:04.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.289+0000 7f7b57a60740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55deed650110 con 0x55deed72f8c0 2024-03-18T16:19:04.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.289+0000 7f7b57a60740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55deed6904f0 con 0x55deed7302b0 2024-03-18T16:19:04.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.289+0000 7f7b57a60740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55deed7383d0 con 0x55deed735420 2024-03-18T16:19:04.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.289+0000 7f7b57a60740 10 client.0 _ll_register_callbacks cb 0x55deed63db70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:19:04.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.290+0000 7f7b4de00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55deed72f8c0 0x55deed739b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:04.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.290+0000 7f7b4de00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55deed72f8c0 0x55deed739b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:49050/0 (socket says 172.21.15.39:49050) 2024-03-18T16:19:04.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.290+0000 7f7b4de00640 1 -- 172.21.15.39:0/3114332501 learned_addr learned my addr 172.21.15.39:0/3114332501 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:19:04.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.290+0000 7f7b45400640 1 --2- 172.21.15.39:0/3114332501 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55deed7302b0 0x55deed73a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:04.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.290+0000 7f7b4de00640 1 -- 172.21.15.39:0/3114332501 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55deed735420 msgr2=0x55deed741090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:19:04.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.290+0000 7f7b4de00640 1 --2- 172.21.15.39:0/3114332501 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55deed735420 0x55deed741090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.290+0000 7f7b4de00640 1 -- 172.21.15.39:0/3114332501 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55deed7302b0 msgr2=0x55deed73a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:04.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.290+0000 7f7b4de00640 1 --2- 172.21.15.39:0/3114332501 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55deed7302b0 0x55deed73a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.290+0000 7f7b4de00640 1 -- 172.21.15.39:0/3114332501 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55deed3a83f0 con 0x55deed72f8c0 2024-03-18T16:19:04.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.290+0000 7f7b45400640 1 --2- 172.21.15.39:0/3114332501 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55deed7302b0 0x55deed73a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:19:04.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.291+0000 7f7b4de00640 1 --2- 172.21.15.39:0/3114332501 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55deed72f8c0 0x55deed739b10 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f7b40000f80 tx=0x7f7b4000feb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8d5c62d52386308 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:04.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.291+0000 7f7b33e00640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-03-18T16:19:04.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.291+0000 7f7b33e00640 1 -- 172.21.15.39:0/3114332501 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b40027030 con 0x55deed72f8c0 2024-03-18T16:19:04.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.291+0000 7f7b33e00640 1 -- 172.21.15.39:0/3114332501 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f7b4001f070 con 0x55deed72f8c0 2024-03-18T16:19:04.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.291+0000 7f7b57a60740 1 -- 172.21.15.39:0/3114332501 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7b40010430 con 0x55deed72f8c0 2024-03-18T16:19:04.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.292+0000 7f7b33e00640 1 -- 172.21.15.39:0/3114332501 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f7b40004c50 con 0x55deed72f8c0 2024-03-18T16:19:04.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.292+0000 7f7b57a60740 10 client.16752 fetch_fsmap learned FSMap version 44 2024-03-18T16:19:04.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.292+0000 7f7b57a60740 1 -- 172.21.15.39:0/3114332501 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x55deed4727d0 con 0x55deed72f8c0 2024-03-18T16:19:04.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.292+0000 7f7b33e00640 1 -- 172.21.15.39:0/3114332501 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b40010430 con 0x55deed72f8c0 2024-03-18T16:19:04.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.292+0000 7f7b33e00640 1 -- 172.21.15.39:0/3114332501 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7f7b40018070 con 0x55deed72f8c0 2024-03-18T16:19:04.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.293+0000 7f7b57a60740 10 client.16752 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:19:04.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.293+0000 7f7b57a60740 10 client.16752 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-18T16:19:04.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.293+0000 7f7b57a60740 20 client.16752 populate_metadata read hostname 'smithi039' 2024-03-18T16:19:04.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.293+0000 7f7b57a60740 1 --2- 172.21.15.39:0/3114332501 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x55deed7456d0 0x55deed747b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:04.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.293+0000 7f7b57a60740 4 client.16752 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:19:04.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.293+0000 7f7b57a60740 1 -- 172.21.15.39:0/3114332501 --> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55deed6a1990 con 0x55deed7456d0 2024-03-18T16:19:04.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.293+0000 7f7b45400640 1 --2- 172.21.15.39:0/3114332501 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x55deed7456d0 0x55deed747b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:04.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.294+0000 7f7b45400640 1 --2- 172.21.15.39:0/3114332501 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x55deed7456d0 0x55deed747b40 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=3681377f60c10490 server_cookie=390c8ac168a1de40 in_seq=0 out_seq=0 2024-03-18T16:19:04.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.294+0000 7f7b33e00640 10 client.16752 ms_handle_connect on v2:172.21.15.194:6836/1301464246 2024-03-18T16:19:04.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.295+0000 7f7b33e00640 1 -- 172.21.15.39:0/3114332501 <== mds.3 v2:172.21.15.194:6836/1301464246 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55deed6a1990 con 0x55deed7456d0 2024-03-18T16:19:04.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.295+0000 7f7b33e00640 10 client.16752 handle_command_reply: tid=0 2024-03-18T16:19:04.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.295+0000 7f7b57a60740 1 client.16752 shutdown 2024-03-18T16:19:04.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.298+0000 7f7b57a60740 1 -- 172.21.15.39:0/3114332501 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55deed72f8c0 msgr2=0x55deed739b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:04.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.298+0000 7f7b57a60740 1 --2- 172.21.15.39:0/3114332501 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55deed72f8c0 0x55deed739b10 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f7b40000f80 tx=0x7f7b4000feb0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.298+0000 7f7b57a60740 1 -- 172.21.15.39:0/3114332501 shutdown_connections 2024-03-18T16:19:04.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.298+0000 7f7b57a60740 1 --2- 172.21.15.39:0/3114332501 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x55deed7456d0 0x55deed747b40 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.299+0000 7f7b57a60740 1 --2- 172.21.15.39:0/3114332501 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55deed7302b0 0x55deed73a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.299+0000 7f7b57a60740 1 --2- 172.21.15.39:0/3114332501 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55deed735420 0x55deed741090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.299+0000 7f7b57a60740 1 --2- 172.21.15.39:0/3114332501 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55deed72f8c0 0x55deed739b10 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.299+0000 7f7b33e00640 0 client.16752 ms_handle_reset on v2:172.21.15.194:6836/1301464246 2024-03-18T16:19:04.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.299+0000 7f7b57a60740 1 -- 172.21.15.39:0/3114332501 >> 172.21.15.39:0/3114332501 conn(0x55deed72ae30 msgr2=0x55deed72a1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:04.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.299+0000 7f7b57a60740 1 -- 172.21.15.39:0/3114332501 shutdown_connections 2024-03-18T16:19:04.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.299+0000 7f7b57a60740 1 -- 172.21.15.39:0/3114332501 wait complete. 2024-03-18T16:19:04.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.300+0000 7f7b57a60740 20 client.16752 trim_cache size 0 max 16384 2024-03-18T16:19:04.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.318+0000 7f7b57a60740 1 Processor -- start 2024-03-18T16:19:04.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.318+0000 7f7b57a60740 1 -- start start 2024-03-18T16:19:04.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.319+0000 7f7b57a60740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55deed72f8c0 0x55deed739e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:04.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.319+0000 7f7b57a60740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55deed7302b0 0x55deed72e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:04.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.319+0000 7f7b57a60740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55deed735420 0x55deed74c400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:04.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.319+0000 7f7b57a60740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f7b501b92d0 con 0x55deed735420 2024-03-18T16:19:04.322 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.319+0000 7f7b57a60740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f7b501bae10 con 0x55deed72f8c0 2024-03-18T16:19:04.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.319+0000 7f7b57a60740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f7b501b99a0 con 0x55deed7302b0 2024-03-18T16:19:04.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.319+0000 7f7b4de00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55deed72f8c0 0x55deed739e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:04.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.319+0000 7f7b45400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55deed7302b0 0x55deed72e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:04.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.319+0000 7f7b4de00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55deed72f8c0 0x55deed739e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:35034/0 (socket says 172.21.15.39:35034) 2024-03-18T16:19:04.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.319+0000 7f7b4de00640 1 -- 172.21.15.39:0/3745422619 learned_addr learned my addr 172.21.15.39:0/3745422619 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:19:04.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.319+0000 7f7b4e800640 1 --2- 172.21.15.39:0/3745422619 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55deed735420 0x55deed74c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:04.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.319+0000 7f7b4de00640 1 -- 172.21.15.39:0/3745422619 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55deed7302b0 msgr2=0x55deed72e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:04.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.319+0000 7f7b4de00640 1 --2- 172.21.15.39:0/3745422619 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55deed7302b0 0x55deed72e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.319+0000 7f7b4de00640 1 -- 172.21.15.39:0/3745422619 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55deed735420 msgr2=0x55deed74c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:04.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.319+0000 7f7b4de00640 1 --2- 172.21.15.39:0/3745422619 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55deed735420 0x55deed74c400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.319+0000 7f7b4de00640 1 -- 172.21.15.39:0/3745422619 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55deed726c10 con 0x55deed72f8c0 2024-03-18T16:19:04.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.320+0000 7f7b4e800640 1 --2- 172.21.15.39:0/3745422619 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55deed735420 0x55deed74c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:19:04.323 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.320+0000 7f7b4de00640 1 --2- 172.21.15.39:0/3745422619 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55deed72f8c0 0x55deed739e10 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f7b40011850 tx=0x7f7b40009fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e1785c82c4253ea2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:04.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.320+0000 7f7b3ea00640 1 -- 172.21.15.39:0/3745422619 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b40027040 con 0x55deed72f8c0 2024-03-18T16:19:04.324 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.320+0000 7f7b3ea00640 1 -- 172.21.15.39:0/3745422619 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b40004df0 con 0x55deed72f8c0 2024-03-18T16:19:04.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.320+0000 7f7b3ea00640 1 -- 172.21.15.39:0/3745422619 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b40003d90 con 0x55deed72f8c0 2024-03-18T16:19:04.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.321+0000 7f7b57a60740 1 -- 172.21.15.39:0/3745422619 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55deed72f8c0 msgr2=0x55deed739e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:04.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.321+0000 7f7b57a60740 1 --2- 172.21.15.39:0/3745422619 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55deed72f8c0 0x55deed739e10 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f7b40011850 tx=0x7f7b40009fd0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.321+0000 7f7b57a60740 1 -- 172.21.15.39:0/3745422619 shutdown_connections 2024-03-18T16:19:04.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.321+0000 7f7b57a60740 1 --2- 172.21.15.39:0/3745422619 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55deed735420 0x55deed74c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.321+0000 7f7b57a60740 1 --2- 172.21.15.39:0/3745422619 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55deed7302b0 0x55deed72e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.321+0000 7f7b57a60740 1 --2- 172.21.15.39:0/3745422619 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55deed72f8c0 0x55deed739e10 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.321+0000 7f7b57a60740 1 -- 172.21.15.39:0/3745422619 >> 172.21.15.39:0/3745422619 conn(0x55deed72ae30 msgr2=0x55deed72e090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:04.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.321+0000 7f7b57a60740 1 -- 172.21.15.39:0/3745422619 shutdown_connections 2024-03-18T16:19:04.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.321+0000 7f7b57a60740 1 -- 172.21.15.39:0/3745422619 wait complete. 2024-03-18T16:19:04.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.322+0000 7f7b57a60740 1 -- start start 2024-03-18T16:19:04.325 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.322+0000 7f7b57a60740 1 Processor -- start 2024-03-18T16:19:04.326 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.322+0000 7f7b57a60740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55deed72f8c0 0x55deed73ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:04.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.322+0000 7f7b57a60740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55deed7302b0 0x55deed73e300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:04.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.322+0000 7f7b57a60740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55deed735420 0x55deed73f310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:04.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.322+0000 7f7b57a60740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f7b501ba740 con 0x55deed72f8c0 2024-03-18T16:19:04.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.322+0000 7f7b57a60740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f7b501ba070 con 0x55deed735420 2024-03-18T16:19:04.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.322+0000 7f7b57a60740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f7b501b72b0 con 0x55deed7302b0 2024-03-18T16:19:04.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.323+0000 7f7b57a60740 10 client.0 _ll_register_callbacks cb 0x7f7b40011780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:19:04.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.323+0000 7f7b45400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55deed7302b0 0x55deed73e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:04.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.323+0000 7f7b4e800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55deed735420 0x55deed73f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:04.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.323+0000 7f7b45400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55deed7302b0 0x55deed73e300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:59454/0 (socket says 172.21.15.39:59454) 2024-03-18T16:19:04.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.323+0000 7f7b45400640 1 -- 172.21.15.39:0/3118104055 learned_addr learned my addr 172.21.15.39:0/3118104055 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:19:04.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.323+0000 7f7b4de00640 1 --2- 172.21.15.39:0/3118104055 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55deed72f8c0 0x55deed73ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:04.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.323+0000 7f7b45400640 1 -- 172.21.15.39:0/3118104055 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55deed735420 msgr2=0x55deed73f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:04.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.323+0000 7f7b45400640 1 --2- 172.21.15.39:0/3118104055 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55deed735420 0x55deed73f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.327 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.323+0000 7f7b45400640 1 -- 172.21.15.39:0/3118104055 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55deed72f8c0 msgr2=0x55deed73ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:04.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.323+0000 7f7b45400640 1 --2- 172.21.15.39:0/3118104055 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55deed72f8c0 0x55deed73ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.323+0000 7f7b45400640 1 -- 172.21.15.39:0/3118104055 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55deed73f900 con 0x55deed7302b0 2024-03-18T16:19:04.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.323+0000 7f7b4de00640 1 --2- 172.21.15.39:0/3118104055 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55deed72f8c0 0x55deed73ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:19:04.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.324+0000 7f7b45400640 1 --2- 172.21.15.39:0/3118104055 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55deed7302b0 0x55deed73e300 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f7b38005690 tx=0x7f7b380edec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c01fcd7dcf765e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:04.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.324+0000 7f7b33e00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:19:04.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.324+0000 7f7b33e00640 1 -- 172.21.15.39:0/3118104055 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b380bcc60 con 0x55deed7302b0 2024-03-18T16:19:04.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.324+0000 7f7b33e00640 1 -- 172.21.15.39:0/3118104055 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f7b380bb190 con 0x55deed7302b0 2024-03-18T16:19:04.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.324+0000 7f7b57a60740 1 -- 172.21.15.39:0/3118104055 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7b40004df0 con 0x55deed7302b0 2024-03-18T16:19:04.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.324+0000 7f7b33e00640 1 -- 172.21.15.39:0/3118104055 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f7b380bc7f0 con 0x55deed7302b0 2024-03-18T16:19:04.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.324+0000 7f7b57a60740 10 client.27370 fetch_fsmap learned FSMap version 44 2024-03-18T16:19:04.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.324+0000 7f7b57a60740 1 -- 172.21.15.39:0/3118104055 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x55deed740cd0 con 0x55deed7302b0 2024-03-18T16:19:04.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.325+0000 7f7b33e00640 1 -- 172.21.15.39:0/3118104055 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b40004df0 con 0x55deed7302b0 2024-03-18T16:19:04.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.325+0000 7f7b33e00640 1 -- 172.21.15.39:0/3118104055 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7f7b380b9500 con 0x55deed7302b0 2024-03-18T16:19:04.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.325+0000 7f7b57a60740 10 client.27370 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:19:04.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.325+0000 7f7b57a60740 10 client.27370 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-18T16:19:04.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.325+0000 7f7b57a60740 20 client.27370 populate_metadata read hostname 'smithi039' 2024-03-18T16:19:04.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.325+0000 7f7b57a60740 1 --2- 172.21.15.39:0/3118104055 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x55deed7abdd0 0x55deed7ae1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:04.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.325+0000 7f7b57a60740 4 client.27370 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:19:04.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.325+0000 7f7b57a60740 1 -- 172.21.15.39:0/3118104055 --> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55deed69bce0 con 0x55deed7abdd0 2024-03-18T16:19:04.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.325+0000 7f7b4de00640 1 --2- 172.21.15.39:0/3118104055 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x55deed7abdd0 0x55deed7ae1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:04.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.326+0000 7f7b4de00640 1 --2- 172.21.15.39:0/3118104055 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x55deed7abdd0 0x55deed7ae1c0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=3f64e80a75e936dc server_cookie=5f0887c1d5f0ec28 in_seq=0 out_seq=0 2024-03-18T16:19:04.330 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.326+0000 7f7b33e00640 10 client.27370 ms_handle_connect on v2:172.21.15.194:6836/1301464246 2024-03-18T16:19:04.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.326+0000 7f7b33e00640 1 -- 172.21.15.39:0/3118104055 <== mds.3 v2:172.21.15.194:6836/1301464246 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55deed69bce0 con 0x55deed7abdd0 2024-03-18T16:19:04.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.326+0000 7f7b33e00640 10 client.27370 handle_command_reply: tid=0 2024-03-18T16:19:04.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.326+0000 7f7b57a60740 1 client.27370 shutdown 2024-03-18T16:19:04.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.327+0000 7f7b57a60740 1 -- 172.21.15.39:0/3118104055 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55deed7302b0 msgr2=0x55deed73e300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:04.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.327+0000 7f7b57a60740 1 --2- 172.21.15.39:0/3118104055 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55deed7302b0 0x55deed73e300 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f7b38005690 tx=0x7f7b380edec0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.327+0000 7f7b57a60740 1 -- 172.21.15.39:0/3118104055 shutdown_connections 2024-03-18T16:19:04.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.327+0000 7f7b57a60740 1 --2- 172.21.15.39:0/3118104055 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x55deed7abdd0 0x55deed7ae1c0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.327+0000 7f7b57a60740 1 --2- 172.21.15.39:0/3118104055 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55deed735420 0x55deed73f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.327+0000 7f7b57a60740 1 --2- 172.21.15.39:0/3118104055 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55deed7302b0 0x55deed73e300 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.327+0000 7f7b57a60740 1 --2- 172.21.15.39:0/3118104055 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55deed72f8c0 0x55deed73ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.327+0000 7f7b57a60740 1 -- 172.21.15.39:0/3118104055 >> 172.21.15.39:0/3118104055 conn(0x55deed72ae30 msgr2=0x55deed72e920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:04.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.327+0000 7f7b33e00640 0 client.27370 ms_handle_reset on v2:172.21.15.194:6836/1301464246 2024-03-18T16:19:04.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.327+0000 7f7b57a60740 1 -- 172.21.15.39:0/3118104055 shutdown_connections 2024-03-18T16:19:04.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.327+0000 7f7b57a60740 1 -- 172.21.15.39:0/3118104055 wait complete. 2024-03-18T16:19:04.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.327+0000 7f7b57a60740 20 client.27370 trim_cache size 0 max 16384 2024-03-18T16:19:04.332 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:19:04.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.328+0000 7f7b54e00640 1 -- 172.21.15.39:0/2015034070 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f7b340a2880 msgr2=0x7f7b340a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:04.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.328+0000 7f7b54e00640 1 --2- 172.21.15.39:0/2015034070 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f7b340a2880 0x7f7b340a4d40 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x55deed730b10 tx=0x7f7b48005f90 comp rx=0 tx=0).stop 2024-03-18T16:19:04.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.328+0000 7f7b54e00640 1 -- 172.21.15.39:0/2015034070 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7b501029d0 msgr2=0x7f7b50111280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:04.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.328+0000 7f7b54e00640 1 --2- 172.21.15.39:0/2015034070 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7b501029d0 0x7f7b50111280 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f7b38036040 tx=0x7f7b3802c650 comp rx=0 tx=0).stop 2024-03-18T16:19:04.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.328+0000 7f7b54e00640 1 -- 172.21.15.39:0/2015034070 shutdown_connections 2024-03-18T16:19:04.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.328+0000 7f7b54e00640 1 --2- 172.21.15.39:0/2015034070 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7b50103390 0x7f7b501162e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.328+0000 7f7b54e00640 1 --2- 172.21.15.39:0/2015034070 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f7b340a2880 0x7f7b340a4d40 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.328+0000 7f7b54e00640 1 --2- 172.21.15.39:0/2015034070 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7b501029d0 0x7f7b50111280 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.328+0000 7f7b54e00640 1 --2- 172.21.15.39:0/2015034070 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7b501017e0 0x7f7b50110d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.328+0000 7f7b54e00640 1 -- 172.21.15.39:0/2015034070 >> 172.21.15.39:0/2015034070 conn(0x7f7b500fcd50 msgr2=0x7f7b50105cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:04.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.329+0000 7f7b54e00640 1 -- 172.21.15.39:0/2015034070 shutdown_connections 2024-03-18T16:19:04.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.329+0000 7f7b54e00640 1 -- 172.21.15.39:0/2015034070 wait complete. 2024-03-18T16:19:04.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:04 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/4210042527' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:19:04.355 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-18T16:19:04.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:04 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/4210042527' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:19:04.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:04 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/4210042527' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:19:04.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.585+0000 7f986e800640 1 -- 172.21.15.39:0/2037361257 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9868103a10 msgr2=0x7f9868103e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:04.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.585+0000 7f986e800640 1 --2- 172.21.15.39:0/2037361257 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9868103a10 0x7f9868103e30 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f9858009860 tx=0x7f985802bb70 comp rx=0 tx=0).stop 2024-03-18T16:19:04.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.586+0000 7f986e800640 1 -- 172.21.15.39:0/2037361257 shutdown_connections 2024-03-18T16:19:04.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.586+0000 7f986e800640 1 --2- 172.21.15.39:0/2037361257 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f98681055c0 0x7f986810be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.586+0000 7f986e800640 1 --2- 172.21.15.39:0/2037361257 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9868104c00 0x7f9868105080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.586+0000 7f986e800640 1 --2- 172.21.15.39:0/2037361257 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9868103a10 0x7f9868103e30 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.586+0000 7f986e800640 1 -- 172.21.15.39:0/2037361257 >> 172.21.15.39:0/2037361257 conn(0x7f98680fefa0 msgr2=0x7f98681013c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:04.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.586+0000 7f986e800640 1 -- 172.21.15.39:0/2037361257 shutdown_connections 2024-03-18T16:19:04.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.586+0000 7f986e800640 1 -- 172.21.15.39:0/2037361257 wait complete. 2024-03-18T16:19:04.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.587+0000 7f986e800640 1 Processor -- start 2024-03-18T16:19:04.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.587+0000 7f986e800640 1 -- start start 2024-03-18T16:19:04.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.588+0000 7f986e800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9868103a10 0x7f9868199cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:04.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.588+0000 7f986e800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9868104c00 0x7f986819a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:04.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.588+0000 7f986e800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f98681055c0 0x7f98681a1260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:04.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.588+0000 7f986e800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f986810f660 con 0x7f9868104c00 2024-03-18T16:19:04.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.588+0000 7f986e800640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f986810f4e0 con 0x7f98681055c0 2024-03-18T16:19:04.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.588+0000 7f986e800640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f986810f7e0 con 0x7f9868103a10 2024-03-18T16:19:04.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.588+0000 7f9867400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9868103a10 0x7f9868199cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:04.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.588+0000 7f9867400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9868103a10 0x7f9868199cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:59460/0 (socket says 172.21.15.39:59460) 2024-03-18T16:19:04.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.588+0000 7f9867400640 1 -- 172.21.15.39:0/1576093314 learned_addr learned my addr 172.21.15.39:0/1576093314 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:19:04.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.588+0000 7f9867e00640 1 --2- 172.21.15.39:0/1576093314 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f98681055c0 0x7f98681a1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:04.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.588+0000 7f9866a00640 1 --2- 172.21.15.39:0/1576093314 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9868104c00 0x7f986819a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:04.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.588+0000 7f9867400640 1 -- 172.21.15.39:0/1576093314 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f98681055c0 msgr2=0x7f98681a1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:04.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.589+0000 7f9867400640 1 --2- 172.21.15.39:0/1576093314 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f98681055c0 0x7f98681a1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.589+0000 7f9867400640 1 -- 172.21.15.39:0/1576093314 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9868104c00 msgr2=0x7f986819a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:04.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.589+0000 7f9867400640 1 --2- 172.21.15.39:0/1576093314 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9868104c00 0x7f986819a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.589+0000 7f9867400640 1 -- 172.21.15.39:0/1576093314 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98681a1900 con 0x7f9868103a10 2024-03-18T16:19:04.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.589+0000 7f9867e00640 1 --2- 172.21.15.39:0/1576093314 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f98681055c0 0x7f98681a1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:19:04.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.589+0000 7f9867400640 1 --2- 172.21.15.39:0/1576093314 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9868103a10 0x7f9868199cf0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f985802c0b0 tx=0x7f9858032430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:04.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.589+0000 7f985fe00640 1 -- 172.21.15.39:0/1576093314 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9858041020 con 0x7f9868103a10 2024-03-18T16:19:04.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.589+0000 7f985fe00640 1 -- 172.21.15.39:0/1576093314 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f985802c9f0 con 0x7f9868103a10 2024-03-18T16:19:04.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.589+0000 7f985fe00640 1 -- 172.21.15.39:0/1576093314 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9858039040 con 0x7f9868103a10 2024-03-18T16:19:04.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.590+0000 7f986e800640 1 -- 172.21.15.39:0/1576093314 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98681a1b90 con 0x7f9868103a10 2024-03-18T16:19:04.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.590+0000 7f986e800640 1 -- 172.21.15.39:0/1576093314 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98681a2030 con 0x7f9868103a10 2024-03-18T16:19:04.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.594+0000 7f985fe00640 1 -- 172.21.15.39:0/1576093314 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f9858044020 con 0x7f9868103a10 2024-03-18T16:19:04.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.595+0000 7f987142c740 1 Processor -- start 2024-03-18T16:19:04.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.596+0000 7f987142c740 1 -- start start 2024-03-18T16:19:04.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.596+0000 7f987142c740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a400ae60d0 0x55a400ae64f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:04.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.596+0000 7f987142c740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a400ae6ac0 0x55a400aeb5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:04.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.596+0000 7f987142c740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a400aebc30 0x55a400aee0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:04.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.596+0000 7f987142c740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55a400aa53b0 con 0x55a400ae6ac0 2024-03-18T16:19:04.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.596+0000 7f987142c740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55a400a42450 con 0x55a400ae60d0 2024-03-18T16:19:04.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.596+0000 7f987142c740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55a4009eacb0 con 0x55a400aebc30 2024-03-18T16:19:04.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.597+0000 7f985fe00640 1 --2- 172.21.15.39:0/1576093314 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f984c0a2950 0x7f984c0a4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:04.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.597+0000 7f9867e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a400ae60d0 0x55a400ae64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:04.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.597+0000 7f985fe00640 1 -- 172.21.15.39:0/1576093314 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f98580f0730 con 0x7f9868103a10 2024-03-18T16:19:04.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.597+0000 7f9867e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a400ae60d0 0x55a400ae64f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:35074/0 (socket says 172.21.15.39:35074) 2024-03-18T16:19:04.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.597+0000 7f9867e00640 1 -- 172.21.15.39:0/524269577 learned_addr learned my addr 172.21.15.39:0/524269577 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:19:04.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.597+0000 7f9867400640 1 --2- 172.21.15.39:0/524269577 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a400ae6ac0 0x55a400aeb5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:04.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.597+0000 7f9867e00640 1 --2- 172.21.15.39:0/1576093314 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f984c0a2950 0x7f984c0a4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:04.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.597+0000 7f9866a00640 1 --2- 172.21.15.39:0/524269577 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a400aebc30 0x55a400aee0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:04.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.598+0000 7f9867e00640 1 -- 172.21.15.39:0/524269577 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a400aebc30 msgr2=0x55a400aee0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:04.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.598+0000 7f9867e00640 1 --2- 172.21.15.39:0/524269577 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a400aebc30 0x55a400aee0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.598+0000 7f9867e00640 1 -- 172.21.15.39:0/524269577 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a400ae6ac0 msgr2=0x55a400aeb5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:04.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.598+0000 7f9867e00640 1 --2- 172.21.15.39:0/524269577 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a400ae6ac0 0x55a400aeb5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.598+0000 7f9867e00640 1 -- 172.21.15.39:0/524269577 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a400a0c1f0 con 0x55a400ae60d0 2024-03-18T16:19:04.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.599+0000 7f9867e00640 1 --2- 172.21.15.39:0/524269577 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a400ae60d0 0x55a400ae64f0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f9860013eb0 tx=0x7f98600117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c0b4df0fc270bf23 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:04.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.599+0000 7f9867e00640 1 --2- 172.21.15.39:0/1576093314 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f984c0a2950 0x7f984c0a4e10 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f9860009fd0 tx=0x7f986001c000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:04.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.599+0000 7f985cc00640 1 -- 172.21.15.39:0/524269577 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f986001e610 con 0x55a400ae60d0 2024-03-18T16:19:04.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.599+0000 7f985cc00640 1 -- 172.21.15.39:0/524269577 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f986000ee50 con 0x55a400ae60d0 2024-03-18T16:19:04.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.599+0000 7f985cc00640 1 -- 172.21.15.39:0/524269577 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9860026a60 con 0x55a400ae60d0 2024-03-18T16:19:04.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.600+0000 7f987142c740 1 -- 172.21.15.39:0/524269577 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a400ae60d0 msgr2=0x55a400ae64f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:04.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.600+0000 7f987142c740 1 --2- 172.21.15.39:0/524269577 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a400ae60d0 0x55a400ae64f0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f9860013eb0 tx=0x7f98600117b0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.600+0000 7f987142c740 1 -- 172.21.15.39:0/524269577 shutdown_connections 2024-03-18T16:19:04.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.600+0000 7f987142c740 1 --2- 172.21.15.39:0/524269577 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a400aebc30 0x55a400aee0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.600+0000 7f987142c740 1 --2- 172.21.15.39:0/524269577 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a400ae6ac0 0x55a400aeb5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.600+0000 7f987142c740 1 --2- 172.21.15.39:0/524269577 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a400ae60d0 0x55a400ae64f0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.600+0000 7f987142c740 1 -- 172.21.15.39:0/524269577 >> 172.21.15.39:0/524269577 conn(0x55a400ae1640 msgr2=0x55a400ae1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:04.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.600+0000 7f987142c740 1 -- 172.21.15.39:0/524269577 shutdown_connections 2024-03-18T16:19:04.603 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.601+0000 7f987142c740 1 -- 172.21.15.39:0/524269577 wait complete. 2024-03-18T16:19:04.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.601+0000 7f987142c740 1 -- start start 2024-03-18T16:19:04.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.601+0000 7f987142c740 1 Processor -- start 2024-03-18T16:19:04.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.602+0000 7f987142c740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a400ae60d0 0x55a400af03d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:04.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.602+0000 7f987142c740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a400ae6ac0 0x55a400af0910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:04.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.602+0000 7f987142c740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a400aebc30 0x55a400af7950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:04.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.602+0000 7f987142c740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55a4009fe110 con 0x55a400ae6ac0 2024-03-18T16:19:04.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.602+0000 7f987142c740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55a400a3e4f0 con 0x55a400ae60d0 2024-03-18T16:19:04.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.602+0000 7f987142c740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55a400aeeb80 con 0x55a400aebc30 2024-03-18T16:19:04.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.602+0000 7f987142c740 10 client.0 _ll_register_callbacks cb 0x55a4009ebb70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:19:04.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.602+0000 7f9867400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a400ae60d0 0x55a400af03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:04.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.602+0000 7f9867400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a400ae60d0 0x55a400af03d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:35086/0 (socket says 172.21.15.39:35086) 2024-03-18T16:19:04.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.602+0000 7f9867e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a400aebc30 0x55a400af7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:04.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.602+0000 7f9867400640 1 -- 172.21.15.39:0/2340940928 learned_addr learned my addr 172.21.15.39:0/2340940928 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:19:04.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.602+0000 7f9866a00640 1 --2- 172.21.15.39:0/2340940928 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a400ae6ac0 0x55a400af0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:04.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.602+0000 7f9867400640 1 -- 172.21.15.39:0/2340940928 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a400aebc30 msgr2=0x55a400af7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:04.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.602+0000 7f9867400640 1 --2- 172.21.15.39:0/2340940928 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a400aebc30 0x55a400af7950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.602+0000 7f9867400640 1 -- 172.21.15.39:0/2340940928 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a400ae6ac0 msgr2=0x55a400af0910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:04.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.602+0000 7f9867400640 1 --2- 172.21.15.39:0/2340940928 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a400ae6ac0 0x55a400af0910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.602+0000 7f9867400640 1 -- 172.21.15.39:0/2340940928 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a4007563f0 con 0x55a400ae60d0 2024-03-18T16:19:04.606 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.602+0000 7f9866a00640 1 --2- 172.21.15.39:0/2340940928 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a400ae6ac0 0x55a400af0910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:19:04.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.603+0000 7f9867400640 1 --2- 172.21.15.39:0/2340940928 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a400ae60d0 0x55a400af03d0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f9858000c00 tx=0x7f98580bcb10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb9745bdc26d8299 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:04.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.603+0000 7f9851600640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:19:04.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.603+0000 7f9851600640 1 -- 172.21.15.39:0/2340940928 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98580bae80 con 0x55a400ae60d0 2024-03-18T16:19:04.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.603+0000 7f9851600640 1 -- 172.21.15.39:0/2340940928 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f985802cba0 con 0x55a400ae60d0 2024-03-18T16:19:04.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.603+0000 7f987142c740 1 -- 172.21.15.39:0/2340940928 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f986000ee50 con 0x55a400ae60d0 2024-03-18T16:19:04.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.603+0000 7f9851600640 1 -- 172.21.15.39:0/2340940928 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f98580bd440 con 0x55a400ae60d0 2024-03-18T16:19:04.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.603+0000 7f987142c740 10 client.27401 fetch_fsmap learned FSMap version 44 2024-03-18T16:19:04.608 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.603+0000 7f987142c740 1 -- 172.21.15.39:0/2340940928 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x55a4008207d0 con 0x55a400ae60d0 2024-03-18T16:19:04.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.604+0000 7f9851600640 1 -- 172.21.15.39:0/2340940928 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f986000ee50 con 0x55a400ae60d0 2024-03-18T16:19:04.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.604+0000 7f9851600640 1 -- 172.21.15.39:0/2340940928 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7f985803c030 con 0x55a400ae60d0 2024-03-18T16:19:04.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.604+0000 7f987142c740 10 client.27401 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:19:04.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.604+0000 7f987142c740 10 client.27401 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-18T16:19:04.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.604+0000 7f987142c740 20 client.27401 populate_metadata read hostname 'smithi039' 2024-03-18T16:19:04.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.604+0000 7f987142c740 1 --2- 172.21.15.39:0/2340940928 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x55a400afc220 0x55a400afe690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:04.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.604+0000 7f987142c740 4 client.27401 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:19:04.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.604+0000 7f987142c740 1 -- 172.21.15.39:0/2340940928 --> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a400a4f990 con 0x55a400afc220 2024-03-18T16:19:04.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.604+0000 7f9866a00640 1 --2- 172.21.15.39:0/2340940928 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x55a400afc220 0x55a400afe690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:04.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.605+0000 7f9866a00640 1 --2- 172.21.15.39:0/2340940928 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x55a400afc220 0x55a400afe690 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=fe33a581e8a33476 server_cookie=1044f66c7e091334 in_seq=0 out_seq=0 2024-03-18T16:19:04.609 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.605+0000 7f9851600640 10 client.27401 ms_handle_connect on v2:172.21.15.196:6836/895247251 2024-03-18T16:19:04.610 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.606+0000 7f9851600640 1 -- 172.21.15.39:0/2340940928 <== mds.2 v2:172.21.15.196:6836/895247251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55a400a4f990 con 0x55a400afc220 2024-03-18T16:19:04.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.606+0000 7f9851600640 10 client.27401 handle_command_reply: tid=0 2024-03-18T16:19:04.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.606+0000 7f987142c740 1 client.27401 shutdown 2024-03-18T16:19:04.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.606+0000 7f987142c740 1 -- 172.21.15.39:0/2340940928 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a400ae60d0 msgr2=0x55a400af03d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:04.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.606+0000 7f987142c740 1 --2- 172.21.15.39:0/2340940928 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a400ae60d0 0x55a400af03d0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f9858000c00 tx=0x7f98580bcb10 comp rx=0 tx=0).stop 2024-03-18T16:19:04.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.606+0000 7f987142c740 1 -- 172.21.15.39:0/2340940928 shutdown_connections 2024-03-18T16:19:04.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.606+0000 7f987142c740 1 --2- 172.21.15.39:0/2340940928 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x55a400afc220 0x55a400afe690 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.606+0000 7f987142c740 1 --2- 172.21.15.39:0/2340940928 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a400aebc30 0x55a400af7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.606+0000 7f987142c740 1 --2- 172.21.15.39:0/2340940928 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a400ae6ac0 0x55a400af0910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.606+0000 7f987142c740 1 --2- 172.21.15.39:0/2340940928 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a400ae60d0 0x55a400af03d0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.606+0000 7f987142c740 1 -- 172.21.15.39:0/2340940928 >> 172.21.15.39:0/2340940928 conn(0x55a400ae1640 msgr2=0x55a400ae2890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:04.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.606+0000 7f9851600640 0 client.27401 ms_handle_reset on v2:172.21.15.196:6836/895247251 2024-03-18T16:19:04.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.607+0000 7f987142c740 1 -- 172.21.15.39:0/2340940928 shutdown_connections 2024-03-18T16:19:04.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.607+0000 7f987142c740 1 -- 172.21.15.39:0/2340940928 wait complete. 2024-03-18T16:19:04.611 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.607+0000 7f987142c740 20 client.27401 trim_cache size 0 max 16384 2024-03-18T16:19:04.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.628+0000 7f987142c740 1 Processor -- start 2024-03-18T16:19:04.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.628+0000 7f987142c740 1 -- start start 2024-03-18T16:19:04.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.629+0000 7f987142c740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a400ae60d0 0x55a400aeffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:04.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.629+0000 7f987142c740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a400ae6ac0 0x55a400af78e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:04.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.629+0000 7f987142c740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a400aebc30 0x55a400af0520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:04.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.629+0000 7f987142c740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f98681a8150 con 0x55a400ae6ac0 2024-03-18T16:19:04.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.629+0000 7f987142c740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f98681a9c90 con 0x55a400ae60d0 2024-03-18T16:19:04.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.629+0000 7f987142c740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f98681a8820 con 0x55a400aebc30 2024-03-18T16:19:04.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.630+0000 7f9867400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a400ae60d0 0x55a400aeffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:04.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.630+0000 7f9867400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a400ae60d0 0x55a400aeffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:35092/0 (socket says 172.21.15.39:35092) 2024-03-18T16:19:04.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.630+0000 7f9867400640 1 -- 172.21.15.39:0/1732805092 learned_addr learned my addr 172.21.15.39:0/1732805092 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:19:04.634 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.630+0000 7f9867e00640 1 --2- 172.21.15.39:0/1732805092 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a400aebc30 0x55a400af0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:04.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.630+0000 7f9866a00640 1 --2- 172.21.15.39:0/1732805092 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a400ae6ac0 0x55a400af78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:04.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.630+0000 7f9867e00640 1 -- 172.21.15.39:0/1732805092 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a400ae60d0 msgr2=0x55a400aeffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:04.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.630+0000 7f9867e00640 1 --2- 172.21.15.39:0/1732805092 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a400ae60d0 0x55a400aeffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.630+0000 7f9867e00640 1 -- 172.21.15.39:0/1732805092 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a400ae6ac0 msgr2=0x55a400af78e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:04.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.630+0000 7f9867e00640 1 --2- 172.21.15.39:0/1732805092 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a400ae6ac0 0x55a400af78e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.630+0000 7f9867e00640 1 -- 172.21.15.39:0/1732805092 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a400ad4c10 con 0x55a400aebc30 2024-03-18T16:19:04.635 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.630+0000 7f9866a00640 1 --2- 172.21.15.39:0/1732805092 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a400ae6ac0 0x55a400af78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:19:04.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.631+0000 7f9867e00640 1 --2- 172.21.15.39:0/1732805092 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a400aebc30 0x55a400af0520 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f9860011cb0 tx=0x7f9860013810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa84a46b2a7d876 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:04.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.631+0000 7f9853e00640 1 -- 172.21.15.39:0/1732805092 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9860004c00 con 0x55a400aebc30 2024-03-18T16:19:04.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.631+0000 7f9853e00640 1 -- 172.21.15.39:0/1732805092 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f986000ee50 con 0x55a400aebc30 2024-03-18T16:19:04.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.631+0000 7f9853e00640 1 -- 172.21.15.39:0/1732805092 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98600326b0 con 0x55a400aebc30 2024-03-18T16:19:04.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.632+0000 7f987142c740 1 -- 172.21.15.39:0/1732805092 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a400aebc30 msgr2=0x55a400af0520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:04.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.632+0000 7f987142c740 1 --2- 172.21.15.39:0/1732805092 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a400aebc30 0x55a400af0520 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f9860011cb0 tx=0x7f9860013810 comp rx=0 tx=0).stop 2024-03-18T16:19:04.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.632+0000 7f987142c740 1 -- 172.21.15.39:0/1732805092 shutdown_connections 2024-03-18T16:19:04.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.632+0000 7f987142c740 1 --2- 172.21.15.39:0/1732805092 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a400aebc30 0x55a400af0520 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.632+0000 7f987142c740 1 --2- 172.21.15.39:0/1732805092 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a400ae6ac0 0x55a400af78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.632+0000 7f987142c740 1 --2- 172.21.15.39:0/1732805092 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a400ae60d0 0x55a400aeffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.632+0000 7f987142c740 1 -- 172.21.15.39:0/1732805092 >> 172.21.15.39:0/1732805092 conn(0x55a400ae1640 msgr2=0x55a400ae1a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:04.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.633+0000 7f987142c740 1 -- 172.21.15.39:0/1732805092 shutdown_connections 2024-03-18T16:19:04.642 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.633+0000 7f987142c740 1 -- 172.21.15.39:0/1732805092 wait complete. 2024-03-18T16:19:04.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.633+0000 7f987142c740 1 -- start start 2024-03-18T16:19:04.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.634+0000 7f987142c740 1 Processor -- start 2024-03-18T16:19:04.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.634+0000 7f987142c740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a400ae60d0 0x55a400af4d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:04.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.634+0000 7f987142c740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a400ae6ac0 0x55a400af5260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:04.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.634+0000 7f987142c740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a400aebc30 0x55a400af6220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:04.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.634+0000 7f987142c740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f98681a95c0 con 0x55a400ae60d0 2024-03-18T16:19:04.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.634+0000 7f987142c740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f98681a8ef0 con 0x55a400ae6ac0 2024-03-18T16:19:04.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.634+0000 7f987142c740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f98681a6130 con 0x55a400aebc30 2024-03-18T16:19:04.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.634+0000 7f987142c740 10 client.0 _ll_register_callbacks cb 0x7f98580097b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:19:04.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.634+0000 7f9867400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a400ae60d0 0x55a400af4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:04.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.634+0000 7f9867400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a400ae60d0 0x55a400af4d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:49118/0 (socket says 172.21.15.39:49118) 2024-03-18T16:19:04.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.634+0000 7f9867400640 1 -- 172.21.15.39:0/1186735851 learned_addr learned my addr 172.21.15.39:0/1186735851 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:19:04.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.635+0000 7f9867e00640 1 --2- 172.21.15.39:0/1186735851 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a400aebc30 0x55a400af6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:04.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.635+0000 7f9866a00640 1 --2- 172.21.15.39:0/1186735851 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a400ae6ac0 0x55a400af5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:04.643 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.635+0000 7f9867400640 1 -- 172.21.15.39:0/1186735851 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a400aebc30 msgr2=0x55a400af6220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:04.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.635+0000 7f9867400640 1 --2- 172.21.15.39:0/1186735851 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a400aebc30 0x55a400af6220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.635+0000 7f9867400640 1 -- 172.21.15.39:0/1186735851 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a400ae6ac0 msgr2=0x55a400af5260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:04.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.635+0000 7f9867400640 1 --2- 172.21.15.39:0/1186735851 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a400ae6ac0 0x55a400af5260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.635+0000 7f9867400640 1 -- 172.21.15.39:0/1186735851 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a400ae23b0 con 0x55a400ae60d0 2024-03-18T16:19:04.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.635+0000 7f9867e00640 1 --2- 172.21.15.39:0/1186735851 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a400aebc30 0x55a400af6220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:19:04.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.635+0000 7f9867400640 1 --2- 172.21.15.39:0/1186735851 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a400ae60d0 0x55a400af4d20 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f98580f0fd0 tx=0x7f98580058d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa87c0dee2f3b094 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:04.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.635+0000 7f9851600640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-03-18T16:19:04.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.636+0000 7f9851600640 1 -- 172.21.15.39:0/1186735851 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98580bae80 con 0x55a400ae60d0 2024-03-18T16:19:04.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.636+0000 7f9851600640 1 -- 172.21.15.39:0/1186735851 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f985802cba0 con 0x55a400ae60d0 2024-03-18T16:19:04.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.637+0000 7f987142c740 1 -- 172.21.15.39:0/1186735851 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f986000ee50 con 0x55a400ae60d0 2024-03-18T16:19:04.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.637+0000 7f9851600640 1 -- 172.21.15.39:0/1186735851 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f98580bc880 con 0x55a400ae60d0 2024-03-18T16:19:04.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.637+0000 7f987142c740 10 client.16773 fetch_fsmap learned FSMap version 44 2024-03-18T16:19:04.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.637+0000 7f987142c740 1 -- 172.21.15.39:0/1186735851 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x55a400af78b0 con 0x55a400ae60d0 2024-03-18T16:19:04.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.637+0000 7f9851600640 1 -- 172.21.15.39:0/1186735851 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f986000ee50 con 0x55a400ae60d0 2024-03-18T16:19:04.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.638+0000 7f9851600640 1 -- 172.21.15.39:0/1186735851 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7f98580cc030 con 0x55a400ae60d0 2024-03-18T16:19:04.644 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.638+0000 7f987142c740 10 client.16773 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:19:04.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.638+0000 7f987142c740 10 client.16773 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-18T16:19:04.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.638+0000 7f987142c740 20 client.16773 populate_metadata read hostname 'smithi039' 2024-03-18T16:19:04.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.638+0000 7f987142c740 1 --2- 172.21.15.39:0/1186735851 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x55a400b62690 0x55a400b64a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:04.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.638+0000 7f987142c740 4 client.16773 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:19:04.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.638+0000 7f987142c740 1 -- 172.21.15.39:0/1186735851 --> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a400ae7ad0 con 0x55a400b62690 2024-03-18T16:19:04.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.638+0000 7f9866a00640 1 --2- 172.21.15.39:0/1186735851 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x55a400b62690 0x55a400b64a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:04.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.639+0000 7f9866a00640 1 --2- 172.21.15.39:0/1186735851 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x55a400b62690 0x55a400b64a80 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=75eb9772249ade27 server_cookie=343c2fc5e5d8da1d in_seq=0 out_seq=0 2024-03-18T16:19:04.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.639+0000 7f9851600640 10 client.16773 ms_handle_connect on v2:172.21.15.196:6836/895247251 2024-03-18T16:19:04.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.639+0000 7f9851600640 1 -- 172.21.15.39:0/1186735851 <== mds.2 v2:172.21.15.196:6836/895247251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a400ae7ad0 con 0x55a400b62690 2024-03-18T16:19:04.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.639+0000 7f9851600640 10 client.16773 handle_command_reply: tid=0 2024-03-18T16:19:04.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.639+0000 7f987142c740 1 client.16773 shutdown 2024-03-18T16:19:04.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.639+0000 7f987142c740 1 -- 172.21.15.39:0/1186735851 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a400ae60d0 msgr2=0x55a400af4d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:04.645 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.639+0000 7f987142c740 1 --2- 172.21.15.39:0/1186735851 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a400ae60d0 0x55a400af4d20 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f98580f0fd0 tx=0x7f98580058d0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.640+0000 7f987142c740 1 -- 172.21.15.39:0/1186735851 shutdown_connections 2024-03-18T16:19:04.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.640+0000 7f987142c740 1 --2- 172.21.15.39:0/1186735851 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a400ae6ac0 0x55a400af5260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.640+0000 7f987142c740 1 --2- 172.21.15.39:0/1186735851 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x55a400b62690 0x55a400b64a80 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.640+0000 7f987142c740 1 --2- 172.21.15.39:0/1186735851 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55a400aebc30 0x55a400af6220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.640+0000 7f987142c740 1 --2- 172.21.15.39:0/1186735851 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55a400ae60d0 0x55a400af4d20 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.646 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.640+0000 7f987142c740 1 -- 172.21.15.39:0/1186735851 >> 172.21.15.39:0/1186735851 conn(0x55a400ae1640 msgr2=0x55a400ad7960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:04.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.640+0000 7f9851600640 0 client.16773 ms_handle_reset on v2:172.21.15.196:6836/895247251 2024-03-18T16:19:04.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.640+0000 7f987142c740 1 -- 172.21.15.39:0/1186735851 shutdown_connections 2024-03-18T16:19:04.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.640+0000 7f987142c740 1 -- 172.21.15.39:0/1186735851 wait complete. 2024-03-18T16:19:04.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.640+0000 7f987142c740 20 client.16773 trim_cache size 0 max 16384 2024-03-18T16:19:04.647 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:19:04.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.641+0000 7f986e800640 1 -- 172.21.15.39:0/1576093314 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f984c0a2950 msgr2=0x7f984c0a4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:04.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.641+0000 7f986e800640 1 --2- 172.21.15.39:0/1576093314 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f984c0a2950 0x7f984c0a4e10 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f9860009fd0 tx=0x7f986001c000 comp rx=0 tx=0).stop 2024-03-18T16:19:04.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.641+0000 7f986e800640 1 -- 172.21.15.39:0/1576093314 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9868103a10 msgr2=0x7f9868199cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:04.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.641+0000 7f986e800640 1 --2- 172.21.15.39:0/1576093314 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9868103a10 0x7f9868199cf0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f985802c0b0 tx=0x7f9858032430 comp rx=0 tx=0).stop 2024-03-18T16:19:04.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.642+0000 7f986e800640 1 -- 172.21.15.39:0/1576093314 shutdown_connections 2024-03-18T16:19:04.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.642+0000 7f986e800640 1 --2- 172.21.15.39:0/1576093314 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f98681055c0 0x7f98681a1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.642+0000 7f986e800640 1 --2- 172.21.15.39:0/1576093314 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f984c0a2950 0x7f984c0a4e10 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.642+0000 7f986e800640 1 --2- 172.21.15.39:0/1576093314 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9868104c00 0x7f986819a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.642+0000 7f986e800640 1 --2- 172.21.15.39:0/1576093314 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9868103a10 0x7f9868199cf0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.642+0000 7f986e800640 1 -- 172.21.15.39:0/1576093314 >> 172.21.15.39:0/1576093314 conn(0x7f98680fefa0 msgr2=0x7f9868107f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:04.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.642+0000 7f986e800640 1 -- 172.21.15.39:0/1576093314 shutdown_connections 2024-03-18T16:19:04.648 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.642+0000 7f986e800640 1 -- 172.21.15.39:0/1576093314 wait complete. 2024-03-18T16:19:04.667 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-18T16:19:04.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.893+0000 7f296ee00640 1 -- 172.21.15.39:0/2627914517 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2968109670 msgr2=0x7f296810bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:04.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.893+0000 7f296ee00640 1 --2- 172.21.15.39:0/2627914517 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2968109670 0x7f296810bb10 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f2960009be0 tx=0x7f296002bf80 comp rx=0 tx=0).stop 2024-03-18T16:19:04.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.893+0000 7f296ee00640 1 -- 172.21.15.39:0/2627914517 shutdown_connections 2024-03-18T16:19:04.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.893+0000 7f296ee00640 1 --2- 172.21.15.39:0/2627914517 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2968109670 0x7f296810bb10 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.893+0000 7f296ee00640 1 --2- 172.21.15.39:0/2627914517 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f29680741e0 0x7f2968074640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.893+0000 7f296ee00640 1 --2- 172.21.15.39:0/2627914517 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f29680757c0 0x7f2968073c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.895+0000 7f296ee00640 1 -- 172.21.15.39:0/2627914517 >> 172.21.15.39:0/2627914517 conn(0x7f29680fcd50 msgr2=0x7f29680ff190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:04.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.895+0000 7f296ee00640 1 -- 172.21.15.39:0/2627914517 shutdown_connections 2024-03-18T16:19:04.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.895+0000 7f296ee00640 1 -- 172.21.15.39:0/2627914517 wait complete. 2024-03-18T16:19:04.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.895+0000 7f296ee00640 1 Processor -- start 2024-03-18T16:19:04.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.895+0000 7f296ee00640 1 -- start start 2024-03-18T16:19:04.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.895+0000 7f296ee00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f29680741e0 0x7f2968199c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:04.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.895+0000 7f296ee00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f29680757c0 0x7f296819a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:04.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.895+0000 7f296ee00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2968109670 0x7f29681a1230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:04.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.895+0000 7f296ee00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f296810f6b0 con 0x7f29680757c0 2024-03-18T16:19:04.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.895+0000 7f296ee00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f296810f530 con 0x7f2968109670 2024-03-18T16:19:04.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.895+0000 7f296ee00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f296810f830 con 0x7f29680741e0 2024-03-18T16:19:04.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.896+0000 7f296ce00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2968109670 0x7f29681a1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:04.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.896+0000 7f2967e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f29680741e0 0x7f2968199c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:04.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.896+0000 7f296ce00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2968109670 0x7f29681a1230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:35128/0 (socket says 172.21.15.39:35128) 2024-03-18T16:19:04.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.896+0000 7f296ce00640 1 -- 172.21.15.39:0/1589815673 learned_addr learned my addr 172.21.15.39:0/1589815673 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:19:04.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.896+0000 7f2967400640 1 --2- 172.21.15.39:0/1589815673 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f29680757c0 0x7f296819a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:04.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.896+0000 7f2967e00640 1 -- 172.21.15.39:0/1589815673 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2968109670 msgr2=0x7f29681a1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:04.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.896+0000 7f2967e00640 1 --2- 172.21.15.39:0/1589815673 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2968109670 0x7f29681a1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.896+0000 7f2967e00640 1 -- 172.21.15.39:0/1589815673 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f29680757c0 msgr2=0x7f296819a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:04.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.896+0000 7f2967e00640 1 --2- 172.21.15.39:0/1589815673 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f29680757c0 0x7f296819a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.896+0000 7f2967e00640 1 -- 172.21.15.39:0/1589815673 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29681a18d0 con 0x7f29680741e0 2024-03-18T16:19:04.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.896+0000 7f2967400640 1 --2- 172.21.15.39:0/1589815673 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f29680757c0 0x7f296819a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:19:04.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.896+0000 7f2967e00640 1 --2- 172.21.15.39:0/1589815673 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f29680741e0 0x7f2968199c90 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f2958009fd0 tx=0x7f295800ec20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:04.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.897+0000 7f2964c00640 1 -- 172.21.15.39:0/1589815673 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f295800ccc0 con 0x7f29680741e0 2024-03-18T16:19:04.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.897+0000 7f2964c00640 1 -- 172.21.15.39:0/1589815673 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f295800ee70 con 0x7f29680741e0 2024-03-18T16:19:04.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.897+0000 7f2964c00640 1 -- 172.21.15.39:0/1589815673 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2958004a00 con 0x7f29680741e0 2024-03-18T16:19:04.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.897+0000 7f296ee00640 1 -- 172.21.15.39:0/1589815673 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29681a1bc0 con 0x7f29680741e0 2024-03-18T16:19:04.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.897+0000 7f296ee00640 1 -- 172.21.15.39:0/1589815673 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29681a20c0 con 0x7f29680741e0 2024-03-18T16:19:04.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.901+0000 7f2971b65740 1 Processor -- start 2024-03-18T16:19:04.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.901+0000 7f2971b65740 1 -- start start 2024-03-18T16:19:04.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.901+0000 7f2971b65740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55614a72d8c0 0x55614a72dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:04.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.901+0000 7f2971b65740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55614a72e2b0 0x55614a732db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:04.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.901+0000 7f2971b65740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55614a733420 0x55614a7358b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:04.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.901+0000 7f2971b65740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55614a6f53b0 con 0x55614a733420 2024-03-18T16:19:04.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.901+0000 7f2971b65740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55614a692450 con 0x55614a72d8c0 2024-03-18T16:19:04.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.901+0000 7f2971b65740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55614a63acb0 con 0x55614a72e2b0 2024-03-18T16:19:04.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.901+0000 7f2964c00640 1 -- 172.21.15.39:0/1589815673 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f2958004ba0 con 0x7f29680741e0 2024-03-18T16:19:04.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.902+0000 7f296ce00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55614a72d8c0 0x55614a72dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:04.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.902+0000 7f296ce00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55614a72d8c0 0x55614a72dce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:35142/0 (socket says 172.21.15.39:35142) 2024-03-18T16:19:04.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.902+0000 7f296ce00640 1 -- 172.21.15.39:0/1050738110 learned_addr learned my addr 172.21.15.39:0/1050738110 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:19:04.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.902+0000 7f2967e00640 1 --2- 172.21.15.39:0/1050738110 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55614a72e2b0 0x55614a732db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:04.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.902+0000 7f2967400640 1 --2- 172.21.15.39:0/1050738110 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55614a733420 0x55614a7358b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:04.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.902+0000 7f296ce00640 1 -- 172.21.15.39:0/1050738110 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55614a72e2b0 msgr2=0x55614a732db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:04.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.902+0000 7f296ce00640 1 --2- 172.21.15.39:0/1050738110 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55614a72e2b0 0x55614a732db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.902+0000 7f296ce00640 1 -- 172.21.15.39:0/1050738110 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55614a733420 msgr2=0x55614a7358b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:04.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.902+0000 7f296ce00640 1 --2- 172.21.15.39:0/1050738110 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55614a733420 0x55614a7358b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.903+0000 7f296ce00640 1 -- 172.21.15.39:0/1050738110 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55614a65c1f0 con 0x55614a72d8c0 2024-03-18T16:19:04.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.903+0000 7f2967e00640 1 --2- 172.21.15.39:0/1050738110 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55614a72e2b0 0x55614a732db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:19:04.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.903+0000 7f296ce00640 1 --2- 172.21.15.39:0/1050738110 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55614a72d8c0 0x55614a72dce0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f2960009370 tx=0x7f2960009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df644957e488b112 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:04.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.903+0000 7f295d600640 1 -- 172.21.15.39:0/1050738110 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2960043070 con 0x55614a72d8c0 2024-03-18T16:19:04.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.904+0000 7f295d600640 1 -- 172.21.15.39:0/1050738110 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f296002ce00 con 0x55614a72d8c0 2024-03-18T16:19:04.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.904+0000 7f2964c00640 1 --2- 172.21.15.39:0/1589815673 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f294c0a2880 0x7f294c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:04.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.904+0000 7f295d600640 1 -- 172.21.15.39:0/1050738110 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29600046e0 con 0x55614a72d8c0 2024-03-18T16:19:04.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.904+0000 7f2964c00640 1 -- 172.21.15.39:0/1589815673 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f29580cf070 con 0x7f29680741e0 2024-03-18T16:19:04.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.904+0000 7f2967e00640 1 --2- 172.21.15.39:0/1589815673 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f294c0a2880 0x7f294c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:04.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.904+0000 7f2971b65740 1 -- 172.21.15.39:0/1050738110 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55614a72d8c0 msgr2=0x55614a72dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:04.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.904+0000 7f2971b65740 1 --2- 172.21.15.39:0/1050738110 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55614a72d8c0 0x55614a72dce0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f2960009370 tx=0x7f2960009840 comp rx=0 tx=0).stop 2024-03-18T16:19:04.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.905+0000 7f2971b65740 1 -- 172.21.15.39:0/1050738110 shutdown_connections 2024-03-18T16:19:04.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.905+0000 7f2971b65740 1 --2- 172.21.15.39:0/1050738110 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55614a733420 0x55614a7358b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.905+0000 7f2971b65740 1 --2- 172.21.15.39:0/1050738110 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55614a72e2b0 0x55614a732db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.905+0000 7f2971b65740 1 --2- 172.21.15.39:0/1050738110 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55614a72d8c0 0x55614a72dce0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.905+0000 7f2971b65740 1 -- 172.21.15.39:0/1050738110 >> 172.21.15.39:0/1050738110 conn(0x55614a728e30 msgr2=0x55614a729230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:04.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.905+0000 7f2971b65740 1 -- 172.21.15.39:0/1050738110 shutdown_connections 2024-03-18T16:19:04.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.905+0000 7f2967e00640 1 --2- 172.21.15.39:0/1589815673 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f294c0a2880 0x7f294c0a4d40 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f29580ccb50 tx=0x7f2958005380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:04.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.905+0000 7f2971b65740 1 -- 172.21.15.39:0/1050738110 wait complete. 2024-03-18T16:19:04.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.906+0000 7f2971b65740 1 -- start start 2024-03-18T16:19:04.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.906+0000 7f2971b65740 1 Processor -- start 2024-03-18T16:19:04.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.907+0000 7f2971b65740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55614a72d8c0 0x55614a737b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:04.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.907+0000 7f296ce00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55614a72d8c0 0x55614a737b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:04.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.907+0000 7f296ce00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55614a72d8c0 0x55614a737b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:35158/0 (socket says 172.21.15.39:35158) 2024-03-18T16:19:04.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.907+0000 7f2971b65740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55614a72e2b0 0x55614a738070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:04.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.908+0000 7f296ce00640 1 -- 172.21.15.39:0/3994132779 learned_addr learned my addr 172.21.15.39:0/3994132779 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:19:04.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.908+0000 7f2971b65740 1 --2- 172.21.15.39:0/3994132779 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55614a733420 0x55614a73f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:04.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.908+0000 7f2971b65740 1 -- 172.21.15.39:0/3994132779 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55614a64e110 con 0x55614a72e2b0 2024-03-18T16:19:04.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.908+0000 7f2971b65740 1 -- 172.21.15.39:0/3994132779 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55614a68e4f0 con 0x55614a72d8c0 2024-03-18T16:19:04.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.908+0000 7f2971b65740 1 -- 172.21.15.39:0/3994132779 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55614a736370 con 0x55614a733420 2024-03-18T16:19:04.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.908+0000 7f2967400640 1 --2- 172.21.15.39:0/3994132779 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55614a72e2b0 0x55614a738070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:04.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.908+0000 7f2971b65740 10 client.0 _ll_register_callbacks cb 0x55614a63bb70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:19:04.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.908+0000 7f296ce00640 1 --2- 172.21.15.39:0/3994132779 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55614a733420 0x55614a73f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:04.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.908+0000 7f296ce00640 1 -- 172.21.15.39:0/3994132779 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55614a733420 msgr2=0x55614a73f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:04.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.908+0000 7f296ce00640 1 --2- 172.21.15.39:0/3994132779 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55614a733420 0x55614a73f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.908+0000 7f296ce00640 1 -- 172.21.15.39:0/3994132779 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55614a72e2b0 msgr2=0x55614a738070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:04.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.908+0000 7f296ce00640 1 --2- 172.21.15.39:0/3994132779 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55614a72e2b0 0x55614a738070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.908+0000 7f296ce00640 1 -- 172.21.15.39:0/3994132779 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55614a3a63f0 con 0x55614a72d8c0 2024-03-18T16:19:04.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.909+0000 7f296ce00640 1 --2- 172.21.15.39:0/3994132779 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55614a72d8c0 0x55614a737b30 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f2960036040 tx=0x7f2960009960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=41ea6939819c2155 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:04.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.909+0000 7f2952000640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:19:04.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.909+0000 7f2952000640 1 -- 172.21.15.39:0/3994132779 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2960051030 con 0x55614a72d8c0 2024-03-18T16:19:04.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.909+0000 7f2952000640 1 -- 172.21.15.39:0/3994132779 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f2960004670 con 0x55614a72d8c0 2024-03-18T16:19:04.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.909+0000 7f2971b65740 1 -- 172.21.15.39:0/3994132779 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f296002ce00 con 0x55614a72d8c0 2024-03-18T16:19:04.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.909+0000 7f2952000640 1 -- 172.21.15.39:0/3994132779 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f296005c050 con 0x55614a72d8c0 2024-03-18T16:19:04.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.911+0000 7f2971b65740 10 client.27419 fetch_fsmap learned FSMap version 44 2024-03-18T16:19:04.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.911+0000 7f2971b65740 1 -- 172.21.15.39:0/3994132779 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x55614a4707d0 con 0x55614a72d8c0 2024-03-18T16:19:04.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.911+0000 7f2952000640 1 -- 172.21.15.39:0/3994132779 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f296002ce00 con 0x55614a72d8c0 2024-03-18T16:19:04.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.911+0000 7f2952000640 1 -- 172.21.15.39:0/3994132779 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7f29600484e0 con 0x55614a72d8c0 2024-03-18T16:19:04.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.912+0000 7f2971b65740 10 client.27419 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:19:04.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.912+0000 7f2971b65740 10 client.27419 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T16:19:04.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.912+0000 7f2971b65740 20 client.27419 populate_metadata read hostname 'smithi039' 2024-03-18T16:19:04.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.912+0000 7f2971b65740 1 --2- 172.21.15.39:0/3994132779 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x55614a7435a0 0x55614a745990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:04.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.912+0000 7f2971b65740 4 client.27419 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:19:04.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.912+0000 7f2971b65740 1 -- 172.21.15.39:0/3994132779 --> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55614a69f990 con 0x55614a7435a0 2024-03-18T16:19:04.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.912+0000 7f2967400640 1 --2- 172.21.15.39:0/3994132779 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x55614a7435a0 0x55614a745990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:04.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.913+0000 7f2967400640 1 --2- 172.21.15.39:0/3994132779 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x55614a7435a0 0x55614a745990 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8ada2cd5c804902f server_cookie=11675aab6b464b94 in_seq=0 out_seq=0 2024-03-18T16:19:04.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.913+0000 7f2952000640 10 client.27419 ms_handle_connect on v2:172.21.15.196:6838/2001427268 2024-03-18T16:19:04.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.914+0000 7f2952000640 1 -- 172.21.15.39:0/3994132779 <== mds.1 v2:172.21.15.196:6838/2001427268 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55614a69f990 con 0x55614a7435a0 2024-03-18T16:19:04.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.914+0000 7f2952000640 10 client.27419 handle_command_reply: tid=0 2024-03-18T16:19:04.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.914+0000 7f2971b65740 1 client.27419 shutdown 2024-03-18T16:19:04.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.914+0000 7f2971b65740 1 -- 172.21.15.39:0/3994132779 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55614a72d8c0 msgr2=0x55614a737b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:04.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.914+0000 7f2971b65740 1 --2- 172.21.15.39:0/3994132779 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55614a72d8c0 0x55614a737b30 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f2960036040 tx=0x7f2960009960 comp rx=0 tx=0).stop 2024-03-18T16:19:04.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.914+0000 7f2971b65740 1 -- 172.21.15.39:0/3994132779 shutdown_connections 2024-03-18T16:19:04.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.914+0000 7f2971b65740 1 --2- 172.21.15.39:0/3994132779 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x55614a7435a0 0x55614a745990 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.914+0000 7f2971b65740 1 --2- 172.21.15.39:0/3994132779 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55614a733420 0x55614a73f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.914+0000 7f2971b65740 1 --2- 172.21.15.39:0/3994132779 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55614a72e2b0 0x55614a738070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.914+0000 7f2971b65740 1 --2- 172.21.15.39:0/3994132779 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55614a72d8c0 0x55614a737b30 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.914+0000 7f2971b65740 1 -- 172.21.15.39:0/3994132779 >> 172.21.15.39:0/3994132779 conn(0x55614a728e30 msgr2=0x55614a7281f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:04.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.914+0000 7f2952000640 0 client.27419 ms_handle_reset on v2:172.21.15.196:6838/2001427268 2024-03-18T16:19:04.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.915+0000 7f2971b65740 1 -- 172.21.15.39:0/3994132779 shutdown_connections 2024-03-18T16:19:04.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.915+0000 7f2971b65740 1 -- 172.21.15.39:0/3994132779 wait complete. 2024-03-18T16:19:04.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.915+0000 7f2971b65740 20 client.27419 trim_cache size 0 max 16384 2024-03-18T16:19:04.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.936+0000 7f2971b65740 1 Processor -- start 2024-03-18T16:19:04.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.937+0000 7f2971b65740 1 -- start start 2024-03-18T16:19:04.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.937+0000 7f2971b65740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55614a72d8c0 0x55614a737fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:04.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.937+0000 7f2971b65740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55614a72e2b0 0x55614a73f070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:04.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.937+0000 7f2971b65740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55614a733420 0x55614a727960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:04.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.937+0000 7f2971b65740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f29681ac540 con 0x55614a733420 2024-03-18T16:19:04.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.937+0000 7f2971b65740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f29681ae080 con 0x55614a72e2b0 2024-03-18T16:19:04.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.937+0000 7f2971b65740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f29681acc10 con 0x55614a72d8c0 2024-03-18T16:19:04.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.937+0000 7f296ce00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55614a72d8c0 0x55614a737fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:04.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.937+0000 7f296ce00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55614a72d8c0 0x55614a737fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:59522/0 (socket says 172.21.15.39:59522) 2024-03-18T16:19:04.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.937+0000 7f296ce00640 1 -- 172.21.15.39:0/164929305 learned_addr learned my addr 172.21.15.39:0/164929305 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:19:04.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.938+0000 7f296ce00640 1 --2- 172.21.15.39:0/164929305 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55614a733420 0x55614a727960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:04.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.938+0000 7f2967400640 1 --2- 172.21.15.39:0/164929305 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55614a72e2b0 0x55614a73f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:04.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.938+0000 7f296ce00640 1 -- 172.21.15.39:0/164929305 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55614a72e2b0 msgr2=0x55614a73f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:04.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.938+0000 7f296ce00640 1 --2- 172.21.15.39:0/164929305 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55614a72e2b0 0x55614a73f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.938+0000 7f296ce00640 1 -- 172.21.15.39:0/164929305 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55614a733420 msgr2=0x55614a727960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:04.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.938+0000 7f296ce00640 1 --2- 172.21.15.39:0/164929305 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55614a733420 0x55614a727960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.938+0000 7f296ce00640 1 -- 172.21.15.39:0/164929305 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55614a724c10 con 0x55614a72d8c0 2024-03-18T16:19:04.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.938+0000 7f296ce00640 1 --2- 172.21.15.39:0/164929305 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55614a72d8c0 0x55614a737fd0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f2960003f60 tx=0x7f296004d040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d6ecb11a3f0f2e1c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:04.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.939+0000 7f295cc00640 1 -- 172.21.15.39:0/164929305 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2960051040 con 0x55614a72d8c0 2024-03-18T16:19:04.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.939+0000 7f295cc00640 1 -- 172.21.15.39:0/164929305 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2960004260 con 0x55614a72d8c0 2024-03-18T16:19:04.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.939+0000 7f295cc00640 1 -- 172.21.15.39:0/164929305 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2960043070 con 0x55614a72d8c0 2024-03-18T16:19:04.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.940+0000 7f2971b65740 1 -- 172.21.15.39:0/164929305 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55614a72d8c0 msgr2=0x55614a737fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:04.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.940+0000 7f2971b65740 1 --2- 172.21.15.39:0/164929305 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55614a72d8c0 0x55614a737fd0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f2960003f60 tx=0x7f296004d040 comp rx=0 tx=0).stop 2024-03-18T16:19:04.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.940+0000 7f2971b65740 1 -- 172.21.15.39:0/164929305 shutdown_connections 2024-03-18T16:19:04.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.940+0000 7f2971b65740 1 --2- 172.21.15.39:0/164929305 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55614a72e2b0 0x55614a73f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.940+0000 7f2971b65740 1 --2- 172.21.15.39:0/164929305 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55614a733420 0x55614a727960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.940+0000 7f2971b65740 1 --2- 172.21.15.39:0/164929305 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55614a72d8c0 0x55614a737fd0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.940+0000 7f2971b65740 1 -- 172.21.15.39:0/164929305 >> 172.21.15.39:0/164929305 conn(0x55614a728e30 msgr2=0x55614a729210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:04.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.940+0000 7f2971b65740 1 -- 172.21.15.39:0/164929305 shutdown_connections 2024-03-18T16:19:04.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.940+0000 7f2971b65740 1 -- 172.21.15.39:0/164929305 wait complete. 2024-03-18T16:19:04.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.940+0000 7f2971b65740 1 -- start start 2024-03-18T16:19:04.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.941+0000 7f2971b65740 1 Processor -- start 2024-03-18T16:19:04.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.941+0000 7f2971b65740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55614a72d8c0 0x55614a73c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:04.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.941+0000 7f2971b65740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55614a72e2b0 0x55614a73cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:04.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.941+0000 7f2971b65740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55614a733420 0x55614a73ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:04.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.941+0000 7f2971b65740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f29681ad9b0 con 0x55614a72e2b0 2024-03-18T16:19:04.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.941+0000 7f2971b65740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f29681ad2e0 con 0x55614a72d8c0 2024-03-18T16:19:04.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.941+0000 7f2971b65740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f29681aa520 con 0x55614a733420 2024-03-18T16:19:04.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.941+0000 7f2971b65740 10 client.0 _ll_register_callbacks cb 0x7f2958005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:19:04.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.942+0000 7f296ce00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55614a72d8c0 0x55614a73c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:04.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.942+0000 7f2967400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55614a72e2b0 0x55614a73cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:04.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.942+0000 7f296ce00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55614a72d8c0 0x55614a73c9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:35178/0 (socket says 172.21.15.39:35178) 2024-03-18T16:19:04.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.942+0000 7f296ce00640 1 -- 172.21.15.39:0/1811359528 learned_addr learned my addr 172.21.15.39:0/1811359528 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:19:04.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.942+0000 7f296ce00640 1 --2- 172.21.15.39:0/1811359528 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55614a733420 0x55614a73ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:04.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.942+0000 7f296ce00640 1 -- 172.21.15.39:0/1811359528 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55614a733420 msgr2=0x55614a73ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:04.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.942+0000 7f296ce00640 1 --2- 172.21.15.39:0/1811359528 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55614a733420 0x55614a73ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.942+0000 7f296ce00640 1 -- 172.21.15.39:0/1811359528 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55614a72e2b0 msgr2=0x55614a73cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:04.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.942+0000 7f296ce00640 1 --2- 172.21.15.39:0/1811359528 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55614a72e2b0 0x55614a73cf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.942+0000 7f296ce00640 1 -- 172.21.15.39:0/1811359528 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55614a72d500 con 0x55614a72d8c0 2024-03-18T16:19:04.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.942+0000 7f296ce00640 1 --2- 172.21.15.39:0/1811359528 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55614a72d8c0 0x55614a73c9f0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f2960048e80 tx=0x7f2960003ec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fc5cff6d099a7181 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:04.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.942+0000 7f2952000640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:19:04.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.943+0000 7f2952000640 1 -- 172.21.15.39:0/1811359528 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2960051070 con 0x55614a72d8c0 2024-03-18T16:19:04.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.943+0000 7f2952000640 1 -- 172.21.15.39:0/1811359528 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f2960048450 con 0x55614a72d8c0 2024-03-18T16:19:04.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.943+0000 7f2971b65740 1 -- 172.21.15.39:0/1811359528 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2960004260 con 0x55614a72d8c0 2024-03-18T16:19:04.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.943+0000 7f2952000640 1 -- 172.21.15.39:0/1811359528 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f2960065040 con 0x55614a72d8c0 2024-03-18T16:19:04.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.943+0000 7f2971b65740 10 client.27425 fetch_fsmap learned FSMap version 44 2024-03-18T16:19:04.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.943+0000 7f2971b65740 1 -- 172.21.15.39:0/1811359528 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x55614a7a9860 con 0x55614a72d8c0 2024-03-18T16:19:04.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.943+0000 7f2952000640 1 -- 172.21.15.39:0/1811359528 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2960004260 con 0x55614a72d8c0 2024-03-18T16:19:04.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.943+0000 7f2952000640 1 -- 172.21.15.39:0/1811359528 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7f2960061440 con 0x55614a72d8c0 2024-03-18T16:19:04.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.944+0000 7f2971b65740 10 client.27425 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:19:04.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.944+0000 7f2971b65740 10 client.27425 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T16:19:04.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.944+0000 7f2971b65740 20 client.27425 populate_metadata read hostname 'smithi039' 2024-03-18T16:19:04.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.944+0000 7f2971b65740 1 --2- 172.21.15.39:0/1811359528 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x55614a7aa0a0 0x55614a7ac4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:04.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.944+0000 7f2971b65740 4 client.27425 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:19:04.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.944+0000 7f2971b65740 1 -- 172.21.15.39:0/1811359528 --> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55614a699ce0 con 0x55614a7aa0a0 2024-03-18T16:19:04.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.945+0000 7f2967400640 1 --2- 172.21.15.39:0/1811359528 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x55614a7aa0a0 0x55614a7ac4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:04.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.945+0000 7f2952000640 10 client.27425 ms_handle_connect on v2:172.21.15.196:6838/2001427268 2024-03-18T16:19:04.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.945+0000 7f2967400640 1 --2- 172.21.15.39:0/1811359528 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x55614a7aa0a0 0x55614a7ac4c0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d26c2b952d954066 server_cookie=8fa1e90c5f1121fd in_seq=0 out_seq=0 2024-03-18T16:19:04.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.945+0000 7f2952000640 1 -- 172.21.15.39:0/1811359528 <== mds.1 v2:172.21.15.196:6838/2001427268 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55614a699ce0 con 0x55614a7aa0a0 2024-03-18T16:19:04.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.945+0000 7f2952000640 10 client.27425 handle_command_reply: tid=0 2024-03-18T16:19:04.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.946+0000 7f2971b65740 1 client.27425 shutdown 2024-03-18T16:19:04.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.946+0000 7f2971b65740 1 -- 172.21.15.39:0/1811359528 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55614a72d8c0 msgr2=0x55614a73c9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:04.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.946+0000 7f2971b65740 1 --2- 172.21.15.39:0/1811359528 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55614a72d8c0 0x55614a73c9f0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f2960048e80 tx=0x7f2960003ec0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.950 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.946+0000 7f2971b65740 1 -- 172.21.15.39:0/1811359528 shutdown_connections 2024-03-18T16:19:04.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.946+0000 7f2971b65740 1 --2- 172.21.15.39:0/1811359528 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x55614a7aa0a0 0x55614a7ac4c0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.946+0000 7f2971b65740 1 --2- 172.21.15.39:0/1811359528 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55614a733420 0x55614a73ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.946+0000 7f2971b65740 1 --2- 172.21.15.39:0/1811359528 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55614a72e2b0 0x55614a73cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.946+0000 7f2952000640 0 client.27425 ms_handle_reset on v2:172.21.15.196:6838/2001427268 2024-03-18T16:19:04.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.946+0000 7f2971b65740 1 --2- 172.21.15.39:0/1811359528 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55614a72d8c0 0x55614a73c9f0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.946+0000 7f2971b65740 1 -- 172.21.15.39:0/1811359528 >> 172.21.15.39:0/1811359528 conn(0x55614a728e30 msgr2=0x55614a727960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:04.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.947+0000 7f2971b65740 1 -- 172.21.15.39:0/1811359528 shutdown_connections 2024-03-18T16:19:04.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.947+0000 7f2971b65740 1 -- 172.21.15.39:0/1811359528 wait complete. 2024-03-18T16:19:04.951 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.947+0000 7f2971b65740 20 client.27425 trim_cache size 0 max 16384 2024-03-18T16:19:04.951 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:19:04.952 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.947+0000 7f296ee00640 1 -- 172.21.15.39:0/1589815673 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f294c0a2880 msgr2=0x7f294c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:04.952 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.947+0000 7f296ee00640 1 --2- 172.21.15.39:0/1589815673 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f294c0a2880 0x7f294c0a4d40 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f29580ccb50 tx=0x7f2958005380 comp rx=0 tx=0).stop 2024-03-18T16:19:04.952 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.947+0000 7f296ee00640 1 -- 172.21.15.39:0/1589815673 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f29680741e0 msgr2=0x7f2968199c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:04.952 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.947+0000 7f296ee00640 1 --2- 172.21.15.39:0/1589815673 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f29680741e0 0x7f2968199c90 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f2958009fd0 tx=0x7f295800ec20 comp rx=0 tx=0).stop 2024-03-18T16:19:04.952 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.948+0000 7f296ee00640 1 -- 172.21.15.39:0/1589815673 shutdown_connections 2024-03-18T16:19:04.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.948+0000 7f296ee00640 1 --2- 172.21.15.39:0/1589815673 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2968109670 0x7f29681a1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.948+0000 7f296ee00640 1 --2- 172.21.15.39:0/1589815673 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f294c0a2880 0x7f294c0a4d40 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.948+0000 7f296ee00640 1 --2- 172.21.15.39:0/1589815673 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f29680757c0 0x7f296819a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.948+0000 7f296ee00640 1 --2- 172.21.15.39:0/1589815673 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f29680741e0 0x7f2968199c90 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:04.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.948+0000 7f296ee00640 1 -- 172.21.15.39:0/1589815673 >> 172.21.15.39:0/1589815673 conn(0x7f29680fcd50 msgr2=0x7f2968105600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:04.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.948+0000 7f296ee00640 1 -- 172.21.15.39:0/1589815673 shutdown_connections 2024-03-18T16:19:04.953 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:04.948+0000 7f296ee00640 1 -- 172.21.15.39:0/1589815673 wait complete. 2024-03-18T16:19:04.972 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-18T16:19:05.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.201+0000 7f66f7a00640 1 -- 172.21.15.39:0/4155780015 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f66f01055b0 msgr2=0x7f66f010be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:05.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.201+0000 7f66f7a00640 1 --2- 172.21.15.39:0/4155780015 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f66f01055b0 0x7f66f010be40 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f66e8009ca0 tx=0x7f66e802c100 comp rx=0 tx=0).stop 2024-03-18T16:19:05.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.201+0000 7f66f7a00640 1 -- 172.21.15.39:0/4155780015 shutdown_connections 2024-03-18T16:19:05.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.201+0000 7f66f7a00640 1 --2- 172.21.15.39:0/4155780015 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f66f01055b0 0x7f66f010be40 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:05.205 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.201+0000 7f66f7a00640 1 --2- 172.21.15.39:0/4155780015 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f66f0104bf0 0x7f66f0105070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:05.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.201+0000 7f66f7a00640 1 --2- 172.21.15.39:0/4155780015 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f66f0103a00 0x7f66f0103e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:05.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.201+0000 7f66f7a00640 1 -- 172.21.15.39:0/4155780015 >> 172.21.15.39:0/4155780015 conn(0x7f66f00fef90 msgr2=0x7f66f01013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:05.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.202+0000 7f66f7a00640 1 -- 172.21.15.39:0/4155780015 shutdown_connections 2024-03-18T16:19:05.206 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.202+0000 7f66f7a00640 1 -- 172.21.15.39:0/4155780015 wait complete. 2024-03-18T16:19:05.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.203+0000 7f66f7a00640 1 Processor -- start 2024-03-18T16:19:05.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.203+0000 7f66f7a00640 1 -- start start 2024-03-18T16:19:05.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.204+0000 7f66f7a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f66f0103a00 0x7f66f0199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:05.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.204+0000 7f66f7a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f66f0104bf0 0x7f66f019a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:05.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.204+0000 7f66f7a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f66f01055b0 0x7f66f01a1210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:05.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.204+0000 7f66f7a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f66f010f660 con 0x7f66f0104bf0 2024-03-18T16:19:05.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.204+0000 7f66f7a00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f66f010f4e0 con 0x7f66f0103a00 2024-03-18T16:19:05.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.204+0000 7f66f7a00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f66f010f7e0 con 0x7f66f01055b0 2024-03-18T16:19:05.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.204+0000 7f66f5000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f66f0103a00 0x7f66f0199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:05.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.204+0000 7f66f5000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f66f0103a00 0x7f66f0199c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:35204/0 (socket says 172.21.15.39:35204) 2024-03-18T16:19:05.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.204+0000 7f66f5000640 1 -- 172.21.15.39:0/1968206912 learned_addr learned my addr 172.21.15.39:0/1968206912 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:19:05.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.204+0000 7f66efe00640 1 --2- 172.21.15.39:0/1968206912 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f66f0104bf0 0x7f66f019a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:05.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.205+0000 7f66f5a00640 1 --2- 172.21.15.39:0/1968206912 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f66f01055b0 0x7f66f01a1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:05.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.205+0000 7f66f5000640 1 -- 172.21.15.39:0/1968206912 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f66f01055b0 msgr2=0x7f66f01a1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:05.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.205+0000 7f66f5000640 1 --2- 172.21.15.39:0/1968206912 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f66f01055b0 0x7f66f01a1210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:05.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.205+0000 7f66f5000640 1 -- 172.21.15.39:0/1968206912 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f66f0104bf0 msgr2=0x7f66f019a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:05.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.205+0000 7f66f5000640 1 --2- 172.21.15.39:0/1968206912 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f66f0104bf0 0x7f66f019a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:05.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.205+0000 7f66f5000640 1 -- 172.21.15.39:0/1968206912 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66f01a18b0 con 0x7f66f0103a00 2024-03-18T16:19:05.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.205+0000 7f66efe00640 1 --2- 172.21.15.39:0/1968206912 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f66f0104bf0 0x7f66f019a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:19:05.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.205+0000 7f66f5000640 1 --2- 172.21.15.39:0/1968206912 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f66f0103a00 0x7f66f0199c70 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f66e400e9c0 tx=0x7f66e400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:05.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.206+0000 7f66ed600640 1 -- 172.21.15.39:0/1968206912 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66e400cd80 con 0x7f66f0103a00 2024-03-18T16:19:05.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.206+0000 7f66ed600640 1 -- 172.21.15.39:0/1968206912 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f66e40035f0 con 0x7f66f0103a00 2024-03-18T16:19:05.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.206+0000 7f66ed600640 1 -- 172.21.15.39:0/1968206912 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66e4010690 con 0x7f66f0103a00 2024-03-18T16:19:05.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.206+0000 7f66f7a00640 1 -- 172.21.15.39:0/1968206912 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66f01a1ba0 con 0x7f66f0103a00 2024-03-18T16:19:05.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.206+0000 7f66f7a00640 1 -- 172.21.15.39:0/1968206912 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66f01a20a0 con 0x7f66f0103a00 2024-03-18T16:19:05.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.210+0000 7f66ed600640 1 -- 172.21.15.39:0/1968206912 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f66e40037a0 con 0x7f66f0103a00 2024-03-18T16:19:05.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.211+0000 7f66fa7d4740 1 Processor -- start 2024-03-18T16:19:05.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.212+0000 7f66fa7d4740 1 -- start start 2024-03-18T16:19:05.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.212+0000 7f66ed600640 1 --2- 172.21.15.39:0/1968206912 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f66d80a2950 0x7f66d80a4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:05.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.213+0000 7f66ed600640 1 -- 172.21.15.39:0/1968206912 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f66e40cf2b0 con 0x7f66f0103a00 2024-03-18T16:19:05.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.213+0000 7f66fa7d4740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55982daf00d0 0x55982daf04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:05.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.213+0000 7f66fa7d4740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55982daf0ac0 0x55982daf55c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:05.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.213+0000 7f66fa7d4740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55982daf5c30 0x55982daf80c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:05.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.213+0000 7f66fa7d4740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55982daaf3b0 con 0x55982daf5c30 2024-03-18T16:19:05.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.213+0000 7f66fa7d4740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55982da4c450 con 0x55982daf00d0 2024-03-18T16:19:05.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.213+0000 7f66fa7d4740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55982d9f4cb0 con 0x55982daf0ac0 2024-03-18T16:19:05.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.213+0000 7f66efe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55982daf0ac0 0x55982daf55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:05.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.213+0000 7f66efe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55982daf0ac0 0x55982daf55c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:59578/0 (socket says 172.21.15.39:59578) 2024-03-18T16:19:05.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.213+0000 7f66efe00640 1 -- 172.21.15.39:0/2522279421 learned_addr learned my addr 172.21.15.39:0/2522279421 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:19:05.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.213+0000 7f66f5a00640 1 --2- 172.21.15.39:0/2522279421 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55982daf5c30 0x55982daf80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:05.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.214+0000 7f66f5000640 1 --2- 172.21.15.39:0/2522279421 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55982daf00d0 0x55982daf04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:05.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.214+0000 7f66efe00640 1 -- 172.21.15.39:0/2522279421 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55982daf00d0 msgr2=0x55982daf04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:05.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.214+0000 7f66efe00640 1 --2- 172.21.15.39:0/2522279421 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55982daf00d0 0x55982daf04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:05.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.214+0000 7f66efe00640 1 -- 172.21.15.39:0/2522279421 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55982daf5c30 msgr2=0x55982daf80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:05.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.214+0000 7f66efe00640 1 --2- 172.21.15.39:0/2522279421 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55982daf5c30 0x55982daf80c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:05.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.214+0000 7f66f5a00640 1 --2- 172.21.15.39:0/1968206912 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f66d80a2950 0x7f66d80a4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:05.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.214+0000 7f66efe00640 1 -- 172.21.15.39:0/2522279421 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55982da161f0 con 0x55982daf0ac0 2024-03-18T16:19:05.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.214+0000 7f66efe00640 1 --2- 172.21.15.39:0/2522279421 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55982daf0ac0 0x55982daf55c0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f66dc009cf0 tx=0x7f66dc0117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9fb2ad2298416479 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:05.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.215+0000 7f66e2000640 1 -- 172.21.15.39:0/2522279421 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66dc012450 con 0x55982daf0ac0 2024-03-18T16:19:05.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.215+0000 7f66e2000640 1 -- 172.21.15.39:0/2522279421 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f66dc012ad0 con 0x55982daf0ac0 2024-03-18T16:19:05.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.215+0000 7f66e2000640 1 -- 172.21.15.39:0/2522279421 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66dc00fc60 con 0x55982daf0ac0 2024-03-18T16:19:05.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.216+0000 7f66fa7d4740 1 -- 172.21.15.39:0/2522279421 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55982daf0ac0 msgr2=0x55982daf55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:05.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.216+0000 7f66fa7d4740 1 --2- 172.21.15.39:0/2522279421 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55982daf0ac0 0x55982daf55c0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f66dc009cf0 tx=0x7f66dc0117b0 comp rx=0 tx=0).stop 2024-03-18T16:19:05.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.216+0000 7f66f5a00640 1 --2- 172.21.15.39:0/1968206912 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f66d80a2950 0x7f66d80a4e10 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f66e802c640 tx=0x7f66e802c100 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:05.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.216+0000 7f66fa7d4740 1 -- 172.21.15.39:0/2522279421 shutdown_connections 2024-03-18T16:19:05.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.216+0000 7f66fa7d4740 1 --2- 172.21.15.39:0/2522279421 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55982daf5c30 0x55982daf80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:05.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.216+0000 7f66fa7d4740 1 --2- 172.21.15.39:0/2522279421 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55982daf0ac0 0x55982daf55c0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:05.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.216+0000 7f66fa7d4740 1 --2- 172.21.15.39:0/2522279421 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55982daf00d0 0x55982daf04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:05.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.216+0000 7f66fa7d4740 1 -- 172.21.15.39:0/2522279421 >> 172.21.15.39:0/2522279421 conn(0x55982daeb640 msgr2=0x55982daeba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:05.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.216+0000 7f66fa7d4740 1 -- 172.21.15.39:0/2522279421 shutdown_connections 2024-03-18T16:19:05.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.216+0000 7f66fa7d4740 1 -- 172.21.15.39:0/2522279421 wait complete. 2024-03-18T16:19:05.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.217+0000 7f66fa7d4740 1 -- start start 2024-03-18T16:19:05.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.217+0000 7f66fa7d4740 1 Processor -- start 2024-03-18T16:19:05.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.217+0000 7f66fa7d4740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55982daf00d0 0x55982dafa3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:05.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.217+0000 7f66fa7d4740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55982daf0ac0 0x55982dafa910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:05.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.218+0000 7f66fa7d4740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55982daf5c30 0x55982db01950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:05.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.218+0000 7f66fa7d4740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55982da08110 con 0x55982daf00d0 2024-03-18T16:19:05.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.218+0000 7f66fa7d4740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55982da484f0 con 0x55982daf5c30 2024-03-18T16:19:05.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.218+0000 7f66fa7d4740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55982daf8be0 con 0x55982daf0ac0 2024-03-18T16:19:05.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.218+0000 7f66fa7d4740 10 client.0 _ll_register_callbacks cb 0x55982d9f5b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:19:05.222 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.218+0000 7f66efe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55982daf0ac0 0x55982dafa910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:05.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.218+0000 7f66efe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55982daf0ac0 0x55982dafa910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:59580/0 (socket says 172.21.15.39:59580) 2024-03-18T16:19:05.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.218+0000 7f66efe00640 1 -- 172.21.15.39:0/507937926 learned_addr learned my addr 172.21.15.39:0/507937926 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:19:05.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.218+0000 7f66f5000640 1 --2- 172.21.15.39:0/507937926 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55982daf00d0 0x55982dafa3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:05.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.218+0000 7f66f5a00640 1 --2- 172.21.15.39:0/507937926 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55982daf5c30 0x55982db01950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:05.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.218+0000 7f66f5000640 1 -- 172.21.15.39:0/507937926 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55982daf0ac0 msgr2=0x55982dafa910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:05.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.218+0000 7f66f5000640 1 --2- 172.21.15.39:0/507937926 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55982daf0ac0 0x55982dafa910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:05.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.219+0000 7f66f5000640 1 -- 172.21.15.39:0/507937926 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55982daf5c30 msgr2=0x55982db01950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:05.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.219+0000 7f66f5000640 1 --2- 172.21.15.39:0/507937926 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55982daf5c30 0x55982db01950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:05.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.219+0000 7f66f5000640 1 -- 172.21.15.39:0/507937926 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55982d7603f0 con 0x55982daf00d0 2024-03-18T16:19:05.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.219+0000 7f66efe00640 1 --2- 172.21.15.39:0/507937926 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55982daf0ac0 0x55982dafa910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:19:05.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.219+0000 7f66f5000640 1 --2- 172.21.15.39:0/507937926 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55982daf00d0 0x55982dafa3d0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f66e40cc9c0 tx=0x7f66e409c020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82cbbf3411bf32bd server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:05.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.219+0000 7f66d6a00640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-03-18T16:19:05.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.219+0000 7f66d6a00640 1 -- 172.21.15.39:0/507937926 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66e40b9060 con 0x55982daf00d0 2024-03-18T16:19:05.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.219+0000 7f66d6a00640 1 -- 172.21.15.39:0/507937926 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f66e401fdb0 con 0x55982daf00d0 2024-03-18T16:19:05.223 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.219+0000 7f66fa7d4740 1 -- 172.21.15.39:0/507937926 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f66dc012ad0 con 0x55982daf00d0 2024-03-18T16:19:05.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.220+0000 7f66d6a00640 1 -- 172.21.15.39:0/507937926 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f66e40b3030 con 0x55982daf00d0 2024-03-18T16:19:05.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.220+0000 7f66fa7d4740 10 client.16803 fetch_fsmap learned FSMap version 44 2024-03-18T16:19:05.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.220+0000 7f66fa7d4740 1 -- 172.21.15.39:0/507937926 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x55982d82a7d0 con 0x55982daf00d0 2024-03-18T16:19:05.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.220+0000 7f66d6a00640 1 -- 172.21.15.39:0/507937926 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f66dc012ad0 con 0x55982daf00d0 2024-03-18T16:19:05.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.221+0000 7f66d6a00640 1 -- 172.21.15.39:0/507937926 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7f66e409ccb0 con 0x55982daf00d0 2024-03-18T16:19:05.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.221+0000 7f66fa7d4740 10 client.16803 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:19:05.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.221+0000 7f66fa7d4740 10 client.16803 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:19:05.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.221+0000 7f66fa7d4740 20 client.16803 populate_metadata read hostname 'smithi039' 2024-03-18T16:19:05.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.221+0000 7f66fa7d4740 1 --2- 172.21.15.39:0/507937926 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55982db17240 0x55982db196d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:05.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.221+0000 7f66fa7d4740 4 client.16803 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:19:05.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.221+0000 7f66fa7d4740 1 -- 172.21.15.39:0/507937926 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55982da59990 con 0x55982db17240 2024-03-18T16:19:05.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.221+0000 7f66efe00640 1 --2- 172.21.15.39:0/507937926 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55982db17240 0x55982db196d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:05.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.223+0000 7f66efe00640 1 --2- 172.21.15.39:0/507937926 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55982db17240 0x55982db196d0 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=59fb4e979bdde2d6 server_cookie=90a14ecb59f54939 in_seq=0 out_seq=0 2024-03-18T16:19:05.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.223+0000 7f66d6a00640 10 client.16803 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:19:05.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.224+0000 7f66d6a00640 1 -- 172.21.15.39:0/507937926 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55982da59990 con 0x55982db17240 2024-03-18T16:19:05.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.224+0000 7f66d6a00640 10 client.16803 handle_command_reply: tid=0 2024-03-18T16:19:05.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.224+0000 7f66fa7d4740 1 client.16803 shutdown 2024-03-18T16:19:05.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.224+0000 7f66fa7d4740 1 -- 172.21.15.39:0/507937926 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55982daf00d0 msgr2=0x55982dafa3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:05.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.224+0000 7f66fa7d4740 1 --2- 172.21.15.39:0/507937926 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55982daf00d0 0x55982dafa3d0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f66e40cc9c0 tx=0x7f66e409c020 comp rx=0 tx=0).stop 2024-03-18T16:19:05.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.225+0000 7f66fa7d4740 1 -- 172.21.15.39:0/507937926 shutdown_connections 2024-03-18T16:19:05.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.225+0000 7f66fa7d4740 1 --2- 172.21.15.39:0/507937926 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55982db17240 0x55982db196d0 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:05.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.225+0000 7f66fa7d4740 1 --2- 172.21.15.39:0/507937926 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55982daf5c30 0x55982db01950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:05.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.225+0000 7f66fa7d4740 1 --2- 172.21.15.39:0/507937926 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55982daf0ac0 0x55982dafa910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:05.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.225+0000 7f66fa7d4740 1 --2- 172.21.15.39:0/507937926 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55982daf00d0 0x55982dafa3d0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:05.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.225+0000 7f66fa7d4740 1 -- 172.21.15.39:0/507937926 >> 172.21.15.39:0/507937926 conn(0x55982daeb640 msgr2=0x55982daec890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:05.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.225+0000 7f66d6a00640 0 client.16803 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:19:05.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.225+0000 7f66fa7d4740 1 -- 172.21.15.39:0/507937926 shutdown_connections 2024-03-18T16:19:05.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.225+0000 7f66fa7d4740 1 -- 172.21.15.39:0/507937926 wait complete. 2024-03-18T16:19:05.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.225+0000 7f66fa7d4740 20 client.16803 trim_cache size 0 max 16384 2024-03-18T16:19:05.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.247+0000 7f66fa7d4740 1 Processor -- start 2024-03-18T16:19:05.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.247+0000 7f66fa7d4740 1 -- start start 2024-03-18T16:19:05.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.247+0000 7f66fa7d4740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55982daf00d0 0x55982dae21f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:05.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.247+0000 7f66fa7d4740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55982daf0ac0 0x55982db018e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:05.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.247+0000 7f66fa7d4740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55982daf5c30 0x55982db1dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:05.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.247+0000 7f66fa7d4740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f66f00711f0 con 0x55982daf5c30 2024-03-18T16:19:05.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.247+0000 7f66fa7d4740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f66f0072d30 con 0x55982daf00d0 2024-03-18T16:19:05.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.247+0000 7f66fa7d4740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f66f00718c0 con 0x55982daf0ac0 2024-03-18T16:19:05.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.248+0000 7f66f5000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55982daf00d0 0x55982dae21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:05.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.248+0000 7f66f5a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55982daf5c30 0x55982db1dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:05.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.248+0000 7f66f5000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55982daf00d0 0x55982dae21f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:35244/0 (socket says 172.21.15.39:35244) 2024-03-18T16:19:05.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.248+0000 7f66f5000640 1 -- 172.21.15.39:0/923115271 learned_addr learned my addr 172.21.15.39:0/923115271 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:19:05.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.248+0000 7f66efe00640 1 --2- 172.21.15.39:0/923115271 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55982daf0ac0 0x55982db018e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:05.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.248+0000 7f66f5000640 1 -- 172.21.15.39:0/923115271 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55982daf0ac0 msgr2=0x55982db018e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:05.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.248+0000 7f66f5000640 1 --2- 172.21.15.39:0/923115271 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55982daf0ac0 0x55982db018e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:05.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.248+0000 7f66f5000640 1 -- 172.21.15.39:0/923115271 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55982daf5c30 msgr2=0x55982db1dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:05.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.248+0000 7f66f5000640 1 --2- 172.21.15.39:0/923115271 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55982daf5c30 0x55982db1dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:05.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.248+0000 7f66f5000640 1 -- 172.21.15.39:0/923115271 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55982dadec10 con 0x55982daf00d0 2024-03-18T16:19:05.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.248+0000 7f66f5000640 1 --2- 172.21.15.39:0/923115271 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55982daf00d0 0x55982dae21f0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f66e40058c0 tx=0x7f66e4004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d9f33945f6a0d049 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:05.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.249+0000 7f66e1600640 1 -- 172.21.15.39:0/923115271 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66e40ad080 con 0x55982daf00d0 2024-03-18T16:19:05.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.249+0000 7f66e1600640 1 -- 172.21.15.39:0/923115271 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f66e40a4000 con 0x55982daf00d0 2024-03-18T16:19:05.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.249+0000 7f66e1600640 1 -- 172.21.15.39:0/923115271 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66e409cb90 con 0x55982daf00d0 2024-03-18T16:19:05.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.250+0000 7f66fa7d4740 1 -- 172.21.15.39:0/923115271 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55982daf00d0 msgr2=0x55982dae21f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:05.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.250+0000 7f66fa7d4740 1 --2- 172.21.15.39:0/923115271 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55982daf00d0 0x55982dae21f0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f66e40058c0 tx=0x7f66e4004730 comp rx=0 tx=0).stop 2024-03-18T16:19:05.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.250+0000 7f66fa7d4740 1 -- 172.21.15.39:0/923115271 shutdown_connections 2024-03-18T16:19:05.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.250+0000 7f66fa7d4740 1 --2- 172.21.15.39:0/923115271 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55982daf5c30 0x55982db1dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:05.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.250+0000 7f66fa7d4740 1 --2- 172.21.15.39:0/923115271 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55982daf0ac0 0x55982db018e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:05.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.250+0000 7f66fa7d4740 1 --2- 172.21.15.39:0/923115271 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55982daf00d0 0x55982dae21f0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:05.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.250+0000 7f66fa7d4740 1 -- 172.21.15.39:0/923115271 >> 172.21.15.39:0/923115271 conn(0x55982daeb640 msgr2=0x55982dafa650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:05.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.250+0000 7f66fa7d4740 1 -- 172.21.15.39:0/923115271 shutdown_connections 2024-03-18T16:19:05.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.250+0000 7f66fa7d4740 1 -- 172.21.15.39:0/923115271 wait complete. 2024-03-18T16:19:05.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.250+0000 7f66fa7d4740 1 -- start start 2024-03-18T16:19:05.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.251+0000 7f66fa7d4740 1 Processor -- start 2024-03-18T16:19:05.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.251+0000 7f66fa7d4740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55982daf00d0 0x55982daff460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:05.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.251+0000 7f66fa7d4740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55982daf0ac0 0x55982daff9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:05.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.251+0000 7f66fa7d4740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55982daf5c30 0x55982db00960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:05.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.251+0000 7f66fa7d4740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f66f0072660 con 0x55982daf00d0 2024-03-18T16:19:05.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.251+0000 7f66fa7d4740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f66f0071f90 con 0x55982daf5c30 2024-03-18T16:19:05.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.251+0000 7f66fa7d4740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f66f006f1d0 con 0x55982daf0ac0 2024-03-18T16:19:05.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.251+0000 7f66fa7d4740 10 client.0 _ll_register_callbacks cb 0x7f66e40097b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:19:05.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.251+0000 7f66f5000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55982daf00d0 0x55982daff460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:05.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.251+0000 7f66efe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55982daf0ac0 0x55982daff9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:05.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.251+0000 7f66f5a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55982daf5c30 0x55982db00960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:05.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.252+0000 7f66efe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55982daf0ac0 0x55982daff9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:59588/0 (socket says 172.21.15.39:59588) 2024-03-18T16:19:05.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.252+0000 7f66efe00640 1 -- 172.21.15.39:0/2172190647 learned_addr learned my addr 172.21.15.39:0/2172190647 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:19:05.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.252+0000 7f66efe00640 1 -- 172.21.15.39:0/2172190647 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55982daf5c30 msgr2=0x55982db00960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:05.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.252+0000 7f66efe00640 1 --2- 172.21.15.39:0/2172190647 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55982daf5c30 0x55982db00960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:05.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.252+0000 7f66efe00640 1 -- 172.21.15.39:0/2172190647 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55982daf00d0 msgr2=0x55982daff460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:05.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.252+0000 7f66efe00640 1 --2- 172.21.15.39:0/2172190647 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55982daf00d0 0x55982daff460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:05.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.252+0000 7f66efe00640 1 -- 172.21.15.39:0/2172190647 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55982db00f50 con 0x55982daf0ac0 2024-03-18T16:19:05.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.252+0000 7f66efe00640 1 --2- 172.21.15.39:0/2172190647 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55982daf0ac0 0x55982daff9a0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f66dc014010 tx=0x7f66dc009740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2bf4ff00134f6663 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:05.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.252+0000 7f66d6a00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:19:05.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.252+0000 7f66d6a00640 1 -- 172.21.15.39:0/2172190647 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66dc005410 con 0x55982daf0ac0 2024-03-18T16:19:05.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.253+0000 7f66fa7d4740 1 -- 172.21.15.39:0/2172190647 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f66e40a4000 con 0x55982daf0ac0 2024-03-18T16:19:05.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.253+0000 7f66d6a00640 1 -- 172.21.15.39:0/2172190647 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f66dc0055b0 con 0x55982daf0ac0 2024-03-18T16:19:05.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.253+0000 7f66d6a00640 1 -- 172.21.15.39:0/2172190647 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f66dc039050 con 0x55982daf0ac0 2024-03-18T16:19:05.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.253+0000 7f66fa7d4740 10 client.27421 fetch_fsmap learned FSMap version 44 2024-03-18T16:19:05.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.253+0000 7f66fa7d4740 1 -- 172.21.15.39:0/2172190647 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x55982db7d9d0 con 0x55982daf0ac0 2024-03-18T16:19:05.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.253+0000 7f66d6a00640 1 -- 172.21.15.39:0/2172190647 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f66e40a4000 con 0x55982daf0ac0 2024-03-18T16:19:05.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.253+0000 7f66d6a00640 1 -- 172.21.15.39:0/2172190647 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7f66dc029640 con 0x55982daf0ac0 2024-03-18T16:19:05.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.253+0000 7f66fa7d4740 10 client.27421 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:19:05.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.253+0000 7f66fa7d4740 10 client.27421 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:19:05.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.254+0000 7f66fa7d4740 20 client.27421 populate_metadata read hostname 'smithi039' 2024-03-18T16:19:05.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.254+0000 7f66fa7d4740 1 --2- 172.21.15.39:0/2172190647 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55982db7e040 0x55982db80430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:05.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.254+0000 7f66fa7d4740 4 client.27421 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:19:05.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.254+0000 7f66fa7d4740 1 -- 172.21.15.39:0/2172190647 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55982da53ce0 con 0x55982db7e040 2024-03-18T16:19:05.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.254+0000 7f66f5000640 1 --2- 172.21.15.39:0/2172190647 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55982db7e040 0x55982db80430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:05.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.256+0000 7f66f5000640 1 --2- 172.21.15.39:0/2172190647 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55982db7e040 0x55982db80430 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b7d0cdc2fba90d1b server_cookie=5cf0667ce1f0fd10 in_seq=0 out_seq=0 2024-03-18T16:19:05.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.256+0000 7f66d6a00640 10 client.27421 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:19:05.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.256+0000 7f66d6a00640 1 -- 172.21.15.39:0/2172190647 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55982da53ce0 con 0x55982db7e040 2024-03-18T16:19:05.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.256+0000 7f66d6a00640 10 client.27421 handle_command_reply: tid=0 2024-03-18T16:19:05.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.256+0000 7f66fa7d4740 1 client.27421 shutdown 2024-03-18T16:19:05.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.257+0000 7f66fa7d4740 1 -- 172.21.15.39:0/2172190647 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55982daf0ac0 msgr2=0x55982daff9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:05.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.257+0000 7f66fa7d4740 1 --2- 172.21.15.39:0/2172190647 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55982daf0ac0 0x55982daff9a0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f66dc014010 tx=0x7f66dc009740 comp rx=0 tx=0).stop 2024-03-18T16:19:05.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.257+0000 7f66fa7d4740 1 -- 172.21.15.39:0/2172190647 shutdown_connections 2024-03-18T16:19:05.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.257+0000 7f66fa7d4740 1 --2- 172.21.15.39:0/2172190647 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55982db7e040 0x55982db80430 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:05.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.257+0000 7f66fa7d4740 1 --2- 172.21.15.39:0/2172190647 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55982daf5c30 0x55982db00960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:05.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.257+0000 7f66fa7d4740 1 --2- 172.21.15.39:0/2172190647 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55982daf0ac0 0x55982daff9a0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:05.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.257+0000 7f66d6a00640 0 client.27421 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:19:05.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.257+0000 7f66fa7d4740 1 --2- 172.21.15.39:0/2172190647 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55982daf00d0 0x55982daff460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:05.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.257+0000 7f66fa7d4740 1 -- 172.21.15.39:0/2172190647 >> 172.21.15.39:0/2172190647 conn(0x55982daeb640 msgr2=0x55982daee8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:05.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.257+0000 7f66fa7d4740 1 -- 172.21.15.39:0/2172190647 shutdown_connections 2024-03-18T16:19:05.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.257+0000 7f66fa7d4740 1 -- 172.21.15.39:0/2172190647 wait complete. 2024-03-18T16:19:05.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.257+0000 7f66fa7d4740 20 client.27421 trim_cache size 0 max 16384 2024-03-18T16:19:05.261 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:19:05.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.258+0000 7f66f7a00640 1 -- 172.21.15.39:0/1968206912 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f66d80a2950 msgr2=0x7f66d80a4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:05.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.258+0000 7f66f7a00640 1 --2- 172.21.15.39:0/1968206912 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f66d80a2950 0x7f66d80a4e10 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f66e802c640 tx=0x7f66e802c100 comp rx=0 tx=0).stop 2024-03-18T16:19:05.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.258+0000 7f66f7a00640 1 -- 172.21.15.39:0/1968206912 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f66f0103a00 msgr2=0x7f66f0199c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:05.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.258+0000 7f66f7a00640 1 --2- 172.21.15.39:0/1968206912 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f66f0103a00 0x7f66f0199c70 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f66e400e9c0 tx=0x7f66e400ee80 comp rx=0 tx=0).stop 2024-03-18T16:19:05.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.258+0000 7f66f7a00640 1 -- 172.21.15.39:0/1968206912 shutdown_connections 2024-03-18T16:19:05.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.258+0000 7f66f7a00640 1 --2- 172.21.15.39:0/1968206912 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f66d80a2950 0x7f66d80a4e10 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:05.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.258+0000 7f66f7a00640 1 --2- 172.21.15.39:0/1968206912 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f66f01055b0 0x7f66f01a1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:05.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.258+0000 7f66f7a00640 1 --2- 172.21.15.39:0/1968206912 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f66f0104bf0 0x7f66f019a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:05.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.258+0000 7f66f7a00640 1 --2- 172.21.15.39:0/1968206912 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f66f0103a00 0x7f66f0199c70 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:05.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.258+0000 7f66f7a00640 1 -- 172.21.15.39:0/1968206912 >> 172.21.15.39:0/1968206912 conn(0x7f66f00fef90 msgr2=0x7f66f0107e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:05.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.258+0000 7f66f7a00640 1 -- 172.21.15.39:0/1968206912 shutdown_connections 2024-03-18T16:19:05.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:05.258+0000 7f66f7a00640 1 -- 172.21.15.39:0/1968206912 wait complete. 2024-03-18T16:19:05.280 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:05 smithi039 ceph-mon[31721]: pgmap v692: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 725 KiB/s wr, 190 op/s 2024-03-18T16:19:05.379 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:05 smithi196 ceph-mon[36307]: pgmap v692: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 725 KiB/s wr, 190 op/s 2024-03-18T16:19:05.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:05 smithi194 ceph-mon[36200]: pgmap v692: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 725 KiB/s wr, 190 op/s 2024-03-18T16:19:06.283 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-18T16:19:06.284 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T16:19:06.284 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-18T16:19:06.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.536+0000 7f768be00640 1 -- 172.21.15.39:0/1439314204 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f768c109570 msgr2=0x7f768c10ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:06.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.536+0000 7f768be00640 1 --2- 172.21.15.39:0/1439314204 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f768c109570 0x7f768c10ba10 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f7684009ca0 tx=0x7f768402c180 comp rx=0 tx=0).stop 2024-03-18T16:19:06.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.536+0000 7f768be00640 1 -- 172.21.15.39:0/1439314204 shutdown_connections 2024-03-18T16:19:06.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.536+0000 7f768be00640 1 --2- 172.21.15.39:0/1439314204 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f768c109570 0x7f768c10ba10 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:06.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.536+0000 7f768be00640 1 --2- 172.21.15.39:0/1439314204 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f768c0741e0 0x7f768c074640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:06.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.536+0000 7f768be00640 1 --2- 172.21.15.39:0/1439314204 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f768c0757c0 0x7f768c073c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:06.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.536+0000 7f768be00640 1 -- 172.21.15.39:0/1439314204 >> 172.21.15.39:0/1439314204 conn(0x7f768c0fcc50 msgr2=0x7f768c0ff090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:06.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.537+0000 7f768be00640 1 -- 172.21.15.39:0/1439314204 shutdown_connections 2024-03-18T16:19:06.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.537+0000 7f768be00640 1 -- 172.21.15.39:0/1439314204 wait complete. 2024-03-18T16:19:06.540 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.538+0000 7f768be00640 1 Processor -- start 2024-03-18T16:19:06.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.538+0000 7f768be00640 1 -- start start 2024-03-18T16:19:06.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.538+0000 7f768be00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f768c0741e0 0x7f768c195800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:06.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.538+0000 7f768be00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f768c0757c0 0x7f768c195d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:06.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.538+0000 7f768be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f768c109570 0x7f768c19cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:06.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.538+0000 7f768be00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f768c10f560 con 0x7f768c109570 2024-03-18T16:19:06.541 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.538+0000 7f768be00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f768c10f3e0 con 0x7f768c0757c0 2024-03-18T16:19:06.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.538+0000 7f768be00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f768c10f6e0 con 0x7f768c0741e0 2024-03-18T16:19:06.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.539+0000 7f768aa00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f768c0741e0 0x7f768c195800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:06.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.539+0000 7f7683e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f768c0757c0 0x7f768c195d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:06.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.539+0000 7f768aa00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f768c0741e0 0x7f768c195800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:59606/0 (socket says 172.21.15.39:59606) 2024-03-18T16:19:06.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.539+0000 7f768aa00640 1 -- 172.21.15.39:0/1363380782 learned_addr learned my addr 172.21.15.39:0/1363380782 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:19:06.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.539+0000 7f768b400640 1 --2- 172.21.15.39:0/1363380782 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f768c109570 0x7f768c19cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:06.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.539+0000 7f7683e00640 1 -- 172.21.15.39:0/1363380782 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f768c0741e0 msgr2=0x7f768c195800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:06.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.539+0000 7f7683e00640 1 --2- 172.21.15.39:0/1363380782 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f768c0741e0 0x7f768c195800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:06.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.539+0000 7f7683e00640 1 -- 172.21.15.39:0/1363380782 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f768c109570 msgr2=0x7f768c19cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:06.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.539+0000 7f7683e00640 1 --2- 172.21.15.39:0/1363380782 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f768c109570 0x7f768c19cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:06.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.539+0000 7f7683e00640 1 -- 172.21.15.39:0/1363380782 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f768c1ae460 con 0x7f768c0757c0 2024-03-18T16:19:06.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.539+0000 7f768b400640 1 --2- 172.21.15.39:0/1363380782 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f768c109570 0x7f768c19cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:19:06.542 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.540+0000 7f7683e00640 1 --2- 172.21.15.39:0/1363380782 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f768c0757c0 0x7f768c195d40 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f767400e9f0 tx=0x7f767400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:06.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.540+0000 7f7683400640 1 -- 172.21.15.39:0/1363380782 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f767400ccc0 con 0x7f768c0757c0 2024-03-18T16:19:06.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.540+0000 7f7683400640 1 -- 172.21.15.39:0/1363380782 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7674010910 con 0x7f768c0757c0 2024-03-18T16:19:06.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.540+0000 7f7683400640 1 -- 172.21.15.39:0/1363380782 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7674018960 con 0x7f768c0757c0 2024-03-18T16:19:06.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.540+0000 7f768be00640 1 -- 172.21.15.39:0/1363380782 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f768c1ae6f0 con 0x7f768c0757c0 2024-03-18T16:19:06.543 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.541+0000 7f768be00640 1 -- 172.21.15.39:0/1363380782 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f768c1aebf0 con 0x7f768c0757c0 2024-03-18T16:19:06.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.545+0000 7f7694a05740 1 Processor -- start 2024-03-18T16:19:06.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.545+0000 7f7694a05740 1 -- start start 2024-03-18T16:19:06.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.546+0000 7f7694a05740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555c11286980 0x555c11286da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:06.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.546+0000 7f7694a05740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555c11287370 0x555c1128be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:06.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.546+0000 7f7694a05740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555c1128c4e0 0x555c1128e970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:06.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.546+0000 7f7694a05740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x555c111e9f80 con 0x555c1128c4e0 2024-03-18T16:19:06.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.546+0000 7f7694a05740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x555c11192ad0 con 0x555c11286980 2024-03-18T16:19:06.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.546+0000 7f7694a05740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x555c111e6020 con 0x555c11287370 2024-03-18T16:19:06.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.546+0000 7f768b400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555c11286980 0x555c11286da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:06.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.546+0000 7f768aa00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555c11287370 0x555c1128be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:06.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.546+0000 7f768aa00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555c11287370 0x555c1128be70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:59608/0 (socket says 172.21.15.39:59608) 2024-03-18T16:19:06.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.546+0000 7f768aa00640 1 -- 172.21.15.39:0/2627511096 learned_addr learned my addr 172.21.15.39:0/2627511096 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:19:06.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.547+0000 7f768aa00640 1 -- 172.21.15.39:0/2627511096 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555c11286980 msgr2=0x555c11286da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:06.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.547+0000 7f768aa00640 1 --2- 172.21.15.39:0/2627511096 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555c11286980 0x555c11286da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:06.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.547+0000 7f768aa00640 1 -- 172.21.15.39:0/2627511096 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555c1128c4e0 msgr2=0x555c1128e970 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:19:06.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.547+0000 7f768aa00640 1 --2- 172.21.15.39:0/2627511096 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555c1128c4e0 0x555c1128e970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:06.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.547+0000 7f768aa00640 1 -- 172.21.15.39:0/2627511096 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555c111aa6f0 con 0x555c11287370 2024-03-18T16:19:06.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.547+0000 7f7683400640 1 -- 172.21.15.39:0/1363380782 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f767401f020 con 0x7f768c0757c0 2024-03-18T16:19:06.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.547+0000 7f768aa00640 1 --2- 172.21.15.39:0/2627511096 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555c11287370 0x555c1128be70 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f767c009d60 tx=0x7f767c0117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ea441beb91fd5702 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:06.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.548+0000 7f767be00640 1 -- 172.21.15.39:0/2627511096 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f767c012650 con 0x555c11287370 2024-03-18T16:19:06.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.548+0000 7f767be00640 1 -- 172.21.15.39:0/2627511096 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f767c009e50 con 0x555c11287370 2024-03-18T16:19:06.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.548+0000 7f767be00640 1 -- 172.21.15.39:0/2627511096 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f767c00fcd0 con 0x555c11287370 2024-03-18T16:19:06.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.548+0000 7f7694a05740 1 -- 172.21.15.39:0/2627511096 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555c11287370 msgr2=0x555c1128be70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:06.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.548+0000 7f7694a05740 1 --2- 172.21.15.39:0/2627511096 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555c11287370 0x555c1128be70 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f767c009d60 tx=0x7f767c0117b0 comp rx=0 tx=0).stop 2024-03-18T16:19:06.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.549+0000 7f7694a05740 1 -- 172.21.15.39:0/2627511096 shutdown_connections 2024-03-18T16:19:06.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.549+0000 7f7694a05740 1 --2- 172.21.15.39:0/2627511096 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555c1128c4e0 0x555c1128e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:06.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.549+0000 7f7694a05740 1 --2- 172.21.15.39:0/2627511096 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555c11287370 0x555c1128be70 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:06.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.549+0000 7f7694a05740 1 --2- 172.21.15.39:0/2627511096 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555c11286980 0x555c11286da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:06.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.549+0000 7f7694a05740 1 -- 172.21.15.39:0/2627511096 >> 172.21.15.39:0/2627511096 conn(0x555c1124e930 msgr2=0x555c112822f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:06.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.549+0000 7f7694a05740 1 -- 172.21.15.39:0/2627511096 shutdown_connections 2024-03-18T16:19:06.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.549+0000 7f7694a05740 1 -- 172.21.15.39:0/2627511096 wait complete. 2024-03-18T16:19:06.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.549+0000 7f7683400640 1 --2- 172.21.15.39:0/1363380782 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f76700a2860 0x7f76700a4d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:06.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.549+0000 7f7694a05740 1 -- start start 2024-03-18T16:19:06.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.550+0000 7f7683400640 1 -- 172.21.15.39:0/1363380782 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f76740cf0f0 con 0x7f768c0757c0 2024-03-18T16:19:06.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.550+0000 7f7694a05740 1 Processor -- start 2024-03-18T16:19:06.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.550+0000 7f768b400640 1 --2- 172.21.15.39:0/1363380782 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f76700a2860 0x7f76700a4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:06.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.551+0000 7f7694a05740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555c11286980 0x555c11290ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:06.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.551+0000 7f7694a05740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555c11287370 0x555c11291010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:06.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.551+0000 7f7694a05740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555c1128c4e0 0x555c11298050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:06.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.551+0000 7f7694a05740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x555c1128f490 con 0x555c11286980 2024-03-18T16:19:06.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.551+0000 7f7694a05740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x555c111a5f30 con 0x555c11287370 2024-03-18T16:19:06.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.551+0000 7f7694a05740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x555c1128f610 con 0x555c1128c4e0 2024-03-18T16:19:06.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.551+0000 7f7694a05740 10 client.0 _ll_register_callbacks cb 0x555c11193bc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:19:06.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.551+0000 7f768aa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555c11286980 0x555c11290ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:06.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.552+0000 7f768aa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555c11286980 0x555c11290ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:49250/0 (socket says 172.21.15.39:49250) 2024-03-18T16:19:06.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.552+0000 7f768aa00640 1 -- 172.21.15.39:0/1616828063 learned_addr learned my addr 172.21.15.39:0/1616828063 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:19:06.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.552+0000 7f768b400640 1 --2- 172.21.15.39:0/1363380782 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f76700a2860 0x7f76700a4d20 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f768402c6c0 tx=0x7f7684032340 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:06.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.552+0000 7f768b400640 1 --2- 172.21.15.39:0/1616828063 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555c1128c4e0 0x555c11298050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:06.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.552+0000 7f7683e00640 1 --2- 172.21.15.39:0/1616828063 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555c11287370 0x555c11291010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:06.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.552+0000 7f768aa00640 1 -- 172.21.15.39:0/1616828063 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555c1128c4e0 msgr2=0x555c11298050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:06.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.552+0000 7f768aa00640 1 --2- 172.21.15.39:0/1616828063 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555c1128c4e0 0x555c11298050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:06.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.552+0000 7f768aa00640 1 -- 172.21.15.39:0/1616828063 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555c11287370 msgr2=0x555c11291010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:06.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.552+0000 7f768aa00640 1 --2- 172.21.15.39:0/1616828063 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555c11287370 0x555c11291010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:06.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.552+0000 7f768aa00640 1 -- 172.21.15.39:0/1616828063 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555c10fc67e0 con 0x555c11286980 2024-03-18T16:19:06.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.553+0000 7f768aa00640 1 --2- 172.21.15.39:0/1616828063 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555c11286980 0x555c11290ad0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f767c011cb0 tx=0x7f767c005270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bf27ffeeb993d80 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:06.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.553+0000 7f7678c00640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-03-18T16:19:06.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.553+0000 7f7678c00640 1 -- 172.21.15.39:0/1616828063 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f767c028020 con 0x555c11286980 2024-03-18T16:19:06.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.553+0000 7f7678c00640 1 -- 172.21.15.39:0/1616828063 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f767c0046c0 con 0x555c11286980 2024-03-18T16:19:06.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.553+0000 7f7694a05740 1 -- 172.21.15.39:0/1616828063 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f767c009e50 con 0x555c11286980 2024-03-18T16:19:06.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.554+0000 7f7678c00640 1 -- 172.21.15.39:0/1616828063 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f767c033050 con 0x555c11286980 2024-03-18T16:19:06.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.554+0000 7f7694a05740 10 client.16818 fetch_fsmap learned FSMap version 44 2024-03-18T16:19:06.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.554+0000 7f7694a05740 1 -- 172.21.15.39:0/1616828063 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x555c11026f70 con 0x555c11286980 2024-03-18T16:19:06.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.554+0000 7f7678c00640 1 -- 172.21.15.39:0/1616828063 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f767c009e50 con 0x555c11286980 2024-03-18T16:19:06.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.555+0000 7f7678c00640 1 -- 172.21.15.39:0/1616828063 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7f767c0048c0 con 0x555c11286980 2024-03-18T16:19:06.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.556+0000 7f7694a05740 10 client.16818 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:19:06.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.556+0000 7f7694a05740 10 client.16818 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:19:06.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.556+0000 7f7694a05740 20 client.16818 populate_metadata read hostname 'smithi039' 2024-03-18T16:19:06.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.556+0000 7f7694a05740 1 --2- 172.21.15.39:0/1616828063 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x555c1129c650 0x555c1129ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:06.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.556+0000 7f7694a05740 4 client.16818 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:19:06.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.556+0000 7f7694a05740 1 -- 172.21.15.39:0/1616828063 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555c111f74c0 con 0x555c1129c650 2024-03-18T16:19:06.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.556+0000 7f7683e00640 1 --2- 172.21.15.39:0/1616828063 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x555c1129c650 0x555c1129ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:06.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.557+0000 7f7683e00640 1 --2- 172.21.15.39:0/1616828063 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x555c1129c650 0x555c1129ea40 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a7a4aad1b927698 server_cookie=b7a6dfb47ab8d13f in_seq=0 out_seq=0 2024-03-18T16:19:06.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.557+0000 7f7678c00640 10 client.16818 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:19:06.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.557+0000 7f7678c00640 1 -- 172.21.15.39:0/1616828063 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x555c111f74c0 con 0x555c1129c650 2024-03-18T16:19:06.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.557+0000 7f7678c00640 10 client.16818 handle_command_reply: tid=0 2024-03-18T16:19:06.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.557+0000 7f7694a05740 1 client.16818 shutdown 2024-03-18T16:19:06.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.558+0000 7f7694a05740 1 -- 172.21.15.39:0/1616828063 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555c11286980 msgr2=0x555c11290ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:06.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.558+0000 7f7694a05740 1 --2- 172.21.15.39:0/1616828063 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555c11286980 0x555c11290ad0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f767c011cb0 tx=0x7f767c005270 comp rx=0 tx=0).stop 2024-03-18T16:19:06.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.558+0000 7f7694a05740 1 -- 172.21.15.39:0/1616828063 shutdown_connections 2024-03-18T16:19:06.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.558+0000 7f7694a05740 1 --2- 172.21.15.39:0/1616828063 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x555c1129c650 0x555c1129ea40 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:06.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.558+0000 7f7694a05740 1 --2- 172.21.15.39:0/1616828063 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555c11287370 0x555c11291010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:06.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.558+0000 7f7694a05740 1 --2- 172.21.15.39:0/1616828063 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555c1128c4e0 0x555c11298050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:06.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.558+0000 7f7678c00640 0 client.16818 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:19:06.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.558+0000 7f7694a05740 1 --2- 172.21.15.39:0/1616828063 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555c11286980 0x555c11290ad0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:06.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.558+0000 7f7694a05740 1 -- 172.21.15.39:0/1616828063 >> 172.21.15.39:0/1616828063 conn(0x555c1124e930 msgr2=0x555c11281570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:06.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.558+0000 7f7694a05740 1 -- 172.21.15.39:0/1616828063 shutdown_connections 2024-03-18T16:19:06.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.558+0000 7f7694a05740 1 -- 172.21.15.39:0/1616828063 wait complete. 2024-03-18T16:19:06.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.558+0000 7f7694a05740 20 client.16818 trim_cache size 0 max 16384 2024-03-18T16:19:06.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.585+0000 7f7694a05740 1 Processor -- start 2024-03-18T16:19:06.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.585+0000 7f7694a05740 1 -- start start 2024-03-18T16:19:06.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.586+0000 7f7694a05740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555c11286980 0x555c112900f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:06.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.586+0000 7f7694a05740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555c11287370 0x555c11298010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:06.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.586+0000 7f7694a05740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555c1128c4e0 0x555c11290630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:06.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.586+0000 7f7694a05740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f768c1b9030 con 0x555c11287370 2024-03-18T16:19:06.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.586+0000 7f7694a05740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f768c1bab70 con 0x555c1128c4e0 2024-03-18T16:19:06.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.586+0000 7f7694a05740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f768c1b9700 con 0x555c11286980 2024-03-18T16:19:06.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.586+0000 7f768b400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555c1128c4e0 0x555c11290630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:06.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.586+0000 7f768aa00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555c11286980 0x555c112900f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:06.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.586+0000 7f768aa00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555c11286980 0x555c112900f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:59616/0 (socket says 172.21.15.39:59616) 2024-03-18T16:19:06.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.586+0000 7f768b400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555c1128c4e0 0x555c11290630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:35294/0 (socket says 172.21.15.39:35294) 2024-03-18T16:19:06.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.586+0000 7f768aa00640 1 -- 172.21.15.39:0/451552492 learned_addr learned my addr 172.21.15.39:0/451552492 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:19:06.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.586+0000 7f7683e00640 1 --2- 172.21.15.39:0/451552492 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555c11287370 0x555c11298010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:06.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.587+0000 7f768aa00640 1 -- 172.21.15.39:0/451552492 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555c1128c4e0 msgr2=0x555c11290630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:06.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.587+0000 7f768aa00640 1 --2- 172.21.15.39:0/451552492 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555c1128c4e0 0x555c11290630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:06.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.587+0000 7f768aa00640 1 -- 172.21.15.39:0/451552492 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555c11287370 msgr2=0x555c11298010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:06.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.587+0000 7f768aa00640 1 --2- 172.21.15.39:0/451552492 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555c11287370 0x555c11298010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:06.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.587+0000 7f768aa00640 1 -- 172.21.15.39:0/451552492 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555c10ebe170 con 0x555c11286980 2024-03-18T16:19:06.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.587+0000 7f768aa00640 1 --2- 172.21.15.39:0/451552492 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555c11286980 0x555c112900f0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f767c0092a0 tx=0x7f767c005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e39441cda68ae97a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:06.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.587+0000 7f767b400640 1 -- 172.21.15.39:0/451552492 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f767c03c070 con 0x555c11286980 2024-03-18T16:19:06.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.587+0000 7f767b400640 1 -- 172.21.15.39:0/451552492 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f767c00f840 con 0x555c11286980 2024-03-18T16:19:06.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.587+0000 7f767b400640 1 -- 172.21.15.39:0/451552492 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f767c028040 con 0x555c11286980 2024-03-18T16:19:06.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.588+0000 7f7694a05740 1 -- 172.21.15.39:0/451552492 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555c11286980 msgr2=0x555c112900f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:06.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.588+0000 7f7694a05740 1 --2- 172.21.15.39:0/451552492 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555c11286980 0x555c112900f0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f767c0092a0 tx=0x7f767c005f90 comp rx=0 tx=0).stop 2024-03-18T16:19:06.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.589+0000 7f7694a05740 1 -- 172.21.15.39:0/451552492 shutdown_connections 2024-03-18T16:19:06.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.589+0000 7f7694a05740 1 --2- 172.21.15.39:0/451552492 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555c1128c4e0 0x555c11290630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:06.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.589+0000 7f7694a05740 1 --2- 172.21.15.39:0/451552492 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555c11287370 0x555c11298010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:06.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.589+0000 7f7694a05740 1 --2- 172.21.15.39:0/451552492 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555c11286980 0x555c112900f0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:06.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.589+0000 7f7694a05740 1 -- 172.21.15.39:0/451552492 >> 172.21.15.39:0/451552492 conn(0x555c1124e930 msgr2=0x555c11281570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:06.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.589+0000 7f7694a05740 1 -- 172.21.15.39:0/451552492 shutdown_connections 2024-03-18T16:19:06.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.589+0000 7f7694a05740 1 -- 172.21.15.39:0/451552492 wait complete. 2024-03-18T16:19:06.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.589+0000 7f7694a05740 1 -- start start 2024-03-18T16:19:06.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.589+0000 7f7694a05740 1 Processor -- start 2024-03-18T16:19:06.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.590+0000 7f7694a05740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555c11286980 0x555c11295570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:06.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.590+0000 7f7694a05740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555c11287370 0x555c11295ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:06.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.590+0000 7f7694a05740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555c1128c4e0 0x555c11296ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:06.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.590+0000 7f7694a05740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f768c1ba4a0 con 0x555c11287370 2024-03-18T16:19:06.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.590+0000 7f7694a05740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f768c1b9dd0 con 0x555c11286980 2024-03-18T16:19:06.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.590+0000 7f7694a05740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f768c1b7010 con 0x555c1128c4e0 2024-03-18T16:19:06.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.590+0000 7f7694a05740 10 client.0 _ll_register_callbacks cb 0x7f7684037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:19:06.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.590+0000 7f768aa00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555c11286980 0x555c11295570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:06.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.590+0000 7f768aa00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555c11286980 0x555c11295570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:35302/0 (socket says 172.21.15.39:35302) 2024-03-18T16:19:06.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.590+0000 7f768aa00640 1 -- 172.21.15.39:0/966420999 learned_addr learned my addr 172.21.15.39:0/966420999 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:19:06.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.590+0000 7f7683e00640 1 --2- 172.21.15.39:0/966420999 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555c11287370 0x555c11295ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:06.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.590+0000 7f768b400640 1 --2- 172.21.15.39:0/966420999 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555c1128c4e0 0x555c11296ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:06.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.590+0000 7f768aa00640 1 -- 172.21.15.39:0/966420999 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555c1128c4e0 msgr2=0x555c11296ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:06.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.590+0000 7f768aa00640 1 --2- 172.21.15.39:0/966420999 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555c1128c4e0 0x555c11296ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:06.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.590+0000 7f768aa00640 1 -- 172.21.15.39:0/966420999 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555c11287370 msgr2=0x555c11295ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:06.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.591+0000 7f768aa00640 1 --2- 172.21.15.39:0/966420999 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555c11287370 0x555c11295ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:06.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.591+0000 7f768aa00640 1 -- 172.21.15.39:0/966420999 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555c112970b0 con 0x555c11286980 2024-03-18T16:19:06.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.591+0000 7f768b400640 1 --2- 172.21.15.39:0/966420999 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555c1128c4e0 0x555c11296ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:19:06.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.591+0000 7f7683e00640 1 --2- 172.21.15.39:0/966420999 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555c11287370 0x555c11295ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:19:06.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.591+0000 7f768aa00640 1 --2- 172.21.15.39:0/966420999 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555c11286980 0x555c11295570 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f767c004d30 tx=0x7f767c0097c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dca7b9269c3ded1a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:06.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.591+0000 7f7678c00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:19:06.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.591+0000 7f7678c00640 1 -- 172.21.15.39:0/966420999 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f767c03c030 con 0x555c11286980 2024-03-18T16:19:06.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.592+0000 7f7694a05740 1 -- 172.21.15.39:0/966420999 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f767c00f840 con 0x555c11286980 2024-03-18T16:19:06.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.592+0000 7f7678c00640 1 -- 172.21.15.39:0/966420999 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f767c004680 con 0x555c11286980 2024-03-18T16:19:06.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.592+0000 7f7678c00640 1 -- 172.21.15.39:0/966420999 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f767c049050 con 0x555c11286980 2024-03-18T16:19:06.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.592+0000 7f7694a05740 10 client.27461 fetch_fsmap learned FSMap version 44 2024-03-18T16:19:06.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.592+0000 7f7694a05740 1 -- 172.21.15.39:0/966420999 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x555c112983f0 con 0x555c11286980 2024-03-18T16:19:06.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.592+0000 7f7678c00640 1 -- 172.21.15.39:0/966420999 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f767c00f840 con 0x555c11286980 2024-03-18T16:19:06.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.592+0000 7f7678c00640 1 -- 172.21.15.39:0/966420999 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7f767c024ba0 con 0x555c11286980 2024-03-18T16:19:06.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.592+0000 7f7694a05740 10 client.27461 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:19:06.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.592+0000 7f7694a05740 10 client.27461 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:19:06.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.592+0000 7f7694a05740 20 client.27461 populate_metadata read hostname 'smithi039' 2024-03-18T16:19:06.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.592+0000 7f7694a05740 1 --2- 172.21.15.39:0/966420999 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x555c11302f70 0x555c11305390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:06.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.592+0000 7f7694a05740 4 client.27461 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-18T16:19:06.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.592+0000 7f7694a05740 1 -- 172.21.15.39:0/966420999 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x555c111f1810 con 0x555c11302f70 2024-03-18T16:19:06.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.593+0000 7f7683e00640 1 --2- 172.21.15.39:0/966420999 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x555c11302f70 0x555c11305390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:06.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.593+0000 7f7683e00640 1 --2- 172.21.15.39:0/966420999 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x555c11302f70 0x555c11305390 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2adac365d53a5cbe server_cookie=726bc9bfbb674f1d in_seq=0 out_seq=0 2024-03-18T16:19:06.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.593+0000 7f7678c00640 10 client.27461 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:19:06.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.594+0000 7f7678c00640 1 -- 172.21.15.39:0/966420999 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x555c111f1810 con 0x555c11302f70 2024-03-18T16:19:06.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.594+0000 7f7678c00640 10 client.27461 handle_command_reply: tid=0 2024-03-18T16:19:06.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.594+0000 7f7694a05740 1 client.27461 shutdown 2024-03-18T16:19:06.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.594+0000 7f7694a05740 1 -- 172.21.15.39:0/966420999 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555c11286980 msgr2=0x555c11295570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:06.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.594+0000 7f7694a05740 1 --2- 172.21.15.39:0/966420999 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555c11286980 0x555c11295570 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f767c004d30 tx=0x7f767c0097c0 comp rx=0 tx=0).stop 2024-03-18T16:19:06.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.595+0000 7f7694a05740 1 -- 172.21.15.39:0/966420999 shutdown_connections 2024-03-18T16:19:06.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.595+0000 7f7694a05740 1 --2- 172.21.15.39:0/966420999 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x555c11302f70 0x555c11305390 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:06.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.595+0000 7f7694a05740 1 --2- 172.21.15.39:0/966420999 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x555c1128c4e0 0x555c11296ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:06.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.595+0000 7f7694a05740 1 --2- 172.21.15.39:0/966420999 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x555c11287370 0x555c11295ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:06.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.595+0000 7f7694a05740 1 --2- 172.21.15.39:0/966420999 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555c11286980 0x555c11295570 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:06.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.595+0000 7f7694a05740 1 -- 172.21.15.39:0/966420999 >> 172.21.15.39:0/966420999 conn(0x555c1124e930 msgr2=0x555c11280ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:06.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.595+0000 7f7678c00640 0 client.27461 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:19:06.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.596+0000 7f7694a05740 1 -- 172.21.15.39:0/966420999 shutdown_connections 2024-03-18T16:19:06.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.596+0000 7f7694a05740 1 -- 172.21.15.39:0/966420999 wait complete. 2024-03-18T16:19:06.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.596+0000 7f7694a05740 20 client.27461 trim_cache size 0 max 16384 2024-03-18T16:19:06.600 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-03-18T16:19:06.600 INFO:teuthology.orchestra.run.smithi039.stdout: "return_code": 0, 2024-03-18T16:19:06.600 INFO:teuthology.orchestra.run.smithi039.stdout: "scrub_tag": "4f10a011-7bc8-4a85-8d00-b669c7b1d918", 2024-03-18T16:19:06.600 INFO:teuthology.orchestra.run.smithi039.stdout: "mode": "asynchronous" 2024-03-18T16:19:06.600 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-03-18T16:19:06.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.598+0000 7f768be00640 1 -- 172.21.15.39:0/1363380782 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f76700a2860 msgr2=0x7f76700a4d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:06.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.598+0000 7f768be00640 1 --2- 172.21.15.39:0/1363380782 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f76700a2860 0x7f76700a4d20 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f768402c6c0 tx=0x7f7684032340 comp rx=0 tx=0).stop 2024-03-18T16:19:06.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.598+0000 7f768be00640 1 -- 172.21.15.39:0/1363380782 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f768c0757c0 msgr2=0x7f768c195d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:06.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.598+0000 7f768be00640 1 --2- 172.21.15.39:0/1363380782 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f768c0757c0 0x7f768c195d40 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f767400e9f0 tx=0x7f767400eeb0 comp rx=0 tx=0).stop 2024-03-18T16:19:06.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.598+0000 7f768be00640 1 -- 172.21.15.39:0/1363380782 shutdown_connections 2024-03-18T16:19:06.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.598+0000 7f768be00640 1 --2- 172.21.15.39:0/1363380782 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f768c0757c0 0x7f768c195d40 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:06.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.598+0000 7f768be00640 1 --2- 172.21.15.39:0/1363380782 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f76700a2860 0x7f76700a4d20 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:06.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.598+0000 7f768be00640 1 --2- 172.21.15.39:0/1363380782 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f768c109570 0x7f768c19cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:06.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.598+0000 7f768be00640 1 --2- 172.21.15.39:0/1363380782 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f768c0741e0 0x7f768c195800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:06.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.598+0000 7f768be00640 1 -- 172.21.15.39:0/1363380782 >> 172.21.15.39:0/1363380782 conn(0x7f768c0fcc50 msgr2=0x7f768c1010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:06.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.598+0000 7f768be00640 1 -- 172.21.15.39:0/1363380782 shutdown_connections 2024-03-18T16:19:06.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.599+0000 7f768be00640 1 -- 172.21.15.39:0/1363380782 wait complete. 2024-03-18T16:19:06.631 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T16:19:06.631 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T16:19:06.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.885+0000 7f406a000640 1 -- 172.21.15.39:0/3386785105 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4064102280 msgr2=0x7f4064104670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:06.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.885+0000 7f406a000640 1 --2- 172.21.15.39:0/3386785105 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4064102280 0x7f4064104670 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f4054009900 tx=0x7f405402bc50 comp rx=0 tx=0).stop 2024-03-18T16:19:06.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.886+0000 7f406a000640 1 -- 172.21.15.39:0/3386785105 shutdown_connections 2024-03-18T16:19:06.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.886+0000 7f406a000640 1 --2- 172.21.15.39:0/3386785105 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f40641074e0 0x7f4064109930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:06.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.886+0000 7f406a000640 1 --2- 172.21.15.39:0/3386785105 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4064104bb0 0x7f4064106fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:06.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.886+0000 7f406a000640 1 --2- 172.21.15.39:0/3386785105 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4064102280 0x7f4064104670 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:06.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.886+0000 7f406a000640 1 -- 172.21.15.39:0/3386785105 >> 172.21.15.39:0/3386785105 conn(0x7f40640fbc90 msgr2=0x7f40640fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:06.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.886+0000 7f406a000640 1 -- 172.21.15.39:0/3386785105 shutdown_connections 2024-03-18T16:19:06.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.886+0000 7f406a000640 1 -- 172.21.15.39:0/3386785105 wait complete. 2024-03-18T16:19:06.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.887+0000 7f406a000640 1 Processor -- start 2024-03-18T16:19:06.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.887+0000 7f406a000640 1 -- start start 2024-03-18T16:19:06.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.887+0000 7f406a000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4064102280 0x7f4064197be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:06.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.887+0000 7f406a000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4064104bb0 0x7f4064198120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:06.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.887+0000 7f406a000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f40641074e0 0x7f406419f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:06.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.887+0000 7f406a000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f406410d4d0 con 0x7f4064102280 2024-03-18T16:19:06.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.887+0000 7f406a000640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f406410d350 con 0x7f40641074e0 2024-03-18T16:19:06.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.887+0000 7f406a000640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f406410d650 con 0x7f4064104bb0 2024-03-18T16:19:06.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.888+0000 7f4063400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4064102280 0x7f4064197be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:06.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.888+0000 7f4062a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4064104bb0 0x7f4064198120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:06.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.888+0000 7f4063400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4064102280 0x7f4064197be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:49284/0 (socket says 172.21.15.39:49284) 2024-03-18T16:19:06.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.888+0000 7f4063400640 1 -- 172.21.15.39:0/1516824854 learned_addr learned my addr 172.21.15.39:0/1516824854 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:19:06.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.888+0000 7f4063e00640 1 --2- 172.21.15.39:0/1516824854 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f40641074e0 0x7f406419f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:06.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.888+0000 7f4063400640 1 -- 172.21.15.39:0/1516824854 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4064104bb0 msgr2=0x7f4064198120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:06.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.888+0000 7f4063400640 1 --2- 172.21.15.39:0/1516824854 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4064104bb0 0x7f4064198120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:06.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.888+0000 7f4063400640 1 -- 172.21.15.39:0/1516824854 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f40641074e0 msgr2=0x7f406419f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:06.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.888+0000 7f4063400640 1 --2- 172.21.15.39:0/1516824854 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f40641074e0 0x7f406419f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:06.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.888+0000 7f4063400640 1 -- 172.21.15.39:0/1516824854 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f406419f760 con 0x7f4064102280 2024-03-18T16:19:06.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.889+0000 7f4063400640 1 --2- 172.21.15.39:0/1516824854 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4064102280 0x7f4064197be0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f405402c260 tx=0x7f405402c670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:06.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.889+0000 7f405be00640 1 -- 172.21.15.39:0/1516824854 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4054041070 con 0x7f4064102280 2024-03-18T16:19:06.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.889+0000 7f405be00640 1 -- 172.21.15.39:0/1516824854 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4054004450 con 0x7f4064102280 2024-03-18T16:19:06.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.889+0000 7f406a000640 1 -- 172.21.15.39:0/1516824854 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f406419f9f0 con 0x7f4064102280 2024-03-18T16:19:06.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.889+0000 7f405be00640 1 -- 172.21.15.39:0/1516824854 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4054034440 con 0x7f4064102280 2024-03-18T16:19:06.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.889+0000 7f406a000640 1 -- 172.21.15.39:0/1516824854 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f406419ff40 con 0x7f4064102280 2024-03-18T16:19:06.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.893+0000 7f406cde3740 1 Processor -- start 2024-03-18T16:19:06.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.894+0000 7f406cde3740 1 -- start start 2024-03-18T16:19:06.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.894+0000 7f406cde3740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564c023b8810 0x564c023b8c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:06.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.894+0000 7f406cde3740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564c023b9200 0x564c023bdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:06.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.894+0000 7f406cde3740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564c023c2780 0x564c023c4c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:06.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.894+0000 7f406cde3740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x564c0231d1f0 con 0x564c023c2780 2024-03-18T16:19:06.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.894+0000 7f406cde3740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x564c022c5a80 con 0x564c023b9200 2024-03-18T16:19:06.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.894+0000 7f406cde3740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x564c02319290 con 0x564c023b8810 2024-03-18T16:19:06.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.894+0000 7f4063e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564c023b8810 0x564c023b8c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:06.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.894+0000 7f4063e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564c023b8810 0x564c023b8c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:59650/0 (socket says 172.21.15.39:59650) 2024-03-18T16:19:06.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.894+0000 7f4063e00640 1 -- 172.21.15.39:0/3785996224 learned_addr learned my addr 172.21.15.39:0/3785996224 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:19:06.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.894+0000 7f405be00640 1 -- 172.21.15.39:0/1516824854 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f4054044070 con 0x7f4064102280 2024-03-18T16:19:06.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.895+0000 7f4062a00640 1 --2- 172.21.15.39:0/3785996224 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564c023c2780 0x564c023c4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:06.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.895+0000 7f4063e00640 1 -- 172.21.15.39:0/3785996224 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564c023b9200 msgr2=0x564c023bdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:06.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.895+0000 7f4063e00640 1 --2- 172.21.15.39:0/3785996224 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564c023b9200 0x564c023bdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:06.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.895+0000 7f4063e00640 1 -- 172.21.15.39:0/3785996224 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564c023c2780 msgr2=0x564c023c4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:06.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.895+0000 7f4063e00640 1 --2- 172.21.15.39:0/3785996224 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564c023c2780 0x564c023c4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:06.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.895+0000 7f4063e00640 1 -- 172.21.15.39:0/3785996224 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564c020fb7d0 con 0x564c023b8810 2024-03-18T16:19:06.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.895+0000 7f4063e00640 1 --2- 172.21.15.39:0/3785996224 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564c023b8810 0x564c023b8c30 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f405c009fd0 tx=0x7f405c0118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c0b367cc81fcfcc4 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:06.898 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.896+0000 7f404fe00640 1 -- 172.21.15.39:0/3785996224 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f405c004d20 con 0x564c023b8810 2024-03-18T16:19:06.898 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.896+0000 7f404fe00640 1 -- 172.21.15.39:0/3785996224 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f405c004050 con 0x564c023b8810 2024-03-18T16:19:06.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.896+0000 7f404fe00640 1 -- 172.21.15.39:0/3785996224 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f405c005240 con 0x564c023b8810 2024-03-18T16:19:06.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.897+0000 7f405be00640 1 --2- 172.21.15.39:0/1516824854 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f40480a2800 0x7f40480a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:06.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.897+0000 7f406cde3740 1 -- 172.21.15.39:0/3785996224 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564c023b8810 msgr2=0x564c023b8c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:06.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.897+0000 7f406cde3740 1 --2- 172.21.15.39:0/3785996224 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564c023b8810 0x564c023b8c30 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f405c009fd0 tx=0x7f405c0118d0 comp rx=0 tx=0).stop 2024-03-18T16:19:06.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.897+0000 7f405be00640 1 -- 172.21.15.39:0/1516824854 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f40540f08a0 con 0x7f4064102280 2024-03-18T16:19:06.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.897+0000 7f406cde3740 1 -- 172.21.15.39:0/3785996224 shutdown_connections 2024-03-18T16:19:06.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.897+0000 7f4063400640 1 --2- 172.21.15.39:0/1516824854 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f40480a2800 0x7f40480a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:06.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.897+0000 7f406cde3740 1 --2- 172.21.15.39:0/3785996224 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564c023b9200 0x564c023bdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:06.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.897+0000 7f406cde3740 1 --2- 172.21.15.39:0/3785996224 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564c023c2780 0x564c023c4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:06.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.897+0000 7f406cde3740 1 --2- 172.21.15.39:0/3785996224 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564c023b8810 0x564c023b8c30 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:06.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.897+0000 7f406cde3740 1 -- 172.21.15.39:0/3785996224 >> 172.21.15.39:0/3785996224 conn(0x564c023b3d80 msgr2=0x564c023b4180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:06.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.897+0000 7f406cde3740 1 -- 172.21.15.39:0/3785996224 shutdown_connections 2024-03-18T16:19:06.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.897+0000 7f406cde3740 1 -- 172.21.15.39:0/3785996224 wait complete. 2024-03-18T16:19:06.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.898+0000 7f4063400640 1 --2- 172.21.15.39:0/1516824854 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f40480a2800 0x7f40480a4cc0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f4054006010 tx=0x7f4054009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:06.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.898+0000 7f406cde3740 1 -- start start 2024-03-18T16:19:06.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.898+0000 7f406cde3740 1 Processor -- start 2024-03-18T16:19:06.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.899+0000 7f406cde3740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564c023b8810 0x564c023bbcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:06.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.899+0000 7f406cde3740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564c023b9200 0x564c023bc200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:06.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.899+0000 7f406cde3740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564c023c2780 0x564c023bd1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:06.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.899+0000 7f406cde3740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x564c023be6a0 con 0x564c023c2780 2024-03-18T16:19:06.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.899+0000 7f406cde3740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x564c022d8ed0 con 0x564c023b8810 2024-03-18T16:19:06.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.899+0000 7f406cde3740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x564c023be820 con 0x564c023b9200 2024-03-18T16:19:06.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.899+0000 7f406cde3740 10 client.0 _ll_register_callbacks cb 0x564c02314220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:19:06.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.899+0000 7f4063e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564c023b8810 0x564c023bbcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:06.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.899+0000 7f4063e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564c023b8810 0x564c023bbcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:35348/0 (socket says 172.21.15.39:35348) 2024-03-18T16:19:06.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.899+0000 7f4063e00640 1 -- 172.21.15.39:0/3478615593 learned_addr learned my addr 172.21.15.39:0/3478615593 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:19:06.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.900+0000 7f4063e00640 1 --2- 172.21.15.39:0/3478615593 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564c023c2780 0x564c023bd1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:06.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.900+0000 7f4062a00640 1 --2- 172.21.15.39:0/3478615593 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564c023b9200 0x564c023bc200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:06.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.900+0000 7f4063e00640 1 -- 172.21.15.39:0/3478615593 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564c023b9200 msgr2=0x564c023bc200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:06.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.900+0000 7f4063e00640 1 --2- 172.21.15.39:0/3478615593 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564c023b9200 0x564c023bc200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:06.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.900+0000 7f4063e00640 1 -- 172.21.15.39:0/3478615593 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564c023c2780 msgr2=0x564c023bd1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:06.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.900+0000 7f4063e00640 1 --2- 172.21.15.39:0/3478615593 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564c023c2780 0x564c023bd1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:06.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.900+0000 7f4063e00640 1 -- 172.21.15.39:0/3478615593 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564c020313f0 con 0x564c023b8810 2024-03-18T16:19:06.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.900+0000 7f4062a00640 1 --2- 172.21.15.39:0/3478615593 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564c023b9200 0x564c023bc200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:19:06.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.900+0000 7f4063e00640 1 --2- 172.21.15.39:0/3478615593 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564c023b8810 0x564c023bbcc0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f405c009fa0 tx=0x7f405c000f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec6c7d37db74e72 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:06.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.901+0000 7f404cc00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:19:06.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.901+0000 7f404cc00640 1 -- 172.21.15.39:0/3478615593 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f405c02c020 con 0x564c023b8810 2024-03-18T16:19:06.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.901+0000 7f404cc00640 1 -- 172.21.15.39:0/3478615593 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f405c00f6e0 con 0x564c023b8810 2024-03-18T16:19:06.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.901+0000 7f406cde3740 1 -- 172.21.15.39:0/3478615593 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f405c004050 con 0x564c023b8810 2024-03-18T16:19:06.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.901+0000 7f404cc00640 1 -- 172.21.15.39:0/3478615593 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f405c00f8e0 con 0x564c023b8810 2024-03-18T16:19:06.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.901+0000 7f406cde3740 10 client.27470 fetch_fsmap learned FSMap version 44 2024-03-18T16:19:06.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.902+0000 7f406cde3740 1 -- 172.21.15.39:0/3478615593 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x564c01ff10f0 con 0x564c023b8810 2024-03-18T16:19:06.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.902+0000 7f404cc00640 1 -- 172.21.15.39:0/3478615593 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f405c004050 con 0x564c023b8810 2024-03-18T16:19:06.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.902+0000 7f404cc00640 1 -- 172.21.15.39:0/3478615593 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7f405c039020 con 0x564c023b8810 2024-03-18T16:19:06.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.902+0000 7f406cde3740 10 client.27470 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:19:06.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.902+0000 7f406cde3740 10 client.27470 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:19:06.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.903+0000 7f406cde3740 20 client.27470 populate_metadata read hostname 'smithi039' 2024-03-18T16:19:06.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.903+0000 7f406cde3740 1 --2- 172.21.15.39:0/3478615593 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x564c023d7150 0x564c023d9540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:06.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.903+0000 7f406cde3740 4 client.27470 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:19:06.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.903+0000 7f406cde3740 1 -- 172.21.15.39:0/3478615593 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564c02330be0 con 0x564c023d7150 2024-03-18T16:19:06.906 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.903+0000 7f4062a00640 1 --2- 172.21.15.39:0/3478615593 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x564c023d7150 0x564c023d9540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:06.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.904+0000 7f4062a00640 1 --2- 172.21.15.39:0/3478615593 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x564c023d7150 0x564c023d9540 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b228028d78e1e956 server_cookie=296b3bcd5899de0a in_seq=0 out_seq=0 2024-03-18T16:19:06.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.904+0000 7f404cc00640 10 client.27470 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:19:06.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.905+0000 7f404cc00640 1 -- 172.21.15.39:0/3478615593 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x564c02330be0 con 0x564c023d7150 2024-03-18T16:19:06.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.905+0000 7f404cc00640 10 client.27470 handle_command_reply: tid=0 2024-03-18T16:19:06.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.905+0000 7f406cde3740 1 client.27470 shutdown 2024-03-18T16:19:06.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.905+0000 7f406cde3740 1 -- 172.21.15.39:0/3478615593 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564c023b8810 msgr2=0x564c023bbcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:06.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.905+0000 7f406cde3740 1 --2- 172.21.15.39:0/3478615593 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564c023b8810 0x564c023bbcc0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f405c009fa0 tx=0x7f405c000f40 comp rx=0 tx=0).stop 2024-03-18T16:19:06.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.905+0000 7f406cde3740 1 -- 172.21.15.39:0/3478615593 shutdown_connections 2024-03-18T16:19:06.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.906+0000 7f406cde3740 1 --2- 172.21.15.39:0/3478615593 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x564c023d7150 0x564c023d9540 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:06.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.906+0000 7f406cde3740 1 --2- 172.21.15.39:0/3478615593 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564c023c2780 0x564c023bd1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:06.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.906+0000 7f406cde3740 1 --2- 172.21.15.39:0/3478615593 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564c023b9200 0x564c023bc200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:06.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.906+0000 7f406cde3740 1 --2- 172.21.15.39:0/3478615593 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564c023b8810 0x564c023bbcc0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:06.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.906+0000 7f406cde3740 1 -- 172.21.15.39:0/3478615593 >> 172.21.15.39:0/3478615593 conn(0x564c023b3d80 msgr2=0x564c023b2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:06.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.906+0000 7f404cc00640 0 client.27470 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:19:06.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.906+0000 7f406cde3740 1 -- 172.21.15.39:0/3478615593 shutdown_connections 2024-03-18T16:19:06.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.906+0000 7f406cde3740 1 -- 172.21.15.39:0/3478615593 wait complete. 2024-03-18T16:19:06.908 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.906+0000 7f406cde3740 20 client.27470 trim_cache size 0 max 16384 2024-03-18T16:19:06.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.932+0000 7f406cde3740 1 Processor -- start 2024-03-18T16:19:06.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.932+0000 7f406cde3740 1 -- start start 2024-03-18T16:19:06.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.932+0000 7f406cde3740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564c023b8810 0x564c023b2d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:06.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.932+0000 7f406cde3740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564c023b9200 0x564c023bb2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:06.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.932+0000 7f406cde3740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564c023c2780 0x564c023bb820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:06.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.932+0000 7f406cde3740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f40641a82d0 con 0x564c023b8810 2024-03-18T16:19:06.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.932+0000 7f406cde3740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f40641a9e10 con 0x564c023c2780 2024-03-18T16:19:06.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.932+0000 7f406cde3740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f40641a89a0 con 0x564c023b9200 2024-03-18T16:19:06.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.933+0000 7f4063e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564c023b8810 0x564c023b2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:06.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.933+0000 7f4062a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564c023b9200 0x564c023bb2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:06.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.933+0000 7f4062a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564c023b9200 0x564c023bb2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:59668/0 (socket says 172.21.15.39:59668) 2024-03-18T16:19:06.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.933+0000 7f4062a00640 1 -- 172.21.15.39:0/1925410216 learned_addr learned my addr 172.21.15.39:0/1925410216 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:19:06.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.933+0000 7f4063e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564c023b8810 0x564c023b2d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:49304/0 (socket says 172.21.15.39:49304) 2024-03-18T16:19:06.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.933+0000 7f4063e00640 1 --2- 172.21.15.39:0/1925410216 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564c023c2780 0x564c023bb820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:06.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.933+0000 7f4063e00640 1 -- 172.21.15.39:0/1925410216 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564c023b9200 msgr2=0x564c023bb2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:06.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.933+0000 7f4063e00640 1 --2- 172.21.15.39:0/1925410216 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564c023b9200 0x564c023bb2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:06.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.933+0000 7f4063e00640 1 -- 172.21.15.39:0/1925410216 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564c023b8810 msgr2=0x564c023b2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:06.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.933+0000 7f4063e00640 1 --2- 172.21.15.39:0/1925410216 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564c023b8810 0x564c023b2d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:06.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.933+0000 7f4063e00640 1 -- 172.21.15.39:0/1925410216 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564c023ae370 con 0x564c023c2780 2024-03-18T16:19:06.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.934+0000 7f4063e00640 1 --2- 172.21.15.39:0/1925410216 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564c023c2780 0x564c023bb820 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f405c030fa0 tx=0x7f405c009c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d28f3a20a828d75d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:06.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.934+0000 7f404f400640 1 -- 172.21.15.39:0/1925410216 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f405c02c040 con 0x564c023c2780 2024-03-18T16:19:06.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.934+0000 7f404f400640 1 -- 172.21.15.39:0/1925410216 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f405c004bb0 con 0x564c023c2780 2024-03-18T16:19:06.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.934+0000 7f404f400640 1 -- 172.21.15.39:0/1925410216 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f405c00f4e0 con 0x564c023c2780 2024-03-18T16:19:06.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.935+0000 7f406cde3740 1 -- 172.21.15.39:0/1925410216 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564c023c2780 msgr2=0x564c023bb820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:06.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.935+0000 7f406cde3740 1 --2- 172.21.15.39:0/1925410216 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564c023c2780 0x564c023bb820 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f405c030fa0 tx=0x7f405c009c80 comp rx=0 tx=0).stop 2024-03-18T16:19:06.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.935+0000 7f406cde3740 1 -- 172.21.15.39:0/1925410216 shutdown_connections 2024-03-18T16:19:06.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.935+0000 7f406cde3740 1 --2- 172.21.15.39:0/1925410216 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564c023c2780 0x564c023bb820 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:06.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.935+0000 7f406cde3740 1 --2- 172.21.15.39:0/1925410216 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564c023b9200 0x564c023bb2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:06.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.935+0000 7f406cde3740 1 --2- 172.21.15.39:0/1925410216 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564c023b8810 0x564c023b2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:06.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.935+0000 7f406cde3740 1 -- 172.21.15.39:0/1925410216 >> 172.21.15.39:0/1925410216 conn(0x564c023b3d80 msgr2=0x564c023b2630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:06.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.935+0000 7f406cde3740 1 -- 172.21.15.39:0/1925410216 shutdown_connections 2024-03-18T16:19:06.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.935+0000 7f406cde3740 1 -- 172.21.15.39:0/1925410216 wait complete. 2024-03-18T16:19:06.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.936+0000 7f406cde3740 1 -- start start 2024-03-18T16:19:06.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.936+0000 7f406cde3740 1 Processor -- start 2024-03-18T16:19:06.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.937+0000 7f406cde3740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564c023b8810 0x564c023ced00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:06.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.937+0000 7f406cde3740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564c023b9200 0x564c023cf240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:06.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.937+0000 7f406cde3740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564c023c2780 0x564c023d2260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:06.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.937+0000 7f406cde3740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f40641a9740 con 0x564c023c2780 2024-03-18T16:19:06.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.937+0000 7f406cde3740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f40641a9070 con 0x564c023b9200 2024-03-18T16:19:06.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.937+0000 7f406cde3740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f40641a62b0 con 0x564c023b8810 2024-03-18T16:19:06.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.937+0000 7f406cde3740 10 client.0 _ll_register_callbacks cb 0x7f405c009ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:19:06.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.937+0000 7f4063e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564c023b8810 0x564c023ced00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:06.939 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.937+0000 7f4063e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564c023c2780 0x564c023d2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:06.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.937+0000 7f4063e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564c023c2780 0x564c023d2260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:49312/0 (socket says 172.21.15.39:49312) 2024-03-18T16:19:06.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.937+0000 7f4063e00640 1 -- 172.21.15.39:0/439690196 learned_addr learned my addr 172.21.15.39:0/439690196 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:19:06.940 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.937+0000 7f4062a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564c023b9200 0x564c023cf240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:06.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.938+0000 7f4062a00640 1 -- 172.21.15.39:0/439690196 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564c023b8810 msgr2=0x564c023ced00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:06.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.938+0000 7f4062a00640 1 --2- 172.21.15.39:0/439690196 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564c023b8810 0x564c023ced00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:06.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.938+0000 7f4062a00640 1 -- 172.21.15.39:0/439690196 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564c023c2780 msgr2=0x564c023d2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:06.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.938+0000 7f4062a00640 1 --2- 172.21.15.39:0/439690196 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564c023c2780 0x564c023d2260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:06.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.938+0000 7f4062a00640 1 -- 172.21.15.39:0/439690196 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564c023d2850 con 0x564c023b9200 2024-03-18T16:19:06.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.938+0000 7f4063e00640 1 --2- 172.21.15.39:0/439690196 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564c023b8810 0x564c023ced00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:19:06.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.938+0000 7f4062a00640 1 --2- 172.21.15.39:0/439690196 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564c023b9200 0x564c023cf240 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x564c023b3d50 tx=0x7f4050012a60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=61dd339b7b5b2a14 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:06.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.938+0000 7f404cc00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:19:06.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.938+0000 7f404cc00640 1 -- 172.21.15.39:0/439690196 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40500223a0 con 0x564c023b9200 2024-03-18T16:19:06.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.938+0000 7f406cde3740 1 -- 172.21.15.39:0/439690196 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f405c004bb0 con 0x564c023b9200 2024-03-18T16:19:06.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.938+0000 7f404cc00640 1 -- 172.21.15.39:0/439690196 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f4050022540 con 0x564c023b9200 2024-03-18T16:19:06.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.940+0000 7f404cc00640 1 -- 172.21.15.39:0/439690196 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f4050022740 con 0x564c023b9200 2024-03-18T16:19:06.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.940+0000 7f406cde3740 10 client.27476 fetch_fsmap learned FSMap version 44 2024-03-18T16:19:06.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.940+0000 7f406cde3740 1 -- 172.21.15.39:0/439690196 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x564c0244e450 con 0x564c023b9200 2024-03-18T16:19:06.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.940+0000 7f404cc00640 1 -- 172.21.15.39:0/439690196 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f405c004bb0 con 0x564c023b9200 2024-03-18T16:19:06.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.940+0000 7f404cc00640 1 -- 172.21.15.39:0/439690196 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7f405002b5a0 con 0x564c023b9200 2024-03-18T16:19:06.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.940+0000 7f406cde3740 10 client.27476 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:19:06.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.940+0000 7f406cde3740 10 client.27476 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:19:06.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.940+0000 7f406cde3740 20 client.27476 populate_metadata read hostname 'smithi039' 2024-03-18T16:19:06.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.940+0000 7f406cde3740 1 --2- 172.21.15.39:0/439690196 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x564c0244edc0 0x564c024511e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:06.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.941+0000 7f406cde3740 4 client.27476 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T16:19:06.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.941+0000 7f406cde3740 1 -- 172.21.15.39:0/439690196 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "scrub status"}) -- 0x564c0232a730 con 0x564c0244edc0 2024-03-18T16:19:06.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.941+0000 7f4063e00640 1 --2- 172.21.15.39:0/439690196 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x564c0244edc0 0x564c024511e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:06.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.941+0000 7f4063e00640 1 --2- 172.21.15.39:0/439690196 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x564c0244edc0 0x564c024511e0 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=55209ed04cb6c9e server_cookie=c7a1f5b0158d7223 in_seq=0 out_seq=0 2024-03-18T16:19:06.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.941+0000 7f404cc00640 10 client.27476 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:19:06.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.942+0000 7f404cc00640 1 -- 172.21.15.39:0/439690196 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x564c0232a730 con 0x564c0244edc0 2024-03-18T16:19:06.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.942+0000 7f404cc00640 10 client.27476 handle_command_reply: tid=0 2024-03-18T16:19:06.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.942+0000 7f406cde3740 1 client.27476 shutdown 2024-03-18T16:19:06.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.942+0000 7f406cde3740 1 -- 172.21.15.39:0/439690196 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564c023b9200 msgr2=0x564c023cf240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:06.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.942+0000 7f406cde3740 1 --2- 172.21.15.39:0/439690196 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564c023b9200 0x564c023cf240 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x564c023b3d50 tx=0x7f4050012a60 comp rx=0 tx=0).stop 2024-03-18T16:19:06.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.942+0000 7f406cde3740 1 -- 172.21.15.39:0/439690196 shutdown_connections 2024-03-18T16:19:06.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.942+0000 7f406cde3740 1 --2- 172.21.15.39:0/439690196 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x564c0244edc0 0x564c024511e0 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:06.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.942+0000 7f406cde3740 1 --2- 172.21.15.39:0/439690196 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564c023b9200 0x564c023cf240 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:06.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.942+0000 7f406cde3740 1 --2- 172.21.15.39:0/439690196 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564c023c2780 0x564c023d2260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:06.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.942+0000 7f404cc00640 0 client.27476 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:19:06.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.942+0000 7f406cde3740 1 --2- 172.21.15.39:0/439690196 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564c023b8810 0x564c023ced00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:06.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.942+0000 7f406cde3740 1 -- 172.21.15.39:0/439690196 >> 172.21.15.39:0/439690196 conn(0x564c023b3d80 msgr2=0x564c023b6fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:06.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.943+0000 7f406cde3740 1 -- 172.21.15.39:0/439690196 shutdown_connections 2024-03-18T16:19:06.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.943+0000 7f406cde3740 1 -- 172.21.15.39:0/439690196 wait complete. 2024-03-18T16:19:06.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.943+0000 7f406cde3740 20 client.27476 trim_cache size 0 max 16384 2024-03-18T16:19:06.946 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-03-18T16:19:06.946 INFO:teuthology.orchestra.run.smithi039.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-18T16:19:06.946 INFO:teuthology.orchestra.run.smithi039.stdout: "scrubs": { 2024-03-18T16:19:06.946 INFO:teuthology.orchestra.run.smithi039.stdout: "4f10a011-7bc8-4a85-8d00-b669c7b1d918": { 2024-03-18T16:19:06.947 INFO:teuthology.orchestra.run.smithi039.stdout: "path": "/", 2024-03-18T16:19:06.947 INFO:teuthology.orchestra.run.smithi039.stdout: "tag": "4f10a011-7bc8-4a85-8d00-b669c7b1d918", 2024-03-18T16:19:06.947 INFO:teuthology.orchestra.run.smithi039.stdout: "options": "recursive,force" 2024-03-18T16:19:06.947 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:19:06.947 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:19:06.947 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-03-18T16:19:06.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.944+0000 7f406a000640 1 -- 172.21.15.39:0/1516824854 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f40480a2800 msgr2=0x7f40480a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:06.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.944+0000 7f406a000640 1 --2- 172.21.15.39:0/1516824854 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f40480a2800 0x7f40480a4cc0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f4054006010 tx=0x7f4054009260 comp rx=0 tx=0).stop 2024-03-18T16:19:06.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.944+0000 7f406a000640 1 -- 172.21.15.39:0/1516824854 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4064102280 msgr2=0x7f4064197be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:06.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.944+0000 7f406a000640 1 --2- 172.21.15.39:0/1516824854 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4064102280 0x7f4064197be0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f405402c260 tx=0x7f405402c670 comp rx=0 tx=0).stop 2024-03-18T16:19:06.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.944+0000 7f406a000640 1 -- 172.21.15.39:0/1516824854 shutdown_connections 2024-03-18T16:19:06.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.944+0000 7f406a000640 1 --2- 172.21.15.39:0/1516824854 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f40641074e0 0x7f406419f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:06.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.944+0000 7f406a000640 1 --2- 172.21.15.39:0/1516824854 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f40480a2800 0x7f40480a4cc0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:06.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.944+0000 7f406a000640 1 --2- 172.21.15.39:0/1516824854 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4064104bb0 0x7f4064198120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:06.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.944+0000 7f406a000640 1 --2- 172.21.15.39:0/1516824854 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4064102280 0x7f4064197be0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:06.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.944+0000 7f406a000640 1 -- 172.21.15.39:0/1516824854 >> 172.21.15.39:0/1516824854 conn(0x7f40640fbc90 msgr2=0x7f4064102be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:06.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.944+0000 7f406a000640 1 -- 172.21.15.39:0/1516824854 shutdown_connections 2024-03-18T16:19:06.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:06.944+0000 7f406a000640 1 -- 172.21.15.39:0/1516824854 wait complete. 2024-03-18T16:19:06.967 INFO:tasks.cephfs.filesystem:scrub status for tag:4f10a011-7bc8-4a85-8d00-b669c7b1d918 - {'path': '/', 'tag': '4f10a011-7bc8-4a85-8d00-b669c7b1d918', 'options': 'recursive,force'} 2024-03-18T16:19:07.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:07 smithi039 ceph-mon[31721]: pgmap v693: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 568 KiB/s wr, 153 op/s 2024-03-18T16:19:07.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:07 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:19:07.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:07 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:07.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:07 smithi196 ceph-mon[36307]: pgmap v693: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 568 KiB/s wr, 153 op/s 2024-03-18T16:19:07.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:07 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:19:07.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:07 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:07.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:07 smithi194 ceph-mon[36200]: pgmap v693: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 568 KiB/s wr, 153 op/s 2024-03-18T16:19:07.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:07 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:19:07.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:07 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:08.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:08 smithi039 ceph-mon[31721]: scrub queued for path: / 2024-03-18T16:19:08.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:08 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:08.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:08 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:19:08.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:08 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:19:08.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:08 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:19:08.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:08 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:19:08.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:08 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:08.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:08 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:08.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:08 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:08.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:08 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:19:08.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:08 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:08.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:08 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:08.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:08 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:19:08.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:08 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:08.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:08 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:19:08.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:08 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:08.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:08 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:19:08.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:08 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:08.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:08 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:19:08.339 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:08 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:08.339 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:08 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:19:08.339 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:08 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:08.339 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:08 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:19:08.339 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:08 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:08.339 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:08 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:19:08.339 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:08 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:08.339 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:08 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:19:08.339 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:08 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:08.339 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:08 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:19:08.339 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:08 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:08.339 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:08 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:19:08.339 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:08 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:08.339 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:08 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:19:08.339 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:08 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:08.340 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:08 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:19:08.340 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:08 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:08.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:08 smithi196 ceph-mon[36307]: scrub queued for path: / 2024-03-18T16:19:08.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:08 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:08.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:08 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:19:08.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:08 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:19:08.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:08 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:19:08.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:08 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:19:08.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:08 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:08.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:08 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:08.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:08 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:08.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:08 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:19:08.480 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:08 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:08.480 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:08 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:08.480 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:08 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:19:08.480 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:08 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:08.480 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:08 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:19:08.480 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:08 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:08.480 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:08 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:19:08.480 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:08 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:08.480 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:08 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:19:08.480 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:08 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:08.480 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:08 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:19:08.480 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:08 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:08.480 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:08 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:19:08.480 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:08 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:08.480 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:08 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:19:08.481 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:08 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:08.481 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:08 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:19:08.481 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:08 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:08.481 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:08 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:19:08.481 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:08 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:08.481 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:08 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:19:08.481 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:08 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:08.481 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:08 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:19:08.481 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:08 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:08.481 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:08 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:19:08.481 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:08 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:08.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:08 smithi194 ceph-mon[36200]: scrub queued for path: / 2024-03-18T16:19:08.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:08 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:08.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:08 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:19:08.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:08 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:19:08.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:08 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:19:08.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:08 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:19:08.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:08 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:08.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:08 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:08.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:08 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:08.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:08 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:19:08.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:08 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:08.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:08 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:08.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:08 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:19:08.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:08 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:08.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:08 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:19:08.492 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:08 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:08.492 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:08 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:19:08.492 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:08 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:08.492 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:08 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:19:08.492 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:08 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:08.492 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:08 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:19:08.492 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:08 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:08.492 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:08 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:19:08.492 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:08 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:08.492 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:08 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:19:08.492 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:08 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:08.492 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:08 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:19:08.492 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:08 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:08.493 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:08 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:19:08.493 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:08 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:08.493 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:08 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:19:08.493 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:08 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:08.493 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:08 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:19:08.493 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:08 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:08.493 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:08 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:19:08.493 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:08 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:09.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:09 smithi039 ceph-mon[31721]: pgmap v694: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 678 KiB/s wr, 180 op/s 2024-03-18T16:19:09.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:09 smithi196 ceph-mon[36307]: pgmap v694: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 678 KiB/s wr, 180 op/s 2024-03-18T16:19:09.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:09 smithi194 ceph-mon[36200]: pgmap v694: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 678 KiB/s wr, 180 op/s 2024-03-18T16:19:11.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:11 smithi039 ceph-mon[31721]: pgmap v695: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 561 KiB/s wr, 151 op/s 2024-03-18T16:19:11.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:11 smithi196 ceph-mon[36307]: pgmap v695: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 561 KiB/s wr, 151 op/s 2024-03-18T16:19:11.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:11 smithi194 ceph-mon[36200]: pgmap v695: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 561 KiB/s wr, 151 op/s 2024-03-18T16:19:13.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:13 smithi039 ceph-mon[31721]: pgmap v696: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 634 KiB/s wr, 169 op/s 2024-03-18T16:19:13.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:13 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:19:13.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:13 smithi196 ceph-mon[36307]: pgmap v696: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 634 KiB/s wr, 169 op/s 2024-03-18T16:19:13.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:13 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:19:13.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:13 smithi194 ceph-mon[36200]: pgmap v696: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 634 KiB/s wr, 169 op/s 2024-03-18T16:19:13.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:13 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:19:14.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:14 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:19:14.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:14 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:19:14.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:14 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:19:14.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:14 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:19:14.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:14 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:19:14.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:14 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:19:14.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:14 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:19:14.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:14 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:19:14.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:14 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:19:15.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:15 smithi039 ceph-mon[31721]: pgmap v697: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 655 KiB/s wr, 174 op/s 2024-03-18T16:19:15.391 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:15 smithi194 ceph-mon[36200]: pgmap v697: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 655 KiB/s wr, 174 op/s 2024-03-18T16:19:15.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:15 smithi196 ceph-mon[36307]: pgmap v697: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 655 KiB/s wr, 174 op/s 2024-03-18T16:19:17.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:17 smithi039 ceph-mon[31721]: pgmap v698: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 562 KiB/s wr, 151 op/s 2024-03-18T16:19:17.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:17 smithi196 ceph-mon[36307]: pgmap v698: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 562 KiB/s wr, 151 op/s 2024-03-18T16:19:17.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:17 smithi194 ceph-mon[36200]: pgmap v698: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 562 KiB/s wr, 151 op/s 2024-03-18T16:19:18.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:18 smithi039 ceph-mon[31721]: scrub summary: idle 2024-03-18T16:19:18.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:18 smithi039 ceph-mon[31721]: scrub summary: idle 2024-03-18T16:19:18.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:18 smithi039 ceph-mon[31721]: scrub summary: idle 2024-03-18T16:19:18.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:18 smithi196 ceph-mon[36307]: scrub summary: idle 2024-03-18T16:19:18.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:18 smithi196 ceph-mon[36307]: scrub summary: idle 2024-03-18T16:19:18.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:18 smithi196 ceph-mon[36307]: scrub summary: idle 2024-03-18T16:19:18.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:18 smithi194 ceph-mon[36200]: scrub summary: idle 2024-03-18T16:19:18.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:18 smithi194 ceph-mon[36200]: scrub summary: idle 2024-03-18T16:19:18.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:18 smithi194 ceph-mon[36200]: scrub summary: idle 2024-03-18T16:19:19.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:19 smithi039 ceph-mon[31721]: scrub summary: idle 2024-03-18T16:19:19.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:19 smithi039 ceph-mon[31721]: scrub summary: idle 2024-03-18T16:19:19.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:19 smithi039 ceph-mon[31721]: pgmap v699: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 660 KiB/s wr, 175 op/s 2024-03-18T16:19:19.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:19 smithi196 ceph-mon[36307]: scrub summary: idle 2024-03-18T16:19:19.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:19 smithi196 ceph-mon[36307]: scrub summary: idle 2024-03-18T16:19:19.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:19 smithi196 ceph-mon[36307]: pgmap v699: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 660 KiB/s wr, 175 op/s 2024-03-18T16:19:19.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:19 smithi194 ceph-mon[36200]: scrub summary: idle 2024-03-18T16:19:19.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:19 smithi194 ceph-mon[36200]: scrub summary: idle 2024-03-18T16:19:19.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:19 smithi194 ceph-mon[36200]: pgmap v699: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 660 KiB/s wr, 175 op/s 2024-03-18T16:19:21.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:21 smithi196 ceph-mon[36307]: pgmap v700: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 551 KiB/s wr, 148 op/s 2024-03-18T16:19:21.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:21 smithi194 ceph-mon[36200]: pgmap v700: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 551 KiB/s wr, 148 op/s 2024-03-18T16:19:21.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:21 smithi039 ceph-mon[31721]: pgmap v700: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 551 KiB/s wr, 148 op/s 2024-03-18T16:19:23.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:23 smithi196 ceph-mon[36307]: pgmap v701: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 617 KiB/s wr, 164 op/s 2024-03-18T16:19:23.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:23 smithi194 ceph-mon[36200]: pgmap v701: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 617 KiB/s wr, 164 op/s 2024-03-18T16:19:23.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:23 smithi039 ceph-mon[31721]: pgmap v701: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 617 KiB/s wr, 164 op/s 2024-03-18T16:19:24.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:24 smithi196 ceph-mon[36307]: pgmap v702: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 643 KiB/s wr, 171 op/s 2024-03-18T16:19:24.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:24 smithi194 ceph-mon[36200]: pgmap v702: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 643 KiB/s wr, 171 op/s 2024-03-18T16:19:24.586 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:24 smithi039 ceph-mon[31721]: pgmap v702: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 643 KiB/s wr, 171 op/s 2024-03-18T16:19:27.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:26 smithi196 ceph-mon[36307]: pgmap v703: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 552 KiB/s wr, 148 op/s 2024-03-18T16:19:27.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:26 smithi194 ceph-mon[36200]: pgmap v703: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 552 KiB/s wr, 148 op/s 2024-03-18T16:19:27.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:26 smithi039 ceph-mon[31721]: pgmap v703: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 552 KiB/s wr, 148 op/s 2024-03-18T16:19:29.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:28 smithi196 ceph-mon[36307]: pgmap v704: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 659 KiB/s wr, 174 op/s 2024-03-18T16:19:29.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:28 smithi194 ceph-mon[36200]: pgmap v704: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 659 KiB/s wr, 174 op/s 2024-03-18T16:19:29.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:28 smithi039 ceph-mon[31721]: pgmap v704: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 659 KiB/s wr, 174 op/s 2024-03-18T16:19:31.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:30 smithi196 ceph-mon[36307]: pgmap v705: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 553 KiB/s wr, 149 op/s 2024-03-18T16:19:31.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:30 smithi194 ceph-mon[36200]: pgmap v705: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 553 KiB/s wr, 149 op/s 2024-03-18T16:19:31.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:30 smithi039 ceph-mon[31721]: pgmap v705: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 553 KiB/s wr, 149 op/s 2024-03-18T16:19:33.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:32 smithi196 ceph-mon[36307]: pgmap v706: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 608 KiB/s wr, 162 op/s 2024-03-18T16:19:33.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:32 smithi194 ceph-mon[36200]: pgmap v706: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 608 KiB/s wr, 162 op/s 2024-03-18T16:19:33.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:32 smithi039 ceph-mon[31721]: pgmap v706: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 608 KiB/s wr, 162 op/s 2024-03-18T16:19:35.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:34 smithi196 ceph-mon[36307]: pgmap v707: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 647 KiB/s wr, 172 op/s 2024-03-18T16:19:35.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:34 smithi194 ceph-mon[36200]: pgmap v707: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 647 KiB/s wr, 172 op/s 2024-03-18T16:19:35.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:34 smithi039 ceph-mon[31721]: pgmap v707: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 647 KiB/s wr, 172 op/s 2024-03-18T16:19:36.968 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T16:19:36.968 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T16:19:37.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.211+0000 7f02cee00640 1 -- 172.21.15.39:0/3095379225 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f02c8102270 msgr2=0x7f02c8104660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:37.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.211+0000 7f02cee00640 1 --2- 172.21.15.39:0/3095379225 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f02c8102270 0x7f02c8104660 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f02b8009900 tx=0x7f02b802bc40 comp rx=0 tx=0).stop 2024-03-18T16:19:37.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.211+0000 7f02cee00640 1 -- 172.21.15.39:0/3095379225 shutdown_connections 2024-03-18T16:19:37.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.211+0000 7f02cee00640 1 --2- 172.21.15.39:0/3095379225 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f02c81074d0 0x7f02c8109920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:37.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.211+0000 7f02cee00640 1 --2- 172.21.15.39:0/3095379225 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f02c8104ba0 0x7f02c8106f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:37.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.211+0000 7f02cee00640 1 --2- 172.21.15.39:0/3095379225 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f02c8102270 0x7f02c8104660 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:37.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.211+0000 7f02cee00640 1 -- 172.21.15.39:0/3095379225 >> 172.21.15.39:0/3095379225 conn(0x7f02c80fbc80 msgr2=0x7f02c80fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:37.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.211+0000 7f02cee00640 1 -- 172.21.15.39:0/3095379225 shutdown_connections 2024-03-18T16:19:37.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.211+0000 7f02cee00640 1 -- 172.21.15.39:0/3095379225 wait complete. 2024-03-18T16:19:37.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.213+0000 7f02cee00640 1 Processor -- start 2024-03-18T16:19:37.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.213+0000 7f02cee00640 1 -- start start 2024-03-18T16:19:37.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.213+0000 7f02cee00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f02c8102270 0x7f02c8101be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:37.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.213+0000 7f02cee00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f02c8104ba0 0x7f02c8100220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:37.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.214+0000 7f02cee00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f02c81074d0 0x7f02c8100760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:37.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.214+0000 7f02cee00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f02c810d470 con 0x7f02c81074d0 2024-03-18T16:19:37.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.214+0000 7f02cee00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f02c810d2f0 con 0x7f02c8102270 2024-03-18T16:19:37.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.214+0000 7f02cee00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f02c810d5f0 con 0x7f02c8104ba0 2024-03-18T16:19:37.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.214+0000 7f02c7e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f02c8102270 0x7f02c8101be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:37.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.214+0000 7f02c7e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f02c8102270 0x7f02c8101be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:51268/0 (socket says 172.21.15.39:51268) 2024-03-18T16:19:37.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.214+0000 7f02c7e00640 1 -- 172.21.15.39:0/1143002672 learned_addr learned my addr 172.21.15.39:0/1143002672 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:19:37.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.214+0000 7f02c7400640 1 --2- 172.21.15.39:0/1143002672 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f02c8104ba0 0x7f02c8100220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:37.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.215+0000 7f02c7400640 1 -- 172.21.15.39:0/1143002672 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f02c8102270 msgr2=0x7f02c8101be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:37.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.215+0000 7f02c7400640 1 --2- 172.21.15.39:0/1143002672 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f02c8102270 0x7f02c8101be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:37.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.215+0000 7f02c7400640 1 -- 172.21.15.39:0/1143002672 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f02c81074d0 msgr2=0x7f02c8100760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:37.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.215+0000 7f02c7400640 1 --2- 172.21.15.39:0/1143002672 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f02c81074d0 0x7f02c8100760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:37.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.215+0000 7f02c7400640 1 -- 172.21.15.39:0/1143002672 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02c8100ca0 con 0x7f02c8104ba0 2024-03-18T16:19:37.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.215+0000 7f02c7400640 1 --2- 172.21.15.39:0/1143002672 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f02c8104ba0 0x7f02c8100220 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f02b400c940 tx=0x7f02b400ce00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:37.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.215+0000 7f02c4c00640 1 -- 172.21.15.39:0/1143002672 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02b4007bb0 con 0x7f02c8104ba0 2024-03-18T16:19:37.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.215+0000 7f02c4c00640 1 -- 172.21.15.39:0/1143002672 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02b4007d50 con 0x7f02c8104ba0 2024-03-18T16:19:37.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.215+0000 7f02c4c00640 1 -- 172.21.15.39:0/1143002672 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02b4018860 con 0x7f02c8104ba0 2024-03-18T16:19:37.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.218+0000 7f02cee00640 1 -- 172.21.15.39:0/1143002672 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02c8100f90 con 0x7f02c8104ba0 2024-03-18T16:19:37.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.218+0000 7f02cee00640 1 -- 172.21.15.39:0/1143002672 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02c81a6480 con 0x7f02c8104ba0 2024-03-18T16:19:37.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.223+0000 7f02d1bd3740 1 Processor -- start 2024-03-18T16:19:37.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.223+0000 7f02d1bd3740 1 -- start start 2024-03-18T16:19:37.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.224+0000 7f02c4c00640 1 -- 172.21.15.39:0/1143002672 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f02b4018a00 con 0x7f02c8104ba0 2024-03-18T16:19:37.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.224+0000 7f02d1bd3740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bca88e9810 0x55bca88e9c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:37.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.224+0000 7f02d1bd3740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bca88ea200 0x55bca88eed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:37.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.224+0000 7f02d1bd3740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bca88ef370 0x55bca88f1800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:37.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.224+0000 7f02d1bd3740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55bca884e1f0 con 0x55bca88e9810 2024-03-18T16:19:37.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.224+0000 7f02d1bd3740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55bca87f6a80 con 0x55bca88ea200 2024-03-18T16:19:37.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.224+0000 7f02d1bd3740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55bca884a290 con 0x55bca88ef370 2024-03-18T16:19:37.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.224+0000 7f02c7400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bca88ef370 0x55bca88f1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:37.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.224+0000 7f02cce00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bca88e9810 0x55bca88e9c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:37.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.224+0000 7f02c7400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bca88ef370 0x55bca88f1800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:46770/0 (socket says 172.21.15.39:46770) 2024-03-18T16:19:37.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.224+0000 7f02c7400640 1 -- 172.21.15.39:0/3939119597 learned_addr learned my addr 172.21.15.39:0/3939119597 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:19:37.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.224+0000 7f02cce00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bca88e9810 0x55bca88e9c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:45056/0 (socket says 172.21.15.39:45056) 2024-03-18T16:19:37.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:36 smithi196 ceph-mon[36307]: pgmap v708: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 556 KiB/s wr, 150 op/s 2024-03-18T16:19:37.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.225+0000 7f02c7400640 1 -- 172.21.15.39:0/3939119597 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bca88ea200 msgr2=0x55bca88eed00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T16:19:37.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.225+0000 7f02c7400640 1 --2- 172.21.15.39:0/3939119597 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bca88ea200 0x55bca88eed00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:37.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.225+0000 7f02c7400640 1 -- 172.21.15.39:0/3939119597 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bca88e9810 msgr2=0x55bca88e9c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:37.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.225+0000 7f02c7400640 1 --2- 172.21.15.39:0/3939119597 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bca88e9810 0x55bca88e9c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:37.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.225+0000 7f02c7400640 1 -- 172.21.15.39:0/3939119597 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bca862c7d0 con 0x55bca88ef370 2024-03-18T16:19:37.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.225+0000 7f02c7400640 1 --2- 172.21.15.39:0/3939119597 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bca88ef370 0x55bca88f1800 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f02b40d2e60 tx=0x7f02b40d7780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6b8350b8464533b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:37.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.225+0000 7f02c4c00640 1 --2- 172.21.15.39:0/1143002672 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f02ac0a2880 0x7f02ac0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:37.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.226+0000 7f02cce00640 1 --2- 172.21.15.39:0/1143002672 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f02ac0a2880 0x7f02ac0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:37.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.226+0000 7f02bd600640 1 -- 172.21.15.39:0/3939119597 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02b4097580 con 0x55bca88ef370 2024-03-18T16:19:37.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.226+0000 7f02bd600640 1 -- 172.21.15.39:0/3939119597 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02b40dc070 con 0x55bca88ef370 2024-03-18T16:19:37.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.226+0000 7f02bd600640 1 -- 172.21.15.39:0/3939119597 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02b409a730 con 0x55bca88ef370 2024-03-18T16:19:37.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.226+0000 7f02c4c00640 1 -- 172.21.15.39:0/1143002672 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f02b40ced30 con 0x7f02c8104ba0 2024-03-18T16:19:37.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.227+0000 7f02d1bd3740 1 -- 172.21.15.39:0/3939119597 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bca88ef370 msgr2=0x55bca88f1800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:37.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.227+0000 7f02d1bd3740 1 --2- 172.21.15.39:0/3939119597 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bca88ef370 0x55bca88f1800 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f02b40d2e60 tx=0x7f02b40d7780 comp rx=0 tx=0).stop 2024-03-18T16:19:37.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.227+0000 7f02d1bd3740 1 -- 172.21.15.39:0/3939119597 shutdown_connections 2024-03-18T16:19:37.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.227+0000 7f02d1bd3740 1 --2- 172.21.15.39:0/3939119597 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bca88ea200 0x55bca88eed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:37.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.227+0000 7f02d1bd3740 1 --2- 172.21.15.39:0/3939119597 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bca88ef370 0x55bca88f1800 secure :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f02b40d2e60 tx=0x7f02b40d7780 comp rx=0 tx=0).stop 2024-03-18T16:19:37.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.227+0000 7f02d1bd3740 1 --2- 172.21.15.39:0/3939119597 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bca88e9810 0x55bca88e9c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:37.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.227+0000 7f02d1bd3740 1 -- 172.21.15.39:0/3939119597 >> 172.21.15.39:0/3939119597 conn(0x55bca88e4d80 msgr2=0x55bca88e5180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:37.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.228+0000 7f02d1bd3740 1 -- 172.21.15.39:0/3939119597 shutdown_connections 2024-03-18T16:19:37.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.228+0000 7f02d1bd3740 1 -- 172.21.15.39:0/3939119597 wait complete. 2024-03-18T16:19:37.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.228+0000 7f02cce00640 1 --2- 172.21.15.39:0/1143002672 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f02ac0a2880 0x7f02ac0a4d40 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x55bca88eaa60 tx=0x7f02c00073a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:37.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.228+0000 7f02d1bd3740 1 -- start start 2024-03-18T16:19:37.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.229+0000 7f02d1bd3740 1 Processor -- start 2024-03-18T16:19:37.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.229+0000 7f02d1bd3740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bca88e9810 0x55bca88f3e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:37.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.229+0000 7f02d1bd3740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bca88ea200 0x55bca88f43c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:37.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.229+0000 7f02d1bd3740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bca88f93f0 0x55bca88fb7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:37.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.229+0000 7f02d1bd3740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55bca88f2320 con 0x55bca88f93f0 2024-03-18T16:19:37.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.229+0000 7f02d1bd3740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55bca8809ed0 con 0x55bca88e9810 2024-03-18T16:19:37.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.229+0000 7f02d1bd3740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55bca88f24a0 con 0x55bca88ea200 2024-03-18T16:19:37.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.229+0000 7f02d1bd3740 10 client.0 _ll_register_callbacks cb 0x55bca8845220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:19:37.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.230+0000 7f02c7400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bca88ea200 0x55bca88f43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:37.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.230+0000 7f02c7e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bca88e9810 0x55bca88f3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:37.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.230+0000 7f02c7400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bca88ea200 0x55bca88f43c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:46780/0 (socket says 172.21.15.39:46780) 2024-03-18T16:19:37.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.230+0000 7f02c7400640 1 -- 172.21.15.39:0/2648992279 learned_addr learned my addr 172.21.15.39:0/2648992279 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:19:37.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.230+0000 7f02cce00640 1 --2- 172.21.15.39:0/2648992279 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bca88f93f0 0x55bca88fb7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:37.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.230+0000 7f02c7400640 1 -- 172.21.15.39:0/2648992279 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bca88e9810 msgr2=0x55bca88f3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:37.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.230+0000 7f02c7400640 1 --2- 172.21.15.39:0/2648992279 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bca88e9810 0x55bca88f3e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:37.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.230+0000 7f02c7400640 1 -- 172.21.15.39:0/2648992279 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bca88f93f0 msgr2=0x55bca88fb7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:37.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.230+0000 7f02c7400640 1 --2- 172.21.15.39:0/2648992279 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bca88f93f0 0x55bca88fb7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:37.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.230+0000 7f02c7400640 1 -- 172.21.15.39:0/2648992279 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bca85623f0 con 0x55bca88ea200 2024-03-18T16:19:37.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.230+0000 7f02cce00640 1 --2- 172.21.15.39:0/2648992279 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bca88f93f0 0x55bca88fb7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:19:37.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.230+0000 7f02c7400640 1 --2- 172.21.15.39:0/2648992279 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bca88ea200 0x55bca88f43c0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f02b409b270 tx=0x7f02b4052540 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f33f5e40bd904d70 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:37.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.230+0000 7f02b2000640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:19:37.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.231+0000 7f02b2000640 1 -- 172.21.15.39:0/2648992279 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02b409aa60 con 0x55bca88ea200 2024-03-18T16:19:37.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.231+0000 7f02b2000640 1 -- 172.21.15.39:0/2648992279 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f02b40cc550 con 0x55bca88ea200 2024-03-18T16:19:37.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.231+0000 7f02d1bd3740 1 -- 172.21.15.39:0/2648992279 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f02b40dc070 con 0x55bca88ea200 2024-03-18T16:19:37.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.231+0000 7f02b2000640 1 -- 172.21.15.39:0/2648992279 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f02b409ac40 con 0x55bca88ea200 2024-03-18T16:19:37.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.231+0000 7f02d1bd3740 10 client.27466 fetch_fsmap learned FSMap version 44 2024-03-18T16:19:37.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.231+0000 7f02d1bd3740 1 -- 172.21.15.39:0/2648992279 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x55bca85220f0 con 0x55bca88ea200 2024-03-18T16:19:37.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.231+0000 7f02b2000640 1 -- 172.21.15.39:0/2648992279 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02b40dc070 con 0x55bca88ea200 2024-03-18T16:19:37.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.231+0000 7f02b2000640 1 -- 172.21.15.39:0/2648992279 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7f02b4020020 con 0x55bca88ea200 2024-03-18T16:19:37.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.232+0000 7f02d1bd3740 10 client.27466 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:19:37.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.232+0000 7f02d1bd3740 10 client.27466 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:19:37.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.232+0000 7f02d1bd3740 20 client.27466 populate_metadata read hostname 'smithi039' 2024-03-18T16:19:37.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.232+0000 7f02d1bd3740 1 --2- 172.21.15.39:0/2648992279 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55bca88ffcd0 0x55bca89020c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:37.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.232+0000 7f02d1bd3740 4 client.27466 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:19:37.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.232+0000 7f02d1bd3740 1 -- 172.21.15.39:0/2648992279 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bca8861be0 con 0x55bca88ffcd0 2024-03-18T16:19:37.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.232+0000 7f02c7e00640 1 --2- 172.21.15.39:0/2648992279 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55bca88ffcd0 0x55bca89020c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:37.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.232+0000 7f02c7e00640 1 --2- 172.21.15.39:0/2648992279 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55bca88ffcd0 0x55bca89020c0 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=94466fad6784744a server_cookie=e499d1d04ad7d2bf in_seq=0 out_seq=0 2024-03-18T16:19:37.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.232+0000 7f02b2000640 10 client.27466 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:19:37.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.233+0000 7f02b2000640 1 -- 172.21.15.39:0/2648992279 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55bca8861be0 con 0x55bca88ffcd0 2024-03-18T16:19:37.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.233+0000 7f02b2000640 10 client.27466 handle_command_reply: tid=0 2024-03-18T16:19:37.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.233+0000 7f02d1bd3740 1 client.27466 shutdown 2024-03-18T16:19:37.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.234+0000 7f02d1bd3740 1 -- 172.21.15.39:0/2648992279 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bca88ea200 msgr2=0x55bca88f43c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:37.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.234+0000 7f02d1bd3740 1 --2- 172.21.15.39:0/2648992279 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bca88ea200 0x55bca88f43c0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f02b409b270 tx=0x7f02b4052540 comp rx=0 tx=0).stop 2024-03-18T16:19:37.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.234+0000 7f02d1bd3740 1 -- 172.21.15.39:0/2648992279 shutdown_connections 2024-03-18T16:19:37.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.234+0000 7f02d1bd3740 1 --2- 172.21.15.39:0/2648992279 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55bca88ffcd0 0x55bca89020c0 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:37.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.234+0000 7f02d1bd3740 1 --2- 172.21.15.39:0/2648992279 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bca88f93f0 0x55bca88fb7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:37.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.234+0000 7f02d1bd3740 1 --2- 172.21.15.39:0/2648992279 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bca88ea200 0x55bca88f43c0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:37.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.234+0000 7f02d1bd3740 1 --2- 172.21.15.39:0/2648992279 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bca88e9810 0x55bca88f3e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:37.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.234+0000 7f02d1bd3740 1 -- 172.21.15.39:0/2648992279 >> 172.21.15.39:0/2648992279 conn(0x55bca88e4d80 msgr2=0x55bca88e3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:37.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.234+0000 7f02b2000640 0 client.27466 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:19:37.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.235+0000 7f02d1bd3740 1 -- 172.21.15.39:0/2648992279 shutdown_connections 2024-03-18T16:19:37.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.235+0000 7f02d1bd3740 1 -- 172.21.15.39:0/2648992279 wait complete. 2024-03-18T16:19:37.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.236+0000 7f02d1bd3740 20 client.27466 trim_cache size 0 max 16384 2024-03-18T16:19:37.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:36 smithi194 ceph-mon[36200]: pgmap v708: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 556 KiB/s wr, 150 op/s 2024-03-18T16:19:37.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.260+0000 7f02d1bd3740 1 Processor -- start 2024-03-18T16:19:37.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.260+0000 7f02d1bd3740 1 -- start start 2024-03-18T16:19:37.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.260+0000 7f02d1bd3740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bca88e9810 0x55bca88dfe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:37.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.261+0000 7f02d1bd3740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bca88ea200 0x55bca88f2840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:37.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.261+0000 7f02d1bd3740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bca88f93f0 0x55bca88f2d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:37.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.261+0000 7f02d1bd3740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f02c81b0980 con 0x55bca88e9810 2024-03-18T16:19:37.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.261+0000 7f02d1bd3740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f02c81b24c0 con 0x55bca88ea200 2024-03-18T16:19:37.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.261+0000 7f02d1bd3740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f02c81b1050 con 0x55bca88f93f0 2024-03-18T16:19:37.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.261+0000 7f02c7400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bca88ea200 0x55bca88f2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:37.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.261+0000 7f02cce00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bca88f93f0 0x55bca88f2d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:37.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.261+0000 7f02c7400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bca88ea200 0x55bca88f2840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:51284/0 (socket says 172.21.15.39:51284) 2024-03-18T16:19:37.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.261+0000 7f02c7400640 1 -- 172.21.15.39:0/4149622215 learned_addr learned my addr 172.21.15.39:0/4149622215 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:19:37.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.261+0000 7f02c7e00640 1 --2- 172.21.15.39:0/4149622215 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bca88e9810 0x55bca88dfe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:37.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.261+0000 7f02c7400640 1 -- 172.21.15.39:0/4149622215 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bca88f93f0 msgr2=0x55bca88f2d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:37.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.261+0000 7f02c7400640 1 --2- 172.21.15.39:0/4149622215 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bca88f93f0 0x55bca88f2d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:37.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.261+0000 7f02c7400640 1 -- 172.21.15.39:0/4149622215 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bca88e9810 msgr2=0x55bca88dfe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:37.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.261+0000 7f02c7400640 1 --2- 172.21.15.39:0/4149622215 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bca88e9810 0x55bca88dfe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:37.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.261+0000 7f02c7400640 1 -- 172.21.15.39:0/4149622215 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bca88df370 con 0x55bca88ea200 2024-03-18T16:19:37.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.262+0000 7f02c7400640 1 --2- 172.21.15.39:0/4149622215 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bca88ea200 0x55bca88f2840 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f02b409afe0 tx=0x7f02b40d7780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a0b3936b585b0ff5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:37.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.262+0000 7f02bcc00640 1 -- 172.21.15.39:0/4149622215 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02b409ae20 con 0x55bca88ea200 2024-03-18T16:19:37.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.262+0000 7f02bcc00640 1 -- 172.21.15.39:0/4149622215 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02b40dc030 con 0x55bca88ea200 2024-03-18T16:19:37.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.262+0000 7f02bcc00640 1 -- 172.21.15.39:0/4149622215 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02b409b0b0 con 0x55bca88ea200 2024-03-18T16:19:37.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.263+0000 7f02d1bd3740 1 -- 172.21.15.39:0/4149622215 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bca88ea200 msgr2=0x55bca88f2840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:37.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.263+0000 7f02d1bd3740 1 --2- 172.21.15.39:0/4149622215 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bca88ea200 0x55bca88f2840 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f02b409afe0 tx=0x7f02b40d7780 comp rx=0 tx=0).stop 2024-03-18T16:19:37.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.263+0000 7f02d1bd3740 1 -- 172.21.15.39:0/4149622215 shutdown_connections 2024-03-18T16:19:37.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.263+0000 7f02d1bd3740 1 --2- 172.21.15.39:0/4149622215 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bca88ea200 0x55bca88f2840 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:37.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.263+0000 7f02d1bd3740 1 --2- 172.21.15.39:0/4149622215 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bca88f93f0 0x55bca88f2d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:37.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.263+0000 7f02d1bd3740 1 --2- 172.21.15.39:0/4149622215 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bca88e9810 0x55bca88dfe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:37.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.263+0000 7f02d1bd3740 1 -- 172.21.15.39:0/4149622215 >> 172.21.15.39:0/4149622215 conn(0x55bca88e4d80 msgr2=0x55bca88eb840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:37.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.263+0000 7f02d1bd3740 1 -- 172.21.15.39:0/4149622215 shutdown_connections 2024-03-18T16:19:37.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.264+0000 7f02d1bd3740 1 -- 172.21.15.39:0/4149622215 wait complete. 2024-03-18T16:19:37.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.264+0000 7f02d1bd3740 1 -- start start 2024-03-18T16:19:37.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.264+0000 7f02d1bd3740 1 Processor -- start 2024-03-18T16:19:37.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.265+0000 7f02d1bd3740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bca88e9810 0x55bca8910710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:37.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.265+0000 7f02d1bd3740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bca88ea200 0x55bca8910c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:37.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.265+0000 7f02d1bd3740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bca88f93f0 0x55bca8911c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:37.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.265+0000 7f02d1bd3740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f02c81b1df0 con 0x55bca88f93f0 2024-03-18T16:19:37.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.265+0000 7f02d1bd3740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f02c81b1720 con 0x55bca88ea200 2024-03-18T16:19:37.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.265+0000 7f02d1bd3740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f02c81ae960 con 0x55bca88e9810 2024-03-18T16:19:37.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.265+0000 7f02d1bd3740 10 client.0 _ll_register_callbacks cb 0x7f02b40d2cb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:19:37.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.265+0000 7f02cce00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bca88f93f0 0x55bca8911c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:37.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.265+0000 7f02cce00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bca88f93f0 0x55bca8911c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:45084/0 (socket says 172.21.15.39:45084) 2024-03-18T16:19:37.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.266+0000 7f02cce00640 1 -- 172.21.15.39:0/1472054042 learned_addr learned my addr 172.21.15.39:0/1472054042 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:19:37.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.266+0000 7f02c7e00640 1 --2- 172.21.15.39:0/1472054042 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bca88e9810 0x55bca8910710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:37.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.266+0000 7f02cce00640 1 -- 172.21.15.39:0/1472054042 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bca88e9810 msgr2=0x55bca8910710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:37.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.266+0000 7f02cce00640 1 --2- 172.21.15.39:0/1472054042 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bca88e9810 0x55bca8910710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:37.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.266+0000 7f02cce00640 1 -- 172.21.15.39:0/1472054042 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bca88ea200 msgr2=0x55bca8910c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:37.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.266+0000 7f02c7400640 1 --2- 172.21.15.39:0/1472054042 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bca88ea200 0x55bca8910c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:37.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.266+0000 7f02cce00640 1 --2- 172.21.15.39:0/1472054042 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bca88ea200 0x55bca8910c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:37.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.266+0000 7f02cce00640 1 -- 172.21.15.39:0/1472054042 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bca8912250 con 0x55bca88f93f0 2024-03-18T16:19:37.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.266+0000 7f02c7400640 1 --2- 172.21.15.39:0/1472054042 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bca88ea200 0x55bca8910c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:19:37.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.267+0000 7f02cce00640 1 --2- 172.21.15.39:0/1472054042 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bca88f93f0 0x55bca8911c60 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f02c001ee00 tx=0x7f02c0005e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13baa566b697daaa server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:37.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.267+0000 7f02b2000640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-03-18T16:19:37.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.268+0000 7f02b2000640 1 -- 172.21.15.39:0/1472054042 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02c0027640 con 0x55bca88f93f0 2024-03-18T16:19:37.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.268+0000 7f02d1bd3740 1 -- 172.21.15.39:0/1472054042 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f02b40dc030 con 0x55bca88f93f0 2024-03-18T16:19:37.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.268+0000 7f02b2000640 1 -- 172.21.15.39:0/1472054042 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f02c00263e0 con 0x55bca88f93f0 2024-03-18T16:19:37.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.269+0000 7f02b2000640 1 -- 172.21.15.39:0/1472054042 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f02c00277e0 con 0x55bca88f93f0 2024-03-18T16:19:37.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.269+0000 7f02d1bd3740 10 client.16851 fetch_fsmap learned FSMap version 44 2024-03-18T16:19:37.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.269+0000 7f02d1bd3740 1 -- 172.21.15.39:0/1472054042 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x55bca8965940 con 0x55bca88f93f0 2024-03-18T16:19:37.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.270+0000 7f02b2000640 1 -- 172.21.15.39:0/1472054042 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02b40dc030 con 0x55bca88f93f0 2024-03-18T16:19:37.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.270+0000 7f02b2000640 1 -- 172.21.15.39:0/1472054042 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7f02c0025070 con 0x55bca88f93f0 2024-03-18T16:19:37.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.270+0000 7f02d1bd3740 10 client.16851 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:19:37.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.270+0000 7f02d1bd3740 10 client.16851 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:19:37.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.270+0000 7f02d1bd3740 20 client.16851 populate_metadata read hostname 'smithi039' 2024-03-18T16:19:37.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.270+0000 7f02d1bd3740 1 --2- 172.21.15.39:0/1472054042 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55bca8966340 0x55bca8968760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:37.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.270+0000 7f02d1bd3740 4 client.16851 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T16:19:37.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.271+0000 7f02d1bd3740 1 -- 172.21.15.39:0/1472054042 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55bca885b730 con 0x55bca8966340 2024-03-18T16:19:37.275 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.271+0000 7f02c7e00640 1 --2- 172.21.15.39:0/1472054042 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55bca8966340 0x55bca8968760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:37.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.272+0000 7f02c7e00640 1 --2- 172.21.15.39:0/1472054042 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55bca8966340 0x55bca8968760 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9cacc1d13120a750 server_cookie=dbb835f046dd1ae3 in_seq=0 out_seq=0 2024-03-18T16:19:37.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.272+0000 7f02b2000640 10 client.16851 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:19:37.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.272+0000 7f02b2000640 1 -- 172.21.15.39:0/1472054042 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55bca885b730 con 0x55bca8966340 2024-03-18T16:19:37.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.272+0000 7f02b2000640 10 client.16851 handle_command_reply: tid=0 2024-03-18T16:19:37.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.272+0000 7f02d1bd3740 1 client.16851 shutdown 2024-03-18T16:19:37.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.273+0000 7f02d1bd3740 1 -- 172.21.15.39:0/1472054042 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bca88f93f0 msgr2=0x55bca8911c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:37.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.273+0000 7f02d1bd3740 1 --2- 172.21.15.39:0/1472054042 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bca88f93f0 0x55bca8911c60 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f02c001ee00 tx=0x7f02c0005e90 comp rx=0 tx=0).stop 2024-03-18T16:19:37.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.273+0000 7f02d1bd3740 1 -- 172.21.15.39:0/1472054042 shutdown_connections 2024-03-18T16:19:37.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.273+0000 7f02d1bd3740 1 --2- 172.21.15.39:0/1472054042 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55bca8966340 0x55bca8968760 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:37.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.273+0000 7f02d1bd3740 1 --2- 172.21.15.39:0/1472054042 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bca88ea200 0x55bca8910c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:37.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.273+0000 7f02d1bd3740 1 --2- 172.21.15.39:0/1472054042 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bca88f93f0 0x55bca8911c60 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:37.276 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.273+0000 7f02d1bd3740 1 --2- 172.21.15.39:0/1472054042 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bca88e9810 0x55bca8910710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:37.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.273+0000 7f02d1bd3740 1 -- 172.21.15.39:0/1472054042 >> 172.21.15.39:0/1472054042 conn(0x55bca88e4d80 msgr2=0x55bca88f2840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:37.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.273+0000 7f02b2000640 0 client.16851 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:19:37.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.274+0000 7f02d1bd3740 1 -- 172.21.15.39:0/1472054042 shutdown_connections 2024-03-18T16:19:37.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.274+0000 7f02d1bd3740 1 -- 172.21.15.39:0/1472054042 wait complete. 2024-03-18T16:19:37.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.274+0000 7f02d1bd3740 20 client.16851 trim_cache size 0 max 16384 2024-03-18T16:19:37.278 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-03-18T16:19:37.278 INFO:teuthology.orchestra.run.smithi039.stdout: "status": "no active scrubs running", 2024-03-18T16:19:37.278 INFO:teuthology.orchestra.run.smithi039.stdout: "scrubs": {} 2024-03-18T16:19:37.278 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-03-18T16:19:37.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.278+0000 7f02cee00640 1 -- 172.21.15.39:0/1143002672 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f02ac0a2880 msgr2=0x7f02ac0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:37.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.278+0000 7f02cee00640 1 --2- 172.21.15.39:0/1143002672 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f02ac0a2880 0x7f02ac0a4d40 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x55bca88eaa60 tx=0x7f02c00073a0 comp rx=0 tx=0).stop 2024-03-18T16:19:37.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.278+0000 7f02cee00640 1 -- 172.21.15.39:0/1143002672 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f02c8104ba0 msgr2=0x7f02c8100220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:37.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.278+0000 7f02cee00640 1 --2- 172.21.15.39:0/1143002672 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f02c8104ba0 0x7f02c8100220 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f02b400c940 tx=0x7f02b400ce00 comp rx=0 tx=0).stop 2024-03-18T16:19:37.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.278+0000 7f02cee00640 1 -- 172.21.15.39:0/1143002672 shutdown_connections 2024-03-18T16:19:37.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.278+0000 7f02cee00640 1 --2- 172.21.15.39:0/1143002672 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f02ac0a2880 0x7f02ac0a4d40 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:37.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.278+0000 7f02cee00640 1 --2- 172.21.15.39:0/1143002672 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f02c81074d0 0x7f02c8100760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:37.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.278+0000 7f02cee00640 1 --2- 172.21.15.39:0/1143002672 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f02c8104ba0 0x7f02c8100220 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:37.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.278+0000 7f02cee00640 1 --2- 172.21.15.39:0/1143002672 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f02c8102270 0x7f02c8101be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:37.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.278+0000 7f02cee00640 1 -- 172.21.15.39:0/1143002672 >> 172.21.15.39:0/1143002672 conn(0x7f02c80fbc80 msgr2=0x7f02c8102bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:37.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.278+0000 7f02cee00640 1 -- 172.21.15.39:0/1143002672 shutdown_connections 2024-03-18T16:19:37.282 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.279+0000 7f02cee00640 1 -- 172.21.15.39:0/1143002672 wait complete. 2024-03-18T16:19:37.300 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:36 smithi039 ceph-mon[31721]: pgmap v708: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 556 KiB/s wr, 150 op/s 2024-03-18T16:19:37.303 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-18T16:19:37.304 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:19:37.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.544+0000 7f49b5000640 1 -- 172.21.15.39:0/2941090235 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f49b0077d80 msgr2=0x7f49b0078620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:37.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.544+0000 7f49b5000640 1 --2- 172.21.15.39:0/2941090235 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f49b0077d80 0x7f49b0078620 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f49a8009be0 tx=0x7f49a802bef0 comp rx=0 tx=0).stop 2024-03-18T16:19:37.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.544+0000 7f49b5000640 1 -- 172.21.15.39:0/2941090235 shutdown_connections 2024-03-18T16:19:37.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.544+0000 7f49b5000640 1 --2- 172.21.15.39:0/2941090235 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f49b0077d80 0x7f49b0078620 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:37.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.544+0000 7f49b5000640 1 --2- 172.21.15.39:0/2941090235 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49b00773e0 0x7f49b0077840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:37.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.544+0000 7f49b5000640 1 --2- 172.21.15.39:0/2941090235 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f49b00761a0 0x7f49b00765c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:37.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.544+0000 7f49b5000640 1 -- 172.21.15.39:0/2941090235 >> 172.21.15.39:0/2941090235 conn(0x7f49b00fef90 msgr2=0x7f49b01013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:37.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.545+0000 7f49b5000640 1 -- 172.21.15.39:0/2941090235 shutdown_connections 2024-03-18T16:19:37.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.545+0000 7f49b5000640 1 -- 172.21.15.39:0/2941090235 wait complete. 2024-03-18T16:19:37.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.546+0000 7f49b5000640 1 Processor -- start 2024-03-18T16:19:37.549 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.547+0000 7f49b5000640 1 -- start start 2024-03-18T16:19:37.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.547+0000 7f49b5000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f49b00761a0 0x7f49b019dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:37.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.547+0000 7f49b5000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f49b00773e0 0x7f49b019e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:37.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.547+0000 7f49b5000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49b0077d80 0x7f49b01a5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:37.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.547+0000 7f49b5000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f49b0113ad0 con 0x7f49b0077d80 2024-03-18T16:19:37.550 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.547+0000 7f49b5000640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f49b0113950 con 0x7f49b00773e0 2024-03-18T16:19:37.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.547+0000 7f49b5000640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f49b0113c50 con 0x7f49b00761a0 2024-03-18T16:19:37.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.547+0000 7f49ad400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f49b00773e0 0x7f49b019e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:37.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.547+0000 7f49ade00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f49b00761a0 0x7f49b019dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:37.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.547+0000 7f49ad400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f49b00773e0 0x7f49b019e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:51308/0 (socket says 172.21.15.39:51308) 2024-03-18T16:19:37.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.547+0000 7f49ad400640 1 -- 172.21.15.39:0/1287244007 learned_addr learned my addr 172.21.15.39:0/1287244007 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:19:37.551 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.547+0000 7f49ae800640 1 --2- 172.21.15.39:0/1287244007 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49b0077d80 0x7f49b01a5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:37.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.548+0000 7f49ade00640 1 -- 172.21.15.39:0/1287244007 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f49b00773e0 msgr2=0x7f49b019e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:37.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.548+0000 7f49ade00640 1 --2- 172.21.15.39:0/1287244007 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f49b00773e0 0x7f49b019e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:37.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.548+0000 7f49ade00640 1 -- 172.21.15.39:0/1287244007 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49b0077d80 msgr2=0x7f49b01a5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:37.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.548+0000 7f49ade00640 1 --2- 172.21.15.39:0/1287244007 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49b0077d80 0x7f49b01a5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:37.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.548+0000 7f49ade00640 1 -- 172.21.15.39:0/1287244007 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49b01a5bd0 con 0x7f49b00761a0 2024-03-18T16:19:37.552 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.548+0000 7f49ad400640 1 --2- 172.21.15.39:0/1287244007 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f49b00773e0 0x7f49b019e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:19:37.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.548+0000 7f49ade00640 1 --2- 172.21.15.39:0/1287244007 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f49b00761a0 0x7f49b019dfb0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f49a000e790 tx=0x7f49a000ec50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:37.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.549+0000 7f49a6a00640 1 -- 172.21.15.39:0/1287244007 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49a000eea0 con 0x7f49b00761a0 2024-03-18T16:19:37.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.549+0000 7f49a6a00640 1 -- 172.21.15.39:0/1287244007 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f49a000ccc0 con 0x7f49b00761a0 2024-03-18T16:19:37.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.549+0000 7f49a6a00640 1 -- 172.21.15.39:0/1287244007 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49a0018780 con 0x7f49b00761a0 2024-03-18T16:19:37.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.549+0000 7f49b5000640 1 -- 172.21.15.39:0/1287244007 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49b01a5ec0 con 0x7f49b00761a0 2024-03-18T16:19:37.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.549+0000 7f49b5000640 1 -- 172.21.15.39:0/1287244007 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49b01a6270 con 0x7f49b00761a0 2024-03-18T16:19:37.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.550+0000 7f49b5000640 1 -- 172.21.15.39:0/1287244007 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49b01103e0 con 0x7f49b00761a0 2024-03-18T16:19:37.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.554+0000 7f49a6a00640 1 -- 172.21.15.39:0/1287244007 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f49a0047a60 con 0x7f49b00761a0 2024-03-18T16:19:37.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.556+0000 7f49a6a00640 1 --2- 172.21.15.39:0/1287244007 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f49940a2880 0x7f49940a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:37.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.556+0000 7f49a6a00640 1 -- 172.21.15.39:0/1287244007 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f49a00ceea0 con 0x7f49b00761a0 2024-03-18T16:19:37.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.556+0000 7f49ad400640 1 --2- 172.21.15.39:0/1287244007 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f49940a2880 0x7f49940a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:37.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.557+0000 7f49ad400640 1 --2- 172.21.15.39:0/1287244007 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f49940a2880 0x7f49940a4d40 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f49b019f4c0 tx=0x7f499c00a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:37.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.558+0000 7f49a6a00640 1 -- 172.21.15.39:0/1287244007 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f49a009b560 con 0x7f49b00761a0 2024-03-18T16:19:37.915 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.913+0000 7f49b5000640 1 -- 172.21.15.39:0/1287244007 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f49b002d1b0 con 0x7f49b00761a0 2024-03-18T16:19:37.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.914+0000 7f49a6a00640 1 -- 172.21.15.39:0/1287244007 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) ==== 94+0+11932 (secure 0 0 0) 0x7f49a009f600 con 0x7f49b00761a0 2024-03-18T16:19:37.917 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:19:37.917 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":44,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2520226608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2520226608},{"type":"v1","addr":"172.21.15.39:6835","nonce":2520226608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/719341763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":719341763},{"type":"v1","addr":"172.21.15.39:6837","nonce":719341763}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":44,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T16:06:44.180425+0000","modified":"2024-03-18T16:15:58.268360+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24575,"mds_1":24556,"mds_2":24535,"mds_3":24524,"mds_4":14463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14445":{"gid":14445,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.39:6839/656513813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":656513813},{"type":"v1","addr":"172.21.15.39:6839","nonce":656513813}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14463":{"gid":14463,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":13,"addr":"172.21.15.39:6841/3515326042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3515326042},{"type":"v1","addr":"172.21.15.39:6841","nonce":3515326042}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6833/931350738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":931350738},{"type":"v1","addr":"172.21.15.194:6833","nonce":931350738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6833/3874209621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":3874209621},{"type":"v1","addr":"172.21.15.196:6833","nonce":3874209621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6835/188385470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":188385470},{"type":"v1","addr":"172.21.15.194:6835","nonce":188385470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6835/3949984309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":3949984309},{"type":"v1","addr":"172.21.15.196:6835","nonce":3949984309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":15,"addr":"172.21.15.194:6837/1301464246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":1301464246},{"type":"v1","addr":"172.21.15.194:6837","nonce":1301464246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":16,"addr":"172.21.15.196:6837/895247251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":895247251},{"type":"v1","addr":"172.21.15.196:6837","nonce":895247251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.196:6839/2001427268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6838","nonce":2001427268},{"type":"v1","addr":"172.21.15.196:6839","nonce":2001427268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24575":{"gid":24575,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.194:6839/274911158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":274911158},{"type":"v1","addr":"172.21.15.194:6839","nonce":274911158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24575,"qdb_cluster":[14463,24524,24575,24556,24535]},"id":1}]} 2024-03-18T16:19:37.917 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 44 2024-03-18T16:19:37.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.916+0000 7f49b5000640 1 -- 172.21.15.39:0/1287244007 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f49940a2880 msgr2=0x7f49940a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:37.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.916+0000 7f49b5000640 1 --2- 172.21.15.39:0/1287244007 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f49940a2880 0x7f49940a4d40 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f49b019f4c0 tx=0x7f499c00a380 comp rx=0 tx=0).stop 2024-03-18T16:19:37.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.916+0000 7f49b5000640 1 -- 172.21.15.39:0/1287244007 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f49b00761a0 msgr2=0x7f49b019dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:37.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.916+0000 7f49b5000640 1 --2- 172.21.15.39:0/1287244007 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f49b00761a0 0x7f49b019dfb0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f49a000e790 tx=0x7f49a000ec50 comp rx=0 tx=0).stop 2024-03-18T16:19:37.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.917+0000 7f49b5000640 1 -- 172.21.15.39:0/1287244007 shutdown_connections 2024-03-18T16:19:37.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.917+0000 7f49b5000640 1 --2- 172.21.15.39:0/1287244007 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f49b00773e0 0x7f49b019e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:37.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.917+0000 7f49b5000640 1 --2- 172.21.15.39:0/1287244007 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f49940a2880 0x7f49940a4d40 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:37.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.917+0000 7f49b5000640 1 --2- 172.21.15.39:0/1287244007 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f49b0077d80 0x7f49b01a5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:37.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.917+0000 7f49b5000640 1 --2- 172.21.15.39:0/1287244007 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f49b00761a0 0x7f49b019dfb0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:37.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.917+0000 7f49b5000640 1 -- 172.21.15.39:0/1287244007 >> 172.21.15.39:0/1287244007 conn(0x7f49b00fef90 msgr2=0x7f49b0100520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:37.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.917+0000 7f49b5000640 1 -- 172.21.15.39:0/1287244007 shutdown_connections 2024-03-18T16:19:37.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:37.917+0000 7f49b5000640 1 -- 172.21.15.39:0/1287244007 wait complete. 2024-03-18T16:19:37.942 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-18T16:19:38.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.172+0000 7fa64fa00640 1 -- 172.21.15.39:0/631975361 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa648103390 msgr2=0x7fa648109c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:38.175 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.172+0000 7fa64fa00640 1 --2- 172.21.15.39:0/631975361 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa648103390 0x7fa648109c20 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fa640009ca0 tx=0x7fa64002c100 comp rx=0 tx=0).stop 2024-03-18T16:19:38.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.174+0000 7fa64fa00640 1 -- 172.21.15.39:0/631975361 shutdown_connections 2024-03-18T16:19:38.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.174+0000 7fa64fa00640 1 --2- 172.21.15.39:0/631975361 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa648103390 0x7fa648109c20 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.174+0000 7fa64fa00640 1 --2- 172.21.15.39:0/631975361 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa6481029d0 0x7fa648102e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.174+0000 7fa64fa00640 1 --2- 172.21.15.39:0/631975361 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa6481017e0 0x7fa648101c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.176 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.174+0000 7fa64fa00640 1 -- 172.21.15.39:0/631975361 >> 172.21.15.39:0/631975361 conn(0x7fa6480fcd50 msgr2=0x7fa6480ff190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:38.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.175+0000 7fa64fa00640 1 -- 172.21.15.39:0/631975361 shutdown_connections 2024-03-18T16:19:38.177 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.175+0000 7fa64fa00640 1 -- 172.21.15.39:0/631975361 wait complete. 2024-03-18T16:19:38.178 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.176+0000 7fa64fa00640 1 Processor -- start 2024-03-18T16:19:38.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.176+0000 7fa64fa00640 1 -- start start 2024-03-18T16:19:38.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.177+0000 7fa64fa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa6481017e0 0x7fa648199ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:38.179 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.177+0000 7fa64fa00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa6481029d0 0x7fa64819a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:38.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.177+0000 7fa64fa00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa648103390 0x7fa6481a1240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:38.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.177+0000 7fa64fa00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fa64810d440 con 0x7fa6481017e0 2024-03-18T16:19:38.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.177+0000 7fa64fa00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fa64810d2c0 con 0x7fa648103390 2024-03-18T16:19:38.180 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.177+0000 7fa64fa00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fa64810d5c0 con 0x7fa6481029d0 2024-03-18T16:19:38.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.177+0000 7fa647e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa6481029d0 0x7fa64819a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:38.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.177+0000 7fa647e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa6481029d0 0x7fa64819a1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:53520/0 (socket says 172.21.15.39:53520) 2024-03-18T16:19:38.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.177+0000 7fa647e00640 1 -- 172.21.15.39:0/3772091101 learned_addr learned my addr 172.21.15.39:0/3772091101 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:19:38.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.178+0000 7fa64da00640 1 --2- 172.21.15.39:0/3772091101 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa648103390 0x7fa6481a1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:38.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.178+0000 7fa64d000640 1 --2- 172.21.15.39:0/3772091101 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa6481017e0 0x7fa648199ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:38.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.178+0000 7fa647e00640 1 -- 172.21.15.39:0/3772091101 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa648103390 msgr2=0x7fa6481a1240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:38.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.178+0000 7fa647e00640 1 --2- 172.21.15.39:0/3772091101 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa648103390 0x7fa6481a1240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.181 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.178+0000 7fa647e00640 1 -- 172.21.15.39:0/3772091101 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa6481017e0 msgr2=0x7fa648199ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:38.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.178+0000 7fa647e00640 1 --2- 172.21.15.39:0/3772091101 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa6481017e0 0x7fa648199ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.178+0000 7fa647e00640 1 -- 172.21.15.39:0/3772091101 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6481a18e0 con 0x7fa6481029d0 2024-03-18T16:19:38.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.178+0000 7fa64d000640 1 --2- 172.21.15.39:0/3772091101 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa6481017e0 0x7fa648199ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:19:38.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.178+0000 7fa647e00640 1 --2- 172.21.15.39:0/3772091101 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa6481029d0 0x7fa64819a1e0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fa63400e9f0 tx=0x7fa63400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:38.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.179+0000 7fa645600640 1 -- 172.21.15.39:0/3772091101 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa63400cd80 con 0x7fa6481029d0 2024-03-18T16:19:38.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.179+0000 7fa645600640 1 -- 172.21.15.39:0/3772091101 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa634010910 con 0x7fa6481029d0 2024-03-18T16:19:38.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.179+0000 7fa645600640 1 -- 172.21.15.39:0/3772091101 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6340188f0 con 0x7fa6481029d0 2024-03-18T16:19:38.182 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.179+0000 7fa64fa00640 1 -- 172.21.15.39:0/3772091101 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6481a1bd0 con 0x7fa6481029d0 2024-03-18T16:19:38.183 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.179+0000 7fa64fa00640 1 -- 172.21.15.39:0/3772091101 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6481a20d0 con 0x7fa6481029d0 2024-03-18T16:19:38.186 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.183+0000 7fa6527cb740 1 Processor -- start 2024-03-18T16:19:38.186 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.184+0000 7fa645600640 1 -- 172.21.15.39:0/3772091101 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fa634010ac0 con 0x7fa6481029d0 2024-03-18T16:19:38.186 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.184+0000 7fa6527cb740 1 -- start start 2024-03-18T16:19:38.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.184+0000 7fa6527cb740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55722e4918c0 0x55722e491ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:38.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.184+0000 7fa6527cb740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55722e4922b0 0x55722e496db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:38.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.184+0000 7fa6527cb740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55722e497420 0x55722e4998b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:38.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.184+0000 7fa6527cb740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55722e4593b0 con 0x55722e4918c0 2024-03-18T16:19:38.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.184+0000 7fa6527cb740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55722e3f6450 con 0x55722e4922b0 2024-03-18T16:19:38.187 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.184+0000 7fa6527cb740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55722e39ecb0 con 0x55722e497420 2024-03-18T16:19:38.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.184+0000 7fa64d000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55722e4922b0 0x55722e496db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:38.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.184+0000 7fa647e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55722e497420 0x55722e4998b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:38.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.185+0000 7fa64d000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55722e4922b0 0x55722e496db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:53250/0 (socket says 172.21.15.39:53250) 2024-03-18T16:19:38.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.185+0000 7fa64d000640 1 -- 172.21.15.39:0/1755801053 learned_addr learned my addr 172.21.15.39:0/1755801053 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:19:38.188 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.185+0000 7fa647e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55722e497420 0x55722e4998b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:53524/0 (socket says 172.21.15.39:53524) 2024-03-18T16:19:38.189 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.185+0000 7fa64da00640 1 --2- 172.21.15.39:0/1755801053 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55722e4918c0 0x55722e491ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:38.189 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.185+0000 7fa64d000640 1 -- 172.21.15.39:0/1755801053 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55722e497420 msgr2=0x55722e4998b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:38.189 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.185+0000 7fa64d000640 1 --2- 172.21.15.39:0/1755801053 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55722e497420 0x55722e4998b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.189 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.185+0000 7fa64d000640 1 -- 172.21.15.39:0/1755801053 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55722e4918c0 msgr2=0x55722e491ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:38.189 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.185+0000 7fa64d000640 1 --2- 172.21.15.39:0/1755801053 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55722e4918c0 0x55722e491ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.189 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.185+0000 7fa64d000640 1 -- 172.21.15.39:0/1755801053 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55722e3c01f0 con 0x55722e4922b0 2024-03-18T16:19:38.189 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.185+0000 7fa64da00640 1 --2- 172.21.15.39:0/1755801053 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55722e4918c0 0x55722e491ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:19:38.189 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.185+0000 7fa64d000640 1 --2- 172.21.15.39:0/1755801053 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55722e4922b0 0x55722e496db0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fa63c0097c0 tx=0x7fa63c009c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=38e1e9278b9d5990 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:38.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.186+0000 7fa645600640 1 --2- 172.21.15.39:0/3772091101 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa6300a2880 0x7fa6300a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:38.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.186+0000 7fa63a000640 1 -- 172.21.15.39:0/1755801053 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa63c00e8d0 con 0x55722e4922b0 2024-03-18T16:19:38.190 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.186+0000 7fa63a000640 1 -- 172.21.15.39:0/1755801053 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa63c005020 con 0x55722e4922b0 2024-03-18T16:19:38.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.186+0000 7fa645600640 1 -- 172.21.15.39:0/3772091101 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fa6340cf180 con 0x7fa6481029d0 2024-03-18T16:19:38.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.187+0000 7fa6527cb740 1 -- 172.21.15.39:0/1755801053 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55722e4922b0 msgr2=0x55722e496db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:38.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.187+0000 7fa6527cb740 1 --2- 172.21.15.39:0/1755801053 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55722e4922b0 0x55722e496db0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fa63c0097c0 tx=0x7fa63c009c80 comp rx=0 tx=0).stop 2024-03-18T16:19:38.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.187+0000 7fa6527cb740 1 -- 172.21.15.39:0/1755801053 shutdown_connections 2024-03-18T16:19:38.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.187+0000 7fa6527cb740 1 --2- 172.21.15.39:0/1755801053 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55722e4922b0 0x55722e496db0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.187+0000 7fa6527cb740 1 --2- 172.21.15.39:0/1755801053 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55722e497420 0x55722e4998b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.187+0000 7fa6527cb740 1 --2- 172.21.15.39:0/1755801053 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55722e4918c0 0x55722e491ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.187+0000 7fa6527cb740 1 -- 172.21.15.39:0/1755801053 >> 172.21.15.39:0/1755801053 conn(0x55722e48ce30 msgr2=0x55722e48d230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:38.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.187+0000 7fa6527cb740 1 -- 172.21.15.39:0/1755801053 shutdown_connections 2024-03-18T16:19:38.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.187+0000 7fa6527cb740 1 -- 172.21.15.39:0/1755801053 wait complete. 2024-03-18T16:19:38.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.187+0000 7fa64da00640 1 --2- 172.21.15.39:0/3772091101 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa6300a2880 0x7fa6300a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:38.191 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.188+0000 7fa6527cb740 1 -- start start 2024-03-18T16:19:38.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.188+0000 7fa6527cb740 1 Processor -- start 2024-03-18T16:19:38.193 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.189+0000 7fa6527cb740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55722e4918c0 0x55722e49bb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:38.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.189+0000 7fa6527cb740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55722e4922b0 0x55722e49c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:38.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.189+0000 7fa6527cb740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55722e497420 0x55722e4a3110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:38.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.189+0000 7fa6527cb740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55722e3b2110 con 0x55722e497420 2024-03-18T16:19:38.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.189+0000 7fa6527cb740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55722e3f24f0 con 0x55722e4918c0 2024-03-18T16:19:38.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.189+0000 7fa6527cb740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55722e49a470 con 0x55722e4922b0 2024-03-18T16:19:38.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.189+0000 7fa6527cb740 10 client.0 _ll_register_callbacks cb 0x55722e39fb70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:19:38.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.189+0000 7fa647e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55722e4922b0 0x55722e49c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:38.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.189+0000 7fa64da00640 1 --2- 172.21.15.39:0/3772091101 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa6300a2880 0x7fa6300a4d40 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fa640005e30 tx=0x7fa64002e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:38.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.190+0000 7fa647e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55722e4922b0 0x55722e49c0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:53526/0 (socket says 172.21.15.39:53526) 2024-03-18T16:19:38.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.190+0000 7fa647e00640 1 -- 172.21.15.39:0/465852463 learned_addr learned my addr 172.21.15.39:0/465852463 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:19:38.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.190+0000 7fa64d000640 1 --2- 172.21.15.39:0/465852463 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55722e4918c0 0x55722e49bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:38.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.190+0000 7fa64da00640 1 --2- 172.21.15.39:0/465852463 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55722e497420 0x55722e4a3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:38.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.190+0000 7fa647e00640 1 -- 172.21.15.39:0/465852463 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55722e4918c0 msgr2=0x55722e49bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:38.194 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.190+0000 7fa647e00640 1 --2- 172.21.15.39:0/465852463 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55722e4918c0 0x55722e49bb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.190+0000 7fa647e00640 1 -- 172.21.15.39:0/465852463 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55722e497420 msgr2=0x55722e4a3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:38.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.190+0000 7fa647e00640 1 --2- 172.21.15.39:0/465852463 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55722e497420 0x55722e4a3110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.190+0000 7fa647e00640 1 -- 172.21.15.39:0/465852463 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55722e10a3f0 con 0x55722e4922b0 2024-03-18T16:19:38.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.190+0000 7fa64da00640 1 --2- 172.21.15.39:0/465852463 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55722e497420 0x55722e4a3110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:19:38.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.190+0000 7fa647e00640 1 --2- 172.21.15.39:0/465852463 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55722e4922b0 0x55722e49c0d0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fa6340a39a0 tx=0x7fa634009700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a46ba44a1c32113c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:38.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.190+0000 7fa62ea00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:19:38.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.191+0000 7fa62ea00640 1 -- 172.21.15.39:0/465852463 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa63409b090 con 0x55722e4922b0 2024-03-18T16:19:38.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.191+0000 7fa62ea00640 1 -- 172.21.15.39:0/465852463 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fa6340d2cb0 con 0x55722e4922b0 2024-03-18T16:19:38.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.191+0000 7fa6527cb740 1 -- 172.21.15.39:0/465852463 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa63c005020 con 0x55722e4922b0 2024-03-18T16:19:38.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.191+0000 7fa62ea00640 1 -- 172.21.15.39:0/465852463 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fa63409be30 con 0x55722e4922b0 2024-03-18T16:19:38.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.191+0000 7fa6527cb740 10 client.27490 fetch_fsmap learned FSMap version 44 2024-03-18T16:19:38.195 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.191+0000 7fa6527cb740 1 -- 172.21.15.39:0/465852463 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x55722e1d47d0 con 0x55722e4922b0 2024-03-18T16:19:38.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.191+0000 7fa62ea00640 1 -- 172.21.15.39:0/465852463 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa63c005020 con 0x55722e4922b0 2024-03-18T16:19:38.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.191+0000 7fa62ea00640 1 -- 172.21.15.39:0/465852463 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7fa63409ab80 con 0x55722e4922b0 2024-03-18T16:19:38.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.192+0000 7fa6527cb740 10 client.27490 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:19:38.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.192+0000 7fa6527cb740 10 client.27490 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-03-18T16:19:38.196 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.192+0000 7fa6527cb740 20 client.27490 populate_metadata read hostname 'smithi039' 2024-03-18T16:19:38.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.192+0000 7fa6527cb740 1 --2- 172.21.15.39:0/465852463 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x55722e4a7960 0x55722e4a9dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:38.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.192+0000 7fa6527cb740 4 client.27490 mds_command: new command op to 14463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:19:38.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.192+0000 7fa6527cb740 1 -- 172.21.15.39:0/465852463 --> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55722e403990 con 0x55722e4a7960 2024-03-18T16:19:38.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.192+0000 7fa64d000640 1 --2- 172.21.15.39:0/465852463 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x55722e4a7960 0x55722e4a9dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:38.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.193+0000 7fa64d000640 1 --2- 172.21.15.39:0/465852463 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x55722e4a7960 0x55722e4a9dc0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=82721b7119091303 server_cookie=311c50462f43bbe8 in_seq=0 out_seq=0 2024-03-18T16:19:38.197 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.193+0000 7fa62ea00640 10 client.27490 ms_handle_connect on v2:172.21.15.39:6840/3515326042 2024-03-18T16:19:38.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.194+0000 7fa62ea00640 1 -- 172.21.15.39:0/465852463 <== mds.4 v2:172.21.15.39:6840/3515326042 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55722e403990 con 0x55722e4a7960 2024-03-18T16:19:38.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.194+0000 7fa62ea00640 10 client.27490 handle_command_reply: tid=0 2024-03-18T16:19:38.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.194+0000 7fa6527cb740 1 client.27490 shutdown 2024-03-18T16:19:38.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.195+0000 7fa6527cb740 1 -- 172.21.15.39:0/465852463 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55722e4922b0 msgr2=0x55722e49c0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:38.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.195+0000 7fa6527cb740 1 --2- 172.21.15.39:0/465852463 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55722e4922b0 0x55722e49c0d0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fa6340a39a0 tx=0x7fa634009700 comp rx=0 tx=0).stop 2024-03-18T16:19:38.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.195+0000 7fa6527cb740 1 -- 172.21.15.39:0/465852463 shutdown_connections 2024-03-18T16:19:38.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.195+0000 7fa6527cb740 1 --2- 172.21.15.39:0/465852463 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x55722e4a7960 0x55722e4a9dc0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.195+0000 7fa6527cb740 1 --2- 172.21.15.39:0/465852463 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55722e497420 0x55722e4a3110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.195+0000 7fa6527cb740 1 --2- 172.21.15.39:0/465852463 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55722e4922b0 0x55722e49c0d0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.195+0000 7fa6527cb740 1 --2- 172.21.15.39:0/465852463 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55722e4918c0 0x55722e49bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.195+0000 7fa62ea00640 0 client.27490 ms_handle_reset on v2:172.21.15.39:6840/3515326042 2024-03-18T16:19:38.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.195+0000 7fa6527cb740 1 -- 172.21.15.39:0/465852463 >> 172.21.15.39:0/465852463 conn(0x55722e48ce30 msgr2=0x55722e48c1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:38.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.195+0000 7fa6527cb740 1 -- 172.21.15.39:0/465852463 shutdown_connections 2024-03-18T16:19:38.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.195+0000 7fa6527cb740 1 -- 172.21.15.39:0/465852463 wait complete. 2024-03-18T16:19:38.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.195+0000 7fa6527cb740 20 client.27490 trim_cache size 0 max 16384 2024-03-18T16:19:38.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.222+0000 7fa6527cb740 1 Processor -- start 2024-03-18T16:19:38.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.223+0000 7fa6527cb740 1 -- start start 2024-03-18T16:19:38.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.223+0000 7fa6527cb740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55722e4918c0 0x55722e496b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:38.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.223+0000 7fa6527cb740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55722e4922b0 0x55722e4a30d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:38.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.223+0000 7fa6527cb740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55722e497420 0x55722e48b960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:38.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.223+0000 7fa6527cb740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fa6481a81d0 con 0x55722e4922b0 2024-03-18T16:19:38.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.223+0000 7fa6527cb740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fa6481a9d10 con 0x55722e497420 2024-03-18T16:19:38.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.223+0000 7fa6527cb740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fa6481a88a0 con 0x55722e4918c0 2024-03-18T16:19:38.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.223+0000 7fa64d000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55722e4918c0 0x55722e496b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:38.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.223+0000 7fa64d000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55722e4918c0 0x55722e496b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:53536/0 (socket says 172.21.15.39:53536) 2024-03-18T16:19:38.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.223+0000 7fa64d000640 1 -- 172.21.15.39:0/3387860770 learned_addr learned my addr 172.21.15.39:0/3387860770 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:19:38.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.224+0000 7fa64da00640 1 --2- 172.21.15.39:0/3387860770 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55722e497420 0x55722e48b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:38.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.224+0000 7fa647e00640 1 --2- 172.21.15.39:0/3387860770 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55722e4922b0 0x55722e4a30d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:38.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.224+0000 7fa64d000640 1 -- 172.21.15.39:0/3387860770 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55722e497420 msgr2=0x55722e48b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:38.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.224+0000 7fa64d000640 1 --2- 172.21.15.39:0/3387860770 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55722e497420 0x55722e48b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.224+0000 7fa64d000640 1 -- 172.21.15.39:0/3387860770 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55722e4922b0 msgr2=0x55722e4a30d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:38.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.224+0000 7fa64d000640 1 --2- 172.21.15.39:0/3387860770 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55722e4922b0 0x55722e4a30d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.224+0000 7fa64d000640 1 -- 172.21.15.39:0/3387860770 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55722e488c10 con 0x55722e4918c0 2024-03-18T16:19:38.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.224+0000 7fa64da00640 1 --2- 172.21.15.39:0/3387860770 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55722e497420 0x55722e48b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:19:38.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.224+0000 7fa647e00640 1 --2- 172.21.15.39:0/3387860770 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55722e4922b0 0x55722e4a30d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:19:38.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.224+0000 7fa64d000640 1 --2- 172.21.15.39:0/3387860770 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55722e4918c0 0x55722e496b10 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fa63c022040 tx=0x7fa63c000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=baf2ead858677909 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:38.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.225+0000 7fa639600640 1 -- 172.21.15.39:0/3387860770 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa63c005850 con 0x55722e4918c0 2024-03-18T16:19:38.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.225+0000 7fa639600640 1 -- 172.21.15.39:0/3387860770 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa63c00ec10 con 0x55722e4918c0 2024-03-18T16:19:38.228 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.225+0000 7fa639600640 1 -- 172.21.15.39:0/3387860770 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa63c020410 con 0x55722e4918c0 2024-03-18T16:19:38.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:37 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/1287244007' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:19:38.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.226+0000 7fa6527cb740 1 -- 172.21.15.39:0/3387860770 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55722e4918c0 msgr2=0x55722e496b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:38.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.226+0000 7fa6527cb740 1 --2- 172.21.15.39:0/3387860770 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55722e4918c0 0x55722e496b10 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fa63c022040 tx=0x7fa63c000bc0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.226+0000 7fa6527cb740 1 -- 172.21.15.39:0/3387860770 shutdown_connections 2024-03-18T16:19:38.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.226+0000 7fa6527cb740 1 --2- 172.21.15.39:0/3387860770 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55722e497420 0x55722e48b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.232 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.226+0000 7fa6527cb740 1 --2- 172.21.15.39:0/3387860770 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55722e4922b0 0x55722e4a30d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.226+0000 7fa6527cb740 1 --2- 172.21.15.39:0/3387860770 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55722e4918c0 0x55722e496b10 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.226+0000 7fa6527cb740 1 -- 172.21.15.39:0/3387860770 >> 172.21.15.39:0/3387860770 conn(0x55722e48ce30 msgr2=0x55722e4935d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:38.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.226+0000 7fa6527cb740 1 -- 172.21.15.39:0/3387860770 shutdown_connections 2024-03-18T16:19:38.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.226+0000 7fa6527cb740 1 -- 172.21.15.39:0/3387860770 wait complete. 2024-03-18T16:19:38.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.226+0000 7fa6527cb740 1 -- start start 2024-03-18T16:19:38.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.227+0000 7fa6527cb740 1 Processor -- start 2024-03-18T16:19:38.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.227+0000 7fa6527cb740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55722e4918c0 0x55722e4a0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:38.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.227+0000 7fa6527cb740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55722e4922b0 0x55722e4a1000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:38.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.227+0000 7fa6527cb740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55722e497420 0x55722e4a2010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:38.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.227+0000 7fa6527cb740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fa6481a9640 con 0x55722e4922b0 2024-03-18T16:19:38.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.227+0000 7fa6527cb740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fa6481a8f70 con 0x55722e497420 2024-03-18T16:19:38.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.227+0000 7fa6527cb740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fa6481a6210 con 0x55722e4918c0 2024-03-18T16:19:38.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.227+0000 7fa6527cb740 10 client.0 _ll_register_callbacks cb 0x7fa6340d28b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:19:38.233 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.227+0000 7fa64d000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55722e4918c0 0x55722e4a0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:38.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.227+0000 7fa64d000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55722e4918c0 0x55722e4a0ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:53552/0 (socket says 172.21.15.39:53552) 2024-03-18T16:19:38.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.227+0000 7fa647e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55722e4922b0 0x55722e4a1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:38.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.228+0000 7fa64d000640 1 -- 172.21.15.39:0/3375119982 learned_addr learned my addr 172.21.15.39:0/3375119982 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:19:38.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.228+0000 7fa64da00640 1 --2- 172.21.15.39:0/3375119982 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55722e497420 0x55722e4a2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:38.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.228+0000 7fa64d000640 1 -- 172.21.15.39:0/3375119982 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55722e497420 msgr2=0x55722e4a2010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:38.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.228+0000 7fa64d000640 1 --2- 172.21.15.39:0/3375119982 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55722e497420 0x55722e4a2010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.228+0000 7fa64d000640 1 -- 172.21.15.39:0/3375119982 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55722e4922b0 msgr2=0x55722e4a1000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:38.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.228+0000 7fa64d000640 1 --2- 172.21.15.39:0/3375119982 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55722e4922b0 0x55722e4a1000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.228+0000 7fa64d000640 1 -- 172.21.15.39:0/3375119982 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55722e4a2600 con 0x55722e4918c0 2024-03-18T16:19:38.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.228+0000 7fa64d000640 1 --2- 172.21.15.39:0/3375119982 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55722e4918c0 0x55722e4a0ac0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fa63c00e6c0 tx=0x7fa63c009400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=306f0242ce3fa559 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:38.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.228+0000 7fa62ea00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:19:38.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.229+0000 7fa62ea00640 1 -- 172.21.15.39:0/3375119982 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa63c03c070 con 0x55722e4918c0 2024-03-18T16:19:38.234 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.229+0000 7fa62ea00640 1 -- 172.21.15.39:0/3375119982 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fa63c01d040 con 0x55722e4918c0 2024-03-18T16:19:38.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.229+0000 7fa6527cb740 1 -- 172.21.15.39:0/3375119982 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa63c00ec10 con 0x55722e4918c0 2024-03-18T16:19:38.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.229+0000 7fa62ea00640 1 -- 172.21.15.39:0/3375119982 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fa63c045050 con 0x55722e4918c0 2024-03-18T16:19:38.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.229+0000 7fa6527cb740 10 client.27496 fetch_fsmap learned FSMap version 44 2024-03-18T16:19:38.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.229+0000 7fa6527cb740 1 -- 172.21.15.39:0/3375119982 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x55722e50ded0 con 0x55722e4918c0 2024-03-18T16:19:38.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.229+0000 7fa62ea00640 1 -- 172.21.15.39:0/3375119982 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa63c00ec10 con 0x55722e4918c0 2024-03-18T16:19:38.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.230+0000 7fa62ea00640 1 -- 172.21.15.39:0/3375119982 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7fa63c00f040 con 0x55722e4918c0 2024-03-18T16:19:38.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.230+0000 7fa6527cb740 10 client.27496 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:19:38.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.230+0000 7fa6527cb740 10 client.27496 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-03-18T16:19:38.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.230+0000 7fa6527cb740 20 client.27496 populate_metadata read hostname 'smithi039' 2024-03-18T16:19:38.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.230+0000 7fa6527cb740 1 --2- 172.21.15.39:0/3375119982 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x55722e50e6c0 0x55722e510ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:38.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.230+0000 7fa6527cb740 4 client.27496 mds_command: new command op to 14463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:19:38.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.230+0000 7fa6527cb740 1 -- 172.21.15.39:0/3375119982 --> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55722e3fdce0 con 0x55722e50e6c0 2024-03-18T16:19:38.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.230+0000 7fa647e00640 1 --2- 172.21.15.39:0/3375119982 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x55722e50e6c0 0x55722e510ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:38.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.231+0000 7fa647e00640 1 --2- 172.21.15.39:0/3375119982 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x55722e50e6c0 0x55722e510ae0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=8f84b32a640def3 server_cookie=4184a2f0cfb974d2 in_seq=0 out_seq=0 2024-03-18T16:19:38.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.231+0000 7fa62ea00640 10 client.27496 ms_handle_connect on v2:172.21.15.39:6840/3515326042 2024-03-18T16:19:38.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.231+0000 7fa62ea00640 1 -- 172.21.15.39:0/3375119982 <== mds.4 v2:172.21.15.39:6840/3515326042 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55722e3fdce0 con 0x55722e50e6c0 2024-03-18T16:19:38.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.231+0000 7fa62ea00640 10 client.27496 handle_command_reply: tid=0 2024-03-18T16:19:38.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.231+0000 7fa6527cb740 1 client.27496 shutdown 2024-03-18T16:19:38.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.231+0000 7fa6527cb740 1 -- 172.21.15.39:0/3375119982 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55722e4918c0 msgr2=0x55722e4a0ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:38.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.231+0000 7fa6527cb740 1 --2- 172.21.15.39:0/3375119982 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55722e4918c0 0x55722e4a0ac0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fa63c00e6c0 tx=0x7fa63c009400 comp rx=0 tx=0).stop 2024-03-18T16:19:38.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.231+0000 7fa6527cb740 1 -- 172.21.15.39:0/3375119982 shutdown_connections 2024-03-18T16:19:38.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.231+0000 7fa6527cb740 1 --2- 172.21.15.39:0/3375119982 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x55722e50e6c0 0x55722e510ae0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.232+0000 7fa6527cb740 1 --2- 172.21.15.39:0/3375119982 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55722e497420 0x55722e4a2010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.232+0000 7fa6527cb740 1 --2- 172.21.15.39:0/3375119982 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55722e4922b0 0x55722e4a1000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.232+0000 7fa6527cb740 1 --2- 172.21.15.39:0/3375119982 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55722e4918c0 0x55722e4a0ac0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.232+0000 7fa62ea00640 0 client.27496 ms_handle_reset on v2:172.21.15.39:6840/3515326042 2024-03-18T16:19:38.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.232+0000 7fa6527cb740 1 -- 172.21.15.39:0/3375119982 >> 172.21.15.39:0/3375119982 conn(0x55722e48ce30 msgr2=0x55722e48b960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:38.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.233+0000 7fa6527cb740 1 -- 172.21.15.39:0/3375119982 shutdown_connections 2024-03-18T16:19:38.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.233+0000 7fa6527cb740 1 -- 172.21.15.39:0/3375119982 wait complete. 2024-03-18T16:19:38.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.233+0000 7fa6527cb740 20 client.27496 trim_cache size 0 max 16384 2024-03-18T16:19:38.238 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:19:38.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.234+0000 7fa64fa00640 1 -- 172.21.15.39:0/3772091101 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa6300a2880 msgr2=0x7fa6300a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:38.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.234+0000 7fa64fa00640 1 --2- 172.21.15.39:0/3772091101 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa6300a2880 0x7fa6300a4d40 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fa640005e30 tx=0x7fa64002e040 comp rx=0 tx=0).stop 2024-03-18T16:19:38.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.234+0000 7fa64fa00640 1 -- 172.21.15.39:0/3772091101 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa6481029d0 msgr2=0x7fa64819a1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:38.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.234+0000 7fa64fa00640 1 --2- 172.21.15.39:0/3772091101 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa6481029d0 0x7fa64819a1e0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fa63400e9f0 tx=0x7fa63400eeb0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.235+0000 7fa64fa00640 1 -- 172.21.15.39:0/3772091101 shutdown_connections 2024-03-18T16:19:38.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.235+0000 7fa64fa00640 1 --2- 172.21.15.39:0/3772091101 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa648103390 0x7fa6481a1240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.235+0000 7fa64fa00640 1 --2- 172.21.15.39:0/3772091101 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa6300a2880 0x7fa6300a4d40 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.235+0000 7fa64fa00640 1 --2- 172.21.15.39:0/3772091101 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa6481029d0 0x7fa64819a1e0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.235+0000 7fa64fa00640 1 --2- 172.21.15.39:0/3772091101 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa6481017e0 0x7fa648199ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.235+0000 7fa64fa00640 1 -- 172.21.15.39:0/3772091101 >> 172.21.15.39:0/3772091101 conn(0x7fa6480fcd50 msgr2=0x7fa648073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:38.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.235+0000 7fa64fa00640 1 -- 172.21.15.39:0/3772091101 shutdown_connections 2024-03-18T16:19:38.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.235+0000 7fa64fa00640 1 -- 172.21.15.39:0/3772091101 wait complete. 2024-03-18T16:19:38.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:37 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/1287244007' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:19:38.253 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:37 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/1287244007' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:19:38.257 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-18T16:19:38.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.474+0000 7f3d36a00640 1 -- 172.21.15.39:0/3818442061 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3d300757e0 msgr2=0x7f3d30073c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:38.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.474+0000 7f3d36a00640 1 --2- 172.21.15.39:0/3818442061 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3d300757e0 0x7f3d30073c30 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f3d20009900 tx=0x7f3d2002bc40 comp rx=0 tx=0).stop 2024-03-18T16:19:38.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.474+0000 7f3d36a00640 1 -- 172.21.15.39:0/3818442061 shutdown_connections 2024-03-18T16:19:38.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.474+0000 7f3d36a00640 1 --2- 172.21.15.39:0/3818442061 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3d30074200 0x7f3d30074680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.474+0000 7f3d36a00640 1 --2- 172.21.15.39:0/3818442061 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3d30109690 0x7f3d3010bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.474+0000 7f3d36a00640 1 --2- 172.21.15.39:0/3818442061 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3d300757e0 0x7f3d30073c30 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.474+0000 7f3d36a00640 1 -- 172.21.15.39:0/3818442061 >> 172.21.15.39:0/3818442061 conn(0x7f3d300fcd50 msgr2=0x7f3d300ff190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:38.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.474+0000 7f3d36a00640 1 -- 172.21.15.39:0/3818442061 shutdown_connections 2024-03-18T16:19:38.477 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.474+0000 7f3d36a00640 1 -- 172.21.15.39:0/3818442061 wait complete. 2024-03-18T16:19:38.478 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.475+0000 7f3d36a00640 1 Processor -- start 2024-03-18T16:19:38.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.476+0000 7f3d36a00640 1 -- start start 2024-03-18T16:19:38.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.476+0000 7f3d36a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3d30074200 0x7f3d30199d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:38.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.476+0000 7f3d36a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3d300757e0 0x7f3d3019a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:38.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.476+0000 7f3d36a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3d30109690 0x7f3d301a1270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:38.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.476+0000 7f3d36a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f3d3010f670 con 0x7f3d30109690 2024-03-18T16:19:38.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.476+0000 7f3d36a00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f3d3010f4f0 con 0x7f3d30074200 2024-03-18T16:19:38.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.476+0000 7f3d36a00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f3d3010f7f0 con 0x7f3d300757e0 2024-03-18T16:19:38.479 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.476+0000 7f3d2f400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3d300757e0 0x7f3d3019a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:38.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.476+0000 7f3d2fe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3d30074200 0x7f3d30199d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:38.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.476+0000 7f3d2f400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3d300757e0 0x7f3d3019a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:53568/0 (socket says 172.21.15.39:53568) 2024-03-18T16:19:38.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.476+0000 7f3d2fe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3d30074200 0x7f3d30199d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:53314/0 (socket says 172.21.15.39:53314) 2024-03-18T16:19:38.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.476+0000 7f3d2f400640 1 -- 172.21.15.39:0/968282965 learned_addr learned my addr 172.21.15.39:0/968282965 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:19:38.480 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.476+0000 7f3d34a00640 1 --2- 172.21.15.39:0/968282965 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3d30109690 0x7f3d301a1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:38.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.477+0000 7f3d2fe00640 1 -- 172.21.15.39:0/968282965 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3d300757e0 msgr2=0x7f3d3019a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:38.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.477+0000 7f3d2fe00640 1 --2- 172.21.15.39:0/968282965 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3d300757e0 0x7f3d3019a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.477+0000 7f3d2fe00640 1 -- 172.21.15.39:0/968282965 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3d30109690 msgr2=0x7f3d301a1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:38.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.477+0000 7f3d2fe00640 1 --2- 172.21.15.39:0/968282965 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3d30109690 0x7f3d301a1270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.477+0000 7f3d2fe00640 1 -- 172.21.15.39:0/968282965 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d301a1910 con 0x7f3d30074200 2024-03-18T16:19:38.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.477+0000 7f3d2fe00640 1 --2- 172.21.15.39:0/968282965 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3d30074200 0x7f3d30199d00 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f3d20036040 tx=0x7f3d2002c660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:38.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.478+0000 7f3d2cc00640 1 -- 172.21.15.39:0/968282965 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d20041070 con 0x7f3d30074200 2024-03-18T16:19:38.481 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.478+0000 7f3d2cc00640 1 -- 172.21.15.39:0/968282965 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d20004450 con 0x7f3d30074200 2024-03-18T16:19:38.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.478+0000 7f3d2cc00640 1 -- 172.21.15.39:0/968282965 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d20034440 con 0x7f3d30074200 2024-03-18T16:19:38.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.478+0000 7f3d36a00640 1 -- 172.21.15.39:0/968282965 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d301a1ba0 con 0x7f3d30074200 2024-03-18T16:19:38.482 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.478+0000 7f3d36a00640 1 -- 172.21.15.39:0/968282965 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d301a2010 con 0x7f3d30074200 2024-03-18T16:19:38.484 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.482+0000 7f3d2cc00640 1 -- 172.21.15.39:0/968282965 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f3d20044070 con 0x7f3d30074200 2024-03-18T16:19:38.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.483+0000 7f3d395f8740 1 Processor -- start 2024-03-18T16:19:38.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.483+0000 7f3d2cc00640 1 --2- 172.21.15.39:0/968282965 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3d140a2880 0x7f3d140a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:38.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.483+0000 7f3d395f8740 1 -- start start 2024-03-18T16:19:38.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.483+0000 7f3d2cc00640 1 -- 172.21.15.39:0/968282965 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f3d200f0880 con 0x7f3d30074200 2024-03-18T16:19:38.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.483+0000 7f3d395f8740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5566d076c8c0 0x5566d076cce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:38.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.483+0000 7f3d395f8740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5566d076d2b0 0x5566d0771db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:38.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.483+0000 7f3d395f8740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5566d0772420 0x5566d07748b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:38.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.483+0000 7f3d395f8740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5566d07343b0 con 0x5566d0772420 2024-03-18T16:19:38.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.483+0000 7f3d395f8740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5566d06d1450 con 0x5566d076c8c0 2024-03-18T16:19:38.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.483+0000 7f3d395f8740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x5566d0679cb0 con 0x5566d076d2b0 2024-03-18T16:19:38.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.484+0000 7f3d34a00640 1 --2- 172.21.15.39:0/968282965 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3d140a2880 0x7f3d140a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:38.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.484+0000 7f3d2fe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5566d076c8c0 0x5566d076cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:38.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.484+0000 7f3d2fe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5566d076c8c0 0x5566d076cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:53322/0 (socket says 172.21.15.39:53322) 2024-03-18T16:19:38.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.484+0000 7f3d2fe00640 1 -- 172.21.15.39:0/2099536670 learned_addr learned my addr 172.21.15.39:0/2099536670 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:19:38.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.484+0000 7f3d2f400640 1 --2- 172.21.15.39:0/2099536670 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5566d076d2b0 0x5566d0771db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:38.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.484+0000 7f3d34a00640 1 --2- 172.21.15.39:0/2099536670 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5566d0772420 0x5566d07748b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:38.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.484+0000 7f3d2fe00640 1 -- 172.21.15.39:0/2099536670 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5566d076d2b0 msgr2=0x5566d0771db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:38.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.484+0000 7f3d2fe00640 1 --2- 172.21.15.39:0/2099536670 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5566d076d2b0 0x5566d0771db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.484+0000 7f3d2fe00640 1 -- 172.21.15.39:0/2099536670 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5566d0772420 msgr2=0x5566d07748b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:38.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.484+0000 7f3d2fe00640 1 --2- 172.21.15.39:0/2099536670 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5566d0772420 0x5566d07748b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.484+0000 7f3d2fe00640 1 -- 172.21.15.39:0/2099536670 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5566d069b1f0 con 0x5566d076c8c0 2024-03-18T16:19:38.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.485+0000 7f3d2fe00640 1 --2- 172.21.15.39:0/2099536670 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5566d076c8c0 0x5566d076cce0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f3d200c50a0 tx=0x7f3d200035f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=effb37cdec3053d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:38.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.485+0000 7f3d34a00640 1 --2- 172.21.15.39:0/968282965 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3d140a2880 0x7f3d140a4d40 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f3d28005fd0 tx=0x7f3d28009b00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:38.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.485+0000 7f3d25200640 1 -- 172.21.15.39:0/2099536670 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d2004b040 con 0x5566d076c8c0 2024-03-18T16:19:38.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.485+0000 7f3d25200640 1 -- 172.21.15.39:0/2099536670 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d200bcee0 con 0x5566d076c8c0 2024-03-18T16:19:38.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.485+0000 7f3d25200640 1 -- 172.21.15.39:0/2099536670 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d200c5140 con 0x5566d076c8c0 2024-03-18T16:19:38.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.486+0000 7f3d395f8740 1 -- 172.21.15.39:0/2099536670 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5566d076c8c0 msgr2=0x5566d076cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:38.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.486+0000 7f3d395f8740 1 --2- 172.21.15.39:0/2099536670 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5566d076c8c0 0x5566d076cce0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f3d200c50a0 tx=0x7f3d200035f0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.486+0000 7f3d395f8740 1 -- 172.21.15.39:0/2099536670 shutdown_connections 2024-03-18T16:19:38.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.486+0000 7f3d395f8740 1 --2- 172.21.15.39:0/2099536670 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5566d0772420 0x5566d07748b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.486+0000 7f3d395f8740 1 --2- 172.21.15.39:0/2099536670 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5566d076d2b0 0x5566d0771db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.486+0000 7f3d395f8740 1 --2- 172.21.15.39:0/2099536670 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5566d076c8c0 0x5566d076cce0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.486+0000 7f3d395f8740 1 -- 172.21.15.39:0/2099536670 >> 172.21.15.39:0/2099536670 conn(0x5566d0767e30 msgr2=0x5566d0768230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:38.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.486+0000 7f3d395f8740 1 -- 172.21.15.39:0/2099536670 shutdown_connections 2024-03-18T16:19:38.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.486+0000 7f3d395f8740 1 -- 172.21.15.39:0/2099536670 wait complete. 2024-03-18T16:19:38.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.487+0000 7f3d395f8740 1 -- start start 2024-03-18T16:19:38.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.487+0000 7f3d395f8740 1 Processor -- start 2024-03-18T16:19:38.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.488+0000 7f3d395f8740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5566d076c8c0 0x5566d077f130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:38.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.488+0000 7f3d395f8740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5566d076d2b0 0x5566d077f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:38.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.488+0000 7f3d395f8740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5566d0772420 0x5566d07866b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:38.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.488+0000 7f3d395f8740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5566d068d110 con 0x5566d0772420 2024-03-18T16:19:38.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.488+0000 7f3d395f8740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5566d06cd4f0 con 0x5566d076d2b0 2024-03-18T16:19:38.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.488+0000 7f3d395f8740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x5566d077d9a0 con 0x5566d076c8c0 2024-03-18T16:19:38.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.488+0000 7f3d395f8740 10 client.0 _ll_register_callbacks cb 0x5566d067ab70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:19:38.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.488+0000 7f3d2f400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5566d076d2b0 0x5566d077f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:38.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.488+0000 7f3d2fe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5566d076c8c0 0x5566d077f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:38.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.488+0000 7f3d2f400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5566d076d2b0 0x5566d077f670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:53330/0 (socket says 172.21.15.39:53330) 2024-03-18T16:19:38.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.488+0000 7f3d2f400640 1 -- 172.21.15.39:0/2566649391 learned_addr learned my addr 172.21.15.39:0/2566649391 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:19:38.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.488+0000 7f3d2fe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5566d076c8c0 0x5566d077f130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:53590/0 (socket says 172.21.15.39:53590) 2024-03-18T16:19:38.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.488+0000 7f3d34a00640 1 --2- 172.21.15.39:0/2566649391 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5566d0772420 0x5566d07866b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:38.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.489+0000 7f3d2fe00640 1 -- 172.21.15.39:0/2566649391 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5566d076d2b0 msgr2=0x5566d077f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:38.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.489+0000 7f3d2fe00640 1 --2- 172.21.15.39:0/2566649391 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5566d076d2b0 0x5566d077f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.489+0000 7f3d2fe00640 1 -- 172.21.15.39:0/2566649391 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5566d0772420 msgr2=0x5566d07866b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:38.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.489+0000 7f3d2fe00640 1 --2- 172.21.15.39:0/2566649391 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5566d0772420 0x5566d07866b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.489+0000 7f3d2fe00640 1 -- 172.21.15.39:0/2566649391 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5566d03e53f0 con 0x5566d076c8c0 2024-03-18T16:19:38.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.489+0000 7f3d2f400640 1 --2- 172.21.15.39:0/2566649391 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5566d076d2b0 0x5566d077f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:19:38.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.489+0000 7f3d34a00640 1 --2- 172.21.15.39:0/2566649391 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5566d0772420 0x5566d07866b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:19:38.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.489+0000 7f3d2fe00640 1 --2- 172.21.15.39:0/2566649391 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5566d076c8c0 0x5566d077f130 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f3d200c50a0 tx=0x7f3d200c53f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4f4e86659b7a21c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:38.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.489+0000 7f3d19600640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:19:38.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.489+0000 7f3d19600640 1 -- 172.21.15.39:0/2566649391 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d200db070 con 0x5566d076c8c0 2024-03-18T16:19:38.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.489+0000 7f3d395f8740 1 -- 172.21.15.39:0/2566649391 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3d200bcee0 con 0x5566d076c8c0 2024-03-18T16:19:38.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.489+0000 7f3d19600640 1 -- 172.21.15.39:0/2566649391 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f3d200bc540 con 0x5566d076c8c0 2024-03-18T16:19:38.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.490+0000 7f3d19600640 1 -- 172.21.15.39:0/2566649391 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f3d2004b070 con 0x5566d076c8c0 2024-03-18T16:19:38.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.490+0000 7f3d395f8740 10 client.27508 fetch_fsmap learned FSMap version 44 2024-03-18T16:19:38.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.490+0000 7f3d395f8740 1 -- 172.21.15.39:0/2566649391 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x5566d04af7d0 con 0x5566d076c8c0 2024-03-18T16:19:38.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.490+0000 7f3d19600640 1 -- 172.21.15.39:0/2566649391 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d200bcee0 con 0x5566d076c8c0 2024-03-18T16:19:38.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.490+0000 7f3d19600640 1 -- 172.21.15.39:0/2566649391 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7f3d200bbda0 con 0x5566d076c8c0 2024-03-18T16:19:38.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.490+0000 7f3d395f8740 10 client.27508 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:19:38.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.490+0000 7f3d395f8740 10 client.27508 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-18T16:19:38.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.490+0000 7f3d395f8740 20 client.27508 populate_metadata read hostname 'smithi039' 2024-03-18T16:19:38.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.490+0000 7f3d395f8740 1 --2- 172.21.15.39:0/2566649391 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x5566d078ad20 0x5566d078d130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:38.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.490+0000 7f3d395f8740 4 client.27508 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:19:38.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.490+0000 7f3d395f8740 1 -- 172.21.15.39:0/2566649391 --> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5566d06de990 con 0x5566d078ad20 2024-03-18T16:19:38.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.491+0000 7f3d2f400640 1 --2- 172.21.15.39:0/2566649391 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x5566d078ad20 0x5566d078d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:38.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.492+0000 7f3d2f400640 1 --2- 172.21.15.39:0/2566649391 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x5566d078ad20 0x5566d078d130 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=f121f9ba96a068e3 server_cookie=de5d8977246557a0 in_seq=0 out_seq=0 2024-03-18T16:19:38.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.492+0000 7f3d19600640 10 client.27508 ms_handle_connect on v2:172.21.15.194:6836/1301464246 2024-03-18T16:19:38.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.493+0000 7f3d19600640 1 -- 172.21.15.39:0/2566649391 <== mds.3 v2:172.21.15.194:6836/1301464246 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5566d06de990 con 0x5566d078ad20 2024-03-18T16:19:38.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.493+0000 7f3d19600640 10 client.27508 handle_command_reply: tid=0 2024-03-18T16:19:38.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.493+0000 7f3d395f8740 1 client.27508 shutdown 2024-03-18T16:19:38.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.493+0000 7f3d395f8740 1 -- 172.21.15.39:0/2566649391 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5566d076c8c0 msgr2=0x5566d077f130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:38.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.493+0000 7f3d395f8740 1 --2- 172.21.15.39:0/2566649391 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5566d076c8c0 0x5566d077f130 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f3d200c50a0 tx=0x7f3d200c53f0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.494+0000 7f3d395f8740 1 -- 172.21.15.39:0/2566649391 shutdown_connections 2024-03-18T16:19:38.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.494+0000 7f3d395f8740 1 --2- 172.21.15.39:0/2566649391 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x5566d078ad20 0x5566d078d130 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.494+0000 7f3d395f8740 1 --2- 172.21.15.39:0/2566649391 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5566d076d2b0 0x5566d077f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.494+0000 7f3d395f8740 1 --2- 172.21.15.39:0/2566649391 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5566d0772420 0x5566d07866b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.494+0000 7f3d395f8740 1 --2- 172.21.15.39:0/2566649391 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5566d076c8c0 0x5566d077f130 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.494+0000 7f3d395f8740 1 -- 172.21.15.39:0/2566649391 >> 172.21.15.39:0/2566649391 conn(0x5566d0767e30 msgr2=0x5566d07671f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:38.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.494+0000 7f3d19600640 0 client.27508 ms_handle_reset on v2:172.21.15.194:6836/1301464246 2024-03-18T16:19:38.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.494+0000 7f3d395f8740 1 -- 172.21.15.39:0/2566649391 shutdown_connections 2024-03-18T16:19:38.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.494+0000 7f3d395f8740 1 -- 172.21.15.39:0/2566649391 wait complete. 2024-03-18T16:19:38.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.494+0000 7f3d395f8740 20 client.27508 trim_cache size 0 max 16384 2024-03-18T16:19:38.522 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.519+0000 7f3d395f8740 1 Processor -- start 2024-03-18T16:19:38.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.519+0000 7f3d395f8740 1 -- start start 2024-03-18T16:19:38.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.519+0000 7f3d395f8740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5566d076c8c0 0x5566d078cd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:38.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.519+0000 7f3d395f8740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5566d076d2b0 0x5566d0786640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:38.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.519+0000 7f3d395f8740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5566d0772420 0x5566d077f450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:38.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.519+0000 7f3d395f8740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f3d301a8200 con 0x5566d0772420 2024-03-18T16:19:38.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.519+0000 7f3d395f8740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f3d301a9d40 con 0x5566d076d2b0 2024-03-18T16:19:38.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.519+0000 7f3d395f8740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f3d301a88d0 con 0x5566d076c8c0 2024-03-18T16:19:38.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.519+0000 7f3d2fe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5566d076c8c0 0x5566d078cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:38.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.519+0000 7f3d2f400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5566d076d2b0 0x5566d0786640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:38.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.519+0000 7f3d2fe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5566d076c8c0 0x5566d078cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:53594/0 (socket says 172.21.15.39:53594) 2024-03-18T16:19:38.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.519+0000 7f3d2f400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5566d076d2b0 0x5566d0786640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:53342/0 (socket says 172.21.15.39:53342) 2024-03-18T16:19:38.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.519+0000 7f3d2fe00640 1 -- 172.21.15.39:0/2058482620 learned_addr learned my addr 172.21.15.39:0/2058482620 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:19:38.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.520+0000 7f3d34a00640 1 --2- 172.21.15.39:0/2058482620 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5566d0772420 0x5566d077f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:38.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.520+0000 7f3d2fe00640 1 -- 172.21.15.39:0/2058482620 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5566d076d2b0 msgr2=0x5566d0786640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:38.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.520+0000 7f3d2fe00640 1 --2- 172.21.15.39:0/2058482620 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5566d076d2b0 0x5566d0786640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.520+0000 7f3d2fe00640 1 -- 172.21.15.39:0/2058482620 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5566d0772420 msgr2=0x5566d077f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:38.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.520+0000 7f3d2fe00640 1 --2- 172.21.15.39:0/2058482620 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5566d0772420 0x5566d077f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.520+0000 7f3d2fe00640 1 -- 172.21.15.39:0/2058482620 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5566d0763c10 con 0x5566d076c8c0 2024-03-18T16:19:38.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.520+0000 7f3d34a00640 1 --2- 172.21.15.39:0/2058482620 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5566d0772420 0x5566d077f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:19:38.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.520+0000 7f3d2fe00640 1 --2- 172.21.15.39:0/2058482620 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5566d076c8c0 0x5566d078cd30 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f3d2002c220 tx=0x7f3d200035f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1d97907d9cc09e6e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:38.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.520+0000 7f3d1be00640 1 -- 172.21.15.39:0/2058482620 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d200db040 con 0x5566d076c8c0 2024-03-18T16:19:38.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.520+0000 7f3d1be00640 1 -- 172.21.15.39:0/2058482620 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d200b94c0 con 0x5566d076c8c0 2024-03-18T16:19:38.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.520+0000 7f3d1be00640 1 -- 172.21.15.39:0/2058482620 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d200c0f80 con 0x5566d076c8c0 2024-03-18T16:19:38.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.521+0000 7f3d395f8740 1 -- 172.21.15.39:0/2058482620 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5566d076c8c0 msgr2=0x5566d078cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:38.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.521+0000 7f3d395f8740 1 --2- 172.21.15.39:0/2058482620 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5566d076c8c0 0x5566d078cd30 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f3d2002c220 tx=0x7f3d200035f0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.521+0000 7f3d395f8740 1 -- 172.21.15.39:0/2058482620 shutdown_connections 2024-03-18T16:19:38.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.521+0000 7f3d395f8740 1 --2- 172.21.15.39:0/2058482620 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5566d076d2b0 0x5566d0786640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.522+0000 7f3d395f8740 1 --2- 172.21.15.39:0/2058482620 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5566d0772420 0x5566d077f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.522+0000 7f3d395f8740 1 --2- 172.21.15.39:0/2058482620 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5566d076c8c0 0x5566d078cd30 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.522+0000 7f3d395f8740 1 -- 172.21.15.39:0/2058482620 >> 172.21.15.39:0/2058482620 conn(0x5566d0767e30 msgr2=0x5566d077ed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:38.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.522+0000 7f3d395f8740 1 -- 172.21.15.39:0/2058482620 shutdown_connections 2024-03-18T16:19:38.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.522+0000 7f3d395f8740 1 -- 172.21.15.39:0/2058482620 wait complete. 2024-03-18T16:19:38.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.522+0000 7f3d395f8740 1 -- start start 2024-03-18T16:19:38.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.523+0000 7f3d395f8740 1 Processor -- start 2024-03-18T16:19:38.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.523+0000 7f3d395f8740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5566d076c8c0 0x5566d07837b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:38.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.523+0000 7f3d395f8740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5566d076d2b0 0x5566d0783cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:38.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.523+0000 7f3d395f8740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5566d0772420 0x5566d0784d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:38.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.523+0000 7f3d395f8740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f3d301a9670 con 0x5566d076c8c0 2024-03-18T16:19:38.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.523+0000 7f3d395f8740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f3d301a8fa0 con 0x5566d0772420 2024-03-18T16:19:38.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.523+0000 7f3d395f8740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f3d301a61e0 con 0x5566d076d2b0 2024-03-18T16:19:38.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.523+0000 7f3d395f8740 10 client.0 _ll_register_callbacks cb 0x7f3d1c009530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:19:38.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.523+0000 7f3d2fe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5566d076c8c0 0x5566d07837b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:38.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.524+0000 7f3d2fe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5566d076c8c0 0x5566d07837b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51354/0 (socket says 172.21.15.39:51354) 2024-03-18T16:19:38.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.524+0000 7f3d2fe00640 1 -- 172.21.15.39:0/2705019357 learned_addr learned my addr 172.21.15.39:0/2705019357 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:19:38.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.524+0000 7f3d34a00640 1 --2- 172.21.15.39:0/2705019357 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5566d0772420 0x5566d0784d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:38.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.524+0000 7f3d2f400640 1 --2- 172.21.15.39:0/2705019357 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5566d076d2b0 0x5566d0783cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:38.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.524+0000 7f3d2fe00640 1 -- 172.21.15.39:0/2705019357 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5566d076d2b0 msgr2=0x5566d0783cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:38.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.524+0000 7f3d2fe00640 1 --2- 172.21.15.39:0/2705019357 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5566d076d2b0 0x5566d0783cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.524+0000 7f3d2fe00640 1 -- 172.21.15.39:0/2705019357 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5566d0772420 msgr2=0x5566d0784d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:38.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.524+0000 7f3d2fe00640 1 --2- 172.21.15.39:0/2705019357 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5566d0772420 0x5566d0784d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.524+0000 7f3d2fe00640 1 -- 172.21.15.39:0/2705019357 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5566d0768ba0 con 0x5566d076c8c0 2024-03-18T16:19:38.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.524+0000 7f3d2fe00640 1 --2- 172.21.15.39:0/2705019357 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5566d076c8c0 0x5566d07837b0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f3d2002e040 tx=0x7f3d200b90f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eee659176e308e7b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:38.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.524+0000 7f3d19600640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-03-18T16:19:38.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.525+0000 7f3d19600640 1 -- 172.21.15.39:0/2705019357 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d200db040 con 0x5566d076c8c0 2024-03-18T16:19:38.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.525+0000 7f3d19600640 1 -- 172.21.15.39:0/2705019357 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f3d200c2130 con 0x5566d076c8c0 2024-03-18T16:19:38.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.525+0000 7f3d395f8740 1 -- 172.21.15.39:0/2705019357 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5566d06d8ce0 con 0x5566d076c8c0 2024-03-18T16:19:38.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.525+0000 7f3d19600640 1 -- 172.21.15.39:0/2705019357 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f3d200c55d0 con 0x5566d076c8c0 2024-03-18T16:19:38.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.525+0000 7f3d395f8740 10 client.16884 fetch_fsmap learned FSMap version 44 2024-03-18T16:19:38.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.525+0000 7f3d395f8740 1 -- 172.21.15.39:0/2705019357 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x5566d0768ba0 con 0x5566d076c8c0 2024-03-18T16:19:38.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.525+0000 7f3d19600640 1 -- 172.21.15.39:0/2705019357 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5566d06d8ce0 con 0x5566d076c8c0 2024-03-18T16:19:38.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.525+0000 7f3d19600640 1 -- 172.21.15.39:0/2705019357 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7f3d200a76a0 con 0x5566d076c8c0 2024-03-18T16:19:38.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.526+0000 7f3d395f8740 10 client.16884 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:19:38.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.526+0000 7f3d395f8740 10 client.16884 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-18T16:19:38.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.526+0000 7f3d395f8740 20 client.16884 populate_metadata read hostname 'smithi039' 2024-03-18T16:19:38.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.526+0000 7f3d395f8740 1 --2- 172.21.15.39:0/2705019357 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x5566d07f1030 0x5566d07f3420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:38.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.526+0000 7f3d395f8740 4 client.16884 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:19:38.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.526+0000 7f3d395f8740 1 -- 172.21.15.39:0/2705019357 --> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5566d0656560 con 0x5566d07f1030 2024-03-18T16:19:38.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.526+0000 7f3d2f400640 1 --2- 172.21.15.39:0/2705019357 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x5566d07f1030 0x5566d07f3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:38.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.527+0000 7f3d2f400640 1 --2- 172.21.15.39:0/2705019357 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x5566d07f1030 0x5566d07f3420 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=2084f65ff082bac2 server_cookie=b5adeda304e3e1aa in_seq=0 out_seq=0 2024-03-18T16:19:38.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.527+0000 7f3d19600640 10 client.16884 ms_handle_connect on v2:172.21.15.194:6836/1301464246 2024-03-18T16:19:38.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.527+0000 7f3d19600640 1 -- 172.21.15.39:0/2705019357 <== mds.3 v2:172.21.15.194:6836/1301464246 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5566d0656560 con 0x5566d07f1030 2024-03-18T16:19:38.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.527+0000 7f3d19600640 10 client.16884 handle_command_reply: tid=0 2024-03-18T16:19:38.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.527+0000 7f3d395f8740 1 client.16884 shutdown 2024-03-18T16:19:38.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.527+0000 7f3d395f8740 1 -- 172.21.15.39:0/2705019357 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5566d076c8c0 msgr2=0x5566d07837b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:38.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.527+0000 7f3d395f8740 1 --2- 172.21.15.39:0/2705019357 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5566d076c8c0 0x5566d07837b0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f3d2002e040 tx=0x7f3d200b90f0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.527+0000 7f3d395f8740 1 -- 172.21.15.39:0/2705019357 shutdown_connections 2024-03-18T16:19:38.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.527+0000 7f3d395f8740 1 --2- 172.21.15.39:0/2705019357 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x5566d07f1030 0x5566d07f3420 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.527+0000 7f3d395f8740 1 --2- 172.21.15.39:0/2705019357 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5566d0772420 0x5566d0784d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.527+0000 7f3d395f8740 1 --2- 172.21.15.39:0/2705019357 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5566d076d2b0 0x5566d0783cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.527+0000 7f3d395f8740 1 --2- 172.21.15.39:0/2705019357 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5566d076c8c0 0x5566d07837b0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.527+0000 7f3d395f8740 1 -- 172.21.15.39:0/2705019357 >> 172.21.15.39:0/2705019357 conn(0x5566d0767e30 msgr2=0x5566d07671f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:38.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.527+0000 7f3d19600640 0 client.16884 ms_handle_reset on v2:172.21.15.194:6836/1301464246 2024-03-18T16:19:38.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.528+0000 7f3d395f8740 1 -- 172.21.15.39:0/2705019357 shutdown_connections 2024-03-18T16:19:38.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.528+0000 7f3d395f8740 1 -- 172.21.15.39:0/2705019357 wait complete. 2024-03-18T16:19:38.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.528+0000 7f3d395f8740 20 client.16884 trim_cache size 0 max 16384 2024-03-18T16:19:38.533 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:19:38.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.528+0000 7f3d36a00640 1 -- 172.21.15.39:0/968282965 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3d140a2880 msgr2=0x7f3d140a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:38.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.528+0000 7f3d36a00640 1 --2- 172.21.15.39:0/968282965 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3d140a2880 0x7f3d140a4d40 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f3d28005fd0 tx=0x7f3d28009b00 comp rx=0 tx=0).stop 2024-03-18T16:19:38.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.529+0000 7f3d36a00640 1 -- 172.21.15.39:0/968282965 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3d30074200 msgr2=0x7f3d30199d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:38.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.529+0000 7f3d36a00640 1 --2- 172.21.15.39:0/968282965 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3d30074200 0x7f3d30199d00 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f3d20036040 tx=0x7f3d2002c660 comp rx=0 tx=0).stop 2024-03-18T16:19:38.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.529+0000 7f3d36a00640 1 -- 172.21.15.39:0/968282965 shutdown_connections 2024-03-18T16:19:38.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.529+0000 7f3d36a00640 1 --2- 172.21.15.39:0/968282965 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f3d140a2880 0x7f3d140a4d40 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.529+0000 7f3d36a00640 1 --2- 172.21.15.39:0/968282965 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3d30109690 0x7f3d301a1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.529+0000 7f3d36a00640 1 --2- 172.21.15.39:0/968282965 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3d300757e0 0x7f3d3019a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.529+0000 7f3d36a00640 1 --2- 172.21.15.39:0/968282965 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3d30074200 0x7f3d30199d00 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.529+0000 7f3d36a00640 1 -- 172.21.15.39:0/968282965 >> 172.21.15.39:0/968282965 conn(0x7f3d300fcd50 msgr2=0x7f3d301055e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:38.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.529+0000 7f3d36a00640 1 -- 172.21.15.39:0/968282965 shutdown_connections 2024-03-18T16:19:38.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.529+0000 7f3d36a00640 1 -- 172.21.15.39:0/968282965 wait complete. 2024-03-18T16:19:38.550 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-18T16:19:38.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.780+0000 7fa42a200640 1 -- 172.21.15.39:0/2401564989 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa4241031d0 msgr2=0x7fa424109a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:38.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.780+0000 7fa42a200640 1 --2- 172.21.15.39:0/2401564989 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa4241031d0 0x7fa424109a60 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fa41c009ca0 tx=0x7fa41c02c100 comp rx=0 tx=0).stop 2024-03-18T16:19:38.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.781+0000 7fa42a200640 1 -- 172.21.15.39:0/2401564989 shutdown_connections 2024-03-18T16:19:38.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.781+0000 7fa42a200640 1 --2- 172.21.15.39:0/2401564989 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa4241031d0 0x7fa424109a60 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.781+0000 7fa42a200640 1 --2- 172.21.15.39:0/2401564989 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa424102810 0x7fa424102c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.781+0000 7fa42a200640 1 --2- 172.21.15.39:0/2401564989 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa424101620 0x7fa424101a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.781+0000 7fa42a200640 1 -- 172.21.15.39:0/2401564989 >> 172.21.15.39:0/2401564989 conn(0x7fa4240fcb90 msgr2=0x7fa4240fefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:38.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.781+0000 7fa42a200640 1 -- 172.21.15.39:0/2401564989 shutdown_connections 2024-03-18T16:19:38.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.781+0000 7fa42a200640 1 -- 172.21.15.39:0/2401564989 wait complete. 2024-03-18T16:19:38.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.782+0000 7fa42a200640 1 Processor -- start 2024-03-18T16:19:38.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.782+0000 7fa42a200640 1 -- start start 2024-03-18T16:19:38.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.783+0000 7fa42a200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa424101620 0x7fa424199a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:38.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.783+0000 7fa42a200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa424102810 0x7fa424199fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:38.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.783+0000 7fa42a200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa4241031d0 0x7fa4241a1030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:38.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.783+0000 7fa42a200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fa42410f4b0 con 0x7fa4241031d0 2024-03-18T16:19:38.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.783+0000 7fa42a200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fa42410f330 con 0x7fa424102810 2024-03-18T16:19:38.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.783+0000 7fa42a200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fa42410f630 con 0x7fa424101620 2024-03-18T16:19:38.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.783+0000 7fa428e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa424101620 0x7fa424199a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:38.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.783+0000 7fa428e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa424101620 0x7fa424199a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:53622/0 (socket says 172.21.15.39:53622) 2024-03-18T16:19:38.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.783+0000 7fa428e00640 1 -- 172.21.15.39:0/4239128191 learned_addr learned my addr 172.21.15.39:0/4239128191 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:19:38.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.783+0000 7fa423e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa424102810 0x7fa424199fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:38.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.783+0000 7fa429800640 1 --2- 172.21.15.39:0/4239128191 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa4241031d0 0x7fa4241a1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:38.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.784+0000 7fa428e00640 1 -- 172.21.15.39:0/4239128191 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa424102810 msgr2=0x7fa424199fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:38.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.784+0000 7fa428e00640 1 --2- 172.21.15.39:0/4239128191 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa424102810 0x7fa424199fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.784+0000 7fa428e00640 1 -- 172.21.15.39:0/4239128191 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa4241031d0 msgr2=0x7fa4241a1030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:38.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.784+0000 7fa428e00640 1 --2- 172.21.15.39:0/4239128191 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa4241031d0 0x7fa4241a1030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.784+0000 7fa428e00640 1 -- 172.21.15.39:0/4239128191 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4241a16a0 con 0x7fa424101620 2024-03-18T16:19:38.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.784+0000 7fa428e00640 1 --2- 172.21.15.39:0/4239128191 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa424101620 0x7fa424199a80 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fa41800e9c0 tx=0x7fa41800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:38.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.784+0000 7fa421600640 1 -- 172.21.15.39:0/4239128191 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa41800cd80 con 0x7fa424101620 2024-03-18T16:19:38.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.785+0000 7fa421600640 1 -- 172.21.15.39:0/4239128191 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa4180035f0 con 0x7fa424101620 2024-03-18T16:19:38.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.785+0000 7fa421600640 1 -- 172.21.15.39:0/4239128191 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa418010690 con 0x7fa424101620 2024-03-18T16:19:38.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.785+0000 7fa42a200640 1 -- 172.21.15.39:0/4239128191 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4241a1990 con 0x7fa424101620 2024-03-18T16:19:38.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.785+0000 7fa42a200640 1 -- 172.21.15.39:0/4239128191 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4241a1e90 con 0x7fa424101620 2024-03-18T16:19:38.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.788+0000 7fa42e57e740 1 Processor -- start 2024-03-18T16:19:38.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.788+0000 7fa42e57e740 1 -- start start 2024-03-18T16:19:38.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.788+0000 7fa42e57e740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556c71a45a90 0x556c71a45eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:38.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.788+0000 7fa42e57e740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556c71a46480 0x556c71a4af80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:38.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.788+0000 7fa42e57e740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556c71a4b5f0 0x556c71a4da80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:38.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.788+0000 7fa42e57e740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x556c71a0d3b0 con 0x556c71a4b5f0 2024-03-18T16:19:38.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.788+0000 7fa42e57e740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x556c719aa450 con 0x556c71a46480 2024-03-18T16:19:38.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.788+0000 7fa42e57e740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x556c71952cb0 con 0x556c71a45a90 2024-03-18T16:19:38.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.789+0000 7fa423e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556c71a4b5f0 0x556c71a4da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:38.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.789+0000 7fa429800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556c71a45a90 0x556c71a45eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:38.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.789+0000 7fa423e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556c71a4b5f0 0x556c71a4da80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51380/0 (socket says 172.21.15.39:51380) 2024-03-18T16:19:38.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.789+0000 7fa423e00640 1 -- 172.21.15.39:0/3289266265 learned_addr learned my addr 172.21.15.39:0/3289266265 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:19:38.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.789+0000 7fa429800640 1 --2- 172.21.15.39:0/3289266265 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556c71a45a90 0x556c71a45eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:53636/0 (socket says 172.21.15.39:53636) 2024-03-18T16:19:38.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.789+0000 7fa423e00640 1 -- 172.21.15.39:0/3289266265 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556c71a45a90 msgr2=0x556c71a45eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:38.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.789+0000 7fa423e00640 1 --2- 172.21.15.39:0/3289266265 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556c71a45a90 0x556c71a45eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.789+0000 7fa423e00640 1 -- 172.21.15.39:0/3289266265 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556c71a46480 msgr2=0x556c71a4af80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:19:38.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.789+0000 7fa423e00640 1 --2- 172.21.15.39:0/3289266265 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556c71a46480 0x556c71a4af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.789+0000 7fa423e00640 1 -- 172.21.15.39:0/3289266265 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556c719741f0 con 0x556c71a4b5f0 2024-03-18T16:19:38.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.789+0000 7fa429800640 1 --2- 172.21.15.39:0/3289266265 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556c71a45a90 0x556c71a45eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:19:38.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.789+0000 7fa423e00640 1 --2- 172.21.15.39:0/3289266265 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556c71a4b5f0 0x556c71a4da80 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fa41000bc90 tx=0x7fa4100107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf613fe895d385da server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:38.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.790+0000 7fa421600640 1 -- 172.21.15.39:0/4239128191 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fa4180037a0 con 0x7fa424101620 2024-03-18T16:19:38.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.790+0000 7fa416000640 1 -- 172.21.15.39:0/3289266265 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa410005220 con 0x556c71a4b5f0 2024-03-18T16:19:38.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.790+0000 7fa416000640 1 -- 172.21.15.39:0/3289266265 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa41000be40 con 0x556c71a4b5f0 2024-03-18T16:19:38.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.790+0000 7fa416000640 1 -- 172.21.15.39:0/3289266265 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa410011840 con 0x556c71a4b5f0 2024-03-18T16:19:38.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.791+0000 7fa42e57e740 1 -- 172.21.15.39:0/3289266265 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556c71a4b5f0 msgr2=0x556c71a4da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:38.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.791+0000 7fa42e57e740 1 --2- 172.21.15.39:0/3289266265 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556c71a4b5f0 0x556c71a4da80 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fa41000bc90 tx=0x7fa4100107a0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.791+0000 7fa42e57e740 1 -- 172.21.15.39:0/3289266265 shutdown_connections 2024-03-18T16:19:38.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.791+0000 7fa42e57e740 1 --2- 172.21.15.39:0/3289266265 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556c71a46480 0x556c71a4af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.791+0000 7fa42e57e740 1 --2- 172.21.15.39:0/3289266265 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556c71a4b5f0 0x556c71a4da80 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.795 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.791+0000 7fa42e57e740 1 --2- 172.21.15.39:0/3289266265 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556c71a45a90 0x556c71a45eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.791+0000 7fa42e57e740 1 -- 172.21.15.39:0/3289266265 >> 172.21.15.39:0/3289266265 conn(0x556c71a41000 msgr2=0x556c71a41400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:38.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.791+0000 7fa42e57e740 1 -- 172.21.15.39:0/3289266265 shutdown_connections 2024-03-18T16:19:38.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.791+0000 7fa42e57e740 1 -- 172.21.15.39:0/3289266265 wait complete. 2024-03-18T16:19:38.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.792+0000 7fa42e57e740 1 -- start start 2024-03-18T16:19:38.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.792+0000 7fa42e57e740 1 Processor -- start 2024-03-18T16:19:38.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.793+0000 7fa421600640 1 --2- 172.21.15.39:0/4239128191 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa40c0a2710 0x7fa40c0a4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:38.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.793+0000 7fa42e57e740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556c71a45a90 0x556c71a4fe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:38.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.793+0000 7fa42e57e740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556c71a46480 0x556c71a50340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:38.811 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.793+0000 7fa421600640 1 -- 172.21.15.39:0/4239128191 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fa4180cf150 con 0x7fa424101620 2024-03-18T16:19:38.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.793+0000 7fa42e57e740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556c71a4b5f0 0x556c71a57380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:38.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.793+0000 7fa42e57e740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x556c71966110 con 0x556c71a4b5f0 2024-03-18T16:19:38.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.793+0000 7fa42e57e740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x556c719a64f0 con 0x556c71a45a90 2024-03-18T16:19:38.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.793+0000 7fa42e57e740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x556c71a4e5a0 con 0x556c71a46480 2024-03-18T16:19:38.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.793+0000 7fa42e57e740 10 client.0 _ll_register_callbacks cb 0x556c71953b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:19:38.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.793+0000 7fa428e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556c71a45a90 0x556c71a4fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:38.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.793+0000 7fa428e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556c71a45a90 0x556c71a4fe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:53370/0 (socket says 172.21.15.39:53370) 2024-03-18T16:19:38.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.793+0000 7fa428e00640 1 -- 172.21.15.39:0/1420219606 learned_addr learned my addr 172.21.15.39:0/1420219606 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:19:38.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.794+0000 7fa423e00640 1 --2- 172.21.15.39:0/1420219606 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556c71a46480 0x556c71a50340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:38.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.794+0000 7fa429800640 1 --2- 172.21.15.39:0/4239128191 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa40c0a2710 0x7fa40c0a4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:38.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.794+0000 7fa428e00640 1 -- 172.21.15.39:0/1420219606 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556c71a46480 msgr2=0x556c71a50340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:38.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.794+0000 7fa428e00640 1 --2- 172.21.15.39:0/1420219606 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556c71a46480 0x556c71a50340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.794+0000 7fa428e00640 1 -- 172.21.15.39:0/1420219606 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556c71a4b5f0 msgr2=0x556c71a57380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:38.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.794+0000 7fa428e00640 1 --2- 172.21.15.39:0/1420219606 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556c71a4b5f0 0x556c71a57380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.794+0000 7fa428e00640 1 -- 172.21.15.39:0/1420219606 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556c716be3f0 con 0x556c71a45a90 2024-03-18T16:19:38.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.794+0000 7fa428e00640 1 --2- 172.21.15.39:0/1420219606 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556c71a45a90 0x556c71a4fe00 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fa4180a39d0 tx=0x7fa41809b180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c83f101a278c404 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:38.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.795+0000 7fa40aa00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:19:38.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.795+0000 7fa40aa00640 1 -- 172.21.15.39:0/1420219606 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4180b4060 con 0x556c71a45a90 2024-03-18T16:19:38.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.795+0000 7fa42e57e740 1 -- 172.21.15.39:0/1420219606 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x556c719b7990 con 0x556c71a45a90 2024-03-18T16:19:38.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.795+0000 7fa40aa00640 1 -- 172.21.15.39:0/1420219606 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fa418003b10 con 0x556c71a45a90 2024-03-18T16:19:38.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.795+0000 7fa429800640 1 --2- 172.21.15.39:0/4239128191 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa40c0a2710 0x7fa40c0a4bd0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fa41c02c640 tx=0x7fa41c0323c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:38.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.795+0000 7fa40aa00640 1 -- 172.21.15.39:0/1420219606 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fa41809f8b0 con 0x556c71a45a90 2024-03-18T16:19:38.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.795+0000 7fa42e57e740 10 client.27536 fetch_fsmap learned FSMap version 44 2024-03-18T16:19:38.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.795+0000 7fa42e57e740 1 -- 172.21.15.39:0/1420219606 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x556c717887d0 con 0x556c71a45a90 2024-03-18T16:19:38.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.795+0000 7fa40aa00640 1 -- 172.21.15.39:0/1420219606 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x556c719b7990 con 0x556c71a45a90 2024-03-18T16:19:38.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.796+0000 7fa40aa00640 1 -- 172.21.15.39:0/1420219606 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7fa4180afb40 con 0x556c71a45a90 2024-03-18T16:19:38.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.796+0000 7fa42e57e740 10 client.27536 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:19:38.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.796+0000 7fa42e57e740 10 client.27536 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-18T16:19:38.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.796+0000 7fa42e57e740 20 client.27536 populate_metadata read hostname 'smithi039' 2024-03-18T16:19:38.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.796+0000 7fa42e57e740 1 --2- 172.21.15.39:0/1420219606 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x556c71a5bb80 0x556c71a5dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:38.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.796+0000 7fa42e57e740 4 client.27536 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:19:38.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.796+0000 7fa42e57e740 1 -- 172.21.15.39:0/1420219606 --> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556c719b1ce0 con 0x556c71a5bb80 2024-03-18T16:19:38.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.796+0000 7fa423e00640 1 --2- 172.21.15.39:0/1420219606 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x556c71a5bb80 0x556c71a5dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:38.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.797+0000 7fa423e00640 1 --2- 172.21.15.39:0/1420219606 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x556c71a5bb80 0x556c71a5dfb0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=29df48ce5ccd6571 server_cookie=a9a2c758551262a1 in_seq=0 out_seq=0 2024-03-18T16:19:38.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.797+0000 7fa40aa00640 10 client.27536 ms_handle_connect on v2:172.21.15.196:6836/895247251 2024-03-18T16:19:38.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.798+0000 7fa40aa00640 1 -- 172.21.15.39:0/1420219606 <== mds.2 v2:172.21.15.196:6836/895247251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556c719b1ce0 con 0x556c71a5bb80 2024-03-18T16:19:38.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.798+0000 7fa40aa00640 10 client.27536 handle_command_reply: tid=0 2024-03-18T16:19:38.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.798+0000 7fa42e57e740 1 client.27536 shutdown 2024-03-18T16:19:38.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.798+0000 7fa42e57e740 1 -- 172.21.15.39:0/1420219606 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556c71a45a90 msgr2=0x556c71a4fe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:38.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.798+0000 7fa42e57e740 1 --2- 172.21.15.39:0/1420219606 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556c71a45a90 0x556c71a4fe00 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fa4180a39d0 tx=0x7fa41809b180 comp rx=0 tx=0).stop 2024-03-18T16:19:38.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.800+0000 7fa42e57e740 1 -- 172.21.15.39:0/1420219606 shutdown_connections 2024-03-18T16:19:38.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.800+0000 7fa42e57e740 1 --2- 172.21.15.39:0/1420219606 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x556c71a5bb80 0x556c71a5dfb0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.800+0000 7fa42e57e740 1 --2- 172.21.15.39:0/1420219606 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556c71a4b5f0 0x556c71a57380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.800+0000 7fa42e57e740 1 --2- 172.21.15.39:0/1420219606 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556c71a46480 0x556c71a50340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.800+0000 7fa42e57e740 1 --2- 172.21.15.39:0/1420219606 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556c71a45a90 0x556c71a4fe00 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.800+0000 7fa40aa00640 0 client.27536 ms_handle_reset on v2:172.21.15.196:6836/895247251 2024-03-18T16:19:38.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.800+0000 7fa42e57e740 1 -- 172.21.15.39:0/1420219606 >> 172.21.15.39:0/1420219606 conn(0x556c71a41000 msgr2=0x556c71a40220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:38.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.800+0000 7fa42e57e740 1 -- 172.21.15.39:0/1420219606 shutdown_connections 2024-03-18T16:19:38.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.800+0000 7fa42e57e740 1 -- 172.21.15.39:0/1420219606 wait complete. 2024-03-18T16:19:38.815 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.800+0000 7fa42e57e740 20 client.27536 trim_cache size 0 max 16384 2024-03-18T16:19:38.830 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.828+0000 7fa42e57e740 1 Processor -- start 2024-03-18T16:19:38.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.828+0000 7fa42e57e740 1 -- start start 2024-03-18T16:19:38.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.828+0000 7fa42e57e740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556c71a41000 0x556c71a502a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:38.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.828+0000 7fa42e57e740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556c71a45a90 0x556c71a57340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:38.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.828+0000 7fa42e57e740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556c71a46480 0x556c71a44970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:38.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.828+0000 7fa42e57e740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fa4241ac450 con 0x556c71a46480 2024-03-18T16:19:38.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.828+0000 7fa42e57e740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fa4241adf90 con 0x556c71a41000 2024-03-18T16:19:38.831 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.828+0000 7fa42e57e740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fa4241acb20 con 0x556c71a45a90 2024-03-18T16:19:38.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.828+0000 7fa428e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556c71a41000 0x556c71a502a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:38.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.828+0000 7fa428e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556c71a41000 0x556c71a502a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:53378/0 (socket says 172.21.15.39:53378) 2024-03-18T16:19:38.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.828+0000 7fa428e00640 1 -- 172.21.15.39:0/1005285425 learned_addr learned my addr 172.21.15.39:0/1005285425 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:19:38.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.829+0000 7fa423e00640 1 --2- 172.21.15.39:0/1005285425 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556c71a45a90 0x556c71a57340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:38.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.829+0000 7fa428e00640 1 -- 172.21.15.39:0/1005285425 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556c71a45a90 msgr2=0x556c71a57340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:38.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.829+0000 7fa428e00640 1 --2- 172.21.15.39:0/1005285425 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556c71a45a90 0x556c71a57340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.829+0000 7fa428e00640 1 -- 172.21.15.39:0/1005285425 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556c71a46480 msgr2=0x556c71a44970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:38.832 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.829+0000 7fa428e00640 1 --2- 172.21.15.39:0/1005285425 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556c71a46480 0x556c71a44970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.833 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.829+0000 7fa428e00640 1 -- 172.21.15.39:0/1005285425 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556c71a46ce0 con 0x556c71a41000 2024-03-18T16:19:38.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.831+0000 7fa428e00640 1 --2- 172.21.15.39:0/1005285425 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556c71a41000 0x556c71a502a0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fa41800c6a0 tx=0x7fa418099c70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb273b66acbfadf0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:38.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.832+0000 7fa415600640 1 -- 172.21.15.39:0/1005285425 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4180b4040 con 0x556c71a41000 2024-03-18T16:19:38.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.832+0000 7fa415600640 1 -- 172.21.15.39:0/1005285425 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa41809abb0 con 0x556c71a41000 2024-03-18T16:19:38.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.832+0000 7fa415600640 1 -- 172.21.15.39:0/1005285425 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa418097dd0 con 0x556c71a41000 2024-03-18T16:19:38.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.833+0000 7fa42e57e740 1 -- 172.21.15.39:0/1005285425 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556c71a41000 msgr2=0x556c71a502a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:38.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.833+0000 7fa42e57e740 1 --2- 172.21.15.39:0/1005285425 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556c71a41000 0x556c71a502a0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fa41800c6a0 tx=0x7fa418099c70 comp rx=0 tx=0).stop 2024-03-18T16:19:38.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.833+0000 7fa42e57e740 1 -- 172.21.15.39:0/1005285425 shutdown_connections 2024-03-18T16:19:38.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.833+0000 7fa42e57e740 1 --2- 172.21.15.39:0/1005285425 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556c71a46480 0x556c71a44970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.833+0000 7fa42e57e740 1 --2- 172.21.15.39:0/1005285425 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556c71a45a90 0x556c71a57340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.833+0000 7fa42e57e740 1 --2- 172.21.15.39:0/1005285425 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556c71a41000 0x556c71a502a0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.833+0000 7fa42e57e740 1 -- 172.21.15.39:0/1005285425 >> 172.21.15.39:0/1005285425 conn(0x556c71a47260 msgr2=0x556c71a44260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:38.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.833+0000 7fa42e57e740 1 -- 172.21.15.39:0/1005285425 shutdown_connections 2024-03-18T16:19:38.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.833+0000 7fa42e57e740 1 -- 172.21.15.39:0/1005285425 wait complete. 2024-03-18T16:19:38.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.834+0000 7fa42e57e740 1 -- start start 2024-03-18T16:19:38.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.834+0000 7fa42e57e740 1 Processor -- start 2024-03-18T16:19:38.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.835+0000 7fa42e57e740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556c71a41000 0x556c71a54d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:38.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.835+0000 7fa42e57e740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556c71a45a90 0x556c71a552b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:38.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.835+0000 7fa42e57e740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556c71a46480 0x556c71a562c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:38.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.835+0000 7fa428e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556c71a41000 0x556c71a54d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:38.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.835+0000 7fa429800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556c71a46480 0x556c71a562c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:38.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.835+0000 7fa423e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556c71a45a90 0x556c71a552b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:38.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.835+0000 7fa423e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556c71a45a90 0x556c71a552b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:53392/0 (socket says 172.21.15.39:53392) 2024-03-18T16:19:38.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.835+0000 7fa423e00640 1 -- 172.21.15.39:0/1930679473 learned_addr learned my addr 172.21.15.39:0/1930679473 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:19:38.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.835+0000 7fa42e57e740 1 -- 172.21.15.39:0/1930679473 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fa4241ad8c0 con 0x556c71a46480 2024-03-18T16:19:38.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.835+0000 7fa42e57e740 1 -- 172.21.15.39:0/1930679473 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fa4241ad1f0 con 0x556c71a45a90 2024-03-18T16:19:38.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.835+0000 7fa42e57e740 1 -- 172.21.15.39:0/1930679473 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fa4241aa430 con 0x556c71a41000 2024-03-18T16:19:38.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.835+0000 7fa42e57e740 10 client.0 _ll_register_callbacks cb 0x7fa41809b5e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:19:38.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.836+0000 7fa429800640 1 -- 172.21.15.39:0/1930679473 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556c71a41000 msgr2=0x556c71a54d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:38.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.836+0000 7fa429800640 1 --2- 172.21.15.39:0/1930679473 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556c71a41000 0x556c71a54d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.836+0000 7fa429800640 1 -- 172.21.15.39:0/1930679473 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556c71a45a90 msgr2=0x556c71a552b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:38.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.836+0000 7fa429800640 1 --2- 172.21.15.39:0/1930679473 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556c71a45a90 0x556c71a552b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.836+0000 7fa429800640 1 -- 172.21.15.39:0/1930679473 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556c71a568b0 con 0x556c71a46480 2024-03-18T16:19:38.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.836+0000 7fa428e00640 1 --2- 172.21.15.39:0/1930679473 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556c71a41000 0x556c71a54d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:19:38.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.837+0000 7fa429800640 1 --2- 172.21.15.39:0/1930679473 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556c71a46480 0x556c71a562c0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fa41c037040 tx=0x7fa41c0327d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6343973ec7f061a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:38.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.837+0000 7fa40aa00640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-03-18T16:19:38.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.837+0000 7fa40aa00640 1 -- 172.21.15.39:0/1930679473 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa41c039040 con 0x556c71a46480 2024-03-18T16:19:38.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.837+0000 7fa40aa00640 1 -- 172.21.15.39:0/1930679473 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fa41c0040c0 con 0x556c71a46480 2024-03-18T16:19:38.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.837+0000 7fa42e57e740 1 -- 172.21.15.39:0/1930679473 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa41809abb0 con 0x556c71a46480 2024-03-18T16:19:38.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.838+0000 7fa40aa00640 1 -- 172.21.15.39:0/1930679473 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fa41c05a050 con 0x556c71a46480 2024-03-18T16:19:38.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.838+0000 7fa42e57e740 10 client.16896 fetch_fsmap learned FSMap version 44 2024-03-18T16:19:38.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.838+0000 7fa42e57e740 1 -- 172.21.15.39:0/1930679473 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x556c71aca980 con 0x556c71a46480 2024-03-18T16:19:38.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.838+0000 7fa40aa00640 1 -- 172.21.15.39:0/1930679473 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa41809abb0 con 0x556c71a46480 2024-03-18T16:19:38.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.838+0000 7fa40aa00640 1 -- 172.21.15.39:0/1930679473 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7fa41c04ca40 con 0x556c71a46480 2024-03-18T16:19:38.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.838+0000 7fa42e57e740 10 client.16896 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:19:38.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.838+0000 7fa42e57e740 10 client.16896 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-18T16:19:38.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.838+0000 7fa42e57e740 20 client.16896 populate_metadata read hostname 'smithi039' 2024-03-18T16:19:38.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.838+0000 7fa42e57e740 1 --2- 172.21.15.39:0/1930679473 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x556c71a4b5f0 0x556c71acd410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:38.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.838+0000 7fa42e57e740 4 client.16896 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:19:38.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.838+0000 7fa42e57e740 1 -- 172.21.15.39:0/1930679473 --> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556c7192f560 con 0x556c71a4b5f0 2024-03-18T16:19:38.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.839+0000 7fa428e00640 1 --2- 172.21.15.39:0/1930679473 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x556c71a4b5f0 0x556c71acd410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:38.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.839+0000 7fa428e00640 1 --2- 172.21.15.39:0/1930679473 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x556c71a4b5f0 0x556c71acd410 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=20bc2450bcf855a5 server_cookie=6bde4e49f0543648 in_seq=0 out_seq=0 2024-03-18T16:19:38.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.839+0000 7fa40aa00640 10 client.16896 ms_handle_connect on v2:172.21.15.196:6836/895247251 2024-03-18T16:19:38.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.840+0000 7fa40aa00640 1 -- 172.21.15.39:0/1930679473 <== mds.2 v2:172.21.15.196:6836/895247251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556c7192f560 con 0x556c71a4b5f0 2024-03-18T16:19:38.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.840+0000 7fa40aa00640 10 client.16896 handle_command_reply: tid=0 2024-03-18T16:19:38.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.840+0000 7fa42e57e740 1 client.16896 shutdown 2024-03-18T16:19:38.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.840+0000 7fa42e57e740 1 -- 172.21.15.39:0/1930679473 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556c71a46480 msgr2=0x556c71a562c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:38.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.840+0000 7fa42e57e740 1 --2- 172.21.15.39:0/1930679473 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556c71a46480 0x556c71a562c0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fa41c037040 tx=0x7fa41c0327d0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.840+0000 7fa42e57e740 1 -- 172.21.15.39:0/1930679473 shutdown_connections 2024-03-18T16:19:38.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.840+0000 7fa42e57e740 1 --2- 172.21.15.39:0/1930679473 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556c71a45a90 0x556c71a552b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.840+0000 7fa42e57e740 1 --2- 172.21.15.39:0/1930679473 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x556c71a4b5f0 0x556c71acd410 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.840+0000 7fa42e57e740 1 --2- 172.21.15.39:0/1930679473 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556c71a46480 0x556c71a562c0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.840+0000 7fa42e57e740 1 --2- 172.21.15.39:0/1930679473 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556c71a41000 0x556c71a54d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.840+0000 7fa42e57e740 1 -- 172.21.15.39:0/1930679473 >> 172.21.15.39:0/1930679473 conn(0x556c71a47260 msgr2=0x556c71a50080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:38.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.840+0000 7fa40aa00640 0 client.16896 ms_handle_reset on v2:172.21.15.196:6836/895247251 2024-03-18T16:19:38.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.840+0000 7fa42e57e740 1 -- 172.21.15.39:0/1930679473 shutdown_connections 2024-03-18T16:19:38.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.840+0000 7fa42e57e740 1 -- 172.21.15.39:0/1930679473 wait complete. 2024-03-18T16:19:38.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.841+0000 7fa42e57e740 20 client.16896 trim_cache size 0 max 16384 2024-03-18T16:19:38.846 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:19:38.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.845+0000 7fa42a200640 1 -- 172.21.15.39:0/4239128191 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa40c0a2710 msgr2=0x7fa40c0a4bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:38.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.845+0000 7fa42a200640 1 --2- 172.21.15.39:0/4239128191 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa40c0a2710 0x7fa40c0a4bd0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fa41c02c640 tx=0x7fa41c0323c0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.845+0000 7fa42a200640 1 -- 172.21.15.39:0/4239128191 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa424101620 msgr2=0x7fa424199a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:38.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.845+0000 7fa42a200640 1 --2- 172.21.15.39:0/4239128191 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa424101620 0x7fa424199a80 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fa41800e9c0 tx=0x7fa41800ee80 comp rx=0 tx=0).stop 2024-03-18T16:19:38.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.845+0000 7fa42a200640 1 -- 172.21.15.39:0/4239128191 shutdown_connections 2024-03-18T16:19:38.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.845+0000 7fa42a200640 1 --2- 172.21.15.39:0/4239128191 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa424102810 0x7fa424199fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.845+0000 7fa42a200640 1 --2- 172.21.15.39:0/4239128191 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa40c0a2710 0x7fa40c0a4bd0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.845+0000 7fa42a200640 1 --2- 172.21.15.39:0/4239128191 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa4241031d0 0x7fa4241a1030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.845+0000 7fa42a200640 1 --2- 172.21.15.39:0/4239128191 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa424101620 0x7fa424199a80 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:38.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.845+0000 7fa42a200640 1 -- 172.21.15.39:0/4239128191 >> 172.21.15.39:0/4239128191 conn(0x7fa4240fcb90 msgr2=0x7fa424105ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:38.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.845+0000 7fa42a200640 1 -- 172.21.15.39:0/4239128191 shutdown_connections 2024-03-18T16:19:38.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:38.845+0000 7fa42a200640 1 -- 172.21.15.39:0/4239128191 wait complete. 2024-03-18T16:19:38.871 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-18T16:19:39.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.195+0000 7f1c1aa00640 1 -- 172.21.15.39:0/415452051 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1c14100a60 msgr2=0x7f1c14100ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:39.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.195+0000 7f1c1aa00640 1 --2- 172.21.15.39:0/415452051 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1c14100a60 0x7f1c14100ec0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f1c00009860 tx=0x7f1c0002bc00 comp rx=0 tx=0).stop 2024-03-18T16:19:39.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.195+0000 7f1c1aa00640 1 -- 172.21.15.39:0/415452051 shutdown_connections 2024-03-18T16:19:39.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.195+0000 7f1c1aa00640 1 --2- 172.21.15.39:0/415452051 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1c14101400 0x7f1c14101880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:39.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.195+0000 7f1c1aa00640 1 --2- 172.21.15.39:0/415452051 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1c14100a60 0x7f1c14100ec0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:39.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.195+0000 7f1c1aa00640 1 --2- 172.21.15.39:0/415452051 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1c14100100 0x7f1c14100520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:39.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.195+0000 7f1c1aa00640 1 -- 172.21.15.39:0/415452051 >> 172.21.15.39:0/415452051 conn(0x7f1c140fbc80 msgr2=0x7f1c140fe0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:39.198 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.195+0000 7f1c1aa00640 1 -- 172.21.15.39:0/415452051 shutdown_connections 2024-03-18T16:19:39.199 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.196+0000 7f1c1aa00640 1 -- 172.21.15.39:0/415452051 wait complete. 2024-03-18T16:19:39.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.197+0000 7f1c1aa00640 1 Processor -- start 2024-03-18T16:19:39.200 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.197+0000 7f1c1aa00640 1 -- start start 2024-03-18T16:19:39.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.198+0000 7f1c1aa00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1c14100100 0x7f1c14199d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:39.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.198+0000 7f1c1aa00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1c14100a60 0x7f1c1419a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:39.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.198+0000 7f1c1aa00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1c14101400 0x7f1c141a1260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:39.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.198+0000 7f1c1aa00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f1c1410f670 con 0x7f1c14100100 2024-03-18T16:19:39.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.198+0000 7f1c1aa00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f1c1410f4f0 con 0x7f1c14100a60 2024-03-18T16:19:39.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.198+0000 7f1c1aa00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f1c1410f7f0 con 0x7f1c14101400 2024-03-18T16:19:39.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.198+0000 7f1c13e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1c14100100 0x7f1c14199d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:39.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.198+0000 7f1c13e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1c14100100 0x7f1c14199d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51426/0 (socket says 172.21.15.39:51426) 2024-03-18T16:19:39.202 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.198+0000 7f1c13e00640 1 -- 172.21.15.39:0/2880273225 learned_addr learned my addr 172.21.15.39:0/2880273225 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:19:39.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.198+0000 7f1c13400640 1 --2- 172.21.15.39:0/2880273225 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1c14100a60 0x7f1c1419a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:39.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.198+0000 7f1c18a00640 1 --2- 172.21.15.39:0/2880273225 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1c14101400 0x7f1c141a1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:39.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.199+0000 7f1c13e00640 1 -- 172.21.15.39:0/2880273225 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1c14101400 msgr2=0x7f1c141a1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:39.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.199+0000 7f1c13e00640 1 --2- 172.21.15.39:0/2880273225 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1c14101400 0x7f1c141a1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:39.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.199+0000 7f1c13e00640 1 -- 172.21.15.39:0/2880273225 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1c14100a60 msgr2=0x7f1c1419a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:39.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.199+0000 7f1c13e00640 1 --2- 172.21.15.39:0/2880273225 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1c14100a60 0x7f1c1419a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:39.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.199+0000 7f1c13e00640 1 -- 172.21.15.39:0/2880273225 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c141a1900 con 0x7f1c14100100 2024-03-18T16:19:39.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.199+0000 7f1c13400640 1 --2- 172.21.15.39:0/2880273225 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1c14100a60 0x7f1c1419a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:19:39.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.199+0000 7f1c13e00640 1 --2- 172.21.15.39:0/2880273225 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1c14100100 0x7f1c14199d10 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f1c0400e9f0 tx=0x7f1c0400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:39.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.199+0000 7f1c10c00640 1 -- 172.21.15.39:0/2880273225 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c0400cd80 con 0x7f1c14100100 2024-03-18T16:19:39.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.199+0000 7f1c10c00640 1 -- 172.21.15.39:0/2880273225 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c04010910 con 0x7f1c14100100 2024-03-18T16:19:39.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.199+0000 7f1c1aa00640 1 -- 172.21.15.39:0/2880273225 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c141a1bf0 con 0x7f1c14100100 2024-03-18T16:19:39.203 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.199+0000 7f1c10c00640 1 -- 172.21.15.39:0/2880273225 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c04018990 con 0x7f1c14100100 2024-03-18T16:19:39.204 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.200+0000 7f1c1aa00640 1 -- 172.21.15.39:0/2880273225 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c141a2160 con 0x7f1c14100100 2024-03-18T16:19:39.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.204+0000 7f1c10c00640 1 -- 172.21.15.39:0/2880273225 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f1c04010ac0 con 0x7f1c14100100 2024-03-18T16:19:39.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.204+0000 7f1c1d7e8740 1 Processor -- start 2024-03-18T16:19:39.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.205+0000 7f1c1d7e8740 1 -- start start 2024-03-18T16:19:39.208 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.205+0000 7f1c1d7e8740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5602b7e3e8c0 0x5602b7e3ece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:39.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.205+0000 7f1c1d7e8740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5602b7e3f2b0 0x5602b7e43db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:39.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.205+0000 7f1c1d7e8740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5602b7e44420 0x5602b7e468b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:39.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.205+0000 7f1c1d7e8740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5602b7e063b0 con 0x5602b7e44420 2024-03-18T16:19:39.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.205+0000 7f1c1d7e8740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5602b7da3450 con 0x5602b7e3e8c0 2024-03-18T16:19:39.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.205+0000 7f1c1d7e8740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x5602b7d4bcb0 con 0x5602b7e3f2b0 2024-03-18T16:19:39.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.206+0000 7f1c18a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5602b7e3e8c0 0x5602b7e3ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:39.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.206+0000 7f1c18a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5602b7e3e8c0 0x5602b7e3ece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:53414/0 (socket says 172.21.15.39:53414) 2024-03-18T16:19:39.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.206+0000 7f1c18a00640 1 -- 172.21.15.39:0/3561037267 learned_addr learned my addr 172.21.15.39:0/3561037267 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:19:39.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.206+0000 7f1c10c00640 1 --2- 172.21.15.39:0/2880273225 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1bf80a2880 0x7f1bf80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:39.209 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.206+0000 7f1c13e00640 1 --2- 172.21.15.39:0/3561037267 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5602b7e3f2b0 0x5602b7e43db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:39.210 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.206+0000 7f1c13400640 1 --2- 172.21.15.39:0/3561037267 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5602b7e44420 0x5602b7e468b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:39.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.206+0000 7f1c10c00640 1 -- 172.21.15.39:0/2880273225 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f1c040cf080 con 0x7f1c14100100 2024-03-18T16:19:39.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.206+0000 7f1c13e00640 1 -- 172.21.15.39:0/3561037267 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5602b7e3e8c0 msgr2=0x5602b7e3ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:39.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.206+0000 7f1c13e00640 1 --2- 172.21.15.39:0/3561037267 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5602b7e3e8c0 0x5602b7e3ece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:39.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.206+0000 7f1c13e00640 1 -- 172.21.15.39:0/3561037267 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5602b7e44420 msgr2=0x5602b7e468b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:39.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.206+0000 7f1c13e00640 1 --2- 172.21.15.39:0/3561037267 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5602b7e44420 0x5602b7e468b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:39.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.206+0000 7f1c13e00640 1 -- 172.21.15.39:0/3561037267 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5602b7d6d1f0 con 0x5602b7e3f2b0 2024-03-18T16:19:39.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.207+0000 7f1c18a00640 1 --2- 172.21.15.39:0/2880273225 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1bf80a2880 0x7f1bf80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:39.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.207+0000 7f1c13e00640 1 --2- 172.21.15.39:0/3561037267 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5602b7e3f2b0 0x5602b7e43db0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f1c04098110 tx=0x7f1c04018420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=51330be297d7b0dc server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:39.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.207+0000 7f1c09200640 1 -- 172.21.15.39:0/3561037267 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c0409b740 con 0x5602b7e3f2b0 2024-03-18T16:19:39.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.207+0000 7f1c09200640 1 -- 172.21.15.39:0/3561037267 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c040a3e30 con 0x5602b7e3f2b0 2024-03-18T16:19:39.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.207+0000 7f1c09200640 1 -- 172.21.15.39:0/3561037267 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c040a3c00 con 0x5602b7e3f2b0 2024-03-18T16:19:39.211 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.208+0000 7f1c1d7e8740 1 -- 172.21.15.39:0/3561037267 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5602b7e3f2b0 msgr2=0x5602b7e43db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:39.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.208+0000 7f1c18a00640 1 --2- 172.21.15.39:0/2880273225 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1bf80a2880 0x7f1bf80a4d40 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x5602b7e3fb10 tx=0x7f1c0c004ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:39.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.208+0000 7f1c1d7e8740 1 --2- 172.21.15.39:0/3561037267 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5602b7e3f2b0 0x5602b7e43db0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f1c04098110 tx=0x7f1c04018420 comp rx=0 tx=0).stop 2024-03-18T16:19:39.212 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.208+0000 7f1c1d7e8740 1 -- 172.21.15.39:0/3561037267 shutdown_connections 2024-03-18T16:19:39.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.208+0000 7f1c1d7e8740 1 --2- 172.21.15.39:0/3561037267 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5602b7e44420 0x5602b7e468b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:39.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.208+0000 7f1c1d7e8740 1 --2- 172.21.15.39:0/3561037267 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5602b7e3f2b0 0x5602b7e43db0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:39.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.208+0000 7f1c1d7e8740 1 --2- 172.21.15.39:0/3561037267 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5602b7e3e8c0 0x5602b7e3ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:39.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.209+0000 7f1c1d7e8740 1 -- 172.21.15.39:0/3561037267 >> 172.21.15.39:0/3561037267 conn(0x5602b7e39e30 msgr2=0x5602b7e3a230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:39.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.209+0000 7f1c1d7e8740 1 -- 172.21.15.39:0/3561037267 shutdown_connections 2024-03-18T16:19:39.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.209+0000 7f1c1d7e8740 1 -- 172.21.15.39:0/3561037267 wait complete. 2024-03-18T16:19:39.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.209+0000 7f1c1d7e8740 1 -- start start 2024-03-18T16:19:39.213 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.210+0000 7f1c1d7e8740 1 Processor -- start 2024-03-18T16:19:39.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.210+0000 7f1c1d7e8740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5602b7e3e8c0 0x5602b7e48b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:39.214 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.211+0000 7f1c1d7e8740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5602b7e3f2b0 0x5602b7e49070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:39.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.211+0000 7f1c1d7e8740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5602b7e44420 0x5602b7e500b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:39.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.211+0000 7f1c1d7e8740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5602b7d5f110 con 0x5602b7e44420 2024-03-18T16:19:39.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.211+0000 7f1c1d7e8740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5602b7d9f4f0 con 0x5602b7e3e8c0 2024-03-18T16:19:39.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.211+0000 7f1c1d7e8740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x5602b7e47310 con 0x5602b7e3f2b0 2024-03-18T16:19:39.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.211+0000 7f1c1d7e8740 10 client.0 _ll_register_callbacks cb 0x5602b7d4cb70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:19:39.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.211+0000 7f1c13e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5602b7e3e8c0 0x5602b7e48b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:39.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.211+0000 7f1c13400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5602b7e3f2b0 0x5602b7e49070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:39.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.211+0000 7f1c13400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5602b7e3f2b0 0x5602b7e49070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:53708/0 (socket says 172.21.15.39:53708) 2024-03-18T16:19:39.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.211+0000 7f1c13400640 1 -- 172.21.15.39:0/2557262856 learned_addr learned my addr 172.21.15.39:0/2557262856 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:19:39.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.211+0000 7f1c18a00640 1 --2- 172.21.15.39:0/2557262856 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5602b7e44420 0x5602b7e500b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:39.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.212+0000 7f1c13400640 1 -- 172.21.15.39:0/2557262856 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5602b7e3e8c0 msgr2=0x5602b7e48b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:39.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.212+0000 7f1c13400640 1 --2- 172.21.15.39:0/2557262856 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5602b7e3e8c0 0x5602b7e48b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:39.215 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.212+0000 7f1c13400640 1 -- 172.21.15.39:0/2557262856 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5602b7e44420 msgr2=0x5602b7e500b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:39.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.212+0000 7f1c13400640 1 --2- 172.21.15.39:0/2557262856 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5602b7e44420 0x5602b7e500b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:39.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.212+0000 7f1c13400640 1 -- 172.21.15.39:0/2557262856 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5602b7ab73f0 con 0x5602b7e3f2b0 2024-03-18T16:19:39.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.212+0000 7f1c13e00640 1 --2- 172.21.15.39:0/2557262856 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5602b7e3e8c0 0x5602b7e48b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:19:39.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.212+0000 7f1c13400640 1 --2- 172.21.15.39:0/2557262856 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5602b7e3f2b0 0x5602b7e49070 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f1c000057c0 tx=0x7f1c00004520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ea5066555f9f17b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:39.216 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.212+0000 7f1bfd600640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:19:39.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.212+0000 7f1bfd600640 1 -- 172.21.15.39:0/2557262856 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c00042070 con 0x5602b7e3f2b0 2024-03-18T16:19:39.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.212+0000 7f1bfd600640 1 -- 172.21.15.39:0/2557262856 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f1c0002c7a0 con 0x5602b7e3f2b0 2024-03-18T16:19:39.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.213+0000 7f1c1d7e8740 1 -- 172.21.15.39:0/2557262856 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1c040a3e30 con 0x5602b7e3f2b0 2024-03-18T16:19:39.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.213+0000 7f1bfd600640 1 -- 172.21.15.39:0/2557262856 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f1c0004e050 con 0x5602b7e3f2b0 2024-03-18T16:19:39.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.213+0000 7f1c1d7e8740 10 client.27544 fetch_fsmap learned FSMap version 44 2024-03-18T16:19:39.217 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.213+0000 7f1c1d7e8740 1 -- 172.21.15.39:0/2557262856 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x5602b7b817d0 con 0x5602b7e3f2b0 2024-03-18T16:19:39.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.214+0000 7f1bfd600640 1 -- 172.21.15.39:0/2557262856 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c040a3e30 con 0x5602b7e3f2b0 2024-03-18T16:19:39.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.214+0000 7f1bfd600640 1 -- 172.21.15.39:0/2557262856 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7f1c0003e5d0 con 0x5602b7e3f2b0 2024-03-18T16:19:39.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.214+0000 7f1c1d7e8740 10 client.27544 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:19:39.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.214+0000 7f1c1d7e8740 10 client.27544 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T16:19:39.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.214+0000 7f1c1d7e8740 20 client.27544 populate_metadata read hostname 'smithi039' 2024-03-18T16:19:39.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.214+0000 7f1c1d7e8740 1 --2- 172.21.15.39:0/2557262856 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x5602b7e54a70 0x5602b7e56ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:39.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.214+0000 7f1c1d7e8740 4 client.27544 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:19:39.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.214+0000 7f1c1d7e8740 1 -- 172.21.15.39:0/2557262856 --> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5602b7db0990 con 0x5602b7e54a70 2024-03-18T16:19:39.218 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.214+0000 7f1c13e00640 1 --2- 172.21.15.39:0/2557262856 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x5602b7e54a70 0x5602b7e56ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:39.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.215+0000 7f1c13e00640 1 --2- 172.21.15.39:0/2557262856 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x5602b7e54a70 0x5602b7e56ed0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=bf246346df54d8c7 server_cookie=fccd8cc6427a279 in_seq=0 out_seq=0 2024-03-18T16:19:39.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.215+0000 7f1bfd600640 10 client.27544 ms_handle_connect on v2:172.21.15.196:6838/2001427268 2024-03-18T16:19:39.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.216+0000 7f1bfd600640 1 -- 172.21.15.39:0/2557262856 <== mds.1 v2:172.21.15.196:6838/2001427268 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5602b7db0990 con 0x5602b7e54a70 2024-03-18T16:19:39.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.216+0000 7f1bfd600640 10 client.27544 handle_command_reply: tid=0 2024-03-18T16:19:39.219 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.216+0000 7f1c1d7e8740 1 client.27544 shutdown 2024-03-18T16:19:39.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.217+0000 7f1c1d7e8740 1 -- 172.21.15.39:0/2557262856 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5602b7e3f2b0 msgr2=0x5602b7e49070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:39.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.217+0000 7f1c1d7e8740 1 --2- 172.21.15.39:0/2557262856 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5602b7e3f2b0 0x5602b7e49070 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f1c000057c0 tx=0x7f1c00004520 comp rx=0 tx=0).stop 2024-03-18T16:19:39.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.217+0000 7f1c1d7e8740 1 -- 172.21.15.39:0/2557262856 shutdown_connections 2024-03-18T16:19:39.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.217+0000 7f1c1d7e8740 1 --2- 172.21.15.39:0/2557262856 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x5602b7e54a70 0x5602b7e56ed0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:39.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.217+0000 7f1c1d7e8740 1 --2- 172.21.15.39:0/2557262856 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5602b7e44420 0x5602b7e500b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:39.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.217+0000 7f1c1d7e8740 1 --2- 172.21.15.39:0/2557262856 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5602b7e3f2b0 0x5602b7e49070 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:39.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.217+0000 7f1c1d7e8740 1 --2- 172.21.15.39:0/2557262856 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5602b7e3e8c0 0x5602b7e48b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:39.220 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.217+0000 7f1c1d7e8740 1 -- 172.21.15.39:0/2557262856 >> 172.21.15.39:0/2557262856 conn(0x5602b7e39e30 msgr2=0x5602b7e391f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:39.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.217+0000 7f1bfd600640 0 client.27544 ms_handle_reset on v2:172.21.15.196:6838/2001427268 2024-03-18T16:19:39.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.217+0000 7f1c1d7e8740 1 -- 172.21.15.39:0/2557262856 shutdown_connections 2024-03-18T16:19:39.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.217+0000 7f1c1d7e8740 1 -- 172.21.15.39:0/2557262856 wait complete. 2024-03-18T16:19:39.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.218+0000 7f1c1d7e8740 20 client.27544 trim_cache size 0 max 16384 2024-03-18T16:19:39.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:38 smithi196 ceph-mon[36307]: pgmap v709: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 660 KiB/s wr, 175 op/s 2024-03-18T16:19:39.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:38 smithi194 ceph-mon[36200]: pgmap v709: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 660 KiB/s wr, 175 op/s 2024-03-18T16:19:39.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.247+0000 7f1c1d7e8740 1 Processor -- start 2024-03-18T16:19:39.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.247+0000 7f1c1d7e8740 1 -- start start 2024-03-18T16:19:39.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.247+0000 7f1c1d7e8740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5602b7e3e8c0 0x5602b7e48510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:39.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.247+0000 7f1c1d7e8740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5602b7e3f2b0 0x5602b7e50070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:39.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.247+0000 7f1c1d7e8740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5602b7e44420 0x5602b7e38960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:39.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.247+0000 7f1c1d7e8740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f1c141ac670 con 0x5602b7e3e8c0 2024-03-18T16:19:39.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.247+0000 7f1c1d7e8740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f1c141ae1b0 con 0x5602b7e3f2b0 2024-03-18T16:19:39.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.247+0000 7f1c1d7e8740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f1c141acd40 con 0x5602b7e44420 2024-03-18T16:19:39.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.248+0000 7f1c13e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5602b7e3e8c0 0x5602b7e48510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:39.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.248+0000 7f1c13400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5602b7e3f2b0 0x5602b7e50070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:39.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.248+0000 7f1c13e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5602b7e3e8c0 0x5602b7e48510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51462/0 (socket says 172.21.15.39:51462) 2024-03-18T16:19:39.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.248+0000 7f1c13400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5602b7e3f2b0 0x5602b7e50070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:53428/0 (socket says 172.21.15.39:53428) 2024-03-18T16:19:39.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.248+0000 7f1c13e00640 1 -- 172.21.15.39:0/4152400843 learned_addr learned my addr 172.21.15.39:0/4152400843 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:19:39.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.248+0000 7f1c18a00640 1 --2- 172.21.15.39:0/4152400843 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5602b7e44420 0x5602b7e38960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:39.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.248+0000 7f1c13e00640 1 -- 172.21.15.39:0/4152400843 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5602b7e44420 msgr2=0x5602b7e38960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:39.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.248+0000 7f1c13e00640 1 --2- 172.21.15.39:0/4152400843 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5602b7e44420 0x5602b7e38960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:39.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.248+0000 7f1c13e00640 1 -- 172.21.15.39:0/4152400843 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5602b7e3f2b0 msgr2=0x5602b7e50070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:39.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.248+0000 7f1c13e00640 1 --2- 172.21.15.39:0/4152400843 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5602b7e3f2b0 0x5602b7e50070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:39.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.248+0000 7f1c13e00640 1 -- 172.21.15.39:0/4152400843 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5602b7e499d0 con 0x5602b7e3e8c0 2024-03-18T16:19:39.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.248+0000 7f1c13400640 1 --2- 172.21.15.39:0/4152400843 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5602b7e3f2b0 0x5602b7e50070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:19:39.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.249+0000 7f1c13e00640 1 --2- 172.21.15.39:0/4152400843 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5602b7e3e8c0 0x5602b7e48510 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f1c0400e9c0 tx=0x7f1c0409e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f371a01f83cc3374 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:39.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.249+0000 7f1bffe00640 1 -- 172.21.15.39:0/4152400843 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c040a3e30 con 0x5602b7e3e8c0 2024-03-18T16:19:39.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.249+0000 7f1bffe00640 1 -- 172.21.15.39:0/4152400843 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c04099a40 con 0x5602b7e3e8c0 2024-03-18T16:19:39.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.249+0000 7f1bffe00640 1 -- 172.21.15.39:0/4152400843 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c0409ab90 con 0x5602b7e3e8c0 2024-03-18T16:19:39.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.250+0000 7f1c1d7e8740 1 -- 172.21.15.39:0/4152400843 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5602b7e3e8c0 msgr2=0x5602b7e48510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:39.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.250+0000 7f1c1d7e8740 1 --2- 172.21.15.39:0/4152400843 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5602b7e3e8c0 0x5602b7e48510 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f1c0400e9c0 tx=0x7f1c0409e040 comp rx=0 tx=0).stop 2024-03-18T16:19:39.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.250+0000 7f1c1d7e8740 1 -- 172.21.15.39:0/4152400843 shutdown_connections 2024-03-18T16:19:39.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.250+0000 7f1c1d7e8740 1 --2- 172.21.15.39:0/4152400843 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5602b7e3f2b0 0x5602b7e50070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:39.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.250+0000 7f1c1d7e8740 1 --2- 172.21.15.39:0/4152400843 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5602b7e44420 0x5602b7e38960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:39.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.250+0000 7f1c1d7e8740 1 --2- 172.21.15.39:0/4152400843 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5602b7e3e8c0 0x5602b7e48510 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:39.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.250+0000 7f1c1d7e8740 1 -- 172.21.15.39:0/4152400843 >> 172.21.15.39:0/4152400843 conn(0x5602b7e39e30 msgr2=0x5602b7e47e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:39.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.250+0000 7f1c1d7e8740 1 -- 172.21.15.39:0/4152400843 shutdown_connections 2024-03-18T16:19:39.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.250+0000 7f1c1d7e8740 1 -- 172.21.15.39:0/4152400843 wait complete. 2024-03-18T16:19:39.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.251+0000 7f1c1d7e8740 1 -- start start 2024-03-18T16:19:39.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.251+0000 7f1c1d7e8740 1 Processor -- start 2024-03-18T16:19:39.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.251+0000 7f1c1d7e8740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5602b7e3f2b0 0x5602b7e4d470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:39.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.251+0000 7f1c1d7e8740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5602b7e44420 0x5602b7e4d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:39.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.251+0000 7f1c1d7e8740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5602b7e4e990 0x5602b7e4ee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:39.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.251+0000 7f1c1d7e8740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f1c141adae0 con 0x5602b7e4e990 2024-03-18T16:19:39.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.251+0000 7f1c1d7e8740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f1c141ad410 con 0x5602b7e3f2b0 2024-03-18T16:19:39.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.251+0000 7f1c1d7e8740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f1c141aa650 con 0x5602b7e44420 2024-03-18T16:19:39.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.251+0000 7f1c1d7e8740 10 client.0 _ll_register_callbacks cb 0x7f1c04005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:19:39.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.252+0000 7f1c13e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5602b7e3f2b0 0x5602b7e4d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:39.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.252+0000 7f1c13e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5602b7e3f2b0 0x5602b7e4d470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:53442/0 (socket says 172.21.15.39:53442) 2024-03-18T16:19:39.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.252+0000 7f1c13400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5602b7e44420 0x5602b7e4d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:39.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.252+0000 7f1c13e00640 1 -- 172.21.15.39:0/3797303916 learned_addr learned my addr 172.21.15.39:0/3797303916 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:19:39.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.252+0000 7f1c18a00640 1 --2- 172.21.15.39:0/3797303916 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5602b7e4e990 0x5602b7e4ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:39.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.252+0000 7f1c13e00640 1 -- 172.21.15.39:0/3797303916 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5602b7e44420 msgr2=0x5602b7e4d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:39.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.252+0000 7f1c13e00640 1 --2- 172.21.15.39:0/3797303916 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5602b7e44420 0x5602b7e4d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:39.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.252+0000 7f1c13e00640 1 -- 172.21.15.39:0/3797303916 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5602b7e4e990 msgr2=0x5602b7e4ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:39.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.252+0000 7f1c13e00640 1 --2- 172.21.15.39:0/3797303916 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5602b7e4e990 0x5602b7e4ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:39.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.252+0000 7f1c13e00640 1 -- 172.21.15.39:0/3797303916 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5602b7e4f350 con 0x5602b7e3f2b0 2024-03-18T16:19:39.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.252+0000 7f1c13e00640 1 --2- 172.21.15.39:0/3797303916 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5602b7e3f2b0 0x5602b7e4d470 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f1c04098110 tx=0x7f1c040973a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eded53c13879befa server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:39.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.252+0000 7f1bfd600640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:19:39.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.253+0000 7f1bfd600640 1 -- 172.21.15.39:0/3797303916 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c04086040 con 0x5602b7e3f2b0 2024-03-18T16:19:39.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.253+0000 7f1c1d7e8740 1 -- 172.21.15.39:0/3797303916 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1c04099a40 con 0x5602b7e3f2b0 2024-03-18T16:19:39.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.253+0000 7f1bfd600640 1 -- 172.21.15.39:0/3797303916 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f1c040b42a0 con 0x5602b7e3f2b0 2024-03-18T16:19:39.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.253+0000 7f1bfd600640 1 -- 172.21.15.39:0/3797303916 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f1c0409b740 con 0x5602b7e3f2b0 2024-03-18T16:19:39.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.253+0000 7f1c1d7e8740 10 client.27560 fetch_fsmap learned FSMap version 44 2024-03-18T16:19:39.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.253+0000 7f1c1d7e8740 1 -- 172.21.15.39:0/3797303916 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x5602b7ec3680 con 0x5602b7e3f2b0 2024-03-18T16:19:39.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.253+0000 7f1bfd600640 1 -- 172.21.15.39:0/3797303916 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c04099a40 con 0x5602b7e3f2b0 2024-03-18T16:19:39.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.253+0000 7f1bfd600640 1 -- 172.21.15.39:0/3797303916 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7f1c04097490 con 0x5602b7e3f2b0 2024-03-18T16:19:39.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.253+0000 7f1c1d7e8740 10 client.27560 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:19:39.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.253+0000 7f1c1d7e8740 10 client.27560 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T16:19:39.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.253+0000 7f1c1d7e8740 20 client.27560 populate_metadata read hostname 'smithi039' 2024-03-18T16:19:39.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.254+0000 7f1c1d7e8740 1 --2- 172.21.15.39:0/3797303916 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x5602b7ec3be0 0x5602b7ec5fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:39.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.254+0000 7f1c1d7e8740 4 client.27560 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:19:39.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.254+0000 7f1c1d7e8740 1 -- 172.21.15.39:0/3797303916 --> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5602b7daace0 con 0x5602b7ec3be0 2024-03-18T16:19:39.259 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.254+0000 7f1c13400640 1 --2- 172.21.15.39:0/3797303916 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x5602b7ec3be0 0x5602b7ec5fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:39.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.255+0000 7f1c13400640 1 --2- 172.21.15.39:0/3797303916 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x5602b7ec3be0 0x5602b7ec5fd0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=eaee54df5e56aa39 server_cookie=6cc8d08fb4c76fcf in_seq=0 out_seq=0 2024-03-18T16:19:39.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.255+0000 7f1bfd600640 10 client.27560 ms_handle_connect on v2:172.21.15.196:6838/2001427268 2024-03-18T16:19:39.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.255+0000 7f1bfd600640 1 -- 172.21.15.39:0/3797303916 <== mds.1 v2:172.21.15.196:6838/2001427268 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5602b7daace0 con 0x5602b7ec3be0 2024-03-18T16:19:39.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.255+0000 7f1bfd600640 10 client.27560 handle_command_reply: tid=0 2024-03-18T16:19:39.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.255+0000 7f1c1d7e8740 1 client.27560 shutdown 2024-03-18T16:19:39.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.256+0000 7f1c1d7e8740 1 -- 172.21.15.39:0/3797303916 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5602b7e3f2b0 msgr2=0x5602b7e4d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:39.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.256+0000 7f1c1d7e8740 1 --2- 172.21.15.39:0/3797303916 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5602b7e3f2b0 0x5602b7e4d470 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f1c04098110 tx=0x7f1c040973a0 comp rx=0 tx=0).stop 2024-03-18T16:19:39.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.256+0000 7f1c1d7e8740 1 -- 172.21.15.39:0/3797303916 shutdown_connections 2024-03-18T16:19:39.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.256+0000 7f1c1d7e8740 1 --2- 172.21.15.39:0/3797303916 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x5602b7ec3be0 0x5602b7ec5fd0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:39.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.256+0000 7f1bfd600640 0 client.27560 ms_handle_reset on v2:172.21.15.196:6838/2001427268 2024-03-18T16:19:39.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.256+0000 7f1c1d7e8740 1 --2- 172.21.15.39:0/3797303916 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5602b7e4e990 0x5602b7e4ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:39.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.256+0000 7f1c1d7e8740 1 --2- 172.21.15.39:0/3797303916 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5602b7e44420 0x5602b7e4d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:39.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.256+0000 7f1c1d7e8740 1 --2- 172.21.15.39:0/3797303916 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5602b7e3f2b0 0x5602b7e4d470 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:39.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.256+0000 7f1c1d7e8740 1 -- 172.21.15.39:0/3797303916 >> 172.21.15.39:0/3797303916 conn(0x5602b7e39e30 msgr2=0x5602b7e38960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:39.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.256+0000 7f1c1d7e8740 1 -- 172.21.15.39:0/3797303916 shutdown_connections 2024-03-18T16:19:39.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.256+0000 7f1c1d7e8740 1 -- 172.21.15.39:0/3797303916 wait complete. 2024-03-18T16:19:39.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.257+0000 7f1c1d7e8740 20 client.27560 trim_cache size 0 max 16384 2024-03-18T16:19:39.261 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:19:39.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.258+0000 7f1c1aa00640 1 -- 172.21.15.39:0/2880273225 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1bf80a2880 msgr2=0x7f1bf80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:39.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.258+0000 7f1c1aa00640 1 --2- 172.21.15.39:0/2880273225 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1bf80a2880 0x7f1bf80a4d40 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x5602b7e3fb10 tx=0x7f1c0c004ab0 comp rx=0 tx=0).stop 2024-03-18T16:19:39.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.259+0000 7f1c1aa00640 1 -- 172.21.15.39:0/2880273225 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1c14100100 msgr2=0x7f1c14199d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:39.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.259+0000 7f1c1aa00640 1 --2- 172.21.15.39:0/2880273225 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1c14100100 0x7f1c14199d10 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f1c0400e9f0 tx=0x7f1c0400eeb0 comp rx=0 tx=0).stop 2024-03-18T16:19:39.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.259+0000 7f1c1aa00640 1 -- 172.21.15.39:0/2880273225 shutdown_connections 2024-03-18T16:19:39.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.259+0000 7f1c1aa00640 1 --2- 172.21.15.39:0/2880273225 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1c14100a60 0x7f1c1419a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:39.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.259+0000 7f1c1aa00640 1 --2- 172.21.15.39:0/2880273225 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1bf80a2880 0x7f1bf80a4d40 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:39.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.259+0000 7f1c1aa00640 1 --2- 172.21.15.39:0/2880273225 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1c14101400 0x7f1c141a1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:39.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.259+0000 7f1c1aa00640 1 --2- 172.21.15.39:0/2880273225 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1c14100100 0x7f1c14199d10 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:39.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.259+0000 7f1c1aa00640 1 -- 172.21.15.39:0/2880273225 >> 172.21.15.39:0/2880273225 conn(0x7f1c140fbc80 msgr2=0x7f1c140fd780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:39.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.260+0000 7f1c1aa00640 1 -- 172.21.15.39:0/2880273225 shutdown_connections 2024-03-18T16:19:39.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.260+0000 7f1c1aa00640 1 -- 172.21.15.39:0/2880273225 wait complete. 2024-03-18T16:19:39.280 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:38 smithi039 ceph-mon[31721]: pgmap v709: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 660 KiB/s wr, 175 op/s 2024-03-18T16:19:39.283 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-18T16:19:39.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.510+0000 7f084a200640 1 -- 172.21.15.39:0/1963444678 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0844103a20 msgr2=0x7f0844103e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:39.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.510+0000 7f084a200640 1 --2- 172.21.15.39:0/1963444678 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0844103a20 0x7f0844103e40 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f0834009860 tx=0x7f083402bb70 comp rx=0 tx=0).stop 2024-03-18T16:19:39.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.511+0000 7f084a200640 1 -- 172.21.15.39:0/1963444678 shutdown_connections 2024-03-18T16:19:39.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.511+0000 7f084a200640 1 --2- 172.21.15.39:0/1963444678 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f08441055d0 0x7f084410be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:39.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.511+0000 7f084a200640 1 --2- 172.21.15.39:0/1963444678 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0844104c10 0x7f0844105090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:39.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.511+0000 7f084a200640 1 --2- 172.21.15.39:0/1963444678 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0844103a20 0x7f0844103e40 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:39.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.511+0000 7f084a200640 1 -- 172.21.15.39:0/1963444678 >> 172.21.15.39:0/1963444678 conn(0x7f08440fef90 msgr2=0x7f08441013d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:39.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.511+0000 7f084a200640 1 -- 172.21.15.39:0/1963444678 shutdown_connections 2024-03-18T16:19:39.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.511+0000 7f084a200640 1 -- 172.21.15.39:0/1963444678 wait complete. 2024-03-18T16:19:39.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.512+0000 7f084a200640 1 Processor -- start 2024-03-18T16:19:39.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.512+0000 7f084a200640 1 -- start start 2024-03-18T16:19:39.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.513+0000 7f084a200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0844103a20 0x7f0844199d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:39.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.513+0000 7f084a200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0844104c10 0x7f084419a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:39.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.513+0000 7f084a200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f08441055d0 0x7f08441a1310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:39.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.513+0000 7f084a200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f084410f690 con 0x7f0844103a20 2024-03-18T16:19:39.516 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.513+0000 7f084a200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f084410f510 con 0x7f0844104c10 2024-03-18T16:19:39.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.513+0000 7f084a200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f084410f810 con 0x7f08441055d0 2024-03-18T16:19:39.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.513+0000 7f0843400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0844103a20 0x7f0844199d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:39.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.513+0000 7f0842a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0844104c10 0x7f084419a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:39.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.513+0000 7f0842a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0844104c10 0x7f084419a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:53458/0 (socket says 172.21.15.39:53458) 2024-03-18T16:19:39.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.513+0000 7f0842a00640 1 -- 172.21.15.39:0/3511089018 learned_addr learned my addr 172.21.15.39:0/3511089018 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:19:39.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.513+0000 7f0843e00640 1 --2- 172.21.15.39:0/3511089018 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f08441055d0 0x7f08441a1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:39.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.514+0000 7f0842a00640 1 -- 172.21.15.39:0/3511089018 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f08441055d0 msgr2=0x7f08441a1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:39.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.514+0000 7f0842a00640 1 --2- 172.21.15.39:0/3511089018 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f08441055d0 0x7f08441a1310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:39.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.514+0000 7f0842a00640 1 -- 172.21.15.39:0/3511089018 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0844103a20 msgr2=0x7f0844199d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:39.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.514+0000 7f0842a00640 1 --2- 172.21.15.39:0/3511089018 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0844103a20 0x7f0844199d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:39.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.514+0000 7f0842a00640 1 -- 172.21.15.39:0/3511089018 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08441a1980 con 0x7f0844104c10 2024-03-18T16:19:39.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.514+0000 7f0842a00640 1 --2- 172.21.15.39:0/3511089018 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0844104c10 0x7f084419a2a0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f083000e9f0 tx=0x7f083000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:39.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.514+0000 7f083be00640 1 -- 172.21.15.39:0/3511089018 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f083000ccc0 con 0x7f0844104c10 2024-03-18T16:19:39.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.514+0000 7f083be00640 1 -- 172.21.15.39:0/3511089018 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0830005500 con 0x7f0844104c10 2024-03-18T16:19:39.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.514+0000 7f083be00640 1 -- 172.21.15.39:0/3511089018 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0830010690 con 0x7f0844104c10 2024-03-18T16:19:39.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.514+0000 7f084a200640 1 -- 172.21.15.39:0/3511089018 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08441a1c70 con 0x7f0844104c10 2024-03-18T16:19:39.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.515+0000 7f084a200640 1 -- 172.21.15.39:0/3511089018 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08441a2170 con 0x7f0844104c10 2024-03-18T16:19:39.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.520+0000 7f083be00640 1 -- 172.21.15.39:0/3511089018 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f08300046c0 con 0x7f0844104c10 2024-03-18T16:19:39.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.521+0000 7f084ce6a740 1 Processor -- start 2024-03-18T16:19:39.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.523+0000 7f083be00640 1 --2- 172.21.15.39:0/3511089018 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f08280a2880 0x7f08280a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:39.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.523+0000 7f084ce6a740 1 -- start start 2024-03-18T16:19:39.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.523+0000 7f083be00640 1 -- 172.21.15.39:0/3511089018 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f08300cf000 con 0x7f0844104c10 2024-03-18T16:19:39.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.523+0000 7f0843e00640 1 --2- 172.21.15.39:0/3511089018 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f08280a2880 0x7f08280a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:39.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.523+0000 7f084ce6a740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556c71c810d0 0x556c71c814f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:39.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.523+0000 7f084ce6a740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556c71c81ac0 0x556c71c865c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:39.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.523+0000 7f084ce6a740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556c71c786b0 0x556c71c7ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:39.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.523+0000 7f084ce6a740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x556c71c403b0 con 0x556c71c81ac0 2024-03-18T16:19:39.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.523+0000 7f084ce6a740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x556c71bdd450 con 0x556c71c786b0 2024-03-18T16:19:39.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.523+0000 7f084ce6a740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x556c71b85cb0 con 0x556c71c810d0 2024-03-18T16:19:39.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.524+0000 7f0843400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556c71c810d0 0x556c71c814f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:39.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.524+0000 7f0843e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556c71c786b0 0x556c71c7ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:39.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.524+0000 7f0843400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556c71c810d0 0x556c71c814f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:53772/0 (socket says 172.21.15.39:53772) 2024-03-18T16:19:39.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.524+0000 7f0843400640 1 -- 172.21.15.39:0/1699261735 learned_addr learned my addr 172.21.15.39:0/1699261735 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:19:39.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.524+0000 7f0843e00640 1 -- 172.21.15.39:0/1699261735 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556c71c810d0 msgr2=0x556c71c814f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:39.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.524+0000 7f0843e00640 1 --2- 172.21.15.39:0/1699261735 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556c71c810d0 0x556c71c814f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:39.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.524+0000 7f0843e00640 1 -- 172.21.15.39:0/1699261735 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556c71c81ac0 msgr2=0x556c71c865c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:39.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.524+0000 7f0843e00640 1 --2- 172.21.15.39:0/1699261735 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556c71c81ac0 0x556c71c865c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:39.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.524+0000 7f0843e00640 1 -- 172.21.15.39:0/1699261735 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556c71ba71f0 con 0x556c71c786b0 2024-03-18T16:19:39.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.525+0000 7f0843e00640 1 --2- 172.21.15.39:0/1699261735 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556c71c786b0 0x556c71c7ab40 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f083c0159d0 tx=0x7f083c015e90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ed60f4a33b39e5be server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:39.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.525+0000 7f082fe00640 1 -- 172.21.15.39:0/1699261735 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f083c012a90 con 0x556c71c786b0 2024-03-18T16:19:39.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.525+0000 7f082fe00640 1 -- 172.21.15.39:0/1699261735 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f083c012c30 con 0x556c71c786b0 2024-03-18T16:19:39.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.525+0000 7f082fe00640 1 -- 172.21.15.39:0/1699261735 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f083c013660 con 0x556c71c786b0 2024-03-18T16:19:39.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.527+0000 7f084ce6a740 1 -- 172.21.15.39:0/1699261735 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556c71c786b0 msgr2=0x556c71c7ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:39.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.527+0000 7f084ce6a740 1 --2- 172.21.15.39:0/1699261735 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556c71c786b0 0x556c71c7ab40 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f083c0159d0 tx=0x7f083c015e90 comp rx=0 tx=0).stop 2024-03-18T16:19:39.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.527+0000 7f0843e00640 1 --2- 172.21.15.39:0/3511089018 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f08280a2880 0x7f08280a4d40 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f083c020000 tx=0x7f083c013d10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:39.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.527+0000 7f084ce6a740 1 -- 172.21.15.39:0/1699261735 shutdown_connections 2024-03-18T16:19:39.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.527+0000 7f084ce6a740 1 --2- 172.21.15.39:0/1699261735 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556c71c786b0 0x556c71c7ab40 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:39.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.527+0000 7f084ce6a740 1 --2- 172.21.15.39:0/1699261735 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556c71c81ac0 0x556c71c865c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:39.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.527+0000 7f084ce6a740 1 --2- 172.21.15.39:0/1699261735 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556c71c810d0 0x556c71c814f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:39.530 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.527+0000 7f084ce6a740 1 -- 172.21.15.39:0/1699261735 >> 172.21.15.39:0/1699261735 conn(0x556c71c73e30 msgr2=0x556c71c74230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:39.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.527+0000 7f084ce6a740 1 -- 172.21.15.39:0/1699261735 shutdown_connections 2024-03-18T16:19:39.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.527+0000 7f084ce6a740 1 -- 172.21.15.39:0/1699261735 wait complete. 2024-03-18T16:19:39.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.527+0000 7f084ce6a740 1 -- start start 2024-03-18T16:19:39.531 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.528+0000 7f084ce6a740 1 Processor -- start 2024-03-18T16:19:39.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.528+0000 7f084ce6a740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556c71c786b0 0x556c71c80200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:39.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.528+0000 7f084ce6a740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556c71c810d0 0x556c71c80740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:39.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.528+0000 7f084ce6a740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556c71c81ac0 0x556c71c7cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:39.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.528+0000 7f084ce6a740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x556c71b99110 con 0x556c71c810d0 2024-03-18T16:19:39.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.528+0000 7f084ce6a740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x556c71bd94f0 con 0x556c71c786b0 2024-03-18T16:19:39.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.528+0000 7f084ce6a740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x556c71c7b660 con 0x556c71c81ac0 2024-03-18T16:19:39.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.528+0000 7f084ce6a740 10 client.0 _ll_register_callbacks cb 0x556c71b86b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:19:39.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.529+0000 7f0843400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556c71c786b0 0x556c71c80200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:39.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.529+0000 7f0843400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556c71c786b0 0x556c71c80200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:53480/0 (socket says 172.21.15.39:53480) 2024-03-18T16:19:39.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.529+0000 7f0843400640 1 -- 172.21.15.39:0/3176405166 learned_addr learned my addr 172.21.15.39:0/3176405166 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:19:39.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.529+0000 7f0842a00640 1 --2- 172.21.15.39:0/3176405166 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556c71c810d0 0x556c71c80740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:39.533 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.529+0000 7f0843e00640 1 --2- 172.21.15.39:0/3176405166 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556c71c81ac0 0x556c71c7cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:39.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.529+0000 7f0843400640 1 -- 172.21.15.39:0/3176405166 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556c71c81ac0 msgr2=0x556c71c7cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:39.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.529+0000 7f0843400640 1 --2- 172.21.15.39:0/3176405166 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556c71c81ac0 0x556c71c7cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:39.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.529+0000 7f0843400640 1 -- 172.21.15.39:0/3176405166 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556c71c810d0 msgr2=0x556c71c80740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:39.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.529+0000 7f0843400640 1 --2- 172.21.15.39:0/3176405166 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556c71c810d0 0x556c71c80740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:39.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.529+0000 7f0843400640 1 -- 172.21.15.39:0/3176405166 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556c718f13f0 con 0x556c71c786b0 2024-03-18T16:19:39.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.529+0000 7f0843400640 1 --2- 172.21.15.39:0/3176405166 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556c71c786b0 0x556c71c80200 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f08340349e0 tx=0x7f0834009520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=49e4ed9a81c4108e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:39.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.529+0000 7f082cc00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:19:39.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.530+0000 7f082cc00640 1 -- 172.21.15.39:0/3176405166 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0834043070 con 0x556c71c786b0 2024-03-18T16:19:39.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.530+0000 7f082cc00640 1 -- 172.21.15.39:0/3176405166 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f0834004040 con 0x556c71c786b0 2024-03-18T16:19:39.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.530+0000 7f084ce6a740 1 -- 172.21.15.39:0/3176405166 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f083c012c30 con 0x556c71c786b0 2024-03-18T16:19:39.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.530+0000 7f082cc00640 1 -- 172.21.15.39:0/3176405166 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f083404f050 con 0x556c71c786b0 2024-03-18T16:19:39.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.530+0000 7f084ce6a740 10 client.27572 fetch_fsmap learned FSMap version 44 2024-03-18T16:19:39.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.530+0000 7f084ce6a740 1 -- 172.21.15.39:0/3176405166 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x556c719bb7d0 con 0x556c71c786b0 2024-03-18T16:19:39.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.530+0000 7f082cc00640 1 -- 172.21.15.39:0/3176405166 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f083c012c30 con 0x556c71c786b0 2024-03-18T16:19:39.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.531+0000 7f082cc00640 1 -- 172.21.15.39:0/3176405166 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7f08340326d0 con 0x556c71c786b0 2024-03-18T16:19:39.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.531+0000 7f084ce6a740 10 client.27572 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:19:39.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.531+0000 7f084ce6a740 10 client.27572 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:19:39.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.531+0000 7f084ce6a740 20 client.27572 populate_metadata read hostname 'smithi039' 2024-03-18T16:19:39.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.531+0000 7f084ce6a740 1 --2- 172.21.15.39:0/3176405166 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x556c71c9f770 0x556c71ca1be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:39.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.531+0000 7f084ce6a740 4 client.27572 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:19:39.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.531+0000 7f084ce6a740 1 -- 172.21.15.39:0/3176405166 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556c71bea990 con 0x556c71c9f770 2024-03-18T16:19:39.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.531+0000 7f0842a00640 1 --2- 172.21.15.39:0/3176405166 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x556c71c9f770 0x556c71ca1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:39.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.532+0000 7f0842a00640 1 --2- 172.21.15.39:0/3176405166 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x556c71c9f770 0x556c71ca1be0 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2e830bba00e244c6 server_cookie=8beaa19f8289b1ab in_seq=0 out_seq=0 2024-03-18T16:19:39.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.532+0000 7f082cc00640 10 client.27572 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:19:39.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.532+0000 7f082cc00640 1 -- 172.21.15.39:0/3176405166 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556c71bea990 con 0x556c71c9f770 2024-03-18T16:19:39.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.532+0000 7f082cc00640 10 client.27572 handle_command_reply: tid=0 2024-03-18T16:19:39.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.532+0000 7f084ce6a740 1 client.27572 shutdown 2024-03-18T16:19:39.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.533+0000 7f084ce6a740 1 -- 172.21.15.39:0/3176405166 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556c71c786b0 msgr2=0x556c71c80200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:39.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.533+0000 7f084ce6a740 1 --2- 172.21.15.39:0/3176405166 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556c71c786b0 0x556c71c80200 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f08340349e0 tx=0x7f0834009520 comp rx=0 tx=0).stop 2024-03-18T16:19:39.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.533+0000 7f084ce6a740 1 -- 172.21.15.39:0/3176405166 shutdown_connections 2024-03-18T16:19:39.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.533+0000 7f084ce6a740 1 --2- 172.21.15.39:0/3176405166 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x556c71c9f770 0x556c71ca1be0 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:39.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.533+0000 7f084ce6a740 1 --2- 172.21.15.39:0/3176405166 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556c71c81ac0 0x556c71c7cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:39.537 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.533+0000 7f082cc00640 0 client.27572 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:19:39.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.533+0000 7f084ce6a740 1 --2- 172.21.15.39:0/3176405166 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556c71c810d0 0x556c71c80740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:39.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.533+0000 7f084ce6a740 1 --2- 172.21.15.39:0/3176405166 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556c71c786b0 0x556c71c80200 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:39.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.533+0000 7f084ce6a740 1 -- 172.21.15.39:0/3176405166 >> 172.21.15.39:0/3176405166 conn(0x556c71c73e30 msgr2=0x556c71c731f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:39.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.533+0000 7f084ce6a740 1 -- 172.21.15.39:0/3176405166 shutdown_connections 2024-03-18T16:19:39.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.533+0000 7f084ce6a740 1 -- 172.21.15.39:0/3176405166 wait complete. 2024-03-18T16:19:39.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.533+0000 7f084ce6a740 20 client.27572 trim_cache size 0 max 16384 2024-03-18T16:19:39.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.555+0000 7f084ce6a740 1 Processor -- start 2024-03-18T16:19:39.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.555+0000 7f084ce6a740 1 -- start start 2024-03-18T16:19:39.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.555+0000 7f084ce6a740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556c71c73e30 0x556c71ca74b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:39.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.555+0000 7f084ce6a740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556c71c786b0 0x556c71c801c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:39.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.555+0000 7f084ce6a740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556c71c810d0 0x556c71c80700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:39.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.555+0000 7f084ce6a740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f08441a82c0 con 0x556c71c810d0 2024-03-18T16:19:39.558 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.555+0000 7f084ce6a740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f08441a9e00 con 0x556c71c786b0 2024-03-18T16:19:39.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.555+0000 7f084ce6a740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f08441a8990 con 0x556c71c73e30 2024-03-18T16:19:39.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.556+0000 7f0843400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556c71c73e30 0x556c71ca74b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:39.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.556+0000 7f0843400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556c71c73e30 0x556c71ca74b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:53802/0 (socket says 172.21.15.39:53802) 2024-03-18T16:19:39.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.556+0000 7f0843400640 1 -- 172.21.15.39:0/690082666 learned_addr learned my addr 172.21.15.39:0/690082666 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:19:39.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.556+0000 7f0843e00640 1 --2- 172.21.15.39:0/690082666 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556c71c810d0 0x556c71c80700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:39.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.556+0000 7f0842a00640 1 --2- 172.21.15.39:0/690082666 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556c71c786b0 0x556c71c801c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:39.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.557+0000 7f0843e00640 1 -- 172.21.15.39:0/690082666 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556c71c73e30 msgr2=0x556c71ca74b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:39.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.557+0000 7f0843e00640 1 --2- 172.21.15.39:0/690082666 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556c71c73e30 0x556c71ca74b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:39.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.557+0000 7f0843e00640 1 -- 172.21.15.39:0/690082666 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556c71c786b0 msgr2=0x556c71c801c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:39.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.557+0000 7f0843e00640 1 --2- 172.21.15.39:0/690082666 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556c71c786b0 0x556c71c801c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:39.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.557+0000 7f0843e00640 1 -- 172.21.15.39:0/690082666 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556c71c7b7e0 con 0x556c71c810d0 2024-03-18T16:19:39.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.557+0000 7f0843400640 1 --2- 172.21.15.39:0/690082666 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556c71c73e30 0x556c71ca74b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:19:39.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.557+0000 7f0842a00640 1 --2- 172.21.15.39:0/690082666 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556c71c786b0 0x556c71c801c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:19:39.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.558+0000 7f0843e00640 1 --2- 172.21.15.39:0/690082666 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556c71c810d0 0x556c71c80700 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f083c012760 tx=0x7f083c012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2038c69a5e44cf7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:39.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.558+0000 7f082f400640 1 -- 172.21.15.39:0/690082666 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f083c026070 con 0x556c71c810d0 2024-03-18T16:19:39.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.558+0000 7f082f400640 1 -- 172.21.15.39:0/690082666 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f083c030070 con 0x556c71c810d0 2024-03-18T16:19:39.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.558+0000 7f082f400640 1 -- 172.21.15.39:0/690082666 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f083c003970 con 0x556c71c810d0 2024-03-18T16:19:39.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.559+0000 7f084ce6a740 1 -- 172.21.15.39:0/690082666 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556c71c810d0 msgr2=0x556c71c80700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:39.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.559+0000 7f084ce6a740 1 --2- 172.21.15.39:0/690082666 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556c71c810d0 0x556c71c80700 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f083c012760 tx=0x7f083c012000 comp rx=0 tx=0).stop 2024-03-18T16:19:39.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.559+0000 7f084ce6a740 1 -- 172.21.15.39:0/690082666 shutdown_connections 2024-03-18T16:19:39.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.559+0000 7f084ce6a740 1 --2- 172.21.15.39:0/690082666 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556c71c786b0 0x556c71c801c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:39.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.559+0000 7f084ce6a740 1 --2- 172.21.15.39:0/690082666 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556c71c810d0 0x556c71c80700 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:39.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.559+0000 7f084ce6a740 1 --2- 172.21.15.39:0/690082666 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556c71c73e30 0x556c71ca74b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:39.562 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.559+0000 7f084ce6a740 1 -- 172.21.15.39:0/690082666 >> 172.21.15.39:0/690082666 conn(0x556c71c6f460 msgr2=0x556c71c77090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:39.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.560+0000 7f084ce6a740 1 -- 172.21.15.39:0/690082666 shutdown_connections 2024-03-18T16:19:39.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.560+0000 7f084ce6a740 1 -- 172.21.15.39:0/690082666 wait complete. 2024-03-18T16:19:39.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.561+0000 7f084ce6a740 1 -- start start 2024-03-18T16:19:39.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.561+0000 7f084ce6a740 1 Processor -- start 2024-03-18T16:19:39.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.562+0000 7f084ce6a740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556c71c73e30 0x556c71cb0a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:39.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.562+0000 7f084ce6a740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556c71c786b0 0x556c71cb0f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:39.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.562+0000 7f084ce6a740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556c71c810d0 0x556c71cb1f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:39.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.562+0000 7f084ce6a740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f08441a9730 con 0x556c71c73e30 2024-03-18T16:19:39.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.562+0000 7f084ce6a740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f08441a9060 con 0x556c71c786b0 2024-03-18T16:19:39.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.562+0000 7f084ce6a740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f08441a62c0 con 0x556c71c810d0 2024-03-18T16:19:39.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.562+0000 7f084ce6a740 10 client.0 _ll_register_callbacks cb 0x7f0830005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:19:39.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.562+0000 7f0842a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556c71c786b0 0x556c71cb0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:39.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.562+0000 7f0842a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556c71c786b0 0x556c71cb0f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:53504/0 (socket says 172.21.15.39:53504) 2024-03-18T16:19:39.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.562+0000 7f0842a00640 1 -- 172.21.15.39:0/872116505 learned_addr learned my addr 172.21.15.39:0/872116505 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:19:39.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.562+0000 7f0843e00640 1 --2- 172.21.15.39:0/872116505 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556c71c810d0 0x556c71cb1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:39.565 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.562+0000 7f0843400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556c71c73e30 0x556c71cb0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:39.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.563+0000 7f0842a00640 1 -- 172.21.15.39:0/872116505 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556c71c810d0 msgr2=0x556c71cb1f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:39.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.563+0000 7f0842a00640 1 --2- 172.21.15.39:0/872116505 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556c71c810d0 0x556c71cb1f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:39.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.563+0000 7f0842a00640 1 -- 172.21.15.39:0/872116505 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556c71c73e30 msgr2=0x556c71cb0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:39.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.563+0000 7f0842a00640 1 --2- 172.21.15.39:0/872116505 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556c71c73e30 0x556c71cb0a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:39.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.563+0000 7f0842a00640 1 -- 172.21.15.39:0/872116505 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556c71cb2540 con 0x556c71c786b0 2024-03-18T16:19:39.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.563+0000 7f0842a00640 1 --2- 172.21.15.39:0/872116505 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556c71c786b0 0x556c71cb0f40 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f08300ccae0 tx=0x7f0830086040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8bae5f7a1b456216 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:39.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.563+0000 7f082cc00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:19:39.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.563+0000 7f082cc00640 1 -- 172.21.15.39:0/872116505 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08300b8070 con 0x556c71c786b0 2024-03-18T16:19:39.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.563+0000 7f082cc00640 1 -- 172.21.15.39:0/872116505 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f0830097c80 con 0x556c71c786b0 2024-03-18T16:19:39.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.563+0000 7f084ce6a740 1 -- 172.21.15.39:0/872116505 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f083c030070 con 0x556c71c786b0 2024-03-18T16:19:39.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.564+0000 7f082cc00640 1 -- 172.21.15.39:0/872116505 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f08300ab050 con 0x556c71c786b0 2024-03-18T16:19:39.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.564+0000 7f084ce6a740 10 client.27578 fetch_fsmap learned FSMap version 44 2024-03-18T16:19:39.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.564+0000 7f084ce6a740 1 -- 172.21.15.39:0/872116505 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x556c71d05b50 con 0x556c71c786b0 2024-03-18T16:19:39.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.564+0000 7f082cc00640 1 -- 172.21.15.39:0/872116505 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f083c030070 con 0x556c71c786b0 2024-03-18T16:19:39.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.564+0000 7f082cc00640 1 -- 172.21.15.39:0/872116505 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7f08300a3920 con 0x556c71c786b0 2024-03-18T16:19:39.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.564+0000 7f084ce6a740 10 client.27578 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:19:39.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.564+0000 7f084ce6a740 10 client.27578 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:19:39.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.564+0000 7f084ce6a740 20 client.27578 populate_metadata read hostname 'smithi039' 2024-03-18T16:19:39.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.564+0000 7f084ce6a740 1 --2- 172.21.15.39:0/872116505 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x556c71c81ac0 0x556c71d08540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:39.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.564+0000 7f084ce6a740 4 client.27578 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:19:39.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.564+0000 7f084ce6a740 1 -- 172.21.15.39:0/872116505 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556c71be4ce0 con 0x556c71c81ac0 2024-03-18T16:19:39.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.565+0000 7f0843400640 1 --2- 172.21.15.39:0/872116505 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x556c71c81ac0 0x556c71d08540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:39.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.565+0000 7f0843400640 1 --2- 172.21.15.39:0/872116505 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x556c71c81ac0 0x556c71d08540 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ef499ef757a461a1 server_cookie=13ba46c90611a6fd in_seq=0 out_seq=0 2024-03-18T16:19:39.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.565+0000 7f082cc00640 10 client.27578 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:19:39.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.565+0000 7f082cc00640 1 -- 172.21.15.39:0/872116505 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556c71be4ce0 con 0x556c71c81ac0 2024-03-18T16:19:39.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.565+0000 7f082cc00640 10 client.27578 handle_command_reply: tid=0 2024-03-18T16:19:39.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.565+0000 7f084ce6a740 1 client.27578 shutdown 2024-03-18T16:19:39.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.566+0000 7f084ce6a740 1 -- 172.21.15.39:0/872116505 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556c71c786b0 msgr2=0x556c71cb0f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:39.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.566+0000 7f084ce6a740 1 --2- 172.21.15.39:0/872116505 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556c71c786b0 0x556c71cb0f40 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f08300ccae0 tx=0x7f0830086040 comp rx=0 tx=0).stop 2024-03-18T16:19:39.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.566+0000 7f084ce6a740 1 -- 172.21.15.39:0/872116505 shutdown_connections 2024-03-18T16:19:39.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.566+0000 7f084ce6a740 1 --2- 172.21.15.39:0/872116505 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x556c71c81ac0 0x556c71d08540 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:39.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.566+0000 7f084ce6a740 1 --2- 172.21.15.39:0/872116505 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556c71c786b0 0x556c71cb0f40 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:39.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.566+0000 7f082cc00640 0 client.27578 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:19:39.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.566+0000 7f084ce6a740 1 --2- 172.21.15.39:0/872116505 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556c71c810d0 0x556c71cb1f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:39.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.566+0000 7f084ce6a740 1 --2- 172.21.15.39:0/872116505 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556c71c73e30 0x556c71cb0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:39.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.566+0000 7f084ce6a740 1 -- 172.21.15.39:0/872116505 >> 172.21.15.39:0/872116505 conn(0x556c71c6f460 msgr2=0x556c71c77090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:39.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.566+0000 7f084ce6a740 1 -- 172.21.15.39:0/872116505 shutdown_connections 2024-03-18T16:19:39.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.566+0000 7f084ce6a740 1 -- 172.21.15.39:0/872116505 wait complete. 2024-03-18T16:19:39.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.566+0000 7f084ce6a740 20 client.27578 trim_cache size 0 max 16384 2024-03-18T16:19:39.571 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:19:39.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.567+0000 7f084a200640 1 -- 172.21.15.39:0/3511089018 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f08280a2880 msgr2=0x7f08280a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:39.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.567+0000 7f084a200640 1 --2- 172.21.15.39:0/3511089018 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f08280a2880 0x7f08280a4d40 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f083c020000 tx=0x7f083c013d10 comp rx=0 tx=0).stop 2024-03-18T16:19:39.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.567+0000 7f084a200640 1 -- 172.21.15.39:0/3511089018 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0844104c10 msgr2=0x7f084419a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:39.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.567+0000 7f084a200640 1 --2- 172.21.15.39:0/3511089018 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0844104c10 0x7f084419a2a0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f083000e9f0 tx=0x7f083000eeb0 comp rx=0 tx=0).stop 2024-03-18T16:19:39.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.567+0000 7f084a200640 1 -- 172.21.15.39:0/3511089018 shutdown_connections 2024-03-18T16:19:39.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.567+0000 7f084a200640 1 --2- 172.21.15.39:0/3511089018 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0844104c10 0x7f084419a2a0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:39.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.567+0000 7f084a200640 1 --2- 172.21.15.39:0/3511089018 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f08280a2880 0x7f08280a4d40 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:39.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.567+0000 7f084a200640 1 --2- 172.21.15.39:0/3511089018 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f08441055d0 0x7f08441a1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:39.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.567+0000 7f084a200640 1 --2- 172.21.15.39:0/3511089018 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0844103a20 0x7f0844199d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:39.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.567+0000 7f084a200640 1 -- 172.21.15.39:0/3511089018 >> 172.21.15.39:0/3511089018 conn(0x7f08440fef90 msgr2=0x7f0844107f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:39.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.567+0000 7f084a200640 1 -- 172.21.15.39:0/3511089018 shutdown_connections 2024-03-18T16:19:39.573 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:39.567+0000 7f084a200640 1 -- 172.21.15.39:0/3511089018 wait complete. 2024-03-18T16:19:40.590 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-18T16:19:40.590 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T16:19:40.590 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-18T16:19:40.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.834+0000 7f4ddb600640 1 -- 172.21.15.39:0/4055158085 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4dd4104c10 msgr2=0x7f4dd4105090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:40.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.834+0000 7f4ddb600640 1 --2- 172.21.15.39:0/4055158085 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4dd4104c10 0x7f4dd4105090 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f4dc0009900 tx=0x7f4dc002bc40 comp rx=0 tx=0).stop 2024-03-18T16:19:40.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.835+0000 7f4ddb600640 1 -- 172.21.15.39:0/4055158085 shutdown_connections 2024-03-18T16:19:40.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.835+0000 7f4ddb600640 1 --2- 172.21.15.39:0/4055158085 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4dd41055d0 0x7f4dd410be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:40.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.835+0000 7f4ddb600640 1 --2- 172.21.15.39:0/4055158085 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4dd4104c10 0x7f4dd4105090 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:40.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.835+0000 7f4ddb600640 1 --2- 172.21.15.39:0/4055158085 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4dd4103a20 0x7f4dd4103e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:40.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.835+0000 7f4ddb600640 1 -- 172.21.15.39:0/4055158085 >> 172.21.15.39:0/4055158085 conn(0x7f4dd40fef90 msgr2=0x7f4dd41013d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:40.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.835+0000 7f4ddb600640 1 -- 172.21.15.39:0/4055158085 shutdown_connections 2024-03-18T16:19:40.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.835+0000 7f4ddb600640 1 -- 172.21.15.39:0/4055158085 wait complete. 2024-03-18T16:19:40.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.836+0000 7f4ddb600640 1 Processor -- start 2024-03-18T16:19:40.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.837+0000 7f4ddb600640 1 -- start start 2024-03-18T16:19:40.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.837+0000 7f4ddb600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4dd4103a20 0x7f4dd4199d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:40.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.837+0000 7f4ddb600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4dd4104c10 0x7f4dd419a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:40.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.837+0000 7f4ddb600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4dd41055d0 0x7f4dd41a1310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:40.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.837+0000 7f4ddb600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f4dd410f6f0 con 0x7f4dd4103a20 2024-03-18T16:19:40.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.837+0000 7f4ddb600640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f4dd410f570 con 0x7f4dd41055d0 2024-03-18T16:19:40.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.837+0000 7f4ddb600640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f4dd410f870 con 0x7f4dd4104c10 2024-03-18T16:19:40.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.837+0000 7f4dd3e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4dd4104c10 0x7f4dd419a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:40.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.837+0000 7f4dd8c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4dd4103a20 0x7f4dd4199d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:40.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.838+0000 7f4dd9600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4dd41055d0 0x7f4dd41a1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:40.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.838+0000 7f4dd3e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4dd4104c10 0x7f4dd419a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:53840/0 (socket says 172.21.15.39:53840) 2024-03-18T16:19:40.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.838+0000 7f4dd3e00640 1 -- 172.21.15.39:0/3190520388 learned_addr learned my addr 172.21.15.39:0/3190520388 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:19:40.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.838+0000 7f4dd9600640 1 -- 172.21.15.39:0/3190520388 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4dd4104c10 msgr2=0x7f4dd419a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:40.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.838+0000 7f4dd9600640 1 --2- 172.21.15.39:0/3190520388 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4dd4104c10 0x7f4dd419a2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:40.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.838+0000 7f4dd9600640 1 -- 172.21.15.39:0/3190520388 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4dd4103a20 msgr2=0x7f4dd4199d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:40.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.838+0000 7f4dd9600640 1 --2- 172.21.15.39:0/3190520388 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4dd4103a20 0x7f4dd4199d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:40.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.838+0000 7f4dd9600640 1 -- 172.21.15.39:0/3190520388 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4dd41a1980 con 0x7f4dd41055d0 2024-03-18T16:19:40.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.838+0000 7f4dd9600640 1 --2- 172.21.15.39:0/3190520388 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4dd41055d0 0x7f4dd41a1310 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f4dcc009fd0 tx=0x7f4dcc00ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:40.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.839+0000 7f4dd1600640 1 -- 172.21.15.39:0/3190520388 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dcc00ce20 con 0x7f4dd41055d0 2024-03-18T16:19:40.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.839+0000 7f4dd1600640 1 -- 172.21.15.39:0/3190520388 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4dcc005020 con 0x7f4dd41055d0 2024-03-18T16:19:40.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.839+0000 7f4ddb600640 1 -- 172.21.15.39:0/3190520388 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4dd41a1c70 con 0x7f4dd41055d0 2024-03-18T16:19:40.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.839+0000 7f4dd1600640 1 -- 172.21.15.39:0/3190520388 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dcc010720 con 0x7f4dd41055d0 2024-03-18T16:19:40.843 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.839+0000 7f4ddb600640 1 -- 172.21.15.39:0/3190520388 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4dd41a2190 con 0x7f4dd41055d0 2024-03-18T16:19:40.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.843+0000 7f4dd1600640 1 -- 172.21.15.39:0/3190520388 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f4dcc005540 con 0x7f4dd41055d0 2024-03-18T16:19:40.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.843+0000 7f4dde225740 1 Processor -- start 2024-03-18T16:19:40.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.843+0000 7f4dde225740 1 -- start start 2024-03-18T16:19:40.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.843+0000 7f4dde225740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c5e8bd77e0 0x55c5e8bd7c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:40.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.843+0000 7f4dde225740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c5e8bd81d0 0x55c5e8bdccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:40.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.843+0000 7f4dde225740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c5e8bdd340 0x55c5e8bdf7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:40.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.843+0000 7f4dde225740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55c5e8b3af80 con 0x55c5e8bdd340 2024-03-18T16:19:40.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.843+0000 7f4dde225740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55c5e8ae3ad0 con 0x55c5e8bd81d0 2024-03-18T16:19:40.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.843+0000 7f4dde225740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55c5e8b37020 con 0x55c5e8bd77e0 2024-03-18T16:19:40.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.844+0000 7f4dd3e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c5e8bd77e0 0x55c5e8bd7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:40.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.844+0000 7f4dd3e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c5e8bd77e0 0x55c5e8bd7c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:53842/0 (socket says 172.21.15.39:53842) 2024-03-18T16:19:40.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.844+0000 7f4dd3e00640 1 -- 172.21.15.39:0/4031817282 learned_addr learned my addr 172.21.15.39:0/4031817282 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:19:40.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.844+0000 7f4dd3e00640 1 --2- 172.21.15.39:0/4031817282 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c5e8bdd340 0x55c5e8bdf7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:40.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.844+0000 7f4dd8c00640 1 --2- 172.21.15.39:0/4031817282 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c5e8bd81d0 0x55c5e8bdccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:40.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.844+0000 7f4dd3e00640 1 -- 172.21.15.39:0/4031817282 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c5e8bd81d0 msgr2=0x55c5e8bdccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:40.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.844+0000 7f4dd3e00640 1 --2- 172.21.15.39:0/4031817282 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c5e8bd81d0 0x55c5e8bdccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:40.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.844+0000 7f4dd3e00640 1 -- 172.21.15.39:0/4031817282 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c5e8bdd340 msgr2=0x55c5e8bdf7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:40.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.844+0000 7f4dd3e00640 1 --2- 172.21.15.39:0/4031817282 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c5e8bdd340 0x55c5e8bdf7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:40.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.844+0000 7f4dd3e00640 1 -- 172.21.15.39:0/4031817282 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c5e8afb6f0 con 0x55c5e8bd77e0 2024-03-18T16:19:40.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.845+0000 7f4dd3e00640 1 --2- 172.21.15.39:0/4031817282 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c5e8bd77e0 0x55c5e8bd7c00 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f4dc0005ec0 tx=0x7f4dc0005b70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ea236271c9aa177 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:40.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.845+0000 7f4dd1600640 1 --2- 172.21.15.39:0/3190520388 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f4dbc0a2800 0x7f4dbc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:40.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.845+0000 7f4dc5c00640 1 -- 172.21.15.39:0/4031817282 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dc0047070 con 0x55c5e8bd77e0 2024-03-18T16:19:40.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.845+0000 7f4dc5c00640 1 -- 172.21.15.39:0/4031817282 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4dc00046c0 con 0x55c5e8bd77e0 2024-03-18T16:19:40.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.845+0000 7f4dc5c00640 1 -- 172.21.15.39:0/4031817282 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dc0005150 con 0x55c5e8bd77e0 2024-03-18T16:19:40.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.845+0000 7f4dd1600640 1 -- 172.21.15.39:0/3190520388 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f4dcc0cf420 con 0x7f4dd41055d0 2024-03-18T16:19:40.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.845+0000 7f4dd8c00640 1 --2- 172.21.15.39:0/3190520388 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f4dbc0a2800 0x7f4dbc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:40.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.846+0000 7f4dde225740 1 -- 172.21.15.39:0/4031817282 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c5e8bd77e0 msgr2=0x55c5e8bd7c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:40.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.846+0000 7f4dde225740 1 --2- 172.21.15.39:0/4031817282 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c5e8bd77e0 0x55c5e8bd7c00 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f4dc0005ec0 tx=0x7f4dc0005b70 comp rx=0 tx=0).stop 2024-03-18T16:19:40.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.846+0000 7f4dde225740 1 -- 172.21.15.39:0/4031817282 shutdown_connections 2024-03-18T16:19:40.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.846+0000 7f4dde225740 1 --2- 172.21.15.39:0/4031817282 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c5e8bd81d0 0x55c5e8bdccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:40.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.846+0000 7f4dde225740 1 --2- 172.21.15.39:0/4031817282 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c5e8bdd340 0x55c5e8bdf7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:40.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.846+0000 7f4dde225740 1 --2- 172.21.15.39:0/4031817282 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c5e8bd77e0 0x55c5e8bd7c00 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:40.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.846+0000 7f4dde225740 1 -- 172.21.15.39:0/4031817282 >> 172.21.15.39:0/4031817282 conn(0x55c5e8b9f930 msgr2=0x55c5e8bd3150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:40.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.846+0000 7f4dd8c00640 1 --2- 172.21.15.39:0/3190520388 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f4dbc0a2800 0x7f4dbc0a4cc0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x55c5e8bd85d0 tx=0x7f4dc8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:40.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.847+0000 7f4dde225740 1 -- 172.21.15.39:0/4031817282 shutdown_connections 2024-03-18T16:19:40.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.847+0000 7f4dde225740 1 -- 172.21.15.39:0/4031817282 wait complete. 2024-03-18T16:19:40.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.847+0000 7f4dde225740 1 -- start start 2024-03-18T16:19:40.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.847+0000 7f4dde225740 1 Processor -- start 2024-03-18T16:19:40.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.848+0000 7f4dde225740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c5e8bd77e0 0x55c5e8be1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:40.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.848+0000 7f4dde225740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c5e8bd81d0 0x55c5e8be2080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:40.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.848+0000 7f4dde225740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c5e8bdd340 0x55c5e8be90c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:40.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.848+0000 7f4dde225740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55c5e8be0290 con 0x55c5e8bd81d0 2024-03-18T16:19:40.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.848+0000 7f4dde225740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55c5e8af6f30 con 0x55c5e8bd77e0 2024-03-18T16:19:40.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.848+0000 7f4dde225740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55c5e8be0410 con 0x55c5e8bdd340 2024-03-18T16:19:40.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.848+0000 7f4dde225740 10 client.0 _ll_register_callbacks cb 0x55c5e8ae4bc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:19:40.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.848+0000 7f4dd8c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c5e8bd77e0 0x55c5e8be1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:40.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.848+0000 7f4dd3e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c5e8bd81d0 0x55c5e8be2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:40.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.848+0000 7f4dd8c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c5e8bd77e0 0x55c5e8be1b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:53548/0 (socket says 172.21.15.39:53548) 2024-03-18T16:19:40.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.848+0000 7f4dd8c00640 1 -- 172.21.15.39:0/2427294708 learned_addr learned my addr 172.21.15.39:0/2427294708 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:19:40.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.848+0000 7f4dd9600640 1 --2- 172.21.15.39:0/2427294708 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c5e8bdd340 0x55c5e8be90c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:40.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.849+0000 7f4dd3e00640 1 -- 172.21.15.39:0/2427294708 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c5e8bdd340 msgr2=0x55c5e8be90c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:40.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.849+0000 7f4dd3e00640 1 --2- 172.21.15.39:0/2427294708 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c5e8bdd340 0x55c5e8be90c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:40.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.849+0000 7f4dd3e00640 1 -- 172.21.15.39:0/2427294708 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c5e8bd77e0 msgr2=0x55c5e8be1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:40.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.849+0000 7f4dd3e00640 1 --2- 172.21.15.39:0/2427294708 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c5e8bd77e0 0x55c5e8be1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:40.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.849+0000 7f4dd3e00640 1 -- 172.21.15.39:0/2427294708 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c5e89177e0 con 0x55c5e8bd81d0 2024-03-18T16:19:40.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.849+0000 7f4dd8c00640 1 --2- 172.21.15.39:0/2427294708 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c5e8bd77e0 0x55c5e8be1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:19:40.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.849+0000 7f4dd3e00640 1 --2- 172.21.15.39:0/2427294708 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c5e8bd81d0 0x55c5e8be2080 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f4dc0005ab0 tx=0x7f4dc002c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1182a324e93f6b4f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:40.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.849+0000 7f4dba000640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-03-18T16:19:40.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.850+0000 7f4dba000640 1 -- 172.21.15.39:0/2427294708 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dc004f020 con 0x55c5e8bd81d0 2024-03-18T16:19:40.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.850+0000 7f4dde225740 1 -- 172.21.15.39:0/2427294708 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4dc00046c0 con 0x55c5e8bd81d0 2024-03-18T16:19:40.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.850+0000 7f4dba000640 1 -- 172.21.15.39:0/2427294708 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f4dc00053c0 con 0x55c5e8bd81d0 2024-03-18T16:19:40.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.850+0000 7f4dba000640 1 -- 172.21.15.39:0/2427294708 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f4dc0057050 con 0x55c5e8bd81d0 2024-03-18T16:19:40.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.850+0000 7f4dde225740 10 client.16941 fetch_fsmap learned FSMap version 44 2024-03-18T16:19:40.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.850+0000 7f4dde225740 1 -- 172.21.15.39:0/2427294708 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x55c5e8977f70 con 0x55c5e8bd81d0 2024-03-18T16:19:40.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.851+0000 7f4dba000640 1 -- 172.21.15.39:0/2427294708 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4dc00046c0 con 0x55c5e8bd81d0 2024-03-18T16:19:40.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.851+0000 7f4dba000640 1 -- 172.21.15.39:0/2427294708 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7f4dc003f410 con 0x55c5e8bd81d0 2024-03-18T16:19:40.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.851+0000 7f4dde225740 10 client.16941 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:19:40.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.851+0000 7f4dde225740 10 client.16941 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:19:40.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.851+0000 7f4dde225740 20 client.16941 populate_metadata read hostname 'smithi039' 2024-03-18T16:19:40.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.851+0000 7f4dde225740 1 --2- 172.21.15.39:0/2427294708 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55c5e8bed5b0 0x55c5e8bef9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:40.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.851+0000 7f4dde225740 4 client.16941 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:19:40.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.851+0000 7f4dde225740 1 -- 172.21.15.39:0/2427294708 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c5e8b484c0 con 0x55c5e8bed5b0 2024-03-18T16:19:40.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.852+0000 7f4dd8c00640 1 --2- 172.21.15.39:0/2427294708 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55c5e8bed5b0 0x55c5e8bef9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:40.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.852+0000 7f4dd8c00640 1 --2- 172.21.15.39:0/2427294708 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55c5e8bed5b0 0x55c5e8bef9a0 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1b65c8cc812d0e5e server_cookie=5967e6d7032a2174 in_seq=0 out_seq=0 2024-03-18T16:19:40.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.853+0000 7f4dba000640 10 client.16941 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:19:40.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.853+0000 7f4dba000640 1 -- 172.21.15.39:0/2427294708 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55c5e8b484c0 con 0x55c5e8bed5b0 2024-03-18T16:19:40.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.853+0000 7f4dba000640 10 client.16941 handle_command_reply: tid=0 2024-03-18T16:19:40.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.854+0000 7f4dde225740 1 client.16941 shutdown 2024-03-18T16:19:40.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.854+0000 7f4dde225740 1 -- 172.21.15.39:0/2427294708 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c5e8bd81d0 msgr2=0x55c5e8be2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:40.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.854+0000 7f4dde225740 1 --2- 172.21.15.39:0/2427294708 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c5e8bd81d0 0x55c5e8be2080 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f4dc0005ab0 tx=0x7f4dc002c890 comp rx=0 tx=0).stop 2024-03-18T16:19:40.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.854+0000 7f4dde225740 1 -- 172.21.15.39:0/2427294708 shutdown_connections 2024-03-18T16:19:40.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.854+0000 7f4dde225740 1 --2- 172.21.15.39:0/2427294708 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55c5e8bed5b0 0x55c5e8bef9a0 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:40.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.854+0000 7f4dde225740 1 --2- 172.21.15.39:0/2427294708 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c5e8bdd340 0x55c5e8be90c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:40.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.854+0000 7f4dde225740 1 --2- 172.21.15.39:0/2427294708 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c5e8bd81d0 0x55c5e8be2080 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:40.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.854+0000 7f4dba000640 0 client.16941 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:19:40.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.854+0000 7f4dde225740 1 --2- 172.21.15.39:0/2427294708 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c5e8bd77e0 0x55c5e8be1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:40.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.854+0000 7f4dde225740 1 -- 172.21.15.39:0/2427294708 >> 172.21.15.39:0/2427294708 conn(0x55c5e8b9f930 msgr2=0x55c5e8bd2350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:40.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.854+0000 7f4dde225740 1 -- 172.21.15.39:0/2427294708 shutdown_connections 2024-03-18T16:19:40.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.854+0000 7f4dde225740 1 -- 172.21.15.39:0/2427294708 wait complete. 2024-03-18T16:19:40.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.855+0000 7f4dde225740 20 client.16941 trim_cache size 0 max 16384 2024-03-18T16:19:40.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.893+0000 7f4dde225740 1 Processor -- start 2024-03-18T16:19:40.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.893+0000 7f4dde225740 1 -- start start 2024-03-18T16:19:40.937 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.893+0000 7f4dde225740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c5e8bd77e0 0x55c5e8bdf730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:40.938 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.893+0000 7f4dde225740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c5e8bd81d0 0x55c5e8be9080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:40.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.893+0000 7f4dde225740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c5e8bdd340 0x55c5e8bd6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:40.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.893+0000 7f4dde225740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f4dd41ac5b0 con 0x55c5e8bdd340 2024-03-18T16:19:40.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.893+0000 7f4dde225740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f4dd41ae0f0 con 0x55c5e8bd77e0 2024-03-18T16:19:40.941 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.893+0000 7f4dde225740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f4dd41acc80 con 0x55c5e8bd81d0 2024-03-18T16:19:40.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.894+0000 7f4dd8c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c5e8bd77e0 0x55c5e8bdf730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:40.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.894+0000 7f4dd3e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c5e8bd81d0 0x55c5e8be9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:40.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.894+0000 7f4dd8c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c5e8bd77e0 0x55c5e8bdf730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:53550/0 (socket says 172.21.15.39:53550) 2024-03-18T16:19:40.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.894+0000 7f4dd8c00640 1 -- 172.21.15.39:0/3378273358 learned_addr learned my addr 172.21.15.39:0/3378273358 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:19:40.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.894+0000 7f4dd9600640 1 --2- 172.21.15.39:0/3378273358 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c5e8bdd340 0x55c5e8bd6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:40.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.894+0000 7f4dd8c00640 1 -- 172.21.15.39:0/3378273358 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c5e8bd81d0 msgr2=0x55c5e8be9080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:40.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.894+0000 7f4dd8c00640 1 --2- 172.21.15.39:0/3378273358 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c5e8bd81d0 0x55c5e8be9080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:40.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.894+0000 7f4dd8c00640 1 -- 172.21.15.39:0/3378273358 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c5e8bdd340 msgr2=0x55c5e8bd6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:40.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.894+0000 7f4dd8c00640 1 --2- 172.21.15.39:0/3378273358 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c5e8bdd340 0x55c5e8bd6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:40.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.894+0000 7f4dd8c00640 1 -- 172.21.15.39:0/3378273358 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c5e880f170 con 0x55c5e8bd77e0 2024-03-18T16:19:40.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.894+0000 7f4dd9600640 1 --2- 172.21.15.39:0/3378273358 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c5e8bdd340 0x55c5e8bd6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:19:40.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.894+0000 7f4dd8c00640 1 --2- 172.21.15.39:0/3378273358 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c5e8bd77e0 0x55c5e8bdf730 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x55c5e8bf0080 tx=0x7f4dc801a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb512da8afd77b2f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:40.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.895+0000 7f4dc5200640 1 -- 172.21.15.39:0/3378273358 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dc8003e30 con 0x55c5e8bd77e0 2024-03-18T16:19:40.942 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.895+0000 7f4dc5200640 1 -- 172.21.15.39:0/3378273358 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4dc802bb90 con 0x55c5e8bd77e0 2024-03-18T16:19:40.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.895+0000 7f4dc5200640 1 -- 172.21.15.39:0/3378273358 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dc80047a0 con 0x55c5e8bd77e0 2024-03-18T16:19:40.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.896+0000 7f4dde225740 1 -- 172.21.15.39:0/3378273358 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c5e8bd77e0 msgr2=0x55c5e8bdf730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:40.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.896+0000 7f4dde225740 1 --2- 172.21.15.39:0/3378273358 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c5e8bd77e0 0x55c5e8bdf730 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x55c5e8bf0080 tx=0x7f4dc801a000 comp rx=0 tx=0).stop 2024-03-18T16:19:40.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.896+0000 7f4dde225740 1 -- 172.21.15.39:0/3378273358 shutdown_connections 2024-03-18T16:19:40.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.896+0000 7f4dde225740 1 --2- 172.21.15.39:0/3378273358 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c5e8bdd340 0x55c5e8bd6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:40.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.896+0000 7f4dde225740 1 --2- 172.21.15.39:0/3378273358 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c5e8bd81d0 0x55c5e8be9080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:40.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.896+0000 7f4dde225740 1 --2- 172.21.15.39:0/3378273358 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c5e8bd77e0 0x55c5e8bdf730 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:40.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.896+0000 7f4dde225740 1 -- 172.21.15.39:0/3378273358 >> 172.21.15.39:0/3378273358 conn(0x55c5e8b9f930 msgr2=0x55c5e8bd94f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:40.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.897+0000 7f4dde225740 1 -- 172.21.15.39:0/3378273358 shutdown_connections 2024-03-18T16:19:40.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.897+0000 7f4dde225740 1 -- 172.21.15.39:0/3378273358 wait complete. 2024-03-18T16:19:40.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.897+0000 7f4dde225740 1 -- start start 2024-03-18T16:19:40.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.897+0000 7f4dde225740 1 Processor -- start 2024-03-18T16:19:40.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.898+0000 7f4dde225740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c5e8bd77e0 0x55c5e8be6960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:40.943 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.898+0000 7f4dde225740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c5e8bd81d0 0x55c5e8be6ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:40.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.898+0000 7f4dde225740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c5e8bdd340 0x55c5e8be7e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:40.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.898+0000 7f4dde225740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f4dd41ada20 con 0x55c5e8bd81d0 2024-03-18T16:19:40.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.898+0000 7f4dde225740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f4dd41ad350 con 0x55c5e8bd77e0 2024-03-18T16:19:40.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.898+0000 7f4dde225740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f4dd41aa590 con 0x55c5e8bdd340 2024-03-18T16:19:40.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.898+0000 7f4dde225740 10 client.0 _ll_register_callbacks cb 0x7f4dc80097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:19:40.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.898+0000 7f4dd8c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c5e8bd77e0 0x55c5e8be6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:40.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.898+0000 7f4dd8c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c5e8bd77e0 0x55c5e8be6960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:53564/0 (socket says 172.21.15.39:53564) 2024-03-18T16:19:40.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.898+0000 7f4dd8c00640 1 -- 172.21.15.39:0/1194027373 learned_addr learned my addr 172.21.15.39:0/1194027373 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:19:40.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.898+0000 7f4dd3e00640 1 --2- 172.21.15.39:0/1194027373 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c5e8bd81d0 0x55c5e8be6ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:40.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.898+0000 7f4dd9600640 1 --2- 172.21.15.39:0/1194027373 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c5e8bdd340 0x55c5e8be7e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:40.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.899+0000 7f4dd8c00640 1 -- 172.21.15.39:0/1194027373 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c5e8bdd340 msgr2=0x55c5e8be7e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:40.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.899+0000 7f4dd8c00640 1 --2- 172.21.15.39:0/1194027373 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c5e8bdd340 0x55c5e8be7e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:40.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.899+0000 7f4dd8c00640 1 -- 172.21.15.39:0/1194027373 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c5e8bd81d0 msgr2=0x55c5e8be6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:40.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.899+0000 7f4dd8c00640 1 --2- 172.21.15.39:0/1194027373 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c5e8bd81d0 0x55c5e8be6ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:40.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.899+0000 7f4dd8c00640 1 -- 172.21.15.39:0/1194027373 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c5e8be8430 con 0x55c5e8bd77e0 2024-03-18T16:19:40.944 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.899+0000 7f4dd9600640 1 --2- 172.21.15.39:0/1194027373 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c5e8bdd340 0x55c5e8be7e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:19:40.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.899+0000 7f4dd8c00640 1 --2- 172.21.15.39:0/1194027373 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c5e8bd77e0 0x55c5e8be6960 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f4dc802d040 tx=0x7f4dc80051a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e111f2f13e9c9bb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:40.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.899+0000 7f4dba000640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:19:40.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.899+0000 7f4dba000640 1 -- 172.21.15.39:0/1194027373 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dc8039070 con 0x55c5e8bd77e0 2024-03-18T16:19:40.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.900+0000 7f4dba000640 1 -- 172.21.15.39:0/1194027373 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f4dc802bd40 con 0x55c5e8bd77e0 2024-03-18T16:19:40.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.900+0000 7f4dde225740 1 -- 172.21.15.39:0/1194027373 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4dc802bb90 con 0x55c5e8bd77e0 2024-03-18T16:19:40.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.900+0000 7f4dba000640 1 -- 172.21.15.39:0/1194027373 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f4dc8045050 con 0x55c5e8bd77e0 2024-03-18T16:19:40.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.900+0000 7f4dde225740 10 client.27596 fetch_fsmap learned FSMap version 44 2024-03-18T16:19:40.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.900+0000 7f4dde225740 1 -- 172.21.15.39:0/1194027373 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x55c5e8c53800 con 0x55c5e8bd77e0 2024-03-18T16:19:40.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.900+0000 7f4dba000640 1 -- 172.21.15.39:0/1194027373 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4dc802bb90 con 0x55c5e8bd77e0 2024-03-18T16:19:40.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.901+0000 7f4dba000640 1 -- 172.21.15.39:0/1194027373 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7f4dc8004db0 con 0x55c5e8bd77e0 2024-03-18T16:19:40.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.901+0000 7f4dde225740 10 client.27596 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:19:40.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.901+0000 7f4dde225740 10 client.27596 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:19:40.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.901+0000 7f4dde225740 20 client.27596 populate_metadata read hostname 'smithi039' 2024-03-18T16:19:40.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.901+0000 7f4dde225740 1 --2- 172.21.15.39:0/1194027373 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55c5e8c54180 0x55c5e8c565a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:40.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.901+0000 7f4dde225740 4 client.27596 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-18T16:19:40.945 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.901+0000 7f4dde225740 1 -- 172.21.15.39:0/1194027373 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55c5e8b42810 con 0x55c5e8c54180 2024-03-18T16:19:40.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.901+0000 7f4dd3e00640 1 --2- 172.21.15.39:0/1194027373 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55c5e8c54180 0x55c5e8c565a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:40.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.902+0000 7f4dd3e00640 1 --2- 172.21.15.39:0/1194027373 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55c5e8c54180 0x55c5e8c565a0 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=12a21f65d8ee9a40 server_cookie=d9b25b253261d4 in_seq=0 out_seq=0 2024-03-18T16:19:40.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.902+0000 7f4dba000640 10 client.27596 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:19:40.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.902+0000 7f4dba000640 1 -- 172.21.15.39:0/1194027373 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55c5e8b42810 con 0x55c5e8c54180 2024-03-18T16:19:40.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.902+0000 7f4dba000640 10 client.27596 handle_command_reply: tid=0 2024-03-18T16:19:40.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.902+0000 7f4dde225740 1 client.27596 shutdown 2024-03-18T16:19:40.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.903+0000 7f4dde225740 1 -- 172.21.15.39:0/1194027373 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c5e8bd77e0 msgr2=0x55c5e8be6960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:40.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.903+0000 7f4dde225740 1 --2- 172.21.15.39:0/1194027373 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c5e8bd77e0 0x55c5e8be6960 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f4dc802d040 tx=0x7f4dc80051a0 comp rx=0 tx=0).stop 2024-03-18T16:19:40.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.903+0000 7f4dde225740 1 -- 172.21.15.39:0/1194027373 shutdown_connections 2024-03-18T16:19:40.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.903+0000 7f4dde225740 1 --2- 172.21.15.39:0/1194027373 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55c5e8c54180 0x55c5e8c565a0 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:40.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.903+0000 7f4dde225740 1 --2- 172.21.15.39:0/1194027373 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55c5e8bdd340 0x55c5e8be7e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:40.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.903+0000 7f4dde225740 1 --2- 172.21.15.39:0/1194027373 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55c5e8bd81d0 0x55c5e8be6ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:40.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.903+0000 7f4dde225740 1 --2- 172.21.15.39:0/1194027373 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c5e8bd77e0 0x55c5e8be6960 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:40.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.903+0000 7f4dde225740 1 -- 172.21.15.39:0/1194027373 >> 172.21.15.39:0/1194027373 conn(0x55c5e8b9f930 msgr2=0x55c5e8bd1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:40.946 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.903+0000 7f4dba000640 0 client.27596 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:19:40.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.904+0000 7f4dde225740 1 -- 172.21.15.39:0/1194027373 shutdown_connections 2024-03-18T16:19:40.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.904+0000 7f4dde225740 1 -- 172.21.15.39:0/1194027373 wait complete. 2024-03-18T16:19:40.947 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.904+0000 7f4dde225740 20 client.27596 trim_cache size 0 max 16384 2024-03-18T16:19:40.947 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-03-18T16:19:40.947 INFO:teuthology.orchestra.run.smithi039.stdout: "return_code": 0, 2024-03-18T16:19:40.947 INFO:teuthology.orchestra.run.smithi039.stdout: "scrub_tag": "5fccb086-7953-49b1-8c46-ce57d303b7a1", 2024-03-18T16:19:40.947 INFO:teuthology.orchestra.run.smithi039.stdout: "mode": "asynchronous" 2024-03-18T16:19:40.947 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-03-18T16:19:40.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.906+0000 7f4ddb600640 1 -- 172.21.15.39:0/3190520388 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f4dbc0a2800 msgr2=0x7f4dbc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:40.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.906+0000 7f4ddb600640 1 --2- 172.21.15.39:0/3190520388 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f4dbc0a2800 0x7f4dbc0a4cc0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x55c5e8bd85d0 tx=0x7f4dc8005f90 comp rx=0 tx=0).stop 2024-03-18T16:19:40.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.906+0000 7f4ddb600640 1 -- 172.21.15.39:0/3190520388 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4dd41055d0 msgr2=0x7f4dd41a1310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:40.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.906+0000 7f4ddb600640 1 --2- 172.21.15.39:0/3190520388 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4dd41055d0 0x7f4dd41a1310 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f4dcc009fd0 tx=0x7f4dcc00ef10 comp rx=0 tx=0).stop 2024-03-18T16:19:40.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.906+0000 7f4ddb600640 1 -- 172.21.15.39:0/3190520388 shutdown_connections 2024-03-18T16:19:40.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.906+0000 7f4ddb600640 1 --2- 172.21.15.39:0/3190520388 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4dd41055d0 0x7f4dd41a1310 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:40.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.906+0000 7f4ddb600640 1 --2- 172.21.15.39:0/3190520388 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f4dbc0a2800 0x7f4dbc0a4cc0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:40.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.906+0000 7f4ddb600640 1 --2- 172.21.15.39:0/3190520388 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4dd4104c10 0x7f4dd419a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:40.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.906+0000 7f4ddb600640 1 --2- 172.21.15.39:0/3190520388 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4dd4103a20 0x7f4dd4199d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:40.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.906+0000 7f4ddb600640 1 -- 172.21.15.39:0/3190520388 >> 172.21.15.39:0/3190520388 conn(0x7f4dd40fef90 msgr2=0x7f4dd4107f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:40.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.906+0000 7f4ddb600640 1 -- 172.21.15.39:0/3190520388 shutdown_connections 2024-03-18T16:19:40.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:40.906+0000 7f4ddb600640 1 -- 172.21.15.39:0/3190520388 wait complete. 2024-03-18T16:19:40.949 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T16:19:40.950 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T16:19:41.207 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.205+0000 7fae37c00640 1 -- 172.21.15.39:0/3498884868 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fae300757c0 msgr2=0x7fae30073c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:41.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.205+0000 7fae37c00640 1 --2- 172.21.15.39:0/3498884868 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fae300757c0 0x7fae30073c10 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fae24009900 tx=0x7fae2402bcd0 comp rx=0 tx=0).stop 2024-03-18T16:19:41.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.205+0000 7fae37c00640 1 -- 172.21.15.39:0/3498884868 shutdown_connections 2024-03-18T16:19:41.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.205+0000 7fae37c00640 1 --2- 172.21.15.39:0/3498884868 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae30109680 0x7fae3010bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:41.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.205+0000 7fae37c00640 1 --2- 172.21.15.39:0/3498884868 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fae300741e0 0x7fae30074640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:41.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.205+0000 7fae37c00640 1 --2- 172.21.15.39:0/3498884868 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fae300757c0 0x7fae30073c10 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:41.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.205+0000 7fae37c00640 1 -- 172.21.15.39:0/3498884868 >> 172.21.15.39:0/3498884868 conn(0x7fae300fcd80 msgr2=0x7fae300ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:41.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.205+0000 7fae37c00640 1 -- 172.21.15.39:0/3498884868 shutdown_connections 2024-03-18T16:19:41.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.205+0000 7fae37c00640 1 -- 172.21.15.39:0/3498884868 wait complete. 2024-03-18T16:19:41.237 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:40 smithi196 ceph-mon[36307]: pgmap v710: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 547 KiB/s wr, 148 op/s 2024-03-18T16:19:41.243 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:40 smithi194 ceph-mon[36200]: pgmap v710: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 547 KiB/s wr, 148 op/s 2024-03-18T16:19:41.245 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.206+0000 7fae37c00640 1 Processor -- start 2024-03-18T16:19:41.245 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.206+0000 7fae37c00640 1 -- start start 2024-03-18T16:19:41.245 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.207+0000 7fae37c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fae300741e0 0x7fae30199d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:41.245 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.207+0000 7fae37c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fae300757c0 0x7fae3019a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:41.245 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.207+0000 7fae37c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae30109680 0x7fae301a12c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:41.245 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.207+0000 7fae37c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fae3010f690 con 0x7fae30109680 2024-03-18T16:19:41.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.207+0000 7fae37c00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fae3010f510 con 0x7fae300741e0 2024-03-18T16:19:41.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.207+0000 7fae37c00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fae3010f810 con 0x7fae300757c0 2024-03-18T16:19:41.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.207+0000 7fae35200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fae300741e0 0x7fae30199d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:41.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.207+0000 7fae2fe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fae300757c0 0x7fae3019a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:41.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.207+0000 7fae35200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fae300741e0 0x7fae30199d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:53574/0 (socket says 172.21.15.39:53574) 2024-03-18T16:19:41.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.207+0000 7fae35200640 1 -- 172.21.15.39:0/3327097819 learned_addr learned my addr 172.21.15.39:0/3327097819 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:19:41.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.207+0000 7fae35c00640 1 --2- 172.21.15.39:0/3327097819 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae30109680 0x7fae301a12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:41.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.208+0000 7fae35200640 1 -- 172.21.15.39:0/3327097819 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fae300757c0 msgr2=0x7fae3019a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:41.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.208+0000 7fae35200640 1 --2- 172.21.15.39:0/3327097819 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fae300757c0 0x7fae3019a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:41.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.208+0000 7fae35200640 1 -- 172.21.15.39:0/3327097819 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae30109680 msgr2=0x7fae301a12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:41.246 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.208+0000 7fae35200640 1 --2- 172.21.15.39:0/3327097819 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae30109680 0x7fae301a12c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:41.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.208+0000 7fae35200640 1 -- 172.21.15.39:0/3327097819 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae301a1960 con 0x7fae300741e0 2024-03-18T16:19:41.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.208+0000 7fae35c00640 1 --2- 172.21.15.39:0/3327097819 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae30109680 0x7fae301a12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:19:41.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.208+0000 7fae35200640 1 --2- 172.21.15.39:0/3327097819 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fae300741e0 0x7fae30199d50 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fae24000c00 tx=0x7fae2402c6f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:41.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.208+0000 7fae2d600640 1 -- 172.21.15.39:0/3327097819 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae24043020 con 0x7fae300741e0 2024-03-18T16:19:41.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.208+0000 7fae2d600640 1 -- 172.21.15.39:0/3327097819 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae240054e0 con 0x7fae300741e0 2024-03-18T16:19:41.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.208+0000 7fae37c00640 1 -- 172.21.15.39:0/3327097819 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae301a1bf0 con 0x7fae300741e0 2024-03-18T16:19:41.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.209+0000 7fae2d600640 1 -- 172.21.15.39:0/3327097819 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae24003c20 con 0x7fae300741e0 2024-03-18T16:19:41.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.209+0000 7fae37c00640 1 -- 172.21.15.39:0/3327097819 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae301a2190 con 0x7fae300741e0 2024-03-18T16:19:41.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.213+0000 7fae3aa2e740 1 Processor -- start 2024-03-18T16:19:41.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.215+0000 7fae3aa2e740 1 -- start start 2024-03-18T16:19:41.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.215+0000 7fae2d600640 1 -- 172.21.15.39:0/3327097819 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fae24046020 con 0x7fae300741e0 2024-03-18T16:19:41.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.215+0000 7fae3aa2e740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561594662810 0x561594662c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:41.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.215+0000 7fae3aa2e740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561594663200 0x561594667d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:41.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.215+0000 7fae3aa2e740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561594668370 0x56159466a800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:41.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.215+0000 7fae3aa2e740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5615945c71f0 con 0x561594663200 2024-03-18T16:19:41.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.215+0000 7fae3aa2e740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x56159456fa80 con 0x561594668370 2024-03-18T16:19:41.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.215+0000 7fae3aa2e740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x5615945c3290 con 0x561594662810 2024-03-18T16:19:41.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.216+0000 7fae35c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561594662810 0x561594662c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:41.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.216+0000 7fae2fe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561594668370 0x56159466a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:41.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.216+0000 7fae35c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561594662810 0x561594662c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:53882/0 (socket says 172.21.15.39:53882) 2024-03-18T16:19:41.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.216+0000 7fae35c00640 1 -- 172.21.15.39:0/2123495373 learned_addr learned my addr 172.21.15.39:0/2123495373 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:19:41.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.216+0000 7fae35200640 1 --2- 172.21.15.39:0/2123495373 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561594663200 0x561594667d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:41.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.216+0000 7fae35c00640 1 -- 172.21.15.39:0/2123495373 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561594668370 msgr2=0x56159466a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:41.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.216+0000 7fae35c00640 1 --2- 172.21.15.39:0/2123495373 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561594668370 0x56159466a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:41.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.216+0000 7fae35c00640 1 -- 172.21.15.39:0/2123495373 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561594663200 msgr2=0x561594667d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:41.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.216+0000 7fae35c00640 1 --2- 172.21.15.39:0/2123495373 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561594663200 0x561594667d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:41.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.216+0000 7fae35c00640 1 -- 172.21.15.39:0/2123495373 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5615943a57d0 con 0x561594662810 2024-03-18T16:19:41.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.216+0000 7fae2fe00640 1 --2- 172.21.15.39:0/2123495373 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561594668370 0x56159466a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:19:41.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.217+0000 7fae35c00640 1 --2- 172.21.15.39:0/2123495373 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561594662810 0x561594662c30 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fae28009fd0 tx=0x7fae280108e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ad583ced42b1a281 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:41.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.217+0000 7fae22000640 1 -- 172.21.15.39:0/2123495373 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae28011760 con 0x561594662810 2024-03-18T16:19:41.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.217+0000 7fae22000640 1 -- 172.21.15.39:0/2123495373 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae28011de0 con 0x561594662810 2024-03-18T16:19:41.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.217+0000 7fae22000640 1 -- 172.21.15.39:0/2123495373 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae2800e3c0 con 0x561594662810 2024-03-18T16:19:41.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.217+0000 7fae2d600640 1 --2- 172.21.15.39:0/3327097819 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fae180a2880 0x7fae180a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:41.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.218+0000 7fae2d600640 1 -- 172.21.15.39:0/3327097819 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fae240f0f90 con 0x7fae300741e0 2024-03-18T16:19:41.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.218+0000 7fae3aa2e740 1 -- 172.21.15.39:0/2123495373 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561594662810 msgr2=0x561594662c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:41.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.218+0000 7fae3aa2e740 1 --2- 172.21.15.39:0/2123495373 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561594662810 0x561594662c30 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fae28009fd0 tx=0x7fae280108e0 comp rx=0 tx=0).stop 2024-03-18T16:19:41.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.218+0000 7fae3aa2e740 1 -- 172.21.15.39:0/2123495373 shutdown_connections 2024-03-18T16:19:41.249 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.218+0000 7fae3aa2e740 1 --2- 172.21.15.39:0/2123495373 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561594668370 0x56159466a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:41.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.218+0000 7fae3aa2e740 1 --2- 172.21.15.39:0/2123495373 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561594663200 0x561594667d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:41.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.218+0000 7fae3aa2e740 1 --2- 172.21.15.39:0/2123495373 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561594662810 0x561594662c30 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:41.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.218+0000 7fae3aa2e740 1 -- 172.21.15.39:0/2123495373 >> 172.21.15.39:0/2123495373 conn(0x56159465dd80 msgr2=0x56159465e180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:41.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.218+0000 7fae3aa2e740 1 -- 172.21.15.39:0/2123495373 shutdown_connections 2024-03-18T16:19:41.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.218+0000 7fae35200640 1 --2- 172.21.15.39:0/3327097819 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fae180a2880 0x7fae180a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:41.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.218+0000 7fae3aa2e740 1 -- 172.21.15.39:0/2123495373 wait complete. 2024-03-18T16:19:41.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.219+0000 7fae3aa2e740 1 -- start start 2024-03-18T16:19:41.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.219+0000 7fae35200640 1 --2- 172.21.15.39:0/3327097819 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fae180a2880 0x7fae180a4d40 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fae240eea70 tx=0x7fae24004ef0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:41.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.219+0000 7fae3aa2e740 1 Processor -- start 2024-03-18T16:19:41.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.220+0000 7fae3aa2e740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561594662810 0x56159466cd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:41.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.220+0000 7fae3aa2e740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561594663200 0x56159466d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:41.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.220+0000 7fae3aa2e740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561594668370 0x561594674310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:41.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.220+0000 7fae3aa2e740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x56159466b320 con 0x561594662810 2024-03-18T16:19:41.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.220+0000 7fae3aa2e740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x561594582ed0 con 0x561594668370 2024-03-18T16:19:41.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.220+0000 7fae3aa2e740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x56159466b4a0 con 0x561594663200 2024-03-18T16:19:41.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.220+0000 7fae3aa2e740 10 client.0 _ll_register_callbacks cb 0x5615945be220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:19:41.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.220+0000 7fae35c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561594662810 0x56159466cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:41.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.220+0000 7fae2fe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561594663200 0x56159466d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:41.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.220+0000 7fae2fe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561594663200 0x56159466d2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:53898/0 (socket says 172.21.15.39:53898) 2024-03-18T16:19:41.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.220+0000 7fae35c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561594662810 0x56159466cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:51630/0 (socket says 172.21.15.39:51630) 2024-03-18T16:19:41.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.220+0000 7fae2fe00640 1 -- 172.21.15.39:0/394425568 learned_addr learned my addr 172.21.15.39:0/394425568 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:19:41.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.221+0000 7fae2fe00640 1 -- 172.21.15.39:0/394425568 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561594668370 msgr2=0x561594674310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:41.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.221+0000 7fae35c00640 1 --2- 172.21.15.39:0/394425568 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561594668370 0x561594674310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:41.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.221+0000 7fae2fe00640 1 --2- 172.21.15.39:0/394425568 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561594668370 0x561594674310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:41.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.221+0000 7fae2fe00640 1 -- 172.21.15.39:0/394425568 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561594662810 msgr2=0x56159466cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:41.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.221+0000 7fae2fe00640 1 --2- 172.21.15.39:0/394425568 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561594662810 0x56159466cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:41.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.221+0000 7fae2fe00640 1 -- 172.21.15.39:0/394425568 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5615942db3f0 con 0x561594663200 2024-03-18T16:19:41.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.221+0000 7fae35c00640 1 --2- 172.21.15.39:0/394425568 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561594668370 0x561594674310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:19:41.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.222+0000 7fae2fe00640 1 --2- 172.21.15.39:0/394425568 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561594663200 0x56159466d2d0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fae1c009c60 tx=0x7fae1c0039c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2336698d05be512d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:41.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.222+0000 7fae16a00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:19:41.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.222+0000 7fae16a00640 1 -- 172.21.15.39:0/394425568 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae1c00f670 con 0x561594663200 2024-03-18T16:19:41.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.222+0000 7fae16a00640 1 -- 172.21.15.39:0/394425568 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fae1c00f810 con 0x561594663200 2024-03-18T16:19:41.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.223+0000 7fae3aa2e740 1 -- 172.21.15.39:0/394425568 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fae28011de0 con 0x561594663200 2024-03-18T16:19:41.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.223+0000 7fae16a00640 1 -- 172.21.15.39:0/394425568 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fae1c00fa10 con 0x561594663200 2024-03-18T16:19:41.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.223+0000 7fae3aa2e740 10 client.27598 fetch_fsmap learned FSMap version 44 2024-03-18T16:19:41.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.223+0000 7fae3aa2e740 1 -- 172.21.15.39:0/394425568 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x56159429b0f0 con 0x561594663200 2024-03-18T16:19:41.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.224+0000 7fae16a00640 1 -- 172.21.15.39:0/394425568 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae28011de0 con 0x561594663200 2024-03-18T16:19:41.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.224+0000 7fae16a00640 1 -- 172.21.15.39:0/394425568 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7fae1c023420 con 0x561594663200 2024-03-18T16:19:41.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.224+0000 7fae3aa2e740 10 client.27598 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:19:41.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.224+0000 7fae3aa2e740 10 client.27598 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:19:41.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.224+0000 7fae3aa2e740 20 client.27598 populate_metadata read hostname 'smithi039' 2024-03-18T16:19:41.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.224+0000 7fae3aa2e740 1 --2- 172.21.15.39:0/394425568 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x561594678ba0 0x56159467b000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:41.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.224+0000 7fae3aa2e740 4 client.27598 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:19:41.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.224+0000 7fae3aa2e740 1 -- 172.21.15.39:0/394425568 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5615945dabe0 con 0x561594678ba0 2024-03-18T16:19:41.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.224+0000 7fae35c00640 1 --2- 172.21.15.39:0/394425568 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x561594678ba0 0x56159467b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:41.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.225+0000 7fae35c00640 1 --2- 172.21.15.39:0/394425568 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x561594678ba0 0x56159467b000 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=806f6c04a47dd616 server_cookie=cc6aa7b0aaa72cb0 in_seq=0 out_seq=0 2024-03-18T16:19:41.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.225+0000 7fae16a00640 10 client.27598 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:19:41.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.226+0000 7fae16a00640 1 -- 172.21.15.39:0/394425568 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5615945dabe0 con 0x561594678ba0 2024-03-18T16:19:41.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.226+0000 7fae16a00640 10 client.27598 handle_command_reply: tid=0 2024-03-18T16:19:41.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.226+0000 7fae3aa2e740 1 client.27598 shutdown 2024-03-18T16:19:41.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.226+0000 7fae3aa2e740 1 -- 172.21.15.39:0/394425568 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561594663200 msgr2=0x56159466d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:41.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.226+0000 7fae3aa2e740 1 --2- 172.21.15.39:0/394425568 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561594663200 0x56159466d2d0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fae1c009c60 tx=0x7fae1c0039c0 comp rx=0 tx=0).stop 2024-03-18T16:19:41.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.227+0000 7fae3aa2e740 1 -- 172.21.15.39:0/394425568 shutdown_connections 2024-03-18T16:19:41.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.227+0000 7fae3aa2e740 1 --2- 172.21.15.39:0/394425568 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x561594678ba0 0x56159467b000 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:41.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.227+0000 7fae3aa2e740 1 --2- 172.21.15.39:0/394425568 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561594668370 0x561594674310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:41.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.227+0000 7fae3aa2e740 1 --2- 172.21.15.39:0/394425568 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561594663200 0x56159466d2d0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:41.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.227+0000 7fae3aa2e740 1 --2- 172.21.15.39:0/394425568 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561594662810 0x56159466cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:41.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.227+0000 7fae3aa2e740 1 -- 172.21.15.39:0/394425568 >> 172.21.15.39:0/394425568 conn(0x56159465dd80 msgr2=0x56159465cec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:41.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.227+0000 7fae16a00640 0 client.27598 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:19:41.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.227+0000 7fae3aa2e740 1 -- 172.21.15.39:0/394425568 shutdown_connections 2024-03-18T16:19:41.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.228+0000 7fae3aa2e740 1 -- 172.21.15.39:0/394425568 wait complete. 2024-03-18T16:19:41.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.228+0000 7fae3aa2e740 20 client.27598 trim_cache size 0 max 16384 2024-03-18T16:19:41.260 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.258+0000 7fae3aa2e740 1 Processor -- start 2024-03-18T16:19:41.261 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.258+0000 7fae3aa2e740 1 -- start start 2024-03-18T16:19:41.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.258+0000 7fae3aa2e740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561594662810 0x5615946647a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:41.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.258+0000 7fae3aa2e740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561594663200 0x561594664ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:41.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.258+0000 7fae3aa2e740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561594668370 0x5615946742d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:41.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.258+0000 7fae3aa2e740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fae301a8390 con 0x561594663200 2024-03-18T16:19:41.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.258+0000 7fae3aa2e740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fae301a9ed0 con 0x561594662810 2024-03-18T16:19:41.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.258+0000 7fae3aa2e740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fae301a8a60 con 0x561594668370 2024-03-18T16:19:41.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.259+0000 7fae35c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561594662810 0x5615946647a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:41.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.259+0000 7fae35c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561594662810 0x5615946647a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:53598/0 (socket says 172.21.15.39:53598) 2024-03-18T16:19:41.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.259+0000 7fae35c00640 1 -- 172.21.15.39:0/1363638522 learned_addr learned my addr 172.21.15.39:0/1363638522 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:19:41.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.259+0000 7fae35c00640 1 --2- 172.21.15.39:0/1363638522 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561594668370 0x5615946742d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:41.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.259+0000 7fae2fe00640 1 --2- 172.21.15.39:0/1363638522 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561594663200 0x561594664ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:41.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.259+0000 7fae35c00640 1 -- 172.21.15.39:0/1363638522 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561594668370 msgr2=0x5615946742d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:41.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.259+0000 7fae35c00640 1 --2- 172.21.15.39:0/1363638522 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561594668370 0x5615946742d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:41.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.259+0000 7fae35c00640 1 -- 172.21.15.39:0/1363638522 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561594663200 msgr2=0x561594664ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:41.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.259+0000 7fae35c00640 1 --2- 172.21.15.39:0/1363638522 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561594663200 0x561594664ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:41.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.259+0000 7fae35c00640 1 -- 172.21.15.39:0/1363638522 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561594658370 con 0x561594662810 2024-03-18T16:19:41.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.260+0000 7fae35c00640 1 --2- 172.21.15.39:0/1363638522 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561594662810 0x5615946647a0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fae28009e10 tx=0x7fae28013040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=24d59d9ff5691955 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:41.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.260+0000 7fae21600640 1 -- 172.21.15.39:0/1363638522 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae28023040 con 0x561594662810 2024-03-18T16:19:41.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.260+0000 7fae21600640 1 -- 172.21.15.39:0/1363638522 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae2801b630 con 0x561594662810 2024-03-18T16:19:41.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.260+0000 7fae21600640 1 -- 172.21.15.39:0/1363638522 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae280117c0 con 0x561594662810 2024-03-18T16:19:41.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.261+0000 7fae3aa2e740 1 -- 172.21.15.39:0/1363638522 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561594662810 msgr2=0x5615946647a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:41.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.261+0000 7fae3aa2e740 1 --2- 172.21.15.39:0/1363638522 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561594662810 0x5615946647a0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fae28009e10 tx=0x7fae28013040 comp rx=0 tx=0).stop 2024-03-18T16:19:41.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.261+0000 7fae3aa2e740 1 -- 172.21.15.39:0/1363638522 shutdown_connections 2024-03-18T16:19:41.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.261+0000 7fae3aa2e740 1 --2- 172.21.15.39:0/1363638522 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561594668370 0x5615946742d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:41.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.261+0000 7fae3aa2e740 1 --2- 172.21.15.39:0/1363638522 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561594663200 0x561594664ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:41.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.261+0000 7fae3aa2e740 1 --2- 172.21.15.39:0/1363638522 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561594662810 0x5615946647a0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:41.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.261+0000 7fae3aa2e740 1 -- 172.21.15.39:0/1363638522 >> 172.21.15.39:0/1363638522 conn(0x56159465dd80 msgr2=0x56159466cd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:41.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.261+0000 7fae3aa2e740 1 -- 172.21.15.39:0/1363638522 shutdown_connections 2024-03-18T16:19:41.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.262+0000 7fae3aa2e740 1 -- 172.21.15.39:0/1363638522 wait complete. 2024-03-18T16:19:41.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.262+0000 7fae3aa2e740 1 -- start start 2024-03-18T16:19:41.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.262+0000 7fae3aa2e740 1 Processor -- start 2024-03-18T16:19:41.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.263+0000 7fae3aa2e740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561594662810 0x561594671750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:41.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.263+0000 7fae3aa2e740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561594663200 0x561594671c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:41.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.263+0000 7fae3aa2e740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561594668370 0x561594672ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:41.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.263+0000 7fae3aa2e740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fae301a9800 con 0x561594663200 2024-03-18T16:19:41.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.263+0000 7fae3aa2e740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fae301a9130 con 0x561594668370 2024-03-18T16:19:41.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.263+0000 7fae3aa2e740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fae301a6370 con 0x561594662810 2024-03-18T16:19:41.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.263+0000 7fae3aa2e740 10 client.0 _ll_register_callbacks cb 0x7fae1c004120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:19:41.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.264+0000 7fae35c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561594662810 0x561594671750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:41.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.264+0000 7fae35c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561594662810 0x561594671750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:53930/0 (socket says 172.21.15.39:53930) 2024-03-18T16:19:41.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.264+0000 7fae35c00640 1 -- 172.21.15.39:0/466681880 learned_addr learned my addr 172.21.15.39:0/466681880 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:19:41.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.264+0000 7fae35c00640 1 --2- 172.21.15.39:0/466681880 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561594668370 0x561594672ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:41.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.264+0000 7fae2fe00640 1 --2- 172.21.15.39:0/466681880 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561594663200 0x561594671c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:41.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.264+0000 7fae35c00640 1 -- 172.21.15.39:0/466681880 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561594668370 msgr2=0x561594672ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:41.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.264+0000 7fae35c00640 1 --2- 172.21.15.39:0/466681880 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561594668370 0x561594672ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:41.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.264+0000 7fae35c00640 1 -- 172.21.15.39:0/466681880 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561594663200 msgr2=0x561594671c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:19:41.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.264+0000 7fae35c00640 1 --2- 172.21.15.39:0/466681880 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561594663200 0x561594671c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:41.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.264+0000 7fae35c00640 1 -- 172.21.15.39:0/466681880 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561594673290 con 0x561594662810 2024-03-18T16:19:41.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.264+0000 7fae2fe00640 1 --2- 172.21.15.39:0/466681880 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561594663200 0x561594671c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:19:41.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.265+0000 7fae35c00640 1 --2- 172.21.15.39:0/466681880 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561594662810 0x561594671750 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fae2801bd80 tx=0x7fae28005f70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9f63768b786c15dc server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:19:41.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.265+0000 7fae16a00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:19:41.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.265+0000 7fae16a00640 1 -- 172.21.15.39:0/466681880 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae2803f060 con 0x561594662810 2024-03-18T16:19:41.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.265+0000 7fae16a00640 1 -- 172.21.15.39:0/466681880 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fae280049c0 con 0x561594662810 2024-03-18T16:19:41.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.265+0000 7fae3aa2e740 1 -- 172.21.15.39:0/466681880 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fae2801b630 con 0x561594662810 2024-03-18T16:19:41.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.265+0000 7fae16a00640 1 -- 172.21.15.39:0/466681880 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7fae2802c3e0 con 0x561594662810 2024-03-18T16:19:41.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.265+0000 7fae3aa2e740 10 client.27604 fetch_fsmap learned FSMap version 44 2024-03-18T16:19:41.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.265+0000 7fae3aa2e740 1 -- 172.21.15.39:0/466681880 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x5615946745a0 con 0x561594662810 2024-03-18T16:19:41.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.266+0000 7fae16a00640 1 -- 172.21.15.39:0/466681880 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fae2801b630 con 0x561594662810 2024-03-18T16:19:41.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.266+0000 7fae16a00640 1 -- 172.21.15.39:0/466681880 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7fae28039040 con 0x561594662810 2024-03-18T16:19:41.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.266+0000 7fae3aa2e740 10 client.27604 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:19:41.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.266+0000 7fae3aa2e740 10 client.27604 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:19:41.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.266+0000 7fae3aa2e740 20 client.27604 populate_metadata read hostname 'smithi039' 2024-03-18T16:19:41.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.266+0000 7fae3aa2e740 1 --2- 172.21.15.39:0/466681880 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5615946df2e0 0x5615946e16d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:19:41.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.266+0000 7fae3aa2e740 4 client.27604 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T16:19:41.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.266+0000 7fae3aa2e740 1 -- 172.21.15.39:0/466681880 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5615945d4730 con 0x5615946df2e0 2024-03-18T16:19:41.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.266+0000 7fae2fe00640 1 --2- 172.21.15.39:0/466681880 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5615946df2e0 0x5615946e16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:19:41.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.267+0000 7fae2fe00640 1 --2- 172.21.15.39:0/466681880 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5615946df2e0 0x5615946e16d0 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b72837db0637ff5e server_cookie=48b177a65387d925 in_seq=0 out_seq=0 2024-03-18T16:19:41.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.267+0000 7fae16a00640 10 client.27604 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:19:41.272 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.267+0000 7fae16a00640 1 -- 172.21.15.39:0/466681880 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5615945d4730 con 0x5615946df2e0 2024-03-18T16:19:41.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.267+0000 7fae16a00640 10 client.27604 handle_command_reply: tid=0 2024-03-18T16:19:41.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.267+0000 7fae3aa2e740 1 client.27604 shutdown 2024-03-18T16:19:41.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.267+0000 7fae3aa2e740 1 -- 172.21.15.39:0/466681880 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561594662810 msgr2=0x561594671750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:41.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.267+0000 7fae3aa2e740 1 --2- 172.21.15.39:0/466681880 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561594662810 0x561594671750 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fae2801bd80 tx=0x7fae28005f70 comp rx=0 tx=0).stop 2024-03-18T16:19:41.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.267+0000 7fae3aa2e740 1 -- 172.21.15.39:0/466681880 shutdown_connections 2024-03-18T16:19:41.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.268+0000 7fae3aa2e740 1 --2- 172.21.15.39:0/466681880 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5615946df2e0 0x5615946e16d0 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:41.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.268+0000 7fae3aa2e740 1 --2- 172.21.15.39:0/466681880 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561594668370 0x561594672ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:41.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.268+0000 7fae3aa2e740 1 --2- 172.21.15.39:0/466681880 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x561594663200 0x561594671c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:41.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.268+0000 7fae3aa2e740 1 --2- 172.21.15.39:0/466681880 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x561594662810 0x561594671750 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:41.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.268+0000 7fae3aa2e740 1 -- 172.21.15.39:0/466681880 >> 172.21.15.39:0/466681880 conn(0x56159465dd80 msgr2=0x56159466cd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:41.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.268+0000 7fae16a00640 0 client.27604 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:19:41.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.268+0000 7fae3aa2e740 1 -- 172.21.15.39:0/466681880 shutdown_connections 2024-03-18T16:19:41.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.268+0000 7fae3aa2e740 1 -- 172.21.15.39:0/466681880 wait complete. 2024-03-18T16:19:41.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.268+0000 7fae3aa2e740 20 client.27604 trim_cache size 0 max 16384 2024-03-18T16:19:41.274 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-03-18T16:19:41.274 INFO:teuthology.orchestra.run.smithi039.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-18T16:19:41.274 INFO:teuthology.orchestra.run.smithi039.stdout: "scrubs": { 2024-03-18T16:19:41.274 INFO:teuthology.orchestra.run.smithi039.stdout: "5fccb086-7953-49b1-8c46-ce57d303b7a1": { 2024-03-18T16:19:41.274 INFO:teuthology.orchestra.run.smithi039.stdout: "path": "/", 2024-03-18T16:19:41.274 INFO:teuthology.orchestra.run.smithi039.stdout: "tag": "5fccb086-7953-49b1-8c46-ce57d303b7a1", 2024-03-18T16:19:41.274 INFO:teuthology.orchestra.run.smithi039.stdout: "options": "recursive,force" 2024-03-18T16:19:41.275 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:19:41.275 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:19:41.275 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-03-18T16:19:41.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.269+0000 7fae37c00640 1 -- 172.21.15.39:0/3327097819 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fae180a2880 msgr2=0x7fae180a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:41.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.269+0000 7fae37c00640 1 --2- 172.21.15.39:0/3327097819 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fae180a2880 0x7fae180a4d40 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fae240eea70 tx=0x7fae24004ef0 comp rx=0 tx=0).stop 2024-03-18T16:19:41.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.269+0000 7fae37c00640 1 -- 172.21.15.39:0/3327097819 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fae300741e0 msgr2=0x7fae30199d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:19:41.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.269+0000 7fae37c00640 1 --2- 172.21.15.39:0/3327097819 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fae300741e0 0x7fae30199d50 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fae24000c00 tx=0x7fae2402c6f0 comp rx=0 tx=0).stop 2024-03-18T16:19:41.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.269+0000 7fae37c00640 1 -- 172.21.15.39:0/3327097819 shutdown_connections 2024-03-18T16:19:41.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.269+0000 7fae37c00640 1 --2- 172.21.15.39:0/3327097819 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fae180a2880 0x7fae180a4d40 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:41.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.269+0000 7fae37c00640 1 --2- 172.21.15.39:0/3327097819 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fae30109680 0x7fae301a12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:41.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.269+0000 7fae37c00640 1 --2- 172.21.15.39:0/3327097819 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fae300757c0 0x7fae3019a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:19:41.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.269+0000 7fae37c00640 1 --2- 172.21.15.39:0/3327097819 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fae300741e0 0x7fae30199d50 secure :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fae24000c00 tx=0x7fae2402c6f0 comp rx=0 tx=0).stop 2024-03-18T16:19:41.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.269+0000 7fae37c00640 1 -- 172.21.15.39:0/3327097819 >> 172.21.15.39:0/3327097819 conn(0x7fae300fcd80 msgr2=0x7fae30105610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:19:41.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.270+0000 7fae37c00640 1 -- 172.21.15.39:0/3327097819 shutdown_connections 2024-03-18T16:19:41.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:19:41.270+0000 7fae37c00640 1 -- 172.21.15.39:0/3327097819 wait complete. 2024-03-18T16:19:41.291 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:40 smithi039 ceph-mon[31721]: pgmap v710: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 547 KiB/s wr, 148 op/s 2024-03-18T16:19:41.294 INFO:tasks.cephfs.filesystem:scrub status for tag:5fccb086-7953-49b1-8c46-ce57d303b7a1 - {'path': '/', 'tag': '5fccb086-7953-49b1-8c46-ce57d303b7a1', 'options': 'recursive,force'} 2024-03-18T16:19:42.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:41 smithi196 ceph-mon[36307]: scrub queued for path: / 2024-03-18T16:19:42.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:41 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:42.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:41 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:19:42.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:41 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:19:42.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:41 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:19:42.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:41 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:19:42.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:41 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:19:42.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:41 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:42.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:41 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:42.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:41 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:19:42.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:41 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:42.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:41 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:42.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:41 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:42.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:41 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:19:42.230 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:41 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:42.230 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:41 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:19:42.230 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:41 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:42.230 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:41 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:19:42.230 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:41 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:42.230 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:41 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:19:42.230 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:41 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:42.230 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:41 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:19:42.230 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:41 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:42.230 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:41 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:19:42.230 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:41 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:42.230 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:41 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:19:42.230 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:41 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:42.230 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:41 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:19:42.230 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:41 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:42.231 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:41 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:19:42.231 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:41 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:42.231 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:41 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:19:42.231 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:41 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:42.231 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:41 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:19:42.231 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:41 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:42.231 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:41 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:19:42.231 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:41 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:42.231 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:41 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:19:42.231 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:41 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:42.231 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:41 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:42.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:41 smithi194 ceph-mon[36200]: scrub queued for path: / 2024-03-18T16:19:42.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:41 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:42.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:41 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:19:42.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:41 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:19:42.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:41 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:19:42.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:41 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:19:42.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:41 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:19:42.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:41 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:42.242 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:41 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:42.242 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:41 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:19:42.242 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:41 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:42.242 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:41 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:42.242 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:41 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:42.242 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:41 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:19:42.242 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:41 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:42.242 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:41 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:19:42.242 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:41 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:42.242 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:41 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:19:42.242 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:41 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:42.242 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:41 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:19:42.242 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:41 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:42.243 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:41 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:19:42.243 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:41 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:42.243 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:41 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:19:42.243 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:41 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:42.243 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:41 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:19:42.243 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:41 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:42.243 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:41 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:19:42.243 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:41 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:42.243 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:41 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:19:42.243 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:41 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:42.243 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:41 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:19:42.243 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:41 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:42.243 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:41 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:19:42.243 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:41 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:42.243 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:41 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:19:42.243 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:41 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:42.244 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:41 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:19:42.244 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:41 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:42.244 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:41 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:42.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:41 smithi039 ceph-mon[31721]: scrub queued for path: / 2024-03-18T16:19:42.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:41 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:42.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:41 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:19:42.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:41 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:19:42.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:41 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:19:42.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:41 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:19:42.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:41 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:19:42.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:41 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:42.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:41 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:42.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:41 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:19:42.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:41 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:42.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:41 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:42.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:41 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:42.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:41 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:19:42.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:41 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:42.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:41 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:19:42.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:41 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:42.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:41 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:19:42.339 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:41 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:42.339 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:41 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:19:42.339 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:41 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:42.339 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:41 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:19:42.339 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:41 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:42.339 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:41 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:19:42.339 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:41 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:42.339 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:41 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:19:42.339 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:41 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:42.339 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:41 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:19:42.339 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:41 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:42.339 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:41 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:19:42.339 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:41 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:42.339 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:41 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:19:42.339 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:41 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:42.340 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:41 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:19:42.340 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:41 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:42.340 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:41 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:19:42.340 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:41 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:42.340 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:41 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:19:42.340 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:41 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:42.340 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:41 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:19:43.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:42 smithi196 ceph-mon[36307]: pgmap v711: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 616 KiB/s wr, 164 op/s 2024-03-18T16:19:43.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:42 smithi194 ceph-mon[36200]: pgmap v711: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 616 KiB/s wr, 164 op/s 2024-03-18T16:19:43.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:42 smithi039 ceph-mon[31721]: pgmap v711: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 616 KiB/s wr, 164 op/s 2024-03-18T16:19:45.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:44 smithi196 ceph-mon[36307]: pgmap v712: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 651 KiB/s wr, 173 op/s 2024-03-18T16:19:45.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:44 smithi194 ceph-mon[36200]: pgmap v712: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 651 KiB/s wr, 173 op/s 2024-03-18T16:19:45.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:44 smithi039 ceph-mon[31721]: pgmap v712: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 651 KiB/s wr, 173 op/s 2024-03-18T16:19:47.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:46 smithi196 ceph-mon[36307]: pgmap v713: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 553 KiB/s wr, 149 op/s 2024-03-18T16:19:47.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:46 smithi194 ceph-mon[36200]: pgmap v713: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 553 KiB/s wr, 149 op/s 2024-03-18T16:19:47.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:46 smithi039 ceph-mon[31721]: pgmap v713: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 553 KiB/s wr, 149 op/s 2024-03-18T16:19:49.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:48 smithi196 ceph-mon[36307]: pgmap v714: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 640 KiB/s wr, 170 op/s 2024-03-18T16:19:49.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:48 smithi194 ceph-mon[36200]: pgmap v714: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 640 KiB/s wr, 170 op/s 2024-03-18T16:19:49.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:48 smithi039 ceph-mon[31721]: pgmap v714: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 640 KiB/s wr, 170 op/s 2024-03-18T16:19:51.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:50 smithi194 ceph-mon[36200]: pgmap v715: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 539 KiB/s wr, 146 op/s 2024-03-18T16:19:51.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:50 smithi039 ceph-mon[31721]: pgmap v715: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 539 KiB/s wr, 146 op/s 2024-03-18T16:19:51.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:50 smithi196 ceph-mon[36307]: pgmap v715: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 539 KiB/s wr, 146 op/s 2024-03-18T16:19:53.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:52 smithi194 ceph-mon[36200]: pgmap v716: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 607 KiB/s wr, 162 op/s 2024-03-18T16:19:53.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:52 smithi194 ceph-mon[36200]: scrub summary: idle 2024-03-18T16:19:53.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:52 smithi194 ceph-mon[36200]: scrub summary: idle 2024-03-18T16:19:53.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:52 smithi194 ceph-mon[36200]: scrub summary: idle 2024-03-18T16:19:53.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:52 smithi039 ceph-mon[31721]: pgmap v716: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 607 KiB/s wr, 162 op/s 2024-03-18T16:19:53.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:52 smithi039 ceph-mon[31721]: scrub summary: idle 2024-03-18T16:19:53.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:52 smithi039 ceph-mon[31721]: scrub summary: idle 2024-03-18T16:19:53.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:52 smithi039 ceph-mon[31721]: scrub summary: idle 2024-03-18T16:19:53.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:52 smithi196 ceph-mon[36307]: pgmap v716: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 607 KiB/s wr, 162 op/s 2024-03-18T16:19:53.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:52 smithi196 ceph-mon[36307]: scrub summary: idle 2024-03-18T16:19:53.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:52 smithi196 ceph-mon[36307]: scrub summary: idle 2024-03-18T16:19:53.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:52 smithi196 ceph-mon[36307]: scrub summary: idle 2024-03-18T16:19:54.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:53 smithi039 ceph-mon[31721]: scrub summary: idle 2024-03-18T16:19:54.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:53 smithi039 ceph-mon[31721]: scrub summary: idle 2024-03-18T16:19:54.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:53 smithi196 ceph-mon[36307]: scrub summary: idle 2024-03-18T16:19:54.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:53 smithi196 ceph-mon[36307]: scrub summary: idle 2024-03-18T16:19:54.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:53 smithi194 ceph-mon[36200]: scrub summary: idle 2024-03-18T16:19:54.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:53 smithi194 ceph-mon[36200]: scrub summary: idle 2024-03-18T16:19:55.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:54 smithi039 ceph-mon[31721]: pgmap v717: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 634 KiB/s wr, 169 op/s 2024-03-18T16:19:55.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:54 smithi196 ceph-mon[36307]: pgmap v717: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 634 KiB/s wr, 169 op/s 2024-03-18T16:19:55.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:54 smithi194 ceph-mon[36200]: pgmap v717: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 634 KiB/s wr, 169 op/s 2024-03-18T16:19:57.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:56 smithi039 ceph-mon[31721]: pgmap v718: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 545 KiB/s wr, 147 op/s 2024-03-18T16:19:57.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:56 smithi196 ceph-mon[36307]: pgmap v718: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 545 KiB/s wr, 147 op/s 2024-03-18T16:19:57.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:56 smithi194 ceph-mon[36200]: pgmap v718: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 545 KiB/s wr, 147 op/s 2024-03-18T16:19:59.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:19:59 smithi039 ceph-mon[31721]: pgmap v719: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 638 KiB/s wr, 169 op/s 2024-03-18T16:19:59.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:19:59 smithi196 ceph-mon[36307]: pgmap v719: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 638 KiB/s wr, 169 op/s 2024-03-18T16:19:59.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:19:59 smithi194 ceph-mon[36200]: pgmap v719: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 638 KiB/s wr, 169 op/s 2024-03-18T16:20:00.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:00 smithi039 ceph-mon[31721]: overall HEALTH_OK 2024-03-18T16:20:00.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:00 smithi196 ceph-mon[36307]: overall HEALTH_OK 2024-03-18T16:20:00.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:00 smithi194 ceph-mon[36200]: overall HEALTH_OK 2024-03-18T16:20:01.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:01 smithi039 ceph-mon[31721]: pgmap v720: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 543 KiB/s wr, 147 op/s 2024-03-18T16:20:01.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:01 smithi196 ceph-mon[36307]: pgmap v720: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 543 KiB/s wr, 147 op/s 2024-03-18T16:20:01.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:01 smithi194 ceph-mon[36200]: pgmap v720: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 543 KiB/s wr, 147 op/s 2024-03-18T16:20:03.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:03 smithi039 ceph-mon[31721]: pgmap v721: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 607 KiB/s wr, 162 op/s 2024-03-18T16:20:03.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:03 smithi196 ceph-mon[36307]: pgmap v721: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 607 KiB/s wr, 162 op/s 2024-03-18T16:20:03.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:03 smithi194 ceph-mon[36200]: pgmap v721: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 607 KiB/s wr, 162 op/s 2024-03-18T16:20:05.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:05 smithi039 ceph-mon[31721]: pgmap v722: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 637 KiB/s wr, 170 op/s 2024-03-18T16:20:05.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:05 smithi196 ceph-mon[36307]: pgmap v722: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 637 KiB/s wr, 170 op/s 2024-03-18T16:20:05.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:05 smithi194 ceph-mon[36200]: pgmap v722: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 637 KiB/s wr, 170 op/s 2024-03-18T16:20:07.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:07 smithi039 ceph-mon[31721]: pgmap v723: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 543 KiB/s wr, 147 op/s 2024-03-18T16:20:07.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:07 smithi196 ceph-mon[36307]: pgmap v723: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 543 KiB/s wr, 147 op/s 2024-03-18T16:20:07.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:07 smithi194 ceph-mon[36200]: pgmap v723: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 543 KiB/s wr, 147 op/s 2024-03-18T16:20:09.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:09 smithi039 ceph-mon[31721]: pgmap v724: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 639 KiB/s wr, 170 op/s 2024-03-18T16:20:09.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:09 smithi196 ceph-mon[36307]: pgmap v724: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 639 KiB/s wr, 170 op/s 2024-03-18T16:20:09.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:09 smithi194 ceph-mon[36200]: pgmap v724: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 639 KiB/s wr, 170 op/s 2024-03-18T16:20:11.210 INFO:tasks.workunit.client.0.smithi039.stdout:Syncing()...1 sec 2024-03-18T16:20:11.211 INFO:tasks.workunit.client.0.smithi039.stdout:FFSB benchmark finished at: Mon Mar 18 16:20:11 2024 2024-03-18T16:20:11.211 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-03-18T16:20:11.211 INFO:tasks.workunit.client.0.smithi039.stdout:Results: 2024-03-18T16:20:11.211 INFO:tasks.workunit.client.0.smithi039.stdout:Benchmark took 468.98 sec 2024-03-18T16:20:11.211 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-03-18T16:20:11.211 INFO:tasks.workunit.client.0.smithi039.stdout:Total Results 2024-03-18T16:20:11.211 INFO:tasks.workunit.client.0.smithi039.stdout:=============== 2024-03-18T16:20:11.211 INFO:tasks.workunit.client.0.smithi039.stdout: Op Name Transactions Trans/sec % Trans % Op Weight Throughput 2024-03-18T16:20:11.211 INFO:tasks.workunit.client.0.smithi039.stdout: ======= ============ ========= ======= =========== ========== 2024-03-18T16:20:11.211 INFO:tasks.workunit.client.0.smithi039.stdout: write : 3421440 7295.52 100.000% 100.000% 28.5MB/sec 2024-03-18T16:20:11.212 INFO:tasks.workunit.client.0.smithi039.stdout:- 2024-03-18T16:20:11.212 INFO:tasks.workunit.client.0.smithi039.stdout:7295.52 Transactions per Second 2024-03-18T16:20:11.212 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-03-18T16:20:11.212 INFO:tasks.workunit.client.0.smithi039.stdout:Throughput Results 2024-03-18T16:20:11.212 INFO:tasks.workunit.client.0.smithi039.stdout:=================== 2024-03-18T16:20:11.212 INFO:tasks.workunit.client.0.smithi039.stdout:Write Throughput: 28.5MB/sec 2024-03-18T16:20:11.273 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-03-18T16:20:11.273 INFO:tasks.workunit.client.0.smithi039.stdout:System Call Latency statistics in millisecs 2024-03-18T16:20:11.273 INFO:tasks.workunit.client.0.smithi039.stdout:===== 2024-03-18T16:20:11.273 INFO:tasks.workunit.client.0.smithi039.stdout: Min Avg Max Total Calls 2024-03-18T16:20:11.273 INFO:tasks.workunit.client.0.smithi039.stdout: ======== ======== ======== ============ 2024-03-18T16:20:11.273 INFO:tasks.workunit.client.0.smithi039.stdout:[ open] 0.020000 1.366961 107.226997 2673 2024-03-18T16:20:11.273 INFO:tasks.workunit.client.0.smithi039.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2024-03-18T16:20:11.273 INFO:tasks.workunit.client.0.smithi039.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2024-03-18T16:20:11.273 INFO:tasks.workunit.client.0.smithi039.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2024-03-18T16:20:11.273 INFO:tasks.workunit.client.0.smithi039.stdout: msec_range[3] 500.000000 - 1000.000000 : 0 2024-03-18T16:20:11.273 INFO:tasks.workunit.client.0.smithi039.stdout: msec_range[4] 200.000000 - 500.000000 : 0 2024-03-18T16:20:11.274 INFO:tasks.workunit.client.0.smithi039.stdout: msec_range[5] 100.000000 - 200.000000 : 1 2024-03-18T16:20:11.274 INFO:tasks.workunit.client.0.smithi039.stdout: msec_range[6] 50.000000 - 100.000000 : 23 2024-03-18T16:20:11.274 INFO:tasks.workunit.client.0.smithi039.stdout: msec_range[7] 20.000000 - 50.000000 : 14 2024-03-18T16:20:11.274 INFO:tasks.workunit.client.0.smithi039.stdout: msec_range[8] 10.000000 - 20.000000 : 21 2024-03-18T16:20:11.274 INFO:tasks.workunit.client.0.smithi039.stdout: msec_range[9] 5.000000 - 10.000000 : 51 2024-03-18T16:20:11.274 INFO:tasks.workunit.client.0.smithi039.stdout: msec_range[10] 2.000000 - 5.000000 : 110 2024-03-18T16:20:11.274 INFO:tasks.workunit.client.0.smithi039.stdout: msec_range[11] 1.000000 - 2.000000 : 150 2024-03-18T16:20:11.274 INFO:tasks.workunit.client.0.smithi039.stdout: msec_range[12] 0.500000 - 1.000000 : 44 2024-03-18T16:20:11.274 INFO:tasks.workunit.client.0.smithi039.stdout: msec_range[13] 0.200000 - 0.500000 : 64 2024-03-18T16:20:11.274 INFO:tasks.workunit.client.0.smithi039.stdout: msec_range[14] 0.100000 - 0.200000 : 286 2024-03-18T16:20:11.274 INFO:tasks.workunit.client.0.smithi039.stdout: msec_range[15] 0.050000 - 0.100000 : 158 2024-03-18T16:20:11.274 INFO:tasks.workunit.client.0.smithi039.stdout: msec_range[16] 0.020000 - 0.050000 : 1751 2024-03-18T16:20:11.274 INFO:tasks.workunit.client.0.smithi039.stdout: msec_range[17] 0.010000 - 0.020000 : 0 2024-03-18T16:20:11.274 INFO:tasks.workunit.client.0.smithi039.stdout: msec_range[18] 0.000000 - 0.010000 : 0 2024-03-18T16:20:11.274 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-03-18T16:20:11.275 INFO:tasks.workunit.client.0.smithi039.stdout:[ write] 0.003000 3.984135 58932.238281 3421440 2024-03-18T16:20:11.275 INFO:tasks.workunit.client.0.smithi039.stdout: msec_range[0] 5000.000000 - 10000.000000 : 155 2024-03-18T16:20:11.275 INFO:tasks.workunit.client.0.smithi039.stdout: msec_range[1] 2000.000000 - 5000.000000 : 262 2024-03-18T16:20:11.275 INFO:tasks.workunit.client.0.smithi039.stdout: msec_range[2] 1000.000000 - 2000.000000 : 147 2024-03-18T16:20:11.275 INFO:tasks.workunit.client.0.smithi039.stdout: msec_range[3] 500.000000 - 1000.000000 : 137 2024-03-18T16:20:11.275 INFO:tasks.workunit.client.0.smithi039.stdout: msec_range[4] 200.000000 - 500.000000 : 4970 2024-03-18T16:20:11.275 INFO:tasks.workunit.client.0.smithi039.stdout: msec_range[5] 100.000000 - 200.000000 : 19566 2024-03-18T16:20:11.275 INFO:tasks.workunit.client.0.smithi039.stdout: msec_range[6] 50.000000 - 100.000000 : 10052 2024-03-18T16:20:11.275 INFO:tasks.workunit.client.0.smithi039.stdout: msec_range[7] 20.000000 - 50.000000 : 8064 2024-03-18T16:20:11.275 INFO:tasks.workunit.client.0.smithi039.stdout: msec_range[8] 10.000000 - 20.000000 : 3385 2024-03-18T16:20:11.275 INFO:tasks.workunit.client.0.smithi039.stdout: msec_range[9] 5.000000 - 10.000000 : 11224 2024-03-18T16:20:11.275 INFO:tasks.workunit.client.0.smithi039.stdout: msec_range[10] 2.000000 - 5.000000 : 4682 2024-03-18T16:20:11.275 INFO:tasks.workunit.client.0.smithi039.stdout: msec_range[11] 1.000000 - 2.000000 : 1585 2024-03-18T16:20:11.275 INFO:tasks.workunit.client.0.smithi039.stdout: msec_range[12] 0.500000 - 1.000000 : 1313 2024-03-18T16:20:11.276 INFO:tasks.workunit.client.0.smithi039.stdout: msec_range[13] 0.200000 - 0.500000 : 2319 2024-03-18T16:20:11.276 INFO:tasks.workunit.client.0.smithi039.stdout: msec_range[14] 0.100000 - 0.200000 : 3696 2024-03-18T16:20:11.276 INFO:tasks.workunit.client.0.smithi039.stdout: msec_range[15] 0.050000 - 0.100000 : 7578 2024-03-18T16:20:11.276 INFO:tasks.workunit.client.0.smithi039.stdout: msec_range[16] 0.020000 - 0.050000 : 37295 2024-03-18T16:20:11.276 INFO:tasks.workunit.client.0.smithi039.stdout: msec_range[17] 0.010000 - 0.020000 : 1459890 2024-03-18T16:20:11.276 INFO:tasks.workunit.client.0.smithi039.stdout: msec_range[18] 0.000000 - 0.010000 : 1844884 2024-03-18T16:20:11.276 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-03-18T16:20:11.276 INFO:tasks.workunit.client.0.smithi039.stdout:[ lseek] 0.000000 0.002990 42.020000 3421440 2024-03-18T16:20:11.276 INFO:tasks.workunit.client.0.smithi039.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2024-03-18T16:20:11.276 INFO:tasks.workunit.client.0.smithi039.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2024-03-18T16:20:11.276 INFO:tasks.workunit.client.0.smithi039.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2024-03-18T16:20:11.276 INFO:tasks.workunit.client.0.smithi039.stdout: msec_range[3] 500.000000 - 1000.000000 : 0 2024-03-18T16:20:11.276 INFO:tasks.workunit.client.0.smithi039.stdout: msec_range[4] 200.000000 - 500.000000 : 0 2024-03-18T16:20:11.276 INFO:tasks.workunit.client.0.smithi039.stdout: msec_range[5] 100.000000 - 200.000000 : 0 2024-03-18T16:20:11.276 INFO:tasks.workunit.client.0.smithi039.stdout: msec_range[6] 50.000000 - 100.000000 : 0 2024-03-18T16:20:11.276 INFO:tasks.workunit.client.0.smithi039.stdout: msec_range[7] 20.000000 - 50.000000 : 55 2024-03-18T16:20:11.277 INFO:tasks.workunit.client.0.smithi039.stdout: msec_range[8] 10.000000 - 20.000000 : 175 2024-03-18T16:20:11.277 INFO:tasks.workunit.client.0.smithi039.stdout: msec_range[9] 5.000000 - 10.000000 : 167 2024-03-18T16:20:11.277 INFO:tasks.workunit.client.0.smithi039.stdout: msec_range[10] 2.000000 - 5.000000 : 275 2024-03-18T16:20:11.277 INFO:tasks.workunit.client.0.smithi039.stdout: msec_range[11] 1.000000 - 2.000000 : 136 2024-03-18T16:20:11.277 INFO:tasks.workunit.client.0.smithi039.stdout: msec_range[12] 0.500000 - 1.000000 : 152 2024-03-18T16:20:11.277 INFO:tasks.workunit.client.0.smithi039.stdout: msec_range[13] 0.200000 - 0.500000 : 251 2024-03-18T16:20:11.277 INFO:tasks.workunit.client.0.smithi039.stdout: msec_range[14] 0.100000 - 0.200000 : 284 2024-03-18T16:20:11.277 INFO:tasks.workunit.client.0.smithi039.stdout: msec_range[15] 0.050000 - 0.100000 : 521 2024-03-18T16:20:11.277 INFO:tasks.workunit.client.0.smithi039.stdout: msec_range[16] 0.020000 - 0.050000 : 1918 2024-03-18T16:20:11.277 INFO:tasks.workunit.client.0.smithi039.stdout: msec_range[17] 0.010000 - 0.020000 : 1476 2024-03-18T16:20:11.277 INFO:tasks.workunit.client.0.smithi039.stdout: msec_range[18] 0.000000 - 0.010000 : 3416030 2024-03-18T16:20:11.277 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-03-18T16:20:11.277 INFO:tasks.workunit.client.0.smithi039.stdout:[ close] 0.003000 0.016854 10.020000 2673 2024-03-18T16:20:11.277 INFO:tasks.workunit.client.0.smithi039.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2024-03-18T16:20:11.277 INFO:tasks.workunit.client.0.smithi039.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2024-03-18T16:20:11.278 INFO:tasks.workunit.client.0.smithi039.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2024-03-18T16:20:11.278 INFO:tasks.workunit.client.0.smithi039.stdout: msec_range[3] 500.000000 - 1000.000000 : 0 2024-03-18T16:20:11.278 INFO:tasks.workunit.client.0.smithi039.stdout: msec_range[4] 200.000000 - 500.000000 : 0 2024-03-18T16:20:11.278 INFO:tasks.workunit.client.0.smithi039.stdout: msec_range[5] 100.000000 - 200.000000 : 0 2024-03-18T16:20:11.278 INFO:tasks.workunit.client.0.smithi039.stdout: msec_range[6] 50.000000 - 100.000000 : 0 2024-03-18T16:20:11.278 INFO:tasks.workunit.client.0.smithi039.stdout: msec_range[7] 20.000000 - 50.000000 : 0 2024-03-18T16:20:11.278 INFO:tasks.workunit.client.0.smithi039.stdout: msec_range[8] 10.000000 - 20.000000 : 1 2024-03-18T16:20:11.278 INFO:tasks.workunit.client.0.smithi039.stdout: msec_range[9] 5.000000 - 10.000000 : 1 2024-03-18T16:20:11.278 INFO:tasks.workunit.client.0.smithi039.stdout: msec_range[10] 2.000000 - 5.000000 : 0 2024-03-18T16:20:11.278 INFO:tasks.workunit.client.0.smithi039.stdout: msec_range[11] 1.000000 - 2.000000 : 0 2024-03-18T16:20:11.278 INFO:tasks.workunit.client.0.smithi039.stdout: msec_range[12] 0.500000 - 1.000000 : 1 2024-03-18T16:20:11.278 INFO:tasks.workunit.client.0.smithi039.stdout: msec_range[13] 0.200000 - 0.500000 : 5 2024-03-18T16:20:11.278 INFO:tasks.workunit.client.0.smithi039.stdout: msec_range[14] 0.100000 - 0.200000 : 2 2024-03-18T16:20:11.278 INFO:tasks.workunit.client.0.smithi039.stdout: msec_range[15] 0.050000 - 0.100000 : 2 2024-03-18T16:20:11.278 INFO:tasks.workunit.client.0.smithi039.stdout: msec_range[16] 0.020000 - 0.050000 : 19 2024-03-18T16:20:11.279 INFO:tasks.workunit.client.0.smithi039.stdout: msec_range[17] 0.010000 - 0.020000 : 977 2024-03-18T16:20:11.279 INFO:tasks.workunit.client.0.smithi039.stdout: msec_range[18] 0.000000 - 0.010000 : 1665 2024-03-18T16:20:11.279 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-03-18T16:20:11.279 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-03-18T16:20:11.279 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-03-18T16:20:11.279 INFO:tasks.workunit.client.0.smithi039.stdout: 2024-03-18T16:20:11.279 INFO:tasks.workunit.client.0.smithi039.stdout:0.6% User Time 2024-03-18T16:20:11.279 INFO:tasks.workunit.client.0.smithi039.stdout:9.5% System Time 2024-03-18T16:20:11.279 INFO:tasks.workunit.client.0.smithi039.stdout:10.1% CPU Utilization 2024-03-18T16:20:11.286 INFO:tasks.workunit.client.0.smithi039.stderr:+ cd .. 2024-03-18T16:20:11.294 INFO:tasks.workunit.client.0.smithi039.stderr:+ rm -r tmp ffsb 2024-03-18T16:20:11.296 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T16:20:11.296 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T16:20:11.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:11 smithi039 ceph-mon[31721]: pgmap v725: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 549 KiB/s wr, 148 op/s 2024-03-18T16:20:11.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:11 smithi196 ceph-mon[36307]: pgmap v725: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 549 KiB/s wr, 148 op/s 2024-03-18T16:20:11.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:11 smithi194 ceph-mon[36200]: pgmap v725: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 549 KiB/s wr, 148 op/s 2024-03-18T16:20:11.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.508+0000 7f11c2200640 1 -- 172.21.15.39:0/3652574114 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f11bc104af0 msgr2=0x7f11bc106ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:11.510 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.508+0000 7f11c2200640 1 --2- 172.21.15.39:0/3652574114 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f11bc104af0 0x7f11bc106ee0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f11a8009900 tx=0x7f11a802bc40 comp rx=0 tx=0).stop 2024-03-18T16:20:11.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.508+0000 7f11c2200640 1 -- 172.21.15.39:0/3652574114 shutdown_connections 2024-03-18T16:20:11.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.508+0000 7f11c2200640 1 --2- 172.21.15.39:0/3652574114 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f11bc104af0 0x7f11bc106ee0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:11.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.509+0000 7f11c2200640 1 --2- 172.21.15.39:0/3652574114 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f11bc107420 0x7f11bc109810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:11.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.509+0000 7f11c2200640 1 --2- 172.21.15.39:0/3652574114 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f11bc1021c0 0x7f11bc1045b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:11.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.509+0000 7f11c2200640 1 -- 172.21.15.39:0/3652574114 >> 172.21.15.39:0/3652574114 conn(0x7f11bc0fbb50 msgr2=0x7f11bc0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:11.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.509+0000 7f11c2200640 1 -- 172.21.15.39:0/3652574114 shutdown_connections 2024-03-18T16:20:11.511 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.509+0000 7f11c2200640 1 -- 172.21.15.39:0/3652574114 wait complete. 2024-03-18T16:20:11.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.510+0000 7f11c2200640 1 Processor -- start 2024-03-18T16:20:11.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.510+0000 7f11c2200640 1 -- start start 2024-03-18T16:20:11.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.510+0000 7f11c2200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f11bc1021c0 0x7f11bc199be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:11.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.510+0000 7f11c2200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f11bc104af0 0x7f11bc19a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:11.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.510+0000 7f11c2200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f11bc107420 0x7f11bc1a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:11.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.510+0000 7f11c2200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f11bc10d310 con 0x7f11bc107420 2024-03-18T16:20:11.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.510+0000 7f11c2200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f11bc10d190 con 0x7f11bc1021c0 2024-03-18T16:20:11.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.510+0000 7f11c2200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f11bc10d490 con 0x7f11bc104af0 2024-03-18T16:20:11.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.510+0000 7f11c0e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f11bc1021c0 0x7f11bc199be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:11.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.510+0000 7f11bbe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f11bc104af0 0x7f11bc19a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:11.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.510+0000 7f11c0e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f11bc1021c0 0x7f11bc199be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:57646/0 (socket says 172.21.15.39:57646) 2024-03-18T16:20:11.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.510+0000 7f11c0e00640 1 -- 172.21.15.39:0/489168301 learned_addr learned my addr 172.21.15.39:0/489168301 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:20:11.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.511+0000 7f11c1800640 1 --2- 172.21.15.39:0/489168301 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f11bc107420 0x7f11bc1a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:11.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.511+0000 7f11c0e00640 1 -- 172.21.15.39:0/489168301 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f11bc104af0 msgr2=0x7f11bc19a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:11.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.511+0000 7f11c0e00640 1 --2- 172.21.15.39:0/489168301 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f11bc104af0 0x7f11bc19a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:11.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.511+0000 7f11c0e00640 1 -- 172.21.15.39:0/489168301 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f11bc107420 msgr2=0x7f11bc1a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:11.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.511+0000 7f11c0e00640 1 --2- 172.21.15.39:0/489168301 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f11bc107420 0x7f11bc1a1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:11.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.511+0000 7f11c0e00640 1 -- 172.21.15.39:0/489168301 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11bc1a1800 con 0x7f11bc1021c0 2024-03-18T16:20:11.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.511+0000 7f11c1800640 1 --2- 172.21.15.39:0/489168301 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f11bc107420 0x7f11bc1a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:20:11.514 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.511+0000 7f11c0e00640 1 --2- 172.21.15.39:0/489168301 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f11bc1021c0 0x7f11bc199be0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f11b000e9f0 tx=0x7f11b000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:11.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.512+0000 7f11b9600640 1 -- 172.21.15.39:0/489168301 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11b000cd80 con 0x7f11bc1021c0 2024-03-18T16:20:11.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.512+0000 7f11b9600640 1 -- 172.21.15.39:0/489168301 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11b0010910 con 0x7f11bc1021c0 2024-03-18T16:20:11.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.512+0000 7f11b9600640 1 -- 172.21.15.39:0/489168301 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11b0018990 con 0x7f11bc1021c0 2024-03-18T16:20:11.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.512+0000 7f11c2200640 1 -- 172.21.15.39:0/489168301 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11bc1a1af0 con 0x7f11bc1021c0 2024-03-18T16:20:11.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.512+0000 7f11c2200640 1 -- 172.21.15.39:0/489168301 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11bc1a2060 con 0x7f11bc1021c0 2024-03-18T16:20:11.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.515+0000 7f11c6410740 1 Processor -- start 2024-03-18T16:20:11.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.515+0000 7f11b9600640 1 -- 172.21.15.39:0/489168301 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f11b0010ac0 con 0x7f11bc1021c0 2024-03-18T16:20:11.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.515+0000 7f11c6410740 1 -- start start 2024-03-18T16:20:11.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.516+0000 7f11c6410740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5585e3511220 0x5585e3511640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:11.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.516+0000 7f11c6410740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5585e3511c10 0x5585e3516710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:11.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.516+0000 7f11c6410740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5585e3516d80 0x5585e3519210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:11.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.516+0000 7f11c6410740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5585e346d1f0 con 0x5585e3511220 2024-03-18T16:20:11.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.516+0000 7f11c6410740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5585e3415a80 con 0x5585e3516d80 2024-03-18T16:20:11.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.516+0000 7f11c6410740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x5585e3469290 con 0x5585e3511c10 2024-03-18T16:20:11.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.516+0000 7f11c0e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5585e3511c10 0x5585e3516710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:11.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.516+0000 7f11bbe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5585e3516d80 0x5585e3519210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:11.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.516+0000 7f11bbe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5585e3516d80 0x5585e3519210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:57652/0 (socket says 172.21.15.39:57652) 2024-03-18T16:20:11.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.516+0000 7f11bbe00640 1 -- 172.21.15.39:0/3626944976 learned_addr learned my addr 172.21.15.39:0/3626944976 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:20:11.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.516+0000 7f11c1800640 1 --2- 172.21.15.39:0/3626944976 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5585e3511220 0x5585e3511640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:11.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.517+0000 7f11c0e00640 1 -- 172.21.15.39:0/3626944976 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5585e3516d80 msgr2=0x5585e3519210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:11.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.517+0000 7f11c0e00640 1 --2- 172.21.15.39:0/3626944976 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5585e3516d80 0x5585e3519210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:11.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.517+0000 7f11c0e00640 1 -- 172.21.15.39:0/3626944976 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5585e3511220 msgr2=0x5585e3511640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:11.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.517+0000 7f11c0e00640 1 --2- 172.21.15.39:0/3626944976 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5585e3511220 0x5585e3511640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:11.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.517+0000 7f11c0e00640 1 -- 172.21.15.39:0/3626944976 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5585e324b7d0 con 0x5585e3511c10 2024-03-18T16:20:11.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.517+0000 7f11c0e00640 1 --2- 172.21.15.39:0/3626944976 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5585e3511c10 0x5585e3516710 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f11b00cc840 tx=0x7f11b00d7780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1914889041d1c6a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:11.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.517+0000 7f11b9600640 1 --2- 172.21.15.39:0/489168301 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f11a40a2880 0x7f11a40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:11.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.517+0000 7f11ae000640 1 -- 172.21.15.39:0/3626944976 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11b00e2070 con 0x5585e3511c10 2024-03-18T16:20:11.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.517+0000 7f11ae000640 1 -- 172.21.15.39:0/3626944976 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11b009be40 con 0x5585e3511c10 2024-03-18T16:20:11.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.517+0000 7f11ae000640 1 -- 172.21.15.39:0/3626944976 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11b00c0c70 con 0x5585e3511c10 2024-03-18T16:20:11.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.517+0000 7f11b9600640 1 -- 172.21.15.39:0/489168301 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f11b00cf190 con 0x7f11bc1021c0 2024-03-18T16:20:11.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.518+0000 7f11c1800640 1 --2- 172.21.15.39:0/489168301 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f11a40a2880 0x7f11a40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:11.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.518+0000 7f11c6410740 1 -- 172.21.15.39:0/3626944976 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5585e3511c10 msgr2=0x5585e3516710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:11.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.518+0000 7f11c6410740 1 --2- 172.21.15.39:0/3626944976 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5585e3511c10 0x5585e3516710 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f11b00cc840 tx=0x7f11b00d7780 comp rx=0 tx=0).stop 2024-03-18T16:20:11.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.518+0000 7f11c6410740 1 -- 172.21.15.39:0/3626944976 shutdown_connections 2024-03-18T16:20:11.522 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.518+0000 7f11c6410740 1 --2- 172.21.15.39:0/3626944976 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5585e3516d80 0x5585e3519210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:11.522 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.518+0000 7f11c6410740 1 --2- 172.21.15.39:0/3626944976 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5585e3511c10 0x5585e3516710 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:11.522 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.518+0000 7f11c6410740 1 --2- 172.21.15.39:0/3626944976 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5585e3511220 0x5585e3511640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:11.522 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.518+0000 7f11c6410740 1 -- 172.21.15.39:0/3626944976 >> 172.21.15.39:0/3626944976 conn(0x5585e3503f80 msgr2=0x5585e3504380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:11.522 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.518+0000 7f11c6410740 1 -- 172.21.15.39:0/3626944976 shutdown_connections 2024-03-18T16:20:11.522 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.518+0000 7f11c1800640 1 --2- 172.21.15.39:0/489168301 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f11a40a2880 0x7f11a40a4d40 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x5585e3512470 tx=0x7f11b4009800 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:11.522 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.518+0000 7f11c6410740 1 -- 172.21.15.39:0/3626944976 wait complete. 2024-03-18T16:20:11.522 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.519+0000 7f11c6410740 1 -- start start 2024-03-18T16:20:11.522 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.519+0000 7f11c6410740 1 Processor -- start 2024-03-18T16:20:11.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.520+0000 7f11c6410740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5585e3511220 0x5585e351b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:11.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.520+0000 7f11c6410740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5585e3511c10 0x5585e351bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:11.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.520+0000 7f11c6410740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5585e3516d80 0x5585e3522ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:11.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.520+0000 7f11c6410740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5585e3519d30 con 0x5585e3511220 2024-03-18T16:20:11.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.520+0000 7f11c6410740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5585e3428ed0 con 0x5585e3511c10 2024-03-18T16:20:11.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.520+0000 7f11c6410740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x5585e3519eb0 con 0x5585e3516d80 2024-03-18T16:20:11.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.520+0000 7f11c6410740 10 client.0 _ll_register_callbacks cb 0x5585e3464220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:20:11.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.520+0000 7f11bbe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5585e3511c10 0x5585e351bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:11.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.520+0000 7f11c1800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5585e3516d80 0x5585e3522ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:11.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.520+0000 7f11c0e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5585e3511220 0x5585e351b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:11.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.520+0000 7f11bbe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5585e3511c10 0x5585e351bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:57660/0 (socket says 172.21.15.39:57660) 2024-03-18T16:20:11.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.520+0000 7f11bbe00640 1 -- 172.21.15.39:0/325807129 learned_addr learned my addr 172.21.15.39:0/325807129 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:20:11.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.520+0000 7f11c1800640 1 -- 172.21.15.39:0/325807129 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5585e3511c10 msgr2=0x5585e351bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:11.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.520+0000 7f11c1800640 1 --2- 172.21.15.39:0/325807129 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5585e3511c10 0x5585e351bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:11.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.520+0000 7f11c1800640 1 -- 172.21.15.39:0/325807129 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5585e3511220 msgr2=0x5585e351b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:11.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.520+0000 7f11c1800640 1 --2- 172.21.15.39:0/325807129 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5585e3511220 0x5585e351b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:11.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.520+0000 7f11c1800640 1 -- 172.21.15.39:0/325807129 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5585e31813f0 con 0x5585e3516d80 2024-03-18T16:20:11.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.521+0000 7f11c1800640 1 --2- 172.21.15.39:0/325807129 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5585e3516d80 0x5585e3522ce0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f11b4009e30 tx=0x7f11b4004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2bd0aa17715be189 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:11.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.521+0000 7f11a2a00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:20:11.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.521+0000 7f11a2a00640 1 -- 172.21.15.39:0/325807129 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11b4020420 con 0x5585e3516d80 2024-03-18T16:20:11.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.521+0000 7f11c6410740 1 -- 172.21.15.39:0/325807129 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f11b009be40 con 0x5585e3516d80 2024-03-18T16:20:11.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.521+0000 7f11a2a00640 1 -- 172.21.15.39:0/325807129 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f11b40205c0 con 0x5585e3516d80 2024-03-18T16:20:11.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.521+0000 7f11a2a00640 1 -- 172.21.15.39:0/325807129 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f11b4033050 con 0x5585e3516d80 2024-03-18T16:20:11.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.521+0000 7f11c6410740 10 client.27616 fetch_fsmap learned FSMap version 44 2024-03-18T16:20:11.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.521+0000 7f11c6410740 1 -- 172.21.15.39:0/325807129 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x5585e31410f0 con 0x5585e3516d80 2024-03-18T16:20:11.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.522+0000 7f11a2a00640 1 -- 172.21.15.39:0/325807129 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11b009be40 con 0x5585e3516d80 2024-03-18T16:20:11.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.522+0000 7f11a2a00640 1 -- 172.21.15.39:0/325807129 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7f11b400f040 con 0x5585e3516d80 2024-03-18T16:20:11.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.522+0000 7f11c6410740 10 client.27616 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:20:11.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.522+0000 7f11c6410740 10 client.27616 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:20:11.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.522+0000 7f11c6410740 20 client.27616 populate_metadata read hostname 'smithi039' 2024-03-18T16:20:11.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.522+0000 7f11c6410740 1 --2- 172.21.15.39:0/325807129 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5585e3527630 0x5585e3529af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:11.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.522+0000 7f11c6410740 4 client.27616 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:20:11.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.522+0000 7f11c6410740 1 -- 172.21.15.39:0/325807129 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5585e3480be0 con 0x5585e3527630 2024-03-18T16:20:11.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.522+0000 7f11c0e00640 1 --2- 172.21.15.39:0/325807129 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5585e3527630 0x5585e3529af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:11.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.523+0000 7f11c0e00640 1 --2- 172.21.15.39:0/325807129 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5585e3527630 0x5585e3529af0 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8be8454ccc9c88aa server_cookie=348a5f077ea107dc in_seq=0 out_seq=0 2024-03-18T16:20:11.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.523+0000 7f11a2a00640 10 client.27616 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:20:11.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.524+0000 7f11a2a00640 1 -- 172.21.15.39:0/325807129 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5585e3480be0 con 0x5585e3527630 2024-03-18T16:20:11.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.524+0000 7f11a2a00640 10 client.27616 handle_command_reply: tid=0 2024-03-18T16:20:11.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.524+0000 7f11c6410740 1 client.27616 shutdown 2024-03-18T16:20:11.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.525+0000 7f11c6410740 1 -- 172.21.15.39:0/325807129 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5585e3516d80 msgr2=0x5585e3522ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:11.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.525+0000 7f11c6410740 1 --2- 172.21.15.39:0/325807129 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5585e3516d80 0x5585e3522ce0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f11b4009e30 tx=0x7f11b4004a40 comp rx=0 tx=0).stop 2024-03-18T16:20:11.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.525+0000 7f11c6410740 1 -- 172.21.15.39:0/325807129 shutdown_connections 2024-03-18T16:20:11.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.525+0000 7f11c6410740 1 --2- 172.21.15.39:0/325807129 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5585e3527630 0x5585e3529af0 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:11.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.525+0000 7f11c6410740 1 --2- 172.21.15.39:0/325807129 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5585e3511c10 0x5585e351bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:11.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.525+0000 7f11c6410740 1 --2- 172.21.15.39:0/325807129 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5585e3516d80 0x5585e3522ce0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:11.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.525+0000 7f11a2a00640 0 client.27616 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:20:11.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.525+0000 7f11c6410740 1 --2- 172.21.15.39:0/325807129 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5585e3511220 0x5585e351b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:11.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.525+0000 7f11c6410740 1 -- 172.21.15.39:0/325807129 >> 172.21.15.39:0/325807129 conn(0x5585e3503f80 msgr2=0x5585e3503110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:11.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.525+0000 7f11c6410740 1 -- 172.21.15.39:0/325807129 shutdown_connections 2024-03-18T16:20:11.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.525+0000 7f11c6410740 1 -- 172.21.15.39:0/325807129 wait complete. 2024-03-18T16:20:11.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.525+0000 7f11c6410740 20 client.27616 trim_cache size 0 max 16384 2024-03-18T16:20:11.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.544+0000 7f11c6410740 1 Processor -- start 2024-03-18T16:20:11.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.544+0000 7f11c6410740 1 -- start start 2024-03-18T16:20:11.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.544+0000 7f11c6410740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5585e3511220 0x5585e3502f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:11.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.544+0000 7f11c6410740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5585e3511c10 0x5585e3522ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:11.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.544+0000 7f11c6410740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5585e3516d80 0x5585e352e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:11.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.544+0000 7f11c6410740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f11bc1a8100 con 0x5585e3516d80 2024-03-18T16:20:11.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.544+0000 7f11c6410740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f11bc1a9c40 con 0x5585e3511220 2024-03-18T16:20:11.546 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.544+0000 7f11c6410740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f11bc1a87d0 con 0x5585e3511c10 2024-03-18T16:20:11.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.545+0000 7f11c0e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5585e3511220 0x5585e3502f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:11.547 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.545+0000 7f11bbe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5585e3511c10 0x5585e3522ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:11.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.545+0000 7f11c0e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5585e3511220 0x5585e3502f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:57674/0 (socket says 172.21.15.39:57674) 2024-03-18T16:20:11.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.545+0000 7f11bbe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5585e3511c10 0x5585e3522ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:37856/0 (socket says 172.21.15.39:37856) 2024-03-18T16:20:11.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.545+0000 7f11c0e00640 1 -- 172.21.15.39:0/1988308805 learned_addr learned my addr 172.21.15.39:0/1988308805 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:20:11.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.545+0000 7f11c1800640 1 --2- 172.21.15.39:0/1988308805 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5585e3516d80 0x5585e352e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:11.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.545+0000 7f11c0e00640 1 -- 172.21.15.39:0/1988308805 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5585e3511c10 msgr2=0x5585e3522ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:11.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.545+0000 7f11c0e00640 1 --2- 172.21.15.39:0/1988308805 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5585e3511c10 0x5585e3522ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:11.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.545+0000 7f11c0e00640 1 -- 172.21.15.39:0/1988308805 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5585e3516d80 msgr2=0x5585e352e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:11.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.545+0000 7f11c0e00640 1 --2- 172.21.15.39:0/1988308805 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5585e3516d80 0x5585e352e3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:11.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.545+0000 7f11c0e00640 1 -- 172.21.15.39:0/1988308805 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5585e34fe560 con 0x5585e3511220 2024-03-18T16:20:11.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.545+0000 7f11c1800640 1 --2- 172.21.15.39:0/1988308805 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5585e3516d80 0x5585e352e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:20:11.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.545+0000 7f11c0e00640 1 --2- 172.21.15.39:0/1988308805 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5585e3511220 0x5585e3502f90 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f11b009b440 tx=0x7f11b00d7780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=96fe93aa6b295536 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:11.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.546+0000 7f11ad600640 1 -- 172.21.15.39:0/1988308805 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11b00c63f0 con 0x5585e3511220 2024-03-18T16:20:11.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.546+0000 7f11ad600640 1 -- 172.21.15.39:0/1988308805 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11b00d2a20 con 0x5585e3511220 2024-03-18T16:20:11.548 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.546+0000 7f11ad600640 1 -- 172.21.15.39:0/1988308805 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11b00c6e20 con 0x5585e3511220 2024-03-18T16:20:11.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.546+0000 7f11c6410740 1 -- 172.21.15.39:0/1988308805 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5585e3511220 msgr2=0x5585e3502f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:11.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.546+0000 7f11c6410740 1 --2- 172.21.15.39:0/1988308805 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5585e3511220 0x5585e3502f90 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f11b009b440 tx=0x7f11b00d7780 comp rx=0 tx=0).stop 2024-03-18T16:20:11.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.546+0000 7f11c6410740 1 -- 172.21.15.39:0/1988308805 shutdown_connections 2024-03-18T16:20:11.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.546+0000 7f11c6410740 1 --2- 172.21.15.39:0/1988308805 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5585e3516d80 0x5585e352e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:11.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.546+0000 7f11c6410740 1 --2- 172.21.15.39:0/1988308805 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5585e3511c10 0x5585e3522ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:11.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.546+0000 7f11c6410740 1 --2- 172.21.15.39:0/1988308805 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5585e3511220 0x5585e3502f90 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:11.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.546+0000 7f11c6410740 1 -- 172.21.15.39:0/1988308805 >> 172.21.15.39:0/1988308805 conn(0x5585e3503f80 msgr2=0x5585e3502880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:11.553 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.547+0000 7f11c6410740 1 -- 172.21.15.39:0/1988308805 shutdown_connections 2024-03-18T16:20:11.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.547+0000 7f11c6410740 1 -- 172.21.15.39:0/1988308805 wait complete. 2024-03-18T16:20:11.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.547+0000 7f11c6410740 1 -- start start 2024-03-18T16:20:11.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.547+0000 7f11c6410740 1 Processor -- start 2024-03-18T16:20:11.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.548+0000 7f11c6410740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5585e3511220 0x5585e3522980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:11.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.548+0000 7f11c6410740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5585e3511c10 0x5585e3520f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:11.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.548+0000 7f11c6410740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5585e3516d80 0x5585e35214d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:11.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.548+0000 7f11c6410740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f11bc1a9570 con 0x5585e3511c10 2024-03-18T16:20:11.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.548+0000 7f11c6410740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f11bc1a8ea0 con 0x5585e3511220 2024-03-18T16:20:11.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.548+0000 7f11c6410740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f11bc1a6100 con 0x5585e3516d80 2024-03-18T16:20:11.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.548+0000 7f11c6410740 10 client.0 _ll_register_callbacks cb 0x7f11b4009cc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:20:11.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.548+0000 7f11c0e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5585e3511220 0x5585e3522980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:11.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.548+0000 7f11c0e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5585e3511220 0x5585e3522980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:57684/0 (socket says 172.21.15.39:57684) 2024-03-18T16:20:11.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.548+0000 7f11c0e00640 1 -- 172.21.15.39:0/1388932258 learned_addr learned my addr 172.21.15.39:0/1388932258 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:20:11.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.548+0000 7f11bbe00640 1 --2- 172.21.15.39:0/1388932258 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5585e3511c10 0x5585e3520f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:11.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.548+0000 7f11c1800640 1 --2- 172.21.15.39:0/1388932258 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5585e3516d80 0x5585e35214d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:11.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.548+0000 7f11c0e00640 1 -- 172.21.15.39:0/1388932258 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5585e3516d80 msgr2=0x5585e35214d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:11.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.548+0000 7f11c0e00640 1 --2- 172.21.15.39:0/1388932258 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5585e3516d80 0x5585e35214d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:11.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.548+0000 7f11c0e00640 1 -- 172.21.15.39:0/1388932258 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5585e3511c10 msgr2=0x5585e3520f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:11.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.548+0000 7f11c0e00640 1 --2- 172.21.15.39:0/1388932258 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5585e3511c10 0x5585e3520f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:11.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.548+0000 7f11c0e00640 1 -- 172.21.15.39:0/1388932258 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5585e3521a90 con 0x5585e3511220 2024-03-18T16:20:11.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.549+0000 7f11c0e00640 1 --2- 172.21.15.39:0/1388932258 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5585e3511220 0x5585e3522980 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f11b00c3360 tx=0x7f11b0097320 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8eedf15b137893f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:11.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.549+0000 7f11a2a00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:20:11.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.549+0000 7f11a2a00640 1 -- 172.21.15.39:0/1388932258 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11b00c73a0 con 0x5585e3511220 2024-03-18T16:20:11.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.549+0000 7f11a2a00640 1 -- 172.21.15.39:0/1388932258 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f11b0099a40 con 0x5585e3511220 2024-03-18T16:20:11.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.549+0000 7f11c6410740 1 -- 172.21.15.39:0/1388932258 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f11b00d2a20 con 0x5585e3511220 2024-03-18T16:20:11.555 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.549+0000 7f11a2a00640 1 -- 172.21.15.39:0/1388932258 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f11b00c54e0 con 0x5585e3511220 2024-03-18T16:20:11.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.549+0000 7f11c6410740 10 client.27629 fetch_fsmap learned FSMap version 44 2024-03-18T16:20:11.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.549+0000 7f11c6410740 1 -- 172.21.15.39:0/1388932258 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x5585e358e7f0 con 0x5585e3511220 2024-03-18T16:20:11.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.549+0000 7f11a2a00640 1 -- 172.21.15.39:0/1388932258 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11b00d2a20 con 0x5585e3511220 2024-03-18T16:20:11.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.550+0000 7f11a2a00640 1 -- 172.21.15.39:0/1388932258 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7f11b0097db0 con 0x5585e3511220 2024-03-18T16:20:11.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.550+0000 7f11c6410740 10 client.27629 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:20:11.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.550+0000 7f11c6410740 10 client.27629 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:20:11.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.550+0000 7f11c6410740 20 client.27629 populate_metadata read hostname 'smithi039' 2024-03-18T16:20:11.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.550+0000 7f11c6410740 1 --2- 172.21.15.39:0/1388932258 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5585e358eb10 0x5585e3590f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:11.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.550+0000 7f11c6410740 4 client.27629 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T16:20:11.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.550+0000 7f11c6410740 1 -- 172.21.15.39:0/1388932258 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5585e347a730 con 0x5585e358eb10 2024-03-18T16:20:11.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.550+0000 7f11bbe00640 1 --2- 172.21.15.39:0/1388932258 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5585e358eb10 0x5585e3590f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:11.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.550+0000 7f11bbe00640 1 --2- 172.21.15.39:0/1388932258 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5585e358eb10 0x5585e3590f30 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a0aadd24e0904c0b server_cookie=47eaf03f82db4139 in_seq=0 out_seq=0 2024-03-18T16:20:11.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.550+0000 7f11a2a00640 10 client.27629 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:20:11.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.551+0000 7f11a2a00640 1 -- 172.21.15.39:0/1388932258 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5585e347a730 con 0x5585e358eb10 2024-03-18T16:20:11.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.551+0000 7f11a2a00640 10 client.27629 handle_command_reply: tid=0 2024-03-18T16:20:11.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.551+0000 7f11c6410740 1 client.27629 shutdown 2024-03-18T16:20:11.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.551+0000 7f11c6410740 1 -- 172.21.15.39:0/1388932258 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5585e3511220 msgr2=0x5585e3522980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:11.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.551+0000 7f11c6410740 1 --2- 172.21.15.39:0/1388932258 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5585e3511220 0x5585e3522980 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f11b00c3360 tx=0x7f11b0097320 comp rx=0 tx=0).stop 2024-03-18T16:20:11.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.551+0000 7f11c6410740 1 -- 172.21.15.39:0/1388932258 shutdown_connections 2024-03-18T16:20:11.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.551+0000 7f11c6410740 1 --2- 172.21.15.39:0/1388932258 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5585e358eb10 0x5585e3590f30 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:11.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.551+0000 7f11a2a00640 0 client.27629 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:20:11.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.551+0000 7f11c6410740 1 --2- 172.21.15.39:0/1388932258 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5585e3516d80 0x5585e35214d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:11.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.551+0000 7f11c6410740 1 --2- 172.21.15.39:0/1388932258 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5585e3511c10 0x5585e3520f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:11.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.551+0000 7f11c6410740 1 --2- 172.21.15.39:0/1388932258 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5585e3511220 0x5585e3522980 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:11.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.551+0000 7f11c6410740 1 -- 172.21.15.39:0/1388932258 >> 172.21.15.39:0/1388932258 conn(0x5585e3503f80 msgr2=0x5585e3503110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:11.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.551+0000 7f11c6410740 1 -- 172.21.15.39:0/1388932258 shutdown_connections 2024-03-18T16:20:11.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.551+0000 7f11c6410740 1 -- 172.21.15.39:0/1388932258 wait complete. 2024-03-18T16:20:11.557 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.551+0000 7f11c6410740 20 client.27629 trim_cache size 0 max 16384 2024-03-18T16:20:11.558 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-03-18T16:20:11.558 INFO:teuthology.orchestra.run.smithi039.stdout: "status": "no active scrubs running", 2024-03-18T16:20:11.558 INFO:teuthology.orchestra.run.smithi039.stdout: "scrubs": {} 2024-03-18T16:20:11.558 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-03-18T16:20:11.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.552+0000 7f11c2200640 1 -- 172.21.15.39:0/489168301 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f11a40a2880 msgr2=0x7f11a40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:11.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.552+0000 7f11c2200640 1 --2- 172.21.15.39:0/489168301 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f11a40a2880 0x7f11a40a4d40 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x5585e3512470 tx=0x7f11b4009800 comp rx=0 tx=0).stop 2024-03-18T16:20:11.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.552+0000 7f11c2200640 1 -- 172.21.15.39:0/489168301 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f11bc1021c0 msgr2=0x7f11bc199be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:11.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.552+0000 7f11c2200640 1 --2- 172.21.15.39:0/489168301 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f11bc1021c0 0x7f11bc199be0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f11b000e9f0 tx=0x7f11b000eeb0 comp rx=0 tx=0).stop 2024-03-18T16:20:11.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.552+0000 7f11c2200640 1 -- 172.21.15.39:0/489168301 shutdown_connections 2024-03-18T16:20:11.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.552+0000 7f11c2200640 1 --2- 172.21.15.39:0/489168301 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f11a40a2880 0x7f11a40a4d40 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:11.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.552+0000 7f11c2200640 1 --2- 172.21.15.39:0/489168301 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f11bc107420 0x7f11bc1a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:11.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.552+0000 7f11c2200640 1 --2- 172.21.15.39:0/489168301 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f11bc104af0 0x7f11bc19a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:11.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.552+0000 7f11c2200640 1 --2- 172.21.15.39:0/489168301 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f11bc1021c0 0x7f11bc199be0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:11.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.552+0000 7f11c2200640 1 -- 172.21.15.39:0/489168301 >> 172.21.15.39:0/489168301 conn(0x7f11bc0fbb50 msgr2=0x7f11bc101890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:11.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.552+0000 7f11c2200640 1 -- 172.21.15.39:0/489168301 shutdown_connections 2024-03-18T16:20:11.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.552+0000 7f11c2200640 1 -- 172.21.15.39:0/489168301 wait complete. 2024-03-18T16:20:11.573 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-18T16:20:11.574 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:20:11.860 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.858+0000 7fa1ec400640 1 -- 172.21.15.39:0/4142434275 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa1e40741c0 msgr2=0x7fa1e4074620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:11.860 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.858+0000 7fa1ec400640 1 --2- 172.21.15.39:0/4142434275 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa1e40741c0 0x7fa1e4074620 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fa1d0009900 tx=0x7fa1d002bcc0 comp rx=0 tx=0).stop 2024-03-18T16:20:11.860 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.858+0000 7fa1ec400640 1 -- 172.21.15.39:0/4142434275 shutdown_connections 2024-03-18T16:20:11.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.858+0000 7fa1ec400640 1 --2- 172.21.15.39:0/4142434275 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa1e40741c0 0x7fa1e4074620 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:11.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.858+0000 7fa1ec400640 1 --2- 172.21.15.39:0/4142434275 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa1e4109690 0x7fa1e410bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:11.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.858+0000 7fa1ec400640 1 --2- 172.21.15.39:0/4142434275 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1e40757a0 0x7fa1e4073bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:11.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.858+0000 7fa1ec400640 1 -- 172.21.15.39:0/4142434275 >> 172.21.15.39:0/4142434275 conn(0x7fa1e40fcd50 msgr2=0x7fa1e40ff190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:11.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.858+0000 7fa1ec400640 1 -- 172.21.15.39:0/4142434275 shutdown_connections 2024-03-18T16:20:11.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.859+0000 7fa1ec400640 1 -- 172.21.15.39:0/4142434275 wait complete. 2024-03-18T16:20:11.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.860+0000 7fa1ec400640 1 Processor -- start 2024-03-18T16:20:11.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.860+0000 7fa1ec400640 1 -- start start 2024-03-18T16:20:11.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.861+0000 7fa1ec400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1e40741c0 0x7fa1e4199d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:11.863 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.861+0000 7fa1ec400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa1e40757a0 0x7fa1e419a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:11.863 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.861+0000 7fa1ec400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa1e4109690 0x7fa1e41a1270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:11.863 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.861+0000 7fa1ec400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fa1e410f690 con 0x7fa1e40741c0 2024-03-18T16:20:11.863 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.861+0000 7fa1ec400640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fa1e410f510 con 0x7fa1e4109690 2024-03-18T16:20:11.863 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.861+0000 7fa1ec400640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fa1e410f810 con 0x7fa1e40757a0 2024-03-18T16:20:11.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.861+0000 7fa1e9000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa1e40757a0 0x7fa1e419a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:11.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.861+0000 7fa1e9000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa1e40757a0 0x7fa1e419a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:37882/0 (socket says 172.21.15.39:37882) 2024-03-18T16:20:11.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.861+0000 7fa1e9000640 1 -- 172.21.15.39:0/4173502874 learned_addr learned my addr 172.21.15.39:0/4173502874 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:20:11.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.861+0000 7fa1ea400640 1 --2- 172.21.15.39:0/4173502874 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa1e4109690 0x7fa1e41a1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:11.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.861+0000 7fa1e9a00640 1 --2- 172.21.15.39:0/4173502874 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1e40741c0 0x7fa1e4199d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:11.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.862+0000 7fa1e9000640 1 -- 172.21.15.39:0/4173502874 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa1e4109690 msgr2=0x7fa1e41a1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:11.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.862+0000 7fa1e9000640 1 --2- 172.21.15.39:0/4173502874 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa1e4109690 0x7fa1e41a1270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:11.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.862+0000 7fa1e9000640 1 -- 172.21.15.39:0/4173502874 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1e40741c0 msgr2=0x7fa1e4199d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:11.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.862+0000 7fa1e9000640 1 --2- 172.21.15.39:0/4173502874 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1e40741c0 0x7fa1e4199d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:11.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.862+0000 7fa1e9000640 1 -- 172.21.15.39:0/4173502874 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1e41a1910 con 0x7fa1e40757a0 2024-03-18T16:20:11.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.863+0000 7fa1e9000640 1 --2- 172.21.15.39:0/4173502874 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa1e40757a0 0x7fa1e419a240 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fa1d0005ab0 tx=0x7fa1d002c750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:11.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.863+0000 7fa1e2000640 1 -- 172.21.15.39:0/4173502874 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1d0043020 con 0x7fa1e40757a0 2024-03-18T16:20:11.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.863+0000 7fa1e2000640 1 -- 172.21.15.39:0/4173502874 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa1d00054e0 con 0x7fa1e40757a0 2024-03-18T16:20:11.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.863+0000 7fa1ec400640 1 -- 172.21.15.39:0/4173502874 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1e41a1ba0 con 0x7fa1e40757a0 2024-03-18T16:20:11.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.863+0000 7fa1ec400640 1 -- 172.21.15.39:0/4173502874 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1e41a2140 con 0x7fa1e40757a0 2024-03-18T16:20:11.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.863+0000 7fa1e2000640 1 -- 172.21.15.39:0/4173502874 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1d0003cc0 con 0x7fa1e40757a0 2024-03-18T16:20:11.867 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.865+0000 7fa1ec400640 1 -- 172.21.15.39:0/4173502874 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1c40052c0 con 0x7fa1e40757a0 2024-03-18T16:20:11.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.868+0000 7fa1e2000640 1 -- 172.21.15.39:0/4173502874 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fa1d0046020 con 0x7fa1e40757a0 2024-03-18T16:20:11.871 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.869+0000 7fa1e2000640 1 --2- 172.21.15.39:0/4173502874 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa1cc0a2880 0x7fa1cc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:11.871 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.869+0000 7fa1e2000640 1 -- 172.21.15.39:0/4173502874 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fa1d00f0ec0 con 0x7fa1e40757a0 2024-03-18T16:20:11.871 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.870+0000 7fa1e9a00640 1 --2- 172.21.15.39:0/4173502874 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa1cc0a2880 0x7fa1cc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:11.872 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.871+0000 7fa1e9a00640 1 --2- 172.21.15.39:0/4173502874 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa1cc0a2880 0x7fa1cc0a4d40 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fa1d8005d00 tx=0x7fa1d8005c70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:11.879 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:11.878+0000 7fa1e2000640 1 -- 172.21.15.39:0/4173502874 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fa1d004d020 con 0x7fa1e40757a0 2024-03-18T16:20:12.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.223+0000 7fa1ec400640 1 -- 172.21.15.39:0/4173502874 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa1c40055b0 con 0x7fa1e40757a0 2024-03-18T16:20:12.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.224+0000 7fa1e2000640 1 -- 172.21.15.39:0/4173502874 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) ==== 94+0+11932 (secure 0 0 0) 0x7fa1d00bdbd0 con 0x7fa1e40757a0 2024-03-18T16:20:12.226 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:20:12.226 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":44,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2520226608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2520226608},{"type":"v1","addr":"172.21.15.39:6835","nonce":2520226608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/719341763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":719341763},{"type":"v1","addr":"172.21.15.39:6837","nonce":719341763}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":44,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T16:06:44.180425+0000","modified":"2024-03-18T16:15:58.268360+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24575,"mds_1":24556,"mds_2":24535,"mds_3":24524,"mds_4":14463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14445":{"gid":14445,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.39:6839/656513813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":656513813},{"type":"v1","addr":"172.21.15.39:6839","nonce":656513813}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14463":{"gid":14463,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":13,"addr":"172.21.15.39:6841/3515326042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3515326042},{"type":"v1","addr":"172.21.15.39:6841","nonce":3515326042}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6833/931350738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":931350738},{"type":"v1","addr":"172.21.15.194:6833","nonce":931350738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6833/3874209621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":3874209621},{"type":"v1","addr":"172.21.15.196:6833","nonce":3874209621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6835/188385470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":188385470},{"type":"v1","addr":"172.21.15.194:6835","nonce":188385470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6835/3949984309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":3949984309},{"type":"v1","addr":"172.21.15.196:6835","nonce":3949984309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":15,"addr":"172.21.15.194:6837/1301464246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":1301464246},{"type":"v1","addr":"172.21.15.194:6837","nonce":1301464246}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":16,"addr":"172.21.15.196:6837/895247251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":895247251},{"type":"v1","addr":"172.21.15.196:6837","nonce":895247251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.196:6839/2001427268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6838","nonce":2001427268},{"type":"v1","addr":"172.21.15.196:6839","nonce":2001427268}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24575":{"gid":24575,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.194:6839/274911158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":274911158},{"type":"v1","addr":"172.21.15.194:6839","nonce":274911158}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24575,"qdb_cluster":[14463,24524,24575,24556,24535]},"id":1}]} 2024-03-18T16:20:12.227 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 44 2024-03-18T16:20:12.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.227+0000 7fa1ec400640 1 -- 172.21.15.39:0/4173502874 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa1cc0a2880 msgr2=0x7fa1cc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:12.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.227+0000 7fa1ec400640 1 --2- 172.21.15.39:0/4173502874 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa1cc0a2880 0x7fa1cc0a4d40 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fa1d8005d00 tx=0x7fa1d8005c70 comp rx=0 tx=0).stop 2024-03-18T16:20:12.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.227+0000 7fa1ec400640 1 -- 172.21.15.39:0/4173502874 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa1e40757a0 msgr2=0x7fa1e419a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:12.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.227+0000 7fa1ec400640 1 --2- 172.21.15.39:0/4173502874 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa1e40757a0 0x7fa1e419a240 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fa1d0005ab0 tx=0x7fa1d002c750 comp rx=0 tx=0).stop 2024-03-18T16:20:12.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.227+0000 7fa1ec400640 1 -- 172.21.15.39:0/4173502874 shutdown_connections 2024-03-18T16:20:12.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.227+0000 7fa1ec400640 1 --2- 172.21.15.39:0/4173502874 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa1e4109690 0x7fa1e41a1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:12.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.227+0000 7fa1ec400640 1 --2- 172.21.15.39:0/4173502874 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa1cc0a2880 0x7fa1cc0a4d40 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:12.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.227+0000 7fa1ec400640 1 --2- 172.21.15.39:0/4173502874 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa1e40757a0 0x7fa1e419a240 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:12.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.227+0000 7fa1ec400640 1 --2- 172.21.15.39:0/4173502874 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1e40741c0 0x7fa1e4199d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:12.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.227+0000 7fa1ec400640 1 -- 172.21.15.39:0/4173502874 >> 172.21.15.39:0/4173502874 conn(0x7fa1e40fcd50 msgr2=0x7fa1e4105620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:12.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.227+0000 7fa1ec400640 1 -- 172.21.15.39:0/4173502874 shutdown_connections 2024-03-18T16:20:12.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.227+0000 7fa1ec400640 1 -- 172.21.15.39:0/4173502874 wait complete. 2024-03-18T16:20:12.250 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-18T16:20:12.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.483+0000 7f17fda00640 1 -- 172.21.15.39:0/650455828 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f17f81017e0 msgr2=0x7f17f8101c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:12.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.483+0000 7f17fda00640 1 --2- 172.21.15.39:0/650455828 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f17f81017e0 0x7f17f8101c00 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f17e8009900 tx=0x7f17e802bc30 comp rx=0 tx=0).stop 2024-03-18T16:20:12.485 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.483+0000 7f17fda00640 1 -- 172.21.15.39:0/650455828 shutdown_connections 2024-03-18T16:20:12.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.483+0000 7f17fda00640 1 --2- 172.21.15.39:0/650455828 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f17f8103390 0x7f17f8109c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:12.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.483+0000 7f17fda00640 1 --2- 172.21.15.39:0/650455828 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f17f81029d0 0x7f17f8102e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:12.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.483+0000 7f17fda00640 1 --2- 172.21.15.39:0/650455828 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f17f81017e0 0x7f17f8101c00 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:12.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.483+0000 7f17fda00640 1 -- 172.21.15.39:0/650455828 >> 172.21.15.39:0/650455828 conn(0x7f17f80fcd70 msgr2=0x7f17f80ff190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:12.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.483+0000 7f17fda00640 1 -- 172.21.15.39:0/650455828 shutdown_connections 2024-03-18T16:20:12.486 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.483+0000 7f17fda00640 1 -- 172.21.15.39:0/650455828 wait complete. 2024-03-18T16:20:12.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.484+0000 7f17fda00640 1 Processor -- start 2024-03-18T16:20:12.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.485+0000 7f17fda00640 1 -- start start 2024-03-18T16:20:12.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.485+0000 7f17fda00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f17f81017e0 0x7f17f8199d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:12.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.485+0000 7f17fda00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f17f81029d0 0x7f17f819a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:12.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.485+0000 7f17fda00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f17f8103390 0x7f17f81a12f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:12.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.485+0000 7f17fda00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f17f810d430 con 0x7f17f8103390 2024-03-18T16:20:12.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.485+0000 7f17fda00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f17f810d2b0 con 0x7f17f81029d0 2024-03-18T16:20:12.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.485+0000 7f17fda00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f17f810d5b0 con 0x7f17f81017e0 2024-03-18T16:20:12.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.485+0000 7f17f6a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f17f81017e0 0x7f17f8199d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:12.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.485+0000 7f17f6000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f17f81029d0 0x7f17f819a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:12.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.485+0000 7f17f6a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f17f81017e0 0x7f17f8199d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:37906/0 (socket says 172.21.15.39:37906) 2024-03-18T16:20:12.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.485+0000 7f17f6a00640 1 -- 172.21.15.39:0/2004727298 learned_addr learned my addr 172.21.15.39:0/2004727298 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:20:12.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.485+0000 7f17f7400640 1 --2- 172.21.15.39:0/2004727298 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f17f8103390 0x7f17f81a12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:12.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.486+0000 7f17f6a00640 1 -- 172.21.15.39:0/2004727298 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f17f81029d0 msgr2=0x7f17f819a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:12.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.486+0000 7f17f6a00640 1 --2- 172.21.15.39:0/2004727298 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f17f81029d0 0x7f17f819a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:12.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.486+0000 7f17f6a00640 1 -- 172.21.15.39:0/2004727298 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f17f8103390 msgr2=0x7f17f81a12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:12.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.486+0000 7f17f6a00640 1 --2- 172.21.15.39:0/2004727298 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f17f8103390 0x7f17f81a12f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:12.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.486+0000 7f17f6a00640 1 -- 172.21.15.39:0/2004727298 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17f81a1960 con 0x7f17f81017e0 2024-03-18T16:20:12.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.486+0000 7f17f7400640 1 --2- 172.21.15.39:0/2004727298 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f17f8103390 0x7f17f81a12f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:20:12.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.486+0000 7f17f6000640 1 --2- 172.21.15.39:0/2004727298 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f17f81029d0 0x7f17f819a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:20:12.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.486+0000 7f17f6a00640 1 --2- 172.21.15.39:0/2004727298 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f17f81017e0 0x7f17f8199d40 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f17e8000c00 tx=0x7f17e802c640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:12.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.486+0000 7f17ef400640 1 -- 172.21.15.39:0/2004727298 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e8041070 con 0x7f17f81017e0 2024-03-18T16:20:12.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.486+0000 7f17ef400640 1 -- 172.21.15.39:0/2004727298 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f17e8004450 con 0x7f17f81017e0 2024-03-18T16:20:12.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.486+0000 7f17ef400640 1 -- 172.21.15.39:0/2004727298 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e8034440 con 0x7f17f81017e0 2024-03-18T16:20:12.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.486+0000 7f17fda00640 1 -- 172.21.15.39:0/2004727298 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17f81a1bf0 con 0x7f17f81017e0 2024-03-18T16:20:12.490 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.487+0000 7f17fda00640 1 -- 172.21.15.39:0/2004727298 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17f81a2130 con 0x7f17f81017e0 2024-03-18T16:20:12.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.490+0000 7f1800779740 1 Processor -- start 2024-03-18T16:20:12.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.490+0000 7f17ef400640 1 -- 172.21.15.39:0/2004727298 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f17e8044070 con 0x7f17f81017e0 2024-03-18T16:20:12.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.490+0000 7f1800779740 1 -- start start 2024-03-18T16:20:12.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.491+0000 7f1800779740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5629f79508c0 0x5629f7950ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:12.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.491+0000 7f1800779740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5629f79512b0 0x5629f7955db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:12.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.491+0000 7f1800779740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5629f7956420 0x5629f79588b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:12.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.491+0000 7f1800779740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5629f79183b0 con 0x5629f7956420 2024-03-18T16:20:12.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.491+0000 7f1800779740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5629f78b5450 con 0x5629f79512b0 2024-03-18T16:20:12.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.491+0000 7f1800779740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x5629f785dcb0 con 0x5629f79508c0 2024-03-18T16:20:12.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.491+0000 7f17f7400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5629f79508c0 0x5629f7950ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:12.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.491+0000 7f17f6000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5629f7956420 0x5629f79588b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:12.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.491+0000 7f17f6a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5629f79512b0 0x5629f7955db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:12.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.491+0000 7f17f7400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5629f79508c0 0x5629f7950ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:37912/0 (socket says 172.21.15.39:37912) 2024-03-18T16:20:12.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.491+0000 7f17f7400640 1 -- 172.21.15.39:0/2074101931 learned_addr learned my addr 172.21.15.39:0/2074101931 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:20:12.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.491+0000 7f17f7400640 1 -- 172.21.15.39:0/2074101931 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5629f79512b0 msgr2=0x5629f7955db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:12.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.491+0000 7f17f7400640 1 --2- 172.21.15.39:0/2074101931 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5629f79512b0 0x5629f7955db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:12.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.491+0000 7f17f7400640 1 -- 172.21.15.39:0/2074101931 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5629f7956420 msgr2=0x5629f79588b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:12.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.491+0000 7f17f7400640 1 --2- 172.21.15.39:0/2074101931 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5629f7956420 0x5629f79588b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:12.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.492+0000 7f17f7400640 1 -- 172.21.15.39:0/2074101931 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5629f787f1f0 con 0x5629f79508c0 2024-03-18T16:20:12.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.492+0000 7f17f6000640 1 --2- 172.21.15.39:0/2074101931 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5629f7956420 0x5629f79588b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:20:12.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.492+0000 7f17f7400640 1 --2- 172.21.15.39:0/2074101931 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5629f79508c0 0x5629f7950ce0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f17f0010450 tx=0x7f17f0010910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=47a550c9ce21c6d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:12.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.492+0000 7f17e3e00640 1 -- 172.21.15.39:0/2074101931 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17f0017070 con 0x5629f79508c0 2024-03-18T16:20:12.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.492+0000 7f17e3e00640 1 -- 172.21.15.39:0/2074101931 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f17f0011c40 con 0x5629f79508c0 2024-03-18T16:20:12.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.492+0000 7f17e3e00640 1 -- 172.21.15.39:0/2074101931 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17f001ad30 con 0x5629f79508c0 2024-03-18T16:20:12.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.492+0000 7f17ef400640 1 --2- 172.21.15.39:0/2004727298 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f17dc0a2880 0x7f17dc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:12.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.493+0000 7f17ef400640 1 -- 172.21.15.39:0/2004727298 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f17e80f0940 con 0x7f17f81017e0 2024-03-18T16:20:12.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.493+0000 7f17f6a00640 1 --2- 172.21.15.39:0/2004727298 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f17dc0a2880 0x7f17dc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:12.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.493+0000 7f1800779740 1 -- 172.21.15.39:0/2074101931 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5629f79508c0 msgr2=0x5629f7950ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:12.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.493+0000 7f1800779740 1 --2- 172.21.15.39:0/2074101931 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5629f79508c0 0x5629f7950ce0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f17f0010450 tx=0x7f17f0010910 comp rx=0 tx=0).stop 2024-03-18T16:20:12.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.493+0000 7f1800779740 1 -- 172.21.15.39:0/2074101931 shutdown_connections 2024-03-18T16:20:12.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.493+0000 7f1800779740 1 --2- 172.21.15.39:0/2074101931 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5629f79512b0 0x5629f7955db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:12.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.493+0000 7f1800779740 1 --2- 172.21.15.39:0/2074101931 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5629f7956420 0x5629f79588b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:12.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.493+0000 7f1800779740 1 --2- 172.21.15.39:0/2074101931 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5629f79508c0 0x5629f7950ce0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:12.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.493+0000 7f1800779740 1 -- 172.21.15.39:0/2074101931 >> 172.21.15.39:0/2074101931 conn(0x5629f794be30 msgr2=0x5629f794c230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:12.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.493+0000 7f1800779740 1 -- 172.21.15.39:0/2074101931 shutdown_connections 2024-03-18T16:20:12.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.493+0000 7f1800779740 1 -- 172.21.15.39:0/2074101931 wait complete. 2024-03-18T16:20:12.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.493+0000 7f17f6a00640 1 --2- 172.21.15.39:0/2004727298 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f17dc0a2880 0x7f17dc0a4d40 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f17e80b93e0 tx=0x7f17e80f6530 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:12.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.494+0000 7f1800779740 1 -- start start 2024-03-18T16:20:12.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.494+0000 7f1800779740 1 Processor -- start 2024-03-18T16:20:12.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.494+0000 7f1800779740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5629f79508c0 0x5629f795ab70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:12.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.494+0000 7f1800779740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5629f79512b0 0x5629f795b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:12.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.494+0000 7f1800779740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5629f7956420 0x5629f79620f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:12.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.494+0000 7f1800779740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5629f7871110 con 0x5629f79512b0 2024-03-18T16:20:12.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.494+0000 7f1800779740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5629f78b14f0 con 0x5629f7956420 2024-03-18T16:20:12.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.494+0000 7f1800779740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x5629f79593d0 con 0x5629f79508c0 2024-03-18T16:20:12.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.495+0000 7f1800779740 10 client.0 _ll_register_callbacks cb 0x5629f785eb70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:20:12.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.495+0000 7f17f7400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5629f79508c0 0x5629f795ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:12.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.495+0000 7f17f7400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5629f79508c0 0x5629f795ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:37922/0 (socket says 172.21.15.39:37922) 2024-03-18T16:20:12.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.495+0000 7f17f7400640 1 -- 172.21.15.39:0/703053799 learned_addr learned my addr 172.21.15.39:0/703053799 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:20:12.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.495+0000 7f17f6000640 1 --2- 172.21.15.39:0/703053799 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5629f79512b0 0x5629f795b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:12.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.495+0000 7f17f7400640 1 --2- 172.21.15.39:0/703053799 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5629f7956420 0x5629f79620f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:12.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.495+0000 7f17f7400640 1 -- 172.21.15.39:0/703053799 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5629f7956420 msgr2=0x5629f79620f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:12.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.495+0000 7f17f7400640 1 --2- 172.21.15.39:0/703053799 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5629f7956420 0x5629f79620f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:12.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.495+0000 7f17f7400640 1 -- 172.21.15.39:0/703053799 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5629f79512b0 msgr2=0x5629f795b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:12.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.495+0000 7f17f7400640 1 --2- 172.21.15.39:0/703053799 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5629f79512b0 0x5629f795b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:12.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.495+0000 7f17f7400640 1 -- 172.21.15.39:0/703053799 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5629f75c93f0 con 0x5629f79508c0 2024-03-18T16:20:12.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.495+0000 7f17f6000640 1 --2- 172.21.15.39:0/703053799 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5629f79512b0 0x5629f795b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:20:12.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.496+0000 7f17f7400640 1 --2- 172.21.15.39:0/703053799 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5629f79508c0 0x5629f795ab70 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f17f001a5f0 tx=0x7f17f0009940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=268cfdb04a930fb7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:12.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.496+0000 7f17e0c00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:20:12.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.496+0000 7f17e0c00640 1 -- 172.21.15.39:0/703053799 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17f0017040 con 0x5629f79508c0 2024-03-18T16:20:12.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.496+0000 7f1800779740 1 -- 172.21.15.39:0/703053799 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f17f0011c40 con 0x5629f79508c0 2024-03-18T16:20:12.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.496+0000 7f17e0c00640 1 -- 172.21.15.39:0/703053799 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f17f0011970 con 0x5629f79508c0 2024-03-18T16:20:12.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.496+0000 7f17e0c00640 1 -- 172.21.15.39:0/703053799 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f17f0004a40 con 0x5629f79508c0 2024-03-18T16:20:12.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.496+0000 7f1800779740 10 client.27640 fetch_fsmap learned FSMap version 44 2024-03-18T16:20:12.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.496+0000 7f1800779740 1 -- 172.21.15.39:0/703053799 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x5629f76937d0 con 0x5629f79508c0 2024-03-18T16:20:12.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.496+0000 7f17e0c00640 1 -- 172.21.15.39:0/703053799 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f17f0011c40 con 0x5629f79508c0 2024-03-18T16:20:12.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.496+0000 7f17e0c00640 1 -- 172.21.15.39:0/703053799 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7f17f0029420 con 0x5629f79508c0 2024-03-18T16:20:12.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.497+0000 7f1800779740 10 client.27640 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:20:12.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.497+0000 7f1800779740 10 client.27640 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-03-18T16:20:12.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.497+0000 7f1800779740 20 client.27640 populate_metadata read hostname 'smithi039' 2024-03-18T16:20:12.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.497+0000 7f1800779740 1 --2- 172.21.15.39:0/703053799 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x5629f79667d0 0x5629f7968bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:12.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.497+0000 7f1800779740 4 client.27640 mds_command: new command op to 14463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:20:12.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.497+0000 7f1800779740 1 -- 172.21.15.39:0/703053799 --> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5629f78c2990 con 0x5629f79667d0 2024-03-18T16:20:12.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.497+0000 7f17f6000640 1 --2- 172.21.15.39:0/703053799 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x5629f79667d0 0x5629f7968bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:12.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.497+0000 7f17f6000640 1 --2- 172.21.15.39:0/703053799 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x5629f79667d0 0x5629f7968bc0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=ce665bba0a8a995 server_cookie=83c40bdae8be8f5c in_seq=0 out_seq=0 2024-03-18T16:20:12.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.497+0000 7f17e0c00640 10 client.27640 ms_handle_connect on v2:172.21.15.39:6840/3515326042 2024-03-18T16:20:12.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.498+0000 7f17e0c00640 1 -- 172.21.15.39:0/703053799 <== mds.4 v2:172.21.15.39:6840/3515326042 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5629f78c2990 con 0x5629f79667d0 2024-03-18T16:20:12.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.498+0000 7f17e0c00640 10 client.27640 handle_command_reply: tid=0 2024-03-18T16:20:12.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.498+0000 7f1800779740 1 client.27640 shutdown 2024-03-18T16:20:12.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.498+0000 7f1800779740 1 -- 172.21.15.39:0/703053799 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5629f79508c0 msgr2=0x5629f795ab70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:12.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.498+0000 7f1800779740 1 --2- 172.21.15.39:0/703053799 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5629f79508c0 0x5629f795ab70 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f17f001a5f0 tx=0x7f17f0009940 comp rx=0 tx=0).stop 2024-03-18T16:20:12.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.498+0000 7f1800779740 1 -- 172.21.15.39:0/703053799 shutdown_connections 2024-03-18T16:20:12.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.498+0000 7f1800779740 1 --2- 172.21.15.39:0/703053799 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x5629f79667d0 0x5629f7968bc0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:12.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.499+0000 7f1800779740 1 --2- 172.21.15.39:0/703053799 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5629f7956420 0x5629f79620f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:12.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.499+0000 7f1800779740 1 --2- 172.21.15.39:0/703053799 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5629f79512b0 0x5629f795b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:12.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.499+0000 7f1800779740 1 --2- 172.21.15.39:0/703053799 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5629f79508c0 0x5629f795ab70 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:12.503 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.499+0000 7f17e0c00640 0 client.27640 ms_handle_reset on v2:172.21.15.39:6840/3515326042 2024-03-18T16:20:12.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.499+0000 7f1800779740 1 -- 172.21.15.39:0/703053799 >> 172.21.15.39:0/703053799 conn(0x5629f794be30 msgr2=0x5629f794b1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:12.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.499+0000 7f1800779740 1 -- 172.21.15.39:0/703053799 shutdown_connections 2024-03-18T16:20:12.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.499+0000 7f1800779740 1 -- 172.21.15.39:0/703053799 wait complete. 2024-03-18T16:20:12.504 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.499+0000 7f1800779740 20 client.27640 trim_cache size 0 max 16384 2024-03-18T16:20:12.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.515+0000 7f1800779740 1 Processor -- start 2024-03-18T16:20:12.517 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.515+0000 7f1800779740 1 -- start start 2024-03-18T16:20:12.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.516+0000 7f1800779740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5629f79508c0 0x5629f795b010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:12.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.516+0000 7f1800779740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5629f79512b0 0x5629f79620b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:12.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.516+0000 7f1800779740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5629f7956420 0x5629f794b070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:12.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.516+0000 7f1800779740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f17f81ac630 con 0x5629f79512b0 2024-03-18T16:20:12.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.516+0000 7f1800779740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f17f81ae170 con 0x5629f79508c0 2024-03-18T16:20:12.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.516+0000 7f1800779740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f17f81acd00 con 0x5629f7956420 2024-03-18T16:20:12.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.516+0000 7f17f7400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5629f79508c0 0x5629f795b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:12.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.516+0000 7f17f6000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5629f79512b0 0x5629f79620b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:12.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.516+0000 7f17f7400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5629f79508c0 0x5629f795b010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:57746/0 (socket says 172.21.15.39:57746) 2024-03-18T16:20:12.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.516+0000 7f17f7400640 1 -- 172.21.15.39:0/1579179209 learned_addr learned my addr 172.21.15.39:0/1579179209 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:20:12.518 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.516+0000 7f17f7400640 1 --2- 172.21.15.39:0/1579179209 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5629f7956420 0x5629f794b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:12.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.516+0000 7f17f6000640 1 -- 172.21.15.39:0/1579179209 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5629f7956420 msgr2=0x5629f794b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:12.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.516+0000 7f17f6000640 1 --2- 172.21.15.39:0/1579179209 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5629f7956420 0x5629f794b070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:12.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.517+0000 7f17f6000640 1 -- 172.21.15.39:0/1579179209 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5629f79508c0 msgr2=0x5629f795b010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:12.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.517+0000 7f17f6000640 1 --2- 172.21.15.39:0/1579179209 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5629f79508c0 0x5629f795b010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:12.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.517+0000 7f17f6000640 1 -- 172.21.15.39:0/1579179209 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5629f7947c10 con 0x5629f79512b0 2024-03-18T16:20:12.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.517+0000 7f17f6000640 1 --2- 172.21.15.39:0/1579179209 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5629f79512b0 0x5629f79620b0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x5629f7969400 tx=0x7f17e4005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbdca9f2b8c44dba server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:12.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.517+0000 7f17e3400640 1 -- 172.21.15.39:0/1579179209 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e400fc20 con 0x5629f79512b0 2024-03-18T16:20:12.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.517+0000 7f17e3400640 1 -- 172.21.15.39:0/1579179209 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f17e400fdc0 con 0x5629f79512b0 2024-03-18T16:20:12.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.517+0000 7f17e3400640 1 -- 172.21.15.39:0/1579179209 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e4010ad0 con 0x5629f79512b0 2024-03-18T16:20:12.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.518+0000 7f1800779740 1 -- 172.21.15.39:0/1579179209 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5629f79512b0 msgr2=0x5629f79620b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:12.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.518+0000 7f1800779740 1 --2- 172.21.15.39:0/1579179209 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5629f79512b0 0x5629f79620b0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x5629f7969400 tx=0x7f17e4005f90 comp rx=0 tx=0).stop 2024-03-18T16:20:12.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.518+0000 7f1800779740 1 -- 172.21.15.39:0/1579179209 shutdown_connections 2024-03-18T16:20:12.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.518+0000 7f1800779740 1 --2- 172.21.15.39:0/1579179209 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5629f7956420 0x5629f794b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:12.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.518+0000 7f1800779740 1 --2- 172.21.15.39:0/1579179209 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5629f79512b0 0x5629f79620b0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:12.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.518+0000 7f1800779740 1 --2- 172.21.15.39:0/1579179209 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5629f79508c0 0x5629f795b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:12.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.518+0000 7f1800779740 1 -- 172.21.15.39:0/1579179209 >> 172.21.15.39:0/1579179209 conn(0x5629f794be30 msgr2=0x5629f794a960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:12.521 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.519+0000 7f1800779740 1 -- 172.21.15.39:0/1579179209 shutdown_connections 2024-03-18T16:20:12.522 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.520+0000 7f1800779740 1 -- 172.21.15.39:0/1579179209 wait complete. 2024-03-18T16:20:12.522 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.520+0000 7f1800779740 1 -- start start 2024-03-18T16:20:12.522 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.520+0000 7f1800779740 1 Processor -- start 2024-03-18T16:20:12.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.521+0000 7f1800779740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5629f79508c0 0x5629f7960140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:12.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.521+0000 7f1800779740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5629f79512b0 0x5629f7960680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:12.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.521+0000 7f1800779740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5629f7956420 0x5629f7961690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:12.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.521+0000 7f1800779740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f17f81adaa0 con 0x5629f79512b0 2024-03-18T16:20:12.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.521+0000 7f1800779740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f17f81ad3d0 con 0x5629f79508c0 2024-03-18T16:20:12.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.521+0000 7f1800779740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f17f81aa610 con 0x5629f7956420 2024-03-18T16:20:12.523 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.521+0000 7f1800779740 10 client.0 _ll_register_callbacks cb 0x7f17e4009570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:20:12.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.522+0000 7f17f7400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5629f79508c0 0x5629f7960140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:12.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.522+0000 7f17f7400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5629f79508c0 0x5629f7960140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:57760/0 (socket says 172.21.15.39:57760) 2024-03-18T16:20:12.524 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.522+0000 7f17f7400640 1 -- 172.21.15.39:0/1521458186 learned_addr learned my addr 172.21.15.39:0/1521458186 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:20:12.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.522+0000 7f17f7400640 1 --2- 172.21.15.39:0/1521458186 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5629f7956420 0x5629f7961690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:12.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.522+0000 7f17f6000640 1 --2- 172.21.15.39:0/1521458186 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5629f79512b0 0x5629f7960680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:12.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.522+0000 7f17f7400640 1 -- 172.21.15.39:0/1521458186 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5629f7956420 msgr2=0x5629f7961690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:12.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.522+0000 7f17f7400640 1 --2- 172.21.15.39:0/1521458186 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5629f7956420 0x5629f7961690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:12.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.522+0000 7f17f7400640 1 -- 172.21.15.39:0/1521458186 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5629f79512b0 msgr2=0x5629f7960680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:12.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.522+0000 7f17f7400640 1 --2- 172.21.15.39:0/1521458186 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5629f79512b0 0x5629f7960680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:12.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.522+0000 7f17f7400640 1 -- 172.21.15.39:0/1521458186 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5629f7950500 con 0x5629f79508c0 2024-03-18T16:20:12.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.522+0000 7f17f6000640 1 --2- 172.21.15.39:0/1521458186 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5629f79512b0 0x5629f7960680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:20:12.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.523+0000 7f17f7400640 1 --2- 172.21.15.39:0/1521458186 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5629f79508c0 0x5629f7960140 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f17f001af80 tx=0x7f17f0005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4cdf6109873b3eb2 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:12.525 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.523+0000 7f17e0c00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:20:12.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.523+0000 7f17e0c00640 1 -- 172.21.15.39:0/1521458186 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17f0017040 con 0x5629f79508c0 2024-03-18T16:20:12.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.523+0000 7f1800779740 1 -- 172.21.15.39:0/1521458186 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f17e400fdc0 con 0x5629f79508c0 2024-03-18T16:20:12.526 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.523+0000 7f17e0c00640 1 -- 172.21.15.39:0/1521458186 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f17f0011970 con 0x5629f79508c0 2024-03-18T16:20:12.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.523+0000 7f17e0c00640 1 -- 172.21.15.39:0/1521458186 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f17f004b050 con 0x5629f79508c0 2024-03-18T16:20:12.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.523+0000 7f1800779740 10 client.27653 fetch_fsmap learned FSMap version 44 2024-03-18T16:20:12.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.524+0000 7f1800779740 1 -- 172.21.15.39:0/1521458186 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x5629f79d59f0 con 0x5629f79508c0 2024-03-18T16:20:12.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.524+0000 7f17e0c00640 1 -- 172.21.15.39:0/1521458186 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f17e400fdc0 con 0x5629f79508c0 2024-03-18T16:20:12.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.524+0000 7f17e0c00640 1 -- 172.21.15.39:0/1521458186 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7f17f0039040 con 0x5629f79508c0 2024-03-18T16:20:12.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.524+0000 7f1800779740 10 client.27653 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:20:12.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.524+0000 7f1800779740 10 client.27653 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-03-18T16:20:12.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.524+0000 7f1800779740 20 client.27653 populate_metadata read hostname 'smithi039' 2024-03-18T16:20:12.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.524+0000 7f1800779740 1 --2- 172.21.15.39:0/1521458186 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x5629f79d6280 0x5629f79d86a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:12.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.524+0000 7f1800779740 4 client.27653 mds_command: new command op to 14463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:20:12.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.524+0000 7f1800779740 1 -- 172.21.15.39:0/1521458186 --> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5629f78bcce0 con 0x5629f79d6280 2024-03-18T16:20:12.527 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.525+0000 7f17f6000640 1 --2- 172.21.15.39:0/1521458186 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x5629f79d6280 0x5629f79d86a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:12.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.526+0000 7f17f6000640 1 --2- 172.21.15.39:0/1521458186 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x5629f79d6280 0x5629f79d86a0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=6f3172b371febcb0 server_cookie=4869544ad495cde8 in_seq=0 out_seq=0 2024-03-18T16:20:12.528 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.526+0000 7f17e0c00640 10 client.27653 ms_handle_connect on v2:172.21.15.39:6840/3515326042 2024-03-18T16:20:12.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.528+0000 7f17e0c00640 1 -- 172.21.15.39:0/1521458186 <== mds.4 v2:172.21.15.39:6840/3515326042 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5629f78bcce0 con 0x5629f79d6280 2024-03-18T16:20:12.529 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.528+0000 7f17e0c00640 10 client.27653 handle_command_reply: tid=0 2024-03-18T16:20:12.532 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.531+0000 7f1800779740 1 client.27653 shutdown 2024-03-18T16:20:12.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.532+0000 7f1800779740 1 -- 172.21.15.39:0/1521458186 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5629f79508c0 msgr2=0x5629f7960140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:12.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.532+0000 7f1800779740 1 --2- 172.21.15.39:0/1521458186 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5629f79508c0 0x5629f7960140 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f17f001af80 tx=0x7f17f0005fd0 comp rx=0 tx=0).stop 2024-03-18T16:20:12.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.532+0000 7f1800779740 1 -- 172.21.15.39:0/1521458186 shutdown_connections 2024-03-18T16:20:12.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.532+0000 7f1800779740 1 --2- 172.21.15.39:0/1521458186 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x5629f79d6280 0x5629f79d86a0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:12.534 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.533+0000 7f1800779740 1 --2- 172.21.15.39:0/1521458186 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5629f7956420 0x5629f7961690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:12.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.533+0000 7f17e0c00640 0 client.27653 ms_handle_reset on v2:172.21.15.39:6840/3515326042 2024-03-18T16:20:12.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.533+0000 7f1800779740 1 --2- 172.21.15.39:0/1521458186 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5629f79512b0 0x5629f7960680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:12.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.533+0000 7f1800779740 1 --2- 172.21.15.39:0/1521458186 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5629f79508c0 0x5629f7960140 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:12.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.533+0000 7f1800779740 1 -- 172.21.15.39:0/1521458186 >> 172.21.15.39:0/1521458186 conn(0x5629f794be30 msgr2=0x5629f794b1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:12.535 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.534+0000 7f1800779740 1 -- 172.21.15.39:0/1521458186 shutdown_connections 2024-03-18T16:20:12.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.534+0000 7f1800779740 1 -- 172.21.15.39:0/1521458186 wait complete. 2024-03-18T16:20:12.536 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.535+0000 7f1800779740 20 client.27653 trim_cache size 0 max 16384 2024-03-18T16:20:12.536 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:20:12.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.536+0000 7f17fda00640 1 -- 172.21.15.39:0/2004727298 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f17dc0a2880 msgr2=0x7f17dc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:12.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.536+0000 7f17fda00640 1 --2- 172.21.15.39:0/2004727298 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f17dc0a2880 0x7f17dc0a4d40 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f17e80b93e0 tx=0x7f17e80f6530 comp rx=0 tx=0).stop 2024-03-18T16:20:12.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.536+0000 7f17fda00640 1 -- 172.21.15.39:0/2004727298 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f17f81017e0 msgr2=0x7f17f8199d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:12.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.536+0000 7f17fda00640 1 --2- 172.21.15.39:0/2004727298 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f17f81017e0 0x7f17f8199d40 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f17e8000c00 tx=0x7f17e802c640 comp rx=0 tx=0).stop 2024-03-18T16:20:12.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.536+0000 7f17fda00640 1 -- 172.21.15.39:0/2004727298 shutdown_connections 2024-03-18T16:20:12.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.536+0000 7f17fda00640 1 --2- 172.21.15.39:0/2004727298 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f17f81029d0 0x7f17f819a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:12.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.536+0000 7f17fda00640 1 --2- 172.21.15.39:0/2004727298 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f17dc0a2880 0x7f17dc0a4d40 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:12.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.536+0000 7f17fda00640 1 --2- 172.21.15.39:0/2004727298 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f17f8103390 0x7f17f81a12f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:12.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.536+0000 7f17fda00640 1 --2- 172.21.15.39:0/2004727298 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f17f81017e0 0x7f17f8199d40 secure :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f17e8000c00 tx=0x7f17e802c640 comp rx=0 tx=0).stop 2024-03-18T16:20:12.538 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.536+0000 7f17fda00640 1 -- 172.21.15.39:0/2004727298 >> 172.21.15.39:0/2004727298 conn(0x7f17f80fcd70 msgr2=0x7f17f8073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:12.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.537+0000 7f17fda00640 1 -- 172.21.15.39:0/2004727298 shutdown_connections 2024-03-18T16:20:12.539 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.537+0000 7f17fda00640 1 -- 172.21.15.39:0/2004727298 wait complete. 2024-03-18T16:20:12.570 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-18T16:20:12.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.879+0000 7f237f200640 1 -- 172.21.15.39:0/3388986133 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f23780757a0 msgr2=0x7f2378073bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:12.881 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.879+0000 7f237f200640 1 --2- 172.21.15.39:0/3388986133 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f23780757a0 0x7f2378073bf0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f2370009ca0 tx=0x7f237002c190 comp rx=0 tx=0).stop 2024-03-18T16:20:12.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.882+0000 7f237f200640 1 -- 172.21.15.39:0/3388986133 shutdown_connections 2024-03-18T16:20:12.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.882+0000 7f237f200640 1 --2- 172.21.15.39:0/3388986133 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2378109690 0x7f237810bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:12.883 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.882+0000 7f237f200640 1 --2- 172.21.15.39:0/3388986133 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f23780741c0 0x7f2378074620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:12.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.882+0000 7f237f200640 1 --2- 172.21.15.39:0/3388986133 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f23780757a0 0x7f2378073bf0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:12.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.882+0000 7f237f200640 1 -- 172.21.15.39:0/3388986133 >> 172.21.15.39:0/3388986133 conn(0x7f23780fcd50 msgr2=0x7f23780ff190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:12.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.882+0000 7f237f200640 1 -- 172.21.15.39:0/3388986133 shutdown_connections 2024-03-18T16:20:12.884 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.882+0000 7f237f200640 1 -- 172.21.15.39:0/3388986133 wait complete. 2024-03-18T16:20:12.885 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.883+0000 7f237f200640 1 Processor -- start 2024-03-18T16:20:12.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.884+0000 7f237f200640 1 -- start start 2024-03-18T16:20:12.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.884+0000 7f237f200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f23780741c0 0x7f2378199d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:12.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.884+0000 7f237f200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f23780757a0 0x7f237819a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:12.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.884+0000 7f237f200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2378109690 0x7f23781a12c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:12.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.884+0000 7f237f200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f237810f6e0 con 0x7f23780757a0 2024-03-18T16:20:12.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.884+0000 7f237f200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f237810f560 con 0x7f2378109690 2024-03-18T16:20:12.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.884+0000 7f237f200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f237810f860 con 0x7f23780741c0 2024-03-18T16:20:12.886 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.884+0000 7f2377e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f23780741c0 0x7f2378199d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:12.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.885+0000 7f2377e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f23780741c0 0x7f2378199d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:37970/0 (socket says 172.21.15.39:37970) 2024-03-18T16:20:12.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.885+0000 7f2377e00640 1 -- 172.21.15.39:0/1445575961 learned_addr learned my addr 172.21.15.39:0/1445575961 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:20:12.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.885+0000 7f237d200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2378109690 0x7f23781a12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:12.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.885+0000 7f2377400640 1 --2- 172.21.15.39:0/1445575961 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f23780757a0 0x7f237819a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:12.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.885+0000 7f2377e00640 1 -- 172.21.15.39:0/1445575961 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2378109690 msgr2=0x7f23781a12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:12.887 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.886+0000 7f2377e00640 1 --2- 172.21.15.39:0/1445575961 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2378109690 0x7f23781a12c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:12.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.886+0000 7f2377e00640 1 -- 172.21.15.39:0/1445575961 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f23780757a0 msgr2=0x7f237819a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:12.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.886+0000 7f2377e00640 1 --2- 172.21.15.39:0/1445575961 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f23780757a0 0x7f237819a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:12.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.886+0000 7f2377e00640 1 -- 172.21.15.39:0/1445575961 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23781a1960 con 0x7f23780741c0 2024-03-18T16:20:12.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.886+0000 7f237d200640 1 --2- 172.21.15.39:0/1445575961 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2378109690 0x7f23781a12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:20:12.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.886+0000 7f2377400640 1 --2- 172.21.15.39:0/1445575961 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f23780757a0 0x7f237819a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:20:12.888 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.887+0000 7f2377e00640 1 --2- 172.21.15.39:0/1445575961 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f23780741c0 0x7f2378199d50 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f23700326d0 tx=0x7f2370003ba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:12.890 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.888+0000 7f2374c00640 1 -- 172.21.15.39:0/1445575961 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2370039040 con 0x7f23780741c0 2024-03-18T16:20:12.890 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.889+0000 7f2374c00640 1 -- 172.21.15.39:0/1445575961 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2370044070 con 0x7f23780741c0 2024-03-18T16:20:12.890 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.889+0000 7f237f200640 1 -- 172.21.15.39:0/1445575961 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23781a1bf0 con 0x7f23780741c0 2024-03-18T16:20:12.891 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.889+0000 7f2374c00640 1 -- 172.21.15.39:0/1445575961 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23700045a0 con 0x7f23780741c0 2024-03-18T16:20:12.892 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.890+0000 7f237f200640 1 -- 172.21.15.39:0/1445575961 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23781a2100 con 0x7f23780741c0 2024-03-18T16:20:12.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.894+0000 7f2381f7e740 1 Processor -- start 2024-03-18T16:20:12.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.895+0000 7f2381f7e740 1 -- start start 2024-03-18T16:20:12.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.895+0000 7f2381f7e740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560d5c8558c0 0x560d5c855ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:12.898 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.895+0000 7f2381f7e740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d5c8562b0 0x560d5c85adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:12.898 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.895+0000 7f2381f7e740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560d5c85b420 0x560d5c85d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:12.898 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.895+0000 7f2381f7e740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x560d5c81d3b0 con 0x560d5c8558c0 2024-03-18T16:20:12.898 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.895+0000 7f2381f7e740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x560d5c7ba450 con 0x560d5c8562b0 2024-03-18T16:20:12.898 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.895+0000 7f2381f7e740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x560d5c762cb0 con 0x560d5c85b420 2024-03-18T16:20:12.898 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.896+0000 7f2377400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560d5c85b420 0x560d5c85d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:12.898 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.896+0000 7f2377400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560d5c85b420 0x560d5c85d8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:37976/0 (socket says 172.21.15.39:37976) 2024-03-18T16:20:12.898 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.896+0000 7f2377400640 1 -- 172.21.15.39:0/2187250833 learned_addr learned my addr 172.21.15.39:0/2187250833 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:20:12.898 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.896+0000 7f237d200640 1 --2- 172.21.15.39:0/2187250833 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560d5c8558c0 0x560d5c855ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:12.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.896+0000 7f2377400640 1 -- 172.21.15.39:0/2187250833 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d5c8562b0 msgr2=0x560d5c85adb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:20:12.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.896+0000 7f2377400640 1 --2- 172.21.15.39:0/2187250833 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d5c8562b0 0x560d5c85adb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:12.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.896+0000 7f2377400640 1 -- 172.21.15.39:0/2187250833 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560d5c8558c0 msgr2=0x560d5c855ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:12.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.896+0000 7f2377400640 1 --2- 172.21.15.39:0/2187250833 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560d5c8558c0 0x560d5c855ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:12.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.896+0000 7f2377400640 1 -- 172.21.15.39:0/2187250833 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d5c7841f0 con 0x560d5c85b420 2024-03-18T16:20:12.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.897+0000 7f2377400640 1 --2- 172.21.15.39:0/2187250833 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560d5c85b420 0x560d5c85d8b0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f2360007850 tx=0x7f2360007d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2add49af20ac88da server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:12.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.898+0000 7f236d600640 1 -- 172.21.15.39:0/2187250833 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f236000d950 con 0x560d5c85b420 2024-03-18T16:20:12.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.898+0000 7f236d600640 1 -- 172.21.15.39:0/2187250833 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2360018070 con 0x560d5c85b420 2024-03-18T16:20:12.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.898+0000 7f236d600640 1 -- 172.21.15.39:0/2187250833 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23600123e0 con 0x560d5c85b420 2024-03-18T16:20:12.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.899+0000 7f2381f7e740 1 -- 172.21.15.39:0/2187250833 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560d5c85b420 msgr2=0x560d5c85d8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:12.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.899+0000 7f2381f7e740 1 --2- 172.21.15.39:0/2187250833 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560d5c85b420 0x560d5c85d8b0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f2360007850 tx=0x7f2360007d10 comp rx=0 tx=0).stop 2024-03-18T16:20:12.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.899+0000 7f2381f7e740 1 -- 172.21.15.39:0/2187250833 shutdown_connections 2024-03-18T16:20:12.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.899+0000 7f2381f7e740 1 --2- 172.21.15.39:0/2187250833 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d5c8562b0 0x560d5c85adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:12.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.899+0000 7f2381f7e740 1 --2- 172.21.15.39:0/2187250833 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560d5c85b420 0x560d5c85d8b0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:12.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.899+0000 7f2381f7e740 1 --2- 172.21.15.39:0/2187250833 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560d5c8558c0 0x560d5c855ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:12.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.899+0000 7f2381f7e740 1 -- 172.21.15.39:0/2187250833 >> 172.21.15.39:0/2187250833 conn(0x560d5c850e30 msgr2=0x560d5c851230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:12.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.900+0000 7f2374c00640 1 -- 172.21.15.39:0/1445575961 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f2370032770 con 0x7f23780741c0 2024-03-18T16:20:12.902 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.901+0000 7f2381f7e740 1 -- 172.21.15.39:0/2187250833 shutdown_connections 2024-03-18T16:20:12.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.903+0000 7f2374c00640 1 --2- 172.21.15.39:0/1445575961 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f235c0a2810 0x7f235c0a4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:12.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.903+0000 7f2374c00640 1 -- 172.21.15.39:0/1445575961 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f23700f1500 con 0x7f23780741c0 2024-03-18T16:20:12.905 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.903+0000 7f2377400640 1 --2- 172.21.15.39:0/1445575961 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f235c0a2810 0x7f235c0a4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:12.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.903+0000 7f2381f7e740 1 -- 172.21.15.39:0/2187250833 wait complete. 2024-03-18T16:20:12.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.904+0000 7f2381f7e740 1 -- start start 2024-03-18T16:20:12.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.904+0000 7f2377400640 1 --2- 172.21.15.39:0/1445575961 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f235c0a2810 0x7f235c0a4cd0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f2360006f60 tx=0x7f2360000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:12.907 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.904+0000 7f2381f7e740 1 Processor -- start 2024-03-18T16:20:12.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.908+0000 7f2381f7e740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560d5c8558c0 0x560d5c868520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:12.910 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.908+0000 7f2381f7e740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d5c85b420 0x560d5c868a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:12.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.908+0000 7f2381f7e740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560d5c86da90 0x560d5c86fe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:12.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.909+0000 7f2381f7e740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x560d5c776110 con 0x560d5c8558c0 2024-03-18T16:20:12.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.909+0000 7f2381f7e740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x560d5c7b64f0 con 0x560d5c85b420 2024-03-18T16:20:12.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.909+0000 7f2381f7e740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x560d5c85e3d0 con 0x560d5c86da90 2024-03-18T16:20:12.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.909+0000 7f237d200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560d5c86da90 0x560d5c86fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:12.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.909+0000 7f237d200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560d5c86da90 0x560d5c86fe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:37986/0 (socket says 172.21.15.39:37986) 2024-03-18T16:20:12.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.909+0000 7f237d200640 1 -- 172.21.15.39:0/2316407446 learned_addr learned my addr 172.21.15.39:0/2316407446 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:20:12.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.909+0000 7f2381f7e740 10 client.0 _ll_register_callbacks cb 0x560d5c763b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:20:12.911 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.909+0000 7f2377400640 1 --2- 172.21.15.39:0/2316407446 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d5c85b420 0x560d5c868a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:12.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.909+0000 7f237d200640 1 -- 172.21.15.39:0/2316407446 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d5c85b420 msgr2=0x560d5c868a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:12.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.909+0000 7f237d200640 1 --2- 172.21.15.39:0/2316407446 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d5c85b420 0x560d5c868a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:12.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.909+0000 7f237d200640 1 -- 172.21.15.39:0/2316407446 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560d5c8558c0 msgr2=0x560d5c868520 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:20:12.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.910+0000 7f2377e00640 1 --2- 172.21.15.39:0/2316407446 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560d5c8558c0 0x560d5c868520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:12.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.911+0000 7f237d200640 1 --2- 172.21.15.39:0/2316407446 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560d5c8558c0 0x560d5c868520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:12.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.911+0000 7f237d200640 1 -- 172.21.15.39:0/2316407446 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560d5c4ce3f0 con 0x560d5c86da90 2024-03-18T16:20:12.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.911+0000 7f2377400640 1 --2- 172.21.15.39:0/2316407446 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d5c85b420 0x560d5c868a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:20:12.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.911+0000 7f2377e00640 1 --2- 172.21.15.39:0/2316407446 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560d5c8558c0 0x560d5c868520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:20:12.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.911+0000 7f237d200640 1 --2- 172.21.15.39:0/2316407446 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560d5c86da90 0x560d5c86fe80 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f236800bcc0 tx=0x7f23680039c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a2dbff95a7883e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:12.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.911+0000 7f2366000640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:20:12.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.911+0000 7f2366000640 1 -- 172.21.15.39:0/2316407446 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2368012860 con 0x560d5c86da90 2024-03-18T16:20:12.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.912+0000 7f2381f7e740 1 -- 172.21.15.39:0/2316407446 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2360018070 con 0x560d5c86da90 2024-03-18T16:20:12.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.912+0000 7f2366000640 1 -- 172.21.15.39:0/2316407446 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f2368012a00 con 0x560d5c86da90 2024-03-18T16:20:12.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.914+0000 7f2366000640 1 -- 172.21.15.39:0/2316407446 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f2368012c00 con 0x560d5c86da90 2024-03-18T16:20:12.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.914+0000 7f2381f7e740 10 client.27658 fetch_fsmap learned FSMap version 44 2024-03-18T16:20:12.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.914+0000 7f2381f7e740 1 -- 172.21.15.39:0/2316407446 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x560d5c5987d0 con 0x560d5c86da90 2024-03-18T16:20:12.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.914+0000 7f2366000640 1 -- 172.21.15.39:0/2316407446 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2360018070 con 0x560d5c86da90 2024-03-18T16:20:12.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.914+0000 7f2366000640 1 -- 172.21.15.39:0/2316407446 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7f2368011490 con 0x560d5c86da90 2024-03-18T16:20:12.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.915+0000 7f2381f7e740 10 client.27658 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:20:12.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.915+0000 7f2381f7e740 10 client.27658 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-18T16:20:12.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.915+0000 7f2381f7e740 20 client.27658 populate_metadata read hostname 'smithi039' 2024-03-18T16:20:12.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.915+0000 7f2381f7e740 1 --2- 172.21.15.39:0/2316407446 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x560d5c874c40 0x560d5c877100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:12.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.915+0000 7f2381f7e740 4 client.27658 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:20:12.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.915+0000 7f2381f7e740 1 -- 172.21.15.39:0/2316407446 --> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560d5c7c7990 con 0x560d5c874c40 2024-03-18T16:20:12.917 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.916+0000 7f2377e00640 1 --2- 172.21.15.39:0/2316407446 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x560d5c874c40 0x560d5c877100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:12.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.916+0000 7f2377e00640 1 --2- 172.21.15.39:0/2316407446 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x560d5c874c40 0x560d5c877100 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=d82761ea2280bb7f server_cookie=a7c2ccc7ad1c0170 in_seq=0 out_seq=0 2024-03-18T16:20:12.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.916+0000 7f2366000640 10 client.27658 ms_handle_connect on v2:172.21.15.194:6836/1301464246 2024-03-18T16:20:12.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.917+0000 7f2366000640 1 -- 172.21.15.39:0/2316407446 <== mds.3 v2:172.21.15.194:6836/1301464246 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560d5c7c7990 con 0x560d5c874c40 2024-03-18T16:20:12.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.917+0000 7f2366000640 10 client.27658 handle_command_reply: tid=0 2024-03-18T16:20:12.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.917+0000 7f2381f7e740 1 client.27658 shutdown 2024-03-18T16:20:12.922 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.920+0000 7f2381f7e740 1 -- 172.21.15.39:0/2316407446 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560d5c86da90 msgr2=0x560d5c86fe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:12.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.920+0000 7f2381f7e740 1 --2- 172.21.15.39:0/2316407446 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560d5c86da90 0x560d5c86fe80 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f236800bcc0 tx=0x7f23680039c0 comp rx=0 tx=0).stop 2024-03-18T16:20:12.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.920+0000 7f2381f7e740 1 -- 172.21.15.39:0/2316407446 shutdown_connections 2024-03-18T16:20:12.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.920+0000 7f2381f7e740 1 --2- 172.21.15.39:0/2316407446 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x560d5c874c40 0x560d5c877100 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:12.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.920+0000 7f2366000640 0 client.27658 ms_handle_reset on v2:172.21.15.194:6836/1301464246 2024-03-18T16:20:12.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.920+0000 7f2381f7e740 1 --2- 172.21.15.39:0/2316407446 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d5c85b420 0x560d5c868a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:12.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.920+0000 7f2381f7e740 1 --2- 172.21.15.39:0/2316407446 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560d5c86da90 0x560d5c86fe80 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:12.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.920+0000 7f2381f7e740 1 --2- 172.21.15.39:0/2316407446 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560d5c8558c0 0x560d5c868520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:12.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.920+0000 7f2381f7e740 1 -- 172.21.15.39:0/2316407446 >> 172.21.15.39:0/2316407446 conn(0x560d5c850e30 msgr2=0x560d5c8501f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:12.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.921+0000 7f2381f7e740 1 -- 172.21.15.39:0/2316407446 shutdown_connections 2024-03-18T16:20:12.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.921+0000 7f2381f7e740 1 -- 172.21.15.39:0/2316407446 wait complete. 2024-03-18T16:20:12.923 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.921+0000 7f2381f7e740 20 client.27658 trim_cache size 0 max 16384 2024-03-18T16:20:12.956 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.955+0000 7f2381f7e740 1 Processor -- start 2024-03-18T16:20:12.956 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.955+0000 7f2381f7e740 1 -- start start 2024-03-18T16:20:12.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.956+0000 7f2381f7e740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d5c8558c0 0x560d5c850070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:12.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.956+0000 7f2381f7e740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560d5c85b420 0x560d5c867030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:12.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.956+0000 7f2381f7e740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560d5c86da90 0x560d5c867570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:12.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.956+0000 7f2381f7e740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f23781a82f0 con 0x560d5c86da90 2024-03-18T16:20:12.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.956+0000 7f2381f7e740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f23781a9e30 con 0x560d5c8558c0 2024-03-18T16:20:12.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.956+0000 7f2381f7e740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f23781a89c0 con 0x560d5c85b420 2024-03-18T16:20:12.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.956+0000 7f2377400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560d5c85b420 0x560d5c867030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:12.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.956+0000 7f2377e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d5c8558c0 0x560d5c850070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:12.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.956+0000 7f2377400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560d5c85b420 0x560d5c867030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:37996/0 (socket says 172.21.15.39:37996) 2024-03-18T16:20:12.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.956+0000 7f2377400640 1 -- 172.21.15.39:0/602512577 learned_addr learned my addr 172.21.15.39:0/602512577 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:20:12.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.957+0000 7f2377e00640 1 -- 172.21.15.39:0/602512577 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560d5c85b420 msgr2=0x560d5c867030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:12.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.957+0000 7f2377e00640 1 --2- 172.21.15.39:0/602512577 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560d5c85b420 0x560d5c867030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:12.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.957+0000 7f2377e00640 1 -- 172.21.15.39:0/602512577 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560d5c86da90 msgr2=0x560d5c867570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:12.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.957+0000 7f2377e00640 1 --2- 172.21.15.39:0/602512577 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560d5c86da90 0x560d5c867570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:12.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.957+0000 7f2377e00640 1 -- 172.21.15.39:0/602512577 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d5c851ba0 con 0x560d5c8558c0 2024-03-18T16:20:12.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.957+0000 7f2377e00640 1 --2- 172.21.15.39:0/602512577 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d5c8558c0 0x560d5c850070 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x560d5c856d20 tx=0x7f2370035f50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c7f544224ed2414 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:12.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.957+0000 7f236cc00640 1 -- 172.21.15.39:0/602512577 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23700bdab0 con 0x560d5c8558c0 2024-03-18T16:20:12.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.958+0000 7f236cc00640 1 -- 172.21.15.39:0/602512577 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f23700c1c00 con 0x560d5c8558c0 2024-03-18T16:20:12.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.958+0000 7f236cc00640 1 -- 172.21.15.39:0/602512577 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23700ba090 con 0x560d5c8558c0 2024-03-18T16:20:12.960 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.959+0000 7f2381f7e740 1 -- 172.21.15.39:0/602512577 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d5c8558c0 msgr2=0x560d5c850070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:12.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.959+0000 7f2381f7e740 1 --2- 172.21.15.39:0/602512577 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d5c8558c0 0x560d5c850070 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x560d5c856d20 tx=0x7f2370035f50 comp rx=0 tx=0).stop 2024-03-18T16:20:12.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.959+0000 7f2381f7e740 1 -- 172.21.15.39:0/602512577 shutdown_connections 2024-03-18T16:20:12.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.959+0000 7f2381f7e740 1 --2- 172.21.15.39:0/602512577 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560d5c86da90 0x560d5c867570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:12.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.959+0000 7f2381f7e740 1 --2- 172.21.15.39:0/602512577 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560d5c85b420 0x560d5c867030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:12.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.959+0000 7f2381f7e740 1 --2- 172.21.15.39:0/602512577 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d5c8558c0 0x560d5c850070 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:12.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.959+0000 7f2381f7e740 1 -- 172.21.15.39:0/602512577 >> 172.21.15.39:0/602512577 conn(0x560d5c850e30 msgr2=0x560d5c84f960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:12.961 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.960+0000 7f2381f7e740 1 -- 172.21.15.39:0/602512577 shutdown_connections 2024-03-18T16:20:12.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.960+0000 7f2381f7e740 1 -- 172.21.15.39:0/602512577 wait complete. 2024-03-18T16:20:12.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.961+0000 7f2381f7e740 1 -- start start 2024-03-18T16:20:12.963 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.961+0000 7f2381f7e740 1 Processor -- start 2024-03-18T16:20:12.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.965+0000 7f2381f7e740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560d5c8558c0 0x560d5c886120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:12.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.966+0000 7f2381f7e740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d5c85b420 0x560d5c886660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:12.967 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.966+0000 7f2381f7e740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560d5c86da90 0x560d5c887670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:12.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.966+0000 7f2377400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d5c85b420 0x560d5c886660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:12.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.966+0000 7f2377400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d5c85b420 0x560d5c886660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:57784/0 (socket says 172.21.15.39:57784) 2024-03-18T16:20:12.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.966+0000 7f2381f7e740 1 -- 172.21.15.39:0/2872873583 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f23781a9760 con 0x560d5c86da90 2024-03-18T16:20:12.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.966+0000 7f2381f7e740 1 -- 172.21.15.39:0/2872873583 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f23781a9090 con 0x560d5c85b420 2024-03-18T16:20:12.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.966+0000 7f2381f7e740 1 -- 172.21.15.39:0/2872873583 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f23781a62d0 con 0x560d5c8558c0 2024-03-18T16:20:12.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.966+0000 7f2377e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560d5c8558c0 0x560d5c886120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:12.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.966+0000 7f2381f7e740 10 client.0 _ll_register_callbacks cb 0x7f2368004160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:20:12.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.966+0000 7f237d200640 1 --2- 172.21.15.39:0/2872873583 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560d5c86da90 0x560d5c887670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:12.968 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.966+0000 7f2377400640 1 -- 172.21.15.39:0/2872873583 learned_addr learned my addr 172.21.15.39:0/2872873583 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:20:12.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.967+0000 7f2377400640 1 -- 172.21.15.39:0/2872873583 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560d5c8558c0 msgr2=0x560d5c886120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:12.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.967+0000 7f2377400640 1 --2- 172.21.15.39:0/2872873583 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560d5c8558c0 0x560d5c886120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:12.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.967+0000 7f2377400640 1 -- 172.21.15.39:0/2872873583 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560d5c86da90 msgr2=0x560d5c887670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:12.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.967+0000 7f2377400640 1 --2- 172.21.15.39:0/2872873583 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560d5c86da90 0x560d5c887670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:12.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.967+0000 7f2377400640 1 -- 172.21.15.39:0/2872873583 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560d5c887c60 con 0x560d5c85b420 2024-03-18T16:20:12.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.968+0000 7f2377400640 1 --2- 172.21.15.39:0/2872873583 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d5c85b420 0x560d5c886660 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f2360017000 tx=0x7f23600047d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2cb487d64db2b8a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:12.969 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.968+0000 7f2366000640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:20:12.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.968+0000 7f2366000640 1 -- 172.21.15.39:0/2872873583 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2360035060 con 0x560d5c85b420 2024-03-18T16:20:12.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.968+0000 7f2381f7e740 1 -- 172.21.15.39:0/2872873583 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f23700c1c00 con 0x560d5c85b420 2024-03-18T16:20:12.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.968+0000 7f2366000640 1 -- 172.21.15.39:0/2872873583 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f2360013600 con 0x560d5c85b420 2024-03-18T16:20:12.970 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.969+0000 7f2366000640 1 -- 172.21.15.39:0/2872873583 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=44) ==== 24+0+0 (secure 0 0 0) 0x7f236003c050 con 0x560d5c85b420 2024-03-18T16:20:12.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.969+0000 7f2381f7e740 10 client.27665 fetch_fsmap learned FSMap version 44 2024-03-18T16:20:12.971 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.969+0000 7f2381f7e740 1 -- 172.21.15.39:0/2872873583 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=44,monmap=4+}) -- 0x560d5c8db2f0 con 0x560d5c85b420 2024-03-18T16:20:12.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.969+0000 7f2366000640 1 -- 172.21.15.39:0/2872873583 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f23700c1c00 con 0x560d5c85b420 2024-03-18T16:20:12.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.971+0000 7f2366000640 1 -- 172.21.15.39:0/2872873583 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 44) ==== 8362+0+0 (secure 0 0 0) 0x7f23600307b0 con 0x560d5c85b420 2024-03-18T16:20:12.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.971+0000 7f2381f7e740 10 client.27665 fetch_fsmap finished waiting for FSMap version 44 2024-03-18T16:20:12.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.971+0000 7f2381f7e740 10 client.27665 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-18T16:20:12.972 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.971+0000 7f2381f7e740 20 client.27665 populate_metadata read hostname 'smithi039' 2024-03-18T16:20:12.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.971+0000 7f2381f7e740 1 --2- 172.21.15.39:0/2872873583 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x560d5c874c40 0x560d5c8ddba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:12.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.971+0000 7f2381f7e740 4 client.27665 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:20:12.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.971+0000 7f2381f7e740 1 -- 172.21.15.39:0/2872873583 --> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560d5c7c1ce0 con 0x560d5c874c40 2024-03-18T16:20:12.973 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.972+0000 7f2377e00640 1 --2- 172.21.15.39:0/2872873583 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x560d5c874c40 0x560d5c8ddba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:12.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.972+0000 7f2377e00640 1 --2- 172.21.15.39:0/2872873583 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x560d5c874c40 0x560d5c8ddba0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=f6bf4794b4efbf4f server_cookie=49f3a7218ef5c6a3 in_seq=0 out_seq=0 2024-03-18T16:20:12.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.972+0000 7f2366000640 10 client.27665 ms_handle_connect on v2:172.21.15.194:6836/1301464246 2024-03-18T16:20:12.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.973+0000 7f2366000640 1 -- 172.21.15.39:0/2872873583 <== mds.3 v2:172.21.15.194:6836/1301464246 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560d5c7c1ce0 con 0x560d5c874c40 2024-03-18T16:20:12.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.973+0000 7f2366000640 10 client.27665 handle_command_reply: tid=0 2024-03-18T16:20:12.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.973+0000 7f2381f7e740 1 client.27665 shutdown 2024-03-18T16:20:12.974 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.973+0000 7f2381f7e740 1 -- 172.21.15.39:0/2872873583 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d5c85b420 msgr2=0x560d5c886660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:12.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.973+0000 7f2381f7e740 1 --2- 172.21.15.39:0/2872873583 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d5c85b420 0x560d5c886660 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f2360017000 tx=0x7f23600047d0 comp rx=0 tx=0).stop 2024-03-18T16:20:12.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.973+0000 7f2381f7e740 1 -- 172.21.15.39:0/2872873583 shutdown_connections 2024-03-18T16:20:12.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.973+0000 7f2381f7e740 1 --2- 172.21.15.39:0/2872873583 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x560d5c874c40 0x560d5c8ddba0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:12.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.973+0000 7f2381f7e740 1 --2- 172.21.15.39:0/2872873583 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d5c85b420 0x560d5c886660 secure :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f2360017000 tx=0x7f23600047d0 comp rx=0 tx=0).stop 2024-03-18T16:20:12.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.973+0000 7f2381f7e740 1 --2- 172.21.15.39:0/2872873583 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560d5c86da90 0x560d5c887670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:12.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.973+0000 7f2381f7e740 1 --2- 172.21.15.39:0/2872873583 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560d5c8558c0 0x560d5c886120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:12.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.973+0000 7f2381f7e740 1 -- 172.21.15.39:0/2872873583 >> 172.21.15.39:0/2872873583 conn(0x560d5c850e30 msgr2=0x560d5c8501f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:12.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.973+0000 7f2366000640 0 client.27665 ms_handle_reset on v2:172.21.15.194:6836/1301464246 2024-03-18T16:20:12.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.974+0000 7f2381f7e740 1 -- 172.21.15.39:0/2872873583 shutdown_connections 2024-03-18T16:20:12.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.974+0000 7f2381f7e740 1 -- 172.21.15.39:0/2872873583 wait complete. 2024-03-18T16:20:12.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.974+0000 7f2381f7e740 20 client.27665 trim_cache size 0 max 16384 2024-03-18T16:20:12.976 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:20:12.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.977+0000 7f237f200640 1 -- 172.21.15.39:0/1445575961 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f235c0a2810 msgr2=0x7f235c0a4cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:12.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.977+0000 7f237f200640 1 --2- 172.21.15.39:0/1445575961 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f235c0a2810 0x7f235c0a4cd0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f2360006f60 tx=0x7f2360000bc0 comp rx=0 tx=0).stop 2024-03-18T16:20:12.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.977+0000 7f237f200640 1 -- 172.21.15.39:0/1445575961 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f23780741c0 msgr2=0x7f2378199d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:12.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.977+0000 7f237f200640 1 --2- 172.21.15.39:0/1445575961 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f23780741c0 0x7f2378199d50 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f23700326d0 tx=0x7f2370003ba0 comp rx=0 tx=0).stop 2024-03-18T16:20:12.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.977+0000 7f237f200640 1 -- 172.21.15.39:0/1445575961 shutdown_connections 2024-03-18T16:20:12.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.977+0000 7f237f200640 1 --2- 172.21.15.39:0/1445575961 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2378109690 0x7f23781a12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:12.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.977+0000 7f237f200640 1 --2- 172.21.15.39:0/1445575961 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f235c0a2810 0x7f235c0a4cd0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:12.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.977+0000 7f237f200640 1 --2- 172.21.15.39:0/1445575961 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f23780757a0 0x7f237819a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:12.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.978+0000 7f237f200640 1 --2- 172.21.15.39:0/1445575961 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f23780741c0 0x7f2378199d50 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:12.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.978+0000 7f237f200640 1 -- 172.21.15.39:0/1445575961 >> 172.21.15.39:0/1445575961 conn(0x7f23780fcd50 msgr2=0x7f2378105620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:12.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.978+0000 7f237f200640 1 -- 172.21.15.39:0/1445575961 shutdown_connections 2024-03-18T16:20:12.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:12.978+0000 7f237f200640 1 -- 172.21.15.39:0/1445575961 wait complete. 2024-03-18T16:20:13.006 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-18T16:20:13.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.329+0000 7f6a8d400640 1 -- 172.21.15.39:0/2140729972 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6a8810c220 msgr2=0x7f6a8810e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:13.331 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.329+0000 7f6a8d400640 1 --2- 172.21.15.39:0/2140729972 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6a8810c220 0x7f6a8810e610 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f6a74009900 tx=0x7f6a7402bc50 comp rx=0 tx=0).stop 2024-03-18T16:20:13.332 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.331+0000 7f6a8d400640 1 -- 172.21.15.39:0/2140729972 shutdown_connections 2024-03-18T16:20:13.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.331+0000 7f6a8d400640 1 --2- 172.21.15.39:0/2140729972 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6a8810eb50 0x7f6a88111010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:13.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.331+0000 7f6a8d400640 1 --2- 172.21.15.39:0/2140729972 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6a8810c220 0x7f6a8810e610 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:13.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.331+0000 7f6a8d400640 1 --2- 172.21.15.39:0/2140729972 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6a881098f0 0x7f6a8810bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:13.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.331+0000 7f6a8d400640 1 -- 172.21.15.39:0/2140729972 >> 172.21.15.39:0/2140729972 conn(0x7f6a8806e860 msgr2=0x7f6a88070ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:13.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.334+0000 7f6a8d400640 1 -- 172.21.15.39:0/2140729972 shutdown_connections 2024-03-18T16:20:13.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.334+0000 7f6a8d400640 1 -- 172.21.15.39:0/2140729972 wait complete. 2024-03-18T16:20:13.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.335+0000 7f6a8d400640 1 Processor -- start 2024-03-18T16:20:13.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:13 smithi039 ceph-mon[31721]: pgmap v726: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 580 KiB/s wr, 155 op/s 2024-03-18T16:20:13.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:13 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/4173502874' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:20:13.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.335+0000 7f6a8d400640 1 -- start start 2024-03-18T16:20:13.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.336+0000 7f6a8d400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6a881098f0 0x7f6a8819f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:13.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.336+0000 7f6a8d400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6a8810c220 0x7f6a8819f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:13.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.336+0000 7f6a8d400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6a8810eb50 0x7f6a881a68d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:13.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.336+0000 7f6a8d400640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f6a88114c50 con 0x7f6a8810eb50 2024-03-18T16:20:13.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.336+0000 7f6a8d400640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f6a88114ad0 con 0x7f6a8810c220 2024-03-18T16:20:13.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.336+0000 7f6a8d400640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f6a88114dd0 con 0x7f6a881098f0 2024-03-18T16:20:13.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.336+0000 7f6a86800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6a881098f0 0x7f6a8819f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:13.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.336+0000 7f6a85e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6a8810c220 0x7f6a8819f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:13.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.336+0000 7f6a85e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6a8810c220 0x7f6a8819f860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:57792/0 (socket says 172.21.15.39:57792) 2024-03-18T16:20:13.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.336+0000 7f6a85e00640 1 -- 172.21.15.39:0/1486042644 learned_addr learned my addr 172.21.15.39:0/1486042644 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:20:13.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.337+0000 7f6a85e00640 1 -- 172.21.15.39:0/1486042644 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6a881098f0 msgr2=0x7f6a8819f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:13.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.337+0000 7f6a85e00640 1 --2- 172.21.15.39:0/1486042644 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6a881098f0 0x7f6a8819f320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:13.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.337+0000 7f6a85e00640 1 -- 172.21.15.39:0/1486042644 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6a8810eb50 msgr2=0x7f6a881a68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:13.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.337+0000 7f6a85e00640 1 --2- 172.21.15.39:0/1486042644 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6a8810eb50 0x7f6a881a68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:13.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.337+0000 7f6a85e00640 1 -- 172.21.15.39:0/1486042644 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a881a6f40 con 0x7f6a8810c220 2024-03-18T16:20:13.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.337+0000 7f6a85e00640 1 --2- 172.21.15.39:0/1486042644 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6a8810c220 0x7f6a8819f860 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f6a74036040 tx=0x7f6a74005d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:13.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.338+0000 7f6a7f400640 1 -- 172.21.15.39:0/1486042644 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a74046070 con 0x7f6a8810c220 2024-03-18T16:20:13.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.338+0000 7f6a7f400640 1 -- 172.21.15.39:0/1486042644 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a74005000 con 0x7f6a8810c220 2024-03-18T16:20:13.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.338+0000 7f6a8d400640 1 -- 172.21.15.39:0/1486042644 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a881087e0 con 0x7f6a8810c220 2024-03-18T16:20:13.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.338+0000 7f6a7f400640 1 -- 172.21.15.39:0/1486042644 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a74003970 con 0x7f6a8810c220 2024-03-18T16:20:13.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.338+0000 7f6a8d400640 1 -- 172.21.15.39:0/1486042644 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a88108da0 con 0x7f6a8810c220 2024-03-18T16:20:13.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.346+0000 7f6a90081740 1 Processor -- start 2024-03-18T16:20:13.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.348+0000 7f6a90081740 1 -- start start 2024-03-18T16:20:13.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.348+0000 7f6a7f400640 1 -- 172.21.15.39:0/1486042644 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f6a74049070 con 0x7f6a8810c220 2024-03-18T16:20:13.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.348+0000 7f6a90081740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564d6aea58c0 0x564d6aea5ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:13.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.348+0000 7f6a90081740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564d6aea62b0 0x564d6aeaadb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:13.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.348+0000 7f6a90081740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564d6aeab420 0x564d6aead8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:13.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.348+0000 7f6a90081740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x564d6ae6d3b0 con 0x564d6aea62b0 2024-03-18T16:20:13.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.348+0000 7f6a90081740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x564d6ae0a450 con 0x564d6aeab420 2024-03-18T16:20:13.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.348+0000 7f6a90081740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x564d6adb2cb0 con 0x564d6aea58c0 2024-03-18T16:20:13.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.349+0000 7f6a86800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564d6aea62b0 0x564d6aeaadb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:13.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.349+0000 7f6a85e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564d6aeab420 0x564d6aead8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:13.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.349+0000 7f6a85e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564d6aeab420 0x564d6aead8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:57804/0 (socket says 172.21.15.39:57804) 2024-03-18T16:20:13.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.349+0000 7f6a85e00640 1 -- 172.21.15.39:0/3207956200 learned_addr learned my addr 172.21.15.39:0/3207956200 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:20:13.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.349+0000 7f6a87200640 1 --2- 172.21.15.39:0/3207956200 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564d6aea58c0 0x564d6aea5ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:13.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.350+0000 7f6a85e00640 1 -- 172.21.15.39:0/3207956200 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564d6aea58c0 msgr2=0x564d6aea5ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:13.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.350+0000 7f6a85e00640 1 --2- 172.21.15.39:0/3207956200 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564d6aea58c0 0x564d6aea5ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:13.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.350+0000 7f6a85e00640 1 -- 172.21.15.39:0/3207956200 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564d6aea62b0 msgr2=0x564d6aeaadb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:13.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.350+0000 7f6a85e00640 1 --2- 172.21.15.39:0/3207956200 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564d6aea62b0 0x564d6aeaadb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:13.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.350+0000 7f6a85e00640 1 -- 172.21.15.39:0/3207956200 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564d6add41f0 con 0x564d6aeab420 2024-03-18T16:20:13.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.350+0000 7f6a7f400640 1 --2- 172.21.15.39:0/1486042644 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f6a6c0a2880 0x7f6a6c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:13.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.351+0000 7f6a7f400640 1 -- 172.21.15.39:0/1486042644 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f6a740f4e50 con 0x7f6a8810c220 2024-03-18T16:20:13.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.351+0000 7f6a85e00640 1 --2- 172.21.15.39:0/3207956200 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564d6aeab420 0x564d6aead8b0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f6a740f8d70 tx=0x7f6a740f8e60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8dd20dbb7bc147b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:13.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.351+0000 7f6a87200640 1 --2- 172.21.15.39:0/1486042644 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f6a6c0a2880 0x7f6a6c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:13.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.352+0000 7f6a73400640 1 -- 172.21.15.39:0/3207956200 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a74050030 con 0x564d6aeab420 2024-03-18T16:20:13.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.352+0000 7f6a73400640 1 -- 172.21.15.39:0/3207956200 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a740c1b00 con 0x564d6aeab420 2024-03-18T16:20:13.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.352+0000 7f6a73400640 1 -- 172.21.15.39:0/3207956200 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a740c0f40 con 0x564d6aeab420 2024-03-18T16:20:13.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.353+0000 7f6a90081740 1 -- 172.21.15.39:0/3207956200 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564d6aeab420 msgr2=0x564d6aead8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:13.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.353+0000 7f6a90081740 1 --2- 172.21.15.39:0/3207956200 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564d6aeab420 0x564d6aead8b0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f6a740f8d70 tx=0x7f6a740f8e60 comp rx=0 tx=0).stop 2024-03-18T16:20:13.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.354+0000 7f6a87200640 1 --2- 172.21.15.39:0/1486042644 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f6a6c0a2880 0x7f6a6c0a4d40 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x564d6aea6b10 tx=0x7f6a80009450 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:13.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.354+0000 7f6a90081740 1 -- 172.21.15.39:0/3207956200 shutdown_connections 2024-03-18T16:20:13.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.354+0000 7f6a90081740 1 --2- 172.21.15.39:0/3207956200 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564d6aeab420 0x564d6aead8b0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:13.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.354+0000 7f6a90081740 1 --2- 172.21.15.39:0/3207956200 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564d6aea62b0 0x564d6aeaadb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:13.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.354+0000 7f6a90081740 1 --2- 172.21.15.39:0/3207956200 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564d6aea58c0 0x564d6aea5ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:13.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.354+0000 7f6a90081740 1 -- 172.21.15.39:0/3207956200 >> 172.21.15.39:0/3207956200 conn(0x564d6aea0e30 msgr2=0x564d6aea1230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:13.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.354+0000 7f6a90081740 1 -- 172.21.15.39:0/3207956200 shutdown_connections 2024-03-18T16:20:13.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.355+0000 7f6a90081740 1 -- 172.21.15.39:0/3207956200 wait complete. 2024-03-18T16:20:13.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.355+0000 7f6a90081740 1 -- start start 2024-03-18T16:20:13.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.356+0000 7f6a90081740 1 Processor -- start 2024-03-18T16:20:13.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.356+0000 7f6a90081740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564d6aea58c0 0x564d6aeaf8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:13.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.356+0000 7f6a90081740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564d6aea62b0 0x564d6aeafe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:13.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.356+0000 7f6a90081740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564d6aeab420 0x564d6aeb6e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:13.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.356+0000 7f6a90081740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x564d6adc6110 con 0x564d6aea62b0 2024-03-18T16:20:13.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.356+0000 7f6a90081740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x564d6ae064f0 con 0x564d6aeab420 2024-03-18T16:20:13.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.356+0000 7f6a90081740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x564d6aeae190 con 0x564d6aea58c0 2024-03-18T16:20:13.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.356+0000 7f6a90081740 10 client.0 _ll_register_callbacks cb 0x564d6adb3b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:20:13.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.357+0000 7f6a87200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564d6aeab420 0x564d6aeb6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:13.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.357+0000 7f6a87200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564d6aeab420 0x564d6aeb6e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:57812/0 (socket says 172.21.15.39:57812) 2024-03-18T16:20:13.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.357+0000 7f6a87200640 1 -- 172.21.15.39:0/2145055487 learned_addr learned my addr 172.21.15.39:0/2145055487 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:20:13.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.357+0000 7f6a87200640 1 -- 172.21.15.39:0/2145055487 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564d6aea58c0 msgr2=0x564d6aeaf8c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T16:20:13.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.357+0000 7f6a87200640 1 --2- 172.21.15.39:0/2145055487 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564d6aea58c0 0x564d6aeaf8c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:13.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.357+0000 7f6a87200640 1 -- 172.21.15.39:0/2145055487 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564d6aea62b0 msgr2=0x564d6aeafe00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:20:13.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.358+0000 7f6a87200640 1 --2- 172.21.15.39:0/2145055487 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564d6aea62b0 0x564d6aeafe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:13.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.358+0000 7f6a87200640 1 -- 172.21.15.39:0/2145055487 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564d6ab1e3f0 con 0x564d6aeab420 2024-03-18T16:20:13.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.358+0000 7f6a87200640 1 --2- 172.21.15.39:0/2145055487 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564d6aeab420 0x564d6aeb6e40 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f6a80009aa0 tx=0x7f6a80004b40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c4de30849b112e3a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:13.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.359+0000 7f6a67e00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:20:13.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.359+0000 7f6a67e00640 1 -- 172.21.15.39:0/2145055487 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a80027060 con 0x564d6aeab420 2024-03-18T16:20:13.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.359+0000 7f6a67e00640 1 -- 172.21.15.39:0/2145055487 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f6a8001d030 con 0x564d6aeab420 2024-03-18T16:20:13.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.360+0000 7f6a90081740 1 -- 172.21.15.39:0/2145055487 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6a740c1b00 con 0x564d6aeab420 2024-03-18T16:20:13.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.360+0000 7f6a67e00640 1 -- 172.21.15.39:0/2145055487 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f6a80027060 con 0x564d6aeab420 2024-03-18T16:20:13.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.360+0000 7f6a90081740 10 client.27674 fetch_fsmap learned FSMap version 45 2024-03-18T16:20:13.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.360+0000 7f6a90081740 1 -- 172.21.15.39:0/2145055487 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x564d6abe87d0 con 0x564d6aeab420 2024-03-18T16:20:13.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.361+0000 7f6a67e00640 1 -- 172.21.15.39:0/2145055487 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a80016a40 con 0x564d6aeab420 2024-03-18T16:20:13.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.361+0000 7f6a67e00640 1 -- 172.21.15.39:0/2145055487 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 45) ==== 8366+0+0 (secure 0 0 0) 0x7f6a80022a20 con 0x564d6aeab420 2024-03-18T16:20:13.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.362+0000 7f6a90081740 10 client.27674 fetch_fsmap finished waiting for FSMap version 45 2024-03-18T16:20:13.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.362+0000 7f6a90081740 10 client.27674 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-18T16:20:13.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.362+0000 7f6a90081740 20 client.27674 populate_metadata read hostname 'smithi039' 2024-03-18T16:20:13.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.362+0000 7f6a90081740 1 --2- 172.21.15.39:0/2145055487 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x564d6aebb730 0x564d6aebdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:13.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.362+0000 7f6a90081740 4 client.27674 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:20:13.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.362+0000 7f6a90081740 1 -- 172.21.15.39:0/2145055487 --> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f6a740c1b00 con 0x564d6aebb730 2024-03-18T16:20:13.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.362+0000 7f6a86800640 1 --2- 172.21.15.39:0/2145055487 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x564d6aebb730 0x564d6aebdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:13.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.364+0000 7f6a86800640 1 --2- 172.21.15.39:0/2145055487 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x564d6aebb730 0x564d6aebdbd0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b4e3fc23f3f01525 server_cookie=38f8f1afdba3aa9d in_seq=0 out_seq=0 2024-03-18T16:20:13.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.364+0000 7f6a67e00640 10 client.27674 ms_handle_connect on v2:172.21.15.196:6836/895247251 2024-03-18T16:20:13.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.365+0000 7f6a67e00640 1 -- 172.21.15.39:0/2145055487 <== mds.2 v2:172.21.15.196:6836/895247251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x7f6a740c1b00 con 0x564d6aebb730 2024-03-18T16:20:13.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.365+0000 7f6a67e00640 10 client.27674 handle_command_reply: tid=0 2024-03-18T16:20:13.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.366+0000 7f6a90081740 1 client.27674 shutdown 2024-03-18T16:20:13.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.366+0000 7f6a90081740 1 -- 172.21.15.39:0/2145055487 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564d6aeab420 msgr2=0x564d6aeb6e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:13.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.366+0000 7f6a90081740 1 --2- 172.21.15.39:0/2145055487 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564d6aeab420 0x564d6aeb6e40 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f6a80009aa0 tx=0x7f6a80004b40 comp rx=0 tx=0).stop 2024-03-18T16:20:13.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.367+0000 7f6a90081740 1 -- 172.21.15.39:0/2145055487 shutdown_connections 2024-03-18T16:20:13.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.367+0000 7f6a90081740 1 --2- 172.21.15.39:0/2145055487 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564d6aeab420 0x564d6aeb6e40 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:13.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.367+0000 7f6a90081740 1 --2- 172.21.15.39:0/2145055487 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x564d6aebb730 0x564d6aebdbd0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:13.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.367+0000 7f6a90081740 1 --2- 172.21.15.39:0/2145055487 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564d6aea62b0 0x564d6aeafe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:13.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.367+0000 7f6a90081740 1 --2- 172.21.15.39:0/2145055487 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564d6aea58c0 0x564d6aeaf8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:13.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.367+0000 7f6a90081740 1 -- 172.21.15.39:0/2145055487 >> 172.21.15.39:0/2145055487 conn(0x564d6aea0e30 msgr2=0x564d6aea01f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:13.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.367+0000 7f6a67e00640 0 client.27674 ms_handle_reset on v2:172.21.15.196:6836/895247251 2024-03-18T16:20:13.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.368+0000 7f6a90081740 1 -- 172.21.15.39:0/2145055487 shutdown_connections 2024-03-18T16:20:13.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.368+0000 7f6a90081740 1 -- 172.21.15.39:0/2145055487 wait complete. 2024-03-18T16:20:13.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.370+0000 7f6a90081740 20 client.27674 trim_cache size 0 max 16384 2024-03-18T16:20:13.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.400+0000 7f6a90081740 1 Processor -- start 2024-03-18T16:20:13.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.400+0000 7f6a90081740 1 -- start start 2024-03-18T16:20:13.402 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.401+0000 7f6a90081740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564d6aea0e30 0x564d6aebd740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:13.403 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.401+0000 7f6a90081740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564d6aea58c0 0x564d6aeb6e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:13.403 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.401+0000 7f6a90081740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564d6aea62b0 0x564d6ae9f960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:13.403 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.401+0000 7f6a90081740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f6a8807d5e0 con 0x564d6aea0e30 2024-03-18T16:20:13.403 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.401+0000 7f6a90081740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f6a8807f150 con 0x564d6aea62b0 2024-03-18T16:20:13.403 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.401+0000 7f6a90081740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f6a8807dcb0 con 0x564d6aea58c0 2024-03-18T16:20:13.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.402+0000 7f6a87200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564d6aea62b0 0x564d6ae9f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:13.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.402+0000 7f6a87200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564d6aea62b0 0x564d6ae9f960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:57826/0 (socket says 172.21.15.39:57826) 2024-03-18T16:20:13.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.402+0000 7f6a87200640 1 -- 172.21.15.39:0/1771256725 learned_addr learned my addr 172.21.15.39:0/1771256725 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:20:13.404 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.402+0000 7f6a85e00640 1 --2- 172.21.15.39:0/1771256725 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564d6aea58c0 0x564d6aeb6e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:13.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.403+0000 7f6a87200640 1 -- 172.21.15.39:0/1771256725 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564d6aea58c0 msgr2=0x564d6aeb6e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:13.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.403+0000 7f6a87200640 1 --2- 172.21.15.39:0/1771256725 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564d6aea58c0 0x564d6aeb6e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:13.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.403+0000 7f6a87200640 1 -- 172.21.15.39:0/1771256725 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564d6aea0e30 msgr2=0x564d6aebd740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:13.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.403+0000 7f6a87200640 1 --2- 172.21.15.39:0/1771256725 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564d6aea0e30 0x564d6aebd740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:13.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.403+0000 7f6a87200640 1 -- 172.21.15.39:0/1771256725 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564d6ae9cc10 con 0x564d6aea62b0 2024-03-18T16:20:13.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.403+0000 7f6a87200640 1 --2- 172.21.15.39:0/1771256725 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564d6aea62b0 0x564d6ae9f960 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f6a8000cc80 tx=0x7f6a8001efb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b3f27f1b1b2bf8e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:13.405 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.403+0000 7f6a72a00640 1 -- 172.21.15.39:0/1771256725 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a80031080 con 0x564d6aea62b0 2024-03-18T16:20:13.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.403+0000 7f6a72a00640 1 -- 172.21.15.39:0/1771256725 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a80007650 con 0x564d6aea62b0 2024-03-18T16:20:13.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.403+0000 7f6a72a00640 1 -- 172.21.15.39:0/1771256725 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a80027060 con 0x564d6aea62b0 2024-03-18T16:20:13.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.405+0000 7f6a90081740 1 -- 172.21.15.39:0/1771256725 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564d6aea62b0 msgr2=0x564d6ae9f960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:13.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.405+0000 7f6a90081740 1 --2- 172.21.15.39:0/1771256725 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564d6aea62b0 0x564d6ae9f960 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f6a8000cc80 tx=0x7f6a8001efb0 comp rx=0 tx=0).stop 2024-03-18T16:20:13.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.405+0000 7f6a90081740 1 -- 172.21.15.39:0/1771256725 shutdown_connections 2024-03-18T16:20:13.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.405+0000 7f6a90081740 1 --2- 172.21.15.39:0/1771256725 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564d6aea62b0 0x564d6ae9f960 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:13.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.405+0000 7f6a90081740 1 --2- 172.21.15.39:0/1771256725 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564d6aea58c0 0x564d6aeb6e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:13.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.405+0000 7f6a90081740 1 --2- 172.21.15.39:0/1771256725 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564d6aea0e30 0x564d6aebd740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:13.407 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.405+0000 7f6a90081740 1 -- 172.21.15.39:0/1771256725 >> 172.21.15.39:0/1771256725 conn(0x564d6aeaaf60 msgr2=0x564d6aeaf8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:13.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.406+0000 7f6a90081740 1 -- 172.21.15.39:0/1771256725 shutdown_connections 2024-03-18T16:20:13.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.406+0000 7f6a90081740 1 -- 172.21.15.39:0/1771256725 wait complete. 2024-03-18T16:20:13.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.406+0000 7f6a90081740 1 -- start start 2024-03-18T16:20:13.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.407+0000 7f6a90081740 1 Processor -- start 2024-03-18T16:20:13.409 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.408+0000 7f6a90081740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564d6aea0e30 0x564d6aeb48a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:13.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.408+0000 7f6a90081740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564d6aea58c0 0x564d6aeb4de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:13.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.408+0000 7f6a90081740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564d6aea62b0 0x564d6aeb5df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:13.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.409+0000 7f6a90081740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f6a8807ea50 con 0x564d6aea62b0 2024-03-18T16:20:13.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.409+0000 7f6a90081740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f6a8807e380 con 0x564d6aea0e30 2024-03-18T16:20:13.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.409+0000 7f6a90081740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f6a8807b580 con 0x564d6aea58c0 2024-03-18T16:20:13.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.409+0000 7f6a86800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564d6aea0e30 0x564d6aeb48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:13.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.409+0000 7f6a86800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564d6aea0e30 0x564d6aeb48a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:57832/0 (socket says 172.21.15.39:57832) 2024-03-18T16:20:13.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.409+0000 7f6a86800640 1 -- 172.21.15.39:0/4021235548 learned_addr learned my addr 172.21.15.39:0/4021235548 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:20:13.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.409+0000 7f6a90081740 10 client.0 _ll_register_callbacks cb 0x7f6a80009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:20:13.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.409+0000 7f6a87200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564d6aea62b0 0x564d6aeb5df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:13.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.410+0000 7f6a86800640 1 -- 172.21.15.39:0/4021235548 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564d6aea58c0 msgr2=0x564d6aeb4de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:13.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.409+0000 7f6a85e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564d6aea58c0 0x564d6aeb4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:13.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.410+0000 7f6a86800640 1 --2- 172.21.15.39:0/4021235548 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564d6aea58c0 0x564d6aeb4de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:13.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.410+0000 7f6a86800640 1 -- 172.21.15.39:0/4021235548 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564d6aea62b0 msgr2=0x564d6aeb5df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:13.413 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.410+0000 7f6a86800640 1 --2- 172.21.15.39:0/4021235548 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564d6aea62b0 0x564d6aeb5df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:13.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.410+0000 7f6a86800640 1 -- 172.21.15.39:0/4021235548 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564d6aeb63e0 con 0x564d6aea0e30 2024-03-18T16:20:13.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.410+0000 7f6a85e00640 1 --2- 172.21.15.39:0/4021235548 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564d6aea58c0 0x564d6aeb4de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:20:13.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.412+0000 7f6a87200640 1 --2- 172.21.15.39:0/4021235548 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564d6aea62b0 0x564d6aeb5df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:20:13.414 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.412+0000 7f6a86800640 1 --2- 172.21.15.39:0/4021235548 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564d6aea0e30 0x564d6aeb48a0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f6a78000c00 tx=0x7f6a78009e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=67629e44d9807f97 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:13.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.412+0000 7f6a67e00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:20:13.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.412+0000 7f6a67e00640 1 -- 172.21.15.39:0/4021235548 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a78016ce0 con 0x564d6aea0e30 2024-03-18T16:20:13.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.412+0000 7f6a67e00640 1 -- 172.21.15.39:0/4021235548 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f6a78014410 con 0x564d6aea0e30 2024-03-18T16:20:13.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.412+0000 7f6a90081740 1 -- 172.21.15.39:0/4021235548 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x564d6ae17990 con 0x564d6aea0e30 2024-03-18T16:20:13.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.413+0000 7f6a67e00640 1 -- 172.21.15.39:0/4021235548 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f6a7802c050 con 0x564d6aea0e30 2024-03-18T16:20:13.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.414+0000 7f6a90081740 10 client.27680 fetch_fsmap learned FSMap version 45 2024-03-18T16:20:13.415 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.414+0000 7f6a90081740 1 -- 172.21.15.39:0/4021235548 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x564d6af2a6d0 con 0x564d6aea0e30 2024-03-18T16:20:13.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.414+0000 7f6a67e00640 1 -- 172.21.15.39:0/4021235548 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x564d6ae17990 con 0x564d6aea0e30 2024-03-18T16:20:13.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.417+0000 7f6a67e00640 1 -- 172.21.15.39:0/4021235548 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 45) ==== 8366+0+0 (secure 0 0 0) 0x7f6a78014610 con 0x564d6aea0e30 2024-03-18T16:20:13.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.418+0000 7f6a90081740 10 client.27680 fetch_fsmap finished waiting for FSMap version 45 2024-03-18T16:20:13.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.418+0000 7f6a90081740 10 client.27680 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-18T16:20:13.419 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.418+0000 7f6a90081740 20 client.27680 populate_metadata read hostname 'smithi039' 2024-03-18T16:20:13.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.418+0000 7f6a90081740 1 --2- 172.21.15.39:0/4021235548 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x564d6aeab420 0x564d6af2d060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:13.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.418+0000 7f6a90081740 4 client.27680 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:20:13.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.418+0000 7f6a90081740 1 -- 172.21.15.39:0/4021235548 --> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564d6ae11ce0 con 0x564d6aeab420 2024-03-18T16:20:13.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.419+0000 7f6a85e00640 1 --2- 172.21.15.39:0/4021235548 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x564d6aeab420 0x564d6af2d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:13.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.420+0000 7f6a85e00640 1 --2- 172.21.15.39:0/4021235548 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x564d6aeab420 0x564d6af2d060 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b6fc0703dc73dc0b server_cookie=ddfb411fbcf7a770 in_seq=0 out_seq=0 2024-03-18T16:20:13.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.420+0000 7f6a67e00640 10 client.27680 ms_handle_connect on v2:172.21.15.196:6836/895247251 2024-03-18T16:20:13.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.421+0000 7f6a67e00640 1 -- 172.21.15.39:0/4021235548 <== mds.2 v2:172.21.15.196:6836/895247251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564d6ae11ce0 con 0x564d6aeab420 2024-03-18T16:20:13.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.421+0000 7f6a67e00640 10 client.27680 handle_command_reply: tid=0 2024-03-18T16:20:13.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.421+0000 7f6a90081740 1 client.27680 shutdown 2024-03-18T16:20:13.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.422+0000 7f6a90081740 1 -- 172.21.15.39:0/4021235548 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564d6aea0e30 msgr2=0x564d6aeb48a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:13.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.422+0000 7f6a90081740 1 --2- 172.21.15.39:0/4021235548 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564d6aea0e30 0x564d6aeb48a0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f6a78000c00 tx=0x7f6a78009e10 comp rx=0 tx=0).stop 2024-03-18T16:20:13.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.422+0000 7f6a90081740 1 -- 172.21.15.39:0/4021235548 shutdown_connections 2024-03-18T16:20:13.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.423+0000 7f6a90081740 1 --2- 172.21.15.39:0/4021235548 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x564d6aeab420 0x564d6af2d060 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:13.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.423+0000 7f6a90081740 1 --2- 172.21.15.39:0/4021235548 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x564d6aea62b0 0x564d6aeb5df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:13.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.423+0000 7f6a90081740 1 --2- 172.21.15.39:0/4021235548 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x564d6aea58c0 0x564d6aeb4de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:13.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.423+0000 7f6a90081740 1 --2- 172.21.15.39:0/4021235548 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564d6aea0e30 0x564d6aeb48a0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:13.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.423+0000 7f6a67e00640 0 client.27680 ms_handle_reset on v2:172.21.15.196:6836/895247251 2024-03-18T16:20:13.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.423+0000 7f6a90081740 1 -- 172.21.15.39:0/4021235548 >> 172.21.15.39:0/4021235548 conn(0x564d6aeaaf60 msgr2=0x564d6ae9f960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:13.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.424+0000 7f6a90081740 1 -- 172.21.15.39:0/4021235548 shutdown_connections 2024-03-18T16:20:13.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.424+0000 7f6a90081740 1 -- 172.21.15.39:0/4021235548 wait complete. 2024-03-18T16:20:13.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.426+0000 7f6a90081740 20 client.27680 trim_cache size 0 max 16384 2024-03-18T16:20:13.428 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:20:13.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.428+0000 7f6a8d400640 1 -- 172.21.15.39:0/1486042644 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f6a6c0a2880 msgr2=0x7f6a6c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:13.429 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.428+0000 7f6a8d400640 1 --2- 172.21.15.39:0/1486042644 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f6a6c0a2880 0x7f6a6c0a4d40 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x564d6aea6b10 tx=0x7f6a80009450 comp rx=0 tx=0).stop 2024-03-18T16:20:13.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.428+0000 7f6a8d400640 1 -- 172.21.15.39:0/1486042644 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6a8810c220 msgr2=0x7f6a8819f860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:13.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.428+0000 7f6a8d400640 1 --2- 172.21.15.39:0/1486042644 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6a8810c220 0x7f6a8819f860 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f6a74036040 tx=0x7f6a74005d50 comp rx=0 tx=0).stop 2024-03-18T16:20:13.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.428+0000 7f6a8d400640 1 -- 172.21.15.39:0/1486042644 shutdown_connections 2024-03-18T16:20:13.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.428+0000 7f6a8d400640 1 --2- 172.21.15.39:0/1486042644 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6a8810c220 0x7f6a8819f860 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:13.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.428+0000 7f6a8d400640 1 --2- 172.21.15.39:0/1486042644 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f6a6c0a2880 0x7f6a6c0a4d40 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:13.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.428+0000 7f6a8d400640 1 --2- 172.21.15.39:0/1486042644 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6a8810eb50 0x7f6a881a68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:13.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.428+0000 7f6a8d400640 1 --2- 172.21.15.39:0/1486042644 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6a881098f0 0x7f6a8819f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:13.430 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.428+0000 7f6a8d400640 1 -- 172.21.15.39:0/1486042644 >> 172.21.15.39:0/1486042644 conn(0x7f6a8806e860 msgr2=0x7f6a88109cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:13.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.428+0000 7f6a8d400640 1 -- 172.21.15.39:0/1486042644 shutdown_connections 2024-03-18T16:20:13.431 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.428+0000 7f6a8d400640 1 -- 172.21.15.39:0/1486042644 wait complete. 2024-03-18T16:20:13.459 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-18T16:20:13.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:13 smithi196 ceph-mon[36307]: pgmap v726: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 580 KiB/s wr, 155 op/s 2024-03-18T16:20:13.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:13 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/4173502874' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:20:13.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:13 smithi194 ceph-mon[36200]: pgmap v726: 129 pgs: 129 active+clean; 13 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 580 KiB/s wr, 155 op/s 2024-03-18T16:20:13.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:13 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/4173502874' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:20:13.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.761+0000 7f93bf200640 1 -- 172.21.15.39:0/2935914674 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f93b8009550 msgr2=0x7f93b8009970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:13.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.761+0000 7f93bf200640 1 --2- 172.21.15.39:0/2935914674 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f93b8009550 0x7f93b8009970 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f93b0009900 tx=0x7f93b002bc80 comp rx=0 tx=0).stop 2024-03-18T16:20:13.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.761+0000 7f93bf200640 1 -- 172.21.15.39:0/2935914674 shutdown_connections 2024-03-18T16:20:13.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.761+0000 7f93bf200640 1 --2- 172.21.15.39:0/2935914674 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f93b800b130 0x7f93b800b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:13.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.761+0000 7f93bf200640 1 --2- 172.21.15.39:0/2935914674 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f93b800a790 0x7f93b800abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:13.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.761+0000 7f93bf200640 1 --2- 172.21.15.39:0/2935914674 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f93b8009550 0x7f93b8009970 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:13.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.761+0000 7f93bf200640 1 -- 172.21.15.39:0/2935914674 >> 172.21.15.39:0/2935914674 conn(0x7f93b80934e0 msgr2=0x7f93b8095920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:13.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.761+0000 7f93bf200640 1 -- 172.21.15.39:0/2935914674 shutdown_connections 2024-03-18T16:20:13.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.761+0000 7f93bf200640 1 -- 172.21.15.39:0/2935914674 wait complete. 2024-03-18T16:20:13.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.763+0000 7f93bf200640 1 Processor -- start 2024-03-18T16:20:13.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.763+0000 7f93bf200640 1 -- start start 2024-03-18T16:20:13.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.763+0000 7f93bf200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f93b8009550 0x7f93b80a3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:13.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.764+0000 7f93bf200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f93b800a790 0x7f93b80a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:13.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.764+0000 7f93bf200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f93b800b130 0x7f93b80a5120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:13.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.764+0000 7f93bf200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f93b8098260 con 0x7f93b8009550 2024-03-18T16:20:13.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.764+0000 7f93bf200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f93b80980e0 con 0x7f93b800a790 2024-03-18T16:20:13.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.764+0000 7f93bf200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f93b80983e0 con 0x7f93b800b130 2024-03-18T16:20:13.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.764+0000 7f93be800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f93b800b130 0x7f93b80a5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:13.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.764+0000 7f93be800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f93b800b130 0x7f93b80a5120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:38072/0 (socket says 172.21.15.39:38072) 2024-03-18T16:20:13.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.764+0000 7f93be800640 1 -- 172.21.15.39:0/4022746686 learned_addr learned my addr 172.21.15.39:0/4022746686 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:20:13.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.764+0000 7f93bd400640 1 --2- 172.21.15.39:0/4022746686 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f93b800a790 0x7f93b80a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:13.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.765+0000 7f93bde00640 1 --2- 172.21.15.39:0/4022746686 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f93b8009550 0x7f93b80a3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:13.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.765+0000 7f93bd400640 1 -- 172.21.15.39:0/4022746686 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f93b800b130 msgr2=0x7f93b80a5120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:13.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.765+0000 7f93bd400640 1 --2- 172.21.15.39:0/4022746686 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f93b800b130 0x7f93b80a5120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:13.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.765+0000 7f93bd400640 1 -- 172.21.15.39:0/4022746686 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f93b8009550 msgr2=0x7f93b80a3ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:13.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.765+0000 7f93bd400640 1 --2- 172.21.15.39:0/4022746686 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f93b8009550 0x7f93b80a3ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:13.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.766+0000 7f93bd400640 1 -- 172.21.15.39:0/4022746686 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93b80a5790 con 0x7f93b800a790 2024-03-18T16:20:13.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.769+0000 7f93bd400640 1 --2- 172.21.15.39:0/4022746686 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f93b800a790 0x7f93b80a40e0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f93ac00d6d0 tx=0x7f93ac00db90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:13.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.769+0000 7f93b6a00640 1 -- 172.21.15.39:0/4022746686 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93ac017070 con 0x7f93b800a790 2024-03-18T16:20:13.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.769+0000 7f93bf200640 1 -- 172.21.15.39:0/4022746686 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93b80a5a80 con 0x7f93b800a790 2024-03-18T16:20:13.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.769+0000 7f93b6a00640 1 -- 172.21.15.39:0/4022746686 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f93ac009820 con 0x7f93b800a790 2024-03-18T16:20:13.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.769+0000 7f93b6a00640 1 -- 172.21.15.39:0/4022746686 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93ac004cd0 con 0x7f93b800a790 2024-03-18T16:20:13.771 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.769+0000 7f93bf200640 1 -- 172.21.15.39:0/4022746686 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93b81544e0 con 0x7f93b800a790 2024-03-18T16:20:13.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.776+0000 7f93b6a00640 1 -- 172.21.15.39:0/4022746686 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f93ac0099d0 con 0x7f93b800a790 2024-03-18T16:20:13.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.777+0000 7f93c7771740 1 Processor -- start 2024-03-18T16:20:13.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.778+0000 7f93c7771740 1 -- start start 2024-03-18T16:20:13.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.778+0000 7f93c7771740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562d79d0ba60 0x562d79d0be80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:13.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.778+0000 7f93c7771740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562d79d0c450 0x562d79d10f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:13.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.778+0000 7f93c7771740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562d79d115c0 0x562d79d13a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:13.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.778+0000 7f93c7771740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x562d79cd33b0 con 0x562d79d115c0 2024-03-18T16:20:13.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.778+0000 7f93c7771740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x562d79c70450 con 0x562d79d0ba60 2024-03-18T16:20:13.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.778+0000 7f93c7771740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x562d79c18cb0 con 0x562d79d0c450 2024-03-18T16:20:13.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.779+0000 7f93bde00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562d79d0c450 0x562d79d10f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:13.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.779+0000 7f93b6a00640 1 --2- 172.21.15.39:0/4022746686 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f93a40a2880 0x7f93a40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:13.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.779+0000 7f93bde00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562d79d0c450 0x562d79d10f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:38080/0 (socket says 172.21.15.39:38080) 2024-03-18T16:20:13.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.779+0000 7f93bd400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562d79d115c0 0x562d79d13a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:13.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.779+0000 7f93bde00640 1 -- 172.21.15.39:0/3335525937 learned_addr learned my addr 172.21.15.39:0/3335525937 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:20:13.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.779+0000 7f93be800640 1 --2- 172.21.15.39:0/3335525937 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562d79d0ba60 0x562d79d0be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:13.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.780+0000 7f93b6a00640 1 -- 172.21.15.39:0/4022746686 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f93ac0cea80 con 0x7f93b800a790 2024-03-18T16:20:13.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.780+0000 7f93bde00640 1 -- 172.21.15.39:0/3335525937 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562d79d0ba60 msgr2=0x562d79d0be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:13.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.780+0000 7f93bde00640 1 --2- 172.21.15.39:0/3335525937 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562d79d0ba60 0x562d79d0be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:13.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.780+0000 7f93bde00640 1 -- 172.21.15.39:0/3335525937 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562d79d115c0 msgr2=0x562d79d13a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:13.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.780+0000 7f93bde00640 1 --2- 172.21.15.39:0/3335525937 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562d79d115c0 0x562d79d13a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:13.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.780+0000 7f93bde00640 1 -- 172.21.15.39:0/3335525937 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562d79c3a1f0 con 0x562d79d0c450 2024-03-18T16:20:13.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.780+0000 7f93be800640 1 --2- 172.21.15.39:0/3335525937 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562d79d0ba60 0x562d79d0be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:20:13.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.780+0000 7f93be800640 1 --2- 172.21.15.39:0/4022746686 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f93a40a2880 0x7f93a40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:13.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.781+0000 7f93bde00640 1 --2- 172.21.15.39:0/3335525937 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562d79d0c450 0x562d79d10f50 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f93b0004b90 tx=0x7f93b0009600 comp rx=0 tx=0).ready entity=mon.1 client_cookie=92f2729e8abe9f97 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:13.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.781+0000 7f93be800640 1 --2- 172.21.15.39:0/4022746686 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f93a40a2880 0x7f93a40a4d40 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f93c00507b0 tx=0x7f93c0067580 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:13.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.782+0000 7f93ab000640 1 -- 172.21.15.39:0/3335525937 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93b0042020 con 0x562d79d0c450 2024-03-18T16:20:13.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.782+0000 7f93ab000640 1 -- 172.21.15.39:0/3335525937 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f93b0004880 con 0x562d79d0c450 2024-03-18T16:20:13.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.782+0000 7f93ab000640 1 -- 172.21.15.39:0/3335525937 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93b00035f0 con 0x562d79d0c450 2024-03-18T16:20:13.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.783+0000 7f93c7771740 1 -- 172.21.15.39:0/3335525937 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562d79d0c450 msgr2=0x562d79d10f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:13.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.783+0000 7f93c7771740 1 --2- 172.21.15.39:0/3335525937 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562d79d0c450 0x562d79d10f50 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f93b0004b90 tx=0x7f93b0009600 comp rx=0 tx=0).stop 2024-03-18T16:20:13.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.783+0000 7f93c7771740 1 -- 172.21.15.39:0/3335525937 shutdown_connections 2024-03-18T16:20:13.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.783+0000 7f93c7771740 1 --2- 172.21.15.39:0/3335525937 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562d79d115c0 0x562d79d13a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:13.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.783+0000 7f93c7771740 1 --2- 172.21.15.39:0/3335525937 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562d79d0c450 0x562d79d10f50 secure :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f93b0004b90 tx=0x7f93b0009600 comp rx=0 tx=0).stop 2024-03-18T16:20:13.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.783+0000 7f93c7771740 1 --2- 172.21.15.39:0/3335525937 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562d79d0ba60 0x562d79d0be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:13.785 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.783+0000 7f93c7771740 1 -- 172.21.15.39:0/3335525937 >> 172.21.15.39:0/3335525937 conn(0x562d79d06fd0 msgr2=0x562d79d073d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:13.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.783+0000 7f93c7771740 1 -- 172.21.15.39:0/3335525937 shutdown_connections 2024-03-18T16:20:13.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.783+0000 7f93c7771740 1 -- 172.21.15.39:0/3335525937 wait complete. 2024-03-18T16:20:13.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.784+0000 7f93c7771740 1 -- start start 2024-03-18T16:20:13.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.784+0000 7f93c7771740 1 Processor -- start 2024-03-18T16:20:13.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.785+0000 7f93c7771740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562d79d0ba60 0x562d79d15f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:13.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.785+0000 7f93c7771740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562d79d115c0 0x562d79d16440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:13.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.785+0000 7f93c7771740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562d79d1b470 0x562d79d1d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:13.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.785+0000 7f93c7771740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x562d79c2c110 con 0x562d79d1b470 2024-03-18T16:20:13.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.785+0000 7f93c7771740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x562d79c6c4f0 con 0x562d79d115c0 2024-03-18T16:20:13.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.785+0000 7f93c7771740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x562d79d14560 con 0x562d79d0ba60 2024-03-18T16:20:13.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.785+0000 7f93c7771740 10 client.0 _ll_register_callbacks cb 0x562d79c19b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:20:13.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.785+0000 7f93bd400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562d79d115c0 0x562d79d16440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:13.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.785+0000 7f93bd400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562d79d115c0 0x562d79d16440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:57884/0 (socket says 172.21.15.39:57884) 2024-03-18T16:20:13.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.785+0000 7f93bd400640 1 -- 172.21.15.39:0/2635155601 learned_addr learned my addr 172.21.15.39:0/2635155601 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:20:13.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.786+0000 7f93bde00640 1 --2- 172.21.15.39:0/2635155601 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562d79d0ba60 0x562d79d15f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:13.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.786+0000 7f93bd400640 1 -- 172.21.15.39:0/2635155601 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562d79d0ba60 msgr2=0x562d79d15f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:13.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.786+0000 7f93bd400640 1 --2- 172.21.15.39:0/2635155601 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562d79d0ba60 0x562d79d15f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:13.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.786+0000 7f93bd400640 1 -- 172.21.15.39:0/2635155601 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562d79d1b470 msgr2=0x562d79d1d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:13.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.786+0000 7f93bd400640 1 --2- 172.21.15.39:0/2635155601 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562d79d1b470 0x562d79d1d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:13.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.786+0000 7f93bd400640 1 -- 172.21.15.39:0/2635155601 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562d799843f0 con 0x562d79d115c0 2024-03-18T16:20:13.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.787+0000 7f93bd400640 1 --2- 172.21.15.39:0/2635155601 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562d79d115c0 0x562d79d16440 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f93ac009610 tx=0x7f93ac09a990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=76bf26b924139e86 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:13.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.787+0000 7f939f400640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:20:13.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.787+0000 7f939f400640 1 -- 172.21.15.39:0/2635155601 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93ac020070 con 0x562d79d115c0 2024-03-18T16:20:13.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.788+0000 7f939f400640 1 -- 172.21.15.39:0/2635155601 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f93ac0976a0 con 0x562d79d115c0 2024-03-18T16:20:13.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.788+0000 7f93c7771740 1 -- 172.21.15.39:0/2635155601 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f93b0004880 con 0x562d79d115c0 2024-03-18T16:20:13.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.788+0000 7f939f400640 1 -- 172.21.15.39:0/2635155601 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f93ac09f180 con 0x562d79d115c0 2024-03-18T16:20:13.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.788+0000 7f93c7771740 10 client.27692 fetch_fsmap learned FSMap version 45 2024-03-18T16:20:13.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.788+0000 7f93c7771740 1 -- 172.21.15.39:0/2635155601 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x562d79a4e7d0 con 0x562d79d115c0 2024-03-18T16:20:13.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.788+0000 7f939f400640 1 -- 172.21.15.39:0/2635155601 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f93b0004880 con 0x562d79d115c0 2024-03-18T16:20:13.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.790+0000 7f939f400640 1 -- 172.21.15.39:0/2635155601 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 45) ==== 8366+0+0 (secure 0 0 0) 0x7f93ac095ba0 con 0x562d79d115c0 2024-03-18T16:20:13.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.791+0000 7f93c7771740 10 client.27692 fetch_fsmap finished waiting for FSMap version 45 2024-03-18T16:20:13.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.791+0000 7f93c7771740 10 client.27692 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T16:20:13.792 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.791+0000 7f93c7771740 20 client.27692 populate_metadata read hostname 'smithi039' 2024-03-18T16:20:13.793 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.791+0000 7f93c7771740 1 --2- 172.21.15.39:0/2635155601 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x562d79d22140 0x562d79d24580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:13.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.793+0000 7f93c7771740 4 client.27692 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:20:13.795 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.793+0000 7f93c7771740 1 -- 172.21.15.39:0/2635155601 --> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562d79c7d990 con 0x562d79d22140 2024-03-18T16:20:13.795 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.793+0000 7f93bde00640 1 --2- 172.21.15.39:0/2635155601 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x562d79d22140 0x562d79d24580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:13.795 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.794+0000 7f93bde00640 1 --2- 172.21.15.39:0/2635155601 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x562d79d22140 0x562d79d24580 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2d4144f9fb2a8ffc server_cookie=cceb9a2a4c0170d3 in_seq=0 out_seq=0 2024-03-18T16:20:13.795 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.794+0000 7f939f400640 10 client.27692 ms_handle_connect on v2:172.21.15.196:6838/2001427268 2024-03-18T16:20:13.797 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.795+0000 7f939f400640 1 -- 172.21.15.39:0/2635155601 <== mds.1 v2:172.21.15.196:6838/2001427268 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562d79c7d990 con 0x562d79d22140 2024-03-18T16:20:13.797 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.795+0000 7f939f400640 10 client.27692 handle_command_reply: tid=0 2024-03-18T16:20:13.797 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.795+0000 7f93c7771740 1 client.27692 shutdown 2024-03-18T16:20:13.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.797+0000 7f93c7771740 1 -- 172.21.15.39:0/2635155601 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562d79d115c0 msgr2=0x562d79d16440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:13.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.797+0000 7f93c7771740 1 --2- 172.21.15.39:0/2635155601 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562d79d115c0 0x562d79d16440 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f93ac009610 tx=0x7f93ac09a990 comp rx=0 tx=0).stop 2024-03-18T16:20:13.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.797+0000 7f93c7771740 1 -- 172.21.15.39:0/2635155601 shutdown_connections 2024-03-18T16:20:13.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.797+0000 7f93c7771740 1 --2- 172.21.15.39:0/2635155601 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x562d79d22140 0x562d79d24580 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:13.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.797+0000 7f93c7771740 1 --2- 172.21.15.39:0/2635155601 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562d79d115c0 0x562d79d16440 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:13.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.797+0000 7f93c7771740 1 --2- 172.21.15.39:0/2635155601 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562d79d1b470 0x562d79d1d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:13.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.797+0000 7f93c7771740 1 --2- 172.21.15.39:0/2635155601 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562d79d0ba60 0x562d79d15f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:13.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.797+0000 7f939f400640 0 client.27692 ms_handle_reset on v2:172.21.15.196:6838/2001427268 2024-03-18T16:20:13.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.797+0000 7f93c7771740 1 -- 172.21.15.39:0/2635155601 >> 172.21.15.39:0/2635155601 conn(0x562d79d06fd0 msgr2=0x562d79d061f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:13.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.797+0000 7f93c7771740 1 -- 172.21.15.39:0/2635155601 shutdown_connections 2024-03-18T16:20:13.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.797+0000 7f93c7771740 1 -- 172.21.15.39:0/2635155601 wait complete. 2024-03-18T16:20:13.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.797+0000 7f93c7771740 20 client.27692 trim_cache size 0 max 16384 2024-03-18T16:20:13.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.833+0000 7f93c7771740 1 Processor -- start 2024-03-18T16:20:13.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.834+0000 7f93c7771740 1 -- start start 2024-03-18T16:20:13.836 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.835+0000 7f93c7771740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562d79d0ba60 0x562d79d0da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:13.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.835+0000 7f93c7771740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562d79d115c0 0x562d79d0df60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:13.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.835+0000 7f93bde00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562d79d0ba60 0x562d79d0da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:13.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.835+0000 7f93c7771740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562d79d1b470 0x562d79d05960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:13.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.835+0000 7f93c7771740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f93b815e6a0 con 0x562d79d1b470 2024-03-18T16:20:13.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.835+0000 7f93c7771740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f93b8160210 con 0x562d79d0ba60 2024-03-18T16:20:13.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.835+0000 7f93bde00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562d79d0ba60 0x562d79d0da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:57894/0 (socket says 172.21.15.39:57894) 2024-03-18T16:20:13.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.835+0000 7f93c7771740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f93b815ed70 con 0x562d79d115c0 2024-03-18T16:20:13.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.835+0000 7f93bde00640 1 -- 172.21.15.39:0/1370983987 learned_addr learned my addr 172.21.15.39:0/1370983987 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:20:13.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.836+0000 7f93bd400640 1 --2- 172.21.15.39:0/1370983987 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562d79d115c0 0x562d79d0df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:13.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.836+0000 7f93be800640 1 --2- 172.21.15.39:0/1370983987 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562d79d1b470 0x562d79d05960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:13.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.837+0000 7f93bde00640 1 -- 172.21.15.39:0/1370983987 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562d79d115c0 msgr2=0x562d79d0df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:13.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.837+0000 7f93bde00640 1 --2- 172.21.15.39:0/1370983987 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562d79d115c0 0x562d79d0df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:13.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.837+0000 7f93bde00640 1 -- 172.21.15.39:0/1370983987 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562d79d1b470 msgr2=0x562d79d05960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:13.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.837+0000 7f93bde00640 1 --2- 172.21.15.39:0/1370983987 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562d79d1b470 0x562d79d05960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:13.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.837+0000 7f93bde00640 1 -- 172.21.15.39:0/1370983987 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562d79d07f50 con 0x562d79d0ba60 2024-03-18T16:20:13.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.837+0000 7f93bde00640 1 --2- 172.21.15.39:0/1370983987 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562d79d0ba60 0x562d79d0da20 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f93b002c290 tx=0x7f93b0004a30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8b86559392a191cc server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:13.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.840+0000 7f93aa600640 1 -- 172.21.15.39:0/1370983987 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93b0042070 con 0x562d79d0ba60 2024-03-18T16:20:13.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.841+0000 7f93aa600640 1 -- 172.21.15.39:0/1370983987 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f93b0004880 con 0x562d79d0ba60 2024-03-18T16:20:13.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.841+0000 7f93aa600640 1 -- 172.21.15.39:0/1370983987 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93b0039040 con 0x562d79d0ba60 2024-03-18T16:20:13.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.842+0000 7f93c7771740 1 -- 172.21.15.39:0/1370983987 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562d79d0ba60 msgr2=0x562d79d0da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:13.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.842+0000 7f93c7771740 1 --2- 172.21.15.39:0/1370983987 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562d79d0ba60 0x562d79d0da20 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f93b002c290 tx=0x7f93b0004a30 comp rx=0 tx=0).stop 2024-03-18T16:20:13.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.842+0000 7f93c7771740 1 -- 172.21.15.39:0/1370983987 shutdown_connections 2024-03-18T16:20:13.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.842+0000 7f93c7771740 1 --2- 172.21.15.39:0/1370983987 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562d79d1b470 0x562d79d05960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:13.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.842+0000 7f93c7771740 1 --2- 172.21.15.39:0/1370983987 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562d79d115c0 0x562d79d0df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:13.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.842+0000 7f93c7771740 1 --2- 172.21.15.39:0/1370983987 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562d79d0ba60 0x562d79d0da20 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:13.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.843+0000 7f93c7771740 1 -- 172.21.15.39:0/1370983987 >> 172.21.15.39:0/1370983987 conn(0x562d79d06fd0 msgr2=0x562d79d15f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:13.847 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.846+0000 7f93c7771740 1 -- 172.21.15.39:0/1370983987 shutdown_connections 2024-03-18T16:20:13.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.847+0000 7f93c7771740 1 -- 172.21.15.39:0/1370983987 wait complete. 2024-03-18T16:20:13.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.847+0000 7f93c7771740 1 -- start start 2024-03-18T16:20:13.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.848+0000 7f93c7771740 1 Processor -- start 2024-03-18T16:20:13.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.849+0000 7f93c7771740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562d79d0ba60 0x562d79d325c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:13.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.849+0000 7f93c7771740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562d79d115c0 0x562d79d32b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:13.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.849+0000 7f93c7771740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562d79d1b470 0x562d79d85e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:13.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.849+0000 7f93bde00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562d79d0ba60 0x562d79d325c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:13.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.850+0000 7f93bde00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562d79d0ba60 0x562d79d325c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:57900/0 (socket says 172.21.15.39:57900) 2024-03-18T16:20:13.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.850+0000 7f93c7771740 1 -- 172.21.15.39:0/1525047546 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f93b815fb10 con 0x562d79d115c0 2024-03-18T16:20:13.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.850+0000 7f93c7771740 1 -- 172.21.15.39:0/1525047546 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f93b815f440 con 0x562d79d0ba60 2024-03-18T16:20:13.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.850+0000 7f93c7771740 1 -- 172.21.15.39:0/1525047546 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f93b815c730 con 0x562d79d1b470 2024-03-18T16:20:13.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.850+0000 7f93bde00640 1 -- 172.21.15.39:0/1525047546 learned_addr learned my addr 172.21.15.39:0/1525047546 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:20:13.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.850+0000 7f93c7771740 10 client.0 _ll_register_callbacks cb 0x7f93ac0bb0b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:20:13.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.850+0000 7f93be800640 1 --2- 172.21.15.39:0/1525047546 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562d79d1b470 0x562d79d85e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:13.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.850+0000 7f93bd400640 1 --2- 172.21.15.39:0/1525047546 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562d79d115c0 0x562d79d32b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:13.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.850+0000 7f93bde00640 1 -- 172.21.15.39:0/1525047546 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562d79d1b470 msgr2=0x562d79d85e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:13.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.850+0000 7f93bde00640 1 --2- 172.21.15.39:0/1525047546 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562d79d1b470 0x562d79d85e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:13.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.850+0000 7f93bde00640 1 -- 172.21.15.39:0/1525047546 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562d79d115c0 msgr2=0x562d79d32b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:13.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.850+0000 7f93bde00640 1 --2- 172.21.15.39:0/1525047546 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562d79d115c0 0x562d79d32b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:13.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.850+0000 7f93bde00640 1 -- 172.21.15.39:0/1525047546 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562d79d863b0 con 0x562d79d0ba60 2024-03-18T16:20:13.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.850+0000 7f93bd400640 1 --2- 172.21.15.39:0/1525047546 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562d79d115c0 0x562d79d32b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:20:13.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.851+0000 7f93bde00640 1 --2- 172.21.15.39:0/1525047546 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562d79d0ba60 0x562d79d325c0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f93b004d040 tx=0x7f93b002cbe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c9ec4f71caf96aeb server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:13.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.851+0000 7f939f400640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:20:13.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.851+0000 7f939f400640 1 -- 172.21.15.39:0/1525047546 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93b004f070 con 0x562d79d0ba60 2024-03-18T16:20:13.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.851+0000 7f93c7771740 1 -- 172.21.15.39:0/1525047546 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f93b0004880 con 0x562d79d0ba60 2024-03-18T16:20:13.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.851+0000 7f939f400640 1 -- 172.21.15.39:0/1525047546 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f93b0004030 con 0x562d79d0ba60 2024-03-18T16:20:13.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.852+0000 7f939f400640 1 -- 172.21.15.39:0/1525047546 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f93b005d050 con 0x562d79d0ba60 2024-03-18T16:20:13.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.852+0000 7f93c7771740 10 client.27698 fetch_fsmap learned FSMap version 45 2024-03-18T16:20:13.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.852+0000 7f93c7771740 1 -- 172.21.15.39:0/1525047546 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x562d79d87690 con 0x562d79d0ba60 2024-03-18T16:20:13.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.854+0000 7f939f400640 1 -- 172.21.15.39:0/1525047546 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f93b0004880 con 0x562d79d0ba60 2024-03-18T16:20:13.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.854+0000 7f939f400640 1 -- 172.21.15.39:0/1525047546 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 45) ==== 8366+0+0 (secure 0 0 0) 0x7f93b0004df0 con 0x562d79d0ba60 2024-03-18T16:20:13.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.854+0000 7f93c7771740 10 client.27698 fetch_fsmap finished waiting for FSMap version 45 2024-03-18T16:20:13.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.854+0000 7f93c7771740 10 client.27698 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T16:20:13.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.854+0000 7f93c7771740 20 client.27698 populate_metadata read hostname 'smithi039' 2024-03-18T16:20:13.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.854+0000 7f93c7771740 1 --2- 172.21.15.39:0/1525047546 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x562d79d87ae0 0x562d79d89ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:13.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.854+0000 7f93c7771740 4 client.27698 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:20:13.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.854+0000 7f93c7771740 1 -- 172.21.15.39:0/1525047546 --> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562d79c77ce0 con 0x562d79d87ae0 2024-03-18T16:20:13.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.855+0000 7f93bd400640 1 --2- 172.21.15.39:0/1525047546 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x562d79d87ae0 0x562d79d89ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:13.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.856+0000 7f93bd400640 1 --2- 172.21.15.39:0/1525047546 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x562d79d87ae0 0x562d79d89ed0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d0b719db872c3287 server_cookie=e707a6bab28b6086 in_seq=0 out_seq=0 2024-03-18T16:20:13.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.856+0000 7f939f400640 10 client.27698 ms_handle_connect on v2:172.21.15.196:6838/2001427268 2024-03-18T16:20:13.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.856+0000 7f939f400640 1 -- 172.21.15.39:0/1525047546 <== mds.1 v2:172.21.15.196:6838/2001427268 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562d79c77ce0 con 0x562d79d87ae0 2024-03-18T16:20:13.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.856+0000 7f939f400640 10 client.27698 handle_command_reply: tid=0 2024-03-18T16:20:13.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.856+0000 7f93c7771740 1 client.27698 shutdown 2024-03-18T16:20:13.859 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.857+0000 7f93c7771740 1 -- 172.21.15.39:0/1525047546 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562d79d0ba60 msgr2=0x562d79d325c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:13.859 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.857+0000 7f93c7771740 1 --2- 172.21.15.39:0/1525047546 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562d79d0ba60 0x562d79d325c0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f93b004d040 tx=0x7f93b002cbe0 comp rx=0 tx=0).stop 2024-03-18T16:20:13.859 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.857+0000 7f93c7771740 1 -- 172.21.15.39:0/1525047546 shutdown_connections 2024-03-18T16:20:13.859 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.857+0000 7f93c7771740 1 --2- 172.21.15.39:0/1525047546 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x562d79d87ae0 0x562d79d89ed0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:13.860 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.857+0000 7f93c7771740 1 --2- 172.21.15.39:0/1525047546 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562d79d1b470 0x562d79d85e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:13.860 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.857+0000 7f93c7771740 1 --2- 172.21.15.39:0/1525047546 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562d79d115c0 0x562d79d32b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:13.860 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.857+0000 7f93c7771740 1 --2- 172.21.15.39:0/1525047546 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562d79d0ba60 0x562d79d325c0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:13.860 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.857+0000 7f93c7771740 1 -- 172.21.15.39:0/1525047546 >> 172.21.15.39:0/1525047546 conn(0x562d79d06fd0 msgr2=0x562d79d15f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:13.860 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.857+0000 7f939f400640 0 client.27698 ms_handle_reset on v2:172.21.15.196:6838/2001427268 2024-03-18T16:20:13.860 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.857+0000 7f93c7771740 1 -- 172.21.15.39:0/1525047546 shutdown_connections 2024-03-18T16:20:13.860 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.857+0000 7f93c7771740 1 -- 172.21.15.39:0/1525047546 wait complete. 2024-03-18T16:20:13.861 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:20:13.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.858+0000 7f93c7771740 20 client.27698 trim_cache size 0 max 16384 2024-03-18T16:20:13.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.860+0000 7f93bf200640 1 -- 172.21.15.39:0/4022746686 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f93a40a2880 msgr2=0x7f93a40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:13.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.860+0000 7f93bf200640 1 --2- 172.21.15.39:0/4022746686 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f93a40a2880 0x7f93a40a4d40 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f93c00507b0 tx=0x7f93c0067580 comp rx=0 tx=0).stop 2024-03-18T16:20:13.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.860+0000 7f93bf200640 1 -- 172.21.15.39:0/4022746686 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f93b800a790 msgr2=0x7f93b80a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:13.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.860+0000 7f93bf200640 1 --2- 172.21.15.39:0/4022746686 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f93b800a790 0x7f93b80a40e0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f93ac00d6d0 tx=0x7f93ac00db90 comp rx=0 tx=0).stop 2024-03-18T16:20:13.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.861+0000 7f93bf200640 1 -- 172.21.15.39:0/4022746686 shutdown_connections 2024-03-18T16:20:13.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.861+0000 7f93bf200640 1 --2- 172.21.15.39:0/4022746686 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f93b800a790 0x7f93b80a40e0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:13.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.861+0000 7f93bf200640 1 --2- 172.21.15.39:0/4022746686 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f93a40a2880 0x7f93a40a4d40 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:13.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.861+0000 7f93bf200640 1 --2- 172.21.15.39:0/4022746686 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f93b800b130 0x7f93b80a5120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:13.863 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.861+0000 7f93bf200640 1 --2- 172.21.15.39:0/4022746686 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f93b8009550 0x7f93b80a3ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:13.863 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.861+0000 7f93bf200640 1 -- 172.21.15.39:0/4022746686 >> 172.21.15.39:0/4022746686 conn(0x7f93b80934e0 msgr2=0x7f93b8094fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:13.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.862+0000 7f93bf200640 1 -- 172.21.15.39:0/4022746686 shutdown_connections 2024-03-18T16:20:13.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:13.862+0000 7f93bf200640 1 -- 172.21.15.39:0/4022746686 wait complete. 2024-03-18T16:20:13.902 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-18T16:20:14.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.247+0000 7fccfce00640 1 -- 172.21.15.39:0/2455494193 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fccf810b150 msgr2=0x7fccf810d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:14.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.247+0000 7fccfce00640 1 --2- 172.21.15.39:0/2455494193 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fccf810b150 0x7fccf810d5c0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fcce4009860 tx=0x7fcce402bb70 comp rx=0 tx=0).stop 2024-03-18T16:20:14.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.250+0000 7fccfce00640 1 -- 172.21.15.39:0/2455494193 shutdown_connections 2024-03-18T16:20:14.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.250+0000 7fccfce00640 1 --2- 172.21.15.39:0/2455494193 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fccf810db00 0x7fccf810df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:14.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.250+0000 7fccfce00640 1 --2- 172.21.15.39:0/2455494193 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fccf810b150 0x7fccf810d5c0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:14.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.250+0000 7fccfce00640 1 --2- 172.21.15.39:0/2455494193 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fccf81087e0 0x7fccf810ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:14.252 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.250+0000 7fccfce00640 1 -- 172.21.15.39:0/2455494193 >> 172.21.15.39:0/2455494193 conn(0x7fccf806e860 msgr2=0x7fccf8070ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:14.253 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.252+0000 7fccfce00640 1 -- 172.21.15.39:0/2455494193 shutdown_connections 2024-03-18T16:20:14.254 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.253+0000 7fccfce00640 1 -- 172.21.15.39:0/2455494193 wait complete. 2024-03-18T16:20:14.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.254+0000 7fccfce00640 1 Processor -- start 2024-03-18T16:20:14.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.254+0000 7fccfce00640 1 -- start start 2024-03-18T16:20:14.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.254+0000 7fccfce00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fccf81087e0 0x7fccf8116ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:14.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.254+0000 7fccfce00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fccf810b150 0x7fccf8117420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:14.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.254+0000 7fccfce00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fccf810db00 0x7fccf81b6930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:14.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.255+0000 7fccfce00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fccf811c220 con 0x7fccf810db00 2024-03-18T16:20:14.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.255+0000 7fccfce00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fccf811c0a0 con 0x7fccf81087e0 2024-03-18T16:20:14.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.255+0000 7fccfce00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fccf811c3a0 con 0x7fccf810b150 2024-03-18T16:20:14.257 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.255+0000 7fccf5400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fccf810b150 0x7fccf8117420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:14.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.255+0000 7fccf5400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fccf810b150 0x7fccf8117420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:38120/0 (socket says 172.21.15.39:38120) 2024-03-18T16:20:14.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.255+0000 7fccf5400640 1 -- 172.21.15.39:0/678961693 learned_addr learned my addr 172.21.15.39:0/678961693 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:20:14.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.255+0000 7fccf5e00640 1 --2- 172.21.15.39:0/678961693 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fccf81087e0 0x7fccf8116ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:14.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.256+0000 7fccf5400640 1 -- 172.21.15.39:0/678961693 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fccf81087e0 msgr2=0x7fccf8116ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:14.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.256+0000 7fccf5400640 1 --2- 172.21.15.39:0/678961693 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fccf81087e0 0x7fccf8116ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:14.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.256+0000 7fccf5400640 1 -- 172.21.15.39:0/678961693 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fccf810db00 msgr2=0x7fccf81b6930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:14.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.256+0000 7fccf5400640 1 --2- 172.21.15.39:0/678961693 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fccf810db00 0x7fccf81b6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:14.258 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.256+0000 7fccf5400640 1 -- 172.21.15.39:0/678961693 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccf81b6e70 con 0x7fccf810b150 2024-03-18T16:20:14.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.260+0000 7fccf5400640 1 --2- 172.21.15.39:0/678961693 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fccf810b150 0x7fccf8117420 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fcce40349e0 tx=0x7fcce4032450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:14.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.261+0000 7fcceea00640 1 -- 172.21.15.39:0/678961693 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcce4041030 con 0x7fccf810b150 2024-03-18T16:20:14.263 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.261+0000 7fcceea00640 1 -- 172.21.15.39:0/678961693 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcce402c510 con 0x7fccf810b150 2024-03-18T16:20:14.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.261+0000 7fcceea00640 1 -- 172.21.15.39:0/678961693 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcce4039040 con 0x7fccf810b150 2024-03-18T16:20:14.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.261+0000 7fccfce00640 1 -- 172.21.15.39:0/678961693 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fccf81b7040 con 0x7fccf810b150 2024-03-18T16:20:14.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.261+0000 7fccfce00640 1 -- 172.21.15.39:0/678961693 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fccf81b7450 con 0x7fccf810b150 2024-03-18T16:20:14.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.269+0000 7fcceea00640 1 -- 172.21.15.39:0/678961693 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fcce4044070 con 0x7fccf810b150 2024-03-18T16:20:14.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.271+0000 7fcceea00640 1 --2- 172.21.15.39:0/678961693 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fccdc0a2880 0x7fccdc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:14.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.272+0000 7fcceea00640 1 -- 172.21.15.39:0/678961693 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fcce40f0360 con 0x7fccf810b150 2024-03-18T16:20:14.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.272+0000 7fccffc1e740 1 Processor -- start 2024-03-18T16:20:14.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.273+0000 7fccffc1e740 1 -- start start 2024-03-18T16:20:14.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.275+0000 7fccf5e00640 1 --2- 172.21.15.39:0/678961693 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fccdc0a2880 0x7fccdc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:14.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.275+0000 7fccffc1e740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fec2bf78c0 0x55fec2bf7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:14.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.275+0000 7fccffc1e740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fec2bf82b0 0x55fec2bfcdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:14.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.275+0000 7fccffc1e740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fec2bfd420 0x55fec2bff8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:14.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.276+0000 7fccffc1e740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55fec2bbf3b0 con 0x55fec2bf82b0 2024-03-18T16:20:14.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.276+0000 7fccf5400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fec2bf82b0 0x55fec2bfcdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:14.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.276+0000 7fccf6800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fec2bfd420 0x55fec2bff8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:14.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.276+0000 7fccf5400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fec2bf82b0 0x55fec2bfcdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:55162/0 (socket says 172.21.15.39:55162) 2024-03-18T16:20:14.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.276+0000 7fccf5400640 1 -- 172.21.15.39:0/2245969561 learned_addr learned my addr 172.21.15.39:0/2245969561 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:20:14.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.276+0000 7fccffc1e740 1 -- 172.21.15.39:0/2245969561 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55fec2b5c450 con 0x55fec2bfd420 2024-03-18T16:20:14.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.276+0000 7fccffc1e740 1 -- 172.21.15.39:0/2245969561 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55fec2b04cb0 con 0x55fec2bf78c0 2024-03-18T16:20:14.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.276+0000 7fccf5e00640 1 --2- 172.21.15.39:0/2245969561 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fec2bf78c0 0x55fec2bf7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:14.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.277+0000 7fccf6800640 1 -- 172.21.15.39:0/2245969561 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fec2bf78c0 msgr2=0x55fec2bf7ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:14.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.277+0000 7fccf6800640 1 --2- 172.21.15.39:0/2245969561 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fec2bf78c0 0x55fec2bf7ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:14.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.277+0000 7fccf6800640 1 -- 172.21.15.39:0/2245969561 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fec2bf82b0 msgr2=0x55fec2bfcdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:14.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.277+0000 7fccf6800640 1 --2- 172.21.15.39:0/2245969561 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fec2bf82b0 0x55fec2bfcdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:14.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.277+0000 7fccf6800640 1 -- 172.21.15.39:0/2245969561 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fec2b261f0 con 0x55fec2bfd420 2024-03-18T16:20:14.281 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.278+0000 7fccf6800640 1 --2- 172.21.15.39:0/2245969561 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fec2bfd420 0x55fec2bff8b0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fcce8009a70 tx=0x7fcce8009f30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3040f6eab1221414 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:14.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.280+0000 7fcce3400640 1 -- 172.21.15.39:0/2245969561 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcce800ecf0 con 0x55fec2bfd420 2024-03-18T16:20:14.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.280+0000 7fcce3400640 1 -- 172.21.15.39:0/2245969561 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcce800ee90 con 0x55fec2bfd420 2024-03-18T16:20:14.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.280+0000 7fcce3400640 1 -- 172.21.15.39:0/2245969561 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcce8010690 con 0x55fec2bfd420 2024-03-18T16:20:14.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.282+0000 7fccffc1e740 1 -- 172.21.15.39:0/2245969561 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fec2bfd420 msgr2=0x55fec2bff8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:14.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.282+0000 7fccffc1e740 1 --2- 172.21.15.39:0/2245969561 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fec2bfd420 0x55fec2bff8b0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fcce8009a70 tx=0x7fcce8009f30 comp rx=0 tx=0).stop 2024-03-18T16:20:14.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.282+0000 7fccffc1e740 1 -- 172.21.15.39:0/2245969561 shutdown_connections 2024-03-18T16:20:14.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.282+0000 7fccffc1e740 1 --2- 172.21.15.39:0/2245969561 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fec2bfd420 0x55fec2bff8b0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:14.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.282+0000 7fccffc1e740 1 --2- 172.21.15.39:0/2245969561 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fec2bf82b0 0x55fec2bfcdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:14.284 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.282+0000 7fccffc1e740 1 --2- 172.21.15.39:0/2245969561 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fec2bf78c0 0x55fec2bf7ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:14.285 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.283+0000 7fccffc1e740 1 -- 172.21.15.39:0/2245969561 >> 172.21.15.39:0/2245969561 conn(0x55fec2bf2e30 msgr2=0x55fec2bf3230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:14.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.285+0000 7fccf5e00640 1 --2- 172.21.15.39:0/678961693 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fccdc0a2880 0x7fccdc0a4d40 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fccf0005fd0 tx=0x7fccf0009f70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:14.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.285+0000 7fccffc1e740 1 -- 172.21.15.39:0/2245969561 shutdown_connections 2024-03-18T16:20:14.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.285+0000 7fccffc1e740 1 -- 172.21.15.39:0/2245969561 wait complete. 2024-03-18T16:20:14.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.286+0000 7fccffc1e740 1 -- start start 2024-03-18T16:20:14.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.286+0000 7fccffc1e740 1 Processor -- start 2024-03-18T16:20:14.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.289+0000 7fccffc1e740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fec2bf78c0 0x55fec2c01b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:14.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.289+0000 7fccffc1e740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fec2bf82b0 0x55fec2c020d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:14.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.290+0000 7fccffc1e740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fec2bfd420 0x55fec2c09110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:14.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.290+0000 7fccffc1e740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55fec2b18110 con 0x55fec2bfd420 2024-03-18T16:20:14.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.290+0000 7fccffc1e740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55fec2b584f0 con 0x55fec2bf78c0 2024-03-18T16:20:14.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.290+0000 7fccffc1e740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55fec2c003d0 con 0x55fec2bf82b0 2024-03-18T16:20:14.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.290+0000 7fccffc1e740 10 client.0 _ll_register_callbacks cb 0x55fec2b05b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:20:14.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.290+0000 7fccf5400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fec2bf82b0 0x55fec2c020d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:14.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.290+0000 7fccf5400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fec2bf82b0 0x55fec2c020d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:38140/0 (socket says 172.21.15.39:38140) 2024-03-18T16:20:14.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.290+0000 7fccf5400640 1 -- 172.21.15.39:0/956838199 learned_addr learned my addr 172.21.15.39:0/956838199 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:20:14.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.291+0000 7fccf5e00640 1 --2- 172.21.15.39:0/956838199 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fec2bf78c0 0x55fec2c01b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:14.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.292+0000 7fccf6800640 1 --2- 172.21.15.39:0/956838199 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fec2bfd420 0x55fec2c09110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:14.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.292+0000 7fccf5400640 1 -- 172.21.15.39:0/956838199 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fec2bf78c0 msgr2=0x55fec2c01b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:14.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.292+0000 7fccf5400640 1 --2- 172.21.15.39:0/956838199 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fec2bf78c0 0x55fec2c01b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:14.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.292+0000 7fccf5400640 1 -- 172.21.15.39:0/956838199 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fec2bfd420 msgr2=0x55fec2c09110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:14.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.292+0000 7fccf5400640 1 --2- 172.21.15.39:0/956838199 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fec2bfd420 0x55fec2c09110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:14.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.292+0000 7fccf5400640 1 -- 172.21.15.39:0/956838199 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fec28703f0 con 0x55fec2bf82b0 2024-03-18T16:20:14.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.293+0000 7fccf5400640 1 --2- 172.21.15.39:0/956838199 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fec2bf82b0 0x55fec2c020d0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fcce4005820 tx=0x7fcce402bb70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a90c6c46b9021a78 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:14.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.293+0000 7fccf6800640 1 --2- 172.21.15.39:0/956838199 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fec2bfd420 0x55fec2c09110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:20:14.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.294+0000 7fccd7e00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:20:14.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.294+0000 7fccd7e00640 1 -- 172.21.15.39:0/956838199 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcce40d5060 con 0x55fec2bf82b0 2024-03-18T16:20:14.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.294+0000 7fccd7e00640 1 -- 172.21.15.39:0/956838199 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fcce402ca30 con 0x55fec2bf82b0 2024-03-18T16:20:14.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.294+0000 7fccffc1e740 1 -- 172.21.15.39:0/956838199 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcce800ee90 con 0x55fec2bf82b0 2024-03-18T16:20:14.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.294+0000 7fccd7e00640 1 -- 172.21.15.39:0/956838199 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7fcce40da050 con 0x55fec2bf82b0 2024-03-18T16:20:14.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.294+0000 7fccffc1e740 10 client.27706 fetch_fsmap learned FSMap version 46 2024-03-18T16:20:14.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.294+0000 7fccffc1e740 1 -- 172.21.15.39:0/956838199 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x55fec293a7d0 con 0x55fec2bf82b0 2024-03-18T16:20:14.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.295+0000 7fccd7e00640 1 -- 172.21.15.39:0/956838199 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcce800ee90 con 0x55fec2bf82b0 2024-03-18T16:20:14.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.296+0000 7fccd7e00640 1 -- 172.21.15.39:0/956838199 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 46) ==== 8370+0+0 (secure 0 0 0) 0x7fcce404e060 con 0x55fec2bf82b0 2024-03-18T16:20:14.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.296+0000 7fccffc1e740 10 client.27706 fetch_fsmap finished waiting for FSMap version 46 2024-03-18T16:20:14.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.296+0000 7fccffc1e740 10 client.27706 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:20:14.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.296+0000 7fccffc1e740 20 client.27706 populate_metadata read hostname 'smithi039' 2024-03-18T16:20:14.298 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.296+0000 7fccffc1e740 1 --2- 172.21.15.39:0/956838199 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55fec2c0d9e0 0x55fec2c0fe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:14.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.296+0000 7fccffc1e740 4 client.27706 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:20:14.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.296+0000 7fccffc1e740 1 -- 172.21.15.39:0/956838199 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fec2b69990 con 0x55fec2c0d9e0 2024-03-18T16:20:14.299 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.297+0000 7fccf5e00640 1 --2- 172.21.15.39:0/956838199 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55fec2c0d9e0 0x55fec2c0fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:14.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.298+0000 7fccf5e00640 1 --2- 172.21.15.39:0/956838199 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55fec2c0d9e0 0x55fec2c0fe40 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2a86a74d3289722d server_cookie=d4bb90b179461282 in_seq=0 out_seq=0 2024-03-18T16:20:14.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.298+0000 7fccd7e00640 10 client.27706 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:20:14.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.303+0000 7fccd7e00640 1 -- 172.21.15.39:0/956838199 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55fec2b69990 con 0x55fec2c0d9e0 2024-03-18T16:20:14.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.303+0000 7fccd7e00640 10 client.27706 handle_command_reply: tid=0 2024-03-18T16:20:14.305 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.303+0000 7fccffc1e740 1 client.27706 shutdown 2024-03-18T16:20:14.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.307+0000 7fccffc1e740 1 -- 172.21.15.39:0/956838199 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fec2bf82b0 msgr2=0x55fec2c020d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:14.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.307+0000 7fccffc1e740 1 --2- 172.21.15.39:0/956838199 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fec2bf82b0 0x55fec2c020d0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fcce4005820 tx=0x7fcce402bb70 comp rx=0 tx=0).stop 2024-03-18T16:20:14.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.308+0000 7fccffc1e740 1 -- 172.21.15.39:0/956838199 shutdown_connections 2024-03-18T16:20:14.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.308+0000 7fccffc1e740 1 --2- 172.21.15.39:0/956838199 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55fec2c0d9e0 0x55fec2c0fe40 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:14.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.308+0000 7fccffc1e740 1 --2- 172.21.15.39:0/956838199 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fec2bfd420 0x55fec2c09110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:14.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.308+0000 7fccffc1e740 1 --2- 172.21.15.39:0/956838199 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fec2bf82b0 0x55fec2c020d0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:14.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.308+0000 7fccffc1e740 1 --2- 172.21.15.39:0/956838199 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fec2bf78c0 0x55fec2c01b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:14.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.308+0000 7fccd7e00640 0 client.27706 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:20:14.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.308+0000 7fccffc1e740 1 -- 172.21.15.39:0/956838199 >> 172.21.15.39:0/956838199 conn(0x55fec2bf2e30 msgr2=0x55fec2bf21f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:14.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.310+0000 7fccffc1e740 1 -- 172.21.15.39:0/956838199 shutdown_connections 2024-03-18T16:20:14.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.310+0000 7fccffc1e740 1 -- 172.21.15.39:0/956838199 wait complete. 2024-03-18T16:20:14.312 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.310+0000 7fccffc1e740 20 client.27706 trim_cache size 0 max 16384 2024-03-18T16:20:14.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:14 smithi039 ceph-mon[31721]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T16:20:14.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:14 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:20:14.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.346+0000 7fccffc1e740 1 Processor -- start 2024-03-18T16:20:14.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.347+0000 7fccffc1e740 1 -- start start 2024-03-18T16:20:14.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.347+0000 7fccffc1e740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fec2bf78c0 0x55fec2bf1960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:14.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.347+0000 7fccffc1e740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fec2bf82b0 0x55fec2c090d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:14.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.347+0000 7fccffc1e740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fec2bfd420 0x55fec2bf1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:14.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.347+0000 7fccffc1e740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fccf81c1b40 con 0x55fec2bfd420 2024-03-18T16:20:14.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.347+0000 7fccffc1e740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fccf81c36b0 con 0x55fec2bf82b0 2024-03-18T16:20:14.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.347+0000 7fccffc1e740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fccf81c2210 con 0x55fec2bf78c0 2024-03-18T16:20:14.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.348+0000 7fccf6800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fec2bfd420 0x55fec2bf1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:14.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.349+0000 7fccf6800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fec2bfd420 0x55fec2bf1ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:55178/0 (socket says 172.21.15.39:55178) 2024-03-18T16:20:14.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.349+0000 7fccf6800640 1 -- 172.21.15.39:0/665355906 learned_addr learned my addr 172.21.15.39:0/665355906 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:20:14.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.349+0000 7fccf6800640 1 -- 172.21.15.39:0/665355906 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fec2bf78c0 msgr2=0x55fec2bf1960 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-18T16:20:14.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.349+0000 7fccf6800640 1 --2- 172.21.15.39:0/665355906 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fec2bf78c0 0x55fec2bf1960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:14.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.349+0000 7fccf6800640 1 -- 172.21.15.39:0/665355906 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fec2bf82b0 msgr2=0x55fec2c090d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:20:14.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.349+0000 7fccf6800640 1 --2- 172.21.15.39:0/665355906 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fec2bf82b0 0x55fec2c090d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:14.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.349+0000 7fccf6800640 1 -- 172.21.15.39:0/665355906 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fec2beec10 con 0x55fec2bfd420 2024-03-18T16:20:14.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.351+0000 7fccf6800640 1 --2- 172.21.15.39:0/665355906 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fec2bfd420 0x55fec2bf1ea0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fcce8000c00 tx=0x7fcce8005d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b1b3e8046de3d4a server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:14.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.353+0000 7fcce2a00640 1 -- 172.21.15.39:0/665355906 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcce8027020 con 0x55fec2bfd420 2024-03-18T16:20:14.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.353+0000 7fcce2a00640 1 -- 172.21.15.39:0/665355906 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcce80096c0 con 0x55fec2bfd420 2024-03-18T16:20:14.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.353+0000 7fcce2a00640 1 -- 172.21.15.39:0/665355906 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcce801d040 con 0x55fec2bfd420 2024-03-18T16:20:14.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.354+0000 7fccffc1e740 1 -- 172.21.15.39:0/665355906 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fec2bfd420 msgr2=0x55fec2bf1ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:14.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.354+0000 7fccffc1e740 1 --2- 172.21.15.39:0/665355906 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fec2bfd420 0x55fec2bf1ea0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fcce8000c00 tx=0x7fcce8005d70 comp rx=0 tx=0).stop 2024-03-18T16:20:14.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.356+0000 7fccffc1e740 1 -- 172.21.15.39:0/665355906 shutdown_connections 2024-03-18T16:20:14.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.356+0000 7fccffc1e740 1 --2- 172.21.15.39:0/665355906 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fec2bf82b0 0x55fec2c090d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:14.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.356+0000 7fccffc1e740 1 --2- 172.21.15.39:0/665355906 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fec2bfd420 0x55fec2bf1ea0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:14.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.356+0000 7fccffc1e740 1 --2- 172.21.15.39:0/665355906 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fec2bf78c0 0x55fec2bf1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:14.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.356+0000 7fccffc1e740 1 -- 172.21.15.39:0/665355906 >> 172.21.15.39:0/665355906 conn(0x55fec2bf2e30 msgr2=0x55fec2bf9710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:14.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.356+0000 7fccffc1e740 1 -- 172.21.15.39:0/665355906 shutdown_connections 2024-03-18T16:20:14.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.356+0000 7fccffc1e740 1 -- 172.21.15.39:0/665355906 wait complete. 2024-03-18T16:20:14.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.357+0000 7fccffc1e740 1 -- start start 2024-03-18T16:20:14.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.357+0000 7fccffc1e740 1 Processor -- start 2024-03-18T16:20:14.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.358+0000 7fccffc1e740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fec2bf78c0 0x55fec2c06a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:14.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.358+0000 7fccffc1e740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fec2bf82b0 0x55fec2c06fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:14.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.358+0000 7fccffc1e740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fec2bfd420 0x55fec2c07fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:14.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.358+0000 7fccffc1e740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fccf81c2fb0 con 0x55fec2bf82b0 2024-03-18T16:20:14.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.358+0000 7fccffc1e740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fccf81c28e0 con 0x55fec2bfd420 2024-03-18T16:20:14.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.358+0000 7fccffc1e740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fccf81bfb20 con 0x55fec2bf78c0 2024-03-18T16:20:14.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.358+0000 7fccffc1e740 10 client.0 _ll_register_callbacks cb 0x7fcce40097b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:20:14.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.358+0000 7fccf6800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fec2bfd420 0x55fec2c07fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:14.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.358+0000 7fccf6800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fec2bfd420 0x55fec2c07fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:57944/0 (socket says 172.21.15.39:57944) 2024-03-18T16:20:14.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.358+0000 7fccf5400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fec2bf82b0 0x55fec2c06fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:14.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.359+0000 7fccf6800640 1 -- 172.21.15.39:0/2611957727 learned_addr learned my addr 172.21.15.39:0/2611957727 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:20:14.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.359+0000 7fccf5e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fec2bf78c0 0x55fec2c06a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:14.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.359+0000 7fccf6800640 1 -- 172.21.15.39:0/2611957727 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fec2bf78c0 msgr2=0x55fec2c06a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:14.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.359+0000 7fccf6800640 1 --2- 172.21.15.39:0/2611957727 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fec2bf78c0 0x55fec2c06a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:14.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.359+0000 7fccf6800640 1 -- 172.21.15.39:0/2611957727 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fec2bf82b0 msgr2=0x55fec2c06fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:14.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.359+0000 7fccf6800640 1 --2- 172.21.15.39:0/2611957727 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fec2bf82b0 0x55fec2c06fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:14.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.359+0000 7fccf6800640 1 -- 172.21.15.39:0/2611957727 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fec2c00510 con 0x55fec2bfd420 2024-03-18T16:20:14.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.360+0000 7fccf6800640 1 --2- 172.21.15.39:0/2611957727 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fec2bfd420 0x55fec2c07fc0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fcce800eff0 tx=0x7fcce8003e40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6993832d80c5e831 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:14.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.360+0000 7fccd7e00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:20:14.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.360+0000 7fccd7e00640 1 -- 172.21.15.39:0/2611957727 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcce80037f0 con 0x55fec2bfd420 2024-03-18T16:20:14.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.360+0000 7fccd7e00640 1 -- 172.21.15.39:0/2611957727 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fcce8003990 con 0x55fec2bfd420 2024-03-18T16:20:14.362 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.360+0000 7fccffc1e740 1 -- 172.21.15.39:0/2611957727 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcce80096c0 con 0x55fec2bfd420 2024-03-18T16:20:14.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.362+0000 7fccd7e00640 1 -- 172.21.15.39:0/2611957727 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=46) ==== 24+0+0 (secure 0 0 0) 0x7fcce803c050 con 0x55fec2bfd420 2024-03-18T16:20:14.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.362+0000 7fccffc1e740 10 client.27713 fetch_fsmap learned FSMap version 46 2024-03-18T16:20:14.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.362+0000 7fccffc1e740 1 -- 172.21.15.39:0/2611957727 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=46,monmap=4+}) -- 0x55fec2c73e30 con 0x55fec2bfd420 2024-03-18T16:20:14.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.362+0000 7fccd7e00640 1 -- 172.21.15.39:0/2611957727 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcce8004570 con 0x55fec2bfd420 2024-03-18T16:20:14.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.364+0000 7fccd7e00640 1 -- 172.21.15.39:0/2611957727 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 46) ==== 8370+0+0 (secure 0 0 0) 0x7fcce8004c60 con 0x55fec2bfd420 2024-03-18T16:20:14.367 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.365+0000 7fccffc1e740 10 client.27713 fetch_fsmap finished waiting for FSMap version 46 2024-03-18T16:20:14.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.365+0000 7fccffc1e740 10 client.27713 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:20:14.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.365+0000 7fccffc1e740 20 client.27713 populate_metadata read hostname 'smithi039' 2024-03-18T16:20:14.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.365+0000 7fccffc1e740 1 --2- 172.21.15.39:0/2611957727 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55fec2c74530 0x55fec2c76950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:14.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.365+0000 7fccffc1e740 4 client.27713 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:20:14.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.365+0000 7fccffc1e740 1 -- 172.21.15.39:0/2611957727 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7fcce80096c0 con 0x55fec2c74530 2024-03-18T16:20:14.368 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.366+0000 7fccf5e00640 1 --2- 172.21.15.39:0/2611957727 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55fec2c74530 0x55fec2c76950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:14.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.368+0000 7fccf5e00640 1 --2- 172.21.15.39:0/2611957727 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55fec2c74530 0x55fec2c76950 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3d703c87316b97fd server_cookie=dad27bb9f62b62b6 in_seq=0 out_seq=0 2024-03-18T16:20:14.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.368+0000 7fccd7e00640 10 client.27713 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:20:14.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.369+0000 7fccd7e00640 1 -- 172.21.15.39:0/2611957727 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7fcce80096c0 con 0x55fec2c74530 2024-03-18T16:20:14.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.369+0000 7fccd7e00640 10 client.27713 handle_command_reply: tid=0 2024-03-18T16:20:14.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.369+0000 7fccffc1e740 1 client.27713 shutdown 2024-03-18T16:20:14.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.369+0000 7fccffc1e740 1 -- 172.21.15.39:0/2611957727 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fec2bfd420 msgr2=0x55fec2c07fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:14.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.369+0000 7fccffc1e740 1 --2- 172.21.15.39:0/2611957727 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fec2bfd420 0x55fec2c07fc0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fcce800eff0 tx=0x7fcce8003e40 comp rx=0 tx=0).stop 2024-03-18T16:20:14.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.369+0000 7fccffc1e740 1 -- 172.21.15.39:0/2611957727 shutdown_connections 2024-03-18T16:20:14.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.369+0000 7fccffc1e740 1 --2- 172.21.15.39:0/2611957727 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55fec2c74530 0x55fec2c76950 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:14.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.369+0000 7fccffc1e740 1 --2- 172.21.15.39:0/2611957727 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55fec2bfd420 0x55fec2c07fc0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:14.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.369+0000 7fccffc1e740 1 --2- 172.21.15.39:0/2611957727 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55fec2bf82b0 0x55fec2c06fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:14.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.369+0000 7fccffc1e740 1 --2- 172.21.15.39:0/2611957727 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55fec2bf78c0 0x55fec2c06a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:14.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.369+0000 7fccffc1e740 1 -- 172.21.15.39:0/2611957727 >> 172.21.15.39:0/2611957727 conn(0x55fec2bf2e30 msgr2=0x55fec2bf1960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:14.371 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.369+0000 7fccd7e00640 0 client.27713 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:20:14.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.369+0000 7fccffc1e740 1 -- 172.21.15.39:0/2611957727 shutdown_connections 2024-03-18T16:20:14.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.371+0000 7fccffc1e740 1 -- 172.21.15.39:0/2611957727 wait complete. 2024-03-18T16:20:14.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.371+0000 7fccffc1e740 20 client.27713 trim_cache size 0 max 16384 2024-03-18T16:20:14.376 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:20:14.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.375+0000 7fccfce00640 1 -- 172.21.15.39:0/678961693 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fccdc0a2880 msgr2=0x7fccdc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:14.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.375+0000 7fccfce00640 1 --2- 172.21.15.39:0/678961693 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fccdc0a2880 0x7fccdc0a4d40 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fccf0005fd0 tx=0x7fccf0009f70 comp rx=0 tx=0).stop 2024-03-18T16:20:14.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.375+0000 7fccfce00640 1 -- 172.21.15.39:0/678961693 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fccf810b150 msgr2=0x7fccf8117420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:14.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.375+0000 7fccfce00640 1 --2- 172.21.15.39:0/678961693 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fccf810b150 0x7fccf8117420 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fcce40349e0 tx=0x7fcce4032450 comp rx=0 tx=0).stop 2024-03-18T16:20:14.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.376+0000 7fccfce00640 1 -- 172.21.15.39:0/678961693 shutdown_connections 2024-03-18T16:20:14.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.376+0000 7fccfce00640 1 --2- 172.21.15.39:0/678961693 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fccdc0a2880 0x7fccdc0a4d40 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:14.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.376+0000 7fccfce00640 1 --2- 172.21.15.39:0/678961693 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fccf810db00 0x7fccf81b6930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:14.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.376+0000 7fccfce00640 1 --2- 172.21.15.39:0/678961693 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fccf810b150 0x7fccf8117420 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:14.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.376+0000 7fccfce00640 1 --2- 172.21.15.39:0/678961693 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fccf81087e0 0x7fccf8116ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:14.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.376+0000 7fccfce00640 1 -- 172.21.15.39:0/678961693 >> 172.21.15.39:0/678961693 conn(0x7fccf806e860 msgr2=0x7fccf810b920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:14.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.376+0000 7fccfce00640 1 -- 172.21.15.39:0/678961693 shutdown_connections 2024-03-18T16:20:14.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:14.376+0000 7fccfce00640 1 -- 172.21.15.39:0/678961693 wait complete. 2024-03-18T16:20:14.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:14 smithi196 ceph-mon[36307]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T16:20:14.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:14 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:20:14.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:14 smithi194 ceph-mon[36200]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T16:20:14.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:14 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:20:15.413 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-18T16:20:15.414 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T16:20:15.414 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-18T16:20:15.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:15 smithi196 ceph-mon[36307]: pgmap v727: 129 pgs: 129 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 587 KiB/s wr, 150 op/s 2024-03-18T16:20:15.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:15 smithi196 ceph-mon[36307]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T16:20:15.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:15 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:20:15.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:15 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:20:15.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:15 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:20:15.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:15 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:20:15.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:15 smithi194 ceph-mon[36200]: pgmap v727: 129 pgs: 129 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 587 KiB/s wr, 150 op/s 2024-03-18T16:20:15.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:15 smithi194 ceph-mon[36200]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T16:20:15.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:15 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:20:15.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:15 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:20:15.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:15 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:20:15.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:15 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:20:15.537 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:15 smithi039 ceph-mon[31721]: pgmap v727: 129 pgs: 129 active+clean; 12 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 587 KiB/s wr, 150 op/s 2024-03-18T16:20:15.537 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:15 smithi039 ceph-mon[31721]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T16:20:15.537 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:15 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:20:15.537 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:15 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:20:15.537 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:15 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:20:15.537 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:15 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:20:15.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.761+0000 7f1b9fe00640 1 -- 172.21.15.39:0/3841152228 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1ba0072bf0 msgr2=0x7f1ba0073010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:15.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.761+0000 7f1b9fe00640 1 --2- 172.21.15.39:0/3841152228 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1ba0072bf0 0x7f1ba0073010 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f1b98009be0 tx=0x7f1b9802bef0 comp rx=0 tx=0).stop 2024-03-18T16:20:15.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.761+0000 7f1b9fe00640 1 -- 172.21.15.39:0/3841152228 shutdown_connections 2024-03-18T16:20:15.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.761+0000 7f1b9fe00640 1 --2- 172.21.15.39:0/3841152228 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1ba0109870 0x7f1ba010e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:15.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.761+0000 7f1b9fe00640 1 --2- 172.21.15.39:0/3841152228 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1ba0108eb0 0x7f1ba0109330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:15.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.761+0000 7f1b9fe00640 1 --2- 172.21.15.39:0/3841152228 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1ba0072bf0 0x7f1ba0073010 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:15.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.761+0000 7f1b9fe00640 1 -- 172.21.15.39:0/3841152228 >> 172.21.15.39:0/3841152228 conn(0x7f1ba006e6c0 msgr2=0x7f1ba0070b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:15.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.761+0000 7f1b9fe00640 1 -- 172.21.15.39:0/3841152228 shutdown_connections 2024-03-18T16:20:15.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.762+0000 7f1b9fe00640 1 -- 172.21.15.39:0/3841152228 wait complete. 2024-03-18T16:20:15.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.763+0000 7f1b9fe00640 1 Processor -- start 2024-03-18T16:20:15.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.763+0000 7f1b9fe00640 1 -- start start 2024-03-18T16:20:15.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.764+0000 7f1b9fe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1ba0072bf0 0x7f1ba01a67d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:15.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.764+0000 7f1b9fe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1ba0108eb0 0x7f1ba01a6d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:15.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.764+0000 7f1b9fe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1ba0109870 0x7f1ba01add80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:15.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.764+0000 7f1b9fe00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f1ba011c180 con 0x7f1ba0072bf0 2024-03-18T16:20:15.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.764+0000 7f1b9fe00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f1ba011c000 con 0x7f1ba0108eb0 2024-03-18T16:20:15.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.764+0000 7f1b9fe00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f1ba011c300 con 0x7f1ba0109870 2024-03-18T16:20:15.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.764+0000 7f1b9e000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1ba0108eb0 0x7f1ba01a6d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:15.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.764+0000 7f1b9e000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1ba0108eb0 0x7f1ba01a6d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:57972/0 (socket says 172.21.15.39:57972) 2024-03-18T16:20:15.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.764+0000 7f1b9e000640 1 -- 172.21.15.39:0/1082395978 learned_addr learned my addr 172.21.15.39:0/1082395978 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:20:15.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.765+0000 7f1b9ea00640 1 --2- 172.21.15.39:0/1082395978 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1ba0072bf0 0x7f1ba01a67d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:15.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.765+0000 7f1b9e000640 1 -- 172.21.15.39:0/1082395978 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1ba0109870 msgr2=0x7f1ba01add80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:15.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.765+0000 7f1b9e000640 1 --2- 172.21.15.39:0/1082395978 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1ba0109870 0x7f1ba01add80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:15.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.765+0000 7f1b9e000640 1 -- 172.21.15.39:0/1082395978 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1ba0072bf0 msgr2=0x7f1ba01a67d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:15.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.765+0000 7f1b9e000640 1 --2- 172.21.15.39:0/1082395978 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1ba0072bf0 0x7f1ba01a67d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:15.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.765+0000 7f1b9e000640 1 -- 172.21.15.39:0/1082395978 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ba01ae3f0 con 0x7f1ba0108eb0 2024-03-18T16:20:15.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.765+0000 7f1b9ea00640 1 --2- 172.21.15.39:0/1082395978 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1ba0072bf0 0x7f1ba01a67d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:20:15.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.765+0000 7f1b9e000640 1 --2- 172.21.15.39:0/1082395978 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1ba0108eb0 0x7f1ba01a6d10 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f1b9000e9c0 tx=0x7f1b9000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:15.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.766+0000 7f1b97400640 1 -- 172.21.15.39:0/1082395978 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b9000ccc0 con 0x7f1ba0108eb0 2024-03-18T16:20:15.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.766+0000 7f1b97400640 1 -- 172.21.15.39:0/1082395978 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1b90005500 con 0x7f1ba0108eb0 2024-03-18T16:20:15.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.766+0000 7f1b97400640 1 -- 172.21.15.39:0/1082395978 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b90010430 con 0x7f1ba0108eb0 2024-03-18T16:20:15.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.766+0000 7f1b9fe00640 1 -- 172.21.15.39:0/1082395978 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ba01ae6e0 con 0x7f1ba0108eb0 2024-03-18T16:20:15.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.766+0000 7f1b9fe00640 1 -- 172.21.15.39:0/1082395978 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ba01aebe0 con 0x7f1ba0108eb0 2024-03-18T16:20:15.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.771+0000 7f1ba8577740 1 Processor -- start 2024-03-18T16:20:15.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.772+0000 7f1ba8577740 1 -- start start 2024-03-18T16:20:15.774 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.772+0000 7f1ba8577740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559787f81980 0x559787f81da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:15.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.772+0000 7f1ba8577740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559787f82370 0x559787f86e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:15.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.772+0000 7f1ba8577740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559787f874e0 0x559787f89970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:15.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.772+0000 7f1ba8577740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x559787ee4f80 con 0x559787f81980 2024-03-18T16:20:15.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.772+0000 7f1ba8577740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x559787e8dad0 con 0x559787f82370 2024-03-18T16:20:15.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.772+0000 7f1ba8577740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x559787ee1020 con 0x559787f874e0 2024-03-18T16:20:15.775 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.772+0000 7f1b97400640 1 -- 172.21.15.39:0/1082395978 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f1b900046c0 con 0x7f1ba0108eb0 2024-03-18T16:20:15.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.773+0000 7f1b9ea00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559787f82370 0x559787f86e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:15.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.773+0000 7f1b9ea00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559787f82370 0x559787f86e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:57984/0 (socket says 172.21.15.39:57984) 2024-03-18T16:20:15.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.773+0000 7f1b9ea00640 1 -- 172.21.15.39:0/28081353 learned_addr learned my addr 172.21.15.39:0/28081353 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:20:15.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.773+0000 7f1b9e000640 1 --2- 172.21.15.39:0/28081353 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559787f874e0 0x559787f89970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:15.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.773+0000 7f1b9f400640 1 --2- 172.21.15.39:0/28081353 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559787f81980 0x559787f81da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:15.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.773+0000 7f1b9e000640 1 -- 172.21.15.39:0/28081353 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559787f82370 msgr2=0x559787f86e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:15.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.773+0000 7f1b9e000640 1 --2- 172.21.15.39:0/28081353 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559787f82370 0x559787f86e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:15.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.773+0000 7f1b9e000640 1 -- 172.21.15.39:0/28081353 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559787f81980 msgr2=0x559787f81da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:15.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.773+0000 7f1b9e000640 1 --2- 172.21.15.39:0/28081353 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559787f81980 0x559787f81da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:15.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.774+0000 7f1b9e000640 1 -- 172.21.15.39:0/28081353 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559787ea56f0 con 0x559787f874e0 2024-03-18T16:20:15.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.774+0000 7f1b9e000640 1 --2- 172.21.15.39:0/28081353 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559787f874e0 0x559787f89970 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f1b900cc8c0 tx=0x7f1b900d7220 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c1386ed7bb0d731 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:15.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.775+0000 7f1b8fe00640 1 -- 172.21.15.39:0/28081353 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b900e2070 con 0x559787f874e0 2024-03-18T16:20:15.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.775+0000 7f1b8fe00640 1 -- 172.21.15.39:0/28081353 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1b900cf640 con 0x559787f874e0 2024-03-18T16:20:15.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.775+0000 7f1b8fe00640 1 -- 172.21.15.39:0/28081353 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b900dd660 con 0x559787f874e0 2024-03-18T16:20:15.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.776+0000 7f1b97400640 1 --2- 172.21.15.39:0/1082395978 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1b840a2900 0x7f1b840a4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:15.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.776+0000 7f1ba8577740 1 -- 172.21.15.39:0/28081353 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559787f874e0 msgr2=0x559787f89970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:15.778 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.776+0000 7f1ba8577740 1 --2- 172.21.15.39:0/28081353 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559787f874e0 0x559787f89970 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f1b900cc8c0 tx=0x7f1b900d7220 comp rx=0 tx=0).stop 2024-03-18T16:20:15.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.776+0000 7f1b97400640 1 -- 172.21.15.39:0/1082395978 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f1b900cf210 con 0x7f1ba0108eb0 2024-03-18T16:20:15.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.777+0000 7f1ba8577740 1 -- 172.21.15.39:0/28081353 shutdown_connections 2024-03-18T16:20:15.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.777+0000 7f1ba8577740 1 --2- 172.21.15.39:0/28081353 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559787f82370 0x559787f86e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:15.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.777+0000 7f1ba8577740 1 --2- 172.21.15.39:0/28081353 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559787f874e0 0x559787f89970 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:15.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.777+0000 7f1ba8577740 1 --2- 172.21.15.39:0/28081353 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559787f81980 0x559787f81da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:15.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.777+0000 7f1ba8577740 1 -- 172.21.15.39:0/28081353 >> 172.21.15.39:0/28081353 conn(0x559787f49930 msgr2=0x559787f7d2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:15.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.777+0000 7f1b9f400640 1 --2- 172.21.15.39:0/1082395978 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1b840a2900 0x7f1b840a4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:15.779 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.777+0000 7f1ba8577740 1 -- 172.21.15.39:0/28081353 shutdown_connections 2024-03-18T16:20:15.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.778+0000 7f1b9f400640 1 --2- 172.21.15.39:0/1082395978 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1b840a2900 0x7f1b840a4dc0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f1b88005d00 tx=0x7f1b88005f30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:15.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.778+0000 7f1ba8577740 1 -- 172.21.15.39:0/28081353 wait complete. 2024-03-18T16:20:15.780 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.779+0000 7f1ba8577740 1 -- start start 2024-03-18T16:20:15.781 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.779+0000 7f1ba8577740 1 Processor -- start 2024-03-18T16:20:15.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.780+0000 7f1ba8577740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559787f81980 0x559787f8bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:15.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.780+0000 7f1ba8577740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559787f82370 0x559787f8c190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:15.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.780+0000 7f1ba8577740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559787f874e0 0x559787f931d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:15.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.780+0000 7f1ba8577740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x559787f8a490 con 0x559787f81980 2024-03-18T16:20:15.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.780+0000 7f1ba8577740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x559787ea0f30 con 0x559787f82370 2024-03-18T16:20:15.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.780+0000 7f1ba8577740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x559787f8a610 con 0x559787f874e0 2024-03-18T16:20:15.782 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.780+0000 7f1ba8577740 10 client.0 _ll_register_callbacks cb 0x559787e8ebc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:20:15.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.781+0000 7f1b9f400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559787f874e0 0x559787f931d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:15.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.781+0000 7f1b9f400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559787f874e0 0x559787f931d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:38184/0 (socket says 172.21.15.39:38184) 2024-03-18T16:20:15.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.781+0000 7f1b9f400640 1 -- 172.21.15.39:0/141091386 learned_addr learned my addr 172.21.15.39:0/141091386 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:20:15.783 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.781+0000 7f1b9e000640 1 --2- 172.21.15.39:0/141091386 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559787f82370 0x559787f8c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:15.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.782+0000 7f1b9e000640 1 -- 172.21.15.39:0/141091386 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559787f874e0 msgr2=0x559787f931d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:15.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.782+0000 7f1b9e000640 1 --2- 172.21.15.39:0/141091386 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559787f874e0 0x559787f931d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:15.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.782+0000 7f1b9e000640 1 -- 172.21.15.39:0/141091386 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559787f81980 msgr2=0x559787f8bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:15.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.782+0000 7f1b9e000640 1 --2- 172.21.15.39:0/141091386 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559787f81980 0x559787f8bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:15.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.782+0000 7f1b9e000640 1 -- 172.21.15.39:0/141091386 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559787cc17e0 con 0x559787f82370 2024-03-18T16:20:15.784 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.783+0000 7f1b9e000640 1 --2- 172.21.15.39:0/141091386 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559787f82370 0x559787f8c190 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f1b90000c00 tx=0x7f1b90009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4dbcee71a6a257a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:15.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.783+0000 7f1b8cc00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:20:15.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.784+0000 7f1b8cc00640 1 -- 172.21.15.39:0/141091386 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b90097a60 con 0x559787f82370 2024-03-18T16:20:15.786 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.785+0000 7f1b8cc00640 1 -- 172.21.15.39:0/141091386 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f1b90016db0 con 0x559787f82370 2024-03-18T16:20:15.787 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.786+0000 7f1ba8577740 1 -- 172.21.15.39:0/141091386 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1b900cf640 con 0x559787f82370 2024-03-18T16:20:15.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.786+0000 7f1b8cc00640 1 -- 172.21.15.39:0/141091386 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f1b9009b120 con 0x559787f82370 2024-03-18T16:20:15.788 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.787+0000 7f1ba8577740 10 client.27725 fetch_fsmap learned FSMap version 47 2024-03-18T16:20:15.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.787+0000 7f1ba8577740 1 -- 172.21.15.39:0/141091386 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x559787d21f70 con 0x559787f82370 2024-03-18T16:20:15.789 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.788+0000 7f1b8cc00640 1 -- 172.21.15.39:0/141091386 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1b900cf640 con 0x559787f82370 2024-03-18T16:20:15.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.789+0000 7f1b8cc00640 1 -- 172.21.15.39:0/141091386 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 47) ==== 8374+0+0 (secure 0 0 0) 0x7f1b90097e30 con 0x559787f82370 2024-03-18T16:20:15.790 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.789+0000 7f1ba8577740 10 client.27725 fetch_fsmap finished waiting for FSMap version 47 2024-03-18T16:20:15.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.789+0000 7f1ba8577740 10 client.27725 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:20:15.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.789+0000 7f1ba8577740 20 client.27725 populate_metadata read hostname 'smithi039' 2024-03-18T16:20:15.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.789+0000 7f1ba8577740 1 --2- 172.21.15.39:0/141091386 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x559787f976c0 0x559787f99ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:15.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.789+0000 7f1ba8577740 4 client.27725 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:20:15.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.789+0000 7f1ba8577740 1 -- 172.21.15.39:0/141091386 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559787ef24c0 con 0x559787f976c0 2024-03-18T16:20:15.791 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.790+0000 7f1b9ea00640 1 --2- 172.21.15.39:0/141091386 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x559787f976c0 0x559787f99ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:15.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.793+0000 7f1b8cc00640 10 client.27725 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:20:15.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.793+0000 7f1b9ea00640 1 --2- 172.21.15.39:0/141091386 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x559787f976c0 0x559787f99ab0 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8eb40d06e2b8b6f2 server_cookie=e270161b4fb9ef35 in_seq=0 out_seq=0 2024-03-18T16:20:15.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.796+0000 7f1b8cc00640 1 -- 172.21.15.39:0/141091386 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x559787ef24c0 con 0x559787f976c0 2024-03-18T16:20:15.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.796+0000 7f1b8cc00640 10 client.27725 handle_command_reply: tid=0 2024-03-18T16:20:15.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.796+0000 7f1ba8577740 1 client.27725 shutdown 2024-03-18T16:20:15.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.797+0000 7f1ba8577740 1 -- 172.21.15.39:0/141091386 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559787f82370 msgr2=0x559787f8c190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:15.798 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.797+0000 7f1ba8577740 1 --2- 172.21.15.39:0/141091386 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559787f82370 0x559787f8c190 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f1b90000c00 tx=0x7f1b90009f90 comp rx=0 tx=0).stop 2024-03-18T16:20:15.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.797+0000 7f1ba8577740 1 -- 172.21.15.39:0/141091386 shutdown_connections 2024-03-18T16:20:15.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.797+0000 7f1ba8577740 1 --2- 172.21.15.39:0/141091386 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x559787f976c0 0x559787f99ab0 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:15.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.797+0000 7f1ba8577740 1 --2- 172.21.15.39:0/141091386 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559787f82370 0x559787f8c190 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:15.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.797+0000 7f1b8cc00640 0 client.27725 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:20:15.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.797+0000 7f1ba8577740 1 --2- 172.21.15.39:0/141091386 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559787f874e0 0x559787f931d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:15.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.797+0000 7f1ba8577740 1 --2- 172.21.15.39:0/141091386 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559787f81980 0x559787f8bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:15.799 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.797+0000 7f1ba8577740 1 -- 172.21.15.39:0/141091386 >> 172.21.15.39:0/141091386 conn(0x559787f49930 msgr2=0x559787f7c570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:15.800 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.798+0000 7f1ba8577740 1 -- 172.21.15.39:0/141091386 shutdown_connections 2024-03-18T16:20:15.800 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.798+0000 7f1ba8577740 1 -- 172.21.15.39:0/141091386 wait complete. 2024-03-18T16:20:15.800 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.798+0000 7f1ba8577740 20 client.27725 trim_cache size 0 max 16384 2024-03-18T16:20:15.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.844+0000 7f1ba8577740 1 Processor -- start 2024-03-18T16:20:15.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.846+0000 7f1ba8577740 1 -- start start 2024-03-18T16:20:15.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.846+0000 7f1ba8577740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559787f81980 0x559787f80530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:15.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.846+0000 7f1ba8577740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559787f82370 0x559787f93190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:15.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.846+0000 7f1ba8577740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559787f874e0 0x559787f80a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:15.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.846+0000 7f1ba8577740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f1ba01b9240 con 0x559787f82370 2024-03-18T16:20:15.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.847+0000 7f1ba8577740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f1ba01badb0 con 0x559787f874e0 2024-03-18T16:20:15.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.847+0000 7f1ba8577740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f1ba01b9910 con 0x559787f81980 2024-03-18T16:20:15.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.847+0000 7f1b9ea00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559787f81980 0x559787f80530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:15.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.847+0000 7f1b9ea00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559787f81980 0x559787f80530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:38192/0 (socket says 172.21.15.39:38192) 2024-03-18T16:20:15.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.847+0000 7f1b9f400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559787f874e0 0x559787f80a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:15.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.847+0000 7f1b9f400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559787f874e0 0x559787f80a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:57992/0 (socket says 172.21.15.39:57992) 2024-03-18T16:20:15.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.847+0000 7f1b9f400640 1 -- 172.21.15.39:0/1382639887 learned_addr learned my addr 172.21.15.39:0/1382639887 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:20:15.851 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.850+0000 7f1b9f400640 1 -- 172.21.15.39:0/1382639887 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559787f81980 msgr2=0x559787f80530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:15.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.850+0000 7f1b9f400640 1 --2- 172.21.15.39:0/1382639887 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559787f81980 0x559787f80530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:15.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.850+0000 7f1b9f400640 1 -- 172.21.15.39:0/1382639887 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559787f82370 msgr2=0x559787f93190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:15.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.850+0000 7f1b9f400640 1 --2- 172.21.15.39:0/1382639887 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559787f82370 0x559787f93190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:15.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.850+0000 7f1b9f400640 1 -- 172.21.15.39:0/1382639887 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559787bb9170 con 0x559787f874e0 2024-03-18T16:20:15.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.850+0000 7f1b9ea00640 1 --2- 172.21.15.39:0/1382639887 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559787f81980 0x559787f80530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:20:15.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.851+0000 7f1b9f400640 1 --2- 172.21.15.39:0/1382639887 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559787f874e0 0x559787f80a70 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f1b880114c0 tx=0x7f1b88012000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=30bf3c1273058a26 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:15.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.851+0000 7f1b8f400640 1 -- 172.21.15.39:0/1382639887 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b88004230 con 0x559787f874e0 2024-03-18T16:20:15.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.852+0000 7f1b8f400640 1 -- 172.21.15.39:0/1382639887 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1b88023070 con 0x559787f874e0 2024-03-18T16:20:15.854 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.852+0000 7f1b8f400640 1 -- 172.21.15.39:0/1382639887 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b8801f860 con 0x559787f874e0 2024-03-18T16:20:15.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.853+0000 7f1ba8577740 1 -- 172.21.15.39:0/1382639887 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559787f874e0 msgr2=0x559787f80a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:15.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.853+0000 7f1ba8577740 1 --2- 172.21.15.39:0/1382639887 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559787f874e0 0x559787f80a70 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f1b880114c0 tx=0x7f1b88012000 comp rx=0 tx=0).stop 2024-03-18T16:20:15.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.853+0000 7f1ba8577740 1 -- 172.21.15.39:0/1382639887 shutdown_connections 2024-03-18T16:20:15.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.853+0000 7f1ba8577740 1 --2- 172.21.15.39:0/1382639887 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559787f874e0 0x559787f80a70 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:15.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.853+0000 7f1ba8577740 1 --2- 172.21.15.39:0/1382639887 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559787f82370 0x559787f93190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:15.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.853+0000 7f1ba8577740 1 --2- 172.21.15.39:0/1382639887 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559787f81980 0x559787f80530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:15.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.853+0000 7f1ba8577740 1 -- 172.21.15.39:0/1382639887 >> 172.21.15.39:0/1382639887 conn(0x559787f49930 msgr2=0x559787f8bc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:15.855 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.853+0000 7f1ba8577740 1 -- 172.21.15.39:0/1382639887 shutdown_connections 2024-03-18T16:20:15.856 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.853+0000 7f1ba8577740 1 -- 172.21.15.39:0/1382639887 wait complete. 2024-03-18T16:20:15.857 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.856+0000 7f1ba8577740 1 -- start start 2024-03-18T16:20:15.858 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.856+0000 7f1ba8577740 1 Processor -- start 2024-03-18T16:20:15.859 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.857+0000 7f1ba8577740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559787f81980 0x559787f90e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:15.859 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.857+0000 7f1ba8577740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559787f82370 0x559787f913a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:15.859 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.857+0000 7f1ba8577740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559787f874e0 0x559787f92360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:15.859 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.857+0000 7f1ba8577740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f1ba01ba6b0 con 0x559787f874e0 2024-03-18T16:20:15.859 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.857+0000 7f1ba8577740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f1ba01b9fe0 con 0x559787f82370 2024-03-18T16:20:15.859 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.857+0000 7f1ba8577740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f1ba01b7220 con 0x559787f81980 2024-03-18T16:20:15.859 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.857+0000 7f1ba8577740 10 client.0 _ll_register_callbacks cb 0x7f1b900e0070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:20:15.860 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.857+0000 7f1b9e000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559787f82370 0x559787f913a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:15.860 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.858+0000 7f1b9e000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559787f82370 0x559787f913a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:58000/0 (socket says 172.21.15.39:58000) 2024-03-18T16:20:15.860 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.858+0000 7f1b9e000640 1 -- 172.21.15.39:0/1637901680 learned_addr learned my addr 172.21.15.39:0/1637901680 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:20:15.860 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.858+0000 7f1b9e000640 1 -- 172.21.15.39:0/1637901680 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559787f81980 msgr2=0x559787f90e60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:20:15.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.858+0000 7f1b9e000640 1 --2- 172.21.15.39:0/1637901680 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559787f81980 0x559787f90e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:15.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.858+0000 7f1b9e000640 1 -- 172.21.15.39:0/1637901680 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559787f874e0 msgr2=0x559787f92360 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:20:15.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.858+0000 7f1b9e000640 1 --2- 172.21.15.39:0/1637901680 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559787f874e0 0x559787f92360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:15.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.858+0000 7f1b9e000640 1 -- 172.21.15.39:0/1637901680 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559787f92950 con 0x559787f82370 2024-03-18T16:20:15.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.859+0000 7f1b9e000640 1 --2- 172.21.15.39:0/1637901680 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559787f82370 0x559787f913a0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f1b9000c6a0 tx=0x7f1b90099cd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2863231e67a16f0b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:15.861 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.859+0000 7f1b8cc00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:20:15.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.861+0000 7f1b8cc00640 1 -- 172.21.15.39:0/1637901680 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b9009b300 con 0x559787f82370 2024-03-18T16:20:15.862 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.861+0000 7f1b8cc00640 1 -- 172.21.15.39:0/1637901680 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f1b900860f0 con 0x559787f82370 2024-03-18T16:20:15.863 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.862+0000 7f1ba8577740 1 -- 172.21.15.39:0/1637901680 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1b88023070 con 0x559787f82370 2024-03-18T16:20:15.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.862+0000 7f1b8cc00640 1 -- 172.21.15.39:0/1637901680 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f1b9009b590 con 0x559787f82370 2024-03-18T16:20:15.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.862+0000 7f1ba8577740 10 client.27731 fetch_fsmap learned FSMap version 47 2024-03-18T16:20:15.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.862+0000 7f1ba8577740 1 -- 172.21.15.39:0/1637901680 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x559787ffdd40 con 0x559787f82370 2024-03-18T16:20:15.864 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.863+0000 7f1b8cc00640 1 -- 172.21.15.39:0/1637901680 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1b88023070 con 0x559787f82370 2024-03-18T16:20:15.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.863+0000 7f1b8cc00640 1 -- 172.21.15.39:0/1637901680 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 47) ==== 8374+0+0 (secure 0 0 0) 0x7f1b900d2350 con 0x559787f82370 2024-03-18T16:20:15.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.863+0000 7f1ba8577740 10 client.27731 fetch_fsmap finished waiting for FSMap version 47 2024-03-18T16:20:15.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.863+0000 7f1ba8577740 10 client.27731 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:20:15.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.863+0000 7f1ba8577740 20 client.27731 populate_metadata read hostname 'smithi039' 2024-03-18T16:20:15.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.863+0000 7f1ba8577740 1 --2- 172.21.15.39:0/1637901680 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x559787ffe580 0x5597880009a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:15.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.863+0000 7f1ba8577740 4 client.27731 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-18T16:20:15.865 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.863+0000 7f1ba8577740 1 -- 172.21.15.39:0/1637901680 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x559787f8b090 con 0x559787ffe580 2024-03-18T16:20:15.866 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.865+0000 7f1b9ea00640 1 --2- 172.21.15.39:0/1637901680 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x559787ffe580 0x5597880009a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:15.867 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.866+0000 7f1b9ea00640 1 --2- 172.21.15.39:0/1637901680 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x559787ffe580 0x5597880009a0 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1db76bccf32f7064 server_cookie=7398774a6f1f59da in_seq=0 out_seq=0 2024-03-18T16:20:15.867 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.866+0000 7f1b8cc00640 10 client.27731 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:20:15.868 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.867+0000 7f1b8cc00640 1 -- 172.21.15.39:0/1637901680 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x559787f8b090 con 0x559787ffe580 2024-03-18T16:20:15.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.867+0000 7f1b8cc00640 10 client.27731 handle_command_reply: tid=0 2024-03-18T16:20:15.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.867+0000 7f1ba8577740 1 client.27731 shutdown 2024-03-18T16:20:15.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.868+0000 7f1ba8577740 1 -- 172.21.15.39:0/1637901680 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559787f82370 msgr2=0x559787f913a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:15.869 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.868+0000 7f1ba8577740 1 --2- 172.21.15.39:0/1637901680 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559787f82370 0x559787f913a0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f1b9000c6a0 tx=0x7f1b90099cd0 comp rx=0 tx=0).stop 2024-03-18T16:20:15.871 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.868+0000 7f1ba8577740 1 -- 172.21.15.39:0/1637901680 shutdown_connections 2024-03-18T16:20:15.871 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.868+0000 7f1ba8577740 1 --2- 172.21.15.39:0/1637901680 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x559787ffe580 0x5597880009a0 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:15.871 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.868+0000 7f1ba8577740 1 --2- 172.21.15.39:0/1637901680 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559787f82370 0x559787f913a0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:15.871 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.868+0000 7f1ba8577740 1 --2- 172.21.15.39:0/1637901680 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x559787f874e0 0x559787f92360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:15.871 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.868+0000 7f1ba8577740 1 --2- 172.21.15.39:0/1637901680 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x559787f81980 0x559787f90e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:15.871 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.868+0000 7f1ba8577740 1 -- 172.21.15.39:0/1637901680 >> 172.21.15.39:0/1637901680 conn(0x559787f49930 msgr2=0x559787f7bce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:15.871 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.868+0000 7f1b8cc00640 0 client.27731 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:20:15.872 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.868+0000 7f1ba8577740 1 -- 172.21.15.39:0/1637901680 shutdown_connections 2024-03-18T16:20:15.872 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.868+0000 7f1ba8577740 1 -- 172.21.15.39:0/1637901680 wait complete. 2024-03-18T16:20:15.872 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.869+0000 7f1ba8577740 20 client.27731 trim_cache size 0 max 16384 2024-03-18T16:20:15.872 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-03-18T16:20:15.872 INFO:teuthology.orchestra.run.smithi039.stdout: "return_code": 0, 2024-03-18T16:20:15.872 INFO:teuthology.orchestra.run.smithi039.stdout: "scrub_tag": "063a4e60-b3a3-4161-8315-0308493ac4f5", 2024-03-18T16:20:15.872 INFO:teuthology.orchestra.run.smithi039.stdout: "mode": "asynchronous" 2024-03-18T16:20:15.872 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-03-18T16:20:15.873 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.871+0000 7f1b9fe00640 1 -- 172.21.15.39:0/1082395978 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1b840a2900 msgr2=0x7f1b840a4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:15.873 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.871+0000 7f1b9fe00640 1 --2- 172.21.15.39:0/1082395978 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1b840a2900 0x7f1b840a4dc0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f1b88005d00 tx=0x7f1b88005f30 comp rx=0 tx=0).stop 2024-03-18T16:20:15.873 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.871+0000 7f1b9fe00640 1 -- 172.21.15.39:0/1082395978 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1ba0108eb0 msgr2=0x7f1ba01a6d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:15.873 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.871+0000 7f1b9fe00640 1 --2- 172.21.15.39:0/1082395978 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1ba0108eb0 0x7f1ba01a6d10 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f1b9000e9c0 tx=0x7f1b9000ee80 comp rx=0 tx=0).stop 2024-03-18T16:20:15.873 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.872+0000 7f1b9fe00640 1 -- 172.21.15.39:0/1082395978 shutdown_connections 2024-03-18T16:20:15.873 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.872+0000 7f1b9fe00640 1 --2- 172.21.15.39:0/1082395978 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1ba0108eb0 0x7f1ba01a6d10 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:15.873 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.872+0000 7f1b9fe00640 1 --2- 172.21.15.39:0/1082395978 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f1b840a2900 0x7f1b840a4dc0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:15.873 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.872+0000 7f1b9fe00640 1 --2- 172.21.15.39:0/1082395978 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1ba0109870 0x7f1ba01add80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:15.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.872+0000 7f1b9fe00640 1 --2- 172.21.15.39:0/1082395978 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f1ba0072bf0 0x7f1ba01a67d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:15.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.872+0000 7f1b9fe00640 1 -- 172.21.15.39:0/1082395978 >> 172.21.15.39:0/1082395978 conn(0x7f1ba006e6c0 msgr2=0x7f1ba0070320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:15.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.872+0000 7f1b9fe00640 1 -- 172.21.15.39:0/1082395978 shutdown_connections 2024-03-18T16:20:15.874 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:15.872+0000 7f1b9fe00640 1 -- 172.21.15.39:0/1082395978 wait complete. 2024-03-18T16:20:15.908 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T16:20:15.908 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T16:20:16.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.238+0000 7f1980000640 1 -- 172.21.15.39:0/366595737 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f19780757c0 msgr2=0x7f1978073c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:16.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.238+0000 7f1980000640 1 --2- 172.21.15.39:0/366595737 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f19780757c0 0x7f1978073c10 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f196c009900 tx=0x7f196c02bcc0 comp rx=0 tx=0).stop 2024-03-18T16:20:16.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.239+0000 7f1980000640 1 -- 172.21.15.39:0/366595737 shutdown_connections 2024-03-18T16:20:16.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.239+0000 7f1980000640 1 --2- 172.21.15.39:0/366595737 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f19781096a0 0x7f197810bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:16.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.239+0000 7f1980000640 1 --2- 172.21.15.39:0/366595737 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f19780741e0 0x7f1978074640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:16.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.239+0000 7f1980000640 1 --2- 172.21.15.39:0/366595737 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f19780757c0 0x7f1978073c10 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:16.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.239+0000 7f1980000640 1 -- 172.21.15.39:0/366595737 >> 172.21.15.39:0/366595737 conn(0x7f19780fcd80 msgr2=0x7f19780ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:16.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.241+0000 7f1980000640 1 -- 172.21.15.39:0/366595737 shutdown_connections 2024-03-18T16:20:16.242 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.241+0000 7f1980000640 1 -- 172.21.15.39:0/366595737 wait complete. 2024-03-18T16:20:16.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.242+0000 7f1980000640 1 Processor -- start 2024-03-18T16:20:16.244 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.243+0000 7f1980000640 1 -- start start 2024-03-18T16:20:16.245 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.243+0000 7f1980000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f19780741e0 0x7f1978195890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:16.245 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.243+0000 7f1980000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f19780757c0 0x7f1978195dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:16.245 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.243+0000 7f197d600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f19780741e0 0x7f1978195890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:16.245 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.243+0000 7f197d600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f19780741e0 0x7f1978195890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:38210/0 (socket says 172.21.15.39:38210) 2024-03-18T16:20:16.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.244+0000 7f197d600640 1 -- 172.21.15.39:0/2990878119 learned_addr learned my addr 172.21.15.39:0/2990878119 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:20:16.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.245+0000 7f1980000640 1 --2- 172.21.15.39:0/2990878119 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f19781096a0 0x7f197819ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:16.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.245+0000 7f1980000640 1 -- 172.21.15.39:0/2990878119 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f197810f6b0 con 0x7f19780757c0 2024-03-18T16:20:16.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.245+0000 7f1980000640 1 -- 172.21.15.39:0/2990878119 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f197810f530 con 0x7f19781096a0 2024-03-18T16:20:16.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.245+0000 7f1980000640 1 -- 172.21.15.39:0/2990878119 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f197810f830 con 0x7f19780741e0 2024-03-18T16:20:16.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.245+0000 7f197e000640 1 --2- 172.21.15.39:0/2990878119 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f19781096a0 0x7f197819ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:16.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.246+0000 7f197d600640 1 -- 172.21.15.39:0/2990878119 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f19781096a0 msgr2=0x7f197819ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:16.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.248+0000 7f197d600640 1 --2- 172.21.15.39:0/2990878119 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f19781096a0 0x7f197819ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:16.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.248+0000 7f197d600640 1 -- 172.21.15.39:0/2990878119 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f19780757c0 msgr2=0x7f1978195dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:16.250 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.248+0000 7f197d600640 1 --2- 172.21.15.39:0/2990878119 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f19780757c0 0x7f1978195dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:16.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.249+0000 7f197d600640 1 -- 172.21.15.39:0/2990878119 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19781ae5b0 con 0x7f19780741e0 2024-03-18T16:20:16.251 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.249+0000 7f197d600640 1 --2- 172.21.15.39:0/2990878119 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f19780741e0 0x7f1978195890 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f196c000c00 tx=0x7f196c005dd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:16.255 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.254+0000 7f1976000640 1 -- 172.21.15.39:0/2990878119 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f196c043020 con 0x7f19780741e0 2024-03-18T16:20:16.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.254+0000 7f1980000640 1 -- 172.21.15.39:0/2990878119 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19781ae780 con 0x7f19780741e0 2024-03-18T16:20:16.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.254+0000 7f1980000640 1 -- 172.21.15.39:0/2990878119 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19781aecf0 con 0x7f19780741e0 2024-03-18T16:20:16.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.254+0000 7f1976000640 1 -- 172.21.15.39:0/2990878119 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f196c004350 con 0x7f19780741e0 2024-03-18T16:20:16.256 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.254+0000 7f1976000640 1 -- 172.21.15.39:0/2990878119 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f196c043020 con 0x7f19780741e0 2024-03-18T16:20:16.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.260+0000 7f1976000640 1 -- 172.21.15.39:0/2990878119 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f196c046020 con 0x7f19780741e0 2024-03-18T16:20:16.264 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.262+0000 7f1982df9740 1 Processor -- start 2024-03-18T16:20:16.265 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.263+0000 7f1982df9740 1 -- start start 2024-03-18T16:20:16.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.264+0000 7f1976000640 1 --2- 172.21.15.39:0/2990878119 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f19600a2b00 0x7f19600a4fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:16.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.264+0000 7f1976000640 1 -- 172.21.15.39:0/2990878119 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f196c0f1710 con 0x7f19780741e0 2024-03-18T16:20:16.266 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.265+0000 7f197e000640 1 --2- 172.21.15.39:0/2990878119 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f19600a2b00 0x7f19600a4fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:16.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.266+0000 7f1982df9740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d2f93ea810 0x55d2f93eac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:16.267 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.266+0000 7f1982df9740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d2f93eb200 0x55d2f93efd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:16.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.266+0000 7f1982df9740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d2f93f0370 0x55d2f93f2800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:16.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.266+0000 7f1982df9740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55d2f934f1f0 con 0x55d2f93ea810 2024-03-18T16:20:16.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.266+0000 7f1982df9740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55d2f92f7a80 con 0x55d2f93eb200 2024-03-18T16:20:16.268 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.266+0000 7f1982df9740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55d2f934b290 con 0x55d2f93f0370 2024-03-18T16:20:16.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.266+0000 7f197e000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d2f93f0370 0x55d2f93f2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:16.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.266+0000 7f197e000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d2f93f0370 0x55d2f93f2800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:38220/0 (socket says 172.21.15.39:38220) 2024-03-18T16:20:16.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.266+0000 7f197e000640 1 -- 172.21.15.39:0/3356854924 learned_addr learned my addr 172.21.15.39:0/3356854924 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:20:16.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.267+0000 7f197e000640 1 -- 172.21.15.39:0/3356854924 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d2f93eb200 msgr2=0x55d2f93efd00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:20:16.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.267+0000 7f197e000640 1 --2- 172.21.15.39:0/3356854924 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d2f93eb200 0x55d2f93efd00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:16.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.267+0000 7f197e000640 1 -- 172.21.15.39:0/3356854924 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d2f93ea810 msgr2=0x55d2f93eac30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:20:16.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.267+0000 7f197e000640 1 --2- 172.21.15.39:0/3356854924 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d2f93ea810 0x55d2f93eac30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:16.269 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.267+0000 7f197e000640 1 -- 172.21.15.39:0/3356854924 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d2f912d7d0 con 0x55d2f93f0370 2024-03-18T16:20:16.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.268+0000 7f197e000640 1 --2- 172.21.15.39:0/3356854924 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d2f93f0370 0x55d2f93f2800 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f197000efd0 tx=0x7f1970004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b802af67108b32b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:16.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.268+0000 7f197e000640 1 --2- 172.21.15.39:0/2990878119 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f19600a2b00 0x7f19600a4fc0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f197000ace0 tx=0x7f197000a2b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:16.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.268+0000 7f196a600640 1 -- 172.21.15.39:0/3356854924 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1970017070 con 0x55d2f93f0370 2024-03-18T16:20:16.270 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.268+0000 7f196a600640 1 -- 172.21.15.39:0/3356854924 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f197000f840 con 0x55d2f93f0370 2024-03-18T16:20:16.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.269+0000 7f1982df9740 1 -- 172.21.15.39:0/3356854924 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d2f93f0370 msgr2=0x55d2f93f2800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:16.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.269+0000 7f1982df9740 1 --2- 172.21.15.39:0/3356854924 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d2f93f0370 0x55d2f93f2800 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f197000efd0 tx=0x7f1970004a40 comp rx=0 tx=0).stop 2024-03-18T16:20:16.271 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.270+0000 7f1982df9740 1 -- 172.21.15.39:0/3356854924 shutdown_connections 2024-03-18T16:20:16.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.270+0000 7f1982df9740 1 --2- 172.21.15.39:0/3356854924 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d2f93eb200 0x55d2f93efd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:16.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.270+0000 7f1982df9740 1 --2- 172.21.15.39:0/3356854924 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d2f93f0370 0x55d2f93f2800 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:16.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.270+0000 7f1982df9740 1 --2- 172.21.15.39:0/3356854924 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d2f93ea810 0x55d2f93eac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:16.273 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.270+0000 7f1982df9740 1 -- 172.21.15.39:0/3356854924 >> 172.21.15.39:0/3356854924 conn(0x55d2f93e5d80 msgr2=0x55d2f93e6180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:16.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.272+0000 7f1982df9740 1 -- 172.21.15.39:0/3356854924 shutdown_connections 2024-03-18T16:20:16.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.272+0000 7f1982df9740 1 -- 172.21.15.39:0/3356854924 wait complete. 2024-03-18T16:20:16.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.273+0000 7f1982df9740 1 -- start start 2024-03-18T16:20:16.274 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.273+0000 7f1982df9740 1 Processor -- start 2024-03-18T16:20:16.277 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.274+0000 7f1982df9740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d2f93ea810 0x55d2f93f4d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:16.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.276+0000 7f1982df9740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d2f93eb200 0x55d2f93f5260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:16.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.276+0000 7f1982df9740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d2f93f0370 0x55d2f93fc260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:16.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.276+0000 7f197d600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d2f93ea810 0x55d2f93f4d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:16.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.276+0000 7f197d600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d2f93ea810 0x55d2f93f4d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:58018/0 (socket says 172.21.15.39:58018) 2024-03-18T16:20:16.278 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.276+0000 7f197d600640 1 -- 172.21.15.39:0/2360825926 learned_addr learned my addr 172.21.15.39:0/2360825926 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:20:16.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.276+0000 7f1982df9740 1 -- 172.21.15.39:0/2360825926 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55d2f93f3360 con 0x55d2f93eb200 2024-03-18T16:20:16.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.276+0000 7f1982df9740 1 -- 172.21.15.39:0/2360825926 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55d2f930aed0 con 0x55d2f93ea810 2024-03-18T16:20:16.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.276+0000 7f1982df9740 1 -- 172.21.15.39:0/2360825926 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55d2f93f34e0 con 0x55d2f93f0370 2024-03-18T16:20:16.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.276+0000 7f1982df9740 10 client.0 _ll_register_callbacks cb 0x55d2f9346220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:20:16.279 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.277+0000 7f197d600640 1 -- 172.21.15.39:0/2360825926 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d2f93f0370 msgr2=0x55d2f93fc260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:16.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.277+0000 7f197d600640 1 --2- 172.21.15.39:0/2360825926 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d2f93f0370 0x55d2f93fc260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:16.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.277+0000 7f197d600640 1 -- 172.21.15.39:0/2360825926 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d2f93eb200 msgr2=0x55d2f93f5260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:16.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.277+0000 7f197d600640 1 --2- 172.21.15.39:0/2360825926 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d2f93eb200 0x55d2f93f5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:16.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.277+0000 7f197d600640 1 -- 172.21.15.39:0/2360825926 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d2f90633f0 con 0x55d2f93ea810 2024-03-18T16:20:16.280 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.278+0000 7f197d600640 1 --2- 172.21.15.39:0/2360825926 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d2f93ea810 0x55d2f93f4d20 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f196c0bd9c0 tx=0x7f196c0bd8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=17b07bde2867872 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:16.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.281+0000 7f195ea00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:20:16.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.281+0000 7f195ea00640 1 -- 172.21.15.39:0/2360825926 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f196c0c1e10 con 0x55d2f93ea810 2024-03-18T16:20:16.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.281+0000 7f195ea00640 1 -- 172.21.15.39:0/2360825926 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f196c0c5fd0 con 0x55d2f93ea810 2024-03-18T16:20:16.283 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.281+0000 7f1982df9740 1 -- 172.21.15.39:0/2360825926 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f197000f840 con 0x55d2f93ea810 2024-03-18T16:20:16.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.285+0000 7f195ea00640 1 -- 172.21.15.39:0/2360825926 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f196c0c6460 con 0x55d2f93ea810 2024-03-18T16:20:16.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.285+0000 7f1982df9740 10 client.27737 fetch_fsmap learned FSMap version 47 2024-03-18T16:20:16.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.285+0000 7f1982df9740 1 -- 172.21.15.39:0/2360825926 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55d2f90230f0 con 0x55d2f93ea810 2024-03-18T16:20:16.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.285+0000 7f195ea00640 1 -- 172.21.15.39:0/2360825926 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f197000f840 con 0x55d2f93ea810 2024-03-18T16:20:16.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.286+0000 7f195ea00640 1 -- 172.21.15.39:0/2360825926 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 47) ==== 8374+0+0 (secure 0 0 0) 0x7f196c0bd110 con 0x55d2f93ea810 2024-03-18T16:20:16.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.286+0000 7f1982df9740 10 client.27737 fetch_fsmap finished waiting for FSMap version 47 2024-03-18T16:20:16.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.286+0000 7f1982df9740 10 client.27737 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:20:16.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.286+0000 7f1982df9740 20 client.27737 populate_metadata read hostname 'smithi039' 2024-03-18T16:20:16.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.286+0000 7f1982df9740 1 --2- 172.21.15.39:0/2360825926 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55d2f9400d30 0x55d2f94031d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:16.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.286+0000 7f1982df9740 4 client.27737 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:20:16.288 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.286+0000 7f1982df9740 1 -- 172.21.15.39:0/2360825926 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d2f9362be0 con 0x55d2f9400d30 2024-03-18T16:20:16.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.286+0000 7f197cc00640 1 --2- 172.21.15.39:0/2360825926 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55d2f9400d30 0x55d2f94031d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:16.289 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.287+0000 7f195ea00640 10 client.27737 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:20:16.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.288+0000 7f197cc00640 1 --2- 172.21.15.39:0/2360825926 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55d2f9400d30 0x55d2f94031d0 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=69ec8103246388c3 server_cookie=b807858d0b868bb7 in_seq=0 out_seq=0 2024-03-18T16:20:16.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.288+0000 7f195ea00640 1 -- 172.21.15.39:0/2360825926 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55d2f9362be0 con 0x55d2f9400d30 2024-03-18T16:20:16.290 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.289+0000 7f195ea00640 10 client.27737 handle_command_reply: tid=0 2024-03-18T16:20:16.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.289+0000 7f1982df9740 1 client.27737 shutdown 2024-03-18T16:20:16.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.289+0000 7f1982df9740 1 -- 172.21.15.39:0/2360825926 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d2f93ea810 msgr2=0x55d2f93f4d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:16.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.289+0000 7f1982df9740 1 --2- 172.21.15.39:0/2360825926 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d2f93ea810 0x55d2f93f4d20 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f196c0bd9c0 tx=0x7f196c0bd8c0 comp rx=0 tx=0).stop 2024-03-18T16:20:16.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.289+0000 7f1982df9740 1 -- 172.21.15.39:0/2360825926 shutdown_connections 2024-03-18T16:20:16.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.289+0000 7f1982df9740 1 --2- 172.21.15.39:0/2360825926 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55d2f9400d30 0x55d2f94031d0 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:16.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.289+0000 7f1982df9740 1 --2- 172.21.15.39:0/2360825926 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d2f93f0370 0x55d2f93fc260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:16.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.289+0000 7f1982df9740 1 --2- 172.21.15.39:0/2360825926 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d2f93eb200 0x55d2f93f5260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:16.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.289+0000 7f1982df9740 1 --2- 172.21.15.39:0/2360825926 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d2f93ea810 0x55d2f93f4d20 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:16.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.289+0000 7f195ea00640 0 client.27737 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:20:16.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.289+0000 7f1982df9740 1 -- 172.21.15.39:0/2360825926 >> 172.21.15.39:0/2360825926 conn(0x55d2f93e5d80 msgr2=0x55d2f93e4ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:16.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.290+0000 7f1982df9740 1 -- 172.21.15.39:0/2360825926 shutdown_connections 2024-03-18T16:20:16.293 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.290+0000 7f1982df9740 1 -- 172.21.15.39:0/2360825926 wait complete. 2024-03-18T16:20:16.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.291+0000 7f1982df9740 20 client.27737 trim_cache size 0 max 16384 2024-03-18T16:20:16.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:16 smithi039 ceph-mon[31721]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T16:20:16.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:16 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:20:16.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:16 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:20:16.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:16 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:20:16.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:16 smithi039 ceph-mon[31721]: pgmap v728: 129 pgs: 129 active+clean; 12 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 456 KiB/s wr, 134 op/s 2024-03-18T16:20:16.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.337+0000 7f1982df9740 1 Processor -- start 2024-03-18T16:20:16.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.338+0000 7f1982df9740 1 -- start start 2024-03-18T16:20:16.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.338+0000 7f1982df9740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d2f93ea810 0x55d2f93f51c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:16.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.338+0000 7f1982df9740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d2f93eb200 0x55d2f93fc130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:16.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.338+0000 7f1982df9740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d2f93f0370 0x55d2f93e4630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:16.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.338+0000 7f1982df9740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f195800fce0 con 0x55d2f93ea810 2024-03-18T16:20:16.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.338+0000 7f1982df9740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f1958011850 con 0x55d2f93eb200 2024-03-18T16:20:16.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.338+0000 7f1982df9740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f19580103b0 con 0x55d2f93f0370 2024-03-18T16:20:16.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.339+0000 7f197cc00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d2f93eb200 0x55d2f93fc130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:16.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.339+0000 7f197cc00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d2f93eb200 0x55d2f93fc130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:58020/0 (socket says 172.21.15.39:58020) 2024-03-18T16:20:16.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.339+0000 7f197cc00640 1 -- 172.21.15.39:0/798719011 learned_addr learned my addr 172.21.15.39:0/798719011 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:20:16.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.339+0000 7f197e000640 1 --2- 172.21.15.39:0/798719011 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d2f93f0370 0x55d2f93e4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:16.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.340+0000 7f197e000640 1 -- 172.21.15.39:0/798719011 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d2f93eb200 msgr2=0x55d2f93fc130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:16.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.340+0000 7f197e000640 1 --2- 172.21.15.39:0/798719011 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d2f93eb200 0x55d2f93fc130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:16.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.340+0000 7f197e000640 1 -- 172.21.15.39:0/798719011 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d2f93ea810 msgr2=0x55d2f93f51c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-18T16:20:16.342 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.341+0000 7f197e000640 1 --2- 172.21.15.39:0/798719011 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d2f93ea810 0x55d2f93f51c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:16.343 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.341+0000 7f197e000640 1 -- 172.21.15.39:0/798719011 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d2f93e58c0 con 0x55d2f93f0370 2024-03-18T16:20:16.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.342+0000 7f197e000640 1 --2- 172.21.15.39:0/798719011 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d2f93f0370 0x55d2f93e4630 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f197000afd0 tx=0x7f19700104e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d16b7e228ab8578 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:16.345 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.344+0000 7f1969c00640 1 -- 172.21.15.39:0/798719011 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f197002c030 con 0x55d2f93f0370 2024-03-18T16:20:16.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.347+0000 7f1969c00640 1 -- 172.21.15.39:0/798719011 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1970004ab0 con 0x55d2f93f0370 2024-03-18T16:20:16.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.347+0000 7f1969c00640 1 -- 172.21.15.39:0/798719011 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1970017070 con 0x55d2f93f0370 2024-03-18T16:20:16.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.349+0000 7f1982df9740 1 -- 172.21.15.39:0/798719011 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d2f93f0370 msgr2=0x55d2f93e4630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:16.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.349+0000 7f1982df9740 1 --2- 172.21.15.39:0/798719011 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d2f93f0370 0x55d2f93e4630 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f197000afd0 tx=0x7f19700104e0 comp rx=0 tx=0).stop 2024-03-18T16:20:16.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.349+0000 7f1982df9740 1 -- 172.21.15.39:0/798719011 shutdown_connections 2024-03-18T16:20:16.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.349+0000 7f1982df9740 1 --2- 172.21.15.39:0/798719011 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d2f93eb200 0x55d2f93fc130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:16.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.349+0000 7f1982df9740 1 --2- 172.21.15.39:0/798719011 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d2f93f0370 0x55d2f93e4630 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:16.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.349+0000 7f1982df9740 1 --2- 172.21.15.39:0/798719011 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d2f93ea810 0x55d2f93f51c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:16.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.349+0000 7f1982df9740 1 -- 172.21.15.39:0/798719011 >> 172.21.15.39:0/798719011 conn(0x55d2f93e5d80 msgr2=0x55d2f93f4ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:16.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.350+0000 7f1982df9740 1 -- 172.21.15.39:0/798719011 shutdown_connections 2024-03-18T16:20:16.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.350+0000 7f1982df9740 1 -- 172.21.15.39:0/798719011 wait complete. 2024-03-18T16:20:16.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.351+0000 7f1982df9740 1 -- start start 2024-03-18T16:20:16.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.354+0000 7f1982df9740 1 Processor -- start 2024-03-18T16:20:16.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.355+0000 7f1982df9740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d2f93ea810 0x55d2f93f9da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:16.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.355+0000 7f1982df9740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d2f93eb200 0x55d2f93fa2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:16.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.356+0000 7f1982df9740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d2f93f0370 0x55d2f93fb2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:16.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.356+0000 7f1982df9740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f1958011150 con 0x55d2f93eb200 2024-03-18T16:20:16.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.356+0000 7f1982df9740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f1958010a80 con 0x55d2f93f0370 2024-03-18T16:20:16.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.356+0000 7f1982df9740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f195800dcc0 con 0x55d2f93ea810 2024-03-18T16:20:16.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.356+0000 7f1982df9740 10 client.0 _ll_register_callbacks cb 0x7f196c0035f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:20:16.358 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.356+0000 7f197d600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d2f93ea810 0x55d2f93f9da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:16.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.356+0000 7f197cc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d2f93eb200 0x55d2f93fa2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:16.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.356+0000 7f197d600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d2f93ea810 0x55d2f93f9da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:38254/0 (socket says 172.21.15.39:38254) 2024-03-18T16:20:16.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.356+0000 7f197d600640 1 -- 172.21.15.39:0/678333089 learned_addr learned my addr 172.21.15.39:0/678333089 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:20:16.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.356+0000 7f197e000640 1 --2- 172.21.15.39:0/678333089 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d2f93f0370 0x55d2f93fb2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:16.359 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.357+0000 7f197d600640 1 -- 172.21.15.39:0/678333089 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d2f93f0370 msgr2=0x55d2f93fb2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:16.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.357+0000 7f197d600640 1 --2- 172.21.15.39:0/678333089 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d2f93f0370 0x55d2f93fb2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:16.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.357+0000 7f197d600640 1 -- 172.21.15.39:0/678333089 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d2f93eb200 msgr2=0x55d2f93fa2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:16.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.358+0000 7f197d600640 1 --2- 172.21.15.39:0/678333089 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d2f93eb200 0x55d2f93fa2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:16.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.358+0000 7f197d600640 1 -- 172.21.15.39:0/678333089 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d2f93fc670 con 0x55d2f93ea810 2024-03-18T16:20:16.360 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.358+0000 7f197cc00640 1 --2- 172.21.15.39:0/678333089 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d2f93eb200 0x55d2f93fa2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:20:16.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.359+0000 7f197d600640 1 --2- 172.21.15.39:0/678333089 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d2f93ea810 0x55d2f93f9da0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f196c02c200 tx=0x7f196c005c90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e37cd666a3d0289c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:16.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.359+0000 7f195ea00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:20:16.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.359+0000 7f195ea00640 1 -- 172.21.15.39:0/678333089 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f196c0bd430 con 0x55d2f93ea810 2024-03-18T16:20:16.361 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.359+0000 7f1982df9740 1 -- 172.21.15.39:0/678333089 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1970004ab0 con 0x55d2f93ea810 2024-03-18T16:20:16.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.361+0000 7f195ea00640 1 -- 172.21.15.39:0/678333089 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f196c0bc280 con 0x55d2f93ea810 2024-03-18T16:20:16.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.361+0000 7f195ea00640 1 -- 172.21.15.39:0/678333089 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f196c0c1e10 con 0x55d2f93ea810 2024-03-18T16:20:16.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.362+0000 7f1982df9740 10 client.27733 fetch_fsmap learned FSMap version 47 2024-03-18T16:20:16.363 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.362+0000 7f1982df9740 1 -- 172.21.15.39:0/678333089 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55d2f93e0370 con 0x55d2f93ea810 2024-03-18T16:20:16.364 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.363+0000 7f195ea00640 1 -- 172.21.15.39:0/678333089 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1970004ab0 con 0x55d2f93ea810 2024-03-18T16:20:16.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.363+0000 7f195ea00640 1 -- 172.21.15.39:0/678333089 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 47) ==== 8374+0+0 (secure 0 0 0) 0x7f196c0bd110 con 0x55d2f93ea810 2024-03-18T16:20:16.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.363+0000 7f1982df9740 10 client.27733 fetch_fsmap finished waiting for FSMap version 47 2024-03-18T16:20:16.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.363+0000 7f1982df9740 10 client.27733 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:20:16.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.363+0000 7f1982df9740 20 client.27733 populate_metadata read hostname 'smithi039' 2024-03-18T16:20:16.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.363+0000 7f1982df9740 1 --2- 172.21.15.39:0/678333089 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55d2f94679b0 0x55d2f9469dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:16.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.363+0000 7f197cc00640 1 --2- 172.21.15.39:0/678333089 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55d2f94679b0 0x55d2f9469dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:16.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.363+0000 7f1982df9740 4 client.27733 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T16:20:16.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.363+0000 7f1982df9740 1 -- 172.21.15.39:0/678333089 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d2f935c730 con 0x55d2f94679b0 2024-03-18T16:20:16.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.364+0000 7f197cc00640 1 --2- 172.21.15.39:0/678333089 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55d2f94679b0 0x55d2f9469dd0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4b20ef9592201560 server_cookie=31d25b588b8d677c in_seq=0 out_seq=0 2024-03-18T16:20:16.369 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.365+0000 7f195ea00640 10 client.27733 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:20:16.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.368+0000 7f195ea00640 1 -- 172.21.15.39:0/678333089 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55d2f935c730 con 0x55d2f94679b0 2024-03-18T16:20:16.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.368+0000 7f195ea00640 10 client.27733 handle_command_reply: tid=0 2024-03-18T16:20:16.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.368+0000 7f1982df9740 1 client.27733 shutdown 2024-03-18T16:20:16.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.368+0000 7f1982df9740 1 -- 172.21.15.39:0/678333089 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d2f93ea810 msgr2=0x55d2f93f9da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:16.370 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.368+0000 7f1982df9740 1 --2- 172.21.15.39:0/678333089 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d2f93ea810 0x55d2f93f9da0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f196c02c200 tx=0x7f196c005c90 comp rx=0 tx=0).stop 2024-03-18T16:20:16.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.370+0000 7f1982df9740 1 -- 172.21.15.39:0/678333089 shutdown_connections 2024-03-18T16:20:16.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.370+0000 7f1982df9740 1 --2- 172.21.15.39:0/678333089 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55d2f94679b0 0x55d2f9469dd0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:16.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.370+0000 7f1982df9740 1 --2- 172.21.15.39:0/678333089 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d2f93f0370 0x55d2f93fb2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:16.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.370+0000 7f1982df9740 1 --2- 172.21.15.39:0/678333089 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55d2f93eb200 0x55d2f93fa2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:16.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.370+0000 7f1982df9740 1 --2- 172.21.15.39:0/678333089 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55d2f93ea810 0x55d2f93f9da0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:16.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.370+0000 7f195ea00640 0 client.27733 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:20:16.372 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.370+0000 7f1982df9740 1 -- 172.21.15.39:0/678333089 >> 172.21.15.39:0/678333089 conn(0x55d2f93e5d80 msgr2=0x55d2f93e4630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:16.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.371+0000 7f1982df9740 1 -- 172.21.15.39:0/678333089 shutdown_connections 2024-03-18T16:20:16.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.371+0000 7f1982df9740 1 -- 172.21.15.39:0/678333089 wait complete. 2024-03-18T16:20:16.373 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.371+0000 7f1982df9740 20 client.27733 trim_cache size 0 max 16384 2024-03-18T16:20:16.374 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-03-18T16:20:16.374 INFO:teuthology.orchestra.run.smithi039.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-18T16:20:16.374 INFO:teuthology.orchestra.run.smithi039.stdout: "scrubs": { 2024-03-18T16:20:16.374 INFO:teuthology.orchestra.run.smithi039.stdout: "063a4e60-b3a3-4161-8315-0308493ac4f5": { 2024-03-18T16:20:16.374 INFO:teuthology.orchestra.run.smithi039.stdout: "path": "/", 2024-03-18T16:20:16.374 INFO:teuthology.orchestra.run.smithi039.stdout: "tag": "063a4e60-b3a3-4161-8315-0308493ac4f5", 2024-03-18T16:20:16.374 INFO:teuthology.orchestra.run.smithi039.stdout: "options": "recursive,force" 2024-03-18T16:20:16.374 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:16.374 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:16.374 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-03-18T16:20:16.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.375+0000 7f1980000640 1 -- 172.21.15.39:0/2990878119 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f19600a2b00 msgr2=0x7f19600a4fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:16.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.375+0000 7f1980000640 1 --2- 172.21.15.39:0/2990878119 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f19600a2b00 0x7f19600a4fc0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f197000ace0 tx=0x7f197000a2b0 comp rx=0 tx=0).stop 2024-03-18T16:20:16.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.375+0000 7f1980000640 1 -- 172.21.15.39:0/2990878119 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f19780741e0 msgr2=0x7f1978195890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:16.377 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.375+0000 7f1980000640 1 --2- 172.21.15.39:0/2990878119 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f19780741e0 0x7f1978195890 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f196c000c00 tx=0x7f196c005dd0 comp rx=0 tx=0).stop 2024-03-18T16:20:16.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.375+0000 7f1980000640 1 -- 172.21.15.39:0/2990878119 shutdown_connections 2024-03-18T16:20:16.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.375+0000 7f1980000640 1 --2- 172.21.15.39:0/2990878119 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f19781096a0 0x7f197819ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:16.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.375+0000 7f1980000640 1 --2- 172.21.15.39:0/2990878119 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f19600a2b00 0x7f19600a4fc0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:16.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.375+0000 7f1980000640 1 --2- 172.21.15.39:0/2990878119 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f19780757c0 0x7f1978195dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:16.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.375+0000 7f1980000640 1 --2- 172.21.15.39:0/2990878119 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f19780741e0 0x7f1978195890 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:16.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.375+0000 7f1980000640 1 -- 172.21.15.39:0/2990878119 >> 172.21.15.39:0/2990878119 conn(0x7f19780fcd80 msgr2=0x7f1978101220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:16.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.376+0000 7f1980000640 1 -- 172.21.15.39:0/2990878119 shutdown_connections 2024-03-18T16:20:16.378 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:16.376+0000 7f1980000640 1 -- 172.21.15.39:0/2990878119 wait complete. 2024-03-18T16:20:16.405 INFO:tasks.cephfs.filesystem:scrub status for tag:063a4e60-b3a3-4161-8315-0308493ac4f5 - {'path': '/', 'tag': '063a4e60-b3a3-4161-8315-0308493ac4f5', 'options': 'recursive,force'} 2024-03-18T16:20:16.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:16 smithi196 ceph-mon[36307]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T16:20:16.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:16 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:20:16.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:16 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:20:16.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:16 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:20:16.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:16 smithi196 ceph-mon[36307]: pgmap v728: 129 pgs: 129 active+clean; 12 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 456 KiB/s wr, 134 op/s 2024-03-18T16:20:16.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:16 smithi194 ceph-mon[36200]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T16:20:16.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:16 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:20:16.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:16 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:20:16.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:16 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:20:16.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:16 smithi194 ceph-mon[36200]: pgmap v728: 129 pgs: 129 active+clean; 12 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 25 KiB/s rd, 456 KiB/s wr, 134 op/s 2024-03-18T16:20:17.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:17 smithi196 ceph-mon[36307]: scrub queued for path: / 2024-03-18T16:20:17.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:17 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:20:17.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:17 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:20:17.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:17 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:20:17.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:17 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:20:17.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:17 smithi196 ceph-mon[36307]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T16:20:17.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:17 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:20:17.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:17 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:20:17.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:17 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:20:17.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:17 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:20:17.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:17 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:20:17.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:17 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:20:17.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:17 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:20:17.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:17 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:20:17.729 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:17 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:20:17.730 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:17 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:20:17.730 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:17 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:20:17.730 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:17 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:20:17.730 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:17 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:20:17.730 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:17 smithi196 ceph-mon[36307]: scrub summary: active paths [/] 2024-03-18T16:20:17.730 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:17 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:20:17.730 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:17 smithi196 ceph-mon[36307]: scrub summary: idle+waiting paths [/] 2024-03-18T16:20:17.739 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:17 smithi039 ceph-mon[31721]: scrub queued for path: / 2024-03-18T16:20:17.739 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:17 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:20:17.740 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:17 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:20:17.740 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:17 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:20:17.740 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:17 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:20:17.740 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:17 smithi039 ceph-mon[31721]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T16:20:17.740 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:17 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:20:17.740 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:17 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:20:17.740 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:17 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:20:17.740 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:17 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:20:17.740 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:17 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:20:17.740 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:17 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:20:17.740 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:17 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:20:17.740 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:17 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:20:17.740 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:17 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:20:17.740 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:17 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:20:17.740 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:17 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:20:17.740 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:17 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:20:17.740 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:17 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:20:17.741 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:17 smithi039 ceph-mon[31721]: scrub summary: active paths [/] 2024-03-18T16:20:17.741 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:17 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:20:17.741 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:17 smithi039 ceph-mon[31721]: scrub summary: idle+waiting paths [/] 2024-03-18T16:20:17.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:17 smithi194 ceph-mon[36200]: scrub queued for path: / 2024-03-18T16:20:17.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:17 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:20:17.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:17 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:20:17.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:17 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:20:17.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:17 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:20:17.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:17 smithi194 ceph-mon[36200]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-18T16:20:17.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:17 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:20:17.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:17 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:20:17.742 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:17 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:20:17.742 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:17 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:20:17.742 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:17 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:20:17.742 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:17 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:20:17.742 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:17 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:20:17.742 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:17 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:20:17.742 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:17 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:20:17.742 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:17 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:20:17.742 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:17 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:20:17.742 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:17 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:20:17.742 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:17 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:20:17.742 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:17 smithi194 ceph-mon[36200]: scrub summary: active paths [/] 2024-03-18T16:20:17.742 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:17 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:20:17.742 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:17 smithi194 ceph-mon[36200]: scrub summary: idle+waiting paths [/] 2024-03-18T16:20:19.090 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:18 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:20:19.090 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:18 smithi039 ceph-mon[31721]: pgmap v729: 129 pgs: 129 active+clean; 9.5 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 430 KiB/s wr, 171 op/s 2024-03-18T16:20:19.090 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:18 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:20:19.091 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:18 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:20:19.091 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:18 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:20:19.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:18 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:20:19.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:18 smithi196 ceph-mon[36307]: pgmap v729: 129 pgs: 129 active+clean; 9.5 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 430 KiB/s wr, 171 op/s 2024-03-18T16:20:19.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:18 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:20:19.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:18 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:20:19.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:18 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:20:19.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:18 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:20:19.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:18 smithi194 ceph-mon[36200]: pgmap v729: 129 pgs: 129 active+clean; 9.5 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 430 KiB/s wr, 171 op/s 2024-03-18T16:20:19.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:18 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:20:19.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:18 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:20:19.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:18 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:20:20.205 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-03-18T16:20:20.205 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-03-18T16:20:20.470 INFO:tasks.workunit:Stopping ['suites/ffsb.sh'] on client.0... 2024-03-18T16:20:20.471 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-18T16:20:20.909 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:20 smithi039 ceph-mon[31721]: pgmap v730: 129 pgs: 129 active+clean; 8.6 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 323 KiB/s wr, 163 op/s 2024-03-18T16:20:20.909 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:20 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:20:20.909 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:20 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:20:21.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:20 smithi196 ceph-mon[36307]: pgmap v730: 129 pgs: 129 active+clean; 8.6 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 323 KiB/s wr, 163 op/s 2024-03-18T16:20:21.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:20 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:20:21.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:20 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:20:21.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:20 smithi194 ceph-mon[36200]: pgmap v730: 129 pgs: 129 active+clean; 8.6 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 323 KiB/s wr, 163 op/s 2024-03-18T16:20:21.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:20 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:20:21.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:20 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:20:22.304 DEBUG:teuthology.parallel:result is None 2024-03-18T16:20:22.305 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-03-18T16:20:22.460 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-03-18T16:20:22.460 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-03-18T16:20:22.470 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-03-18T16:20:23.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:22 smithi196 ceph-mon[36307]: pgmap v731: 129 pgs: 129 active+clean; 6.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 273 KiB/s wr, 198 op/s 2024-03-18T16:20:23.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:22 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:20:23.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:22 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:20:23.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:22 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:20:23.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:22 smithi194 ceph-mon[36200]: pgmap v731: 129 pgs: 129 active+clean; 6.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 273 KiB/s wr, 198 op/s 2024-03-18T16:20:23.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:22 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:20:23.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:22 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:20:23.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:22 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:20:23.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:22 smithi039 ceph-mon[31721]: pgmap v731: 129 pgs: 129 active+clean; 6.5 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 273 KiB/s wr, 198 op/s 2024-03-18T16:20:23.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:22 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:20:23.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:22 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:20:23.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:22 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:20:25.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:24 smithi196 ceph-mon[36307]: pgmap v732: 129 pgs: 129 active+clean; 5.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 245 KiB/s wr, 217 op/s 2024-03-18T16:20:25.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:24 smithi194 ceph-mon[36200]: pgmap v732: 129 pgs: 129 active+clean; 5.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 245 KiB/s wr, 217 op/s 2024-03-18T16:20:25.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:24 smithi039 ceph-mon[31721]: pgmap v732: 129 pgs: 129 active+clean; 5.1 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 245 KiB/s wr, 217 op/s 2024-03-18T16:20:27.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:26 smithi196 ceph-mon[36307]: pgmap v733: 129 pgs: 129 active+clean; 4.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 127 KiB/s wr, 219 op/s 2024-03-18T16:20:27.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:26 smithi194 ceph-mon[36200]: pgmap v733: 129 pgs: 129 active+clean; 4.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 127 KiB/s wr, 219 op/s 2024-03-18T16:20:27.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:26 smithi039 ceph-mon[31721]: pgmap v733: 129 pgs: 129 active+clean; 4.0 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 127 KiB/s wr, 219 op/s 2024-03-18T16:20:28.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:27 smithi196 ceph-mon[36307]: scrub summary: idle 2024-03-18T16:20:28.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:27 smithi196 ceph-mon[36307]: scrub summary: idle 2024-03-18T16:20:28.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:27 smithi194 ceph-mon[36200]: scrub summary: idle 2024-03-18T16:20:28.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:27 smithi194 ceph-mon[36200]: scrub summary: idle 2024-03-18T16:20:28.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:27 smithi039 ceph-mon[31721]: scrub summary: idle 2024-03-18T16:20:28.425 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:27 smithi039 ceph-mon[31721]: scrub summary: idle 2024-03-18T16:20:29.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:28 smithi196 ceph-mon[36307]: scrub summary: idle 2024-03-18T16:20:29.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:28 smithi196 ceph-mon[36307]: pgmap v734: 129 pgs: 129 active+clean; 1.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 115 KiB/s wr, 272 op/s 2024-03-18T16:20:29.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:28 smithi194 ceph-mon[36200]: scrub summary: idle 2024-03-18T16:20:29.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:28 smithi194 ceph-mon[36200]: pgmap v734: 129 pgs: 129 active+clean; 1.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 115 KiB/s wr, 272 op/s 2024-03-18T16:20:29.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:28 smithi039 ceph-mon[31721]: scrub summary: idle 2024-03-18T16:20:29.352 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:28 smithi039 ceph-mon[31721]: pgmap v734: 129 pgs: 129 active+clean; 1.0 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 151 KiB/s rd, 115 KiB/s wr, 272 op/s 2024-03-18T16:20:31.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:30 smithi196 ceph-mon[36307]: pgmap v735: 129 pgs: 129 active+clean; 437 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 95 KiB/s wr, 239 op/s 2024-03-18T16:20:31.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:30 smithi194 ceph-mon[36200]: pgmap v735: 129 pgs: 129 active+clean; 437 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 95 KiB/s wr, 239 op/s 2024-03-18T16:20:31.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:30 smithi039 ceph-mon[31721]: pgmap v735: 129 pgs: 129 active+clean; 437 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 95 KiB/s wr, 239 op/s 2024-03-18T16:20:33.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:32 smithi196 ceph-mon[36307]: pgmap v736: 129 pgs: 129 active+clean; 20 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 55 KiB/s wr, 224 op/s 2024-03-18T16:20:33.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:32 smithi194 ceph-mon[36200]: pgmap v736: 129 pgs: 129 active+clean; 20 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 55 KiB/s wr, 224 op/s 2024-03-18T16:20:33.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:32 smithi039 ceph-mon[31721]: pgmap v736: 129 pgs: 129 active+clean; 20 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 55 KiB/s wr, 224 op/s 2024-03-18T16:20:35.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:34 smithi196 ceph-mon[36307]: pgmap v737: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 55 KiB/s wr, 176 op/s 2024-03-18T16:20:35.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:34 smithi194 ceph-mon[36200]: pgmap v737: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 55 KiB/s wr, 176 op/s 2024-03-18T16:20:35.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:34 smithi039 ceph-mon[31721]: pgmap v737: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 55 KiB/s wr, 176 op/s 2024-03-18T16:20:37.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:36 smithi196 ceph-mon[36307]: pgmap v738: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 2.4 KiB/s wr, 137 op/s 2024-03-18T16:20:37.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:36 smithi194 ceph-mon[36200]: pgmap v738: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 2.4 KiB/s wr, 137 op/s 2024-03-18T16:20:37.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:36 smithi039 ceph-mon[31721]: pgmap v738: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 2.4 KiB/s wr, 137 op/s 2024-03-18T16:20:39.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:38 smithi194 ceph-mon[36200]: pgmap v739: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 2.4 KiB/s wr, 112 op/s 2024-03-18T16:20:39.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:38 smithi039 ceph-mon[31721]: pgmap v739: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 2.4 KiB/s wr, 112 op/s 2024-03-18T16:20:39.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:38 smithi196 ceph-mon[36307]: pgmap v739: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 2.4 KiB/s wr, 112 op/s 2024-03-18T16:20:41.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:40 smithi194 ceph-mon[36200]: pgmap v740: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 2.0 KiB/s wr, 40 op/s 2024-03-18T16:20:41.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:40 smithi039 ceph-mon[31721]: pgmap v740: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 2.0 KiB/s wr, 40 op/s 2024-03-18T16:20:41.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:40 smithi196 ceph-mon[36307]: pgmap v740: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 2.0 KiB/s wr, 40 op/s 2024-03-18T16:20:43.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:42 smithi039 ceph-mon[31721]: pgmap v741: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1023 B/s wr, 23 op/s 2024-03-18T16:20:43.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:42 smithi196 ceph-mon[36307]: pgmap v741: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1023 B/s wr, 23 op/s 2024-03-18T16:20:43.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:42 smithi194 ceph-mon[36200]: pgmap v741: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1023 B/s wr, 23 op/s 2024-03-18T16:20:45.292 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:45 smithi194 ceph-mon[36200]: pgmap v742: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.4 KiB/s wr, 10 op/s 2024-03-18T16:20:45.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:45 smithi039 ceph-mon[31721]: pgmap v742: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.4 KiB/s wr, 10 op/s 2024-03-18T16:20:45.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:45 smithi196 ceph-mon[36307]: pgmap v742: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1.4 KiB/s wr, 10 op/s 2024-03-18T16:20:46.406 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-18T16:20:46.407 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-18T16:20:46.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.649+0000 7fa206600640 1 -- 172.21.15.39:0/316652143 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa200104a70 msgr2=0x7fa200106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:46.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.649+0000 7fa206600640 1 --2- 172.21.15.39:0/316652143 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa200104a70 0x7fa200106e60 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec009900 tx=0x7fa1ec02bc30 comp rx=0 tx=0).stop 2024-03-18T16:20:46.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.649+0000 7fa206600640 1 -- 172.21.15.39:0/316652143 shutdown_connections 2024-03-18T16:20:46.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.649+0000 7fa206600640 1 --2- 172.21.15.39:0/316652143 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa2001073a0 0x7fa2001097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:46.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.649+0000 7fa206600640 1 --2- 172.21.15.39:0/316652143 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa200104a70 0x7fa200106e60 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:46.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.649+0000 7fa206600640 1 --2- 172.21.15.39:0/316652143 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa200102140 0x7fa200104530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:46.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.649+0000 7fa206600640 1 -- 172.21.15.39:0/316652143 >> 172.21.15.39:0/316652143 conn(0x7fa2000fbb50 msgr2=0x7fa2000fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:46.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.649+0000 7fa206600640 1 -- 172.21.15.39:0/316652143 shutdown_connections 2024-03-18T16:20:46.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.649+0000 7fa206600640 1 -- 172.21.15.39:0/316652143 wait complete. 2024-03-18T16:20:46.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.650+0000 7fa206600640 1 Processor -- start 2024-03-18T16:20:46.653 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.651+0000 7fa206600640 1 -- start start 2024-03-18T16:20:46.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.651+0000 7fa206600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa200102140 0x7fa200101ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:46.654 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.651+0000 7fa206600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa200104a70 0x7fa2001000f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:46.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.651+0000 7fa206600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa2001073a0 0x7fa200100630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:46.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.651+0000 7fa206600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fa20010d3a0 con 0x7fa200104a70 2024-03-18T16:20:46.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.651+0000 7fa206600640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fa20010d220 con 0x7fa2001073a0 2024-03-18T16:20:46.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.651+0000 7fa206600640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fa20010d520 con 0x7fa200102140 2024-03-18T16:20:46.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.651+0000 7fa205200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa200102140 0x7fa200101ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:46.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.651+0000 7fa205200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa200102140 0x7fa200101ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:45002/0 (socket says 172.21.15.39:45002) 2024-03-18T16:20:46.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.651+0000 7fa205200640 1 -- 172.21.15.39:0/4202170725 learned_addr learned my addr 172.21.15.39:0/4202170725 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:20:46.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.651+0000 7fa205c00640 1 --2- 172.21.15.39:0/4202170725 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa2001073a0 0x7fa200100630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:46.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.651+0000 7fa1ffe00640 1 --2- 172.21.15.39:0/4202170725 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa200104a70 0x7fa2001000f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:46.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.652+0000 7fa205200640 1 -- 172.21.15.39:0/4202170725 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa2001073a0 msgr2=0x7fa200100630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:46.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.652+0000 7fa205200640 1 --2- 172.21.15.39:0/4202170725 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa2001073a0 0x7fa200100630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:46.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.652+0000 7fa205200640 1 -- 172.21.15.39:0/4202170725 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa200104a70 msgr2=0x7fa2001000f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:46.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.652+0000 7fa205200640 1 --2- 172.21.15.39:0/4202170725 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa200104a70 0x7fa2001000f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:46.655 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.652+0000 7fa205200640 1 -- 172.21.15.39:0/4202170725 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa200100ec0 con 0x7fa200102140 2024-03-18T16:20:46.656 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.652+0000 7fa205200640 1 --2- 172.21.15.39:0/4202170725 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa200102140 0x7fa200101ab0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fa1f400e9f0 tx=0x7fa1f400edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:46.656 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.652+0000 7fa1fd600640 1 -- 172.21.15.39:0/4202170725 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1f400cd80 con 0x7fa200102140 2024-03-18T16:20:46.656 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.652+0000 7fa1fd600640 1 -- 172.21.15.39:0/4202170725 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa1f4005500 con 0x7fa200102140 2024-03-18T16:20:46.656 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.652+0000 7fa1fd600640 1 -- 172.21.15.39:0/4202170725 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1f4010690 con 0x7fa200102140 2024-03-18T16:20:46.656 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.652+0000 7fa206600640 1 -- 172.21.15.39:0/4202170725 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2001a1b00 con 0x7fa200102140 2024-03-18T16:20:46.656 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.653+0000 7fa206600640 1 -- 172.21.15.39:0/4202170725 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2001a2020 con 0x7fa200102140 2024-03-18T16:20:46.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.656+0000 7fa1fd600640 1 -- 172.21.15.39:0/4202170725 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fa1f40046c0 con 0x7fa200102140 2024-03-18T16:20:46.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.656+0000 7fa20a9b9740 1 Processor -- start 2024-03-18T16:20:46.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.656+0000 7fa20a9b9740 1 -- start start 2024-03-18T16:20:46.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.657+0000 7fa20a9b9740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f7632caa10 0x55f7632cae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:46.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.657+0000 7fa20a9b9740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f7632cb400 0x55f7632cff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:46.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.657+0000 7fa20a9b9740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f7632d0570 0x55f7632d2a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:46.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.657+0000 7fa20a9b9740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55f76322f1f0 con 0x55f7632d0570 2024-03-18T16:20:46.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.657+0000 7fa20a9b9740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55f7631d7a80 con 0x55f7632cb400 2024-03-18T16:20:46.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.657+0000 7fa20a9b9740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55f76322b290 con 0x55f7632caa10 2024-03-18T16:20:46.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.657+0000 7fa205c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f7632caa10 0x55f7632cae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:46.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.657+0000 7fa205200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f7632cb400 0x55f7632cff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:46.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.657+0000 7fa1ffe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f7632d0570 0x55f7632d2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:46.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.657+0000 7fa205c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f7632caa10 0x55f7632cae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:45018/0 (socket says 172.21.15.39:45018) 2024-03-18T16:20:46.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.657+0000 7fa205c00640 1 -- 172.21.15.39:0/311219943 learned_addr learned my addr 172.21.15.39:0/311219943 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:20:46.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.657+0000 7fa205c00640 1 -- 172.21.15.39:0/311219943 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f7632cb400 msgr2=0x55f7632cff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:46.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.657+0000 7fa205c00640 1 --2- 172.21.15.39:0/311219943 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f7632cb400 0x55f7632cff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:46.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.657+0000 7fa205c00640 1 -- 172.21.15.39:0/311219943 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f7632d0570 msgr2=0x55f7632d2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:46.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.657+0000 7fa205c00640 1 --2- 172.21.15.39:0/311219943 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f7632d0570 0x55f7632d2a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:46.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.657+0000 7fa205c00640 1 -- 172.21.15.39:0/311219943 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f76300d7d0 con 0x55f7632caa10 2024-03-18T16:20:46.661 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.658+0000 7fa205c00640 1 --2- 172.21.15.39:0/311219943 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f7632caa10 0x55f7632cae30 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fa1f8009fd0 tx=0x7fa1f80118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d9681c611ea1882e server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:46.665 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.658+0000 7fa1f2000640 1 -- 172.21.15.39:0/311219943 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1f8004d20 con 0x55f7632caa10 2024-03-18T16:20:46.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.658+0000 7fa1f2000640 1 -- 172.21.15.39:0/311219943 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa1f8004050 con 0x55f7632caa10 2024-03-18T16:20:46.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.658+0000 7fa1f2000640 1 -- 172.21.15.39:0/311219943 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1f8005240 con 0x55f7632caa10 2024-03-18T16:20:46.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.658+0000 7fa1fd600640 1 --2- 172.21.15.39:0/4202170725 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa1e80a2880 0x7fa1e80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:46.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.658+0000 7fa1fd600640 1 -- 172.21.15.39:0/4202170725 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fa1f40cf2a0 con 0x7fa200102140 2024-03-18T16:20:46.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.658+0000 7fa205200640 1 --2- 172.21.15.39:0/4202170725 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa1e80a2880 0x7fa1e80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:46.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.659+0000 7fa20a9b9740 1 -- 172.21.15.39:0/311219943 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f7632caa10 msgr2=0x55f7632cae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:46.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.659+0000 7fa20a9b9740 1 --2- 172.21.15.39:0/311219943 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f7632caa10 0x55f7632cae30 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fa1f8009fd0 tx=0x7fa1f80118d0 comp rx=0 tx=0).stop 2024-03-18T16:20:46.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.659+0000 7fa20a9b9740 1 -- 172.21.15.39:0/311219943 shutdown_connections 2024-03-18T16:20:46.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.659+0000 7fa20a9b9740 1 --2- 172.21.15.39:0/311219943 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f7632cb400 0x55f7632cff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:46.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.659+0000 7fa20a9b9740 1 --2- 172.21.15.39:0/311219943 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f7632d0570 0x55f7632d2a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:46.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.659+0000 7fa20a9b9740 1 --2- 172.21.15.39:0/311219943 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f7632caa10 0x55f7632cae30 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:46.702 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.659+0000 7fa20a9b9740 1 -- 172.21.15.39:0/311219943 >> 172.21.15.39:0/311219943 conn(0x55f7632c5f80 msgr2=0x55f7632c6380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:46.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.659+0000 7fa20a9b9740 1 -- 172.21.15.39:0/311219943 shutdown_connections 2024-03-18T16:20:46.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.659+0000 7fa20a9b9740 1 -- 172.21.15.39:0/311219943 wait complete. 2024-03-18T16:20:46.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.659+0000 7fa205200640 1 --2- 172.21.15.39:0/4202170725 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa1e80a2880 0x7fa1e80a4d40 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fa1f40a3ac0 tx=0x7fa1f40d2ce0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:46.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.659+0000 7fa20a9b9740 1 -- start start 2024-03-18T16:20:46.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.660+0000 7fa20a9b9740 1 Processor -- start 2024-03-18T16:20:46.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.660+0000 7fa20a9b9740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f7632caa10 0x55f7632d4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:46.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.660+0000 7fa20a9b9740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f7632cb400 0x55f7632d5430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:46.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.660+0000 7fa20a9b9740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f7632d0570 0x55f7632dc470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:46.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.660+0000 7fa20a9b9740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55f7632d3520 con 0x55f7632d0570 2024-03-18T16:20:46.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.660+0000 7fa20a9b9740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55f7631eaed0 con 0x55f7632cb400 2024-03-18T16:20:46.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.660+0000 7fa20a9b9740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55f7632d36a0 con 0x55f7632caa10 2024-03-18T16:20:46.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.660+0000 7fa20a9b9740 10 client.0 _ll_register_callbacks cb 0x55f763226220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:20:46.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.661+0000 7fa1ffe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f7632cb400 0x55f7632d5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:46.703 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.661+0000 7fa1ffe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f7632cb400 0x55f7632d5430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:53342/0 (socket says 172.21.15.39:53342) 2024-03-18T16:20:46.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.661+0000 7fa1ffe00640 1 -- 172.21.15.39:0/1702577082 learned_addr learned my addr 172.21.15.39:0/1702577082 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:20:46.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.661+0000 7fa205c00640 1 --2- 172.21.15.39:0/1702577082 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f7632d0570 0x55f7632dc470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:46.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.661+0000 7fa205c00640 1 --2- 172.21.15.39:0/1702577082 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f7632caa10 0x55f7632d4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:46.717 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.661+0000 7fa1ffe00640 1 -- 172.21.15.39:0/1702577082 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f7632caa10 msgr2=0x55f7632d4ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:46.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.661+0000 7fa1ffe00640 1 --2- 172.21.15.39:0/1702577082 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f7632caa10 0x55f7632d4ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:46.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.661+0000 7fa1ffe00640 1 -- 172.21.15.39:0/1702577082 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f7632d0570 msgr2=0x55f7632dc470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:46.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.661+0000 7fa1ffe00640 1 --2- 172.21.15.39:0/1702577082 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f7632d0570 0x55f7632dc470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:46.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.661+0000 7fa1ffe00640 1 -- 172.21.15.39:0/1702577082 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f762f433f0 con 0x55f7632cb400 2024-03-18T16:20:46.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.661+0000 7fa205c00640 1 --2- 172.21.15.39:0/1702577082 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f7632d0570 0x55f7632dc470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:20:46.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.661+0000 7fa1ffe00640 1 --2- 172.21.15.39:0/1702577082 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f7632cb400 0x55f7632d5430 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec005ab0 tx=0x7fa1ec004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f2e340a47fe24d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:46.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.661+0000 7fa1e6a00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:20:46.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.662+0000 7fa1e6a00640 1 -- 172.21.15.39:0/1702577082 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1ec005d50 con 0x55f7632cb400 2024-03-18T16:20:46.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.662+0000 7fa20a9b9740 1 -- 172.21.15.39:0/1702577082 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa1f8004050 con 0x55f7632cb400 2024-03-18T16:20:46.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.662+0000 7fa1e6a00640 1 -- 172.21.15.39:0/1702577082 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fa1ec005220 con 0x55f7632cb400 2024-03-18T16:20:46.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.662+0000 7fa1e6a00640 1 -- 172.21.15.39:0/1702577082 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7fa1ec050050 con 0x55f7632cb400 2024-03-18T16:20:46.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.662+0000 7fa20a9b9740 10 client.27755 fetch_fsmap learned FSMap version 48 2024-03-18T16:20:46.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.662+0000 7fa20a9b9740 1 -- 172.21.15.39:0/1702577082 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x55f762f030f0 con 0x55f7632cb400 2024-03-18T16:20:46.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.662+0000 7fa1e6a00640 1 -- 172.21.15.39:0/1702577082 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa1f8004050 con 0x55f7632cb400 2024-03-18T16:20:46.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.662+0000 7fa1e6a00640 1 -- 172.21.15.39:0/1702577082 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 48) ==== 8386+0+0 (secure 0 0 0) 0x7fa1ec0035f0 con 0x55f7632cb400 2024-03-18T16:20:46.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.663+0000 7fa20a9b9740 10 client.27755 fetch_fsmap finished waiting for FSMap version 48 2024-03-18T16:20:46.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.663+0000 7fa20a9b9740 10 client.27755 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:20:46.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.663+0000 7fa20a9b9740 20 client.27755 populate_metadata read hostname 'smithi039' 2024-03-18T16:20:46.752 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.663+0000 7fa20a9b9740 1 --2- 172.21.15.39:0/1702577082 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55f7632e0d00 0x55f7632e3160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:46.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.663+0000 7fa20a9b9740 4 client.27755 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:20:46.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.663+0000 7fa20a9b9740 1 -- 172.21.15.39:0/1702577082 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f763242be0 con 0x55f7632e0d00 2024-03-18T16:20:46.753 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.663+0000 7fa205c00640 1 --2- 172.21.15.39:0/1702577082 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55f7632e0d00 0x55f7632e3160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:46.759 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.663+0000 7fa205c00640 1 --2- 172.21.15.39:0/1702577082 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55f7632e0d00 0x55f7632e3160 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1de07e9d03af50ae server_cookie=134f53662c9ba2f1 in_seq=0 out_seq=0 2024-03-18T16:20:46.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.663+0000 7fa1e6a00640 10 client.27755 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:20:46.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.664+0000 7fa1e6a00640 1 -- 172.21.15.39:0/1702577082 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55f763242be0 con 0x55f7632e0d00 2024-03-18T16:20:46.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.664+0000 7fa1e6a00640 10 client.27755 handle_command_reply: tid=0 2024-03-18T16:20:46.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.664+0000 7fa20a9b9740 1 client.27755 shutdown 2024-03-18T16:20:46.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.664+0000 7fa20a9b9740 1 -- 172.21.15.39:0/1702577082 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f7632cb400 msgr2=0x55f7632d5430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:46.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.664+0000 7fa20a9b9740 1 --2- 172.21.15.39:0/1702577082 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f7632cb400 0x55f7632d5430 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec005ab0 tx=0x7fa1ec004270 comp rx=0 tx=0).stop 2024-03-18T16:20:46.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.665+0000 7fa20a9b9740 1 -- 172.21.15.39:0/1702577082 shutdown_connections 2024-03-18T16:20:46.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.665+0000 7fa20a9b9740 1 --2- 172.21.15.39:0/1702577082 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55f7632e0d00 0x55f7632e3160 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:46.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.665+0000 7fa20a9b9740 1 --2- 172.21.15.39:0/1702577082 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f7632cb400 0x55f7632d5430 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:46.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.665+0000 7fa1e6a00640 0 client.27755 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:20:46.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.665+0000 7fa20a9b9740 1 --2- 172.21.15.39:0/1702577082 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f7632d0570 0x55f7632dc470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:46.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.665+0000 7fa20a9b9740 1 --2- 172.21.15.39:0/1702577082 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f7632caa10 0x55f7632d4ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:46.760 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.665+0000 7fa20a9b9740 1 -- 172.21.15.39:0/1702577082 >> 172.21.15.39:0/1702577082 conn(0x55f7632c5f80 msgr2=0x55f7632c5110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:46.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.665+0000 7fa20a9b9740 1 -- 172.21.15.39:0/1702577082 shutdown_connections 2024-03-18T16:20:46.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.665+0000 7fa20a9b9740 1 -- 172.21.15.39:0/1702577082 wait complete. 2024-03-18T16:20:46.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.665+0000 7fa20a9b9740 20 client.27755 trim_cache size 0 max 16384 2024-03-18T16:20:46.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.693+0000 7fa20a9b9740 1 Processor -- start 2024-03-18T16:20:46.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.694+0000 7fa20a9b9740 1 -- start start 2024-03-18T16:20:46.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.694+0000 7fa20a9b9740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f7632caa10 0x55f7632c6360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:46.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.694+0000 7fa20a9b9740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f7632cb400 0x55f7632dc430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:46.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.694+0000 7fa20a9b9740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f7632d0570 0x55f7632c4880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:46.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.694+0000 7fa20a9b9740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fa2001ac440 con 0x55f7632d0570 2024-03-18T16:20:46.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.694+0000 7fa20a9b9740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fa2001ae040 con 0x55f7632caa10 2024-03-18T16:20:46.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.694+0000 7fa20a9b9740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fa2001acb10 con 0x55f7632cb400 2024-03-18T16:20:46.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.694+0000 7fa1ffe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f7632cb400 0x55f7632dc430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:46.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.694+0000 7fa1ffe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f7632cb400 0x55f7632dc430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:45036/0 (socket says 172.21.15.39:45036) 2024-03-18T16:20:46.761 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.694+0000 7fa1ffe00640 1 -- 172.21.15.39:0/280761864 learned_addr learned my addr 172.21.15.39:0/280761864 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:20:46.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.694+0000 7fa205c00640 1 --2- 172.21.15.39:0/280761864 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f7632caa10 0x55f7632c6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:46.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.695+0000 7fa205c00640 1 --2- 172.21.15.39:0/280761864 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f7632d0570 0x55f7632c4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:46.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.695+0000 7fa1ffe00640 1 -- 172.21.15.39:0/280761864 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f7632caa10 msgr2=0x55f7632c6360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:46.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.695+0000 7fa1ffe00640 1 --2- 172.21.15.39:0/280761864 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f7632caa10 0x55f7632c6360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:46.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.695+0000 7fa1ffe00640 1 -- 172.21.15.39:0/280761864 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f7632d0570 msgr2=0x55f7632c4880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:46.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.695+0000 7fa1ffe00640 1 --2- 172.21.15.39:0/280761864 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f7632d0570 0x55f7632c4880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:46.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.695+0000 7fa1ffe00640 1 -- 172.21.15.39:0/280761864 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f7632cbc90 con 0x55f7632cb400 2024-03-18T16:20:46.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.695+0000 7fa205c00640 1 --2- 172.21.15.39:0/280761864 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f7632d0570 0x55f7632c4880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:20:46.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.695+0000 7fa1ffe00640 1 --2- 172.21.15.39:0/280761864 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f7632cb400 0x55f7632dc430 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec02c8e0 tx=0x7fa1ec009620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b111cbb84a670fa7 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:46.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.695+0000 7fa1f1600640 1 -- 172.21.15.39:0/280761864 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1ec005d50 con 0x55f7632cb400 2024-03-18T16:20:46.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.695+0000 7fa1f1600640 1 -- 172.21.15.39:0/280761864 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa1ec009780 con 0x55f7632cb400 2024-03-18T16:20:46.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.695+0000 7fa1f1600640 1 -- 172.21.15.39:0/280761864 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1ec004920 con 0x55f7632cb400 2024-03-18T16:20:46.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.696+0000 7fa20a9b9740 1 -- 172.21.15.39:0/280761864 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f7632cb400 msgr2=0x55f7632dc430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:46.762 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.696+0000 7fa20a9b9740 1 --2- 172.21.15.39:0/280761864 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f7632cb400 0x55f7632dc430 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fa1ec02c8e0 tx=0x7fa1ec009620 comp rx=0 tx=0).stop 2024-03-18T16:20:46.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.696+0000 7fa20a9b9740 1 -- 172.21.15.39:0/280761864 shutdown_connections 2024-03-18T16:20:46.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.696+0000 7fa20a9b9740 1 --2- 172.21.15.39:0/280761864 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f7632d0570 0x55f7632c4880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:46.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.696+0000 7fa20a9b9740 1 --2- 172.21.15.39:0/280761864 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f7632cb400 0x55f7632dc430 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:46.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.696+0000 7fa20a9b9740 1 --2- 172.21.15.39:0/280761864 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f7632caa10 0x55f7632c6360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:46.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.696+0000 7fa20a9b9740 1 -- 172.21.15.39:0/280761864 >> 172.21.15.39:0/280761864 conn(0x55f7632c5f80 msgr2=0x55f7632cc8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:46.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.696+0000 7fa20a9b9740 1 -- 172.21.15.39:0/280761864 shutdown_connections 2024-03-18T16:20:46.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.696+0000 7fa20a9b9740 1 -- 172.21.15.39:0/280761864 wait complete. 2024-03-18T16:20:46.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.697+0000 7fa20a9b9740 1 -- start start 2024-03-18T16:20:46.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.697+0000 7fa20a9b9740 1 Processor -- start 2024-03-18T16:20:46.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.698+0000 7fa20a9b9740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f7632caa10 0x55f7632d9df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:46.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.698+0000 7fa20a9b9740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f7632cb400 0x55f7632da330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:46.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.698+0000 7fa20a9b9740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f7632d0570 0x55f7632db2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:46.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.698+0000 7fa20a9b9740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fa2001ad940 con 0x55f7632cb400 2024-03-18T16:20:46.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.698+0000 7fa20a9b9740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fa2001ad240 con 0x55f7632caa10 2024-03-18T16:20:46.763 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.698+0000 7fa20a9b9740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fa2001aa420 con 0x55f7632d0570 2024-03-18T16:20:46.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.698+0000 7fa20a9b9740 10 client.0 _ll_register_callbacks cb 0x7fa1f4005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:20:46.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.698+0000 7fa205c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f7632caa10 0x55f7632d9df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:46.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.698+0000 7fa1ffe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f7632cb400 0x55f7632da330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:46.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.698+0000 7fa205c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f7632caa10 0x55f7632d9df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:53362/0 (socket says 172.21.15.39:53362) 2024-03-18T16:20:46.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.698+0000 7fa205c00640 1 -- 172.21.15.39:0/4144152719 learned_addr learned my addr 172.21.15.39:0/4144152719 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:20:46.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.698+0000 7fa205c00640 1 --2- 172.21.15.39:0/4144152719 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f7632d0570 0x55f7632db2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:46.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.698+0000 7fa1ffe00640 1 -- 172.21.15.39:0/4144152719 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f7632d0570 msgr2=0x55f7632db2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:46.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.698+0000 7fa1ffe00640 1 --2- 172.21.15.39:0/4144152719 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f7632d0570 0x55f7632db2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:46.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.698+0000 7fa1ffe00640 1 -- 172.21.15.39:0/4144152719 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f7632caa10 msgr2=0x55f7632d9df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:46.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.698+0000 7fa1ffe00640 1 --2- 172.21.15.39:0/4144152719 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f7632caa10 0x55f7632d9df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:46.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.698+0000 7fa1ffe00640 1 -- 172.21.15.39:0/4144152719 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f7632db8e0 con 0x55f7632cb400 2024-03-18T16:20:46.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.698+0000 7fa205c00640 1 --2- 172.21.15.39:0/4144152719 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f7632caa10 0x55f7632d9df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:20:46.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.699+0000 7fa1ffe00640 1 --2- 172.21.15.39:0/4144152719 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f7632cb400 0x55f7632da330 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x55f7632e3760 tx=0x7fa1ec02bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4896104fc5bab688 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:46.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.699+0000 7fa1e6a00640 10 client.0 ms_handle_connect on v2:172.21.15.39:3300/0 2024-03-18T16:20:46.764 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.699+0000 7fa1e6a00640 1 -- 172.21.15.39:0/4144152719 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1ec061060 con 0x55f7632cb400 2024-03-18T16:20:46.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.699+0000 7fa20a9b9740 1 -- 172.21.15.39:0/4144152719 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa1ec009780 con 0x55f7632cb400 2024-03-18T16:20:46.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.699+0000 7fa1e6a00640 1 -- 172.21.15.39:0/4144152719 <== mon.0 v2:172.21.15.39:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fa1ec034440 con 0x55f7632cb400 2024-03-18T16:20:46.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.699+0000 7fa1e6a00640 1 -- 172.21.15.39:0/4144152719 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7fa1ec034640 con 0x55f7632cb400 2024-03-18T16:20:46.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.699+0000 7fa20a9b9740 10 client.17067 fetch_fsmap learned FSMap version 48 2024-03-18T16:20:46.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.699+0000 7fa20a9b9740 1 -- 172.21.15.39:0/4144152719 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x55f76334fa80 con 0x55f7632cb400 2024-03-18T16:20:46.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.700+0000 7fa1e6a00640 1 -- 172.21.15.39:0/4144152719 <== mon.0 v2:172.21.15.39:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa1ec009780 con 0x55f7632cb400 2024-03-18T16:20:46.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.700+0000 7fa1e6a00640 1 -- 172.21.15.39:0/4144152719 <== mon.0 v2:172.21.15.39:3300/0 5 ==== fsmap(e 48) ==== 8386+0+0 (secure 0 0 0) 0x7fa1ec0497c0 con 0x55f7632cb400 2024-03-18T16:20:46.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.700+0000 7fa20a9b9740 10 client.17067 fetch_fsmap finished waiting for FSMap version 48 2024-03-18T16:20:46.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.700+0000 7fa20a9b9740 10 client.17067 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:20:46.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.700+0000 7fa20a9b9740 20 client.17067 populate_metadata read hostname 'smithi039' 2024-03-18T16:20:46.765 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.700+0000 7fa20a9b9740 1 --2- 172.21.15.39:0/4144152719 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55f763350360 0x55f763352780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:46.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.700+0000 7fa20a9b9740 4 client.17067 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-18T16:20:46.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.700+0000 7fa20a9b9740 1 -- 172.21.15.39:0/4144152719 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f76323c730 con 0x55f763350360 2024-03-18T16:20:46.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.700+0000 7fa205c00640 1 --2- 172.21.15.39:0/4144152719 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55f763350360 0x55f763352780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:46.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.701+0000 7fa205c00640 1 --2- 172.21.15.39:0/4144152719 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55f763350360 0x55f763352780 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=35cc260184767b6a server_cookie=72343eb9bf36d8c in_seq=0 out_seq=0 2024-03-18T16:20:46.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.701+0000 7fa1e6a00640 10 client.17067 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:20:46.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.701+0000 7fa1e6a00640 1 -- 172.21.15.39:0/4144152719 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55f76323c730 con 0x55f763350360 2024-03-18T16:20:46.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.701+0000 7fa1e6a00640 10 client.17067 handle_command_reply: tid=0 2024-03-18T16:20:46.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.701+0000 7fa20a9b9740 1 client.17067 shutdown 2024-03-18T16:20:46.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.701+0000 7fa20a9b9740 1 -- 172.21.15.39:0/4144152719 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f7632cb400 msgr2=0x55f7632da330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:46.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.701+0000 7fa20a9b9740 1 --2- 172.21.15.39:0/4144152719 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f7632cb400 0x55f7632da330 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x55f7632e3760 tx=0x7fa1ec02bc30 comp rx=0 tx=0).stop 2024-03-18T16:20:46.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.701+0000 7fa20a9b9740 1 -- 172.21.15.39:0/4144152719 shutdown_connections 2024-03-18T16:20:46.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.701+0000 7fa20a9b9740 1 --2- 172.21.15.39:0/4144152719 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55f763350360 0x55f763352780 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:46.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.701+0000 7fa20a9b9740 1 --2- 172.21.15.39:0/4144152719 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55f7632d0570 0x55f7632db2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:46.766 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.701+0000 7fa20a9b9740 1 --2- 172.21.15.39:0/4144152719 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55f7632cb400 0x55f7632da330 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:46.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.701+0000 7fa20a9b9740 1 --2- 172.21.15.39:0/4144152719 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f7632caa10 0x55f7632d9df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:46.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.701+0000 7fa1e6a00640 0 client.17067 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:20:46.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.701+0000 7fa20a9b9740 1 -- 172.21.15.39:0/4144152719 >> 172.21.15.39:0/4144152719 conn(0x55f7632c5f80 msgr2=0x55f7632c4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:46.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.701+0000 7fa20a9b9740 1 -- 172.21.15.39:0/4144152719 shutdown_connections 2024-03-18T16:20:46.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.701+0000 7fa20a9b9740 1 -- 172.21.15.39:0/4144152719 wait complete. 2024-03-18T16:20:46.767 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.702+0000 7fa20a9b9740 20 client.17067 trim_cache size 0 max 16384 2024-03-18T16:20:46.767 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-03-18T16:20:46.768 INFO:teuthology.orchestra.run.smithi039.stdout: "status": "no active scrubs running", 2024-03-18T16:20:46.768 INFO:teuthology.orchestra.run.smithi039.stdout: "scrubs": {} 2024-03-18T16:20:46.768 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-03-18T16:20:46.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.702+0000 7fa206600640 1 -- 172.21.15.39:0/4202170725 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa1e80a2880 msgr2=0x7fa1e80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:46.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.702+0000 7fa206600640 1 --2- 172.21.15.39:0/4202170725 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa1e80a2880 0x7fa1e80a4d40 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fa1f40a3ac0 tx=0x7fa1f40d2ce0 comp rx=0 tx=0).stop 2024-03-18T16:20:46.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.702+0000 7fa206600640 1 -- 172.21.15.39:0/4202170725 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa200102140 msgr2=0x7fa200101ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:46.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.702+0000 7fa206600640 1 --2- 172.21.15.39:0/4202170725 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa200102140 0x7fa200101ab0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fa1f400e9f0 tx=0x7fa1f400edf0 comp rx=0 tx=0).stop 2024-03-18T16:20:46.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.702+0000 7fa206600640 1 -- 172.21.15.39:0/4202170725 shutdown_connections 2024-03-18T16:20:46.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.702+0000 7fa206600640 1 --2- 172.21.15.39:0/4202170725 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa2001073a0 0x7fa200100630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:46.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.702+0000 7fa206600640 1 --2- 172.21.15.39:0/4202170725 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa1e80a2880 0x7fa1e80a4d40 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:46.769 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.702+0000 7fa206600640 1 --2- 172.21.15.39:0/4202170725 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa200104a70 0x7fa2001000f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:46.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.702+0000 7fa206600640 1 --2- 172.21.15.39:0/4202170725 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa200102140 0x7fa200101ab0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:46.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.702+0000 7fa206600640 1 -- 172.21.15.39:0/4202170725 >> 172.21.15.39:0/4202170725 conn(0x7fa2000fbb50 msgr2=0x7fa200102a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:46.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.703+0000 7fa206600640 1 -- 172.21.15.39:0/4202170725 shutdown_connections 2024-03-18T16:20:46.770 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:46.703+0000 7fa206600640 1 -- 172.21.15.39:0/4202170725 wait complete. 2024-03-18T16:20:46.770 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-18T16:20:46.770 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:20:47.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.005+0000 7fc44ae00640 1 -- 172.21.15.39:0/2484948960 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc4440ffe60 msgr2=0x7fc444102290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:47.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.005+0000 7fc44ae00640 1 --2- 172.21.15.39:0/2484948960 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc4440ffe60 0x7fc444102290 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fc438009900 tx=0x7fc43802bc30 comp rx=0 tx=0).stop 2024-03-18T16:20:47.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.005+0000 7fc44ae00640 1 -- 172.21.15.39:0/2484948960 shutdown_connections 2024-03-18T16:20:47.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.005+0000 7fc44ae00640 1 --2- 172.21.15.39:0/2484948960 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc4441027d0 0x7fc444104c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:47.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.005+0000 7fc44ae00640 1 --2- 172.21.15.39:0/2484948960 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc444105180 0x7fc444105600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:47.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.005+0000 7fc44ae00640 1 --2- 172.21.15.39:0/2484948960 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc4440ffe60 0x7fc444102290 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:47.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.005+0000 7fc44ae00640 1 -- 172.21.15.39:0/2484948960 >> 172.21.15.39:0/2484948960 conn(0x7fc4440fba60 msgr2=0x7fc4440fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:47.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.005+0000 7fc44ae00640 1 -- 172.21.15.39:0/2484948960 shutdown_connections 2024-03-18T16:20:47.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.006+0000 7fc44ae00640 1 -- 172.21.15.39:0/2484948960 wait complete. 2024-03-18T16:20:47.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.007+0000 7fc44ae00640 1 Processor -- start 2024-03-18T16:20:47.036 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.007+0000 7fc44ae00640 1 -- start start 2024-03-18T16:20:47.038 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.007+0000 7fc44ae00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc4440ffe60 0x7fc44419deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:47.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.007+0000 7fc44ae00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc4441027d0 0x7fc44419e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:47.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.007+0000 7fc44ae00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc444105180 0x7fc4441a5460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:47.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.007+0000 7fc44ae00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fc4441138b0 con 0x7fc4440ffe60 2024-03-18T16:20:47.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.007+0000 7fc44ae00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fc444113730 con 0x7fc444105180 2024-03-18T16:20:47.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.007+0000 7fc44ae00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fc444113a30 con 0x7fc4441027d0 2024-03-18T16:20:47.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.008+0000 7fc449a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc4440ffe60 0x7fc44419deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:47.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.008+0000 7fc449000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc4441027d0 0x7fc44419e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:47.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.008+0000 7fc449000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc4441027d0 0x7fc44419e3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:45070/0 (socket says 172.21.15.39:45070) 2024-03-18T16:20:47.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.008+0000 7fc44a400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc444105180 0x7fc4441a5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:47.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.008+0000 7fc449000640 1 -- 172.21.15.39:0/2774971440 learned_addr learned my addr 172.21.15.39:0/2774971440 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:20:47.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.008+0000 7fc449000640 1 -- 172.21.15.39:0/2774971440 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc444105180 msgr2=0x7fc4441a5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:47.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.008+0000 7fc449000640 1 --2- 172.21.15.39:0/2774971440 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc444105180 0x7fc4441a5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:47.039 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.008+0000 7fc449000640 1 -- 172.21.15.39:0/2774971440 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc4440ffe60 msgr2=0x7fc44419deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:47.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.008+0000 7fc449000640 1 --2- 172.21.15.39:0/2774971440 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc4440ffe60 0x7fc44419deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:47.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.008+0000 7fc449000640 1 -- 172.21.15.39:0/2774971440 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4441a5ad0 con 0x7fc4441027d0 2024-03-18T16:20:47.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.008+0000 7fc449000640 1 --2- 172.21.15.39:0/2774971440 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc4441027d0 0x7fc44419e3f0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fc43000e9f0 tx=0x7fc43000edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:47.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.009+0000 7fc442000640 1 -- 172.21.15.39:0/2774971440 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc43000cd80 con 0x7fc4441027d0 2024-03-18T16:20:47.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.009+0000 7fc442000640 1 -- 172.21.15.39:0/2774971440 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc430005500 con 0x7fc4441027d0 2024-03-18T16:20:47.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.009+0000 7fc44ae00640 1 -- 172.21.15.39:0/2774971440 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4441a5dc0 con 0x7fc4441027d0 2024-03-18T16:20:47.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.009+0000 7fc442000640 1 -- 172.21.15.39:0/2774971440 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc430010690 con 0x7fc4441027d0 2024-03-18T16:20:47.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.010+0000 7fc44ae00640 1 -- 172.21.15.39:0/2774971440 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4441a6280 con 0x7fc4441027d0 2024-03-18T16:20:47.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.011+0000 7fc44ae00640 1 -- 172.21.15.39:0/2774971440 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4440fd550 con 0x7fc4441027d0 2024-03-18T16:20:47.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.012+0000 7fc442000640 1 -- 172.21.15.39:0/2774971440 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fc4300046c0 con 0x7fc4441027d0 2024-03-18T16:20:47.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.013+0000 7fc442000640 1 --2- 172.21.15.39:0/2774971440 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fc42c0a2880 0x7fc42c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:47.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.014+0000 7fc442000640 1 -- 172.21.15.39:0/2774971440 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fc4300cf160 con 0x7fc4441027d0 2024-03-18T16:20:47.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.014+0000 7fc449a00640 1 --2- 172.21.15.39:0/2774971440 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fc42c0a2880 0x7fc42c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:47.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.015+0000 7fc449a00640 1 --2- 172.21.15.39:0/2774971440 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fc42c0a2880 0x7fc42c0a4d40 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fc43802c240 tx=0x7fc43802df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:47.040 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.018+0000 7fc442000640 1 -- 172.21.15.39:0/2774971440 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fc43009be70 con 0x7fc4441027d0 2024-03-18T16:20:47.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:47 smithi194 ceph-mon[36200]: pgmap v743: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 596 B/s wr, 10 op/s 2024-03-18T16:20:47.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:47 smithi039 ceph-mon[31721]: pgmap v743: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 596 B/s wr, 10 op/s 2024-03-18T16:20:47.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.353+0000 7fc44ae00640 1 -- 172.21.15.39:0/2774971440 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc444062480 con 0x7fc4441027d0 2024-03-18T16:20:47.408 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.354+0000 7fc442000640 1 -- 172.21.15.39:0/2774971440 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) ==== 94+0+11939 (secure 0 0 0) 0x7fc43009b770 con 0x7fc4441027d0 2024-03-18T16:20:47.408 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:20:47.409 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2520226608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2520226608},{"type":"v1","addr":"172.21.15.39:6835","nonce":2520226608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/719341763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":719341763},{"type":"v1","addr":"172.21.15.39:6837","nonce":719341763}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":48,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T16:06:44.180425+0000","modified":"2024-03-18T16:20:16.222033+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24575,"mds_1":24556,"mds_2":24535,"mds_3":24524,"mds_4":14463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14445":{"gid":14445,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.39:6839/656513813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":656513813},{"type":"v1","addr":"172.21.15.39:6839","nonce":656513813}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14463":{"gid":14463,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":13,"addr":"172.21.15.39:6841/3515326042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3515326042},{"type":"v1","addr":"172.21.15.39:6841","nonce":3515326042}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6833/931350738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":931350738},{"type":"v1","addr":"172.21.15.194:6833","nonce":931350738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6833/3874209621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":3874209621},{"type":"v1","addr":"172.21.15.196:6833","nonce":3874209621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6835/188385470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":188385470},{"type":"v1","addr":"172.21.15.194:6835","nonce":188385470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6835/3949984309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":3949984309},{"type":"v1","addr":"172.21.15.196:6835","nonce":3949984309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":15,"addr":"172.21.15.194:6837/1301464246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":1301464246},{"type":"v1","addr":"172.21.15.194:6837","nonce":1301464246}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":16,"addr":"172.21.15.196:6837/895247251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":895247251},{"type":"v1","addr":"172.21.15.196:6837","nonce":895247251}]},"join_fscid":-1,"export_targets":[1,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.196:6839/2001427268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6838","nonce":2001427268},{"type":"v1","addr":"172.21.15.196:6839","nonce":2001427268}]},"join_fscid":-1,"export_targets":[4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24575":{"gid":24575,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.194:6839/274911158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":274911158},{"type":"v1","addr":"172.21.15.194:6839","nonce":274911158}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24575,"qdb_cluster":[14463,24524,24575,24556,24535]},"id":1}]} 2024-03-18T16:20:47.410 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 48 2024-03-18T16:20:47.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.357+0000 7fc44ae00640 1 -- 172.21.15.39:0/2774971440 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fc42c0a2880 msgr2=0x7fc42c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:47.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.357+0000 7fc44ae00640 1 --2- 172.21.15.39:0/2774971440 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fc42c0a2880 0x7fc42c0a4d40 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fc43802c240 tx=0x7fc43802df90 comp rx=0 tx=0).stop 2024-03-18T16:20:47.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.357+0000 7fc44ae00640 1 -- 172.21.15.39:0/2774971440 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc4441027d0 msgr2=0x7fc44419e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:47.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.357+0000 7fc44ae00640 1 --2- 172.21.15.39:0/2774971440 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc4441027d0 0x7fc44419e3f0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fc43000e9f0 tx=0x7fc43000edf0 comp rx=0 tx=0).stop 2024-03-18T16:20:47.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.357+0000 7fc44ae00640 1 -- 172.21.15.39:0/2774971440 shutdown_connections 2024-03-18T16:20:47.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.357+0000 7fc44ae00640 1 --2- 172.21.15.39:0/2774971440 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc444105180 0x7fc4441a5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:47.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.357+0000 7fc44ae00640 1 --2- 172.21.15.39:0/2774971440 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fc42c0a2880 0x7fc42c0a4d40 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:47.410 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.357+0000 7fc44ae00640 1 --2- 172.21.15.39:0/2774971440 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc4441027d0 0x7fc44419e3f0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:47.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.357+0000 7fc44ae00640 1 --2- 172.21.15.39:0/2774971440 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fc4440ffe60 0x7fc44419deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:47.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.357+0000 7fc44ae00640 1 -- 172.21.15.39:0/2774971440 >> 172.21.15.39:0/2774971440 conn(0x7fc4440fba60 msgr2=0x7fc444103160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:47.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.358+0000 7fc44ae00640 1 -- 172.21.15.39:0/2774971440 shutdown_connections 2024-03-18T16:20:47.411 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.358+0000 7fc44ae00640 1 -- 172.21.15.39:0/2774971440 wait complete. 2024-03-18T16:20:47.411 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-18T16:20:47.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:47 smithi196 ceph-mon[36307]: pgmap v743: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 596 B/s wr, 10 op/s 2024-03-18T16:20:47.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.615+0000 7fa2fb200640 1 -- 172.21.15.39:0/750502366 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa2fc100800 msgr2=0x7fa2fc100c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:47.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.615+0000 7fa2fb200640 1 --2- 172.21.15.39:0/750502366 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa2fc100800 0x7fa2fc100c60 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fa2e4009900 tx=0x7fa2e402bc40 comp rx=0 tx=0).stop 2024-03-18T16:20:47.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.615+0000 7fa2fb200640 1 -- 172.21.15.39:0/750502366 shutdown_connections 2024-03-18T16:20:47.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.615+0000 7fa2fb200640 1 --2- 172.21.15.39:0/750502366 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa2fc1011a0 0x7fa2fc101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:47.618 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.615+0000 7fa2fb200640 1 --2- 172.21.15.39:0/750502366 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa2fc100800 0x7fa2fc100c60 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:47.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.615+0000 7fa2fb200640 1 --2- 172.21.15.39:0/750502366 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa2fc0ffea0 0x7fa2fc1002c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:47.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.615+0000 7fa2fb200640 1 -- 172.21.15.39:0/750502366 >> 172.21.15.39:0/750502366 conn(0x7fa2fc0fba80 msgr2=0x7fa2fc0fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:47.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.615+0000 7fa2fb200640 1 -- 172.21.15.39:0/750502366 shutdown_connections 2024-03-18T16:20:47.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.615+0000 7fa2fb200640 1 -- 172.21.15.39:0/750502366 wait complete. 2024-03-18T16:20:47.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.617+0000 7fa2fb200640 1 Processor -- start 2024-03-18T16:20:47.619 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.617+0000 7fa2fb200640 1 -- start start 2024-03-18T16:20:47.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.617+0000 7fa2fb200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa2fc0ffea0 0x7fa2fc199aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:47.620 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.617+0000 7fa2fb200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa2fc100800 0x7fa2fc199fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:47.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.617+0000 7fa2fb200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa2fc1011a0 0x7fa2fc1a1050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:47.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.617+0000 7fa2fb200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fa2fc10f460 con 0x7fa2fc100800 2024-03-18T16:20:47.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.617+0000 7fa2fb200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fa2fc10f2e0 con 0x7fa2fc1011a0 2024-03-18T16:20:47.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.617+0000 7fa2fb200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fa2fc10f5e0 con 0x7fa2fc0ffea0 2024-03-18T16:20:47.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.618+0000 7fa2f9e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa2fc0ffea0 0x7fa2fc199aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:47.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.618+0000 7fa2fa800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa2fc1011a0 0x7fa2fc1a1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:47.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.618+0000 7fa2f3e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa2fc100800 0x7fa2fc199fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:47.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.618+0000 7fa2f9e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa2fc0ffea0 0x7fa2fc199aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:45086/0 (socket says 172.21.15.39:45086) 2024-03-18T16:20:47.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.618+0000 7fa2fa800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa2fc1011a0 0x7fa2fc1a1050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:53394/0 (socket says 172.21.15.39:53394) 2024-03-18T16:20:47.621 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.618+0000 7fa2f9e00640 1 -- 172.21.15.39:0/1579441835 learned_addr learned my addr 172.21.15.39:0/1579441835 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:20:47.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.618+0000 7fa2fa800640 1 -- 172.21.15.39:0/1579441835 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa2fc0ffea0 msgr2=0x7fa2fc199aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:47.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.618+0000 7fa2fa800640 1 --2- 172.21.15.39:0/1579441835 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa2fc0ffea0 0x7fa2fc199aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:47.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.618+0000 7fa2fa800640 1 -- 172.21.15.39:0/1579441835 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa2fc100800 msgr2=0x7fa2fc199fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:47.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.618+0000 7fa2fa800640 1 --2- 172.21.15.39:0/1579441835 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa2fc100800 0x7fa2fc199fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:47.622 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.618+0000 7fa2fa800640 1 -- 172.21.15.39:0/1579441835 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2fc1a16c0 con 0x7fa2fc1011a0 2024-03-18T16:20:47.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.618+0000 7fa2f9e00640 1 --2- 172.21.15.39:0/1579441835 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa2fc0ffea0 0x7fa2fc199aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:20:47.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.619+0000 7fa2fa800640 1 --2- 172.21.15.39:0/1579441835 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa2fc1011a0 0x7fa2fc1a1050 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fa2f400ec50 tx=0x7fa2f400c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:47.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.619+0000 7fa2f2a00640 1 -- 172.21.15.39:0/1579441835 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2f4003b70 con 0x7fa2fc1011a0 2024-03-18T16:20:47.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.619+0000 7fa2f2a00640 1 -- 172.21.15.39:0/1579441835 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa2f4003d10 con 0x7fa2fc1011a0 2024-03-18T16:20:47.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.619+0000 7fa2fb200640 1 -- 172.21.15.39:0/1579441835 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2fc1a19b0 con 0x7fa2fc1011a0 2024-03-18T16:20:47.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.619+0000 7fa2f2a00640 1 -- 172.21.15.39:0/1579441835 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2f4010890 con 0x7fa2fc1011a0 2024-03-18T16:20:47.623 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.619+0000 7fa2fb200640 1 -- 172.21.15.39:0/1579441835 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2fc1a1ed0 con 0x7fa2fc1011a0 2024-03-18T16:20:47.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.622+0000 7fa30381f740 1 Processor -- start 2024-03-18T16:20:47.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.622+0000 7fa2f2a00640 1 -- 172.21.15.39:0/1579441835 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fa2f4003fe0 con 0x7fa2fc1011a0 2024-03-18T16:20:47.625 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.623+0000 7fa30381f740 1 -- start start 2024-03-18T16:20:47.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.623+0000 7fa30381f740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562a9d874a90 0x562a9d874eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:47.627 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.623+0000 7fa30381f740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a9d875480 0x562a9d879f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:47.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.623+0000 7fa30381f740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562a9d87a5f0 0x562a9d87ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:47.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.623+0000 7fa30381f740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x562a9d83c3b0 con 0x562a9d874a90 2024-03-18T16:20:47.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.623+0000 7fa30381f740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x562a9d7d9450 con 0x562a9d875480 2024-03-18T16:20:47.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.623+0000 7fa30381f740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x562a9d781cb0 con 0x562a9d87a5f0 2024-03-18T16:20:47.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.624+0000 7fa2f9e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a9d875480 0x562a9d879f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:47.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.624+0000 7fa2f9e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a9d875480 0x562a9d879f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:53396/0 (socket says 172.21.15.39:53396) 2024-03-18T16:20:47.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.624+0000 7fa2f9e00640 1 -- 172.21.15.39:0/3007552789 learned_addr learned my addr 172.21.15.39:0/3007552789 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:20:47.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.624+0000 7fa2f3e00640 1 --2- 172.21.15.39:0/3007552789 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562a9d874a90 0x562a9d874eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:47.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.624+0000 7fa2f3e00640 1 --2- 172.21.15.39:0/3007552789 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562a9d87a5f0 0x562a9d87ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:47.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.624+0000 7fa2f9e00640 1 -- 172.21.15.39:0/3007552789 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562a9d87a5f0 msgr2=0x562a9d87ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:47.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.624+0000 7fa2f9e00640 1 --2- 172.21.15.39:0/3007552789 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562a9d87a5f0 0x562a9d87ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:47.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.624+0000 7fa2f9e00640 1 -- 172.21.15.39:0/3007552789 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562a9d874a90 msgr2=0x562a9d874eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:47.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.624+0000 7fa2f9e00640 1 --2- 172.21.15.39:0/3007552789 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562a9d874a90 0x562a9d874eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:47.628 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.624+0000 7fa2f9e00640 1 -- 172.21.15.39:0/3007552789 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a9d7a31f0 con 0x562a9d875480 2024-03-18T16:20:47.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.624+0000 7fa2f3e00640 1 --2- 172.21.15.39:0/3007552789 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562a9d87a5f0 0x562a9d87ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:20:47.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.624+0000 7fa2f2a00640 1 --2- 172.21.15.39:0/1579441835 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa2e00a2880 0x7fa2e00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:47.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.624+0000 7fa2f9e00640 1 --2- 172.21.15.39:0/3007552789 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a9d875480 0x562a9d879f80 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fa2ec011650 tx=0x7fa2ec009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=84f8f10c53512568 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:47.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.625+0000 7fa2f2a00640 1 -- 172.21.15.39:0/1579441835 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fa2f40cf370 con 0x7fa2fc1011a0 2024-03-18T16:20:47.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.625+0000 7fa2f3e00640 1 --2- 172.21.15.39:0/1579441835 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa2e00a2880 0x7fa2e00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:47.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.625+0000 7fa2eb000640 1 -- 172.21.15.39:0/3007552789 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2ec01e070 con 0x562a9d875480 2024-03-18T16:20:47.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.625+0000 7fa2eb000640 1 -- 172.21.15.39:0/3007552789 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa2ec00fba0 con 0x562a9d875480 2024-03-18T16:20:47.629 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.625+0000 7fa2eb000640 1 -- 172.21.15.39:0/3007552789 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2ec019cd0 con 0x562a9d875480 2024-03-18T16:20:47.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.626+0000 7fa30381f740 1 -- 172.21.15.39:0/3007552789 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a9d875480 msgr2=0x562a9d879f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:47.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.626+0000 7fa2f3e00640 1 --2- 172.21.15.39:0/1579441835 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa2e00a2880 0x7fa2e00a4d40 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fa2e402c250 tx=0x7fa2e402e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:47.631 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.626+0000 7fa30381f740 1 --2- 172.21.15.39:0/3007552789 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a9d875480 0x562a9d879f80 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7fa2ec011650 tx=0x7fa2ec009f90 comp rx=0 tx=0).stop 2024-03-18T16:20:47.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.626+0000 7fa30381f740 1 -- 172.21.15.39:0/3007552789 shutdown_connections 2024-03-18T16:20:47.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.626+0000 7fa30381f740 1 --2- 172.21.15.39:0/3007552789 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a9d875480 0x562a9d879f80 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:47.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.626+0000 7fa30381f740 1 --2- 172.21.15.39:0/3007552789 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562a9d87a5f0 0x562a9d87ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:47.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.626+0000 7fa30381f740 1 --2- 172.21.15.39:0/3007552789 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562a9d874a90 0x562a9d874eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:47.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.626+0000 7fa30381f740 1 -- 172.21.15.39:0/3007552789 >> 172.21.15.39:0/3007552789 conn(0x562a9d870000 msgr2=0x562a9d870400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:47.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.626+0000 7fa30381f740 1 -- 172.21.15.39:0/3007552789 shutdown_connections 2024-03-18T16:20:47.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.626+0000 7fa30381f740 1 -- 172.21.15.39:0/3007552789 wait complete. 2024-03-18T16:20:47.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.627+0000 7fa30381f740 1 -- start start 2024-03-18T16:20:47.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.627+0000 7fa30381f740 1 Processor -- start 2024-03-18T16:20:47.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.628+0000 7fa30381f740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562a9d874a90 0x562a9d87ed30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:47.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.628+0000 7fa30381f740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562a9d875480 0x562a9d87f270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:47.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.628+0000 7fa30381f740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a9d87a5f0 0x562a9d886270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:47.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.628+0000 7fa30381f740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x562a9d795110 con 0x562a9d874a90 2024-03-18T16:20:47.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.628+0000 7fa30381f740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x562a9d7d54f0 con 0x562a9d87a5f0 2024-03-18T16:20:47.632 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.628+0000 7fa30381f740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x562a9d87d5a0 con 0x562a9d875480 2024-03-18T16:20:47.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.628+0000 7fa30381f740 10 client.0 _ll_register_callbacks cb 0x562a9d782b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:20:47.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.628+0000 7fa2f9e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562a9d874a90 0x562a9d87ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:47.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.628+0000 7fa2f9e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562a9d874a90 0x562a9d87ed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:60412/0 (socket says 172.21.15.39:60412) 2024-03-18T16:20:47.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.628+0000 7fa2f9e00640 1 -- 172.21.15.39:0/1373756329 learned_addr learned my addr 172.21.15.39:0/1373756329 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:20:47.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.628+0000 7fa2f3e00640 1 --2- 172.21.15.39:0/1373756329 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562a9d875480 0x562a9d87f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:47.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.628+0000 7fa2fa800640 1 --2- 172.21.15.39:0/1373756329 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a9d87a5f0 0x562a9d886270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:47.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.629+0000 7fa2f3e00640 1 -- 172.21.15.39:0/1373756329 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a9d87a5f0 msgr2=0x562a9d886270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:47.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.629+0000 7fa2f3e00640 1 --2- 172.21.15.39:0/1373756329 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a9d87a5f0 0x562a9d886270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:47.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.629+0000 7fa2f3e00640 1 -- 172.21.15.39:0/1373756329 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562a9d874a90 msgr2=0x562a9d87ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:47.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.629+0000 7fa2f3e00640 1 --2- 172.21.15.39:0/1373756329 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562a9d874a90 0x562a9d87ed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:47.633 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.629+0000 7fa2f3e00640 1 -- 172.21.15.39:0/1373756329 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a9d4ed3f0 con 0x562a9d875480 2024-03-18T16:20:47.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.629+0000 7fa2f3e00640 1 --2- 172.21.15.39:0/1373756329 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562a9d875480 0x562a9d87f270 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fa2e4009230 tx=0x7fa2e4000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e3444d01a529c230 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:47.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.629+0000 7fa2df400640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:20:47.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.629+0000 7fa2df400640 1 -- 172.21.15.39:0/1373756329 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2e40050f0 con 0x562a9d875480 2024-03-18T16:20:47.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.629+0000 7fa30381f740 1 -- 172.21.15.39:0/1373756329 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa2ec00fba0 con 0x562a9d875480 2024-03-18T16:20:47.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.630+0000 7fa2df400640 1 -- 172.21.15.39:0/1373756329 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fa2e4005290 con 0x562a9d875480 2024-03-18T16:20:47.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.630+0000 7fa2df400640 1 -- 172.21.15.39:0/1373756329 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7fa2e405e050 con 0x562a9d875480 2024-03-18T16:20:47.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.630+0000 7fa30381f740 10 client.27766 fetch_fsmap learned FSMap version 48 2024-03-18T16:20:47.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.630+0000 7fa30381f740 1 -- 172.21.15.39:0/1373756329 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x562a9d5b77d0 con 0x562a9d875480 2024-03-18T16:20:47.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.630+0000 7fa2df400640 1 -- 172.21.15.39:0/1373756329 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa2ec00fba0 con 0x562a9d875480 2024-03-18T16:20:47.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.630+0000 7fa2df400640 1 -- 172.21.15.39:0/1373756329 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 48) ==== 8386+0+0 (secure 0 0 0) 0x7fa2e4034440 con 0x562a9d875480 2024-03-18T16:20:47.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.630+0000 7fa30381f740 10 client.27766 fetch_fsmap finished waiting for FSMap version 48 2024-03-18T16:20:47.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.630+0000 7fa30381f740 10 client.27766 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-03-18T16:20:47.636 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.630+0000 7fa30381f740 20 client.27766 populate_metadata read hostname 'smithi039' 2024-03-18T16:20:47.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.630+0000 7fa30381f740 1 --2- 172.21.15.39:0/1373756329 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x562a9d88aa80 0x562a9d88cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:47.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.630+0000 7fa30381f740 4 client.27766 mds_command: new command op to 14463 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:20:47.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.630+0000 7fa30381f740 1 -- 172.21.15.39:0/1373756329 --> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562a9d7e6990 con 0x562a9d88aa80 2024-03-18T16:20:47.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.631+0000 7fa2f9e00640 1 --2- 172.21.15.39:0/1373756329 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x562a9d88aa80 0x562a9d88cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:47.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.632+0000 7fa2f9e00640 1 --2- 172.21.15.39:0/1373756329 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x562a9d88aa80 0x562a9d88cf40 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=153e1ac9eba973d5 server_cookie=4f52426e85c71767 in_seq=0 out_seq=0 2024-03-18T16:20:47.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.632+0000 7fa2df400640 10 client.27766 ms_handle_connect on v2:172.21.15.39:6840/3515326042 2024-03-18T16:20:47.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.632+0000 7fa2df400640 1 -- 172.21.15.39:0/1373756329 <== mds.4 v2:172.21.15.39:6840/3515326042 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x562a9d7e6990 con 0x562a9d88aa80 2024-03-18T16:20:47.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.632+0000 7fa2df400640 10 client.27766 handle_command_reply: tid=0 2024-03-18T16:20:47.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.632+0000 7fa30381f740 1 client.27766 shutdown 2024-03-18T16:20:47.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.633+0000 7fa30381f740 1 -- 172.21.15.39:0/1373756329 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562a9d875480 msgr2=0x562a9d87f270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:47.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.633+0000 7fa30381f740 1 --2- 172.21.15.39:0/1373756329 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562a9d875480 0x562a9d87f270 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fa2e4009230 tx=0x7fa2e4000bc0 comp rx=0 tx=0).stop 2024-03-18T16:20:47.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.633+0000 7fa30381f740 1 -- 172.21.15.39:0/1373756329 shutdown_connections 2024-03-18T16:20:47.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.633+0000 7fa30381f740 1 --2- 172.21.15.39:0/1373756329 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x562a9d88aa80 0x562a9d88cf40 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:47.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.633+0000 7fa30381f740 1 --2- 172.21.15.39:0/1373756329 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a9d87a5f0 0x562a9d886270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:47.637 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.633+0000 7fa30381f740 1 --2- 172.21.15.39:0/1373756329 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562a9d875480 0x562a9d87f270 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:47.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.633+0000 7fa30381f740 1 --2- 172.21.15.39:0/1373756329 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562a9d874a90 0x562a9d87ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:47.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.633+0000 7fa2df400640 0 client.27766 ms_handle_reset on v2:172.21.15.39:6840/3515326042 2024-03-18T16:20:47.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.633+0000 7fa30381f740 1 -- 172.21.15.39:0/1373756329 >> 172.21.15.39:0/1373756329 conn(0x562a9d870000 msgr2=0x562a9d86f220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:47.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.633+0000 7fa30381f740 1 -- 172.21.15.39:0/1373756329 shutdown_connections 2024-03-18T16:20:47.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.633+0000 7fa30381f740 1 -- 172.21.15.39:0/1373756329 wait complete. 2024-03-18T16:20:47.638 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.633+0000 7fa30381f740 20 client.27766 trim_cache size 0 max 16384 2024-03-18T16:20:47.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.655+0000 7fa30381f740 1 Processor -- start 2024-03-18T16:20:47.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.655+0000 7fa30381f740 1 -- start start 2024-03-18T16:20:47.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.655+0000 7fa30381f740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a9d874a90 0x562a9d86f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:47.658 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.655+0000 7fa30381f740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562a9d875480 0x562a9d886230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:47.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.655+0000 7fa30381f740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562a9d87a5f0 0x562a9d891840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:47.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.655+0000 7fa30381f740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fa2fc1ac3f0 con 0x562a9d87a5f0 2024-03-18T16:20:47.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.655+0000 7fa30381f740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fa2fc1adff0 con 0x562a9d874a90 2024-03-18T16:20:47.659 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.655+0000 7fa30381f740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fa2fc1acac0 con 0x562a9d875480 2024-03-18T16:20:47.660 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.656+0000 7fa2f9e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a9d874a90 0x562a9d86f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:47.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.656+0000 7fa2f3e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562a9d875480 0x562a9d886230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:47.669 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.656+0000 7fa2f9e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a9d874a90 0x562a9d86f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:53422/0 (socket says 172.21.15.39:53422) 2024-03-18T16:20:47.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.656+0000 7fa2f9e00640 1 -- 172.21.15.39:0/172443850 learned_addr learned my addr 172.21.15.39:0/172443850 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:20:47.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.656+0000 7fa2fa800640 1 --2- 172.21.15.39:0/172443850 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562a9d87a5f0 0x562a9d891840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:47.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.656+0000 7fa2f3e00640 1 -- 172.21.15.39:0/172443850 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a9d874a90 msgr2=0x562a9d86f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:47.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.656+0000 7fa2f3e00640 1 --2- 172.21.15.39:0/172443850 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a9d874a90 0x562a9d86f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:47.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.656+0000 7fa2f3e00640 1 -- 172.21.15.39:0/172443850 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562a9d87a5f0 msgr2=0x562a9d891840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:47.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.656+0000 7fa2f3e00640 1 --2- 172.21.15.39:0/172443850 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562a9d87a5f0 0x562a9d891840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:47.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.656+0000 7fa2f3e00640 1 -- 172.21.15.39:0/172443850 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a9d86f950 con 0x562a9d875480 2024-03-18T16:20:47.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.656+0000 7fa2f9e00640 1 --2- 172.21.15.39:0/172443850 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a9d874a90 0x562a9d86f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:20:47.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.657+0000 7fa2f3e00640 1 --2- 172.21.15.39:0/172443850 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562a9d875480 0x562a9d886230 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fa2e4034a00 tx=0x7fa2e402cf90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7ae13ff548d8e3c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:47.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.657+0000 7fa2ea600640 1 -- 172.21.15.39:0/172443850 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2e4066020 con 0x562a9d875480 2024-03-18T16:20:47.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.657+0000 7fa2ea600640 1 -- 172.21.15.39:0/172443850 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa2e4009780 con 0x562a9d875480 2024-03-18T16:20:47.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.657+0000 7fa2ea600640 1 -- 172.21.15.39:0/172443850 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2e40055e0 con 0x562a9d875480 2024-03-18T16:20:47.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.658+0000 7fa30381f740 1 -- 172.21.15.39:0/172443850 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562a9d875480 msgr2=0x562a9d886230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:47.670 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.658+0000 7fa30381f740 1 --2- 172.21.15.39:0/172443850 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562a9d875480 0x562a9d886230 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7fa2e4034a00 tx=0x7fa2e402cf90 comp rx=0 tx=0).stop 2024-03-18T16:20:47.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.658+0000 7fa30381f740 1 -- 172.21.15.39:0/172443850 shutdown_connections 2024-03-18T16:20:47.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.658+0000 7fa30381f740 1 --2- 172.21.15.39:0/172443850 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562a9d87a5f0 0x562a9d891840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:47.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.658+0000 7fa30381f740 1 --2- 172.21.15.39:0/172443850 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562a9d875480 0x562a9d886230 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:47.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.658+0000 7fa30381f740 1 --2- 172.21.15.39:0/172443850 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a9d874a90 0x562a9d86f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:47.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.658+0000 7fa30381f740 1 -- 172.21.15.39:0/172443850 >> 172.21.15.39:0/172443850 conn(0x562a9d870000 msgr2=0x562a9d86e990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:47.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.658+0000 7fa30381f740 1 -- 172.21.15.39:0/172443850 shutdown_connections 2024-03-18T16:20:47.671 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.658+0000 7fa30381f740 1 -- 172.21.15.39:0/172443850 wait complete. 2024-03-18T16:20:47.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.658+0000 7fa30381f740 1 -- start start 2024-03-18T16:20:47.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.658+0000 7fa30381f740 1 Processor -- start 2024-03-18T16:20:47.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.659+0000 7fa30381f740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562a9d874a90 0x562a9d883e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:47.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.659+0000 7fa30381f740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a9d875480 0x562a9d8843a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:47.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.659+0000 7fa30381f740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562a9d87a5f0 0x562a9d8853b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:47.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.659+0000 7fa30381f740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fa2fc1ad8f0 con 0x562a9d874a90 2024-03-18T16:20:47.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.659+0000 7fa30381f740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fa2fc1ad1f0 con 0x562a9d875480 2024-03-18T16:20:47.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.659+0000 7fa30381f740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fa2fc1aa3d0 con 0x562a9d87a5f0 2024-03-18T16:20:47.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.659+0000 7fa30381f740 10 client.0 _ll_register_callbacks cb 0x7fa2f40093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:20:47.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.659+0000 7fa2f9e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562a9d874a90 0x562a9d883e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:47.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.659+0000 7fa2fa800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562a9d87a5f0 0x562a9d8853b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:47.675 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.659+0000 7fa2fa800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562a9d87a5f0 0x562a9d8853b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:45142/0 (socket says 172.21.15.39:45142) 2024-03-18T16:20:47.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.659+0000 7fa2fa800640 1 -- 172.21.15.39:0/1230478109 learned_addr learned my addr 172.21.15.39:0/1230478109 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:20:47.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.659+0000 7fa2f3e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a9d875480 0x562a9d8843a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:47.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.660+0000 7fa2fa800640 1 -- 172.21.15.39:0/1230478109 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a9d875480 msgr2=0x562a9d8843a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:47.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.660+0000 7fa2fa800640 1 --2- 172.21.15.39:0/1230478109 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a9d875480 0x562a9d8843a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:47.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.660+0000 7fa2fa800640 1 -- 172.21.15.39:0/1230478109 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562a9d874a90 msgr2=0x562a9d883e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:47.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.660+0000 7fa2fa800640 1 --2- 172.21.15.39:0/1230478109 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562a9d874a90 0x562a9d883e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:47.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.660+0000 7fa2fa800640 1 -- 172.21.15.39:0/1230478109 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a9d86fba0 con 0x562a9d87a5f0 2024-03-18T16:20:47.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.660+0000 7fa2f3e00640 1 --2- 172.21.15.39:0/1230478109 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a9d875480 0x562a9d8843a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:20:47.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.660+0000 7fa2fa800640 1 --2- 172.21.15.39:0/1230478109 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562a9d87a5f0 0x562a9d8853b0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fa2f4004af0 tx=0x7fa2f409b350 comp rx=0 tx=0).ready entity=mon.1 client_cookie=734dd2aaa74de34d server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:47.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.660+0000 7fa2df400640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:20:47.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.660+0000 7fa2df400640 1 -- 172.21.15.39:0/1230478109 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2f40b9060 con 0x562a9d87a5f0 2024-03-18T16:20:47.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.660+0000 7fa2df400640 1 -- 172.21.15.39:0/1230478109 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fa2f40042d0 con 0x562a9d87a5f0 2024-03-18T16:20:47.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.660+0000 7fa30381f740 1 -- 172.21.15.39:0/1230478109 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa2e4009780 con 0x562a9d87a5f0 2024-03-18T16:20:47.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.660+0000 7fa2df400640 1 -- 172.21.15.39:0/1230478109 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7fa2f40044d0 con 0x562a9d87a5f0 2024-03-18T16:20:47.676 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.660+0000 7fa30381f740 10 client.27772 fetch_fsmap learned FSMap version 48 2024-03-18T16:20:47.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.660+0000 7fa30381f740 1 -- 172.21.15.39:0/1230478109 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x562a9d8f1250 con 0x562a9d87a5f0 2024-03-18T16:20:47.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.661+0000 7fa2df400640 1 -- 172.21.15.39:0/1230478109 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa2e4009780 con 0x562a9d87a5f0 2024-03-18T16:20:47.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.661+0000 7fa2df400640 1 -- 172.21.15.39:0/1230478109 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 48) ==== 8386+0+0 (secure 0 0 0) 0x7fa2f4004b40 con 0x562a9d87a5f0 2024-03-18T16:20:47.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.661+0000 7fa30381f740 10 client.27772 fetch_fsmap finished waiting for FSMap version 48 2024-03-18T16:20:47.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.661+0000 7fa30381f740 10 client.27772 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.a 2024-03-18T16:20:47.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.661+0000 7fa30381f740 20 client.27772 populate_metadata read hostname 'smithi039' 2024-03-18T16:20:47.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.661+0000 7fa30381f740 1 --2- 172.21.15.39:0/1230478109 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x562a9d8f1c30 0x562a9d8f4050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:47.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.661+0000 7fa30381f740 4 client.27772 mds_command: new command op to 14463 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:20:47.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.661+0000 7fa30381f740 1 -- 172.21.15.39:0/1230478109 --> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562a9d7e0ce0 con 0x562a9d8f1c30 2024-03-18T16:20:47.677 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.661+0000 7fa2f9e00640 1 --2- 172.21.15.39:0/1230478109 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x562a9d8f1c30 0x562a9d8f4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:47.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.662+0000 7fa2f9e00640 1 --2- 172.21.15.39:0/1230478109 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x562a9d8f1c30 0x562a9d8f4050 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=79d1c63b0f154341 server_cookie=ebb8ebe2442c9941 in_seq=0 out_seq=0 2024-03-18T16:20:47.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.662+0000 7fa2df400640 10 client.27772 ms_handle_connect on v2:172.21.15.39:6840/3515326042 2024-03-18T16:20:47.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.662+0000 7fa2df400640 1 -- 172.21.15.39:0/1230478109 <== mds.4 v2:172.21.15.39:6840/3515326042 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562a9d7e0ce0 con 0x562a9d8f1c30 2024-03-18T16:20:47.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.662+0000 7fa2df400640 10 client.27772 handle_command_reply: tid=0 2024-03-18T16:20:47.697 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.662+0000 7fa30381f740 1 client.27772 shutdown 2024-03-18T16:20:47.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.662+0000 7fa30381f740 1 -- 172.21.15.39:0/1230478109 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562a9d87a5f0 msgr2=0x562a9d8853b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:47.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.662+0000 7fa30381f740 1 --2- 172.21.15.39:0/1230478109 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562a9d87a5f0 0x562a9d8853b0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fa2f4004af0 tx=0x7fa2f409b350 comp rx=0 tx=0).stop 2024-03-18T16:20:47.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.662+0000 7fa30381f740 1 -- 172.21.15.39:0/1230478109 shutdown_connections 2024-03-18T16:20:47.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.662+0000 7fa30381f740 1 --2- 172.21.15.39:0/1230478109 >> [v2:172.21.15.39:6840/3515326042,v1:172.21.15.39:6841/3515326042] conn(0x562a9d8f1c30 0x562a9d8f4050 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:47.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.662+0000 7fa30381f740 1 --2- 172.21.15.39:0/1230478109 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a9d875480 0x562a9d8843a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:47.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.662+0000 7fa2df400640 0 client.27772 ms_handle_reset on v2:172.21.15.39:6840/3515326042 2024-03-18T16:20:47.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.662+0000 7fa30381f740 1 --2- 172.21.15.39:0/1230478109 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x562a9d87a5f0 0x562a9d8853b0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:47.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.662+0000 7fa30381f740 1 --2- 172.21.15.39:0/1230478109 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x562a9d874a90 0x562a9d883e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:47.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.662+0000 7fa30381f740 1 -- 172.21.15.39:0/1230478109 >> 172.21.15.39:0/1230478109 conn(0x562a9d870000 msgr2=0x562a9d86f220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:47.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.663+0000 7fa30381f740 1 -- 172.21.15.39:0/1230478109 shutdown_connections 2024-03-18T16:20:47.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.663+0000 7fa30381f740 1 -- 172.21.15.39:0/1230478109 wait complete. 2024-03-18T16:20:47.698 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.663+0000 7fa30381f740 20 client.27772 trim_cache size 0 max 16384 2024-03-18T16:20:47.699 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:20:47.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.663+0000 7fa2fb200640 1 -- 172.21.15.39:0/1579441835 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa2e00a2880 msgr2=0x7fa2e00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:47.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.663+0000 7fa2fb200640 1 --2- 172.21.15.39:0/1579441835 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa2e00a2880 0x7fa2e00a4d40 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fa2e402c250 tx=0x7fa2e402e040 comp rx=0 tx=0).stop 2024-03-18T16:20:47.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.663+0000 7fa2fb200640 1 -- 172.21.15.39:0/1579441835 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa2fc1011a0 msgr2=0x7fa2fc1a1050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:47.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.663+0000 7fa2fb200640 1 --2- 172.21.15.39:0/1579441835 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa2fc1011a0 0x7fa2fc1a1050 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fa2f400ec50 tx=0x7fa2f400c6a0 comp rx=0 tx=0).stop 2024-03-18T16:20:47.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.664+0000 7fa2fb200640 1 -- 172.21.15.39:0/1579441835 shutdown_connections 2024-03-18T16:20:47.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.664+0000 7fa2fb200640 1 --2- 172.21.15.39:0/1579441835 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa2fc1011a0 0x7fa2fc1a1050 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:47.700 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.664+0000 7fa2fb200640 1 --2- 172.21.15.39:0/1579441835 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fa2e00a2880 0x7fa2e00a4d40 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:47.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.664+0000 7fa2fb200640 1 --2- 172.21.15.39:0/1579441835 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa2fc100800 0x7fa2fc199fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:47.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.664+0000 7fa2fb200640 1 --2- 172.21.15.39:0/1579441835 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa2fc0ffea0 0x7fa2fc199aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:47.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.664+0000 7fa2fb200640 1 -- 172.21.15.39:0/1579441835 >> 172.21.15.39:0/1579441835 conn(0x7fa2fc0fba80 msgr2=0x7fa2fc0fd110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:47.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.664+0000 7fa2fb200640 1 -- 172.21.15.39:0/1579441835 shutdown_connections 2024-03-18T16:20:47.701 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.664+0000 7fa2fb200640 1 -- 172.21.15.39:0/1579441835 wait complete. 2024-03-18T16:20:47.701 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-18T16:20:47.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.908+0000 7f9ed6800640 1 -- 172.21.15.39:0/2436455064 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9ed01039f0 msgr2=0x7f9ed0103e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:47.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.908+0000 7f9ed6800640 1 --2- 172.21.15.39:0/2436455064 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9ed01039f0 0x7f9ed0103e10 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f9ec0009900 tx=0x7f9ec002bc30 comp rx=0 tx=0).stop 2024-03-18T16:20:47.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.909+0000 7f9ed6800640 1 -- 172.21.15.39:0/2436455064 shutdown_connections 2024-03-18T16:20:47.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.909+0000 7f9ed6800640 1 --2- 172.21.15.39:0/2436455064 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9ed01055a0 0x7f9ed010be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:47.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.909+0000 7f9ed6800640 1 --2- 172.21.15.39:0/2436455064 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9ed0104be0 0x7f9ed0105060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:47.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.909+0000 7f9ed6800640 1 --2- 172.21.15.39:0/2436455064 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9ed01039f0 0x7f9ed0103e10 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:47.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.909+0000 7f9ed6800640 1 -- 172.21.15.39:0/2436455064 >> 172.21.15.39:0/2436455064 conn(0x7f9ed00fefa0 msgr2=0x7f9ed01013c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:47.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.909+0000 7f9ed6800640 1 -- 172.21.15.39:0/2436455064 shutdown_connections 2024-03-18T16:20:47.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.909+0000 7f9ed6800640 1 -- 172.21.15.39:0/2436455064 wait complete. 2024-03-18T16:20:47.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.910+0000 7f9ed6800640 1 Processor -- start 2024-03-18T16:20:47.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.910+0000 7f9ed6800640 1 -- start start 2024-03-18T16:20:47.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.910+0000 7f9ed6800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9ed01039f0 0x7f9ed0199cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:47.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.910+0000 7f9ed6800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9ed0104be0 0x7f9ed019a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:47.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.910+0000 7f9ed6800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9ed01055a0 0x7f9ed01a1260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:47.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.910+0000 7f9ed6800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f9ed010f690 con 0x7f9ed01039f0 2024-03-18T16:20:47.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.910+0000 7f9ed6800640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f9ed010f510 con 0x7f9ed01055a0 2024-03-18T16:20:47.918 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.910+0000 7f9ed6800640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f9ed010f810 con 0x7f9ed0104be0 2024-03-18T16:20:47.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.911+0000 7f9ecf400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9ed01039f0 0x7f9ed0199cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:47.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.911+0000 7f9ecfe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9ed01055a0 0x7f9ed01a1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:47.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.911+0000 7f9ecea00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9ed0104be0 0x7f9ed019a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:47.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.911+0000 7f9ecf400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9ed01039f0 0x7f9ed0199cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:60434/0 (socket says 172.21.15.39:60434) 2024-03-18T16:20:47.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.911+0000 7f9ecf400640 1 -- 172.21.15.39:0/2868569497 learned_addr learned my addr 172.21.15.39:0/2868569497 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:20:47.919 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.911+0000 7f9ecfe00640 1 -- 172.21.15.39:0/2868569497 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9ed0104be0 msgr2=0x7f9ed019a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:47.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.911+0000 7f9ecfe00640 1 --2- 172.21.15.39:0/2868569497 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9ed0104be0 0x7f9ed019a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:47.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.911+0000 7f9ecfe00640 1 -- 172.21.15.39:0/2868569497 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9ed01039f0 msgr2=0x7f9ed0199cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:47.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.911+0000 7f9ecfe00640 1 --2- 172.21.15.39:0/2868569497 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9ed01039f0 0x7f9ed0199cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:47.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.911+0000 7f9ecfe00640 1 -- 172.21.15.39:0/2868569497 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ed01a1900 con 0x7f9ed01055a0 2024-03-18T16:20:47.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.911+0000 7f9ecf400640 1 --2- 172.21.15.39:0/2868569497 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9ed01039f0 0x7f9ed0199cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:20:47.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.911+0000 7f9ecfe00640 1 --2- 172.21.15.39:0/2868569497 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9ed01055a0 0x7f9ed01a1260 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f9ec8009fd0 tx=0x7f9ec800ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:47.920 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.912+0000 7f9ec7e00640 1 -- 172.21.15.39:0/2868569497 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ec800ce20 con 0x7f9ed01055a0 2024-03-18T16:20:47.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.912+0000 7f9ec7e00640 1 -- 172.21.15.39:0/2868569497 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ec8005020 con 0x7f9ed01055a0 2024-03-18T16:20:47.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.912+0000 7f9ed6800640 1 -- 172.21.15.39:0/2868569497 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ed01a1bf0 con 0x7f9ed01055a0 2024-03-18T16:20:47.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.912+0000 7f9ec7e00640 1 -- 172.21.15.39:0/2868569497 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ec8010720 con 0x7f9ed01055a0 2024-03-18T16:20:47.921 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.912+0000 7f9ed6800640 1 -- 172.21.15.39:0/2868569497 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ed01a20f0 con 0x7f9ed01055a0 2024-03-18T16:20:47.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.915+0000 7f9ec7e00640 1 -- 172.21.15.39:0/2868569497 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f9ec8005540 con 0x7f9ed01055a0 2024-03-18T16:20:47.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.917+0000 7f9ed946e740 1 Processor -- start 2024-03-18T16:20:47.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.917+0000 7f9ec7e00640 1 --2- 172.21.15.39:0/2868569497 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f9eb40a2800 0x7f9eb40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:47.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.917+0000 7f9ed946e740 1 -- start start 2024-03-18T16:20:47.931 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.917+0000 7f9ec7e00640 1 -- 172.21.15.39:0/2868569497 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f9ec80cf5c0 con 0x7f9ed01055a0 2024-03-18T16:20:47.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.917+0000 7f9ecea00640 1 --2- 172.21.15.39:0/2868569497 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f9eb40a2800 0x7f9eb40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:47.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.917+0000 7f9ed946e740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc679388c0 0x55bc67938ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:47.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.917+0000 7f9ed946e740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bc679392b0 0x55bc6793ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:47.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.917+0000 7f9ed946e740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bc6793e420 0x55bc679408b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:47.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.917+0000 7f9ed946e740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55bc679003b0 con 0x55bc679388c0 2024-03-18T16:20:47.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.917+0000 7f9ed946e740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55bc6789d450 con 0x55bc679392b0 2024-03-18T16:20:47.932 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.917+0000 7f9ed946e740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55bc67845cb0 con 0x55bc6793e420 2024-03-18T16:20:47.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.918+0000 7f9ecea00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bc679392b0 0x55bc6793ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:47.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.918+0000 7f9ecfe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bc6793e420 0x55bc679408b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:47.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.918+0000 7f9ecea00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bc679392b0 0x55bc6793ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:53460/0 (socket says 172.21.15.39:53460) 2024-03-18T16:20:47.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.918+0000 7f9ecea00640 1 -- 172.21.15.39:0/235858615 learned_addr learned my addr 172.21.15.39:0/235858615 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:20:47.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.918+0000 7f9ecf400640 1 --2- 172.21.15.39:0/235858615 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc679388c0 0x55bc67938ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:47.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.918+0000 7f9ecfe00640 1 -- 172.21.15.39:0/235858615 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bc679392b0 msgr2=0x55bc6793ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:47.933 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.918+0000 7f9ecfe00640 1 --2- 172.21.15.39:0/235858615 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bc679392b0 0x55bc6793ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:47.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.918+0000 7f9ecfe00640 1 -- 172.21.15.39:0/235858615 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc679388c0 msgr2=0x55bc67938ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:47.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.918+0000 7f9ecfe00640 1 --2- 172.21.15.39:0/235858615 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc679388c0 0x55bc67938ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:47.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.918+0000 7f9ecfe00640 1 -- 172.21.15.39:0/235858615 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bc678671f0 con 0x55bc6793e420 2024-03-18T16:20:47.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.919+0000 7f9ecfe00640 1 --2- 172.21.15.39:0/235858615 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bc6793e420 0x55bc679408b0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f9ec809b870 tx=0x7f9ec809c270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d7779464f98401b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:47.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.919+0000 7f9ecea00640 1 --2- 172.21.15.39:0/2868569497 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f9eb40a2800 0x7f9eb40a4cc0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f9ebc013a30 tx=0x7f9ebc009bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:47.934 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.919+0000 7f9ec4c00640 1 -- 172.21.15.39:0/235858615 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ec80a4310 con 0x55bc6793e420 2024-03-18T16:20:47.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.919+0000 7f9ec4c00640 1 -- 172.21.15.39:0/235858615 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ec8099e70 con 0x55bc6793e420 2024-03-18T16:20:47.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.919+0000 7f9ec4c00640 1 -- 172.21.15.39:0/235858615 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ec80a3e80 con 0x55bc6793e420 2024-03-18T16:20:47.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.920+0000 7f9ed946e740 1 -- 172.21.15.39:0/235858615 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bc6793e420 msgr2=0x55bc679408b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:47.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.920+0000 7f9ed946e740 1 --2- 172.21.15.39:0/235858615 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bc6793e420 0x55bc679408b0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f9ec809b870 tx=0x7f9ec809c270 comp rx=0 tx=0).stop 2024-03-18T16:20:47.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.920+0000 7f9ed946e740 1 -- 172.21.15.39:0/235858615 shutdown_connections 2024-03-18T16:20:47.935 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.920+0000 7f9ed946e740 1 --2- 172.21.15.39:0/235858615 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bc679392b0 0x55bc6793ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:47.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.920+0000 7f9ed946e740 1 --2- 172.21.15.39:0/235858615 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bc6793e420 0x55bc679408b0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:47.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.920+0000 7f9ed946e740 1 --2- 172.21.15.39:0/235858615 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc679388c0 0x55bc67938ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:47.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.920+0000 7f9ed946e740 1 -- 172.21.15.39:0/235858615 >> 172.21.15.39:0/235858615 conn(0x55bc67933e30 msgr2=0x55bc67934230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:47.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.920+0000 7f9ed946e740 1 -- 172.21.15.39:0/235858615 shutdown_connections 2024-03-18T16:20:47.936 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.920+0000 7f9ed946e740 1 -- 172.21.15.39:0/235858615 wait complete. 2024-03-18T16:20:47.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.920+0000 7f9ed946e740 1 -- start start 2024-03-18T16:20:47.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.921+0000 7f9ed946e740 1 Processor -- start 2024-03-18T16:20:47.975 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.921+0000 7f9ed946e740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bc679388c0 0x55bc67942b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:47.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.921+0000 7f9ed946e740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bc679392b0 0x55bc679430d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:47.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.921+0000 7f9ed946e740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc6793e420 0x55bc6794a110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:47.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.921+0000 7f9ed946e740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55bc67859110 con 0x55bc6793e420 2024-03-18T16:20:47.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.921+0000 7f9ed946e740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55bc678994f0 con 0x55bc679388c0 2024-03-18T16:20:47.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.921+0000 7f9ed946e740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55bc679413d0 con 0x55bc679392b0 2024-03-18T16:20:47.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.921+0000 7f9ed946e740 10 client.0 _ll_register_callbacks cb 0x55bc67846b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:20:47.976 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.922+0000 7f9ecea00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bc679392b0 0x55bc679430d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:47.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.922+0000 7f9ecf400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bc679388c0 0x55bc67942b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:47.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.922+0000 7f9ecea00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bc679392b0 0x55bc679430d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:45174/0 (socket says 172.21.15.39:45174) 2024-03-18T16:20:47.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.922+0000 7f9ecea00640 1 -- 172.21.15.39:0/2150753596 learned_addr learned my addr 172.21.15.39:0/2150753596 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:20:47.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.922+0000 7f9ecfe00640 1 --2- 172.21.15.39:0/2150753596 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc6793e420 0x55bc6794a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:47.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.922+0000 7f9ecea00640 1 -- 172.21.15.39:0/2150753596 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bc679388c0 msgr2=0x55bc67942b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:47.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.922+0000 7f9ecea00640 1 --2- 172.21.15.39:0/2150753596 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bc679388c0 0x55bc67942b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:47.977 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.922+0000 7f9ecea00640 1 -- 172.21.15.39:0/2150753596 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc6793e420 msgr2=0x55bc6794a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:47.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.922+0000 7f9ecea00640 1 --2- 172.21.15.39:0/2150753596 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc6793e420 0x55bc6794a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:47.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.922+0000 7f9ecea00640 1 -- 172.21.15.39:0/2150753596 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bc675b13f0 con 0x55bc679392b0 2024-03-18T16:20:47.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.922+0000 7f9ecea00640 1 --2- 172.21.15.39:0/2150753596 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bc679392b0 0x55bc679430d0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f9ebc00ff80 tx=0x7f9ebc004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5db21bc139b17124 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:47.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.922+0000 7f9eb9600640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:20:47.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.923+0000 7f9eb9600640 1 -- 172.21.15.39:0/2150753596 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ebc025070 con 0x55bc679392b0 2024-03-18T16:20:47.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.923+0000 7f9ed946e740 1 -- 172.21.15.39:0/2150753596 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9ec8099e70 con 0x55bc679392b0 2024-03-18T16:20:47.978 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.923+0000 7f9eb9600640 1 -- 172.21.15.39:0/2150753596 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f9ebc003d80 con 0x55bc679392b0 2024-03-18T16:20:47.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.923+0000 7f9eb9600640 1 -- 172.21.15.39:0/2150753596 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f9ebc031050 con 0x55bc679392b0 2024-03-18T16:20:47.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.923+0000 7f9ed946e740 10 client.27784 fetch_fsmap learned FSMap version 48 2024-03-18T16:20:47.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.923+0000 7f9ed946e740 1 -- 172.21.15.39:0/2150753596 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x55bc6767b7d0 con 0x55bc679392b0 2024-03-18T16:20:47.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.923+0000 7f9eb9600640 1 -- 172.21.15.39:0/2150753596 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ec8099e70 con 0x55bc679392b0 2024-03-18T16:20:47.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.923+0000 7f9eb9600640 1 -- 172.21.15.39:0/2150753596 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 48) ==== 8386+0+0 (secure 0 0 0) 0x7f9ebc01f040 con 0x55bc679392b0 2024-03-18T16:20:47.979 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.923+0000 7f9ed946e740 10 client.27784 fetch_fsmap finished waiting for FSMap version 48 2024-03-18T16:20:47.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.923+0000 7f9ed946e740 10 client.27784 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-18T16:20:47.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.923+0000 7f9ed946e740 20 client.27784 populate_metadata read hostname 'smithi039' 2024-03-18T16:20:47.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.924+0000 7f9ed946e740 1 --2- 172.21.15.39:0/2150753596 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x55bc6794eb90 0x55bc67950fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:47.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.924+0000 7f9ed946e740 4 client.27784 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:20:47.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.924+0000 7f9ed946e740 1 -- 172.21.15.39:0/2150753596 --> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bc678aa990 con 0x55bc6794eb90 2024-03-18T16:20:47.980 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.924+0000 7f9ecf400640 1 --2- 172.21.15.39:0/2150753596 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x55bc6794eb90 0x55bc67950fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:47.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.925+0000 7f9ecf400640 1 --2- 172.21.15.39:0/2150753596 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x55bc6794eb90 0x55bc67950fd0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=906c8b8ccacbdd72 server_cookie=2615e2a7354fa3a3 in_seq=0 out_seq=0 2024-03-18T16:20:47.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.925+0000 7f9eb9600640 10 client.27784 ms_handle_connect on v2:172.21.15.194:6836/1301464246 2024-03-18T16:20:47.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.926+0000 7f9eb9600640 1 -- 172.21.15.39:0/2150753596 <== mds.3 v2:172.21.15.194:6836/1301464246 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55bc678aa990 con 0x55bc6794eb90 2024-03-18T16:20:47.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.926+0000 7f9eb9600640 10 client.27784 handle_command_reply: tid=0 2024-03-18T16:20:47.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.926+0000 7f9ed946e740 1 client.27784 shutdown 2024-03-18T16:20:47.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.926+0000 7f9ed946e740 1 -- 172.21.15.39:0/2150753596 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bc679392b0 msgr2=0x55bc679430d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:47.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.926+0000 7f9ed946e740 1 --2- 172.21.15.39:0/2150753596 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bc679392b0 0x55bc679430d0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f9ebc00ff80 tx=0x7f9ebc004730 comp rx=0 tx=0).stop 2024-03-18T16:20:47.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.926+0000 7f9ed946e740 1 -- 172.21.15.39:0/2150753596 shutdown_connections 2024-03-18T16:20:47.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.926+0000 7f9ed946e740 1 --2- 172.21.15.39:0/2150753596 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x55bc6794eb90 0x55bc67950fd0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:47.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.926+0000 7f9ed946e740 1 --2- 172.21.15.39:0/2150753596 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc6793e420 0x55bc6794a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:47.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.926+0000 7f9ed946e740 1 --2- 172.21.15.39:0/2150753596 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bc679392b0 0x55bc679430d0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:47.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.926+0000 7f9eb9600640 0 client.27784 ms_handle_reset on v2:172.21.15.194:6836/1301464246 2024-03-18T16:20:47.981 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.926+0000 7f9ed946e740 1 --2- 172.21.15.39:0/2150753596 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bc679388c0 0x55bc67942b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:47.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.926+0000 7f9ed946e740 1 -- 172.21.15.39:0/2150753596 >> 172.21.15.39:0/2150753596 conn(0x55bc67933e30 msgr2=0x55bc679331f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:47.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.926+0000 7f9ed946e740 1 -- 172.21.15.39:0/2150753596 shutdown_connections 2024-03-18T16:20:47.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.926+0000 7f9ed946e740 1 -- 172.21.15.39:0/2150753596 wait complete. 2024-03-18T16:20:47.982 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.927+0000 7f9ed946e740 20 client.27784 trim_cache size 0 max 16384 2024-03-18T16:20:47.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.949+0000 7f9ed946e740 1 Processor -- start 2024-03-18T16:20:47.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.949+0000 7f9ed946e740 1 -- start start 2024-03-18T16:20:47.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.949+0000 7f9ed946e740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bc679388c0 0x55bc67933070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:47.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.949+0000 7f9ed946e740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bc679392b0 0x55bc6794a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:47.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.949+0000 7f9ed946e740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc6793e420 0x55bc679558d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:47.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.949+0000 7f9ed946e740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f9ed01ac5b0 con 0x55bc6793e420 2024-03-18T16:20:47.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.949+0000 7f9ed946e740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f9ed01ae1b0 con 0x55bc679388c0 2024-03-18T16:20:48.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.949+0000 7f9ed946e740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f9ed01acc80 con 0x55bc679392b0 2024-03-18T16:20:48.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.950+0000 7f9ecea00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bc679392b0 0x55bc6794a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:48.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.950+0000 7f9ecea00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bc679392b0 0x55bc6794a0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:45190/0 (socket says 172.21.15.39:45190) 2024-03-18T16:20:48.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.950+0000 7f9ecf400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bc679388c0 0x55bc67933070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:48.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.950+0000 7f9ecea00640 1 -- 172.21.15.39:0/2864902959 learned_addr learned my addr 172.21.15.39:0/2864902959 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:20:48.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.950+0000 7f9ecfe00640 1 --2- 172.21.15.39:0/2864902959 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc6793e420 0x55bc679558d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:48.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.950+0000 7f9ecea00640 1 -- 172.21.15.39:0/2864902959 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bc679388c0 msgr2=0x55bc67933070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:48.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.950+0000 7f9ecea00640 1 --2- 172.21.15.39:0/2864902959 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bc679388c0 0x55bc67933070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.950+0000 7f9ecea00640 1 -- 172.21.15.39:0/2864902959 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc6793e420 msgr2=0x55bc679558d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:48.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.950+0000 7f9ecea00640 1 --2- 172.21.15.39:0/2864902959 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc6793e420 0x55bc679558d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.950+0000 7f9ecea00640 1 -- 172.21.15.39:0/2864902959 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bc6792f670 con 0x55bc679392b0 2024-03-18T16:20:48.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.950+0000 7f9ecea00640 1 --2- 172.21.15.39:0/2864902959 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bc679392b0 0x55bc6794a0d0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f9ebc010ad0 tx=0x7f9ebc00fdf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7be3264a8c42f85b server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:48.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.950+0000 7f9ebbe00640 1 -- 172.21.15.39:0/2864902959 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ebc025070 con 0x55bc679392b0 2024-03-18T16:20:48.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.951+0000 7f9ebbe00640 1 -- 172.21.15.39:0/2864902959 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ebc0039c0 con 0x55bc679392b0 2024-03-18T16:20:48.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.951+0000 7f9ebbe00640 1 -- 172.21.15.39:0/2864902959 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ebc0212d0 con 0x55bc679392b0 2024-03-18T16:20:48.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.952+0000 7f9ed946e740 1 -- 172.21.15.39:0/2864902959 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bc679392b0 msgr2=0x55bc6794a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:48.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.952+0000 7f9ed946e740 1 --2- 172.21.15.39:0/2864902959 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bc679392b0 0x55bc6794a0d0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f9ebc010ad0 tx=0x7f9ebc00fdf0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.952+0000 7f9ed946e740 1 -- 172.21.15.39:0/2864902959 shutdown_connections 2024-03-18T16:20:48.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.952+0000 7f9ed946e740 1 --2- 172.21.15.39:0/2864902959 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc6793e420 0x55bc679558d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.952+0000 7f9ed946e740 1 --2- 172.21.15.39:0/2864902959 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bc679392b0 0x55bc6794a0d0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.952+0000 7f9ed946e740 1 --2- 172.21.15.39:0/2864902959 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bc679388c0 0x55bc67933070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.952+0000 7f9ed946e740 1 -- 172.21.15.39:0/2864902959 >> 172.21.15.39:0/2864902959 conn(0x55bc67933e30 msgr2=0x55bc67932960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:48.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.952+0000 7f9ed946e740 1 -- 172.21.15.39:0/2864902959 shutdown_connections 2024-03-18T16:20:48.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.952+0000 7f9ed946e740 1 -- 172.21.15.39:0/2864902959 wait complete. 2024-03-18T16:20:48.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.952+0000 7f9ed946e740 1 -- start start 2024-03-18T16:20:48.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.953+0000 7f9ed946e740 1 Processor -- start 2024-03-18T16:20:48.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.953+0000 7f9ed946e740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bc679388c0 0x55bc67947b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:48.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.953+0000 7f9ed946e740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bc679392b0 0x55bc679480d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:48.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.953+0000 7f9ed946e740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc6793e420 0x55bc67949090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:48.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.953+0000 7f9ed946e740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f9ed01adab0 con 0x55bc6793e420 2024-03-18T16:20:48.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.953+0000 7f9ed946e740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f9ed01ad3b0 con 0x55bc679392b0 2024-03-18T16:20:48.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.953+0000 7f9ed946e740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f9ed01aa590 con 0x55bc679388c0 2024-03-18T16:20:48.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.954+0000 7f9ed946e740 10 client.0 _ll_register_callbacks cb 0x7f9ebc00fe60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:20:48.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.954+0000 7f9ecea00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bc679392b0 0x55bc679480d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:48.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.954+0000 7f9ecea00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bc679392b0 0x55bc679480d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:53498/0 (socket says 172.21.15.39:53498) 2024-03-18T16:20:48.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.954+0000 7f9ecea00640 1 -- 172.21.15.39:0/125883850 learned_addr learned my addr 172.21.15.39:0/125883850 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:20:48.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.954+0000 7f9ecf400640 1 --2- 172.21.15.39:0/125883850 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bc679388c0 0x55bc67947b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:48.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.954+0000 7f9ecfe00640 1 --2- 172.21.15.39:0/125883850 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc6793e420 0x55bc67949090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:48.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.954+0000 7f9ecea00640 1 -- 172.21.15.39:0/125883850 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bc679388c0 msgr2=0x55bc67947b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:48.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.954+0000 7f9ecea00640 1 --2- 172.21.15.39:0/125883850 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bc679388c0 0x55bc67947b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.010 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.954+0000 7f9ecea00640 1 -- 172.21.15.39:0/125883850 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc6793e420 msgr2=0x55bc67949090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:48.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.954+0000 7f9ecea00640 1 --2- 172.21.15.39:0/125883850 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc6793e420 0x55bc67949090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.954+0000 7f9ecea00640 1 -- 172.21.15.39:0/125883850 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bc67949680 con 0x55bc679392b0 2024-03-18T16:20:48.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.954+0000 7f9ecf400640 1 --2- 172.21.15.39:0/125883850 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bc679388c0 0x55bc67947b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:20:48.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.954+0000 7f9ecea00640 1 --2- 172.21.15.39:0/125883850 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bc679392b0 0x55bc679480d0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f9ebc010bf0 tx=0x7f9ebc0106e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d80ba5f1694bfa68 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:48.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.955+0000 7f9eb9600640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:20:48.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.955+0000 7f9eb9600640 1 -- 172.21.15.39:0/125883850 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ebc025050 con 0x55bc679392b0 2024-03-18T16:20:48.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.955+0000 7f9ed946e740 1 -- 172.21.15.39:0/125883850 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9ebc0039c0 con 0x55bc679392b0 2024-03-18T16:20:48.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.955+0000 7f9eb9600640 1 -- 172.21.15.39:0/125883850 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f9ebc005450 con 0x55bc679392b0 2024-03-18T16:20:48.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.955+0000 7f9eb9600640 1 -- 172.21.15.39:0/125883850 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f9ebc046050 con 0x55bc679392b0 2024-03-18T16:20:48.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.955+0000 7f9ed946e740 10 client.27803 fetch_fsmap learned FSMap version 48 2024-03-18T16:20:48.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.955+0000 7f9ed946e740 1 -- 172.21.15.39:0/125883850 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x55bc679b52c0 con 0x55bc679392b0 2024-03-18T16:20:48.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.956+0000 7f9eb9600640 1 -- 172.21.15.39:0/125883850 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ebc0039c0 con 0x55bc679392b0 2024-03-18T16:20:48.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.956+0000 7f9eb9600640 1 -- 172.21.15.39:0/125883850 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 48) ==== 8386+0+0 (secure 0 0 0) 0x7f9ebc020050 con 0x55bc679392b0 2024-03-18T16:20:48.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.956+0000 7f9ed946e740 10 client.27803 fetch_fsmap finished waiting for FSMap version 48 2024-03-18T16:20:48.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.956+0000 7f9ed946e740 10 client.27803 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-03-18T16:20:48.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.956+0000 7f9ed946e740 20 client.27803 populate_metadata read hostname 'smithi039' 2024-03-18T16:20:48.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.956+0000 7f9ed946e740 1 --2- 172.21.15.39:0/125883850 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x55bc679b5c60 0x55bc679b8080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:48.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.956+0000 7f9ed946e740 4 client.27803 mds_command: new command op to 24524 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:20:48.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.956+0000 7f9ed946e740 1 -- 172.21.15.39:0/125883850 --> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55bc678a4ce0 con 0x55bc679b5c60 2024-03-18T16:20:48.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.956+0000 7f9ecf400640 1 --2- 172.21.15.39:0/125883850 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x55bc679b5c60 0x55bc679b8080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:48.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.957+0000 7f9ecf400640 1 --2- 172.21.15.39:0/125883850 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x55bc679b5c60 0x55bc679b8080 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=a14cdf76f694ee65 server_cookie=5fb5111b52196d38 in_seq=0 out_seq=0 2024-03-18T16:20:48.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.957+0000 7f9eb9600640 10 client.27803 ms_handle_connect on v2:172.21.15.194:6836/1301464246 2024-03-18T16:20:48.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.957+0000 7f9eb9600640 1 -- 172.21.15.39:0/125883850 <== mds.3 v2:172.21.15.194:6836/1301464246 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55bc678a4ce0 con 0x55bc679b5c60 2024-03-18T16:20:48.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.957+0000 7f9eb9600640 10 client.27803 handle_command_reply: tid=0 2024-03-18T16:20:48.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.957+0000 7f9ed946e740 1 client.27803 shutdown 2024-03-18T16:20:48.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.957+0000 7f9ed946e740 1 -- 172.21.15.39:0/125883850 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bc679392b0 msgr2=0x55bc679480d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:48.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.957+0000 7f9ed946e740 1 --2- 172.21.15.39:0/125883850 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bc679392b0 0x55bc679480d0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f9ebc010bf0 tx=0x7f9ebc0106e0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.958+0000 7f9ed946e740 1 -- 172.21.15.39:0/125883850 shutdown_connections 2024-03-18T16:20:48.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.958+0000 7f9ed946e740 1 --2- 172.21.15.39:0/125883850 >> [v2:172.21.15.194:6836/1301464246,v1:172.21.15.194:6837/1301464246] conn(0x55bc679b5c60 0x55bc679b8080 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.958+0000 7f9ed946e740 1 --2- 172.21.15.39:0/125883850 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bc679392b0 0x55bc679480d0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.958+0000 7f9eb9600640 0 client.27803 ms_handle_reset on v2:172.21.15.194:6836/1301464246 2024-03-18T16:20:48.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.958+0000 7f9ed946e740 1 --2- 172.21.15.39:0/125883850 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x55bc6793e420 0x55bc67949090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.958+0000 7f9ed946e740 1 --2- 172.21.15.39:0/125883850 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55bc679388c0 0x55bc67947b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.958+0000 7f9ed946e740 1 -- 172.21.15.39:0/125883850 >> 172.21.15.39:0/125883850 conn(0x55bc67933e30 msgr2=0x55bc679331f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:48.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.958+0000 7f9ed946e740 1 -- 172.21.15.39:0/125883850 shutdown_connections 2024-03-18T16:20:48.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.958+0000 7f9ed946e740 1 -- 172.21.15.39:0/125883850 wait complete. 2024-03-18T16:20:48.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.958+0000 7f9ed946e740 20 client.27803 trim_cache size 0 max 16384 2024-03-18T16:20:48.014 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:20:48.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.959+0000 7f9ed6800640 1 -- 172.21.15.39:0/2868569497 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f9eb40a2800 msgr2=0x7f9eb40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:48.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.959+0000 7f9ed6800640 1 --2- 172.21.15.39:0/2868569497 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f9eb40a2800 0x7f9eb40a4cc0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f9ebc013a30 tx=0x7f9ebc009bc0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.959+0000 7f9ed6800640 1 -- 172.21.15.39:0/2868569497 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9ed01055a0 msgr2=0x7f9ed01a1260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:48.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.959+0000 7f9ed6800640 1 --2- 172.21.15.39:0/2868569497 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9ed01055a0 0x7f9ed01a1260 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f9ec8009fd0 tx=0x7f9ec800ef10 comp rx=0 tx=0).stop 2024-03-18T16:20:48.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.959+0000 7f9ed6800640 1 -- 172.21.15.39:0/2868569497 shutdown_connections 2024-03-18T16:20:48.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.959+0000 7f9ed6800640 1 --2- 172.21.15.39:0/2868569497 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9ed01055a0 0x7f9ed01a1260 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.959+0000 7f9ed6800640 1 --2- 172.21.15.39:0/2868569497 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f9eb40a2800 0x7f9eb40a4cc0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.959+0000 7f9ed6800640 1 --2- 172.21.15.39:0/2868569497 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9ed0104be0 0x7f9ed019a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.959+0000 7f9ed6800640 1 --2- 172.21.15.39:0/2868569497 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f9ed01039f0 0x7f9ed0199cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.959+0000 7f9ed6800640 1 -- 172.21.15.39:0/2868569497 >> 172.21.15.39:0/2868569497 conn(0x7f9ed00fefa0 msgr2=0x7f9ed0107ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:48.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.959+0000 7f9ed6800640 1 -- 172.21.15.39:0/2868569497 shutdown_connections 2024-03-18T16:20:48.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:47.959+0000 7f9ed6800640 1 -- 172.21.15.39:0/2868569497 wait complete. 2024-03-18T16:20:48.016 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-18T16:20:48.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.218+0000 7f602be00640 1 -- 172.21.15.39:0/2759819142 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f60240761a0 msgr2=0x7f60240765c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:48.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.218+0000 7f602be00640 1 --2- 172.21.15.39:0/2759819142 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f60240761a0 0x7f60240765c0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f6018009900 tx=0x7f601802bc30 comp rx=0 tx=0).stop 2024-03-18T16:20:48.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.218+0000 7f602be00640 1 -- 172.21.15.39:0/2759819142 shutdown_connections 2024-03-18T16:20:48.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.218+0000 7f602be00640 1 --2- 172.21.15.39:0/2759819142 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f60240773e0 0x7f6024077840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.218+0000 7f602be00640 1 --2- 172.21.15.39:0/2759819142 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6024077d80 0x7f6024078620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.218+0000 7f602be00640 1 --2- 172.21.15.39:0/2759819142 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f60240761a0 0x7f60240765c0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.218+0000 7f602be00640 1 -- 172.21.15.39:0/2759819142 >> 172.21.15.39:0/2759819142 conn(0x7f60240fefa0 msgr2=0x7f60241013c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:48.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.218+0000 7f602be00640 1 -- 172.21.15.39:0/2759819142 shutdown_connections 2024-03-18T16:20:48.221 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.218+0000 7f602be00640 1 -- 172.21.15.39:0/2759819142 wait complete. 2024-03-18T16:20:48.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.219+0000 7f602be00640 1 Processor -- start 2024-03-18T16:20:48.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.219+0000 7f602be00640 1 -- start start 2024-03-18T16:20:48.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.219+0000 7f602be00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f60240761a0 0x7f602419e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:48.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.219+0000 7f602be00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f60240773e0 0x7f602419e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:48.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.219+0000 7f602be00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6024077d80 0x7f60241a5620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:48.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.219+0000 7f602be00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f6024113a80 con 0x7f60240773e0 2024-03-18T16:20:48.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.219+0000 7f602be00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f6024113900 con 0x7f60240761a0 2024-03-18T16:20:48.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.219+0000 7f602be00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f6024113c00 con 0x7f6024077d80 2024-03-18T16:20:48.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.220+0000 7f6029400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f60240761a0 0x7f602419e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:48.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.220+0000 7f6029e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6024077d80 0x7f60241a5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:48.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.220+0000 7f6029400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f60240761a0 0x7f602419e070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:38988/0 (socket says 172.21.15.39:38988) 2024-03-18T16:20:48.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.220+0000 7f6028a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f60240773e0 0x7f602419e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:48.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.220+0000 7f6029400640 1 -- 172.21.15.39:0/885741760 learned_addr learned my addr 172.21.15.39:0/885741760 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:20:48.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.220+0000 7f6029400640 1 -- 172.21.15.39:0/885741760 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6024077d80 msgr2=0x7f60241a5620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:48.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.220+0000 7f6029400640 1 --2- 172.21.15.39:0/885741760 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6024077d80 0x7f60241a5620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.220+0000 7f6029400640 1 -- 172.21.15.39:0/885741760 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f60240773e0 msgr2=0x7f602419e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:48.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.220+0000 7f6029400640 1 --2- 172.21.15.39:0/885741760 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f60240773e0 0x7f602419e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.220+0000 7f6029400640 1 -- 172.21.15.39:0/885741760 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60241a5c90 con 0x7f60240761a0 2024-03-18T16:20:48.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.220+0000 7f6029e00640 1 --2- 172.21.15.39:0/885741760 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6024077d80 0x7f60241a5620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:20:48.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.221+0000 7f6029400640 1 --2- 172.21.15.39:0/885741760 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f60240761a0 0x7f602419e070 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f6018000c00 tx=0x7f601802c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:48.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.221+0000 7f6022000640 1 -- 172.21.15.39:0/885741760 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6018043020 con 0x7f60240761a0 2024-03-18T16:20:48.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.221+0000 7f6022000640 1 -- 172.21.15.39:0/885741760 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f60180054e0 con 0x7f60240761a0 2024-03-18T16:20:48.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.221+0000 7f602be00640 1 -- 172.21.15.39:0/885741760 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60241a5f20 con 0x7f60240761a0 2024-03-18T16:20:48.225 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.221+0000 7f6022000640 1 -- 172.21.15.39:0/885741760 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6018003c40 con 0x7f60240761a0 2024-03-18T16:20:48.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.221+0000 7f602be00640 1 -- 172.21.15.39:0/885741760 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60241a6380 con 0x7f60240761a0 2024-03-18T16:20:48.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.225+0000 7f602ec3a740 1 Processor -- start 2024-03-18T16:20:48.227 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.225+0000 7f602ec3a740 1 -- start start 2024-03-18T16:20:48.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.225+0000 7f602ec3a740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5587004fd8c0 0x5587004fdce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:48.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.225+0000 7f602ec3a740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5587004fe2b0 0x558700502db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:48.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.225+0000 7f602ec3a740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558700503420 0x5587005058b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:48.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.225+0000 7f602ec3a740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5587004c53b0 con 0x558700503420 2024-03-18T16:20:48.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.225+0000 7f602ec3a740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x558700462450 con 0x5587004fd8c0 2024-03-18T16:20:48.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.225+0000 7f602ec3a740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55870040acb0 con 0x5587004fe2b0 2024-03-18T16:20:48.229 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.225+0000 7f6022000640 1 -- 172.21.15.39:0/885741760 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f6018046020 con 0x7f60240761a0 2024-03-18T16:20:48.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.226+0000 7f6029e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5587004fd8c0 0x5587004fdce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:48.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.226+0000 7f6029e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5587004fd8c0 0x5587004fdce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:38992/0 (socket says 172.21.15.39:38992) 2024-03-18T16:20:48.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.226+0000 7f6029e00640 1 -- 172.21.15.39:0/584542685 learned_addr learned my addr 172.21.15.39:0/584542685 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:20:48.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.226+0000 7f6028a00640 1 --2- 172.21.15.39:0/584542685 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558700503420 0x5587005058b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:48.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.226+0000 7f6029400640 1 --2- 172.21.15.39:0/584542685 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5587004fe2b0 0x558700502db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:48.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.226+0000 7f6029e00640 1 -- 172.21.15.39:0/584542685 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5587004fe2b0 msgr2=0x558700502db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:48.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.226+0000 7f6029e00640 1 --2- 172.21.15.39:0/584542685 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5587004fe2b0 0x558700502db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.226+0000 7f6029e00640 1 -- 172.21.15.39:0/584542685 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558700503420 msgr2=0x5587005058b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:48.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.226+0000 7f6029e00640 1 --2- 172.21.15.39:0/584542685 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558700503420 0x5587005058b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.226+0000 7f6029e00640 1 -- 172.21.15.39:0/584542685 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55870042c1f0 con 0x5587004fd8c0 2024-03-18T16:20:48.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.226+0000 7f6028a00640 1 --2- 172.21.15.39:0/584542685 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558700503420 0x5587005058b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:20:48.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.227+0000 7f6029e00640 1 --2- 172.21.15.39:0/584542685 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5587004fd8c0 0x5587004fdce0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f601c0117e0 tx=0x7f601c011ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=49e9e6ce0eb973f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:48.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.227+0000 7f6016600640 1 -- 172.21.15.39:0/584542685 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f601c011310 con 0x5587004fd8c0 2024-03-18T16:20:48.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.227+0000 7f6016600640 1 -- 172.21.15.39:0/584542685 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f601c003e50 con 0x5587004fd8c0 2024-03-18T16:20:48.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.227+0000 7f6016600640 1 -- 172.21.15.39:0/584542685 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f601c004620 con 0x5587004fd8c0 2024-03-18T16:20:48.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.227+0000 7f6022000640 1 --2- 172.21.15.39:0/885741760 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f600c0a2800 0x7f600c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:48.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.228+0000 7f6022000640 1 -- 172.21.15.39:0/885741760 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f60180f0e90 con 0x7f60240761a0 2024-03-18T16:20:48.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.228+0000 7f602ec3a740 1 -- 172.21.15.39:0/584542685 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5587004fd8c0 msgr2=0x5587004fdce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:48.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.228+0000 7f602ec3a740 1 --2- 172.21.15.39:0/584542685 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5587004fd8c0 0x5587004fdce0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f601c0117e0 tx=0x7f601c011ca0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.228+0000 7f6029400640 1 --2- 172.21.15.39:0/885741760 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f600c0a2800 0x7f600c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:48.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.228+0000 7f602ec3a740 1 -- 172.21.15.39:0/584542685 shutdown_connections 2024-03-18T16:20:48.235 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.228+0000 7f602ec3a740 1 --2- 172.21.15.39:0/584542685 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x558700503420 0x5587005058b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.228+0000 7f602ec3a740 1 --2- 172.21.15.39:0/584542685 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5587004fe2b0 0x558700502db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.228+0000 7f602ec3a740 1 --2- 172.21.15.39:0/584542685 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5587004fd8c0 0x5587004fdce0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.228+0000 7f602ec3a740 1 -- 172.21.15.39:0/584542685 >> 172.21.15.39:0/584542685 conn(0x5587004f8e30 msgr2=0x5587004f9230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:48.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.228+0000 7f602ec3a740 1 -- 172.21.15.39:0/584542685 shutdown_connections 2024-03-18T16:20:48.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.228+0000 7f602ec3a740 1 -- 172.21.15.39:0/584542685 wait complete. 2024-03-18T16:20:48.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.229+0000 7f602ec3a740 1 -- start start 2024-03-18T16:20:48.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.229+0000 7f6029400640 1 --2- 172.21.15.39:0/885741760 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f600c0a2800 0x7f600c0a4cc0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f60180bd3d0 tx=0x7f60180b9040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:48.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.229+0000 7f602ec3a740 1 Processor -- start 2024-03-18T16:20:48.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.229+0000 7f602ec3a740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5587004fd8c0 0x558700507c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:48.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.229+0000 7f602ec3a740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5587004fe2b0 0x558700508160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:48.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.229+0000 7f602ec3a740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558700503420 0x55870050f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:48.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.229+0000 7f602ec3a740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55870041e110 con 0x5587004fd8c0 2024-03-18T16:20:48.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.229+0000 7f602ec3a740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55870045e4f0 con 0x5587004fe2b0 2024-03-18T16:20:48.236 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.229+0000 7f602ec3a740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x5587005063d0 con 0x558700503420 2024-03-18T16:20:48.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.229+0000 7f602ec3a740 10 client.0 _ll_register_callbacks cb 0x55870040bb70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:20:48.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.230+0000 7f6028a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5587004fe2b0 0x558700508160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:48.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.230+0000 7f6029e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5587004fd8c0 0x558700507c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:48.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.230+0000 7f6028a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5587004fe2b0 0x558700508160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:38996/0 (socket says 172.21.15.39:38996) 2024-03-18T16:20:48.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.230+0000 7f6028a00640 1 -- 172.21.15.39:0/2272355974 learned_addr learned my addr 172.21.15.39:0/2272355974 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:20:48.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.230+0000 7f6029e00640 1 --2- 172.21.15.39:0/2272355974 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558700503420 0x55870050f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:48.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.230+0000 7f6028a00640 1 -- 172.21.15.39:0/2272355974 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558700503420 msgr2=0x55870050f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:48.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.230+0000 7f6028a00640 1 --2- 172.21.15.39:0/2272355974 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558700503420 0x55870050f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.230+0000 7f6028a00640 1 -- 172.21.15.39:0/2272355974 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5587004fd8c0 msgr2=0x558700507c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:48.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.230+0000 7f6028a00640 1 --2- 172.21.15.39:0/2272355974 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5587004fd8c0 0x558700507c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.230+0000 7f6028a00640 1 -- 172.21.15.39:0/2272355974 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5587001763f0 con 0x5587004fe2b0 2024-03-18T16:20:48.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.230+0000 7f6028a00640 1 --2- 172.21.15.39:0/2272355974 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5587004fe2b0 0x558700508160 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f6010012700 tx=0x7f6010012bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f6d504b35aff05b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:48.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.230+0000 7f600aa00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:20:48.237 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.231+0000 7f600aa00640 1 -- 172.21.15.39:0/2272355974 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f601000f9a0 con 0x5587004fe2b0 2024-03-18T16:20:48.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.231+0000 7f600aa00640 1 -- 172.21.15.39:0/2272355974 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f601000fb40 con 0x5587004fe2b0 2024-03-18T16:20:48.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.231+0000 7f602ec3a740 1 -- 172.21.15.39:0/2272355974 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f601c003e50 con 0x5587004fe2b0 2024-03-18T16:20:48.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.231+0000 7f600aa00640 1 -- 172.21.15.39:0/2272355974 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f6010024050 con 0x5587004fe2b0 2024-03-18T16:20:48.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.231+0000 7f602ec3a740 10 client.27815 fetch_fsmap learned FSMap version 48 2024-03-18T16:20:48.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.231+0000 7f602ec3a740 1 -- 172.21.15.39:0/2272355974 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x5587002407d0 con 0x5587004fe2b0 2024-03-18T16:20:48.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.231+0000 7f600aa00640 1 -- 172.21.15.39:0/2272355974 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f601c003e50 con 0x5587004fe2b0 2024-03-18T16:20:48.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.231+0000 7f600aa00640 1 -- 172.21.15.39:0/2272355974 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 48) ==== 8386+0+0 (secure 0 0 0) 0x7f601001a370 con 0x5587004fe2b0 2024-03-18T16:20:48.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.232+0000 7f602ec3a740 10 client.27815 fetch_fsmap finished waiting for FSMap version 48 2024-03-18T16:20:48.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.232+0000 7f602ec3a740 10 client.27815 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-18T16:20:48.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.232+0000 7f602ec3a740 20 client.27815 populate_metadata read hostname 'smithi039' 2024-03-18T16:20:48.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.232+0000 7f602ec3a740 1 --2- 172.21.15.39:0/2272355974 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x558700513b10 0x558700515f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:48.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.232+0000 7f602ec3a740 4 client.27815 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:20:48.238 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.232+0000 7f602ec3a740 1 -- 172.21.15.39:0/2272355974 --> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55870046f990 con 0x558700513b10 2024-03-18T16:20:48.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.232+0000 7f6029e00640 1 --2- 172.21.15.39:0/2272355974 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x558700513b10 0x558700515f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:48.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.232+0000 7f6029e00640 1 --2- 172.21.15.39:0/2272355974 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x558700513b10 0x558700515f50 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=66589246e8764ec0 server_cookie=49fdaecbb3c7fbd1 in_seq=0 out_seq=0 2024-03-18T16:20:48.239 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.232+0000 7f600aa00640 10 client.27815 ms_handle_connect on v2:172.21.15.196:6836/895247251 2024-03-18T16:20:48.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.233+0000 7f600aa00640 1 -- 172.21.15.39:0/2272355974 <== mds.2 v2:172.21.15.196:6836/895247251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x55870046f990 con 0x558700513b10 2024-03-18T16:20:48.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.233+0000 7f600aa00640 10 client.27815 handle_command_reply: tid=0 2024-03-18T16:20:48.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.233+0000 7f602ec3a740 1 client.27815 shutdown 2024-03-18T16:20:48.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.234+0000 7f602ec3a740 1 -- 172.21.15.39:0/2272355974 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5587004fe2b0 msgr2=0x558700508160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:48.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.234+0000 7f602ec3a740 1 --2- 172.21.15.39:0/2272355974 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5587004fe2b0 0x558700508160 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f6010012700 tx=0x7f6010012bc0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.234+0000 7f602ec3a740 1 -- 172.21.15.39:0/2272355974 shutdown_connections 2024-03-18T16:20:48.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.234+0000 7f602ec3a740 1 --2- 172.21.15.39:0/2272355974 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5587004fe2b0 0x558700508160 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.240 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.234+0000 7f602ec3a740 1 --2- 172.21.15.39:0/2272355974 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x558700513b10 0x558700515f50 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.234+0000 7f602ec3a740 1 --2- 172.21.15.39:0/2272355974 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558700503420 0x55870050f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.234+0000 7f602ec3a740 1 --2- 172.21.15.39:0/2272355974 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5587004fd8c0 0x558700507c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.234+0000 7f602ec3a740 1 -- 172.21.15.39:0/2272355974 >> 172.21.15.39:0/2272355974 conn(0x5587004f8e30 msgr2=0x5587004f81f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:48.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.234+0000 7f600aa00640 0 client.27815 ms_handle_reset on v2:172.21.15.196:6836/895247251 2024-03-18T16:20:48.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.234+0000 7f602ec3a740 1 -- 172.21.15.39:0/2272355974 shutdown_connections 2024-03-18T16:20:48.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.234+0000 7f602ec3a740 1 -- 172.21.15.39:0/2272355974 wait complete. 2024-03-18T16:20:48.241 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.234+0000 7f602ec3a740 20 client.27815 trim_cache size 0 max 16384 2024-03-18T16:20:48.241 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:48 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/2774971440' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:20:48.262 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.258+0000 7f602ec3a740 1 Processor -- start 2024-03-18T16:20:48.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.258+0000 7f602ec3a740 1 -- start start 2024-03-18T16:20:48.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.258+0000 7f602ec3a740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5587004fd8c0 0x5587004f8070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:48.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.258+0000 7f602ec3a740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5587004fe2b0 0x55870050f160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:48.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.258+0000 7f602ec3a740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558700503420 0x55870051a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:48.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.258+0000 7f602ec3a740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f60241b09c0 con 0x5587004fe2b0 2024-03-18T16:20:48.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.258+0000 7f602ec3a740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f60241b25c0 con 0x5587004fd8c0 2024-03-18T16:20:48.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.258+0000 7f602ec3a740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f60241b1090 con 0x558700503420 2024-03-18T16:20:48.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.258+0000 7f6029e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5587004fd8c0 0x5587004f8070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:48.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.259+0000 7f6029e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5587004fd8c0 0x5587004f8070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:38998/0 (socket says 172.21.15.39:38998) 2024-03-18T16:20:48.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.259+0000 7f6029e00640 1 -- 172.21.15.39:0/3407715731 learned_addr learned my addr 172.21.15.39:0/3407715731 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:20:48.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.259+0000 7f6029e00640 1 --2- 172.21.15.39:0/3407715731 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558700503420 0x55870051a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:48.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.259+0000 7f6028a00640 1 --2- 172.21.15.39:0/3407715731 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5587004fe2b0 0x55870050f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:48.301 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.259+0000 7f6029e00640 1 -- 172.21.15.39:0/3407715731 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558700503420 msgr2=0x55870051a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:48.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.259+0000 7f6029e00640 1 --2- 172.21.15.39:0/3407715731 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558700503420 0x55870051a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.259+0000 7f6029e00640 1 -- 172.21.15.39:0/3407715731 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5587004fe2b0 msgr2=0x55870050f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:48.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.259+0000 7f6029e00640 1 --2- 172.21.15.39:0/3407715731 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5587004fe2b0 0x55870050f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.259+0000 7f6029e00640 1 -- 172.21.15.39:0/3407715731 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5587004f4c10 con 0x5587004fd8c0 2024-03-18T16:20:48.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.259+0000 7f6028a00640 1 --2- 172.21.15.39:0/3407715731 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5587004fe2b0 0x55870050f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-18T16:20:48.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.259+0000 7f6029e00640 1 --2- 172.21.15.39:0/3407715731 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5587004fd8c0 0x5587004f8070 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f601c014040 tx=0x7f601c004080 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e68378d8e8dbb850 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:48.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.259+0000 7f6015c00640 1 -- 172.21.15.39:0/3407715731 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f601c023040 con 0x5587004fd8c0 2024-03-18T16:20:48.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.259+0000 7f6015c00640 1 -- 172.21.15.39:0/3407715731 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f601c00f9a0 con 0x5587004fd8c0 2024-03-18T16:20:48.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.259+0000 7f6015c00640 1 -- 172.21.15.39:0/3407715731 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f601c005440 con 0x5587004fd8c0 2024-03-18T16:20:48.306 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.260+0000 7f602ec3a740 1 -- 172.21.15.39:0/3407715731 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5587004fd8c0 msgr2=0x5587004f8070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:48.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.260+0000 7f602ec3a740 1 --2- 172.21.15.39:0/3407715731 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5587004fd8c0 0x5587004f8070 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f601c014040 tx=0x7f601c004080 comp rx=0 tx=0).stop 2024-03-18T16:20:48.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.260+0000 7f602ec3a740 1 -- 172.21.15.39:0/3407715731 shutdown_connections 2024-03-18T16:20:48.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.260+0000 7f602ec3a740 1 --2- 172.21.15.39:0/3407715731 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x558700503420 0x55870051a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.260+0000 7f602ec3a740 1 --2- 172.21.15.39:0/3407715731 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5587004fe2b0 0x55870050f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.260+0000 7f602ec3a740 1 --2- 172.21.15.39:0/3407715731 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5587004fd8c0 0x5587004f8070 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.260+0000 7f602ec3a740 1 -- 172.21.15.39:0/3407715731 >> 172.21.15.39:0/3407715731 conn(0x5587004f8e30 msgr2=0x5587004f7960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:48.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.260+0000 7f602ec3a740 1 -- 172.21.15.39:0/3407715731 shutdown_connections 2024-03-18T16:20:48.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.260+0000 7f602ec3a740 1 -- 172.21.15.39:0/3407715731 wait complete. 2024-03-18T16:20:48.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.261+0000 7f602ec3a740 1 -- start start 2024-03-18T16:20:48.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.261+0000 7f602ec3a740 1 Processor -- start 2024-03-18T16:20:48.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.261+0000 7f602ec3a740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5587004fd8c0 0x55870050c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:48.307 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.261+0000 7f602ec3a740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5587004fe2b0 0x55870050ce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:48.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.261+0000 7f602ec3a740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558700503420 0x55870050de60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:48.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.261+0000 7f602ec3a740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f60241b1ec0 con 0x5587004fe2b0 2024-03-18T16:20:48.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.261+0000 7f602ec3a740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f60241b17c0 con 0x558700503420 2024-03-18T16:20:48.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.261+0000 7f602ec3a740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f60241ae9a0 con 0x5587004fd8c0 2024-03-18T16:20:48.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.261+0000 7f602ec3a740 10 client.0 _ll_register_callbacks cb 0x7f6010009570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:20:48.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.262+0000 7f6029e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5587004fd8c0 0x55870050c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:48.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.262+0000 7f6029e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5587004fd8c0 0x55870050c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:47624/0 (socket says 172.21.15.39:47624) 2024-03-18T16:20:48.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.262+0000 7f6029e00640 1 -- 172.21.15.39:0/3397535870 learned_addr learned my addr 172.21.15.39:0/3397535870 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:20:48.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.262+0000 7f6028a00640 1 --2- 172.21.15.39:0/3397535870 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5587004fe2b0 0x55870050ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:48.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.262+0000 7f6029e00640 1 --2- 172.21.15.39:0/3397535870 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558700503420 0x55870050de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:48.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.262+0000 7f6029e00640 1 -- 172.21.15.39:0/3397535870 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558700503420 msgr2=0x55870050de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:48.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.262+0000 7f6029e00640 1 --2- 172.21.15.39:0/3397535870 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558700503420 0x55870050de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.308 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.262+0000 7f6029e00640 1 -- 172.21.15.39:0/3397535870 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5587004fe2b0 msgr2=0x55870050ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:48.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.262+0000 7f6029e00640 1 --2- 172.21.15.39:0/3397535870 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5587004fe2b0 0x55870050ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.262+0000 7f6029e00640 1 -- 172.21.15.39:0/3397535870 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55870050e450 con 0x5587004fd8c0 2024-03-18T16:20:48.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.263+0000 7f6029e00640 1 --2- 172.21.15.39:0/3397535870 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5587004fd8c0 0x55870050c910 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f601c006010 tx=0x7f601c003ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e6343a3f44fd069 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:48.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.263+0000 7f600aa00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:20:48.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.263+0000 7f600aa00640 1 -- 172.21.15.39:0/3397535870 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f601c023070 con 0x5587004fd8c0 2024-03-18T16:20:48.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.263+0000 7f602ec3a740 1 -- 172.21.15.39:0/3397535870 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f601c00f9a0 con 0x5587004fd8c0 2024-03-18T16:20:48.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.263+0000 7f600aa00640 1 -- 172.21.15.39:0/3397535870 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f601c00fb50 con 0x5587004fd8c0 2024-03-18T16:20:48.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.263+0000 7f600aa00640 1 -- 172.21.15.39:0/3397535870 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f601c00fd50 con 0x5587004fd8c0 2024-03-18T16:20:48.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.263+0000 7f602ec3a740 10 client.27808 fetch_fsmap learned FSMap version 48 2024-03-18T16:20:48.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.263+0000 7f602ec3a740 1 -- 172.21.15.39:0/3397535870 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x558700579e90 con 0x5587004fd8c0 2024-03-18T16:20:48.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.263+0000 7f600aa00640 1 -- 172.21.15.39:0/3397535870 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f601c00f9a0 con 0x5587004fd8c0 2024-03-18T16:20:48.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.263+0000 7f600aa00640 1 -- 172.21.15.39:0/3397535870 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 48) ==== 8386+0+0 (secure 0 0 0) 0x7f601c018020 con 0x5587004fd8c0 2024-03-18T16:20:48.309 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.264+0000 7f602ec3a740 10 client.27808 fetch_fsmap finished waiting for FSMap version 48 2024-03-18T16:20:48.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.264+0000 7f602ec3a740 10 client.27808 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-18T16:20:48.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.264+0000 7f602ec3a740 20 client.27808 populate_metadata read hostname 'smithi039' 2024-03-18T16:20:48.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.264+0000 7f602ec3a740 1 --2- 172.21.15.39:0/3397535870 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x55870057a690 0x55870057cab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:48.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.264+0000 7f602ec3a740 4 client.27808 mds_command: new command op to 24535 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:20:48.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.264+0000 7f602ec3a740 1 -- 172.21.15.39:0/3397535870 --> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558700469ce0 con 0x55870057a690 2024-03-18T16:20:48.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.264+0000 7f6028a00640 1 --2- 172.21.15.39:0/3397535870 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x55870057a690 0x55870057cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:48.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.265+0000 7f600aa00640 10 client.27808 ms_handle_connect on v2:172.21.15.196:6836/895247251 2024-03-18T16:20:48.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.265+0000 7f6028a00640 1 --2- 172.21.15.39:0/3397535870 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x55870057a690 0x55870057cab0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=cad30b6b1fdc6e77 server_cookie=a64a2de106ee1dcd in_seq=0 out_seq=0 2024-03-18T16:20:48.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.265+0000 7f600aa00640 1 -- 172.21.15.39:0/3397535870 <== mds.2 v2:172.21.15.196:6836/895247251 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558700469ce0 con 0x55870057a690 2024-03-18T16:20:48.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.265+0000 7f600aa00640 10 client.27808 handle_command_reply: tid=0 2024-03-18T16:20:48.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.265+0000 7f602ec3a740 1 client.27808 shutdown 2024-03-18T16:20:48.310 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.265+0000 7f602ec3a740 1 -- 172.21.15.39:0/3397535870 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5587004fd8c0 msgr2=0x55870050c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:48.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.265+0000 7f602ec3a740 1 --2- 172.21.15.39:0/3397535870 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5587004fd8c0 0x55870050c910 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f601c006010 tx=0x7f601c003ec0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.265+0000 7f602ec3a740 1 -- 172.21.15.39:0/3397535870 shutdown_connections 2024-03-18T16:20:48.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.265+0000 7f602ec3a740 1 --2- 172.21.15.39:0/3397535870 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558700503420 0x55870050de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.265+0000 7f602ec3a740 1 --2- 172.21.15.39:0/3397535870 >> [v2:172.21.15.196:6836/895247251,v1:172.21.15.196:6837/895247251] conn(0x55870057a690 0x55870057cab0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.265+0000 7f602ec3a740 1 --2- 172.21.15.39:0/3397535870 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5587004fe2b0 0x55870050ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.265+0000 7f602ec3a740 1 --2- 172.21.15.39:0/3397535870 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x5587004fd8c0 0x55870050c910 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.265+0000 7f600aa00640 0 client.27808 ms_handle_reset on v2:172.21.15.196:6836/895247251 2024-03-18T16:20:48.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.265+0000 7f602ec3a740 1 -- 172.21.15.39:0/3397535870 >> 172.21.15.39:0/3397535870 conn(0x5587004f8e30 msgr2=0x5587004f81f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:48.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.265+0000 7f602ec3a740 1 -- 172.21.15.39:0/3397535870 shutdown_connections 2024-03-18T16:20:48.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.265+0000 7f602ec3a740 1 -- 172.21.15.39:0/3397535870 wait complete. 2024-03-18T16:20:48.311 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.265+0000 7f602ec3a740 20 client.27808 trim_cache size 0 max 16384 2024-03-18T16:20:48.312 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:20:48.312 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.266+0000 7f602be00640 1 -- 172.21.15.39:0/885741760 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f600c0a2800 msgr2=0x7f600c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:48.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.266+0000 7f602be00640 1 --2- 172.21.15.39:0/885741760 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f600c0a2800 0x7f600c0a4cc0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f60180bd3d0 tx=0x7f60180b9040 comp rx=0 tx=0).stop 2024-03-18T16:20:48.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.266+0000 7f602be00640 1 -- 172.21.15.39:0/885741760 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f60240761a0 msgr2=0x7f602419e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:48.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.266+0000 7f602be00640 1 --2- 172.21.15.39:0/885741760 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f60240761a0 0x7f602419e070 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f6018000c00 tx=0x7f601802c650 comp rx=0 tx=0).stop 2024-03-18T16:20:48.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.266+0000 7f602be00640 1 -- 172.21.15.39:0/885741760 shutdown_connections 2024-03-18T16:20:48.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.266+0000 7f602be00640 1 --2- 172.21.15.39:0/885741760 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f600c0a2800 0x7f600c0a4cc0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.266+0000 7f602be00640 1 --2- 172.21.15.39:0/885741760 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6024077d80 0x7f60241a5620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.266+0000 7f602be00640 1 --2- 172.21.15.39:0/885741760 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f60240773e0 0x7f602419e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.266+0000 7f602be00640 1 --2- 172.21.15.39:0/885741760 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f60240761a0 0x7f602419e070 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.266+0000 7f602be00640 1 -- 172.21.15.39:0/885741760 >> 172.21.15.39:0/885741760 conn(0x7f60240fefa0 msgr2=0x7f6024100aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:48.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.266+0000 7f602be00640 1 -- 172.21.15.39:0/885741760 shutdown_connections 2024-03-18T16:20:48.313 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.267+0000 7f602be00640 1 -- 172.21.15.39:0/885741760 wait complete. 2024-03-18T16:20:48.314 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:48 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/2774971440' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:20:48.315 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-18T16:20:48.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:48 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/2774971440' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:20:48.512 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.509+0000 7f7e04a00640 1 -- 172.21.15.39:0/2822231800 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7e00101800 msgr2=0x7f7e00101c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:48.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.509+0000 7f7e04a00640 1 --2- 172.21.15.39:0/2822231800 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7e00101800 0x7f7e00101c20 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f7df0009900 tx=0x7f7df002bc30 comp rx=0 tx=0).stop 2024-03-18T16:20:48.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.509+0000 7f7e04a00640 1 -- 172.21.15.39:0/2822231800 shutdown_connections 2024-03-18T16:20:48.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.509+0000 7f7e04a00640 1 --2- 172.21.15.39:0/2822231800 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7e001033b0 0x7f7e00109c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.509+0000 7f7e04a00640 1 --2- 172.21.15.39:0/2822231800 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7e001029f0 0x7f7e00102e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.509+0000 7f7e04a00640 1 --2- 172.21.15.39:0/2822231800 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7e00101800 0x7f7e00101c20 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.509+0000 7f7e04a00640 1 -- 172.21.15.39:0/2822231800 >> 172.21.15.39:0/2822231800 conn(0x7f7e000fcd70 msgr2=0x7f7e000ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:48.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.509+0000 7f7e04a00640 1 -- 172.21.15.39:0/2822231800 shutdown_connections 2024-03-18T16:20:48.554 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.509+0000 7f7e04a00640 1 -- 172.21.15.39:0/2822231800 wait complete. 2024-03-18T16:20:48.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.511+0000 7f7e04a00640 1 Processor -- start 2024-03-18T16:20:48.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.511+0000 7f7e04a00640 1 -- start start 2024-03-18T16:20:48.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.511+0000 7f7e04a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7e00101800 0x7f7e00075590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:48.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.511+0000 7f7e04a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7e001029f0 0x7f7e00073bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:48.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.512+0000 7f7e04a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7e001033b0 0x7f7e00074110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:48.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.512+0000 7f7e04a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f7e0010d4a0 con 0x7f7e001033b0 2024-03-18T16:20:48.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.512+0000 7f7e04a00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f7e0010d320 con 0x7f7e00101800 2024-03-18T16:20:48.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.512+0000 7f7e04a00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f7e0010d620 con 0x7f7e001029f0 2024-03-18T16:20:48.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.512+0000 7f7dfde00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7e00101800 0x7f7e00075590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:48.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.512+0000 7f7dfde00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7e00101800 0x7f7e00075590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:39016/0 (socket says 172.21.15.39:39016) 2024-03-18T16:20:48.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.512+0000 7f7dfde00640 1 -- 172.21.15.39:0/2082187410 learned_addr learned my addr 172.21.15.39:0/2082187410 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:20:48.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.512+0000 7f7dfd400640 1 --2- 172.21.15.39:0/2082187410 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7e001029f0 0x7f7e00073bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:48.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.512+0000 7f7dfe800640 1 --2- 172.21.15.39:0/2082187410 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7e001033b0 0x7f7e00074110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:48.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.512+0000 7f7dfde00640 1 -- 172.21.15.39:0/2082187410 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7e001029f0 msgr2=0x7f7e00073bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:48.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.512+0000 7f7dfde00640 1 --2- 172.21.15.39:0/2082187410 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7e001029f0 0x7f7e00073bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.512+0000 7f7dfde00640 1 -- 172.21.15.39:0/2082187410 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7e001033b0 msgr2=0x7f7e00074110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:48.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.512+0000 7f7dfde00640 1 --2- 172.21.15.39:0/2082187410 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7e001033b0 0x7f7e00074110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.512+0000 7f7dfde00640 1 -- 172.21.15.39:0/2082187410 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e000749a0 con 0x7f7e00101800 2024-03-18T16:20:48.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.513+0000 7f7dfde00640 1 --2- 172.21.15.39:0/2082187410 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7e00101800 0x7f7e00075590 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f7df0000c00 tx=0x7f7df002c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:48.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.513+0000 7f7df6a00640 1 -- 172.21.15.39:0/2082187410 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7df0043020 con 0x7f7e00101800 2024-03-18T16:20:48.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.513+0000 7f7df6a00640 1 -- 172.21.15.39:0/2082187410 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7df00054e0 con 0x7f7e00101800 2024-03-18T16:20:48.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.513+0000 7f7e04a00640 1 -- 172.21.15.39:0/2082187410 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e001a1c30 con 0x7f7e00101800 2024-03-18T16:20:48.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.513+0000 7f7df6a00640 1 -- 172.21.15.39:0/2082187410 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7df0003c40 con 0x7f7e00101800 2024-03-18T16:20:48.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.513+0000 7f7e04a00640 1 -- 172.21.15.39:0/2082187410 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e001a21d0 con 0x7f7e00101800 2024-03-18T16:20:48.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.516+0000 7f7df6a00640 1 -- 172.21.15.39:0/2082187410 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f7df0046020 con 0x7f7e00101800 2024-03-18T16:20:48.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.516+0000 7f7e077ce740 1 Processor -- start 2024-03-18T16:20:48.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.517+0000 7f7e077ce740 1 -- start start 2024-03-18T16:20:48.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.517+0000 7f7e077ce740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560e2e88a8c0 0x560e2e88ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:48.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.517+0000 7f7e077ce740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560e2e88b2b0 0x560e2e88fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:48.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.517+0000 7f7e077ce740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560e2e890420 0x560e2e8928b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:48.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.517+0000 7f7e077ce740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x560e2e8523b0 con 0x560e2e88a8c0 2024-03-18T16:20:48.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.517+0000 7f7e077ce740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x560e2e7ef450 con 0x560e2e890420 2024-03-18T16:20:48.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.517+0000 7f7e077ce740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x560e2e797cb0 con 0x560e2e88b2b0 2024-03-18T16:20:48.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.517+0000 7f7dfde00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560e2e88b2b0 0x560e2e88fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:48.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.517+0000 7f7dfde00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560e2e88b2b0 0x560e2e88fdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:47642/0 (socket says 172.21.15.39:47642) 2024-03-18T16:20:48.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.517+0000 7f7dfde00640 1 -- 172.21.15.39:0/3519494455 learned_addr learned my addr 172.21.15.39:0/3519494455 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:20:48.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.517+0000 7f7dfe800640 1 --2- 172.21.15.39:0/3519494455 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560e2e88a8c0 0x560e2e88ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:48.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.517+0000 7f7dfd400640 1 --2- 172.21.15.39:0/3519494455 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560e2e890420 0x560e2e8928b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:48.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.518+0000 7f7dfde00640 1 -- 172.21.15.39:0/3519494455 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560e2e890420 msgr2=0x560e2e8928b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:48.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.518+0000 7f7dfde00640 1 --2- 172.21.15.39:0/3519494455 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560e2e890420 0x560e2e8928b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.518+0000 7f7dfde00640 1 -- 172.21.15.39:0/3519494455 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560e2e88a8c0 msgr2=0x560e2e88ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:48.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.518+0000 7f7dfde00640 1 --2- 172.21.15.39:0/3519494455 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560e2e88a8c0 0x560e2e88ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.518+0000 7f7dfde00640 1 -- 172.21.15.39:0/3519494455 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560e2e7b91f0 con 0x560e2e88b2b0 2024-03-18T16:20:48.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.518+0000 7f7dfde00640 1 --2- 172.21.15.39:0/3519494455 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560e2e88b2b0 0x560e2e88fdb0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f7df00f4d80 tx=0x7f7df00f4e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=76316f9f766aa416 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:48.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.518+0000 7f7deb000640 1 -- 172.21.15.39:0/3519494455 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7df00f7b00 con 0x560e2e88b2b0 2024-03-18T16:20:48.588 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.518+0000 7f7deb000640 1 -- 172.21.15.39:0/3519494455 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7df00f7ca0 con 0x560e2e88b2b0 2024-03-18T16:20:48.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.518+0000 7f7deb000640 1 -- 172.21.15.39:0/3519494455 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7df01015d0 con 0x560e2e88b2b0 2024-03-18T16:20:48.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.519+0000 7f7df6a00640 1 --2- 172.21.15.39:0/2082187410 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f7de40a2880 0x7f7de40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:48.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.519+0000 7f7df6a00640 1 -- 172.21.15.39:0/2082187410 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f7df00f0e60 con 0x7f7e00101800 2024-03-18T16:20:48.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.519+0000 7f7e077ce740 1 -- 172.21.15.39:0/3519494455 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560e2e88b2b0 msgr2=0x560e2e88fdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:48.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.519+0000 7f7e077ce740 1 --2- 172.21.15.39:0/3519494455 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560e2e88b2b0 0x560e2e88fdb0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f7df00f4d80 tx=0x7f7df00f4e70 comp rx=0 tx=0).stop 2024-03-18T16:20:48.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.519+0000 7f7dfe800640 1 --2- 172.21.15.39:0/2082187410 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f7de40a2880 0x7f7de40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:48.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.520+0000 7f7e077ce740 1 -- 172.21.15.39:0/3519494455 shutdown_connections 2024-03-18T16:20:48.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.520+0000 7f7e077ce740 1 --2- 172.21.15.39:0/3519494455 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560e2e890420 0x560e2e8928b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.520+0000 7f7e077ce740 1 --2- 172.21.15.39:0/3519494455 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560e2e88b2b0 0x560e2e88fdb0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.520+0000 7f7e077ce740 1 --2- 172.21.15.39:0/3519494455 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560e2e88a8c0 0x560e2e88ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.520+0000 7f7e077ce740 1 -- 172.21.15.39:0/3519494455 >> 172.21.15.39:0/3519494455 conn(0x560e2e885e30 msgr2=0x560e2e886230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:48.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.520+0000 7f7e077ce740 1 -- 172.21.15.39:0/3519494455 shutdown_connections 2024-03-18T16:20:48.589 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.520+0000 7f7e077ce740 1 -- 172.21.15.39:0/3519494455 wait complete. 2024-03-18T16:20:48.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.520+0000 7f7dfe800640 1 --2- 172.21.15.39:0/2082187410 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f7de40a2880 0x7f7de40a4d40 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x560e2e88bb10 tx=0x7f7df8009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:48.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.520+0000 7f7e077ce740 1 -- start start 2024-03-18T16:20:48.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.521+0000 7f7e077ce740 1 Processor -- start 2024-03-18T16:20:48.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.521+0000 7f7e077ce740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560e2e88a8c0 0x560e2e894880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:48.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.521+0000 7f7e077ce740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560e2e88b2b0 0x560e2e894dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:48.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.521+0000 7f7e077ce740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560e2e890420 0x560e2e89be00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:48.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.521+0000 7f7e077ce740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x560e2e7ab110 con 0x560e2e88b2b0 2024-03-18T16:20:48.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.521+0000 7f7e077ce740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x560e2e7eb4f0 con 0x560e2e88a8c0 2024-03-18T16:20:48.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.521+0000 7f7e077ce740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x560e2e893190 con 0x560e2e890420 2024-03-18T16:20:48.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.521+0000 7f7e077ce740 10 client.0 _ll_register_callbacks cb 0x560e2e798b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:20:48.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.521+0000 7f7dfde00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560e2e88a8c0 0x560e2e894880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:48.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.521+0000 7f7dfe800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560e2e890420 0x560e2e89be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:48.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.521+0000 7f7dfd400640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560e2e88b2b0 0x560e2e894dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:48.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.521+0000 7f7dfde00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560e2e88a8c0 0x560e2e894880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:39024/0 (socket says 172.21.15.39:39024) 2024-03-18T16:20:48.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.521+0000 7f7dfe800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560e2e890420 0x560e2e89be00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:47648/0 (socket says 172.21.15.39:47648) 2024-03-18T16:20:48.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.521+0000 7f7dfde00640 1 -- 172.21.15.39:0/2410973950 learned_addr learned my addr 172.21.15.39:0/2410973950 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:20:48.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.522+0000 7f7dfde00640 1 -- 172.21.15.39:0/2410973950 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560e2e890420 msgr2=0x560e2e89be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:48.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.522+0000 7f7dfde00640 1 --2- 172.21.15.39:0/2410973950 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560e2e890420 0x560e2e89be00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.522+0000 7f7dfde00640 1 -- 172.21.15.39:0/2410973950 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560e2e88b2b0 msgr2=0x560e2e894dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:48.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.522+0000 7f7dfde00640 1 --2- 172.21.15.39:0/2410973950 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560e2e88b2b0 0x560e2e894dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.522+0000 7f7dfde00640 1 -- 172.21.15.39:0/2410973950 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560e2e5033f0 con 0x560e2e88a8c0 2024-03-18T16:20:48.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.522+0000 7f7dfde00640 1 --2- 172.21.15.39:0/2410973950 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560e2e88a8c0 0x560e2e894880 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f7df002ceb0 tx=0x7f7df002bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a202964827cce30 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:48.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.522+0000 7f7ddf400640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:20:48.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.522+0000 7f7ddf400640 1 -- 172.21.15.39:0/2410973950 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7df00b96b0 con 0x560e2e88a8c0 2024-03-18T16:20:48.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.522+0000 7f7ddf400640 1 -- 172.21.15.39:0/2410973950 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f7df00bb4b0 con 0x560e2e88a8c0 2024-03-18T16:20:48.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.522+0000 7f7e077ce740 1 -- 172.21.15.39:0/2410973950 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7df00f7ca0 con 0x560e2e88a8c0 2024-03-18T16:20:48.591 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.523+0000 7f7ddf400640 1 -- 172.21.15.39:0/2410973950 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f7df00f7b00 con 0x560e2e88a8c0 2024-03-18T16:20:48.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.523+0000 7f7e077ce740 10 client.27833 fetch_fsmap learned FSMap version 48 2024-03-18T16:20:48.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.523+0000 7f7e077ce740 1 -- 172.21.15.39:0/2410973950 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x560e2e5cd7d0 con 0x560e2e88a8c0 2024-03-18T16:20:48.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.523+0000 7f7ddf400640 1 -- 172.21.15.39:0/2410973950 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7df00f7ca0 con 0x560e2e88a8c0 2024-03-18T16:20:48.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.523+0000 7f7ddf400640 1 -- 172.21.15.39:0/2410973950 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 48) ==== 8386+0+0 (secure 0 0 0) 0x7f7df00f3060 con 0x560e2e88a8c0 2024-03-18T16:20:48.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.523+0000 7f7e077ce740 10 client.27833 fetch_fsmap finished waiting for FSMap version 48 2024-03-18T16:20:48.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.523+0000 7f7e077ce740 10 client.27833 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T16:20:48.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.523+0000 7f7e077ce740 20 client.27833 populate_metadata read hostname 'smithi039' 2024-03-18T16:20:48.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.523+0000 7f7e077ce740 1 --2- 172.21.15.39:0/2410973950 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x560e2e8a0510 0x560e2e8a2940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:48.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.523+0000 7f7e077ce740 4 client.27833 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:20:48.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.523+0000 7f7e077ce740 1 -- 172.21.15.39:0/2410973950 --> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560e2e7fc990 con 0x560e2e8a0510 2024-03-18T16:20:48.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.524+0000 7f7dfd400640 1 --2- 172.21.15.39:0/2410973950 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x560e2e8a0510 0x560e2e8a2940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:48.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.524+0000 7f7dfd400640 1 --2- 172.21.15.39:0/2410973950 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x560e2e8a0510 0x560e2e8a2940 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f76fc2a7b6ee0c82 server_cookie=6febba38fa70ba86 in_seq=0 out_seq=0 2024-03-18T16:20:48.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.524+0000 7f7ddf400640 10 client.27833 ms_handle_connect on v2:172.21.15.196:6838/2001427268 2024-03-18T16:20:48.592 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.525+0000 7f7ddf400640 1 -- 172.21.15.39:0/2410973950 <== mds.1 v2:172.21.15.196:6838/2001427268 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x560e2e7fc990 con 0x560e2e8a0510 2024-03-18T16:20:48.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.525+0000 7f7ddf400640 10 client.27833 handle_command_reply: tid=0 2024-03-18T16:20:48.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.525+0000 7f7e077ce740 1 client.27833 shutdown 2024-03-18T16:20:48.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.525+0000 7f7e077ce740 1 -- 172.21.15.39:0/2410973950 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560e2e88a8c0 msgr2=0x560e2e894880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:48.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.525+0000 7f7e077ce740 1 --2- 172.21.15.39:0/2410973950 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560e2e88a8c0 0x560e2e894880 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f7df002ceb0 tx=0x7f7df002bc30 comp rx=0 tx=0).stop 2024-03-18T16:20:48.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.525+0000 7f7e077ce740 1 -- 172.21.15.39:0/2410973950 shutdown_connections 2024-03-18T16:20:48.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.525+0000 7f7e077ce740 1 --2- 172.21.15.39:0/2410973950 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x560e2e8a0510 0x560e2e8a2940 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.525+0000 7f7e077ce740 1 --2- 172.21.15.39:0/2410973950 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560e2e890420 0x560e2e89be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.525+0000 7f7e077ce740 1 --2- 172.21.15.39:0/2410973950 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560e2e88b2b0 0x560e2e894dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.525+0000 7f7e077ce740 1 --2- 172.21.15.39:0/2410973950 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560e2e88a8c0 0x560e2e894880 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.525+0000 7f7ddf400640 0 client.27833 ms_handle_reset on v2:172.21.15.196:6838/2001427268 2024-03-18T16:20:48.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.525+0000 7f7e077ce740 1 -- 172.21.15.39:0/2410973950 >> 172.21.15.39:0/2410973950 conn(0x560e2e885e30 msgr2=0x560e2e8851f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:48.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.525+0000 7f7e077ce740 1 -- 172.21.15.39:0/2410973950 shutdown_connections 2024-03-18T16:20:48.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.525+0000 7f7e077ce740 1 -- 172.21.15.39:0/2410973950 wait complete. 2024-03-18T16:20:48.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.526+0000 7f7e077ce740 20 client.27833 trim_cache size 0 max 16384 2024-03-18T16:20:48.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.540+0000 7f7e077ce740 1 Processor -- start 2024-03-18T16:20:48.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.540+0000 7f7e077ce740 1 -- start start 2024-03-18T16:20:48.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.540+0000 7f7e077ce740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560e2e88a8c0 0x560e2e884960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:48.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.540+0000 7f7e077ce740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560e2e88b2b0 0x560e2e89bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:48.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.540+0000 7f7e077ce740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560e2e890420 0x560e2e884ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:48.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.541+0000 7f7e077ce740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f7e001ac6d0 con 0x560e2e88a8c0 2024-03-18T16:20:48.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.541+0000 7f7e077ce740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f7e001ae2d0 con 0x560e2e88b2b0 2024-03-18T16:20:48.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.541+0000 7f7e077ce740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f7e001acda0 con 0x560e2e890420 2024-03-18T16:20:48.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.541+0000 7f7dfd400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560e2e88b2b0 0x560e2e89bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:48.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.541+0000 7f7dfde00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560e2e88a8c0 0x560e2e884960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:48.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.541+0000 7f7dfe800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560e2e890420 0x560e2e884ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:48.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.541+0000 7f7dfd400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560e2e88b2b0 0x560e2e89bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:39040/0 (socket says 172.21.15.39:39040) 2024-03-18T16:20:48.594 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.541+0000 7f7dfd400640 1 -- 172.21.15.39:0/3683165140 learned_addr learned my addr 172.21.15.39:0/3683165140 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:20:48.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.541+0000 7f7dfd400640 1 -- 172.21.15.39:0/3683165140 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560e2e890420 msgr2=0x560e2e884ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:48.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.541+0000 7f7dfd400640 1 --2- 172.21.15.39:0/3683165140 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560e2e890420 0x560e2e884ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.541+0000 7f7dfd400640 1 -- 172.21.15.39:0/3683165140 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560e2e88a8c0 msgr2=0x560e2e884960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:48.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.541+0000 7f7dfd400640 1 --2- 172.21.15.39:0/3683165140 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560e2e88a8c0 0x560e2e884960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.541+0000 7f7dfd400640 1 -- 172.21.15.39:0/3683165140 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560e2e881c10 con 0x560e2e88b2b0 2024-03-18T16:20:48.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.541+0000 7f7dfde00640 1 --2- 172.21.15.39:0/3683165140 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560e2e88a8c0 0x560e2e884960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:20:48.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.542+0000 7f7dfd400640 1 --2- 172.21.15.39:0/3683165140 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560e2e88b2b0 0x560e2e89bd90 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f7dec000c00 tx=0x7f7dec005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=614c16e00fb4e8cc server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:48.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.542+0000 7f7dea600640 1 -- 172.21.15.39:0/3683165140 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dec023070 con 0x560e2e88b2b0 2024-03-18T16:20:48.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.542+0000 7f7dea600640 1 -- 172.21.15.39:0/3683165140 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7dec0047a0 con 0x560e2e88b2b0 2024-03-18T16:20:48.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.542+0000 7f7dea600640 1 -- 172.21.15.39:0/3683165140 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7dec01f4c0 con 0x560e2e88b2b0 2024-03-18T16:20:48.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.543+0000 7f7e077ce740 1 -- 172.21.15.39:0/3683165140 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560e2e88b2b0 msgr2=0x560e2e89bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:48.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.543+0000 7f7e077ce740 1 --2- 172.21.15.39:0/3683165140 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560e2e88b2b0 0x560e2e89bd90 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f7dec000c00 tx=0x7f7dec005f90 comp rx=0 tx=0).stop 2024-03-18T16:20:48.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.543+0000 7f7e077ce740 1 -- 172.21.15.39:0/3683165140 shutdown_connections 2024-03-18T16:20:48.595 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.543+0000 7f7e077ce740 1 --2- 172.21.15.39:0/3683165140 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560e2e88b2b0 0x560e2e89bd90 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.543+0000 7f7e077ce740 1 --2- 172.21.15.39:0/3683165140 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560e2e890420 0x560e2e884ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.543+0000 7f7e077ce740 1 --2- 172.21.15.39:0/3683165140 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560e2e88a8c0 0x560e2e884960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.543+0000 7f7e077ce740 1 -- 172.21.15.39:0/3683165140 >> 172.21.15.39:0/3683165140 conn(0x560e2e885e30 msgr2=0x560e2e894490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:48.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.543+0000 7f7e077ce740 1 -- 172.21.15.39:0/3683165140 shutdown_connections 2024-03-18T16:20:48.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.543+0000 7f7e077ce740 1 -- 172.21.15.39:0/3683165140 wait complete. 2024-03-18T16:20:48.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.544+0000 7f7e077ce740 1 -- start start 2024-03-18T16:20:48.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.544+0000 7f7e077ce740 1 Processor -- start 2024-03-18T16:20:48.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.544+0000 7f7e077ce740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560e2e88a8c0 0x560e2e8992f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:48.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.544+0000 7f7e077ce740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560e2e88b2b0 0x560e2e899830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:48.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.544+0000 7f7e077ce740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560e2e890420 0x560e2e89a840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:48.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.544+0000 7f7e077ce740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f7e001adbd0 con 0x560e2e88b2b0 2024-03-18T16:20:48.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.544+0000 7f7e077ce740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f7e001ad4d0 con 0x560e2e890420 2024-03-18T16:20:48.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.544+0000 7f7e077ce740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f7e001aa6b0 con 0x560e2e88a8c0 2024-03-18T16:20:48.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.544+0000 7f7e077ce740 10 client.0 _ll_register_callbacks cb 0x7f7dec0097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:20:48.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.544+0000 7f7dfde00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560e2e88a8c0 0x560e2e8992f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:48.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.544+0000 7f7dfe800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560e2e890420 0x560e2e89a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:48.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.545+0000 7f7dfe800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560e2e890420 0x560e2e89a840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:39056/0 (socket says 172.21.15.39:39056) 2024-03-18T16:20:48.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.545+0000 7f7dfe800640 1 -- 172.21.15.39:0/2759072765 learned_addr learned my addr 172.21.15.39:0/2759072765 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:20:48.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.545+0000 7f7dfde00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560e2e88a8c0 0x560e2e8992f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:47680/0 (socket says 172.21.15.39:47680) 2024-03-18T16:20:48.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.545+0000 7f7dfd400640 1 --2- 172.21.15.39:0/2759072765 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560e2e88b2b0 0x560e2e899830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:48.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.545+0000 7f7dfe800640 1 -- 172.21.15.39:0/2759072765 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560e2e88a8c0 msgr2=0x560e2e8992f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:48.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.545+0000 7f7dfe800640 1 --2- 172.21.15.39:0/2759072765 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560e2e88a8c0 0x560e2e8992f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.545+0000 7f7dfe800640 1 -- 172.21.15.39:0/2759072765 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560e2e88b2b0 msgr2=0x560e2e899830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:48.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.545+0000 7f7dfe800640 1 --2- 172.21.15.39:0/2759072765 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560e2e88b2b0 0x560e2e899830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.545+0000 7f7dfe800640 1 -- 172.21.15.39:0/2759072765 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560e2e8a02f0 con 0x560e2e890420 2024-03-18T16:20:48.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.545+0000 7f7dfd400640 1 --2- 172.21.15.39:0/2759072765 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560e2e88b2b0 0x560e2e899830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:20:48.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.545+0000 7f7dfe800640 1 --2- 172.21.15.39:0/2759072765 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560e2e890420 0x560e2e89a840 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f7df8012c40 tx=0x7f7df8004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1eca0835c89a62d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:48.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.545+0000 7f7ddf400640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:20:48.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.545+0000 7f7ddf400640 1 -- 172.21.15.39:0/2759072765 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7df802c060 con 0x560e2e890420 2024-03-18T16:20:48.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.546+0000 7f7e077ce740 1 -- 172.21.15.39:0/2759072765 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7dec0047a0 con 0x560e2e890420 2024-03-18T16:20:48.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.546+0000 7f7ddf400640 1 -- 172.21.15.39:0/2759072765 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f7df8027400 con 0x560e2e890420 2024-03-18T16:20:48.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.546+0000 7f7ddf400640 1 -- 172.21.15.39:0/2759072765 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f7df8027600 con 0x560e2e890420 2024-03-18T16:20:48.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.546+0000 7f7e077ce740 10 client.27839 fetch_fsmap learned FSMap version 48 2024-03-18T16:20:48.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.546+0000 7f7e077ce740 1 -- 172.21.15.39:0/2759072765 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x560e2e89bfd0 con 0x560e2e890420 2024-03-18T16:20:48.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.546+0000 7f7ddf400640 1 -- 172.21.15.39:0/2759072765 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7dec0047a0 con 0x560e2e890420 2024-03-18T16:20:48.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.546+0000 7f7ddf400640 1 -- 172.21.15.39:0/2759072765 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 48) ==== 8386+0+0 (secure 0 0 0) 0x7f7df8019040 con 0x560e2e890420 2024-03-18T16:20:48.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.546+0000 7f7e077ce740 10 client.27839 fetch_fsmap finished waiting for FSMap version 48 2024-03-18T16:20:48.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.546+0000 7f7e077ce740 10 client.27839 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-03-18T16:20:48.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.546+0000 7f7e077ce740 20 client.27839 populate_metadata read hostname 'smithi039' 2024-03-18T16:20:48.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.546+0000 7f7e077ce740 1 --2- 172.21.15.39:0/2759072765 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x560e2e906f10 0x560e2e909330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:48.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.546+0000 7f7e077ce740 4 client.27839 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:20:48.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.546+0000 7f7e077ce740 1 -- 172.21.15.39:0/2759072765 --> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560e2e7f6ce0 con 0x560e2e906f10 2024-03-18T16:20:48.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.547+0000 7f7dfde00640 1 --2- 172.21.15.39:0/2759072765 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x560e2e906f10 0x560e2e909330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:48.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.547+0000 7f7dfde00640 1 --2- 172.21.15.39:0/2759072765 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x560e2e906f10 0x560e2e909330 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e46f5bfa8445825d server_cookie=6d94e58ade837c2 in_seq=0 out_seq=0 2024-03-18T16:20:48.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.547+0000 7f7ddf400640 10 client.27839 ms_handle_connect on v2:172.21.15.196:6838/2001427268 2024-03-18T16:20:48.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.547+0000 7f7ddf400640 1 -- 172.21.15.39:0/2759072765 <== mds.1 v2:172.21.15.196:6838/2001427268 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560e2e7f6ce0 con 0x560e2e906f10 2024-03-18T16:20:48.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.547+0000 7f7ddf400640 10 client.27839 handle_command_reply: tid=0 2024-03-18T16:20:48.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.547+0000 7f7e077ce740 1 client.27839 shutdown 2024-03-18T16:20:48.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.547+0000 7f7e077ce740 1 -- 172.21.15.39:0/2759072765 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560e2e890420 msgr2=0x560e2e89a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:48.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.547+0000 7f7e077ce740 1 --2- 172.21.15.39:0/2759072765 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560e2e890420 0x560e2e89a840 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f7df8012c40 tx=0x7f7df8004a40 comp rx=0 tx=0).stop 2024-03-18T16:20:48.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.548+0000 7f7e077ce740 1 -- 172.21.15.39:0/2759072765 shutdown_connections 2024-03-18T16:20:48.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.548+0000 7f7e077ce740 1 --2- 172.21.15.39:0/2759072765 >> [v2:172.21.15.196:6838/2001427268,v1:172.21.15.196:6839/2001427268] conn(0x560e2e906f10 0x560e2e909330 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.548+0000 7f7e077ce740 1 --2- 172.21.15.39:0/2759072765 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560e2e890420 0x560e2e89a840 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.548+0000 7f7e077ce740 1 --2- 172.21.15.39:0/2759072765 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x560e2e88b2b0 0x560e2e899830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.548+0000 7f7ddf400640 0 client.27839 ms_handle_reset on v2:172.21.15.196:6838/2001427268 2024-03-18T16:20:48.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.548+0000 7f7e077ce740 1 --2- 172.21.15.39:0/2759072765 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x560e2e88a8c0 0x560e2e8992f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.548+0000 7f7e077ce740 1 -- 172.21.15.39:0/2759072765 >> 172.21.15.39:0/2759072765 conn(0x560e2e885e30 msgr2=0x560e2e884960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:48.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.548+0000 7f7e077ce740 1 -- 172.21.15.39:0/2759072765 shutdown_connections 2024-03-18T16:20:48.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.548+0000 7f7e077ce740 1 -- 172.21.15.39:0/2759072765 wait complete. 2024-03-18T16:20:48.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.548+0000 7f7e077ce740 20 client.27839 trim_cache size 0 max 16384 2024-03-18T16:20:48.600 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:20:48.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.549+0000 7f7e04a00640 1 -- 172.21.15.39:0/2082187410 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f7de40a2880 msgr2=0x7f7de40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:48.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.549+0000 7f7e04a00640 1 --2- 172.21.15.39:0/2082187410 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f7de40a2880 0x7f7de40a4d40 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x560e2e88bb10 tx=0x7f7df8009500 comp rx=0 tx=0).stop 2024-03-18T16:20:48.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.549+0000 7f7e04a00640 1 -- 172.21.15.39:0/2082187410 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7e00101800 msgr2=0x7f7e00075590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:48.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.549+0000 7f7e04a00640 1 --2- 172.21.15.39:0/2082187410 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7e00101800 0x7f7e00075590 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f7df0000c00 tx=0x7f7df002c650 comp rx=0 tx=0).stop 2024-03-18T16:20:48.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.549+0000 7f7e04a00640 1 -- 172.21.15.39:0/2082187410 shutdown_connections 2024-03-18T16:20:48.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.549+0000 7f7e04a00640 1 --2- 172.21.15.39:0/2082187410 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f7de40a2880 0x7f7de40a4d40 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.549+0000 7f7e04a00640 1 --2- 172.21.15.39:0/2082187410 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7e001033b0 0x7f7e00074110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.549+0000 7f7e04a00640 1 --2- 172.21.15.39:0/2082187410 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7e001029f0 0x7f7e00073bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.549+0000 7f7e04a00640 1 --2- 172.21.15.39:0/2082187410 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7e00101800 0x7f7e00075590 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.549+0000 7f7e04a00640 1 -- 172.21.15.39:0/2082187410 >> 172.21.15.39:0/2082187410 conn(0x7f7e000fcd70 msgr2=0x7f7e00105cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:48.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.549+0000 7f7e04a00640 1 -- 172.21.15.39:0/2082187410 shutdown_connections 2024-03-18T16:20:48.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.549+0000 7f7e04a00640 1 -- 172.21.15.39:0/2082187410 wait complete. 2024-03-18T16:20:48.603 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-18T16:20:48.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.832+0000 7f92b7200640 1 -- 172.21.15.39:0/360478132 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f92b81004d0 msgr2=0x7f92b81008f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:48.834 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.832+0000 7f92b7200640 1 --2- 172.21.15.39:0/360478132 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f92b81004d0 0x7f92b81008f0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f92a8009900 tx=0x7f92a802bc40 comp rx=0 tx=0).stop 2024-03-18T16:20:48.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.832+0000 7f92b7200640 1 -- 172.21.15.39:0/360478132 shutdown_connections 2024-03-18T16:20:48.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.832+0000 7f92b7200640 1 --2- 172.21.15.39:0/360478132 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f92b8102080 0x7f92b8108910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.832+0000 7f92b7200640 1 --2- 172.21.15.39:0/360478132 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f92b81016c0 0x7f92b8101b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.832+0000 7f92b7200640 1 --2- 172.21.15.39:0/360478132 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f92b81004d0 0x7f92b81008f0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.832+0000 7f92b7200640 1 -- 172.21.15.39:0/360478132 >> 172.21.15.39:0/360478132 conn(0x7f92b80fba80 msgr2=0x7f92b80fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:48.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.832+0000 7f92b7200640 1 -- 172.21.15.39:0/360478132 shutdown_connections 2024-03-18T16:20:48.835 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.832+0000 7f92b7200640 1 -- 172.21.15.39:0/360478132 wait complete. 2024-03-18T16:20:48.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.833+0000 7f92b7200640 1 Processor -- start 2024-03-18T16:20:48.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.833+0000 7f92b7200640 1 -- start start 2024-03-18T16:20:48.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.834+0000 7f92b7200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f92b81004d0 0x7f92b8199b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:48.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.834+0000 7f92b7200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f92b81016c0 0x7f92b819a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:48.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.834+0000 7f92b7200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f92b8102080 0x7f92b81a10b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:48.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.834+0000 7f92b7200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f92b810c1b0 con 0x7f92b81016c0 2024-03-18T16:20:48.837 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.834+0000 7f92b7200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f92b810c030 con 0x7f92b8102080 2024-03-18T16:20:48.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.834+0000 7f92b7200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f92b810c330 con 0x7f92b81004d0 2024-03-18T16:20:48.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.834+0000 7f92b5e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f92b81004d0 0x7f92b8199b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:48.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.834+0000 7f92afe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f92b81016c0 0x7f92b819a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:48.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.834+0000 7f92b5e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f92b81004d0 0x7f92b8199b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:47698/0 (socket says 172.21.15.39:47698) 2024-03-18T16:20:48.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.834+0000 7f92b5e00640 1 -- 172.21.15.39:0/2687048642 learned_addr learned my addr 172.21.15.39:0/2687048642 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:20:48.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.834+0000 7f92b6800640 1 --2- 172.21.15.39:0/2687048642 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f92b8102080 0x7f92b81a10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:48.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.834+0000 7f92b5e00640 1 -- 172.21.15.39:0/2687048642 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f92b8102080 msgr2=0x7f92b81a10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:48.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.835+0000 7f92b5e00640 1 --2- 172.21.15.39:0/2687048642 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f92b8102080 0x7f92b81a10b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.835+0000 7f92b5e00640 1 -- 172.21.15.39:0/2687048642 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f92b81016c0 msgr2=0x7f92b819a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:48.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.835+0000 7f92b5e00640 1 --2- 172.21.15.39:0/2687048642 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f92b81016c0 0x7f92b819a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.835+0000 7f92b5e00640 1 -- 172.21.15.39:0/2687048642 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92b81a1720 con 0x7f92b81004d0 2024-03-18T16:20:48.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.835+0000 7f92b6800640 1 --2- 172.21.15.39:0/2687048642 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f92b8102080 0x7f92b81a10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:20:48.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.835+0000 7f92b5e00640 1 --2- 172.21.15.39:0/2687048642 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f92b81004d0 0x7f92b8199b00 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f92a802efd0 tx=0x7f92a8005d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:48.838 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.835+0000 7f92aea00640 1 -- 172.21.15.39:0/2687048642 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92a8041070 con 0x7f92b81004d0 2024-03-18T16:20:48.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.835+0000 7f92aea00640 1 -- 172.21.15.39:0/2687048642 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f92a80044d0 con 0x7f92b81004d0 2024-03-18T16:20:48.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.835+0000 7f92b7200640 1 -- 172.21.15.39:0/2687048642 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92b81a19b0 con 0x7f92b81004d0 2024-03-18T16:20:48.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.835+0000 7f92aea00640 1 -- 172.21.15.39:0/2687048642 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92a8034440 con 0x7f92b81004d0 2024-03-18T16:20:48.839 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.835+0000 7f92b7200640 1 -- 172.21.15.39:0/2687048642 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92b81a1ec0 con 0x7f92b81004d0 2024-03-18T16:20:48.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.839+0000 7f92bfe68740 1 Processor -- start 2024-03-18T16:20:48.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.839+0000 7f92aea00640 1 -- 172.21.15.39:0/2687048642 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f92a8044070 con 0x7f92b81004d0 2024-03-18T16:20:48.840 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.839+0000 7f92bfe68740 1 -- start start 2024-03-18T16:20:48.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.839+0000 7f92bfe68740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556418492a90 0x556418492eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:48.841 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.839+0000 7f92bfe68740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556418493480 0x556418497f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:48.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.839+0000 7f92bfe68740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5564184985f0 0x55641849aa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:48.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.839+0000 7f92bfe68740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x55641845a3b0 con 0x556418493480 2024-03-18T16:20:48.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.839+0000 7f92bfe68740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5564183f7450 con 0x5564184985f0 2024-03-18T16:20:48.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.839+0000 7f92bfe68740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55641839fcb0 con 0x556418492a90 2024-03-18T16:20:48.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.839+0000 7f92b6800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556418492a90 0x556418492eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:48.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.840+0000 7f92b6800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556418492a90 0x556418492eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:47700/0 (socket says 172.21.15.39:47700) 2024-03-18T16:20:48.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.840+0000 7f92afe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5564184985f0 0x55641849aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:48.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.840+0000 7f92b6800640 1 -- 172.21.15.39:0/1325497139 learned_addr learned my addr 172.21.15.39:0/1325497139 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:20:48.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.840+0000 7f92b5e00640 1 --2- 172.21.15.39:0/1325497139 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556418493480 0x556418497f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:48.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.840+0000 7f92afe00640 1 -- 172.21.15.39:0/1325497139 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556418492a90 msgr2=0x556418492eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:48.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.840+0000 7f92afe00640 1 --2- 172.21.15.39:0/1325497139 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556418492a90 0x556418492eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.840+0000 7f92afe00640 1 -- 172.21.15.39:0/1325497139 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556418493480 msgr2=0x556418497f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:48.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.840+0000 7f92afe00640 1 --2- 172.21.15.39:0/1325497139 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556418493480 0x556418497f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.842 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.840+0000 7f92afe00640 1 -- 172.21.15.39:0/1325497139 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5564183c11f0 con 0x5564184985f0 2024-03-18T16:20:48.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.840+0000 7f92afe00640 1 --2- 172.21.15.39:0/1325497139 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5564184985f0 0x55641849aa80 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f92a000bbc0 tx=0x7f92a00107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea9e85b47041c679 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:48.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.841+0000 7f92a7400640 1 -- 172.21.15.39:0/1325497139 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92a0004030 con 0x5564184985f0 2024-03-18T16:20:48.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.841+0000 7f92a7400640 1 -- 172.21.15.39:0/1325497139 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f92a00041d0 con 0x5564184985f0 2024-03-18T16:20:48.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.841+0000 7f92a7400640 1 -- 172.21.15.39:0/1325497139 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92a00116a0 con 0x5564184985f0 2024-03-18T16:20:48.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.841+0000 7f92aea00640 1 --2- 172.21.15.39:0/2687048642 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f929c0a2880 0x7f929c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:48.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.841+0000 7f92aea00640 1 -- 172.21.15.39:0/2687048642 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f92a80f0960 con 0x7f92b81004d0 2024-03-18T16:20:48.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.841+0000 7f92b6800640 1 --2- 172.21.15.39:0/2687048642 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f929c0a2880 0x7f929c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:48.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.842+0000 7f92bfe68740 1 -- 172.21.15.39:0/1325497139 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5564184985f0 msgr2=0x55641849aa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:48.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.842+0000 7f92bfe68740 1 --2- 172.21.15.39:0/1325497139 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5564184985f0 0x55641849aa80 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f92a000bbc0 tx=0x7f92a00107a0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.842+0000 7f92bfe68740 1 -- 172.21.15.39:0/1325497139 shutdown_connections 2024-03-18T16:20:48.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.842+0000 7f92bfe68740 1 --2- 172.21.15.39:0/1325497139 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5564184985f0 0x55641849aa80 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.842+0000 7f92bfe68740 1 --2- 172.21.15.39:0/1325497139 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556418493480 0x556418497f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.842+0000 7f92bfe68740 1 --2- 172.21.15.39:0/1325497139 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556418492a90 0x556418492eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.842+0000 7f92bfe68740 1 -- 172.21.15.39:0/1325497139 >> 172.21.15.39:0/1325497139 conn(0x55641848e000 msgr2=0x55641848e400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:48.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.842+0000 7f92bfe68740 1 -- 172.21.15.39:0/1325497139 shutdown_connections 2024-03-18T16:20:48.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.842+0000 7f92b6800640 1 --2- 172.21.15.39:0/2687048642 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f929c0a2880 0x7f929c0a4d40 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x556418493ce0 tx=0x7f92b0009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:48.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.842+0000 7f92bfe68740 1 -- 172.21.15.39:0/1325497139 wait complete. 2024-03-18T16:20:48.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.843+0000 7f92bfe68740 1 -- start start 2024-03-18T16:20:48.845 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.843+0000 7f92bfe68740 1 Processor -- start 2024-03-18T16:20:48.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.843+0000 7f92bfe68740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556418492a90 0x55641849cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:48.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.843+0000 7f92bfe68740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556418493480 0x55641849d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:48.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.843+0000 7f92bfe68740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5564184985f0 0x5564184a4290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:48.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.843+0000 7f92bfe68740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x5564183b3110 con 0x556418493480 2024-03-18T16:20:48.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.843+0000 7f92bfe68740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5564183f34f0 con 0x5564184985f0 2024-03-18T16:20:48.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.843+0000 7f92bfe68740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55641849b5a0 con 0x556418492a90 2024-03-18T16:20:48.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.843+0000 7f92bfe68740 10 client.0 _ll_register_callbacks cb 0x5564183a0b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:20:48.848 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.844+0000 7f92b5e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556418492a90 0x55641849cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:48.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.844+0000 7f92b5e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556418492a90 0x55641849cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:47702/0 (socket says 172.21.15.39:47702) 2024-03-18T16:20:48.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.844+0000 7f92b5e00640 1 -- 172.21.15.39:0/3255314151 learned_addr learned my addr 172.21.15.39:0/3255314151 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:20:48.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.844+0000 7f92afe00640 1 --2- 172.21.15.39:0/3255314151 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556418493480 0x55641849d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:48.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.844+0000 7f92b6800640 1 --2- 172.21.15.39:0/3255314151 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5564184985f0 0x5564184a4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:48.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.844+0000 7f92b5e00640 1 -- 172.21.15.39:0/3255314151 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5564184985f0 msgr2=0x5564184a4290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:48.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.844+0000 7f92b5e00640 1 --2- 172.21.15.39:0/3255314151 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5564184985f0 0x5564184a4290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.844+0000 7f92b5e00640 1 -- 172.21.15.39:0/3255314151 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556418493480 msgr2=0x55641849d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:48.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.844+0000 7f92b5e00640 1 --2- 172.21.15.39:0/3255314151 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556418493480 0x55641849d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.844+0000 7f92b5e00640 1 -- 172.21.15.39:0/3255314151 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55641810b3f0 con 0x556418492a90 2024-03-18T16:20:48.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.844+0000 7f92b5e00640 1 --2- 172.21.15.39:0/3255314151 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556418492a90 0x55641849cd50 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f92a802c180 tx=0x7f92a80b9260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6a05fe5cb6e6913c server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:48.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.844+0000 7f929be00640 10 client.0 ms_handle_connect on v2:172.21.15.196:3300/0 2024-03-18T16:20:48.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.845+0000 7f929be00640 1 -- 172.21.15.39:0/3255314151 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92a80f2e60 con 0x556418492a90 2024-03-18T16:20:48.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.845+0000 7f929be00640 1 -- 172.21.15.39:0/3255314151 <== mon.1 v2:172.21.15.196:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f92a80c5420 con 0x556418492a90 2024-03-18T16:20:48.849 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.845+0000 7f92bfe68740 1 -- 172.21.15.39:0/3255314151 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f92a00041d0 con 0x556418492a90 2024-03-18T16:20:48.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.845+0000 7f929be00640 1 -- 172.21.15.39:0/3255314151 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f92a804b030 con 0x556418492a90 2024-03-18T16:20:48.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.845+0000 7f92bfe68740 10 client.27838 fetch_fsmap learned FSMap version 48 2024-03-18T16:20:48.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.845+0000 7f92bfe68740 1 -- 172.21.15.39:0/3255314151 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x5564181d57d0 con 0x556418492a90 2024-03-18T16:20:48.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.845+0000 7f929be00640 1 -- 172.21.15.39:0/3255314151 <== mon.1 v2:172.21.15.196:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f92a00041d0 con 0x556418492a90 2024-03-18T16:20:48.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.845+0000 7f929be00640 1 -- 172.21.15.39:0/3255314151 <== mon.1 v2:172.21.15.196:3300/0 5 ==== fsmap(e 48) ==== 8386+0+0 (secure 0 0 0) 0x7f92a80d2020 con 0x556418492a90 2024-03-18T16:20:48.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.845+0000 7f92bfe68740 10 client.27838 fetch_fsmap finished waiting for FSMap version 48 2024-03-18T16:20:48.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.845+0000 7f92bfe68740 10 client.27838 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:20:48.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.845+0000 7f92bfe68740 20 client.27838 populate_metadata read hostname 'smithi039' 2024-03-18T16:20:48.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.845+0000 7f92bfe68740 1 --2- 172.21.15.39:0/3255314151 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5564184a8b30 0x5564184aaff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:48.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.846+0000 7f92bfe68740 4 client.27838 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-18T16:20:48.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.846+0000 7f92bfe68740 1 -- 172.21.15.39:0/3255314151 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556418404990 con 0x5564184a8b30 2024-03-18T16:20:48.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.846+0000 7f92afe00640 1 --2- 172.21.15.39:0/3255314151 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5564184a8b30 0x5564184aaff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:48.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.846+0000 7f92afe00640 1 --2- 172.21.15.39:0/3255314151 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5564184a8b30 0x5564184aaff0 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d6a48c4b3f5e9cfc server_cookie=496fb3a66339738e in_seq=0 out_seq=0 2024-03-18T16:20:48.850 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.846+0000 7f929be00640 10 client.27838 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:20:48.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.847+0000 7f929be00640 1 -- 172.21.15.39:0/3255314151 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x556418404990 con 0x5564184a8b30 2024-03-18T16:20:48.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.847+0000 7f929be00640 10 client.27838 handle_command_reply: tid=0 2024-03-18T16:20:48.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.847+0000 7f92bfe68740 1 client.27838 shutdown 2024-03-18T16:20:48.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.848+0000 7f92bfe68740 1 -- 172.21.15.39:0/3255314151 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556418492a90 msgr2=0x55641849cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:48.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.848+0000 7f92bfe68740 1 --2- 172.21.15.39:0/3255314151 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556418492a90 0x55641849cd50 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f92a802c180 tx=0x7f92a80b9260 comp rx=0 tx=0).stop 2024-03-18T16:20:48.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.848+0000 7f92bfe68740 1 -- 172.21.15.39:0/3255314151 shutdown_connections 2024-03-18T16:20:48.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.848+0000 7f92bfe68740 1 --2- 172.21.15.39:0/3255314151 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x5564184a8b30 0x5564184aaff0 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.848+0000 7f92bfe68740 1 --2- 172.21.15.39:0/3255314151 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5564184985f0 0x5564184a4290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.848+0000 7f92bfe68740 1 --2- 172.21.15.39:0/3255314151 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556418493480 0x55641849d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.848+0000 7f929be00640 0 client.27838 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:20:48.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.848+0000 7f92bfe68740 1 --2- 172.21.15.39:0/3255314151 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556418492a90 0x55641849cd50 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:48.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.848+0000 7f92bfe68740 1 -- 172.21.15.39:0/3255314151 >> 172.21.15.39:0/3255314151 conn(0x55641848e000 msgr2=0x55641848d220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:48.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.848+0000 7f92bfe68740 1 -- 172.21.15.39:0/3255314151 shutdown_connections 2024-03-18T16:20:48.852 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.848+0000 7f92bfe68740 1 -- 172.21.15.39:0/3255314151 wait complete. 2024-03-18T16:20:48.853 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.848+0000 7f92bfe68740 20 client.27838 trim_cache size 0 max 16384 2024-03-18T16:20:48.868 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.866+0000 7f92bfe68740 1 Processor -- start 2024-03-18T16:20:49.001 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.866+0000 7f92bfe68740 1 -- start start 2024-03-18T16:20:49.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.866+0000 7f92bfe68740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556418492a90 0x55641848c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:49.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.866+0000 7f92bfe68740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556418493480 0x5564184a4250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:49.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.866+0000 7f92bfe68740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5564184985f0 0x55641848ced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:49.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.866+0000 7f92bfe68740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f92b81ac3f0 con 0x556418493480 2024-03-18T16:20:49.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.866+0000 7f92bfe68740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f92b81adff0 con 0x5564184985f0 2024-03-18T16:20:49.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.866+0000 7f92bfe68740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f92b81acac0 con 0x556418492a90 2024-03-18T16:20:49.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.866+0000 7f92b6800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5564184985f0 0x55641848ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:49.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.866+0000 7f92b5e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556418492a90 0x55641848c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:49.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.866+0000 7f92afe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556418493480 0x5564184a4250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:49.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.866+0000 7f92b6800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5564184985f0 0x55641848ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:39100/0 (socket says 172.21.15.39:39100) 2024-03-18T16:20:49.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.866+0000 7f92b6800640 1 -- 172.21.15.39:0/3366361392 learned_addr learned my addr 172.21.15.39:0/3366361392 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:20:49.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.867+0000 7f92b6800640 1 -- 172.21.15.39:0/3366361392 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556418492a90 msgr2=0x55641848c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:49.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.867+0000 7f92b6800640 1 --2- 172.21.15.39:0/3366361392 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556418492a90 0x55641848c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:49.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.867+0000 7f92b6800640 1 -- 172.21.15.39:0/3366361392 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556418493480 msgr2=0x5564184a4250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:49.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.867+0000 7f92b6800640 1 --2- 172.21.15.39:0/3366361392 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556418493480 0x5564184a4250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:49.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.867+0000 7f92b6800640 1 -- 172.21.15.39:0/3366361392 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55641848d950 con 0x5564184985f0 2024-03-18T16:20:49.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.867+0000 7f92b6800640 1 --2- 172.21.15.39:0/3366361392 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5564184985f0 0x55641848ced0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f92b00119b0 tx=0x7f92b0016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=811e81e7ef4fe3f server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:49.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.867+0000 7f92a6a00640 1 -- 172.21.15.39:0/3366361392 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92b001c590 con 0x5564184985f0 2024-03-18T16:20:49.013 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.867+0000 7f92a6a00640 1 -- 172.21.15.39:0/3366361392 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f92b002d070 con 0x5564184985f0 2024-03-18T16:20:49.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.867+0000 7f92a6a00640 1 -- 172.21.15.39:0/3366361392 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92b0028b40 con 0x5564184985f0 2024-03-18T16:20:49.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.868+0000 7f92bfe68740 1 -- 172.21.15.39:0/3366361392 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5564184985f0 msgr2=0x55641848ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:49.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.868+0000 7f92bfe68740 1 --2- 172.21.15.39:0/3366361392 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5564184985f0 0x55641848ced0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f92b00119b0 tx=0x7f92b0016000 comp rx=0 tx=0).stop 2024-03-18T16:20:49.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.868+0000 7f92bfe68740 1 -- 172.21.15.39:0/3366361392 shutdown_connections 2024-03-18T16:20:49.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.868+0000 7f92bfe68740 1 --2- 172.21.15.39:0/3366361392 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5564184985f0 0x55641848ced0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:49.014 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.868+0000 7f92bfe68740 1 --2- 172.21.15.39:0/3366361392 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x556418493480 0x5564184a4250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:49.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.868+0000 7f92bfe68740 1 --2- 172.21.15.39:0/3366361392 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556418492a90 0x55641848c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:49.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.868+0000 7f92bfe68740 1 -- 172.21.15.39:0/3366361392 >> 172.21.15.39:0/3366361392 conn(0x55641848e000 msgr2=0x55641848e3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:49.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.868+0000 7f92bfe68740 1 -- 172.21.15.39:0/3366361392 shutdown_connections 2024-03-18T16:20:49.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.868+0000 7f92bfe68740 1 -- 172.21.15.39:0/3366361392 wait complete. 2024-03-18T16:20:49.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.869+0000 7f92bfe68740 1 -- start start 2024-03-18T16:20:49.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.869+0000 7f92bfe68740 1 Processor -- start 2024-03-18T16:20:49.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.869+0000 7f92bfe68740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556418492a90 0x5564184a2040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:49.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.869+0000 7f92bfe68740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556418493480 0x5564184a2580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:49.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.869+0000 7f92bfe68740 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5564184985f0 0x5564184a3590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:49.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.869+0000 7f92bfe68740 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f92b81ad8f0 con 0x5564184985f0 2024-03-18T16:20:49.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.869+0000 7f92bfe68740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f92b81ad1f0 con 0x556418493480 2024-03-18T16:20:49.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.869+0000 7f92bfe68740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f92b81aa3d0 con 0x556418492a90 2024-03-18T16:20:49.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.870+0000 7f92bfe68740 10 client.0 _ll_register_callbacks cb 0x7f92a8005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-18T16:20:49.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.870+0000 7f92afe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556418493480 0x5564184a2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:49.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.870+0000 7f92b5e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556418492a90 0x5564184a2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:49.016 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.870+0000 7f92afe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556418493480 0x5564184a2580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:39108/0 (socket says 172.21.15.39:39108) 2024-03-18T16:20:49.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.870+0000 7f92afe00640 1 -- 172.21.15.39:0/2442538018 learned_addr learned my addr 172.21.15.39:0/2442538018 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:20:49.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.870+0000 7f92b5e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556418492a90 0x5564184a2040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:47724/0 (socket says 172.21.15.39:47724) 2024-03-18T16:20:49.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.870+0000 7f92b6800640 1 --2- 172.21.15.39:0/2442538018 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5564184985f0 0x5564184a3590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:49.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.870+0000 7f92afe00640 1 -- 172.21.15.39:0/2442538018 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556418492a90 msgr2=0x5564184a2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:49.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.870+0000 7f92afe00640 1 --2- 172.21.15.39:0/2442538018 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556418492a90 0x5564184a2040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:49.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.870+0000 7f92afe00640 1 -- 172.21.15.39:0/2442538018 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5564184985f0 msgr2=0x5564184a3590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-18T16:20:49.017 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.870+0000 7f92afe00640 1 --2- 172.21.15.39:0/2442538018 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5564184985f0 0x5564184a3590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:49.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.870+0000 7f92afe00640 1 -- 172.21.15.39:0/2442538018 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55641848dba0 con 0x556418493480 2024-03-18T16:20:49.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.870+0000 7f92b5e00640 1 --2- 172.21.15.39:0/2442538018 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556418492a90 0x5564184a2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:20:49.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.871+0000 7f92afe00640 1 --2- 172.21.15.39:0/2442538018 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556418493480 0x5564184a2580 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x55641848ef70 tx=0x7f92a000b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b00b5a76b2958b85 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:49.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.871+0000 7f929be00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-03-18T16:20:49.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.871+0000 7f929be00640 1 -- 172.21.15.39:0/2442538018 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92a0005250 con 0x556418493480 2024-03-18T16:20:49.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.871+0000 7f929be00640 1 -- 172.21.15.39:0/2442538018 <== mon.2 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f92a00053f0 con 0x556418493480 2024-03-18T16:20:49.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.871+0000 7f92bfe68740 1 -- 172.21.15.39:0/2442538018 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f92b002d070 con 0x556418493480 2024-03-18T16:20:49.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.871+0000 7f929be00640 1 -- 172.21.15.39:0/2442538018 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) ==== 24+0+0 (secure 0 0 0) 0x7f92a0039050 con 0x556418493480 2024-03-18T16:20:49.018 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.871+0000 7f92bfe68740 10 client.27857 fetch_fsmap learned FSMap version 48 2024-03-18T16:20:49.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.871+0000 7f92bfe68740 1 -- 172.21.15.39:0/2442538018 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=4+}) -- 0x55641850f4c0 con 0x556418493480 2024-03-18T16:20:49.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.872+0000 7f929be00640 1 -- 172.21.15.39:0/2442538018 <== mon.2 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f92b002d070 con 0x556418493480 2024-03-18T16:20:49.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.872+0000 7f929be00640 1 -- 172.21.15.39:0/2442538018 <== mon.2 v2:172.21.15.194:3300/0 5 ==== fsmap(e 48) ==== 8386+0+0 (secure 0 0 0) 0x7f92a0003950 con 0x556418493480 2024-03-18T16:20:49.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.872+0000 7f92bfe68740 10 client.27857 fetch_fsmap finished waiting for FSMap version 48 2024-03-18T16:20:49.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.872+0000 7f92bfe68740 10 client.27857 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-18T16:20:49.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.872+0000 7f92bfe68740 20 client.27857 populate_metadata read hostname 'smithi039' 2024-03-18T16:20:49.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.872+0000 7f92bfe68740 1 --2- 172.21.15.39:0/2442538018 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55641850fc20 0x556418512040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:49.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.872+0000 7f92bfe68740 4 client.27857 mds_command: new command op to 24575 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-18T16:20:49.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.872+0000 7f92bfe68740 1 -- 172.21.15.39:0/2442538018 --> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5564183fece0 con 0x55641850fc20 2024-03-18T16:20:49.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.872+0000 7f92b5e00640 1 --2- 172.21.15.39:0/2442538018 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55641850fc20 0x556418512040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:49.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.873+0000 7f92b5e00640 1 --2- 172.21.15.39:0/2442538018 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55641850fc20 0x556418512040 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf7779467b9af377 server_cookie=e1552769b5cc1a44 in_seq=0 out_seq=0 2024-03-18T16:20:49.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.873+0000 7f929be00640 10 client.27857 ms_handle_connect on v2:172.21.15.194:6838/274911158 2024-03-18T16:20:49.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.873+0000 7f929be00640 1 -- 172.21.15.39:0/2442538018 <== mds.0 v2:172.21.15.194:6838/274911158 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5564183fece0 con 0x55641850fc20 2024-03-18T16:20:49.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.873+0000 7f929be00640 10 client.27857 handle_command_reply: tid=0 2024-03-18T16:20:49.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.873+0000 7f92bfe68740 1 client.27857 shutdown 2024-03-18T16:20:49.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.873+0000 7f92bfe68740 1 -- 172.21.15.39:0/2442538018 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556418493480 msgr2=0x5564184a2580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:49.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.873+0000 7f92bfe68740 1 --2- 172.21.15.39:0/2442538018 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556418493480 0x5564184a2580 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x55641848ef70 tx=0x7f92a000b800 comp rx=0 tx=0).stop 2024-03-18T16:20:49.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.873+0000 7f92bfe68740 1 -- 172.21.15.39:0/2442538018 shutdown_connections 2024-03-18T16:20:49.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.873+0000 7f92bfe68740 1 --2- 172.21.15.39:0/2442538018 >> [v2:172.21.15.194:6838/274911158,v1:172.21.15.194:6839/274911158] conn(0x55641850fc20 0x556418512040 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:49.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.873+0000 7f92bfe68740 1 --2- 172.21.15.39:0/2442538018 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556418493480 0x5564184a2580 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:49.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.873+0000 7f92bfe68740 1 --2- 172.21.15.39:0/2442538018 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x5564184985f0 0x5564184a3590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:49.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.873+0000 7f92bfe68740 1 --2- 172.21.15.39:0/2442538018 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556418492a90 0x5564184a2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:49.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.873+0000 7f92bfe68740 1 -- 172.21.15.39:0/2442538018 >> 172.21.15.39:0/2442538018 conn(0x55641848e000 msgr2=0x55641848c990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:49.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.873+0000 7f929be00640 0 client.27857 ms_handle_reset on v2:172.21.15.194:6838/274911158 2024-03-18T16:20:49.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.873+0000 7f92bfe68740 1 -- 172.21.15.39:0/2442538018 shutdown_connections 2024-03-18T16:20:49.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.873+0000 7f92bfe68740 1 -- 172.21.15.39:0/2442538018 wait complete. 2024-03-18T16:20:49.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.874+0000 7f92bfe68740 20 client.27857 trim_cache size 0 max 16384 2024-03-18T16:20:49.022 INFO:teuthology.orchestra.run.smithi039.stdout:[] 2024-03-18T16:20:49.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.874+0000 7f92b7200640 1 -- 172.21.15.39:0/2687048642 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f929c0a2880 msgr2=0x7f929c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:49.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.874+0000 7f92b7200640 1 --2- 172.21.15.39:0/2687048642 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f929c0a2880 0x7f929c0a4d40 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x556418493ce0 tx=0x7f92b0009500 comp rx=0 tx=0).stop 2024-03-18T16:20:49.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.874+0000 7f92b7200640 1 -- 172.21.15.39:0/2687048642 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f92b81004d0 msgr2=0x7f92b8199b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:49.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.874+0000 7f92b7200640 1 --2- 172.21.15.39:0/2687048642 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f92b81004d0 0x7f92b8199b00 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f92a802efd0 tx=0x7f92a8005d50 comp rx=0 tx=0).stop 2024-03-18T16:20:49.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.874+0000 7f92b7200640 1 -- 172.21.15.39:0/2687048642 shutdown_connections 2024-03-18T16:20:49.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.874+0000 7f92b7200640 1 --2- 172.21.15.39:0/2687048642 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f92b8102080 0x7f92b81a10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:49.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.874+0000 7f92b7200640 1 --2- 172.21.15.39:0/2687048642 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f929c0a2880 0x7f929c0a4d40 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:49.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.874+0000 7f92b7200640 1 --2- 172.21.15.39:0/2687048642 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f92b81016c0 0x7f92b819a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:49.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.874+0000 7f92b7200640 1 --2- 172.21.15.39:0/2687048642 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f92b81004d0 0x7f92b8199b00 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:49.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.874+0000 7f92b7200640 1 -- 172.21.15.39:0/2687048642 >> 172.21.15.39:0/2687048642 conn(0x7f92b80fba80 msgr2=0x7f92b80fd4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:49.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.875+0000 7f92b7200640 1 -- 172.21.15.39:0/2687048642 shutdown_connections 2024-03-18T16:20:49.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:48.875+0000 7f92b7200640 1 -- 172.21.15.39:0/2687048642 wait complete. 2024-03-18T16:20:49.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:49 smithi039 ceph-mon[31721]: pgmap v744: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 597 B/s wr, 10 op/s 2024-03-18T16:20:49.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:49 smithi196 ceph-mon[36307]: pgmap v744: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 597 B/s wr, 10 op/s 2024-03-18T16:20:49.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:49 smithi194 ceph-mon[36200]: pgmap v744: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 597 B/s wr, 10 op/s 2024-03-18T16:20:50.025 INFO:tasks.fwd_scrub.fs.[cephfs]:end scrubbing fs: cephfs 2024-03-18T16:20:50.026 INFO:tasks.fwd_scrub:done joining 2024-03-18T16:20:50.026 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-03-18T16:20:50.053 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T16:20:50.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.289+0000 7f8a05e00640 1 -- 172.21.15.39:0/3169809695 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8a00103a00 msgr2=0x7f8a00103e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:50.291 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.289+0000 7f8a05e00640 1 --2- 172.21.15.39:0/3169809695 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8a00103a00 0x7f8a00103e20 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f89f0009900 tx=0x7f89f002bc30 comp rx=0 tx=0).stop 2024-03-18T16:20:50.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.289+0000 7f8a05e00640 1 -- 172.21.15.39:0/3169809695 shutdown_connections 2024-03-18T16:20:50.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.289+0000 7f8a05e00640 1 --2- 172.21.15.39:0/3169809695 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8a001055b0 0x7f8a0010be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:50.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.289+0000 7f8a05e00640 1 --2- 172.21.15.39:0/3169809695 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a00104bf0 0x7f8a00105070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:50.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.289+0000 7f8a05e00640 1 --2- 172.21.15.39:0/3169809695 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8a00103a00 0x7f8a00103e20 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:50.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.289+0000 7f8a05e00640 1 -- 172.21.15.39:0/3169809695 >> 172.21.15.39:0/3169809695 conn(0x7f8a000fef90 msgr2=0x7f8a001013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:50.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.289+0000 7f8a05e00640 1 -- 172.21.15.39:0/3169809695 shutdown_connections 2024-03-18T16:20:50.292 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.289+0000 7f8a05e00640 1 -- 172.21.15.39:0/3169809695 wait complete. 2024-03-18T16:20:50.294 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.290+0000 7f8a05e00640 1 Processor -- start 2024-03-18T16:20:50.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.291+0000 7f8a05e00640 1 -- start start 2024-03-18T16:20:50.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.291+0000 7f8a05e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8a00103a00 0x7f8a00199db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:50.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.291+0000 7f8a05e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a00104bf0 0x7f8a0019a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:50.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.291+0000 7f8a05e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8a001055b0 0x7f8a001a1310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:50.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.291+0000 7f8a05e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f8a0010f680 con 0x7f8a00104bf0 2024-03-18T16:20:50.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.291+0000 7f8a05e00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f8a0010f500 con 0x7f8a001055b0 2024-03-18T16:20:50.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.291+0000 7f8a05e00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f8a0010f800 con 0x7f8a00103a00 2024-03-18T16:20:50.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.291+0000 7f89fea00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8a00103a00 0x7f8a00199db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:50.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.291+0000 7f89fea00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8a00103a00 0x7f8a00199db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:47740/0 (socket says 172.21.15.39:47740) 2024-03-18T16:20:50.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.291+0000 7f89fea00640 1 -- 172.21.15.39:0/3860595456 learned_addr learned my addr 172.21.15.39:0/3860595456 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:20:50.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.291+0000 7f89fe000640 1 --2- 172.21.15.39:0/3860595456 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a00104bf0 0x7f8a0019a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:50.295 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.291+0000 7f89ff400640 1 --2- 172.21.15.39:0/3860595456 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8a001055b0 0x7f8a001a1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:50.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.292+0000 7f89fea00640 1 -- 172.21.15.39:0/3860595456 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8a001055b0 msgr2=0x7f8a001a1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:50.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.292+0000 7f89fea00640 1 --2- 172.21.15.39:0/3860595456 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8a001055b0 0x7f8a001a1310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:50.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.292+0000 7f89fea00640 1 -- 172.21.15.39:0/3860595456 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a00104bf0 msgr2=0x7f8a0019a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:50.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.292+0000 7f89fea00640 1 --2- 172.21.15.39:0/3860595456 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a00104bf0 0x7f8a0019a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:50.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.292+0000 7f89fea00640 1 -- 172.21.15.39:0/3860595456 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a001a1980 con 0x7f8a00103a00 2024-03-18T16:20:50.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.292+0000 7f89fea00640 1 --2- 172.21.15.39:0/3860595456 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8a00103a00 0x7f8a00199db0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f89f0000c00 tx=0x7f89f002c640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:50.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.292+0000 7f8a04a00640 1 -- 172.21.15.39:0/3860595456 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89f0041070 con 0x7f8a00103a00 2024-03-18T16:20:50.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.292+0000 7f8a04a00640 1 -- 172.21.15.39:0/3860595456 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89f0004450 con 0x7f8a00103a00 2024-03-18T16:20:50.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.293+0000 7f8a04a00640 1 -- 172.21.15.39:0/3860595456 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89f0034440 con 0x7f8a00103a00 2024-03-18T16:20:50.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.293+0000 7f8a05e00640 1 -- 172.21.15.39:0/3860595456 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a001a1c10 con 0x7f8a00103a00 2024-03-18T16:20:50.296 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.293+0000 7f8a05e00640 1 -- 172.21.15.39:0/3860595456 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a001a2100 con 0x7f8a00103a00 2024-03-18T16:20:50.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.294+0000 7f8a05e00640 1 -- 172.21.15.39:0/3860595456 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a0010ad70 con 0x7f8a00103a00 2024-03-18T16:20:50.297 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.296+0000 7f8a04a00640 1 -- 172.21.15.39:0/3860595456 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f89f0044070 con 0x7f8a00103a00 2024-03-18T16:20:50.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.298+0000 7f8a04a00640 1 --2- 172.21.15.39:0/3860595456 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f89e40a2800 0x7f89e40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:50.365 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.298+0000 7f8a04a00640 1 -- 172.21.15.39:0/3860595456 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f89f00f0910 con 0x7f8a00103a00 2024-03-18T16:20:50.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.298+0000 7f89fe000640 1 --2- 172.21.15.39:0/3860595456 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f89e40a2800 0x7f89e40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:50.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.299+0000 7f89fe000640 1 --2- 172.21.15.39:0/3860595456 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f89e40a2800 0x7f89e40a4cc0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f8a0019b270 tx=0x7f89ec005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:50.366 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.301+0000 7f8a04a00640 1 -- 172.21.15.39:0/3860595456 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f89f00bd000 con 0x7f8a00103a00 2024-03-18T16:20:50.513 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.511+0000 7f8a05e00640 1 -- 172.21.15.39:0/3860595456 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8a00062410 con 0x7f8a00103a00 2024-03-18T16:20:50.515 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.514+0000 7f8a04a00640 1 -- 172.21.15.39:0/3860595456 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24762 (secure 0 0 0) 0x7f89f00c10a0 con 0x7f8a00103a00 2024-03-18T16:20:50.516 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:20:50.516 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":93,"fsid":"1892d8fc-e540-11ee-95c9-87774f69a715","created":"2024-03-18T15:57:36.507008+0000","modified":"2024-03-18T16:08:47.874967+0000","last_up_change":"2024-03-18T16:05:45.250629+0000","last_in_change":"2024-03-18T16:05:24.841337+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T16:01:30.048247+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T16:06:42.191791+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T16:06:43.670325+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T16:06:47.523388+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"771e0558-be62-4436-94b0-6364751911c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6803","nonce":4090743635}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6805","nonce":4090743635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6809","nonce":4090743635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6807","nonce":4090743635}]},"public_addr":"172.21.15.39:6803/4090743635","cluster_addr":"172.21.15.39:6805/4090743635","heartbeat_back_addr":"172.21.15.39:6809/4090743635","heartbeat_front_addr":"172.21.15.39:6807/4090743635","state":["exists","up"]},{"osd":1,"uuid":"0430eee1-cf34-4277-b255-c7d9f07ad19a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6801","nonce":2800329805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6803","nonce":2800329805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6807","nonce":2800329805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6805","nonce":2800329805}]},"public_addr":"172.21.15.194:6801/2800329805","cluster_addr":"172.21.15.194:6803/2800329805","heartbeat_back_addr":"172.21.15.194:6807/2800329805","heartbeat_front_addr":"172.21.15.194:6805/2800329805","state":["exists","up"]},{"osd":2,"uuid":"3a718da4-2511-4ec1-b1a8-eafd6cce0df0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6801","nonce":3935874719}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6803","nonce":3935874719}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6807","nonce":3935874719}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6805","nonce":3935874719}]},"public_addr":"172.21.15.196:6801/3935874719","cluster_addr":"172.21.15.196:6803/3935874719","heartbeat_back_addr":"172.21.15.196:6807/3935874719","heartbeat_front_addr":"172.21.15.196:6805/3935874719","state":["exists","up"]},{"osd":3,"uuid":"6f9c37f9-0ab9-4038-8919-afd4b39c4165","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6811","nonce":893345939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6813","nonce":893345939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6817","nonce":893345939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6815","nonce":893345939}]},"public_addr":"172.21.15.39:6811/893345939","cluster_addr":"172.21.15.39:6813/893345939","heartbeat_back_addr":"172.21.15.39:6817/893345939","heartbeat_front_addr":"172.21.15.39:6815/893345939","state":["exists","up"]},{"osd":4,"uuid":"e795bb0e-4791-4537-acda-1a321e19bc9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6809","nonce":800423088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6811","nonce":800423088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6815","nonce":800423088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6813","nonce":800423088}]},"public_addr":"172.21.15.194:6809/800423088","cluster_addr":"172.21.15.194:6811/800423088","heartbeat_back_addr":"172.21.15.194:6815/800423088","heartbeat_front_addr":"172.21.15.194:6813/800423088","state":["exists","up"]},{"osd":5,"uuid":"026e175e-6acb-4b4c-b235-8ccee8ae1149","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6809","nonce":2746224525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6811","nonce":2746224525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6815","nonce":2746224525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6813","nonce":2746224525}]},"public_addr":"172.21.15.196:6809/2746224525","cluster_addr":"172.21.15.196:6811/2746224525","heartbeat_back_addr":"172.21.15.196:6815/2746224525","heartbeat_front_addr":"172.21.15.196:6813/2746224525","state":["exists","up"]},{"osd":6,"uuid":"c1b183ff-383f-4c5c-9075-f76bab1c0bb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6819","nonce":1213371331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6821","nonce":1213371331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6825","nonce":1213371331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6823","nonce":1213371331}]},"public_addr":"172.21.15.39:6819/1213371331","cluster_addr":"172.21.15.39:6821/1213371331","heartbeat_back_addr":"172.21.15.39:6825/1213371331","heartbeat_front_addr":"172.21.15.39:6823/1213371331","state":["exists","up"]},{"osd":7,"uuid":"612f8697-957d-4ef4-a87f-241b36f9d13d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6817","nonce":3299162444}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6819","nonce":3299162444}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6823","nonce":3299162444}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6821","nonce":3299162444}]},"public_addr":"172.21.15.194:6817/3299162444","cluster_addr":"172.21.15.194:6819/3299162444","heartbeat_back_addr":"172.21.15.194:6823/3299162444","heartbeat_front_addr":"172.21.15.194:6821/3299162444","state":["exists","up"]},{"osd":8,"uuid":"836a9107-4d4f-4594-b8dc-0301e64f59d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6817","nonce":2456143145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6819","nonce":2456143145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6823","nonce":2456143145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6821","nonce":2456143145}]},"public_addr":"172.21.15.196:6817/2456143145","cluster_addr":"172.21.15.196:6819/2456143145","heartbeat_back_addr":"172.21.15.196:6823/2456143145","heartbeat_front_addr":"172.21.15.196:6821/2456143145","state":["exists","up"]},{"osd":9,"uuid":"4f30b960-12e3-4b16-bd28-fcc988139b58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6827","nonce":1848634433}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6829","nonce":1848634433}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6833","nonce":1848634433}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6831","nonce":1848634433}]},"public_addr":"172.21.15.39:6827/1848634433","cluster_addr":"172.21.15.39:6829/1848634433","heartbeat_back_addr":"172.21.15.39:6833/1848634433","heartbeat_front_addr":"172.21.15.39:6831/1848634433","state":["exists","up"]},{"osd":10,"uuid":"620f90e8-23e6-4bc0-8269-7219b9e1aebd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6825","nonce":3725064809}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6827","nonce":3725064809}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6831","nonce":3725064809}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6829","nonce":3725064809}]},"public_addr":"172.21.15.194:6825/3725064809","cluster_addr":"172.21.15.194:6827/3725064809","heartbeat_back_addr":"172.21.15.194:6831/3725064809","heartbeat_front_addr":"172.21.15.194:6829/3725064809","state":["exists","up"]},{"osd":11,"uuid":"867a2535-8235-4ceb-a698-eb86869d6d20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6825","nonce":2501212422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6827","nonce":2501212422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6831","nonce":2501212422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6829","nonce":2501212422}]},"public_addr":"172.21.15.196:6825/2501212422","cluster_addr":"172.21.15.196:6827/2501212422","heartbeat_back_addr":"172.21.15.196:6831/2501212422","heartbeat_front_addr":"172.21.15.196:6829/2501212422","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:00:36.931566+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:01:02.614024+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:01:26.193265+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:01:58.308833+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:02:25.049546+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:02:49.748750+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:03:23.582153+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:03:50.447321+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:04:15.812129+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:04:49.998047+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:05:17.087924+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:05:43.167637+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":4,"to":7}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.a","mappings":[{"from":5,"to":8}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.39:0/3482086140":"2024-03-19T15:58:45.340294+0000","172.21.15.39:0/3844306284":"2024-03-19T15:58:45.340294+0000","172.21.15.39:6801/1810238546":"2024-03-19T15:58:45.340294+0000","172.21.15.39:6800/1810238546":"2024-03-19T15:58:45.340294+0000","172.21.15.39:0/2528397488":"2024-03-19T15:58:10.264798+0000","172.21.15.39:0/2240981254":"2024-03-19T15:58:10.264798+0000","172.21.15.39:6800/1217503189":"2024-03-19T15:58:10.264798+0000","172.21.15.39:0/3754876370":"2024-03-19T15:58:10.264798+0000","172.21.15.39:6801/1217503189":"2024-03-19T15:58:10.264798+0000","172.21.15.39:0/1874637628":"2024-03-19T15:58:45.340294+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T16:20:50.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.517+0000 7f8a05e00640 1 -- 172.21.15.39:0/3860595456 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f89e40a2800 msgr2=0x7f89e40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:50.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.517+0000 7f8a05e00640 1 --2- 172.21.15.39:0/3860595456 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f89e40a2800 0x7f89e40a4cc0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f8a0019b270 tx=0x7f89ec005f90 comp rx=0 tx=0).stop 2024-03-18T16:20:50.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.517+0000 7f8a05e00640 1 -- 172.21.15.39:0/3860595456 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8a00103a00 msgr2=0x7f8a00199db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:50.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.517+0000 7f8a05e00640 1 --2- 172.21.15.39:0/3860595456 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8a00103a00 0x7f8a00199db0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f89f0000c00 tx=0x7f89f002c640 comp rx=0 tx=0).stop 2024-03-18T16:20:50.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.517+0000 7f8a05e00640 1 -- 172.21.15.39:0/3860595456 shutdown_connections 2024-03-18T16:20:50.519 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.517+0000 7f8a05e00640 1 --2- 172.21.15.39:0/3860595456 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8a001055b0 0x7f8a001a1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:50.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.517+0000 7f8a05e00640 1 --2- 172.21.15.39:0/3860595456 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f89e40a2800 0x7f89e40a4cc0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:50.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.517+0000 7f8a05e00640 1 --2- 172.21.15.39:0/3860595456 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8a00104bf0 0x7f8a0019a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:50.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.517+0000 7f8a05e00640 1 --2- 172.21.15.39:0/3860595456 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8a00103a00 0x7f8a00199db0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:50.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.517+0000 7f8a05e00640 1 -- 172.21.15.39:0/3860595456 >> 172.21.15.39:0/3860595456 conn(0x7f8a000fef90 msgr2=0x7f8a00107f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:50.520 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.517+0000 7f8a05e00640 1 -- 172.21.15.39:0/3860595456 shutdown_connections 2024-03-18T16:20:50.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.517+0000 7f8a05e00640 1 -- 172.21.15.39:0/3860595456 wait complete. 2024-03-18T16:20:50.569 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T16:01:30.048247+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T16:06:42.191791+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T16:06:43.670325+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T16:06:47.523388+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T16:20:50.569 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T16:20:50.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.800+0000 7f8699000640 1 -- 172.21.15.39:0/3609655642 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f86941038e0 msgr2=0x7f8694103d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:50.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.800+0000 7f8699000640 1 --2- 172.21.15.39:0/3609655642 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f86941038e0 0x7f8694103d00 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f8684009900 tx=0x7f868402bc30 comp rx=0 tx=0).stop 2024-03-18T16:20:50.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.800+0000 7f8699000640 1 -- 172.21.15.39:0/3609655642 shutdown_connections 2024-03-18T16:20:50.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.800+0000 7f8699000640 1 --2- 172.21.15.39:0/3609655642 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8694105490 0x7f869410bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:50.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.800+0000 7f8699000640 1 --2- 172.21.15.39:0/3609655642 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8694104ad0 0x7f8694104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:50.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.800+0000 7f8699000640 1 --2- 172.21.15.39:0/3609655642 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f86941038e0 0x7f8694103d00 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:50.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.800+0000 7f8699000640 1 -- 172.21.15.39:0/3609655642 >> 172.21.15.39:0/3609655642 conn(0x7f86940fee50 msgr2=0x7f8694101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:50.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.800+0000 7f8699000640 1 -- 172.21.15.39:0/3609655642 shutdown_connections 2024-03-18T16:20:50.802 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.800+0000 7f8699000640 1 -- 172.21.15.39:0/3609655642 wait complete. 2024-03-18T16:20:50.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.802+0000 7f8699000640 1 Processor -- start 2024-03-18T16:20:50.803 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.802+0000 7f8699000640 1 -- start start 2024-03-18T16:20:50.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.802+0000 7f8699000640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f86941038e0 0x7f8694199bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:50.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.802+0000 7f8699000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8694104ad0 0x7f869419a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:50.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.802+0000 7f8699000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8694105490 0x7f86941a1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:50.805 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.802+0000 7f8699000640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f869410f5b0 con 0x7f8694105490 2024-03-18T16:20:50.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.802+0000 7f8699000640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f869410f430 con 0x7f8694104ad0 2024-03-18T16:20:50.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.802+0000 7f8699000640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f869410f730 con 0x7f86941038e0 2024-03-18T16:20:50.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.802+0000 7f8693400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f86941038e0 0x7f8694199bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:50.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.802+0000 7f8692a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8694104ad0 0x7f869419a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:50.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.803+0000 7f8693400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f86941038e0 0x7f8694199bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:47764/0 (socket says 172.21.15.39:47764) 2024-03-18T16:20:50.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.803+0000 7f8692a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8694104ad0 0x7f869419a110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:39150/0 (socket says 172.21.15.39:39150) 2024-03-18T16:20:50.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.803+0000 7f8693400640 1 -- 172.21.15.39:0/2356417186 learned_addr learned my addr 172.21.15.39:0/2356417186 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:20:50.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.803+0000 7f8693e00640 1 --2- 172.21.15.39:0/2356417186 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8694105490 0x7f86941a1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:50.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.803+0000 7f8693400640 1 -- 172.21.15.39:0/2356417186 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8694104ad0 msgr2=0x7f869419a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:50.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.803+0000 7f8693400640 1 --2- 172.21.15.39:0/2356417186 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8694104ad0 0x7f869419a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:50.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.803+0000 7f8693400640 1 -- 172.21.15.39:0/2356417186 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8694105490 msgr2=0x7f86941a1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:50.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.803+0000 7f8693400640 1 --2- 172.21.15.39:0/2356417186 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8694105490 0x7f86941a1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:50.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.803+0000 7f8693400640 1 -- 172.21.15.39:0/2356417186 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86941a17f0 con 0x7f86941038e0 2024-03-18T16:20:50.806 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.803+0000 7f8692a00640 1 --2- 172.21.15.39:0/2356417186 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8694104ad0 0x7f869419a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:20:50.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.803+0000 7f8693400640 1 --2- 172.21.15.39:0/2356417186 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f86941038e0 0x7f8694199bd0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f8684000c00 tx=0x7f868402c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:50.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.804+0000 7f868be00640 1 -- 172.21.15.39:0/2356417186 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8684043020 con 0x7f86941038e0 2024-03-18T16:20:50.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.804+0000 7f868be00640 1 -- 172.21.15.39:0/2356417186 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f86840054e0 con 0x7f86941038e0 2024-03-18T16:20:50.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.804+0000 7f868be00640 1 -- 172.21.15.39:0/2356417186 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8684003c40 con 0x7f86941038e0 2024-03-18T16:20:50.807 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.804+0000 7f8699000640 1 -- 172.21.15.39:0/2356417186 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86941a1a80 con 0x7f86941038e0 2024-03-18T16:20:50.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.804+0000 7f8699000640 1 -- 172.21.15.39:0/2356417186 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86941a1f80 con 0x7f86941038e0 2024-03-18T16:20:50.808 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.805+0000 7f8699000640 1 -- 172.21.15.39:0/2356417186 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f869410ac00 con 0x7f86941038e0 2024-03-18T16:20:50.810 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.809+0000 7f868be00640 1 -- 172.21.15.39:0/2356417186 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f8684046020 con 0x7f86941038e0 2024-03-18T16:20:50.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.810+0000 7f868be00640 1 --2- 172.21.15.39:0/2356417186 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f86780a2800 0x7f86780a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:50.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.811+0000 7f868be00640 1 -- 172.21.15.39:0/2356417186 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f86840f1200 con 0x7f86941038e0 2024-03-18T16:20:50.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.811+0000 7f8692a00640 1 --2- 172.21.15.39:0/2356417186 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f86780a2800 0x7f86780a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:50.813 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.812+0000 7f8692a00640 1 --2- 172.21.15.39:0/2356417186 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f86780a2800 0x7f86780a4cc0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f869419b0e0 tx=0x7f8680005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:50.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:50.815+0000 7f868be00640 1 -- 172.21.15.39:0/2356417186 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f868404d020 con 0x7f86941038e0 2024-03-18T16:20:51.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.019+0000 7f8699000640 1 -- 172.21.15.39:0/2356417186 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f869402d340 con 0x7f86941038e0 2024-03-18T16:20:51.021 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.019+0000 7f868be00640 1 -- 172.21.15.39:0/2356417186 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7f86840bdeb0 con 0x7f86941038e0 2024-03-18T16:20:51.021 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2024-03-18T16:20:51.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.022+0000 7f8699000640 1 -- 172.21.15.39:0/2356417186 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f86780a2800 msgr2=0x7f86780a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:51.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.022+0000 7f8699000640 1 --2- 172.21.15.39:0/2356417186 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f86780a2800 0x7f86780a4cc0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f869419b0e0 tx=0x7f8680005f90 comp rx=0 tx=0).stop 2024-03-18T16:20:51.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.022+0000 7f8699000640 1 -- 172.21.15.39:0/2356417186 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f86941038e0 msgr2=0x7f8694199bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:51.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.022+0000 7f8699000640 1 --2- 172.21.15.39:0/2356417186 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f86941038e0 0x7f8694199bd0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f8684000c00 tx=0x7f868402c650 comp rx=0 tx=0).stop 2024-03-18T16:20:51.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.023+0000 7f8699000640 1 -- 172.21.15.39:0/2356417186 shutdown_connections 2024-03-18T16:20:51.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.023+0000 7f8699000640 1 --2- 172.21.15.39:0/2356417186 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8694104ad0 0x7f869419a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:51.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.023+0000 7f8699000640 1 --2- 172.21.15.39:0/2356417186 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f86780a2800 0x7f86780a4cc0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:51.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.023+0000 7f8699000640 1 --2- 172.21.15.39:0/2356417186 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8694105490 0x7f86941a1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:51.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.023+0000 7f8699000640 1 --2- 172.21.15.39:0/2356417186 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f86941038e0 0x7f8694199bd0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:51.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.023+0000 7f8699000640 1 -- 172.21.15.39:0/2356417186 >> 172.21.15.39:0/2356417186 conn(0x7f86940fee50 msgr2=0x7f8694107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:51.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.023+0000 7f8699000640 1 -- 172.21.15.39:0/2356417186 shutdown_connections 2024-03-18T16:20:51.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.023+0000 7f8699000640 1 -- 172.21.15.39:0/2356417186 wait complete. 2024-03-18T16:20:51.050 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T16:20:51.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.298+0000 7fbeb3e00640 1 -- 172.21.15.39:0/892930546 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbeb40741c0 msgr2=0x7fbeb4074620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:51.300 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.298+0000 7fbeb3e00640 1 --2- 172.21.15.39:0/892930546 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbeb40741c0 0x7fbeb4074620 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fbe9c009900 tx=0x7fbe9c02bcd0 comp rx=0 tx=0).stop 2024-03-18T16:20:51.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.299+0000 7fbeb3e00640 1 -- 172.21.15.39:0/892930546 shutdown_connections 2024-03-18T16:20:51.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.299+0000 7fbeb3e00640 1 --2- 172.21.15.39:0/892930546 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbeb40741c0 0x7fbeb4074620 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:51.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.299+0000 7fbeb3e00640 1 --2- 172.21.15.39:0/892930546 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbeb4109590 0x7fbeb410ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:51.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.299+0000 7fbeb3e00640 1 --2- 172.21.15.39:0/892930546 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbeb40757a0 0x7fbeb4073bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:51.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.299+0000 7fbeb3e00640 1 -- 172.21.15.39:0/892930546 >> 172.21.15.39:0/892930546 conn(0x7fbeb40fcc50 msgr2=0x7fbeb40ff090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:51.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.299+0000 7fbeb3e00640 1 -- 172.21.15.39:0/892930546 shutdown_connections 2024-03-18T16:20:51.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.299+0000 7fbeb3e00640 1 -- 172.21.15.39:0/892930546 wait complete. 2024-03-18T16:20:51.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.300+0000 7fbeb3e00640 1 Processor -- start 2024-03-18T16:20:51.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.300+0000 7fbeb3e00640 1 -- start start 2024-03-18T16:20:51.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.300+0000 7fbeb3e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbeb40741c0 0x7fbeb4199c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:51.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.300+0000 7fbeb3e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbeb40757a0 0x7fbeb419a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:51.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.300+0000 7fbeb3e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbeb4109590 0x7fbeb41a1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:51.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.300+0000 7fbeb3e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fbeb410f540 con 0x7fbeb4109590 2024-03-18T16:20:51.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.300+0000 7fbeb3e00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fbeb410f3c0 con 0x7fbeb40757a0 2024-03-18T16:20:51.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.300+0000 7fbeb3e00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fbeb410f6c0 con 0x7fbeb40741c0 2024-03-18T16:20:51.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.300+0000 7fbeb2a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbeb40741c0 0x7fbeb4199c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:51.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.300+0000 7fbeb2a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbeb40741c0 0x7fbeb4199c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:47782/0 (socket says 172.21.15.39:47782) 2024-03-18T16:20:51.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.300+0000 7fbeb2a00640 1 -- 172.21.15.39:0/2999833436 learned_addr learned my addr 172.21.15.39:0/2999833436 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:20:51.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.300+0000 7fbeabe00640 1 --2- 172.21.15.39:0/2999833436 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbeb40757a0 0x7fbeb419a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:51.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.301+0000 7fbeb3400640 1 --2- 172.21.15.39:0/2999833436 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbeb4109590 0x7fbeb41a1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:51.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.301+0000 7fbeb2a00640 1 -- 172.21.15.39:0/2999833436 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbeb40757a0 msgr2=0x7fbeb419a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:51.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.301+0000 7fbeb2a00640 1 --2- 172.21.15.39:0/2999833436 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbeb40757a0 0x7fbeb419a150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:51.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.301+0000 7fbeb2a00640 1 -- 172.21.15.39:0/2999833436 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbeb4109590 msgr2=0x7fbeb41a1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:51.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.301+0000 7fbeb2a00640 1 --2- 172.21.15.39:0/2999833436 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbeb4109590 0x7fbeb41a1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:51.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.301+0000 7fbeb2a00640 1 -- 172.21.15.39:0/2999833436 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbeb41a17e0 con 0x7fbeb40741c0 2024-03-18T16:20:51.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.301+0000 7fbeb2a00640 1 --2- 172.21.15.39:0/2999833436 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbeb40741c0 0x7fbeb4199c10 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fbea400e9f0 tx=0x7fbea400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:51.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.301+0000 7fbeab400640 1 -- 172.21.15.39:0/2999833436 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbea400cd80 con 0x7fbeb40741c0 2024-03-18T16:20:51.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.301+0000 7fbeab400640 1 -- 172.21.15.39:0/2999833436 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbea4010910 con 0x7fbeb40741c0 2024-03-18T16:20:51.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.301+0000 7fbeb3e00640 1 -- 172.21.15.39:0/2999833436 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbeb41a1ad0 con 0x7fbeb40741c0 2024-03-18T16:20:51.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.302+0000 7fbeab400640 1 -- 172.21.15.39:0/2999833436 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbea4018990 con 0x7fbeb40741c0 2024-03-18T16:20:51.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.302+0000 7fbeb3e00640 1 -- 172.21.15.39:0/2999833436 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbeb41a1ff0 con 0x7fbeb40741c0 2024-03-18T16:20:51.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.303+0000 7fbeb3e00640 1 -- 172.21.15.39:0/2999833436 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbeb4108350 con 0x7fbeb40741c0 2024-03-18T16:20:51.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.304+0000 7fbeab400640 1 -- 172.21.15.39:0/2999833436 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fbea4010ac0 con 0x7fbeb40741c0 2024-03-18T16:20:51.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.306+0000 7fbeab400640 1 --2- 172.21.15.39:0/2999833436 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fbe980a2800 0x7fbe980a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:51.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.306+0000 7fbeab400640 1 -- 172.21.15.39:0/2999833436 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fbea40cf180 con 0x7fbeb40741c0 2024-03-18T16:20:51.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.306+0000 7fbeabe00640 1 --2- 172.21.15.39:0/2999833436 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fbe980a2800 0x7fbe980a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:51.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.307+0000 7fbeabe00640 1 --2- 172.21.15.39:0/2999833436 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fbe980a2800 0x7fbe980a4cc0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fbe9c02c850 tx=0x7fbe9c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:51.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.310+0000 7fbeab400640 1 -- 172.21.15.39:0/2999833436 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fbea409b7e0 con 0x7fbeb40741c0 2024-03-18T16:20:51.339 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:51 smithi039 ceph-mon[31721]: pgmap v745: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 767 B/s wr, 11 op/s 2024-03-18T16:20:51.339 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:51 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/3860595456' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:20:51.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:51 smithi196 ceph-mon[36307]: pgmap v745: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 767 B/s wr, 11 op/s 2024-03-18T16:20:51.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:51 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/3860595456' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:20:51.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:51 smithi194 ceph-mon[36200]: pgmap v745: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 767 B/s wr, 11 op/s 2024-03-18T16:20:51.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:51 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/3860595456' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:20:51.564 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.562+0000 7fbeb3e00640 1 -- 172.21.15.39:0/2999833436 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fbeb40623e0 con 0x7fbeb40741c0 2024-03-18T16:20:51.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.563+0000 7fbeab400640 1 -- 172.21.15.39:0/2999833436 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7fbea409f880 con 0x7fbeb40741c0 2024-03-18T16:20:51.600 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 32 2024-03-18T16:20:51.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.565+0000 7fbeb3e00640 1 -- 172.21.15.39:0/2999833436 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fbe980a2800 msgr2=0x7fbe980a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:51.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.565+0000 7fbeb3e00640 1 --2- 172.21.15.39:0/2999833436 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fbe980a2800 0x7fbe980a4cc0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fbe9c02c850 tx=0x7fbe9c000bc0 comp rx=0 tx=0).stop 2024-03-18T16:20:51.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.565+0000 7fbeb3e00640 1 -- 172.21.15.39:0/2999833436 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbeb40741c0 msgr2=0x7fbeb4199c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:51.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.565+0000 7fbeb3e00640 1 --2- 172.21.15.39:0/2999833436 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbeb40741c0 0x7fbeb4199c10 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fbea400e9f0 tx=0x7fbea400eeb0 comp rx=0 tx=0).stop 2024-03-18T16:20:51.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.565+0000 7fbeb3e00640 1 -- 172.21.15.39:0/2999833436 shutdown_connections 2024-03-18T16:20:51.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.565+0000 7fbeb3e00640 1 --2- 172.21.15.39:0/2999833436 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbeb40757a0 0x7fbeb419a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:51.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.565+0000 7fbeb3e00640 1 --2- 172.21.15.39:0/2999833436 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fbe980a2800 0x7fbe980a4cc0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:51.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.565+0000 7fbeb3e00640 1 --2- 172.21.15.39:0/2999833436 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fbeb4109590 0x7fbeb41a1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:51.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.565+0000 7fbeb3e00640 1 --2- 172.21.15.39:0/2999833436 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbeb40741c0 0x7fbeb4199c10 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:51.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.565+0000 7fbeb3e00640 1 -- 172.21.15.39:0/2999833436 >> 172.21.15.39:0/2999833436 conn(0x7fbeb40fcc50 msgr2=0x7fbeb4105520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:51.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.566+0000 7fbeb3e00640 1 -- 172.21.15.39:0/2999833436 shutdown_connections 2024-03-18T16:20:51.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.566+0000 7fbeb3e00640 1 -- 172.21.15.39:0/2999833436 wait complete. 2024-03-18T16:20:51.602 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T16:20:51.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.841+0000 7f8f5b200640 1 -- 172.21.15.39:0/681196125 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8f5c105490 msgr2=0x7f8f5c10bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:51.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.841+0000 7f8f5b200640 1 --2- 172.21.15.39:0/681196125 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8f5c105490 0x7f8f5c10bd20 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f8f54009ca0 tx=0x7f8f5402c0f0 comp rx=0 tx=0).stop 2024-03-18T16:20:51.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.842+0000 7f8f5b200640 1 -- 172.21.15.39:0/681196125 shutdown_connections 2024-03-18T16:20:51.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.842+0000 7f8f5b200640 1 --2- 172.21.15.39:0/681196125 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8f5c105490 0x7f8f5c10bd20 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:51.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.842+0000 7f8f5b200640 1 --2- 172.21.15.39:0/681196125 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8f5c104ad0 0x7f8f5c104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:51.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.842+0000 7f8f5b200640 1 --2- 172.21.15.39:0/681196125 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8f5c1038e0 0x7f8f5c103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:51.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.842+0000 7f8f5b200640 1 -- 172.21.15.39:0/681196125 >> 172.21.15.39:0/681196125 conn(0x7f8f5c0fee50 msgr2=0x7f8f5c101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:51.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.842+0000 7f8f5b200640 1 -- 172.21.15.39:0/681196125 shutdown_connections 2024-03-18T16:20:51.844 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.842+0000 7f8f5b200640 1 -- 172.21.15.39:0/681196125 wait complete. 2024-03-18T16:20:51.846 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.843+0000 7f8f5b200640 1 Processor -- start 2024-03-18T16:20:51.912 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.843+0000 7f8f5b200640 1 -- start start 2024-03-18T16:20:51.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.843+0000 7f8f5b200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8f5c1038e0 0x7f8f5c199b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:51.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.843+0000 7f8f5b200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8f5c104ad0 0x7f8f5c19a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:51.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.843+0000 7f8f5b200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8f5c105490 0x7f8f5c1a1130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:51.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.843+0000 7f8f5b200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f8f5c10f540 con 0x7f8f5c105490 2024-03-18T16:20:51.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.843+0000 7f8f5b200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f8f5c10f3c0 con 0x7f8f5c1038e0 2024-03-18T16:20:51.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.843+0000 7f8f5b200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f8f5c10f6c0 con 0x7f8f5c104ad0 2024-03-18T16:20:51.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.843+0000 7f8f59e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8f5c1038e0 0x7f8f5c199b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:51.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.843+0000 7f8f51400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8f5c104ad0 0x7f8f5c19a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:51.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.843+0000 7f8f59e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8f5c1038e0 0x7f8f5c199b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:39186/0 (socket says 172.21.15.39:39186) 2024-03-18T16:20:51.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.843+0000 7f8f59e00640 1 -- 172.21.15.39:0/1520816213 learned_addr learned my addr 172.21.15.39:0/1520816213 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:20:51.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.844+0000 7f8f5a800640 1 --2- 172.21.15.39:0/1520816213 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8f5c105490 0x7f8f5c1a1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:51.913 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.844+0000 7f8f59e00640 1 -- 172.21.15.39:0/1520816213 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8f5c104ad0 msgr2=0x7f8f5c19a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:51.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.844+0000 7f8f59e00640 1 --2- 172.21.15.39:0/1520816213 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8f5c104ad0 0x7f8f5c19a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:51.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.844+0000 7f8f59e00640 1 -- 172.21.15.39:0/1520816213 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8f5c105490 msgr2=0x7f8f5c1a1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:51.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.844+0000 7f8f59e00640 1 --2- 172.21.15.39:0/1520816213 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8f5c105490 0x7f8f5c1a1130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:51.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.844+0000 7f8f59e00640 1 -- 172.21.15.39:0/1520816213 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f5c1a17a0 con 0x7f8f5c1038e0 2024-03-18T16:20:51.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.844+0000 7f8f51400640 1 --2- 172.21.15.39:0/1520816213 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8f5c104ad0 0x7f8f5c19a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:20:51.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.844+0000 7f8f5a800640 1 --2- 172.21.15.39:0/1520816213 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8f5c105490 0x7f8f5c1a1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:20:51.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.844+0000 7f8f59e00640 1 --2- 172.21.15.39:0/1520816213 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8f5c1038e0 0x7f8f5c199b80 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f8f4c00e9c0 tx=0x7f8f4c00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:51.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.844+0000 7f8f53400640 1 -- 172.21.15.39:0/1520816213 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f4c00cd80 con 0x7f8f5c1038e0 2024-03-18T16:20:51.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.845+0000 7f8f5b200640 1 -- 172.21.15.39:0/1520816213 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f5c1a1a90 con 0x7f8f5c1038e0 2024-03-18T16:20:51.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.845+0000 7f8f53400640 1 -- 172.21.15.39:0/1520816213 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f4c0035f0 con 0x7f8f5c1038e0 2024-03-18T16:20:51.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.845+0000 7f8f53400640 1 -- 172.21.15.39:0/1520816213 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f4c010690 con 0x7f8f5c1038e0 2024-03-18T16:20:51.914 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.845+0000 7f8f5b200640 1 -- 172.21.15.39:0/1520816213 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f5c1a2030 con 0x7f8f5c1038e0 2024-03-18T16:20:51.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.846+0000 7f8f5b200640 1 -- 172.21.15.39:0/1520816213 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f5c10ac00 con 0x7f8f5c1038e0 2024-03-18T16:20:51.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.848+0000 7f8f53400640 1 -- 172.21.15.39:0/1520816213 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f8f4c0037a0 con 0x7f8f5c1038e0 2024-03-18T16:20:51.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.849+0000 7f8f53400640 1 --2- 172.21.15.39:0/1520816213 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f8f400a2880 0x7f8f400a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:51.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.849+0000 7f8f53400640 1 -- 172.21.15.39:0/1520816213 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f8f4c0cf230 con 0x7f8f5c1038e0 2024-03-18T16:20:51.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.850+0000 7f8f51400640 1 --2- 172.21.15.39:0/1520816213 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f8f400a2880 0x7f8f400a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:51.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.850+0000 7f8f51400640 1 --2- 172.21.15.39:0/1520816213 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f8f400a2880 0x7f8f400a4d40 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c19b090 tx=0x7f8f44005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:51.916 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:51.858+0000 7f8f53400640 1 -- 172.21.15.39:0/1520816213 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f8f4c09bee0 con 0x7f8f5c1038e0 2024-03-18T16:20:52.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:52.066+0000 7f8f5b200640 1 -- 172.21.15.39:0/1520816213 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f8f5c02d430 con 0x7f8f5c1038e0 2024-03-18T16:20:52.068 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:52.067+0000 7f8f53400640 1 -- 172.21.15.39:0/1520816213 <== mon.2 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7f8f4c09b7e0 con 0x7f8f5c1038e0 2024-03-18T16:20:52.069 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 64 2024-03-18T16:20:52.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:52.069+0000 7f8f5b200640 1 -- 172.21.15.39:0/1520816213 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f8f400a2880 msgr2=0x7f8f400a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:52.071 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:52.069+0000 7f8f5b200640 1 --2- 172.21.15.39:0/1520816213 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f8f400a2880 0x7f8f400a4d40 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c19b090 tx=0x7f8f44005f90 comp rx=0 tx=0).stop 2024-03-18T16:20:52.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:52.069+0000 7f8f5b200640 1 -- 172.21.15.39:0/1520816213 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8f5c1038e0 msgr2=0x7f8f5c199b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:52.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:52.069+0000 7f8f5b200640 1 --2- 172.21.15.39:0/1520816213 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8f5c1038e0 0x7f8f5c199b80 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f8f4c00e9c0 tx=0x7f8f4c00ee80 comp rx=0 tx=0).stop 2024-03-18T16:20:52.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:52.070+0000 7f8f5b200640 1 -- 172.21.15.39:0/1520816213 shutdown_connections 2024-03-18T16:20:52.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:52.070+0000 7f8f5b200640 1 --2- 172.21.15.39:0/1520816213 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f8f400a2880 0x7f8f400a4d40 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:52.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:52.070+0000 7f8f5b200640 1 --2- 172.21.15.39:0/1520816213 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8f5c105490 0x7f8f5c1a1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:52.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:52.070+0000 7f8f5b200640 1 --2- 172.21.15.39:0/1520816213 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8f5c104ad0 0x7f8f5c19a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:52.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:52.070+0000 7f8f5b200640 1 --2- 172.21.15.39:0/1520816213 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8f5c1038e0 0x7f8f5c199b80 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:52.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:52.070+0000 7f8f5b200640 1 -- 172.21.15.39:0/1520816213 >> 172.21.15.39:0/1520816213 conn(0x7f8f5c0fee50 msgr2=0x7f8f5c107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:52.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:52.070+0000 7f8f5b200640 1 -- 172.21.15.39:0/1520816213 shutdown_connections 2024-03-18T16:20:52.072 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:52.070+0000 7f8f5b200640 1 -- 172.21.15.39:0/1520816213 wait complete. 2024-03-18T16:20:52.098 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T16:20:52.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:52.331+0000 7f7c7dc00640 1 -- 172.21.15.39:0/2846891475 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7c781055b0 msgr2=0x7f7c7810be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:52.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:52.331+0000 7f7c7dc00640 1 --2- 172.21.15.39:0/2846891475 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7c781055b0 0x7f7c7810be40 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f7c70009ca0 tx=0x7f7c7002c100 comp rx=0 tx=0).stop 2024-03-18T16:20:52.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:52.331+0000 7f7c7dc00640 1 -- 172.21.15.39:0/2846891475 shutdown_connections 2024-03-18T16:20:52.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:52.331+0000 7f7c7dc00640 1 --2- 172.21.15.39:0/2846891475 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7c781055b0 0x7f7c7810be40 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:52.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:52.331+0000 7f7c7dc00640 1 --2- 172.21.15.39:0/2846891475 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7c78104bf0 0x7f7c78105070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:52.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:52.331+0000 7f7c7dc00640 1 --2- 172.21.15.39:0/2846891475 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7c78103a00 0x7f7c78103e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:52.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:52.331+0000 7f7c7dc00640 1 -- 172.21.15.39:0/2846891475 >> 172.21.15.39:0/2846891475 conn(0x7f7c780fef90 msgr2=0x7f7c781013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:52.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:52.331+0000 7f7c7dc00640 1 -- 172.21.15.39:0/2846891475 shutdown_connections 2024-03-18T16:20:52.334 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:52.331+0000 7f7c7dc00640 1 -- 172.21.15.39:0/2846891475 wait complete. 2024-03-18T16:20:52.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:52.332+0000 7f7c7dc00640 1 Processor -- start 2024-03-18T16:20:52.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:52.333+0000 7f7c7dc00640 1 -- start start 2024-03-18T16:20:52.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:52.333+0000 7f7c7dc00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7c78103a00 0x7f7c78199d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:52.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:52.333+0000 7f7c7dc00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7c78104bf0 0x7f7c7819a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:52.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:52.333+0000 7f7c7dc00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7c781055b0 0x7f7c781a1290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:52.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:52.333+0000 7f7c7dc00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f7c7810f660 con 0x7f7c78103a00 2024-03-18T16:20:52.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:52.333+0000 7f7c7dc00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f7c7810f4e0 con 0x7f7c78104bf0 2024-03-18T16:20:52.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:52.333+0000 7f7c7dc00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f7c7810f7e0 con 0x7f7c781055b0 2024-03-18T16:20:52.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:52.333+0000 7f7c6fe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7c78104bf0 0x7f7c7819a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:52.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:52.333+0000 7f7c6fe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7c78104bf0 0x7f7c7819a260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:39210/0 (socket says 172.21.15.39:39210) 2024-03-18T16:20:52.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:52.333+0000 7f7c6fe00640 1 -- 172.21.15.39:0/4053565556 learned_addr learned my addr 172.21.15.39:0/4053565556 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:20:52.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:52.333+0000 7f7c76a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7c78103a00 0x7f7c78199d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:52.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:52.333+0000 7f7c77400640 1 --2- 172.21.15.39:0/4053565556 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7c781055b0 0x7f7c781a1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:52.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:52.334+0000 7f7c6fe00640 1 -- 172.21.15.39:0/4053565556 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7c781055b0 msgr2=0x7f7c781a1290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:52.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:52.334+0000 7f7c6fe00640 1 --2- 172.21.15.39:0/4053565556 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7c781055b0 0x7f7c781a1290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:52.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:52.334+0000 7f7c6fe00640 1 -- 172.21.15.39:0/4053565556 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7c78103a00 msgr2=0x7f7c78199d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:52.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:52.334+0000 7f7c6fe00640 1 --2- 172.21.15.39:0/4053565556 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7c78103a00 0x7f7c78199d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:52.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:52.334+0000 7f7c6fe00640 1 -- 172.21.15.39:0/4053565556 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c781a1930 con 0x7f7c78104bf0 2024-03-18T16:20:52.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:52.334+0000 7f7c76a00640 1 --2- 172.21.15.39:0/4053565556 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7c78103a00 0x7f7c78199d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:20:52.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:52.334+0000 7f7c6fe00640 1 --2- 172.21.15.39:0/4053565556 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7c78104bf0 0x7f7c7819a260 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f7c6000e9f0 tx=0x7f7c6000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:52.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:52.334+0000 7f7c6f400640 1 -- 172.21.15.39:0/4053565556 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c6000cd80 con 0x7f7c78104bf0 2024-03-18T16:20:52.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:52.334+0000 7f7c6f400640 1 -- 172.21.15.39:0/4053565556 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c60010910 con 0x7f7c78104bf0 2024-03-18T16:20:52.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:52.334+0000 7f7c6f400640 1 -- 172.21.15.39:0/4053565556 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c60018990 con 0x7f7c78104bf0 2024-03-18T16:20:52.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:52.334+0000 7f7c7dc00640 1 -- 172.21.15.39:0/4053565556 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c781a1c20 con 0x7f7c78104bf0 2024-03-18T16:20:52.338 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:52.335+0000 7f7c7dc00640 1 -- 172.21.15.39:0/4053565556 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c781a21c0 con 0x7f7c78104bf0 2024-03-18T16:20:52.339 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:52 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/2356417186' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T16:20:52.339 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:52 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/2999833436' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T16:20:52.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:52.336+0000 7f7c7dc00640 1 -- 172.21.15.39:0/4053565556 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c7810acc0 con 0x7f7c78104bf0 2024-03-18T16:20:52.339 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:52.337+0000 7f7c6f400640 1 -- 172.21.15.39:0/4053565556 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f7c60010ac0 con 0x7f7c78104bf0 2024-03-18T16:20:52.340 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:52.339+0000 7f7c6f400640 1 --2- 172.21.15.39:0/4053565556 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f7c5c0a2880 0x7f7c5c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:52.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:52.339+0000 7f7c6f400640 1 -- 172.21.15.39:0/4053565556 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f7c600cf240 con 0x7f7c78104bf0 2024-03-18T16:20:52.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:52.339+0000 7f7c76a00640 1 --2- 172.21.15.39:0/4053565556 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f7c5c0a2880 0x7f7c5c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:52.341 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:52.340+0000 7f7c76a00640 1 --2- 172.21.15.39:0/4053565556 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f7c5c0a2880 0x7f7c5c0a4d40 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f7c68005fd0 tx=0x7f7c68005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:52.344 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:52.343+0000 7f7c6f400640 1 -- 172.21.15.39:0/4053565556 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7c6009bef0 con 0x7f7c78104bf0 2024-03-18T16:20:52.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:52 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/2356417186' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T16:20:52.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:52 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/2999833436' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T16:20:52.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:52 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/2356417186' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T16:20:52.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:52 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/2999833436' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T16:20:52.706 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:52.705+0000 7f7c7dc00640 1 -- 172.21.15.39:0/4053565556 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f7c780623e0 con 0x7f7c78104bf0 2024-03-18T16:20:52.707 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:52.705+0000 7f7c6f400640 1 -- 172.21.15.39:0/4053565556 <== mon.2 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7f7c6009b7f0 con 0x7f7c78104bf0 2024-03-18T16:20:52.708 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 32 2024-03-18T16:20:52.710 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:52.708+0000 7f7c7dc00640 1 -- 172.21.15.39:0/4053565556 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f7c5c0a2880 msgr2=0x7f7c5c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:52.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:52.708+0000 7f7c7dc00640 1 --2- 172.21.15.39:0/4053565556 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f7c5c0a2880 0x7f7c5c0a4d40 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f7c68005fd0 tx=0x7f7c68005850 comp rx=0 tx=0).stop 2024-03-18T16:20:52.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:52.708+0000 7f7c7dc00640 1 -- 172.21.15.39:0/4053565556 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7c78104bf0 msgr2=0x7f7c7819a260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:52.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:52.708+0000 7f7c7dc00640 1 --2- 172.21.15.39:0/4053565556 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7c78104bf0 0x7f7c7819a260 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f7c6000e9f0 tx=0x7f7c6000eeb0 comp rx=0 tx=0).stop 2024-03-18T16:20:52.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:52.708+0000 7f7c7dc00640 1 -- 172.21.15.39:0/4053565556 shutdown_connections 2024-03-18T16:20:52.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:52.708+0000 7f7c7dc00640 1 --2- 172.21.15.39:0/4053565556 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7c78104bf0 0x7f7c7819a260 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:52.711 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:52.708+0000 7f7c7dc00640 1 --2- 172.21.15.39:0/4053565556 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f7c5c0a2880 0x7f7c5c0a4d40 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:52.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:52.708+0000 7f7c7dc00640 1 --2- 172.21.15.39:0/4053565556 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7c781055b0 0x7f7c781a1290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:52.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:52.708+0000 7f7c7dc00640 1 --2- 172.21.15.39:0/4053565556 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7c78103a00 0x7f7c78199d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:52.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:52.708+0000 7f7c7dc00640 1 -- 172.21.15.39:0/4053565556 >> 172.21.15.39:0/4053565556 conn(0x7f7c780fef90 msgr2=0x7f7c78107e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:52.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:52.708+0000 7f7c7dc00640 1 -- 172.21.15.39:0/4053565556 shutdown_connections 2024-03-18T16:20:52.712 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:52.708+0000 7f7c7dc00640 1 -- 172.21.15.39:0/4053565556 wait complete. 2024-03-18T16:20:52.728 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-03-18T16:20:53.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:53.017+0000 7f2fb6200640 1 -- 172.21.15.39:0/600500362 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2fb0104a60 msgr2=0x7f2fb0106e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:53.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:53.017+0000 7f2fb6200640 1 --2- 172.21.15.39:0/600500362 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2fb0104a60 0x7f2fb0106e50 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f2f9c009900 tx=0x7f2f9c02bc30 comp rx=0 tx=0).stop 2024-03-18T16:20:53.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:53.017+0000 7f2fb6200640 1 -- 172.21.15.39:0/600500362 shutdown_connections 2024-03-18T16:20:53.019 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:53.017+0000 7f2fb6200640 1 --2- 172.21.15.39:0/600500362 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2fb0107390 0x7f2fb0109780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:53.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:53.017+0000 7f2fb6200640 1 --2- 172.21.15.39:0/600500362 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2fb0104a60 0x7f2fb0106e50 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:53.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:53.017+0000 7f2fb6200640 1 --2- 172.21.15.39:0/600500362 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2fb0102130 0x7f2fb0104520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:53.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:53.017+0000 7f2fb6200640 1 -- 172.21.15.39:0/600500362 >> 172.21.15.39:0/600500362 conn(0x7f2fb00fbac0 msgr2=0x7f2fb00fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:53.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:53.017+0000 7f2fb6200640 1 -- 172.21.15.39:0/600500362 shutdown_connections 2024-03-18T16:20:53.020 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:53.017+0000 7f2fb6200640 1 -- 172.21.15.39:0/600500362 wait complete. 2024-03-18T16:20:53.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:53.019+0000 7f2fb6200640 1 Processor -- start 2024-03-18T16:20:53.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:53.019+0000 7f2fb6200640 1 -- start start 2024-03-18T16:20:53.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:53.019+0000 7f2fb6200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2fb0102130 0x7f2fb0197950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:53.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:53.019+0000 7f2fb6200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2fb0104a60 0x7f2fb0197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:53.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:53.019+0000 7f2fb6200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2fb0107390 0x7f2fb019ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:53.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:53.019+0000 7f2fb6200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f2fb010d300 con 0x7f2fb0104a60 2024-03-18T16:20:53.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:53.019+0000 7f2fb6200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f2fb010d180 con 0x7f2fb0102130 2024-03-18T16:20:53.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:53.019+0000 7f2fb6200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f2fb010d480 con 0x7f2fb0107390 2024-03-18T16:20:53.022 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:53.020+0000 7f2fb4e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2fb0102130 0x7f2fb0197950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:53.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:53.020+0000 7f2fafe00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2fb0104a60 0x7f2fb0197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:53.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:53.020+0000 7f2fb4e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2fb0102130 0x7f2fb0197950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:39228/0 (socket says 172.21.15.39:39228) 2024-03-18T16:20:53.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:53.020+0000 7f2fb4e00640 1 -- 172.21.15.39:0/371867407 learned_addr learned my addr 172.21.15.39:0/371867407 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:20:53.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:53.020+0000 7f2fb5800640 1 --2- 172.21.15.39:0/371867407 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2fb0107390 0x7f2fb019ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:53.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:53.020+0000 7f2fafe00640 1 -- 172.21.15.39:0/371867407 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2fb0107390 msgr2=0x7f2fb019ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:53.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:53.020+0000 7f2fafe00640 1 --2- 172.21.15.39:0/371867407 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2fb0107390 0x7f2fb019ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:53.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:53.020+0000 7f2fafe00640 1 -- 172.21.15.39:0/371867407 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2fb0102130 msgr2=0x7f2fb0197950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:53.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:53.020+0000 7f2fafe00640 1 --2- 172.21.15.39:0/371867407 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2fb0102130 0x7f2fb0197950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:53.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:53.020+0000 7f2fafe00640 1 -- 172.21.15.39:0/371867407 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fb019f570 con 0x7f2fb0104a60 2024-03-18T16:20:53.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:53.020+0000 7f2fb4e00640 1 --2- 172.21.15.39:0/371867407 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2fb0102130 0x7f2fb0197950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:20:53.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:53.020+0000 7f2fb5800640 1 --2- 172.21.15.39:0/371867407 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2fb0107390 0x7f2fb019ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:20:53.023 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:53.020+0000 7f2fafe00640 1 --2- 172.21.15.39:0/371867407 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2fb0104a60 0x7f2fb0197e90 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f2f9c009230 tx=0x7f2f9c02c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:53.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:53.021+0000 7f2fad600640 1 -- 172.21.15.39:0/371867407 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f9c043020 con 0x7f2fb0104a60 2024-03-18T16:20:53.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:53.021+0000 7f2fb6200640 1 -- 172.21.15.39:0/371867407 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fb019f800 con 0x7f2fb0104a60 2024-03-18T16:20:53.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:53.021+0000 7f2fad600640 1 -- 172.21.15.39:0/371867407 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f9c0054e0 con 0x7f2fb0104a60 2024-03-18T16:20:53.024 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:53.021+0000 7f2fb6200640 1 -- 172.21.15.39:0/371867407 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fb019fd70 con 0x7f2fb0104a60 2024-03-18T16:20:53.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:53.021+0000 7f2fad600640 1 -- 172.21.15.39:0/371867407 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f9c003b00 con 0x7f2fb0104a60 2024-03-18T16:20:53.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:53.023+0000 7f2fa2a00640 1 -- 172.21.15.39:0/371867407 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f8c005290 con 0x7f2fb0104a60 2024-03-18T16:20:53.025 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:53.024+0000 7f2fad600640 1 -- 172.21.15.39:0/371867407 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f2f9c046020 con 0x7f2fb0104a60 2024-03-18T16:20:53.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:53.026+0000 7f2fad600640 1 --2- 172.21.15.39:0/371867407 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f2f980a2800 0x7f2f980a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:20:53.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:53.026+0000 7f2fad600640 1 -- 172.21.15.39:0/371867407 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f2f9c0f0db0 con 0x7f2fb0104a60 2024-03-18T16:20:53.028 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:53.026+0000 7f2fb4e00640 1 --2- 172.21.15.39:0/371867407 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f2f980a2800 0x7f2f980a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:20:53.029 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:53.027+0000 7f2fb4e00640 1 --2- 172.21.15.39:0/371867407 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f2f980a2800 0x7f2f980a4cc0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f2fa40095f0 tx=0x7f2fa4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:20:53.032 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:53.031+0000 7f2fad600640 1 -- 172.21.15.39:0/371867407 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f2f9c0f5070 con 0x7f2fb0104a60 2024-03-18T16:20:53.287 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:53.285+0000 7f2fa2a00640 1 -- 172.21.15.39:0/371867407 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7f2f8c005580 con 0x7f2fb0104a60 2024-03-18T16:20:53.318 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:53.316+0000 7f2fad600640 1 -- 172.21.15.39:0/371867407 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v18) ==== 81+0+739830 (secure 0 0 0) 0x7f2f9c04d020 con 0x7f2fb0104a60 2024-03-18T16:20:53.334 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:20:53.385 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-03-18T16:20:53.385 INFO:teuthology.orchestra.run.smithi039.stdout: "epoch": 18, 2024-03-18T16:20:53.385 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.385 INFO:teuthology.orchestra.run.smithi039.stdout: "active_gid": 14152, 2024-03-18T16:20:53.385 INFO:teuthology.orchestra.run.smithi039.stdout: "active_name": "x", 2024-03-18T16:20:53.385 INFO:teuthology.orchestra.run.smithi039.stdout: "active_addrs": { 2024-03-18T16:20:53.386 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-03-18T16:20:53.386 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:53.386 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-03-18T16:20:53.386 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6800", 2024-03-18T16:20:53.386 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 2749214286 2024-03-18T16:20:53.386 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.386 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:53.386 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v1", 2024-03-18T16:20:53.386 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:6801", 2024-03-18T16:20:53.386 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 2749214286 2024-03-18T16:20:53.386 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:53.386 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-03-18T16:20:53.386 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.386 INFO:teuthology.orchestra.run.smithi039.stdout: "active_addr": "172.21.15.39:6801/2749214286", 2024-03-18T16:20:53.387 INFO:teuthology.orchestra.run.smithi039.stdout: "active_change": "2024-03-18T15:58:45.340556+0000", 2024-03-18T16:20:53.387 INFO:teuthology.orchestra.run.smithi039.stdout: "active_mgr_features": 4540701547738038271, 2024-03-18T16:20:53.387 INFO:teuthology.orchestra.run.smithi039.stdout: "available": true, 2024-03-18T16:20:53.387 INFO:teuthology.orchestra.run.smithi039.stdout: "standbys": [ 2024-03-18T16:20:53.387 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:53.387 INFO:teuthology.orchestra.run.smithi039.stdout: "gid": 24109, 2024-03-18T16:20:53.387 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "z", 2024-03-18T16:20:53.387 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr_features": 4540701547738038271, 2024-03-18T16:20:53.387 INFO:teuthology.orchestra.run.smithi039.stdout: "available_modules": [ 2024-03-18T16:20:53.387 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:53.387 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "alerts", 2024-03-18T16:20:53.387 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:53.387 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:53.387 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:53.387 INFO:teuthology.orchestra.run.smithi039.stdout: "interval": { 2024-03-18T16:20:53.388 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "interval", 2024-03-18T16:20:53.388 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-03-18T16:20:53.388 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.388 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:53.388 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "60", 2024-03-18T16:20:53.388 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.388 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.388 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.388 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "How frequently to reexamine health status", 2024-03-18T16:20:53.388 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.388 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.388 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.388 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.388 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:53.389 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:53.389 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:53.389 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.389 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:53.389 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:53.389 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.389 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.389 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:53.389 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:53.389 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:53.389 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:53.389 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:53.389 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:53.389 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:53.389 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:53.389 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:53.390 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.390 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.390 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.390 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.390 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:53.390 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:53.390 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:53.390 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.390 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:53.390 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:53.390 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.390 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.390 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.390 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:53.391 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.391 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.391 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.391 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.391 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:53.391 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:53.391 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:53.391 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.391 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:53.391 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:53.391 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.391 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.391 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:53.391 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:53.391 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:53.392 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:53.392 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:53.392 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:53.392 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:53.392 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:53.392 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:53.392 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.392 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.392 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.392 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.392 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:53.392 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:53.392 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:53.392 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.392 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:53.393 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:53.393 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.393 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.393 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.393 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:53.393 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.393 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.393 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.393 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.393 INFO:teuthology.orchestra.run.smithi039.stdout: "smtp_destination": { 2024-03-18T16:20:53.393 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "smtp_destination", 2024-03-18T16:20:53.393 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:53.393 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.393 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:53.393 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:53.394 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.394 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.394 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.394 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Email address to send alerts to", 2024-03-18T16:20:53.394 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.394 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.394 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.394 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.394 INFO:teuthology.orchestra.run.smithi039.stdout: "smtp_from_name": { 2024-03-18T16:20:53.394 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "smtp_from_name", 2024-03-18T16:20:53.394 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:53.394 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.394 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:53.394 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "Ceph", 2024-03-18T16:20:53.394 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.395 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.395 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.395 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Email From: name", 2024-03-18T16:20:53.395 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.395 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.395 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.395 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.395 INFO:teuthology.orchestra.run.smithi039.stdout: "smtp_host": { 2024-03-18T16:20:53.395 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "smtp_host", 2024-03-18T16:20:53.395 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:53.395 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.395 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:53.395 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:53.395 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.395 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.396 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.396 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "SMTP server", 2024-03-18T16:20:53.396 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.396 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.396 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.396 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.396 INFO:teuthology.orchestra.run.smithi039.stdout: "smtp_password": { 2024-03-18T16:20:53.396 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "smtp_password", 2024-03-18T16:20:53.396 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:53.396 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.397 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:53.397 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:53.397 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.397 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.397 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.397 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Password to authenticate with", 2024-03-18T16:20:53.397 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.397 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.397 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.398 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.398 INFO:teuthology.orchestra.run.smithi039.stdout: "smtp_port": { 2024-03-18T16:20:53.398 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "smtp_port", 2024-03-18T16:20:53.398 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:53.398 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.398 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:53.398 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "465", 2024-03-18T16:20:53.398 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.398 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.398 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.398 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "SMTP port", 2024-03-18T16:20:53.398 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.398 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.398 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.398 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.399 INFO:teuthology.orchestra.run.smithi039.stdout: "smtp_sender": { 2024-03-18T16:20:53.399 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "smtp_sender", 2024-03-18T16:20:53.399 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:53.399 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.399 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:53.399 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:53.399 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.399 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.399 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.399 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "SMTP envelope sender", 2024-03-18T16:20:53.399 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.399 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.400 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.400 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.400 INFO:teuthology.orchestra.run.smithi039.stdout: "smtp_ssl": { 2024-03-18T16:20:53.400 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "smtp_ssl", 2024-03-18T16:20:53.400 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:53.400 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.400 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:53.400 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:53.400 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.400 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.400 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.400 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-18T16:20:53.400 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.400 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.400 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.401 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.401 INFO:teuthology.orchestra.run.smithi039.stdout: "smtp_user": { 2024-03-18T16:20:53.401 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "smtp_user", 2024-03-18T16:20:53.401 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:53.401 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.401 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:53.401 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:53.401 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.401 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.401 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.401 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "User to authenticate as", 2024-03-18T16:20:53.401 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.401 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.401 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.401 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:53.401 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:53.402 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.402 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:53.402 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "balancer", 2024-03-18T16:20:53.402 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:53.402 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:53.402 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:53.402 INFO:teuthology.orchestra.run.smithi039.stdout: "active": { 2024-03-18T16:20:53.402 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "active", 2024-03-18T16:20:53.402 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:53.402 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.402 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:53.402 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:53.402 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.402 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.402 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.403 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "automatically balance PGs across cluster", 2024-03-18T16:20:53.403 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.403 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.403 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.403 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.403 INFO:teuthology.orchestra.run.smithi039.stdout: "begin_time": { 2024-03-18T16:20:53.403 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "begin_time", 2024-03-18T16:20:53.403 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:53.403 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.403 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:53.403 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0000", 2024-03-18T16:20:53.403 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.403 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.403 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.403 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "beginning time of day to automatically balance", 2024-03-18T16:20:53.404 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-18T16:20:53.404 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.404 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.404 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.404 INFO:teuthology.orchestra.run.smithi039.stdout: "begin_weekday": { 2024-03-18T16:20:53.404 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "begin_weekday", 2024-03-18T16:20:53.404 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "uint", 2024-03-18T16:20:53.404 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.404 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:53.404 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-03-18T16:20:53.404 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "0", 2024-03-18T16:20:53.404 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "6", 2024-03-18T16:20:53.404 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.404 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-18T16:20:53.404 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-18T16:20:53.405 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.405 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.405 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.405 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_compat_max_iterations": { 2024-03-18T16:20:53.405 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "crush_compat_max_iterations", 2024-03-18T16:20:53.405 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "uint", 2024-03-18T16:20:53.405 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.405 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:53.405 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "25", 2024-03-18T16:20:53.405 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "1", 2024-03-18T16:20:53.405 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "250", 2024-03-18T16:20:53.405 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.405 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-18T16:20:53.405 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.405 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.406 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.406 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.406 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_compat_metrics": { 2024-03-18T16:20:53.406 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "crush_compat_metrics", 2024-03-18T16:20:53.406 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:53.406 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.406 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:53.406 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "pgs,objects,bytes", 2024-03-18T16:20:53.406 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.406 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.406 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.406 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-18T16:20:53.406 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-18T16:20:53.406 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.407 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.407 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.407 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_compat_step": { 2024-03-18T16:20:53.407 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "crush_compat_step", 2024-03-18T16:20:53.407 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "float", 2024-03-18T16:20:53.407 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.407 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:53.407 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0.5", 2024-03-18T16:20:53.407 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "0.001", 2024-03-18T16:20:53.407 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "0.999", 2024-03-18T16:20:53.407 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.407 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "aggressiveness of optimization", 2024-03-18T16:20:53.407 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-18T16:20:53.407 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.407 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.407 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.408 INFO:teuthology.orchestra.run.smithi039.stdout: "end_time": { 2024-03-18T16:20:53.408 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "end_time", 2024-03-18T16:20:53.408 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:53.408 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.408 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:53.408 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "2359", 2024-03-18T16:20:53.408 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.408 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.408 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.408 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "ending time of day to automatically balance", 2024-03-18T16:20:53.408 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-18T16:20:53.408 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.408 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.443 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.443 INFO:teuthology.orchestra.run.smithi039.stdout: "end_weekday": { 2024-03-18T16:20:53.444 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "end_weekday", 2024-03-18T16:20:53.444 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "uint", 2024-03-18T16:20:53.444 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.444 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:53.444 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-03-18T16:20:53.444 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "0", 2024-03-18T16:20:53.444 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "6", 2024-03-18T16:20:53.444 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.444 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-18T16:20:53.444 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-18T16:20:53.444 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.444 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.444 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.444 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:53.445 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:53.445 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:53.445 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.445 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:53.445 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:53.445 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.445 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.445 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:53.445 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:53.445 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:53.445 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:53.445 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:53.445 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:53.445 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:53.445 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:53.446 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:53.446 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.446 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.446 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.446 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.446 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:53.446 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:53.446 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:53.446 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.446 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:53.446 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:53.446 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.446 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.447 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.447 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:53.447 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.447 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.447 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.447 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.447 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:53.447 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:53.447 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:53.447 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.447 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:53.447 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:53.447 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.447 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.447 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:53.448 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:53.448 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:53.448 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:53.448 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:53.448 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:53.448 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:53.448 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:53.448 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:53.448 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.448 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.448 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.448 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.448 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:53.448 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:53.448 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:53.449 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.449 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:53.449 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:53.449 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.449 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.449 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.449 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:53.449 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.449 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.449 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.449 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.449 INFO:teuthology.orchestra.run.smithi039.stdout: "min_score": { 2024-03-18T16:20:53.449 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "min_score", 2024-03-18T16:20:53.449 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "float", 2024-03-18T16:20:53.449 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.450 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:53.450 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-03-18T16:20:53.450 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.450 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.450 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.450 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-18T16:20:53.450 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.450 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.450 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.450 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.450 INFO:teuthology.orchestra.run.smithi039.stdout: "mode": { 2024-03-18T16:20:53.450 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "mode", 2024-03-18T16:20:53.450 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:53.450 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.451 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:53.451 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "upmap", 2024-03-18T16:20:53.451 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.451 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.451 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:53.451 INFO:teuthology.orchestra.run.smithi039.stdout: "crush-compat", 2024-03-18T16:20:53.451 INFO:teuthology.orchestra.run.smithi039.stdout: "none", 2024-03-18T16:20:53.451 INFO:teuthology.orchestra.run.smithi039.stdout: "read", 2024-03-18T16:20:53.451 INFO:teuthology.orchestra.run.smithi039.stdout: "upmap", 2024-03-18T16:20:53.451 INFO:teuthology.orchestra.run.smithi039.stdout: "upmap-read" 2024-03-18T16:20:53.451 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:53.451 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Balancer mode", 2024-03-18T16:20:53.451 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.451 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.451 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.452 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.452 INFO:teuthology.orchestra.run.smithi039.stdout: "pool_ids": { 2024-03-18T16:20:53.452 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "pool_ids", 2024-03-18T16:20:53.452 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:53.452 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.452 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:53.452 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:53.452 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.452 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.452 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.452 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-18T16:20:53.452 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.452 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.452 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.452 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.452 INFO:teuthology.orchestra.run.smithi039.stdout: "sleep_interval": { 2024-03-18T16:20:53.453 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sleep_interval", 2024-03-18T16:20:53.453 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-03-18T16:20:53.453 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.453 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:53.453 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "60", 2024-03-18T16:20:53.453 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.453 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.453 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.453 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-18T16:20:53.453 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.453 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.453 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.453 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.453 INFO:teuthology.orchestra.run.smithi039.stdout: "upmap_max_deviation": { 2024-03-18T16:20:53.577 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "upmap_max_deviation", 2024-03-18T16:20:53.578 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:53.578 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.578 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:53.578 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "5", 2024-03-18T16:20:53.578 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "1", 2024-03-18T16:20:53.578 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.578 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.578 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-18T16:20:53.578 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-18T16:20:53.578 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.578 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.578 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.578 INFO:teuthology.orchestra.run.smithi039.stdout: "upmap_max_optimizations": { 2024-03-18T16:20:53.578 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "upmap_max_optimizations", 2024-03-18T16:20:53.579 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "uint", 2024-03-18T16:20:53.579 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.579 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:53.579 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "10", 2024-03-18T16:20:53.579 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.579 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.579 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.579 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-18T16:20:53.579 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.579 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.579 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.579 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:53.579 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:53.579 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.579 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:53.580 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "cephadm", 2024-03-18T16:20:53.580 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:53.580 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:53.580 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:53.580 INFO:teuthology.orchestra.run.smithi039.stdout: "agent_down_multiplier": { 2024-03-18T16:20:53.580 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "agent_down_multiplier", 2024-03-18T16:20:53.580 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "float", 2024-03-18T16:20:53.580 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.580 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.580 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "3.0", 2024-03-18T16:20:53.580 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.580 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.580 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.580 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-18T16:20:53.580 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.581 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.581 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.581 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.581 INFO:teuthology.orchestra.run.smithi039.stdout: "agent_refresh_rate": { 2024-03-18T16:20:53.581 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "agent_refresh_rate", 2024-03-18T16:20:53.581 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-03-18T16:20:53.581 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.581 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.581 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "20", 2024-03-18T16:20:53.581 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.581 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.581 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.581 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-18T16:20:53.581 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.581 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.581 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.582 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.582 INFO:teuthology.orchestra.run.smithi039.stdout: "agent_starting_port": { 2024-03-18T16:20:53.582 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "agent_starting_port", 2024-03-18T16:20:53.582 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:53.582 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.582 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.582 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "4721", 2024-03-18T16:20:53.582 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.582 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.582 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.582 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-18T16:20:53.582 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.583 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.583 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.583 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.583 INFO:teuthology.orchestra.run.smithi039.stdout: "allow_ptrace": { 2024-03-18T16:20:53.583 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "allow_ptrace", 2024-03-18T16:20:53.583 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:53.583 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.583 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.583 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:53.583 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.583 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.583 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.583 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-18T16:20:53.583 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-18T16:20:53.583 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.584 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.584 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.584 INFO:teuthology.orchestra.run.smithi039.stdout: "autotune_interval": { 2024-03-18T16:20:53.584 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "autotune_interval", 2024-03-18T16:20:53.584 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-03-18T16:20:53.584 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.584 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.584 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "600", 2024-03-18T16:20:53.584 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.584 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.584 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.584 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-18T16:20:53.584 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.584 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.585 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.585 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.585 INFO:teuthology.orchestra.run.smithi039.stdout: "autotune_memory_target_ratio": { 2024-03-18T16:20:53.585 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "autotune_memory_target_ratio", 2024-03-18T16:20:53.585 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "float", 2024-03-18T16:20:53.585 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.585 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.585 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0.7", 2024-03-18T16:20:53.585 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.585 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.585 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.585 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-18T16:20:53.585 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.585 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.585 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.586 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.586 INFO:teuthology.orchestra.run.smithi039.stdout: "cephadm_log_destination": { 2024-03-18T16:20:53.586 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "cephadm_log_destination", 2024-03-18T16:20:53.586 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:53.586 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.586 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.586 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:53.586 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.586 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.586 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:53.586 INFO:teuthology.orchestra.run.smithi039.stdout: "file", 2024-03-18T16:20:53.586 INFO:teuthology.orchestra.run.smithi039.stdout: "file,syslog", 2024-03-18T16:20:53.586 INFO:teuthology.orchestra.run.smithi039.stdout: "syslog" 2024-03-18T16:20:53.586 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:53.586 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-18T16:20:53.587 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.587 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.587 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.587 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.587 INFO:teuthology.orchestra.run.smithi039.stdout: "cgroups_split": { 2024-03-18T16:20:53.587 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "cgroups_split", 2024-03-18T16:20:53.587 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:53.587 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.587 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.587 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:53.587 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.587 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.587 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.587 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-18T16:20:53.587 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.588 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.588 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.588 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.588 INFO:teuthology.orchestra.run.smithi039.stdout: "config_checks_enabled": { 2024-03-18T16:20:53.588 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "config_checks_enabled", 2024-03-18T16:20:53.588 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:53.588 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.588 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.588 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:53.588 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.588 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.588 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.588 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-18T16:20:53.588 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.588 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.589 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.589 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.589 INFO:teuthology.orchestra.run.smithi039.stdout: "config_dashboard": { 2024-03-18T16:20:53.589 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "config_dashboard", 2024-03-18T16:20:53.589 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:53.589 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.589 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.589 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:53.589 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.650 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.650 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.650 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-18T16:20:53.651 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.651 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.651 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.651 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.651 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_alertmanager": { 2024-03-18T16:20:53.651 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_alertmanager", 2024-03-18T16:20:53.651 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:53.651 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.651 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.651 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-18T16:20:53.651 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.651 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.651 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.651 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Prometheus container image", 2024-03-18T16:20:53.651 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.652 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.652 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.652 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.652 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_base": { 2024-03-18T16:20:53.699 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_base", 2024-03-18T16:20:53.699 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:53.700 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.700 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:53.700 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-18T16:20:53.700 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.700 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.700 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.700 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Container image name, without the tag", 2024-03-18T16:20:53.700 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.700 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.701 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.701 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.701 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_elasticsearch": { 2024-03-18T16:20:53.701 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_elasticsearch", 2024-03-18T16:20:53.701 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:53.701 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.701 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.701 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-18T16:20:53.701 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.701 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.701 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.701 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "elasticsearch container image", 2024-03-18T16:20:53.701 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.701 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.702 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.702 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.702 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_grafana": { 2024-03-18T16:20:53.711 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_grafana", 2024-03-18T16:20:53.711 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:53.711 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.711 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.711 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-18T16:20:53.711 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.711 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.711 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.711 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Prometheus container image", 2024-03-18T16:20:53.711 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.711 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.712 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.712 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.712 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_haproxy": { 2024-03-18T16:20:53.712 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_haproxy", 2024-03-18T16:20:53.712 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:53.712 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.712 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.712 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-18T16:20:53.712 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.712 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.712 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.713 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "HAproxy container image", 2024-03-18T16:20:53.713 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.713 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.713 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.713 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.713 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_jaeger_agent": { 2024-03-18T16:20:53.713 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_jaeger_agent", 2024-03-18T16:20:53.713 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:53.713 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.713 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.713 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-18T16:20:53.713 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.713 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.713 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.714 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Jaeger agent container image", 2024-03-18T16:20:53.714 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.714 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.714 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.714 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.714 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_jaeger_collector": { 2024-03-18T16:20:53.714 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_jaeger_collector", 2024-03-18T16:20:53.714 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:53.714 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.714 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.714 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-18T16:20:53.714 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.714 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.714 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.715 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Jaeger collector container image", 2024-03-18T16:20:53.715 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.715 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.715 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.715 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.715 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_jaeger_query": { 2024-03-18T16:20:53.715 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_jaeger_query", 2024-03-18T16:20:53.715 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:53.715 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.715 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.715 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-18T16:20:53.715 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.715 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.715 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.716 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Jaeger query container image", 2024-03-18T16:20:53.716 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.716 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.716 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.716 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.716 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_keepalived": { 2024-03-18T16:20:53.716 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_keepalived", 2024-03-18T16:20:53.716 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:53.716 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.716 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.716 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-18T16:20:53.716 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.716 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.716 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.717 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Keepalived container image", 2024-03-18T16:20:53.717 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.717 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.717 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.717 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.717 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_loki": { 2024-03-18T16:20:53.717 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_loki", 2024-03-18T16:20:53.717 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:53.717 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.717 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.717 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-18T16:20:53.717 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.717 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.717 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.717 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Loki container image", 2024-03-18T16:20:53.718 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.718 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.718 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.718 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.718 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_node_exporter": { 2024-03-18T16:20:53.718 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_node_exporter", 2024-03-18T16:20:53.718 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:53.718 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.718 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.718 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-18T16:20:53.718 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.718 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.718 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.719 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Prometheus container image", 2024-03-18T16:20:53.719 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.719 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.719 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.719 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.719 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_nvmeof": { 2024-03-18T16:20:53.719 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_nvmeof", 2024-03-18T16:20:53.719 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:53.719 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.719 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.719 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-18T16:20:53.719 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.719 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.719 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.720 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Nvme-of container image", 2024-03-18T16:20:53.720 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.720 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.720 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.720 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.720 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_prometheus": { 2024-03-18T16:20:53.720 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_prometheus", 2024-03-18T16:20:53.720 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:53.720 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.720 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.720 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-18T16:20:53.720 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.720 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.720 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.720 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Prometheus container image", 2024-03-18T16:20:53.721 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.721 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.721 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.721 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.721 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_promtail": { 2024-03-18T16:20:53.721 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_promtail", 2024-03-18T16:20:53.721 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:53.721 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.721 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.721 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-18T16:20:53.721 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.721 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.721 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.721 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Promtail container image", 2024-03-18T16:20:53.722 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.722 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.722 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.722 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.722 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_snmp_gateway": { 2024-03-18T16:20:53.722 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_snmp_gateway", 2024-03-18T16:20:53.722 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:53.722 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.722 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.722 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-18T16:20:53.722 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.722 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.722 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.723 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "SNMP Gateway container image", 2024-03-18T16:20:53.723 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.723 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.723 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.723 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.723 INFO:teuthology.orchestra.run.smithi039.stdout: "container_init": { 2024-03-18T16:20:53.723 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_init", 2024-03-18T16:20:53.723 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:53.723 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.723 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.723 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:53.723 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.723 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.723 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.724 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Run podman/docker with `--init`", 2024-03-18T16:20:53.724 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.724 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.724 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.724 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.724 INFO:teuthology.orchestra.run.smithi039.stdout: "daemon_cache_timeout": { 2024-03-18T16:20:53.724 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "daemon_cache_timeout", 2024-03-18T16:20:53.724 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-03-18T16:20:53.724 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.724 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.724 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "600", 2024-03-18T16:20:53.724 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.725 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.725 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.725 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-18T16:20:53.725 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.725 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.725 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.725 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.725 INFO:teuthology.orchestra.run.smithi039.stdout: "default_cephadm_command_timeout": { 2024-03-18T16:20:53.725 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "default_cephadm_command_timeout", 2024-03-18T16:20:53.725 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:53.725 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.725 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.725 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "900", 2024-03-18T16:20:53.725 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.726 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.726 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.726 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-18T16:20:53.726 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.726 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.726 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.726 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.726 INFO:teuthology.orchestra.run.smithi039.stdout: "default_registry": { 2024-03-18T16:20:53.726 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "default_registry", 2024-03-18T16:20:53.726 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:53.726 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.726 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.726 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "docker.io", 2024-03-18T16:20:53.726 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.727 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.727 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.727 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-18T16:20:53.727 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.727 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.727 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.727 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.727 INFO:teuthology.orchestra.run.smithi039.stdout: "device_cache_timeout": { 2024-03-18T16:20:53.727 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "device_cache_timeout", 2024-03-18T16:20:53.727 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-03-18T16:20:53.727 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.727 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.727 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "1800", 2024-03-18T16:20:53.727 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.728 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.737 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.737 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "seconds to cache device inventory", 2024-03-18T16:20:53.738 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.738 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.738 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.738 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.738 INFO:teuthology.orchestra.run.smithi039.stdout: "device_enhanced_scan": { 2024-03-18T16:20:53.738 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "device_enhanced_scan", 2024-03-18T16:20:53.738 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:53.738 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.738 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.738 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:53.738 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.738 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.738 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.738 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-18T16:20:53.739 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.739 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.739 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.739 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.739 INFO:teuthology.orchestra.run.smithi039.stdout: "facts_cache_timeout": { 2024-03-18T16:20:53.739 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "facts_cache_timeout", 2024-03-18T16:20:53.739 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-03-18T16:20:53.739 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.739 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.739 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "60", 2024-03-18T16:20:53.745 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.745 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.745 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.745 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "seconds to cache host facts data", 2024-03-18T16:20:53.745 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.745 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.745 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.745 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.745 INFO:teuthology.orchestra.run.smithi039.stdout: "host_check_interval": { 2024-03-18T16:20:53.745 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "host_check_interval", 2024-03-18T16:20:53.746 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-03-18T16:20:53.746 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.746 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.746 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "600", 2024-03-18T16:20:53.746 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.746 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.746 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.746 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "how frequently to perform a host check", 2024-03-18T16:20:53.746 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.746 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.746 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.746 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.746 INFO:teuthology.orchestra.run.smithi039.stdout: "hw_monitoring": { 2024-03-18T16:20:53.746 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "hw_monitoring", 2024-03-18T16:20:53.747 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:53.747 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.747 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.747 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:53.747 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.747 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.747 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.747 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-18T16:20:53.747 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.747 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.747 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.747 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.747 INFO:teuthology.orchestra.run.smithi039.stdout: "inventory_list_all": { 2024-03-18T16:20:53.747 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "inventory_list_all", 2024-03-18T16:20:53.748 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:53.748 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.748 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.748 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:53.748 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.748 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.748 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.748 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-18T16:20:53.748 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.748 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.748 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.748 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.748 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:53.748 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:53.748 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:53.749 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.749 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:53.749 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:53.749 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.749 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.749 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:53.749 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:53.749 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:53.749 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:53.749 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:53.749 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:53.749 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:53.749 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:53.749 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:53.749 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.750 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.750 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.750 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.750 INFO:teuthology.orchestra.run.smithi039.stdout: "log_refresh_metadata": { 2024-03-18T16:20:53.750 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_refresh_metadata", 2024-03-18T16:20:53.750 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:53.750 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.750 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.750 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:53.750 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.750 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.750 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.751 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-18T16:20:53.751 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.751 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.751 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.751 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.751 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:53.751 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:53.751 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:53.751 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.751 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.751 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:53.751 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.751 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.751 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.752 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-18T16:20:53.752 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.752 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.752 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.752 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.752 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:53.752 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:53.752 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:53.752 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.752 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:53.752 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:53.752 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.753 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.753 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:53.753 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:53.753 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:53.753 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:53.753 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:53.753 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:53.753 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:53.753 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:53.753 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:53.753 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.753 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.753 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.753 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.753 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:53.754 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:53.754 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:53.754 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.754 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:53.754 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:53.754 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.754 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.754 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.754 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:53.754 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.754 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.754 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.754 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.754 INFO:teuthology.orchestra.run.smithi039.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-18T16:20:53.755 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-18T16:20:53.755 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:53.755 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.755 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.755 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:53.755 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.755 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.755 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.755 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-18T16:20:53.755 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.755 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.755 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.755 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.756 INFO:teuthology.orchestra.run.smithi039.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-18T16:20:53.756 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-18T16:20:53.756 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:53.756 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.756 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.756 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "*", 2024-03-18T16:20:53.756 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.756 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.756 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.756 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-18T16:20:53.756 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.756 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.756 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.756 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.757 INFO:teuthology.orchestra.run.smithi039.stdout: "max_count_per_host": { 2024-03-18T16:20:53.757 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "max_count_per_host", 2024-03-18T16:20:53.757 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:53.757 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.757 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.757 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "10", 2024-03-18T16:20:53.757 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.757 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.757 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.757 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "max number of daemons per service per host", 2024-03-18T16:20:53.757 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.757 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.757 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.757 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.758 INFO:teuthology.orchestra.run.smithi039.stdout: "max_osd_draining_count": { 2024-03-18T16:20:53.758 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "max_osd_draining_count", 2024-03-18T16:20:53.758 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:53.758 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.758 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.758 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "10", 2024-03-18T16:20:53.758 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.758 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.758 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.758 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-18T16:20:53.758 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.758 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.758 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.759 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.759 INFO:teuthology.orchestra.run.smithi039.stdout: "migration_current": { 2024-03-18T16:20:53.759 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "migration_current", 2024-03-18T16:20:53.759 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:53.759 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.759 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.759 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:53.759 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.759 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.759 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.759 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "internal - do not modify", 2024-03-18T16:20:53.759 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.759 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.759 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.759 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.760 INFO:teuthology.orchestra.run.smithi039.stdout: "mode": { 2024-03-18T16:20:53.760 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "mode", 2024-03-18T16:20:53.760 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:53.760 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.760 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.760 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "root", 2024-03-18T16:20:53.760 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.760 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.760 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:53.760 INFO:teuthology.orchestra.run.smithi039.stdout: "cephadm-package", 2024-03-18T16:20:53.760 INFO:teuthology.orchestra.run.smithi039.stdout: "root" 2024-03-18T16:20:53.760 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:53.760 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "mode for remote execution of cephadm", 2024-03-18T16:20:53.760 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.761 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.761 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.761 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.761 INFO:teuthology.orchestra.run.smithi039.stdout: "oob_default_addr": { 2024-03-18T16:20:53.761 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "oob_default_addr", 2024-03-18T16:20:53.761 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:53.761 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.761 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.761 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "169.254.1.1", 2024-03-18T16:20:53.761 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.761 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.761 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.761 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-18T16:20:53.761 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.761 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.762 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.762 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.762 INFO:teuthology.orchestra.run.smithi039.stdout: "prometheus_alerts_path": { 2024-03-18T16:20:53.762 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "prometheus_alerts_path", 2024-03-18T16:20:53.762 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:53.762 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.762 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.762 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-18T16:20:53.762 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.762 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.762 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.762 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-18T16:20:53.762 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.762 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.762 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.763 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.763 INFO:teuthology.orchestra.run.smithi039.stdout: "registry_insecure": { 2024-03-18T16:20:53.763 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "registry_insecure", 2024-03-18T16:20:53.763 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:53.763 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.763 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.763 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:53.763 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.763 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.763 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.763 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-18T16:20:53.763 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.763 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.763 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.764 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.764 INFO:teuthology.orchestra.run.smithi039.stdout: "registry_password": { 2024-03-18T16:20:53.764 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "registry_password", 2024-03-18T16:20:53.764 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:53.764 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.764 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.764 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:53.764 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.764 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.764 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.764 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-18T16:20:53.764 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.764 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.764 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.764 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.765 INFO:teuthology.orchestra.run.smithi039.stdout: "registry_url": { 2024-03-18T16:20:53.765 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "registry_url", 2024-03-18T16:20:53.765 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:53.765 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.765 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.765 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:53.765 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.765 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.765 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.765 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-18T16:20:53.765 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.765 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.765 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.766 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.766 INFO:teuthology.orchestra.run.smithi039.stdout: "registry_username": { 2024-03-18T16:20:53.766 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "registry_username", 2024-03-18T16:20:53.766 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:53.766 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.766 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.766 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:53.766 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.766 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.766 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.766 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-18T16:20:53.766 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.766 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.766 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.766 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.767 INFO:teuthology.orchestra.run.smithi039.stdout: "secure_monitoring_stack": { 2024-03-18T16:20:53.767 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "secure_monitoring_stack", 2024-03-18T16:20:53.767 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:53.767 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.767 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.767 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:53.767 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.767 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.767 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.767 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-18T16:20:53.767 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.767 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.767 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.767 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.768 INFO:teuthology.orchestra.run.smithi039.stdout: "service_discovery_port": { 2024-03-18T16:20:53.768 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "service_discovery_port", 2024-03-18T16:20:53.768 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:53.768 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.768 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.768 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "8765", 2024-03-18T16:20:53.768 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.768 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.768 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.768 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "cephadm service discovery port", 2024-03-18T16:20:53.768 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.768 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.768 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.768 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.768 INFO:teuthology.orchestra.run.smithi039.stdout: "ssh_config_file": { 2024-03-18T16:20:53.769 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "ssh_config_file", 2024-03-18T16:20:53.769 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:53.769 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.769 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.769 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:53.813 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.813 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.813 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.813 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-18T16:20:53.822 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.822 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.822 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.822 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.822 INFO:teuthology.orchestra.run.smithi039.stdout: "use_agent": { 2024-03-18T16:20:53.822 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "use_agent", 2024-03-18T16:20:53.822 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:53.822 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.823 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.823 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:53.823 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.823 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.823 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.823 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-18T16:20:53.823 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.823 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.823 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.823 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.823 INFO:teuthology.orchestra.run.smithi039.stdout: "use_repo_digest": { 2024-03-18T16:20:53.823 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "use_repo_digest", 2024-03-18T16:20:53.823 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:53.824 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.824 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.824 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:53.824 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.824 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.824 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.824 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-18T16:20:53.824 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.824 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.824 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.824 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.824 INFO:teuthology.orchestra.run.smithi039.stdout: "warn_on_failed_host_check": { 2024-03-18T16:20:53.824 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "warn_on_failed_host_check", 2024-03-18T16:20:53.824 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:53.824 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.825 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.825 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:53.825 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.825 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.825 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.825 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "raise a health warning if the host check fails", 2024-03-18T16:20:53.825 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.825 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.825 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.825 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.825 INFO:teuthology.orchestra.run.smithi039.stdout: "warn_on_stray_daemons": { 2024-03-18T16:20:53.825 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "warn_on_stray_daemons", 2024-03-18T16:20:53.825 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:53.825 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.826 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.826 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:53.826 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.826 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.826 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.826 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-18T16:20:53.826 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.826 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.826 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.826 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.826 INFO:teuthology.orchestra.run.smithi039.stdout: "warn_on_stray_hosts": { 2024-03-18T16:20:53.826 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "warn_on_stray_hosts", 2024-03-18T16:20:53.826 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:53.827 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.827 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.827 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:53.827 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.827 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.827 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.827 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-18T16:20:53.827 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.827 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.827 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.827 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:53.827 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:53.827 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.827 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:53.828 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "crash", 2024-03-18T16:20:53.828 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:53.828 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:53.828 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:53.828 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:53.828 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:53.828 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:53.828 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.828 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:53.828 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:53.828 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.828 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.828 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:53.828 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:53.828 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:53.829 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:53.829 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:53.829 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:53.829 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:53.829 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:53.829 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:53.829 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.829 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.829 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.829 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.829 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:53.829 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:53.829 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:53.829 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.829 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:53.830 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:53.830 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.830 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.830 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.830 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:53.830 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.830 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.830 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.830 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.830 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:53.830 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:53.830 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:53.831 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.831 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:53.831 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:53.831 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.831 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.831 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:53.831 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:53.831 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:53.831 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:53.831 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:53.831 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:53.831 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:53.831 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:53.831 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:53.831 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.832 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.832 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.832 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.832 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:53.832 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:53.832 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:53.832 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.832 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:53.832 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:53.832 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.832 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.832 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.832 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:53.832 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.833 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.833 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.833 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.833 INFO:teuthology.orchestra.run.smithi039.stdout: "retain_interval": { 2024-03-18T16:20:53.833 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "retain_interval", 2024-03-18T16:20:53.833 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-03-18T16:20:53.833 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.833 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:53.833 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "31536000", 2024-03-18T16:20:53.833 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.833 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.833 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.833 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-18T16:20:53.833 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.833 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.834 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.834 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.834 INFO:teuthology.orchestra.run.smithi039.stdout: "warn_recent_interval": { 2024-03-18T16:20:53.834 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "warn_recent_interval", 2024-03-18T16:20:53.834 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-03-18T16:20:53.834 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.834 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:53.834 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "1209600", 2024-03-18T16:20:53.834 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.834 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.834 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.834 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-18T16:20:53.834 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.834 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.835 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.835 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:53.835 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:53.835 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.835 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:53.835 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "dashboard", 2024-03-18T16:20:53.835 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:53.835 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:53.835 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:53.835 INFO:teuthology.orchestra.run.smithi039.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-18T16:20:53.835 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-18T16:20:53.835 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:53.835 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.835 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.836 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "10", 2024-03-18T16:20:53.836 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.836 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.836 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.836 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:53.836 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.836 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.836 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.836 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.836 INFO:teuthology.orchestra.run.smithi039.stdout: "ALERTMANAGER_API_HOST": { 2024-03-18T16:20:53.836 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-18T16:20:53.836 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:53.836 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.836 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.836 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:53.836 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.837 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.837 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.837 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:53.837 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.837 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.837 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.837 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.837 INFO:teuthology.orchestra.run.smithi039.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-18T16:20:53.837 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-18T16:20:53.837 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:53.837 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.837 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.837 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:53.837 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.837 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.838 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.838 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:53.838 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.838 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.838 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.838 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.838 INFO:teuthology.orchestra.run.smithi039.stdout: "AUDIT_API_ENABLED": { 2024-03-18T16:20:53.838 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "AUDIT_API_ENABLED", 2024-03-18T16:20:53.838 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:53.838 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.838 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.838 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:53.838 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.838 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.838 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.839 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:53.839 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.839 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.839 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.839 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.839 INFO:teuthology.orchestra.run.smithi039.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-18T16:20:53.839 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-18T16:20:53.839 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:53.839 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.839 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.839 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:53.839 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.839 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.839 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.839 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:53.840 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.840 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.840 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.840 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.840 INFO:teuthology.orchestra.run.smithi039.stdout: "ENABLE_BROWSABLE_API": { 2024-03-18T16:20:53.840 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-18T16:20:53.840 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:53.840 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.840 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.840 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:53.840 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.840 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.840 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.840 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:53.840 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.841 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.841 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.841 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.841 INFO:teuthology.orchestra.run.smithi039.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-18T16:20:53.841 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-18T16:20:53.841 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:53.841 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.841 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.841 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:53.841 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.841 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.841 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.841 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:53.841 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.841 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.842 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.842 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.842 INFO:teuthology.orchestra.run.smithi039.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-18T16:20:53.842 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-18T16:20:53.842 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:53.842 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.842 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.842 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:53.842 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.842 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.842 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.842 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:53.842 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.842 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.842 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.843 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.843 INFO:teuthology.orchestra.run.smithi039.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-18T16:20:53.843 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-18T16:20:53.843 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:53.843 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.843 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.843 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:53.843 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.843 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.843 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.843 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:53.843 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.843 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.844 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.844 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.844 INFO:teuthology.orchestra.run.smithi039.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-18T16:20:53.844 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-18T16:20:53.844 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:53.844 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.844 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.844 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:53.844 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.844 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.844 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.844 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:53.844 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.844 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.844 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.845 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.845 INFO:teuthology.orchestra.run.smithi039.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-18T16:20:53.845 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-18T16:20:53.845 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:53.845 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.845 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.845 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:53.845 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.845 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.845 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.845 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:53.845 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.845 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.845 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.845 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.846 INFO:teuthology.orchestra.run.smithi039.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-18T16:20:53.846 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-18T16:20:53.846 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:53.846 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.846 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.846 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:53.846 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.846 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.846 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.846 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:53.846 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.846 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.846 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.846 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.847 INFO:teuthology.orchestra.run.smithi039.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-18T16:20:53.847 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-18T16:20:53.847 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:53.847 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.847 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.847 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:53.847 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.847 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.847 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.847 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:53.847 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.847 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.847 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.847 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.848 INFO:teuthology.orchestra.run.smithi039.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-18T16:20:53.848 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-18T16:20:53.848 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:53.848 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.848 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.848 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:53.848 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.848 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.848 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.848 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:53.848 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.848 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.848 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.848 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.848 INFO:teuthology.orchestra.run.smithi039.stdout: "GRAFANA_API_PASSWORD": { 2024-03-18T16:20:53.848 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-18T16:20:53.849 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:53.849 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.849 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.849 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "admin", 2024-03-18T16:20:53.849 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.849 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.849 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.849 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:53.849 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.849 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.849 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.849 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.849 INFO:teuthology.orchestra.run.smithi039.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-18T16:20:53.849 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-18T16:20:53.850 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:53.850 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.850 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.850 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:53.850 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.850 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.850 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.850 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:53.850 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.850 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.850 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.850 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.850 INFO:teuthology.orchestra.run.smithi039.stdout: "GRAFANA_API_URL": { 2024-03-18T16:20:53.851 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "GRAFANA_API_URL", 2024-03-18T16:20:53.851 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:53.851 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.851 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.851 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:53.851 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.851 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.851 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.851 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:53.880 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.933 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.933 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.933 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.933 INFO:teuthology.orchestra.run.smithi039.stdout: "GRAFANA_API_USERNAME": { 2024-03-18T16:20:53.933 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-18T16:20:53.933 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:53.933 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.933 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.933 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "admin", 2024-03-18T16:20:53.933 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.934 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.934 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.934 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:53.934 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.934 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.934 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.934 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.934 INFO:teuthology.orchestra.run.smithi039.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-18T16:20:53.934 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-18T16:20:53.934 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:53.934 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.934 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.934 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:53.934 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.934 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.935 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.935 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:53.935 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.935 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.935 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.935 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.935 INFO:teuthology.orchestra.run.smithi039.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-18T16:20:53.935 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-18T16:20:53.935 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:53.935 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.935 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.935 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:53.935 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.935 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.935 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.936 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:53.936 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.936 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.936 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.936 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.936 INFO:teuthology.orchestra.run.smithi039.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-18T16:20:53.936 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-18T16:20:53.936 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:53.936 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.936 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.936 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:53.936 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.936 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.936 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.936 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:53.937 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.937 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.937 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.937 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.937 INFO:teuthology.orchestra.run.smithi039.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-18T16:20:53.937 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-18T16:20:53.937 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:53.937 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.937 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.937 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:53.937 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.937 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.938 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.938 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:53.938 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.938 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.938 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.938 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.938 INFO:teuthology.orchestra.run.smithi039.stdout: "MULTICLUSTER_CONFIG": { 2024-03-18T16:20:53.938 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-18T16:20:53.938 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:53.938 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.938 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.938 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "{}", 2024-03-18T16:20:53.938 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.939 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.939 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.939 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:53.939 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.939 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.939 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.939 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.939 INFO:teuthology.orchestra.run.smithi039.stdout: "PROMETHEUS_API_HOST": { 2024-03-18T16:20:53.939 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-18T16:20:53.939 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:53.939 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.939 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.939 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:53.939 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.939 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.940 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.940 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:53.940 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.940 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.940 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.940 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.940 INFO:teuthology.orchestra.run.smithi039.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-18T16:20:53.940 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-18T16:20:53.940 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:53.940 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.940 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.940 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:53.940 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.940 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.941 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.941 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:53.941 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.941 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.941 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.941 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.941 INFO:teuthology.orchestra.run.smithi039.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-18T16:20:53.941 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-18T16:20:53.941 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:53.941 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.941 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.941 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:53.941 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.941 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.941 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.942 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:53.942 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.942 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.942 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.942 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.942 INFO:teuthology.orchestra.run.smithi039.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-18T16:20:53.942 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-18T16:20:53.942 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:53.942 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.942 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.942 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:53.942 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.942 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.942 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.943 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:53.943 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.943 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.943 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.943 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.943 INFO:teuthology.orchestra.run.smithi039.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-18T16:20:53.943 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-18T16:20:53.943 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:53.943 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.943 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.943 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:53.943 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.943 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.943 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.943 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:53.944 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.944 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.944 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.944 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.944 INFO:teuthology.orchestra.run.smithi039.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-18T16:20:53.944 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-18T16:20:53.944 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:53.944 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.944 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.944 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:53.944 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.944 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.944 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.944 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:53.944 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.945 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.945 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.945 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.945 INFO:teuthology.orchestra.run.smithi039.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-18T16:20:53.945 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-18T16:20:53.945 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:53.945 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.945 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.945 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:53.945 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.945 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.945 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.945 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:53.945 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.945 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.946 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.946 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.946 INFO:teuthology.orchestra.run.smithi039.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-18T16:20:53.946 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-18T16:20:53.946 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:53.946 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.946 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.946 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:53.946 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.946 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.946 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.946 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:53.946 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.946 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.946 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.947 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.947 INFO:teuthology.orchestra.run.smithi039.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-18T16:20:53.947 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-18T16:20:53.947 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:53.947 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.947 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.947 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:53.947 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.947 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.947 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.947 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:53.947 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.947 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.947 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.947 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.948 INFO:teuthology.orchestra.run.smithi039.stdout: "PWD_POLICY_ENABLED": { 2024-03-18T16:20:53.948 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-18T16:20:53.948 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:53.948 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.948 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.948 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:53.948 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.948 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.948 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.948 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:53.948 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.948 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.948 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.948 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.948 INFO:teuthology.orchestra.run.smithi039.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-18T16:20:53.949 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-18T16:20:53.949 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:53.949 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.949 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.949 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-18T16:20:53.949 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.949 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.949 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.949 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:53.949 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.949 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.949 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.949 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.949 INFO:teuthology.orchestra.run.smithi039.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-18T16:20:53.949 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-18T16:20:53.950 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:53.950 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.950 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.950 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "10", 2024-03-18T16:20:53.950 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.950 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.950 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.950 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:53.950 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.950 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.950 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.950 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.950 INFO:teuthology.orchestra.run.smithi039.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-18T16:20:53.950 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-18T16:20:53.951 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:53.951 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.951 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.951 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "8", 2024-03-18T16:20:53.951 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.951 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.951 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.951 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:53.951 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.951 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.951 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.951 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.951 INFO:teuthology.orchestra.run.smithi039.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-18T16:20:53.951 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-18T16:20:53.951 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:53.951 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.952 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.952 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "45", 2024-03-18T16:20:53.952 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.952 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.952 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.985 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:53.985 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.985 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.985 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.985 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.986 INFO:teuthology.orchestra.run.smithi039.stdout: "RGW_API_ACCESS_KEY": { 2024-03-18T16:20:53.986 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-18T16:20:53.986 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:53.986 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.986 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.986 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:53.986 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.986 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.986 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.986 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:53.986 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.986 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.986 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.986 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.987 INFO:teuthology.orchestra.run.smithi039.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-18T16:20:53.987 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-18T16:20:53.987 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:53.987 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.987 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.987 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "admin", 2024-03-18T16:20:53.987 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.987 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.987 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.987 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:53.987 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.987 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.987 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.987 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.987 INFO:teuthology.orchestra.run.smithi039.stdout: "RGW_API_SECRET_KEY": { 2024-03-18T16:20:53.988 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-18T16:20:53.988 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:53.988 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.988 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.988 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:53.988 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.988 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.988 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.988 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:53.988 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.988 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.988 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.996 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.996 INFO:teuthology.orchestra.run.smithi039.stdout: "RGW_API_SSL_VERIFY": { 2024-03-18T16:20:53.997 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-18T16:20:53.997 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:53.997 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.997 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.997 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:53.997 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.997 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.997 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.997 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:53.997 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.997 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.997 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.997 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.997 INFO:teuthology.orchestra.run.smithi039.stdout: "UNSAFE_TLS_v1_2": { 2024-03-18T16:20:53.998 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-18T16:20:53.998 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:53.998 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.998 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.998 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:53.998 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.998 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.998 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.998 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:53.998 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.998 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.998 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.998 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.998 INFO:teuthology.orchestra.run.smithi039.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-18T16:20:53.999 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-18T16:20:53.999 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:53.999 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:53.999 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:53.999 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-03-18T16:20:53.999 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:53.999 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:53.999 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:53.999 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:53.999 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:53.999 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:53.999 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:53.999 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:53.999 INFO:teuthology.orchestra.run.smithi039.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-18T16:20:54.000 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-18T16:20:54.000 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:54.000 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.000 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.000 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "10", 2024-03-18T16:20:54.000 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.000 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.000 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.000 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.000 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.000 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.000 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.000 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.000 INFO:teuthology.orchestra.run.smithi039.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-18T16:20:54.000 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-18T16:20:54.001 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:54.001 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.059 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.060 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "5", 2024-03-18T16:20:54.060 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.060 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.060 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.060 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.060 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.060 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.060 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.060 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.060 INFO:teuthology.orchestra.run.smithi039.stdout: "cross_origin_url": { 2024-03-18T16:20:54.060 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "cross_origin_url", 2024-03-18T16:20:54.060 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.061 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.061 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.061 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.061 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.061 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.061 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.061 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.061 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.061 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.061 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.061 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.061 INFO:teuthology.orchestra.run.smithi039.stdout: "crt_file": { 2024-03-18T16:20:54.061 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "crt_file", 2024-03-18T16:20:54.061 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.061 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.062 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.062 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.062 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.062 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.062 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.062 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.062 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.062 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.062 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.062 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.062 INFO:teuthology.orchestra.run.smithi039.stdout: "debug": { 2024-03-18T16:20:54.062 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "debug", 2024-03-18T16:20:54.062 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.062 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.063 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.063 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.063 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.063 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.063 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.063 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Enable/disable debug options", 2024-03-18T16:20:54.063 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.063 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.063 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.063 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.063 INFO:teuthology.orchestra.run.smithi039.stdout: "jwt_token_ttl": { 2024-03-18T16:20:54.064 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "jwt_token_ttl", 2024-03-18T16:20:54.064 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:54.064 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.064 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.064 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "28800", 2024-03-18T16:20:54.064 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.064 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.064 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.064 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.064 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.064 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.064 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.064 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.064 INFO:teuthology.orchestra.run.smithi039.stdout: "key_file": { 2024-03-18T16:20:54.065 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "key_file", 2024-03-18T16:20:54.065 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.065 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.065 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.065 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.065 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.065 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.065 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.065 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.065 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.065 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.065 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.065 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.065 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:54.066 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:54.066 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.066 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.066 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.066 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.066 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.066 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.073 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:54.081 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:54.109 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:54.109 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:54.109 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:54.109 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:54.109 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:54.109 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:54.109 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.109 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.109 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.109 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.109 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.109 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:54.109 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:54.110 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.110 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.110 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.110 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.110 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.110 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.110 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.110 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.110 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.110 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.110 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.110 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.110 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:54.110 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:54.111 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.111 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.111 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.111 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:54.111 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.111 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.111 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:54.111 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:54.111 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:54.111 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:54.111 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:54.111 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:54.111 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:54.111 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:54.111 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.112 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.112 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.112 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.112 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.112 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:54.112 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:54.112 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.112 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.112 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.112 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.112 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.112 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.112 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.112 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.112 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.113 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.113 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.113 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.113 INFO:teuthology.orchestra.run.smithi039.stdout: "motd": { 2024-03-18T16:20:54.113 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "motd", 2024-03-18T16:20:54.113 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.113 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.113 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.113 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.113 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.113 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.113 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.113 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "The message of the day", 2024-03-18T16:20:54.113 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.113 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.114 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.114 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.114 INFO:teuthology.orchestra.run.smithi039.stdout: "redirect_resolve_ip_addr": { 2024-03-18T16:20:54.114 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "redirect_resolve_ip_addr", 2024-03-18T16:20:54.114 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.114 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.114 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.114 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.114 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.114 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.114 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.114 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.114 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.114 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.114 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.115 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.115 INFO:teuthology.orchestra.run.smithi039.stdout: "server_addr": { 2024-03-18T16:20:54.115 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "server_addr", 2024-03-18T16:20:54.115 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.115 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.115 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.115 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "::", 2024-03-18T16:20:54.115 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.115 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.115 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.115 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.115 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.115 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.115 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.115 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.116 INFO:teuthology.orchestra.run.smithi039.stdout: "server_port": { 2024-03-18T16:20:54.116 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "server_port", 2024-03-18T16:20:54.116 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:54.116 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.116 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.116 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "8080", 2024-03-18T16:20:54.116 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.116 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.116 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.116 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.116 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.116 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.116 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.116 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.116 INFO:teuthology.orchestra.run.smithi039.stdout: "ssl": { 2024-03-18T16:20:54.116 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "ssl", 2024-03-18T16:20:54.117 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.117 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.117 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.117 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:54.117 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.117 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.117 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.117 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.117 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.117 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.117 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.117 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.117 INFO:teuthology.orchestra.run.smithi039.stdout: "ssl_server_port": { 2024-03-18T16:20:54.117 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "ssl_server_port", 2024-03-18T16:20:54.117 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:54.118 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.118 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.118 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "8443", 2024-03-18T16:20:54.118 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.118 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.118 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.118 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.118 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.118 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.118 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.118 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.118 INFO:teuthology.orchestra.run.smithi039.stdout: "standby_behaviour": { 2024-03-18T16:20:54.118 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "standby_behaviour", 2024-03-18T16:20:54.119 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.119 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.119 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.119 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "redirect", 2024-03-18T16:20:54.119 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.119 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.119 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:54.119 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:54.119 INFO:teuthology.orchestra.run.smithi039.stdout: "redirect" 2024-03-18T16:20:54.119 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:54.119 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.119 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.119 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.119 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.119 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.119 INFO:teuthology.orchestra.run.smithi039.stdout: "standby_error_status_code": { 2024-03-18T16:20:54.120 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "standby_error_status_code", 2024-03-18T16:20:54.120 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:54.120 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.120 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.120 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "500", 2024-03-18T16:20:54.120 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "400", 2024-03-18T16:20:54.120 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "599", 2024-03-18T16:20:54.120 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.120 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.120 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.120 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.120 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.120 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.120 INFO:teuthology.orchestra.run.smithi039.stdout: "url_prefix": { 2024-03-18T16:20:54.121 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "url_prefix", 2024-03-18T16:20:54.121 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.162 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.162 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.162 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.162 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.162 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.162 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.162 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.162 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.162 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.163 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.163 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:54.163 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:54.163 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.163 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:54.163 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "devicehealth", 2024-03-18T16:20:54.163 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:54.163 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:54.163 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:54.163 INFO:teuthology.orchestra.run.smithi039.stdout: "enable_monitoring": { 2024-03-18T16:20:54.163 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "enable_monitoring", 2024-03-18T16:20:54.163 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.163 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.163 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.164 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:54.164 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.164 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.164 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.164 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "monitor device health metrics", 2024-03-18T16:20:54.164 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.164 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.164 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.164 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.164 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:54.164 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:54.164 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.164 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.164 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.165 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.165 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.165 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.165 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:54.165 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:54.165 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:54.165 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:54.165 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:54.165 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:54.165 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:54.165 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:54.165 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.165 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.165 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.165 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.165 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.166 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:54.166 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:54.166 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.166 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.166 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.166 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.166 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.166 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.166 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.166 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.166 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.166 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.166 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.166 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.166 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:54.167 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:54.167 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.167 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.167 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.167 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:54.167 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.167 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.167 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:54.167 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:54.167 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:54.167 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:54.167 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:54.167 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:54.167 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:54.167 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:54.168 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.168 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.168 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.168 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.168 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.168 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:54.168 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:54.168 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.168 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.168 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.168 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.168 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.168 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.168 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.168 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.168 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.169 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.169 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.169 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.169 INFO:teuthology.orchestra.run.smithi039.stdout: "mark_out_threshold": { 2024-03-18T16:20:54.169 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "mark_out_threshold", 2024-03-18T16:20:54.169 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-03-18T16:20:54.169 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.169 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.169 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "2419200", 2024-03-18T16:20:54.169 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.169 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.169 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.170 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-18T16:20:54.170 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.170 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.170 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.170 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.170 INFO:teuthology.orchestra.run.smithi039.stdout: "pool_name": { 2024-03-18T16:20:54.170 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "pool_name", 2024-03-18T16:20:54.170 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.170 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.170 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.170 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "device_health_metrics", 2024-03-18T16:20:54.170 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.170 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.170 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.170 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-18T16:20:54.171 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.171 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.171 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.171 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.171 INFO:teuthology.orchestra.run.smithi039.stdout: "retention_period": { 2024-03-18T16:20:54.171 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "retention_period", 2024-03-18T16:20:54.171 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-03-18T16:20:54.171 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.171 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.171 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "15552000", 2024-03-18T16:20:54.171 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.171 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.171 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.171 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "how long to retain device health metrics", 2024-03-18T16:20:54.171 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.172 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.172 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.172 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.172 INFO:teuthology.orchestra.run.smithi039.stdout: "scrape_frequency": { 2024-03-18T16:20:54.172 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "scrape_frequency", 2024-03-18T16:20:54.172 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-03-18T16:20:54.172 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.172 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.172 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "86400", 2024-03-18T16:20:54.172 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.172 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.172 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.172 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-18T16:20:54.173 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.173 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.173 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.173 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.173 INFO:teuthology.orchestra.run.smithi039.stdout: "self_heal": { 2024-03-18T16:20:54.173 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "self_heal", 2024-03-18T16:20:54.173 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.173 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.173 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.173 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:54.173 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.173 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.173 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.173 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-18T16:20:54.173 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.174 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.174 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.174 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.174 INFO:teuthology.orchestra.run.smithi039.stdout: "sleep_interval": { 2024-03-18T16:20:54.174 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sleep_interval", 2024-03-18T16:20:54.174 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-03-18T16:20:54.174 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.174 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.174 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "600", 2024-03-18T16:20:54.174 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.174 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.174 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.174 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "how frequently to wake up and check device health", 2024-03-18T16:20:54.175 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.175 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.175 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.175 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.175 INFO:teuthology.orchestra.run.smithi039.stdout: "warn_threshold": { 2024-03-18T16:20:54.175 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "warn_threshold", 2024-03-18T16:20:54.175 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-03-18T16:20:54.175 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.175 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.175 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "7257600", 2024-03-18T16:20:54.175 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.175 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.175 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.175 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-18T16:20:54.175 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.176 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.176 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.176 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:54.176 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:54.176 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.176 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:54.176 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "diskprediction_local", 2024-03-18T16:20:54.176 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:54.176 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:54.176 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:54.176 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:54.176 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:54.176 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.176 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.177 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.177 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.177 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.177 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.177 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:54.177 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:54.177 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:54.177 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:54.177 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:54.177 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:54.177 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:54.177 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:54.177 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.177 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.177 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.178 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.178 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.178 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:54.178 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:54.178 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.178 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.178 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.178 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.178 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.178 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.178 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.178 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.178 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.178 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.178 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.179 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.179 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:54.179 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:54.179 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.179 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.179 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.179 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:54.179 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.179 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.179 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:54.179 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:54.179 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:54.179 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:54.179 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:54.179 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:54.180 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:54.180 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:54.180 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.180 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.180 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.180 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.180 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.180 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:54.180 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:54.180 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.180 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.180 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.180 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.180 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.180 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.212 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.213 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.214 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.214 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.214 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.214 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.214 INFO:teuthology.orchestra.run.smithi039.stdout: "predict_interval": { 2024-03-18T16:20:54.214 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "predict_interval", 2024-03-18T16:20:54.214 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.214 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.214 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.214 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "86400", 2024-03-18T16:20:54.214 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.214 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.214 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.215 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.215 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.215 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.215 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.215 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.215 INFO:teuthology.orchestra.run.smithi039.stdout: "predictor_model": { 2024-03-18T16:20:54.215 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "predictor_model", 2024-03-18T16:20:54.215 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.215 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.215 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.215 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "prophetstor", 2024-03-18T16:20:54.215 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.216 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.216 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.216 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.216 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.216 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.216 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.216 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.216 INFO:teuthology.orchestra.run.smithi039.stdout: "sleep_interval": { 2024-03-18T16:20:54.216 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sleep_interval", 2024-03-18T16:20:54.216 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.216 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.216 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.216 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "600", 2024-03-18T16:20:54.216 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.216 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.217 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.217 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.217 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.217 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.217 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.217 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:54.217 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:54.217 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.217 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:54.217 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "influx", 2024-03-18T16:20:54.217 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": false, 2024-03-18T16:20:54.217 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "influxdb python module not found", 2024-03-18T16:20:54.217 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:54.217 INFO:teuthology.orchestra.run.smithi039.stdout: "batch_size": { 2024-03-18T16:20:54.217 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "batch_size", 2024-03-18T16:20:54.218 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:54.218 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.218 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.218 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "5000", 2024-03-18T16:20:54.218 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.218 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.218 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.218 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-18T16:20:54.218 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.218 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.218 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.218 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.218 INFO:teuthology.orchestra.run.smithi039.stdout: "database": { 2024-03-18T16:20:54.218 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "database", 2024-03-18T16:20:54.218 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.219 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.219 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.219 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "ceph", 2024-03-18T16:20:54.219 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.219 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.219 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.219 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-18T16:20:54.219 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.219 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.219 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.219 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.219 INFO:teuthology.orchestra.run.smithi039.stdout: "hostname": { 2024-03-18T16:20:54.220 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "hostname", 2024-03-18T16:20:54.220 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.220 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.220 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.220 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.220 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.220 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.220 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.220 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "InfluxDB server hostname", 2024-03-18T16:20:54.220 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.220 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.220 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.220 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.220 INFO:teuthology.orchestra.run.smithi039.stdout: "interval": { 2024-03-18T16:20:54.220 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "interval", 2024-03-18T16:20:54.221 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-03-18T16:20:54.221 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.221 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.221 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "30", 2024-03-18T16:20:54.221 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "5", 2024-03-18T16:20:54.221 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.221 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.221 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-18T16:20:54.221 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.221 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.221 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.221 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.221 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:54.221 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:54.221 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.222 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.222 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.222 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.222 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.222 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.222 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:54.222 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:54.222 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:54.222 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:54.222 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:54.229 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:54.234 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:54.234 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:54.234 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.234 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.234 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.234 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.235 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.235 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:54.235 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:54.235 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.235 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.235 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.235 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.235 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.235 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.235 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.235 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.235 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.235 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.235 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.236 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.236 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:54.236 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:54.236 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.236 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.236 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.236 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:54.236 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.236 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.236 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:54.236 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:54.236 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:54.236 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:54.236 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:54.237 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:54.237 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:54.237 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:54.237 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.237 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.237 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.237 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.237 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.237 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:54.237 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:54.237 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.237 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.237 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.237 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.237 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.238 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.238 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.238 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.238 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.238 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.238 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.238 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.238 INFO:teuthology.orchestra.run.smithi039.stdout: "password": { 2024-03-18T16:20:54.238 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "password", 2024-03-18T16:20:54.238 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.238 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.238 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.238 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.238 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.239 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.239 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.239 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "password of InfluxDB server user", 2024-03-18T16:20:54.239 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.239 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.239 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.239 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.239 INFO:teuthology.orchestra.run.smithi039.stdout: "port": { 2024-03-18T16:20:54.239 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "port", 2024-03-18T16:20:54.239 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:54.239 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.239 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.239 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "8086", 2024-03-18T16:20:54.239 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.240 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.240 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.240 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "InfluxDB server port", 2024-03-18T16:20:54.240 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.240 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.240 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.240 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.240 INFO:teuthology.orchestra.run.smithi039.stdout: "ssl": { 2024-03-18T16:20:54.240 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "ssl", 2024-03-18T16:20:54.240 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.240 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.240 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.240 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "false", 2024-03-18T16:20:54.240 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.241 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.241 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.241 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-18T16:20:54.241 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.241 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.241 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.241 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.241 INFO:teuthology.orchestra.run.smithi039.stdout: "threads": { 2024-03-18T16:20:54.241 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "threads", 2024-03-18T16:20:54.241 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:54.241 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.241 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.241 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "5", 2024-03-18T16:20:54.241 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "1", 2024-03-18T16:20:54.241 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "32", 2024-03-18T16:20:54.242 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.242 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-18T16:20:54.242 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.242 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.242 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.242 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.242 INFO:teuthology.orchestra.run.smithi039.stdout: "username": { 2024-03-18T16:20:54.242 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "username", 2024-03-18T16:20:54.242 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.242 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.242 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.242 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.242 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.242 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.243 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.243 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "username of InfluxDB server user", 2024-03-18T16:20:54.243 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.243 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.243 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.243 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.243 INFO:teuthology.orchestra.run.smithi039.stdout: "verify_ssl": { 2024-03-18T16:20:54.243 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "verify_ssl", 2024-03-18T16:20:54.243 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.243 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.243 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.243 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "true", 2024-03-18T16:20:54.243 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.244 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.244 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.244 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-18T16:20:54.244 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.244 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.244 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.244 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:54.244 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:54.244 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.244 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:54.244 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "insights", 2024-03-18T16:20:54.244 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:54.244 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:54.244 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:54.245 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:54.245 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:54.245 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.245 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.245 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.245 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.245 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.245 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.245 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:54.245 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:54.245 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:54.245 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:54.245 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:54.245 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:54.246 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:54.246 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:54.246 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.246 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.246 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.246 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.246 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.246 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:54.246 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:54.246 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.246 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.246 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.246 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.247 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.247 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.247 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.247 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.247 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.247 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.247 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.247 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.247 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:54.247 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:54.247 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.247 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.247 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.247 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:54.248 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.248 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.248 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:54.248 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:54.248 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:54.248 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:54.248 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:54.248 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:54.248 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:54.248 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:54.248 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.248 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.248 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.248 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.249 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.249 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:54.249 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:54.249 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.249 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.249 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.249 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.249 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.249 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.249 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.249 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.249 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.249 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.250 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.250 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:54.250 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:54.250 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.250 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:54.250 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "iostat", 2024-03-18T16:20:54.250 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:54.250 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:54.250 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:54.250 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:54.250 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:54.250 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.250 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.251 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.251 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.251 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.251 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.251 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:54.251 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:54.251 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:54.251 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:54.251 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:54.251 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:54.251 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:54.251 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:54.251 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.251 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.251 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.252 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.252 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.252 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:54.252 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:54.252 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.252 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.252 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.252 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.252 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.252 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.252 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.252 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.252 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.253 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.253 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.253 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.253 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:54.253 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:54.253 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.253 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.253 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.253 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:54.253 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.253 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.253 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:54.253 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:54.253 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:54.253 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:54.254 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:54.254 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:54.254 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:54.254 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:54.254 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.254 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.254 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.254 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.254 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.254 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:54.254 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:54.254 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.254 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.255 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.255 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.255 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.255 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.255 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.255 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.255 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.255 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.255 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.255 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:54.255 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:54.255 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.255 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:54.255 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "k8sevents", 2024-03-18T16:20:54.255 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:54.256 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:54.256 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:54.256 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph_event_retention_days": { 2024-03-18T16:20:54.256 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "ceph_event_retention_days", 2024-03-18T16:20:54.256 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:54.256 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.256 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.256 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "7", 2024-03-18T16:20:54.256 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.256 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.256 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.256 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-18T16:20:54.256 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.256 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.256 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.257 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.257 INFO:teuthology.orchestra.run.smithi039.stdout: "config_check_secs": { 2024-03-18T16:20:54.257 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "config_check_secs", 2024-03-18T16:20:54.257 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:54.257 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.257 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.257 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "10", 2024-03-18T16:20:54.257 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "10", 2024-03-18T16:20:54.257 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.257 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.257 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-18T16:20:54.257 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.258 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.258 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.258 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.258 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:54.258 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:54.258 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.258 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.258 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.258 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.258 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.258 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.258 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:54.258 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:54.258 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:54.259 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:54.259 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:54.259 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:54.259 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:54.259 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:54.259 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.259 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.259 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.259 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.259 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.259 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:54.259 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:54.259 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.260 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.260 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.260 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.260 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.260 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.260 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.260 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.260 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.260 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.260 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.260 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.260 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:54.260 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:54.260 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.260 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.261 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.261 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:54.261 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.261 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.261 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:54.261 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:54.261 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:54.261 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:54.261 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:54.261 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:54.261 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:54.261 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:54.261 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.261 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.262 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.262 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.262 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.262 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:54.262 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:54.262 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.262 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.262 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.262 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.262 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.262 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.262 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.262 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.262 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.263 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.263 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.263 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:54.263 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:54.263 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.263 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:54.263 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "localpool", 2024-03-18T16:20:54.263 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:54.263 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:54.263 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:54.263 INFO:teuthology.orchestra.run.smithi039.stdout: "failure_domain": { 2024-03-18T16:20:54.263 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "failure_domain", 2024-03-18T16:20:54.263 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.263 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.263 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.264 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "host", 2024-03-18T16:20:54.264 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.264 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.264 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.264 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "failure domain for any created local pool", 2024-03-18T16:20:54.264 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-18T16:20:54.264 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.264 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.264 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.264 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:54.264 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:54.264 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.264 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.264 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.265 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.265 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.265 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.265 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:54.265 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:54.265 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:54.265 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:54.265 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:54.265 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:54.265 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:54.265 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:54.265 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.265 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.265 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.265 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.266 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.266 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:54.266 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:54.266 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.266 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.266 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.266 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.266 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.266 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.266 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.266 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.266 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.266 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.266 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.267 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.267 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:54.267 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:54.267 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.267 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.267 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.267 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:54.267 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.267 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.267 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:54.267 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:54.267 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:54.267 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:54.267 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:54.267 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:54.268 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:54.268 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:54.268 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.268 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.268 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.268 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.268 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.268 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:54.268 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:54.268 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.268 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.268 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.268 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.268 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.268 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.269 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.269 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.269 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.269 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.269 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.269 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.269 INFO:teuthology.orchestra.run.smithi039.stdout: "min_size": { 2024-03-18T16:20:54.269 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "min_size", 2024-03-18T16:20:54.269 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:54.269 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.269 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.269 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.269 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.269 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.269 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.270 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "default min_size for any created local pool", 2024-03-18T16:20:54.270 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-18T16:20:54.270 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.270 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.270 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.270 INFO:teuthology.orchestra.run.smithi039.stdout: "num_rep": { 2024-03-18T16:20:54.270 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "num_rep", 2024-03-18T16:20:54.270 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:54.270 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.270 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.270 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "3", 2024-03-18T16:20:54.270 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.270 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.270 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.271 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "default replica count for any created local pool", 2024-03-18T16:20:54.271 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.271 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.271 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.271 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.271 INFO:teuthology.orchestra.run.smithi039.stdout: "pg_num": { 2024-03-18T16:20:54.271 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "pg_num", 2024-03-18T16:20:54.271 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:54.271 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.271 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.271 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "128", 2024-03-18T16:20:54.271 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.271 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.272 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.272 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "default pg_num for any created local pool", 2024-03-18T16:20:54.272 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.272 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.272 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.272 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.272 INFO:teuthology.orchestra.run.smithi039.stdout: "prefix": { 2024-03-18T16:20:54.272 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "prefix", 2024-03-18T16:20:54.272 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.272 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.272 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.272 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.272 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.272 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.273 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.273 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "name prefix for any created local pool", 2024-03-18T16:20:54.273 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.273 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.278 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.279 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.279 INFO:teuthology.orchestra.run.smithi039.stdout: "subtree": { 2024-03-18T16:20:54.279 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "subtree", 2024-03-18T16:20:54.279 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.279 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.279 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.279 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "rack", 2024-03-18T16:20:54.279 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.279 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.279 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.279 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-18T16:20:54.279 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-18T16:20:54.279 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.280 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.280 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:54.280 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:54.280 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.280 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:54.280 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "mds_autoscaler", 2024-03-18T16:20:54.280 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:54.280 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:54.280 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:54.280 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:54.280 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:54.280 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.280 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.280 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.280 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.281 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.281 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.281 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:54.281 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:54.281 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:54.281 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:54.281 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:54.281 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:54.281 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:54.281 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:54.281 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.281 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.281 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.281 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.281 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.282 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:54.282 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:54.282 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.282 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.282 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.282 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.282 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.282 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.282 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.282 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.282 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.282 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.282 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.282 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.283 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:54.283 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:54.283 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.283 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.283 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.283 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:54.283 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.283 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.283 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:54.283 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:54.283 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:54.283 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:54.283 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:54.283 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:54.283 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:54.284 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:54.284 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.284 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.284 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.284 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.284 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.284 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:54.284 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:54.284 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.284 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.284 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.284 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.284 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.284 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.284 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.285 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.285 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.285 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.285 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.285 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:54.285 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:54.285 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.285 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:54.285 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "mirroring", 2024-03-18T16:20:54.285 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:54.285 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:54.285 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:54.285 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:54.285 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:54.285 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.286 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.286 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.286 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.286 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.286 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.286 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:54.286 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:54.286 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:54.286 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:54.286 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:54.286 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:54.286 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:54.286 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:54.286 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.287 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.287 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.287 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.287 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.287 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:54.287 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:54.287 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.287 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.287 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.287 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.287 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.287 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.287 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.287 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.287 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.287 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.288 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.308 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.308 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:54.308 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:54.309 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.309 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.309 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.309 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:54.309 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.309 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.309 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:54.309 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:54.309 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:54.309 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:54.309 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:54.310 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:54.310 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:54.310 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:54.310 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.310 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.310 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.310 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.310 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.311 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:54.311 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:54.311 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.311 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.311 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.312 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.312 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.312 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.313 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.313 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.313 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.313 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.314 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.314 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:54.314 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:54.314 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.314 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:54.315 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "nfs", 2024-03-18T16:20:54.315 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:54.315 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:54.315 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:54.316 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:54.316 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:54.316 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.316 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.316 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.317 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.317 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.317 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.317 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:54.318 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:54.318 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:54.318 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:54.318 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:54.318 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:54.318 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:54.318 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:54.319 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.319 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.319 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.319 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.319 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.319 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:54.319 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:54.320 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.320 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.320 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.320 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.320 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.320 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.321 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.321 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.321 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.321 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.321 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.321 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.322 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:54.322 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:54.322 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.322 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.322 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.322 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:54.323 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.323 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.323 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:54.323 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:54.323 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:54.324 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:54.324 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:54.324 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:54.324 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:54.324 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:54.324 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.325 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.325 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.325 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.325 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.325 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:54.325 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:54.326 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.326 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.326 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.326 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.326 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.326 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.327 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.327 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.327 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.327 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.327 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.327 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:54.328 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:54.328 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.328 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:54.328 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "orchestrator", 2024-03-18T16:20:54.328 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:54.328 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:54.329 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:54.329 INFO:teuthology.orchestra.run.smithi039.stdout: "fail_fs": { 2024-03-18T16:20:54.329 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "fail_fs", 2024-03-18T16:20:54.329 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.329 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.330 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.330 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.330 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.330 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.330 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.330 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-18T16:20:54.331 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.331 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.331 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.331 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.331 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:54.331 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:54.332 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.332 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.332 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.332 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.332 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.332 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.333 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:54.333 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:54.333 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:54.333 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:54.333 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:54.333 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:54.333 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:54.333 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:54.334 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.334 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.334 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.334 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.334 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.334 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:54.334 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:54.335 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.335 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.335 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.335 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.335 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.335 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.335 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.335 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.336 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.336 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.336 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.336 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.336 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:54.336 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:54.336 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.336 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.337 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.341 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:54.341 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.341 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.341 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:54.341 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:54.341 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:54.341 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:54.341 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:54.341 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:54.341 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:54.342 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:54.342 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.342 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.342 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.342 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.342 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.342 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:54.342 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:54.342 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.342 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.342 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.342 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.342 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.343 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.343 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.343 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.343 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.343 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.343 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.343 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.343 INFO:teuthology.orchestra.run.smithi039.stdout: "orchestrator": { 2024-03-18T16:20:54.343 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "orchestrator", 2024-03-18T16:20:54.343 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.343 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.343 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.343 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.343 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.344 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.344 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:54.344 INFO:teuthology.orchestra.run.smithi039.stdout: "cephadm", 2024-03-18T16:20:54.344 INFO:teuthology.orchestra.run.smithi039.stdout: "rook", 2024-03-18T16:20:54.344 INFO:teuthology.orchestra.run.smithi039.stdout: "test_orchestrator" 2024-03-18T16:20:54.344 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:54.344 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Orchestrator backend", 2024-03-18T16:20:54.344 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.344 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.344 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.344 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:54.344 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:54.344 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.344 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:54.344 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "osd_perf_query", 2024-03-18T16:20:54.345 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:54.345 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:54.345 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:54.345 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:54.345 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:54.345 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.345 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.345 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.345 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.345 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.345 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.345 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:54.345 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:54.346 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:54.346 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:54.349 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:54.349 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:54.349 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:54.349 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:54.349 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.349 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.349 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.349 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.349 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.350 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:54.350 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:54.350 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.350 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.350 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.350 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.350 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.350 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.350 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.350 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.350 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.350 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.350 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.350 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.351 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:54.351 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:54.351 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.351 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.351 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.351 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:54.351 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.351 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.351 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:54.351 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:54.351 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:54.351 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:54.351 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:54.352 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:54.352 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:54.352 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:54.352 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.352 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.352 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.352 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.352 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.352 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:54.352 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:54.352 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.352 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.353 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.353 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.353 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.353 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.353 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.353 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.353 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.353 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.353 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.353 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:54.353 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:54.353 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.353 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:54.353 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "osd_support", 2024-03-18T16:20:54.353 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:54.354 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:54.354 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:54.354 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:54.354 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:54.354 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.354 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.354 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.354 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.354 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.354 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.354 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:54.354 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:54.354 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:54.354 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:54.354 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:54.355 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:54.355 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:54.355 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:54.355 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.355 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.355 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.355 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.355 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.355 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:54.355 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:54.355 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.355 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.355 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.355 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.355 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.356 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.356 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.356 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.356 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.356 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.356 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.356 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.356 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:54.356 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:54.356 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.356 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.356 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.357 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:54.357 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.357 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.357 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:54.357 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:54.357 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:54.357 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:54.357 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:54.357 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:54.357 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:54.357 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:54.357 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.357 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.357 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.357 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.358 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.358 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:54.358 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:54.358 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.358 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.358 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.358 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.358 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.358 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.358 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.358 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.358 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.358 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.358 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.359 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:54.359 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:54.359 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.359 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:54.359 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "pg_autoscaler", 2024-03-18T16:20:54.359 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:54.359 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:54.359 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:54.359 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:54.359 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:54.359 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.359 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.359 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.359 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.359 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.360 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.360 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:54.360 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:54.360 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:54.360 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:54.360 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:54.360 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:54.360 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:54.360 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:54.360 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.360 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.360 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.360 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.360 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.361 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:54.361 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:54.361 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.361 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.361 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.361 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.361 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.361 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.361 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.361 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.361 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.361 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.361 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.361 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.362 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:54.362 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:54.362 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.362 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.362 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.362 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:54.362 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.362 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.362 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:54.363 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:54.363 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:54.363 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:54.363 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:54.363 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:54.363 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:54.363 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:54.363 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.363 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.363 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.363 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.363 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.363 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:54.363 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:54.364 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.364 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.364 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.364 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.364 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.364 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.364 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.364 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.364 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.364 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.364 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.364 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.364 INFO:teuthology.orchestra.run.smithi039.stdout: "sleep_interval": { 2024-03-18T16:20:54.364 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sleep_interval", 2024-03-18T16:20:54.364 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-03-18T16:20:54.365 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.365 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.365 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "60", 2024-03-18T16:20:54.365 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.365 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.365 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.365 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.365 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.365 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.365 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.365 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.365 INFO:teuthology.orchestra.run.smithi039.stdout: "threshold": { 2024-03-18T16:20:54.365 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "threshold", 2024-03-18T16:20:54.365 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "float", 2024-03-18T16:20:54.366 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.366 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.366 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "3.0", 2024-03-18T16:20:54.366 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "1.0", 2024-03-18T16:20:54.366 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.366 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.366 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "scaling threshold", 2024-03-18T16:20:54.366 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-18T16:20:54.366 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.366 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.366 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:54.366 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:54.366 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.366 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:54.367 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "progress", 2024-03-18T16:20:54.367 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:54.367 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:54.367 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:54.367 INFO:teuthology.orchestra.run.smithi039.stdout: "allow_pg_recovery_event": { 2024-03-18T16:20:54.367 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "allow_pg_recovery_event", 2024-03-18T16:20:54.367 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.367 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.367 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.367 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.367 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.367 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.367 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.367 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-18T16:20:54.367 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.368 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.368 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.368 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.368 INFO:teuthology.orchestra.run.smithi039.stdout: "enabled": { 2024-03-18T16:20:54.368 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "enabled", 2024-03-18T16:20:54.368 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.368 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.368 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.368 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:54.368 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.368 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.368 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.368 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.368 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.368 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.368 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.369 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.369 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:54.369 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:54.369 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.369 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.369 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.369 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.369 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.369 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.369 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:54.369 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:54.369 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:54.369 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:54.369 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:54.369 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:54.369 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:54.370 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:54.370 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.370 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.370 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.370 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.370 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.370 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:54.370 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:54.370 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.370 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.370 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.370 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.370 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.370 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.370 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.371 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.371 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.371 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.371 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.371 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.371 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:54.371 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:54.371 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.371 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.371 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.371 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:54.371 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.371 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.371 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:54.372 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:54.372 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:54.372 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:54.372 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:54.372 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:54.372 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:54.372 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:54.372 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.372 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.372 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.372 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.372 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.372 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:54.373 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:54.373 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.373 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.373 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.373 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.373 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.373 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.373 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.373 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.373 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.373 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.373 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.374 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.374 INFO:teuthology.orchestra.run.smithi039.stdout: "max_completed_events": { 2024-03-18T16:20:54.374 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "max_completed_events", 2024-03-18T16:20:54.374 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:54.374 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.374 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.374 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "50", 2024-03-18T16:20:54.374 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.374 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.374 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.374 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "number of past completed events to remember", 2024-03-18T16:20:54.374 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.374 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.374 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.375 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.375 INFO:teuthology.orchestra.run.smithi039.stdout: "sleep_interval": { 2024-03-18T16:20:54.375 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sleep_interval", 2024-03-18T16:20:54.375 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-03-18T16:20:54.375 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.375 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.375 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "5", 2024-03-18T16:20:54.375 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.375 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.375 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.375 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "how long the module is going to sleep", 2024-03-18T16:20:54.375 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.375 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.375 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.376 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:54.376 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:54.376 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.376 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:54.376 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "prometheus", 2024-03-18T16:20:54.376 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:54.376 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:54.376 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:54.376 INFO:teuthology.orchestra.run.smithi039.stdout: "cache": { 2024-03-18T16:20:54.376 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "cache", 2024-03-18T16:20:54.376 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.376 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.376 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.376 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:54.376 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.377 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.377 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.377 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.377 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.377 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.377 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.377 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.377 INFO:teuthology.orchestra.run.smithi039.stdout: "exclude_perf_counters": { 2024-03-18T16:20:54.377 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "exclude_perf_counters", 2024-03-18T16:20:54.377 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.377 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.377 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.378 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:54.378 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.378 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.378 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.378 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-18T16:20:54.378 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-18T16:20:54.378 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.378 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.378 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.378 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:54.378 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:54.378 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.378 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.378 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.379 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.379 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.379 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.379 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:54.379 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:54.379 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:54.379 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:54.379 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:54.379 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:54.379 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:54.379 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:54.379 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.379 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.379 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.379 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.380 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.380 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:54.380 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:54.380 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.380 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.380 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.380 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.380 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.380 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.380 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.380 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.380 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.380 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.380 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.380 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.381 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:54.381 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:54.381 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.381 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.381 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.381 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:54.381 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.381 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.381 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:54.381 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:54.381 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:54.381 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:54.382 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:54.382 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:54.382 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:54.382 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:54.382 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.382 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.382 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.382 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.382 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.382 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:54.382 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:54.382 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.383 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.383 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.383 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.383 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.383 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.383 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.383 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.383 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.383 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.383 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.383 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.383 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_stats_pools": { 2024-03-18T16:20:54.383 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "rbd_stats_pools", 2024-03-18T16:20:54.383 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.383 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.384 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.384 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.384 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.384 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.384 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.384 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.384 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.384 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.384 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.384 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.384 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-18T16:20:54.384 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-18T16:20:54.384 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:54.384 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.384 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.385 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "300", 2024-03-18T16:20:54.385 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.385 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.385 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.385 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.385 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.385 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.385 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.385 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.385 INFO:teuthology.orchestra.run.smithi039.stdout: "scrape_interval": { 2024-03-18T16:20:54.385 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "scrape_interval", 2024-03-18T16:20:54.385 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "float", 2024-03-18T16:20:54.385 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.385 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.385 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "15.0", 2024-03-18T16:20:54.385 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.386 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.386 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.386 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.386 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.386 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.386 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.386 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.386 INFO:teuthology.orchestra.run.smithi039.stdout: "server_addr": { 2024-03-18T16:20:54.386 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "server_addr", 2024-03-18T16:20:54.386 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.386 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.386 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.386 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "::", 2024-03-18T16:20:54.386 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.387 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.387 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.387 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-18T16:20:54.387 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.387 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.387 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.387 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.387 INFO:teuthology.orchestra.run.smithi039.stdout: "server_port": { 2024-03-18T16:20:54.387 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "server_port", 2024-03-18T16:20:54.387 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:54.387 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.387 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.387 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "9283", 2024-03-18T16:20:54.387 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.387 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.388 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.388 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-18T16:20:54.388 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.388 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.388 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.388 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.388 INFO:teuthology.orchestra.run.smithi039.stdout: "stale_cache_strategy": { 2024-03-18T16:20:54.388 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "stale_cache_strategy", 2024-03-18T16:20:54.388 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.388 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.388 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.388 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "log", 2024-03-18T16:20:54.388 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.389 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.389 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.389 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.389 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.389 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.389 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.389 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.389 INFO:teuthology.orchestra.run.smithi039.stdout: "standby_behaviour": { 2024-03-18T16:20:54.389 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "standby_behaviour", 2024-03-18T16:20:54.389 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.389 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.389 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.389 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "default", 2024-03-18T16:20:54.389 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.389 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.390 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:54.390 INFO:teuthology.orchestra.run.smithi039.stdout: "default", 2024-03-18T16:20:54.390 INFO:teuthology.orchestra.run.smithi039.stdout: "error" 2024-03-18T16:20:54.390 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:54.390 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.390 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.390 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.390 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.390 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.390 INFO:teuthology.orchestra.run.smithi039.stdout: "standby_error_status_code": { 2024-03-18T16:20:54.390 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "standby_error_status_code", 2024-03-18T16:20:54.390 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:54.390 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.390 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.391 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "500", 2024-03-18T16:20:54.391 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "400", 2024-03-18T16:20:54.391 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "599", 2024-03-18T16:20:54.391 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.391 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.391 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.391 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.391 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.391 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:54.391 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:54.391 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.391 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:54.391 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "rbd_support", 2024-03-18T16:20:54.391 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:54.392 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:54.392 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:54.392 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:54.392 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:54.392 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.392 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.392 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.392 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.392 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.392 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.392 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:54.392 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:54.392 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:54.393 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:54.393 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:54.393 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:54.393 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:54.393 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:54.393 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.393 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.393 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.393 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.393 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.393 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:54.393 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:54.393 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.393 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.393 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.394 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.394 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.394 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.394 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.394 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.394 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.394 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.394 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.394 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.394 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:54.394 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:54.394 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.394 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.394 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.395 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:54.395 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.395 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.395 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:54.395 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:54.395 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:54.395 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:54.395 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:54.395 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:54.395 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:54.395 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:54.395 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.395 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.395 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.395 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.395 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.396 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:54.396 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:54.396 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.396 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.396 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.396 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.396 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.396 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.396 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.396 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.396 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.396 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.396 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.396 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.396 INFO:teuthology.orchestra.run.smithi039.stdout: "max_concurrent_snap_create": { 2024-03-18T16:20:54.397 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "max_concurrent_snap_create", 2024-03-18T16:20:54.397 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:54.397 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.397 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.397 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "10", 2024-03-18T16:20:54.397 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.397 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.397 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.397 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.397 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.397 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.397 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.397 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.397 INFO:teuthology.orchestra.run.smithi039.stdout: "mirror_snapshot_schedule": { 2024-03-18T16:20:54.397 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "mirror_snapshot_schedule", 2024-03-18T16:20:54.428 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.428 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.428 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.429 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.429 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.429 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.429 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.429 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.429 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.429 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.429 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.429 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.429 INFO:teuthology.orchestra.run.smithi039.stdout: "trash_purge_schedule": { 2024-03-18T16:20:54.429 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "trash_purge_schedule", 2024-03-18T16:20:54.429 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.429 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.429 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.429 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.430 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.430 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.430 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.430 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.430 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.430 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.430 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.430 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:54.430 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:54.430 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.430 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:54.430 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "restful", 2024-03-18T16:20:54.430 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:54.430 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:54.431 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:54.431 INFO:teuthology.orchestra.run.smithi039.stdout: "enable_auth": { 2024-03-18T16:20:54.431 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "enable_auth", 2024-03-18T16:20:54.431 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.431 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.431 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.431 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:54.431 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.431 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.431 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.431 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.431 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.431 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.431 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.432 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.432 INFO:teuthology.orchestra.run.smithi039.stdout: "key_file": { 2024-03-18T16:20:54.432 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "key_file", 2024-03-18T16:20:54.432 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.432 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.432 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.432 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.432 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.432 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.432 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.432 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.432 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.432 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.432 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.433 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.433 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:54.433 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:54.433 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.433 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.433 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.433 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.433 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.433 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.433 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:54.434 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:54.434 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:54.434 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:54.434 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:54.434 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:54.434 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:54.434 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:54.434 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.434 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.434 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.434 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.434 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.434 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:54.434 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:54.434 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.435 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.435 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.435 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.435 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.435 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.435 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.435 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.435 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.435 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.435 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.435 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.435 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:54.435 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:54.435 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.435 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.436 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.436 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:54.436 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.436 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.436 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:54.436 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:54.436 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:54.436 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:54.436 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:54.436 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:54.436 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:54.436 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:54.436 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.436 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.436 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.437 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.437 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.437 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:54.437 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:54.437 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.437 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.437 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.437 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.437 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.437 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.437 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.437 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.437 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.437 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.437 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.438 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.438 INFO:teuthology.orchestra.run.smithi039.stdout: "server_addr": { 2024-03-18T16:20:54.438 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "server_addr", 2024-03-18T16:20:54.438 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.438 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.438 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.438 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.438 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.438 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.438 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.438 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.438 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.438 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.438 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.439 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.439 INFO:teuthology.orchestra.run.smithi039.stdout: "server_port": { 2024-03-18T16:20:54.439 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "server_port", 2024-03-18T16:20:54.439 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.439 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.439 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.439 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.439 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.439 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.439 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.439 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.439 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.439 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.439 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.439 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:54.440 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:54.440 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.440 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:54.440 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "rgw", 2024-03-18T16:20:54.440 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:54.440 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:54.440 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:54.440 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:54.440 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:54.440 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.440 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.440 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.440 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.440 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.441 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.441 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:54.441 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:54.441 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:54.441 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:54.441 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:54.441 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:54.441 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:54.441 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:54.441 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.441 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.441 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.441 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.441 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.441 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:54.442 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:54.442 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.442 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.442 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.442 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.442 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.442 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.442 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.442 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.442 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.442 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.442 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.443 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.443 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:54.443 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:54.443 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.443 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.443 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.443 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:54.443 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.443 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.443 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:54.443 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:54.443 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:54.443 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:54.443 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:54.443 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:54.443 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:54.444 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:54.444 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.444 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.444 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.444 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.444 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.444 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:54.444 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:54.444 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.444 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.444 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.444 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.444 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.444 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.444 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.445 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.445 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.445 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.445 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.445 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:54.445 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:54.445 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.445 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:54.445 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "rook", 2024-03-18T16:20:54.445 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:54.445 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:54.445 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:54.445 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:54.445 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:54.445 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.446 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.446 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.446 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.446 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.446 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.446 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:54.446 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:54.446 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:54.446 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:54.446 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:54.446 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:54.446 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:54.446 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:54.446 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.446 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.447 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.447 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.447 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.447 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:54.447 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:54.447 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.447 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.447 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.447 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.447 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.447 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.447 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.448 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.448 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.448 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.448 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.448 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.448 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:54.448 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:54.448 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.448 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.448 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.448 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:54.448 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.448 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.448 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:54.449 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:54.449 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:54.449 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:54.449 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:54.449 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:54.449 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:54.449 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:54.449 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.449 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.449 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.449 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.449 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.449 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:54.449 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:54.450 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.450 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.450 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.450 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.450 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.450 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.450 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.450 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.450 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.450 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.450 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.450 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.450 INFO:teuthology.orchestra.run.smithi039.stdout: "storage_class": { 2024-03-18T16:20:54.450 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "storage_class", 2024-03-18T16:20:54.450 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.451 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.451 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.451 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "local", 2024-03-18T16:20:54.451 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.451 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.451 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.451 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-18T16:20:54.451 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.451 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.451 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.451 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:54.451 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:54.451 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.451 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:54.452 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "selftest", 2024-03-18T16:20:54.452 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:54.452 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:54.452 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:54.452 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:54.452 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:54.452 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.452 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.452 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.452 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.452 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.452 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.453 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:54.453 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:54.453 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:54.453 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:54.453 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:54.453 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:54.453 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:54.453 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:54.453 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.453 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.453 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.453 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.453 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.453 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:54.453 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:54.454 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.454 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.454 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.454 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.454 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.454 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.454 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.454 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.454 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.454 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.454 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.454 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.454 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:54.454 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:54.454 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.455 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.455 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.455 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:54.455 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.455 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.455 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:54.455 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:54.455 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:54.455 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:54.455 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:54.455 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:54.455 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:54.455 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:54.455 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.455 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.456 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.456 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.456 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.456 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:54.456 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:54.456 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.456 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.456 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.456 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.456 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.456 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.456 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.456 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.456 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.456 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.456 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.457 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.457 INFO:teuthology.orchestra.run.smithi039.stdout: "roption1": { 2024-03-18T16:20:54.457 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "roption1", 2024-03-18T16:20:54.457 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.457 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.457 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.457 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.457 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.457 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.457 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.458 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.458 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.458 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.458 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.458 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.458 INFO:teuthology.orchestra.run.smithi039.stdout: "roption2": { 2024-03-18T16:20:54.458 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "roption2", 2024-03-18T16:20:54.458 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.458 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.458 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.458 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "xyz", 2024-03-18T16:20:54.458 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.458 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.458 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.458 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.459 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.459 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.459 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.459 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.459 INFO:teuthology.orchestra.run.smithi039.stdout: "rwoption1": { 2024-03-18T16:20:54.459 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "rwoption1", 2024-03-18T16:20:54.459 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.459 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.459 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.459 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.459 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.459 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.459 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.459 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.459 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.459 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.460 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.460 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.460 INFO:teuthology.orchestra.run.smithi039.stdout: "rwoption2": { 2024-03-18T16:20:54.460 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "rwoption2", 2024-03-18T16:20:54.460 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:54.460 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.460 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.460 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.460 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.460 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.460 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.461 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.461 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.461 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.461 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.461 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.461 INFO:teuthology.orchestra.run.smithi039.stdout: "rwoption3": { 2024-03-18T16:20:54.461 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "rwoption3", 2024-03-18T16:20:54.461 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "float", 2024-03-18T16:20:54.461 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.461 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.461 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.461 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.461 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.462 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.462 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.462 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.462 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.462 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.462 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.462 INFO:teuthology.orchestra.run.smithi039.stdout: "rwoption4": { 2024-03-18T16:20:54.462 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "rwoption4", 2024-03-18T16:20:54.462 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.462 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.462 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.462 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.463 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.463 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.463 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.463 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.463 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.463 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.463 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.463 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.463 INFO:teuthology.orchestra.run.smithi039.stdout: "rwoption5": { 2024-03-18T16:20:54.463 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "rwoption5", 2024-03-18T16:20:54.463 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.463 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.463 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.463 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.463 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.464 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.464 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.464 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.464 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.464 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.464 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.464 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.464 INFO:teuthology.orchestra.run.smithi039.stdout: "rwoption6": { 2024-03-18T16:20:54.464 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "rwoption6", 2024-03-18T16:20:54.464 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.464 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.464 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.464 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:54.464 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.464 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.465 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.465 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.465 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.465 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.465 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.465 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.465 INFO:teuthology.orchestra.run.smithi039.stdout: "rwoption7": { 2024-03-18T16:20:54.465 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "rwoption7", 2024-03-18T16:20:54.465 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:54.466 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.466 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.466 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.466 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "1", 2024-03-18T16:20:54.466 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "42", 2024-03-18T16:20:54.466 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.466 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.466 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.467 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.467 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.467 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.467 INFO:teuthology.orchestra.run.smithi039.stdout: "testkey": { 2024-03-18T16:20:54.467 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "testkey", 2024-03-18T16:20:54.467 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.467 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.468 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.468 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.468 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.468 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.468 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.468 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.468 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.468 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.468 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.468 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.468 INFO:teuthology.orchestra.run.smithi039.stdout: "testlkey": { 2024-03-18T16:20:54.468 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "testlkey", 2024-03-18T16:20:54.468 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.468 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.468 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.469 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.469 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.469 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.469 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.469 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.469 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.469 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.469 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.469 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.469 INFO:teuthology.orchestra.run.smithi039.stdout: "testnewline": { 2024-03-18T16:20:54.469 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "testnewline", 2024-03-18T16:20:54.469 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.469 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.469 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.469 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.469 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.470 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.470 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.470 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.470 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.470 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.470 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.470 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:54.470 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:54.470 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.470 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:54.470 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "snap_schedule", 2024-03-18T16:20:54.470 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:54.470 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:54.470 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:54.470 INFO:teuthology.orchestra.run.smithi039.stdout: "allow_m_granularity": { 2024-03-18T16:20:54.471 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "allow_m_granularity", 2024-03-18T16:20:54.471 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.471 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.471 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.471 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.471 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.471 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.471 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.471 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "allow minute scheduled snapshots", 2024-03-18T16:20:54.471 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.471 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.471 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.471 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.472 INFO:teuthology.orchestra.run.smithi039.stdout: "dump_on_update": { 2024-03-18T16:20:54.472 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "dump_on_update", 2024-03-18T16:20:54.472 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.472 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.472 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.472 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.472 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.472 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.472 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.472 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "dump database to debug log on update", 2024-03-18T16:20:54.472 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.472 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.472 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.473 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.473 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:54.473 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:54.473 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.473 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.473 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.473 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.473 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.473 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.473 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:54.473 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:54.473 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:54.473 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:54.473 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:54.473 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:54.474 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:54.474 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:54.474 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.474 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.474 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.474 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.474 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.474 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:54.474 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:54.474 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.474 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.474 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.475 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.475 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.475 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.475 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.475 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.475 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.475 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.475 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.475 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.475 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:54.475 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:54.475 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.475 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.475 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.476 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:54.476 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.476 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.476 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:54.476 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:54.476 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:54.476 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:54.476 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:54.476 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:54.476 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:54.476 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:54.476 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.476 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.477 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.477 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.477 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.477 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:54.477 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:54.477 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.477 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.477 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.477 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.477 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.477 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.477 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.477 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.477 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.478 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.478 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.478 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:54.478 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:54.478 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.478 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:54.478 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "stats", 2024-03-18T16:20:54.478 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:54.478 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:54.478 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:54.478 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:54.478 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:54.479 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.479 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.479 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.479 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.479 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.479 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.479 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:54.479 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:54.479 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:54.479 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:54.479 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:54.479 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:54.479 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:54.479 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:54.479 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.480 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.480 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.480 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.480 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.480 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:54.480 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:54.480 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.480 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.480 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.480 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.480 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.480 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.480 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.480 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.480 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.481 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.481 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.481 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.481 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:54.481 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:54.481 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.481 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.481 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.481 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:54.481 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.481 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.481 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:54.481 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:54.481 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:54.482 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:54.482 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:54.482 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:54.482 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:54.482 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:54.482 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.482 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.482 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.482 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.482 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.482 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:54.482 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:54.482 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.482 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.483 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.483 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.483 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.483 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.483 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.483 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.560 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.561 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.561 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.561 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:54.561 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:54.561 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.561 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:54.561 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "status", 2024-03-18T16:20:54.561 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:54.561 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:54.561 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:54.561 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:54.562 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:54.562 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.562 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.562 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.562 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.562 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.562 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.562 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:54.562 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:54.562 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:54.562 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:54.563 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:54.563 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:54.563 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:54.563 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:54.563 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.563 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.563 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.563 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.563 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.563 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:54.563 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:54.563 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.563 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.563 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.563 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.563 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.564 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.564 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.564 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.564 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.564 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.564 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.564 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.564 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:54.564 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:54.564 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.564 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.564 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.564 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:54.564 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.564 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.565 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:54.565 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:54.565 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:54.565 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:54.565 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:54.565 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:54.565 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:54.565 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:54.565 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.590 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.590 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.590 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.590 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.590 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:54.591 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:54.591 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.591 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.591 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.591 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.591 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.591 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.591 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.591 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.591 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.591 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.591 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.591 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:54.591 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:54.591 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.592 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:54.592 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "telegraf", 2024-03-18T16:20:54.592 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:54.592 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:54.592 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:54.592 INFO:teuthology.orchestra.run.smithi039.stdout: "address": { 2024-03-18T16:20:54.592 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "address", 2024-03-18T16:20:54.592 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.592 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.592 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.592 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-18T16:20:54.592 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.592 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.592 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.593 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.593 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.593 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.593 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.593 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.593 INFO:teuthology.orchestra.run.smithi039.stdout: "interval": { 2024-03-18T16:20:54.593 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "interval", 2024-03-18T16:20:54.593 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-03-18T16:20:54.593 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.593 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.593 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "15", 2024-03-18T16:20:54.593 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.593 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.593 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.593 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.594 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.594 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.594 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.594 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.594 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:54.594 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:54.594 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.594 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.594 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.594 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.594 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.594 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.594 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:54.594 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:54.595 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:54.595 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:54.595 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:54.595 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:54.595 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:54.595 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:54.595 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.595 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.595 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.595 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.595 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.595 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:54.595 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:54.595 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.596 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.596 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.596 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.596 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.596 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.596 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.596 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.596 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.596 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.596 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.596 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.596 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:54.596 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:54.596 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.596 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.597 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.597 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:54.597 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.597 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.597 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:54.597 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:54.597 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:54.597 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:54.597 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:54.597 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:54.597 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:54.597 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:54.597 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.597 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.597 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.598 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.598 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.598 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:54.598 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:54.598 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.598 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.598 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.598 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.598 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.598 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.598 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.598 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.598 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.598 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.599 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.599 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:54.599 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:54.599 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.599 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:54.599 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "telemetry", 2024-03-18T16:20:54.599 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:54.599 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:54.599 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:54.599 INFO:teuthology.orchestra.run.smithi039.stdout: "channel_basic": { 2024-03-18T16:20:54.599 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "channel_basic", 2024-03-18T16:20:54.599 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.599 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.599 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.599 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:54.599 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.600 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.600 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.600 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-18T16:20:54.600 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.600 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.600 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.600 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.600 INFO:teuthology.orchestra.run.smithi039.stdout: "channel_crash": { 2024-03-18T16:20:54.600 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "channel_crash", 2024-03-18T16:20:54.600 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.600 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.600 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.600 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:54.600 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.600 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.601 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.601 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-18T16:20:54.601 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.601 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.601 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.601 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.601 INFO:teuthology.orchestra.run.smithi039.stdout: "channel_device": { 2024-03-18T16:20:54.601 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "channel_device", 2024-03-18T16:20:54.601 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.601 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.601 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.601 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:54.601 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.601 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.601 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.602 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-18T16:20:54.602 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.602 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.602 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.602 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.602 INFO:teuthology.orchestra.run.smithi039.stdout: "channel_ident": { 2024-03-18T16:20:54.602 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "channel_ident", 2024-03-18T16:20:54.602 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.602 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.602 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.602 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.602 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.602 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.602 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.602 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-18T16:20:54.603 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.603 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.603 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.603 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.603 INFO:teuthology.orchestra.run.smithi039.stdout: "channel_perf": { 2024-03-18T16:20:54.603 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "channel_perf", 2024-03-18T16:20:54.603 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.603 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.603 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.603 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.603 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.603 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.603 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.603 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-18T16:20:54.603 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.604 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.604 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.604 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.604 INFO:teuthology.orchestra.run.smithi039.stdout: "contact": { 2024-03-18T16:20:54.604 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "contact", 2024-03-18T16:20:54.604 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.604 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.604 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.604 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.604 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.604 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.604 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.604 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.604 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.604 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.604 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.605 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.605 INFO:teuthology.orchestra.run.smithi039.stdout: "description": { 2024-03-18T16:20:54.605 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "description", 2024-03-18T16:20:54.605 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.605 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.605 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.605 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.605 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.605 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.605 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.605 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.605 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.605 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.605 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.606 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.606 INFO:teuthology.orchestra.run.smithi039.stdout: "device_url": { 2024-03-18T16:20:54.606 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "device_url", 2024-03-18T16:20:54.606 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.606 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.606 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.606 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-18T16:20:54.606 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.606 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.606 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.606 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.606 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.606 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.606 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.606 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.607 INFO:teuthology.orchestra.run.smithi039.stdout: "enabled": { 2024-03-18T16:20:54.607 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "enabled", 2024-03-18T16:20:54.607 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.607 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.607 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.607 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.607 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.607 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.607 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.607 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.607 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.607 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.607 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.607 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.607 INFO:teuthology.orchestra.run.smithi039.stdout: "interval": { 2024-03-18T16:20:54.608 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "interval", 2024-03-18T16:20:54.608 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:54.608 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.608 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.608 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "24", 2024-03-18T16:20:54.608 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "8", 2024-03-18T16:20:54.608 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.608 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.608 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.608 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.608 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.608 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.608 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.608 INFO:teuthology.orchestra.run.smithi039.stdout: "last_opt_revision": { 2024-03-18T16:20:54.608 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "last_opt_revision", 2024-03-18T16:20:54.608 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:54.609 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.609 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.609 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "1", 2024-03-18T16:20:54.609 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.609 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.609 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.609 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.609 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.609 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.609 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.609 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.609 INFO:teuthology.orchestra.run.smithi039.stdout: "leaderboard": { 2024-03-18T16:20:54.609 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "leaderboard", 2024-03-18T16:20:54.609 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.609 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.610 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.610 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.610 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.610 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.610 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.610 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.610 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.610 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.610 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.610 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.610 INFO:teuthology.orchestra.run.smithi039.stdout: "leaderboard_description": { 2024-03-18T16:20:54.610 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "leaderboard_description", 2024-03-18T16:20:54.610 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.610 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.610 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.611 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.611 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.611 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.611 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.611 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.611 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.611 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.611 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.611 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.611 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:54.611 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:54.611 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.611 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.611 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.611 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.611 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.612 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.612 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:54.612 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:54.612 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:54.612 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:54.612 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:54.612 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:54.612 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:54.612 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:54.612 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.612 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.612 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.612 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.612 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.612 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:54.613 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:54.613 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.613 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.613 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.613 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.613 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.613 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.613 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.613 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.613 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.613 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.613 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.613 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.613 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:54.613 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:54.614 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.614 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.614 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.614 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:54.614 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.614 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.614 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:54.614 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:54.614 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:54.614 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:54.614 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:54.614 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:54.614 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:54.614 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:54.614 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.615 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.615 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.615 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.615 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.615 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:54.615 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:54.615 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.615 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.615 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.615 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.615 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.615 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.615 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.615 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.615 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.615 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.616 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.616 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.616 INFO:teuthology.orchestra.run.smithi039.stdout: "organization": { 2024-03-18T16:20:54.616 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "organization", 2024-03-18T16:20:54.616 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.616 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.616 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.616 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.616 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.616 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.616 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.616 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.616 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.616 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.616 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.617 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.617 INFO:teuthology.orchestra.run.smithi039.stdout: "proxy": { 2024-03-18T16:20:54.617 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "proxy", 2024-03-18T16:20:54.617 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.617 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.617 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.617 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.617 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.617 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.617 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.617 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.617 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.617 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.617 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.617 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.618 INFO:teuthology.orchestra.run.smithi039.stdout: "url": { 2024-03-18T16:20:54.618 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "url", 2024-03-18T16:20:54.618 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.618 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.618 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.618 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-18T16:20:54.618 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.618 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.618 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.618 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.618 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.618 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.618 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.618 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:54.619 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:54.619 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.619 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:54.619 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "test_orchestrator", 2024-03-18T16:20:54.619 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:54.619 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:54.619 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:54.619 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:54.619 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:54.619 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.619 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.619 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.619 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.619 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.619 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.620 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:54.620 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:54.621 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:54.621 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:54.621 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:54.621 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:54.621 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:54.621 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:54.621 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.622 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.622 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.622 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.622 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.622 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:54.622 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:54.622 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.622 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.622 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.622 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.622 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.622 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.622 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.622 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.622 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.623 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.623 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.623 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.623 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:54.623 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:54.623 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.623 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.623 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.623 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:54.623 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.623 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.623 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:54.623 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:54.623 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:54.623 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:54.624 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:54.624 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:54.624 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:54.624 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:54.624 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.624 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.624 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.624 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.624 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.624 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:54.624 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:54.624 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.624 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.624 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.624 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.625 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.625 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.625 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.625 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.625 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.625 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.625 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.625 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:54.625 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:54.625 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.625 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:54.625 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "volumes", 2024-03-18T16:20:54.625 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:54.625 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:54.625 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:54.625 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:54.626 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:54.626 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.626 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.626 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.626 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.626 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.626 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.626 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:54.626 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:54.626 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:54.626 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:54.626 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:54.626 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:54.626 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:54.626 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:54.627 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.627 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.627 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.627 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.627 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.627 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:54.627 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:54.627 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.627 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.627 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.627 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.627 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.627 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.627 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.627 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.628 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.628 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.628 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.628 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.628 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:54.628 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:54.628 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.628 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.628 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.628 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:54.628 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.628 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.628 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:54.628 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:54.628 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:54.629 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:54.629 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:54.629 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:54.629 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:54.629 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:54.629 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.629 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.629 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.629 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.629 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.629 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:54.629 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:54.629 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.629 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.629 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.630 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.630 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.630 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.630 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.630 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.630 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.630 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.630 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.630 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.630 INFO:teuthology.orchestra.run.smithi039.stdout: "max_concurrent_clones": { 2024-03-18T16:20:54.630 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "max_concurrent_clones", 2024-03-18T16:20:54.630 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:54.630 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.630 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.630 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "4", 2024-03-18T16:20:54.631 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.631 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.631 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.631 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-18T16:20:54.631 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.631 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.631 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.631 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.631 INFO:teuthology.orchestra.run.smithi039.stdout: "periodic_async_work": { 2024-03-18T16:20:54.631 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "periodic_async_work", 2024-03-18T16:20:54.631 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.631 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.637 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.637 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.637 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.637 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.637 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.637 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Periodically check for async work", 2024-03-18T16:20:54.637 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.638 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.638 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.638 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.638 INFO:teuthology.orchestra.run.smithi039.stdout: "snapshot_clone_delay": { 2024-03-18T16:20:54.638 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "snapshot_clone_delay", 2024-03-18T16:20:54.638 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:54.638 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.638 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.638 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-03-18T16:20:54.638 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.638 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.638 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.638 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-18T16:20:54.639 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.639 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.639 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.639 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.639 INFO:teuthology.orchestra.run.smithi039.stdout: "snapshot_clone_no_wait": { 2024-03-18T16:20:54.639 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "snapshot_clone_no_wait", 2024-03-18T16:20:54.639 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.639 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.639 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.639 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:54.639 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.639 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.639 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.639 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-18T16:20:54.639 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.640 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.640 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.640 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:54.640 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:54.640 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.640 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:54.640 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "zabbix", 2024-03-18T16:20:54.640 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:54.640 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:54.640 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:54.640 INFO:teuthology.orchestra.run.smithi039.stdout: "discovery_interval": { 2024-03-18T16:20:54.640 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "discovery_interval", 2024-03-18T16:20:54.640 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "uint", 2024-03-18T16:20:54.640 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.640 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.641 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "100", 2024-03-18T16:20:54.641 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.641 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.641 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.641 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.641 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.641 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.641 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.641 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.641 INFO:teuthology.orchestra.run.smithi039.stdout: "identifier": { 2024-03-18T16:20:54.641 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "identifier", 2024-03-18T16:20:54.641 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.641 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.641 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.641 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.641 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.642 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.642 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.642 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.642 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.642 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.642 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.642 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.642 INFO:teuthology.orchestra.run.smithi039.stdout: "interval": { 2024-03-18T16:20:54.642 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "interval", 2024-03-18T16:20:54.642 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-03-18T16:20:54.642 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.642 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.642 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "60", 2024-03-18T16:20:54.642 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.643 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.643 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.643 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.643 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.643 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.643 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.643 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.643 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:54.643 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:54.643 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.643 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.643 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.643 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.643 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.644 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.644 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:54.644 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:54.644 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:54.644 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:54.644 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:54.644 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:54.644 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:54.644 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:54.644 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.644 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.644 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.644 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.644 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.644 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:54.644 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:54.645 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.645 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.645 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.645 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.645 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.645 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.645 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.645 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.645 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.645 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.645 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.645 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.645 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:54.645 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:54.645 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.646 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.646 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.646 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:54.646 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.646 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.646 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:54.646 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:54.646 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:54.646 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:54.646 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:54.646 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:54.646 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:54.646 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:54.646 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.646 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.647 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.647 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.647 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.647 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:54.647 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:54.647 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.647 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.647 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.647 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.647 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.647 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.647 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.647 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.647 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.647 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.647 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.648 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.648 INFO:teuthology.orchestra.run.smithi039.stdout: "zabbix_host": { 2024-03-18T16:20:54.648 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "zabbix_host", 2024-03-18T16:20:54.648 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.648 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.648 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.648 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.648 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.660 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.661 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.661 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.661 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.661 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.661 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.661 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.661 INFO:teuthology.orchestra.run.smithi039.stdout: "zabbix_port": { 2024-03-18T16:20:54.661 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "zabbix_port", 2024-03-18T16:20:54.661 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:54.661 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.661 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.661 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "10051", 2024-03-18T16:20:54.661 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.661 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.661 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.662 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.662 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.662 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.662 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.662 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.662 INFO:teuthology.orchestra.run.smithi039.stdout: "zabbix_sender": { 2024-03-18T16:20:54.662 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "zabbix_sender", 2024-03-18T16:20:54.662 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.662 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.662 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.662 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-18T16:20:54.662 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.662 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.663 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.663 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.663 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.663 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.663 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.663 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:54.663 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:54.663 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:54.663 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-03-18T16:20:54.663 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.663 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:54.663 INFO:teuthology.orchestra.run.smithi039.stdout: "gid": 24113, 2024-03-18T16:20:54.663 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "y", 2024-03-18T16:20:54.664 INFO:teuthology.orchestra.run.smithi039.stdout: "mgr_features": 4540701547738038271, 2024-03-18T16:20:54.664 INFO:teuthology.orchestra.run.smithi039.stdout: "available_modules": [ 2024-03-18T16:20:54.664 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:54.664 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "alerts", 2024-03-18T16:20:54.664 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:54.664 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:54.664 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:54.664 INFO:teuthology.orchestra.run.smithi039.stdout: "interval": { 2024-03-18T16:20:54.664 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "interval", 2024-03-18T16:20:54.664 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-03-18T16:20:54.664 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.664 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.664 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "60", 2024-03-18T16:20:54.664 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.664 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.665 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.665 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "How frequently to reexamine health status", 2024-03-18T16:20:54.665 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.665 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.665 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.665 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.665 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:54.665 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:54.665 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.665 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.665 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.665 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.665 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.665 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.666 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:54.666 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:54.666 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:54.666 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:54.666 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:54.666 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:54.666 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:54.666 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:54.666 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.666 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.666 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.666 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.666 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.666 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:54.666 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:54.667 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.667 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.667 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.667 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.667 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.667 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.667 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.667 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.667 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.667 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.667 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.667 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.667 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:54.667 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:54.668 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.668 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.668 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.668 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:54.668 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.668 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.668 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:54.668 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:54.668 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:54.668 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:54.668 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:54.668 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:54.668 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:54.668 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:54.669 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.669 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.669 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.669 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.669 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.669 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:54.669 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:54.669 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.669 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.669 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.669 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.669 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.669 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.669 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.669 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.670 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.670 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.670 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.670 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.670 INFO:teuthology.orchestra.run.smithi039.stdout: "smtp_destination": { 2024-03-18T16:20:54.670 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "smtp_destination", 2024-03-18T16:20:54.670 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.670 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.670 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.670 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.670 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.670 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.670 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.670 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Email address to send alerts to", 2024-03-18T16:20:54.671 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.671 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.671 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.671 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.671 INFO:teuthology.orchestra.run.smithi039.stdout: "smtp_from_name": { 2024-03-18T16:20:54.671 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "smtp_from_name", 2024-03-18T16:20:54.671 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.671 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.671 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.671 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "Ceph", 2024-03-18T16:20:54.671 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.671 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.671 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.671 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Email From: name", 2024-03-18T16:20:54.671 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.672 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.672 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.672 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.672 INFO:teuthology.orchestra.run.smithi039.stdout: "smtp_host": { 2024-03-18T16:20:54.672 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "smtp_host", 2024-03-18T16:20:54.672 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.672 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.672 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.672 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.672 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.672 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.672 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.672 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "SMTP server", 2024-03-18T16:20:54.672 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.672 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.673 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.673 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.673 INFO:teuthology.orchestra.run.smithi039.stdout: "smtp_password": { 2024-03-18T16:20:54.673 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "smtp_password", 2024-03-18T16:20:54.673 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.673 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.673 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.673 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.673 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.673 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.673 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.673 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Password to authenticate with", 2024-03-18T16:20:54.673 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.673 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.673 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.674 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.674 INFO:teuthology.orchestra.run.smithi039.stdout: "smtp_port": { 2024-03-18T16:20:54.674 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "smtp_port", 2024-03-18T16:20:54.674 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:54.674 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.674 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.674 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "465", 2024-03-18T16:20:54.674 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.674 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.674 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.674 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "SMTP port", 2024-03-18T16:20:54.674 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.674 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.674 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.674 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.675 INFO:teuthology.orchestra.run.smithi039.stdout: "smtp_sender": { 2024-03-18T16:20:54.675 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "smtp_sender", 2024-03-18T16:20:54.675 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.675 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.675 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.675 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.675 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.675 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.675 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.675 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "SMTP envelope sender", 2024-03-18T16:20:54.675 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.675 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.675 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.675 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.676 INFO:teuthology.orchestra.run.smithi039.stdout: "smtp_ssl": { 2024-03-18T16:20:54.676 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "smtp_ssl", 2024-03-18T16:20:54.676 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.676 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.676 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.676 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:54.676 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.676 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.676 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.676 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-18T16:20:54.676 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.676 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.676 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.676 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.676 INFO:teuthology.orchestra.run.smithi039.stdout: "smtp_user": { 2024-03-18T16:20:54.677 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "smtp_user", 2024-03-18T16:20:54.677 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.677 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.677 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.677 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.677 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.677 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.677 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.677 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "User to authenticate as", 2024-03-18T16:20:54.677 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.677 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.677 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.677 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:54.678 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:54.678 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.678 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:54.678 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "balancer", 2024-03-18T16:20:54.678 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:54.678 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:54.678 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:54.737 INFO:teuthology.orchestra.run.smithi039.stdout: "active": { 2024-03-18T16:20:54.737 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "active", 2024-03-18T16:20:54.737 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.737 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.737 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.737 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:54.738 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.738 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.738 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.738 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "automatically balance PGs across cluster", 2024-03-18T16:20:54.738 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.738 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.738 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.738 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.738 INFO:teuthology.orchestra.run.smithi039.stdout: "begin_time": { 2024-03-18T16:20:54.738 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "begin_time", 2024-03-18T16:20:54.738 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.738 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.738 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.738 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0000", 2024-03-18T16:20:54.739 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.739 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.739 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.739 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "beginning time of day to automatically balance", 2024-03-18T16:20:54.739 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-18T16:20:54.739 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.739 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.739 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.739 INFO:teuthology.orchestra.run.smithi039.stdout: "begin_weekday": { 2024-03-18T16:20:54.739 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "begin_weekday", 2024-03-18T16:20:54.739 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "uint", 2024-03-18T16:20:54.739 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.739 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.739 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-03-18T16:20:54.740 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "0", 2024-03-18T16:20:54.740 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "6", 2024-03-18T16:20:54.740 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.740 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-18T16:20:54.740 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-18T16:20:54.740 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.740 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.740 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.740 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_compat_max_iterations": { 2024-03-18T16:20:54.740 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "crush_compat_max_iterations", 2024-03-18T16:20:54.740 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "uint", 2024-03-18T16:20:54.740 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.740 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.740 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "25", 2024-03-18T16:20:54.740 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "1", 2024-03-18T16:20:54.741 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "250", 2024-03-18T16:20:54.741 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.741 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-18T16:20:54.741 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.741 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.741 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.741 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.741 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_compat_metrics": { 2024-03-18T16:20:54.741 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "crush_compat_metrics", 2024-03-18T16:20:54.741 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.741 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.741 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.741 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "pgs,objects,bytes", 2024-03-18T16:20:54.741 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.742 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.742 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.742 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-18T16:20:54.742 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-18T16:20:54.742 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.742 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.742 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.742 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_compat_step": { 2024-03-18T16:20:54.742 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "crush_compat_step", 2024-03-18T16:20:54.742 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "float", 2024-03-18T16:20:54.742 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.742 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.742 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0.5", 2024-03-18T16:20:54.742 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "0.001", 2024-03-18T16:20:54.743 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "0.999", 2024-03-18T16:20:54.743 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.743 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "aggressiveness of optimization", 2024-03-18T16:20:54.743 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-18T16:20:54.743 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.743 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.743 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.743 INFO:teuthology.orchestra.run.smithi039.stdout: "end_time": { 2024-03-18T16:20:54.743 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "end_time", 2024-03-18T16:20:54.743 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.743 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.743 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.743 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "2359", 2024-03-18T16:20:54.743 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.743 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.744 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.744 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "ending time of day to automatically balance", 2024-03-18T16:20:54.744 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-18T16:20:54.744 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.744 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.744 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.744 INFO:teuthology.orchestra.run.smithi039.stdout: "end_weekday": { 2024-03-18T16:20:54.744 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "end_weekday", 2024-03-18T16:20:54.744 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "uint", 2024-03-18T16:20:54.744 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.744 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.744 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-03-18T16:20:54.744 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "0", 2024-03-18T16:20:54.744 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "6", 2024-03-18T16:20:54.745 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.745 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-18T16:20:54.745 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-18T16:20:54.745 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.745 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.745 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.745 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:54.745 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:54.745 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.745 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.745 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.745 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.745 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.745 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.745 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:54.746 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:54.746 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:54.746 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:54.746 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:54.746 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:54.746 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:54.746 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:54.746 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.746 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.746 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.746 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.746 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.746 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:54.746 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:54.746 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.747 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.747 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.747 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.747 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.747 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.747 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.747 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.747 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.747 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.747 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.747 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.747 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:54.747 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:54.747 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.747 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.748 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.748 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:54.748 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.748 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.748 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:54.748 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:54.748 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:54.748 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:54.748 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:54.748 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:54.748 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:54.748 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:54.748 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.749 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.749 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.749 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.749 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.749 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:54.749 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:54.749 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.749 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.749 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.749 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.749 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.749 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.749 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.749 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.749 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.750 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.750 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.750 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.750 INFO:teuthology.orchestra.run.smithi039.stdout: "min_score": { 2024-03-18T16:20:54.750 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "min_score", 2024-03-18T16:20:54.750 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "float", 2024-03-18T16:20:54.750 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.750 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.750 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-03-18T16:20:54.750 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.750 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.750 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.750 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-18T16:20:54.750 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.751 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.751 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.751 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.751 INFO:teuthology.orchestra.run.smithi039.stdout: "mode": { 2024-03-18T16:20:54.751 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "mode", 2024-03-18T16:20:54.751 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.751 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.751 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.751 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "upmap", 2024-03-18T16:20:54.751 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.751 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.751 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:54.751 INFO:teuthology.orchestra.run.smithi039.stdout: "crush-compat", 2024-03-18T16:20:54.751 INFO:teuthology.orchestra.run.smithi039.stdout: "none", 2024-03-18T16:20:54.752 INFO:teuthology.orchestra.run.smithi039.stdout: "read", 2024-03-18T16:20:54.752 INFO:teuthology.orchestra.run.smithi039.stdout: "upmap", 2024-03-18T16:20:54.752 INFO:teuthology.orchestra.run.smithi039.stdout: "upmap-read" 2024-03-18T16:20:54.752 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:54.752 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Balancer mode", 2024-03-18T16:20:54.752 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.752 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.752 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.752 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.752 INFO:teuthology.orchestra.run.smithi039.stdout: "pool_ids": { 2024-03-18T16:20:54.752 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "pool_ids", 2024-03-18T16:20:54.752 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.752 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.752 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.752 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.753 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.753 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.753 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.753 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-18T16:20:54.753 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.753 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.753 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.753 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.753 INFO:teuthology.orchestra.run.smithi039.stdout: "sleep_interval": { 2024-03-18T16:20:54.753 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sleep_interval", 2024-03-18T16:20:54.753 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-03-18T16:20:54.753 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.753 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.753 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "60", 2024-03-18T16:20:54.753 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.754 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.754 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.754 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-18T16:20:54.754 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.754 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.754 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.754 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.754 INFO:teuthology.orchestra.run.smithi039.stdout: "upmap_max_deviation": { 2024-03-18T16:20:54.754 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "upmap_max_deviation", 2024-03-18T16:20:54.754 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:54.754 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.754 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.760 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "5", 2024-03-18T16:20:54.760 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "1", 2024-03-18T16:20:54.760 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.760 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.760 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-18T16:20:54.760 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-18T16:20:54.760 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.760 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.760 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.760 INFO:teuthology.orchestra.run.smithi039.stdout: "upmap_max_optimizations": { 2024-03-18T16:20:54.760 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "upmap_max_optimizations", 2024-03-18T16:20:54.760 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "uint", 2024-03-18T16:20:54.760 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.761 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.761 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "10", 2024-03-18T16:20:54.761 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.761 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.761 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.761 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-18T16:20:54.761 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.761 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.761 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.761 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:54.761 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:54.761 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.761 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:54.761 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "cephadm", 2024-03-18T16:20:54.761 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:54.762 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:54.762 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:54.762 INFO:teuthology.orchestra.run.smithi039.stdout: "agent_down_multiplier": { 2024-03-18T16:20:54.762 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "agent_down_multiplier", 2024-03-18T16:20:54.762 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "float", 2024-03-18T16:20:54.762 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.762 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.762 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "3.0", 2024-03-18T16:20:54.762 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.762 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.762 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.762 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-18T16:20:54.762 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.762 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.763 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.763 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.763 INFO:teuthology.orchestra.run.smithi039.stdout: "agent_refresh_rate": { 2024-03-18T16:20:54.763 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "agent_refresh_rate", 2024-03-18T16:20:54.763 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-03-18T16:20:54.763 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.763 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.763 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "20", 2024-03-18T16:20:54.763 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.763 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.763 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.763 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-18T16:20:54.763 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.763 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.763 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.764 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.764 INFO:teuthology.orchestra.run.smithi039.stdout: "agent_starting_port": { 2024-03-18T16:20:54.764 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "agent_starting_port", 2024-03-18T16:20:54.764 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:54.764 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.764 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.764 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "4721", 2024-03-18T16:20:54.764 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.764 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.764 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.764 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-18T16:20:54.764 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.764 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.764 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.764 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.765 INFO:teuthology.orchestra.run.smithi039.stdout: "allow_ptrace": { 2024-03-18T16:20:54.765 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "allow_ptrace", 2024-03-18T16:20:54.765 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.765 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.765 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.765 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.765 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.765 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.765 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.765 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-18T16:20:54.765 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-18T16:20:54.765 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.765 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.766 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.766 INFO:teuthology.orchestra.run.smithi039.stdout: "autotune_interval": { 2024-03-18T16:20:54.766 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "autotune_interval", 2024-03-18T16:20:54.766 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-03-18T16:20:54.766 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.766 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.766 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "600", 2024-03-18T16:20:54.766 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.766 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.766 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.766 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-18T16:20:54.766 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.766 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.766 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.767 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.767 INFO:teuthology.orchestra.run.smithi039.stdout: "autotune_memory_target_ratio": { 2024-03-18T16:20:54.767 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "autotune_memory_target_ratio", 2024-03-18T16:20:54.767 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "float", 2024-03-18T16:20:54.767 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.767 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.767 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0.7", 2024-03-18T16:20:54.767 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.767 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.767 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.767 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-18T16:20:54.767 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.767 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.767 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.767 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.768 INFO:teuthology.orchestra.run.smithi039.stdout: "cephadm_log_destination": { 2024-03-18T16:20:54.768 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "cephadm_log_destination", 2024-03-18T16:20:54.768 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.768 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.768 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.768 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.768 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.768 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.768 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:54.768 INFO:teuthology.orchestra.run.smithi039.stdout: "file", 2024-03-18T16:20:54.768 INFO:teuthology.orchestra.run.smithi039.stdout: "file,syslog", 2024-03-18T16:20:54.768 INFO:teuthology.orchestra.run.smithi039.stdout: "syslog" 2024-03-18T16:20:54.768 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:54.768 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-18T16:20:54.768 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.769 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.769 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.769 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.769 INFO:teuthology.orchestra.run.smithi039.stdout: "cgroups_split": { 2024-03-18T16:20:54.769 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "cgroups_split", 2024-03-18T16:20:54.769 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.769 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.769 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.769 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:54.769 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.769 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.769 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.769 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-18T16:20:54.769 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.770 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.770 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.770 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.770 INFO:teuthology.orchestra.run.smithi039.stdout: "config_checks_enabled": { 2024-03-18T16:20:54.770 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "config_checks_enabled", 2024-03-18T16:20:54.770 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.770 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.770 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.770 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.770 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.770 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.770 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.770 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-18T16:20:54.770 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.770 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.771 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.771 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.771 INFO:teuthology.orchestra.run.smithi039.stdout: "config_dashboard": { 2024-03-18T16:20:54.771 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "config_dashboard", 2024-03-18T16:20:54.771 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.771 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.771 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.771 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:54.771 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.771 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.771 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.771 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-18T16:20:54.771 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.771 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.771 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.772 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.772 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_alertmanager": { 2024-03-18T16:20:54.772 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_alertmanager", 2024-03-18T16:20:54.772 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.772 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.772 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.772 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-18T16:20:54.772 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.772 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.772 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.772 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Prometheus container image", 2024-03-18T16:20:54.772 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.772 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.772 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.773 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.773 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_base": { 2024-03-18T16:20:54.773 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_base", 2024-03-18T16:20:54.773 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.773 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.773 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.773 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-18T16:20:54.773 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.773 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.773 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.773 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Container image name, without the tag", 2024-03-18T16:20:54.773 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.773 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.773 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.773 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.774 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_elasticsearch": { 2024-03-18T16:20:54.774 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_elasticsearch", 2024-03-18T16:20:54.774 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.774 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.774 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.774 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-18T16:20:54.774 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.774 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.774 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.774 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "elasticsearch container image", 2024-03-18T16:20:54.774 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.774 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.780 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.780 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.780 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_grafana": { 2024-03-18T16:20:54.780 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_grafana", 2024-03-18T16:20:54.780 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.780 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.780 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.781 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-18T16:20:54.781 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.781 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.781 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.781 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Prometheus container image", 2024-03-18T16:20:54.781 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.781 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.781 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.781 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.781 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_haproxy": { 2024-03-18T16:20:54.781 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_haproxy", 2024-03-18T16:20:54.781 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.781 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.781 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.781 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-18T16:20:54.782 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.782 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.782 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.782 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "HAproxy container image", 2024-03-18T16:20:54.782 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.782 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.782 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.782 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.782 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_jaeger_agent": { 2024-03-18T16:20:54.782 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_jaeger_agent", 2024-03-18T16:20:54.782 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.782 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.782 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.782 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-18T16:20:54.783 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.783 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.783 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.783 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Jaeger agent container image", 2024-03-18T16:20:54.783 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.783 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.783 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.783 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.783 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_jaeger_collector": { 2024-03-18T16:20:54.783 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_jaeger_collector", 2024-03-18T16:20:54.783 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.783 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.783 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.784 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-18T16:20:54.784 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.784 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.784 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.784 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Jaeger collector container image", 2024-03-18T16:20:54.784 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.784 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.784 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.784 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.784 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_jaeger_query": { 2024-03-18T16:20:54.784 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_jaeger_query", 2024-03-18T16:20:54.784 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.784 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.784 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.785 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-18T16:20:54.785 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.785 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.785 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.785 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Jaeger query container image", 2024-03-18T16:20:54.785 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.785 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.785 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.785 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.785 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_keepalived": { 2024-03-18T16:20:54.785 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_keepalived", 2024-03-18T16:20:54.785 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.785 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.785 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.785 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-18T16:20:54.786 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.786 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.786 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.786 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Keepalived container image", 2024-03-18T16:20:54.786 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.786 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.786 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.786 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.786 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_loki": { 2024-03-18T16:20:54.786 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_loki", 2024-03-18T16:20:54.786 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.786 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.786 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.787 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-18T16:20:54.787 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.787 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.787 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.787 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Loki container image", 2024-03-18T16:20:54.787 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.787 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.787 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.787 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.787 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_node_exporter": { 2024-03-18T16:20:54.787 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_node_exporter", 2024-03-18T16:20:54.787 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.787 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.787 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.787 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-18T16:20:54.788 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.788 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.788 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.788 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Prometheus container image", 2024-03-18T16:20:54.788 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.788 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.788 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.788 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.788 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_nvmeof": { 2024-03-18T16:20:54.788 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_nvmeof", 2024-03-18T16:20:54.788 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.788 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.788 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.788 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-18T16:20:54.788 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.789 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.789 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.789 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Nvme-of container image", 2024-03-18T16:20:54.789 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.789 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.789 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.789 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.789 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_prometheus": { 2024-03-18T16:20:54.789 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_prometheus", 2024-03-18T16:20:54.789 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.789 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.789 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.789 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-18T16:20:54.789 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.789 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.790 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.790 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Prometheus container image", 2024-03-18T16:20:54.790 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.790 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.790 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.790 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.790 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_promtail": { 2024-03-18T16:20:54.790 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_promtail", 2024-03-18T16:20:54.790 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.790 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.790 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.790 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-18T16:20:54.790 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.790 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.791 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.791 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Promtail container image", 2024-03-18T16:20:54.791 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.791 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.791 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.791 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.791 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_snmp_gateway": { 2024-03-18T16:20:54.791 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_snmp_gateway", 2024-03-18T16:20:54.791 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.791 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.791 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.791 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-18T16:20:54.791 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.791 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.792 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.792 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "SNMP Gateway container image", 2024-03-18T16:20:54.792 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.792 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.792 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.792 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.792 INFO:teuthology.orchestra.run.smithi039.stdout: "container_init": { 2024-03-18T16:20:54.792 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_init", 2024-03-18T16:20:54.792 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.792 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.792 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.792 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:54.792 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.792 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.792 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.793 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Run podman/docker with `--init`", 2024-03-18T16:20:54.793 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.793 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.793 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.793 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.793 INFO:teuthology.orchestra.run.smithi039.stdout: "daemon_cache_timeout": { 2024-03-18T16:20:54.793 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "daemon_cache_timeout", 2024-03-18T16:20:54.793 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-03-18T16:20:54.793 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.793 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.793 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "600", 2024-03-18T16:20:54.793 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.793 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.793 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.793 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-18T16:20:54.794 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.794 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.794 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.794 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.794 INFO:teuthology.orchestra.run.smithi039.stdout: "default_cephadm_command_timeout": { 2024-03-18T16:20:54.794 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "default_cephadm_command_timeout", 2024-03-18T16:20:54.794 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:54.794 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.794 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.794 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "900", 2024-03-18T16:20:54.794 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.794 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.794 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.794 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-18T16:20:54.795 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.795 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.795 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.795 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.795 INFO:teuthology.orchestra.run.smithi039.stdout: "default_registry": { 2024-03-18T16:20:54.795 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "default_registry", 2024-03-18T16:20:54.795 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.795 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.795 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.795 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "docker.io", 2024-03-18T16:20:54.795 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.795 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.795 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.795 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-18T16:20:54.795 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.796 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.796 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.796 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.796 INFO:teuthology.orchestra.run.smithi039.stdout: "device_cache_timeout": { 2024-03-18T16:20:54.796 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "device_cache_timeout", 2024-03-18T16:20:54.796 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-03-18T16:20:54.796 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.796 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.796 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "1800", 2024-03-18T16:20:54.796 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.796 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.796 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.796 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "seconds to cache device inventory", 2024-03-18T16:20:54.796 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.797 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.797 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.797 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.797 INFO:teuthology.orchestra.run.smithi039.stdout: "device_enhanced_scan": { 2024-03-18T16:20:54.797 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "device_enhanced_scan", 2024-03-18T16:20:54.797 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.797 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.797 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.797 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.797 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.797 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.797 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.797 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-18T16:20:54.797 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.797 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.798 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.798 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.798 INFO:teuthology.orchestra.run.smithi039.stdout: "facts_cache_timeout": { 2024-03-18T16:20:54.798 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "facts_cache_timeout", 2024-03-18T16:20:54.798 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-03-18T16:20:54.798 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.798 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.798 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "60", 2024-03-18T16:20:54.798 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.798 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.798 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.798 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "seconds to cache host facts data", 2024-03-18T16:20:54.798 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.799 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.799 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.799 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.799 INFO:teuthology.orchestra.run.smithi039.stdout: "host_check_interval": { 2024-03-18T16:20:54.799 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "host_check_interval", 2024-03-18T16:20:54.799 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-03-18T16:20:54.799 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.799 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.799 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "600", 2024-03-18T16:20:54.799 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.799 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.799 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.799 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "how frequently to perform a host check", 2024-03-18T16:20:54.799 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.799 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.800 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.800 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.800 INFO:teuthology.orchestra.run.smithi039.stdout: "hw_monitoring": { 2024-03-18T16:20:54.800 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "hw_monitoring", 2024-03-18T16:20:54.800 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.800 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.800 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.800 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.800 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.800 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.800 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.800 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-18T16:20:54.800 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.800 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.800 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.801 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.801 INFO:teuthology.orchestra.run.smithi039.stdout: "inventory_list_all": { 2024-03-18T16:20:54.801 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "inventory_list_all", 2024-03-18T16:20:54.801 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.801 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.801 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.801 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.801 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.801 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.801 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.801 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-18T16:20:54.801 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.801 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.801 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.801 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.802 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:54.802 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:54.802 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.802 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.802 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.802 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.802 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.802 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.802 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:54.802 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:54.802 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:54.802 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:54.802 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:54.802 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:54.802 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:54.803 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:54.803 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.803 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.803 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.803 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.803 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.803 INFO:teuthology.orchestra.run.smithi039.stdout: "log_refresh_metadata": { 2024-03-18T16:20:54.803 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_refresh_metadata", 2024-03-18T16:20:54.803 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.803 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.803 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.803 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.803 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.803 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.803 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.804 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-18T16:20:54.804 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.804 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.804 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.804 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.804 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:54.804 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:54.804 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.804 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.804 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.804 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:54.804 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.804 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.804 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.804 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-18T16:20:54.805 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.805 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.805 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.805 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.805 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:54.805 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:54.805 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.805 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.805 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.805 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:54.805 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.805 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.805 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:54.805 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:54.805 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:54.806 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:54.806 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:54.806 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:54.806 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:54.806 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:54.806 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.806 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.806 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.806 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.806 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.806 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:54.806 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:54.806 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.806 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.807 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.807 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.807 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.807 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.807 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.807 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.807 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.807 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.807 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.807 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.808 INFO:teuthology.orchestra.run.smithi039.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-18T16:20:54.808 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-18T16:20:54.808 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.808 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.808 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.808 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.808 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.808 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.808 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.808 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-18T16:20:54.808 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.808 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.808 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.808 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.809 INFO:teuthology.orchestra.run.smithi039.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-18T16:20:54.809 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-18T16:20:54.809 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.809 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.809 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.809 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "*", 2024-03-18T16:20:54.809 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.809 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.809 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.809 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-18T16:20:54.809 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.809 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.810 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.810 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.810 INFO:teuthology.orchestra.run.smithi039.stdout: "max_count_per_host": { 2024-03-18T16:20:54.810 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "max_count_per_host", 2024-03-18T16:20:54.810 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:54.810 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.810 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.810 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "10", 2024-03-18T16:20:54.810 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.810 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.810 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.810 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "max number of daemons per service per host", 2024-03-18T16:20:54.810 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.810 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.811 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.811 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.811 INFO:teuthology.orchestra.run.smithi039.stdout: "max_osd_draining_count": { 2024-03-18T16:20:54.811 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "max_osd_draining_count", 2024-03-18T16:20:54.811 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:54.811 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.811 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.811 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "10", 2024-03-18T16:20:54.811 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.811 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.811 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.811 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-18T16:20:54.811 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.811 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.811 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.812 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.812 INFO:teuthology.orchestra.run.smithi039.stdout: "migration_current": { 2024-03-18T16:20:54.812 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "migration_current", 2024-03-18T16:20:54.812 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:54.812 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.812 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.812 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.812 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.812 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.812 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.812 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "internal - do not modify", 2024-03-18T16:20:54.812 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.812 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.812 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.812 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.813 INFO:teuthology.orchestra.run.smithi039.stdout: "mode": { 2024-03-18T16:20:54.813 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "mode", 2024-03-18T16:20:54.813 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.813 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.813 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.813 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "root", 2024-03-18T16:20:54.813 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.813 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.813 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:54.813 INFO:teuthology.orchestra.run.smithi039.stdout: "cephadm-package", 2024-03-18T16:20:54.813 INFO:teuthology.orchestra.run.smithi039.stdout: "root" 2024-03-18T16:20:54.813 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:54.813 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "mode for remote execution of cephadm", 2024-03-18T16:20:54.813 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.813 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.814 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.814 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.814 INFO:teuthology.orchestra.run.smithi039.stdout: "oob_default_addr": { 2024-03-18T16:20:54.814 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "oob_default_addr", 2024-03-18T16:20:54.814 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.814 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.814 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.814 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "169.254.1.1", 2024-03-18T16:20:54.814 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.814 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.814 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.814 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-18T16:20:54.814 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.814 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.815 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.815 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.815 INFO:teuthology.orchestra.run.smithi039.stdout: "prometheus_alerts_path": { 2024-03-18T16:20:54.815 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "prometheus_alerts_path", 2024-03-18T16:20:54.815 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.815 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.815 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.815 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-18T16:20:54.815 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.815 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.815 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.815 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-18T16:20:54.815 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.815 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.816 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.816 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.816 INFO:teuthology.orchestra.run.smithi039.stdout: "registry_insecure": { 2024-03-18T16:20:54.816 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "registry_insecure", 2024-03-18T16:20:54.816 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.816 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.816 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.816 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.816 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.816 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.816 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.816 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-18T16:20:54.816 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.816 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.816 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.817 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.817 INFO:teuthology.orchestra.run.smithi039.stdout: "registry_password": { 2024-03-18T16:20:54.817 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "registry_password", 2024-03-18T16:20:54.817 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.817 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.817 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.817 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.817 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.817 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.817 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.817 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-18T16:20:54.817 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.817 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.817 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.817 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.817 INFO:teuthology.orchestra.run.smithi039.stdout: "registry_url": { 2024-03-18T16:20:54.818 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "registry_url", 2024-03-18T16:20:54.818 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.818 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.818 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.818 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.818 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.818 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.818 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.818 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-18T16:20:54.818 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.818 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.818 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.818 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.818 INFO:teuthology.orchestra.run.smithi039.stdout: "registry_username": { 2024-03-18T16:20:54.818 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "registry_username", 2024-03-18T16:20:54.819 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.819 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.819 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.819 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.819 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.819 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.819 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.819 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-18T16:20:54.819 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.819 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.819 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.819 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.819 INFO:teuthology.orchestra.run.smithi039.stdout: "secure_monitoring_stack": { 2024-03-18T16:20:54.819 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "secure_monitoring_stack", 2024-03-18T16:20:54.820 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.820 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.820 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.820 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.820 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.820 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.820 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.820 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-18T16:20:54.820 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.820 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.820 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.820 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.820 INFO:teuthology.orchestra.run.smithi039.stdout: "service_discovery_port": { 2024-03-18T16:20:54.820 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "service_discovery_port", 2024-03-18T16:20:54.820 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:54.821 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.821 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.821 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "8765", 2024-03-18T16:20:54.821 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.821 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.821 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.821 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "cephadm service discovery port", 2024-03-18T16:20:54.821 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.821 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.821 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.821 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.821 INFO:teuthology.orchestra.run.smithi039.stdout: "ssh_config_file": { 2024-03-18T16:20:54.821 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "ssh_config_file", 2024-03-18T16:20:54.821 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.821 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.821 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.822 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.822 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.822 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.822 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.822 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-18T16:20:54.822 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.822 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.822 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.822 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.822 INFO:teuthology.orchestra.run.smithi039.stdout: "use_agent": { 2024-03-18T16:20:54.823 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "use_agent", 2024-03-18T16:20:54.823 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.823 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.823 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.823 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.823 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.823 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.823 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.823 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-18T16:20:54.823 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.823 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.824 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.824 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.824 INFO:teuthology.orchestra.run.smithi039.stdout: "use_repo_digest": { 2024-03-18T16:20:54.824 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "use_repo_digest", 2024-03-18T16:20:54.824 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.824 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.824 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.824 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:54.824 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.824 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.825 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.825 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-18T16:20:54.825 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.825 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.825 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.825 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.825 INFO:teuthology.orchestra.run.smithi039.stdout: "warn_on_failed_host_check": { 2024-03-18T16:20:54.825 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "warn_on_failed_host_check", 2024-03-18T16:20:54.825 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.825 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.826 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.826 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:54.826 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.826 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.826 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.826 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "raise a health warning if the host check fails", 2024-03-18T16:20:54.826 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.826 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.826 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.826 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.826 INFO:teuthology.orchestra.run.smithi039.stdout: "warn_on_stray_daemons": { 2024-03-18T16:20:54.826 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "warn_on_stray_daemons", 2024-03-18T16:20:54.827 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.827 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.827 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.827 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:54.827 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.827 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.827 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.827 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-18T16:20:54.827 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.827 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.827 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.827 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.827 INFO:teuthology.orchestra.run.smithi039.stdout: "warn_on_stray_hosts": { 2024-03-18T16:20:54.827 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "warn_on_stray_hosts", 2024-03-18T16:20:54.827 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.827 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.828 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.828 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:54.828 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.828 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.828 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.828 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-18T16:20:54.828 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.828 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.828 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.828 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:54.828 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:54.828 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.828 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:54.828 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "crash", 2024-03-18T16:20:54.828 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:54.829 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:54.829 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:54.829 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:54.829 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:54.829 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.829 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.829 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.829 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.829 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.829 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.829 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:54.829 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:54.829 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:54.829 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:54.829 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:54.830 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:54.830 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:54.830 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:54.830 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.830 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.830 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.830 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.830 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.830 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:54.830 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:54.830 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.830 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.830 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.830 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.830 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.831 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.831 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.831 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.831 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.831 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.831 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.831 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.831 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:54.831 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:54.831 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.831 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.831 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.831 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:54.831 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.831 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.832 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:54.832 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:54.832 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:54.832 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:54.832 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:54.832 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:54.832 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:54.832 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:54.832 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.832 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.832 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.832 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.832 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.832 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:54.833 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:54.833 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.833 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.833 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.833 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.833 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.833 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.833 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.833 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.833 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.833 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.833 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.833 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.833 INFO:teuthology.orchestra.run.smithi039.stdout: "retain_interval": { 2024-03-18T16:20:54.833 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "retain_interval", 2024-03-18T16:20:54.834 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-03-18T16:20:54.834 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.834 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.834 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "31536000", 2024-03-18T16:20:54.834 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.834 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.834 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.834 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-18T16:20:54.834 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.834 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.834 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.834 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.834 INFO:teuthology.orchestra.run.smithi039.stdout: "warn_recent_interval": { 2024-03-18T16:20:54.835 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "warn_recent_interval", 2024-03-18T16:20:54.835 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-03-18T16:20:54.835 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.835 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.835 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "1209600", 2024-03-18T16:20:54.835 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.835 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.835 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.835 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-18T16:20:54.835 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.835 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.835 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.835 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:54.835 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:54.835 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.836 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:54.836 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "dashboard", 2024-03-18T16:20:54.836 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:54.836 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:54.836 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:54.836 INFO:teuthology.orchestra.run.smithi039.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-18T16:20:54.836 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-18T16:20:54.836 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:54.836 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.836 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.836 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "10", 2024-03-18T16:20:54.836 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.836 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.837 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.837 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.837 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.837 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.837 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.837 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.837 INFO:teuthology.orchestra.run.smithi039.stdout: "ALERTMANAGER_API_HOST": { 2024-03-18T16:20:54.837 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-18T16:20:54.837 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.837 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.837 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.837 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.837 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.838 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.838 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.838 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.838 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.838 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.838 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.838 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.838 INFO:teuthology.orchestra.run.smithi039.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-18T16:20:54.838 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-18T16:20:54.838 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.838 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.838 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.838 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:54.838 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.839 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.839 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.839 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.839 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.839 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.839 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.839 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.839 INFO:teuthology.orchestra.run.smithi039.stdout: "AUDIT_API_ENABLED": { 2024-03-18T16:20:54.839 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "AUDIT_API_ENABLED", 2024-03-18T16:20:54.839 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.839 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.839 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.839 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.839 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.839 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.839 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.840 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.840 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.840 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.840 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.840 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.840 INFO:teuthology.orchestra.run.smithi039.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-18T16:20:54.840 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-18T16:20:54.840 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.840 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.840 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.840 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:54.847 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.847 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.847 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.847 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.847 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.847 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.847 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.847 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.847 INFO:teuthology.orchestra.run.smithi039.stdout: "ENABLE_BROWSABLE_API": { 2024-03-18T16:20:54.847 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-18T16:20:54.847 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.848 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.848 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.848 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:54.848 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.848 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.848 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.848 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.848 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.848 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.848 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.848 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.848 INFO:teuthology.orchestra.run.smithi039.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-18T16:20:54.848 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-18T16:20:54.848 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.848 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.849 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.849 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:54.849 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.849 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.849 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.849 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.849 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.849 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.849 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.849 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.849 INFO:teuthology.orchestra.run.smithi039.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-18T16:20:54.849 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-18T16:20:54.849 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.849 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.850 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.850 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:54.850 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.850 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.850 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.850 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.850 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.850 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.850 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.850 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.850 INFO:teuthology.orchestra.run.smithi039.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-18T16:20:54.850 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-18T16:20:54.850 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.850 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.850 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.851 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:54.851 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.851 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.851 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.851 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.851 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.851 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.851 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.851 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.851 INFO:teuthology.orchestra.run.smithi039.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-18T16:20:54.851 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-18T16:20:54.851 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.851 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.851 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.852 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:54.852 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.852 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.852 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.852 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.852 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.852 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.852 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.852 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.852 INFO:teuthology.orchestra.run.smithi039.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-18T16:20:54.852 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-18T16:20:54.852 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.852 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.852 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.852 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:54.853 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.853 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.853 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.853 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.853 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.853 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.853 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.853 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.853 INFO:teuthology.orchestra.run.smithi039.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-18T16:20:54.853 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-18T16:20:54.853 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.853 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.853 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.853 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:54.853 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.854 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.854 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.854 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.854 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.854 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.854 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.854 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.854 INFO:teuthology.orchestra.run.smithi039.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-18T16:20:54.854 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-18T16:20:54.854 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.854 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.854 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.854 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:54.854 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.855 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.855 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.855 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.855 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.855 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.855 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.855 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.855 INFO:teuthology.orchestra.run.smithi039.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-18T16:20:54.855 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-18T16:20:54.855 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.855 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.855 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.855 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.855 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.855 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.856 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.856 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.856 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.856 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.856 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.856 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.856 INFO:teuthology.orchestra.run.smithi039.stdout: "GRAFANA_API_PASSWORD": { 2024-03-18T16:20:54.856 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-18T16:20:54.856 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.856 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.856 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.856 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "admin", 2024-03-18T16:20:54.856 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.856 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.856 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.857 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.857 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.857 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.857 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.857 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.857 INFO:teuthology.orchestra.run.smithi039.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-18T16:20:54.857 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-18T16:20:54.857 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.857 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.857 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.857 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:54.857 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.857 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.857 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.857 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.858 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.858 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.858 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.858 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.858 INFO:teuthology.orchestra.run.smithi039.stdout: "GRAFANA_API_URL": { 2024-03-18T16:20:54.858 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "GRAFANA_API_URL", 2024-03-18T16:20:54.858 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.858 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.858 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.858 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.858 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.858 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.858 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.858 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.858 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.859 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.859 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.859 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.859 INFO:teuthology.orchestra.run.smithi039.stdout: "GRAFANA_API_USERNAME": { 2024-03-18T16:20:54.859 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-18T16:20:54.859 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.859 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.859 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.859 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "admin", 2024-03-18T16:20:54.859 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.859 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.859 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.859 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.859 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.859 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.860 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.860 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.860 INFO:teuthology.orchestra.run.smithi039.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-18T16:20:54.860 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-18T16:20:54.860 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.860 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.860 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.860 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.860 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.860 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.860 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.860 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.860 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.860 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.860 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.860 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.861 INFO:teuthology.orchestra.run.smithi039.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-18T16:20:54.861 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-18T16:20:54.861 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.861 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.861 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.861 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.861 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.861 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.861 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.861 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.861 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.861 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.861 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.862 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.862 INFO:teuthology.orchestra.run.smithi039.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-18T16:20:54.862 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-18T16:20:54.862 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.862 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.862 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.862 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:54.862 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.862 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.862 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.862 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.862 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.862 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.862 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.862 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.863 INFO:teuthology.orchestra.run.smithi039.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-18T16:20:54.863 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-18T16:20:54.863 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.863 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.863 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.863 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.863 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.863 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.863 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.863 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.863 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.863 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.863 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.863 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.864 INFO:teuthology.orchestra.run.smithi039.stdout: "MULTICLUSTER_CONFIG": { 2024-03-18T16:20:54.864 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-18T16:20:54.864 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.864 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.864 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.864 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "{}", 2024-03-18T16:20:54.864 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.864 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.864 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.864 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.864 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.864 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.864 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.865 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.865 INFO:teuthology.orchestra.run.smithi039.stdout: "PROMETHEUS_API_HOST": { 2024-03-18T16:20:54.865 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-18T16:20:54.865 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.865 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.865 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.865 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.865 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.865 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.865 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.865 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.865 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.865 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.865 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.865 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.866 INFO:teuthology.orchestra.run.smithi039.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-18T16:20:54.866 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-18T16:20:54.866 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.866 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.866 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.866 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:54.866 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.866 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.866 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.866 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.866 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.866 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.866 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.866 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.866 INFO:teuthology.orchestra.run.smithi039.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-18T16:20:54.867 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-18T16:20:54.867 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.867 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.867 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.867 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.867 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.867 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.867 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.867 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.867 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.867 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.867 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.867 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.867 INFO:teuthology.orchestra.run.smithi039.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-18T16:20:54.867 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-18T16:20:54.868 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.868 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.868 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.868 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.868 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.868 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.868 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.868 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.868 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.868 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.868 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.868 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.868 INFO:teuthology.orchestra.run.smithi039.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-18T16:20:54.868 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-18T16:20:54.868 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.868 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.869 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.869 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:54.869 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.869 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.869 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.869 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.869 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.869 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.869 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.869 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.869 INFO:teuthology.orchestra.run.smithi039.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-18T16:20:54.869 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-18T16:20:54.869 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.869 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.870 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.870 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:54.870 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.870 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.870 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.913 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.913 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.914 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.914 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.914 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.914 INFO:teuthology.orchestra.run.smithi039.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-18T16:20:54.914 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-18T16:20:54.914 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.914 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.914 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.914 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.914 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.914 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.914 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.914 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.914 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.915 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.915 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.915 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.915 INFO:teuthology.orchestra.run.smithi039.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-18T16:20:54.915 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-18T16:20:54.915 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.915 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.915 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.915 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.915 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.915 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.915 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.915 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.916 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.916 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.916 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.916 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.916 INFO:teuthology.orchestra.run.smithi039.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-18T16:20:54.916 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-18T16:20:54.916 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.916 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.916 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.916 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.916 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.916 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.916 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.916 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.916 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.917 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.917 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.917 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.917 INFO:teuthology.orchestra.run.smithi039.stdout: "PWD_POLICY_ENABLED": { 2024-03-18T16:20:54.917 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-18T16:20:54.917 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.917 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.917 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.917 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:54.917 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.917 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.917 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.917 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.917 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.917 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.917 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.918 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.918 INFO:teuthology.orchestra.run.smithi039.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-18T16:20:54.918 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-18T16:20:54.918 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.918 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.918 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.918 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-18T16:20:54.918 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.918 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.918 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.918 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.918 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.918 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.918 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.919 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.919 INFO:teuthology.orchestra.run.smithi039.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-18T16:20:54.919 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-18T16:20:54.919 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:54.919 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.919 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.919 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "10", 2024-03-18T16:20:54.919 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.919 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.919 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.919 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.919 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.919 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.919 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.919 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.920 INFO:teuthology.orchestra.run.smithi039.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-18T16:20:54.920 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-18T16:20:54.920 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:54.920 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.920 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.920 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "8", 2024-03-18T16:20:54.920 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.920 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.920 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.920 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.920 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.920 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.920 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.920 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.921 INFO:teuthology.orchestra.run.smithi039.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-18T16:20:54.921 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-18T16:20:54.921 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:54.921 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.921 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.921 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "45", 2024-03-18T16:20:54.921 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.921 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.921 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.921 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.921 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.921 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.921 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.921 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.921 INFO:teuthology.orchestra.run.smithi039.stdout: "RGW_API_ACCESS_KEY": { 2024-03-18T16:20:54.921 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-18T16:20:54.922 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.922 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.922 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.922 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.922 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.922 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.922 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.922 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.922 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.922 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.922 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.922 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.922 INFO:teuthology.orchestra.run.smithi039.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-18T16:20:54.922 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-18T16:20:54.923 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.923 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.923 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.923 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "admin", 2024-03-18T16:20:54.923 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.923 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.923 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.923 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.923 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.923 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.923 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.923 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.923 INFO:teuthology.orchestra.run.smithi039.stdout: "RGW_API_SECRET_KEY": { 2024-03-18T16:20:54.923 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-18T16:20:54.923 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.924 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.924 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.924 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.924 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.924 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.924 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.924 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.924 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.924 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.946 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.946 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.946 INFO:teuthology.orchestra.run.smithi039.stdout: "RGW_API_SSL_VERIFY": { 2024-03-18T16:20:54.946 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-18T16:20:54.946 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.947 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.947 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.947 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:54.947 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.947 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.947 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.947 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.947 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.947 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.947 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.947 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.947 INFO:teuthology.orchestra.run.smithi039.stdout: "UNSAFE_TLS_v1_2": { 2024-03-18T16:20:54.947 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-18T16:20:54.947 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.948 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.948 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.948 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.948 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.948 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.948 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.948 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.948 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.948 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.948 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.948 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.948 INFO:teuthology.orchestra.run.smithi039.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-18T16:20:54.948 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-18T16:20:54.948 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:54.948 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.949 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.949 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-03-18T16:20:54.949 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.949 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.949 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.949 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.949 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.949 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.949 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.949 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.949 INFO:teuthology.orchestra.run.smithi039.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-18T16:20:54.949 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-18T16:20:54.949 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:54.949 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.949 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.950 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "10", 2024-03-18T16:20:54.950 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.950 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.950 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.950 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.950 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.950 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.950 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.950 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.950 INFO:teuthology.orchestra.run.smithi039.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-18T16:20:54.950 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-18T16:20:54.950 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:54.950 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.950 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.950 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "5", 2024-03-18T16:20:54.951 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.951 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.951 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.951 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.951 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.951 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.951 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.951 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.951 INFO:teuthology.orchestra.run.smithi039.stdout: "cross_origin_url": { 2024-03-18T16:20:54.951 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "cross_origin_url", 2024-03-18T16:20:54.951 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.951 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.951 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.951 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.951 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.952 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.952 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.952 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.952 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.952 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.952 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.952 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.952 INFO:teuthology.orchestra.run.smithi039.stdout: "crt_file": { 2024-03-18T16:20:54.952 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "crt_file", 2024-03-18T16:20:54.952 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.952 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.952 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.952 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.952 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.952 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.953 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.953 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.953 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.953 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.953 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.953 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.953 INFO:teuthology.orchestra.run.smithi039.stdout: "debug": { 2024-03-18T16:20:54.953 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "debug", 2024-03-18T16:20:54.953 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.953 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.953 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.953 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.953 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.953 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.954 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.954 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Enable/disable debug options", 2024-03-18T16:20:54.954 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.954 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.954 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.954 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.954 INFO:teuthology.orchestra.run.smithi039.stdout: "jwt_token_ttl": { 2024-03-18T16:20:54.954 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "jwt_token_ttl", 2024-03-18T16:20:54.954 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:54.954 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.954 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.954 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "28800", 2024-03-18T16:20:54.954 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.954 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.955 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.955 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.955 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.955 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.955 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.955 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.955 INFO:teuthology.orchestra.run.smithi039.stdout: "key_file": { 2024-03-18T16:20:54.955 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "key_file", 2024-03-18T16:20:54.955 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.955 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.955 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.955 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.955 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.955 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.956 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.956 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.956 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.956 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.956 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.956 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.956 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:54.956 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:54.956 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.956 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.956 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.956 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.956 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.956 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.956 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:54.957 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:54.957 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:54.957 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:54.957 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:54.957 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:54.957 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:54.957 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:54.957 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.957 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.957 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.957 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.957 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.957 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:54.957 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:54.957 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.958 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.958 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.958 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.958 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.958 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.958 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.958 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.958 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.958 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.958 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.958 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.958 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:54.958 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:54.958 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.959 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.959 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.959 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:54.959 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.959 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.959 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:54.959 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:54.959 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:54.959 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:54.960 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:54.960 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:54.960 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:54.960 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:54.960 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.960 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.960 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.960 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.960 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.961 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:54.961 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:54.961 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.961 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.961 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.961 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.961 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.961 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.962 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.962 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.962 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.962 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.962 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.962 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.962 INFO:teuthology.orchestra.run.smithi039.stdout: "motd": { 2024-03-18T16:20:54.962 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "motd", 2024-03-18T16:20:54.963 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.963 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.963 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.963 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.963 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.963 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.963 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.963 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "The message of the day", 2024-03-18T16:20:54.963 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.963 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.963 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.963 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.963 INFO:teuthology.orchestra.run.smithi039.stdout: "redirect_resolve_ip_addr": { 2024-03-18T16:20:54.964 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "redirect_resolve_ip_addr", 2024-03-18T16:20:54.964 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.964 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.964 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.964 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.964 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.964 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.964 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.964 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.964 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.964 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.964 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.964 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.964 INFO:teuthology.orchestra.run.smithi039.stdout: "server_addr": { 2024-03-18T16:20:54.964 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "server_addr", 2024-03-18T16:20:54.965 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.965 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.965 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.965 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "::", 2024-03-18T16:20:54.965 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.965 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.965 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.965 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.965 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.965 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.965 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.965 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.965 INFO:teuthology.orchestra.run.smithi039.stdout: "server_port": { 2024-03-18T16:20:54.965 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "server_port", 2024-03-18T16:20:54.966 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:54.966 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.966 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.966 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "8080", 2024-03-18T16:20:54.966 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.966 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.966 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.966 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.966 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.966 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.966 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.966 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.966 INFO:teuthology.orchestra.run.smithi039.stdout: "ssl": { 2024-03-18T16:20:54.966 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "ssl", 2024-03-18T16:20:54.966 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.967 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.967 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.967 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:54.967 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.967 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.967 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.967 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.967 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.967 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.967 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.967 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.967 INFO:teuthology.orchestra.run.smithi039.stdout: "ssl_server_port": { 2024-03-18T16:20:54.967 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "ssl_server_port", 2024-03-18T16:20:54.967 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:54.967 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.967 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.968 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "8443", 2024-03-18T16:20:54.968 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.968 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.968 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.968 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.968 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.968 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.968 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.968 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.968 INFO:teuthology.orchestra.run.smithi039.stdout: "standby_behaviour": { 2024-03-18T16:20:54.968 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "standby_behaviour", 2024-03-18T16:20:54.968 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.968 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.968 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.968 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "redirect", 2024-03-18T16:20:54.969 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.969 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.969 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:54.969 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:54.969 INFO:teuthology.orchestra.run.smithi039.stdout: "redirect" 2024-03-18T16:20:54.969 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:54.969 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.969 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.969 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.969 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.969 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.969 INFO:teuthology.orchestra.run.smithi039.stdout: "standby_error_status_code": { 2024-03-18T16:20:54.969 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "standby_error_status_code", 2024-03-18T16:20:54.969 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:54.970 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.970 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.970 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "500", 2024-03-18T16:20:54.970 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "400", 2024-03-18T16:20:54.970 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "599", 2024-03-18T16:20:54.970 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.970 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.970 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.970 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.970 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.970 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.970 INFO:teuthology.orchestra.run.smithi039.stdout: "url_prefix": { 2024-03-18T16:20:54.970 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "url_prefix", 2024-03-18T16:20:54.970 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.971 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.971 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:54.971 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.971 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.971 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.971 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.971 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.971 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.971 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.971 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.971 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:54.971 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:54.971 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.971 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:54.971 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "devicehealth", 2024-03-18T16:20:54.972 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:54.972 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:54.972 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:54.972 INFO:teuthology.orchestra.run.smithi039.stdout: "enable_monitoring": { 2024-03-18T16:20:54.972 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "enable_monitoring", 2024-03-18T16:20:54.972 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.972 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.972 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.972 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:54.972 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.972 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.972 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.972 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "monitor device health metrics", 2024-03-18T16:20:54.972 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.972 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.973 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.973 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.973 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:54.973 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:54.973 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.973 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.973 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.973 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:54.973 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.973 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.973 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:54.973 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:54.973 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:54.973 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:54.973 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:54.974 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:54.974 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:54.974 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:54.974 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.974 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.974 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.974 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.974 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.974 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:54.974 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:54.974 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.974 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.974 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.974 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.974 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.975 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.975 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.975 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.975 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.975 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.975 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.975 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.975 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:54.975 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:54.975 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.975 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.975 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.975 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:54.975 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.975 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.976 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:54.976 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:54.976 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:54.976 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:54.976 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:54.976 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:54.976 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:54.976 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:54.976 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.976 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.976 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.976 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.976 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.976 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:54.976 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:54.976 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.977 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.977 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.977 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:54.977 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.977 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.977 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.977 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:54.977 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.977 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.977 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.977 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.977 INFO:teuthology.orchestra.run.smithi039.stdout: "mark_out_threshold": { 2024-03-18T16:20:54.977 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "mark_out_threshold", 2024-03-18T16:20:54.977 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-03-18T16:20:54.977 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.978 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.978 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "2419200", 2024-03-18T16:20:54.978 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.978 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.978 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.978 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-18T16:20:54.978 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.978 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.978 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.978 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.978 INFO:teuthology.orchestra.run.smithi039.stdout: "pool_name": { 2024-03-18T16:20:54.978 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "pool_name", 2024-03-18T16:20:54.978 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:54.978 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.978 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.979 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "device_health_metrics", 2024-03-18T16:20:54.979 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.979 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.979 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.979 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-18T16:20:54.979 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.979 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.979 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.979 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.979 INFO:teuthology.orchestra.run.smithi039.stdout: "retention_period": { 2024-03-18T16:20:54.979 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "retention_period", 2024-03-18T16:20:54.979 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-03-18T16:20:54.979 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.979 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.979 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "15552000", 2024-03-18T16:20:54.980 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.980 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.980 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.980 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "how long to retain device health metrics", 2024-03-18T16:20:54.980 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.980 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.980 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.980 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.980 INFO:teuthology.orchestra.run.smithi039.stdout: "scrape_frequency": { 2024-03-18T16:20:54.980 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "scrape_frequency", 2024-03-18T16:20:54.980 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-03-18T16:20:54.980 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.980 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.980 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "86400", 2024-03-18T16:20:54.980 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.981 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.981 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.981 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-18T16:20:54.981 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.981 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.981 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.981 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.981 INFO:teuthology.orchestra.run.smithi039.stdout: "self_heal": { 2024-03-18T16:20:54.981 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "self_heal", 2024-03-18T16:20:54.981 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:54.981 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.981 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.981 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:54.981 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.981 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.981 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.982 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-18T16:20:54.982 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.982 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:54.982 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:54.982 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:54.982 INFO:teuthology.orchestra.run.smithi039.stdout: "sleep_interval": { 2024-03-18T16:20:54.982 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sleep_interval", 2024-03-18T16:20:54.982 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-03-18T16:20:54.982 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:54.982 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:54.982 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "600", 2024-03-18T16:20:54.982 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:54.982 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:54.982 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:54.983 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "how frequently to wake up and check device health", 2024-03-18T16:20:54.983 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:54.983 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.001 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.011 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.011 INFO:teuthology.orchestra.run.smithi039.stdout: "warn_threshold": { 2024-03-18T16:20:55.011 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "warn_threshold", 2024-03-18T16:20:55.011 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-03-18T16:20:55.011 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.011 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.012 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "7257600", 2024-03-18T16:20:55.012 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.012 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.012 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.012 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-18T16:20:55.012 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.012 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.012 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.013 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:55.013 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:55.013 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.013 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:55.013 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "diskprediction_local", 2024-03-18T16:20:55.013 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:55.013 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:55.013 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:55.014 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:55.014 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:55.014 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.014 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.014 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.014 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:55.014 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.014 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.015 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:55.015 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:55.015 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:55.015 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:55.015 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:55.015 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:55.015 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:55.016 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:55.016 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.016 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.016 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.016 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.016 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.016 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:55.016 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:55.017 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.017 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.017 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.017 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.017 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.017 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.017 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.017 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.017 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.018 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.018 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.018 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.018 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:55.018 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:55.018 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.018 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.019 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.019 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:55.019 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.019 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.019 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:55.019 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:55.019 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:55.019 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:55.019 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:55.020 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:55.020 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:55.020 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:55.020 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.020 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.020 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.020 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.020 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.021 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:55.021 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:55.021 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.021 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.021 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.021 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.021 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.021 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.022 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.022 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.022 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.022 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.022 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.022 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.022 INFO:teuthology.orchestra.run.smithi039.stdout: "predict_interval": { 2024-03-18T16:20:55.022 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "predict_interval", 2024-03-18T16:20:55.023 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.023 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.023 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.023 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "86400", 2024-03-18T16:20:55.023 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.023 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.023 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.023 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.024 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.024 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.024 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.024 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.024 INFO:teuthology.orchestra.run.smithi039.stdout: "predictor_model": { 2024-03-18T16:20:55.024 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "predictor_model", 2024-03-18T16:20:55.024 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.024 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.025 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.025 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "prophetstor", 2024-03-18T16:20:55.025 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.025 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.025 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.025 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.025 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.025 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.026 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.026 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.026 INFO:teuthology.orchestra.run.smithi039.stdout: "sleep_interval": { 2024-03-18T16:20:55.026 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sleep_interval", 2024-03-18T16:20:55.026 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.026 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.026 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.027 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "600", 2024-03-18T16:20:55.027 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.027 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.027 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.027 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.027 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.027 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.027 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.027 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:55.027 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:55.027 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.028 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:55.028 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "influx", 2024-03-18T16:20:55.029 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": false, 2024-03-18T16:20:55.029 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "influxdb python module not found", 2024-03-18T16:20:55.029 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:55.030 INFO:teuthology.orchestra.run.smithi039.stdout: "batch_size": { 2024-03-18T16:20:55.030 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "batch_size", 2024-03-18T16:20:55.030 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:55.030 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.030 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.030 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "5000", 2024-03-18T16:20:55.030 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.030 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.031 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.031 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-18T16:20:55.031 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.031 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.031 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.031 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.031 INFO:teuthology.orchestra.run.smithi039.stdout: "database": { 2024-03-18T16:20:55.031 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "database", 2024-03-18T16:20:55.032 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.032 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.032 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.032 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "ceph", 2024-03-18T16:20:55.032 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.032 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.032 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.032 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-18T16:20:55.032 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.033 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.033 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.033 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.033 INFO:teuthology.orchestra.run.smithi039.stdout: "hostname": { 2024-03-18T16:20:55.033 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "hostname", 2024-03-18T16:20:55.033 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.033 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.034 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.034 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:55.034 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.034 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.034 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.034 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "InfluxDB server hostname", 2024-03-18T16:20:55.034 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.034 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.035 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.035 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.035 INFO:teuthology.orchestra.run.smithi039.stdout: "interval": { 2024-03-18T16:20:55.035 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "interval", 2024-03-18T16:20:55.035 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-03-18T16:20:55.035 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.035 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.035 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "30", 2024-03-18T16:20:55.036 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "5", 2024-03-18T16:20:55.036 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.036 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.036 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-18T16:20:55.036 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.036 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.036 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.036 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.037 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:55.037 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:55.037 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.037 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.037 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.037 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:55.037 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.037 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.038 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:55.038 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:55.038 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:55.038 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:55.038 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:55.038 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:55.038 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:55.038 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:55.039 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.039 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.039 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.039 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.039 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.039 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:55.039 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:55.040 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.040 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.040 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.040 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.040 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.040 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.040 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.040 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.041 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.041 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.041 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.041 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.041 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:55.041 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:55.041 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.041 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.042 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.042 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:55.042 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.042 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.042 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:55.042 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:55.042 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:55.042 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:55.043 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:55.043 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:55.043 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:55.043 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:55.043 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.043 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.043 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.043 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.044 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.044 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:55.044 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:55.044 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.044 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.044 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.044 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.044 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.045 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.045 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.045 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.045 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.045 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.045 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.045 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.045 INFO:teuthology.orchestra.run.smithi039.stdout: "password": { 2024-03-18T16:20:55.046 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "password", 2024-03-18T16:20:55.046 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.046 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.046 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.046 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:55.046 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.046 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.046 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.047 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "password of InfluxDB server user", 2024-03-18T16:20:55.047 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.047 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.047 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.047 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.047 INFO:teuthology.orchestra.run.smithi039.stdout: "port": { 2024-03-18T16:20:55.047 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "port", 2024-03-18T16:20:55.047 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:55.048 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.048 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.048 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "8086", 2024-03-18T16:20:55.048 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.048 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.048 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.048 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "InfluxDB server port", 2024-03-18T16:20:55.048 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.048 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.049 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.049 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.049 INFO:teuthology.orchestra.run.smithi039.stdout: "ssl": { 2024-03-18T16:20:55.049 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "ssl", 2024-03-18T16:20:55.049 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.049 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.049 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.049 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "false", 2024-03-18T16:20:55.050 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.050 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.050 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.050 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-18T16:20:55.050 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.050 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.050 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.050 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.051 INFO:teuthology.orchestra.run.smithi039.stdout: "threads": { 2024-03-18T16:20:55.051 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "threads", 2024-03-18T16:20:55.051 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:55.051 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.051 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.051 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "5", 2024-03-18T16:20:55.051 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "1", 2024-03-18T16:20:55.051 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "32", 2024-03-18T16:20:55.052 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.052 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-18T16:20:55.052 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.052 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.052 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.052 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.052 INFO:teuthology.orchestra.run.smithi039.stdout: "username": { 2024-03-18T16:20:55.052 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "username", 2024-03-18T16:20:55.053 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.053 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.053 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.053 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:55.053 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.053 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.053 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.053 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "username of InfluxDB server user", 2024-03-18T16:20:55.054 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.054 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.054 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.054 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.054 INFO:teuthology.orchestra.run.smithi039.stdout: "verify_ssl": { 2024-03-18T16:20:55.054 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "verify_ssl", 2024-03-18T16:20:55.054 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.054 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.055 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.055 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "true", 2024-03-18T16:20:55.055 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.055 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.055 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.055 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-18T16:20:55.055 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.055 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.056 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.056 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:55.056 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:55.056 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.056 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:55.056 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "insights", 2024-03-18T16:20:55.056 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:55.056 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:55.056 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:55.057 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:55.057 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:55.057 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.057 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.057 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.057 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:55.057 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.057 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.058 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:55.058 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:55.058 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:55.058 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:55.058 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:55.058 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:55.058 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:55.059 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:55.059 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.059 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.059 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.059 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.059 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.059 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:55.059 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:55.060 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.060 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.070 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.070 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.070 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.070 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.070 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.070 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.070 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.071 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.071 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.071 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.071 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:55.071 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:55.071 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.071 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.071 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.072 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:55.072 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.072 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.072 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:55.072 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:55.072 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:55.072 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:55.072 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:55.073 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:55.073 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:55.073 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:55.073 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.073 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.073 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.073 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.073 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.074 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:55.074 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:55.074 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.074 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.074 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.074 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.074 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.074 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.074 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.074 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.074 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.074 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.074 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.075 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:55.075 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:55.075 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.075 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:55.075 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "iostat", 2024-03-18T16:20:55.075 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:55.075 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:55.075 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:55.075 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:55.075 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:55.075 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.075 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.075 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.075 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:55.076 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.076 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.076 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:55.076 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:55.076 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:55.076 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:55.076 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:55.076 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:55.076 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:55.076 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:55.076 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.076 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.076 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.076 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.076 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.077 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:55.077 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:55.077 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.077 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.077 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.077 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.077 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.077 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.077 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.077 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.077 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.077 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.077 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.077 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.077 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:55.078 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:55.078 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.078 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.078 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.078 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:55.078 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.078 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.078 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:55.078 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:55.078 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:55.078 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:55.078 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:55.078 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:55.078 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:55.078 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:55.079 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.079 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.079 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.079 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.079 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.079 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:55.079 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:55.079 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.079 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.079 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.079 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.079 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.079 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.079 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.079 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.080 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.080 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.080 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.080 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:55.080 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:55.080 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.080 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:55.080 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "k8sevents", 2024-03-18T16:20:55.080 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:55.080 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:55.080 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:55.080 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph_event_retention_days": { 2024-03-18T16:20:55.080 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "ceph_event_retention_days", 2024-03-18T16:20:55.080 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:55.080 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.081 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.081 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "7", 2024-03-18T16:20:55.081 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.081 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.081 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.081 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-18T16:20:55.081 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.081 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.081 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.081 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.081 INFO:teuthology.orchestra.run.smithi039.stdout: "config_check_secs": { 2024-03-18T16:20:55.081 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "config_check_secs", 2024-03-18T16:20:55.081 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:55.081 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.082 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.082 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "10", 2024-03-18T16:20:55.082 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "10", 2024-03-18T16:20:55.082 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.082 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.082 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-18T16:20:55.082 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.082 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.082 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.082 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.082 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:55.082 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:55.082 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.083 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.083 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.083 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:55.083 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.083 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.083 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:55.083 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:55.083 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:55.083 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:55.083 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:55.083 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:55.083 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:55.083 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:55.084 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.084 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.084 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.084 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.084 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.084 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:55.084 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:55.084 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.084 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.084 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.084 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.084 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.084 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.084 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.084 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.085 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.085 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.085 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.085 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.085 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:55.085 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:55.085 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.085 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.085 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.085 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:55.085 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.085 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.085 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:55.085 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:55.085 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:55.086 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:55.086 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:55.086 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:55.086 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:55.086 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:55.086 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.086 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.086 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.086 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.086 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.086 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:55.086 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:55.086 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.086 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.087 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.087 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.087 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.087 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.087 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.087 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.087 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.087 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.087 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.087 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:55.087 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:55.087 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.087 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:55.087 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "localpool", 2024-03-18T16:20:55.087 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:55.088 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:55.088 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:55.088 INFO:teuthology.orchestra.run.smithi039.stdout: "failure_domain": { 2024-03-18T16:20:55.088 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "failure_domain", 2024-03-18T16:20:55.088 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.094 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.094 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.094 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "host", 2024-03-18T16:20:55.094 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.094 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.095 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.095 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "failure domain for any created local pool", 2024-03-18T16:20:55.095 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-18T16:20:55.095 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.095 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.095 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.095 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:55.096 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:55.096 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.096 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.096 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.096 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:55.096 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.096 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.097 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:55.097 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:55.097 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:55.097 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:55.097 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:55.097 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:55.097 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:55.098 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:55.098 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.098 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.098 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.098 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.098 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.098 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:55.098 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:55.098 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.099 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.099 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.099 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.099 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.099 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.099 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.099 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.100 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.100 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.100 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.100 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.100 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:55.100 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:55.100 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.101 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.101 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.101 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:55.101 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.101 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.101 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:55.101 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:55.102 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:55.102 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:55.102 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:55.102 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:55.102 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:55.102 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:55.102 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.102 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.102 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.102 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.103 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.103 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:55.103 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:55.103 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.103 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.103 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.103 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.103 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.104 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.104 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.104 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.104 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.104 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.104 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.104 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.104 INFO:teuthology.orchestra.run.smithi039.stdout: "min_size": { 2024-03-18T16:20:55.105 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "min_size", 2024-03-18T16:20:55.105 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:55.105 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.105 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.105 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:55.105 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.105 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.106 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.106 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "default min_size for any created local pool", 2024-03-18T16:20:55.106 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-18T16:20:55.106 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.106 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.106 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.106 INFO:teuthology.orchestra.run.smithi039.stdout: "num_rep": { 2024-03-18T16:20:55.107 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "num_rep", 2024-03-18T16:20:55.107 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:55.107 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.107 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.107 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "3", 2024-03-18T16:20:55.107 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.107 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.107 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.107 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "default replica count for any created local pool", 2024-03-18T16:20:55.107 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.108 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.108 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.108 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.108 INFO:teuthology.orchestra.run.smithi039.stdout: "pg_num": { 2024-03-18T16:20:55.108 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "pg_num", 2024-03-18T16:20:55.108 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:55.109 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.109 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.109 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "128", 2024-03-18T16:20:55.109 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.109 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.109 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.110 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "default pg_num for any created local pool", 2024-03-18T16:20:55.110 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.110 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.110 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.111 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.111 INFO:teuthology.orchestra.run.smithi039.stdout: "prefix": { 2024-03-18T16:20:55.111 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "prefix", 2024-03-18T16:20:55.111 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.111 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.111 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.111 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:55.112 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.112 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.112 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.112 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "name prefix for any created local pool", 2024-03-18T16:20:55.112 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.112 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.112 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.113 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.113 INFO:teuthology.orchestra.run.smithi039.stdout: "subtree": { 2024-03-18T16:20:55.113 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "subtree", 2024-03-18T16:20:55.113 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.113 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.113 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.114 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "rack", 2024-03-18T16:20:55.114 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.114 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.114 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.114 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-18T16:20:55.114 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-18T16:20:55.114 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.115 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.115 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:55.115 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:55.115 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.115 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:55.115 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "mds_autoscaler", 2024-03-18T16:20:55.115 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:55.115 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:55.116 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:55.116 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:55.116 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:55.116 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.116 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.116 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.116 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:55.117 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.117 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.117 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:55.117 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:55.117 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:55.117 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:55.118 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:55.118 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:55.118 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:55.118 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:55.118 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.118 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.119 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.119 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.119 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.119 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:55.119 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:55.119 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.119 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.119 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.120 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.120 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.120 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.120 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.120 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.120 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.120 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.121 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.121 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.121 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:55.121 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:55.121 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.121 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.122 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.122 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:55.122 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.122 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.122 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:55.122 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:55.122 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:55.123 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:55.123 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:55.123 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:55.123 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:55.123 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:55.123 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.123 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.124 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.124 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.124 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.124 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:55.124 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:55.124 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.124 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.125 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.125 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.125 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.125 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.125 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.125 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.126 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.126 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.126 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.126 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:55.126 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:55.126 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.126 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:55.127 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "mirroring", 2024-03-18T16:20:55.127 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:55.127 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:55.127 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:55.127 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:55.127 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:55.127 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.128 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.128 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.128 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:55.128 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.128 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.128 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:55.128 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:55.128 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:55.129 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:55.129 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:55.129 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:55.129 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:55.129 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:55.129 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.129 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.130 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.130 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.130 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.130 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:55.130 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:55.130 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.131 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.131 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.131 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.131 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.131 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.131 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.131 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.131 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.132 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.132 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.132 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.132 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:55.132 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:55.132 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.132 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.133 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.133 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:55.133 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.133 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.139 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:55.140 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:55.140 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:55.140 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:55.140 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:55.140 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:55.140 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:55.140 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:55.141 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.141 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.141 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.141 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.141 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.141 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:55.141 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:55.141 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.142 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.142 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.142 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.142 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.142 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.142 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.142 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.142 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.142 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.143 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.143 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:55.143 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:55.143 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.143 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:55.143 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "nfs", 2024-03-18T16:20:55.143 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:55.143 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:55.144 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:55.144 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:55.144 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:55.144 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.144 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.144 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.144 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:55.145 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.145 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.145 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:55.145 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:55.145 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:55.145 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:55.145 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:55.145 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:55.146 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:55.146 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:55.146 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.146 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.146 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.146 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.146 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.147 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:55.147 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:55.147 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.147 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.147 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.147 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.147 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.148 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.148 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.148 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.148 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.148 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.148 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.148 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.148 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:55.149 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:55.149 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.149 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.149 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.149 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:55.149 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.149 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.149 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:55.150 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:55.150 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:55.150 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:55.150 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:55.150 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:55.150 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:55.150 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:55.151 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.151 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.151 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.151 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.151 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.151 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:55.151 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:55.151 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.152 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.152 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.152 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.152 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.152 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.152 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.152 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.153 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.153 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.153 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.153 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:55.153 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:55.153 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.153 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:55.153 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "orchestrator", 2024-03-18T16:20:55.154 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:55.154 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:55.154 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:55.154 INFO:teuthology.orchestra.run.smithi039.stdout: "fail_fs": { 2024-03-18T16:20:55.154 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "fail_fs", 2024-03-18T16:20:55.154 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.154 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.155 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.155 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.155 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.155 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.155 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.155 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-18T16:20:55.155 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.156 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.156 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.156 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.156 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:55.156 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:55.156 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.156 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.156 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.157 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:55.157 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.157 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.157 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:55.157 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:55.157 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:55.157 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:55.157 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:55.158 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:55.158 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:55.158 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:55.158 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.158 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.158 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.158 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.159 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.159 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:55.159 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:55.159 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.159 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.159 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.159 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.159 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.160 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.160 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.160 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.160 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.160 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.160 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.160 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.160 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:55.161 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:55.161 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.161 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.161 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.161 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:55.161 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.161 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.162 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:55.162 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:55.162 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:55.162 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:55.162 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:55.162 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:55.162 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:55.163 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:55.163 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.163 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.163 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.163 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.163 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.164 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:55.164 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:55.164 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.164 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.164 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.164 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.165 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.165 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.165 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.165 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.165 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.211 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.220 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.220 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.220 INFO:teuthology.orchestra.run.smithi039.stdout: "orchestrator": { 2024-03-18T16:20:55.220 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "orchestrator", 2024-03-18T16:20:55.220 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.220 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.220 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.220 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:55.220 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.220 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.220 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:55.221 INFO:teuthology.orchestra.run.smithi039.stdout: "cephadm", 2024-03-18T16:20:55.221 INFO:teuthology.orchestra.run.smithi039.stdout: "rook", 2024-03-18T16:20:55.221 INFO:teuthology.orchestra.run.smithi039.stdout: "test_orchestrator" 2024-03-18T16:20:55.221 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:55.221 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Orchestrator backend", 2024-03-18T16:20:55.221 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.221 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.221 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.221 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:55.221 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:55.221 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.221 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:55.221 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "osd_perf_query", 2024-03-18T16:20:55.221 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:55.222 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:55.222 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:55.222 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:55.222 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:55.222 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.222 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.222 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.222 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:55.222 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.222 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.222 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:55.222 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:55.222 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:55.223 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:55.223 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:55.223 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:55.223 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:55.223 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:55.223 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.223 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.223 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.223 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.223 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.223 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:55.223 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:55.223 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.224 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.224 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.224 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.224 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.224 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.224 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.224 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.224 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.224 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.224 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.224 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.224 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:55.224 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:55.225 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.225 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.225 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.225 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:55.225 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.225 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.225 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:55.225 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:55.225 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:55.225 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:55.225 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:55.225 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:55.225 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:55.225 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:55.226 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.226 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.226 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.226 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.226 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.226 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:55.226 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:55.226 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.226 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.226 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.226 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.226 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.226 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.226 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.227 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.227 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.227 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.227 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.227 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:55.227 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:55.227 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.227 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:55.227 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "osd_support", 2024-03-18T16:20:55.227 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:55.227 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:55.227 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:55.228 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:55.228 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:55.228 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.228 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.228 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.228 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:55.228 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.228 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.228 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:55.228 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:55.228 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:55.228 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:55.228 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:55.228 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:55.229 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:55.229 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:55.229 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.229 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.229 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.229 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.229 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.229 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:55.229 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:55.229 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.229 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.229 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.229 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.229 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.230 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.230 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.230 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.230 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.230 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.230 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.230 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.230 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:55.230 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:55.230 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.230 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.230 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.230 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:55.230 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.231 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.231 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:55.231 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:55.231 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:55.231 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:55.231 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:55.231 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:55.231 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:55.231 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:55.231 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.231 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.231 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.231 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.231 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.232 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:55.232 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:55.233 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.233 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.233 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.233 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.234 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.234 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.234 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.234 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.234 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.234 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.234 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.234 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:55.234 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:55.234 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.234 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:55.234 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "pg_autoscaler", 2024-03-18T16:20:55.234 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:55.234 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:55.235 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:55.235 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:55.235 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:55.235 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.235 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.235 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.235 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:55.235 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.235 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.235 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:55.235 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:55.235 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:55.235 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:55.235 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:55.236 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:55.236 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:55.236 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:55.236 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.236 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.236 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.236 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.236 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.236 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:55.236 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:55.236 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.236 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.236 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.236 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.237 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.237 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.237 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.237 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.237 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.237 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.237 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.237 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.237 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:55.237 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:55.237 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.237 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.237 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.237 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:55.237 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.238 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.238 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:55.238 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:55.238 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:55.238 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:55.238 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:55.238 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:55.238 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:55.238 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:55.238 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.238 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.238 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.238 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.239 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.239 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:55.239 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:55.239 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.239 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.239 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.239 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.239 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.239 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.239 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.239 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.239 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.240 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.240 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.240 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.240 INFO:teuthology.orchestra.run.smithi039.stdout: "sleep_interval": { 2024-03-18T16:20:55.240 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sleep_interval", 2024-03-18T16:20:55.240 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-03-18T16:20:55.240 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.240 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.240 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "60", 2024-03-18T16:20:55.240 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.240 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.240 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.240 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.240 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.240 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.241 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.241 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.241 INFO:teuthology.orchestra.run.smithi039.stdout: "threshold": { 2024-03-18T16:20:55.241 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "threshold", 2024-03-18T16:20:55.241 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "float", 2024-03-18T16:20:55.241 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.241 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.241 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "3.0", 2024-03-18T16:20:55.241 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "1.0", 2024-03-18T16:20:55.241 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.241 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.241 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "scaling threshold", 2024-03-18T16:20:55.241 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-18T16:20:55.242 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.242 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.242 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:55.242 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:55.242 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.242 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:55.242 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "progress", 2024-03-18T16:20:55.242 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:55.242 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:55.242 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:55.242 INFO:teuthology.orchestra.run.smithi039.stdout: "allow_pg_recovery_event": { 2024-03-18T16:20:55.242 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "allow_pg_recovery_event", 2024-03-18T16:20:55.242 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.242 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.243 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.243 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.243 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.243 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.243 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.243 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-18T16:20:55.243 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.243 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.243 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.243 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.272 INFO:teuthology.orchestra.run.smithi039.stdout: "enabled": { 2024-03-18T16:20:55.272 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "enabled", 2024-03-18T16:20:55.272 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.272 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.272 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.272 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:55.273 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.273 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.273 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.273 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.273 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.274 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.274 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.274 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.274 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:55.274 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:55.274 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.274 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.274 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.274 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:55.274 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.275 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.275 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:55.275 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:55.275 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:55.275 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:55.275 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:55.275 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:55.275 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:55.275 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:55.275 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.275 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.275 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.275 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.275 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.276 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:55.276 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:55.276 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.276 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.276 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.276 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.276 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.276 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.276 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.276 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.276 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.276 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.276 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.276 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.276 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:55.277 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:55.277 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.277 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.277 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.277 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:55.277 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.277 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.277 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:55.277 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:55.277 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:55.277 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:55.277 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:55.277 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:55.277 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:55.277 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:55.278 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.278 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.278 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.278 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.278 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.278 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:55.278 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:55.278 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.278 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.278 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.278 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.278 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.278 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.278 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.278 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.279 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.279 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.279 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.279 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.279 INFO:teuthology.orchestra.run.smithi039.stdout: "max_completed_events": { 2024-03-18T16:20:55.279 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "max_completed_events", 2024-03-18T16:20:55.279 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:55.279 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.279 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.279 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "50", 2024-03-18T16:20:55.279 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.279 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.279 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.279 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "number of past completed events to remember", 2024-03-18T16:20:55.279 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.280 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.280 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.280 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.280 INFO:teuthology.orchestra.run.smithi039.stdout: "sleep_interval": { 2024-03-18T16:20:55.280 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sleep_interval", 2024-03-18T16:20:55.280 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-03-18T16:20:55.280 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.280 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.280 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "5", 2024-03-18T16:20:55.280 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.280 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.280 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.280 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "how long the module is going to sleep", 2024-03-18T16:20:55.280 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.281 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.281 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.281 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:55.281 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:55.281 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.281 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:55.281 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "prometheus", 2024-03-18T16:20:55.281 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:55.281 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:55.281 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:55.281 INFO:teuthology.orchestra.run.smithi039.stdout: "cache": { 2024-03-18T16:20:55.281 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "cache", 2024-03-18T16:20:55.281 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.281 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.281 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.282 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:55.282 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.282 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.282 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.282 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.282 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.282 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.282 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.282 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.282 INFO:teuthology.orchestra.run.smithi039.stdout: "exclude_perf_counters": { 2024-03-18T16:20:55.282 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "exclude_perf_counters", 2024-03-18T16:20:55.282 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.282 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.283 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.283 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:55.283 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.283 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.283 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.283 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-18T16:20:55.283 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-18T16:20:55.283 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.283 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.283 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.283 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:55.283 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:55.283 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.283 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.284 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.284 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:55.284 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.284 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.284 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:55.284 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:55.284 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:55.284 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:55.284 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:55.284 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:55.284 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:55.284 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:55.284 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.284 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.285 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.285 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.285 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.285 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:55.285 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:55.285 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.285 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.285 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.285 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.285 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.285 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.285 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.286 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.286 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.286 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.286 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.286 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.286 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:55.286 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:55.287 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.287 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.287 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.287 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:55.287 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.292 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.338 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:55.338 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:55.338 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:55.338 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:55.338 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:55.338 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:55.338 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:55.338 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:55.338 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.339 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.339 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.339 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.339 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.339 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:55.339 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:55.339 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.339 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.339 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.339 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.339 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.339 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.339 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.339 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.340 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.340 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.340 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.340 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.340 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_stats_pools": { 2024-03-18T16:20:55.340 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "rbd_stats_pools", 2024-03-18T16:20:55.340 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.340 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.340 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.340 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:55.340 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.340 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.341 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.341 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.341 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.341 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.341 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.341 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.341 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-18T16:20:55.341 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-18T16:20:55.341 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:55.341 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.341 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.342 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "300", 2024-03-18T16:20:55.342 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.342 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.342 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.342 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.342 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.342 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.342 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.342 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.342 INFO:teuthology.orchestra.run.smithi039.stdout: "scrape_interval": { 2024-03-18T16:20:55.342 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "scrape_interval", 2024-03-18T16:20:55.342 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "float", 2024-03-18T16:20:55.343 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.343 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.343 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "15.0", 2024-03-18T16:20:55.343 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.343 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.343 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.343 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.343 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.344 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.344 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.344 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.344 INFO:teuthology.orchestra.run.smithi039.stdout: "server_addr": { 2024-03-18T16:20:55.344 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "server_addr", 2024-03-18T16:20:55.344 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.344 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.344 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.344 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "::", 2024-03-18T16:20:55.344 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.344 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.344 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.345 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-18T16:20:55.345 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.345 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.345 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.345 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.345 INFO:teuthology.orchestra.run.smithi039.stdout: "server_port": { 2024-03-18T16:20:55.345 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "server_port", 2024-03-18T16:20:55.345 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:55.345 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.345 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.345 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "9283", 2024-03-18T16:20:55.345 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.345 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.345 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.346 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-18T16:20:55.346 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.346 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.346 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.346 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.346 INFO:teuthology.orchestra.run.smithi039.stdout: "stale_cache_strategy": { 2024-03-18T16:20:55.346 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "stale_cache_strategy", 2024-03-18T16:20:55.346 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.346 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.346 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.346 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "log", 2024-03-18T16:20:55.346 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.346 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.346 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.347 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.347 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.347 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.347 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.347 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.347 INFO:teuthology.orchestra.run.smithi039.stdout: "standby_behaviour": { 2024-03-18T16:20:55.347 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "standby_behaviour", 2024-03-18T16:20:55.347 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.347 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.347 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.347 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "default", 2024-03-18T16:20:55.347 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.347 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.348 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:55.348 INFO:teuthology.orchestra.run.smithi039.stdout: "default", 2024-03-18T16:20:55.348 INFO:teuthology.orchestra.run.smithi039.stdout: "error" 2024-03-18T16:20:55.348 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:55.348 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.348 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.348 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.348 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.348 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.348 INFO:teuthology.orchestra.run.smithi039.stdout: "standby_error_status_code": { 2024-03-18T16:20:55.348 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "standby_error_status_code", 2024-03-18T16:20:55.348 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:55.348 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.348 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.349 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "500", 2024-03-18T16:20:55.349 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "400", 2024-03-18T16:20:55.349 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "599", 2024-03-18T16:20:55.349 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.349 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.349 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.349 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.349 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.349 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:55.349 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:55.349 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.349 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:55.350 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "rbd_support", 2024-03-18T16:20:55.350 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:55.350 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:55.350 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:55.350 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:55.350 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:55.350 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.350 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.350 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.350 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:55.350 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.350 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.350 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:55.350 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:55.351 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:55.351 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:55.351 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:55.351 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:55.351 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:55.351 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:55.351 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.351 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.351 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.351 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.351 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.351 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:55.351 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:55.352 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.352 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.352 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.352 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.352 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.352 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.352 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.352 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.352 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.352 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.352 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.352 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.352 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:55.352 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:55.352 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.353 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.353 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.353 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:55.353 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.353 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.353 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:55.353 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:55.353 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:55.353 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:55.353 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:55.353 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:55.353 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:55.353 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:55.353 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.353 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.354 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.354 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.354 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.354 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:55.354 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:55.354 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.354 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.354 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.354 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.354 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.354 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.354 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.354 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.355 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.355 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.355 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.355 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.355 INFO:teuthology.orchestra.run.smithi039.stdout: "max_concurrent_snap_create": { 2024-03-18T16:20:55.355 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "max_concurrent_snap_create", 2024-03-18T16:20:55.355 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:55.355 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.355 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.355 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "10", 2024-03-18T16:20:55.355 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.355 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.355 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.355 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.355 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.356 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.356 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.356 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.356 INFO:teuthology.orchestra.run.smithi039.stdout: "mirror_snapshot_schedule": { 2024-03-18T16:20:55.356 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "mirror_snapshot_schedule", 2024-03-18T16:20:55.356 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.356 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.356 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.356 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:55.356 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.356 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.356 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.356 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.356 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.356 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.357 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.357 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.357 INFO:teuthology.orchestra.run.smithi039.stdout: "trash_purge_schedule": { 2024-03-18T16:20:55.357 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "trash_purge_schedule", 2024-03-18T16:20:55.357 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.357 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.357 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.357 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:55.357 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.357 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.357 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.357 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.357 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.357 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.357 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.358 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:55.358 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:55.358 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.358 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:55.358 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "restful", 2024-03-18T16:20:55.358 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:55.358 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:55.358 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:55.358 INFO:teuthology.orchestra.run.smithi039.stdout: "enable_auth": { 2024-03-18T16:20:55.358 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "enable_auth", 2024-03-18T16:20:55.358 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.358 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.358 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.359 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:55.359 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.359 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.359 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.359 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.359 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.359 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.359 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.359 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.359 INFO:teuthology.orchestra.run.smithi039.stdout: "key_file": { 2024-03-18T16:20:55.359 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "key_file", 2024-03-18T16:20:55.359 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.360 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.360 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.360 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:55.360 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.360 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.360 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.360 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.360 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.360 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.360 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.360 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.360 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:55.361 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:55.361 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.361 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.361 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.361 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:55.361 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.361 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.361 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:55.361 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:55.361 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:55.361 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:55.361 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:55.361 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:55.361 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:55.362 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:55.362 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.362 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.362 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.362 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.362 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.362 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:55.362 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:55.362 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.362 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.362 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.363 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.363 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.363 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.363 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.363 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.363 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.363 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.363 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.363 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.363 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:55.364 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:55.364 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.364 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.364 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.364 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:55.364 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.364 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.364 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:55.364 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:55.364 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:55.364 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:55.364 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:55.364 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:55.365 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:55.365 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:55.365 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.365 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.365 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.365 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.365 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.365 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:55.365 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:55.365 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.365 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.365 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.365 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.365 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.365 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.366 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.414 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.414 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.414 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.414 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.414 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.414 INFO:teuthology.orchestra.run.smithi039.stdout: "server_addr": { 2024-03-18T16:20:55.414 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "server_addr", 2024-03-18T16:20:55.415 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.415 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.415 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.415 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:55.415 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.415 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.415 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.415 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.415 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.415 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.415 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.415 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.415 INFO:teuthology.orchestra.run.smithi039.stdout: "server_port": { 2024-03-18T16:20:55.415 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "server_port", 2024-03-18T16:20:55.416 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.416 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.416 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.416 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:55.416 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.416 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.416 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.416 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.416 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.416 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.416 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.416 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:55.416 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:55.417 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.417 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:55.417 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "rgw", 2024-03-18T16:20:55.417 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:55.417 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:55.417 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:55.417 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:55.417 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:55.417 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.417 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.417 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.417 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:55.417 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.417 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.418 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:55.418 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:55.418 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:55.418 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:55.418 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:55.418 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:55.418 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:55.418 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:55.418 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.418 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.418 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.418 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.419 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.419 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:55.419 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:55.419 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.419 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.419 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.419 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.419 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.419 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.419 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.419 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.419 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.419 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.419 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.420 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.420 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:55.420 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:55.420 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.420 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.420 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.420 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:55.420 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.420 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.420 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:55.420 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:55.420 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:55.421 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:55.421 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:55.421 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:55.421 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:55.421 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:55.421 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.421 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.421 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.421 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.421 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.421 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:55.421 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:55.422 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.422 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.422 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.422 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.422 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.422 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.422 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.422 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.422 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.422 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.422 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.423 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:55.423 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:55.423 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.423 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:55.423 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "rook", 2024-03-18T16:20:55.423 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:55.423 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:55.424 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:55.424 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:55.424 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:55.424 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.424 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.424 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.424 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:55.424 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.424 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.424 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:55.424 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:55.424 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:55.424 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:55.424 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:55.425 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:55.425 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:55.425 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:55.425 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.425 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.425 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.425 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.425 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.425 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:55.425 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:55.425 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.425 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.425 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.425 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.425 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.426 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.426 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.426 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.426 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.426 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.426 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.426 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.426 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:55.426 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:55.426 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.426 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.426 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.427 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:55.427 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.427 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.427 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:55.427 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:55.427 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:55.427 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:55.427 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:55.427 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:55.427 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:55.427 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:55.428 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.428 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.428 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.428 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.428 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.428 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:55.428 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:55.428 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.428 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.428 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.428 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.428 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.428 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.428 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.428 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.429 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.429 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.429 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.429 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.429 INFO:teuthology.orchestra.run.smithi039.stdout: "storage_class": { 2024-03-18T16:20:55.429 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "storage_class", 2024-03-18T16:20:55.429 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.429 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.429 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.429 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "local", 2024-03-18T16:20:55.429 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.429 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.429 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.429 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-18T16:20:55.430 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.430 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.430 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.430 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:55.430 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:55.431 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.431 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:55.431 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "selftest", 2024-03-18T16:20:55.431 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:55.431 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:55.431 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:55.431 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:55.431 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:55.431 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.431 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.431 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.431 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:55.432 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.432 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.432 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:55.432 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:55.432 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:55.432 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:55.432 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:55.432 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:55.432 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:55.432 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:55.432 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.432 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.432 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.433 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.433 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.433 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:55.433 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:55.433 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.433 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.433 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.433 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.433 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.433 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.433 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.434 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.434 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.434 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.434 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.434 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.434 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:55.434 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:55.434 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.434 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.434 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.435 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:55.435 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.435 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.435 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:55.435 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:55.435 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:55.435 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:55.435 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:55.435 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:55.435 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:55.435 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:55.435 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.436 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.436 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.436 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.436 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.436 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:55.436 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:55.436 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.436 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.436 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.436 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.436 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.436 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.437 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.437 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.437 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.437 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.437 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.437 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.437 INFO:teuthology.orchestra.run.smithi039.stdout: "roption1": { 2024-03-18T16:20:55.437 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "roption1", 2024-03-18T16:20:55.437 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.437 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.437 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.437 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:55.437 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.438 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.438 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.438 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.438 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.438 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.438 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.438 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.438 INFO:teuthology.orchestra.run.smithi039.stdout: "roption2": { 2024-03-18T16:20:55.438 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "roption2", 2024-03-18T16:20:55.438 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.438 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.438 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.438 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "xyz", 2024-03-18T16:20:55.438 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.439 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.439 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.439 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.439 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.439 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.439 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.439 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.439 INFO:teuthology.orchestra.run.smithi039.stdout: "rwoption1": { 2024-03-18T16:20:55.439 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "rwoption1", 2024-03-18T16:20:55.439 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.440 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.440 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.440 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:55.440 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.440 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.440 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.440 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.440 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.440 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.440 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.467 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.467 INFO:teuthology.orchestra.run.smithi039.stdout: "rwoption2": { 2024-03-18T16:20:55.467 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "rwoption2", 2024-03-18T16:20:55.467 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:55.467 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.467 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.467 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:55.467 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.467 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.467 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.467 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.470 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.470 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.471 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.471 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.471 INFO:teuthology.orchestra.run.smithi039.stdout: "rwoption3": { 2024-03-18T16:20:55.471 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "rwoption3", 2024-03-18T16:20:55.471 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "float", 2024-03-18T16:20:55.471 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.471 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.471 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:55.471 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.471 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.471 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.471 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.471 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.471 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.472 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.472 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.472 INFO:teuthology.orchestra.run.smithi039.stdout: "rwoption4": { 2024-03-18T16:20:55.472 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "rwoption4", 2024-03-18T16:20:55.472 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.472 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.472 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.472 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:55.472 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.472 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.472 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.472 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.472 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.472 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.473 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.473 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.473 INFO:teuthology.orchestra.run.smithi039.stdout: "rwoption5": { 2024-03-18T16:20:55.473 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "rwoption5", 2024-03-18T16:20:55.473 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.473 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.473 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.473 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:55.473 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.473 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.473 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.473 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.473 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.473 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.474 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.474 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.474 INFO:teuthology.orchestra.run.smithi039.stdout: "rwoption6": { 2024-03-18T16:20:55.474 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "rwoption6", 2024-03-18T16:20:55.474 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.474 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.474 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.474 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:55.474 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.474 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.474 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.474 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.474 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.474 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.474 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.475 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.475 INFO:teuthology.orchestra.run.smithi039.stdout: "rwoption7": { 2024-03-18T16:20:55.475 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "rwoption7", 2024-03-18T16:20:55.475 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:55.475 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.475 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.475 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:55.475 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "1", 2024-03-18T16:20:55.475 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "42", 2024-03-18T16:20:55.475 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.475 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.475 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.475 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.475 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.475 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.476 INFO:teuthology.orchestra.run.smithi039.stdout: "testkey": { 2024-03-18T16:20:55.476 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "testkey", 2024-03-18T16:20:55.476 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.476 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.476 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.476 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:55.476 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.476 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.476 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.476 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.476 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.476 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.476 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.476 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.476 INFO:teuthology.orchestra.run.smithi039.stdout: "testlkey": { 2024-03-18T16:20:55.477 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "testlkey", 2024-03-18T16:20:55.477 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.477 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.477 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.477 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:55.477 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.477 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.477 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.477 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.477 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.477 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.477 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.477 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.477 INFO:teuthology.orchestra.run.smithi039.stdout: "testnewline": { 2024-03-18T16:20:55.477 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "testnewline", 2024-03-18T16:20:55.478 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.478 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.478 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.478 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:55.478 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.478 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.478 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.478 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.478 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.478 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.478 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.478 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:55.478 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:55.478 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.478 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:55.479 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "snap_schedule", 2024-03-18T16:20:55.479 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:55.479 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:55.479 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:55.479 INFO:teuthology.orchestra.run.smithi039.stdout: "allow_m_granularity": { 2024-03-18T16:20:55.479 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "allow_m_granularity", 2024-03-18T16:20:55.479 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.479 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.479 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.479 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.479 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.479 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.479 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.479 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "allow minute scheduled snapshots", 2024-03-18T16:20:55.479 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.480 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.480 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.480 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.480 INFO:teuthology.orchestra.run.smithi039.stdout: "dump_on_update": { 2024-03-18T16:20:55.480 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "dump_on_update", 2024-03-18T16:20:55.480 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.480 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.480 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.480 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.480 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.480 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.480 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.480 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "dump database to debug log on update", 2024-03-18T16:20:55.480 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.480 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.480 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.481 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.481 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:55.481 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:55.481 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.481 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.481 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.481 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:55.481 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.481 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.481 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:55.481 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:55.481 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:55.481 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:55.481 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:55.481 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:55.482 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:55.482 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:55.482 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.482 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.482 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.482 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.482 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.482 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:55.482 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:55.482 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.482 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.482 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.482 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.482 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.482 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.483 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.483 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.483 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.483 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.483 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.483 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.483 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:55.483 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:55.483 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.483 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.483 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.483 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:55.483 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.483 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.483 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:55.483 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:55.484 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:55.484 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:55.484 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:55.484 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:55.484 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:55.484 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:55.484 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.484 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.484 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.484 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.484 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.484 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:55.484 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:55.484 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.485 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.485 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.485 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.485 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.485 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.485 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.485 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.485 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.485 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.485 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.485 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:55.485 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:55.485 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.485 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:55.485 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "stats", 2024-03-18T16:20:55.486 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:55.486 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:55.486 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:55.486 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:55.486 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:55.486 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.486 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.486 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.486 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:55.486 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.486 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.486 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:55.486 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:55.486 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:55.486 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:55.487 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:55.487 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:55.487 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:55.487 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:55.487 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.487 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.487 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.487 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.487 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.487 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:55.487 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:55.487 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.487 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.487 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.487 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.488 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.488 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.488 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.488 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.488 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.488 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.488 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.488 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.488 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:55.488 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:55.488 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.488 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.488 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.488 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:55.488 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.489 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.489 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:55.489 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:55.489 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:55.489 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:55.489 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:55.489 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:55.489 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:55.489 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:55.489 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.489 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.489 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.489 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.489 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.489 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:55.490 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:55.490 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.490 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.490 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.490 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.490 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.490 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.490 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.490 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.490 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.490 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.498 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.498 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:55.498 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:55.498 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.498 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:55.499 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "status", 2024-03-18T16:20:55.499 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:55.499 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:55.499 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:55.499 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:55.499 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:55.499 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.499 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.500 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.500 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:55.500 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.500 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.500 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:55.500 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:55.500 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:55.500 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:55.500 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:55.501 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:55.501 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:55.501 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:55.501 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.501 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.501 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.501 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.501 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.501 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:55.501 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:55.502 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.502 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.502 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.502 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.502 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.502 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.502 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.502 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.502 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.502 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.503 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.503 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.503 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:55.503 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:55.503 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.503 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.503 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.503 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:55.503 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.504 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.504 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:55.504 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:55.504 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:55.504 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:55.504 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:55.504 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:55.504 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:55.504 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:55.505 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.505 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.505 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.505 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.505 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.505 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:55.505 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:55.505 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.505 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.505 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.506 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.506 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.506 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.506 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.506 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.506 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.506 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.506 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.507 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:55.507 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:55.507 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.507 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:55.507 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "telegraf", 2024-03-18T16:20:55.507 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:55.507 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:55.507 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:55.507 INFO:teuthology.orchestra.run.smithi039.stdout: "address": { 2024-03-18T16:20:55.508 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "address", 2024-03-18T16:20:55.508 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.508 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.508 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.508 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-18T16:20:55.508 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.508 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.509 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.509 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.509 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.509 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.509 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.509 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.509 INFO:teuthology.orchestra.run.smithi039.stdout: "interval": { 2024-03-18T16:20:55.509 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "interval", 2024-03-18T16:20:55.509 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-03-18T16:20:55.510 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.510 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.510 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "15", 2024-03-18T16:20:55.510 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.510 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.510 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.510 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.510 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.511 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.511 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.511 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.511 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:55.511 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:55.511 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.511 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.511 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.511 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:55.512 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.512 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.512 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:55.512 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:55.512 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:55.512 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:55.512 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:55.512 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:55.513 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:55.513 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:55.513 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.513 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.513 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.513 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.513 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.513 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:55.514 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:55.514 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.514 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.523 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.523 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.524 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.524 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.524 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.524 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.524 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.524 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.524 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.524 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.524 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:55.524 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:55.524 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.524 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.524 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.524 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:55.524 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.525 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.525 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:55.525 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:55.525 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:55.525 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:55.525 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:55.525 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:55.525 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:55.525 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:55.525 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.525 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.525 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.525 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.525 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.525 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:55.526 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:55.526 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.526 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.526 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.526 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.526 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.526 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.526 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.526 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.526 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.526 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.526 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.526 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:55.526 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:55.527 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.527 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:55.527 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "telemetry", 2024-03-18T16:20:55.527 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:55.527 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:55.527 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:55.527 INFO:teuthology.orchestra.run.smithi039.stdout: "channel_basic": { 2024-03-18T16:20:55.527 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "channel_basic", 2024-03-18T16:20:55.527 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.527 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.527 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.527 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:55.527 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.527 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.528 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.528 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-18T16:20:55.528 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.528 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.528 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.528 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.528 INFO:teuthology.orchestra.run.smithi039.stdout: "channel_crash": { 2024-03-18T16:20:55.528 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "channel_crash", 2024-03-18T16:20:55.528 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.528 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.528 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.528 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:55.528 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.528 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.528 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.529 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-18T16:20:55.529 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.529 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.529 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.529 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.529 INFO:teuthology.orchestra.run.smithi039.stdout: "channel_device": { 2024-03-18T16:20:55.529 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "channel_device", 2024-03-18T16:20:55.529 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.529 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.529 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.529 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:55.529 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.529 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.529 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.529 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-18T16:20:55.530 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.530 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.530 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.530 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.530 INFO:teuthology.orchestra.run.smithi039.stdout: "channel_ident": { 2024-03-18T16:20:55.530 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "channel_ident", 2024-03-18T16:20:55.530 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.530 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.530 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.530 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.530 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.530 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.530 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.530 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-18T16:20:55.530 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.531 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.531 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.531 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.531 INFO:teuthology.orchestra.run.smithi039.stdout: "channel_perf": { 2024-03-18T16:20:55.531 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "channel_perf", 2024-03-18T16:20:55.531 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.531 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.531 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.531 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.531 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.531 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.531 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.531 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-18T16:20:55.531 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.531 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.532 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.532 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.532 INFO:teuthology.orchestra.run.smithi039.stdout: "contact": { 2024-03-18T16:20:55.532 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "contact", 2024-03-18T16:20:55.532 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.532 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.532 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.532 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:55.532 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.532 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.532 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.532 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.532 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.532 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.532 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.533 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.533 INFO:teuthology.orchestra.run.smithi039.stdout: "description": { 2024-03-18T16:20:55.533 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "description", 2024-03-18T16:20:55.533 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.533 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.533 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.533 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:55.533 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.533 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.533 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.533 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.533 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.533 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.533 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.533 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.534 INFO:teuthology.orchestra.run.smithi039.stdout: "device_url": { 2024-03-18T16:20:55.534 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "device_url", 2024-03-18T16:20:55.534 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.534 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.534 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.534 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-18T16:20:55.534 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.534 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.534 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.534 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.534 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.534 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.534 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.535 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.535 INFO:teuthology.orchestra.run.smithi039.stdout: "enabled": { 2024-03-18T16:20:55.535 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "enabled", 2024-03-18T16:20:55.535 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.535 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.535 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.535 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.535 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.535 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.535 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.535 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.535 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.535 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.535 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.535 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.535 INFO:teuthology.orchestra.run.smithi039.stdout: "interval": { 2024-03-18T16:20:55.536 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "interval", 2024-03-18T16:20:55.536 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:55.536 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.536 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.536 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "24", 2024-03-18T16:20:55.536 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "8", 2024-03-18T16:20:55.536 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.536 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.536 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.536 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.536 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.536 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.536 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.536 INFO:teuthology.orchestra.run.smithi039.stdout: "last_opt_revision": { 2024-03-18T16:20:55.537 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "last_opt_revision", 2024-03-18T16:20:55.537 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:55.537 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.537 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.537 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "1", 2024-03-18T16:20:55.537 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.537 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.537 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.537 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.537 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.537 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.537 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.537 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.537 INFO:teuthology.orchestra.run.smithi039.stdout: "leaderboard": { 2024-03-18T16:20:55.537 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "leaderboard", 2024-03-18T16:20:55.538 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.538 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.538 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.538 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.538 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.538 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.538 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.538 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.538 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.538 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.538 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.538 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.538 INFO:teuthology.orchestra.run.smithi039.stdout: "leaderboard_description": { 2024-03-18T16:20:55.538 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "leaderboard_description", 2024-03-18T16:20:55.539 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.539 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.539 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.539 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:55.539 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.539 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.539 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.539 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.539 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.539 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.539 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.539 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.539 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:55.539 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:55.539 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.540 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.540 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.540 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:55.540 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.540 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.540 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:55.540 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:55.540 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:55.540 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:55.540 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:55.540 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:55.540 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:55.540 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:55.540 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.541 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.541 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.541 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.541 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.541 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:55.541 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:55.541 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.541 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.541 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.541 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.541 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.541 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.541 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.541 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.541 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.541 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.542 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.542 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.542 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:55.542 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:55.542 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.542 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.542 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.542 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:55.542 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.542 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.543 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:55.543 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:55.543 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:55.543 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:55.566 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:55.566 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:55.566 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:55.566 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:55.566 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.566 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.566 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.566 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.566 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.567 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:55.567 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:55.567 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.567 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.567 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.567 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.567 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.567 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.567 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.567 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.567 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.567 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.567 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.568 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.568 INFO:teuthology.orchestra.run.smithi039.stdout: "organization": { 2024-03-18T16:20:55.568 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "organization", 2024-03-18T16:20:55.568 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.568 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.568 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.568 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:55.568 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.608 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.608 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.608 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.608 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.608 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.609 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.609 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.609 INFO:teuthology.orchestra.run.smithi039.stdout: "proxy": { 2024-03-18T16:20:55.609 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "proxy", 2024-03-18T16:20:55.609 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.609 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.609 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.609 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:55.609 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.609 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.609 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.609 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.610 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.610 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.610 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.610 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.610 INFO:teuthology.orchestra.run.smithi039.stdout: "url": { 2024-03-18T16:20:55.610 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "url", 2024-03-18T16:20:55.610 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.610 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.610 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.610 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-18T16:20:55.610 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.611 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.611 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.611 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.611 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.611 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.611 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.611 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:55.611 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:55.611 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.611 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:55.611 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "test_orchestrator", 2024-03-18T16:20:55.612 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:55.612 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:55.612 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:55.612 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:55.612 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:55.612 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.612 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.612 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.612 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:55.612 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.613 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.613 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:55.613 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:55.613 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:55.613 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:55.613 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:55.613 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:55.613 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:55.613 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:55.613 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.613 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.613 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.614 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.614 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.614 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:55.614 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:55.614 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.614 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.614 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.614 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.614 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.614 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.614 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.615 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.615 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.615 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.615 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.615 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.615 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:55.615 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:55.615 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.615 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.615 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.617 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:55.617 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.617 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.617 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:55.618 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:55.618 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:55.618 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:55.618 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:55.618 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:55.618 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:55.618 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:55.618 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.618 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.618 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.618 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.618 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.618 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:55.618 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:55.619 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.619 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.619 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.619 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.619 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.619 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.619 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.619 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.633 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.633 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.633 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.634 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:55.634 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:55.634 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.634 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:55.634 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "volumes", 2024-03-18T16:20:55.634 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:55.634 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:55.634 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:55.634 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:55.634 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:55.634 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.634 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.635 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.635 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:55.635 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.635 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.635 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:55.635 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:55.635 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:55.635 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:55.635 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:55.635 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:55.635 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:55.635 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:55.635 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.636 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.636 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.636 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.636 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.636 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:55.636 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:55.636 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.636 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.636 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.636 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.636 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.636 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.636 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.637 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.637 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.637 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.637 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.637 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.637 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:55.637 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:55.637 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.637 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.637 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.637 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:55.638 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.638 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.638 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:55.638 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:55.638 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:55.638 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:55.638 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:55.638 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:55.638 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:55.638 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:55.638 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.638 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.638 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.639 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.639 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.639 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:55.639 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:55.639 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.639 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.639 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.639 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.639 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.639 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.639 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.639 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.639 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.639 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.639 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.639 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.640 INFO:teuthology.orchestra.run.smithi039.stdout: "max_concurrent_clones": { 2024-03-18T16:20:55.640 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "max_concurrent_clones", 2024-03-18T16:20:55.640 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:55.640 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.640 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.640 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "4", 2024-03-18T16:20:55.640 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.640 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.640 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.640 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-18T16:20:55.640 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.641 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.641 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.641 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.641 INFO:teuthology.orchestra.run.smithi039.stdout: "periodic_async_work": { 2024-03-18T16:20:55.641 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "periodic_async_work", 2024-03-18T16:20:55.641 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.641 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.641 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.641 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.641 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.641 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.641 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.641 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Periodically check for async work", 2024-03-18T16:20:55.641 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.642 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.642 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.642 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.642 INFO:teuthology.orchestra.run.smithi039.stdout: "snapshot_clone_delay": { 2024-03-18T16:20:55.642 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "snapshot_clone_delay", 2024-03-18T16:20:55.642 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:55.642 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.642 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.642 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-03-18T16:20:55.642 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.642 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.642 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.643 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-18T16:20:55.643 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.643 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.643 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.643 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.643 INFO:teuthology.orchestra.run.smithi039.stdout: "snapshot_clone_no_wait": { 2024-03-18T16:20:55.643 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "snapshot_clone_no_wait", 2024-03-18T16:20:55.643 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.643 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.643 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.643 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:55.643 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.643 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.644 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.644 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-18T16:20:55.644 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.644 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.644 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.644 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:55.644 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:55.644 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.644 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:55.644 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "zabbix", 2024-03-18T16:20:55.644 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:55.644 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:55.644 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:55.644 INFO:teuthology.orchestra.run.smithi039.stdout: "discovery_interval": { 2024-03-18T16:20:55.645 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "discovery_interval", 2024-03-18T16:20:55.645 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "uint", 2024-03-18T16:20:55.645 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.645 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.645 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "100", 2024-03-18T16:20:55.645 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.645 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.645 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.645 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.645 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.645 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.645 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.645 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.645 INFO:teuthology.orchestra.run.smithi039.stdout: "identifier": { 2024-03-18T16:20:55.646 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "identifier", 2024-03-18T16:20:55.646 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.646 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.646 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.646 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:55.646 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.646 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.646 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.646 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.646 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.646 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.646 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.646 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.646 INFO:teuthology.orchestra.run.smithi039.stdout: "interval": { 2024-03-18T16:20:55.647 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "interval", 2024-03-18T16:20:55.647 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-03-18T16:20:55.647 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.647 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.647 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "60", 2024-03-18T16:20:55.647 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.647 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.668 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.668 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.668 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.668 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.668 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.669 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.669 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:55.669 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:55.669 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.669 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.669 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.669 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:55.669 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.669 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.669 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:55.669 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:55.669 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:55.670 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:55.670 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:55.670 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:55.670 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:55.670 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:55.670 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.670 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.670 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.670 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.670 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.670 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:55.671 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:55.671 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.671 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.671 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.671 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.671 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.671 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.671 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.671 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.671 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.671 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.671 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.671 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.672 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:55.672 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:55.672 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.672 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.672 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.672 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:55.672 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.672 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.672 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:55.672 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:55.672 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:55.672 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:55.672 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:55.672 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:55.673 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:55.673 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:55.673 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.673 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.673 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.673 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.673 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.673 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:55.673 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:55.673 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.673 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.673 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.673 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.673 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.673 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.674 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.674 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.674 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.674 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.674 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.674 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.674 INFO:teuthology.orchestra.run.smithi039.stdout: "zabbix_host": { 2024-03-18T16:20:55.674 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "zabbix_host", 2024-03-18T16:20:55.674 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.674 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.674 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.675 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:55.675 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.675 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.675 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.675 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.675 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.675 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.675 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.675 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.675 INFO:teuthology.orchestra.run.smithi039.stdout: "zabbix_port": { 2024-03-18T16:20:55.675 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "zabbix_port", 2024-03-18T16:20:55.675 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:55.675 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.676 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.676 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "10051", 2024-03-18T16:20:55.676 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.676 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.676 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.676 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.676 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.676 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.676 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.676 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.676 INFO:teuthology.orchestra.run.smithi039.stdout: "zabbix_sender": { 2024-03-18T16:20:55.676 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "zabbix_sender", 2024-03-18T16:20:55.676 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.676 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.676 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.677 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-18T16:20:55.677 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.677 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.677 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.677 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.677 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.677 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.677 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.677 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:55.677 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:55.677 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:55.677 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-03-18T16:20:55.677 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:55.677 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:55.677 INFO:teuthology.orchestra.run.smithi039.stdout: "modules": [ 2024-03-18T16:20:55.678 INFO:teuthology.orchestra.run.smithi039.stdout: "cephadm", 2024-03-18T16:20:55.678 INFO:teuthology.orchestra.run.smithi039.stdout: "dashboard", 2024-03-18T16:20:55.678 INFO:teuthology.orchestra.run.smithi039.stdout: "iostat", 2024-03-18T16:20:55.678 INFO:teuthology.orchestra.run.smithi039.stdout: "nfs", 2024-03-18T16:20:55.678 INFO:teuthology.orchestra.run.smithi039.stdout: "restful" 2024-03-18T16:20:55.678 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:55.678 INFO:teuthology.orchestra.run.smithi039.stdout: "available_modules": [ 2024-03-18T16:20:55.678 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:55.678 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "alerts", 2024-03-18T16:20:55.678 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:55.678 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:55.678 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:55.678 INFO:teuthology.orchestra.run.smithi039.stdout: "interval": { 2024-03-18T16:20:55.678 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "interval", 2024-03-18T16:20:55.679 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-03-18T16:20:55.679 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.679 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.679 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "60", 2024-03-18T16:20:55.679 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.679 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.679 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.679 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "How frequently to reexamine health status", 2024-03-18T16:20:55.679 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.679 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.679 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.679 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.679 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:55.679 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:55.680 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.680 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.680 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.680 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:55.680 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.680 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.680 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:55.680 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:55.680 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:55.680 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:55.680 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:55.680 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:55.680 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:55.680 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:55.680 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.680 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.681 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.681 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.681 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.681 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:55.681 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:55.681 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.681 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.681 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.681 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.681 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.681 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.681 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.681 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.682 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.682 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.682 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.682 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.682 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:55.682 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:55.682 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.682 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.682 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.682 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:55.682 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.682 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.682 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:55.682 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:55.682 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:55.683 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:55.683 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:55.683 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:55.683 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:55.683 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:55.683 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.683 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.683 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.683 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.683 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.683 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:55.683 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:55.683 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.683 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.684 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.684 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.684 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.684 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.684 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.684 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.684 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.684 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.684 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.684 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.684 INFO:teuthology.orchestra.run.smithi039.stdout: "smtp_destination": { 2024-03-18T16:20:55.684 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "smtp_destination", 2024-03-18T16:20:55.685 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.685 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.685 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.685 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:55.685 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.685 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.685 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.685 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Email address to send alerts to", 2024-03-18T16:20:55.685 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.685 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.685 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.686 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.686 INFO:teuthology.orchestra.run.smithi039.stdout: "smtp_from_name": { 2024-03-18T16:20:55.686 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "smtp_from_name", 2024-03-18T16:20:55.686 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.686 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.686 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.686 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "Ceph", 2024-03-18T16:20:55.686 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.686 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.686 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.686 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Email From: name", 2024-03-18T16:20:55.686 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.686 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.686 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.687 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.687 INFO:teuthology.orchestra.run.smithi039.stdout: "smtp_host": { 2024-03-18T16:20:55.687 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "smtp_host", 2024-03-18T16:20:55.687 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.687 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.687 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.687 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:55.687 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.688 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.688 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.688 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "SMTP server", 2024-03-18T16:20:55.688 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.688 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.688 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.688 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.688 INFO:teuthology.orchestra.run.smithi039.stdout: "smtp_password": { 2024-03-18T16:20:55.688 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "smtp_password", 2024-03-18T16:20:55.688 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.688 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.688 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.688 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:55.688 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.689 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.689 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.689 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Password to authenticate with", 2024-03-18T16:20:55.689 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.689 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.689 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.689 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.689 INFO:teuthology.orchestra.run.smithi039.stdout: "smtp_port": { 2024-03-18T16:20:55.689 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "smtp_port", 2024-03-18T16:20:55.689 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:55.689 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.689 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.689 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "465", 2024-03-18T16:20:55.689 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.690 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.690 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.690 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "SMTP port", 2024-03-18T16:20:55.690 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.690 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.690 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.690 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.690 INFO:teuthology.orchestra.run.smithi039.stdout: "smtp_sender": { 2024-03-18T16:20:55.690 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "smtp_sender", 2024-03-18T16:20:55.690 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.690 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.690 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.690 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:55.690 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.691 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.691 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.691 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "SMTP envelope sender", 2024-03-18T16:20:55.691 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.691 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.691 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.691 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.691 INFO:teuthology.orchestra.run.smithi039.stdout: "smtp_ssl": { 2024-03-18T16:20:55.691 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "smtp_ssl", 2024-03-18T16:20:55.691 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.691 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.691 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.692 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:55.692 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.692 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.692 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.709 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-18T16:20:55.709 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.709 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.709 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.709 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.709 INFO:teuthology.orchestra.run.smithi039.stdout: "smtp_user": { 2024-03-18T16:20:55.709 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "smtp_user", 2024-03-18T16:20:55.710 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.710 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.710 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.710 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:55.710 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.710 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.710 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.710 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "User to authenticate as", 2024-03-18T16:20:55.710 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.710 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.710 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.710 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:55.710 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:55.710 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.711 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:55.711 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "balancer", 2024-03-18T16:20:55.711 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:55.711 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:55.711 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:55.711 INFO:teuthology.orchestra.run.smithi039.stdout: "active": { 2024-03-18T16:20:55.711 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "active", 2024-03-18T16:20:55.711 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.711 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.711 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.711 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:55.711 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.711 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.712 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.712 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "automatically balance PGs across cluster", 2024-03-18T16:20:55.712 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.712 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.712 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.712 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.712 INFO:teuthology.orchestra.run.smithi039.stdout: "begin_time": { 2024-03-18T16:20:55.712 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "begin_time", 2024-03-18T16:20:55.712 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.712 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.712 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.712 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0000", 2024-03-18T16:20:55.713 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.713 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.713 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.713 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "beginning time of day to automatically balance", 2024-03-18T16:20:55.713 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-18T16:20:55.713 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.713 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.713 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.713 INFO:teuthology.orchestra.run.smithi039.stdout: "begin_weekday": { 2024-03-18T16:20:55.713 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "begin_weekday", 2024-03-18T16:20:55.713 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "uint", 2024-03-18T16:20:55.713 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.713 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.714 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-03-18T16:20:55.714 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "0", 2024-03-18T16:20:55.714 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "6", 2024-03-18T16:20:55.714 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.714 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-18T16:20:55.714 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-18T16:20:55.714 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.714 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.714 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.714 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_compat_max_iterations": { 2024-03-18T16:20:55.714 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "crush_compat_max_iterations", 2024-03-18T16:20:55.715 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "uint", 2024-03-18T16:20:55.715 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.715 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.715 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "25", 2024-03-18T16:20:55.715 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "1", 2024-03-18T16:20:55.715 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "250", 2024-03-18T16:20:55.715 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.715 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-18T16:20:55.715 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.715 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.715 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.716 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.716 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_compat_metrics": { 2024-03-18T16:20:55.716 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "crush_compat_metrics", 2024-03-18T16:20:55.716 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.716 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.716 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.716 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "pgs,objects,bytes", 2024-03-18T16:20:55.716 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.716 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.716 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.716 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-18T16:20:55.716 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-18T16:20:55.717 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.717 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.717 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.717 INFO:teuthology.orchestra.run.smithi039.stdout: "crush_compat_step": { 2024-03-18T16:20:55.717 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "crush_compat_step", 2024-03-18T16:20:55.717 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "float", 2024-03-18T16:20:55.717 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.717 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.717 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0.5", 2024-03-18T16:20:55.717 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "0.001", 2024-03-18T16:20:55.717 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "0.999", 2024-03-18T16:20:55.717 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.717 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "aggressiveness of optimization", 2024-03-18T16:20:55.717 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-18T16:20:55.718 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.718 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.718 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.718 INFO:teuthology.orchestra.run.smithi039.stdout: "end_time": { 2024-03-18T16:20:55.718 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "end_time", 2024-03-18T16:20:55.718 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.718 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.718 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.718 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "2359", 2024-03-18T16:20:55.718 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.718 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.718 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.718 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "ending time of day to automatically balance", 2024-03-18T16:20:55.718 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-18T16:20:55.719 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.719 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.719 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.719 INFO:teuthology.orchestra.run.smithi039.stdout: "end_weekday": { 2024-03-18T16:20:55.719 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "end_weekday", 2024-03-18T16:20:55.719 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "uint", 2024-03-18T16:20:55.719 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.719 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.719 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-03-18T16:20:55.719 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "0", 2024-03-18T16:20:55.719 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "6", 2024-03-18T16:20:55.719 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.719 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-18T16:20:55.720 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-18T16:20:55.720 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.720 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.720 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.720 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:55.720 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:55.720 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.720 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.720 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.720 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:55.720 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.720 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.720 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:55.721 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:55.721 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:55.721 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:55.721 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:55.721 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:55.721 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:55.721 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:55.721 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.721 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.721 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.721 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.721 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.721 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:55.721 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:55.721 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.722 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.722 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.722 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.722 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.722 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.722 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.722 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.722 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.722 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.722 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.722 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.722 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:55.722 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:55.723 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.723 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.723 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.723 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:55.723 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.724 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.724 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:55.724 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:55.724 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:55.755 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:55.773 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:55.773 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:55.773 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:55.773 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:55.773 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.773 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.773 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.773 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.773 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.774 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:55.774 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:55.774 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.774 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.774 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.774 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.774 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.774 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.774 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.774 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.774 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.774 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.774 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.774 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.775 INFO:teuthology.orchestra.run.smithi039.stdout: "min_score": { 2024-03-18T16:20:55.775 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "min_score", 2024-03-18T16:20:55.775 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "float", 2024-03-18T16:20:55.775 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.775 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.775 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-03-18T16:20:55.775 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.775 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.775 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.775 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-18T16:20:55.775 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.775 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.775 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.775 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.775 INFO:teuthology.orchestra.run.smithi039.stdout: "mode": { 2024-03-18T16:20:55.776 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "mode", 2024-03-18T16:20:55.776 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.776 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.776 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.776 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "upmap", 2024-03-18T16:20:55.776 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.776 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.776 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:55.776 INFO:teuthology.orchestra.run.smithi039.stdout: "crush-compat", 2024-03-18T16:20:55.776 INFO:teuthology.orchestra.run.smithi039.stdout: "none", 2024-03-18T16:20:55.776 INFO:teuthology.orchestra.run.smithi039.stdout: "read", 2024-03-18T16:20:55.776 INFO:teuthology.orchestra.run.smithi039.stdout: "upmap", 2024-03-18T16:20:55.776 INFO:teuthology.orchestra.run.smithi039.stdout: "upmap-read" 2024-03-18T16:20:55.776 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:55.776 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Balancer mode", 2024-03-18T16:20:55.777 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.777 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.777 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.777 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.777 INFO:teuthology.orchestra.run.smithi039.stdout: "pool_ids": { 2024-03-18T16:20:55.777 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "pool_ids", 2024-03-18T16:20:55.777 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.777 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.777 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.777 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:55.777 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.777 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.777 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.777 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-18T16:20:55.777 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.777 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.778 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.778 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.778 INFO:teuthology.orchestra.run.smithi039.stdout: "sleep_interval": { 2024-03-18T16:20:55.778 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sleep_interval", 2024-03-18T16:20:55.778 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-03-18T16:20:55.778 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.778 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.778 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "60", 2024-03-18T16:20:55.778 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.778 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.778 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.778 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-18T16:20:55.778 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.778 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.779 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.779 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.779 INFO:teuthology.orchestra.run.smithi039.stdout: "upmap_max_deviation": { 2024-03-18T16:20:55.779 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "upmap_max_deviation", 2024-03-18T16:20:55.779 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:55.779 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.779 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.779 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "5", 2024-03-18T16:20:55.779 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "1", 2024-03-18T16:20:55.779 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.779 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.779 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-18T16:20:55.779 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-18T16:20:55.779 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.780 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.780 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.780 INFO:teuthology.orchestra.run.smithi039.stdout: "upmap_max_optimizations": { 2024-03-18T16:20:55.780 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "upmap_max_optimizations", 2024-03-18T16:20:55.780 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "uint", 2024-03-18T16:20:55.780 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.780 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.780 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "10", 2024-03-18T16:20:55.780 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.780 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.780 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.780 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-18T16:20:55.780 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.780 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.780 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.780 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:55.781 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:55.781 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.781 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:55.781 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "cephadm", 2024-03-18T16:20:55.781 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:55.781 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:55.781 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:55.781 INFO:teuthology.orchestra.run.smithi039.stdout: "agent_down_multiplier": { 2024-03-18T16:20:55.781 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "agent_down_multiplier", 2024-03-18T16:20:55.781 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "float", 2024-03-18T16:20:55.781 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.781 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.781 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "3.0", 2024-03-18T16:20:55.781 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.781 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.781 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.782 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-18T16:20:55.782 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.782 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.782 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.782 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.782 INFO:teuthology.orchestra.run.smithi039.stdout: "agent_refresh_rate": { 2024-03-18T16:20:55.782 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "agent_refresh_rate", 2024-03-18T16:20:55.782 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-03-18T16:20:55.782 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.782 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.782 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "20", 2024-03-18T16:20:55.782 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.782 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.782 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.783 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-18T16:20:55.783 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.783 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.783 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.783 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.783 INFO:teuthology.orchestra.run.smithi039.stdout: "agent_starting_port": { 2024-03-18T16:20:55.783 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "agent_starting_port", 2024-03-18T16:20:55.783 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:55.783 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.783 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.783 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "4721", 2024-03-18T16:20:55.783 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.783 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.783 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.783 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-18T16:20:55.784 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.784 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.784 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.784 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.784 INFO:teuthology.orchestra.run.smithi039.stdout: "allow_ptrace": { 2024-03-18T16:20:55.784 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "allow_ptrace", 2024-03-18T16:20:55.784 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.784 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.784 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.784 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.784 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.784 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.784 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.784 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-18T16:20:55.785 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-18T16:20:55.785 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.785 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.785 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.785 INFO:teuthology.orchestra.run.smithi039.stdout: "autotune_interval": { 2024-03-18T16:20:55.785 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "autotune_interval", 2024-03-18T16:20:55.785 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-03-18T16:20:55.785 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.785 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.785 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "600", 2024-03-18T16:20:55.785 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.785 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.785 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.785 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-18T16:20:55.785 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.786 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.786 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.786 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.786 INFO:teuthology.orchestra.run.smithi039.stdout: "autotune_memory_target_ratio": { 2024-03-18T16:20:55.786 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "autotune_memory_target_ratio", 2024-03-18T16:20:55.786 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "float", 2024-03-18T16:20:55.786 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.786 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.786 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0.7", 2024-03-18T16:20:55.786 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.786 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.786 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.786 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-18T16:20:55.786 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.786 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.786 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.787 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.787 INFO:teuthology.orchestra.run.smithi039.stdout: "cephadm_log_destination": { 2024-03-18T16:20:55.787 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "cephadm_log_destination", 2024-03-18T16:20:55.787 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.787 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.787 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.787 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:55.787 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.787 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.787 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:55.787 INFO:teuthology.orchestra.run.smithi039.stdout: "file", 2024-03-18T16:20:55.788 INFO:teuthology.orchestra.run.smithi039.stdout: "file,syslog", 2024-03-18T16:20:55.788 INFO:teuthology.orchestra.run.smithi039.stdout: "syslog" 2024-03-18T16:20:55.788 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:55.788 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-18T16:20:55.788 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.788 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.788 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.788 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.788 INFO:teuthology.orchestra.run.smithi039.stdout: "cgroups_split": { 2024-03-18T16:20:55.788 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "cgroups_split", 2024-03-18T16:20:55.788 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.788 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.788 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.789 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:55.789 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.789 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.789 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.789 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-18T16:20:55.789 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.789 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.789 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.789 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.789 INFO:teuthology.orchestra.run.smithi039.stdout: "config_checks_enabled": { 2024-03-18T16:20:55.789 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "config_checks_enabled", 2024-03-18T16:20:55.789 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.789 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.789 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.790 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.790 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.790 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.790 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.790 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-18T16:20:55.790 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.790 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.790 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.790 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.790 INFO:teuthology.orchestra.run.smithi039.stdout: "config_dashboard": { 2024-03-18T16:20:55.790 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "config_dashboard", 2024-03-18T16:20:55.790 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.790 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.791 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.791 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:55.791 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.791 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.791 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.791 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-18T16:20:55.791 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.791 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.791 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.791 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.791 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_alertmanager": { 2024-03-18T16:20:55.791 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_alertmanager", 2024-03-18T16:20:55.791 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.791 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.791 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.792 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-18T16:20:55.792 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.792 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.792 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.792 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Prometheus container image", 2024-03-18T16:20:55.792 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.792 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.792 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.792 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.792 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_base": { 2024-03-18T16:20:55.792 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_base", 2024-03-18T16:20:55.792 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.792 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.792 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.792 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-18T16:20:55.793 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.793 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.793 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.793 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Container image name, without the tag", 2024-03-18T16:20:55.793 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.793 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.793 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.793 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.793 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_elasticsearch": { 2024-03-18T16:20:55.793 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_elasticsearch", 2024-03-18T16:20:55.793 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.793 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.793 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.794 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-18T16:20:55.794 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.794 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.794 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.794 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "elasticsearch container image", 2024-03-18T16:20:55.794 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.794 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.794 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.794 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.794 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_grafana": { 2024-03-18T16:20:55.794 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_grafana", 2024-03-18T16:20:55.794 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.794 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.794 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.795 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-18T16:20:55.795 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.795 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.795 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.795 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Prometheus container image", 2024-03-18T16:20:55.795 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.795 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.795 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.795 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.795 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_haproxy": { 2024-03-18T16:20:55.795 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_haproxy", 2024-03-18T16:20:55.795 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.796 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.796 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.796 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-18T16:20:55.796 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.796 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.796 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.796 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "HAproxy container image", 2024-03-18T16:20:55.796 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.796 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.796 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.797 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.797 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_jaeger_agent": { 2024-03-18T16:20:55.797 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_jaeger_agent", 2024-03-18T16:20:55.797 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.797 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.797 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.797 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-18T16:20:55.797 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.797 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.797 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.797 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Jaeger agent container image", 2024-03-18T16:20:55.797 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.797 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.797 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.797 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.798 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_jaeger_collector": { 2024-03-18T16:20:55.798 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_jaeger_collector", 2024-03-18T16:20:55.798 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.798 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.798 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.798 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-18T16:20:55.798 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.798 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.798 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.798 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Jaeger collector container image", 2024-03-18T16:20:55.798 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.798 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.798 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.799 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.799 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_jaeger_query": { 2024-03-18T16:20:55.799 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_jaeger_query", 2024-03-18T16:20:55.799 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.799 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.799 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.799 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-18T16:20:55.799 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.799 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.799 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.799 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Jaeger query container image", 2024-03-18T16:20:55.799 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.799 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.799 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.800 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.800 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_keepalived": { 2024-03-18T16:20:55.800 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_keepalived", 2024-03-18T16:20:55.800 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.800 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.800 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.800 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-18T16:20:55.800 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.800 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.800 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.800 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Keepalived container image", 2024-03-18T16:20:55.800 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.800 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.800 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.801 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.801 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_loki": { 2024-03-18T16:20:55.807 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_loki", 2024-03-18T16:20:55.808 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.808 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.808 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.808 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-18T16:20:55.808 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.808 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.808 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.808 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Loki container image", 2024-03-18T16:20:55.808 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.808 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.808 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.808 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.809 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_node_exporter": { 2024-03-18T16:20:55.809 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_node_exporter", 2024-03-18T16:20:55.809 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.809 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.809 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.809 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-18T16:20:55.809 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.809 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.809 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.810 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Prometheus container image", 2024-03-18T16:20:55.810 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.810 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.810 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.810 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.810 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_nvmeof": { 2024-03-18T16:20:55.810 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_nvmeof", 2024-03-18T16:20:55.810 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.810 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.810 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.810 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-18T16:20:55.811 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.811 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.811 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.811 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Nvme-of container image", 2024-03-18T16:20:55.811 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.811 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.811 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.811 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.811 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_prometheus": { 2024-03-18T16:20:55.811 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_prometheus", 2024-03-18T16:20:55.812 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.812 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.812 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.812 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-18T16:20:55.812 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.812 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.812 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.812 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Prometheus container image", 2024-03-18T16:20:55.812 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.812 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.812 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.812 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.812 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_promtail": { 2024-03-18T16:20:55.812 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_promtail", 2024-03-18T16:20:55.812 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.813 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.813 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.813 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-18T16:20:55.813 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.813 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.813 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.813 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Promtail container image", 2024-03-18T16:20:55.813 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.813 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.813 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.814 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.814 INFO:teuthology.orchestra.run.smithi039.stdout: "container_image_snmp_gateway": { 2024-03-18T16:20:55.814 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_image_snmp_gateway", 2024-03-18T16:20:55.814 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.814 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.814 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.815 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-18T16:20:55.815 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.815 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.815 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.815 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "SNMP Gateway container image", 2024-03-18T16:20:55.815 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.815 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.815 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.815 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.815 INFO:teuthology.orchestra.run.smithi039.stdout: "container_init": { 2024-03-18T16:20:55.816 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "container_init", 2024-03-18T16:20:55.816 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.816 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.816 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.816 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:55.816 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.816 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.816 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.816 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Run podman/docker with `--init`", 2024-03-18T16:20:55.816 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.816 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.816 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.816 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.816 INFO:teuthology.orchestra.run.smithi039.stdout: "daemon_cache_timeout": { 2024-03-18T16:20:55.816 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "daemon_cache_timeout", 2024-03-18T16:20:55.817 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-03-18T16:20:55.817 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.817 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.817 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "600", 2024-03-18T16:20:55.817 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.817 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.817 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.817 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-18T16:20:55.817 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.817 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.817 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.817 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.817 INFO:teuthology.orchestra.run.smithi039.stdout: "default_cephadm_command_timeout": { 2024-03-18T16:20:55.817 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "default_cephadm_command_timeout", 2024-03-18T16:20:55.818 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:55.818 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.818 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.818 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "900", 2024-03-18T16:20:55.818 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.818 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.818 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.818 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-18T16:20:55.818 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.818 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.818 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.818 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.818 INFO:teuthology.orchestra.run.smithi039.stdout: "default_registry": { 2024-03-18T16:20:55.818 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "default_registry", 2024-03-18T16:20:55.819 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.819 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.819 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.819 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "docker.io", 2024-03-18T16:20:55.819 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.819 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.819 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.819 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-18T16:20:55.819 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.819 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.819 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.819 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.819 INFO:teuthology.orchestra.run.smithi039.stdout: "device_cache_timeout": { 2024-03-18T16:20:55.820 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "device_cache_timeout", 2024-03-18T16:20:55.820 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-03-18T16:20:55.820 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.820 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.820 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "1800", 2024-03-18T16:20:55.820 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.820 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.820 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.820 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "seconds to cache device inventory", 2024-03-18T16:20:55.820 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.820 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.820 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.820 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.820 INFO:teuthology.orchestra.run.smithi039.stdout: "device_enhanced_scan": { 2024-03-18T16:20:55.821 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "device_enhanced_scan", 2024-03-18T16:20:55.821 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.821 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.821 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.821 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.821 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.821 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.821 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.821 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-18T16:20:55.821 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.821 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.821 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.821 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.821 INFO:teuthology.orchestra.run.smithi039.stdout: "facts_cache_timeout": { 2024-03-18T16:20:55.821 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "facts_cache_timeout", 2024-03-18T16:20:55.822 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-03-18T16:20:55.822 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.822 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.822 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "60", 2024-03-18T16:20:55.822 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.822 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.822 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.822 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "seconds to cache host facts data", 2024-03-18T16:20:55.822 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.822 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.822 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.822 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.822 INFO:teuthology.orchestra.run.smithi039.stdout: "host_check_interval": { 2024-03-18T16:20:55.822 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "host_check_interval", 2024-03-18T16:20:55.822 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-03-18T16:20:55.823 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.823 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.823 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "600", 2024-03-18T16:20:55.823 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.823 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.823 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.823 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "how frequently to perform a host check", 2024-03-18T16:20:55.823 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.823 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.823 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.823 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.823 INFO:teuthology.orchestra.run.smithi039.stdout: "hw_monitoring": { 2024-03-18T16:20:55.823 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "hw_monitoring", 2024-03-18T16:20:55.823 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.824 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.824 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.824 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.824 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.824 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.824 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.824 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-18T16:20:55.824 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.824 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.824 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.824 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.824 INFO:teuthology.orchestra.run.smithi039.stdout: "inventory_list_all": { 2024-03-18T16:20:55.824 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "inventory_list_all", 2024-03-18T16:20:55.824 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.825 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.825 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.825 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.825 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.825 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.825 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.825 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-18T16:20:55.825 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.825 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.825 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.825 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.825 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:55.825 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:55.825 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.825 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.826 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.826 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:55.826 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.826 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.826 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:55.826 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:55.826 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:55.826 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:55.826 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:55.826 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:55.826 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:55.826 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:55.826 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.826 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.826 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.827 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.827 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.827 INFO:teuthology.orchestra.run.smithi039.stdout: "log_refresh_metadata": { 2024-03-18T16:20:55.827 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_refresh_metadata", 2024-03-18T16:20:55.827 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.827 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.827 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.827 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.827 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.827 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.827 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.827 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-18T16:20:55.827 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.827 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.828 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.828 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.828 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:55.828 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:55.828 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.828 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.828 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.828 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:55.828 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.828 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.828 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.828 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-18T16:20:55.828 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.829 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.829 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.829 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.829 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:55.829 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:55.829 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.829 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.829 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.829 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:55.829 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.829 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.829 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:55.829 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:55.829 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:55.830 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:55.830 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:55.830 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:55.830 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:55.830 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:55.830 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.830 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.830 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.830 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.830 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.830 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:55.830 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:55.830 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.830 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.831 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.831 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.857 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.857 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.857 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.857 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.857 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.857 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.857 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.858 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.858 INFO:teuthology.orchestra.run.smithi039.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-18T16:20:55.858 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-18T16:20:55.858 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.858 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.858 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.858 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.858 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.858 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.858 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.858 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-18T16:20:55.858 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.859 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.859 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.859 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.859 INFO:teuthology.orchestra.run.smithi039.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-18T16:20:55.859 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-18T16:20:55.859 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.859 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.859 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.859 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "*", 2024-03-18T16:20:55.859 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.859 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.859 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.859 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-18T16:20:55.860 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.860 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.860 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.860 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.860 INFO:teuthology.orchestra.run.smithi039.stdout: "max_count_per_host": { 2024-03-18T16:20:55.860 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "max_count_per_host", 2024-03-18T16:20:55.860 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:55.860 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.860 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.860 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "10", 2024-03-18T16:20:55.860 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.860 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.860 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.861 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "max number of daemons per service per host", 2024-03-18T16:20:55.861 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.861 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.861 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.861 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.861 INFO:teuthology.orchestra.run.smithi039.stdout: "max_osd_draining_count": { 2024-03-18T16:20:55.861 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "max_osd_draining_count", 2024-03-18T16:20:55.861 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:55.861 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.861 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.861 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "10", 2024-03-18T16:20:55.861 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.862 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.862 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.862 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-18T16:20:55.862 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.862 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.862 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.862 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.862 INFO:teuthology.orchestra.run.smithi039.stdout: "migration_current": { 2024-03-18T16:20:55.862 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "migration_current", 2024-03-18T16:20:55.862 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:55.862 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.862 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.863 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:55.863 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.863 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.863 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.863 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "internal - do not modify", 2024-03-18T16:20:55.863 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.863 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.863 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.863 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.863 INFO:teuthology.orchestra.run.smithi039.stdout: "mode": { 2024-03-18T16:20:55.863 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "mode", 2024-03-18T16:20:55.863 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.863 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.864 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.864 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "root", 2024-03-18T16:20:55.864 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.864 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.864 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:55.864 INFO:teuthology.orchestra.run.smithi039.stdout: "cephadm-package", 2024-03-18T16:20:55.864 INFO:teuthology.orchestra.run.smithi039.stdout: "root" 2024-03-18T16:20:55.864 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:55.864 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "mode for remote execution of cephadm", 2024-03-18T16:20:55.869 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.870 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.870 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.870 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.870 INFO:teuthology.orchestra.run.smithi039.stdout: "oob_default_addr": { 2024-03-18T16:20:55.870 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "oob_default_addr", 2024-03-18T16:20:55.870 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.870 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.870 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.871 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "169.254.1.1", 2024-03-18T16:20:55.871 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.871 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.871 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.871 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-18T16:20:55.871 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.871 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.871 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.871 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.871 INFO:teuthology.orchestra.run.smithi039.stdout: "prometheus_alerts_path": { 2024-03-18T16:20:55.871 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "prometheus_alerts_path", 2024-03-18T16:20:55.871 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.872 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.872 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.872 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-18T16:20:55.872 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.872 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.872 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.872 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-18T16:20:55.872 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.872 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.872 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.872 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.872 INFO:teuthology.orchestra.run.smithi039.stdout: "registry_insecure": { 2024-03-18T16:20:55.872 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "registry_insecure", 2024-03-18T16:20:55.872 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.872 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.873 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.873 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.873 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.873 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.873 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.873 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-18T16:20:55.873 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.873 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.873 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.873 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.873 INFO:teuthology.orchestra.run.smithi039.stdout: "registry_password": { 2024-03-18T16:20:55.873 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "registry_password", 2024-03-18T16:20:55.873 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.873 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.873 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.874 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:55.874 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.874 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.874 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.874 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-18T16:20:55.874 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.874 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.874 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.874 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.874 INFO:teuthology.orchestra.run.smithi039.stdout: "registry_url": { 2024-03-18T16:20:55.875 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "registry_url", 2024-03-18T16:20:55.875 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.875 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.875 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.875 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:55.875 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.875 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.875 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.875 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-18T16:20:55.875 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.875 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.875 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.875 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.875 INFO:teuthology.orchestra.run.smithi039.stdout: "registry_username": { 2024-03-18T16:20:55.875 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "registry_username", 2024-03-18T16:20:55.876 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.876 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.876 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.876 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:55.876 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.876 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.876 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.876 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-18T16:20:55.876 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.876 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.876 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.876 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.876 INFO:teuthology.orchestra.run.smithi039.stdout: "secure_monitoring_stack": { 2024-03-18T16:20:55.877 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "secure_monitoring_stack", 2024-03-18T16:20:55.877 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.877 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.877 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.877 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.877 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.877 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.882 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.882 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-18T16:20:55.882 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.883 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.883 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.883 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.883 INFO:teuthology.orchestra.run.smithi039.stdout: "service_discovery_port": { 2024-03-18T16:20:55.883 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "service_discovery_port", 2024-03-18T16:20:55.883 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:55.883 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.883 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.883 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "8765", 2024-03-18T16:20:55.883 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.884 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.884 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.884 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "cephadm service discovery port", 2024-03-18T16:20:55.884 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.884 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.884 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.884 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.884 INFO:teuthology.orchestra.run.smithi039.stdout: "ssh_config_file": { 2024-03-18T16:20:55.884 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "ssh_config_file", 2024-03-18T16:20:55.884 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.885 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.885 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.885 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:55.885 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.885 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.885 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.885 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-18T16:20:55.885 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.885 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.885 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.885 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.886 INFO:teuthology.orchestra.run.smithi039.stdout: "use_agent": { 2024-03-18T16:20:55.886 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "use_agent", 2024-03-18T16:20:55.886 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.886 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.886 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.886 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.886 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.886 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.886 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.886 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-18T16:20:55.886 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.886 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.886 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.887 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.887 INFO:teuthology.orchestra.run.smithi039.stdout: "use_repo_digest": { 2024-03-18T16:20:55.887 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "use_repo_digest", 2024-03-18T16:20:55.887 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.887 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.887 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.887 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:55.887 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.887 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.887 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.887 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-18T16:20:55.887 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.887 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.887 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.887 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.888 INFO:teuthology.orchestra.run.smithi039.stdout: "warn_on_failed_host_check": { 2024-03-18T16:20:55.888 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "warn_on_failed_host_check", 2024-03-18T16:20:55.888 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.888 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.888 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.888 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:55.888 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.888 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.888 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.888 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "raise a health warning if the host check fails", 2024-03-18T16:20:55.888 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.888 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.888 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.888 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.888 INFO:teuthology.orchestra.run.smithi039.stdout: "warn_on_stray_daemons": { 2024-03-18T16:20:55.889 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "warn_on_stray_daemons", 2024-03-18T16:20:55.889 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.889 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.889 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.889 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:55.889 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.889 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.889 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.889 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-18T16:20:55.889 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.889 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.889 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.889 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.889 INFO:teuthology.orchestra.run.smithi039.stdout: "warn_on_stray_hosts": { 2024-03-18T16:20:55.889 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "warn_on_stray_hosts", 2024-03-18T16:20:55.890 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.890 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.890 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.890 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:55.890 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.890 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.890 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.890 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-18T16:20:55.890 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.890 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.890 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.890 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:55.890 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:55.890 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.891 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:55.891 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "crash", 2024-03-18T16:20:55.891 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:55.891 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:55.891 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:55.891 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:55.891 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:55.891 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.891 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.891 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.892 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:55.892 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.892 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.892 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:55.892 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:55.892 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:55.892 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:55.892 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:55.892 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:55.892 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:55.893 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:55.893 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.893 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.893 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.893 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.893 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.893 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:55.893 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:55.893 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.894 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.894 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.894 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.894 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.894 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.894 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.894 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.894 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.894 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.894 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.894 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.894 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:55.895 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:55.895 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.895 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.895 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.895 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:55.895 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.895 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.895 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:55.895 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:55.895 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:55.895 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:55.895 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:55.895 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:55.895 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:55.895 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:55.896 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.896 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.896 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.896 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.896 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.896 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:55.896 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:55.896 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.896 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.896 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.896 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.896 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.896 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.896 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.897 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.897 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.897 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.897 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.897 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.897 INFO:teuthology.orchestra.run.smithi039.stdout: "retain_interval": { 2024-03-18T16:20:55.897 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "retain_interval", 2024-03-18T16:20:55.897 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-03-18T16:20:55.897 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.897 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.897 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "31536000", 2024-03-18T16:20:55.898 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.898 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.898 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.898 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-18T16:20:55.898 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.898 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.898 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.898 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.898 INFO:teuthology.orchestra.run.smithi039.stdout: "warn_recent_interval": { 2024-03-18T16:20:55.898 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "warn_recent_interval", 2024-03-18T16:20:55.898 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-03-18T16:20:55.899 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.899 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:55.899 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "1209600", 2024-03-18T16:20:55.899 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.899 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.899 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.899 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-18T16:20:55.899 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.899 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.899 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.899 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:55.899 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:55.899 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.899 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:55.899 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "dashboard", 2024-03-18T16:20:55.899 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:55.900 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:55.900 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:55.900 INFO:teuthology.orchestra.run.smithi039.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-18T16:20:55.900 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-18T16:20:55.900 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:55.900 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.900 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.900 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "10", 2024-03-18T16:20:55.900 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.900 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.900 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.900 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.900 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.900 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.900 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.901 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.901 INFO:teuthology.orchestra.run.smithi039.stdout: "ALERTMANAGER_API_HOST": { 2024-03-18T16:20:55.901 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-18T16:20:55.901 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.901 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.901 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.901 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:55.901 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.901 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.901 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.901 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.901 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.901 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.901 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.901 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.902 INFO:teuthology.orchestra.run.smithi039.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-18T16:20:55.902 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-18T16:20:55.902 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.902 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.902 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.902 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:55.902 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.902 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.902 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.902 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.902 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.902 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.902 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.902 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.903 INFO:teuthology.orchestra.run.smithi039.stdout: "AUDIT_API_ENABLED": { 2024-03-18T16:20:55.903 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "AUDIT_API_ENABLED", 2024-03-18T16:20:55.903 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.903 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.903 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.903 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.903 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.903 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.903 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.903 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.903 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.903 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.904 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.904 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.904 INFO:teuthology.orchestra.run.smithi039.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-18T16:20:55.904 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-18T16:20:55.904 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.904 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.904 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.904 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:55.904 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.904 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.904 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.904 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.904 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.904 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.905 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.905 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.905 INFO:teuthology.orchestra.run.smithi039.stdout: "ENABLE_BROWSABLE_API": { 2024-03-18T16:20:55.905 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-18T16:20:55.905 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.905 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.905 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.905 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:55.905 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.905 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.905 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.905 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.905 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.905 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.906 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.906 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.906 INFO:teuthology.orchestra.run.smithi039.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-18T16:20:55.906 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-18T16:20:55.906 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.906 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.906 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.906 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:55.906 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.906 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.906 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.906 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.906 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.906 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.906 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.907 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.907 INFO:teuthology.orchestra.run.smithi039.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-18T16:20:55.907 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-18T16:20:55.907 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.907 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.932 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.933 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:55.933 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.933 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.933 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.933 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.933 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.933 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.933 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.933 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.933 INFO:teuthology.orchestra.run.smithi039.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-18T16:20:55.934 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-18T16:20:55.934 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.934 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.934 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.934 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:55.934 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.934 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.934 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.934 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.934 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.934 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.934 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.935 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.935 INFO:teuthology.orchestra.run.smithi039.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-18T16:20:55.935 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-18T16:20:55.935 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.935 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.935 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.935 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:55.935 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.935 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.935 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.935 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.935 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.935 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.936 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.936 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.936 INFO:teuthology.orchestra.run.smithi039.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-18T16:20:55.936 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-18T16:20:55.936 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.936 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.936 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.936 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:55.936 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.936 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.936 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.936 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.936 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.937 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.937 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.937 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.937 INFO:teuthology.orchestra.run.smithi039.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-18T16:20:55.937 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-18T16:20:55.937 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.937 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.937 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.937 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:55.937 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.938 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.938 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.938 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.938 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.938 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.938 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.938 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.938 INFO:teuthology.orchestra.run.smithi039.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-18T16:20:55.939 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-18T16:20:55.939 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.939 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.939 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.939 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:55.939 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.940 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.940 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.986 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.986 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.986 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.986 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.986 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.986 INFO:teuthology.orchestra.run.smithi039.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-18T16:20:55.986 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-18T16:20:55.986 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.986 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.987 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.987 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:55.987 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.987 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.987 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.987 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.987 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.987 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.987 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.987 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.987 INFO:teuthology.orchestra.run.smithi039.stdout: "GRAFANA_API_PASSWORD": { 2024-03-18T16:20:55.987 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-18T16:20:55.987 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.988 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.988 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.988 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "admin", 2024-03-18T16:20:55.988 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.988 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.988 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.988 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.988 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.988 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.988 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.988 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.988 INFO:teuthology.orchestra.run.smithi039.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-18T16:20:55.988 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-18T16:20:55.988 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.989 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.989 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.989 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:55.989 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.989 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.989 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.989 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.989 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.989 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.989 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.989 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.989 INFO:teuthology.orchestra.run.smithi039.stdout: "GRAFANA_API_URL": { 2024-03-18T16:20:55.989 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "GRAFANA_API_URL", 2024-03-18T16:20:55.989 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.990 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.990 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.990 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:55.990 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.990 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.990 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.990 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.990 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.990 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.990 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.990 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.990 INFO:teuthology.orchestra.run.smithi039.stdout: "GRAFANA_API_USERNAME": { 2024-03-18T16:20:55.990 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-18T16:20:55.990 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.990 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.991 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.991 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "admin", 2024-03-18T16:20:55.991 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.991 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.991 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.991 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.991 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.991 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.991 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.991 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.991 INFO:teuthology.orchestra.run.smithi039.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-18T16:20:55.991 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-18T16:20:55.991 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.991 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.991 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.992 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:55.992 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.992 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.992 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.992 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.992 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.992 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.992 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.992 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.992 INFO:teuthology.orchestra.run.smithi039.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-18T16:20:55.992 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-18T16:20:55.992 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.992 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.992 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.992 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.993 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.993 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.993 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.993 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.993 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.993 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.993 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.993 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.993 INFO:teuthology.orchestra.run.smithi039.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-18T16:20:55.993 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-18T16:20:55.993 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.993 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.993 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.993 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:55.994 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.994 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.994 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.994 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.994 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.994 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.994 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.994 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.994 INFO:teuthology.orchestra.run.smithi039.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-18T16:20:55.994 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-18T16:20:55.994 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.994 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.994 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.994 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:55.995 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.995 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.995 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.995 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.995 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.995 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.995 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.995 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.995 INFO:teuthology.orchestra.run.smithi039.stdout: "MULTICLUSTER_CONFIG": { 2024-03-18T16:20:55.995 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-18T16:20:55.995 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.995 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.995 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.995 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "{}", 2024-03-18T16:20:55.996 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.996 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.996 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.996 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.996 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.996 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.996 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.996 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.996 INFO:teuthology.orchestra.run.smithi039.stdout: "PROMETHEUS_API_HOST": { 2024-03-18T16:20:55.996 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-18T16:20:55.996 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:55.996 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.996 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.996 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:55.996 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.997 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.997 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.997 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.997 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.997 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.997 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.997 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.997 INFO:teuthology.orchestra.run.smithi039.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-18T16:20:55.997 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-18T16:20:55.997 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.997 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.997 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.997 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:55.997 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.998 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.998 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.998 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.998 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.998 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.998 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.998 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.998 INFO:teuthology.orchestra.run.smithi039.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-18T16:20:55.998 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-18T16:20:55.998 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.998 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.998 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.998 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.998 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.999 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:55.999 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:55.999 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:55.999 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:55.999 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:55.999 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:55.999 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:55.999 INFO:teuthology.orchestra.run.smithi039.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-18T16:20:55.999 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-18T16:20:55.999 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:55.999 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:55.999 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:55.999 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:55.999 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:55.999 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.000 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.000 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.000 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.000 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.000 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.000 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.000 INFO:teuthology.orchestra.run.smithi039.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-18T16:20:56.000 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-18T16:20:56.000 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.000 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.000 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.000 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:56.000 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.000 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.000 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.001 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.001 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.001 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.001 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.001 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.001 INFO:teuthology.orchestra.run.smithi039.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-18T16:20:56.001 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-18T16:20:56.001 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.001 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.001 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.001 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:56.001 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.001 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.001 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.001 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.002 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.002 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.002 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.002 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.002 INFO:teuthology.orchestra.run.smithi039.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-18T16:20:56.002 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-18T16:20:56.002 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.002 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.002 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.002 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:56.002 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.002 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.003 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.003 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.003 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.003 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.003 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.003 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.003 INFO:teuthology.orchestra.run.smithi039.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-18T16:20:56.003 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-18T16:20:56.003 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.003 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.003 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.003 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:56.004 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.004 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.004 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.004 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.004 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.004 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.004 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.004 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.004 INFO:teuthology.orchestra.run.smithi039.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-18T16:20:56.004 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-18T16:20:56.004 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.004 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.004 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.004 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:56.005 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.005 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.005 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.005 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.005 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.005 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.005 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.005 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.005 INFO:teuthology.orchestra.run.smithi039.stdout: "PWD_POLICY_ENABLED": { 2024-03-18T16:20:56.005 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-18T16:20:56.005 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.005 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.005 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.005 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:56.005 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.005 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.006 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.006 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.006 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.006 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.006 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.006 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.006 INFO:teuthology.orchestra.run.smithi039.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-18T16:20:56.006 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-18T16:20:56.006 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.006 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.006 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.006 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-18T16:20:56.006 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.007 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.007 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.007 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.007 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.007 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.007 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.007 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.007 INFO:teuthology.orchestra.run.smithi039.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-18T16:20:56.007 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-18T16:20:56.007 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:56.007 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.007 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.007 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "10", 2024-03-18T16:20:56.007 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.007 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.008 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.008 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.008 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.008 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.008 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.008 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.008 INFO:teuthology.orchestra.run.smithi039.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-18T16:20:56.008 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-18T16:20:56.008 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:56.008 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.008 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.008 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "8", 2024-03-18T16:20:56.008 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.008 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.009 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.009 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.009 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.009 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.009 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.009 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.009 INFO:teuthology.orchestra.run.smithi039.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-18T16:20:56.009 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-18T16:20:56.009 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:56.009 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.009 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.009 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "45", 2024-03-18T16:20:56.009 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.009 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.010 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.010 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.010 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.010 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.010 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.010 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.010 INFO:teuthology.orchestra.run.smithi039.stdout: "RGW_API_ACCESS_KEY": { 2024-03-18T16:20:56.010 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-18T16:20:56.010 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.010 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.010 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.010 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:56.010 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.010 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.011 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.011 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.011 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.011 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.011 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.011 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.011 INFO:teuthology.orchestra.run.smithi039.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-18T16:20:56.011 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-18T16:20:56.011 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.011 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.011 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.011 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "admin", 2024-03-18T16:20:56.011 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.011 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.011 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.012 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.012 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.012 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.012 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.012 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.012 INFO:teuthology.orchestra.run.smithi039.stdout: "RGW_API_SECRET_KEY": { 2024-03-18T16:20:56.012 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-18T16:20:56.012 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.012 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.012 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.012 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:56.012 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.012 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.013 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.013 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.013 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.013 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.013 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.013 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.013 INFO:teuthology.orchestra.run.smithi039.stdout: "RGW_API_SSL_VERIFY": { 2024-03-18T16:20:56.013 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-18T16:20:56.013 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.013 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.013 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.013 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:56.013 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.014 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.014 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.014 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.014 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.014 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.014 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.014 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.014 INFO:teuthology.orchestra.run.smithi039.stdout: "UNSAFE_TLS_v1_2": { 2024-03-18T16:20:56.014 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-18T16:20:56.014 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.014 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.014 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.014 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:56.014 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.015 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.015 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.015 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.015 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.015 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.015 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.015 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.015 INFO:teuthology.orchestra.run.smithi039.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-18T16:20:56.015 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-18T16:20:56.015 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:56.015 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.015 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.015 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-03-18T16:20:56.015 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.030 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.030 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.030 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.031 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.031 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.031 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.031 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.031 INFO:teuthology.orchestra.run.smithi039.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-18T16:20:56.031 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-18T16:20:56.031 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:56.031 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.031 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.032 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "10", 2024-03-18T16:20:56.032 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.032 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.032 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.032 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.032 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.032 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.032 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.032 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.032 INFO:teuthology.orchestra.run.smithi039.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-18T16:20:56.032 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-18T16:20:56.032 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:56.032 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.032 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.032 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "5", 2024-03-18T16:20:56.033 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.033 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.033 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.033 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.033 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.033 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.033 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.033 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.033 INFO:teuthology.orchestra.run.smithi039.stdout: "cross_origin_url": { 2024-03-18T16:20:56.033 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "cross_origin_url", 2024-03-18T16:20:56.033 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.033 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.033 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.033 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:56.033 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.034 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.034 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.034 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.034 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.034 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.034 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.034 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.034 INFO:teuthology.orchestra.run.smithi039.stdout: "crt_file": { 2024-03-18T16:20:56.034 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "crt_file", 2024-03-18T16:20:56.034 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.034 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.034 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.034 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:56.034 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.035 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.035 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.035 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.035 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.035 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.035 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.035 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.035 INFO:teuthology.orchestra.run.smithi039.stdout: "debug": { 2024-03-18T16:20:56.035 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "debug", 2024-03-18T16:20:56.035 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.035 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.035 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.035 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:56.035 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.035 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.036 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.036 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Enable/disable debug options", 2024-03-18T16:20:56.036 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.036 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.036 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.036 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.036 INFO:teuthology.orchestra.run.smithi039.stdout: "jwt_token_ttl": { 2024-03-18T16:20:56.036 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "jwt_token_ttl", 2024-03-18T16:20:56.036 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:56.036 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.036 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.036 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "28800", 2024-03-18T16:20:56.036 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.036 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.036 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.037 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.037 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.037 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.037 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.037 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.037 INFO:teuthology.orchestra.run.smithi039.stdout: "key_file": { 2024-03-18T16:20:56.037 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "key_file", 2024-03-18T16:20:56.037 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.037 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.037 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.037 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:56.037 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.037 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.037 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.038 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.038 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.038 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.038 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.038 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.038 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:56.038 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:56.038 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.038 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.038 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.038 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:56.038 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.038 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.038 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:56.038 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:56.039 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:56.039 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:56.039 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:56.039 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:56.039 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:56.039 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:56.039 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.039 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.039 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.039 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.039 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.039 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:56.039 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:56.039 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.040 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.040 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.040 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:56.040 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.040 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.040 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.040 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.040 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.040 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.040 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.040 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.040 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:56.040 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:56.040 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.041 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.041 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.041 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:56.041 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.041 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.041 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:56.041 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:56.041 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:56.041 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:56.041 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:56.041 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:56.041 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:56.041 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:56.041 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.041 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.041 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.042 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.042 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.042 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:56.042 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:56.042 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.042 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.042 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.042 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:56.042 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.042 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.042 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.042 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.043 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.043 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.043 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.043 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.043 INFO:teuthology.orchestra.run.smithi039.stdout: "motd": { 2024-03-18T16:20:56.043 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "motd", 2024-03-18T16:20:56.043 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.043 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.043 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.043 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:56.043 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.043 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.043 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.043 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "The message of the day", 2024-03-18T16:20:56.044 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.044 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.044 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.044 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.044 INFO:teuthology.orchestra.run.smithi039.stdout: "redirect_resolve_ip_addr": { 2024-03-18T16:20:56.044 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "redirect_resolve_ip_addr", 2024-03-18T16:20:56.044 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.044 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.044 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.044 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:56.044 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.044 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.044 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.044 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.044 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.045 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.045 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.045 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.045 INFO:teuthology.orchestra.run.smithi039.stdout: "server_addr": { 2024-03-18T16:20:56.045 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "server_addr", 2024-03-18T16:20:56.045 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.045 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.045 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.045 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "::", 2024-03-18T16:20:56.045 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.045 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.045 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.045 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.046 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.046 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.046 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.046 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.046 INFO:teuthology.orchestra.run.smithi039.stdout: "server_port": { 2024-03-18T16:20:56.046 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "server_port", 2024-03-18T16:20:56.046 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:56.046 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.046 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.046 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "8080", 2024-03-18T16:20:56.046 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.046 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.046 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.046 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.047 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.047 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.047 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.047 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.047 INFO:teuthology.orchestra.run.smithi039.stdout: "ssl": { 2024-03-18T16:20:56.047 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "ssl", 2024-03-18T16:20:56.047 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.047 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.047 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.047 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:56.047 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.047 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.047 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.048 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.048 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.048 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.048 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.048 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.048 INFO:teuthology.orchestra.run.smithi039.stdout: "ssl_server_port": { 2024-03-18T16:20:56.048 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "ssl_server_port", 2024-03-18T16:20:56.048 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:56.048 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.048 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.048 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "8443", 2024-03-18T16:20:56.048 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.048 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.048 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.048 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.049 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.049 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.049 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.049 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.049 INFO:teuthology.orchestra.run.smithi039.stdout: "standby_behaviour": { 2024-03-18T16:20:56.049 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "standby_behaviour", 2024-03-18T16:20:56.049 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.049 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.049 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.049 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "redirect", 2024-03-18T16:20:56.049 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.049 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.049 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:56.050 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:56.050 INFO:teuthology.orchestra.run.smithi039.stdout: "redirect" 2024-03-18T16:20:56.050 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:56.050 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.050 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.050 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.050 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.050 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.050 INFO:teuthology.orchestra.run.smithi039.stdout: "standby_error_status_code": { 2024-03-18T16:20:56.050 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "standby_error_status_code", 2024-03-18T16:20:56.050 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:56.050 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.050 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.050 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "500", 2024-03-18T16:20:56.051 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "400", 2024-03-18T16:20:56.051 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "599", 2024-03-18T16:20:56.051 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.051 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.051 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.051 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.051 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.051 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.051 INFO:teuthology.orchestra.run.smithi039.stdout: "url_prefix": { 2024-03-18T16:20:56.051 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "url_prefix", 2024-03-18T16:20:56.051 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.051 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.051 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.051 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:56.051 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.052 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.052 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.052 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.060 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.060 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.060 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.060 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:56.060 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:56.061 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.061 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:56.061 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "devicehealth", 2024-03-18T16:20:56.061 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:56.061 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:56.061 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:56.061 INFO:teuthology.orchestra.run.smithi039.stdout: "enable_monitoring": { 2024-03-18T16:20:56.061 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "enable_monitoring", 2024-03-18T16:20:56.061 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.061 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.061 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.061 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:56.061 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.061 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.062 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.062 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "monitor device health metrics", 2024-03-18T16:20:56.062 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.062 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.062 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.062 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.062 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:56.062 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:56.062 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.062 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.062 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.062 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:56.062 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.062 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.063 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:56.063 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:56.063 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:56.063 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:56.063 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:56.063 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:56.063 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:56.063 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:56.063 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.063 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.063 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.063 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.063 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.063 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:56.064 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:56.064 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.064 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.064 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.064 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:56.064 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.064 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.064 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.064 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.064 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.064 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.064 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.065 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.065 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:56.065 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:56.065 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.065 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.065 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.065 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:56.065 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.065 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.065 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:56.066 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:56.066 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:56.066 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:56.066 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:56.066 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:56.066 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:56.066 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:56.066 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.066 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.066 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.066 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.067 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.067 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:56.067 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:56.067 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.067 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.067 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.067 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:56.067 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.067 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.067 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.067 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.068 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.068 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.068 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.068 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.068 INFO:teuthology.orchestra.run.smithi039.stdout: "mark_out_threshold": { 2024-03-18T16:20:56.068 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "mark_out_threshold", 2024-03-18T16:20:56.068 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-03-18T16:20:56.068 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.068 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.068 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "2419200", 2024-03-18T16:20:56.068 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.068 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.069 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.069 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-18T16:20:56.069 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.069 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.069 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.069 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.069 INFO:teuthology.orchestra.run.smithi039.stdout: "pool_name": { 2024-03-18T16:20:56.069 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "pool_name", 2024-03-18T16:20:56.069 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.069 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.069 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.070 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "device_health_metrics", 2024-03-18T16:20:56.070 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.070 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.070 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.070 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-18T16:20:56.070 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.070 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.070 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.070 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.070 INFO:teuthology.orchestra.run.smithi039.stdout: "retention_period": { 2024-03-18T16:20:56.070 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "retention_period", 2024-03-18T16:20:56.071 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-03-18T16:20:56.071 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.071 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.071 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "15552000", 2024-03-18T16:20:56.071 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.071 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.071 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.071 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "how long to retain device health metrics", 2024-03-18T16:20:56.071 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.071 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.071 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.071 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.072 INFO:teuthology.orchestra.run.smithi039.stdout: "scrape_frequency": { 2024-03-18T16:20:56.072 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "scrape_frequency", 2024-03-18T16:20:56.072 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-03-18T16:20:56.072 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.072 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.072 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "86400", 2024-03-18T16:20:56.072 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.072 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.072 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.072 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-18T16:20:56.072 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.073 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.073 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.073 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.073 INFO:teuthology.orchestra.run.smithi039.stdout: "self_heal": { 2024-03-18T16:20:56.073 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "self_heal", 2024-03-18T16:20:56.073 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.073 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.073 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.073 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:56.073 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.073 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.074 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.074 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-18T16:20:56.074 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.074 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.074 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.074 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.074 INFO:teuthology.orchestra.run.smithi039.stdout: "sleep_interval": { 2024-03-18T16:20:56.074 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sleep_interval", 2024-03-18T16:20:56.074 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-03-18T16:20:56.074 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.074 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.074 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "600", 2024-03-18T16:20:56.075 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.075 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.075 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.075 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "how frequently to wake up and check device health", 2024-03-18T16:20:56.075 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.075 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.075 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.075 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.075 INFO:teuthology.orchestra.run.smithi039.stdout: "warn_threshold": { 2024-03-18T16:20:56.075 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "warn_threshold", 2024-03-18T16:20:56.075 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-03-18T16:20:56.075 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.076 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.076 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "7257600", 2024-03-18T16:20:56.076 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.076 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.076 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.076 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-18T16:20:56.076 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.076 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.076 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.076 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:56.076 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:56.076 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.077 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:56.077 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "diskprediction_local", 2024-03-18T16:20:56.077 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:56.077 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:56.077 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:56.077 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:56.077 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:56.077 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.077 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.077 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.077 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:56.077 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.077 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.077 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:56.078 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:56.078 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:56.078 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:56.078 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:56.078 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:56.078 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:56.078 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:56.078 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.078 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.078 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.078 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.078 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.078 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:56.078 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:56.079 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.079 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.079 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.079 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:56.079 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.079 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.079 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.080 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.080 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.080 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.080 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.080 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.080 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:56.080 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:56.080 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.080 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.080 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.080 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:56.080 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.080 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.080 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:56.081 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:56.081 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:56.081 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:56.081 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:56.081 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:56.081 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:56.081 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:56.081 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.081 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.081 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.081 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.081 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.081 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:56.081 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:56.082 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.082 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.082 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.082 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:56.082 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.082 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.082 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.082 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.082 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.082 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.082 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.082 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.082 INFO:teuthology.orchestra.run.smithi039.stdout: "predict_interval": { 2024-03-18T16:20:56.083 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "predict_interval", 2024-03-18T16:20:56.083 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.083 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.083 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.083 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "86400", 2024-03-18T16:20:56.083 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.083 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.083 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.083 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.083 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.083 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.083 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.083 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.083 INFO:teuthology.orchestra.run.smithi039.stdout: "predictor_model": { 2024-03-18T16:20:56.083 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "predictor_model", 2024-03-18T16:20:56.084 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.084 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.084 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.084 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "prophetstor", 2024-03-18T16:20:56.084 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.084 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.084 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.084 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.084 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.084 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.085 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.085 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.085 INFO:teuthology.orchestra.run.smithi039.stdout: "sleep_interval": { 2024-03-18T16:20:56.085 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sleep_interval", 2024-03-18T16:20:56.085 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.085 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.085 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.085 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "600", 2024-03-18T16:20:56.085 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.085 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.085 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.085 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.086 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.086 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.086 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.086 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:56.086 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:56.086 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.086 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:56.086 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "influx", 2024-03-18T16:20:56.086 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": false, 2024-03-18T16:20:56.086 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "influxdb python module not found", 2024-03-18T16:20:56.086 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:56.087 INFO:teuthology.orchestra.run.smithi039.stdout: "batch_size": { 2024-03-18T16:20:56.087 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "batch_size", 2024-03-18T16:20:56.087 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:56.087 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.087 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.087 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "5000", 2024-03-18T16:20:56.087 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.087 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.087 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.087 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-18T16:20:56.087 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.088 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.088 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.088 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.088 INFO:teuthology.orchestra.run.smithi039.stdout: "database": { 2024-03-18T16:20:56.088 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "database", 2024-03-18T16:20:56.088 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.088 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.088 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.088 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "ceph", 2024-03-18T16:20:56.088 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.089 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.089 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.089 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-18T16:20:56.089 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.089 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.089 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.089 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.089 INFO:teuthology.orchestra.run.smithi039.stdout: "hostname": { 2024-03-18T16:20:56.089 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "hostname", 2024-03-18T16:20:56.090 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.090 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.090 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.090 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:56.090 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.090 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.090 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.090 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "InfluxDB server hostname", 2024-03-18T16:20:56.090 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.090 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.090 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.091 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.091 INFO:teuthology.orchestra.run.smithi039.stdout: "interval": { 2024-03-18T16:20:56.091 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "interval", 2024-03-18T16:20:56.091 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-03-18T16:20:56.091 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.097 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.097 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "30", 2024-03-18T16:20:56.097 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "5", 2024-03-18T16:20:56.097 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.097 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.097 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-18T16:20:56.097 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.098 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.098 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.098 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.098 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:56.098 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:56.098 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.098 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.098 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.098 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:56.098 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.099 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.099 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:56.099 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:56.099 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:56.099 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:56.099 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:56.099 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:56.099 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:56.099 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:56.099 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.099 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.099 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.100 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.100 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.100 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:56.100 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:56.100 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.100 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.100 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.100 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:56.100 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.100 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.100 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.100 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.101 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.101 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.101 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.101 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.101 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:56.101 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:56.101 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.101 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.101 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.101 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:56.102 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.102 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.102 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:56.102 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:56.102 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:56.102 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:56.102 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:56.102 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:56.102 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:56.102 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:56.102 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.102 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.103 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.103 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.103 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.103 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:56.103 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:56.103 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.103 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.103 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.103 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:56.103 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.103 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.103 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.104 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.104 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.104 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.104 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.104 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.104 INFO:teuthology.orchestra.run.smithi039.stdout: "password": { 2024-03-18T16:20:56.104 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "password", 2024-03-18T16:20:56.104 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.104 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.104 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.104 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:56.105 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.105 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.105 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.105 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "password of InfluxDB server user", 2024-03-18T16:20:56.105 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.105 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.105 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.105 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.105 INFO:teuthology.orchestra.run.smithi039.stdout: "port": { 2024-03-18T16:20:56.105 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "port", 2024-03-18T16:20:56.105 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:56.106 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.106 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.106 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "8086", 2024-03-18T16:20:56.106 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.106 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.106 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.106 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "InfluxDB server port", 2024-03-18T16:20:56.106 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.106 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.106 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.106 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.107 INFO:teuthology.orchestra.run.smithi039.stdout: "ssl": { 2024-03-18T16:20:56.107 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "ssl", 2024-03-18T16:20:56.107 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.107 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.107 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.107 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "false", 2024-03-18T16:20:56.107 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.107 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.107 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.107 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-18T16:20:56.107 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.107 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.107 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.107 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.108 INFO:teuthology.orchestra.run.smithi039.stdout: "threads": { 2024-03-18T16:20:56.108 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "threads", 2024-03-18T16:20:56.108 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:56.108 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.108 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.108 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "5", 2024-03-18T16:20:56.108 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "1", 2024-03-18T16:20:56.108 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "32", 2024-03-18T16:20:56.108 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.108 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-18T16:20:56.108 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.108 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.108 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.108 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.108 INFO:teuthology.orchestra.run.smithi039.stdout: "username": { 2024-03-18T16:20:56.109 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "username", 2024-03-18T16:20:56.109 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.109 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.109 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.109 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:56.109 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.109 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.109 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.109 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "username of InfluxDB server user", 2024-03-18T16:20:56.109 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.109 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.109 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.109 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.109 INFO:teuthology.orchestra.run.smithi039.stdout: "verify_ssl": { 2024-03-18T16:20:56.110 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "verify_ssl", 2024-03-18T16:20:56.110 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.110 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.110 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.110 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "true", 2024-03-18T16:20:56.110 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.110 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.110 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.110 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-18T16:20:56.110 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.110 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.110 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.110 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:56.110 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:56.111 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.111 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:56.111 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "insights", 2024-03-18T16:20:56.111 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:56.111 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:56.111 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:56.111 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:56.111 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:56.111 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.111 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.111 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.111 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:56.111 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.111 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.112 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:56.112 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:56.112 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:56.112 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:56.112 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:56.112 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:56.112 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:56.112 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:56.112 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.112 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.112 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.112 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.112 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.112 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:56.112 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:56.112 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.113 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.113 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.113 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:56.113 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.113 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.113 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.113 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.113 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.113 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.113 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.113 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.113 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:56.113 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:56.113 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.113 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.114 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.114 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:56.114 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.114 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.114 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:56.114 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:56.114 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:56.114 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:56.114 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:56.114 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:56.114 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:56.114 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:56.114 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.114 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.115 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.115 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.115 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.115 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:56.115 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:56.115 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.115 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.115 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.115 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:56.115 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.115 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.116 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.116 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.116 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.116 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.116 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.116 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:56.116 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:56.116 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.116 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:56.116 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "iostat", 2024-03-18T16:20:56.116 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:56.117 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:56.117 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:56.117 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:56.117 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:56.117 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.117 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.117 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.117 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:56.117 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.117 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.118 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:56.118 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:56.118 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:56.118 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:56.118 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:56.118 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:56.118 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:56.118 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:56.118 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.119 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.119 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.119 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.119 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.119 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:56.119 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:56.119 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.119 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.119 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.119 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:56.119 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.119 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.119 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.120 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.120 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.120 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.120 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.120 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.120 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:56.120 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:56.120 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.120 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.121 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.121 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:56.121 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.121 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.121 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:56.121 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:56.121 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:56.121 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:56.121 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:56.121 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:56.121 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:56.122 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:56.122 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.122 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.122 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.122 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.122 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.122 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:56.122 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:56.122 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.123 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.123 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.123 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:56.123 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.123 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.123 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.123 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.123 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.123 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.123 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.123 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:56.124 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:56.124 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.124 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:56.124 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "k8sevents", 2024-03-18T16:20:56.124 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:56.124 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:56.124 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:56.124 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph_event_retention_days": { 2024-03-18T16:20:56.124 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "ceph_event_retention_days", 2024-03-18T16:20:56.124 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:56.125 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.125 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.125 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "7", 2024-03-18T16:20:56.125 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.125 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.125 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.125 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-18T16:20:56.125 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.125 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.126 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.126 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.126 INFO:teuthology.orchestra.run.smithi039.stdout: "config_check_secs": { 2024-03-18T16:20:56.126 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "config_check_secs", 2024-03-18T16:20:56.126 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:56.126 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.126 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.126 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "10", 2024-03-18T16:20:56.126 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "10", 2024-03-18T16:20:56.127 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.127 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.127 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-18T16:20:56.127 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.127 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.127 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.127 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.127 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:56.176 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:56.176 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.176 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.176 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.176 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:56.177 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.177 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.177 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:56.177 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:56.177 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:56.177 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:56.177 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:56.177 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:56.177 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:56.177 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:56.177 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.177 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.178 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.178 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.178 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.178 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:56.178 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:56.178 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.178 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.178 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.178 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:56.178 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.178 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.179 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.179 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.179 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.179 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.179 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.179 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.179 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:56.179 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:56.179 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.179 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.179 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.180 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:56.180 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.180 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.180 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:56.180 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:56.180 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:56.180 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:56.180 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:56.180 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:56.180 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:56.180 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:56.180 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.181 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.181 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.181 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.181 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.181 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:56.181 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:56.181 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.181 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.181 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.181 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:56.181 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.181 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.181 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.181 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.181 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.182 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.182 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.182 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:56.182 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:56.182 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.182 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:56.182 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "localpool", 2024-03-18T16:20:56.182 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:56.182 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:56.182 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:56.182 INFO:teuthology.orchestra.run.smithi039.stdout: "failure_domain": { 2024-03-18T16:20:56.182 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "failure_domain", 2024-03-18T16:20:56.182 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.182 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.183 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.183 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "host", 2024-03-18T16:20:56.183 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.183 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.183 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.183 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "failure domain for any created local pool", 2024-03-18T16:20:56.183 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-18T16:20:56.183 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.183 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.183 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.183 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:56.183 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:56.183 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.183 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.183 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.184 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:56.184 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.184 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.184 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:56.184 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:56.184 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:56.184 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:56.184 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:56.184 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:56.184 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:56.184 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:56.184 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.184 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.185 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.185 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.185 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.185 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:56.185 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:56.185 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.185 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.185 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.185 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:56.185 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.185 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.185 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.185 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.185 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.186 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.186 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.186 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.186 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:56.186 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:56.186 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.186 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.186 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.186 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:56.186 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.186 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.186 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:56.186 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:56.186 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:56.187 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:56.187 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:56.187 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:56.187 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:56.187 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:56.187 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.187 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.187 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.188 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.188 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.188 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:56.188 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:56.188 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.188 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.188 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.188 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:56.188 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.188 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.188 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.188 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.188 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.189 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.189 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.189 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.189 INFO:teuthology.orchestra.run.smithi039.stdout: "min_size": { 2024-03-18T16:20:56.189 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "min_size", 2024-03-18T16:20:56.189 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:56.189 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.189 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.189 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:56.189 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.189 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.189 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.190 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "default min_size for any created local pool", 2024-03-18T16:20:56.190 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-18T16:20:56.190 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.190 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.190 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.190 INFO:teuthology.orchestra.run.smithi039.stdout: "num_rep": { 2024-03-18T16:20:56.190 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "num_rep", 2024-03-18T16:20:56.190 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:56.190 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.190 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.191 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "3", 2024-03-18T16:20:56.191 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.191 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.191 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.191 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "default replica count for any created local pool", 2024-03-18T16:20:56.191 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.191 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.191 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.191 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.191 INFO:teuthology.orchestra.run.smithi039.stdout: "pg_num": { 2024-03-18T16:20:56.191 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "pg_num", 2024-03-18T16:20:56.191 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:56.191 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.192 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.192 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "128", 2024-03-18T16:20:56.192 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.192 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.192 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.192 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "default pg_num for any created local pool", 2024-03-18T16:20:56.192 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.192 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.192 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.192 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.193 INFO:teuthology.orchestra.run.smithi039.stdout: "prefix": { 2024-03-18T16:20:56.193 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "prefix", 2024-03-18T16:20:56.193 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.193 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.193 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.193 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:56.193 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.193 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.193 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.193 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "name prefix for any created local pool", 2024-03-18T16:20:56.194 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.194 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.194 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.194 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.194 INFO:teuthology.orchestra.run.smithi039.stdout: "subtree": { 2024-03-18T16:20:56.194 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "subtree", 2024-03-18T16:20:56.194 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.194 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.194 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.194 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "rack", 2024-03-18T16:20:56.195 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.195 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.195 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.195 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-18T16:20:56.195 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-18T16:20:56.195 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.195 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.195 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:56.195 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:56.195 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.196 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:56.196 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "mds_autoscaler", 2024-03-18T16:20:56.196 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:56.196 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:56.196 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:56.196 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:56.196 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:56.196 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.196 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.197 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.197 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:56.197 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.197 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.197 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:56.197 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:56.197 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:56.197 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:56.197 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:56.197 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:56.198 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:56.198 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:56.198 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.198 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.198 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.198 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.198 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.198 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:56.199 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:56.199 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.199 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.199 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.199 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:56.199 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.199 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.199 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.200 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.200 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.200 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.200 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.200 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.200 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:56.200 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:56.200 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.200 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.200 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.201 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:56.201 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.201 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.201 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:56.201 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:56.201 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:56.201 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:56.201 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:56.201 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:56.201 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:56.201 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:56.201 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.202 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.202 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.202 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.202 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.202 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:56.202 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:56.202 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.202 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.202 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.202 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:56.202 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.203 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.203 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.203 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.203 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.203 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.203 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.203 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:56.203 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:56.264 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.264 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:56.265 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "mirroring", 2024-03-18T16:20:56.265 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:56.265 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:56.265 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:56.265 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:56.265 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:56.265 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.265 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.265 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.265 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:56.266 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.266 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.266 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:56.266 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:56.266 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:56.266 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:56.266 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:56.266 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:56.266 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:56.267 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:56.267 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.267 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.267 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.267 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.267 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.267 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:56.267 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:56.267 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.268 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.268 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.268 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:56.268 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.268 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.268 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.268 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.268 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.268 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.269 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.269 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.269 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:56.269 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:56.269 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.269 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.269 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.269 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:56.269 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.270 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.270 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:56.270 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:56.270 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:56.270 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:56.270 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:56.270 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:56.270 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:56.270 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:56.271 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.271 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.271 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.271 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.271 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.271 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:56.271 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:56.271 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.271 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.271 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.271 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:56.271 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.272 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.272 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.272 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.272 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.272 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.272 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.272 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:56.272 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:56.272 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.272 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:56.272 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "nfs", 2024-03-18T16:20:56.273 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:56.273 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:56.273 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:56.273 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:56.273 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:56.273 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.273 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.273 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.273 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:56.273 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.274 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.274 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:56.274 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:56.274 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:56.274 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:56.274 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:56.274 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:56.274 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:56.275 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:56.275 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.275 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.275 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.275 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.275 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.275 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:56.275 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:56.275 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.275 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.275 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.275 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:56.276 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.276 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.276 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.276 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.276 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.276 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.276 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.276 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.276 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:56.276 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:56.276 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.276 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.277 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.277 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:56.277 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.277 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.277 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:56.277 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:56.277 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:56.277 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:56.277 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:56.277 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:56.277 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:56.277 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:56.278 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.278 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.278 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.278 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.356 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.356 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:56.356 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:56.356 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.356 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.356 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.356 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:56.356 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.356 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.356 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.357 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.357 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.357 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.357 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.357 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:56.357 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:56.357 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.357 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:56.357 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "orchestrator", 2024-03-18T16:20:56.357 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:56.357 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:56.357 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:56.357 INFO:teuthology.orchestra.run.smithi039.stdout: "fail_fs": { 2024-03-18T16:20:56.357 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "fail_fs", 2024-03-18T16:20:56.358 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.358 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.358 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.358 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:56.358 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.358 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.358 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.358 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-18T16:20:56.364 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.364 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.364 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.364 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.364 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:56.364 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:56.364 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.364 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.365 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.365 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:56.365 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.365 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.365 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:56.365 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:56.365 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:56.366 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:56.366 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:56.366 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:56.366 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:56.366 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:56.366 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.366 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.366 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.366 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.366 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.367 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:56.367 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:56.367 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.367 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.367 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.367 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:56.367 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.367 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.367 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.367 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.368 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.368 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.368 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.368 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.368 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:56.368 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:56.368 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.368 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.368 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.368 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:56.369 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.369 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.369 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:56.369 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:56.369 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:56.369 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:56.369 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:56.369 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:56.370 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:56.370 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:56.370 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.370 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.376 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.377 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.377 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.377 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:56.377 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:56.377 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.377 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.377 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.377 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:56.377 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.377 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.377 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.378 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.378 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.378 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.378 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.378 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.378 INFO:teuthology.orchestra.run.smithi039.stdout: "orchestrator": { 2024-03-18T16:20:56.378 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "orchestrator", 2024-03-18T16:20:56.378 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.378 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.378 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.378 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:56.378 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.378 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.379 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:56.379 INFO:teuthology.orchestra.run.smithi039.stdout: "cephadm", 2024-03-18T16:20:56.379 INFO:teuthology.orchestra.run.smithi039.stdout: "rook", 2024-03-18T16:20:56.379 INFO:teuthology.orchestra.run.smithi039.stdout: "test_orchestrator" 2024-03-18T16:20:56.379 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:56.379 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Orchestrator backend", 2024-03-18T16:20:56.379 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.379 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.379 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.379 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:56.379 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:56.379 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.379 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:56.379 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "osd_perf_query", 2024-03-18T16:20:56.379 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:56.380 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:56.380 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:56.380 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:56.380 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:56.380 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.380 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.380 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.380 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:56.380 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.380 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.380 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:56.380 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:56.380 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:56.380 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:56.380 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:56.381 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:56.381 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:56.381 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:56.381 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.381 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.381 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.381 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.381 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.381 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:56.381 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:56.381 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.381 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.381 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.381 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:56.381 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.382 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.382 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.382 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.382 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.382 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.382 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.382 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.382 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:56.382 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:56.382 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.382 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.382 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.382 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:56.382 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.383 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.383 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:56.383 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:56.383 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:56.383 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:56.383 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:56.383 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:56.383 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:56.383 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:56.383 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.383 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.383 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.383 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.383 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.383 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:56.383 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:56.384 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.384 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.384 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.384 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:56.384 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.384 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.384 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.384 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.384 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.384 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.384 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.384 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:56.384 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:56.384 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.384 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:56.385 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "osd_support", 2024-03-18T16:20:56.385 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:56.385 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:56.385 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:56.385 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:56.385 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:56.385 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.385 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.385 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.385 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:56.385 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.385 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.385 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:56.385 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:56.385 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:56.385 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:56.386 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:56.386 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:56.386 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:56.386 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:56.386 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.386 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.386 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.386 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.386 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.386 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:56.386 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:56.386 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.386 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.386 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.387 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:56.387 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.387 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.387 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.387 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.387 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.387 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.387 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.387 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.387 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:56.387 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:56.387 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.387 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.387 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.387 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:56.388 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.388 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.388 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:56.388 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:56.388 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:56.388 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:56.388 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:56.388 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:56.388 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:56.388 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:56.388 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.388 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.388 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.388 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.388 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.389 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:56.389 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:56.389 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.389 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.389 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.389 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:56.389 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.389 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.389 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.389 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.389 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.389 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.389 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.389 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:56.389 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:56.389 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.390 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:56.390 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "pg_autoscaler", 2024-03-18T16:20:56.390 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:56.390 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:56.390 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:56.390 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:56.390 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:56.390 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.390 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.390 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.390 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:56.390 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.390 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.390 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:56.390 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:56.391 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:56.391 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:56.391 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:56.391 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:56.391 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:56.391 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:56.391 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.391 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.391 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.391 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.391 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.391 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:56.391 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:56.391 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.391 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.392 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.392 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:56.392 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.392 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.392 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.392 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.392 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.392 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.392 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.392 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.392 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:56.392 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:56.392 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.392 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.392 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.393 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:56.393 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.393 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.393 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:56.393 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:56.393 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:56.393 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:56.393 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:56.393 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:56.393 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:56.393 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:56.393 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.393 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.393 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.393 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.394 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.394 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:56.394 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:56.394 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.394 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.394 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.394 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:56.394 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.394 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.394 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.394 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.394 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.394 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.394 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.394 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.395 INFO:teuthology.orchestra.run.smithi039.stdout: "sleep_interval": { 2024-03-18T16:20:56.395 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sleep_interval", 2024-03-18T16:20:56.395 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-03-18T16:20:56.395 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.395 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.395 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "60", 2024-03-18T16:20:56.395 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.395 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.395 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.395 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.395 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.395 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.395 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.395 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.395 INFO:teuthology.orchestra.run.smithi039.stdout: "threshold": { 2024-03-18T16:20:56.396 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "threshold", 2024-03-18T16:20:56.396 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "float", 2024-03-18T16:20:56.396 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.396 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.396 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "3.0", 2024-03-18T16:20:56.396 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "1.0", 2024-03-18T16:20:56.396 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.396 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.396 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "scaling threshold", 2024-03-18T16:20:56.396 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-18T16:20:56.396 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.396 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.396 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:56.396 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:56.396 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.397 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:56.397 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "progress", 2024-03-18T16:20:56.397 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:56.397 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:56.397 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:56.397 INFO:teuthology.orchestra.run.smithi039.stdout: "allow_pg_recovery_event": { 2024-03-18T16:20:56.397 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "allow_pg_recovery_event", 2024-03-18T16:20:56.397 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.397 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.397 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.397 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:56.397 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.397 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.397 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.397 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-18T16:20:56.398 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.398 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.398 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.398 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.398 INFO:teuthology.orchestra.run.smithi039.stdout: "enabled": { 2024-03-18T16:20:56.398 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "enabled", 2024-03-18T16:20:56.398 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.398 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.398 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.398 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:56.398 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.398 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.398 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.398 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.398 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.399 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.399 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.399 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.399 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:56.399 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:56.399 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.399 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.399 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.399 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:56.399 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.399 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.399 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:56.399 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:56.399 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:56.400 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:56.400 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:56.400 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:56.400 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:56.400 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:56.400 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.400 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.400 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.400 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.400 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.400 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:56.400 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:56.401 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.401 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.401 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.401 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:56.401 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.401 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.401 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.401 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.401 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.401 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.401 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.402 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.402 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:56.402 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:56.402 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.402 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.402 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.402 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:56.402 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.402 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.402 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:56.402 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:56.402 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:56.402 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:56.403 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:56.403 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:56.403 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:56.403 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:56.403 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.403 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.403 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.403 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.403 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.403 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:56.403 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:56.403 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.403 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.403 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.403 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:56.404 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.404 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.404 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.404 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.404 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.404 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.404 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.404 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.404 INFO:teuthology.orchestra.run.smithi039.stdout: "max_completed_events": { 2024-03-18T16:20:56.404 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "max_completed_events", 2024-03-18T16:20:56.404 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:56.404 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.404 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.404 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "50", 2024-03-18T16:20:56.404 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.405 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.405 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.405 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "number of past completed events to remember", 2024-03-18T16:20:56.405 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.405 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.405 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.405 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.405 INFO:teuthology.orchestra.run.smithi039.stdout: "sleep_interval": { 2024-03-18T16:20:56.405 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "sleep_interval", 2024-03-18T16:20:56.405 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-03-18T16:20:56.405 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.405 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.405 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "5", 2024-03-18T16:20:56.406 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.406 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.406 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.406 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "how long the module is going to sleep", 2024-03-18T16:20:56.406 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.406 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.406 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.406 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:56.406 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:56.406 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.406 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:56.406 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "prometheus", 2024-03-18T16:20:56.407 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:56.407 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:56.407 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:56.407 INFO:teuthology.orchestra.run.smithi039.stdout: "cache": { 2024-03-18T16:20:56.407 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "cache", 2024-03-18T16:20:56.407 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.407 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.407 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.407 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:56.407 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.407 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.407 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.407 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.407 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.407 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.408 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.408 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.408 INFO:teuthology.orchestra.run.smithi039.stdout: "exclude_perf_counters": { 2024-03-18T16:20:56.408 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "exclude_perf_counters", 2024-03-18T16:20:56.408 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.408 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.408 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.408 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:56.408 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.408 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.408 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.408 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-18T16:20:56.408 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-18T16:20:56.408 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.408 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.409 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.409 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:56.409 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:56.409 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.409 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.409 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.409 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:56.409 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.409 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.409 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:56.409 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:56.409 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:56.409 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:56.410 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:56.410 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:56.410 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:56.410 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:56.410 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.410 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.410 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.410 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.410 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.410 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:56.410 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:56.411 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.411 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.411 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.411 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:56.411 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.411 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.411 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.411 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.411 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.411 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.411 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.411 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.411 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:56.411 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:56.411 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.412 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.412 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.412 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:56.412 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.412 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.412 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:56.412 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:56.412 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:56.412 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:56.412 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:56.413 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:56.413 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:56.413 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:56.413 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.413 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.413 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.413 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.413 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.413 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:56.413 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:56.413 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.413 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.413 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.414 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:56.414 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.414 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.414 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.414 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.414 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.414 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.414 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.414 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.414 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_stats_pools": { 2024-03-18T16:20:56.414 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "rbd_stats_pools", 2024-03-18T16:20:56.414 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.414 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.420 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.420 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:56.420 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.420 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.420 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.420 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.420 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.421 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.421 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.421 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.421 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-18T16:20:56.421 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-18T16:20:56.421 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:56.421 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.421 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.421 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "300", 2024-03-18T16:20:56.421 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.421 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.421 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.421 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.421 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.421 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.422 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.422 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.422 INFO:teuthology.orchestra.run.smithi039.stdout: "scrape_interval": { 2024-03-18T16:20:56.422 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "scrape_interval", 2024-03-18T16:20:56.422 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "float", 2024-03-18T16:20:56.422 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.422 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.422 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "15.0", 2024-03-18T16:20:56.422 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.422 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.422 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.422 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.422 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.422 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.422 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.423 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.423 INFO:teuthology.orchestra.run.smithi039.stdout: "server_addr": { 2024-03-18T16:20:56.423 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "server_addr", 2024-03-18T16:20:56.423 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.423 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.423 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.423 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "::", 2024-03-18T16:20:56.423 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.423 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.423 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.423 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-18T16:20:56.423 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.423 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.423 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.423 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.424 INFO:teuthology.orchestra.run.smithi039.stdout: "server_port": { 2024-03-18T16:20:56.424 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "server_port", 2024-03-18T16:20:56.424 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:56.424 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.424 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.424 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "9283", 2024-03-18T16:20:56.424 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.424 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.424 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.424 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-18T16:20:56.424 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.424 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.424 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.424 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.424 INFO:teuthology.orchestra.run.smithi039.stdout: "stale_cache_strategy": { 2024-03-18T16:20:56.425 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "stale_cache_strategy", 2024-03-18T16:20:56.425 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.425 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.425 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.425 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "log", 2024-03-18T16:20:56.425 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.425 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.425 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.425 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.425 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.425 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.425 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.425 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.425 INFO:teuthology.orchestra.run.smithi039.stdout: "standby_behaviour": { 2024-03-18T16:20:56.425 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "standby_behaviour", 2024-03-18T16:20:56.426 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.426 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.426 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.426 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "default", 2024-03-18T16:20:56.426 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.426 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.426 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:56.426 INFO:teuthology.orchestra.run.smithi039.stdout: "default", 2024-03-18T16:20:56.426 INFO:teuthology.orchestra.run.smithi039.stdout: "error" 2024-03-18T16:20:56.426 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:56.426 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.426 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.426 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.426 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.426 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.427 INFO:teuthology.orchestra.run.smithi039.stdout: "standby_error_status_code": { 2024-03-18T16:20:56.427 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "standby_error_status_code", 2024-03-18T16:20:56.427 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:56.427 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.427 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.427 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "500", 2024-03-18T16:20:56.427 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "400", 2024-03-18T16:20:56.427 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "599", 2024-03-18T16:20:56.427 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.427 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.427 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.427 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.427 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.427 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:56.427 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:56.428 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.428 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:56.428 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "rbd_support", 2024-03-18T16:20:56.428 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:56.428 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:56.428 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:56.428 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:56.428 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:56.428 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.428 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.428 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.428 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:56.428 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.428 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.429 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:56.429 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:56.429 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:56.429 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:56.429 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:56.429 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:56.429 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:56.429 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:56.429 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.429 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.429 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.429 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.429 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.429 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:56.429 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:56.429 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.430 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.430 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.430 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:56.430 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.430 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.430 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.430 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.430 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.430 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.430 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.430 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.430 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:56.431 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:56.431 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.431 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.431 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.431 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:56.431 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.431 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.431 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:56.431 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:56.431 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:56.431 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:56.431 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:56.432 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:56.432 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:56.432 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:56.432 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.432 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.432 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.432 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.432 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.432 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:56.432 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:56.432 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.432 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.432 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.432 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:56.433 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.433 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.433 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.433 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.433 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.433 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.433 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.433 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.433 INFO:teuthology.orchestra.run.smithi039.stdout: "max_concurrent_snap_create": { 2024-03-18T16:20:56.433 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "max_concurrent_snap_create", 2024-03-18T16:20:56.433 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:56.433 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.433 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.434 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "10", 2024-03-18T16:20:56.434 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.434 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.434 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.434 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.434 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.434 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.434 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.434 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.434 INFO:teuthology.orchestra.run.smithi039.stdout: "mirror_snapshot_schedule": { 2024-03-18T16:20:56.434 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "mirror_snapshot_schedule", 2024-03-18T16:20:56.434 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.434 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.435 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.435 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:56.435 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.435 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.435 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.435 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.435 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.435 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.435 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.436 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.436 INFO:teuthology.orchestra.run.smithi039.stdout: "trash_purge_schedule": { 2024-03-18T16:20:56.436 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "trash_purge_schedule", 2024-03-18T16:20:56.436 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.436 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.436 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.436 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:56.436 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.436 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.436 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.436 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.436 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.436 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.436 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.437 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:56.437 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:56.437 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.437 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:56.437 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "restful", 2024-03-18T16:20:56.437 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:56.437 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:56.437 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:56.437 INFO:teuthology.orchestra.run.smithi039.stdout: "enable_auth": { 2024-03-18T16:20:56.437 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "enable_auth", 2024-03-18T16:20:56.437 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.438 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.438 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.438 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:56.438 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.438 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.438 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.438 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.438 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.438 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.438 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.438 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.439 INFO:teuthology.orchestra.run.smithi039.stdout: "key_file": { 2024-03-18T16:20:56.439 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "key_file", 2024-03-18T16:20:56.439 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.439 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.439 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.439 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:56.439 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.439 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.439 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.440 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.440 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.440 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.440 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.440 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.440 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:56.440 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:56.440 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.440 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.440 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.440 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:56.440 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.440 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.440 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:56.441 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:56.441 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:56.441 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:56.441 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:56.441 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:56.441 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:56.441 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:56.441 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.441 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.441 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.441 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.442 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.442 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:56.442 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:56.442 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.442 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.442 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.442 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:56.442 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.442 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.442 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.442 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.442 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.443 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.443 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.443 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.443 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:56.443 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:56.443 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.443 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.451 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.451 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:56.451 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.451 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.451 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:56.452 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:56.452 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:56.452 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:56.452 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:56.452 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:56.452 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:56.452 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:56.452 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.452 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.452 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.452 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.452 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.452 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:56.452 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:56.452 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.453 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.453 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.453 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:56.453 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.453 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.453 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.453 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.453 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.453 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.453 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.453 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.453 INFO:teuthology.orchestra.run.smithi039.stdout: "server_addr": { 2024-03-18T16:20:56.453 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "server_addr", 2024-03-18T16:20:56.453 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.454 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.454 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.454 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:56.454 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.454 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.454 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.454 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.454 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.454 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.454 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.454 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.454 INFO:teuthology.orchestra.run.smithi039.stdout: "server_port": { 2024-03-18T16:20:56.454 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "server_port", 2024-03-18T16:20:56.454 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.455 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.455 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.455 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:56.455 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.455 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.455 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.455 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.455 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.455 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.455 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.455 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:56.455 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:56.455 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.455 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:56.455 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "rgw", 2024-03-18T16:20:56.455 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:56.456 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:56.457 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:56.457 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:56.457 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:56.457 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.457 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.457 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.457 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:56.458 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.458 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.458 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:56.458 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:56.458 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:56.458 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:56.458 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:56.458 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:56.458 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:56.458 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:56.458 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.458 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.458 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.458 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.459 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.459 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:56.459 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:56.459 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.459 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.459 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.459 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:56.459 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.459 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.459 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.459 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.459 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.459 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.459 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.526 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.526 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:56.526 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:56.526 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.526 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.526 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.526 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:56.526 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.526 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.526 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:56.527 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:56.527 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:56.527 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:56.527 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:56.527 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:56.527 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:56.527 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:56.527 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.527 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.527 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.527 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.527 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.527 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:56.527 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:56.527 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.528 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.528 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.528 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:56.528 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.528 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.528 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.528 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.528 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.528 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.528 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.528 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:56.528 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:56.528 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.528 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:56.528 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "rook", 2024-03-18T16:20:56.529 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:56.529 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:56.529 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:56.529 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:56.529 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:56.529 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.529 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.529 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.529 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:56.529 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.529 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.529 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:56.529 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:56.529 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:56.529 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:56.530 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:56.530 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:56.530 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:56.530 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:56.530 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.530 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.530 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.530 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.530 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.530 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:56.530 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:56.530 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.530 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.530 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.530 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:56.531 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.531 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.531 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.531 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.531 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.531 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.531 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.531 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.531 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:56.531 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:56.531 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.531 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.531 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.531 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:56.532 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.532 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.532 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:56.532 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:56.532 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:56.532 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:56.532 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:56.532 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:56.532 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:56.532 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:56.532 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.532 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.532 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.533 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.533 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.533 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:56.533 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:56.533 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.533 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.533 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.533 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:56.533 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.533 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.533 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.533 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.533 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.533 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.533 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.533 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.534 INFO:teuthology.orchestra.run.smithi039.stdout: "storage_class": { 2024-03-18T16:20:56.534 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "storage_class", 2024-03-18T16:20:56.534 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.534 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.534 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.534 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "local", 2024-03-18T16:20:56.534 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.534 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.534 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.534 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-18T16:20:56.534 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.534 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.534 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.534 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:56.535 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:56.535 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.535 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:56.535 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "selftest", 2024-03-18T16:20:56.535 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:56.535 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:56.535 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:56.535 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:56.535 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:56.535 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.535 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.535 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.535 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:56.535 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.535 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.535 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:56.536 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:56.536 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:56.536 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:56.536 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:56.536 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:56.536 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:56.536 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:56.536 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.536 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.536 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.536 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.536 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.536 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:56.536 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:56.536 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.536 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.537 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.537 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:56.537 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.537 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.537 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.537 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.537 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.537 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.580 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.580 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.580 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:56.580 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:56.580 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.580 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.580 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.581 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:56.581 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.581 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.581 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:56.581 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:56.581 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:56.581 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:56.581 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:56.581 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:56.581 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:56.581 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:56.581 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.581 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.581 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.582 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.582 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.582 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:56.582 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:56.582 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.582 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.582 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.582 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:56.582 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.582 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.582 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.582 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.583 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.583 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.583 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.583 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.583 INFO:teuthology.orchestra.run.smithi039.stdout: "roption1": { 2024-03-18T16:20:56.583 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "roption1", 2024-03-18T16:20:56.583 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.583 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.583 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.583 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:56.583 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.583 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.583 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.583 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.583 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.584 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.584 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.584 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.584 INFO:teuthology.orchestra.run.smithi039.stdout: "roption2": { 2024-03-18T16:20:56.584 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "roption2", 2024-03-18T16:20:56.584 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.584 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.584 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.584 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "xyz", 2024-03-18T16:20:56.584 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.584 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.584 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.584 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.584 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.585 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.585 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.585 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.585 INFO:teuthology.orchestra.run.smithi039.stdout: "rwoption1": { 2024-03-18T16:20:56.585 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "rwoption1", 2024-03-18T16:20:56.585 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.585 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.585 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.585 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:56.585 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.585 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.585 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.585 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.585 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.585 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.586 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.586 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.586 INFO:teuthology.orchestra.run.smithi039.stdout: "rwoption2": { 2024-03-18T16:20:56.586 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "rwoption2", 2024-03-18T16:20:56.586 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:56.586 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.586 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.586 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:56.586 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.586 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.586 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.586 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.586 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.587 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.587 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.587 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.587 INFO:teuthology.orchestra.run.smithi039.stdout: "rwoption3": { 2024-03-18T16:20:56.587 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "rwoption3", 2024-03-18T16:20:56.587 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "float", 2024-03-18T16:20:56.587 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.587 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.587 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:56.587 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.587 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.587 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.587 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.587 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.587 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.588 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.588 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.588 INFO:teuthology.orchestra.run.smithi039.stdout: "rwoption4": { 2024-03-18T16:20:56.588 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "rwoption4", 2024-03-18T16:20:56.588 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.588 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.588 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.588 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:56.588 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.589 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.589 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.589 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.589 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.589 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.589 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.589 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.589 INFO:teuthology.orchestra.run.smithi039.stdout: "rwoption5": { 2024-03-18T16:20:56.589 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "rwoption5", 2024-03-18T16:20:56.589 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.589 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.589 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.651 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:56.651 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.651 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.651 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.651 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.651 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.651 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.651 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.651 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.651 INFO:teuthology.orchestra.run.smithi039.stdout: "rwoption6": { 2024-03-18T16:20:56.651 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "rwoption6", 2024-03-18T16:20:56.651 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.652 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.652 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.652 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:56.652 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.652 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.652 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.652 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.652 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.652 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.652 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.652 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.652 INFO:teuthology.orchestra.run.smithi039.stdout: "rwoption7": { 2024-03-18T16:20:56.652 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "rwoption7", 2024-03-18T16:20:56.652 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:56.653 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.653 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.653 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:56.653 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "1", 2024-03-18T16:20:56.653 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "42", 2024-03-18T16:20:56.653 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.653 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.653 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.653 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.653 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.653 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.653 INFO:teuthology.orchestra.run.smithi039.stdout: "testkey": { 2024-03-18T16:20:56.653 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "testkey", 2024-03-18T16:20:56.653 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.653 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.653 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.654 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:56.654 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.654 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.654 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.654 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.654 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.654 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.654 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.654 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.654 INFO:teuthology.orchestra.run.smithi039.stdout: "testlkey": { 2024-03-18T16:20:56.654 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "testlkey", 2024-03-18T16:20:56.654 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.654 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.654 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.655 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:56.655 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.655 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.655 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.655 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.655 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.655 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.655 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.655 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.655 INFO:teuthology.orchestra.run.smithi039.stdout: "testnewline": { 2024-03-18T16:20:56.655 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "testnewline", 2024-03-18T16:20:56.655 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.655 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.655 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.655 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:56.656 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.656 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.656 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.656 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.656 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.656 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.656 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.656 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:56.656 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:56.656 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.656 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:56.656 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "snap_schedule", 2024-03-18T16:20:56.656 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:56.656 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:56.656 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:56.657 INFO:teuthology.orchestra.run.smithi039.stdout: "allow_m_granularity": { 2024-03-18T16:20:56.657 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "allow_m_granularity", 2024-03-18T16:20:56.657 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.657 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.657 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.657 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:56.657 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.657 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.657 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.657 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "allow minute scheduled snapshots", 2024-03-18T16:20:56.657 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.657 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.657 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.657 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.657 INFO:teuthology.orchestra.run.smithi039.stdout: "dump_on_update": { 2024-03-18T16:20:56.657 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "dump_on_update", 2024-03-18T16:20:56.658 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.658 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.658 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.658 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:56.658 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.658 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.658 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.658 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "dump database to debug log on update", 2024-03-18T16:20:56.658 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.658 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.658 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.658 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.658 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:56.658 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:56.659 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.659 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.659 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.659 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:56.659 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.659 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.659 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:56.659 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:56.659 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:56.659 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:56.659 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:56.659 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:56.659 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:56.659 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:56.659 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.659 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.660 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.660 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.660 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.660 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:56.660 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:56.660 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.660 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.660 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.660 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:56.660 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.660 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.660 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.660 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.660 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.660 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.661 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.661 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.661 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:56.661 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:56.661 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.661 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.661 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.661 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:56.661 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.661 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.661 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:56.661 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:56.661 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:56.661 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:56.661 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:56.662 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:56.662 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:56.662 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:56.662 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.662 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.662 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.662 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.662 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.662 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:56.662 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:56.662 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.662 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.662 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.662 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:56.663 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.663 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.663 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.663 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.663 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.663 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.663 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.663 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:56.663 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:56.663 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.663 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:56.663 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "stats", 2024-03-18T16:20:56.663 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:56.663 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:56.664 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:56.664 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:56.664 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:56.664 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.664 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.664 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.664 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:56.664 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.664 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.664 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:56.664 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:56.664 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:56.664 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:56.664 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:56.664 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:56.665 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:56.665 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:56.665 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.665 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.665 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.665 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.665 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.665 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:56.665 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:56.665 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.665 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.665 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.665 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:56.665 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.666 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.666 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.666 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.666 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.666 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.666 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.666 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.666 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:56.666 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:56.666 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.666 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.666 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.666 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:56.666 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.667 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.667 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:56.667 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:56.667 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:56.667 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:56.667 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:56.667 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:56.667 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:56.667 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:56.667 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.667 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.667 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.667 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.667 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.667 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:56.668 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:56.668 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.668 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.668 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.668 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:56.668 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.668 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.668 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.668 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.668 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.668 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.668 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.668 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:56.668 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:56.668 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.669 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:56.669 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "status", 2024-03-18T16:20:56.669 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:56.669 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:56.669 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:56.669 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:56.669 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:56.669 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.669 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.669 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.669 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:56.669 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.669 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.669 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:56.669 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:56.670 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:56.670 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:56.670 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:56.670 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:56.670 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:56.670 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:56.670 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.670 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.670 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.670 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.670 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.670 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:56.670 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:56.670 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.670 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.671 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.671 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:56.671 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.671 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.671 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.671 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.671 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.671 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.671 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.671 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.671 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:56.671 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:56.671 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.671 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.671 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.671 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:56.672 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.672 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.672 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:56.672 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:56.672 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:56.672 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:56.672 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:56.672 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:56.672 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:56.672 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:56.672 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.672 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.672 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.673 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.673 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.673 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:56.673 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:56.673 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.673 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.673 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.673 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:56.673 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.673 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.673 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.673 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.673 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.673 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.674 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.674 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:56.674 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:56.674 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.674 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:56.674 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "telegraf", 2024-03-18T16:20:56.674 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:56.674 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:56.674 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:56.674 INFO:teuthology.orchestra.run.smithi039.stdout: "address": { 2024-03-18T16:20:56.674 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "address", 2024-03-18T16:20:56.674 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.674 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.674 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.675 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-18T16:20:56.675 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.675 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.675 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.675 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.675 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.675 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.675 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.675 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.675 INFO:teuthology.orchestra.run.smithi039.stdout: "interval": { 2024-03-18T16:20:56.675 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "interval", 2024-03-18T16:20:56.675 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-03-18T16:20:56.675 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.675 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.675 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "15", 2024-03-18T16:20:56.675 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.676 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.676 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.676 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.676 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.676 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.676 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.676 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.676 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:56.676 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:56.676 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.676 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.676 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.676 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:56.676 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.676 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.677 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:56.677 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:56.677 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:56.677 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:56.677 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:56.677 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:56.677 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:56.677 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:56.677 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.677 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.677 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.677 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.677 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.677 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:56.677 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:56.677 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.678 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.678 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.678 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:56.678 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.678 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.678 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.678 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.678 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.678 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.678 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.678 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.678 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:56.678 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:56.678 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.678 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.679 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.679 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:56.679 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.679 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.679 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:56.679 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:56.679 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:56.679 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:56.679 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:56.679 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:56.679 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:56.679 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:56.679 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.679 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.680 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.680 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.680 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.680 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:56.680 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:56.680 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.680 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.680 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.680 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:56.680 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.680 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.680 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.680 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.680 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.680 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.680 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.681 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:56.681 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:56.681 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.681 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:56.681 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "telemetry", 2024-03-18T16:20:56.681 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:56.681 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:56.681 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:56.681 INFO:teuthology.orchestra.run.smithi039.stdout: "channel_basic": { 2024-03-18T16:20:56.681 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "channel_basic", 2024-03-18T16:20:56.681 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.681 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.681 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.681 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:56.681 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.682 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.682 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.682 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-18T16:20:56.682 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.682 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.682 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.705 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.705 INFO:teuthology.orchestra.run.smithi039.stdout: "channel_crash": { 2024-03-18T16:20:56.705 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "channel_crash", 2024-03-18T16:20:56.705 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.705 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.705 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.705 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:56.705 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.705 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.705 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.706 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-18T16:20:56.706 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.706 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.706 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.706 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.706 INFO:teuthology.orchestra.run.smithi039.stdout: "channel_device": { 2024-03-18T16:20:56.706 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "channel_device", 2024-03-18T16:20:56.706 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.706 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.706 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.706 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:56.707 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.707 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.707 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.707 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-18T16:20:56.707 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.707 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.707 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.707 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.707 INFO:teuthology.orchestra.run.smithi039.stdout: "channel_ident": { 2024-03-18T16:20:56.707 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "channel_ident", 2024-03-18T16:20:56.707 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.707 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.708 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.708 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:56.708 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.708 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.708 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.708 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-18T16:20:56.708 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.708 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.708 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.708 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.708 INFO:teuthology.orchestra.run.smithi039.stdout: "channel_perf": { 2024-03-18T16:20:56.708 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "channel_perf", 2024-03-18T16:20:56.708 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.709 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.709 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.709 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:56.709 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.709 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.709 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.709 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-18T16:20:56.709 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.709 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.709 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.709 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.709 INFO:teuthology.orchestra.run.smithi039.stdout: "contact": { 2024-03-18T16:20:56.709 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "contact", 2024-03-18T16:20:56.710 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.710 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.710 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.710 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:56.710 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.710 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.710 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.710 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.710 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.710 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.710 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.710 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.710 INFO:teuthology.orchestra.run.smithi039.stdout: "description": { 2024-03-18T16:20:56.710 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "description", 2024-03-18T16:20:56.711 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.711 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.711 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.711 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:56.711 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.711 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.711 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.711 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.711 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.711 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.711 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.711 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.711 INFO:teuthology.orchestra.run.smithi039.stdout: "device_url": { 2024-03-18T16:20:56.711 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "device_url", 2024-03-18T16:20:56.711 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.711 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.712 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.712 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-18T16:20:56.712 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.712 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.712 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.712 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.712 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.712 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.712 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.712 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.712 INFO:teuthology.orchestra.run.smithi039.stdout: "enabled": { 2024-03-18T16:20:56.712 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "enabled", 2024-03-18T16:20:56.712 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.712 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.712 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.713 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:56.713 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.713 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.713 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.713 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.713 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.713 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.713 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.713 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.713 INFO:teuthology.orchestra.run.smithi039.stdout: "interval": { 2024-03-18T16:20:56.713 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "interval", 2024-03-18T16:20:56.713 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:56.713 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.713 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.713 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "24", 2024-03-18T16:20:56.714 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "8", 2024-03-18T16:20:56.714 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.714 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.714 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.714 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.714 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.714 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.724 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.724 INFO:teuthology.orchestra.run.smithi039.stdout: "last_opt_revision": { 2024-03-18T16:20:56.724 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "last_opt_revision", 2024-03-18T16:20:56.725 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:56.725 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.725 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.725 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "1", 2024-03-18T16:20:56.725 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.725 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.725 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.725 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.725 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.725 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.725 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.725 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.726 INFO:teuthology.orchestra.run.smithi039.stdout: "leaderboard": { 2024-03-18T16:20:56.726 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "leaderboard", 2024-03-18T16:20:56.726 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.726 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.726 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.726 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:56.726 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.726 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.727 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.727 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.727 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.727 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.727 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.727 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.727 INFO:teuthology.orchestra.run.smithi039.stdout: "leaderboard_description": { 2024-03-18T16:20:56.727 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "leaderboard_description", 2024-03-18T16:20:56.727 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.727 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.727 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.727 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:56.728 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.728 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.728 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.728 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.728 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.728 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.728 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.728 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.728 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:56.728 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:56.729 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.729 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.729 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.729 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:56.729 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.729 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.729 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:56.729 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:56.729 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:56.729 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:56.729 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:56.729 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:56.730 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:56.730 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:56.730 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.730 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.730 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.730 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.730 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.730 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:56.730 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:56.730 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.730 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.731 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.731 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:56.731 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.731 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.731 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.731 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.731 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.731 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.731 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.731 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.731 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:56.732 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:56.732 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.732 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.732 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.732 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:56.732 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.732 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.732 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:56.732 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:56.732 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:56.732 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:56.732 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:56.733 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:56.733 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:56.733 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:56.733 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.733 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.733 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.733 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.733 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.733 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:56.733 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:56.733 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.734 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.734 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.734 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:56.734 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.734 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.734 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.734 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.734 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.734 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.734 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.734 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.734 INFO:teuthology.orchestra.run.smithi039.stdout: "organization": { 2024-03-18T16:20:56.734 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "organization", 2024-03-18T16:20:56.734 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.735 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.735 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.735 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:56.735 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.735 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.735 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.735 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.735 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.735 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.735 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.735 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.735 INFO:teuthology.orchestra.run.smithi039.stdout: "proxy": { 2024-03-18T16:20:56.735 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "proxy", 2024-03-18T16:20:56.735 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.735 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.736 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.736 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:56.736 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.736 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.736 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.736 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.736 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.736 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.736 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.736 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.736 INFO:teuthology.orchestra.run.smithi039.stdout: "url": { 2024-03-18T16:20:56.736 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "url", 2024-03-18T16:20:56.736 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.736 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.736 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.737 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-18T16:20:56.737 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.737 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.737 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.737 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.737 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.737 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.737 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.737 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:56.737 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:56.737 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.737 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:56.737 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "test_orchestrator", 2024-03-18T16:20:56.737 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:56.737 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:56.738 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:56.738 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:56.738 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:56.738 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.738 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.738 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.738 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:56.738 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.738 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.738 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:56.739 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:56.739 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:56.739 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:56.739 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:56.739 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:56.739 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:56.739 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:56.739 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.739 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.739 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.740 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.740 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.740 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:56.740 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:56.740 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.740 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.740 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.740 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:56.740 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.740 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.740 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.740 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.740 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.741 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.741 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.741 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.741 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:56.741 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:56.741 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.741 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.742 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.742 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:56.742 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.742 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.742 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:56.742 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:56.742 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:56.742 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:56.742 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:56.742 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:56.742 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:56.742 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:56.742 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.742 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.743 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.743 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.743 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.743 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:56.743 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:56.743 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.743 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.743 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.744 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:56.744 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.744 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.744 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.744 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.744 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.744 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.744 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.744 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:56.744 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:56.744 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.744 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:56.744 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "volumes", 2024-03-18T16:20:56.744 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:56.745 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:56.745 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:56.745 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:56.745 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:56.745 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.745 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.745 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.745 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:56.746 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.746 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.746 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:56.746 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:56.746 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:56.746 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:56.746 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:56.746 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:56.746 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:56.746 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:56.746 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.747 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.747 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.747 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.747 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.747 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:56.747 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:56.747 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.747 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.747 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.747 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:56.747 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.747 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.747 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.747 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.748 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.748 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.748 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.748 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.748 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:56.748 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:56.748 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.748 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.748 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.748 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:56.748 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.748 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.748 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:56.748 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:56.748 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:56.749 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:56.749 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:56.749 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:56.749 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:56.749 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:56.749 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.749 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.749 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.749 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.749 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.749 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:56.749 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:56.749 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.749 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.749 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.750 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:56.750 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.750 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.750 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.750 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.750 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.750 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.750 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.750 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.750 INFO:teuthology.orchestra.run.smithi039.stdout: "max_concurrent_clones": { 2024-03-18T16:20:56.750 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "max_concurrent_clones", 2024-03-18T16:20:56.750 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:56.750 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.750 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.751 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "4", 2024-03-18T16:20:56.751 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.751 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.751 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.751 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-18T16:20:56.751 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.751 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.751 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.751 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.751 INFO:teuthology.orchestra.run.smithi039.stdout: "periodic_async_work": { 2024-03-18T16:20:56.751 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "periodic_async_work", 2024-03-18T16:20:56.751 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.751 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.751 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.751 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:56.751 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.752 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.752 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.752 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Periodically check for async work", 2024-03-18T16:20:56.752 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.752 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.752 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.752 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.752 INFO:teuthology.orchestra.run.smithi039.stdout: "snapshot_clone_delay": { 2024-03-18T16:20:56.752 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "snapshot_clone_delay", 2024-03-18T16:20:56.752 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:56.753 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.753 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.753 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "0", 2024-03-18T16:20:56.753 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.753 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.753 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.753 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-18T16:20:56.753 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.753 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.753 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.753 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.753 INFO:teuthology.orchestra.run.smithi039.stdout: "snapshot_clone_no_wait": { 2024-03-18T16:20:56.753 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "snapshot_clone_no_wait", 2024-03-18T16:20:56.753 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.753 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.754 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.754 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "True", 2024-03-18T16:20:56.754 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.754 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.754 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.754 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-18T16:20:56.754 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.755 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.755 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.755 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:56.755 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:56.755 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.755 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:56.755 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "zabbix", 2024-03-18T16:20:56.755 INFO:teuthology.orchestra.run.smithi039.stdout: "can_run": true, 2024-03-18T16:20:56.755 INFO:teuthology.orchestra.run.smithi039.stdout: "error_string": "", 2024-03-18T16:20:56.755 INFO:teuthology.orchestra.run.smithi039.stdout: "module_options": { 2024-03-18T16:20:56.755 INFO:teuthology.orchestra.run.smithi039.stdout: "discovery_interval": { 2024-03-18T16:20:56.755 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "discovery_interval", 2024-03-18T16:20:56.756 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "uint", 2024-03-18T16:20:56.756 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.756 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.756 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "100", 2024-03-18T16:20:56.756 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.756 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.764 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.764 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.764 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.764 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.764 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.765 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.765 INFO:teuthology.orchestra.run.smithi039.stdout: "identifier": { 2024-03-18T16:20:56.765 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "identifier", 2024-03-18T16:20:56.765 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.765 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.765 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.765 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:56.765 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.765 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.766 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.766 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.766 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.766 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.766 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.766 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.766 INFO:teuthology.orchestra.run.smithi039.stdout: "interval": { 2024-03-18T16:20:56.766 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "interval", 2024-03-18T16:20:56.767 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "secs", 2024-03-18T16:20:56.767 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.767 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.767 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "60", 2024-03-18T16:20:56.767 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.767 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.767 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.767 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.767 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.768 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.768 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.768 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.768 INFO:teuthology.orchestra.run.smithi039.stdout: "log_level": { 2024-03-18T16:20:56.768 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_level", 2024-03-18T16:20:56.768 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.768 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.768 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.768 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:56.768 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.768 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.768 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:56.768 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:56.768 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:56.769 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:56.769 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:56.769 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:56.769 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:56.769 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:56.769 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.769 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.769 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.769 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.769 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.769 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster": { 2024-03-18T16:20:56.779 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster", 2024-03-18T16:20:56.779 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.780 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.780 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.780 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:56.780 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.780 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.780 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.780 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.780 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.780 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.780 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.780 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.780 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_cluster_level": { 2024-03-18T16:20:56.781 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_cluster_level", 2024-03-18T16:20:56.781 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.781 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.781 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.781 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "info", 2024-03-18T16:20:56.781 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.781 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.781 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [ 2024-03-18T16:20:56.781 INFO:teuthology.orchestra.run.smithi039.stdout: "", 2024-03-18T16:20:56.781 INFO:teuthology.orchestra.run.smithi039.stdout: "critical", 2024-03-18T16:20:56.781 INFO:teuthology.orchestra.run.smithi039.stdout: "debug", 2024-03-18T16:20:56.782 INFO:teuthology.orchestra.run.smithi039.stdout: "error", 2024-03-18T16:20:56.782 INFO:teuthology.orchestra.run.smithi039.stdout: "info", 2024-03-18T16:20:56.782 INFO:teuthology.orchestra.run.smithi039.stdout: "warning" 2024-03-18T16:20:56.782 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:56.782 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.782 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.782 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.782 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.782 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.782 INFO:teuthology.orchestra.run.smithi039.stdout: "log_to_file": { 2024-03-18T16:20:56.782 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "log_to_file", 2024-03-18T16:20:56.782 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "bool", 2024-03-18T16:20:56.782 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.783 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 1, 2024-03-18T16:20:56.783 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "False", 2024-03-18T16:20:56.783 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.783 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.783 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.783 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.783 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.783 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.783 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.783 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.783 INFO:teuthology.orchestra.run.smithi039.stdout: "zabbix_host": { 2024-03-18T16:20:56.783 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "zabbix_host", 2024-03-18T16:20:56.783 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.783 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.784 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.784 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "", 2024-03-18T16:20:56.784 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.784 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.784 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.784 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.784 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.784 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.784 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.784 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.784 INFO:teuthology.orchestra.run.smithi039.stdout: "zabbix_port": { 2024-03-18T16:20:56.784 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "zabbix_port", 2024-03-18T16:20:56.784 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "int", 2024-03-18T16:20:56.784 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.784 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.785 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "10051", 2024-03-18T16:20:56.785 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.785 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.785 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.785 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.785 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.785 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.785 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.785 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.785 INFO:teuthology.orchestra.run.smithi039.stdout: "zabbix_sender": { 2024-03-18T16:20:56.785 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "zabbix_sender", 2024-03-18T16:20:56.785 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "str", 2024-03-18T16:20:56.785 INFO:teuthology.orchestra.run.smithi039.stdout: "level": "advanced", 2024-03-18T16:20:56.785 INFO:teuthology.orchestra.run.smithi039.stdout: "flags": 0, 2024-03-18T16:20:56.785 INFO:teuthology.orchestra.run.smithi039.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-18T16:20:56.786 INFO:teuthology.orchestra.run.smithi039.stdout: "min": "", 2024-03-18T16:20:56.786 INFO:teuthology.orchestra.run.smithi039.stdout: "max": "", 2024-03-18T16:20:56.786 INFO:teuthology.orchestra.run.smithi039.stdout: "enum_allowed": [], 2024-03-18T16:20:56.786 INFO:teuthology.orchestra.run.smithi039.stdout: "desc": "", 2024-03-18T16:20:56.786 INFO:teuthology.orchestra.run.smithi039.stdout: "long_desc": "", 2024-03-18T16:20:56.786 INFO:teuthology.orchestra.run.smithi039.stdout: "tags": [], 2024-03-18T16:20:56.786 INFO:teuthology.orchestra.run.smithi039.stdout: "see_also": [] 2024-03-18T16:20:56.786 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:56.786 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:56.786 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:56.786 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:56.786 INFO:teuthology.orchestra.run.smithi039.stdout: "services": { 2024-03-18T16:20:56.786 INFO:teuthology.orchestra.run.smithi039.stdout: "dashboard": "https://172.21.15.39:8443/" 2024-03-18T16:20:56.786 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.787 INFO:teuthology.orchestra.run.smithi039.stdout: "always_on_modules": { 2024-03-18T16:20:56.787 INFO:teuthology.orchestra.run.smithi039.stdout: "octopus": [ 2024-03-18T16:20:56.787 INFO:teuthology.orchestra.run.smithi039.stdout: "balancer", 2024-03-18T16:20:56.787 INFO:teuthology.orchestra.run.smithi039.stdout: "crash", 2024-03-18T16:20:56.787 INFO:teuthology.orchestra.run.smithi039.stdout: "devicehealth", 2024-03-18T16:20:56.787 INFO:teuthology.orchestra.run.smithi039.stdout: "orchestrator", 2024-03-18T16:20:56.787 INFO:teuthology.orchestra.run.smithi039.stdout: "pg_autoscaler", 2024-03-18T16:20:56.787 INFO:teuthology.orchestra.run.smithi039.stdout: "progress", 2024-03-18T16:20:56.787 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_support", 2024-03-18T16:20:56.787 INFO:teuthology.orchestra.run.smithi039.stdout: "status", 2024-03-18T16:20:56.787 INFO:teuthology.orchestra.run.smithi039.stdout: "telemetry", 2024-03-18T16:20:56.787 INFO:teuthology.orchestra.run.smithi039.stdout: "volumes" 2024-03-18T16:20:56.787 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:56.787 INFO:teuthology.orchestra.run.smithi039.stdout: "pacific": [ 2024-03-18T16:20:56.787 INFO:teuthology.orchestra.run.smithi039.stdout: "balancer", 2024-03-18T16:20:56.788 INFO:teuthology.orchestra.run.smithi039.stdout: "crash", 2024-03-18T16:20:56.788 INFO:teuthology.orchestra.run.smithi039.stdout: "devicehealth", 2024-03-18T16:20:56.788 INFO:teuthology.orchestra.run.smithi039.stdout: "orchestrator", 2024-03-18T16:20:56.788 INFO:teuthology.orchestra.run.smithi039.stdout: "pg_autoscaler", 2024-03-18T16:20:56.788 INFO:teuthology.orchestra.run.smithi039.stdout: "progress", 2024-03-18T16:20:56.788 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_support", 2024-03-18T16:20:56.788 INFO:teuthology.orchestra.run.smithi039.stdout: "status", 2024-03-18T16:20:56.788 INFO:teuthology.orchestra.run.smithi039.stdout: "telemetry", 2024-03-18T16:20:56.788 INFO:teuthology.orchestra.run.smithi039.stdout: "volumes" 2024-03-18T16:20:56.788 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:56.788 INFO:teuthology.orchestra.run.smithi039.stdout: "quincy": [ 2024-03-18T16:20:56.788 INFO:teuthology.orchestra.run.smithi039.stdout: "balancer", 2024-03-18T16:20:56.788 INFO:teuthology.orchestra.run.smithi039.stdout: "crash", 2024-03-18T16:20:56.788 INFO:teuthology.orchestra.run.smithi039.stdout: "devicehealth", 2024-03-18T16:20:56.788 INFO:teuthology.orchestra.run.smithi039.stdout: "orchestrator", 2024-03-18T16:20:56.789 INFO:teuthology.orchestra.run.smithi039.stdout: "pg_autoscaler", 2024-03-18T16:20:56.789 INFO:teuthology.orchestra.run.smithi039.stdout: "progress", 2024-03-18T16:20:56.789 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_support", 2024-03-18T16:20:56.789 INFO:teuthology.orchestra.run.smithi039.stdout: "status", 2024-03-18T16:20:56.789 INFO:teuthology.orchestra.run.smithi039.stdout: "telemetry", 2024-03-18T16:20:56.789 INFO:teuthology.orchestra.run.smithi039.stdout: "volumes" 2024-03-18T16:20:56.789 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:56.789 INFO:teuthology.orchestra.run.smithi039.stdout: "reef": [ 2024-03-18T16:20:56.789 INFO:teuthology.orchestra.run.smithi039.stdout: "balancer", 2024-03-18T16:20:56.789 INFO:teuthology.orchestra.run.smithi039.stdout: "crash", 2024-03-18T16:20:56.789 INFO:teuthology.orchestra.run.smithi039.stdout: "devicehealth", 2024-03-18T16:20:56.789 INFO:teuthology.orchestra.run.smithi039.stdout: "orchestrator", 2024-03-18T16:20:56.789 INFO:teuthology.orchestra.run.smithi039.stdout: "pg_autoscaler", 2024-03-18T16:20:56.789 INFO:teuthology.orchestra.run.smithi039.stdout: "progress", 2024-03-18T16:20:56.789 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_support", 2024-03-18T16:20:56.790 INFO:teuthology.orchestra.run.smithi039.stdout: "status", 2024-03-18T16:20:56.790 INFO:teuthology.orchestra.run.smithi039.stdout: "telemetry", 2024-03-18T16:20:56.790 INFO:teuthology.orchestra.run.smithi039.stdout: "volumes" 2024-03-18T16:20:56.790 INFO:teuthology.orchestra.run.smithi039.stdout: ], 2024-03-18T16:20:56.790 INFO:teuthology.orchestra.run.smithi039.stdout: "squid": [ 2024-03-18T16:20:56.790 INFO:teuthology.orchestra.run.smithi039.stdout: "balancer", 2024-03-18T16:20:56.790 INFO:teuthology.orchestra.run.smithi039.stdout: "crash", 2024-03-18T16:20:56.790 INFO:teuthology.orchestra.run.smithi039.stdout: "devicehealth", 2024-03-18T16:20:56.790 INFO:teuthology.orchestra.run.smithi039.stdout: "orchestrator", 2024-03-18T16:20:56.790 INFO:teuthology.orchestra.run.smithi039.stdout: "pg_autoscaler", 2024-03-18T16:20:56.790 INFO:teuthology.orchestra.run.smithi039.stdout: "progress", 2024-03-18T16:20:56.790 INFO:teuthology.orchestra.run.smithi039.stdout: "rbd_support", 2024-03-18T16:20:56.790 INFO:teuthology.orchestra.run.smithi039.stdout: "status", 2024-03-18T16:20:56.790 INFO:teuthology.orchestra.run.smithi039.stdout: "telemetry", 2024-03-18T16:20:56.790 INFO:teuthology.orchestra.run.smithi039.stdout: "volumes" 2024-03-18T16:20:56.791 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-03-18T16:20:56.791 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.791 INFO:teuthology.orchestra.run.smithi039.stdout: "last_failure_osd_epoch": 3, 2024-03-18T16:20:56.791 INFO:teuthology.orchestra.run.smithi039.stdout: "active_clients": [ 2024-03-18T16:20:56.791 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:56.791 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "devicehealth", 2024-03-18T16:20:56.791 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-03-18T16:20:56.791 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:56.791 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-03-18T16:20:56.791 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:0", 2024-03-18T16:20:56.791 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 3746088120 2024-03-18T16:20:56.791 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:56.791 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-03-18T16:20:56.791 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.791 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:56.792 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "libcephsqlite", 2024-03-18T16:20:56.792 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-03-18T16:20:56.792 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:56.792 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-03-18T16:20:56.792 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:0", 2024-03-18T16:20:56.792 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 265589236 2024-03-18T16:20:56.792 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:56.792 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-03-18T16:20:56.792 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.792 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:56.792 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "rbd_support", 2024-03-18T16:20:56.792 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-03-18T16:20:56.792 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:56.792 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-03-18T16:20:56.792 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:0", 2024-03-18T16:20:56.793 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 1969807796 2024-03-18T16:20:56.793 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:56.793 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-03-18T16:20:56.793 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:56.793 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:56.793 INFO:teuthology.orchestra.run.smithi039.stdout: "name": "volumes", 2024-03-18T16:20:56.793 INFO:teuthology.orchestra.run.smithi039.stdout: "addrvec": [ 2024-03-18T16:20:56.793 INFO:teuthology.orchestra.run.smithi039.stdout: { 2024-03-18T16:20:56.793 INFO:teuthology.orchestra.run.smithi039.stdout: "type": "v2", 2024-03-18T16:20:56.793 INFO:teuthology.orchestra.run.smithi039.stdout: "addr": "172.21.15.39:0", 2024-03-18T16:20:56.793 INFO:teuthology.orchestra.run.smithi039.stdout: "nonce": 2682283296 2024-03-18T16:20:56.793 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:56.793 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-03-18T16:20:56.793 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:56.793 INFO:teuthology.orchestra.run.smithi039.stdout: ] 2024-03-18T16:20:56.793 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-03-18T16:20:56.794 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:53.332+0000 7f2fa2a00640 1 -- 172.21.15.39:0/371867407 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f2f980a2800 msgr2=0x7f2f980a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:56.795 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:53.332+0000 7f2fa2a00640 1 --2- 172.21.15.39:0/371867407 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f2f980a2800 0x7f2f980a4cc0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f2fa40095f0 tx=0x7f2fa4005f90 comp rx=0 tx=0).stop 2024-03-18T16:20:56.795 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:53.332+0000 7f2fa2a00640 1 -- 172.21.15.39:0/371867407 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2fb0104a60 msgr2=0x7f2fb0197e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:20:56.795 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:53.332+0000 7f2fa2a00640 1 --2- 172.21.15.39:0/371867407 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2fb0104a60 0x7f2fb0197e90 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f2f9c009230 tx=0x7f2f9c02c5d0 comp rx=0 tx=0).stop 2024-03-18T16:20:56.795 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:53.332+0000 7f2fa2a00640 1 -- 172.21.15.39:0/371867407 shutdown_connections 2024-03-18T16:20:56.795 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:53.332+0000 7f2fa2a00640 1 --2- 172.21.15.39:0/371867407 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f2f980a2800 0x7f2f980a4cc0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:56.795 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:53.332+0000 7f2fa2a00640 1 --2- 172.21.15.39:0/371867407 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2fb0107390 0x7f2fb019ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:56.795 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:53.332+0000 7f2fa2a00640 1 --2- 172.21.15.39:0/371867407 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2fb0104a60 0x7f2fb0197e90 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:56.795 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:53.332+0000 7f2fa2a00640 1 --2- 172.21.15.39:0/371867407 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2fb0102130 0x7f2fb0197950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:20:56.795 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:53.332+0000 7f2fa2a00640 1 -- 172.21.15.39:0/371867407 >> 172.21.15.39:0/371867407 conn(0x7f2fb00fbac0 msgr2=0x7f2fb01029d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:20:56.795 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:53.333+0000 7f2fa2a00640 1 -- 172.21.15.39:0/371867407 shutdown_connections 2024-03-18T16:20:56.795 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:20:53.333+0000 7f2fa2a00640 1 -- 172.21.15.39:0/371867407 wait complete. 2024-03-18T16:20:56.796 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:53 smithi039 ceph-mon[31721]: pgmap v746: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 767 B/s wr, 10 op/s 2024-03-18T16:20:56.796 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:53 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/1520816213' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T16:20:56.796 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:53 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/4053565556' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T16:20:56.802 DEBUG:tasks.check_counter:Getting stats from a 2024-03-18T16:20:56.803 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph daemon mds.a perf dump 2024-03-18T16:20:56.803 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:54 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/371867407' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-18T16:20:56.804 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:55 smithi039 ceph-mon[31721]: pgmap v747: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 938 B/s wr, 10 op/s 2024-03-18T16:20:56.804 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:53 smithi194 ceph-mon[36200]: pgmap v746: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 767 B/s wr, 10 op/s 2024-03-18T16:20:56.804 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:53 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/1520816213' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T16:20:56.804 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:53 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/4053565556' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T16:20:56.804 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:54 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/371867407' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-18T16:20:56.804 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:55 smithi194 ceph-mon[36200]: pgmap v747: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 938 B/s wr, 10 op/s 2024-03-18T16:20:56.805 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:53 smithi196 ceph-mon[36307]: pgmap v746: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 767 B/s wr, 10 op/s 2024-03-18T16:20:56.805 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:53 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/1520816213' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T16:20:56.805 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:53 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/4053565556' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T16:20:56.805 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:54 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/371867407' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-18T16:20:56.805 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:55 smithi196 ceph-mon[36307]: pgmap v747: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 938 B/s wr, 10 op/s 2024-03-18T16:20:57.305 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T16:20:57.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:57 smithi039 ceph-mon[31721]: pgmap v748: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 511 B/s wr, 10 op/s 2024-03-18T16:20:57.418 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:57 smithi194 ceph-mon[36200]: pgmap v748: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 511 B/s wr, 10 op/s 2024-03-18T16:20:57.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:57 smithi196 ceph-mon[36307]: pgmap v748: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 511 B/s wr, 10 op/s 2024-03-18T16:20:59.214 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:20:59 smithi039 ceph-mon[31721]: pgmap v749: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 341 B/s wr, 10 op/s 2024-03-18T16:20:59.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:20:59 smithi196 ceph-mon[36307]: pgmap v749: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 341 B/s wr, 10 op/s 2024-03-18T16:20:59.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:20:59 smithi194 ceph-mon[36200]: pgmap v749: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 341 B/s wr, 10 op/s 2024-03-18T16:20:59.943 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-03-18T16:20:59.943 INFO:teuthology.orchestra.run.smithi039.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T16:20:59.943 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_messages": 3028, 2024-03-18T16:20:59.943 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_messages": 4122, 2024-03-18T16:20:59.943 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_bytes": 1066537, 2024-03-18T16:20:59.944 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_bytes": 4329638, 2024-03-18T16:20:59.944 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_created_connections": 17, 2024-03-18T16:20:59.944 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_active_connections": 2, 2024-03-18T16:20:59.944 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_total_time": 1.272739826, 2024-03-18T16:20:59.944 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_send_time": 0.815332838, 2024-03-18T16:20:59.944 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_recv_time": 0.273109699, 2024-03-18T16:20:59.944 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_fast_dispatch_time": 0.106491502, 2024-03-18T16:20:59.944 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T16:20:59.944 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 4120, 2024-03-18T16:20:59.944 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 3.234121623, 2024-03-18T16:20:59.944 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000784980 2024-03-18T16:20:59.944 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:59.944 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_handle_ack_lat": { 2024-03-18T16:20:59.944 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 3102, 2024-03-18T16:20:59.945 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.003247459, 2024-03-18T16:20:59.945 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000001046 2024-03-18T16:20:59.945 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:59.945 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_encrypted_bytes": 452400, 2024-03-18T16:20:59.945 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_encrypted_bytes": 1058560 2024-03-18T16:20:59.945 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:59.945 INFO:teuthology.orchestra.run.smithi039.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T16:20:59.945 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_messages": 4198, 2024-03-18T16:20:59.945 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_messages": 5563, 2024-03-18T16:20:59.945 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_bytes": 1453203, 2024-03-18T16:20:59.945 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_bytes": 2264837, 2024-03-18T16:20:59.945 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_created_connections": 11, 2024-03-18T16:20:59.945 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_active_connections": 6, 2024-03-18T16:20:59.945 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_total_time": 0.765450394, 2024-03-18T16:20:59.945 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_send_time": 0.316996511, 2024-03-18T16:20:59.946 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_recv_time": 0.330469707, 2024-03-18T16:20:59.946 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_fast_dispatch_time": 0.101283344, 2024-03-18T16:20:59.946 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T16:20:59.946 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 5564, 2024-03-18T16:20:59.946 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 2.345018298, 2024-03-18T16:20:59.946 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000421462 2024-03-18T16:20:59.946 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:59.946 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_handle_ack_lat": { 2024-03-18T16:20:59.946 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 5747, 2024-03-18T16:20:59.946 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.003008596, 2024-03-18T16:20:59.946 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000523 2024-03-18T16:20:59.947 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:59.947 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_encrypted_bytes": 3152, 2024-03-18T16:20:59.947 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_encrypted_bytes": 384704 2024-03-18T16:20:59.947 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:59.947 INFO:teuthology.orchestra.run.smithi039.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T16:20:59.947 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_messages": 2767, 2024-03-18T16:20:59.947 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_messages": 3273, 2024-03-18T16:20:59.947 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_bytes": 1638810, 2024-03-18T16:20:59.947 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_bytes": 1567898, 2024-03-18T16:20:59.947 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_created_connections": 40, 2024-03-18T16:20:59.947 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_active_connections": 11, 2024-03-18T16:20:59.947 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_total_time": 0.548160311, 2024-03-18T16:20:59.947 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_send_time": 0.185112783, 2024-03-18T16:20:59.947 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_recv_time": 0.224569354, 2024-03-18T16:20:59.947 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_fast_dispatch_time": 0.111133708, 2024-03-18T16:20:59.948 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T16:20:59.948 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 3270, 2024-03-18T16:20:59.948 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 1.585385555, 2024-03-18T16:20:59.948 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000484827 2024-03-18T16:20:59.948 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:59.948 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_handle_ack_lat": { 2024-03-18T16:20:59.948 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 2064, 2024-03-18T16:20:59.948 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000648461, 2024-03-18T16:20:59.948 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000314 2024-03-18T16:20:59.948 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:59.948 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_encrypted_bytes": 663536, 2024-03-18T16:20:59.948 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_encrypted_bytes": 161728 2024-03-18T16:20:59.948 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:59.948 INFO:teuthology.orchestra.run.smithi039.stdout: "cct": { 2024-03-18T16:20:59.948 INFO:teuthology.orchestra.run.smithi039.stdout: "total_workers": 1, 2024-03-18T16:20:59.949 INFO:teuthology.orchestra.run.smithi039.stdout: "unhealthy_workers": 0 2024-03-18T16:20:59.949 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:59.949 INFO:teuthology.orchestra.run.smithi039.stdout: "finisher-MDSRank": { 2024-03-18T16:20:59.949 INFO:teuthology.orchestra.run.smithi039.stdout: "queue_len": 0, 2024-03-18T16:20:59.949 INFO:teuthology.orchestra.run.smithi039.stdout: "complete_latency": { 2024-03-18T16:20:59.949 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 3022, 2024-03-18T16:20:59.949 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 2.618405105, 2024-03-18T16:20:59.949 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000866447 2024-03-18T16:20:59.949 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:59.949 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:59.949 INFO:teuthology.orchestra.run.smithi039.stdout: "finisher-PurgeQueue": { 2024-03-18T16:20:59.949 INFO:teuthology.orchestra.run.smithi039.stdout: "queue_len": 0, 2024-03-18T16:20:59.949 INFO:teuthology.orchestra.run.smithi039.stdout: "complete_latency": { 2024-03-18T16:20:59.949 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 963, 2024-03-18T16:20:59.950 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.146906411, 2024-03-18T16:20:59.950 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000152550 2024-03-18T16:20:59.950 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:20:59.950 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:59.950 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": { 2024-03-18T16:20:59.950 INFO:teuthology.orchestra.run.smithi039.stdout: "request": 465, 2024-03-18T16:20:59.950 INFO:teuthology.orchestra.run.smithi039.stdout: "reply": 446, 2024-03-18T16:20:59.950 INFO:teuthology.orchestra.run.smithi039.stdout: "reply_latency": { 2024-03-18T16:20:59.950 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 446, 2024-03-18T16:20:59.950 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 7.541369249, 2024-03-18T16:20:59.950 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.016908899 2024-03-18T16:20:59.950 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:59.950 INFO:teuthology.orchestra.run.smithi039.stdout: "slow_reply": 0, 2024-03-18T16:20:59.950 INFO:teuthology.orchestra.run.smithi039.stdout: "forward": 19, 2024-03-18T16:20:59.950 INFO:teuthology.orchestra.run.smithi039.stdout: "dir_fetch_complete": 0, 2024-03-18T16:20:59.951 INFO:teuthology.orchestra.run.smithi039.stdout: "dir_fetch_keys": 0, 2024-03-18T16:20:59.951 INFO:teuthology.orchestra.run.smithi039.stdout: "dir_commit": 11, 2024-03-18T16:20:59.951 INFO:teuthology.orchestra.run.smithi039.stdout: "dir_split": 0, 2024-03-18T16:20:59.951 INFO:teuthology.orchestra.run.smithi039.stdout: "dir_merge": 0, 2024-03-18T16:20:59.951 INFO:teuthology.orchestra.run.smithi039.stdout: "inodes": 26, 2024-03-18T16:20:59.951 INFO:teuthology.orchestra.run.smithi039.stdout: "inodes_top": 1, 2024-03-18T16:20:59.951 INFO:teuthology.orchestra.run.smithi039.stdout: "inodes_bottom": 3, 2024-03-18T16:20:59.951 INFO:teuthology.orchestra.run.smithi039.stdout: "inodes_pin_tail": 22, 2024-03-18T16:20:59.951 INFO:teuthology.orchestra.run.smithi039.stdout: "inodes_pinned": 24, 2024-03-18T16:20:59.951 INFO:teuthology.orchestra.run.smithi039.stdout: "inodes_expired": 20, 2024-03-18T16:20:59.951 INFO:teuthology.orchestra.run.smithi039.stdout: "inodes_with_caps": 2, 2024-03-18T16:20:59.951 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": 2, 2024-03-18T16:20:59.951 INFO:teuthology.orchestra.run.smithi039.stdout: "subtrees": 4, 2024-03-18T16:20:59.951 INFO:teuthology.orchestra.run.smithi039.stdout: "traverse": 737, 2024-03-18T16:20:59.951 INFO:teuthology.orchestra.run.smithi039.stdout: "traverse_hit": 587, 2024-03-18T16:20:59.952 INFO:teuthology.orchestra.run.smithi039.stdout: "traverse_forward": 0, 2024-03-18T16:20:59.952 INFO:teuthology.orchestra.run.smithi039.stdout: "traverse_discover": 48, 2024-03-18T16:20:59.952 INFO:teuthology.orchestra.run.smithi039.stdout: "traverse_dir_fetch": 0, 2024-03-18T16:20:59.952 INFO:teuthology.orchestra.run.smithi039.stdout: "traverse_remote_ino": 0, 2024-03-18T16:20:59.952 INFO:teuthology.orchestra.run.smithi039.stdout: "traverse_lock": 0, 2024-03-18T16:20:59.952 INFO:teuthology.orchestra.run.smithi039.stdout: "load_cent": 3004, 2024-03-18T16:20:59.952 INFO:teuthology.orchestra.run.smithi039.stdout: "q": 3, 2024-03-18T16:20:59.952 INFO:teuthology.orchestra.run.smithi039.stdout: "exported": 3, 2024-03-18T16:20:59.952 INFO:teuthology.orchestra.run.smithi039.stdout: "exported_inodes": 18, 2024-03-18T16:20:59.952 INFO:teuthology.orchestra.run.smithi039.stdout: "imported": 69, 2024-03-18T16:20:59.952 INFO:teuthology.orchestra.run.smithi039.stdout: "imported_inodes": 392, 2024-03-18T16:20:59.952 INFO:teuthology.orchestra.run.smithi039.stdout: "openino_dir_fetch": 0, 2024-03-18T16:20:59.952 INFO:teuthology.orchestra.run.smithi039.stdout: "openino_backtrace_fetch": 3159, 2024-03-18T16:20:59.952 INFO:teuthology.orchestra.run.smithi039.stdout: "openino_peer_discover": 1, 2024-03-18T16:20:59.952 INFO:teuthology.orchestra.run.smithi039.stdout: "root_rfiles": 112, 2024-03-18T16:20:59.953 INFO:teuthology.orchestra.run.smithi039.stdout: "root_rbytes": 933339, 2024-03-18T16:20:59.953 INFO:teuthology.orchestra.run.smithi039.stdout: "root_rsnaps": 0, 2024-03-18T16:20:59.953 INFO:teuthology.orchestra.run.smithi039.stdout: "scrub_backtrace_fetch": 3159, 2024-03-18T16:20:59.953 INFO:teuthology.orchestra.run.smithi039.stdout: "scrub_set_tag": 0, 2024-03-18T16:20:59.953 INFO:teuthology.orchestra.run.smithi039.stdout: "scrub_backtrace_repaired": 0, 2024-03-18T16:20:59.953 INFO:teuthology.orchestra.run.smithi039.stdout: "scrub_inotable_repaired": 0, 2024-03-18T16:20:59.953 INFO:teuthology.orchestra.run.smithi039.stdout: "scrub_dir_inodes": 993, 2024-03-18T16:20:59.953 INFO:teuthology.orchestra.run.smithi039.stdout: "scrub_dir_base_inodes": 0, 2024-03-18T16:20:59.953 INFO:teuthology.orchestra.run.smithi039.stdout: "scrub_dirfrag_rstats": 993, 2024-03-18T16:20:59.953 INFO:teuthology.orchestra.run.smithi039.stdout: "scrub_file_inodes": 2166, 2024-03-18T16:20:59.953 INFO:teuthology.orchestra.run.smithi039.stdout: "handle_inode_file_caps": 40, 2024-03-18T16:20:59.953 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph_cap_op_revoke": 158, 2024-03-18T16:20:59.953 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph_cap_op_grant": 545, 2024-03-18T16:20:59.953 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph_cap_op_trunc": 0, 2024-03-18T16:20:59.954 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-18T16:20:59.954 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-18T16:20:59.954 INFO:teuthology.orchestra.run.smithi039.stdout: "handle_client_caps": 640, 2024-03-18T16:20:59.954 INFO:teuthology.orchestra.run.smithi039.stdout: "handle_client_caps_dirty": 371, 2024-03-18T16:20:59.954 INFO:teuthology.orchestra.run.smithi039.stdout: "handle_client_cap_release": 20, 2024-03-18T16:20:59.954 INFO:teuthology.orchestra.run.smithi039.stdout: "process_request_cap_release": 397 2024-03-18T16:20:59.954 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:59.954 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_cache": { 2024-03-18T16:20:59.954 INFO:teuthology.orchestra.run.smithi039.stdout: "dir_update": 0, 2024-03-18T16:20:59.954 INFO:teuthology.orchestra.run.smithi039.stdout: "dir_update_receipt": 3, 2024-03-18T16:20:59.954 INFO:teuthology.orchestra.run.smithi039.stdout: "dir_try_discover": 2, 2024-03-18T16:20:59.954 INFO:teuthology.orchestra.run.smithi039.stdout: "dir_send_discover": 90, 2024-03-18T16:20:59.954 INFO:teuthology.orchestra.run.smithi039.stdout: "dir_handle_discover": 2, 2024-03-18T16:20:59.954 INFO:teuthology.orchestra.run.smithi039.stdout: "num_strays": 1, 2024-03-18T16:20:59.954 INFO:teuthology.orchestra.run.smithi039.stdout: "num_strays_delayed": 0, 2024-03-18T16:20:59.955 INFO:teuthology.orchestra.run.smithi039.stdout: "num_strays_enqueuing": 0, 2024-03-18T16:20:59.955 INFO:teuthology.orchestra.run.smithi039.stdout: "strays_created": 267, 2024-03-18T16:20:59.955 INFO:teuthology.orchestra.run.smithi039.stdout: "strays_enqueued": 266, 2024-03-18T16:20:59.955 INFO:teuthology.orchestra.run.smithi039.stdout: "strays_reintegrated": 0, 2024-03-18T16:20:59.955 INFO:teuthology.orchestra.run.smithi039.stdout: "strays_migrated": 0, 2024-03-18T16:20:59.955 INFO:teuthology.orchestra.run.smithi039.stdout: "num_recovering_processing": 0, 2024-03-18T16:20:59.955 INFO:teuthology.orchestra.run.smithi039.stdout: "num_recovering_enqueued": 0, 2024-03-18T16:20:59.955 INFO:teuthology.orchestra.run.smithi039.stdout: "num_recovering_prioritized": 0, 2024-03-18T16:20:59.955 INFO:teuthology.orchestra.run.smithi039.stdout: "recovery_started": 0, 2024-03-18T16:20:59.955 INFO:teuthology.orchestra.run.smithi039.stdout: "recovery_completed": 0, 2024-03-18T16:20:59.955 INFO:teuthology.orchestra.run.smithi039.stdout: "ireq_quiesce_path": 0, 2024-03-18T16:20:59.955 INFO:teuthology.orchestra.run.smithi039.stdout: "ireq_quiesce_inode": 0, 2024-03-18T16:20:59.955 INFO:teuthology.orchestra.run.smithi039.stdout: "ireq_enqueue_scrub": 0, 2024-03-18T16:20:59.955 INFO:teuthology.orchestra.run.smithi039.stdout: "ireq_exportdir": 3, 2024-03-18T16:20:59.955 INFO:teuthology.orchestra.run.smithi039.stdout: "ireq_flush": 0, 2024-03-18T16:20:59.956 INFO:teuthology.orchestra.run.smithi039.stdout: "ireq_fragmentdir": 0, 2024-03-18T16:20:59.956 INFO:teuthology.orchestra.run.smithi039.stdout: "ireq_fragstats": 0, 2024-03-18T16:20:59.956 INFO:teuthology.orchestra.run.smithi039.stdout: "ireq_inodestats": 0 2024-03-18T16:20:59.956 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:59.956 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_log": { 2024-03-18T16:20:59.956 INFO:teuthology.orchestra.run.smithi039.stdout: "evlrg": 0, 2024-03-18T16:20:59.956 INFO:teuthology.orchestra.run.smithi039.stdout: "evadd": 1394, 2024-03-18T16:20:59.956 INFO:teuthology.orchestra.run.smithi039.stdout: "evex": 0, 2024-03-18T16:20:59.956 INFO:teuthology.orchestra.run.smithi039.stdout: "evtrm": 0, 2024-03-18T16:20:59.956 INFO:teuthology.orchestra.run.smithi039.stdout: "ev": 1394, 2024-03-18T16:20:59.956 INFO:teuthology.orchestra.run.smithi039.stdout: "evexg": 0, 2024-03-18T16:20:59.956 INFO:teuthology.orchestra.run.smithi039.stdout: "evexd": 0, 2024-03-18T16:20:59.956 INFO:teuthology.orchestra.run.smithi039.stdout: "segadd": 3, 2024-03-18T16:20:59.956 INFO:teuthology.orchestra.run.smithi039.stdout: "segex": 0, 2024-03-18T16:20:59.956 INFO:teuthology.orchestra.run.smithi039.stdout: "segtrm": 0, 2024-03-18T16:20:59.957 INFO:teuthology.orchestra.run.smithi039.stdout: "seg": 3, 2024-03-18T16:20:59.957 INFO:teuthology.orchestra.run.smithi039.stdout: "segmjr": 2, 2024-03-18T16:20:59.957 INFO:teuthology.orchestra.run.smithi039.stdout: "segexg": 0, 2024-03-18T16:20:59.957 INFO:teuthology.orchestra.run.smithi039.stdout: "segexd": 0, 2024-03-18T16:20:59.957 INFO:teuthology.orchestra.run.smithi039.stdout: "expos": 4194304, 2024-03-18T16:20:59.957 INFO:teuthology.orchestra.run.smithi039.stdout: "wrpos": 6645882, 2024-03-18T16:20:59.957 INFO:teuthology.orchestra.run.smithi039.stdout: "rdpos": 0, 2024-03-18T16:20:59.989 INFO:teuthology.orchestra.run.smithi039.stdout: "jlat": { 2024-03-18T16:20:59.990 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 350, 2024-03-18T16:20:59.990 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 66.776928311, 2024-03-18T16:20:59.990 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.190791223 2024-03-18T16:20:59.990 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:59.990 INFO:teuthology.orchestra.run.smithi039.stdout: "replayed": 0 2024-03-18T16:20:59.990 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:59.990 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_mem": { 2024-03-18T16:20:59.991 INFO:teuthology.orchestra.run.smithi039.stdout: "ino": 26, 2024-03-18T16:20:59.991 INFO:teuthology.orchestra.run.smithi039.stdout: "ino+": 293, 2024-03-18T16:20:59.991 INFO:teuthology.orchestra.run.smithi039.stdout: "ino-": 267, 2024-03-18T16:20:59.991 INFO:teuthology.orchestra.run.smithi039.stdout: "dir": 21, 2024-03-18T16:20:59.991 INFO:teuthology.orchestra.run.smithi039.stdout: "dir+": 103, 2024-03-18T16:20:59.991 INFO:teuthology.orchestra.run.smithi039.stdout: "dir-": 82, 2024-03-18T16:20:59.991 INFO:teuthology.orchestra.run.smithi039.stdout: "dn": 26, 2024-03-18T16:20:59.991 INFO:teuthology.orchestra.run.smithi039.stdout: "dn+": 673, 2024-03-18T16:20:59.991 INFO:teuthology.orchestra.run.smithi039.stdout: "dn-": 647, 2024-03-18T16:20:59.992 INFO:teuthology.orchestra.run.smithi039.stdout: "cap": 2, 2024-03-18T16:20:59.992 INFO:teuthology.orchestra.run.smithi039.stdout: "cap+": 377, 2024-03-18T16:20:59.992 INFO:teuthology.orchestra.run.smithi039.stdout: "cap-": 375, 2024-03-18T16:20:59.992 INFO:teuthology.orchestra.run.smithi039.stdout: "rss": 59720, 2024-03-18T16:20:59.992 INFO:teuthology.orchestra.run.smithi039.stdout: "heap": 223612 2024-03-18T16:20:59.992 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:59.992 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_server": { 2024-03-18T16:20:59.993 INFO:teuthology.orchestra.run.smithi039.stdout: "dispatch_client_request": 990, 2024-03-18T16:20:59.993 INFO:teuthology.orchestra.run.smithi039.stdout: "dispatch_server_request": 52, 2024-03-18T16:20:59.993 INFO:teuthology.orchestra.run.smithi039.stdout: "handle_client_request": 465, 2024-03-18T16:20:59.993 INFO:teuthology.orchestra.run.smithi039.stdout: "handle_client_session": 9, 2024-03-18T16:20:59.993 INFO:teuthology.orchestra.run.smithi039.stdout: "handle_peer_request": 170, 2024-03-18T16:20:59.993 INFO:teuthology.orchestra.run.smithi039.stdout: "req_create_latency": { 2024-03-18T16:20:59.993 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-03-18T16:20:59.993 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-03-18T16:20:59.993 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-03-18T16:20:59.994 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:59.994 INFO:teuthology.orchestra.run.smithi039.stdout: "req_getattr_latency": { 2024-03-18T16:20:59.994 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-03-18T16:20:59.994 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-03-18T16:20:59.994 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-03-18T16:20:59.994 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:59.994 INFO:teuthology.orchestra.run.smithi039.stdout: "req_getfilelock_latency": { 2024-03-18T16:20:59.995 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-03-18T16:20:59.995 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-03-18T16:20:59.995 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-03-18T16:20:59.995 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:59.995 INFO:teuthology.orchestra.run.smithi039.stdout: "req_link_latency": { 2024-03-18T16:20:59.995 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-03-18T16:20:59.995 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-03-18T16:20:59.995 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-03-18T16:20:59.995 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:59.996 INFO:teuthology.orchestra.run.smithi039.stdout: "req_lookup_latency": { 2024-03-18T16:20:59.996 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 121, 2024-03-18T16:20:59.996 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.156718878, 2024-03-18T16:20:59.996 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.001295197 2024-03-18T16:20:59.996 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:59.996 INFO:teuthology.orchestra.run.smithi039.stdout: "req_lookuphash_latency": { 2024-03-18T16:20:59.996 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-03-18T16:20:59.996 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-03-18T16:20:59.997 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-03-18T16:20:59.997 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:59.997 INFO:teuthology.orchestra.run.smithi039.stdout: "req_lookupino_latency": { 2024-03-18T16:20:59.997 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-03-18T16:20:59.997 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-03-18T16:20:59.997 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-03-18T16:20:59.997 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:59.997 INFO:teuthology.orchestra.run.smithi039.stdout: "req_lookupname_latency": { 2024-03-18T16:20:59.997 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-03-18T16:20:59.998 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-03-18T16:20:59.998 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-03-18T16:20:59.998 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:59.998 INFO:teuthology.orchestra.run.smithi039.stdout: "req_lookupparent_latency": { 2024-03-18T16:20:59.998 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-03-18T16:20:59.998 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-03-18T16:20:59.998 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-03-18T16:20:59.999 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:59.999 INFO:teuthology.orchestra.run.smithi039.stdout: "req_lookupsnap_latency": { 2024-03-18T16:20:59.999 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-03-18T16:20:59.999 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-03-18T16:20:59.999 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-03-18T16:20:59.999 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:20:59.999 INFO:teuthology.orchestra.run.smithi039.stdout: "req_lssnap_latency": { 2024-03-18T16:20:59.999 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-03-18T16:21:00.001 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-03-18T16:21:00.001 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-03-18T16:21:00.001 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:00.001 INFO:teuthology.orchestra.run.smithi039.stdout: "req_mkdir_latency": { 2024-03-18T16:21:00.001 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-03-18T16:21:00.001 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-03-18T16:21:00.002 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-03-18T16:21:00.002 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:00.002 INFO:teuthology.orchestra.run.smithi039.stdout: "req_mknod_latency": { 2024-03-18T16:21:00.002 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-03-18T16:21:00.002 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-03-18T16:21:00.002 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-03-18T16:21:00.002 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:00.002 INFO:teuthology.orchestra.run.smithi039.stdout: "req_mksnap_latency": { 2024-03-18T16:21:00.003 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-03-18T16:21:00.003 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-03-18T16:21:00.003 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-03-18T16:21:00.003 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:00.003 INFO:teuthology.orchestra.run.smithi039.stdout: "req_open_latency": { 2024-03-18T16:21:00.003 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 9, 2024-03-18T16:21:00.003 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.014250646, 2024-03-18T16:21:00.003 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.001583405 2024-03-18T16:21:00.004 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:00.004 INFO:teuthology.orchestra.run.smithi039.stdout: "req_readdir_latency": { 2024-03-18T16:21:00.004 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 49, 2024-03-18T16:21:00.004 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.224511549, 2024-03-18T16:21:00.004 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.004581868 2024-03-18T16:21:00.004 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:00.004 INFO:teuthology.orchestra.run.smithi039.stdout: "req_rename_latency": { 2024-03-18T16:21:00.004 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-03-18T16:21:00.004 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-03-18T16:21:00.005 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-03-18T16:21:00.005 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:00.005 INFO:teuthology.orchestra.run.smithi039.stdout: "req_renamesnap_latency": { 2024-03-18T16:21:00.005 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-03-18T16:21:00.005 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-03-18T16:21:00.005 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-03-18T16:21:00.005 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:00.005 INFO:teuthology.orchestra.run.smithi039.stdout: "req_snapdiff_latency": { 2024-03-18T16:21:00.006 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-03-18T16:21:00.006 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-03-18T16:21:00.006 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-03-18T16:21:00.006 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:00.006 INFO:teuthology.orchestra.run.smithi039.stdout: "req_rmdir_latency": { 2024-03-18T16:21:00.006 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 81, 2024-03-18T16:21:00.006 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 6.415832129, 2024-03-18T16:21:00.006 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.079207804 2024-03-18T16:21:00.006 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:00.007 INFO:teuthology.orchestra.run.smithi039.stdout: "req_rmsnap_latency": { 2024-03-18T16:21:00.007 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-03-18T16:21:00.007 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-03-18T16:21:00.007 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-03-18T16:21:00.007 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:00.007 INFO:teuthology.orchestra.run.smithi039.stdout: "req_rmxattr_latency": { 2024-03-18T16:21:00.007 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-03-18T16:21:00.007 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-03-18T16:21:00.008 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-03-18T16:21:00.008 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:00.008 INFO:teuthology.orchestra.run.smithi039.stdout: "req_setattr_latency": { 2024-03-18T16:21:00.008 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-03-18T16:21:00.008 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-03-18T16:21:00.008 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-03-18T16:21:00.008 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:00.008 INFO:teuthology.orchestra.run.smithi039.stdout: "req_setdirlayout_latency": { 2024-03-18T16:21:00.008 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-03-18T16:21:00.009 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-03-18T16:21:00.009 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-03-18T16:21:00.009 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:00.009 INFO:teuthology.orchestra.run.smithi039.stdout: "req_setfilelock_latency": { 2024-03-18T16:21:00.009 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-03-18T16:21:00.009 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-03-18T16:21:00.009 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-03-18T16:21:00.009 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:00.009 INFO:teuthology.orchestra.run.smithi039.stdout: "req_setlayout_latency": { 2024-03-18T16:21:00.010 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-03-18T16:21:00.010 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-03-18T16:21:00.010 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-03-18T16:21:00.010 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:00.010 INFO:teuthology.orchestra.run.smithi039.stdout: "req_setxattr_latency": { 2024-03-18T16:21:00.010 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-03-18T16:21:00.010 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-03-18T16:21:00.011 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-03-18T16:21:00.011 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:00.011 INFO:teuthology.orchestra.run.smithi039.stdout: "req_symlink_latency": { 2024-03-18T16:21:00.011 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-03-18T16:21:00.011 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-03-18T16:21:00.011 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-03-18T16:21:00.011 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:00.011 INFO:teuthology.orchestra.run.smithi039.stdout: "req_unlink_latency": { 2024-03-18T16:21:00.011 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 186, 2024-03-18T16:21:00.012 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.730056047, 2024-03-18T16:21:00.012 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.003925032 2024-03-18T16:21:00.012 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:00.012 INFO:teuthology.orchestra.run.smithi039.stdout: "cap_revoke_eviction": 0, 2024-03-18T16:21:00.012 INFO:teuthology.orchestra.run.smithi039.stdout: "cap_acquisition_throttle": 0, 2024-03-18T16:21:00.012 INFO:teuthology.orchestra.run.smithi039.stdout: "req_getvxattr_latency": { 2024-03-18T16:21:00.012 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-03-18T16:21:00.012 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-03-18T16:21:00.012 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-03-18T16:21:00.013 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:21:00.013 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:00.013 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_sessions": { 2024-03-18T16:21:00.013 INFO:teuthology.orchestra.run.smithi039.stdout: "session_count": 1, 2024-03-18T16:21:00.013 INFO:teuthology.orchestra.run.smithi039.stdout: "session_add": 1, 2024-03-18T16:21:00.013 INFO:teuthology.orchestra.run.smithi039.stdout: "session_remove": 0, 2024-03-18T16:21:00.013 INFO:teuthology.orchestra.run.smithi039.stdout: "sessions_open": 1, 2024-03-18T16:21:00.013 INFO:teuthology.orchestra.run.smithi039.stdout: "sessions_stale": 0, 2024-03-18T16:21:00.014 INFO:teuthology.orchestra.run.smithi039.stdout: "total_load": 389, 2024-03-18T16:21:00.014 INFO:teuthology.orchestra.run.smithi039.stdout: "average_load": 389, 2024-03-18T16:21:00.014 INFO:teuthology.orchestra.run.smithi039.stdout: "avg_session_uptime": 653, 2024-03-18T16:21:00.014 INFO:teuthology.orchestra.run.smithi039.stdout: "mdthresh_evicted": 0 2024-03-18T16:21:00.014 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:00.014 INFO:teuthology.orchestra.run.smithi039.stdout: "mempool": { 2024-03-18T16:21:00.014 INFO:teuthology.orchestra.run.smithi039.stdout: "bloom_filter_bytes": 0, 2024-03-18T16:21:00.014 INFO:teuthology.orchestra.run.smithi039.stdout: "bloom_filter_items": 0, 2024-03-18T16:21:00.014 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T16:21:00.014 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_alloc_items": 0, 2024-03-18T16:21:00.014 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T16:21:00.014 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_data_items": 0, 2024-03-18T16:21:00.014 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T16:21:00.015 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T16:21:00.015 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T16:21:00.015 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T16:21:00.015 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T16:21:00.015 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_other_items": 0, 2024-03-18T16:21:00.015 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T16:21:00.015 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T16:21:00.015 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_extent_bytes": 0, 2024-03-18T16:21:00.015 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_extent_items": 0, 2024-03-18T16:21:00.015 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_blob_bytes": 0, 2024-03-18T16:21:00.015 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_blob_items": 0, 2024-03-18T16:21:00.015 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T16:21:00.015 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T16:21:00.015 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T16:21:00.015 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T16:21:00.016 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T16:21:00.016 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_fsck_items": 0, 2024-03-18T16:21:00.016 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_txc_bytes": 0, 2024-03-18T16:21:00.016 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_txc_items": 0, 2024-03-18T16:21:00.016 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T16:21:00.016 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T16:21:00.016 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_writing_bytes": 0, 2024-03-18T16:21:00.016 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_writing_items": 0, 2024-03-18T16:21:00.016 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_bytes": 0, 2024-03-18T16:21:00.016 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_items": 0, 2024-03-18T16:21:00.016 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T16:21:00.016 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_file_reader_items": 0, 2024-03-18T16:21:00.016 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T16:21:00.016 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_file_writer_items": 0, 2024-03-18T16:21:00.016 INFO:teuthology.orchestra.run.smithi039.stdout: "buffer_anon_bytes": 319732, 2024-03-18T16:21:00.017 INFO:teuthology.orchestra.run.smithi039.stdout: "buffer_anon_items": 130, 2024-03-18T16:21:00.017 INFO:teuthology.orchestra.run.smithi039.stdout: "buffer_meta_bytes": 0, 2024-03-18T16:21:00.017 INFO:teuthology.orchestra.run.smithi039.stdout: "buffer_meta_items": 0, 2024-03-18T16:21:00.017 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_bytes": 0, 2024-03-18T16:21:00.017 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_items": 0, 2024-03-18T16:21:00.017 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_mapbl_bytes": 0, 2024-03-18T16:21:00.017 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_mapbl_items": 0, 2024-03-18T16:21:00.017 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_pglog_bytes": 0, 2024-03-18T16:21:00.017 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_pglog_items": 0, 2024-03-18T16:21:00.017 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_bytes": 8888, 2024-03-18T16:21:00.017 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_items": 170, 2024-03-18T16:21:00.017 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T16:21:00.017 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_mapping_items": 0, 2024-03-18T16:21:00.017 INFO:teuthology.orchestra.run.smithi039.stdout: "pgmap_bytes": 0, 2024-03-18T16:21:00.017 INFO:teuthology.orchestra.run.smithi039.stdout: "pgmap_items": 0, 2024-03-18T16:21:00.018 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_co_bytes": 107771, 2024-03-18T16:21:00.018 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_co_items": 760, 2024-03-18T16:21:00.018 INFO:teuthology.orchestra.run.smithi039.stdout: "unittest_1_bytes": 0, 2024-03-18T16:21:00.018 INFO:teuthology.orchestra.run.smithi039.stdout: "unittest_1_items": 0, 2024-03-18T16:21:00.018 INFO:teuthology.orchestra.run.smithi039.stdout: "unittest_2_bytes": 0, 2024-03-18T16:21:00.018 INFO:teuthology.orchestra.run.smithi039.stdout: "unittest_2_items": 0 2024-03-18T16:21:00.018 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:00.018 INFO:teuthology.orchestra.run.smithi039.stdout: "objecter": { 2024-03-18T16:21:00.018 INFO:teuthology.orchestra.run.smithi039.stdout: "op_active": 0, 2024-03-18T16:21:00.018 INFO:teuthology.orchestra.run.smithi039.stdout: "op_laggy": 0, 2024-03-18T16:21:00.018 INFO:teuthology.orchestra.run.smithi039.stdout: "op_send": 4566, 2024-03-18T16:21:00.018 INFO:teuthology.orchestra.run.smithi039.stdout: "op_send_bytes": 2530559, 2024-03-18T16:21:00.018 INFO:teuthology.orchestra.run.smithi039.stdout: "op_resend": 33, 2024-03-18T16:21:00.018 INFO:teuthology.orchestra.run.smithi039.stdout: "op_reply": 4566, 2024-03-18T16:21:00.018 INFO:teuthology.orchestra.run.smithi039.stdout: "op_latency": { 2024-03-18T16:21:00.019 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 4566, 2024-03-18T16:21:00.019 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 386.298420511, 2024-03-18T16:21:00.019 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.084603245 2024-03-18T16:21:00.019 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:00.019 INFO:teuthology.orchestra.run.smithi039.stdout: "op_inflight": 0, 2024-03-18T16:21:00.019 INFO:teuthology.orchestra.run.smithi039.stdout: "oplen_avg": { 2024-03-18T16:21:00.019 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 4566, 2024-03-18T16:21:00.019 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 4662 2024-03-18T16:21:00.019 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:00.019 INFO:teuthology.orchestra.run.smithi039.stdout: "op": 4566, 2024-03-18T16:21:00.019 INFO:teuthology.orchestra.run.smithi039.stdout: "op_r": 3175, 2024-03-18T16:21:00.019 INFO:teuthology.orchestra.run.smithi039.stdout: "op_w": 1391, 2024-03-18T16:21:00.019 INFO:teuthology.orchestra.run.smithi039.stdout: "op_rmw": 0, 2024-03-18T16:21:00.019 INFO:teuthology.orchestra.run.smithi039.stdout: "op_pg": 0, 2024-03-18T16:21:00.019 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_stat": 0, 2024-03-18T16:21:00.020 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_create": 10, 2024-03-18T16:21:00.020 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_read": 16, 2024-03-18T16:21:00.020 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_write": 366, 2024-03-18T16:21:00.020 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_writefull": 39, 2024-03-18T16:21:00.020 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_writesame": 0, 2024-03-18T16:21:00.020 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_append": 0, 2024-03-18T16:21:00.020 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_zero": 0, 2024-03-18T16:21:00.020 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_truncate": 0, 2024-03-18T16:21:00.020 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_delete": 896, 2024-03-18T16:21:00.020 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_mapext": 0, 2024-03-18T16:21:00.020 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_sparse_read": 0, 2024-03-18T16:21:00.020 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_clonerange": 0, 2024-03-18T16:21:00.020 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_getxattr": 3159, 2024-03-18T16:21:00.020 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_setxattr": 20, 2024-03-18T16:21:00.020 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_cmpxattr": 0, 2024-03-18T16:21:00.021 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_rmxattr": 0, 2024-03-18T16:21:00.021 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_resetxattrs": 0, 2024-03-18T16:21:00.021 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_call": 0, 2024-03-18T16:21:00.021 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_watch": 0, 2024-03-18T16:21:00.021 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_notify": 0, 2024-03-18T16:21:00.021 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_src_cmpxattr": 0, 2024-03-18T16:21:00.021 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_pgls": 0, 2024-03-18T16:21:00.021 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_pgls_filter": 0, 2024-03-18T16:21:00.021 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_other": 0, 2024-03-18T16:21:00.021 INFO:teuthology.orchestra.run.smithi039.stdout: "linger_active": 0, 2024-03-18T16:21:00.021 INFO:teuthology.orchestra.run.smithi039.stdout: "linger_send": 0, 2024-03-18T16:21:00.021 INFO:teuthology.orchestra.run.smithi039.stdout: "linger_resend": 0, 2024-03-18T16:21:00.021 INFO:teuthology.orchestra.run.smithi039.stdout: "linger_ping": 0, 2024-03-18T16:21:00.021 INFO:teuthology.orchestra.run.smithi039.stdout: "poolop_active": 0, 2024-03-18T16:21:00.021 INFO:teuthology.orchestra.run.smithi039.stdout: "poolop_send": 0, 2024-03-18T16:21:00.022 INFO:teuthology.orchestra.run.smithi039.stdout: "poolop_resend": 0, 2024-03-18T16:21:00.022 INFO:teuthology.orchestra.run.smithi039.stdout: "poolstat_active": 0, 2024-03-18T16:21:00.022 INFO:teuthology.orchestra.run.smithi039.stdout: "poolstat_send": 0, 2024-03-18T16:21:00.022 INFO:teuthology.orchestra.run.smithi039.stdout: "poolstat_resend": 0, 2024-03-18T16:21:00.022 INFO:teuthology.orchestra.run.smithi039.stdout: "statfs_active": 0, 2024-03-18T16:21:00.022 INFO:teuthology.orchestra.run.smithi039.stdout: "statfs_send": 0, 2024-03-18T16:21:00.022 INFO:teuthology.orchestra.run.smithi039.stdout: "statfs_resend": 0, 2024-03-18T16:21:00.022 INFO:teuthology.orchestra.run.smithi039.stdout: "command_active": 0, 2024-03-18T16:21:00.022 INFO:teuthology.orchestra.run.smithi039.stdout: "command_send": 0, 2024-03-18T16:21:00.022 INFO:teuthology.orchestra.run.smithi039.stdout: "command_resend": 0, 2024-03-18T16:21:00.022 INFO:teuthology.orchestra.run.smithi039.stdout: "map_epoch": 93, 2024-03-18T16:21:00.022 INFO:teuthology.orchestra.run.smithi039.stdout: "map_full": 0, 2024-03-18T16:21:00.022 INFO:teuthology.orchestra.run.smithi039.stdout: "map_inc": 4, 2024-03-18T16:21:00.022 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_sessions": 12, 2024-03-18T16:21:00.022 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_session_open": 12, 2024-03-18T16:21:00.023 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_session_close": 0, 2024-03-18T16:21:00.023 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_laggy": 0, 2024-03-18T16:21:00.023 INFO:teuthology.orchestra.run.smithi039.stdout: "omap_wr": 110, 2024-03-18T16:21:00.023 INFO:teuthology.orchestra.run.smithi039.stdout: "omap_rd": 0, 2024-03-18T16:21:00.023 INFO:teuthology.orchestra.run.smithi039.stdout: "omap_del": 46 2024-03-18T16:21:00.023 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:00.023 INFO:teuthology.orchestra.run.smithi039.stdout: "oft": { 2024-03-18T16:21:00.023 INFO:teuthology.orchestra.run.smithi039.stdout: "omap_total_objs": 1, 2024-03-18T16:21:00.023 INFO:teuthology.orchestra.run.smithi039.stdout: "omap_total_kv_pairs": 0, 2024-03-18T16:21:00.023 INFO:teuthology.orchestra.run.smithi039.stdout: "omap_total_updates": 0, 2024-03-18T16:21:00.024 INFO:teuthology.orchestra.run.smithi039.stdout: "omap_total_removes": 0 2024-03-18T16:21:00.024 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:00.024 INFO:teuthology.orchestra.run.smithi039.stdout: "osd-slow-ops": { 2024-03-18T16:21:00.024 INFO:teuthology.orchestra.run.smithi039.stdout: "slow_ops_count": 0 2024-03-18T16:21:00.024 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:00.024 INFO:teuthology.orchestra.run.smithi039.stdout: "purge_queue": { 2024-03-18T16:21:00.024 INFO:teuthology.orchestra.run.smithi039.stdout: "pq_executing_ops": 0, 2024-03-18T16:21:00.024 INFO:teuthology.orchestra.run.smithi039.stdout: "pq_executing_ops_high_water": 26, 2024-03-18T16:21:00.024 INFO:teuthology.orchestra.run.smithi039.stdout: "pq_executing": 0, 2024-03-18T16:21:00.024 INFO:teuthology.orchestra.run.smithi039.stdout: "pq_executing_high_water": 8, 2024-03-18T16:21:00.024 INFO:teuthology.orchestra.run.smithi039.stdout: "pq_executed_ops": 884, 2024-03-18T16:21:00.024 INFO:teuthology.orchestra.run.smithi039.stdout: "pq_executed": 266, 2024-03-18T16:21:00.025 INFO:teuthology.orchestra.run.smithi039.stdout: "pq_item_in_journal": 0 2024-03-18T16:21:00.025 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:00.025 INFO:teuthology.orchestra.run.smithi039.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T16:21:00.025 INFO:teuthology.orchestra.run.smithi039.stdout: "val": 0, 2024-03-18T16:21:00.025 INFO:teuthology.orchestra.run.smithi039.stdout: "max": 104857600, 2024-03-18T16:21:00.025 INFO:teuthology.orchestra.run.smithi039.stdout: "get_started": 0, 2024-03-18T16:21:00.025 INFO:teuthology.orchestra.run.smithi039.stdout: "get": 9989, 2024-03-18T16:21:00.025 INFO:teuthology.orchestra.run.smithi039.stdout: "get_sum": 3190047, 2024-03-18T16:21:00.026 INFO:teuthology.orchestra.run.smithi039.stdout: "get_or_fail_fail": 0, 2024-03-18T16:21:00.026 INFO:teuthology.orchestra.run.smithi039.stdout: "get_or_fail_success": 9989, 2024-03-18T16:21:00.026 INFO:teuthology.orchestra.run.smithi039.stdout: "take": 0, 2024-03-18T16:21:00.026 INFO:teuthology.orchestra.run.smithi039.stdout: "take_sum": 0, 2024-03-18T16:21:00.026 INFO:teuthology.orchestra.run.smithi039.stdout: "put": 9989, 2024-03-18T16:21:00.026 INFO:teuthology.orchestra.run.smithi039.stdout: "put_sum": 3190047, 2024-03-18T16:21:00.026 INFO:teuthology.orchestra.run.smithi039.stdout: "wait": { 2024-03-18T16:21:00.026 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-03-18T16:21:00.027 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-03-18T16:21:00.027 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-03-18T16:21:00.027 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:21:00.027 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:00.027 INFO:teuthology.orchestra.run.smithi039.stdout: "throttle-objecter_bytes": { 2024-03-18T16:21:00.027 INFO:teuthology.orchestra.run.smithi039.stdout: "val": 0, 2024-03-18T16:21:00.027 INFO:teuthology.orchestra.run.smithi039.stdout: "max": 104857600, 2024-03-18T16:21:00.027 INFO:teuthology.orchestra.run.smithi039.stdout: "get_started": 0, 2024-03-18T16:21:00.027 INFO:teuthology.orchestra.run.smithi039.stdout: "get": 0, 2024-03-18T16:21:00.028 INFO:teuthology.orchestra.run.smithi039.stdout: "get_sum": 0, 2024-03-18T16:21:00.028 INFO:teuthology.orchestra.run.smithi039.stdout: "get_or_fail_fail": 0, 2024-03-18T16:21:00.028 INFO:teuthology.orchestra.run.smithi039.stdout: "get_or_fail_success": 0, 2024-03-18T16:21:00.028 INFO:teuthology.orchestra.run.smithi039.stdout: "take": 4566, 2024-03-18T16:21:00.028 INFO:teuthology.orchestra.run.smithi039.stdout: "take_sum": 2558913, 2024-03-18T16:21:00.028 INFO:teuthology.orchestra.run.smithi039.stdout: "put": 4566, 2024-03-18T16:21:00.028 INFO:teuthology.orchestra.run.smithi039.stdout: "put_sum": 2558913, 2024-03-18T16:21:00.028 INFO:teuthology.orchestra.run.smithi039.stdout: "wait": { 2024-03-18T16:21:00.029 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-03-18T16:21:00.029 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-03-18T16:21:00.029 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-03-18T16:21:00.029 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:21:00.029 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:00.029 INFO:teuthology.orchestra.run.smithi039.stdout: "throttle-objecter_ops": { 2024-03-18T16:21:00.029 INFO:teuthology.orchestra.run.smithi039.stdout: "val": 0, 2024-03-18T16:21:00.029 INFO:teuthology.orchestra.run.smithi039.stdout: "max": 1024, 2024-03-18T16:21:00.030 INFO:teuthology.orchestra.run.smithi039.stdout: "get_started": 0, 2024-03-18T16:21:00.040 INFO:teuthology.orchestra.run.smithi039.stdout: "get": 0, 2024-03-18T16:21:00.042 INFO:teuthology.orchestra.run.smithi039.stdout: "get_sum": 0, 2024-03-18T16:21:00.042 INFO:teuthology.orchestra.run.smithi039.stdout: "get_or_fail_fail": 0, 2024-03-18T16:21:00.042 INFO:teuthology.orchestra.run.smithi039.stdout: "get_or_fail_success": 0, 2024-03-18T16:21:00.042 INFO:teuthology.orchestra.run.smithi039.stdout: "take": 4566, 2024-03-18T16:21:00.043 INFO:teuthology.orchestra.run.smithi039.stdout: "take_sum": 4566, 2024-03-18T16:21:00.043 INFO:teuthology.orchestra.run.smithi039.stdout: "put": 4566, 2024-03-18T16:21:00.043 INFO:teuthology.orchestra.run.smithi039.stdout: "put_sum": 4566, 2024-03-18T16:21:00.043 INFO:teuthology.orchestra.run.smithi039.stdout: "wait": { 2024-03-18T16:21:00.043 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-03-18T16:21:00.044 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-03-18T16:21:00.044 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-03-18T16:21:00.044 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:21:00.044 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:00.044 INFO:teuthology.orchestra.run.smithi039.stdout: "throttle-write_buf_throttle": { 2024-03-18T16:21:00.045 INFO:teuthology.orchestra.run.smithi039.stdout: "val": 0, 2024-03-18T16:21:00.045 INFO:teuthology.orchestra.run.smithi039.stdout: "max": 3758096384, 2024-03-18T16:21:00.045 INFO:teuthology.orchestra.run.smithi039.stdout: "get_started": 0, 2024-03-18T16:21:00.045 INFO:teuthology.orchestra.run.smithi039.stdout: "get": 266, 2024-03-18T16:21:00.045 INFO:teuthology.orchestra.run.smithi039.stdout: "get_sum": 28354, 2024-03-18T16:21:00.045 INFO:teuthology.orchestra.run.smithi039.stdout: "get_or_fail_fail": 0, 2024-03-18T16:21:00.046 INFO:teuthology.orchestra.run.smithi039.stdout: "get_or_fail_success": 266, 2024-03-18T16:21:00.046 INFO:teuthology.orchestra.run.smithi039.stdout: "take": 0, 2024-03-18T16:21:00.046 INFO:teuthology.orchestra.run.smithi039.stdout: "take_sum": 0, 2024-03-18T16:21:00.046 INFO:teuthology.orchestra.run.smithi039.stdout: "put": 16, 2024-03-18T16:21:00.046 INFO:teuthology.orchestra.run.smithi039.stdout: "put_sum": 28354, 2024-03-18T16:21:00.046 INFO:teuthology.orchestra.run.smithi039.stdout: "wait": { 2024-03-18T16:21:00.046 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-03-18T16:21:00.047 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-03-18T16:21:00.047 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-03-18T16:21:00.047 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:21:00.047 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:00.047 INFO:teuthology.orchestra.run.smithi039.stdout: "throttle-write_buf_throttle-0x5635e0be8500": { 2024-03-18T16:21:00.047 INFO:teuthology.orchestra.run.smithi039.stdout: "val": 0, 2024-03-18T16:21:00.047 INFO:teuthology.orchestra.run.smithi039.stdout: "max": 3758096384, 2024-03-18T16:21:00.048 INFO:teuthology.orchestra.run.smithi039.stdout: "get_started": 0, 2024-03-18T16:21:00.048 INFO:teuthology.orchestra.run.smithi039.stdout: "get": 1394, 2024-03-18T16:21:00.048 INFO:teuthology.orchestra.run.smithi039.stdout: "get_sum": 2451578, 2024-03-18T16:21:00.048 INFO:teuthology.orchestra.run.smithi039.stdout: "get_or_fail_fail": 0, 2024-03-18T16:21:00.048 INFO:teuthology.orchestra.run.smithi039.stdout: "get_or_fail_success": 1394, 2024-03-18T16:21:00.048 INFO:teuthology.orchestra.run.smithi039.stdout: "take": 0, 2024-03-18T16:21:00.048 INFO:teuthology.orchestra.run.smithi039.stdout: "take_sum": 0, 2024-03-18T16:21:00.048 INFO:teuthology.orchestra.run.smithi039.stdout: "put": 350, 2024-03-18T16:21:00.048 INFO:teuthology.orchestra.run.smithi039.stdout: "put_sum": 2451578, 2024-03-18T16:21:00.049 INFO:teuthology.orchestra.run.smithi039.stdout: "wait": { 2024-03-18T16:21:00.049 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-03-18T16:21:00.049 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-03-18T16:21:00.049 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-03-18T16:21:00.049 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:21:00.049 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:21:00.049 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-03-18T16:21:01.001 INFO:tasks.check_counter:Daemon mds.a mds.exported=3 2024-03-18T16:21:01.001 INFO:tasks.check_counter:Daemon mds.a mds.imported=69 2024-03-18T16:21:01.001 INFO:tasks.check_counter:Daemon mds.a mds.dir_split=0 2024-03-18T16:21:01.001 DEBUG:tasks.check_counter:Getting stats from d 2024-03-18T16:21:01.001 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph daemon mds.d perf dump 2024-03-18T16:21:01.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:01 smithi039 ceph-mon[31721]: pgmap v750: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-18T16:21:01.467 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T16:21:01.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:01 smithi196 ceph-mon[36307]: pgmap v750: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-18T16:21:01.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:01 smithi194 ceph-mon[36200]: pgmap v750: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 426 B/s wr, 10 op/s 2024-03-18T16:21:03.320 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:03 smithi039 ceph-mon[31721]: pgmap v751: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 255 B/s wr, 9 op/s 2024-03-18T16:21:03.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:03 smithi196 ceph-mon[36307]: pgmap v751: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 255 B/s wr, 9 op/s 2024-03-18T16:21:03.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:03 smithi194 ceph-mon[36200]: pgmap v751: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 255 B/s wr, 9 op/s 2024-03-18T16:21:03.910 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-03-18T16:21:03.910 INFO:teuthology.orchestra.run.smithi039.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T16:21:03.910 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_messages": 0, 2024-03-18T16:21:03.911 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_messages": 2, 2024-03-18T16:21:03.911 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_bytes": 0, 2024-03-18T16:21:03.911 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_bytes": 0, 2024-03-18T16:21:03.911 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_created_connections": 4, 2024-03-18T16:21:03.911 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_active_connections": 0, 2024-03-18T16:21:03.911 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_total_time": 0.003988951, 2024-03-18T16:21:03.911 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_send_time": 0.000000000, 2024-03-18T16:21:03.911 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_recv_time": 0.000806323, 2024-03-18T16:21:03.911 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-18T16:21:03.911 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T16:21:03.911 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-03-18T16:21:03.911 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-03-18T16:21:03.911 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-03-18T16:21:03.912 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:03.912 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_handle_ack_lat": { 2024-03-18T16:21:03.912 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-03-18T16:21:03.929 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-03-18T16:21:03.929 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-03-18T16:21:03.930 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:03.930 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-18T16:21:03.930 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_encrypted_bytes": 0 2024-03-18T16:21:03.930 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:03.930 INFO:teuthology.orchestra.run.smithi039.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T16:21:03.930 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_messages": 5, 2024-03-18T16:21:03.930 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_messages": 184, 2024-03-18T16:21:03.930 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_bytes": 3152, 2024-03-18T16:21:03.930 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_bytes": 110144, 2024-03-18T16:21:03.930 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_created_connections": 3, 2024-03-18T16:21:03.930 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_active_connections": 2, 2024-03-18T16:21:03.930 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_total_time": 0.027554084, 2024-03-18T16:21:03.930 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_send_time": 0.022033852, 2024-03-18T16:21:03.931 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_recv_time": 0.002958702, 2024-03-18T16:21:03.931 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-18T16:21:03.931 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T16:21:03.931 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 183, 2024-03-18T16:21:03.931 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.029456682, 2024-03-18T16:21:03.931 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000160965 2024-03-18T16:21:03.931 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:03.931 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_handle_ack_lat": { 2024-03-18T16:21:03.931 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-03-18T16:21:03.931 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-03-18T16:21:03.948 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-03-18T16:21:03.948 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:03.949 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_encrypted_bytes": 3152, 2024-03-18T16:21:03.949 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_encrypted_bytes": 110144 2024-03-18T16:21:03.949 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:03.949 INFO:teuthology.orchestra.run.smithi039.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T16:21:03.949 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_messages": 241, 2024-03-18T16:21:03.949 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_messages": 238, 2024-03-18T16:21:03.949 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_bytes": 482528, 2024-03-18T16:21:03.949 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_bytes": 147104, 2024-03-18T16:21:03.949 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_created_connections": 2, 2024-03-18T16:21:03.949 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_active_connections": 1, 2024-03-18T16:21:03.949 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_total_time": 0.075229384, 2024-03-18T16:21:03.949 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_send_time": 0.033865352, 2024-03-18T16:21:03.949 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_recv_time": 0.035661733, 2024-03-18T16:21:03.950 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_fast_dispatch_time": 0.009097589, 2024-03-18T16:21:03.950 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T16:21:03.950 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 237, 2024-03-18T16:21:03.950 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.031107973, 2024-03-18T16:21:03.950 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000131257 2024-03-18T16:21:03.950 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:03.950 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_handle_ack_lat": { 2024-03-18T16:21:03.950 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-03-18T16:21:03.950 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-03-18T16:21:03.950 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-03-18T16:21:03.950 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:03.950 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_encrypted_bytes": 482528, 2024-03-18T16:21:03.950 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_encrypted_bytes": 147104 2024-03-18T16:21:03.950 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:03.951 INFO:teuthology.orchestra.run.smithi039.stdout: "cct": { 2024-03-18T16:21:03.951 INFO:teuthology.orchestra.run.smithi039.stdout: "total_workers": 0, 2024-03-18T16:21:03.951 INFO:teuthology.orchestra.run.smithi039.stdout: "unhealthy_workers": 0 2024-03-18T16:21:03.951 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:03.951 INFO:teuthology.orchestra.run.smithi039.stdout: "mempool": { 2024-03-18T16:21:03.951 INFO:teuthology.orchestra.run.smithi039.stdout: "bloom_filter_bytes": 0, 2024-03-18T16:21:03.951 INFO:teuthology.orchestra.run.smithi039.stdout: "bloom_filter_items": 0, 2024-03-18T16:21:03.951 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T16:21:03.951 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_alloc_items": 0, 2024-03-18T16:21:03.951 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T16:21:03.951 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_data_items": 0, 2024-03-18T16:21:03.951 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T16:21:03.951 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T16:21:03.951 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T16:21:03.952 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T16:21:03.952 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T16:21:03.952 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_other_items": 0, 2024-03-18T16:21:03.952 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T16:21:03.952 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T16:21:03.952 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_extent_bytes": 0, 2024-03-18T16:21:03.952 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_extent_items": 0, 2024-03-18T16:21:03.952 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_blob_bytes": 0, 2024-03-18T16:21:03.952 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_blob_items": 0, 2024-03-18T16:21:03.952 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T16:21:03.952 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T16:21:03.952 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T16:21:03.952 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T16:21:03.952 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T16:21:03.952 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_fsck_items": 0, 2024-03-18T16:21:03.953 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_txc_bytes": 0, 2024-03-18T16:21:03.953 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_txc_items": 0, 2024-03-18T16:21:03.953 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T16:21:03.953 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T16:21:03.953 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_writing_bytes": 0, 2024-03-18T16:21:03.953 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_writing_items": 0, 2024-03-18T16:21:03.953 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_bytes": 0, 2024-03-18T16:21:03.953 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_items": 0, 2024-03-18T16:21:03.953 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T16:21:03.953 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_file_reader_items": 0, 2024-03-18T16:21:03.953 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T16:21:03.953 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_file_writer_items": 0, 2024-03-18T16:21:03.953 INFO:teuthology.orchestra.run.smithi039.stdout: "buffer_anon_bytes": 148967, 2024-03-18T16:21:03.953 INFO:teuthology.orchestra.run.smithi039.stdout: "buffer_anon_items": 29, 2024-03-18T16:21:03.953 INFO:teuthology.orchestra.run.smithi039.stdout: "buffer_meta_bytes": 0, 2024-03-18T16:21:03.954 INFO:teuthology.orchestra.run.smithi039.stdout: "buffer_meta_items": 0, 2024-03-18T16:21:03.954 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_bytes": 0, 2024-03-18T16:21:03.954 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_items": 0, 2024-03-18T16:21:03.954 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_mapbl_bytes": 0, 2024-03-18T16:21:03.954 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_mapbl_items": 0, 2024-03-18T16:21:03.954 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_pglog_bytes": 0, 2024-03-18T16:21:03.954 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_pglog_items": 0, 2024-03-18T16:21:03.954 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_bytes": 0, 2024-03-18T16:21:03.954 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_items": 0, 2024-03-18T16:21:03.954 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T16:21:03.954 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_mapping_items": 0, 2024-03-18T16:21:03.954 INFO:teuthology.orchestra.run.smithi039.stdout: "pgmap_bytes": 0, 2024-03-18T16:21:03.954 INFO:teuthology.orchestra.run.smithi039.stdout: "pgmap_items": 0, 2024-03-18T16:21:03.955 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_co_bytes": 928, 2024-03-18T16:21:03.955 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_co_items": 2, 2024-03-18T16:21:03.955 INFO:teuthology.orchestra.run.smithi039.stdout: "unittest_1_bytes": 0, 2024-03-18T16:21:03.955 INFO:teuthology.orchestra.run.smithi039.stdout: "unittest_1_items": 0, 2024-03-18T16:21:03.955 INFO:teuthology.orchestra.run.smithi039.stdout: "unittest_2_bytes": 0, 2024-03-18T16:21:03.955 INFO:teuthology.orchestra.run.smithi039.stdout: "unittest_2_items": 0 2024-03-18T16:21:03.955 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:03.955 INFO:teuthology.orchestra.run.smithi039.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T16:21:03.955 INFO:teuthology.orchestra.run.smithi039.stdout: "val": 0, 2024-03-18T16:21:03.955 INFO:teuthology.orchestra.run.smithi039.stdout: "max": 104857600, 2024-03-18T16:21:03.955 INFO:teuthology.orchestra.run.smithi039.stdout: "get_started": 0, 2024-03-18T16:21:03.955 INFO:teuthology.orchestra.run.smithi039.stdout: "get": 242, 2024-03-18T16:21:03.955 INFO:teuthology.orchestra.run.smithi039.stdout: "get_sum": 448455, 2024-03-18T16:21:03.955 INFO:teuthology.orchestra.run.smithi039.stdout: "get_or_fail_fail": 0, 2024-03-18T16:21:03.956 INFO:teuthology.orchestra.run.smithi039.stdout: "get_or_fail_success": 242, 2024-03-18T16:21:03.956 INFO:teuthology.orchestra.run.smithi039.stdout: "take": 0, 2024-03-18T16:21:03.956 INFO:teuthology.orchestra.run.smithi039.stdout: "take_sum": 0, 2024-03-18T16:21:03.956 INFO:teuthology.orchestra.run.smithi039.stdout: "put": 242, 2024-03-18T16:21:03.956 INFO:teuthology.orchestra.run.smithi039.stdout: "put_sum": 448455, 2024-03-18T16:21:03.956 INFO:teuthology.orchestra.run.smithi039.stdout: "wait": { 2024-03-18T16:21:03.956 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-03-18T16:21:03.956 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-03-18T16:21:03.956 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-03-18T16:21:03.956 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:21:03.956 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:21:03.956 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-03-18T16:21:04.794 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-03-18T16:21:04.794 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-03-18T16:21:04.794 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.d 2024-03-18T16:21:04.794 DEBUG:tasks.check_counter:Getting stats from g 2024-03-18T16:21:04.794 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph daemon mds.g perf dump 2024-03-18T16:21:05.280 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T16:21:05.436 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:05 smithi039 ceph-mon[31721]: pgmap v752: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 255 B/s wr, 9 op/s 2024-03-18T16:21:05.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:05 smithi196 ceph-mon[36307]: pgmap v752: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 255 B/s wr, 9 op/s 2024-03-18T16:21:05.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:05 smithi194 ceph-mon[36200]: pgmap v752: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 255 B/s wr, 9 op/s 2024-03-18T16:21:07.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:07 smithi039 ceph-mon[31721]: pgmap v753: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 85 B/s wr, 10 op/s 2024-03-18T16:21:07.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:07 smithi196 ceph-mon[36307]: pgmap v753: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 85 B/s wr, 10 op/s 2024-03-18T16:21:07.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:07 smithi194 ceph-mon[36200]: pgmap v753: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 85 B/s wr, 10 op/s 2024-03-18T16:21:07.782 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-03-18T16:21:07.782 INFO:teuthology.orchestra.run.smithi039.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T16:21:07.782 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_messages": 704, 2024-03-18T16:21:07.782 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_messages": 704, 2024-03-18T16:21:07.782 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_bytes": 238137, 2024-03-18T16:21:07.783 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_bytes": 223049, 2024-03-18T16:21:07.783 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_created_connections": 6, 2024-03-18T16:21:07.783 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_active_connections": 2, 2024-03-18T16:21:07.783 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_total_time": 0.144288755, 2024-03-18T16:21:07.783 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_send_time": 0.059642353, 2024-03-18T16:21:07.783 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_recv_time": 0.031783951, 2024-03-18T16:21:07.783 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_fast_dispatch_time": 0.038891666, 2024-03-18T16:21:07.783 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T16:21:07.783 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 703, 2024-03-18T16:21:07.783 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.095844856, 2024-03-18T16:21:07.783 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000136336 2024-03-18T16:21:07.783 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:07.783 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_handle_ack_lat": { 2024-03-18T16:21:07.784 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-03-18T16:21:07.784 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-03-18T16:21:07.784 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-03-18T16:21:07.784 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:07.784 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-03-18T16:21:07.784 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_encrypted_bytes": 448 2024-03-18T16:21:07.784 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:07.784 INFO:teuthology.orchestra.run.smithi039.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T16:21:07.784 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_messages": 1358, 2024-03-18T16:21:07.784 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_messages": 1536, 2024-03-18T16:21:07.784 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_bytes": 2796808, 2024-03-18T16:21:07.784 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_bytes": 818287, 2024-03-18T16:21:07.784 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_created_connections": 5, 2024-03-18T16:21:07.785 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_active_connections": 4, 2024-03-18T16:21:07.785 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_total_time": 0.206556158, 2024-03-18T16:21:07.785 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_send_time": 0.079310139, 2024-03-18T16:21:07.785 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_recv_time": 0.050444828, 2024-03-18T16:21:07.785 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_fast_dispatch_time": 0.061387659, 2024-03-18T16:21:07.785 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T16:21:07.785 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 1534, 2024-03-18T16:21:07.785 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.194919020, 2024-03-18T16:21:07.785 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000127065 2024-03-18T16:21:07.785 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:07.785 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_handle_ack_lat": { 2024-03-18T16:21:07.785 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-03-18T16:21:07.785 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-03-18T16:21:07.785 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-03-18T16:21:07.786 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:07.786 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-18T16:21:07.786 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_encrypted_bytes": 386704 2024-03-18T16:21:07.786 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:07.786 INFO:teuthology.orchestra.run.smithi039.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T16:21:07.786 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_messages": 270, 2024-03-18T16:21:07.786 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_messages": 246, 2024-03-18T16:21:07.786 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_bytes": 642996, 2024-03-18T16:21:07.786 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_bytes": 147045, 2024-03-18T16:21:07.786 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_created_connections": 4, 2024-03-18T16:21:07.786 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_active_connections": 3, 2024-03-18T16:21:07.786 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_total_time": 0.085745286, 2024-03-18T16:21:07.786 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_send_time": 0.038060774, 2024-03-18T16:21:07.787 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_recv_time": 0.029851705, 2024-03-18T16:21:07.787 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_fast_dispatch_time": 0.009163681, 2024-03-18T16:21:07.787 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T16:21:07.787 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 245, 2024-03-18T16:21:07.787 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.054263865, 2024-03-18T16:21:07.787 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000221485 2024-03-18T16:21:07.787 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:07.787 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_handle_ack_lat": { 2024-03-18T16:21:07.787 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-03-18T16:21:07.787 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-03-18T16:21:07.787 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-03-18T16:21:07.787 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:07.788 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_encrypted_bytes": 641648, 2024-03-18T16:21:07.788 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_encrypted_bytes": 146080 2024-03-18T16:21:07.788 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:07.788 INFO:teuthology.orchestra.run.smithi039.stdout: "cct": { 2024-03-18T16:21:07.788 INFO:teuthology.orchestra.run.smithi039.stdout: "total_workers": 1, 2024-03-18T16:21:07.788 INFO:teuthology.orchestra.run.smithi039.stdout: "unhealthy_workers": 0 2024-03-18T16:21:07.788 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:07.788 INFO:teuthology.orchestra.run.smithi039.stdout: "finisher-MDSRank": { 2024-03-18T16:21:07.788 INFO:teuthology.orchestra.run.smithi039.stdout: "queue_len": 0, 2024-03-18T16:21:07.788 INFO:teuthology.orchestra.run.smithi039.stdout: "complete_latency": { 2024-03-18T16:21:07.788 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 3466, 2024-03-18T16:21:07.788 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.144131443, 2024-03-18T16:21:07.788 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000041584 2024-03-18T16:21:07.788 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:21:07.789 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:07.789 INFO:teuthology.orchestra.run.smithi039.stdout: "finisher-PurgeQueue": { 2024-03-18T16:21:07.789 INFO:teuthology.orchestra.run.smithi039.stdout: "queue_len": 0, 2024-03-18T16:21:07.789 INFO:teuthology.orchestra.run.smithi039.stdout: "complete_latency": { 2024-03-18T16:21:07.789 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-03-18T16:21:07.789 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-03-18T16:21:07.789 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-03-18T16:21:07.789 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:21:07.789 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:07.789 INFO:teuthology.orchestra.run.smithi039.stdout: "mds": { 2024-03-18T16:21:07.789 INFO:teuthology.orchestra.run.smithi039.stdout: "request": 0, 2024-03-18T16:21:07.789 INFO:teuthology.orchestra.run.smithi039.stdout: "reply": 0, 2024-03-18T16:21:07.789 INFO:teuthology.orchestra.run.smithi039.stdout: "reply_latency": { 2024-03-18T16:21:07.790 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-03-18T16:21:07.790 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-03-18T16:21:07.790 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-03-18T16:21:07.790 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:07.790 INFO:teuthology.orchestra.run.smithi039.stdout: "slow_reply": 0, 2024-03-18T16:21:07.790 INFO:teuthology.orchestra.run.smithi039.stdout: "forward": 0, 2024-03-18T16:21:07.790 INFO:teuthology.orchestra.run.smithi039.stdout: "dir_fetch_complete": 0, 2024-03-18T16:21:07.790 INFO:teuthology.orchestra.run.smithi039.stdout: "dir_fetch_keys": 0, 2024-03-18T16:21:07.790 INFO:teuthology.orchestra.run.smithi039.stdout: "dir_commit": 0, 2024-03-18T16:21:07.790 INFO:teuthology.orchestra.run.smithi039.stdout: "dir_split": 0, 2024-03-18T16:21:07.790 INFO:teuthology.orchestra.run.smithi039.stdout: "dir_merge": 0, 2024-03-18T16:21:07.790 INFO:teuthology.orchestra.run.smithi039.stdout: "inodes": 415, 2024-03-18T16:21:07.790 INFO:teuthology.orchestra.run.smithi039.stdout: "inodes_top": 0, 2024-03-18T16:21:07.791 INFO:teuthology.orchestra.run.smithi039.stdout: "inodes_bottom": 415, 2024-03-18T16:21:07.791 INFO:teuthology.orchestra.run.smithi039.stdout: "inodes_pin_tail": 0, 2024-03-18T16:21:07.791 INFO:teuthology.orchestra.run.smithi039.stdout: "inodes_pinned": 415, 2024-03-18T16:21:07.791 INFO:teuthology.orchestra.run.smithi039.stdout: "inodes_expired": 0, 2024-03-18T16:21:07.791 INFO:teuthology.orchestra.run.smithi039.stdout: "inodes_with_caps": 0, 2024-03-18T16:21:07.791 INFO:teuthology.orchestra.run.smithi039.stdout: "caps": 0, 2024-03-18T16:21:07.791 INFO:teuthology.orchestra.run.smithi039.stdout: "subtrees": 3, 2024-03-18T16:21:07.791 INFO:teuthology.orchestra.run.smithi039.stdout: "traverse": 0, 2024-03-18T16:21:07.791 INFO:teuthology.orchestra.run.smithi039.stdout: "traverse_hit": 0, 2024-03-18T16:21:07.791 INFO:teuthology.orchestra.run.smithi039.stdout: "traverse_forward": 0, 2024-03-18T16:21:07.792 INFO:teuthology.orchestra.run.smithi039.stdout: "traverse_discover": 0, 2024-03-18T16:21:07.792 INFO:teuthology.orchestra.run.smithi039.stdout: "traverse_dir_fetch": 0, 2024-03-18T16:21:07.792 INFO:teuthology.orchestra.run.smithi039.stdout: "traverse_remote_ino": 0, 2024-03-18T16:21:07.792 INFO:teuthology.orchestra.run.smithi039.stdout: "traverse_lock": 0, 2024-03-18T16:21:07.792 INFO:teuthology.orchestra.run.smithi039.stdout: "load_cent": 0, 2024-03-18T16:21:07.792 INFO:teuthology.orchestra.run.smithi039.stdout: "q": 0, 2024-03-18T16:21:07.792 INFO:teuthology.orchestra.run.smithi039.stdout: "exported": 0, 2024-03-18T16:21:07.792 INFO:teuthology.orchestra.run.smithi039.stdout: "exported_inodes": 0, 2024-03-18T16:21:07.792 INFO:teuthology.orchestra.run.smithi039.stdout: "imported": 0, 2024-03-18T16:21:07.792 INFO:teuthology.orchestra.run.smithi039.stdout: "imported_inodes": 0, 2024-03-18T16:21:07.793 INFO:teuthology.orchestra.run.smithi039.stdout: "openino_dir_fetch": 0, 2024-03-18T16:21:07.793 INFO:teuthology.orchestra.run.smithi039.stdout: "openino_backtrace_fetch": 0, 2024-03-18T16:21:07.793 INFO:teuthology.orchestra.run.smithi039.stdout: "openino_peer_discover": 0, 2024-03-18T16:21:07.793 INFO:teuthology.orchestra.run.smithi039.stdout: "root_rfiles": 0, 2024-03-18T16:21:07.793 INFO:teuthology.orchestra.run.smithi039.stdout: "root_rbytes": 0, 2024-03-18T16:21:07.793 INFO:teuthology.orchestra.run.smithi039.stdout: "root_rsnaps": 0, 2024-03-18T16:21:07.793 INFO:teuthology.orchestra.run.smithi039.stdout: "scrub_backtrace_fetch": 0, 2024-03-18T16:21:07.793 INFO:teuthology.orchestra.run.smithi039.stdout: "scrub_set_tag": 0, 2024-03-18T16:21:07.793 INFO:teuthology.orchestra.run.smithi039.stdout: "scrub_backtrace_repaired": 0, 2024-03-18T16:21:07.793 INFO:teuthology.orchestra.run.smithi039.stdout: "scrub_inotable_repaired": 0, 2024-03-18T16:21:07.793 INFO:teuthology.orchestra.run.smithi039.stdout: "scrub_dir_inodes": 0, 2024-03-18T16:21:07.793 INFO:teuthology.orchestra.run.smithi039.stdout: "scrub_dir_base_inodes": 0, 2024-03-18T16:21:07.793 INFO:teuthology.orchestra.run.smithi039.stdout: "scrub_dirfrag_rstats": 0, 2024-03-18T16:21:07.793 INFO:teuthology.orchestra.run.smithi039.stdout: "scrub_file_inodes": 0, 2024-03-18T16:21:07.794 INFO:teuthology.orchestra.run.smithi039.stdout: "handle_inode_file_caps": 0, 2024-03-18T16:21:07.794 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph_cap_op_revoke": 0, 2024-03-18T16:21:07.794 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph_cap_op_grant": 0, 2024-03-18T16:21:07.794 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph_cap_op_trunc": 0, 2024-03-18T16:21:07.794 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-18T16:21:07.794 INFO:teuthology.orchestra.run.smithi039.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-18T16:21:07.794 INFO:teuthology.orchestra.run.smithi039.stdout: "handle_client_caps": 0, 2024-03-18T16:21:07.794 INFO:teuthology.orchestra.run.smithi039.stdout: "handle_client_caps_dirty": 0, 2024-03-18T16:21:07.794 INFO:teuthology.orchestra.run.smithi039.stdout: "handle_client_cap_release": 0, 2024-03-18T16:21:07.794 INFO:teuthology.orchestra.run.smithi039.stdout: "process_request_cap_release": 0 2024-03-18T16:21:07.794 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:07.794 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_cache": { 2024-03-18T16:21:07.794 INFO:teuthology.orchestra.run.smithi039.stdout: "dir_update": 0, 2024-03-18T16:21:07.794 INFO:teuthology.orchestra.run.smithi039.stdout: "dir_update_receipt": 0, 2024-03-18T16:21:07.795 INFO:teuthology.orchestra.run.smithi039.stdout: "dir_try_discover": 0, 2024-03-18T16:21:07.795 INFO:teuthology.orchestra.run.smithi039.stdout: "dir_send_discover": 0, 2024-03-18T16:21:07.795 INFO:teuthology.orchestra.run.smithi039.stdout: "dir_handle_discover": 0, 2024-03-18T16:21:07.795 INFO:teuthology.orchestra.run.smithi039.stdout: "num_strays": 0, 2024-03-18T16:21:07.795 INFO:teuthology.orchestra.run.smithi039.stdout: "num_strays_delayed": 0, 2024-03-18T16:21:07.795 INFO:teuthology.orchestra.run.smithi039.stdout: "num_strays_enqueuing": 0, 2024-03-18T16:21:07.795 INFO:teuthology.orchestra.run.smithi039.stdout: "strays_created": 0, 2024-03-18T16:21:07.795 INFO:teuthology.orchestra.run.smithi039.stdout: "strays_enqueued": 0, 2024-03-18T16:21:07.795 INFO:teuthology.orchestra.run.smithi039.stdout: "strays_reintegrated": 0, 2024-03-18T16:21:07.795 INFO:teuthology.orchestra.run.smithi039.stdout: "strays_migrated": 0, 2024-03-18T16:21:07.795 INFO:teuthology.orchestra.run.smithi039.stdout: "num_recovering_processing": 0, 2024-03-18T16:21:07.795 INFO:teuthology.orchestra.run.smithi039.stdout: "num_recovering_enqueued": 0, 2024-03-18T16:21:07.795 INFO:teuthology.orchestra.run.smithi039.stdout: "num_recovering_prioritized": 0, 2024-03-18T16:21:07.795 INFO:teuthology.orchestra.run.smithi039.stdout: "recovery_started": 0, 2024-03-18T16:21:07.796 INFO:teuthology.orchestra.run.smithi039.stdout: "recovery_completed": 0, 2024-03-18T16:21:07.796 INFO:teuthology.orchestra.run.smithi039.stdout: "ireq_quiesce_path": 0, 2024-03-18T16:21:07.796 INFO:teuthology.orchestra.run.smithi039.stdout: "ireq_quiesce_inode": 0, 2024-03-18T16:21:07.796 INFO:teuthology.orchestra.run.smithi039.stdout: "ireq_enqueue_scrub": 0, 2024-03-18T16:21:07.796 INFO:teuthology.orchestra.run.smithi039.stdout: "ireq_exportdir": 0, 2024-03-18T16:21:07.796 INFO:teuthology.orchestra.run.smithi039.stdout: "ireq_flush": 0, 2024-03-18T16:21:07.796 INFO:teuthology.orchestra.run.smithi039.stdout: "ireq_fragmentdir": 0, 2024-03-18T16:21:07.796 INFO:teuthology.orchestra.run.smithi039.stdout: "ireq_fragstats": 0, 2024-03-18T16:21:07.796 INFO:teuthology.orchestra.run.smithi039.stdout: "ireq_inodestats": 0 2024-03-18T16:21:07.796 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:07.796 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_log": { 2024-03-18T16:21:07.796 INFO:teuthology.orchestra.run.smithi039.stdout: "evlrg": 0, 2024-03-18T16:21:07.797 INFO:teuthology.orchestra.run.smithi039.stdout: "evadd": 0, 2024-03-18T16:21:07.797 INFO:teuthology.orchestra.run.smithi039.stdout: "evex": 0, 2024-03-18T16:21:07.797 INFO:teuthology.orchestra.run.smithi039.stdout: "evtrm": 0, 2024-03-18T16:21:07.797 INFO:teuthology.orchestra.run.smithi039.stdout: "ev": 1394, 2024-03-18T16:21:07.797 INFO:teuthology.orchestra.run.smithi039.stdout: "evexg": 0, 2024-03-18T16:21:07.797 INFO:teuthology.orchestra.run.smithi039.stdout: "evexd": 0, 2024-03-18T16:21:07.797 INFO:teuthology.orchestra.run.smithi039.stdout: "segadd": 0, 2024-03-18T16:21:07.797 INFO:teuthology.orchestra.run.smithi039.stdout: "segex": 0, 2024-03-18T16:21:07.797 INFO:teuthology.orchestra.run.smithi039.stdout: "segtrm": 0, 2024-03-18T16:21:07.797 INFO:teuthology.orchestra.run.smithi039.stdout: "seg": 3, 2024-03-18T16:21:07.797 INFO:teuthology.orchestra.run.smithi039.stdout: "segmjr": 2, 2024-03-18T16:21:07.797 INFO:teuthology.orchestra.run.smithi039.stdout: "segexg": 0, 2024-03-18T16:21:07.797 INFO:teuthology.orchestra.run.smithi039.stdout: "segexd": 0, 2024-03-18T16:21:07.797 INFO:teuthology.orchestra.run.smithi039.stdout: "expos": 4194304, 2024-03-18T16:21:07.797 INFO:teuthology.orchestra.run.smithi039.stdout: "wrpos": 6645882, 2024-03-18T16:21:07.798 INFO:teuthology.orchestra.run.smithi039.stdout: "rdpos": 6645308, 2024-03-18T16:21:07.798 INFO:teuthology.orchestra.run.smithi039.stdout: "jlat": { 2024-03-18T16:21:07.798 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-03-18T16:21:07.798 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-03-18T16:21:07.798 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-03-18T16:21:07.798 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:07.798 INFO:teuthology.orchestra.run.smithi039.stdout: "replayed": 1394 2024-03-18T16:21:07.798 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:07.798 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_mem": { 2024-03-18T16:21:07.798 INFO:teuthology.orchestra.run.smithi039.stdout: "ino": 16, 2024-03-18T16:21:07.798 INFO:teuthology.orchestra.run.smithi039.stdout: "ino+": 285, 2024-03-18T16:21:07.798 INFO:teuthology.orchestra.run.smithi039.stdout: "ino-": 269, 2024-03-18T16:21:07.798 INFO:teuthology.orchestra.run.smithi039.stdout: "dir": 11, 2024-03-18T16:21:07.798 INFO:teuthology.orchestra.run.smithi039.stdout: "dir+": 94, 2024-03-18T16:21:07.799 INFO:teuthology.orchestra.run.smithi039.stdout: "dir-": 83, 2024-03-18T16:21:07.799 INFO:teuthology.orchestra.run.smithi039.stdout: "dn": 415, 2024-03-18T16:21:07.799 INFO:teuthology.orchestra.run.smithi039.stdout: "dn+": 665, 2024-03-18T16:21:07.799 INFO:teuthology.orchestra.run.smithi039.stdout: "dn-": 250, 2024-03-18T16:21:07.799 INFO:teuthology.orchestra.run.smithi039.stdout: "cap": 0, 2024-03-18T16:21:07.799 INFO:teuthology.orchestra.run.smithi039.stdout: "cap+": 0, 2024-03-18T16:21:07.799 INFO:teuthology.orchestra.run.smithi039.stdout: "cap-": 0, 2024-03-18T16:21:07.799 INFO:teuthology.orchestra.run.smithi039.stdout: "rss": 52028, 2024-03-18T16:21:07.799 INFO:teuthology.orchestra.run.smithi039.stdout: "heap": 215420 2024-03-18T16:21:07.799 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:07.799 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_server": { 2024-03-18T16:21:07.799 INFO:teuthology.orchestra.run.smithi039.stdout: "dispatch_client_request": 0, 2024-03-18T16:21:07.799 INFO:teuthology.orchestra.run.smithi039.stdout: "dispatch_server_request": 0, 2024-03-18T16:21:07.799 INFO:teuthology.orchestra.run.smithi039.stdout: "handle_client_request": 0, 2024-03-18T16:21:07.800 INFO:teuthology.orchestra.run.smithi039.stdout: "handle_client_session": 0, 2024-03-18T16:21:07.800 INFO:teuthology.orchestra.run.smithi039.stdout: "handle_peer_request": 0, 2024-03-18T16:21:07.800 INFO:teuthology.orchestra.run.smithi039.stdout: "req_create_latency": { 2024-03-18T16:21:07.800 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-03-18T16:21:07.800 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-03-18T16:21:07.800 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-03-18T16:21:07.800 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:07.800 INFO:teuthology.orchestra.run.smithi039.stdout: "req_getattr_latency": { 2024-03-18T16:21:07.800 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-03-18T16:21:07.800 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-03-18T16:21:07.800 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-03-18T16:21:07.800 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:07.800 INFO:teuthology.orchestra.run.smithi039.stdout: "req_getfilelock_latency": { 2024-03-18T16:21:07.800 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-03-18T16:21:07.801 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-03-18T16:21:07.801 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-03-18T16:21:07.801 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:07.801 INFO:teuthology.orchestra.run.smithi039.stdout: "req_link_latency": { 2024-03-18T16:21:07.801 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-03-18T16:21:07.801 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-03-18T16:21:07.801 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-03-18T16:21:07.801 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:07.801 INFO:teuthology.orchestra.run.smithi039.stdout: "req_lookup_latency": { 2024-03-18T16:21:07.801 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-03-18T16:21:07.801 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-03-18T16:21:07.801 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-03-18T16:21:07.801 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:07.801 INFO:teuthology.orchestra.run.smithi039.stdout: "req_lookuphash_latency": { 2024-03-18T16:21:07.801 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-03-18T16:21:07.802 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-03-18T16:21:07.802 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-03-18T16:21:07.802 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:07.802 INFO:teuthology.orchestra.run.smithi039.stdout: "req_lookupino_latency": { 2024-03-18T16:21:07.802 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-03-18T16:21:07.802 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-03-18T16:21:07.802 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-03-18T16:21:07.802 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:07.802 INFO:teuthology.orchestra.run.smithi039.stdout: "req_lookupname_latency": { 2024-03-18T16:21:07.802 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-03-18T16:21:07.802 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-03-18T16:21:07.802 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-03-18T16:21:07.802 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:07.802 INFO:teuthology.orchestra.run.smithi039.stdout: "req_lookupparent_latency": { 2024-03-18T16:21:07.803 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-03-18T16:21:07.803 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-03-18T16:21:07.803 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-03-18T16:21:07.803 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:07.803 INFO:teuthology.orchestra.run.smithi039.stdout: "req_lookupsnap_latency": { 2024-03-18T16:21:07.803 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-03-18T16:21:07.803 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-03-18T16:21:07.803 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-03-18T16:21:07.803 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:07.803 INFO:teuthology.orchestra.run.smithi039.stdout: "req_lssnap_latency": { 2024-03-18T16:21:07.803 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-03-18T16:21:07.803 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-03-18T16:21:07.803 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-03-18T16:21:07.803 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:07.803 INFO:teuthology.orchestra.run.smithi039.stdout: "req_mkdir_latency": { 2024-03-18T16:21:07.804 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-03-18T16:21:07.804 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-03-18T16:21:07.804 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-03-18T16:21:07.804 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:07.804 INFO:teuthology.orchestra.run.smithi039.stdout: "req_mknod_latency": { 2024-03-18T16:21:07.804 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-03-18T16:21:07.804 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-03-18T16:21:07.804 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-03-18T16:21:07.804 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:07.804 INFO:teuthology.orchestra.run.smithi039.stdout: "req_mksnap_latency": { 2024-03-18T16:21:07.804 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-03-18T16:21:07.804 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-03-18T16:21:07.804 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-03-18T16:21:07.805 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:07.805 INFO:teuthology.orchestra.run.smithi039.stdout: "req_open_latency": { 2024-03-18T16:21:07.805 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-03-18T16:21:07.805 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-03-18T16:21:07.805 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-03-18T16:21:07.805 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:07.805 INFO:teuthology.orchestra.run.smithi039.stdout: "req_readdir_latency": { 2024-03-18T16:21:07.805 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-03-18T16:21:07.805 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-03-18T16:21:07.805 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-03-18T16:21:07.805 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:07.805 INFO:teuthology.orchestra.run.smithi039.stdout: "req_rename_latency": { 2024-03-18T16:21:07.805 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-03-18T16:21:07.805 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-03-18T16:21:07.805 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-03-18T16:21:07.806 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:07.806 INFO:teuthology.orchestra.run.smithi039.stdout: "req_renamesnap_latency": { 2024-03-18T16:21:07.806 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-03-18T16:21:07.806 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-03-18T16:21:07.806 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-03-18T16:21:07.806 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:07.806 INFO:teuthology.orchestra.run.smithi039.stdout: "req_snapdiff_latency": { 2024-03-18T16:21:07.806 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-03-18T16:21:07.806 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-03-18T16:21:07.806 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-03-18T16:21:07.806 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:07.806 INFO:teuthology.orchestra.run.smithi039.stdout: "req_rmdir_latency": { 2024-03-18T16:21:07.807 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-03-18T16:21:07.807 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-03-18T16:21:07.807 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-03-18T16:21:07.807 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:07.807 INFO:teuthology.orchestra.run.smithi039.stdout: "req_rmsnap_latency": { 2024-03-18T16:21:07.807 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-03-18T16:21:07.807 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-03-18T16:21:07.807 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-03-18T16:21:07.807 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:07.807 INFO:teuthology.orchestra.run.smithi039.stdout: "req_rmxattr_latency": { 2024-03-18T16:21:07.807 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-03-18T16:21:07.807 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-03-18T16:21:07.807 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-03-18T16:21:07.808 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:07.808 INFO:teuthology.orchestra.run.smithi039.stdout: "req_setattr_latency": { 2024-03-18T16:21:07.808 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-03-18T16:21:07.808 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-03-18T16:21:07.808 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-03-18T16:21:07.808 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:07.808 INFO:teuthology.orchestra.run.smithi039.stdout: "req_setdirlayout_latency": { 2024-03-18T16:21:07.808 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-03-18T16:21:07.808 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-03-18T16:21:07.808 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-03-18T16:21:07.808 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:07.808 INFO:teuthology.orchestra.run.smithi039.stdout: "req_setfilelock_latency": { 2024-03-18T16:21:07.808 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-03-18T16:21:07.809 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-03-18T16:21:07.809 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-03-18T16:21:07.809 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:07.809 INFO:teuthology.orchestra.run.smithi039.stdout: "req_setlayout_latency": { 2024-03-18T16:21:07.809 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-03-18T16:21:07.809 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-03-18T16:21:07.809 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-03-18T16:21:07.809 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:07.809 INFO:teuthology.orchestra.run.smithi039.stdout: "req_setxattr_latency": { 2024-03-18T16:21:07.809 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-03-18T16:21:07.809 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-03-18T16:21:07.809 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-03-18T16:21:07.809 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:07.810 INFO:teuthology.orchestra.run.smithi039.stdout: "req_symlink_latency": { 2024-03-18T16:21:07.810 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-03-18T16:21:07.810 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-03-18T16:21:07.810 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-03-18T16:21:07.810 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:07.810 INFO:teuthology.orchestra.run.smithi039.stdout: "req_unlink_latency": { 2024-03-18T16:21:07.810 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-03-18T16:21:07.810 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-03-18T16:21:07.810 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-03-18T16:21:07.810 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:07.810 INFO:teuthology.orchestra.run.smithi039.stdout: "cap_revoke_eviction": 0, 2024-03-18T16:21:07.810 INFO:teuthology.orchestra.run.smithi039.stdout: "cap_acquisition_throttle": 0, 2024-03-18T16:21:07.810 INFO:teuthology.orchestra.run.smithi039.stdout: "req_getvxattr_latency": { 2024-03-18T16:21:07.810 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-03-18T16:21:07.811 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-03-18T16:21:07.811 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-03-18T16:21:07.811 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:21:07.811 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:07.811 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_sessions": { 2024-03-18T16:21:07.811 INFO:teuthology.orchestra.run.smithi039.stdout: "session_count": 1, 2024-03-18T16:21:07.811 INFO:teuthology.orchestra.run.smithi039.stdout: "session_add": 1, 2024-03-18T16:21:07.811 INFO:teuthology.orchestra.run.smithi039.stdout: "session_remove": 0, 2024-03-18T16:21:07.811 INFO:teuthology.orchestra.run.smithi039.stdout: "sessions_open": 1, 2024-03-18T16:21:07.811 INFO:teuthology.orchestra.run.smithi039.stdout: "sessions_stale": 0, 2024-03-18T16:21:07.811 INFO:teuthology.orchestra.run.smithi039.stdout: "total_load": 0, 2024-03-18T16:21:07.811 INFO:teuthology.orchestra.run.smithi039.stdout: "average_load": 0, 2024-03-18T16:21:07.811 INFO:teuthology.orchestra.run.smithi039.stdout: "avg_session_uptime": 2050, 2024-03-18T16:21:07.812 INFO:teuthology.orchestra.run.smithi039.stdout: "mdthresh_evicted": 0 2024-03-18T16:21:07.812 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:07.812 INFO:teuthology.orchestra.run.smithi039.stdout: "mempool": { 2024-03-18T16:21:07.812 INFO:teuthology.orchestra.run.smithi039.stdout: "bloom_filter_bytes": 0, 2024-03-18T16:21:07.812 INFO:teuthology.orchestra.run.smithi039.stdout: "bloom_filter_items": 0, 2024-03-18T16:21:07.812 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T16:21:07.812 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_alloc_items": 0, 2024-03-18T16:21:07.812 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T16:21:07.812 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_data_items": 0, 2024-03-18T16:21:07.812 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T16:21:07.812 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T16:21:07.812 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T16:21:07.812 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T16:21:07.812 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T16:21:07.812 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_other_items": 0, 2024-03-18T16:21:07.813 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T16:21:07.813 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T16:21:07.813 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_extent_bytes": 0, 2024-03-18T16:21:07.813 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_extent_items": 0, 2024-03-18T16:21:07.813 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_blob_bytes": 0, 2024-03-18T16:21:07.813 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_blob_items": 0, 2024-03-18T16:21:07.813 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T16:21:07.813 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T16:21:07.813 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T16:21:07.813 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T16:21:07.813 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T16:21:07.813 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_fsck_items": 0, 2024-03-18T16:21:07.813 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_txc_bytes": 0, 2024-03-18T16:21:07.813 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_txc_items": 0, 2024-03-18T16:21:07.814 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T16:21:07.814 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T16:21:07.814 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_writing_bytes": 0, 2024-03-18T16:21:07.814 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_writing_items": 0, 2024-03-18T16:21:07.814 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_bytes": 0, 2024-03-18T16:21:07.814 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_items": 0, 2024-03-18T16:21:07.814 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T16:21:07.814 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_file_reader_items": 0, 2024-03-18T16:21:07.814 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T16:21:07.814 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_file_writer_items": 0, 2024-03-18T16:21:07.814 INFO:teuthology.orchestra.run.smithi039.stdout: "buffer_anon_bytes": 198531, 2024-03-18T16:21:07.814 INFO:teuthology.orchestra.run.smithi039.stdout: "buffer_anon_items": 56, 2024-03-18T16:21:07.814 INFO:teuthology.orchestra.run.smithi039.stdout: "buffer_meta_bytes": 0, 2024-03-18T16:21:07.815 INFO:teuthology.orchestra.run.smithi039.stdout: "buffer_meta_items": 0, 2024-03-18T16:21:07.815 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_bytes": 0, 2024-03-18T16:21:07.815 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_items": 0, 2024-03-18T16:21:07.815 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_mapbl_bytes": 0, 2024-03-18T16:21:07.815 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_mapbl_items": 0, 2024-03-18T16:21:07.815 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_pglog_bytes": 0, 2024-03-18T16:21:07.815 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_pglog_items": 0, 2024-03-18T16:21:07.815 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_bytes": 8888, 2024-03-18T16:21:07.815 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_items": 170, 2024-03-18T16:21:07.815 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T16:21:07.815 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_mapping_items": 0, 2024-03-18T16:21:07.815 INFO:teuthology.orchestra.run.smithi039.stdout: "pgmap_bytes": 0, 2024-03-18T16:21:07.815 INFO:teuthology.orchestra.run.smithi039.stdout: "pgmap_items": 0, 2024-03-18T16:21:07.815 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_co_bytes": 341893, 2024-03-18T16:21:07.815 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_co_items": 2027, 2024-03-18T16:21:07.816 INFO:teuthology.orchestra.run.smithi039.stdout: "unittest_1_bytes": 0, 2024-03-18T16:21:07.816 INFO:teuthology.orchestra.run.smithi039.stdout: "unittest_1_items": 0, 2024-03-18T16:21:07.816 INFO:teuthology.orchestra.run.smithi039.stdout: "unittest_2_bytes": 0, 2024-03-18T16:21:07.816 INFO:teuthology.orchestra.run.smithi039.stdout: "unittest_2_items": 0 2024-03-18T16:21:07.816 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:07.816 INFO:teuthology.orchestra.run.smithi039.stdout: "objecter": { 2024-03-18T16:21:07.816 INFO:teuthology.orchestra.run.smithi039.stdout: "op_active": 0, 2024-03-18T16:21:07.816 INFO:teuthology.orchestra.run.smithi039.stdout: "op_laggy": 0, 2024-03-18T16:21:07.816 INFO:teuthology.orchestra.run.smithi039.stdout: "op_send": 2060, 2024-03-18T16:21:07.816 INFO:teuthology.orchestra.run.smithi039.stdout: "op_send_bytes": 21, 2024-03-18T16:21:07.816 INFO:teuthology.orchestra.run.smithi039.stdout: "op_resend": 3, 2024-03-18T16:21:07.816 INFO:teuthology.orchestra.run.smithi039.stdout: "op_reply": 2060, 2024-03-18T16:21:07.816 INFO:teuthology.orchestra.run.smithi039.stdout: "op_latency": { 2024-03-18T16:21:07.816 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 2060, 2024-03-18T16:21:07.817 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 128.270472523, 2024-03-18T16:21:07.817 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.062267219 2024-03-18T16:21:07.817 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:07.817 INFO:teuthology.orchestra.run.smithi039.stdout: "op_inflight": 0, 2024-03-18T16:21:07.817 INFO:teuthology.orchestra.run.smithi039.stdout: "oplen_avg": { 2024-03-18T16:21:07.817 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 2060, 2024-03-18T16:21:07.817 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 2061 2024-03-18T16:21:07.817 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:07.817 INFO:teuthology.orchestra.run.smithi039.stdout: "op": 2060, 2024-03-18T16:21:07.817 INFO:teuthology.orchestra.run.smithi039.stdout: "op_r": 2060, 2024-03-18T16:21:07.817 INFO:teuthology.orchestra.run.smithi039.stdout: "op_w": 0, 2024-03-18T16:21:07.817 INFO:teuthology.orchestra.run.smithi039.stdout: "op_rmw": 0, 2024-03-18T16:21:07.817 INFO:teuthology.orchestra.run.smithi039.stdout: "op_pg": 0, 2024-03-18T16:21:07.817 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_stat": 1233, 2024-03-18T16:21:07.818 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_create": 0, 2024-03-18T16:21:07.818 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_read": 825, 2024-03-18T16:21:07.818 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_write": 0, 2024-03-18T16:21:07.818 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_writefull": 0, 2024-03-18T16:21:07.818 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_writesame": 0, 2024-03-18T16:21:07.818 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_append": 0, 2024-03-18T16:21:07.818 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_zero": 0, 2024-03-18T16:21:07.818 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_truncate": 0, 2024-03-18T16:21:07.818 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_delete": 0, 2024-03-18T16:21:07.818 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_mapext": 0, 2024-03-18T16:21:07.818 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_sparse_read": 0, 2024-03-18T16:21:07.818 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_clonerange": 0, 2024-03-18T16:21:07.818 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_getxattr": 1, 2024-03-18T16:21:07.818 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_setxattr": 0, 2024-03-18T16:21:07.819 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_cmpxattr": 0, 2024-03-18T16:21:07.819 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_rmxattr": 0, 2024-03-18T16:21:07.819 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_resetxattrs": 0, 2024-03-18T16:21:07.819 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_call": 0, 2024-03-18T16:21:07.819 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_watch": 0, 2024-03-18T16:21:07.819 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_notify": 0, 2024-03-18T16:21:07.819 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_src_cmpxattr": 0, 2024-03-18T16:21:07.819 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_pgls": 0, 2024-03-18T16:21:07.819 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_pgls_filter": 0, 2024-03-18T16:21:07.819 INFO:teuthology.orchestra.run.smithi039.stdout: "osdop_other": 0, 2024-03-18T16:21:07.819 INFO:teuthology.orchestra.run.smithi039.stdout: "linger_active": 0, 2024-03-18T16:21:07.819 INFO:teuthology.orchestra.run.smithi039.stdout: "linger_send": 0, 2024-03-18T16:21:07.819 INFO:teuthology.orchestra.run.smithi039.stdout: "linger_resend": 0, 2024-03-18T16:21:07.819 INFO:teuthology.orchestra.run.smithi039.stdout: "linger_ping": 0, 2024-03-18T16:21:07.819 INFO:teuthology.orchestra.run.smithi039.stdout: "poolop_active": 0, 2024-03-18T16:21:07.819 INFO:teuthology.orchestra.run.smithi039.stdout: "poolop_send": 0, 2024-03-18T16:21:07.820 INFO:teuthology.orchestra.run.smithi039.stdout: "poolop_resend": 0, 2024-03-18T16:21:07.820 INFO:teuthology.orchestra.run.smithi039.stdout: "poolstat_active": 0, 2024-03-18T16:21:07.820 INFO:teuthology.orchestra.run.smithi039.stdout: "poolstat_send": 0, 2024-03-18T16:21:07.820 INFO:teuthology.orchestra.run.smithi039.stdout: "poolstat_resend": 0, 2024-03-18T16:21:07.820 INFO:teuthology.orchestra.run.smithi039.stdout: "statfs_active": 0, 2024-03-18T16:21:07.820 INFO:teuthology.orchestra.run.smithi039.stdout: "statfs_send": 0, 2024-03-18T16:21:07.820 INFO:teuthology.orchestra.run.smithi039.stdout: "statfs_resend": 0, 2024-03-18T16:21:07.820 INFO:teuthology.orchestra.run.smithi039.stdout: "command_active": 0, 2024-03-18T16:21:07.820 INFO:teuthology.orchestra.run.smithi039.stdout: "command_send": 0, 2024-03-18T16:21:07.820 INFO:teuthology.orchestra.run.smithi039.stdout: "command_resend": 0, 2024-03-18T16:21:07.820 INFO:teuthology.orchestra.run.smithi039.stdout: "map_epoch": 93, 2024-03-18T16:21:07.820 INFO:teuthology.orchestra.run.smithi039.stdout: "map_full": 0, 2024-03-18T16:21:07.821 INFO:teuthology.orchestra.run.smithi039.stdout: "map_inc": 4, 2024-03-18T16:21:07.821 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_sessions": 6, 2024-03-18T16:21:07.821 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_session_open": 6, 2024-03-18T16:21:07.821 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_session_close": 0, 2024-03-18T16:21:07.821 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_laggy": 0, 2024-03-18T16:21:07.821 INFO:teuthology.orchestra.run.smithi039.stdout: "omap_wr": 0, 2024-03-18T16:21:07.821 INFO:teuthology.orchestra.run.smithi039.stdout: "omap_rd": 2, 2024-03-18T16:21:07.821 INFO:teuthology.orchestra.run.smithi039.stdout: "omap_del": 0 2024-03-18T16:21:07.821 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:07.821 INFO:teuthology.orchestra.run.smithi039.stdout: "oft": { 2024-03-18T16:21:07.821 INFO:teuthology.orchestra.run.smithi039.stdout: "omap_total_objs": 0, 2024-03-18T16:21:07.821 INFO:teuthology.orchestra.run.smithi039.stdout: "omap_total_kv_pairs": 0, 2024-03-18T16:21:07.821 INFO:teuthology.orchestra.run.smithi039.stdout: "omap_total_updates": 0, 2024-03-18T16:21:07.822 INFO:teuthology.orchestra.run.smithi039.stdout: "omap_total_removes": 0 2024-03-18T16:21:07.822 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:07.822 INFO:teuthology.orchestra.run.smithi039.stdout: "osd-slow-ops": { 2024-03-18T16:21:07.822 INFO:teuthology.orchestra.run.smithi039.stdout: "slow_ops_count": 0 2024-03-18T16:21:07.822 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:07.822 INFO:teuthology.orchestra.run.smithi039.stdout: "purge_queue": { 2024-03-18T16:21:07.822 INFO:teuthology.orchestra.run.smithi039.stdout: "pq_executing_ops": 0, 2024-03-18T16:21:07.822 INFO:teuthology.orchestra.run.smithi039.stdout: "pq_executing_ops_high_water": 0, 2024-03-18T16:21:07.822 INFO:teuthology.orchestra.run.smithi039.stdout: "pq_executing": 0, 2024-03-18T16:21:07.822 INFO:teuthology.orchestra.run.smithi039.stdout: "pq_executing_high_water": 0, 2024-03-18T16:21:07.822 INFO:teuthology.orchestra.run.smithi039.stdout: "pq_executed_ops": 0, 2024-03-18T16:21:07.822 INFO:teuthology.orchestra.run.smithi039.stdout: "pq_executed": 0, 2024-03-18T16:21:07.822 INFO:teuthology.orchestra.run.smithi039.stdout: "pq_item_in_journal": 0 2024-03-18T16:21:07.822 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:07.823 INFO:teuthology.orchestra.run.smithi039.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T16:21:07.823 INFO:teuthology.orchestra.run.smithi039.stdout: "val": 0, 2024-03-18T16:21:07.823 INFO:teuthology.orchestra.run.smithi039.stdout: "max": 104857600, 2024-03-18T16:21:07.823 INFO:teuthology.orchestra.run.smithi039.stdout: "get_started": 0, 2024-03-18T16:21:07.823 INFO:teuthology.orchestra.run.smithi039.stdout: "get": 2328, 2024-03-18T16:21:07.823 INFO:teuthology.orchestra.run.smithi039.stdout: "get_sum": 3451776, 2024-03-18T16:21:07.823 INFO:teuthology.orchestra.run.smithi039.stdout: "get_or_fail_fail": 0, 2024-03-18T16:21:07.823 INFO:teuthology.orchestra.run.smithi039.stdout: "get_or_fail_success": 2328, 2024-03-18T16:21:07.823 INFO:teuthology.orchestra.run.smithi039.stdout: "take": 0, 2024-03-18T16:21:07.823 INFO:teuthology.orchestra.run.smithi039.stdout: "take_sum": 0, 2024-03-18T16:21:07.823 INFO:teuthology.orchestra.run.smithi039.stdout: "put": 2328, 2024-03-18T16:21:07.823 INFO:teuthology.orchestra.run.smithi039.stdout: "put_sum": 3451776, 2024-03-18T16:21:07.823 INFO:teuthology.orchestra.run.smithi039.stdout: "wait": { 2024-03-18T16:21:07.823 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-03-18T16:21:07.823 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-03-18T16:21:07.824 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-03-18T16:21:07.824 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:21:07.824 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:07.824 INFO:teuthology.orchestra.run.smithi039.stdout: "throttle-objecter_bytes": { 2024-03-18T16:21:07.824 INFO:teuthology.orchestra.run.smithi039.stdout: "val": 0, 2024-03-18T16:21:07.824 INFO:teuthology.orchestra.run.smithi039.stdout: "max": 104857600, 2024-03-18T16:21:07.824 INFO:teuthology.orchestra.run.smithi039.stdout: "get_started": 0, 2024-03-18T16:21:07.824 INFO:teuthology.orchestra.run.smithi039.stdout: "get": 0, 2024-03-18T16:21:07.824 INFO:teuthology.orchestra.run.smithi039.stdout: "get_sum": 0, 2024-03-18T16:21:07.824 INFO:teuthology.orchestra.run.smithi039.stdout: "get_or_fail_fail": 0, 2024-03-18T16:21:07.824 INFO:teuthology.orchestra.run.smithi039.stdout: "get_or_fail_success": 0, 2024-03-18T16:21:07.824 INFO:teuthology.orchestra.run.smithi039.stdout: "take": 2060, 2024-03-18T16:21:07.824 INFO:teuthology.orchestra.run.smithi039.stdout: "take_sum": 2451583, 2024-03-18T16:21:07.824 INFO:teuthology.orchestra.run.smithi039.stdout: "put": 2060, 2024-03-18T16:21:07.825 INFO:teuthology.orchestra.run.smithi039.stdout: "put_sum": 2451583, 2024-03-18T16:21:07.825 INFO:teuthology.orchestra.run.smithi039.stdout: "wait": { 2024-03-18T16:21:07.825 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-03-18T16:21:07.825 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-03-18T16:21:07.825 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-03-18T16:21:07.825 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:21:07.825 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:07.825 INFO:teuthology.orchestra.run.smithi039.stdout: "throttle-objecter_ops": { 2024-03-18T16:21:07.825 INFO:teuthology.orchestra.run.smithi039.stdout: "val": 0, 2024-03-18T16:21:07.825 INFO:teuthology.orchestra.run.smithi039.stdout: "max": 1024, 2024-03-18T16:21:07.825 INFO:teuthology.orchestra.run.smithi039.stdout: "get_started": 0, 2024-03-18T16:21:07.825 INFO:teuthology.orchestra.run.smithi039.stdout: "get": 0, 2024-03-18T16:21:07.825 INFO:teuthology.orchestra.run.smithi039.stdout: "get_sum": 0, 2024-03-18T16:21:07.825 INFO:teuthology.orchestra.run.smithi039.stdout: "get_or_fail_fail": 0, 2024-03-18T16:21:07.825 INFO:teuthology.orchestra.run.smithi039.stdout: "get_or_fail_success": 0, 2024-03-18T16:21:07.826 INFO:teuthology.orchestra.run.smithi039.stdout: "take": 2060, 2024-03-18T16:21:07.826 INFO:teuthology.orchestra.run.smithi039.stdout: "take_sum": 2060, 2024-03-18T16:21:07.826 INFO:teuthology.orchestra.run.smithi039.stdout: "put": 2060, 2024-03-18T16:21:07.826 INFO:teuthology.orchestra.run.smithi039.stdout: "put_sum": 2060, 2024-03-18T16:21:07.826 INFO:teuthology.orchestra.run.smithi039.stdout: "wait": { 2024-03-18T16:21:07.826 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-03-18T16:21:07.826 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-03-18T16:21:07.826 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-03-18T16:21:07.826 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:21:07.826 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:07.826 INFO:teuthology.orchestra.run.smithi039.stdout: "throttle-write_buf_throttle": { 2024-03-18T16:21:07.826 INFO:teuthology.orchestra.run.smithi039.stdout: "val": 0, 2024-03-18T16:21:07.826 INFO:teuthology.orchestra.run.smithi039.stdout: "max": 3758096384, 2024-03-18T16:21:07.827 INFO:teuthology.orchestra.run.smithi039.stdout: "get_started": 0, 2024-03-18T16:21:07.827 INFO:teuthology.orchestra.run.smithi039.stdout: "get": 0, 2024-03-18T16:21:07.827 INFO:teuthology.orchestra.run.smithi039.stdout: "get_sum": 0, 2024-03-18T16:21:07.827 INFO:teuthology.orchestra.run.smithi039.stdout: "get_or_fail_fail": 0, 2024-03-18T16:21:07.827 INFO:teuthology.orchestra.run.smithi039.stdout: "get_or_fail_success": 0, 2024-03-18T16:21:07.827 INFO:teuthology.orchestra.run.smithi039.stdout: "take": 0, 2024-03-18T16:21:07.827 INFO:teuthology.orchestra.run.smithi039.stdout: "take_sum": 0, 2024-03-18T16:21:07.827 INFO:teuthology.orchestra.run.smithi039.stdout: "put": 0, 2024-03-18T16:21:07.827 INFO:teuthology.orchestra.run.smithi039.stdout: "put_sum": 0, 2024-03-18T16:21:07.827 INFO:teuthology.orchestra.run.smithi039.stdout: "wait": { 2024-03-18T16:21:07.827 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-03-18T16:21:07.827 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-03-18T16:21:07.827 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-03-18T16:21:07.828 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:21:07.828 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:07.828 INFO:teuthology.orchestra.run.smithi039.stdout: "throttle-write_buf_throttle-0x563cd3720780": { 2024-03-18T16:21:07.828 INFO:teuthology.orchestra.run.smithi039.stdout: "val": 0, 2024-03-18T16:21:07.828 INFO:teuthology.orchestra.run.smithi039.stdout: "max": 3758096384, 2024-03-18T16:21:07.828 INFO:teuthology.orchestra.run.smithi039.stdout: "get_started": 0, 2024-03-18T16:21:07.828 INFO:teuthology.orchestra.run.smithi039.stdout: "get": 0, 2024-03-18T16:21:07.828 INFO:teuthology.orchestra.run.smithi039.stdout: "get_sum": 0, 2024-03-18T16:21:07.828 INFO:teuthology.orchestra.run.smithi039.stdout: "get_or_fail_fail": 0, 2024-03-18T16:21:07.828 INFO:teuthology.orchestra.run.smithi039.stdout: "get_or_fail_success": 0, 2024-03-18T16:21:07.828 INFO:teuthology.orchestra.run.smithi039.stdout: "take": 0, 2024-03-18T16:21:07.828 INFO:teuthology.orchestra.run.smithi039.stdout: "take_sum": 0, 2024-03-18T16:21:07.828 INFO:teuthology.orchestra.run.smithi039.stdout: "put": 0, 2024-03-18T16:21:07.828 INFO:teuthology.orchestra.run.smithi039.stdout: "put_sum": 0, 2024-03-18T16:21:07.828 INFO:teuthology.orchestra.run.smithi039.stdout: "wait": { 2024-03-18T16:21:07.829 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-03-18T16:21:07.829 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-03-18T16:21:07.829 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-03-18T16:21:07.829 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:21:07.829 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:21:07.829 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-03-18T16:21:08.775 INFO:tasks.check_counter:Daemon mds.g mds.exported=0 2024-03-18T16:21:08.775 INFO:tasks.check_counter:Daemon mds.g mds.imported=0 2024-03-18T16:21:08.775 INFO:tasks.check_counter:Daemon mds.g mds.dir_split=0 2024-03-18T16:21:08.776 DEBUG:tasks.check_counter:Getting stats from j 2024-03-18T16:21:08.776 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph daemon mds.j perf dump 2024-03-18T16:21:09.257 INFO:teuthology.orchestra.run.smithi039.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.a/config 2024-03-18T16:21:09.412 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:09 smithi039 ceph-mon[31721]: pgmap v754: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 10 op/s 2024-03-18T16:21:09.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:09 smithi196 ceph-mon[36307]: pgmap v754: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 10 op/s 2024-03-18T16:21:09.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:09 smithi194 ceph-mon[36200]: pgmap v754: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 10 op/s 2024-03-18T16:21:11.219 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:11 smithi039 ceph-mon[31721]: pgmap v755: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 85 B/s wr, 10 op/s 2024-03-18T16:21:11.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:11 smithi196 ceph-mon[36307]: pgmap v755: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 85 B/s wr, 10 op/s 2024-03-18T16:21:11.773 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:11 smithi194 ceph-mon[36200]: pgmap v755: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 85 B/s wr, 10 op/s 2024-03-18T16:21:11.844 INFO:teuthology.orchestra.run.smithi039.stdout:{ 2024-03-18T16:21:11.844 INFO:teuthology.orchestra.run.smithi039.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T16:21:11.845 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_messages": 0, 2024-03-18T16:21:11.845 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_messages": 2, 2024-03-18T16:21:11.845 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_bytes": 0, 2024-03-18T16:21:11.845 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_bytes": 0, 2024-03-18T16:21:11.845 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_created_connections": 4, 2024-03-18T16:21:11.845 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_active_connections": 0, 2024-03-18T16:21:11.845 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_total_time": 0.004360245, 2024-03-18T16:21:11.846 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_send_time": 0.000000000, 2024-03-18T16:21:11.846 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_recv_time": 0.000965652, 2024-03-18T16:21:11.846 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-18T16:21:11.846 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T16:21:11.846 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-03-18T16:21:11.846 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-03-18T16:21:11.846 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-03-18T16:21:11.846 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:11.846 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_handle_ack_lat": { 2024-03-18T16:21:11.847 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-03-18T16:21:11.847 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-03-18T16:21:11.847 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-03-18T16:21:11.847 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:11.847 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-18T16:21:11.847 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_encrypted_bytes": 0 2024-03-18T16:21:11.847 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:11.847 INFO:teuthology.orchestra.run.smithi039.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T16:21:11.848 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_messages": 245, 2024-03-18T16:21:11.848 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_messages": 242, 2024-03-18T16:21:11.848 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_bytes": 484240, 2024-03-18T16:21:11.848 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_bytes": 149360, 2024-03-18T16:21:11.848 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_created_connections": 2, 2024-03-18T16:21:11.848 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_active_connections": 1, 2024-03-18T16:21:11.848 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_total_time": 0.077781472, 2024-03-18T16:21:11.848 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_send_time": 0.036151044, 2024-03-18T16:21:11.848 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_recv_time": 0.036921715, 2024-03-18T16:21:11.849 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_fast_dispatch_time": 0.008737536, 2024-03-18T16:21:11.849 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T16:21:11.849 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 241, 2024-03-18T16:21:11.849 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.027256591, 2024-03-18T16:21:11.849 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000113097 2024-03-18T16:21:11.849 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:11.849 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_handle_ack_lat": { 2024-03-18T16:21:11.849 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-03-18T16:21:11.849 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-03-18T16:21:11.849 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-03-18T16:21:11.850 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:11.850 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_encrypted_bytes": 484240, 2024-03-18T16:21:11.850 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_encrypted_bytes": 149360 2024-03-18T16:21:11.850 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:11.850 INFO:teuthology.orchestra.run.smithi039.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T16:21:11.850 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_messages": 6, 2024-03-18T16:21:11.850 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_messages": 187, 2024-03-18T16:21:11.850 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_bytes": 3312, 2024-03-18T16:21:11.850 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_bytes": 188688, 2024-03-18T16:21:11.851 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_created_connections": 4, 2024-03-18T16:21:11.851 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_active_connections": 2, 2024-03-18T16:21:11.851 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_total_time": 0.031031608, 2024-03-18T16:21:11.851 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_send_time": 0.023016630, 2024-03-18T16:21:11.851 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_recv_time": 0.005012813, 2024-03-18T16:21:11.851 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-18T16:21:11.851 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T16:21:11.851 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 186, 2024-03-18T16:21:11.851 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.026952931, 2024-03-18T16:21:11.851 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000144908 2024-03-18T16:21:11.852 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:11.852 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_handle_ack_lat": { 2024-03-18T16:21:11.852 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-03-18T16:21:11.852 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-03-18T16:21:11.852 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-03-18T16:21:11.852 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:11.852 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-03-18T16:21:11.852 INFO:teuthology.orchestra.run.smithi039.stdout: "msgr_send_encrypted_bytes": 188688 2024-03-18T16:21:11.852 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:11.852 INFO:teuthology.orchestra.run.smithi039.stdout: "cct": { 2024-03-18T16:21:11.853 INFO:teuthology.orchestra.run.smithi039.stdout: "total_workers": 0, 2024-03-18T16:21:11.853 INFO:teuthology.orchestra.run.smithi039.stdout: "unhealthy_workers": 0 2024-03-18T16:21:11.853 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:11.853 INFO:teuthology.orchestra.run.smithi039.stdout: "mempool": { 2024-03-18T16:21:11.853 INFO:teuthology.orchestra.run.smithi039.stdout: "bloom_filter_bytes": 0, 2024-03-18T16:21:11.853 INFO:teuthology.orchestra.run.smithi039.stdout: "bloom_filter_items": 0, 2024-03-18T16:21:11.853 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T16:21:11.853 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_alloc_items": 0, 2024-03-18T16:21:11.853 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T16:21:11.854 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_data_items": 0, 2024-03-18T16:21:11.854 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T16:21:11.854 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T16:21:11.854 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T16:21:11.854 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T16:21:11.854 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T16:21:11.854 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_other_items": 0, 2024-03-18T16:21:11.854 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T16:21:11.854 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T16:21:11.854 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_extent_bytes": 0, 2024-03-18T16:21:11.855 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_extent_items": 0, 2024-03-18T16:21:11.855 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_blob_bytes": 0, 2024-03-18T16:21:11.855 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_blob_items": 0, 2024-03-18T16:21:11.855 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T16:21:11.855 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T16:21:11.855 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T16:21:11.855 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T16:21:11.855 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T16:21:11.855 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_fsck_items": 0, 2024-03-18T16:21:11.856 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_txc_bytes": 0, 2024-03-18T16:21:11.856 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_txc_items": 0, 2024-03-18T16:21:11.856 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T16:21:11.856 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T16:21:11.856 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_writing_bytes": 0, 2024-03-18T16:21:11.856 INFO:teuthology.orchestra.run.smithi039.stdout: "bluestore_writing_items": 0, 2024-03-18T16:21:11.856 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_bytes": 0, 2024-03-18T16:21:11.856 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_items": 0, 2024-03-18T16:21:11.856 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T16:21:11.856 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_file_reader_items": 0, 2024-03-18T16:21:11.857 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T16:21:11.857 INFO:teuthology.orchestra.run.smithi039.stdout: "bluefs_file_writer_items": 0, 2024-03-18T16:21:11.857 INFO:teuthology.orchestra.run.smithi039.stdout: "buffer_anon_bytes": 148967, 2024-03-18T16:21:11.857 INFO:teuthology.orchestra.run.smithi039.stdout: "buffer_anon_items": 29, 2024-03-18T16:21:11.857 INFO:teuthology.orchestra.run.smithi039.stdout: "buffer_meta_bytes": 0, 2024-03-18T16:21:11.857 INFO:teuthology.orchestra.run.smithi039.stdout: "buffer_meta_items": 0, 2024-03-18T16:21:11.857 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_bytes": 0, 2024-03-18T16:21:11.857 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_items": 0, 2024-03-18T16:21:11.857 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_mapbl_bytes": 0, 2024-03-18T16:21:11.857 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_mapbl_items": 0, 2024-03-18T16:21:11.858 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_pglog_bytes": 0, 2024-03-18T16:21:11.858 INFO:teuthology.orchestra.run.smithi039.stdout: "osd_pglog_items": 0, 2024-03-18T16:21:11.858 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_bytes": 0, 2024-03-18T16:21:11.858 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_items": 0, 2024-03-18T16:21:11.858 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T16:21:11.858 INFO:teuthology.orchestra.run.smithi039.stdout: "osdmap_mapping_items": 0, 2024-03-18T16:21:11.858 INFO:teuthology.orchestra.run.smithi039.stdout: "pgmap_bytes": 0, 2024-03-18T16:21:11.858 INFO:teuthology.orchestra.run.smithi039.stdout: "pgmap_items": 0, 2024-03-18T16:21:11.858 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_co_bytes": 928, 2024-03-18T16:21:11.858 INFO:teuthology.orchestra.run.smithi039.stdout: "mds_co_items": 2, 2024-03-18T16:21:11.859 INFO:teuthology.orchestra.run.smithi039.stdout: "unittest_1_bytes": 0, 2024-03-18T16:21:11.859 INFO:teuthology.orchestra.run.smithi039.stdout: "unittest_1_items": 0, 2024-03-18T16:21:11.859 INFO:teuthology.orchestra.run.smithi039.stdout: "unittest_2_bytes": 0, 2024-03-18T16:21:11.859 INFO:teuthology.orchestra.run.smithi039.stdout: "unittest_2_items": 0 2024-03-18T16:21:11.859 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2024-03-18T16:21:11.859 INFO:teuthology.orchestra.run.smithi039.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T16:21:11.859 INFO:teuthology.orchestra.run.smithi039.stdout: "val": 0, 2024-03-18T16:21:11.859 INFO:teuthology.orchestra.run.smithi039.stdout: "max": 104857600, 2024-03-18T16:21:11.859 INFO:teuthology.orchestra.run.smithi039.stdout: "get_started": 0, 2024-03-18T16:21:11.859 INFO:teuthology.orchestra.run.smithi039.stdout: "get": 247, 2024-03-18T16:21:11.860 INFO:teuthology.orchestra.run.smithi039.stdout: "get_sum": 449639, 2024-03-18T16:21:11.860 INFO:teuthology.orchestra.run.smithi039.stdout: "get_or_fail_fail": 0, 2024-03-18T16:21:11.860 INFO:teuthology.orchestra.run.smithi039.stdout: "get_or_fail_success": 247, 2024-03-18T16:21:11.860 INFO:teuthology.orchestra.run.smithi039.stdout: "take": 0, 2024-03-18T16:21:11.860 INFO:teuthology.orchestra.run.smithi039.stdout: "take_sum": 0, 2024-03-18T16:21:11.860 INFO:teuthology.orchestra.run.smithi039.stdout: "put": 247, 2024-03-18T16:21:11.860 INFO:teuthology.orchestra.run.smithi039.stdout: "put_sum": 449639, 2024-03-18T16:21:11.860 INFO:teuthology.orchestra.run.smithi039.stdout: "wait": { 2024-03-18T16:21:11.860 INFO:teuthology.orchestra.run.smithi039.stdout: "avgcount": 0, 2024-03-18T16:21:11.861 INFO:teuthology.orchestra.run.smithi039.stdout: "sum": 0.000000000, 2024-03-18T16:21:11.861 INFO:teuthology.orchestra.run.smithi039.stdout: "avgtime": 0.000000000 2024-03-18T16:21:11.861 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:21:11.861 INFO:teuthology.orchestra.run.smithi039.stdout: } 2024-03-18T16:21:11.861 INFO:teuthology.orchestra.run.smithi039.stdout:} 2024-03-18T16:21:12.869 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-03-18T16:21:12.869 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-03-18T16:21:12.869 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.j 2024-03-18T16:21:12.870 DEBUG:tasks.check_counter:Getting stats from b 2024-03-18T16:21:12.870 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph daemon mds.b perf dump 2024-03-18T16:21:13.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:13 smithi194 ceph-mon[36200]: pgmap v756: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T16:21:13.252 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T16:21:13.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:13 smithi196 ceph-mon[36307]: pgmap v756: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T16:21:13.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:13 smithi039 ceph-mon[31721]: pgmap v756: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T16:21:15.087 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-03-18T16:21:15.088 INFO:teuthology.orchestra.run.smithi194.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T16:21:15.088 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_messages": 3351, 2024-03-18T16:21:15.088 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_messages": 3247, 2024-03-18T16:21:15.088 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_bytes": 1729225, 2024-03-18T16:21:15.088 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_bytes": 5339180, 2024-03-18T16:21:15.088 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_created_connections": 10, 2024-03-18T16:21:15.088 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_active_connections": 5, 2024-03-18T16:21:15.088 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_total_time": 0.259685095, 2024-03-18T16:21:15.088 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_send_time": 0.079628305, 2024-03-18T16:21:15.088 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_recv_time": 0.105897898, 2024-03-18T16:21:15.088 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_fast_dispatch_time": 0.064374447, 2024-03-18T16:21:15.089 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T16:21:15.089 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 3246, 2024-03-18T16:21:15.089 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.481994268, 2024-03-18T16:21:15.089 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000148488 2024-03-18T16:21:15.089 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:15.089 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_handle_ack_lat": { 2024-03-18T16:21:15.089 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 2426, 2024-03-18T16:21:15.089 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000443016, 2024-03-18T16:21:15.089 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000182 2024-03-18T16:21:15.089 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:15.089 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_encrypted_bytes": 695168, 2024-03-18T16:21:15.089 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_encrypted_bytes": 174496 2024-03-18T16:21:15.089 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:15.090 INFO:teuthology.orchestra.run.smithi194.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T16:21:15.090 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_messages": 3104, 2024-03-18T16:21:15.090 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_messages": 2800, 2024-03-18T16:21:15.090 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_bytes": 1178186, 2024-03-18T16:21:15.090 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_bytes": 1582110, 2024-03-18T16:21:15.090 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_created_connections": 10, 2024-03-18T16:21:15.090 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_active_connections": 5, 2024-03-18T16:21:15.090 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_total_time": 0.207712173, 2024-03-18T16:21:15.090 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_send_time": 0.063248002, 2024-03-18T16:21:15.090 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_recv_time": 0.098070215, 2024-03-18T16:21:15.090 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_fast_dispatch_time": 0.052041624, 2024-03-18T16:21:15.090 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T16:21:15.090 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 2803, 2024-03-18T16:21:15.090 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.363376613, 2024-03-18T16:21:15.091 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000129638 2024-03-18T16:21:15.091 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:15.091 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_handle_ack_lat": { 2024-03-18T16:21:15.091 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 3718, 2024-03-18T16:21:15.091 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000627043, 2024-03-18T16:21:15.091 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000168 2024-03-18T16:21:15.091 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:15.091 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-18T16:21:15.091 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_encrypted_bytes": 483072 2024-03-18T16:21:15.091 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:15.091 INFO:teuthology.orchestra.run.smithi194.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T16:21:15.091 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_messages": 7781, 2024-03-18T16:21:15.091 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_messages": 10058, 2024-03-18T16:21:15.091 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_bytes": 2959545, 2024-03-18T16:21:15.092 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_bytes": 12355536, 2024-03-18T16:21:15.092 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_created_connections": 157, 2024-03-18T16:21:15.092 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_active_connections": 10, 2024-03-18T16:21:15.092 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_total_time": 0.647474392, 2024-03-18T16:21:15.092 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_send_time": 0.238792212, 2024-03-18T16:21:15.092 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_recv_time": 0.391441867, 2024-03-18T16:21:15.092 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_fast_dispatch_time": 0.090117884, 2024-03-18T16:21:15.092 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T16:21:15.092 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 10051, 2024-03-18T16:21:15.092 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.487276174, 2024-03-18T16:21:15.092 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000048480 2024-03-18T16:21:15.092 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:15.092 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_handle_ack_lat": { 2024-03-18T16:21:15.092 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 12620, 2024-03-18T16:21:15.093 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.002092123, 2024-03-18T16:21:15.093 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000165 2024-03-18T16:21:15.093 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:15.093 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_encrypted_bytes": 2180288, 2024-03-18T16:21:15.093 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_encrypted_bytes": 5603104 2024-03-18T16:21:15.093 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:15.093 INFO:teuthology.orchestra.run.smithi194.stdout: "cct": { 2024-03-18T16:21:15.093 INFO:teuthology.orchestra.run.smithi194.stdout: "total_workers": 1, 2024-03-18T16:21:15.093 INFO:teuthology.orchestra.run.smithi194.stdout: "unhealthy_workers": 0 2024-03-18T16:21:15.093 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:15.093 INFO:teuthology.orchestra.run.smithi194.stdout: "finisher-MDSRank": { 2024-03-18T16:21:15.093 INFO:teuthology.orchestra.run.smithi194.stdout: "queue_len": 0, 2024-03-18T16:21:15.093 INFO:teuthology.orchestra.run.smithi194.stdout: "complete_latency": { 2024-03-18T16:21:15.093 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 2193, 2024-03-18T16:21:15.094 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 2.215542361, 2024-03-18T16:21:15.094 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.001010279 2024-03-18T16:21:15.094 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-03-18T16:21:15.094 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:15.094 INFO:teuthology.orchestra.run.smithi194.stdout: "finisher-PurgeQueue": { 2024-03-18T16:21:15.094 INFO:teuthology.orchestra.run.smithi194.stdout: "queue_len": 0, 2024-03-18T16:21:15.094 INFO:teuthology.orchestra.run.smithi194.stdout: "complete_latency": { 2024-03-18T16:21:15.094 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 1660, 2024-03-18T16:21:15.094 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.096451308, 2024-03-18T16:21:15.094 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000058103 2024-03-18T16:21:15.094 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-03-18T16:21:15.094 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:15.094 INFO:teuthology.orchestra.run.smithi194.stdout: "mds": { 2024-03-18T16:21:15.094 INFO:teuthology.orchestra.run.smithi194.stdout: "request": 1996, 2024-03-18T16:21:15.095 INFO:teuthology.orchestra.run.smithi194.stdout: "reply": 1994, 2024-03-18T16:21:15.095 INFO:teuthology.orchestra.run.smithi194.stdout: "reply_latency": { 2024-03-18T16:21:15.095 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 1994, 2024-03-18T16:21:15.095 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 112.033767210, 2024-03-18T16:21:15.095 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.056185439 2024-03-18T16:21:15.095 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:15.095 INFO:teuthology.orchestra.run.smithi194.stdout: "slow_reply": 0, 2024-03-18T16:21:15.095 INFO:teuthology.orchestra.run.smithi194.stdout: "forward": 2, 2024-03-18T16:21:15.095 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_fetch_complete": 0, 2024-03-18T16:21:15.095 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_fetch_keys": 0, 2024-03-18T16:21:15.095 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_commit": 12, 2024-03-18T16:21:15.095 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_split": 0, 2024-03-18T16:21:15.095 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_merge": 0, 2024-03-18T16:21:15.095 INFO:teuthology.orchestra.run.smithi194.stdout: "inodes": 77, 2024-03-18T16:21:15.095 INFO:teuthology.orchestra.run.smithi194.stdout: "inodes_top": 2, 2024-03-18T16:21:15.096 INFO:teuthology.orchestra.run.smithi194.stdout: "inodes_bottom": 1, 2024-03-18T16:21:15.096 INFO:teuthology.orchestra.run.smithi194.stdout: "inodes_pin_tail": 74, 2024-03-18T16:21:15.096 INFO:teuthology.orchestra.run.smithi194.stdout: "inodes_pinned": 74, 2024-03-18T16:21:15.096 INFO:teuthology.orchestra.run.smithi194.stdout: "inodes_expired": 483, 2024-03-18T16:21:15.096 INFO:teuthology.orchestra.run.smithi194.stdout: "inodes_with_caps": 0, 2024-03-18T16:21:15.096 INFO:teuthology.orchestra.run.smithi194.stdout: "caps": 0, 2024-03-18T16:21:15.096 INFO:teuthology.orchestra.run.smithi194.stdout: "subtrees": 5, 2024-03-18T16:21:15.096 INFO:teuthology.orchestra.run.smithi194.stdout: "traverse": 2727, 2024-03-18T16:21:15.096 INFO:teuthology.orchestra.run.smithi194.stdout: "traverse_hit": 2577, 2024-03-18T16:21:15.096 INFO:teuthology.orchestra.run.smithi194.stdout: "traverse_forward": 0, 2024-03-18T16:21:15.096 INFO:teuthology.orchestra.run.smithi194.stdout: "traverse_discover": 1, 2024-03-18T16:21:15.097 INFO:teuthology.orchestra.run.smithi194.stdout: "traverse_dir_fetch": 0, 2024-03-18T16:21:15.097 INFO:teuthology.orchestra.run.smithi194.stdout: "traverse_remote_ino": 0, 2024-03-18T16:21:15.097 INFO:teuthology.orchestra.run.smithi194.stdout: "traverse_lock": 50, 2024-03-18T16:21:15.097 INFO:teuthology.orchestra.run.smithi194.stdout: "load_cent": 0, 2024-03-18T16:21:15.097 INFO:teuthology.orchestra.run.smithi194.stdout: "q": 0, 2024-03-18T16:21:15.097 INFO:teuthology.orchestra.run.smithi194.stdout: "exported": 7, 2024-03-18T16:21:15.097 INFO:teuthology.orchestra.run.smithi194.stdout: "exported_inodes": 452, 2024-03-18T16:21:15.097 INFO:teuthology.orchestra.run.smithi194.stdout: "imported": 1, 2024-03-18T16:21:15.097 INFO:teuthology.orchestra.run.smithi194.stdout: "imported_inodes": 28, 2024-03-18T16:21:15.097 INFO:teuthology.orchestra.run.smithi194.stdout: "openino_dir_fetch": 0, 2024-03-18T16:21:15.097 INFO:teuthology.orchestra.run.smithi194.stdout: "openino_backtrace_fetch": 849, 2024-03-18T16:21:15.097 INFO:teuthology.orchestra.run.smithi194.stdout: "openino_peer_discover": 0, 2024-03-18T16:21:15.097 INFO:teuthology.orchestra.run.smithi194.stdout: "root_rfiles": 4, 2024-03-18T16:21:15.097 INFO:teuthology.orchestra.run.smithi194.stdout: "root_rbytes": 234, 2024-03-18T16:21:15.097 INFO:teuthology.orchestra.run.smithi194.stdout: "root_rsnaps": 0, 2024-03-18T16:21:15.097 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_backtrace_fetch": 849, 2024-03-18T16:21:15.098 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_set_tag": 0, 2024-03-18T16:21:15.098 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_backtrace_repaired": 0, 2024-03-18T16:21:15.098 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_inotable_repaired": 0, 2024-03-18T16:21:15.098 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_dir_inodes": 156, 2024-03-18T16:21:15.098 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_dir_base_inodes": 17, 2024-03-18T16:21:15.098 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_dirfrag_rstats": 139, 2024-03-18T16:21:15.098 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_file_inodes": 693, 2024-03-18T16:21:15.098 INFO:teuthology.orchestra.run.smithi194.stdout: "handle_inode_file_caps": 9, 2024-03-18T16:21:15.098 INFO:teuthology.orchestra.run.smithi194.stdout: "ceph_cap_op_revoke": 280, 2024-03-18T16:21:15.098 INFO:teuthology.orchestra.run.smithi194.stdout: "ceph_cap_op_grant": 1939, 2024-03-18T16:21:15.098 INFO:teuthology.orchestra.run.smithi194.stdout: "ceph_cap_op_trunc": 6, 2024-03-18T16:21:15.098 INFO:teuthology.orchestra.run.smithi194.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-18T16:21:15.098 INFO:teuthology.orchestra.run.smithi194.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-18T16:21:15.098 INFO:teuthology.orchestra.run.smithi194.stdout: "handle_client_caps": 2471, 2024-03-18T16:21:15.098 INFO:teuthology.orchestra.run.smithi194.stdout: "handle_client_caps_dirty": 1707, 2024-03-18T16:21:15.099 INFO:teuthology.orchestra.run.smithi194.stdout: "handle_client_cap_release": 110, 2024-03-18T16:21:15.099 INFO:teuthology.orchestra.run.smithi194.stdout: "process_request_cap_release": 972 2024-03-18T16:21:15.099 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:15.099 INFO:teuthology.orchestra.run.smithi194.stdout: "mds_cache": { 2024-03-18T16:21:15.099 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_update": 4, 2024-03-18T16:21:15.099 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_update_receipt": 0, 2024-03-18T16:21:15.099 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_try_discover": 0, 2024-03-18T16:21:15.099 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_send_discover": 1, 2024-03-18T16:21:15.099 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_handle_discover": 32, 2024-03-18T16:21:15.099 INFO:teuthology.orchestra.run.smithi194.stdout: "num_strays": 0, 2024-03-18T16:21:15.099 INFO:teuthology.orchestra.run.smithi194.stdout: "num_strays_delayed": 0, 2024-03-18T16:21:15.099 INFO:teuthology.orchestra.run.smithi194.stdout: "num_strays_enqueuing": 0, 2024-03-18T16:21:15.099 INFO:teuthology.orchestra.run.smithi194.stdout: "strays_created": 558, 2024-03-18T16:21:15.099 INFO:teuthology.orchestra.run.smithi194.stdout: "strays_enqueued": 497, 2024-03-18T16:21:15.099 INFO:teuthology.orchestra.run.smithi194.stdout: "strays_reintegrated": 61, 2024-03-18T16:21:15.100 INFO:teuthology.orchestra.run.smithi194.stdout: "strays_migrated": 0, 2024-03-18T16:21:15.100 INFO:teuthology.orchestra.run.smithi194.stdout: "num_recovering_processing": 0, 2024-03-18T16:21:15.100 INFO:teuthology.orchestra.run.smithi194.stdout: "num_recovering_enqueued": 0, 2024-03-18T16:21:15.100 INFO:teuthology.orchestra.run.smithi194.stdout: "num_recovering_prioritized": 0, 2024-03-18T16:21:15.100 INFO:teuthology.orchestra.run.smithi194.stdout: "recovery_started": 0, 2024-03-18T16:21:15.100 INFO:teuthology.orchestra.run.smithi194.stdout: "recovery_completed": 0, 2024-03-18T16:21:15.100 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_quiesce_path": 0, 2024-03-18T16:21:15.100 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_quiesce_inode": 0, 2024-03-18T16:21:15.100 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_enqueue_scrub": 17, 2024-03-18T16:21:15.100 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_exportdir": 7, 2024-03-18T16:21:15.100 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_flush": 0, 2024-03-18T16:21:15.100 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_fragmentdir": 0, 2024-03-18T16:21:15.100 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_fragstats": 0, 2024-03-18T16:21:15.100 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_inodestats": 0 2024-03-18T16:21:15.100 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:15.101 INFO:teuthology.orchestra.run.smithi194.stdout: "mds_log": { 2024-03-18T16:21:15.101 INFO:teuthology.orchestra.run.smithi194.stdout: "evlrg": 0, 2024-03-18T16:21:15.101 INFO:teuthology.orchestra.run.smithi194.stdout: "evadd": 4235, 2024-03-18T16:21:15.101 INFO:teuthology.orchestra.run.smithi194.stdout: "evex": 0, 2024-03-18T16:21:15.101 INFO:teuthology.orchestra.run.smithi194.stdout: "evtrm": 0, 2024-03-18T16:21:15.101 INFO:teuthology.orchestra.run.smithi194.stdout: "ev": 4235, 2024-03-18T16:21:15.101 INFO:teuthology.orchestra.run.smithi194.stdout: "evexg": 0, 2024-03-18T16:21:15.101 INFO:teuthology.orchestra.run.smithi194.stdout: "evexd": 0, 2024-03-18T16:21:15.101 INFO:teuthology.orchestra.run.smithi194.stdout: "segadd": 6, 2024-03-18T16:21:15.101 INFO:teuthology.orchestra.run.smithi194.stdout: "segex": 0, 2024-03-18T16:21:15.101 INFO:teuthology.orchestra.run.smithi194.stdout: "segtrm": 0, 2024-03-18T16:21:15.101 INFO:teuthology.orchestra.run.smithi194.stdout: "seg": 6, 2024-03-18T16:21:15.101 INFO:teuthology.orchestra.run.smithi194.stdout: "segmjr": 2, 2024-03-18T16:21:15.101 INFO:teuthology.orchestra.run.smithi194.stdout: "segexg": 0, 2024-03-18T16:21:15.101 INFO:teuthology.orchestra.run.smithi194.stdout: "segexd": 0, 2024-03-18T16:21:15.102 INFO:teuthology.orchestra.run.smithi194.stdout: "expos": 4194304, 2024-03-18T16:21:15.102 INFO:teuthology.orchestra.run.smithi194.stdout: "wrpos": 12487696, 2024-03-18T16:21:15.102 INFO:teuthology.orchestra.run.smithi194.stdout: "rdpos": 0, 2024-03-18T16:21:15.102 INFO:teuthology.orchestra.run.smithi194.stdout: "jlat": { 2024-03-18T16:21:15.102 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 753, 2024-03-18T16:21:15.102 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 85.357730066, 2024-03-18T16:21:15.102 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.113356879 2024-03-18T16:21:15.102 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:15.102 INFO:teuthology.orchestra.run.smithi194.stdout: "replayed": 0 2024-03-18T16:21:15.102 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:15.102 INFO:teuthology.orchestra.run.smithi194.stdout: "mds_mem": { 2024-03-18T16:21:15.102 INFO:teuthology.orchestra.run.smithi194.stdout: "ino": 21, 2024-03-18T16:21:15.102 INFO:teuthology.orchestra.run.smithi194.stdout: "ino+": 844, 2024-03-18T16:21:15.102 INFO:teuthology.orchestra.run.smithi194.stdout: "ino-": 823, 2024-03-18T16:21:15.103 INFO:teuthology.orchestra.run.smithi194.stdout: "dir": 20, 2024-03-18T16:21:15.103 INFO:teuthology.orchestra.run.smithi194.stdout: "dir+": 122, 2024-03-18T16:21:15.103 INFO:teuthology.orchestra.run.smithi194.stdout: "dir-": 102, 2024-03-18T16:21:15.103 INFO:teuthology.orchestra.run.smithi194.stdout: "dn": 77, 2024-03-18T16:21:15.103 INFO:teuthology.orchestra.run.smithi194.stdout: "dn+": 1139, 2024-03-18T16:21:15.103 INFO:teuthology.orchestra.run.smithi194.stdout: "dn-": 1062, 2024-03-18T16:21:15.103 INFO:teuthology.orchestra.run.smithi194.stdout: "cap": 0, 2024-03-18T16:21:15.103 INFO:teuthology.orchestra.run.smithi194.stdout: "cap+": 826, 2024-03-18T16:21:15.103 INFO:teuthology.orchestra.run.smithi194.stdout: "cap-": 826, 2024-03-18T16:21:15.103 INFO:teuthology.orchestra.run.smithi194.stdout: "rss": 61508, 2024-03-18T16:21:15.103 INFO:teuthology.orchestra.run.smithi194.stdout: "heap": 231804 2024-03-18T16:21:15.103 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:15.103 INFO:teuthology.orchestra.run.smithi194.stdout: "mds_server": { 2024-03-18T16:21:15.103 INFO:teuthology.orchestra.run.smithi194.stdout: "dispatch_client_request": 2415, 2024-03-18T16:21:15.103 INFO:teuthology.orchestra.run.smithi194.stdout: "dispatch_server_request": 8, 2024-03-18T16:21:15.103 INFO:teuthology.orchestra.run.smithi194.stdout: "handle_client_request": 1996, 2024-03-18T16:21:15.104 INFO:teuthology.orchestra.run.smithi194.stdout: "handle_client_session": 25, 2024-03-18T16:21:15.104 INFO:teuthology.orchestra.run.smithi194.stdout: "handle_peer_request": 47, 2024-03-18T16:21:15.104 INFO:teuthology.orchestra.run.smithi194.stdout: "req_create_latency": { 2024-03-18T16:21:15.104 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 663, 2024-03-18T16:21:15.104 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.829765913, 2024-03-18T16:21:15.104 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.001251532 2024-03-18T16:21:15.104 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:15.104 INFO:teuthology.orchestra.run.smithi194.stdout: "req_getattr_latency": { 2024-03-18T16:21:15.104 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 130, 2024-03-18T16:21:15.104 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 94.359502246, 2024-03-18T16:21:15.104 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.725842324 2024-03-18T16:21:15.104 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:15.104 INFO:teuthology.orchestra.run.smithi194.stdout: "req_getfilelock_latency": { 2024-03-18T16:21:15.104 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:15.104 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:15.105 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:15.105 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:15.105 INFO:teuthology.orchestra.run.smithi194.stdout: "req_link_latency": { 2024-03-18T16:21:15.105 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 61, 2024-03-18T16:21:15.105 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.074683231, 2024-03-18T16:21:15.105 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.001224315 2024-03-18T16:21:15.105 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:15.105 INFO:teuthology.orchestra.run.smithi194.stdout: "req_lookup_latency": { 2024-03-18T16:21:15.105 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 139, 2024-03-18T16:21:15.105 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.173960841, 2024-03-18T16:21:15.105 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.001251516 2024-03-18T16:21:15.105 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:15.105 INFO:teuthology.orchestra.run.smithi194.stdout: "req_lookuphash_latency": { 2024-03-18T16:21:15.105 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:15.105 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:15.106 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:15.106 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:15.106 INFO:teuthology.orchestra.run.smithi194.stdout: "req_lookupino_latency": { 2024-03-18T16:21:15.106 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 1, 2024-03-18T16:21:15.106 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000197984, 2024-03-18T16:21:15.106 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000197984 2024-03-18T16:21:15.106 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:15.106 INFO:teuthology.orchestra.run.smithi194.stdout: "req_lookupname_latency": { 2024-03-18T16:21:15.106 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:15.106 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:15.106 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:15.106 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:15.106 INFO:teuthology.orchestra.run.smithi194.stdout: "req_lookupparent_latency": { 2024-03-18T16:21:15.106 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:15.107 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:15.107 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:15.107 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:15.107 INFO:teuthology.orchestra.run.smithi194.stdout: "req_lookupsnap_latency": { 2024-03-18T16:21:15.107 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:15.107 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:15.107 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:15.107 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:15.107 INFO:teuthology.orchestra.run.smithi194.stdout: "req_lssnap_latency": { 2024-03-18T16:21:15.107 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:15.108 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:15.159 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:15.159 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:15.159 INFO:teuthology.orchestra.run.smithi194.stdout: "req_mkdir_latency": { 2024-03-18T16:21:15.159 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 105, 2024-03-18T16:21:15.159 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.132265598, 2024-03-18T16:21:15.159 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.001259672 2024-03-18T16:21:15.160 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:15.160 INFO:teuthology.orchestra.run.smithi194.stdout: "req_mknod_latency": { 2024-03-18T16:21:15.160 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:15.160 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:15.160 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:15.160 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:15.160 INFO:teuthology.orchestra.run.smithi194.stdout: "req_mksnap_latency": { 2024-03-18T16:21:15.160 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:15.160 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:15.160 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:15.160 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:15.160 INFO:teuthology.orchestra.run.smithi194.stdout: "req_open_latency": { 2024-03-18T16:21:15.160 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 63, 2024-03-18T16:21:15.160 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.020792629, 2024-03-18T16:21:15.160 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000330041 2024-03-18T16:21:15.161 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:15.161 INFO:teuthology.orchestra.run.smithi194.stdout: "req_readdir_latency": { 2024-03-18T16:21:15.161 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 161, 2024-03-18T16:21:15.161 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 1.079948336, 2024-03-18T16:21:15.161 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.006707753 2024-03-18T16:21:15.161 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:15.161 INFO:teuthology.orchestra.run.smithi194.stdout: "req_rename_latency": { 2024-03-18T16:21:15.161 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 113, 2024-03-18T16:21:15.161 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 14.242140923, 2024-03-18T16:21:15.161 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.126036645 2024-03-18T16:21:15.161 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:15.161 INFO:teuthology.orchestra.run.smithi194.stdout: "req_renamesnap_latency": { 2024-03-18T16:21:15.161 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:15.161 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:15.161 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:15.162 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:15.162 INFO:teuthology.orchestra.run.smithi194.stdout: "req_snapdiff_latency": { 2024-03-18T16:21:15.162 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:15.162 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:15.162 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:15.162 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:15.162 INFO:teuthology.orchestra.run.smithi194.stdout: "req_rmdir_latency": { 2024-03-18T16:21:15.162 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 16, 2024-03-18T16:21:15.162 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.190212944, 2024-03-18T16:21:15.162 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.011888309 2024-03-18T16:21:15.162 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:15.162 INFO:teuthology.orchestra.run.smithi194.stdout: "req_rmsnap_latency": { 2024-03-18T16:21:15.162 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:15.162 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:15.162 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:15.163 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:15.163 INFO:teuthology.orchestra.run.smithi194.stdout: "req_rmxattr_latency": { 2024-03-18T16:21:15.163 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:15.163 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:15.163 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:15.163 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:15.163 INFO:teuthology.orchestra.run.smithi194.stdout: "req_setattr_latency": { 2024-03-18T16:21:15.163 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 8, 2024-03-18T16:21:15.163 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.032244192, 2024-03-18T16:21:15.163 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.004030524 2024-03-18T16:21:15.163 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:15.163 INFO:teuthology.orchestra.run.smithi194.stdout: "req_setdirlayout_latency": { 2024-03-18T16:21:15.163 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:15.163 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:15.163 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:15.164 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:15.164 INFO:teuthology.orchestra.run.smithi194.stdout: "req_setfilelock_latency": { 2024-03-18T16:21:15.164 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 4, 2024-03-18T16:21:15.164 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.001698235, 2024-03-18T16:21:15.164 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000424558 2024-03-18T16:21:15.164 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:15.164 INFO:teuthology.orchestra.run.smithi194.stdout: "req_setlayout_latency": { 2024-03-18T16:21:15.164 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:15.164 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:15.164 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:15.164 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:15.164 INFO:teuthology.orchestra.run.smithi194.stdout: "req_setxattr_latency": { 2024-03-18T16:21:15.164 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 8, 2024-03-18T16:21:15.164 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.018664632, 2024-03-18T16:21:15.164 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.002333079 2024-03-18T16:21:15.165 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:15.165 INFO:teuthology.orchestra.run.smithi194.stdout: "req_symlink_latency": { 2024-03-18T16:21:15.165 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 13, 2024-03-18T16:21:15.165 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.008703474, 2024-03-18T16:21:15.165 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000669498 2024-03-18T16:21:15.165 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:15.165 INFO:teuthology.orchestra.run.smithi194.stdout: "req_unlink_latency": { 2024-03-18T16:21:15.165 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 509, 2024-03-18T16:21:15.165 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.868986032, 2024-03-18T16:21:15.165 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.001707241 2024-03-18T16:21:15.165 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:15.165 INFO:teuthology.orchestra.run.smithi194.stdout: "cap_revoke_eviction": 0, 2024-03-18T16:21:15.165 INFO:teuthology.orchestra.run.smithi194.stdout: "cap_acquisition_throttle": 0, 2024-03-18T16:21:15.165 INFO:teuthology.orchestra.run.smithi194.stdout: "req_getvxattr_latency": { 2024-03-18T16:21:15.165 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:15.166 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:15.166 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:15.166 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-03-18T16:21:15.166 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:15.166 INFO:teuthology.orchestra.run.smithi194.stdout: "mds_sessions": { 2024-03-18T16:21:15.166 INFO:teuthology.orchestra.run.smithi194.stdout: "session_count": 1, 2024-03-18T16:21:15.166 INFO:teuthology.orchestra.run.smithi194.stdout: "session_add": 3, 2024-03-18T16:21:15.166 INFO:teuthology.orchestra.run.smithi194.stdout: "session_remove": 2, 2024-03-18T16:21:15.166 INFO:teuthology.orchestra.run.smithi194.stdout: "sessions_open": 1, 2024-03-18T16:21:15.166 INFO:teuthology.orchestra.run.smithi194.stdout: "sessions_stale": 0, 2024-03-18T16:21:15.166 INFO:teuthology.orchestra.run.smithi194.stdout: "total_load": 64, 2024-03-18T16:21:15.166 INFO:teuthology.orchestra.run.smithi194.stdout: "average_load": 64, 2024-03-18T16:21:15.166 INFO:teuthology.orchestra.run.smithi194.stdout: "avg_session_uptime": 678, 2024-03-18T16:21:15.166 INFO:teuthology.orchestra.run.smithi194.stdout: "mdthresh_evicted": 0 2024-03-18T16:21:15.166 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:15.167 INFO:teuthology.orchestra.run.smithi194.stdout: "mempool": { 2024-03-18T16:21:15.167 INFO:teuthology.orchestra.run.smithi194.stdout: "bloom_filter_bytes": 0, 2024-03-18T16:21:15.167 INFO:teuthology.orchestra.run.smithi194.stdout: "bloom_filter_items": 0, 2024-03-18T16:21:15.167 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T16:21:15.167 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_alloc_items": 0, 2024-03-18T16:21:15.167 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T16:21:15.167 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_data_items": 0, 2024-03-18T16:21:15.168 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T16:21:15.168 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T16:21:15.168 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T16:21:15.168 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T16:21:15.168 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T16:21:15.169 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_other_items": 0, 2024-03-18T16:21:15.169 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T16:21:15.169 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T16:21:15.169 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_extent_bytes": 0, 2024-03-18T16:21:15.169 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_extent_items": 0, 2024-03-18T16:21:15.169 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_blob_bytes": 0, 2024-03-18T16:21:15.169 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_blob_items": 0, 2024-03-18T16:21:15.169 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T16:21:15.169 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T16:21:15.169 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T16:21:15.169 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T16:21:15.169 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T16:21:15.169 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_fsck_items": 0, 2024-03-18T16:21:15.169 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_txc_bytes": 0, 2024-03-18T16:21:15.170 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_txc_items": 0, 2024-03-18T16:21:15.170 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T16:21:15.170 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T16:21:15.170 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_writing_bytes": 0, 2024-03-18T16:21:15.170 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_writing_items": 0, 2024-03-18T16:21:15.170 INFO:teuthology.orchestra.run.smithi194.stdout: "bluefs_bytes": 0, 2024-03-18T16:21:15.170 INFO:teuthology.orchestra.run.smithi194.stdout: "bluefs_items": 0, 2024-03-18T16:21:15.170 INFO:teuthology.orchestra.run.smithi194.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T16:21:15.170 INFO:teuthology.orchestra.run.smithi194.stdout: "bluefs_file_reader_items": 0, 2024-03-18T16:21:15.170 INFO:teuthology.orchestra.run.smithi194.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T16:21:15.170 INFO:teuthology.orchestra.run.smithi194.stdout: "bluefs_file_writer_items": 0, 2024-03-18T16:21:15.171 INFO:teuthology.orchestra.run.smithi194.stdout: "buffer_anon_bytes": 281671, 2024-03-18T16:21:15.171 INFO:teuthology.orchestra.run.smithi194.stdout: "buffer_anon_items": 112, 2024-03-18T16:21:15.171 INFO:teuthology.orchestra.run.smithi194.stdout: "buffer_meta_bytes": 176, 2024-03-18T16:21:15.171 INFO:teuthology.orchestra.run.smithi194.stdout: "buffer_meta_items": 2, 2024-03-18T16:21:15.171 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_bytes": 0, 2024-03-18T16:21:15.171 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_items": 0, 2024-03-18T16:21:15.171 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_mapbl_bytes": 0, 2024-03-18T16:21:15.171 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_mapbl_items": 0, 2024-03-18T16:21:15.171 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_pglog_bytes": 0, 2024-03-18T16:21:15.171 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_pglog_items": 0, 2024-03-18T16:21:15.171 INFO:teuthology.orchestra.run.smithi194.stdout: "osdmap_bytes": 8888, 2024-03-18T16:21:15.171 INFO:teuthology.orchestra.run.smithi194.stdout: "osdmap_items": 170, 2024-03-18T16:21:15.171 INFO:teuthology.orchestra.run.smithi194.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T16:21:15.172 INFO:teuthology.orchestra.run.smithi194.stdout: "osdmap_mapping_items": 0, 2024-03-18T16:21:15.172 INFO:teuthology.orchestra.run.smithi194.stdout: "pgmap_bytes": 0, 2024-03-18T16:21:15.172 INFO:teuthology.orchestra.run.smithi194.stdout: "pgmap_items": 0, 2024-03-18T16:21:15.172 INFO:teuthology.orchestra.run.smithi194.stdout: "mds_co_bytes": 132093, 2024-03-18T16:21:15.172 INFO:teuthology.orchestra.run.smithi194.stdout: "mds_co_items": 1133, 2024-03-18T16:21:15.172 INFO:teuthology.orchestra.run.smithi194.stdout: "unittest_1_bytes": 0, 2024-03-18T16:21:15.172 INFO:teuthology.orchestra.run.smithi194.stdout: "unittest_1_items": 0, 2024-03-18T16:21:15.172 INFO:teuthology.orchestra.run.smithi194.stdout: "unittest_2_bytes": 0, 2024-03-18T16:21:15.172 INFO:teuthology.orchestra.run.smithi194.stdout: "unittest_2_items": 0 2024-03-18T16:21:15.172 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:15.172 INFO:teuthology.orchestra.run.smithi194.stdout: "objecter": { 2024-03-18T16:21:15.172 INFO:teuthology.orchestra.run.smithi194.stdout: "op_active": 0, 2024-03-18T16:21:15.172 INFO:teuthology.orchestra.run.smithi194.stdout: "op_laggy": 0, 2024-03-18T16:21:15.172 INFO:teuthology.orchestra.run.smithi194.stdout: "op_send": 4062, 2024-03-18T16:21:15.172 INFO:teuthology.orchestra.run.smithi194.stdout: "op_send_bytes": 8460019, 2024-03-18T16:21:15.173 INFO:teuthology.orchestra.run.smithi194.stdout: "op_resend": 37, 2024-03-18T16:21:15.173 INFO:teuthology.orchestra.run.smithi194.stdout: "op_reply": 4062, 2024-03-18T16:21:15.173 INFO:teuthology.orchestra.run.smithi194.stdout: "op_latency": { 2024-03-18T16:21:15.173 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 4062, 2024-03-18T16:21:15.173 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 323.995469539, 2024-03-18T16:21:15.173 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.079762547 2024-03-18T16:21:15.173 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:15.173 INFO:teuthology.orchestra.run.smithi194.stdout: "op_inflight": 0, 2024-03-18T16:21:15.173 INFO:teuthology.orchestra.run.smithi194.stdout: "oplen_avg": { 2024-03-18T16:21:15.173 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 4062, 2024-03-18T16:21:15.173 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 4223 2024-03-18T16:21:15.173 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:15.173 INFO:teuthology.orchestra.run.smithi194.stdout: "op": 4062, 2024-03-18T16:21:15.173 INFO:teuthology.orchestra.run.smithi194.stdout: "op_r": 906, 2024-03-18T16:21:15.173 INFO:teuthology.orchestra.run.smithi194.stdout: "op_w": 3156, 2024-03-18T16:21:15.174 INFO:teuthology.orchestra.run.smithi194.stdout: "op_rmw": 0, 2024-03-18T16:21:15.174 INFO:teuthology.orchestra.run.smithi194.stdout: "op_pg": 0, 2024-03-18T16:21:15.174 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_stat": 0, 2024-03-18T16:21:15.174 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_create": 11, 2024-03-18T16:21:15.174 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_read": 40, 2024-03-18T16:21:15.174 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_write": 776, 2024-03-18T16:21:15.174 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_writefull": 58, 2024-03-18T16:21:15.174 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_writesame": 0, 2024-03-18T16:21:15.174 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_append": 0, 2024-03-18T16:21:15.174 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_zero": 0, 2024-03-18T16:21:15.174 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_truncate": 0, 2024-03-18T16:21:15.174 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_delete": 2176, 2024-03-18T16:21:15.174 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_mapext": 0, 2024-03-18T16:21:15.174 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_sparse_read": 0, 2024-03-18T16:21:15.174 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_clonerange": 0, 2024-03-18T16:21:15.175 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_getxattr": 866, 2024-03-18T16:21:15.175 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_setxattr": 22, 2024-03-18T16:21:15.175 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_cmpxattr": 0, 2024-03-18T16:21:15.175 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_rmxattr": 0, 2024-03-18T16:21:15.175 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_resetxattrs": 0, 2024-03-18T16:21:15.175 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_call": 0, 2024-03-18T16:21:15.175 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_watch": 0, 2024-03-18T16:21:15.175 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_notify": 0, 2024-03-18T16:21:15.175 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_src_cmpxattr": 0, 2024-03-18T16:21:15.175 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_pgls": 0, 2024-03-18T16:21:15.175 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_pgls_filter": 0, 2024-03-18T16:21:15.175 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_other": 6, 2024-03-18T16:21:15.175 INFO:teuthology.orchestra.run.smithi194.stdout: "linger_active": 0, 2024-03-18T16:21:15.175 INFO:teuthology.orchestra.run.smithi194.stdout: "linger_send": 0, 2024-03-18T16:21:15.175 INFO:teuthology.orchestra.run.smithi194.stdout: "linger_resend": 0, 2024-03-18T16:21:15.175 INFO:teuthology.orchestra.run.smithi194.stdout: "linger_ping": 0, 2024-03-18T16:21:15.176 INFO:teuthology.orchestra.run.smithi194.stdout: "poolop_active": 0, 2024-03-18T16:21:15.176 INFO:teuthology.orchestra.run.smithi194.stdout: "poolop_send": 0, 2024-03-18T16:21:15.176 INFO:teuthology.orchestra.run.smithi194.stdout: "poolop_resend": 0, 2024-03-18T16:21:15.176 INFO:teuthology.orchestra.run.smithi194.stdout: "poolstat_active": 0, 2024-03-18T16:21:15.176 INFO:teuthology.orchestra.run.smithi194.stdout: "poolstat_send": 0, 2024-03-18T16:21:15.176 INFO:teuthology.orchestra.run.smithi194.stdout: "poolstat_resend": 0, 2024-03-18T16:21:15.176 INFO:teuthology.orchestra.run.smithi194.stdout: "statfs_active": 0, 2024-03-18T16:21:15.176 INFO:teuthology.orchestra.run.smithi194.stdout: "statfs_send": 0, 2024-03-18T16:21:15.176 INFO:teuthology.orchestra.run.smithi194.stdout: "statfs_resend": 0, 2024-03-18T16:21:15.176 INFO:teuthology.orchestra.run.smithi194.stdout: "command_active": 0, 2024-03-18T16:21:15.176 INFO:teuthology.orchestra.run.smithi194.stdout: "command_send": 0, 2024-03-18T16:21:15.176 INFO:teuthology.orchestra.run.smithi194.stdout: "command_resend": 0, 2024-03-18T16:21:15.176 INFO:teuthology.orchestra.run.smithi194.stdout: "map_epoch": 93, 2024-03-18T16:21:15.176 INFO:teuthology.orchestra.run.smithi194.stdout: "map_full": 0, 2024-03-18T16:21:15.176 INFO:teuthology.orchestra.run.smithi194.stdout: "map_inc": 9, 2024-03-18T16:21:15.177 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_sessions": 12, 2024-03-18T16:21:15.177 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_session_open": 12, 2024-03-18T16:21:15.177 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_session_close": 0, 2024-03-18T16:21:15.177 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_laggy": 0, 2024-03-18T16:21:15.177 INFO:teuthology.orchestra.run.smithi194.stdout: "omap_wr": 192, 2024-03-18T16:21:15.177 INFO:teuthology.orchestra.run.smithi194.stdout: "omap_rd": 0, 2024-03-18T16:21:15.177 INFO:teuthology.orchestra.run.smithi194.stdout: "omap_del": 76 2024-03-18T16:21:15.177 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:15.177 INFO:teuthology.orchestra.run.smithi194.stdout: "oft": { 2024-03-18T16:21:15.177 INFO:teuthology.orchestra.run.smithi194.stdout: "omap_total_objs": 1, 2024-03-18T16:21:15.177 INFO:teuthology.orchestra.run.smithi194.stdout: "omap_total_kv_pairs": 0, 2024-03-18T16:21:15.177 INFO:teuthology.orchestra.run.smithi194.stdout: "omap_total_updates": 0, 2024-03-18T16:21:15.177 INFO:teuthology.orchestra.run.smithi194.stdout: "omap_total_removes": 0 2024-03-18T16:21:15.177 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:15.177 INFO:teuthology.orchestra.run.smithi194.stdout: "osd-slow-ops": { 2024-03-18T16:21:15.177 INFO:teuthology.orchestra.run.smithi194.stdout: "slow_ops_count": 1 2024-03-18T16:21:15.178 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:15.178 INFO:teuthology.orchestra.run.smithi194.stdout: "purge_queue": { 2024-03-18T16:21:15.178 INFO:teuthology.orchestra.run.smithi194.stdout: "pq_executing_ops": 0, 2024-03-18T16:21:15.178 INFO:teuthology.orchestra.run.smithi194.stdout: "pq_executing_ops_high_water": 52, 2024-03-18T16:21:15.178 INFO:teuthology.orchestra.run.smithi194.stdout: "pq_executing": 0, 2024-03-18T16:21:15.178 INFO:teuthology.orchestra.run.smithi194.stdout: "pq_executing_high_water": 9, 2024-03-18T16:21:15.178 INFO:teuthology.orchestra.run.smithi194.stdout: "pq_executed_ops": 1663, 2024-03-18T16:21:15.178 INFO:teuthology.orchestra.run.smithi194.stdout: "pq_executed": 497, 2024-03-18T16:21:15.178 INFO:teuthology.orchestra.run.smithi194.stdout: "pq_item_in_journal": 0 2024-03-18T16:21:15.178 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:15.178 INFO:teuthology.orchestra.run.smithi194.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T16:21:15.178 INFO:teuthology.orchestra.run.smithi194.stdout: "val": 0, 2024-03-18T16:21:15.178 INFO:teuthology.orchestra.run.smithi194.stdout: "max": 104857600, 2024-03-18T16:21:15.178 INFO:teuthology.orchestra.run.smithi194.stdout: "get_started": 0, 2024-03-18T16:21:15.179 INFO:teuthology.orchestra.run.smithi194.stdout: "get": 14232, 2024-03-18T16:21:15.179 INFO:teuthology.orchestra.run.smithi194.stdout: "get_sum": 4317392, 2024-03-18T16:21:15.179 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_fail": 0, 2024-03-18T16:21:15.179 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_success": 14232, 2024-03-18T16:21:15.179 INFO:teuthology.orchestra.run.smithi194.stdout: "take": 0, 2024-03-18T16:21:15.179 INFO:teuthology.orchestra.run.smithi194.stdout: "take_sum": 0, 2024-03-18T16:21:15.179 INFO:teuthology.orchestra.run.smithi194.stdout: "put": 14232, 2024-03-18T16:21:15.179 INFO:teuthology.orchestra.run.smithi194.stdout: "put_sum": 4317392, 2024-03-18T16:21:15.180 INFO:teuthology.orchestra.run.smithi194.stdout: "wait": { 2024-03-18T16:21:15.186 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:15.186 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:15.186 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:15.186 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-03-18T16:21:15.187 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:15.187 INFO:teuthology.orchestra.run.smithi194.stdout: "throttle-objecter_bytes": { 2024-03-18T16:21:15.187 INFO:teuthology.orchestra.run.smithi194.stdout: "val": 0, 2024-03-18T16:21:15.187 INFO:teuthology.orchestra.run.smithi194.stdout: "max": 104857600, 2024-03-18T16:21:15.187 INFO:teuthology.orchestra.run.smithi194.stdout: "get_started": 0, 2024-03-18T16:21:15.187 INFO:teuthology.orchestra.run.smithi194.stdout: "get": 0, 2024-03-18T16:21:15.187 INFO:teuthology.orchestra.run.smithi194.stdout: "get_sum": 0, 2024-03-18T16:21:15.187 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_fail": 0, 2024-03-18T16:21:15.187 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_success": 0, 2024-03-18T16:21:15.187 INFO:teuthology.orchestra.run.smithi194.stdout: "take": 4062, 2024-03-18T16:21:15.187 INFO:teuthology.orchestra.run.smithi194.stdout: "take_sum": 8513960, 2024-03-18T16:21:15.187 INFO:teuthology.orchestra.run.smithi194.stdout: "put": 4062, 2024-03-18T16:21:15.187 INFO:teuthology.orchestra.run.smithi194.stdout: "put_sum": 8513960, 2024-03-18T16:21:15.188 INFO:teuthology.orchestra.run.smithi194.stdout: "wait": { 2024-03-18T16:21:15.188 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:15.188 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:15.188 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:15.188 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-03-18T16:21:15.188 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:15.188 INFO:teuthology.orchestra.run.smithi194.stdout: "throttle-objecter_ops": { 2024-03-18T16:21:15.188 INFO:teuthology.orchestra.run.smithi194.stdout: "val": 0, 2024-03-18T16:21:15.188 INFO:teuthology.orchestra.run.smithi194.stdout: "max": 1024, 2024-03-18T16:21:15.188 INFO:teuthology.orchestra.run.smithi194.stdout: "get_started": 0, 2024-03-18T16:21:15.188 INFO:teuthology.orchestra.run.smithi194.stdout: "get": 0, 2024-03-18T16:21:15.188 INFO:teuthology.orchestra.run.smithi194.stdout: "get_sum": 0, 2024-03-18T16:21:15.188 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_fail": 0, 2024-03-18T16:21:15.189 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_success": 0, 2024-03-18T16:21:15.189 INFO:teuthology.orchestra.run.smithi194.stdout: "take": 4062, 2024-03-18T16:21:15.189 INFO:teuthology.orchestra.run.smithi194.stdout: "take_sum": 4062, 2024-03-18T16:21:15.189 INFO:teuthology.orchestra.run.smithi194.stdout: "put": 4062, 2024-03-18T16:21:15.189 INFO:teuthology.orchestra.run.smithi194.stdout: "put_sum": 4062, 2024-03-18T16:21:15.189 INFO:teuthology.orchestra.run.smithi194.stdout: "wait": { 2024-03-18T16:21:15.189 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:15.189 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:15.189 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:15.189 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-03-18T16:21:15.189 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:15.190 INFO:teuthology.orchestra.run.smithi194.stdout: "throttle-write_buf_throttle": { 2024-03-18T16:21:15.190 INFO:teuthology.orchestra.run.smithi194.stdout: "val": 0, 2024-03-18T16:21:15.190 INFO:teuthology.orchestra.run.smithi194.stdout: "max": 3758096384, 2024-03-18T16:21:15.190 INFO:teuthology.orchestra.run.smithi194.stdout: "get_started": 0, 2024-03-18T16:21:15.190 INFO:teuthology.orchestra.run.smithi194.stdout: "get": 497, 2024-03-18T16:21:15.190 INFO:teuthology.orchestra.run.smithi194.stdout: "get_sum": 53941, 2024-03-18T16:21:15.190 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_fail": 0, 2024-03-18T16:21:15.190 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_success": 497, 2024-03-18T16:21:15.190 INFO:teuthology.orchestra.run.smithi194.stdout: "take": 0, 2024-03-18T16:21:15.190 INFO:teuthology.orchestra.run.smithi194.stdout: "take_sum": 0, 2024-03-18T16:21:15.190 INFO:teuthology.orchestra.run.smithi194.stdout: "put": 23, 2024-03-18T16:21:15.190 INFO:teuthology.orchestra.run.smithi194.stdout: "put_sum": 53941, 2024-03-18T16:21:15.191 INFO:teuthology.orchestra.run.smithi194.stdout: "wait": { 2024-03-18T16:21:15.191 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:15.191 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:15.191 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:15.191 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-03-18T16:21:15.191 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:15.191 INFO:teuthology.orchestra.run.smithi194.stdout: "throttle-write_buf_throttle-0x55e7494c0500": { 2024-03-18T16:21:15.191 INFO:teuthology.orchestra.run.smithi194.stdout: "val": 0, 2024-03-18T16:21:15.191 INFO:teuthology.orchestra.run.smithi194.stdout: "max": 3758096384, 2024-03-18T16:21:15.191 INFO:teuthology.orchestra.run.smithi194.stdout: "get_started": 0, 2024-03-18T16:21:15.191 INFO:teuthology.orchestra.run.smithi194.stdout: "get": 4235, 2024-03-18T16:21:15.191 INFO:teuthology.orchestra.run.smithi194.stdout: "get_sum": 8293392, 2024-03-18T16:21:15.192 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_fail": 0, 2024-03-18T16:21:15.192 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_success": 4235, 2024-03-18T16:21:15.192 INFO:teuthology.orchestra.run.smithi194.stdout: "take": 0, 2024-03-18T16:21:15.192 INFO:teuthology.orchestra.run.smithi194.stdout: "take_sum": 0, 2024-03-18T16:21:15.192 INFO:teuthology.orchestra.run.smithi194.stdout: "put": 753, 2024-03-18T16:21:15.192 INFO:teuthology.orchestra.run.smithi194.stdout: "put_sum": 8293392, 2024-03-18T16:21:15.192 INFO:teuthology.orchestra.run.smithi194.stdout: "wait": { 2024-03-18T16:21:15.192 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:15.192 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:15.192 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:15.192 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-03-18T16:21:15.192 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-03-18T16:21:15.192 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-03-18T16:21:15.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:15 smithi194 ceph-mon[36200]: pgmap v757: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-18T16:21:15.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:15 smithi196 ceph-mon[36307]: pgmap v757: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-18T16:21:15.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:15 smithi039 ceph-mon[31721]: pgmap v757: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-18T16:21:15.913 INFO:tasks.check_counter:Daemon mds.b mds.exported=7 2024-03-18T16:21:15.913 INFO:tasks.check_counter:Daemon mds.b mds.imported=1 2024-03-18T16:21:15.913 INFO:tasks.check_counter:Daemon mds.b mds.dir_split=0 2024-03-18T16:21:15.914 DEBUG:tasks.check_counter:Getting stats from e 2024-03-18T16:21:15.914 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph daemon mds.e perf dump 2024-03-18T16:21:16.174 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T16:21:16.428 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:16 smithi194 ceph-mon[36200]: pgmap v758: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-18T16:21:16.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:16 smithi196 ceph-mon[36307]: pgmap v758: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-18T16:21:16.587 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:16 smithi039 ceph-mon[31721]: pgmap v758: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-18T16:21:18.033 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-03-18T16:21:18.033 INFO:teuthology.orchestra.run.smithi194.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T16:21:18.033 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_messages": 2444, 2024-03-18T16:21:18.033 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_messages": 2446, 2024-03-18T16:21:18.033 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_bytes": 7048352, 2024-03-18T16:21:18.033 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_bytes": 777199, 2024-03-18T16:21:18.033 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_created_connections": 6, 2024-03-18T16:21:18.033 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_active_connections": 2, 2024-03-18T16:21:18.033 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_total_time": 0.166149269, 2024-03-18T16:21:18.033 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_send_time": 0.044821786, 2024-03-18T16:21:18.033 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_recv_time": 0.046497366, 2024-03-18T16:21:18.033 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_fast_dispatch_time": 0.064381510, 2024-03-18T16:21:18.034 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T16:21:18.034 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 2444, 2024-03-18T16:21:18.034 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.061068847, 2024-03-18T16:21:18.034 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000024987 2024-03-18T16:21:18.034 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:18.034 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_handle_ack_lat": { 2024-03-18T16:21:18.034 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:18.034 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:18.034 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:18.034 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:18.035 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-18T16:21:18.035 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_encrypted_bytes": 0 2024-03-18T16:21:18.035 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:18.035 INFO:teuthology.orchestra.run.smithi194.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T16:21:18.035 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_messages": 4, 2024-03-18T16:21:18.035 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_messages": 188, 2024-03-18T16:21:18.035 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_bytes": 876, 2024-03-18T16:21:18.035 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_bytes": 557178, 2024-03-18T16:21:18.035 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_created_connections": 6, 2024-03-18T16:21:18.035 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_active_connections": 3, 2024-03-18T16:21:18.035 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_total_time": 0.010357681, 2024-03-18T16:21:18.035 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_send_time": 0.006608999, 2024-03-18T16:21:18.035 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_recv_time": 0.002333613, 2024-03-18T16:21:18.035 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_fast_dispatch_time": 0.000051340, 2024-03-18T16:21:18.036 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T16:21:18.036 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 186, 2024-03-18T16:21:18.036 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.011133187, 2024-03-18T16:21:18.036 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000059855 2024-03-18T16:21:18.036 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:18.036 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_handle_ack_lat": { 2024-03-18T16:21:18.036 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:18.036 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:18.036 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:18.036 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:18.036 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-18T16:21:18.036 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_encrypted_bytes": 556480 2024-03-18T16:21:18.036 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:18.036 INFO:teuthology.orchestra.run.smithi194.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T16:21:18.036 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_messages": 282, 2024-03-18T16:21:18.037 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_messages": 253, 2024-03-18T16:21:18.037 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_bytes": 665292, 2024-03-18T16:21:18.037 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_bytes": 151022, 2024-03-18T16:21:18.037 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_created_connections": 3, 2024-03-18T16:21:18.037 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_active_connections": 2, 2024-03-18T16:21:18.037 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_total_time": 0.031571776, 2024-03-18T16:21:18.037 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_send_time": 0.011328294, 2024-03-18T16:21:18.037 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_recv_time": 0.016312092, 2024-03-18T16:21:18.037 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_fast_dispatch_time": 0.005003303, 2024-03-18T16:21:18.037 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T16:21:18.037 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 253, 2024-03-18T16:21:18.037 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.010754459, 2024-03-18T16:21:18.037 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000042507 2024-03-18T16:21:18.037 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:18.038 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_handle_ack_lat": { 2024-03-18T16:21:18.038 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:18.038 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:18.038 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:18.038 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:18.038 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_encrypted_bytes": 665024, 2024-03-18T16:21:18.038 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_encrypted_bytes": 150704 2024-03-18T16:21:18.038 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:18.038 INFO:teuthology.orchestra.run.smithi194.stdout: "cct": { 2024-03-18T16:21:18.038 INFO:teuthology.orchestra.run.smithi194.stdout: "total_workers": 1, 2024-03-18T16:21:18.038 INFO:teuthology.orchestra.run.smithi194.stdout: "unhealthy_workers": 0 2024-03-18T16:21:18.038 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:18.038 INFO:teuthology.orchestra.run.smithi194.stdout: "finisher-MDSRank": { 2024-03-18T16:21:18.038 INFO:teuthology.orchestra.run.smithi194.stdout: "queue_len": 0, 2024-03-18T16:21:18.038 INFO:teuthology.orchestra.run.smithi194.stdout: "complete_latency": { 2024-03-18T16:21:18.039 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 4164, 2024-03-18T16:21:18.039 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.087423632, 2024-03-18T16:21:18.039 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000020995 2024-03-18T16:21:18.039 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-03-18T16:21:18.039 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:18.039 INFO:teuthology.orchestra.run.smithi194.stdout: "finisher-PurgeQueue": { 2024-03-18T16:21:18.039 INFO:teuthology.orchestra.run.smithi194.stdout: "queue_len": 0, 2024-03-18T16:21:18.039 INFO:teuthology.orchestra.run.smithi194.stdout: "complete_latency": { 2024-03-18T16:21:18.039 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:18.039 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:18.039 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:18.039 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-03-18T16:21:18.039 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:18.039 INFO:teuthology.orchestra.run.smithi194.stdout: "mds": { 2024-03-18T16:21:18.039 INFO:teuthology.orchestra.run.smithi194.stdout: "request": 0, 2024-03-18T16:21:18.040 INFO:teuthology.orchestra.run.smithi194.stdout: "reply": 0, 2024-03-18T16:21:18.040 INFO:teuthology.orchestra.run.smithi194.stdout: "reply_latency": { 2024-03-18T16:21:18.040 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:18.040 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:18.040 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:18.040 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:18.040 INFO:teuthology.orchestra.run.smithi194.stdout: "slow_reply": 0, 2024-03-18T16:21:18.040 INFO:teuthology.orchestra.run.smithi194.stdout: "forward": 0, 2024-03-18T16:21:18.040 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_fetch_complete": 0, 2024-03-18T16:21:18.040 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_fetch_keys": 0, 2024-03-18T16:21:18.040 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_commit": 0, 2024-03-18T16:21:18.040 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_split": 0, 2024-03-18T16:21:18.040 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_merge": 0, 2024-03-18T16:21:18.040 INFO:teuthology.orchestra.run.smithi194.stdout: "inodes": 49, 2024-03-18T16:21:18.041 INFO:teuthology.orchestra.run.smithi194.stdout: "inodes_top": 0, 2024-03-18T16:21:18.041 INFO:teuthology.orchestra.run.smithi194.stdout: "inodes_bottom": 49, 2024-03-18T16:21:18.041 INFO:teuthology.orchestra.run.smithi194.stdout: "inodes_pin_tail": 0, 2024-03-18T16:21:18.041 INFO:teuthology.orchestra.run.smithi194.stdout: "inodes_pinned": 49, 2024-03-18T16:21:18.041 INFO:teuthology.orchestra.run.smithi194.stdout: "inodes_expired": 1, 2024-03-18T16:21:18.041 INFO:teuthology.orchestra.run.smithi194.stdout: "inodes_with_caps": 0, 2024-03-18T16:21:18.041 INFO:teuthology.orchestra.run.smithi194.stdout: "caps": 0, 2024-03-18T16:21:18.041 INFO:teuthology.orchestra.run.smithi194.stdout: "subtrees": 1, 2024-03-18T16:21:18.041 INFO:teuthology.orchestra.run.smithi194.stdout: "traverse": 0, 2024-03-18T16:21:18.041 INFO:teuthology.orchestra.run.smithi194.stdout: "traverse_hit": 0, 2024-03-18T16:21:18.041 INFO:teuthology.orchestra.run.smithi194.stdout: "traverse_forward": 0, 2024-03-18T16:21:18.041 INFO:teuthology.orchestra.run.smithi194.stdout: "traverse_discover": 0, 2024-03-18T16:21:18.041 INFO:teuthology.orchestra.run.smithi194.stdout: "traverse_dir_fetch": 0, 2024-03-18T16:21:18.041 INFO:teuthology.orchestra.run.smithi194.stdout: "traverse_remote_ino": 0, 2024-03-18T16:21:18.041 INFO:teuthology.orchestra.run.smithi194.stdout: "traverse_lock": 0, 2024-03-18T16:21:18.041 INFO:teuthology.orchestra.run.smithi194.stdout: "load_cent": 0, 2024-03-18T16:21:18.042 INFO:teuthology.orchestra.run.smithi194.stdout: "q": 0, 2024-03-18T16:21:18.042 INFO:teuthology.orchestra.run.smithi194.stdout: "exported": 0, 2024-03-18T16:21:18.042 INFO:teuthology.orchestra.run.smithi194.stdout: "exported_inodes": 0, 2024-03-18T16:21:18.042 INFO:teuthology.orchestra.run.smithi194.stdout: "imported": 0, 2024-03-18T16:21:18.042 INFO:teuthology.orchestra.run.smithi194.stdout: "imported_inodes": 0, 2024-03-18T16:21:18.042 INFO:teuthology.orchestra.run.smithi194.stdout: "openino_dir_fetch": 0, 2024-03-18T16:21:18.042 INFO:teuthology.orchestra.run.smithi194.stdout: "openino_backtrace_fetch": 0, 2024-03-18T16:21:18.042 INFO:teuthology.orchestra.run.smithi194.stdout: "openino_peer_discover": 0, 2024-03-18T16:21:18.042 INFO:teuthology.orchestra.run.smithi194.stdout: "root_rfiles": 0, 2024-03-18T16:21:18.042 INFO:teuthology.orchestra.run.smithi194.stdout: "root_rbytes": 0, 2024-03-18T16:21:18.042 INFO:teuthology.orchestra.run.smithi194.stdout: "root_rsnaps": 0, 2024-03-18T16:21:18.042 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_backtrace_fetch": 0, 2024-03-18T16:21:18.042 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_set_tag": 0, 2024-03-18T16:21:18.042 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_backtrace_repaired": 0, 2024-03-18T16:21:18.042 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_inotable_repaired": 0, 2024-03-18T16:21:18.043 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_dir_inodes": 0, 2024-03-18T16:21:18.043 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_dir_base_inodes": 0, 2024-03-18T16:21:18.043 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_dirfrag_rstats": 0, 2024-03-18T16:21:18.043 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_file_inodes": 0, 2024-03-18T16:21:18.043 INFO:teuthology.orchestra.run.smithi194.stdout: "handle_inode_file_caps": 0, 2024-03-18T16:21:18.043 INFO:teuthology.orchestra.run.smithi194.stdout: "ceph_cap_op_revoke": 0, 2024-03-18T16:21:18.043 INFO:teuthology.orchestra.run.smithi194.stdout: "ceph_cap_op_grant": 0, 2024-03-18T16:21:18.043 INFO:teuthology.orchestra.run.smithi194.stdout: "ceph_cap_op_trunc": 0, 2024-03-18T16:21:18.043 INFO:teuthology.orchestra.run.smithi194.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-18T16:21:18.043 INFO:teuthology.orchestra.run.smithi194.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-18T16:21:18.043 INFO:teuthology.orchestra.run.smithi194.stdout: "handle_client_caps": 0, 2024-03-18T16:21:18.043 INFO:teuthology.orchestra.run.smithi194.stdout: "handle_client_caps_dirty": 0, 2024-03-18T16:21:18.043 INFO:teuthology.orchestra.run.smithi194.stdout: "handle_client_cap_release": 0, 2024-03-18T16:21:18.043 INFO:teuthology.orchestra.run.smithi194.stdout: "process_request_cap_release": 0 2024-03-18T16:21:18.043 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:18.044 INFO:teuthology.orchestra.run.smithi194.stdout: "mds_cache": { 2024-03-18T16:21:18.044 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_update": 0, 2024-03-18T16:21:18.044 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_update_receipt": 0, 2024-03-18T16:21:18.044 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_try_discover": 0, 2024-03-18T16:21:18.044 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_send_discover": 0, 2024-03-18T16:21:18.044 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_handle_discover": 0, 2024-03-18T16:21:18.044 INFO:teuthology.orchestra.run.smithi194.stdout: "num_strays": 0, 2024-03-18T16:21:18.044 INFO:teuthology.orchestra.run.smithi194.stdout: "num_strays_delayed": 0, 2024-03-18T16:21:18.044 INFO:teuthology.orchestra.run.smithi194.stdout: "num_strays_enqueuing": 0, 2024-03-18T16:21:18.044 INFO:teuthology.orchestra.run.smithi194.stdout: "strays_created": 0, 2024-03-18T16:21:18.044 INFO:teuthology.orchestra.run.smithi194.stdout: "strays_enqueued": 0, 2024-03-18T16:21:18.044 INFO:teuthology.orchestra.run.smithi194.stdout: "strays_reintegrated": 0, 2024-03-18T16:21:18.044 INFO:teuthology.orchestra.run.smithi194.stdout: "strays_migrated": 0, 2024-03-18T16:21:18.044 INFO:teuthology.orchestra.run.smithi194.stdout: "num_recovering_processing": 0, 2024-03-18T16:21:18.044 INFO:teuthology.orchestra.run.smithi194.stdout: "num_recovering_enqueued": 0, 2024-03-18T16:21:18.045 INFO:teuthology.orchestra.run.smithi194.stdout: "num_recovering_prioritized": 0, 2024-03-18T16:21:18.045 INFO:teuthology.orchestra.run.smithi194.stdout: "recovery_started": 0, 2024-03-18T16:21:18.045 INFO:teuthology.orchestra.run.smithi194.stdout: "recovery_completed": 0, 2024-03-18T16:21:18.045 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_quiesce_path": 0, 2024-03-18T16:21:18.045 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_quiesce_inode": 0, 2024-03-18T16:21:18.045 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_enqueue_scrub": 0, 2024-03-18T16:21:18.045 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_exportdir": 0, 2024-03-18T16:21:18.045 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_flush": 0, 2024-03-18T16:21:18.045 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_fragmentdir": 0, 2024-03-18T16:21:18.045 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_fragstats": 0, 2024-03-18T16:21:18.045 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_inodestats": 0 2024-03-18T16:21:18.045 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:18.045 INFO:teuthology.orchestra.run.smithi194.stdout: "mds_log": { 2024-03-18T16:21:18.045 INFO:teuthology.orchestra.run.smithi194.stdout: "evlrg": 0, 2024-03-18T16:21:18.045 INFO:teuthology.orchestra.run.smithi194.stdout: "evadd": 0, 2024-03-18T16:21:18.046 INFO:teuthology.orchestra.run.smithi194.stdout: "evex": 0, 2024-03-18T16:21:18.046 INFO:teuthology.orchestra.run.smithi194.stdout: "evtrm": 0, 2024-03-18T16:21:18.046 INFO:teuthology.orchestra.run.smithi194.stdout: "ev": 3016, 2024-03-18T16:21:18.046 INFO:teuthology.orchestra.run.smithi194.stdout: "evexg": 0, 2024-03-18T16:21:18.046 INFO:teuthology.orchestra.run.smithi194.stdout: "evexd": 0, 2024-03-18T16:21:18.046 INFO:teuthology.orchestra.run.smithi194.stdout: "segadd": 0, 2024-03-18T16:21:18.046 INFO:teuthology.orchestra.run.smithi194.stdout: "segex": 0, 2024-03-18T16:21:18.046 INFO:teuthology.orchestra.run.smithi194.stdout: "segtrm": 0, 2024-03-18T16:21:18.046 INFO:teuthology.orchestra.run.smithi194.stdout: "seg": 5, 2024-03-18T16:21:18.046 INFO:teuthology.orchestra.run.smithi194.stdout: "segmjr": 2, 2024-03-18T16:21:18.046 INFO:teuthology.orchestra.run.smithi194.stdout: "segexg": 0, 2024-03-18T16:21:18.047 INFO:teuthology.orchestra.run.smithi194.stdout: "segexd": 0, 2024-03-18T16:21:18.047 INFO:teuthology.orchestra.run.smithi194.stdout: "expos": 4194304, 2024-03-18T16:21:18.047 INFO:teuthology.orchestra.run.smithi194.stdout: "wrpos": 10556918, 2024-03-18T16:21:18.047 INFO:teuthology.orchestra.run.smithi194.stdout: "rdpos": 10554754, 2024-03-18T16:21:18.047 INFO:teuthology.orchestra.run.smithi194.stdout: "jlat": { 2024-03-18T16:21:18.047 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:18.047 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:18.047 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:18.047 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:18.047 INFO:teuthology.orchestra.run.smithi194.stdout: "replayed": 3016 2024-03-18T16:21:18.047 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:18.047 INFO:teuthology.orchestra.run.smithi194.stdout: "mds_mem": { 2024-03-18T16:21:18.047 INFO:teuthology.orchestra.run.smithi194.stdout: "ino": 5, 2024-03-18T16:21:18.047 INFO:teuthology.orchestra.run.smithi194.stdout: "ino+": 378, 2024-03-18T16:21:18.047 INFO:teuthology.orchestra.run.smithi194.stdout: "ino-": 373, 2024-03-18T16:21:18.048 INFO:teuthology.orchestra.run.smithi194.stdout: "dir": 3, 2024-03-18T16:21:18.048 INFO:teuthology.orchestra.run.smithi194.stdout: "dir+": 133, 2024-03-18T16:21:18.048 INFO:teuthology.orchestra.run.smithi194.stdout: "dir-": 130, 2024-03-18T16:21:18.048 INFO:teuthology.orchestra.run.smithi194.stdout: "dn": 49, 2024-03-18T16:21:18.048 INFO:teuthology.orchestra.run.smithi194.stdout: "dn+": 618, 2024-03-18T16:21:18.048 INFO:teuthology.orchestra.run.smithi194.stdout: "dn-": 569, 2024-03-18T16:21:18.048 INFO:teuthology.orchestra.run.smithi194.stdout: "cap": 0, 2024-03-18T16:21:18.048 INFO:teuthology.orchestra.run.smithi194.stdout: "cap+": 0, 2024-03-18T16:21:18.048 INFO:teuthology.orchestra.run.smithi194.stdout: "cap-": 0, 2024-03-18T16:21:18.048 INFO:teuthology.orchestra.run.smithi194.stdout: "rss": 50204, 2024-03-18T16:21:18.048 INFO:teuthology.orchestra.run.smithi194.stdout: "heap": 215420 2024-03-18T16:21:18.048 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:18.048 INFO:teuthology.orchestra.run.smithi194.stdout: "mds_server": { 2024-03-18T16:21:18.048 INFO:teuthology.orchestra.run.smithi194.stdout: "dispatch_client_request": 0, 2024-03-18T16:21:18.049 INFO:teuthology.orchestra.run.smithi194.stdout: "dispatch_server_request": 0, 2024-03-18T16:21:18.049 INFO:teuthology.orchestra.run.smithi194.stdout: "handle_client_request": 0, 2024-03-18T16:21:18.049 INFO:teuthology.orchestra.run.smithi194.stdout: "handle_client_session": 0, 2024-03-18T16:21:18.049 INFO:teuthology.orchestra.run.smithi194.stdout: "handle_peer_request": 0, 2024-03-18T16:21:18.049 INFO:teuthology.orchestra.run.smithi194.stdout: "req_create_latency": { 2024-03-18T16:21:18.049 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:18.049 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:18.049 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:18.049 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:18.049 INFO:teuthology.orchestra.run.smithi194.stdout: "req_getattr_latency": { 2024-03-18T16:21:18.049 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:18.049 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:18.049 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:18.049 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:18.049 INFO:teuthology.orchestra.run.smithi194.stdout: "req_getfilelock_latency": { 2024-03-18T16:21:18.049 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:18.050 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:18.050 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:18.050 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:18.050 INFO:teuthology.orchestra.run.smithi194.stdout: "req_link_latency": { 2024-03-18T16:21:18.050 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:18.050 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:18.050 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:18.050 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:18.050 INFO:teuthology.orchestra.run.smithi194.stdout: "req_lookup_latency": { 2024-03-18T16:21:18.050 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:18.050 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:18.050 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:18.050 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:18.050 INFO:teuthology.orchestra.run.smithi194.stdout: "req_lookuphash_latency": { 2024-03-18T16:21:18.051 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:18.051 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:18.051 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:18.051 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:18.051 INFO:teuthology.orchestra.run.smithi194.stdout: "req_lookupino_latency": { 2024-03-18T16:21:18.051 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:18.051 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:18.051 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:18.051 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:18.051 INFO:teuthology.orchestra.run.smithi194.stdout: "req_lookupname_latency": { 2024-03-18T16:21:18.051 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:18.051 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:18.051 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:18.051 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:18.051 INFO:teuthology.orchestra.run.smithi194.stdout: "req_lookupparent_latency": { 2024-03-18T16:21:18.052 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:18.052 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:18.052 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:18.052 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:18.052 INFO:teuthology.orchestra.run.smithi194.stdout: "req_lookupsnap_latency": { 2024-03-18T16:21:18.052 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:18.052 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:18.052 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:18.052 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:18.052 INFO:teuthology.orchestra.run.smithi194.stdout: "req_lssnap_latency": { 2024-03-18T16:21:18.052 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:18.052 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:18.052 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:18.052 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:18.052 INFO:teuthology.orchestra.run.smithi194.stdout: "req_mkdir_latency": { 2024-03-18T16:21:18.052 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:18.053 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:18.053 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:18.053 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:18.053 INFO:teuthology.orchestra.run.smithi194.stdout: "req_mknod_latency": { 2024-03-18T16:21:18.053 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:18.054 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:18.054 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:18.054 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:18.054 INFO:teuthology.orchestra.run.smithi194.stdout: "req_mksnap_latency": { 2024-03-18T16:21:18.054 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:18.054 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:18.054 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:18.054 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:18.054 INFO:teuthology.orchestra.run.smithi194.stdout: "req_open_latency": { 2024-03-18T16:21:18.054 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:18.054 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:18.054 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:18.054 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:18.054 INFO:teuthology.orchestra.run.smithi194.stdout: "req_readdir_latency": { 2024-03-18T16:21:18.054 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:18.055 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:18.055 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:18.055 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:18.055 INFO:teuthology.orchestra.run.smithi194.stdout: "req_rename_latency": { 2024-03-18T16:21:18.055 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:18.055 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:18.055 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:18.055 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:18.055 INFO:teuthology.orchestra.run.smithi194.stdout: "req_renamesnap_latency": { 2024-03-18T16:21:18.055 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:18.055 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:18.055 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:18.055 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:18.055 INFO:teuthology.orchestra.run.smithi194.stdout: "req_snapdiff_latency": { 2024-03-18T16:21:18.055 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:18.056 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:18.056 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:18.056 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:18.056 INFO:teuthology.orchestra.run.smithi194.stdout: "req_rmdir_latency": { 2024-03-18T16:21:18.056 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:18.056 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:18.056 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:18.056 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:18.056 INFO:teuthology.orchestra.run.smithi194.stdout: "req_rmsnap_latency": { 2024-03-18T16:21:18.056 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:18.056 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:18.056 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:18.056 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:18.056 INFO:teuthology.orchestra.run.smithi194.stdout: "req_rmxattr_latency": { 2024-03-18T16:21:18.056 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:18.057 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:18.057 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:18.057 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:18.057 INFO:teuthology.orchestra.run.smithi194.stdout: "req_setattr_latency": { 2024-03-18T16:21:18.057 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:18.057 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:18.057 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:18.057 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:18.057 INFO:teuthology.orchestra.run.smithi194.stdout: "req_setdirlayout_latency": { 2024-03-18T16:21:18.057 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:18.057 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:18.057 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:18.057 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:18.057 INFO:teuthology.orchestra.run.smithi194.stdout: "req_setfilelock_latency": { 2024-03-18T16:21:18.057 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:18.058 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:18.058 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:18.058 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:18.058 INFO:teuthology.orchestra.run.smithi194.stdout: "req_setlayout_latency": { 2024-03-18T16:21:18.058 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:18.058 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:18.058 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:18.058 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:18.058 INFO:teuthology.orchestra.run.smithi194.stdout: "req_setxattr_latency": { 2024-03-18T16:21:18.058 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:18.059 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:18.059 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:18.059 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:18.059 INFO:teuthology.orchestra.run.smithi194.stdout: "req_symlink_latency": { 2024-03-18T16:21:18.059 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:18.059 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:18.059 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:18.059 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:18.059 INFO:teuthology.orchestra.run.smithi194.stdout: "req_unlink_latency": { 2024-03-18T16:21:18.059 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:18.059 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:18.059 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:18.059 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:18.059 INFO:teuthology.orchestra.run.smithi194.stdout: "cap_revoke_eviction": 0, 2024-03-18T16:21:18.059 INFO:teuthology.orchestra.run.smithi194.stdout: "cap_acquisition_throttle": 0, 2024-03-18T16:21:18.060 INFO:teuthology.orchestra.run.smithi194.stdout: "req_getvxattr_latency": { 2024-03-18T16:21:18.060 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:18.060 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:18.060 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:18.060 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-03-18T16:21:18.060 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:18.060 INFO:teuthology.orchestra.run.smithi194.stdout: "mds_sessions": { 2024-03-18T16:21:18.060 INFO:teuthology.orchestra.run.smithi194.stdout: "session_count": 1, 2024-03-18T16:21:18.060 INFO:teuthology.orchestra.run.smithi194.stdout: "session_add": 2, 2024-03-18T16:21:18.060 INFO:teuthology.orchestra.run.smithi194.stdout: "session_remove": 1, 2024-03-18T16:21:18.060 INFO:teuthology.orchestra.run.smithi194.stdout: "sessions_open": 1, 2024-03-18T16:21:18.060 INFO:teuthology.orchestra.run.smithi194.stdout: "sessions_stale": 0, 2024-03-18T16:21:18.060 INFO:teuthology.orchestra.run.smithi194.stdout: "total_load": 0, 2024-03-18T16:21:18.060 INFO:teuthology.orchestra.run.smithi194.stdout: "average_load": 0, 2024-03-18T16:21:18.060 INFO:teuthology.orchestra.run.smithi194.stdout: "avg_session_uptime": 2470, 2024-03-18T16:21:18.061 INFO:teuthology.orchestra.run.smithi194.stdout: "mdthresh_evicted": 0 2024-03-18T16:21:18.061 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:18.061 INFO:teuthology.orchestra.run.smithi194.stdout: "mempool": { 2024-03-18T16:21:18.061 INFO:teuthology.orchestra.run.smithi194.stdout: "bloom_filter_bytes": 0, 2024-03-18T16:21:18.061 INFO:teuthology.orchestra.run.smithi194.stdout: "bloom_filter_items": 0, 2024-03-18T16:21:18.061 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T16:21:18.061 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_alloc_items": 0, 2024-03-18T16:21:18.061 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T16:21:18.061 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_data_items": 0, 2024-03-18T16:21:18.061 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T16:21:18.061 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T16:21:18.061 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T16:21:18.061 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T16:21:18.061 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T16:21:18.062 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_other_items": 0, 2024-03-18T16:21:18.062 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T16:21:18.062 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T16:21:18.062 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_extent_bytes": 0, 2024-03-18T16:21:18.062 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_extent_items": 0, 2024-03-18T16:21:18.062 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_blob_bytes": 0, 2024-03-18T16:21:18.062 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_blob_items": 0, 2024-03-18T16:21:18.062 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T16:21:18.062 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T16:21:18.062 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T16:21:18.062 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T16:21:18.062 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T16:21:18.062 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_fsck_items": 0, 2024-03-18T16:21:18.062 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_txc_bytes": 0, 2024-03-18T16:21:18.063 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_txc_items": 0, 2024-03-18T16:21:18.063 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T16:21:18.063 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T16:21:18.063 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_writing_bytes": 0, 2024-03-18T16:21:18.063 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_writing_items": 0, 2024-03-18T16:21:18.063 INFO:teuthology.orchestra.run.smithi194.stdout: "bluefs_bytes": 0, 2024-03-18T16:21:18.063 INFO:teuthology.orchestra.run.smithi194.stdout: "bluefs_items": 0, 2024-03-18T16:21:18.063 INFO:teuthology.orchestra.run.smithi194.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T16:21:18.063 INFO:teuthology.orchestra.run.smithi194.stdout: "bluefs_file_reader_items": 0, 2024-03-18T16:21:18.063 INFO:teuthology.orchestra.run.smithi194.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T16:21:18.063 INFO:teuthology.orchestra.run.smithi194.stdout: "bluefs_file_writer_items": 0, 2024-03-18T16:21:18.063 INFO:teuthology.orchestra.run.smithi194.stdout: "buffer_anon_bytes": 181321, 2024-03-18T16:21:18.063 INFO:teuthology.orchestra.run.smithi194.stdout: "buffer_anon_items": 47, 2024-03-18T16:21:18.063 INFO:teuthology.orchestra.run.smithi194.stdout: "buffer_meta_bytes": 0, 2024-03-18T16:21:18.063 INFO:teuthology.orchestra.run.smithi194.stdout: "buffer_meta_items": 0, 2024-03-18T16:21:18.064 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_bytes": 0, 2024-03-18T16:21:18.064 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_items": 0, 2024-03-18T16:21:18.064 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_mapbl_bytes": 0, 2024-03-18T16:21:18.064 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_mapbl_items": 0, 2024-03-18T16:21:18.064 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_pglog_bytes": 0, 2024-03-18T16:21:18.064 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_pglog_items": 0, 2024-03-18T16:21:18.064 INFO:teuthology.orchestra.run.smithi194.stdout: "osdmap_bytes": 8888, 2024-03-18T16:21:18.064 INFO:teuthology.orchestra.run.smithi194.stdout: "osdmap_items": 170, 2024-03-18T16:21:18.064 INFO:teuthology.orchestra.run.smithi194.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T16:21:18.064 INFO:teuthology.orchestra.run.smithi194.stdout: "osdmap_mapping_items": 0, 2024-03-18T16:21:18.064 INFO:teuthology.orchestra.run.smithi194.stdout: "pgmap_bytes": 0, 2024-03-18T16:21:18.064 INFO:teuthology.orchestra.run.smithi194.stdout: "pgmap_items": 0, 2024-03-18T16:21:18.064 INFO:teuthology.orchestra.run.smithi194.stdout: "mds_co_bytes": 49968, 2024-03-18T16:21:18.065 INFO:teuthology.orchestra.run.smithi194.stdout: "mds_co_items": 235, 2024-03-18T16:21:18.065 INFO:teuthology.orchestra.run.smithi194.stdout: "unittest_1_bytes": 0, 2024-03-18T16:21:18.065 INFO:teuthology.orchestra.run.smithi194.stdout: "unittest_1_items": 0, 2024-03-18T16:21:18.065 INFO:teuthology.orchestra.run.smithi194.stdout: "unittest_2_bytes": 0, 2024-03-18T16:21:18.065 INFO:teuthology.orchestra.run.smithi194.stdout: "unittest_2_items": 0 2024-03-18T16:21:18.065 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:18.065 INFO:teuthology.orchestra.run.smithi194.stdout: "objecter": { 2024-03-18T16:21:18.065 INFO:teuthology.orchestra.run.smithi194.stdout: "op_active": 0, 2024-03-18T16:21:18.065 INFO:teuthology.orchestra.run.smithi194.stdout: "op_laggy": 0, 2024-03-18T16:21:18.065 INFO:teuthology.orchestra.run.smithi194.stdout: "op_send": 2447, 2024-03-18T16:21:18.065 INFO:teuthology.orchestra.run.smithi194.stdout: "op_send_bytes": 21, 2024-03-18T16:21:18.065 INFO:teuthology.orchestra.run.smithi194.stdout: "op_resend": 3, 2024-03-18T16:21:18.065 INFO:teuthology.orchestra.run.smithi194.stdout: "op_reply": 2447, 2024-03-18T16:21:18.066 INFO:teuthology.orchestra.run.smithi194.stdout: "op_latency": { 2024-03-18T16:21:18.066 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 2447, 2024-03-18T16:21:18.066 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 60.408784932, 2024-03-18T16:21:18.066 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.024686875 2024-03-18T16:21:18.066 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:18.066 INFO:teuthology.orchestra.run.smithi194.stdout: "op_inflight": 0, 2024-03-18T16:21:18.066 INFO:teuthology.orchestra.run.smithi194.stdout: "oplen_avg": { 2024-03-18T16:21:18.066 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 2447, 2024-03-18T16:21:18.066 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 2448 2024-03-18T16:21:18.066 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:18.066 INFO:teuthology.orchestra.run.smithi194.stdout: "op": 2447, 2024-03-18T16:21:18.066 INFO:teuthology.orchestra.run.smithi194.stdout: "op_r": 2447, 2024-03-18T16:21:18.066 INFO:teuthology.orchestra.run.smithi194.stdout: "op_w": 0, 2024-03-18T16:21:18.066 INFO:teuthology.orchestra.run.smithi194.stdout: "op_rmw": 0, 2024-03-18T16:21:18.067 INFO:teuthology.orchestra.run.smithi194.stdout: "op_pg": 0, 2024-03-18T16:21:18.067 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_stat": 1480, 2024-03-18T16:21:18.067 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_create": 0, 2024-03-18T16:21:18.067 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_read": 965, 2024-03-18T16:21:18.067 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_write": 0, 2024-03-18T16:21:18.067 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_writefull": 0, 2024-03-18T16:21:18.067 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_writesame": 0, 2024-03-18T16:21:18.067 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_append": 0, 2024-03-18T16:21:18.067 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_zero": 0, 2024-03-18T16:21:18.067 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_truncate": 0, 2024-03-18T16:21:18.067 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_delete": 0, 2024-03-18T16:21:18.067 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_mapext": 0, 2024-03-18T16:21:18.068 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_sparse_read": 0, 2024-03-18T16:21:18.068 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_clonerange": 0, 2024-03-18T16:21:18.068 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_getxattr": 1, 2024-03-18T16:21:18.068 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_setxattr": 0, 2024-03-18T16:21:18.068 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_cmpxattr": 0, 2024-03-18T16:21:18.068 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_rmxattr": 0, 2024-03-18T16:21:18.068 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_resetxattrs": 0, 2024-03-18T16:21:18.068 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_call": 0, 2024-03-18T16:21:18.068 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_watch": 0, 2024-03-18T16:21:18.068 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_notify": 0, 2024-03-18T16:21:18.068 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_src_cmpxattr": 0, 2024-03-18T16:21:18.068 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_pgls": 0, 2024-03-18T16:21:18.068 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_pgls_filter": 0, 2024-03-18T16:21:18.068 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_other": 0, 2024-03-18T16:21:18.068 INFO:teuthology.orchestra.run.smithi194.stdout: "linger_active": 0, 2024-03-18T16:21:18.069 INFO:teuthology.orchestra.run.smithi194.stdout: "linger_send": 0, 2024-03-18T16:21:18.069 INFO:teuthology.orchestra.run.smithi194.stdout: "linger_resend": 0, 2024-03-18T16:21:18.069 INFO:teuthology.orchestra.run.smithi194.stdout: "linger_ping": 0, 2024-03-18T16:21:18.069 INFO:teuthology.orchestra.run.smithi194.stdout: "poolop_active": 0, 2024-03-18T16:21:18.069 INFO:teuthology.orchestra.run.smithi194.stdout: "poolop_send": 0, 2024-03-18T16:21:18.069 INFO:teuthology.orchestra.run.smithi194.stdout: "poolop_resend": 0, 2024-03-18T16:21:18.069 INFO:teuthology.orchestra.run.smithi194.stdout: "poolstat_active": 0, 2024-03-18T16:21:18.069 INFO:teuthology.orchestra.run.smithi194.stdout: "poolstat_send": 0, 2024-03-18T16:21:18.069 INFO:teuthology.orchestra.run.smithi194.stdout: "poolstat_resend": 0, 2024-03-18T16:21:18.069 INFO:teuthology.orchestra.run.smithi194.stdout: "statfs_active": 0, 2024-03-18T16:21:18.069 INFO:teuthology.orchestra.run.smithi194.stdout: "statfs_send": 0, 2024-03-18T16:21:18.069 INFO:teuthology.orchestra.run.smithi194.stdout: "statfs_resend": 0, 2024-03-18T16:21:18.069 INFO:teuthology.orchestra.run.smithi194.stdout: "command_active": 0, 2024-03-18T16:21:18.069 INFO:teuthology.orchestra.run.smithi194.stdout: "command_send": 0, 2024-03-18T16:21:18.069 INFO:teuthology.orchestra.run.smithi194.stdout: "command_resend": 0, 2024-03-18T16:21:18.070 INFO:teuthology.orchestra.run.smithi194.stdout: "map_epoch": 93, 2024-03-18T16:21:18.070 INFO:teuthology.orchestra.run.smithi194.stdout: "map_full": 0, 2024-03-18T16:21:18.070 INFO:teuthology.orchestra.run.smithi194.stdout: "map_inc": 4, 2024-03-18T16:21:18.070 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_sessions": 4, 2024-03-18T16:21:18.070 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_session_open": 4, 2024-03-18T16:21:18.070 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_session_close": 0, 2024-03-18T16:21:18.070 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_laggy": 0, 2024-03-18T16:21:18.070 INFO:teuthology.orchestra.run.smithi194.stdout: "omap_wr": 0, 2024-03-18T16:21:18.070 INFO:teuthology.orchestra.run.smithi194.stdout: "omap_rd": 2, 2024-03-18T16:21:18.070 INFO:teuthology.orchestra.run.smithi194.stdout: "omap_del": 0 2024-03-18T16:21:18.070 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:18.070 INFO:teuthology.orchestra.run.smithi194.stdout: "oft": { 2024-03-18T16:21:18.070 INFO:teuthology.orchestra.run.smithi194.stdout: "omap_total_objs": 0, 2024-03-18T16:21:18.070 INFO:teuthology.orchestra.run.smithi194.stdout: "omap_total_kv_pairs": 0, 2024-03-18T16:21:18.070 INFO:teuthology.orchestra.run.smithi194.stdout: "omap_total_updates": 0, 2024-03-18T16:21:18.071 INFO:teuthology.orchestra.run.smithi194.stdout: "omap_total_removes": 0 2024-03-18T16:21:18.071 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:18.071 INFO:teuthology.orchestra.run.smithi194.stdout: "osd-slow-ops": { 2024-03-18T16:21:18.071 INFO:teuthology.orchestra.run.smithi194.stdout: "slow_ops_count": 0 2024-03-18T16:21:18.071 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:18.071 INFO:teuthology.orchestra.run.smithi194.stdout: "purge_queue": { 2024-03-18T16:21:18.071 INFO:teuthology.orchestra.run.smithi194.stdout: "pq_executing_ops": 0, 2024-03-18T16:21:18.071 INFO:teuthology.orchestra.run.smithi194.stdout: "pq_executing_ops_high_water": 0, 2024-03-18T16:21:18.071 INFO:teuthology.orchestra.run.smithi194.stdout: "pq_executing": 0, 2024-03-18T16:21:18.071 INFO:teuthology.orchestra.run.smithi194.stdout: "pq_executing_high_water": 0, 2024-03-18T16:21:18.071 INFO:teuthology.orchestra.run.smithi194.stdout: "pq_executed_ops": 0, 2024-03-18T16:21:18.071 INFO:teuthology.orchestra.run.smithi194.stdout: "pq_executed": 0, 2024-03-18T16:21:18.071 INFO:teuthology.orchestra.run.smithi194.stdout: "pq_item_in_journal": 0 2024-03-18T16:21:18.071 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:18.071 INFO:teuthology.orchestra.run.smithi194.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T16:21:18.072 INFO:teuthology.orchestra.run.smithi194.stdout: "val": 0, 2024-03-18T16:21:18.072 INFO:teuthology.orchestra.run.smithi194.stdout: "max": 104857600, 2024-03-18T16:21:18.072 INFO:teuthology.orchestra.run.smithi194.stdout: "get_started": 0, 2024-03-18T16:21:18.072 INFO:teuthology.orchestra.run.smithi194.stdout: "get": 2726, 2024-03-18T16:21:18.072 INFO:teuthology.orchestra.run.smithi194.stdout: "get_sum": 7452006, 2024-03-18T16:21:18.072 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_fail": 0, 2024-03-18T16:21:18.072 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_success": 2726, 2024-03-18T16:21:18.072 INFO:teuthology.orchestra.run.smithi194.stdout: "take": 0, 2024-03-18T16:21:18.072 INFO:teuthology.orchestra.run.smithi194.stdout: "take_sum": 0, 2024-03-18T16:21:18.072 INFO:teuthology.orchestra.run.smithi194.stdout: "put": 2726, 2024-03-18T16:21:18.072 INFO:teuthology.orchestra.run.smithi194.stdout: "put_sum": 7452006, 2024-03-18T16:21:18.072 INFO:teuthology.orchestra.run.smithi194.stdout: "wait": { 2024-03-18T16:21:18.072 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:18.073 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:18.073 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:18.073 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-03-18T16:21:18.073 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:18.073 INFO:teuthology.orchestra.run.smithi194.stdout: "throttle-objecter_bytes": { 2024-03-18T16:21:18.073 INFO:teuthology.orchestra.run.smithi194.stdout: "val": 0, 2024-03-18T16:21:18.073 INFO:teuthology.orchestra.run.smithi194.stdout: "max": 104857600, 2024-03-18T16:21:18.073 INFO:teuthology.orchestra.run.smithi194.stdout: "get_started": 0, 2024-03-18T16:21:18.073 INFO:teuthology.orchestra.run.smithi194.stdout: "get": 0, 2024-03-18T16:21:18.073 INFO:teuthology.orchestra.run.smithi194.stdout: "get_sum": 0, 2024-03-18T16:21:18.073 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_fail": 0, 2024-03-18T16:21:18.073 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_success": 0, 2024-03-18T16:21:18.073 INFO:teuthology.orchestra.run.smithi194.stdout: "take": 2447, 2024-03-18T16:21:18.074 INFO:teuthology.orchestra.run.smithi194.stdout: "take_sum": 6363352, 2024-03-18T16:21:18.074 INFO:teuthology.orchestra.run.smithi194.stdout: "put": 2447, 2024-03-18T16:21:18.074 INFO:teuthology.orchestra.run.smithi194.stdout: "put_sum": 6363352, 2024-03-18T16:21:18.074 INFO:teuthology.orchestra.run.smithi194.stdout: "wait": { 2024-03-18T16:21:18.074 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:18.074 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:18.074 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:18.074 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-03-18T16:21:18.074 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:18.074 INFO:teuthology.orchestra.run.smithi194.stdout: "throttle-objecter_ops": { 2024-03-18T16:21:18.074 INFO:teuthology.orchestra.run.smithi194.stdout: "val": 0, 2024-03-18T16:21:18.075 INFO:teuthology.orchestra.run.smithi194.stdout: "max": 1024, 2024-03-18T16:21:18.075 INFO:teuthology.orchestra.run.smithi194.stdout: "get_started": 0, 2024-03-18T16:21:18.075 INFO:teuthology.orchestra.run.smithi194.stdout: "get": 0, 2024-03-18T16:21:18.075 INFO:teuthology.orchestra.run.smithi194.stdout: "get_sum": 0, 2024-03-18T16:21:18.075 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_fail": 0, 2024-03-18T16:21:18.075 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_success": 0, 2024-03-18T16:21:18.075 INFO:teuthology.orchestra.run.smithi194.stdout: "take": 2447, 2024-03-18T16:21:18.075 INFO:teuthology.orchestra.run.smithi194.stdout: "take_sum": 2447, 2024-03-18T16:21:18.075 INFO:teuthology.orchestra.run.smithi194.stdout: "put": 2447, 2024-03-18T16:21:18.075 INFO:teuthology.orchestra.run.smithi194.stdout: "put_sum": 2447, 2024-03-18T16:21:18.075 INFO:teuthology.orchestra.run.smithi194.stdout: "wait": { 2024-03-18T16:21:18.075 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:18.075 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:18.075 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:18.076 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-03-18T16:21:18.076 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:18.076 INFO:teuthology.orchestra.run.smithi194.stdout: "throttle-write_buf_throttle": { 2024-03-18T16:21:18.076 INFO:teuthology.orchestra.run.smithi194.stdout: "val": 0, 2024-03-18T16:21:18.076 INFO:teuthology.orchestra.run.smithi194.stdout: "max": 3758096384, 2024-03-18T16:21:18.076 INFO:teuthology.orchestra.run.smithi194.stdout: "get_started": 0, 2024-03-18T16:21:18.076 INFO:teuthology.orchestra.run.smithi194.stdout: "get": 0, 2024-03-18T16:21:18.076 INFO:teuthology.orchestra.run.smithi194.stdout: "get_sum": 0, 2024-03-18T16:21:18.076 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_fail": 0, 2024-03-18T16:21:18.076 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_success": 0, 2024-03-18T16:21:18.076 INFO:teuthology.orchestra.run.smithi194.stdout: "take": 0, 2024-03-18T16:21:18.077 INFO:teuthology.orchestra.run.smithi194.stdout: "take_sum": 0, 2024-03-18T16:21:18.077 INFO:teuthology.orchestra.run.smithi194.stdout: "put": 0, 2024-03-18T16:21:18.077 INFO:teuthology.orchestra.run.smithi194.stdout: "put_sum": 0, 2024-03-18T16:21:18.077 INFO:teuthology.orchestra.run.smithi194.stdout: "wait": { 2024-03-18T16:21:18.077 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:18.077 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:18.077 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:18.077 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-03-18T16:21:18.077 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:18.077 INFO:teuthology.orchestra.run.smithi194.stdout: "throttle-write_buf_throttle-0x55aaf0daa6e0": { 2024-03-18T16:21:18.077 INFO:teuthology.orchestra.run.smithi194.stdout: "val": 0, 2024-03-18T16:21:18.077 INFO:teuthology.orchestra.run.smithi194.stdout: "max": 3758096384, 2024-03-18T16:21:18.077 INFO:teuthology.orchestra.run.smithi194.stdout: "get_started": 0, 2024-03-18T16:21:18.077 INFO:teuthology.orchestra.run.smithi194.stdout: "get": 0, 2024-03-18T16:21:18.078 INFO:teuthology.orchestra.run.smithi194.stdout: "get_sum": 0, 2024-03-18T16:21:18.078 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_fail": 0, 2024-03-18T16:21:18.078 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_success": 0, 2024-03-18T16:21:18.078 INFO:teuthology.orchestra.run.smithi194.stdout: "take": 0, 2024-03-18T16:21:18.078 INFO:teuthology.orchestra.run.smithi194.stdout: "take_sum": 0, 2024-03-18T16:21:18.078 INFO:teuthology.orchestra.run.smithi194.stdout: "put": 0, 2024-03-18T16:21:18.078 INFO:teuthology.orchestra.run.smithi194.stdout: "put_sum": 0, 2024-03-18T16:21:18.078 INFO:teuthology.orchestra.run.smithi194.stdout: "wait": { 2024-03-18T16:21:18.078 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:18.079 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:18.079 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:18.079 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-03-18T16:21:18.079 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-03-18T16:21:18.079 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-03-18T16:21:18.851 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-03-18T16:21:18.851 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-03-18T16:21:18.852 INFO:tasks.check_counter:Daemon mds.e mds.dir_split=0 2024-03-18T16:21:18.852 DEBUG:tasks.check_counter:Getting stats from h 2024-03-18T16:21:18.852 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph daemon mds.h perf dump 2024-03-18T16:21:19.125 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T16:21:19.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:18 smithi196 ceph-mon[36307]: pgmap v759: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T16:21:19.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:18 smithi194 ceph-mon[36200]: pgmap v759: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T16:21:19.289 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:18 smithi039 ceph-mon[31721]: pgmap v759: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T16:21:20.904 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-03-18T16:21:20.904 INFO:teuthology.orchestra.run.smithi194.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T16:21:20.904 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_messages": 2173, 2024-03-18T16:21:20.904 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_messages": 2644, 2024-03-18T16:21:20.904 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_bytes": 716531, 2024-03-18T16:21:20.904 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_bytes": 3415603, 2024-03-18T16:21:20.904 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_created_connections": 22, 2024-03-18T16:21:20.904 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_active_connections": 5, 2024-03-18T16:21:20.905 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_total_time": 0.180652373, 2024-03-18T16:21:20.905 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_send_time": 0.065591512, 2024-03-18T16:21:20.905 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_recv_time": 0.115532377, 2024-03-18T16:21:20.905 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_fast_dispatch_time": 0.031513171, 2024-03-18T16:21:20.905 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T16:21:20.905 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 2640, 2024-03-18T16:21:20.905 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.353007448, 2024-03-18T16:21:20.905 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000133714 2024-03-18T16:21:20.905 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:20.905 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_handle_ack_lat": { 2024-03-18T16:21:20.905 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 2176, 2024-03-18T16:21:20.905 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000627189, 2024-03-18T16:21:20.905 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000288 2024-03-18T16:21:20.905 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:20.905 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_encrypted_bytes": 469264, 2024-03-18T16:21:20.906 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_encrypted_bytes": 758464 2024-03-18T16:21:20.906 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:20.906 INFO:teuthology.orchestra.run.smithi194.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T16:21:20.906 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_messages": 2455, 2024-03-18T16:21:20.906 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_messages": 1902, 2024-03-18T16:21:20.906 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_bytes": 1446961, 2024-03-18T16:21:20.906 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_bytes": 1053000, 2024-03-18T16:21:20.906 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_created_connections": 9, 2024-03-18T16:21:20.906 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_active_connections": 6, 2024-03-18T16:21:20.906 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_total_time": 0.149868079, 2024-03-18T16:21:20.906 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_send_time": 0.050106833, 2024-03-18T16:21:20.906 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_recv_time": 0.107381432, 2024-03-18T16:21:20.906 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_fast_dispatch_time": 0.022417884, 2024-03-18T16:21:20.906 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T16:21:20.907 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 1903, 2024-03-18T16:21:20.907 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.127154965, 2024-03-18T16:21:20.909 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000066818 2024-03-18T16:21:20.909 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:20.909 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_handle_ack_lat": { 2024-03-18T16:21:20.909 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 2261, 2024-03-18T16:21:20.909 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000361356, 2024-03-18T16:21:20.909 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000159 2024-03-18T16:21:20.909 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:20.909 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_encrypted_bytes": 674000, 2024-03-18T16:21:20.909 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_encrypted_bytes": 166864 2024-03-18T16:21:20.909 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:20.909 INFO:teuthology.orchestra.run.smithi194.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T16:21:20.909 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_messages": 1964, 2024-03-18T16:21:20.909 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_messages": 2613, 2024-03-18T16:21:20.910 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_bytes": 1024957, 2024-03-18T16:21:20.910 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_bytes": 1841917, 2024-03-18T16:21:20.910 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_created_connections": 38, 2024-03-18T16:21:20.910 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_active_connections": 10, 2024-03-18T16:21:20.910 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_total_time": 0.161536905, 2024-03-18T16:21:20.910 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_send_time": 0.064531953, 2024-03-18T16:21:20.910 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_recv_time": 0.089299562, 2024-03-18T16:21:20.910 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_fast_dispatch_time": 0.015575764, 2024-03-18T16:21:20.910 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T16:21:20.910 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 2612, 2024-03-18T16:21:20.911 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.197273415, 2024-03-18T16:21:20.911 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000075525 2024-03-18T16:21:20.911 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:20.911 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_handle_ack_lat": { 2024-03-18T16:21:20.911 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 3279, 2024-03-18T16:21:20.911 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000973479, 2024-03-18T16:21:20.911 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000296 2024-03-18T16:21:20.911 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:20.911 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_encrypted_bytes": 3152, 2024-03-18T16:21:20.911 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_encrypted_bytes": 472080 2024-03-18T16:21:20.911 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:20.911 INFO:teuthology.orchestra.run.smithi194.stdout: "cct": { 2024-03-18T16:21:20.911 INFO:teuthology.orchestra.run.smithi194.stdout: "total_workers": 1, 2024-03-18T16:21:20.911 INFO:teuthology.orchestra.run.smithi194.stdout: "unhealthy_workers": 0 2024-03-18T16:21:20.912 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:20.912 INFO:teuthology.orchestra.run.smithi194.stdout: "finisher-MDSRank": { 2024-03-18T16:21:20.912 INFO:teuthology.orchestra.run.smithi194.stdout: "queue_len": 0, 2024-03-18T16:21:20.912 INFO:teuthology.orchestra.run.smithi194.stdout: "complete_latency": { 2024-03-18T16:21:20.912 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 1225, 2024-03-18T16:21:20.912 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.418343346, 2024-03-18T16:21:20.912 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000341504 2024-03-18T16:21:20.912 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-03-18T16:21:20.912 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:20.912 INFO:teuthology.orchestra.run.smithi194.stdout: "finisher-PurgeQueue": { 2024-03-18T16:21:20.912 INFO:teuthology.orchestra.run.smithi194.stdout: "queue_len": 0, 2024-03-18T16:21:20.912 INFO:teuthology.orchestra.run.smithi194.stdout: "complete_latency": { 2024-03-18T16:21:20.912 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 827, 2024-03-18T16:21:20.912 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.028612364, 2024-03-18T16:21:20.912 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000034597 2024-03-18T16:21:20.913 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-03-18T16:21:20.913 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:20.913 INFO:teuthology.orchestra.run.smithi194.stdout: "mds": { 2024-03-18T16:21:20.913 INFO:teuthology.orchestra.run.smithi194.stdout: "request": 121, 2024-03-18T16:21:20.913 INFO:teuthology.orchestra.run.smithi194.stdout: "reply": 114, 2024-03-18T16:21:20.913 INFO:teuthology.orchestra.run.smithi194.stdout: "reply_latency": { 2024-03-18T16:21:20.913 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 114, 2024-03-18T16:21:20.913 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.110344727, 2024-03-18T16:21:20.913 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000967936 2024-03-18T16:21:20.913 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:20.913 INFO:teuthology.orchestra.run.smithi194.stdout: "slow_reply": 0, 2024-03-18T16:21:20.913 INFO:teuthology.orchestra.run.smithi194.stdout: "forward": 7, 2024-03-18T16:21:20.913 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_fetch_complete": 0, 2024-03-18T16:21:20.913 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_fetch_keys": 0, 2024-03-18T16:21:20.913 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_commit": 11, 2024-03-18T16:21:20.914 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_split": 0, 2024-03-18T16:21:20.914 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_merge": 0, 2024-03-18T16:21:20.914 INFO:teuthology.orchestra.run.smithi194.stdout: "inodes": 17, 2024-03-18T16:21:20.914 INFO:teuthology.orchestra.run.smithi194.stdout: "inodes_top": 1, 2024-03-18T16:21:20.914 INFO:teuthology.orchestra.run.smithi194.stdout: "inodes_bottom": 2, 2024-03-18T16:21:20.914 INFO:teuthology.orchestra.run.smithi194.stdout: "inodes_pin_tail": 14, 2024-03-18T16:21:20.914 INFO:teuthology.orchestra.run.smithi194.stdout: "inodes_pinned": 14, 2024-03-18T16:21:20.914 INFO:teuthology.orchestra.run.smithi194.stdout: "inodes_expired": 328, 2024-03-18T16:21:20.914 INFO:teuthology.orchestra.run.smithi194.stdout: "inodes_with_caps": 1, 2024-03-18T16:21:20.914 INFO:teuthology.orchestra.run.smithi194.stdout: "caps": 1, 2024-03-18T16:21:20.914 INFO:teuthology.orchestra.run.smithi194.stdout: "subtrees": 5, 2024-03-18T16:21:20.914 INFO:teuthology.orchestra.run.smithi194.stdout: "traverse": 254, 2024-03-18T16:21:20.914 INFO:teuthology.orchestra.run.smithi194.stdout: "traverse_hit": 175, 2024-03-18T16:21:20.914 INFO:teuthology.orchestra.run.smithi194.stdout: "traverse_forward": 6, 2024-03-18T16:21:20.915 INFO:teuthology.orchestra.run.smithi194.stdout: "traverse_discover": 10, 2024-03-18T16:21:20.915 INFO:teuthology.orchestra.run.smithi194.stdout: "traverse_dir_fetch": 0, 2024-03-18T16:21:20.915 INFO:teuthology.orchestra.run.smithi194.stdout: "traverse_remote_ino": 0, 2024-03-18T16:21:20.915 INFO:teuthology.orchestra.run.smithi194.stdout: "traverse_lock": 0, 2024-03-18T16:21:20.915 INFO:teuthology.orchestra.run.smithi194.stdout: "load_cent": 0, 2024-03-18T16:21:20.915 INFO:teuthology.orchestra.run.smithi194.stdout: "q": 0, 2024-03-18T16:21:20.915 INFO:teuthology.orchestra.run.smithi194.stdout: "exported": 66, 2024-03-18T16:21:20.915 INFO:teuthology.orchestra.run.smithi194.stdout: "exported_inodes": 296, 2024-03-18T16:21:20.915 INFO:teuthology.orchestra.run.smithi194.stdout: "imported": 6, 2024-03-18T16:21:20.915 INFO:teuthology.orchestra.run.smithi194.stdout: "imported_inodes": 296, 2024-03-18T16:21:20.915 INFO:teuthology.orchestra.run.smithi194.stdout: "openino_dir_fetch": 0, 2024-03-18T16:21:20.915 INFO:teuthology.orchestra.run.smithi194.stdout: "openino_backtrace_fetch": 734, 2024-03-18T16:21:20.915 INFO:teuthology.orchestra.run.smithi194.stdout: "openino_peer_discover": 0, 2024-03-18T16:21:20.915 INFO:teuthology.orchestra.run.smithi194.stdout: "root_rfiles": 112, 2024-03-18T16:21:20.915 INFO:teuthology.orchestra.run.smithi194.stdout: "root_rbytes": 933339, 2024-03-18T16:21:20.916 INFO:teuthology.orchestra.run.smithi194.stdout: "root_rsnaps": 0, 2024-03-18T16:21:20.916 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_backtrace_fetch": 734, 2024-03-18T16:21:20.916 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_set_tag": 0, 2024-03-18T16:21:20.916 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_backtrace_repaired": 0, 2024-03-18T16:21:20.916 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_inotable_repaired": 0, 2024-03-18T16:21:20.916 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_dir_inodes": 160, 2024-03-18T16:21:20.916 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_dir_base_inodes": 0, 2024-03-18T16:21:20.916 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_dirfrag_rstats": 160, 2024-03-18T16:21:20.916 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_file_inodes": 574, 2024-03-18T16:21:20.916 INFO:teuthology.orchestra.run.smithi194.stdout: "handle_inode_file_caps": 0, 2024-03-18T16:21:20.916 INFO:teuthology.orchestra.run.smithi194.stdout: "ceph_cap_op_revoke": 66, 2024-03-18T16:21:20.916 INFO:teuthology.orchestra.run.smithi194.stdout: "ceph_cap_op_grant": 344, 2024-03-18T16:21:20.916 INFO:teuthology.orchestra.run.smithi194.stdout: "ceph_cap_op_trunc": 0, 2024-03-18T16:21:20.917 INFO:teuthology.orchestra.run.smithi194.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-18T16:21:20.917 INFO:teuthology.orchestra.run.smithi194.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-18T16:21:20.917 INFO:teuthology.orchestra.run.smithi194.stdout: "handle_client_caps": 996, 2024-03-18T16:21:20.917 INFO:teuthology.orchestra.run.smithi194.stdout: "handle_client_caps_dirty": 673, 2024-03-18T16:21:20.917 INFO:teuthology.orchestra.run.smithi194.stdout: "handle_client_cap_release": 11, 2024-03-18T16:21:20.917 INFO:teuthology.orchestra.run.smithi194.stdout: "process_request_cap_release": 65 2024-03-18T16:21:20.917 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:20.917 INFO:teuthology.orchestra.run.smithi194.stdout: "mds_cache": { 2024-03-18T16:21:20.917 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_update": 0, 2024-03-18T16:21:20.917 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_update_receipt": 3, 2024-03-18T16:21:20.917 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_try_discover": 2, 2024-03-18T16:21:20.917 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_send_discover": 13, 2024-03-18T16:21:20.917 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_handle_discover": 78, 2024-03-18T16:21:20.917 INFO:teuthology.orchestra.run.smithi194.stdout: "num_strays": 0, 2024-03-18T16:21:20.918 INFO:teuthology.orchestra.run.smithi194.stdout: "num_strays_delayed": 0, 2024-03-18T16:21:20.918 INFO:teuthology.orchestra.run.smithi194.stdout: "num_strays_enqueuing": 0, 2024-03-18T16:21:20.918 INFO:teuthology.orchestra.run.smithi194.stdout: "strays_created": 31, 2024-03-18T16:21:20.918 INFO:teuthology.orchestra.run.smithi194.stdout: "strays_enqueued": 31, 2024-03-18T16:21:20.918 INFO:teuthology.orchestra.run.smithi194.stdout: "strays_reintegrated": 0, 2024-03-18T16:21:20.918 INFO:teuthology.orchestra.run.smithi194.stdout: "strays_migrated": 0, 2024-03-18T16:21:20.918 INFO:teuthology.orchestra.run.smithi194.stdout: "num_recovering_processing": 0, 2024-03-18T16:21:20.918 INFO:teuthology.orchestra.run.smithi194.stdout: "num_recovering_enqueued": 0, 2024-03-18T16:21:20.918 INFO:teuthology.orchestra.run.smithi194.stdout: "num_recovering_prioritized": 0, 2024-03-18T16:21:20.918 INFO:teuthology.orchestra.run.smithi194.stdout: "recovery_started": 0, 2024-03-18T16:21:20.918 INFO:teuthology.orchestra.run.smithi194.stdout: "recovery_completed": 0, 2024-03-18T16:21:20.918 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_quiesce_path": 0, 2024-03-18T16:21:20.918 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_quiesce_inode": 0, 2024-03-18T16:21:20.918 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_enqueue_scrub": 0, 2024-03-18T16:21:20.918 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_exportdir": 66, 2024-03-18T16:21:20.919 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_flush": 0, 2024-03-18T16:21:20.919 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_fragmentdir": 0, 2024-03-18T16:21:20.919 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_fragstats": 0, 2024-03-18T16:21:20.919 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_inodestats": 0 2024-03-18T16:21:20.919 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:20.919 INFO:teuthology.orchestra.run.smithi194.stdout: "mds_log": { 2024-03-18T16:21:20.919 INFO:teuthology.orchestra.run.smithi194.stdout: "evlrg": 0, 2024-03-18T16:21:20.919 INFO:teuthology.orchestra.run.smithi194.stdout: "evadd": 1158, 2024-03-18T16:21:20.919 INFO:teuthology.orchestra.run.smithi194.stdout: "evex": 0, 2024-03-18T16:21:20.919 INFO:teuthology.orchestra.run.smithi194.stdout: "evtrm": 0, 2024-03-18T16:21:20.919 INFO:teuthology.orchestra.run.smithi194.stdout: "ev": 1158, 2024-03-18T16:21:20.919 INFO:teuthology.orchestra.run.smithi194.stdout: "evexg": 0, 2024-03-18T16:21:20.919 INFO:teuthology.orchestra.run.smithi194.stdout: "evexd": 0, 2024-03-18T16:21:20.919 INFO:teuthology.orchestra.run.smithi194.stdout: "segadd": 3, 2024-03-18T16:21:20.920 INFO:teuthology.orchestra.run.smithi194.stdout: "segex": 0, 2024-03-18T16:21:20.920 INFO:teuthology.orchestra.run.smithi194.stdout: "segtrm": 0, 2024-03-18T16:21:20.920 INFO:teuthology.orchestra.run.smithi194.stdout: "seg": 3, 2024-03-18T16:21:20.920 INFO:teuthology.orchestra.run.smithi194.stdout: "segmjr": 2, 2024-03-18T16:21:20.920 INFO:teuthology.orchestra.run.smithi194.stdout: "segexg": 0, 2024-03-18T16:21:20.920 INFO:teuthology.orchestra.run.smithi194.stdout: "segexd": 0, 2024-03-18T16:21:20.920 INFO:teuthology.orchestra.run.smithi194.stdout: "expos": 4194304, 2024-03-18T16:21:20.920 INFO:teuthology.orchestra.run.smithi194.stdout: "wrpos": 6419165, 2024-03-18T16:21:20.920 INFO:teuthology.orchestra.run.smithi194.stdout: "rdpos": 0, 2024-03-18T16:21:20.920 INFO:teuthology.orchestra.run.smithi194.stdout: "jlat": { 2024-03-18T16:21:20.920 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 222, 2024-03-18T16:21:20.920 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 19.535587344, 2024-03-18T16:21:20.920 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.087998141 2024-03-18T16:21:20.920 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:20.920 INFO:teuthology.orchestra.run.smithi194.stdout: "replayed": 0 2024-03-18T16:21:20.920 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:20.921 INFO:teuthology.orchestra.run.smithi194.stdout: "mds_mem": { 2024-03-18T16:21:20.921 INFO:teuthology.orchestra.run.smithi194.stdout: "ino": 22, 2024-03-18T16:21:20.921 INFO:teuthology.orchestra.run.smithi194.stdout: "ino+": 237, 2024-03-18T16:21:20.921 INFO:teuthology.orchestra.run.smithi194.stdout: "ino-": 215, 2024-03-18T16:21:20.921 INFO:teuthology.orchestra.run.smithi194.stdout: "dir": 21, 2024-03-18T16:21:20.921 INFO:teuthology.orchestra.run.smithi194.stdout: "dir+": 103, 2024-03-18T16:21:20.921 INFO:teuthology.orchestra.run.smithi194.stdout: "dir-": 82, 2024-03-18T16:21:20.921 INFO:teuthology.orchestra.run.smithi194.stdout: "dn": 17, 2024-03-18T16:21:20.921 INFO:teuthology.orchestra.run.smithi194.stdout: "dn+": 376, 2024-03-18T16:21:20.921 INFO:teuthology.orchestra.run.smithi194.stdout: "dn-": 359, 2024-03-18T16:21:20.921 INFO:teuthology.orchestra.run.smithi194.stdout: "cap": 1, 2024-03-18T16:21:20.921 INFO:teuthology.orchestra.run.smithi194.stdout: "cap+": 232, 2024-03-18T16:21:20.921 INFO:teuthology.orchestra.run.smithi194.stdout: "cap-": 231, 2024-03-18T16:21:20.921 INFO:teuthology.orchestra.run.smithi194.stdout: "rss": 55284, 2024-03-18T16:21:20.921 INFO:teuthology.orchestra.run.smithi194.stdout: "heap": 223612 2024-03-18T16:21:20.922 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:20.922 INFO:teuthology.orchestra.run.smithi194.stdout: "mds_server": { 2024-03-18T16:21:20.922 INFO:teuthology.orchestra.run.smithi194.stdout: "dispatch_client_request": 246, 2024-03-18T16:21:20.922 INFO:teuthology.orchestra.run.smithi194.stdout: "dispatch_server_request": 2, 2024-03-18T16:21:20.922 INFO:teuthology.orchestra.run.smithi194.stdout: "handle_client_request": 121, 2024-03-18T16:21:20.922 INFO:teuthology.orchestra.run.smithi194.stdout: "handle_client_session": 68, 2024-03-18T16:21:20.922 INFO:teuthology.orchestra.run.smithi194.stdout: "handle_peer_request": 41, 2024-03-18T16:21:20.922 INFO:teuthology.orchestra.run.smithi194.stdout: "req_create_latency": { 2024-03-18T16:21:20.922 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:20.922 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:20.922 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:20.922 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:20.922 INFO:teuthology.orchestra.run.smithi194.stdout: "req_getattr_latency": { 2024-03-18T16:21:20.922 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 1, 2024-03-18T16:21:20.922 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.001008353, 2024-03-18T16:21:20.923 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.001008353 2024-03-18T16:21:20.923 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:20.923 INFO:teuthology.orchestra.run.smithi194.stdout: "req_getfilelock_latency": { 2024-03-18T16:21:20.923 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:20.923 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:20.923 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:20.923 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:20.923 INFO:teuthology.orchestra.run.smithi194.stdout: "req_link_latency": { 2024-03-18T16:21:20.923 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:20.923 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:20.923 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:20.923 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:20.923 INFO:teuthology.orchestra.run.smithi194.stdout: "req_lookup_latency": { 2024-03-18T16:21:20.923 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 58, 2024-03-18T16:21:20.923 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.022662550, 2024-03-18T16:21:20.924 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000390733 2024-03-18T16:21:20.924 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:20.924 INFO:teuthology.orchestra.run.smithi194.stdout: "req_lookuphash_latency": { 2024-03-18T16:21:20.924 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:20.924 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:20.924 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:20.924 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:20.924 INFO:teuthology.orchestra.run.smithi194.stdout: "req_lookupino_latency": { 2024-03-18T16:21:20.924 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:20.924 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:20.924 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:20.924 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:20.924 INFO:teuthology.orchestra.run.smithi194.stdout: "req_lookupname_latency": { 2024-03-18T16:21:20.924 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:20.924 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:20.925 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:20.925 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:20.925 INFO:teuthology.orchestra.run.smithi194.stdout: "req_lookupparent_latency": { 2024-03-18T16:21:20.925 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:20.925 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:20.925 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:20.925 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:20.925 INFO:teuthology.orchestra.run.smithi194.stdout: "req_lookupsnap_latency": { 2024-03-18T16:21:20.925 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:20.925 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:20.925 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:20.925 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:20.925 INFO:teuthology.orchestra.run.smithi194.stdout: "req_lssnap_latency": { 2024-03-18T16:21:20.925 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:20.925 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:20.926 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:20.926 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:20.926 INFO:teuthology.orchestra.run.smithi194.stdout: "req_mkdir_latency": { 2024-03-18T16:21:20.926 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:20.926 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:20.927 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:20.927 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:20.927 INFO:teuthology.orchestra.run.smithi194.stdout: "req_mknod_latency": { 2024-03-18T16:21:20.927 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:20.927 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:20.927 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:20.927 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:20.927 INFO:teuthology.orchestra.run.smithi194.stdout: "req_mksnap_latency": { 2024-03-18T16:21:20.927 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:20.927 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:20.927 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:20.927 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:20.927 INFO:teuthology.orchestra.run.smithi194.stdout: "req_open_latency": { 2024-03-18T16:21:20.927 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 22, 2024-03-18T16:21:20.927 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.017275149, 2024-03-18T16:21:20.928 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000785234 2024-03-18T16:21:20.928 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:20.928 INFO:teuthology.orchestra.run.smithi194.stdout: "req_readdir_latency": { 2024-03-18T16:21:20.928 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 2, 2024-03-18T16:21:20.928 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.001051360, 2024-03-18T16:21:20.928 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000525680 2024-03-18T16:21:20.928 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:20.928 INFO:teuthology.orchestra.run.smithi194.stdout: "req_rename_latency": { 2024-03-18T16:21:20.928 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:20.928 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:20.928 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:20.928 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:20.928 INFO:teuthology.orchestra.run.smithi194.stdout: "req_renamesnap_latency": { 2024-03-18T16:21:20.928 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:20.929 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:20.929 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:20.929 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:20.929 INFO:teuthology.orchestra.run.smithi194.stdout: "req_snapdiff_latency": { 2024-03-18T16:21:20.929 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:20.929 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:20.929 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:20.929 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:20.929 INFO:teuthology.orchestra.run.smithi194.stdout: "req_rmdir_latency": { 2024-03-18T16:21:20.929 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:20.929 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:20.929 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:20.929 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:20.929 INFO:teuthology.orchestra.run.smithi194.stdout: "req_rmsnap_latency": { 2024-03-18T16:21:20.929 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:20.930 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:20.930 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:20.930 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:20.930 INFO:teuthology.orchestra.run.smithi194.stdout: "req_rmxattr_latency": { 2024-03-18T16:21:20.930 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:20.930 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:20.930 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:20.930 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:20.930 INFO:teuthology.orchestra.run.smithi194.stdout: "req_setattr_latency": { 2024-03-18T16:21:20.930 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:20.930 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:20.930 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:20.930 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:20.930 INFO:teuthology.orchestra.run.smithi194.stdout: "req_setdirlayout_latency": { 2024-03-18T16:21:20.930 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:20.931 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:20.931 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:20.931 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:20.931 INFO:teuthology.orchestra.run.smithi194.stdout: "req_setfilelock_latency": { 2024-03-18T16:21:20.931 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:20.931 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:20.931 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:20.931 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:20.931 INFO:teuthology.orchestra.run.smithi194.stdout: "req_setlayout_latency": { 2024-03-18T16:21:20.931 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:20.931 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:20.931 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:20.931 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:20.931 INFO:teuthology.orchestra.run.smithi194.stdout: "req_setxattr_latency": { 2024-03-18T16:21:20.931 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:20.932 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:20.932 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:20.932 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:20.932 INFO:teuthology.orchestra.run.smithi194.stdout: "req_symlink_latency": { 2024-03-18T16:21:20.932 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:20.932 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:20.932 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:20.932 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:20.932 INFO:teuthology.orchestra.run.smithi194.stdout: "req_unlink_latency": { 2024-03-18T16:21:20.932 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 31, 2024-03-18T16:21:20.932 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.068347315, 2024-03-18T16:21:20.932 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.002204752 2024-03-18T16:21:20.932 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:20.932 INFO:teuthology.orchestra.run.smithi194.stdout: "cap_revoke_eviction": 0, 2024-03-18T16:21:20.932 INFO:teuthology.orchestra.run.smithi194.stdout: "cap_acquisition_throttle": 0, 2024-03-18T16:21:20.933 INFO:teuthology.orchestra.run.smithi194.stdout: "req_getvxattr_latency": { 2024-03-18T16:21:20.933 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:20.933 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:20.933 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:20.933 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-03-18T16:21:20.933 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:20.933 INFO:teuthology.orchestra.run.smithi194.stdout: "mds_sessions": { 2024-03-18T16:21:20.933 INFO:teuthology.orchestra.run.smithi194.stdout: "session_count": 1, 2024-03-18T16:21:20.933 INFO:teuthology.orchestra.run.smithi194.stdout: "session_add": 1, 2024-03-18T16:21:20.933 INFO:teuthology.orchestra.run.smithi194.stdout: "session_remove": 0, 2024-03-18T16:21:20.933 INFO:teuthology.orchestra.run.smithi194.stdout: "sessions_open": 1, 2024-03-18T16:21:20.933 INFO:teuthology.orchestra.run.smithi194.stdout: "sessions_stale": 0, 2024-03-18T16:21:20.933 INFO:teuthology.orchestra.run.smithi194.stdout: "total_load": 66, 2024-03-18T16:21:20.933 INFO:teuthology.orchestra.run.smithi194.stdout: "average_load": 66, 2024-03-18T16:21:20.933 INFO:teuthology.orchestra.run.smithi194.stdout: "avg_session_uptime": 684, 2024-03-18T16:21:20.934 INFO:teuthology.orchestra.run.smithi194.stdout: "mdthresh_evicted": 0 2024-03-18T16:21:20.934 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:20.934 INFO:teuthology.orchestra.run.smithi194.stdout: "mempool": { 2024-03-18T16:21:20.934 INFO:teuthology.orchestra.run.smithi194.stdout: "bloom_filter_bytes": 0, 2024-03-18T16:21:20.934 INFO:teuthology.orchestra.run.smithi194.stdout: "bloom_filter_items": 0, 2024-03-18T16:21:20.934 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T16:21:20.934 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_alloc_items": 0, 2024-03-18T16:21:20.934 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T16:21:20.934 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_data_items": 0, 2024-03-18T16:21:20.934 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T16:21:20.934 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T16:21:20.934 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T16:21:20.934 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T16:21:20.934 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T16:21:20.935 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_other_items": 0, 2024-03-18T16:21:20.935 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T16:21:20.935 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T16:21:20.935 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_extent_bytes": 0, 2024-03-18T16:21:20.935 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_extent_items": 0, 2024-03-18T16:21:20.935 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_blob_bytes": 0, 2024-03-18T16:21:20.935 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_blob_items": 0, 2024-03-18T16:21:20.935 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T16:21:20.935 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T16:21:20.935 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T16:21:20.935 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T16:21:20.935 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T16:21:20.935 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_fsck_items": 0, 2024-03-18T16:21:20.935 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_txc_bytes": 0, 2024-03-18T16:21:20.935 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_txc_items": 0, 2024-03-18T16:21:20.935 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T16:21:20.936 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T16:21:20.936 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_writing_bytes": 0, 2024-03-18T16:21:20.936 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_writing_items": 0, 2024-03-18T16:21:20.936 INFO:teuthology.orchestra.run.smithi194.stdout: "bluefs_bytes": 0, 2024-03-18T16:21:20.936 INFO:teuthology.orchestra.run.smithi194.stdout: "bluefs_items": 0, 2024-03-18T16:21:20.936 INFO:teuthology.orchestra.run.smithi194.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T16:21:20.936 INFO:teuthology.orchestra.run.smithi194.stdout: "bluefs_file_reader_items": 0, 2024-03-18T16:21:20.936 INFO:teuthology.orchestra.run.smithi194.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T16:21:20.936 INFO:teuthology.orchestra.run.smithi194.stdout: "bluefs_file_writer_items": 0, 2024-03-18T16:21:20.936 INFO:teuthology.orchestra.run.smithi194.stdout: "buffer_anon_bytes": 292839, 2024-03-18T16:21:20.936 INFO:teuthology.orchestra.run.smithi194.stdout: "buffer_anon_items": 121, 2024-03-18T16:21:20.936 INFO:teuthology.orchestra.run.smithi194.stdout: "buffer_meta_bytes": 0, 2024-03-18T16:21:20.936 INFO:teuthology.orchestra.run.smithi194.stdout: "buffer_meta_items": 0, 2024-03-18T16:21:20.936 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_bytes": 0, 2024-03-18T16:21:20.936 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_items": 0, 2024-03-18T16:21:20.937 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_mapbl_bytes": 0, 2024-03-18T16:21:20.937 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_mapbl_items": 0, 2024-03-18T16:21:20.937 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_pglog_bytes": 0, 2024-03-18T16:21:20.937 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_pglog_items": 0, 2024-03-18T16:21:20.937 INFO:teuthology.orchestra.run.smithi194.stdout: "osdmap_bytes": 8888, 2024-03-18T16:21:20.937 INFO:teuthology.orchestra.run.smithi194.stdout: "osdmap_items": 170, 2024-03-18T16:21:20.937 INFO:teuthology.orchestra.run.smithi194.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T16:21:20.937 INFO:teuthology.orchestra.run.smithi194.stdout: "osdmap_mapping_items": 0, 2024-03-18T16:21:20.937 INFO:teuthology.orchestra.run.smithi194.stdout: "pgmap_bytes": 0, 2024-03-18T16:21:20.937 INFO:teuthology.orchestra.run.smithi194.stdout: "pgmap_items": 0, 2024-03-18T16:21:20.937 INFO:teuthology.orchestra.run.smithi194.stdout: "mds_co_bytes": 88957, 2024-03-18T16:21:20.937 INFO:teuthology.orchestra.run.smithi194.stdout: "mds_co_items": 368, 2024-03-18T16:21:20.937 INFO:teuthology.orchestra.run.smithi194.stdout: "unittest_1_bytes": 0, 2024-03-18T16:21:20.937 INFO:teuthology.orchestra.run.smithi194.stdout: "unittest_1_items": 0, 2024-03-18T16:21:20.937 INFO:teuthology.orchestra.run.smithi194.stdout: "unittest_2_bytes": 0, 2024-03-18T16:21:20.938 INFO:teuthology.orchestra.run.smithi194.stdout: "unittest_2_items": 0 2024-03-18T16:21:20.938 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:20.938 INFO:teuthology.orchestra.run.smithi194.stdout: "objecter": { 2024-03-18T16:21:20.938 INFO:teuthology.orchestra.run.smithi194.stdout: "op_active": 0, 2024-03-18T16:21:20.938 INFO:teuthology.orchestra.run.smithi194.stdout: "op_laggy": 0, 2024-03-18T16:21:20.938 INFO:teuthology.orchestra.run.smithi194.stdout: "op_send": 1931, 2024-03-18T16:21:20.938 INFO:teuthology.orchestra.run.smithi194.stdout: "op_send_bytes": 2270695, 2024-03-18T16:21:20.938 INFO:teuthology.orchestra.run.smithi194.stdout: "op_resend": 33, 2024-03-18T16:21:20.938 INFO:teuthology.orchestra.run.smithi194.stdout: "op_reply": 1931, 2024-03-18T16:21:20.938 INFO:teuthology.orchestra.run.smithi194.stdout: "op_latency": { 2024-03-18T16:21:20.938 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 1931, 2024-03-18T16:21:20.938 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 265.025669191, 2024-03-18T16:21:20.938 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.137247886 2024-03-18T16:21:20.938 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:20.938 INFO:teuthology.orchestra.run.smithi194.stdout: "op_inflight": 0, 2024-03-18T16:21:20.939 INFO:teuthology.orchestra.run.smithi194.stdout: "oplen_avg": { 2024-03-18T16:21:20.939 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 1931, 2024-03-18T16:21:20.939 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 2054 2024-03-18T16:21:20.939 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:20.939 INFO:teuthology.orchestra.run.smithi194.stdout: "op": 1931, 2024-03-18T16:21:20.939 INFO:teuthology.orchestra.run.smithi194.stdout: "op_r": 735, 2024-03-18T16:21:20.939 INFO:teuthology.orchestra.run.smithi194.stdout: "op_w": 1196, 2024-03-18T16:21:20.939 INFO:teuthology.orchestra.run.smithi194.stdout: "op_rmw": 0, 2024-03-18T16:21:20.939 INFO:teuthology.orchestra.run.smithi194.stdout: "op_pg": 0, 2024-03-18T16:21:20.939 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_stat": 0, 2024-03-18T16:21:20.939 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_create": 10, 2024-03-18T16:21:20.939 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_read": 1, 2024-03-18T16:21:20.939 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_write": 223, 2024-03-18T16:21:20.939 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_writefull": 36, 2024-03-18T16:21:20.939 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_writesame": 0, 2024-03-18T16:21:20.940 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_append": 0, 2024-03-18T16:21:20.940 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_zero": 0, 2024-03-18T16:21:20.940 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_truncate": 0, 2024-03-18T16:21:20.940 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_delete": 818, 2024-03-18T16:21:20.940 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_mapext": 0, 2024-03-18T16:21:20.940 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_sparse_read": 0, 2024-03-18T16:21:20.940 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_clonerange": 0, 2024-03-18T16:21:20.940 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_getxattr": 734, 2024-03-18T16:21:20.940 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_setxattr": 20, 2024-03-18T16:21:20.940 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_cmpxattr": 0, 2024-03-18T16:21:20.940 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_rmxattr": 0, 2024-03-18T16:21:20.940 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_resetxattrs": 0, 2024-03-18T16:21:20.940 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_call": 0, 2024-03-18T16:21:20.940 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_watch": 0, 2024-03-18T16:21:20.940 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_notify": 0, 2024-03-18T16:21:20.941 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_src_cmpxattr": 0, 2024-03-18T16:21:20.941 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_pgls": 0, 2024-03-18T16:21:20.941 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_pgls_filter": 0, 2024-03-18T16:21:20.941 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_other": 0, 2024-03-18T16:21:20.941 INFO:teuthology.orchestra.run.smithi194.stdout: "linger_active": 0, 2024-03-18T16:21:20.941 INFO:teuthology.orchestra.run.smithi194.stdout: "linger_send": 0, 2024-03-18T16:21:20.941 INFO:teuthology.orchestra.run.smithi194.stdout: "linger_resend": 0, 2024-03-18T16:21:20.941 INFO:teuthology.orchestra.run.smithi194.stdout: "linger_ping": 0, 2024-03-18T16:21:20.941 INFO:teuthology.orchestra.run.smithi194.stdout: "poolop_active": 0, 2024-03-18T16:21:20.941 INFO:teuthology.orchestra.run.smithi194.stdout: "poolop_send": 0, 2024-03-18T16:21:20.941 INFO:teuthology.orchestra.run.smithi194.stdout: "poolop_resend": 0, 2024-03-18T16:21:20.941 INFO:teuthology.orchestra.run.smithi194.stdout: "poolstat_active": 0, 2024-03-18T16:21:20.941 INFO:teuthology.orchestra.run.smithi194.stdout: "poolstat_send": 0, 2024-03-18T16:21:20.941 INFO:teuthology.orchestra.run.smithi194.stdout: "poolstat_resend": 0, 2024-03-18T16:21:20.941 INFO:teuthology.orchestra.run.smithi194.stdout: "statfs_active": 0, 2024-03-18T16:21:21.044 INFO:teuthology.orchestra.run.smithi194.stdout: "statfs_send": 0, 2024-03-18T16:21:21.044 INFO:teuthology.orchestra.run.smithi194.stdout: "statfs_resend": 0, 2024-03-18T16:21:21.044 INFO:teuthology.orchestra.run.smithi194.stdout: "command_active": 0, 2024-03-18T16:21:21.044 INFO:teuthology.orchestra.run.smithi194.stdout: "command_send": 0, 2024-03-18T16:21:21.044 INFO:teuthology.orchestra.run.smithi194.stdout: "command_resend": 0, 2024-03-18T16:21:21.044 INFO:teuthology.orchestra.run.smithi194.stdout: "map_epoch": 93, 2024-03-18T16:21:21.044 INFO:teuthology.orchestra.run.smithi194.stdout: "map_full": 0, 2024-03-18T16:21:21.044 INFO:teuthology.orchestra.run.smithi194.stdout: "map_inc": 4, 2024-03-18T16:21:21.044 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_sessions": 12, 2024-03-18T16:21:21.044 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_session_open": 12, 2024-03-18T16:21:21.044 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_session_close": 0, 2024-03-18T16:21:21.044 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_laggy": 0, 2024-03-18T16:21:21.044 INFO:teuthology.orchestra.run.smithi194.stdout: "omap_wr": 137, 2024-03-18T16:21:21.045 INFO:teuthology.orchestra.run.smithi194.stdout: "omap_rd": 0, 2024-03-18T16:21:21.045 INFO:teuthology.orchestra.run.smithi194.stdout: "omap_del": 75 2024-03-18T16:21:21.045 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:21.045 INFO:teuthology.orchestra.run.smithi194.stdout: "oft": { 2024-03-18T16:21:21.045 INFO:teuthology.orchestra.run.smithi194.stdout: "omap_total_objs": 1, 2024-03-18T16:21:21.045 INFO:teuthology.orchestra.run.smithi194.stdout: "omap_total_kv_pairs": 0, 2024-03-18T16:21:21.045 INFO:teuthology.orchestra.run.smithi194.stdout: "omap_total_updates": 0, 2024-03-18T16:21:21.045 INFO:teuthology.orchestra.run.smithi194.stdout: "omap_total_removes": 0 2024-03-18T16:21:21.045 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:21.045 INFO:teuthology.orchestra.run.smithi194.stdout: "osd-slow-ops": { 2024-03-18T16:21:21.045 INFO:teuthology.orchestra.run.smithi194.stdout: "slow_ops_count": 0 2024-03-18T16:21:21.045 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:21.045 INFO:teuthology.orchestra.run.smithi194.stdout: "purge_queue": { 2024-03-18T16:21:21.045 INFO:teuthology.orchestra.run.smithi194.stdout: "pq_executing_ops": 0, 2024-03-18T16:21:21.046 INFO:teuthology.orchestra.run.smithi194.stdout: "pq_executing_ops_high_water": 26, 2024-03-18T16:21:21.046 INFO:teuthology.orchestra.run.smithi194.stdout: "pq_executing": 0, 2024-03-18T16:21:21.046 INFO:teuthology.orchestra.run.smithi194.stdout: "pq_executing_high_water": 1, 2024-03-18T16:21:21.046 INFO:teuthology.orchestra.run.smithi194.stdout: "pq_executed_ops": 806, 2024-03-18T16:21:21.046 INFO:teuthology.orchestra.run.smithi194.stdout: "pq_executed": 31, 2024-03-18T16:21:21.046 INFO:teuthology.orchestra.run.smithi194.stdout: "pq_item_in_journal": 0 2024-03-18T16:21:21.046 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:21.046 INFO:teuthology.orchestra.run.smithi194.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T16:21:21.046 INFO:teuthology.orchestra.run.smithi194.stdout: "val": 0, 2024-03-18T16:21:21.046 INFO:teuthology.orchestra.run.smithi194.stdout: "max": 104857600, 2024-03-18T16:21:21.046 INFO:teuthology.orchestra.run.smithi194.stdout: "get_started": 0, 2024-03-18T16:21:21.046 INFO:teuthology.orchestra.run.smithi194.stdout: "get": 6588, 2024-03-18T16:21:21.046 INFO:teuthology.orchestra.run.smithi194.stdout: "get_sum": 2523394, 2024-03-18T16:21:21.046 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_fail": 0, 2024-03-18T16:21:21.046 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_success": 6588, 2024-03-18T16:21:21.047 INFO:teuthology.orchestra.run.smithi194.stdout: "take": 0, 2024-03-18T16:21:21.047 INFO:teuthology.orchestra.run.smithi194.stdout: "take_sum": 0, 2024-03-18T16:21:21.047 INFO:teuthology.orchestra.run.smithi194.stdout: "put": 6588, 2024-03-18T16:21:21.047 INFO:teuthology.orchestra.run.smithi194.stdout: "put_sum": 2523394, 2024-03-18T16:21:21.047 INFO:teuthology.orchestra.run.smithi194.stdout: "wait": { 2024-03-18T16:21:21.047 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:21.047 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:21.047 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:21.047 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-03-18T16:21:21.047 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:21.047 INFO:teuthology.orchestra.run.smithi194.stdout: "throttle-objecter_bytes": { 2024-03-18T16:21:21.047 INFO:teuthology.orchestra.run.smithi194.stdout: "val": 0, 2024-03-18T16:21:21.047 INFO:teuthology.orchestra.run.smithi194.stdout: "max": 104857600, 2024-03-18T16:21:21.047 INFO:teuthology.orchestra.run.smithi194.stdout: "get_started": 0, 2024-03-18T16:21:21.047 INFO:teuthology.orchestra.run.smithi194.stdout: "get": 0, 2024-03-18T16:21:21.048 INFO:teuthology.orchestra.run.smithi194.stdout: "get_sum": 0, 2024-03-18T16:21:21.048 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_fail": 0, 2024-03-18T16:21:21.048 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_success": 0, 2024-03-18T16:21:21.048 INFO:teuthology.orchestra.run.smithi194.stdout: "take": 1931, 2024-03-18T16:21:21.048 INFO:teuthology.orchestra.run.smithi194.stdout: "take_sum": 2274074, 2024-03-18T16:21:21.048 INFO:teuthology.orchestra.run.smithi194.stdout: "put": 1931, 2024-03-18T16:21:21.048 INFO:teuthology.orchestra.run.smithi194.stdout: "put_sum": 2274074, 2024-03-18T16:21:21.048 INFO:teuthology.orchestra.run.smithi194.stdout: "wait": { 2024-03-18T16:21:21.048 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:21.048 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:21.048 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:21.048 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-03-18T16:21:21.048 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:21.048 INFO:teuthology.orchestra.run.smithi194.stdout: "throttle-objecter_ops": { 2024-03-18T16:21:21.049 INFO:teuthology.orchestra.run.smithi194.stdout: "val": 0, 2024-03-18T16:21:21.049 INFO:teuthology.orchestra.run.smithi194.stdout: "max": 1024, 2024-03-18T16:21:21.049 INFO:teuthology.orchestra.run.smithi194.stdout: "get_started": 0, 2024-03-18T16:21:21.049 INFO:teuthology.orchestra.run.smithi194.stdout: "get": 0, 2024-03-18T16:21:21.049 INFO:teuthology.orchestra.run.smithi194.stdout: "get_sum": 0, 2024-03-18T16:21:21.049 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_fail": 0, 2024-03-18T16:21:21.049 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_success": 0, 2024-03-18T16:21:21.049 INFO:teuthology.orchestra.run.smithi194.stdout: "take": 1931, 2024-03-18T16:21:21.049 INFO:teuthology.orchestra.run.smithi194.stdout: "take_sum": 1931, 2024-03-18T16:21:21.049 INFO:teuthology.orchestra.run.smithi194.stdout: "put": 1931, 2024-03-18T16:21:21.049 INFO:teuthology.orchestra.run.smithi194.stdout: "put_sum": 1931, 2024-03-18T16:21:21.049 INFO:teuthology.orchestra.run.smithi194.stdout: "wait": { 2024-03-18T16:21:21.049 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:21.049 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:21.049 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:21.050 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-03-18T16:21:21.050 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:21.050 INFO:teuthology.orchestra.run.smithi194.stdout: "throttle-write_buf_throttle": { 2024-03-18T16:21:21.050 INFO:teuthology.orchestra.run.smithi194.stdout: "val": 0, 2024-03-18T16:21:21.050 INFO:teuthology.orchestra.run.smithi194.stdout: "max": 3758096384, 2024-03-18T16:21:21.050 INFO:teuthology.orchestra.run.smithi194.stdout: "get_started": 0, 2024-03-18T16:21:21.050 INFO:teuthology.orchestra.run.smithi194.stdout: "get": 31, 2024-03-18T16:21:21.050 INFO:teuthology.orchestra.run.smithi194.stdout: "get_sum": 3379, 2024-03-18T16:21:21.050 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_fail": 0, 2024-03-18T16:21:21.050 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_success": 31, 2024-03-18T16:21:21.050 INFO:teuthology.orchestra.run.smithi194.stdout: "take": 0, 2024-03-18T16:21:21.050 INFO:teuthology.orchestra.run.smithi194.stdout: "take_sum": 0, 2024-03-18T16:21:21.050 INFO:teuthology.orchestra.run.smithi194.stdout: "put": 1, 2024-03-18T16:21:21.050 INFO:teuthology.orchestra.run.smithi194.stdout: "put_sum": 3379, 2024-03-18T16:21:21.050 INFO:teuthology.orchestra.run.smithi194.stdout: "wait": { 2024-03-18T16:21:21.051 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:21.051 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:21.051 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:21.051 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-03-18T16:21:21.051 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:21.051 INFO:teuthology.orchestra.run.smithi194.stdout: "throttle-write_buf_throttle-0x5580153fc500": { 2024-03-18T16:21:21.051 INFO:teuthology.orchestra.run.smithi194.stdout: "val": 0, 2024-03-18T16:21:21.051 INFO:teuthology.orchestra.run.smithi194.stdout: "max": 3758096384, 2024-03-18T16:21:21.051 INFO:teuthology.orchestra.run.smithi194.stdout: "get_started": 0, 2024-03-18T16:21:21.051 INFO:teuthology.orchestra.run.smithi194.stdout: "get": 1158, 2024-03-18T16:21:21.051 INFO:teuthology.orchestra.run.smithi194.stdout: "get_sum": 2224861, 2024-03-18T16:21:21.051 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_fail": 0, 2024-03-18T16:21:21.051 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_success": 1158, 2024-03-18T16:21:21.051 INFO:teuthology.orchestra.run.smithi194.stdout: "take": 0, 2024-03-18T16:21:21.051 INFO:teuthology.orchestra.run.smithi194.stdout: "take_sum": 0, 2024-03-18T16:21:21.052 INFO:teuthology.orchestra.run.smithi194.stdout: "put": 222, 2024-03-18T16:21:21.052 INFO:teuthology.orchestra.run.smithi194.stdout: "put_sum": 2224861, 2024-03-18T16:21:21.052 INFO:teuthology.orchestra.run.smithi194.stdout: "wait": { 2024-03-18T16:21:21.052 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:21.052 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:21.052 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:21.052 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-03-18T16:21:21.052 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-03-18T16:21:21.052 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-03-18T16:21:21.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:20 smithi194 ceph-mon[36200]: pgmap v760: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-18T16:21:21.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:20 smithi039 ceph-mon[31721]: pgmap v760: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-18T16:21:21.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:20 smithi196 ceph-mon[36307]: pgmap v760: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-18T16:21:21.661 INFO:tasks.check_counter:Daemon mds.h mds.exported=66 2024-03-18T16:21:21.662 INFO:tasks.check_counter:Daemon mds.h mds.imported=6 2024-03-18T16:21:21.662 INFO:tasks.check_counter:Daemon mds.h mds.dir_split=0 2024-03-18T16:21:21.662 DEBUG:tasks.check_counter:Getting stats from k 2024-03-18T16:21:21.662 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph daemon mds.k perf dump 2024-03-18T16:21:21.907 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.b/config 2024-03-18T16:21:23.150 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:22 smithi194 ceph-mon[36200]: pgmap v761: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T16:21:23.151 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:22 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:21:23.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:22 smithi039 ceph-mon[31721]: pgmap v761: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T16:21:23.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:22 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:21:23.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:22 smithi196 ceph-mon[36307]: pgmap v761: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T16:21:23.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:22 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-18T16:21:23.885 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-03-18T16:21:23.886 INFO:teuthology.orchestra.run.smithi194.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T16:21:23.886 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_messages": 1745, 2024-03-18T16:21:23.886 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_messages": 1747, 2024-03-18T16:21:23.886 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_bytes": 2740768, 2024-03-18T16:21:23.886 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_bytes": 554911, 2024-03-18T16:21:23.886 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_created_connections": 6, 2024-03-18T16:21:23.886 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_active_connections": 2, 2024-03-18T16:21:23.886 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_total_time": 0.131280294, 2024-03-18T16:21:23.886 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_send_time": 0.040950598, 2024-03-18T16:21:23.886 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_recv_time": 0.034695797, 2024-03-18T16:21:23.886 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_fast_dispatch_time": 0.045467649, 2024-03-18T16:21:23.887 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T16:21:23.887 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 1745, 2024-03-18T16:21:23.887 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.043745464, 2024-03-18T16:21:23.887 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000025069 2024-03-18T16:21:23.887 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:23.887 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_handle_ack_lat": { 2024-03-18T16:21:23.887 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:23.887 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:23.887 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:23.887 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:23.887 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-18T16:21:23.887 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_encrypted_bytes": 0 2024-03-18T16:21:23.887 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:23.887 INFO:teuthology.orchestra.run.smithi194.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T16:21:23.888 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_messages": 667, 2024-03-18T16:21:23.888 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_messages": 854, 2024-03-18T16:21:23.888 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_bytes": 164648, 2024-03-18T16:21:23.888 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_bytes": 604992, 2024-03-18T16:21:23.888 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_created_connections": 5, 2024-03-18T16:21:23.888 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_active_connections": 4, 2024-03-18T16:21:23.888 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_total_time": 0.055867789, 2024-03-18T16:21:23.888 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_send_time": 0.018346818, 2024-03-18T16:21:23.888 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_recv_time": 0.011342211, 2024-03-18T16:21:23.888 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_fast_dispatch_time": 0.020276506, 2024-03-18T16:21:23.888 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T16:21:23.888 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 852, 2024-03-18T16:21:23.888 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.029041437, 2024-03-18T16:21:23.888 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000034086 2024-03-18T16:21:23.888 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:23.889 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_handle_ack_lat": { 2024-03-18T16:21:23.889 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:23.889 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:23.889 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:23.889 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:23.889 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_encrypted_bytes": 160, 2024-03-18T16:21:23.889 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_encrypted_bytes": 393136 2024-03-18T16:21:23.889 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:23.889 INFO:teuthology.orchestra.run.smithi194.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T16:21:23.889 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_messages": 287, 2024-03-18T16:21:23.889 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_messages": 258, 2024-03-18T16:21:23.889 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_bytes": 666652, 2024-03-18T16:21:23.889 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_bytes": 154222, 2024-03-18T16:21:23.889 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_created_connections": 3, 2024-03-18T16:21:23.890 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_active_connections": 2, 2024-03-18T16:21:23.890 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_total_time": 0.032760284, 2024-03-18T16:21:23.890 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_send_time": 0.011117391, 2024-03-18T16:21:23.890 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_recv_time": 0.017816226, 2024-03-18T16:21:23.890 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_fast_dispatch_time": 0.005099319, 2024-03-18T16:21:23.890 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T16:21:23.890 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 258, 2024-03-18T16:21:23.890 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.009319717, 2024-03-18T16:21:23.890 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000036122 2024-03-18T16:21:23.890 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:23.890 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_handle_ack_lat": { 2024-03-18T16:21:23.890 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:23.890 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:23.890 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:23.891 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:23.891 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_encrypted_bytes": 666384, 2024-03-18T16:21:23.891 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_encrypted_bytes": 153904 2024-03-18T16:21:23.891 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:23.891 INFO:teuthology.orchestra.run.smithi194.stdout: "cct": { 2024-03-18T16:21:23.891 INFO:teuthology.orchestra.run.smithi194.stdout: "total_workers": 1, 2024-03-18T16:21:23.891 INFO:teuthology.orchestra.run.smithi194.stdout: "unhealthy_workers": 0 2024-03-18T16:21:23.891 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:23.891 INFO:teuthology.orchestra.run.smithi194.stdout: "finisher-MDSRank": { 2024-03-18T16:21:23.891 INFO:teuthology.orchestra.run.smithi194.stdout: "queue_len": 0, 2024-03-18T16:21:23.891 INFO:teuthology.orchestra.run.smithi194.stdout: "complete_latency": { 2024-03-18T16:21:23.891 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 4100, 2024-03-18T16:21:23.891 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.097234604, 2024-03-18T16:21:23.891 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000023715 2024-03-18T16:21:23.891 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-03-18T16:21:23.892 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:23.892 INFO:teuthology.orchestra.run.smithi194.stdout: "finisher-PurgeQueue": { 2024-03-18T16:21:23.892 INFO:teuthology.orchestra.run.smithi194.stdout: "queue_len": 0, 2024-03-18T16:21:23.892 INFO:teuthology.orchestra.run.smithi194.stdout: "complete_latency": { 2024-03-18T16:21:23.892 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:23.892 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:23.892 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:23.892 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-03-18T16:21:23.892 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:23.892 INFO:teuthology.orchestra.run.smithi194.stdout: "mds": { 2024-03-18T16:21:23.892 INFO:teuthology.orchestra.run.smithi194.stdout: "request": 0, 2024-03-18T16:21:23.892 INFO:teuthology.orchestra.run.smithi194.stdout: "reply": 0, 2024-03-18T16:21:23.892 INFO:teuthology.orchestra.run.smithi194.stdout: "reply_latency": { 2024-03-18T16:21:23.892 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:23.892 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:23.893 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:23.893 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:23.893 INFO:teuthology.orchestra.run.smithi194.stdout: "slow_reply": 0, 2024-03-18T16:21:23.893 INFO:teuthology.orchestra.run.smithi194.stdout: "forward": 0, 2024-03-18T16:21:23.893 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_fetch_complete": 0, 2024-03-18T16:21:23.893 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_fetch_keys": 0, 2024-03-18T16:21:23.893 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_commit": 0, 2024-03-18T16:21:23.893 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_split": 0, 2024-03-18T16:21:23.893 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_merge": 0, 2024-03-18T16:21:23.893 INFO:teuthology.orchestra.run.smithi194.stdout: "inodes": 32, 2024-03-18T16:21:23.893 INFO:teuthology.orchestra.run.smithi194.stdout: "inodes_top": 0, 2024-03-18T16:21:23.893 INFO:teuthology.orchestra.run.smithi194.stdout: "inodes_bottom": 32, 2024-03-18T16:21:23.894 INFO:teuthology.orchestra.run.smithi194.stdout: "inodes_pin_tail": 0, 2024-03-18T16:21:23.894 INFO:teuthology.orchestra.run.smithi194.stdout: "inodes_pinned": 32, 2024-03-18T16:21:23.894 INFO:teuthology.orchestra.run.smithi194.stdout: "inodes_expired": 0, 2024-03-18T16:21:23.894 INFO:teuthology.orchestra.run.smithi194.stdout: "inodes_with_caps": 0, 2024-03-18T16:21:23.894 INFO:teuthology.orchestra.run.smithi194.stdout: "caps": 0, 2024-03-18T16:21:23.894 INFO:teuthology.orchestra.run.smithi194.stdout: "subtrees": 1, 2024-03-18T16:21:23.894 INFO:teuthology.orchestra.run.smithi194.stdout: "traverse": 0, 2024-03-18T16:21:23.894 INFO:teuthology.orchestra.run.smithi194.stdout: "traverse_hit": 0, 2024-03-18T16:21:23.894 INFO:teuthology.orchestra.run.smithi194.stdout: "traverse_forward": 0, 2024-03-18T16:21:23.894 INFO:teuthology.orchestra.run.smithi194.stdout: "traverse_discover": 0, 2024-03-18T16:21:23.894 INFO:teuthology.orchestra.run.smithi194.stdout: "traverse_dir_fetch": 0, 2024-03-18T16:21:23.895 INFO:teuthology.orchestra.run.smithi194.stdout: "traverse_remote_ino": 0, 2024-03-18T16:21:23.895 INFO:teuthology.orchestra.run.smithi194.stdout: "traverse_lock": 0, 2024-03-18T16:21:23.895 INFO:teuthology.orchestra.run.smithi194.stdout: "load_cent": 0, 2024-03-18T16:21:23.895 INFO:teuthology.orchestra.run.smithi194.stdout: "q": 0, 2024-03-18T16:21:23.895 INFO:teuthology.orchestra.run.smithi194.stdout: "exported": 0, 2024-03-18T16:21:23.895 INFO:teuthology.orchestra.run.smithi194.stdout: "exported_inodes": 0, 2024-03-18T16:21:23.895 INFO:teuthology.orchestra.run.smithi194.stdout: "imported": 0, 2024-03-18T16:21:23.895 INFO:teuthology.orchestra.run.smithi194.stdout: "imported_inodes": 0, 2024-03-18T16:21:23.895 INFO:teuthology.orchestra.run.smithi194.stdout: "openino_dir_fetch": 0, 2024-03-18T16:21:23.895 INFO:teuthology.orchestra.run.smithi194.stdout: "openino_backtrace_fetch": 0, 2024-03-18T16:21:23.895 INFO:teuthology.orchestra.run.smithi194.stdout: "openino_peer_discover": 0, 2024-03-18T16:21:23.895 INFO:teuthology.orchestra.run.smithi194.stdout: "root_rfiles": 0, 2024-03-18T16:21:23.896 INFO:teuthology.orchestra.run.smithi194.stdout: "root_rbytes": 0, 2024-03-18T16:21:23.896 INFO:teuthology.orchestra.run.smithi194.stdout: "root_rsnaps": 0, 2024-03-18T16:21:23.896 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_backtrace_fetch": 0, 2024-03-18T16:21:23.896 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_set_tag": 0, 2024-03-18T16:21:23.896 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_backtrace_repaired": 0, 2024-03-18T16:21:23.896 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_inotable_repaired": 0, 2024-03-18T16:21:23.896 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_dir_inodes": 0, 2024-03-18T16:21:23.896 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_dir_base_inodes": 0, 2024-03-18T16:21:23.896 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_dirfrag_rstats": 0, 2024-03-18T16:21:23.896 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_file_inodes": 0, 2024-03-18T16:21:23.896 INFO:teuthology.orchestra.run.smithi194.stdout: "handle_inode_file_caps": 0, 2024-03-18T16:21:23.896 INFO:teuthology.orchestra.run.smithi194.stdout: "ceph_cap_op_revoke": 0, 2024-03-18T16:21:23.897 INFO:teuthology.orchestra.run.smithi194.stdout: "ceph_cap_op_grant": 0, 2024-03-18T16:21:23.897 INFO:teuthology.orchestra.run.smithi194.stdout: "ceph_cap_op_trunc": 0, 2024-03-18T16:21:23.897 INFO:teuthology.orchestra.run.smithi194.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-18T16:21:23.897 INFO:teuthology.orchestra.run.smithi194.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-18T16:21:23.897 INFO:teuthology.orchestra.run.smithi194.stdout: "handle_client_caps": 0, 2024-03-18T16:21:23.897 INFO:teuthology.orchestra.run.smithi194.stdout: "handle_client_caps_dirty": 0, 2024-03-18T16:21:23.897 INFO:teuthology.orchestra.run.smithi194.stdout: "handle_client_cap_release": 0, 2024-03-18T16:21:23.897 INFO:teuthology.orchestra.run.smithi194.stdout: "process_request_cap_release": 0 2024-03-18T16:21:23.897 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:23.897 INFO:teuthology.orchestra.run.smithi194.stdout: "mds_cache": { 2024-03-18T16:21:23.897 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_update": 0, 2024-03-18T16:21:23.897 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_update_receipt": 0, 2024-03-18T16:21:23.898 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_try_discover": 0, 2024-03-18T16:21:23.898 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_send_discover": 0, 2024-03-18T16:21:23.898 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_handle_discover": 0, 2024-03-18T16:21:23.898 INFO:teuthology.orchestra.run.smithi194.stdout: "num_strays": 0, 2024-03-18T16:21:23.898 INFO:teuthology.orchestra.run.smithi194.stdout: "num_strays_delayed": 0, 2024-03-18T16:21:23.898 INFO:teuthology.orchestra.run.smithi194.stdout: "num_strays_enqueuing": 0, 2024-03-18T16:21:23.898 INFO:teuthology.orchestra.run.smithi194.stdout: "strays_created": 0, 2024-03-18T16:21:23.898 INFO:teuthology.orchestra.run.smithi194.stdout: "strays_enqueued": 0, 2024-03-18T16:21:23.898 INFO:teuthology.orchestra.run.smithi194.stdout: "strays_reintegrated": 0, 2024-03-18T16:21:23.898 INFO:teuthology.orchestra.run.smithi194.stdout: "strays_migrated": 0, 2024-03-18T16:21:23.898 INFO:teuthology.orchestra.run.smithi194.stdout: "num_recovering_processing": 0, 2024-03-18T16:21:23.898 INFO:teuthology.orchestra.run.smithi194.stdout: "num_recovering_enqueued": 0, 2024-03-18T16:21:23.899 INFO:teuthology.orchestra.run.smithi194.stdout: "num_recovering_prioritized": 0, 2024-03-18T16:21:23.899 INFO:teuthology.orchestra.run.smithi194.stdout: "recovery_started": 0, 2024-03-18T16:21:23.899 INFO:teuthology.orchestra.run.smithi194.stdout: "recovery_completed": 0, 2024-03-18T16:21:23.899 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_quiesce_path": 0, 2024-03-18T16:21:23.899 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_quiesce_inode": 0, 2024-03-18T16:21:23.899 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_enqueue_scrub": 0, 2024-03-18T16:21:23.899 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_exportdir": 0, 2024-03-18T16:21:23.899 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_flush": 0, 2024-03-18T16:21:23.899 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_fragmentdir": 0, 2024-03-18T16:21:23.899 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_fragstats": 0, 2024-03-18T16:21:23.899 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_inodestats": 0 2024-03-18T16:21:23.899 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:23.900 INFO:teuthology.orchestra.run.smithi194.stdout: "mds_log": { 2024-03-18T16:21:23.900 INFO:teuthology.orchestra.run.smithi194.stdout: "evlrg": 0, 2024-03-18T16:21:23.900 INFO:teuthology.orchestra.run.smithi194.stdout: "evadd": 0, 2024-03-18T16:21:23.900 INFO:teuthology.orchestra.run.smithi194.stdout: "evex": 0, 2024-03-18T16:21:23.900 INFO:teuthology.orchestra.run.smithi194.stdout: "evtrm": 0, 2024-03-18T16:21:23.900 INFO:teuthology.orchestra.run.smithi194.stdout: "ev": 1158, 2024-03-18T16:21:23.900 INFO:teuthology.orchestra.run.smithi194.stdout: "evexg": 0, 2024-03-18T16:21:23.900 INFO:teuthology.orchestra.run.smithi194.stdout: "evexd": 0, 2024-03-18T16:21:23.900 INFO:teuthology.orchestra.run.smithi194.stdout: "segadd": 0, 2024-03-18T16:21:23.900 INFO:teuthology.orchestra.run.smithi194.stdout: "segex": 0, 2024-03-18T16:21:23.900 INFO:teuthology.orchestra.run.smithi194.stdout: "segtrm": 0, 2024-03-18T16:21:23.900 INFO:teuthology.orchestra.run.smithi194.stdout: "seg": 3, 2024-03-18T16:21:23.901 INFO:teuthology.orchestra.run.smithi194.stdout: "segmjr": 2, 2024-03-18T16:21:23.901 INFO:teuthology.orchestra.run.smithi194.stdout: "segexg": 0, 2024-03-18T16:21:23.901 INFO:teuthology.orchestra.run.smithi194.stdout: "segexd": 0, 2024-03-18T16:21:23.901 INFO:teuthology.orchestra.run.smithi194.stdout: "expos": 4194304, 2024-03-18T16:21:23.901 INFO:teuthology.orchestra.run.smithi194.stdout: "wrpos": 6419165, 2024-03-18T16:21:23.901 INFO:teuthology.orchestra.run.smithi194.stdout: "rdpos": 6417514, 2024-03-18T16:21:23.901 INFO:teuthology.orchestra.run.smithi194.stdout: "jlat": { 2024-03-18T16:21:23.901 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:23.901 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:23.901 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:23.902 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:23.902 INFO:teuthology.orchestra.run.smithi194.stdout: "replayed": 1158 2024-03-18T16:21:23.902 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:23.902 INFO:teuthology.orchestra.run.smithi194.stdout: "mds_mem": { 2024-03-18T16:21:23.902 INFO:teuthology.orchestra.run.smithi194.stdout: "ino": 4, 2024-03-18T16:21:23.902 INFO:teuthology.orchestra.run.smithi194.stdout: "ino+": 227, 2024-03-18T16:21:23.902 INFO:teuthology.orchestra.run.smithi194.stdout: "ino-": 223, 2024-03-18T16:21:23.902 INFO:teuthology.orchestra.run.smithi194.stdout: "dir": 2, 2024-03-18T16:21:23.902 INFO:teuthology.orchestra.run.smithi194.stdout: "dir+": 92, 2024-03-18T16:21:23.903 INFO:teuthology.orchestra.run.smithi194.stdout: "dir-": 90, 2024-03-18T16:21:23.903 INFO:teuthology.orchestra.run.smithi194.stdout: "dn": 32, 2024-03-18T16:21:23.903 INFO:teuthology.orchestra.run.smithi194.stdout: "dn+": 343, 2024-03-18T16:21:23.903 INFO:teuthology.orchestra.run.smithi194.stdout: "dn-": 311, 2024-03-18T16:21:23.903 INFO:teuthology.orchestra.run.smithi194.stdout: "cap": 0, 2024-03-18T16:21:23.903 INFO:teuthology.orchestra.run.smithi194.stdout: "cap+": 0, 2024-03-18T16:21:23.903 INFO:teuthology.orchestra.run.smithi194.stdout: "cap-": 0, 2024-03-18T16:21:23.903 INFO:teuthology.orchestra.run.smithi194.stdout: "rss": 52740, 2024-03-18T16:21:23.903 INFO:teuthology.orchestra.run.smithi194.stdout: "heap": 215420 2024-03-18T16:21:23.903 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:23.903 INFO:teuthology.orchestra.run.smithi194.stdout: "mds_server": { 2024-03-18T16:21:23.904 INFO:teuthology.orchestra.run.smithi194.stdout: "dispatch_client_request": 0, 2024-03-18T16:21:23.904 INFO:teuthology.orchestra.run.smithi194.stdout: "dispatch_server_request": 0, 2024-03-18T16:21:23.904 INFO:teuthology.orchestra.run.smithi194.stdout: "handle_client_request": 0, 2024-03-18T16:21:23.904 INFO:teuthology.orchestra.run.smithi194.stdout: "handle_client_session": 0, 2024-03-18T16:21:23.904 INFO:teuthology.orchestra.run.smithi194.stdout: "handle_peer_request": 0, 2024-03-18T16:21:23.904 INFO:teuthology.orchestra.run.smithi194.stdout: "req_create_latency": { 2024-03-18T16:21:23.904 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:23.904 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:23.904 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:23.904 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:23.905 INFO:teuthology.orchestra.run.smithi194.stdout: "req_getattr_latency": { 2024-03-18T16:21:23.905 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:23.905 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:23.905 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:23.905 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:23.905 INFO:teuthology.orchestra.run.smithi194.stdout: "req_getfilelock_latency": { 2024-03-18T16:21:23.905 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:23.905 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:23.905 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:23.905 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:23.905 INFO:teuthology.orchestra.run.smithi194.stdout: "req_link_latency": { 2024-03-18T16:21:23.905 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:23.905 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:23.906 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:23.906 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:23.906 INFO:teuthology.orchestra.run.smithi194.stdout: "req_lookup_latency": { 2024-03-18T16:21:23.906 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:23.906 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:23.906 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:23.906 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:23.906 INFO:teuthology.orchestra.run.smithi194.stdout: "req_lookuphash_latency": { 2024-03-18T16:21:23.907 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:23.907 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:23.907 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:23.907 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:23.907 INFO:teuthology.orchestra.run.smithi194.stdout: "req_lookupino_latency": { 2024-03-18T16:21:23.907 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:23.907 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:23.907 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:23.907 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:23.907 INFO:teuthology.orchestra.run.smithi194.stdout: "req_lookupname_latency": { 2024-03-18T16:21:23.907 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:23.907 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:23.908 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:23.908 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:23.908 INFO:teuthology.orchestra.run.smithi194.stdout: "req_lookupparent_latency": { 2024-03-18T16:21:23.908 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:23.908 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:23.908 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:23.908 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:23.908 INFO:teuthology.orchestra.run.smithi194.stdout: "req_lookupsnap_latency": { 2024-03-18T16:21:23.908 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:23.908 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:23.908 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:23.909 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:23.909 INFO:teuthology.orchestra.run.smithi194.stdout: "req_lssnap_latency": { 2024-03-18T16:21:23.909 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:23.909 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:23.909 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:23.909 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:23.909 INFO:teuthology.orchestra.run.smithi194.stdout: "req_mkdir_latency": { 2024-03-18T16:21:23.909 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:23.910 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:23.910 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:23.910 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:23.910 INFO:teuthology.orchestra.run.smithi194.stdout: "req_mknod_latency": { 2024-03-18T16:21:23.910 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:23.910 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:23.911 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:23.911 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:23.911 INFO:teuthology.orchestra.run.smithi194.stdout: "req_mksnap_latency": { 2024-03-18T16:21:23.911 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:23.911 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:23.911 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:23.911 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:23.911 INFO:teuthology.orchestra.run.smithi194.stdout: "req_open_latency": { 2024-03-18T16:21:23.911 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:23.911 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:23.911 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:23.912 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:23.912 INFO:teuthology.orchestra.run.smithi194.stdout: "req_readdir_latency": { 2024-03-18T16:21:23.912 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:23.912 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:23.912 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:23.912 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:23.912 INFO:teuthology.orchestra.run.smithi194.stdout: "req_rename_latency": { 2024-03-18T16:21:23.912 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:23.912 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:23.912 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:23.912 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:23.913 INFO:teuthology.orchestra.run.smithi194.stdout: "req_renamesnap_latency": { 2024-03-18T16:21:23.913 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:23.913 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:23.913 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:23.913 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:23.913 INFO:teuthology.orchestra.run.smithi194.stdout: "req_snapdiff_latency": { 2024-03-18T16:21:23.913 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:23.913 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:23.913 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:23.913 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:23.914 INFO:teuthology.orchestra.run.smithi194.stdout: "req_rmdir_latency": { 2024-03-18T16:21:23.914 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:23.914 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:23.914 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:23.914 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:23.914 INFO:teuthology.orchestra.run.smithi194.stdout: "req_rmsnap_latency": { 2024-03-18T16:21:23.914 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:23.914 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:23.914 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:23.914 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:23.914 INFO:teuthology.orchestra.run.smithi194.stdout: "req_rmxattr_latency": { 2024-03-18T16:21:23.914 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:23.914 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:23.914 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:23.915 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:23.915 INFO:teuthology.orchestra.run.smithi194.stdout: "req_setattr_latency": { 2024-03-18T16:21:23.915 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:23.915 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:23.915 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:23.915 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:23.915 INFO:teuthology.orchestra.run.smithi194.stdout: "req_setdirlayout_latency": { 2024-03-18T16:21:23.915 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:23.915 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:23.915 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:23.915 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:23.915 INFO:teuthology.orchestra.run.smithi194.stdout: "req_setfilelock_latency": { 2024-03-18T16:21:23.915 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:23.915 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:23.915 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:23.915 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:23.916 INFO:teuthology.orchestra.run.smithi194.stdout: "req_setlayout_latency": { 2024-03-18T16:21:23.916 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:23.916 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:23.916 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:23.916 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:23.916 INFO:teuthology.orchestra.run.smithi194.stdout: "req_setxattr_latency": { 2024-03-18T16:21:23.916 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:23.916 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:23.916 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:23.916 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:23.916 INFO:teuthology.orchestra.run.smithi194.stdout: "req_symlink_latency": { 2024-03-18T16:21:23.916 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:23.916 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:23.917 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:23.917 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:23.917 INFO:teuthology.orchestra.run.smithi194.stdout: "req_unlink_latency": { 2024-03-18T16:21:23.917 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:23.917 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:23.917 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:23.917 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:23.917 INFO:teuthology.orchestra.run.smithi194.stdout: "cap_revoke_eviction": 0, 2024-03-18T16:21:23.917 INFO:teuthology.orchestra.run.smithi194.stdout: "cap_acquisition_throttle": 0, 2024-03-18T16:21:23.917 INFO:teuthology.orchestra.run.smithi194.stdout: "req_getvxattr_latency": { 2024-03-18T16:21:23.917 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:23.917 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:23.918 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:23.918 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-03-18T16:21:23.918 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:23.918 INFO:teuthology.orchestra.run.smithi194.stdout: "mds_sessions": { 2024-03-18T16:21:23.918 INFO:teuthology.orchestra.run.smithi194.stdout: "session_count": 1, 2024-03-18T16:21:23.918 INFO:teuthology.orchestra.run.smithi194.stdout: "session_add": 1, 2024-03-18T16:21:23.918 INFO:teuthology.orchestra.run.smithi194.stdout: "session_remove": 0, 2024-03-18T16:21:23.918 INFO:teuthology.orchestra.run.smithi194.stdout: "sessions_open": 1, 2024-03-18T16:21:23.918 INFO:teuthology.orchestra.run.smithi194.stdout: "sessions_stale": 0, 2024-03-18T16:21:23.918 INFO:teuthology.orchestra.run.smithi194.stdout: "total_load": 0, 2024-03-18T16:21:23.918 INFO:teuthology.orchestra.run.smithi194.stdout: "average_load": 0, 2024-03-18T16:21:23.918 INFO:teuthology.orchestra.run.smithi194.stdout: "avg_session_uptime": 2479, 2024-03-18T16:21:23.919 INFO:teuthology.orchestra.run.smithi194.stdout: "mdthresh_evicted": 0 2024-03-18T16:21:23.919 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:23.919 INFO:teuthology.orchestra.run.smithi194.stdout: "mempool": { 2024-03-18T16:21:23.919 INFO:teuthology.orchestra.run.smithi194.stdout: "bloom_filter_bytes": 0, 2024-03-18T16:21:23.919 INFO:teuthology.orchestra.run.smithi194.stdout: "bloom_filter_items": 0, 2024-03-18T16:21:23.919 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T16:21:23.919 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_alloc_items": 0, 2024-03-18T16:21:23.919 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T16:21:23.919 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_data_items": 0, 2024-03-18T16:21:23.919 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T16:21:23.919 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T16:21:23.919 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T16:21:23.919 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T16:21:23.920 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T16:21:23.920 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_other_items": 0, 2024-03-18T16:21:23.920 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T16:21:23.920 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T16:21:23.920 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_extent_bytes": 0, 2024-03-18T16:21:23.920 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_extent_items": 0, 2024-03-18T16:21:23.920 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_blob_bytes": 0, 2024-03-18T16:21:23.920 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_blob_items": 0, 2024-03-18T16:21:23.920 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T16:21:23.920 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T16:21:23.921 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T16:21:23.921 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T16:21:23.921 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T16:21:23.921 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_fsck_items": 0, 2024-03-18T16:21:23.921 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_txc_bytes": 0, 2024-03-18T16:21:23.921 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_txc_items": 0, 2024-03-18T16:21:23.921 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T16:21:23.921 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T16:21:23.921 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_writing_bytes": 0, 2024-03-18T16:21:23.921 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_writing_items": 0, 2024-03-18T16:21:23.921 INFO:teuthology.orchestra.run.smithi194.stdout: "bluefs_bytes": 0, 2024-03-18T16:21:23.921 INFO:teuthology.orchestra.run.smithi194.stdout: "bluefs_items": 0, 2024-03-18T16:21:23.922 INFO:teuthology.orchestra.run.smithi194.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T16:21:23.922 INFO:teuthology.orchestra.run.smithi194.stdout: "bluefs_file_reader_items": 0, 2024-03-18T16:21:23.922 INFO:teuthology.orchestra.run.smithi194.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T16:21:23.922 INFO:teuthology.orchestra.run.smithi194.stdout: "bluefs_file_writer_items": 0, 2024-03-18T16:21:23.922 INFO:teuthology.orchestra.run.smithi194.stdout: "buffer_anon_bytes": 189339, 2024-03-18T16:21:23.922 INFO:teuthology.orchestra.run.smithi194.stdout: "buffer_anon_items": 50, 2024-03-18T16:21:23.922 INFO:teuthology.orchestra.run.smithi194.stdout: "buffer_meta_bytes": 0, 2024-03-18T16:21:23.922 INFO:teuthology.orchestra.run.smithi194.stdout: "buffer_meta_items": 0, 2024-03-18T16:21:23.922 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_bytes": 0, 2024-03-18T16:21:23.922 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_items": 0, 2024-03-18T16:21:23.923 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_mapbl_bytes": 0, 2024-03-18T16:21:23.923 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_mapbl_items": 0, 2024-03-18T16:21:23.923 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_pglog_bytes": 0, 2024-03-18T16:21:23.923 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_pglog_items": 0, 2024-03-18T16:21:23.923 INFO:teuthology.orchestra.run.smithi194.stdout: "osdmap_bytes": 8888, 2024-03-18T16:21:23.923 INFO:teuthology.orchestra.run.smithi194.stdout: "osdmap_items": 170, 2024-03-18T16:21:23.923 INFO:teuthology.orchestra.run.smithi194.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T16:21:23.923 INFO:teuthology.orchestra.run.smithi194.stdout: "osdmap_mapping_items": 0, 2024-03-18T16:21:23.923 INFO:teuthology.orchestra.run.smithi194.stdout: "pgmap_bytes": 0, 2024-03-18T16:21:23.923 INFO:teuthology.orchestra.run.smithi194.stdout: "pgmap_items": 0, 2024-03-18T16:21:23.923 INFO:teuthology.orchestra.run.smithi194.stdout: "mds_co_bytes": 34512, 2024-03-18T16:21:23.923 INFO:teuthology.orchestra.run.smithi194.stdout: "mds_co_items": 156, 2024-03-18T16:21:23.924 INFO:teuthology.orchestra.run.smithi194.stdout: "unittest_1_bytes": 0, 2024-03-18T16:21:23.924 INFO:teuthology.orchestra.run.smithi194.stdout: "unittest_1_items": 0, 2024-03-18T16:21:23.924 INFO:teuthology.orchestra.run.smithi194.stdout: "unittest_2_bytes": 0, 2024-03-18T16:21:23.924 INFO:teuthology.orchestra.run.smithi194.stdout: "unittest_2_items": 0 2024-03-18T16:21:23.924 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:23.924 INFO:teuthology.orchestra.run.smithi194.stdout: "objecter": { 2024-03-18T16:21:23.924 INFO:teuthology.orchestra.run.smithi194.stdout: "op_active": 0, 2024-03-18T16:21:23.924 INFO:teuthology.orchestra.run.smithi194.stdout: "op_laggy": 0, 2024-03-18T16:21:23.924 INFO:teuthology.orchestra.run.smithi194.stdout: "op_send": 2412, 2024-03-18T16:21:23.924 INFO:teuthology.orchestra.run.smithi194.stdout: "op_send_bytes": 21, 2024-03-18T16:21:23.924 INFO:teuthology.orchestra.run.smithi194.stdout: "op_resend": 3, 2024-03-18T16:21:23.924 INFO:teuthology.orchestra.run.smithi194.stdout: "op_reply": 2412, 2024-03-18T16:21:23.924 INFO:teuthology.orchestra.run.smithi194.stdout: "op_latency": { 2024-03-18T16:21:23.925 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 2412, 2024-03-18T16:21:23.925 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 27.189453069, 2024-03-18T16:21:23.925 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.011272575 2024-03-18T16:21:23.925 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:23.925 INFO:teuthology.orchestra.run.smithi194.stdout: "op_inflight": 0, 2024-03-18T16:21:23.925 INFO:teuthology.orchestra.run.smithi194.stdout: "oplen_avg": { 2024-03-18T16:21:23.925 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 2412, 2024-03-18T16:21:23.925 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 2413 2024-03-18T16:21:23.925 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:23.925 INFO:teuthology.orchestra.run.smithi194.stdout: "op": 2412, 2024-03-18T16:21:23.925 INFO:teuthology.orchestra.run.smithi194.stdout: "op_r": 2412, 2024-03-18T16:21:23.925 INFO:teuthology.orchestra.run.smithi194.stdout: "op_w": 0, 2024-03-18T16:21:23.925 INFO:teuthology.orchestra.run.smithi194.stdout: "op_rmw": 0, 2024-03-18T16:21:23.925 INFO:teuthology.orchestra.run.smithi194.stdout: "op_pg": 0, 2024-03-18T16:21:23.925 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_stat": 1480, 2024-03-18T16:21:23.926 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_create": 0, 2024-03-18T16:21:23.926 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_read": 930, 2024-03-18T16:21:23.926 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_write": 0, 2024-03-18T16:21:23.926 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_writefull": 0, 2024-03-18T16:21:23.926 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_writesame": 0, 2024-03-18T16:21:23.926 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_append": 0, 2024-03-18T16:21:23.926 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_zero": 0, 2024-03-18T16:21:23.926 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_truncate": 0, 2024-03-18T16:21:23.926 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_delete": 0, 2024-03-18T16:21:23.926 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_mapext": 0, 2024-03-18T16:21:23.926 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_sparse_read": 0, 2024-03-18T16:21:23.926 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_clonerange": 0, 2024-03-18T16:21:23.926 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_getxattr": 1, 2024-03-18T16:21:23.926 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_setxattr": 0, 2024-03-18T16:21:23.926 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_cmpxattr": 0, 2024-03-18T16:21:23.927 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_rmxattr": 0, 2024-03-18T16:21:23.927 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_resetxattrs": 0, 2024-03-18T16:21:23.927 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_call": 0, 2024-03-18T16:21:23.927 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_watch": 0, 2024-03-18T16:21:23.927 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_notify": 0, 2024-03-18T16:21:23.927 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_src_cmpxattr": 0, 2024-03-18T16:21:23.927 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_pgls": 0, 2024-03-18T16:21:23.927 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_pgls_filter": 0, 2024-03-18T16:21:23.927 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_other": 0, 2024-03-18T16:21:23.927 INFO:teuthology.orchestra.run.smithi194.stdout: "linger_active": 0, 2024-03-18T16:21:23.927 INFO:teuthology.orchestra.run.smithi194.stdout: "linger_send": 0, 2024-03-18T16:21:23.927 INFO:teuthology.orchestra.run.smithi194.stdout: "linger_resend": 0, 2024-03-18T16:21:23.927 INFO:teuthology.orchestra.run.smithi194.stdout: "linger_ping": 0, 2024-03-18T16:21:23.928 INFO:teuthology.orchestra.run.smithi194.stdout: "poolop_active": 0, 2024-03-18T16:21:23.928 INFO:teuthology.orchestra.run.smithi194.stdout: "poolop_send": 0, 2024-03-18T16:21:23.928 INFO:teuthology.orchestra.run.smithi194.stdout: "poolop_resend": 0, 2024-03-18T16:21:23.928 INFO:teuthology.orchestra.run.smithi194.stdout: "poolstat_active": 0, 2024-03-18T16:21:23.928 INFO:teuthology.orchestra.run.smithi194.stdout: "poolstat_send": 0, 2024-03-18T16:21:23.928 INFO:teuthology.orchestra.run.smithi194.stdout: "poolstat_resend": 0, 2024-03-18T16:21:23.928 INFO:teuthology.orchestra.run.smithi194.stdout: "statfs_active": 0, 2024-03-18T16:21:23.928 INFO:teuthology.orchestra.run.smithi194.stdout: "statfs_send": 0, 2024-03-18T16:21:23.928 INFO:teuthology.orchestra.run.smithi194.stdout: "statfs_resend": 0, 2024-03-18T16:21:23.928 INFO:teuthology.orchestra.run.smithi194.stdout: "command_active": 0, 2024-03-18T16:21:23.928 INFO:teuthology.orchestra.run.smithi194.stdout: "command_send": 0, 2024-03-18T16:21:23.929 INFO:teuthology.orchestra.run.smithi194.stdout: "command_resend": 0, 2024-03-18T16:21:23.929 INFO:teuthology.orchestra.run.smithi194.stdout: "map_epoch": 93, 2024-03-18T16:21:23.929 INFO:teuthology.orchestra.run.smithi194.stdout: "map_full": 0, 2024-03-18T16:21:23.929 INFO:teuthology.orchestra.run.smithi194.stdout: "map_inc": 4, 2024-03-18T16:21:23.929 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_sessions": 5, 2024-03-18T16:21:23.929 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_session_open": 5, 2024-03-18T16:21:23.929 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_session_close": 0, 2024-03-18T16:21:23.929 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_laggy": 0, 2024-03-18T16:21:23.929 INFO:teuthology.orchestra.run.smithi194.stdout: "omap_wr": 0, 2024-03-18T16:21:23.929 INFO:teuthology.orchestra.run.smithi194.stdout: "omap_rd": 2, 2024-03-18T16:21:23.929 INFO:teuthology.orchestra.run.smithi194.stdout: "omap_del": 0 2024-03-18T16:21:23.930 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:23.930 INFO:teuthology.orchestra.run.smithi194.stdout: "oft": { 2024-03-18T16:21:23.930 INFO:teuthology.orchestra.run.smithi194.stdout: "omap_total_objs": 0, 2024-03-18T16:21:23.930 INFO:teuthology.orchestra.run.smithi194.stdout: "omap_total_kv_pairs": 0, 2024-03-18T16:21:23.930 INFO:teuthology.orchestra.run.smithi194.stdout: "omap_total_updates": 0, 2024-03-18T16:21:23.930 INFO:teuthology.orchestra.run.smithi194.stdout: "omap_total_removes": 0 2024-03-18T16:21:23.930 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:23.930 INFO:teuthology.orchestra.run.smithi194.stdout: "osd-slow-ops": { 2024-03-18T16:21:23.930 INFO:teuthology.orchestra.run.smithi194.stdout: "slow_ops_count": 0 2024-03-18T16:21:23.931 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:23.931 INFO:teuthology.orchestra.run.smithi194.stdout: "purge_queue": { 2024-03-18T16:21:23.931 INFO:teuthology.orchestra.run.smithi194.stdout: "pq_executing_ops": 0, 2024-03-18T16:21:23.931 INFO:teuthology.orchestra.run.smithi194.stdout: "pq_executing_ops_high_water": 0, 2024-03-18T16:21:23.931 INFO:teuthology.orchestra.run.smithi194.stdout: "pq_executing": 0, 2024-03-18T16:21:23.931 INFO:teuthology.orchestra.run.smithi194.stdout: "pq_executing_high_water": 0, 2024-03-18T16:21:23.931 INFO:teuthology.orchestra.run.smithi194.stdout: "pq_executed_ops": 0, 2024-03-18T16:21:23.931 INFO:teuthology.orchestra.run.smithi194.stdout: "pq_executed": 0, 2024-03-18T16:21:23.931 INFO:teuthology.orchestra.run.smithi194.stdout: "pq_item_in_journal": 0 2024-03-18T16:21:23.931 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:23.931 INFO:teuthology.orchestra.run.smithi194.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T16:21:23.931 INFO:teuthology.orchestra.run.smithi194.stdout: "val": 0, 2024-03-18T16:21:23.931 INFO:teuthology.orchestra.run.smithi194.stdout: "max": 104857600, 2024-03-18T16:21:23.932 INFO:teuthology.orchestra.run.smithi194.stdout: "get_started": 0, 2024-03-18T16:21:23.932 INFO:teuthology.orchestra.run.smithi194.stdout: "get": 2695, 2024-03-18T16:21:23.932 INFO:teuthology.orchestra.run.smithi194.stdout: "get_sum": 3312127, 2024-03-18T16:21:23.932 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_fail": 0, 2024-03-18T16:21:23.932 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_success": 2695, 2024-03-18T16:21:23.932 INFO:teuthology.orchestra.run.smithi194.stdout: "take": 0, 2024-03-18T16:21:23.932 INFO:teuthology.orchestra.run.smithi194.stdout: "take_sum": 0, 2024-03-18T16:21:23.932 INFO:teuthology.orchestra.run.smithi194.stdout: "put": 2695, 2024-03-18T16:21:23.932 INFO:teuthology.orchestra.run.smithi194.stdout: "put_sum": 3312127, 2024-03-18T16:21:23.932 INFO:teuthology.orchestra.run.smithi194.stdout: "wait": { 2024-03-18T16:21:23.932 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:23.932 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:23.932 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:23.932 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-03-18T16:21:23.933 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:23.933 INFO:teuthology.orchestra.run.smithi194.stdout: "throttle-objecter_bytes": { 2024-03-18T16:21:23.933 INFO:teuthology.orchestra.run.smithi194.stdout: "val": 0, 2024-03-18T16:21:23.933 INFO:teuthology.orchestra.run.smithi194.stdout: "max": 104857600, 2024-03-18T16:21:23.933 INFO:teuthology.orchestra.run.smithi194.stdout: "get_started": 0, 2024-03-18T16:21:23.933 INFO:teuthology.orchestra.run.smithi194.stdout: "get": 0, 2024-03-18T16:21:23.933 INFO:teuthology.orchestra.run.smithi194.stdout: "get_sum": 0, 2024-03-18T16:21:23.933 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_fail": 0, 2024-03-18T16:21:23.933 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_success": 0, 2024-03-18T16:21:23.933 INFO:teuthology.orchestra.run.smithi194.stdout: "take": 2412, 2024-03-18T16:21:23.933 INFO:teuthology.orchestra.run.smithi194.stdout: "take_sum": 2224866, 2024-03-18T16:21:23.933 INFO:teuthology.orchestra.run.smithi194.stdout: "put": 2412, 2024-03-18T16:21:23.933 INFO:teuthology.orchestra.run.smithi194.stdout: "put_sum": 2224866, 2024-03-18T16:21:23.934 INFO:teuthology.orchestra.run.smithi194.stdout: "wait": { 2024-03-18T16:21:23.934 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:23.934 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:23.934 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:23.934 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-03-18T16:21:23.934 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:23.934 INFO:teuthology.orchestra.run.smithi194.stdout: "throttle-objecter_ops": { 2024-03-18T16:21:23.934 INFO:teuthology.orchestra.run.smithi194.stdout: "val": 0, 2024-03-18T16:21:23.934 INFO:teuthology.orchestra.run.smithi194.stdout: "max": 1024, 2024-03-18T16:21:23.934 INFO:teuthology.orchestra.run.smithi194.stdout: "get_started": 0, 2024-03-18T16:21:23.935 INFO:teuthology.orchestra.run.smithi194.stdout: "get": 0, 2024-03-18T16:21:23.935 INFO:teuthology.orchestra.run.smithi194.stdout: "get_sum": 0, 2024-03-18T16:21:23.935 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_fail": 0, 2024-03-18T16:21:23.935 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_success": 0, 2024-03-18T16:21:23.935 INFO:teuthology.orchestra.run.smithi194.stdout: "take": 2412, 2024-03-18T16:21:23.935 INFO:teuthology.orchestra.run.smithi194.stdout: "take_sum": 2412, 2024-03-18T16:21:23.935 INFO:teuthology.orchestra.run.smithi194.stdout: "put": 2412, 2024-03-18T16:21:23.935 INFO:teuthology.orchestra.run.smithi194.stdout: "put_sum": 2412, 2024-03-18T16:21:23.935 INFO:teuthology.orchestra.run.smithi194.stdout: "wait": { 2024-03-18T16:21:23.935 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:23.935 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:23.935 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:23.936 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-03-18T16:21:23.936 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:23.936 INFO:teuthology.orchestra.run.smithi194.stdout: "throttle-write_buf_throttle": { 2024-03-18T16:21:23.936 INFO:teuthology.orchestra.run.smithi194.stdout: "val": 0, 2024-03-18T16:21:23.936 INFO:teuthology.orchestra.run.smithi194.stdout: "max": 3758096384, 2024-03-18T16:21:23.936 INFO:teuthology.orchestra.run.smithi194.stdout: "get_started": 0, 2024-03-18T16:21:23.936 INFO:teuthology.orchestra.run.smithi194.stdout: "get": 0, 2024-03-18T16:21:23.936 INFO:teuthology.orchestra.run.smithi194.stdout: "get_sum": 0, 2024-03-18T16:21:23.936 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_fail": 0, 2024-03-18T16:21:23.936 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_success": 0, 2024-03-18T16:21:23.936 INFO:teuthology.orchestra.run.smithi194.stdout: "take": 0, 2024-03-18T16:21:23.937 INFO:teuthology.orchestra.run.smithi194.stdout: "take_sum": 0, 2024-03-18T16:21:23.937 INFO:teuthology.orchestra.run.smithi194.stdout: "put": 0, 2024-03-18T16:21:23.937 INFO:teuthology.orchestra.run.smithi194.stdout: "put_sum": 0, 2024-03-18T16:21:23.937 INFO:teuthology.orchestra.run.smithi194.stdout: "wait": { 2024-03-18T16:21:23.937 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:23.937 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:23.937 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:23.937 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-03-18T16:21:23.937 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-03-18T16:21:23.937 INFO:teuthology.orchestra.run.smithi194.stdout: "throttle-write_buf_throttle-0x56167e7666e0": { 2024-03-18T16:21:23.937 INFO:teuthology.orchestra.run.smithi194.stdout: "val": 0, 2024-03-18T16:21:23.937 INFO:teuthology.orchestra.run.smithi194.stdout: "max": 3758096384, 2024-03-18T16:21:23.938 INFO:teuthology.orchestra.run.smithi194.stdout: "get_started": 0, 2024-03-18T16:21:23.938 INFO:teuthology.orchestra.run.smithi194.stdout: "get": 0, 2024-03-18T16:21:23.938 INFO:teuthology.orchestra.run.smithi194.stdout: "get_sum": 0, 2024-03-18T16:21:23.938 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_fail": 0, 2024-03-18T16:21:23.938 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_success": 0, 2024-03-18T16:21:23.938 INFO:teuthology.orchestra.run.smithi194.stdout: "take": 0, 2024-03-18T16:21:23.938 INFO:teuthology.orchestra.run.smithi194.stdout: "take_sum": 0, 2024-03-18T16:21:23.938 INFO:teuthology.orchestra.run.smithi194.stdout: "put": 0, 2024-03-18T16:21:23.938 INFO:teuthology.orchestra.run.smithi194.stdout: "put_sum": 0, 2024-03-18T16:21:23.938 INFO:teuthology.orchestra.run.smithi194.stdout: "wait": { 2024-03-18T16:21:23.938 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-03-18T16:21:23.939 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-03-18T16:21:23.939 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-03-18T16:21:23.939 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-03-18T16:21:23.939 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-03-18T16:21:23.939 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-03-18T16:21:24.276 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:23 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:21:24.276 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:23 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:21:24.276 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:23 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:21:24.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:23 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:21:24.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:23 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:21:24.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:23 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:21:24.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:23 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-18T16:21:24.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:23 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-18T16:21:24.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:23 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' 2024-03-18T16:21:24.641 INFO:tasks.check_counter:Daemon mds.k mds.exported=0 2024-03-18T16:21:24.641 INFO:tasks.check_counter:Daemon mds.k mds.imported=0 2024-03-18T16:21:24.641 INFO:tasks.check_counter:Daemon mds.k mds.dir_split=0 2024-03-18T16:21:24.641 DEBUG:tasks.check_counter:Getting stats from c 2024-03-18T16:21:24.641 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph daemon mds.c perf dump 2024-03-18T16:21:25.041 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T16:21:25.164 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:25 smithi196 ceph-mon[36307]: pgmap v762: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-18T16:21:25.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:25 smithi039 ceph-mon[31721]: pgmap v762: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-18T16:21:25.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:25 smithi194 ceph-mon[36200]: pgmap v762: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-18T16:21:26.977 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-03-18T16:21:26.978 INFO:teuthology.orchestra.run.smithi196.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T16:21:26.978 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_recv_messages": 2071, 2024-03-18T16:21:26.978 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_messages": 2172, 2024-03-18T16:21:26.978 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_recv_bytes": 1425757, 2024-03-18T16:21:26.978 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_bytes": 2417622, 2024-03-18T16:21:26.978 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_created_connections": 41, 2024-03-18T16:21:26.978 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_active_connections": 3, 2024-03-18T16:21:26.978 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_total_time": 0.180718154, 2024-03-18T16:21:26.978 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_send_time": 0.062725107, 2024-03-18T16:21:26.978 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_recv_time": 0.108633718, 2024-03-18T16:21:26.978 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_fast_dispatch_time": 0.028097077, 2024-03-18T16:21:26.978 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T16:21:26.978 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 2173, 2024-03-18T16:21:26.978 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.198503859, 2024-03-18T16:21:26.979 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000091350 2024-03-18T16:21:26.979 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:26.979 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_handle_ack_lat": { 2024-03-18T16:21:26.979 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 2149, 2024-03-18T16:21:26.979 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000445884, 2024-03-18T16:21:26.979 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000207 2024-03-18T16:21:26.979 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:26.979 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_recv_encrypted_bytes": 680928, 2024-03-18T16:21:26.979 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_encrypted_bytes": 171872 2024-03-18T16:21:26.979 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:26.979 INFO:teuthology.orchestra.run.smithi196.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T16:21:26.979 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_recv_messages": 538, 2024-03-18T16:21:26.979 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_messages": 721, 2024-03-18T16:21:26.979 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_recv_bytes": 220534, 2024-03-18T16:21:26.979 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_bytes": 876757, 2024-03-18T16:21:26.980 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_created_connections": 16, 2024-03-18T16:21:26.980 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_active_connections": 6, 2024-03-18T16:21:26.980 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_total_time": 0.062152623, 2024-03-18T16:21:26.980 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_send_time": 0.020331268, 2024-03-18T16:21:26.980 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_recv_time": 0.019910556, 2024-03-18T16:21:26.980 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_fast_dispatch_time": 0.018422935, 2024-03-18T16:21:26.980 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T16:21:26.980 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 719, 2024-03-18T16:21:26.980 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.069653807, 2024-03-18T16:21:26.980 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000096875 2024-03-18T16:21:26.980 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:26.980 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_handle_ack_lat": { 2024-03-18T16:21:26.980 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 114, 2024-03-18T16:21:26.980 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000017689, 2024-03-18T16:21:26.980 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000155 2024-03-18T16:21:26.981 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:26.981 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_recv_encrypted_bytes": 3472, 2024-03-18T16:21:26.981 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_encrypted_bytes": 561744 2024-03-18T16:21:26.981 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:26.981 INFO:teuthology.orchestra.run.smithi196.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T16:21:26.981 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_recv_messages": 1963, 2024-03-18T16:21:26.981 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_messages": 2213, 2024-03-18T16:21:26.981 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_recv_bytes": 737814, 2024-03-18T16:21:26.981 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_bytes": 1004648, 2024-03-18T16:21:26.981 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_created_connections": 13, 2024-03-18T16:21:26.981 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_active_connections": 7, 2024-03-18T16:21:26.981 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_total_time": 0.157588549, 2024-03-18T16:21:26.981 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_send_time": 0.055794768, 2024-03-18T16:21:26.981 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_recv_time": 0.117439163, 2024-03-18T16:21:26.981 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_fast_dispatch_time": 0.013764239, 2024-03-18T16:21:26.982 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T16:21:26.982 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 2209, 2024-03-18T16:21:26.982 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.453115695, 2024-03-18T16:21:26.982 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000205122 2024-03-18T16:21:26.982 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:26.982 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_handle_ack_lat": { 2024-03-18T16:21:26.982 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 3045, 2024-03-18T16:21:26.982 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000784530, 2024-03-18T16:21:26.982 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000257 2024-03-18T16:21:26.982 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:26.982 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_recv_encrypted_bytes": 363344, 2024-03-18T16:21:26.982 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_encrypted_bytes": 488624 2024-03-18T16:21:26.982 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:26.982 INFO:teuthology.orchestra.run.smithi196.stdout: "cct": { 2024-03-18T16:21:26.982 INFO:teuthology.orchestra.run.smithi196.stdout: "total_workers": 1, 2024-03-18T16:21:26.983 INFO:teuthology.orchestra.run.smithi196.stdout: "unhealthy_workers": 0 2024-03-18T16:21:26.983 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:26.983 INFO:teuthology.orchestra.run.smithi196.stdout: "finisher-MDSRank": { 2024-03-18T16:21:26.983 INFO:teuthology.orchestra.run.smithi196.stdout: "queue_len": 0, 2024-03-18T16:21:26.983 INFO:teuthology.orchestra.run.smithi196.stdout: "complete_latency": { 2024-03-18T16:21:26.983 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 744, 2024-03-18T16:21:26.983 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.279891719, 2024-03-18T16:21:26.983 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000376198 2024-03-18T16:21:26.983 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-03-18T16:21:26.983 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:26.983 INFO:teuthology.orchestra.run.smithi196.stdout: "finisher-PurgeQueue": { 2024-03-18T16:21:26.983 INFO:teuthology.orchestra.run.smithi196.stdout: "queue_len": 0, 2024-03-18T16:21:26.983 INFO:teuthology.orchestra.run.smithi196.stdout: "complete_latency": { 2024-03-18T16:21:26.983 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 703, 2024-03-18T16:21:26.983 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.025900540, 2024-03-18T16:21:26.984 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000036842 2024-03-18T16:21:26.984 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-03-18T16:21:26.984 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:26.984 INFO:teuthology.orchestra.run.smithi196.stdout: "mds": { 2024-03-18T16:21:26.984 INFO:teuthology.orchestra.run.smithi196.stdout: "request": 103, 2024-03-18T16:21:26.984 INFO:teuthology.orchestra.run.smithi196.stdout: "reply": 97, 2024-03-18T16:21:26.984 INFO:teuthology.orchestra.run.smithi196.stdout: "reply_latency": { 2024-03-18T16:21:26.984 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 97, 2024-03-18T16:21:26.984 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.113248310, 2024-03-18T16:21:26.984 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.001167508 2024-03-18T16:21:26.984 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:26.984 INFO:teuthology.orchestra.run.smithi196.stdout: "slow_reply": 0, 2024-03-18T16:21:26.984 INFO:teuthology.orchestra.run.smithi196.stdout: "forward": 6, 2024-03-18T16:21:26.984 INFO:teuthology.orchestra.run.smithi196.stdout: "dir_fetch_complete": 0, 2024-03-18T16:21:26.985 INFO:teuthology.orchestra.run.smithi196.stdout: "dir_fetch_keys": 0, 2024-03-18T16:21:26.985 INFO:teuthology.orchestra.run.smithi196.stdout: "dir_commit": 11, 2024-03-18T16:21:26.985 INFO:teuthology.orchestra.run.smithi196.stdout: "dir_split": 0, 2024-03-18T16:21:26.985 INFO:teuthology.orchestra.run.smithi196.stdout: "dir_merge": 0, 2024-03-18T16:21:26.985 INFO:teuthology.orchestra.run.smithi196.stdout: "inodes": 12, 2024-03-18T16:21:26.985 INFO:teuthology.orchestra.run.smithi196.stdout: "inodes_top": 1, 2024-03-18T16:21:26.985 INFO:teuthology.orchestra.run.smithi196.stdout: "inodes_bottom": 1, 2024-03-18T16:21:26.985 INFO:teuthology.orchestra.run.smithi196.stdout: "inodes_pin_tail": 10, 2024-03-18T16:21:26.985 INFO:teuthology.orchestra.run.smithi196.stdout: "inodes_pinned": 10, 2024-03-18T16:21:26.985 INFO:teuthology.orchestra.run.smithi196.stdout: "inodes_expired": 41, 2024-03-18T16:21:26.985 INFO:teuthology.orchestra.run.smithi196.stdout: "inodes_with_caps": 1, 2024-03-18T16:21:26.985 INFO:teuthology.orchestra.run.smithi196.stdout: "caps": 1, 2024-03-18T16:21:26.985 INFO:teuthology.orchestra.run.smithi196.stdout: "subtrees": 4, 2024-03-18T16:21:26.985 INFO:teuthology.orchestra.run.smithi196.stdout: "traverse": 224, 2024-03-18T16:21:26.985 INFO:teuthology.orchestra.run.smithi196.stdout: "traverse_hit": 152, 2024-03-18T16:21:26.986 INFO:teuthology.orchestra.run.smithi196.stdout: "traverse_forward": 6, 2024-03-18T16:21:26.986 INFO:teuthology.orchestra.run.smithi196.stdout: "traverse_discover": 6, 2024-03-18T16:21:26.986 INFO:teuthology.orchestra.run.smithi196.stdout: "traverse_dir_fetch": 0, 2024-03-18T16:21:26.986 INFO:teuthology.orchestra.run.smithi196.stdout: "traverse_remote_ino": 0, 2024-03-18T16:21:26.986 INFO:teuthology.orchestra.run.smithi196.stdout: "traverse_lock": 0, 2024-03-18T16:21:26.986 INFO:teuthology.orchestra.run.smithi196.stdout: "load_cent": 0, 2024-03-18T16:21:26.986 INFO:teuthology.orchestra.run.smithi196.stdout: "q": 0, 2024-03-18T16:21:26.986 INFO:teuthology.orchestra.run.smithi196.stdout: "exported": 4, 2024-03-18T16:21:26.986 INFO:teuthology.orchestra.run.smithi196.stdout: "exported_inodes": 26, 2024-03-18T16:21:26.986 INFO:teuthology.orchestra.run.smithi196.stdout: "imported": 4, 2024-03-18T16:21:26.986 INFO:teuthology.orchestra.run.smithi196.stdout: "imported_inodes": 32, 2024-03-18T16:21:26.986 INFO:teuthology.orchestra.run.smithi196.stdout: "openino_dir_fetch": 0, 2024-03-18T16:21:26.986 INFO:teuthology.orchestra.run.smithi196.stdout: "openino_backtrace_fetch": 358, 2024-03-18T16:21:26.986 INFO:teuthology.orchestra.run.smithi196.stdout: "openino_peer_discover": 0, 2024-03-18T16:21:26.986 INFO:teuthology.orchestra.run.smithi196.stdout: "root_rfiles": 112, 2024-03-18T16:21:26.987 INFO:teuthology.orchestra.run.smithi196.stdout: "root_rbytes": 933339, 2024-03-18T16:21:26.987 INFO:teuthology.orchestra.run.smithi196.stdout: "root_rsnaps": 0, 2024-03-18T16:21:26.987 INFO:teuthology.orchestra.run.smithi196.stdout: "scrub_backtrace_fetch": 358, 2024-03-18T16:21:26.987 INFO:teuthology.orchestra.run.smithi196.stdout: "scrub_set_tag": 0, 2024-03-18T16:21:26.987 INFO:teuthology.orchestra.run.smithi196.stdout: "scrub_backtrace_repaired": 0, 2024-03-18T16:21:26.987 INFO:teuthology.orchestra.run.smithi196.stdout: "scrub_inotable_repaired": 0, 2024-03-18T16:21:26.987 INFO:teuthology.orchestra.run.smithi196.stdout: "scrub_dir_inodes": 0, 2024-03-18T16:21:26.987 INFO:teuthology.orchestra.run.smithi196.stdout: "scrub_dir_base_inodes": 0, 2024-03-18T16:21:26.987 INFO:teuthology.orchestra.run.smithi196.stdout: "scrub_dirfrag_rstats": 0, 2024-03-18T16:21:26.987 INFO:teuthology.orchestra.run.smithi196.stdout: "scrub_file_inodes": 358, 2024-03-18T16:21:26.987 INFO:teuthology.orchestra.run.smithi196.stdout: "handle_inode_file_caps": 0, 2024-03-18T16:21:26.987 INFO:teuthology.orchestra.run.smithi196.stdout: "ceph_cap_op_revoke": 5, 2024-03-18T16:21:26.987 INFO:teuthology.orchestra.run.smithi196.stdout: "ceph_cap_op_grant": 287, 2024-03-18T16:21:26.987 INFO:teuthology.orchestra.run.smithi196.stdout: "ceph_cap_op_trunc": 0, 2024-03-18T16:21:26.987 INFO:teuthology.orchestra.run.smithi196.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-18T16:21:26.988 INFO:teuthology.orchestra.run.smithi196.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-18T16:21:26.988 INFO:teuthology.orchestra.run.smithi196.stdout: "handle_client_caps": 789, 2024-03-18T16:21:26.988 INFO:teuthology.orchestra.run.smithi196.stdout: "handle_client_caps_dirty": 613, 2024-03-18T16:21:26.988 INFO:teuthology.orchestra.run.smithi196.stdout: "handle_client_cap_release": 9, 2024-03-18T16:21:26.988 INFO:teuthology.orchestra.run.smithi196.stdout: "process_request_cap_release": 62 2024-03-18T16:21:26.988 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:26.988 INFO:teuthology.orchestra.run.smithi196.stdout: "mds_cache": { 2024-03-18T16:21:26.988 INFO:teuthology.orchestra.run.smithi196.stdout: "dir_update": 0, 2024-03-18T16:21:26.988 INFO:teuthology.orchestra.run.smithi196.stdout: "dir_update_receipt": 3, 2024-03-18T16:21:26.988 INFO:teuthology.orchestra.run.smithi196.stdout: "dir_try_discover": 2, 2024-03-18T16:21:26.988 INFO:teuthology.orchestra.run.smithi196.stdout: "dir_send_discover": 9, 2024-03-18T16:21:26.988 INFO:teuthology.orchestra.run.smithi196.stdout: "dir_handle_discover": 2, 2024-03-18T16:21:26.988 INFO:teuthology.orchestra.run.smithi196.stdout: "num_strays": 0, 2024-03-18T16:21:26.988 INFO:teuthology.orchestra.run.smithi196.stdout: "num_strays_delayed": 0, 2024-03-18T16:21:26.988 INFO:teuthology.orchestra.run.smithi196.stdout: "num_strays_enqueuing": 0, 2024-03-18T16:21:26.988 INFO:teuthology.orchestra.run.smithi196.stdout: "strays_created": 30, 2024-03-18T16:21:26.989 INFO:teuthology.orchestra.run.smithi196.stdout: "strays_enqueued": 30, 2024-03-18T16:21:26.989 INFO:teuthology.orchestra.run.smithi196.stdout: "strays_reintegrated": 0, 2024-03-18T16:21:26.989 INFO:teuthology.orchestra.run.smithi196.stdout: "strays_migrated": 0, 2024-03-18T16:21:26.989 INFO:teuthology.orchestra.run.smithi196.stdout: "num_recovering_processing": 0, 2024-03-18T16:21:26.989 INFO:teuthology.orchestra.run.smithi196.stdout: "num_recovering_enqueued": 0, 2024-03-18T16:21:26.989 INFO:teuthology.orchestra.run.smithi196.stdout: "num_recovering_prioritized": 0, 2024-03-18T16:21:26.989 INFO:teuthology.orchestra.run.smithi196.stdout: "recovery_started": 0, 2024-03-18T16:21:26.989 INFO:teuthology.orchestra.run.smithi196.stdout: "recovery_completed": 0, 2024-03-18T16:21:26.989 INFO:teuthology.orchestra.run.smithi196.stdout: "ireq_quiesce_path": 0, 2024-03-18T16:21:26.989 INFO:teuthology.orchestra.run.smithi196.stdout: "ireq_quiesce_inode": 0, 2024-03-18T16:21:26.989 INFO:teuthology.orchestra.run.smithi196.stdout: "ireq_enqueue_scrub": 0, 2024-03-18T16:21:26.989 INFO:teuthology.orchestra.run.smithi196.stdout: "ireq_exportdir": 4, 2024-03-18T16:21:26.989 INFO:teuthology.orchestra.run.smithi196.stdout: "ireq_flush": 0, 2024-03-18T16:21:26.989 INFO:teuthology.orchestra.run.smithi196.stdout: "ireq_fragmentdir": 0, 2024-03-18T16:21:26.989 INFO:teuthology.orchestra.run.smithi196.stdout: "ireq_fragstats": 0, 2024-03-18T16:21:26.990 INFO:teuthology.orchestra.run.smithi196.stdout: "ireq_inodestats": 0 2024-03-18T16:21:26.990 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:26.990 INFO:teuthology.orchestra.run.smithi196.stdout: "mds_log": { 2024-03-18T16:21:26.990 INFO:teuthology.orchestra.run.smithi196.stdout: "evlrg": 0, 2024-03-18T16:21:26.990 INFO:teuthology.orchestra.run.smithi196.stdout: "evadd": 931, 2024-03-18T16:21:26.990 INFO:teuthology.orchestra.run.smithi196.stdout: "evex": 0, 2024-03-18T16:21:26.990 INFO:teuthology.orchestra.run.smithi196.stdout: "evtrm": 0, 2024-03-18T16:21:26.990 INFO:teuthology.orchestra.run.smithi196.stdout: "ev": 931, 2024-03-18T16:21:26.990 INFO:teuthology.orchestra.run.smithi196.stdout: "evexg": 0, 2024-03-18T16:21:26.990 INFO:teuthology.orchestra.run.smithi196.stdout: "evexd": 0, 2024-03-18T16:21:26.990 INFO:teuthology.orchestra.run.smithi196.stdout: "segadd": 2, 2024-03-18T16:21:26.990 INFO:teuthology.orchestra.run.smithi196.stdout: "segex": 0, 2024-03-18T16:21:26.990 INFO:teuthology.orchestra.run.smithi196.stdout: "segtrm": 0, 2024-03-18T16:21:26.990 INFO:teuthology.orchestra.run.smithi196.stdout: "seg": 2, 2024-03-18T16:21:26.991 INFO:teuthology.orchestra.run.smithi196.stdout: "segmjr": 2, 2024-03-18T16:21:26.991 INFO:teuthology.orchestra.run.smithi196.stdout: "segexg": 0, 2024-03-18T16:21:26.991 INFO:teuthology.orchestra.run.smithi196.stdout: "segexd": 0, 2024-03-18T16:21:26.991 INFO:teuthology.orchestra.run.smithi196.stdout: "expos": 4194304, 2024-03-18T16:21:26.991 INFO:teuthology.orchestra.run.smithi196.stdout: "wrpos": 5343855, 2024-03-18T16:21:26.991 INFO:teuthology.orchestra.run.smithi196.stdout: "rdpos": 0, 2024-03-18T16:21:26.991 INFO:teuthology.orchestra.run.smithi196.stdout: "jlat": { 2024-03-18T16:21:26.991 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 148, 2024-03-18T16:21:26.991 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 2.187299138, 2024-03-18T16:21:26.991 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.014779048 2024-03-18T16:21:26.991 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:26.991 INFO:teuthology.orchestra.run.smithi196.stdout: "replayed": 0 2024-03-18T16:21:26.991 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:26.991 INFO:teuthology.orchestra.run.smithi196.stdout: "mds_mem": { 2024-03-18T16:21:26.991 INFO:teuthology.orchestra.run.smithi196.stdout: "ino": 17, 2024-03-18T16:21:26.992 INFO:teuthology.orchestra.run.smithi196.stdout: "ino+": 58, 2024-03-18T16:21:26.992 INFO:teuthology.orchestra.run.smithi196.stdout: "ino-": 41, 2024-03-18T16:21:26.992 INFO:teuthology.orchestra.run.smithi196.stdout: "dir": 16, 2024-03-18T16:21:26.992 INFO:teuthology.orchestra.run.smithi196.stdout: "dir+": 29, 2024-03-18T16:21:26.992 INFO:teuthology.orchestra.run.smithi196.stdout: "dir-": 13, 2024-03-18T16:21:26.992 INFO:teuthology.orchestra.run.smithi196.stdout: "dn": 12, 2024-03-18T16:21:26.992 INFO:teuthology.orchestra.run.smithi196.stdout: "dn+": 89, 2024-03-18T16:21:26.992 INFO:teuthology.orchestra.run.smithi196.stdout: "dn-": 77, 2024-03-18T16:21:26.992 INFO:teuthology.orchestra.run.smithi196.stdout: "cap": 1, 2024-03-18T16:21:26.992 INFO:teuthology.orchestra.run.smithi196.stdout: "cap+": 50, 2024-03-18T16:21:26.992 INFO:teuthology.orchestra.run.smithi196.stdout: "cap-": 49, 2024-03-18T16:21:26.992 INFO:teuthology.orchestra.run.smithi196.stdout: "rss": 54720, 2024-03-18T16:21:26.992 INFO:teuthology.orchestra.run.smithi196.stdout: "heap": 223612 2024-03-18T16:21:26.992 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:26.992 INFO:teuthology.orchestra.run.smithi196.stdout: "mds_server": { 2024-03-18T16:21:26.992 INFO:teuthology.orchestra.run.smithi196.stdout: "dispatch_client_request": 217, 2024-03-18T16:21:26.993 INFO:teuthology.orchestra.run.smithi196.stdout: "dispatch_server_request": 4, 2024-03-18T16:21:26.993 INFO:teuthology.orchestra.run.smithi196.stdout: "handle_client_request": 103, 2024-03-18T16:21:26.993 INFO:teuthology.orchestra.run.smithi196.stdout: "handle_client_session": 9, 2024-03-18T16:21:26.993 INFO:teuthology.orchestra.run.smithi196.stdout: "handle_peer_request": 42, 2024-03-18T16:21:26.993 INFO:teuthology.orchestra.run.smithi196.stdout: "req_create_latency": { 2024-03-18T16:21:26.993 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:26.993 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:26.993 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:26.993 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:26.993 INFO:teuthology.orchestra.run.smithi196.stdout: "req_getattr_latency": { 2024-03-18T16:21:26.993 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:26.993 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:26.993 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:26.993 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:26.993 INFO:teuthology.orchestra.run.smithi196.stdout: "req_getfilelock_latency": { 2024-03-18T16:21:26.994 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:26.994 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:26.994 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:26.994 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:26.994 INFO:teuthology.orchestra.run.smithi196.stdout: "req_link_latency": { 2024-03-18T16:21:26.994 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:26.994 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:26.994 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:26.994 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:26.994 INFO:teuthology.orchestra.run.smithi196.stdout: "req_lookup_latency": { 2024-03-18T16:21:26.994 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 51, 2024-03-18T16:21:26.994 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.025119858, 2024-03-18T16:21:26.994 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000492546 2024-03-18T16:21:26.994 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:26.994 INFO:teuthology.orchestra.run.smithi196.stdout: "req_lookuphash_latency": { 2024-03-18T16:21:26.995 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:26.995 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:26.995 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:26.995 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:26.995 INFO:teuthology.orchestra.run.smithi196.stdout: "req_lookupino_latency": { 2024-03-18T16:21:26.995 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 1, 2024-03-18T16:21:26.995 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000094217, 2024-03-18T16:21:26.995 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000094217 2024-03-18T16:21:26.995 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:26.995 INFO:teuthology.orchestra.run.smithi196.stdout: "req_lookupname_latency": { 2024-03-18T16:21:26.995 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:26.995 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:26.995 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:26.995 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:26.995 INFO:teuthology.orchestra.run.smithi196.stdout: "req_lookupparent_latency": { 2024-03-18T16:21:26.996 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:26.996 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:26.996 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:26.996 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:26.996 INFO:teuthology.orchestra.run.smithi196.stdout: "req_lookupsnap_latency": { 2024-03-18T16:21:26.996 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:26.996 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:26.996 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:26.996 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:26.996 INFO:teuthology.orchestra.run.smithi196.stdout: "req_lssnap_latency": { 2024-03-18T16:21:26.996 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:26.996 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:26.996 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:26.997 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:26.997 INFO:teuthology.orchestra.run.smithi196.stdout: "req_mkdir_latency": { 2024-03-18T16:21:26.997 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:26.997 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:26.997 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:26.997 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:26.997 INFO:teuthology.orchestra.run.smithi196.stdout: "req_mknod_latency": { 2024-03-18T16:21:26.997 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:26.997 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:26.998 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:26.998 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:26.998 INFO:teuthology.orchestra.run.smithi196.stdout: "req_mksnap_latency": { 2024-03-18T16:21:26.998 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:26.998 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:26.998 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:26.998 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:26.998 INFO:teuthology.orchestra.run.smithi196.stdout: "req_open_latency": { 2024-03-18T16:21:26.998 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 11, 2024-03-18T16:21:26.998 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.009003797, 2024-03-18T16:21:26.998 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000818527 2024-03-18T16:21:26.998 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:26.998 INFO:teuthology.orchestra.run.smithi196.stdout: "req_readdir_latency": { 2024-03-18T16:21:26.998 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 4, 2024-03-18T16:21:26.998 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.002606993, 2024-03-18T16:21:26.999 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000651748 2024-03-18T16:21:26.999 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:26.999 INFO:teuthology.orchestra.run.smithi196.stdout: "req_rename_latency": { 2024-03-18T16:21:26.999 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:26.999 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:26.999 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:26.999 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:26.999 INFO:teuthology.orchestra.run.smithi196.stdout: "req_renamesnap_latency": { 2024-03-18T16:21:26.999 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:26.999 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:26.999 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:26.999 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:26.999 INFO:teuthology.orchestra.run.smithi196.stdout: "req_snapdiff_latency": { 2024-03-18T16:21:26.999 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:26.999 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:27.000 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:27.000 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:27.000 INFO:teuthology.orchestra.run.smithi196.stdout: "req_rmdir_latency": { 2024-03-18T16:21:27.000 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:27.000 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:27.000 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:27.000 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:27.000 INFO:teuthology.orchestra.run.smithi196.stdout: "req_rmsnap_latency": { 2024-03-18T16:21:27.000 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:27.000 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:27.000 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:27.000 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:27.000 INFO:teuthology.orchestra.run.smithi196.stdout: "req_rmxattr_latency": { 2024-03-18T16:21:27.000 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:27.000 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:27.001 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:27.001 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:27.001 INFO:teuthology.orchestra.run.smithi196.stdout: "req_setattr_latency": { 2024-03-18T16:21:27.001 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:27.001 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:27.001 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:27.001 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:27.001 INFO:teuthology.orchestra.run.smithi196.stdout: "req_setdirlayout_latency": { 2024-03-18T16:21:27.001 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:27.001 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:27.001 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:27.001 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:27.001 INFO:teuthology.orchestra.run.smithi196.stdout: "req_setfilelock_latency": { 2024-03-18T16:21:27.001 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:27.002 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:27.002 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:27.002 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:27.002 INFO:teuthology.orchestra.run.smithi196.stdout: "req_setlayout_latency": { 2024-03-18T16:21:27.002 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:27.002 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:27.002 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:27.002 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:27.002 INFO:teuthology.orchestra.run.smithi196.stdout: "req_setxattr_latency": { 2024-03-18T16:21:27.002 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:27.002 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:27.002 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:27.002 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:27.002 INFO:teuthology.orchestra.run.smithi196.stdout: "req_symlink_latency": { 2024-03-18T16:21:27.003 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:27.003 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:27.003 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:27.003 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:27.003 INFO:teuthology.orchestra.run.smithi196.stdout: "req_unlink_latency": { 2024-03-18T16:21:27.003 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 30, 2024-03-18T16:21:27.003 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.076423445, 2024-03-18T16:21:27.003 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.002547448 2024-03-18T16:21:27.003 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:27.003 INFO:teuthology.orchestra.run.smithi196.stdout: "cap_revoke_eviction": 0, 2024-03-18T16:21:27.003 INFO:teuthology.orchestra.run.smithi196.stdout: "cap_acquisition_throttle": 0, 2024-03-18T16:21:27.003 INFO:teuthology.orchestra.run.smithi196.stdout: "req_getvxattr_latency": { 2024-03-18T16:21:27.003 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:27.003 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:27.003 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:27.003 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-03-18T16:21:27.004 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:27.004 INFO:teuthology.orchestra.run.smithi196.stdout: "mds_sessions": { 2024-03-18T16:21:27.004 INFO:teuthology.orchestra.run.smithi196.stdout: "session_count": 1, 2024-03-18T16:21:27.004 INFO:teuthology.orchestra.run.smithi196.stdout: "session_add": 1, 2024-03-18T16:21:27.004 INFO:teuthology.orchestra.run.smithi196.stdout: "session_remove": 0, 2024-03-18T16:21:27.004 INFO:teuthology.orchestra.run.smithi196.stdout: "sessions_open": 1, 2024-03-18T16:21:27.004 INFO:teuthology.orchestra.run.smithi196.stdout: "sessions_stale": 0, 2024-03-18T16:21:27.004 INFO:teuthology.orchestra.run.smithi196.stdout: "total_load": 62, 2024-03-18T16:21:27.004 INFO:teuthology.orchestra.run.smithi196.stdout: "average_load": 62, 2024-03-18T16:21:27.004 INFO:teuthology.orchestra.run.smithi196.stdout: "avg_session_uptime": 679, 2024-03-18T16:21:27.004 INFO:teuthology.orchestra.run.smithi196.stdout: "mdthresh_evicted": 0 2024-03-18T16:21:27.004 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:27.004 INFO:teuthology.orchestra.run.smithi196.stdout: "mempool": { 2024-03-18T16:21:27.004 INFO:teuthology.orchestra.run.smithi196.stdout: "bloom_filter_bytes": 0, 2024-03-18T16:21:27.005 INFO:teuthology.orchestra.run.smithi196.stdout: "bloom_filter_items": 0, 2024-03-18T16:21:27.005 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T16:21:27.005 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_alloc_items": 0, 2024-03-18T16:21:27.005 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T16:21:27.005 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_cache_data_items": 0, 2024-03-18T16:21:27.005 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T16:21:27.005 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T16:21:27.005 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T16:21:27.005 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T16:21:27.005 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T16:21:27.005 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_cache_other_items": 0, 2024-03-18T16:21:27.005 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T16:21:27.005 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T16:21:27.005 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_extent_bytes": 0, 2024-03-18T16:21:27.005 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_extent_items": 0, 2024-03-18T16:21:27.006 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_blob_bytes": 0, 2024-03-18T16:21:27.006 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_blob_items": 0, 2024-03-18T16:21:27.006 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T16:21:27.006 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T16:21:27.006 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T16:21:27.006 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T16:21:27.006 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T16:21:27.006 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_fsck_items": 0, 2024-03-18T16:21:27.006 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_txc_bytes": 0, 2024-03-18T16:21:27.006 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_txc_items": 0, 2024-03-18T16:21:27.006 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T16:21:27.006 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T16:21:27.006 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_writing_bytes": 0, 2024-03-18T16:21:27.006 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_writing_items": 0, 2024-03-18T16:21:27.006 INFO:teuthology.orchestra.run.smithi196.stdout: "bluefs_bytes": 0, 2024-03-18T16:21:27.007 INFO:teuthology.orchestra.run.smithi196.stdout: "bluefs_items": 0, 2024-03-18T16:21:27.007 INFO:teuthology.orchestra.run.smithi196.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T16:21:27.007 INFO:teuthology.orchestra.run.smithi196.stdout: "bluefs_file_reader_items": 0, 2024-03-18T16:21:27.007 INFO:teuthology.orchestra.run.smithi196.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T16:21:27.007 INFO:teuthology.orchestra.run.smithi196.stdout: "bluefs_file_writer_items": 0, 2024-03-18T16:21:27.007 INFO:teuthology.orchestra.run.smithi196.stdout: "buffer_anon_bytes": 293095, 2024-03-18T16:21:27.007 INFO:teuthology.orchestra.run.smithi196.stdout: "buffer_anon_items": 117, 2024-03-18T16:21:27.007 INFO:teuthology.orchestra.run.smithi196.stdout: "buffer_meta_bytes": 0, 2024-03-18T16:21:27.007 INFO:teuthology.orchestra.run.smithi196.stdout: "buffer_meta_items": 0, 2024-03-18T16:21:27.007 INFO:teuthology.orchestra.run.smithi196.stdout: "osd_bytes": 0, 2024-03-18T16:21:27.007 INFO:teuthology.orchestra.run.smithi196.stdout: "osd_items": 0, 2024-03-18T16:21:27.007 INFO:teuthology.orchestra.run.smithi196.stdout: "osd_mapbl_bytes": 0, 2024-03-18T16:21:27.007 INFO:teuthology.orchestra.run.smithi196.stdout: "osd_mapbl_items": 0, 2024-03-18T16:21:27.007 INFO:teuthology.orchestra.run.smithi196.stdout: "osd_pglog_bytes": 0, 2024-03-18T16:21:27.007 INFO:teuthology.orchestra.run.smithi196.stdout: "osd_pglog_items": 0, 2024-03-18T16:21:27.008 INFO:teuthology.orchestra.run.smithi196.stdout: "osdmap_bytes": 8888, 2024-03-18T16:21:27.008 INFO:teuthology.orchestra.run.smithi196.stdout: "osdmap_items": 170, 2024-03-18T16:21:27.008 INFO:teuthology.orchestra.run.smithi196.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T16:21:27.008 INFO:teuthology.orchestra.run.smithi196.stdout: "osdmap_mapping_items": 0, 2024-03-18T16:21:27.008 INFO:teuthology.orchestra.run.smithi196.stdout: "pgmap_bytes": 0, 2024-03-18T16:21:27.008 INFO:teuthology.orchestra.run.smithi196.stdout: "pgmap_items": 0, 2024-03-18T16:21:27.008 INFO:teuthology.orchestra.run.smithi196.stdout: "mds_co_bytes": 67952, 2024-03-18T16:21:27.008 INFO:teuthology.orchestra.run.smithi196.stdout: "mds_co_items": 271, 2024-03-18T16:21:27.008 INFO:teuthology.orchestra.run.smithi196.stdout: "unittest_1_bytes": 0, 2024-03-18T16:21:27.008 INFO:teuthology.orchestra.run.smithi196.stdout: "unittest_1_items": 0, 2024-03-18T16:21:27.008 INFO:teuthology.orchestra.run.smithi196.stdout: "unittest_2_bytes": 0, 2024-03-18T16:21:27.008 INFO:teuthology.orchestra.run.smithi196.stdout: "unittest_2_items": 0 2024-03-18T16:21:27.008 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:27.008 INFO:teuthology.orchestra.run.smithi196.stdout: "objecter": { 2024-03-18T16:21:27.008 INFO:teuthology.orchestra.run.smithi196.stdout: "op_active": 0, 2024-03-18T16:21:27.008 INFO:teuthology.orchestra.run.smithi196.stdout: "op_laggy": 0, 2024-03-18T16:21:27.009 INFO:teuthology.orchestra.run.smithi196.stdout: "op_send": 1333, 2024-03-18T16:21:27.009 INFO:teuthology.orchestra.run.smithi196.stdout: "op_send_bytes": 1178540, 2024-03-18T16:21:27.009 INFO:teuthology.orchestra.run.smithi196.stdout: "op_resend": 33, 2024-03-18T16:21:27.009 INFO:teuthology.orchestra.run.smithi196.stdout: "op_reply": 1333, 2024-03-18T16:21:27.009 INFO:teuthology.orchestra.run.smithi196.stdout: "op_latency": { 2024-03-18T16:21:27.009 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 1333, 2024-03-18T16:21:27.009 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 172.334513327, 2024-03-18T16:21:27.009 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.129283205 2024-03-18T16:21:27.009 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:27.009 INFO:teuthology.orchestra.run.smithi196.stdout: "op_inflight": 0, 2024-03-18T16:21:27.009 INFO:teuthology.orchestra.run.smithi196.stdout: "oplen_avg": { 2024-03-18T16:21:27.009 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 1333, 2024-03-18T16:21:27.009 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 1429 2024-03-18T16:21:27.009 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:27.010 INFO:teuthology.orchestra.run.smithi196.stdout: "op": 1333, 2024-03-18T16:21:27.010 INFO:teuthology.orchestra.run.smithi196.stdout: "op_r": 359, 2024-03-18T16:21:27.010 INFO:teuthology.orchestra.run.smithi196.stdout: "op_w": 974, 2024-03-18T16:21:27.010 INFO:teuthology.orchestra.run.smithi196.stdout: "op_rmw": 0, 2024-03-18T16:21:27.010 INFO:teuthology.orchestra.run.smithi196.stdout: "op_pg": 0, 2024-03-18T16:21:27.010 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_stat": 0, 2024-03-18T16:21:27.010 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_create": 10, 2024-03-18T16:21:27.010 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_read": 1, 2024-03-18T16:21:27.010 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_write": 149, 2024-03-18T16:21:27.010 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_writefull": 34, 2024-03-18T16:21:27.010 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_writesame": 0, 2024-03-18T16:21:27.010 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_append": 0, 2024-03-18T16:21:27.010 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_zero": 0, 2024-03-18T16:21:27.010 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_truncate": 0, 2024-03-18T16:21:27.010 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_delete": 696, 2024-03-18T16:21:27.011 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_mapext": 0, 2024-03-18T16:21:27.011 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_sparse_read": 0, 2024-03-18T16:21:27.011 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_clonerange": 0, 2024-03-18T16:21:27.011 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_getxattr": 358, 2024-03-18T16:21:27.011 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_setxattr": 20, 2024-03-18T16:21:27.011 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_cmpxattr": 0, 2024-03-18T16:21:27.011 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_rmxattr": 0, 2024-03-18T16:21:27.011 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_resetxattrs": 0, 2024-03-18T16:21:27.011 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_call": 0, 2024-03-18T16:21:27.011 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_watch": 0, 2024-03-18T16:21:27.011 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_notify": 0, 2024-03-18T16:21:27.011 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_src_cmpxattr": 0, 2024-03-18T16:21:27.011 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_pgls": 0, 2024-03-18T16:21:27.011 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_pgls_filter": 0, 2024-03-18T16:21:27.011 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_other": 0, 2024-03-18T16:21:27.012 INFO:teuthology.orchestra.run.smithi196.stdout: "linger_active": 0, 2024-03-18T16:21:27.012 INFO:teuthology.orchestra.run.smithi196.stdout: "linger_send": 0, 2024-03-18T16:21:27.012 INFO:teuthology.orchestra.run.smithi196.stdout: "linger_resend": 0, 2024-03-18T16:21:27.012 INFO:teuthology.orchestra.run.smithi196.stdout: "linger_ping": 0, 2024-03-18T16:21:27.012 INFO:teuthology.orchestra.run.smithi196.stdout: "poolop_active": 0, 2024-03-18T16:21:27.012 INFO:teuthology.orchestra.run.smithi196.stdout: "poolop_send": 0, 2024-03-18T16:21:27.012 INFO:teuthology.orchestra.run.smithi196.stdout: "poolop_resend": 0, 2024-03-18T16:21:27.012 INFO:teuthology.orchestra.run.smithi196.stdout: "poolstat_active": 0, 2024-03-18T16:21:27.012 INFO:teuthology.orchestra.run.smithi196.stdout: "poolstat_send": 0, 2024-03-18T16:21:27.012 INFO:teuthology.orchestra.run.smithi196.stdout: "poolstat_resend": 0, 2024-03-18T16:21:27.012 INFO:teuthology.orchestra.run.smithi196.stdout: "statfs_active": 0, 2024-03-18T16:21:27.012 INFO:teuthology.orchestra.run.smithi196.stdout: "statfs_send": 0, 2024-03-18T16:21:27.012 INFO:teuthology.orchestra.run.smithi196.stdout: "statfs_resend": 0, 2024-03-18T16:21:27.012 INFO:teuthology.orchestra.run.smithi196.stdout: "command_active": 0, 2024-03-18T16:21:27.012 INFO:teuthology.orchestra.run.smithi196.stdout: "command_send": 0, 2024-03-18T16:21:27.012 INFO:teuthology.orchestra.run.smithi196.stdout: "command_resend": 0, 2024-03-18T16:21:27.013 INFO:teuthology.orchestra.run.smithi196.stdout: "map_epoch": 93, 2024-03-18T16:21:27.013 INFO:teuthology.orchestra.run.smithi196.stdout: "map_full": 0, 2024-03-18T16:21:27.013 INFO:teuthology.orchestra.run.smithi196.stdout: "map_inc": 4, 2024-03-18T16:21:27.013 INFO:teuthology.orchestra.run.smithi196.stdout: "osd_sessions": 12, 2024-03-18T16:21:27.013 INFO:teuthology.orchestra.run.smithi196.stdout: "osd_session_open": 12, 2024-03-18T16:21:27.013 INFO:teuthology.orchestra.run.smithi196.stdout: "osd_session_close": 0, 2024-03-18T16:21:27.013 INFO:teuthology.orchestra.run.smithi196.stdout: "osd_laggy": 0, 2024-03-18T16:21:27.013 INFO:teuthology.orchestra.run.smithi196.stdout: "omap_wr": 108, 2024-03-18T16:21:27.013 INFO:teuthology.orchestra.run.smithi196.stdout: "omap_rd": 0, 2024-03-18T16:21:27.013 INFO:teuthology.orchestra.run.smithi196.stdout: "omap_del": 53 2024-03-18T16:21:27.013 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:27.013 INFO:teuthology.orchestra.run.smithi196.stdout: "oft": { 2024-03-18T16:21:27.013 INFO:teuthology.orchestra.run.smithi196.stdout: "omap_total_objs": 1, 2024-03-18T16:21:27.013 INFO:teuthology.orchestra.run.smithi196.stdout: "omap_total_kv_pairs": 0, 2024-03-18T16:21:27.013 INFO:teuthology.orchestra.run.smithi196.stdout: "omap_total_updates": 0, 2024-03-18T16:21:27.014 INFO:teuthology.orchestra.run.smithi196.stdout: "omap_total_removes": 0 2024-03-18T16:21:27.014 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:27.014 INFO:teuthology.orchestra.run.smithi196.stdout: "osd-slow-ops": { 2024-03-18T16:21:27.014 INFO:teuthology.orchestra.run.smithi196.stdout: "slow_ops_count": 0 2024-03-18T16:21:27.014 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:27.014 INFO:teuthology.orchestra.run.smithi196.stdout: "purge_queue": { 2024-03-18T16:21:27.014 INFO:teuthology.orchestra.run.smithi196.stdout: "pq_executing_ops": 0, 2024-03-18T16:21:27.014 INFO:teuthology.orchestra.run.smithi196.stdout: "pq_executing_ops_high_water": 26, 2024-03-18T16:21:27.014 INFO:teuthology.orchestra.run.smithi196.stdout: "pq_executing": 0, 2024-03-18T16:21:27.014 INFO:teuthology.orchestra.run.smithi196.stdout: "pq_executing_high_water": 4, 2024-03-18T16:21:27.014 INFO:teuthology.orchestra.run.smithi196.stdout: "pq_executed_ops": 684, 2024-03-18T16:21:27.014 INFO:teuthology.orchestra.run.smithi196.stdout: "pq_executed": 30, 2024-03-18T16:21:27.014 INFO:teuthology.orchestra.run.smithi196.stdout: "pq_item_in_journal": 0 2024-03-18T16:21:27.014 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:27.014 INFO:teuthology.orchestra.run.smithi196.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T16:21:27.015 INFO:teuthology.orchestra.run.smithi196.stdout: "val": 0, 2024-03-18T16:21:27.015 INFO:teuthology.orchestra.run.smithi196.stdout: "max": 104857600, 2024-03-18T16:21:27.015 INFO:teuthology.orchestra.run.smithi196.stdout: "get_started": 0, 2024-03-18T16:21:27.015 INFO:teuthology.orchestra.run.smithi196.stdout: "get": 4568, 2024-03-18T16:21:27.015 INFO:teuthology.orchestra.run.smithi196.stdout: "get_sum": 1913428, 2024-03-18T16:21:27.015 INFO:teuthology.orchestra.run.smithi196.stdout: "get_or_fail_fail": 0, 2024-03-18T16:21:27.015 INFO:teuthology.orchestra.run.smithi196.stdout: "get_or_fail_success": 4568, 2024-03-18T16:21:27.015 INFO:teuthology.orchestra.run.smithi196.stdout: "take": 0, 2024-03-18T16:21:27.015 INFO:teuthology.orchestra.run.smithi196.stdout: "take_sum": 0, 2024-03-18T16:21:27.015 INFO:teuthology.orchestra.run.smithi196.stdout: "put": 4568, 2024-03-18T16:21:27.015 INFO:teuthology.orchestra.run.smithi196.stdout: "put_sum": 1913428, 2024-03-18T16:21:27.015 INFO:teuthology.orchestra.run.smithi196.stdout: "wait": { 2024-03-18T16:21:27.015 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:27.015 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:27.015 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:27.016 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-03-18T16:21:27.016 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:27.016 INFO:teuthology.orchestra.run.smithi196.stdout: "throttle-objecter_bytes": { 2024-03-18T16:21:27.016 INFO:teuthology.orchestra.run.smithi196.stdout: "val": 0, 2024-03-18T16:21:27.016 INFO:teuthology.orchestra.run.smithi196.stdout: "max": 104857600, 2024-03-18T16:21:27.016 INFO:teuthology.orchestra.run.smithi196.stdout: "get_started": 0, 2024-03-18T16:21:27.016 INFO:teuthology.orchestra.run.smithi196.stdout: "get": 0, 2024-03-18T16:21:27.016 INFO:teuthology.orchestra.run.smithi196.stdout: "get_sum": 0, 2024-03-18T16:21:27.016 INFO:teuthology.orchestra.run.smithi196.stdout: "get_or_fail_fail": 0, 2024-03-18T16:21:27.016 INFO:teuthology.orchestra.run.smithi196.stdout: "get_or_fail_success": 0, 2024-03-18T16:21:27.016 INFO:teuthology.orchestra.run.smithi196.stdout: "take": 1333, 2024-03-18T16:21:27.016 INFO:teuthology.orchestra.run.smithi196.stdout: "take_sum": 1181810, 2024-03-18T16:21:27.016 INFO:teuthology.orchestra.run.smithi196.stdout: "put": 1333, 2024-03-18T16:21:27.016 INFO:teuthology.orchestra.run.smithi196.stdout: "put_sum": 1181810, 2024-03-18T16:21:27.016 INFO:teuthology.orchestra.run.smithi196.stdout: "wait": { 2024-03-18T16:21:27.017 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:27.017 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:27.017 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:27.017 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-03-18T16:21:27.017 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:27.017 INFO:teuthology.orchestra.run.smithi196.stdout: "throttle-objecter_ops": { 2024-03-18T16:21:27.017 INFO:teuthology.orchestra.run.smithi196.stdout: "val": 0, 2024-03-18T16:21:27.017 INFO:teuthology.orchestra.run.smithi196.stdout: "max": 1024, 2024-03-18T16:21:27.017 INFO:teuthology.orchestra.run.smithi196.stdout: "get_started": 0, 2024-03-18T16:21:27.017 INFO:teuthology.orchestra.run.smithi196.stdout: "get": 0, 2024-03-18T16:21:27.017 INFO:teuthology.orchestra.run.smithi196.stdout: "get_sum": 0, 2024-03-18T16:21:27.017 INFO:teuthology.orchestra.run.smithi196.stdout: "get_or_fail_fail": 0, 2024-03-18T16:21:27.017 INFO:teuthology.orchestra.run.smithi196.stdout: "get_or_fail_success": 0, 2024-03-18T16:21:27.017 INFO:teuthology.orchestra.run.smithi196.stdout: "take": 1333, 2024-03-18T16:21:27.017 INFO:teuthology.orchestra.run.smithi196.stdout: "take_sum": 1333, 2024-03-18T16:21:27.018 INFO:teuthology.orchestra.run.smithi196.stdout: "put": 1333, 2024-03-18T16:21:27.018 INFO:teuthology.orchestra.run.smithi196.stdout: "put_sum": 1333, 2024-03-18T16:21:27.018 INFO:teuthology.orchestra.run.smithi196.stdout: "wait": { 2024-03-18T16:21:27.018 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:27.018 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:27.018 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:27.018 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-03-18T16:21:27.018 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:27.018 INFO:teuthology.orchestra.run.smithi196.stdout: "throttle-write_buf_throttle": { 2024-03-18T16:21:27.018 INFO:teuthology.orchestra.run.smithi196.stdout: "val": 0, 2024-03-18T16:21:27.018 INFO:teuthology.orchestra.run.smithi196.stdout: "max": 3758096384, 2024-03-18T16:21:27.018 INFO:teuthology.orchestra.run.smithi196.stdout: "get_started": 0, 2024-03-18T16:21:27.018 INFO:teuthology.orchestra.run.smithi196.stdout: "get": 30, 2024-03-18T16:21:27.018 INFO:teuthology.orchestra.run.smithi196.stdout: "get_sum": 3270, 2024-03-18T16:21:27.018 INFO:teuthology.orchestra.run.smithi196.stdout: "get_or_fail_fail": 0, 2024-03-18T16:21:27.019 INFO:teuthology.orchestra.run.smithi196.stdout: "get_or_fail_success": 30, 2024-03-18T16:21:27.019 INFO:teuthology.orchestra.run.smithi196.stdout: "take": 0, 2024-03-18T16:21:27.019 INFO:teuthology.orchestra.run.smithi196.stdout: "take_sum": 0, 2024-03-18T16:21:27.019 INFO:teuthology.orchestra.run.smithi196.stdout: "put": 1, 2024-03-18T16:21:27.019 INFO:teuthology.orchestra.run.smithi196.stdout: "put_sum": 3270, 2024-03-18T16:21:27.019 INFO:teuthology.orchestra.run.smithi196.stdout: "wait": { 2024-03-18T16:21:27.019 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:27.019 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:27.019 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:27.019 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-03-18T16:21:27.019 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:27.019 INFO:teuthology.orchestra.run.smithi196.stdout: "throttle-write_buf_throttle-0x5639dcda6500": { 2024-03-18T16:21:27.019 INFO:teuthology.orchestra.run.smithi196.stdout: "val": 0, 2024-03-18T16:21:27.019 INFO:teuthology.orchestra.run.smithi196.stdout: "max": 3758096384, 2024-03-18T16:21:27.019 INFO:teuthology.orchestra.run.smithi196.stdout: "get_started": 0, 2024-03-18T16:21:27.020 INFO:teuthology.orchestra.run.smithi196.stdout: "get": 931, 2024-03-18T16:21:27.020 INFO:teuthology.orchestra.run.smithi196.stdout: "get_sum": 1149551, 2024-03-18T16:21:27.020 INFO:teuthology.orchestra.run.smithi196.stdout: "get_or_fail_fail": 0, 2024-03-18T16:21:27.020 INFO:teuthology.orchestra.run.smithi196.stdout: "get_or_fail_success": 931, 2024-03-18T16:21:27.020 INFO:teuthology.orchestra.run.smithi196.stdout: "take": 0, 2024-03-18T16:21:27.020 INFO:teuthology.orchestra.run.smithi196.stdout: "take_sum": 0, 2024-03-18T16:21:27.020 INFO:teuthology.orchestra.run.smithi196.stdout: "put": 148, 2024-03-18T16:21:27.020 INFO:teuthology.orchestra.run.smithi196.stdout: "put_sum": 1149551, 2024-03-18T16:21:27.020 INFO:teuthology.orchestra.run.smithi196.stdout: "wait": { 2024-03-18T16:21:27.020 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:27.020 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:27.020 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:27.020 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-03-18T16:21:27.020 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-03-18T16:21:27.020 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-03-18T16:21:27.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:27 smithi196 ceph-mon[36307]: pgmap v763: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-18T16:21:27.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:27 smithi039 ceph-mon[31721]: pgmap v763: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-18T16:21:27.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:27 smithi194 ceph-mon[36200]: pgmap v763: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-18T16:21:27.930 INFO:tasks.check_counter:Daemon mds.c mds.exported=4 2024-03-18T16:21:27.930 INFO:tasks.check_counter:Daemon mds.c mds.imported=4 2024-03-18T16:21:27.930 INFO:tasks.check_counter:Daemon mds.c mds.dir_split=0 2024-03-18T16:21:27.930 DEBUG:tasks.check_counter:Getting stats from f 2024-03-18T16:21:27.930 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph daemon mds.f perf dump 2024-03-18T16:21:28.198 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T16:21:29.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:29 smithi039 ceph-mon[31721]: pgmap v764: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T16:21:29.374 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:29 smithi196 ceph-mon[36307]: pgmap v764: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T16:21:29.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:29 smithi194 ceph-mon[36200]: pgmap v764: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T16:21:30.085 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-03-18T16:21:30.086 INFO:teuthology.orchestra.run.smithi196.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T16:21:30.086 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_recv_messages": 1068, 2024-03-18T16:21:30.086 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_messages": 1068, 2024-03-18T16:21:30.086 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_recv_bytes": 4537412, 2024-03-18T16:21:30.086 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_bytes": 338810, 2024-03-18T16:21:30.086 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_created_connections": 6, 2024-03-18T16:21:30.086 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_active_connections": 2, 2024-03-18T16:21:30.086 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_total_time": 0.082912763, 2024-03-18T16:21:30.086 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_send_time": 0.023595085, 2024-03-18T16:21:30.086 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_recv_time": 0.024293077, 2024-03-18T16:21:30.086 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_fast_dispatch_time": 0.027770858, 2024-03-18T16:21:30.087 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T16:21:30.087 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 1067, 2024-03-18T16:21:30.087 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.030847580, 2024-03-18T16:21:30.087 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000028910 2024-03-18T16:21:30.087 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:30.087 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_handle_ack_lat": { 2024-03-18T16:21:30.087 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:30.087 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:30.087 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:30.087 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:30.087 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-03-18T16:21:30.087 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_encrypted_bytes": 448 2024-03-18T16:21:30.087 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:30.087 INFO:teuthology.orchestra.run.smithi196.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T16:21:30.088 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_recv_messages": 562, 2024-03-18T16:21:30.088 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_messages": 748, 2024-03-18T16:21:30.088 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_recv_bytes": 138752, 2024-03-18T16:21:30.088 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_bytes": 650738, 2024-03-18T16:21:30.088 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_created_connections": 6, 2024-03-18T16:21:30.088 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_active_connections": 4, 2024-03-18T16:21:30.088 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_total_time": 0.053578386, 2024-03-18T16:21:30.088 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_send_time": 0.020028401, 2024-03-18T16:21:30.088 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_recv_time": 0.010862395, 2024-03-18T16:21:30.088 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_fast_dispatch_time": 0.018369033, 2024-03-18T16:21:30.088 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T16:21:30.088 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 745, 2024-03-18T16:21:30.088 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.036724618, 2024-03-18T16:21:30.088 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000049294 2024-03-18T16:21:30.089 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:30.089 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_handle_ack_lat": { 2024-03-18T16:21:30.089 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:30.089 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:30.089 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:30.089 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:30.089 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-18T16:21:30.089 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_encrypted_bytes": 472592 2024-03-18T16:21:30.089 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:30.089 INFO:teuthology.orchestra.run.smithi196.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T16:21:30.089 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_recv_messages": 1247, 2024-03-18T16:21:30.089 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_messages": 1220, 2024-03-18T16:21:30.089 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_recv_bytes": 5008699, 2024-03-18T16:21:30.089 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_bytes": 459525, 2024-03-18T16:21:30.090 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_created_connections": 3, 2024-03-18T16:21:30.090 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_active_connections": 2, 2024-03-18T16:21:30.090 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_total_time": 0.102700307, 2024-03-18T16:21:30.090 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_send_time": 0.034318096, 2024-03-18T16:21:30.090 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_recv_time": 0.035557308, 2024-03-18T16:21:30.090 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_fast_dispatch_time": 0.028860858, 2024-03-18T16:21:30.090 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T16:21:30.090 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 1219, 2024-03-18T16:21:30.090 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.045490597, 2024-03-18T16:21:30.090 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000037317 2024-03-18T16:21:30.090 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:30.090 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_handle_ack_lat": { 2024-03-18T16:21:30.090 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:30.090 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:30.091 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:30.091 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:30.091 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_recv_encrypted_bytes": 663008, 2024-03-18T16:21:30.091 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_encrypted_bytes": 152336 2024-03-18T16:21:30.091 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:30.091 INFO:teuthology.orchestra.run.smithi196.stdout: "cct": { 2024-03-18T16:21:30.091 INFO:teuthology.orchestra.run.smithi196.stdout: "total_workers": 1, 2024-03-18T16:21:30.091 INFO:teuthology.orchestra.run.smithi196.stdout: "unhealthy_workers": 0 2024-03-18T16:21:30.091 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:30.091 INFO:teuthology.orchestra.run.smithi196.stdout: "finisher-MDSRank": { 2024-03-18T16:21:30.091 INFO:teuthology.orchestra.run.smithi196.stdout: "queue_len": 0, 2024-03-18T16:21:30.091 INFO:teuthology.orchestra.run.smithi196.stdout: "complete_latency": { 2024-03-18T16:21:30.091 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 4316, 2024-03-18T16:21:30.091 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.096488031, 2024-03-18T16:21:30.091 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000022355 2024-03-18T16:21:30.092 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-03-18T16:21:30.092 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:30.092 INFO:teuthology.orchestra.run.smithi196.stdout: "finisher-PurgeQueue": { 2024-03-18T16:21:30.092 INFO:teuthology.orchestra.run.smithi196.stdout: "queue_len": 0, 2024-03-18T16:21:30.092 INFO:teuthology.orchestra.run.smithi196.stdout: "complete_latency": { 2024-03-18T16:21:30.092 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:30.092 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:30.092 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:30.092 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-03-18T16:21:30.092 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:30.092 INFO:teuthology.orchestra.run.smithi196.stdout: "mds": { 2024-03-18T16:21:30.092 INFO:teuthology.orchestra.run.smithi196.stdout: "request": 0, 2024-03-18T16:21:30.092 INFO:teuthology.orchestra.run.smithi196.stdout: "reply": 0, 2024-03-18T16:21:30.092 INFO:teuthology.orchestra.run.smithi196.stdout: "reply_latency": { 2024-03-18T16:21:30.093 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:30.093 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:30.093 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:30.093 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:30.093 INFO:teuthology.orchestra.run.smithi196.stdout: "slow_reply": 0, 2024-03-18T16:21:30.093 INFO:teuthology.orchestra.run.smithi196.stdout: "forward": 0, 2024-03-18T16:21:30.093 INFO:teuthology.orchestra.run.smithi196.stdout: "dir_fetch_complete": 0, 2024-03-18T16:21:30.093 INFO:teuthology.orchestra.run.smithi196.stdout: "dir_fetch_keys": 0, 2024-03-18T16:21:30.093 INFO:teuthology.orchestra.run.smithi196.stdout: "dir_commit": 0, 2024-03-18T16:21:30.093 INFO:teuthology.orchestra.run.smithi196.stdout: "dir_split": 0, 2024-03-18T16:21:30.093 INFO:teuthology.orchestra.run.smithi196.stdout: "dir_merge": 0, 2024-03-18T16:21:30.093 INFO:teuthology.orchestra.run.smithi196.stdout: "inodes": 563, 2024-03-18T16:21:30.093 INFO:teuthology.orchestra.run.smithi196.stdout: "inodes_top": 0, 2024-03-18T16:21:30.093 INFO:teuthology.orchestra.run.smithi196.stdout: "inodes_bottom": 563, 2024-03-18T16:21:30.093 INFO:teuthology.orchestra.run.smithi196.stdout: "inodes_pin_tail": 0, 2024-03-18T16:21:30.094 INFO:teuthology.orchestra.run.smithi196.stdout: "inodes_pinned": 563, 2024-03-18T16:21:30.094 INFO:teuthology.orchestra.run.smithi196.stdout: "inodes_expired": 0, 2024-03-18T16:21:30.094 INFO:teuthology.orchestra.run.smithi196.stdout: "inodes_with_caps": 0, 2024-03-18T16:21:30.094 INFO:teuthology.orchestra.run.smithi196.stdout: "caps": 0, 2024-03-18T16:21:30.094 INFO:teuthology.orchestra.run.smithi196.stdout: "subtrees": 3, 2024-03-18T16:21:30.094 INFO:teuthology.orchestra.run.smithi196.stdout: "traverse": 0, 2024-03-18T16:21:30.094 INFO:teuthology.orchestra.run.smithi196.stdout: "traverse_hit": 0, 2024-03-18T16:21:30.094 INFO:teuthology.orchestra.run.smithi196.stdout: "traverse_forward": 0, 2024-03-18T16:21:30.094 INFO:teuthology.orchestra.run.smithi196.stdout: "traverse_discover": 0, 2024-03-18T16:21:30.094 INFO:teuthology.orchestra.run.smithi196.stdout: "traverse_dir_fetch": 0, 2024-03-18T16:21:30.094 INFO:teuthology.orchestra.run.smithi196.stdout: "traverse_remote_ino": 0, 2024-03-18T16:21:30.094 INFO:teuthology.orchestra.run.smithi196.stdout: "traverse_lock": 0, 2024-03-18T16:21:30.094 INFO:teuthology.orchestra.run.smithi196.stdout: "load_cent": 0, 2024-03-18T16:21:30.094 INFO:teuthology.orchestra.run.smithi196.stdout: "q": 0, 2024-03-18T16:21:30.095 INFO:teuthology.orchestra.run.smithi196.stdout: "exported": 0, 2024-03-18T16:21:30.095 INFO:teuthology.orchestra.run.smithi196.stdout: "exported_inodes": 0, 2024-03-18T16:21:30.095 INFO:teuthology.orchestra.run.smithi196.stdout: "imported": 0, 2024-03-18T16:21:30.095 INFO:teuthology.orchestra.run.smithi196.stdout: "imported_inodes": 0, 2024-03-18T16:21:30.095 INFO:teuthology.orchestra.run.smithi196.stdout: "openino_dir_fetch": 0, 2024-03-18T16:21:30.095 INFO:teuthology.orchestra.run.smithi196.stdout: "openino_backtrace_fetch": 0, 2024-03-18T16:21:30.095 INFO:teuthology.orchestra.run.smithi196.stdout: "openino_peer_discover": 0, 2024-03-18T16:21:30.095 INFO:teuthology.orchestra.run.smithi196.stdout: "root_rfiles": 4, 2024-03-18T16:21:30.095 INFO:teuthology.orchestra.run.smithi196.stdout: "root_rbytes": 234, 2024-03-18T16:21:30.095 INFO:teuthology.orchestra.run.smithi196.stdout: "root_rsnaps": 0, 2024-03-18T16:21:30.095 INFO:teuthology.orchestra.run.smithi196.stdout: "scrub_backtrace_fetch": 0, 2024-03-18T16:21:30.095 INFO:teuthology.orchestra.run.smithi196.stdout: "scrub_set_tag": 0, 2024-03-18T16:21:30.095 INFO:teuthology.orchestra.run.smithi196.stdout: "scrub_backtrace_repaired": 0, 2024-03-18T16:21:30.095 INFO:teuthology.orchestra.run.smithi196.stdout: "scrub_inotable_repaired": 0, 2024-03-18T16:21:30.096 INFO:teuthology.orchestra.run.smithi196.stdout: "scrub_dir_inodes": 0, 2024-03-18T16:21:30.096 INFO:teuthology.orchestra.run.smithi196.stdout: "scrub_dir_base_inodes": 0, 2024-03-18T16:21:30.096 INFO:teuthology.orchestra.run.smithi196.stdout: "scrub_dirfrag_rstats": 0, 2024-03-18T16:21:30.096 INFO:teuthology.orchestra.run.smithi196.stdout: "scrub_file_inodes": 0, 2024-03-18T16:21:30.096 INFO:teuthology.orchestra.run.smithi196.stdout: "handle_inode_file_caps": 0, 2024-03-18T16:21:30.096 INFO:teuthology.orchestra.run.smithi196.stdout: "ceph_cap_op_revoke": 0, 2024-03-18T16:21:30.096 INFO:teuthology.orchestra.run.smithi196.stdout: "ceph_cap_op_grant": 0, 2024-03-18T16:21:30.096 INFO:teuthology.orchestra.run.smithi196.stdout: "ceph_cap_op_trunc": 0, 2024-03-18T16:21:30.096 INFO:teuthology.orchestra.run.smithi196.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-18T16:21:30.096 INFO:teuthology.orchestra.run.smithi196.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-18T16:21:30.096 INFO:teuthology.orchestra.run.smithi196.stdout: "handle_client_caps": 0, 2024-03-18T16:21:30.096 INFO:teuthology.orchestra.run.smithi196.stdout: "handle_client_caps_dirty": 0, 2024-03-18T16:21:30.096 INFO:teuthology.orchestra.run.smithi196.stdout: "handle_client_cap_release": 0, 2024-03-18T16:21:30.096 INFO:teuthology.orchestra.run.smithi196.stdout: "process_request_cap_release": 0 2024-03-18T16:21:30.097 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:30.097 INFO:teuthology.orchestra.run.smithi196.stdout: "mds_cache": { 2024-03-18T16:21:30.097 INFO:teuthology.orchestra.run.smithi196.stdout: "dir_update": 0, 2024-03-18T16:21:30.097 INFO:teuthology.orchestra.run.smithi196.stdout: "dir_update_receipt": 0, 2024-03-18T16:21:30.097 INFO:teuthology.orchestra.run.smithi196.stdout: "dir_try_discover": 0, 2024-03-18T16:21:30.097 INFO:teuthology.orchestra.run.smithi196.stdout: "dir_send_discover": 0, 2024-03-18T16:21:30.097 INFO:teuthology.orchestra.run.smithi196.stdout: "dir_handle_discover": 0, 2024-03-18T16:21:30.097 INFO:teuthology.orchestra.run.smithi196.stdout: "num_strays": 0, 2024-03-18T16:21:30.097 INFO:teuthology.orchestra.run.smithi196.stdout: "num_strays_delayed": 0, 2024-03-18T16:21:30.097 INFO:teuthology.orchestra.run.smithi196.stdout: "num_strays_enqueuing": 0, 2024-03-18T16:21:30.097 INFO:teuthology.orchestra.run.smithi196.stdout: "strays_created": 0, 2024-03-18T16:21:30.097 INFO:teuthology.orchestra.run.smithi196.stdout: "strays_enqueued": 0, 2024-03-18T16:21:30.097 INFO:teuthology.orchestra.run.smithi196.stdout: "strays_reintegrated": 0, 2024-03-18T16:21:30.098 INFO:teuthology.orchestra.run.smithi196.stdout: "strays_migrated": 0, 2024-03-18T16:21:30.098 INFO:teuthology.orchestra.run.smithi196.stdout: "num_recovering_processing": 0, 2024-03-18T16:21:30.098 INFO:teuthology.orchestra.run.smithi196.stdout: "num_recovering_enqueued": 0, 2024-03-18T16:21:30.098 INFO:teuthology.orchestra.run.smithi196.stdout: "num_recovering_prioritized": 0, 2024-03-18T16:21:30.098 INFO:teuthology.orchestra.run.smithi196.stdout: "recovery_started": 0, 2024-03-18T16:21:30.098 INFO:teuthology.orchestra.run.smithi196.stdout: "recovery_completed": 0, 2024-03-18T16:21:30.098 INFO:teuthology.orchestra.run.smithi196.stdout: "ireq_quiesce_path": 0, 2024-03-18T16:21:30.098 INFO:teuthology.orchestra.run.smithi196.stdout: "ireq_quiesce_inode": 0, 2024-03-18T16:21:30.098 INFO:teuthology.orchestra.run.smithi196.stdout: "ireq_enqueue_scrub": 0, 2024-03-18T16:21:30.098 INFO:teuthology.orchestra.run.smithi196.stdout: "ireq_exportdir": 0, 2024-03-18T16:21:30.098 INFO:teuthology.orchestra.run.smithi196.stdout: "ireq_flush": 0, 2024-03-18T16:21:30.098 INFO:teuthology.orchestra.run.smithi196.stdout: "ireq_fragmentdir": 0, 2024-03-18T16:21:30.098 INFO:teuthology.orchestra.run.smithi196.stdout: "ireq_fragstats": 0, 2024-03-18T16:21:30.098 INFO:teuthology.orchestra.run.smithi196.stdout: "ireq_inodestats": 0 2024-03-18T16:21:30.098 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:30.099 INFO:teuthology.orchestra.run.smithi196.stdout: "mds_log": { 2024-03-18T16:21:30.099 INFO:teuthology.orchestra.run.smithi196.stdout: "evlrg": 0, 2024-03-18T16:21:30.099 INFO:teuthology.orchestra.run.smithi196.stdout: "evadd": 0, 2024-03-18T16:21:30.099 INFO:teuthology.orchestra.run.smithi196.stdout: "evex": 0, 2024-03-18T16:21:30.099 INFO:teuthology.orchestra.run.smithi196.stdout: "evtrm": 0, 2024-03-18T16:21:30.099 INFO:teuthology.orchestra.run.smithi196.stdout: "ev": 4235, 2024-03-18T16:21:30.099 INFO:teuthology.orchestra.run.smithi196.stdout: "evexg": 0, 2024-03-18T16:21:30.099 INFO:teuthology.orchestra.run.smithi196.stdout: "evexd": 0, 2024-03-18T16:21:30.099 INFO:teuthology.orchestra.run.smithi196.stdout: "segadd": 0, 2024-03-18T16:21:30.099 INFO:teuthology.orchestra.run.smithi196.stdout: "segex": 0, 2024-03-18T16:21:30.099 INFO:teuthology.orchestra.run.smithi196.stdout: "segtrm": 0, 2024-03-18T16:21:30.099 INFO:teuthology.orchestra.run.smithi196.stdout: "seg": 6, 2024-03-18T16:21:30.099 INFO:teuthology.orchestra.run.smithi196.stdout: "segmjr": 2, 2024-03-18T16:21:30.099 INFO:teuthology.orchestra.run.smithi196.stdout: "segexg": 0, 2024-03-18T16:21:30.100 INFO:teuthology.orchestra.run.smithi196.stdout: "segexd": 0, 2024-03-18T16:21:30.100 INFO:teuthology.orchestra.run.smithi196.stdout: "expos": 4194304, 2024-03-18T16:21:30.100 INFO:teuthology.orchestra.run.smithi196.stdout: "wrpos": 12487696, 2024-03-18T16:21:30.100 INFO:teuthology.orchestra.run.smithi196.stdout: "rdpos": 12486044, 2024-03-18T16:21:30.100 INFO:teuthology.orchestra.run.smithi196.stdout: "jlat": { 2024-03-18T16:21:30.100 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:30.100 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:30.100 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:30.100 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:30.100 INFO:teuthology.orchestra.run.smithi196.stdout: "replayed": 4235 2024-03-18T16:21:30.100 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:30.100 INFO:teuthology.orchestra.run.smithi196.stdout: "mds_mem": { 2024-03-18T16:21:30.100 INFO:teuthology.orchestra.run.smithi196.stdout: "ino": 8, 2024-03-18T16:21:30.100 INFO:teuthology.orchestra.run.smithi196.stdout: "ino+": 819, 2024-03-18T16:21:30.100 INFO:teuthology.orchestra.run.smithi196.stdout: "ino-": 811, 2024-03-18T16:21:30.101 INFO:teuthology.orchestra.run.smithi196.stdout: "dir": 7, 2024-03-18T16:21:30.101 INFO:teuthology.orchestra.run.smithi196.stdout: "dir+": 114, 2024-03-18T16:21:30.101 INFO:teuthology.orchestra.run.smithi196.stdout: "dir-": 107, 2024-03-18T16:21:30.101 INFO:teuthology.orchestra.run.smithi196.stdout: "dn": 563, 2024-03-18T16:21:30.101 INFO:teuthology.orchestra.run.smithi196.stdout: "dn+": 1090, 2024-03-18T16:21:30.101 INFO:teuthology.orchestra.run.smithi196.stdout: "dn-": 527, 2024-03-18T16:21:30.101 INFO:teuthology.orchestra.run.smithi196.stdout: "cap": 0, 2024-03-18T16:21:30.101 INFO:teuthology.orchestra.run.smithi196.stdout: "cap+": 0, 2024-03-18T16:21:30.101 INFO:teuthology.orchestra.run.smithi196.stdout: "cap-": 0, 2024-03-18T16:21:30.101 INFO:teuthology.orchestra.run.smithi196.stdout: "rss": 53252, 2024-03-18T16:21:30.101 INFO:teuthology.orchestra.run.smithi196.stdout: "heap": 215420 2024-03-18T16:21:30.101 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:30.101 INFO:teuthology.orchestra.run.smithi196.stdout: "mds_server": { 2024-03-18T16:21:30.101 INFO:teuthology.orchestra.run.smithi196.stdout: "dispatch_client_request": 0, 2024-03-18T16:21:30.102 INFO:teuthology.orchestra.run.smithi196.stdout: "dispatch_server_request": 0, 2024-03-18T16:21:30.102 INFO:teuthology.orchestra.run.smithi196.stdout: "handle_client_request": 0, 2024-03-18T16:21:30.102 INFO:teuthology.orchestra.run.smithi196.stdout: "handle_client_session": 0, 2024-03-18T16:21:30.102 INFO:teuthology.orchestra.run.smithi196.stdout: "handle_peer_request": 0, 2024-03-18T16:21:30.102 INFO:teuthology.orchestra.run.smithi196.stdout: "req_create_latency": { 2024-03-18T16:21:30.102 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:30.102 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:30.102 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:30.102 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:30.103 INFO:teuthology.orchestra.run.smithi196.stdout: "req_getattr_latency": { 2024-03-18T16:21:30.103 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:30.103 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:30.103 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:30.103 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:30.103 INFO:teuthology.orchestra.run.smithi196.stdout: "req_getfilelock_latency": { 2024-03-18T16:21:30.103 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:30.103 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:30.104 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:30.104 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:30.104 INFO:teuthology.orchestra.run.smithi196.stdout: "req_link_latency": { 2024-03-18T16:21:30.104 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:30.104 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:30.104 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:30.104 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:30.104 INFO:teuthology.orchestra.run.smithi196.stdout: "req_lookup_latency": { 2024-03-18T16:21:30.104 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:30.104 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:30.104 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:30.104 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:30.104 INFO:teuthology.orchestra.run.smithi196.stdout: "req_lookuphash_latency": { 2024-03-18T16:21:30.104 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:30.105 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:30.105 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:30.105 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:30.105 INFO:teuthology.orchestra.run.smithi196.stdout: "req_lookupino_latency": { 2024-03-18T16:21:30.105 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:30.105 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:30.105 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:30.105 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:30.105 INFO:teuthology.orchestra.run.smithi196.stdout: "req_lookupname_latency": { 2024-03-18T16:21:30.105 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:30.105 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:30.105 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:30.105 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:30.105 INFO:teuthology.orchestra.run.smithi196.stdout: "req_lookupparent_latency": { 2024-03-18T16:21:30.105 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:30.106 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:30.106 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:30.106 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:30.106 INFO:teuthology.orchestra.run.smithi196.stdout: "req_lookupsnap_latency": { 2024-03-18T16:21:30.108 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:30.108 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:30.108 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:30.108 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:30.108 INFO:teuthology.orchestra.run.smithi196.stdout: "req_lssnap_latency": { 2024-03-18T16:21:30.108 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:30.108 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:30.108 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:30.108 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:30.108 INFO:teuthology.orchestra.run.smithi196.stdout: "req_mkdir_latency": { 2024-03-18T16:21:30.109 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:30.109 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:30.109 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:30.109 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:30.110 INFO:teuthology.orchestra.run.smithi196.stdout: "req_mknod_latency": { 2024-03-18T16:21:30.110 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:30.110 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:30.110 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:30.110 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:30.110 INFO:teuthology.orchestra.run.smithi196.stdout: "req_mksnap_latency": { 2024-03-18T16:21:30.110 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:30.110 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:30.110 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:30.110 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:30.110 INFO:teuthology.orchestra.run.smithi196.stdout: "req_open_latency": { 2024-03-18T16:21:30.110 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:30.110 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:30.110 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:30.111 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:30.111 INFO:teuthology.orchestra.run.smithi196.stdout: "req_readdir_latency": { 2024-03-18T16:21:30.111 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:30.111 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:30.111 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:30.111 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:30.111 INFO:teuthology.orchestra.run.smithi196.stdout: "req_rename_latency": { 2024-03-18T16:21:30.111 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:30.111 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:30.111 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:30.111 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:30.111 INFO:teuthology.orchestra.run.smithi196.stdout: "req_renamesnap_latency": { 2024-03-18T16:21:30.111 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:30.111 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:30.112 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:30.112 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:30.112 INFO:teuthology.orchestra.run.smithi196.stdout: "req_snapdiff_latency": { 2024-03-18T16:21:30.112 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:30.112 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:30.112 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:30.112 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:30.112 INFO:teuthology.orchestra.run.smithi196.stdout: "req_rmdir_latency": { 2024-03-18T16:21:30.112 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:30.112 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:30.112 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:30.112 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:30.112 INFO:teuthology.orchestra.run.smithi196.stdout: "req_rmsnap_latency": { 2024-03-18T16:21:30.112 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:30.112 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:30.113 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:30.113 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:30.113 INFO:teuthology.orchestra.run.smithi196.stdout: "req_rmxattr_latency": { 2024-03-18T16:21:30.113 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:30.113 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:30.113 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:30.113 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:30.113 INFO:teuthology.orchestra.run.smithi196.stdout: "req_setattr_latency": { 2024-03-18T16:21:30.113 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:30.113 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:30.113 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:30.113 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:30.113 INFO:teuthology.orchestra.run.smithi196.stdout: "req_setdirlayout_latency": { 2024-03-18T16:21:30.113 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:30.114 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:30.114 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:30.114 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:30.114 INFO:teuthology.orchestra.run.smithi196.stdout: "req_setfilelock_latency": { 2024-03-18T16:21:30.114 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:30.114 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:30.114 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:30.114 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:30.114 INFO:teuthology.orchestra.run.smithi196.stdout: "req_setlayout_latency": { 2024-03-18T16:21:30.114 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:30.114 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:30.114 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:30.114 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:30.114 INFO:teuthology.orchestra.run.smithi196.stdout: "req_setxattr_latency": { 2024-03-18T16:21:30.115 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:30.115 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:30.115 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:30.115 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:30.115 INFO:teuthology.orchestra.run.smithi196.stdout: "req_symlink_latency": { 2024-03-18T16:21:30.115 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:30.115 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:30.115 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:30.115 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:30.115 INFO:teuthology.orchestra.run.smithi196.stdout: "req_unlink_latency": { 2024-03-18T16:21:30.115 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:30.115 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:30.115 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:30.115 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:30.115 INFO:teuthology.orchestra.run.smithi196.stdout: "cap_revoke_eviction": 0, 2024-03-18T16:21:30.116 INFO:teuthology.orchestra.run.smithi196.stdout: "cap_acquisition_throttle": 0, 2024-03-18T16:21:30.116 INFO:teuthology.orchestra.run.smithi196.stdout: "req_getvxattr_latency": { 2024-03-18T16:21:30.116 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:30.116 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:30.116 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:30.116 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-03-18T16:21:30.116 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:30.116 INFO:teuthology.orchestra.run.smithi196.stdout: "mds_sessions": { 2024-03-18T16:21:30.116 INFO:teuthology.orchestra.run.smithi196.stdout: "session_count": 1, 2024-03-18T16:21:30.116 INFO:teuthology.orchestra.run.smithi196.stdout: "session_add": 3, 2024-03-18T16:21:30.116 INFO:teuthology.orchestra.run.smithi196.stdout: "session_remove": 2, 2024-03-18T16:21:30.116 INFO:teuthology.orchestra.run.smithi196.stdout: "sessions_open": 1, 2024-03-18T16:21:30.116 INFO:teuthology.orchestra.run.smithi196.stdout: "sessions_stale": 0, 2024-03-18T16:21:30.116 INFO:teuthology.orchestra.run.smithi196.stdout: "total_load": 0, 2024-03-18T16:21:30.117 INFO:teuthology.orchestra.run.smithi196.stdout: "average_load": 0, 2024-03-18T16:21:30.117 INFO:teuthology.orchestra.run.smithi196.stdout: "avg_session_uptime": 2480, 2024-03-18T16:21:30.117 INFO:teuthology.orchestra.run.smithi196.stdout: "mdthresh_evicted": 0 2024-03-18T16:21:30.117 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:30.117 INFO:teuthology.orchestra.run.smithi196.stdout: "mempool": { 2024-03-18T16:21:30.117 INFO:teuthology.orchestra.run.smithi196.stdout: "bloom_filter_bytes": 0, 2024-03-18T16:21:30.117 INFO:teuthology.orchestra.run.smithi196.stdout: "bloom_filter_items": 0, 2024-03-18T16:21:30.117 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T16:21:30.117 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_alloc_items": 0, 2024-03-18T16:21:30.117 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T16:21:30.117 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_cache_data_items": 0, 2024-03-18T16:21:30.117 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T16:21:30.117 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T16:21:30.117 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T16:21:30.118 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T16:21:30.118 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T16:21:30.118 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_cache_other_items": 0, 2024-03-18T16:21:30.118 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T16:21:30.118 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T16:21:30.118 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_extent_bytes": 0, 2024-03-18T16:21:30.118 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_extent_items": 0, 2024-03-18T16:21:30.118 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_blob_bytes": 0, 2024-03-18T16:21:30.118 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_blob_items": 0, 2024-03-18T16:21:30.118 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T16:21:30.118 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T16:21:30.118 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T16:21:30.118 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T16:21:30.118 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T16:21:30.118 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_fsck_items": 0, 2024-03-18T16:21:30.119 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_txc_bytes": 0, 2024-03-18T16:21:30.119 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_txc_items": 0, 2024-03-18T16:21:30.119 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T16:21:30.119 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T16:21:30.119 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_writing_bytes": 0, 2024-03-18T16:21:30.119 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_writing_items": 0, 2024-03-18T16:21:30.119 INFO:teuthology.orchestra.run.smithi196.stdout: "bluefs_bytes": 0, 2024-03-18T16:21:30.119 INFO:teuthology.orchestra.run.smithi196.stdout: "bluefs_items": 0, 2024-03-18T16:21:30.119 INFO:teuthology.orchestra.run.smithi196.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T16:21:30.119 INFO:teuthology.orchestra.run.smithi196.stdout: "bluefs_file_reader_items": 0, 2024-03-18T16:21:30.119 INFO:teuthology.orchestra.run.smithi196.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T16:21:30.119 INFO:teuthology.orchestra.run.smithi196.stdout: "bluefs_file_writer_items": 0, 2024-03-18T16:21:30.119 INFO:teuthology.orchestra.run.smithi196.stdout: "buffer_anon_bytes": 189383, 2024-03-18T16:21:30.119 INFO:teuthology.orchestra.run.smithi196.stdout: "buffer_anon_items": 50, 2024-03-18T16:21:30.119 INFO:teuthology.orchestra.run.smithi196.stdout: "buffer_meta_bytes": 0, 2024-03-18T16:21:30.120 INFO:teuthology.orchestra.run.smithi196.stdout: "buffer_meta_items": 0, 2024-03-18T16:21:30.120 INFO:teuthology.orchestra.run.smithi196.stdout: "osd_bytes": 0, 2024-03-18T16:21:30.120 INFO:teuthology.orchestra.run.smithi196.stdout: "osd_items": 0, 2024-03-18T16:21:30.120 INFO:teuthology.orchestra.run.smithi196.stdout: "osd_mapbl_bytes": 0, 2024-03-18T16:21:30.120 INFO:teuthology.orchestra.run.smithi196.stdout: "osd_mapbl_items": 0, 2024-03-18T16:21:30.120 INFO:teuthology.orchestra.run.smithi196.stdout: "osd_pglog_bytes": 0, 2024-03-18T16:21:30.120 INFO:teuthology.orchestra.run.smithi196.stdout: "osd_pglog_items": 0, 2024-03-18T16:21:30.120 INFO:teuthology.orchestra.run.smithi196.stdout: "osdmap_bytes": 8888, 2024-03-18T16:21:30.120 INFO:teuthology.orchestra.run.smithi196.stdout: "osdmap_items": 170, 2024-03-18T16:21:30.120 INFO:teuthology.orchestra.run.smithi196.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T16:21:30.120 INFO:teuthology.orchestra.run.smithi196.stdout: "osdmap_mapping_items": 0, 2024-03-18T16:21:30.120 INFO:teuthology.orchestra.run.smithi196.stdout: "pgmap_bytes": 0, 2024-03-18T16:21:30.120 INFO:teuthology.orchestra.run.smithi196.stdout: "pgmap_items": 0, 2024-03-18T16:21:30.120 INFO:teuthology.orchestra.run.smithi196.stdout: "mds_co_bytes": 423744, 2024-03-18T16:21:30.121 INFO:teuthology.orchestra.run.smithi196.stdout: "mds_co_items": 2334, 2024-03-18T16:21:30.121 INFO:teuthology.orchestra.run.smithi196.stdout: "unittest_1_bytes": 0, 2024-03-18T16:21:30.121 INFO:teuthology.orchestra.run.smithi196.stdout: "unittest_1_items": 0, 2024-03-18T16:21:30.121 INFO:teuthology.orchestra.run.smithi196.stdout: "unittest_2_bytes": 0, 2024-03-18T16:21:30.121 INFO:teuthology.orchestra.run.smithi196.stdout: "unittest_2_items": 0 2024-03-18T16:21:30.121 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:30.121 INFO:teuthology.orchestra.run.smithi196.stdout: "objecter": { 2024-03-18T16:21:30.121 INFO:teuthology.orchestra.run.smithi196.stdout: "op_active": 0, 2024-03-18T16:21:30.121 INFO:teuthology.orchestra.run.smithi196.stdout: "op_laggy": 0, 2024-03-18T16:21:30.121 INFO:teuthology.orchestra.run.smithi196.stdout: "op_send": 2590, 2024-03-18T16:21:30.121 INFO:teuthology.orchestra.run.smithi196.stdout: "op_send_bytes": 26, 2024-03-18T16:21:30.121 INFO:teuthology.orchestra.run.smithi196.stdout: "op_resend": 4, 2024-03-18T16:21:30.121 INFO:teuthology.orchestra.run.smithi196.stdout: "op_reply": 2590, 2024-03-18T16:21:30.121 INFO:teuthology.orchestra.run.smithi196.stdout: "op_latency": { 2024-03-18T16:21:30.121 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 2590, 2024-03-18T16:21:30.122 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 55.668862102, 2024-03-18T16:21:30.122 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.021493769 2024-03-18T16:21:30.122 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:30.122 INFO:teuthology.orchestra.run.smithi196.stdout: "op_inflight": 0, 2024-03-18T16:21:30.122 INFO:teuthology.orchestra.run.smithi196.stdout: "oplen_avg": { 2024-03-18T16:21:30.122 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 2590, 2024-03-18T16:21:30.122 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 2591 2024-03-18T16:21:30.122 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:30.122 INFO:teuthology.orchestra.run.smithi196.stdout: "op": 2590, 2024-03-18T16:21:30.122 INFO:teuthology.orchestra.run.smithi196.stdout: "op_r": 2590, 2024-03-18T16:21:30.122 INFO:teuthology.orchestra.run.smithi196.stdout: "op_w": 0, 2024-03-18T16:21:30.122 INFO:teuthology.orchestra.run.smithi196.stdout: "op_rmw": 0, 2024-03-18T16:21:30.122 INFO:teuthology.orchestra.run.smithi196.stdout: "op_pg": 0, 2024-03-18T16:21:30.122 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_stat": 1590, 2024-03-18T16:21:30.123 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_create": 0, 2024-03-18T16:21:30.123 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_read": 997, 2024-03-18T16:21:30.123 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_write": 0, 2024-03-18T16:21:30.123 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_writefull": 0, 2024-03-18T16:21:30.123 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_writesame": 0, 2024-03-18T16:21:30.123 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_append": 0, 2024-03-18T16:21:30.123 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_zero": 0, 2024-03-18T16:21:30.123 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_truncate": 0, 2024-03-18T16:21:30.123 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_delete": 0, 2024-03-18T16:21:30.123 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_mapext": 0, 2024-03-18T16:21:30.123 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_sparse_read": 0, 2024-03-18T16:21:30.123 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_clonerange": 0, 2024-03-18T16:21:30.123 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_getxattr": 2, 2024-03-18T16:21:30.123 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_setxattr": 0, 2024-03-18T16:21:30.123 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_cmpxattr": 0, 2024-03-18T16:21:30.124 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_rmxattr": 0, 2024-03-18T16:21:30.124 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_resetxattrs": 0, 2024-03-18T16:21:30.124 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_call": 0, 2024-03-18T16:21:30.124 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_watch": 0, 2024-03-18T16:21:30.124 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_notify": 0, 2024-03-18T16:21:30.124 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_src_cmpxattr": 0, 2024-03-18T16:21:30.124 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_pgls": 0, 2024-03-18T16:21:30.124 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_pgls_filter": 0, 2024-03-18T16:21:30.124 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_other": 0, 2024-03-18T16:21:30.124 INFO:teuthology.orchestra.run.smithi196.stdout: "linger_active": 0, 2024-03-18T16:21:30.124 INFO:teuthology.orchestra.run.smithi196.stdout: "linger_send": 0, 2024-03-18T16:21:30.124 INFO:teuthology.orchestra.run.smithi196.stdout: "linger_resend": 0, 2024-03-18T16:21:30.124 INFO:teuthology.orchestra.run.smithi196.stdout: "linger_ping": 0, 2024-03-18T16:21:30.124 INFO:teuthology.orchestra.run.smithi196.stdout: "poolop_active": 0, 2024-03-18T16:21:30.125 INFO:teuthology.orchestra.run.smithi196.stdout: "poolop_send": 0, 2024-03-18T16:21:30.125 INFO:teuthology.orchestra.run.smithi196.stdout: "poolop_resend": 0, 2024-03-18T16:21:30.125 INFO:teuthology.orchestra.run.smithi196.stdout: "poolstat_active": 0, 2024-03-18T16:21:30.125 INFO:teuthology.orchestra.run.smithi196.stdout: "poolstat_send": 0, 2024-03-18T16:21:30.125 INFO:teuthology.orchestra.run.smithi196.stdout: "poolstat_resend": 0, 2024-03-18T16:21:30.125 INFO:teuthology.orchestra.run.smithi196.stdout: "statfs_active": 0, 2024-03-18T16:21:30.125 INFO:teuthology.orchestra.run.smithi196.stdout: "statfs_send": 0, 2024-03-18T16:21:30.125 INFO:teuthology.orchestra.run.smithi196.stdout: "statfs_resend": 0, 2024-03-18T16:21:30.125 INFO:teuthology.orchestra.run.smithi196.stdout: "command_active": 0, 2024-03-18T16:21:30.125 INFO:teuthology.orchestra.run.smithi196.stdout: "command_send": 0, 2024-03-18T16:21:30.125 INFO:teuthology.orchestra.run.smithi196.stdout: "command_resend": 0, 2024-03-18T16:21:30.125 INFO:teuthology.orchestra.run.smithi196.stdout: "map_epoch": 93, 2024-03-18T16:21:30.125 INFO:teuthology.orchestra.run.smithi196.stdout: "map_full": 0, 2024-03-18T16:21:30.125 INFO:teuthology.orchestra.run.smithi196.stdout: "map_inc": 4, 2024-03-18T16:21:30.125 INFO:teuthology.orchestra.run.smithi196.stdout: "osd_sessions": 5, 2024-03-18T16:21:30.125 INFO:teuthology.orchestra.run.smithi196.stdout: "osd_session_open": 5, 2024-03-18T16:21:30.126 INFO:teuthology.orchestra.run.smithi196.stdout: "osd_session_close": 0, 2024-03-18T16:21:30.126 INFO:teuthology.orchestra.run.smithi196.stdout: "osd_laggy": 0, 2024-03-18T16:21:30.126 INFO:teuthology.orchestra.run.smithi196.stdout: "omap_wr": 0, 2024-03-18T16:21:30.126 INFO:teuthology.orchestra.run.smithi196.stdout: "omap_rd": 2, 2024-03-18T16:21:30.126 INFO:teuthology.orchestra.run.smithi196.stdout: "omap_del": 0 2024-03-18T16:21:30.126 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:30.126 INFO:teuthology.orchestra.run.smithi196.stdout: "oft": { 2024-03-18T16:21:30.126 INFO:teuthology.orchestra.run.smithi196.stdout: "omap_total_objs": 0, 2024-03-18T16:21:30.126 INFO:teuthology.orchestra.run.smithi196.stdout: "omap_total_kv_pairs": 0, 2024-03-18T16:21:30.126 INFO:teuthology.orchestra.run.smithi196.stdout: "omap_total_updates": 0, 2024-03-18T16:21:30.126 INFO:teuthology.orchestra.run.smithi196.stdout: "omap_total_removes": 0 2024-03-18T16:21:30.126 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:30.126 INFO:teuthology.orchestra.run.smithi196.stdout: "osd-slow-ops": { 2024-03-18T16:21:30.126 INFO:teuthology.orchestra.run.smithi196.stdout: "slow_ops_count": 0 2024-03-18T16:21:30.127 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:30.127 INFO:teuthology.orchestra.run.smithi196.stdout: "purge_queue": { 2024-03-18T16:21:30.127 INFO:teuthology.orchestra.run.smithi196.stdout: "pq_executing_ops": 0, 2024-03-18T16:21:30.127 INFO:teuthology.orchestra.run.smithi196.stdout: "pq_executing_ops_high_water": 0, 2024-03-18T16:21:30.127 INFO:teuthology.orchestra.run.smithi196.stdout: "pq_executing": 0, 2024-03-18T16:21:30.127 INFO:teuthology.orchestra.run.smithi196.stdout: "pq_executing_high_water": 0, 2024-03-18T16:21:30.127 INFO:teuthology.orchestra.run.smithi196.stdout: "pq_executed_ops": 0, 2024-03-18T16:21:30.127 INFO:teuthology.orchestra.run.smithi196.stdout: "pq_executed": 0, 2024-03-18T16:21:30.127 INFO:teuthology.orchestra.run.smithi196.stdout: "pq_item_in_journal": 0 2024-03-18T16:21:30.127 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:30.127 INFO:teuthology.orchestra.run.smithi196.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T16:21:30.127 INFO:teuthology.orchestra.run.smithi196.stdout: "val": 0, 2024-03-18T16:21:30.127 INFO:teuthology.orchestra.run.smithi196.stdout: "max": 104857600, 2024-03-18T16:21:30.127 INFO:teuthology.orchestra.run.smithi196.stdout: "get_started": 0, 2024-03-18T16:21:30.127 INFO:teuthology.orchestra.run.smithi196.stdout: "get": 2873, 2024-03-18T16:21:30.128 INFO:teuthology.orchestra.run.smithi196.stdout: "get_sum": 9408917, 2024-03-18T16:21:30.128 INFO:teuthology.orchestra.run.smithi196.stdout: "get_or_fail_fail": 0, 2024-03-18T16:21:30.128 INFO:teuthology.orchestra.run.smithi196.stdout: "get_or_fail_success": 2873, 2024-03-18T16:21:30.128 INFO:teuthology.orchestra.run.smithi196.stdout: "take": 0, 2024-03-18T16:21:30.128 INFO:teuthology.orchestra.run.smithi196.stdout: "take_sum": 0, 2024-03-18T16:21:30.128 INFO:teuthology.orchestra.run.smithi196.stdout: "put": 2873, 2024-03-18T16:21:30.128 INFO:teuthology.orchestra.run.smithi196.stdout: "put_sum": 9408917, 2024-03-18T16:21:30.128 INFO:teuthology.orchestra.run.smithi196.stdout: "wait": { 2024-03-18T16:21:30.128 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:30.128 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:30.128 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:30.128 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-03-18T16:21:30.128 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:30.128 INFO:teuthology.orchestra.run.smithi196.stdout: "throttle-objecter_bytes": { 2024-03-18T16:21:30.129 INFO:teuthology.orchestra.run.smithi196.stdout: "val": 0, 2024-03-18T16:21:30.129 INFO:teuthology.orchestra.run.smithi196.stdout: "max": 104857600, 2024-03-18T16:21:30.129 INFO:teuthology.orchestra.run.smithi196.stdout: "get_started": 0, 2024-03-18T16:21:30.129 INFO:teuthology.orchestra.run.smithi196.stdout: "get": 0, 2024-03-18T16:21:30.129 INFO:teuthology.orchestra.run.smithi196.stdout: "get_sum": 0, 2024-03-18T16:21:30.129 INFO:teuthology.orchestra.run.smithi196.stdout: "get_or_fail_fail": 0, 2024-03-18T16:21:30.129 INFO:teuthology.orchestra.run.smithi196.stdout: "get_or_fail_success": 0, 2024-03-18T16:21:30.129 INFO:teuthology.orchestra.run.smithi196.stdout: "take": 2590, 2024-03-18T16:21:30.129 INFO:teuthology.orchestra.run.smithi196.stdout: "take_sum": 8295779, 2024-03-18T16:21:30.129 INFO:teuthology.orchestra.run.smithi196.stdout: "put": 2590, 2024-03-18T16:21:30.129 INFO:teuthology.orchestra.run.smithi196.stdout: "put_sum": 8295779, 2024-03-18T16:21:30.129 INFO:teuthology.orchestra.run.smithi196.stdout: "wait": { 2024-03-18T16:21:30.129 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:30.129 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:30.129 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:30.130 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-03-18T16:21:30.130 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:30.130 INFO:teuthology.orchestra.run.smithi196.stdout: "throttle-objecter_ops": { 2024-03-18T16:21:30.130 INFO:teuthology.orchestra.run.smithi196.stdout: "val": 0, 2024-03-18T16:21:30.130 INFO:teuthology.orchestra.run.smithi196.stdout: "max": 1024, 2024-03-18T16:21:30.130 INFO:teuthology.orchestra.run.smithi196.stdout: "get_started": 0, 2024-03-18T16:21:30.130 INFO:teuthology.orchestra.run.smithi196.stdout: "get": 0, 2024-03-18T16:21:30.130 INFO:teuthology.orchestra.run.smithi196.stdout: "get_sum": 0, 2024-03-18T16:21:30.130 INFO:teuthology.orchestra.run.smithi196.stdout: "get_or_fail_fail": 0, 2024-03-18T16:21:30.130 INFO:teuthology.orchestra.run.smithi196.stdout: "get_or_fail_success": 0, 2024-03-18T16:21:30.130 INFO:teuthology.orchestra.run.smithi196.stdout: "take": 2590, 2024-03-18T16:21:30.130 INFO:teuthology.orchestra.run.smithi196.stdout: "take_sum": 2590, 2024-03-18T16:21:30.130 INFO:teuthology.orchestra.run.smithi196.stdout: "put": 2590, 2024-03-18T16:21:30.130 INFO:teuthology.orchestra.run.smithi196.stdout: "put_sum": 2590, 2024-03-18T16:21:30.131 INFO:teuthology.orchestra.run.smithi196.stdout: "wait": { 2024-03-18T16:21:30.131 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:30.131 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:30.131 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:30.131 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-03-18T16:21:30.131 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:30.131 INFO:teuthology.orchestra.run.smithi196.stdout: "throttle-write_buf_throttle": { 2024-03-18T16:21:30.131 INFO:teuthology.orchestra.run.smithi196.stdout: "val": 0, 2024-03-18T16:21:30.131 INFO:teuthology.orchestra.run.smithi196.stdout: "max": 3758096384, 2024-03-18T16:21:30.131 INFO:teuthology.orchestra.run.smithi196.stdout: "get_started": 0, 2024-03-18T16:21:30.131 INFO:teuthology.orchestra.run.smithi196.stdout: "get": 0, 2024-03-18T16:21:30.131 INFO:teuthology.orchestra.run.smithi196.stdout: "get_sum": 0, 2024-03-18T16:21:30.131 INFO:teuthology.orchestra.run.smithi196.stdout: "get_or_fail_fail": 0, 2024-03-18T16:21:30.131 INFO:teuthology.orchestra.run.smithi196.stdout: "get_or_fail_success": 0, 2024-03-18T16:21:30.131 INFO:teuthology.orchestra.run.smithi196.stdout: "take": 0, 2024-03-18T16:21:30.132 INFO:teuthology.orchestra.run.smithi196.stdout: "take_sum": 0, 2024-03-18T16:21:30.132 INFO:teuthology.orchestra.run.smithi196.stdout: "put": 0, 2024-03-18T16:21:30.132 INFO:teuthology.orchestra.run.smithi196.stdout: "put_sum": 0, 2024-03-18T16:21:30.132 INFO:teuthology.orchestra.run.smithi196.stdout: "wait": { 2024-03-18T16:21:30.132 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:30.132 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:30.132 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:30.132 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-03-18T16:21:30.132 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:30.132 INFO:teuthology.orchestra.run.smithi196.stdout: "throttle-write_buf_throttle-0x55f61791a6e0": { 2024-03-18T16:21:30.132 INFO:teuthology.orchestra.run.smithi196.stdout: "val": 0, 2024-03-18T16:21:30.132 INFO:teuthology.orchestra.run.smithi196.stdout: "max": 3758096384, 2024-03-18T16:21:30.132 INFO:teuthology.orchestra.run.smithi196.stdout: "get_started": 0, 2024-03-18T16:21:30.132 INFO:teuthology.orchestra.run.smithi196.stdout: "get": 0, 2024-03-18T16:21:30.132 INFO:teuthology.orchestra.run.smithi196.stdout: "get_sum": 0, 2024-03-18T16:21:30.133 INFO:teuthology.orchestra.run.smithi196.stdout: "get_or_fail_fail": 0, 2024-03-18T16:21:30.133 INFO:teuthology.orchestra.run.smithi196.stdout: "get_or_fail_success": 0, 2024-03-18T16:21:30.133 INFO:teuthology.orchestra.run.smithi196.stdout: "take": 0, 2024-03-18T16:21:30.133 INFO:teuthology.orchestra.run.smithi196.stdout: "take_sum": 0, 2024-03-18T16:21:30.133 INFO:teuthology.orchestra.run.smithi196.stdout: "put": 0, 2024-03-18T16:21:30.133 INFO:teuthology.orchestra.run.smithi196.stdout: "put_sum": 0, 2024-03-18T16:21:30.133 INFO:teuthology.orchestra.run.smithi196.stdout: "wait": { 2024-03-18T16:21:30.133 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:30.133 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:30.133 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:30.133 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-03-18T16:21:30.133 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-03-18T16:21:30.133 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-03-18T16:21:30.890 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-03-18T16:21:30.890 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-03-18T16:21:30.890 INFO:tasks.check_counter:Daemon mds.f mds.dir_split=0 2024-03-18T16:21:30.890 DEBUG:tasks.check_counter:Getting stats from i 2024-03-18T16:21:30.891 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph daemon mds.i perf dump 2024-03-18T16:21:31.141 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T16:21:31.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:31 smithi196 ceph-mon[36307]: pgmap v765: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-18T16:21:31.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:31 smithi039 ceph-mon[31721]: pgmap v765: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-18T16:21:31.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:31 smithi194 ceph-mon[36200]: pgmap v765: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-18T16:21:32.842 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-03-18T16:21:32.842 INFO:teuthology.orchestra.run.smithi196.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T16:21:32.842 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_recv_messages": 5911, 2024-03-18T16:21:32.842 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_messages": 7183, 2024-03-18T16:21:32.842 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_recv_bytes": 2139229, 2024-03-18T16:21:32.842 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_bytes": 9496130, 2024-03-18T16:21:32.842 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_created_connections": 22, 2024-03-18T16:21:32.843 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_active_connections": 2, 2024-03-18T16:21:32.843 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_total_time": 0.500561952, 2024-03-18T16:21:32.843 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_send_time": 0.199568813, 2024-03-18T16:21:32.843 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_recv_time": 0.340470918, 2024-03-18T16:21:32.843 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_fast_dispatch_time": 0.053998427, 2024-03-18T16:21:32.843 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T16:21:32.844 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 7182, 2024-03-18T16:21:32.844 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.633087985, 2024-03-18T16:21:32.844 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000088149 2024-03-18T16:21:32.844 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:32.844 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_handle_ack_lat": { 2024-03-18T16:21:32.844 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 7948, 2024-03-18T16:21:32.844 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.001928552, 2024-03-18T16:21:32.844 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000242 2024-03-18T16:21:32.844 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:32.845 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_recv_encrypted_bytes": 1323488, 2024-03-18T16:21:32.845 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_encrypted_bytes": 2127904 2024-03-18T16:21:32.845 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:32.845 INFO:teuthology.orchestra.run.smithi196.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T16:21:32.845 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_recv_messages": 1982, 2024-03-18T16:21:32.845 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_messages": 1557, 2024-03-18T16:21:32.845 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_recv_bytes": 1337021, 2024-03-18T16:21:32.845 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_bytes": 980085, 2024-03-18T16:21:32.845 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_created_connections": 11, 2024-03-18T16:21:32.845 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_active_connections": 6, 2024-03-18T16:21:32.845 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_total_time": 0.125279973, 2024-03-18T16:21:32.845 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_send_time": 0.042398931, 2024-03-18T16:21:32.845 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_recv_time": 0.081133427, 2024-03-18T16:21:32.846 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_fast_dispatch_time": 0.019232111, 2024-03-18T16:21:32.846 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T16:21:32.846 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 1556, 2024-03-18T16:21:32.846 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.125840061, 2024-03-18T16:21:32.846 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000080874 2024-03-18T16:21:32.847 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:32.847 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_handle_ack_lat": { 2024-03-18T16:21:32.847 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 1865, 2024-03-18T16:21:32.847 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000291562, 2024-03-18T16:21:32.847 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000156 2024-03-18T16:21:32.847 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:32.847 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_recv_encrypted_bytes": 685600, 2024-03-18T16:21:32.847 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_encrypted_bytes": 199408 2024-03-18T16:21:32.848 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:32.848 INFO:teuthology.orchestra.run.smithi196.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T16:21:32.848 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_recv_messages": 4464, 2024-03-18T16:21:32.849 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_messages": 3937, 2024-03-18T16:21:32.849 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_recv_bytes": 1729864, 2024-03-18T16:21:32.849 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_bytes": 2314229, 2024-03-18T16:21:32.849 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_created_connections": 37, 2024-03-18T16:21:32.849 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_active_connections": 6, 2024-03-18T16:21:32.849 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_total_time": 0.275765021, 2024-03-18T16:21:32.849 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_send_time": 0.101496857, 2024-03-18T16:21:32.849 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_recv_time": 0.192274557, 2024-03-18T16:21:32.850 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_fast_dispatch_time": 0.025386703, 2024-03-18T16:21:32.850 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T16:21:32.850 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 3934, 2024-03-18T16:21:32.850 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.249655361, 2024-03-18T16:21:32.850 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000063460 2024-03-18T16:21:32.850 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:32.851 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_handle_ack_lat": { 2024-03-18T16:21:32.851 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 6290, 2024-03-18T16:21:32.851 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.001135928, 2024-03-18T16:21:32.852 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000180 2024-03-18T16:21:32.852 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:32.852 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-18T16:21:32.852 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_encrypted_bytes": 477248 2024-03-18T16:21:32.852 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:32.852 INFO:teuthology.orchestra.run.smithi196.stdout: "cct": { 2024-03-18T16:21:32.852 INFO:teuthology.orchestra.run.smithi196.stdout: "total_workers": 1, 2024-03-18T16:21:32.853 INFO:teuthology.orchestra.run.smithi196.stdout: "unhealthy_workers": 0 2024-03-18T16:21:32.853 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:32.853 INFO:teuthology.orchestra.run.smithi196.stdout: "finisher-MDSRank": { 2024-03-18T16:21:32.853 INFO:teuthology.orchestra.run.smithi196.stdout: "queue_len": 0, 2024-03-18T16:21:32.853 INFO:teuthology.orchestra.run.smithi196.stdout: "complete_latency": { 2024-03-18T16:21:32.853 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 3128, 2024-03-18T16:21:32.853 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 1.253501329, 2024-03-18T16:21:32.854 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000400735 2024-03-18T16:21:32.854 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-03-18T16:21:32.854 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:32.855 INFO:teuthology.orchestra.run.smithi196.stdout: "finisher-PurgeQueue": { 2024-03-18T16:21:32.855 INFO:teuthology.orchestra.run.smithi196.stdout: "queue_len": 0, 2024-03-18T16:21:32.855 INFO:teuthology.orchestra.run.smithi196.stdout: "complete_latency": { 2024-03-18T16:21:32.855 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 727, 2024-03-18T16:21:32.856 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.036145274, 2024-03-18T16:21:32.856 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000049718 2024-03-18T16:21:32.856 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-03-18T16:21:32.856 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:32.856 INFO:teuthology.orchestra.run.smithi196.stdout: "mds": { 2024-03-18T16:21:32.856 INFO:teuthology.orchestra.run.smithi196.stdout: "request": 350, 2024-03-18T16:21:32.857 INFO:teuthology.orchestra.run.smithi196.stdout: "reply": 343, 2024-03-18T16:21:32.857 INFO:teuthology.orchestra.run.smithi196.stdout: "reply_latency": { 2024-03-18T16:21:32.857 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 343, 2024-03-18T16:21:32.857 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 1.235991187, 2024-03-18T16:21:32.857 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.003603472 2024-03-18T16:21:32.857 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:32.858 INFO:teuthology.orchestra.run.smithi196.stdout: "slow_reply": 0, 2024-03-18T16:21:32.858 INFO:teuthology.orchestra.run.smithi196.stdout: "forward": 7, 2024-03-18T16:21:32.858 INFO:teuthology.orchestra.run.smithi196.stdout: "dir_fetch_complete": 0, 2024-03-18T16:21:32.858 INFO:teuthology.orchestra.run.smithi196.stdout: "dir_fetch_keys": 0, 2024-03-18T16:21:32.858 INFO:teuthology.orchestra.run.smithi196.stdout: "dir_commit": 20, 2024-03-18T16:21:32.858 INFO:teuthology.orchestra.run.smithi196.stdout: "dir_split": 1, 2024-03-18T16:21:32.859 INFO:teuthology.orchestra.run.smithi196.stdout: "dir_merge": 0, 2024-03-18T16:21:32.859 INFO:teuthology.orchestra.run.smithi196.stdout: "inodes": 18, 2024-03-18T16:21:32.859 INFO:teuthology.orchestra.run.smithi196.stdout: "inodes_top": 2, 2024-03-18T16:21:32.859 INFO:teuthology.orchestra.run.smithi196.stdout: "inodes_bottom": 6, 2024-03-18T16:21:32.859 INFO:teuthology.orchestra.run.smithi196.stdout: "inodes_pin_tail": 10, 2024-03-18T16:21:32.859 INFO:teuthology.orchestra.run.smithi196.stdout: "inodes_pinned": 13, 2024-03-18T16:21:32.859 INFO:teuthology.orchestra.run.smithi196.stdout: "inodes_expired": 496, 2024-03-18T16:21:32.859 INFO:teuthology.orchestra.run.smithi196.stdout: "inodes_with_caps": 0, 2024-03-18T16:21:32.860 INFO:teuthology.orchestra.run.smithi196.stdout: "caps": 0, 2024-03-18T16:21:32.860 INFO:teuthology.orchestra.run.smithi196.stdout: "subtrees": 6, 2024-03-18T16:21:32.860 INFO:teuthology.orchestra.run.smithi196.stdout: "traverse": 594, 2024-03-18T16:21:32.860 INFO:teuthology.orchestra.run.smithi196.stdout: "traverse_hit": 500, 2024-03-18T16:21:32.860 INFO:teuthology.orchestra.run.smithi196.stdout: "traverse_forward": 6, 2024-03-18T16:21:32.860 INFO:teuthology.orchestra.run.smithi196.stdout: "traverse_discover": 23, 2024-03-18T16:21:32.860 INFO:teuthology.orchestra.run.smithi196.stdout: "traverse_dir_fetch": 0, 2024-03-18T16:21:32.860 INFO:teuthology.orchestra.run.smithi196.stdout: "traverse_remote_ino": 0, 2024-03-18T16:21:32.860 INFO:teuthology.orchestra.run.smithi196.stdout: "traverse_lock": 0, 2024-03-18T16:21:32.860 INFO:teuthology.orchestra.run.smithi196.stdout: "load_cent": 0, 2024-03-18T16:21:32.861 INFO:teuthology.orchestra.run.smithi196.stdout: "q": 0, 2024-03-18T16:21:32.861 INFO:teuthology.orchestra.run.smithi196.stdout: "exported": 25, 2024-03-18T16:21:32.861 INFO:teuthology.orchestra.run.smithi196.stdout: "exported_inodes": 369, 2024-03-18T16:21:32.861 INFO:teuthology.orchestra.run.smithi196.stdout: "imported": 25, 2024-03-18T16:21:32.861 INFO:teuthology.orchestra.run.smithi196.stdout: "imported_inodes": 413, 2024-03-18T16:21:32.861 INFO:teuthology.orchestra.run.smithi196.stdout: "openino_dir_fetch": 0, 2024-03-18T16:21:32.861 INFO:teuthology.orchestra.run.smithi196.stdout: "openino_backtrace_fetch": 667, 2024-03-18T16:21:32.861 INFO:teuthology.orchestra.run.smithi196.stdout: "openino_peer_discover": 0, 2024-03-18T16:21:32.861 INFO:teuthology.orchestra.run.smithi196.stdout: "root_rfiles": 112, 2024-03-18T16:21:32.862 INFO:teuthology.orchestra.run.smithi196.stdout: "root_rbytes": 933339, 2024-03-18T16:21:32.862 INFO:teuthology.orchestra.run.smithi196.stdout: "root_rsnaps": 0, 2024-03-18T16:21:32.862 INFO:teuthology.orchestra.run.smithi196.stdout: "scrub_backtrace_fetch": 667, 2024-03-18T16:21:32.862 INFO:teuthology.orchestra.run.smithi196.stdout: "scrub_set_tag": 0, 2024-03-18T16:21:32.862 INFO:teuthology.orchestra.run.smithi196.stdout: "scrub_backtrace_repaired": 0, 2024-03-18T16:21:32.862 INFO:teuthology.orchestra.run.smithi196.stdout: "scrub_inotable_repaired": 0, 2024-03-18T16:21:32.862 INFO:teuthology.orchestra.run.smithi196.stdout: "scrub_dir_inodes": 26, 2024-03-18T16:21:32.863 INFO:teuthology.orchestra.run.smithi196.stdout: "scrub_dir_base_inodes": 0, 2024-03-18T16:21:32.863 INFO:teuthology.orchestra.run.smithi196.stdout: "scrub_dirfrag_rstats": 26, 2024-03-18T16:21:32.863 INFO:teuthology.orchestra.run.smithi196.stdout: "scrub_file_inodes": 641, 2024-03-18T16:21:32.863 INFO:teuthology.orchestra.run.smithi196.stdout: "handle_inode_file_caps": 8, 2024-03-18T16:21:32.863 INFO:teuthology.orchestra.run.smithi196.stdout: "ceph_cap_op_revoke": 47, 2024-03-18T16:21:32.863 INFO:teuthology.orchestra.run.smithi196.stdout: "ceph_cap_op_grant": 1313, 2024-03-18T16:21:32.863 INFO:teuthology.orchestra.run.smithi196.stdout: "ceph_cap_op_trunc": 0, 2024-03-18T16:21:32.864 INFO:teuthology.orchestra.run.smithi196.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-18T16:21:32.864 INFO:teuthology.orchestra.run.smithi196.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-18T16:21:32.864 INFO:teuthology.orchestra.run.smithi196.stdout: "handle_client_caps": 2663, 2024-03-18T16:21:32.864 INFO:teuthology.orchestra.run.smithi196.stdout: "handle_client_caps_dirty": 2261, 2024-03-18T16:21:32.864 INFO:teuthology.orchestra.run.smithi196.stdout: "handle_client_cap_release": 22, 2024-03-18T16:21:32.865 INFO:teuthology.orchestra.run.smithi196.stdout: "process_request_cap_release": 97 2024-03-18T16:21:32.865 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:32.865 INFO:teuthology.orchestra.run.smithi196.stdout: "mds_cache": { 2024-03-18T16:21:32.865 INFO:teuthology.orchestra.run.smithi196.stdout: "dir_update": 0, 2024-03-18T16:21:32.865 INFO:teuthology.orchestra.run.smithi196.stdout: "dir_update_receipt": 3, 2024-03-18T16:21:32.865 INFO:teuthology.orchestra.run.smithi196.stdout: "dir_try_discover": 2, 2024-03-18T16:21:32.866 INFO:teuthology.orchestra.run.smithi196.stdout: "dir_send_discover": 26, 2024-03-18T16:21:32.866 INFO:teuthology.orchestra.run.smithi196.stdout: "dir_handle_discover": 25, 2024-03-18T16:21:32.866 INFO:teuthology.orchestra.run.smithi196.stdout: "num_strays": 0, 2024-03-18T16:21:32.867 INFO:teuthology.orchestra.run.smithi196.stdout: "num_strays_delayed": 0, 2024-03-18T16:21:32.867 INFO:teuthology.orchestra.run.smithi196.stdout: "num_strays_enqueuing": 0, 2024-03-18T16:21:32.867 INFO:teuthology.orchestra.run.smithi196.stdout: "strays_created": 47, 2024-03-18T16:21:32.867 INFO:teuthology.orchestra.run.smithi196.stdout: "strays_enqueued": 47, 2024-03-18T16:21:32.867 INFO:teuthology.orchestra.run.smithi196.stdout: "strays_reintegrated": 0, 2024-03-18T16:21:32.867 INFO:teuthology.orchestra.run.smithi196.stdout: "strays_migrated": 0, 2024-03-18T16:21:32.868 INFO:teuthology.orchestra.run.smithi196.stdout: "num_recovering_processing": 0, 2024-03-18T16:21:32.868 INFO:teuthology.orchestra.run.smithi196.stdout: "num_recovering_enqueued": 0, 2024-03-18T16:21:32.868 INFO:teuthology.orchestra.run.smithi196.stdout: "num_recovering_prioritized": 0, 2024-03-18T16:21:32.869 INFO:teuthology.orchestra.run.smithi196.stdout: "recovery_started": 0, 2024-03-18T16:21:32.869 INFO:teuthology.orchestra.run.smithi196.stdout: "recovery_completed": 0, 2024-03-18T16:21:32.869 INFO:teuthology.orchestra.run.smithi196.stdout: "ireq_quiesce_path": 0, 2024-03-18T16:21:32.870 INFO:teuthology.orchestra.run.smithi196.stdout: "ireq_quiesce_inode": 0, 2024-03-18T16:21:32.870 INFO:teuthology.orchestra.run.smithi196.stdout: "ireq_enqueue_scrub": 0, 2024-03-18T16:21:32.871 INFO:teuthology.orchestra.run.smithi196.stdout: "ireq_exportdir": 25, 2024-03-18T16:21:32.872 INFO:teuthology.orchestra.run.smithi196.stdout: "ireq_flush": 0, 2024-03-18T16:21:32.873 INFO:teuthology.orchestra.run.smithi196.stdout: "ireq_fragmentdir": 1, 2024-03-18T16:21:32.873 INFO:teuthology.orchestra.run.smithi196.stdout: "ireq_fragstats": 0, 2024-03-18T16:21:32.873 INFO:teuthology.orchestra.run.smithi196.stdout: "ireq_inodestats": 0 2024-03-18T16:21:32.873 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:32.873 INFO:teuthology.orchestra.run.smithi196.stdout: "mds_log": { 2024-03-18T16:21:32.873 INFO:teuthology.orchestra.run.smithi196.stdout: "evlrg": 0, 2024-03-18T16:21:32.873 INFO:teuthology.orchestra.run.smithi196.stdout: "evadd": 3016, 2024-03-18T16:21:32.874 INFO:teuthology.orchestra.run.smithi196.stdout: "evex": 0, 2024-03-18T16:21:32.874 INFO:teuthology.orchestra.run.smithi196.stdout: "evtrm": 0, 2024-03-18T16:21:32.874 INFO:teuthology.orchestra.run.smithi196.stdout: "ev": 3016, 2024-03-18T16:21:32.874 INFO:teuthology.orchestra.run.smithi196.stdout: "evexg": 0, 2024-03-18T16:21:32.875 INFO:teuthology.orchestra.run.smithi196.stdout: "evexd": 0, 2024-03-18T16:21:32.875 INFO:teuthology.orchestra.run.smithi196.stdout: "segadd": 5, 2024-03-18T16:21:32.875 INFO:teuthology.orchestra.run.smithi196.stdout: "segex": 0, 2024-03-18T16:21:32.875 INFO:teuthology.orchestra.run.smithi196.stdout: "segtrm": 0, 2024-03-18T16:21:32.875 INFO:teuthology.orchestra.run.smithi196.stdout: "seg": 5, 2024-03-18T16:21:32.876 INFO:teuthology.orchestra.run.smithi196.stdout: "segmjr": 2, 2024-03-18T16:21:32.876 INFO:teuthology.orchestra.run.smithi196.stdout: "segexg": 0, 2024-03-18T16:21:32.876 INFO:teuthology.orchestra.run.smithi196.stdout: "segexd": 0, 2024-03-18T16:21:32.876 INFO:teuthology.orchestra.run.smithi196.stdout: "expos": 4194304, 2024-03-18T16:21:32.877 INFO:teuthology.orchestra.run.smithi196.stdout: "wrpos": 10556918, 2024-03-18T16:21:32.877 INFO:teuthology.orchestra.run.smithi196.stdout: "rdpos": 0, 2024-03-18T16:21:32.877 INFO:teuthology.orchestra.run.smithi196.stdout: "jlat": { 2024-03-18T16:21:32.877 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 1194, 2024-03-18T16:21:32.877 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 34.482731043, 2024-03-18T16:21:32.877 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.028880009 2024-03-18T16:21:32.878 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:32.878 INFO:teuthology.orchestra.run.smithi196.stdout: "replayed": 0 2024-03-18T16:21:32.878 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:32.878 INFO:teuthology.orchestra.run.smithi196.stdout: "mds_mem": { 2024-03-18T16:21:32.878 INFO:teuthology.orchestra.run.smithi196.stdout: "ino": 25, 2024-03-18T16:21:32.879 INFO:teuthology.orchestra.run.smithi196.stdout: "ino+": 378, 2024-03-18T16:21:32.879 INFO:teuthology.orchestra.run.smithi196.stdout: "ino-": 353, 2024-03-18T16:21:32.879 INFO:teuthology.orchestra.run.smithi196.stdout: "dir": 23, 2024-03-18T16:21:32.879 INFO:teuthology.orchestra.run.smithi196.stdout: "dir+": 134, 2024-03-18T16:21:32.880 INFO:teuthology.orchestra.run.smithi196.stdout: "dir-": 111, 2024-03-18T16:21:32.880 INFO:teuthology.orchestra.run.smithi196.stdout: "dn": 18, 2024-03-18T16:21:32.880 INFO:teuthology.orchestra.run.smithi196.stdout: "dn+": 706, 2024-03-18T16:21:32.880 INFO:teuthology.orchestra.run.smithi196.stdout: "dn-": 688, 2024-03-18T16:21:32.881 INFO:teuthology.orchestra.run.smithi196.stdout: "cap": 0, 2024-03-18T16:21:32.881 INFO:teuthology.orchestra.run.smithi196.stdout: "cap+": 386, 2024-03-18T16:21:32.881 INFO:teuthology.orchestra.run.smithi196.stdout: "cap-": 386, 2024-03-18T16:21:32.881 INFO:teuthology.orchestra.run.smithi196.stdout: "rss": 61456, 2024-03-18T16:21:32.881 INFO:teuthology.orchestra.run.smithi196.stdout: "heap": 223612 2024-03-18T16:21:32.881 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:32.882 INFO:teuthology.orchestra.run.smithi196.stdout: "mds_server": { 2024-03-18T16:21:32.882 INFO:teuthology.orchestra.run.smithi196.stdout: "dispatch_client_request": 439, 2024-03-18T16:21:32.882 INFO:teuthology.orchestra.run.smithi196.stdout: "dispatch_server_request": 126, 2024-03-18T16:21:32.882 INFO:teuthology.orchestra.run.smithi196.stdout: "handle_client_request": 350, 2024-03-18T16:21:32.883 INFO:teuthology.orchestra.run.smithi196.stdout: "handle_client_session": 21, 2024-03-18T16:21:32.883 INFO:teuthology.orchestra.run.smithi196.stdout: "handle_peer_request": 303, 2024-03-18T16:21:32.883 INFO:teuthology.orchestra.run.smithi196.stdout: "req_create_latency": { 2024-03-18T16:21:32.883 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 101, 2024-03-18T16:21:32.883 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.148494390, 2024-03-18T16:21:32.883 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.001470241 2024-03-18T16:21:32.883 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:32.883 INFO:teuthology.orchestra.run.smithi196.stdout: "req_getattr_latency": { 2024-03-18T16:21:32.883 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 2, 2024-03-18T16:21:32.883 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000567042, 2024-03-18T16:21:32.883 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000283521 2024-03-18T16:21:32.883 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:32.883 INFO:teuthology.orchestra.run.smithi196.stdout: "req_getfilelock_latency": { 2024-03-18T16:21:32.883 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:32.883 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:32.883 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:32.884 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:32.884 INFO:teuthology.orchestra.run.smithi196.stdout: "req_link_latency": { 2024-03-18T16:21:32.884 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:32.884 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:32.884 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:32.884 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:32.884 INFO:teuthology.orchestra.run.smithi196.stdout: "req_lookup_latency": { 2024-03-18T16:21:32.884 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 137, 2024-03-18T16:21:32.884 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.289214868, 2024-03-18T16:21:32.884 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.002111057 2024-03-18T16:21:32.884 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:32.884 INFO:teuthology.orchestra.run.smithi196.stdout: "req_lookuphash_latency": { 2024-03-18T16:21:32.884 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:32.884 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:32.884 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:32.885 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:32.885 INFO:teuthology.orchestra.run.smithi196.stdout: "req_lookupino_latency": { 2024-03-18T16:21:32.885 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:32.885 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:32.885 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:32.885 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:32.885 INFO:teuthology.orchestra.run.smithi196.stdout: "req_lookupname_latency": { 2024-03-18T16:21:32.885 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:32.885 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:32.885 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:32.885 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:32.885 INFO:teuthology.orchestra.run.smithi196.stdout: "req_lookupparent_latency": { 2024-03-18T16:21:32.885 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:32.885 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:32.885 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:32.886 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:32.886 INFO:teuthology.orchestra.run.smithi196.stdout: "req_lookupsnap_latency": { 2024-03-18T16:21:32.886 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:32.886 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:32.886 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:32.886 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:32.886 INFO:teuthology.orchestra.run.smithi196.stdout: "req_lssnap_latency": { 2024-03-18T16:21:32.886 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:32.887 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:32.887 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:32.887 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:32.887 INFO:teuthology.orchestra.run.smithi196.stdout: "req_mkdir_latency": { 2024-03-18T16:21:32.887 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:32.887 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:32.887 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:32.887 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:32.887 INFO:teuthology.orchestra.run.smithi196.stdout: "req_mknod_latency": { 2024-03-18T16:21:32.887 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:32.887 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:32.888 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:32.888 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:32.888 INFO:teuthology.orchestra.run.smithi196.stdout: "req_mksnap_latency": { 2024-03-18T16:21:32.888 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:32.888 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:32.888 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:32.888 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:32.888 INFO:teuthology.orchestra.run.smithi196.stdout: "req_open_latency": { 2024-03-18T16:21:32.888 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 37, 2024-03-18T16:21:32.888 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.044228262, 2024-03-18T16:21:32.888 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.001195358 2024-03-18T16:21:32.888 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:32.888 INFO:teuthology.orchestra.run.smithi196.stdout: "req_readdir_latency": { 2024-03-18T16:21:32.888 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 19, 2024-03-18T16:21:32.888 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.008911898, 2024-03-18T16:21:32.889 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000469047 2024-03-18T16:21:32.889 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:32.889 INFO:teuthology.orchestra.run.smithi196.stdout: "req_rename_latency": { 2024-03-18T16:21:32.889 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:32.889 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:32.889 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:32.889 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:32.889 INFO:teuthology.orchestra.run.smithi196.stdout: "req_renamesnap_latency": { 2024-03-18T16:21:32.889 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:32.889 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:32.889 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:32.889 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:32.889 INFO:teuthology.orchestra.run.smithi196.stdout: "req_snapdiff_latency": { 2024-03-18T16:21:32.889 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:32.889 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:32.890 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:32.890 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:32.890 INFO:teuthology.orchestra.run.smithi196.stdout: "req_rmdir_latency": { 2024-03-18T16:21:32.890 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 2, 2024-03-18T16:21:32.890 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.535385091, 2024-03-18T16:21:32.890 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.267692545 2024-03-18T16:21:32.890 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:32.890 INFO:teuthology.orchestra.run.smithi196.stdout: "req_rmsnap_latency": { 2024-03-18T16:21:32.890 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:32.890 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:32.890 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:32.890 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:32.890 INFO:teuthology.orchestra.run.smithi196.stdout: "req_rmxattr_latency": { 2024-03-18T16:21:32.890 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:32.890 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:32.891 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:32.891 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:32.891 INFO:teuthology.orchestra.run.smithi196.stdout: "req_setattr_latency": { 2024-03-18T16:21:32.891 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:32.891 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:32.891 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:32.891 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:32.891 INFO:teuthology.orchestra.run.smithi196.stdout: "req_setdirlayout_latency": { 2024-03-18T16:21:32.891 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:32.891 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:32.891 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:32.891 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:32.891 INFO:teuthology.orchestra.run.smithi196.stdout: "req_setfilelock_latency": { 2024-03-18T16:21:32.891 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:32.891 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:32.892 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:32.892 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:32.892 INFO:teuthology.orchestra.run.smithi196.stdout: "req_setlayout_latency": { 2024-03-18T16:21:32.892 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:32.892 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:32.892 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:32.892 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:32.892 INFO:teuthology.orchestra.run.smithi196.stdout: "req_setxattr_latency": { 2024-03-18T16:21:32.892 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:32.892 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:32.892 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:32.892 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:32.892 INFO:teuthology.orchestra.run.smithi196.stdout: "req_symlink_latency": { 2024-03-18T16:21:32.892 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:32.892 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:32.892 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:32.893 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:32.893 INFO:teuthology.orchestra.run.smithi196.stdout: "req_unlink_latency": { 2024-03-18T16:21:32.893 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 45, 2024-03-18T16:21:32.893 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.209189636, 2024-03-18T16:21:32.893 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.004648658 2024-03-18T16:21:32.893 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:32.893 INFO:teuthology.orchestra.run.smithi196.stdout: "cap_revoke_eviction": 0, 2024-03-18T16:21:32.893 INFO:teuthology.orchestra.run.smithi196.stdout: "cap_acquisition_throttle": 0, 2024-03-18T16:21:32.893 INFO:teuthology.orchestra.run.smithi196.stdout: "req_getvxattr_latency": { 2024-03-18T16:21:32.893 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:32.893 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:32.893 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:32.893 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-03-18T16:21:32.893 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:32.893 INFO:teuthology.orchestra.run.smithi196.stdout: "mds_sessions": { 2024-03-18T16:21:32.894 INFO:teuthology.orchestra.run.smithi196.stdout: "session_count": 1, 2024-03-18T16:21:32.894 INFO:teuthology.orchestra.run.smithi196.stdout: "session_add": 2, 2024-03-18T16:21:32.894 INFO:teuthology.orchestra.run.smithi196.stdout: "session_remove": 1, 2024-03-18T16:21:32.894 INFO:teuthology.orchestra.run.smithi196.stdout: "sessions_open": 1, 2024-03-18T16:21:32.894 INFO:teuthology.orchestra.run.smithi196.stdout: "sessions_stale": 0, 2024-03-18T16:21:32.894 INFO:teuthology.orchestra.run.smithi196.stdout: "total_load": 91, 2024-03-18T16:21:32.894 INFO:teuthology.orchestra.run.smithi196.stdout: "average_load": 91, 2024-03-18T16:21:32.895 INFO:teuthology.orchestra.run.smithi196.stdout: "avg_session_uptime": 609, 2024-03-18T16:21:32.895 INFO:teuthology.orchestra.run.smithi196.stdout: "mdthresh_evicted": 0 2024-03-18T16:21:32.895 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:32.895 INFO:teuthology.orchestra.run.smithi196.stdout: "mempool": { 2024-03-18T16:21:32.895 INFO:teuthology.orchestra.run.smithi196.stdout: "bloom_filter_bytes": 0, 2024-03-18T16:21:32.895 INFO:teuthology.orchestra.run.smithi196.stdout: "bloom_filter_items": 0, 2024-03-18T16:21:32.895 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T16:21:32.895 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_alloc_items": 0, 2024-03-18T16:21:32.896 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T16:21:32.896 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_cache_data_items": 0, 2024-03-18T16:21:32.896 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T16:21:32.896 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T16:21:32.896 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T16:21:32.896 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T16:21:32.896 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T16:21:32.896 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_cache_other_items": 0, 2024-03-18T16:21:32.896 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T16:21:32.896 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T16:21:32.896 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_extent_bytes": 0, 2024-03-18T16:21:32.897 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_extent_items": 0, 2024-03-18T16:21:32.897 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_blob_bytes": 0, 2024-03-18T16:21:32.897 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_blob_items": 0, 2024-03-18T16:21:32.897 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T16:21:32.897 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T16:21:32.897 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T16:21:32.897 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T16:21:32.897 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T16:21:32.897 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_fsck_items": 0, 2024-03-18T16:21:32.897 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_txc_bytes": 0, 2024-03-18T16:21:32.898 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_txc_items": 0, 2024-03-18T16:21:32.898 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T16:21:32.898 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T16:21:32.898 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_writing_bytes": 0, 2024-03-18T16:21:32.898 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_writing_items": 0, 2024-03-18T16:21:32.898 INFO:teuthology.orchestra.run.smithi196.stdout: "bluefs_bytes": 0, 2024-03-18T16:21:32.898 INFO:teuthology.orchestra.run.smithi196.stdout: "bluefs_items": 0, 2024-03-18T16:21:32.898 INFO:teuthology.orchestra.run.smithi196.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T16:21:32.898 INFO:teuthology.orchestra.run.smithi196.stdout: "bluefs_file_reader_items": 0, 2024-03-18T16:21:32.898 INFO:teuthology.orchestra.run.smithi196.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T16:21:32.898 INFO:teuthology.orchestra.run.smithi196.stdout: "bluefs_file_writer_items": 0, 2024-03-18T16:21:32.899 INFO:teuthology.orchestra.run.smithi196.stdout: "buffer_anon_bytes": 315047, 2024-03-18T16:21:32.899 INFO:teuthology.orchestra.run.smithi196.stdout: "buffer_anon_items": 119, 2024-03-18T16:21:32.899 INFO:teuthology.orchestra.run.smithi196.stdout: "buffer_meta_bytes": 0, 2024-03-18T16:21:32.899 INFO:teuthology.orchestra.run.smithi196.stdout: "buffer_meta_items": 0, 2024-03-18T16:21:32.899 INFO:teuthology.orchestra.run.smithi196.stdout: "osd_bytes": 0, 2024-03-18T16:21:32.899 INFO:teuthology.orchestra.run.smithi196.stdout: "osd_items": 0, 2024-03-18T16:21:32.899 INFO:teuthology.orchestra.run.smithi196.stdout: "osd_mapbl_bytes": 0, 2024-03-18T16:21:32.899 INFO:teuthology.orchestra.run.smithi196.stdout: "osd_mapbl_items": 0, 2024-03-18T16:21:32.899 INFO:teuthology.orchestra.run.smithi196.stdout: "osd_pglog_bytes": 0, 2024-03-18T16:21:32.899 INFO:teuthology.orchestra.run.smithi196.stdout: "osd_pglog_items": 0, 2024-03-18T16:21:32.899 INFO:teuthology.orchestra.run.smithi196.stdout: "osdmap_bytes": 8888, 2024-03-18T16:21:32.899 INFO:teuthology.orchestra.run.smithi196.stdout: "osdmap_items": 170, 2024-03-18T16:21:32.900 INFO:teuthology.orchestra.run.smithi196.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T16:21:32.900 INFO:teuthology.orchestra.run.smithi196.stdout: "osdmap_mapping_items": 0, 2024-03-18T16:21:32.900 INFO:teuthology.orchestra.run.smithi196.stdout: "pgmap_bytes": 0, 2024-03-18T16:21:32.900 INFO:teuthology.orchestra.run.smithi196.stdout: "pgmap_items": 0, 2024-03-18T16:21:32.900 INFO:teuthology.orchestra.run.smithi196.stdout: "mds_co_bytes": 98909, 2024-03-18T16:21:32.900 INFO:teuthology.orchestra.run.smithi196.stdout: "mds_co_items": 409, 2024-03-18T16:21:32.900 INFO:teuthology.orchestra.run.smithi196.stdout: "unittest_1_bytes": 0, 2024-03-18T16:21:32.900 INFO:teuthology.orchestra.run.smithi196.stdout: "unittest_1_items": 0, 2024-03-18T16:21:32.900 INFO:teuthology.orchestra.run.smithi196.stdout: "unittest_2_bytes": 0, 2024-03-18T16:21:32.900 INFO:teuthology.orchestra.run.smithi196.stdout: "unittest_2_items": 0 2024-03-18T16:21:32.900 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:32.900 INFO:teuthology.orchestra.run.smithi196.stdout: "objecter": { 2024-03-18T16:21:32.901 INFO:teuthology.orchestra.run.smithi196.stdout: "op_active": 0, 2024-03-18T16:21:32.901 INFO:teuthology.orchestra.run.smithi196.stdout: "op_laggy": 0, 2024-03-18T16:21:32.901 INFO:teuthology.orchestra.run.smithi196.stdout: "op_send": 2787, 2024-03-18T16:21:32.901 INFO:teuthology.orchestra.run.smithi196.stdout: "op_send_bytes": 6571657, 2024-03-18T16:21:32.901 INFO:teuthology.orchestra.run.smithi196.stdout: "op_resend": 33, 2024-03-18T16:21:32.901 INFO:teuthology.orchestra.run.smithi196.stdout: "op_reply": 2787, 2024-03-18T16:21:32.901 INFO:teuthology.orchestra.run.smithi196.stdout: "op_latency": { 2024-03-18T16:21:32.901 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 2787, 2024-03-18T16:21:32.901 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 187.669193814, 2024-03-18T16:21:32.901 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.067337349 2024-03-18T16:21:32.901 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:32.901 INFO:teuthology.orchestra.run.smithi196.stdout: "op_inflight": 0, 2024-03-18T16:21:32.902 INFO:teuthology.orchestra.run.smithi196.stdout: "oplen_avg": { 2024-03-18T16:21:32.902 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 2787, 2024-03-18T16:21:32.902 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 2989 2024-03-18T16:21:32.902 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:32.902 INFO:teuthology.orchestra.run.smithi196.stdout: "op": 2787, 2024-03-18T16:21:32.902 INFO:teuthology.orchestra.run.smithi196.stdout: "op_r": 674, 2024-03-18T16:21:32.902 INFO:teuthology.orchestra.run.smithi196.stdout: "op_w": 2113, 2024-03-18T16:21:32.902 INFO:teuthology.orchestra.run.smithi196.stdout: "op_rmw": 0, 2024-03-18T16:21:32.902 INFO:teuthology.orchestra.run.smithi196.stdout: "op_pg": 0, 2024-03-18T16:21:32.902 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_stat": 0, 2024-03-18T16:21:32.902 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_create": 10, 2024-03-18T16:21:32.902 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_read": 7, 2024-03-18T16:21:32.902 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_write": 1201, 2024-03-18T16:21:32.902 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_writefull": 38, 2024-03-18T16:21:32.903 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_writesame": 0, 2024-03-18T16:21:32.903 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_append": 0, 2024-03-18T16:21:32.903 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_zero": 0, 2024-03-18T16:21:32.903 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_truncate": 1, 2024-03-18T16:21:32.903 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_delete": 713, 2024-03-18T16:21:32.903 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_mapext": 0, 2024-03-18T16:21:32.903 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_sparse_read": 0, 2024-03-18T16:21:32.903 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_clonerange": 0, 2024-03-18T16:21:32.903 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_getxattr": 667, 2024-03-18T16:21:32.903 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_setxattr": 20, 2024-03-18T16:21:32.903 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_cmpxattr": 0, 2024-03-18T16:21:32.903 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_rmxattr": 0, 2024-03-18T16:21:32.903 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_resetxattrs": 0, 2024-03-18T16:21:32.904 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_call": 0, 2024-03-18T16:21:32.904 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_watch": 0, 2024-03-18T16:21:32.904 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_notify": 0, 2024-03-18T16:21:32.904 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_src_cmpxattr": 0, 2024-03-18T16:21:32.904 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_pgls": 0, 2024-03-18T16:21:32.904 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_pgls_filter": 0, 2024-03-18T16:21:32.904 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_other": 0, 2024-03-18T16:21:32.904 INFO:teuthology.orchestra.run.smithi196.stdout: "linger_active": 0, 2024-03-18T16:21:32.904 INFO:teuthology.orchestra.run.smithi196.stdout: "linger_send": 0, 2024-03-18T16:21:32.904 INFO:teuthology.orchestra.run.smithi196.stdout: "linger_resend": 0, 2024-03-18T16:21:32.904 INFO:teuthology.orchestra.run.smithi196.stdout: "linger_ping": 0, 2024-03-18T16:21:32.904 INFO:teuthology.orchestra.run.smithi196.stdout: "poolop_active": 0, 2024-03-18T16:21:32.905 INFO:teuthology.orchestra.run.smithi196.stdout: "poolop_send": 0, 2024-03-18T16:21:32.905 INFO:teuthology.orchestra.run.smithi196.stdout: "poolop_resend": 0, 2024-03-18T16:21:32.905 INFO:teuthology.orchestra.run.smithi196.stdout: "poolstat_active": 0, 2024-03-18T16:21:32.905 INFO:teuthology.orchestra.run.smithi196.stdout: "poolstat_send": 0, 2024-03-18T16:21:32.905 INFO:teuthology.orchestra.run.smithi196.stdout: "poolstat_resend": 0, 2024-03-18T16:21:32.905 INFO:teuthology.orchestra.run.smithi196.stdout: "statfs_active": 0, 2024-03-18T16:21:32.905 INFO:teuthology.orchestra.run.smithi196.stdout: "statfs_send": 0, 2024-03-18T16:21:32.905 INFO:teuthology.orchestra.run.smithi196.stdout: "statfs_resend": 0, 2024-03-18T16:21:32.905 INFO:teuthology.orchestra.run.smithi196.stdout: "command_active": 0, 2024-03-18T16:21:32.905 INFO:teuthology.orchestra.run.smithi196.stdout: "command_send": 0, 2024-03-18T16:21:32.905 INFO:teuthology.orchestra.run.smithi196.stdout: "command_resend": 0, 2024-03-18T16:21:32.905 INFO:teuthology.orchestra.run.smithi196.stdout: "map_epoch": 93, 2024-03-18T16:21:32.905 INFO:teuthology.orchestra.run.smithi196.stdout: "map_full": 0, 2024-03-18T16:21:32.905 INFO:teuthology.orchestra.run.smithi196.stdout: "map_inc": 4, 2024-03-18T16:21:32.906 INFO:teuthology.orchestra.run.smithi196.stdout: "osd_sessions": 12, 2024-03-18T16:21:32.906 INFO:teuthology.orchestra.run.smithi196.stdout: "osd_session_open": 12, 2024-03-18T16:21:32.906 INFO:teuthology.orchestra.run.smithi196.stdout: "osd_session_close": 0, 2024-03-18T16:21:32.906 INFO:teuthology.orchestra.run.smithi196.stdout: "osd_laggy": 0, 2024-03-18T16:21:32.906 INFO:teuthology.orchestra.run.smithi196.stdout: "omap_wr": 228, 2024-03-18T16:21:32.906 INFO:teuthology.orchestra.run.smithi196.stdout: "omap_rd": 0, 2024-03-18T16:21:32.906 INFO:teuthology.orchestra.run.smithi196.stdout: "omap_del": 104 2024-03-18T16:21:32.906 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:32.906 INFO:teuthology.orchestra.run.smithi196.stdout: "oft": { 2024-03-18T16:21:32.906 INFO:teuthology.orchestra.run.smithi196.stdout: "omap_total_objs": 1, 2024-03-18T16:21:32.906 INFO:teuthology.orchestra.run.smithi196.stdout: "omap_total_kv_pairs": 0, 2024-03-18T16:21:32.907 INFO:teuthology.orchestra.run.smithi196.stdout: "omap_total_updates": 0, 2024-03-18T16:21:32.907 INFO:teuthology.orchestra.run.smithi196.stdout: "omap_total_removes": 0 2024-03-18T16:21:32.907 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:32.907 INFO:teuthology.orchestra.run.smithi196.stdout: "osd-slow-ops": { 2024-03-18T16:21:32.907 INFO:teuthology.orchestra.run.smithi196.stdout: "slow_ops_count": 0 2024-03-18T16:21:32.907 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:32.907 INFO:teuthology.orchestra.run.smithi196.stdout: "purge_queue": { 2024-03-18T16:21:32.907 INFO:teuthology.orchestra.run.smithi196.stdout: "pq_executing_ops": 0, 2024-03-18T16:21:32.907 INFO:teuthology.orchestra.run.smithi196.stdout: "pq_executing_ops_high_water": 27, 2024-03-18T16:21:32.907 INFO:teuthology.orchestra.run.smithi196.stdout: "pq_executing": 0, 2024-03-18T16:21:32.907 INFO:teuthology.orchestra.run.smithi196.stdout: "pq_executing_high_water": 5, 2024-03-18T16:21:32.907 INFO:teuthology.orchestra.run.smithi196.stdout: "pq_executed_ops": 700, 2024-03-18T16:21:32.907 INFO:teuthology.orchestra.run.smithi196.stdout: "pq_executed": 47, 2024-03-18T16:21:32.908 INFO:teuthology.orchestra.run.smithi196.stdout: "pq_item_in_journal": 0 2024-03-18T16:21:32.908 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:32.908 INFO:teuthology.orchestra.run.smithi196.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T16:21:32.908 INFO:teuthology.orchestra.run.smithi196.stdout: "val": 0, 2024-03-18T16:21:32.908 INFO:teuthology.orchestra.run.smithi196.stdout: "max": 104857600, 2024-03-18T16:21:32.908 INFO:teuthology.orchestra.run.smithi196.stdout: "get_started": 0, 2024-03-18T16:21:32.908 INFO:teuthology.orchestra.run.smithi196.stdout: "get": 12353, 2024-03-18T16:21:32.908 INFO:teuthology.orchestra.run.smithi196.stdout: "get_sum": 3928974, 2024-03-18T16:21:32.908 INFO:teuthology.orchestra.run.smithi196.stdout: "get_or_fail_fail": 0, 2024-03-18T16:21:32.908 INFO:teuthology.orchestra.run.smithi196.stdout: "get_or_fail_success": 12353, 2024-03-18T16:21:32.908 INFO:teuthology.orchestra.run.smithi196.stdout: "take": 0, 2024-03-18T16:21:32.908 INFO:teuthology.orchestra.run.smithi196.stdout: "take_sum": 0, 2024-03-18T16:21:32.909 INFO:teuthology.orchestra.run.smithi196.stdout: "put": 12353, 2024-03-18T16:21:32.909 INFO:teuthology.orchestra.run.smithi196.stdout: "put_sum": 3928974, 2024-03-18T16:21:32.909 INFO:teuthology.orchestra.run.smithi196.stdout: "wait": { 2024-03-18T16:21:32.909 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:32.909 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:32.909 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:32.909 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-03-18T16:21:32.909 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:32.909 INFO:teuthology.orchestra.run.smithi196.stdout: "throttle-objecter_bytes": { 2024-03-18T16:21:32.909 INFO:teuthology.orchestra.run.smithi196.stdout: "val": 0, 2024-03-18T16:21:32.909 INFO:teuthology.orchestra.run.smithi196.stdout: "max": 104857600, 2024-03-18T16:21:32.909 INFO:teuthology.orchestra.run.smithi196.stdout: "get_started": 0, 2024-03-18T16:21:32.909 INFO:teuthology.orchestra.run.smithi196.stdout: "get": 0, 2024-03-18T16:21:32.909 INFO:teuthology.orchestra.run.smithi196.stdout: "get_sum": 0, 2024-03-18T16:21:32.909 INFO:teuthology.orchestra.run.smithi196.stdout: "get_or_fail_fail": 0, 2024-03-18T16:21:32.910 INFO:teuthology.orchestra.run.smithi196.stdout: "get_or_fail_success": 0, 2024-03-18T16:21:32.910 INFO:teuthology.orchestra.run.smithi196.stdout: "take": 2787, 2024-03-18T16:21:32.910 INFO:teuthology.orchestra.run.smithi196.stdout: "take_sum": 6576772, 2024-03-18T16:21:32.910 INFO:teuthology.orchestra.run.smithi196.stdout: "put": 2787, 2024-03-18T16:21:32.910 INFO:teuthology.orchestra.run.smithi196.stdout: "put_sum": 6576772, 2024-03-18T16:21:32.910 INFO:teuthology.orchestra.run.smithi196.stdout: "wait": { 2024-03-18T16:21:32.910 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:32.910 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:32.910 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:32.910 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-03-18T16:21:32.910 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:32.910 INFO:teuthology.orchestra.run.smithi196.stdout: "throttle-objecter_ops": { 2024-03-18T16:21:32.911 INFO:teuthology.orchestra.run.smithi196.stdout: "val": 0, 2024-03-18T16:21:32.911 INFO:teuthology.orchestra.run.smithi196.stdout: "max": 1024, 2024-03-18T16:21:32.911 INFO:teuthology.orchestra.run.smithi196.stdout: "get_started": 0, 2024-03-18T16:21:32.911 INFO:teuthology.orchestra.run.smithi196.stdout: "get": 0, 2024-03-18T16:21:32.911 INFO:teuthology.orchestra.run.smithi196.stdout: "get_sum": 0, 2024-03-18T16:21:32.911 INFO:teuthology.orchestra.run.smithi196.stdout: "get_or_fail_fail": 0, 2024-03-18T16:21:32.911 INFO:teuthology.orchestra.run.smithi196.stdout: "get_or_fail_success": 0, 2024-03-18T16:21:32.911 INFO:teuthology.orchestra.run.smithi196.stdout: "take": 2787, 2024-03-18T16:21:32.911 INFO:teuthology.orchestra.run.smithi196.stdout: "take_sum": 2787, 2024-03-18T16:21:32.911 INFO:teuthology.orchestra.run.smithi196.stdout: "put": 2787, 2024-03-18T16:21:32.911 INFO:teuthology.orchestra.run.smithi196.stdout: "put_sum": 2787, 2024-03-18T16:21:32.911 INFO:teuthology.orchestra.run.smithi196.stdout: "wait": { 2024-03-18T16:21:32.911 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:32.912 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:32.912 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:32.912 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-03-18T16:21:32.912 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:32.912 INFO:teuthology.orchestra.run.smithi196.stdout: "throttle-write_buf_throttle": { 2024-03-18T16:21:32.912 INFO:teuthology.orchestra.run.smithi196.stdout: "val": 0, 2024-03-18T16:21:32.912 INFO:teuthology.orchestra.run.smithi196.stdout: "max": 3758096384, 2024-03-18T16:21:32.912 INFO:teuthology.orchestra.run.smithi196.stdout: "get_started": 0, 2024-03-18T16:21:32.912 INFO:teuthology.orchestra.run.smithi196.stdout: "get": 47, 2024-03-18T16:21:32.912 INFO:teuthology.orchestra.run.smithi196.stdout: "get_sum": 5115, 2024-03-18T16:21:32.912 INFO:teuthology.orchestra.run.smithi196.stdout: "get_or_fail_fail": 0, 2024-03-18T16:21:32.912 INFO:teuthology.orchestra.run.smithi196.stdout: "get_or_fail_success": 47, 2024-03-18T16:21:32.912 INFO:teuthology.orchestra.run.smithi196.stdout: "take": 0, 2024-03-18T16:21:32.913 INFO:teuthology.orchestra.run.smithi196.stdout: "take_sum": 0, 2024-03-18T16:21:32.913 INFO:teuthology.orchestra.run.smithi196.stdout: "put": 7, 2024-03-18T16:21:32.913 INFO:teuthology.orchestra.run.smithi196.stdout: "put_sum": 5115, 2024-03-18T16:21:32.913 INFO:teuthology.orchestra.run.smithi196.stdout: "wait": { 2024-03-18T16:21:32.913 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:32.913 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:32.913 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:32.913 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-03-18T16:21:32.913 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:32.913 INFO:teuthology.orchestra.run.smithi196.stdout: "throttle-write_buf_throttle-0x555aecce8500": { 2024-03-18T16:21:32.913 INFO:teuthology.orchestra.run.smithi196.stdout: "val": 0, 2024-03-18T16:21:32.913 INFO:teuthology.orchestra.run.smithi196.stdout: "max": 3758096384, 2024-03-18T16:21:32.913 INFO:teuthology.orchestra.run.smithi196.stdout: "get_started": 0, 2024-03-18T16:21:32.913 INFO:teuthology.orchestra.run.smithi196.stdout: "get": 3016, 2024-03-18T16:21:32.914 INFO:teuthology.orchestra.run.smithi196.stdout: "get_sum": 6362614, 2024-03-18T16:21:32.914 INFO:teuthology.orchestra.run.smithi196.stdout: "get_or_fail_fail": 0, 2024-03-18T16:21:32.914 INFO:teuthology.orchestra.run.smithi196.stdout: "get_or_fail_success": 3016, 2024-03-18T16:21:32.914 INFO:teuthology.orchestra.run.smithi196.stdout: "take": 0, 2024-03-18T16:21:32.914 INFO:teuthology.orchestra.run.smithi196.stdout: "take_sum": 0, 2024-03-18T16:21:32.914 INFO:teuthology.orchestra.run.smithi196.stdout: "put": 1194, 2024-03-18T16:21:32.914 INFO:teuthology.orchestra.run.smithi196.stdout: "put_sum": 6362614, 2024-03-18T16:21:32.914 INFO:teuthology.orchestra.run.smithi196.stdout: "wait": { 2024-03-18T16:21:32.914 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:32.914 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:32.914 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:32.914 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-03-18T16:21:32.914 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-03-18T16:21:32.914 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-03-18T16:21:33.230 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:33 smithi196 ceph-mon[36307]: pgmap v766: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T16:21:33.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:33 smithi039 ceph-mon[31721]: pgmap v766: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T16:21:33.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:33 smithi194 ceph-mon[36200]: pgmap v766: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T16:21:33.758 INFO:tasks.check_counter:Daemon mds.i mds.exported=25 2024-03-18T16:21:33.758 INFO:tasks.check_counter:Daemon mds.i mds.imported=25 2024-03-18T16:21:33.758 INFO:tasks.check_counter:Daemon mds.i mds.dir_split=1 2024-03-18T16:21:33.758 DEBUG:tasks.check_counter:Getting stats from l 2024-03-18T16:21:33.758 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af shell --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 -- ceph daemon mds.l perf dump 2024-03-18T16:21:34.019 INFO:teuthology.orchestra.run.smithi196.stderr:Inferring config /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/mon.c/config 2024-03-18T16:21:35.209 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:35 smithi196 ceph-mon[36307]: pgmap v767: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-18T16:21:35.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:35 smithi039 ceph-mon[31721]: pgmap v767: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-18T16:21:35.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:35 smithi194 ceph-mon[36200]: pgmap v767: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-03-18T16:21:35.827 INFO:teuthology.orchestra.run.smithi196.stdout:{ 2024-03-18T16:21:35.827 INFO:teuthology.orchestra.run.smithi196.stdout: "AsyncMessenger::Worker-0": { 2024-03-18T16:21:35.827 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_recv_messages": 286, 2024-03-18T16:21:35.827 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_messages": 259, 2024-03-18T16:21:35.827 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_recv_bytes": 665042, 2024-03-18T16:21:35.828 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_bytes": 155700, 2024-03-18T16:21:35.828 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_created_connections": 5, 2024-03-18T16:21:35.828 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_active_connections": 1, 2024-03-18T16:21:35.828 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_total_time": 0.033056315, 2024-03-18T16:21:35.828 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_send_time": 0.011634694, 2024-03-18T16:21:35.828 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_recv_time": 0.016443943, 2024-03-18T16:21:35.828 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_fast_dispatch_time": 0.005256006, 2024-03-18T16:21:35.828 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T16:21:35.828 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 258, 2024-03-18T16:21:35.828 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.008136862, 2024-03-18T16:21:35.828 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000031538 2024-03-18T16:21:35.828 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:35.828 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_handle_ack_lat": { 2024-03-18T16:21:35.828 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:35.829 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:35.829 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:35.829 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:35.829 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_recv_encrypted_bytes": 664208, 2024-03-18T16:21:35.829 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_encrypted_bytes": 155376 2024-03-18T16:21:35.829 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:35.829 INFO:teuthology.orchestra.run.smithi196.stdout: "AsyncMessenger::Worker-1": { 2024-03-18T16:21:35.829 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_recv_messages": 670, 2024-03-18T16:21:35.829 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_messages": 858, 2024-03-18T16:21:35.829 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_recv_bytes": 166764, 2024-03-18T16:21:35.829 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_bytes": 686835, 2024-03-18T16:21:35.829 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_created_connections": 6, 2024-03-18T16:21:35.829 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_active_connections": 4, 2024-03-18T16:21:35.829 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_total_time": 0.063006688, 2024-03-18T16:21:35.829 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_send_time": 0.019407989, 2024-03-18T16:21:35.829 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_recv_time": 0.014273506, 2024-03-18T16:21:35.830 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_fast_dispatch_time": 0.023233597, 2024-03-18T16:21:35.830 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T16:21:35.830 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 857, 2024-03-18T16:21:35.830 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.034602091, 2024-03-18T16:21:35.830 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000040375 2024-03-18T16:21:35.830 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:35.830 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_handle_ack_lat": { 2024-03-18T16:21:35.830 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:35.830 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:35.830 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:35.830 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:35.830 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_recv_encrypted_bytes": 3152, 2024-03-18T16:21:35.830 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_encrypted_bytes": 475360 2024-03-18T16:21:35.830 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:35.830 INFO:teuthology.orchestra.run.smithi196.stdout: "AsyncMessenger::Worker-2": { 2024-03-18T16:21:35.831 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_recv_messages": 1765, 2024-03-18T16:21:35.831 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_messages": 1767, 2024-03-18T16:21:35.831 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_recv_bytes": 1671714, 2024-03-18T16:21:35.831 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_bytes": 561328, 2024-03-18T16:21:35.831 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_created_connections": 4, 2024-03-18T16:21:35.831 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_active_connections": 3, 2024-03-18T16:21:35.831 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_total_time": 0.133672233, 2024-03-18T16:21:35.831 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_send_time": 0.036593770, 2024-03-18T16:21:35.831 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_recv_time": 0.036290726, 2024-03-18T16:21:35.831 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_running_fast_dispatch_time": 0.050859575, 2024-03-18T16:21:35.831 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_messages_queue_lat": { 2024-03-18T16:21:35.831 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 1765, 2024-03-18T16:21:35.831 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.061368436, 2024-03-18T16:21:35.831 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000034769 2024-03-18T16:21:35.831 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:35.831 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_handle_ack_lat": { 2024-03-18T16:21:35.831 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:35.832 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:35.832 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:35.832 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:35.832 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-18T16:21:35.832 INFO:teuthology.orchestra.run.smithi196.stdout: "msgr_send_encrypted_bytes": 0 2024-03-18T16:21:35.832 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:35.832 INFO:teuthology.orchestra.run.smithi196.stdout: "cct": { 2024-03-18T16:21:35.832 INFO:teuthology.orchestra.run.smithi196.stdout: "total_workers": 1, 2024-03-18T16:21:35.832 INFO:teuthology.orchestra.run.smithi196.stdout: "unhealthy_workers": 0 2024-03-18T16:21:35.832 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:35.832 INFO:teuthology.orchestra.run.smithi196.stdout: "finisher-MDSRank": { 2024-03-18T16:21:35.832 INFO:teuthology.orchestra.run.smithi196.stdout: "queue_len": 0, 2024-03-18T16:21:35.832 INFO:teuthology.orchestra.run.smithi196.stdout: "complete_latency": { 2024-03-18T16:21:35.832 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 4166, 2024-03-18T16:21:35.833 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.096376729, 2024-03-18T16:21:35.833 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000023134 2024-03-18T16:21:35.833 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-03-18T16:21:35.833 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:35.833 INFO:teuthology.orchestra.run.smithi196.stdout: "finisher-PurgeQueue": { 2024-03-18T16:21:35.833 INFO:teuthology.orchestra.run.smithi196.stdout: "queue_len": 0, 2024-03-18T16:21:35.833 INFO:teuthology.orchestra.run.smithi196.stdout: "complete_latency": { 2024-03-18T16:21:35.833 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:35.833 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:35.833 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:35.833 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-03-18T16:21:35.833 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:35.833 INFO:teuthology.orchestra.run.smithi196.stdout: "mds": { 2024-03-18T16:21:35.833 INFO:teuthology.orchestra.run.smithi196.stdout: "request": 0, 2024-03-18T16:21:35.833 INFO:teuthology.orchestra.run.smithi196.stdout: "reply": 0, 2024-03-18T16:21:35.833 INFO:teuthology.orchestra.run.smithi196.stdout: "reply_latency": { 2024-03-18T16:21:35.833 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:35.834 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:35.834 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:35.834 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:35.834 INFO:teuthology.orchestra.run.smithi196.stdout: "slow_reply": 0, 2024-03-18T16:21:35.834 INFO:teuthology.orchestra.run.smithi196.stdout: "forward": 0, 2024-03-18T16:21:35.834 INFO:teuthology.orchestra.run.smithi196.stdout: "dir_fetch_complete": 0, 2024-03-18T16:21:35.834 INFO:teuthology.orchestra.run.smithi196.stdout: "dir_fetch_keys": 0, 2024-03-18T16:21:35.834 INFO:teuthology.orchestra.run.smithi196.stdout: "dir_commit": 0, 2024-03-18T16:21:35.834 INFO:teuthology.orchestra.run.smithi196.stdout: "dir_split": 0, 2024-03-18T16:21:35.834 INFO:teuthology.orchestra.run.smithi196.stdout: "dir_merge": 0, 2024-03-18T16:21:35.834 INFO:teuthology.orchestra.run.smithi196.stdout: "inodes": 31, 2024-03-18T16:21:35.834 INFO:teuthology.orchestra.run.smithi196.stdout: "inodes_top": 0, 2024-03-18T16:21:35.834 INFO:teuthology.orchestra.run.smithi196.stdout: "inodes_bottom": 31, 2024-03-18T16:21:35.834 INFO:teuthology.orchestra.run.smithi196.stdout: "inodes_pin_tail": 0, 2024-03-18T16:21:35.835 INFO:teuthology.orchestra.run.smithi196.stdout: "inodes_pinned": 31, 2024-03-18T16:21:35.835 INFO:teuthology.orchestra.run.smithi196.stdout: "inodes_expired": 0, 2024-03-18T16:21:35.835 INFO:teuthology.orchestra.run.smithi196.stdout: "inodes_with_caps": 0, 2024-03-18T16:21:35.835 INFO:teuthology.orchestra.run.smithi196.stdout: "caps": 0, 2024-03-18T16:21:35.835 INFO:teuthology.orchestra.run.smithi196.stdout: "subtrees": 1, 2024-03-18T16:21:35.835 INFO:teuthology.orchestra.run.smithi196.stdout: "traverse": 0, 2024-03-18T16:21:35.835 INFO:teuthology.orchestra.run.smithi196.stdout: "traverse_hit": 0, 2024-03-18T16:21:35.835 INFO:teuthology.orchestra.run.smithi196.stdout: "traverse_forward": 0, 2024-03-18T16:21:35.835 INFO:teuthology.orchestra.run.smithi196.stdout: "traverse_discover": 0, 2024-03-18T16:21:35.835 INFO:teuthology.orchestra.run.smithi196.stdout: "traverse_dir_fetch": 0, 2024-03-18T16:21:35.835 INFO:teuthology.orchestra.run.smithi196.stdout: "traverse_remote_ino": 0, 2024-03-18T16:21:35.835 INFO:teuthology.orchestra.run.smithi196.stdout: "traverse_lock": 0, 2024-03-18T16:21:35.835 INFO:teuthology.orchestra.run.smithi196.stdout: "load_cent": 0, 2024-03-18T16:21:35.835 INFO:teuthology.orchestra.run.smithi196.stdout: "q": 0, 2024-03-18T16:21:35.835 INFO:teuthology.orchestra.run.smithi196.stdout: "exported": 0, 2024-03-18T16:21:35.835 INFO:teuthology.orchestra.run.smithi196.stdout: "exported_inodes": 0, 2024-03-18T16:21:35.835 INFO:teuthology.orchestra.run.smithi196.stdout: "imported": 0, 2024-03-18T16:21:35.836 INFO:teuthology.orchestra.run.smithi196.stdout: "imported_inodes": 0, 2024-03-18T16:21:35.836 INFO:teuthology.orchestra.run.smithi196.stdout: "openino_dir_fetch": 0, 2024-03-18T16:21:35.836 INFO:teuthology.orchestra.run.smithi196.stdout: "openino_backtrace_fetch": 0, 2024-03-18T16:21:35.836 INFO:teuthology.orchestra.run.smithi196.stdout: "openino_peer_discover": 0, 2024-03-18T16:21:35.836 INFO:teuthology.orchestra.run.smithi196.stdout: "root_rfiles": 0, 2024-03-18T16:21:35.836 INFO:teuthology.orchestra.run.smithi196.stdout: "root_rbytes": 0, 2024-03-18T16:21:35.836 INFO:teuthology.orchestra.run.smithi196.stdout: "root_rsnaps": 0, 2024-03-18T16:21:35.836 INFO:teuthology.orchestra.run.smithi196.stdout: "scrub_backtrace_fetch": 0, 2024-03-18T16:21:35.836 INFO:teuthology.orchestra.run.smithi196.stdout: "scrub_set_tag": 0, 2024-03-18T16:21:35.836 INFO:teuthology.orchestra.run.smithi196.stdout: "scrub_backtrace_repaired": 0, 2024-03-18T16:21:35.836 INFO:teuthology.orchestra.run.smithi196.stdout: "scrub_inotable_repaired": 0, 2024-03-18T16:21:35.836 INFO:teuthology.orchestra.run.smithi196.stdout: "scrub_dir_inodes": 0, 2024-03-18T16:21:35.836 INFO:teuthology.orchestra.run.smithi196.stdout: "scrub_dir_base_inodes": 0, 2024-03-18T16:21:35.836 INFO:teuthology.orchestra.run.smithi196.stdout: "scrub_dirfrag_rstats": 0, 2024-03-18T16:21:35.836 INFO:teuthology.orchestra.run.smithi196.stdout: "scrub_file_inodes": 0, 2024-03-18T16:21:35.836 INFO:teuthology.orchestra.run.smithi196.stdout: "handle_inode_file_caps": 0, 2024-03-18T16:21:35.837 INFO:teuthology.orchestra.run.smithi196.stdout: "ceph_cap_op_revoke": 0, 2024-03-18T16:21:35.837 INFO:teuthology.orchestra.run.smithi196.stdout: "ceph_cap_op_grant": 0, 2024-03-18T16:21:35.837 INFO:teuthology.orchestra.run.smithi196.stdout: "ceph_cap_op_trunc": 0, 2024-03-18T16:21:35.837 INFO:teuthology.orchestra.run.smithi196.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-18T16:21:35.837 INFO:teuthology.orchestra.run.smithi196.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-18T16:21:35.837 INFO:teuthology.orchestra.run.smithi196.stdout: "handle_client_caps": 0, 2024-03-18T16:21:35.837 INFO:teuthology.orchestra.run.smithi196.stdout: "handle_client_caps_dirty": 0, 2024-03-18T16:21:35.837 INFO:teuthology.orchestra.run.smithi196.stdout: "handle_client_cap_release": 0, 2024-03-18T16:21:35.837 INFO:teuthology.orchestra.run.smithi196.stdout: "process_request_cap_release": 0 2024-03-18T16:21:35.837 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:35.837 INFO:teuthology.orchestra.run.smithi196.stdout: "mds_cache": { 2024-03-18T16:21:35.837 INFO:teuthology.orchestra.run.smithi196.stdout: "dir_update": 0, 2024-03-18T16:21:35.837 INFO:teuthology.orchestra.run.smithi196.stdout: "dir_update_receipt": 0, 2024-03-18T16:21:35.837 INFO:teuthology.orchestra.run.smithi196.stdout: "dir_try_discover": 0, 2024-03-18T16:21:35.837 INFO:teuthology.orchestra.run.smithi196.stdout: "dir_send_discover": 0, 2024-03-18T16:21:35.837 INFO:teuthology.orchestra.run.smithi196.stdout: "dir_handle_discover": 0, 2024-03-18T16:21:35.838 INFO:teuthology.orchestra.run.smithi196.stdout: "num_strays": 0, 2024-03-18T16:21:35.838 INFO:teuthology.orchestra.run.smithi196.stdout: "num_strays_delayed": 0, 2024-03-18T16:21:35.838 INFO:teuthology.orchestra.run.smithi196.stdout: "num_strays_enqueuing": 0, 2024-03-18T16:21:35.838 INFO:teuthology.orchestra.run.smithi196.stdout: "strays_created": 0, 2024-03-18T16:21:35.838 INFO:teuthology.orchestra.run.smithi196.stdout: "strays_enqueued": 0, 2024-03-18T16:21:35.838 INFO:teuthology.orchestra.run.smithi196.stdout: "strays_reintegrated": 0, 2024-03-18T16:21:35.838 INFO:teuthology.orchestra.run.smithi196.stdout: "strays_migrated": 0, 2024-03-18T16:21:35.838 INFO:teuthology.orchestra.run.smithi196.stdout: "num_recovering_processing": 0, 2024-03-18T16:21:35.838 INFO:teuthology.orchestra.run.smithi196.stdout: "num_recovering_enqueued": 0, 2024-03-18T16:21:35.838 INFO:teuthology.orchestra.run.smithi196.stdout: "num_recovering_prioritized": 0, 2024-03-18T16:21:35.838 INFO:teuthology.orchestra.run.smithi196.stdout: "recovery_started": 0, 2024-03-18T16:21:35.838 INFO:teuthology.orchestra.run.smithi196.stdout: "recovery_completed": 0, 2024-03-18T16:21:35.839 INFO:teuthology.orchestra.run.smithi196.stdout: "ireq_quiesce_path": 0, 2024-03-18T16:21:35.839 INFO:teuthology.orchestra.run.smithi196.stdout: "ireq_quiesce_inode": 0, 2024-03-18T16:21:35.839 INFO:teuthology.orchestra.run.smithi196.stdout: "ireq_enqueue_scrub": 0, 2024-03-18T16:21:35.839 INFO:teuthology.orchestra.run.smithi196.stdout: "ireq_exportdir": 0, 2024-03-18T16:21:35.839 INFO:teuthology.orchestra.run.smithi196.stdout: "ireq_flush": 0, 2024-03-18T16:21:35.839 INFO:teuthology.orchestra.run.smithi196.stdout: "ireq_fragmentdir": 0, 2024-03-18T16:21:35.839 INFO:teuthology.orchestra.run.smithi196.stdout: "ireq_fragstats": 0, 2024-03-18T16:21:35.839 INFO:teuthology.orchestra.run.smithi196.stdout: "ireq_inodestats": 0 2024-03-18T16:21:35.839 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:35.840 INFO:teuthology.orchestra.run.smithi196.stdout: "mds_log": { 2024-03-18T16:21:35.840 INFO:teuthology.orchestra.run.smithi196.stdout: "evlrg": 0, 2024-03-18T16:21:35.840 INFO:teuthology.orchestra.run.smithi196.stdout: "evadd": 0, 2024-03-18T16:21:35.840 INFO:teuthology.orchestra.run.smithi196.stdout: "evex": 0, 2024-03-18T16:21:35.840 INFO:teuthology.orchestra.run.smithi196.stdout: "evtrm": 0, 2024-03-18T16:21:35.840 INFO:teuthology.orchestra.run.smithi196.stdout: "ev": 931, 2024-03-18T16:21:35.840 INFO:teuthology.orchestra.run.smithi196.stdout: "evexg": 0, 2024-03-18T16:21:35.840 INFO:teuthology.orchestra.run.smithi196.stdout: "evexd": 0, 2024-03-18T16:21:35.840 INFO:teuthology.orchestra.run.smithi196.stdout: "segadd": 0, 2024-03-18T16:21:35.840 INFO:teuthology.orchestra.run.smithi196.stdout: "segex": 0, 2024-03-18T16:21:35.840 INFO:teuthology.orchestra.run.smithi196.stdout: "segtrm": 0, 2024-03-18T16:21:35.840 INFO:teuthology.orchestra.run.smithi196.stdout: "seg": 2, 2024-03-18T16:21:35.840 INFO:teuthology.orchestra.run.smithi196.stdout: "segmjr": 2, 2024-03-18T16:21:35.840 INFO:teuthology.orchestra.run.smithi196.stdout: "segexg": 0, 2024-03-18T16:21:35.840 INFO:teuthology.orchestra.run.smithi196.stdout: "segexd": 0, 2024-03-18T16:21:35.840 INFO:teuthology.orchestra.run.smithi196.stdout: "expos": 4194304, 2024-03-18T16:21:35.840 INFO:teuthology.orchestra.run.smithi196.stdout: "wrpos": 5343855, 2024-03-18T16:21:35.841 INFO:teuthology.orchestra.run.smithi196.stdout: "rdpos": 5342204, 2024-03-18T16:21:35.841 INFO:teuthology.orchestra.run.smithi196.stdout: "jlat": { 2024-03-18T16:21:35.841 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:35.841 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:35.841 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:35.841 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:35.841 INFO:teuthology.orchestra.run.smithi196.stdout: "replayed": 931 2024-03-18T16:21:35.841 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:35.841 INFO:teuthology.orchestra.run.smithi196.stdout: "mds_mem": { 2024-03-18T16:21:35.841 INFO:teuthology.orchestra.run.smithi196.stdout: "ino": 4, 2024-03-18T16:21:35.841 INFO:teuthology.orchestra.run.smithi196.stdout: "ino+": 49, 2024-03-18T16:21:35.841 INFO:teuthology.orchestra.run.smithi196.stdout: "ino-": 45, 2024-03-18T16:21:35.841 INFO:teuthology.orchestra.run.smithi196.stdout: "dir": 2, 2024-03-18T16:21:35.841 INFO:teuthology.orchestra.run.smithi196.stdout: "dir+": 19, 2024-03-18T16:21:35.841 INFO:teuthology.orchestra.run.smithi196.stdout: "dir-": 17, 2024-03-18T16:21:35.842 INFO:teuthology.orchestra.run.smithi196.stdout: "dn": 31, 2024-03-18T16:21:35.842 INFO:teuthology.orchestra.run.smithi196.stdout: "dn+": 79, 2024-03-18T16:21:35.842 INFO:teuthology.orchestra.run.smithi196.stdout: "dn-": 48, 2024-03-18T16:21:35.842 INFO:teuthology.orchestra.run.smithi196.stdout: "cap": 0, 2024-03-18T16:21:35.842 INFO:teuthology.orchestra.run.smithi196.stdout: "cap+": 0, 2024-03-18T16:21:35.842 INFO:teuthology.orchestra.run.smithi196.stdout: "cap-": 0, 2024-03-18T16:21:35.842 INFO:teuthology.orchestra.run.smithi196.stdout: "rss": 51528, 2024-03-18T16:21:35.842 INFO:teuthology.orchestra.run.smithi196.stdout: "heap": 215420 2024-03-18T16:21:35.842 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:35.842 INFO:teuthology.orchestra.run.smithi196.stdout: "mds_server": { 2024-03-18T16:21:35.842 INFO:teuthology.orchestra.run.smithi196.stdout: "dispatch_client_request": 0, 2024-03-18T16:21:35.842 INFO:teuthology.orchestra.run.smithi196.stdout: "dispatch_server_request": 0, 2024-03-18T16:21:35.843 INFO:teuthology.orchestra.run.smithi196.stdout: "handle_client_request": 0, 2024-03-18T16:21:35.843 INFO:teuthology.orchestra.run.smithi196.stdout: "handle_client_session": 0, 2024-03-18T16:21:35.843 INFO:teuthology.orchestra.run.smithi196.stdout: "handle_peer_request": 0, 2024-03-18T16:21:35.843 INFO:teuthology.orchestra.run.smithi196.stdout: "req_create_latency": { 2024-03-18T16:21:35.843 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:35.843 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:35.843 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:35.843 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:35.843 INFO:teuthology.orchestra.run.smithi196.stdout: "req_getattr_latency": { 2024-03-18T16:21:35.843 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:35.843 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:35.843 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:35.843 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:35.843 INFO:teuthology.orchestra.run.smithi196.stdout: "req_getfilelock_latency": { 2024-03-18T16:21:35.844 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:35.844 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:35.844 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:35.844 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:35.844 INFO:teuthology.orchestra.run.smithi196.stdout: "req_link_latency": { 2024-03-18T16:21:35.844 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:35.844 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:35.844 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:35.844 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:35.844 INFO:teuthology.orchestra.run.smithi196.stdout: "req_lookup_latency": { 2024-03-18T16:21:35.844 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:35.844 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:35.844 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:35.844 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:35.844 INFO:teuthology.orchestra.run.smithi196.stdout: "req_lookuphash_latency": { 2024-03-18T16:21:35.844 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:35.844 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:35.845 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:35.845 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:35.845 INFO:teuthology.orchestra.run.smithi196.stdout: "req_lookupino_latency": { 2024-03-18T16:21:35.845 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:35.845 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:35.845 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:35.845 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:35.845 INFO:teuthology.orchestra.run.smithi196.stdout: "req_lookupname_latency": { 2024-03-18T16:21:35.845 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:35.845 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:35.845 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:35.845 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:35.845 INFO:teuthology.orchestra.run.smithi196.stdout: "req_lookupparent_latency": { 2024-03-18T16:21:35.845 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:35.845 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:35.845 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:35.846 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:35.846 INFO:teuthology.orchestra.run.smithi196.stdout: "req_lookupsnap_latency": { 2024-03-18T16:21:35.846 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:35.846 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:35.846 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:35.846 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:35.846 INFO:teuthology.orchestra.run.smithi196.stdout: "req_lssnap_latency": { 2024-03-18T16:21:35.846 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:35.846 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:35.846 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:35.846 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:35.846 INFO:teuthology.orchestra.run.smithi196.stdout: "req_mkdir_latency": { 2024-03-18T16:21:35.846 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:35.847 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:35.847 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:35.847 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:35.847 INFO:teuthology.orchestra.run.smithi196.stdout: "req_mknod_latency": { 2024-03-18T16:21:35.847 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:35.847 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:35.847 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:35.847 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:35.847 INFO:teuthology.orchestra.run.smithi196.stdout: "req_mksnap_latency": { 2024-03-18T16:21:35.847 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:35.848 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:35.848 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:35.848 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:35.848 INFO:teuthology.orchestra.run.smithi196.stdout: "req_open_latency": { 2024-03-18T16:21:35.848 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:35.848 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:35.848 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:35.848 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:35.848 INFO:teuthology.orchestra.run.smithi196.stdout: "req_readdir_latency": { 2024-03-18T16:21:35.848 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:35.848 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:35.848 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:35.849 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:35.849 INFO:teuthology.orchestra.run.smithi196.stdout: "req_rename_latency": { 2024-03-18T16:21:35.849 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:35.849 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:35.849 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:35.849 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:35.849 INFO:teuthology.orchestra.run.smithi196.stdout: "req_renamesnap_latency": { 2024-03-18T16:21:35.849 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:35.849 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:35.849 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:35.849 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:35.849 INFO:teuthology.orchestra.run.smithi196.stdout: "req_snapdiff_latency": { 2024-03-18T16:21:35.849 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:35.849 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:35.849 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:35.850 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:35.850 INFO:teuthology.orchestra.run.smithi196.stdout: "req_rmdir_latency": { 2024-03-18T16:21:35.850 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:35.850 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:35.850 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:35.850 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:35.850 INFO:teuthology.orchestra.run.smithi196.stdout: "req_rmsnap_latency": { 2024-03-18T16:21:35.850 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:35.850 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:35.850 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:35.850 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:35.850 INFO:teuthology.orchestra.run.smithi196.stdout: "req_rmxattr_latency": { 2024-03-18T16:21:35.850 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:35.851 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:35.851 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:35.851 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:35.851 INFO:teuthology.orchestra.run.smithi196.stdout: "req_setattr_latency": { 2024-03-18T16:21:35.851 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:35.851 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:35.851 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:35.851 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:35.851 INFO:teuthology.orchestra.run.smithi196.stdout: "req_setdirlayout_latency": { 2024-03-18T16:21:35.851 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:35.851 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:35.851 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:35.851 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:35.851 INFO:teuthology.orchestra.run.smithi196.stdout: "req_setfilelock_latency": { 2024-03-18T16:21:35.852 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:35.852 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:35.852 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:35.852 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:35.852 INFO:teuthology.orchestra.run.smithi196.stdout: "req_setlayout_latency": { 2024-03-18T16:21:35.852 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:35.852 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:35.852 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:35.852 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:35.852 INFO:teuthology.orchestra.run.smithi196.stdout: "req_setxattr_latency": { 2024-03-18T16:21:35.852 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:35.852 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:35.852 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:35.852 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:35.853 INFO:teuthology.orchestra.run.smithi196.stdout: "req_symlink_latency": { 2024-03-18T16:21:35.853 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:35.853 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:35.853 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:35.853 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:35.853 INFO:teuthology.orchestra.run.smithi196.stdout: "req_unlink_latency": { 2024-03-18T16:21:35.853 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:35.853 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:35.853 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:35.853 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:35.853 INFO:teuthology.orchestra.run.smithi196.stdout: "cap_revoke_eviction": 0, 2024-03-18T16:21:35.853 INFO:teuthology.orchestra.run.smithi196.stdout: "cap_acquisition_throttle": 0, 2024-03-18T16:21:35.854 INFO:teuthology.orchestra.run.smithi196.stdout: "req_getvxattr_latency": { 2024-03-18T16:21:35.854 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:35.854 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:35.854 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:35.854 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-03-18T16:21:35.854 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:35.854 INFO:teuthology.orchestra.run.smithi196.stdout: "mds_sessions": { 2024-03-18T16:21:35.854 INFO:teuthology.orchestra.run.smithi196.stdout: "session_count": 1, 2024-03-18T16:21:35.854 INFO:teuthology.orchestra.run.smithi196.stdout: "session_add": 1, 2024-03-18T16:21:35.854 INFO:teuthology.orchestra.run.smithi196.stdout: "session_remove": 0, 2024-03-18T16:21:35.854 INFO:teuthology.orchestra.run.smithi196.stdout: "sessions_open": 1, 2024-03-18T16:21:35.855 INFO:teuthology.orchestra.run.smithi196.stdout: "sessions_stale": 0, 2024-03-18T16:21:35.855 INFO:teuthology.orchestra.run.smithi196.stdout: "total_load": 0, 2024-03-18T16:21:35.855 INFO:teuthology.orchestra.run.smithi196.stdout: "average_load": 0, 2024-03-18T16:21:35.855 INFO:teuthology.orchestra.run.smithi196.stdout: "avg_session_uptime": 2488, 2024-03-18T16:21:35.855 INFO:teuthology.orchestra.run.smithi196.stdout: "mdthresh_evicted": 0 2024-03-18T16:21:35.855 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:35.855 INFO:teuthology.orchestra.run.smithi196.stdout: "mempool": { 2024-03-18T16:21:35.855 INFO:teuthology.orchestra.run.smithi196.stdout: "bloom_filter_bytes": 0, 2024-03-18T16:21:35.855 INFO:teuthology.orchestra.run.smithi196.stdout: "bloom_filter_items": 0, 2024-03-18T16:21:35.855 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_alloc_bytes": 0, 2024-03-18T16:21:35.855 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_alloc_items": 0, 2024-03-18T16:21:35.855 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_cache_data_bytes": 0, 2024-03-18T16:21:35.855 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_cache_data_items": 0, 2024-03-18T16:21:35.855 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-18T16:21:35.855 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_cache_onode_items": 0, 2024-03-18T16:21:35.856 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-18T16:21:35.856 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_cache_meta_items": 0, 2024-03-18T16:21:35.856 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_cache_other_bytes": 0, 2024-03-18T16:21:35.856 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_cache_other_items": 0, 2024-03-18T16:21:35.856 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-18T16:21:35.856 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_cache_buffer_items": 0, 2024-03-18T16:21:35.856 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_extent_bytes": 0, 2024-03-18T16:21:35.856 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_extent_items": 0, 2024-03-18T16:21:35.856 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_blob_bytes": 0, 2024-03-18T16:21:35.856 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_blob_items": 0, 2024-03-18T16:21:35.856 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-18T16:21:35.856 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_shared_blob_items": 0, 2024-03-18T16:21:35.856 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-18T16:21:35.856 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_inline_bl_items": 0, 2024-03-18T16:21:35.856 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_fsck_bytes": 0, 2024-03-18T16:21:35.856 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_fsck_items": 0, 2024-03-18T16:21:35.856 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_txc_bytes": 0, 2024-03-18T16:21:35.857 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_txc_items": 0, 2024-03-18T16:21:35.857 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-18T16:21:35.857 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_writing_deferred_items": 0, 2024-03-18T16:21:35.857 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_writing_bytes": 0, 2024-03-18T16:21:35.857 INFO:teuthology.orchestra.run.smithi196.stdout: "bluestore_writing_items": 0, 2024-03-18T16:21:35.857 INFO:teuthology.orchestra.run.smithi196.stdout: "bluefs_bytes": 0, 2024-03-18T16:21:35.857 INFO:teuthology.orchestra.run.smithi196.stdout: "bluefs_items": 0, 2024-03-18T16:21:35.857 INFO:teuthology.orchestra.run.smithi196.stdout: "bluefs_file_reader_bytes": 0, 2024-03-18T16:21:35.857 INFO:teuthology.orchestra.run.smithi196.stdout: "bluefs_file_reader_items": 0, 2024-03-18T16:21:35.857 INFO:teuthology.orchestra.run.smithi196.stdout: "bluefs_file_writer_bytes": 0, 2024-03-18T16:21:35.857 INFO:teuthology.orchestra.run.smithi196.stdout: "bluefs_file_writer_items": 0, 2024-03-18T16:21:35.857 INFO:teuthology.orchestra.run.smithi196.stdout: "buffer_anon_bytes": 190162, 2024-03-18T16:21:35.857 INFO:teuthology.orchestra.run.smithi196.stdout: "buffer_anon_items": 53, 2024-03-18T16:21:35.857 INFO:teuthology.orchestra.run.smithi196.stdout: "buffer_meta_bytes": 0, 2024-03-18T16:21:35.857 INFO:teuthology.orchestra.run.smithi196.stdout: "buffer_meta_items": 0, 2024-03-18T16:21:35.857 INFO:teuthology.orchestra.run.smithi196.stdout: "osd_bytes": 0, 2024-03-18T16:21:35.857 INFO:teuthology.orchestra.run.smithi196.stdout: "osd_items": 0, 2024-03-18T16:21:35.858 INFO:teuthology.orchestra.run.smithi196.stdout: "osd_mapbl_bytes": 0, 2024-03-18T16:21:35.858 INFO:teuthology.orchestra.run.smithi196.stdout: "osd_mapbl_items": 0, 2024-03-18T16:21:35.858 INFO:teuthology.orchestra.run.smithi196.stdout: "osd_pglog_bytes": 0, 2024-03-18T16:21:35.858 INFO:teuthology.orchestra.run.smithi196.stdout: "osd_pglog_items": 0, 2024-03-18T16:21:35.858 INFO:teuthology.orchestra.run.smithi196.stdout: "osdmap_bytes": 8888, 2024-03-18T16:21:35.858 INFO:teuthology.orchestra.run.smithi196.stdout: "osdmap_items": 170, 2024-03-18T16:21:35.858 INFO:teuthology.orchestra.run.smithi196.stdout: "osdmap_mapping_bytes": 0, 2024-03-18T16:21:35.858 INFO:teuthology.orchestra.run.smithi196.stdout: "osdmap_mapping_items": 0, 2024-03-18T16:21:35.858 INFO:teuthology.orchestra.run.smithi196.stdout: "pgmap_bytes": 0, 2024-03-18T16:21:35.858 INFO:teuthology.orchestra.run.smithi196.stdout: "pgmap_items": 0, 2024-03-18T16:21:35.858 INFO:teuthology.orchestra.run.smithi196.stdout: "mds_co_bytes": 33808, 2024-03-18T16:21:35.858 INFO:teuthology.orchestra.run.smithi196.stdout: "mds_co_items": 152, 2024-03-18T16:21:35.858 INFO:teuthology.orchestra.run.smithi196.stdout: "unittest_1_bytes": 0, 2024-03-18T16:21:35.858 INFO:teuthology.orchestra.run.smithi196.stdout: "unittest_1_items": 0, 2024-03-18T16:21:35.858 INFO:teuthology.orchestra.run.smithi196.stdout: "unittest_2_bytes": 0, 2024-03-18T16:21:35.859 INFO:teuthology.orchestra.run.smithi196.stdout: "unittest_2_items": 0 2024-03-18T16:21:35.859 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:35.859 INFO:teuthology.orchestra.run.smithi196.stdout: "objecter": { 2024-03-18T16:21:35.859 INFO:teuthology.orchestra.run.smithi196.stdout: "op_active": 0, 2024-03-18T16:21:35.859 INFO:teuthology.orchestra.run.smithi196.stdout: "op_laggy": 0, 2024-03-18T16:21:35.859 INFO:teuthology.orchestra.run.smithi196.stdout: "op_send": 2431, 2024-03-18T16:21:35.859 INFO:teuthology.orchestra.run.smithi196.stdout: "op_send_bytes": 21, 2024-03-18T16:21:35.859 INFO:teuthology.orchestra.run.smithi196.stdout: "op_resend": 3, 2024-03-18T16:21:35.859 INFO:teuthology.orchestra.run.smithi196.stdout: "op_reply": 2431, 2024-03-18T16:21:35.859 INFO:teuthology.orchestra.run.smithi196.stdout: "op_latency": { 2024-03-18T16:21:35.859 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 2431, 2024-03-18T16:21:35.859 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 29.399397741, 2024-03-18T16:21:35.859 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.012093540 2024-03-18T16:21:35.859 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:35.859 INFO:teuthology.orchestra.run.smithi196.stdout: "op_inflight": 0, 2024-03-18T16:21:35.859 INFO:teuthology.orchestra.run.smithi196.stdout: "oplen_avg": { 2024-03-18T16:21:35.860 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 2431, 2024-03-18T16:21:35.860 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 2432 2024-03-18T16:21:35.860 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:35.860 INFO:teuthology.orchestra.run.smithi196.stdout: "op": 2431, 2024-03-18T16:21:35.860 INFO:teuthology.orchestra.run.smithi196.stdout: "op_r": 2431, 2024-03-18T16:21:35.860 INFO:teuthology.orchestra.run.smithi196.stdout: "op_w": 0, 2024-03-18T16:21:35.860 INFO:teuthology.orchestra.run.smithi196.stdout: "op_rmw": 0, 2024-03-18T16:21:35.860 INFO:teuthology.orchestra.run.smithi196.stdout: "op_pg": 0, 2024-03-18T16:21:35.860 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_stat": 1492, 2024-03-18T16:21:35.860 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_create": 0, 2024-03-18T16:21:35.860 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_read": 937, 2024-03-18T16:21:35.860 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_write": 0, 2024-03-18T16:21:35.860 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_writefull": 0, 2024-03-18T16:21:35.860 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_writesame": 0, 2024-03-18T16:21:35.860 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_append": 0, 2024-03-18T16:21:35.860 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_zero": 0, 2024-03-18T16:21:35.861 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_truncate": 0, 2024-03-18T16:21:35.861 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_delete": 0, 2024-03-18T16:21:35.861 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_mapext": 0, 2024-03-18T16:21:35.861 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_sparse_read": 0, 2024-03-18T16:21:35.861 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_clonerange": 0, 2024-03-18T16:21:35.861 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_getxattr": 1, 2024-03-18T16:21:35.861 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_setxattr": 0, 2024-03-18T16:21:35.861 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_cmpxattr": 0, 2024-03-18T16:21:35.861 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_rmxattr": 0, 2024-03-18T16:21:35.861 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_resetxattrs": 0, 2024-03-18T16:21:35.861 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_call": 0, 2024-03-18T16:21:35.861 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_watch": 0, 2024-03-18T16:21:35.861 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_notify": 0, 2024-03-18T16:21:35.861 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_src_cmpxattr": 0, 2024-03-18T16:21:35.862 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_pgls": 0, 2024-03-18T16:21:35.862 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_pgls_filter": 0, 2024-03-18T16:21:35.862 INFO:teuthology.orchestra.run.smithi196.stdout: "osdop_other": 0, 2024-03-18T16:21:35.862 INFO:teuthology.orchestra.run.smithi196.stdout: "linger_active": 0, 2024-03-18T16:21:35.862 INFO:teuthology.orchestra.run.smithi196.stdout: "linger_send": 0, 2024-03-18T16:21:35.862 INFO:teuthology.orchestra.run.smithi196.stdout: "linger_resend": 0, 2024-03-18T16:21:35.862 INFO:teuthology.orchestra.run.smithi196.stdout: "linger_ping": 0, 2024-03-18T16:21:35.862 INFO:teuthology.orchestra.run.smithi196.stdout: "poolop_active": 0, 2024-03-18T16:21:35.862 INFO:teuthology.orchestra.run.smithi196.stdout: "poolop_send": 0, 2024-03-18T16:21:35.862 INFO:teuthology.orchestra.run.smithi196.stdout: "poolop_resend": 0, 2024-03-18T16:21:35.862 INFO:teuthology.orchestra.run.smithi196.stdout: "poolstat_active": 0, 2024-03-18T16:21:35.862 INFO:teuthology.orchestra.run.smithi196.stdout: "poolstat_send": 0, 2024-03-18T16:21:35.862 INFO:teuthology.orchestra.run.smithi196.stdout: "poolstat_resend": 0, 2024-03-18T16:21:35.862 INFO:teuthology.orchestra.run.smithi196.stdout: "statfs_active": 0, 2024-03-18T16:21:35.863 INFO:teuthology.orchestra.run.smithi196.stdout: "statfs_send": 0, 2024-03-18T16:21:35.863 INFO:teuthology.orchestra.run.smithi196.stdout: "statfs_resend": 0, 2024-03-18T16:21:35.863 INFO:teuthology.orchestra.run.smithi196.stdout: "command_active": 0, 2024-03-18T16:21:35.863 INFO:teuthology.orchestra.run.smithi196.stdout: "command_send": 0, 2024-03-18T16:21:35.863 INFO:teuthology.orchestra.run.smithi196.stdout: "command_resend": 0, 2024-03-18T16:21:35.863 INFO:teuthology.orchestra.run.smithi196.stdout: "map_epoch": 93, 2024-03-18T16:21:35.863 INFO:teuthology.orchestra.run.smithi196.stdout: "map_full": 0, 2024-03-18T16:21:35.863 INFO:teuthology.orchestra.run.smithi196.stdout: "map_inc": 4, 2024-03-18T16:21:35.863 INFO:teuthology.orchestra.run.smithi196.stdout: "osd_sessions": 5, 2024-03-18T16:21:35.863 INFO:teuthology.orchestra.run.smithi196.stdout: "osd_session_open": 5, 2024-03-18T16:21:35.863 INFO:teuthology.orchestra.run.smithi196.stdout: "osd_session_close": 0, 2024-03-18T16:21:35.863 INFO:teuthology.orchestra.run.smithi196.stdout: "osd_laggy": 0, 2024-03-18T16:21:35.863 INFO:teuthology.orchestra.run.smithi196.stdout: "omap_wr": 0, 2024-03-18T16:21:35.863 INFO:teuthology.orchestra.run.smithi196.stdout: "omap_rd": 2, 2024-03-18T16:21:35.863 INFO:teuthology.orchestra.run.smithi196.stdout: "omap_del": 0 2024-03-18T16:21:35.864 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:35.864 INFO:teuthology.orchestra.run.smithi196.stdout: "oft": { 2024-03-18T16:21:35.864 INFO:teuthology.orchestra.run.smithi196.stdout: "omap_total_objs": 0, 2024-03-18T16:21:35.864 INFO:teuthology.orchestra.run.smithi196.stdout: "omap_total_kv_pairs": 0, 2024-03-18T16:21:35.864 INFO:teuthology.orchestra.run.smithi196.stdout: "omap_total_updates": 0, 2024-03-18T16:21:35.864 INFO:teuthology.orchestra.run.smithi196.stdout: "omap_total_removes": 0 2024-03-18T16:21:35.864 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:35.864 INFO:teuthology.orchestra.run.smithi196.stdout: "osd-slow-ops": { 2024-03-18T16:21:35.864 INFO:teuthology.orchestra.run.smithi196.stdout: "slow_ops_count": 0 2024-03-18T16:21:35.864 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:35.864 INFO:teuthology.orchestra.run.smithi196.stdout: "purge_queue": { 2024-03-18T16:21:35.864 INFO:teuthology.orchestra.run.smithi196.stdout: "pq_executing_ops": 0, 2024-03-18T16:21:35.864 INFO:teuthology.orchestra.run.smithi196.stdout: "pq_executing_ops_high_water": 0, 2024-03-18T16:21:35.864 INFO:teuthology.orchestra.run.smithi196.stdout: "pq_executing": 0, 2024-03-18T16:21:35.864 INFO:teuthology.orchestra.run.smithi196.stdout: "pq_executing_high_water": 0, 2024-03-18T16:21:35.865 INFO:teuthology.orchestra.run.smithi196.stdout: "pq_executed_ops": 0, 2024-03-18T16:21:35.865 INFO:teuthology.orchestra.run.smithi196.stdout: "pq_executed": 0, 2024-03-18T16:21:35.865 INFO:teuthology.orchestra.run.smithi196.stdout: "pq_item_in_journal": 0 2024-03-18T16:21:35.865 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:35.865 INFO:teuthology.orchestra.run.smithi196.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-18T16:21:35.865 INFO:teuthology.orchestra.run.smithi196.stdout: "val": 0, 2024-03-18T16:21:35.865 INFO:teuthology.orchestra.run.smithi196.stdout: "max": 104857600, 2024-03-18T16:21:35.865 INFO:teuthology.orchestra.run.smithi196.stdout: "get_started": 0, 2024-03-18T16:21:35.865 INFO:teuthology.orchestra.run.smithi196.stdout: "get": 2717, 2024-03-18T16:21:35.865 INFO:teuthology.orchestra.run.smithi196.stdout: "get_sum": 2241443, 2024-03-18T16:21:35.865 INFO:teuthology.orchestra.run.smithi196.stdout: "get_or_fail_fail": 0, 2024-03-18T16:21:35.865 INFO:teuthology.orchestra.run.smithi196.stdout: "get_or_fail_success": 2717, 2024-03-18T16:21:35.865 INFO:teuthology.orchestra.run.smithi196.stdout: "take": 0, 2024-03-18T16:21:35.865 INFO:teuthology.orchestra.run.smithi196.stdout: "take_sum": 0, 2024-03-18T16:21:35.865 INFO:teuthology.orchestra.run.smithi196.stdout: "put": 2717, 2024-03-18T16:21:35.866 INFO:teuthology.orchestra.run.smithi196.stdout: "put_sum": 2241443, 2024-03-18T16:21:35.866 INFO:teuthology.orchestra.run.smithi196.stdout: "wait": { 2024-03-18T16:21:35.866 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:35.866 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:35.866 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:35.866 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-03-18T16:21:35.866 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:35.866 INFO:teuthology.orchestra.run.smithi196.stdout: "throttle-objecter_bytes": { 2024-03-18T16:21:35.866 INFO:teuthology.orchestra.run.smithi196.stdout: "val": 0, 2024-03-18T16:21:35.866 INFO:teuthology.orchestra.run.smithi196.stdout: "max": 104857600, 2024-03-18T16:21:35.866 INFO:teuthology.orchestra.run.smithi196.stdout: "get_started": 0, 2024-03-18T16:21:35.866 INFO:teuthology.orchestra.run.smithi196.stdout: "get": 0, 2024-03-18T16:21:35.866 INFO:teuthology.orchestra.run.smithi196.stdout: "get_sum": 0, 2024-03-18T16:21:35.866 INFO:teuthology.orchestra.run.smithi196.stdout: "get_or_fail_fail": 0, 2024-03-18T16:21:35.866 INFO:teuthology.orchestra.run.smithi196.stdout: "get_or_fail_success": 0, 2024-03-18T16:21:35.866 INFO:teuthology.orchestra.run.smithi196.stdout: "take": 2431, 2024-03-18T16:21:35.867 INFO:teuthology.orchestra.run.smithi196.stdout: "take_sum": 1149556, 2024-03-18T16:21:35.867 INFO:teuthology.orchestra.run.smithi196.stdout: "put": 2431, 2024-03-18T16:21:35.867 INFO:teuthology.orchestra.run.smithi196.stdout: "put_sum": 1149556, 2024-03-18T16:21:35.867 INFO:teuthology.orchestra.run.smithi196.stdout: "wait": { 2024-03-18T16:21:35.867 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:35.867 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:35.867 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:35.867 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-03-18T16:21:35.867 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:35.867 INFO:teuthology.orchestra.run.smithi196.stdout: "throttle-objecter_ops": { 2024-03-18T16:21:35.867 INFO:teuthology.orchestra.run.smithi196.stdout: "val": 0, 2024-03-18T16:21:35.867 INFO:teuthology.orchestra.run.smithi196.stdout: "max": 1024, 2024-03-18T16:21:35.867 INFO:teuthology.orchestra.run.smithi196.stdout: "get_started": 0, 2024-03-18T16:21:35.867 INFO:teuthology.orchestra.run.smithi196.stdout: "get": 0, 2024-03-18T16:21:35.867 INFO:teuthology.orchestra.run.smithi196.stdout: "get_sum": 0, 2024-03-18T16:21:35.867 INFO:teuthology.orchestra.run.smithi196.stdout: "get_or_fail_fail": 0, 2024-03-18T16:21:35.868 INFO:teuthology.orchestra.run.smithi196.stdout: "get_or_fail_success": 0, 2024-03-18T16:21:35.868 INFO:teuthology.orchestra.run.smithi196.stdout: "take": 2431, 2024-03-18T16:21:35.868 INFO:teuthology.orchestra.run.smithi196.stdout: "take_sum": 2431, 2024-03-18T16:21:35.868 INFO:teuthology.orchestra.run.smithi196.stdout: "put": 2431, 2024-03-18T16:21:35.868 INFO:teuthology.orchestra.run.smithi196.stdout: "put_sum": 2431, 2024-03-18T16:21:35.868 INFO:teuthology.orchestra.run.smithi196.stdout: "wait": { 2024-03-18T16:21:35.868 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:35.868 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:35.868 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:35.868 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-03-18T16:21:35.868 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:35.868 INFO:teuthology.orchestra.run.smithi196.stdout: "throttle-write_buf_throttle": { 2024-03-18T16:21:35.868 INFO:teuthology.orchestra.run.smithi196.stdout: "val": 0, 2024-03-18T16:21:35.868 INFO:teuthology.orchestra.run.smithi196.stdout: "max": 3758096384, 2024-03-18T16:21:35.868 INFO:teuthology.orchestra.run.smithi196.stdout: "get_started": 0, 2024-03-18T16:21:35.869 INFO:teuthology.orchestra.run.smithi196.stdout: "get": 0, 2024-03-18T16:21:35.869 INFO:teuthology.orchestra.run.smithi196.stdout: "get_sum": 0, 2024-03-18T16:21:35.869 INFO:teuthology.orchestra.run.smithi196.stdout: "get_or_fail_fail": 0, 2024-03-18T16:21:35.869 INFO:teuthology.orchestra.run.smithi196.stdout: "get_or_fail_success": 0, 2024-03-18T16:21:35.869 INFO:teuthology.orchestra.run.smithi196.stdout: "take": 0, 2024-03-18T16:21:35.869 INFO:teuthology.orchestra.run.smithi196.stdout: "take_sum": 0, 2024-03-18T16:21:35.869 INFO:teuthology.orchestra.run.smithi196.stdout: "put": 0, 2024-03-18T16:21:35.869 INFO:teuthology.orchestra.run.smithi196.stdout: "put_sum": 0, 2024-03-18T16:21:35.869 INFO:teuthology.orchestra.run.smithi196.stdout: "wait": { 2024-03-18T16:21:35.869 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:35.869 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:35.869 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:35.869 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-03-18T16:21:35.869 INFO:teuthology.orchestra.run.smithi196.stdout: }, 2024-03-18T16:21:35.870 INFO:teuthology.orchestra.run.smithi196.stdout: "throttle-write_buf_throttle-0x5594ffd706e0": { 2024-03-18T16:21:35.870 INFO:teuthology.orchestra.run.smithi196.stdout: "val": 0, 2024-03-18T16:21:35.870 INFO:teuthology.orchestra.run.smithi196.stdout: "max": 3758096384, 2024-03-18T16:21:35.870 INFO:teuthology.orchestra.run.smithi196.stdout: "get_started": 0, 2024-03-18T16:21:35.870 INFO:teuthology.orchestra.run.smithi196.stdout: "get": 0, 2024-03-18T16:21:35.870 INFO:teuthology.orchestra.run.smithi196.stdout: "get_sum": 0, 2024-03-18T16:21:35.870 INFO:teuthology.orchestra.run.smithi196.stdout: "get_or_fail_fail": 0, 2024-03-18T16:21:35.870 INFO:teuthology.orchestra.run.smithi196.stdout: "get_or_fail_success": 0, 2024-03-18T16:21:35.870 INFO:teuthology.orchestra.run.smithi196.stdout: "take": 0, 2024-03-18T16:21:35.870 INFO:teuthology.orchestra.run.smithi196.stdout: "take_sum": 0, 2024-03-18T16:21:35.870 INFO:teuthology.orchestra.run.smithi196.stdout: "put": 0, 2024-03-18T16:21:35.870 INFO:teuthology.orchestra.run.smithi196.stdout: "put_sum": 0, 2024-03-18T16:21:35.870 INFO:teuthology.orchestra.run.smithi196.stdout: "wait": { 2024-03-18T16:21:35.870 INFO:teuthology.orchestra.run.smithi196.stdout: "avgcount": 0, 2024-03-18T16:21:35.870 INFO:teuthology.orchestra.run.smithi196.stdout: "sum": 0.000000000, 2024-03-18T16:21:35.870 INFO:teuthology.orchestra.run.smithi196.stdout: "avgtime": 0.000000000 2024-03-18T16:21:35.871 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-03-18T16:21:35.871 INFO:teuthology.orchestra.run.smithi196.stdout: } 2024-03-18T16:21:35.871 INFO:teuthology.orchestra.run.smithi196.stdout:} 2024-03-18T16:21:36.647 INFO:tasks.check_counter:Daemon mds.l mds.exported=0 2024-03-18T16:21:36.647 INFO:tasks.check_counter:Daemon mds.l mds.imported=0 2024-03-18T16:21:36.647 INFO:tasks.check_counter:Daemon mds.l mds.dir_split=0 2024-03-18T16:21:36.648 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-03-18T16:21:36.658 INFO:tasks.kclient:Unmounting kernel clients... 2024-03-18T16:21:36.658 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-03-18T16:21:36.658 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T16:21:36.732 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2024-03-18T16:21:36.732 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-18T16:21:36.786 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T16:21:37.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:36.999+0000 7faa8d800640 1 -- 172.21.15.39:0/4083434474 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7faa881011c0 msgr2=0x7faa88101640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:37.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:36.999+0000 7faa8d800640 1 --2- 172.21.15.39:0/4083434474 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7faa881011c0 0x7faa88101640 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7faa80009ca0 tx=0x7faa8002c0f0 comp rx=0 tx=0).stop 2024-03-18T16:21:37.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:36.999+0000 7faa8d800640 1 -- 172.21.15.39:0/4083434474 shutdown_connections 2024-03-18T16:21:37.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:36.999+0000 7faa8d800640 1 --2- 172.21.15.39:0/4083434474 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7faa881011c0 0x7faa88101640 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:37.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:36.999+0000 7faa8d800640 1 --2- 172.21.15.39:0/4083434474 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faa88100820 0x7faa88100c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:37.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:36.999+0000 7faa8d800640 1 --2- 172.21.15.39:0/4083434474 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7faa880ffec0 0x7faa881002e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:37.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:36.999+0000 7faa8d800640 1 -- 172.21.15.39:0/4083434474 >> 172.21.15.39:0/4083434474 conn(0x7faa880fba60 msgr2=0x7faa880fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:37.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:36.999+0000 7faa8d800640 1 -- 172.21.15.39:0/4083434474 shutdown_connections 2024-03-18T16:21:37.002 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:36.999+0000 7faa8d800640 1 -- 172.21.15.39:0/4083434474 wait complete. 2024-03-18T16:21:37.003 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.001+0000 7faa8d800640 1 Processor -- start 2024-03-18T16:21:37.004 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.001+0000 7faa8d800640 1 -- start start 2024-03-18T16:21:37.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.002+0000 7faa8d800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faa880ffec0 0x7faa88199a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:37.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.002+0000 7faa8d800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7faa88100820 0x7faa88199f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:37.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.002+0000 7faa8d800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7faa881011c0 0x7faa881a0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:37.006 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.002+0000 7faa8d800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7faa8810f440 con 0x7faa880ffec0 2024-03-18T16:21:37.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.002+0000 7faa8d800640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7faa8810f2c0 con 0x7faa881011c0 2024-03-18T16:21:37.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.002+0000 7faa8d800640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7faa8810f5c0 con 0x7faa88100820 2024-03-18T16:21:37.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.002+0000 7faa87e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faa880ffec0 0x7faa88199a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:37.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.002+0000 7faa7fe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7faa88100820 0x7faa88199f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:37.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.002+0000 7faa87e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faa880ffec0 0x7faa88199a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:46416/0 (socket says 172.21.15.39:46416) 2024-03-18T16:21:37.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.002+0000 7faa87e00640 1 -- 172.21.15.39:0/1298060338 learned_addr learned my addr 172.21.15.39:0/1298060338 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:21:37.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.002+0000 7faa8ce00640 1 --2- 172.21.15.39:0/1298060338 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7faa881011c0 0x7faa881a0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:37.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.003+0000 7faa7fe00640 1 -- 172.21.15.39:0/1298060338 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7faa881011c0 msgr2=0x7faa881a0fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:37.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.003+0000 7faa7fe00640 1 --2- 172.21.15.39:0/1298060338 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7faa881011c0 0x7faa881a0fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:37.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.003+0000 7faa7fe00640 1 -- 172.21.15.39:0/1298060338 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faa880ffec0 msgr2=0x7faa88199a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:37.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.003+0000 7faa7fe00640 1 --2- 172.21.15.39:0/1298060338 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faa880ffec0 0x7faa88199a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:37.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.003+0000 7faa7fe00640 1 -- 172.21.15.39:0/1298060338 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa881a1640 con 0x7faa88100820 2024-03-18T16:21:37.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.003+0000 7faa8ce00640 1 --2- 172.21.15.39:0/1298060338 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7faa881011c0 0x7faa881a0fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:21:37.007 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.003+0000 7faa7fe00640 1 --2- 172.21.15.39:0/1298060338 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7faa88100820 0x7faa88199f60 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7faa7000e9f0 tx=0x7faa7000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:37.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.003+0000 7faa85600640 1 -- 172.21.15.39:0/1298060338 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa7000ccc0 con 0x7faa88100820 2024-03-18T16:21:37.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.003+0000 7faa85600640 1 -- 172.21.15.39:0/1298060338 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa70010910 con 0x7faa88100820 2024-03-18T16:21:37.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.003+0000 7faa8d800640 1 -- 172.21.15.39:0/1298060338 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa881a1930 con 0x7faa88100820 2024-03-18T16:21:37.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.004+0000 7faa85600640 1 -- 172.21.15.39:0/1298060338 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa70018960 con 0x7faa88100820 2024-03-18T16:21:37.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.004+0000 7faa8d800640 1 -- 172.21.15.39:0/1298060338 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa881a1ed0 con 0x7faa88100820 2024-03-18T16:21:37.008 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.005+0000 7faa8d800640 1 -- 172.21.15.39:0/1298060338 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa8810ac10 con 0x7faa88100820 2024-03-18T16:21:37.009 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.007+0000 7faa85600640 1 -- 172.21.15.39:0/1298060338 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7faa70010ac0 con 0x7faa88100820 2024-03-18T16:21:37.011 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.009+0000 7faa85600640 1 --2- 172.21.15.39:0/1298060338 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7faa6c0a2880 0x7faa6c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:37.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.009+0000 7faa85600640 1 -- 172.21.15.39:0/1298060338 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7faa700cf160 con 0x7faa88100820 2024-03-18T16:21:37.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.009+0000 7faa87e00640 1 --2- 172.21.15.39:0/1298060338 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7faa6c0a2880 0x7faa6c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:37.012 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.010+0000 7faa87e00640 1 --2- 172.21.15.39:0/1298060338 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7faa6c0a2880 0x7faa6c0a4d40 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7faa780095f0 tx=0x7faa78005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:37.015 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.012+0000 7faa85600640 1 -- 172.21.15.39:0/1298060338 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7faa7009b7c0 con 0x7faa88100820 2024-03-18T16:21:37.224 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.222+0000 7faa8d800640 1 -- 172.21.15.39:0/1298060338 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7faa880623e0 con 0x7faa88100820 2024-03-18T16:21:37.226 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.223+0000 7faa85600640 1 -- 172.21.15.39:0/1298060338 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24762 (secure 0 0 0) 0x7faa7009f860 con 0x7faa88100820 2024-03-18T16:21:37.227 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:21:37.227 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":93,"fsid":"1892d8fc-e540-11ee-95c9-87774f69a715","created":"2024-03-18T15:57:36.507008+0000","modified":"2024-03-18T16:08:47.874967+0000","last_up_change":"2024-03-18T16:05:45.250629+0000","last_in_change":"2024-03-18T16:05:24.841337+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T16:01:30.048247+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T16:06:42.191791+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T16:06:43.670325+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T16:06:47.523388+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"771e0558-be62-4436-94b0-6364751911c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6803","nonce":4090743635}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6805","nonce":4090743635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6809","nonce":4090743635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6807","nonce":4090743635}]},"public_addr":"172.21.15.39:6803/4090743635","cluster_addr":"172.21.15.39:6805/4090743635","heartbeat_back_addr":"172.21.15.39:6809/4090743635","heartbeat_front_addr":"172.21.15.39:6807/4090743635","state":["exists","up"]},{"osd":1,"uuid":"0430eee1-cf34-4277-b255-c7d9f07ad19a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6801","nonce":2800329805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6803","nonce":2800329805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6807","nonce":2800329805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6805","nonce":2800329805}]},"public_addr":"172.21.15.194:6801/2800329805","cluster_addr":"172.21.15.194:6803/2800329805","heartbeat_back_addr":"172.21.15.194:6807/2800329805","heartbeat_front_addr":"172.21.15.194:6805/2800329805","state":["exists","up"]},{"osd":2,"uuid":"3a718da4-2511-4ec1-b1a8-eafd6cce0df0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6801","nonce":3935874719}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6803","nonce":3935874719}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6807","nonce":3935874719}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6805","nonce":3935874719}]},"public_addr":"172.21.15.196:6801/3935874719","cluster_addr":"172.21.15.196:6803/3935874719","heartbeat_back_addr":"172.21.15.196:6807/3935874719","heartbeat_front_addr":"172.21.15.196:6805/3935874719","state":["exists","up"]},{"osd":3,"uuid":"6f9c37f9-0ab9-4038-8919-afd4b39c4165","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6811","nonce":893345939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6813","nonce":893345939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6817","nonce":893345939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6815","nonce":893345939}]},"public_addr":"172.21.15.39:6811/893345939","cluster_addr":"172.21.15.39:6813/893345939","heartbeat_back_addr":"172.21.15.39:6817/893345939","heartbeat_front_addr":"172.21.15.39:6815/893345939","state":["exists","up"]},{"osd":4,"uuid":"e795bb0e-4791-4537-acda-1a321e19bc9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6809","nonce":800423088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6811","nonce":800423088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6815","nonce":800423088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6813","nonce":800423088}]},"public_addr":"172.21.15.194:6809/800423088","cluster_addr":"172.21.15.194:6811/800423088","heartbeat_back_addr":"172.21.15.194:6815/800423088","heartbeat_front_addr":"172.21.15.194:6813/800423088","state":["exists","up"]},{"osd":5,"uuid":"026e175e-6acb-4b4c-b235-8ccee8ae1149","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6809","nonce":2746224525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6811","nonce":2746224525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6815","nonce":2746224525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6813","nonce":2746224525}]},"public_addr":"172.21.15.196:6809/2746224525","cluster_addr":"172.21.15.196:6811/2746224525","heartbeat_back_addr":"172.21.15.196:6815/2746224525","heartbeat_front_addr":"172.21.15.196:6813/2746224525","state":["exists","up"]},{"osd":6,"uuid":"c1b183ff-383f-4c5c-9075-f76bab1c0bb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6819","nonce":1213371331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6821","nonce":1213371331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6825","nonce":1213371331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6823","nonce":1213371331}]},"public_addr":"172.21.15.39:6819/1213371331","cluster_addr":"172.21.15.39:6821/1213371331","heartbeat_back_addr":"172.21.15.39:6825/1213371331","heartbeat_front_addr":"172.21.15.39:6823/1213371331","state":["exists","up"]},{"osd":7,"uuid":"612f8697-957d-4ef4-a87f-241b36f9d13d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6817","nonce":3299162444}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6819","nonce":3299162444}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6823","nonce":3299162444}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6821","nonce":3299162444}]},"public_addr":"172.21.15.194:6817/3299162444","cluster_addr":"172.21.15.194:6819/3299162444","heartbeat_back_addr":"172.21.15.194:6823/3299162444","heartbeat_front_addr":"172.21.15.194:6821/3299162444","state":["exists","up"]},{"osd":8,"uuid":"836a9107-4d4f-4594-b8dc-0301e64f59d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6817","nonce":2456143145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6819","nonce":2456143145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6823","nonce":2456143145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6821","nonce":2456143145}]},"public_addr":"172.21.15.196:6817/2456143145","cluster_addr":"172.21.15.196:6819/2456143145","heartbeat_back_addr":"172.21.15.196:6823/2456143145","heartbeat_front_addr":"172.21.15.196:6821/2456143145","state":["exists","up"]},{"osd":9,"uuid":"4f30b960-12e3-4b16-bd28-fcc988139b58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6827","nonce":1848634433}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6829","nonce":1848634433}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6833","nonce":1848634433}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6831","nonce":1848634433}]},"public_addr":"172.21.15.39:6827/1848634433","cluster_addr":"172.21.15.39:6829/1848634433","heartbeat_back_addr":"172.21.15.39:6833/1848634433","heartbeat_front_addr":"172.21.15.39:6831/1848634433","state":["exists","up"]},{"osd":10,"uuid":"620f90e8-23e6-4bc0-8269-7219b9e1aebd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6825","nonce":3725064809}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6827","nonce":3725064809}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6831","nonce":3725064809}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6829","nonce":3725064809}]},"public_addr":"172.21.15.194:6825/3725064809","cluster_addr":"172.21.15.194:6827/3725064809","heartbeat_back_addr":"172.21.15.194:6831/3725064809","heartbeat_front_addr":"172.21.15.194:6829/3725064809","state":["exists","up"]},{"osd":11,"uuid":"867a2535-8235-4ceb-a698-eb86869d6d20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6825","nonce":2501212422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6827","nonce":2501212422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6831","nonce":2501212422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6829","nonce":2501212422}]},"public_addr":"172.21.15.196:6825/2501212422","cluster_addr":"172.21.15.196:6827/2501212422","heartbeat_back_addr":"172.21.15.196:6831/2501212422","heartbeat_front_addr":"172.21.15.196:6829/2501212422","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:00:36.931566+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:01:02.614024+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:01:26.193265+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:01:58.308833+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:02:25.049546+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:02:49.748750+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:03:23.582153+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:03:50.447321+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:04:15.812129+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:04:49.998047+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:05:17.087924+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:05:43.167637+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":4,"to":7}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.a","mappings":[{"from":5,"to":8}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.39:0/3482086140":"2024-03-19T15:58:45.340294+0000","172.21.15.39:0/3844306284":"2024-03-19T15:58:45.340294+0000","172.21.15.39:6801/1810238546":"2024-03-19T15:58:45.340294+0000","172.21.15.39:6800/1810238546":"2024-03-19T15:58:45.340294+0000","172.21.15.39:0/2528397488":"2024-03-19T15:58:10.264798+0000","172.21.15.39:0/2240981254":"2024-03-19T15:58:10.264798+0000","172.21.15.39:6800/1217503189":"2024-03-19T15:58:10.264798+0000","172.21.15.39:0/3754876370":"2024-03-19T15:58:10.264798+0000","172.21.15.39:6801/1217503189":"2024-03-19T15:58:10.264798+0000","172.21.15.39:0/1874637628":"2024-03-19T15:58:45.340294+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T16:21:37.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.227+0000 7faa8d800640 1 -- 172.21.15.39:0/1298060338 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7faa6c0a2880 msgr2=0x7faa6c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:37.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.227+0000 7faa8d800640 1 --2- 172.21.15.39:0/1298060338 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7faa6c0a2880 0x7faa6c0a4d40 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7faa780095f0 tx=0x7faa78005f90 comp rx=0 tx=0).stop 2024-03-18T16:21:37.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.227+0000 7faa8d800640 1 -- 172.21.15.39:0/1298060338 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7faa88100820 msgr2=0x7faa88199f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:37.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.227+0000 7faa8d800640 1 --2- 172.21.15.39:0/1298060338 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7faa88100820 0x7faa88199f60 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7faa7000e9f0 tx=0x7faa7000eeb0 comp rx=0 tx=0).stop 2024-03-18T16:21:37.230 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.227+0000 7faa8d800640 1 -- 172.21.15.39:0/1298060338 shutdown_connections 2024-03-18T16:21:37.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.227+0000 7faa8d800640 1 --2- 172.21.15.39:0/1298060338 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7faa881011c0 0x7faa881a0fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:37.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.227+0000 7faa8d800640 1 --2- 172.21.15.39:0/1298060338 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7faa6c0a2880 0x7faa6c0a4d40 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:37.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.227+0000 7faa8d800640 1 --2- 172.21.15.39:0/1298060338 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7faa88100820 0x7faa88199f60 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:37.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.227+0000 7faa8d800640 1 --2- 172.21.15.39:0/1298060338 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faa880ffec0 0x7faa88199a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:37.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.227+0000 7faa8d800640 1 -- 172.21.15.39:0/1298060338 >> 172.21.15.39:0/1298060338 conn(0x7faa880fba60 msgr2=0x7faa880fd560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:37.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.228+0000 7faa8d800640 1 -- 172.21.15.39:0/1298060338 shutdown_connections 2024-03-18T16:21:37.231 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.228+0000 7faa8d800640 1 -- 172.21.15.39:0/1298060338 wait complete. 2024-03-18T16:21:37.254 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:37 smithi039 ceph-mon[31721]: pgmap v768: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-18T16:21:37.259 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-18T16:01:30.048247+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-18T16:06:42.191791+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-18T16:06:43.670325+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-18T16:06:47.523388+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-18T16:21:37.259 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-18T16:21:37.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:37 smithi196 ceph-mon[36307]: pgmap v768: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-18T16:21:37.487 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.484+0000 7f18d6600640 1 -- 172.21.15.39:0/3697308811 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f18d01008f0 msgr2=0x7f18d0100d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:37.487 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.484+0000 7f18d6600640 1 --2- 172.21.15.39:0/3697308811 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f18d01008f0 0x7f18d0100d50 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f18bc009900 tx=0x7f18bc02bc40 comp rx=0 tx=0).stop 2024-03-18T16:21:37.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.485+0000 7f18d6600640 1 -- 172.21.15.39:0/3697308811 shutdown_connections 2024-03-18T16:21:37.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.485+0000 7f18d6600640 1 --2- 172.21.15.39:0/3697308811 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f18d0101290 0x7f18d0101710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:37.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.485+0000 7f18d6600640 1 --2- 172.21.15.39:0/3697308811 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f18d01008f0 0x7f18d0100d50 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:37.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.485+0000 7f18d6600640 1 --2- 172.21.15.39:0/3697308811 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f18d00fff90 0x7f18d01003b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:37.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.485+0000 7f18d6600640 1 -- 172.21.15.39:0/3697308811 >> 172.21.15.39:0/3697308811 conn(0x7f18d00fbb50 msgr2=0x7f18d00fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:37.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.485+0000 7f18d6600640 1 -- 172.21.15.39:0/3697308811 shutdown_connections 2024-03-18T16:21:37.488 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.485+0000 7f18d6600640 1 -- 172.21.15.39:0/3697308811 wait complete. 2024-03-18T16:21:37.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.486+0000 7f18d6600640 1 Processor -- start 2024-03-18T16:21:37.489 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.486+0000 7f18d6600640 1 -- start start 2024-03-18T16:21:37.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.486+0000 7f18d6600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f18d00fff90 0x7f18d0199ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:37.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.487+0000 7f18d6600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f18d01008f0 0x7f18d019a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:37.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.487+0000 7f18d6600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f18d0101290 0x7f18d01a1150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:37.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.487+0000 7f18d6600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f18d010f580 con 0x7f18d0101290 2024-03-18T16:21:37.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.487+0000 7f18d6600640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f18d010f400 con 0x7f18d00fff90 2024-03-18T16:21:37.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.487+0000 7f18d6600640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f18d010f700 con 0x7f18d01008f0 2024-03-18T16:21:37.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.487+0000 7f18d5200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f18d00fff90 0x7f18d0199ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:37.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.487+0000 7f18d5200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f18d00fff90 0x7f18d0199ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:33910/0 (socket says 172.21.15.39:33910) 2024-03-18T16:21:37.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.487+0000 7f18d5200640 1 -- 172.21.15.39:0/1509541967 learned_addr learned my addr 172.21.15.39:0/1509541967 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:21:37.491 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.487+0000 7f18cfe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f18d01008f0 0x7f18d019a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:37.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.487+0000 7f18d5c00640 1 --2- 172.21.15.39:0/1509541967 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f18d0101290 0x7f18d01a1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:37.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.487+0000 7f18d5200640 1 -- 172.21.15.39:0/1509541967 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f18d01008f0 msgr2=0x7f18d019a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:37.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.487+0000 7f18d5200640 1 --2- 172.21.15.39:0/1509541967 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f18d01008f0 0x7f18d019a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:37.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.487+0000 7f18d5200640 1 -- 172.21.15.39:0/1509541967 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f18d0101290 msgr2=0x7f18d01a1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:37.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.487+0000 7f18d5200640 1 --2- 172.21.15.39:0/1509541967 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f18d0101290 0x7f18d01a1150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:37.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.487+0000 7f18d5200640 1 -- 172.21.15.39:0/1509541967 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18d01a17c0 con 0x7f18d00fff90 2024-03-18T16:21:37.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.488+0000 7f18d5200640 1 --2- 172.21.15.39:0/1509541967 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f18d00fff90 0x7f18d0199ba0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f18c400d950 tx=0x7f18c400de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:37.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.488+0000 7f18cd600640 1 -- 172.21.15.39:0/1509541967 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18c4017070 con 0x7f18d00fff90 2024-03-18T16:21:37.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.488+0000 7f18cd600640 1 -- 172.21.15.39:0/1509541967 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18c4009c30 con 0x7f18d00fff90 2024-03-18T16:21:37.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.488+0000 7f18cd600640 1 -- 172.21.15.39:0/1509541967 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18c400f6b0 con 0x7f18d00fff90 2024-03-18T16:21:37.492 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.488+0000 7f18d6600640 1 -- 172.21.15.39:0/1509541967 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18d01a1ab0 con 0x7f18d00fff90 2024-03-18T16:21:37.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.488+0000 7f18d6600640 1 -- 172.21.15.39:0/1509541967 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18d01a1f80 con 0x7f18d00fff90 2024-03-18T16:21:37.493 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:37 smithi194 ceph-mon[36200]: pgmap v768: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-18T16:21:37.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.489+0000 7f18d6600640 1 -- 172.21.15.39:0/1509541967 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18d010ae30 con 0x7f18d00fff90 2024-03-18T16:21:37.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.491+0000 7f18cd600640 1 -- 172.21.15.39:0/1509541967 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f18c40109d0 con 0x7f18d00fff90 2024-03-18T16:21:37.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.492+0000 7f18cd600640 1 --2- 172.21.15.39:0/1509541967 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f18b80a2880 0x7f18b80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:37.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.492+0000 7f18cd600640 1 -- 172.21.15.39:0/1509541967 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f18c40ce970 con 0x7f18d00fff90 2024-03-18T16:21:37.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.493+0000 7f18cfe00640 1 --2- 172.21.15.39:0/1509541967 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f18b80a2880 0x7f18b80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:37.496 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.493+0000 7f18cfe00640 1 --2- 172.21.15.39:0/1509541967 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f18b80a2880 0x7f18b80a4d40 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f18bc02c250 tx=0x7f18bc02cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:37.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.496+0000 7f18cd600640 1 -- 172.21.15.39:0/1509541967 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f18c4009de0 con 0x7f18d00fff90 2024-03-18T16:21:37.704 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.701+0000 7f18d6600640 1 -- 172.21.15.39:0/1509541967 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f18d002d500 con 0x7f18d00fff90 2024-03-18T16:21:37.705 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.702+0000 7f18cd600640 1 -- 172.21.15.39:0/1509541967 <== mon.2 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7f18d002d500 con 0x7f18d00fff90 2024-03-18T16:21:37.705 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2024-03-18T16:21:37.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.705+0000 7f18d6600640 1 -- 172.21.15.39:0/1509541967 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f18b80a2880 msgr2=0x7f18b80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:37.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.705+0000 7f18d6600640 1 --2- 172.21.15.39:0/1509541967 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f18b80a2880 0x7f18b80a4d40 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f18bc02c250 tx=0x7f18bc02cf90 comp rx=0 tx=0).stop 2024-03-18T16:21:37.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.705+0000 7f18d6600640 1 -- 172.21.15.39:0/1509541967 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f18d00fff90 msgr2=0x7f18d0199ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:37.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.705+0000 7f18d6600640 1 --2- 172.21.15.39:0/1509541967 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f18d00fff90 0x7f18d0199ba0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f18c400d950 tx=0x7f18c400de10 comp rx=0 tx=0).stop 2024-03-18T16:21:37.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.705+0000 7f18d6600640 1 -- 172.21.15.39:0/1509541967 shutdown_connections 2024-03-18T16:21:37.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.705+0000 7f18d6600640 1 --2- 172.21.15.39:0/1509541967 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f18b80a2880 0x7f18b80a4d40 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:37.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.705+0000 7f18d6600640 1 --2- 172.21.15.39:0/1509541967 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f18d0101290 0x7f18d01a1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:37.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.705+0000 7f18d6600640 1 --2- 172.21.15.39:0/1509541967 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f18d01008f0 0x7f18d019a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:37.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.705+0000 7f18d6600640 1 --2- 172.21.15.39:0/1509541967 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f18d00fff90 0x7f18d0199ba0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:37.708 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.705+0000 7f18d6600640 1 -- 172.21.15.39:0/1509541967 >> 172.21.15.39:0/1509541967 conn(0x7f18d00fbb50 msgr2=0x7f18d00fd210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:37.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.705+0000 7f18d6600640 1 -- 172.21.15.39:0/1509541967 shutdown_connections 2024-03-18T16:21:37.709 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.705+0000 7f18d6600640 1 -- 172.21.15.39:0/1509541967 wait complete. 2024-03-18T16:21:37.728 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-18T16:21:37.948 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.945+0000 7eff90200640 1 -- 172.21.15.39:0/3424301249 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7eff88104c10 msgr2=0x7eff88105090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:37.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.945+0000 7eff90200640 1 --2- 172.21.15.39:0/3424301249 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7eff88104c10 0x7eff88105090 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7eff74009900 tx=0x7eff7402bc30 comp rx=0 tx=0).stop 2024-03-18T16:21:37.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.946+0000 7eff90200640 1 -- 172.21.15.39:0/3424301249 shutdown_connections 2024-03-18T16:21:37.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.946+0000 7eff90200640 1 --2- 172.21.15.39:0/3424301249 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7eff881055d0 0x7eff8810be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:37.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.946+0000 7eff90200640 1 --2- 172.21.15.39:0/3424301249 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7eff88104c10 0x7eff88105090 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:37.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.946+0000 7eff90200640 1 --2- 172.21.15.39:0/3424301249 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7eff88103a20 0x7eff88103e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:37.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.946+0000 7eff90200640 1 -- 172.21.15.39:0/3424301249 >> 172.21.15.39:0/3424301249 conn(0x7eff880fef90 msgr2=0x7eff881013d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:37.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.946+0000 7eff90200640 1 -- 172.21.15.39:0/3424301249 shutdown_connections 2024-03-18T16:21:37.949 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.946+0000 7eff90200640 1 -- 172.21.15.39:0/3424301249 wait complete. 2024-03-18T16:21:37.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.947+0000 7eff90200640 1 Processor -- start 2024-03-18T16:21:37.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.947+0000 7eff90200640 1 -- start start 2024-03-18T16:21:37.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.947+0000 7eff90200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7eff88103a20 0x7eff88199d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:37.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.947+0000 7eff90200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7eff88104c10 0x7eff8819a260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:37.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.947+0000 7eff90200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7eff881055d0 0x7eff881a1290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:37.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.947+0000 7eff90200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7eff8810f6f0 con 0x7eff88103a20 2024-03-18T16:21:37.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.947+0000 7eff90200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7eff8810f570 con 0x7eff88104c10 2024-03-18T16:21:37.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.947+0000 7eff90200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7eff8810f870 con 0x7eff881055d0 2024-03-18T16:21:37.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.947+0000 7eff8e200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7eff881055d0 0x7eff881a1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:37.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.947+0000 7eff8ce00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7eff88104c10 0x7eff8819a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:37.957 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.947+0000 7eff8e200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7eff881055d0 0x7eff881a1290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:35236/0 (socket says 172.21.15.39:35236) 2024-03-18T16:21:37.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.947+0000 7eff8e200640 1 -- 172.21.15.39:0/4268008829 learned_addr learned my addr 172.21.15.39:0/4268008829 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:21:37.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.947+0000 7eff8d800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7eff88103a20 0x7eff88199d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:37.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.948+0000 7eff8e200640 1 -- 172.21.15.39:0/4268008829 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7eff88104c10 msgr2=0x7eff8819a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:37.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.948+0000 7eff8e200640 1 --2- 172.21.15.39:0/4268008829 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7eff88104c10 0x7eff8819a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:37.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.948+0000 7eff8e200640 1 -- 172.21.15.39:0/4268008829 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7eff88103a20 msgr2=0x7eff88199d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:37.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.948+0000 7eff8e200640 1 --2- 172.21.15.39:0/4268008829 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7eff88103a20 0x7eff88199d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:37.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.948+0000 7eff8e200640 1 -- 172.21.15.39:0/4268008829 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff881a1930 con 0x7eff881055d0 2024-03-18T16:21:37.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.948+0000 7eff8ce00640 1 --2- 172.21.15.39:0/4268008829 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7eff88104c10 0x7eff8819a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:21:37.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.948+0000 7eff8e200640 1 --2- 172.21.15.39:0/4268008829 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7eff881055d0 0x7eff881a1290 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7eff8000eb10 tx=0x7eff8000c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:37.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.948+0000 7eff86000640 1 -- 172.21.15.39:0/4268008829 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff80004b20 con 0x7eff881055d0 2024-03-18T16:21:37.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.949+0000 7eff86000640 1 -- 172.21.15.39:0/4268008829 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff80004cc0 con 0x7eff881055d0 2024-03-18T16:21:37.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.949+0000 7eff86000640 1 -- 172.21.15.39:0/4268008829 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff80010760 con 0x7eff881055d0 2024-03-18T16:21:37.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.949+0000 7eff90200640 1 -- 172.21.15.39:0/4268008829 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff881a1c20 con 0x7eff881055d0 2024-03-18T16:21:37.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.949+0000 7eff90200640 1 -- 172.21.15.39:0/4268008829 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff881a21c0 con 0x7eff881055d0 2024-03-18T16:21:37.958 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.950+0000 7eff90200640 1 -- 172.21.15.39:0/4268008829 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff8810ad40 con 0x7eff881055d0 2024-03-18T16:21:37.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.952+0000 7eff86000640 1 -- 172.21.15.39:0/4268008829 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7eff800459b0 con 0x7eff881055d0 2024-03-18T16:21:37.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.954+0000 7eff86000640 1 --2- 172.21.15.39:0/4268008829 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7eff700a2880 0x7eff700a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:37.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.954+0000 7eff86000640 1 -- 172.21.15.39:0/4268008829 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7eff800cf490 con 0x7eff881055d0 2024-03-18T16:21:37.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.954+0000 7eff8d800640 1 --2- 172.21.15.39:0/4268008829 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7eff700a2880 0x7eff700a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:37.959 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.955+0000 7eff8d800640 1 --2- 172.21.15.39:0/4268008829 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7eff700a2880 0x7eff700a4d40 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7eff8819b1f0 tx=0x7eff7c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:37.962 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:37.960+0000 7eff86000640 1 -- 172.21.15.39:0/4268008829 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7eff8009c140 con 0x7eff881055d0 2024-03-18T16:21:38.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.165+0000 7eff90200640 1 -- 172.21.15.39:0/4268008829 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7eff8802d360 con 0x7eff881055d0 2024-03-18T16:21:38.168 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.166+0000 7eff86000640 1 -- 172.21.15.39:0/4268008829 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7eff8009ba40 con 0x7eff881055d0 2024-03-18T16:21:38.169 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 32 2024-03-18T16:21:38.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.168+0000 7eff90200640 1 -- 172.21.15.39:0/4268008829 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7eff700a2880 msgr2=0x7eff700a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:38.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.168+0000 7eff90200640 1 --2- 172.21.15.39:0/4268008829 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7eff700a2880 0x7eff700a4d40 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7eff8819b1f0 tx=0x7eff7c00b410 comp rx=0 tx=0).stop 2024-03-18T16:21:38.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.168+0000 7eff90200640 1 -- 172.21.15.39:0/4268008829 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7eff881055d0 msgr2=0x7eff881a1290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:38.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.168+0000 7eff90200640 1 --2- 172.21.15.39:0/4268008829 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7eff881055d0 0x7eff881a1290 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7eff8000eb10 tx=0x7eff8000c2d0 comp rx=0 tx=0).stop 2024-03-18T16:21:38.171 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.168+0000 7eff90200640 1 -- 172.21.15.39:0/4268008829 shutdown_connections 2024-03-18T16:21:38.172 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.168+0000 7eff90200640 1 --2- 172.21.15.39:0/4268008829 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7eff88104c10 0x7eff8819a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:38.172 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.168+0000 7eff90200640 1 --2- 172.21.15.39:0/4268008829 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7eff700a2880 0x7eff700a4d40 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:38.172 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.168+0000 7eff90200640 1 --2- 172.21.15.39:0/4268008829 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7eff881055d0 0x7eff881a1290 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:38.172 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.168+0000 7eff90200640 1 --2- 172.21.15.39:0/4268008829 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7eff88103a20 0x7eff88199d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:38.172 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.168+0000 7eff90200640 1 -- 172.21.15.39:0/4268008829 >> 172.21.15.39:0/4268008829 conn(0x7eff880fef90 msgr2=0x7eff88107f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:38.172 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.169+0000 7eff90200640 1 -- 172.21.15.39:0/4268008829 shutdown_connections 2024-03-18T16:21:38.172 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.169+0000 7eff90200640 1 -- 172.21.15.39:0/4268008829 wait complete. 2024-03-18T16:21:38.194 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-18T16:21:38.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:38 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/1298060338' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:21:38.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:38 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/1509541967' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T16:21:38.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.414+0000 7f3399e00640 1 -- 172.21.15.39:0/3778359210 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f33941029e0 msgr2=0x7f3394102e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:38.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.414+0000 7f3399e00640 1 --2- 172.21.15.39:0/3778359210 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f33941029e0 0x7f3394102e60 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f337c009860 tx=0x7f337c02bb70 comp rx=0 tx=0).stop 2024-03-18T16:21:38.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.415+0000 7f3399e00640 1 -- 172.21.15.39:0/3778359210 shutdown_connections 2024-03-18T16:21:38.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.415+0000 7f3399e00640 1 --2- 172.21.15.39:0/3778359210 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f33941033a0 0x7f3394109c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:38.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.415+0000 7f3399e00640 1 --2- 172.21.15.39:0/3778359210 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f33941029e0 0x7f3394102e60 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:38.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.415+0000 7f3399e00640 1 --2- 172.21.15.39:0/3778359210 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f33941017f0 0x7f3394101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:38.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.415+0000 7f3399e00640 1 -- 172.21.15.39:0/3778359210 >> 172.21.15.39:0/3778359210 conn(0x7f33940fcd60 msgr2=0x7f33940ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:38.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.415+0000 7f3399e00640 1 -- 172.21.15.39:0/3778359210 shutdown_connections 2024-03-18T16:21:38.418 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.415+0000 7f3399e00640 1 -- 172.21.15.39:0/3778359210 wait complete. 2024-03-18T16:21:38.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.416+0000 7f3399e00640 1 Processor -- start 2024-03-18T16:21:38.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.416+0000 7f3399e00640 1 -- start start 2024-03-18T16:21:38.420 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.416+0000 7f3399e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f33941017f0 0x7f3394197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:38.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.416+0000 7f3399e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f33941029e0 0x7f3394198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:38.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.416+0000 7f3399e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f33941033a0 0x7f339419f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:38.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.416+0000 7f3399e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f339410d450 con 0x7f33941029e0 2024-03-18T16:21:38.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.416+0000 7f3399e00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f339410d2d0 con 0x7f33941033a0 2024-03-18T16:21:38.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.416+0000 7f3399e00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f339410d5d0 con 0x7f33941017f0 2024-03-18T16:21:38.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.416+0000 7f3392a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f33941017f0 0x7f3394197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:38.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.416+0000 7f3392a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f33941017f0 0x7f3394197b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:41562/0 (socket says 172.21.15.39:41562) 2024-03-18T16:21:38.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.416+0000 7f3392a00640 1 -- 172.21.15.39:0/4142415784 learned_addr learned my addr 172.21.15.39:0/4142415784 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:21:38.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.416+0000 7f3392000640 1 --2- 172.21.15.39:0/4142415784 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f33941029e0 0x7f3394198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:38.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.416+0000 7f3393400640 1 --2- 172.21.15.39:0/4142415784 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f33941033a0 0x7f339419f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:38.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.417+0000 7f3392a00640 1 -- 172.21.15.39:0/4142415784 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f33941033a0 msgr2=0x7f339419f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:38.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.417+0000 7f3392a00640 1 --2- 172.21.15.39:0/4142415784 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f33941033a0 0x7f339419f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:38.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.417+0000 7f3392a00640 1 -- 172.21.15.39:0/4142415784 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f33941029e0 msgr2=0x7f3394198090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:38.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.417+0000 7f3392a00640 1 --2- 172.21.15.39:0/4142415784 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f33941029e0 0x7f3394198090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:38.421 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.417+0000 7f3392a00640 1 -- 172.21.15.39:0/4142415784 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f339419f770 con 0x7f33941017f0 2024-03-18T16:21:38.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.417+0000 7f3392a00640 1 --2- 172.21.15.39:0/4142415784 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f33941017f0 0x7f3394197b50 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f338400c3a0 tx=0x7f338400c860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:38.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.417+0000 7f3398a00640 1 -- 172.21.15.39:0/4142415784 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3384003970 con 0x7f33941017f0 2024-03-18T16:21:38.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.417+0000 7f3398a00640 1 -- 172.21.15.39:0/4142415784 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3384004020 con 0x7f33941017f0 2024-03-18T16:21:38.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.417+0000 7f3399e00640 1 -- 172.21.15.39:0/4142415784 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f339419fa60 con 0x7f33941017f0 2024-03-18T16:21:38.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.417+0000 7f3398a00640 1 -- 172.21.15.39:0/4142415784 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f338400b9f0 con 0x7f33941017f0 2024-03-18T16:21:38.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.418+0000 7f3399e00640 1 -- 172.21.15.39:0/4142415784 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f339419ffd0 con 0x7f33941017f0 2024-03-18T16:21:38.422 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.419+0000 7f3399e00640 1 -- 172.21.15.39:0/4142415784 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3394108b60 con 0x7f33941017f0 2024-03-18T16:21:38.423 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.421+0000 7f3398a00640 1 -- 172.21.15.39:0/4142415784 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f338401d020 con 0x7f33941017f0 2024-03-18T16:21:38.425 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.423+0000 7f3398a00640 1 --2- 172.21.15.39:0/4142415784 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f33780a2880 0x7f33780a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:38.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.423+0000 7f3392000640 1 --2- 172.21.15.39:0/4142415784 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f33780a2880 0x7f33780a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:38.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.423+0000 7f3398a00640 1 -- 172.21.15.39:0/4142415784 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f33840cefd0 con 0x7f33941017f0 2024-03-18T16:21:38.426 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.424+0000 7f3392000640 1 --2- 172.21.15.39:0/4142415784 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f33780a2880 0x7f33780a4d40 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f337c009830 tx=0x7f337c034ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:38.427 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.424+0000 7f3398a00640 1 -- 172.21.15.39:0/4142415784 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f338409bc80 con 0x7f33941017f0 2024-03-18T16:21:38.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:38 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/1298060338' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:21:38.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:38 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/1509541967' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T16:21:38.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:38 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/1298060338' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:21:38.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:38 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/1509541967' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-18T16:21:38.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.644+0000 7f3399e00640 1 -- 172.21.15.39:0/4142415784 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f339402d420 con 0x7f33941017f0 2024-03-18T16:21:38.647 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.645+0000 7f3398a00640 1 -- 172.21.15.39:0/4142415784 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7f338409b580 con 0x7f33941017f0 2024-03-18T16:21:38.647 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 64 2024-03-18T16:21:38.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.648+0000 7f3399e00640 1 -- 172.21.15.39:0/4142415784 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f33780a2880 msgr2=0x7f33780a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:38.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.648+0000 7f3399e00640 1 --2- 172.21.15.39:0/4142415784 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f33780a2880 0x7f33780a4d40 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f337c009830 tx=0x7f337c034ab0 comp rx=0 tx=0).stop 2024-03-18T16:21:38.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.648+0000 7f3399e00640 1 -- 172.21.15.39:0/4142415784 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f33941017f0 msgr2=0x7f3394197b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:38.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.648+0000 7f3399e00640 1 --2- 172.21.15.39:0/4142415784 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f33941017f0 0x7f3394197b50 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f338400c3a0 tx=0x7f338400c860 comp rx=0 tx=0).stop 2024-03-18T16:21:38.651 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.648+0000 7f3399e00640 1 -- 172.21.15.39:0/4142415784 shutdown_connections 2024-03-18T16:21:38.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.648+0000 7f3399e00640 1 --2- 172.21.15.39:0/4142415784 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f33941033a0 0x7f339419f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:38.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.648+0000 7f3399e00640 1 --2- 172.21.15.39:0/4142415784 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f33780a2880 0x7f33780a4d40 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:38.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.648+0000 7f3399e00640 1 --2- 172.21.15.39:0/4142415784 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f33941029e0 0x7f3394198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:38.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.648+0000 7f3399e00640 1 --2- 172.21.15.39:0/4142415784 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f33941017f0 0x7f3394197b50 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:38.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.648+0000 7f3399e00640 1 -- 172.21.15.39:0/4142415784 >> 172.21.15.39:0/4142415784 conn(0x7f33940fcd60 msgr2=0x7f3394105d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:38.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.649+0000 7f3399e00640 1 -- 172.21.15.39:0/4142415784 shutdown_connections 2024-03-18T16:21:38.652 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.649+0000 7f3399e00640 1 -- 172.21.15.39:0/4142415784 wait complete. 2024-03-18T16:21:38.672 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-18T16:21:38.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.892+0000 7f4ad2e00640 1 -- 172.21.15.39:0/1868010575 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4acc1016e0 msgr2=0x7f4acc101b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:38.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.892+0000 7f4ad2e00640 1 --2- 172.21.15.39:0/1868010575 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4acc1016e0 0x7f4acc101b00 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f4ac0009900 tx=0x7f4ac002bc30 comp rx=0 tx=0).stop 2024-03-18T16:21:38.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.892+0000 7f4ad2e00640 1 -- 172.21.15.39:0/1868010575 shutdown_connections 2024-03-18T16:21:38.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.892+0000 7f4ad2e00640 1 --2- 172.21.15.39:0/1868010575 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4acc1028d0 0x7f4acc102d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:38.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.892+0000 7f4ad2e00640 1 --2- 172.21.15.39:0/1868010575 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4acc103290 0x7f4acc109b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:38.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.892+0000 7f4ad2e00640 1 --2- 172.21.15.39:0/1868010575 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4acc1016e0 0x7f4acc101b00 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:38.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.892+0000 7f4ad2e00640 1 -- 172.21.15.39:0/1868010575 >> 172.21.15.39:0/1868010575 conn(0x7f4acc0fcc50 msgr2=0x7f4acc0ff090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:38.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.892+0000 7f4ad2e00640 1 -- 172.21.15.39:0/1868010575 shutdown_connections 2024-03-18T16:21:38.895 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.892+0000 7f4ad2e00640 1 -- 172.21.15.39:0/1868010575 wait complete. 2024-03-18T16:21:38.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.893+0000 7f4ad2e00640 1 Processor -- start 2024-03-18T16:21:38.896 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.894+0000 7f4ad2e00640 1 -- start start 2024-03-18T16:21:38.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.894+0000 7f4ad2e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4acc1016e0 0x7f4acc1979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:38.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.894+0000 7f4ad2e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4acc1028d0 0x7f4acc197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:38.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.894+0000 7f4ad2e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4acc103290 0x7f4acc19efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:38.897 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.894+0000 7f4ad2e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f4acc10d380 con 0x7f4acc103290 2024-03-18T16:21:38.898 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.894+0000 7f4ad2e00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f4acc10d200 con 0x7f4acc1028d0 2024-03-18T16:21:38.898 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.894+0000 7f4ad2e00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f4acc10d500 con 0x7f4acc1016e0 2024-03-18T16:21:38.898 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.894+0000 7f4ad1a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4acc1016e0 0x7f4acc1979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:38.898 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.894+0000 7f4ad1a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4acc1016e0 0x7f4acc1979f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:41576/0 (socket says 172.21.15.39:41576) 2024-03-18T16:21:38.898 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.894+0000 7f4ad1a00640 1 -- 172.21.15.39:0/160944757 learned_addr learned my addr 172.21.15.39:0/160944757 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:21:38.898 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.894+0000 7f4ad1000640 1 --2- 172.21.15.39:0/160944757 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4acc1028d0 0x7f4acc197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:38.898 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.894+0000 7f4ad2400640 1 --2- 172.21.15.39:0/160944757 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4acc103290 0x7f4acc19efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:38.898 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.895+0000 7f4ad1a00640 1 -- 172.21.15.39:0/160944757 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4acc1028d0 msgr2=0x7f4acc197f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:38.898 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.895+0000 7f4ad1a00640 1 --2- 172.21.15.39:0/160944757 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4acc1028d0 0x7f4acc197f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:38.898 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.895+0000 7f4ad1a00640 1 -- 172.21.15.39:0/160944757 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4acc103290 msgr2=0x7f4acc19efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:38.898 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.895+0000 7f4ad1a00640 1 --2- 172.21.15.39:0/160944757 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4acc103290 0x7f4acc19efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:38.898 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.895+0000 7f4ad1a00640 1 -- 172.21.15.39:0/160944757 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4acc19f610 con 0x7f4acc1016e0 2024-03-18T16:21:38.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.895+0000 7f4ad1a00640 1 --2- 172.21.15.39:0/160944757 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4acc1016e0 0x7f4acc1979f0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f4ac0000c00 tx=0x7f4ac002c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:38.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.895+0000 7f4aca000640 1 -- 172.21.15.39:0/160944757 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ac0043020 con 0x7f4acc1016e0 2024-03-18T16:21:38.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.896+0000 7f4aca000640 1 -- 172.21.15.39:0/160944757 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ac00054e0 con 0x7f4acc1016e0 2024-03-18T16:21:38.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.896+0000 7f4aca000640 1 -- 172.21.15.39:0/160944757 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ac0003c40 con 0x7f4acc1016e0 2024-03-18T16:21:38.899 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.896+0000 7f4ad2e00640 1 -- 172.21.15.39:0/160944757 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4acc19f8a0 con 0x7f4acc1016e0 2024-03-18T16:21:38.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.896+0000 7f4ad2e00640 1 -- 172.21.15.39:0/160944757 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4acc19fe40 con 0x7f4acc1016e0 2024-03-18T16:21:38.900 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.897+0000 7f4ad2e00640 1 -- 172.21.15.39:0/160944757 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4acc0684f0 con 0x7f4acc1016e0 2024-03-18T16:21:38.901 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.899+0000 7f4aca000640 1 -- 172.21.15.39:0/160944757 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f4ac0046020 con 0x7f4acc1016e0 2024-03-18T16:21:38.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.900+0000 7f4aca000640 1 --2- 172.21.15.39:0/160944757 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f4ab40a2880 0x7f4ab40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:38.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.900+0000 7f4aca000640 1 -- 172.21.15.39:0/160944757 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f4ac00f12d0 con 0x7f4acc1016e0 2024-03-18T16:21:38.903 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.901+0000 7f4ad1000640 1 --2- 172.21.15.39:0/160944757 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f4ab40a2880 0x7f4ab40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:38.904 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.901+0000 7f4ad1000640 1 --2- 172.21.15.39:0/160944757 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f4ab40a2880 0x7f4ab40a4d40 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f4acc198f00 tx=0x7f4ab8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:38.909 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:38.907+0000 7f4aca000640 1 -- 172.21.15.39:0/160944757 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f4ac004d020 con 0x7f4acc1016e0 2024-03-18T16:21:39.116 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:39.114+0000 7f4ad2e00640 1 -- 172.21.15.39:0/160944757 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f4acc02d1b0 con 0x7f4acc1016e0 2024-03-18T16:21:39.117 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:39.115+0000 7f4aca000640 1 -- 172.21.15.39:0/160944757 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7f4ac00bdf80 con 0x7f4acc1016e0 2024-03-18T16:21:39.117 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 32 2024-03-18T16:21:39.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:39.117+0000 7f4ad2e00640 1 -- 172.21.15.39:0/160944757 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f4ab40a2880 msgr2=0x7f4ab40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:39.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:39.117+0000 7f4ad2e00640 1 --2- 172.21.15.39:0/160944757 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f4ab40a2880 0x7f4ab40a4d40 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f4acc198f00 tx=0x7f4ab8005f90 comp rx=0 tx=0).stop 2024-03-18T16:21:39.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:39.117+0000 7f4ad2e00640 1 -- 172.21.15.39:0/160944757 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4acc1016e0 msgr2=0x7f4acc1979f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:39.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:39.117+0000 7f4ad2e00640 1 --2- 172.21.15.39:0/160944757 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4acc1016e0 0x7f4acc1979f0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f4ac0000c00 tx=0x7f4ac002c650 comp rx=0 tx=0).stop 2024-03-18T16:21:39.120 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:39.117+0000 7f4ad2e00640 1 -- 172.21.15.39:0/160944757 shutdown_connections 2024-03-18T16:21:39.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:39.117+0000 7f4ad2e00640 1 --2- 172.21.15.39:0/160944757 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4acc1028d0 0x7f4acc197f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:39.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:39.117+0000 7f4ad2e00640 1 --2- 172.21.15.39:0/160944757 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f4ab40a2880 0x7f4ab40a4d40 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:39.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:39.117+0000 7f4ad2e00640 1 --2- 172.21.15.39:0/160944757 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f4acc103290 0x7f4acc19efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:39.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:39.117+0000 7f4ad2e00640 1 --2- 172.21.15.39:0/160944757 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4acc1016e0 0x7f4acc1979f0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:39.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:39.117+0000 7f4ad2e00640 1 -- 172.21.15.39:0/160944757 >> 172.21.15.39:0/160944757 conn(0x7f4acc0fcc50 msgr2=0x7f4acc105c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:39.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:39.118+0000 7f4ad2e00640 1 -- 172.21.15.39:0/160944757 shutdown_connections 2024-03-18T16:21:39.121 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:39.118+0000 7f4ad2e00640 1 -- 172.21.15.39:0/160944757 wait complete. 2024-03-18T16:21:39.141 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-18T16:21:39.336 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:39 smithi039 ceph-mon[31721]: pgmap v769: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T16:21:39.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:39 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/4268008829' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T16:21:39.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:39 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/4142415784' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T16:21:39.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:39.343+0000 7f6dc8c00640 1 -- 172.21.15.39:0/2224207936 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6dc41016e0 msgr2=0x7f6dc4101b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:39.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:39.343+0000 7f6dc8c00640 1 --2- 172.21.15.39:0/2224207936 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6dc41016e0 0x7f6dc4101b00 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f6db4009900 tx=0x7f6db402bc40 comp rx=0 tx=0).stop 2024-03-18T16:21:39.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:39.343+0000 7f6dc8c00640 1 -- 172.21.15.39:0/2224207936 shutdown_connections 2024-03-18T16:21:39.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:39.343+0000 7f6dc8c00640 1 --2- 172.21.15.39:0/2224207936 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6dc4103290 0x7f6dc4109b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:39.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:39.343+0000 7f6dc8c00640 1 --2- 172.21.15.39:0/2224207936 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6dc41028d0 0x7f6dc4102d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:39.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:39.343+0000 7f6dc8c00640 1 --2- 172.21.15.39:0/2224207936 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6dc41016e0 0x7f6dc4101b00 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:39.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:39.343+0000 7f6dc8c00640 1 -- 172.21.15.39:0/2224207936 >> 172.21.15.39:0/2224207936 conn(0x7f6dc40fcc50 msgr2=0x7f6dc40ff090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:39.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:39.343+0000 7f6dc8c00640 1 -- 172.21.15.39:0/2224207936 shutdown_connections 2024-03-18T16:21:39.346 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:39.343+0000 7f6dc8c00640 1 -- 172.21.15.39:0/2224207936 wait complete. 2024-03-18T16:21:39.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:39.344+0000 7f6dc8c00640 1 Processor -- start 2024-03-18T16:21:39.348 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:39.344+0000 7f6dc8c00640 1 -- start start 2024-03-18T16:21:39.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:39.345+0000 7f6dc8c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6dc41016e0 0x7f6dc4199c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:39.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:39.345+0000 7f6dc8c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6dc41028d0 0x7f6dc419a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:39.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:39.345+0000 7f6dc8c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6dc4103290 0x7f6dc41a11c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:39.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:39.345+0000 7f6dc8c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f6dc410d380 con 0x7f6dc4103290 2024-03-18T16:21:39.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:39.345+0000 7f6dc8c00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f6dc410d200 con 0x7f6dc41016e0 2024-03-18T16:21:39.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:39.345+0000 7f6dc8c00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f6dc410d500 con 0x7f6dc41028d0 2024-03-18T16:21:39.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:39.345+0000 7f6dc3400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6dc41016e0 0x7f6dc4199c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:39.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:39.345+0000 7f6dc2a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6dc41028d0 0x7f6dc419a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:39.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:39.345+0000 7f6dc3400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6dc41016e0 0x7f6dc4199c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:37014/0 (socket says 172.21.15.39:37014) 2024-03-18T16:21:39.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:39.345+0000 7f6dc3400640 1 -- 172.21.15.39:0/3861972972 learned_addr learned my addr 172.21.15.39:0/3861972972 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:21:39.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:39.345+0000 7f6dc3e00640 1 --2- 172.21.15.39:0/3861972972 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6dc4103290 0x7f6dc41a11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:39.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:39.346+0000 7f6dc3400640 1 -- 172.21.15.39:0/3861972972 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6dc41028d0 msgr2=0x7f6dc419a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:39.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:39.346+0000 7f6dc3400640 1 --2- 172.21.15.39:0/3861972972 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6dc41028d0 0x7f6dc419a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:39.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:39.346+0000 7f6dc3400640 1 -- 172.21.15.39:0/3861972972 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6dc4103290 msgr2=0x7f6dc41a11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:39.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:39.346+0000 7f6dc3400640 1 --2- 172.21.15.39:0/3861972972 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6dc4103290 0x7f6dc41a11c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:39.349 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:39.346+0000 7f6dc3400640 1 -- 172.21.15.39:0/3861972972 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dc41a1860 con 0x7f6dc41016e0 2024-03-18T16:21:39.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:39.346+0000 7f6dc2a00640 1 --2- 172.21.15.39:0/3861972972 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6dc41028d0 0x7f6dc419a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:21:39.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:39.346+0000 7f6dc3e00640 1 --2- 172.21.15.39:0/3861972972 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6dc4103290 0x7f6dc41a11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:21:39.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:39.346+0000 7f6dc3400640 1 --2- 172.21.15.39:0/3861972972 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6dc41016e0 0x7f6dc4199c50 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f6db402efd0 tx=0x7f6db4005d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:39.350 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:39.346+0000 7f6dbbe00640 1 -- 172.21.15.39:0/3861972972 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6db4041070 con 0x7f6dc41016e0 2024-03-18T16:21:39.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:39.346+0000 7f6dbbe00640 1 -- 172.21.15.39:0/3861972972 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6db40044d0 con 0x7f6dc41016e0 2024-03-18T16:21:39.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:39.346+0000 7f6dbbe00640 1 -- 172.21.15.39:0/3861972972 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6db4034440 con 0x7f6dc41016e0 2024-03-18T16:21:39.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:39.346+0000 7f6dc8c00640 1 -- 172.21.15.39:0/3861972972 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6dc41a1af0 con 0x7f6dc41016e0 2024-03-18T16:21:39.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:39.346+0000 7f6dc8c00640 1 -- 172.21.15.39:0/3861972972 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6dc41a2000 con 0x7f6dc41016e0 2024-03-18T16:21:39.351 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:39.348+0000 7f6dc8c00640 1 -- 172.21.15.39:0/3861972972 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6dc41a21d0 con 0x7f6dc41016e0 2024-03-18T16:21:39.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:39.349+0000 7f6dbbe00640 1 -- 172.21.15.39:0/3861972972 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f6db4044070 con 0x7f6dc41016e0 2024-03-18T16:21:39.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:39.350+0000 7f6dbbe00640 1 --2- 172.21.15.39:0/3861972972 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f6da80a2880 0x7f6da80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:39.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:39.351+0000 7f6dbbe00640 1 -- 172.21.15.39:0/3861972972 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f6db40f0aa0 con 0x7f6dc41016e0 2024-03-18T16:21:39.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:39.351+0000 7f6dc2a00640 1 --2- 172.21.15.39:0/3861972972 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f6da80a2880 0x7f6da80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:39.356 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:39.352+0000 7f6dc2a00640 1 --2- 172.21.15.39:0/3861972972 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f6da80a2880 0x7f6da80a4d40 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f6db0009fc0 tx=0x7f6db0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:39.357 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:39.355+0000 7f6dbbe00640 1 -- 172.21.15.39:0/3861972972 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f6db40bd750 con 0x7f6dc41016e0 2024-03-18T16:21:39.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:39 smithi196 ceph-mon[36307]: pgmap v769: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T16:21:39.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:39 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/4268008829' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T16:21:39.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:39 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/4142415784' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T16:21:39.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:39 smithi194 ceph-mon[36200]: pgmap v769: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-18T16:21:39.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:39 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/4268008829' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-18T16:21:39.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:39 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/4142415784' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-18T16:21:39.578 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:39.575+0000 7f6dc8c00640 1 -- 172.21.15.39:0/3861972972 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f6dc41a23a0 con 0x7f6dc41016e0 2024-03-18T16:21:39.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:39.576+0000 7f6dbbe00640 1 -- 172.21.15.39:0/3861972972 <== mon.2 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v93) ==== 81+0+592 (secure 0 0 0) 0x7f6db40bd050 con 0x7f6dc41016e0 2024-03-18T16:21:39.579 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/3482086140 2024-03-19T15:58:45.340294+0000 2024-03-18T16:21:39.579 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/3844306284 2024-03-19T15:58:45.340294+0000 2024-03-18T16:21:39.579 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:6801/1810238546 2024-03-19T15:58:45.340294+0000 2024-03-18T16:21:39.579 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:6800/1810238546 2024-03-19T15:58:45.340294+0000 2024-03-18T16:21:39.579 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/2528397488 2024-03-19T15:58:10.264798+0000 2024-03-18T16:21:39.579 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/2240981254 2024-03-19T15:58:10.264798+0000 2024-03-18T16:21:39.579 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:6800/1217503189 2024-03-19T15:58:10.264798+0000 2024-03-18T16:21:39.579 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/3754876370 2024-03-19T15:58:10.264798+0000 2024-03-18T16:21:39.579 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:6801/1217503189 2024-03-19T15:58:10.264798+0000 2024-03-18T16:21:39.579 INFO:teuthology.orchestra.run.smithi039.stdout:172.21.15.39:0/1874637628 2024-03-19T15:58:45.340294+0000 2024-03-18T16:21:39.580 INFO:teuthology.orchestra.run.smithi039.stderr:listed 10 entries 2024-03-18T16:21:39.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:39.580+0000 7f6dc8c00640 1 -- 172.21.15.39:0/3861972972 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f6da80a2880 msgr2=0x7f6da80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:39.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:39.580+0000 7f6dc8c00640 1 --2- 172.21.15.39:0/3861972972 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f6da80a2880 0x7f6da80a4d40 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f6db0009fc0 tx=0x7f6db0005f90 comp rx=0 tx=0).stop 2024-03-18T16:21:39.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:39.580+0000 7f6dc8c00640 1 -- 172.21.15.39:0/3861972972 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6dc41016e0 msgr2=0x7f6dc4199c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:39.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:39.580+0000 7f6dc8c00640 1 --2- 172.21.15.39:0/3861972972 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6dc41016e0 0x7f6dc4199c50 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f6db402efd0 tx=0x7f6db4005d50 comp rx=0 tx=0).stop 2024-03-18T16:21:39.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:39.580+0000 7f6dc8c00640 1 -- 172.21.15.39:0/3861972972 shutdown_connections 2024-03-18T16:21:39.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:39.580+0000 7f6dc8c00640 1 --2- 172.21.15.39:0/3861972972 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f6da80a2880 0x7f6da80a4d40 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:39.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:39.580+0000 7f6dc8c00640 1 --2- 172.21.15.39:0/3861972972 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f6dc4103290 0x7f6dc41a11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:39.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:39.580+0000 7f6dc8c00640 1 --2- 172.21.15.39:0/3861972972 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6dc41028d0 0x7f6dc419a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:39.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:39.580+0000 7f6dc8c00640 1 --2- 172.21.15.39:0/3861972972 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6dc41016e0 0x7f6dc4199c50 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:39.582 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:39.580+0000 7f6dc8c00640 1 -- 172.21.15.39:0/3861972972 >> 172.21.15.39:0/3861972972 conn(0x7f6dc40fcc50 msgr2=0x7f6dc4073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:39.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:39.580+0000 7f6dc8c00640 1 -- 172.21.15.39:0/3861972972 shutdown_connections 2024-03-18T16:21:39.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:39.580+0000 7f6dc8c00640 1 -- 172.21.15.39:0/3861972972 wait complete. 2024-03-18T16:21:39.602 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-18T16:21:39.602 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T16:21:39.602 DEBUG:teuthology.orchestra.run.smithi039:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-18T16:21:39.787 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi039.front.sepia.ceph.com 2024-03-18T16:21:39.787 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T16:21:39.787 DEBUG:teuthology.orchestra.run.smithi039:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-18T16:21:39.867 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T16:21:39.867 DEBUG:teuthology.orchestra.run.smithi039:> ip netns list 2024-03-18T16:21:39.938 INFO:teuthology.orchestra.run.smithi039.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-18T16:21:39.939 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T16:21:39.939 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-18T16:21:40.015 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-18T16:21:40.015 DEBUG:teuthology.orchestra.run.smithi039:> sudo ip link delete ceph-brx 2024-03-18T16:21:40.145 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-18T16:21:40.155 INFO:tasks.cephadm:Teardown begin 2024-03-18T16:21:40.156 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-18T16:21:40.156 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:21:40.262 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:40 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/160944757' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T16:21:40.262 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:40 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/3861972972' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-18T16:21:40.386 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.384+0000 7f7327200640 1 -- 172.21.15.39:0/2005945195 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f73201097d0 msgr2=0x7f7320110060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:40.386 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.384+0000 7f7327200640 1 --2- 172.21.15.39:0/2005945195 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f73201097d0 0x7f7320110060 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f731c009ca0 tx=0x7f731c02c0f0 comp rx=0 tx=0).stop 2024-03-18T16:21:40.386 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.384+0000 7f7327200640 1 -- 172.21.15.39:0/2005945195 shutdown_connections 2024-03-18T16:21:40.386 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.384+0000 7f7327200640 1 --2- 172.21.15.39:0/2005945195 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f73201097d0 0x7f7320110060 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:40.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.384+0000 7f7327200640 1 --2- 172.21.15.39:0/2005945195 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7320108e10 0x7f7320109290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:40.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.384+0000 7f7327200640 1 --2- 172.21.15.39:0/2005945195 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7320107c20 0x7f7320108040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:40.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.384+0000 7f7327200640 1 -- 172.21.15.39:0/2005945195 >> 172.21.15.39:0/2005945195 conn(0x7f7320076bb0 msgr2=0x7f7320078ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:40.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.384+0000 7f7327200640 1 -- 172.21.15.39:0/2005945195 shutdown_connections 2024-03-18T16:21:40.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.385+0000 7f7327200640 1 -- 172.21.15.39:0/2005945195 wait complete. 2024-03-18T16:21:40.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.386+0000 7f7327200640 1 Processor -- start 2024-03-18T16:21:40.387 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.386+0000 7f7327200640 1 -- start start 2024-03-18T16:21:40.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.386+0000 7f7327200640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7320107c20 0x7f732019def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:40.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.386+0000 7f7327200640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7320108e10 0x7f732019e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:40.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.386+0000 7f7327200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f73201097d0 0x7f73201a54a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:40.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.386+0000 7f7327200640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f73201138a0 con 0x7f7320108e10 2024-03-18T16:21:40.389 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.386+0000 7f7327200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f7320113720 con 0x7f73201097d0 2024-03-18T16:21:40.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.386+0000 7f7327200640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f7320113a20 con 0x7f7320107c20 2024-03-18T16:21:40.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.386+0000 7f7325e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7320107c20 0x7f732019def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:40.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.387+0000 7f7325e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7320107c20 0x7f732019def0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:41614/0 (socket says 172.21.15.39:41614) 2024-03-18T16:21:40.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.387+0000 7f7325e00640 1 -- 172.21.15.39:0/1755569039 learned_addr learned my addr 172.21.15.39:0/1755569039 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:21:40.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.387+0000 7f7326800640 1 --2- 172.21.15.39:0/1755569039 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f73201097d0 0x7f73201a54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:40.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.387+0000 7f7325400640 1 --2- 172.21.15.39:0/1755569039 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7320108e10 0x7f732019e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:40.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.387+0000 7f7325e00640 1 -- 172.21.15.39:0/1755569039 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f73201097d0 msgr2=0x7f73201a54a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:40.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.387+0000 7f7325e00640 1 --2- 172.21.15.39:0/1755569039 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f73201097d0 0x7f73201a54a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:40.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.387+0000 7f7325e00640 1 -- 172.21.15.39:0/1755569039 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7320108e10 msgr2=0x7f732019e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:40.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.387+0000 7f7325e00640 1 --2- 172.21.15.39:0/1755569039 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7320108e10 0x7f732019e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:40.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.387+0000 7f7325e00640 1 -- 172.21.15.39:0/1755569039 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73201a5b10 con 0x7f7320107c20 2024-03-18T16:21:40.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.387+0000 7f7325400640 1 --2- 172.21.15.39:0/1755569039 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7320108e10 0x7f732019e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:21:40.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.387+0000 7f7325e00640 1 --2- 172.21.15.39:0/1755569039 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7320107c20 0x7f732019def0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f731400d950 tx=0x7f731400de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:40.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.387+0000 7f731aa00640 1 -- 172.21.15.39:0/1755569039 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7314017020 con 0x7f7320107c20 2024-03-18T16:21:40.390 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.388+0000 7f731aa00640 1 -- 172.21.15.39:0/1755569039 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7314009c30 con 0x7f7320107c20 2024-03-18T16:21:40.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.388+0000 7f7327200640 1 -- 172.21.15.39:0/1755569039 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73201a5e00 con 0x7f7320107c20 2024-03-18T16:21:40.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.388+0000 7f731aa00640 1 -- 172.21.15.39:0/1755569039 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f731400f720 con 0x7f7320107c20 2024-03-18T16:21:40.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.388+0000 7f7327200640 1 -- 172.21.15.39:0/1755569039 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73201a6260 con 0x7f7320107c20 2024-03-18T16:21:40.391 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.389+0000 7f7327200640 1 -- 172.21.15.39:0/1755569039 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f732010eee0 con 0x7f7320107c20 2024-03-18T16:21:40.392 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.391+0000 7f731aa00640 1 -- 172.21.15.39:0/1755569039 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f731401a020 con 0x7f7320107c20 2024-03-18T16:21:40.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.392+0000 7f731aa00640 1 --2- 172.21.15.39:0/1755569039 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f73080a2800 0x7f73080a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:40.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.392+0000 7f731aa00640 1 -- 172.21.15.39:0/1755569039 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f73140cedd0 con 0x7f7320107c20 2024-03-18T16:21:40.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.392+0000 7f7325400640 1 --2- 172.21.15.39:0/1755569039 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f73080a2800 0x7f73080a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:40.394 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.393+0000 7f7325400640 1 --2- 172.21.15.39:0/1755569039 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f73080a2800 0x7f73080a4cc0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f732019f400 tx=0x7f7310005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:40.397 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.396+0000 7f731aa00640 1 -- 172.21.15.39:0/1755569039 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f7314021020 con 0x7f7320107c20 2024-03-18T16:21:40.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:40 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/160944757' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T16:21:40.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:40 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/3861972972' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-18T16:21:40.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:40 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/160944757' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-18T16:21:40.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:40 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/3861972972' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-18T16:21:40.750 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.749+0000 7f7327200640 1 -- 172.21.15.39:0/1755569039 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7320062480 con 0x7f7320107c20 2024-03-18T16:21:40.751 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.749+0000 7f731aa00640 1 -- 172.21.15.39:0/1755569039 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) ==== 94+0+11939 (secure 0 0 0) 0x7f7314009de0 con 0x7f7320107c20 2024-03-18T16:21:40.752 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:21:40.752 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2520226608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2520226608},{"type":"v1","addr":"172.21.15.39:6835","nonce":2520226608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/719341763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":719341763},{"type":"v1","addr":"172.21.15.39:6837","nonce":719341763}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":48,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T16:06:44.180425+0000","modified":"2024-03-18T16:20:16.222033+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24575,"mds_1":24556,"mds_2":24535,"mds_3":24524,"mds_4":14463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14445":{"gid":14445,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.39:6839/656513813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":656513813},{"type":"v1","addr":"172.21.15.39:6839","nonce":656513813}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14463":{"gid":14463,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":13,"addr":"172.21.15.39:6841/3515326042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3515326042},{"type":"v1","addr":"172.21.15.39:6841","nonce":3515326042}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6833/931350738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":931350738},{"type":"v1","addr":"172.21.15.194:6833","nonce":931350738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6833/3874209621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":3874209621},{"type":"v1","addr":"172.21.15.196:6833","nonce":3874209621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6835/188385470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":188385470},{"type":"v1","addr":"172.21.15.194:6835","nonce":188385470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6835/3949984309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":3949984309},{"type":"v1","addr":"172.21.15.196:6835","nonce":3949984309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":15,"addr":"172.21.15.194:6837/1301464246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":1301464246},{"type":"v1","addr":"172.21.15.194:6837","nonce":1301464246}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":16,"addr":"172.21.15.196:6837/895247251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":895247251},{"type":"v1","addr":"172.21.15.196:6837","nonce":895247251}]},"join_fscid":-1,"export_targets":[1,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.196:6839/2001427268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6838","nonce":2001427268},{"type":"v1","addr":"172.21.15.196:6839","nonce":2001427268}]},"join_fscid":-1,"export_targets":[4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24575":{"gid":24575,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.194:6839/274911158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":274911158},{"type":"v1","addr":"172.21.15.194:6839","nonce":274911158}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24575,"qdb_cluster":[14463,24524,24575,24556,24535]},"id":1}]} 2024-03-18T16:21:40.752 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 48 2024-03-18T16:21:40.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.752+0000 7f7327200640 1 -- 172.21.15.39:0/1755569039 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f73080a2800 msgr2=0x7f73080a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:40.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.752+0000 7f7327200640 1 --2- 172.21.15.39:0/1755569039 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f73080a2800 0x7f73080a4cc0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f732019f400 tx=0x7f7310005f90 comp rx=0 tx=0).stop 2024-03-18T16:21:40.754 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.752+0000 7f7327200640 1 -- 172.21.15.39:0/1755569039 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7320107c20 msgr2=0x7f732019def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:40.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.752+0000 7f7327200640 1 --2- 172.21.15.39:0/1755569039 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7320107c20 0x7f732019def0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f731400d950 tx=0x7f731400de10 comp rx=0 tx=0).stop 2024-03-18T16:21:40.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.752+0000 7f7327200640 1 -- 172.21.15.39:0/1755569039 shutdown_connections 2024-03-18T16:21:40.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.752+0000 7f7327200640 1 --2- 172.21.15.39:0/1755569039 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f73201097d0 0x7f73201a54a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:40.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.752+0000 7f7327200640 1 --2- 172.21.15.39:0/1755569039 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f73080a2800 0x7f73080a4cc0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:40.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.752+0000 7f7327200640 1 --2- 172.21.15.39:0/1755569039 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7320108e10 0x7f732019e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:40.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.752+0000 7f7327200640 1 --2- 172.21.15.39:0/1755569039 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7320107c20 0x7f732019def0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:40.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.752+0000 7f7327200640 1 -- 172.21.15.39:0/1755569039 >> 172.21.15.39:0/1755569039 conn(0x7f7320076bb0 msgr2=0x7f732010c0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:40.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.752+0000 7f7327200640 1 -- 172.21.15.39:0/1755569039 shutdown_connections 2024-03-18T16:21:40.755 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.753+0000 7f7327200640 1 -- 172.21.15.39:0/1755569039 wait complete. 2024-03-18T16:21:40.775 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-18T16:21:40.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.985+0000 7f15c1800640 1 -- 172.21.15.39:0/949640973 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f15bc107e00 msgr2=0x7f15bc108220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:40.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.985+0000 7f15c1800640 1 --2- 172.21.15.39:0/949640973 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f15bc107e00 0x7f15bc108220 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f15ac009900 tx=0x7f15ac02bc30 comp rx=0 tx=0).stop 2024-03-18T16:21:40.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.985+0000 7f15c1800640 1 -- 172.21.15.39:0/949640973 shutdown_connections 2024-03-18T16:21:40.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.985+0000 7f15c1800640 1 --2- 172.21.15.39:0/949640973 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f15bc1099b0 0x7f15bc110240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:40.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.985+0000 7f15c1800640 1 --2- 172.21.15.39:0/949640973 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f15bc108ff0 0x7f15bc109470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:40.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.985+0000 7f15c1800640 1 --2- 172.21.15.39:0/949640973 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f15bc107e00 0x7f15bc108220 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:40.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.985+0000 7f15c1800640 1 -- 172.21.15.39:0/949640973 >> 172.21.15.39:0/949640973 conn(0x7f15bc076dc0 msgr2=0x7f15bc0791e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:40.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.985+0000 7f15c1800640 1 -- 172.21.15.39:0/949640973 shutdown_connections 2024-03-18T16:21:40.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.985+0000 7f15c1800640 1 -- 172.21.15.39:0/949640973 wait complete. 2024-03-18T16:21:40.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.986+0000 7f15c1800640 1 Processor -- start 2024-03-18T16:21:40.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.986+0000 7f15c1800640 1 -- start start 2024-03-18T16:21:40.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.987+0000 7f15c1800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f15bc107e00 0x7f15bc19e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:40.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.987+0000 7f15c1800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f15bc108ff0 0x7f15bc19e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:40.989 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.987+0000 7f15c1800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f15bc1099b0 0x7f15bc1a5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:40.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.987+0000 7f15c1800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f15bc113a50 con 0x7f15bc108ff0 2024-03-18T16:21:40.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.987+0000 7f15c1800640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f15bc1138d0 con 0x7f15bc107e00 2024-03-18T16:21:40.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.987+0000 7f15c1800640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f15bc113bd0 con 0x7f15bc1099b0 2024-03-18T16:21:40.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.987+0000 7f15baa00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f15bc107e00 0x7f15bc19e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:40.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.987+0000 7f15bb400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f15bc1099b0 0x7f15bc1a5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:40.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.987+0000 7f15baa00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f15bc107e00 0x7f15bc19e0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:37030/0 (socket says 172.21.15.39:37030) 2024-03-18T16:21:40.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.987+0000 7f15baa00640 1 -- 172.21.15.39:0/258369497 learned_addr learned my addr 172.21.15.39:0/258369497 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:21:40.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.987+0000 7f15ba000640 1 --2- 172.21.15.39:0/258369497 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f15bc108ff0 0x7f15bc19e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:40.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.988+0000 7f15bb400640 1 -- 172.21.15.39:0/258369497 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f15bc107e00 msgr2=0x7f15bc19e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:40.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.988+0000 7f15bb400640 1 --2- 172.21.15.39:0/258369497 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f15bc107e00 0x7f15bc19e0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:40.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.988+0000 7f15bb400640 1 -- 172.21.15.39:0/258369497 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f15bc108ff0 msgr2=0x7f15bc19e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:40.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.988+0000 7f15bb400640 1 --2- 172.21.15.39:0/258369497 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f15bc108ff0 0x7f15bc19e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:40.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.988+0000 7f15bb400640 1 -- 172.21.15.39:0/258369497 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15bc1a5ce0 con 0x7f15bc1099b0 2024-03-18T16:21:40.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.988+0000 7f15baa00640 1 --2- 172.21.15.39:0/258369497 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f15bc107e00 0x7f15bc19e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:21:40.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.988+0000 7f15bb400640 1 --2- 172.21.15.39:0/258369497 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f15bc1099b0 0x7f15bc1a5670 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f15b4009fd0 tx=0x7f15b400ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:40.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.988+0000 7f15b3400640 1 -- 172.21.15.39:0/258369497 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15b400ce20 con 0x7f15bc1099b0 2024-03-18T16:21:40.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.988+0000 7f15b3400640 1 -- 172.21.15.39:0/258369497 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f15b4005020 con 0x7f15bc1099b0 2024-03-18T16:21:40.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.988+0000 7f15c1800640 1 -- 172.21.15.39:0/258369497 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15bc1a5fd0 con 0x7f15bc1099b0 2024-03-18T16:21:40.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.988+0000 7f15b3400640 1 -- 172.21.15.39:0/258369497 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15b4010720 con 0x7f15bc1099b0 2024-03-18T16:21:40.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.988+0000 7f15c1800640 1 -- 172.21.15.39:0/258369497 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15bc1a6440 con 0x7f15bc1099b0 2024-03-18T16:21:40.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.989+0000 7f15c1800640 1 -- 172.21.15.39:0/258369497 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15bc10f120 con 0x7f15bc1099b0 2024-03-18T16:21:40.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.991+0000 7f15b3400640 1 -- 172.21.15.39:0/258369497 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f15b4005540 con 0x7f15bc1099b0 2024-03-18T16:21:40.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.992+0000 7f15b3400640 1 --2- 172.21.15.39:0/258369497 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f15a00a2800 0x7f15a00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:40.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.993+0000 7f15b3400640 1 -- 172.21.15.39:0/258369497 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f15b40cf4d0 con 0x7f15bc1099b0 2024-03-18T16:21:40.994 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.993+0000 7f15baa00640 1 --2- 172.21.15.39:0/258369497 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f15a00a2800 0x7f15a00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:40.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.993+0000 7f15baa00640 1 --2- 172.21.15.39:0/258369497 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f15a00a2800 0x7f15a00a4cc0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f15ac02c8e0 tx=0x7f15ac005b50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:40.998 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:40.996+0000 7f15b3400640 1 -- 172.21.15.39:0/258369497 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f15b409c1e0 con 0x7f15bc1099b0 2024-03-18T16:21:41.243 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.242+0000 7f15c1800640 1 -- 172.21.15.39:0/258369497 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f15bc02d1b0 con 0x7f15bc1099b0 2024-03-18T16:21:41.245 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.243+0000 7f15b3400640 1 -- 172.21.15.39:0/258369497 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) ==== 94+0+11939 (secure 0 0 0) 0x7f15b409bae0 con 0x7f15bc1099b0 2024-03-18T16:21:41.245 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:21:41.245 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":48,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2520226608","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2520226608},{"type":"v1","addr":"172.21.15.39:6835","nonce":2520226608}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14391,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6837/719341763","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6836","nonce":719341763},{"type":"v1","addr":"172.21.15.39:6837","nonce":719341763}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":48,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-18T16:06:44.180425+0000","modified":"2024-03-18T16:20:16.222033+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24575,"mds_1":24556,"mds_2":24535,"mds_3":24524,"mds_4":14463},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14445":{"gid":14445,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.39:6839/656513813","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6838","nonce":656513813},{"type":"v1","addr":"172.21.15.39:6839","nonce":656513813}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_14463":{"gid":14463,"name":"a","rank":4,"incarnation":25,"state":"up:active","state_seq":13,"addr":"172.21.15.39:6841/3515326042","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6840","nonce":3515326042},{"type":"v1","addr":"172.21.15.39:6841","nonce":3515326042}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24446":{"gid":24446,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6833/931350738","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":931350738},{"type":"v1","addr":"172.21.15.194:6833","nonce":931350738}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24463":{"gid":24463,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6833/3874209621","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":3874209621},{"type":"v1","addr":"172.21.15.196:6833","nonce":3874209621}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24494":{"gid":24494,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6835/188385470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":188385470},{"type":"v1","addr":"172.21.15.194:6835","nonce":188385470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.196:6835/3949984309","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":3949984309},{"type":"v1","addr":"172.21.15.196:6835","nonce":3949984309}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24524":{"gid":24524,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":15,"addr":"172.21.15.194:6837/1301464246","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":1301464246},{"type":"v1","addr":"172.21.15.194:6837","nonce":1301464246}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":16,"addr":"172.21.15.196:6837/895247251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":895247251},{"type":"v1","addr":"172.21.15.196:6837","nonce":895247251}]},"join_fscid":-1,"export_targets":[1,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.196:6839/2001427268","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6838","nonce":2001427268},{"type":"v1","addr":"172.21.15.196:6839","nonce":2001427268}]},"join_fscid":-1,"export_targets":[4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24575":{"gid":24575,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.194:6839/274911158","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":274911158},{"type":"v1","addr":"172.21.15.194:6839","nonce":274911158}]},"join_fscid":-1,"export_targets":[1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24575,"qdb_cluster":[14463,24524,24575,24556,24535]},"id":1}]} 2024-03-18T16:21:41.245 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 48 2024-03-18T16:21:41.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.245+0000 7f15c1800640 1 -- 172.21.15.39:0/258369497 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f15a00a2800 msgr2=0x7f15a00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:41.247 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.245+0000 7f15c1800640 1 --2- 172.21.15.39:0/258369497 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f15a00a2800 0x7f15a00a4cc0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f15ac02c8e0 tx=0x7f15ac005b50 comp rx=0 tx=0).stop 2024-03-18T16:21:41.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.245+0000 7f15c1800640 1 -- 172.21.15.39:0/258369497 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f15bc1099b0 msgr2=0x7f15bc1a5670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:41.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.245+0000 7f15c1800640 1 --2- 172.21.15.39:0/258369497 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f15bc1099b0 0x7f15bc1a5670 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f15b4009fd0 tx=0x7f15b400ef10 comp rx=0 tx=0).stop 2024-03-18T16:21:41.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.245+0000 7f15c1800640 1 -- 172.21.15.39:0/258369497 shutdown_connections 2024-03-18T16:21:41.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.245+0000 7f15c1800640 1 --2- 172.21.15.39:0/258369497 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f15a00a2800 0x7f15a00a4cc0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:41.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.245+0000 7f15c1800640 1 --2- 172.21.15.39:0/258369497 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f15bc1099b0 0x7f15bc1a5670 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:41.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.245+0000 7f15c1800640 1 --2- 172.21.15.39:0/258369497 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f15bc108ff0 0x7f15bc19e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:41.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.246+0000 7f15c1800640 1 --2- 172.21.15.39:0/258369497 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f15bc107e00 0x7f15bc19e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:41.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.246+0000 7f15c1800640 1 -- 172.21.15.39:0/258369497 >> 172.21.15.39:0/258369497 conn(0x7f15bc076dc0 msgr2=0x7f15bc10c2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:41.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.246+0000 7f15c1800640 1 -- 172.21.15.39:0/258369497 shutdown_connections 2024-03-18T16:21:41.248 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.246+0000 7f15c1800640 1 -- 172.21.15.39:0/258369497 wait complete. 2024-03-18T16:21:41.267 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-18T16:21:41.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:41 smithi039 ceph-mon[31721]: pgmap v770: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-18T16:21:41.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:41 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/1755569039' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:21:41.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:41 smithi196 ceph-mon[36307]: pgmap v770: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-18T16:21:41.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:41 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/1755569039' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:21:41.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:41 smithi194 ceph-mon[36200]: pgmap v770: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-03-18T16:21:41.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:41 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/1755569039' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:21:41.493 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.491+0000 7f025ee00640 1 -- 172.21.15.39:0/2887615053 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0258103a00 msgr2=0x7f0258103e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:41.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.491+0000 7f025ee00640 1 --2- 172.21.15.39:0/2887615053 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0258103a00 0x7f0258103e20 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f0248009900 tx=0x7f024802bc40 comp rx=0 tx=0).stop 2024-03-18T16:21:41.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.492+0000 7f025ee00640 1 -- 172.21.15.39:0/2887615053 shutdown_connections 2024-03-18T16:21:41.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.492+0000 7f025ee00640 1 --2- 172.21.15.39:0/2887615053 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f02581055b0 0x7f025810be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:41.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.492+0000 7f025ee00640 1 --2- 172.21.15.39:0/2887615053 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0258104bf0 0x7f0258105070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:41.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.492+0000 7f025ee00640 1 --2- 172.21.15.39:0/2887615053 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0258103a00 0x7f0258103e20 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:41.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.492+0000 7f025ee00640 1 -- 172.21.15.39:0/2887615053 >> 172.21.15.39:0/2887615053 conn(0x7f02580fef90 msgr2=0x7f02581013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:41.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.492+0000 7f025ee00640 1 -- 172.21.15.39:0/2887615053 shutdown_connections 2024-03-18T16:21:41.494 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.492+0000 7f025ee00640 1 -- 172.21.15.39:0/2887615053 wait complete. 2024-03-18T16:21:41.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.493+0000 7f025ee00640 1 Processor -- start 2024-03-18T16:21:41.495 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.493+0000 7f025ee00640 1 -- start start 2024-03-18T16:21:41.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.494+0000 7f025ee00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0258103a00 0x7f0258199d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:41.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.494+0000 7f025ee00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0258104bf0 0x7f025819a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:41.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.494+0000 7f025ee00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f02581055b0 0x7f02581a12a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:41.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.494+0000 7f025ee00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f025810f6a0 con 0x7f0258103a00 2024-03-18T16:21:41.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.494+0000 7f025ee00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f025810f520 con 0x7f0258104bf0 2024-03-18T16:21:41.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.494+0000 7f025ee00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f025810f820 con 0x7f02581055b0 2024-03-18T16:21:41.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.494+0000 7f0257e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0258103a00 0x7f0258199d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:41.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.494+0000 7f0257400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0258104bf0 0x7f025819a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:41.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.494+0000 7f0257400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0258104bf0 0x7f025819a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:37054/0 (socket says 172.21.15.39:37054) 2024-03-18T16:21:41.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.494+0000 7f0257400640 1 -- 172.21.15.39:0/4265079278 learned_addr learned my addr 172.21.15.39:0/4265079278 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:21:41.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.494+0000 7f025ce00640 1 --2- 172.21.15.39:0/4265079278 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f02581055b0 0x7f02581a12a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:41.497 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.494+0000 7f025ce00640 1 -- 172.21.15.39:0/4265079278 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0258104bf0 msgr2=0x7f025819a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:41.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.494+0000 7f025ce00640 1 --2- 172.21.15.39:0/4265079278 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0258104bf0 0x7f025819a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:41.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.494+0000 7f025ce00640 1 -- 172.21.15.39:0/4265079278 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0258103a00 msgr2=0x7f0258199d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:41.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.494+0000 7f025ce00640 1 --2- 172.21.15.39:0/4265079278 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0258103a00 0x7f0258199d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:41.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.494+0000 7f025ce00640 1 -- 172.21.15.39:0/4265079278 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02581a1940 con 0x7f02581055b0 2024-03-18T16:21:41.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.495+0000 7f0257400640 1 --2- 172.21.15.39:0/4265079278 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0258104bf0 0x7f025819a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:21:41.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.495+0000 7f025ce00640 1 --2- 172.21.15.39:0/4265079278 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f02581055b0 0x7f02581a12a0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f025000ec50 tx=0x7f025000c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:41.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.495+0000 7f0254c00640 1 -- 172.21.15.39:0/4265079278 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0250003b70 con 0x7f02581055b0 2024-03-18T16:21:41.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.495+0000 7f0254c00640 1 -- 172.21.15.39:0/4265079278 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0250003d10 con 0x7f02581055b0 2024-03-18T16:21:41.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.495+0000 7f025ee00640 1 -- 172.21.15.39:0/4265079278 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02581a1c30 con 0x7f02581055b0 2024-03-18T16:21:41.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.495+0000 7f0254c00640 1 -- 172.21.15.39:0/4265079278 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0250010890 con 0x7f02581055b0 2024-03-18T16:21:41.498 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.495+0000 7f025ee00640 1 -- 172.21.15.39:0/4265079278 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02581a2100 con 0x7f02581055b0 2024-03-18T16:21:41.499 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.497+0000 7f025ee00640 1 -- 172.21.15.39:0/4265079278 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f025810ada0 con 0x7f02581055b0 2024-03-18T16:21:41.500 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.498+0000 7f0254c00640 1 -- 172.21.15.39:0/4265079278 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f0250003fc0 con 0x7f02581055b0 2024-03-18T16:21:41.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.499+0000 7f0254c00640 1 --2- 172.21.15.39:0/4265079278 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f023c0a2800 0x7f023c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:41.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.500+0000 7f0254c00640 1 -- 172.21.15.39:0/4265079278 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7f02500cf400 con 0x7f02581055b0 2024-03-18T16:21:41.501 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.500+0000 7f0257e00640 1 --2- 172.21.15.39:0/4265079278 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f023c0a2800 0x7f023c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:41.502 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.501+0000 7f0257e00640 1 --2- 172.21.15.39:0/4265079278 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f023c0a2800 0x7f023c0a4cc0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f024802c180 tx=0x7f024802cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:41.509 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.508+0000 7f0254c00640 1 -- 172.21.15.39:0/4265079278 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f025009c110 con 0x7f02581055b0 2024-03-18T16:21:41.736 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.735+0000 7f025ee00640 1 -- 172.21.15.39:0/4265079278 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f02580623e0 con 0x7f02581055b0 2024-03-18T16:21:41.739 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.737+0000 7f0254c00640 1 -- 172.21.15.39:0/4265079278 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24762 (secure 0 0 0) 0x7f025009ba10 con 0x7f02581055b0 2024-03-18T16:21:41.740 INFO:teuthology.orchestra.run.smithi039.stdout: 2024-03-18T16:21:41.740 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":93,"fsid":"1892d8fc-e540-11ee-95c9-87774f69a715","created":"2024-03-18T15:57:36.507008+0000","modified":"2024-03-18T16:08:47.874967+0000","last_up_change":"2024-03-18T16:05:45.250629+0000","last_in_change":"2024-03-18T16:05:24.841337+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-18T16:01:30.048247+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-18T16:06:42.191791+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-18T16:06:43.670325+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-18T16:06:47.523388+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"771e0558-be62-4436-94b0-6364751911c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6803","nonce":4090743635}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6805","nonce":4090743635}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6809","nonce":4090743635}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":4090743635},{"type":"v1","addr":"172.21.15.39:6807","nonce":4090743635}]},"public_addr":"172.21.15.39:6803/4090743635","cluster_addr":"172.21.15.39:6805/4090743635","heartbeat_back_addr":"172.21.15.39:6809/4090743635","heartbeat_front_addr":"172.21.15.39:6807/4090743635","state":["exists","up"]},{"osd":1,"uuid":"0430eee1-cf34-4277-b255-c7d9f07ad19a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6801","nonce":2800329805}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6803","nonce":2800329805}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6807","nonce":2800329805}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":2800329805},{"type":"v1","addr":"172.21.15.194:6805","nonce":2800329805}]},"public_addr":"172.21.15.194:6801/2800329805","cluster_addr":"172.21.15.194:6803/2800329805","heartbeat_back_addr":"172.21.15.194:6807/2800329805","heartbeat_front_addr":"172.21.15.194:6805/2800329805","state":["exists","up"]},{"osd":2,"uuid":"3a718da4-2511-4ec1-b1a8-eafd6cce0df0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6801","nonce":3935874719}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6803","nonce":3935874719}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6807","nonce":3935874719}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":3935874719},{"type":"v1","addr":"172.21.15.196:6805","nonce":3935874719}]},"public_addr":"172.21.15.196:6801/3935874719","cluster_addr":"172.21.15.196:6803/3935874719","heartbeat_back_addr":"172.21.15.196:6807/3935874719","heartbeat_front_addr":"172.21.15.196:6805/3935874719","state":["exists","up"]},{"osd":3,"uuid":"6f9c37f9-0ab9-4038-8919-afd4b39c4165","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6811","nonce":893345939}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6813","nonce":893345939}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6817","nonce":893345939}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":893345939},{"type":"v1","addr":"172.21.15.39:6815","nonce":893345939}]},"public_addr":"172.21.15.39:6811/893345939","cluster_addr":"172.21.15.39:6813/893345939","heartbeat_back_addr":"172.21.15.39:6817/893345939","heartbeat_front_addr":"172.21.15.39:6815/893345939","state":["exists","up"]},{"osd":4,"uuid":"e795bb0e-4791-4537-acda-1a321e19bc9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6809","nonce":800423088}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6811","nonce":800423088}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6815","nonce":800423088}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":800423088},{"type":"v1","addr":"172.21.15.194:6813","nonce":800423088}]},"public_addr":"172.21.15.194:6809/800423088","cluster_addr":"172.21.15.194:6811/800423088","heartbeat_back_addr":"172.21.15.194:6815/800423088","heartbeat_front_addr":"172.21.15.194:6813/800423088","state":["exists","up"]},{"osd":5,"uuid":"026e175e-6acb-4b4c-b235-8ccee8ae1149","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6809","nonce":2746224525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6811","nonce":2746224525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6815","nonce":2746224525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":2746224525},{"type":"v1","addr":"172.21.15.196:6813","nonce":2746224525}]},"public_addr":"172.21.15.196:6809/2746224525","cluster_addr":"172.21.15.196:6811/2746224525","heartbeat_back_addr":"172.21.15.196:6815/2746224525","heartbeat_front_addr":"172.21.15.196:6813/2746224525","state":["exists","up"]},{"osd":6,"uuid":"c1b183ff-383f-4c5c-9075-f76bab1c0bb3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6819","nonce":1213371331}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6821","nonce":1213371331}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6825","nonce":1213371331}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":1213371331},{"type":"v1","addr":"172.21.15.39:6823","nonce":1213371331}]},"public_addr":"172.21.15.39:6819/1213371331","cluster_addr":"172.21.15.39:6821/1213371331","heartbeat_back_addr":"172.21.15.39:6825/1213371331","heartbeat_front_addr":"172.21.15.39:6823/1213371331","state":["exists","up"]},{"osd":7,"uuid":"612f8697-957d-4ef4-a87f-241b36f9d13d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6817","nonce":3299162444}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6819","nonce":3299162444}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6823","nonce":3299162444}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":3299162444},{"type":"v1","addr":"172.21.15.194:6821","nonce":3299162444}]},"public_addr":"172.21.15.194:6817/3299162444","cluster_addr":"172.21.15.194:6819/3299162444","heartbeat_back_addr":"172.21.15.194:6823/3299162444","heartbeat_front_addr":"172.21.15.194:6821/3299162444","state":["exists","up"]},{"osd":8,"uuid":"836a9107-4d4f-4594-b8dc-0301e64f59d9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6817","nonce":2456143145}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6819","nonce":2456143145}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6823","nonce":2456143145}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":2456143145},{"type":"v1","addr":"172.21.15.196:6821","nonce":2456143145}]},"public_addr":"172.21.15.196:6817/2456143145","cluster_addr":"172.21.15.196:6819/2456143145","heartbeat_back_addr":"172.21.15.196:6823/2456143145","heartbeat_front_addr":"172.21.15.196:6821/2456143145","state":["exists","up"]},{"osd":9,"uuid":"4f30b960-12e3-4b16-bd28-fcc988139b58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6827","nonce":1848634433}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6829","nonce":1848634433}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6833","nonce":1848634433}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":1848634433},{"type":"v1","addr":"172.21.15.39:6831","nonce":1848634433}]},"public_addr":"172.21.15.39:6827/1848634433","cluster_addr":"172.21.15.39:6829/1848634433","heartbeat_back_addr":"172.21.15.39:6833/1848634433","heartbeat_front_addr":"172.21.15.39:6831/1848634433","state":["exists","up"]},{"osd":10,"uuid":"620f90e8-23e6-4bc0-8269-7219b9e1aebd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6825","nonce":3725064809}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6827","nonce":3725064809}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6831","nonce":3725064809}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":3725064809},{"type":"v1","addr":"172.21.15.194:6829","nonce":3725064809}]},"public_addr":"172.21.15.194:6825/3725064809","cluster_addr":"172.21.15.194:6827/3725064809","heartbeat_back_addr":"172.21.15.194:6831/3725064809","heartbeat_front_addr":"172.21.15.194:6829/3725064809","state":["exists","up"]},{"osd":11,"uuid":"867a2535-8235-4ceb-a698-eb86869d6d20","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6825","nonce":2501212422}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6827","nonce":2501212422}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6831","nonce":2501212422}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":2501212422},{"type":"v1","addr":"172.21.15.196:6829","nonce":2501212422}]},"public_addr":"172.21.15.196:6825/2501212422","cluster_addr":"172.21.15.196:6827/2501212422","heartbeat_back_addr":"172.21.15.196:6831/2501212422","heartbeat_front_addr":"172.21.15.196:6829/2501212422","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:00:36.931566+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:01:02.614024+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:01:26.193265+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:01:58.308833+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:02:25.049546+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:02:49.748750+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:03:23.582153+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:03:50.447321+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:04:15.812129+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:04:49.998047+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:05:17.087924+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-18T16:05:43.167637+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.f","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.c","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.1d","mappings":[{"from":4,"to":7}]},{"pgid":"3.20","mappings":[{"from":5,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"4.5","mappings":[{"from":9,"to":10}]},{"pgid":"4.a","mappings":[{"from":5,"to":8}]},{"pgid":"4.f","mappings":[{"from":9,"to":8}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.39:0/3482086140":"2024-03-19T15:58:45.340294+0000","172.21.15.39:0/3844306284":"2024-03-19T15:58:45.340294+0000","172.21.15.39:6801/1810238546":"2024-03-19T15:58:45.340294+0000","172.21.15.39:6800/1810238546":"2024-03-19T15:58:45.340294+0000","172.21.15.39:0/2528397488":"2024-03-19T15:58:10.264798+0000","172.21.15.39:0/2240981254":"2024-03-19T15:58:10.264798+0000","172.21.15.39:6800/1217503189":"2024-03-19T15:58:10.264798+0000","172.21.15.39:0/3754876370":"2024-03-19T15:58:10.264798+0000","172.21.15.39:6801/1217503189":"2024-03-19T15:58:10.264798+0000","172.21.15.39:0/1874637628":"2024-03-19T15:58:45.340294+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-18T16:21:41.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.740+0000 7f025ee00640 1 -- 172.21.15.39:0/4265079278 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f023c0a2800 msgr2=0x7f023c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:41.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.740+0000 7f025ee00640 1 --2- 172.21.15.39:0/4265079278 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f023c0a2800 0x7f023c0a4cc0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f024802c180 tx=0x7f024802cf90 comp rx=0 tx=0).stop 2024-03-18T16:21:41.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.740+0000 7f025ee00640 1 -- 172.21.15.39:0/4265079278 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f02581055b0 msgr2=0x7f02581a12a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:41.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.740+0000 7f025ee00640 1 --2- 172.21.15.39:0/4265079278 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f02581055b0 0x7f02581a12a0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f025000ec50 tx=0x7f025000c6a0 comp rx=0 tx=0).stop 2024-03-18T16:21:41.742 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.740+0000 7f025ee00640 1 -- 172.21.15.39:0/4265079278 shutdown_connections 2024-03-18T16:21:41.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.740+0000 7f025ee00640 1 --2- 172.21.15.39:0/4265079278 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0258104bf0 0x7f025819a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:41.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.740+0000 7f025ee00640 1 --2- 172.21.15.39:0/4265079278 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f023c0a2800 0x7f023c0a4cc0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:41.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.740+0000 7f025ee00640 1 --2- 172.21.15.39:0/4265079278 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f02581055b0 0x7f02581a12a0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:41.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.740+0000 7f025ee00640 1 --2- 172.21.15.39:0/4265079278 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0258103a00 0x7f0258199d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:41.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.740+0000 7f025ee00640 1 -- 172.21.15.39:0/4265079278 >> 172.21.15.39:0/4265079278 conn(0x7f02580fef90 msgr2=0x7f0258107f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:41.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.741+0000 7f025ee00640 1 -- 172.21.15.39:0/4265079278 shutdown_connections 2024-03-18T16:21:41.743 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.741+0000 7f025ee00640 1 -- 172.21.15.39:0/4265079278 wait complete. 2024-03-18T16:21:41.762 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-18T16:21:41.987 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.985+0000 7fb044a00640 1 -- 172.21.15.39:0/2194979790 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb040074200 msgr2=0x7fb040074680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:41.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.985+0000 7fb044a00640 1 --2- 172.21.15.39:0/2194979790 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb040074200 0x7fb040074680 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fb02c009900 tx=0x7fb02c02bc50 comp rx=0 tx=0).stop 2024-03-18T16:21:41.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.985+0000 7fb044a00640 1 -- 172.21.15.39:0/2194979790 shutdown_connections 2024-03-18T16:21:41.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.985+0000 7fb044a00640 1 --2- 172.21.15.39:0/2194979790 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb0401096a0 0x7fb04010bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:41.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.985+0000 7fb044a00640 1 --2- 172.21.15.39:0/2194979790 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb040074200 0x7fb040074680 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:41.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.985+0000 7fb044a00640 1 --2- 172.21.15.39:0/2194979790 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb0400757e0 0x7fb040073c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:41.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.985+0000 7fb044a00640 1 -- 172.21.15.39:0/2194979790 >> 172.21.15.39:0/2194979790 conn(0x7fb0400fcd80 msgr2=0x7fb0400ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:41.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.986+0000 7fb044a00640 1 -- 172.21.15.39:0/2194979790 shutdown_connections 2024-03-18T16:21:41.988 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.986+0000 7fb044a00640 1 -- 172.21.15.39:0/2194979790 wait complete. 2024-03-18T16:21:41.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.987+0000 7fb044a00640 1 Processor -- start 2024-03-18T16:21:41.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.987+0000 7fb044a00640 1 -- start start 2024-03-18T16:21:41.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.987+0000 7fb044a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb040074200 0x7fb040199d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:41.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.987+0000 7fb044a00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb0400757e0 0x7fb04019a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:41.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.987+0000 7fb044a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb0401096a0 0x7fb0401a1270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:41.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.987+0000 7fb044a00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fb04010f680 con 0x7fb040074200 2024-03-18T16:21:41.990 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.987+0000 7fb044a00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fb04010f500 con 0x7fb0401096a0 2024-03-18T16:21:41.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.987+0000 7fb044a00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fb04010f800 con 0x7fb0400757e0 2024-03-18T16:21:41.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.987+0000 7fb03d400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb0400757e0 0x7fb04019a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:41.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.987+0000 7fb03de00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb040074200 0x7fb040199d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:41.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.987+0000 7fb03d400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb0400757e0 0x7fb04019a240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:41674/0 (socket says 172.21.15.39:41674) 2024-03-18T16:21:41.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.987+0000 7fb03d400640 1 -- 172.21.15.39:0/2363212428 learned_addr learned my addr 172.21.15.39:0/2363212428 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:21:41.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.988+0000 7fb03e800640 1 --2- 172.21.15.39:0/2363212428 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb0401096a0 0x7fb0401a1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:41.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.988+0000 7fb03d400640 1 -- 172.21.15.39:0/2363212428 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb0401096a0 msgr2=0x7fb0401a1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:41.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.988+0000 7fb03d400640 1 --2- 172.21.15.39:0/2363212428 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb0401096a0 0x7fb0401a1270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:41.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.988+0000 7fb03d400640 1 -- 172.21.15.39:0/2363212428 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb040074200 msgr2=0x7fb040199d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:41.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.988+0000 7fb03d400640 1 --2- 172.21.15.39:0/2363212428 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb040074200 0x7fb040199d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:41.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.988+0000 7fb03d400640 1 -- 172.21.15.39:0/2363212428 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0401a1910 con 0x7fb0400757e0 2024-03-18T16:21:41.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.988+0000 7fb03de00640 1 --2- 172.21.15.39:0/2363212428 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb040074200 0x7fb040199d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:21:41.991 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.988+0000 7fb03d400640 1 --2- 172.21.15.39:0/2363212428 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb0400757e0 0x7fb04019a240 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fb02c02c260 tx=0x7fb02c02c670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:41.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.989+0000 7fb036a00640 1 -- 172.21.15.39:0/2363212428 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb02c041070 con 0x7fb0400757e0 2024-03-18T16:21:41.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.989+0000 7fb036a00640 1 -- 172.21.15.39:0/2363212428 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb02c004450 con 0x7fb0400757e0 2024-03-18T16:21:41.992 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.989+0000 7fb036a00640 1 -- 172.21.15.39:0/2363212428 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb02c034440 con 0x7fb0400757e0 2024-03-18T16:21:41.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.989+0000 7fb044a00640 1 -- 172.21.15.39:0/2363212428 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0401a1ba0 con 0x7fb0400757e0 2024-03-18T16:21:41.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.989+0000 7fb044a00640 1 -- 172.21.15.39:0/2363212428 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0401a20a0 con 0x7fb0400757e0 2024-03-18T16:21:41.993 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.990+0000 7fb044a00640 1 -- 172.21.15.39:0/2363212428 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb040108420 con 0x7fb0400757e0 2024-03-18T16:21:41.995 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.994+0000 7fb036a00640 1 -- 172.21.15.39:0/2363212428 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fb02c044070 con 0x7fb0400757e0 2024-03-18T16:21:41.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.995+0000 7fb036a00640 1 --2- 172.21.15.39:0/2363212428 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb0240a2880 0x7fb0240a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:41.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.995+0000 7fb036a00640 1 -- 172.21.15.39:0/2363212428 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9150+0+0 (secure 0 0 0) 0x7fb02c0f08f0 con 0x7fb0400757e0 2024-03-18T16:21:41.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.995+0000 7fb03de00640 1 --2- 172.21.15.39:0/2363212428 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb0240a2880 0x7fb0240a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:41.997 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:41.996+0000 7fb03de00640 1 --2- 172.21.15.39:0/2363212428 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb0240a2880 0x7fb0240a4d40 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fb04019b1d0 tx=0x7fb030005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:42.005 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:42.003+0000 7fb036a00640 1 -- 172.21.15.39:0/2363212428 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb02c0bcfb0 con 0x7fb0400757e0 2024-03-18T16:21:42.302 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:42.300+0000 7fb044a00640 1 -- 172.21.15.39:0/2363212428 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fb040062480 con 0x7fb0400757e0 2024-03-18T16:21:42.312 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:42.310+0000 7fb036a00640 1 -- 172.21.15.39:0/2363212428 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v49) ==== 161+0+0 (secure 0 0 0) 0x7fb02c0c1050 con 0x7fb0400757e0 2024-03-18T16:21:42.315 INFO:teuthology.orchestra.run.smithi039.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-18T16:21:42.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:42.314+0000 7fb044a00640 1 -- 172.21.15.39:0/2363212428 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb0240a2880 msgr2=0x7fb0240a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:42.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:42.314+0000 7fb044a00640 1 --2- 172.21.15.39:0/2363212428 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb0240a2880 0x7fb0240a4d40 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fb04019b1d0 tx=0x7fb030005f90 comp rx=0 tx=0).stop 2024-03-18T16:21:42.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:42.314+0000 7fb044a00640 1 -- 172.21.15.39:0/2363212428 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb0400757e0 msgr2=0x7fb04019a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:42.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:42.314+0000 7fb044a00640 1 --2- 172.21.15.39:0/2363212428 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb0400757e0 0x7fb04019a240 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fb02c02c260 tx=0x7fb02c02c670 comp rx=0 tx=0).stop 2024-03-18T16:21:42.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:42.314+0000 7fb044a00640 1 -- 172.21.15.39:0/2363212428 shutdown_connections 2024-03-18T16:21:42.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:42.314+0000 7fb044a00640 1 --2- 172.21.15.39:0/2363212428 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb0401096a0 0x7fb0401a1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:42.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:42.314+0000 7fb044a00640 1 --2- 172.21.15.39:0/2363212428 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb0240a2880 0x7fb0240a4d40 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:42.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:42.314+0000 7fb044a00640 1 --2- 172.21.15.39:0/2363212428 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb0400757e0 0x7fb04019a240 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:42.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:42.314+0000 7fb044a00640 1 --2- 172.21.15.39:0/2363212428 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb040074200 0x7fb040199d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:42.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:42.314+0000 7fb044a00640 1 -- 172.21.15.39:0/2363212428 >> 172.21.15.39:0/2363212428 conn(0x7fb0400fcd80 msgr2=0x7fb0401055f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:42.316 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:42.314+0000 7fb044a00640 1 -- 172.21.15.39:0/2363212428 shutdown_connections 2024-03-18T16:21:42.317 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:42.314+0000 7fb044a00640 1 -- 172.21.15.39:0/2363212428 wait complete. 2024-03-18T16:21:42.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:42 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/258369497' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:21:42.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:42 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/4265079278' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:21:42.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:42 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mon-a[31716]: 2024-03-18T16:21:42.302+0000 7fc86c400700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-18T16:21:42.343 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-18T16:21:42.377 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:42 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/258369497' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:21:42.377 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:42 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/4265079278' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:21:42.378 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:42 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/258369497' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-18T16:21:42.378 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:42 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/4265079278' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-18T16:21:42.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:42.554+0000 7fb114800640 1 -- 172.21.15.39:0/1469472865 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb10c077d80 msgr2=0x7fb10c078620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:42.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:42.554+0000 7fb114800640 1 --2- 172.21.15.39:0/1469472865 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb10c077d80 0x7fb10c078620 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fb108009ca0 tx=0x7fb10802c100 comp rx=0 tx=0).stop 2024-03-18T16:21:42.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:42.554+0000 7fb114800640 1 -- 172.21.15.39:0/1469472865 shutdown_connections 2024-03-18T16:21:42.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:42.554+0000 7fb114800640 1 --2- 172.21.15.39:0/1469472865 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb10c0773e0 0x7fb10c077840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:42.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:42.554+0000 7fb114800640 1 --2- 172.21.15.39:0/1469472865 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb10c077d80 0x7fb10c078620 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:42.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:42.554+0000 7fb114800640 1 --2- 172.21.15.39:0/1469472865 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb10c0761a0 0x7fb10c0765c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:42.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:42.554+0000 7fb114800640 1 -- 172.21.15.39:0/1469472865 >> 172.21.15.39:0/1469472865 conn(0x7fb10c0fef90 msgr2=0x7fb10c1013d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:42.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:42.554+0000 7fb114800640 1 -- 172.21.15.39:0/1469472865 shutdown_connections 2024-03-18T16:21:42.556 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:42.554+0000 7fb114800640 1 -- 172.21.15.39:0/1469472865 wait complete. 2024-03-18T16:21:42.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:42.555+0000 7fb114800640 1 Processor -- start 2024-03-18T16:21:42.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:42.555+0000 7fb114800640 1 -- start start 2024-03-18T16:21:42.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:42.555+0000 7fb114800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb10c0761a0 0x7fb10c19e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:42.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:42.555+0000 7fb114800640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb10c0773e0 0x7fb10c19e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:42.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:42.555+0000 7fb114800640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb10c077d80 0x7fb10c1a5600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:42.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:42.555+0000 7fb114800640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fb10c113ab0 con 0x7fb10c0773e0 2024-03-18T16:21:42.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:42.555+0000 7fb114800640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fb10c113930 con 0x7fb10c0761a0 2024-03-18T16:21:42.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:42.555+0000 7fb114800640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fb10c113c30 con 0x7fb10c077d80 2024-03-18T16:21:42.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:42.556+0000 7fb111e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb10c0761a0 0x7fb10c19e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:42.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:42.556+0000 7fb111e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb10c0761a0 0x7fb10c19e050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:37150/0 (socket says 172.21.15.39:37150) 2024-03-18T16:21:42.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:42.556+0000 7fb111e00640 1 -- 172.21.15.39:0/78832104 learned_addr learned my addr 172.21.15.39:0/78832104 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:21:42.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:42.556+0000 7fb111400640 1 --2- 172.21.15.39:0/78832104 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb10c0773e0 0x7fb10c19e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:42.559 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:42.556+0000 7fb112800640 1 --2- 172.21.15.39:0/78832104 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb10c077d80 0x7fb10c1a5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:42.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:42.556+0000 7fb111e00640 1 -- 172.21.15.39:0/78832104 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb10c077d80 msgr2=0x7fb10c1a5600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:42.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:42.556+0000 7fb111e00640 1 --2- 172.21.15.39:0/78832104 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb10c077d80 0x7fb10c1a5600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:42.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:42.556+0000 7fb111e00640 1 -- 172.21.15.39:0/78832104 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb10c0773e0 msgr2=0x7fb10c19e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:42.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:42.556+0000 7fb111e00640 1 --2- 172.21.15.39:0/78832104 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb10c0773e0 0x7fb10c19e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:42.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:42.556+0000 7fb111e00640 1 -- 172.21.15.39:0/78832104 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb10c1a5c70 con 0x7fb10c0761a0 2024-03-18T16:21:42.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:42.556+0000 7fb112800640 1 --2- 172.21.15.39:0/78832104 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb10c077d80 0x7fb10c1a5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-18T16:21:42.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:42.556+0000 7fb111e00640 1 --2- 172.21.15.39:0/78832104 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb10c0761a0 0x7fb10c19e050 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fb10000e9f0 tx=0x7fb10000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:42.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:42.557+0000 7fb106a00640 1 -- 172.21.15.39:0/78832104 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb10000cd80 con 0x7fb10c0761a0 2024-03-18T16:21:42.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:42.557+0000 7fb114800640 1 -- 172.21.15.39:0/78832104 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb10c1a5f60 con 0x7fb10c0761a0 2024-03-18T16:21:42.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:42.557+0000 7fb106a00640 1 -- 172.21.15.39:0/78832104 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb100010910 con 0x7fb10c0761a0 2024-03-18T16:21:42.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:42.557+0000 7fb106a00640 1 -- 172.21.15.39:0/78832104 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1000188f0 con 0x7fb10c0761a0 2024-03-18T16:21:42.560 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:42.557+0000 7fb114800640 1 -- 172.21.15.39:0/78832104 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb10c1a6420 con 0x7fb10c0761a0 2024-03-18T16:21:42.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:42.558+0000 7fb114800640 1 -- 172.21.15.39:0/78832104 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb10c110390 con 0x7fb10c0761a0 2024-03-18T16:21:42.561 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:42.560+0000 7fb106a00640 1 -- 172.21.15.39:0/78832104 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fb100010ac0 con 0x7fb10c0761a0 2024-03-18T16:21:42.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:42.561+0000 7fb106a00640 1 --2- 172.21.15.39:0/78832104 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb0f40a2880 0x7fb0f40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:42.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:42.561+0000 7fb106a00640 1 -- 172.21.15.39:0/78832104 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9580+0+0 (secure 0 0 0) 0x7fb1000cf3d0 con 0x7fb10c0761a0 2024-03-18T16:21:42.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:42.561+0000 7fb111400640 1 --2- 172.21.15.39:0/78832104 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb0f40a2880 0x7fb0f40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:42.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:42.562+0000 7fb111400640 1 --2- 172.21.15.39:0/78832104 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb0f40a2880 0x7fb0f40a4d40 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fb10c19f560 tx=0x7fb0fc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:42.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:42.565+0000 7fb106a00640 1 -- 172.21.15.39:0/78832104 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fb10009bed0 con 0x7fb10c0761a0 2024-03-18T16:21:42.728 INFO:journalctl@ceph.mds.c.smithi196.stdout:Mar 18 16:21:42 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-c[66685]: ignoring --setuser ceph since I am not root 2024-03-18T16:21:42.729 INFO:journalctl@ceph.mds.c.smithi196.stdout:Mar 18 16:21:42 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-c[66685]: ignoring --setgroup ceph since I am not root 2024-03-18T16:21:42.729 INFO:journalctl@ceph.mds.c.smithi196.stdout:Mar 18 16:21:42 smithi196 ceph-mds[66689]: -- 172.21.15.196:0/686562804 <== mon.2 v2:172.21.15.194:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5624caec1a40 con 0x5624cae94800 2024-03-18T16:21:42.729 INFO:journalctl@ceph.mds.c.smithi196.stdout:Mar 18 16:21:42 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-c[66685]: starting mds.c at 2024-03-18T16:21:42.729 INFO:journalctl@ceph.mds.i.smithi196.stdout:Mar 18 16:21:42 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-i[66987]: ignoring --setuser ceph since I am not root 2024-03-18T16:21:42.729 INFO:journalctl@ceph.mds.i.smithi196.stdout:Mar 18 16:21:42 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-i[66987]: ignoring --setgroup ceph since I am not root 2024-03-18T16:21:42.729 INFO:journalctl@ceph.mds.i.smithi196.stdout:Mar 18 16:21:42 smithi196 ceph-mds[66991]: -- 172.21.15.196:0/4035425044 <== mon.1 v2:172.21.15.196:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b1ab7bfa40 con 0x55b1ab792c00 2024-03-18T16:21:42.729 INFO:journalctl@ceph.mds.i.smithi196.stdout:Mar 18 16:21:42 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-i[66987]: starting mds.i at 2024-03-18T16:21:42.729 INFO:journalctl@ceph.mds.f.smithi196.stdout:Mar 18 16:21:42 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-f[66376]: ignoring --setuser ceph since I am not root 2024-03-18T16:21:42.730 INFO:journalctl@ceph.mds.f.smithi196.stdout:Mar 18 16:21:42 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-f[66376]: ignoring --setgroup ceph since I am not root 2024-03-18T16:21:42.730 INFO:journalctl@ceph.mds.f.smithi196.stdout:Mar 18 16:21:42 smithi196 ceph-mds[66380]: -- 172.21.15.196:0/1927832664 <== mon.1 v2:172.21.15.196:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563716267a40 con 0x56371623a800 2024-03-18T16:21:42.730 INFO:journalctl@ceph.mds.f.smithi196.stdout:Mar 18 16:21:42 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-f[66376]: starting mds.f at 2024-03-18T16:21:42.740 INFO:journalctl@ceph.mds.e.smithi194.stdout:Mar 18 16:21:42 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-e[66675]: ignoring --setuser ceph since I am not root 2024-03-18T16:21:42.740 INFO:journalctl@ceph.mds.e.smithi194.stdout:Mar 18 16:21:42 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-e[66675]: ignoring --setgroup ceph since I am not root 2024-03-18T16:21:42.741 INFO:journalctl@ceph.mds.e.smithi194.stdout:Mar 18 16:21:42 smithi194 ceph-mds[66679]: -- 172.21.15.194:0/2111189293 <== mon.2 v2:172.21.15.194:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55adb43ffa40 con 0x55adb43d3000 2024-03-18T16:21:42.741 INFO:journalctl@ceph.mds.e.smithi194.stdout:Mar 18 16:21:42 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-e[66675]: starting mds.e at 2024-03-18T16:21:42.741 INFO:journalctl@ceph.mds.k.smithi194.stdout:Mar 18 16:21:42 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-k[66356]: ignoring --setuser ceph since I am not root 2024-03-18T16:21:42.741 INFO:journalctl@ceph.mds.k.smithi194.stdout:Mar 18 16:21:42 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-k[66356]: ignoring --setgroup ceph since I am not root 2024-03-18T16:21:42.741 INFO:journalctl@ceph.mds.k.smithi194.stdout:Mar 18 16:21:42 smithi194 ceph-mds[66360]: -- 172.21.15.194:0/3747737448 <== mon.2 v2:172.21.15.194:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b93e485a40 con 0x55b93e458800 2024-03-18T16:21:42.741 INFO:journalctl@ceph.mds.k.smithi194.stdout:Mar 18 16:21:42 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-k[66356]: starting mds.k at 2024-03-18T16:21:42.741 INFO:journalctl@ceph.mds.b.smithi194.stdout:Mar 18 16:21:42 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-b[67287]: ignoring --setuser ceph since I am not root 2024-03-18T16:21:42.742 INFO:journalctl@ceph.mds.b.smithi194.stdout:Mar 18 16:21:42 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-b[67287]: ignoring --setgroup ceph since I am not root 2024-03-18T16:21:42.742 INFO:journalctl@ceph.mds.b.smithi194.stdout:Mar 18 16:21:42 smithi194 ceph-mds[67291]: -- 172.21.15.194:0/2708524556 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55948f7684e0 con 0x55948f73ec00 2024-03-18T16:21:42.742 INFO:journalctl@ceph.mds.b.smithi194.stdout:Mar 18 16:21:42 smithi194 ceph-mds[67291]: -- 172.21.15.194:0/2708524556 <== mon.2 v2:172.21.15.194:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55948f76ba40 con 0x55948f73ec00 2024-03-18T16:21:42.742 INFO:journalctl@ceph.mds.b.smithi194.stdout:Mar 18 16:21:42 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-b[67287]: starting mds.b at 2024-03-18T16:21:42.742 INFO:journalctl@ceph.mds.h.smithi194.stdout:Mar 18 16:21:42 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-h[66980]: ignoring --setuser ceph since I am not root 2024-03-18T16:21:42.742 INFO:journalctl@ceph.mds.h.smithi194.stdout:Mar 18 16:21:42 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-h[66980]: ignoring --setgroup ceph since I am not root 2024-03-18T16:21:42.742 INFO:journalctl@ceph.mds.h.smithi194.stdout:Mar 18 16:21:42 smithi194 ceph-mds[66984]: -- 172.21.15.194:0/3848894221 <== mon.2 v2:172.21.15.194:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55831fc1ba40 con 0x55831fbee800 2024-03-18T16:21:42.742 INFO:journalctl@ceph.mds.h.smithi194.stdout:Mar 18 16:21:42 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-h[66980]: starting mds.h at 2024-03-18T16:21:42.814 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:42.812+0000 7fb114800640 1 -- 172.21.15.39:0/78832104 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fb10c0624e0 con 0x7fb10c0761a0 2024-03-18T16:21:42.837 INFO:journalctl@ceph.mds.a.smithi039.stdout:Mar 18 16:21:42 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-a[71287]: ignoring --setuser ceph since I am not root 2024-03-18T16:21:42.837 INFO:journalctl@ceph.mds.a.smithi039.stdout:Mar 18 16:21:42 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-a[71287]: ignoring --setgroup ceph since I am not root 2024-03-18T16:21:42.837 INFO:journalctl@ceph.mds.a.smithi039.stdout:Mar 18 16:21:42 smithi039 ceph-mds[71310]: -- 172.21.15.39:0/991017907 <== mon.2 v2:172.21.15.194:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56271fc43a40 con 0x56271fc16800 2024-03-18T16:21:42.837 INFO:journalctl@ceph.mds.a.smithi039.stdout:Mar 18 16:21:42 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-a[71287]: starting mds.a at 2024-03-18T16:21:42.838 INFO:journalctl@ceph.mds.g.smithi039.stdout:Mar 18 16:21:42 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-g[70920]: ignoring --setuser ceph since I am not root 2024-03-18T16:21:42.838 INFO:journalctl@ceph.mds.g.smithi039.stdout:Mar 18 16:21:42 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-g[70920]: ignoring --setgroup ceph since I am not root 2024-03-18T16:21:42.838 INFO:journalctl@ceph.mds.g.smithi039.stdout:Mar 18 16:21:42 smithi039 ceph-mds[70926]: -- 172.21.15.39:0/2555037116 <== mon.1 v2:172.21.15.196:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55fb98b9ba40 con 0x55fb98b6e800 2024-03-18T16:21:42.838 INFO:journalctl@ceph.mds.g.smithi039.stdout:Mar 18 16:21:42 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-g[70920]: starting mds.g at 2024-03-18T16:21:43.315 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:43.313+0000 7fb106a00640 1 -- 172.21.15.39:0/78832104 <== mon.2 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v50) ==== 104+0+0 (secure 0 0 0) 0x7fb10009b7d0 con 0x7fb10c0761a0 2024-03-18T16:21:43.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:43.317+0000 7fb114800640 1 -- 172.21.15.39:0/78832104 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb0f40a2880 msgr2=0x7fb0f40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:43.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:43.317+0000 7fb114800640 1 --2- 172.21.15.39:0/78832104 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb0f40a2880 0x7fb0f40a4d40 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fb10c19f560 tx=0x7fb0fc005f90 comp rx=0 tx=0).stop 2024-03-18T16:21:43.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:43.317+0000 7fb114800640 1 -- 172.21.15.39:0/78832104 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb10c0761a0 msgr2=0x7fb10c19e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:43.319 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:43.317+0000 7fb114800640 1 --2- 172.21.15.39:0/78832104 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb10c0761a0 0x7fb10c19e050 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fb10000e9f0 tx=0x7fb10000eeb0 comp rx=0 tx=0).stop 2024-03-18T16:21:43.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:43.318+0000 7fb114800640 1 -- 172.21.15.39:0/78832104 shutdown_connections 2024-03-18T16:21:43.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:43.318+0000 7fb114800640 1 --2- 172.21.15.39:0/78832104 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fb0f40a2880 0x7fb0f40a4d40 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:43.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:43.318+0000 7fb114800640 1 --2- 172.21.15.39:0/78832104 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb10c077d80 0x7fb10c1a5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:43.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:43.318+0000 7fb114800640 1 --2- 172.21.15.39:0/78832104 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb10c0773e0 0x7fb10c19e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:43.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:43.318+0000 7fb114800640 1 --2- 172.21.15.39:0/78832104 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb10c0761a0 0x7fb10c19e050 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:43.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:43.318+0000 7fb114800640 1 -- 172.21.15.39:0/78832104 >> 172.21.15.39:0/78832104 conn(0x7fb10c0fef90 msgr2=0x7fb10c100a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:43.320 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:43.318+0000 7fb114800640 1 -- 172.21.15.39:0/78832104 shutdown_connections 2024-03-18T16:21:43.321 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:43.319+0000 7fb114800640 1 -- 172.21.15.39:0/78832104 wait complete. 2024-03-18T16:21:43.337 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:43 smithi039 ceph-mon[31721]: pgmap v771: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-18T16:21:43.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:43 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/2363212428' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-18T16:21:43.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:43 smithi039 ceph-mon[31721]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-18T16:21:43.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:43 smithi039 ceph-mon[31721]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-18T16:21:43.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:43 smithi039 ceph-mon[31721]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-18T16:21:43.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:43 smithi039 ceph-mon[31721]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-18T16:21:43.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:43 smithi039 ceph-mon[31721]: osdmap e94: 12 total, 12 up, 12 in 2024-03-18T16:21:43.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:43 smithi039 ceph-mon[31721]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-03-18T16:21:43.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:43 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/78832104' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-18T16:21:43.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:43 smithi039 ceph-mon[31721]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-18T16:21:43.344 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-18T16:21:43.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:43 smithi196 ceph-mon[36307]: pgmap v771: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-18T16:21:43.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:43 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/2363212428' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-18T16:21:43.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:43 smithi196 ceph-mon[36307]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-18T16:21:43.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:43 smithi196 ceph-mon[36307]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-18T16:21:43.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:43 smithi196 ceph-mon[36307]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-18T16:21:43.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:43 smithi196 ceph-mon[36307]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-18T16:21:43.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:43 smithi196 ceph-mon[36307]: osdmap e94: 12 total, 12 up, 12 in 2024-03-18T16:21:43.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:43 smithi196 ceph-mon[36307]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-03-18T16:21:43.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:43 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/78832104' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-18T16:21:43.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:43 smithi196 ceph-mon[36307]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-18T16:21:43.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:43 smithi194 ceph-mon[36200]: pgmap v771: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-18T16:21:43.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:43 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/2363212428' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-18T16:21:43.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:43 smithi194 ceph-mon[36200]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-18T16:21:43.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:43 smithi194 ceph-mon[36200]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-18T16:21:43.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:43 smithi194 ceph-mon[36200]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-18T16:21:43.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:43 smithi194 ceph-mon[36200]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-18T16:21:43.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:43 smithi194 ceph-mon[36200]: osdmap e94: 12 total, 12 up, 12 in 2024-03-18T16:21:43.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:43 smithi194 ceph-mon[36200]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-03-18T16:21:43.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:43 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/78832104' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-18T16:21:43.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:43 smithi194 ceph-mon[36200]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-18T16:21:43.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:43.560+0000 7f66f5600640 1 -- 172.21.15.39:0/3577129773 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f66f0103a20 msgr2=0x7f66f0103e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:43.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:43.560+0000 7f66f5600640 1 --2- 172.21.15.39:0/3577129773 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f66f0103a20 0x7f66f0103e40 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f66e0009ca0 tx=0x7f66e002c0f0 comp rx=0 tx=0).stop 2024-03-18T16:21:43.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:43.561+0000 7f66f5600640 1 -- 172.21.15.39:0/3577129773 shutdown_connections 2024-03-18T16:21:43.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:43.561+0000 7f66f5600640 1 --2- 172.21.15.39:0/3577129773 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f66f01055d0 0x7f66f010be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:43.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:43.561+0000 7f66f5600640 1 --2- 172.21.15.39:0/3577129773 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f66f0104c10 0x7f66f0105090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:43.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:43.561+0000 7f66f5600640 1 --2- 172.21.15.39:0/3577129773 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f66f0103a20 0x7f66f0103e40 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:43.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:43.561+0000 7f66f5600640 1 -- 172.21.15.39:0/3577129773 >> 172.21.15.39:0/3577129773 conn(0x7f66f00fef90 msgr2=0x7f66f01013d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:43.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:43.561+0000 7f66f5600640 1 -- 172.21.15.39:0/3577129773 shutdown_connections 2024-03-18T16:21:43.563 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:43.561+0000 7f66f5600640 1 -- 172.21.15.39:0/3577129773 wait complete. 2024-03-18T16:21:43.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:43.562+0000 7f66f5600640 1 Processor -- start 2024-03-18T16:21:43.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:43.562+0000 7f66f5600640 1 -- start start 2024-03-18T16:21:43.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:43.562+0000 7f66f5600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f66f0103a20 0x7f66f0199dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:43.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:43.562+0000 7f66f5600640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f66f0104c10 0x7f66f019a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:43.566 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:43.562+0000 7f66f5600640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f66f01055d0 0x7f66f01a1330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:43.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:43.562+0000 7f66f5600640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f66f010f690 con 0x7f66f0104c10 2024-03-18T16:21:43.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:43.562+0000 7f66f5600640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f66f010f510 con 0x7f66f0103a20 2024-03-18T16:21:43.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:43.562+0000 7f66f5600640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f66f010f810 con 0x7f66f01055d0 2024-03-18T16:21:43.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:43.563+0000 7f66ee000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f66f0104c10 0x7f66f019a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:43.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:43.563+0000 7f66eea00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f66f0103a20 0x7f66f0199dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:43.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:43.563+0000 7f66eea00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f66f0103a20 0x7f66f0199dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.39:37156/0 (socket says 172.21.15.39:37156) 2024-03-18T16:21:43.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:43.563+0000 7f66ef400640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f66f01055d0 0x7f66f01a1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:43.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:43.563+0000 7f66ee000640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f66f0104c10 0x7f66f019a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:36194/0 (socket says 172.21.15.39:36194) 2024-03-18T16:21:43.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:43.563+0000 7f66eea00640 1 -- 172.21.15.39:0/2897927242 learned_addr learned my addr 172.21.15.39:0/2897927242 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:21:43.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:43.563+0000 7f66eea00640 1 -- 172.21.15.39:0/2897927242 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f66f01055d0 msgr2=0x7f66f01a1330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:43.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:43.563+0000 7f66eea00640 1 --2- 172.21.15.39:0/2897927242 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f66f01055d0 0x7f66f01a1330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:43.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:43.563+0000 7f66eea00640 1 -- 172.21.15.39:0/2897927242 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f66f0104c10 msgr2=0x7f66f019a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:43.567 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:43.563+0000 7f66eea00640 1 --2- 172.21.15.39:0/2897927242 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f66f0104c10 0x7f66f019a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:43.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:43.563+0000 7f66eea00640 1 -- 172.21.15.39:0/2897927242 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66f01a19d0 con 0x7f66f0103a20 2024-03-18T16:21:43.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:43.563+0000 7f66ef400640 1 --2- 172.21.15.39:0/2897927242 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f66f01055d0 0x7f66f01a1330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:21:43.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:43.563+0000 7f66ee000640 1 --2- 172.21.15.39:0/2897927242 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f66f0104c10 0x7f66f019a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:21:43.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:43.563+0000 7f66eea00640 1 --2- 172.21.15.39:0/2897927242 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f66f0103a20 0x7f66f0199dc0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f66e00324c0 tx=0x7f66e0032f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:43.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:43.564+0000 7f66e7400640 1 -- 172.21.15.39:0/2897927242 <== mon.2 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66e0041020 con 0x7f66f0103a20 2024-03-18T16:21:43.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:43.564+0000 7f66f5600640 1 -- 172.21.15.39:0/2897927242 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66f01a1c60 con 0x7f66f0103a20 2024-03-18T16:21:43.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:43.564+0000 7f66e7400640 1 -- 172.21.15.39:0/2897927242 <== mon.2 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f66e0034cd0 con 0x7f66f0103a20 2024-03-18T16:21:43.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:43.564+0000 7f66e7400640 1 -- 172.21.15.39:0/2897927242 <== mon.2 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66e0039040 con 0x7f66f0103a20 2024-03-18T16:21:43.568 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:43.564+0000 7f66f5600640 1 -- 172.21.15.39:0/2897927242 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66f01a2150 con 0x7f66f0103a20 2024-03-18T16:21:43.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:43.565+0000 7f66f5600640 1 -- 172.21.15.39:0/2897927242 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66f010ad90 con 0x7f66f0103a20 2024-03-18T16:21:43.569 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:43.567+0000 7f66e7400640 1 -- 172.21.15.39:0/2897927242 <== mon.2 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f66e0044020 con 0x7f66f0103a20 2024-03-18T16:21:43.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:43.568+0000 7f66e7400640 1 --2- 172.21.15.39:0/2897927242 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f66d40a2800 0x7f66d40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:43.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:43.569+0000 7f66e7400640 1 -- 172.21.15.39:0/2897927242 <== mon.2 v2:172.21.15.194:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9580+0+0 (secure 0 0 0) 0x7f66e00f1310 con 0x7f66f0103a20 2024-03-18T16:21:43.570 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:43.569+0000 7f66ee000640 1 --2- 172.21.15.39:0/2897927242 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f66d40a2800 0x7f66d40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:43.571 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:43.569+0000 7f66ee000640 1 --2- 172.21.15.39:0/2897927242 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f66d40a2800 0x7f66d40a4cc0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f66d8009fc0 tx=0x7f66d8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:43.572 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:43.571+0000 7f66e7400640 1 -- 172.21.15.39:0/2897927242 <== mon.2 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f66e004b020 con 0x7f66f0103a20 2024-03-18T16:21:43.777 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:43.775+0000 7f66f5600640 1 -- 172.21.15.39:0/2897927242 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f66f002d1b0 con 0x7f66f0103a20 2024-03-18T16:21:44.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:44.326+0000 7f66e7400640 1 -- 172.21.15.39:0/2897927242 <== mon.2 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v95) ==== 181+0+0 (secure 0 0 0) 0x7f66e00bde10 con 0x7f66f0103a20 2024-03-18T16:21:44.328 INFO:teuthology.orchestra.run.smithi039.stderr:pool 'cephfs_metadata' removed 2024-03-18T16:21:44.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:44.331+0000 7f66f5600640 1 -- 172.21.15.39:0/2897927242 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f66d40a2800 msgr2=0x7f66d40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:44.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:44.331+0000 7f66f5600640 1 --2- 172.21.15.39:0/2897927242 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f66d40a2800 0x7f66d40a4cc0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f66d8009fc0 tx=0x7f66d8005f90 comp rx=0 tx=0).stop 2024-03-18T16:21:44.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:44.331+0000 7f66f5600640 1 -- 172.21.15.39:0/2897927242 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f66f0103a20 msgr2=0x7f66f0199dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:44.333 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:44.331+0000 7f66f5600640 1 --2- 172.21.15.39:0/2897927242 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f66f0103a20 0x7f66f0199dc0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f66e00324c0 tx=0x7f66e0032f40 comp rx=0 tx=0).stop 2024-03-18T16:21:44.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:44.333+0000 7f66f5600640 1 -- 172.21.15.39:0/2897927242 shutdown_connections 2024-03-18T16:21:44.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:44.333+0000 7f66f5600640 1 --2- 172.21.15.39:0/2897927242 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f66d40a2800 0x7f66d40a4cc0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:44.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:44.333+0000 7f66f5600640 1 --2- 172.21.15.39:0/2897927242 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f66f01055d0 0x7f66f01a1330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:44.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:44.333+0000 7f66f5600640 1 --2- 172.21.15.39:0/2897927242 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f66f0104c10 0x7f66f019a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:44.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:44.333+0000 7f66f5600640 1 --2- 172.21.15.39:0/2897927242 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f66f0103a20 0x7f66f0199dc0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:44.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:44.333+0000 7f66f5600640 1 -- 172.21.15.39:0/2897927242 >> 172.21.15.39:0/2897927242 conn(0x7f66f00fef90 msgr2=0x7f66f0107f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:44.335 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:44.333+0000 7f66f5600640 1 -- 172.21.15.39:0/2897927242 shutdown_connections 2024-03-18T16:21:44.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:44.334+0000 7f66f5600640 1 -- 172.21.15.39:0/2897927242 wait complete. 2024-03-18T16:21:44.339 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:44 smithi039 ceph-mon[31721]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-18T16:21:44.339 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:44 smithi039 ceph-mon[31721]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-18T16:21:44.339 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:44 smithi039 ceph-mon[31721]: Cluster is now healthy 2024-03-18T16:21:44.339 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:44 smithi039 ceph-mon[31721]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-18T16:21:44.339 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:44 smithi039 ceph-mon[31721]: mds.? [v2:172.21.15.194:6832/650048920,v1:172.21.15.194:6835/650048920] up:boot 2024-03-18T16:21:44.339 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:44 smithi039 ceph-mon[31721]: mds.? [v2:172.21.15.196:6832/3660583303,v1:172.21.15.196:6834/3660583303] up:boot 2024-03-18T16:21:44.339 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:44 smithi039 ceph-mon[31721]: mds.? [v2:172.21.15.194:6833/3522540765,v1:172.21.15.194:6834/3522540765] up:boot 2024-03-18T16:21:44.339 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:44 smithi039 ceph-mon[31721]: mds.? [v2:172.21.15.196:6833/681867401,v1:172.21.15.196:6835/681867401] up:boot 2024-03-18T16:21:44.339 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:44 smithi039 ceph-mon[31721]: mds.? [v2:172.21.15.196:6837/2507170468,v1:172.21.15.196:6839/2507170468] up:boot 2024-03-18T16:21:44.339 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:44 smithi039 ceph-mon[31721]: mds.? [v2:172.21.15.196:6836/3524782434,v1:172.21.15.196:6838/3524782434] up:boot 2024-03-18T16:21:44.339 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:44 smithi039 ceph-mon[31721]: mds.? [v2:172.21.15.194:6836/3621710687,v1:172.21.15.194:6838/3621710687] up:boot 2024-03-18T16:21:44.339 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:44 smithi039 ceph-mon[31721]: mds.? [v2:172.21.15.194:6837/977009080,v1:172.21.15.194:6839/977009080] up:boot 2024-03-18T16:21:44.340 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:44 smithi039 ceph-mon[31721]: mds.? [v2:172.21.15.39:6838/2971407205,v1:172.21.15.39:6839/2971407205] up:boot 2024-03-18T16:21:44.340 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:44 smithi039 ceph-mon[31721]: mds.? [v2:172.21.15.39:6840/3453419851,v1:172.21.15.39:6841/3453419851] up:boot 2024-03-18T16:21:44.340 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:44 smithi039 ceph-mon[31721]: fsmap 12 up:standby 2024-03-18T16:21:44.340 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:44 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-18T16:21:44.340 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:44 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-18T16:21:44.340 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:44 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-18T16:21:44.340 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:44 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-18T16:21:44.340 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:44 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-18T16:21:44.340 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:44 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-18T16:21:44.340 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:44 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-18T16:21:44.340 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:44 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-18T16:21:44.340 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:44 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-18T16:21:44.340 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:44 smithi039 ceph-mon[31721]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-18T16:21:44.340 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:44 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/2897927242' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T16:21:44.341 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:44 smithi039 ceph-mon[31721]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T16:21:44.360 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-18T16:21:44.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:44 smithi196 ceph-mon[36307]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-18T16:21:44.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:44 smithi196 ceph-mon[36307]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-18T16:21:44.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:44 smithi196 ceph-mon[36307]: Cluster is now healthy 2024-03-18T16:21:44.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:44 smithi196 ceph-mon[36307]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-18T16:21:44.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:44 smithi196 ceph-mon[36307]: mds.? [v2:172.21.15.194:6832/650048920,v1:172.21.15.194:6835/650048920] up:boot 2024-03-18T16:21:44.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:44 smithi196 ceph-mon[36307]: mds.? [v2:172.21.15.196:6832/3660583303,v1:172.21.15.196:6834/3660583303] up:boot 2024-03-18T16:21:44.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:44 smithi196 ceph-mon[36307]: mds.? [v2:172.21.15.194:6833/3522540765,v1:172.21.15.194:6834/3522540765] up:boot 2024-03-18T16:21:44.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:44 smithi196 ceph-mon[36307]: mds.? [v2:172.21.15.196:6833/681867401,v1:172.21.15.196:6835/681867401] up:boot 2024-03-18T16:21:44.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:44 smithi196 ceph-mon[36307]: mds.? [v2:172.21.15.196:6837/2507170468,v1:172.21.15.196:6839/2507170468] up:boot 2024-03-18T16:21:44.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:44 smithi196 ceph-mon[36307]: mds.? [v2:172.21.15.196:6836/3524782434,v1:172.21.15.196:6838/3524782434] up:boot 2024-03-18T16:21:44.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:44 smithi196 ceph-mon[36307]: mds.? [v2:172.21.15.194:6836/3621710687,v1:172.21.15.194:6838/3621710687] up:boot 2024-03-18T16:21:44.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:44 smithi196 ceph-mon[36307]: mds.? [v2:172.21.15.194:6837/977009080,v1:172.21.15.194:6839/977009080] up:boot 2024-03-18T16:21:44.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:44 smithi196 ceph-mon[36307]: mds.? [v2:172.21.15.39:6838/2971407205,v1:172.21.15.39:6839/2971407205] up:boot 2024-03-18T16:21:44.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:44 smithi196 ceph-mon[36307]: mds.? [v2:172.21.15.39:6840/3453419851,v1:172.21.15.39:6841/3453419851] up:boot 2024-03-18T16:21:44.480 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:44 smithi196 ceph-mon[36307]: fsmap 12 up:standby 2024-03-18T16:21:44.480 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:44 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-18T16:21:44.480 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:44 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-18T16:21:44.480 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:44 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-18T16:21:44.480 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:44 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-18T16:21:44.480 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:44 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-18T16:21:44.480 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:44 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-18T16:21:44.480 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:44 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-18T16:21:44.480 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:44 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-18T16:21:44.480 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:44 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-18T16:21:44.480 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:44 smithi196 ceph-mon[36307]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-18T16:21:44.480 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:44 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/2897927242' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T16:21:44.480 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:44 smithi196 ceph-mon[36307]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T16:21:44.490 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:44 smithi194 ceph-mon[36200]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-18T16:21:44.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:44 smithi194 ceph-mon[36200]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-18T16:21:44.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:44 smithi194 ceph-mon[36200]: Cluster is now healthy 2024-03-18T16:21:44.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:44 smithi194 ceph-mon[36200]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-18T16:21:44.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:44 smithi194 ceph-mon[36200]: mds.? [v2:172.21.15.194:6832/650048920,v1:172.21.15.194:6835/650048920] up:boot 2024-03-18T16:21:44.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:44 smithi194 ceph-mon[36200]: mds.? [v2:172.21.15.196:6832/3660583303,v1:172.21.15.196:6834/3660583303] up:boot 2024-03-18T16:21:44.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:44 smithi194 ceph-mon[36200]: mds.? [v2:172.21.15.194:6833/3522540765,v1:172.21.15.194:6834/3522540765] up:boot 2024-03-18T16:21:44.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:44 smithi194 ceph-mon[36200]: mds.? [v2:172.21.15.196:6833/681867401,v1:172.21.15.196:6835/681867401] up:boot 2024-03-18T16:21:44.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:44 smithi194 ceph-mon[36200]: mds.? [v2:172.21.15.196:6837/2507170468,v1:172.21.15.196:6839/2507170468] up:boot 2024-03-18T16:21:44.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:44 smithi194 ceph-mon[36200]: mds.? [v2:172.21.15.196:6836/3524782434,v1:172.21.15.196:6838/3524782434] up:boot 2024-03-18T16:21:44.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:44 smithi194 ceph-mon[36200]: mds.? [v2:172.21.15.194:6836/3621710687,v1:172.21.15.194:6838/3621710687] up:boot 2024-03-18T16:21:44.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:44 smithi194 ceph-mon[36200]: mds.? [v2:172.21.15.194:6837/977009080,v1:172.21.15.194:6839/977009080] up:boot 2024-03-18T16:21:44.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:44 smithi194 ceph-mon[36200]: mds.? [v2:172.21.15.39:6838/2971407205,v1:172.21.15.39:6839/2971407205] up:boot 2024-03-18T16:21:44.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:44 smithi194 ceph-mon[36200]: mds.? [v2:172.21.15.39:6840/3453419851,v1:172.21.15.39:6841/3453419851] up:boot 2024-03-18T16:21:44.491 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:44 smithi194 ceph-mon[36200]: fsmap 12 up:standby 2024-03-18T16:21:44.492 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:44 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-18T16:21:44.492 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:44 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-18T16:21:44.492 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:44 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-18T16:21:44.492 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:44 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-18T16:21:44.492 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:44 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-18T16:21:44.492 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:44 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-18T16:21:44.492 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:44 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-18T16:21:44.492 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:44 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-18T16:21:44.492 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:44 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-18T16:21:44.492 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:44 smithi194 ceph-mon[36200]: from='mgr.14152 172.21.15.39:0/3534052698' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-18T16:21:44.492 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:44 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/2897927242' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T16:21:44.492 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:44 smithi194 ceph-mon[36200]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T16:21:44.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:44.590+0000 7fd687e00640 1 -- 172.21.15.39:0/1938637293 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd688108ef0 msgr2=0x7fd688109370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:44.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:44.590+0000 7fd687e00640 1 --2- 172.21.15.39:0/1938637293 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd688108ef0 0x7fd688109370 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fd674009900 tx=0x7fd67402bc30 comp rx=0 tx=0).stop 2024-03-18T16:21:44.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:44.591+0000 7fd687e00640 1 -- 172.21.15.39:0/1938637293 shutdown_connections 2024-03-18T16:21:44.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:44.591+0000 7fd687e00640 1 --2- 172.21.15.39:0/1938637293 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd6881098b0 0x7fd688110140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:44.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:44.591+0000 7fd687e00640 1 --2- 172.21.15.39:0/1938637293 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd688108ef0 0x7fd688109370 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:44.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:44.591+0000 7fd687e00640 1 --2- 172.21.15.39:0/1938637293 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd688107d00 0x7fd688108120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:44.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:44.591+0000 7fd687e00640 1 -- 172.21.15.39:0/1938637293 >> 172.21.15.39:0/1938637293 conn(0x7fd688076cb0 msgr2=0x7fd6880790d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:44.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:44.591+0000 7fd687e00640 1 -- 172.21.15.39:0/1938637293 shutdown_connections 2024-03-18T16:21:44.593 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:44.591+0000 7fd687e00640 1 -- 172.21.15.39:0/1938637293 wait complete. 2024-03-18T16:21:44.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:44.592+0000 7fd687e00640 1 Processor -- start 2024-03-18T16:21:44.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:44.592+0000 7fd687e00640 1 -- start start 2024-03-18T16:21:44.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:44.592+0000 7fd687e00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd688107d00 0x7fd68819dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:44.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:44.592+0000 7fd687e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd688108ef0 0x7fd68819e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:44.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:44.592+0000 7fd687e00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd6881098b0 0x7fd6881a55a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:44.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:44.593+0000 7fd687e00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7fd6881139d0 con 0x7fd688107d00 2024-03-18T16:21:44.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:44.593+0000 7fd687e00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fd688113850 con 0x7fd688108ef0 2024-03-18T16:21:44.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:44.593+0000 7fd687e00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fd688113b50 con 0x7fd6881098b0 2024-03-18T16:21:44.596 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:44.593+0000 7fd686a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd688107d00 0x7fd68819dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:44.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:44.593+0000 7fd686a00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd688107d00 0x7fd68819dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:36214/0 (socket says 172.21.15.39:36214) 2024-03-18T16:21:44.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:44.593+0000 7fd686a00640 1 -- 172.21.15.39:0/1805000410 learned_addr learned my addr 172.21.15.39:0/1805000410 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:21:44.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:44.593+0000 7fd686000640 1 --2- 172.21.15.39:0/1805000410 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd688108ef0 0x7fd68819e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:44.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:44.593+0000 7fd687400640 1 --2- 172.21.15.39:0/1805000410 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd6881098b0 0x7fd6881a55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:44.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:44.593+0000 7fd686a00640 1 -- 172.21.15.39:0/1805000410 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd6881098b0 msgr2=0x7fd6881a55a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:44.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:44.593+0000 7fd686a00640 1 --2- 172.21.15.39:0/1805000410 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd6881098b0 0x7fd6881a55a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:44.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:44.593+0000 7fd686a00640 1 -- 172.21.15.39:0/1805000410 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd688108ef0 msgr2=0x7fd68819e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:44.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:44.593+0000 7fd686a00640 1 --2- 172.21.15.39:0/1805000410 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd688108ef0 0x7fd68819e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:44.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:44.593+0000 7fd686a00640 1 -- 172.21.15.39:0/1805000410 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6881a5c10 con 0x7fd688107d00 2024-03-18T16:21:44.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:44.594+0000 7fd686a00640 1 --2- 172.21.15.39:0/1805000410 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd688107d00 0x7fd68819dff0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fd67800e9c0 tx=0x7fd67800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:44.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:44.594+0000 7fd67f400640 1 -- 172.21.15.39:0/1805000410 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd67800cd80 con 0x7fd688107d00 2024-03-18T16:21:44.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:44.594+0000 7fd67f400640 1 -- 172.21.15.39:0/1805000410 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd6780035f0 con 0x7fd688107d00 2024-03-18T16:21:44.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:44.594+0000 7fd67f400640 1 -- 172.21.15.39:0/1805000410 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd678010690 con 0x7fd688107d00 2024-03-18T16:21:44.597 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:44.594+0000 7fd687e00640 1 -- 172.21.15.39:0/1805000410 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6881a5f00 con 0x7fd688107d00 2024-03-18T16:21:44.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:44.594+0000 7fd687e00640 1 -- 172.21.15.39:0/1805000410 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6881a6310 con 0x7fd688107d00 2024-03-18T16:21:44.598 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:44.595+0000 7fd687e00640 1 -- 172.21.15.39:0/1805000410 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd68810f020 con 0x7fd688107d00 2024-03-18T16:21:44.599 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:44.597+0000 7fd67f400640 1 -- 172.21.15.39:0/1805000410 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7fd6780037a0 con 0x7fd688107d00 2024-03-18T16:21:44.600 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:44.599+0000 7fd67f400640 1 --2- 172.21.15.39:0/1805000410 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd66c0a2800 0x7fd66c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:44.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:44.599+0000 7fd67f400640 1 -- 172.21.15.39:0/1805000410 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9084+0+0 (secure 0 0 0) 0x7fd6780cf1a0 con 0x7fd688107d00 2024-03-18T16:21:44.601 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:44.599+0000 7fd686000640 1 --2- 172.21.15.39:0/1805000410 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd66c0a2800 0x7fd66c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:44.602 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:44.600+0000 7fd686000640 1 --2- 172.21.15.39:0/1805000410 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd66c0a2800 0x7fd66c0a4cc0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fd67402c240 tx=0x7fd67402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:44.605 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:44.604+0000 7fd67f400640 1 -- 172.21.15.39:0/1805000410 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7fd67809be90 con 0x7fd688107d00 2024-03-18T16:21:44.812 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:44.810+0000 7fd687e00640 1 -- 172.21.15.39:0/1805000410 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fd68802d1b0 con 0x7fd688107d00 2024-03-18T16:21:45.328 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:45.326+0000 7fd67f400640 1 -- 172.21.15.39:0/1805000410 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v96) ==== 169+0+0 (secure 0 0 0) 0x7fd67809b790 con 0x7fd688107d00 2024-03-18T16:21:45.328 INFO:teuthology.orchestra.run.smithi039.stderr:pool 'cephfs_data' removed 2024-03-18T16:21:45.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:45.332+0000 7fd687e00640 1 -- 172.21.15.39:0/1805000410 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd66c0a2800 msgr2=0x7fd66c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:45.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:45.332+0000 7fd687e00640 1 --2- 172.21.15.39:0/1805000410 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd66c0a2800 0x7fd66c0a4cc0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fd67402c240 tx=0x7fd67402df90 comp rx=0 tx=0).stop 2024-03-18T16:21:45.336 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:45.332+0000 7fd687e00640 1 -- 172.21.15.39:0/1805000410 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd688107d00 msgr2=0x7fd68819dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:45.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:45.332+0000 7fd687e00640 1 --2- 172.21.15.39:0/1805000410 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd688107d00 0x7fd68819dff0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fd67800e9c0 tx=0x7fd67800ee80 comp rx=0 tx=0).stop 2024-03-18T16:21:45.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:45.333+0000 7fd687e00640 1 -- 172.21.15.39:0/1805000410 shutdown_connections 2024-03-18T16:21:45.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:45.333+0000 7fd687e00640 1 --2- 172.21.15.39:0/1805000410 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd688108ef0 0x7fd68819e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:45.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:45.333+0000 7fd687e00640 1 --2- 172.21.15.39:0/1805000410 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7fd66c0a2800 0x7fd66c0a4cc0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:45.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:45.333+0000 7fd687e00640 1 --2- 172.21.15.39:0/1805000410 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd6881098b0 0x7fd6881a55a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:45.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:45.333+0000 7fd687e00640 1 --2- 172.21.15.39:0/1805000410 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd688107d00 0x7fd68819dff0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:45.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:45.333+0000 7fd687e00640 1 -- 172.21.15.39:0/1805000410 >> 172.21.15.39:0/1805000410 conn(0x7fd688076cb0 msgr2=0x7fd68810c1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:45.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:45.334+0000 7fd687e00640 1 -- 172.21.15.39:0/1805000410 shutdown_connections 2024-03-18T16:21:45.337 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:45.335+0000 7fd687e00640 1 -- 172.21.15.39:0/1805000410 wait complete. 2024-03-18T16:21:45.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:45 smithi039 ceph-mon[31721]: pgmap v773: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 818 B/s wr, 11 op/s 2024-03-18T16:21:45.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:45 smithi039 ceph-mon[31721]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-18T16:21:45.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:45 smithi039 ceph-mon[31721]: osdmap e95: 12 total, 12 up, 12 in 2024-03-18T16:21:45.338 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:45 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/1805000410' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T16:21:45.370 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-03-18T16:21:45.391 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:45 smithi194 ceph-mon[36200]: pgmap v773: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 818 B/s wr, 11 op/s 2024-03-18T16:21:45.391 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:45 smithi194 ceph-mon[36200]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-18T16:21:45.392 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:45 smithi194 ceph-mon[36200]: osdmap e95: 12 total, 12 up, 12 in 2024-03-18T16:21:45.392 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:45 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/1805000410' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T16:21:45.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:45 smithi196 ceph-mon[36307]: pgmap v773: 129 pgs: 129 active+clean; 20 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 818 B/s wr, 11 op/s 2024-03-18T16:21:45.478 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:45 smithi196 ceph-mon[36307]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-18T16:21:45.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:45 smithi196 ceph-mon[36307]: osdmap e95: 12 total, 12 up, 12 in 2024-03-18T16:21:45.479 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:45 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/1805000410' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T16:21:45.579 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:45.577+0000 7f5246c00640 1 -- 172.21.15.39:0/1402897251 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f52401033a0 msgr2=0x7f5240109c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:45.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:45.577+0000 7f5246c00640 1 --2- 172.21.15.39:0/1402897251 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f52401033a0 0x7f5240109c30 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f5238009ca0 tx=0x7f523802c0f0 comp rx=0 tx=0).stop 2024-03-18T16:21:45.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:45.577+0000 7f5246c00640 1 -- 172.21.15.39:0/1402897251 shutdown_connections 2024-03-18T16:21:45.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:45.577+0000 7f5246c00640 1 --2- 172.21.15.39:0/1402897251 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f52401033a0 0x7f5240109c30 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:45.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:45.577+0000 7f5246c00640 1 --2- 172.21.15.39:0/1402897251 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f52401029e0 0x7f5240102e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:45.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:45.577+0000 7f5246c00640 1 --2- 172.21.15.39:0/1402897251 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f52401017f0 0x7f5240101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:45.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:45.577+0000 7f5246c00640 1 -- 172.21.15.39:0/1402897251 >> 172.21.15.39:0/1402897251 conn(0x7f52400fcd80 msgr2=0x7f52400ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:45.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:45.577+0000 7f5246c00640 1 -- 172.21.15.39:0/1402897251 shutdown_connections 2024-03-18T16:21:45.580 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:45.577+0000 7f5246c00640 1 -- 172.21.15.39:0/1402897251 wait complete. 2024-03-18T16:21:45.581 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:45.578+0000 7f5246c00640 1 Processor -- start 2024-03-18T16:21:45.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:45.578+0000 7f5246c00640 1 -- start start 2024-03-18T16:21:45.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:45.578+0000 7f5246c00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f52401017f0 0x7f5240199ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:45.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:45.579+0000 7f5246c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f52401029e0 0x7f524019a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:45.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:45.579+0000 7f5246c00640 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f52401033a0 0x7f52401a1240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:45.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:45.579+0000 7f5246c00640 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 -- 0x7f524010d470 con 0x7f52401033a0 2024-03-18T16:21:45.583 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:45.579+0000 7f5246c00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f524010d2f0 con 0x7f52401029e0 2024-03-18T16:21:45.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:45.579+0000 7f5246c00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f524010d5f0 con 0x7f52401017f0 2024-03-18T16:21:45.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:45.579+0000 7f523fe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f52401017f0 0x7f5240199ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:45.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:45.579+0000 7f523fe00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f52401017f0 0x7f5240199ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.39:41776/0 (socket says 172.21.15.39:41776) 2024-03-18T16:21:45.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:45.579+0000 7f523f400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f52401029e0 0x7f524019a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:45.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:45.579+0000 7f523fe00640 1 -- 172.21.15.39:0/3897144819 learned_addr learned my addr 172.21.15.39:0/3897144819 (peer_addr_for_me v2:172.21.15.39:0/0) 2024-03-18T16:21:45.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:45.579+0000 7f5244c00640 1 --2- 172.21.15.39:0/3897144819 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f52401033a0 0x7f52401a1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:45.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:45.579+0000 7f523fe00640 1 -- 172.21.15.39:0/3897144819 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f52401029e0 msgr2=0x7f524019a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:45.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:45.579+0000 7f523fe00640 1 --2- 172.21.15.39:0/3897144819 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f52401029e0 0x7f524019a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:45.584 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:45.579+0000 7f523fe00640 1 -- 172.21.15.39:0/3897144819 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f52401033a0 msgr2=0x7f52401a1240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:45.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:45.579+0000 7f523fe00640 1 --2- 172.21.15.39:0/3897144819 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f52401033a0 0x7f52401a1240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:45.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:45.579+0000 7f523fe00640 1 -- 172.21.15.39:0/3897144819 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52401a18e0 con 0x7f52401017f0 2024-03-18T16:21:45.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:45.579+0000 7f523f400640 1 --2- 172.21.15.39:0/3897144819 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f52401029e0 0x7f524019a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-18T16:21:45.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:45.580+0000 7f523fe00640 1 --2- 172.21.15.39:0/3897144819 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f52401017f0 0x7f5240199ca0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f523000e9c0 tx=0x7f523000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:45.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:45.580+0000 7f523cc00640 1 -- 172.21.15.39:0/3897144819 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f523000cd80 con 0x7f52401017f0 2024-03-18T16:21:45.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:45.580+0000 7f523cc00640 1 -- 172.21.15.39:0/3897144819 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f52300035f0 con 0x7f52401017f0 2024-03-18T16:21:45.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:45.580+0000 7f5246c00640 1 -- 172.21.15.39:0/3897144819 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52401a1bd0 con 0x7f52401017f0 2024-03-18T16:21:45.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:45.580+0000 7f523cc00640 1 -- 172.21.15.39:0/3897144819 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5230010690 con 0x7f52401017f0 2024-03-18T16:21:45.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:45.580+0000 7f5246c00640 1 -- 172.21.15.39:0/3897144819 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52401a2120 con 0x7f52401017f0 2024-03-18T16:21:45.585 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:45.581+0000 7f5246c00640 1 -- 172.21.15.39:0/3897144819 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52401a2350 con 0x7f52401017f0 2024-03-18T16:21:45.586 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:45.583+0000 7f523cc00640 1 -- 172.21.15.39:0/3897144819 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 18) ==== 137290+0+0 (secure 0 0 0) 0x7f52300037a0 con 0x7f52401017f0 2024-03-18T16:21:45.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:45.584+0000 7f523cc00640 1 --2- 172.21.15.39:0/3897144819 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f52240a2880 0x7f52240a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-18T16:21:45.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:45.585+0000 7f523cc00640 1 -- 172.21.15.39:0/3897144819 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 8496+0+0 (secure 0 0 0) 0x7f52300cee70 con 0x7f52401017f0 2024-03-18T16:21:45.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:45.585+0000 7f523f400640 1 --2- 172.21.15.39:0/3897144819 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f52240a2880 0x7f52240a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-18T16:21:45.587 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:45.585+0000 7f523f400640 1 --2- 172.21.15.39:0/3897144819 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f52240a2880 0x7f52240a4d40 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f524019b1a0 tx=0x7f522c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-18T16:21:45.590 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:45.589+0000 7f523cc00640 1 -- 172.21.15.39:0/3897144819 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186149 (secure 0 0 0) 0x7f523009bdd0 con 0x7f52401017f0 2024-03-18T16:21:45.817 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:45.815+0000 7f5246c00640 1 -- 172.21.15.39:0/3897144819 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) -- 0x7f52401a2520 con 0x7f52401017f0 2024-03-18T16:21:46.347 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:46.345+0000 7f523cc00640 1 -- 172.21.15.39:0/3897144819 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v97) ==== 178+0+0 (secure 0 0 0) 0x7f523009b6d0 con 0x7f52401017f0 2024-03-18T16:21:46.348 INFO:teuthology.orchestra.run.smithi039.stderr:pool 'cephfs_data_ec' removed 2024-03-18T16:21:46.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:46.350+0000 7f5246c00640 1 -- 172.21.15.39:0/3897144819 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f52240a2880 msgr2=0x7f52240a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:46.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:46.350+0000 7f5246c00640 1 --2- 172.21.15.39:0/3897144819 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f52240a2880 0x7f52240a4d40 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f524019b1a0 tx=0x7f522c005f90 comp rx=0 tx=0).stop 2024-03-18T16:21:46.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:46.350+0000 7f5246c00640 1 -- 172.21.15.39:0/3897144819 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f52401017f0 msgr2=0x7f5240199ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-18T16:21:46.352 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:46.350+0000 7f5246c00640 1 --2- 172.21.15.39:0/3897144819 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f52401017f0 0x7f5240199ca0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f523000e9c0 tx=0x7f523000ee80 comp rx=0 tx=0).stop 2024-03-18T16:21:46.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:46.351+0000 7f5246c00640 1 -- 172.21.15.39:0/3897144819 shutdown_connections 2024-03-18T16:21:46.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:46.351+0000 7f5246c00640 1 --2- 172.21.15.39:0/3897144819 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f52401029e0 0x7f524019a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:46.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:46.351+0000 7f5246c00640 1 --2- 172.21.15.39:0/3897144819 >> [v2:172.21.15.39:6800/2749214286,v1:172.21.15.39:6801/2749214286] conn(0x7f52240a2880 0x7f52240a4d40 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:46.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:46.351+0000 7f5246c00640 1 --2- 172.21.15.39:0/3897144819 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f52401033a0 0x7f52401a1240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-18T16:21:46.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:46.351+0000 7f5246c00640 1 --2- 172.21.15.39:0/3897144819 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f52401017f0 0x7f5240199ca0 secure :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f523000e9c0 tx=0x7f523000ee80 comp rx=0 tx=0).stop 2024-03-18T16:21:46.353 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:46.351+0000 7f5246c00640 1 -- 172.21.15.39:0/3897144819 >> 172.21.15.39:0/3897144819 conn(0x7f52400fcd80 msgr2=0x7f5240073a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-18T16:21:46.354 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:46.352+0000 7f5246c00640 1 -- 172.21.15.39:0/3897144819 shutdown_connections 2024-03-18T16:21:46.355 INFO:teuthology.orchestra.run.smithi039.stderr:2024-03-18T16:21:46.353+0000 7f5246c00640 1 -- 172.21.15.39:0/3897144819 wait complete. 2024-03-18T16:21:46.387 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-18T16:21:46.525 DEBUG:teuthology.orchestra.run.smithi194:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-18T16:21:46.588 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:46 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/1805000410' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-18T16:21:46.589 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:46 smithi039 ceph-mon[31721]: osdmap e96: 12 total, 12 up, 12 in 2024-03-18T16:21:46.589 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:46 smithi039 ceph-mon[31721]: from='client.? 172.21.15.39:0/3897144819' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T16:21:46.589 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:46 smithi039 ceph-mon[31721]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T16:21:46.589 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:46 smithi039 ceph-mon[31721]: pgmap v776: 33 pgs: 33 active+clean; 577 KiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:21:46.595 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:46 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/1805000410' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-18T16:21:46.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:46 smithi194 ceph-mon[36200]: osdmap e96: 12 total, 12 up, 12 in 2024-03-18T16:21:46.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:46 smithi194 ceph-mon[36200]: from='client.? 172.21.15.39:0/3897144819' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T16:21:46.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:46 smithi194 ceph-mon[36200]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T16:21:46.596 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:46 smithi194 ceph-mon[36200]: pgmap v776: 33 pgs: 33 active+clean; 577 KiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:21:46.631 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-18T16:21:46.701 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:46 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/1805000410' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-18T16:21:46.701 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:46 smithi196 ceph-mon[36307]: osdmap e96: 12 total, 12 up, 12 in 2024-03-18T16:21:46.701 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:46 smithi196 ceph-mon[36307]: from='client.? 172.21.15.39:0/3897144819' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T16:21:46.701 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:46 smithi196 ceph-mon[36307]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-18T16:21:46.701 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:46 smithi196 ceph-mon[36307]: pgmap v776: 33 pgs: 33 active+clean; 577 KiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail 2024-03-18T16:21:46.720 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-18T16:21:46.720 DEBUG:teuthology.orchestra.run.smithi039:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-18T16:21:46.779 DEBUG:teuthology.orchestra.run.smithi194:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-18T16:21:46.801 DEBUG:teuthology.orchestra.run.smithi196:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-18T16:21:46.826 INFO:tasks.cephadm:Stopping all daemons... 2024-03-18T16:21:46.826 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-18T16:21:46.826 DEBUG:teuthology.orchestra.run.smithi039:> sudo systemctl stop ceph-1892d8fc-e540-11ee-95c9-87774f69a715@mon.a 2024-03-18T16:21:47.339 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:46 smithi039 systemd[1]: Stopping Ceph mon.a for 1892d8fc-e540-11ee-95c9-87774f69a715... 2024-03-18T16:21:47.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:47 smithi196 ceph-mon[36307]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]': finished 2024-03-18T16:21:47.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:47 smithi196 ceph-mon[36307]: osdmap e97: 12 total, 12 up, 12 in 2024-03-18T16:21:47.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:47 smithi194 ceph-mon[36200]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]': finished 2024-03-18T16:21:47.741 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:47 smithi194 ceph-mon[36200]: osdmap e97: 12 total, 12 up, 12 in 2024-03-18T16:21:47.839 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:47 smithi039 ceph-mon[31721]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]': finished 2024-03-18T16:21:47.839 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:47 smithi039 ceph-mon[31721]: osdmap e97: 12 total, 12 up, 12 in 2024-03-18T16:21:47.839 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:47 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mon-a[31716]: 2024-03-18T16:21:47.503+0000 7fc873200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T16:21:47.840 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:47 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mon-a[31716]: 2024-03-18T16:21:47.504+0000 7fc873200700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-18T16:21:47.840 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:47 smithi039 podman[101180]: 2024-03-18 16:21:47.593773222 +0000 UTC m=+0.245893800 container died c8c8b05ddbd74ac24f563dc2794fd9878b25451a36ad900909048bb8acfa9618 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mon-a, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307) 2024-03-18T16:21:48.561 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:48 smithi039 podman[101180]: 2024-03-18 16:21:48.257183006 +0000 UTC m=+0.909303606 container cleanup c8c8b05ddbd74ac24f563dc2794fd9878b25451a36ad900909048bb8acfa9618 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mon-a, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=) 2024-03-18T16:21:48.561 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:48 smithi039 bash[101180]: ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mon-a 2024-03-18T16:21:48.837 INFO:journalctl@ceph.mon.a.smithi039.stdout:Mar 18 16:21:48 smithi039 podman[101193]: 2024-03-18 16:21:48.558882853 +0000 UTC m=+0.950401375 container remove c8c8b05ddbd74ac24f563dc2794fd9878b25451a36ad900909048bb8acfa9618 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mon-a, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=) 2024-03-18T16:21:49.072 DEBUG:teuthology.orchestra.run.smithi039:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1892d8fc-e540-11ee-95c9-87774f69a715@mon.a.service' 2024-03-18T16:21:49.218 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T16:21:49.218 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-18T16:21:49.218 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-03-18T16:21:49.219 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ceph-1892d8fc-e540-11ee-95c9-87774f69a715@mon.b 2024-03-18T16:21:49.572 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:49 smithi194 systemd[1]: Stopping Ceph mon.b for 1892d8fc-e540-11ee-95c9-87774f69a715... 2024-03-18T16:21:49.947 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:49 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mon-b[36196]: 2024-03-18T16:21:49.569+0000 7fce23869700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T16:21:49.948 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:49 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mon-b[36196]: 2024-03-18T16:21:49.569+0000 7fce23869700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-03-18T16:21:49.948 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:49 smithi194 podman[71966]: 2024-03-18 16:21:49.671804225 +0000 UTC m=+0.222790240 container died 171673f257b832d7b4f3919b261fcc05d3f32115e8c9b4c0eb2bffa122103658 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mon-b, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-18T16:21:50.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:50 smithi194 podman[71966]: 2024-03-18 16:21:50.045124023 +0000 UTC m=+0.596110033 container cleanup 171673f257b832d7b4f3919b261fcc05d3f32115e8c9b4c0eb2bffa122103658 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mon-b, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-18T16:21:50.240 INFO:journalctl@ceph.mon.b.smithi194.stdout:Mar 18 16:21:50 smithi194 bash[71966]: ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mon-b 2024-03-18T16:21:50.464 DEBUG:teuthology.orchestra.run.smithi194:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1892d8fc-e540-11ee-95c9-87774f69a715@mon.b.service' 2024-03-18T16:21:50.516 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T16:21:50.517 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-03-18T16:21:50.517 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-03-18T16:21:50.517 DEBUG:teuthology.orchestra.run.smithi196:> sudo systemctl stop ceph-1892d8fc-e540-11ee-95c9-87774f69a715@mon.c 2024-03-18T16:21:50.864 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:50 smithi196 systemd[1]: Stopping Ceph mon.c for 1892d8fc-e540-11ee-95c9-87774f69a715... 2024-03-18T16:21:51.228 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:50 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mon-c[36303]: 2024-03-18T16:21:50.862+0000 7ff3de622700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T16:21:51.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:50 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mon-c[36303]: 2024-03-18T16:21:50.862+0000 7ff3de622700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-03-18T16:21:51.229 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:51 smithi196 podman[71650]: 2024-03-18 16:21:51.077766905 +0000 UTC m=+0.327760154 container died 83e5d1a96c4638f038cca2c77e66c118994e205a95cd7d1088ea996a47cb513b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mon-c, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-18T16:21:51.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:51 smithi196 podman[71650]: 2024-03-18 16:21:51.505142048 +0000 UTC m=+0.755135283 container cleanup 83e5d1a96c4638f038cca2c77e66c118994e205a95cd7d1088ea996a47cb513b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mon-c, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-18T16:21:51.728 INFO:journalctl@ceph.mon.c.smithi196.stdout:Mar 18 16:21:51 smithi196 bash[71650]: ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mon-c 2024-03-18T16:21:51.959 DEBUG:teuthology.orchestra.run.smithi196:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1892d8fc-e540-11ee-95c9-87774f69a715@mon.c.service' 2024-03-18T16:21:52.011 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T16:21:52.011 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-03-18T16:21:52.012 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-18T16:21:52.012 DEBUG:teuthology.orchestra.run.smithi039:> sudo systemctl stop ceph-1892d8fc-e540-11ee-95c9-87774f69a715@mgr.x 2024-03-18T16:21:52.587 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 16:21:52 smithi039 systemd[1]: Stopping Ceph mgr.x for 1892d8fc-e540-11ee-95c9-87774f69a715... 2024-03-18T16:21:53.087 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 16:21:52 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T16:21:52.694+0000 7f6f1a200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T16:21:53.087 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 16:21:52 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x[31959]: 2024-03-18T16:21:52.694+0000 7f6f1a200700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-18T16:21:53.088 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 16:21:52 smithi039 podman[101310]: 2024-03-18 16:21:52.788999409 +0000 UTC m=+0.322291363 container died 0ad604841d44848bd8834bb6b35f810ea8be6ba0a381cb69c436dacfc47017b9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-18T16:21:53.587 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 16:21:53 smithi039 podman[101310]: 2024-03-18 16:21:53.440542075 +0000 UTC m=+0.973833954 container cleanup 0ad604841d44848bd8834bb6b35f810ea8be6ba0a381cb69c436dacfc47017b9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T16:21:53.587 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 16:21:53 smithi039 bash[101310]: ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x 2024-03-18T16:21:54.087 INFO:journalctl@ceph.mgr.x.smithi039.stdout:Mar 18 16:21:53 smithi039 podman[101324]: 2024-03-18 16:21:53.717607988 +0000 UTC m=+0.911717762 container remove 0ad604841d44848bd8834bb6b35f810ea8be6ba0a381cb69c436dacfc47017b9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3d78f5ba07bf18880172c07d4d7b4ac00c9d30af, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-x, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T16:21:54.279 DEBUG:teuthology.orchestra.run.smithi039:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1892d8fc-e540-11ee-95c9-87774f69a715@mgr.x.service' 2024-03-18T16:21:54.410 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T16:21:54.411 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-18T16:21:54.411 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-18T16:21:54.411 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ceph-1892d8fc-e540-11ee-95c9-87774f69a715@mgr.y 2024-03-18T16:21:54.740 INFO:journalctl@ceph.mgr.y.smithi194.stdout:Mar 18 16:21:54 smithi194 systemd[1]: Stopping Ceph mgr.y for 1892d8fc-e540-11ee-95c9-87774f69a715... 2024-03-18T16:21:55.060 INFO:journalctl@ceph.mgr.y.smithi194.stdout:Mar 18 16:21:54 smithi194 podman[72089]: 2024-03-18 16:21:54.754256046 +0000 UTC m=+0.152172664 container died d24c93dbcadc45d14bcf4862cfcae33e74e10adeeae40f5f599c2e4c3724ddfa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-y, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-18T16:21:55.395 INFO:journalctl@ceph.mgr.y.smithi194.stdout:Mar 18 16:21:55 smithi194 podman[72089]: 2024-03-18 16:21:55.175393959 +0000 UTC m=+0.573310577 container cleanup d24c93dbcadc45d14bcf4862cfcae33e74e10adeeae40f5f599c2e4c3724ddfa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-y, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-18T16:21:55.395 INFO:journalctl@ceph.mgr.y.smithi194.stdout:Mar 18 16:21:55 smithi194 bash[72089]: ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-y 2024-03-18T16:21:55.634 DEBUG:teuthology.orchestra.run.smithi194:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1892d8fc-e540-11ee-95c9-87774f69a715@mgr.y.service' 2024-03-18T16:21:55.657 INFO:journalctl@ceph.mgr.y.smithi194.stdout:Mar 18 16:21:55 smithi194 podman[72103]: 2024-03-18 16:21:55.393352535 +0000 UTC m=+0.632175468 container remove d24c93dbcadc45d14bcf4862cfcae33e74e10adeeae40f5f599c2e4c3724ddfa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-y, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T16:21:55.658 INFO:journalctl@ceph.mgr.y.smithi194.stdout:Mar 18 16:21:55 smithi194 systemd[1]: ceph-1892d8fc-e540-11ee-95c9-87774f69a715@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-03-18T16:21:55.658 INFO:journalctl@ceph.mgr.y.smithi194.stdout:Mar 18 16:21:55 smithi194 systemd[1]: ceph-1892d8fc-e540-11ee-95c9-87774f69a715@mgr.y.service: Failed with result 'exit-code'. 2024-03-18T16:21:55.658 INFO:journalctl@ceph.mgr.y.smithi194.stdout:Mar 18 16:21:55 smithi194 systemd[1]: Stopped Ceph mgr.y for 1892d8fc-e540-11ee-95c9-87774f69a715. 2024-03-18T16:21:55.658 INFO:journalctl@ceph.mgr.y.smithi194.stdout:Mar 18 16:21:55 smithi194 systemd[1]: ceph-1892d8fc-e540-11ee-95c9-87774f69a715@mgr.y.service: Consumed 12.999s CPU time. 2024-03-18T16:21:55.671 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T16:21:55.671 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-18T16:21:55.671 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-03-18T16:21:55.671 DEBUG:teuthology.orchestra.run.smithi196:> sudo systemctl stop ceph-1892d8fc-e540-11ee-95c9-87774f69a715@mgr.z 2024-03-18T16:21:55.978 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 18 16:21:55 smithi196 systemd[1]: Stopping Ceph mgr.z for 1892d8fc-e540-11ee-95c9-87774f69a715... 2024-03-18T16:21:56.312 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 18 16:21:56 smithi196 podman[71773]: 2024-03-18 16:21:56.005509527 +0000 UTC m=+0.146767522 container died fce4512a2469ffb976389bcb604d420920bf42ecada7daf775dd69e34accc9bc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-z, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-18T16:21:56.721 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 18 16:21:56 smithi196 podman[71773]: 2024-03-18 16:21:56.418787941 +0000 UTC m=+0.560045955 container cleanup fce4512a2469ffb976389bcb604d420920bf42ecada7daf775dd69e34accc9bc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-z, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-18T16:21:56.721 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 18 16:21:56 smithi196 bash[71773]: ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-z 2024-03-18T16:21:56.951 DEBUG:teuthology.orchestra.run.smithi196:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1892d8fc-e540-11ee-95c9-87774f69a715@mgr.z.service' 2024-03-18T16:21:56.975 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 18 16:21:56 smithi196 podman[71788]: 2024-03-18 16:21:56.720152037 +0000 UTC m=+0.713068464 container remove fce4512a2469ffb976389bcb604d420920bf42ecada7daf775dd69e34accc9bc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mgr-z, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-18T16:21:56.975 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 18 16:21:56 smithi196 systemd[1]: ceph-1892d8fc-e540-11ee-95c9-87774f69a715@mgr.z.service: Main process exited, code=exited, status=143/n/a 2024-03-18T16:21:56.975 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 18 16:21:56 smithi196 systemd[1]: ceph-1892d8fc-e540-11ee-95c9-87774f69a715@mgr.z.service: Failed with result 'exit-code'. 2024-03-18T16:21:56.975 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 18 16:21:56 smithi196 systemd[1]: Stopped Ceph mgr.z for 1892d8fc-e540-11ee-95c9-87774f69a715. 2024-03-18T16:21:56.975 INFO:journalctl@ceph.mgr.z.smithi196.stdout:Mar 18 16:21:56 smithi196 systemd[1]: ceph-1892d8fc-e540-11ee-95c9-87774f69a715@mgr.z.service: Consumed 13.214s CPU time. 2024-03-18T16:21:56.988 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T16:21:56.988 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-03-18T16:21:56.988 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-18T16:21:56.989 DEBUG:teuthology.orchestra.run.smithi039:> sudo systemctl stop ceph-1892d8fc-e540-11ee-95c9-87774f69a715@osd.0 2024-03-18T16:21:57.337 INFO:journalctl@ceph.osd.0.smithi039.stdout:Mar 18 16:21:57 smithi039 systemd[1]: Stopping Ceph osd.0 for 1892d8fc-e540-11ee-95c9-87774f69a715... 2024-03-18T16:21:57.838 INFO:journalctl@ceph.osd.0.smithi039.stdout:Mar 18 16:21:57 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-0[43741]: 2024-03-18T16:21:57.569+0000 7f8636600700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T16:21:57.912 INFO:journalctl@ceph.osd.0.smithi039.stdout:Mar 18 16:21:57 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-0[43741]: 2024-03-18T16:21:57.569+0000 7f8636600700 -1 osd.0 97 *** Got signal Terminated *** 2024-03-18T16:21:57.912 INFO:journalctl@ceph.osd.0.smithi039.stdout:Mar 18 16:21:57 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-0[43741]: 2024-03-18T16:21:57.569+0000 7f8636600700 -1 osd.0 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T16:22:03.720 INFO:journalctl@ceph.osd.0.smithi039.stdout:Mar 18 16:22:03 smithi039 podman[101430]: 2024-03-18 16:22:03.41807738 +0000 UTC m=+5.981229381 container died 2bd2b2c0bfbe7bdf4a71dd50f3c0315de66759820c7b2f3db9ef52532373a592 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-18T16:22:04.087 INFO:journalctl@ceph.osd.0.smithi039.stdout:Mar 18 16:22:03 smithi039 podman[101430]: 2024-03-18 16:22:03.846298113 +0000 UTC m=+6.409450175 container cleanup 2bd2b2c0bfbe7bdf4a71dd50f3c0315de66759820c7b2f3db9ef52532373a592 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-18T16:22:04.087 INFO:journalctl@ceph.osd.0.smithi039.stdout:Mar 18 16:22:03 smithi039 bash[101430]: ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-0 2024-03-18T16:22:04.587 INFO:journalctl@ceph.osd.0.smithi039.stdout:Mar 18 16:22:04 smithi039 podman[101452]: 2024-03-18 16:22:04.156021673 +0000 UTC m=+0.722974144 container remove 2bd2b2c0bfbe7bdf4a71dd50f3c0315de66759820c7b2f3db9ef52532373a592 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-0, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307) 2024-03-18T16:22:05.337 INFO:journalctl@ceph.osd.0.smithi039.stdout:Mar 18 16:22:05 smithi039 podman[101541]: 2024-03-18 16:22:04.930776741 +0000 UTC m=+0.037037707 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T16:22:05.337 INFO:journalctl@ceph.osd.0.smithi039.stdout:Mar 18 16:22:05 smithi039 podman[101541]: 2024-03-18 16:22:05.207355093 +0000 UTC m=+0.313616096 container create 1cfc7a6b9533f838cdc5b64b062853ea02f1a569dbb042c54fac7061d3a410db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-0-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-18T16:22:05.646 INFO:journalctl@ceph.osd.0.smithi039.stdout:Mar 18 16:22:05 smithi039 podman[101541]: 2024-03-18 16:22:05.523438247 +0000 UTC m=+0.629699257 container init 1cfc7a6b9533f838cdc5b64b062853ea02f1a569dbb042c54fac7061d3a410db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-0-deactivate, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-18T16:22:05.646 INFO:journalctl@ceph.osd.0.smithi039.stdout:Mar 18 16:22:05 smithi039 podman[101541]: 2024-03-18 16:22:05.535036085 +0000 UTC m=+0.641297082 container start 1cfc7a6b9533f838cdc5b64b062853ea02f1a569dbb042c54fac7061d3a410db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-0-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-18T16:22:05.646 INFO:journalctl@ceph.osd.0.smithi039.stdout:Mar 18 16:22:05 smithi039 podman[101541]: 2024-03-18 16:22:05.643782469 +0000 UTC m=+0.750043502 container attach 1cfc7a6b9533f838cdc5b64b062853ea02f1a569dbb042c54fac7061d3a410db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-0-deactivate, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-18T16:22:06.837 INFO:journalctl@ceph.osd.0.smithi039.stdout:Mar 18 16:22:06 smithi039 podman[101541]: 2024-03-18 16:22:06.458732573 +0000 UTC m=+1.564993568 container died 1cfc7a6b9533f838cdc5b64b062853ea02f1a569dbb042c54fac7061d3a410db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-18T16:22:07.395 DEBUG:teuthology.orchestra.run.smithi039:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1892d8fc-e540-11ee-95c9-87774f69a715@osd.0.service' 2024-03-18T16:22:07.529 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T16:22:07.529 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-18T16:22:07.529 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-18T16:22:07.529 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ceph-1892d8fc-e540-11ee-95c9-87774f69a715@osd.1 2024-03-18T16:22:07.839 INFO:journalctl@ceph.osd.1.smithi194.stdout:Mar 18 16:22:07 smithi194 systemd[1]: Stopping Ceph osd.1 for 1892d8fc-e540-11ee-95c9-87774f69a715... 2024-03-18T16:22:08.241 INFO:journalctl@ceph.osd.1.smithi194.stdout:Mar 18 16:22:07 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-1[40784]: 2024-03-18T16:22:07.836+0000 7f3cd54ea700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T16:22:08.378 INFO:journalctl@ceph.osd.1.smithi194.stdout:Mar 18 16:22:07 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-1[40784]: 2024-03-18T16:22:07.836+0000 7f3cd54ea700 -1 osd.1 97 *** Got signal Terminated *** 2024-03-18T16:22:08.378 INFO:journalctl@ceph.osd.1.smithi194.stdout:Mar 18 16:22:07 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-1[40784]: 2024-03-18T16:22:07.836+0000 7f3cd54ea700 -1 osd.1 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T16:22:13.674 INFO:journalctl@ceph.osd.1.smithi194.stdout:Mar 18 16:22:13 smithi194 podman[72211]: 2024-03-18 16:22:13.347996225 +0000 UTC m=+5.625408107 container died 082da29464fe715ec5abb8737208f27f8c113d174e95a57967c3989e36eb4f8a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-1, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307) 2024-03-18T16:22:13.991 INFO:journalctl@ceph.osd.1.smithi194.stdout:Mar 18 16:22:13 smithi194 podman[72211]: 2024-03-18 16:22:13.770684937 +0000 UTC m=+6.048096809 container cleanup 082da29464fe715ec5abb8737208f27f8c113d174e95a57967c3989e36eb4f8a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-1, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, ceph=True) 2024-03-18T16:22:13.991 INFO:journalctl@ceph.osd.1.smithi194.stdout:Mar 18 16:22:13 smithi194 bash[72211]: ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-1 2024-03-18T16:22:14.444 INFO:journalctl@ceph.osd.1.smithi194.stdout:Mar 18 16:22:14 smithi194 podman[72228]: 2024-03-18 16:22:14.022289215 +0000 UTC m=+0.676625860 container remove 082da29464fe715ec5abb8737208f27f8c113d174e95a57967c3989e36eb4f8a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-1, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-18T16:22:14.741 INFO:journalctl@ceph.osd.1.smithi194.stdout:Mar 18 16:22:14 smithi194 podman[72321]: 2024-03-18 16:22:14.350649224 +0000 UTC m=+0.022403104 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T16:22:14.747 INFO:journalctl@ceph.osd.1.smithi194.stdout:Mar 18 16:22:14 smithi194 podman[72321]: 2024-03-18 16:22:14.481538166 +0000 UTC m=+0.153292070 container create a80295b0b68c51560cd99f41e96f18d9a68cdbd51001890fd63fc15f7e55ed06 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-1-deactivate, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-18T16:22:15.240 INFO:journalctl@ceph.osd.1.smithi194.stdout:Mar 18 16:22:14 smithi194 podman[72321]: 2024-03-18 16:22:14.769158929 +0000 UTC m=+0.440912790 container init a80295b0b68c51560cd99f41e96f18d9a68cdbd51001890fd63fc15f7e55ed06 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-1-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-18T16:22:15.241 INFO:journalctl@ceph.osd.1.smithi194.stdout:Mar 18 16:22:14 smithi194 podman[72321]: 2024-03-18 16:22:14.77227556 +0000 UTC m=+0.444029416 container start a80295b0b68c51560cd99f41e96f18d9a68cdbd51001890fd63fc15f7e55ed06 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-1-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-18T16:22:15.241 INFO:journalctl@ceph.osd.1.smithi194.stdout:Mar 18 16:22:14 smithi194 podman[72321]: 2024-03-18 16:22:14.893241118 +0000 UTC m=+0.564995000 container attach a80295b0b68c51560cd99f41e96f18d9a68cdbd51001890fd63fc15f7e55ed06 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-1-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True) 2024-03-18T16:22:15.650 INFO:journalctl@ceph.osd.1.smithi194.stdout:Mar 18 16:22:15 smithi194 podman[72321]: 2024-03-18 16:22:15.382076308 +0000 UTC m=+1.053830178 container died a80295b0b68c51560cd99f41e96f18d9a68cdbd51001890fd63fc15f7e55ed06 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, io.buildah.version=1.33.5) 2024-03-18T16:22:16.043 DEBUG:teuthology.orchestra.run.smithi194:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1892d8fc-e540-11ee-95c9-87774f69a715@osd.1.service' 2024-03-18T16:22:16.084 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T16:22:16.084 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-18T16:22:16.084 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-18T16:22:16.085 DEBUG:teuthology.orchestra.run.smithi196:> sudo systemctl stop ceph-1892d8fc-e540-11ee-95c9-87774f69a715@osd.2 2024-03-18T16:22:16.449 INFO:journalctl@ceph.osd.2.smithi196.stdout:Mar 18 16:22:16 smithi196 systemd[1]: Stopping Ceph osd.2 for 1892d8fc-e540-11ee-95c9-87774f69a715... 2024-03-18T16:22:16.706 INFO:journalctl@ceph.osd.2.smithi196.stdout:Mar 18 16:22:16 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-2[41457]: 2024-03-18T16:22:16.446+0000 7fc0efe97700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T16:22:16.706 INFO:journalctl@ceph.osd.2.smithi196.stdout:Mar 18 16:22:16 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-2[41457]: 2024-03-18T16:22:16.446+0000 7fc0efe97700 -1 osd.2 97 *** Got signal Terminated *** 2024-03-18T16:22:16.706 INFO:journalctl@ceph.osd.2.smithi196.stdout:Mar 18 16:22:16 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-2[41457]: 2024-03-18T16:22:16.446+0000 7fc0efe97700 -1 osd.2 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T16:22:22.228 INFO:journalctl@ceph.osd.2.smithi196.stdout:Mar 18 16:22:21 smithi196 podman[71897]: 2024-03-18 16:22:21.949579256 +0000 UTC m=+5.677763365 container died 2041433785f64f1440fd4e7c2b31c108e647a5496f9680c0d6015c1fed9d426e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-18T16:22:22.624 INFO:journalctl@ceph.osd.2.smithi196.stdout:Mar 18 16:22:22 smithi196 podman[71897]: 2024-03-18 16:22:22.330107479 +0000 UTC m=+6.058291579 container cleanup 2041433785f64f1440fd4e7c2b31c108e647a5496f9680c0d6015c1fed9d426e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T16:22:22.625 INFO:journalctl@ceph.osd.2.smithi196.stdout:Mar 18 16:22:22 smithi196 bash[71897]: ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-2 2024-03-18T16:22:22.934 INFO:journalctl@ceph.osd.2.smithi196.stdout:Mar 18 16:22:22 smithi196 podman[71927]: 2024-03-18 16:22:22.623328189 +0000 UTC m=+0.675222895 container remove 2041433785f64f1440fd4e7c2b31c108e647a5496f9680c0d6015c1fed9d426e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-2, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-18T16:22:23.195 INFO:journalctl@ceph.osd.2.smithi196.stdout:Mar 18 16:22:22 smithi196 podman[72017]: 2024-03-18 16:22:22.835447627 +0000 UTC m=+0.017887690 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T16:22:23.196 INFO:journalctl@ceph.osd.2.smithi196.stdout:Mar 18 16:22:22 smithi196 podman[72017]: 2024-03-18 16:22:22.9576524 +0000 UTC m=+0.140092500 container create 73543ef13b00f7bf71b3327bd8743d78cc1573215e8182b1f37803f44e0cbac1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-2-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T16:22:23.196 INFO:journalctl@ceph.osd.2.smithi196.stdout:Mar 18 16:22:23 smithi196 podman[72017]: 2024-03-18 16:22:23.193423379 +0000 UTC m=+0.375863451 container init 73543ef13b00f7bf71b3327bd8743d78cc1573215e8182b1f37803f44e0cbac1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-2-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-18T16:22:23.479 INFO:journalctl@ceph.osd.2.smithi196.stdout:Mar 18 16:22:23 smithi196 podman[72017]: 2024-03-18 16:22:23.196771969 +0000 UTC m=+0.379212033 container start 73543ef13b00f7bf71b3327bd8743d78cc1573215e8182b1f37803f44e0cbac1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-2-deactivate, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-18T16:22:23.479 INFO:journalctl@ceph.osd.2.smithi196.stdout:Mar 18 16:22:23 smithi196 podman[72017]: 2024-03-18 16:22:23.318824873 +0000 UTC m=+0.501264951 container attach 73543ef13b00f7bf71b3327bd8743d78cc1573215e8182b1f37803f44e0cbac1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-2-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-18T16:22:24.074 INFO:journalctl@ceph.osd.2.smithi196.stdout:Mar 18 16:22:23 smithi196 podman[72017]: 2024-03-18 16:22:23.769574536 +0000 UTC m=+0.952014608 container died 73543ef13b00f7bf71b3327bd8743d78cc1573215e8182b1f37803f44e0cbac1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-2-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-18T16:22:24.378 DEBUG:teuthology.orchestra.run.smithi196:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1892d8fc-e540-11ee-95c9-87774f69a715@osd.2.service' 2024-03-18T16:22:24.421 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T16:22:24.421 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-18T16:22:24.421 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-18T16:22:24.422 DEBUG:teuthology.orchestra.run.smithi039:> sudo systemctl stop ceph-1892d8fc-e540-11ee-95c9-87774f69a715@osd.3 2024-03-18T16:22:24.837 INFO:journalctl@ceph.osd.3.smithi039.stdout:Mar 18 16:22:24 smithi039 systemd[1]: Stopping Ceph osd.3 for 1892d8fc-e540-11ee-95c9-87774f69a715... 2024-03-18T16:22:25.224 INFO:journalctl@ceph.osd.3.smithi039.stdout:Mar 18 16:22:25 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-3[50694]: 2024-03-18T16:22:25.005+0000 7f8459400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T16:22:25.224 INFO:journalctl@ceph.osd.3.smithi039.stdout:Mar 18 16:22:25 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-3[50694]: 2024-03-18T16:22:25.005+0000 7f8459400700 -1 osd.3 97 *** Got signal Terminated *** 2024-03-18T16:22:25.224 INFO:journalctl@ceph.osd.3.smithi039.stdout:Mar 18 16:22:25 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-3[50694]: 2024-03-18T16:22:25.005+0000 7f8459400700 -1 osd.3 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T16:22:31.088 INFO:journalctl@ceph.osd.3.smithi039.stdout:Mar 18 16:22:30 smithi039 podman[101646]: 2024-03-18 16:22:30.811246793 +0000 UTC m=+5.951391582 container died ba1dc46ed7d3cab63f5a9e9d985d5035574e2f97a9f5be7d8b4111b11d5b0376 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-3, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0) 2024-03-18T16:22:31.551 INFO:journalctl@ceph.osd.3.smithi039.stdout:Mar 18 16:22:31 smithi039 podman[101646]: 2024-03-18 16:22:31.2679532 +0000 UTC m=+6.408098053 container cleanup ba1dc46ed7d3cab63f5a9e9d985d5035574e2f97a9f5be7d8b4111b11d5b0376 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-3, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-18T16:22:31.551 INFO:journalctl@ceph.osd.3.smithi039.stdout:Mar 18 16:22:31 smithi039 bash[101646]: ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-3 2024-03-18T16:22:31.837 INFO:journalctl@ceph.osd.3.smithi039.stdout:Mar 18 16:22:31 smithi039 podman[101676]: 2024-03-18 16:22:31.548735396 +0000 UTC m=+0.720687357 container remove ba1dc46ed7d3cab63f5a9e9d985d5035574e2f97a9f5be7d8b4111b11d5b0376 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-3, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T16:22:32.837 INFO:journalctl@ceph.osd.3.smithi039.stdout:Mar 18 16:22:32 smithi039 podman[101765]: 2024-03-18 16:22:32.371243499 +0000 UTC m=+0.041699136 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T16:22:32.837 INFO:journalctl@ceph.osd.3.smithi039.stdout:Mar 18 16:22:32 smithi039 podman[101765]: 2024-03-18 16:22:32.486995387 +0000 UTC m=+0.157451060 container create f9d7365986065ac259b2e7b2cfdc926be2e26fce056b6ddcc774f5645719bcf0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-3-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-18T16:22:33.337 INFO:journalctl@ceph.osd.3.smithi039.stdout:Mar 18 16:22:32 smithi039 podman[101765]: 2024-03-18 16:22:32.932974644 +0000 UTC m=+0.603430324 container init f9d7365986065ac259b2e7b2cfdc926be2e26fce056b6ddcc774f5645719bcf0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-3-deactivate, GIT_CLEAN=True, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-18T16:22:33.337 INFO:journalctl@ceph.osd.3.smithi039.stdout:Mar 18 16:22:32 smithi039 podman[101765]: 2024-03-18 16:22:32.944566909 +0000 UTC m=+0.615022566 container start f9d7365986065ac259b2e7b2cfdc926be2e26fce056b6ddcc774f5645719bcf0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-3-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-18T16:22:33.338 INFO:journalctl@ceph.osd.3.smithi039.stdout:Mar 18 16:22:33 smithi039 podman[101765]: 2024-03-18 16:22:33.065003399 +0000 UTC m=+0.735459105 container attach f9d7365986065ac259b2e7b2cfdc926be2e26fce056b6ddcc774f5645719bcf0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-3-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-18T16:22:33.818 INFO:journalctl@ceph.osd.3.smithi039.stdout:Mar 18 16:22:33 smithi039 podman[101765]: 2024-03-18 16:22:33.404134494 +0000 UTC m=+1.074590174 container died f9d7365986065ac259b2e7b2cfdc926be2e26fce056b6ddcc774f5645719bcf0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-3-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T16:22:34.383 DEBUG:teuthology.orchestra.run.smithi039:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1892d8fc-e540-11ee-95c9-87774f69a715@osd.3.service' 2024-03-18T16:22:34.514 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T16:22:34.514 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-18T16:22:34.515 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-18T16:22:34.515 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ceph-1892d8fc-e540-11ee-95c9-87774f69a715@osd.4 2024-03-18T16:22:34.842 INFO:journalctl@ceph.osd.4.smithi194.stdout:Mar 18 16:22:34 smithi194 systemd[1]: Stopping Ceph osd.4 for 1892d8fc-e540-11ee-95c9-87774f69a715... 2024-03-18T16:22:35.240 INFO:journalctl@ceph.osd.4.smithi194.stdout:Mar 18 16:22:34 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-4[47784]: 2024-03-18T16:22:34.840+0000 7f588af58700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T16:22:35.241 INFO:journalctl@ceph.osd.4.smithi194.stdout:Mar 18 16:22:34 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-4[47784]: 2024-03-18T16:22:34.840+0000 7f588af58700 -1 osd.4 97 *** Got signal Terminated *** 2024-03-18T16:22:35.241 INFO:journalctl@ceph.osd.4.smithi194.stdout:Mar 18 16:22:34 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-4[47784]: 2024-03-18T16:22:34.840+0000 7f588af58700 -1 osd.4 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T16:22:40.702 INFO:journalctl@ceph.osd.4.smithi194.stdout:Mar 18 16:22:40 smithi194 podman[72439]: 2024-03-18 16:22:40.387591079 +0000 UTC m=+5.670274396 container died 9cac6602916932fa7aa2ca2b62062656f55425a1d230994f2ac052baf4ef9e2e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-4, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-18T16:22:40.991 INFO:journalctl@ceph.osd.4.smithi194.stdout:Mar 18 16:22:40 smithi194 podman[72439]: 2024-03-18 16:22:40.799173727 +0000 UTC m=+6.081857036 container cleanup 9cac6602916932fa7aa2ca2b62062656f55425a1d230994f2ac052baf4ef9e2e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-4, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-18T16:22:40.991 INFO:journalctl@ceph.osd.4.smithi194.stdout:Mar 18 16:22:40 smithi194 bash[72439]: ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-4 2024-03-18T16:22:41.377 INFO:journalctl@ceph.osd.4.smithi194.stdout:Mar 18 16:22:41 smithi194 podman[72458]: 2024-03-18 16:22:41.067311231 +0000 UTC m=+0.680601246 container remove 9cac6602916932fa7aa2ca2b62062656f55425a1d230994f2ac052baf4ef9e2e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-4, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307) 2024-03-18T16:22:41.714 INFO:journalctl@ceph.osd.4.smithi194.stdout:Mar 18 16:22:41 smithi194 podman[72549]: 2024-03-18 16:22:41.278394916 +0000 UTC m=+0.017802414 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T16:22:41.714 INFO:journalctl@ceph.osd.4.smithi194.stdout:Mar 18 16:22:41 smithi194 podman[72549]: 2024-03-18 16:22:41.393199638 +0000 UTC m=+0.132607154 container create 3b9aeb9c11e7e03d57c042016f120f6c86b3bd97bdd95e957f88b0ee2318fa87 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-4-deactivate, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T16:22:41.714 INFO:journalctl@ceph.osd.4.smithi194.stdout:Mar 18 16:22:41 smithi194 podman[72549]: 2024-03-18 16:22:41.612473585 +0000 UTC m=+0.351881093 container init 3b9aeb9c11e7e03d57c042016f120f6c86b3bd97bdd95e957f88b0ee2318fa87 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-4-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0) 2024-03-18T16:22:41.714 INFO:journalctl@ceph.osd.4.smithi194.stdout:Mar 18 16:22:41 smithi194 podman[72549]: 2024-03-18 16:22:41.615724035 +0000 UTC m=+0.355131533 container start 3b9aeb9c11e7e03d57c042016f120f6c86b3bd97bdd95e957f88b0ee2318fa87 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-4-deactivate, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-18T16:22:41.714 INFO:journalctl@ceph.osd.4.smithi194.stdout:Mar 18 16:22:41 smithi194 podman[72549]: 2024-03-18 16:22:41.712721278 +0000 UTC m=+0.452128791 container attach 3b9aeb9c11e7e03d57c042016f120f6c86b3bd97bdd95e957f88b0ee2318fa87 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True) 2024-03-18T16:22:41.991 INFO:journalctl@ceph.osd.4.smithi194.stdout:Mar 18 16:22:41 smithi194 podman[72549]: 2024-03-18 16:22:41.905718628 +0000 UTC m=+0.645126124 container died 3b9aeb9c11e7e03d57c042016f120f6c86b3bd97bdd95e957f88b0ee2318fa87 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-4-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240307, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS) 2024-03-18T16:22:42.518 DEBUG:teuthology.orchestra.run.smithi194:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1892d8fc-e540-11ee-95c9-87774f69a715@osd.4.service' 2024-03-18T16:22:42.560 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T16:22:42.561 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-18T16:22:42.561 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-18T16:22:42.561 DEBUG:teuthology.orchestra.run.smithi196:> sudo systemctl stop ceph-1892d8fc-e540-11ee-95c9-87774f69a715@osd.5 2024-03-18T16:22:42.978 INFO:journalctl@ceph.osd.5.smithi196.stdout:Mar 18 16:22:42 smithi196 systemd[1]: Stopping Ceph osd.5 for 1892d8fc-e540-11ee-95c9-87774f69a715... 2024-03-18T16:22:43.478 INFO:journalctl@ceph.osd.5.smithi196.stdout:Mar 18 16:22:42 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-5[48008]: 2024-03-18T16:22:42.981+0000 7f95d047f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T16:22:43.479 INFO:journalctl@ceph.osd.5.smithi196.stdout:Mar 18 16:22:42 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-5[48008]: 2024-03-18T16:22:42.981+0000 7f95d047f700 -1 osd.5 97 *** Got signal Terminated *** 2024-03-18T16:22:43.479 INFO:journalctl@ceph.osd.5.smithi196.stdout:Mar 18 16:22:42 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-5[48008]: 2024-03-18T16:22:42.981+0000 7f95d047f700 -1 osd.5 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T16:22:48.885 INFO:journalctl@ceph.osd.5.smithi196.stdout:Mar 18 16:22:48 smithi196 podman[72123]: 2024-03-18 16:22:48.598255119 +0000 UTC m=+5.845074485 container died 223e035c854b86bb40b383244d6f7e74799de8556b104cf45bc43895c8d3bc8a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-5, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-18T16:22:49.193 INFO:journalctl@ceph.osd.5.smithi196.stdout:Mar 18 16:22:48 smithi196 podman[72123]: 2024-03-18 16:22:48.973954966 +0000 UTC m=+6.220774327 container cleanup 223e035c854b86bb40b383244d6f7e74799de8556b104cf45bc43895c8d3bc8a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240307, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-18T16:22:49.193 INFO:journalctl@ceph.osd.5.smithi196.stdout:Mar 18 16:22:48 smithi196 bash[72123]: ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-5 2024-03-18T16:22:49.478 INFO:journalctl@ceph.osd.5.smithi196.stdout:Mar 18 16:22:49 smithi196 podman[72138]: 2024-03-18 16:22:49.192125184 +0000 UTC m=+0.596051869 container remove 223e035c854b86bb40b383244d6f7e74799de8556b104cf45bc43895c8d3bc8a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0) 2024-03-18T16:22:49.843 INFO:journalctl@ceph.osd.5.smithi196.stdout:Mar 18 16:22:49 smithi196 podman[72230]: 2024-03-18 16:22:49.484192187 +0000 UTC m=+0.017115396 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T16:22:49.843 INFO:journalctl@ceph.osd.5.smithi196.stdout:Mar 18 16:22:49 smithi196 podman[72230]: 2024-03-18 16:22:49.593553499 +0000 UTC m=+0.126476702 container create 2fb565fb3ff8f6646c3dbaefbda0ccc776f169bc527b11827defd4e76027244e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-5-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T16:22:50.135 INFO:journalctl@ceph.osd.5.smithi196.stdout:Mar 18 16:22:49 smithi196 podman[72230]: 2024-03-18 16:22:49.842077697 +0000 UTC m=+0.375000898 container init 2fb565fb3ff8f6646c3dbaefbda0ccc776f169bc527b11827defd4e76027244e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-5-deactivate, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-18T16:22:50.136 INFO:journalctl@ceph.osd.5.smithi196.stdout:Mar 18 16:22:49 smithi196 podman[72230]: 2024-03-18 16:22:49.845325967 +0000 UTC m=+0.378249176 container start 2fb565fb3ff8f6646c3dbaefbda0ccc776f169bc527b11827defd4e76027244e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-5-deactivate, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T16:22:50.136 INFO:journalctl@ceph.osd.5.smithi196.stdout:Mar 18 16:22:49 smithi196 podman[72230]: 2024-03-18 16:22:49.946010286 +0000 UTC m=+0.478933511 container attach 2fb565fb3ff8f6646c3dbaefbda0ccc776f169bc527b11827defd4e76027244e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-5-deactivate, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T16:22:50.398 INFO:journalctl@ceph.osd.5.smithi196.stdout:Mar 18 16:22:50 smithi196 podman[72230]: 2024-03-18 16:22:50.134631216 +0000 UTC m=+0.667554429 container died 2fb565fb3ff8f6646c3dbaefbda0ccc776f169bc527b11827defd4e76027244e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T16:22:50.818 DEBUG:teuthology.orchestra.run.smithi196:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1892d8fc-e540-11ee-95c9-87774f69a715@osd.5.service' 2024-03-18T16:22:50.859 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T16:22:50.859 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-18T16:22:50.859 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-18T16:22:50.859 DEBUG:teuthology.orchestra.run.smithi039:> sudo systemctl stop ceph-1892d8fc-e540-11ee-95c9-87774f69a715@osd.6 2024-03-18T16:22:51.337 INFO:journalctl@ceph.osd.6.smithi039.stdout:Mar 18 16:22:50 smithi039 systemd[1]: Stopping Ceph osd.6 for 1892d8fc-e540-11ee-95c9-87774f69a715... 2024-03-18T16:22:51.837 INFO:journalctl@ceph.osd.6.smithi039.stdout:Mar 18 16:22:51 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-6[57699]: 2024-03-18T16:22:51.537+0000 7f68d9e00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T16:22:51.837 INFO:journalctl@ceph.osd.6.smithi039.stdout:Mar 18 16:22:51 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-6[57699]: 2024-03-18T16:22:51.537+0000 7f68d9e00700 -1 osd.6 97 *** Got signal Terminated *** 2024-03-18T16:22:51.837 INFO:journalctl@ceph.osd.6.smithi039.stdout:Mar 18 16:22:51 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-6[57699]: 2024-03-18T16:22:51.537+0000 7f68d9e00700 -1 osd.6 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T16:22:57.309 INFO:journalctl@ceph.osd.6.smithi039.stdout:Mar 18 16:22:57 smithi039 podman[101868]: 2024-03-18 16:22:57.010986554 +0000 UTC m=+5.677867410 container died 789035e5d60fff56a3cbb0d8a70e62da06d6dca3992ea16507d8783b6f236b8c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-6, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-18T16:22:57.587 INFO:journalctl@ceph.osd.6.smithi039.stdout:Mar 18 16:22:57 smithi039 podman[101868]: 2024-03-18 16:22:57.480570492 +0000 UTC m=+6.147451311 container cleanup 789035e5d60fff56a3cbb0d8a70e62da06d6dca3992ea16507d8783b6f236b8c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-6, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-18T16:22:57.587 INFO:journalctl@ceph.osd.6.smithi039.stdout:Mar 18 16:22:57 smithi039 bash[101868]: ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-6 2024-03-18T16:22:58.087 INFO:journalctl@ceph.osd.6.smithi039.stdout:Mar 18 16:22:57 smithi039 podman[101884]: 2024-03-18 16:22:57.815393463 +0000 UTC m=+0.792103955 container remove 789035e5d60fff56a3cbb0d8a70e62da06d6dca3992ea16507d8783b6f236b8c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-6, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T16:22:58.698 INFO:journalctl@ceph.osd.6.smithi039.stdout:Mar 18 16:22:58 smithi039 podman[101970]: 2024-03-18 16:22:58.34799647 +0000 UTC m=+0.039970582 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T16:22:58.699 INFO:journalctl@ceph.osd.6.smithi039.stdout:Mar 18 16:22:58 smithi039 podman[101970]: 2024-03-18 16:22:58.50793804 +0000 UTC m=+0.199912190 container create 046cc6ca79134197b8c04f1a8ec79e6fc571bad4e5e6b24a67f4253575e69d32 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-6-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-18T16:22:59.087 INFO:journalctl@ceph.osd.6.smithi039.stdout:Mar 18 16:22:58 smithi039 podman[101970]: 2024-03-18 16:22:58.820968012 +0000 UTC m=+0.512942166 container init 046cc6ca79134197b8c04f1a8ec79e6fc571bad4e5e6b24a67f4253575e69d32 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-6-deactivate, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T16:22:59.087 INFO:journalctl@ceph.osd.6.smithi039.stdout:Mar 18 16:22:58 smithi039 podman[101970]: 2024-03-18 16:22:58.835096452 +0000 UTC m=+0.527070597 container start 046cc6ca79134197b8c04f1a8ec79e6fc571bad4e5e6b24a67f4253575e69d32 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-6-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-18T16:22:59.087 INFO:journalctl@ceph.osd.6.smithi039.stdout:Mar 18 16:22:58 smithi039 podman[101970]: 2024-03-18 16:22:58.902065452 +0000 UTC m=+0.594039639 container attach 046cc6ca79134197b8c04f1a8ec79e6fc571bad4e5e6b24a67f4253575e69d32 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-6-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-18T16:22:59.587 INFO:journalctl@ceph.osd.6.smithi039.stdout:Mar 18 16:22:59 smithi039 podman[101970]: 2024-03-18 16:22:59.283301478 +0000 UTC m=+0.975275632 container died 046cc6ca79134197b8c04f1a8ec79e6fc571bad4e5e6b24a67f4253575e69d32 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-6-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T16:23:00.103 DEBUG:teuthology.orchestra.run.smithi039:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1892d8fc-e540-11ee-95c9-87774f69a715@osd.6.service' 2024-03-18T16:23:00.211 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T16:23:00.212 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-18T16:23:00.212 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-18T16:23:00.212 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ceph-1892d8fc-e540-11ee-95c9-87774f69a715@osd.7 2024-03-18T16:23:00.530 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 18 16:23:00 smithi194 systemd[1]: Stopping Ceph osd.7 for 1892d8fc-e540-11ee-95c9-87774f69a715... 2024-03-18T16:23:00.796 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 18 16:23:00 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-7[54818]: 2024-03-18T16:23:00.527+0000 7f28cdf53700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T16:23:00.797 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 18 16:23:00 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-7[54818]: 2024-03-18T16:23:00.528+0000 7f28cdf53700 -1 osd.7 97 *** Got signal Terminated *** 2024-03-18T16:23:00.797 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 18 16:23:00 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-7[54818]: 2024-03-18T16:23:00.528+0000 7f28cdf53700 -1 osd.7 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T16:23:05.228 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 18 16:23:04 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-11[61670]: 2024-03-18T16:23:04.826+0000 7fb127c8b700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:21:59.309915+0000 front 2024-03-18T16:21:59.310166+0000 (oldest deadline 2024-03-18T16:23:04.609811+0000) 2024-03-18T16:23:05.740 INFO:journalctl@ceph.osd.10.smithi194.stdout:Mar 18 16:23:05 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-10[61903]: 2024-03-18T16:23:05.409+0000 7f86c63d5700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:22:01.305530+0000 front 2024-03-18T16:22:01.305781+0000 (oldest deadline 2024-03-18T16:23:05.405546+0000) 2024-03-18T16:23:05.838 INFO:journalctl@ceph.osd.9.smithi039.stdout:Mar 18 16:23:05 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-9[64747]: 2024-03-18T16:23:05.471+0000 7f2568000700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:22:01.432115+0000 front 2024-03-18T16:22:01.432982+0000 (oldest deadline 2024-03-18T16:23:04.833583+0000) 2024-03-18T16:23:06.228 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 18 16:23:05 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-11[61670]: 2024-03-18T16:23:05.852+0000 7fb127c8b700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:21:59.309915+0000 front 2024-03-18T16:21:59.310166+0000 (oldest deadline 2024-03-18T16:23:04.609811+0000) 2024-03-18T16:23:06.304 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 18 16:23:05 smithi194 podman[72653]: 2024-03-18 16:23:05.995730644 +0000 UTC m=+5.596288408 container died 770cf6249f6f7068d57a36fd4a315eccb27a4f425c4fc6bf4111e513a0c58ac0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-7, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T16:23:06.572 INFO:journalctl@ceph.osd.10.smithi194.stdout:Mar 18 16:23:06 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-10[61903]: 2024-03-18T16:23:06.397+0000 7f86c63d5700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:22:01.305530+0000 front 2024-03-18T16:22:01.305781+0000 (oldest deadline 2024-03-18T16:23:05.405546+0000) 2024-03-18T16:23:06.573 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 18 16:23:06 smithi194 podman[72653]: 2024-03-18 16:23:06.303203664 +0000 UTC m=+5.903761475 container cleanup 770cf6249f6f7068d57a36fd4a315eccb27a4f425c4fc6bf4111e513a0c58ac0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-7, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T16:23:06.573 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 18 16:23:06 smithi194 bash[72653]: ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-7 2024-03-18T16:23:06.837 INFO:journalctl@ceph.osd.9.smithi039.stdout:Mar 18 16:23:06 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-9[64747]: 2024-03-18T16:23:06.514+0000 7f2568000700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:22:01.432115+0000 front 2024-03-18T16:22:01.432982+0000 (oldest deadline 2024-03-18T16:23:04.833583+0000) 2024-03-18T16:23:06.965 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 18 16:23:06 smithi194 podman[72683]: 2024-03-18 16:23:06.571125515 +0000 UTC m=+0.574961466 container remove 770cf6249f6f7068d57a36fd4a315eccb27a4f425c4fc6bf4111e513a0c58ac0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-7, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-18T16:23:07.228 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 18 16:23:06 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-11[61670]: 2024-03-18T16:23:06.884+0000 7fb127c8b700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:21:59.309915+0000 front 2024-03-18T16:21:59.310166+0000 (oldest deadline 2024-03-18T16:23:04.609811+0000) 2024-03-18T16:23:07.229 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 18 16:23:06 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-8[54576]: 2024-03-18T16:23:06.839+0000 7fb10fdcd700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:22:00.351611+0000 front 2024-03-18T16:22:00.351844+0000 (oldest deadline 2024-03-18T16:23:06.251616+0000) 2024-03-18T16:23:07.241 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 18 16:23:06 smithi194 podman[72770]: 2024-03-18 16:23:06.963623705 +0000 UTC m=+0.109943845 container create c86624844df21b6d8f31b01d5d6f134cb5412f6e7b670e0453fc126b14a6aba4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-7-deactivate, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-18T16:23:07.241 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 18 16:23:06 smithi194 podman[72770]: 2024-03-18 16:23:06.871671195 +0000 UTC m=+0.017991306 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T16:23:07.241 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 18 16:23:07 smithi194 podman[72770]: 2024-03-18 16:23:07.165770621 +0000 UTC m=+0.312090718 container init c86624844df21b6d8f31b01d5d6f134cb5412f6e7b670e0453fc126b14a6aba4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-7-deactivate, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-18T16:23:07.241 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 18 16:23:07 smithi194 podman[72770]: 2024-03-18 16:23:07.16880276 +0000 UTC m=+0.315122868 container start c86624844df21b6d8f31b01d5d6f134cb5412f6e7b670e0453fc126b14a6aba4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-7-deactivate, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-03-18T16:23:07.741 INFO:journalctl@ceph.osd.10.smithi194.stdout:Mar 18 16:23:07 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-10[61903]: 2024-03-18T16:23:07.440+0000 7f86c63d5700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:22:01.305530+0000 front 2024-03-18T16:22:01.305781+0000 (oldest deadline 2024-03-18T16:23:05.405546+0000) 2024-03-18T16:23:07.741 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 18 16:23:07 smithi194 podman[72770]: 2024-03-18 16:23:07.257252758 +0000 UTC m=+0.403572875 container attach c86624844df21b6d8f31b01d5d6f134cb5412f6e7b670e0453fc126b14a6aba4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-18T16:23:07.741 INFO:journalctl@ceph.osd.7.smithi194.stdout:Mar 18 16:23:07 smithi194 podman[72770]: 2024-03-18 16:23:07.46132482 +0000 UTC m=+0.607644919 container died c86624844df21b6d8f31b01d5d6f134cb5412f6e7b670e0453fc126b14a6aba4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-7-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-18T16:23:07.837 INFO:journalctl@ceph.osd.9.smithi039.stdout:Mar 18 16:23:07 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-9[64747]: 2024-03-18T16:23:07.485+0000 7f2568000700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:22:01.432115+0000 front 2024-03-18T16:22:01.432982+0000 (oldest deadline 2024-03-18T16:23:04.833583+0000) 2024-03-18T16:23:08.075 DEBUG:teuthology.orchestra.run.smithi194:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1892d8fc-e540-11ee-95c9-87774f69a715@osd.7.service' 2024-03-18T16:23:08.120 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T16:23:08.120 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-18T16:23:08.120 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-03-18T16:23:08.120 DEBUG:teuthology.orchestra.run.smithi196:> sudo systemctl stop ceph-1892d8fc-e540-11ee-95c9-87774f69a715@osd.8 2024-03-18T16:23:08.150 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 18 16:23:07 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-8[54576]: 2024-03-18T16:23:07.792+0000 7fb10fdcd700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:22:00.351611+0000 front 2024-03-18T16:22:00.351844+0000 (oldest deadline 2024-03-18T16:23:06.251616+0000) 2024-03-18T16:23:08.150 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 18 16:23:07 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-11[61670]: 2024-03-18T16:23:07.927+0000 7fb127c8b700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:21:59.309915+0000 front 2024-03-18T16:21:59.310166+0000 (oldest deadline 2024-03-18T16:23:04.609811+0000) 2024-03-18T16:23:08.452 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 18 16:23:08 smithi196 systemd[1]: Stopping Ceph osd.8 for 1892d8fc-e540-11ee-95c9-87774f69a715... 2024-03-18T16:23:08.728 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 18 16:23:08 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-8[54576]: 2024-03-18T16:23:08.450+0000 7fb1177e1700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T16:23:08.728 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 18 16:23:08 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-8[54576]: 2024-03-18T16:23:08.450+0000 7fb1177e1700 -1 osd.8 97 *** Got signal Terminated *** 2024-03-18T16:23:08.729 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 18 16:23:08 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-8[54576]: 2024-03-18T16:23:08.450+0000 7fb1177e1700 -1 osd.8 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T16:23:08.740 INFO:journalctl@ceph.osd.10.smithi194.stdout:Mar 18 16:23:08 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-10[61903]: 2024-03-18T16:23:08.467+0000 7f86c63d5700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:22:01.305530+0000 front 2024-03-18T16:22:01.305781+0000 (oldest deadline 2024-03-18T16:23:05.405546+0000) 2024-03-18T16:23:08.837 INFO:journalctl@ceph.osd.9.smithi039.stdout:Mar 18 16:23:08 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-9[64747]: 2024-03-18T16:23:08.464+0000 7f2568000700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:22:01.432115+0000 front 2024-03-18T16:22:01.432982+0000 (oldest deadline 2024-03-18T16:23:04.833583+0000) 2024-03-18T16:23:09.228 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 18 16:23:08 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-8[54576]: 2024-03-18T16:23:08.787+0000 7fb10fdcd700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:22:00.351611+0000 front 2024-03-18T16:22:00.351844+0000 (oldest deadline 2024-03-18T16:23:06.251616+0000) 2024-03-18T16:23:09.229 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 18 16:23:08 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-11[61670]: 2024-03-18T16:23:08.923+0000 7fb127c8b700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:21:59.309915+0000 front 2024-03-18T16:21:59.310166+0000 (oldest deadline 2024-03-18T16:23:04.609811+0000) 2024-03-18T16:23:09.837 INFO:journalctl@ceph.osd.9.smithi039.stdout:Mar 18 16:23:09 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-9[64747]: 2024-03-18T16:23:09.493+0000 7f2568000700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:22:01.432115+0000 front 2024-03-18T16:22:01.432982+0000 (oldest deadline 2024-03-18T16:23:04.833583+0000) 2024-03-18T16:23:09.990 INFO:journalctl@ceph.osd.10.smithi194.stdout:Mar 18 16:23:09 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-10[61903]: 2024-03-18T16:23:09.497+0000 7f86c63d5700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:22:01.305530+0000 front 2024-03-18T16:22:01.305781+0000 (oldest deadline 2024-03-18T16:23:05.405546+0000) 2024-03-18T16:23:10.228 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 18 16:23:09 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-11[61670]: 2024-03-18T16:23:09.889+0000 7fb127c8b700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:21:59.309915+0000 front 2024-03-18T16:21:59.310166+0000 (oldest deadline 2024-03-18T16:23:04.609811+0000) 2024-03-18T16:23:10.229 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 18 16:23:09 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-8[54576]: 2024-03-18T16:23:09.740+0000 7fb10fdcd700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:22:00.351611+0000 front 2024-03-18T16:22:00.351844+0000 (oldest deadline 2024-03-18T16:23:06.251616+0000) 2024-03-18T16:23:10.837 INFO:journalctl@ceph.osd.9.smithi039.stdout:Mar 18 16:23:10 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-9[64747]: 2024-03-18T16:23:10.502+0000 7f2568000700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:22:01.432115+0000 front 2024-03-18T16:22:01.432982+0000 (oldest deadline 2024-03-18T16:23:04.833583+0000) 2024-03-18T16:23:10.990 INFO:journalctl@ceph.osd.10.smithi194.stdout:Mar 18 16:23:10 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-10[61903]: 2024-03-18T16:23:10.490+0000 7f86c63d5700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:22:01.305530+0000 front 2024-03-18T16:22:01.305781+0000 (oldest deadline 2024-03-18T16:23:05.405546+0000) 2024-03-18T16:23:11.228 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 18 16:23:10 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-8[54576]: 2024-03-18T16:23:10.757+0000 7fb10fdcd700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:22:00.351611+0000 front 2024-03-18T16:22:00.351844+0000 (oldest deadline 2024-03-18T16:23:06.251616+0000) 2024-03-18T16:23:11.228 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 18 16:23:10 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-11[61670]: 2024-03-18T16:23:10.846+0000 7fb127c8b700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:21:59.309915+0000 front 2024-03-18T16:21:59.310166+0000 (oldest deadline 2024-03-18T16:23:04.609811+0000) 2024-03-18T16:23:11.837 INFO:journalctl@ceph.osd.9.smithi039.stdout:Mar 18 16:23:11 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-9[64747]: 2024-03-18T16:23:11.529+0000 7f2568000700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:22:01.432115+0000 front 2024-03-18T16:22:01.432982+0000 (oldest deadline 2024-03-18T16:23:04.833583+0000) 2024-03-18T16:23:11.990 INFO:journalctl@ceph.osd.10.smithi194.stdout:Mar 18 16:23:11 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-10[61903]: 2024-03-18T16:23:11.521+0000 7f86c63d5700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:22:01.305530+0000 front 2024-03-18T16:22:01.305781+0000 (oldest deadline 2024-03-18T16:23:05.405546+0000) 2024-03-18T16:23:12.228 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 18 16:23:11 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-11[61670]: 2024-03-18T16:23:11.895+0000 7fb127c8b700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:21:59.309915+0000 front 2024-03-18T16:21:59.310166+0000 (oldest deadline 2024-03-18T16:23:04.609811+0000) 2024-03-18T16:23:12.228 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 18 16:23:11 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-8[54576]: 2024-03-18T16:23:11.748+0000 7fb10fdcd700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:22:00.351611+0000 front 2024-03-18T16:22:00.351844+0000 (oldest deadline 2024-03-18T16:23:06.251616+0000) 2024-03-18T16:23:12.740 INFO:journalctl@ceph.osd.10.smithi194.stdout:Mar 18 16:23:12 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-10[61903]: 2024-03-18T16:23:12.483+0000 7f86c63d5700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:22:01.305530+0000 front 2024-03-18T16:22:01.305781+0000 (oldest deadline 2024-03-18T16:23:05.405546+0000) 2024-03-18T16:23:12.837 INFO:journalctl@ceph.osd.9.smithi039.stdout:Mar 18 16:23:12 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-9[64747]: 2024-03-18T16:23:12.516+0000 7f2568000700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:22:01.432115+0000 front 2024-03-18T16:22:01.432982+0000 (oldest deadline 2024-03-18T16:23:04.833583+0000) 2024-03-18T16:23:13.228 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 18 16:23:12 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-11[61670]: 2024-03-18T16:23:12.851+0000 7fb127c8b700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:21:59.309915+0000 front 2024-03-18T16:21:59.310166+0000 (oldest deadline 2024-03-18T16:23:04.609811+0000) 2024-03-18T16:23:13.229 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 18 16:23:12 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-8[54576]: 2024-03-18T16:23:12.739+0000 7fb10fdcd700 -1 osd.8 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:22:00.351611+0000 front 2024-03-18T16:22:00.351844+0000 (oldest deadline 2024-03-18T16:23:06.251616+0000) 2024-03-18T16:23:13.837 INFO:journalctl@ceph.osd.9.smithi039.stdout:Mar 18 16:23:13 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-9[64747]: 2024-03-18T16:23:13.519+0000 7f2568000700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:22:01.432115+0000 front 2024-03-18T16:22:01.432982+0000 (oldest deadline 2024-03-18T16:23:04.833583+0000) 2024-03-18T16:23:13.990 INFO:journalctl@ceph.osd.10.smithi194.stdout:Mar 18 16:23:13 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-10[61903]: 2024-03-18T16:23:13.532+0000 7f86c63d5700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:22:01.305530+0000 front 2024-03-18T16:22:01.305781+0000 (oldest deadline 2024-03-18T16:23:05.405546+0000) 2024-03-18T16:23:14.228 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 18 16:23:13 smithi196 podman[72346]: 2024-03-18 16:23:13.955703577 +0000 UTC m=+5.634281358 container died b9c41d44520c0477cd44b1c1a1903429a9514a311c5f32f4ddb6e126165be6e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-8, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-18T16:23:14.229 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 18 16:23:13 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-11[61670]: 2024-03-18T16:23:13.813+0000 7fb127c8b700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:21:59.309915+0000 front 2024-03-18T16:21:59.310166+0000 (oldest deadline 2024-03-18T16:23:04.609811+0000) 2024-03-18T16:23:14.478 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 18 16:23:14 smithi196 podman[72346]: 2024-03-18 16:23:14.300486828 +0000 UTC m=+5.979064612 container cleanup b9c41d44520c0477cd44b1c1a1903429a9514a311c5f32f4ddb6e126165be6e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-8, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-18T16:23:14.478 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 18 16:23:14 smithi196 bash[72346]: ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-8 2024-03-18T16:23:14.740 INFO:journalctl@ceph.osd.10.smithi194.stdout:Mar 18 16:23:14 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-10[61903]: 2024-03-18T16:23:14.487+0000 7f86c63d5700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:22:01.305530+0000 front 2024-03-18T16:22:01.305781+0000 (oldest deadline 2024-03-18T16:23:05.405546+0000) 2024-03-18T16:23:14.741 INFO:journalctl@ceph.osd.10.smithi194.stdout:Mar 18 16:23:14 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-10[61903]: 2024-03-18T16:23:14.487+0000 7f86c63d5700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.194:6804 osd.1 since back 2024-03-18T16:22:10.106502+0000 front 2024-03-18T16:22:10.106530+0000 (oldest deadline 2024-03-18T16:23:14.206661+0000) 2024-03-18T16:23:14.837 INFO:journalctl@ceph.osd.9.smithi039.stdout:Mar 18 16:23:14 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-9[64747]: 2024-03-18T16:23:14.523+0000 7f2568000700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:22:01.432115+0000 front 2024-03-18T16:22:01.432982+0000 (oldest deadline 2024-03-18T16:23:04.833583+0000) 2024-03-18T16:23:14.837 INFO:journalctl@ceph.osd.9.smithi039.stdout:Mar 18 16:23:14 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-9[64747]: 2024-03-18T16:23:14.523+0000 7f2568000700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.194:6804 osd.1 since back 2024-03-18T16:22:10.036658+0000 front 2024-03-18T16:22:10.036823+0000 (oldest deadline 2024-03-18T16:23:14.136825+0000) 2024-03-18T16:23:14.895 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 18 16:23:14 smithi196 podman[72363]: 2024-03-18 16:23:14.604097443 +0000 UTC m=+0.647492790 container remove b9c41d44520c0477cd44b1c1a1903429a9514a311c5f32f4ddb6e126165be6e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-8, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T16:23:14.895 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 18 16:23:14 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-11[61670]: 2024-03-18T16:23:14.799+0000 7fb127c8b700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:21:59.309915+0000 front 2024-03-18T16:21:59.310166+0000 (oldest deadline 2024-03-18T16:23:04.609811+0000) 2024-03-18T16:23:15.150 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 18 16:23:14 smithi196 podman[72452]: 2024-03-18 16:23:14.796591703 +0000 UTC m=+0.017423221 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T16:23:15.150 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 18 16:23:14 smithi196 podman[72452]: 2024-03-18 16:23:14.896585382 +0000 UTC m=+0.117416929 container create ac78e2325835fb5f4ccd13ae458cb05b99db222c66bc9a911946baa6e6841932 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-8-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-18T16:23:15.444 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 18 16:23:15 smithi196 podman[72452]: 2024-03-18 16:23:15.149219547 +0000 UTC m=+0.370051055 container init ac78e2325835fb5f4ccd13ae458cb05b99db222c66bc9a911946baa6e6841932 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-8-deactivate, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-18T16:23:15.444 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 18 16:23:15 smithi196 podman[72452]: 2024-03-18 16:23:15.152344774 +0000 UTC m=+0.373176283 container start ac78e2325835fb5f4ccd13ae458cb05b99db222c66bc9a911946baa6e6841932 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-8-deactivate, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-18T16:23:15.444 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 18 16:23:15 smithi196 podman[72452]: 2024-03-18 16:23:15.232584568 +0000 UTC m=+0.453416093 container attach ac78e2325835fb5f4ccd13ae458cb05b99db222c66bc9a911946baa6e6841932 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-8-deactivate, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T16:23:15.728 INFO:journalctl@ceph.osd.8.smithi196.stdout:Mar 18 16:23:15 smithi196 podman[72452]: 2024-03-18 16:23:15.442482357 +0000 UTC m=+0.663313866 container died ac78e2325835fb5f4ccd13ae458cb05b99db222c66bc9a911946baa6e6841932 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-8-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-18T16:23:15.740 INFO:journalctl@ceph.osd.10.smithi194.stdout:Mar 18 16:23:15 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-10[61903]: 2024-03-18T16:23:15.443+0000 7f86c63d5700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:22:01.305530+0000 front 2024-03-18T16:22:01.305781+0000 (oldest deadline 2024-03-18T16:23:05.405546+0000) 2024-03-18T16:23:15.740 INFO:journalctl@ceph.osd.10.smithi194.stdout:Mar 18 16:23:15 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-10[61903]: 2024-03-18T16:23:15.443+0000 7f86c63d5700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.194:6804 osd.1 since back 2024-03-18T16:22:10.106502+0000 front 2024-03-18T16:22:10.106530+0000 (oldest deadline 2024-03-18T16:23:14.206661+0000) 2024-03-18T16:23:15.837 INFO:journalctl@ceph.osd.9.smithi039.stdout:Mar 18 16:23:15 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-9[64747]: 2024-03-18T16:23:15.518+0000 7f2568000700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:22:01.432115+0000 front 2024-03-18T16:22:01.432982+0000 (oldest deadline 2024-03-18T16:23:04.833583+0000) 2024-03-18T16:23:15.837 INFO:journalctl@ceph.osd.9.smithi039.stdout:Mar 18 16:23:15 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-9[64747]: 2024-03-18T16:23:15.518+0000 7f2568000700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.194:6804 osd.1 since back 2024-03-18T16:22:10.036658+0000 front 2024-03-18T16:22:10.036823+0000 (oldest deadline 2024-03-18T16:23:14.136825+0000) 2024-03-18T16:23:16.079 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 18 16:23:15 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-11[61670]: 2024-03-18T16:23:15.834+0000 7fb127c8b700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:21:59.309915+0000 front 2024-03-18T16:21:59.310166+0000 (oldest deadline 2024-03-18T16:23:04.609811+0000) 2024-03-18T16:23:16.105 DEBUG:teuthology.orchestra.run.smithi196:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1892d8fc-e540-11ee-95c9-87774f69a715@osd.8.service' 2024-03-18T16:23:16.187 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T16:23:16.187 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-03-18T16:23:16.187 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-03-18T16:23:16.187 DEBUG:teuthology.orchestra.run.smithi039:> sudo systemctl stop ceph-1892d8fc-e540-11ee-95c9-87774f69a715@osd.9 2024-03-18T16:23:16.556 INFO:journalctl@ceph.osd.9.smithi039.stdout:Mar 18 16:23:16 smithi039 systemd[1]: Stopping Ceph osd.9 for 1892d8fc-e540-11ee-95c9-87774f69a715... 2024-03-18T16:23:16.742 INFO:journalctl@ceph.osd.10.smithi194.stdout:Mar 18 16:23:16 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-10[61903]: 2024-03-18T16:23:16.460+0000 7f86c63d5700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:22:01.305530+0000 front 2024-03-18T16:22:01.305781+0000 (oldest deadline 2024-03-18T16:23:05.405546+0000) 2024-03-18T16:23:16.742 INFO:journalctl@ceph.osd.10.smithi194.stdout:Mar 18 16:23:16 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-10[61903]: 2024-03-18T16:23:16.460+0000 7f86c63d5700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.194:6804 osd.1 since back 2024-03-18T16:22:10.106502+0000 front 2024-03-18T16:22:10.106530+0000 (oldest deadline 2024-03-18T16:23:14.206661+0000) 2024-03-18T16:23:16.837 INFO:journalctl@ceph.osd.9.smithi039.stdout:Mar 18 16:23:16 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-9[64747]: 2024-03-18T16:23:16.552+0000 7f2568000700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:22:01.432115+0000 front 2024-03-18T16:22:01.432982+0000 (oldest deadline 2024-03-18T16:23:04.833583+0000) 2024-03-18T16:23:16.838 INFO:journalctl@ceph.osd.9.smithi039.stdout:Mar 18 16:23:16 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-9[64747]: 2024-03-18T16:23:16.552+0000 7f2568000700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.194:6804 osd.1 since back 2024-03-18T16:22:10.036658+0000 front 2024-03-18T16:22:10.036823+0000 (oldest deadline 2024-03-18T16:23:14.136825+0000) 2024-03-18T16:23:16.838 INFO:journalctl@ceph.osd.9.smithi039.stdout:Mar 18 16:23:16 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-9[64747]: 2024-03-18T16:23:16.789+0000 7f256c800700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T16:23:16.838 INFO:journalctl@ceph.osd.9.smithi039.stdout:Mar 18 16:23:16 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-9[64747]: 2024-03-18T16:23:16.789+0000 7f256c800700 -1 osd.9 97 *** Got signal Terminated *** 2024-03-18T16:23:16.838 INFO:journalctl@ceph.osd.9.smithi039.stdout:Mar 18 16:23:16 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-9[64747]: 2024-03-18T16:23:16.789+0000 7f256c800700 -1 osd.9 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T16:23:17.228 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 18 16:23:16 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-11[61670]: 2024-03-18T16:23:16.864+0000 7fb127c8b700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:21:59.309915+0000 front 2024-03-18T16:21:59.310166+0000 (oldest deadline 2024-03-18T16:23:04.609811+0000) 2024-03-18T16:23:17.837 INFO:journalctl@ceph.osd.9.smithi039.stdout:Mar 18 16:23:17 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-9[64747]: 2024-03-18T16:23:17.581+0000 7f2568000700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:22:01.432115+0000 front 2024-03-18T16:22:01.432982+0000 (oldest deadline 2024-03-18T16:23:04.833583+0000) 2024-03-18T16:23:17.837 INFO:journalctl@ceph.osd.9.smithi039.stdout:Mar 18 16:23:17 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-9[64747]: 2024-03-18T16:23:17.581+0000 7f2568000700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.194:6804 osd.1 since back 2024-03-18T16:22:10.036658+0000 front 2024-03-18T16:22:10.036823+0000 (oldest deadline 2024-03-18T16:23:14.136825+0000) 2024-03-18T16:23:17.990 INFO:journalctl@ceph.osd.10.smithi194.stdout:Mar 18 16:23:17 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-10[61903]: 2024-03-18T16:23:17.509+0000 7f86c63d5700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:22:01.305530+0000 front 2024-03-18T16:22:01.305781+0000 (oldest deadline 2024-03-18T16:23:05.405546+0000) 2024-03-18T16:23:17.991 INFO:journalctl@ceph.osd.10.smithi194.stdout:Mar 18 16:23:17 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-10[61903]: 2024-03-18T16:23:17.509+0000 7f86c63d5700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.194:6804 osd.1 since back 2024-03-18T16:22:10.106502+0000 front 2024-03-18T16:22:10.106530+0000 (oldest deadline 2024-03-18T16:23:14.206661+0000) 2024-03-18T16:23:18.228 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 18 16:23:17 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-11[61670]: 2024-03-18T16:23:17.883+0000 7fb127c8b700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:21:59.309915+0000 front 2024-03-18T16:21:59.310166+0000 (oldest deadline 2024-03-18T16:23:04.609811+0000) 2024-03-18T16:23:18.837 INFO:journalctl@ceph.osd.9.smithi039.stdout:Mar 18 16:23:18 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-9[64747]: 2024-03-18T16:23:18.553+0000 7f2568000700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:22:01.432115+0000 front 2024-03-18T16:22:01.432982+0000 (oldest deadline 2024-03-18T16:23:04.833583+0000) 2024-03-18T16:23:18.837 INFO:journalctl@ceph.osd.9.smithi039.stdout:Mar 18 16:23:18 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-9[64747]: 2024-03-18T16:23:18.553+0000 7f2568000700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.194:6804 osd.1 since back 2024-03-18T16:22:10.036658+0000 front 2024-03-18T16:22:10.036823+0000 (oldest deadline 2024-03-18T16:23:14.136825+0000) 2024-03-18T16:23:18.991 INFO:journalctl@ceph.osd.10.smithi194.stdout:Mar 18 16:23:18 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-10[61903]: 2024-03-18T16:23:18.552+0000 7f86c63d5700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:22:01.305530+0000 front 2024-03-18T16:22:01.305781+0000 (oldest deadline 2024-03-18T16:23:05.405546+0000) 2024-03-18T16:23:18.991 INFO:journalctl@ceph.osd.10.smithi194.stdout:Mar 18 16:23:18 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-10[61903]: 2024-03-18T16:23:18.552+0000 7f86c63d5700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.194:6804 osd.1 since back 2024-03-18T16:22:10.106502+0000 front 2024-03-18T16:22:10.106530+0000 (oldest deadline 2024-03-18T16:23:14.206661+0000) 2024-03-18T16:23:19.228 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 18 16:23:18 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-11[61670]: 2024-03-18T16:23:18.854+0000 7fb127c8b700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:21:59.309915+0000 front 2024-03-18T16:21:59.310166+0000 (oldest deadline 2024-03-18T16:23:04.609811+0000) 2024-03-18T16:23:19.837 INFO:journalctl@ceph.osd.9.smithi039.stdout:Mar 18 16:23:19 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-9[64747]: 2024-03-18T16:23:19.509+0000 7f2568000700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:22:01.432115+0000 front 2024-03-18T16:22:01.432982+0000 (oldest deadline 2024-03-18T16:23:04.833583+0000) 2024-03-18T16:23:19.837 INFO:journalctl@ceph.osd.9.smithi039.stdout:Mar 18 16:23:19 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-9[64747]: 2024-03-18T16:23:19.509+0000 7f2568000700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.194:6804 osd.1 since back 2024-03-18T16:22:10.036658+0000 front 2024-03-18T16:22:10.036823+0000 (oldest deadline 2024-03-18T16:23:14.136825+0000) 2024-03-18T16:23:19.990 INFO:journalctl@ceph.osd.10.smithi194.stdout:Mar 18 16:23:19 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-10[61903]: 2024-03-18T16:23:19.588+0000 7f86c63d5700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:22:01.305530+0000 front 2024-03-18T16:22:01.305781+0000 (oldest deadline 2024-03-18T16:23:05.405546+0000) 2024-03-18T16:23:19.991 INFO:journalctl@ceph.osd.10.smithi194.stdout:Mar 18 16:23:19 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-10[61903]: 2024-03-18T16:23:19.588+0000 7f86c63d5700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.194:6804 osd.1 since back 2024-03-18T16:22:10.106502+0000 front 2024-03-18T16:22:10.106530+0000 (oldest deadline 2024-03-18T16:23:14.206661+0000) 2024-03-18T16:23:20.228 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 18 16:23:19 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-11[61670]: 2024-03-18T16:23:19.850+0000 7fb127c8b700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:21:59.309915+0000 front 2024-03-18T16:21:59.310166+0000 (oldest deadline 2024-03-18T16:23:04.609811+0000) 2024-03-18T16:23:20.837 INFO:journalctl@ceph.osd.9.smithi039.stdout:Mar 18 16:23:20 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-9[64747]: 2024-03-18T16:23:20.526+0000 7f2568000700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:22:01.432115+0000 front 2024-03-18T16:22:01.432982+0000 (oldest deadline 2024-03-18T16:23:04.833583+0000) 2024-03-18T16:23:20.837 INFO:journalctl@ceph.osd.9.smithi039.stdout:Mar 18 16:23:20 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-9[64747]: 2024-03-18T16:23:20.526+0000 7f2568000700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.194:6804 osd.1 since back 2024-03-18T16:22:10.036658+0000 front 2024-03-18T16:22:10.036823+0000 (oldest deadline 2024-03-18T16:23:14.136825+0000) 2024-03-18T16:23:20.990 INFO:journalctl@ceph.osd.10.smithi194.stdout:Mar 18 16:23:20 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-10[61903]: 2024-03-18T16:23:20.555+0000 7f86c63d5700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:22:01.305530+0000 front 2024-03-18T16:22:01.305781+0000 (oldest deadline 2024-03-18T16:23:05.405546+0000) 2024-03-18T16:23:20.990 INFO:journalctl@ceph.osd.10.smithi194.stdout:Mar 18 16:23:20 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-10[61903]: 2024-03-18T16:23:20.555+0000 7f86c63d5700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.194:6804 osd.1 since back 2024-03-18T16:22:10.106502+0000 front 2024-03-18T16:22:10.106530+0000 (oldest deadline 2024-03-18T16:23:14.206661+0000) 2024-03-18T16:23:21.228 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 18 16:23:20 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-11[61670]: 2024-03-18T16:23:20.884+0000 7fb127c8b700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:21:59.309915+0000 front 2024-03-18T16:21:59.310166+0000 (oldest deadline 2024-03-18T16:23:04.609811+0000) 2024-03-18T16:23:21.837 INFO:journalctl@ceph.osd.9.smithi039.stdout:Mar 18 16:23:21 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-9[64747]: 2024-03-18T16:23:21.518+0000 7f2568000700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:22:01.432115+0000 front 2024-03-18T16:22:01.432982+0000 (oldest deadline 2024-03-18T16:23:04.833583+0000) 2024-03-18T16:23:21.837 INFO:journalctl@ceph.osd.9.smithi039.stdout:Mar 18 16:23:21 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-9[64747]: 2024-03-18T16:23:21.518+0000 7f2568000700 -1 osd.9 97 heartbeat_check: no reply from 172.21.15.194:6804 osd.1 since back 2024-03-18T16:22:10.036658+0000 front 2024-03-18T16:22:10.036823+0000 (oldest deadline 2024-03-18T16:23:14.136825+0000) 2024-03-18T16:23:21.990 INFO:journalctl@ceph.osd.10.smithi194.stdout:Mar 18 16:23:21 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-10[61903]: 2024-03-18T16:23:21.586+0000 7f86c63d5700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:22:01.305530+0000 front 2024-03-18T16:22:01.305781+0000 (oldest deadline 2024-03-18T16:23:05.405546+0000) 2024-03-18T16:23:21.990 INFO:journalctl@ceph.osd.10.smithi194.stdout:Mar 18 16:23:21 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-10[61903]: 2024-03-18T16:23:21.586+0000 7f86c63d5700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.194:6804 osd.1 since back 2024-03-18T16:22:10.106502+0000 front 2024-03-18T16:22:10.106530+0000 (oldest deadline 2024-03-18T16:23:14.206661+0000) 2024-03-18T16:23:22.228 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 18 16:23:21 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-11[61670]: 2024-03-18T16:23:21.914+0000 7fb127c8b700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:21:59.309915+0000 front 2024-03-18T16:21:59.310166+0000 (oldest deadline 2024-03-18T16:23:04.609811+0000) 2024-03-18T16:23:22.991 INFO:journalctl@ceph.osd.10.smithi194.stdout:Mar 18 16:23:22 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-10[61903]: 2024-03-18T16:23:22.632+0000 7f86c63d5700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:22:01.305530+0000 front 2024-03-18T16:22:01.305781+0000 (oldest deadline 2024-03-18T16:23:05.405546+0000) 2024-03-18T16:23:23.557 INFO:journalctl@ceph.osd.10.smithi194.stdout:Mar 18 16:23:22 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-10[61903]: 2024-03-18T16:23:22.632+0000 7f86c63d5700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.194:6804 osd.1 since back 2024-03-18T16:22:10.106502+0000 front 2024-03-18T16:22:10.106530+0000 (oldest deadline 2024-03-18T16:23:14.206661+0000) 2024-03-18T16:23:23.559 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 18 16:23:22 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-11[61670]: 2024-03-18T16:23:22.947+0000 7fb127c8b700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:21:59.309915+0000 front 2024-03-18T16:21:59.310166+0000 (oldest deadline 2024-03-18T16:23:04.609811+0000) 2024-03-18T16:23:23.559 INFO:journalctl@ceph.osd.9.smithi039.stdout:Mar 18 16:23:22 smithi039 podman[102087]: 2024-03-18 16:23:22.756356386 +0000 UTC m=+6.103545644 container died 12c25915865a56a24022b1e9169faf6ab3894afb15213da862d8fa1aacbbe415 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-9, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-18T16:23:23.559 INFO:journalctl@ceph.osd.9.smithi039.stdout:Mar 18 16:23:23 smithi039 podman[102087]: 2024-03-18 16:23:23.166798101 +0000 UTC m=+6.513987355 container cleanup 12c25915865a56a24022b1e9169faf6ab3894afb15213da862d8fa1aacbbe415 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-9, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2) 2024-03-18T16:23:23.559 INFO:journalctl@ceph.osd.9.smithi039.stdout:Mar 18 16:23:23 smithi039 bash[102087]: ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-9 2024-03-18T16:23:23.837 INFO:journalctl@ceph.osd.9.smithi039.stdout:Mar 18 16:23:23 smithi039 podman[102106]: 2024-03-18 16:23:23.434847249 +0000 UTC m=+0.668377980 container remove 12c25915865a56a24022b1e9169faf6ab3894afb15213da862d8fa1aacbbe415 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-9, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-18T16:23:23.990 INFO:journalctl@ceph.osd.10.smithi194.stdout:Mar 18 16:23:23 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-10[61903]: 2024-03-18T16:23:23.660+0000 7f86c63d5700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:22:01.305530+0000 front 2024-03-18T16:22:01.305781+0000 (oldest deadline 2024-03-18T16:23:05.405546+0000) 2024-03-18T16:23:23.991 INFO:journalctl@ceph.osd.10.smithi194.stdout:Mar 18 16:23:23 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-10[61903]: 2024-03-18T16:23:23.660+0000 7f86c63d5700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.194:6804 osd.1 since back 2024-03-18T16:22:10.106502+0000 front 2024-03-18T16:22:10.106530+0000 (oldest deadline 2024-03-18T16:23:14.206661+0000) 2024-03-18T16:23:24.228 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 18 16:23:23 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-11[61670]: 2024-03-18T16:23:23.901+0000 7fb127c8b700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:21:59.309915+0000 front 2024-03-18T16:21:59.310166+0000 (oldest deadline 2024-03-18T16:23:04.609811+0000) 2024-03-18T16:23:24.587 INFO:journalctl@ceph.osd.9.smithi039.stdout:Mar 18 16:23:24 smithi039 podman[102192]: 2024-03-18 16:23:24.187085888 +0000 UTC m=+0.041154243 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T16:23:24.587 INFO:journalctl@ceph.osd.9.smithi039.stdout:Mar 18 16:23:24 smithi039 podman[102192]: 2024-03-18 16:23:24.302597917 +0000 UTC m=+0.156666271 container create 9ac6e7576402d769a1a87577e6a225ca624bef175808c3a5f0294410c424a106 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-9-deactivate, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-18T16:23:24.990 INFO:journalctl@ceph.osd.10.smithi194.stdout:Mar 18 16:23:24 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-10[61903]: 2024-03-18T16:23:24.684+0000 7f86c63d5700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:22:01.305530+0000 front 2024-03-18T16:22:01.305781+0000 (oldest deadline 2024-03-18T16:23:05.405546+0000) 2024-03-18T16:23:24.991 INFO:journalctl@ceph.osd.10.smithi194.stdout:Mar 18 16:23:24 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-10[61903]: 2024-03-18T16:23:24.684+0000 7f86c63d5700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.194:6804 osd.1 since back 2024-03-18T16:22:10.106502+0000 front 2024-03-18T16:22:10.106530+0000 (oldest deadline 2024-03-18T16:23:14.206661+0000) 2024-03-18T16:23:25.052 INFO:journalctl@ceph.osd.9.smithi039.stdout:Mar 18 16:23:24 smithi039 podman[102192]: 2024-03-18 16:23:24.615260306 +0000 UTC m=+0.469328689 container init 9ac6e7576402d769a1a87577e6a225ca624bef175808c3a5f0294410c424a106 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-9-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-18T16:23:25.052 INFO:journalctl@ceph.osd.9.smithi039.stdout:Mar 18 16:23:24 smithi039 podman[102192]: 2024-03-18 16:23:24.627969031 +0000 UTC m=+0.482037380 container start 9ac6e7576402d769a1a87577e6a225ca624bef175808c3a5f0294410c424a106 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-9-deactivate, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T16:23:25.052 INFO:journalctl@ceph.osd.9.smithi039.stdout:Mar 18 16:23:24 smithi039 podman[102192]: 2024-03-18 16:23:24.705139014 +0000 UTC m=+0.559207450 container attach 9ac6e7576402d769a1a87577e6a225ca624bef175808c3a5f0294410c424a106 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-9-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-18T16:23:25.228 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 18 16:23:24 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-11[61670]: 2024-03-18T16:23:24.914+0000 7fb127c8b700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:21:59.309915+0000 front 2024-03-18T16:21:59.310166+0000 (oldest deadline 2024-03-18T16:23:04.609811+0000) 2024-03-18T16:23:25.228 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 18 16:23:24 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-11[61670]: 2024-03-18T16:23:24.914+0000 7fb127c8b700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-18T16:22:19.614338+0000 front 2024-03-18T16:22:19.614585+0000 (oldest deadline 2024-03-18T16:23:24.814982+0000) 2024-03-18T16:23:25.337 INFO:journalctl@ceph.osd.9.smithi039.stdout:Mar 18 16:23:25 smithi039 podman[102226]: 2024-03-18 16:23:25.138577923 +0000 UTC m=+0.033019398 container died 9ac6e7576402d769a1a87577e6a225ca624bef175808c3a5f0294410c424a106 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-9-deactivate, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-18T16:23:25.757 DEBUG:teuthology.orchestra.run.smithi039:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1892d8fc-e540-11ee-95c9-87774f69a715@osd.9.service' 2024-03-18T16:23:25.870 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T16:23:25.871 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-03-18T16:23:25.871 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-03-18T16:23:25.871 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ceph-1892d8fc-e540-11ee-95c9-87774f69a715@osd.10 2024-03-18T16:23:25.925 INFO:journalctl@ceph.osd.10.smithi194.stdout:Mar 18 16:23:25 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-10[61903]: 2024-03-18T16:23:25.660+0000 7f86c63d5700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:22:01.305530+0000 front 2024-03-18T16:22:01.305781+0000 (oldest deadline 2024-03-18T16:23:05.405546+0000) 2024-03-18T16:23:25.925 INFO:journalctl@ceph.osd.10.smithi194.stdout:Mar 18 16:23:25 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-10[61903]: 2024-03-18T16:23:25.661+0000 7f86c63d5700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.194:6804 osd.1 since back 2024-03-18T16:22:10.106502+0000 front 2024-03-18T16:22:10.106530+0000 (oldest deadline 2024-03-18T16:23:14.206661+0000) 2024-03-18T16:23:25.925 INFO:journalctl@ceph.osd.10.smithi194.stdout:Mar 18 16:23:25 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-10[61903]: 2024-03-18T16:23:25.661+0000 7f86c63d5700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-18T16:22:17.707799+0000 front 2024-03-18T16:22:17.707814+0000 (oldest deadline 2024-03-18T16:23:25.308612+0000) 2024-03-18T16:23:26.228 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 18 16:23:25 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-11[61670]: 2024-03-18T16:23:25.883+0000 7fb127c8b700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:21:59.309915+0000 front 2024-03-18T16:21:59.310166+0000 (oldest deadline 2024-03-18T16:23:04.609811+0000) 2024-03-18T16:23:26.229 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 18 16:23:25 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-11[61670]: 2024-03-18T16:23:25.883+0000 7fb127c8b700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-18T16:22:19.614338+0000 front 2024-03-18T16:22:19.614585+0000 (oldest deadline 2024-03-18T16:23:24.814982+0000) 2024-03-18T16:23:26.240 INFO:journalctl@ceph.osd.10.smithi194.stdout:Mar 18 16:23:25 smithi194 systemd[1]: Stopping Ceph osd.10 for 1892d8fc-e540-11ee-95c9-87774f69a715... 2024-03-18T16:23:26.709 INFO:journalctl@ceph.osd.10.smithi194.stdout:Mar 18 16:23:26 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-10[61903]: 2024-03-18T16:23:26.288+0000 7f86cdde9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T16:23:26.710 INFO:journalctl@ceph.osd.10.smithi194.stdout:Mar 18 16:23:26 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-10[61903]: 2024-03-18T16:23:26.288+0000 7f86cdde9700 -1 osd.10 97 *** Got signal Terminated *** 2024-03-18T16:23:26.710 INFO:journalctl@ceph.osd.10.smithi194.stdout:Mar 18 16:23:26 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-10[61903]: 2024-03-18T16:23:26.288+0000 7f86cdde9700 -1 osd.10 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T16:23:26.710 INFO:journalctl@ceph.osd.10.smithi194.stdout:Mar 18 16:23:26 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-10[61903]: 2024-03-18T16:23:26.707+0000 7f86c63d5700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:22:01.305530+0000 front 2024-03-18T16:22:01.305781+0000 (oldest deadline 2024-03-18T16:23:05.405546+0000) 2024-03-18T16:23:26.710 INFO:journalctl@ceph.osd.10.smithi194.stdout:Mar 18 16:23:26 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-10[61903]: 2024-03-18T16:23:26.707+0000 7f86c63d5700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.194:6804 osd.1 since back 2024-03-18T16:22:10.106502+0000 front 2024-03-18T16:22:10.106530+0000 (oldest deadline 2024-03-18T16:23:14.206661+0000) 2024-03-18T16:23:26.710 INFO:journalctl@ceph.osd.10.smithi194.stdout:Mar 18 16:23:26 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-10[61903]: 2024-03-18T16:23:26.707+0000 7f86c63d5700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-18T16:22:17.707799+0000 front 2024-03-18T16:22:17.707814+0000 (oldest deadline 2024-03-18T16:23:25.308612+0000) 2024-03-18T16:23:27.229 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 18 16:23:26 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-11[61670]: 2024-03-18T16:23:26.921+0000 7fb127c8b700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:21:59.309915+0000 front 2024-03-18T16:21:59.310166+0000 (oldest deadline 2024-03-18T16:23:04.609811+0000) 2024-03-18T16:23:27.229 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 18 16:23:26 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-11[61670]: 2024-03-18T16:23:26.921+0000 7fb127c8b700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-18T16:22:19.614338+0000 front 2024-03-18T16:22:19.614585+0000 (oldest deadline 2024-03-18T16:23:24.814982+0000) 2024-03-18T16:23:27.990 INFO:journalctl@ceph.osd.10.smithi194.stdout:Mar 18 16:23:27 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-10[61903]: 2024-03-18T16:23:27.669+0000 7f86c63d5700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:22:01.305530+0000 front 2024-03-18T16:22:01.305781+0000 (oldest deadline 2024-03-18T16:23:05.405546+0000) 2024-03-18T16:23:27.991 INFO:journalctl@ceph.osd.10.smithi194.stdout:Mar 18 16:23:27 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-10[61903]: 2024-03-18T16:23:27.669+0000 7f86c63d5700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.194:6804 osd.1 since back 2024-03-18T16:22:10.106502+0000 front 2024-03-18T16:22:10.106530+0000 (oldest deadline 2024-03-18T16:23:14.206661+0000) 2024-03-18T16:23:27.991 INFO:journalctl@ceph.osd.10.smithi194.stdout:Mar 18 16:23:27 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-10[61903]: 2024-03-18T16:23:27.669+0000 7f86c63d5700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-18T16:22:17.707799+0000 front 2024-03-18T16:22:17.707814+0000 (oldest deadline 2024-03-18T16:23:25.308612+0000) 2024-03-18T16:23:28.228 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 18 16:23:27 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-11[61670]: 2024-03-18T16:23:27.951+0000 7fb127c8b700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:21:59.309915+0000 front 2024-03-18T16:21:59.310166+0000 (oldest deadline 2024-03-18T16:23:04.609811+0000) 2024-03-18T16:23:28.229 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 18 16:23:27 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-11[61670]: 2024-03-18T16:23:27.951+0000 7fb127c8b700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-18T16:22:19.614338+0000 front 2024-03-18T16:22:19.614585+0000 (oldest deadline 2024-03-18T16:23:24.814982+0000) 2024-03-18T16:23:28.990 INFO:journalctl@ceph.osd.10.smithi194.stdout:Mar 18 16:23:28 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-10[61903]: 2024-03-18T16:23:28.667+0000 7f86c63d5700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:22:01.305530+0000 front 2024-03-18T16:22:01.305781+0000 (oldest deadline 2024-03-18T16:23:05.405546+0000) 2024-03-18T16:23:28.991 INFO:journalctl@ceph.osd.10.smithi194.stdout:Mar 18 16:23:28 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-10[61903]: 2024-03-18T16:23:28.667+0000 7f86c63d5700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.194:6804 osd.1 since back 2024-03-18T16:22:10.106502+0000 front 2024-03-18T16:22:10.106530+0000 (oldest deadline 2024-03-18T16:23:14.206661+0000) 2024-03-18T16:23:28.991 INFO:journalctl@ceph.osd.10.smithi194.stdout:Mar 18 16:23:28 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-10[61903]: 2024-03-18T16:23:28.667+0000 7f86c63d5700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-18T16:22:17.707799+0000 front 2024-03-18T16:22:17.707814+0000 (oldest deadline 2024-03-18T16:23:25.308612+0000) 2024-03-18T16:23:29.283 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 18 16:23:28 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-11[61670]: 2024-03-18T16:23:28.992+0000 7fb127c8b700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:21:59.309915+0000 front 2024-03-18T16:21:59.310166+0000 (oldest deadline 2024-03-18T16:23:04.609811+0000) 2024-03-18T16:23:29.283 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 18 16:23:28 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-11[61670]: 2024-03-18T16:23:28.992+0000 7fb127c8b700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-18T16:22:19.614338+0000 front 2024-03-18T16:22:19.614585+0000 (oldest deadline 2024-03-18T16:23:24.814982+0000) 2024-03-18T16:23:29.990 INFO:journalctl@ceph.osd.10.smithi194.stdout:Mar 18 16:23:29 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-10[61903]: 2024-03-18T16:23:29.711+0000 7f86c63d5700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:22:01.305530+0000 front 2024-03-18T16:22:01.305781+0000 (oldest deadline 2024-03-18T16:23:05.405546+0000) 2024-03-18T16:23:29.991 INFO:journalctl@ceph.osd.10.smithi194.stdout:Mar 18 16:23:29 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-10[61903]: 2024-03-18T16:23:29.711+0000 7f86c63d5700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.194:6804 osd.1 since back 2024-03-18T16:22:10.106502+0000 front 2024-03-18T16:22:10.106530+0000 (oldest deadline 2024-03-18T16:23:14.206661+0000) 2024-03-18T16:23:29.991 INFO:journalctl@ceph.osd.10.smithi194.stdout:Mar 18 16:23:29 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-10[61903]: 2024-03-18T16:23:29.711+0000 7f86c63d5700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-18T16:22:17.707799+0000 front 2024-03-18T16:22:17.707814+0000 (oldest deadline 2024-03-18T16:23:25.308612+0000) 2024-03-18T16:23:30.478 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 18 16:23:30 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-11[61670]: 2024-03-18T16:23:30.033+0000 7fb127c8b700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:21:59.309915+0000 front 2024-03-18T16:21:59.310166+0000 (oldest deadline 2024-03-18T16:23:04.609811+0000) 2024-03-18T16:23:30.478 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 18 16:23:30 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-11[61670]: 2024-03-18T16:23:30.033+0000 7fb127c8b700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-18T16:22:19.614338+0000 front 2024-03-18T16:22:19.614585+0000 (oldest deadline 2024-03-18T16:23:24.814982+0000) 2024-03-18T16:23:30.990 INFO:journalctl@ceph.osd.10.smithi194.stdout:Mar 18 16:23:30 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-10[61903]: 2024-03-18T16:23:30.699+0000 7f86c63d5700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:22:01.305530+0000 front 2024-03-18T16:22:01.305781+0000 (oldest deadline 2024-03-18T16:23:05.405546+0000) 2024-03-18T16:23:30.991 INFO:journalctl@ceph.osd.10.smithi194.stdout:Mar 18 16:23:30 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-10[61903]: 2024-03-18T16:23:30.699+0000 7f86c63d5700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.194:6804 osd.1 since back 2024-03-18T16:22:10.106502+0000 front 2024-03-18T16:22:10.106530+0000 (oldest deadline 2024-03-18T16:23:14.206661+0000) 2024-03-18T16:23:30.991 INFO:journalctl@ceph.osd.10.smithi194.stdout:Mar 18 16:23:30 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-10[61903]: 2024-03-18T16:23:30.699+0000 7f86c63d5700 -1 osd.10 97 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-18T16:22:17.707799+0000 front 2024-03-18T16:22:17.707814+0000 (oldest deadline 2024-03-18T16:23:25.308612+0000) 2024-03-18T16:23:31.478 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 18 16:23:31 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-11[61670]: 2024-03-18T16:23:31.027+0000 7fb127c8b700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:21:59.309915+0000 front 2024-03-18T16:21:59.310166+0000 (oldest deadline 2024-03-18T16:23:04.609811+0000) 2024-03-18T16:23:31.478 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 18 16:23:31 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-11[61670]: 2024-03-18T16:23:31.027+0000 7fb127c8b700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-18T16:22:19.614338+0000 front 2024-03-18T16:22:19.614585+0000 (oldest deadline 2024-03-18T16:23:24.814982+0000) 2024-03-18T16:23:31.991 INFO:journalctl@ceph.osd.10.smithi194.stdout:Mar 18 16:23:31 smithi194 podman[72875]: 2024-03-18 16:23:31.69988683 +0000 UTC m=+5.632983933 container died f5126ff7bc170402a3829cf19a0cef1031f79e3b0d9a1f883e883bdb824eea0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-10, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-18T16:23:32.350 INFO:journalctl@ceph.osd.10.smithi194.stdout:Mar 18 16:23:32 smithi194 podman[72875]: 2024-03-18 16:23:32.088166595 +0000 UTC m=+6.021263707 container cleanup f5126ff7bc170402a3829cf19a0cef1031f79e3b0d9a1f883e883bdb824eea0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-10, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-18T16:23:32.350 INFO:journalctl@ceph.osd.10.smithi194.stdout:Mar 18 16:23:32 smithi194 bash[72875]: ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-10 2024-03-18T16:23:32.478 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 18 16:23:31 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-11[61670]: 2024-03-18T16:23:31.996+0000 7fb127c8b700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:21:59.309915+0000 front 2024-03-18T16:21:59.310166+0000 (oldest deadline 2024-03-18T16:23:04.609811+0000) 2024-03-18T16:23:32.478 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 18 16:23:31 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-11[61670]: 2024-03-18T16:23:31.996+0000 7fb127c8b700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-18T16:22:19.614338+0000 front 2024-03-18T16:22:19.614585+0000 (oldest deadline 2024-03-18T16:23:24.814982+0000) 2024-03-18T16:23:32.666 INFO:journalctl@ceph.osd.10.smithi194.stdout:Mar 18 16:23:32 smithi194 podman[72905]: 2024-03-18 16:23:32.348006491 +0000 UTC m=+0.638737672 container remove f5126ff7bc170402a3829cf19a0cef1031f79e3b0d9a1f883e883bdb824eea0c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-10, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-18T16:23:32.961 INFO:journalctl@ceph.osd.10.smithi194.stdout:Mar 18 16:23:32 smithi194 podman[73000]: 2024-03-18 16:23:32.66526553 +0000 UTC m=+0.121547947 container create 97980be3250c3ecdc8adf12629282ee8c8063f14e5272aae6825234a8cde3e5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-10-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2) 2024-03-18T16:23:32.961 INFO:journalctl@ceph.osd.10.smithi194.stdout:Mar 18 16:23:32 smithi194 podman[73000]: 2024-03-18 16:23:32.561221552 +0000 UTC m=+0.017503986 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T16:23:32.961 INFO:journalctl@ceph.osd.10.smithi194.stdout:Mar 18 16:23:32 smithi194 podman[73000]: 2024-03-18 16:23:32.876622539 +0000 UTC m=+0.332904946 container init 97980be3250c3ecdc8adf12629282ee8c8063f14e5272aae6825234a8cde3e5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-10-deactivate, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240307) 2024-03-18T16:23:32.961 INFO:journalctl@ceph.osd.10.smithi194.stdout:Mar 18 16:23:32 smithi194 podman[73000]: 2024-03-18 16:23:32.879599299 +0000 UTC m=+0.335881716 container start 97980be3250c3ecdc8adf12629282ee8c8063f14e5272aae6825234a8cde3e5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-10-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-18T16:23:32.961 INFO:journalctl@ceph.osd.10.smithi194.stdout:Mar 18 16:23:32 smithi194 podman[73000]: 2024-03-18 16:23:32.959789513 +0000 UTC m=+0.416071950 container attach 97980be3250c3ecdc8adf12629282ee8c8063f14e5272aae6825234a8cde3e5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-10-deactivate, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T16:23:33.240 INFO:journalctl@ceph.osd.10.smithi194.stdout:Mar 18 16:23:33 smithi194 podman[73000]: 2024-03-18 16:23:33.16773028 +0000 UTC m=+0.624012698 container died 97980be3250c3ecdc8adf12629282ee8c8063f14e5272aae6825234a8cde3e5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-10-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-18T16:23:33.478 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 18 16:23:33 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-11[61670]: 2024-03-18T16:23:33.001+0000 7fb127c8b700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:21:59.309915+0000 front 2024-03-18T16:21:59.310166+0000 (oldest deadline 2024-03-18T16:23:04.609811+0000) 2024-03-18T16:23:33.479 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 18 16:23:33 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-11[61670]: 2024-03-18T16:23:33.001+0000 7fb127c8b700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-18T16:22:19.614338+0000 front 2024-03-18T16:22:19.614585+0000 (oldest deadline 2024-03-18T16:23:24.814982+0000) 2024-03-18T16:23:33.755 INFO:journalctl@ceph.osd.10.smithi194.stdout:Mar 18 16:23:33 smithi194 podman[73026]: 2024-03-18 16:23:33.7295127 +0000 UTC m=+0.556937610 container remove 97980be3250c3ecdc8adf12629282ee8c8063f14e5272aae6825234a8cde3e5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-10-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2) 2024-03-18T16:23:33.756 INFO:journalctl@ceph.osd.10.smithi194.stdout:Mar 18 16:23:33 smithi194 systemd[1]: ceph-1892d8fc-e540-11ee-95c9-87774f69a715@osd.10.service: Deactivated successfully. 2024-03-18T16:23:33.756 INFO:journalctl@ceph.osd.10.smithi194.stdout:Mar 18 16:23:33 smithi194 systemd[1]: Stopped Ceph osd.10 for 1892d8fc-e540-11ee-95c9-87774f69a715. 2024-03-18T16:23:33.756 INFO:journalctl@ceph.osd.10.smithi194.stdout:Mar 18 16:23:33 smithi194 systemd[1]: ceph-1892d8fc-e540-11ee-95c9-87774f69a715@osd.10.service: Consumed 3min 42.244s CPU time. 2024-03-18T16:23:33.758 DEBUG:teuthology.orchestra.run.smithi194:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1892d8fc-e540-11ee-95c9-87774f69a715@osd.10.service' 2024-03-18T16:23:33.799 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T16:23:33.799 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-03-18T16:23:33.799 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-03-18T16:23:33.799 DEBUG:teuthology.orchestra.run.smithi196:> sudo systemctl stop ceph-1892d8fc-e540-11ee-95c9-87774f69a715@osd.11 2024-03-18T16:23:34.123 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 18 16:23:33 smithi196 systemd[1]: Stopping Ceph osd.11 for 1892d8fc-e540-11ee-95c9-87774f69a715... 2024-03-18T16:23:34.123 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 18 16:23:33 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-11[61670]: 2024-03-18T16:23:33.957+0000 7fb127c8b700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:21:59.309915+0000 front 2024-03-18T16:21:59.310166+0000 (oldest deadline 2024-03-18T16:23:04.609811+0000) 2024-03-18T16:23:34.123 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 18 16:23:33 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-11[61670]: 2024-03-18T16:23:33.957+0000 7fb127c8b700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-18T16:22:19.614338+0000 front 2024-03-18T16:22:19.614585+0000 (oldest deadline 2024-03-18T16:23:24.814982+0000) 2024-03-18T16:23:34.123 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 18 16:23:33 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-11[61670]: 2024-03-18T16:23:33.957+0000 7fb127c8b700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-03-18T16:22:29.317362+0000 front 2024-03-18T16:22:29.317759+0000 (oldest deadline 2024-03-18T16:23:33.417236+0000) 2024-03-18T16:23:34.478 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 18 16:23:34 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-11[61670]: 2024-03-18T16:23:34.120+0000 7fb12f69f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T16:23:34.479 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 18 16:23:34 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-11[61670]: 2024-03-18T16:23:34.120+0000 7fb12f69f700 -1 osd.11 97 *** Got signal Terminated *** 2024-03-18T16:23:34.479 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 18 16:23:34 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-11[61670]: 2024-03-18T16:23:34.120+0000 7fb12f69f700 -1 osd.11 97 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-18T16:23:35.228 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 18 16:23:34 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-11[61670]: 2024-03-18T16:23:34.962+0000 7fb127c8b700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:21:59.309915+0000 front 2024-03-18T16:21:59.310166+0000 (oldest deadline 2024-03-18T16:23:04.609811+0000) 2024-03-18T16:23:35.228 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 18 16:23:34 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-11[61670]: 2024-03-18T16:23:34.962+0000 7fb127c8b700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-18T16:22:19.614338+0000 front 2024-03-18T16:22:19.614585+0000 (oldest deadline 2024-03-18T16:23:24.814982+0000) 2024-03-18T16:23:35.228 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 18 16:23:34 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-11[61670]: 2024-03-18T16:23:34.962+0000 7fb127c8b700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-03-18T16:22:29.317362+0000 front 2024-03-18T16:22:29.317759+0000 (oldest deadline 2024-03-18T16:23:33.417236+0000) 2024-03-18T16:23:36.478 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 18 16:23:35 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-11[61670]: 2024-03-18T16:23:35.980+0000 7fb127c8b700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:21:59.309915+0000 front 2024-03-18T16:21:59.310166+0000 (oldest deadline 2024-03-18T16:23:04.609811+0000) 2024-03-18T16:23:36.478 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 18 16:23:35 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-11[61670]: 2024-03-18T16:23:35.980+0000 7fb127c8b700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-18T16:22:19.614338+0000 front 2024-03-18T16:22:19.614585+0000 (oldest deadline 2024-03-18T16:23:24.814982+0000) 2024-03-18T16:23:36.478 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 18 16:23:35 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-11[61670]: 2024-03-18T16:23:35.980+0000 7fb127c8b700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-03-18T16:22:29.317362+0000 front 2024-03-18T16:22:29.317759+0000 (oldest deadline 2024-03-18T16:23:33.417236+0000) 2024-03-18T16:23:37.478 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 18 16:23:36 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-11[61670]: 2024-03-18T16:23:36.993+0000 7fb127c8b700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:21:59.309915+0000 front 2024-03-18T16:21:59.310166+0000 (oldest deadline 2024-03-18T16:23:04.609811+0000) 2024-03-18T16:23:37.479 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 18 16:23:36 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-11[61670]: 2024-03-18T16:23:36.993+0000 7fb127c8b700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-18T16:22:19.614338+0000 front 2024-03-18T16:22:19.614585+0000 (oldest deadline 2024-03-18T16:23:24.814982+0000) 2024-03-18T16:23:37.479 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 18 16:23:36 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-11[61670]: 2024-03-18T16:23:36.993+0000 7fb127c8b700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-03-18T16:22:29.317362+0000 front 2024-03-18T16:22:29.317759+0000 (oldest deadline 2024-03-18T16:23:33.417236+0000) 2024-03-18T16:23:38.478 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 18 16:23:37 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-11[61670]: 2024-03-18T16:23:37.987+0000 7fb127c8b700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:21:59.309915+0000 front 2024-03-18T16:21:59.310166+0000 (oldest deadline 2024-03-18T16:23:04.609811+0000) 2024-03-18T16:23:38.478 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 18 16:23:37 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-11[61670]: 2024-03-18T16:23:37.987+0000 7fb127c8b700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-18T16:22:19.614338+0000 front 2024-03-18T16:22:19.614585+0000 (oldest deadline 2024-03-18T16:23:24.814982+0000) 2024-03-18T16:23:38.478 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 18 16:23:37 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-11[61670]: 2024-03-18T16:23:37.987+0000 7fb127c8b700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-03-18T16:22:29.317362+0000 front 2024-03-18T16:22:29.317759+0000 (oldest deadline 2024-03-18T16:23:33.417236+0000) 2024-03-18T16:23:39.228 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 18 16:23:38 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-11[61670]: 2024-03-18T16:23:38.966+0000 7fb127c8b700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.39:6806 osd.0 since back 2024-03-18T16:21:59.309915+0000 front 2024-03-18T16:21:59.310166+0000 (oldest deadline 2024-03-18T16:23:04.609811+0000) 2024-03-18T16:23:39.228 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 18 16:23:38 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-11[61670]: 2024-03-18T16:23:38.966+0000 7fb127c8b700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.196:6804 osd.2 since back 2024-03-18T16:22:19.614338+0000 front 2024-03-18T16:22:19.614585+0000 (oldest deadline 2024-03-18T16:23:24.814982+0000) 2024-03-18T16:23:39.228 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 18 16:23:38 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-11[61670]: 2024-03-18T16:23:38.966+0000 7fb127c8b700 -1 osd.11 97 heartbeat_check: no reply from 172.21.15.39:6814 osd.3 since back 2024-03-18T16:22:29.317362+0000 front 2024-03-18T16:22:29.317759+0000 (oldest deadline 2024-03-18T16:23:33.417236+0000) 2024-03-18T16:23:39.856 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 18 16:23:39 smithi196 podman[72570]: 2024-03-18 16:23:39.588488736 +0000 UTC m=+5.596703277 container died c87aa051aa63a6e71b0e2301b9ebc9034a90ef7d557d4009d7bbc8010ecd9729 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-11, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-18T16:23:40.139 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 18 16:23:39 smithi196 podman[72570]: 2024-03-18 16:23:39.945423661 +0000 UTC m=+5.953638190 container cleanup c87aa051aa63a6e71b0e2301b9ebc9034a90ef7d557d4009d7bbc8010ecd9729 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-11, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5) 2024-03-18T16:23:40.139 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 18 16:23:39 smithi196 bash[72570]: ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-11 2024-03-18T16:23:40.478 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 18 16:23:40 smithi196 podman[72587]: 2024-03-18 16:23:40.138150379 +0000 UTC m=+0.542192427 container remove c87aa051aa63a6e71b0e2301b9ebc9034a90ef7d557d4009d7bbc8010ecd9729 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-11, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0) 2024-03-18T16:23:40.801 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 18 16:23:40 smithi196 podman[72677]: 2024-03-18 16:23:40.514120989 +0000 UTC m=+0.102864397 container create 0a22f0ed9ce586367793ec9de54a264116463e135ecf6aee7b7e41f1c8da5109 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-11-deactivate, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-18T16:23:40.801 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 18 16:23:40 smithi196 podman[72677]: 2024-03-18 16:23:40.42888788 +0000 UTC m=+0.017631309 image pull be86dc8cb22b274cd03b7f0beb3c2008b61e12c6812f73ca001a16ae92cb3dae quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513 2024-03-18T16:23:40.801 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 18 16:23:40 smithi196 podman[72677]: 2024-03-18 16:23:40.733700392 +0000 UTC m=+0.322443817 container init 0a22f0ed9ce586367793ec9de54a264116463e135ecf6aee7b7e41f1c8da5109 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-11-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-18T16:23:40.801 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 18 16:23:40 smithi196 podman[72677]: 2024-03-18 16:23:40.737005663 +0000 UTC m=+0.325749080 container start 0a22f0ed9ce586367793ec9de54a264116463e135ecf6aee7b7e41f1c8da5109 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-11-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-18T16:23:40.801 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 18 16:23:40 smithi196 podman[72677]: 2024-03-18 16:23:40.799784622 +0000 UTC m=+0.388528057 container attach 0a22f0ed9ce586367793ec9de54a264116463e135ecf6aee7b7e41f1c8da5109 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-11-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-18T16:23:41.228 INFO:journalctl@ceph.osd.11.smithi196.stdout:Mar 18 16:23:41 smithi196 podman[72677]: 2024-03-18 16:23:41.023327929 +0000 UTC m=+0.612071351 container died 0a22f0ed9ce586367793ec9de54a264116463e135ecf6aee7b7e41f1c8da5109 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-osd-11-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T16:23:41.611 DEBUG:teuthology.orchestra.run.smithi196:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1892d8fc-e540-11ee-95c9-87774f69a715@osd.11.service' 2024-03-18T16:23:41.651 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T16:23:41.651 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-03-18T16:23:41.652 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-03-18T16:23:41.652 DEBUG:teuthology.orchestra.run.smithi039:> sudo systemctl stop ceph-1892d8fc-e540-11ee-95c9-87774f69a715@mds.a 2024-03-18T16:23:42.087 INFO:journalctl@ceph.mds.a.smithi039.stdout:Mar 18 16:23:41 smithi039 systemd[1]: Stopping Ceph mds.a for 1892d8fc-e540-11ee-95c9-87774f69a715... 2024-03-18T16:23:42.587 INFO:journalctl@ceph.mds.a.smithi039.stdout:Mar 18 16:23:42 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-a[71287]: 2024-03-18T16:23:42.252+0000 7f5659800700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T16:23:42.587 INFO:journalctl@ceph.mds.a.smithi039.stdout:Mar 18 16:23:42 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-a[71287]: 2024-03-18T16:23:42.252+0000 7f5659800700 -1 mds.a *** got signal Terminated *** 2024-03-18T16:23:46.587 INFO:journalctl@ceph.mds.a.smithi039.stdout:Mar 18 16:23:46 smithi039 podman[102307]: 2024-03-18 16:23:46.287125116 +0000 UTC m=+4.167480541 container died 26e6fb6d37c490c0f3548d10178c69baef4828e6f3c7fde4e02aa5e1f466ba39 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-a, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD) 2024-03-18T16:23:46.970 INFO:journalctl@ceph.mds.a.smithi039.stdout:Mar 18 16:23:46 smithi039 podman[102307]: 2024-03-18 16:23:46.74368928 +0000 UTC m=+4.624044721 container cleanup 26e6fb6d37c490c0f3548d10178c69baef4828e6f3c7fde4e02aa5e1f466ba39 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240307) 2024-03-18T16:23:46.970 INFO:journalctl@ceph.mds.a.smithi039.stdout:Mar 18 16:23:46 smithi039 bash[102307]: ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-a 2024-03-18T16:23:47.337 INFO:journalctl@ceph.mds.a.smithi039.stdout:Mar 18 16:23:46 smithi039 podman[102321]: 2024-03-18 16:23:46.96817809 +0000 UTC m=+0.663232357 container remove 26e6fb6d37c490c0f3548d10178c69baef4828e6f3c7fde4e02aa5e1f466ba39 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-a, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS) 2024-03-18T16:23:47.509 DEBUG:teuthology.orchestra.run.smithi039:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1892d8fc-e540-11ee-95c9-87774f69a715@mds.a.service' 2024-03-18T16:23:47.640 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T16:23:47.640 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-03-18T16:23:47.640 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-03-18T16:23:47.640 DEBUG:teuthology.orchestra.run.smithi039:> sudo systemctl stop ceph-1892d8fc-e540-11ee-95c9-87774f69a715@mds.d 2024-03-18T16:23:48.087 INFO:journalctl@ceph.mds.d.smithi039.stdout:Mar 18 16:23:47 smithi039 systemd[1]: Stopping Ceph mds.d for 1892d8fc-e540-11ee-95c9-87774f69a715... 2024-03-18T16:23:48.587 INFO:journalctl@ceph.mds.d.smithi039.stdout:Mar 18 16:23:48 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-d[70266]: 2024-03-18T16:23:48.237+0000 7f3df4000700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T16:23:48.587 INFO:journalctl@ceph.mds.d.smithi039.stdout:Mar 18 16:23:48 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-d[70266]: 2024-03-18T16:23:48.237+0000 7f3df4000700 -1 mds.d *** got signal Terminated *** 2024-03-18T16:23:51.837 INFO:journalctl@ceph.mds.d.smithi039.stdout:Mar 18 16:23:51 smithi039 podman[102427]: 2024-03-18 16:23:51.539083643 +0000 UTC m=+3.420452618 container died 312ac9d705b0775801f310404d320dc8dd3396a702b0398dc7d4ac34500f8df6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-d, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-18T16:23:52.259 INFO:journalctl@ceph.mds.d.smithi039.stdout:Mar 18 16:23:52 smithi039 podman[102427]: 2024-03-18 16:23:52.064273085 +0000 UTC m=+3.945642073 container cleanup 312ac9d705b0775801f310404d320dc8dd3396a702b0398dc7d4ac34500f8df6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-d, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T16:23:52.259 INFO:journalctl@ceph.mds.d.smithi039.stdout:Mar 18 16:23:52 smithi039 bash[102427]: ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-d 2024-03-18T16:23:52.587 INFO:journalctl@ceph.mds.d.smithi039.stdout:Mar 18 16:23:52 smithi039 podman[102441]: 2024-03-18 16:23:52.257299604 +0000 UTC m=+0.707959082 container remove 312ac9d705b0775801f310404d320dc8dd3396a702b0398dc7d4ac34500f8df6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-d, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T16:23:52.765 DEBUG:teuthology.orchestra.run.smithi039:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1892d8fc-e540-11ee-95c9-87774f69a715@mds.d.service' 2024-03-18T16:23:52.882 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T16:23:52.882 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-03-18T16:23:52.882 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-03-18T16:23:52.882 DEBUG:teuthology.orchestra.run.smithi039:> sudo systemctl stop ceph-1892d8fc-e540-11ee-95c9-87774f69a715@mds.g 2024-03-18T16:23:53.087 INFO:journalctl@ceph.mds.g.smithi039.stdout:Mar 18 16:23:52 smithi039 systemd[1]: Stopping Ceph mds.g for 1892d8fc-e540-11ee-95c9-87774f69a715... 2024-03-18T16:23:53.837 INFO:journalctl@ceph.mds.g.smithi039.stdout:Mar 18 16:23:53 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-g[70920]: 2024-03-18T16:23:53.482+0000 7f36d5400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T16:23:53.837 INFO:journalctl@ceph.mds.g.smithi039.stdout:Mar 18 16:23:53 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-g[70920]: 2024-03-18T16:23:53.482+0000 7f36d5400700 -1 mds.g *** got signal Terminated *** 2024-03-18T16:23:54.809 INFO:journalctl@ceph.mds.g.smithi039.stdout:Mar 18 16:23:54 smithi039 podman[102549]: 2024-03-18 16:23:54.55742905 +0000 UTC m=+1.259387561 container died d8fdf2cc8f12929b85f928ed3fca3609551801f4285ccd1e7b3eaab8cab9fcdf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-g, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T16:23:55.087 INFO:journalctl@ceph.mds.g.smithi039.stdout:Mar 18 16:23:54 smithi039 podman[102549]: 2024-03-18 16:23:54.969154963 +0000 UTC m=+1.671113517 container cleanup d8fdf2cc8f12929b85f928ed3fca3609551801f4285ccd1e7b3eaab8cab9fcdf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-g, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-03-18T16:23:55.087 INFO:journalctl@ceph.mds.g.smithi039.stdout:Mar 18 16:23:54 smithi039 bash[102549]: ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-g 2024-03-18T16:23:55.587 INFO:journalctl@ceph.mds.g.smithi039.stdout:Mar 18 16:23:55 smithi039 podman[102563]: 2024-03-18 16:23:55.13692724 +0000 UTC m=+0.568624849 container remove d8fdf2cc8f12929b85f928ed3fca3609551801f4285ccd1e7b3eaab8cab9fcdf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-g, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T16:23:55.698 DEBUG:teuthology.orchestra.run.smithi039:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1892d8fc-e540-11ee-95c9-87774f69a715@mds.g.service' 2024-03-18T16:23:55.822 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T16:23:55.823 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-03-18T16:23:55.823 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-03-18T16:23:55.823 DEBUG:teuthology.orchestra.run.smithi039:> sudo systemctl stop ceph-1892d8fc-e540-11ee-95c9-87774f69a715@mds.j 2024-03-18T16:23:56.337 INFO:journalctl@ceph.mds.j.smithi039.stdout:Mar 18 16:23:55 smithi039 systemd[1]: Stopping Ceph mds.j for 1892d8fc-e540-11ee-95c9-87774f69a715... 2024-03-18T16:23:56.837 INFO:journalctl@ceph.mds.j.smithi039.stdout:Mar 18 16:23:56 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-j[69918]: 2024-03-18T16:23:56.495+0000 7f64a9a00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T16:23:56.837 INFO:journalctl@ceph.mds.j.smithi039.stdout:Mar 18 16:23:56 smithi039 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-j[69918]: 2024-03-18T16:23:56.495+0000 7f64a9a00700 -1 mds.j *** got signal Terminated *** 2024-03-18T16:23:59.841 INFO:journalctl@ceph.mds.j.smithi039.stdout:Mar 18 16:23:59 smithi039 podman[102669]: 2024-03-18 16:23:59.572214212 +0000 UTC m=+3.274137481 container died 035dc69f8af4d5f70d5c9118e036e407dff1166cc04312a2ed1143a20af63cc2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-j, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T16:24:00.286 INFO:journalctl@ceph.mds.j.smithi039.stdout:Mar 18 16:24:00 smithi039 podman[102669]: 2024-03-18 16:24:00.031794187 +0000 UTC m=+3.733717445 container cleanup 035dc69f8af4d5f70d5c9118e036e407dff1166cc04312a2ed1143a20af63cc2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-j, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T16:24:00.286 INFO:journalctl@ceph.mds.j.smithi039.stdout:Mar 18 16:24:00 smithi039 bash[102669]: ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-j 2024-03-18T16:24:00.587 INFO:journalctl@ceph.mds.j.smithi039.stdout:Mar 18 16:24:00 smithi039 podman[102683]: 2024-03-18 16:24:00.282585691 +0000 UTC m=+0.694641746 container remove 035dc69f8af4d5f70d5c9118e036e407dff1166cc04312a2ed1143a20af63cc2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-j, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T16:24:00.800 DEBUG:teuthology.orchestra.run.smithi039:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1892d8fc-e540-11ee-95c9-87774f69a715@mds.j.service' 2024-03-18T16:24:00.922 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T16:24:00.922 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-03-18T16:24:00.922 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-03-18T16:24:00.923 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ceph-1892d8fc-e540-11ee-95c9-87774f69a715@mds.b 2024-03-18T16:24:01.242 INFO:journalctl@ceph.mds.b.smithi194.stdout:Mar 18 16:24:00 smithi194 systemd[1]: Stopping Ceph mds.b for 1892d8fc-e540-11ee-95c9-87774f69a715... 2024-03-18T16:24:01.501 INFO:journalctl@ceph.mds.b.smithi194.stdout:Mar 18 16:24:01 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-b[67287]: 2024-03-18T16:24:01.295+0000 7f11c1df2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T16:24:01.502 INFO:journalctl@ceph.mds.b.smithi194.stdout:Mar 18 16:24:01 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-b[67287]: 2024-03-18T16:24:01.295+0000 7f11c1df2700 -1 mds.b *** got signal Terminated *** 2024-03-18T16:24:02.680 INFO:journalctl@ceph.mds.b.smithi194.stdout:Mar 18 16:24:02 smithi194 podman[73105]: 2024-03-18 16:24:02.393420797 +0000 UTC m=+1.283570109 container died 13068c0a3b14b351dd6554019e42de5779fda2bec182626b7bb1ce9e2c71edf4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-b, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True) 2024-03-18T16:24:02.680 INFO:journalctl@ceph.mds.b.smithi194.stdout:Mar 18 16:24:02 smithi194 podman[73105]: 2024-03-18 16:24:02.679039429 +0000 UTC m=+1.569188731 container cleanup 13068c0a3b14b351dd6554019e42de5779fda2bec182626b7bb1ce9e2c71edf4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-b, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-18T16:24:02.948 INFO:journalctl@ceph.mds.b.smithi194.stdout:Mar 18 16:24:02 smithi194 bash[73105]: ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-b 2024-03-18T16:24:03.094 DEBUG:teuthology.orchestra.run.smithi194:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1892d8fc-e540-11ee-95c9-87774f69a715@mds.b.service' 2024-03-18T16:24:03.133 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T16:24:03.133 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-03-18T16:24:03.133 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-03-18T16:24:03.133 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ceph-1892d8fc-e540-11ee-95c9-87774f69a715@mds.e 2024-03-18T16:24:03.490 INFO:journalctl@ceph.mds.e.smithi194.stdout:Mar 18 16:24:03 smithi194 systemd[1]: Stopping Ceph mds.e for 1892d8fc-e540-11ee-95c9-87774f69a715... 2024-03-18T16:24:03.491 INFO:journalctl@ceph.mds.e.smithi194.stdout:Mar 18 16:24:03 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-e[66675]: 2024-03-18T16:24:03.457+0000 7fd59b10c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T16:24:03.491 INFO:journalctl@ceph.mds.e.smithi194.stdout:Mar 18 16:24:03 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-e[66675]: 2024-03-18T16:24:03.457+0000 7fd59b10c700 -1 mds.e *** got signal Terminated *** 2024-03-18T16:24:06.685 INFO:journalctl@ceph.mds.e.smithi194.stdout:Mar 18 16:24:06 smithi194 podman[73239]: 2024-03-18 16:24:06.398930621 +0000 UTC m=+3.046618564 container died 315ebcc53fe14edef9440845eb72064d8ea0151204ec81fc0ba8aaf4e00f67ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-e, GIT_BRANCH=HEAD, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-18T16:24:06.990 INFO:journalctl@ceph.mds.e.smithi194.stdout:Mar 18 16:24:06 smithi194 podman[73239]: 2024-03-18 16:24:06.774130286 +0000 UTC m=+3.421818221 container cleanup 315ebcc53fe14edef9440845eb72064d8ea0151204ec81fc0ba8aaf4e00f67ec (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-e, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-18T16:24:06.990 INFO:journalctl@ceph.mds.e.smithi194.stdout:Mar 18 16:24:06 smithi194 bash[73239]: ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-e 2024-03-18T16:24:07.191 DEBUG:teuthology.orchestra.run.smithi194:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1892d8fc-e540-11ee-95c9-87774f69a715@mds.e.service' 2024-03-18T16:24:07.226 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T16:24:07.226 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-03-18T16:24:07.226 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-03-18T16:24:07.226 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ceph-1892d8fc-e540-11ee-95c9-87774f69a715@mds.h 2024-03-18T16:24:07.490 INFO:journalctl@ceph.mds.h.smithi194.stdout:Mar 18 16:24:07 smithi194 systemd[1]: Stopping Ceph mds.h for 1892d8fc-e540-11ee-95c9-87774f69a715... 2024-03-18T16:24:07.990 INFO:journalctl@ceph.mds.h.smithi194.stdout:Mar 18 16:24:07 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-h[66980]: 2024-03-18T16:24:07.551+0000 7efd9d542700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T16:24:07.990 INFO:journalctl@ceph.mds.h.smithi194.stdout:Mar 18 16:24:07 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-h[66980]: 2024-03-18T16:24:07.551+0000 7efd9d542700 -1 mds.h *** got signal Terminated *** 2024-03-18T16:24:10.740 INFO:journalctl@ceph.mds.h.smithi194.stdout:Mar 18 16:24:10 smithi194 podman[73360]: 2024-03-18 16:24:10.410432828 +0000 UTC m=+2.969349950 container died a4ae79a1fecdae374922290280fba25ba516e5bf5454f2790d249690e0ee6d11 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-h, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-18T16:24:11.346 INFO:journalctl@ceph.mds.h.smithi194.stdout:Mar 18 16:24:11 smithi194 podman[73360]: 2024-03-18 16:24:11.077495798 +0000 UTC m=+3.636412941 container cleanup a4ae79a1fecdae374922290280fba25ba516e5bf5454f2790d249690e0ee6d11 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-h, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS) 2024-03-18T16:24:11.347 INFO:journalctl@ceph.mds.h.smithi194.stdout:Mar 18 16:24:11 smithi194 bash[73360]: ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-h 2024-03-18T16:24:11.493 DEBUG:teuthology.orchestra.run.smithi194:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1892d8fc-e540-11ee-95c9-87774f69a715@mds.h.service' 2024-03-18T16:24:11.528 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T16:24:11.528 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-03-18T16:24:11.528 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-03-18T16:24:11.528 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ceph-1892d8fc-e540-11ee-95c9-87774f69a715@mds.k 2024-03-18T16:24:11.891 INFO:journalctl@ceph.mds.k.smithi194.stdout:Mar 18 16:24:11 smithi194 systemd[1]: Stopping Ceph mds.k for 1892d8fc-e540-11ee-95c9-87774f69a715... 2024-03-18T16:24:12.240 INFO:journalctl@ceph.mds.k.smithi194.stdout:Mar 18 16:24:11 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-k[66356]: 2024-03-18T16:24:11.888+0000 7f9043226700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T16:24:12.240 INFO:journalctl@ceph.mds.k.smithi194.stdout:Mar 18 16:24:11 smithi194 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-k[66356]: 2024-03-18T16:24:11.888+0000 7f9043226700 -1 mds.k *** got signal Terminated *** 2024-03-18T16:24:14.700 INFO:journalctl@ceph.mds.k.smithi194.stdout:Mar 18 16:24:14 smithi194 podman[73481]: 2024-03-18 16:24:14.407733606 +0000 UTC m=+2.669905241 container died e2b93397987b5b558f9b45764540063e06cd63ba9e482acef598b9dbb7819433 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-k, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b) 2024-03-18T16:24:14.990 INFO:journalctl@ceph.mds.k.smithi194.stdout:Mar 18 16:24:14 smithi194 podman[73481]: 2024-03-18 16:24:14.797603896 +0000 UTC m=+3.059775528 container cleanup e2b93397987b5b558f9b45764540063e06cd63ba9e482acef598b9dbb7819433 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-k, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-18T16:24:14.990 INFO:journalctl@ceph.mds.k.smithi194.stdout:Mar 18 16:24:14 smithi194 bash[73481]: ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-k 2024-03-18T16:24:15.126 DEBUG:teuthology.orchestra.run.smithi194:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1892d8fc-e540-11ee-95c9-87774f69a715@mds.k.service' 2024-03-18T16:24:15.159 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T16:24:15.160 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-03-18T16:24:15.160 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-03-18T16:24:15.160 DEBUG:teuthology.orchestra.run.smithi196:> sudo systemctl stop ceph-1892d8fc-e540-11ee-95c9-87774f69a715@mds.c 2024-03-18T16:24:15.469 INFO:journalctl@ceph.mds.c.smithi196.stdout:Mar 18 16:24:15 smithi196 systemd[1]: Stopping Ceph mds.c for 1892d8fc-e540-11ee-95c9-87774f69a715... 2024-03-18T16:24:15.728 INFO:journalctl@ceph.mds.c.smithi196.stdout:Mar 18 16:24:15 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-c[66685]: 2024-03-18T16:24:15.467+0000 7feeb21aa700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T16:24:15.728 INFO:journalctl@ceph.mds.c.smithi196.stdout:Mar 18 16:24:15 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-c[66685]: 2024-03-18T16:24:15.467+0000 7feeb21aa700 -1 mds.c *** got signal Terminated *** 2024-03-18T16:24:18.686 INFO:journalctl@ceph.mds.c.smithi196.stdout:Mar 18 16:24:18 smithi196 podman[72782]: 2024-03-18 16:24:18.400948792 +0000 UTC m=+3.044460664 container died 4a10438865ac82f5d005ddf2d90f1604f3e5471805c97ae5dac394aef7a40de5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-c, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307) 2024-03-18T16:24:18.978 INFO:journalctl@ceph.mds.c.smithi196.stdout:Mar 18 16:24:18 smithi196 podman[72782]: 2024-03-18 16:24:18.775793006 +0000 UTC m=+3.419304880 container cleanup 4a10438865ac82f5d005ddf2d90f1604f3e5471805c97ae5dac394aef7a40de5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-c, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-18T16:24:18.978 INFO:journalctl@ceph.mds.c.smithi196.stdout:Mar 18 16:24:18 smithi196 bash[72782]: ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-c 2024-03-18T16:24:19.145 DEBUG:teuthology.orchestra.run.smithi196:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1892d8fc-e540-11ee-95c9-87774f69a715@mds.c.service' 2024-03-18T16:24:19.184 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T16:24:19.184 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-03-18T16:24:19.184 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-03-18T16:24:19.184 DEBUG:teuthology.orchestra.run.smithi196:> sudo systemctl stop ceph-1892d8fc-e540-11ee-95c9-87774f69a715@mds.f 2024-03-18T16:24:19.728 INFO:journalctl@ceph.mds.f.smithi196.stdout:Mar 18 16:24:19 smithi196 systemd[1]: Stopping Ceph mds.f for 1892d8fc-e540-11ee-95c9-87774f69a715... 2024-03-18T16:24:19.729 INFO:journalctl@ceph.mds.f.smithi196.stdout:Mar 18 16:24:19 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-f[66376]: 2024-03-18T16:24:19.512+0000 7f12f32c2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T16:24:19.729 INFO:journalctl@ceph.mds.f.smithi196.stdout:Mar 18 16:24:19 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-f[66376]: 2024-03-18T16:24:19.512+0000 7f12f32c2700 -1 mds.f *** got signal Terminated *** 2024-03-18T16:24:22.681 INFO:journalctl@ceph.mds.f.smithi196.stdout:Mar 18 16:24:22 smithi196 podman[72914]: 2024-03-18 16:24:22.403514749 +0000 UTC m=+3.000233453 container died e19ea3e4220c92249765c0552b5daae2f25fd82c2d9c066eb3a22ef2ff308fc2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-f, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-18T16:24:22.948 INFO:journalctl@ceph.mds.f.smithi196.stdout:Mar 18 16:24:22 smithi196 podman[72914]: 2024-03-18 16:24:22.754405212 +0000 UTC m=+3.351123910 container cleanup e19ea3e4220c92249765c0552b5daae2f25fd82c2d9c066eb3a22ef2ff308fc2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-f, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-18T16:24:22.948 INFO:journalctl@ceph.mds.f.smithi196.stdout:Mar 18 16:24:22 smithi196 bash[72914]: ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-f 2024-03-18T16:24:23.173 DEBUG:teuthology.orchestra.run.smithi196:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1892d8fc-e540-11ee-95c9-87774f69a715@mds.f.service' 2024-03-18T16:24:23.198 INFO:journalctl@ceph.mds.f.smithi196.stdout:Mar 18 16:24:22 smithi196 podman[72929]: 2024-03-18 16:24:22.947018063 +0000 UTC m=+0.536179035 container remove e19ea3e4220c92249765c0552b5daae2f25fd82c2d9c066eb3a22ef2ff308fc2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-f, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-18T16:24:23.198 INFO:journalctl@ceph.mds.f.smithi196.stdout:Mar 18 16:24:23 smithi196 systemd[1]: ceph-1892d8fc-e540-11ee-95c9-87774f69a715@mds.f.service: Deactivated successfully. 2024-03-18T16:24:23.199 INFO:journalctl@ceph.mds.f.smithi196.stdout:Mar 18 16:24:23 smithi196 systemd[1]: Stopped Ceph mds.f for 1892d8fc-e540-11ee-95c9-87774f69a715. 2024-03-18T16:24:23.199 INFO:journalctl@ceph.mds.f.smithi196.stdout:Mar 18 16:24:23 smithi196 systemd[1]: ceph-1892d8fc-e540-11ee-95c9-87774f69a715@mds.f.service: Consumed 2.767s CPU time. 2024-03-18T16:24:23.208 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T16:24:23.209 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-03-18T16:24:23.209 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-03-18T16:24:23.209 DEBUG:teuthology.orchestra.run.smithi196:> sudo systemctl stop ceph-1892d8fc-e540-11ee-95c9-87774f69a715@mds.i 2024-03-18T16:24:23.478 INFO:journalctl@ceph.mds.i.smithi196.stdout:Mar 18 16:24:23 smithi196 systemd[1]: Stopping Ceph mds.i for 1892d8fc-e540-11ee-95c9-87774f69a715... 2024-03-18T16:24:23.978 INFO:journalctl@ceph.mds.i.smithi196.stdout:Mar 18 16:24:23 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-i[66987]: 2024-03-18T16:24:23.532+0000 7f7258481700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T16:24:23.978 INFO:journalctl@ceph.mds.i.smithi196.stdout:Mar 18 16:24:23 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-i[66987]: 2024-03-18T16:24:23.532+0000 7f7258481700 -1 mds.i *** got signal Terminated *** 2024-03-18T16:24:26.728 INFO:journalctl@ceph.mds.i.smithi196.stdout:Mar 18 16:24:26 smithi196 podman[73038]: 2024-03-18 16:24:26.392968367 +0000 UTC m=+2.972181115 container stop d4b43a4f0cf797248641000842847c04b69a1313be7ff0d780c8e3358420527d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-i, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-18T16:24:26.728 INFO:journalctl@ceph.mds.i.smithi196.stdout:Mar 18 16:24:26 smithi196 podman[73038]: 2024-03-18 16:24:26.418603971 +0000 UTC m=+2.997816723 container died d4b43a4f0cf797248641000842847c04b69a1313be7ff0d780c8e3358420527d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-i, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.build-date=20240307) 2024-03-18T16:24:27.044 INFO:journalctl@ceph.mds.i.smithi196.stdout:Mar 18 16:24:26 smithi196 podman[73038]: 2024-03-18 16:24:26.841648308 +0000 UTC m=+3.420861053 container cleanup d4b43a4f0cf797248641000842847c04b69a1313be7ff0d780c8e3358420527d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-i, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-18T16:24:27.044 INFO:journalctl@ceph.mds.i.smithi196.stdout:Mar 18 16:24:26 smithi196 bash[73038]: ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-i 2024-03-18T16:24:27.283 DEBUG:teuthology.orchestra.run.smithi196:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1892d8fc-e540-11ee-95c9-87774f69a715@mds.i.service' 2024-03-18T16:24:27.307 INFO:journalctl@ceph.mds.i.smithi196.stdout:Mar 18 16:24:27 smithi196 podman[73052]: 2024-03-18 16:24:27.042693422 +0000 UTC m=+0.639433498 container remove d4b43a4f0cf797248641000842847c04b69a1313be7ff0d780c8e3358420527d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-i, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240307) 2024-03-18T16:24:27.307 INFO:journalctl@ceph.mds.i.smithi196.stdout:Mar 18 16:24:27 smithi196 systemd[1]: ceph-1892d8fc-e540-11ee-95c9-87774f69a715@mds.i.service: Deactivated successfully. 2024-03-18T16:24:27.307 INFO:journalctl@ceph.mds.i.smithi196.stdout:Mar 18 16:24:27 smithi196 systemd[1]: Stopped Ceph mds.i for 1892d8fc-e540-11ee-95c9-87774f69a715. 2024-03-18T16:24:27.307 INFO:journalctl@ceph.mds.i.smithi196.stdout:Mar 18 16:24:27 smithi196 systemd[1]: ceph-1892d8fc-e540-11ee-95c9-87774f69a715@mds.i.service: Consumed 4min 43.850s CPU time. 2024-03-18T16:24:27.318 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T16:24:27.318 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-03-18T16:24:27.319 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-03-18T16:24:27.319 DEBUG:teuthology.orchestra.run.smithi196:> sudo systemctl stop ceph-1892d8fc-e540-11ee-95c9-87774f69a715@mds.l 2024-03-18T16:24:27.654 INFO:journalctl@ceph.mds.l.smithi196.stdout:Mar 18 16:24:27 smithi196 systemd[1]: Stopping Ceph mds.l for 1892d8fc-e540-11ee-95c9-87774f69a715... 2024-03-18T16:24:27.978 INFO:journalctl@ceph.mds.l.smithi196.stdout:Mar 18 16:24:27 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-l[66072]: 2024-03-18T16:24:27.652+0000 7fb6da928700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-18T16:24:27.978 INFO:journalctl@ceph.mds.l.smithi196.stdout:Mar 18 16:24:27 smithi196 ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-l[66072]: 2024-03-18T16:24:27.652+0000 7fb6da928700 -1 mds.l *** got signal Terminated *** 2024-03-18T16:24:30.680 INFO:journalctl@ceph.mds.l.smithi196.stdout:Mar 18 16:24:30 smithi196 podman[73159]: 2024-03-18 16:24:30.399556074 +0000 UTC m=+2.862671631 container died cefba4accc94891db27089414ac76a7264d6ff8c6e6a832f586d5fd739e89387 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-l, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-18T16:24:30.978 INFO:journalctl@ceph.mds.l.smithi196.stdout:Mar 18 16:24:30 smithi196 podman[73159]: 2024-03-18 16:24:30.761449113 +0000 UTC m=+3.224564672 container cleanup cefba4accc94891db27089414ac76a7264d6ff8c6e6a832f586d5fd739e89387 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f23dacf9be3936dccb4fcc712e538df27762e9b507b84b5cef1b0402d46f6513, name=ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-l, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-batrick-testing-20240314.183243-3d78f5b, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-18T16:24:30.979 INFO:journalctl@ceph.mds.l.smithi196.stdout:Mar 18 16:24:30 smithi196 bash[73159]: ceph-1892d8fc-e540-11ee-95c9-87774f69a715-mds-l 2024-03-18T16:24:31.145 DEBUG:teuthology.orchestra.run.smithi196:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1892d8fc-e540-11ee-95c9-87774f69a715@mds.l.service' 2024-03-18T16:24:31.190 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-18T16:24:31.190 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-03-18T16:24:31.191 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 --force --keep-logs 2024-03-18T16:24:31.539 INFO:teuthology.orchestra.run.smithi039.stdout:Deleting cluster with fsid: 1892d8fc-e540-11ee-95c9-87774f69a715 2024-03-18T16:24:36.682 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 --force --keep-logs 2024-03-18T16:24:36.890 INFO:teuthology.orchestra.run.smithi194.stdout:Deleting cluster with fsid: 1892d8fc-e540-11ee-95c9-87774f69a715 2024-03-18T16:24:39.425 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 --force --keep-logs 2024-03-18T16:24:39.631 INFO:teuthology.orchestra.run.smithi196.stdout:Deleting cluster with fsid: 1892d8fc-e540-11ee-95c9-87774f69a715 2024-03-18T16:24:42.077 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-18T16:24:42.189 DEBUG:teuthology.orchestra.run.smithi194:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-18T16:24:42.225 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-18T16:24:42.254 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-18T16:24:42.256 DEBUG:teuthology.misc:Transferring archived files from smithi039:/var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608907/remote/smithi039/crash 2024-03-18T16:24:42.257 DEBUG:teuthology.orchestra.run.smithi039:> sudo tar c -f - -C /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/crash -- . 2024-03-18T16:24:42.365 INFO:teuthology.orchestra.run.smithi039.stderr:tar: /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/crash: Cannot open: No such file or directory 2024-03-18T16:24:42.365 INFO:teuthology.orchestra.run.smithi039.stderr:tar: Error is not recoverable: exiting now 2024-03-18T16:24:42.371 DEBUG:teuthology.misc:Transferring archived files from smithi194:/var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608907/remote/smithi194/crash 2024-03-18T16:24:42.372 DEBUG:teuthology.orchestra.run.smithi194:> sudo tar c -f - -C /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/crash -- . 2024-03-18T16:24:42.416 INFO:teuthology.orchestra.run.smithi194.stderr:tar: /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/crash: Cannot open: No such file or directory 2024-03-18T16:24:42.416 INFO:teuthology.orchestra.run.smithi194.stderr:tar: Error is not recoverable: exiting now 2024-03-18T16:24:42.418 DEBUG:teuthology.misc:Transferring archived files from smithi196:/var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/crash to /home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608907/remote/smithi196/crash 2024-03-18T16:24:42.419 DEBUG:teuthology.orchestra.run.smithi196:> sudo tar c -f - -C /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/crash -- . 2024-03-18T16:24:42.454 INFO:teuthology.orchestra.run.smithi196.stderr:tar: /var/lib/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/crash: Cannot open: No such file or directory 2024-03-18T16:24:42.455 INFO:teuthology.orchestra.run.smithi196.stderr:tar: Error is not recoverable: exiting now 2024-03-18T16:24:42.456 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-18T16:24:42.456 DEBUG:teuthology.orchestra.run.smithi039:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'mons down, quorum' | egrep -v 'out of quorum' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-18T16:24:42.649 INFO:teuthology.orchestra.run.smithi039.stdout:2024-03-18T16:03:54.424048+0000 mon.a (mon.0) 681 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-03-18T16:24:42.652 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-18T16:24:42.652 DEBUG:teuthology.orchestra.run.smithi039:> sudo egrep '\[SEC\]' /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'mons down, quorum' | egrep -v 'out of quorum' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-18T16:24:42.770 DEBUG:teuthology.orchestra.run.smithi039:> sudo egrep '\[ERR\]' /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'mons down, quorum' | egrep -v 'out of quorum' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-18T16:24:42.866 DEBUG:teuthology.orchestra.run.smithi039:> sudo egrep '\[WRN\]' /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'mons down, quorum' | egrep -v 'out of quorum' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-18T16:24:42.958 INFO:teuthology.orchestra.run.smithi039.stdout:2024-03-18T16:03:54.424048+0000 mon.a (mon.0) 681 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-03-18T16:24:42.959 INFO:tasks.cephadm:Compressing logs... 2024-03-18T16:24:42.959 DEBUG:teuthology.orchestra.run.smithi039:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-18T16:24:42.963 DEBUG:teuthology.orchestra.run.smithi194:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-18T16:24:42.966 DEBUG:teuthology.orchestra.run.smithi196:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-18T16:24:43.022 INFO:teuthology.orchestra.run.smithi196.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-18T16:24:43.022 INFO:teuthology.orchestra.run.smithi196.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-18T16:24:43.022 INFO:teuthology.orchestra.run.smithi196.stderr:gzip -5 --verbose -- /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-mds.l.log 2024-03-18T16:24:43.022 INFO:teuthology.orchestra.run.smithi196.stderr:gzip -5 --verbose -- /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph.audit.log 2024-03-18T16:24:43.022 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-mds.l.log: gzip -5 --verbose -- /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-mds.i.log 2024-03-18T16:24:43.026 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-osd.11.log 2024-03-18T16:24:43.026 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-mgr.z.log 2024-03-18T16:24:43.026 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-osd.8.log 2024-03-18T16:24:43.026 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-osd.5.log 2024-03-18T16:24:43.032 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-mds.f.log 2024-03-18T16:24:43.032 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph.cephadm.log 2024-03-18T16:24:43.032 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph.log 2024-03-18T16:24:43.032 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-osd.2.log 2024-03-18T16:24:43.032 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-mds.c.log 2024-03-18T16:24:43.032 INFO:teuthology.orchestra.run.smithi196.stderr:gzip -5 --verbose -- /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-volume.log/var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-osd.2.log: 2024-03-18T16:24:43.032 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-mon.c.log 2024-03-18T16:24:43.032 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-volume.log: /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-mon.c.log: 92.9% -- replaced with /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-mgr.z.log.gz 2024-03-18T16:24:43.034 INFO:teuthology.orchestra.run.smithi194.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-18T16:24:43.035 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-18T16:24:43.035 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-osd.10.log 2024-03-18T16:24:43.035 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph.audit.log 2024-03-18T16:24:43.035 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-mds.b.log 2024-03-18T16:24:43.035 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-osd.4.log 2024-03-18T16:24:43.035 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-osd.1.log 2024-03-18T16:24:43.035 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-mds.k.log 2024-03-18T16:24:43.035 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-mds.h.log 2024-03-18T16:24:43.035 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-mds.k.log: gzip -5 --verbose -- /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-osd.7.log 2024-03-18T16:24:43.035 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-mds.e.log 2024-03-18T16:24:43.036 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-mgr.y.log 2024-03-18T16:24:43.036 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph.cephadm.log 2024-03-18T16:24:43.036 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph.log 2024-03-18T16:24:43.036 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-volume.log 2024-03-18T16:24:43.036 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-mon.b.log 2024-03-18T16:24:43.036 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-volume.log: /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-mon.b.log: 92.8% -- replaced with /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-mgr.y.log.gz 2024-03-18T16:24:43.051 INFO:teuthology.orchestra.run.smithi039.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-18T16:24:43.051 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-18T16:24:43.053 INFO:teuthology.orchestra.run.smithi039.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.log 2024-03-18T16:24:43.054 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-osd.6.log 2024-03-18T16:24:43.055 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/ceph-client.admin.log: gzip -5 --verbose -- /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-mon.a.log 2024-03-18T16:24:43.056 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph.audit.log 2024-03-18T16:24:43.057 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-mds.g.log 2024-03-18T16:24:43.058 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-mds.j.log 2024-03-18T16:24:43.059 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-osd.3.log 2024-03-18T16:24:43.060 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-mgr.x.log 2024-03-18T16:24:43.062 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-osd.0.log 2024-03-18T16:24:43.063 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph.cephadm.log 2024-03-18T16:24:43.064 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph.log 2024-03-18T16:24:43.064 INFO:teuthology.orchestra.run.smithi039.stderr: 94.5% -- replaced with /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-mds.j.log.gz 2024-03-18T16:24:43.065 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-volume.log 2024-03-18T16:24:43.066 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-osd.9.log 2024-03-18T16:24:43.067 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-mds.d.log 2024-03-18T16:24:43.068 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-mds.a.log 2024-03-18T16:24:43.069 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-mds.d.log: 87.5% -- replaced with /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph.log.gz 2024-03-18T16:24:43.072 INFO:teuthology.orchestra.run.smithi039.stderr:/var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-mds.a.log: 94.5% -- replaced with /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-mds.d.log.gz 2024-03-18T16:24:43.121 INFO:teuthology.orchestra.run.smithi039.stderr: 82.8% -- replaced with /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph.cephadm.log.gz 2024-03-18T16:24:43.163 INFO:teuthology.orchestra.run.smithi194.stderr: 82.4% -- replaced with /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph.cephadm.log.gz 2024-03-18T16:24:43.232 INFO:teuthology.orchestra.run.smithi196.stderr: 90.9% -- replaced with /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-volume.log.gz 2024-03-18T16:24:43.235 INFO:teuthology.orchestra.run.smithi194.stderr: 90.9% -- replaced with /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-volume.log.gz 2024-03-18T16:24:43.246 INFO:teuthology.orchestra.run.smithi039.stderr: 93.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-18T16:24:43.256 INFO:teuthology.orchestra.run.smithi196.stderr: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-18T16:24:43.259 INFO:teuthology.orchestra.run.smithi196.stderr: 82.3% -- replaced with /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph.cephadm.log.gz 2024-03-18T16:24:43.280 INFO:teuthology.orchestra.run.smithi039.stderr: 90.2% -- replaced with /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph.audit.log.gz 2024-03-18T16:24:43.327 INFO:teuthology.orchestra.run.smithi039.stderr: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.log.gz 2024-03-18T16:24:43.330 INFO:teuthology.orchestra.run.smithi039.stderr: 90.7% -- replaced with /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-volume.log.gz 2024-03-18T16:24:43.348 INFO:teuthology.orchestra.run.smithi194.stderr: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-18T16:24:43.375 INFO:teuthology.orchestra.run.smithi194.stderr: 90.3% -- replaced with /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph.audit.log.gz 2024-03-18T16:24:43.381 INFO:teuthology.orchestra.run.smithi194.stderr: 87.5% -- replaced with /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph.log.gz 2024-03-18T16:24:43.438 INFO:teuthology.orchestra.run.smithi196.stderr: 94.2% -- replaced with /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-mds.l.log.gz 2024-03-18T16:24:43.477 INFO:teuthology.orchestra.run.smithi194.stderr: 94.5% -- replaced with /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-mds.k.log.gz 2024-03-18T16:24:43.951 INFO:teuthology.orchestra.run.smithi039.stderr: 94.3% -- replaced with /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-mds.g.log.gz 2024-03-18T16:24:44.145 INFO:teuthology.orchestra.run.smithi196.stderr: 87.5% -- replaced with /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph.log.gz 2024-03-18T16:24:44.195 INFO:teuthology.orchestra.run.smithi196.stderr: 90.3% -- replaced with /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph.audit.log.gz 2024-03-18T16:24:44.250 INFO:teuthology.orchestra.run.smithi039.stderr: 90.2% -- replaced with /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-mgr.x.log.gz 2024-03-18T16:24:44.563 INFO:teuthology.orchestra.run.smithi194.stderr: 94.9% -- replaced with /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-mds.e.log.gz 2024-03-18T16:24:45.242 INFO:teuthology.orchestra.run.smithi196.stderr: 94.9% -- replaced with /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-mds.f.log.gz 2024-03-18T16:24:45.554 INFO:teuthology.orchestra.run.smithi194.stderr: 91.7% -- replaced with /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-mon.b.log.gz 2024-03-18T16:24:46.689 INFO:teuthology.orchestra.run.smithi196.stderr: 91.7% -- replaced with /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-mon.c.log.gz 2024-03-18T16:24:47.561 INFO:teuthology.orchestra.run.smithi196.stderr: 94.3% -- replaced with /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-mds.c.log.gz 2024-03-18T16:24:48.475 INFO:teuthology.orchestra.run.smithi194.stderr: 94.3% -- replaced with /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-mds.h.log.gz 2024-03-18T16:24:50.193 INFO:teuthology.orchestra.run.smithi039.stderr: 90.3% -- replaced with /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-mon.a.log.gz 2024-03-18T16:24:51.668 INFO:teuthology.orchestra.run.smithi039.stderr: 93.9% -- replaced with /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-mds.a.log.gz 2024-03-18T16:24:54.103 INFO:teuthology.orchestra.run.smithi196.stderr: 94.0% -- replaced with /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-mds.i.log.gz 2024-03-18T16:24:59.354 INFO:teuthology.orchestra.run.smithi194.stderr: 93.7% -- replaced with /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-mds.b.log.gz 2024-03-18T16:31:08.301 INFO:teuthology.orchestra.run.smithi194.stderr: 92.8% -- replaced with /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-osd.10.log.gz 2024-03-18T16:32:34.215 INFO:teuthology.orchestra.run.smithi196.stderr: 92.3% -- replaced with /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-osd.8.log.gz 2024-03-18T16:32:41.438 INFO:teuthology.orchestra.run.smithi194.stderr: 92.5% -- replaced with /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-osd.1.log.gz 2024-03-18T16:32:44.561 INFO:teuthology.orchestra.run.smithi196.stderr: 92.0% -- replaced with /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-osd.11.log.gz 2024-03-18T16:33:52.978 INFO:teuthology.orchestra.run.smithi194.stderr: 92.3% -- replaced with /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-osd.7.log.gz 2024-03-18T16:34:12.390 INFO:teuthology.orchestra.run.smithi194.stderr: 91.7% -- replaced with /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-osd.4.log.gz 2024-03-18T16:34:12.393 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-03-18T16:34:12.393 INFO:teuthology.orchestra.run.smithi194.stderr:real 9m29.413s 2024-03-18T16:34:12.393 INFO:teuthology.orchestra.run.smithi194.stderr:user 5m53.676s 2024-03-18T16:34:12.393 INFO:teuthology.orchestra.run.smithi194.stderr:sys 0m13.827s 2024-03-18T16:34:55.902 INFO:teuthology.orchestra.run.smithi196.stderr: 91.9% -- replaced with /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-osd.2.log.gz 2024-03-18T16:36:03.682 INFO:teuthology.orchestra.run.smithi196.stderr: 91.9% -- replaced with /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-osd.5.log.gz 2024-03-18T16:36:03.706 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-18T16:36:03.706 INFO:teuthology.orchestra.run.smithi196.stderr:real 11m20.717s 2024-03-18T16:36:03.706 INFO:teuthology.orchestra.run.smithi196.stderr:user 6m39.886s 2024-03-18T16:36:03.706 INFO:teuthology.orchestra.run.smithi196.stderr:sys 0m15.961s 2024-03-18T16:39:34.603 INFO:teuthology.orchestra.run.smithi039.stderr: 92.1% -- replaced with /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-osd.0.log.gz 2024-03-18T16:40:20.554 INFO:teuthology.orchestra.run.smithi039.stderr: 92.0% -- replaced with /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-osd.3.log.gz 2024-03-18T16:40:37.296 INFO:teuthology.orchestra.run.smithi039.stderr: 92.1% -- replaced with /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-osd.6.log.gz 2024-03-18T16:41:30.443 INFO:teuthology.orchestra.run.smithi039.stderr: 91.4% -- replaced with /var/log/ceph/1892d8fc-e540-11ee-95c9-87774f69a715/ceph-osd.9.log.gz 2024-03-18T16:41:30.447 INFO:teuthology.orchestra.run.smithi039.stderr: 2024-03-18T16:41:30.448 INFO:teuthology.orchestra.run.smithi039.stderr:real 16m47.453s 2024-03-18T16:41:30.448 INFO:teuthology.orchestra.run.smithi039.stderr:user 20m24.805s 2024-03-18T16:41:30.448 INFO:teuthology.orchestra.run.smithi039.stderr:sys 2m27.495s 2024-03-18T16:41:30.449 INFO:tasks.cephadm:Archiving logs... 2024-03-18T16:41:30.450 DEBUG:teuthology.misc:Transferring archived files from smithi039:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608907/remote/smithi039/log 2024-03-18T16:41:30.451 DEBUG:teuthology.orchestra.run.smithi039:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-18T16:42:46.287 DEBUG:teuthology.misc:Transferring archived files from smithi194:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608907/remote/smithi194/log 2024-03-18T16:42:46.288 DEBUG:teuthology.orchestra.run.smithi194:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-18T16:43:41.915 DEBUG:teuthology.misc:Transferring archived files from smithi196:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608907/remote/smithi196/log 2024-03-18T16:43:41.916 DEBUG:teuthology.orchestra.run.smithi196:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-18T16:44:48.117 INFO:tasks.cephadm:Removing cluster... 2024-03-18T16:44:48.118 DEBUG:teuthology.orchestra.run.smithi039:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 --force 2024-03-18T16:44:48.714 INFO:teuthology.orchestra.run.smithi039.stdout:Deleting cluster with fsid: 1892d8fc-e540-11ee-95c9-87774f69a715 2024-03-18T16:44:50.708 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 --force 2024-03-18T16:44:51.048 INFO:teuthology.orchestra.run.smithi194.stdout:Deleting cluster with fsid: 1892d8fc-e540-11ee-95c9-87774f69a715 2024-03-18T16:44:51.711 DEBUG:teuthology.orchestra.run.smithi196:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 1892d8fc-e540-11ee-95c9-87774f69a715 --force 2024-03-18T16:44:52.166 INFO:teuthology.orchestra.run.smithi196.stdout:Deleting cluster with fsid: 1892d8fc-e540-11ee-95c9-87774f69a715 2024-03-18T16:44:52.887 INFO:tasks.cephadm:Removing cephadm ... 2024-03-18T16:44:52.888 DEBUG:teuthology.orchestra.run.smithi039:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-18T16:44:52.948 DEBUG:teuthology.orchestra.run.smithi194:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-18T16:44:52.968 DEBUG:teuthology.orchestra.run.smithi196:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-18T16:44:52.983 INFO:tasks.cephadm:Teardown complete 2024-03-18T16:44:52.983 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-18T16:44:52.994 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-18T16:44:52.994 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-18T16:44:53.002 DEBUG:teuthology.orchestra.run.smithi194:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-18T16:44:53.011 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-18T16:44:53.203 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-18T16:44:53.214 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-18T16:44:53.214 DEBUG:teuthology.orchestra.run.smithi039:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-18T16:44:53.222 DEBUG:teuthology.orchestra.run.smithi194:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-18T16:44:53.224 DEBUG:teuthology.orchestra.run.smithi196:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-18T16:44:53.238 INFO:teuthology.orchestra.run.smithi194.stderr:bash: line 1: ntpq: command not found 2024-03-18T16:44:53.238 INFO:teuthology.orchestra.run.smithi196.stderr:bash: line 1: ntpq: command not found 2024-03-18T16:44:53.271 INFO:teuthology.orchestra.run.smithi039.stderr:bash: line 1: ntpq: command not found 2024-03-18T16:44:53.352 INFO:teuthology.orchestra.run.smithi196.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-18T16:44:53.353 INFO:teuthology.orchestra.run.smithi196.stdout:=============================================================================== 2024-03-18T16:44:53.353 INFO:teuthology.orchestra.run.smithi196.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 8 +2643us[+2643us] +/- 92ms 2024-03-18T16:44:53.353 INFO:teuthology.orchestra.run.smithi196.stdout:^* hv02.front.sepia.ceph.com 3 6 377 9 -1204us[-1172us] +/- 58ms 2024-03-18T16:44:53.353 INFO:teuthology.orchestra.run.smithi196.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 8 -568us[ -568us] +/- 85ms 2024-03-18T16:44:53.353 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T16:44:53.358 INFO:teuthology.orchestra.run.smithi194.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-18T16:44:53.358 INFO:teuthology.orchestra.run.smithi194.stdout:=============================================================================== 2024-03-18T16:44:53.358 INFO:teuthology.orchestra.run.smithi194.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 15 +2647us[+2654us] +/- 92ms 2024-03-18T16:44:53.359 INFO:teuthology.orchestra.run.smithi194.stdout:^* hv02.front.sepia.ceph.com 3 6 377 14 -1209us[-1202us] +/- 58ms 2024-03-18T16:44:53.359 INFO:teuthology.orchestra.run.smithi194.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 14 -569us[ -562us] +/- 84ms 2024-03-18T16:44:53.359 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T16:44:53.382 INFO:teuthology.orchestra.run.smithi039.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-18T16:44:53.382 INFO:teuthology.orchestra.run.smithi039.stdout:=============================================================================== 2024-03-18T16:44:53.382 INFO:teuthology.orchestra.run.smithi039.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 8 +2609us[+2609us] +/- 92ms 2024-03-18T16:44:53.382 INFO:teuthology.orchestra.run.smithi039.stdout:^* hv02.front.sepia.ceph.com 3 6 377 11 -1242us[-1266us] +/- 58ms 2024-03-18T16:44:53.382 INFO:teuthology.orchestra.run.smithi039.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 10 -577us[ -577us] +/- 85ms 2024-03-18T16:44:53.382 INFO:teuthology.orchestra.run.smithi039.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-18T16:44:53.385 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-18T16:44:53.396 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-18T16:44:53.396 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-18T16:44:53.415 DEBUG:teuthology.orchestra.run.smithi039:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-18T16:44:53.714 DEBUG:teuthology.orchestra.run.smithi194:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-18T16:44:53.814 DEBUG:teuthology.orchestra.run.smithi196:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-18T16:44:53.917 DEBUG:teuthology.orchestra.run.smithi039:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-18T16:44:54.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T16:44:54.029 DEBUG:teuthology.orchestra.run.smithi194:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-18T16:44:54.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T16:44:54.063 DEBUG:teuthology.orchestra.run.smithi196:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-18T16:44:54.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T16:44:54.094 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-18T16:44:54.105 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-18T16:44:54.145 INFO:teuthology.task.internal:Duration was 3423.716950 seconds 2024-03-18T16:44:54.145 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-18T16:44:54.166 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-18T16:44:54.166 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-18T16:44:54.174 DEBUG:teuthology.orchestra.run.smithi194:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-18T16:44:54.176 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-18T16:44:54.234 INFO:teuthology.orchestra.run.smithi194.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-18T16:44:54.238 INFO:teuthology.orchestra.run.smithi196.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-18T16:44:54.367 INFO:teuthology.orchestra.run.smithi039.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-18T16:44:54.703 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-18T16:44:54.703 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi039.front.sepia.ceph.com 2024-03-18T16:44:54.704 DEBUG:teuthology.orchestra.run.smithi039:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-18T16:44:54.771 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi194.front.sepia.ceph.com 2024-03-18T16:44:54.771 DEBUG:teuthology.orchestra.run.smithi194:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-18T16:44:54.799 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi196.front.sepia.ceph.com 2024-03-18T16:44:54.799 DEBUG:teuthology.orchestra.run.smithi196:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-18T16:44:54.823 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-18T16:44:54.823 DEBUG:teuthology.orchestra.run.smithi039:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-18T16:44:54.831 DEBUG:teuthology.orchestra.run.smithi194:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-18T16:44:54.842 DEBUG:teuthology.orchestra.run.smithi196:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-18T16:44:54.938 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-18T16:44:54.938 DEBUG:teuthology.orchestra.run.smithi039:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-18T16:44:54.944 DEBUG:teuthology.orchestra.run.smithi194:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-18T16:44:54.946 DEBUG:teuthology.orchestra.run.smithi196:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-18T16:44:55.476 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-18T16:44:55.487 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-18T16:44:55.487 DEBUG:teuthology.orchestra.run.smithi039:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-18T16:44:55.616 DEBUG:teuthology.orchestra.run.smithi194:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-18T16:44:55.666 DEBUG:teuthology.orchestra.run.smithi196:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-18T16:44:55.695 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-18T16:44:55.708 DEBUG:teuthology.orchestra.run.smithi039:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-18T16:44:55.716 DEBUG:teuthology.orchestra.run.smithi194:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-18T16:44:55.718 DEBUG:teuthology.orchestra.run.smithi196:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-18T16:44:55.755 INFO:teuthology.orchestra.run.smithi194.stdout:kernel.core_pattern = core 2024-03-18T16:44:55.781 INFO:teuthology.orchestra.run.smithi196.stdout:kernel.core_pattern = core 2024-03-18T16:44:55.840 INFO:teuthology.orchestra.run.smithi039.stdout:kernel.core_pattern = core 2024-03-18T16:44:55.907 DEBUG:teuthology.orchestra.run.smithi039:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-18T16:44:55.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T16:44:55.959 DEBUG:teuthology.orchestra.run.smithi194:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-18T16:44:55.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T16:44:55.973 DEBUG:teuthology.orchestra.run.smithi196:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-18T16:44:55.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-18T16:44:55.987 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-18T16:44:55.998 INFO:teuthology.task.internal:Transferring archived files... 2024-03-18T16:44:55.999 DEBUG:teuthology.misc:Transferring archived files from smithi039:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608907/remote/smithi039 2024-03-18T16:44:55.999 DEBUG:teuthology.orchestra.run.smithi039:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-18T16:44:56.117 DEBUG:teuthology.misc:Transferring archived files from smithi194:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608907/remote/smithi194 2024-03-18T16:44:56.117 DEBUG:teuthology.orchestra.run.smithi194:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-18T16:44:56.156 DEBUG:teuthology.misc:Transferring archived files from smithi196:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608907/remote/smithi196 2024-03-18T16:44:56.156 DEBUG:teuthology.orchestra.run.smithi196:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-18T16:44:56.194 INFO:teuthology.task.internal:Removing archive directory... 2024-03-18T16:44:56.194 DEBUG:teuthology.orchestra.run.smithi039:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-18T16:44:56.202 DEBUG:teuthology.orchestra.run.smithi194:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-18T16:44:56.205 DEBUG:teuthology.orchestra.run.smithi196:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-18T16:44:56.255 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-18T16:44:56.269 INFO:teuthology.task.internal:Not uploading archives. 2024-03-18T16:44:56.269 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-18T16:44:56.279 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-18T16:44:56.280 DEBUG:teuthology.orchestra.run.smithi039:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-18T16:44:56.287 DEBUG:teuthology.orchestra.run.smithi194:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-18T16:44:56.289 DEBUG:teuthology.orchestra.run.smithi196:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-18T16:44:56.302 INFO:teuthology.orchestra.run.smithi194.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 18 16:44 /home/ubuntu/cephtest 2024-03-18T16:44:56.303 INFO:teuthology.orchestra.run.smithi196.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 18 16:44 /home/ubuntu/cephtest 2024-03-18T16:44:56.336 INFO:teuthology.orchestra.run.smithi039.stdout: 393229 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 18 16:44 /home/ubuntu/cephtest 2024-03-18T16:44:56.340 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-18T16:44:56.351 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-18T16:44:56.383 INFO:teuthology.nuke:Checking targets against current locks 2024-03-18T16:44:56.409 DEBUG:teuthology.nuke:shortname: smithi039 2024-03-18T16:44:56.409 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-18T16:44:56.437 DEBUG:teuthology.nuke:shortname: smithi194 2024-03-18T16:44:56.437 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-18T16:44:56.442 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi039.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608907', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-18 15:36:02.329550', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHY1MDDrpKaPt02pg1LDMFSgGmc2dWtanXhMrgFGTcvqcQ0jx/3Sv8LUP0LN1UAcD79+qY1LEySKgfl2VsGvS4o='} 2024-03-18T16:44:56.459 DEBUG:teuthology.nuke:shortname: smithi196 2024-03-18T16:44:56.459 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-18T16:44:56.466 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi194.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608907', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-18 15:36:02.332011', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBfF0Vvutt5ltBy/buw4FS4VynvDztlWx3R/4TOA1t6sFlVqtej5vckm1rwGO6Aahuqy0ZWUiRVT+0nF1HTpNvM='} 2024-03-18T16:44:56.479 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi196.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-18_14:19:46-fs-wip-batrick-testing-20240314.183243-distro-default-smithi/7608907', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-18 15:36:02.333123', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKcVJ3bbZUAz5uSEqaEYjMFAq4WHt9Vv/N04A/u1+3Jbn4w7pTCkeOmOXuHBfzVxz6J1WmPK7gwbfF8ZPEPLLSA='} 2024-03-18T16:44:56.490 INFO:teuthology.orchestra.console.smithi039:Power off 2024-03-18T16:44:56.490 DEBUG:teuthology.orchestra.console.smithi039:pexpect command: ipmitool -H smithi039.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-18T16:44:56.514 DEBUG:teuthology.orchestra.console.smithi039:power off output: Chassis Power Control: Down/Off 2024-03-18T16:44:56.514 DEBUG:teuthology.orchestra.console.smithi039:pexpect command: ipmitool -H smithi039.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T16:44:56.531 INFO:teuthology.orchestra.console.smithi194:Power off 2024-03-18T16:44:56.531 DEBUG:teuthology.orchestra.console.smithi194:pexpect command: ipmitool -H smithi194.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-18T16:44:56.543 INFO:teuthology.orchestra.console.smithi196:Power off 2024-03-18T16:44:56.544 DEBUG:teuthology.orchestra.console.smithi196:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-18T16:44:56.552 DEBUG:teuthology.orchestra.console.smithi039:check power output: Chassis Power is on 2024-03-18T16:44:56.553 DEBUG:teuthology.orchestra.console.smithi194:power off output: Chassis Power Control: Down/Off 2024-03-18T16:44:56.553 DEBUG:teuthology.orchestra.console.smithi194:pexpect command: ipmitool -H smithi194.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T16:44:56.565 DEBUG:teuthology.orchestra.console.smithi196:power off output: Chassis Power Control: Down/Off 2024-03-18T16:44:56.565 DEBUG:teuthology.orchestra.console.smithi196:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T16:44:56.574 DEBUG:teuthology.orchestra.console.smithi194:check power output: Chassis Power is on 2024-03-18T16:44:56.587 DEBUG:teuthology.orchestra.console.smithi196:check power output: Chassis Power is on 2024-03-18T16:45:00.553 DEBUG:teuthology.orchestra.console.smithi039:pexpect command: ipmitool -H smithi039.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T16:45:00.575 DEBUG:teuthology.orchestra.console.smithi194:pexpect command: ipmitool -H smithi194.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T16:45:00.588 DEBUG:teuthology.orchestra.console.smithi196:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T16:45:00.664 DEBUG:teuthology.orchestra.console.smithi039:check power output: Chassis Power is on 2024-03-18T16:45:00.685 DEBUG:teuthology.orchestra.console.smithi194:check power output: Chassis Power is on 2024-03-18T16:45:00.696 DEBUG:teuthology.orchestra.console.smithi196:check power output: Chassis Power is on 2024-03-18T16:45:04.669 DEBUG:teuthology.orchestra.console.smithi039:pexpect command: ipmitool -H smithi039.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T16:45:04.686 DEBUG:teuthology.orchestra.console.smithi194:pexpect command: ipmitool -H smithi194.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T16:45:04.697 DEBUG:teuthology.orchestra.console.smithi196:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-18T16:45:04.781 DEBUG:teuthology.orchestra.console.smithi039:check power output: Chassis Power is off 2024-03-18T16:45:04.794 DEBUG:teuthology.orchestra.console.smithi194:check power output: Chassis Power is off 2024-03-18T16:45:04.807 DEBUG:teuthology.orchestra.console.smithi196:check power output: Chassis Power is off 2024-03-18T16:45:04.883 INFO:teuthology.orchestra.console.smithi039:Power off completed 2024-03-18T16:45:04.894 INFO:teuthology.orchestra.console.smithi194:Power off completed 2024-03-18T16:45:04.908 INFO:teuthology.orchestra.console.smithi196:Power off completed 2024-03-18T16:45:05.013 INFO:teuthology.run:Summary data: client.0-kernel-sha1: 3845b7ac715a3ff198582638d199a90e30e254da description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/yes} objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/no 5-workunit/suites/ffsb}} duration: 3423.716949939728 failure_reason: '"2024-03-18T16:03:54.424048+0000 mon.a (mon.0) 681 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY)" in cluster log' flavor: default owner: scheduled_pdonnell@teuthology success: false 2024-03-18T16:45:05.013 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-18T16:45:05.099 INFO:teuthology.run:FAIL